- CASE-CONTROL STUDY
- Overview
- A case-control study is an observational study where researchers select a number of subjects with an outcome of interest (cases), and a number of subjects without the outcome of interest (controls). The prevalence of exposure to a risk factor is then compared between the two groups.
- If the cases have a much higher rate of exposure to the risk factor than the controls, it may mean that the outcome is related to the risk factor
- Example:
- To evaluate if cell phones cause brain cancer, researchers select two groups of subjects
- Case group : 100 people with brain cancer
- Control group: 100 people without brain cancer
- Researchers then compare cell phone use between the two groups
- If cell phone use is much higher in the case group then there may be a link between brain cancer and cell phone use
- Much easier and cheaper to perform than a randomized controlled trial
- With modern medical databases, a large number of subjects can be evaluated with very little effort
- Outcomes with a low incidence can be evaluated. This is typically not feasible with a randomized controlled trial.
- Exposures that are not ethical in a randomized controlled trial can be evaluated. For example, no randomized controlled trial is going to randomize patients to smoking or known carcinogens.
- Does not prove causality
- Data for patient variables may be lacking or missing and therefore must be estimated
- Difficult or impossible to control for confounding
- Patients are not randomized so there is no way to control for unmeasured covariates and hidden bias
- COHORT STUDY
- Overview
- A cohort study is an observational study that is used to evaluate whether an exposure is associated with a particular outcome
- In a cohort study, two groups of subjects are formed based on exposure to a factor. One group has been exposed to a factor, and the other group has not been exposed to the factor.
- For other variables (ex. age, sex, race, etc.), the groups are matched as closely as possible
- Neither group has the outcome at the beginning of the study
- The two groups are followed for a length of time, and the incidence of the outcome is compared between the two groups
- A higher incidence in the exposed group may mean a link exists between the factor and the outcome
- Example:
- Researchers want to evaluate if smoking causes bladder cancer
- They identify 2 cohorts of people:
- Exposed cohort: 100 smokers
- Control cohort: 100 nonsmokers
- They follow the two cohorts for a number of years and compare the incidence of bladder cancer between them
- Much easier and cheaper to perform than a randomized controlled trial
- With modern medical databases, a large number of subjects can be evaluated with very little effort
- Outcomes with a low incidence can be evaluated. This is typically not feasible with a randomized controlled trial.
- Exposures that are not ethical in a randomized controlled trial can be evaluated. For example, no randomized controlled trial is going to randomize patients to smoking or known carcinogens.
- Does not prove causality
- Data for patient variables may be lacking or missing and therefore must be estimated
- Difficult or impossible to control for confounding
- Patients are not randomized so there is no way to control for unmeasured covariates and hidden bias
- META-ANALYSIS
- Overview
- A meta-analysis is a statistical study where the results from a number of related studies are pooled together and analyzed in order to draw conclusions from a broader set of data
- A meta-analysis may pool the results of cohort studies, case-control studies, randomized controlled trials, or a combination of the three. Cohort and Case-control studies may be combined in a meta-analysis, but it is not common (nor recommended) to combine randomized controlled trials with other types of studies
- Often times, different studies on the same condition will come to different conclusions. A meta-analysis is a way to combine the results of these studies so that an overall effect can be estimated.
- If a trial has a small number of subjects, it may not be powerful enough to find a statistically significant result. A meta-analysis allows small trials to be pooled together so that they have sufficient power to find a significant effect if one exists.
- The overall effect found in meta-analyses is often less than what is seen in individual trials. When done properly, the effect in the meta-analysis is more likely to represent the true effect in a large population of patients.
- Studies differ in length, design, intervention, population characteristics, and outcome measures. This can make it difficult to determine which studies to include.
- Combining studies that differ in design may lead to conclusions that are invalid
- Meta-analyses are subject to publication bias. Publication bias occurs when only studies that found a significant effect or a large effect were published.
- Meta-analyses are only as good as the studies they evaluate. If there are not enough good studies to perform a valid meta-analysis, the authors should be willing to concede this and abandon the meta-analysis.
- It can be argued that the results from a large, randomized controlled trial are more valid than the results of a meta-analysis on the same topic
- NESTED CASE-CONTROL STUDY
- Overview
- A nested case-control study is a case-control study where the subjects for the study are sampled from a cohort of patients that has been observed for a period of time. The study is said to be "nested" within the cohort.
- Researchers form two groups from the cohort of patients. One group has the outcome of interest (cases), and one group has not developed the outcome (controls). Exposure to a risk factor can then be compared between the two groups. If the cases have a significantly greater exposure to the risk factor, then the risk factor may be related to the outcome.
- When selecting the control group in a nested case-control study, the researchers typically select 1 - 4 controls per case as opposed to comparing cases to the entire group of controls. Controls are selected that match the cases on a number of measured variables (ex. age, medical conditions, sex, etc). This helps to control for confounding.
- Example:
- Researchers have observed a cohort of 100,000 women for 10 years
- At the start of the study, all women gave blood samples
- Over the course of the 10 years, 2000 women developed breast cancer
- The researchers are interested in finding out if there is a link between mercury exposure and breast cancer
- The researchers decide to test the blood samples for mercury levels and compare rates of breast cancer between women with high levels and those with normal levels
- Testing 100,000 blood samples for mercury levels would cost about $2 million dollars
- To save money, the researchers decide to do a nested case-control study where they select 2 controls for every case. Controls are selected that match the cases for a number of variables. This way they only have to test 6000 blood samples (2000 cases and 4000 controls).
- In most cases, the nested case-control study is much cheaper to perform than a full case-cohort study
- By selecting controls that match the cases on a number of covariates, there is some control for confounding and statistical efficiency should be improved
- Because the entire cohort is not used (for the controls), some precision is lost in the statistical measures, and the confidence intervals for the study outcomes will be wider
- Like regular case-control studies, nested case-control studies are observational studies and have all the same limitations
- OBSERVATIONAL STUDY
- Overview
- An observational study is a study where patients and outcomes are observed and no intervention is applied by the investigators. In observational studies, groups of subjects are formed according to specific exposures (cohort study) or according to specific outcomes (case-control study). This is in contrast to a randomized controlled trial where subjects are randomly assigned to a specific group and then followed.
- Observational studies may be prospective (groups are identified and then observed over time) or retrospective (groups are identified and outcomes are compared)
- Much easier and cheaper to perform than a randomized controlled trial
- With modern medical databases, a large number of subjects can be evaluated with very little effort
- Outcomes with a low incidence can be evaluated. This is typically not feasible with a randomized controlled trial.
- Exposures that are not ethical in a randomized controlled trial can be evaluated. For example, no randomized controlled trial is going to randomize patients to smoking or known carcinogens.
- The main disadvantage of observational studies is that there is no way to control for hidden bias and unmeasured covariates. In randomized controlled trials, the process of randomization helps to alleviate the effect of these issues.
- Observational studies do not prove causality
- RANDOMIZED CONTROLLED TRIAL (RCT)
- Overview
- A RCT is a prospective study where subjects are randomly assigned to either a treatment group or a control group
- In most studies, the two groups are matched as closely as possible by variables such as age, medical problems, lab values, blood pressures, etc. If the study is very large (ex. > 500 subjects), then these factors will often balance automatically through randomization.
- Randomization is very important because it is the only way to control for unmeasured covariates. Measured variables (ex. weight, age, medical history, etc.) can be adjusted for, but there may be variables that are not measured because the researchers are not aware that they can effect the study outcome. If the study is randomized and large enough, then unmeasured covariates should be distributed equally between the groups, and they will not bias the outcome.
- Randomization also prevents hidden bias. In observational studies, there is no way to prevent or control for hidden bias.
- In a RCT, the treatment group is given an intervention (drug or procedure)
- The control group is not given the intervention, or it is given a placebo or sham procedure
- The effect of the intervention is then measured over a predetermined time period by a predetermined set of outcomes
- RCTs are considered the gold standard for clinical trials, and if done correctly, their findings are much more consequential than those of other study designs (ex. cohort studies, case-control studies)
- The best RCTs are placebo-controlled and double-blinded where possible
- Able to control for confounding from unmeasured covariates
- Hidden bias can be prevented
- Criteria and outcomes can be clearly defined and accurately measured
- Large RCTs will give the best measure of an intervention's effect in the intended population
- Expensive and time-consuming
- Can take many years to complete
- If an outcome has a low incidence, RCTs are typically not feasible because the study size would need to be very large, and it would make the trial very expensive
- RCT are not ethical in some situations. Example: Randomizing subjects to smoking or other interventions that are known to be harmful.
- DATABASE AND REGISTRY STUDIES
- Overview
- Certain countries, HMOs, and insurance companies maintain large databases of medical information on their citizens/patients/clients
- This information can be used to conduct observational studies
- Observational studies using medical databases as a source of information have become very popular because they allow investigators to examine a very large amount of information with very little effort, expense, and time
- Information from medical databases has a number of weaknesses that should be considered
- Cheap and easy to perform
- Able to evaluate a large amount of information with very little effort
- Diagnostic codes are often used as criteria. Use of these codes is not well-standardized.
- Pertinent patient information is often missing or not recorded and must be estimated
- Only can be used for observational studies
- AS-TREATED ANALYSIS
- Overview
- In an as-treated analysis, outcomes for subjects in a study are counted toward the treatment they actually receive. If a subject is randomized to one treatment group, but crosses over during the study and receives a competing treatment, then their outcome counts toward the competing treatment.
- This differs from an intention-to-treat analysis where a participant's outcome counts toward their original assigned group regardless of whether they crossover
- As-treated analyses are often performed in studies where there is a high amount of crossover between treatment groups
- On the surface, as-treated analyses seem to make sense, but in reality, they are susceptible to a large amount of bias
- A trial enrolls 200 people with sciatic nerve pain
- 100 people are randomized to physical therapy, and 100 people are randomized to back surgery
- The subjects are followed for one year. At the end of one year, differences in back pain and disability are compared.
- During the course of the year, 30 people assigned to physical therapy end up undergoing back surgery, and 10 people assigned to surgery never actually have surgery and instead receive physical therapy (crossovers)
- The researchers decide to do an as-treated analysis of their data so that the crossovers will be counted toward the therapy they actually received
- Bias in as-treated analyses
- In this example, all the patients who crossed-over are counted toward the treatment they actually received: 30 people from the physical therapy group are counted toward surgery, and 10 people from the surgery group are counted toward physical therapy
- This appears to be logical since the outcomes for crossovers will count toward the treatment they actually received. In reality, the analysis will likely be biased.
- People often enter trials because they hope to receive one of the treatments being offered. Trial protocols often allow for crossover at some point because this makes it easier to get people to participate. If a patient believes the only thing that will make his sciatic pain better is surgery (the perception that an invasive treatment is actually doing something where other treatments are not), and they are assigned to the physical therapy group, then that patient may be disappointed and is already biased toward not improving. The same patient may later crossover and receive surgery and only then get better because "something has been done." If this patient is counted toward the surgery group, then bias has been introduced.
- Another issue with as-treated analyses is that randomization is compromised. Randomization is one of the most important statistical methods that protects against bias and confounding (see randomization below)
Example:
- COMPETING RISKS
- Survival analysis
- The Kaplan–Meier estimator is a popular way to evaluate outcomes in prospective studies
- The method was originally developed to evaluate survival over a period of time with overall mortality being the primary outcome of interest
- Use of the method has expanded, and it is often used to evaluate outcomes other than overall mortality
- It's important to understand the concept of censored patients in survival analysis. All patients entering the study are assumed to experience the primary outcome at some point. If the patients are censored (lost from the study for some reason without experiencing the outcome), they are still treated as having the primary outcome at some point after they are censored. Patients who make it to the end of the study without being censored or having the primary outcome are assumed to have the primary outcome at some point after the study ends.
- This concept holds true if the primary outcome is overall mortality, because all patients will die at some point. If the primary outcome is something other than overall mortality, then the assumption that all patients will eventually experience the outcome or that they are always at risk for the outcome is not necessarily true.
- Example:
- A study is designed to look at the risk of hip replacement in elderly patients over time
- Patients are followed for a period of time and the Kaplan-Meier method is used to estimate risk with hip replacement being the primary outcome
- Because the patient population is elderly, a significant number of subjects die during the study without ever having a hip replacement
- The dead subjects are no longer at risk for a hip replacement, yet in the Kaplan-Meier analysis, they are still considered to be at risk for a hip replacement
- The Kaplan-Meier analysis will overestimate the risk of a hip replacement in this population
- Competing risk
- In the example above, death is a competing risk for hip replacement
- In order to correct for the fact that patients who die are no longer at risk for a hip replacement, a different kind of analysis needs to be performed
- In most cases, that analysis is the cumulative incidence competing risk (CICR) estimate
- CIRC analysis accounts for events that affect the risk of the primary event, and it lowers the probability of the primary event when a competing event occurs. In CIRC analysis, patients experiencing the competing event are no longer considered at risk for the primary event.
- DOUBLE-BLIND TRIALS
- In double-blind trials, neither the subject nor the researcher evaluating the subject knows which intervention the subject is receiving
- In single-blind trials, only the subject is blinded as to what intervention they are receiving
- Double-blinding helps prevent bias on the part of the subjects and the researchers
- When possible, randomized controlled trials should be double-blinded
- Double-blinding is not always possible. For example, when comparing a surgical intervention to a medical one.
- INTENTION-TO-TREAT
- Overview
- Intention-to-treat is a method in randomized controlled trials where the outcomes for all participants who are randomized to a certain treatment or control group are counted toward that group regardless of whether they complete their assigned treatment protocol or not
- In an intention-to-treat analysis, outcomes for patients who stop treatment or drop out of treatment are still counted toward their original randomized group. Even outcomes from patients who crossover and receive a competing treatment are still counted toward the original assigned group.
- Intention-to-treat analysis is the gold standard for analysis, because other forms of analyses (see per-protocol analysis, on-treatment analysis, modified intention-to-treat analysis, and as-treated analysis) can introduce bias into a study
- Example:
- A trial enrolls 200 people with sciatic nerve pain
- 100 people are randomized to physical therapy, and 100 people are randomized to back surgery
- The subjects are followed for one year. At the end of one year, differences in back pain and disability are compared.
- During the course of the year, 30 people assigned to physical therapy end up undergoing back surgery, and 10 people assigned to surgery never actually have surgery and instead receive physical therapy (crossovers)
- Intention-to-treat
- In an intention-to-treat analysis, outcomes for the 30 people who crossed-over from the physical therapy group and received surgery would still be counted toward the physical therapy group
- Outcomes for the 10 people who never underwent surgery will still be counted toward the surgery group
- Limits bias - In the above example, if an as-treated analysis was performed where all the patients who crossed-over were counted toward the group they ended up in, then bias would be introduced. People often enter trials because they hope to receive one of the treatments being offered. Trial protocols often allow for crossover at some point because this makes it easier to get people to participate. If a patient believes the only thing that will make his sciatic pain better is surgery (the perception that an invasive treatment is actually doing something where other treatments are not), and they are assigned to the physical therapy group, then that patient may be disappointed and is already biased toward not improving. The same patient may later crossover and receive surgery and only then get better. If this patient is counted toward the surgery group, then bias has been introduced.
- High crossover or dropout rates can bias a study towards the null (no effect) - If a study has a high crossover or dropout rate, then the study may be biased towards the null, or "no effect." In the above example, the 30 patients who crossed over to surgery are still counted toward physical therapy. If a true difference in outcomes exists between the two treatments, it will be harder to discern since positive surgery outcomes may count toward physical therapy. In the end, there is no perfect way to deal with crossovers and dropouts, but the gold standard for evaluating subjects in a trial is intention-to-treat.
- MENDELIAN RANDOMIZATION
- Overview
- Mendelian randomization is an observational study technique that uses variations in genetic makeup to determine if a risk factor is associated with an outcome
- For example, in population studies, HDL cholesterol levels are inversely associated with heart disease risk; as HDL levels increase, risk of heart disease goes down
- A number of genetic variants are associated with HDL levels. Some variants raise levels and others lower them. Through Mendelian inheritance, these variants are distributed randomly across the population. If a researcher wants to determine if HDL levels are directly associated with heart disease risk, they can form patient cohorts based on genetic variants that affect HDL levels. Cohorts with HDL-promoting variants will have higher HDL levels simply because of their genetic makeup. Other variables that affect HDL levels (e.g. exercise, diet) will be distributed equally among the individuals making the cohorts "randomized" for potential confounders. If a causal relationship between HDL levels and heart disease exists, then HDL-promoting variants will be associated with lower heart disease risk. A number of studies have actually looked at this, and they have found that HDL-promoting variants are not associated with lower heart disease risk calling into question the association between HDL and heart disease.
- In order for Mendelian randomization to work, the genetic variants being studied must only be associated with the risk factor in question and not other variables that may affect the outcome. For example, if HDL-promoting variants are also associated with lower LDL levels or lower blood pressure, then confounding exists and incorrect conclusions may be drawn. When a variant affects the outcome through pathways other than the risk factor being studied, this is referred to as pleiotropy.
- Although Mendelian randomization studies are observational cohort studies, cohort formation based on genetic variants offers some degree of randomization
- Mendelian randomization studies may be used to do an exploratory analysis of a possible intervention that affects a risk factor. For example, drugs have been developed that raise HDL levels. Before doing expensive randomized controlled trials to see if the drugs lower heart disease risk, researchers may do a Mendelian randomization study to see if HDL levels are directly related to heart disease risk.
- If pleiotropy exists, results are not valid. Pleiotropy may not be easy to detect.
- Statistical power can be difficult to achieve and often requires the use of multiple variants which increases the risk of pleiotropy [8]
- MODIFIED INTENTION-TO-TREAT
- Overview
- Modified intention-to-treat is when investigators in a trial use an intention-to-treat analysis, but they modify who they include in this analysis
- Example:
- A study wants to compare the effects of a new drug on postmenopausal vaginal dryness
- 100 women are randomized to the drug, and 100 women are randomized to placebo
- At the start of the study, all women have a vaginal smear performed
- After 6 months, changes in vaginal symptoms are measured
- The researchers present their significant findings in a "modified intention-to-treat" analysis where only women who had ≤ 5 superficial cells on their vaginal smear were included in the analysis
- In the above example, the researchers decided to only include a subset of the randomized women in their analysis
- A modified intention-to-treat analysis should raise red flags because it may mean the researchers did not find a significant result with their intention-to-treat analysis, and they are now digging for a significant results by changing the study's criteria. This is bad form because it can introduce bias.
- A reviewer should also ask themselves if the modified criteria is clinically relevant. In the above example, most doctors are not going to do a vaginal smear before prescribing a menopause drug, so the fact that it may work in this subset of women is irrelevant.
- In some cases, a modified intention-to-treat analysis has little impact. Example: researchers do a modified intention-to-treat analysis that only includes randomized patients who received at least one dose of a study drug.
- ON-TREATMENT ANALYSIS
- Overview
- On-treatment analysis is when investigators only include data from patients while they are taking their assigned treatment. If patients stop their assigned treatment for a period, data from that time period is excluded.
- On-treatment analysis is similar to per-protocol analysis
- On the surface, on-treatment analyses makes sense - if someone stops treatment then their results should not count since the intervention can not help them if they do not take it
- In reality, on-treatment analysis introduces bias (See Examples below)
- A study is comparing a new drug to prevent migraines to an older drug that prevents migraines
- 100 people are randomly assigned the new drug, and 100 people are randomly assigned the old drug
- The patients are supposed to take the drugs for 6 months, and the number of migraines in that time period are recorded
- 30 people taking the new drug and 10 people taking the old drug stop taking it before the end of the trial
- The researchers decide to do a on-treatment analysis that excludes data from people while they stopped taking their drugs
- In this circumstance, the on-treatment analysis has introduced bias
- People stop study medications for a variety of reasons (e.g. side effects, lack of perceived efficacy, etc.). If a on-treatment analysis is performed, the study has been biased towards "responders" and it does not reflect the true efficacy of the drug in the intended population. This may also lead to spurious significant results.
- A study is comparing aspirin to placebo for the prevention of deep vein thrombosis
- 1500 people are given aspirin and 1500 people are given placebo. The 2 groups are followed for 2 years and the rate of deep vein thrombosis is compared.
- 300 people in the aspirin group stopped taking their aspirin during the trial
- The researchers do a on-treatment analysis that excludes data from people while they stopped their aspirin. The analysis shows that when compared to placebo, patients who actually took their aspirin had better outcomes than the whole aspirin group (takers and non-takers).
- On the surface, this analysis seems legitimate. The outcome is objective, and for the most part, people tolerate aspirin well, so comparing people who actually took the drug to the controls makes sense.
- The problem is that past studies have shown that outcomes in the placebo group also differ between compliant and noncompliant patients. Patients who are compliant in taking their placebo are also more likely to have better outcomes than patients who do not (even for objective outcomes like heart attack). This phenomenon likely occurs because of differences in unmeasured confounders between compliant and noncompliant patients. Because of this phenomenon, the on-treatment analysis leads to erroneous results and is biased.
- PER-PROTOCOL ANALYSIS
- Overview
- In a per-protocol analysis, only subjects who completed the protocol they were assigned to are included in the statistical analyses
- Patients who vary from the protocol (ex. stop medication), drop out of the study, or crossover to another therapy are not counted in the analyses
- Per-protocol analyses are similar to on-treatment analyses
- Per-protocol analyses introduce bias in many cases (see Example #2 and Example #3 below)
- Per-protocol analyses may be more meaningful in conditions that are difficult to treat and/or when there are few treatment options
- Noninferiority trials
- In noninferiority trials, per-protocol analyses should be reported with intention-to-treat analyses
- Noninferiority trials often compare new treatments to standard treatments (ex. new anticoagulant drug compared to warfarin for atrial fibrillation)
- During the trial, patients assigned to the new treatment may be allowed to crossover to the standard treatment for various reasons (e.g. side effects, new indication, etc.). In intention-to-treat analyses, crossovers can bias a study towards the null, or in the case of noninferiority trials, towards noninferiority.
- Per-protocol analyses should be reported with intention-to-treat analyses to help measure the sensitivity of the results to protocol violations
- A study is comparing a new drug + standard therapy to standard therapy alone in a difficult to treat cancer
- The new drug has many side effects that make it intolerable for a number of patients. These patients stop taking the drug.
- The researchers decide to to do a per-protocol analysis to see if the new drug improved survival in patients who actually completed therapy
- In this case, the per-protocol analysis is meaningful because it will show whether people who can tolerate the therapy actually benefit from it
- A study is comparing a new drug to prevent migraines to an older drug that prevents migraines
- 100 people are randomly assigned the new drug, and 100 people are randomly assigned the old drug
- The patients are supposed to take the drugs for 6 months, and the number of migraines in that time period are recorded
- 30 people taking the new drug and 10 people taking the old drug stop taking it before the end of the trial
- The researchers decide to do a per-protocol analysis that would exclude the people who stopped taking their drugs
- In this circumstance, the per-protocol analysis has introduced bias
- People stop study medications for a variety of reasons (e.g. side effects, lack of perceived efficacy, etc.). If a per-protocol analysis is performed, the study has been biased towards "responders" and it does not reflect the true efficacy of the drug in the intended population. This may also lead to spurious significant results.
- A study is comparing aspirin to placebo for the prevention of deep vein thrombosis
- 1500 people are given aspirin and 1500 people are given placebo. The 2 groups are followed for 2 years and the rate of deep vein thrombosis is compared.
- 300 people in the aspirin group stopped taking their aspirin during the trial
- The researchers do a per-protocol analysis that excludes the people who stopped their aspirin. The analysis shows that when compared to placebo, patients who actually took their aspirin had better outcomes than the whole aspirin group (takers and non-takers).
- On the surface, this analysis seems legitimate. The outcome is objective, and for the most part, people tolerate aspirin well, so comparing people who actually took the drug to the controls makes sense.
- The problem is that past studies have shown that outcomes in the placebo group also differ between compliant and noncompliant patients. Patients who are compliant in taking their placebo are also more likely to have better outcomes than patients who do not (even for objective outcomes like heart attack). This phenomenon likely occurs because of differences in unmeasured confounders between compliant and noncompliant patients. Because of this phenomenon, the per-protocol analysis leads to erroneous results.
- PLACEBO-CONTROLLED TRIALS
- In a placebo-controlled trial, subjects who are not randomized to an active treatment (controls) are given a fake or sham treatment instead (Ex. an inert pill or a procedure that is faked)
- Placebo-control prevents subjects in a trial from knowing whether they are receiving the active treatment. This prevents bias that can arise from a subject's expectations that an intervention works.
- Placebo-control is very important in trials where outcomes are subjective (ex. pain, disability, frequency of events that are not biologically measurable)
- It should be applied when possible in trials with objective outcomes
- PROPENSITY SCORE MATCHING (PSM)
- Overview
- Propensity score matching (PSM) is a statistical technique used in observational studies. It helps to control for selection bias.
- In PSM, subjects who received a treatment are matched with subjects who did not receive the treatment based on the probability of receiving the treatment. The probability of receiving the treatment is reflected in the propensity score. The propensity score is calculated by combining the probability of receiving treatment for a number of measured covariates.
- When subjects with the same propensity score are compared, selection bias is limited
- Determine the probability of receiving the treatment for each measured covariate in the study (ex. age, sex, medical conditions, lab values, etc)
- For each subject, combine the probabilities of receiving the treatment based on their individual covariates. Derive the propensity score from this combination (usually with logistic regression)
- Create groups of subjects with the same propensity score. Subjects with the same propensity score will typically match closely on a number of covariates.
- Compare treated subjects to untreated subjects within these groups. There are a number of ways of doing this. See Propensity score techniques below.
- Perform multivariate analysis on the propensity-matched sample
- Propensity score techniques
- Direct comparison
- In direct comparisons, treated and untreated individuals with the same score are matched and compared. Matching can be one-to-one, one-to-as many that match, and so on.
- Groups can also be stratified by propensity score and then strata can be compared directly
- A major disadvantage of direct matching is that depending on the method used, all of the available data may not be utilized. In addition, if strata are compared, residual confounding may occur.
- Propensity score as a covariate
- The propensity score can be used as a covariate in a multivariate model
- In this case, the outcome is the dependent variable, and the independent variables are the exposure/treatment and the propensity score
- If the probability of receiving the treatment (propensity score) is strongly associated with the outcome, then covariate adjustment for the propensity score will decrease the strength of association between the treatment and the outcome
- Propensity score weighting
- Propensity score weighting is a method where each individual in the sample is weighted by the inverse of their propensity score
- Using this method, subjects who are more likely to receive the treatment carry less weight, and subjects who are less likely to receive the treatment receive the most weight
- With inverse weighting, covariates that are associated with receiving the treatment/exposure (higher propensity score) are neutralized while covariates that are weakly associated with receiving the treatment (low propensity score) gain value. This helps to neutralize selection bias, creating a sample where covariate distribution is independent of treatment selection. [5,6]
- PSM is helpful when a large number of covariates are measured because it creates a comparison that minimizes selection bias regardless of whether patients match exactly on all of the covariates
- PSM simulates randomization of treatment for measured covariates (not for unmeasured covariates)
- PSM is an observational technique, and it cannot control for unmeasured covariates and hidden bias
- RANDOMIZATION
- TERMINOLOGY:
- Measured variables - patient variables (ex. age, sex, medical problems) that are measured before a trial begins
- Unmeasured covariates - patient variables that are not measured, but may be related to (and have an influence on) the outcome being evaluated. There may be several reasons these variables are not measured. Researchers may not be aware that they can affect the outcome, or the variables may be expensive to obtain (ex. genetic profiles on every patient enrolled)
- Hidden bias - any form of bias that may influence the outcome of a trial (ex. researchers assigning healthier patients to the treatment group (a form of selection bias))
- RANDOMIZATION
- Randomization is a process in clinical trials where patients are randomly assigned to different groups (ex. treatment and placebo)
- Ideally, the person enrolling the patients in the trial should have no idea which group the patient will be assigned to
- Randomization is a critical component of controlled trials because it is the only way to control for unmeasured covariates. Randomization also prevents hidden bias.
- If the trial is large (> 500 subjects), then randomization will typically create groups that are equal for a number of measured variables (ex. age, sex, medical problems, etc)
- Unmeasured covariates can also affect the trial outcome. If the trial is large enough, randomization will automatically balance unmeasured covariates between groups so that they will not affect the outcome.
- Randomization prevents hidden bias from occurring
- Randomization is the main reason randomized controlled trials are the gold standard for clinical studies. Observational studies cannot randomize participants, therefore they are always subject to unmeasured covariates and hidden bias.
- SINGLE-BLINDED TRIALS
- In single-blinded trials, only the patient is blinded as to what treatment they are receiving. The investigators or treating doctors are aware of the treatment given.
- Contrast with double-blinded trials where the investigator and subject are blinded to the treatment allocation
- Double-blinding is preferred when possible because it prevents investigator bias
- ABSOLUTE VS RELATIVE RISK
- Overview
- When discussing absolute risk and relative risk, it helps to consider the two measures together
- Absolute risk is the overall incidence of an outcome in an entire group or population
- Relative risk is the ratio of the incidence of an outcome between two groups
- Relative risk only considers the proportion of the participants who have the outcome in question where absolute risk considers the entire population being studied. This can lead to some large difference between the two measures.
- Conclusions
- As one can see from the example above, the relative risk reduction and the absolute risk reduction can be quite different (46% vs 2.3%)
- It's important to understand the difference, because medical literature will often emphasize a large relative risk reduction when the absolute risk reduction may be quite small and insignificant. From the example above, the drug company for Livelong will likely say that they "reduce the risk of heart attack by 46%." While this may be true, someone taking Livelong will only reduce their overall risk of heart attack by 2.3%.
- INCIDENCE AND PREVALENCE
- INCIDENCE
- Incidence is the proportion of a group that develops a disease (or experiences an event) over a specified period of time
- Example:
- 100 people without diabetes are followed for 5 years
- At the end of 5 years, 10 people have developed diabetes
- The incidence of diabetes over the 5 years is 10%
- PREVALENCE
- Prevalence is the proportion of a group that has the disease (or has experienced an event) at a single time point
- Example:
- 100 people are selected randomly
- 10 of the people in the group have diabetes
- The prevalence of diabetes in the group is 10%
- LIKELIHOOD RATIO (LR)
- Overview
- A likelihood ratio is a statistical ratio used to quantify the predictive value of a medical test (typically a screening test)
- The likelihood ratio is the factor by which the post-test odds of a condition being present (positive likelihood ratio) or absent (negative likelihood ratio) are increased or decreased based on the results of a test
- The likelihood ratio can be multiplied by the pre-test odds of the condition being present to determine the post-test odds
- Post-test odds of disease = Pre-test odds X Likelihood ratio
- Mathematically, the likelihood ratio is calculated using the sensitivity and specificity of the test:
- Example:
- Test A is a screening test for heart disease
- Paul is a 60 year-old patient who based on his family history, age, and medical conditions has a probability of 25% of having heart disease
- Convert 25% probability to odds = 25/75 = 0.33
- Test A has a negative likelihood ratio of 0.1 and a positive likelihood ratio of 1.9
- Equivalent to a Sensitivity of 95% and a specificity of 50%
- If Paul has a negative test A, then his post-test odds of heart disease are 0.1 X 0.33 = 0.033
- If Paul has a positive test A, then his post-test odds of heart disease are 1.9 X 0.33 = 0.627
- Converting back to probability:
- negative test = 0.033/1.033 = 0.032 or 3.2% probability
- positive test = 0.627/1.627 = 0.385 or 38.5% probability
- In Paul's case, a positive test increases his probability of disease from 25% to 38.5% - not very helpful
- A negative test decreased his probability of disease from 25% to 3.2% - more helpful
- This is true because the test had a high sensitivity (low false negatives) and low specificity (high false positives)
- NET RECLASSIFICATION INDEX/IMPROVEMENT (NRI)
- The Net Reclassification Improvement (NRI), (also called "Index") is a statistical measure used to evaluate whether adding a measure to a predictive model will improve the predictive accuracy of the model. An example would be adding Coronary Artery Calcium Scores (CACS) to the Framingham risk model to see if the addition of the CACS improves the Framingham's ability to predict a heart attack. The NRI takes into account both correct and incorrect reclassifications.
- The NRI is calculated in the following manner:
- In a study, take the people who had an event or outcome (events)
- Calculate the number of events who were reclassified into a higher risk category when the new measure was included
- Subtract the number of events who were reclassified into a lower risk category when the new measure was added
- Divide this number by the total number of people who had an event
- Now take the number of people who did not have an event or outcome (nonevents)
- Calculate the number of nonevents who were reclassified into a lower risk category when the new measure was added
- Subtract the number of nonevents who were reclassified into a higher risk category when the new measure was added
- Divide this number by the total number of people who did not have an event
- Add the two proportions together and you have the NRI
- The formula for the NRI is as follows:
- Example
- The Framingham risk model uses age, gender, total and HDL cholesterol, smoking status, and systolic blood pressure to predict a person's 10-year risk of heart attack
- Researchers want to know if adding Coronary Artery Calcium Scores (CACS) to the model improves the risk prediction of the original Framingham model
- A cohort of patients is formed and their Framingham 10-year risk is calculated using the original risk model
- The patients also have their CACS measured at baseline
- The patients are followed for 10 years and the incidence of heart attacks is measured
- Each patient's predicted risk of heart attack is calculated using the original Framingham model. The risk is also calculated using a model that incorporates the CACS.
- The NRI is then calculated by comparing the two models
- Interpretation of the NRI
- It's important to understand what the NRI actually means because it is commonly misinterpreted
- In the example above, let's assume the addition of the CACS to the Framingham model had the following effect:
- 71 patients with a heart attack are reclassified as higher risk
- 24 patients with a heart attack are reclassified as lower risk
- 790 patients without a heart attack are reclassified as lower risk
- 657 patients without a heart attack are reclassified as higher risk
- Total number of patients who had a heart attack - 209
- Total number of patients who did not have a heart attack - 5669
- The calculation of the NRI is as follows:
- Some people mistakenly interpret this as the addition of CACS correctly reclassifies 25% of the population studied
- This is not correct. The addition of the CACS correctly reclassified 22.5% of the population who had an event (0.225 X 209 = 47) and 2.3% of the population that did not have an event (0.023 X 5669 = 130)
- The percent of the total population correctly reclassified is 177/5878 = .03 or 3%
- Whether or not 3% is clinically significant will depend on the cost and convenience of the additional measure
- NUMBER NEEDED TO TREAT/HARM (NNT/NNH)
- Overview
- The number needed to treat (often abbreviated "NNT") is the number of patients who must be treated with an intervention in order for one patient to benefit
- If the intervention is a screening test, it is the number of people who would have to be screened in order for one person to benefit from screening
- The NNT can also be referred to as the "Number Needed to Harm" (NNH) when talking about a side effect or adverse event
- The NNT is calculated by taking the reciprocal of the absolute risk reduction
- The NNH is calculated by taking the reciprocal of the absolute increase in adverse events
- Example:
- The recent NLST trial (detailed here) found that yearly CT scan screening reduced the absolute risk of lung cancer death by 0.33% in heavy smokers
- The number needed to screen is = 1/0.0033 = 303 patients
- This means 303 patients would have to be screened in order to prevent 1 lung cancer death
- ODDS RATIO
- OVERVIEW
- When most people think of comparing two outcomes, they think in terms of relative risk
- Relative risk is a comparison that uses fractions. It is the most intuitive way to compare two outcomes.
- Example:
- Drug A has been shown to reduce the risk of heart attack by 20%
- Most people understand that people taking Drug A will have 20% fewer heart attacks than people not taking Drug A
- ODDS
- When discussing odds ratios, it helps to review what odds are
- Odds are ratios. Odds are not fractions. This can be confusing.
- Odds = the number of times that an event will occur for the number of times that it will not occur
- Example:
- 3:1 odds means the event is likely to occur 3 times for every one time it does not occur
- So for every 4 tries, there will be 3 events and 1 non-event. This equals a probability of 75% (3/4).
- To convert between odds and probability:
- ODDS RATIO
- An odds ratio is a ratio of two odds. Relative risk (sometimes called hazard ratios) is a ratio of two fractions.
- Because of this distinction, the odds ratio and the relative risk ratio do not always approximate each other. This can be confusing because many people mistakenly interpret odds ratios as relative risk ratios.
- As the prevalence of an outcome increases above 10%, the odds ratio and the relative risk start to diverge. This occurs because odds above 1:1 (0.50 probability) can run to infinity where a fraction (of a whole) is always between 0 and 1. This can lead to wide discrepancies between the odds ratio and the relative risk.
- Example:
- Let's assume a group of diabetics are followed for a year and 80% of them suffer a heart attack
- A group of nondiabetics is followed for a year and 20% of them suffer a heart attack
- We want to compare the risk of heart attack between the two groups
- The relative risk of heart attack in the diabetic group compared to the nondiabetics is (0.8)/(0.2) = 4. Diabetics have 4 times the risk of heart attack when compared to nondiabetics. Most people understand this.
- The odds ratio of a heart attack when comparing the diabetics to nondiabetics is (4)/(0.25) = 16. This differs from the relative risk. It means diabetics have 16 times the odds of a heart attack than nondiabetics. This comparison is not as intuitive for most people and will often be misinterpreted as 16 times the risk of heart attack.
- Rules for interpreting odds ratios:
- When the incidence of the outcome of interest is < 10% in the study population, then the odds ratio and the relative risk can be considered equivalent
- When the incidence of the outcome of interest is > 10% in the study population, and the odds ratio is > 2.5 or < 0.5, then the odds ratio will tend to exaggerate the magnitude of the association. In some circumstances the odds ratio can be converted to a relative risk (see below).
- ODDS RATIO IN MEDICAL STUDIES
- Odds ratios are always reported in case-control studies because risk is not measured in these studies. The case group has 100% probability of the disease and the control group has 0% probability. The odds of exposure to a risk factor is compared between the two groups, therefore the results are reported as an odds ratio.
- Cohort studies often utilize logistic regression to control for covariates. Logistic regression yields an odds ratio. In some cases, the odds ratio can be converted to a relative risk. See formula below. [2]
- CONVERTING AN ODDS RATIO TO RELATIVE RISK
- If a cohort study uses logistic regression and reports an odds ratio, it can be converted to relative risk if the incidence of the outcome in the unexposed (control) group is known
- OUTCOME MEASURES (SIGNIFICANT CLINICAL OUTCOMES VS SURROGATE ENDPOINTS)
- SIGNIFICANT CLINICAL OUTCOMES
- A significant clinical outcome is an outcome that has a significant effect on a person's health, functioning, or well-being
- Examples of significant clinical outcomes include: death, heart attack, stroke, blindness, paralysis, cancer
- SURROGATE ENDPOINTS
- Surrogate endpoints are measures that have been shown to be related to a significant clinical outcome
- Surrogate Endpoints are often used in clinical trials because they are easier to measure and require less time to demonstrate an effect than significant clinical outcomes
| Surrogate endpoint | Related significant clinical outcome |
|---|---|
| Cholesterol levels | heart attack, stroke, mortality |
| blood pressure | heart attack, stroke, mortality |
| carotid intima thickness | heart attack and stroke |
| Hemoglobin A1C level | diabetes complications |
- Example:
- High cholesterol levels are associated with heart attacks
- A trial with a new cholesterol drug will measure its effect on cholesterol levels (surrogate endpoint)
- This takes far less time (weeks to months) than measuring its effect on heart attack rates (years)
- ISSUES WITH SURROGATE ENDPOINTS
- Problems arise when interventions improve surrogate endpoints but do not improve (or may even worsen) significant clinical outcomes
- The surrogate endpoint may be related to the outcome through a factor that the intervention does not affect
- The surrogate endpoint may be a marker of disease severity as opposed to being a cause of the disease
- The intervention may be harmful in a way that is unrelated to the surrogate endpoint
- Examples of interventions that improve surrogate endpoints, but do not improve significant clinical outcomes:
- Folic acid and Vitamin B supplements
- High homocysteine levels are associated with an increased risk of heart attack
- Folic acid and Vitamin B supplementation lower homocysteine levels (surrogate endpoint)
- Folic acid and Vitamin B supplementation have not been shown to decrease the risk of heart attack (significant clinical outcome) in patients with high homocysteine levels
- Niacin and fibrates
- High HDL levels (good cholesterol) are associated with a lower risk of heart attack
- Niacin and fibrates raise HDL levels (surrogate endpoint)
- These drugs have not been shown to improve overall mortality (significant clinical outcome) in patients with coronary heart disease
- P-VALUE
- Overview
- The p-value is an important statistical measure used to compare findings between two outcomes
- The p-value is expressed as a fraction (ex. p=0.01) which can be converted into a percentage (ex. p=0.01 same as p = 1.0%)
- The p-value has the following meaning: If there really is no difference between two groups, the p-value is the probability that the observed difference (or one greater) occurred by chance.
- Example:
- A study compares a new drug to prevent deep vein thrombosis to placebo
- 1000 people take the new drug for a year, and 1000 people take placebo for a year
- At the end of the trial, the new drug group had 20 fewer deep vein thromboses than the placebo group
- The researchers report the following p-value for the difference between the two groups: p=0.02
- This means that if there really is no difference between the two groups (the new drug is ineffective), then the probability of observing the results found in this trial is 2%
- Statistical significance
- In most cases, a p-value < 0.05 is considered statistically significant. In the medical literature, it is often stated that the difference was "significant."
- This cutoff value is somewhat arbitrary. For example, a p-value of 0.05001 would be considered nonsignificant, while a value of 0.04999 would be considered significant. The difference between these two values is miniscule. It's better to look at the p-value and consider its meaning.
- A common guideline for considering p-values is presented below
| p-value | Significance |
|---|---|
| 0.01 ≤ p < 0.05 | significant |
| 0.001 ≤ p < 0.01 | highly significant |
| p < 0.001 | very highly significant |
| p > 0.05 | not significant |
| 0.05 ≤ p < 0.10 | trend towards significance |
- POSITIVE PREDICTIVE VALUE (PPV) AND NEGATIVE PREDICTIVE VALUE (NPV)
- Positive predictive value (PPV) - the proportion of patients with a positive test who actually have the disease
- Negative predictive value (NPV) - the proportion of patients with a negative test who do not have the disease
- Like sensitivity and specificity, the predictive value is a measure of test accuracy (typically a screening test)
- The predictive value takes into account the prevalence of the disease in the population that is being tested. The prevalence of the disease affects how well a test will perform.
- By combining these three measures - test sensitivity, test specificity, and disease prevalence - the predictive value gives a better measure of how a test will perform in a specific population
- The main drawback to predictive values is that the prevalence of the disease in the tested population must be known or estimated with some accuracy
- Formulas for predictive values:
- SENSITIVITY AND SPECIFICITY
- SENSITIVITY
- Sensitivity is a measure of how well a test identifies people with a disease
- A test with a high sensitivity will be positive for most people with the disease
- In statistical terms, it is the fraction of subjects with the disease who get a positive test result
- If a test has a high sensitivity, a negative test result means the disease is unlikely (few false negatives)
- SPECIFICITY
- Specificity is a measure of how well a test excludes people without a disease
- If a test has a high specificity, it will give a negative result for most people without the disease
- In statistical terms, it is the fraction of subjects without the disease who get a negative test result
- If a test has a high specificity, a positive test means the disease is likely (few false positives)
- INVESTIGATOR BIAS
- Investigator bias occurs when the researchers in a study desire a particular outcome for the study
- Investigator bias can occur at every level in the research process - design, data gathering, data analysis, and outcome reporting
- Well-designed studies with careful blinding can help limit investigator bias
- Investigator bias can also occur when researchers emphasize secondary outcomes that are significant when their primary outcome was not
- Example:
- Investigators are studying a new screening test for breast cancer
- Outcome measures include overall mortality and mortality from breast cancer
- If the study finds that the screening test decreases breast cancer mortality, but has no effect on overall mortality, investigators may be tempted to report the decrease in breast cancer mortality and disregard the null effect on overall mortality. If they report both, they may emphasize the effect on breast cancer mortality and pay little notice to the effect on overall mortality. This practice is quite common.
- PROTOPATHIC BIAS
- Protopathic bias occurs when an intervention is used to treat the first symptoms of a disease that has not yet been diagnosed. The intervention may appear to be a risk factor for the disease when it is actually associated with symptom treatment.
- Example:
- A study looks at the association of Depo-Provera® and uterine fibroids
- The study finds that Depo-Provera® is significantly associated with uterine fibroids and declares it a risk factor
- Depo-Provera® is often used to treat excessive vaginal bleeding, a presenting symptom of uterine fibroids
- Protopathic bias may have influenced the findings in this study
- RECALL BIAS
- Recall bias occurs when a person with an outcome is more likely to recall an exposure than someone without the outcome
- Recall bias is a major concern in observational studies, particularly case-control studies, because it may bias results toward a false positive effect (Type 1 error)
- Example:
- A case-control study is exploring the relationship between over-the-counter NSAID use in pregnancy and birth defects
- Women who have children with birth defects (cases) and women who have children without birth defects (controls) are asked about their over-the-counter NSAID use during pregnancy
- In this study, recall bias may occur because women who have children with birth defects may be more likely to thoroughly contemplate their NSAID use during pregnancy than women who have normal children
- This difference in attention to recall may bias the study toward a positive effect
- REGRESSION DILUTION BIAS (ATTENUATION BIAS)
- Regression dilution bias (also referred to as attenuation bias) is a form of bias seen in regression analysis. It occurs when there is an excessive amount of measurement error in the independent variable. The excessive error leads to an underestimation of the slope (coefficient) of the regression line.
- Regression dilution can be corrected for in several ways. To decrease the amount of measurement error, the independent variable can be measured several times on each subject and an average can be used. If remeasuring the entire sample is not feasible, then a smaller subset of randomly selected individuals can be used. In other methods, subjects at 20 - 30% of the extremes are remeasured. The average of these measurements can then be used to correct the slope of the regression line.
- Correcting for regression dilution is not usually necessary in hypothesis testing for linear relationships (slope ≠ 0)
- Corrections should be performed when regression coefficients are used to measure effect sizes [7]
- REPORTING BIAS
- Reporting bias occurs when events associated with one treatment are more likely to be reported than events associated with another treatment
- There are several reasons reporting bias may occur. If a treatment is newer and less familiar, then people who use the treatment or doctors who prescribe it may be more inclined to report an adverse event they believe to be associated with the treatment. If they are familiar with a treatment, then they may be less inclined to report an adverse event associated with the treatment. Because of this bias, the newer treatment may have more events reported than the older treatment when the underlying incidence of the event is really no different between the two treatments. This phenomenon occurs frequently with new drugs in the FDA's Adverse Event Reporting System.
- Reporting bias may also occur if there is publicity surrounding a treatment-related event. If providers have recently been made aware of a possible adverse event associated with a treatment, then they may be more inclined to report events associated with the publicized treatment than with events from other treatments.
- SAMPLING/ASCERTAINMENT BIAS
- Sampling bias (also referred to as "ascertainment bias") occurs when the procedure for sampling a population inadvertently leads to the over/under representation of a subgroup of that population
- Example:
- Researchers want to study a new drug for chronic back pain
- To recruit patients, the researchers decide to enroll new patients presenting to pain management clinics for chronic back pain
- By recruiting patients from pain management clinics, the researchers run a high-risk of sampling bias
- Patients who are referred to pain management clinics often have complicated histories that might include opiate medication-seeking behavior
- If the medication they are studying does not have opioid properties (ex. a new NSAID), the medication may not perform well simply because a large proportion of the study population is seeking opiate medications
- If the medication is an opiate, it may perform better than expected simply because the population is seeking this type of medication
- In either case, sampling bias has occurred because a subpopulation of patients with chronic back pain (opiate-seekers) is overrepresented in the sample
- SELECTION BIAS
- Selection bias occurs when an intervention or treatment is more likely to be given to one person over another because they differ in some way. The underlying difference between the individuals may not be obvious or even measurable.
- Selection bias is a big concern in observational studies because patients are not randomized
- Example:
- An observational study compares outcomes between diabetics who received Medication A and those who did not
- Medication A requires that patients return to the clinic every 3 months for blood tests
- The study shows that diabetics who took Medication A had better outcomes
- There is a big concern for selection bias in this study because it may be that doctors were more likely to prescribe Medication A to their patients who were more compliant
- The difference in compliance between the two groups can bias the outcomes
- If Medication A is expensive, patients in a higher socioeconomic class may have been more likely to receive it. This can also bias outcomes.
- SURVEILLANCE BIAS
- Surveillance bias occurs when a proposed risk factor for a condition is also independently associated with an increase in medical diagnostic testing (ex. labs, X-rays, screening, etc.)
- The risk factor may appear to be associated with conditions that are discovered from the testing, when in reality, the conditions are only diagnosed more frequently secondary to increased testing (surveillance)
- Example:
- A study published in the Annals of Rheumatic Diseases in 2014 provides a perfect example of a study where there was a high-risk for surveillance bias (PMID 23842460)
- In the study, researchers evaluated the association of polymyalgia rheumatica with cancer
- The study was a cohort study that compared the incidence of cancer diagnosis between patients with polymyalgia rheumatica and matched controls without polymyalgia rheumatica. Data for the study was gathered from a medical database.
- Results of the study found that patients with polymyalgia rheumatica were more likely to be diagnosed with cancer within 6 months of their polymyalgia rheumatica diagnosis when compared to matched controls
- The authors concluded that polymyalgia rheumatica may be associated with cancer
- This study has a high-risk for surveillance bias
- Polymyalgia rheumatica can be a difficult condition to diagnose, and patients will often undergo extensive diagnostic workups. These workups may lead to other diagnoses, including cancer, that may have otherwise gone undetected.
- Once diagnosed, polymyalgia rheumatica treatment often involves frequent doctor's visits and an overall increased exposure to medical care. This increased exposure will typically lead to more testing and screening.
- Because of these factors, there is a high-risk for surveillance bias in this study. It's unclear if polymyalgia rheumatica increases the risk of cancer, or if the increased medical surveillance associated with its workup and diagnosis leads to more cancer detection.
- CONFOUNDING BY INDICATION
- Overview
- Confounding by indication can occur when a condition is a risk factor for an outcome among both exposed/treated individuals and unexposed/untreated controls. If the condition is associated with the exposure/treatment, then confounding by indication may occur, given that the condition is not an intermediate step in the causal pathway to the outcome. [4]
- Example:
- Researchers want to evaluate if Byetta® is associated with pancreatitis
- They conduct a case-control study where they compare the risk of pancreatitis between diabetics on Byetta® and diabetics not on Byetta®
- The study finds that diabetics who took Byetta® were at greater risk for pancreatitis
- There is a risk for confounding by indication in this study. Obesity is a risk factor for pancreatitis. Byetta® causes modest weight loss. It may be that doctors were more likely to prescribe Byetta® to obese diabetics. Because these patients were already predisposed to pancreatitis, confounding by indication may have occurred.
- Confounding by severity
- Confounding by indication is sometimes used to refer to confounding by severity
- In confounding by severity, patients who are sicker may be more prone to receive an intervention
- This may cause the intervention to appear less effective, or even inferior in some cases [4]
- Example:
- A study looks at the association of minoxidil and stroke
- It finds that patients who took minoxidil as compared to other blood pressure medications were more likely to have a stroke
- Minoxidil is typically reserved for patients who have resistant, difficult to treat hypertension
- Confounding by severity is likely because patients with more severe disease are more likely to receive minoxidil. This would bias the results against minoxidil.
- RESIDUAL CONFOUNDING
- Residual confounding occurs when a variable that is related to an outcome is not stratified appropriately
- Confounding is present because subjects grouped together in a strata have different risks for the outcome
- The risk for residual confounding is particularly high when a variable and the risk for an outcome have a non-linear relationship (ex. J-shaped or bimodal)
- Example:
- Researchers want to look at the association of BMI with mortality
- They form two cohorts of patients: Group 1 - BMI < 27; Group 2 - BMI ≥ 27
- After observing the two cohorts for 10 years, they find no difference in the risk of mortality and declare that body weight does not influence mortality
- Residual confounding is very likely in this study because the researchers did not stratify patients appropriately
- BMI has been shown to have a J-shaped relationship with mortality. Patients with a very low BMI have a higher mortality rate than patients with a normal BMI, and patients with a high BMI have a higher mortality rate than patients with a normal BMI
- Because the authors only formed two BMI groups (< 27, ≥ 27), they came to the erroneous conclusion that BMI is not related to mortality. If they had stratified patients into five groups of BMIs (e.g. < 20, 21-25, 26-30, 31-35, ≥ 36), they would have come to a different conclusion.
- COVARIATE AND CONFOUNDER
- Covariate - a covariate is a patient variable (ex. age, sex, race, etc.) that may or may not be related to the outcome being studied. If the covariate is related to both the exposure/risk factor and the outcome, then the covariate becomes a confounder.
- Confounder - A confounder is a covariate that is related to both the outcome and the exposure/risk factor. A confounder may either increase or decrease the likelihood of the outcome.
- IMPORTANT POINTS ABOUT DRUG INTERACTIONS
- DRUG INTERACTIONS ARE CHALLENGING
- Information on drug interactions can be difficult to assimilate
- Certain drug interactions and metabolic pathways are well-documented, while many are not
- New drugs
- When a new drug is being developed, the FDA requires that it be tested for drug interactions with a small number of medications that are known to have significant interactions
- For obvious reasons, there is no way to test a medication in every possible drug combination that may occur
- This means most drugs come to market with incomplete drug interaction profiles
- After a medication is prescribed to a large number of people, other drug interactions are inevitably discovered
- Research
- Much of the research involving drug metabolism and drug interactions occurs in vitro meaning in a lab, or outside of the human body
- Animal models and cell cultures are often used to test drugs for metabolic pathways and interactions
- Findings from in vitro experiments do not always translate into what actually happens in the human body, or in vivo
- Evolving information
- Drug metabolism is an evolving field of medicine and pharmacology
- Researchers are just beginning to understand all the different systems that are involved in how the body metabolizes and eliminates drugs
- Cell transport systems (ex. p-glycoprotein, OAT, etc.) are a relatively new area of pharmacology and information about how these systems affect drug elimination is evolving
- SUMMARY
- Not all drug interactions are known or can be predicted
- Good information on possible drug interactions may not be available
- Not all drug interactions are significant
- Always consult your physician or pharmacist before changing your medication if you are concerned about a possible drug interaction
Factors that can make drug interactions challenging include the following:
- FDA ADVERSE EVENT REPORTING SYSTEM (AERS)
- FDA AERS
- The FDA's AERS is a database run by the FDA where adverse events thought to be related to drugs are recorded and followed
- Reporting of adverse events by the general public is voluntary
- Anyone who thinks they know of an adverse drug event can report it to the system
- When drug companies receive reports of adverse drug events, they are required to report them to the system
- Based on information received in the database, the FDA sometimes takes action on a drug. Actions may include changes to the drug labeling, restricting use of the drug, issuing warnings to healthcare professionals and patients, and removing drugs from the market.
- IMPORTANT POINTS
- When the FDA issues a warning based on information from its AERS, it often draws headlines and causes concern among patients and healthcare providers alike
- While the concern is warranted, it's important to understand that the database in no way, shape, or form constitutes a statistical analysis of the adverse event being reported
- The purpose of the AERS is to flag trends in adverse events that need to be explored further with proper studies
- It does not prove causality
- REPORTING BIAS
- Adverse event reporting is susceptible to a tremendous amount of reporting bias
- Reporting bias occurs when adverse events associated with a new drug or treatment are more likely to be reported than adverse events associated with older, more familiar drugs and treatments
- Because the prescriber is less familiar with the new drug, he may be more inclined to report an adverse event he thinks is associated with the new drug
- A prescriber may be less inclined to report an adverse event for a drug that he has used for a long time. He may be biased in thinking that the drug is safe, and therefore he does not associate it with the adverse event, or the adverse event may already be known, and he doesn't see the point in reporting it.
- Example:
- Doctor prescribes brand new drug to patient who subsequently develops pancreatitis
- Doctor may be inclined to report pancreatitis as an adverse event for the drug because it is new, and he is unfamiliar with it
- If a warning has already been issued for the drug, this may also bias his reporting of the event
- Doctor prescribes drug he has used for years to patient who subsequently develops pancreatitis
- Doctor may not consider the drug as a possible cause since he has used it for years and is comfortable with it
- In the AERS, pancreatitis is being reported for the new drug, while older drugs are not receiving adverse reports
- In reality, there may be no difference in the incidence of pancreatitis between the new drug and the older drug
- OVERDIAGNOSIS
- DEFINITION
- Overdiagnosis occurs when a condition is diagnosed that would otherwise not go on to cause symptoms or death
- Overdiagnosis has become a larger issue as the utilization of screening tests has increased
- Overdiagnosis is a particular problem in cancer screening since almost all cancers are treated. Treatment of all cancers leads to overtreatment in some cases. Overtreatment can adversely affect a person's quality of life.
- Example:
- Patient A is a 60 year-old man with slow-growing prostate cancer and heart failure
- Patient B is a 60 year-old man with slow-growing prostate cancer and heart failure
- Patient A has a screening PSA test that detects his prostate cancer. Patient A then undergoes radiation and androgen deprivation therapy. Patient A develops impotence, hot flashes, and gynecomastia from the treatment. He dies five years later from his heart failure.
- Patient B is never screened for prostate cancer and never knows that he has it. He dies six years later from heart failure
- Patient A was overdiagnosed because his prostate cancer would not have led to his eventual death. The treatment of his prostate cancer may have hastened his death, and his quality of life was adversely affected.
- MEASURING OVERDIAGNOSIS
- Overdiagnosis is best measured in comparative clinical trials where one group is screened and another group is not screened
- Consider a randomized controlled trial that compared a screening test for breast cancer in one group (screened) to no screening in another group (control). At the conclusion of the screening phase of the trial, the screening group will have more breast cancer diagnoses than the control group, because screening tests advance the time of cancer diagnosis.
- If there is no overdiagnosis, then as time goes by, the control group will catch-up with the screened group, and the number of breast cancers diagnosed in each group will be equal
- If overdiagnosis has occurred, then the control group will have a lower breast cancer incidence than the screening group. This means the screening group had cancers detected that would have never led to clinically detectable disease.
- ISSUES WITH OVERDIAGNOSIS
- Measuring overdiagnosis in a trial can be difficult because it requires many years of follow-up. In order to accurately predict the incidence of cancer in each group, the length of follow-up must allow for slow-growing cancers to become clinically apparent. This can take up to 7 or more years in some cases.
- At the time of diagnosis, it is often not possible for a doctor to know if the cancer will cause the patient to die, or if the patient will die from something else
- Overdiagnosis assumes that the undiagnosed cancer did not contribute to the patient's death or morbidity [3]
- BIAS WITH OVERDIAGNOSIS
- Overdiagnosis can lead to bias in survival statistics
- If a screening test detects a condition in a significant number of people who will never die from the condition (overdiagnosis), then it will inflate the survival advantage of the screening test
- The link below is to an illustration from Wikipedia that demonstrates this phenomenon
- As detailed in the illustration, the screening test detected a large number of patients who did not die from lung cancer (referred to as "pseudodisease"). This inflated the 10-year survival for the screened group.
- Screening tests can also inflate survival statistics because they diagnose a condition earlier. For example, assume a cancer has absolutely no effective treatment. Patients who are screened for the cancer may have it diagnosed years before patients who are not screened. The screened group will have a longer "survival time" simply because they were diagnosed earlier. In reality, screening had no real effect on cancer mortality.
- PMID
- PMID is short for "PubMed Identification #"
- PubMed is an online resource developed and maintained by the National Center for Biotechnology Information
- PubMed comprises over 20 million citations for biomedical literature from MEDLINE, life science journals, and online books
- Citations on this site are listed by their PubMed Identification number. Example - PMID 123400978
- Citations can quickly be retrieved by searching PubMed with the ID number
- PRIMARY PREVENTION
- Primary prevention involves taking measures (ex. medications, exercise, weight loss) to prevent the occurrence of an event (ex. heart attack) or illness (ex. cancer) before the underlying disease has been established in the patient
- Primary prevention differs from secondary prevention where the underlying disease has already been established
- Example:
- Patient has never had a heart attack, but he is concerned about it because his father had one
- Patient starts taking aspirin every day to prevent a heart attack (Primary prevention)
- SECONDARY PREVENTION
- Secondary prevention involves taking measures (ex. medications) to prevent the recurrence of an event (ex. heart attack) or illness (ex. cancer) after the underlying disease has already been established in the patient
- Secondary prevention differs from primary prevention where the underlying disease has not been established in the patient
- Example:
- Patient has a stroke
- Patient then starts taking aspirin every day to prevent another stroke (secondary prevention)
- BIBLIOGRAPHY
What is PMID? PI = Manufacturer's Package Insert
- # PMID
1 - Fundamentals of Biostatistics 7th ed.
2 - 9832001 RR paper
3 - 20413742
4 - 10355372
5 - 26501539 - PSM JAMA
6 - 17909367 - PS inverse weight
7 - 22401135 - Regression dilution bias
8 - 29164242 - Mendelian Randomization, JAMA (2017)