Medical studies

CASE-CONTROL STUDY

Overview

A case-control study is an observational study where researchers select a number of subjects with an outcome of interest (cases), and a number of subjects without the outcome of interest (controls). The prevalence of exposure to a risk factor is then compared between the two groups.
If the cases have a much higher rate of exposure to the risk factor than the controls, it may mean that the outcome is related to the risk factor

Example

To evaluate if cell phones cause brain cancer, researchers select two groups of subjects
Case group : 100 people with brain cancer
Control group: 100 people without brain cancer
Researchers then compare cell phone use between the two groups
If cell phone use is much higher in the case group then there may be a link between brain cancer and cell phone use

Advantages

Much easier and cheaper to perform than a randomized controlled trial
With modern medical databases, a large number of subjects can be evaluated with very little effort
Outcomes with a low incidence can be evaluated. This is typically not feasible with a randomized controlled trial.
Exposures that are not ethical in a randomized controlled trial can be evaluated. For example, no randomized controlled trial is going to randomize patients to smoking or other known carcinogens.

Disadvantages

Does not prove causality
Data for patient variables may be lacking or missing and therefore must be estimated
Difficult or impossible to control for confounding
Patients are not randomized so there is no way to control for unmeasured covariates and hidden bias

COHORT STUDY

Overview

A cohort study is an observational study that is used to evaluate whether an exposure is associated with a particular outcome
In a cohort study, two groups of subjects are formed based on exposure to a factor. One group has been exposed to a factor, and the other group has not been exposed to the factor.
For other variables (ex. age, sex, race, etc.), the groups are matched as closely as possible
Neither group has the outcome at the beginning of the study
The two groups are followed for a length of time, and the incidence of the outcome is compared between the two groups
A higher incidence in the exposed group may mean a link exists between the factor and the outcome

Example

Researchers want to evaluate if smoking causes bladder cancer
They identify 2 cohorts of people:
Exposed cohort: 100 smokers
Control cohort: 100 nonsmokers
They follow the two cohorts for a number of years and compare the incidence of bladder cancer between them

Advantages

Much easier and cheaper to perform than a randomized controlled trial
With modern medical databases, a large number of subjects can be evaluated with very little effort
Outcomes with a low incidence can be evaluated. This is typically not feasible with a randomized controlled trial.
Exposures that are not ethical in a randomized controlled trial can be evaluated. For example, no randomized controlled trial is going to randomize patients to smoking or known carcinogens.

Disadvantages

Does not prove causality
Data for patient variables may be lacking or missing and therefore must be estimated
Difficult or impossible to control for confounding
Patients are not randomized so there is no way to control for unmeasured covariates and hidden bias

META-ANALYSIS

Overview

A meta-analysis is a statistical study where the results from a number of related studies are pooled together and analyzed in order to draw conclusions from a broader set of data
A meta-analysis may pool the results of cohort studies, case-control studies, randomized controlled trials, or a combination of the three. Cohort and case-control studies may be combined in a meta-analysis, but it is not common (nor recommended) to combine randomized controlled trials with other types of studies

Advantages

Often times, different studies on the same condition will come to different conclusions. A meta-analysis is a way to combine the results of these studies so that an overall effect can be estimated.
If a trial has a small number of subjects, it may not be powerful enough to find a statistically significant result. A meta-analysis allows small trials to be pooled together so that they have sufficient power to find a significant effect if one exists.
The overall effect found in meta-analyses is often less than what is seen in individual trials. When done properly, the effect in the meta-analysis is more likely to represent the true effect in a large population of patients.

Disadvantages

Studies differ in length, design, intervention, population characteristics, and outcome measures. This can make it difficult to determine which studies to include.
Combining studies that differ in design may lead to conclusions that are invalid
Meta-analyses are subject to publication bias. Publication bias occurs when only studies that found a significant effect or a large effect were published.
Meta-analyses are only as good as the studies they evaluate. If there are not enough good studies to perform a valid meta-analysis, the authors should be willing to concede this and abandon the meta-analysis.
It can be argued that the results from a large, randomized controlled trial are more valid than the results of a meta-analysis on the same topic

NESTED CASE-CONTROL STUDY

Overview

A nested case-control study is a case-control study where the subjects for the study are sampled from a cohort of patients that has been observed for a period of time. The study is said to be "nested" within the cohort.
Researchers form two groups from the cohort of patients. One group has the outcome of interest (cases), and one group has not developed the outcome (controls). Exposure to a risk factor can then be compared between the two groups. If the cases have a significantly greater exposure to the risk factor, then the risk factor may be related to the outcome.
When selecting the control group in a nested case-control study, the researchers typically select 1 - 4 controls per case as opposed to comparing cases to the entire group of controls. Controls are selected that match the cases on a number of measured variables (ex. age, medical conditions, sex, etc). This helps to control for confounding.

Example

Researchers have observed a cohort of 100,000 women for 10 years
At the start of the study, all women gave blood samples
Over the course of the 10 years, 2000 women developed breast cancer
The researchers are interested in finding out if there is a link between mercury exposure and breast cancer
The researchers decide to test the blood samples for mercury levels and compare rates of breast cancer between women with high levels and those with normal levels
Testing 100,000 blood samples for mercury levels would cost about $2 million dollars
To save money, the researchers decide to do a nested case-control study where they select 2 controls for every case. Controls are selected that match the cases for a number of variables. This way they only have to test 6000 blood samples (2000 cases and 4000 controls).

Advantages

In most cases, the nested case-control study is much cheaper to perform than a full case-cohort study
By selecting controls that match the cases on a number of covariates, there is some control for confounding and statistical efficiency should be improved

Disadvantages

Because the entire cohort is not used (for the controls), some precision is lost in the statistical measures, and the confidence intervals for the study outcomes will be wider
Like regular case-control studies, nested case-control studies are observational studies and have all the same limitations

OBSERVATIONAL STUDY

Overview

An observational study is a study where patients and outcomes are observed and no intervention is applied by the investigators. In observational studies, groups of subjects are formed according to specific exposures (cohort study) or according to specific outcomes (case-control study). This is in contrast to a randomized controlled trial where subjects are randomly assigned to a specific group and then followed.
Observational studies may be prospective (groups are identified and then observed over time) or retrospective (groups are identified and outcomes are compared)

Advantages

Much easier and cheaper to perform than a randomized controlled trial
With modern medical databases, a large number of subjects can be evaluated with very little effort
Outcomes with a low incidence can be evaluated. This is typically not feasible with a randomized controlled trial.
Exposures that are not ethical in a randomized controlled trial can be evaluated. For example, no randomized controlled trial is going to randomize patients to smoking or known carcinogens.

Disadvantages

The main disadvantage of observational studies is that there is no way to control for hidden bias and unmeasured covariates. In randomized controlled trials, the process of randomization helps to alleviate the effect of these issues.
Observational studies do not prove causality

RANDOMIZED CONTROLLED TRIAL (RCT)

Overview

A RCT is a prospective study where subjects are randomly assigned to either a treatment group or a control group
In most studies, the two groups are matched as closely as possible by variables such as age, medical problems, lab values, blood pressures, etc. If the study is very large (ex. > 500 subjects), then these factors will often balance automatically through randomization.
Randomization is very important because it is the only way to control for unmeasured covariates. Measured variables (ex. weight, age, medical history, etc.) can be adjusted for, but there may be variables that are not measured because the researchers are not aware that they can affect the study outcome. If the study is randomized and large enough, then unmeasured covariates should be distributed equally between the groups, and they will not bias the outcome.
Randomization also prevents hidden bias. In observational studies, there is no way to prevent or control for hidden bias.
In a RCT, the treatment group is given an intervention (drug or procedure)
The control group is not given the intervention, or it is given a placebo or sham procedure
The effect of the intervention is then measured over a predetermined time period by a predetermined set of outcomes
RCTs are considered the gold standard for clinical trials, and if done correctly, their findings are much more consequential than those of other study designs (ex. cohort studies, case-control studies)
The best RCTs are placebo-controlled and double-blinded when possible

Advantages

Able to control for confounding from unmeasured covariates
Hidden bias can be prevented
Criteria and outcomes can be clearly defined and accurately measured
Large RCTs will give the best measure of an intervention's effect in the intended population

Disadvantages

Expensive and time-consuming
Can take many years to complete
If an outcome has a low incidence, RCTs are typically not feasible because the study size would need to be very large, and it would make the trial very expensive
RCT are not ethical in some situations. For example, randomizing subjects to smoking or other interventions that are known to be harmful.

REGISTRY-BASED RANDOMIZED CONTROLLED TRIAL (RRCT)

Overview

RRCTs are randomized controlled trials that use large medical registries/databases to help streamline and simplify data collection
Some countries and health organizations maintain large databases of information on patients in their healthcare system. This information can be used in performing a RRCT.
In RRCTs, registry information is used to identify eligible patients, log baseline information, determine randomization, and track outcomes

Advantages

Less expensive and time-consuming than a traditional RCT
The centralized enrollment process makes it easier to enroll and recruit patients
The registry information helps to identify more eligible patients
Follow-up rates are higher because outcomes are tracked in the registry and investigator-specific follow-up is not required
Results may be more pragmatic since the studies occur during the course of typical patient care and do not include specific methodology

Disadvantages

Outcome events are based on registry entries, and therefore, they are not adjudicated and must typically remain definitive or broad (e.g. overall mortality)
Blinding and placebo control are difficult to implement
Not amenable to all types of outcome data

DATABASE AND REGISTRY STUDIES

Overview

Certain countries, HMOs, and insurance companies maintain large databases of medical information on their citizens/patients/clients
This information is often used to conduct observational studies, and more recently, it has been utilized in conducting randomized controlled trials (see registry-based randomized controlled trials)
Observational studies using medical databases are very popular because they allow investigators to examine a large amount of information with very little effort, expense, and time. These studies have many weaknesses though, and their results do not prove causality.

Advantages

Cheap and easy to perform
Able to evaluate a large amount of information with very little effort

Disadvantages

Diagnostic codes are often used as criteria. Use of these codes is not well-standardized.
Pertinent patient information is often missing or not recorded and must be estimated

PAIRED AVAILABILITY DESIGN STUDY

Overview

Paired availability design studies are a type of observational study that uses the time period before an intervention is available (or widely available) as a historical control
In a paired availability design study, outcomes are measured in all patients who are candidates for an intervention both before and after the intervention is widely available. The two sets of outcome data are then compared. All candidates are included in the analysis whether they receive the intervention or not (this includes both before widespread availability and after). By including all patients in the data set, selection bias is limited.
Data for paired availability design studies may be collected prospectively, retrospectively, or through a combination of the two

Advantages

Cheaper and easier than randomized controlled trials
Data collection may be facilitated by a registry
Less prone to selection bias than a cohort study

Disadvantages

In order for the results to be valid, a number of assumptions must hold true including:

Stable population meaning the before and after patient population does not change in some significant way. Because of this, these studies are typically performed at hospitals that serve a specific patient population or geographic region.
Stable treatment meaning other treatments or care does not change significantly between the before and after periods
Stable evaluation meaning diagnostic and monitoring modalities are not different between the two periods. Improvements in diagnosis and monitoring can improve outcomes.
Stable preference meaning new data/information does not influence whether a patient chooses the intervention
The availability of the intervention should not affect its efficacy. This applies when some people in the before group receive the intervention. Increased availability may lead to earlier treatment in the after group which can affect outcomes. Interventions that have a learning curve may not perform as well in the before group when compared to the after group. [9]

AS-TREATED ANALYSIS

Overview

In an as-treated analysis, outcomes for subjects in a study are counted toward the treatment they actually receive. If a subject is randomized to one treatment group, but crosses over during the study and receives a competing treatment, then their outcome counts toward the competing treatment.
This differs from an intention-to-treat analysis where a participant's outcome counts toward their original assigned group regardless of whether they crossover
As-treated analyses are often performed in studies where there is a high amount of crossover between treatment groups
On the surface, as-treated analyses seem to make sense, but in reality, they are susceptible to a large amount of bias

Example

A trial enrolls 200 people with sciatic nerve pain
100 people are randomized to physical therapy, and 100 people are randomized to back surgery
The subjects are followed for one year. At the end of one year, differences in back pain and disability are compared.
During the course of the year, 30 people assigned to physical therapy end up undergoing back surgery, and 10 people assigned to surgery never actually have surgery and instead receive physical therapy (crossovers)
The researchers decide to do an as-treated analysis of their data so that the crossovers will be counted toward the therapy they actually received

Bias in as-treated analyses

In this example, all the patients who crossed-over are counted toward the treatment they actually received: 30 people from the physical therapy group are counted toward surgery, and 10 people from the surgery group are counted toward physical therapy
This appears to be logical since the outcomes for crossovers will count toward the treatment they actually received. In reality, the analysis will likely be biased.
People often enter trials because they hope to receive one of the treatments being offered. Trial protocols often allow for crossover at some point because this makes it easier to get people to participate. If a patient believes the only thing that will make his sciatic pain better is surgery (the perception that an invasive treatment is actually doing something where other treatments are not), and they are assigned to the physical therapy group, then that patient may be disappointed and is already biased toward not improving. The same patient may later crossover and receive surgery and only then get better because "something has been done." If this patient is counted toward the surgery group, then bias has been introduced.
Another issue with as-treated analyses is that randomization is compromised. Randomization is one of the most important statistical methods that protects against bias and confounding (see randomization below)

COMPETING RISKS

Survival analysis

The Kaplan–Meier estimator is a popular way to evaluate outcomes in prospective studies
The method was originally developed to evaluate survival over a period of time with overall mortality being the primary outcome of interest
Use of the method has expanded, and it is often used to evaluate outcomes other than overall mortality
It's important to understand the concept of censored patients in survival analysis. All patients entering the study are assumed to experience the primary outcome at some point. If the patients are censored (lost from the study for some reason without experiencing the outcome), they are still treated as having the primary outcome at some point after they are censored. Patients who make it to the end of the study without being censored or having the primary outcome are assumed to have the primary outcome at some point after the study ends.
This concept holds true if the primary outcome is overall mortality, because all patients will die at some point. If the primary outcome is something other than overall mortality, then the assumption that all patients will eventually experience the outcome or that they are always at risk for the outcome is not necessarily true.

Example

A study is designed to look at the risk of hip replacement in elderly patients over time
Patients are followed for a period of time and the Kaplan-Meier method is used to estimate risk with hip replacement being the primary outcome
Because the patient population is elderly, a significant number of subjects die during the study without ever having a hip replacement
The dead subjects are no longer at risk for a hip replacement, yet in the Kaplan-Meier analysis, they are still considered to be at risk for a hip replacement
The Kaplan-Meier analysis will overestimate the risk of a hip replacement in this population

Competing risk

In the example above, death is a competing risk for hip replacement
In order to correct for the fact that patients who die are no longer at risk for a hip replacement, a different kind of analysis needs to be performed
In most cases, that analysis is the cumulative incidence competing risk (CICR) estimate
CIRC analysis accounts for events that affect the risk of the primary event, and it lowers the probability of the primary event when a competing event occurs. In CIRC analysis, patients experiencing the competing event are no longer considered at risk for the primary event.

DOUBLE-BLIND TRIALS

In double-blind trials, neither the subject nor the researcher evaluating the subject knows which intervention the subject is receiving
In single-blind trials, only the subject is blinded as to what intervention they are receiving
Double-blinding helps prevent bias on the part of the subjects and the researchers
When possible, randomized controlled trials should be double-blinded
Double-blinding is not always possible. For example, when comparing a surgical intervention to a medical one.

INSTRUMENTAL VARIABLE ANALYSIS

Overview

A major limitation of observational studies is the inability to detect and control for unmeasured confounders. Randomization, which is not possible in observational studies, is the only method that can completely eliminate the effects of unmeasured confounders.
A technique called instrumental variable analysis has been developed to help reduce the effects of unmeasured confounders in observational studies. An instrumental variable is a patient variable that is strongly associated with the treatment a patient receives while at the same time having no association with the outcome being measured. For example, a new diabetes drug may only be covered by a few insurance companies. Patients with insurance that covers the drug are far more likely to receive the drug than patients without coverage. In this case, insurance drug coverage is an instrumental variable because it is strongly associated with treatment received but should have no association with diabetic outcomes like A1C values and CVD unless the new drug is truly superior to established therapies.
Once an instrumental variable is identified, researchers can "randomize" patients by comparing outcomes between patients grouped by the instrumental variable. [11]

Example

Researchers want to see if gallbladder removal for cholecystitis within 12 hours of presentation is associated with better outcomes than removal beyond twelve hours
They collect data on patients who have had their gallbladder removed and divide them into 2 groups based on whether they had it removed within 12 hours or after 12 hours
The two groups differ on a number of covariates so the researchers decide to look for an instrumental variable. They find that patients admitted on the weekend were far more likely to have their gallbladder removed after twelve hours when compared to patients admitted during the week. Since day of hospital admission should not have a direct effect on outcomes, they decide to use it as their instrumental variable. An analysis of the data is then performed with patients grouped by whether they were admitted on the weekend or on a weekday.

INTENTION-TO-TREAT

Overview

Intention-to-treat is a method in randomized controlled trials where the outcomes for all participants who are randomized to a certain treatment or control group are counted toward that group regardless of whether they complete their assigned treatment protocol or not
In an intention-to-treat analysis, outcomes for patients who stop treatment or drop out of treatment are still counted toward their original randomized group. Even outcomes from patients who crossover and receive a competing treatment are still counted toward the original assigned group.
Intention-to-treat analysis is the gold standard for analysis, because other forms of analyses (see per-protocol analysis, on-treatment analysis, modified intention-to-treat analysis, and as-treated analysis) can introduce bias into a study

Example

A trial enrolls 200 people with sciatic nerve pain
100 people are randomized to physical therapy, and 100 people are randomized to back surgery
The subjects are followed for one year. At the end of one year, differences in back pain and disability are compared.
During the course of the year, 30 people assigned to physical therapy end up undergoing back surgery, and 10 people assigned to surgery never actually have surgery and instead receive physical therapy (crossovers)

Intention-to-treat

In an intention-to-treat analysis, outcomes for the 30 people who crossed-over from the physical therapy group and received surgery would still be counted toward the physical therapy group
Outcomes for the 10 people who never underwent surgery will still be counted toward the surgery group

Advantages

Limits bias - In the above example, if an as-treated analysis was performed where all the patients who crossed-over were counted toward the group they ended up in, then bias would be introduced. People often enter trials because they hope to receive one of the treatments being offered. Trial protocols often allow for crossover at some point because this makes it easier to get people to participate. If a patient believes the only thing that will make his sciatic pain better is surgery (the perception that an invasive treatment is actually doing something where other treatments are not), and they are assigned to the physical therapy group, then that patient may be disappointed and is already biased toward not improving. The same patient may later crossover and receive surgery and only then get better. If this patient is counted toward the surgery group, then bias has been introduced.

Disadvantages

High crossover or dropout rates can bias a study towards the null (no effect) - If a study has a high crossover or dropout rate, then the study may be biased towards the null, or "no effect." In the above example, the 30 patients who crossed over to surgery are still counted toward physical therapy. If a true difference in outcomes exists between the two treatments, it will be harder to discern since positive surgery outcomes may count toward physical therapy. In the end, there is no perfect way to deal with crossovers and dropouts, but the gold standard for evaluating subjects in a trial is intention-to-treat.

MENDELIAN RANDOMIZATION

Overview

Mendelian randomization is an observational study technique that uses variations in genetic makeup to determine if a risk factor is associated with an outcome
For example, in population studies, HDL cholesterol levels are inversely associated with heart disease risk; as HDL levels increase, risk of heart disease goes down
A number of genetic variants are associated with HDL levels. Some variants raise levels and others lower them. Through Mendelian inheritance, these variants are distributed randomly across the population. If a researcher wants to determine if HDL levels are directly associated with heart disease risk, they can form patient cohorts based on genetic variants that affect HDL levels. Cohorts with HDL-promoting variants will have higher HDL levels simply because of their genetic makeup. Other variables that affect HDL levels (e.g. exercise, diet) will be distributed equally among the individuals making the cohorts "randomized" for potential confounders. If a causal relationship between HDL levels and heart disease exists, then HDL-promoting variants will be associated with lower heart disease risk. A number of studies have actually looked at this, and they have found that HDL-promoting variants are not associated with lower heart disease risk calling into question the association between HDL and heart disease.
In order for Mendelian randomization to work, the genetic variants being studied must only be associated with the risk factor in question and not other variables that may affect the outcome. For example, if HDL-promoting variants are also associated with lower LDL levels or lower blood pressure, then confounding exists and incorrect conclusions may be drawn. When a variant affects the outcome through pathways other than the risk factor being studied, this is referred to as pleiotropy.

Advantages

Although Mendelian randomization studies are observational cohort studies, cohort formation based on genetic variants offers some degree of randomization
Mendelian randomization studies may be used to do an exploratory analysis of a possible intervention that affects a risk factor. For example, drugs have been developed that raise HDL levels. Before doing expensive randomized controlled trials to see if the drugs lower heart disease risk, researchers may do a Mendelian randomization study to see if HDL levels are directly related to heart disease risk.

Disadvantages

If pleiotropy exists, results are not valid. Pleiotropy may not be easy to detect.
Statistical power can be difficult to achieve and often requires the use of multiple variants which increases the risk of pleiotropy [8]

MODIFIED INTENTION-TO-TREAT

Overview

Modified intention-to-treat is when investigators in a trial use an intention-to-treat analysis, but they modify who they include in this analysis

Example

A study wants to compare the effects of a new drug on postmenopausal vaginal dryness
100 women are randomized to the drug, and 100 women are randomized to placebo
At the start of the study, all women have a vaginal smear performed
After 6 months, changes in vaginal symptoms are measured
The researchers present their significant findings in a "modified intention-to-treat" analysis where only women who had ≤ 5 superficial cells on their vaginal smear were included in the analysis

Modified intention-to-treat

In the above example, the researchers decided to only include a subset of the randomized women in their analysis
A modified intention-to-treat analysis should raise red flags because it may mean the researchers did not find a significant result with their intention-to-treat analysis, and they are now digging for a significant results by changing the study's criteria. This is bad form because it can introduce bias.
A reviewer should also ask themselves if the modified criteria is clinically relevant. In the above example, most doctors are not going to do a vaginal smear before prescribing a menopause drug, so the fact that it may work in this subset of women is irrelevant.
In some cases, a modified intention-to-treat analysis has little impact. For example, researchers do a modified intention-to-treat analysis that only includes randomized patients who received at least one dose of a study drug.

ON-TREATMENT ANALYSIS

Overview

On-treatment analysis is when investigators only include data from patients while they are taking their assigned treatment. If patients stop their assigned treatment for a period, data from that time period is excluded.
On-treatment analysis is similar to per-protocol analysis
On the surface, on-treatment analyses makes sense - if someone stops treatment then their results should not count since the intervention can not help them if they do not take it
In reality, on-treatment analysis introduces bias (See Examples below)

Example #1

A study is comparing a new drug to prevent migraines to an older drug that prevents migraines
100 people are randomly assigned the new drug, and 100 people are randomly assigned the old drug
The patients are supposed to take the drugs for 6 months, and the number of migraines in that time period are recorded
30 people taking the new drug and 10 people taking the old drug stop taking it before the end of the trial
The researchers decide to do a on-treatment analysis that excludes data from people while they stopped taking their drugs
In this circumstance, the on-treatment analysis has introduced bias
People stop study medications for a variety of reasons (e.g. side effects, lack of perceived efficacy, etc.). If a on-treatment analysis is performed, the study has been biased towards "responders" and it does not reflect the true efficacy of the drug in the intended population. This may also lead to spurious significant results.

Example #2

A study is comparing aspirin to placebo for the prevention of deep vein thrombosis
1500 people are given aspirin and 1500 people are given placebo. The 2 groups are followed for 2 years and the rate of deep vein thrombosis is compared.
300 people in the aspirin group stopped taking their aspirin during the trial
The researchers do a on-treatment analysis that excludes data from people while they stopped their aspirin. The analysis shows that when compared to placebo, patients who actually took their aspirin had better outcomes than the whole aspirin group (takers and non-takers).
On the surface, this analysis seems legitimate. The outcome is objective, and for the most part, people tolerate aspirin well, so comparing people who actually took the drug to the controls makes sense.
The problem is that past studies have shown that outcomes in the placebo group also differ between compliant and noncompliant patients. Patients who are compliant in taking their placebo are also more likely to have better outcomes than patients who do not (even for objective outcomes like heart attack). This phenomenon likely occurs because of differences in unmeasured confounders between compliant and noncompliant patients. Because of this phenomenon, the on-treatment analysis leads to erroneous results and is biased.

PER-PROTOCOL ANALYSIS

Overview

In a per-protocol analysis, only subjects who completed the protocol they were assigned to are included in the statistical analyses
Patients who vary from the protocol (ex. stop medication), drop out of the study, or crossover to another therapy are not counted in the analyses
Per-protocol analyses are similar to on-treatment analyses
Per-protocol analyses introduce bias in many cases (see Example #2 and Example #3 below)
Per-protocol analyses may be more meaningful in conditions that are difficult to treat and/or when there are few treatment options (see Example #1 below)

Noninferiority trials

In noninferiority trials, per-protocol analyses should be reported with intention-to-treat analyses
Noninferiority trials often compare new treatments to standard treatments (ex. new anticoagulant drug compared to warfarin for atrial fibrillation)
During the trial, patients assigned to the new treatment may be allowed to crossover to the standard treatment for various reasons (e.g. side effects, new indication, etc.). In intention-to-treat analyses, crossovers can bias a study towards the null, or in the case of noninferiority trials, towards noninferiority.
Per-protocol analyses should be reported with intention-to-treat analyses to help measure the sensitivity of the results to protocol violations

Example #1

A study is comparing a new drug + standard therapy to standard therapy alone in a difficult to treat cancer
The new drug has many side effects that make it intolerable for a number of patients. These patients stop taking the drug.
The researchers decide to to do a per-protocol analysis to see if the new drug improved survival in patients who actually completed therapy
In this case, the per-protocol analysis is meaningful because it will show whether people who can tolerate the therapy actually benefit from it

Example #2

A study is comparing a new drug to prevent migraines to an older drug that prevents migraines
100 people are randomly assigned the new drug, and 100 people are randomly assigned the old drug
The patients are supposed to take the drugs for 6 months, and the number of migraines in that time period are recorded
30 people taking the new drug and 10 people taking the old drug stop taking it before the end of the trial
The researchers decide to do a per-protocol analysis that would exclude the people who stopped taking their drugs
In this circumstance, the per-protocol analysis has introduced bias
People stop study medications for a variety of reasons (e.g. side effects, lack of perceived efficacy, etc.). If a per-protocol analysis is performed, the study has been biased towards "responders" and it does not reflect the true efficacy of the drug in the intended population. This may also lead to spurious significant results.

Example #3

A study is comparing aspirin to placebo for the prevention of deep vein thrombosis
1500 people are given aspirin and 1500 people are given placebo. The 2 groups are followed for 2 years and the rate of deep vein thrombosis is compared.
300 people in the aspirin group stopped taking their aspirin during the trial
The researchers do a per-protocol analysis that excludes the people who stopped their aspirin. The analysis shows that when compared to placebo, patients who actually took their aspirin had better outcomes than the whole aspirin group (takers and non-takers).
On the surface, this analysis seems legitimate. The outcome is objective, and for the most part, people tolerate aspirin well, so comparing people who actually took the drug to the controls makes sense.
The problem is that past studies have shown that outcomes in the placebo group also differ between compliant and noncompliant patients. Patients who are compliant in taking their placebo are also more likely to have better outcomes than patients who do not (even for objective outcomes like heart attack). This phenomenon likely occurs because of differences in unmeasured confounders between compliant and noncompliant patients. Because of this phenomenon, the per-protocol analysis leads to erroneous results.

PLACEBO-CONTROLLED TRIALS

In a placebo-controlled trial, subjects who are not randomized to an active treatment (controls) are given a fake or sham treatment instead (Ex. an inert pill or a procedure that is faked)
Placebo-control prevents subjects in a trial from knowing whether they are receiving the active treatment. This prevents bias that can arise from a subject's expectations that an intervention works.
Placebo-control is very important in trials where outcomes are subjective (ex. pain, disability, frequency of events that are not biologically measurable)
It should be applied when possible in trials with objective outcomes

PROPENSITY SCORE MATCHING (PSM)

Overview

Propensity score matching (PSM) is a statistical technique used in observational studies. It helps to control for selection bias.
In PSM, subjects who received a treatment are matched with subjects who did not receive the treatment based on the probability of receiving the treatment. The probability of receiving the treatment is reflected in the propensity score. The propensity score is calculated by combining the probability of receiving treatment for a number of measured covariates.
When subjects with the same propensity score are compared, selection bias is limited

Procedure

Determine the probability of receiving the treatment for each measured covariate in the study (ex. age, sex, medical conditions, lab values, etc)
For each subject, combine the probabilities of receiving the treatment based on their individual covariates. Derive the propensity score from this combination (usually with logistic regression)
Create groups of subjects with the same propensity score. Subjects with the same propensity score will typically match closely on a number of covariates.
Compare treated subjects to untreated subjects within these groups. There are a number of ways of doing this. See Propensity score techniques below.
Perform multivariate analysis on the propensity-matched sample

Propensity score techniques

Direct comparison

In direct comparisons, treated and untreated individuals with the same score are matched and compared. Matching can be one-to-one, one-to-as many that match, and so on.
Groups can also be stratified by propensity score and then strata can be compared directly
A major disadvantage of direct matching is that depending on the method used, all of the available data may not be utilized. In addition, if strata are compared, residual confounding may occur.

Propensity score as a covariate

The propensity score can be used as a covariate in a multivariate model
In this case, the outcome is the dependent variable, and the independent variables are the exposure/treatment and the propensity score
If the probability of receiving the treatment (propensity score) is strongly associated with the outcome, then covariate adjustment for the propensity score will decrease the strength of association between the treatment and the outcome

Propensity score weighting

Propensity score weighting is a method where each individual in the sample is weighted by the inverse of their propensity score
Using this method, subjects who are more likely to receive the treatment carry less weight, and subjects who are less likely to receive the treatment receive the most weight
With inverse weighting, covariates that are associated with receiving the treatment/exposure (higher propensity score) are neutralized while covariates that are weakly associated with receiving the treatment (low propensity score) gain value. This helps to neutralize selection bias, creating a sample where covariate distribution is independent of treatment selection. [5,6]

Advantages

PSM is helpful when a large number of covariates are measured because it creates a comparison that minimizes selection bias regardless of whether patients match exactly on all of the covariates
PSM simulates randomization of treatment for measured covariates (not for unmeasured covariates)

Disadvantages

PSM is an observational technique, and it cannot control for unmeasured covariates and hidden bias

RANDOMIZATION

Definitions:

Measured variables - patient variables (ex. age, sex, medical problems) that are measured before a trial begins
Unmeasured covariates - patient variables that are not measured, but may be related to (and have an influence on) the outcome being evaluated. There may be several reasons these variables are not measured. Researchers may not be aware that they can affect the outcome, or the variables may be expensive to obtain (ex. genetic profiles on every patient enrolled).
Unmeasured confounder - if an unmeasured covariate is associated with both the exposure and the outcome measure, then it becomes an unmeasured confounder (see covariates and confounders below)
Hidden bias - any form of bias that may influence the outcome of a trial (e.g. researchers assigning healthier patients to the treatment group, a form of selection bias)

Randomization

Randomization is a process in clinical trials where patients are randomly assigned to different groups (ex. treatment and placebo)
Ideally, the person enrolling the patients in the trial should have no idea which group the patient will be assigned to
Randomization is a critical component of controlled trials because it is the only way to control for unmeasured covariates and confounders. Randomization also prevents hidden bias.
If a trial is large enough (> 500 subjects), randomization will typically create groups that are equal for a number of measured variables (e.g. age, sex, medical problems). It will also equally distribute unmeasured covariates and confounders between the groups so that these unmeasured variables do not affect the outcome.
Randomization is the main reason randomized controlled trials are the gold standard for clinical studies. Observational studies cannot randomize participants; therefore, they are always subject to unmeasured covariates, confounders, and hidden bias.

SINGLE-BLINDED TRIALS

In single-blinded trials, only the patient is blinded as to what treatment they are receiving. The investigators or treating doctors are aware of the treatment given.
Contrast with double-blinded trials where the investigator and subject are blinded to the treatment allocation
Double-blinding is preferred when possible because it prevents investigator bias

ABSOLUTE VS RELATIVE RISK

Overview

When discussing absolute risk and relative risk, it helps to consider the two measures together
Absolute risk is the overall incidence of an outcome in an entire group or population
Relative risk is the ratio of the incidence of an outcome between two groups
Relative risk only considers the proportion of the participants who have the outcome in question where absolute risk considers the entire population being studied. This can lead to some large difference between the two measures.

Conclusions

As one can see from the example above, the relative risk reduction and the absolute risk reduction can be quite different (46% vs 2.3%)
It's important to understand the difference, because medical literature will often emphasize a large relative risk reduction when the absolute risk reduction may be quite small and insignificant. From the example above, the drug company for Livelong will likely say that they "reduce the risk of heart attack by 46%." While this may be true, someone taking Livelong will only reduce their overall risk of heart attack by 2.3%.

E-VALUE

Overview

The E-value is a statistical measure that can be used in observational studies to help assess the effect of unmeasured confounders
The E-value is a risk ratio that quantifies the minimal amount of correlation (while controlling for measured covariates) that an unmeasured confounder must have with the exposure and the outcome in order to negate the observed association between the exposure and outcome
If the E-value is low (risk ratio close to 1), then it is more likely that an unmeasured confounder could affect the observed results. If the E-value is higher, then it is less likely that an unmeasured confounder could affect the results. [10]

Example

An observational study looks at the association between exercise and cancer
The study finds that exercise significantly lowers the risk of cancer (HR 0.70 95%CI [0.50 - 0.88])
The authors calculate an E-value and it comes back as 3.2. This means an unmeasured confounder would have to have at the very least, a hazard ratio of 3.2 between exercise and cancer in order for it to affect the observed results.
In the same study, the authors calculated a hazard ratio of 1.90 for the association between smoking and cancer. Since smoking is a strong risk factor for cancer, it's unlikely that an unmeasured confounder would have a stronger association; therefore, the authors conclude that the E-value supports their observed findings.

INCIDENCE AND PREVALENCE

Incidence

Incidence is the proportion of a group that develops a disease (or experiences an event) over a specified period of time

Example

100 people without diabetes are followed for 5 years
At the end of 5 years, 10 people have developed diabetes
The incidence of diabetes over the 5 years is 10%

Prevalence

Prevalence is the proportion of a group that has the disease (or has experienced an event) at a single time point

Example

100 people are selected randomly
10 of the people in the group have diabetes
The prevalence of diabetes in the group is 10%

LIKELIHOOD RATIO (LR)

Overview

A likelihood ratio is a statistical ratio used to quantify the predictive value of a medical test (typically a screening test)
The likelihood ratio is the factor by which the post-test odds of a condition being present (positive likelihood ratio) or absent (negative likelihood ratio) are increased or decreased based on the results of a test
The likelihood ratio can be multiplied by the pre-test odds of the condition being present to determine the post-test odds
Post-test odds of disease = Pre-test odds X Likelihood ratio

Mathematically, the likelihood ratio is calculated using the sensitivity and specificity of the test:

Example

Test A is a screening test for heart disease
Paul is a 60 year-old patient who based on his family history, age, and medical conditions has a probability of 25% of having heart disease

Convert 25% probability to odds = 25/75 = 0.33

Test A has a negative likelihood ratio of 0.1 and a positive likelihood ratio of 1.9

Equivalent to a Sensitivity of 95% and a specificity of 50%

If Paul has a negative test A, then his post-test odds of heart disease are 0.1 X 0.33 = 0.033
If Paul has a positive test A, then his post-test odds of heart disease are 1.9 X 0.33 = 0.627
Converting back to probability:

negative test = 0.033/1.033 = 0.032 or 3.2% probability
positive test = 0.627/1.627 = 0.385 or 38.5% probability

In Paul's case, a positive test increases his probability of disease from 25% to 38.5% - not very helpful
A negative test decreased his probability of disease from 25% to 3.2% - more helpful
This is true because the test had a high sensitivity (low false negatives) and low specificity (high false positives)

NET RECLASSIFICATION INDEX/IMPROVEMENT (NRI)

Overview

The Net Reclassification Improvement (NRI), (also called "Index") is a statistical measure used to evaluate whether adding a measure to a predictive model will improve the predictive accuracy of the model. An example would be adding Coronary Artery Calcium Scores (CACS) to the Framingham risk model to see if the addition of the CACS improves the Framingham's ability to predict a heart attack. The NRI takes into account both correct and incorrect reclassifications.

The NRI is calculated in the following manner

1. In a study, take the people who had an event or outcome (events)
2. Calculate the number of events who were reclassified into a higher risk category when the new measure was included
3. Subtract the number of events who were reclassified into a lower risk category when the new measure was added
4. Divide this number by the total number of people who had an event
5. Now take the number of people who did not have an event or outcome (nonevents)
6. Calculate the number of nonevents who were reclassified into a lower risk category when the new measure was added
7. Subtract the number of nonevents who were reclassified into a higher risk category when the new measure was added
8. Divide this number by the total number of people who did not have an event
9. Add the two proportions together and you have the NRI

The formula for the NRI is as follows:

Example

The Framingham risk model uses age, gender, total and HDL cholesterol, smoking status, and systolic blood pressure to predict a person's 10-year risk of heart attack
Researchers want to know if adding Coronary Artery Calcium Scores (CACS) to the model improves the risk prediction of the original Framingham model
A cohort of patients is formed and their Framingham 10-year risk is calculated using the original risk model
The patients also have their CACS measured at baseline
The patients are followed for 10 years and the incidence of heart attacks is measured
Each patient's predicted risk of heart attack is calculated using the original Framingham model. The risk is also calculated using a model that incorporates the CACS.
The NRI is then calculated by comparing the two models

Interpretation of the NRI

It's important to understand what the NRI actually means because it is commonly misinterpreted

In the example above, let's assume the addition of the CACS to the Framingham model had the following effect:

71 patients with a heart attack are reclassified as higher risk
24 patients with a heart attack are reclassified as lower risk
790 patients without a heart attack are reclassified as lower risk
657 patients without a heart attack are reclassified as higher risk
Total number of patients who had a heart attack - 209
Total number of patients who did not have a heart attack - 5669

The calculation of the NRI is as follows:

Some people mistakenly interpret this as the addition of CACS correctly reclassifies 25% of the population studied
This is not correct. The addition of the CACS correctly reclassified 22.5% of the population who had an event (0.225 X 209 = 47) and 2.3% of the population that did not have an event (0.023 X 5669 = 130)
The percent of the total population correctly reclassified is 177/5878 = .03 or 3%
Whether or not 3% is clinically significant will depend on the cost and convenience of the additional measure

NUMBER NEEDED TO TREAT/HARM (NNT/NNH)

Overview

The number needed to treat (often abbreviated "NNT") is the number of patients who must be treated with an intervention in order for one patient to benefit
If the intervention is a screening test, it is the number of people who would have to be screened in order for one person to benefit from screening
The NNT can also be referred to as the "Number Needed to Harm" (NNH) when talking about a side effect or adverse event
The NNT is calculated by taking the reciprocal of the absolute risk reduction
The NNH is calculated by taking the reciprocal of the absolute increase in adverse events

Example

The recent NLST trial (detailed here) found that yearly CT scan screening reduced the absolute risk of lung cancer death by 0.33% in heavy smokers
The number needed to screen is = 1/0.0033 = 303 patients
This means 303 patients would have to be screened in order to prevent 1 lung cancer death

ODDS RATIO

Overview

When most people think of comparing two outcomes, they think in terms of relative risk
Relative risk is a comparison that uses fractions. It is the most intuitive way to compare two outcomes.

Example

Drug A has been shown to reduce the risk of heart attack by 20%
Most people understand that people taking Drug A will have 20% fewer heart attacks than people not taking Drug A

Definition of odds

When discussing odds ratios, it helps to review what odds are
Odds are ratios. Odds are not fractions. This can be confusing.

Odds = the number of times that an event will occur for the number of times that it will not occur
Example

3:1 odds means the event is likely to occur 3 times for every one time it does not occur
So for every 4 tries, there will be 3 events and 1 non-event. This equals a probability of 75% (3/4).

Convert between odds and probability

Odds ratio

An odds ratio is a ratio of two odds. Relative risk (sometimes called hazard ratios) is a ratio of two fractions.
Because of this distinction, the odds ratio and the relative risk ratio do not always approximate each other. This can be confusing because many people mistakenly interpret odds ratios as relative risk ratios.
As the prevalence of an outcome increases above 10%, the odds ratio and the relative risk start to diverge. This occurs because odds above 1:1 (0.50 probability) can run to infinity where a fraction (of a whole) is always between 0 and 1. This can lead to wide discrepancies between the odds ratio and the relative risk.

Example

Let's assume a group of diabetics are followed for a year and 80% of them suffer a heart attack
A group of nondiabetics is followed for a year and 20% of them suffer a heart attack
We want to compare the risk of heart attack between the two groups
The relative risk of heart attack in the diabetic group compared to the nondiabetics is (0.8)/(0.2) = 4. Diabetics have 4 times the risk of heart attack when compared to nondiabetics. Most people understand this.
The odds ratio of a heart attack when comparing the diabetics to nondiabetics is (4)/(0.25) = 16. This differs from the relative risk. It means diabetics have 16 times the odds of a heart attack than nondiabetics. This comparison is not as intuitive for most people and will often be misinterpreted as 16 times the risk of heart attack.

Rules for interpreting odds ratios:

When the incidence of the outcome of interest is < 10% in the study population, then the odds ratio and the relative risk can be considered equivalent
When the incidence of the outcome of interest is > 10% in the study population, and the odds ratio is > 2.5 or < 0.5, then the odds ratio will tend to exaggerate the magnitude of the association. In some circumstances the odds ratio can be converted to a relative risk (see below).

Odds ratio in medical studies

Case-control studies

Odds ratios are always reported in case-control studies because risk is not measured in these studies. The case group has a 100% probability of the disease and the control group has a 0% probability. The odds of exposure to a risk factor is compared between the two groups, therefore the results are reported as an odds ratio.

Cohort studies

Cohort studies often utilize logistic regression to control for covariates. Logistic regression yields an odds ratio. In some cases, the odds ratio can be converted to a relative risk. See formula below. [2]

Converting an odds ratio to relative risk

If a cohort study uses logistic regression and reports an odds ratio, it can be converted to relative risk if the incidence of the outcome in the unexposed (control) group is known

SURROGATE ENDPOINTS

Significant clinical outcomes

A significant clinical outcome is an outcome that has a significant effect on a person's health, functioning, or well-being
Examples of significant clinical outcomes include: death, heart attack, stroke, blindness, paralysis, cancer

Surrogate endpoints

Surrogate endpoints are measures that have been shown to be related to a significant clinical outcome
Surrogate Endpoints are often used in clinical trials because they are easier to measure and require less time to demonstrate an effect than significant clinical outcomes

Surrogate endpoint	Related significant clinical outcome
Cholesterol levels	heart attack, stroke, mortality
blood pressure	heart attack, stroke, mortality
carotid intima thickness	heart attack and stroke
Hemoglobin A1C level	diabetes complications

Example

High cholesterol levels are associated with heart attacks
A trial with a new cholesterol drug will measure its effect on cholesterol levels (surrogate endpoint)
This takes far less time (weeks to months) than measuring its effect on heart attack rates (years)

Issues with surrogate endpoints

Problems arise when interventions improve surrogate endpoints but do not improve (or may even worsen) significant clinical outcomes

This may occur for several reasons:

The surrogate endpoint may be related to the outcome through a factor that the intervention does not affect
The surrogate endpoint may be a marker of disease severity as opposed to being a cause of the disease
The intervention may be harmful in a way that is unrelated to the surrogate endpoint

Examples of interventions that improve surrogate endpoints, but do not improve significant clinical outcomes:

Folic acid and Vitamin B supplements

High homocysteine levels are associated with an increased risk of heart attack
Folic acid and Vitamin B supplementation lower homocysteine levels (surrogate endpoint)
Folic acid and Vitamin B supplementation have not been shown to decrease the risk of heart attack (significant clinical outcome) in patients with high homocysteine levels

Niacin and fibrates

High HDL levels (good cholesterol) are associated with a lower risk of heart attack
Niacin and fibrates raise HDL levels (surrogate endpoint)
These drugs have not been shown to improve overall mortality (significant clinical outcome) in patients with coronary heart disease

P-VALUE

Overview

The p-value is an important statistical measure used to compare findings between two outcomes
The p-value is expressed as a fraction (ex. p=0.01) which can be converted into a percentage (ex. p=0.01 same as p = 1.0%)
The p-value has the following meaning: If there really is no difference between the two groups, the p-value is the probability that the observed difference (or one greater) occurred by chance

Example

A study compares a new drug to prevent deep vein thrombosis to placebo
1000 people take the new drug for a year, and 1000 people take placebo for a year
At the end of the trial, the new drug group had 20 fewer deep vein thromboses than the placebo group
The researchers report the following p-value for the difference between the two groups: p=0.02
This means that if there really is no difference between the two groups (the new drug is ineffective), then the probability of observing the results found in this trial is 2%

Statistical significance

In most cases, a p-value < 0.05 is considered statistically significant. In the medical literature, it is often stated that the difference was "significant."
This cutoff value is somewhat arbitrary. For example, a p-value of 0.05001 would be considered nonsignificant, while a value of 0.04999 would be considered significant. The difference between these two values is miniscule. It's better to look at the p-value and consider its meaning.
A common guideline for considering p-values is presented below

Reference [1]
p-value	Significance
0.01 ≤ p < 0.05	significant
0.001 ≤ p < 0.01	highly significant
p < 0.001	very highly significant
p > 0.05	not significant
0.05 ≤ p < 0.10	trend towards significance

POSITIVE PREDICTIVE VALUE (PPV) AND NEGATIVE PREDICTIVE VALUE (NPV)

Definition

Positive predictive value (PPV) - the proportion of patients with a positive test who actually have the disease
Negative predictive value (NPV) - the proportion of patients with a negative test who do not have the disease

Overview

Like sensitivity and specificity, the predictive value is a measure of test accuracy
The predictive value takes into account the prevalence of the disease in the population that is being tested. This gives a more meaningful interpretation of test results than sensitivity and specificity alone.
The main drawback to predictive values is that the prevalence of the disease in the tested population must be known or estimated with some accuracy

Example

Let's assume a colon cancer screening test has a sensitivity of 100% and a specificity of 97%. This test would be positive in everyone with colon cancer, and it would only be positive in 3% of people without colon cancer (3% false-positive). If someone has a positive test, they may assume they probably have colon cancer because the test is only wrong 3% of the time. This assumption would likely be false.
In order to understand the real meaning of a positive test, the prevalence of colon cancer in the population being screened must be considered
Let's assume the prevalence of colon cancer in the population being screened is 0.5%. That means for every 10,000 people screened, 50 of them will have colon cancer. Our test will be positive in all 50 people with colon cancer (100% sensitivity), and it will be positive in 3% of people without colon cancer. This means 3% of 9,950 people (299 people) will have a false-positive test. The percent of people with a positive test who actually have colon cancer (the PPV) is 14% (50/349 X 100).

Formulas for predictive values:

SENSITIVITY AND SPECIFICITY

Sensitivity

Sensitivity is a measure of how well a test identifies people with a disease
A test with a high sensitivity will be positive for most people with the disease
In statistical terms, it is the fraction of subjects with the disease who get a positive test result
If a test has a high sensitivity, a negative test result means the disease is unlikely (few false negatives)

Specificity

Specificity is a measure of how well a test excludes people without a disease
If a test has a high specificity, it will give a negative result for most people without the disease
In statistical terms, it is the fraction of subjects without the disease who get a negative test result
If a test has a high specificity, a positive test means the disease is likely (few false positives)

Sensitivity and specificity figure

Sensitivity / specificity and predictive value

It's important to understand how to apply the sensitivity and specificity of a test to a given population because it is easy to misinterpret the meaning of the measures
Let's assume a colon cancer screening test has a sensitivity of 100% and a specificity of 97%. This test would be positive in everyone with colon cancer, and it would only be positive in 3% of people without colon cancer (3% false-positive). If someone has a positive test, they may assume they probably have colon cancer because the test is only wrong 3% of the time. This assumption would likely be false.
In order to understand the real meaning of a positive test, the prevalence of colon cancer in the population being screened must be considered
Let's assume the prevalence of colon cancer in the population being screened is 0.5%. That means for every 10,000 people screened, 50 of them will have colon cancer. Our test will be positive in all 50 people with colon cancer (100% sensitivity), and it will be positive in 3% of people without colon cancer. This means 3% of 9,950 people (299 people) will have a false-positive test. The percent of people with a positive test who actually have colon cancer is 14% (50/349 X 100). This measure is called the positive predictive value (PPV). The PPV takes into account the prevalence of the condition in the screened population, and it gives a more meaningful interpretation of a positive test.

HAWTHORNE EFFECT

Overview

The Hawthorne effect is a phenomenon that occurs when subjects in a study modify their behavior because they are aware that they are being observed
The Hawthorne effect is particularly important in studies where a placebo control is either not used or is not feasible. Common examples include studies where patients are randomized to an intervention versus "usual care," studies where baseline measurements are used as controls, and studies where treatment groups are unblinded.
The Hawthorne effect may also increase the effect size of shorter trials when compared to longer trials because it tends to wane over time

Example one (usual care)

Researchers have developed an EMR prompt system that provides evidence-based information on heart failure to doctors when they are entering CHF orders. The researchers want to test whether the system improves CHF outcomes so they randomize a group of hospitals to the EMR prompt system and another group of hospitals to "usual care." The Hawthorne effect may occur in the "usual care" group because the doctors know their outcomes are being observed. The "usual care" doctors may change their behavior when they treat CHF patients so that they are more in-line with clinical guidelines. In this case, the Hawthorne effect may decrease the observed effect size of the prompt system.

Example two (unblinded treatment groups)

Studies in renal denervation exemplify the power of the Hawthorne effect. Renal denervation is a procedure where the sympathetic nerves that innervate the kidneys are ablated. The procedure theoretically lowers blood pressure and is used to treat resistant hypertension. A study published in 2010 randomized 106 patients with resistant hypertension to renal denervation + continuation of current meds or continuation of current meds only. At 6 months, the renal denervation group's blood pressure decreased by 33/11 mmHg when compared to control. [PMID 21093036] In 2014, another similar study was published, but in this study, patients in the control group received a sham procedure so that all subjects were unaware if they actually received the procedure. At 6 months, there was no significant difference in SBP reduction (14 vs 11 mmHg). [PMID 24678939] In both studies, blood pressure meds were continued unchanged during the 6 month follow-up period.
So how does one explain the large discrepancy in results? The Hawthorne effect likely played a role. In the first trial, patients who did not receive the procedure were likely disappointed and lost interest in the study. Their behaviors were less inclined to change. In the second trial, patients didn't know if they received the treatment, and they were more likely to remain engaged. Their behaviors likely changed due to a heightened interest in the treatment effect.

INVESTIGATOR BIAS

Overview

Investigator bias occurs when the researchers in a study desire a particular outcome for the study
Investigator bias can occur at every level in the research process - design, data gathering, data analysis, and outcome reporting
Well-designed studies with careful blinding can help limit investigator bias
Investigator bias can also occur when researchers emphasize secondary outcomes that are significant when their primary outcome was not

Example

Investigators are studying a new screening test for breast cancer
Outcome measures include overall mortality and mortality from breast cancer
If the study finds that the screening test decreases breast cancer mortality, but has no effect on overall mortality, investigators may be tempted to report the decrease in breast cancer mortality and disregard the null effect on overall mortality. If they report both, they may emphasize the effect on breast cancer mortality and pay little notice to the effect on overall mortality. This practice is quite common.

PROTOPATHIC BIAS

Overview

Protopathic bias occurs when an intervention is used to treat the first symptoms of a disease that has not yet been diagnosed. The intervention may appear to be a risk factor for the disease when it is actually associated with symptom treatment.

Example

A study looks at the association of Depo-Provera® and uterine fibroids
The study finds that Depo-Provera® is significantly associated with uterine fibroids and declares it a risk factor
Depo-Provera® is often used to treat excessive vaginal bleeding, a presenting symptom of uterine fibroids
Protopathic bias may have influenced the findings in this study

RECALL BIAS

Overview

Recall bias occurs when a person with an outcome is more likely to recall an exposure than someone without the outcome
Recall bias is a major concern in observational studies, particularly case-control studies, because it may bias results toward a false positive effect (Type 1 error)

Example

A case-control study is exploring the relationship between over-the-counter NSAID use in pregnancy and birth defects
Women who have children with birth defects (cases) and women who have children without birth defects (controls) are asked about their over-the-counter NSAID use during pregnancy
In this study, recall bias may occur because women who have children with birth defects may be more likely to thoroughly contemplate their NSAID use during pregnancy than women who have normal children
This difference in attention to recall may bias the study toward a positive effect

REGRESSION DILUTION BIAS (ATTENUATION BIAS)

Regression dilution bias (also referred to as attenuation bias) is a form of bias seen in regression analysis. It occurs when there is an excessive amount of measurement error in the independent variable. The excessive error leads to an underestimation of the slope (coefficient) of the regression line.
Regression dilution can be corrected for in several ways. To decrease the amount of measurement error, the independent variable can be measured several times on each subject and an average can be used. If remeasuring the entire sample is not feasible, then a smaller subset of randomly selected individuals can be used. In other methods, subjects at 20 - 30% of the extremes are remeasured. The average of these measurements can then be used to correct the slope of the regression line.
Correcting for regression dilution is not usually necessary in hypothesis testing for linear relationships (slope ≠ 0)
Corrections should be performed when regression coefficients are used to measure effect sizes [7]

REPORTING BIAS

Reporting bias occurs when events associated with one treatment are more likely to be reported than events associated with another treatment
There are several reasons reporting bias may occur. If a treatment is newer and less familiar, then people who use the treatment or doctors who prescribe it may be more inclined to report an adverse event they believe to be associated with the treatment. If they are familiar with a treatment, then they may be less inclined to report an adverse event associated with the treatment. Because of this bias, the newer treatment may have more events reported than the older treatment when the underlying incidence of the event is really no different between the two treatments. This phenomenon occurs frequently with new drugs in the FDA's Adverse Event Reporting System.
Reporting bias may also occur if there is publicity surrounding a treatment-related event. If providers have recently been made aware of a possible adverse event associated with a treatment, then they may be more inclined to report events associated with the publicized treatment than with events from other treatments.

SAMPLING/ASCERTAINMENT BIAS

Overview

Sampling bias (also referred to as "ascertainment bias") occurs when the procedure for sampling a population inadvertently leads to the over/under representation of a subgroup of that population

Example

Researchers want to study a new drug for chronic back pain
To recruit patients, the researchers decide to enroll new patients presenting to pain management clinics for chronic back pain
By recruiting patients from pain management clinics, the researchers run a high-risk of sampling bias
Patients who are referred to pain management clinics often have complicated histories that might include opiate medication-seeking behavior
If the medication they are studying does not have opioid properties (ex. a new NSAID), the medication may not perform well simply because a large proportion of the study population is seeking opiate medications
If the medication is an opiate, it may perform better than expected simply because the population is seeking this type of medication
In either case, sampling bias has occurred because a subpopulation of patients with chronic back pain (opiate-seekers) is overrepresented in the sample

SELECTION BIAS

Overview

Selection bias occurs when an intervention or treatment is more likely to be given to one person over another because they differ in some way. The underlying difference between the individuals may not be obvious or even measurable.
Selection bias is a big concern in observational studies because patients are not randomized

Example

An observational study compares outcomes between diabetics who received Medication A and those who did not
Medication A requires that patients return to the clinic every 3 months for blood tests
The study shows that diabetics who took Medication A had better outcomes
There is a big concern for selection bias in this study because it may be that doctors were more likely to prescribe Medication A to their patients who were more compliant
The difference in compliance between the two groups can bias the outcomes
If Medication A is expensive, patients in a higher socioeconomic class may have been more likely to receive it. This can also bias outcomes.

SURVEILLANCE BIAS

Overview

Surveillance bias occurs when a proposed risk factor for a condition is also independently associated with an increase in medical diagnostic testing (ex. labs, X-rays, screening, etc.)
The risk factor may appear to be associated with conditions that are discovered from the testing, when in reality, the conditions are only diagnosed more frequently secondary to increased testing (surveillance)

Example

A study published in the Annals of Rheumatic Diseases in 2014 provides a perfect example of a study where there was a high-risk for surveillance bias (PMID 23842460)
In the study, researchers evaluated the association of polymyalgia rheumatica with cancer
The study was a cohort study that compared the incidence of cancer diagnosis between patients with polymyalgia rheumatica and matched controls without polymyalgia rheumatica. Data for the study was gathered from a medical database.
Results of the study found that patients with polymyalgia rheumatica were more likely to be diagnosed with cancer within 6 months of their polymyalgia rheumatica diagnosis when compared to matched controls
The authors concluded that polymyalgia rheumatica may be associated with cancer
This study has a high-risk for surveillance bias
Polymyalgia rheumatica can be a difficult condition to diagnose, and patients will often undergo extensive diagnostic workups. These workups may lead to other diagnoses, including cancer, that may have otherwise gone undetected.
Once diagnosed, polymyalgia rheumatica treatment often involves frequent doctor's visits and an overall increased exposure to medical care. This increased exposure will typically lead to more testing and screening.
Because of these factors, there is a high-risk for surveillance bias in this study. It's unclear if polymyalgia rheumatica increases the risk of cancer, or if the increased medical surveillance associated with its workup and diagnosis leads to more cancer detection.

CONFOUNDING BY INDICATION

Overview

Confounding by indication can occur when a condition is a risk factor for an outcome among both exposed/treated individuals and unexposed/untreated controls. If the condition is associated with the exposure/treatment, then confounding by indication may occur, given that the condition is not an intermediate step in the causal pathway to the outcome. [4]

Example

Researchers want to evaluate if Byetta® is associated with pancreatitis
They conduct a case-control study where they compare the risk of pancreatitis between diabetics on Byetta® and diabetics not on Byetta®
The study finds that diabetics who took Byetta® were at greater risk for pancreatitis
There is a risk for confounding by indication in this study. Obesity is a risk factor for pancreatitis. Byetta® causes modest weight loss. It may be that doctors were more likely to prescribe Byetta® to obese diabetics. Because these patients were already predisposed to pancreatitis, confounding by indication may have occurred.

CONFOUNDING BY SEVERITY

Overview

Confounding by severity is a type of confounding by indication
In confounding by severity, patients who are sicker may be more prone to receive an intervention
This may cause the intervention to appear less effective, or even inferior in some cases [4]

Example

A study looks at the association of minoxidil and stroke
It finds that patients who took minoxidil as compared to other blood pressure medications were more likely to have a stroke
Minoxidil is typically reserved for patients who have resistant, difficult to treat hypertension
Confounding by severity is likely because patients with more severe disease are more likely to receive minoxidil. This would bias the results against minoxidil.

RESIDUAL CONFOUNDING

Overview

Residual confounding occurs when a variable that is related to an outcome is not stratified appropriately
Confounding is present because subjects grouped together in a strata have different risks for the outcome
The risk for residual confounding is particularly high when a variable and the risk for an outcome have a nonlinear relationship (ex. J-shaped or bimodal)

Example

Researchers want to look at the association of BMI with mortality
They form two cohorts of patients: Group 1 - BMI < 27; Group 2 - BMI ≥ 27
After observing the two cohorts for 10 years, they find no difference in the risk of mortality and declare that body weight does not influence mortality
Residual confounding is very likely in this study because the researchers did not stratify patients appropriately
BMI has been shown to have a J-shaped relationship with mortality. Patients with a very low BMI have a higher mortality rate than patients with a normal BMI, and patients with a high BMI have a higher mortality rate than patients with a normal BMI
Because the authors only formed two BMI groups (< 27, ≥ 27), they came to the erroneous conclusion that BMI is not related to mortality. If they had stratified patients into five groups of BMIs (e.g. < 20, 21-25, 26-30, 31-35, ≥ 36), they would have come to a different conclusion.

COVARIATE AND CONFOUNDER

Covariate - a covariate is a patient variable (ex. age, sex, race, etc.) that may or may not be related to the outcome being studied. If the covariate is related to both the exposure/risk factor and the outcome, then the covariate becomes a confounder.
Confounder - A confounder is a covariate that is related to both the outcome and the exposure/risk factor. A confounder may either increase or decrease the likelihood of the outcome.

Illustration of covariate-confounder relationship

IMPORTANT POINTS ABOUT DRUG INTERACTIONS

Drug interactions are challenging

Information on drug interactions can be difficult to assimilate
Certain drug interactions and metabolic pathways are well-documented, while many are not

Factors that can make drug interactions challenging include the following:

New drugs

When a new drug is being developed, the FDA requires that it be tested for drug interactions with a small number of medications that are known to have significant interactions
For obvious reasons, there is no way to test a medication in every possible drug combination that may occur
This means most drugs come to market with incomplete drug interaction profiles
After a medication is prescribed to a large number of people, other drug interactions are inevitably discovered

Research

Much of the research involving drug metabolism and drug interactions occurs in vitro meaning in a lab, or outside of the human body
Animal models and cell cultures are often used to test drugs for metabolic pathways and interactions
Findings from in vitro experiments do not always translate into what actually happens in the human body, or in vivo

Evolving information

Drug metabolism is an evolving field of medicine and pharmacology
Researchers are just beginning to understand all the different systems that are involved in how the body metabolizes and eliminates drugs
Cell transport systems (ex. p-glycoprotein, OAT, etc.) are a relatively new area of pharmacology and information about how these systems affect drug elimination is evolving

Summary

Not all drug interactions are known or can be predicted
Good information on possible drug interactions may not be available
Not all drug interactions are significant
Always consult your physician or pharmacist before changing your medication if you are concerned about a possible drug interaction

FDA ADVERSE EVENT REPORTING SYSTEM (AERS)

Overview

The FDA's AERS is a database run by the FDA where adverse events thought to be related to drugs are recorded and followed
Reporting of adverse events by the general public is voluntary
Anyone who thinks they know of an adverse drug event can report it to the system
When drug companies receive reports of adverse drug events, they are required to report them to the system
Based on information received in the database, the FDA sometimes takes action on a drug. Actions may include changes to the drug labeling, restricting use of the drug, issuing warnings to healthcare professionals and patients, and removing drugs from the market.

Important points

When the FDA issues a warning based on information from its AERS, it often draws headlines and causes concern among patients and healthcare providers alike
While the concern is warranted, it's important to understand that the database in no way, shape, or form constitutes a statistical analysis of the adverse event being reported
The purpose of the AERS is to flag trends in adverse events that need to be explored further with proper studies
It does not prove causality

Reporting bias

Adverse event reporting is susceptible to a tremendous amount of reporting bias
Reporting bias occurs when adverse events associated with a new drug or treatment are more likely to be reported than adverse events associated with older, more familiar drugs and treatments
Because the prescriber is less familiar with the new drug, he may be more inclined to report an adverse event he thinks is associated with the new drug
A prescriber may be less inclined to report an adverse event for a drug that he has used for a long time. He may be biased in thinking that the drug is safe, and therefore he does not associate it with the adverse event, or the adverse event may already be known, and he doesn't see the point in reporting it.

Example

Scenario one

Doctor prescribes brand new drug to patient who subsequently develops pancreatitis
Doctor may be inclined to report pancreatitis as an adverse event for the drug because it is new, and he is unfamiliar with it
If a warning has already been issued for the drug, this may also bias his reporting of the event

Scenario two

Doctor prescribes drug he has used for years to patient who subsequently develops pancreatitis
Doctor may not consider the drug as a possible cause since he has used it for years and is comfortable with it

Result

In the AERS, pancreatitis is being reported for the new drug, while older drugs are not receiving adverse reports
In reality, there may be no difference in the incidence of pancreatitis between the new drug and the older drug

OVERDIAGNOSIS

Definition

Overdiagnosis occurs when a condition is diagnosed that would otherwise not go on to cause symptoms or death
Overdiagnosis has become a larger issue as the utilization of screening tests has increased
Overdiagnosis is a particular problem in cancer screening since almost all cancers are treated. Treatment of all cancers leads to overtreatment in some cases. Overtreatment can adversely affect a person's quality of life.

Example

Patient A is a 60 year-old man with slow-growing prostate cancer and heart failure
Patient B is a 60 year-old man with slow-growing prostate cancer and heart failure
Patient A has a screening PSA test that detects his prostate cancer. Patient A then undergoes radiation and androgen deprivation therapy. Patient A develops impotence, hot flashes, and gynecomastia from the treatment. He dies five years later from his heart failure.
Patient B is never screened for prostate cancer and never knows that he has it. He dies six years later from heart failure
Patient A was overdiagnosed because his prostate cancer would not have led to his eventual death. The treatment of his prostate cancer may have hastened his death, and his quality of life was adversely affected.

Measuring overdiagnosis

Overdiagnosis is best measured in comparative clinical trials where one group is screened and another group is not screened
Consider a randomized controlled trial that compared a screening test for breast cancer in one group (screened) to no screening in another group (control). At the conclusion of the screening phase of the trial, the screening group will have more breast cancer diagnoses than the control group, because screening tests advance the time of cancer diagnosis.
If there is no overdiagnosis, then as time goes by, the control group will catch-up with the screened group, and the number of breast cancers diagnosed in each group will be equal
If overdiagnosis has occurred, then the control group will have a lower breast cancer incidence than the screening group. This means the screening group had cancers detected that would have never led to clinically detectable disease.

Issues with overdiagnosis

Measuring overdiagnosis in a trial can be difficult because it requires many years of follow-up. In order to accurately predict the incidence of cancer in each group, the length of follow-up must allow for slow-growing cancers to become clinically apparent. This can take up to 7 or more years in some cases.
At the time of diagnosis, it is often not possible for a doctor to know if the cancer will cause the patient to die, or if the patient will die from something else
Overdiagnosis assumes that the undiagnosed cancer did not contribute to the patient's death or morbidity [3]

Bias with overdiagnosis

Overdiagnosis can lead to bias in survival statistics
If a screening test detects a condition in a significant number of people who will never die from the condition (overdiagnosis), then it will inflate the survival advantage of the screening test
The link below is to an illustration from Wikipedia that demonstrates this phenomenon

Bias in overdiagnosis illustration

As detailed in the illustration, the screening test detected a large number of patients who did not die from lung cancer (referred to as "pseudodisease"). This inflated the 10-year survival for the screened group.
Screening tests can also inflate survival statistics because they diagnose a condition earlier. For example, assume a cancer has absolutely no effective treatment. Patients who are screened for the cancer may have it diagnosed years before patients who are not screened. The screened group will have a longer "survival time" simply because they were diagnosed earlier. In reality, screening had no real effect on cancer mortality.

PMID

PMID is short for "PubMed Identification #"
PubMed is an online resource developed and maintained by the National Center for Biotechnology Information
PubMed comprises over 20 million citations for biomedical literature from MEDLINE, life science journals, and online books
Citations on this site are listed by their PubMed Identification number. Example - PMID 123400978
Citations can quickly be retrieved by searching PubMed with the ID number

PRIMARY PREVENTION

Overview

Primary prevention involves taking measures (ex. medications, exercise, weight loss) to prevent the occurrence of an event (ex. heart attack) or illness (ex. cancer) before the underlying disease has been established in the patient
Primary prevention differs from secondary prevention where the underlying disease has already been established

Example

Patient has never had a heart attack, but he is concerned about it because his father had one
Patient starts taking aspirin every day to prevent a heart attack (Primary prevention)

SECONDARY PREVENTION

Overview

Secondary prevention involves taking measures (ex. medications) to prevent the recurrence of an event (ex. heart attack) or illness (ex. cancer) after the underlying disease has already been established in the patient
Secondary prevention differs from primary prevention where the underlying disease has not been established in the patient

Example

Patient has a stroke
Patient then starts taking aspirin every day to prevent another stroke (secondary prevention)

BIBLIOGRAPHY

1 - Fundamentals of Biostatistics 7th ed.
2 - PMID 9832001 RR paper
3 - PMID 20413742
4 - PMID 10355372
5 - PMID 26501539 - PSM JAMA
6 - PMID 17909367 - PS inverse weight
7 - PMID 22401135 - Regression dilution bias
8 - PMID 29164242 - Mendelian Randomization, JAMA (2017)
9 - PMID 11602018 - The paired availability design for historical controls, BMC Med Res Methodol (2001)
10 - PMID 30676631 - Using the E-Value to Assess the Potential Effect of Unmeasured Confounding in Observational Studies, JAMA (2019)
11 - PMID 31046064 - Using Instrumental Variables to Address Bias From Unobserved Confounders, JAMA (2019)

MEDICAL STUDIES