Medical studies

CASE-CONTROL STUDY

Overview

A case-control study is an observational study where researchers identify two groups of subjects: one group has an outcome of interest (cases), and the other does not (controls). The prevalence of an exposure is then compared between the groups. The exposure can be just about anything (e.g. medications, environmental toxins, medical conditions), and the outcome is something thought to be related to the exposure (e.g. death, heart disease, cancer). If the exposure is more common in one of the groups, it may be associated with the outcome in some way.

Example

Researchers want to evaluate if cell phone use can cause brain cancer, so they identify two groups of subjects:

Case group: 1000 people with brain cancer
Control group: 1000 people without brain cancer

They give each group a questionnaire that asks about their cell phone use history. Reported cell phone use is much higher in the case group, so they conclude that cell phone use may be associated with brain cancer.

Advantages

Case-control studies are far cheaper and easier to perform than randomized controlled trials. Nowadays, entities that provide healthcare (e.g. governments, HMOs, insurance companies) maintain large databases of medical information on their clients, providing researchers with an enormous amount of data that can be used to perform studies with minimal effort and resources.
Unlike a prospective cohort study, there is no observation period, so results are available as soon as the data is analyzed
Outcomes with a low incidence (< 3%) can be evaluated. This may be unfeasible in a randomized controlled trial because it requires a very large number of participants.
Able to evaluate exposures that would be unethical in a randomized controlled trial. For example, no trial is going to randomize patients to smoking or other known carcinogens.
Able to evaluate exposures that would be impractical in a randomized controlled trial. For example, it would be impossible to randomize people to drinking alcohol or abstinence, as these behaviors are unlikely to change overnight.
Able to evaluate associations between conditions and outcomes. You can’t randomize people to diseases like rheumatoid arthritis or ulcerative colitis, so the only way to look for a link between these conditions and an outcome like cancer or heart disease is with observational data.

Disadvantages

Does not prove causality
Data for patient variables may be lacking or missing and must be estimated
The incidence of the outcome is not measured, so group comparisons are expressed as odds ratios, which are less intuitive for quantifying risk
Patients are not randomized, so there is no way to control for unmeasured covariates, confounders, and hidden bias (see randomization for definitions)

COHORT STUDY

Overview

A cohort study is an observational study that evaluates the association between an exposure and an outcome. The exposure can be just about anything (e.g. medications, tobacco smoke, sunlight, environmental toxins), and the outcome is something thought to be related to the exposure (e.g. death, heart disease, cancer). Groups of people called cohorts are identified based on whether or not they have been exposed to the factor of interest. The cohorts are then followed for a period of time, and the incidence of the outcome is measured. If the outcome is more common in one of the groups, the exposure may be associated with the outcome in some way.
Cohort studies may be prospective or retrospective. In prospective studies, cohorts are identified before the observation period begins. In retrospective studies, cohorts and outcomes are identified after the observation period has occurred; this is typically done with a medical registry or database.

Example

Researchers want to see if smoking is associated with bladder cancer, so they identify two cohorts of people:

Exposed cohort: 1000 smokers
Control cohort: 1000 nonsmokers

They follow the two cohorts for 10 years and compare the incidence of bladder cancer over that time. Bladder cancer is more common in the smoker cohort, so they determine that smoking may cause bladder cancer.

Advantages

Cohort studies are far cheaper and easier to perform than randomized controlled trials. Nowadays, entities that provide healthcare (e.g. governments, HMOs, insurance companies) maintain large databases of medical information on their clients, providing researchers with an enormous amount of data that can be used to perform studies with minimal effort and resources.
Outcomes with a low incidence (< 3%) can be evaluated. This may be unfeasible in a randomized controlled trial because it requires a very large number of participants.
Able to evaluate exposures that would be unethical in a randomized controlled trial. For example, no trial is going to randomize patients to smoking or other known carcinogens.
Able to evaluate exposures that would be impractical in a randomized controlled trial. For example, it would be impossible to randomize people to drinking alcohol or abstinence, as these behaviors are unlikely to change overnight.
Able to evaluate associations between conditions and outcomes. You can’t randomize people to diseases like rheumatoid arthritis or ulcerative colitis, so the only way to look for a link between these conditions and an outcome like cancer or heart disease is with observational data.

Disadvantages

Cannot prove causality
Data for patient variables may be lacking or missing and must be estimated
Patients are not randomized, so there is no way to control for unmeasured covariates, confounders, and hidden bias (see randomization for definitions)

META-ANALYSIS

Overview

A meta-analysis is an observational study where the results from a number of related studies are pooled and analyzed in order to draw conclusions from a broader set of data. Study types may include randomized controlled trials, cohort studies, case-control studies, or a combination of the three. A meta-analysis uses special statistical techniques to manage variance within and across studies and to combine outcome measures that differ in type (e.g. means, odds ratios, correlations).

Example

A meta-analysis from the Cochrane review [PMID 24953955] evaluated the effects of vitamin D supplementation on cancer risk. Study authors searched the medical literature for randomized controlled trials comparing vitamin D to placebo or no treatment that included cancer as an outcome. They found 18 studies that met their criteria and were able to pool results for 50,623 patients. Using special statistical techniques, they analyzed the data and found no significant overall effect of vitamin D on cancer occurrence.

Advantages

When studies on the same or similar topic come to different conclusions, a meta-analysis offers a way to combine results so that an overall effect can be estimated
If a topic has been evaluated in a number of small trials, a meta-analysis can pool data from those trials, increasing study power and the chance of finding a significant effect
The effect size in a meta-analysis, which is often less than what is seen in individual trials, may provide a truer estimate of the overall effect in a large population

Disadvantages

Combining results from trials that used different criteria, protocols, and outcome measures lowers precision and increases the chance of erroneous conclusions
Meta-analyses are subject to publication bias, which occurs when only studies that found a significant or large effect are published. To account for this, researchers often try to obtain unpublished studies.
Meta-analyses are only as good as the studies they evaluate. If there are not enough good studies to perform a valid meta-analysis, the authors should be willing to abandon the study.
If a large randomized controlled trial has been performed on the topic, its results are more consequential

NETWORK META-ANALYSIS

Overview

A network meta-analysis is a type of meta-analysis where three or more treatments are compared. What makes a network meta-analysis unique is that interventions that have not been compared directly in a trial can be compared indirectly across a network of connected studies. When stated plainly, this may be difficult to understand. The example below helps to explain the concept.

Example

There are few head-to-head antidepressant trials, so researchers decide to compare medications using a network meta-analysis. They identify five studies comparing Paxil to placebo, three comparing Lexapro to placebo, and one comparing Lexapro to Zoloft. In a network meta-analysis, Paxil and Lexapro are indirectly compared by contrasting each drug's effect size against their combined placebo groups. Furthermore, a comparison between Zoloft and Paxil can be made because they are connected through the network. The illustration below shows how the drugs are connected, with the size of the nodes being proportional to the number of participants in each group and the width of the lines relative to the number of trials between the drugs. The dotted lines represent indirect comparisons.

Advantages

The main advantage of a network meta-analysis is that it provides a way to compare the effects of therapies that have not been evaluated in head-to-head trials

Disadvantages

A network meta-analysis carries all the same disadvantages of a typical meta-analysis along with an additional layer of error and uncertainty that comes with making indirect comparisons [13]

NESTED CASE-CONTROL STUDY

Overview

A nested case-control study is a case-control study where the subjects for the study are sampled from a cohort of patients that has been observed for a period of time. The study is said to be "nested" within the cohort.
Researchers form two groups from the cohort of patients. One group has the outcome of interest (cases), and one group has not developed the outcome (controls). Exposure to a risk factor can then be compared between the two groups. If the cases have a significantly greater exposure to the risk factor, then the risk factor may be related to the outcome.
When selecting the control group in a nested case-control study, the researchers typically select 1 - 4 controls per case as opposed to comparing cases to the entire group of controls. Controls are selected that match the cases on a number of measured variables (ex. age, medical conditions, sex, etc). This helps to control for confounding.

Example

Researchers have observed a cohort of 100,000 women for 10 years
At the start of the study, all women gave blood samples
Over the course of the 10 years, 2000 women developed breast cancer
The researchers are interested in finding out if there is a link between mercury exposure and breast cancer
The researchers decide to test the blood samples for mercury levels and compare rates of breast cancer between women with high levels and those with normal levels
Testing 100,000 blood samples for mercury levels would cost about $2 million dollars
To save money, the researchers decide to do a nested case-control study where they select 2 controls for every case. Controls are selected that match the cases for a number of variables. This way they only have to test 6000 blood samples (2000 cases and 4000 controls).

Advantages

In most cases, the nested case-control study is much cheaper to perform than a full case-cohort study
By selecting controls that match the cases on a number of covariates, there is some control for confounding and statistical efficiency should be improved

Disadvantages

Because the entire cohort is not used (for the controls), some precision is lost in the statistical measures, and the confidence intervals for the study outcomes will be wider
Like regular case-control studies, nested case-control studies are observational studies and have all the same limitations

OBSERVATIONAL STUDY

Overview

An observational study is a study where patients and outcomes are observed and no intervention is applied by the investigators. In observational studies, groups of subjects are formed according to specific exposures (cohort study) or according to specific outcomes (case-control study). This is in contrast to a randomized controlled trial where subjects are randomly assigned to a specific group and then followed.
Observational studies may be prospective (groups are identified and then observed over time) or retrospective (groups are identified and outcomes are compared)

Advantages

Much easier and cheaper to perform than a randomized controlled trial
With modern medical databases, a large number of subjects can be evaluated with very little effort
Outcomes with a low incidence can be evaluated. This is typically not feasible with a randomized controlled trial.
Exposures that are not ethical in a randomized controlled trial can be evaluated. For example, no randomized controlled trial is going to randomize patients to smoking or known carcinogens.

Disadvantages

The main disadvantage of observational studies is that there is no way to control for hidden bias and unmeasured covariates. In randomized controlled trials, the process of randomization helps to alleviate the effect of these issues.
Observational studies do not prove causality

RANDOMIZED CONTROLLED TRIAL (RCT)

Overview

A RCT is a prospective study where subjects are randomly assigned to either a treatment group or a control group
In most studies, the two groups are matched as closely as possible by variables such as age, medical problems, lab values, blood pressures, etc. If the study is very large (ex. > 500 subjects), then these factors will often balance automatically through randomization.
Randomization is very important because it is the only way to control for unmeasured covariates. Measured variables (ex. weight, age, medical history, etc.) can be adjusted for, but there may be variables that are not measured because the researchers are not aware that they can affect the study outcome. If the study is randomized and large enough, then unmeasured covariates should be distributed equally between the groups, and they will not bias the outcome.
Randomization also prevents hidden bias. In observational studies, there is no way to prevent or control for hidden bias.
In a RCT, the treatment group is given an intervention (drug or procedure)
The control group is not given the intervention, or it is given a placebo or sham procedure
The effect of the intervention is then measured over a predetermined time period by a predetermined set of outcomes
RCTs are considered the gold standard for clinical trials, and if done correctly, their findings are much more consequential than those of other study designs (ex. cohort studies, case-control studies)
The best RCTs are placebo-controlled and double-blinded when possible

Advantages

Able to control for confounding from unmeasured covariates
Hidden bias can be prevented
Criteria and outcomes can be clearly defined and accurately measured
Large RCTs will give the best measure of an intervention's effect in the intended population

Disadvantages

Expensive and time-consuming
Can take many years to complete
If an outcome has a low incidence, RCTs are typically not feasible because the study size would need to be very large, and it would make the trial very expensive
RCT are not ethical in some situations. For example, randomizing subjects to smoking or other interventions that are known to be harmful.

REGISTRY-BASED RANDOMIZED CONTROLLED TRIAL (RRCT)

Overview

RRCTs are randomized controlled trials that use large medical registries/databases to help streamline and simplify data collection
Some countries and health organizations maintain large databases of information on patients in their healthcare system. This information can be used in performing a RRCT.
In RRCTs, registry information is used to identify eligible patients, log baseline information, determine randomization, and track outcomes

Advantages

Less expensive and time-consuming than a traditional RCT
The centralized enrollment process makes it easier to enroll and recruit patients
The registry information helps to identify more eligible patients
Follow-up rates are higher because outcomes are tracked in the registry and investigator-specific follow-up is not required
Results may be more pragmatic since the studies occur during the course of typical patient care and do not include specific methodology

Disadvantages

Outcome events are based on registry entries, and therefore, they are not adjudicated and must typically remain definitive or broad (e.g. overall mortality)
Blinding and placebo control are difficult to implement
Not amenable to all types of outcome data

DATABASE AND REGISTRY STUDIES

Overview

Certain countries, HMOs, and insurance companies maintain large databases of medical information on their citizens/patients/clients
This information is often used to conduct observational studies, and more recently, it has been utilized in conducting randomized controlled trials (see registry-based randomized controlled trials)
Observational studies using medical databases are very popular because they allow investigators to examine a large amount of information with very little effort, expense, and time. These studies have many weaknesses though, and their results do not prove causality.

Advantages

Cheap and easy to perform
Able to evaluate a large amount of information with very little effort

Disadvantages

Diagnostic codes are often used as criteria. Use of these codes is not well-standardized.
Pertinent patient information is often missing or not recorded and must be estimated

PAIRED AVAILABILITY DESIGN STUDY

Overview

Paired availability design studies are a type of observational study that uses the time period before an intervention is available (or widely available) as a historical control
In a paired availability design study, outcomes are measured in all patients who are candidates for an intervention both before and after the intervention is widely available. The two sets of outcome data are then compared. All candidates are included in the analysis whether they receive the intervention or not (this includes both before widespread availability and after). By including all patients in the data set, selection bias is limited.
Data for paired availability design studies may be collected prospectively, retrospectively, or through a combination of the two

Advantages

Cheaper and easier than randomized controlled trials
Data collection may be facilitated by a registry
Less prone to selection bias than a cohort study

Disadvantages

In order for the results to be valid, a number of assumptions must hold true including:

Stable population meaning the before and after patient population does not change in some significant way. Because of this, these studies are typically performed at hospitals that serve a specific patient population or geographic region.
Stable treatment meaning other treatments or care does not change significantly between the before and after periods
Stable evaluation meaning diagnostic and monitoring modalities are not different between the two periods. Improvements in diagnosis and monitoring can improve outcomes.
Stable preference meaning new data/information does not influence whether a patient chooses the intervention
The availability of the intervention should not affect its efficacy. This applies when some people in the before group receive the intervention. Increased availability may lead to earlier treatment in the after group which can affect outcomes. Interventions that have a learning curve may not perform as well in the before group when compared to the after group. [9]

AS-TREATED ANALYSIS

Overview

In an as-treated analysis, subject outcomes are counted toward the treatment they actually receive. If a subject is randomized to one treatment group but crosses over during the study to a competing treatment, then their outcome counts toward the competing treatment. This differs from intention-to-treat analysis, where outcomes count toward the participant's original assigned group regardless of whether they crossover.
As-treated analyses are often performed in studies with high crossover rates. On the surface, as-treated analyses seem to be a sensible way to handle crossovers, but in reality, they can introduce bias (see example below).

Example

A trial enrolls 200 people with sciatic nerve pain
100 people are randomized to physical therapy, and 100 people are randomized to back surgery
The subjects are followed for one year, and at the end of the year, differences in back pain and disability are compared
Over the course of the study, 30 people assigned to physical therapy end up receiving back surgery, and 10 people assigned to surgery never have surgery and instead receive physical therapy (crossovers)
The researchers decide to do an as-treated analysis so that crossovers count toward the therapy they actually received

Bias in as-treated analyses

In this example, results from 30 people in the physical therapy group are counted toward surgery, and results from 10 people in the surgery group are counted toward physical therapy. This appears logical since the outcomes for crossovers now count toward the treatment they received. In reality, the analysis is likely biased. People often enter trials because they hope to receive one of the treatments being offered. Researchers know this, so they allow crossovers to increase the chance someone will participate. In the example above, patients who believe invasive treatments are better may be disappointed if they are assigned to physical therapy. They are biased toward surgery, and if they cross over, the as-teated analysis will also be biased.
Another issue with as-treated analyses is that randomization is compromised, and this may introduce unmeasured confounding

COMPETING RISKS

Survival analysis

The Kaplan–Meier estimator is a popular way to evaluate outcomes in prospective studies
The method was originally developed to evaluate survival over a period of time with overall mortality being the primary outcome of interest
Use of the method has expanded, and it is often used to evaluate outcomes other than overall mortality
It's important to understand the concept of censored patients in survival analysis. All patients entering the study are assumed to experience the primary outcome at some point. If the patients are censored (lost from the study for some reason without experiencing the outcome), they are still treated as having the primary outcome at some point after they are censored. Patients who make it to the end of the study without being censored or having the primary outcome are assumed to have the primary outcome at some point after the study ends.
This concept holds true if the primary outcome is overall mortality, because all patients will die at some point. If the primary outcome is something other than overall mortality, then the assumption that all patients will eventually experience the outcome or that they are always at risk for the outcome is not necessarily true.

Example

A study is designed to look at the risk of hip replacement in elderly patients over time
Patients are followed for a period of time and the Kaplan-Meier method is used to estimate risk with hip replacement being the primary outcome
Because the patient population is elderly, a significant number of subjects die during the study without ever having a hip replacement
The dead subjects are no longer at risk for a hip replacement, yet in the Kaplan-Meier analysis, they are still considered to be at risk for a hip replacement
The Kaplan-Meier analysis will overestimate the risk of a hip replacement in this population

Competing risk

In the example above, death is a competing risk for hip replacement
In order to correct for the fact that patients who die are no longer at risk for a hip replacement, a different kind of analysis needs to be performed
In most cases, that analysis is the cumulative incidence competing risk (CICR) estimate
CIRC analysis accounts for events that affect the risk of the primary event, and it lowers the probability of the primary event when a competing event occurs. In CIRC analysis, patients experiencing the competing event are no longer considered at risk for the primary event.

DOUBLE-BLIND TRIALS

In double-blind trials, neither the subject nor the researcher evaluating the subject knows which intervention the subject is receiving
In single-blind trials, only the subject is blinded as to what intervention they are receiving
Double-blinding helps prevent bias on the part of the subjects and the researchers
When possible, randomized controlled trials should be double-blinded
Double-blinding is not always possible. For example, when comparing a surgical intervention to a medical one.

INSTRUMENTAL VARIABLE ANALYSIS

Overview

A major limitation of observational studies is the inability to detect and control for unmeasured confounders. Randomization, which is not possible in observational studies, is the only method that can completely eliminate the effects of unmeasured confounders.
A technique called instrumental variable analysis has been developed to help reduce the effects of unmeasured confounders in observational studies. An instrumental variable is a patient variable that is strongly associated with the treatment a patient receives while at the same time having no association with the outcome being measured. For example, a new diabetes drug may only be covered by a few insurance companies. Patients with insurance that covers the drug are far more likely to receive the drug than patients without coverage. In this case, insurance drug coverage is an instrumental variable because it is strongly associated with treatment received but should have no association with diabetic outcomes like A1C values and CVD unless the new drug is truly superior to established therapies.
Once an instrumental variable is identified, researchers can "randomize" patients by comparing outcomes between patients grouped by the instrumental variable. [11]

Example

Researchers want to see if gallbladder removal for cholecystitis within 12 hours of presentation is associated with better outcomes than removal beyond twelve hours
They collect data on patients who have had their gallbladder removed and divide them into 2 groups based on whether they had it removed within 12 hours or after 12 hours
The two groups differ on a number of covariates so the researchers decide to look for an instrumental variable. They find that patients admitted on the weekend were far more likely to have their gallbladder removed after twelve hours when compared to patients admitted during the week. Since day of hospital admission should not have a direct effect on outcomes, they decide to use it as their instrumental variable. An analysis of the data is then performed with patients grouped by whether they were admitted on the weekend or on a weekday.

INTENTION-TO-TREAT ANALYSIS

Overview

In intention-to-treat analysis, participant outcomes are counted towards their original assigned group regardless of whether they are adherent to their randomized treatment or not. This means outcomes for patients who stop their assigned treatment, crossover to other treatment groups, or violate other study protocols (e.g. receiving nontrial treatments) are still counted toward the patient's original randomized group.
Intention-to-treat analysis is the gold standard of study analysis because it is the only method that can control for unmeasured confounders and bias. Its main drawback occurs when a study has a high number of crossovers; in this case, results can become biased towards the null (no effect). Other analytical methods are often used to adjust for high crossover rates (e.g. per-protocol analysis, on-treatment analysis, modified intention-to-treat analysis, as-treated analysis), but they too have weaknesses.

Example

A trial enrolls 200 people with sciatic nerve pain
100 people are randomized to physical therapy, and 100 people are randomized to back surgery
The subjects are followed for one year, and at the end of the year, differences in back pain and disability are compared
During the course of the study, 30 people assigned to physical therapy receive back surgery, and 10 people assigned to surgery never have surgery and receive physical therapy instead (crossovers)

Intention-to-treat

In an intention-to-treat analysis, outcomes for the 30 people who crossed over from physical therapy to surgery still count toward the physical therapy group, and outcomes for the 10 people who never received surgery still count toward the surgery group

INVERSE PROBABILITY OF CENSORING WEIGHTING

Inverse probability of censoring weighting (IPCW) is a statistical technique that is used to correct for nonadherence in randomized controlled trials. The majority of randomized controlled trials use an intention-to-treat analysis which means that participant outcomes are counted towards their original assigned group regardless of whether they are adherent to their randomized treatment or not. Nonadherence is introduced when participants stop their assigned treatment, crossover to other treatment groups, or violate the study protocol in some other way (e.g. receiving nontrial treatments). If nonadherence is significant (e.g. > 10% crossovers), the study results can become invalid and/or biased. Per-protocol analysis, as-treated analysis, and on-treatment analysis are three methods that are often used to adjust for nonadherence, but they all have their own disadvantages and shortcomings.

IPCW tries to adjust for nonadherence in the following manner:

Data from participants who are nonadherent is censored at the time they become nonadherent and on
Participants in a treatment arm are grouped based on their risks (e.g. age, sex, lifestyle habits, comorbidities) for developing the outcome of interest
Within each risk group, adherent and nonadherent participants are identified
Outcomes for subjects in each risk group that were adherent are given more weight (upweighted) than nonadherent subjects through inverse probability. By doing this, adherent participants that are similar to nonadherent participants receive more weighting to account for the data lost when nonadherent participants are censored. [12]

In order for the IPCW method to be valid, investigators must identify participant characteristics that are associated with the outcome of interest both at the beginning and during the trial. The method also assumes there are no unmeasured confounders which can never really be known.
Like other techniques used to account for nonadherence, IPCW shares some of the same disadvantages in that some degree of randomization is lost, results may be biased toward responders, and unmeasured differences between adherers and nonadherers may confound results

MENDELIAN RANDOMIZATION

Overview

Mendelian randomization is an observational study technique that uses variations in genetic makeup to determine if a risk factor is associated with an outcome
For example, in population studies, HDL cholesterol levels are inversely associated with heart disease risk; as HDL levels increase, risk of heart disease goes down
A number of genetic variants are associated with HDL levels. Some variants raise levels and others lower them. Through Mendelian inheritance, these variants are distributed randomly across the population. If a researcher wants to determine if HDL levels are directly associated with heart disease risk, they can form patient cohorts based on genetic variants that affect HDL levels. Cohorts with HDL-promoting variants will have higher HDL levels simply because of their genetic makeup. Other variables that affect HDL levels (e.g. exercise, diet) will be distributed equally among the individuals making the cohorts "randomized" for potential confounders. If a causal relationship between HDL levels and heart disease exists, then HDL-promoting variants will be associated with lower heart disease risk. A number of studies have actually looked at this, and they have found that HDL-promoting variants are not associated with lower heart disease risk calling into question the association between HDL and heart disease.
In order for Mendelian randomization to work, the genetic variants being studied must only be associated with the risk factor in question and not other variables that may affect the outcome. For example, if HDL-promoting variants are also associated with lower LDL levels or lower blood pressure, then confounding exists and incorrect conclusions may be drawn. When a variant affects the outcome through pathways other than the risk factor being studied, this is referred to as pleiotropy.

Advantages

Although Mendelian randomization studies are observational cohort studies, cohort formation based on genetic variants offers some degree of randomization
Mendelian randomization studies may be used to do an exploratory analysis of a possible intervention that affects a risk factor. For example, drugs have been developed that raise HDL levels. Before doing expensive randomized controlled trials to see if the drugs lower heart disease risk, researchers may do a Mendelian randomization study to see if HDL levels are directly related to heart disease risk.

Disadvantages

If pleiotropy exists, results are not valid. Pleiotropy may not be easy to detect.
Statistical power can be difficult to achieve and often requires the use of multiple variants which increases the risk of pleiotropy [8]

MODIFIED INTENTION-TO-TREAT ANALYSIS

In a modified intention-to-treat analysis, investigators in a trial use an intention-to-treat analysis, but they modify who is included in the analysis in some way
A modified intention-to-treat analysis should raise red flags because it may mean the researchers did not find a significant result with their intention-to-treat analysis, and they are now digging for significant results by changing the study's criteria. This is bad form because it can introduce bias.
In some cases, a modified intention-to-treat analysis has little impact. For example, researchers often do a modified intention-to-treat analysis that only includes randomized patients who received at least one dose of a study drug.

ON-TREATMENT ANALYSIS

Overview

In an on-treatment analysis, only data from the time periods where a patient was compliant with their assigned treatment are counted. Data from periods where a participant was not taking their treatment are excluded.
While an on-treatment analysis seems like a logical method for handling protocol violators, it introduces bias in many cases

Example

A study is comparing a new migraine prevention drug to an older one
100 people are randomly assigned to the new drug, and 100 people are randomly assigned to the old drug
The patients are told to take the drugs for 6 months, and the number of migraines in that time period are recorded
During the course of the trial, 30 people assigned to the new drug and 10 people assigned to the old drug stop taking their assigned medication
The researchers decide to do an on-treatment analysis so that time periods where patients were not compliant with their medication are excluded

Bias in on-treatment analysis

In this example, the on-treatment analysis is biased. People who felt that their treatment was working (responders) are more likely to keep taking it. People who did not perceive a benefit from the medication are more likely to stop it. An on-treatment analysis biases the study toward responders because they account for more data in the outcome measures.

PER-PROTOCOL ANALYSIS

Overview

In per-protocol analysis, only subjects who complete the protocol they are assigned to are included in the outcome data. Patients who vary from the protocol (e.g. stop treatment, crossover, drop out) are excluded, and their outcomes do not count. Per-protocol analyses differ from on-treatment analyses and as-treated analyses in that patients who violate protocol are not counted at all, whereas in the other two methods, data from protocol violators is still used, but it may be truncated or applied to different treatment groups.
Per-protocol analysis seems like a logical method for handling protocol violators, but they can introduce bias (see Example #1 and Example #2 below). Per-protocol analyses are more meaningful in conditions that are difficult to treat and/or when there are few treatment options (see Example #3 below).
In noninferiority trials where established treatments are compared to new treatments, per-protocol analyses should always be reported with intention-to-treat analyses because they test the sensitivity of the results to protocol violations

Example #1

A study is comparing a new migraine prevention drug to an older one
100 people are randomly assigned to the new drug, and 100 people are randomly assigned to the old drug
The patients are told to take the drugs for 6 months, and the number of migraines in that time period are recorded
During the course of the trial, 30 people assigned to the new drug and 10 people assigned to the old drug stop taking their assigned medication
The researchers decide to do a per-protocol analysis so that people who stopped their medication are excluded

Bias in per-protocol analysis

In this example, the per-protocol analysis is biased. People stop study medications for a variety of reasons (e.g. side effects, lack of perceived efficacy), and excluding data from nonadherers biases the study towards those who responded to the medication; this does not reflect the true efficacy of the drug in the intended population.

Example #2

A study is comparing aspirin to placebo for the prevention of deep vein thrombosis
1500 people are given aspirin and 1500 people are given placebo. The 2 groups are followed for 2 years and the rate of deep vein thrombosis is compared.
300 people in the aspirin group stop taking aspirin during the trial
The researchers do a per-protocol analysis that excludes people who stopped aspirin. The analysis shows that compliant aspirin patients had lower rates of deep vein thrombosis than the placebo group.

Bias in per-protocol analysis

On the surface, this analysis seems legitimate. The outcome is objective, and for the most part, people tolerate aspirin well, so comparing compliant aspirin patients to controls makes sense. The problem is that past studies have shown that outcomes between compliant and noncompliant patients differ even when they are receiving a placebo. This phenomenon likely occurs because of differences between the two in unmeasured confounders. Excluding noncompliant patients in the aspirin group biases the study toward aspirin. The researchers might account for this by excluding noncompliant patients in both groups, but this will cause the study to lose power and some degree of randomization.

Example #3

A study is comparing a new drug + standard therapy to standard therapy alone in a difficult to treat cancer
The new drug has many side effects that make it intolerable for a number of patients, and they are forced to stop taking it
The researchers decide to to do a per-protocol analysis to see if the new drug improved survival in patients who actually completed therapy
In this case, the per-protocol analysis is meaningful because it will show whether people who can tolerate the therapy actually benefit from it

PLACEBO-CONTROLLED TRIALS

In a placebo-controlled trial, subjects who are not randomized to an active treatment (controls) are given a fake or sham treatment instead (Ex. an inert pill or a procedure that is faked)
Placebo-control prevents subjects in a trial from knowing whether they are receiving the active treatment. This prevents bias that can arise from a subject's expectations that an intervention works.
Placebo-control is very important in trials where outcomes are subjective (ex. pain, disability, frequency of events that are not biologically measurable)
It should be applied when possible in trials with objective outcomes

PROPENSITY SCORE MATCHING (PSM)

Overview

Propensity score matching (PSM) is a statistical technique used in observational studies. It helps to control for selection bias.
In PSM, subjects who received a treatment are matched with subjects who did not receive the treatment based on the probability of receiving the treatment. The probability of receiving the treatment is reflected in the propensity score. The propensity score is calculated by combining the probability of receiving treatment for a number of measured covariates.
When subjects with the same propensity score are compared, selection bias is limited

Procedure

Determine the probability of receiving the treatment for each measured covariate in the study (ex. age, sex, medical conditions, lab values, etc)
For each subject, combine the probabilities of receiving the treatment based on their individual covariates. Derive the propensity score from this combination (usually with logistic regression)
Create groups of subjects with the same propensity score. Subjects with the same propensity score will typically match closely on a number of covariates.
Compare treated subjects to untreated subjects within these groups. There are a number of ways of doing this. See Propensity score techniques below.
Perform multivariate analysis on the propensity-matched sample

Propensity score techniques

Direct comparison

In direct comparisons, treated and untreated individuals with the same score are matched and compared. Matching can be one-to-one, one-to-as many that match, and so on.
Groups can also be stratified by propensity score and then strata can be compared directly
A major disadvantage of direct matching is that depending on the method used, all of the available data may not be utilized. In addition, if strata are compared, residual confounding may occur.

Propensity score as a covariate

The propensity score can be used as a covariate in a multivariate model
In this case, the outcome is the dependent variable, and the independent variables are the exposure/treatment and the propensity score
If the probability of receiving the treatment (propensity score) is strongly associated with the outcome, then covariate adjustment for the propensity score will decrease the strength of association between the treatment and the outcome

Propensity score weighting

Propensity score weighting is a method where each individual in the sample is weighted by the inverse of their propensity score
Using this method, subjects who are more likely to receive the treatment carry less weight, and subjects who are less likely to receive the treatment receive the most weight
With inverse weighting, covariates that are associated with receiving the treatment/exposure (higher propensity score) are neutralized while covariates that are weakly associated with receiving the treatment (low propensity score) gain value. This helps to neutralize selection bias, creating a sample where covariate distribution is independent of treatment selection. [5,6]

Advantages

PSM is helpful when a large number of covariates are measured because it creates a comparison that minimizes selection bias regardless of whether patients match exactly on all of the covariates
PSM simulates randomization of treatment for measured covariates (not for unmeasured covariates)

Disadvantages

PSM is an observational technique, and it cannot control for unmeasured covariates and hidden bias

RANDOMIZATION

Definitions:

Measured covariates - measured covariates are patient variables (ex. age, sex, medical problems) that are recorded before a trial begins
Unmeasured covariates - unmeasured covariates are patient variables that are not measured but may be related to (and influence) the outcome being evaluated. There may be a number of reasons these variables are not measured; researchers may not be aware that the variable affects the outcome, the variable may be difficult to quantify (e.g. patient attitudes towards certain treatments), or the variable may be expensive to obtain (e.g. genetic profiles on every patient enrolled).
Confounder - a confounder is a covariate that is associated with both the exposure and the outcome. (see covariates and confounders below)
Hidden bias - hidden bias is any form of bias that may influence the outcome of a trial (e.g. researchers assigning healthier patients to the treatment group, a form of selection bias)

Randomization

Randomization is a process in controlled trials where patients are randomly assigned to different treatment groups. Randomization is critical because it is the only way to control for unmeasured covariates and confounders. If a trial is large enough (> 500 subjects), randomization will create groups that have an equal proportion of measured variables (e.g. age, sex, medical problems). It will also equally distribute unmeasured covariates and confounders between the groups so that these factors do not affect the outcome. To prevent hidden bias, the person enrolling the patient should have no idea which group the patient will be assigned to.
Randomization is the main reason randomized controlled trials are the gold standard for clinical studies. Observational studies cannot randomize participants; therefore, they are always subject to unmeasured covariates, confounders, and hidden bias.

SINGLE-BLINDED TRIALS

In single-blinded trials, only the patient is blinded as to what treatment they are receiving. The investigators or treating doctors are aware of the treatment given.
Contrast with double-blinded trials where the investigator and subject are blinded to the treatment allocation
Double-blinding is preferred when possible because it prevents investigator bias

ABSOLUTE VS RELATIVE RISK

Overview

When discussing absolute risk and relative risk, it helps to consider the two measures together
Absolute risk is the overall incidence of an outcome in an entire group or population
Relative risk is the ratio of the incidence of an outcome between two groups
Relative risk only considers the proportion of the participants who have the outcome in question where absolute risk considers the entire population being studied. This can lead to some large difference between the two measures.

Conclusions

As one can see from the example above, the relative risk reduction and the absolute risk reduction can be quite different (46% vs 2.3%)
It's important to understand the difference, because medical literature will often emphasize a large relative risk reduction when the absolute risk reduction may be quite small and insignificant. From the example above, the drug company for Livelong will likely say that they "reduce the risk of heart attack by 46%." While this may be true, someone taking Livelong will only reduce their overall risk of heart attack by 2.3%.

Relative risk and hazard ratios

Hazard ratios are often interpreted as relative risks, but they are not completely the same. For example, if subjects given a treatment have a hazard ratio of 0.75 for an outcome, it is often stated that the treatment lowers the risk of the outcome by 25%. In most cases, hazard ratios and relative risk are close, but not always identical. The difference lies in the fact that relative risk is based on the cumulative number of events over the entire study period regardless of when they occurred while hazard ratios represent the average risk over the course of the study.

E-VALUE

Overview

The E-value is a statistical measure that can be used in observational studies to help assess the effect of unmeasured confounders
The E-value is a risk ratio that quantifies the minimal amount of correlation (while controlling for measured covariates) that an unmeasured confounder must have with the exposure and the outcome in order to negate the observed association between the exposure and outcome
If the E-value is low (risk ratio close to 1), then it is more likely that an unmeasured confounder could affect the observed results. If the E-value is higher, then it is less likely that an unmeasured confounder could affect the results. [10]

Example

An observational study looks at the association between exercise and cancer
The study finds that exercise significantly lowers the risk of cancer (HR 0.70 95%CI [0.50 - 0.88])
The authors calculate an E-value and it comes back as 3.2. This means an unmeasured confounder would have to have at the very least, a hazard ratio of 3.2 between exercise and cancer in order for it to affect the observed results.
In the same study, the authors calculated a hazard ratio of 1.90 for the association between smoking and cancer. Since smoking is a strong risk factor for cancer, it's unlikely that an unmeasured confounder would have a stronger association; therefore, the authors conclude that the E-value supports their observed findings.

INCIDENCE AND PREVALENCE

Incidence

Incidence is the proportion of a group that develops a disease (or experiences an event) over a specified period of time

Example

100 people without diabetes are followed for 5 years
At the end of 5 years, 10 people have developed diabetes
The incidence of diabetes over the 5 years is 10%

Prevalence

Prevalence is the proportion of a group that has the disease (or has experienced an event) at a single time point

Example

100 people are selected randomly
10 of the people in the group have diabetes
The prevalence of diabetes in the group is 10%

LIKELIHOOD RATIO (LR)

Overview

A likelihood ratio is a statistical ratio used to quantify the predictive value of a medical test (typically a screening test)
The likelihood ratio is the factor by which the post-test odds of a condition being present (positive likelihood ratio) or absent (negative likelihood ratio) are increased or decreased based on the results of a test
The likelihood ratio can be multiplied by the pre-test odds of the condition being present to determine the post-test odds
Post-test odds of disease = Pre-test odds X Likelihood ratio

Mathematically, the likelihood ratio is calculated using the sensitivity and specificity of the test:

Example

Test A is a screening test for heart disease
Paul is a 60 year-old patient who based on his family history, age, and medical conditions has a probability of 25% of having heart disease

Convert 25% probability to odds = 25/75 = 0.33

Test A has a negative likelihood ratio of 0.1 and a positive likelihood ratio of 1.9

Equivalent to a Sensitivity of 95% and a specificity of 50%

If Paul has a negative test A, then his post-test odds of heart disease are 0.1 X 0.33 = 0.033
If Paul has a positive test A, then his post-test odds of heart disease are 1.9 X 0.33 = 0.627
Converting back to probability:

negative test = 0.033/1.033 = 0.032 or 3.2% probability
positive test = 0.627/1.627 = 0.385 or 38.5% probability

In Paul's case, a positive test increases his probability of disease from 25% to 38.5% - not very helpful
A negative test decreased his probability of disease from 25% to 3.2% - more helpful
This is true because the test had a high sensitivity (low false negatives) and low specificity (high false positives)

NET RECLASSIFICATION INDEX/IMPROVEMENT (NRI)

Overview

The Net Reclassification Improvement (NRI), (also called "Index") is a statistical measure used to evaluate whether adding a measure to a predictive model will improve the predictive accuracy of the model. An example would be adding Coronary Artery Calcium Scores (CACS) to the Framingham risk model to see if the addition of the CACS improves the Framingham's ability to predict a heart attack. The NRI takes into account both correct and incorrect reclassifications.

The NRI is calculated in the following manner

1. In a study, take the people who had an event or outcome (events)
2. Calculate the number of events who were reclassified into a higher risk category when the new measure was included
3. Subtract the number of events who were reclassified into a lower risk category when the new measure was added
4. Divide this number by the total number of people who had an event
5. Now take the number of people who did not have an event or outcome (nonevents)
6. Calculate the number of nonevents who were reclassified into a lower risk category when the new measure was added
7. Subtract the number of nonevents who were reclassified into a higher risk category when the new measure was added
8. Divide this number by the total number of people who did not have an event
9. Add the two proportions together and you have the NRI

The formula for the NRI is as follows:

Example

The Framingham risk model uses age, gender, total and HDL cholesterol, smoking status, and systolic blood pressure to predict a person's 10-year risk of heart attack
Researchers want to know if adding Coronary Artery Calcium Scores (CACS) to the model improves the risk prediction of the original Framingham model
A cohort of patients is formed and their Framingham 10-year risk is calculated using the original risk model
The patients also have their CACS measured at baseline
The patients are followed for 10 years and the incidence of heart attacks is measured
Each patient's predicted risk of heart attack is calculated using the original Framingham model. The risk is also calculated using a model that incorporates the CACS.
The NRI is then calculated by comparing the two models

Interpretation of the NRI

It's important to understand what the NRI actually means because it is commonly misinterpreted

In the example above, let's assume the addition of the CACS to the Framingham model had the following effect:

71 patients with a heart attack are reclassified as higher risk
24 patients with a heart attack are reclassified as lower risk
790 patients without a heart attack are reclassified as lower risk
657 patients without a heart attack are reclassified as higher risk
Total number of patients who had a heart attack - 209
Total number of patients who did not have a heart attack - 5669

The calculation of the NRI is as follows:

Some people mistakenly interpret this as the addition of CACS correctly reclassifies 25% of the population studied
This is not correct. The addition of the CACS correctly reclassified 22.5% of the population who had an event (0.225 X 209 = 47) and 2.3% of the population that did not have an event (0.023 X 5669 = 130)
The percent of the total population correctly reclassified is 177/5878 = .03 or 3%
Whether or not 3% is clinically significant will depend on the cost and convenience of the additional measure

NUMBER NEEDED TO TREAT/HARM (NNT/NNH)

Overview

The number needed to treat (often abbreviated "NNT") is the number of patients who must be treated with an intervention in order for one patient to benefit
If the intervention is a screening test, it is the number of people who would have to be screened in order for one person to benefit from screening
The NNT can also be referred to as the "Number Needed to Harm" (NNH) when talking about a side effect or adverse event
The NNT is calculated by taking the reciprocal of the absolute risk reduction
The NNH is calculated by taking the reciprocal of the absolute increase in adverse events

Example

The recent NLST trial (detailed here) found that yearly CT scan screening reduced the absolute risk of lung cancer death by 0.33% in heavy smokers
The number needed to screen is = 1/0.0033 = 303 patients
This means 303 patients would have to be screened in order to prevent 1 lung cancer death

ODDS RATIO

Overview

When most people think of comparing two outcomes, they think in terms of relative risk
Relative risk is a comparison that uses fractions. It is the most intuitive way to compare two outcomes.

Example

Drug A has been shown to reduce the risk of heart attack by 20%
Most people understand that people taking Drug A will have 20% fewer heart attacks than people not taking Drug A

Definition of odds

When discussing odds ratios, it helps to review what odds are
Odds are ratios. Odds are not fractions. This can be confusing.

Odds = the number of times that an event will occur for the number of times that it will not occur
Example

3:1 odds means the event is likely to occur 3 times for every one time it does not occur
So for every 4 tries, there will be 3 events and 1 non-event. This equals a probability of 75% (3/4).

Convert between odds and probability

Odds ratio

An odds ratio is a ratio of two odds. Relative risk (sometimes called hazard ratios) is a ratio of two fractions.
Because of this distinction, the odds ratio and the relative risk ratio do not always approximate each other. This can be confusing because many people mistakenly interpret odds ratios as relative risk ratios.
As the prevalence of an outcome increases above 10%, the odds ratio and the relative risk start to diverge. This occurs because odds above 1:1 (0.50 probability) can run to infinity where a fraction (of a whole) is always between 0 and 1. This can lead to wide discrepancies between the odds ratio and the relative risk.

Example

Let's assume a group of diabetics are followed for a year and 80% of them suffer a heart attack
A group of nondiabetics is followed for a year and 20% of them suffer a heart attack
We want to compare the risk of heart attack between the two groups
The relative risk of heart attack in the diabetic group compared to the nondiabetics is (0.8)/(0.2) = 4. Diabetics have 4 times the risk of heart attack when compared to nondiabetics. Most people understand this.
The odds ratio of a heart attack when comparing the diabetics to nondiabetics is (4)/(0.25) = 16. This differs from the relative risk. It means diabetics have 16 times the odds of a heart attack than nondiabetics. This comparison is not as intuitive for most people and will often be misinterpreted as 16 times the risk of heart attack.

Rules for interpreting odds ratios:

When the incidence of the outcome of interest is < 10% in the study population, then the odds ratio and the relative risk can be considered equivalent
When the incidence of the outcome of interest is > 10% in the study population, and the odds ratio is > 2.5 or < 0.5, then the odds ratio will tend to exaggerate the magnitude of the association. In some circumstances the odds ratio can be converted to a relative risk (see below).

Odds ratio in medical studies

Case-control studies

Odds ratios are always reported in case-control studies because risk is not measured in these studies. The case group has a 100% probability of the disease and the control group has a 0% probability. The odds of exposure to a risk factor is compared between the two groups, therefore the results are reported as an odds ratio.

Cohort studies

Cohort studies often utilize logistic regression to control for covariates. Logistic regression yields an odds ratio. In some cases, the odds ratio can be converted to a relative risk. See formula below. [2]

Converting an odds ratio to relative risk

If a cohort study uses logistic regression and reports an odds ratio, it can be converted to relative risk if the incidence of the outcome in the unexposed (control) group is known

SURROGATE ENDPOINTS

Significant clinical outcomes

A significant clinical outcome is an outcome that has a significant effect on a person's health, functioning, or well-being
Examples of significant clinical outcomes include: death, heart attack, stroke, blindness, paralysis, cancer

Surrogate endpoints

Surrogate endpoints are measures that have been shown to be related to a significant clinical outcome
Surrogate Endpoints are often used in clinical trials because they are easier to measure and require less time to demonstrate an effect than significant clinical outcomes

Surrogate endpoint	Related significant clinical outcome
Cholesterol levels	heart attack, stroke, mortality
blood pressure	heart attack, stroke, mortality
carotid intima thickness	heart attack and stroke
Hemoglobin A1C level	diabetes complications

Example

High cholesterol levels are associated with heart attacks
A trial with a new cholesterol drug will measure its effect on cholesterol levels (surrogate endpoint)
This takes far less time (weeks to months) than measuring its effect on heart attack rates (years)

Issues with surrogate endpoints

Problems arise when interventions improve surrogate endpoints but do not improve (or may even worsen) significant clinical outcomes

This may occur for several reasons:

The surrogate endpoint may be related to the outcome through a factor that the intervention does not affect
The surrogate endpoint may be a marker of disease severity as opposed to being a cause of the disease
The intervention may be harmful in a way that is unrelated to the surrogate endpoint

Examples of interventions that improve surrogate endpoints, but do not improve significant clinical outcomes:

Folic acid and Vitamin B supplements

High homocysteine levels are associated with an increased risk of heart attack
Folic acid and Vitamin B supplementation lower homocysteine levels (surrogate endpoint)
Folic acid and Vitamin B supplementation have not been shown to decrease the risk of heart attack (significant clinical outcome) in patients with high homocysteine levels

Niacin and fibrates

High HDL levels (good cholesterol) are associated with a lower risk of heart attack
Niacin and fibrates raise HDL levels (surrogate endpoint)
These drugs have not been shown to improve overall mortality (significant clinical outcome) in patients with coronary heart disease

P-VALUE

Overview

The p-value is an important statistical measure used to compare findings between two outcomes
The p-value is expressed as a fraction (ex. p=0.01) which can be converted into a percentage (ex. p=0.01 same as p = 1.0%)
The p-value has the following meaning: If there really is no difference between the two groups, the p-value is the probability that the observed difference (or one greater) occurred by chance

Example

A study compares a new drug to prevent deep vein thrombosis to placebo
1000 people take the new drug for a year, and 1000 people take placebo for a year
At the end of the trial, the new drug group had 20 fewer deep vein thromboses than the placebo group
The researchers report the following p-value for the difference between the two groups: p=0.02
This means that if there really is no difference between the two groups (the new drug is ineffective), then the probability of observing the results found in this trial is 2%

Statistical significance

In most cases, a p-value < 0.05 is considered statistically significant. In the medical literature, it is often stated that the difference was "significant."
This cutoff value is somewhat arbitrary. For example, a p-value of 0.05001 would be considered nonsignificant, while a value of 0.04999 would be considered significant. The difference between these two values is miniscule. It's better to look at the p-value and consider its meaning.
A common guideline for considering p-values is presented below

Reference [1]
p-value	Significance
< 0.001	Very highly significant
0.001 to <0.01	Highly significant
0.01 to <0.05	Significant
0.05 to 0.10	Trend toward significance
> 0.10	Nonsignificant

POSITIVE PREDICTIVE VALUE (PPV) AND NEGATIVE PREDICTIVE VALUE (NPV)

Definition

Positive predictive value (PPV) - the proportion of patients with a positive test who actually have the disease
Negative predictive value (NPV) - the proportion of patients with a negative test who do not have the disease

Overview

Like sensitivity and specificity, the predictive value is a measure of test accuracy
The predictive value takes into account the prevalence of the disease in the population that is being tested. This gives a more meaningful interpretation of test results than sensitivity and specificity alone.
The main drawback to predictive values is that the prevalence of the disease in the tested population must be known or estimated with some accuracy

Example

Let's assume a colon cancer screening test has a sensitivity of 100% and a specificity of 97%. This test would be positive in everyone with colon cancer, and it would only be positive in 3% of people without colon cancer (3% false-positive). If someone has a positive test, they may assume they probably have colon cancer because the test is only wrong 3% of the time. This assumption would likely be false.
In order to understand the real meaning of a positive test, the prevalence of colon cancer in the population being screened must be considered
Let's assume the prevalence of colon cancer in the population being screened is 0.5%. That means for every 10,000 people screened, 50 of them will have colon cancer. Our test will be positive in all 50 people with colon cancer (100% sensitivity), and it will be positive in 3% of people without colon cancer. This means 3% of 9,950 people (299 people) will have a false-positive test. The percent of people with a positive test who actually have colon cancer (the PPV) is 14% (50/349 X 100).

Formulas for predictive values:

SENSITIVITY AND SPECIFICITY

Sensitivity

Sensitivity is a measure of how well a test identifies people with a disease
A test with a high sensitivity will be positive for most people with the disease
In statistical terms, it is the fraction of subjects with the disease who get a positive test result
If a test has a high sensitivity, a negative test result means the disease is unlikely (few false negatives)

Specificity

Specificity is a measure of how well a test excludes people without a disease
If a test has a high specificity, it will give a negative result for most people without the disease
In statistical terms, it is the fraction of subjects without the disease who get a negative test result
If a test has a high specificity, a positive test means the disease is likely (few false positives)

Sensitivity and specificity figure

Sensitivity / specificity and predictive value

It's important to understand how to apply the sensitivity and specificity of a test to a given population because it is easy to misinterpret the meaning of the measures
Let's assume a colon cancer screening test has a sensitivity of 100% and a specificity of 97%. This test would be positive in everyone with colon cancer, and it would only be positive in 3% of people without colon cancer (3% false-positive). If someone has a positive test, they may assume they probably have colon cancer because the test is only wrong 3% of the time. This assumption would likely be false.
In order to understand the real meaning of a positive test, the prevalence of colon cancer in the population being screened must be considered
Let's assume the prevalence of colon cancer in the population being screened is 0.5%. That means for every 10,000 people screened, 50 of them will have colon cancer. Our test will be positive in all 50 people with colon cancer (100% sensitivity), and it will be positive in 3% of people without colon cancer. This means 3% of 9,950 people (299 people) will have a false-positive test. The percent of people with a positive test who actually have colon cancer is 14% (50/349 X 100). This measure is called the positive predictive value (PPV). The PPV takes into account the prevalence of the condition in the screened population, and it gives a more meaningful interpretation of a positive test.

HAWTHORNE EFFECT

Overview

The Hawthorne effect is a phenomenon where subjects in a study modify their behavior because they know they are being observed. It is particularly relevant in unblinded studies because participants may behave differently based on whether they received an intervention (treatment group) or not (control group). The Hawthorne effect can also increase the effect size of shorter trials compared to longer ones because it tends to wane over time.

Example one (usual care)

Researchers have developed an EMR prompt system that provides doctors with evidence-based information on congestive heart failure (CHF) when they enter CHF orders. The researchers want to test whether the system improves CHF outcomes, so they randomize a group of hospitals to the prompt system and another group to usual care. The Hawthorne effect may cause the usual care doctors to order more evidence-based treatments because they know they are being observed, thus decreasing the impact of the prompt system.

Example two (unblinded treatment groups)

Studies in renal denervation exemplify the power of the Hawthorne effect. Renal denervation is a procedure where the sympathetic nerves innervating the kidneys are ablated. Decreased renal sympathetic tone reduces stimulation of the renin-angiotensin-aldosterone system (RAAS) and, in theory, lowers blood pressure. A study published in 2010 randomized 106 patients with resistant hypertension to renal denervation + continuation of current meds or continuation of current meds only. At 6 months, patients treated with renal denervation had office-based blood pressure reductions of 32/12 mmHg from baseline, and the control group had no change. [PMID 21093036] In 2014, a similar study was published, except that patients in the control group received a sham procedure so that all subjects were blinded to their treatment assignment. At 6 months, there was no significant difference in SBP reduction (14 vs 11 mmHg). [PMID 24678939] In both studies, blood pressure meds were continued unchanged during the 6-month follow-up period.
So how does one explain the large discrepancy in results? The Hawthorne effect likely played a role. In the first trial, patients who did not receive the procedure were likely disappointed and lost interest in the study. This may have caused differences in medication compliance, with the treated group being more compliant than the untreated. In the second trial, patients were blinded to the treatment they received, making them more likely to remain engaged in both arms.

IMMORTAL TIME BIAS

Overview

Immortal time is a period of time after study enrollment that patients in the treatment/exposure group cannot experience the outcome. It can occur when study enrollment and exposure/treatment are asynchronous. It is best explained using an example.

Example

Researchers want to look at whether a Drug A prevents rehospitalization for heart failure
To go about this, they take patients who have been hospitalized for heart failure and divide them into 2 cohorts by whether or not they were prescribed Drug A at discharge. They then follow the two cohorts to compare rehospitalization rates. In the treated cohort, there is a gap of time between hospital discharge and when the prescription for Drug A is filled. This gap of time is considered immortal time if the study design is such that patients in the treatment cohort who are rehospitalized before they have their prescription filled are excluded from the study.

Discussion

In the example above, excluding patients who have the outcome before they fill their prescription for Drug A creates bias because the study design has selected for patients who have gone event-free during the immortal time. The study outcomes are now biased toward the treatment group. To correct for this, the time between hospital discharge and prescription fulfillment should be counted toward the untreated (control) group for all participants.

INVESTIGATOR BIAS

Overview

Investigator bias occurs when the researchers in a study desire a particular outcome for the study
Investigator bias can occur at every level in the research process - design, data gathering, data analysis, and outcome reporting
Well-designed studies with careful blinding can help limit investigator bias
Investigator bias can also occur when researchers emphasize secondary outcomes that are significant when their primary outcome was not

Example

Investigators are studying a new screening test for breast cancer
Outcome measures include overall mortality and mortality from breast cancer
If the study finds that the screening test decreases breast cancer mortality, but has no effect on overall mortality, investigators may be tempted to report the decrease in breast cancer mortality and disregard the null effect on overall mortality. If they report both, they may emphasize the effect on breast cancer mortality and pay little notice to the effect on overall mortality. This practice is quite common.

PROTOPATHIC BIAS

Overview

Protopathic bias occurs when an intervention is used to treat the first symptoms of a disease that has not yet been diagnosed. The intervention may appear to be a risk factor for the disease when it is actually associated with symptom treatment.

Example

A study looks at the association of Depo-Provera® and uterine fibroids
The study finds that Depo-Provera® is significantly associated with uterine fibroids and declares it a risk factor
Depo-Provera® is often used to treat excessive vaginal bleeding, a presenting symptom of uterine fibroids
Protopathic bias may have influenced the findings in this study

RECALL BIAS

Overview

Recall bias occurs when a person with an outcome is more likely to recall an exposure than someone without the outcome
Recall bias is a major concern in observational studies, particularly case-control studies, because it may bias results toward a false positive effect (Type 1 error)

Example

A case-control study is exploring the relationship between over-the-counter NSAID use in pregnancy and birth defects
Women who have children with birth defects (cases) and women who have children without birth defects (controls) are asked about their over-the-counter NSAID use during pregnancy
In this study, recall bias may occur because women who have children with birth defects may be more likely to thoroughly contemplate their NSAID use during pregnancy than women who have normal children
This difference in attention to recall may bias the study toward a positive effect

REGRESSION DILUTION BIAS (ATTENUATION BIAS)

Regression dilution bias (also referred to as attenuation bias) is a form of bias seen in regression analysis. It occurs when there is an excessive amount of measurement error in the independent variable. The excessive error leads to an underestimation of the slope (coefficient) of the regression line.
Regression dilution can be corrected for in several ways. To decrease the amount of measurement error, the independent variable can be measured several times on each subject and an average can be used. If remeasuring the entire sample is not feasible, then a smaller subset of randomly selected individuals can be used. In other methods, subjects at 20 - 30% of the extremes are remeasured. The average of these measurements can then be used to correct the slope of the regression line.
Correcting for regression dilution is not usually necessary in hypothesis testing for linear relationships (slope ≠ 0)
Corrections should be performed when regression coefficients are used to measure effect sizes [7]

REPORTING BIAS

Definition

Reporting bias occurs when an event, typically a side effect, is more likely to be reported when it happens with therapy A as opposed to therapy B. This can make it appear as though the event has a stronger association with therapy A when there really may be no underlying difference. Reporting bias is often seen with new therapies because physicians and patients are unfamiliar with their effects.

Example

After the GLP-1 drugs (GLP-1 analogs and DPP-4 inhibitors) were introduced around 2005, the FDA began to receive postmarketing reports of pancreatitis and pancreatic cancer in patients treated with these agents. This prompted the FDA to issue a warning to providers about a possible association. After years of collecting and reviewing data, the FDA published a joint paper with its European counterpart in 2013 that concluded there was no definitive link between GLP therapies and pancreatitis or pancreatic cancer. [PMID 24571751]
This example illustrates two potential instances of reporting bias. In the first case, doctors may have been more inclined to report pancreatic events in patients receiving GLP drugs because they were less familiar with them and their side effects. Consider this, if two patients developed pancreatitis, one on a GLP-1 analog and the other on a sulfonylurea, which case was more likely to get reported? The GLP-1 analog because providers have less experience with it than sulfonylureas, which have been around forever and aren’t known to cause pancreatitis. This bias in reporting makes it look like GLP-1 analogs are associated with pancreatitis, when in reality, the incidence of pancreatitis among patients receiving GLP-1 analogs and sulfonylureas may be no different. The other instance of reporting bias probably occurred after the FDA issued a warning about the association. Drug warnings raise awareness, increasing the likelihood that an event will be identified and reported. That’s why it’s important for patients and providers to understand that drug warnings based on case reports in the FDA’s Adverse Event Reporting System are prone to reporting bias and often later disproved by data from better sources, as was the case with GLP drugs.

SAMPLING/ASCERTAINMENT BIAS

Overview

Sampling bias (also referred to as "ascertainment bias") occurs when the procedure for sampling a population inadvertently leads to the over/under-representation of a subgroup of that population

Example

Researchers want to study a new drug for chronic back pain
To recruit subjects, the researchers decide to enroll patients presenting to pain management clinics for chronic back pain
By recruiting patients from this select population, the researchers run a high risk of sampling bias
Patients who are referred to pain management clinics often have complicated histories that might include opioid-seeking behavior
If the medication they are studying does not have opioid properties (e.g. an NSAID), it may not perform well simply because a large portion of the population is seeking an opioid medication. Conversely, if the medication is an opioid, it may perform better than expected because it is the desired drug type.
In either case, sampling bias has occurred because a subpopulation of patients with chronic back pain (opioid-seekers) is overrepresented in the sample

SELECTION BIAS

Overview

Selection bias occurs when an intervention or treatment is more likely to be given to one person over another because they differ in some way. The underlying difference between the individuals may not be obvious or even measurable.
Selection bias is a big concern in observational studies because patients are not randomized

Example

An observational study compares outcomes between diabetics who received Medication A and those who did not
Medication A requires that patients return to the clinic every 3 months for blood tests
The study shows that diabetics who took Medication A had better outcomes
There is a big concern for selection bias in this study because it may be that doctors were more likely to prescribe Medication A to their patients who were more compliant
The difference in compliance between the two groups can bias the outcomes
If Medication A is expensive, patients in a higher socioeconomic class may have been more likely to receive it. This can also bias outcomes.

SURVEILLANCE BIAS

Overview

Surveillance bias occurs when a proposed risk factor for a condition is independently associated with increased medical testing (e.g. labs, X-rays, screening). The prevalence of the condition may appear to be greater in patients with the risk factor when in reality, differences in medical testing (surveillance) are the reason for the discrepancy.

Example

A cohort study published in the Annals of Rheumatic Diseases, where researchers looked for an association between polymyalgia rheumatica (PMR) and cancer, provides the perfect example of surveillance bias. [PMID 23842460] Using a database, researchers formed two cohorts: 2877 PMR patients and 9942 matched controls without PMR. They followed the cohorts for a median of 7.8 years and compared the incidence of cancer between them. At the end of the study, patients with PMR were more likely to be diagnosed with cancer than controls, so the authors concluded that PMR might increase the risk of cancer.
This study runs a high risk of surveillance bias because it's likely that PMR patients had more medical testing than controls. In fact, the authors found an interaction between time and cancer diagnosis that showed diagnoses were significantly greater in PMR patients within the first 6 months of their PMR diagnosis but not later. Cancers that were more prevalent in PMR patients included hematologic and urinary tract neoplasms, conditions that cause abnormalities on routine blood work. Most telling, though, is prostate cancer, which was almost 4 times more likely in PMR patients. There is no reason to check a PSA or do a rectal exam on patients with PMR symptoms, so this shows that these patients received more screening because of their interactions with providers.
In this example, PMR is the proposed risk factor, cancer is the condition, and surveillance bias exists because PMR is independently associated with more medical testing.

CONFOUNDING BY INDICATION

Overview

Confounding by indication can occur when a condition is a risk factor for an outcome among both exposed/treated individuals and unexposed/untreated controls. If the condition is associated with the exposure/treatment, then confounding by indication may occur, given that the condition is not an intermediate step in the causal pathway to the outcome. [4]

Example

Researchers want to evaluate if Byetta® is associated with pancreatitis
They conduct a case-control study where they compare the risk of pancreatitis between diabetics on Byetta® and diabetics not on Byetta®
The study finds that diabetics who took Byetta® were at greater risk for pancreatitis
There is a risk for confounding by indication in this study. Obesity is a risk factor for pancreatitis. Byetta® causes modest weight loss. It may be that doctors were more likely to prescribe Byetta® to obese diabetics. Because these patients were already predisposed to pancreatitis, confounding by indication may have occurred.

CONFOUNDING BY SEVERITY

Overview

Confounding by severity is a type of confounding by indication
In confounding by severity, patients who are sicker may be more prone to receive an intervention
This may cause the intervention to appear less effective, or even inferior in some cases [4]

Example

A study looks at the association of minoxidil and stroke
It finds that patients who took minoxidil as compared to other blood pressure medications were more likely to have a stroke
Minoxidil is typically reserved for patients who have resistant, difficult to treat hypertension
Confounding by severity is likely because patients with more severe disease are more likely to receive minoxidil. This would bias the results against minoxidil.

RESIDUAL CONFOUNDING

Overview

Residual confounding occurs when a variable that is related to an outcome is not stratified appropriately
Confounding is present because subjects grouped together in a strata have different risks for the outcome
The risk for residual confounding is particularly high when a variable and the risk for an outcome have a nonlinear relationship (ex. J-shaped or bimodal)

Example

Researchers want to look at the association of BMI with mortality
They form two cohorts of patients: Group 1 - BMI < 27; Group 2 - BMI ≥ 27
After observing the two cohorts for 10 years, they find no difference in the risk of mortality and declare that body weight does not influence mortality
Residual confounding is very likely in this study because the researchers did not stratify patients appropriately
BMI has been shown to have a J-shaped relationship with mortality. Patients with a very low BMI have a higher mortality rate than patients with a normal BMI, and patients with a high BMI have a higher mortality rate than patients with a normal BMI
Because the authors only formed two BMI groups (< 27, ≥ 27), they came to the erroneous conclusion that BMI is not related to mortality. If they had stratified patients into five groups of BMIs (e.g. < 20, 21-25, 26-30, 31-35, ≥ 36), they would have come to a different conclusion.

COVARIATE AND CONFOUNDER

Definitions

Covariate - a covariate is an independent patient variable that may or may not influence the subject’s susceptibility to an outcome. Some covariates are easily measured (e.g. age, sex, race, medical history), while others are not (e.g. patient attitudes and personality traits, compliance). When designing a study, researchers must identify covariates they think are relevant and do their best to distribute them equally between study arms.
Confounder - a confounder is a covariate that is associated with both the outcome and the exposure. A confounder may either increase or decrease the likelihood of the outcome.

The illustration below describes their relationship

illustration of covariates and confounders and how they are related to outcomes and exposures

Example

Researchers want to see if vitamin D supplements prevent melanoma. Using a medical database, they identify two large cohorts of patients. One cohort takes a daily vitamin D supplement, and the other does not.
The researchers follow the two cohorts for ten years and compare the incidence of melanoma between them

In our example, we have the following:

Outcome - melanoma
Exposure - daily vitamin D supplement

There are a number of covariates we should consider, but we are going to focus on two: Race and Sun exposure

Using our illustration, we have the following:

Covariate - Race

Race is associated with melanoma risk, as light-skinned races have a much higher risk than dark-skinned races
Race is not associated with taking daily vitamin D supplements
Race is a covariate but not a confounder
The two cohorts were formed based on daily vitamin D use. If the cohorts are large enough, the racial composition of each cohort will be similar, and race will not influence the outcome. If the cohorts differ significantly in racial makeup, the analysis should be adjusted for ethnicity.

Confounder - Sun exposure

Sun exposure is associated with melanoma risk
Sun exposure is also associated with taking daily vitamin D supplements. Possible reasons include the following: (1) people who take vitamin D supplements are more health-conscious, and therefore, they also avoid excessive sun exposure; (2) people who wear sunscreen and avoid the sun may have read that they are not making enough vitamin D in their skin, so they take supplements; (3) people who avoid the sun may have lower vitamin D levels, causing their doctor to recommend a supplement.
Since sun exposure is associated with melanoma risk and taking daily vitamin D, it is a confounder
When the data is analyzed, it shows that vitamin D users have a lower risk of melanoma. This may lead researchers to conclude that vitamin D supplementation lowers melanoma risk, when in fact, reduced sun exposure among users is the real reason for the lower risk

Illustration of covariate-confounder relationship

IMPORTANT POINTS ABOUT DRUG INTERACTIONS

Drug interactions are challenging

Information on drug interactions can be difficult to assimilate
Certain drug interactions and metabolic pathways are well-defined while others are not

Factors that can make drug interactions challenging

New drugs

During drug development, the FDA requires interaction testing with medications that are known to have significant effects on CYP enzymes and transporters. Obviously, there is no way to test a new drug with every medication available, meaning most drugs come to market with incomplete interaction profiles. After the medication is prescribed to a large number of people, other drug interactions are inevitably discovered.

Research

Drug research often occurs in a laboratory setting (in vitro) with animals and cell cultures. Findings from these experiments do not always reflect what happens in the human body (in vivo).

Evolving information

Drug metabolism is an evolving field, and researchers are just beginning to understand all the different ways the body eliminates medications. Cytochrome P450 enzymes have been studied for a while, but cell transport systems (e.g. p-glycoprotein, OAT) are a relatively new area of pharmacology, and their role in drug elimination has not been completely elucidated.

Important points

Not all drug interactions are known or can be predicted
Good information on possible drug interactions may not be available
Not all drug interactions are clinically significant, and patients should consult their healthcare provider if they are concerned about a possible interaction
Drug interaction checkers provide the most efficient and practical way to check for interactions among multiple medications. A free interaction checker is available from Drugs.com (see Drugs.com interactions checker).

FDA ADVERSE EVENT REPORTING SYSTEM (AERS)

Overview

The AERS is a database run by the FDA where adverse events thought to be related to drugs are recorded and followed. Reporting of adverse events by the general public is voluntary, and anyone who thinks they have experienced an event can report it. Drug companies are required to report any information they receive about events to the system.
Based on information in the database, the FDA sometimes takes action on a drug. Actions may include changes to the drug labeling, restricting use of the drug, issuing warnings to healthcare professionals and patients, and removing the drug from the market.

Important points

When the FDA issues a warning based on information from its AERS, it often draws headlines and causes concern among patients and healthcare providers alike
While the concern is warranted, it's important to understand that the database in no way, shape, or form constitutes a statistical analysis of the adverse event being reported
The purpose of the AERS is to flag trends in adverse events that need to be explored further with proper studies
It does not prove causality

Reporting bias

Adverse event reporting is susceptible to a tremendous amount of reporting bias
Reporting bias occurs when adverse events associated with a new drug or treatment are more likely to be reported than adverse events associated with older, more familiar drugs and treatments
Because the prescriber is less familiar with the new drug, he may be more inclined to report an adverse event he thinks is associated with the new drug
A prescriber may be less inclined to report an adverse event for a drug that he has used for a long time. He may be biased in thinking that the drug is safe, and therefore he does not associate it with the adverse event, or the adverse event may already be known, and he doesn't see the point in reporting it.

Example

Scenario one

Doctor prescribes brand new drug to patient who subsequently develops pancreatitis
Doctor may be inclined to report pancreatitis as an adverse event for the drug because it is new, and he is unfamiliar with it
If a warning has already been issued for the drug, this may also bias his reporting of the event

Scenario two

Doctor prescribes drug he has used for years to patient who subsequently develops pancreatitis
Doctor may not consider the drug as a possible cause since he has used it for years and is comfortable with it

Result

In the AERS, pancreatitis is being reported for the new drug, while older drugs are not receiving adverse reports
In reality, there may be no difference in the incidence of pancreatitis between the new drug and the older drug

OVERDIAGNOSIS

Definition

Overdiagnosis occurs when a condition is diagnosed that would otherwise not go on to cause symptoms or death
Overdiagnosis has become a larger issue as the utilization of screening tests has increased
Overdiagnosis is a particular problem in cancer screening since almost all cancers are treated. Treatment of all cancers leads to overtreatment in some cases. Overtreatment can adversely affect a person's quality of life.

Example

Patient A is a 60 year-old man with slow-growing prostate cancer and heart failure
Patient B is a 60 year-old man with slow-growing prostate cancer and heart failure
Patient A has a screening PSA test that detects his prostate cancer. Patient A then undergoes radiation and androgen deprivation therapy. Patient A develops impotence, hot flashes, and gynecomastia from the treatment. He dies five years later from his heart failure.
Patient B is never screened for prostate cancer and never knows that he has it. He dies six years later from heart failure
Patient A was overdiagnosed because his prostate cancer would not have led to his eventual death. The treatment of his prostate cancer may have hastened his death, and his quality of life was adversely affected.

Measuring overdiagnosis

Overdiagnosis is best measured in comparative clinical trials where one group is screened and another group is not screened
Consider a randomized controlled trial that compared a screening test for breast cancer in one group (screened) to no screening in another group (control). At the conclusion of the screening phase of the trial, the screening group will have more breast cancer diagnoses than the control group, because screening tests advance the time of cancer diagnosis.
If there is no overdiagnosis, then as time goes by, the control group will catch-up with the screened group, and the number of breast cancers diagnosed in each group will be equal
If overdiagnosis has occurred, then the control group will have a lower breast cancer incidence than the screening group. This means the screening group had cancers detected that would have never led to clinically detectable disease.

Issues with overdiagnosis

Measuring overdiagnosis in a trial can be difficult because it requires many years of follow-up. In order to accurately predict the incidence of cancer in each group, the length of follow-up must allow for slow-growing cancers to become clinically apparent. This can take up to 7 or more years in some cases.
At the time of diagnosis, it is often not possible for a doctor to know if the cancer will cause the patient to die, or if the patient will die from something else
Overdiagnosis assumes that the undiagnosed cancer did not contribute to the patient's death or morbidity [3]

Bias with overdiagnosis

Overdiagnosis can lead to bias in survival statistics
If a screening test detects a condition in a significant number of people who will never die from the condition (overdiagnosis), then it will inflate the survival advantage of the screening test
The link below is to an illustration from Wikipedia that demonstrates this phenomenon

Bias in overdiagnosis illustration

As detailed in the illustration, the screening test detected a large number of patients who did not die from lung cancer (referred to as "pseudodisease"). This inflated the 10-year survival for the screened group.
Screening tests can also inflate survival statistics because they diagnose a condition earlier. For example, assume a cancer has absolutely no effective treatment. Patients who are screened for the cancer may have it diagnosed years before patients who are not screened. The screened group will have a longer "survival time" simply because they were diagnosed earlier. In reality, screening had no real effect on cancer mortality.

PMID

PMID is short for "PubMed Identification #"
PubMed is an online resource developed and maintained by the National Center for Biotechnology Information
PubMed comprises over 20 million citations for biomedical literature from MEDLINE, life science journals, and online books
Citations on this site are listed by their PubMed Identification number. Example - PMID 123400978
Citations can quickly be retrieved by searching PubMed with the ID number

PRIMARY PREVENTION

Overview

Primary prevention involves taking measures (ex. medications, exercise, weight loss) to prevent the occurrence of an event (ex. heart attack) or illness (ex. cancer) before the underlying disease has been established in the patient
Primary prevention differs from secondary prevention where the underlying disease has already been established

Example

Patient has never had a heart attack, but he is concerned about it because his father had one
Patient starts taking aspirin every day to prevent a heart attack (Primary prevention)

SECONDARY PREVENTION

Overview

Secondary prevention involves taking measures (ex. medications) to prevent the recurrence of an event (ex. heart attack) or illness (ex. cancer) after the underlying disease has already been established in the patient
Secondary prevention differs from primary prevention where the underlying disease has not been established in the patient

Example

Patient has a stroke
Patient then starts taking aspirin every day to prevent another stroke (secondary prevention)

BIBLIOGRAPHY

1 - Fundamentals of Biostatistics 7th ed.
2 - PMID 9832001 RR paper
3 - PMID 20413742
4 - PMID 10355372
5 - PMID 26501539 - PSM JAMA
6 - PMID 17909367 - PS inverse weight
7 - PMID 22401135 - Regression dilution bias
8 - PMID 29164242 - Mendelian Randomization, JAMA (2017)
9 - PMID 11602018 - The paired availability design for historical controls, BMC Med Res Methodol (2001)
10 - PMID 30676631 - Using the E-Value to Assess the Potential Effect of Unmeasured Confounding in Observational Studies, JAMA (2019)
11 - PMID 31046064 - Using Instrumental Variables to Address Bias From Unobserved Confounders, JAMA (2019)
12 - PMID 34032845 - Adjusting for Nonadherence or Stopping Treatments in Randomized Clinical Trials, JAMA (2021)
13 - Chaimani A, Caldwell DM, Li T, Higgins JPT, Salanti G. Chapter 11: Undertaking network meta-analyses. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (editors). Cochrane Handbook for Systematic Reviews of Interventions version 6.3 (updated February 2022). Cochrane, 2022. Available from https://training.cochrane.org/handbook/current/chapter-11

MEDICAL STUDIES