





































34    AMWA Journal / V36 N1 / 2021 / amwa.org        

Efficacy and Effectiveness
In our previous article (Schindler TM, Bridge H. The evalu-

ation of efficacy, or how do we know whether a treatment 

works? Part 1. AMWA J. 2020;35(2):82-86), we described how 

randomized clinical trials (RCTs) determine efficacy of a 

treatment. Efficacy is generally understood as the ability of 

a drug or other intervention to reliably produce a positive 

effect in patients with a defined disease under controlled 

conditions. In other words, clinical trials are good tools to 

evaluate efficacy. Clinical trials provide a well-controlled 

framework that is characterized by the following: 

• Selecting participants according to clearly defined  

eligibility criteria

• Randomly allocating participants to treatment groups

• Concealing study treatments to participants and study 

doctors (blinding)

• Clearly defining endpoint measures before study start

 Provided a drug consistently demonstrates efficacy in 

several clinical studies and appears safe, regulators will 

approve it and permit its marketing for the treatment of a 

disease. However, once a drug is on the market, it will be 

used for the entire spectrum of patients and under circum-

stances that might not have been tested in the clinical stud-

ies. For example, the new drug will be used in patients who 

have several comorbidities in addition to the approved indi-

cation. The evaluation of a drug’s effects under everyday 

conditions is called effectiveness research.

Why Does a Treatment’s Effectiveness Differ  
From Its Efficacy?
There are many reasons why the effects of a drug in everyday 

clinical practice are different from those seen in clinical trials. 

In short, the entire societal-medical context contributes to the 

effectiveness of a drug or intervention. Before we look at some 

of the key factors, we need to remind ourselves that both “effi-

cacy” and “effectiveness” are determined in groups of people 

and not individuals. While efficacy pertains to groups of par-

ticipants in clinical trials, effectiveness pertains to all people 

with a disease who could be treated with the drug. 

Issue 1: Physical Availability and Accessibility

Above all else, to be effective, a drug needs to be available to 

patients, at best to all patients with the disease it was devel-

oped to treat. There are national, regional, and local aspects 

of availability of a treatment. Approval of a drug by regula-

tors permits its marketing and selling in a certain country. 

For example, a drug that is approved in the United States 

may not be available in Canada or Mexico. 

 Before the new drug reaches a patient, a number of 

obstacles in the supply chain need to be overcome. The first 

one is reimbursement, ie, the negotiation of price for the 

new drug with government agencies, insurance companies, 

and pharma wholesalers. Discussions on price may have 

a drastic effect on availability. If a pharmaceutical com-

pany considers the proposed price too low to ever regain its 

investments, it may decide not to provide the drug at all. 

 Regional differences in availability may arise when 

regional pharma wholesale companies decide on stocking 

of medicines. They may decide not to stock a certain drug 

in certain locations because they do not believe that it will 

be widely used there. Even if a drug is available via a whole-

saler and the physician is willing to prescribe it, a patient’s 

health insurance plan may not cover the entire cost for the 

new treatment. Likewise, if a patient is in hospital, she might 

not get the drug because the company that runs the hospi-

tal may not see its benefits and may therefore not include it 

in their treatment plans and offer an alternative treatment. 

Furthermore, health care providers may not believe in the 

benefits of a certain drug and may prescribe treatments that 

they consider to have superior therapeutic effects instead. 

Thomas M. Schindler1 and Helen Bridge2 / 1Innovation Medical Writing, Boehringer Ingelheim Pharma,  

Biberach a.d. Riss, Germany; 2Medical Communications, AstraZeneca, Cambridge, United Kingdom

STATISTICALLY SPEAKING

The Evaluation of Effectiveness, or How Do We Know 
Whether a Treatment Works in Real Life? Part 2



      AMWA Journal / V36 N1 / 2021 / amwa.org    35

Issue 2: Medical Tradition

Medical doctors, ie, the prescribers of medicines, undergo 

many years of training, and they apply this knowledge in their 

clinical practice, thereby maintaining a certain medical tra-

dition. Such traditions are tenacious, and it may take a long 

time before new therapeutic options are widely accepted and 

offered to all patients. Conversely, health care providers and 

hospitals may continue to offer treatments to their patients 

that have long been shown to be less effective because this is 

their established practice, offers financial advantages, or is 

expected by patients. 

 Medical practitioners follow the insights they gain in their 

medical practice. If they have the impression that a particular 

medicine works for their patients, they will continue to pre-

scribe it. They will stick to what they believe is helpful and may 

not use a new drug with which they have little experience. This 

is particularly likely when a new drug has only been tested 

against placebo and not against established treatments. A new 

drug may not be given to a patient because the physician has 

not heard about it or does not believe it is superior to the drugs 

that she usually prescribes.

Issue 3: Treatment Adherence 

In clinical trials, participants are closely monitored with regard 

to how and when and at which dose they take their medica-

tion. For example, study participants are reminded to take 

their dose at the same time every day, with or without food, 

to achieve the optimal effect. In normal life, things tend to be 

different, and patients might occasionally forget to take their 

medication. On days when their disease is particularly dis-

comforting, they may be inclined to increase the dose of their 

medicine. If their lives are busy, patients may forget to renew 

their prescription in time and may therefore not take any med-

ication for a while. Thus, although the drug is available to the 

patients, its effects may be smaller than those observed in the 

clinical studies because of limited adherence to medication 

plans. Furthermore, patients might be taking additional drugs 

to treat other conditions, and these drugs may influence the 

effects of the new treatment.

Real-World Data and Real-World Evidence to 
Determine Effectiveness
Many different factors affect the use of a drug in everyday 

medical practice. It is therefore very difficult to determine how 

effective a drug is “out there.” Post-marketing (Phase 4) studies 

usually focus on safety rather than effectiveness. However, for 

many stakeholders in the health care system, it is important to 

know how well a drug works in the real world: 

• patients taking the medicine want to know whether it works, 

• payers like health insurance companies need to understand 

whether they are paying for an effective treatment,

• government agencies want to know how the new medicine 

affects public health, 

• pharmaceutical companies want to understand to what 

extent available medicines meet patients’ needs and 

whether there is space for additional medicines.

 Recently, there has been much discussion about the use-

fulness of “real-world data” (RWD) and ”real-world evidence” 

(RWE) to evaluate effectiveness of treatments and support 

health care decisions. RWD and RWE are generally understood 

to refer to data and evidence from sources other than tradi-

tional clinical trials, but it has not always been clear exactly 

what these categories include. Over a number of years, regu-

lators were pressed to delineate ways to incorporate nonclas-

sical data in their evaluations. In 2018, the US Food and Drug 

Administration (FDA) released the framework for the Agency’s 

Real-World Evidence Program. There, they give the following 

definitions:

• RWD is “data relating to patient health status and/or the 

delivery of health care routinely collected from a variety 

of sources.”

• RWE is “the clinical evidence about the usage and potential 

benefits or risks of a medical product derived from analysis 

of RWD.”

Real-World Data: Abundant, Promising, but Messy

Sources of RWD include registries, electronic health records, 

medical claims databases, mobile devices, and so on. The 

recent excitement about RWD and RWE has focused on the 

availability of apparently rich sources of data, particularly 

those resulting from digitization. Sources such as electronic 

health records from entire health systems or wearable devices 

that generate data on a plethora of variables at frequent inter-

vals promise an abundance of easily collected data.

 There is a growing tendency to believe that data collected in 

the course of routine health care, whether via the records kept 

by health care providers or directly from patients using apps, 

better reflect how treatments actually perform than data col-

lected in the research settings of clinical trials. Large quantities 

of RWD can easily be collected from thousands, or even mil-

lions, of patients, and such “big data” seem to promise greater 

representativeness than data from the comparatively small, 

selected populations of patients included in clinical trials.

 There are also clear practical and economic grounds for 

exploring RWD: vast quantities of such data can be collected 

and accessed quickly and cheaply by comparison with the  

laborious and costly collection of data in traditional clinical 

trials. RWD therefore holds appeal for companies keen to reduce 

the expense and time taken to bring treatments to market.

STATISTICALLY SPEAKING



36 AMWA Journal / V36 N1 / 2021 / amwa.org 

 Easy and cost-effective access to large quantities of medi-

cal data collected from the full range of users of a treatment 

in real-world conditions may sound too good to be true. Sure 

enough, there are several major obstacles to using RWD to 

reach reliable conclusions regarding the effectiveness of  

treatments.

Obstacle 1: Data Quality

One of the factors behind the expense of running clinical trials 

is the quest for good data quality and the consequent rigor 

with which data are collected. The clinical trial protocol speci-

fies precisely which data are to be collected for each patient, 

the time points at which they are to be collected, and the 

methods, often including the precise equipment, that are to 

be used. A trial is designed to answer specific scientific ques-

tions, and the data to be collected are those that are required 

to answer these questions. Data are recorded on a case report 

form designed specifically for the trial. Strict procedures are 

followed to ensure data are collected in accordance with 

Good Clinical Practice principles, and clinical trial monitors 

verify the data for completeness and accuracy. These meth-

ods ensure that clinical trial data are highly standardized from 

patient to patient and from center to center, with the same 

variables measured and recorded in the same way and at the 

same time points. These provisions make clinical trial data 

reliable and trustworthy.

 Data from real-world sources are unlikely to share any of 

these qualities. Table 1 outlines the main problems with qual-

ity of RWD. These deficiencies introduce noise and bias that 

make it difficult to draw reliable conclusions about the effec-

tiveness of a treatment. 

Obstacle 2: Lack of Randomization

Medical research generally falls into 1 of 2 broad categories: 

clinical trials and observational studies (Figure 1). The key 

feature that separates these approaches is the presence or 

absence of randomization. A recent opinion piece in the New 

England Journal of Medicine insightfully contrasts the “magic 

of randomization” in RCTs with the “myth of real-world evi-

dence” from observational studies. Randomization ensures 

that there are no systematic differences between treatment 

groups with regard to patients’ characteristics that may affect 

efficacy outcomes. It is likely impossible to achieve such a 

balance in an observational study because the groups that 

are compared did not result from randomization. Whenever 

the treatment is a choice, whether by a doctor or the patient, 

groups of patients taking different treatments are likely to 

differ systematically, often in ways that are difficult to identify. 

Moreover, the reasons for the choice of a particular treatment 

are almost never entered into health records or databases. This 

is likely to result in a biased comparison of the treatments. 

Indeed, there have been well-publicized cases in which obser-

vational studies and RCTs have come to opposing conclusions 

about particular treatments.

 An obstacle to using RWD to evaluate effectiveness is that 

these data originate in routine health care contexts in which 

patients are not randomized to treatments. Although such 

data have long been used by regulators to evaluate the safety 

of treatments, using them to arrive at unbiased evaluations of 

effectiveness is challenging and requires sophisticated con-

siderations on methodology. In recent years, statisticians 

have worked on developing new study designs and complex 

analysis methods, including so-called “causal inference” and 

machine-learning methods, to help overcome some of the 

limitations of analyzing observational data. The FDA has com-

mitted (in its RWE framework document) to evaluating “the 

potential role of observational studies in contributing to evi-

dence of drug product effectiveness” and has supported a 

series of workshops that included discussion of methods for 

assessing and minimizing bias in observational studies.

STATISTICALLY SPEAKING

Table 1. Real-World Data: Common Problems With Data Quality

Characteristic of Data Likely Problems With RWD

Relevance to research 
questions

Data are collected for purposes other 
than research and will not be optimal for 
answering the question of interest. There 
is a risk of allowing the available data to 
determine the research questions that are 
asked (rather than defining the question 
first and then looking for the data).

Reliability Inaccuracies in the data may result from 
human error or faulty devices. Data 
collected cannot be verified by comparison 
with source data. Random errors and 
systematic bias are difficult to identify  
and impossible to correct.

Completeness RWD are characterized by a high quantity 
of missing values, with no information as 
to why data are missing.

Consistency and 
interoperability

Variables recorded, measurement methods, 
and data formats and data standards 
usually vary greatly across patients, health 
care providers, devices, companies, etc. 
The lack of common data standards, ie, 
the way data are structured, stored, and 
summarized, makes it challenging to 
collate and analyze the data.



      AMWA Journal / V36 N1 / 2021 / amwa.org    37

 

Obstacle 3: The Need for Rigorous Research Methodology

No amount of data can, in itself, help us evaluate treatments. 

Data only become evidence once they are used within a 

methodological framework or research study to answer spe-

cific questions, such as “Is treatment X effective in patients 

with heart failure?” or “How much more effective is treat-

ment A than treatment B at extending survival in patients with 

advanced non-small cell lung cancer?” For RWD to provide 

evidence of effectiveness, they need to be analyzed using an 

appropriate research methodology. This includes the defini-

tion of data formats for datasets, the availability of a compre-

hensive study protocol, and detailed analysis plans to ensure 

reproducibility of results. If data are derived from novel data 

sources such as wearables, these data need to accurately 

reflect the clinical outcome that is being investigated.

 

Combining Clinical Trials and Real-World Data for 
Insight into Effectiveness 
Observational studies are not the only way of using RWD to 

tackle the question of effectiveness. A promising alternative 

approach advocated in the FDA framework document as a way 

of generating RWE is to make clinical trials more “real life.” 

This can be done by simplifying the trial design to become a 

“large, simple trial” or by otherwise incorporating pragmatic 

elements in the design so that patients’ treatment within the 

trial closely resembles routine clinical practice (Figure 1).  

Alternatively, hybrid designs can be used whereby health 

data that are routinely collected are used in the trial, together 

with data specified by the trial protocol (for example, effi-

cacy-to-effectiveness or efficacy-and-effectiveness-too trials). 

Combining clinical trial methodology, notably randomization, 

with the collection and evaluation of RWD has the benefit of 

enabling a more unbiased evaluation of treatment effects in 

settings that are close to real-world clinical practice. 

 Particularly in rare diseases, it is often impossible to con-

duct RCTs because of the low number of patients available. 

Recruiting a sufficient number of patients into a study may 

take too long to yield useful results. In these instances, sin-

gle-arm open-label studies may be conducted and the results 

compared with external controls, ie, RWD collected outside 

of the study. The control data could come from registries, 

medical records, scientific literature, or expanded access pro-

grams. A recent example of such an approach is the approval 

of avelumab in metastatic Merkel cell carcinoma. The drug 

was approved in 2017 based on a single-arm open-label study 

that compared the study outcomes with historical controls 

retrieved from electronic health records. In 2019, approval of 

palbociclib for HR+, HER2- advanced breast cancer in men, a 

label extension, was based on post-marketing reports and elec-

tronic health records.

 Responses to the coronavirus disease 2019 (COVID-19) 

pandemic provide a further, highly topical example of how 

randomized trials that incorporate real-world elements can 

generate reliable evidence of effectiveness to guide clinical 

decision-making about treatments. At the time of writing, a 

number of large, simple trials to evaluate various potential 

Figure 1. Idealized theoretical framework for the generation of evidence for efficacy and effectiveness. Hybrid trials are 
studies that combine efficacy and effectiveness assessments. E2E, efficacy-to-effectiveness trial (sequential assessment); 
EE2, efficacy-and-effectiveness-too trial (simultaneous assessment); EHR, electronic health records.

STATISTICALLY SPEAKING

Continuity of evidence

Classic RCT

Pragmatic
trials

Large simple
trials

Hybrid trials,
E2E, EE2

Data 
observed 
for research 
purpose

Electronic 
data, EHR, 
insurance 
claims, etc

Unstructured 
data

EFFICACY

Randomized trials Observational studies

EFFECTIVENESS

©TMSchindler & HBridge 2021



38 AMWA Journal / V36 N1 / 2021 / amwa.org 

treatments for COVID-19 are ongoing. For example, the global 

Solidarity trial, initiated by the World Health Organization, 

had recruited over 12,000 patients by October 2, 2020, with 

116 countries having joined or expressed an interest in join-

ing the trial. In the United Kingdom, the RECOVERY trial is 

being conducted at all major hospitals and had enrolled over 

20,000 patients by December 2020. These trials have simple 

protocols and heavily streamlined procedures. The aim is to 

maximize recruitment and minimize the burden of participa-

tion on health care staff. The RECOVERY trial, for example, has 

minimal eligibility criteria, simple and quick informed consent 

and randomization processes, and minimal data collection 

requirements, with follow-up information to be recorded at 

a single time point. Within 3 months of trial initiation, results 

were released showing the effectiveness of dexamethasone for 

reducing mortality in patients on mechanical ventilation or 

supplemental oxygen. The trial has also shown hydroxychloro-

quine, lopinavir and ritonavir, and azithromycin to be ineffec-

tive at reducing mortality from COVID-19.

 These examples show the potential of combining RCTs 

with RWD to produce robust evidence of effectiveness that 

can inform decision-making about treatments. They also indi-

cate that the use of RWD, while informative as a supplement to 

RCTs in certain contexts, is unlikely to replace traditional RCTs. 

Until a treatment is widely used in clinical practice, there are 

no RWD relating to its use. Consequently, data to support first-

time marketing approvals for new drugs have to come from 

RCTs. The context of a clinical trial also allows close monitor-

ing of patients, which is essential for their safety with drugs 

that have not yet received marketing authorization. As the 

examples given show, the situation is very different when a 

new drug is to be compared with a widely used drug or when 

well-established drugs are to be evaluated as potential treat-

ments for a new disease.

Author declaration and disclosures: The authors note no commercial 
associations that may pose a conflict of interest in relation to this article.

Author contacts: Thomas.Schindler@boehringer-ingelheim.com; Helen.
Bridge@astrazeneca.com

Resources

Baumfeld Andre E, Reynolds R, Caubel P, Azoulay L, Dreyer NA. Trial designs 

using real-world data: The changing landscape of the regulatory approval 

process. Pharmacoepidemiol Drug Saf. 2020;29(10):1201-1212.

Beaulieu-Jones BK, Finlayson SG, Yuan W, et al. Examining the use of 

real-world evidence in the regulatory process. Clin Pharmacol Ther. 

2020;107(4):843-852.

Collins R, Bowman L, Landray M, Peto R. The magic of randomization versus 

the myth of real-world evidence. N Engl J Med. 2020;382(7):674-678.

Epstein D, ProPublica. When evidence says no, but doctors say yes. The 

Atlantic website. Published February 22, 2017. Accessed January 8, 2021. 

https://www.theatlantic.com/health/archive/2017/02/when-evidence-says-

no-but-doctors-say-yes/517368/ 

Ford I, Norrie J. Pragmatic trials. N Engl J Med. 2016;375(5):454-463.

Franklin JM, Schneeweiss S. When and how can real world data analyses 

substitute for randomized controlled trials? Clin Pharmacol Ther. 

2017;102(6):924-933.

Najafzadeh M, Gagne JJ, Schneeweiss S. Synergies from integrating 

randomized controlled trials and real-world data analyses. Clin Pharmacol 

Ther. 2017;102(6):914-916.

Randomised evaluation of COVID-19 therapy (RECOVERY) Clinical Study 

Protocol, V12.1. RECOVERY Trial website. Published December 16, 2020. 

Accessed January 8, 2021. https://www.recoverytrial.net/files/recovery-

protocol-v12-1-2020-12-16.pdf 

Reynolds MW, Bourke A, Dreyer NA. Considerations when evaluating real-

world data quality in the context of fitness for purpose. Pharmacoepidemiol 

Drug Saf. 2020;29(10):1316-1318.

Saesen R, Huys I. COVID-19 clinical trials: See it big and keep it simple. BMJ 

Evid Based Med. Published online July 8, 2020. doi:10.1136/bmjebm-2020-

111447

Schindler TM, Bridge H. The evaluation of efficacy, or how do we know 

whether a treatment works? Part 1. AMWA J. 2020;35(2):82-86.

Scott IA, Attia J. Cautionary tales in the interpretation of observational 

studies of effects of clinical interventions. Intern Med J. 2017;47(2):144-157.

Selker HP, Eichler H-G, Stockbridge NL, et al. Efficacy and effectiveness 

too trials: Clinical trial designs to generate evidence on efficacy and on 

effectiveness in wide practice. Clin Pharmacol Ther. 2019;105(4):857-866.

Sherman RE, Anderson SA, Dal Pan GJ, et al. Real-world evidence – What is it 

and what can it tell us? New Engl J Med. 2016;375(23):2293-2297.

World Health Organization. “Solidarity” clinical trial for COVID-19 

treatments. Published 2020. Accessed January 8, 2021. https://www.who.int/

emergencies/diseases/novel-coronavirus-2019/global-research-on-novel-

coronavirus-2019-ncov/solidarity-clinical-trial-for-covid-19-treatments 

US Food and Drug Administration. Framework for FDA’s real-world evidence 

program. Published December 2018. Accessed January 8, 2021. https://www.

fda.gov/media/120060/download 

Wapner J. How prescription drugs get their prices, explained. Newsweek 

website. Published April 14, 2017. Accessed January 8, 2021. https://

www.newsweek.com/2017/04/14/prescription-drug-pricing-569444.

html#:~:text=Reimbursement%20is%20the%20amount%20the,as%20a%20

pharmacy%20benefit%20manager

STATISTICALLY SPEAKING

mailto:Thomas.Schindler@boehringer-ingelheim.com
mailto:Bridge@astrazeneca.com
https://www.theatlantic.com/health/archive/2017/02/when-evidence-says-no-but-doctors-say-yes/517368/
https://www.theatlantic.com/health/archive/2017/02/when-evidence-says-no-but-doctors-say-yes/517368/
https://www.theatlantic.com/health/archive/2017/02/when-evidence-says-no-but-doctors-say-yes/517368/
https://www.recoverytrial.net/files/recovery-protocol-v12-1-2020-12-16.pdf
https://www.recoverytrial.net/files/recovery-protocol-v12-1-2020-12-16.pdf
https://www.recoverytrial.net/files/recovery-protocol-v12-1-2020-12-16.pdf
https://www.who.int/emergencies/diseases/novel-coronavirus-2019/global-research-on-novel-coronavirus-2019-ncov/solidarity-clinical-trial-for-covid-19-treatments
https://www.fda.gov/media/120060/download
https://www.newsweek.com/2017/04/14/prescription-drug-pricing-569444
https://www.newsweek.com/2017/04/14/prescription-drug-pricing-569444.html#:~:text=Reimbursement%20is%20the%20amount%20the,as%20a%20pharmacy%20benefit%20manager
https://ebm.bmj.com/content/early/2020/07/08/bmjebm-2020-111447



