Article

Weighing the Benefits and Risks of Proliferating Observational Treatment Assessments: Observational Cacophony, Randomized Harmony

Authors:
To read the full-text of this research, you can request a copy directly from the authors.

Abstract

Amid the severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) pandemic, substantial effort is being directed toward mining databases and publishing case series and reports that may provide insights into the epidemiology and clinical management of coronavirus disease 2019 (COVID-19). However, there is growing concern about whether attempts to infer causation about the benefits and risks of potential therapeutics from nonrandomized studies are providing insights that improve clinical knowledge and accelerate the search for needed answers, or whether these reports just add noise, confusion, and false confidence. Most of these studies include a caveat indicating that “randomized clinical trials are needed.” But disclaimers aside, does this approach help make the case for well-designed randomized clinical trials (RCTs) and accelerate their delivery?¹ Or do observational studies reduce the likelihood of a properly designed trial being performed, thereby delaying the discovery of reliable truth?

No full-text available

Request Full-text Paper PDF

To read the full-text of this research,
you can request a copy directly from the authors.

... This large and groundbreaking RCT is long overdue and has largely been warranted for decades. Given the potential biases inherent to nonrandomized studies [93][94][95], they may not be valid when the expected effects are likely to be actually null or only moderate, with relative risks less than two-fold [95]. This is most probably the case for the overall effects of light to moderate alcohol intake. ...
... A clear and definitive answer based on This large and groundbreaking RCT is long overdue and has largely been warranted for decades. Given the potential biases inherent to nonrandomized studies [93][94][95], they may not be valid when the expected effects are likely to be actually null or only moderate, with relative risks less than two-fold [95]. This is most probably the case for the overall effects of light to moderate alcohol intake. ...
Article
Full-text available
This review discusses the inconsistent recommendations on alcohol consumption and its association with chronic disease, highlighting the need for an evidence-based consensus. Alcohol is an addictive substance consumed worldwide, especially in European countries. Recommendations on alcohol consumption are controversial. On one hand, many nonrandomized studies defend that moderate consumption has a beneficial cardiovascular effect or a lower risk of all-cause mortality. On the other hand, alcohol is associated with an increased risk of cancer, neurological diseases, or injuries, among others. For years, efforts have been made to answer the question regarding the safe amount of alcohol intake, but controversies remain. Observational studies advocate moderate alcohol consumption following a Mediterranean pattern (red wine with meals avoiding binge drinking) as the best option for current drinkers. However, agencies such as the IARC recommend abstention from alcohol as it is a potent carcinogen. In this context, more randomized trial with larger sample size and hard clinical endpoints should be conducted to clarify the available evidence and provide clinicians with support for their clinical practice.
... would be overwhelming. 3 You are not alone in your inveracity relative to this Surgisphere fraud. Immediately after its publication the provaccine Washington Post featured the sputum from Professor William Schaffner, a professor of preventive medicine and infectious diseases at Vanderbilt Medical Center, who declared the "investigation" as, "rigorously done." ...
... (p.6) Therefore, in its recommendation that could have devastating civil, social and medical consequences, when enforced on the scale of the world population, the WHO violated the Golden Rule of medical ethics: "You don't recommend an intervention without policy-grade evidence for both harms and benefits".Regarding the said Golden Rule of medical ethics, allow me to quote the most authoritative voices of Califf, Hernandez and Landray, discussing medical-treatmentprotocol assessment during COVID-19, and writing in the prestigious Journal of the American Medical Association (JAMA) on 31 July 2020:[3] ...
... would be overwhelming. 3 You are not alone in your inveracity relative to this Surgisphere fraud. Immediately after its publication the provaccine Washington Post featured the sputum from Professor William Schaffner, a professor of preventive medicine and infectious diseases at Vanderbilt Medical Center, who declared the "investigation" as, "rigorously done." ...
... (p.6) Therefore, in its recommendation that could have devastating civil, social and medical consequences, when enforced on the scale of the world population, the WHO violated the Golden Rule of medical ethics: "You don't recommend an intervention without policy-grade evidence for both harms and benefits".Regarding the said Golden Rule of medical ethics, allow me to quote the most authoritative voices of Califf, Hernandez and Landray, discussing medical-treatmentprotocol assessment during COVID-19, and writing in the prestigious Journal of the American Medical Association (JAMA) on 31 July 2020:[3] ...
Technical Report
Full-text available
Alleged "COVID-19 Pandemic" mandated government enforced lockdowns of citizens, leading to massive but ignored K -12 suicide deaths of our children is connectable to Dr. Anthony Fauci.
... NIH-NeuroCOVID will not replace efforts to identify effective treatments via well-designed and conducted RCTs. There is no substitute for gold-standard RCT evidence (8). But NIH-NeuroCOVID offers an important adjunct, which may be more agile and able to quickly provide critical pieces of information. ...
Article
Full-text available
Patients suffering from COVID-19 experience a wide range of symptoms and sequelae, including increasingly recognized neurological problems. A concerted effort is necessary to identify and characterize these issues, whether newly appearing as a result of COVID-19 disease or exacerbations of underlying conditions. A national resource to collect information and/or biospecimens regarding neurological complications of COVID-19 offers an opportunity for broad representation, harmonization, and rapid learning, all while ensuring robust protection of confidential information through the use of global unique identifiers to protect patient privacy.
... Users of a DIY vaccine might also be unwilling or ineligible to participate in future clinical trials for traditional vaccines. The COVID-19 pandemic has already seen widespread off-protocol use of unproven interventions frustrate attempts to rigorously evaluate those or other interventions (12). At the same time, polls show that many are reluctant to take any COVID-19 vaccine. ...
... These in themselves provide lessons in retrospect, but perhaps the most positive influence that the pandemic could have on future ARDS management is to socialize and normalize enrolment in randomized trials in the intensive care environment. The creation and success of ongoing platform trials such as RECOVERY, REMAP-CAP, and ACTIV during the pandemic will hopefully spur clinicians, patients and their families, regulators, funders, and research ethics boards to demand enrolment into randomized clinical trials, [16] as is commonly the case in the cancer field. ...
... Claims made on the basis of supposedly impressive clinical outcomes of COVID-19 infected patients treated with Chloroquine and Hydroxychloroquine were viewed with skepticism, and the contradictory data that were later published about these drugs, including some that had to be retracted [85], confirmed that skepticism was indeed in order, and that scientific standards could not be lowered as a result of the pandemic [82]. Observational studies, even when conducted with care, can be so misleading that some authors have argued a moratorium should be placed on reporting them [20]. And indeed, to counteract exaggerated claims based on uncontrolled data, some wide-ranging national or international collaborations were quickly put in place for the conduct of large-scale trials [115]. ...
Article
Full-text available
Starting from historic reflections, the current SARS-CoV-2 induced COVID-19 pandemic is examined from various perspectives, in terms of what it implies for the implementation of non-pharmaceutical interventions, the modeling and monitoring of the epidemic, the development of early-warning systems, the study of mortality, prevalence estimation, diagnostic and serological testing, vaccine development, and ultimately clinical trials. Emphasis is placed on how the pandemic had led to unprecedented speed in methodological and clinical development, the pitfalls thereof, but also the opportunities that it engenders for national and international collaboration, and how it has simplified and sped up procedures. We also study the impact of the pandemic on clinical trials in other indications. We note that it has placed biostatistics, epidemiology, virology, infectiology, and vaccinology, and related fields in the spotlight in an unprecedented way, implying great opportunities, but also the need to communicate effectively, often amidst controversy.
... Prospective and retrospective real-world observational studies also have their own limitations (33). It is critically important to understand that while large numbers of patients in these studies may increase the precision of efficacy and safety estimates, they do not address and may actually reinforce systematic errors due to patient selection and other biases that are unavoidably present and result in confounding of results from both unknown factors and those that are known but perhaps not captured in the data repository. ...
... Some are related to the nature of the data; for example, any inferences on therapies through data aggregation across institutions must take into account the difficulties in accounting for important confounders related to practice variations, availability of medications, patient socioeconomic status, type of insurance coverage and other biases. Thus EHR-based evidence must not be interpreted as a substitute for well-designed randomized trials [17]. On the other hand, EHRs can provide an important resource to describe outcomes or interventions in real life, since strict eligibility criteria and design constraints do not always allow extrapolation of clinical trial findings to the population at large. ...
Article
Full-text available
Our goal is to summarize the collective experience of fifteen organizations in dealing with uncoordinated efforts that result in unnecessary delays in understanding, predicting, preparing for, containing, and mitigating the COVID-19 pandemic in the USA. Response efforts involve the collection and analysis of data corresponding to healthcare organizations, public health departments, socioeconomic indicators, as well as additional signals collected directly from individuals and communities. We focused on Electronic Health Records (EHRs) data, since EHRs can be leveraged and scaled to improve clinical care, research, and to inform public health decision-making. We outline the current challenges in the data ecosystem and the technology infrastructure that are relevant to COVID-19, as witnessed in our fifteen institutions. The infrastructure includes registries and clinical data networks to support population-level analyses. We propose a specific set of strategic next steps to increase interoperability, overall organization and efficiencies.
... Moreover, the public demand for an effective intervention can generate unwarranted visibility for sensational results from small, unblinded, or nonrandomized trials, as illustrated with hydroxychloroquine. But to be confident that an intervention is effective for COVID-19, as Califf et al 9 have suggested, requires the reliance on evidence from only the highest-quality randomized trials. ...
... In assessing the clinical effectiveness of an intervention, the clinician will want to consider the types of study designs used to generate evidence of effectiveness [28]. Traditionally, randomized controlled trials are considered the gold standard in evidence assessment [29], followed by observational studies such as cohort, cross-sectional, and case-control studies, and ending with descriptive studies such as surveillance, surveys, and case reports [30]. However, a nuanced that takes into account the size of the study and the rigor of the study design, recognizes that large, well-designed observational studies can yield among the highest-quality clinical evidence. ...
Chapter
Full-text available
This chapter will introduce the clinician to the quickly expanding field of musculoskeletal-focused digital apps (MDA), with an eye towards helping the clinician select and recommend MDAs for optimal patient care. MDAs are increasingly being used for physical therapy and rehabilitation, telehealth, pain management, behavioral health, and remote patient monitoring. The COVID-19 pandemic has vastly accelerated the adoption of telehealth and digital health apps by patients and clinicians, and the digital health field will only continue to expand as developers increasingly harness artificial intelligence (AI) and machine learning (ML) capabilities, coupled with precision medicine capabilities that integrate personal health data tracking and genomics insights. Here we begin with an overview of several types of MDA, before discussing the epidemiology of musculoskeletal conditions and injuries, clinical considerations in selecting a digital health solution, payor reimbursement for digital apps, and regulatory oversight of digital health apps.
... RCTs and OSs do not necessarily yield the same results when assessing the healthcare outcomes of treatments [20][21][22] although, on average, they do reach the same conclusions [23]. Indeed, various RCTs on the same topic can yield different results due to variable characteristics of their included populations and outcome assessments [24,25], or simply because the intervention under study is too subtle to be associated with a sufficiently large effect, for example, the use of CYP2C19 genotyping to optimize clopidogrel prescription [26]. ...
Article
Full-text available
For millennia, medicine was empirical and dominated by beliefs, dogmas and sorcery. Evidence-based medicine then emerged over the last two centuries as a major source of progress towards enlightened clinical practice. Yet, the design of randomized clinical trials and observational studies and, even more so, their methods of data collection and analyses, and the way they are presented in the media, have become sources of important deviancies and inaccuracies. This may be explained by the extreme pressure exerted by both academic and commercial concerns on clinical research and researchers. Although there is no justification to contest the importance of evidence-based medicine, there is certainly a need to learn how to detect the ways in which medical information can be distorted to convey messages that do not reflect reality. This article discusses how and why evidence-based medicine can be deceitful, and provides examples to illustrate this point of view. Key words (MeSH): Clinical study; Conflict of interest; Data accuracy; Endpoint determination; Epidemiology; Evidence-based medicine; Journalism; Medical; Methods; Reading; Research design
... 5,6 These concerns have been highlighted by the rapid execution and dissemination of a large volume of nonrandomized assessments of treatments for coronavirus disease 2019 (COVID- 19) with highly variable quality. 7,8 Calibration of RWE studies against a known treatment effect is one way to evaluate whether RWE can support causal conclusions in select circumstances if conducted using robust methodology. Several systematic reviews have compared the findings of published noninterventional studies with randomized, controlled trial (RCT) findings, [9][10][11][12][13][14][15] but they provided limited insights because they identified a wide variety of trials and compared them against published noninterventional studies that often differed substantially in terms of targeted populations, outcomes, or treatment strategies. ...
Article
Full-text available
Background: Regulators are evaluating the use of non-interventional real-world evidence (RWE) studies to assess the effectiveness of medical products. The RCT-DUPLICATE initiative uses a structured process to design RWE studies emulating randomized controlled trials (RCTs) and compare results. Here, we report findings of the first 10 trial emulations, evaluating cardiovascular outcomes of antidiabetic or antiplatelet medications. Methods: We selected 3 active-controlled and 7 placebo-controlled RCTs for replication. Using patient-level claims data from US commercial and Medicare payers, we implemented inclusion/exclusion criteria, selected primary endpoints, and comparator populations to emulate those of each corresponding RCT. Within the trial-mimicking populations, we conducted propensity score matching to control for >120 pre-exposure confounders. All study parameters were prospectively defined and protocols registered before hazard ratios (HRs) and 95% confidence intervals (CIs) were computed. Success criteria for the primary analysis were pre-specified for each replication. Results: Despite attempts to emulate RCT design as closely as possible, differences between the RCT and corresponding RWE study populations remained. The regulatory conclusions were equivalent in 6 of 10. The RWE emulations achieved a HR estimate that was within the 95% CI from the corresponding RCT in 8 of 10 studies. In 9 of 10, either the regulatory or estimate agreement success criteria were fulfilled. The largest differences in effect estimates were found for RCTs where second-generation sulfonylureas were used as a proxy for placebo regarding cardiovascular effects. Nine of 10 replications had a standardized difference between effect estimates of <2, which suggests differences within expected random variation. Conclusions: Agreement between RCT and RWE findings varies depending on which agreement metric is used. Interim findings indicate that selection of active comparator therapies with similar indications and use patterns enhances the validity of RWE. Even in the context of active comparators, concordance between RCT and RWE findings is not guaranteed, partially because trials are not emulated exactly. More trial emulations are needed to understand how often and in what contexts RWE findings match RCTs. Clinical Trial Registration: URL: https://clinicaltrials.gov Unique Identifiers: NCT03936049, NCT04215523, NCT04215536, NCT03936010, NCT03936036, NCT03936062, NCT03936023, NCT03648424, NCT04237935, NCT04237922
... In the months that followed the FDA's EAP issuance, CCP use increased beyond expectations, leading to criticism that this modality was being deployed clinically without sufficiently rigorous efficacy trials. 9 In this perspective, we review how CCP emerged as a leading COVID-19 therapy and consider the issues encountered in establishing its efficacy, with particular emphasis on the unique complexities involved in conducting randomized clinical trials with a heterogeneous product during a pandemic with limited information on the conditions for ideal use. ...
Article
Antibody-based therapy for infectious diseases predates modern antibiotics and, in the absence of other therapeutic options, was deployed early in the SARS-CoV-2 pandemic through COVID-19 convalescent plasma (CCP) administration. Although most studies have demonstrated signals of efficacy for CCP, definitive assessment has proved difficult under pandemic conditions, with rapid changes in disease incidence and the knowledge base complicating the design and implementation of randomized controlled trials. Nevertheless, evidence from a variety of studies demonstrates that CCP is as safe as ordinary plasma and strongly suggests that it can reduce mortality if given early and with sufficient antibody content.
... It has also contributed to a "cacophony" of poorquality research (Califf et al. 2020). In an interview with the New York Times Magazine, the charismatic French physician Didier Raoult explained that it is "unnecessary, in addition to being unethical, to run randomized controlled trials, or R.C.T.s, of treatments for deadly infectious diseases" saying, "We're not going to tell someone, 'Listen, today's not your lucky day, you're getting the placebo, you're going to be dying'" (Sayare 2020). ...
Article
Statements of the core ethical and professional responsibilities of medical professionals are incomplete in ways that threaten fundamental goals of medicine. First, in the absence of explicit guidance for responding to cases in which there is significant uncertainty or disagreement about the relative therapeutic, prophylactic or diagnostic merits of available interventions they perpetuate self-defeating practices. Second, without addressing the role of advertising in shaping patient and community preferences they risk creating moral loopholes that bypass and undermine professional duties of fidelity, honesty and transparency. In both cases, these flaws are exacerbated by an individualism that ignores the critical role of health systems in managing and reducing uncertainty and conflict over best practices, and in communicating with and shaping the expectations of the public. These points are illustrated with examples from the response to COVID-19 and suggestions for reform are proposed.
... Regarding the said Golden Rule of medical ethics, allow me to quote the most authoritative voices of Califf, Hernandez and Landray, discussing medical-treatmentprotocol assessment during COVID-19, and writing in the prestigious Journal of the American Medical Association (JAMA) on 31 July 2020: [3] […] However, there is growing concern about whether attempts to infer causation about the benefits and risks of potential therapeutics from nonrandomized studies are providing insights that improve clinical knowledge and accelerate the search for needed answers, or whether these reports just add noise, confusion, and false confidence. Most of these studies include a caveat indicating that "randomized clinical trials are needed." ...
Technical Report
A vile new mantra is on the lips of every public health official and politician in the global campaign to force universal masking on the general public: “there is a growing body of evidence”. This propagandistic phrase is a vector designed to achieve five main goals: - Give the false impression that a balance of evidence now proves that masks reduce the transmission of COVID-19 - Falsely assimilate commentary made in scientific venues with “evidence” - Hide the fact that a decade’s worth of policy-grade evidence proves the opposite: that masks are ineffective with viral respiratory diseases - Hide the fact that there is now direct observational proof that cloth masks do not prevent exhalation of clouds of suspended aerosol particles; above, below and through the masks - Deter attention away from the considerable known harms and risks due to face masks, applied to entire populations The said harms and risks include that a cloth mask becomes a culture medium for a large variety of bacterial pathogens, and a collector of viral pathogens; given the hot and humid environment and the constant source, where home fabrics are hydrophilic whereas medical masks are hydrophobic. In short, I argue: op-eds are not “evidence”, irrelevance does not help, and more bias does not remove bias. Their mantra of “a growing body of evidence” is a self-serving contrivance that impedes good science and threatens public safety. I prove that there is no policy-grade evidence to support forced masking on the general population, and that all the latest-decade’s policy-grade evidence points to the opposite: NOT recommending forced masking of the general population. Therefore, the politicians and health authorities are acting without legitimacy and recklessly.
... For example, several studies of COVID-19 treatments assessed inclusion/exclusion criteria or treatment assignment during study follow-up, a design that clearly would not be possible in a randomized trial. 23 These design issues may account for many of the conflicting findings of such studies. ...
Article
Full-text available
The randomized controlled trial (RCT) is the gold standard for evaluating the causal effects of medications. Limitations of RCTs have led to increasing interst in using real‐world evidence (RWE) to augment RCT evidence and inform decision‐making on medications. Although RWE can be either randomized or nonrandomized, nonrandomized RWE can capitalize on the recent proliferation of large healthcare databases and can often answer questions that cannot be answered in randomized studies due to resource constraints. However, the results of nonrandomized studies are much more likely to be impacted by confounding bias, and the existence of unmeasured confounders can never be completely ruled out. Furthermore, non‐randomized studies require more complex design considerations which can sometimes result in design‐related biases.We discuss questions that can help investigators or evidence consumers evaluate the potential impact of confounding or other biases on their findings: Does the design emulate a hypothetical randomized trial design? Is the comparator or control condition appropriate? Does the primary analysis adjust for measured confounders? Do sensitivity analyses quantify the potential impact of residual confounding? Are methods open to inspection and (if possible) replication? Designing a high‐quality nonrandomized study of medications remains challenging and requires broad expertise across a range of disciplines, including relevant clinical areas, epidemiology, and biostatistics. The questions posed in this paper provide a guiding framework for assessing the credibility of nonranomzied RWE and could be applied across many clinical questions.
... This on-going need, however, should be tempered with scientific and ethical oversight that is at least as rigorous as normal times with a focus on well-designed trials and not rapid dissemination of low-quality data. The potential harms of producing multiple iterations of lower-quality studies have been identified, including wasting of resources, lapses in the ethical standard of scientific reporting, delaying the conduct of higher-level evidence trials, diluting the quality of available evidence, and endangering the ethical responsibility to patients who enroll in trials with the expectation of assisting in medical and scientific advancement [6,12,13]. Researchers should endeavour to maintain high-quality research methods by increasing collaboration across multiple centres, helping to overcome limitations that may exist from single-centre efforts [3,14]. International teams working in concert and not in competition on welldesigned studies would greatly improve the capacity to detect clinically meaningful effects to inform the international health system's efforts against COVID-19. ...
Article
Full-text available
Background The COVID-19 pandemic has yielded an unprecedented quantity of new publications, contributing to an overwhelming quantity of information and leading to the rapid dissemination of less stringently validated information. Yet, a formal analysis of how the medical literature has changed during the pandemic is lacking. In this analysis, we aimed to quantify how scientific publications changed at the outset of the COVID-19 pandemic. Methods We performed a cross-sectional bibliometric study of published studies in four high-impact medical journals to identify differences in the characteristics of COVID-19 related publications compared to non-pandemic studies. Original investigations related to SARS-CoV-2 and COVID-19 published in March and April 2020 were identified and compared to non-COVID-19 research publications over the same two-month period in 2019 and 2020. Extracted data included publication characteristics, study characteristics, author characteristics, and impact metrics. Our primary measure was principal component analysis (PCA) of publication characteristics and impact metrics across groups. Results We identified 402 publications that met inclusion criteria: 76 were related to COVID-19; 154 and 172 were non-COVID publications over the same period in 2020 and 2019, respectively. PCA utilizing the collected bibliometric data revealed segregation of the COVID-19 literature subset from both groups of non-COVID literature (2019 and 2020). COVID-19 publications were more likely to describe prospective observational (31.6%) or case series (41.8%) studies without industry funding as compared with non-COVID articles, which were represented primarily by randomized controlled trials (32.5% and 36.6% in the non-COVID literature from 2020 and 2019, respectively). Conclusions In this cross-sectional study of publications in four general medical journals, COVID-related articles were significantly different from non-COVID articles based on article characteristics and impact metrics. COVID-related studies were generally shorter articles reporting observational studies with less literature cited and fewer study sites, suggestive of more limited scientific support. They nevertheless had much higher dissemination.
... 10 Some observational comparisons have yielded encouraging and highly publicized results, 37,38 accompanied by concern regarding residual confounding by indication-and sometimes followed by discordant results from subsequent randomized comparisons. 9 These three dimensions of RWD may be linked for practical reasons. For example, retrospective nonrandomized comparisons are often practically limited to use of existing healthcare records regarding treatments delivered under real-world conditions. ...
Article
Full-text available
Concerns regarding both the limited generalizability and the slow pace of traditional randomized trials have led to calls for greater use of real‐world evidence (RWE) in the evaluation of new treatments or products. The RWE label has been used to refer to a variety of departures from the methods of traditional randomized controlled trials. Recognizing this complexity and potential confusion, the National Academies of Science, Engineering, and Medicine convened a series of workshops to clarify and address questions regarding the use of RWE to evaluate new medical treatments. Those workshops identified three specific dimensions in which RWE studies might differ from traditional clinical trials: use of real‐world data (data extracted from health system records or data captured by mobile devices), delivery of real‐world treatment (open‐label treatments delivered in community settings by community practitioners), and real‐world treatment assignment (including non‐randomized comparisons and variations on random assignment such as before‐after or stepped‐wedge designs). For any RWE study, decisions regarding each of these dimensions depends on the specific research question, characteristics of the potential study settings, and characteristics of the settings where study results would be applied.
... While many COVID-19 trials have seen slow or failed enrollment(11, 12), trials allowing more real-world treatment delivery and data collection have yielded rapid and actionable results(10). Some observational comparisons have yielded encouraging and highly publicized results (37,38), accompanied by concern regarding residual confounding by indication -and sometimes followed by discordant results from subsequent randomized comparisons (9). ...
Article
Full-text available
Concerns regarding both the limited generalizability and the slow pace of traditional randomized trials have led to calls for greater use of real-world evidence (RWE) in the evaluation of new treatments or products. RWE studies often rely on real-world data (RWD), including data extracted from healthcare records or data captured by mobile phones or other consumer devices. Global assessments of RWD sources are not helpful in assessing whether any specific RWD element is fit for any specific purpose. Instead, evidence generators and evidence consumers should clearly identify the specific health state or clinical phenomenon of interest and then consider each step between that clinical phenomenon and its representation in a research database. We propose specific questions regarding potential error or bias affecting each of those steps: Would a person experiencing this clinical phenomenon present for care in this setting or interact with this recording device? Would this clinical phenomenon be accurately recognized or assessed? How might the recording environment or tools affect accurate and consistent recording of this clinical phenomenon? Can data elements from different sources be harmonized, both technically (same format) and semantically (same meaning)? Can the original data elements be consistently reduced to a useful clinical phenotype? Addressing these questions requires a range of clinical, organizational, and technical expertise. Transparency regarding each step in the creation of RWD is essential if evidence consumers are to rely on RWE studies.
... The contrasting mortality benefit between matched-control studies and larger multicenter RCTs raises the possibility that locally sourced plasma administered shortly after donation may increase convalescent plasma efficacy against local variants in real time as they emerge (17). Compared to RCTs, matched-control studies are inherently predisposed to greater bias risk (77). However, several studies observed convalescent plasma efficacy using rigorous propensity matching for key variables such as age, sex, disease severity and oxygen requirements, and comorbidities (52,64,69,70). ...
Article
Full-text available
Convalescent plasma has been used worldwide to treat patients hospitalized with coronavirus disease 2019 (COVID-19) and prevent disease progression. Despite global usage, uncertainty remains regarding plasma efficacy, as randomized controlled trials (RCTs) have provided divergent evidence regarding the survival benefit of convalescent plasma. Here, we argue that during a global health emergency, the mosaic of evidence originating from multiple levels of the epistemic hierarchy should inform contemporary policy and healthcare decisions. Indeed, worldwide matched-control studies have generally found convalescent plasma to improve COVID-19 patient survival, and RCTs have demonstrated a survival benefit when transfused early in the disease course but limited or no benefit later in the disease course when patients required greater supportive therapies. RCTs have also revealed that convalescent plasma transfusion contributes to improved symptomatology and viral clearance. To further investigate the effect of convalescent plasma on patient mortality, we performed a meta-analytical approach to pool daily survival data from all controlled studies that reported Kaplan–Meier survival plots. Qualitative inspection of all available Kaplan–Meier survival data and an aggregate Kaplan–Meier survival plot revealed a directionally consistent pattern among studies arising from multiple levels of the epistemic hierarchy, whereby convalescent plasma transfusion was generally associated with greater patient survival. Given that convalescent plasma has a similar safety profile as standard plasma, convalescent plasma should be implemented within weeks of the onset of future infectious disease outbreaks.
... Observational analyses began to emerge and to be acted upon, despite them being unable to address unmeasured or unknown confounders. [15][16][17][18] Hence, RCTs are necessary to resolve such uncertainty. 19,20 Moderate treatment effects (ie proportional reductions in mortality of 20%-25%) were plausible, and still hugely relevant. ...
Article
When COVID-19 hit the UK in early 2020, there were no known treatments for a condition that results in the death of around one in four patients hospitalised with this disease. Around the world, possible treatments were administered to huge numbers of patients, without any reliable assessments of safety and efficacy. The rapid generation of high-quality evidence was vital. RECOVERY is a streamlined, pragmatic, randomised controlled trial, which was set up in response to this challenge. As of April 2021, over 39,000 patients have been enrolled from 178 hospital sites in the UK. Within 100 days of its initiation, RECOVERY demonstrated that dexamethasone improves survival for patients with severe disease; a result that was rapidly implemented in the UK and internationally saving hundreds of thousands of lives. Importantly, it also showed that other widely used treatments (such as hydroxychloroquine and azithromycin) have no meaningful benefit for hospitalised patients. This was only possible through randomisation of large numbers of patients and the adoption of streamlined and pragmatic procedures focused on quality, together with widespread collaboration focused on a single goal. RECOVERY illustrates how clinical trials and healthcare can be integrated, even in a pandemic. This approach provides new opportunities to generate the evidence needed for high-quality healthcare not only for a pandemic but for the many other conditions that place a burden on patients and the healthcare system.
... To this point, a review of characteristics and expected strength of evidence of COVID-19 studies registered on ClinicalTrials.gov revealed few large multicenter trials had the potential to generate highquality evidence and a large proportion of studies with an expected low level of evidence [104]. Caution was raised about the rapid dissemination of low-quality evidence due to potential harmful infl uence on public opinion, government actions, and clinical practice. ...
... 5,6 These concerns have been highlighted by the rapid execution and dissemination of a large volume of nonrandomized assessments of treatments for coronavirus disease 2019 (COVID- 19) with highly variable quality. 7,8 Calibration of RWE studies against a known treatment effect is one way to evaluate whether RWE can support causal conclusions in select circumstances if conducted using robust methodology. Several systematic reviews have compared the findings of published noninterventional studies with randomized, controlled trial (RCT) findings, [9][10][11][12][13][14][15] but they provided limited insights because they identified a wide variety of trials and compared them against published noninterventional studies that often differed substantially in terms of targeted populations, outcomes, or treatment strategies. ...
Article
Full-text available
In Response:We are grateful for the interest in our research and the thoughtful letters to the editor regarding our recently published article.[1] We very much agree with the main points of Olson et al that emulation success was the most critical factor determining whether the randomized controlled trial (RCT) results could be rep-licated by real-world evidence (RWE) and that, among those RCTs that could be emulated well, RWE replicated RCT results in general. Observational RWE studies can typically emulate pragmatic RCTs better than highly controlled RCTs.[2] Olson et al argue that, in light of these findings, more effort should be made “ensuring that comparators and end points are adequately emulated.” As Olson et al and other RWE researchers know, emulation is not the purpose of RWE. Although we do these comparisons to demonstrate when and how RWE studies come to causal conclusions, the strength of RWE is its ability to answer questions that cannot be answered by RCTs, as in cases where randomiza-tion would be difficult or unethical, or questions that will not be answered by RCTs, because of high costs and a lack of incentives. The conclusion that there should be greater focus on judging the design and measurement similarity of RCT and RWE studies when interpreting differences between RWE and RCT findings is important and aligns well with past work that has shown that better emulation of RCT design can lead to better concordance between RWE and RCT findings.
Article
Full-text available
The emergence and global spread of the severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) has resulted in an urgent need for evidence on medical interventions and outcomes of the resulting disease, COVID-19. While many randomized controlled trials (RCTs) evaluating treatments and vaccines for COVID-19 are already in-progress, the number of clinical questions of interest greatly outpaces the available resources to conduct RCTs. Therefore, there is growing interest in whether nonrandomized real-world evidence (RWE) can be used to supplement RCT evidence and aid in clinical decision-making, but concerns about nonrandomized RWE have been highlighted by a proliferation of RWE studies on medications and COVID-19 outcomes with widely varying conclusions. The objective of this paper is to review some clinical questions of interest, potential data types, challenges, and merits of RWE in COVID-19, resulting in recommendations for nonrandomized RWE designs and analyses based on established RWE principles.
Article
Large randomized controlled trials (RCTs) remain the gold standard for evaluating treatment efficacy. However, observational studies including non-randomized cohort studies as well as small RCTs have gained increasing attention especially during the SARS-CoV-2 pandemic where critical evaluation of limited therapeutic options are sought to improve patient care while awaiting results for subsequent randomized controlled trials. As the authors have previously discussed, RCTs and observational studies are complementary approaches which often appear synergistic with one another. While not all real-world studies are the same, the results of observational studies are notoriously subject to both known and unknown confounding factors. The utilization of COVID-9 Convalescent Plasma is a timely illustration of evaluating the efficacy and safety of a COVID-19 therapy given the dangerous and often lethal effects of the virus and the limited approved therapeutic options for the disease. While awaiting the results of large RCTS of convalescent plasma, serval observational cohorts and small RCTs have attempted to assess the efficacy and safety of this approach with very mixed results. Among the likely reasons for this failure to provide a definitive answer concerning the value of convalescent plasma are the many limitations inherent to addressing treatment efficacy in non-randomized studies. While such studies are often able to capture information on large numbers of individuals rapidly, it is important to understand that although larger numbers may enhance the precision of estimates provided, larger numbers, in and of themselves, do not increase the accuracy of estimates due to patient selection and other biases. At the same time, both observational studies and small RCTS are constantly at risk for publication bias due to investigator, reviewer and editorial bias toward positive studies. In this commentary we discuss the advantages and limitations of these methodologic approaches when addressing urgently needed evidence on the effectiveness and safety of therapies in a crisis such as the COVID-19 pandemic.
Article
Clopidogrel, prasugrel, and ticagrelor are oral platelet P2Y12 receptor inhibitors that decrease the risk of platelet-mediated coronary artery thrombosis. Clinical guidelines have recommended ticagrelor or prasugrel over clopidogrel in combination with aspirin as dual antiplatelet therapy (DAPT) for 1 year after acute coronary syndrome (ACS), whether or not percutaneous coronary intervention (PCI) is performed.¹ The ticagrelor recommendation was based on the Study of Platelet Inhibition and Patient Outcomes (PLATO) trial, which enrolled 18 624 patients with ACS and randomized them to receive DAPT with either clopidogrel and aspirin or ticagrelor and aspirin.² The 1-year primary composite efficacy end point of death from vascular causes, myocardial infarction (MI), or stroke favored ticagrelor and aspirn vs clopidogrel and aspirin (9.8% vs 11.7%; hazard ratio, 0.84 [95% CI, 0.77-0.92]; P < .001), but major bleeding not related to coronary artery bypass graft surgery was increased with ticagrelor and aspirin vs clopidogrel and aspirin (4.5% vs 3.8%; P = .03).
Preprint
Full-text available
OBJECTIVE We explored longitudinal trends in sociodemographic characteristics, reported symptoms, laboratory findings, pharmacological and non-pharmacological treatment, comorbidities, and 30-day in-hospital mortality among hospitalized patients with coronavirus disease 2019 (COVID-19). METHODS This retrospective cohort study included 43,267 patients diagnosed with COVID-19 in the Veterans Health Administration between 03/01/20 and 08/31/20 and followed until 09/30/20. We focused our analysis on patients that were subsequently hospitalized, and categorized them into groups based on the month of hospitalization. We summarized our findings through descriptive statistics. We used a nonparametric rank-sum test for trend to examine any differences in the distribution of our study variables across the six months. RESULTS During our study period, 8,240 patients were hospitalized, and 1,081 (13.1%) died within 30 days of admission. Hospitalizations increased over time, but the proportion of patients that died consistently declined from March (N=221/890, 24.8%) to August (N=111/1,396, 8.0%). Patients hospitalized in March compared to August were younger on average, mostly black, and symptomatic. They also had a higher frequency of baseline comorbidities, including hypertension and diabetes, and were more likely to present with abnormal laboratory findings including low lymphocyte counts and elevated creatinine. Lastly, receipt of mechanical ventilation and Hydroxychloroquine declined from March to August, while treatment with Dexamethasone and Remdesivir increased. CONCLUSION We found evidence of declining COVID-19 severity and fatality over time within a national health care system.
Preprint
Full-text available
Background: Hydroxychloroquine and chloroquine have been proposed as treatments for coronavirus disease 2019 (COVID-19) on the basis of in vitro activity, uncontrolled data, and small randomized studies. Methods: The Randomised Evaluation of COVID-19 therapy (RECOVERY) trial is a randomized, controlled, open-label, platform trial comparing a range of possible treatments with usual care in patients hospitalized with COVID-19. We report the preliminary results for the comparison of hydroxychloroquine vs. usual care alone. The primary outcome was 28-day mortality. Results: 1561 patients randomly allocated to receive hydroxychloroquine were compared with 3155 patients concurrently allocated to usual care. Overall, 418 (26.8%) patients allocated hydroxychloroquine and 788 (25.0%) patients allocated usual care died within 28 days (rate ratio 1.09; 95% confidence interval [CI] 0.96 to 1.23; P=0.18). Consistent results were seen in all pre-specified subgroups of patients. Patients allocated to hydroxychloroquine were less likely to be discharged from hospital alive within 28 days (60.3% vs. 62.8%; rate ratio 0.92; 95% CI 0.85-0.99) and those not on invasive mechanical ventilation at baseline were more likely to reach the composite endpoint of invasive mechanical ventilation or death (29.8% vs. 26.5%; risk ratio 1.12; 95% CI 1.01-1.25). There was no excess of new major cardiac arrhythmia. Conclusions: In patients hospitalized with COVID-19, hydroxychloroquine was not associated with reductions in 28-day mortality but was associated with an increased length of hospital stay and increased risk of progressing to invasive mechanical ventilation or death.
Preprint
Full-text available
Background: Coronavirus disease 2019 (COVID-19) is associated with diffuse lung damage. Corticosteroids may modulate immune-mediated lung injury and reducing progression to respiratory failure and death. Methods: The Randomised Evaluation of COVID-19 therapy (RECOVERY) trial is a randomized, controlled, open-label, adaptive, platform trial comparing a range of possible treatments with usual care in patients hospitalized with COVID-19. We report the preliminary results for the comparison of dexamethasone 6 mg given once daily for up to ten days vs. usual care alone. The primary outcome was 28-day mortality. Results: 2104 patients randomly allocated to receive dexamethasone were compared with 4321 patients concurrently allocated to usual care. Overall, 454 (21.6%) patients allocated dexamethasone and 1065 (24.6%) patients allocated usual care died within 28 days (age-adjusted rate ratio [RR] 0.83; 95% confidence interval [CI] 0.74 to 0.92; P<0.001). The proportional and absolute mortality rate reductions varied significantly depending on level of respiratory support at randomization (test for trend p<0.001): Dexamethasone reduced deaths by one-third in patients receiving invasive mechanical ventilation (29.0% vs. 40.7%, RR 0.65 [95% CI 0.51 to 0.82]; p<0.001), by one-fifth in patients receiving oxygen without invasive mechanical ventilation (21.5% vs. 25.0%, RR 0.80 [95% CI 0.70 to 0.92]; p=0.002), but did not reduce mortality in patients not receiving respiratory support at randomization (17.0% vs. 13.2%, RR 1.22 [95% CI 0.93 to 1.61]; p=0.14). Conclusions: In patients hospitalized with COVID-19, dexamethasone reduced 28-day mortality among those receiving invasive mechanical ventilation or oxygen at randomization, but not among patients not receiving respiratory support.
Article
Objective To provide an update on key safety metrics after transfusion of convalescent plasma in hospitalized COVID-19 patients, having previously demonstrated safety in 5,000 hospitalized patients. Patients and Methods From April 3 to June 2, 2020, the US FDA Expanded Access Program for COVID-19 convalescent plasma transfused a convenience sample of 20,000 hospitalized patients with COVID-19 convalescent plasma. Results The incidence of all serious adverse events was low; these included transfusion reactions (n=89; <1%), thromboembolic or thrombotic events (n=87; <1%), and cardiac events (n=680, ∼3%). Notably, the vast majority of the thromboembolic or thrombotic events (n=55) and cardiac events (n=562) were judged to be unrelated to the plasma transfusion per se. The seven-day mortality rate was 8.6% (8.2%, 9.0%), and was higher among more critically-ill patients relative to less ill counterparts, including patients admitted to the intensive care unit vs. not admitted (10.5% vs. 6.0%), mechanically ventilated vs. not ventilated (12.1% vs. 6.2%), and with septic shock or multiple organ dysfunction/failure vs. those without dysfunction/failure (14.0% vs. 7.6%). Conclusion These updated data provide robust evidence that transfusion of convalescent plasma is safe in hospitalized patients with COVID-19, and support the notion that earlier administration of plasma within the clinical course of COVID-19 is more likely to reduce mortality.
Article
After publication of our Lancet Article,1 several concerns were raised with respect to the veracity of the data and analyses conducted by Surgisphere Corporation and its founder and our co-author, Sapan Desai, in our publication. We launched an independent third-party peer review of Surgisphere with the consent of Sapan Desai to evaluate the origination of the database elements, to confirm the completeness of the database, and to replicate the analyses presented in the paper. Our independent peer reviewers informed us that Surgisphere would not transfer the full dataset, client contracts, and the full ISO audit report to their servers for analysis as such transfer would violate client agreements and confidentiality requirements. As such, our reviewers were not able to conduct an independent and private peer review and therefore notified us of their withdrawal from the peer-review process. We always aspire to perform our research in accordance with the highest ethical and professional guidelines. We can never forget the responsibility we have as researchers to scrupulously ensure that we rely on data sources that adhere to our high standards. Based on this development, we can no longer vouch for the veracity of the primary data sources. Due to this unfortunate development, the authors request that the paper be retracted. We all entered this collaboration to contribute in good faith and at a time of great need during the COVID-19 pandemic. We deeply apologise to you, the editors, and the journal readership for any embarrassment or inconvenience that this may have caused. MRM reports personal fees from Abbott, Medtronic, Janssen, Roivant, Triple Gene, Mesoblast, Baim Institute for Clinical Research, Portola, Bayer, NupulseCV, FineHeart, and Leviticus. FR has been paid for time spent as a committee member for clinical trials, advisory boards, other forms of consulting, and lectures or presentations; these payments were made directly to the University of Zurich and no personal payments were received in relation to these trials or other activities since 2018. Before 2018 FR reports grants and personal fees from SJM/Abbott, grants and personal fees from Servier, personal fees from Zoll, personal fees from Astra Zeneca, personal fees from Sanofi, grants and personal fees from Novartis, personal fees from Amgen, personal fees from BMS, personal fees from Pfizer, personal fees from Fresenius, personal fees from Vifor, personal fees from Roche, grants and personal fees from Bayer, personal fees from Cardiorentis, personal fees from Boehringer Ingelheim, other from Heartware, and grants from Mars. ANP declares no competing interests.
Article
Crises are no excuse for lowering scientific standards
Article
Nonrandomized observational analyses have been promoted as alternatives to randomized clinical trials. However, randomization ensures balance between groups, whereas nonrandomized studies are often biased by between-group differences. Efforts to reduce the cost and complexity of clinical trials are preferable to relying on observational studies.
Article
For hundreds of years, the development of new medical treatments relied on “real-world” experience. Discoveries such as citrus fruit curing scurvy described in the 1700s or insulin as a treatment for diabetes in the 1920s long preceded the advent of the modern randomized clinical trial. What these diseases had in common was a reliable method of diagnosis, a predictable clinical course, and a large and obvious effect of the treatment.