The Debate Highlights Some Basic Issues About Clinical Trial Research
Back in August, on the night before the Republican National Convention, the president of the United States and the director of the U.S. Food and Drug Administration (FDA) conducted a press conference in which they announced an emergency use authorization (EUA) for convalescent plasma to treat patients with COVID-19. Statements made at the press conference triggered a flood of criticism from medical and scientific experts. As it had when hydroxychloroquine was touted as a breakthrough treatment for COVID-19, politics once again entered the clinical trials arena.
The controversy over convalescent plasma allows us to highlight some of the basics about how a new drug is tested and the results interpreted.
Convalescent Plasma Explained
First, let’s explain what convalescent plasma is. Blood contains a host of different cells floating around in a liquid called plasma. These cells include the red blood cells that carry oxygen to the body’s organs, white blood cells that form the basis of the immune system, and platelets, necessary for blood to clot. If you spin blood in a centrifuge, the cells separate out from the plasma. What is left in plasma is a lot of proteins, some of which is composed of antibodies against a huge variety of foreign invaders, including viruses and bacteria. If you remove various clotting factors from the plasma, you get a clear, yellowish liquid called serum.
Convalescent plasma is plasma obtained from someone who has recovered from an illness that contains antibodies against the specific virus or bacteria that caused the illness. It has been used to treat infectious diseases since the late 19th century, including during the 1918 flue epidemic. Antibodies, also called immunoglobulins, are proteins that latch onto pathogens like viruses that get into our bodies and neutralizes them. They are produced by a specific type of white blood cell, called the B cell, and form the main engine of what is called the humoral immune system.
It is known that at least some people who contract COVID-19 develop neutralizing antibodies against the virus that causes it, SARS-CoV-2. For months, studies have been ongoing to test the possibility that convalescent plasma containing neutralizing antibodies against SARS-CoV-2 could be taken from people who recovered from COVID-19 and administered to patients acutely ill with the disease in order to reduce the severity of symptoms, prevent death, and speed recovery. One such study was published online in August. It was conducted by the Mayo Clinic and involved multiple centers across the country, enrolling at that point about 35,000 patients hospitalized with severe COVID-19 symptoms.
Misstating Trial Results
The data from that study formed the basis of the EUA for convalescent plasma and showed that among patients who received convalescent plasma (within three days of being diagnosed with COVID-19, 8.7% died by seven days compared to 11.9% who received the antibody transfusion four or more days after diagnosis. That difference is statistically significant. At 30 days, those mortality figures were 21.6% and 26.7% respectively for patients who received antibodies less than and more than four days after diagnosis, also a statistically significant difference. Importantly, the higher the amount (or titer) of antibodies against the coronavirus in the plasma received, the greater were the chances of survival.
As was widely reported, the director of the FDA, Stephen Hahn, misstated the size of the benefit from convalescent serum in this study when he said that it produced a 35% reduction in deaths, or, as he put it, for every 100 patients with COVID-19 transfused with convalescent serum, 35 would survive who would have died without receiving plasma.
We can use this clinical trial and the controversy surrounding statements about it to illustrate several important points about clinical trial research, starting with why Hahn’s statement was incorrect (note that he admitted publicly a few days later that he had made a mistake). It is based on the difference between relative and absolute risk. The relative reduction in risk for death by 7 days in the Mayo Clinic study is indeed 35% in the subgroup of patients included in the analysis. But that only tells you that a person with COVID-19 has a 35% chance of doing better if they receive convalescent serum. This does not tell us how many actually would survive if given the treatment.
Let’s say that an imaginary virus kills one person who gets an imaginary antiviral drug in a study that enrolls 10,000 people compared to two people who don’t get the drug. That’s obviously a relative reduction in mortality of 50%, which sounds like a very big number and one can imagine headlines blaring that the drug “reduces death by 50%.” In fact, of course, death from the virus in this imaginary case turns out to be rare and the drug only saved one more person out of 10,000. The headline should read “drug does not produce a meaningful survival benefit.” On the other hand, if in the same study 100 people die without drug compared to only 50 people given drug, that is still a 50% reduction, but this time the difference is 50 patients and the drug would appear to offer an important benefit.
So instead of looking at relative differences, we need to look at absolute differences. In the case of the Mayo Clinic study, the difference in death rates at 7 days is actually 3.2%, meaning that instead of 35 out of 100 potentially being saved, it is really only three patients out of 100. Saving three people isn’t bad but relying on this finding now leads us to two additional problems, one having to do with study design and the other with multiple testing. We’ll consider these in turn.
An RCT is Needed
In the Mayo Clinic study all of the patients received the antibody transfusion. Why did some people get it by three days and others at four or more days? Apparently, that was based on a clinical decision, which means that some set of unknown and unmeasured factors were in play influencing the decision. It could be simply how quickly the hospital was able to obtain convalescent plasma. But other factors could also be involved that might have influenced the decision. Did the doctors taking care of these patients perhaps decide that people with some characteristics were more likely to benefit from immediate transfusion and could those factors, rather than the transfusion itself, be responsible for the difference in survival outcome? In order for us to be certain that no such confounding factors influencing the decision about when to transfuse occurred, the decision about how many days after diagnosing someone would have to be made completely randomly and the two groups would have to be similar in most characteristics except for the number of days after diagnosis that they received the transfusion.
The best way to satisfy these criteria would be to randomize the patients to receive either a transfusion of serum or plasma containing antibodies or to a transfusion of a similar appearing substance that does not contain antibodies at the same number of days after diagnosis. This would be the most powerful test of whether transfusing convalescent plasma is beneficial because the only difference between the two groups would be whether antibodies are received.
Without a randomized, placebo-controlled clinical trial, we simply do not know whether the small difference in survival observed in the Mayo study is real. Such studies are now being conducted and the first one reported failed to find a difference in any outcome between those transfused with antibodies and those given a placebo. However, in that study participants received transfusions more than three days after diagnosis, so it is not directly comparable to the Mayo Clinic. More about the possible importance of the number of days that elapse between diagnosis and transfusion later, but first let’s tackle the question of multiple testing of the data.
Don’t Test Too Often
In a clinical trial with a randomized, placebo-controlled design (known as an RCT), researchers must declare before starting the study what their primary outcome measure is, for example the difference in 7-day survival. Absent that declaration, investigators could pick any outcome measure they wish after the study is completed. Why not try survival at three days and if that does produce a statistically significant difference between drug and placebo, let’s try four days and then 7 days and then 30 days. Or let’s look only at the subgroup of patients who needed to be on a mechanical respirator. Or perhaps just the group who developed severe respiratory distress.
This is what statisticians sometimes call “data massage”: keep dividing the study group into different subgroups until a statistically significant finding emerges. The problem is that for statistical reasons if you keep doing that you will ultimately get a statistically significant finding purely by chance. And then once again you have no idea whether the finding is real or fluke. It turns out that the subgroup of patients included in the Mayo analysis was comprised of only a small subgroup of the total approximately 35,000 enrolled in the study and there is no indication that this group and the three day cutoff were part of a primary outcome measure declared before the study started. So if those investigators kept testing the data until something emerged, what they have is an interesting—or more technically, exploratory—preliminary finding that may or may not be a real finding. How do you solve that problem? Conduct an RCT next.
Experts pointed out that at the point the president and FDA director held their press conference and the EUA was granted, we did not have (and still at the time of this writing last month do not have) enough data to reasonably conclude whether or not convalescent serum works to treat patients with COVID-19. The risk of prematurely issuing a EUA, especially when the drug it is applied to is hyped by high ranking federal officials, is that patients will shun RCT’s in which there is a 50% chance they will receive a placebo. That means that large numbers of COVID-19 patients who are very sick and hospitalized will be given a treatment about which we are not yet sure it works. It could be, for instance, that antibodies, which as we mentioned above are part of the humoral immune response based in B cells, are not the only or even main component of a successful anti-COVID-19 immune response. There is evidence that the other main branch of the immune system, called cellular immunity and driven by T cells, is important, but T cells are deliberately removed from convalescent plasma and serum. It is clearly a bad idea to keep throwing unproven treatments at patients, as was the case with the hydroxychloroquine experience, without ensuring that RCTs are done. We need to be resolute about completing clinical trials that will tell us what does and doesn’t work and what is safe.
None of this means, of course, that convalescent plasma doesn’t work. The findings that it works best if given early and with high titers of neutralizing antibodies make sense. It is possible that in early days of COVID infection the damage is done by the virus itself and neutralizing antibodies are one of the body’s defenses in this phase. Then, after a few days an overwhelming immune response may get out of control and become the problem, rather than the virus itself. At that point, neutralizing antibodies would no longer offer benefit and anti-inflammatory drugs, some of which are being tested now, could be the best avenue to symptom reduction. Hopefully, something like this will prove true in the RCTs now being conducted and a valuable intervention to prevent death from COVID-19 will be found.
Politicians need to take their lead from scientists when drug development is concerned and not the other way around. The intrusion of politics into decisions about what constitutes an effective and safe treatment increases the chance that useless and/or dangerous medications will be foisted on very ill people. One of our missions at Critica is to increase the use of scientific evidence in public policymaking. Right now, that is needed more than ever.
criticascience 