Additional reporting on how the seroprevalence study in California may have been impacted by methodological challenges. New seroprevalence studies are likely to be unveiled in the coming weeks that will give us truer insight into the prevalence of #covid19 https://t.co/KBLjKS3E8B— Scott Gottlieb, MD (@ScottGottliebMD) April 24, 2020
Woops! Was I wrong to single out Stanford?
New York City’s Department of Health sent an alert to providers advising them against using antibody tests to diagnose prior covid infection or to assess immunity because of low reliability and high rates of false positive results. This is a good regulatory action by the city. pic.twitter.com/2o68NowbtO— Scott Gottlieb, MD (@ScottGottliebMD) April 24, 2020
Many science studies can’t be replicated, especially in psychology.
ReplyDeleteGood news is everyday we are learning more.
Perhaps NYC purchased their test kits from China's version of "As Seen on TV"?
ReplyDeleteVery possibly. It's still where we're getting most of our meds.
DeleteRe the title of your post, Mark, what exactly did the Stanford prof who was the author of the study do that was specifically 'unethical'? Seems like a harsh criticism...Just wondering...
ReplyDeleteDoes what he did seem like scientific sampling methodology to you? Have your read any of the critiques of the author by other scientists and statisticians? I linked to one some days ago. Sample:
Delete“I think the authors owe us all an apology… not just to us, but to Stanford,” wrote Andrew Gelman, a professor of statistics and political science and director of the Applied Statistics Center at Columbia University.“
Harsh?
"Mapping the Mortality Maze," in the American Spectator could be a worthwhile read. I can't say they're right. But his defense of the Stanford study warrants scrutiny.
Deletehttps://spectator.org/mapping-the-mortality-maze-how-deadly-is-covid-19/
The authors--a grab bag of non-infectious specialists--make some valid points--all of which I've been making on my own from the beginning. But they fails to address the scientific critiques of the Stanford "study". Such as the low specificity of the (Chinese?) test used, and the decidedly non-random sampling. I addressed that--you haven't.
DeleteI don't have time to write a detailed critique of the article--much of which I happen to agree with. However, my point all along has been that there is unlikely to be wide and rapid spread of the virus across America, and certainly not at any level that would lead to herd immunity. There are arguments to be made for widespread reopening, and I've made them--including that the virus is not as easily transmissible as feared, although that too was known fairly early. Those arguments for reopening can be made without touting "studies" that use suspect methods.
BTW, I've cited Scott Gottlieb in the past for the proposition that the virus would spread to about 5% of the general population. Many of those "studies" are coming in below that, and instead advance the meaningless claim of "50-85" times higher than previously thought. That 5% level (NY being the exception) isn't remotely close to affording any sort of general immunity. Again, arguments for reopening need to reexamine the data and rethink their assumptions. It can be done, but it requires leaving the conceptual box they're stuck in: "herd immunity" as a well near infallible process.
I'm trying to present honest assessments here, not agenda driven ones that rely on largely libertarian assumptions.
Look Titan, here's just one example of the flaws in the article--and I'm trying to be very polite.
DeleteThe authors present a nice graph to show a "fever spike" in early to mid March, then state:
"This could be evidence into how much more prevalent COVID-19 is than early testing showed."
Oh yeah? HOW MUCH more prevalent? They don't say--data free innuendo is about what that amounts to. None of the "studies" go beyond the nationwide rate that Gottliebe posits: ~ 5%.
The article's defense of the Stanford "study" warrants dismissal.
Here's an article which comprehensively summarizes the criticism of the Stanford study.
Deletehttps://www.mercurynews.com/2020/04/20/feud-over-stanford-coronavirus-study-the-authors-owe-us-all-an-apology/
Since the study was published before finalization, we'll have to see where it comes out after adjusting for the criticisms received by the authors.
I guess I'm a little more old school, Mark. I would reserve the judgment of 'unethical' until all the evidence is in. But that's just me. We can agree to differ on this one, too.
Reread Gottlieb's tweet. Former FDA commissioner. Diplomatically saying: methodologicially challenged, doesn't provide true insights.
DeleteHarsh? Or honest?
For a scientist to knowingly publish a "methodologically challenged" study is unethical. Call me old school, if you wish.
DeleteMark, thank you for your refreshingly well-reasoned analysis and criticism. That is why I come here.
DeleteOh, you're very welcome.
DeleteMark, respectfully, you might want to slow down and take a breath.
ReplyDeleteScott Gottlieb looks to have been caught up with Buzzfeed clickbait, and an allegation of unethical conduct against the Santa Clara study is over the top, IMO. The Buzzfeed article is silly.
The study was based on 3,300 volunteers, who were made aware of the testing on short notice by various means of communication. One posting on a middle school listserv with some forgettable wording--out of the dozens of avenues used to recruit volunteers--does not invalidate a study whose results were reported, as adjusted based on county-wide social-economic, geographic, and gender/age demography.
Whatever overrepresentation occurred--whether by one middle school, or other demographic groups--was adjusted in the reported results.
There may be other reasons to critique the study. The Mercury News article posted above by Cassander didn't convince me. It consists of opinions entirely devoid of substance--great for clickbait media, insubstantial in providing an argument, or otherwise recommending a contrary result.
"They owe us an apology" is not convincing. "How NOT to do statistics" without attribution is less persuasive, while at the same time, extremely inflammatory.
There's a narrative trap people are getting caught in, defending or disagreeing based on their cognitive bias, unable to bear any examples that contradict their settled position. This is not helpful, nor healthy.
Critics--of all sorts--are "making the perfect the enemy of the good." Good data is being dismissed until we have perfect data--which doesn't exist.
Meanwhile, the goal posts keep moving until perfection arrives. Flatten the curve. Decrease the growth in infections. Decline in infection. No new infections. Herd immunity. Limit testing to the seriously ill. Limit testing to those presenting symptoms. Test those with elevated body temperature. Do serology to determine prevalence, and possibly herd immunity. Don't do serology because it doesn't diagnose exiting infections. Don't do serology testing, better tests are on the way.
Every question answered triggers the nest question. Meanwhile the healthy sit at home, devoid of income and livelihood, destroying small businesses all across the country--while PhDs argue over how many angels fit on the top of a pin.
Cheers.
But you don't address the very low specificity of the test and what that means for the low % results of this "study--which I discussed in another post. Also the hyped and tendentious interpretation of the findings: 50-85% higher--than an extremely low level. The important point is that 50-85x higher still doesn't get you anywhere remotely close to 'herd' immunity. And that--the claim that the virus was 'everywhere' and we' already achieved 'herd' immunity--was the very point that the "study" was supposed to make, as the authors pre-hyped in the WSJ article that preceded any actual data being taken.
DeleteThey very obviously went into their "study" intending to find a certain result, and when despite their best efforts that result wasn't forthcoming they offered tendentious interpretations--50-85x higher--to make it sound like their pre-study hype had been correct.
I think you're being unfair to Gottlieb in suggesting that a guy with his background fell for leftwing "clickbait" instead of reaching an independent judgment.
Here's another and longer Buzzfeed article that was linked in the original, and which goes into the very real problems with the Stanford "study," usefully contrasting them to the way the LA study was done. It gets into very specific criticisms that I'm quite confident Gottlieb was aware of:
https://www.buzzfeednews.com/article/stephaniemlee/coronavirus-antibody-test-santa-clara-los-angeles-stanford
I'll repeat: Unethical. Unethical because, as Gelman wrote: "I think they need to apologize because these were avoidable screw-ups. They’re the kind of screw-ups that happen if you want to leap out with an exciting finding and you don’t look too carefully at what you might have done wrong.”
My overall point remains, that it's foolish to be trying to lower the overall death rate with these kinds of studies. The major focus should be how virulent the virus is and who is at most risk. All studies so far show that Gottlieb is correct in predicting a 5% rate of infection in the general population--a number that I'm sure he didn't just make up. Anyone claiming something approaching 'herd' immunity rates of >50% is most likely driven more by ideology and desires rather than evidence. The gap is too great for any other explanation.
Unethical.
Also, here's Gelman's blog on the subject, which you apparently didn't even glance at:
Deletehttps://statmodeling.stat.columbia.edu/2020/04/19/fatal-flaws-in-stanford-study-of-coronavirus-prevalence/
Well, Mark...its your blog. And its a darn good one...
ReplyDeleteAs it happens, I largely agree with your conclusions regarding Spygate or whatever you want to call it. I think its the largest scandal in our political history and threatens to bring the Republic down. I have enjoyed reading your thoughtful posts and sharing my reactions with you and your many commenters. I have learned a lot.
As for the pandemic, I think you are overly wedded to your own narrative: that this pandemic is substantially more dangerous than seasonal flu. It may turn out to be, but I have an open mind and am listening to arguments that, for example, the death rate may not turn out to be substantially greater than seasonal flu or, if greater, not so much greater as has been feared.
While I can understand some of the arguments made by those who criticise the Santa Clara study, unlike you I am not prepared to accuse all of the authors: Eran Bendavid, Bianca Mulaney, Neeraj Sood, Soleil Shah, Emilia Ling, Rebecca Bromley-Dulfano, Cara Lai, Zoe Weissberg, Rodrigo Saavedra, James Tedrow, Dona Tversky, Andrew Bogan, Thomas Kupiec, Daniel Eichner, Ribhav Gupta, John Ioannidis, and Jay Bhattacharya of unethical conduct.
I understand that the article was published as a pre-print and that the authors have publicly indicated that they are considering the criticisms. Furthermore, there are so many variables in play here and so much we still don't know about the behavior of this virus that I am inclined at this point to step back and see what a rigorously debated scientific inquiry ultimately concludes.
Neither am I very comfortable personally continuing this discussion in such an overheated atmosphere. And perhaps I'm overly invested in my point of view. So I think I'll take the advice the estimable Forbes has offered above...I think I'll slow down and take a rest and take (another) time out as far as my participation here is concerned.
I do however wish you the very best and hope that your prodigious efforts here are appropriately rewarded.
"I think you are overly wedded to your own narrative"
DeleteBut why misrepresent my 'narrative'? My narrative has always been that the death rate will be moderated by the lower degree of virulence (transmissibility}. As it is, we know that the death toll so far is already as bad as a very bad flu season--but in a much shorter time span. The fact that that death toll is so geographically concentrated surely speaks to the question of transmissibility (despite the overall lack of transparency), and that impacts the question of advisable public health measures. That approach, I have maintained, is preferrable to wedding oneself to 'rates' that cannot at this time be established with 'scientific rigor.'
"the authors have publicly indicated that they are considering the criticisms."
That however is part of the issue. The fact that the authors are "considering" the criticisms is an indication that they know the criticisms are coming from weighty quarters and not just from Leftist opinion writers. What the critics are saying, with specificity, is that the "study" was constructed in defiance of well known methodological principles. The criticisms were easily avoidable by following those principles in the first place, if the authors had been willing to take a bit more time.
As it is, Gelman expresses himself as perfectly willing to view the results as they are as plausible--but simply objects to the lack of scientific rigor. For my part, the results claimed (which also support Gottlieb's estimates) simply support my own 'narrative'--that, however lethal the disease may be, the lower degree of transmissibility will mitigate deaths, in part by mitigating the overall spread. All the testing for prevalence so far, despite the methodological flaws and tendentious attempts to overbroad applicability, indicate that the virus has not spread across the land like wildfire. That is my own view, based on early reports, i.e., it's not my own 'narrative.' It also happens to be at variance with the presuppositions of the modelers, whom I have criticized from my own standpoint.
"see what a rigorously debated scientific inquiry ultimately concludes."
And yet both you and Forbes decline to address, in any way, the serious quesions raised by the articles I have linked or by my 'narrative.' I actually do have a background in philosophy of science going back to my undergrad days, so I'm not inclined to give what I view as undue deference to "scientists." That's part of why I've felt free to disagree with--and question the motives of--both the modelers and their critics when I believe their are grounds for criticism. That has always been part of the scientific process despite the efforts of the scientific establishment to suppress criticism and mystify their own status.
I agree. Mark has an absolutely terrific blog the content of which never fails to be thoughtful, illuminating, and necessary.
DeleteOn the Standford study, I lean toward Forbes and Cassander's side of the street. Apart from that, an issue that will be resolved in time, one of many, I cannot for the life of me uncouple our response to the virus from the virus itself. I have no words for what we have done to the economy.
It's the economy that's keeping me awake at night, not the virus. I'm a codger. I'm all set (assuming they don't outlaw ammunition). It's the graduating class of 2020 and beyond I'm worried about. It's a good bet we have shattered their future.
My other concern is simply logical. How is it that the ruinously inept men and women who have been running our institutions into the ground, at the same time they've been wrong about every issue under the sun, for decades, suddenly managed, under extreme pressure, to get it right?
It passeth understanding.
I'm doing a new, long, blog that I hope will clear up some issues. However, I'm taking a bit of a break till later in the morning. Please be patient.
Delete"How is it that the ruinously inept men and women who have been running our institutions into the ground, at the same time they've been wrong about every issue under the sun, for decades, suddenly managed, under extreme pressure, to get it right?"
DeleteI hope you're not suggesting that's my 'narrative'?
Absolutely not.
Delete