A journal has issued an expression of concern for an Australian study that supported mask mandates after researchers raised several potential problems with the design and methodology of the study.
The article, “The introduction of a mandatory mask policy was associated with significantly reduced COVID-19 cases in a major metropolitan city,” was published in the journal PLOS ONE in July 2021. It has been cited 12 times, according to Clarivate’s Web of Science.
In comments to Retraction Watch, the authors of the paper stood by their work, but a key critic said he still thought the work should have been retracted.
The research was led by scientists at the Burnet Institute in Melbourne, and examined the impact of a mask mandate on COVID-19 cases in the Australian state of Victoria.
Along with using case data from the Victorian Department of Health and Human Services, the team estimated mask use from photographs taken for the newspaper The Age before and after the government imposed a mask mandate. They also looked at an online survey of self-reported mask use.
The researchers concluded that the “study strongly supports the use of masks for controlling epidemics in the broader community.”.
In October 2021, Australian doctor and medical researcher Kyle Sheldrick, who has flagged issues with other COVID-19 research, raised concerns about the study on PubPeer, writing that “no weight should be given to the results.”
His comment went on to say, in part:
All the photos taken before the intervention were taken in a [sic] the late afternoon between approx 2pm and 4pm. All the photos after the intervention were taken between 830am and 1230pm. This is an unresolvable problem. Where there is a bias in time (with one group having *some *more or fewer earlier or later photos) corrections can be made and relative contribution can be assessed,. However, where there is complete separation between cohorts on another potentially causal factor (as there is in this case) drawing any causal inference is simply invalid and indefensible.
Sheldrick raised several other concerns, including an apparent discrepancy in the data collection period, that editorial photos taken for a newspaper are not unbiased and therefore not an appropriate source of data, and how small the self-reported masking data sample was. Sheldrick also expressed doubt about the claim in the paper that masking in Victoria reached 100% while other states apparently had 0% masking. And he noted that similar case trends existed in other parts of Australia that did not have mask mandates.
Other researchers had raised concerns about the study when it was published. An article in The Sydney Morning Herald quoted William Bevens, an epidemiology researcher at the University of Melbourne, saying: “I don’t think the evidence they put forward demonstrates their conclusion.”
The expression of concern, published on April 21, stated, in part:
After publication of this article, readers raised a number of concerns, including about the methodology, the limitations of the study design, and whether the conclusions are fully supported. The PLOS ONE Editors consulted with two members of the editorial board and a statistical advisor who advised that the study design is associated with a number of weaknesses that are discussed in the article, and which are unavoidable because of ethical issues that would be associated with a randomized controlled trial in the context of a pandemic, but that there were also additional weaknesses. The authors have provided further discussion of these issues upon editorial follow up.
The notice addressed many of the concerns Sheldrick raised in his comment, particularly the small sample size of the online survey data and other potential factors that could have accounted for the observed reduction in COVID-19 cases, such as the closure of public gathering spaces.
The editors concluded that although aspects of the study present real limitations, they do not completely invalidate the study’s findings:
According to the cumulative expert input received by the PLOS ONE Editors, the results of the published study contribute to the field of mask evaluation research, provided results are not overinterpreted and limitations are acknowledged. The PLOS ONE Editors felt that the conclusions, including those that imply causation, a direct correlation between COVID-19 cases and mask mandates, and the ability of masks for controlling epidemics, were not suitably tempered in light of the limitations of the study design. The PLOS ONE Editors issue this Expression of Concern to inform readers about the above considerations regarding study design and interpretation of the results.
The expression of concern also corrected several errors in the study, including the issue with the image data collection dates that Sheldrick noted. Whereas the study originally said that images were collected between July 10 and August 2, which it claimed was 14 days before and 14 days after the mask mandate was imposed, the correction acknowledges that the correct numbers were 13 days before and 11 days after the mask mandate. It also noted that in some instances, an absence of data for people outside Victoria was originally represented as 0% mask-wearing, which was not accurate.
Both in the expression of concern and in response to an email from Retraction Watch, the authors of the study stood by its validity. In a statement to Retraction Watch, which co-lead author Allan Saul said was written and/or reviewed by six of the authors, the group wrote, in part:
When the article was published it attracted a vigorous, highly political and at times, nasty, social media response from a vocal minority who were anti-COVID public health measures. This commentary did not address the main findings of the study. Some of this commentary spilled over to comments posted on the PLOS web site resulting in PLOS re-reviewing the article. The review found no errors in the primary analyses about the effects a mask mandate had on reducing COVID-19 caseloads in Melbourne during the second wave. The main point of our paper is that the timing of the introduction of a mask mandate correlated with a statistically significant reduction in cases. There were some minor errors picked up that were subsequently corrected (e.g., the dates for collection of photographs of people wearing masks were incorrect reported (although the actual dates used were correct) and an error in a graph with results of a survey of mask data where no responses had been plotted as “zero” instead of no line at all. Presumably it was this external pressure, that resulted in PLOS posting the “Expression of Concern”. We are disappointed that PLOS has posted the Expression of Concern despite the editors agreeing within the documented expression of concern itself, that many of the technical criticisms were unjustified.
Raina MacIntyre, another author of the study, said she sees the backlash to the study as “an example of the information warfare that has abounded about masks since SARS in 2003.” She said she has received criticism for her work on masking since the 2000s, adding that the topic became political again in 2014, during the Ebola outbreak in Western Africa. She said she sees the response to the study as another example of that politicization and said that “the complaints themselves about the paper came from people without any relevant expertise in infectious diseases, epidemiology or masks.”
In reply to an email from Retraction Watch, PLOS said, in part:
As described in the notice, the PLOS ONE Editors received input from three expert advisors; overall, the input indicated that the study design had weaknesses and that some potential limitations might not have been adequately discussed in the original article, but that the results nevertheless made a valid contribution to the field of mask evaluation research. The Expression of Concern provides transparency as to the issues raised post-publication, the authors’ responses, and the outcome of our editorial assessment, and informs readers about considerations regarding the study design and interpretation of the results. The notice also corrected some reporting errors and provided additional underlying data.
Sheldrick said that he was “dissatisfied with PLOS’s response” and believes the paper should have been retracted, writing again of concerns over using editorial photographs, which he told Retraction Watch is “close to the most non-random sample imaginable.”
Sheldrick also noted that though the researchers expected to find that cases in rural Victoria were not impacted by the mask mandate, which only applied to Melbourne, they actually found that the small number of cases they detected decreased, in some cases even more than in the city. He wrote:
Ultimately the authors identified a control group in the same state with no mask mandate. That group had a faster fall in cases. That means the intervention failed.
There is no within-group change that can overcome a failure to outperform control.
The expression of concern did not specifically address this point. In the study, the researchers said that these cases in rural Victoria, which accounted for about 5% of total cases in Victoria in the study period, made no overall difference to the results.
In their message to Retraction Watch, the researchers wrote that as part of the study, they checked to see if there was a comparable decrease in cases “in rural populations that were not subject to the mask mandate. None was found.”
Like Retraction Watch? You can make a tax-deductible contribution to support our work, follow us on Twitter, like us on Facebook, add us to your RSS reader, or subscribe to our daily digest. If you find a retraction that’s not in our database, you can let us know here. For comments or feedback, email us at team@retractionwatch.com.
Related
2023-05-05 04:01:32