Another PRO-MASK Study Debunked

You may recall some years ago the New England Journal of Medicine went bonkers crazy trying to push masks on kids based on a single study that was put out there. They put out multiple editorials and tried to justify the massive mandates that were foisted on kids as young as two years old.

A recent study published in the Annals of Internal Medicine by Dr. Ambarish Chandra, Dr. Tracy Beth Høeg, and colleagues critically examines the effectiveness of school mask mandates in reducing COVID-19 transmission among students and staff. The research addresses the challenges of using observational data and difference-in-differences (DiD) analysis to estimate the impact of public health interventions, particularly in rapidly changing situations like a pandemic.

The difference-in-differences (DiD) methodology is a statistical technique commonly used in social sciences to estimate the causal effect of a policy intervention or treatment when randomization is not possible. It compares the changes in outcomes over time between a group that is exposed to a treatment (intervention group) and a group that is not (control group).

Background

Mask mandates in schools have been a contentious public health measure during the COVID-19 pandemic. While some observational studies have suggested benefits (very few), others have found mixed results. Notably, randomized controlled trials (RCTs), considered the gold standard in clinical research, have not provided strong evidence supporting the effectiveness of mask mandates in community settings or among children. In fact, the studies show - they don’t do much.

Objective of the New Study

The primary objective was to assess the validity of previous studies that used DiD methodology to claim causal reductions in COVID-19 cases due to school mask mandates. The authors aimed to:

• Highlight the shortcomings of DiD analysis when applied to ecological data with potential confounders.

• Reanalyze previous data using alternative methods and control groups to test the robustness of earlier conclusions.

Methodology

1. Data Source and Setting:

   • The study focused on school districts in the greater Boston area and Massachusetts during the 2021–2022 academic year.

   • Data included COVID-19 case rates among students and staff in districts that did and did not lift mask mandates.

2. Analysis Approach:

   • The authors recreated key figures from a prior study and extended the analysis to the entire academic year.

   • They defined three additional control groups based on geographic proximity to Boston:

     • Districts within 50 km of Boston.

     • Districts within 80 km of Boston.

     • The remainder of Massachusetts.

   • Statistical analyses included calculating case rates, absolute differences, incidence rate ratios, and ratios of incidence rate ratios (RRRs).

   • The study examined potential confounders such as prior community infection rates, vaccination rates, testing practices, and demographic differences (race and income).

Key Findings

1. Variability in Results Based on Control Groups:

   • Depending on the choice of control group and whether students or staff were analyzed, lifting mask mandates was associated with a wide range of outcomes:

     • An increase of 5.64 cases per 1,000 persons.

     • A decrease of 2.74 cases per 1,000 persons.

   • This variability suggests that conclusions about the effectiveness of mask mandates are highly sensitive to the selected control groups and analytical methods.

2. Challenges with DiD Methodology:

   • The DiD analysis requires assumptions such as parallel trends and no confounding factors, which were not met in this context.

   • Significant differences in racial composition and income between intervention and control groups likely introduced confounding variables.

   • Temporal variations, such as prior immunity from infections and changes in testing practices, further compromised the validity of causal inferences.

3. Influence of Prior Community Cases:

   • Communities that maintained mask mandates (Boston and Chelsea) had higher cumulative community case rates before the policy change.

   • This suggests that higher prior infection rates could have conferred greater immunity, affecting subsequent case rates independently of mask policies.

4. Testing Practices as a Confounder:

   • Consent to school SARS-CoV-2 testing varied significantly by race, with lower participation among Black students.

   • Such discrepancies could have led to underreporting of cases in certain districts, skewing the data.

5. Inconsistent Trends Over Time:

   • Extending the analysis to the full academic year showed inconsistent differences in case rates across districts, violating the parallel trends assumption necessary for DiD analysis.

Discussion

• Limitations of Observational Studies:

  • The study highlights the pitfalls of relying on observational data and DiD methodology in the presence of confounders and rapidly changing circumstances.

  • Ecological data, which aggregates data at a group level rather than individual level, can obscure underlying variations and introduce biases.

• Need for High-Quality Evidence:

  • The authors emphasize the importance of conducting randomized controlled trials (RCTs) to evaluate the effectiveness of public health interventions like mask mandates.

  • Previous RCTs on mask effectiveness in community settings have not demonstrated significant benefits, underscoring the need for robust evidence before implementing widespread policies.

• Alternative Research Designs:

  • The study suggests that designs such as regression discontinuity or crossover studies may provide more reliable insights by reducing confounding variables.

  • Such studies have not found substantial benefits of mask mandates in school settings.

Conclusions

• The effectiveness of school mask mandates in reducing COVID-19 transmission remains uncertain due to methodological challenges and potential confounders in observational studies.

• Policymakers should exercise caution when interpreting findings from studies using DiD analysis of ecological data, especially when underlying assumptions are not met.

• There is a critical need for well-designed RCTs to provide definitive answers on the effectiveness of mask mandates and inform public health decisions.

Implications for Public Health Policy

• Decisions regarding mask mandates in schools should be guided by high-quality evidence that accounts for potential confounders and biases.

• Transparency in communication about the benefits and limitations of public health interventions is essential to maintain public trust.

• Future policy implementations should consider the feasibility of conducting RCTs or other robust study designs to evaluate interventions before widespread adoption.

References

1. Chandra, A., Høeg, T. B., Ladhani, S., Prasad, V., & Duriseti, R. (2023). School Mask Mandates and COVID-19: The Challenge of Using Difference-in-Differences Analysis of Observational Data to Estimate the Effectiveness of a Public Health Intervention. Annals of Internal Medicine. https://doi.org/10.7326/M23-2907

2. Cowger, T. L., et al. (2022). Impact of Lifting School Masking Requirements on Incidence of COVID-19 Among Students and Staff. New England Journal of Medicine, 387, 1935-1946. https://doi.org/10.1056/NEJMoa2211029

Note to Readers

This summary aims to provide an objective overview of the recent study examining the challenges of evaluating school mask mandates using observational data. The findings underscore the importance of rigorous research methodologies in public health to ensure policies are based on reliable evidence.