Health Data Analysis and Forecasting Initiative

Health Data Analysis and Forecasting Initiative is sponsored by Centers for Disease Control and Prevention (CDC). Funds projects that analyze health data to predict trends and inform public health decisions.

Prepared by the Illinois Public Health Institute | August 27, 2021 # Forecasting and Modeling # FINAL REPORT WITH TECHNICAL NOTES Contents Background And Report Overview 3 High-Level Cross-Cutting Themes 7 Proposed Function: Predict 8 Modeling And Forecasting 8 Data Collection And Accessibility 12 Real-Time And Granular Data 14 Covid-19 And Other Diseases 14 Proposed Function : Connect 16 Proposed Function: Inform 18 Responses To Additional Questions 20 Appendix A: List Of Participants 24 Appendix B: Methodology For Listening Sessions And Qualitative/Thematic Analysis 26 Appendix C: Listening Session Technical Notes 28 Appendix D: Article “Infrastructure As Social Sensor” 76 “Modernizing Health Data Analytics and Forecasting.

Forecasting and Modeling Listening Sessions – Final Report with Technical Notes. ” August 27, 2021. M O D E R N I Z I N G H E A LT H D ATA A N A LY T I C S A N D F O R E C A S T I N G 3 Real-time, actionable evidence serves as the cornerstone for improving baseline population health as well as preventing and responding to epidemics, outbreaks and pandemics.

In January 2021, noting the importance of such evidence, the United States (U.S.) Government announced plans for establishing an interagency National Center for Epidemic Forecasting and Outbreak Analytics (hereafter “Center”) to modernize global early warning and trigger systems to prevent, detect and respond to biological threats 1 As an independent nonprofit that mobilizes philanthropic and private-sector resources to support the Centers for Disease Control and Prevention (CDC) and public health, CDC Foundation held two three- hour, multi-sector listening sessions to seek input on needs and gaps in public health forecasting.

The listening sessions were held on July 22 and 28, 2021. These listening sessions engaged stakeholders across the private, public health, healthcare and academic sectors. Participants discussed ways to catalyze transformative advances in forecasting, mathematical modeling and other analytical capabilities in the United States and globally.

Discussions ranged across topics to assure broad feedback from the diverse subject matter experts in attendance, focusing notably on the nexus of forecasting, data access and data utility.

Topics included 1) data collection, prioritization, integration and analyses; 2) short-term forecasts that may be unconditional predictions and longer-term forecasts that will be conditional predictions; 3) infectious and noninfectious diseases and syndromes; and 4) engagement and coordination across partners, sectors, stakeholders and jurisdictions.

This report summarizes high-level themes from the listening sessions and incorporates key takeaways from each theme. High-level themes included: Pandemic-Preparedness pdf 1. Modeling and forecasting 3 Data collection and accessibility 4.

Real-time and granular data 5.

COVID-19 and other diseases “You cannot predict the future, but you —PETER DRUCKER M O D E R N I Z I N G H E A LT H D ATA A N A LY T I C S A N D F O R E C A S T I N G 4 With that thematic approach, the report aims to provide multi-sector insights into important consid - erations for public health as leaders and practitioners work toward real-time, actionable evidence that can better meet ongoing and emerging needs of disease prevention and health promotion.

While the listening sessions yielded rich discussion resulting in the themes, this report represents the available input based on the current phase in the process. The nature of the data modernization work is dynamic. As additional opportunities for input are made available, new information will be added and recommendations may be refined.

M O D E R N I Z I N G H E A LT H D ATA A N A LY T I C S A N D F O R E C A S T I N G 5 At the start of both listening sessions, Judith Monroe, MD, President and CEO at CDC Foundation and Daniel B. Jernigan, MD, MPH, Deputy Director for Public Health Science and Surveillance at the CDC welcomed the group and provided background and level-setting remarks about the importance of advancing forecasting in the public health.

The below summarizes those opening remarks. JUDITH MONROE, MD, PRESIDENT AND CEO, CDC FOUNDATION CDC Foundation is committed to identifying critical improvements needed to reinforce and support vital public health infrastructure for data modernization Timely and actionable health-related data is the cornerstone for an effective response for the numerous conditions and diseases that negatively impact people’s health and the economy.

Forecasting, mathematical modeling and other analytical tools are valuable in interpreting and translating data into action to respond effectively to existing and emerging public health threats.

In addition, the COVID-19 pandemic has demonstrated that this need for timely and actionable data is imperative for a broad range of decision makers, such as elected officials, institutional and organizational leaders, grassroots community organizers and others across a broad range of sectors, such as schools, workplaces, industry sectors and other institutions across communities.

CDC Foundation sits at the intersection of the government, philanthropic and private sectors; working together these combined resources, insights and flexibility allow for tackling the toughest health challenges including modernizing health data. # Opening Remarks M O D E R N I Z I N G H E A LT H D ATA A N A LY T I C S A N D F O R E C A S T I N G 6 DANIEL B.

JERNIGAN, MD, MPH, DEPUTY DIRECTOR FOR PUBLIC HEALTH SCIENCE The Data Modernization Initiative falls under the purview of Public Health Science and Surveillance at the Centers for Disease Control and Prevention (CDC). Years of preparatory work is now being applied, with recent appropriations and funding intended to support public health data surveillance, analytics infrastructure and other modernization initiatives at CDC.

To date, there have been several challenges in addressing data modernization as part of the COVID-19 response, including: # • Limitations across mission critical public health responses such as detecting, tracking, intervening and preventing. # • Vastly different inputs leading to different estimates used to predict disease transmission and impact, forecast supplies and design countermeasures.

There is not a coordinated effort and there are not enough benchmarks in place. # • Lack of coordination in the translation of models and forecasts to inform decision makers and to direct resources effectively. This has resulted in varying recommendations and efforts among jurisdictions.

# • Limited and not easily accessible modeling and forecasting expertise.

The appropriations and funding directed at data modernization allows for establishing, expanding and maintaining efforts to modernize the United States’ disease warning system to forecast and track hotspots for COVID–19 and its variants and other emerging biological threats, improving academic and workforce support for analytics and informatics infrastructure and improving data collection systems.

To that end, the current Administration is looking to create an interagency National Center for Epidemic Forecasting and Outbreak Analytics. The goals for the Center include: # • Modeling and forecasting public health concerns and sharing information in real time to trigger governmental, private sector and public actions to respond within the United States and abroad.

# • Advancing the use of forecast and outbreak analytics in public health decision-making with the goal of supporting more efficient and effective outbreak responses. # • Bringing together next-generation public health data scientists, expert disease modelers, public health emergency responders and high-quality communications experts to meet the needs of decision makers and translate information for key audiences.

# • The proposed functions of the Center include: Predict: Modeling and forecasting to determine the foundational data sources needed; support research and innovation in outbreak analytics and science for real-time action; and establish appropriate forecasting horizons. This includes surveillance and case data.

# • Data inputs include public health departments, laboratories, health care, weather, animal outbreaks, social mobility and consumer data. Connect: Broad capability for data sharing and integration; maximizing interoperability with data standards and utilizing open-source software and application programming interface (API) capabilities with existing and new data streams from the public health ecosystem and beyond.

Inform: Translating and communicating forecasts; connect with key decision makers across sectors including government, businesses and nonprofits, along with individuals with strong intergovernmental affairs and communication capacity for action.

Take what can be forecasted and, through scenario modeling, translate that into action by decision makers and to inform the public M O D E R N I Z I N G H E A LT H D ATA A N A LY T I C S A N D F O R E C A S T I N G 7 Th emes that emerged across the listening sessions include: # • Modeling and forecasting # • Data collection and accessibility # • Real-time and granular data # • COVID-19 and other diseases These themes (figure 1) are categorized within the three proposed functions of the Center described above.

The remainder of the report details the feedback received in the listening sessions on each of the themes. Multiple participant comments related to the same idea are rolled up into one statement to support each theme. Key takeaways were derived primarily from recommendations that are more specific and actionable and, in some cases, reflect participant consensus.

For more information on the methodology of this report, see Appendix B .

> K E Y T H E M E S F R O M L I S T E N I N G S E S S I O N S Real-time and Granular Data COVID-19 and Other Diseases # Cutting Themes M O D E R N I Z I N G H E A LT H D ATA A N A LY T I C S A N D F O R E C A S T I N G 8 Modeling and forecasting are analytical capabilities that can help guide policy and planning in responding to infectious diseases and other biological threats.

Modeling and forecasting have the potential to improve epidemic and pandemic management by preventing deaths and severe illness and reducing the public health and economic impacts of these threats.

Across the two listening sessions there were almost 200 comments related to this topic that aligned with the following subcategories: learnings from previous and existing models, precise and accurate interpretations, an integrated systems approach, individual vs. population-level forecasting and intentional matching between stakeholder needs and approaches to modeling and forecasting.

Specific recommendations from each subcategory are highlighted below: Learnings from previous and existing models # • Research, evaluate and learn from past and existing models, for example, weather forecasting models and applications, flu modeling and forecasting, MIDAS (Mixed Data Sampling) and RAPIDD (Research and Policy for Infectious Disease Dynamics).

Also evaluate and learn from where things have gone wrong in generating forecasts. For example, flu forecasting could have brought different teams together from forecast hubs and other efforts.

# • Evaluate approaches to the COVID-19 pandemic to assess the success of the various models # • Evaluate successful modeling and forecasting techniques implemented in other countries, such # • Implement an impact assessment to gauge if particular models were/are effective. Prioritize precise and accurate interpretations # • Communicate forecasting assumptions to stakeholders.

Stochasticity that exists in the real world cannot always be emulated in a model. Similarly, it is important to present both the positives and limitations of the forecast. These can be taken into consideration for future forecasts.

# • Create modeling systems that are more objective and less susceptible to bias. # • Recommend nowcasting over forecasting.

Real-time and Granular Data COVID-19 and Other Diseases Communications M O D E R N I Z I N G H E A LT H D ATA A N A LY T I C S A N D F O R E C A S T I N G 9 # • Share whether forecast is counterfactual or factual in interpreting and communicating out # • Interpret and use modeling and forecasting correctly and in a uniform manner. # • Create indicators/measures to assess accuracy of interpretations.

Integrated systems approach # • Develop an integrated systems approach and build out capacity and systems for simultaneous analytical practices, modeling and forecasting techniques. # • Coordinate among parallel modeling centers. There is currently extensive duplication in modeling and forecasting.

# • Realign incentives with infectious disease modeling efforts in community and in academia. # • Look at what predictions and forecasts look like next to one another. # • Evaluate interagency links that will allow for more effective delivery of information that is wanted and/or important to local constituents.

The 2020 study Ensemble Forecasts of Coronavirus DISEASE 2019 (COVID-19) in the U.S. supports the above points by identifying that real-time ensemble forecasts, i.e. combining multiple probabilistic models and assessing forecast skills at different prediction horizons, can provide robust short-term predictions of relevant indicators to public health decision makers.

2 Individual vs. population-level forecasting # • Ensure research capacity includes modeling at several levels, from ecological/population-level models to individual-based models. Ensure modeling incorporates variability in human actions and interactions that influence emergent social phenomena. # • Recommend individual-based models and network-based models that show how individual interactions influence the whole.

# • Consider that local modeling requires local communication (local people translate data for # • Recommend starting with forecasting goals that are meaningful and add value at the local level. Intentional matching between needs and approaches # • Look at the key questions that need to be addressed and the best data that can support that modeling, prior to collecting data.

Approaches to forecasting must be directly related to the # • Clarify to all involved the prioritization, modeling targets and goals, and questions to be answered. Clarify what types of data are needed ahead of time. # • Have structured relationships that focus on how questions are addressed together.

> 2Ray, Evan L, et al. “Ensemble Forecasts of Coronavirus DISEASE 2019 (COVID-19) in the U.S.” MedRxiv , 2020, doi:10. 1101/20 > 20.

08. 19. 20177493.

M O D E R N I Z I N G H E A LT H D ATA A N A LY T I C S A N D F O R E C A S T I N G 1 0 In planning for the National Center for Epidemic Forecasting and Outbreak Analytics evaluate and emulate, as appropriate, existing forecasting models such as weather and flu modeling. The plan should include an integrated systems approach such that there is coordination across modeling centers and incentives are aligned.

Ahead of collecting data, recommend intentional matching between the question at hand and approaches to and sources of collecting data. All relevant stakeholders should have clarity on modeling targets and goals. Consider modeling and forecasting at both individual and population-based levels.

Assumptions and limitations must be communicated to those utilizing the forecasts. “There is increasing consensus that infrastructure is crucial for connectivity, and that access to infrastructure is asymmetric. Therefore, tracking the natural buildup of physical infrastructure (and gaps in digital infrastructure) may serve as a crucial means for more rapid pandemic response.

” 3 Another top theme that emerged across the two listening sessions was the emphasis on infrastructure, with over 150 comments on this theme. Infrastructure in this context comprises resources, academic and workforce support and funding needed for the operations of the National Center for Epidemic Forecasting and Outbreak Analytics.

Specific feedback on each of these areas included: # • Tap into existing resources and platforms until we can increase knowledge and understanding of modeling and forecasting. # • Prioritize what modernized data systems and infrastructures are needed for success, such as technology, a data lake, a system that links laboratory and electronic medical record (EMR) data, a data stockpile and/or a data hub.

# • Recognize that smaller health departments may not have the capacity to work on urgent projects. # • Enhance the data pipelines in clinical infrastructure that enables faster reporting. # • Invest in global capacity development in modeling.

Academic and Workforce Support # • Grow the workforce across universities and other stakeholders. There is currently a workforce gap.

Need capacity and expertise for different types of modeling, especially # • Incorporate modeling and forecasting as topics into curricula for public health professionals # • Have statisticians, epidemiologists, data analysts etc. observe and learn forecasting # • Implement more research-based training programs and workgroups to support the workforce, including education around modeling.

# • Consider incentives, particularly for academics and EMR custodians, for implementation. # • Offer workshops where modelers and frontline public health officials can come together to learn > 3Armanios, Daniel E, and Nicola Ritsch. Infrastructure as Social Sensor: The Case for Better Collection and Integration of > Infrastructure Data for Improving Pandemic Response.

Real-time and Granular Data COVID-19 and Other Diseases Communications M O D E R N I Z I N G H E A LT H D ATA A N A LY T I C S A N D F O R E C A S T I N G 1 1 # • Invest in operational structures, infectious disease analytics, modeling and forecasting. Data priorities are dependent on funding, investment and capacity. This work takes continued time, investment and communication.

# • Build a structure during “peace” time such that it can be leveraged during concern periods. # • Fund data collection, aggregation and analysis on an ongoing basis. # • Invest substantially in state and local public health since they do not currently have the capacity to generate the needed data.

# • Create operational structures to make technology and science work in a sensible way. # • Develop operational systems and integrated platforms in “peace” time for data curating, sharing, analytics, modeling and forecasting. # • Maintain all components in terms of people who are employed to do the day-to-day operational work, as well as the go-betweens (between the academics doing research and the operational side).

# • Be aware that there is currently a lot of duplication in the work along with a lack of coordination and centralization of tools. # • Support interdisciplinary contributions in an ongoing way, with regular collaboration and feedback and an organized structure. # • Emphasize that multiple aspects of forecasting and modeling need to be a continuous, # • Ensure continuous and transparent dialogue, evaluation and feedback.

# • Prioritize operations. As noted in Armanios and Ritsch’s report (see Appendix D ), a focus on and enhancements to infrastructure can also be linked to improved economic and health outcomes and plays an important role in social determinants of health. Utilize existing tools and resources related to modeling and forecasting while growing the academic and workforce support.

Offer workshops whereby modelers and frontline public health officials come together to learn from one another. Implement a data lake and a system that links laboratory and EMR data. Make short- and long-term investments in a more robust informatics workforce and maintain these investments, and the Center’s capacity and readiness, even when there are no major events or crises.

Prioritize operations.

M O D E R N I Z I N G H E A LT H D ATA A N A LY T I C S A N D F O R E C A S T I N G 1 2 As part of modeling and forecasting, participants gave special attention in their feedback to data collection at national and local levels, electronic medical records (EMRs) and healthcare data, social determinants of health data, geographic considerations, access to data (identified as the biggest bottleneck to progress) and data standards.

There were approximately 250 comments on this theme. Specific feedback included: # • Adapt data collection systems to county level. A counter perspective noted that local level data analysis can be difficult.

# • Obtain line level health data. # • Obtain data on health seeking behaviors. # • Collect and integrate infrastructure data.

# • Gather data on conditions beyond COVID-19-related conditions. # • Make data streams flexible enough to allow for one-off types of data collection. # • Understand how data are generated.

# • Keep in mind that data are subject to misinterpretation and errors. # • Obtain county level transmission data, vaccination rates, count of new cases, re-infections, admissions and bed use data. # • Create a data repository so that stakeholders can build on what was learned during COVID and # • Evaluate what policies need to change to allow for collection of data.

# • Ensure there is clear guidance and downward policy support for changes at the state level. Electronic Medical Records and Healthcare Data # • Implement a system that collects EMR data throughout the country and have a national system in place that interfaces and queries health information exchanges. In the United States there is a very weak connection between the healthcare system, healthcare data and public health data.

# • Emphasize healthcare data (especially EMR data) as they are massive valuable assets and often an untapped data source. # • Generate live and aggregated healthcare episode data. # • Create a national patient identifier for patient matching.

Social Determinants of Health Data # • Prioritize and focus on addressing and closing social, racial, economic and structural drivers of inequities. To do this, we need social determinants of health data, for example: race, ethnicity, household crowding, comorbidities, access and socioeconomic status data.

# • Assess the conditions that lead to disparate outcomes and the impact of disparate outcomes Real-time and Granular Data COVID-19 and Other Diseases Communications M O D E R N I Z I N G H E A LT H D ATA A N A LY T I C S A N D F O R E C A S T I N G 1 3 # • Keep in mind the importance of geographic scope; this ranges from as local as possible to global data that may offer guidance into new threats.

# • Understand how data, including health data, are captured at local levels and what types of analysis/forecasting models should be done at each level; there is utility to forecasting at the zip code level. Get into a virtuous cycle where data modeling benefits state and local individuals. This provides the incentive for state and local individuals to provide the data.

This worked well during the Malaria Atlas Project. # • Conduct a landscape analysis at the local/grassroots level to understand gaps, capabilities, # • Educate and collaborate with county governments and other key stakeholders so they are on board and involved in decision-making from the beginning.

# • Ensure that data collected from community members gets fed back to the community; otherwise run the risk of resistance in providing data in the future. # • Support local authorities. # • Consider what policies need to be changed to allow access to data needed.

There exists a tremendous amount of data, but there are issues accessing it and therefore not using it. A part of access issues is legal barriers including different regulations on standards for data collection # • Obtain local data in real time. # • Prioritize complete laboratory and epidemiological data.

Often electronic laboratory reporting # • Focus on better data connectivity. # • Address surveillance data barriers but evaluate guardrails and caveats and how we should be # • Establish a national standard for data reporting including supporting guidelines and resources. Have minimum data sets and definitions.

Currently there is no uniform approach to the standardization of data elements and standards vary by jurisdiction. A plethora of data exist, but the biggest bottleneck is access to these data. Consider collecting local-level data for guidance into new threats.

At the same time ensure that data get fed back to communities, so communities understand the value of providing data. Evaluate what policies need to be changed to allow for increased access and data connectivity Data must be collected on variables related to social determinants of health to close the gaps in drivers of health inequities.

A national standard for data reporting is needed, including minimum data sets and definitions to improve data access and sharing. M O D E R N I Z I N G H E A LT H D ATA A N A LY T I C S A N D F O R E C A S T I N G 1 4 REAL-TIME AND GRANULAR DATA Listening session participants emphasized obtaining granular data, disaggregated data and data at the state and local level that can be collected in real time.

There were approximately 70 comments related to this theme. Some participants felt that data become less meaningful when in aggregate and that a flat system where everyone has access to real-time raw data is important. Other participants reflected that public health reporting streams, which are thought of as real time, are outdated.

Additional feedback on this topic included: # • Model human behavior and feedback loops and how they generate and interact with infectious disease dynamics and adoption of practices/restrictions. # • Collect data on human behavior during COVID-19. # • Utilize metagenomic and genomic data to put together clusters of how individual transmission events have been related to each other.

# • Implement homomorphic aggregation technology. # • Obtain mobility travel data at the local level (currently only available at the national level). # • Prioritize low latency and reliable data reporting.

Real-time data have had long lag times from collection to actual reporting, so what is real-time may be outdated. There is currently hesitation with reporting preliminary data that leads to data latency and revisions. It may be easier to report data without revisions, but we can use data set with revisions more effectively.

The traditional reporting process causes a loss of granularity. # • Prioritize timeliness over cleanliness and accuracy; corrections can be applied retrospectively. # • Provide metadata about timing and nature of updates.

# • Build infrastructure and modeling system such that it considers early noise and bias. Obtain disaggregated and state- and local-level data. Include human behavior and feedback loops into modeling for infectious disease dynamics.

Metagenomic and genomic data can be used to create clusters to better understand the relationship of individual transmissions Be willing to report preliminary data without revisions to reduce latency and use data with revisions more effectively. Build modeling systems to include early noise and bias.

COVID-19 AND OTHER DISEASES There were approximately 70 comments related to COVID-19 and more broadly other infectious diseases, current and future novel diseases. The context of the summary below relates to the scope the Center should consider as high priority. Note that better understanding of transmissibility of infectious diseases was highlighted across these comments.

# • Analyze periods of surges. # • Examine COVID-related health conditions and long-term impacts of COVID-19. # • Understand what policies have been effective in slowing down transmission, minimizing deaths and hospitalizations, and improving equity around COVID-19.

# • Consider how current COVID data impacts future forecasts on COVID.

# • Study the relationship between vaccination and natural immunity against COVID Real-time and Granular Data COVID-19 and Other Diseases Communications M O D E R N I Z I N G H E A LT H D ATA A N A LY T I C S A N D F O R E C A S T I N G 1 5 # • Work with academia and the private sector to understand vector-borne diseases and other # • Focus on noninfectious diseases and conditions as well as infectious diseases.

# • Investigate epidemiological-related questions such as transmissibility, disparities in severity and progression of illness, to enhance understanding of how infectious diseases move # • Maintain advances in pooled testing, which allows for more people to be tested quickly using # • Analyze the impact from noncompliant individuals (e.g., mask wearing). # • Evaluate transmission changes and areas of high transmission.

Analyze COVID-19-related policies that have been effective in minimizing transmissibility, severe illness, death and inequities, and apply these policies to other infectious and noninfectious diseases. Consider how to address challenges of noncompliant individuals. Emphasize cross-sector collaboration to understand vector-borne and other zoonotic diseases.

Maintain pooled testing capacity. There were approximately 20 comments on the topic of surveillance. Participants noted that it is a bold goal to have a national syndromic surveillance platform and the current state of funding and resources remains challenging to achieve this goal while trying to build up surveillance.

People who are conducting surveillance should be more familiar with surveillance techniques and the provision of data needed for forecasting. Integrate routine surveillance in the United States and globally, including genetic, serological, active and sentinel surveillance to help with case follow-up.

Integrate modeling into existing Real-time and Granular Data COVID-19 and Other Diseases Communications M O D E R N I Z I N G H E A LT H D ATA A N A LY T I C S A N D F O R E C A S T I N G 1 6 There were approximately 60 comments related to partnerships.

Partnership for the purpose of developing a plan for the Center means evaluating and using existing relationships among those working in modeling, policy and public health and learning from each other to inform future work and allow for increased access to data.

Recommendations included having a systematic approach to look for peer organizations across government and having structured relationships that focus on addressing questions together. Other comments included: # • Build trust and sustained relationships through ongoing efforts, not solely in a crisis.

# • Facilitate a nationwide collaborative with key members; for example, discuss ways health departments, hospital associations and other regional systems share data. # • Leverage partnerships in academia and the private sector, for example, pharmaceuticals, to examine data and diseases and bring faster capabilities. # • Work with local communities to identify critical health priorities.

# • Have an advisory committee that represents diverse perspectives; implement one or two collaborations with state and local groups to be