Lung Cancer Screening Before and After an Electronic Health Record Intervention (2024)

Key Points

Question Is a multifaceted lung cancer screening intervention, including a shared decision-making tool, clinician-facing reminders, narrative guidance provided in the ordering process, and patient-facing reminders, associated with improved screening-related care (care gap closure) for lung cancer screening?

Findings In this nonrandomized controlled trial of 1865 patients aged 55 to 80 years who smoked 30 pack-years or more, the care gap closure rate for lung cancer screening improved from 15.9% to 46.9%.

Meaning This study found that a multifaceted electronic health record–integrated intervention was associated with increased care gap closure for lung cancer screening.

Importance Lung cancer is the deadliest cancer in the US. Early-stage lung cancer detection with lung cancer screening (LCS) through low-dose computed tomography (LDCT) improves outcomes.

Objective To assess the association of a multifaceted clinical decision support intervention with rates of identification and completion of recommended LCS-related services.

Design, Setting, and Participants This nonrandomized controlled trial used an interrupted time series design, including 3 study periods from August 24, 2019, to April 27, 2022: baseline (12 months), period 1 (11 months), and period 2 (9 months). Outcome changes were reported as shifts in the outcome level at the beginning of each period and changes in monthly trend (ie, slope). The study was conducted at primary care and pulmonary clinics at a health care system headquartered in Salt Lake City, Utah, among patients aged 55 to 80 years who had smoked 30 pack-years or more and were current smokers or had quit smoking in the past 15 years. Data were analyzed from September 2023 through February 2024.

Interventions Interventions in period 1 included clinician-facing preventive care reminders, an electronic health record–integrated shared decision-making tool, and narrative LCS guidance provided in the LDCT ordering screen. Interventions in period 2 included the same clinician-facing interventions and patient-facing reminders for LCS discussion and LCS.

Main Outcome and Measure The primary outcome was LCS care gap closure, defined as the identification and completion of recommended care services. LCS care gap closure could be achieved through LDCT completion, other chest CT completion, or LCS shared decision-making.

Results The study included 1865 patients (median [IQR] age, 64 [60-70] years; 759 female [40.7%]). The clinician-facing intervention (period 1) was not associated with changes in level but was associated with an increase in slope of 2.6 percentage points (95% CI, 2.4-2.7 percentage points) per month in care gap closure through any means and 1.6 percentage points (95% CI, 1.4-1.8 percentage points) per month in closure through LDCT. In period 2, introduction of patient-facing reminders was associated with an immediate increase in care gap closure (2.3 percentage points; 95% CI, 1.0-3.6 percentage points) and closure through LDCT (2.4 percentage points; 95% CI, 0.9-3.9 percentage points) but was not associated with an increase in slope. The overall care gap closure rate was 175 of 1104 patients (15.9%) at the end of the baseline period vs 588 of 1255 patients (46.9%) at the end of period 2.

Conclusions and Relevance In this study, a multifaceted intervention was associated with an improvement in LCS care gap closure.

Trial Registration ClinicalTrials.gov Identifier: NCT04498052

Lung cancer, the leading cause of cancer-related deaths in the US, continues to be a formidable health challenge.¹ The US Preventive Services Task Force (USPSTF) recommends offering lung cancer screening (LCS) through low-dose computed tomography (LDCT) to individuals with a significant tobacco use history.^2,3 Although LCS reduces mortality by as much as 20%,^4,5 the LCS rate among eligible individuals was approximately 6.5% in 2020.⁶

To address low LCS rates, health care systems across the US are beginning to implement clinical decision support (CDS) interventions to support LCS.^7-10 In a study at University of Utah Health (UUH), a multifaceted intervention, including clinician-facing reminders and an electronic health record (EHR)–integrated shared decision-making (SDM) tool, was associated with an increase in LDCT ordering (adjusted odds ratio [OR], 4.9, 95% CI, 3.4-6.9).^7,8 Another study reported that implementing EHR prompts was associated with an increase in LDCT ordering (adjusted OR, 1.04; 95% CI, 1.01-1.07).⁹

These prior studies of EHR-integrated CDS interventions focused on LCS ordering and completion among patients who had not undergone LCS in the past year.^8,9 While LDCT is central to meeting patient needs related to LCS, it is not the only acceptable approach for closing this care gap. Specifically, patients can be screened for lung cancer through other chest CTs (eg, diagnostic chest CTs) conducted for purposes other than LCS. Furthermore, patients may decline screening after SDM given that there are substantial potential downsides to screening and wide individual variation in patient lung cancer risk, life expectancy, and potential net benefit from screening.¹¹ Therefore, other chest CTs or LCS SDM should be considered in addition to LDCT as valid approaches to LCS care gap closure.

This study’s primary goal was to provide a more holistic understanding of the association of a multifaceted, EHR-based CDS intervention with LCS care gap closure by accounting for these various approaches. A secondary goal was to evaluate whether providing simple patient portal reminders for LCS-eligible patients was associated with additional improvements in care cap closure. The primary study objective was to evaluate changes in LCS care gap closure via any means (ie, LDCT, other chest CT, or SDM) after the introduction of a multifaceted, EHR-integrated clinician-facing intervention (period 1) and the addition of patient-facing EHR patient portal reminders (period 2).

This nonrandomized controlled trial followed a interrupted time series (ITS) design, encompassing a baseline period (usual care) lasting 12 months (August 24, 2019, to August 23, 2020), an intervention period 1 (clinician-facing intervention) lasting 11 months (August 24, 2020, to July 27, 2021), and an intervention period 2 (patient-facing intervention) lasting 9 months (July 28, 2021, to April 27, 2022). The study was approved by the University of Utah Institutional Review Board and registered with ClinicalTrials.gov (NCT04498052; see trial protocol in Supplement 1). There were no significant deviations from the registered trial. A waiver of consent was approved by the University of Utah Institutional Review Board because the intervention, which follows USPSTF guidelines, does not add more risk than the current standard of care and measures were in place to minimize privacy risks. Data were obtained from the UUH Enterprise Data Warehouse on September 14, 2023. This report follows the Transparent Reporting of Evaluations With Nonrandomized Designs (TREND) reporting guideline.

The research was carried out across 28 primary care and 4 pulmonary clinics located at 12 UUH locations. UUH uses a decentralized approach to LCS whereby frontline clinicians, such as primary care clinicians and pulmonologists, refer eligible and interested patients for LCS. The UUH LCS program is accredited by the American College of Radiology, and the Huntsman Cancer Institute maintains a registry of patients who have undergone LCS. There were 2 clinics at 1 location that started the intervention but were excluded because they closed before study completion.

UUH uses the Epic EHR system (versions February 2019, August 2019, February 2020, August 2020, February 2021, and May 2022). The creation of the multifaceted LCS CDS intervention was led by a collaborative initiative known as ReImagine EHR,¹² which applies interoperable EHR innovations to patient care.

Study eligibility criteria were evaluated at the level of primary care visits (office visits or telehealth visits) during the study period. Patients were eligible if they met 2013 USPSTF LCS eligibility criteria at the time of the visit and had at least 1 primary care visit in the preceding year. Per USPSTF guidelines, a person qualified for LCS if they were aged 55 to 80 years, had a smoking history of 30 pack-years or greater, actively smoked or had quit in the previous 15 years, and had not been diagnosed with lung cancer.² While USPSTF expanded these criteria in 2021,¹³ this analysis used 2013 criteria to maintain comparable patient populations across study periods. Inclusion and exclusion criteria were determined using EHR data on smoking, demographics, problem list entries, medical history, and encounter diagnoses.

Period 1 CDS tools included clinician-facing reminders for LCS and LCS discussion in the EHR system Health Maintenance module, an EHR-integrated SDM tool (Figure 1), and narrative guidance provided in the LDCT ordering screen regarding LCS guidelines, including requirements from the Centers for Medicare & Medicaid Services to conduct SDM using a decision aid prior to initiating screening.¹¹ The period 1 intervention was described previously⁸ and is detailed in the eMethods in Supplement 2.

In period 2, patient-facing reminders were added. These reminders were part of the EHR system Health Maintenance module, in which care gaps can be optionally presented to patients on the main portal screen. Reminders can also be accessed through the patient portal menu. Patient reminders consisted of notifications on the need for Lung Cancer Screening or Lung Cancer Screening Discussion based on whether patients required screening or an initial discussion on screening. No further information (eg, an explanation of LCS) was provided due to patient portal limitations.

In period 1, the SDM tool and clinician reminders supported 2013 USPSTF guidelines.² Individualized predictions of net benefit in the SDM tool were based on the Bach risk model.^14,15 In period 2, the SDM tool and clinician reminders were updated to support 2021 USPSTF guidelines.¹³ The SDM tool was updated to use the life-years gained from screening (LYFS)-CT model at this time to account for the higher risk of lung cancer among Black individuals; a threshold of at least 16.2 days of life expected to be gained from screening was used to identify patients expected to have a high benefit.¹⁶ Study interventions were communicated to users via typical channels for notifying clinicians regarding EHR system updates, including EHR-integrated prompts described previously and inclusion in brief updates on new EHR features presented at clinician meetings.

The primary outcome was LCS care gap closure through any means. Care gap closure through any means could be achieved through LDCT completion in the past year, completion of another chest CT in the past year, or SDM documentation in the past 3 years. To assess population care gap closure levels at the end of each study period, we estimated the care gap closure status for all patients who had primary care visits in the 12 months preceding the last day of the period. Using structured EHR data, we considered SDM documented if a clinician noted that the need for LCS discussion was addressed, the patient declined screening, or LCS was not appropriate.⁸ The primary hypothesis was that introduction of the multifaceted, EHR-integrated intervention would be associated with increased LCS care gap closure.

Secondary outcomes included component mechanisms for achieving care gap closure (ie, through LDCTs, other chest CTs, and SDM), 3 mechanisms for unclosed care gaps (LCS ordered in past year but not completed, LCS elected in past 3 years but not ordered, and LCS not ordered and SDM not completed), and SDM tool use in the past 3 years. Of care gap closure mechanisms, closure through LDCTs was of particular interest.

To ensure that the patient population was stable across study periods, we assessed patient characteristics, including estimated screening benefit level,¹⁶ receipt of care in pulmonary clinics, sex, race and ethnicity, age, tobacco use, comorbidities, body mass index (calculated as weight in kilograms divided by height in meters squared), insurance status, family history of lung cancer, and patient portal use in the past year. The screening benefit level was reported in accordance with guidance from the American College of Chest Physicians to identify patients who might obtain the most benefit from screening (patients expected to have a high benefit).¹⁷ To identify these patients, we used LYFS-CT models, with a benchmark of an anticipated gain of at least 16.2 days of life.¹⁶ We identified 13 medical conditions or comorbidities from the problem list, medical history, and visit diagnoses. Race and ethnicity were derived from the EHR, where they were documented based on patient self-report as a part of routine clinical care. Race was documented as American Indian and Alaska Native, Asian, Black or African American, Native Hawaiian and Other Pacific Islander, White or Caucasian, and choose not to disclose. Ethnicity was documented as Hispanic or Latino, not Hispanic or Latino, and choose not to disclose. We aggregated race and ethnicity data into Hispanic or Latino, Black or African American, White or Caucasian, and other. The other race and ethnicity category included individuals of non-Hispanic ethnicity who selected a race of American Indian and Alaska Native, Asian, Native Hawaiian and Pacific Islander, or choose not to disclose. Non-Hispanic individuals were also categorized as other race and ethnicity if they selected more than 1 race or had no race selection in the system. Additionally, we reported Utah COVID-19 hospitalization rates as the 7-day mean of hospitalizations during the patient’s last eligible visit during each period.¹⁸ Patient characteristics were calculated as of the last date of each study period, with the last known observation carried forward for categorical variables. For overall patient demographics, we used the last date of the baseline period.

We used median and IQR to summarize continuous characteristics and count and percentage to summarize categorical characteristics. To compare characteristics across study periods, we used generalized linear models. To compare period 1 with baseline and period 2 with baseline, indicators of whether patients had visits in both periods were added to the model.

To evaluate the association of interventions with care gap closure rates, we conducted ITS analysis using the segmented regression approach. We chose segmented regression analysis because it allowed us to assess preexisting trends and changes in the slope and level of outcomes.¹⁹ Each patient was assigned a value of 1 (for care gap closure) or 0 (for care gap nonclosure) at the end of each month. The mean of these values for each month was found, and values were fitted into segmented linear regression models with study periods constituting 3 segments. We used the segmented least squares approach with parameters for intercept, baseline trend, and changes in the level and trend after the intervention. Care gap closure rates were stratified by estimated screening benefit level. We expected that implementing the tool would be associated with a higher rate of care gap closure in study period 1 compared with baseline, period 2 compared with period 1, and among patients with a high benefit compared with patients who were eligible but with intermediate benefit.

For primary and secondary outcome measures, we also estimated adjusted intervention outcomes after controlling for all characteristics using the covariate balancing propensity score (CBPS) approach.²⁰ For binary outcomes, we used logistic regression. Improvements in covariate balance after propensity score adjustment are summarized in eFigure 2 in Supplement 2.

Statistical significance was defined at alpha = .05. We used R statistical software version 4.3.2 (R Project for Statistical Computing) and tidyverse, WeightIt, nnet, and betareg extension packages. Data were analyzed from September 2023 through February 2024.

There were 19 008 patients who met inclusion criteria (eTable 1 in Supplement 2). Using exclusion criteria consecutively, we excluded 7698 patients (40.5%) due to insufficient detailed smoking data in the EHR to determine eligibility, 10 847 patients (57.1%) due to not meeting USPSTF criteria for lung cancer screening based on detailed smoking data, and 75 patients (0.4%) due to lung cancer diagnosis. The study included 1865 patients (median [IQR] age, 64 [60-70] years; 759 female [40.7%] and 1106 male [59.3%]; 98 Hispanic [5.3%], 36 non-Hispanic Black or African American [1.9%], 1574 non-Hispanic White [84.4%], and 157 other race or ethnicity [8.4%]) who met eligibility criteria (9.8%) (eFigure 1, eTable 1, and eTable 2 in Supplement 2). A substantial proportion of patients had insufficient EHR smoking data to determine eligibility. Populations were similar in the 3 periods (eg, the median [IQR] age was 65.0 [60.8-71.0] at baseline, 65.0 [60.0-70.0] years in period 1, and 65.0 [60.0-70.0] years in period 2) (Table 1). The COVID-19 pandemic started during period 1. The pandemic was associated with increased use of the patient portal and telehealth visits.

In the baseline period, the LCS care gap closure through any means and LCS care gap closure through LDCT were 17.8% (95% CI, 17.0% to 18.6%) and 9.2% (95% CI, 19.0% to 20.8%), respectively, and their slopes were −0.3% (95% CI, −0.5% to −0.2%) per month and −0.2% (95% CI, −0.4% to −0.1%) per month, respectively (Table 2 and Figure 2). In intervention period 1 (clinician-facing intervention), changes in care gap closure and LDCT-based closure levels were not significant, but changes in their slopes were significant, at 2.6 percentage points (95% CI (95% CI, 2.4 to 2.7 percentage points) per month and 1.6 percentage points (95% CI, 1.4 to 1.8 percentage points) per month, respectively. In intervention period 2 (clinician- and patient-facing reminders), care gap closure and LDCT-based closure levels increased by 2.3 percentage points (95% CI, (1.0 to 3.6 percentage points) and 2.4 percentage points (95% CI, 0.9 to 3.9 percentage points), respectively, and their slopes decreased by −1.7 percentage points (95% CI, −1.9 to −1.5 percentage points) per month and −1.2 percentage points (95% CI, −1.4 to −0.9 percentage points) per month, respectively. Analyses stratified by patient benefit level showed similar trends as the overall analysis. The increase in the slope of LDCT completion in period 1 was higher for patients with a high benefit compared with those with an intermediate benefit (1.9 percentage points; 95% CI, 1.6 to 2.1 percentage points per month vs 1.1 percentage points; 95% CI, 0.9 to 1.4 percentage points per month).

Regression analysis results are summarized in Table 3 with adjusted ORs estimated via CBPS. This analysis confirmed results for the ITS analysis. The intervention was associated with an increase in LCS care gap closure, from 175 of 1104 patients (15.9%) at the end of the baseline period to 510 of 1219 patients (41.8%) at the end of intervention period 1 (adjusted OR vs baseline, 3.7; 95% CI, 3.04.6) and 588 of 1255 patients (46.9%) at the end of intervention period 2 (adjusted OR vs baseline, 4.3; 95% CI, 3.5,-5.3). At the end of intervention period 2, most patients (298 patients [23.7%]) achieved care gap closure though LDCT, followed by documentation of SDM in Health Maintenance modules (159 patients [12.7%]) and other chest CT (131 patients [10.4%]). For 107 patients in period 2 [8.5%], LCS was ordered but not completed. The number of patients for whom neither SDM nor LCS were completed decreased from 889 patients (80.5%) at baseline to 547 patients (43.6%) at the end of intervention period 2 (adjusted OR vs baseline, 0.2; 95% CI, 0.2-0.2). The SDM tool was used for 168 of 1255 eligible patients (13.4%) in period 2.

Most of the increase in care-gap closure was contributed by increases in LDCT and LCS SDM. Care gap closure through LDCT increased from 75 patients (6.8%) in the baseline period to 278 patients (22.8%) in period 1 (adjusted OR vs baseline, 3.6; 95% CI, 2.7-4.7) and 298 patients (23.7%) in period 2 (adjusted OR vs baseline, 3.5; 95% CI, 2.6-4.6), while care gap closure through LCS SDM increased from 7 patients (0.6%) in the baseline period to 106 patients (8.7%) in period 1 (adjusted OR vs baseline, 25.3; 95% CI, 9.8-65.8) and 159 patients (12.7%) in period 2 (adjusted OR vs baseline, 29.0; 95% CI, 11.3-74.3) and care-gap closure through other chest CT increased from 93 patients (8.4%) in the baseline period to 126 patients (10.3%) in period 1 (adjusted OR vs baseline, 1.2; 95% CI, 0.9-1.6) and 131 patients (10.4%) in period 2 (adjusted OR vs baseline, 1.4; 95% CI, 1.0-1.9).

This nonrandomized controlled trial found that introduction of a multifaceted, EHR-integrated intervention was associated with improved LCS care gap closure in an academic health care system. Introduction of clinician-facing interventions was associated with improvement (period 1), with a slight further increase associated with the addition of patient-facing reminders (period 2). Although the care gap closure rate slowed in period 2, eligible patients continued to have their care gaps closed throughout the study. While both clinician- and patient-facing LCS interventions showed potential benefits, 43.6% of screening-eligible patients still lacked LDCT orders or discussions at the end of period 2. These findings suggest that further implementation strategies are needed to improve LCS care gap closure.

Study analyses considered care gap closure as a multifactor outcome, including LDCT completion, other chest CTs, and SDM documentation. These alternative approaches to care gap closure accounted for a substantial proportion of overall closures, with 10.4% attributed to other chest CTs and 12.7% to SDM documentation, compared with 23.7% for LDCT. Notably, the National Committee for Quality Assurance announced plans to introduce a Healthcare Effectiveness Data and Information Set quality measure for LCS; however, it is unclear whether these measures would address alternative strategies to care gap closure.²¹ While such quality measures are defined, findings of our study suggest the need to account for LCS care gap closures through not only LDCT, but also other chest CTs and SDM.

This study has several strengths. It evaluated a multifaceted, standards-based, and EHR-integrated intervention that could potentially be widely scaled.⁸ Furthermore, this study evaluated additive outcomes associated with simple patient portal reminders using detailed smoking history in the EHR to assess eligibility; this has not been studied to date, to our knowledge. Additionally, this study found that other chest CTs and SDM combined were approximately as common (23.1%) as LDCTs (23.7%) for closing LCS care gaps in phase 2. This is consistent with prior literature identifying that most patients have lung cancer identified through chest imaging outside of LDCTs.²²

This study identified several areas of need for further research and improvement. As identified by us and others previously^23-26 and as underscored by the large number of patients with unknown LCS eligibility in this study, there is a need to improve the documentation of detailed smoking history in the EHR. Moreover, 8.5% of patients in this study had an LDCT ordered in the past year but did not complete it, indicating the need for improving follow-through after LDCT ordering. Even with clinician- and patient-facing interventions, LCS care gaps remained in more than half of LCS-eligible patients, and the LCS tool was used with only 13.4% of eligible patients, suggesting the need for further research to test implementation strategies to improve SDM and LCS. Additionally, as shown by the modest change in outcome associated with the simple patient portal reminder, more research may be needed on improving patient education and empowerment through more engaging patient-facing interventions. Accordingly, a new multisite trial is underway to evaluate the additive outcome associated with offering more robust patient-facing interventions, including individualized patient education through the patient portal.

This study has several limitations. First, we used a nonrandomized ITS study design without parallel controls. The onset of the COVID-19 pandemic during the baseline period posed potential biases, such as reduced LCS rates due to diminished clinician capacity. We believe that the small increase in other chest CTs may be attributed to the COVID-19 pandemic rather than to our intervention. Furthermore, USPSTF guideline changes in 2021 resulted in a larger number of patients becoming eligible for LCS. We used more stringent 2013 USPSTF criteria to maintain compatibility with period 1, but we were unable to fully separate the intervention outcome from potential outcomes associated with the guideline change itself. To mitigate the impact of this limitation, we used propensity score analysis techniques to account for differences in covariates. Second, our study took place in a single academic health care system, limiting generalizability. Third, we depended on EHR smoking history data, which may underestimate patient eligibility due to inaccuracies.²³ Fourth, the quality of SDM was not evaluated. Fifth, due to the limited quality of EHR smoking data, a large proportion of individuals had insufficient detailed smoking history data to determine their eligibility. We are currently implementing follow-up interventions aimed at improving the identification of patients eligible for LCS.

In this nonrandomized controlled trial, implementing a comprehensive intervention that combined clinician-facing EHR reminders, an EHR-integrated SDM tool for personalized screening, narrative guidance presented in the LDCT order screen, and patient-facing reminders was associated with an increase in LCS care gap closure. Further research is needed to improve LCS, including through improved documentation of detailed smoking history in the EHR, improved LDCT follow-through rates, and more effective engagement of patients in their care.

Accepted for Publication: March 24, 2024.

Published: June 7, 2024. doi:10.1001/jamanetworkopen.2024.15383

Open Access: This is an open access article distributed under the terms of the CC-BY License. © 2024 Kukhareva PV et al. JAMA Network Open.

Corresponding Author: Kensaku Kawamoto, MD, PhD, MHS, Department of Biomedical Informatics, University of Utah, 421 Wakara Way, Ste 108, Salt Lake City, UT 84108 (kensaku.kawamoto@utah.edu).

Author Contributions: Dr Kawamoto had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

Concept and design: Kukhareva, Caverly, fa*gerlin, Del Fiol, Zhang, Butler, Flynn, Reddy, Choi, Nanjo, Kawamoto.

Acquisition, analysis, or interpretation of data: Kukhareva, Li, Caverly, Del Fiol, Hess, Zhang, Schlechter, Balbin, I. Warner, P. Warner, Kawamoto.

Drafting of the manuscript: Kukhareva, Li, Zhang, Kawamoto.

Critical review of the manuscript for important intellectual content: Kukhareva, Caverly, fa*gerlin, Del Fiol, Hess, Zhang, Butler, Schlechter, Flynn, Reddy, Choi, Balbin, I. Warner, P. Warner, Nanjo, Kawamoto.

Statistical analysis: Kukhareva, Li, Zhang.

Obtained funding: Del Fiol, Butler, Kawamoto.

Administrative, technical, or material support: Kukhareva, Del Fiol, Flynn, Balbin, I. Warner, P. Warner, Nanjo, Kawamoto.

Supervision: Caverly, Zhang, Reddy, Kawamoto.

Conflict of Interest Disclosures: Drs Caverly and fa*gerlin reported being co-creators of DecisionPrecision, a freely available and open-source tool to enable personalized shared decision-making for lung cancer screening. Dr Hess reported receiving grants from the National Institutes of Health during the conduct of the study and personal fees from the Astellas Pharmaceuticals data and safety monitoring board outside the submitted work. Dr Butler reported receiving consulting fees from the University of California, San Francisco, outside the submitted work. Dr Choi reported receiving grants from the National Library of Medicine during the conduct of the study. Dr Kawamoto reported receiving grants from Hitachi; personal fees from Pfizer, RTI International, the University of California at San Francisco, Indiana University, NORC at the University of Chicago, the University of Pennsylvania, Yale University, the Regenstrief Foundation, the Korean Society of Medical Informatics, the University of Nebraska, and the U.S. Office of the National Coordinator for Health Information Technology (via Security Risk Solutions); a book chapter honorarium from Elsevier; codevelopment from MD Aware; and licensing from Custom Clinical Decision Support outside the submitted work; serving as an unpaid board member of the nonprofit Health Level Seven International health information technology standard development organization and the US Health Information Technology Advisory Committee; and developing health information technology tools that may be commercialized to enable wider impact. No other disclosures were reported.

Funding/Support: This study was supported in part by grants R18HS026198 and R18HS028791 from the Agency for Healthcare Research and Quality. Dr Caverly was supported by Veterans Affairs Health Services Research and Development Career Development Award 16-151.

Role of the Funder/Sponsor: The funders had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

Disclaimer: The content is solely the responsibility of the authors and does not necessarily represent the official views of the organizations involved.

Data Sharing Statement: See Supplement 3.

Additional Contributions: Tony Jones, Angela Wigren, and Mark Bittinger, MS (University of Utah), contributed to the development of the clinician-facing prompts and order forms. There was no compensation beyond their basic salaries.