������Ƶ

1. Introduction

The Quality and Safety Review System (QSRS) was designed to identify the occurrence of specified adverse events to gain a better understanding of patient safety in the hospital setting.¹ In collaboration with the Centers for Medicare & Medicaid Services (CMS), the Agency for Healthcare Research and Quality (AHRQ) gathers data in the QSRS to produce national estimates of adverse events that occur when a patient is hospitalized. QSRS data are collected through retrospective manual abstraction of inpatient records. Human abstractors answer questions related to objective information from the medical record. Algorithms are coded into QSRS that use the answers to these questions to identify whether an AE occurred during a particular hospital stay. Analyses from these data are made available to the public.² The purpose of this document is to provide additional technical information regarding the sampling design and weighting methodology of QSRS data from 2020 through 2024.³

2. Sample Design

CMS conducts the QSRS sampling. The 2020–2024 QSRS national sampling frame represents Medicare beneficiaries (both fee-for-service and Medicare Advantage) ages 18 and older with hospitals stays less than 120 days, who were discharged from acute care hospitals between January 1, 2020, and December 31, 2024.⁴ Records included any discharge status, including death. 

Participating hospitals are stratified into five cohorts: (1) Medicare Rural Acute Care Hospitals (“RURAL”) participating in the Inpatient Prospective Payment System (IPPS); (2) Medicare Targeted Urban Acute Care Hospitals (“TARGETED URBAN”) participating in IPPS and also eligible for Hospital Quality Improvement Contractor (HQIC) services; (3) Critical Access Hospitals (CRITICAL ACCESS); (4) Indian Health Service (IHS) hospitals; and (5) Medicare Other Acute Care Hospitals (“OTHER”) participating in IPPS. By definition, Department of Defense and Department of Veterans Affairs hospitals are excluded. The first four hospital cohorts are types of hospitals that CMS targets as part of the HQIC Program, which aims to improve the effectiveness, efficiency, cost-effectiveness, and quality of services delivered to Medicare beneficiaries.⁵ More specifically, through September 2024, the program supported small rural and critical access hospitals and facilities that care for vulnerable and underserved patients through contractors who focus on patient safety, among other initiatives.⁶ The OTHER cohort consists of all other acute care hospitals and are included to generate national estimates that are representative of acute care hospitals.

Each hospital cohort is defined as follows:

• Rural Acute Care Hospitals are located outside of a Metropolitan Statistical Area, as defined by the Federal Office of Management and Budget. These hospitals do not include rural census tracts within metropolitan counties.⁷
• Targeted Urban Acute Care Hospitals are hospitals targeted in urban areas that represented a significant share of their healthcare market. These hospitals also had identified areas for improvement over a specified period of time. They were targeted by the CMS Hospital Quality Improvement Contractors program consistent with the program’s priorities for quality and safety.
• Critical Access Hospitals are rural hospitals designated by CMS as a separate provider type with their own Medicare Conditions of Participation as well as a separate payment method.⁸
• Medicare Other Acute Care Hospitals consist of all other acute care hospitals that accept Medicare.
• Twenty-two acute care hospitals receiving services through the Indian Health Service (IHS).⁹ 

The estimates for 2021–2024 QSRS-reported analyses are based on annual samples of medical records from four of the five hospital cohorts noted above: RURAL, TARGETED URBAN, CRITICAL ACCESS, and OTHER. While records from IHS hospitals were abstracted using QSRS, information from these hospitals was excluded due to differences in sampling strategies.¹⁰ For the remaining four hospital cohorts, CMS implemented a stratified, two-stage cluster sampling design. The sampling frame includes all discharges by Medicare beneficiaries 18 years and older from acute care hospitals where length of stay was less than 120 days. Any duplicate discharge records were removed. The data are derived from the National Claims History Part A inpatient claims database. 

The hospital selection process begins with the exclusion of ineligible hospitals. Only acute care hospitals are included; other facilities such as children’s hospitals, cancer hospitals, inpatient psychiatric facilities, and long-term care facilities are excluded. Among acute care hospitals, excluded hospitals include those that closed, were selected for the Inpatient Quality Reporting Program, located in an area experiencing a natural disaster (known as a Federal Emergency Management Agency waiver), or those not designated in the IPPS. Probability-proportion-to-size (PPS) sampling was used to randomly select hospitals from within each of the four cohorts of interest each month. PPS sampling is a method of sampling from a finite population in which a size measure is available for each population unit before sampling and where the probability of selecting a unit is proportional to its size.¹¹ The QSRS size measure is the size of the hospital defined by its number of inpatient beds. Larger hospitals were given an increased proportionate chance of being included in the monthly random sample of hospitals. 

The number of hospitals selected each year was determined by CMS’ available budget. The original target sample size per month for these four cohorts of hospitals were: 40 RURAL, 15 TARGETED URBAN, 30 CRITICAL ACCESS, and 70 OTHER. Beginning with the June 2021 sample, the target sample size increased to 44 RURAL, 62 TARGETED URBAN, 44 CRITICAL ACCESS, and 70 OTHER. All hospitals are chosen on an annual basis and then randomly assigned to months 1, 2, and 3 of a quarter equally. When a hospital is selected for the sample, records are requested from that hospital once a quarter for the month they are assigned. 

Simple random sampling is used to select inpatient records from within each hospital. Ten discharge records are selected from each hospital for the month they are assigned to in each quarter (i.e., 40 records per year per hospital) to minimize hospital burden, as hospital submission of the records is not voluntary.   

Sampled hospitals that closed or received a FEMA waiver after initial selection were replaced with other hospitals sampled from within the same cohort. 

3. Weighting Methodology

CMS’ sampling plan for the hospital cohorts does not reflect the true percentage of these hospital cohorts in the United States. Thus, for the QSRS data to produce adverse event estimates that are nationally representative, weighting is required to account for the over- or underrepresentation of certain hospital types included in the sample. As it pertains to QSRS data, the sampling unit of interest is the patient record. Hence, individual patients discharged from a hospital could theoretically be represented more than once in the QSRS sample. The weighting methodology considers the exact probability of a given hospital and associated number of records that were selected and available for a given month. The weighting methodology ensures that estimates are not affected by variation across years in: 

• The percentage of patients in the samples of records representing each of the hospital cohorts.
• The monthly and total number of discharges for each hospital.
• The exclusion of some sampled hospitals due to natural disasters or other emergency waivers or changes in CMS priorities and/or budgetary considerations.

The weights applied to the QSRS estimates account for the above issues and consider the exact probability of a given hospital and associated number of records that were selected and available for a given month. Additionally, the weights account for nonresponse bias and differences between the final sample distribution and the target population distribution. Monthly weighted rates are combined to provide an annual weighted rate. The control totals used for weighting come from all hospitals that submitted Medicare claims for adult patients, 18 years and older, who received care at one of the hospital cohorts included in the study and who were eligible for sampling. SAS^® code was created to deploy the weighting methodology. 

There are three steps to the weighting methodology. The first step is to calculate the base sampling weight for the hospitals. The second step is to calculate the sampling weight for each selected record. The final step is to adjust the sampling weight for bias attributed to missing records.

Step 1: Base Sampling Weights

The initial base weight for each selected hospital within a cohort(h), equals the inverse of the sampling rate W_h within the sampling domain. 

W_h = 1/r_hk, where r_hk = (# hospitals selected in the h^th cohort / total number of hospitals in the universe of the h^th cohort), h = 1 … 5   and k = 1…m(k^th hospital selected in the h^th cohort).

Step 2: QSRS Sampling Weight per Discharge Record

The sampling weight for each discharge record selected during the 2^nd stage of the sampling process is conditioned on the event of belonging to a hospital that was selected during the 1^st stage of the sampling process. Hence the sampling weight rate for the j^th discharge record (W_hj) equals the inverse of the sampling rate for the jth discharge record (W_hj). This equates to inverting the conditional probability of selecting the j^th discharge record given that the k^th hospital was selected within the h^th cohort.

W_hj =    1/Probability (selecting the j^th discharge record | that the k^th hospital was selected within the h^th cohort) = 1/r_j|hk

Computationally, Whj is calculated as the ratio of the number of inpatient discharge records in the universe / the total number inpatient discharge records randomly selected. 

Step 3: Adjusted QSRS Sampling Weight per Discharge Record

During the data abstraction process of collecting information from an inpatient discharge record, some discharge records will be completely missing or lack most of the patient’s relevant medical history. The non-sampling bias attributable to missing inpatient discharge records is expected to be minimal but cannot be ignored. To adjust for this possible bias, we invert the conditional probability of selecting the j^th discharge record and the j^th discharge record is non-missing, given that the k^th hospital was selected within the h^th cohort.

adj W_hj = 1/r_j|hk  where 1/r_{j|hk =}  1/Probability (selecting the j^th discharge record and the j^th discharge record is non-missing | that the k^th hospital was selected within the h^th cohort)

4. Reliability of Estimates 

Researchers and analysts should also be aware of the reliability or unreliability of survey estimates. For QSRS data, an estimated adverse event is considered reliable if it has a relative standard error of 30 percent or less (i.e., the standard error is no more than 30 percent of the estimate) and it is based on at least 12 patient records and a population at risk of at least 100 hospital stays.

1. Quality and Safety Review System (QSRS). Content last reviewed April 2025. Agency for Healthcare Research and Quality, Rockville, MD. /patient-safety/quality-measures/qsrs/index.html.
2. See, e.g.: National Action Alliance for Patient and Workforce Safety. Agency for Healthcare Research and Quality, Rockville, MD. Rodrick D, Phojanakong, P, Timashenka A, Umscheid CA. Adverse Events Among Medicare Hospitalizations in 2021–2023. ������ƵPublication No. 25-0067. Rockville, MD: Agency for Healthcare Research and Quality; September 2025. /sites/default/files/wysiwyg/patient-safety/quality-measures/qsrs/qsrs-2021-2023-adverse-event-data-report.pdf
3. For more information on the descriptions of the events captured in QSRS, go to the Common Formats for Surveillance – Hospital Version 1.0: .
4. While this sample design was intended to be deployed throughout 2020, CMS suspended data collection from January to August 2020 due to the COVID-19 pandemic.
5. U.S. Department of Health and Human Services, "Report to Congress: The Administration, Cost, and Impact of the Quality Fiscal Year 2023," 2024. .
6. Centers for Medicare & Medicaid Services, "Quality Improvement Organizations: Who We Help–Hospitals," [Online]. Available: .
7. Centers for Medicare & Medicaid Services, "Publication #100-07 State Operations Manual," [Online]. Available: .  Health Resources & Services Administration, "FAQ: Rural Residency Planning and Development Program," [Online]. Available: ).
8. Centers for Medicare & Medicaid Services, "Critical Access Hospitals," [Online]. Available: .
9. Indian Health Service, "Indian Health Service: A Quick Look." [Online]. Available: .
10. For Indian Health Service Hospitals, a simple random sample of 100 Medicare beneficiary discharges each month is selected.  In months where fewer than 100 Medicare beneficiaries receive acute inpatient care in an IHS hospital, all discharges were selected.
11. Brewer, K. and Gregoire, T.G. (2009) Introduction to survey sampling, in Sample Surveys: Design, Methods and Applications, vol. 29A (eds D. Pfeffermann and C.R. Rao), Elsevier, Amsterdam, pp. 9–37.  Kish, L. (1965) Survey Sampling, Chapter 7, John Wiley & Sons, Inc., New York.