GRACE Principles
Background
A Validated Checklist
for Evaluating the Quality of Observational
Cohort Studies for Decision-Making Support
GRACE: Good ReseArch for Comparative Effectiveness
The GRACE Checklist is designed for the assessment of observational studies
of comparative effectiveness in terms of their quality and usefulness for
decision-making. The checklist was developed from a review of the literature
with guidance from recognized experts in this field. The content includes
questions about data and methods. One hundred and thirteen (113) volunteer
testers have rated 280 articles. Validation activities have documented the
usefulness of all 11 questions in this checklist. Approaches to scoring are
under consideration. The GRACE Initiative has been spearheaded by Quintiles
Outcome in collaboration with the National Pharmaceutical Council. GRACE
contributors represent perspectives from academic, government, and private
sectors in the US, Europe, Asia and Africa. A listing of contributors and
collaborators can be found at www.graceprinciples.org. More information is
available in the American Journal of Managed Care 2010; 16(6): 467-471
(Dreyer NA, Schneeweiss S, McNeil B et al.) The methods and results for the
validation have been submitted for publication.
To join the GRACE Initiative or for more information, please contact
them at
[email protected]
Nancy A. Dreyer
Leader, GRACE Initiative
Data
D1. Were treatment and/or important details of treatment exposure
adequately recorded for the study purpose in the data source(s)?
Note: not all details of treatment are required for all research
questions.
Yes: reasonably necessary information to
determine treatment or intervention was adequately recorded for study
purposes (e.g., for drugs, sufficient detail on dose, days supplied, route
or other data important. For vaccines, consider the importance of batch,
dose, route and site of administration, etc. For devices, consider type of
device, placement, surgical procedure used, serial number, etc.).
No: data source clearly deficient or not enough information
in
article.
D2. Were the primary outcomes adequately recorded for the study
purpose (e.g., available in sufficient detail through data
source(s))?
Yes: information to ascertain
outcomes were adequately recorded in the data source (e.g., if
clinical outcomes were ascertained using ICD-9 diagnosis code(s) in
an administrative database, the level of sensitivity and specificity
captured by the code(s) was sufficient for assessing the outcome of
interest.)
No: data source clearly deficient
(e.g., the code(s) captured a range of conditions that was too broad
or narrow, and supplementary information such as that from medical
charts was not available), or not enough information in article
D3. Was the primary clinical outcome(s) measured objectively rather than
subject to clinical judgment (e.g., opinion about whether the patient’s
condition has improved)?
Yes:
clinical outcomes were measured objectively (e.g., hospitalization,
mortality) N/A--primary outcome not clinical (e.g., PROs)
No:
e.g., clinical opinion about whether patient’s condition improved, or not
enough information in article
D4. Were primary outcomes validated, adjudicated, or otherwise known
to be valid in a similar population?
Yes: outcomes were
validated, adjudicated, or based on medical chart abstractions with clear
definitions, e.g. a validated instrument was used to assess patient-reported
outcomes (e.g., SF-12 Health Survey); a clinical diagnosis via ICD-9 code
was used, with formal medical record adjudication by committee to confirm
diagnosis or other procedures to achieve reasonable sensitivity and
specificity; billing data were used to assess health resource utilization,
etc.
No: No, or not enough information in article
D5. Was the primary outcome(s) measured or identified in an
equivalent manner between the treatment/ intervention group and the
comparison group(s)?
Yes.
No or not enough information in article
D6. Were important covariates that may be known confounders or effect
modifiers available and recorded? Important covariates depend on the
treatment and/or outcome of interest, (e.g., body mass index should be
available and recorded for studies of diabetes; race should be available and
recorded for studies of hypertension and glaucoma).
Yes. most if not all important
known confounders and effect modifiers available and recorded, e.g.,
measures of medication dose and duration.
No at least one important known confounder or effect modifier
not available and recorded (as noted by authors or as determined by user’s
clinical knowledge), or not enough information in article
Methods
M1. Was the study (or analysis) population restricted to new initiators
of treatment or those starting a new course of treatment? Efforts to include
only new initiators may include restricting the cohort to those who had a
washout period (specified period of medication nonuse) prior to the
beginning of study follow-up.
Yes. Only new initiators of the treatment of interest were included
in the cohort, or for surgical procedures and devices, only patients who
never had the treatment before the start of study follow-up were included.
No or not enough information in article
M2. If one or more comparison groups were used, were they
concurrent comparators? If not, did the authors justify the use of
historical comparisons group(s)?
Yes. Yes--data were collected during the same time period as the
treatment group (“concurrent”) or historical comparators were used with
reasonable justification, e.g., when it is impossible for researchers to
identify current users of older treatments or when a concurrent comparison
group is not valid--(i.e., uptake of new product is so rapid that concurrent
comparators differ greatly on factors related to the outcome)
No. No--historical comparators used without being scientifically
justifiable, or not enough information in article
M3. Were important covariates, confounding and effect modifying
variables taken into account in the design and/or analysis? Appropriate
methods to take these variables into account may include: restriction,
stratification, interaction terms, multivariate analysis, propensity score
matching, instrumental variables or other approaches.
Yes.
Yes--most if not all important covariates that would be likely to change the
effect estimate substantially were accounted for, e.g., measures of
medication dose and duration.
No.
No--some important covariates were available
for analysis but not analyzed appropriately, or at least one important
covariate was not measured, or not enough information in article
M4. Is the classification of exposed and unexposed
person-time free of “immortal time bias”? Immortal time in
epidemiology refers to a period of cohort follow-up time during
which death (or an outcome that determines end of follow-up) cannot
occur.
Yes.
No.
No, or not enough information in article
M5. Were any meaningful analyses conducted to test key
assumptions on which primary results are based? E.g., were some analyses
reported to evaluate the potential for a biased assessment of exposure or
outcome, such as analyses where the impact of varying exposure and/or
outcome definitions was tested to examine the impact on results.
Yes (1). Yes--and primary results did not substantially change
Yes (2) Yes--and primary results changed substantially
No.
None reported, or not enough information in article
References
Dreyer NA, Bryant A, Velentgas P. The GRACE checklist: a validated
assessment tool for high quality observational studies of comparative
effectiveness. J Managed Care & Specialty Pharmacy 2016; 22(10):1107-13.
