Control of Allergic Rhinitis and Asthma Test: A COSMIN Analysis
Overview
The Control of Allergic Rhinitis and Asthma Test (CARAT) serves as a patient-reported outcome measurement (PROM) to evaluate the management of asthma and allergic rhinitis (AR) within a 4-week timeframe. This systematic review aimed to assess CARAT’s measurement properties. Adhering to PRISMA and COSMIN guidelines, a thorough search across five bibliographic databases was conducted, focusing on studies related to CARAT’s development, property assessment, validation, and potential cultural adaptations. Rigorous evaluations of methodological quality, measurement property quality, and overall evidence quality were undertaken. Additionally, meta-analysis of CARAT’s measurement properties was performed. A total of 16 studies were included in the analysis.
The analysis revealed that the Control of Allergic Rhinitis and Asthma Test demonstrated adequate content validity and strong consistency (meta-analytical Cronbach’s alpha = 0.83; 95% CI = 0.80–0.86; I² = 62.6%). In terms of control of allergic rhinitis and asthma, the meta-analytical intraclass correlation coefficient was 0.91 (95% CI = 0.64–0.98; I² = 93.7%). Notably, it showcased favorable construct validity, particularly in its correlations with patient-reported outcome measures focusing on asthma (absolute Spearman correlation coefficients ranging from 0.67 to 0.73; moderate quality of evidence). Moreover, the test demonstrated good responsiveness, with a minimal important difference of 3.5.
Overall, the Control of Allergic Rhinitis and Asthma Test exhibited strong internal consistency, reliability, construct validity, and responsiveness, despite varying degrees of evidence quality. This suggests that CARAT can be effectively utilized to evaluate the management of asthma and allergic rhinitis. As a pioneering effort, this meta-analysis of CARAT’s measurement properties significantly enhances the level of evidence for questionnaires targeting asthma and/or AR control.
Introduction
Patient-reported outcome measures (PROMs) have been developed to quantify how a specific disease or group of diseases impact patients from their own perspective. These measures offer valuable insights into various aspects of diseases, from treatment effectiveness to quality of life, playing a crucial role in guiding clinical decisions. When it comes to assessing the control of asthma and allergic rhinitis (AR), several PROMs are available, such as the Asthma Control Test (ACT), the Asthma Control Questionnaire (ACQ), the Allergic Rhinitis Control Test (ARCT), and the Rhinitis Control Assessment Test (RCAT). However, these PROMs focus on assessing asthma and AR separately.
The challenge arises from the fact that many asthma patients also have AR, necessitating a comprehensive evaluation of both conditions simultaneously. The Allergic Rhinitis and Its Impact on Asthma (ARIA) initiative recommends a holistic assessment using a single tool. Among the available PROMs, the Control of Allergic Rhinitis and Asthma Test (CARAT) stands out as the only measure designed to assess control over both asthma and AR. CARAT features 10 questions addressing symptoms in the upper and lower airways, sleep disturbances, activity limitations, and medication adjustments over the past 4 weeks. The total score ranges from 0 to 30, with scores above 24 indicating good control of both conditions. The development of CARAT has been meticulously documented and validated by various independent studies.
Moreover, CARAT has gained widespread adoption in clinical practice and scientific research, leading to its translation and adaptation based on international recommendations and best practices. It’s available in both paper and digital formats, accessible through websites and mobile health apps, allowing patients to use it between medical visits. As the use of CARAT continues to grow, there is a need for a comprehensive assessment of its measurement properties. Hence, the objective of this systematic review was to rigorously evaluate CARAT’s measurement properties using the COSMIN methodology, following systematic review guidelines for PROMs.
Method
This systematic review, coupled with a meta-analysis, adhered to the guidelines outlined by the Preferred Reporting in Systematic Reviews and Meta-Analyses (PRISMA) as well as the COSMIN methodology for systematic reviews of patient-reported outcome measures (PROMs). The COSMIN methodology provides specific directives for evaluating the bias risk within primary studies, rating the measurement properties, and determining the overall quality of evidence for each property.
To conduct this comprehensive review, a meticulous search was carried out in January 2022 across five prominent bibliographic databases: Ovid/MEDLINE, Web of Science, Scopus, ClinicalTrials.Gov, and the Cochrane Central Register of Controlled Trials (CENTRAL). Moreover, the references of the studies included in the review were meticulously screened to uncover potentially relevant research. In an effort to be thorough, a manual exploration was also undertaken on Google Scholar to identify any additional studies referencing the primary studies encompassing CARAT’s development, validation, or cultural adaptation. This diligent approach aligns with the review’s commitment to a comprehensive analysis.
From each included primary study, data extraction was conducted independently by two authors (RJV and ACF) using a custom-designed form. This encompassed variables like sample size, age and gender distribution of participants, prevalence of asthma and/or allergic rhinitis (AR), healthcare setting, country, and the language of questionnaire administration. In addition, information about CARAT’s measurement properties garnered from each study was retrieved. In cases where multiple reports evaluated the same participants or overlapped, data were primarily taken from the article with a larger sample, while other articles were scrutinized for supplementary insights not covered in the main report.
Assessment of CARAT’s measurement properties was undertaken by RJV and ACF, following the COSMIN methodology. This evaluation consisted of three steps: methodological quality assessment of primary studies, an overall rating of CARAT’s properties, and an evaluation of the generated evidence quality. The methodological quality pertained to the risk of bias evaluation in the included studies (inclusive of those related to CARAT’s development). This was assessed based on items such as statistical procedures, sampling, and study size, rated from ‘very good (V)’ to ‘inadequate (I)’ using the COSMIN risk of bias checklist. The lowest rating among the items for each property determined the overall rating. The overall rating relied on quantifying the psychometric properties against predetermined criteria for adequacy. Ratings of ‘sufficient (+)’, ‘insufficient (-)’, or ‘indeterminate (?)’ were assigned based on quantitative results for each property.
Specific criteria guided content validity assessment, drawing from the COSMIN methodology’s recommendations. Structural validity required a Root Mean Square Error of Approximation <0.06 or Standardized Root Mean Residuals <0.08 for a ‘sufficient (+)’ rating. Internal consistency necessitated low evidence for structural validity and a Cronbach’s alpha ≥0.70. Likewise, reliability required an intraclass correlation coefficient (ICC) of at least 0.70. For construct validity and responsiveness hypothesis testing, correlations with instruments measuring akin constructs were expected to be ≥0.50 or 0.30–0.50 for related but disparate constructs, or an area under the receiver operating characteristic (ROC) curve ≥0.70. Results were summarized per property with an overall rating of ‘sufficient’ (+) or ‘insufficient’ (-) if concurrence exceeded 75%, an ‘inconsistent’ (+/-) rating if no single rating surpassed 75% without an explanatory rationale, and an ‘indeterminate’ (?) rating if all study results were inconclusive.
Lastly, the evidence quality was evaluated employing the Grading of Recommendations Assessment, Development and Evaluation (GRADE) approach, considering methodological quality, result inconsistency, imprecision, and indirectness. This comprehensive process adheres to rigorous standards for objective evaluation.
Inclusion Criteria
The study encompassed original research articles that met specific criteria. The criteria included studies involving adolescents (12 years and older) and adults with asthma and/or allergic rhinitis (AR). The research needed to either focus on the development, assessment of properties (like validity, reliability, consistency, and responsiveness), and/or cultural adaptation and validation of the Control of Allergic Rhinitis and Asthma Test (CARAT), or employ CARAT alongside other patient-reported outcome measures (PROMs) as an endpoint. Additionally, the studies needed to utilize CARAT to evaluate the control of asthma and/or AR within a 4-week recall period. Notably, conference abstracts were excluded in line with COSMIN recommendations, and no constraints were placed on publication date or language.
Following the removal of duplicates, articles were subjected to a dual screening process. Authors RJV and CJ independently assessed titles and abstracts, while full texts of non-excluded articles were independently reviewed by authors RJV and ACF. If articles were inaccessible, attempts were made to contact the researchers. For articles in languages unfamiliar to the reviewers, translation to English was facilitated either by native speakers of that language or through online translation tools. In the event of disagreements, consensus was reached through discussion between the authors. This meticulous selection process upheld rigorous standards and ensured the thoroughness of the study’s approach.
Statistical Analysis
In order to quantitatively consolidate evidence pertaining to the internal consistency, reliability, construct validity, and responsiveness of CARAT and its subscales (‘CARAT upper airway’ and ‘CARAT lower airway’), the study conducted meta-analyses for key parameters. Cronbach’s alphas (indicative of internal consistency), intraclass correlation coefficients (ICC; indicating reliability), and Spearman correlation coefficients (representing construct validity and responsiveness) were analyzed. However, due to an insufficient number of included primary studies addressing other properties (e.g., measurement error), meta-analysis for those properties could not be performed.
A random-effects model, employing the restricted maximum likelihood method, was used for the meta-analyses. Given that the original studies didn’t provide confidence intervals or standard errors with their effect size measurements, certain transformations were applied. For instance, Spearman correlation coefficients and ICCs underwent initial transformations before back-transforming results into the original scale. The estimation of variances was facilitated through calculated confidence interval limits for Cronbach alphas.
Heterogeneity, a measure of result diversity, was assessed using the I² statistic and the p-value for the Q-Cochran statistic. An I² exceeding 50% and a p-value below 0.10 indicated substantial heterogeneity. Sensitivity analyses were conducted for both asthma and non-asthma patient groups when applicable. Additionally, to ensure the inclusion of studies with comparable methodologies, the main meta-analytical outcomes didn’t incorporate mHealth data when outcomes were assessed through different data retrieval strategies.
All analyses were executed using the software R (version 4.0), specifically employing the metafor package. This approach aimed to consolidate data and provide a comprehensive understanding of CARAT’s psychometric properties in a methodologically sound manner.
Result
Study selection involved a rigorous process. Initially, 283 search results were obtained from the database query. After eliminating duplicates, a total of 136 references underwent assessment based on titles and abstracts, leading to the complete reading of 48 articles. Moreover, 216 distinct articles were identified and screened via Google Scholar’s reference search, with 30 receiving comprehensive scrutiny. In total, 16 original studies, which encompassed 23 reports, were included in the systematic review.
Diverse versions of CARAT were employed across the studies. The Portuguese version featured in 8 studies (11 reports), while the Italian, German, and Dutch versions were assessed in 2 studies (3 reports). The Turkish version was evaluated in a single study, and a notable study, published across 3 reports, employed data from a mobile app, involving patients from 25 countries with CARAT available in various languages. The participant pool consisted of 4467 individuals, with ages ranging from 15 to 55 years. A subset of four studies included 2622 participants with AR, while six studies encompassed 1245 participants with asthma. Moreover, 508 participants were diagnosed with both AR and asthma.
Methodological quality ratings varied for each psychometric property in individual studies, yet an overall low risk of bias was observed. The quality rating for PROM development for CARAT was based on its development study, spanning from ‘adequate’ to ‘very good’ for general design requirements. Content validity was rated ‘sufficient’ albeit with very low evidence due to a lack of independent studies assessing this aspect. Structural validity, assessed in three studies, affirmed CARAT’s two-factorial scale structure.
Internal consistency was assessed across seven studies (8 reports), demonstrating good consistency (meta-analytical Cronbach alpha = 0.83; 95% CI = 0.80; 0.86), while subscales exhibited similar patterns. Reliability was rated ‘adequate’ with an ICC of 0.70 or higher in all three studies.
Construct validity, examined in 14 studies (20 reports), indicated ‘sufficient’ evidence in 92% of cases. Meta-analytical Spearman coefficients showcased strong correlations. Responsiveness was adequately addressed in three studies, demonstrating high responsiveness across various outcomes.
Cross-cultural validity studies exhibited consistent properties across four countries where CARAT was validated, aside from some responsiveness heterogeneity. Furthermore, CARAT found application in 57 clinical studies from 15 countries and was integrated into a mobile app (MASK-air®) available in 27 countries.
Missing item percentages were minimal (0% to 9.7%) across studies, while the ceiling score ranged from 2.6% to 8.7%. With its concise format comprising 10 questions, taking under three minutes to complete, CARAT offers a practical tool for assessment without requiring prior authorization for clinical use.
Conclusion
This comprehensive systematic review adhered to COSMIN guidelines, presenting the first-ever evaluation of measurement properties for asthma and/or allergic rhinitis (AR) using the Control of Allergic Rhinitis and Asthma Test (CARAT). Despite the heterogeneous nature of the studies, the analysis revealed CARAT’s robustness in terms of internal consistency, reliability, construct validity, and responsiveness. This confirms CARAT’s suitability for assessing asthma and AR control over a 4-week period.
CARAT’s origin lies in its Portuguese version, developed with physician and patient input to ensure relevance. While its concept elicitation phase was rated ‘doubtful’ due to non-verbatim transcription, other aspects adhered to COSMIN recommendations. Notably, content validity, crucial in capturing a construct’s essence, was deemed ‘sufficient’ based on the development study. Though primary studies lacked results on cross-cultural validity assessment, consistent global adoption suggests its successful implementation.
The questionnaire’s structural validity, fundamental for internal consistency evaluation, aligned with a two-factor scale structure confirmed in three studies. While heterogeneity across studies limited the determination of structural validity quality, meta-analyses demonstrated CARAT’s strong internal consistency.
Construct validity, validated by comparisons with other established PROMs, showed promising correlations with minimal heterogeneity. CARAT’s ability to measure related but distinct constructs was evidenced by its lower correlation with EQ-5D VAS.
Reliability and responsiveness assessment demonstrated CARAT’s reliability and effectiveness, despite some heterogeneity. Cross-cultural validation highlighted consistent results across diverse countries. CARAT’s performance paralleled that of comparable PROMs like ACT and ACQ, solidifying its validity for clinical use.
Despite its strengths, this review faced limitations arising from primary study data gaps, and the diverse participant populations and data collection methods. While providing comprehensive insight into CARAT, further research is needed, particularly concerning its application in patients solely diagnosed with asthma or AR. Additionally, a need for similar evaluations of other PROMs in the field was identified.
In summary, this systematic review, adhering to rigorous guidelines, establishes CARAT’s measurement properties for asthma and AR control. The synthesized evidence affirms CARAT’s credibility, yet further exploration is warranted to refine its utility in varying patient scenarios.
Oncology Related Tools
Other
Latest Research
