APPRAISAL OF GUIDELINES FOR RESEARCH & EVALUATION
INTRODUCTION
- Purpose of the AGREE II Instrument
Clinical practice guidelines (‘guidelines’) are systematically developed statements to assist practitioner and patient decisions about appropriate health care for specific clinical circumstances (1). In addition, guidelines can play an important role in health policy formation (2,3) and have evolved to cover topics across the health care continuum (e.g., health promotion, screening, diagnosis).
The potential benefits of guidelines are only as good as the quality of the guidelines themselves. Appropriate methodologies and rigorous strategies in the guideline development process are important for the successful implementation of the resulting recommendations (4-6). The quality of guidelines can be extremely variable and some often fall short of basic standards (7-9).
The Appraisal of Guidelines for REsearch & Evaluation (AGREE) Instrument (10) was developed to address the issue of variability in guideline quality. To that end, the AGREE instrument is a tool that assesses the methodological rigour and transparency in which a guideline is developed. The original AGREE instrument has been refined, which has resulted in the new AGREE II and includes a new User’s Manual (11).
The purpose of the AGREE II, is to provide a framework to:
- assess the quality of guidelines;
- provide a methodological strategy for the development of guidelines; and
- inform what information and how information ought to be reported in guidelines.
The AGREE II replaces the original instrument as the preferred tool and can be used as part of an overall quality mandate aimed to improve health care.
- ii) History of the AGREE Project
The original AGREE Instrument was published in 2003 by a group of international guideline developers and researchers, the AGREE Collaboration (10) . The objective of the Collaboration was to develop a tool to assess the quality of guidelines. The AGREE Collaboration defined quality of guidelines as the confidence that the potential biases of guideline development have been addressed adequately and that the recommendations are both internally and externally valid, and are feasible for practice (10). The assessment includes judgments about the methods used for developing the guidelines, the components of the final recommendations, and the factors that are linked to their uptake. The result of the Collaboration’s effort was the original AGREE Instrument, a 23-item tool comprising 6 quality domains. The AGREE Instrument has been translated into many languages, has been cited in well over 100 publications, and is endorsed by several health care organizations. More details about the original instrument and related publications are available on the Web site of the AGREE Research Trust (http://www.agreetrust.org/), the official body managing the interests of the AGREE Instrument.
As with any new assessment tool, it was recognized that ongoing development was required to strengthen the measurement properties of the instrument and to ensure its usability and feasibility among intended users. This led several members of the original team to form the AGREE Next Steps Consortium (Consortium). The objectives of the Consortium were to further improve the measurement properties of the instrument, including its reliability and validity; to refine the instrument’s items to better meet the needs of the intended users; and to improve the supporting documentation (i.e., original training manual and user’s guide) to facilitate the ability of users to implement the instrument with confidence.
The result of these efforts is the AGREE II, which is comprised of the new User’s Manual and 23 item tool organized into the same six domains, described here. The User’s Manual is a significant modification of the original training manual and user’s guide and provides explicit information for each of the 23 items. Table 1 compares the items of the original AGREE to the items in the AGREE II.
Table 1. Comparison of original AGREE and AGREE II items.
| Original AGREE Item | AGREE II Item | |
| Domain 1. Scope and Purpose | ||
| 1. | The overall objective(s) of the guideline is (are) | No change |
| specifically described. | ||
| 2. | The clinical question(s) covered by the guideline is (are) | The health question(s) covered by the guideline is (are) |
| specifically described. | specifically described. | |
| 3. | The patients to whom the guideline is meant to apply | The population (patients, public, etc.) to whom the guideline is |
| are specifically described. | meant to apply is specifically described. | |
| Domain 2. Stakeholder Involvement | ||
| 4. | The guideline development group includes individuals | No change |
| from all the relevant professional groups. | ||
| 5. | The patients’ views and preferences have been sought. | The views and preferences of the target population (patients, |
| public, etc.) have been sought. | ||
| 6. | The target users of the guideline are clearly defined. | No change |
| 7. | The guideline has been piloted among end users. | Delete item. Incorporated into user guide description of item |
| 19. | ||
| Domain 3. Rigour of Development | ||
| 8. | Systematic methods were used to search for evidence. | No change in item. Renumber to 7. |
| 9. | The criteria for selecting the evidence are clearly | No change in item. Renumber to 8. |
| described. | ||
| NEW Item 9. The strengths and limitations of the body of | ||
| evidence are clearly described. | ||
| 10. | The methods for formulating the recommendations are | No change |
| clearly described. | ||
| 11. | The health benefits, side effects, and risks have been | No change |
| considered in formulating the recommendations. | ||
2
| Original AGREE Item | AGREE II Item | |
| 12. | There is an explicit link between the recommendations | No change |
| and the supporting evidence. | ||
| 13. | The guideline has been externally reviewed by experts | No change |
| prior to its publication. | ||
| 14. | A procedure for updating the guideline is provided. | No change |
| Domain 4. Clarity of Presentation | ||
| 15. | The recommendations are specific and unambiguous. | No change |
| 16. | The different options for management of the condition | The different options for management of the condition or health |
| are clearly presented. | issue are clearly presented. | |
| 17. | Key recommendations are easily identifiable. | No change |
| Domain 5. Applicability | ||
| 18. | The guideline is supported with tools for application. | The guideline provides advice and/or tools on how the |
| recommendations can be put into practice. | ||
| AND Change in domain (from Clarity of Presentation) AND | ||
| renumber to 19 | ||
| 19. | The potential organizational barriers in applying the | The guideline describes facilitators and barriers to its |
| recommendations have been discussed. | application. | |
| AND change in order – renumber to 18 | ||
| 20. | The potential cost implications of applying the | The potential resource implications of applying the |
| recommendations have been considered. | ||
| 21. | The guideline presents key review criteria for monitoring | The guideline presents monitoring and/ or auditing criteria. |
| and/ or audit purposes. | ||
| Domain 6. Editorial Independence | ||
| 22. | The guideline is editorially independent from the funding | The views of the funding body have not influenced the content |
| body. | of the guideline. | |
| 23. | Conflicts of interest of guideline development members | Competing interests of guideline development group members |
| have been recorded. | have been recorded and addressed. | |
APPLYING THE AGREE II
- Which guidelines can be appraised with the AGREE II?
As with the original instrument, AGREE II is designed to assess guidelines developed by local, regional, national or international groups or affiliated governmental organizations. These include original versions of and updates of existing guidelines.
The AGREE II is generic and can be applied to guidelines in any disease area targeting any step in the health care continuum, including those for health promotion, public health, screening, diagnosis, treatment or interventions. It is suitable for guidelines presented in paper or electronic format. At this stage, the AGREE II has not been designed to assess the quality of guidance documents that address health care organizational issues. Its role in the assessment of health technology assessments has not yet been formally evaluated.
- ii) Who can use the AGREE II?
The AGREE II is intended to be used by the following stakeholder groups:
- by health care providers who wish to undertake their own assessment of a guideline before adopting its recommendations into their practice;
- by guideline developers to follow a structured and rigorous development methodology, to conduct an internal assessment to ensure that their guidelines are sound, or to evaluate guidelines from other groups for potential adaptation to their own context;
- by policy makers to help them decide which guidelines could be recommended for use in practice or to inform policy decisions; and
- by educators to help enhance critical appraisal skills amongst health professionals and to teach core competencies in guideline development and reporting.
- KEY RESOURCES AND REFERENCES
- i) AGREE Research Trust
The AGREE Research Trust (ART) is an independent body established in 2004 at the conclusion of the activities of the original AGREE Collaboration. ART endorses the AGREE II and manages the interests of the AGREE enterprise, supports a research agenda regarding its development, and serves as the holder of its copyright.
The AGREE Research Trust web site http://www.agreetrust.org provides:
- free downloadable copies of AGREE II
- links to the AGREE II on-line training tool
- reference lists citing AGREE II and the original AGREE Instrument
- free downloadable copies of the original AGREE Instrument
- information about AGREE projects, the AGREE Next Steps Consortium and the original AGREE Collaboration
- ii) How to cite the AGREE II
AGREE Next Steps Consortium (2009). The AGREE II Instrument [Electronic version].
Retrieved <Month, Day, Year>, from http://www.agreetrust.org .
iii) AGREE II On-Line Training Tool
For access to the AGREE II On-Line Training Tool, please visit http://www.agreetrust.org .
- iv) References related to the AGREE II
AGREE II: Advancing guideline development, reporting and evaluation in healthcare. Parallel publications in progress
- v) Primary reference related to the original AGREE Instrument
AGREE Collaboration. Development and validation of an international appraisal instrument for assessing the quality of clinical practice guidelines: the AGREE project. Qual Saf Health Care. 2003 Feb;12(1):18-23.
REFERENCES
- Woolf SH, Grol R, Hutchinson A, Eccles M, Grimshaw J. Clinical guidelines: potential benefits, limitations, and harms of clinical guidelines. 1999;318(7182):527-530.
- Committee to Advise the Public Health Service on Clinical Practice Guidelines IoM. Clinical practice guidelines: directions for a new program. Washington: National Academy Press;
- Browman GP, Snider A, Ellis P. Negotiating for change. The healthcare manager as catalyst for evidence-based practice: changing the healthcare environment and sharing experience. Healthc Pap. 2003;3(3):10-22.
- Grol R. Success and failures in the implementation of evidence-based guidelines for clinical practice. Med Care. 2001;39(8 Suppl 2):1146-54.
- Davis DA, Taylor-Vaisey A. Translating guidelines into practice: a systematic review of theoretic concepts, practice experience and research evidence in the adoption of clinical practice guidelines. CMAJ. 1997;157(4):408-16.
- Grimshaw J,.Russell I. Effect of clinical guidelines on medical practice: a systematic review of rigorous evaluations. Lancet. 1993;342:1317-22.
- Shaneyfelt TM, Mayo-Smith MF Rothwangl J. Are guidelines following guidelines? The methodological quality of clinical practice guidelines in the peer-reviewed medical literature. JAMA 1999:281(20):1900-5.
- Grilli R, Magrini N, Penna A, Mura G, Liberati A. Practice guidelines developed by specialty societies: the need for critical appraisal. 2000;355:103-6.
- Burgers JS, Fervers B, Haugh M, Brouwers M, Browman G, Phillip T, Cluzeau FA. International assessment of the quality of clinical practice guidelines in oncology using the Appraisal of Guidelines and Research and Evaluation Instrument. J Clin Oncol. 2004;22:2000-7.
- AGREE Collaboration. Development and validation of an international appraisal instrument for assessing the quality of clinical practice guidelines: the AGREE project. Qual Saf Health Care. 2003 Feb;12(1):18-23.
- AGREE II: Advancing the guideline development, reporting and evaluation in healthcare. Parallel publications in progress.
UPDATE: September 2013
In 2013, the AGREE marked its 10th anniversary since the original AGREE Instrument was first published and made available for use. To mark this anniversary, we provide a summary of
activities that have taken place over the past 10 years and an update to the references originally noted in the AGREE II 2009 version.
