Introduction

The AGREE-collaboration has developed an instrument for the appraisal of clinical guidelines. By filling-in the instrument, a global appraisal of guidelines is obtained. This is because the instrument is focused on the guideline as a whole, not on the separate individual recommendations.

There is a risk that a global appraisal does not adequately reflect the quality of the individual recommendations of the guideline. For example, while a guideline may generally be built on strong evidence, certain important individual recommendations may rest on weak evidence only. In such cases, appraisal of the individual recommendations, instead of the guideline as a whole, may improve the validity of the appraisal.

The project 'Appraisal of Individual Recommendations of Clinical Guidelines' can be seen as a validation study of the global AGREE-instrument (version May 1999), leading to suggestions for further improvement of the validity of this instrument. An article is in preparation and a poster will be presented in March 2001 during the European Forum on Quality in Health Care in Bologna, Italy.

Objectives

The key objectives of this project were;

1. To develop appraisal criteria for individual recommendations reflecting the dimensions of the AGREE global appraisal instrument
2. To appraise guidelines using the appraisal criteria for individual recommendations and the global AGREE appraisal instrument
3. To explore the relationship between the quality of individual recommendations and the global quality of the guideline
4. To identify additional resources required for appraising recommendations
5. To determine whether the benefits of appraising recommendations justify additional resources needed compared to global appraisal.

Methods

Seven countries planned to appraise the individual recommendations of four guidelines, which had already been appraised using the global AGREE-instrument. Appraisers had to fulfil certain criteria. For the appraisal of individual recommendations, an instrument based on the global instrument and it's first version of dimensions was developed. Scores from both types of appraisal were compared. If there was a considerable difference between scores, a member of the AGREE-collaboration would look for an explanation.

Preliminary conclusions

Sixteen appraisals have been completed by now. Additional appraisals are underway. Our preliminary conclusions from this material are:

- There was a significant relationship between the scores derived by the two ways of appraising clinical guidelines.
- The global AGREE-instrument consists of 22 questions, each leading to a score. About 20% of these scores differed from the score derived by appraising individual recommendations.
- The global instrument nearly always gives a higher (=more positive) score for a guideline than if recommendations were appraised individually. On average the score of the global instrument was 20% higher than the score of the appraisal of individual recommendations.
- The questions with the greatest differences between scores were:
           - The recommendations are specific and unambiguous?
           - The guideline is supported by tools for application?
           - Review criteria have been provided with the guideline to assess the adherence to the guideline?
- Whenever there was a difference between scores of the two instruments, the score from individual appraisals was more often regarded accurate.
- The format of the instrument to appraise individual recommendations was not considered helpful by most appraisers.
- Finding the causes for the differences between scores of the two instruments has lead to a series of suggestions for improvement of the global AGREE-instrument. Examples are:
          - In the introduction of the AGREE-instrument you must warn appraisers about how easily they can get a too positive impression of a guideline, when an item (for example evidence) is sometimes present.
          - You have to explain what is meant by review criteria. Some appraisers think that a minimal data set is the same as review criteria, which is not so.

In order to prevent the causes which have led to the score differences, the wording of the global AGREE-instrument has been changed. Version August 2000 has been developed and is currently being validated.

Preparation group

P. ten Have, N. Klazinga, H. Varonen, J. Grimshaw, M. Makela, B. Burnand, G. Ollenschläger, on behalf of the AGREE Collaboration.