gms | German Medical Science

Background: The concepts of Clinical Research and Health Services Research are well known. Clinical Research investigates and describes effects under Experimental Study Conditions (ESC) while Health Services Research investigates and describes effects under Real World Conditions (RWC). The effects under ESC are known as efficacy and the effects under RWC are known as effectiveness.

There is a consensus on the appropriate tool for assessment of efficacy, but not for assessment of effectiveness. The most likely reason for the lack of the later is the focused perspective of many researchers on randomization only. The 2008 version of the CONSORT statement recommended RCTs for completion of both explanatory and pragmatic studies (1). However, RCT is not a good match for pragmatic studies since, by definition, the allocation of treatment in RCT must be at random (which doesn't happen in the real world). This sole focus on RCTs would exclude all pragmatic studies that are essential for health services research.

Aim of study: In this study we aim to investigate the distinction between efficacy and effectiveness in the medical literature and to demonstrate the impact on the establishment of guidelines when the distinction is not clear.

Methods: To achieve our aims we completed three projects and a conscientious analysis of the 2008 CONSORT Statement (1).

In the first project we analyzed five corresponding pairs of terms recommended in six reviews on the reporting on outcomes of clinical trials (1-6). These five pairs were

1.

efficacy/effectiveness outcome,

2.

experimental/observational trial,

3.

explanatory/pragmatic attitude,

4.

randomized/non-randomized allocation of intervention, and

5.

analytical/descriptive trial.

In the second project we analyzed four corresponding triplets of terms used in 100 published quality of life (QoL) studies (7). The four triplets of terms were

1.

explanatory/pragmatic/not defined attitude,

2.

randomized/non-randomized/not defined allocation of intervention,

3.

primary/ secondary/not defined endpoint, and

4.

efficacy/effectiveness/not defined outcome.

In the third project we analyzed the congruence of recommendations from 24 treatment guidelines (of eleven countries on three types of cancer) in 330 scenarios. A recommendation was considered congruent (or not defined) when at least 66% of corresponding recommendations of all countries were congruent (or were not defined) (8).

Results: The first project demonstrated that none of the six reviews included all six pairs of terms. The pairs explanatory/pragmatic, and randomized/non-randomized were included in five of six reviews (1,2,3,5,6). The pair efficacy/effectiveness was included in four of six reviews (1,2,3,6). The two pairs experimental/observational and analytical/descriptive were included in two reviews (4,5). The review by Schwartz & Lellouch (2) included only the pair of explanatory/pragmatic and discussed efficacy but not effectiveness. The 2008 CONSORT statement (1) included the pairs of explanatory/ pragmatic, randomized/not randomized, and efficacy/effectiveness but did not discuss the meaning of efficacy/effectiveness. Each of the three remaining reviews (3,4,6) included three pairs of terms, but not the same pairs.

The results of the second project showed that an experimental study design (RCT) was used in 91% of the investigated 100 QoL studies implying that the effect was investigated mainly under experimental conditions, which is not a good fit for QoL studies. QoL was the primary endpoint in only 29% of the trials. Furthermore, 60% of the studies did not state whether they were assessing efficacy or effectiveness. Only 14% of the studies claimed to assess efficacy and 25% to assess effectiveness. With an endpoint of QoL, however, most clinicians would agree that effectiveness is more important than efficacy in a study on the effects of different treatments. Only 6% of the studies were classified as pragmatic trials (9).

The third project revealed that 15% of the corresponding guidelines were congruent, 60% were incongruent and 25% were not defined (8).

Discussion: Our results confirm the problem in describing efficacy and effectiveness in clinical studies and review articles. This lack of congruence is likely to confound the authors of clinical guidelines. The CONSORT recommendations utilize the term "randomized pragmatic studies". This decision is not supported by appropriate evidence and is unlikely to promote health services research because health services research depends on pragmatic rather than randomized trials. In view of this an update of CONSORT has to be considered.

Practical implications: We propose two solutions. First, a consensus on the clear definition of the structural and functional criteria of efficacy and effectiveness should be reached. Second, new proposals for the assessment of effectiveness under real world conditions such as Pragmatic Controlled Trials (PCTs) should be discussed (10,11).