Canada is frequently defined by its healthcare system, which can be seen by Canadians as a source of both pride and frustration. Fans and critics of Canadian healthcare often compare our system to those of other countries in order to emphasize the strengths and weaknesses of our model. Various organizations have attempted to formally evaluate and compare the performance of numerous national healthcare systems. The end product of this type of evaluation, usually in the form of a simple media-friendly table, is an international ranking of healthcare systems. Unfortunately, these Olympics-style rankings are usually plagued by significant flaws that limit their validity and usefulness (1). In this essay, I will discuss international rankings of healthcare systems, and I will argue that fundamental methodological issues annul many of their conclusions. I will then evaluate the usefulness and implications of such rankings before suggesting possible alternatives.
In 2000, the World Health Organization published The World Health Report 2000- Health Systems: Improving Performance, which introduced a framework for evaluating and ranking healthcare with the stated objective of “improving the performance of health systems around the world” (2). The performances of 191 national healthcare systems were quantitatively assessed based on five factors: overall level of health, health disparities, responsiveness to population expectations, distribution of responsiveness, and distribution of financial contribution. Other organizations, such as The Commonwealth Fund (3) and Bloomberg (4) have also made attempts at ranking healthcare systems, but they have not made the same claims of scientific, evidence-based evaluation as the WHO report, nor have they had a similar level of subsequent media exposure and political influence (8). Canada was ranked 30th in the WHO report, a position that has been widely and repeatedly cited, from media outlets, to the general public, to politicians and researchers, as an indication of a flawed healthcare system in need of reform (1).
Despite its global scope and admirable goals, the WHO report attracted a great deal of critical attention. Interestingly, one of the most damning criticisms of the report came from one of its principal authors, Philip Musgrove, who explained that many of the results were calculated using imputed data, and that “only 39% of the indicator values represent real data” (5). Another significant criticism related to the ranking of health system responsiveness and distribution for 191 countries using the judgment of a non-representative sample of key informants from only 35 of these countries (6). Other methodological concerns included ignoring sub-population characteristics and equity when measuring health inequality, and using a questionable definition of ‘fair financing’ that fails to consider the use of services relative to need (6). The face validity of the rankings was also called into question, with Spain ranking high despite ongoing, unprecedented demonstrations against the country’s healthcare authorities, and Canada somehow ranking lower than Colombia in fairness of financing (7).
An argument in favor of ranking healthcare systems by country is that it is an effective way of stimulating analysis, and placing health-system performance higher on political and research agendas. Reform initiatives in the healthcare systems of China, Mexico, and Iran, for example, seem to have been catalyzed by the WHO rankings (9). Nevertheless, publications claiming to provide evidence-based measurements and rankings should do more than simply stimulate further research and exert political influence–they should present a reliable, objective assessment of reality. Otherwise, these publications would more closely represent opinion pieces, or be considered socio-political, rather than scientific literature.
In reality, national healthcare systems do not lend themselves to the broad and simple characterization necessary for an objective comparison between countries. In the first place, it is difficult to even speak of a single national healthcare system in some countries. In Canada, healthcare must follow the five overarching principles of the Canada Health Act (public administration, comprehensiveness, universality, portability, and accessibility), but healthcare systems are otherwise entirely directed by their respective provinces or territories (10). Canada actually has thirteen different healthcare systems. Furthermore, ranking healthcare systems oversimplifies and disregards the context of the system. Health and healthcare do not exist in a vacuum–they influence and are influenced by a variety of country, province, or territory-specific factors, including, but not limited to, culture, economy, politics, and environment. Ranking healthcare systems implies an oversimplified understanding of health policy that amounts to medicalization of a more complex set of problems (11). Without appropriately considering the context of each country’s healthcare system, one might think that the country ranking first overall has a healthcare system that should be adopted by all other countries–a conclusion that is as irrational as it is impossible.
One of the potential consequences of international rankings is increased attention to the need to evaluate individual healthcare systems. Comparing countries and providing provocative rankings of healthcare systems are technically and methodologically contentious methods of evaluation, in addition to being conceptually flawed. Instead of comparing countries, which may contain a multitude of different systems, it makes more sense to compare the actual individual healthcare systems. Furthermore, these comparisons must take into account the particular context and complex environment of the given healthcare system. One possible way of taking complex environments into account is by performing, or adopting certain aspects of a realist synthesis. A realist synthesis of literature focuses on the development and refinement of theories in order to understand the mechanism by which an intervention works, taking into account the context and outcome of the intervention (12). This model of evaluating complex policy interventions “provides an explanatory analysis aimed at discerning what works for whom, in what circumstances, in what respects and how” (13). Although realist syntheses are not simple, they can allow for a deeper understanding of why certain aspects of a healthcare system works in one context, and how they can potentially be adopted to another. Another potential alternative, which may allow for a more valid comparison of healthcare systems and reforms is performing a step-wedge trial, involving the “sequential roll-out of an intervention to participants (individuals or clusters) over a number of time periods” (14). Rather than implementing reforms on a national level, a country can use this method of systematically implementing reform, while also allowing for built-in evaluation of the new system. For example, this type of study proved to be an effective and pragmatic way of implementing and evaluating a community health insurance scheme in Burkina Faso (15). While these suggestions may present problems themselves, they represent a more appropriate and valid method of evaluating healthcare systems, rather than the oversimplified and needlessly competitive notion of comparing and ranking countries.
Evaluating healthcare systems represents a complex, but worthwhile scientific endeavour. Despite admirable intentions, attempts at ranking national healthcare systems by country have been methodologically flawed and conceptually inappropriate. Healthcare systems are too complex and context-specific to be reduced to a simple measure of performance that can be compared and ranked between countries. Evaluation and comparison must reflect these complexities, and a realist synthesis is a more appropriate method of doing so. Alternatively, a step-wedge trial can also help to determine the effectiveness of a particular healthcare reform within a country or region.
As much as Canadians can be proud, or frustrated, by our model of healthcare, we should also be aware that it is not an international competition. Canadian provinces and territories can and should continue to strive to improve their healthcare systems. We should avoid, however, pursuing the appealing, but effectively meaningless, designation of “best in the world.”