What is wrong with QALYs?

A paper by Rand and Kesselheim (2021) in Health Affairs this month conducts a systematic literature review to answer this question. Based on 113 articles they identified in peer-reviewed journals, they identify the following 10 criticisms categories.

The graph above has each criticism category and the number of peer-reviewed articles that mention this critique type. Each of these criticism categories are subdivided into specific critiques:

Criticism category Specific criticism
Health state measurement Validity of tools to measure health state;
Reliability among tools;
Difficulty of self-reporting for some groups; (e.g., children, dementia);
Insensitivity to specific conditions or changes in health;
Prescriptive definition of health-related quality of life (excludes some health domains and well-being beyond health);
Utility Measurement Validity of utility measurement;
Reliability between utility measures: different methods produce different results;
Elicited utilities are changed by the perspective the question or responder takes, including whether ill health happens now or later & risk-aversion;
Utility scores elicited from the public are different from those of patients who experience the health state;
Utility values may change over the life course or not be fixed;
Individual preferences are not compatible with the utility theory of QALYs: People do not hold time and quality of life in a constant proportional, linear trade-off;
Measures utilities per health state rather than utility of moving between health states;
Death should not be treated as a quality-of-life utility; it is a time measurement
Unclear how to value certain conditions or outcomes, such as stillbirth
Nonhealth effects Social utility and externalities are not measured;
Effects on dependents and benefits to others not measured;
Scientific spillover (future innovation or learning) not measured
Equal value of QALY regardlesss of recepients Neutrality overall: equal value to the QALY irrespective of other characteristics of the recipients; QALYs ignore distribution of health
Severity of the condition is not accounted for in QALYs (that is, starting health state not reflected in QALY gain);
QALYs ignore endpoint health state (whether treatment results in a poor health state or whether a better state would have been possible);
QALY gains reflect capacity to benefit and magnitude rather than need or past ill health;
Aggregation of outcomes QALYs in total counted, not individual lives; assumes that gains can compensate for losses;
Does not draw a distinction between saving or extending lives and improving quality of life;
Ignores whether a health state was improved or decline prevented;
Aggregation of individual preferences Individuals’ values & preferences are different, but they are assumed to value QALYs equally;
Utilities should not be interpersonally compared and aggregated;
Individuals’ preferences should not apply to others;
Effects for individuals may be different from the average (outliers);
QALYs are insensitive to health distribution, do not account for equity or fairness;
QALY neutrality does not reflect public preferences about health equity
Ageism Bias by age because remaining life expectancy means younger populations can produce more QALYs;
QALYs should weight health gains for younger populations to achieve equity in life expectancy;
condition discrimination
A disability or chronic condition reduces quality of life and therefore potential to gain QALYs;
People with disabilities are often unwilling to trade off time for quality of life, leading to high utility scores for their health state;

To find out more about each of these criticisms, do read the whole paper here.