Appraisal checklist — the eleven principles

Back to the workshop

Use this on any published economic evaluation. Work down the list, write what the paper actually did in the fourth column, and mark the last. A paper that fails a principle is not necessarily a bad paper, but the failure should change how much weight you give its conclusion, and you should be able to say how.

Derived from Sharma D, Prinja S, Aggarwal AK, Rajsekar K, Bahuguna P. Development of the Indian Reference Case for undertaking economic evaluation for health technology assessment. Lancet Reg Health Southeast Asia 2023;16:100241. PMID 37694178

Paper

Reviewer Date

#	Principle	What India asks for
1	Decision problem	Stated as who does what, to whom, where, when, how often. Policy question, condition, setting, target population, and any subgroups named in advance.
2	Comparator	Current practice in use, from standard treatment guidelines or the most widely used option. Not "do nothing" unless that is genuinely current practice. Dose, frequency, duration and route reported for drugs.
3	Perspective	Abridged societal in the base case: direct medical and direct non-medical costs borne by payer and patient, including out-of-pocket. Healthcare payer perspective reported separately.
4	Source of effectiveness evidence	Systematic review and meta-analysis of RCTs preferred; network meta-analysis if no head-to-head trial. Level of evidence stated. Expert opinion avoided.
5	Measuring costs	NHSCD / CHSI for health-system costs, NSS 75th round for out-of-pocket. Capital costs annuitised. Indirect costs in sensitivity analysis only, by the human capital approach. Intangible costs excluded, being already in the QALY.
6	Health outcomes	Cost–utility analysis with QALYs; EQ-5D-5L with the Indian value set. EQ-5D-Y for 8 to 15 (proxy 4 to 8), adult Indian value set for 4 to 15, DALYs permitted under 4.
7	Time horizon	Long enough to capture all differential costs and effects. Lifetime where mortality or survival is affected. Shorter horizons justified and tested in scenario analysis.
8	Discounting	3% on both costs and outcomes beyond one year. Varied 0 to 5% in sensitivity analysis. Undiscounted results also reported.
9	Heterogeneity	Subgroup analysis, with subgroups identified at the planning stage and the basis justified. Not post hoc.
10	Uncertainty	PSA for all analyses, presented as a CEAC. DSA for key structural assumptions and influential parameters. Scenario analysis including best and worst case.
11	Equity	All lives and QALYs valued equally. Distributional CEA, or extended CEA for financial risk protection, where relevant.
12	Presentation	Disaggregated and aggregated results. Total costs and total outcomes given separately, not only the ratio. Graphs used. Departures from the Reference Case justified.

Eleven principles, twelve recommendations. Principles 1 and 2 share a recommendation in the source document, which is why the numbering runs to 12.

The five questions to answer out loud when you report back

What is the comparator, and is it what actually happens now? If not, every ICER in the paper is flattering.
Whose costs are counted? If out-of-pocket payments are missing in an Indian study, say what that hides.
How long is the time horizon, and does the benefit have time to arrive? Prevention studied over five years is a rigged question.
Is the outcome a QALY or a DALY? You cannot put the two in the same league table.
How was uncertainty handled, and what was the probability of being cost-effective? A point estimate with no PSA is a claim without an error bar.

And then the question that matters

Do you believe the conclusion? Not "is the arithmetic right". Is the decision it recommends one you would defend in front of the people whose care would be displaced to pay for it?