Handling uncertainty

By the end of this session

1. Say why an ICER is a point estimate, and name the kinds of uncertainty
2. Tell heterogeneity from uncertainty, and say why they differ
3. Read a tornado diagram and say what it cannot tell you
4. Describe what a probabilistic sensitivity analysis does
5. Read a cost-effectiveness acceptability curve and say how it is built
6. Say what "a 57% probability of being cost-effective" does not mean

Every number was estimated

The ICER inherits all of it, and reports none of it.

₹26,154 per QALY is the answer if every parameter sits at its central value at the same moment. That is one draw from a very large space.

Four things that get muddled

Probabilistic analysis handles the first of these and nothing else. Say so when you report it.

9 Heterogeneity: decide subgroups first

An intervention may be cost-effective in one subgroup and nowhere else. That is a targeting opportunity, not a nuisance.

Identify subgroups at the planning stage, so stratified costs, effects and utilities are collected for each.

Picking subgroups after seeing the results is data dredging with an economics vocabulary. The Reference Case says so plainly. Justify the basis for each subgroup and have a clinician confirm it is sensible.

Heterogeneity and equity are, on published appraisals, among the weakest parts of Indian economic evaluations. Handling them well is a way to stand out.

10 What India actually requires

Only 73% of the Indian evaluations reviewed had done any sensitivity analysis at all.

Sharma et al., 2023. Proportion from Shah et al., Frontiers in Public Health, 2022.

One-way analysis: move one thing at a time

Take a parameter, swing it across its plausible range, hold everything else at its central value, and see what happens to the ICER.

Repeat for every parameter. Cheap, transparent, and it answers the question a decision-maker asks first:

Which number does this conclusion actually depend on?

The range must come from somewhere defensible: a published confidence interval, the spread across facilities in a costing study, the range of prices actually quoted. Not from a round number you liked.

The tornado diagram

One bar per parameter, showing the ICER at its low and high value. Sorted widest at the top, which is where the name comes from.

In the 2011 Indian rotavirus analysis the two parameters that mattered were vaccine price and efficacy against severe disease.

Esposito, Tate, Kang, Parashar, Clinical Infectious Diseases, 2011.

Read the tornado

One model, five uncertain parameters, shown three ways. Tab 1 is the tornado: read the widest bar first, and ask what you would do about that parameter.

Tab 2 comes later. On your phone: sph-aiimsbhopal.github.io/intro-economic-evaluation-health

Why one-way analysis is not enough

• Parameters do not take turns. Several are wrong at once, and the joint effect is not the sum of the separate ones
• Some are correlated. A facility with high staff costs usually has high overheads too
• It cannot give you a probability. A tornado says the ICER could be anywhere from ₹1.05 lakh to ₹4.06 lakh, and nothing about how likely either end is

For a probability you must vary everything at once, thousands of times.

Probabilistic sensitivity analysis

1. Give every uncertain parameter a distribution, not a single value
2. Draw one value from each: a complete, internally coherent set of inputs
3. Run the model once with that set. Record ΔC and ΔE
4. Repeat a thousand times, or ten thousand

The output is not a number. It is a distribution of possible answers.

Which is the honest representation, and the reason the Reference Case wants it for every analysis.

Which distribution, and why

Using a normal distribution for a cost is a real error. It lets the model draw a negative cost.

The result is a cloud, not a point

Plot every simulation on the cost-effectiveness plane. Each dot is one coherent version of the world.

The counting is done on net monetary benefit, not on the ratio.

The cloud, and the curve it generates

Same data as the tornado. Drag λ: the curve's height at any λ is just the share of the cloud below the dashed line.

Re-run to resample the cloud  ·  sph-aiimsbhopal.github.io/intro-economic-evaluation-health

The acceptability curve

Threshold on the horizontal axis. Probability of being cost-effective on the vertical. One point for every value λ could take.

To build it: pick a λ, count the simulations with positive net monetary benefit, plot the proportion. Move λ. Repeat.

P(cost-effective at λ) = proportion of runs with (λ × ΔE_i) − ΔC_i > 0

It saves the decision-maker from committing to a threshold before seeing the answer.

Useful in India now, where the threshold is unsettled. You can say: below ₹1 lakh a poor bet, above ₹3 lakh a good one, in between it depends on a number not yet agreed.

What a curve does not tell you

• Not the probability that the intervention works. Only that it is good value, given the model
• Not structural uncertainty. If the model is wrong, all ten thousand simulations are wrong together
• Not whether you can afford it. That is budget impact, a separate calculation
• Not a decision. A curve near 50% at every plausible threshold is a call for more research

And an 80% probability of being cost-effective still means a one-in-five chance you are wrong.

The Indian example worth remembering

The trastuzumab table from Session 5, with the column that was missing then:

Same drug, same country, same year.

Nine weeks of trastuzumab was cost-effective, and the authors recommended including it in guidelines. A point estimate alone would have hidden all of this.

Gupta, Verma, Gupta, Prinja, JCO Global Oncology, 2020.

Two more to know the names of

Naming the tool you did not use is better than pretending it does not exist.

11 Equity and 12 presentation

In 2023 the Reference Case had to say no recognised threshold for India existed. Three years later, Chugh and colleagues published one.

You are learning this subject at the moment it is being settled.

Recap

• An ICER is a point estimate that inherits every input's uncertainty
• Heterogeneity is not uncertainty. It gets subgroup analysis, planned in advance
• One-way analysis shows what the answer turns on. It cannot give a probability
• Probabilistic analysis gives every parameter a distribution. India asks for it in all analyses
• The curve is just the cloud counted at every threshold, using net monetary benefit

Where you started, and where you are

Take home: appraise one Indian evaluation against the eleven principles. Bring the flaw you found.

Sources

1. Drummond MF, Sculpher MJ, Claxton K, Stoddart GL, Torrance GW. Methods for the Economic Evaluation of Health Care Programmes. 4th ed. Oxford University Press; 2015.
2. Sharma D, Prinja S, Aggarwal AK, Rajsekar K, Bahuguna P. Development of the Indian Reference Case for undertaking economic evaluation for health technology assessment. Lancet Reg Health Southeast Asia 2023;16:100241. PMID 37694178
3. Gupta N, Verma RK, Gupta S, Prinja S. Cost effectiveness of trastuzumab for management of breast cancer in India. JCO Glob Oncol 2020;6:205–16. PMID 32045547
4. Esposito DH, Tate JE, Kang G, Parashar UD. Projected impact and cost-effectiveness of a rotavirus vaccination program in India, 2008. Clin Infect Dis 2011;52(2):171–7. PMID 21288839
5. Shah K, Singh M, Kotwani P, et al. Comprehensive league table of cost-utility ratios for health policy decisions in India. Front Public Health 2022;10:831254. PMID 36311623
6. Chugh Y, Jyani G, Kar SS, Rajsekar K, Prinja S, et al. Estimating the willingness-to-pay per quality-adjusted life-year to aid health technology assessment in India. Value Health 2026;29(3):457–66. PMID 41176029

Parameters in the uncertainty widget are illustrative, chosen to show the mechanism. The trastuzumab figures above are the real Indian example.