In 2020, I was asked by a dear colleague to facilitate the second in a three-part series of Listen Events organized by the BC SUPPORT Unit Health Economics and Simulation Modelling Methods Cluster. The purpose of these events, which took place in spring and summer 2021 on Zoom, is to provide an opportunity for individuals and groups whose voices are not always heard to speak with health economists, on their terms, about the issues they want to raise. The first event comprised discussions with leaders from diverse Indigenous communities, the second with members of the UBC Social Justice Institute; the third event will be taking place on June 29, 2021, and involve community stakeholders representing Black-led non-profits and social justice groups as well as members of the Black community of BC at large. In this blog post, I share three reflections I, Taq Kaur Bhandal, learned as a participant and witness at the first event and facilitator of the second event (with members of the UBC Social Justice Institute).

I can name here that I currently live on Mi’kmaq Treaty Territories in Halifax, Canada, where I continue to teach and write on the topics of social justice, sustainability, pelvic health, and mental health & addictions. Mere poorvaj tan tubar (my ancestors and family) are from the Punjab region of modern day India and Pakistan. We came to the current borders of Canada primarily due to ongoing persecution of Sikh religious minorities in our homelands. Today, I write this blog post while holding the tension of being a settler, displaced person, and diaspora. My reflections below come from this body of experience.

Reflection # 1: We can listen for guidance.

In the Listen Event I facilitated, I noticed how I was able to open up and hold space for multiple perspectives and answers to questions about social justice. Through the process of listening (not just waiting to speak), my body shifted and I found myself releasing limiting beliefs about what is possible in the field of health economics. In this way, I can reflect that when we listen (and tune out the static of our defensive ego), we can hear the guidance on how to engage in research, teaching, and everyday life with social justice in mind.

Reflection # 2: Working together across university departments is no longer optional. It’s essential.

The Listen Events team met through shared connections at the University of British Columbia and Simon Fraser University. Both institutions are located in Metro Vancouver, Canada on the unceded territories of the xʷməθkʷəy̓əm (Musqueam), Sḵwx̱wú7mesh Úxwumixw (Squamish), səl̓ilw̓ ətaʔɬ (Tsleil-Waututh), q̓íc̓əy̓ (Katzie), kʷikʷəƛ̓əm (Kwikwetlem), Qayqayt, Kwantlen, Semiahmoo and Tsawwassen peoples. By acknowledging the land and First Peoples, we can make conscious the reality of Canadian universities as places where social and environmental injustices are produced.

At the same time, through my own academic and personal life, I’ve discovered that universities can also be places where we can train ourselves to actually unlearn harm, violence, scarcity thinking and re-learn social justice, abundance, and healthy relationships with land. In the case of health economics, the Listen Events taught me that this can happen through an awareness & acceptance of our own positionalities (social identity in the world). Moreover, through taking radical action towards creating solidarities and alliances across departmental “borders”.

Reflection # 3: We can circulate university funding with intention.

Finally, from my standpoint, the Listen Events also provided an example of how academic funding can be circulated intentionally towards social justice. We made sure facilitators and participants were paid generously for time. As a facilitator, I also inquired and encouraged the group to name self-care (listening & caring for our bodies/minds/souls). I understand this as a reflection of how there is always enough money to create the futures/worlds/health/healthy economies/health care systems we know are theoretically possible.

In conclusion, the Listen Event created space for reflection and personal growth in three ways. First, I found that when I listen deeply and with intention, I can hear guidance on how to engage in research, teaching, and health economics practice with social justice in mind and body. Second, working together across university departments is no longer optional. It’s essential. Third, we can circulate university funding and health care funding in a way that honours the principles and practices of social justice. I truly appreciate you taking the time to read my reflections, and I can sense that something has shifted with you as well. Now I will pass it on to you: What reflections (in body and mind) came up for you while taking in the When We Listen blog post? Feel free to share them with me!

[SH]: So Nick, if I understand correctly, you and your team have developed a tool that helps communicate the results of clinical prediction models. Is that right?

[NB]: That's right. Clinical prediction models are known by various different names: risk models, prognostic models, algorithms, calculators. We've created this open-source software that enables the results of these models to go into a tool that can be used by clinicians and patients in an understandable way.

So much research and money goes into creating these models, but too many do not have the sort of the impact that they should be having. Why? Because this final step of actually informing patients’ decisions doesn't happen. This software is trying to bridge that gap.

Can you tell me more about what a clinical prediction model is?

Prediction models come from a large set of data: collected information about people, and what has happened to those people, such as clinical outcomes. The data that’s collected can vary from simple demographic details, such as age and ethnicity, to complex data such as blood counts and genetic profiles. Researchers take these data and they build models to see how these characteristics relate to outcomes. So you can predict, for a new person, given their characteristics, how likely these outcomes might be to happen to them in the future.

How are these models used in practice?

There are different ways these models are applied. In public health, models are often used to predict future occurrence of a disease­­— so if you have a high likelihood of having the disease, you could use a preventive treatment. For example, a frequently used model calculates the risk of you having a heart attack or stroke in the next 10 years, based on your age and sex and ethnicity and various pieces of information about your other conditions such as having high blood pressure. If you are at high risk of heart attack or stroke, then you might be prescribed a drug like a statin or recommended to change your diet or things like this. That's public health.

In clinical practice, models tend to be used in two different ways. They can be used to decide whether you should have a more invasive procedure to see if you have an underlying disease. Take, for example, lung cancer: the more definitive procedure to determine if you have lung cancer actually has risks in itself, so you don't want to have that procedure unless there is a fairly good chance that you might have lung cancer. So there's a prediction model, that that uses your age and smoking history and other data to determine what your underlying risk of lung cancer is.

Another way models are used is to help determine if you should use a treatment. For example, they can help patients with COPD know if they have a high risk of developing bad outcomes in the future. The results can help patients and clinicians determine what the impact of lifestyle changes, such as stopping smoking, might have on your future risk of these outcomes. Or whether in taking preventative treatment—which has risks itself—the benefits might outweigh those harms.

What’s the relationship between a clinical prediction model and a patient decision aid?

A prediction model tends to end with a ‘risk’, a ‘probability’ of a poor outcome—that is, whether an individual will have a future outcome or event over a certain time period. Sometimes that number, that probability, is all you need. If your probability is 0%, or 100%, it can be pretty simple to know what you do next. But more often than not, this number needs to be contextualized with the options that are available, how the patient feels about those options, the features of those options, and the consequences of those options.

So that's what a decision aid does. It embeds the risks into a tool that actually helps inform the decision that needs to be made. So it's information on the options, what matters to the patient, so that a clinician can make a shared decision with their patient.

For example, if somebody has a 24% chance of developing severe lung disease over the next 10 years, what does that 24% mean? And how does this compare to the risk of side effects from taking preventative treatment? How much does that preventative treatment actually reduce my risk of severe lung disease? Does it go to 0% or does it go to 10% or 20%? Then the treatment itself: is the treatment easy to take? Does it cost? There's always other factors that go into a decision aid.

So to clarify: your tool both helps to present model results and it helps to create decision aids?

The right tool will depend on different contexts. For some contexts, a simple clinical prediction tool that just describes the risk might be sufficient—what’s wanted by both clinicians and patients. But in other contexts, just that risk may not be enough. That's where a decision aid might be more useful. So our tool has a number of different options. It can go from just describing risks, to actually understanding patients’ values, describing all the options, and creating short summaries that can then be used by clinicians in consultations.

Does the software have a name?

Yeah, we call it Decideapp, and you can read more on decideapp.org. You can download it on your own servers or we are hosting it on UBC servers as well.

Does your tool work with any of the clinical prediction models on the Peer Models Network?

Right now we've just created a demonstration of how it can be used with one of the models on the Peer Models Network (the ACCEPT model). That’s just for demonstration, but it could be applied to any of the models on the Peer Models Network, on any clinical prediction model that people are interested in.

That said, there's no cookie cutter approach to these tools. Each one needs to be considered carefully: how it might be used with clinicians, the patients that might be using it, the situation or circumstances they might be using it in. There needs to be some consideration of what needs to go in and how the risk needs to be interpreted. But once you've done that stage, the software can be used to create that tool.

It’s important to know that clinical prediction models can’t “predict the future” for an individual patient. How do you think that should be explained, or affect how models are used in practice?

That's a great example of how to describe risk. Research shows that most clinicians explain risk poorly to patients. It's almost impossible to describe what might happen to you. But it is best to explain risk by what happens to people like you or: “Imagine there's a hundred people like you. This is what our best estimates expect what will happen to those hundred people”. What these prediction models are really doing is looking back. What happened to people like you before might be useful for understanding what happens to you in the future.

Were patients involved in developing the Decideapp?

We've developed nearly 20 different tools with this software so far and in nearly all of them patients have been involved in various ways in the development. And I would really recommend that people who are considering developing their own tool engage patients early and throughout the process.

One example I can think of is our is one of our risk graphics. You may have seen these graphics before. It’s an image of a hundred people and some of the people are illustrated in a different color— to describe the number of people who will have an event, based on probability. Research shows that these graphics are helpful for people to understand risks. But when we worked with patients and showed them this graphic, we still found a good proportion of people didn't understand what that meant. So we developed an animated version of this graphic with patients, which describes what it means for them. We wouldn't have done that without patients being involved in in a development of these tools.

If we paired the Decideapp with models on the Peer Models Network, could patients give input on their experience using them?

Yes! I think all these tools that use the results of models need to always be improved upon. I think we also need to recognize that no one tool will be the best way of describing the model results to all patients. We might actually realize that we need to create different versions of these tools, for people who are more or less numerate, or who have a preference for graphics over writing, for example. So that we create something that is accessible to more people. I think that's where I think getting feedback from patients about their experiences—both good and bad— would be really helpful.

If the sky was the limit, what would the future of this tool look like?

If the sky was the limit, I would ensure that every research project that is developing a clinical prediction model uses a tool like ours. And works with patients from the very start, so that research is has the highest chance of actually being implemented into practice.

Introduction: Overview, Perspective & Sources

Hi everyone, thank you for having me today. My name is Stephanie and I'm a fellow at UBC. My interest is in health economics, but more specifically in talking about health economics with the public and understanding the social and ethical value judgments involved in building health economics models. Hopefully what I have to say will be a useful introduction to the topic of science, policy, and epidemiology. When I talk about ‘models’ today, I'm referring quite generally to the sort of computer models used in the health sciences, so both statistical models and simulation models that have a time component. At one point, I'll talk specifically about decision models in health economics.

First, I'll review some very basic principles of public health, and health economics in particular. Then I’ll tell you about a qualitative study that I did that applied the philosophy on values in science to health economics modelling. This shaped my reasoning about why there are social value judgments throughout the modelling process. Then I'll make some more general comments about the reasons it's hard to simply "follow the science" to inform public health policy. Last, I want to say why I think it would be good to have more interaction between philosophers and health economists, and better public conversations about values in health-policy-oriented science.

Just so you know who you’re hearing from today: I'm not a high up person in any domain, but I’ve worked in public health since 2005 and I did a PhD focused on economic evaluation in health, specifically on economic evaluations of treatment recommendations. So I've spent a lot of time thinking about the costs and consequences of specific types of health advice. My sources for today are standard references in health economics1,2 and some recent commentaries on the pandemic by health economists.3-5. I'll also be drawing on numerous philosophers who write about values in science.6

The ‘101’

Public health is concerned with health at the population level, so it's sensitive to the difference between what will benefit an individual and what will benefit a group. The research often focuses on the effect of 'health interventions', which can take a variety of forms, and it looks at a range of different 'health outcomes'- phenomena or constructs of various levels of complexity. And, very generally, the purpose of doing this is to inform health policy decisions.

So, let's take an example of a health policy decision: who exactly should get a drug to make sure they don't get bat rabies? Back in 2007, one population that drew some attention was people who wake up from a deep sleep, find out there's a bat in their bedroom, and can't confirm they didn't touch it.7,8

This concern arose because some data had shown that only a third of people with bat rabies had an obvious bat bite. This led people to think that possibly a bat bite could be so minor that it would escape a person's attention while they were sleeping. So it was recommended that people with "bedroom bat exposure without recognized contact" should get Post-Exposure Prophylaxis (PEP). Now, this seemed to me to be a pretty reasonable policy.

However, a group of public health researchers thought to consider certain other information, including how frequently people wake up to find a bat in their bedroom.7,8 So they did a telephone survey of people in Quebec and based on their results, they concluded this might happen to roughly one out of 1000 people every year. They also looked at epidemiological data, which showed that between 1990 and 2007, only 2 people with bat rabies got it specifically from "bedroom bat exposure without recognized contact". So they estimated that the incidence of this was roughly ~1 case per 2.7 billion person-years. This meant that to prevent 1 case of bat rabies due to it, you would have to give PEP to 2.7 million people. Furthermore, to do that would cost almost 2 billion dollars.

So this research team concluded that the PEP guidelines should be revisited. Specifically, they noted that this bedroom recommendation effectively implied that no level of risk was acceptable, and that this would come at a very high cost.7 I think this example shows that, sometimes, our views about a health policy can change once we consider both epidemiological data and cost data alongside each other.

This insight is the foundation of health economics. Health economics is the branch of public health science that asks whether a given health intervention is worth doing, considering the other things we could do with the same resources.

It asks this question under the assumption that resources are scarce: that spending comes at an ‘opportunity cost’. This is a just way of saying that if you spend a dollar on something, you've lost the opportunity to spend it on something else. That's the sacrifice.

Not all public health models include costs, but they all inform decisions with health and economic consequences. So health economics is sometimes viewed as a cold or crude branch of public health science, but one can also view public health models that don't explicitly include costs as incomplete and even problematically incomplete. So the ‘101’ of health economics involves couple of core concepts. One is opportunity cost. Another is ‘efficiency’: minimizing input for a given output, and producing cost-effective outputs at an 'optimal' rate and allocating them in an 'optimal' way, whatever that means.1,2

Another one is ‘marginal’ or ‘incremental analysis’: here, the focus is on the costs and consequences of having ‘a little bit more or a little bit less’ 1 of something in the health system, and on the optimal balance of health programs. One common way of comparing two health interventions is the incremental cost effectiveness ratio, or “ICER”, which is the difference in costs over the difference in effects.

In the context of the pandemic, health economists have advanced certain critiques of public health responses, which reflect their thinking. Some common themes are: first, we should openly acknowledge that there are trade-offs being made in any pandemic policy; second, we should do marginal analysis to quantify these trade-offs. For example, the trade-offs involved in imposing 'lockdowns' at different levels of contagion. They've also pointed out that the most common ways of evaluating health interventions, using outcome measures like, ‘Quality Adjusted Life Years’, have not been applied in the pandemic.

For example, Chilton and others write that "Evaluation tools are there to aid decision makers by providing clear information that highlights the nature of the trade-offs being made.... As well as the already noted fiscal costs, the opportunity costs will include social isolation, deepening inequalities, cancer treatments displaced, elective surgeries cancelled, and so forth."3

Reddy that "An ‘optimal’ programme of non-pharmaceutical interventions would be arrived at by weighing the marginal health benefits and costs entailed by varying each dimension of their implementation...taking note of both direct and indirect consequences."4

And Donaldson and Mitton that "Unsurprisingly, a ‘great big marginal analysis’, of the sort currently required...has never, to our knowledge, been undertaken."4 This speaks to the fact that measuring costs and benefits in the context of a global problem like this is an absolutely daunting task.

Already, it should be clear that health economics involves some obvious social value judgment- notably, deciding what to define as a benefit and what distribution of benefits is desirable. A key question is whether a health benefit is of equal value no matter who gets it, or whether health benefits should be valued more highly among certain groups.

Two dominant approaches in health economics are referred to as ‘welfarist’ and ‘extra-welfarist’.1,9 On the welfarist approach the desired benefit is the utility associated with the consumption of goods and services, since health services are regarded like any other goods and services - so benefits are expressed in terms of ‘willingness to pay’. On the extra-welfarist approach, the desired benefit is health itself: health is conceived as its own thing and valued in its own right. In practice, both of these schools have ignored distributional issues and just focused on maximizing total benefits regardless of who gets them.

Extra-welfarism, specifically, raises the conundrum of how exactly to define and value ‘health’. This requires other numerous value judgments like 1) what categories of functioning should be considered as parts of health, 2) what weights should be applied to different health states, and 3) who should decide, and who should decide who decides?2

In general, a number of different outcomes can be used as health indicators. One of these is deaths- and certainly deaths are very often desired information. However, deaths are not informative in all contexts. For example, in end-of-life care or in physiotherapy, the outcome of interest would not typically be death. One alternative that gives additional information is life years gained, but this also leaves out other information, like quality of life.

There's also a desire in health economics to be able to compare cost-effectiveness across treatments for different conditions. For example, if one drug reduces swollen joints, and another reduces lung attacks, health economists want a way to compare their value and help choose which one to invest in.

This has led to the development of various standardized health outcome measures, among which a very influential one is the Quality Adjusted Life Year.

To calculate a QALY you need both data on length of life and quality of life, but more specifically the health-related 'utility' or preference associated with a health state. This type of utility is expressed on a scale of 0-1, where 0 implies death and 1 implies full health. And it's measured indirectly with generic quality of life questionnaires. The two most common ones that I know of are the EQ5D and the SF-36, each of which considers slightly different things as dimensions of health.

(Now, I really wanted to include in this presentation the details of how quality of life data are converted into utility scores- but I was told that this would be burdensome to listen to. However, if you want to know what's under the hood of the QALY, please Google the "Time Trade Off" and the "Standard Gamble".)

To summarize, to get a QALY, utility scores are multiplied by data on length of life. So, if my treatment extends my life for 1 year, but my subsequent health state has a utility value of 0.5, I gain half a QALY.

In practice, QALYs go into ICER calculations. This gives average cost per additional QALY produced by a new treatment or health program. This information is used to help decide whether the new treatment should be funded. In practice, it's very common to assess the value of health programs by comparing the ICER to a threshold value that represents an acceptable cost per additional QALY. This raises the question: how much is an extra QALY worth? Furthermore, in practice, thresholds are just a guide, and decision-makers can always make exceptions. When should we make an exception?

One idea is that we have an obligation- or at least a motivation- to rescue certain people regardless of cost. For example, when we can identify specific individuals in immediate danger, we might not consider cost. This is referred to as the ‘Rule of Rescue’.10 At the same time, there’s a tension between cost-effectiveness and the Rule of Rescue. Between the threshold approach and the Rule of Rescue, it's clear that we can end up with health policies that result in increased expenditures without increases in aggregate population health.

That brings me to the end of my health economics ‘101’. What I hope you took away from it was that the discipline centres around social value judgments. Even if you didn't Google the Time Trade-Off or Standard Gamble, you know that health economists have a big role in defining what a health benefit is and influencing how they're distributed. And we haven't gotten into the specifics of modelling. So that's what I'll focus on now.

Values in Health Economics Modelling

A long-held ideal is that we should restrict the influence of social values to certain stages in science. So, some people have the idea that even though there are obvious value judgments in health policy-making, the modelling process that informs health policy can be value-free. Numerous philosophers have argued that this is not possible. That said, philosophers have each expressed their reasons in slightly different terms, and not with reference to health economics modelling.

In 2019, I did a study where I interviewed 22 health economics modellers on this topic.5 First, I briefed them about different arguments that philosophers make about why there are value judgments throughout the scientific process. Then I asked them if they had any examples from their modelling practice that supported those arguments. So, in the paper, examples from modelling practice are grouped by philosophical arguments about values in science. But one problem with doing that way was that philosophers' arguments are not totally distinct from one another, and articulating the overlap between them is difficult. So, I've been focused on how to express, particularly to the public, why there are social value judgments throughout health economics modelling. I'm going to say it here in 3 categories: what to model, how to model it, and what to conclude. In this context, I'm using examples from what health economists call decision-analytic modelling, which are simulation models that incorporate data from multiple sources and show things over time.

What to Model?

‘What to model’ refers to decisions about what to include in the model. In my view, these decisions are all straightforward extensions of the research question, which effectively everyone agrees involves a social value judgment. In health economics, these are sometimes quite obvious moral judgments about what options are acceptable or even possible to consider. In the context of the pandemic, I imagine we can all think of different strategies that only some people think are even okay to consider.

First we have to specify in a very high level of detail what the alternative strategies are. Here, a lot of moral micro-questions pop up. For example, it should be clear that "lockdown" is not one clear strategy that you could model the costs and effects of. You would have to grapple with specific questions about who exactly will be restricted from doing what and under what conditions. And this is broadly true in health economics: treatment strategies have to be specified in terms of dose, mode of administration, target population, and so on, all of which can be the subject of ethical disagreements.

Once alternative strategies are specified in detail, you then have to specify exactly what costs you're going to include. In health economics, this is discussed under the header 'the perspective'. So studies are done from the health insurer's perspective or the societal perspective, etc., because they include the costs that would be relevant from that point of view. In a study done from the ‘societal perspective’ common questions are things like, “will we include patients' out-of-pocket costs?”

Finally, just like costs, we need to decide what outcome or outcomes are going to be represented in the model. So, we have decide what we think is important to know. Furthermore, we have make trade-offs. Outcomes like deaths might be relatively straightforward to measure but they may not be informative enough for our purposes. We might want information on quality of life, but we may question the specifics of the questionnaires, or the other assumptions that are under the hood of the QALY.

Decisions around what to include in the model are shot through with social values, because they determine what you consider to be definitely relevant to policy-making. Furthermore, you could always make the overarching judgment that the right data don't exist for a particular model and so it would be wrong to build it at all. So going ahead with building a model is a value judgment.

This is something that participants in the interviews spoke to. For example, one participant said, "We made the decision not to simulate anything in that area because the lack of evidence would just produce results that were just fantasy."

How to Model It?

Once we've decided what will be included in the model, we have to decide how to include it. In practice, this overlaps with 'What to Model' decisions, since those are informed by data availability. Nonetheless, I think it's useful to think about decisions about things once they're already chosen for inclusion in a model. So, the first question is what data sources are we going to use to estimate probabilities of events and costs and outcomes associated with alternative strategies? Here, we face all of the unique problems with different data sources. For example, we may want to use randomized controlled trial data, because we expect it to be less affected by confounding, but we may worry that RCT data isn't widely generalizable. So there are trade-offs involved. And the overall judgment is always whether a data source is adequate for our purposes.

Another key decision in this type of modelling concerns the so-called ‘time horizon’. So how long into the future is the model is going to represent things. Different health programs will have different ‘time profiles’, meaning their costs and consequences will kick in at different times. So, models with different time horizons will represent different costs and consequences.

A related issue is that most data sources in health will represent things over a relatively short period of observation. However, health policy-makers often care about effects five years or more after a decision. So the choice of time horizon invites more trade-offs. For example, there may be concerns about whether it's appropriate to extrapolate RCT data over time, but also concerns about not providing policy-makers the desired information. So again, there's a meta-judgment about when exactly a model might cross into fantasy, in an ethically bad way.

Related to the choice of time horizon is the practice called discounting. This where costs and benefits that happen far into the future are reduced by some percentage. And people will disagree about whether this should be done, and by what amount.

Another a major question concerns the representation of uncertainty. So often, there will be uncertainty around what range of values something might take in the world. For example, we might be unsure about how much a treatment will actually cost. So the practice is to do sensitivity analyses to see the effect of varying that parameter. But a few questions pop up: first, what value will be represented as being the most plausible? Then, how many different parameters will be varied, and by how much? Often, model results will be sensitive to the values that are chosen. And often results of different sensitive analyses would support different policy decisions.

A related decision has to do with representing differences between people. There might be evidence that costs and outcomes will vary significantly between different groups. So the question is whether this will be modelled, and if it will be, exactly how will different groups be defined. Depending on how groupings are done, you might see different results.

To summarize, these are some just major decisions that pertain to how model things in health economics. My interpretation is that each one of these decisions has an ethical dimension. Each one is a decision about what representation of things is going to be adequate for our purpose. And our purpose is to inform these policy decisions with health and economic consequences.

What to Conclude?

The last category of value judgments that I would carve out has to do with what to conclude. I'll divide them into the modelling context and the policy context, even though I think these overlap in practice.

First, on the basis of a finished model, what should we conclude is a 'fact'? This is a decision that the modellers or the scientists have a central role in making. And it has a direct influence on policy decisions.

Next, what are we going to do on the basis of the model results? This is a decision that policy-makers have a central role in making. But it's clear that health economists do sometimes directly advise policy-makers. I'll briefly mention two issues here.

First, how should we make decisions under uncertainty? One popular view among health economists is that we should focus on whatever looks like will give the most benefit, regardless of the uncertainty around it.12 And the rationale is that one of the alternatives has to be chosen and the decision can't be deferred, or deferring it is a decision in its own right.

Second, what factors should we consider that were not explicitly modelled? This is important, because often things are excluded from models because they're too time consuming or expensive to model, or simply impossible to model. And it's often said that policy-makers never consider only cost-effectiveness. They might also consider budget impact, equity, public opinion, fear of specific outcomes, and so on. This is important, because it raises the question of how to explain to the public what actually drove a policy decision, when ostensibly it was informed by a model that maybe they heard about.

So, I think that decisions around what to conclude all have an ethical dimension. Decisions around 'what to do' are straightforwardly ethical decisions. And I think decisions about what is a 'fact' have a special influence on those decisions.

‘Follow the Science’: Obstacles

So that's the end of my comments on values in modelling. I hope what you took away it is that there are social value judgments throughout the process. To summarize, I'll use a distinction that Eric Winsberg and I have made recently. First, decisions about what to model and how to model it are decisions about representation.13 They're decisions about what's adequate for our purpose, more than decisions about what's true or false- so they invoke the social values that link to our purposes. Second, after a model is complete, we might conclude what is a 'fact'. And these decisions have a unique social significance, and a different significance than decisions about representation.

So, now I want to comment more generally on why it's not straightforward to simply ‘Follow the Science’ in a health policy context. This is something that people do still recommend doing, at least on social media where it's a hashtag. So it's good to review the obstacles.

First, sometimes we hear that science can't tell us what to do. I think this is true, but I think it's a bit too simply put. Depending on how health policy questions are framed, it can sometimes seem like it's totally obvious what we should want to do, so it's not taken as a serious problem that science doesn't provide this direction. For example, it's sometimes been taken as obvious that we should aim to avoid COVID deaths, and follow whatever strategy we think will get those numbers down as far as possible. But the situation is more complicated, since that strategy has costs and consequences that not everyone thinks they're prepared to accept. This is leaving aside, for the moment, the fact that those costs and consequences are difficult and maybe impossible to estimate.

So, at the very least, it's a bit more specific to say, science can't tell us what to sacrifice or what trade-offs to make. Furthermore, we might do what Martha Nussbaum suggests and separate what she calls the obvious question of 'what to do' from the 'tragic' question of whether any of our options are morally acceptable.14 Like she says, cost-benefit analysis does not aim to answer whether any policy options avoid moral wrongdoing. And neither does science. So that's an overarching obstacle worth mentioning first.

Another obstacle is that the constructs in health policy-oriented science are value-laden. So there are causal claims that come pre-packaged with normative assumptions. Anna Alexandrova calls these 'mixed' claims.15 And I think this is the specialty of health economics. Every claim about what moves the QALY up and down is a mixed claim, since the QALY packs in a huge number of normative assumptions. But it's also a morally significantly choice to count deaths rather than to count QALYs or something else. In most health-policy oriented contexts, in order to ‘Follow the Science’ you also have to ‘Follow the Normative Presuppositions’.

A related obstacle has to do with the fact that different scientific study designs and representations make different trade- offs, in terms of what kinds of questions they help to answer. Donal Kohswrowi and Julian Reiss discuss this.16 For example, they note that data from randomized controlled trials is preferred for guiding policy because it helps control for confounding- but it only gives good evidence about average effects and doesn't it address distributive issues. So there's an ‘epistemic’ trade-off with a social implication. That's a core problem in health economics.

A related risk is what Eric and I have called 'Representational Risk'.13 This is the risk that a decision about scientific representation will not be adequate for purpose. For example, if we are interested in distributive effects, a model of RCT data may not be adequate for purpose- and there are social implications.

So to conclude I just want to say a few words about why I would like to see more interaction between health economists and philosophers. First, I'd like to see them inform the constructs that end up on the centre stage in health policy. For example, there's no philosophical theory behind the Quality-Adjusted Life Year and I wonder what we would use instead if we had philosophers' input.

Second, I'd like to know how philosophers think health economists should manage epistemic trade-offs and representational risk. And I think it would be good to involve philosophers in some of the conversations that health economists are being encouraged to have with the public about issues like this. An overarching question is to what extent value judgments should be made based on philosophical reasoning, on the views of the general public, and/or something else. Right now, health economics models are heavily influenced by what health economists and decision-makers think is right. And this is surely insufficient. So I'd like to see philosophers and health economists interact on this question.

Last, I think the biggest reason goes back to Nussbaum. Again, she points out that the obvious question of what to do is different from the tragic question of whether any option is morally acceptable. And she argues that facing the tragic question is important, because it informs our future actions. I think she means that policy-makers should think about how to limit tragic predicaments where there's no morally acceptable option. And they should consider when they might owe reparations to the people wronged in tragic predicaments. In my experience, this is a type of thinking that I don't encounter in health economics: the focus in that field is entirely on the obvious question and never on the tragic question. So I think interaction with philosophers could change how health economists think, and how they help health policy-makers make decisions and also plan for the future.