My Harkness year coincided with Donald Trump’s election and the 2017 Republican attempt to repeal Obamacare. So I promptly ditched one of my Harkness projects in November 2016 and decided to understand better what Republicans actually wanted national health policy to do. I asked state politicians across the US about their priorities for the goals of health policy. The mathematically aggregated results from 182 Republican and 192 Democrat politicians were illuminating, highlighting enormous differences on the role of government and tackling disparities.
Importance map derived from legislators’ ranks of different possible goals for health policy. The main goals are highlighted – the lighter grey goals consist of a variety of goals related to the quality and safety of health care. First shown in Pagel, Harvard Health Policy Review, Oct 15th 2017
Once back in the UK, my thoughts turned to Brexit. I was frustrated that polls asked people about their preferences regarding the Single Market or Customs Union vs controls on immigration when to me the real question was what people wanted the UK to achieve. Binary questions explicitly assume the pollster knows what voters’ top priorities are. I wanted to ask voters instead, and thanks to the Harkness work, I had the methodology ready to analyse ranked responses.
This ended in two large surveys carried out independently over the summer, the first supported by UK in a Changing Europe and the second supported by the People’s Vote Campaign. Both surveys asked people (over 15,000 in total) to rank a list of 13 challenges the UK faces over the next 5 years. We then explored how ranks correlated with referendum and party votes and what this might mean for the Brexit debate.
Importance map derived from referendum voter ranks of 7 different challenges for the UK. We only show 7 out of 13 challenges for ease of visualisation. Adapted from Pagel and Cooper, UK in a Changing Europe, July 2018.
The maths is cool and the results to both the US and the UK work were fascinating, but neither project would have worked without several months of careful development of the survey questions. For both projects, I had to acknowledge that I was biased, with strong opinions of my own on both the future of US health policy and Brexit. That meant that it was crucial that I talked with people holding different opinions and tested the survey language thoroughly for relevance, potential for misinterpretation and that the priorities were meaningful and complete.
To do this in the US, I first compiled an initial list of 24 candidate health policy goals taken from official policy documents and the World Health Organization. I then had a series of one-on-one structured phone calls and meetings with twelve key people, all experts in health policy, where I discussed each potential goal in detail, refining the set of goals between each conversation. People were chosen to reflect different perspectives (party affiliation, expertise, area of work e.g. federal government, local government, hospital system). The list of goals needed to be comprehensive and not associated (if possible) with specific policy solutions, to avoid people just picking policies their parties supported. Towards the end of the process, I spoke to six current state legislators as a focus group, three Democrats and three Republicans from six geographically diverse states, to finalise the selected list.
In the UK, where I was doing this in my spare time, I needed a different approach. I used the website prolific.ac to run mini-surveys on groups of Leave and Remain voters in the UK, testing out different versions of a list of challenges for the UK. Again, I wanted a list that was reasonably comprehensive, not tied too closely with policy and to contain challenges that might be affected by Brexit. As well as asking respondents to rank the prototype list, I asked them what was missing or what was unclear. I also ran separate surveys asking respondents to describe what they thought key challenges meant to tighten the language. Overall, I engaged with almost 200 people in developing the survey, weighted towards Leave voters (70 Remain Voters, 120 Leave Voters) not least because, as an academic in London, I hardly knew any Leavers.
This engagement definitely changed the priorities chosen. For the US for instance, we added “Reduce the role of government” as a goal – and this turned out to be second highest priority for Republicans. It clearly represents a fundamental value in a way that I just hadn’t understood. In the UK, I included two immigration questions – one around reducing overall number of immigrants and the other restricting immigration to just high-skilled immigrants – because Leave voters described and ranked these differently in the test surveys. I also added “UK Control over laws & Regulations” because Leave voters told us this was their number one issue. And indeed, in our surveys of over 15,000 people, control over laws was much more important to leavers (and a significant number of remainers) than immigration.
The fact that the results made sense, providing new insight and, in the UK, were consistent across two large surveys, shows that the care taken in choosing the questions was worth it. It’s so tempting to skimp on this step, to rush straight into surveying and answers. But misleading, or leading, questions lead to unreliable answers and, potentially, damaging policy. No amount of clever mathematics can compensate for that.