Marcel Das is the director of CentERdata and professor of Econometrics and Data Collection at the department of Econometrics and OR.
At CentERdata, the institute for data collection and applied economic research, we had never been involved in the many polls that pop up in the period towards elections. That is, until the 2017 Dutch parliamentary election. Before, we felt that interfering in the daily debate of who is winning and who is losing did not contribute to either society or science. Last year our opinion changed, inspired by a new method that was used by the Center for Economic and Social Research at the University of Southern California.
This method turned out to be quite successful in predicting how Americans would be voting in November 2016: for Donald Trump or Hillary Clinton. It did become a scientifically interesting challenge: a scientific experiment to examine the performance of an alternative method was born.
Traditionally, election polls ask for which party one would vote if (parliamentary) elections were held today. The respondent can then select one of the listed parties. This may be problematic in case you are still uncertain on what to vote and leave your options open. In the alternative method respondents are asked to give the percent chance that one would vote for each of the listed parties (if parliamentary elections were held today). The total probabilities should add up to 100%. And before answering this question, the respondent was also asked the probability that he or she would vote at all.
Next to the different method of asking we also used a specific and unique sample for this study: members of the LISS panel. As I explained in my previous column, this panel deviates from the usual online panels in the sense that the panel is not based on a self-selected sample but on a random sample drawn from the population register by Statistics Netherlands. In addition, households that could not otherwise participate are provided with the necessary equipment, such as a computer and/or broadband connection. The panel that was used in the United States for the election poll is similar on these aspects.
There are many more details I could share with you, but a column has a limited number of words and I guess the only thing you are curious about is the outcome. Did the alternative method perform better than the method used by the commercial pollsters? Long story short: ‘no’ and ‘yes’. Our predictions were not closest to the actual votes. That may sound disappointing, but actually here also science starts (again) because one wants to know why. One of the reasons is that we followed the design that was also used by the US research team. Data collection was done on a daily basis (rotating groups) and the predictions were calculated as the average of all of the prior week’s responses. In case voters doubt until the last moment – which is definitely the case, perfectly shown by the alternative probability questions – then crucial events prior to the day of elections may have a large impact. And that’s what happened: the diplomatic crisis between Turkey and the Netherlands during the weekend before Election Day had most probably changed opinions, at least for a part of the voting population. Our running average gave insufficient weight to this event.
But there were also positive results from using the alternative method. Jochem de Bresser and Arthur van Soest, both affiliated to the department of Econometrics, analyzed the data and found clear evidence that at the level of the individual, subjective probabilities are substantially better predictors of actual voting than deterministic statements. They also analyzed whether this consistently hold for the entire sample, and found substantial heterogeneity in the predictive power: the alternative method works better for those with higher probability numeracy.
Want to analyze the data yourself, looking for other (different) patterns? The data are freely available in the LISS Data Archive (https://www.dataarchive.lissdata.nl/). You may also visit the website https://verkiezingen.lissdata.nl/ for more information. I’m curious whether you find interesting and other insights. And maybe you will come up with an even better way of predicting the outcome of elections!
Text by: Marcel Das