The FIFA World Cup of 2018 seems a lifetime ago. So much has changed since, with at least two years lost to a global pandemic. However, one thing hasn’t changed: Video Assistant Refereeing (VAR) keeps getting decisions wrong.
At the last World Cup, VAR was introduced on the world stage. Up until that point, football, as one of the world’s most established and popular sports, was well behind the times in its use of video technology to aid decision making. And its introduction was not without controversy, with many divisive incidents over the course of the competition.
Cognitive biases in VAR
Following the tournament, we put together a review of the technology’s implementation from the perspective of behavioural science. That is to say, how good VAR was at removing cognitive biases from verdicts. Needless to say, we weren’t particularly impressed with how FIFA had enacted the system, even if FIFA President Gianni Infantino has said himself that it was ”an undisputable success”. Rather than removing cognitive biases, the approaches in place to help decision making were often reinforcing them. We highlighted confirmation, framing and anchoring biases.
Furthermore, we showed a number of incidents, particularly around handballs, where ambiguous rules led to highly subjective interpretation and decision making. One of these handball infractions occurred on the biggest stage of all, the World Cup Final, where France was awarded a penalty for a less than clear cut handball by a Croatian player, after Blaise Matuidi had headed the ball onto his arm.
We outlined three recommendations for improving the system:
- Reduce information overload – the referee has plenty to process under pressure without adding more stimulus, e.g. replays on the pitch-side monitor
- Leave decisions to the 4th official in charge of VAR – this will help ensure decisions remain as objective as possible by their independence from the on-field referee
- Make the rules clearer – VAR works best resolving binary scenarios; ambiguity around some laws like ‘handball’ leave decisions too open to interpretation and vulnerable to biases
In subsequent years, and despite years of fans bemoaning the lack of technology in football, VAR hasn’t exactly covered itself in glory. Many of the issues are down to the above suggestions not being addressed. Or, as with amend to the handball law at the start of the 20/21 Premier League season, not being implemented in a way favourable with fans. In fact, from a poll of more than 33,000 supporters by the FSA, 95% of those who had experienced VAR in-stadium said that VAR had made watching football less enjoyable.
VAR changes for the Qatar World Cup
So, what can we expect from this year’s tournament? The good (or bad) news is that some changes are being implemented. FIFA has announced that semi-automated offside technology will be used in Qatar, “offering a support tool for the video match officials and the on-field officials to help them make faster, more accurate and more reproducible offside decisions”. The new technology uses 12 dedicated tracking cameras and an Inertial Measurement Unit (IMU) sensor in the ball to track its position and those of up to 29 data points of each individual player. 50 times per second, the exact position of all data points is known on the pitch, providing an automated offside alert to the video match officials.
This feels like a welcome addition; even if a fraction of a toenail rules someone offside, a least there will be greater objectivity than the crudely drawn on lines we’ve grown used to. The speed of offside decisions – and often whether we can celebrate a goal or not – will be much quicker. But is there anything else that the footballing powers should be considering beyond our 2018 suggestions?
Noise: A flaw in human judgement
In the last year, the ‘Godfather’ of behavioural science, Daniel Kahneman added to his magnum opus: Thinking, Fast and Slow - the bible on cognitive bias - with a book on Noise: A Flaw in Human Judgement. Luckily for you, I’ve read it so you don’t have to (it is a rather dry tome after all), but there are further lessons in it for improving judgements.
In this new book, Kahneman, with collaborators Olivier Sibony and Cass R Sunstein, starts by using a series of targets to explain the difference between bias and noise. He says: ”Put simply, bias is the average error in judgments. If you look at many judgments, and errors in those judgments all follow in the same direction, that is bias. By contrast, noise is the variability of error. If you look at many judgments, and the errors in those judgments follow in many different directions, that is noise.“
In other words, we would hope decision makers in the same role (e.g. football referees) to be interchangeable; that is, faced with the same case they would reach the same or very similar judgments. In reality, not only do different referees make different judgements, the same referee (as a flawed human) can be strongly influenced by irrelevant factors, such as their current mood, the time since their last meal, and the weather when judging an identical case in a different scenario. As established, we call this variability of judgment ‘noise’. Noise doesn’t just cause inconsistent judgements, but comes on top of biases that incline judgement towards a particular outcome already.
Reducing noise in VAR decisions
Experienced professionals tend to have high confidence in the accuracy of their own judgments, and they also have high regard for their colleagues’ intelligence. This combination inevitably leads to an overestimation of agreement and so will be the case with referees and VAR. We’ve highlighted possible ways that VAR can improve with a reduction in biases, but how can it potentially address noise? Following lessons learnt in the book, we’ve outlined below some thoughts for FIFA to consider.
- Perform a ‘noise audit’ on VAR practices and officials - a review of multiple decisions across multiple referees to first and foremost identify where noise is being generated. In essence, looking at the frequency and scale of variability in judgements to help address the inconsistencies.
- Formalise decision making by replacing human judgement with algorithms. Of course, this is the aim of technologies being introduced, particularly the new semi-automated offside system. However, these algorithms or ‘reasoned rules’ don’t have to be too elaborate or indeed automated. A set of rules or criteria can be set (and met) to determine particular decisions. Each variable can be fed into a predictive formula that calculates their various weights and produces a more consistent decision. Of course, time would be of the essence in sport, but this could be worked around with individuals responsible for just one input into a decision-making system for speedy judgements.
- Linked to the above is aggregated decision making – essentially, taking an average or majority decision from a panel of judges. Hardly ground-breaking, and used in many sports already, it seems sensible that if decisions are already being ‘sent upstairs’ to a VAR room then having a panel of referees in the room ruling independently (this is critical), rather than as a collective isn’t too unrealistic. The diversity, as well as complementary nature of these independent decision makers, can also be carefully selected to reduce noise.
- Finally, and probably most controversially, FIFA could vet referees or VAR decision makers more thoroughly based on their intelligence and their ‘active open mindedness’. Intelligence is highly correlated with better performance in all walks of life, however, it is the addition of cognitive style, specifically being ‘actively open-minded’, that has been found to be most predictive of decision making accuracy. As Noise says, ”To be actively open-minded is to actively search for information that contradicts your pre-existing hypotheses. Such information includes the dissenting opinions of others and the careful weighing of new evidence against old beliefs.” This isn’t a universal trait amongst intelligent people, so it stands to reason that the best referees (or at least VAR arbiters) be recruited with this trait.
In summary, with humans involved in judgements, there is always going to be error. And whether this is what makes sport unpredictable and exciting is also a point of debate. However, I think it’s hard to argue VAR could not be improved drastically by reducing both biases and noise. What the upcoming Qatar World Cup holds in store, we’ll have to wait and see, but I’d bet on there being one or two dubious refereeing decisions.
