In the academic discipline of decision-making, experts in general have a mixed reputation. A well-known result is that when comparing predictions of human experts with the predictions of statistical models (based on the same information), models outperform human experts in about 40% of the cases while experts outperform human experts only in about 10% of the cases (Grove et al., 2000).
Recently, we decided to give a particular type of expert a chance to defend the reputation of human experts: real-estate agents. As part of a series of three Bachelor Thesis projects, we did the following: we first scraped about 8,000 housing advertisements, all houses in the Eindhoven region. We then tried to model the asking price as dependent on the characteristics of the houses: size, year of build, number of rooms, etc. Most real-estate agents are used to model-based predictions in their daily practice, as two model-based predictions need to be added to an official valuation of a house. The models that provide these predictions do not reveal the way in which they calculate predictions but do claim to use recent selling prices of houses near the house in question. We used a relatively straightforward model, so it might have been better if we had mined a bit more, but it turned out that the models did not perform too great. For the better models, the percentage of houses with an estimate within 5% of the actual price was only about 25%. A golden opportunity for our real-estate agents!
In total 51 participants took part in an experiment of which 23 were real-estate agents from Eindhoven and 8 were real-estate agents from outside Eindhoven. As a reference category we also asked 28 novices (that is, people without any particular knowledge about the housing market). Each participant had to predict the market value of 13 houses. For 3 houses, participants only got summary information (address, area/volume, kind of house/apartment, and arrangement of the rooms). For 5 other houses, participants received a much more extensive list of variables to give the participants a complete overview of the characteristics of the house. For 5 still other house, participants also received pictures of the house. All houses were chosen from the scraped list of actual houses for sale in the Eindhoven region. Ok - so what happened?
With hardly any information available, even our mediocre model outperformed all humans. The average error for all humans, expert or not, was about 20% (!), whereas a model based on the same restricted amount of info halved that error. With more information available, the model still outperformed all humans, although only beating the real-estate agents by a narrow margin. The novices did not improve with more information and scored 20% error again. Real-estate agents, both from the Eindhoven region and outside that region, averaged an error of 14.7%. The model scored 14.2%. Finally, we had the houses with elaborate info and pictures. The model could only predict based on the info and not the pictures, so the humans in a way had an unfair advantage here. But at last, they did prevail! Novices still did not get any better (a bit worse actually): 23% error. The real-estate agents from outside Eindhoven, who were evaluating houses outside of their regular working area, scored an error percentage of 16.7%. The model scored 13.1% but the real-estate agents from Eindhoven performed best with an error rate of 9.5%.
So who won? Well, the literature on human experts versus statistical models only makes comparisons when both model and expert have the same information. In all these cases, models won (even the relatively bad ones that we used). On the other hand, give the true experts some pictures and they do get better than our model. So let us call it a tie.
There is much more to say about the experiment. For instance about the extent to which humans improve if you give them a model-based prediction (they do, but not always), or which humans are more likely to change their prediction, how much a model would improve if it would incorporate more local characteristics, and many more interesting issues. Want to read the full Bachelor Thesis? You can download them here.