Brokers eschew Zillow’s AVM stating a lack of accuracy and the confusion it causes in the marketplace as their major complaints. So let’s investigate how real estate brokers and Zillow’s AVM stack up against each other in terms of valuation accuracy.
Zillow claims a median error rate of 8%. Per Zillow’s website this means half of all automated valuations are within 8% of sale price and half are beyond 8% of sale price (values can be higher or lower than actual sale price). It certainly must be the half of errors greater than 8% that antagonizes the brokers, as my own research shows that brokers’ own estimates of value have a median error of 5% and an average error rate of 7%.
To get some idea of brokers’ own accuracy I pulled six months of sale data directly from the MLS (Mid-October 2014 to Mid-April 2015). (As a note this particular MLS prohibits brokers from pulling more than 5,000 properties at a time, so I stopped there.)
Since this MLS allows input from areas outside of the direct metro area I used only cities truly within the metro area, feeling that outlying areas may demonstrate more variance.
However I purposely included all geographic areas within the metro area and all price ranges. So there’s a solid mix of high, medium, and low priced properties from a wide swath of physical locations.
Additionally I used the “original list price” as the brokers’ valuations and the “sold price” as the accuracy factor. I felt no need to worry about closing costs or other seller paid contributions made as I was trying to purely estimate brokers’ accuracy in valuing properties.
There were a few properties I truncated from the analysis as they were obvious typos (i.e.: asking price $1,000,000; sale price $10,000 sort of errors). I do think it’s sad that the MLS does not catch such obvious errors though.
This MLS stated approximately 45,000 sales in 2014, but this includes outlying areas and data sharing between MLS systems. My sample size was 4,802 properties, again within the most recent six month period, while removing outlying areas and obvious typing errors. Thus my sample was well over ten-percent of the true metro area sales.
How do brokers compare to Zillow? If Zillow’s claim of 8% median error is correct (I do not have the means to test this), and with brokers’ own initial price in the MLS having a median error rate of 5%, certainly brokers offer superior valuations, but perhaps not to the extreme they may suggest based on their publically pronounced disdain for the Zestimate.
A few additional broker accuracy notes from my study: 710 properties undervalued by an average of (-5%); 3483 properties overvalued by an average of (8%); and 609 properties valued exactly at sale price.
(Median error rate was used for the comparison since this is how Zillow publishes their results. Using a median error rate moderates the impact of the larger errors toward the tails of the bell curve if you envision the error rate being plotted that way.)
While certainly the 3% difference between the brokers’ 5% median error and Zillow’s 8% median error is not tiny, it does not seem as extreme as the banter by the brokers implies.
Where the Zestimate gets in trouble in the marketplace is when the user sees a large discrepancy between what they think the home is worth and the automated valuation. Most home owners and actually many seasoned buyers have a pretty decent idea of property values in their neighborhood. Zillow covers this incongruence statistically by providing a 95% confidence level range of values, but many users don’t notice that “small print”, and Zillow never explains why there’s such a massive spread in potential values in terms the average seller or buyer can relate to—specifically what causes such a wide range of potential values. Brokers have the opportunity provide such an explanation live with the client.
To create a truly equal comparison it would be ideal to have the 100 largest MLS companies run a full year’s analysis on all homes sold, using the initial asking price as one factor and the actual sold price as the other. It would be their call, but keeping the dataset to residential properties only, both detached single family and attached single family, all within the generally accepted boundaries of the metro area, should make for a very interesting study.
I do realize that many brokers may object to using the initial asking price in the MLS as their valuation, as the asking price is often “negotiated” between broker and home seller. But to this I respond that there’s no other way to complete this study in large scale, and no broker is forced to accept any listing agreement if they feel the home is not worthy of the price they input into the MLS. The broker has accepted and published the initial price.