Can Pre-Employment Tests Predict Employee Success Better than a Human?

Still trusting your gut when hiring? That's long been a tagline of ours when discussing the predictive power of incorporating pre-employment tests into the hiring process. Psychological researchers have consistently demonstrated that pre-employment tests provide relevant and objective data about job candidates that are ultimately more predictive of employee success than a host of other criteria commonly used in making hiring decisions. And now, a compelling new study from the National Bureau of Economic Research (NBER) reaffirms the idea that the use of pre-employment tests leads to tangible improvements in hiring results.

A piece in Bloomberg Businessweek describes the study's finding under the provocative byline "Machines are Better than Humans at Hiring the Best Employees." The article is the latest in a series of articles heralding the advent of algorithm-driven hiring. (See another example from the Harvard Business Review.) However, the actual content of the study had little to do with "Big Data" or algorithms that incorporate data from social media profiles or job applications. The study simply endorsed a finding that has long been uncontroversial among organizational psychologists: pre-employment tests are better at predicting workplace performance than other more subjective criteria.

The impact of this study, however, may be greater than many of its predecessors because of the pedigree and credibility of its source. The NBER is a private nonprofit and the largest economics research organization in the United States. Its members include no fewer than 22 Nobel Prize Winners in Economics.

For this study, the NBER partnered with researchers from Harvard, Yale, and the University of Toronto* to investigate how the hiring decisions of (human) managers compared with the hiring decisions made by a pre-employment test's algorithm. The study's conclusion: the more a manager strayed from the test's recommendation, the worse the hiring outcome. In other words, the test selected better job applicants than the hiring managers.

The Study

The research focused on 15 firms hiring entry-level service sector employees. The firms gradually started to administer a pre-employment test to applicants with the goal of reducing turnover. The test assessed cognitive ability, personality traits, technical skills, and other factors to come up with a green, yellow, or red score. The data consisted of an enormous sample of 300,000 hires and 555 hiring managers.

The first goal of the study was to show that as firms adopted pre-employment testing, the quality of their hires improved. Quality of hire was judged by longer tenure rates and greater hourly productivity. The staggered adoption of the testing program allowed for time series analyses that clearly show a tipping point once the firm was testing at least half of prospective employees. What they discovered was that simply incorporating job tests into the hiring process lead to improvements in employee retention – employees who were hired with job testing stayed 15% longer in their positions than those hired without testing.

Furthermore, the color classifications were predictive of employee retention, as candidates who received green scores stayed an average of 12 days (11%) longer than those who received yellow scores. Furthermore, those who received yellow scores stayed 17 days (18%) longer than those who received red scores. In an industry where the median worker only stays for 99 days, or a little over 3 months, this marks a substantial difference.

The study then investigated how much "discretion" hiring managers used when selecting job applicants by passing over higher rated candidates. Managers differed in the rate at which they exercised their own discretion, with some following the test's recommendations exactly, and others regularly choosing yellow and even red candidates even when green ones were available. The study found that the greater the degree of discretion exercised, the lower the benefits of testing to the firms. The conclusion is that whatever rationale the hiring managers were using to override the test scores – be it likeability, social fit of the candidate, intuition, or "gut feeling"- the managers were clearly not acting on better information that would improve their desired business outcome, in this case better employee retention.

What This Means for Employers

The popular and business press has recently made a lot of noise about the arrival of "Big Data" and its application to hiring because the "man vs machine" angle makes for an intriguing narrative. The findings of the NBER study, however, are far from new; the study can more accurately be described as an endorsement of the predictive validity of pre-employment tests specifically, rather than an endorsement of algorithm-driven hiring in general. Half a century ago, the University of Minnesota psychologist Paul Meehl extensively documented the superiority of statistically-driven approaches to personnel selection. More recently, the most extensive meta-analysis of the predictive utility of pre-employment tests (Hunter and Schmidt)** concluded that cognitive aptitude tests, in particular, were far better predictors of work performance than commonly used selection criteria such as interviews, work experience, and education level.

The recent NBER study convincingly affirms the value of pre-employment testing. Through its massive sample of data, this new study takes a unique approach by analyzing the behavior of hiring managers to determine how bias affects their hiring decisions and, ultimately, employee outcomes. What it finds is not surprising – humans are naturally biased and tend to trust their intuition, or "gut," when making judgments about people.

The recent NBER study convincingly affirms the value of pre-employment testing.

This does not mean that hiring managers should completely discard their intuition in favor of an algorithm, or that HR professionals should feel in any way threatened by, or overly beholden to, the verdicts of pre-employment tests. Human Resources professionals have ways to gather vast amounts of job-related information on candidates that cannot be measured by pre-employment tests. However, this study suggest they must be careful to incorporate such additional insights in methodical and rigorous ways, in order to avoid being misled by subjective impressions that may not be job-related. Pre-employment tests offer great predictive validity when compared to traditional means of gathering information on candidates, including resumes and interviews, but ultimately should be only one part of a comprehensive set of criteria used to evaluate prospective candidates. Pre-employment tests have the advantage of providing objective and standardized data that hiring managers can use to make more informed — and more effective — hiring decisions.

*Hoffman, M., Kahn, L.B., & Li, D. (2015). Discretion in Hiring (Working Paper No. 21709). Retrieved from National Bureau of Economic Research website: http://www.nber.org/papers/w21709
**Schmidt, F. & Hunter, J. (1998). The validity and utility of selection methods in personnel psychology: Practical and Theoretical Implications of 85 years of research findings. Psychological Bulletin, 124(2), 262-274.