By Jeffrey Ross and David Jones [1]

There are tremendous advantages in effectively screening, testing and assessing candidates. Finding new hires who learn quickly, hit targets and advance is a major payoff. Reducing turnover, whether voluntary or involuntary, is another. Finding new hires who engage with their job, coworkers and organization is yet another. What’s key is doing it the right way.

There are any number of high-quality vendors in the world of employment testing and assessment. But we also see growing examples of pitfalls that employers need to avoid when they shop for the right program. Some sales pitches lead to claims that boil down to: “Trust us, our tests work.” That’s a sign of a buyer beware market.

Accepting claims like these can drive bad hiring decisions and raise the legal compliance risks of an employer’s hiring process. Too often we look at a claim about what a vendor’s test does and are concerned that the employer could be vulnerable if challenged.

To avoid wasting money and creating risk with poor employment testing tools, ask the vendor our questions below. Also seek proof that a vendor’s proposed testing tools actually work before moving forward. This will help you avoid Sham, Scam and Khazaam! claims such as:

1. “Try Our Test with Your Top Performers”

When asked to explain how their testing tool works, some vendors tell the employer that all it needs to do is test the organization’s very best performers, and then compare future candidates to the resulting top-performer profile. The claim sounds convincing when presented by a talented sales person, but when you think about it, it’s truly a sham.

How would we know whether the poorest-performing employees score any differently than top performers? After all, don’t we want to be sure that the test actually distinguishes between the best versus the worst?

We were asked to help defend an employer where a vendor made such a claim. Yet the data showed that the employer’s top performers scored no differently on the test than its poorest performers. The result? Wasted money, bad hiring decisions and an invitation to legal challenge when the test produced different selection results for candidates of different ethnicity.

2. “Try Our Test with Your Top and Bottom Performers”

Sometimes a vendor suggests testing a few of the employer’s best and worst performers. This sounds more sensible. But it’s almost as bad in its payoff and defensibility.

Picking just a few employees to compare, choosing extremes from the workforce or simply finding chance differences instead of statistically reliable ones will produce the same result: an unacceptable claim that the assessment tool really works.

There are clear standards that need to be met if the test is challenged. Using data based on small, atypical employee groups just won’t stand up. Instead, ask the vendor how to decide where to set the standard for candidates in the middle of the range and whether they’ve assessed a large enough group to produce statistically reliable results.

3. “Our Tests Predict Candidate Performance”

Start with a basic concept: To defend a test from a disparate impact discrimination claim, an employer needs to show that the test predicts candidates’ on-the-job performance. This is the concept of “validity.” If the test has validity, it’s much more likely to be defensible. From a business point of view, if the test doesn’t predict a candidate’s job performance, why spend the money?

But what is the vendor claiming that its validated assessment tool actually predicts? Sometimes what’s predicted is a sham because it’s not really important. For example, vendors sometimes claim that candidate scores on their tool match the way supervisors evaluate employee performance. This can be useful, provided the evaluations reflect important work outcomes. But sometimes these claims simply show that the way candidates describe themselves on a test matches how their supervisors describe them. Is this really useful? What about linking test scores with objective measures such as productivity, performance targets or absenteeism? Employers need to ask a vendor:

• What aspects of on-the-job performance can we expect your assessment tool to predict?
• How accurate will these predictions be? What level of “hits” versus “misses” can we expect?

4. “This Test Is Totally Legal”

Unfortunately, there is no such thing as a “legal” test. Whether a test is legal depends on where and how it is used. A test may be perfectly legal when used in one setting or in a given way, and absolutely illegal when used in another. But test vendors are not lawyers. Testing tools usually are created by psychologists, then marketed by sales staff. They generally do not understand the esoteric legal requirements applicable to testing. And that’s where the trouble starts.

Some vendors make bold representations that their test does not cause an adverse impact on any legally protected group. That may be true, but whether a test causes an adverse impact often depends on the composition of the candidate population being tested. And candidate populations differ widely.

Whether a test has an adverse impact also depends on the lawful application of the test. We reviewed the work of one vendor that eliminated adverse impact on African-Americans and Hispanics. How? The vendor used a minority group-based “fudge factor” to alter test scores. But Congress specifically outlawed that type of scam!

Another vendor claimed to perform adverse impact analyses based on “ethnic background.” But their technical report only analyzed data as “white vs. nonwhite” or “white vs. minority.” It provided no breakdown by race or ethnicity (i.e., African-Americans, Hispanics, Asians, etc.). So there was no way to determine from the report whether the test has an adverse impact on any legally protected group.

Then there was a testing expert who advised that it was lawful to continue using a vendor’s test as long as it caused no adverse impact, even though the test had no validity. It’s hard to imagine professional advice that is so truthful, yet at the same time so dysfunctional.

Some vendors are more creative. A vendor of a so-called personality test took liberties with the validation data to develop marketing materials that blatantly misrepresented the traits measured. But if you think that’s bad, the same vendor’s marketing materials also represented, without any basis, that the test would improve tax and regulatory compliance. You can’t make up this kind of scam. Employers can avoid many of these problems simply by posing questions like the following:

• Will you document in writing your claim that it would be lawful to use your test for the jobs we have identified?

What will your firm do if we face a legal challenge for using your test?

Bottom Line: Ask Probing Questions

Employment testing and assessment can produce tremendous benefits. And many vendors are high-quality professionals who would never consider using Sham, Scam and Khazaam! strategies.

Too often, though, we find employers misled by claims of research and statistical evidence that fall outside their areas of expertise. Making “people decisions” based on vendor promises of “trust me, this will work” can produce some of the biggest losses you’ll ever see, and can keep you from growing a truly high-performance organization.

The principle is simple: Don’t settle for generalities. Find vendors who focus on tracking and continually improving how their product works for you. How you select people needs the same accuracy, monitoring and continuous improvement you use in growing your organization’s other high-value assets.

[1]               This article is adapted from one published by Mr. Ross and Dr. Jones in the February 2015 edition of Corporate Counsel.