The conversation about diversity in tech is getting hijacked by bad research

Recruiting companies offering one easy fix to gender disparities are getting taken way too seriously.

Sunday’s New York Times Magazine contained a long feature by Clive Thompson on the history of women in tech and the industry’s more recent efforts to grapple with software engineering’s gender gap.

One statistic in the piece jumped out at me:

That’s an astounding statistic — so astounding that it seemed literally incredible. Research into “blinded” résumés has sometimes found that it reduces bias — but never by a margin like that. Yes, blinded auditions for orchestras saw the share of women in the top orchestras grow from 5 percent to 25 percent, but that was over the course of 40 years. This effect was more pronounced than that, and it happened overnight.

If these statistics were right, a relatively simple change to job applications could close the whole gender gap in tech, if not reverse it. I dug into it and found that the whole statistic was fishy — based on a shadow study that no one seems to have actually seen.

In 2016, Speak With a Geek, a tech recruiting company that has since closed its doors, announced that it had conducted an experiment with blind auditions. CNET was the first to report on the experiment (everyone else who reported on it cited CNET, rather than seeing the original experiment):

This claimed effect is much larger than the largest disparities ever found in reputable studies about the effects of résumé blinding. Speak With a Geek never published this experiment anywhere — including on its own website or social media. People who asked to see the results at the time were ignored. And some things about the study are confusing even aside from the shocking size of the effect: Who would select any candidate for an interview when “names, experience and background” weren’t provided? Companies were shown the exact same candidates twice? These are questions we could answer by looking at the methodology in more detail, if it were available. But so far, it doesn’t seem to exist.

Read more

Our videos