In 1915, a hilariously caustic review of a science book appeared in the pages of the American Journal of Sociology. “It must have been a sense of humor which led the publishers to put this volume in their ‘Science Series’,” began sociologist A. B. Wolfe. You get some idea of where he was going.

The volume in question was Sex Antagonism, written by the highly respected British reproductive biologist and fellow of the Royal Society, Walter Heape. It appeared to be a work of biology, but it happened to be published during a moment of great political upheaval. Women were only a few years away from getting the vote, after persistent and sometimes violent struggle, and this wasn’t welcomed by all—including, it seemed, the eminent Heape.

Sex Antagonism was his attempt to throw his two cents (perhaps a little more) into the debate. Using some decidedly tortured scientific logic, Heape argued that suffragettes were damaging their childbearing potential by channeling all that energy into activism. Instead, he wrote, women should focus on being mothers, since that’s what they were biologically designed for. Don’t be offended. This was science, after all, not mere opinion.

In case this wasn’t enough to convince the suffragettes to lay down their banners, Heape ended his book with a thinly-veiled threat: “For the present the man is very patient with his woman-kind, and is remarkably silent concerning the discomforts he himself is subject to; perhaps for these very reasons he will act with all the more force when the proper time comes for action.”

You can almost hear Wolfe’s suppressed guffaws. “It is a fine illustration of the sort of mental pathology a scientist, especially a biologist, can exhibit when, with slight acquaintance with other fields than his own, he ventures to dictate from ‘natural law’ (with which Mr. Heape claims to be in most intimate acquaintance) what social and ethical relation shall be,” he wrote in his review, adding: “He sees only disaster and perversion in the modern woman [sic] movement.”

Wolfe highlighted a problem that we live with to this day. Society listens to what scientists have to say because we imagine that they are motivated purely by the truth, without any of the messy personal prejudices that plague the rest of us. What’s more, we like to think that a smart person is smart about everything. Is Trump good for America? Ask Stephen Hawking! Should I invest in mortgage-based securities? Again, ask Stephen Hawking!

The recent controversy over a memo written by former Google software engineer James Damore, in which he suggested “human nature” might explain part of the shortfall of women in Silicon Valley, revealed exactly the kind of “mental pathology” that Wolfe condemned. With the scantest scientific evidence, Damore felt able to preach to Google exactly how it should be running its diversity programs. “But he’s a Harvard graduate!” his supporters shouted, as though this made his opinion more reliable.

The history of women in computing, the social pressures that draw girls away from science, the macho sexism of Silicon Valley, were all expected to dissolve in the powerful acid of Damore’s scrappy scientific knowledge of sex and gender.

Having spent the last few years researching the science of sex differences, I have interviewed firsthand some of those whose work Damore cites. What is clear is that we know relatively little. What is also clear is that the evidence so far does not suggest that whatever small psychological sex differences there are between women and men can reliably explain the enormous gender inequalities we see in society. There are too many other factors at play, most of which have nothing to do with biology.

Yet in the gap between what is known and unknown, speculation is rife. The press release attached to one high-profile paper published in 2014, claiming to see significant sex differences in white matter, erroneously suggested that women are better at multi-tasking—which one of the paper’s authors himself much later told me was incorrect. In the early nineteenth century, neurologists speculated that women must be less intelligent because their brains weigh on average five ounces less than men’s.

It is dangerously easy to marry the small amounts of data we have about sex difference with our firmly rooted gender stereotypes. As sociologist Wolfe proved in his blistering review, studying humans is of no use if social, historical and cultural context is removed.

The social sciences, including philosophy, tend to be more humble in their proclamations about human nature. It’s a humility that is sadly lacking in some parts of biology—and particularly these days in evolutionary psychology and neuroscience. Scientists are dismally slow to admit that they might be affected by bias, and curiously silent when their work is used by others to defend dangerous political agendas.

This doesn’t mean that scientists can’t be right about topics that are important to society. If the evidence is on your side, feel free to brandish it. But it takes more than a little learning to be sure that it is.