Most people never become mathematicians, but everyone has a stake in mathematics. Almost since the dawn of human civilization, societies have vested special authority in mathematical experts. The question of how and why the public should support elite mathematics remains as pertinent as ever, and in the last five centuries (especially the last two) it has been joined by the related question of what mathematics most members of the public should know.

Why does mathematics matter to society at large? Listen to mathematicians, policymakers, and educators and the answer seems unanimous: mathematics is everywhere, therefore everyone should care about it. Books and articles abound with examples of the math that their authors claim is hidden in every facet of everyday life or unlocks powerful truths and technologies that shape the fates of individuals and nations. Take math professor Jordan Ellenberg, author of the bestselling book How Not to Be Wrong, who asserts “you can find math everywhere you look.”

To be sure, numbers and measurement figure regularly in most people’s lives, but this risks conflating basic numeracy with the kind of math that most affects your life. When we talk about math in public policy, especially the public’s investment in mathematical training and research, we are not talking about simple sums and measures. For most of its history, the mathematics that makes the most difference to society has been the province of the exceptional few. Societies have valued and cultivated math not because it is everywhere and for everyone but because it is difficult and exclusive. Recognizing that math has elitism built into its historical core, rather than pretending it is hidden all around us, furnishes a more realistic understanding of how math fits into society and can help the public demand a more responsible and inclusive discipline.

In the first agricultural societies in the cradle of civilization, math connected the heavens and the earth. Priests used astronomical calculations to mark the seasons and interpret divine will, and their special command of mathematics gave them power and privilege in their societies. As early economies grew larger and more complex, merchants and craftsmen incorporated more and more basic mathematics into their work, but for them mathematics was a trick of the trade rather than a public good. For millennia, advanced math remained the concern of the well-off, as either a philosophical pastime or a means to assert special authority.

The first relatively widespread suggestions that anything beyond simple practical math ought to have a wider reach date to what historians call the Early Modern period, beginning around five centuries ago, when many of our modern social structures and institutions started to take shape. Just as Martin Luther and other early Protestants began to insist that Scripture should be available to the masses in their own languages, scientific writers like Welsh polymath Robert Recorde used the relatively new technology of the printing press to promote math for the people. Recorde’s 1543 English arithmetic textbook began with an argument that “no man can do any thing alone, and much less talk or bargain with another, but he shall still have to do with number” and that numbers’ uses were “unnumerable” (pun intended).

Far more influential and representative of this period, however, was Recorde’s contemporary John Dee, who used his mathematical reputation to gain a powerful position advising Queen Elizabeth I. Dee hewed so closely to the idea of math as a secret and privileged kind of knowledge that his detractors accused him of conjuring and other occult practices. In the seventeenth century’s Scientific Revolution, the new promoters of an experimental science that was (at least in principle) open to any observer were suspicious of mathematical arguments as inaccessible, tending to shut down diverse perspectives with a false sense of certainty. During the eighteenth-century Enlightenment, by contrast, the savants of the French Academy of Sciences parlayed their mastery of difficult mathematics into a special place of authority in public life, weighing in on philosophical debates and civic affairs alike while closing their ranks to women, minorities, and the lower social classes.

Societies across the world were transformed in the nineteenth century by wave after wave of political and economic revolution, but the French model of privileged mathematical expertise in service to the state endured. The difference was in who got to be part of that mathematical elite. Being born into the right family continued to help, but in the wake of the French Revolution successive governments also took a greater interest in primary and secondary education, and strong performance in examinations could help some students rise despite their lower birth. Political and military leaders received a uniform education in advanced mathematics at a few distinguished academies which prepared them to tackle the specialized problems of modern states, and this French model of state involvement in mass education combined with special mathematical training for the very best found imitators across Europe and even across the Atlantic. Even while basic math reached more and more people through mass education, math remained something special that set the elite apart. More people could potentially become elites, but math was definitely not for everyone.

Entering the twentieth century, the system of channeling students through elite training continued to gain importance across the Western world, but mathematics itself became less central to that training. Partly this reflected the changing priorities of government, but partly it was a matter of advanced mathematics leaving the problems of government behind. Where once Enlightenment mathematicians counted practical and technological questions alongside their more philosophical inquiries, later modern mathematicians turned increasingly to forbiddingly abstract theories without the pretense of addressing worldly matters directly.

The next turning point, which continues in many ways to define the relations between math and society today, was World War II. Fighting a war on that scale, the major combatants encountered new problems in logistics, weapons design and use, and other areas that mathematicians proved especially capable of solving. It wasn’t that the most advanced mathematics suddenly got more practical, but that states found new uses for those with advanced mathematical training and mathematicians found new ways to appeal to states for support. After the war, mathematicians won substantial support from the United States and other governments on the premise that regardless of whether their peacetime research was useful, they now had proof that highly trained mathematicians would be needed in the next war.

Some of those wartime activities continue to occupy mathematical professionals, both in and beyond the state—from security scientists and code-breakers at technology companies and the NSA to operations researchers optimizing factories and supply chains across the global economy. Postwar electronic computing offered another area where mathematicians became essential. In all of these areas, it is the special mathematical advances of an elite few that motivate the public investments mathematicians continue to receive today. It would be great if everyone were confident with numbers, could write a computer program, and evaluate statistical evidence, and these are all important aims for primary and secondary education. But we should not confuse these with the main goals and rationales of public support for mathematics, which have always been about math at the top rather than math for everyone.

Imagining math to be everywhere makes it all too easy to ignore the very real politics of who gets to be part of the mathematical elite that really count—for technology, security, and economics, for the last war and the next one. Instead, if we see that this kind of mathematics has historically been built by and for the very few, we are called to ask who gets to be part of that few and what are the responsibilities that come with their expertise. We have to recognize that elite mathematics today, while much more inclusive than it was one or five or fifty centuries ago, remains a discipline that vests special authority in those who, by virtue of gender, race, and class, are often already among our society’s most powerful. If math were really everywhere, it would already belong to everyone equally. But when it comes to accessing and supporting math, there is much work to be done. Math isn’t everywhere.