The Cambridge Analytica scandal is more than a “breach,” as Facebook executives have defined it. It exemplifies the possibility of using online data to algorithmically predict and influence human behavior in a manner that circumvents users’ awareness of such influence. Using an intermediary app, Cambridge Analytica was able to harvest large data volumes—over 50 million raw profiles—and use big data analytics to create psychographic profiles in order to subsequently target users with customized digital ads and other manipulative information. According to some observers, this massive data analytics tactic might have been used to purposively swing election campaigns around the world. The reports are still incomplete and more is likely to come to light in the next days.

Although different in scale and scope, this scandal is not entirely new. In 2014, Facebook conducted a colossal online psychosocial experiment with researchers at Cornell University on almost seven hundred thousand unaware users, algorithmically modifying their newsfeeds to observe changes in their emotions. The study results, published in the prestigious Proceedings of the National Academy of Sciences (PNAS), showed the ability of the social network to make people happier or sadder on a massive scale and without their awareness—a phenomenon that was labeled “emotional contagion.” As in the Cambridge Analytica case, Facebook’s emotional contagion study sparked harsh criticism, with experts calling for new standards of oversight and accountability for social-computing research.

A common lesson from these two different cases is that Facebook’s privacy policy is no absolute guarantee of data protection: in 2014, it allowed the reuse of data for research purposes even though “research” was not listed in the company’s Data Use Policy at the time of data collection. A couple of years later, it allowed an abusive app to collect data not only on users who signed up for it, but also on their friends. Mark Zuckerberg himself has admitted that the data were not protected as they should have been. In trying to make sense of this scandal, however, there are two more subtle considerations that go beyond data protection:

First, accepting terms of service (ToS) and privacy policies (PP) is a prerequisite for using most online services, Facebook included. Nonetheless, it is no secret that most people accept ToSs without even scrolling to the end of the page. This well-known phenomenon raises the question of whether online agreements qualify as informed consent. Berggruen Prize winner Onora O’Neill has argued that “the point of consent procedures is to limit deception and coercion,” hence they should be designed to give people “control over the amount of information they receive and opportunity to rescind consent already given.”

Online services, from the most mainstream like Facebook and Twitter to the most shady like Cambridge Analytica, seem to do exactly the opposite. As a German Court has recently ruled, there is no guarantee that people are sufficiently informed about Facebook’s privacy-related options before registering for the service, hence informed consent might be undermined. On top of that, the platform hosts activities that use online manipulation to reduce people’s rational control over the information they generate or receive, be it in the form of micro-targeted advertising or fake-news-spreading social bots. Instead of being limited, deception is normalized.

In this ever-evolving online environment characterized by weakened consent, conventional data protection measures might be insufficient. What data are used for is unlikely to be controlled by the users who provided the data in the first place. Data access boards, monitoring boards and other mechanisms have a better chance to control and respond to undesirable uses. Such mechanisms have to be part of a more systemic oversight plan that spreads throughout the continuum of regulatory activities and responds to unexpected events across the life cycle of data uses. This approach can target new types of risk and emerging forms of vulnerability arising in the online data ecosystem. 

The second consideration is that not just Cambridge Analytica, but most of the current online ecosystem, is an arm’s race to the unconscious mind: notifications, microtargeted ads, autoplay plugins, are all strategies designed to induce addictive behavior, hence to manipulate. Researchers have called for adaptive regulatory frameworks that can limit information extraction from and modulation of someone’s mind using experimental neurotechnologies. Social computing shows that you don’t necessarily have to read people’s brains to influence their choices. It is sufficient to collect and mine the data they regularly—and often unwittingly—share online.

Therefore we need to consider whether we should set for the digital space a firm threshold for cognitive liberty. Cognitive liberty highlights the freedom to control one’s own cognitive dimension (including preferences, choices and beliefs) and to be protected from manipulative strategies that are designed to bypass one’s cognitive defenses. This is precisely what Cambridge Analytica’s attempted to do, as their managing director revealed during an undercover investigation by Channel 4 News: the company’s aim, he admitted, is to “to take information onboard effectively” using “two fundamental human drivers,” namely “hopes and fears,” which are often “unspoken or even unconscious.”

Attempts to manipulate other people’s unconscious mind and associated behavior are as old as human history. In Ancient Greece, Plato warned against demagogues: political leaders who build consensus by appealing to popular desires and prejudices instead of rational deliberation. However, the only tool demagogues ancient Athens could use to bypass rational deliberation was the art of persuasion.

In today’s digital ecosystem, wannabe demagogues can use big data analytics to uncover cognitive vulnerabilities from large user datasets and effectively exploit them in a manner that bypasses individual rational control. For example, machine learning can be used to identify deep-rooted fears among pre-profiled user groups which social-media bots can subsequently exploit to foment anger and intolerance.

The recently adopted EU General Data Protection Regulation, with its principle of purpose limitation (data collectors are required to specify the purpose of collecting personal information at the time of collection) is likely to partly defuse the current toxic digital environment. However, determining where persuasion ends and manipulation begins, is a question which goes, as recently admitted by the European Data Protection Supervisor (EDPS), “well beyond the right to data protection.”

The EDPS has underscored that microtargeting and other online strategies “point towards a culture of manipulation in the online environment” in which “most individuals are unaware of how they are being used.” If recklessly applied to the electoral domain, they could even change reduce “the space for debate and interchange of ideas,” a risk which “urgently requires a democratic debate on the use and exploitation of data for political campaign and decision-making.” Last year, international experts have addressed the question of whether democracy will survive big data and artificial intelligence. The answer will partly depend on how we govern data flows and protect the liberty of the individual mind.