To summarize: models should be used with parsimony–that is, only when we really need them–and their role should not [be] exaggerated. Models can be useful to organize the data and clarify simple logical relations between basic concepts; but they cannot replace the historical narrative, which in my view must be the real core of the analysis (and which I consider to be the core of my book). The complexity and multidimensionality of historical, social, and political processes in real-world societies are so great that there is no way they can be adequately described by mathematical language alone: one needs to use primarily the natural language of the social sciences (and sometimes the language of literature and movies, which, as I try to show in my book, can be viewed as an additional and complementary way to grasp social and historical realities, just like mathematical language).-Thomas Piketty, in his contribution to After Piketty: The agenda for economics and inequality, p. 554
I’ve been posting a bit lately about data and theory, and the other week I excerpted the Stanford Encyclopedia of Philosophy’s entry on big data and science. I want to return to that topic through the lens of economics.
In short, the proliferation of data can be thought of as an economic shock and basic economic theory would then predict that it would play a greater role in science.
In an article that became the book Prediction Machines, economists Ajay Agrawal, Joshua Gans, and Avi Goldfarb talk about AI as a drop in the cost of prediction:
Technological revolutions tend to involve some important activity becoming cheap, like the cost of communication or finding information… When the cost of any input falls so precipitously, there are two other well-established economic implications. First, we will start using prediction to perform tasks where we previously didn’t. Second, the value of other things that complement prediction will rise…
As a historical example, consider semiconductors, an area of technological advance that caused a significant drop in the cost of a different input: arithmetic. With semiconductors we could calculate cheaply, so activities for which arithmetic was a key input, such as data analysis and accounting, became much cheaper. However, we also started using the newly cheap arithmetic to solve problems that were not historically arithmetic problems. An example is photography. We shifted from a film-oriented, chemistry-based approach to a digital-oriented, arithmetic-based approach. Other new applications for cheap arithmetic include communications, music, and drug discovery.
What does that mean for science and the role of data? As the cost of collecting data drops, scientists will use it more. For example, as the Stanford entry suggests, some see data-driven exploration as a substitute for traditional methods of hypothesis generation. If that’s the case, economic theory would expect the former to become more common and the latter less so. But what about theory? Most people would say theory is a complement to data, not a substitute, in which case its value should rise. This offers a sort of a synthesis position between advocates of data and theory at present: data-driven methods will and should become more common. But that shift makes theory more important, not less.
Obviously, this is all super speculative. Just thinking through the analogy.
Benjamin Peirce (father of Charles Sanders Peirce) on induction, deduction, and the role of math in the scientific process:
Observation supplies fact. Induction ascends from fact to law. Deduction, by applying the pure logic of mathematics, reverses the process and descends from law to fact. The facts of observation are liable to the uncertainties and inaccuracies of the human senses; and the first inductions of law are rough approximations to the truth. The law is freed from the defects of observation and converted by the speculations of the geometer into exact form. But it has ceased to be pure induction, and has become ideal hypothesis. Deductions are made from it with syllogistic precision, and consequent facts are logically evolved without immediate reference to the actual events of Nature. If the results of computation coincide, not merely qualitatively but quantitatively, with observation, the law is established as a reality, and is restored to the domain of induction.
Via The Metaphysical Club, p. 155-156. The original text is from 1881.
First up, “A problem in theory,” an essay from 2019 blaming the replication crisis in psychology research largely on lack of theory:
The replication crisis facing the psychological sciences is widely regarded as rooted in methodological or statistical shortcomings. We argue that a large part of the problem is the lack of a cumulative theoretical framework or frameworks. Without an overarching theoretical framework that generates hypotheses across diverse domains, empirical programs spawn and grow from personal intuitions and culturally biased folk theories. By providing ways to develop clear predictions, including through the use of formal modelling, theoretical frameworks set expectations that determine whether a new finding is confirmatory, nicely integrating with existing lines of research, or surprising, and therefore requiring further replication and scrutiny. Such frameworks also prioritize certain research foci, motivate the use diverse empirical approaches and, often, provide a natural means to integrate across the sciences. Thus, overarching theoretical frameworks pave the way toward a more general theory of human behaviour. We illustrate one such a theoretical framework: dual inheritance theory.
Second, the Stanford Encylopedia of Philosophy’s entry on big data and scientific research (long quote coming):
6. Big Data, Knowledge and Inquiry
Let us now return to the idea of data-driven inquiry, often suggested as a counterpoint to hypothesis-driven science (e.g., Hey et al. 2009). Kevin Elliot and colleagues have offered a brief history of hypothesis-driven inquiry (Elliott et al. 2016), emphasising how scientific institutions (including funding programmes and publication venues) have pushed researchers towards a Popperian conceptualisation of inquiry as the formulation and testing of a strong hypothesis. Big data analysis clearly points to a different and arguably Baconian understanding of the role of hypothesis in science. Theoretical expectations are no longer seen as driving the process of inquiry and empirical input is recognised as primary in determining the direction of research and the phenomena—and related hypotheses—considered by researchers.
The emphasis on data as a central component of research poses a significant challenge to one of the best-established philosophical views on scientific knowledge. According to this view, which I shall label the theory-centric view of science, scientific knowledge consists of justified true beliefs about the world. These beliefs are obtained through empirical methods aiming to test the validity and reliability of statements that describe or explain aspects of reality. Hence scientific knowledge is conceptualised as inherently propositional: what counts as an output are claims published in books and journals, which are also typically presented as solutions to hypothesis-driven inquiry. This view acknowledges the significance of methods, data, models, instruments and materials within scientific investigations, but ultimately regards them as means towards one end: the achievement of true claims about the world. Reichenbach’s seminal distinction between contexts of discovery and justification exemplifies this position (Reichenbach 1938). Theory-centrism recognises research components such as data and related practical skills as essential to discovery, and more specifically to the messy, irrational part of scientific work that involves value judgements, trial-and-error, intuition and exploration and within which the very phenomena to be investigated may not have been stabilised. The justification of claims, by contrast, involves the rational reconstruction of the research that has been performed, so that it conforms to established norms of inferential reasoning. Importantly, within the context of justification, only data that support the claims of interest are explicitly reported and discussed: everything else—including the vast majority of data produced in the course of inquiry—is lost to the chaotic context of discovery.
Much recent philosophy of science, and particularly modelling and experimentation, has challenged theory-centrism by highlighting the role of models, methods and modes of intervention as research outputs rather than simple tools, and stressing the importance of expanding philosophical understandings of scientific knowledge to include these elements alongside propositional claims. The rise of big data offers another opportunity to reframe understandings of scientific knowledge as not necessarily centred on theories and to include non-propositional components—thus, in Cartwright’s paraphrase of Gilbert Ryle’s famous distinction, refocusing on knowing-how over knowing-that (Cartwright 2019). One way to construe data-centric methods is indeed to embrace a conception of knowledge as ability, such as promoted by early pragmatists like John Dewey and more recently reprised by Chang, who specifically highlighted it as the broader category within which the understanding of knowledge-as-information needs to be placed (Chang 2017).
Another way to interpret the rise of big data is as a vindication of inductivism in the face of the barrage of philosophical criticism levelled against theory-free reasoning over the centuries. For instance, Jon Williamson (2004: 88) has argued that advances in automation, combined with the emergence of big data, lend plausibility to inductivist philosophy of science. Wolfgang Pietsch agrees with this view and provided a sophisticated framework to understand just what kind of inductive reasoning is instigated by big data and related machine learning methods such as decision trees (Pietsch 2015). Following John Stuart Mill, he calls this approach variational induction and presents it as common to both big data approaches and exploratory experimentation, though the former can handle a much larger number of variables (Pietsch 2015: 913). Pietsch concludes that the problem of theory-ladenness in machine learning can be addressed by determining under which theoretical assumptions variational induction works (2015: 910ff).
Others are less inclined to see theory-ladenness as a problem that can be mitigated by data-intensive methods, and rather see it as a constitutive part of the process of empirical inquiry. Arching back to the extensive literature on perspectivism and experimentation (Gooding 1990; Giere 2006; Radder 2006; Massimi 2012), Werner Callebaut has forcefully argued that the most sophisticated and standardised measurements embody a specific theoretical perspective, and this is no less true of big data (Callebaut 2012). Elliott and colleagues emphasise that conceptualising big data analysis as atheoretical risks encouraging unsophisticated attitudes to empirical investigation as a
“fishing expedition”, having a high probability of leading to nonsense results or spurious correlations, being reliant on scientists who do not have adequate expertise in data analysis, and yielding data biased by the mode of collection. (Elliott et al. 2016: 880)
To address related worries in genetic analysis, Ken Waters has provided the useful characterisation of “theory-informed” inquiry (Waters 2007), which can be invoked to stress how theory informs the methods used to extract meaningful patterns from big data, and yet does not necessarily determine either the starting point or the outcomes of data-intensive science. This does not resolve the question of what role theory actually plays. Rob Kitchin (2014) has proposed to see big data as linked to a new mode of hypothesis generation within a hypothetical-deductive framework. Leonelli is more sceptical of attempts to match big data approaches, which are many and diverse, with a specific type of inferential logic. She rather focused on the extent to which the theoretical apparatus at work within big data analysis rests on conceptual decisions about how to order and classify data—and proposed that such decisions can give rise to a particular form of theorization, which she calls classificatory theory (Leonelli 2016).
These disagreements point to big data as eliciting diverse understandings of the nature of knowledge and inquiry, and the complex iterations through which different inferential methods build on each other. Again, in the words of Elliot and colleagues,
attempting to draw a sharp distinction between hypothesis-driven and data-intensive science is misleading; these modes of research are not in fact orthogonal and often intertwine in actual scientific practice. (Elliott et al. 2016: 881, see also O’Malley et al. 2009, Elliott 2012)
Vox’s Future Perfect newsletter reports:
“Just carefully reading a paper — even as a layperson without deep knowledge of the field — is sufficient to form a pretty accurate guess about whether the study will replicate.
Meanwhile, DARPA’s replication markets found that guessing which papers will hold up and which won’t is often just a matter of looking at whether the study makes any sense. Some important statistics to take note of: Did the researchers squeeze out a result barely below the significance threshold of p = 0.05? (A paper can often claim a “significant” result if this threshold is met, and many use various statistical tricks to push their paper across that line.) Did they find no effects in most groups but significant effects for a tiny, hyper-specific subgroup?
“Predicting replication is easy,” Menard writes. “There’s no need for a deep dive into the statistical methodology or a rigorous examination of the data, no need to scrutinize esoteric theories for subtle errors—these papers have obvious, surface-level problems.”
This is important work and I get the point but in a way it’s studying things backwards. It’s assessing whether laypeople can do better than random at predicting which studies will replicate, which, again, is important. But the test of the studies’ usefulness is really whether they can help people improve their judgments, not the other way around.
The study I’d like to see would work like this: A group of people is asked to predict the result of a forthcoming study which, unbeknownst to them, is a replication of a past study. They’re asked to predict the effect that some intervention has on some outcome variable. One group, the control, makes this prediction just based on their knowledge of the world. The other group, the treatment, gets access to the original study. They can read it, see its result and methodology, and then incorporate that (if they want to) in making their prediction.
Would access to the original studies improve peoples’ predictions?
In late 2009, a little more than a decade ago, I started this blog. It wasn’t my first time blogging, but it was my first sustained effort and I might have started it sooner if it hadn’t been for the difficulty of picking a suitable name. After far too much deliberation, someone close to me suggested “Beyond the times,” which I liked because it captured my interest in the media and in the future.
My subject, as I announced it to quite literally no one, was to cover “The Internet, Information, and the Public Sphere.”
I’ve written more than 350 posts over the intervening years, and a lot has changed since then. When I started, I was writing about the media from outside of it. But almost a year after launching the blog, I wrote a post about dating algorithms, in response to a piece on The Atlantic’s newly launched tech vertical. That led to some contributions to the section, which led to a job reporting on tech for a news startup, which led to jobs at HBR and now Quartz.
Since joining the media eight years ago, I’ve written less about it. I still have lots of opinions about media and journalism, of course. But my writing has focused on innovation and the economy, and today I’m renaming the blog to reflect that.
This blog’s name is now Nonrival, to reflect my current focus on economics and my continued interest in information and innovation.
Most economic goods are “rivalrous,” meaning if one person consumes them then another person can’t. If you and I have an apple, you can eat it or I can eat it or we can split it. We can’t both eat the whole apple. But nonrivalrous goods are different. If I share an idea with you, we both get to enjoy it. If you share it with someone else, it doesn’t take anything away from me. Digital goods are nonrival.* A Netflix episode is more like an idea than an apple. The new name captures my focus not just on the internet but its economic effects.
And to the extent that “nonrival” has any meaning colloquially, it’s one I like, too. One of the topics I blogged about most in the early days was collaboration, and “nonrival” gets at some of that spirit.
I’m hoping to add a Nonrival newsletter soon, too. You can sign up in advance here.
*OK, sure, not totally. Server space and various other physical goods that support digital ones may be rivalrous.
Just a post to clip together some resources…
In this article, we build on research on social change,1 including our own research,2 for which we studied hundreds of social change initiatives over multiple years and interviewed social entrepreneurs, civil society leaders, and public officials around the world. We identify three distinct roles played by those who participate in movements for social change: agitator, innovator, and orchestrator. An agitator brings the grievances of specific individuals or groups to the forefront of public awareness. An innovator creates an actionable solution3 to address these grievances. And an orchestrator coordinates action across groups, organizations, and sectors to scale the proposed solution. Any pathway to social change requires all three. Agitation without innovation means complaints without ways forward, and innovation without orchestration means ideas without impact.
Cass Sunstein’s book, How Change Happens.
Here’s political scientist Henry Farrell on Tyler Cowen’s podcast:
FARRELL: Analytic Marxism, I think, is underrated. I think it’s going to come back. Now, when I say analytic Marxism, let me be specific about that. There’s a lot of Marx which I think is flat-out wrong. And the analytic Marxist project itself drove itself into a hole. If you look at people like [Jon] Elster, the other people who are strongly committed to it, they eventually ended up figuring that if you wanted to make sense of Marx, you’re going to come to the conclusion that eventually there wasn’t much point, there wasn’t much sense to be made of it.
But nonetheless, the basic underlying idea, which is that if you bring together rationalist perspectives with a direct concern with power relations and a desire to understand power relations in the Marxist and the Weberian way, that this is something which is coming to the fore again, which is extremely valuable.
Somebody else who I think is underrated in this context is Mancur Olson. For example, if you want to understand where Elizabeth Warren is going, you want to go back to Mancur Olson’s book, The Rise and Decline of Nations, because I think what Elizabeth Warren is pursuing is very much an Olsonian view of how markets work: that drag and dross and corruption builds up and that in order to allow markets to achieve their full potential, you basically need to cleanse them at a certain point.
I picked up Olson’s The Rise and Decline of Nations on that recommendation, and want to post a few notes from it. He helpfully summarizes his own argument on page 74 (more books should do this!):
- There will be no countries that attain symmetrical organization of all groups with a common interest and thereby attain optimal outcomes through comprehensive bargaining.
- Stable societies with unchanged boundaries ten to accumulate more collusions and organizations for collective action over time.
- Members of ‘small’ groups have disproportionate organizational power for collective action, and this disproportion diminishes but does not disappear over time in stable societies.
- On balance, special-interest organizations and collusions reduce efficiency and aggregate income in the societies in which they operate and make political life more divisive.
- Encompassing organizations have some incentive to make the society in which they operate more prosperous, and an incentive to redistribute income to their members with as little excess burden as possible, and to cease such redistribution unless the amount redistributed is substantial in relation to the social cost of the redistribution.
- Distributional coalitions make decisions more slowly than the individuals and firms of which they are comprised, tend to have crowded agendas and bargaining tables, and more often fix prices than quantities.
- Distributional coalitions slow down a society’s capacity to adopt new technologies and to reallocate resources in response to changing conditions, and thereby reduce the rate of economic growth.
- Distributional coalitions, once big enough to succeed, are exclusive, and seek to limit the diversity of incomes and values of their membership.
- The accumulation of distributional coalitions increases the complexity of regulation, the role of government, and the complexity of understandings, and changes the direction of social evolution.
The typical organization for collective action will do nothing to eliminate the social loss or ‘public bad’ its effort to get a larger share of the social output brings about. The familiar image of the slicing of the social pie does not really capture the essence of the situation; it is perhaps better to think of wrestlers struggling over the contents of a china shop. (p. 43-44)
To borrow an evocative phrase from Marx, there is an “internal contradiction” in the development of stable societies. This is not the contradiction that Marx claimed to have found, but rather an inherent conflict between the colossal economic and political advantages of peace and stability and the longer-term losses that come from the accumulating networks of distributional coalitions that can survive only in stable environments. (p. 145)
Pretty much every step in this argument can be contested, starting with Olson’s core idea that collective action is harder with larger groups. Nonetheless, it was a very worthwhile recommendation.
From Hans Morgenthau’s classic text on international relations, Politics among Nations: The Struggle for Power and Peace:
The theory, in other words, must be judged not by some preconceived abstract principle or concept unrelated to reality, but by its purpose: to bring order and meaning to a mass of phenomena which without it would remain disconnected and unintelligible. It must meet a dual test, an empirical and a logical one: Do the facts as they actually are lend themselves to the interpretation the theory has put upon them, and do the conclusions at which the theory arrives follow with logical necessity from its premises? In short, is the theory consistent with the facts and within itself?
Here are a few posts I’ve done on theories, models, and evidence:
In the new Foreign Affairs, Felix Salmon reviews Darkness by Design: The Hidden Power in Global Capital Markets, by political scientist Walter Mattli. Salmon and Mattli share the view that more competition among stock exchanges has been bad for financial markets. Here’s Salmon:
Up until that point, the exchange was a mutual society: firms could buy seats, and the exchange was owned by its members. After 2005, it demutualized, stopped selling seats, and became just one among many exchanges, most of which were owned and operated by enormous global broker-dealers–think Credit Suisse, Goldman Sachs, and Merrill Lynch–that had spent limitless hours and dollars on lobbying the SEC to push Reg NMS through. Rather than being a utility owned by its members, the NYSE was now a profit-maximizing entity like all the other exchanges.
There’s a parallel here to today’s tech platforms. They’re big and powerful, and some argue that they should be broken up. But would more (but smaller) platforms be a good thing? Would competition help?
Salmon and Mattlie argue that having a marketplace (the stock market) competing in a market of its own (the market for trades) has lots of downsides. Whatever you think of that in the context of stock exchanges, it’s worth considering for tech. Would competition between mini-Facebooks shift power toward advertisers and, as a result, further erode privacy? What might the mini-Facebooks do in order to win the business of key publishers or to gain access to particular markets?
Of course, none of the tech platforms are currently run as mutual societies. And so the conversation tends to be about either breaking them up to encourage competition, or regulating them as utilities.
But, just for the fun of it, imagine what a mutual society model might look like. What if Twitter were run for the benefit of its users, with major publishers “buying seats” and individual users electing representatives to advance their interests? You can imagine all sorts of reasons that might not work. (A version of this has been proposed.) But as Salmon and Mattlie suggest, counting on competition between platforms isn’t always a good thing either.