When AI Takes Over Scientific Discovery: Will We Know What Works Without Knowing Why?
If autonomous AI delivers a molecule that cures disease or a recipe for room-temperature superconductivity, but we cannot understand the causal mechanism, is that a scientific triumph or a high-dimensional version of making fire by trial and error?
science
In protein-structure prediction, crystal screening, and turbulence simulation, large models and autonomous agents are demonstrating remarkable empirical capabilities. Given a sequence or a set of boundary conditions, they can produce detailed physical structures or experimental recipes. Alongside this leap in efficiency comes an unsettling philosophical question: if AI eventually makes most scientific discoveries autonomously, while humans receive only the results and cannot understand how billions of floating-point weights produced them, will our epistemic position resemble a technologically advanced version of a pre-scientific society?
The question cuts directly into modern science’s aspirations. To assess it without prejudging the answer, we should put aside romantic accounts of scientific progress and examine the historical record. Over the past several centuries, did we first understand nature’s laws and then build tools? Or did we first stumble upon effective recipes in the dark, without knowing why they worked?
1. High-Dimensional Fire-Making: When Black-Box Prediction Outruns Causal Understanding
Ancient people used fire, fermented grain, and smelted bronze without knowing about free-radical oxidation chains, anaerobic yeast metabolism, or dislocations in metallic lattices. They possessed empirical mappings accumulated through costly trial and error: mix tin and copper ores in particular proportions, heat them until the flame has a certain color, and obtain a hard blade; boil willow bark and drink the liquid to relieve fever and pain. They could reliably use natural phenomena while nature itself remained a black box.
Today, a biologist may ask a gene-regulation foundation model to design a cell-type-specific enhancer, or a materials scientist may use a graph neural network to screen millions of candidates for stable inorganic crystals (Nature, 1). A subtle symmetry appears. The model supplies sequences or compositions that can withstand experimental testing, yet when a researcher asks why substituting adenine at nucleotide 42 resists chromatin silencing, attention matrices spread across dozens of layers rarely yield a compact causal explanation a human can grasp.
Viewed only as an information structure—enter an instruction, obtain the expected physical result, and remain unable to explain the mechanism—a modern laboratory heavily dependent on an AI oracle resembles an ancient craft workshop tending a fire. The difference seems to be that earlier artisans needed generations of accidental experimentation to establish local knowledge, whereas AI can explore vast parameter spaces in days using high-dimensional tensor operations.
But concluding that AI will reduce science to primitive sorcery would be premature. Industrial and scientific history shows that knowing what works without knowing why is not peculiar to ancient societies. It has repeatedly been the leading edge of modern engineering and production.
2. The Historical Ledger: How Much Progress Came From Theory, and How Much From Experimentation?
Textbooks often present a standard sequence: physicists and chemists derive fundamental laws on blackboards, then engineers turn those laws into steam engines, chemical plants, oil-recovery methods, and medicines. Historical records frequently reverse this linear model.
A particularly revealing example is surfactant flooding, a major form of tertiary or enhanced oil recovery (EOR). After extraction using natural reservoir pressure and then water injection, more than half the oil may remain trapped by capillary forces as small disconnected droplets in microscopic rock pores. In the 1960s and 1970s, with energy shortages mounting, petroleum engineers began large-scale field trials adding petroleum sulfonates and other surfactants, together with polymers, to injection water (Journal of Petroleum Technology, 2).
Field engineers understood the practical effect: empirical phase-behavior experiments, including formulations producing Winsor III microemulsions, could lower oil–water interfacial tension by orders of magnitude and mobilize residual oil (SPE Journal, 3). Yet for decades, their picture of the microscopic processes inside real porous rock remained incomplete. Why did formulations that performed well in laboratory core floods sometimes fail in heterogeneous reservoirs? Dynamic surfactant adsorption on rough rock, the suppression of capillary snap-off by viscoelastic fluids, and multiphase percolation through complex pore networks became far more accessible to detailed physical analysis with synchrotron X-ray microtomography and pore-scale numerical simulation in the early twenty-first century (Advances in Water Resources, 4). The practical act of adding something like detergent to water to displace oil preceded this detailed microscopic picture by several decades.
Examples of engineering preceding theoretical understanding recur throughout technological history:
- Steam engines and thermodynamics. Thomas Newcomen built a practical atmospheric engine for pumping water out of mines in 1712, and James Watt’s separate condenser followed in 1769. Sadi Carnot did not publish Reflections on the Motive Power of Fire until 1824, and he still worked within the mistaken caloric theory. Clausius and Lord Kelvin developed the foundations of modern thermodynamics in the 1850s. Steam engines had already been operating for more than a century without a correct thermodynamic theory. As the biochemist Lawrence Henderson observed, before 1850 the steam engine had contributed more to science than science had contributed to the steam engine.
- Aspirin and general anesthesia. Felix Hoffmann at Bayer synthesized acetylsalicylic acid in 1897. Seventy-four years later, John Vane’s experiments linked aspirin’s action to inhibition of prostaglandin synthesis, a mechanism associated with cyclooxygenase (COX) inhibition (Nature New Biology, 5). Ether anesthesia had entered surgery in 1846, while molecular accounts of how volatile anesthetics act on targets such as GABAA receptors and two-pore-domain potassium channels emerged much later, toward the end of the twentieth century and into the twenty-first (Nature Reviews Neuroscience, 6).
To quantify how much technological progress depended on basic scientific understanding, the U.S. Department of Defense launched Project Hindsight in the mid-1960s. Researchers traced 710 important technical events behind 20 modern weapons systems, using a retrospective window of roughly 20 years. Results published in Science in 1967 were striking: about 91% of the events came from mission-driven technological development and empirical improvements, about 8.7% from applied science, and only 0.3% directly from recent undirected basic research (7).
The National Science Foundation responded by commissioning the Illinois Institute of Technology Research Institute’s TRACES study—Technology in Retrospect and Critical Events in Science—in 1968. It examined five major innovations, including the electron microscope, magnetic ferrites, and oral contraceptives, over a period extending more than 50 years. The picture changed: over the longer horizon, undirected basic science accounted for roughly 70% of the foundational contributions. Such discoveries often needed 20 to 30 years to become tools that engineers would eventually take for granted.
Economic historian Joel Mokyr, in The Gifts of Athena: Historical Origins of the Knowledge Economy (Princeton University Press, 8), offers a useful framework for interpreting these apparently contradictory findings. He distinguishes propositional knowledge (Ω)—knowledge about what exists and why it behaves as it does—from prescriptive knowledge (λ): the recipes, procedures, and instructions for achieving a particular result.
**Concept Propositional and Prescriptive Knowledge** In Mokyr’s distinction, propositional knowledge (Ω) concerns the regularities and explanations of nature. Prescriptive knowledge (λ) consists of executable instructions: the artisan’s or engineer’s knowledge of how to obtain a desired effect.
Mokyr argues that before the eighteenth-century Industrial Enlightenment, much productive progress accumulated through trial-and-error additions to prescriptive knowledge. Such knowledge could develop independently of a broad theoretical foundation and support remarkable local achievements, from Chinese porcelain to medieval European cathedrals. Its weakness was rapidly diminishing returns. Once artisans exhausted the familiar parameters of a process, they lacked a systematic basis for choosing where to search next, and a technique could stagnate for centuries.
3. Nature Already Computes Itself: Why Simulate One Black Box With Another?
This brings us to a second question. Nature is already “computing” itself in real time: stars enact nuclear fusion and hydrostatic balance, while cells continuously enact thermodynamics and molecular dynamics. If we build a hundred-billion-parameter AI model to reproduce these phenomena, but understand neither the cell nor the model, what have we gained from this second-order black box?
Jorge Luis Borges’s 1946 miniature story On Exactitude in Science describes cartographers who pursue accuracy until their map is as large as the empire itself. Mountains coincide with mountains, buildings with buildings. Later generations find the map useless and leave it to decay. If AI were merely an equally incomprehensible, one-to-one copy of nature, it would indeed resemble that map.
But the engineering realities of physical and biological computation reveal three important asymmetries between an artificial black box and a natural one—even when we cannot interpret every internal model parameter.
First, irreversible experiments with atoms can become reversible probes with bits. Nature’s computations consume real matter, physical time, and sometimes living organisms. A compound administered to a person can cause real liver or kidney failure; a plasma disruption in a fusion device can damage real first-wall material. A model acting as a shadow simulator makes it possible to test counterfactual interventions without directly exposing those physical systems to each trial, and potentially much faster than the corresponding real-world process.
Second, models can connect information across scales and reduce noise. Measurements of nature are noisy and often destructive. Once a cell is lysed for sequencing, its subsequent trajectory cannot be observed. Seismic measurements record only sparse vibrations at the surface. Through self-supervised learning across fragmented datasets, AI can join disconnected local observations into a continuous, differentiable interpolation space.
Third, AI expands the search radius of prescriptive knowledge. When theory lags, engineers have traditionally been limited to modest experimental throughput—like petroleum engineers testing a dozen surfactant formulations in bottles each day. Autonomous scientific AI can act as a powerful generator of candidate recipes, accelerating the search for effective procedures by orders of magnitude.
Simulating nature with AI therefore has substantial engineering value even when the process remains partly opaque. The dimension of science under pressure is a different one: understanding nature.
4. The Ceiling of Empirical Alchemy: Why Recipes Without Understanding Eventually Hit Limits
If AI can keep supplying effective prescriptive knowledge, why not abandon the pursuit of our own theoretical understanding? Could we simply enjoy its technological output, much as ancient people benefited from fire without understanding combustion?
From the perspectives of complex systems and philosophy of science, that path has serious limits. If AI expands prescriptive knowledge far beyond the propositional knowledge humans can understand, three barriers emerge.
The first is catastrophic failure outside the training distribution. An empirical recipe does not necessarily reveal the limits of its own validity. Consider a surfactant formulation that improves recovery by 15% in a cool, low-salinity sandstone reservoir. Without understanding electrostatic double-layer screening and wettability changes, engineers might transfer it to a hot, highly saline carbonate reservoir, where precipitation could obstruct pores and cause losses worth tens of millions of dollars. A purely empirical model without physical constraints can be unreliable when conditions cross a phase boundary. Understanding causal invariants helps us recognize when a successful interpolation may become a dangerous extrapolation.
The second is the loss of composability and clear grounds for refutation. Karl Popper’s emphasis on criticism and David Deutsch’s account of good explanations in The Beginning of Infinity point toward the importance of theories that are difficult to vary arbitrarily: each component has a specific logical role. Compact scientific theories can also be combined across fields. Electromagnetism and statistical thermodynamics can be brought together in the study of black-body radiation; quantum mechanics and chemical bonding can be combined to explain semiconductor bands.
Two neural networks, each containing ten billion parameters, do not by themselves yield a third cross-scale theorem on a blackboard. Gregory Chaitin captured a related intuition through algorithmic information theory: “comprehension is compression” (Complexity, 9). Tycho Brahe’s volumes of planetary observations were data. Kepler compressed their patterns into three geometric laws. Newton’s gravitational law, F = GMm/r², connected celestial orbits and falling objects in a few symbols. If reproducing a phenomenon requires a representation nearly as complex as the phenomenon itself, it has not achieved the kind of compression that makes scientific understanding portable.
**Concept Epistemic Compression** The extraction of compact mathematical relations or causal mechanisms from high-dimensional observations or model representations—relations that can be combined with other knowledge and remain valid across different boundary conditions. Greater compression with fewer exceptions is a sign of deeper understanding.
The third is that the costs and responsibilities of engineering cannot be outsourced to probabilities. When AI proposes ten materials or drug candidates with equally impressive virtual scores, people must still choose which to take into a multibillion-dollar chemical plant or a human clinical trial. They bear the physical and legal consequences. Causal understanding remains an important foundation for safety margins and redundant protection; without it, large-scale deployment risks becoming an uncontrolled gamble.
5. A Moving Scientific Bottleneck: From Predicting Nature to Extracting Understanding From Models
The historical record and these epistemic limits allow a more layered answer to the opening question.
Autonomous scientific AI need not return humanity permanently to a pre-scientific state. It may, however, open a new golden age of empiricism—or high-dimensional alchemy. Historically, recipes often preceded theory by decades within particular crafts. Over the next ten to twenty years, AI could expand that gap across biology, materials, chemistry, and complex fluids simultaneously. We may increasingly encounter working materials, effective interventions, and accurate forecasts before textbooks can fully explain them.
This would not end science. It would change the central task of human scientists.
Nature already enacts its own complexity. Science’s distinctive ambition is to compress natural phenomena and model predictions into causal principles that the human mind can grasp.
For much of the past three centuries, limited computational power made solving complex systems from known theory a central bottleneck. We could write down the Schrödinger equation or the Navier–Stokes equations without being able to calculate a real protein’s folding or a typhoon’s trajectory from them.
As AI takes on more high-dimensional prediction, the hardest problem may move from forward prediction to reverse distillation. Scientists will face two black boxes: nature and models that have implicitly encoded aspects of nature in their weights. Compared with noisy, expensive, irreversible experiments on the world, the model is an accessible intermediate object. It can be paused, examined layer by layer, and subjected to repeated virtual ablation experiments.
Early examples of treating AI as an experimental object are already visible. Researchers associated with Princeton and DeepMind trained graph neural networks to predict many-body physical systems and aspects of dark-matter structure, then used symbolic regression to examine the learned message-passing representations. They extracted concise analytical expressions, including a relation that improved on an existing empirical formula (NeurIPS, 10). In single-cell and virtual-cell modeling, researchers are likewise exploring whether interpretability methods can recover readable gene-regulatory relationships from large learned representations.
As long as we want more than instructions delivered at the push of a button—as long as we need to combine ideas across disciplines and reason safely at unfamiliar boundaries—building human understanding will remain essential. An explosion of AI-generated discoveries could make that understanding one of science’s scarcest and most valuable achievements.
Key Takeaways
-
Engineering has often preceded explanation. Steam engines, aspirin, and surfactant-based oil recovery illustrate how prescriptive knowledge can advance decades before a fuller theoretical account. AI could greatly amplify this pattern across disciplines.
-
Simulation and understanding serve different purposes. AI can replace some costly physical trials with reversible virtual probes. Yet recipes alone leave unresolved problems of out-of-distribution behavior, combining knowledge across fields, and engineering accountability. Compact causal explanations remain valuable.
-
The scientist’s role may shift toward theory extraction. As AI improves prediction and candidate generation, a central challenge becomes studying models themselves and distilling their representations into understandable, falsifiable scientific relations.
References
- Scaling deep learning for materials discovery — Nature.
- Use of Micellar Solutions to Recover Oil — Journal of Petroleum Technology.
- Physicochemical Aspects of Microemulsion Flooding — SPE Journal.
- Advances in Water Resources — source on pore-scale flow.
- Inhibition of Prostaglandin Synthesis as a Mechanism of Action for Aspirin-like Drugs — Nature New Biology.
- General anaesthesia: from molecular targets to neuronal pathways of sleep and arousal — Nature Reviews Neuroscience.
- Project Hindsight — Science.
- The Gifts of Athena — Princeton University Press.
- Complexity — source on comprehension and compression.
- Discovering Symbolic Models from Deep Learning with Inductive Biases — arXiv / NeurIPS.