Drug Discovery

Drug discovery is the process by which new pharmaceutical compounds are identified, characterized, and developed from initial biological hypothesis to clinical candidate. It is the domain where molecular biology, pharmacology, organic chemistry, and clinical medicine converge — and it is a domain where the gap between scientific understanding and practical outcome has been, historically, more dramatic than in almost any other field of applied science.

The central fact about drug discovery is that it fails most of the time. The probability of a compound entering Phase I clinical trials eventually receiving regulatory approval is approximately 10%. The probability of a compound identified in early discovery research reaching the patient is closer to 1 in 10,000. These failure rates have not improved markedly over the past four decades despite enormous increases in mechanistic understanding, computational power, and the sophistication of screening technologies. Understanding why discovery fails so reliably is as important as understanding how it occasionally succeeds.

The History of How Drugs Were Actually Found

The received narrative of drug discovery presents it as an orderly progression from mechanistic understanding to therapeutic intervention: identify a disease pathway, find a molecular target in that pathway, design a compound that modulates the target, test it in cells, test it in animals, test it in humans. This rational drug design framework is the organizing ideology of pharmaceutical research since the 1980s.

It is largely false as a historical description. Most of the drugs that changed medicine were found by other routes.

Aspirin was in medical use as a folk remedy (willow bark) for centuries before its mechanism — inhibition of cyclooxygenase enzymes — was elucidated in 1971 by John Vane, who received a Nobel Prize for explaining what the compound had been doing all along. Penicillin was found by Alexander Fleming through careful observation of a fungal contamination, not through a mechanistic hypothesis about bacterial cell wall synthesis. The mechanism of beta-lactam antibiotics was worked out decades after the drugs were in clinical use. The statins — among the most prescribed drugs in history — were discovered by Akira Endo by screening microbial fermentation products for HMG-CoA reductase inhibition, a mechanism he chose to target based on understanding of cholesterol biosynthesis. This is closer to rational design, but it involved testing thousands of fungal extracts to find the active compound — a process that is more craft than algorithm.

The antidepressant revolution of the 1950s and 1960s was launched by compounds found through serendipity: iproniazid, developed as an antituberculosis agent, was observed to have mood-elevating effects in patients; imipramine, a phenothiazine derivative initially screened as an antipsychotic, was found by Roland Kuhn to have antidepressant rather than antipsychotic effects. The mechanism — monoamine oxidase inhibition in one case, tricyclic reuptake inhibition in the other — was understood only later. The SSRIs that followed in the 1980s represented genuine rational design, but they succeeded partly because the mechanistic framework had been retrospectively constructed from the earlier serendipitous discoveries.

Target-Based Drug Discovery and Its Limitations

The dominant framework in pharmaceutical research since the 1990s has been target-based drug discovery: identify a protein (or nucleic acid, or pathway) causally involved in a disease, develop high-throughput screening assays for compounds that modulate that target, optimize hit compounds through medicinal chemistry, and advance optimized leads through preclinical development. This approach has advantages: it is systematizable, amenable to automation, and generates mechanistic understanding alongside the compound.

Its limitation is fundamental. A target that is causally involved in disease pathology in a cell line, or in a mouse model, may not be druggable in the pharmacological sense; the compound that modulates it may not reach its target at therapeutic doses in a human; and even if it reaches the target and modulates it, the disease may not respond as the cellular and animal models predicted.

This last failure mode — target validation failure, or the gap between the model and the disease — is responsible for a substantial fraction of late-stage clinical failures and constitutes the deepest problem in contemporary drug discovery. Alzheimer's disease has been a case study in target validation failure: the amyloid hypothesis, which posited that beta-amyloid plaques cause neurodegeneration and that clearing them would halt progression, generated a large investment in compounds that successfully cleared amyloid in humans. The trials failed. Patients with cleared amyloid plaques did not recover or stabilize significantly better than controls. The target was modulated; the disease was not. Whether this means the hypothesis is wrong, the targets were wrong, the intervention timing was wrong, or the patient populations were wrong remains an active and deeply contested empirical question.

Phenotypic Screening: The Return to Empiricism

The recognition that target-based discovery has structural limitations has driven a partial return to phenotypic screening: testing compounds for their effects on cells, tissues, or organisms without requiring advance specification of the molecular target. This is closer to how most historical drugs were actually found — a compound that produces a desired cellular effect is identified, and the mechanism is worked out afterward.

Phenotypic screening has been most successful in areas where the relevant biological readout is well-defined and accessible: certain infectious diseases, cancer cell killing, neurological endpoints in model organisms. It is more difficult to apply in diseases where the relevant endpoint is not measurable in cultured cells or simple organisms.

The systems pharmacology approach attempts to integrate both frameworks: build computational models of disease-relevant biological networks, use those models to predict compound effects across multiple targets and pathways simultaneously, and use phenotypic screens to validate the predictions. This is conceptually attractive, and there are early successes. The limiting factor is model accuracy: biological networks are incompletely characterized, the parameters governing their dynamics are poorly measured, and the models that exist tend to be accurate for the well-studied parts of biology and unreliable for the parts that matter in the diseases we have not yet conquered.

The Economics of Discovery and Their Consequences

Drug discovery is not a purely scientific enterprise. It is conducted primarily by organizations — pharmaceutical companies, biotechnology companies, academic laboratories funded by commercial interests — with financial constraints that shape what gets discovered. This has well-documented consequences for the portfolio of diseases that receive discovery effort.

Diseases primarily affecting wealthy populations in wealthy countries receive disproportionate research investment relative to their global disease burden. Neglected tropical diseases affecting hundreds of millions of people in low-income countries receive a tiny fraction of the discovery investment that cardiovascular disease or cancer attracts, despite causing comparable or greater global burden. This is not primarily a scientific failure — the biology of these diseases is tractable. It is a market failure: the expected return on investment is insufficient to justify the cost.

The patent system that finances drug development creates a further structural bias: it incentivizes development of compounds that are patentable and can command high prices, which tends to favor novel chemical entities over repurposed generics and favors diseases where the patient population is large and wealthy. The result is a portfolio of drugs that is well-adapted to the commercial environment in which it was developed, not to the disease burden it nominally addresses.

Any serious account of drug discovery must grapple with the fact that the drugs we do not have are not primarily the result of scientific failure. They are partly the result of a discovery apparatus that is designed to find commercially viable drugs, not the most medically important ones. These are systematically different objectives, and the gap between them is filled by people who are sick and cannot afford what exists, or cannot access what exists, or need something that was never developed because the market was too small.

The history of drug discovery reveals a field whose most important achievements were mostly not the result of the intellectual frameworks used to justify its current organization, and whose most conspicuous contemporary failures are not correctable by better science alone. A rational drug discovery enterprise would begin not from what is mechanistically tractable but from what burdens of disease are most urgent — and it would require institutions that do not yet exist.