The Second Oldest Profession: Spies and Spying in the Twentieth Century

The Second Oldest Profession - Spies and Spying in the Twentieth Century by Phillip Knightley uncovers the tangled origins, explosive growth, and self-sustaining mythology of intelligence agencies as they shaped the world’s great power rivalries and internal politics. Knightley draws from decades of investigative journalism, interviews with master spies, and deep archival research to reveal the infrastructure and psychology behind the secret world of twentieth-century espionage.

The Invention of the Intelligence Agency

Phillip Knightley locates the formal birth of the intelligence agency in early twentieth-century Britain, as imperial anxieties and the popular imagination fused to create a permanent bureaucratic structure for spying. The British government convened a sub-committee of top officials, alarmed by rumors of German espionage, and weighed the evidence presented by Colonel James Edmonds. Edmonds, lacking resources and hard proof, supplied a mix of anecdote, rumor, and allusions to fictional accounts such as those by William Le Queux. The government, under pressure to take action, moved to create an official secret service: an agency equipped, funded, and mandated to gather secrets abroad and protect the homeland.

Le Queux’s spy novels did not merely influence the public—they shaped the beliefs and testimony of policymakers. The synergy between fiction and government fact-finding built momentum for a bureaucratic solution, producing an agency that would endure in peace and war.

Public Fear, Law, and Bureaucratic Momentum

Once the intelligence agency emerged, public anxiety fueled its expansion. Le Queux and other writers produced works that blurred the boundary between entertainment and statecraft, igniting what Knightley calls “spy fever.” The resulting atmosphere drove the passage of sweeping new laws, such as the Official Secrets Act of 1911, which enabled prosecution based on suspicion and association. Legislators moved rapidly and quietly, bundling the act through parliament with minimal debate. In this climate, rumor acquired the force of evidence, and authorities operated with wide discretion.

Knightley chronicles specific cases such as Max Schultz and Robert Blackburn, whose supposed espionage often relied on circumstantial or trivial acts, yet resulted in conviction and punishment. In this legal environment, the agency and the state gained the power to define, detect, and destroy perceived threats, reinforcing their own necessity.

The Rise of Institutional Power

Knightley demonstrates that once created, intelligence agencies accumulate power through secrecy, budgetary expansion, and organizational growth. Their leaders, conscious of the benefits and status conferred by secrecy, manipulate access to information, shape media narratives, and secure resources far beyond original mandates. By the outbreak of the First World War, intelligence budgets and staffing had multiplied, often with little external oversight or public accounting.

Agencies become adept at constructing the perception of indispensability. They produce plausible threats, justify increased funding, and reinterpret ambiguous results as evidence of success or proof of the need for improvement. Successes justify expansion, while failures prompt calls for more resources or looser operational constraints.

Espionage, Myth, and the Mass Media

Knightley identifies the feedback loop between spy fiction and reality as a fundamental driver of agency growth. Novels and news accounts generate new fears; those fears translate into political pressure and policy; agencies provide stories to the press that reinforce their mystique. Agencies selectively release information, allowing stories of triumph or menace to shape public opinion. When international tension eases, agencies often “go public,” releasing stories designed to sustain anxiety and preserve institutional relevance.

The narrative of the master spy, unmasking traitors and averting disasters, gains momentum with each cycle. Intelligence failures disappear into classified archives, while select victories become the foundation for legend. The resulting mythology enables agencies to resist attempts at reform, abolition, or external control.

The Global Intelligence Race

Knightley charts the international spread of intelligence agencies as nations respond to one another’s initiatives. Germany organizes its service in 1913, Russia in 1917, France in 1935, and the United States only in 1947, after World War II. National pride, bureaucratic rivalry, and the perceived need to match rivals drive the process.

Intelligence work, once an ad hoc or amateur affair, institutionalizes and proliferates. Agencies compete with each other, sometimes more fiercely than with foreign adversaries. Internal rivalries, jurisdictional disputes, and overlapping mandates characterize the new intelligence landscape. The structure of government shifts as intelligence services claim authority over foreign policy, military planning, and domestic security.

The Economics of Secrecy

Knightley scrutinizes the vast sums allocated to intelligence, observing that governments rarely know the true cost or scope of their agencies. The CIA’s annual spending, by his estimate, exceeds $30 billion, rivaled by the NSA’s similar budget. British agencies, officially budgeted at £120 million, likely spend far more. Intelligence agencies use creative accounting and cooperative arrangements with allied services, complicating efforts to track headcount or expenditure.

Within these structures, tens of thousands draw salaries, and even more participate as informers, contractors, or military support. These numbers underscore the size and power of the intelligence community, which acts as a distinct, self-interested group within government, invested in its own survival and growth.

The Professionalization and Bureaucratization of Espionage

As agencies matured, they shifted from swashbuckling adventurism to professionalized, bureaucratic operations. Modern spies, Knightley asserts, resemble technocrats and analysts more than fictional heroes. They design and operate systems for intercepting electronic communications, managing informant networks, and processing vast flows of data.

Knightley tracks this evolution through the Cold War, when technological advances supercharged surveillance and data collection. Agencies such as the NSA transformed espionage into a high-technology industry, capable of vacuuming up global communications. As the volume of intercepted information soared, the problem of analysis and synthesis became acute. Agencies faced the risk of “drowning in their own intelligence,” overwhelmed by the sheer quantity of collected data.

The Political and Psychological Uses of Intelligence

Knightley dissects the ways agencies use intelligence as a tool of political persuasion, both within government and in public. Agencies argue for resources by presenting scenarios of looming threats, real or fabricated. They claim that timely warning of an attack or crisis enabled policy successes, or, if failure occurs, that the government failed to act on their accurate assessments. When funding or authority comes under threat, agencies present the argument that only more resources and greater secrecy can ensure national safety.

This dynamic insulates agencies from scrutiny and makes it difficult to measure results. Legislators and journalists, denied access to key information, find themselves unable to challenge agency claims. Agency insiders use secrecy laws and employment contracts to silence dissent. Occasionally, agencies orchestrate media campaigns or strategically release information to shift political opinion or forestall reforms.

The Cold War and the Expansion of Espionage

With the advent of the Cold War, intelligence agencies gained unprecedented influence over government policy and public consciousness. The US and USSR, locked in ideological and military competition, each invested heavily in surveillance, covert action, and propaganda. Knightley describes a world in which agencies monitor both enemies and allies, seeking information and influence everywhere. They become both actors and observers, shaping events and responding to them in real time.

The Cold War environment rewards paranoia and rewards those who anticipate the next crisis or uncover the next plot. Agencies justify their existence through continual assertion of threat, building careers and reputations on the discovery or prevention of espionage and subversion.

Failures, Cover-Ups, and the Myth of Infallibility

Knightley challenges the myth of intelligence agency infallibility by recounting numerous failures, cover-ups, and misjudgments. Agencies struggle to convert raw information into actionable insight. Policy makers sometimes ignore accurate intelligence, and agencies sometimes misinterpret or mishandle what they collect. Yet failures, when exposed, rarely result in punishment or reform. The structure of secrecy allows agencies to explain failures as government errors or as a result of inadequate funding.

Knightley explores incidents where agencies failed to predict invasions, coups, or major diplomatic shifts, only to claim after the fact that they had issued sufficient warnings. The circular logic of threat and justification persists, unchallenged by external review.

The Age of Surveillance and Data Overload

In the late twentieth century, intelligence agencies become global collectors of information, building massive archives of intercepted messages, photographs, and digital data. Knightley identifies the paradox that as agencies increase their collection capabilities, they lose the capacity to distinguish the significant from the trivial. The bureaucratic imperative to collect and archive everything creates a glut of information that no human or machine can effectively manage.

He cites the problem of data overload at agencies like the NSA, where analysts struggle to find meaning in the deluge of signals intelligence. Despite the sophistication of technology, the human factor—interpretation, synthesis, judgment—remains the limiting constraint. The risk emerges that agencies, in pursuit of omniscience, become less effective and more vulnerable to error.

Legacy and Impact on Democracy

Knightley concludes that intelligence agencies, born in secrecy and sustained by myth, now function as quasi-autonomous centers of power within democratic societies. Their influence shapes national security policy, legislation, and even public perceptions of reality. The persistence of the spy myth—fueled by fiction, media, and agency storytelling—makes it difficult to disentangle truth from legend, fact from narrative.

Agencies succeed in preserving their autonomy and resources through continual engagement with public anxiety and political necessity. They serve as guardians, manipulators, and sometimes creators of the threats they claim to counter. The result is a security apparatus whose growth, logic, and operations challenge the transparency and accountability of government.

Phillip Knightley’s The Second Oldest Profession compels readers to confront the structures and stories that have defined modern espionage. Knightley’s analysis reveals a world shaped by anxiety, ambition, and the machinery of secrecy. His work underscores the enduring need to question, investigate, and understand the forces that operate beyond the visible boundaries of public life. Where do the boundaries of legitimate secrecy end, and the imperatives of public accountability begin? Knightley’s chronicle insists on the urgency of the question, urging vigilance in the face of institutional power that thrives in the shadows.