Barack Obama may have struck a delicate balance with Robert Mueller, but the 104-year history of the FBI is littered with clashes between presidents and feds – and it's the Bureau that comes out on top
One agency in the US has the power to invade and investigate the White House: the Federal Bureau of Investigation. Its extraordinary powers make it the closest thing the country has to a secret police. With the intelligence at its command, the FBI can make presidents, and break them.
Information is power; secret information is power squared. And eyes-only intelligence secured for the president – or against him – is the political equivalent of a nuclear warhead. Its mere possession is sufficient to deter an enemy, and the FBI's J Edgar Hoover had an arsenal unsurpassed in the annals of 20th-century America. Forty years after his death, the FBI still has that power, along with the keys and codes to unleash it. It was the FBI that toppled President Richard Nixon, made felony cases against President Ronald Reagan's national security team, and withdrew the blood and the DNA from President Bill Clinton that brought him to impeachment for lying about sex.
The FBI has been with us for 104 years, and for much of the past century its relationships with presidents have been marked by a ferocious tug-of-war. Only in the past three years, under Obama, has a delicate balance between national security and civil liberties approached equilibrium. No other Democratic president has been free of the fear of the power of the FBI since Lyndon B Johnson took office in 1963. Harry Truman worried aloud that the Bureau would become an "American Gestapo". John F Kennedy knew Hoover had 20-year-old sex tapes of his liaison with a suspected Nazi agent, and he shared his national security adviser's opinion that Hoover was "a goddamned sewer", collecting and disseminating dirt. But most presidents have taken pleasure in using the FBI as a sword and a shield to protect and defend their powers.
Franklin D Roosevelt gave J Edgar Hoover the power to eavesdrop, plant hidden microphones, and purloin secrets through burglary. When the Supreme Court outlawed telephone wiretaps, Roosevelt told Hoover, in so many words, to hell with the court. He relished the political intelligence and the unsavoury scuttlebutt that Hoover brought him. He knew that secret agents can be scofflaws; yet their techniques are useful against terrorists.
When Hoover ruled the FBI – as he did for 48 years, until his death in 1972 – he was confounded by the American electorate only once. After Roosevelt died in office, near the end of the second world war, Hoover, like his fellow Americans, assumed the Republican candidate, Thomas Dewey, would defeat President Truman in November 1948. Dewey, who had made his name as a crime–fighting prosecutor, would have been the first rock-ribbed conservative elected to the White House in 20 years. Hoover was working behind the scenes to support Dewey, who shared Hoover's views on the national emergency confronting the US in the early days of the cold war.
The autumn of 1948 was a dangerous moment in American democracy. As the civilian leader in the war on communism, Hoover was no longer obeying the president. "He wasn't taking orders from Truman or anybody else," said Stephen Spingarn, an army counterintelligence officer serving as a White House adviser. Hoover sought sweeping national security powers over law enforcement and intelligence, sufficient to make him a secret police tsar. The White House pushed back. "That was contrary to our whole tradition," Spingarn said. "You did that in communist and fascist countries, but you don't do that in the United States."
Hoover wanted to detain thousands of politically suspect American citizens in the event of a crisis with the Soviets. The broad outlines of Soviet espionage in the US were beginning to come clear; Stalin's spies had stolen American atomic secrets. Hoover now drew up plans for the mass detention of political suspects in military stockades: a secret prison system for jailing American citizens, including the suspension of the ancient writ of habeas corpus. Hoover's national security assistant, Mickey Ladd, began working out the details for an American Guantánamo in October 1948. The FBI and the army would hold the detainees at military bases in and around New York, San Francisco and Los Angeles. The FBI, the CIA, and army intelligence officers would share the duties of carrying out the thousands upon thousands of interrogations.
Truman looked powerless and politically spent as the election approached. Travelling on a long whistle-stop campaign, with the election four weeks away, he caught a glimpse of a Newsweek magazine poll of America's 50 most prominent political reporters. Their unanimous prediction: Dewey defeats Truman. Hoover went to sleep on election night confident in the outcome. But at 11.14am on Wednesday, 3 November 1948, the bulletin went out across the world: Truman had won the biggest upset in the history of the American presidency. A shift of only 33,000 voters in California, Illinois, and Ohio would have given Dewey victory. When Hoover heard the news, he left his desk at FBI headquarters and did not come back for two weeks. He simply disappeared.
Hoover would not let that happen again. He served to ensure the election of Lyndon B Johnson, his long-time Washington neighbour, their friendship bonded over sips of Jack Daniel's and backyard barbecues. Johnson loved secret intelligence; he craved it as he did whisky and cigarettes.
After the assassination of President John F Kennedy, both men feared that his brother, Attorney General Robert F Kennedy, would seize the fallen mantle. Hoover hated Bobby Kennedy with a passion and ceased to communicate with him. Though the attorney general was his titular superior, Hoover effectively cut off his supply of information and power, ending the possibility that Kennedy could command the Democratic party.
Then an FBI intelligence operation in the Deep South provided Johnson with essential political ammunition for his election. Johnson was furiously pushing civil rights legislation that would allow black Americans the right to vote and deny racist senators the ability to block them from the polls. He needed to destroy the Ku Klux Klan, which was murdering civil rights workers, blowing up black churches, and ruling by fear throughout the South.
On 2 July 1964, Johnson ordered Hoover to go to Mississippi and proclaim the omnipotence of the FBI. The director was dubious; he thought the civil rights workers, not the racists, were the primary problem. "Whatever you do, you're going to be damned," Hoover said. "Can't satisfy both sides." Then he got a direct order from the president. "Ain't nobody going to damn you," said Johnson, who was secretly taping the telephone call. "Ain't anybody in this country has the respect you have."
Johnson knew how to twist Hoover's arm: "Now I don't want these Klansmen to open their mouths without your knowing what they're sayin'. Now nobody needs to know it but you, maybe, but we ought to have intelligence on that state … I want you to have the same kind of intelligence that you have on the communists."
Johnson was telling Hoover to go after the Klan in language he understood. Hoover obeyed. He would subvert them and sabotage them, so long as Johnson commanded that it be done. The FBI broke the Ku Klux Klan like dry twigs. The civil rights laws were passed, and Johnson won in a landslide, with the votes of millions of white liberals and newly enfranchised black citizens.
Days before he left office, Johnson told his successor, Richard M Nixon: "If it hadn't been for Edgar Hoover, I couldn't carry out my responsibilities as commander-in-chief – period. Dick, you will come to depend on Edgar. He is a pillar of strength in a city of weak men."
Hoover had helped Nixon gain power from 1947 onward, feeding him secret intelligence on American communists such as Alger Hiss. But Hoover died in May 1972, and six weeks later, Nixon's henchmen were arrested breaking into the offices of the Democratic National Committee at the Watergate hotel. They had burglary tools and a gadget that the police thought was a bomb disguised as a smoke detector: it was a sophisticated electronic eavesdropping device. The suspects had crisp $100 bills and Watergate hotel keys in their pockets. Their ringleaders were Gordon Liddy, a former FBI agent, and Howard Hunt, a former CIA officer. The FBI quickly determined that both men worked for the president.
An FBI supervisor named Daniel Bledsoe was running the major crimes desk at the FBI on the morning of Sunday, 17 June 1972, when he picked up the overnight report of the break-in. He recognised Liddy's name; he had met him at the FBI a decade before. When he heard that the burglars had been caught with eavesdropping equipment, he immediately opened a case under the federal wiretapping statutes. At about four in the afternoon, his secretary answered the phone and told him the White House was calling.
"This is Agent Supervisor Dan Bledsoe," he said. "Who am I speaking with?"
"You are speaking with John Ehrlichman. Do you know who I am?"
"Yes. You are the chief of staff there at the White House."
"That's right. I have a mandate from the President of the United States," Ehrlichman said. "The FBI is to terminate the investigation of the break-in." Bledsoe was silent.
"Did you hear what I said?" Ehrlichman thundered. "Are you going to terminate the investigation?"
"No," Bledsoe said.
"Do you know that you are saying 'no' to the President of the United States?"
"Yes," the FBI agent replied.
The FBI investigated the break-in. Nixon's resignation came two summers thereafter.
Fast-forward 40 years: the White House, 12 March 2004.
The FBI director, Robert Swan Mueller III, who had taken office on 4 September 2001, walked into the Oval Office with a handwritten letter of resignation in his breast pocket. He had determined, along with the attorney general, that the electronic-eavesdropping programmes created in great secrecy by President George W Bush after the 9/11 attacks had violated the constitution's protections against unreasonable searches and seizures. He would not consent to their use unless the president scaled them back to the boundaries of the law.
Mueller told Bush face-to-face that he would resign if the FBI was ordered to continue warrantless searches on Americans. Both men had sworn upon taking office to faithfully execute the laws of the US. Only one still held to his oath.
Bush promised to put the programmes on a legal footing. This did not happen overnight. It took years. But he backed down. Mueller has never breathed a word of what happened at the White House. But he is an exemplar of what the FBI must do every day.
The man who was then the acting attorney general, James Comey, later described what Mueller had heard from Bush: "If we don't do this, people will die."
"You can all supply your own 'this'," Comey said. "'If we don't collect this type of information' or 'If we don't use this technique' or 'If we don't extend this authority.' It is extraordinarily difficult to be the attorney standing in front of the freight train that is the need for this," Comey said. "It takes far more than a sharp legal mind to say 'no' when it matters most. It takes moral character. It takes an ability to see the future. It takes an appreciation of the damage that will flow from an unjustified 'yes'. It takes an understanding that, in the long run, intelligence under law is the only sustainable intelligence in this country."
The FBI today is still calibrating the balance between liberty and security. Barack Obama, a Democrat grounded in civil liberties, and Mueller, still the civilian commander of domestic security, are two very different men who manage to see eye-to-eye. Mueller, born to wealth and privilege, joined the US Marines and led troops in combat at the height of the war in Vietnam, where he was decorated for valour. Marine officers follow a code: if your troops go up the hill, you go up the hill. Over the past decade, Mueller has led his agents to believe they must fight the threat of terrorism without trampling civil rights. By and large he has succeeded. Mitt Romney and the Republicans have thus far failed to credibly attack Obama's right flank on issues of national security. In this way, the FBI has protected the power of the president.
Obama and Mueller have never clashed on any significant issue of counterterrorism or civil liberties. The two men meet most mornings in the White House; unlike the presidencies of Clinton and Jimmy Carter, the discourse is not a dialogue of the deaf. Obama, who never came on to the FBI's radar until he started running for the White House, is intensely attuned to the difference between existential threats and idle chatter; 99% of what US intelligence hears is the latter. Mueller works on the same wavelength; he spent much of his first five years in office after the 9/11 attacks trying to filter out raw intelligence reporting that led the FBI to investigate pizza deliverymen and delusional sociopaths. The focus on analysing actual threats from potential terrorists has reduced the fear factor in daily political discourse, both at the White House and on the 24-hour ravings on the airwaves at Rupert Murdoch's Fox and its affiliates.
The balance between national security and civil liberties remains delicate. The authors of the constitution foresaw this struggle 10 generations ago. A free people must have both security and liberty. They are opposing forces, yet we cannot have one without the other. Mueller has said we will not win the war on terrorism if we lose our freedoms in the battle. He knows Americans will surrender guarantees of liberty for promises of security. They may feel more safe, but they will be less free. He lives each day in this state of continual conflict. We all do. No free republic in the history of civilisation has survived for more than 300 years.
• Tim Weiner is the author of Enemies: A History of the FBI (Allen Lane, £25/$30), out now.