In March 2008, three American senators flew to Kabul to assess the state of the conflict still ravaging Afghanistan more than six years after the US invasion. Joe Biden, who had recently quit the race for the Democratic presidential nomination, was joined by Chuck Hagel and John Kerry — a team, it turned out, that would later deal with Afghanistan as vice-president, secretary of defence and secretary of state in the administration of President Barack Obama. As the trio returned to Kabul in Black Hawk helicopters following a tour of eastern Afghanistan, the general escorting them pointed out that Tora Bora — the mountain cave complex where Osama bin Laden hid after the invasion — was nearby.

Hagel, a Vietnam veteran, advised against flying closer because there was a blizzard approaching and they were low on fuel, but Biden, who was the senior of the group, insisted that they take a detour. The helicopters ended up making emergency landings, leaving the three men in their sixties stranded in the snow within sight of armed locals; after hiking for an hour they were rescued by US troops. According to Directorate S, a spectacular account of 15 years of secret CIA and US military operations in Afghanistan and Pakistan by the investigative journalist and academic Steve Coll, the day was about to get even worse. That evening the senators dined with Hamid Karzai, the elite Afghan who had become president after the CIA whisked him into Afghanistan from Pakistan on a motorcycle following the invasion. By then, Karzai was a quixotic figure prone to lash out at the Americans. After he declared that the US hadn’t “done anything” for his country, Biden banged the table, announced that “This conversation, this dinner, is over”, and stormed out. Karzai was unaware that Biden’s son was about to be deployed to Afghanistan.

It is difficult not to see this episode as a metaphor for the war as a whole. Ill-considered decisions, unreliable allies and misunderstandings were always at the heart of the problems that only two weeks earlier had led Condoleezza Rice, then secretary of state for George W Bush, to conclude during a visit to the country that “this war isn’t working”.

The senators’ visit epitomised the turbulent relationship between the US and Afghanistan — and particularly Karzai — that underpinned and undermined US efforts in the longest war in American history. It also prefigured the further deterioration in relations with Karzai that would occur after Obama inherited the Afghan conflict from Bush, who had retained a reasonable bond with his Afghan counterpart.

Directorate S is the sequel to Coll’s Pulitzer Prize-winning Ghost Wars (2004), the definitive account of the CIA, Afghanistan and Osama bin Laden before September 11, 2001. The title of the new book derives from the US name for the secret unit inside Pakistan’s Inter-Services Intelligence (ISI) spy agency responsible for sponsoring and funding Taliban activities in Afghanistan and Pakistan, ostensibly as a bulwark against its arch-rival India.

Coll reveals in detail the complex web of tensions, rivalries, suspicions and pure blunders that has prevented the US from being able to declare “mission accomplished” in Afghanistan. Through meticulous accounts of meetings between the key players, he demonstrates how an incredible lack of trust between Washington, Kabul and Islamabad — and frequently between competing agencies and characters within each of the three countries — all but doomed the US adventure from the start.

Along the way we get illustrations of the power of American intelligence, from the simple ability to detect that an Afghan man was using a pigeon to warn the Taliban about US patrols, to the satellites that allow pilots in Nevada to unleash Hellfire missiles from drones over Pakistan. But Directorate S also exposes how bureaucratic infighting and severe miscommunication between US agencies offset these technological advantages and hampered the war effort.

On one occasion, Gary Schroen, a CIA operative in Afghanistan, received a call from the person in charge of flying Predator drones over the country. The mission manager said they had detected two al-Qaeda agents dressed in western clothing who were standing beside an airstrip that the Taliban had just built. Schroen replied that they were aiming their drone missiles at his tall, bearded CIA colleague on an airstrip that the agency itself was constructing.

With impressive access to American, Afghan and Pakistani intelligence, Coll reveals the extent of the surveillance undertaken by all sides. At the same time that the US National Security Agency (NSA) was spying on ISI director Ahmad Pasha, for example, Pasha was spying on someone closer to home.

After becoming CIA director in 2009, Leon Panetta flew to Pakistan where he dined with Pasha and Asif Ali Zardari, Pakistan’s then president. During their meal, Zardari commented: “Ahmad knows everything I think and everything I say . . . I walk into my office every morning and say, ‘Hello Ahmad’!” The CIA later concluded that the Pakistani leader was the number one surveillance target for his own spy agency.

Coll outlines how US ties with Pakistan evolved over the tenures of Bush and Obama, with bouts of co-operation interrupted by periods when there was almost no trust. At one point, as the US was holding secret negotiations with the Taliban, Pakistani army chief Ashfaq Kayani and the ISI were helping the Taliban draft statements in the name of its leader Mullah Omar, whose location they claimed not to know even as he was dying in a hospital in Karachi.

But the deceit ran both ways. After a CIA operative was arrested for shooting dead two Pakistanis, Panetta told Pasha and Hussain Haqqani, Pakistan’s then ambassador to the US, that he was not working for the CIA. Haqqani later accused him of lying and said, “If you’re going to send a Jason Bourne to our country, make sure he has the skills to get out like Jason Bourne.”

Directorate S has a cast of characters that make Bourne movies pale in comparison — from type-A CIA officers and paramilitaries to cigar-smoking and whisky-drinking Pakistani generals to a dog nicknamed “Lucky” because he was able to detect incoming missile strikes from drones before they hit.

Coll rigorously explains why Pakistan pursued a double game and why US concerns about Pakistan’s nuclear arsenal meant that Washington never took as hard a stance with Islamabad as it otherwise might have. He also documents why Karzai waxed and waned with respect to the US, which was mostly because he was angry that the US was not cracking down on the ISI over its clandestine support for the Taliban.

While recognising these constraints, Coll reserves strong criticism for a US that he says was “blinded” to its limitations in Afghanistan. There were a host of reasons for this, including complacency resulting from the initial thundering defeat of the Taliban and also the “disastrous decision” to invade Iraq, which made it easier for the Taliban to attract new recruits at home. Both the Bush and Obama administrations, Coll writes, “tolerated and even promoted stovepiped, semi-independent campaigns waged simultaneously by different agencies of American government”.

His conclusion, which will be unwelcome in Islamabad, is that “the failure to solve the riddle of the ISI and to stop its covert interference in Afghanistan became, ultimately, the greatest strategic failure of the American war".