I have spent most of my adult life working on, and in, interventions. I began as a junior diplomat with East Timor, served in the Balkans and inIraq, then spent a few years in Afghanistan. But none of this made me feel I could predict the future of Libya as I entered Tripoli in August. There were echoes of Baghdad in the masked men holding on to truck-mounted anti-aircraft guns and shouting Allahu Akbar at an angry crowd outside the bank. Was this the prelude to a sudden flurry of looting, then, after a few months, sullen resentment, riots, roadside bombs and rockets falling into the foreign compounds? Would Libya, like the Iraq or Afghan interventions, eventually suck in billions of dollars, thousands of lives, and achieve little more than trauma, corruption and insecurity?

Or could it be, like the Balkans, a success? After all, in 1995 there was a civil war in Bosnia in which 100,000 people were killed; there were three ethnic armies and 419,000 men under arms. Then the west intervened. Today it has a single army of fewer than 15,000. A million refugees have returned and more than 200,000 homes have been given back to their owners. Karadzic, Milosevic and Mladic have been caught and tried as war criminals. You can drive from one end of Bosnia to the other: the checkpoints are gone. The war has completely ended. And all this was achieved at a cost of zero American and Nato lives. Are there lessons from the last two decades that could guide us to success in Libya?

Two dominant theories of intervention have been studied by those responsible for “post-conflict strategy” in Libya. The first – championed by one of the world’s most influential thinktanks, the Rand Corporation – emphasises resources and planning. It holds that the Balkans succeeded because of sufficient troops, money and good management. Iraq failed because there was “no postwar plan”; Afghanistan because of insufficient resources (“We were distracted by Iraq”). The implication for Libya is to plan better and, if necessary, to “surge”. The new US theory of counterinsurgency, of General Petraeus, is this view on steroids. He and countless American politicians, from the president down, emphasise strategy, leadership and above all resources.

The second – equally influential – lesson stresses the need for heroic nation-builders. For former high representative in Bosnia Paddy Ashdown, the key is to “go in hard”, establish the rule of law rapidly, through bold and charismatic international leadership, then “avoid setting deadlines and settle in for the long haul. Peacekeeping needs to be measured not in months but decades.” In 2003, Ashdown wrote to secretary of defence Donald Rumsfeld, suggesting the postponement of elections in Iraq until the rule of law was established. In 2008, he warned that in Afghanistan there was a need to find “a single person to head up the international effort, with the authority to bash international heads together”.

Both theories place the emphasis on foreigners. They are not optimistic about local capacities – local people and leaders are usually portrayed negatively, as criminals or victims – but are highly optimistic about the “international community”, which is assumed to be likely to succeed anywhere in the world, provided it has the right strategy, resources and confidence. As the Rand Corporation wrote in its revealingly titled Beginner’s Guide To Nation-Building: “Iraq was not the first but the seventh society in a little more than a decade that the United States had entered to liberate and rebuild… Six of these seven societies were Muslim. Thus, by the time US troops entered Iraq, no country in the world had more modern experience in nation-building than the United States.”

The fundamental flaw in these theories is that they do not reflect how little foreigners know, can do, or have the legitimacy to do, in someone else’s country. In my experience, foreigners are far more isolated and limited than they like to acknowledge. When I was in Afghanistan, you could begin at dawn running in formation inside the perimeter of a military base; you could spend afternoons gazing at a video-conference screen, linked to your country’s capital. You could sleep on a Chinese foam mattress on the floor of a house in the 70s suburb of Kartai-e-se, or in a shipping container on the gravel flats near the Jalalabad Road. But whatever the rhythm of your day, as a foreigner you were inevitably isolated from Afghan life: by your tour length, security restrictions, career structures, education, and simply through being a foreigner. Even the most determined foreigners had far less contact with the reality of daily life than the most junior policymaker working at home on domestic affairs. I knew this because I had been one of these isolated foreigners myself.

Home civil servants have usually been born in the country and have spent their whole life within its governmental and social institutions. They speak the language fluently (because it is their own), travel on the buses, inhabit the nuances of national manners and social expectations. In short, home civil servants have a kinship with the culture that surrounds them. The officials in Iraq or Afghanistan from 2001 to 2011 had none of these advantages. They were on very short tours. Even members of the US military did not stay for longer than 15 months. Civilians were locked within guarded embassy compounds. And in the rare cases when diplomats were able to spend extended time with locals, they were often unable to speak to them. In 2009, according to the British Foreign Office’s own assessment, the British required no Pashto speakers to work effectively in Afghanistan, even though it was the language of Helmand, where Britain was fighting. The Foreign Office reckoned it required six “operational” Dari speakers, but only three of its diplomats in Kabul were proficient at this level. This was out of an embassy of more than 300 staff, in a country where very few people spoke English.

International civilians in general had little knowledge of Iraqi or Afghan anthropology, geography, history, language, literature or theology. They were not expected to know, for example, the exact content of the Sunni prejudices against the Ismailis, nor to take an interest in the honour codes of gangsters in old Kabul. Instead most international civilians were experts in fields that barely existed as recently as the 50s and which are hardly household names today: governance, gender, conflict resolution, civil society and public administration. They were not experts in gender or governance in Iraq or Afghanistan: they were experts on gender and governance in the abstract.

This did not mean foreigners could do nothing. Even in Afghanistan, there were dramatic improvements in finance, health, public works, education and telecommunications in the first two years after the intervention of 2001. But these were areas in which foreign or Afghan experts in capital cities had particular advantages. Creating a central bank and stabilising the currency was roughly the same all over the world, and not dependent on the exact social structure of an Afghan village. Highly trained foreign doctors improved healthcare, and foreign engineers and architects contributed to bridge design, generator construction and the restoration of historic monuments. Simply changing the Taliban law that prohibited female education brought one and a half million more children to school. Under the Taliban there was essentially no mobile phone coverage. Opening the radio spectrum meant the number of subscribers leapt to one million, then five million. The service is now better in Kabul than Cumbria. The international community even met its initial counter-terrorism goals by driving Osama bin Laden and al-Qaida out of Afghanistan and into Pakistan in 2001; then, over the next nine years, killing or capturing almost every senior member of the leadership, including Bin Laden himself.

But the isolation of foreign lives and their highly abstract ideas meant foreigners performed much worse, the closer they came to the real structures of Afghan rural life. They did well at stabilising the currency, but very poorly at establishing honest local policemen, weaning farmers off opium poppy production, creating “good governance”, rule of law and the other elements of “nation-building”. And, tragically, that was the core of their self-imposed mission. They were committed to fix the “root causes” of the conflict and the fundamental structures of the Afghan state, because they believed if the state remained weak, or the Taliban re-emerged, all their work would be in vain and al-Qaida would return and pose an “existential threat” to the US and its allies. Thus President Obama, in unveiling his Afghan strategy of 2009, wanted to “promote a more capable and accountable Afghan government… advance security, opportunity and justice… develop an economy”.

The international community spent hundreds of millions, for example, attempting to create the rule of law in Afghanistan. An American friend who was employed as a rule-of-law consultant calculated that the total cost of him, his single colleague, their accommodation, support and security team was $1.5m a year. The hundreds of millions of dollars spent over a decade allowed the west to lecture Afghan lawyers; take Afghan judges to meet state judges in the midwest; hold seminars; republish Afghan laws; introduce new codes and administrative regulations; build prisons; train policemen. At the end of the period, however, the justice delivered by young Taliban commanders under trees was consistently rated as fairer and more efficient than that of the infrastructure of the state to which the international community had devoted so much time and money. The Afghan judiciary was still astonishingly corrupt. A senior judge in Helmand conceded in 2009 that the only reason anyone came to his court was to get a passport application form. More and more Afghans said, “At least there was security and justice under the Taliban.” Across the country, 85% of Afghans continued to focus on informal systems. Similar stories can be told of the failure of the west to create “good governance” or disarm and demobilise the armed groups.

Ultimately the more ambitious international programmes, which sought to transform the state, were not simply futile: they were damaging. The international community had exaggerated its fears of Afghan terrorism and regional instability to the point that failure was not an option. Guilt at lost lives and promises dragged them ever deeper into the religion of “nation-building” and encouraged Nato to deploy more and more troops to Helmand and the south. Similar fears and dreams drove the counterinsurgency doctrine and the surge of 2009, sucking up hundreds of billions of dollars and 150,000 troops. This in turn fed corruption in the Afghan government, discouraged Afghan leaders from taking responsibility, distorted Afghan society and government structures, and exacerbated instability. Thousands were killed. And insurgents were able to present themselves as fighting for Islam and Afghanistan against a foreign military occupation.

The same foreign isolation from Afghan lives, which led us to fail, made it very difficult to acknowledge failure. Nowhere was this tendency clearer than with the military. Each new general in Afghanistan from 2002 to 2011 suggested that the situation he had inherited was dismal, implied that this was because his predecessor had had the wrong resources or strategy, but asserted that he now had the resources, strategy and leadership to deliver a decisive year. In 2004, the new International Security Assistance Force commander, General Barno, said that “without question” 2004 would be a “decisive year”. General Abuzaid thought 2005 would be a “decisive year”, General Richards that 2006 would be the “crunch year” for the Taliban. Major General Champoux predicted that 2008 would be a “decisive year”. In 2009, General McChrystalstated: “The Taliban… no longer has the initiative… We are knee-deep in the decisive year.” Both the Nato secretary-general and the UK foreign secretary, David Miliband, predicted 2010 would be a “decisive year”. At the end of 2010, President Obama concluded: “For the first time in years, we’ve put in place the strategy and the resources.” German foreign minister Guido Westerwelle predicted that 2011 would be a “decisive year”.

Given, then, the propensities of western governments – given that their staff are so isolated from local society, hypnotised by abstract ideas of state-building, prone to guilt at lost lives, inclined to ignore failure and dig themselves ever deeper – how could we ever succeed in Libya or anywhere? Perhaps by learning from the Balkans. The foreigners were no different in the Balkans – often it was the very same individuals (I was one of tens of thousands of people who served in the Balkans as well as Iraq and Afghanistan). In postwar Kosovo, just as in Afghanistan, foreigners were often hardly aware of the local forms of security, administration and justice that predated the crisis and which continued to support communities. In Bosnia, too, foreigners tended to focus on long-term structural problems – such as unemployment, poor education, gender discrimination or weak service delivery – without acknowledging that these were not the prime cause of the conflict. In every intervention, foreigners attempted to impose solutions through overwhelming force – or ever more absolute legal powers.

But the secret in Bosnia may have been that these dangerous traits in the psyche of the international community were not given full rein. The US population and Congress were very reluctant to become involved in Bosnia – they were still haunted by the humiliation of Black Hawk Down in Somalia, and even by ghosts of Vietnam. When they intervened, they did so tentatively. Their mission was humanitarian: to end a war. The Balkans were not considered an “existential threat”. So there was neither the will nor the mandate to force through a radical programme of “nation-building” in the face of local opposition. It is true that very large numbers of troops were deployed in the Balkans – far more per head of the population than in Afghanistan, for example – and Rand has interpreted this as a sign of great international power. But in fact the Balkan troops were under orders to take few risks: the biggest challenges for US army doctors in Bosnia looking after 20,000 soldiers were sports injuries.

In the first year, the international soldiers did not disarm special police forces, intelligence services or even the rented mobs made up of nationalist war veterans. Croat and Serb hardliners kept control of paramilitary police. There was no move to ban the leading Serb party (SDS), founded by Karadzic, seen as responsible for Srebrenica, or even to stop it participating in elections. Until the end of 1995, the war crimes tribunal had only one indicted criminal in jail, and there was little effort on refugee return. The international administration in Bosnia never had any police or prisons, still less an Abu Ghraib. When Paddy Ashdown talked about “going in hard”, he did not mean physical arrests. By the time he arrived in Sarajevo, there were only 12,000 international soldiers left in Bosnia. This lack of resolve and mandate, rather than undermining the intervention, in fact provided the space for Bosnians to take the lead in key areas, and allowed time for changes to take place in politics and the region, without which reform would have been impossible.

It was Bosnians who pushed through the refugee returns at a time when many senior international migration experts still thought it risky. It was political fractures within the Bosnian Serb entity, between the leaders Karadzic and Plavsic, that weakened the special police forces in the Bosnian Serb territory and allowed the international troops – two years after the intervention – to gradually assert control. And it was CroatianPresident Tudjman‘s death and the collapse of his regime in 1999, then the overthrow of President Milosevic in Serbia, that finally brought stability, because it was those Yugoslav leaders who had planned and driven the ethnic cleansing. Their fall crippled their Bosnian proxies and opened the door to a flood of arrests and trials.

None of these events or developments could have been confidently predicted, even by Bosnians themselves. None of them could be replicated in Iraq or Afghanistan. A different reaction to the foreign troops from the armed groups in the Balkans could have led to an insurgency. If the international community had been forced to protect its soldiers and civilians from years of roadside bombs and assassinations, and justify thousands of deaths and a bloody counter-insurgency campaign alongside its mission, it might have achieved next to nothing in Bosnia.

The central lesson for Libya is that success is dependent far less on the plans or genius of the foreigners, and far more on local context – a context that is intrinsically chaotic and unpredictable. Intervention is far more uncertain than any conceivable domestic policy. You are charging into a dark room, unsure whether there is even a floor beneath your feet. There is no magic plan, nor quantity of resources, that can guarantee success. There could never be (pace the Rand Corporation) a universal formula for intervention, specifying the exact approach or quantity of international resources required for each hypothetical country. But one strength of Bosnia seems to have been that the initial foreign assistance was limited, cautious and incremental, whereas in Iraq, and after 2005 in Afghanistan, the occupation was insistent and overpowering.

The best thing I saw in Libya was the celebration in Tripoli, because it was such a Libyan affair. No foreign Humvees blocked the streets, no foreign bodyguards surrounded the minister of the interior, no UN staff bustled back and forth to check the microphones on the podium. Instead speech after speech was spoken in Arabic. Apart from half a dozen journalists, there were no foreigners to be seen and almost no reference to foreign governments. The speeches suggested that Libyans and no one else had got rid of Gaddafi. In Iraq, large angry crowds of poor Shia screamed their rage against Saddam and the Ba’ath party. Here hundreds of young men, seated on great cranes, stretched over the square (once built to hold a world record-sized photograph of Gaddafi), had painted faces and flags, and chanted good-humoured anthems of celebration.

Off the square, not everything was positive: I wondered if the hotel, which a militia group had occupied, was only the first of many assets that would be seized from “Gaddafi sympathisers”; whether the anti-foreign instinct of the Islamist commanders would prove dangerous; how the complacency of the new government, or tribal rivalries or corruption, would affect the future. But the members of the new Libyan government, unlike their Iraqi or Afghan equivalents, neither denied these problems nor pretended to be able to eliminate them. In Iraq and Afghanistan, the international community created a dependency on foreign assistance and fostered an obsession with nation-building. The US tended to frustrate compromises with opposition groups – whether the Sadrists or the Taliban. There was little sign of such phenomena in Libya, presumably in part because there was so little international presence on the ground. The members of the transitional council I met – working in health, in finance – talked modestly about working with what they had: making compromises, negotiating, forgiving.

None of this should suggest that either Bosnia or Libya provide a universal template for intervention. We must remain cautious about the tempting abstractions of “the responsibility to protect”. We must ignore those who are already suggesting that perhaps the secret in Libya was a smaller population than Iraq and no ethnic or sectarian divisions. (They should be asked to explain the horrors of neighbouring Algeria which has no Shia population and has experienced decades of civil war.) We must restrain the military planners already using Libya as an argument for more investment in targeted air-power.

If there is a lesson from all these countries, it is that there is no substitute for detailed experience in a particular place. The best the west can do is to rely on more people who are focused on the specific history and culture of Libya, more attentive to the realities of rural life. A massive influx of international money and staff could make this group of specialists a tiny minority, drowned by vast, rapidly expanding organisations with eye-watering budgets. The international community must bring such learned outsiders, who have worked for years on the ground, back into the policy centre to minimise the abstraction and isolation of our elites. These advisers’ role in Libya should not be to predict whether there will be anarchy after the fall of the regime – that would be almost impossible even for a Libyan – but instead to sense the direction in which politics is moving. Could it be corruption, rather than pumping technology, that is impeding the petrol supplies? What signs might there be that the transitional government is losing legitimacy? Is it worth taking the risk of an early election? When is there nothing more that foreign support can do – when is it in fact making the situation worse?

If the only common theme in intervention is radical uncertainty, then the only rule must be humility and restraint. The Libyan no-fly zone strayed towards targeting Gaddafi but it was not pulled into a ground intervention. No Nato lives were lost, so there was none of the guilt that often prolongs a doomed mission reluctant to concede a soldier died “in vain”. The investment was significant – $200m – but it was one 20th of our annual investment in Afghanistan. Because our involvement was limited, the option – just – remained of reassessing the situation, spotting the dangers, even privately acknowledging failure, and having the confidence, if necessary, to pull back. It has certainly allowed more space for Libyan leadership, Libyan pride and Libyan imaginations. And it may have kept us sufficiently involved to usefully support local initiatives and opportunities when they arise. If the Libyan intervention works, however, and is justified, it may be precisely because, compared with Iraq or Afghanistan, it is hardly an intervention at all.