My parents gave me a subscription to The Spectator in 1984, when I was 11. When I was 12, I wrote a letter to the editor, criticising the progressive views of the Bishop of Durham, and Charles Moore — who had just become the editor at the age of 27 — published it under the headline ‘Very young fogey’. Who knows what a weekly diet of The Spectator did to my impressionable mind? Is Taki responsible for my taking up martial arts? Or Roger Scruton for my views on ugly buildings? I think it was the book reviewers, so unintimidated by even the grandest book, who made the greatest impression. They made the most flattering assumptions about readers — that I would have read Osip Mandelstam, and had spent months with curlews, and made a close study of Pugin interiors. When I edited my school magazine I copied The Spectator right down to the font and the column shape. Such reverence is hard to eliminate: I was embarrassingly honoured when Charles Moore gave me high marks as a ‘real conservative’ two months ago.

Except I am not technically a Conservative any more. I voted against no deal. (For the record I am pushing for Brexit and against Remain but think no deal would lead to uncertainty, damage and division. This is particularly true for farmers in my constituency.) The Prime Minister responded by deselecting me as the Conservative MP for Penrith and the Border, and trying to call an election. So having resigned from the cabinet only six weeks ago, I may find myself out of what remains of my job in six weeks’ time. I will be particularly sorry about this because I feel I am beginning to learn about some of the really tough hidden issues in Britain. Running for the leadership in June gave me the opportunity to explore areas outside my constituency — in Northern Ireland, England and Scotland, from Derry/Londonderry to Wigan. Last week, the last of the parliamentary holiday, I have been walking through communities in the north-east between Newcastle and Hartlepool.

Last Thursday, I was in a food bank, in a church, which was feeding 70 people. The young Nigerian man who was studying aeronautical engineering was fit and confident. But he was the exception. A homeless man could have been any age between 20 and 30 – his skin was a raw red, he was hunched, one eye was half-closed. His flat stare implied that he had combined his heroin with street Valium. In Glasgow, I was told that the homeless simply sit, with their cup out, and when it has accumulated £10, their dealer comes and empties the cup and drops off the heroin. In a park by Saltmarket I found half an acre of ground, covered in discarded needles, and equipment for cooking drugs. The life expectancy of the heaviest drinkers is 29, and of the heroin addicts, 46. At least in Sunderland and Hartlepool there are some services, which you can access – for addiction or support. But I found that there was almost nothing for the addicts in Easington colliery, where houses cost £15,000, and nothing has replaced the mine.

It was difficult to learn from professionals in charities – they seemed so allergic to Tory politicians that any questions were immediately diverted into furious attacks on austerity.  I had more luck with volunteers. Mary chatted to me while handing over bags of condensed milk and cereal. She had been coming to help in the food bank every week for the last two years. And seemed to know everyone. ‘This girl has removed herself from school at 14 – I’m trying to convince her to go back. Them? They’re drunk (a flailing fist-fight had erupted between two women near the statue of Our Lady). I’m not sure why they are queuing for the kids’ food – they don’t have kids. How did she afford those hair extensions? They cost £70…’ I loved her energy and cheerfulness – and lack of solemn piety. ‘Right’ she said at the end, ‘that’s my guilt dealt with for the week – I’m going paddleboarding.’ The answer has to begin with recruiting, training and backing more Marys: thoughtful, realistic, good-humoured people – some in social, youth and probation work, some volunteers – patiently helping people through all the challenges: from addiction to a simple lack of hope.

I’ve just finished reading Blitzed, a beautifully researched book about drugs in Nazi Germany. The Blitzkrieg was fuelled, it seems, on crystal meth — it allowed the panzer drivers to go for three days without sleep and gave suicidal confidence to the waves of attacking German troops. The success of a reckless charge, against all odds, just strengthened their optimism. By 1944 Hitler was maintaining the Führer confidence with six injections a day. Some were cocktails of pig’s blood (which didn’t affect his belief that he was a vegetarian). But increasingly it was combinations of heroin and methamphetamine. The generals — astonished at his optimism in the face of all negative reports on every front — assumed he must have a secret weapon. The secret was chemically induced.

Yesterday, I was worrying about my cherry trees — what we call ‘Gean’ in my part of Britain and what the Spectator reader would call prunus avium. We planted 40 of them to mark our wedding seven years ago. One is thick, healthy, and 20 feet high. The others are in terrible states — black and wilting leaves, leaves shredded by aphids, barks torn to the core by roe-deer. They all share the same sun, in the same field, and I have put new shelters around all of them, but after seven years some are barely four foot tall. Twice I have replaced some of the most sickly, so there are now three batches struggling in the field. Every spring I hope the problem will be solved. I suspect the soil and the cherries simply don’t agree. But I cannot seem to change course. I am a great one for accusing other people of fairytale optimism, about Iraq, Afghanistan or no-deal Brexit. But when it comes to my cherry trees my megalomaniac fantasies are on full display: failure is not an option.

What is wrong with us? The United Kingdom should be one of the most impressive democracies on Earth. We have incisive and apparently incorruptible judges; an undeferential, boisterous and intelligent media; and an extraordinary culture of voluntary activity – supplemented by charitable superpowers such as Oxfam and the Royal Society for the Protection of Birds. Our civil servants, diplomats and soldiers are thoughtful and skilful. Our parliament is more diverse than at any time in its history. Our economy is stable. Only the US can rival our universities; and only the US and China, our success in science. London remains one of the greatest cities on Earth. Our citizens have never seemed so healthy or better educated. So why does it suddenly seem to be so difficult to make what once, at least, seemed obvious, sensible arguments from the centre ground?

Three weeks ago, I found myself in the final five for the leadership of the Conservative Party. I went into a BBC debate against the other four candidates – all of whom were promising unfunded tax cuts and increases in public spending. They also rejected the current European Union withdrawal agreement and insisted on retaining no-deal as a threat against the EU. All the others claimed to be able to get a new deal out of Brussels by the end of October (or in one case, the end of the year). All insisted that in the absence of such a deal, they would want to leave in 2019 with no deal, and that they would be able to drive no deal through parliament.

How could I lose against such arguments? Most of the public and 90 per cent of my parliamentary colleagues agreed – or at least had recently agreed – that we could not get no deal through parliament. Very few people were comfortable with unfunded tax or spending promises. Or with a no-deal Brexit. Or with suspending parliament. Nobody seriously believed that Brussels would offer an entirely new deal by October (even Nigel Farage agreed with me on that).

But I failed to win any of these arguments, and within 24 hours I was knocked out of the contest. There have been explanations for my failure. One, made by the pollster John Curtice, is that public opinion simply makes a “centrist” position such as mine impossibly quixotic. Public opinion was traditionally – in the time of Tony Blair and David Cameron – a bell curve with all the votes located in the centre ground. Now the bell shape has collapsed, like an unstable soufflé, into a U-shape, leaving voters only on the extremes.

Forty per cent of voters want to ignore the referendum result and remain in the European Union; and 40 per cent of the population – and around 80 per cent of Conservative members – apparently favour no deal if the alternative is remain. There was almost no constituency for someone trying to argue for a moderate and pragmatic Brexit among the public – and therefore there could not be among practical MPs.

I favour a different explanation. Which was that I had forgotten all the lessons of the many walks I had done around the country in the previous weeks – from Derry to Derby, from Edinburgh to Peterborough – and had tried simply to rehearse what I saw as the facts. I talked about the impact of unfunded tax cuts and spending pledges on our fiscal position. I tried and failed to explain in a few seconds how higher tariffs following a no-deal Brexit would lead to inflation, pressure on incomes, interest rate rises, and ultimately negative impacts on GDP.

In other words, as some of my friends argued when they were being more polite, I was “off my game”.

The lesson is that it is possible to change minds, and defeat extremist positions; but not by explaining tariff levels. If 72 per cent of voters are dissatisfied with the UK democratic system – half believing that the government doesn’t care about them, and more than half saying that “Britain needs a strong leader willing to break the rules” – you cannot expect to win simply on technocratic arguments. But nor should you feel forced to respond with nonsense and fairy tales. What I had seen, walking around the country, is that democratic life is neither about echoing and deepening pre-existing prejudices, nor only about communicating economics.

To support a sensible, pragmatic position, you have to begin by rediscovering a sense of anger and shame. And acknowledge how unforgiveable and appalling many things are in modern Britain. You need to see the line of broken windows in cell after cell in a Liverpool prison, and see the blood on the floor in one in Birmingham, to feel the right kind of shame at the state of our prisons. You need to get out to Poplar in east London to see the white tent in the rough grass between the buildings of a housing estate, pitched over the body of a dead man; the air ambulance, rotor blades still spinning, waiting to take him away; and listen to the older man explain what it was like to walk out of his mosque and see someone lying on the ground, bleeding to death from a knife wound.

Second, you need to be very much in a hurry to fix things. You cannot accept that it is impossible to reduce violence in prisons in a year. Or that the fast train connection between Leeds and Manchester will not begin to be built until 2035. Or that Britain should have slower broadband speeds than Madagascar. Or that almost every young man around that Poplar estate is carrying a knife. Nor should you accept smaller injustices – such as the idea that a constituent visiting a dying parent should receive a parking fine in the hospital car park.

Third, you need to be ambitious. You should aim to plant not 11 million trees but 110 million trees; build not a million homes in five years, but two million and build them with government money, and make them beautiful.

And if there is a single extreme disgrace, such as the great unfinished revolution of adult social care – left for decades after the creation of the NHS – then sort it out. It is not defensible that the frail elderly should only get 15-minute daily care visits in their homes – hardly long enough to wash them, let alone talk to them. So, reach boldly across parties, and agree on how to finance a proper system of care.

All of this – seeing the horrors, feeling the shame, getting on with it at scale – allows you occasionally to do something more fundamental, which is to be truthful about your own failings, and the obstacles that stand in the way of getting anything significant done. And all of this – from shame to truth – helps you to believe in what you are doing again.

Perhaps because the centre ground had seemed so inevitable and successful for so long, little emotional or moral energy was needed to make the case for moderation. Instead the arguments had long been surrendered to policy think tanks and civil servants who produced intelligent papers, in a minor key – about learning lessons from Scandinavia, or recommending better use of technology, or more efficiency. The only “value” that had been discussed for two decades was value for money. Patriotism, liberty, courage, and the will of the people were ideas that were abandoned – left on a dusty shelf – to be picked up again by the more divisive politicians.

It was all too easy for the extremists to use punchy language to disguise their vacuous fantasies, making the impractical seem practical, the negative positive, and the airy-fairy down to earth. Take “no-deal Brexit”, for example. It is a phrase that sounds like a thing, a fine Anglo-Saxon fact, not an abstraction like “a temporary backstop”, but it is in fact the opposite of a thing. No deal is an absence that pretends to be a presence; the negation of a deal that pretends to be a type of deal.

It is a phrase that floats recklessly free from any connection to any particular person in a local place, or at a particular time. It cannot ever tell, say, Chris Harrison, a sheep farmer in Alston, Cumbria, whether he would have to pay European tariffs of 40 per cent on his Swaledale lambs; or whether Brazilian beef would come in import-free. But it is vague enough to allow its proponents to claim it is democratic despite the majority of the population being against it, and patriotic even though it is against the best traditions and interests of our country.

Like almost every other argument made today – even in the mainstream of the Labour and Conservative parties – it builds on the fantasy of the victim, that you would be better off on your own, if only you could rid yourself of other classes, other groups and other nations. It shares the instinct that made some Scottish nationalists feel that the many challenges of a modern nation could be solved by simply getting rid of England. And these desires to turn away from relationships seemed an aspect of a deeper desire to turn away from reality – from the questions of who or what we really are today as a people, what we share, and who we can realistically be in the future.

I’ve been struggling to communicate my sense that the centre ground should not be simply a midpoint between these empty abstractions  – some grey smudge between the blacks and whites of the extreme; that it has a reality, a connective power and a maturity that is denied to the other positions. It is something that exists on a different dimension – of reality as opposed to fairy tale. And because it is grounded in reality, it can be sustained and successful in a way populism cannot, because it is founded in truth.

But the centre should also not be a narrowing project that seeks to belittle the intuitions, the understanding or the motivations of people who currently favour a no-deal Brexit or Jeremy Corbyn.

Faced with a democracy that sometimes feels as though all the votes are gathered at two opposite ends of a stick, the centre ground must not be simply the midpoint of the stick, whose only merit is being as far away as possible from each extreme. It should instead involve a project of bending the stick and connecting both ends with a string to make a bow. The centre should then be at the midpoint of the string, the point of greatest potential energy – which comes not from excluding the two extremes, nor from linking them loosely together, but instead from harnessing the tension of two opposing forces.

Our country has entered a midlife crisis. The answer cannot be to try to lurch back to an adolescent fantasy of being saved by superheroes, but instead to move forward into maturity. Such a maturity begins with a recognition of our shame, our failures, and our successes in surviving and battling the constraints of the world and time. A maturity that turns from comforting and ambiguous abstractions to face, unflinchingly, reality. And to say, like Samuel Beckett’s Krapp, reflecting on whether he missed the excitement of his youth, “not with the fire that’s in me now”.