A film director once told Salman Rushdie that all movies made from novels are rubbish. With so many screenplays based on books triumphing at the Oscars this week and Slumdog Millionaire stealing the show, he asks is there such a thing as a good adaptation

Adaptation, the process by which one thing develops into another thing, by which one shape or form changes into a different form, is a commonplace artistic activity. Books are turned into plays and films all the time, plays are turned into movies and also sometimes into musicals, movies are turned into Broadway shows and even, by the ugly method known as “novelisation”, into books as well. We live in a world of such transformations and metamorphoses. Good movies – Lolita, The Pink Panther – are remade as bad movies; bad movies – The Incredible Hulk, Deep Throat – are remade as even worse movies; British TV comedy series are turned into American TV comedy series, so that The Office becomes a different The Office, and Ricky Gervais turns into Steve Carell, just as, long ago, the British working-class racist Alf Garnett in Till Death Us Do Part turned into the American blue-collar bigot Archie Bunker in All In the Family. British reality programmes are adapted to suit American audiences as well; Pop Idol becomes American Idol when it crosses the Atlantic, Strictly Come Dancing becomes Dancing With the Stars – a programme which, it may interest you to know, invited me to appear on it last season, an invitation I declined.

Songs by great artists are covered by lesser artists; on inauguration day this year, Beyoncé performed her version of Etta James’s classic “At Last” to the considerable irritation of Etta James herself (but then, James seemed even more irritated by the election of Barack Obama, so perhaps she was just in a bad mood). All of these are examples of the myriad variations of adaptation, an insatiable process which can sometimes seem voracious, world-swallowing, as if we now live in a culture that endlessly cannibalises itself, so that, eventually, it will have eaten itself up completely. Anyone can make a list of the many catastrophic adaptations they have seen – my personal favourites being David Lean’s ridiculous film of A Passage to India, in which Alec Guinness as a Hindu wise man dangles his feet blasphemously in the waters of a sacred water tank; and the Merchant Ivory emasculation of Kazuo Ishiguro’s The Remains of the Day, in which Ishiguro’s guilty-as-hell British Nazi aristocrat is portrayed as a lovable, misguided, deceived old bugger more deserving of our sympathy than our scorn.

But adaptation can be a creative as well as a destructive force. Rod Stewart singing “Downtown Train” is almost the equal of Tom Waits, and Joe Cocker singing “With a Little Help from My Friends” achieves the rare feat of singing a Beatles song better than the Beatles did, which is less impressive when you remember that the original singer was Ringo Starr. I’m currently teaching a course that highlights some of the instances in which fine books have been adapted into fine films – Edith Wharton’s The Age of Innocence mutated into Martin Scorsese’s The Age of Innocence; Giuseppe di Lampedusa’s portrait of Sicily in 1860, The Leopard, turned into Luchino Visconti’s greatest film; Flannery O’Connor’s Wise Blood became a wonderful John Huston movie; and, in his film of Great Expectations, Lean produced a classic that can stand alongside the Dickens novel without any sense of inferiority, a film that allows this film-goer, at least, to forgive him for the later blunders of A Passage to India.

There are many other examples of successful adaptation. Few people these days read Jan Potocki’s 19th-century Franco-Polish masterpiece The Manuscript Found at Saragossa, but I urge you to discover it for its playfulness and bizarrerie, its surreal, supernatural, gothic, picaresque world of Gypsies, thieves, hallucinations, inquisitions and a pair of unbelievably beautiful sisters who are, unfortunately for the men they seduce, only ghosts. Its qualities are perfectly captured by the Polish film director Wojciech Has in his 1965 film The Saragossa Manuscript, which you should seek out at once. Satyajit Ray’s 1955 film Pather Panchali (“The Song of the Little Road”) not only equalled but bettered the 1929 Bengali classic by Bibhutibhushan Bandhopadyahya from which it was adapted. Huston seems to have been a particularly gifted adapter of good literature, and his film of Joyce’s “The Dead”, perhaps the greatest short story in the English language, brings it vividly, passionately to life; although right at the end, when the camera moves out through a window to watch the falling snow, and Joyce’s famous words take over from Huston’s images, speaking of the snow that was falling faintly through the universe and faintly falling, like the descent of their last end, upon all the living and the dead, we are reminded of the difference between excellence and genius. The Dead is an excellent film, but the last lines of Joyce’s story surpass it effortlessly.

The question raised by the adaptive excesses of Adaptation is the question at the heart of the entire subject of adaptation – that is to say, the question of essence. “Poetry is what gets lost in translation,” said Robert Frost, but Joseph Brodsky retorted: “Poetry is what is gained in translation,” and the battle-lines could not be more clearly drawn. My own view has always been that whether we are talking about a poem moving across a language border to become another poem in another tongue, a book crossing the frontier between the world of print and celluloid, or human beings migrating from one world to another, both Frost and Brodsky are right. Something is always lost in translation; and yet something can also be gained. I am defining adaptation very broadly, to include translation, migration and metamorphosis, all the means by which one thing becomes another. In my novel Midnight’s Children the narrator Saleem discusses the making of pickles as this sort of adaptive process: “I reconcile myself,” he says, “to the inevitable distortions of the pickling process. To pickle is to give immortality, after all: fish, vegetables, fruit hang embalmed in spice-and-vinegar; a certain alteration, a slight intensification of taste, is a small matter, surely? The art is to change the flavour in degree, but not in kind; and above all (in my thirty jars and a jar) to give it shape and form – that is to say, meaning.”

The question of essences remains at the heart of the adaptive act: how to make a second version of a first thing, of a book or film or poem or vegetable, or of yourself, that is successfully its own, new thing and yet carries with it the essence, the spirit, the soul of the first thing, the thing that you yourself, or your book or poem or film or your pre-pickle mango or lime, originally were.

Is it impossible? Is the intangible in our arts and our natures, the space between our words, the things seen in between the things shown, inevitably discarded in the remaking process, and if so can it be filled up with other spaces, other visions, that satisfy or even enrich us enough so that we do not mind the loss? To look at adaptation in this broad-spectrum way, to take it beyond the realm of art into the rest of life, is to see that all the meanings of the word deal with the question of what is essential – in a work adapted to another form, in an individual adapting to a new home, in a society adapting to a new age. What do you preserve? What do you jettison? What is changeable, and where must you draw the line? The questions are always the same, and the way we answer them determines the quality of the adaptation, of the book, the poem, or of our own lives.

So what of the adaptations in this week’s Oscars? In 1921, F Scott Fitzgerald wrote an odd little story called “The Curious Case of Benjamin Button”, about the birth, to “young Mr and Mrs Roger Button”, of a male baby who is born as a 70-year-old man and who then lives backwards, getting younger all the time, until at the end of his life, baby-sized and shrinking slowly in his white crib, he is sucked away into nothingness. In 2008, this little squib of a tale was turned by Brad Pitt and the director David Fincher into a $200m film.

However, the difference between the story and the movie is unusually great. In Fitzgerald’s story, Benjamin is born as a full-sized septuagenarian male. It is never explained how Mrs Button managed to give birth to such a large baby without being torn in half. Indeed, Mrs Button never gets a look-in. In the story, Benjamin’s life is lived largely in the private sphere, apart from an excursion to fight in the Spanish-American war, while in the movie he becomes involved in so many of the public events of his time that the picture might almost have been called Zelig in Reverse, or perhaps Forrest Gump Goes Backwards. (The screenwriter of Forrest Gump, Eric Roth, who adapted that screenplay from the novel by Winston Groom, is also responsible for Benjamin Button

Perhaps the biggest difference between the two works is that, other than sharing the idea of a man who lives backwards in time, their stories are entirely different; the film is not really an adaptation of the book, but almost entirely Roth’s creation. And while Roth and Fincher’s film is essentially a bravura special-effects performance helped by two fine acting performances, by Pitt and Cate Blanchett, it doesn’t finally have anything in particular to say. Fitzgerald’s story is at least a comedy of snobbery and embarrassment which, while maintaining a deliberately frothy and light tone, enjoyably satirises the social attitudes of late 19th and early 20th-century Baltimore.

Everyone accepts that stories and films are different things, and that the source material must be modified, even radically modified, to be effective in the new medium. The only interesting questions are “how?” and “how much?” However, when the original is virtually discarded, it’s difficult to know if the result can be called an adaptation at all.

There are, after all, other well-known stories of time-reversal that precede the Fincher/Roth film. In Martin Amis’s 1991 novel Time’s Arrow, the story of the Holocaust is told in reverse, so that, in one extraordinary scene, kindly Nazi doctors in a concentration camp fetch gold from their private stores and use it to put fillings into the teeth of Jewish dental patients. But in Time’s Arrow everything, and not just one single life, goes backwards. Perhaps the best known example of another Button-style reversal is the character of the wizard Merlyn in TH White’s 1938 classic The Sword in the Stone, itself the subject of a Disneyfied adaptation over which it would be best to draw a veil. Merlyn, the teacher of the boy known as Wart, the future King Arthur, lives backwards in time, and thus has the great advantage of knowing the future while being confused about the past. Benjamin Button has no such luck. He’s old and robotic, but as ignorant as any new-born babe. On the other hand, he grows into Brad Pitt, so things are not all bad.

What can one say about Slumdog Millionaire, adapted from the novel Q&A by the Indian diplomat Vikas Swarup and directed by Danny Boyle and Loveleen Tandan, which won eight Oscars, including best picture? A feelgood movie about the dreadful Bombay slums, an opulently photographed movie about extreme poverty, a romantic, Bollywoodised look at the harsh, unromantic underbelly of India – well – it feels good, right? And, just to clinch it, there’s a nifty Bollywood dance sequence at the end. (Actually, it’s an amazingly second-rate dance sequence even by Bollywood’s standards, but never mind.) It’s probably pointless to go up against such a popular film, but let me try.

The problems begin with the work being adapted. Swarup’s novel is a corny potboiler, with a plot that defies belief: a boy from the slums somehow manages to get on to the hit Indian version of Who Wants to Be a Millionaire and answers all his questions correctly because the random accidents of his life have, in a series of outrageous coincidences, given him the information he needs, and are conveniently asked in the order that allows his flashbacks to occur in chronological sequence. This is a patently ridiculous conceit, the kind of fantasy writing that gives fantasy writing a bad name. It is a plot device faithfully preserved by the film-makers, and lies at the heart of the weirdly renamed Slumdog Millionaire. As a result the film, too, beggars belief.

It used to be the case that western movies about India were about blonde women arriving there to find, almost at once, a maharajah to fall in love with, the supply of such maharajahs being apparently endless and specially provided for English or American blondes; or they were about European women accusing non-maharajah Indians of rape, perhaps because they were so indignant at having being approached by a non-maharajah; or they were about dashing white men galloping about the colonies firing pistols and unsheathing sabres, to varying effect. Now that sort of exoticism has lost its appeal; people want, instead, enough grit and violence to convince themselves that what they are seeing is authentic; but it’s still tourism. If the earlier films were raj tourism, maharajah-tourism, then we, today, have slum tourism instead. In an interview conducted at the Telluride film festival last autumn, Boyle, when asked why he had chosen a project so different from his usual material, answered that he had never been to India and knew nothing about it, so he thought this project was a great opportunity. Listening to him, I imagined an Indian film director making a movie about New York low-life and saying that he had done so because he knew nothing about New York and had indeed never been there. He would have been torn limb from limb by critical opinion. But for a first world director to say that about the third world is considered praiseworthy, an indication of his artistic daring. The double standards of post-colonial attitudes have not yet wholly faded away.

There is a widely held view among movie-lovers that films made from original screenplays are and must be held to be superior to films made by adapting plays or books. The brilliant books of recent times that have undergone cinematic transmutation include – to offer a very incomplete list – Martin Amis’s The Rachel Papers, Ian McEwan’s Atonement, Ishiguro’s The Remains of the Day, Graham Swift’s Last Orders, Peter Carey’s Oscar and Lucinda, Patrick McGrath’s Spider, Günter Grass’s The Tin Drum, Gabriel García Márquez’s Love in the Time of Cholera, Innocent Eréndira and Chronicle of a Death Foretold, Philip Roth’s The Human Stain and Short Cuts from the stories of Raymond Carver. (Independence Day, the movie, was of course not an adaptation of Richard Ford’s award-winning novel, which unfortunately came out at much the same time as the film, so that, according to legend, when customers in bookstores requested the book, the booksellers were obliged to ask: “With or without aliens?”)

Of this particular list, however, perhaps only Volker Schlöndorff’s film of The Tin Drum is worth talking about as a film, and this imbalance between good and bad adaptations strengthens the argument of the anti-adaptation lobby. Short Cuts betrays Carver’s vision by moving most of his characters up the social scale, where their barely suppressed despair looks like self-indulgence. And down at the very bottom of the barrel is the film of The Human Stain, which casts, in the role of an African-American man who manages to pass for white for much of his life, the actor Anthony Hopkins, a light-skinned Welshman.

The anti-adaptation, pro-original-screenplay argument was once expounded to me with immense vehemence by a somewhat inebriated British film producer, who said, with a certain amount of fist-pounding on our hosts’ dinner-table, that all movies made from books are shit. It is certainly possible to make a strong argument for the Shit Position. The Human Stain does not stand alone. The films of almost all the books I’ve just mentioned are failures, tedious, lazy and limp, where the originals are gripping, energetic and taut. The films of García Márquez’s masterpieces, in particular, are travesties, replacing the writer’s imaginative precision with a lazy exoticism that betrays the originals profoundly without even knowing it is doing so.

However, Schlöndorff’s Tin Drum stands as a magnificent exception with, at its heart, the electric performance of David Bennent as Oskar Matzerath, the Peter Pan among the million lost boys and murderous pirates of Nazi Germany: little, stunted Oskar, the other boy in classic literature who never grew up. I’ve tried to find more films that disprove the British producer’s dictum, and could add, for example, the Coen brothers’ No Country for Old Men, a film that succeeds by keeping very close, scene by scene, line of dialogue by line of dialogue, to Cormac McCarthy’s novel, and There Will Be Blood, which succeeds by the opposite method, making a free, loose and largely successful adaptation of Upton Sinclair’s novel Oil!; but the failures are so much more frequent than the successes.

The auteur theory of film-making was first expressed by François Truffaut in Cahiers du Cinéma in the late 1950s, and amplified, first as film theory and then in the making of actual films, by a group of critics who would turn into some of the world’s most important film-makers: Truffaut, Jean-Luc Godard, Claude Chabrol, Eric Rohmer and Jacques Rivette. But even though the idea of the superiority of scripts written as original screenplays rather than adaptations lay at or near the heart of the French New Wave, many of the finest works of French, and indeed world cinema in the 50s and 60s were, in fact, successful adaptations. Godard, a devotee of the original screenplay, had his greatest commercial success with Le Mépris (“Contempt”) which was based on a novel by Alberto Moravia. Chabrol made a terrific film from Cecil Day Lewis’s pseudonymously written thriller The Beast Must Die, or, in French, Que la Bête Meurt; Rohmer brilliantly filmed the classic novella by Heinrich von Kleist, Die Marquise von O …; and then there’s Jules et Jim, from the novel by Henri-Pierre Roché.

The immensely rich world cinema of the same era likewise went some distance towards exploding the Shit Principle. Kurosawa’s early samurai masterpieces Yojimbo and Sanjuro had literary originals, although The Seven Samurai came from an original screenplay; and Rashomon was made by combining two short stories by Ryunosuke Akutagawa. Satyajit Ray took much from classic Bengali literature, and some of his greatest films, such as Charulata and The Home and the World, are adapted more or less faithfully from originals by Rabindranath Tagore. Ingmar Bergman and Federico Fellini invariably filmed their own original screenplays, but Luis Buñuel was less dogmatic and made some of his most successful films by allying his own anarchic, surrealist tendencies to classic European literature, adapting for instance Belle de Jour by Joseph Kessel.

The case against film adaptations thus remains unproven and, when we look below the level of great literature, a plausible argument can be made that many cinematic adaptations are better than their prose source materials. I would suggest that Peter Jackson’s Lord of the Rings films surpass Tolkien’s originals, because, to be blunt, Jackson makes films better than Tolkien writes; Jackson’s cinematic style, sweeping, lyrical, by turns intimate and epic, is greatly preferable to Tolkien’s prose style, which veers alarmingly between windbaggery, archness, pomposity, and achieves something like humanity, and ordinary English, only in the parts about hobbits, the little people who are our representatives in the saga to a far greater degree than its grandly heroic (or snivellingly crooked) men.

My personal experiences with adaptation have been … well, mixed, though they are improving. Things got off to a bad start. One of the producers of Richard Attenborough’s Gandhi said she was keen to make a film of Midnight’s Children, except for one small part, which she found weak and redundant. Unfortunately, this small part was the climax of the novel, in which the Indian prime minister, Indira Gandhi, was, more or less, the villain. “You just don’t need that,” the producer said, “the book is so much better without it.” That project did not go forward. In the years that followed there were other abortive attempts to film the novel. The nearest miss was in 1997. There was a plan for a five-part BBC mini-series, for which I wrote the adaptation myself. This taught me much of what I know about adaptation, in particular the need for ruthlessness with the extremely long original, combined with a determination to fillet out and preserve its essence. The series was never made because of last-minute political problems in Sri Lanka, where the principal photography was to have taken place, and that was a huge disappointment.

A couple of years later, however, I was able to use that experience, and some of the work, in a theatre adaptation of Midnight’s Children, directed by Tim Supple, which the Royal Shakespeare Company performed both in Britain and in America. Theatre is a different beast – it’s so present, the play’s being right there in front of you makes it such an insistently declaratory form (except in the hands of a Beckett or Pinter, who turn its normal rules upside down); and what is true of the theatre in general is doubly true of epic theatre. As a result the stage adaptation of Midnight’s Children differed in two striking ways from the book: first, it was much more noisily, obviously political, putting the political material front and centre instead of using it more suggestively, in the background, as the novel often does; and, second, there was a lot more sex. I mean: a lot more.

Speaking as the author of the adaptation as well as the novel, I liked these differences. I thought of the play as a sort of second cousin of the book – perhaps its illegitimate child; its relative, not its mirror-image, and I thought its brasher, more aggressively in-your-face style was powerful and effective and properly theatrical, while remaining true to the book’s spirit. The response from audiences was interestingly divided. It soon became clear that the people who most enjoyed the show were those who had not read the novel.

This production was in marked contrast to Supple’s, marvellous, fluid, magical adaptation of Haroun and the Sea of Stories for the National Theatre. So perhaps the problem was me. We’ll find out soon enough, because I’m about to do it again, this time for the movies. There’s a new project to film Midnight’s Children, this time with my friend Deepa Mehta, director of the Oscar-nominated Water, and in a few months’ time she and I will be settling down to work out how you preserve the essence of a 600-page novel in a 100-page screenplay. There are some obvious decisions to be made. Can we really tell the story of all the novel’s three generations, or ought we to concentrate on Saleem’s own life? But then, would it be Midnight’s Children, especially without the “perforated sheet” – the episode in which Saleem’s grandparents fall in love through a sheet with a hole in it? And again: what language should the film be in, as many of the book’s characters would not really be speaking in English? (Here we may actually have been helped by Slumdog Millionaire, which is accustoming international cinema-goers to an Indian movie in which the dialogue is partly in English and partly in subtitled Marathi and Hindi.) And what shall we do about Saleem’s enormous nose?

The essence of a work to be adapted may lie anywhere – in the frame-stories that tell us, for example, how Superman became super, why Batman became batty, or why the Joker jokes. It may lie in a story’s unique atmospherics – the depression-era bigotries of a small Alabama town as seen through a young girl’s eyes – or it may lie in a character’s interiority, the inner life of Holden Caulfield or of Proust’s narrator Marcel. That these essences can be understood and captured on film is exemplified by, for example, Raul Ruiz’s great film of Proust’s Time Regained, or Robert Mulligan’s film of To Kill a Mockingbird, or Heath Ledger’s extraordinary incarnation of the Joker in The Dark Knight

Most difficult of all for the adapter are those texts whose essences reside in language, and this may explain why all those García Márquez movies were so bad, why there have never been good films made from the work of Italo Calvino or Thomas Pynchon or Evelyn Waugh (though there are many snobbery-choked versions of Brideshead Revisited), why movies of Hemingway so often misfire (I’m thinking of The Old Man and the Sea, with Spencer Tracy cast horribly adrift with a dead fish), and why even a really good try such as Joseph Strick’s 1967 attempt to film Joyce’s Ulysses doesn’t fully match up to the original, even though it is perfectly cast, with Milo O’Shea as an uncannily good incarnation of Leopold Bloom, and Maurice Roëves as a more than adequate Stephen Dedalus. When it does succeed, it does so, like Huston at the end of The Dead, by surrendering to Joyce’s language completely. In the final scene of Ulysses, Barbara Jefford as Molly Bloom lounges and rolls promiscuously upon her marital bed, and delivers in voiceover the grandest soliloquy in any novel, and as yes she says yes she says yes, the world of Joyce’s tongue comes fully alive at last.

What is essential? It’s one of the great questions of life, and, as I’ve suggested, it’s a question that crops up in other adaptations than artistic ones. The text is human society and the human self, in isolation or in groups, the essence to be preserved is a human essence, and the result is the pluralist, hybridised, mixed-up world in which we all now live. Adaptation as metaphor, to paraphrase Susan Sontag, adaptation as carrying across, which is the literal meaning of the word “metaphor”, from the Greek, and of the related word “translation”, another form of carrying across, this time derived from Latin.

What are the things we think of as essential in our lives? The answers could be: our children, a daily walk in the park, a good stiff drink, the reading of books, a job, a vacation, a baseball team, a cigarette, or love. And yet life has a way of making us rethink. Our children move away from home, we move away from our favourite park, the doctor forbids us to drink or smoke, we lose our eyesight, we get fired, there’s no time or money to take a vacation, our baseball team sucks, our heart is broken. At such times our picture of the world hangs crookedly on the wall. Then, if we can manage it, we adapt. And what this shows us is that essence is something deeper than any of that, it’s the thing that gets us through. The 12 separate varieties of finches that Charles Darwin found on the Galápagos Islands had all made local adaptations, but when the ornithologist John Gould examined Darwin’s specimens in 1837, he could see that these were not different birds, but 12 variations of the same bird. In spite of random mutation and natural selection, their finchness, their essence, was intact.

As individuals, as communities, as nations, we are the constant adapters of ourselves, and must constantly ask ourselves the question wherein does our finchness lie: what are the things we cannot ever give up unless we wish to cease to be ourselves?

We can learn this much from the poets who translate the poetry of others, from the screenwriters and film-makers who turn words on the page into images on a screen, from all those who carry across one thing into another state: an adaptation works best when it is a genuine transaction between the old and the new, carried out by persons who understand and care for both, who can help the thing adapted to leap the gulf and shine again in a different light. In other words, the process of social, cultural and individual adaptation, just like artistic adaptation, needs to be free, not rigid, if it is to succeed. Those who cling too fiercely to the old text, the thing to be adapted, the old ways, the past, are doomed to produce something that does not work, an unhappiness, an alienation, a quarrel, a failure, a loss.

But those who do not know who they are, are doomed too: individuals who sacrifice themselves for the sake of pleasing others, comedians who stop telling jokes because they find themselves in a humourless world, serious people who start trying to tell jokes because they fear being thought humourless, people in a new situation, a new relationship, a new university, who act against their natures because they think that’s the way to make things easy for themselves.

Whole societies can lose their way through a process of bad adaptation. Striving to save themselves, they can oppress others. Hoping to defend themselves, they can damage the very liberties they believed to be under attack. Claiming to defend freedom, they can make themselves and others less free. Or, seeking to calm the violent hotheads in their midst, societies can try to appease them, and so give the violent hotheads the notion that their violence and hotheadedness is effective. Wishing to create better understanding between peoples, they can seek to prevent the expression of opinions unpalatable to some of their members, and so immediately make others even angrier than they were before.

Societies in motion, at a time of rapid change such as the present day, succeed, as all good adaptations do, by knowing what is essential, what cannot be compromised, what all their citizens must accept as the price of membership. For many years now, I’m sorry to say, we have lived through an era of bad social adaptations, of appeasements and surrenders on the one hand, of arrogant excesses and coercions on the other.

We can only hope that the worst is over, and that better movies, better musicals and better times lie ahead.

James Lasdun on a provocative retelling of the Holocaust’s horrors through Nazi eyes

One approaches the fictionalisation of any aspect of the Holocaust with suspicion. Art is always at some level entertainment, and the idea of being entertained, however skilfully, by this particular set of horrors seems inherently objectionable.

Jonathan Littell’s The Kindly Ones reprises the familiar atrocities, in graphic detail and at massive length, from the viewpoint of an SS officer intimately involved in their execution. The book, which has already won the Prix Goncourt (Littell grew up in Paris and wrote it in French), is certainly skilful. It’s certainly objectionable too, deliberately so: the resentment and repugnance it arouses are evidently a part of its underlying calculus. One’s abiding sense of something pornographic about the whole enterprise is orchestrated – cleverly, horribly – into the comprehensive disgust at one’s own species that it seems intent on arousing. Despite the title (the Kindly Ones are the Eumenides, agents of catharsis in Greek drama), this is not one of those works that set out to leave you with a feeling of teary uplift about the Holocaust (a strong point in its favour).

The officer, Max Aue, a lawyer in civilian life, presents himself as a cultivated man; a Nazi by conviction rather than expedience, intellectually scrupulous, highly disciplined but also sensitive (the mass killings he observes and helps to organise sicken him to the point of repeated breakdown, even as he constantly affirms, explains and re-explains his belief in their necessity). In its coupling of high culture with demonic brutality, his character seems premised on Celan’s line about the preservation of Goethe’s oak tree in Buchenwald: “They build the camp, they respect the oak.”

The story he tells, roughly speaking, begins with his acceptance of an assignment on the ill-fated eastern campaign, to write reports on the implementation of steadily darkening directives concerning the “Jewish question”. As the early victories in the Caucasus (where the unpleasant business of butchering Jews is alleviated by enlightening opportunities to visit, say, the site of Lermontov’s duel) stall in the face of Russian resistance and winter, Aue falls foul of his commanding officer, and finds himself dispatched to Stalingrad. Here, after witnessing freshly apocalyptic levels of horror, he is shot through the head, but miraculously survives, waking up back in Berlin to find Himmler pinning a medal to his chest. Following his recovery he decides, after some hesitation (by now he is afflicted by nightmares, hallucinations, constant diarrhoea and vomiting), to devote his talents to ironing out the various administrative difficulties raised by the Endlösung, the final solution. In this capacity, he becomes embroiled in a bureaucratic wrangle between Eichmann, who wants to exterminate as many Jews as possible, and Albert Speer, who wants to keep as many as possible alive to use as slave labour: one of many grotesque moral dilemmas the book explores with gloomy brilliance.

A common effect of reading histories of the Holocaust is the helpless desire for exegesis that they leave you with. The eagerness with which Hannah Arendt’s line about the “banality of evil” has been seized on as holy writ is a measure of the intensity of this need. One way of looking at Littell’s novel is as an attempt to use the resources of fiction to supply this missing dimension: a kind of gigantic thought-experiment whereby the reader is situated in the Nazi nightmare subjectively, via the consciousness of a living, thinking, tormentedly willing participant, whose nervous system reacts more or less humanly to the inferno surrounding him, whose mind is endowed with a conveniently encyclopedic frame of reference to help us make sense of it. He perhaps even, thanks to his head injury, possesses the legendary pineal “third eye”, capable of penetrating into the spiritual essence of things.

In this respect the book rises impressively, even magnificently, to its own occasions, building out of its fact-crammed but stately sentences (the impersonal prose resembles that of a mandarin memoir) vast and phosphorescent tableaux vivants seething with Dantesque detail. If you have an interest in feeling your way into the administrative manoeuvring, the pseudo-scientific argumentation about language and race, and the mass of period-specific social and sensory minutiae that comprised the human reality out of which arose, say, the massacre at Babi Yar, or the final death march from Auschwitz; if you care to revive, in your own psyche, the finer points of cannibalism in Stalingrad or the emotional impact on the war-weary Gauleiters of Himmler’s call, at Posen, for total genocide, then this is undoubtedly a book you should read.

It does, however, have some large flaws. The most serious, for me, was Littell’s decision to equip his protagonist with a radically abnormal set of psycho-sexual characteristics. An erotic obsession (briefly consummated) with his twin sister Una, a murderous hatred of his mother and stepfather, and an inconsolable grief for his absconded father, form the basic elements of his inner life. Rejected by Una as a lover, he compensates by taking male lovers in order to pretend to himself that he is Una, as they penetrate him. And on a sick leave in the south of France he hacks his mother and step-father to death with an axe (he denies this, but is pursued by a pair of semi-comic furies in the form of two doggedly relentless Kripo cops, throughout the rest of the war).

These quirks establish Aue as a familiar literary type: the rococo personality who believes himself to be impeccably classical. They also fulfil the standard trope equating Nazism with extreme kinkiness. The problem is that they undermine Aue’s repeated claim (one that seems to represent the basic philosophical claim of the book) that we, the readers, are no different and would have behaved just as he did under similar circumstances: “The real danger for mankind is me, is you.” This Baudelairean assertion may be valid in theory, but a character who gets his kicks watching mother and stepfather eating the sausages he’s just sodomised himself with seems perhaps not the most persuasive basis for a claim that we’re all alike. At least Eichmann’s “banality” is something most people can relate to.

What this novel offers instead is a study in the exoticism of evil. The more perverse it gets, the less representative Aue seems of anything other than himself. By the end, after extended scenes of him having sex with half the furniture in his sister’s abandoned house, we’re left with a pure singularity: a ghoul belonging more to the fictional universe of, say, Bret Easton Ellis’s American Psycho (there’s an interesting affinity between Aue’s cultural name-dropping and the brand-name obsession of Ellis’s Patrick Bateman) than the real-world squalor of Nazi Germany.

Related to this dubious claim of kinship is an intermittent suggestion of larger historical mirrorings. An anachronistic use of the word “terrorists” for the Maquis seems slyly aimed at current political discourse, as does the word “surge” in a remark about the floundering German war effort. And several scenes of soldiers photographing their own atrocities inevitably recall Abu Ghraib.

All of which is interestingly provocative. More troubling is a persistent effort to establish underlying reciprocities between the Nazis and the Jews. In one of his hallucinatory moments, Aue sees Hitler metamorphosing into a rabbi. Another character, citing Disraeli’s Coningsby as a proto-Nazi text with its paean to the Chosen People as “an unmixed race … the aristocracy of nature”, observes that the Jews “are our only real competitors … Our only serious rivals”. Eichmann worries that sparing the strongest Jews for slave labour will create “the strongest biological pool”, and that “in 50 years everything will start all over again”.

Is that intended as an oblique reference to the present? Maybe, maybe not. But this, from Aue himself as he reflects on the Warsaw uprising, surely is: “It’s the Jews who are becoming warriors again, who are becoming cruel, who also are becoming killers. I find that very beautiful.” It’s difficult to read that without thinking of certain contemporary commentators who take pleasure in likening modern Israel to Nazi Germany; harder still to figure out where Littell himself stands in relation to his protagonist’s sentiments. At moments like this, the authorial detachment he cultivates with such magisterial elegance seems evasive.

At one point, Aue quotes the critic Maurice Blanchot’s description of Moby-Dick as a work that “presents the ironic quality of an enigma and reveals itself only by the questions it raises”. I suspect this may express something of Littell’s ambitions for his own monumental inquiry into evil. To say that it falls short of Melville’s visionary originality (and lacks, also, the breadth and vitality of Tolstoy, despite the claims of some reviewers) is hardly a criticism. It’s a rare book that even invites such comparisons, and for all its faults, for all its problematic use of history, The Kindly Ones does just that.

• James Lasdun’s short story collection It’s Beginning to Hurt is published by Cape in April.

Charlotte Higgins studies the latest theories of the death of Athens’s great philosopher

Early Greek history “sets a special challenge to the disciplined mind. It is a game with very few pieces, where the skill of the players lies in complicating the rules”. So wrote Iris Murdoch in her novel The Nice and the Good. Strictly speaking, she was referring to the archaic period. But in practical terms, it could be extended to embrace the whole of ancient history, where sources are few; or, rather, appear in a sudden floods (usually associated with a very well-preserved writer such as Cicero) closely followed by frustrating periods of drought. Historians must wring every last drop of juice from this or that inscription, potsherd, or literary source, proceeding with painstaking care and engaging in minute acts of close-reading. But then the fun of it is that they may make the most extraordinary leaps of the imagination to bridge the gaps. This process, of almost pettifogging exactitude combined with what some might regard as little short of fantasy, can be frustrating. But it is this marriage of precision, abstract thinking and creativity that makes ancient history so absorbing and endlessly fresh. Someone is always coming along and knocking down the fragile house of cards constructed by the last thinker, and boldly building another elegant edifice.

Robin Waterfield, in Why Socrates Died, has his moments of unbridled creation. For instance, with an attractive flourish, he creates, from hints and later writings, a putative text of one of the prosecution speeches at Socrates’s trial in 399BC. If we had the real document, many mysteries from more than two millennia ago would be solved. It is a mark of the clarity, confident arguing and good sense of Waterfield – known for his translations of Plato and Herodotus as well as a previous historical work of non-fiction, Xenophon’s Retreat – that the version he invents reads so plausibly at the climax of his book.

It is almost impossible to overestimate the historical importance of the trial and execution of Socrates. Plato more or less invented philosophy as we know it in the wake of, and because of, his teacher’s death. As Emily Wilson wrote in her excellent book The Death of Socrates, “the only death of comparable importance in our history is that of Jesus”. Wilson’s work is primarily concerned with Socrates’s posthumous career as (variously) martyr, hero, villain and saint. By contrast, Waterfield’s book (and the two make good companion pieces) is an investigation into the reasons he was killed.

These reasons, on the face of it, are opaque. What harm did Socrates ever do anybody? Famously, he was a philosopher who never wrote anything; he refused money for his teachings; and he took no active part in politics. All he did was wander around Athens talking to people. For admirers of Athens’s radical democracy, his execution remains a traumatic subject. How could a society that championed free speech condemn an apparently innocent 70-year-old?

The charges against him were of not acknowledging the city’s gods; of introducing new gods; and of corrupting the young men of Athens. But what precisely did that all mean? The charge of introducing new gods seems particularly peculiar – true, Socrates talked of his daimonion, the little voice in his head that guided his actions, but that hardly seems worthy of the death penalty. Equally, Socrates emerges from Plato and Xenophon’s writings as perfectly observant of conventional religious ritual. (His dying wish, according to Plato, was that a cock be sacrificed to Asclepius – to which mysterious injunction we shall return.) Which leaves us with the corrupting of the young.

The cleverness of Waterfield’s richly told and enjoyable book is that he uses the death of Socrates as a way of introducing a wonderfully full picture of Athens in the fifth century. His contention is that to understand Socrates’s demise we need to understand the city – its legal system, its politics, its generation of rich, clever young-men-in-a-hurry, its aristocratic culture of late-night partying, and, in particular, its war. In as clear an exegesis of the Peloponnesian war as the general reader will find, Waterfield builds up a cogent picture of a power-hungry, restless democracy that came under unbearable stress through its exhausting war with Sparta, and put itself at the mercy of ambitious, often unscrupulous politicians – not least among them Alciabiades: fashion icon, sexually voracious bon vivant, national traitor, mercurial military commander.

The most important reason Socrates was condemned, argues Waterfield, was his association with this young generation of controversial men such as Alciabiades. He skilfully draws out Socrates’s probable anti-democratic leanings in his vivid description of the brutal oligarchic revolutions that engulfed the city in 411 and 404. Critias, one of the most bloody figures of that second coup, was a pupil of Socrates. In 399, the philosopher was unfinished business; a sore on the face of the restored democracy. That is why, argues Waterfield, he had to die. And the cock sacrificed to Asclepius? Waterfield’s intriguing theory is that the gesture relates to Socrates’s offering himself as a scapegoat, a sort of self-sacrifice to heal the wounds of the bruised city-state. But no doubt someone will be along soon to overturn that particular house of cards.

• Charlotte Higgins is the author of It’s All Greek to Me (Short Books). To order Why Socrates Died for £18 with free UK p&p call Guardian book service on 0330 333 6846 or go to guardian.co.uk/bookshop

Hilary Mantel enjoys a well-crafted story of loneliness that harks back to an unhappy childhood

When Paul Sturgis lies down at night in the hope of sleep, he walks back in memory to his childhood house. He peers at his mother’s dressing table, and grows nostalgic over what he lacks in his present life: “a hallway with a proper hallstand”. Seventy-two when the book begins, Paul is a retired banker. He is comfortably off, like most of Brookner’s characters; necessity doesn’t coerce them, so if they make shoddy moral choices they have nothing external to blame.

Never married, Paul has only one living relative, Helena. Every six weeks or so he treks down from Hampstead to visit her in Kensington. Helena pretends to have a full social diary, but is practically a recluse: another elderly, unfriended person. We anticipate with relish the pain and embarrassment these two will cause each other: one a crass egotist, the other reticent and given to pained self-examination. But our pity for Paul’s loneliness is coloured, before long, by our perception that he too is a monster of egotism who hides from himself in a mist of self-deprecation. Like other Brookner creations, he is essentially homeless; indoors, he yearns for the street, and in the street he fears he may die. Sustained only by grilled sole, the pale repast of the dead, he haunts galleries and museums. Then, in the hopeless cause of cheering himself up, he locks up his gloomy flat and takes off to Venice.

Venice, in novels, is usually the cue for a flood of events, or metaphors at least. It is where Heather, in Brookner’s novel A Friend from England, learns some home truths. But Paul comes back from Italy much as he left England. We hear the barely suppressed sound of the author laughing up her sleeve. Though Paul’s depression has not lifted, one thing has occurred. In Venice he has given his phone number to a woman called Vicky, a chattering, pretty divorcee. Unreliable, and of no fixed abode since her marriage broke up, Vicky is full of restless appetites. Put food before her, and she eats it. “Inroads had been made into the ciabatta . . .” Back in London Paul takes her out for tea, pointing out on the way a house where Henry James lived. No gift for Jamesian circumlocution rubs off on Vicky. When they get back: “She indicated the bed. ‘I could stay if you like.’”

It is a tribute to Brookner’s stealthy technique that the moment is profoundly shocking. It is placed precisely one-third of the way through the book, and the reader feels almost as if a new book might begin, but – again, that authorial snigger – what Brookner arranges for Paul is not an affair but a bereavement. His cousin Helena dies, leaves him her flat and her money. If he was free before to please himself, he is now doubly so. He has turned down Vicky: “Her gesture towards the bed had had a utilitarian element, like that of a waiter indicating a vacant table.” But he can’t dismiss her from his life; restless, seeking what she calls “compensation” for her failed marriage, she goes on her travels, but always leaves a bag at his flat, then two bags; he is a man haunted by suitcases, mute leathery witnesses to all his vacillations. Whenever Vicky bobs into view, his arm shoots up in greeting – straight from his sea of solipsism, not waving but drowning.

In Jermyn Street he meets an old flame, Sarah, now disconcertingly aged. Sarah had once rejected him as being “too nice”; she meant too suburban, he suspects. Sarah seems to have got through life without rigorous examination of it. Cheating, surely? But now she is vulnerable, she might value his dependability. Should he perhaps marry her, or marry the flighty but vital Vicky, who might be suited to be his nurse in old age, if only she would stay in one place? Like Freud, Paul asks: “What do women want?” He wishes he’d had a sister, to explain their mysteries. In fact he has, but they are in other Brookner novels. It’s Freudian myths that bind and choke these people, glumly playing out the desire of daughters for fathers, mothers for sons, poisoned nostalgia at the root of their lives. The childhood Paul remembers, in that family house in Camberwell, was not a happy one. He tries and fails to move the skittish Vicky into Helena’s old flat. It’s “like a morgue”, she cheerfully complains, flitting elsewhere. When Paul goes in to clear up after her he finds a used teabag on the side of the bath. He imagines himself sitting in Helena’s flat all day, “awaiting some spectral visitor such as himself”. It is the most dismaying image in a narrative full of phantoms.

Brookner’s theme is stated in the early chapters and is then restated in sentences almost identical but not quite, each a comic and plangent variation on what has gone before. In this book as elsewhere, she subverts her characters ruthlessly and exposes them to humiliation, not only in the eyes of other characters but the eyes of her reader. Her technique as a novelist is so sure and so quietly commanding that we forget the essential artifice, and as soon as we meet the inhabitants of her rooms we start to advise them on redecoration; to psychoanalyse them; to fantasise about introducing them to other Brookner characters, whom surely they have narrowly avoided bumping into at the Wallace Collection or the Royal Academy. Her subtle, uncomfortable high comedy is poised on the brink of tragedy, but her characters would never claim tragic status for themselves. It would be ill-mannered, it might hint that they want something, if only a tear: something that impinged on others, to the point of the taking out of a handkerchief.

“Fate is rarely kind, and nature never,” she concludes briskly. Brookner is the sort of artist described as minor by people who read her books only once. A decade ago it was fashionable for critics to stamp their feet at her and insist she do something different. Her characters should get out more, cheer up or take some Prozac, go shopping, get a makeover. Readers more attuned to her could only shake their heads and warn: “You’ll understand it better, my dears, when you grow up.” This is her 23rd novel. You can start with any one of them. The singular quality of each, as well as the integrity of the project, is established. Each book is a prayer bead on a string, and each prayer is a secular, circumspect prayer, a prayer and a protest and a charm against encroaching night.

• Hilary Mantel’s novel Wolf Hall will be published by Fourth Estate in May.

‘Writing is the best rush I’ve ever found. I’m utterly, hopelessly addicted to it. I go into a kind of dream every day’

Life, says TC Boyle, “is tragic and absurd and none of it has any purpose at all.” He is sitting contentedly with a glass of wine in the west room of his Frank Lloyd Wright house in Montecito, California. “Science has killed religion, there’s no hope for the future with seven billion of us on the planet, and the only thing you can do is to laugh in the face of it all.”

T Coraghessan Boyle, as he used to call himself, has always enjoyed making mincemeat of conventional pieties. He emerged in the 1980s as a satirical novelist and short-story writer with a black sense of comedy and an exuberant prose style. He dressed like a rock star, and his self-chosen middle name, pronounced Cor-rag-essan, sounded like a battle cry. In 1993 he gave a famous free reading in Central Park with Patti Smith, and today, at 60, with 12 successful novels and a 750-page volume of short stories lined up in hardback on the burnished redwood shelf above his fireplace, he still looks like a punk Mephistopheles.

The house is a low, spreading, cruciform structure of redwood and glass, built in the prairie style with a Japanese influence, and Boyle’s latest novel, The Women, is about its architect. “I really didn’t know much about Frank Lloyd Wright when we bought the house in ‘93. Living here, I got curious and started reading about him and found out what a bizarre, outlandish character he was, with all this incredible turmoil in his personal life, and I knew I had to write about him.”

Architecture is touched on in The Women, but the novel’s main concern is Wright’s scandal-racked love life and how it was experienced by the four women involved. “All the events in the book are taken from the newspaper accounts and biographies, and I really put my soul into trying to keep the details accurate,” Boyle says. “Where the fictional process is at work is when I enter the heads of the characters and imagine what they were thinking, and why they did what they did.” He based his main narrator, a Japanese apprentice called Tadashi Sato, on the many international architecture students that Wright charged for the privilege of doing his cooking and cleaning, and who were required to obey all his commands without question.

Wright’s first wife was the long-suffering Kitty Tobin. They married young and had six children, and then he fell in love with one of her best friends, an early feminist called Mamah Borthwick Cheney, who was also married with children. Publicly announcing their freedom to follow their hearts and hounded by the press, Frank and Mamah went off to live together at Taliesen, a shimmering country estate in Wisconsin that Wright built as his own private utopia. In 1914, while Wright was away on business, Mamah was murdered there by a crazed manservant with an axe. In the same rampage, he killed her two visiting children and four other adults, wounding two more and setting a fire that burned Taliesen almost to the ground.

The next woman in his life was Maude Miriam Noel, a passionate, morphine-addicted Southern belle, and for Boyle, the most enjoyable character in the novel to write. “Miriam was beautiful, delusional, heartbreaking, and she did all these wild, insane things which to her made perfect sense. She came to dominate my life and the book because I found it so interesting being inside her head.”

After she left Wright, and he realised he no longer wanted her back, Miriam became consumed by vengefulness and spent the rest of her life trying to destroy him with increasingly deranged lawsuits, criminal complaints and media campaigns. Wright, meanwhile, had taken up with Olgivanna Milanoff, a statuesque Montenegrin beauty and follower of the Russian mystic Gurdjieff, who bore him two more children and became known as “the Dragon Lady” among the coterie of apprentices at the rebuilt Taliesen.

“Wright was a classic narcissistic personality,” says Boyle. “The kind of person who doesn’t care what other people want, or who they are, and can’t even imagine that they might have emotions and desires of their own. Other people existed only to serve his needs, and I find that fascinating in a cautionary way.”

Frank Lloyd Wright is not the first domineering genius to move from the pages of history into a TC Boyle novel. That distinction goes to John Harvey Kellogg, the inventor of corn flakes, who was the subject of his 1993 novel The Road to Wellville and the film of the same name. Then came the sex researcher Alfred C Kinsey in The Inner Circle, published in 2004. “I suppose the three of them do make a trinity, although I didn’t realise it when I started on Wright,” Boyle says. “They’re the great egomaniacs of the 20th century. I don’t think any of them would have made a good companion, let alone a husband, and if the three of them had ever met, they probably would have killed and eaten each other.”

All three surrounded themselves with acolytes whom they abused in various ways, and all three were genuine visionaries, who permanently changed the way we see personal health, sex and the possibilities of architecture. “The most bizarre was certainly Kellogg with his enema regimes and his crazed health-food obsessions, but he also had some good ideas – that we should eat less meat, take exercise and get fresh air. Kinsey was essentially a sexual predator who was bisexual at a time when that couldn’t be admitted, especially in his position as a respected professor of sex research. And Wright was a con man and he had to be. For me to make my art, all I need is a room, a computer or a typewriter and a ream of paper. For him to make his art, he had to convince a patron to lay out all this money, and it was never enough for what he wanted to do.”

Wright had very few repeat clients, and it wasn’t just because of financial chicanery. “He was so much of a control freak that he hated the idea that someone was going to move into his house, bring in their baggage and ruin his beautiful design. In a couple of cases he got all his own furniture made for a house and even designed the clothing of the housewife. It’s like a kid playing with a dollhouse and manipulating figures who aren’t really human.”

Similar criticism has been levelled against Boyle’s fiction. “Boyle is not psychological,” Lorrie Moore has written. “He’s all demography and zeitgeist.” The critic Bill Seligman has argued: “[He] can write and he can imagine, with more energy than any of his contemporaries. But energy isn’t enough; there’s only so far you can go on sheer technique. And until he goes further, he’ll remain a satirist cut off from the oxygen of morality.”

He has been accused of lacking proper sympathy for his characters and taking too much pleasure in heaping calamities on them and watching them squirm and flail. “It’s my universe, and by god they’re going to suffer,” Boyle says with a laugh. “Look, when I write funny, satirical stuff, I get criticised for not being serious. When I write moving, naturalistic stories, I get criticised for not being funny.”

More broadly, he’s been denigrated as an entertainer, a crowd-pleaser and laugh-getter, and to this he pleads enthusiastically guilty. “If we lose sight of the fact that writing is entertainment, then writing is doomed. Books are up against TV and movies and video games and a multimedia society that is so busy that people don’t have contemplative time any more. I worry deeply about this. In fact I worry about everything all the time. I used to be a punk. All I wanted to do was tear everything down, and that was so much easier.”

Boyle grew up in the leafy suburbs of Westchester County north of New York City. Born in 1948, he was a child of the 1960s and alcoholic parents. When he was young, he tried particularly hard to please them, as the children of alcoholics often do, and then at 15 he rebelled, rejecting Catholicism and embracing vandalism, alcohol, drugs, maniacal driving and the writing of Aldous Huxley, JD Salinger and Jack Kerouac. At 17 he arrived, saxophone in hand, at a small liberal arts college in Potsdam, New York, intending to study music and become a musician. He failed the audition and signed up instead for history and English, which had been his only good subjects at high school.

“One of the classes was the American short story, and that’s where I discovered Updike and Bellow and Flannery O’Connor, and it really changed everything. Then I got into black humour, Beckett and John Barth and Robert Coover, and the Latin Americans like García Márquez and Borges, and it was all a big inspirational stew that kept getting stirred. Then I blundered into a creative writing class and here I am.”

It wasn’t quite that simple. There was a weekend heroin habit that lasted two years until a friend overdosed and scared him into cleaning up, which took another two years and a lot of pills and alcohol. He wrote a story about his heroin experiences called “The OD and Hepatitis Railroad or Bust”, which was published by the North American Review. That inspired him to apply to the Iowa Writers’ Workshop, where so many of his literary heroes had studied, or taught, or both. He was accepted on the strength of that one story.

“Iowa is like a conservatory for writers instead of musicians. You go there to study with a master and that master may impart nothing to you, or he may be your coach and push you on your way, and you take your chances. I had three teachers – Vance Bourjaily, John Irving and John Cheever – all of whom were extremely generous to me and essentially said what I needed to hear: you’ve got talent, you’re on the right track, keep it up. I got time to learn, and time to write, and be in a place where writing is revered, and so many great writers came through there to read their work and stumble around drunk.”

He spent five and a half years at Iowa and left with a degree in creative writing, a PhD in 19th-century British literature and a friendship with Raymond Carver. “He was a very unpretentious, shy, demon-haunted and beautiful man, and I admired him greatly.” Carver was the leading short-story writer of his generation, well known for his bleak, minimalist style. Boyle yearned to emulate him but his style was already in the opposite camp – hectic and garrulous, full of quips and asides – and when he left Iowa, he hurled himself into a novel, writing in the morning for four or five hours, seven days a week.

That first novel was Water Music, a picaresque comedy about the 18th-century explorer Mungo Park, published in 1981, and Boyle has been working to the same schedule ever since. Despite the pessimism of his worldview, he counts himself as a happy and fortunate man, and this is because he takes such pleasure in his daily hours of writing. “It’s the best rush I’ve ever found and I’m utterly, hopelessly addicted to it. I go into a kind of dream every day. It’s wonderful.”

He writes in his study upstairs, always to music – “gloom, rain and suicidal cello concertos are best” – or in a remote house in the mountains of northern California, where he sequesters himself for weeks at a time, hiking, snowshoeing and fishing in the afternoons. Like so many contemporary American writers, he also teaches creative writing and is currently professor of literature at the University of Southern California, with a very light teaching burden.

“I have this wild-man image and I am a little crazy,” he says. “But at the same time I’m a tenured professor, hardworking and diligent and a good family man. Karen and I have three grown children and I must be the only American writer of my generation who has had only one wife.”

Unlike Frank Lloyd Wright, who required chaos and tumult to create his art, Boyle needs calm and order, a good dog and a restful night’s sleep. He begins his novels in a burst of creativity, slows down in the middle as he works out the irksome problems of plot and theme, and then, with the end in sight, goes into a frenzy to reach it. “I’m too exhausted at that point to begin another novel, so I write short stories instead. And when those peter out, it’s usually time to begin a new novel. It’s a good cycle for me. It keeps me from having that horrible blockage and downtime that so many novelists have after finishing a project.”

A new collection of his stories, Wild Child, will be published next year, and he has amassed another volume of his lifetime collected stories. This summer he hopes to complete his 13th novel, about ecological restoration in the Channel Islands off the California coast. “More and more what I write about is man’s relationship to nature, and my take on it is extremely depressing,” he says. He tackled climate change and ecological collapse in A Friend of the Earth, published in 2000, and now he has even less hope that an apocalyptic future can be averted. “I think it’s going to turn out like Cormac McCarthy’s The Road within 50 years. We’ll eat everything left to eat and then we’ll eat each other. But my plan, personally, is to die. That’s how I’m going to deal with it.”

Boyle on Boyle

None of the doctors could help her in Los Angeles or the provincial outpost of San Diego either, little people all of them, sniveling types, handwringers, an army of effete bald-headed men in spectacles who were mortified of the law – as if this law had any more right to exist than Prohibition, because who was the federal government to dictate what people could and couldn’t do with their bodies, their own minds, their personal needs and wants and compulsions? Were they going to regulate needs, then? Dole them out? Tax them? Miriam was so furious, so burned up and blistered with the outrage of it that she must have been overly severe with the cabman – the driver with his hat cocked back on his head and his trace of a Valentino moustache – because when they got to the border at Tijuana, he stopped the car, turned around in his seat and demanded payment in full. Insolently. Out of insolent little pig’s eyes.

• From The Woman, published by Bloomsbury

“This is my first chance to deeply inhabit a close third-person point of view of Miriam, the crazy harpie wife who would ultimately try to destroy Frank Lloyd Wright. She is clearly outraged about something but the reader doesn’t yet know what it is. Miriam, my favourite character in the book, is a woman with multitudinous problems, but here, as I introduce her, her problem is very simple. She needs morphine.”

Seventy-five years after C Day Lewis’s manifesto A Hope for Poetry, Peter Stanford asks what the collision of poetry and politics in that ‘low, dishonest decade’ has to tell us about our own straitened times

Literary and artistic movements are fond of issuing manifestos. There are several texts with a claim to set out the beliefs of the left-leaning “Thirties Poets”, a group whose legacy is still debated – most recently in tributes to their close collaborator, the writer Edward Upward, who died two weeks ago at the age of 105. However, A Hope for Poetry, Cecil Day Lewis’s slim but passionate statement of their shared aims, is the most enduring.

A Hope for Poetry, published three-quarters of a century ago, had an extraordinary impact. Day Lewis’s assertion that poets (and, in particular, himself and his colleagues WH Auden, Stephen Spender and Louis MacNeice) had a prominent role to play in articulating the pressing challenges of economic recession and rising unemployment ran to six editions in the years up to the second world war, three more before 1945 and was still being reprinted in the mid-1970s. A few months after it came out, the Evening Standard reported on a meeting between TE Lawrence and Winston Churchill. They were bemoaning the state of Britain when Lawrence remarked that he had been reading a book by “the one great man in these lands – his name is Cecil Day Lewis”.

Today it seems extraordinary that so much hope could be invested in poetry. Soon a new poet laureate will be named, but, despite Britain finding itself in an economic mess with clear parallels to the 1930s, no one is suggesting that the successful candidate will line up alongside politicians meaningfully to address the recession.

To the modern reader, there is a lost idealism – and a naivety – running through A Hope for Poetry. For all their claim to be speaking for the common man, the Thirties Poets, along with the prose writers Christopher Isherwood and Edward Upward who made up their circle, were Oxbridge-educated and middle class. Most were eking out their literary earnings by teaching in private schools. As revolutionary figures, their lives hardly matched their rhetoric – as MI5 discovered. Alerted by the attention the poets were generating with their radical talk, its agents were sent to spy on Auden, Day Lewis and Spender as subversives. The best “proof” that the agent who followed Day Lewis could come up with, however, was that he “seldom wears a hat and [is] not altogether smart in dress”.

Others pointed out publicly the irony of the poets’ status as would-be revolutionaries. Julian Bell, son of Clive and Vanessa Bell, later killed fighting on the republican side in the Spanish civil war, had been an early associate of the “Auden gang”, but subsequently condemned its political posturing as that of “enthusiastic boy scouts”.

A Hope for Poetry is in this regard not without self-knowledge. In a section defending the role of the poet in conveying the suffering caused by the slump, Day Lewis writes: “This kind of poetry . . . is animated by unsentimental pity and sacred indignation. It does not wish to make poetic capital out of the suffering of others.” He then quotes a Spender poem about the unemployed:

No, I shall weave no tracery of pen-ornament
To make them birds upon my
singing tree.

This, he claims, “probably gets as near to communist poetry as bourgeois writers under a bourgeois regime can hope to get”.

Yet it is a rare moment of restraint. The fighting talk that characterises the book is all about rejecting “the vague cri-de-coeur for a new world” in favour of “poems which show that the writer has emotionally experienced a political situation and assimilated it through his specific function into the substance of poetry”. The core of the argument is that the poets it seeks to promote are more than willing to dirty their own hands in turbulent times by abandoning “pure” poetry in favour of verse that may be “impure”, and may even border on propaganda, but which connects, on a human level, with what is going on in the country.

At the centre of the group was the didactic, compelling Wystan Auden. “I willingly became his disciple where poetry was concerned,” Day Lewis later wrote of their early meetings in Auden’s rooms in Christ Church, Oxford. Auden had much the same effect on Spender and MacNeice. Nevill Coghill, his tutor, would recall how Auden’s sayings “would appear in the essays of other pupils. These being cross-examined and their nonsense laid bare, [they] still held the trump: ‘Well, that’s what Wystan says’.” It was Auden, too, who provided the link between the young Oxford poets and their contemporaries Isherwood and Upward, who had been collaborating on writing their “Mortmere” stories at Cambridge. Auden was fond of reading out these macabre tales to his admirers.

The notion of them all as a single movement, however, is an exaggeration. History may lump their names together – most memorably as “MacSpaunday”, the beast dreamt up by an embittered rival, the South African poet Roy Campbell – but the first time Auden, Day Lewis and Spender were all captured in a single photograph was in Venice in 1949. And MacNeice nearly didn’t make it into A Hope for Poetry because, until then, he hadn’t met Day Lewis.

A Hope for Poetry is noticeably short on concrete proposals. By the time of its publication, MacNeice had already rejected communism in his 1933 poem “To a Communist”. Auden was still flirting with it. His opening remark from The Orators (1932) is quoted with approval in the manifesto: “What do you think about England, this country of ours where nobody is well?” One of the poems in The Orators was dedicated to Upward, who became a member of the party in 1932.

Day Lewis joined in 1936, as did Spender, and spoke at its rallies in industrial towns. While Spender tired of it quickly – and was disillusioned when he joined the trail of leftwing writers going to Spain to support the republicans in the civil war – Day Lewis stayed in longer, earning himself the label “Red Cecil”. Within the wider group, Upward stayed longest. He resigned only in 1948, and then because he believed the party to be insufficiently revolutionary.

It wasn’t only political and social concerns that brought the four MacSpaunday poets together, however. They shared much as writers. Again the dominant influence was Auden with his often savage satire, his habit of almost telegraphing his message, using the poetic equivalent of shorthand, and, above all, his liking for modern industrial imagery. The poems of all four from this period are full of trains, factories, mines, iron, steel and – in a memorable image from Spender – pylons. Day Lewis argues in A Hope for Poetry that there is “a social energy working through contemporary imagery so integrated with the poem as to release into it the external life they represent”.

There was in these poets at this time a touch of the pulpit, of evangelising for a better world. Three of them – Auden, Day Lewis and MacNeice – were sons or grandsons of clergymen. All had rejected conventional religion as undergraduates, but retained its impetus towards activism in the cause of social justice. In exploring this in verse, they had to navigate modernism, and their relationship to it was a complicated one. They resisted its urge to break poetic form in order to discover new meaning. Such an undertaking can, A Hope for Poetry warns, make modernist verse difficult, if not unintelligible, especially to a working-class readership.

They also questioned what they saw as modernism’s tendency, even when identifying political, social and economic issues, to do so as an observer rather than as an active participant for change. “We obscurely felt the need,” Day Lewis wrote later of this period, “to do more with the fragments than shoring them against our ruin.” Yet all admired The Waste Land, which had reignited enthusiasm for poetry in the post-war period. And three of the four Thirties Poets were published by Eliot at Faber.

What is most striking today is the furore that surrounded A Hope for Poetry. The poets were discussed on both the literary and political pages of newspapers and magazines. They were showered with invitations and plaudits. And, inevitably, it turned their heads: any doubts about the synthesis of poetry and politics were buried.

Yet it is now tempting to see the book less as an opening salvo than as a kind of swansong for a group whose moment was passing. The political climate that had made communism seem a plausible alternative for a polarised Britain changed as reports of Stalin’s brutality began to leak out of the Soviet Union. Auden was starting out on the path that led him back to Christianity and to the remark, in his 1940 elegy for WB Yeats, that “poetry makes nothing happen”. Later he was to suppress, or rewrite, some of his poems from that “low, dishonest decade” and played down suggestions that he had ever been part of a movement.

For the author of A Hope for Poetry disillusion was longer coming. In 1936, Day Lewis wrote a verse play, Noah and the Waters, which contained, whole and undigested, parts of The Communist Manifesto. It was panned by the critics, and a struggle ensued. On the one hand was a belief among the 30s writers – set out most forcefully by Upward in his 1937 essay “Sketch for a Marxist Interpretation of Literature” – that art and politics must be as one and that the only decent writing would come from the left. Day Lewis had great sympathy with such a view, as Noah and the Waters had shown. On the other hand, those close to Day Lewis – including Virginia Woolf – counselled him that communist propaganda and oratory were damaging his poetry. If he continued along that course, he risked being eclipsed – which was Upward’s fate, as a result of his determination, long after the rest of that 30s generation had fallen by the wayside, to produce prose that was true to his political principles. Upward’s 1938 novel, Journey to the Border, “dulled him into silence”, as Valentine Cunningham has put it, and it was to be many more decades before he appeared in print again.

Day Lewis, by contrast, chose literature over politics and, in 1938, withdrew from the fray to rural Devon. It was, most critics agree, the making of him as a poet. His next collection, 1943’s Word Over All, devoid of overt politics and full of the frailties of the human heart as he embarked on a love affair with the novelist Rosamond Lehmann, is widely regarded as his best.

Does A Hope for Poetry, 75 years on, still have a relevance? One admirer, the Irish poet and Oxford don Bernard O’Donoghue, believes it does. “It is a matter,” he says, “of seeing where hope lay, and for that matter where it lies for writers today. This sounds sentimental; but it represents something quite specific and indispensable in poetic discussion. What I mean by hope, in the context of Day Lewis’s book, is the view that poetry has something distinctive to offer to humanity… common endeavour, and is not, in Yeats’s tempting phrase, merely ’self-delighting, self-appeasing, self-affrighting’.”

But the last word rightly belongs to A Hope for Poetry itself. In a postscript added to the 1936 edition, Day Lewis wrote: “To the idea of poetry as exclusive, esoteric, amoral, the private affair of the poet, moving in a different world from prose, creative of its own reality, I should oppose the idea of poetry as catholic, diverse in function, moral, everyone’s business (potentially at any rate), assimilating not rejecting prose meaning, a way of synthesising and communicating reality.”