The Art of Conjecturing

Reading Katy Börner’s Atlas of Forecasts: Modeling and mapping desirable futures for New Scientist, 18 August 2021

My leafy, fairly affluent corner of south London has a traffic congestion problem, and to solve it, there’s a plan to close certain roads. You can imagine the furore: the trunk of every kerbside tree sports a protest sign. How can shutting off roads improve traffic flows?

The German mathematician Dietrich Braess answered this one back in 1968, with a graph that kept track of travel times and densities for each road link, and distinguished between flows that are optimal for all cars, and flows optimised for each individual car.

On a Paradox of Traffic Planning is a fine example of how a mathematical model predicts and resolves a real-world problem.

This and over 1,300 other models, maps and forecasts feature in the references to Katy Börner’s latest atlas, which is the third to be derived from Indiana University’s traveling exhibit Places & Spaces: Mapping Science.

Atlas of Science: Visualizing What We Know (2010) revealed the power of maps in science; Atlas of Knowledge: Anyone Can Map (2015), focused on visualisation. In her third and final foray, Börner is out to show how models, maps and forecasts inform decision-making in education, science, technology, and policymaking. It’s a well-structured, heavyweight argument, supported by descriptions of over 300 model applications.

Some entries, like Bernard H. Porter’s Map of Physics of 1939, earn their place thanks purely to their beauty and for the insights they offer. Mostly, though, Börner chooses models that were applied in practice and made a positive difference.

Her historical range is impressive. We begin at equations (did you know Newton’s law of universal gravitation has been applied to human migration patterns and international trade?) and move through the centuries, tipping a wink to Jacob Bernoulli’s “The Art of Conjecturing” of 1713 (which introduced probability theory) and James Clerk Maxwell’s 1868 paper “On Governors” (an early gesture at cybernetics) until we arrive at our current era of massive computation and ever-more complex model building.

It’s here that interesting questions start to surface. To forecast the behaviour of complex systems, especially those which contain a human component, many current researchers reach for something called “agent-based modeling” (ABM) in which discrete autonomous agents interact with each other and with their common (digitally modelled) environment.

Heady stuff, no doubt. But, says Börner, “ABMs in general have very few analytical tools by which they can be studied, and often no backward sensitivity analysis can be performed because of the large number of parameters and dynamical rules involved.”

In other words, an ABM model offers the researcher an exquisitely detailed forecast, but no clear way of knowing why the model has drawn the conclusions it has — a risky state of affairs, given that all its data is ultimately provided by eccentric, foible-ridden human beings.

Börner’s sumptuous, detailed book tackles issues of error and bias head-on, but she left me tugging at a still bigger problem, represented by those irate protest signs smothering my neighbourhood.

If, over 50 years since the maths was published, reasonably wealthy, mostly well-educated people in comfortable surroundings have remained ignorant of how traffic flows work, what are the chances that the rest of us, industrious and preoccupied as we are, will ever really understand, or trust, all the many other models which increasingly dictate our civic life?

Borner argues that modelling data can counteract misinformation, tribalism, authoritarianism, demonization, and magical thinking.

I can’t for the life of me see how. Albert Einstein said, “Everything should be made as simple as possible, but no simpler.” What happens when a model reaches such complexity, only an expert can really understand it, or when even the expert can’t be entirely sure why the forecast is saying what it’s saying?

We have enough difficulty understanding climate forecasts, let alone explaining them. To apply these technologies to the civic realm begs a host of problems that are nothing to do with the technology, and everything to do with whether anyone will be listening.

Sod provenance

Is the digital revolution that Pixar began with Toy Story stifling art – or saving it? An article for the Telegraph, 24 July 2021

In 2011 the Westfield shopping mall in Stratford, East London, acquired a new public artwork: a digital waterfall by the Shoreditch-based Jason Bruges Studio. The liquid-crystal facets of the 12 metre high sculpture form a subtle semi-random flickering display, as though water were pouring down its sides. Depending on the shopper’s mood, this either slakes their visual appetite, or leaves them gasping for a glimpse of real rocks, real water, real life.

Over its ten-year life, Bruges’s piece has gone from being a comment about natural processes (so soothing, so various, so predictable!) to being a comment about digital images, a nagging reminder that underneath the apparent smoothness of our media lurks the jagged line and the stair-stepped edge, the grid, the square: the pixel, in other words.

We suspect that the digital world is grainier than the real, coarser, more constricted, and stubbornly rectilinear. But this is a prejudice, and one that’s neatly punctured by a new book by electrical engineer and Pixar co-founder Alvy Ray Smith, “A Biography of the Pixel”. This eccentric work traces the intellectual genealogy of Toy Story (Pixar’s first feature-length computer animation in 1995) over bump-maps and around occlusions, along traced rays and through endless samples, computations and transformations, back to the mathematics of the eighteenth century.

Smith’s whig history is a little hard to take — as though, say, Joseph Fourier’s efforts in 1822 to visualise how heat passed through solids were merely a way-station on the way to Buzz Lightyear’s calamitous launch from the banister rail — but it’s a superb short-hand in which to explain the science.

We can use Fourier’s mathematics to record an image as a series of waves. (Visual patterns, patterns of light and shade and movement, “can be represented by the voltage patterns in a machine,” Smith explains.) And we can recreate these waves, and the image they represent, with perfect fidelity, so long as we have a record of the points at the crests and troughs of each wave.

The locations of these high- and low-points, recorded as numerical coordinates, are pixels. (The little dots you see if you stare far too closely at your computer screen are not pixels; strictly speaking, they’re “display elements”.)

Digital media do not cut up the world into little squares. (Only crappy screens do that). They don’t paint by numbers. On the contrary, they faithfully mimic patterns in the real world.

This leads Smith to his wonderfully upside-down-sounding catch-line: “Reality,” he says, ”is just a convenient measure of complexity.”

Once pixels are converted to images on a screen, they can be used to create any world, rooted in any geometry, and obeying any physics. And yet these possibilities remain largely unexplored. Almost every computer animation is shot through a fictitious “camera lens”, faithfully recording a Euclidean landscape. Why are digital animations so conservative?

I think this is the wrong question: its assumptions are faulty. I think the ability to ape reality at such high fidelity creates compelling and radical possibilities of its own.

I discussed some of these possibilities with Paul Franklin, co-founder of the SFX company DNEG, and who won Oscars for his work on Christopher Nolan’s sci-fi blockbusters Interstellar (2014) and Inception (2010). Franklin says the digital technologies appearing on film sets in the past decade — from lighter cameras and cooler lights to 3-D printed props and LED front-projection screens — are positively disrupting the way films are made. They are making film sets creative spaces once again, and giving the director and camera crew more opportunities for on-the-fly creative decision making. “We used a front-projection screen on the film Interstellar, so the actors could see what visual effects they were supposed to be responding to,” he remembers. “The actors loved being able to see the super-massive black hole they were supposed to be hurtling towards. Then we realised that we could capture an image of the rotating black hole’s disc reflecting in Matthew McConaughey’s helmet: now that’s not the sort of shot you plan.”

Now those projection screens are interactive. Franklin explains: “Say I’m looking down a big corridor. As I move the camera across the screen, instead of it flattening off and giving away the fact that it’s actually just a scenic backing, the corridor moves with the correct perspective, creating the illusion of a huge volume of space beyond the screen itself.“

Effects can be added to a shot in real-time, and in full view of cast and crew. More to the point, what the director sees through their viewfinder is what the audience gets. This encourages the sort of disciplined and creative filmmaking Melies and Chaplin would recognise, and spells an end to the deplorable industry habit of kicking important creative decisions into the long grass of post-production.

What’s taking shape here isn’t a “good enough for TV” reality. This is a “good enough to reveal truths” reality. (Gargantua, the spinning black hole at Interstellar’s climax, was calculated and rendered so meticulously, it ended up in a paper for the journal Classical and Quantum Gravity.) In some settings, digital facsimile is becoming, literally, a replacement reality.

In 2012 the EU High Representative Baroness Ashton gave a physical facsimile of the burial chamber of Tutankhamun to the people of Egypt. The digital studio responsible for its creation, Factum Foundation, has been working in the Valley of the Kings since 2001, creating ever-more faithful copies of places that were never meant to be visited. They also print paintings (by Velasquez, by Murillo, by Raphael…) that are indistinguishable from the originals.

From the perspective of this burgeoning replacement reality, much that is currently considered radical in the art world appears no more than a frantic shoring-up of old ideas and exhausted values. A couple of days ago Damien Hirst launched The Currency, a physical set of dot paintings the digitally tokenised images of which can be purchased, traded, and exchanged for the real paintings.

Eventually the purchaser has to choose whether to retain the token, or trade it in for the physical picture. They can’t own both. This, says Hirst, is supposed to challenge the concept of value through money and art. Every participant is confronted with their perception of value, and how it influences their decision.

But hang on: doesn’t money already do this? Isn’t this what money actually is?

It can be no accident that non-fungible tokens (NFTs), which make bits of the internet ownable, have emerged even as the same digital technologies are actually erasing the value of provenance in the real world. There is nothing sillier, or more dated looking, than the Neues Museum’s scan of its iconic bust of Nefertiti, released free to the public after a complex three-year legal battle. It comes complete with a copyright license in the bottom of the bust itself — a copyright claim to the scan of a 3,000-year-old sculpture created 3,000 miles away.

Digital technologies will not destroy art, but they will erode and ultimately extinguish the value of an artwork’s physical provenance. Once facsimiles become indistinguishable from originals, then originals will be considered mere “first editions”.

Of course literature has thrived for many centuries in such an environment; why should the same environment damage art? That would happen only if art had somehow already been reduced to a mere vehicle for financial speculation. As if!

 

Dispersing the crowds

Considering the fate of museums and galleries under covid laockdown for New Scientist, 3 February 2021

In November 2020, the International Council of Museums estimated that 6.1 per cent of museums globally were resigned to permanent closure due to the pandemic. The figure was welcomed with enthusiasm: in May, it had reported nearly 13 per cent faced demise.

Something is changing for the better. This isn’t a story about how galleries and museums have used technology to save themselves during lockdowns (many didn’t try; many couldn’t afford to try; many tried and failed). But it is a story of how they weathered lockdowns and ongoing restrictions by using tech to future-proof themselves.

One key tool turned out to be virtual tours. Before 2020, they were under-resourced novelties; quickly, they became one of the few ways for galleries and museums to engage with the public. The best is arguably one through the Tomb of Pharaoh Ramses VI, by the Egyptian Tourism Authority and Cairo-based studio VRTEEK.

And while interfaces remain clunky, they improved throughout the year, as exhibition-goers can see in the 360-degree virtual tour created by the Museum of Fine Arts Ghent in Belgium to draw people through its otherwise-mothballed Van Eyck exhibition.

The past year has also forced the hands of curators, pushing them into uncharted territory where the distinctions between the real and the virtual become progressively more ambiguous.

With uncanny timing, the V&A in London had chosen Lewis Carroll’s Alice books for its 2020 summer show. Forced into the virtual realm by covid-19 restrictions, the V&A, working with HTC Vive Arts, created a VR game based in Wonderland, where people can follow their own White Rabbit, solve the caterpillar’s mind-bending riddles, visit the Queen of Hearts’ croquet garden and more. Curious Alice is available through Viveport; the real-world show is slated to open on 27 March.

Will museums grow their online experiences into commercial offerings? Almost all such tours are free at the moment, or are used to build community. If this format is really going to make an impact, it will probably have to develop a consolidated subscription service – a sort of arts Netflix or Spotify.

What the price point should be is anyone’s guess. It doesn’t help for institutions to muddy the waters by calling their video tours virtual tours.

But the advantages are obvious. The crowded conditions in galleries and museums have been miserable for years – witness the Mona Lisa, imprisoned behind bulletproof glass under low-level diffuse lighting and protected by barricades. Art isn’t “available” in any real sense when you can only spend 10 seconds with a piece. I can’t be alone in having staggered out of some exhibitions with no clear idea of what I had seen or why. Imagine if that was your first experience of fine art.

Why do we go to museums and galleries expecting to see originals? The Victorians didn’t. They knew the value of copies and reproductions. In the US in particular, museums lacked “real” antiquities, and plaster casts were highly valued. The casts aren’t indistinguishable from the original, but what if we produced copies that were exact in information as well as appearance? As British art critic Jonathan Jones says: “This is not a new age of fakery. It’s a new era of knowledge.”

With lidar, photogrammetry and new printing techniques, great statues, frescoes and chapels can be recreated anywhere. This promises to spread the crowds and give local museums and galleries a new lease of life. At last, they can become places where we think about art – not merely gawp at it.

Langlands & Bell move the furniture

Talking with Ben Langlands and Nikki Bell for the Financial Times, 29 September 2020

Ben Langlands and Nikki Bell have spent forty years making architecturally inspired art: meticulous cardboard models of real buildings hung in hardwood frames, or inset under glass in the seats of sculptural kitchen chairs you cannot sit on.

What’s at the centre of their work? What have they done with its heart?

Langlands and Bell split their time between Whitechapel and their studio in Kent. In person, they are warm and funny and garrulous. But just as their house (called Untitled) is designed to melt unobtrusively into the landscape, so their art has a tendency to vanish into the warp and weft of things. This is literally true of their show “Degrees of Truth”, which opened at Sir John Soane’s Museum in London just days before the Coronavirus lockdown. On 1 October, when the museum reopens, visitors may need a minute or two to adjust to the fiendish way the partnership’s new and old work has hidden itself among the eighteenth-century architect’s eclectic collection of stones, carvings, statues, curios and models.

“Soane liked models; we like models,” Langlands says. “But while Soane used models for understanding how to build something, we use them for understanding how and why it was built.”

The couple’s investigative process invites comparisons with Trevor Paglen (who tilts at surveillance systems) and Forensic Architecture (who rebuild erased moments in history using point-clouds and plaster). But while they are known for tackling big political themes, their preferred environment is the domestic interior. They renovated houses for money when they were students; days before the lockdown they invited me to their place in Kent, a self-built house-studio-gallery they designed themselves. No-one at the time imagined they would soon find themselves marooned there. (“We got a lot done over the summer”, Ben Langlands admits over the phone, “but the virtual and the real are different places. It’s unreasonable to expect that you’ll get real-world inspiration off the internet.”)

They say they are neither architects, nor designers, and in the same breath they say their work is about the relationships people establish with each other through their buildings and their furniture. The first work they ever collaborated on, as students at Middlesex Polytechnic (the former Hornsey College of Art) was a pair of kitchens: through a window let in to the wall of the worn, grimy old one, you got a fleeting glimpse the brand-spanking new one. They believe (rightly) that it is possible to read a world of social relations into, say, the position of chairs around a table.

Not all architectures are material, and this may be why the point of their work sometimes vanishes from sight. Globe Table (2020) is a piece made for the show which now languishes behind the locked doors of Sir John Soane’s Museum. It is a giant white marble laced with black lines, marking the world’s major air routes. Nearby, in a case that once held a pistol that supposedly belonged to Napoleon, Virtual World, Medal of Dishonour (2008) is a disc whose enamel rings combine three categories of codes; codes for airports, like LHR or JFK or LAX, then for NGOs involved in reconstruction or disaster relief like UN WHO or USAID; finally the acronyms of geopolitical players: IRA, CIA, ETA, ISIS.

A bit of a jump, that, from seating arrangements to airline schedules and security agencies. But this is the territory Langlands and Bell have staked out. Since 1990 they have been exploring the space where physical structures, images, logos and acronyms bleed into each other. Their next show, opening at CCA Kitakyushu in Japan on 16 November, is a museological skit built around the signatures of curators they’ve run into over a 45 year career.

They began from a place of high seriousness. Logoworks (1990), modelled the new corporate offices rising in Frankfurt, and they garnered headlines again when they tackled some iconic West Coast companies in Internet Giants: Masters of the Universe (2018) showing how, in Bell’s words, “companies subliminally bring their identity to the forms of their buildings”.

And the seriousness becomes positively deadly in an upcoming show at Gallery 1957 in Accra. “The Past is Never Dead…” brings to the slave forts of the Ghanaian ‘Gold Coast’ the same forensic eye the artists applied to the “campuses” of Apple and Facebook. There’s the same consistency in typology on show, only this time the buildings’ spiny, angular forms are driven, not by brand marketing, but by the need to defend against sea-borne cannon attack.

“I think architecture is changing,” says Nikki Bell, “in that it’s becoming more object-based.” Algorithmic design encourages planners and architects to treat buildings like scaleless objects, like those vector graphics that expand endlessly without loss of resolution. Some of the most ambitious buildings of our age are, architecturally speaking, simply scaled-up logos.

And, says Ben Langlands, there is another, even more powerful force eroding architecture. “Up until now the most profound influences exercised on us culturally have come from architecture,” he says, “so tangible, so enduring, so powerful, so massive, so complicated and expensive, that it has huge effects on us that last for centuries. ” Today, however, those relations are being shaped much more powerfully by social media, “a new kind of architecture which is much more stealthy and hidden.”

Langlands and Bell are committed to studying the world in aesthetic terms, and everything else they might feel or think or say follows from their way of seeing. Once I stop ransacking their work for ideological Easter eggs (are they dystopian? are they anti-capitalist? are they neo-Luddite?), I begin to see what they’re up to. They are looking for beauty, sincere in their conviction that through beauty they will find truth.

Between Logoworks and Internet Giants, and in the shadow of the first Gulf War, the artists began making reliefs, “two-and-a-half-dimensional” wall sculptures that reflected how reconnaissance planes at high altitude saw structures tens of thousands of feet below. “You would get these collapsed views at very compressed angles which now appear very typical of that time.”

Marseille, Cité Radieuse (2001), for example, presents a distorted view of the facade of Le Corbusier’s Unité d’Habitation in Marseille. It’s a model made from a photograph taken at an angle, a meticulous white sculpture that depends almost entirely on the way it’s lit to be at all comprehensible.

Developments in photography and in architecture since the 1990s have replaced those evocative compressed-angle images with drone footage, and this has in its turn informed the tiresome surveillance typology of umpteen videogames — though not before Langlands and Bell earned a Turner Prize nomination for The House of Osama bin Laden. which included a virtual render, explorable via joystick, of a lake-side house bin Laden once occupied.

Forty years into their career, Langlands and Bell continue to chip away at the world with tools that Sir John Soane would have recognised: a sense of form, light, movement and beauty. They say they like new things to investigate. They say they are always travelling, always exploring. But what else can an architecturally minded artist do, once the very idea of architecture has begun to dissolve?

“Langlands & Bell: Degrees of Truth” at Sir John Soane’s Museum, 13 Lincoln’s Inn Fields, London WC2A 3BP, from 1 October 2020.

“Curators Signatures” at CCA Kitakyushu from 16 November 2020 to 22 January 2021.

“’The Past is Never Dead…’ the architecture of the Slave Castles of the Ghanaian Gold Coast”, at Gallery 1957, Accra, Ghana, will open in 2021.

Pretty tragic

Is Michel Comte’s past celebrity a burden? “You carry it on your fucking back,” he says. “It took ten years for people to notice I was visiting Africa for months at a time. It took twenty years before people starting listening to what I’ve been saying since my first gallery show.”

A conversation for the Financial Times, 20 May 2020.

Can you use a bottle opener?

Visiting the Baltic in Gateshead for Animalesque: Art across species and beings.
For New Scientist, 15 January 2020.

EXHIBITIONS about our relationship with the environment tend to be bombastic. Either they preach doom and destruction, or they reckon our children will soon be living lives of plenty on artificial atolls.

Animalesque at the Baltic Centre for Contemporary Art in Gateshead, UK, knows better than this. In an international selection of art, sculpture and film, curator Filipa Ramos points out how little we know about other species, and how much we might still learn. With this humility comes hope that we can reform our relations with Earth.

Research has a major role to play, but it can only go so far. One unassuming TV monitor is screening a video from Tupilakosaurus, a long-running project by Danish-Greenlandic artist Pia Arke. It is a telling but not unsympathetic satirical film, in which examinations of a fossil dinosaur throw up folk tales, mangled histories and surreal mountains of paperwork as researchers try to represent and classify the Arctic’s life and history.

Often, we find out about other species only as we are evicting and replacing them. This happened to the Malayan tiger, which now numbers just some 300 wild cats in the Malay Peninsula. 2 or 3 Tigers (2015) by Singaporean artist Ho Tzu Nyen is a 19-minute, two- screen video, made using CGI and some very dodgy operatic singing, about the were-tigers of Malayan folklore. As ancestors, companions, competitors, protectors, destroyers and gods, tigers were central to the indigenous culture. Western settlers couldn’t find any there, however, until one sprang out of the forest in 1835 and attacked a hapless surveyor’s theodolite.

Our most stable cross-species relationships are with domesticated animals, even if they are sometimes discomforting or guilt-ridden affairs. In French artist Pierre Huyghe’s Untitled (Human Mask) (2014), a macaque explores an abandoned restaurant in Fukushima, Japan, an area gutted by the 2011 tsunami. Identifying the species of our protagonist takes a while. You would be forgiven for thinking you were watching a girl, because the macaque is wearing a wig and an eerily beautiful mask (pictured above).

The uncanny collision of categories (girl and pet, puppet and creature) only becomes more dizzying when you discover that Huyghe recruited his “star” from a Tokyo restaurant where the macaque spent many apparently happy hours working as a waiter.

It is a film of great pathos, more moving and less disturbing than this bald description suggests. It speaks to our difficulty understanding other animals, steeped as we are in human concerns.

The difficulty is real, can research help us? Degreecoordinates, Shared traits of the Hominini (humans, bonobos and chimpanzees) (2015) attempts it. For this, UK artist Marcus Coates worked with primatologist Volker Sommer to list questions relevant to all three: do you resolve conflicts using sex? Can you use a bottle opener? Do you kiss? Are you preoccupied with hierarchy and status?

Human answers vary, but so do those gleaned from studying individual chimps and bonobos. The differences between individuals of each of the three species far exceed those across species. Animalesque celebrates what we share – and what we can learn.

 

Breakfast with Ryoji Ikeda

Meeting the artist Ryoji Ikeda for the Financial Times, 29 November 2019

At breakfast in a Paris café, the artist and composer Ryoji Ikeda looks ageless in a soft black cap and impenetrably dark glasses, dressed all in black so as to resemble the avatar from an indie video game.

His work too is severe, the spectrum reduced to grayscale, light to pixels, sound to spikes. Yet Ikeda is no minimalist: he is interested in the complexity that explodes the moment you reduce things to their underlying mathematics.

An artist in light, video, sound and haptics (his works often tremble beneath your feet), Ikeda is out to make you dizzy, to overload your senses, to convey, in the most visceral manner (through beats, high volumes, bright lights and image-blizzards) the blooming, buzzing confusion of the world. “I like playing around with the thresholds of perception,” he says. “If it’s too safe, it’s boring. But you have to know what you’re doing. You can hurt people.”

Ikeda’s stringent approach to his work began in the deafening underground clubs of Kyoto. There, in the mid-1990s, he made throbbing sonic experiences with Dumb Type, a coalition of technologically adept experimental artists. And he can still be this immediate when he wants to be: visitors to the main pavilion at this year’s Venice Biennale found themselves squeezed through “Spectra III” (first assembled in 2008), a white corridor so evenly and brightly lit your eyes rejected what they saw, leaving you groping your way out as if in total darkness.

These days, though, he is better known for installations that go straight for the cerebral and mathematical. His ongoing “data-verse” project consists of three massively complex computer animations. The first part, “data-verse 1”, is based on static data from CERN, Nasa, the Human Genome Project and other open sources. “data-verse” contains animations, tables, graphs, matrices, 3D models, Lidar projections, maps. But what is being depicted here: something very small, or very big? There’s no way to tell. The data have peeled away from the things they represent and are dancing their own pixelated dance. Numbers have become rivers. At last the viewer’s mind surrenders to the flow and rhythm of this frenetic 12-minute piece.

It would be polite to say that “data-verse” is beautiful — but it isn’t. Rather, it is sublime, evoking a world stripped back to its mathematical bones. “If it’s beautiful, you can handle it; the sublime, you cannot,” Ikeda says. “If you stand in some great whited-out landscape in Lapland, the Sahara or the Alps, you feel something like fear. You’re trying to draw inform­ation from the world, but it’s something that your brain cannot handle.”

Similarly, the symmetrical, self-similar “data-verse” is an artwork that your mind struggles to navigate, tugging at every locked door in an attempt to regain purchase on the world.

“You try to understand, but you give up — and then it’s nice. Because now you are experiencing this piece the same way you listen to music,” Ikeda says. “It’s simply a manipulation of numbers and relationships, like a musical composition. It’s very different from the sort of visual art where you’re looking through the surface of the painting or the sculpture to see what it represents.”

When we meet, Ikeda is on his way to Tokyo Midtown, and the unveiling of “data-verse 2” (this one based on dynamic data “like the weather, or stock exchanges”). The venue is Beyond Watchmaking, an exhibition arranged by his patron, the eccentric family-run Swiss watchmaker Audemars Piguet. The third part of data-verse is due to be unveiled next year.

It is a vastly ambitious project but Ikeda has always tended towards the expansive. He pulls out of his suitcase an enormously heavy encyclopedia of sonic visualisations. “I wanted you to see this,” he says with a touching pride, leafing through page after page of meticulously documented oscilloscoped forms. Encyclopedia Cyclo.id was compiled with his friend Carsten Nicolai, the German multimedia artist, in 1999. Each figure here represents a particular sound. The more complex figures resemble watch faces. “It’s for designers, really,” Ikeda shrugs, shutting the book, “and architects.”

And the point of this? That lawful, timeless mathematics underpins the world and all our activities within it.

Ikeda spends 10 months out of every 12 travelling: “I really work in the airport or the kitchen. I don’t like the studio.” Months spent working out problems on paper and in his head are interspersed with intense, collaborative “cooking sessions” with a coterie of exceptional coders — creative sessions in which all previous assumptions are there to be challenged.

However, “data-verse” is likely to be Ikeda’s last intensely technological artwork. At the moment he is inclining more towards music and has been arranging some late compositions by John Cage in a purely acoustic project. As comfortable as he is around microphones, amps and computers, Ikeda isn’t particularly affiliated to machines.

“For a long time, I was put in the media-art category,” he says, “and I was so uncomfortable, because so much of that work is toylike, no depth to it at all. I’m absolutely not like this.”

Ikeda’s art, built not from things but from quantities and patterns, has afforded him much freedom. But he is acutely aware that others have more freedom still: “Mathematicians,” he sighs, “they don’t care about a thing. They don’t even care about time. It’s very interesting.”

668 televisions (some of them broken)

Visiting the Nam June Paik exhibition at Tate Modern for New Scientist, 27 November 2019

A short drive out of Washington DC, in an anonymous industrial unit, there is an enormous storage space crammed to the brim with broken television sets, and rolling stack shelving piled with typewriters, sewing machines and crudely carved coyotes.

This is the archive of the estate of Nam June Paik, the man who predicted the internet, the Web, YouTube, MOOCs, and most other icons of the current information age; an artist who spent much of his time engineering, dismantling, reusing, swapping out components, replacing old technology with better technology, delivering what he could of his vision with the components available to him. Cathode ray tube televisions. Neon. Copper. FORTRAN punch cards. And a video synthesizer, designed with the Tokyo artist-engineer Shuya Abe in 1969. The signature psychedelic video effects of Top of the Pops and MTV began life here.

Paik was born in Seoul in 1932, during the Japanese occupation of Korea, and educated in Germany, where he met the composers Karl-Heinz Stockhausen and John Cage. A fascinating retrospective show currently at London’s Tate Modern celebrates his involvement with that loose confederacy of artist-anarchists known as Fluxus. (Yoko Ono was a patron. David Bowie and Laurie Anderson were hangers-on.)

Beneath Paik’s celebrated, and celebrity-stuffed concerts, openings and “happenings” — there’s what amounts — in the absence of Paik’s controlling intelligence (he died in 2006) — to a pile of junk. 668 televisions, some of them broken. A black box the size of a double refrigerator, containing the hardware to drive one of Paik’s massive “matrices”, Megatron/Matrix, an eight-channel, 215-screen video wall, in pieces now, a nightmare to catalogue, never mind reconstruct, stored in innumerable tea chests.

The trick for Saisha Grayson, the Smithsonian American Art Museum’s curator of time-based media, and Lynn Putney its associate registrar, is to distinguish the raw material of Paik’s work from the work itself. Then curators like Tate Modern’s Sook Kyung Lee must interpret that work for a new generation, using new technology. Because let’s face it: in the end, more or less everything Paik used to make his art will end up in the bin. Consumer electronics aren’t like a painter’s pigments, which can be analysed and copied, or like a sculptor’s marble, which can, at a pinch, be repaired.

“Through Paik’s estate we are getting advice and guidance about what the artist really intended to achieve,” Lee explains, “and then we are simulating those things with new technology.”

Paik’s video walls — the works by which he’s best remembered, are monstrously heavy and absurdly delicate. But the Tate has been able to recreate Paik’s Sistine Chapel for this show. Video projectors to fill a room with a blizzard of cultural and pop-cultural imagery from around the world — a visual melting pot reflective of Paik’s vision of a technological utopia, in which “telecommunication will become our springboard for new and surprising human endeavors.” The projectors are new but the feel of this recreated piece is not so very different to the 1994 original.

To stand here, bombarded by Bowie and Nixon and Mongolian throat singers and all the other flitting, flickering icons of Paik’s madcap future, is to remember all our hopes for the information age: “Video-telephones, fax machines, interactive two-way television… and many other variations of this kind of technology are going to turn the television set into an «expanded-media» telephone system with thousands of novel uses,” Paik enthused in 1974, “not only to serve our daily needs, but to enrich the quality of life itself.”

Nam June Paik: Doing away with structure once and for all

Visiting Nam June Paik at Tate Modern for the Financial Times, 24 October 2019

In 1963 one of the more notorious members of Darmstadt’s new music community, Nam June Paik, stuck around fifty strips of audio tape to the wall of the Galerie Parnass in Wuppertal in Germany.

“I wanted to let the audience… act and play by itself,” he wrote, “so I have resigned the performance of music… I made various kinds of musical instruments… to expose them in a room so that the congregation may play them as they please.”

Exhausted and alienated by the difficult musics coming out of Darmstadt — Pierre Boulez, Karlheinz Stockhausen, serialism and all the rest — visitors lapped up Paik’s free-wheeling alternative. You’d go up to the wall and rub the playback head of a dismantled tape recorder along the strips, back and forth, hunting for sounds, scratches, white noise, and hey presto! you almost became a composer.

“You have to be a lot rougher with this than you think,” a gallery worker explained, showing me the Tate’s recreated Random Access. “Really scrape.”

So I scraped. And I still couldn’t get much of a sound out of the wall-mounted speakers, and now the gallery wall is covered in dirty brown ferrous oxide streaks.

The original wasn’t very effective, either. The point was that Paik was giving you permission to play, to experiment. The Swiss artist and career eccentric Josef Beuys took Paik at his word and destroyed one of the the pianos in Paik’s first solo show with an axe. And Paik dug it; they became lifelong friends.

How do you represent an artist whose chosen medium is the audience? Who spends his time chivvying it into life by gestures, situations, shocks, pornography? How do you preserve Zen of Head (1962), in which Paik dipped his head in black ink and used it to draw a line on a length of paper? How do you honour his nearly thirty-year collaboration with the cellist and performance artist Charlotte Moorman, when Variations on a Theme by Saint-Saens (1969) involves her climbing up a ladder and vanishing into a water-filled oil drum?

The many representative works gathered by the Tate can only go so far to represent Paik’s whole practice. TV Buddha (1974) is a statuette of a seated Buddha, gazing at its own televised image. Three Eggs (1975-82) — one real, one nested in an empty television, and the third a televised image of the first egg — goes beyond mere solipsism to suggest something more complex. There are robots made from TV sets here, lines of code from early experiments at Bell Labs in New Jersey, abd TV bras and TV spectacles that seem to have fallen out of one of the calmer moments of the Japanese cyberpunk horror flick Tetsuo: The Iron Man. Newcomers would be left hopelessly at sea were it not that the Tate has also assembled a huge amount of documentation, and arranged it in a fashion that is not just informative: it’s revelatory.

Programmes. Posters. Photographs. Snatches of 8mm. Mostly they record events in tiny rooms, the visitors all crammed together, everyone laughing, having a good time. Wall by wall, case by case, we begin to understand what we missed.

Paik was a collector, a collaborator, an impresario. He urged others to enact the strangest dreams. In New York, in 1964, a topless Charlotte Moorman saws away at her cello, and Alison Knowles sheds her panties and shoves them down the throat of the least talented art critic in the room.

But Paik had other dreams, too, which which for years he kept strictly to himself. As early as 1961 he had given up studying art and was avidly reading Popular Mechanics. In Tokyo, with the engineer Shuya Abe, he co-invented the Paik-Abe Video Synthesizer. This added single-colour channels to broadcast images in real time, distorted, colorised, and superimposed multiple images, and was in essence the technology that would soon give Top of the Pops and the MTV music channel their visual signature.

Paik’s use of TV as a medium is now what everyone most remembers about him, thanks mostly to his monumental “matrices”: sculptural video collages assembled using steel gantries and neon tubing and multiple cathode-ray televisions. There’s a late example here called Internet Dream (1994), and nearby, a recreation of the video installation Sistine Chapel, which in 1993 graced the pavilion of a newly-unified Germany at the Venice Biennale. Thrown across walls and ceiling by TV projectors, disembodied David Bowies and Janis Joplins, Lou Reeds and Ryuichi Sakamotos jostle for space with parties of Gobi desert Mongolians. It’s intoxicating. Dated. Kitsch. It’s the fruit both in flower and in rot.

“Thanks to Paik,” he wrote about himself (never a good sign) ” we discover that our entire world can become sound — or rather that it *is* sound… he does away with structure once and for all.”

And, oh dear, just look where that liquefaction has led. By giving us permission to create, Paik stripped away the structures that let us receive, appreciate, and judge. His mentor John Cage did much the same for music. And around Cage and Paik, Moorman and Beuys swirled a loose, revolutionary band of brothers and sisters who, under the banner of a movement called Fluxus, abandoned the commodified single art object and sought to create democratic art; an art of the everyday.

The idea that audiences also knew something about art filled these self-appointed shamans with impatience. The audience’s ideas were third-hand, third-rate, bourgeois prisons from which they might yet be liberated.

Liberated into what, though? Into boredom? Into consumption? All you can do with this work is participate in it. Swallow it. Go see In Real Life, Olafur Eliasson’s collection of kid-friendly novelties, if you want to see where this attitude leads. It runs next door till January 5.

As I left Paik’s show, I paused by a wall-mounted TV, where pianist Manon-Liu Winter plays her own composition on Paik’s prepared piano (now too fragile to travel). The one with the barbed wire, whose keyboard once triggered sirens, heaters, ventilators and tape recorders.

Now, though, it’s just a ruined piano. Winter picks her way across its atrocious keyboard like Jack Skellington, trying to discover the secret of Christmas by measuring the presents under the tree with a tape measure. This is indeed a revelatory exhibition — but you may come away liking Paik less.

“Intelligence is the wrong metaphor for what we’ve built”

Travelling From Apple to Anomaly, Trevor Paglen’s installation at the Barbican’s Curve gallery in London, for New Scientist, 9 October 2019

A COUPLE of days before the opening of Trevor Paglen’s latest photographic installation, From “Apple” to “Anomaly”, a related project by the artist found itself splashed all over the papers.

ImageNet Roulette is an online collaboration with artificial intelligence researcher Kate Crawford at New York University. The website invites you to provide an image of your face. An algorithm will then compare your face against a database called ImageNet and assign you to one or two of its 21,000 categories.

ImageNet has become one of the most influential visual data sets in the fields of deep learning and AI. Its creators at Stanford, Princeton and other US universities harvested more than 14 million photographs from photo upload sites and other internet sources, then had them manually categorised by some 25,000 workers on Amazon’s crowdsourcing labour site Mechanical Turk. ImageNet is widely used as a training data set for image-based AI systems and is the secret sauce within many key applications, from phone filters to medical imaging, biometrics and autonomous cars.

According to ImageNet Roulette, I look like a “political scientist” and a “historian”. Both descriptions are sort-of-accurate and highly flattering. I was impressed. Mind you, I’m a white man. We are all over the internet, and the neural net had plenty of “my sort” to go on.

Spare a thought for Guardian journalist Julia Carrie Wong, however. According to ImageNet Roulette she was a “gook” and a “slant-eye”. In its attempt to identify Wong’s “sort”, ImageNet Roulette had innocently turned up some racist labels.

From “Apple” to “Anomaly” also takes ImageNet to task. Paglen took a selection of 35,000 photos from ImageNet’s archive, printed them out and stuck them to the wall of the Curve gallery at the Barbican in London in a 50-metre-long collage.

The entry point is images labelled “apple” – a category that, unsurprisingly, yields mostly pictures of apples – but the piece then works through increasingly abstract and controversial categories such as “sister” and “racist”. (Among the “racists” are Roger Moore and Barack Obama; my guess is that being over-represented in a data set carries its own set of risks.) Paglen explains: “We can all look at an apple and call it by its name. An apple is an apple. But what about a noun like ‘sister’, which is a relational concept? What might seem like a simple idea – categorising objects or naming pictures – quickly becomes a process of judgement.”

The final category in the show is “anomaly”. There is, of course, no such thing as an anomaly in nature. Anomalies are simply things that don’t conform to the classification systems we set up.

Halfway along the vast, gallery-spanning collage of photographs, the slew of predominantly natural and environmental images peters out, replaced by human faces. Discrete labels here and there indicate which of ImageNet’s categories are being illustrated. At one point of transition, the group labelled “bottom feeder” consists entirely of headshots of media figures – there isn’t one aquatic creature in evidence.

Scanning From “Apple” to “Anomaly” gives gallery-goers many such unexpected, disconcerting insights into the way language parcels up the world. Sometimes, these threaten to undermine the piece itself. Passing seamlessly from “android” to “minibar”, one might suppose that we are passing from category to category according to the logic of a visual algorithm. After all, a metal man and a minibar are not so dissimilar. At other times – crossing from “coffee” to “poultry”, for example – the division between categories is sharp, leaving me unsure how we moved from one to another, and whose decision it was. Was some algorithm making an obscure connection between hens and beans?

Well, no: the categories were chosen and arranged by Paglen. Only the choice of images within each category was made by a trained neural network.

This set me wondering whether the ImageNet data set wasn’t simply being used as a foil for Paglen’s sense of mischief. Why else would a cheerleader dominate the “saboteur” category? And do all “divorce lawyers” really wear red ties?

This is a problem for art built around artificial intelligence: it can be hard to tell where the algorithm ends and the artist begins. Mind you, you could say the same about the entire AI field. “A lot of the ideology around AI, and what people imagine it can do, has to do with that simple word ‘intelligence’,” says Paglen, a US artist now based in Berlin, whose interest in computer vision and surveillance culture sprung from his academic career as a geographer. “Intelligence is the wrong metaphor for what we’ve built, but it’s one we’ve inherited from the 1960s.”

Paglen fears the way the word intelligence implies some kind of superhuman agency and infallibility to what are in essence giant statistical engines. “This is terribly dangerous,” he says, “and also very convenient for people trying to raise money to build all sorts of shoddy, ill-advised applications with it.”

Asked what concerns him more, intelligent machines or the people who use them, Paglen answers: “I worry about the people who make money from them. Artificial intelligence is not about making computers smart. It’s about extracting value from data, from images, from patterns of life. The point is not seeing. The point is to make money or to amplify power.”

It is a point by no means lost on a creator of ImageNet itself, Fei-Fei Li at Stanford University in California, who, when I spoke to Paglen, was in London to celebrate ImageNet’s 10th birthday at the Photographers’ Gallery. Far from being the face of predatory surveillance capitalism, Li leads efforts to correct the malevolent biases lurking in her creation. Wong, incidentally, won’t get that racist slur again, following ImageNet’s announcement that it was removing more than half of the 1.2 million pictures of people in its collection.

Paglen is sympathetic to the challenge Li faces. “We’re not normally aware of the very narrow parameters that are built into computer vision and artificial intelligence systems,” he says. His job as artist-cum-investigative reporter is, he says, to help reveal the failures and biases and forms of politics built into such systems.

Some might feel that such work feeds an easy and unexamined public paranoia. Peter Skomoroch, former principal data scientist at LinkedIn, thinks so. He calls ImageNet Roulette junk science, and wrote on Twitter: “Intentionally building a broken demo that gives bad results for shock value reminds me of Edison’s war of the currents.”

Paglen believes, on the contrary, that we have a long way to go before we are paranoid enough about the world we are creating.

Fifty years ago it was very difficult for marketing companies to get information about what kind of television shows you watched, what kinds of drinking habits you might have or how you drove your car. Now giant companies are trying to extract value from that information. “I think,” says Paglen, “that we’re going through something akin to England and Wales’s Inclosure Acts, when what had been de facto public spaces were fenced off by the state and by capital.”