Big data needs big judgement

There is an experimental platform imagined in the classic 80s film Tron where all forms of research can be carried out at unparalleled speeds. It was called The Grid, and 30 years on, this futuristic machine has become a reality.

Down in a series of vast underground caverns and tunnels beneath France and Switzerland, the Large Hadron Collider is about to switch on again after two years of fine-tuning and upgrades. When it does, it will be operating at twice the energy and producing an enormous, mind-blowing quantity of data. And all of that data will be processed by the Grid.

The Grid

The numbers involved are huge.

When the switch is flicked in the new year the Grid will start to consume 160,000 gigabytes of information every day. That works out at 70 petabytes a year. To give you an idea of size, one petabyte is big enough to store the entire DNA material of everyone in the USA.

But how does the Grid cope with such a large amount of data?

Big data

The Grid is a global network of computers, originating in CERN, that quickly processes and shares the big data generated by the trillions of collisions in the Large Hadron Collider. Like its futuristic forbear in Tron, the Grid is incredibly quick. It is composed of hundreds of thousands of processors working in parallel, and it also comprised of a series of Tiers that pass the data along the line, refining as it goes. This enables the information from the experiment to be shared almost in real time with scientists anywhere around the globe.

Handling the data

But the key to its success is not archiving all the data, but reducing it to a manageable level.

CERN’s data expert Pierre Vande Vyvre has spoken recently about the future for big data processing. He says that the role will change to standard algorithms that can do the heavy lifting first and reduce the data to a more manageable size before it gets close to human analysis.

“The big science workflows are mainly data reduction. Currently just 1% of data from collision events [at the Large Hadron Collider] are selected for analysis. The archiving of raw data is not the standard anymore.”

What can big science teach us about big data?

CERN is well prepared. They can handle their data. Like the best big science projects, they have a clear theory which they are testing against the data, and they have set up clear parameters that can narrow the data. They also have the in-house skills to analyse what they find.

It’s an important framework for dealing with big data that companies and governments could learn from, all of which are setting up their own big data initiatives to help them make predictions about the future.

Big predictions

The Obama administration recently launched a $200 million R&D initiative to ‘improve the government’s ability to extract insights from digital data’, Google has been trying to map flu trends by studying search queries, and Amazon has been using customer data to try and automate our choices for years. All of them are chasing the big data dream.

But it’s when you start to use big data to try and predict future behaviour that you get stuck.

The data is based on social systems which are fraught with unpredictable changes. Unless you understand all future variables, big data for prediction can only tell us more about the past or the present than the future. As Juan Mateos-Garcia argues, “Big data is lurking with biases, mirages and self-fulfilling prophesies. Avoiding them requires access to the right skills and organisation”.

Big data needs big judgement

Operations like the Grid at CERN point the way to better management and analysis of huge data sets. Crucially, they not only have the skills to build the sensors and the computing architecture to deal with the massive flow of data, they also have the judgement to reduce the data and to analyse it afterwards.

Big data is still a judgement call. Despite companies investing heavily in big data processes, most do not have the skills to analyse what is coming in. It’s a problem, and an opportunity, that we’ve set out in our report Model Workers.

The end of theory

Chris Anderson – the idealist TED leader – made the bold call that big data will spell the end of all theory. Like the Master Control Programme in Tron, big data algorithms will work many times faster than a human and will be able to answer our global problems before they arise.

It is misplaced because it is naïve to assume that machines can make decisions about what data to collect and how to interpret the patterns in that data as information for other machines to use. It feels like the techno-utopian belief that brought the world market to its knees in 2008. Here the idea that computers can measure, control and self-stabilise societies was proved dramatically wrong when the complex algorithms that ran financial markets failed to predict the giant debt bubble.

Adam Curtis carefully dismantled this dream of computers running things in his brilliant TV series ‘All Watched Over by Machines of Loving Grace’ (2011). The danger now is that the predictive power of big data is feeding the same fantasy that leads to the same distortion and simplification of the world around us.

Keeping big data human

When it is switched back on the data processed by the Grid will make the Large Hadron Collider the biggest of the Big Science experiments in the world.

But it won’t be the machines peering into the underlying structure of the universe.

That’s up to us.

Image: The Grid’s Tier 0 data center courtesy of CERN. 

This blog was first published on www.nesta.org.uk 

Technology won’t save us, people will

It’s a truth that’s often misunderstood by the tech evangelists, the singularity obsessives, and all the dystopian bandwagoners who think that technology is an alien force that we have to fight to control, otherwise it will eventually control us. This couldn’t be further from the truth.

Technology is an enabling force that allows us to improve the world around us. It is part of our human fabric, not some alien species.

As Kathryn Myronuk said at a recent Wired event: “We can combat global challenges through the lens of tech, not with technology alone.”

We’ve been carrying this enormous human potential in our pockets for years now. And despite a few seismic shifts, what do we use it for most? LOLcats.

Tech heroes

That’s why I wanted to pick out some of the brightest and best talents around the UK and show the great ideas these people have come up with that use digital technology as the enabling force to improve how we live.

I teamed up with my friends around Nesta to help me draw up an initial longlist which I then boiled down.

I think it’s a really strong list that shows not only the refreshing diversity of the tech for good scene, but also some of its current trends:

1. Open data.

This underpins many of the ideas on the list, from creating open platforms for sharing civic data to taking more control of your medical data. It’s also the mechanism for the brilliant citizen science platform Galaxy Zoo co-founded by Chris Lintott.

2. Edutech.

New tools for education is also a big influence on the list – from the groundbreaking work of Eben Upton with Raspberry Pi to the world of online training courses such as Code Club, co-founded by Clare Sutcliffe and Linda Sandvik.
Show me the criteria

How’s it work?

I wanted to keep this first list local to the UK and as tech-lite as possible. That’s why I’ve included leaders of teams, not just developers and coders.

The impact criteria is much harder to judge, but me and my team at Nesta agreed that impact should be based on an idea that is out of Beta, that is currently in use, and that has been distributed across more than one region in the UK.

Here are the criteria in full:

• They need to have produced a digital product or service that benefits society
• Their product or service needs to have demonstrated impact
• They should be a leader of their idea, not just a team member
• They need to be based in the UK (for this version at least)

Take a look at the full Top 10 Tech Heroes for Good here and tell me who I’ve missed out.

This blog was originally published on the Nesta website here: http://www.nesta.org.uk/blog/technology-wont-save-us-people-will