Governments fail to capitalise on swaths of open data

London was the obvious choice for Ian McGill to start his business, Spend Network. The subscription service, launched in 2010, collects public sector spending records of more than 300 bodies. Subscribers and the company’s own researchers use this data to analyse the activities of government suppliers to help secure better deals in procurement.

The UK pioneered the movement to make such data more freely available. In 2010, in an effort to boost public trust in politics, the then prime minister David Cameron wrote to government departments asking them to open up their data to “enable the public to hold politicians and public bodies to account”.

Yet Mr McGill struggles to hide his frustration at the data’s patchy quality. Public bodies in the UK are required to publish details of procurement contracts awarded but, he adds “we have seen only an handful of those contracts”. Data is often incomplete, not standardised or has other serious quality flaws, says McGill.

Behind the push for open data is a desire to make governments more transparent, accountable and efficient — but also to allow businesses to create products and services that spark economic development. The global annual opportunity cost of failing to do this effectively is about $5tn, according to one estimate from McKinsey, the consultancy.

The UK is not the only country falling short, says the Open Data Barometer, which monitors the status of government data across the world. Among the 30 leading governments — those that have championed the open data movement and have made progress over five years — “less than a quarter of the data with the biggest potential for social and economic impact” is truly open. 

This goal of transparency, it seems, has not proved sufficient for “creating value” — the movement’s latest focus.

In 2015, nearly a decade after advocates first discussed the principles of open government data, 62 countries adopted the six Open Data Charter principles — which called for data to be open by default, usable and comparable.

Yet Carlos Iglesias, senior research manager at the Web Foundation, which campaigns for the provision of internet access as a basic right and runs the ODB, says global government data “is usually incomplete, out of date, of low quality and fragmented”.

While most databases are easy to find and free to use, only half are available in a machine-readable format, the barometer’s 2018 global report found. Of those, only one in four can be downloaded in bulk to be reused. 

The new mantra among supporters of the initiative is “publish with a purpose”, says Barbara Ubaldi, economist at the OECD, the Paris-based club of mostly rich nations.

Private companies have a role to play here, too. Theresa May, the UK prime minister, proposed last month that certain companies disclose the pay gap between staff of different ethnicities — in a similar vein to the requirement to report gender pay, which came into effect in earlier this year.

Ania Calderon, executive director at the Open Data Charter, says: “Sometimes the most valuable data is held by private companies and where there is the potential to create public value there is a role in government to play in legislating how that data should be made open.”

The use of open data has already bore fruit for some countries. In 2015, Japan’s ministry of land, infrastructure and transport set up an open data site aimed at disabled and elderly people. The 7,000 data points published are downloadable and the service can be used to generate a map that shows which passenger terminals on train, bus and ferry networksprovide barrier-free access.

In the US, The Climate Corporation, a digital agriculture company, combined 30 years of weather data and 60 years of crop yield data to help farmers increase their productivity. And in the UK, subscription service Land Insight merges different sources of land data to help individuals and developers compare property information, forecast selling prices, contact land owners and track planning applications.

The European Commission estimated the market value of open data in the EU in 2016 at close to €200bn — with a projection of up to €285bn by 2020.

Various monitors are trying to assess impact in different ways. The OECD tracks government efforts in supporting the reuse of data among individuals, NGOs and private companies, explains Ms Ubaldi. Both the ODB and the European Commission trackers, meanwhile, measure apps and services created using government data, and their political, social and economic value.

New seams of data are helping to create new businesses, with some countries performing better than others.

South Korea’s data policy has helped the nation create an active business ecosystem. Ms Calderon says it is “focused exclusively on companies developing services . . . [using] data that they produce”.

Open Data 500, an international network of organisations that studies the use and impact of open data, reveals that private companies in South Korea are using government agency data, with technology, advertising and business services among the biggest users.

It shows, for example, that Archidraw, a four-year-old Seoul-based company that provides 3D visualisation tools for interior design and property remodelling, has used mapping data from the Ministry of Land, Infrastructure and Transport.

Germany lags behind on most assessments as the move towards open data was carried out at state level with little national co-ordination, but it has pushed ahead with legislation. The country’s open data law came into effect last summer, but the requirement to publish data within two years applies only to public authorities subject to the federal government, and excludes Germany’s regional authorities.

Such legislation is something the UK lacks, says Jeni Tennison, chief executive of the Open Data Institute, co-founded by inventor of the internet Sir Tim Berners-Lee to promote the use of open data worldwide. “There is no legislation that forces a public sector department to update their open data or to produce it to a certain quality.”

Mr McGill agrees: “The regulation is there but not the checks. It’s like having a speed limit, but no police on the road and no monitoring.” 

Better-quality open data could help unlock an annual $3.2tn-$5.4tn in economic value globally across seven sectors, according to a study by McKinsey, and includes transportation, consumer finance and healthcare.

The UK is among those that could do much more to bridge this gap, Mr McGill believes. “There are large potentials in the UK [from better open government data] that are simply not met,” he says.

With open data’s leading governments treating the initiative as a “side project”, in the words of the ODB, it will be some while yet before its full economic potential is realised.