I’m fortunate to have been involved with the Civil Service Data Challenge from the beginning. Over the first two years more than three hundred ideas were submitted by civil servants who spotted opportunities to use data more effectively in their day jobs. I’m sometimes asked if there are any trends in the entries and what we can learn from about government from this unique snapshot. Here are a few thoughts.

Innovation is infectious

We can see in the pattern of entries where a flurry of related ideas have been submitted in quick succession – clearly originating from one team. We also find this effect when working with departments across Whitehall – where the art of the possible in data exploitation becomes more widely understood – or a particular data product generates ideas for the next. The explosion of interest in generative AI is just the latest example of this phenomenon. This effect also works at the individual level: entrants who had some success in year one, came back for more in year two.

We need to share success stories

Part of the idea triage process is to check that an initiative is not already underway or in fact implemented – and this can take some digging. A surprising number of ideas turn out to be partially implemented already without the person submitting the idea being aware. A good example was one of the finalists last year: the team wanted to on-board self-employment data from HMRC to tackle missing child maintenance payments. The idea originated in the DWP’s Child Maintenance Service but in fact DWP already had this information and just needed to take the step of using it in a new way.

There’s a sweet spot for ambition

When it comes to the Data Challenge, bigger is not always better. Ambitious ideas that require co-ordination right across the Civil Service may be the stuff of manifestos but the prize money does not deliver that level of transformation. On the other hand, judges take note of the criteria for ‘replicable and scalable, with potential applications across government’. So an idea that can start small but scales up is looked on favourably.

Diversity is healthy

Teams are chosen to support the shortlisted ideas from among those who have either submitted unsuccessful ideas, or who have volunteered to be part of a team. Teams deliberately contain a mix of skills, grades and departments and those involved find this is one of the most enriching parts of the experience. Diverse teams make for ideas that are better refined, researched and presented – and those who have never presented before get fantastic coaching.

The impacts are real

Those who have been involved with the Data Challenge before will no doubt know of Tom Duffield’s Pharos idea which was a runner-up in year one. Pharos is an FCDO platform to estimate the number of UK citizens in cities and counties around the world at any one time. It’s useful for long term planning but also to deal with emergency evacuations of UK citizens – such as that in progress from Israel and Gaza. I’ve been privileged to work with Tom and his team to develop the first POC which has been used for emergency responses already – and now to develop the strategic platform. As we suspected during the Discovery phase, a number of other departments are now interested in this same data for their own purposes. The Home Office have been essential partners in sharing data – so if you’re reading this and think that government data sharing is in the ‘too hard’ category, take heart – it can be done.

Bill Wilson

Head of Data & Intelligence Solutions, NTT DATA. October 2023