Open Data Issues

UK open data has a tonne of untapped potential. However, the challenges associated with handling large data volumes and overwhelming data complexity can become a major obstacle when trying to realise its value.

Data Volume

Most open data is small in volume, but it is often the very large datasets that contain the most impactful and actionable insights; they are good candidates for building business intelligence and advanced analytics solutions. The size of the data alone is not the issue; it is the challenge of efficiently transforming millions or billions of data points into an analysis-ready state. This is a task that typically requires a data engineering skill set and, even for a seasoned data engineer, creating, maintaining, monitoring, and enhancing the necessary data pipelines is a considerable amount of work.

Data Complexity

Although open, the data and context is often scattered across several semi-structured and unstructured data sources such as CSVs, Excel workbooks, Word documents, web pages, and even PDFs. It is quite common for...

Continue reading this article on our Open Data Blend Blog.