Solving the challenges of data sprawl

Exponential data growth isn’t a new idea, but the pandemic and the move to hybrid workplaces has accelerated the adoption of data sources, particularly structured data, and we’re now at a point where most organizations need to invest in tools to help manage the sprawl.

This is according to records and information management professionals surveyed for the Pulse of the Industry Report 2023. According to the Report, the average organization adopted two or more new data sources over the course of the pandemic, with some introducing more than five new sources of critical data.

As a result, more than 36% of respondents saw between 100% and 500% data growth in the data for which they were responsible. Most respondents see this as a trend that will continue, expecting 50-100% growth in the next two years.

Most respondents’ organizations (81%) are now operating a hybrid model, with both cloud services and on-premises systems.

But while 72% said they increased the number of data sources, only 36% of organizations said they were doing a good job of managing the situation. Let’s take a closer look at the results.

Unstructured data is growing

The use of unstructured data sources, like documents (for example, Word, Excel, PDF) in collaboration suites, SharePoint sites, Microsoft Teams, OneDrive, and file shares, continues to be significant. For most respondents (60%) unstructured data represents between 50 and 100% of total data in their organisations.

The challenges of unstructured data

When it comes to collaboration tools or file shares, the primary challenge for records and information managers is that the tools lack the functionality to ensure robust record controls and policies.

Applications that output unstructured data lack basic records management features such as versioning, and their metadata is related to the file properties. Information related to the business context of the data is held within the file itself. The burden of categorization or classification is then on the user or the records and information management professional, who must open the file and investigate its context.

There is also an added challenge for collaboration tools such as Slack or Teams: how do you define a record? Is a Slack approval a formal business decision? What if that approval comes in the form of a poll, or an emoji reaction?

What about structured data?

A little confusingly, while most respondents rated their organizations well when it comes to managing structured data, only 26% of respondents had tools and processes for managing structured data across all the systems within their respective organizations. These systems include Customer Relationship Management systems (CRM), Financial systems, and Line of Business systems.

Structured data differs from unstructured data, as it is much more consistent: it is in a standardized format, has a well-defined structure and complies to a data model. This data is generally stored in a database.

Here too, we face challenges with systems that may lack functionality to apply records management controls. Often, records and information management professionals were not involved in the definition or decision making when acquiring these systems, so questions like how the data within should be managed may not have been considered. As a result, the organization adopts systems which lack the capabilities to allow compliance and defensible disposition.  

Regulations apply to all data, regardless of structure

But regulators don’t care which tool you are using or whether the data is structured or unstructured, they only care that the data is managed appropriately. They are levying significant fines for those organizations who fail to meet this standard.

These issues are important from a records management point of view, but they really start to matter when you consider emerging privacy legislation, for example the General Data Protection Regulation (GDPR) in Europe or the California Consumer Privacy Act (CCPA) in California. These regulations are becoming the norm around the world.

Organizations subject to these regulations and others like them need to be prepared for a consumer, resident, or citizen to request all the data the organization has related to them.

If 10 people make that request simultaneously, it will be difficult for the average organization to handle. If 100 people make the request at the same time, it becomes highly challenging. If you’re in a country like Australia with a population of 25 million people, and a significant fraction of them decide to make that request, it starts to become a risk to business operations.

We need to invest in tools

How do you overcome this challenge? Not manually. At this scale the traditional method of putting the work of classification and categorization on the user no longer works, if it ever did. The finance team are focused on paying invoices, so they are not worried about classifying the document that went into the finance system. An asset manager is focused on their outcome, not the data they have used or created along the way.

We need to consider technological solutions, rather than relying on users. Artificial Intelligence (AI) or machine-learning tools such as those offered by RecordPoint can now classify data accurately, consistently and at speed.

Professionals are either late to the realization or are struggling to get their organization to understand why records management matters. Of respondents, 28% said their organization’s use of artificial intelligence and machine learning was very immature, with just 2% reporting their organisation uses machine learning and AI extensively. Most organisations (35%) said they needed to improve how they automated governance and information management.

While we have discussed this from a risk point of view, it is also worth asking: isn’t this data an asset? It may be time to start acting like it, by investing in the tools to make use of your valuable data to achieve business goals.

RecordPoint can solve your data sprawl challenges

Is your business struggling with data challenges? RecordPoint allows for consistent classification, data minimization, in-place management and more, making it easier to manage information efficiently and effectively, and demonstrate how your activity contributes to business outcomes.

‍