Using data governance to overcome data quality and preparation challenges

data governance visual

Enterprises continue to migrate mission-critical data and applications to the cloud, which offers more efficiency, scalability and accessibility than on-premises computing environments.

More than two-thirds of IT professionals (68%) in a recent survey on data usage said they are using the cloud to store “more or all of their data”, and to host their artificial intelligence, machine learning and analytics initiatives (66%), and data management functions (69%). Nearly nine out of 10 respondents (88%) to the survey by data wrangling company Trifacta expect most or all of their data to be stored in the cloud two years from now.

The survey also reveals that poor data quality is undermining many cloud-based AI and ML initiatives as well as analytics-based decision-making because organizations are spending too much time on data cleansing and preparation. Nearly half of the respondents say they spend more than 10 hours (22%) preparing data for analytics and AI/ML initiatives, with 24% reporting they spend more than 20 hours on data prep.

In addition to the bottleneck poor data quality creates, survey respondents say it has led to errors in forecasting demand (59%) and targeting prospects (26%). Unsurprisingly, these problems are rattling the confidence of C-suite members in their organizations’ data operation. Three out of four (75%) survey respondents say they lack confidence in the quality of the data from which they are supposed to be making critical business decisions. That’s a tough way to run an enterprise.

So how can enterprises make data prep more efficient to 1) ensure the success of AI and ML initiatives in the cloud, and 2) to improve analytics-based decision-making? The answer is to implement an effective and ongoing data governance program.

Successful data governance requires a comprehensive approach that involves data analysts and scientists, data quality professionals, and an overarching data strategy to guide the governance program.

Organizations lacking the necessary data governance and management skills must either hire from outside or help current employees attain these skills. However, both options can be time-consuming. One way to overcome this skills deficit without engaging in lengthy hiring and training processes is to partner with a data governance services provider.

With or without an external partner, organizations must develop a data strategy as a prerequisite to forming and implementing a data governance plan. The goal of a data strategy is to identify where an enterprise’s valuable data resides (remember, most data now is based in the cloud) and to create roadmaps that connect high-value data with business objectives while accounting for IT and external requirements.

Once the organization has a defined data strategy, it must implement processes for managing metadata to make it easier to determine from where the data originated and to make the data more accessible to applications. This helps prevent data from being overlooked or misinterpreted.

From there organizations can begin to build a data governance framework to define who is responsible for data across the value chain, and to implement diagnostics to gauge data quality. Data governance programs also should include risk assessments and establishment of rules to safeguard data.

The sheer volume of data can overwhelm organizations unprepared to deal with it. Poor-quality data compounds this problem, causing delays that sidetrack cloud-based AI and ML initiatives and presenting enterprise leaders with inaccurate or incomplete information that can precipitate disastrous decisions.

A strong data governance program can mitigate and even eliminate these data volume and quality issues. The results should be more transparency into an organization’s cloud-based and on-premises data, greater confidence in that data, and better data-based business decisions.

Comments

  1. Joe Yacura says:

    Excellent article.

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: