Strategy and Big Data

Early this year I wrote a post arguing that business strategy and business needs must drive data collection. In this post I explained that the data and information available in a business intelligence (BI) solution must be the right type of data; the data and information needed to make informed decisions. The BI solution should be based on the information needs rather than simply reporting on the data from the existing information systems.

This week I read another article that reminded me of this issue. This article discussed the technical challenges of developing a Big Data solution. Just like big data is an extension of BI but with new problems to solve, the data issue I described associated with BI solutions is further extended compounded in a big data solution.

The issue of business needs driving big data solutions is far grater in big data solutions than for BI solutions. Both big data and BI solutions require the right type of data to yield systems that provide meaningful results and lead to improved decision making. The data issues with Big data solutions are compounded due to significantly larger volume of data, significantly wider scope of data collected, and the mixed type of data collected (structured and unstructured). Big data solutions require more processing horsepower than traditional BI solutions.

It is the processing horsepower and storage limitation issues that further require a focus on the end result when designing and implementing big data solutions. The volumes of data can be reduced if only data contributing to the decision making for strategic efforts are collected. The reduced data collection also leads to reduction in the data compression and filtering, data storage, data cleaning and integration, and data representation needed to process all of the data in the big data solution.

If we allow our business strategies to determine the types of decisions we need to make we will then be able to better focus our big data efforts. With focused efforts we reduce the data and processing needed to yield meaningful outcomes. This results in improved decision making in areas that are strategically important to the organization. The strategic focus reduces the big data load and improves the value derived by the organization.