Strategic challenges in big data business intelligence

Over the past few decades, the use of computing systems for big data business intelligence has increased exponentially. Today, even complex...

Over the past few decades, the use of computing systems for big data business intelligence has increased exponentially. Today, even complex structures like bridges and railway tracks have systems embedded within them to indicate, for example, wear and tear, and these systems can produce terabytes of data every day (Zikopoulos and Eaton 2011). Today, we often hear the phrase “Data is the new oil”. Data is a natural resource that is growing bigger. But, like any other resource, data is difficult to extract. The term “Big Data” is a misnomer, as it indicates that Big Data stands for huge data sets. However, there are many huge data sets that are not Big Data. Generally, Big Data is big where it needs to be distributed across several machines and it cannot be processed manually. However, everything that benefits us poses us with challenges. Most people mistake challenges for the characteristics of Big Data (volume, variety, velocity, and veracity, also known as the Four Vs). But the Four Vs are a tiny subset of the challenges posed by Big Data. As an example, let us attempt to have a better understanding of the challenges and opportunities in the Big Data domain.

big data business intelligence
Image Credit: Pixabay

Here, we consider the video rental service stores from the early 2000s. The shopkeepers of these stores used to track and keep a record of the videos that were often rented and those which were not. Analyzing these records, the shopkeeper could know the stock availability. The factors that could affect the sales of the rental services were the store locality, price of the products, and customer satisfaction, which could be improved by observing how the competitors handled the same elements. Since the emergence of online rental services, however, the understanding of customers has evolved to another level. Online rental services can not only track consumer watch history and their preferences but also can track activities like watch duration, the number of replays of a video, device on which the video was watched, location where the video was played, how the reviews affected the users, and the number of people who navigated to the site or application through advertisements. Similar to analyses done by shopkeepers from traditional stores, online rental services developed algorithms to predict a customer’s video queue, which would be recommended to them based on various factors as given above. This is one of the major reasons why the advent of online rental services like Netflix led to the bankruptcy of many traditional rental services.

The companies born digital, like Netflix, Google, and Amazon, have accomplished things for which business executives from a decade ago could only dream. This is a small example of the impact of the emergence of Big Data business intelligence. In addition, there have been enormous impacts on manufacturing, telecommunication, and pharmaceutical companies and government agencies. Manufacturers are using Big Data business intelligence to determine optimum maintenance cycles to replace component parts before they fail, thus increasing customer satisfaction. Telecommunication companies are using Big Data to monetize communication traffic data. Pharmaceuticals are using Big Data business intelligence to accelerate drug discovery and provide more personalized medicines to individuals. Government agencies are using Big Data business intelligence for protection against cyber attacks. This is one of the many reasons why all traditional companies are trying to move online.

As we can understand from the example mentioned above, big data business intelligence can provide us with far more insights than traditional analytics. Hence, we can make better predictions and smarter decisions than before and make Big Data business intelligence a management revolution. The challenges in becoming a Big Data-enabled organization can be quite enormous, but the benefits it provides far outweigh the challenges. Some of the important ones are complexity of coding, difficulty in deployment, and rapid improvement in technology. Construction of an optimum code to transform an unstructured and semistructured format into an understandable format can be a laborious task. Deployment is the next milestone. According to Informatica, more than half of the constructed codes never get deployed because of various compatibility issues and hindrances. Rapid improvements in technology are leading to many optimized solutions. Companies should always stay alert and be sure to adopt these optimized solutions.

Big data business intelligence: Data Challenges

When working with Big Data business intelligence, a data analyst is faced with many questions, such as, “How can we store these data?” or “How do we draw insights from these data?” or “How do we present these insights to business executives?” These questions present us with challenges that can be classified as data challenges. Data challenges and characteristics of Big Data (the Four Vs of Big Data) are considered the same. They may be inextricably intertwined, but one can differentiate them in several respects, which we explore in this section. Also, we list some key data challenges.

Storing the Data

Storing data can closely be associated with the volume of Big Data business intelligence. According to IBM, 2.5 quintillion bytes of data are generated every day (Lu et al. 2014). By 2018, it is expected that 50,000 gigabytes of data will be generated per second. This sheer volume of data is generated from two main sources, which can be classified as internal and external data sources. Internal data sources are mainly comprised by transactions, log data, and emails. External data sources include social media, audio, photos, and videos. Most of the organizations focus on analyzing internal data to extract exact insights, whereas fewer organizations also focus on analyzing external data, which can include social networks like Twitter or Facebook. This insights can reveal valuable insights that can change the pace of an entire organization. In the present day, we store almost everything: environmental data, financial data, medical data, surveillance data, and the list goes on and on (Zicari 2014). Every organization is now facing massive volumes of data. There is an essential need to capture, process, store, and analyze these huge data sets. This cannot be done using conventional database systems. Thus, organizations and enterprises must use the right technology to analyze the sheer volume of data in order to attain a perfect understanding of one’s business, customers, and services. As we require new storage mediums with higher input/output speeds and storage capacities to meet the volume and velocity requirements of Big Data business intelligence, several new data storage and capture techniques, like directattached storage (DAS), network-attached storage (NAS), and storage area networks (SAN), have been introduced. However, these systems had many limitations and drawbacks when they were implemented in large-scale distributed systems. Later, techniques for data access optimization techniques, including data replication, migration, distribution, and access parallelism, were designed, and data access platforms, such as CASTOR, dCache, GPFS, and Scalla/Xrootd, were employed for scalability and low-latency, highbandwidth access. Many companies preferred cloud storage to traditional storage, which helped them in decreasing costs, but on the other side, cloud storage also led to data security problems.

Velocity of the Data

Any event that generates digital data contributes to the ever-growing velocity of Big Data. The rate at which these events take place can be termed the velocity. Presently, the data are being generated at an increasingly massive rate, and thus the difficulties in collecting and processing these data are increasing. The speed at which the data are stored and analyzed has a stunning impact on the respective enterprise or organization. Even a minute difference of a few microseconds in analyzing and processing data may adversely affect an organization and lead to a major loss. The organization should be able to identify the most efficient solution to handle the data in order to reserve its place in the market. Any small delay in identifying insights changes the entire pace and position of an organization in a business sector. Thus, the actual efficiency lies in handling the sheer volume and variety of “data in motion” rather than “data at rest.” For example, GPS data are refreshed in real time, via satellite communication, etc.

Data Variety

The sheer volume of big data is mainly constituted by three categories: structured, semistructured, and unstructured data. Structured data can easily be processed using traditional methods that are based on relational databases. This type of data constitutes around 5% of total digital data. The structured data that do not conform with the formal structures of traditional data models can be considered semistructured data. Semistructured data also constitute about 5% of total data. Some the examples include XML, JSON, etc. The remaining whopping 90% is occupied by unstructured data, which are difficult to process in comparison to structured data. These include sensor data, log data, data generated by social media, etc. Traditional analytic methods cannot handle these various categories of data at one time. Big Data business intelligence technologies are efficient enough to handle the combination of both structured and unstructured data. Any organization’s growth largely depends upon its ability to handle, process, and extract insights from the various forms of data available to it.

Computational Power

Most data-driven companies or organizations find sheer volumes of data whose rate is increasing every second. Accessing these data at high speeds requires large computational power, which can be implemented in many ways. The first approach is to improve the hardware. There has been a huge shift in technologies, like replacement of hard disk drives with solid-state drives for increased random access speeds. Because there has been very few improvements in clock speeds in the past decade, a few companies use powerful parallel processing to increase their data-processing speed exponentially, while some other companies use a grid computing approach. All these approaches allow organizations to explore huge data volumes and gain business insights in near-real time.

Understanding the Data

To draw insights from data, one must understand various aspects of the data, like the sources, its genuineness, etc. For example, if the obtained data are from navigation systems, the analyst must know some details about the user, e.g., the user is a construction worker traveling to the work site, and so you must understand what insights you need from the particular data. Without any context, it is less likely that the analyst will be able to draw valid insights. Most data-driven companies hire a domain analyst for this issue, so that the analyst can understand the data sources, target audience, and how the audience perceives the information.

Data Quality

Data quality is closely related to data veracity. Before analyzing any data, one has to ensure that the data are refined. Data veracity or quality defines the trustworthiness, authenticity, and integrity of the data (Demchenko et al. 2013). The main focus of every provider of decision-supporting technologies is for their data to be accurate. According to IBM, one in three business leaders don’t trust the information they use to make their decisions, because the increasing volume of data makes it difficult for providers to achieve the intended accuracy. Security infrastructure is a key factor in determining the data veracity, since unauthorized access to the data of an organization will cause degradation in data veracity. Many leaders and business executives focus only on the technical aspects of Big Data before storing it, but it is always better to have quality data rather than a very large irrelevant data set, so that better results and conclusions can be drawn. This leads to questions like “How can one know if the data are relevant?” and “What is the minimum threshold of data required to perform data analysis?” In order to combat this challenge, companies need to have a data governance or information management process similar to quality management teams in other sectors to ensure the quality of the data before any problems arise.

Data Visualization

Data analysts need to present the insights drawn from the data in an easy-to-understand format for the managers and business executives. So, the main objective of this challenge is to represent the knowledge obtained more intuitively and effectively. Most data analysts use graphs for this purpose. However, Big Data  business intelligence visualization tools mostly have poor performance with regard to functionalities, scalability, and response time. This is one of many reasons why even large organizations turn to data visualization platforms for their visualization. LinkedIn, a business and employment-oriented social networking service, generates a lot of log data. In order to visualize these data, LinkedIn turned to the data visualization platform Tableau, which represents the data as intuitive graphs and pictures. According to Michael Li, Senior Director of Business Analytics at LinkedIn, around 80 to 90% of LinkedIn’s sales team accesses data on the Tableau server, allowing them to get instant insights.

The availability of new in-memory technology and high-performance analytics that use data visualization is providing a better way to analyze data more quickly than ever. To tackle these challenges effectively, several big data business intelligence technologies and techniques have been developed and many more are still under development.

Big data business intelligence: Management Challenges

Today, many companies and businesses are Big Data enabled. Therefore, for a company to stand at the top of its business chain, it should be able to obtain the most benefits from big data business intelligence, which can only be possible when a company is able to manage change effectively. The five areas listed below are particularly important for that process.


A Big Data analysis system can help in erasing most but not all uncertainity on any issue, and hence, there is a need for vision and human insights. For a company to make the most of being Big Data enabled, it must not only have better data than its competitors, but also better leaders who can draw better insights to help the company move forward. Most good leaders have spotted a great opportunity with big data business intelligence and can understand how the market is evolving, think creatively and propose truly novel offerings, articulate a compelling vision, persuade people to embrace it and work hard to realize it, and deal effectively with customers, employees, stockholders, and other stakeholders (McAfee et al. 2012). One of the reasons behind the success of a company or an organization will always be its leaders, who do all the above-mentioned actions while helping the company adapt to the Big Data business intelligence era.

Talent Management

A sudden increase in competition among companies around the data analytics sector has given birth to the profession of data scientist. A data scientist is a high-ranking professional with the training and curiosity to make discoveries in the world of Big Data business intelligence (Davenport and Patil 2012). The enthusiasm for big data business intelligence focuses on technologies that make taming it possible, including Hadoop and related open source tools, cloud computing, and data visualization. While these technologies are important, people with the right skill set who can put these technologies to good use are equally as important. Since a large number of data scientists have already been hired by startups and well-established corporations, there is a shortage and hence demand for data scientists in a few sectors. The challenges for managers in talent management include learning how to identify talent, attracting it to their enterprise, and making it productive. A data scientist can be thought of as an amalgamation of a data hacker, analyst, communicator, and trusted advisor. Perhaps the most regarded skills of a data scientist are statistics knowledge (most of which is not taught in a regular statistics course) and knowledge of methods for cleaning and organizing large data sets.


The technologies available to handle Big Data business intelligence have greatly increased and improved over the years. Most of these technologies are not very expensive, and much of them are open source. Hadoop, one of the most commonly used frameworks, combines commodity hardware with open source software. Data scientists often fashion their own tools and even conduct academic-style research. Yahoo’s data scientists made huge contributions in developing Hadoop. Facebook’s data team created Hive for programming Hadoop projects. Many other data scientists, at companies such as Google, Amazon, Microsoft, Walmart, eBay, LinkedIn, and Twitter, have added to and refined the Hadoop toolkit. Many IT companies have as their sole focus Big Data business intelligence technologies, and hence this part is generally overfocused. Although overattention to technology is not advisable, it remains to a necessary component of a Big Data business intelligence strategy.

Decision Making

As mentioned in the Leadership section above, big data business intelligence can eliminate most but not all uncertainties to help make predictions. For example, consider an oil and gas company. By incorporating and analyzing historic yield information or geological information, the system can create a far more accurate picture of the likely outcome of any given well or mine. With the ability to predict both quality and quantity of output, the commodities business is in a better position to decide with which producers they will deal, how to find an appropriate buyer, enter into advanced agreements, and negotiate better pricing, as well as employing optimized logistics planning (Schwartz and Clore 2015). This information may help in making decisions, but it too can have uncertainties which couldn’t have been predicted by analyzing historical data. Therefore, an effective organization puts information and the relevant decision rights in the same location. In the Big Data era, information is created and transferred, and expertise is not where it used to be. An artful leader will create an organization flexible enough to maximize the necessary cross-functional cooperation (McAfee et al. 2012). In addition, a reliance on only data insights to make decisions is not advised. Instead, a blend of both data insights and human insights is required.

Company Culture

One of the biggest aspects of big data business intelligence is how it supports decision making. When data are scarce, expensive to obtain, or are not documented in digital form, managers tend to make decisions on the basis of their experience, observations, and intuition. For a company to be truly data-driven, executives need to override their own intuition when the data don’t agree with it. The new role of executives in a data-driven company is to challenge the authenticity of the data, its sources, and the results. Also, executives must not spice up reports with lots of data that support their intuition. The HiPPO (the highestpaid person’s opinion) system must be fully abolished within a data-driven company. The executives must embrace these facts or should be replaced by those who do.

There are additional challenges, like privacy and security concerns, that have become more significant by the day since the emergence of Big Data business intelligence. Many people protest against big data business intelligence, as they believe in it being unethical for breaching the privacy of people. Privacy advocates believe that engineers can develop new techniques for data analytics that can minimize the costs to privacy.

Big data business intelligence: Process Challenges

Even after significant exploration and complex decision processes by experienced data scientists, capturing the right analysis model is an extremely difficult task. Success cannot be guaranteed even after considerable amount of analysis on huge data sets. Failure of Google Flu is one such example. Google Flu was designed to provide real-time monitoring of flu cases around the world. The fact that people with the flu will probably go online to find out about treatments, symptoms, and other related information was exploited by Google to track such behavior, hoping to predict flu outbreaks faster than traditional health authorities. According to Google, there exists a close relationship between the number of people searching for flu-related topics and the number of people with flu symptoms. The comparison between these query counts with the traditional flu surveillance systems revealed that these search queries tended to be most popular exactly when flu season was happening. With the help of these query counts, Google Flu estimates how flu is circulating in different countries. But just a few months after announcing Google Flu, the world was hit by the 2009 swine flu pandemic, caused by H1N1 virus, which it couldn’t track (Salzberg 2014). So, failures when dealing with big data business intelligence are quite common, as even minute flaws can change the whole game. The major process challenges include the following:

  • Capturing the data from different internal and external data sources;
  • Transforming the captured data into an analyzable form;
  • Deriving and understanding the insights and visualizing them; and
  • Usage of the new insights to serve the desired purpose.

Even after overcoming all these challenges, no one can ensure fulfillment of the desired goal. It depends upon numerous factors that include accuracy of analysis, the degree of its importance, and the impact on people and organizations.



Analytics Case Study Content Experience How-To Mobile Marketing Social Media Strategy Strategy
The Digital Media Strategy Blog: Strategic challenges in big data business intelligence
Strategic challenges in big data business intelligence
The Digital Media Strategy Blog
Not found any posts VIEW ALL Readmore Reply Cancel reply Delete By Home PAGES POSTS View All RECOMMENDED FOR YOU LABEL ARCHIVE SEARCH ALL POSTS Not found any post match with your request Back Home Sunday Monday Tuesday Wednesday Thursday Friday Saturday Sun Mon Tue Wed Thu Fri Sat January February March April May June July August September October November December Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec just now 1 minute ago $$1$$ minutes ago 1 hour ago $$1$$ hours ago Yesterday $$1$$ days ago $$1$$ weeks ago more than 5 weeks ago Followers Follow THIS CONTENT IS PREMIUM Please share to unlock Copy All Code Select All Code All codes were copied to your clipboard Can not copy the codes / texts, please press [CTRL]+[C] (or CMD+C with Mac) to copy