The upstream oil and gas industry needs to focus on data democratisation to gain return on innovation through big data, says Dr. Satyam Priyadarshy, chief data scientist, Halliburton.
The rise of big data is featuraing strongly in the upstream oil and gas industry, and along with it comes the challenge of growth in dark data. Gartner defines dark data as the information assets organisations collect, process and store during regular business activities, but generally fail to use for other purposes. Dark data can provide many insights that have not been leveraged by the industry, because it has not been mined and analysed effectively.
Traditionally, the data collected during exploration phases is analysed until a decision is made to do exploration drilling of the oil well. Similarly, the data collected from various BHA (bottom hole assembly) sensors like LWD, MWD, etc. are analysed in realtime to take decisions on the drilling in progress. If there are problems or challenges during the drilling process, those are addressed by the subject matter experts, based on their experience and intuition. Rarely is a detailed data-driven approach used to make live decisions.
The typical decision-making process works reasonably well because the domain experts have significant tacit knowledge due to their experience. However, this tacit knowledge is on the decline in the industry due to a large number of workforce retirements predicted in the near future. Articles in industry journals are suggesting that the industry has not invested enough in preparing the future workforce. Often these decisions are made with an emphasis on controlling the shortterm costs versus strategic cost savings for the enterprise. For building strategic cost saving, a big data-driven approach is needed.
A large number of papers have been written on data-driven approaches used to look at industry problems. However, many of these approaches have been tried on very limited data sets and in isolation of the big picture. The cost challenge of upstream oil and gas is not a silo problem; it is a holistic problem.
Unfortunately, the data is locked in silos due to factors such as contractual agreements, storage policies of yesterday, outdated storage technology, and lack of understanding of the value in interconnected data sets. For understanding and deriving insights for this holistic challenge, one needs to have access to data along the vertical, horizontal and time domain.
While the E&P industry watches the growth of big data and its value creation for other industries such as e-commerce, social media, consumer goods and airlines, and appreciates their reliance on data-driven approaches, we fail to see the underlying phenomenon that these industries apply to create value for the stakeholders.
This underlying phenomenon is called data democratisation. I define data democratisation as the process of creating value from all the data by opening data silos for enabling the creation of new patterns from the data, and thus gaining additional knowledge. This is possible by leveraging emerging technologies and a broader audience than typical business intelligence professionals.
In contrast to these industries, the oil and gas industry usually owns the data, but fails to connect all the data for a holistic view. Significant value can be realised by exploring hidden inefficiencies during different phases of the life cycle of the oil well, if one connects the data from different wells in the same field, and further connects the insights from various fields through a data-driven approach. Insights created through big data-driven approaches become more valuable than the raw data.
These insights can be called predictive models, for those who are mathematically inclined. The predictive models are by definition more anonymous then the raw data itself. The democratisation of these data-driven models can benefit the upstream oil and gas industry significantly.
A properly crafted data-driven approach honours the companys or enterprises data governance, privacy, and security, even when one is creating an open data culture. Today, when emerging technologies are providing real-time monitoring of massive amounts of logs, and the downturn in the price of hydrocarbons dictates the elimination of hidden inefficiencies, it is imperative to focus on data democratisation for the future good of the industry.