A New Approach to Securing Success for Big Data Initiatives

This thought leadership article is brought to you by ClearFunnel delivering end-to-end Big Data solutions-as-a-service on subscription fee model. ClearFunnel.

It is no secret that a majority of Big Data investments and projects fail to realize their goals. On the one hand, organizations need for Big Data solutions is rapidly growing and on the other, the success rate of such initiatives remains abysmally low. This is a double whammy at its best for product managers whose business use cases depend on successful Big Data implementations.

This paper explains why newer and more exotic Big Data technologies and platforms are not effective responses to this crisis. Instead, the answer lies in learning from solution models already successfully implemented for several other business functions and by providing an end-to-end, turnkey Big Data solution as-a-service based on usage-based subscription fee. In addition to having a much lower total cost of ownership (TCO), the Solution as-a-service model provides key financial flexibility to move away from risky and large Capex to a low-risk and a steady Opex.

Today, most business managers who attempt to implement analytics based solutions, which require either applying Decision Science at a Big Data scale or leveraging a Big Data platform, almost immediately face a wall of obstacles in terms of complex and unwieldy technology choices, new and expensive talent needs, big and growing budget demands, longer project schedules and above all high levels of uncertainty and risk to their core business use cases. The technology and the IT projects perspective of this established issue has been widely covered in the media.

Let us now look at this issue from the perspective of business managers who seem to almost always get caught in a Big Data whirlwind of sorts when all they seek are a timely, reliable, and a reasonable solution for their underlying analytics requirement. During the lifecycle of most Big Data engagements, it is not uncommon to see the focus of the business team shifting to tame the fallouts created by the Big Data behemoth instead of continuing on driving the objectives of the business case.

Steve is one such business manager at a small and medium business (SMB) who has been through such a Big Data quagmire and he breaks-down the problem in 3 key areas (note that this list is from a business managers perspective):

1. High Complexity and Risks

Steve has huge data assets and though he understands what value he can get out of these assets, all technology options available to him for executing this Big Data project is highly complex in terms of several moving parts, long timelines and big budgets. Even then all available options provide only minimal guarantees for the expected outcomes. This challenge ultimately leads to scaling down of the business case itself, sometimes diluting it to such an extent that it finally becomes unfeasible.

2. Exorbitant Costs

Today, the volume, variety, and veracity of Big Data is not only seen in large enterprises, but also in SMBs who deal with equally large data volumes and complex algorithms for their Big Data analytics use cases. However, while large enterprises continue to spend millions of Dollars in building complex Big Data ecosystems (and also in experimentation), this luxury is not readily available to the SMBs. In fact, most of the traditional options for executing and running Big Data projects soon become out of reach to the SMBs. The overall cost of enabling just the first Big Data use case soon adds-up and risks undermining the business case itself when one carefully takes into account the costs of all the different building blocks required for creating and sustaining such a solution:

• Big Data technology and Data Science talent,
• Multiple technologies for each of the different layers of a complex architecture,
• Hardware and software infrastructure (in-house or cloud-based),
• Cost of experimentation and prototyping,
• Systems integration with other applications,
• Day-to-day operations,
• Ongoing care and feed of the system,
• Conformance to security and compliance requirement, and
• Maintaining expensive redundancies in the system for ensuring high availability.

3. Solution and Data Obsolescence

Often, by the time all the challenges of a Big Data analytics project are overcome and a solution is deployed, months pass and by then, several assumptions including some of the input data sources of the business case become obsolete thereby, rendering the Big Data solution to be a white elephant. This key fallout of the existing approaches to deliver Big Data projects severely limits Steves ability to focus on rapidly launching and adapting the revenue generating products which are dependent on Big Data solutions.

In the final tally, it is the Organization in general and the business manager in particular that ultimately bear the high cost of missed opportunities and pay the penalties for large sunk costs in Big Data systems.

Most leading reports on Big Data trends (including Gartner Predicts 2015: Big Data Challenges Move From Technology to the Organization November 2014) have quantified such failures prevalent in Big Data initiatives across enterprises. For business managers like Steve, it is important to be aware that a majority of Big Data projects are not yielding measurable value or providing competitive differentiation proportionate to the investments being made in creating complex Big Data ecosystems within organizations.

The problem that Steve is facing is not unique to his business or industry. These problems are prevalent across industries and faced by a multitude of business managers, for example:

• Product Managers who are constantly challenged to launch innovative new products and new revenue streams based on exploding data assets.
• Data Scientists who are able to gain new insights using sample data sets but are unable to efficiently scale it with Terabytes and Petabytes of actual data to derive value for the enterprise.
• SMBs which are challenged with the same Big Data problems as the large enterprises. However, what SMBs really need are Big Data solutions without the traditional Big Data budgets and risks.

So, the Big Data problems, after all, are not very different across the large, medium, and small enterprises. Thanks to the deluge of data in all organizations, the challenges of deriving a meaningful solution from those data sets are very similar across different types of businesses. Today, companies of all sizes need to cross the chasm from using the traditional, BI-type thinking to a Big Data-enabled approach for finding solutions to their data-driven analytics business cases.

Contrary to the popular trend, finding answers to these problems does not lie in investing in newer and more exotic Big Data products. Evolution of new Big Data products and platforms attempt to solve technology issues, which will not adequately address all the business problems outlined above.

Today, there is already a flood of Big Data products and implementers with varying capabilities, and their mixed bag of technologies and solution approaches only add more complexities for a business that is trying to implement what should have otherwise been a straightforward Big Data solution to power their business case. The Big Data solution options available today provide limited answers to the key business asks of achieving rapid time-to-market, guaranteed and risk-free implementation, and a healthy return on investment.

A successful Big Data solution involves bringing various moving parts together, which include expertise in Data Science, Big Data and analytics technologies, data management, solution development, infrastructure and cloud management, systems integration, and ongoing operations and maintenance. However, building these expensive and complex capabilities is not the core focus area for most businesses which need Big Data solutions. Even with strong in-house data science and subject matter expertise, several corporations and certainly most SMBs are still unable to successfully execute their Big Data plans due to upfront, large, and risky investments required for creating Big Data ecosystems consisting of large infrastructure, complex technology stacks, and expensive and hard to find talent. For example: a typical Hadoop installation requires experts skilled in Java, Map Reduce, Pig / Hive, enterprise integration, data management, server administration, security, and cloud / infrastructure management.

What option does Steve really have? He understands that:

• He needs to focus on the analytics solution enabling his business case instead of trying to invest in creating all the building blocks required to operationalize the underlying Big Data technology.
• An upfront large investment in building a Big Data ecosystem for his company will not be productive in enabling new revenue streams without incurring a large opportunity cost due to sunk investments, locked financial resources, and risky outcomes.
• The cost of developing, operating, and maintaining a complex Big Data solution and equally important, the cash flow of such an investment and expense must be directly tied to the cash flow of the expected revenue stream and also be paid for from the real value derived from such a solution.

So, from Steves perspective, what defines a successful Big Data Solution? His key ask is that the cost of such a solution must be proportional to the business value derived from it. With existing Big Data products and with the availability of cloud-based infrastructure, we can boil the data ocean with ease, but the total cost (as illustrated above ) soon becomes prohibitive if it is not directly tied to the business outcome.

One way to tie cost to outcome is a transactional model of pricing or pay-per-use based fees. Successfully delivering such a value proposition will require us to rethink the approach to delivering Big Data solutions. Subscription fee and transaction pricing model have already been successfully implemented in the areas of Software as a Service (SaaS) powered Office application suites, Email, CRM, ERP, Accounting, Payroll and HR solutions, Team Collaboration, Service Desk, Issue Management, Source Control, and Files Storage needs. These business functions have successfully traversed the classic technology journey from using in-house captive applications, then gradually outsourcing systems development, leveraging cloud-based infrastructure and platforms, to finally using hosted, turnkey SaaS models. The success of SaaS model across these above-mentioned business functions amply prove that organizations of all sizes trust and embrace pay-per-use model, especially where they get a turnkey solution at a low subscription fee.

In order to fully address all of Steves problems (as listed above), we need to go one step beyond where most existing Big Data as-a-service offerings have stopped, and really enable Big Data Solutions as-a-Service. The challenge lies in the fact that an industrial-grade Big Data Analytics back-end requires a complex web of algorithms, technologies, infrastructure, and operations building blocks (and all of those need to be kept secured and up to date). However, by successfully packaging all these algorithm and technology complexities, nuances, challenges, risks, and unknowns of an enterprise-class Big Data solution into a simple, pay-per-use interface will provide a holistic answer to Steves business case.

A well-implemented Big Data Solution as-a-Service model:

• Gives an enormous advantage to Steve by reducing his total cost for developing a Big Data solution and more importantly, tying the cost of such a solution to the business outcomes derived from it, effectively allowing Steve to pay for the solution directly from the revenue stream enabled by it.
• Accelerates product development for the business case and shrinks the gestation period to realizing returns on investment.
• Greatly reduces business risks and the transactional subscription fee model ensures that the cost of a solutions failure is negligible to the business.
• Allows Steve to effectively respond to the demands of evolving business use cases by leveraging an underlying Big Data Solution that is more agile, adaptive, and responsive. This also avoids building solutions that turn into white elephants and contribute to the solution and data obsolescence syndrome described above.
• Shields Steve and his business from all the complexities of building and maintaining a complex Big Data solution, allowing him to focus on what he does best create and grow his business.
• Provides an essential financial flexibility to move away from huge capital investment that strains cash flow to a lower and a steady operating expense of a simple subscription fee model.