• Skip to main content
  • Skip to secondary menu
  • Skip to primary sidebar
  • Skip to footer
  • Articles
  • News
  • Events
  • Advertize
  • Jobs
  • Courses
  • Contact
  • (0)
  • LoginRegister
    • Facebook
    • LinkedIn
    • RSS
      Articles
      News
      Events
      Job Posts
    • Twitter
Datafloq

Datafloq

Data and Technology Insights

  • Categories
    • Big Data
    • Blockchain
    • Cloud
    • Internet Of Things
    • Metaverse
    • Robotics
    • Cybersecurity
    • Startups
    • Strategy
    • Technical
  • Big Data
  • Blockchain
  • Cloud
  • Metaverse
  • Internet Of Things
  • Robotics
  • Cybersecurity
  • Startups
  • Strategy
  • Technical

Is Real-Time Analytics A Money Pit?

Bill Franks / 4 min read.
February 15, 2018
Datafloq AI Score
×

Datafloq AI Score: 84.33

Datafloq enables anyone to contribute articles, but we value high-quality content. This means that we do not accept SEO link building content, spammy articles, clickbait, articles written by bots and especially not misinformation. Therefore, we have developed an AI, built using multiple built open-source and proprietary tools to instantly define whether an article is written by a human or a bot and determine the level of bias, objectivity, whether it is fact-based or not, sentiment and overall quality.

Articles published on Datafloq need to have a minimum AI score of 60% and we provide this graph to give more detailed information on how we rate this article. Please note that this is a work in progress and if you have any suggestions, feel free to contact us.

floq.to/E9tjY

Certainly, it is important to have analytics available in the timeframe needed for making decisions. For many years, it was too difficult and expensive to execute analytics anywhere near real-time and so everything was done using infrequent batch processes. As processing power has increased exponentially and costs have dropped to unprecedented levels, it is feasible to perform a wide array of enterprise analytics on a near real-time basis. However, many organizations today are vastly over-utilizing real-time analytics and are paying a price for it that, unfortunately, isn’t always recognized.

Forget Real Time. Focus on Decision Time!

Naturally, I am a proponent of ensuring that business decisions are made in a timely fashion. However, many decisions do not need to be made in anything approaching real- time. Just because something can be analyzed in real-time does not mean that it should be analyzed in real-time. Creating real-time processes where they aren’t needed leads to a lot of additional complexity and cost. Careful consideration is needed to decide how fast or frequently a given analytics process should run.

The first step is to identify the true speed required to meet business needs. I like to call this required speed Decision Time . While you might be able to make a decision faster than Decision Time, there is typically no benefit in doing so. An early decision will simply be held until it is needed. Contrary to the hype, there are surprisingly many cases in the real world where real-time speed is not needed. Here are some examples that illustrate different Decision Times:

  • Milliseconds/real-time. Optimizing ads on a website prior to the next page loading

  • Seconds. A predictive maintenance algorithm identifying if an abnormal situation is present in a car’s engine and suggesting a corrective action

  • Minutes. A courier approaching a package pickup needs the optimal delivery route, taking into account additional pickups that might be needed along the way

  • Hours. An AI process to detect cancer in a brain scan must be completed for the doctor before he returns in the morning

  • Days. A ship will arrive end of week, so analytics must decide which distribution centers are sent what percentage of the shipment given anticipated local demand and inventory levels


    Interested in what the future will bring? Download our 2023 Technology Trends eBook for free.

    Consent

  • Weeks. Analytics must determine if a US tax return is fraudulent before paying a refund in several weeks

  • Indefinite. Particle accelerator data must be analyzed to determine if an experiment was successful in creating a new particle

Why Over-use of Real-Time Analytics Can Lead to Excess Costs

Of the prior examples, only one requires real-time analytics. Why does engineering a solution to support real-time analytics have the potential to cost so much? Let’s explore the example of detecting tax fraud to illustrate.

The IRS typically pays refunds several weeks after submission of a return. Sure, fraud analytics could run within milliseconds of submission. But, the typical pattern for the IRS is that there will be a massive spike of activity on the mid-April due date, with very high activity the few days before and after. A steady increase of activity will occur in the weeks prior to the due date as well. For the remainder of the year, there will be, relatively speaking, no submissions to handle.

A system to handle returns in real-time will have to be built to support the huge influx of returns on the peak days and also all of the associated analytics. This will result in a system sized such that it will have massive unneeded capacity most of the year. That’s expensive! Even if the solution is deployed on the cloud, it will still require lots of temporary capacity, which will cost more than a committed base level of usage. In addition, since the analytics must be guaranteed to complete in real-time, a range of redundancies and contingency plans will have to be made in case the primary systems go down, adding even more cost and complexity.

Alternatively, returns can be accepted during peak times with none of the analytics completed until later. Once the rush is over, there are still days or weeks to work through the returns and run the fraud analytics. An internal system of much smaller size can handle this processing over a period of time. Or, a cloud-based system can handle it within typical committed capacity. Overall, it is less complex, cheaper, and just as effective to execute the analytics on a delayed basis. And, all analytics will still be completed within the required Decision Time window.

Always Build to the Required Decision Time

Business owners may be enamored with the idea of building real-time analytics. Many analytics professionals are too. However, as analytic professionals it is upon us to meet business needs while also being good stewards of resources. If we understand when a decision is needed, and analytics processes are built to meet that need, business owners will be thrilled. They may ask, Is this running in real-time ? A simple response is, It is running as close to real-time as we need it to . That’s really all that matters. The savings compared to the cost to make the process real-time can then be put toward an additional analytics effort targeted for completion in Decision Time. That will make everyone happy.

Originally published by the International Institute for Analytics

Categories: Big Data, Strategy
Tags: analytics, Big Data, real-time, real-time analytics

About Bill Franks

Bill Franks is an internationally recognized chief analytics officer who is a thought leader, speaker, consultant, and author focused on analytics and data science. Franks is also the author of Winning The Room, 97 Things About Ethics Everyone In Data Science Should Know, Taming The Big Data Tidal Wave, and The Analytics Revolution. His work has spanned clients in a variety of industries for companies ranging in size from Fortune 100 companies to small non-profit organizations. You can learn more at https://www.bill-franks.com.

Primary Sidebar

E-mail Newsletter

Sign up to receive email updates daily and to hear what's going on with us!

Publish
AN Article
Submit
a press release
List
AN Event
Create
A Job Post
Host your website with Managed WordPress for $1.00/mo with GoDaddy!

Related Articles

The Advantages of IT Staff Augmentation Over Traditional Hiring

May 4, 2023 By Mukesh Ram

The State of Digital Asset Management in 2023

May 3, 2023 By pimcoremkt

Test Data Management – Implementation Challenges and Tools Available

May 1, 2023 By yash.mehta262

Related Jobs

  • Software Engineer | South Yorkshire, GB - February 07, 2023
  • Software Engineer with C# .net Investment House | London, GB - February 07, 2023
  • Senior Java Developer | London, GB - February 07, 2023
  • Software Engineer – Growing Digital Media Company | London, GB - February 07, 2023
  • LBG Returners – Senior Data Analyst | Chester Moor, GB - February 07, 2023
More Jobs

Tags

AI Amazon analysis analytics app application Artificial Intelligence BI Big Data business China Cloud Companies company costs crypto customers Data design development digital environment experience future Google+ government information learning machine learning market mobile Musk news Other public research sales security share social social media software strategy technology twitter

Related Events

  • 6th Middle East Banking AI & Analytics Summit 2023 | Riyadh, Saudi Arabia - May 10, 2023
  • Data Science Salon NYC: AI & Machine Learning in Finance & Technology | The Theater Center - December 7, 2022
  • Big Data LDN 2023 | Olympia London - September 20, 2023
More events

Related Online Courses

  • Oracle Cloud Data Management Foundations Workshop
  • Data Science at Scale
  • Statistics with Python
More courses

Footer


Datafloq is the one-stop source for big data, blockchain and artificial intelligence. We offer information, insights and opportunities to drive innovation with emerging technologies.

  • Facebook
  • LinkedIn
  • RSS
  • Twitter

Recent

  • 5 Reasons Why Modern Data Integration Gives You a Competitive Advantage
  • 5 Most Common Database Structures for Small Businesses
  • 6 Ways to Reduce IT Costs Through Observability
  • How is Big Data Analytics Used in Business? These 5 Use Cases Share Valuable Insights
  • How Realistic Are Self-Driving Cars?

Search

Tags

AI Amazon analysis analytics app application Artificial Intelligence BI Big Data business China Cloud Companies company costs crypto customers Data design development digital environment experience future Google+ government information learning machine learning market mobile Musk news Other public research sales security share social social media software strategy technology twitter

Copyright © 2023 Datafloq
HTML Sitemap| Privacy| Terms| Cookies

  • Facebook
  • Twitter
  • LinkedIn
  • WhatsApp

In order to optimize the website and to continuously improve Datafloq, we use cookies. For more information click here.

Dear visitor,
Thank you for visiting Datafloq. If you find our content interesting, please subscribe to our weekly newsletter:

Did you know that you can publish job posts for free on Datafloq? You can start immediately and find the best candidates for free! Click here to get started.

Not Now Subscribe

Thanks for visiting Datafloq
If you enjoyed our content on emerging technologies, why not subscribe to our weekly newsletter to receive the latest news straight into your mailbox?

Subscribe

No thanks

Privacy Overview

This website uses cookies so that we can provide you with the best user experience possible. Cookie information is stored in your browser and performs functions such as recognising you when you return to our website and helping our team to understand which sections of the website you find most interesting and useful.

Necessary Cookies

Strictly Necessary Cookie should be enabled at all times so that we can save your preferences for cookie settings.

If you disable this cookie, we will not be able to save your preferences. This means that every time you visit this website you will need to enable or disable cookies again.

Marketing cookies

This website uses Google Analytics to collect anonymous information such as the number of visitors to the site, and the most popular pages.

Keeping this cookie enabled helps us to improve our website.

Please enable Strictly Necessary Cookies first so that we can save your preferences!