This is an annotated excerpt from PeopleInsight CEO, John Pensom's interview with the new podcast series, Eagle Tech Talks - which helps recruitment professionals gain greater depth and understanding of the IT industry.
This Edition of the podcast, with John and Morley, will be live here the 4th week of March, 2021. In the meantime check out the current episodes covering All-Things-IT-for-Recruiters.
Morley Surcon (Eagle Tech Talks host): What does "Big Data" mean today? How do you suggest that we think about Data Analytics?
Firstly, let's level set on terminology:
There's a conflation and misuse of terms such as big data, business intelligence, analytics, predictive, data science, AI, ML etc. They all mean something kind of different in the space of data science and data management.
Big data is simply one component, and is really meant to be a description of a "data set".
For example, a shopping list is a data set.
You create it, line by line, you write it down on a piece of paper, put it in a spreadsheet, or make some notes on your phone. This is a data set, but, it can hardly be considered big.
Big data was traditionally characterised by the 4 Vs
So to understand if a spreadsheet-based shopping list for your family is big data, you have to ask the questions....does the shopping list have big volume? Variety? Velocity? Value?
It's pretty clear to see that your personal shopping list wouldn’t be considered big data.
However, think of this from Loblaws, Amazon or Walmart’s perspective.
Let’s say they have 1 million families who have set up online accounts. All of which have some “repeat purchases' ' like milk ,they type of chicken you like, the coffee beans you need to refill every 2 weeks….The data set gets bigger, right? Volume increases.
And then at any given time, you have 500-1000 of these families accessing those individual profiles and actually shopping online. Well, you now have velocity.
And what you are actually doing is NOT JUST processing a transactional order - what is actually happening is that the system does a search that "boneless skinless" is available at your store, that they can commit 3 packages of inventory to you, followed by some financial processing, and then they issue a purchase order to the boneless skinless supplier for 3 more to replace the inventory which you've just bought…..there’s a lot going on - and this needs to happen across every store, every product, every supplier, in a synchronicity.
This is the variety side of big data. And it is complex.
Finally, big data should be used to create value.
Value in terms of relevant, timely information. To keep up with the grocery shopping scenario at the start of the pandemic - how can big data help us shift - or digitally transform - our grocery delivery business so we can continue to serve our customers and sustain our business?
So with that said, let me attempt to answer the question on clarifying “data analytics” or “business intelligence”.
Firstly, let’s break things down into 2 types of technologies (see the diagram below).
- Technology A has the sole purpose of helping you perform a "transaction", which basically creates data. An example of which is when a customer actually purchases the chicken from a grocery store.
- Alternatively, Technology B has the sole purpose of managing, searching, accessing and analyzing that transactional data.
An example I frequently use to highlight this difference is online flight bookings.
You go to Expedia or Google Flights or whatever, and you look for inventory. You set your filters, your criteria, your price sensitivity, etc. And once you have a list of dozens of options to get you from Tucson to New York, you book it.
Everything you do prior to committing that booking to the shopping cart - is reliant upon big data and business intelligence. Global flight inventories, airport connection time-thresholds, etc…..that’s all in searching and discovering masses of data points and the flight capacity of an entire industry. Incredibly complex from many different angles - but all focused on segmenting flight information from an endless series of dimensions - in real time.
This is business intelligence. Searching, accessing, visualizing and analyzing flight data.
Now take that one step further - when you click "buy now", you move out of search, and into a financial transaction.
Now there are elements of this which are also big data, but we now move into a financial system which is actually transacting the payment for you…..and a transactional payment system creates data in a logical and linear way from a system perspective.
There are a few data validations and checks that need to be done back and forth (so data is certainly a part of the flow) but this is more about following the standard transactional process of credit card processing. There’s really no “big data” associated with the financial transaction side.
Finally, transactional systems are typically used to transact a process or activity - basically creating a record for every transaction and storing that data in a very sequential way.
And this is important - the way that these transactional systems record and save the newly created data is really quite simple - and “filed” in a sequential way. This simple filing does not support business intelligence.
Some of your will remember this as the old world (like 15-20 years ago) view of IT - when we focused entirely on building systems to simply automate and transact a workflow, with "reporting" as an afterthought.
But as we know, we want to use that series of transactional records and data in a more powerful way, to understand trends, counts, rates, segments, probabilities & statistics - basically to extract value from this data - and this cannot be done with the database structures in a transactional-first technology.
The data must be taken out of these transactional systems and put into a “data first” storage technology - which can be called a data warehouse, or a data lake or some other catchy phrase.
The bottom line is that big data, business intelligence and analytics can only be truly enabled through a data-first technology.
Access Eagle Tech Talks here.