Skip to main content

Why is HR so desperate to jump on the big data bandwagon?

By 04/09/2013June 22nd, 2021Technology
big data

Many businesses today are grappling with how to understand big data and what it means to their business. Over the past months, there has been an increase in the amount of talk about big data relative to HR and recruitment. Indeed CIPD’s People Management magazine published an article this month entitled “We LOVE big data” which really got us thinking.

As techies, there is nothing we love more than new technology but big data in the context of HR has really got us puzzled and a little frustrated. We can see why technology companies are keen to sell the idea of big data for HR purposes but is there really any grounding in the idea or is it all just sales patter and hot air?

To begin with let’s look at some definitions of what big data is (there are many, many varied definitions!) – Wikipedia refers to big data as “data sets with sizes beyond the ability of commonly used software tools to capture, curate, manage, and process the data within a tolerable elapsed time. Big data sizes are a constantly moving target, as of 2012 ranging from a few dozen terabytes to many petabytes of data in a single data set” whilst Gartner in 2012 updated its definition to “high volume, high velocity, and/or high variety information assets that require new forms of processing to enable enhanced decision making, insight discovery and process optimization.”

Whilst this may seem like tech jargon to some, the crux of this is that big data is simply dataset terabytes or petabytes in size which is too big, moves too fast, or simply doesn’t fit into “traditional” databases like Oracle, Microsoft SQL or MySQL.

Enterprise-class databases are designed to deal with datasets such as records which can easily be stored in tables (put simply rows and columns) and each will easily cope with billions of records in a single table whilst a single database can contain thousands of tables so how many datasets are there in HR which can exceed those limitations?

The aforementioned People Management article refers to an example where a retailer with 90,000 employees is using swipe card data to store information on their employees generating 40m records per year. If we take these figures on face value that is only 110,000 records per day – spread over even an 8 hour day that is less than 15,000 records per hour. Whilst it is a lot of data and something which only the largest companies will ever need to worry about it is not beyond the realms of a traditional database so is it really big data or is it just a lot of data?

Another example in the article mentions a theoretical use case around staff turnover and retention. It specifically talks about analysing “that person’s previous three performance reviews and comments from their peers, you might be able to gather that they’re a bit hacked off”. Is that valuable? Yes! Is it big data? You decide.

Wikipedia also lists some examples of the kinds of datasets where big data is being used currently – these include things such as Facebook which handles 50 billion photos and Walmart which handles more than 1m customer transactions every hour.

Take the Walmart example to the next level and combine with those 1m transactions per hour things like your historical buying habits, the weather and what customers usually buy in that weather, trending topics on social media or in the press, their own stock levels and promotions, your physical location and so on and this is both incredibly powerful and ideal for big data – this just wouldn’t fit into a normal database. The sheer volume of data and number of updates for millions of customers per day would overwhelm most normal systems so big data is ideal in this use case.

We are not saying that data in HR is worthless nor that data analysis does not provide value to companies large and small but the above two examples are simply data analytics and that is easily within the reach of any company large or small without needing to pay to get a “big data” solution from a vendor trying to justify the premium which inevitably comes with each and every new fad.