Big Data to Connected Data to Globally Connected Perspectives

Estimated reading time: 3 mins

We just went through a severe impact of all respects, due to an unprecedented crisis. One that we weren’t prepared for, and one that caught us completely off-guard! This crisis, which still has not fazed, has been severe enough to shake some of the fundamentals around which we have been building our lives and living.

During this period of Covid 19, Big Data suddenly shot into prominence. The silver lining of this pandemic was the pace at which we evolved in leaps and bounds in handling and leveraging this big data. And, one of the first (but very difficult) steps has been the ‘Connected Data’!

Not that we hadn’t known the power or concept of ‘Connected Data’ . It’s just that during this crisis, a lot of use cases (and the absence of any primary data!) forced us to figure out the ‘how’ a bit better.

Journey of Big Data to Connected Data

A famous quote by Abraham Lincoln, “Give me six hours to chop down a tree and I will spend the first four sharpening the axe” is of supreme relevance in this case (preparing for Connected Data!).

The journey towards Connected Data in Market Research starts before the data gathering and collection stage itself. Knowing the purpose for data collection, and the manner in which one plans to ‘connect’ the various dots, is a must at the beginning itself. Typically, with the option of collecting Primary Data as per your design, this phase was not given as much consideration. One has to consider all aspects that would be needed, right from data sources to model features, in this stage.

Understand how Connected Data in a highly connected and contextual environment expedites resolution of business issues in complex, data-driven business environments.
Watch the panel discussion video for Connected Data for better Insights, now >>

During this stage, we typically draw out an ‘Information & Source’ matrix, whereby we list the kind of information we need to answer the key business question in rows, and detail out various types of sources in the columns. As a best practice, we should ensure that there is an optimum balance between the intermediate cells (that is, no particular row or column should be too heavy or light, in terms of dependence).

The next step within this stage itself is around identifying the data formats, availability and integrity available from different sources. High clarity at this stage would save many hours later on during the analysis phase. Subsequent to the stage of planning, the next step is around data collection, curation and storage. As this part is already well-oiled given current technology and practices, I would not spend much time here.

Now comes the most interesting part, which includes not only Connecting Data, but also ensuring that we connect Relevant data…that builds up to the business question that we aim to answer through this exercise. Unification of data from multiple sources, leads to Amplification of the insights and intelligence.

We typically connect disparate data sources using either a ‘soft’ or a ‘hard’ connections. A ‘soft’ connection is more of a visual connection, where we plot two different data sources on the same graph (same x-axis!), and try to visualize any kind of lag or leading connections between them. An example of a soft connection would be the share price of Datamatics, and the GDP growth rate of USA. Both are plotted on the same time axis, and one could see if and the extent the economy of US influences the stock price of Datamatics. A ‘hard’ connection, on the other hand, is more when we are able to integrate multiple sources of data, in different formats (audio, video, text, image, etc.) into a single row. An example of that would be able to integrate the banking habits of a particular customer, along with his pizza ordering transactions and his daily drive route between his home & office. Such data is usually connected at a ‘meta’ level, and not at the primary level.

While the ‘soft’ data connection in most cases is ‘two-dimensional’, a hard connection across different data sources can result in an ‘n-dimension’ distribution pattern.

Thankfully, the technology has evolved today to not only extract the meta-data from disparate multi-source data, but also to plot it, while analysing and comprehending it.

This step towards Connected Data is a tiny one, as of now! But, this field is expanding and evolving at a rapid pace, where the ultimate goal of establishing a cause and effect relationship between all that matters would be within reach! But, alas, the questions would be evolving too!!

Simply put

The Big Data journey starts from the data gathering, collection, and preparation stages itself. Information and Source matrix before the commencement of the exercise is of paramount importance. This is the stage which determines the success of the Big Data to Connected Data journey. The soft and hard connects that emerge during the process of assembling information from disparate sources provides the much needed contextual reference for the success of the entire Connected Data project and offer solutions to business issues in a complex, data-driven environment.

Next reading

• Connected Data – A Connected Ecosystem – Better Bounce-back
• “Frankenstein” or “Alice”? – Identify early-on with Data Sciences and Augmented Analytics!
• How technology is disrupting Market Research operations
• Big Data? - Get Big Insights!
• A beginner’s guide to the Role of Augmented Analytics in Consumer Behavior Analysis
• Use Artificial Intelligence and Analytics to unlock value from unstructured data