Discover Geeks Academy’s articles on: Blockchain, Coding, Cybersecurity, Cloud, Big Data, Artificial Intelligence, Gaming, Digital Innovation
Approaching the world of big data with no experience makes the learning curve steeper. The continuous creation of millions of data on a daily basis has to be stored somehow. To achieve positive results, data analysis splits in multiple stages and processes. Working in this field requires basic knowledge of the various stages that data go through, however filling a role within a data-driven company requires an excellent knowledge of at least one of the phases. What creates anxiety, confusion or fear in those unaware of the many facets of data analysis is indeed a lack of knowledge surrounding the industry, terminology, definitions and, in general, the language used and required to understand the data industry.
Big Data
Big data generally refers to a large collection of data and information that it requires the use of specific tools, technologies and analysis methods to be interpreted. A perfect example of big data is a social network, where every single user generates a huge amount of information on a daily basis through numerous and different kinds of interactions. In terms of social media, it is important to underline the distinction between structured and unstructured data: structured data are those with which we have interacted at least once in our workplace. A classic example is the Excel spreadsheet, where each piece of information (record) corresponds to a row belonging to a particular column; unstructured data, on the other hand, are heterogeneous information and do not correspond to a specific file type. They can be images, videos or text documents, which refer to a single user, as happens on social networks indeed. According to recent studies, today the volume of unstructured data is about 80-90% of the total amount of data globally.
The evolution of big data and the constant increase of unstructured data has led to a significant development of data science, particularly in the fields of artificial intelligence (AI) and machine learning.
Types of analysis
In data analytics, there are four types of analysis, which provide different but complementary points of view and results:
- Descriptive analytics
- Diagnostic analytics
- Predictive analytics
- Prescriptive analytics
Each of these analyses answers a different question. To understand the past of your company, the use of descriptive analytics allows us to understand what happened. Going deeper, we can analyse the data acquired from the previous analysis to understand why certain events occurred (diagnostic analytics). Having learned about our company's past, the next question we ask is what is likely to happen? Therefore, based on historical data, we are able to make a forecast on the company's potential future, thanks to predictive analytics. Finally, we can make use of prescriptive analytics, which is perhaps the most complex. Not only do we foresee the future but we also develop different scenarios by still relying on the previous analyses’ results, trying to find solutions and answers for every kind of situation in which the company may find itself in the short, medium or long term.
It should be noted that the progress of AI and machine learning has eased the analytical process, especially at a predictive and prescriptive level.
The stages of data analysis
Given that classification, it is now clear how complex data analysis is as well as the need to split those processes into various stages. Probably, in your first data analysis job you have met (or will meet) only one type of analysis and at most two stages. We can split the analytical process into six key steps:
- Defining the goals
- Data collection
- Data cleaning
- Data exploration
- Data mining
- Data visualization
First of all, before we start an analysis, we need to set goals. In order to agree on business goals, it is necessary to understand which is or which are the problems to be solved. Once our analysis’ main goals have been established, we move on to data collection. Our project’s data sources are often numerous and the type of data is not always the same. Based on business needs, we will have to split and classify data, thus arriving at the next stage, data cleaning. In this stage, raw data has to be manipulated, simplified and cleaned up so that it can be analysed afterwards using business intelligence (BI) tools. We often hear this definition in relation to the acronym ETL. ETL means extract, transform, load and is perhaps the most delicate and important part of the whole analytical process. Through BI tools, we are able to normalize data and therefore increase its utility and quality. Once the initial phase of data cleaning and preparation has been completed, it's time to test our data and move on to the data exploration stage. During this step, we test data in order to get useful insights, which could help us in the following steps. As soon as we complete data exploration and processing, we move on to the second-to-last stage, data mining. Here the data scientist comes in, using algorithms to train a predictive model and reach accurate predictions on the company's potential future. By training the algorithm to find patterns that are useful for our purposes, we finally reach the last stage, data visualization. Data visualization allows you to transform a huge amount of figures and information into pleasant and intuitive visualizations. These representations allow you to present the project in a simple way, emphasizing the interesting discoveries made through the whole analysis process and basically creating a story with data. If you have heard of data storytelling, this is where the magic happens. In addition to analytical knowledge, the person who presents and displays the data requires excellent communication skills. In fact, many of the professional profiles specialising in data visualization come from either the communication and design sectors.
In the stages of data analysis, there is certainly a hierarchy to follow but, by gaining experience, you will notice how these stages take place in a different order or even simultaneously. Depending on the project size, its development and the amount of resources available, the workflow might change. Furthermore, by facing issues along the way, you are generally forced to take a step back or repeat some of the stages following a different sequence.
The future is Big Data
During the past years, the total amount of data created has soared, mostly due to the expansion of the Internet of Things (IoT). The data market is therefore growing dramatically and, according to the latest estimates, is going to reach a value of $103 billion by 2027. Here are some figures about the world of data:
- Businesses generate approximately 2,000,000,000,000,000,000 bytes of data per day.
- 97.2% of companies invest in AI and Big Data.
While the numbers speak for themselves, companies struggle to keep up with the ceaseless creation of new data:
- About 95% of companies report an inability to understand and manage unstructured data.
- Only about 26% of companies say they have achieved a data-driven culture.
Don't live the future as a sidekick... be a superhero! Discover Geeks Academy’s training offer in AI & Big Data:
Sources:
https://www.impactmybiz.com/blog/what-is-the-difference-between-big-data-and-business-intelligence/