The use of data science for solving real-world problems and enhancing business productivity is becoming the norm. To achieve the goals of data science projects, Python is widely used for programming purposes in order to design and deploy machine learning (ML) models. Python is a very friendly language (it literally is), which makes it easy to do programming.

One of the key features of Python is that it allows use of a number of libraries (e.g., Numpy, Pandas, etc.) to run various commands and functions to achieve the results that one desires from the designed programs. 

In simplistic terms, Python libraries can be seen as a collection of ready-to-use codes. An estimated 137,000 Python libraries are available today. What that means is that the platform is there to save time and effort to code everything from scratch for building programs. As long as one is inclined and able to understand what is possible through the use of the available libraries, one can get started and get hands-on. Hence, anyone with some level of interest in using Python for programming can actually start doing programming in a short time by leveraging the benefits of a multitude of available libraries.

Given the above, it won’t come as a surprise that different libraries provide different types of functional leverage, such as understanding data, data visualization, data manipulation, and performing various mathematical and statistical functions. The intent here is not to go into a technical explanation of Python libraries, we will leave that bit for some other time. Instead, we will focus on briefly discussing how to use the power of these libraries to do basic data science tasks such as understanding data (in the context of project-related data).

It goes without saying that to be able to do something with the data in order to solve a problem at hand, understanding the data is the most vital step. What we mean by understanding the data is to know about its structure (i.e., variables in rows and columns), what the data is about, type of data, quality of data (i.e., clean, not skewed, do not have missing values), and outliers or any other issues that are inherent in the data that may obscure gaining meaningful and reliable insights from the data. 

Certainly, a variety of tools, such as Excel, provide features and functionalities to gain an understanding of data. But the use of Python libraries makes it quite easy to use available codes to analyze data from different angles to get a grasp of what we are working with and decide how to proceed with analysis. Further, as data science is becoming mainstream, there is a need to become conversant in using it to some extent to be able to keep up with the pace of time. Then the question is what one can do to understand project-related data using a data science-based approach. To answer, below we discuss some of the possibilities of using various available functions in Python libraries (e.g., Matplotlib, Seaborn) to get a better handle on the data.

1. Understand the structure of the data

The first step in the process of understanding data is to find out the structure of the data. It could be as simple as knowing how many records we have in the dataset. What are the variables, and how many of them? What are the types of variables? Are there any missing data points?

A variety of very simple functions, such as shape, info, describe, head, and tail, provide basic but useful information about the structure of the data. For instance, for a project risk-related dataset based on synthetic data, Python was used to get a basic understanding of data. 

The use of the shape function provided the following information: (610, 11). This suggests that the dataset has 610 records for 11 variables. The use of the info() function showed the type of variables we have in the dataset, which gave some idea about the numerical or string (categorical) focused variables.

Another simple function, i.e., describe(), provided very useful information, including the mean, standard deviation, minimum, and maximum values for each of the variables in the data set. This initial information can be very useful to understand the averages and variations in the data that we will need to work with. A snapshot of the output in Python is given below.

2. Understanding variations in the data

The real-world data is full of variations, and understanding such variations is important to see if the data is somehow affected in any way and has abnormal peaks and lows. Ideally, one would like to work with data that is distributed normally. But it is not always the case. Therefore, understanding the variations in the data is one of the first steps in its analysis.

For this purpose, a number of useful functions are available in Python libraries for data visualization. These include functions such as scatter, hist, plot, and bar. For statistical data visualization, one can use boxplot, pairplot, or stripplot functions by calling upon the Seaborn library.

The use of the above-mentioned functions will yield graphs (e.g., histogram, scatter plot), which will allow us to see if the data follows a normal distribution, is skewed (left or right), suffers from outliers, etc. Such an understanding will drive further thinking about what to do with the data to improve its utility.

3. Data preparation

Another key step at this stage is to prepare data for use. The information obtained through the use of the above-discussed functions will help in developing strategies for data amputation for missing values, treating outliers, or adding more data to make the dataset palatable for modeling and inference purposes. For example, in some cases, we can use the mean, median, or mode to replace missing values. In other cases, we can use Python functions to drop records with missing values. Similarly, where suitable, we can remove the records with outliers if it does not affect further analysis capabilities. 

Concluding thoughts: 

Developments in the field of data science are occurring at a rapid pace. One needs to keep up with the pace of these developments to stay competitive. A simple way to do that is to slowly and gradually learn the basics and try to implement them for day-to-day job purposes.

As projects are conceptualized and delivered on an ongoing basis in every business domain, a lot of data is created. Such data makes it possible to use it for data analysis purposes and to do machine learning modeling to improve decision-making efficiencies. With that in mind, in this article we have looked at some of the simple ways to understand the features of data that we have in order to see if it is of any use for gaining insights from the data.

Given the benefits of using Python, we have discussed a few functions that are available in Python libraries to show how we can understand the structure and quality of data. Certainly, the discussion is neither conclusive nor exhaustive. But it is meant to show the possibilities of working with Python libraries for project related datasets to make use of available data for enhancing project efficiencies.

© 2024 Jiwat Ram, All Rights Reserved.