Data Quality

Installation

To install Data Quality package:

$ pip install eazyml-data-quality

Available APIs

The ez_data_quality function processes a dataset to assess its quality based on various parameters and returns the results in a structured response. Here’s a summary of what it does:

Parameter Validation:
- It checks that required parameters (train_data and outcome) are provided, and if not, returns an error message.
- It also validates the options argument (if provided) to ensure it’s a dictionary and contains valid keys.
Configuration Setup:
- It initializes configuration options, including handling specific keys related to data quality (e.g., data_quality_options, prediction_data).
- If certain keys are invalid or have incorrect data types, it returns an error.
Data Processing:

Based on the options specified (e.g., data_shape, data_emptiness, remove_outliers, data_balance, outcome_correlation), it performs various checks or transformations on the data:
- data_shape_quality: Analyzes the shape of the data.
- data_emptiness_quality: Checks for missing values and applies imputation if specified.
- data_outliers_quality: Identifies and handles outliers.
- data_balance_quality: Assesses the balance of the outcome variable.
- data_correlation_quality: Analyzes the correlation of the data with the outcome variable.
Alert Generation:

After evaluating the dataset, it generates quality alerts based on the results, flagging any issues related to the data.
Response:
- It returns a structured response in JSON format indicating whether the data quality checks were successful or if there were any issues.
- If an error occurs during processing, it returns an exception.

ez_init(access_key: str | None = None, usage_share_consent: bool | None = None, usage_delete: bool = False)

Initialize EazyML package by passing access_key

Args:

access_key (str): The access key to be set as an environment variable for EazyML.
usage_share_consent (bool): User’s agreement to allow their usage information to be shared. If consent is given, only OS information, Python version, and EazyML packages API call counts are collected.

Returns:

A dictionary containing the results of the initialization process with the following fields:

success (bool): Indicates whether the operation was successful.
message (str): A message describing the success or failure of the operation.

Example:

from eazyml_data_quality import ez_init

# Initialize the EazyML library with the access key.
# This sets the `EAZYML_ACCESS_KEY` environment variable
access_key = "your_access_key_here"  # Replace with your actual access key
_ = ez_init(access_key)

Notes:

Make sure to call this function before using other functionalities of the EazyML library that require a valid access key.
The access key will be stored in the environment, and other functions in EazyML will automatically use it when required.

ez_data_quality(train_data, outcome, options={})

Performs a series of data quality checks on the given dataset and returns a JSON response indicating the results of these checks.

Args:

train_data (DataFrame or str): A pandas DataFrame containing the training dataset. Alternatively, you can provide the file path of training dataset (as a string).
outcome (str): The target variable for the data quality.
options (dict, optional): A dictionary of options to configure the data quality process. If not provided, the function will use default settings. Supported keys include:
- data_shape (str, optional): The default is no. If yes, the function will perform a data shape check.
- data_balance (str, optional): The default is no. If yes, the function will perform a data balance check.
- data_emptiness (str, optional): The default is no. If yes, the function will perform a data emptiness check.
- impute (str, optional): The default is no. If yes, the function will perform imputation on training dataset.
- data_outliers (str, optional): The default is no. If yes, the function will perform a data outliers check.
- remove_outliers (str, optional): The default is no. If yes, the function will remove outliers from training dataset.
- outcome_correlation (str, optional): The default is no. If yes, the function will perform a data correlation check.
- data_drift (str, optional): The default is no. If yes, the function will perform a data drift check.
- model_drift (str, optional): The default is no. If yes, the function will perform a model drift check.
- prediction_data (DataFrame or str, optional): A pandas DataFrame containing the test dataset. Alternatively, you can provide the file path of test dataset (as a string).
- data_completeness (str, optional): The default is no. If yes, the function will perform a data completeness check.
- data_correctness (str, optional): The default is no. If yes, the function will perform a data correctness check.

Returns:

dict: A dictionary containing the results of the explanations with the following fields:
- success (bool): Indicates whether the operation was successful.
- message (str): A message describing the success or failure of the operation.
On Success:
data_shape_quality (dict): Contains results of data shape quality checks.

data_emptiness_quality (dict): Includes results of data emptiness checks, such as the presence of missing or null values.

data_outliers_quality (dict): Provides insights into the presence of outliers.

data_balance_quality (dict): Contains information about the balance of data.

data_correlation_quality (dict): Includes results of correlation checks, identifying highly correlated features or potential redundancies.

data_completeness_quality (dict): Includes results of data completeness checks.

data_correctness_quality (dict): Includes results of data correctness checks.

drift_quality (dict): Includes results of data drift and model drift checks.

data_bad_quality_alerts (dict): Summarizes critical quality issues detected, with the following fields:

data_shape_alert (bool): Indicates if there are structural issues with the data (e.g., mismatched dimensions, irregular shapes).

data_balance_alert (bool): Flags issues with data balance (e.g., uneven class distributions).

data_emptiness_alert (bool): Signals significant levels of missing or null data.

data_outliers_alert (bool): Highlights the presence of extreme outliers that may affect data quality.

data_correlation_alert (bool): Flags excessive correlation among features that could lead to redundancy or multicollinearity.

data_drift_alert (bool): Flags data drift alerts based on ks data drift and psi data drift.

model_drift_alert (bool): Flags model drift alerts based on interval and distributional model drift.

Example:

from eazyml_data_quality import ez_data_quality

# Define training data path (make sure the file path is correct).
train_file_path = "path_to_your_training_data.csv"  # Replace with the correct file path

# Define the outcome (target variable)
outcome = "target"  # Replace with your actual target variable name

# Define test data path (make sure the file path is correct).
test_file_path = "path_to_your_test_data.csv"  # Replace with the correct file path

# Set the options for performing the data quality
dqa_options = {
               "data_shape": "yes",
               "data_balance": "yes",
               "data_emptiness": "yes",
               "data_outliers": "yes",
               "remove_outliers": "yes",
               "outcome_correlation": "yes",
               "data_drift": "yes",
               "model_drift": "yes",
               "prediction_data": test_file_path,
               "data_completeness": "yes",
               "data_correctness": "yes"
              }

# Call the EazyML function to perform data quality
dqa_response = ez_data_quality(train_file_path, outcome, options=dqa_options)

# dqa_response is a dictionary object with following keys.
# print (dqa_response.keys())
# dict_keys(['success', 'message', 'data_shape_quality', 'data_emptiness_quality', 'data_outliers_quality', 'data_balance_quality', 'data_correlation_quality', 'data_completeness_quality', 'data_correctness_quality', 'drift_quality', 'data_bad_quality_alerts'])