Documentation | Discord | Stack Overflow | Latest changelog
Do you like this project? Show us your love and give feedback!
`fg-data-profiling` primary goal is to provide a one-line Exploratory Data Analysis (EDA) experience in a consistent and fast solution. Like pandas `df.describe()` function, that is so handy, fg-data-profiling delivers an extended analysis of a DataFrame while allowing the data analysis to be exported in different formats such as **html** and **json**. The package outputs a simple and digested analysis of a dataset, including **time-series** and **text**. ## Migration Guide ### 1. Uninstall the old package ```bash pip uninstall ydata-profiling ``` ### 2. Install the new package ```bash pip install fg-data-profiling ``` ### 3. Update your imports Find and replace all occurrences of the old import in your codebase: ```python # Before import ydata_profiling from ydata_profiling import ProfileReport # After import data_profiling from data_profiling import ProfileReport ``` You can use this one-liner to find all affected files: ```bash grep -r "ydata_profiling" . --include="*.py" ``` ## ▶️ Quickstart ### Install ```cmd pip install fg-data-profiling ``` or ```cmd conda install -c conda-forge fg-data-profiling ``` ### Start profiling Start by loading your pandas `DataFrame` as you normally would, e.g. by using: ```python import numpy as np import pandas as pd from data_profiling import ProfileReport df = pd.DataFrame(np.random.rand(100, 5), columns=["a", "b", "c", "d", "e"]) ``` To generate the standard profiling report, merely run: ```python profile = ProfileReport(df, title="Profiling Report") ``` ## 📊 Key features - **Type inference**: automatic detection of columns' data types (*Categorical*, *Numerical*, *Date*, etc.) - **Warnings**: A summary of the problems/challenges in the data that you might need to work on (*missing data*, *inaccuracies*, *skewness*, etc.) - **Univariate analysis**: including descriptive statistics (mean, median, mode, etc) and informative visualizations such as distribution histograms - **Multivariate analysis**: including correlations, a detailed analysis of missing data, duplicate rows, and visual support for variables pairwise interaction - **Time-Series**: including different statistical information relative to time dependent data such as auto-correlation and seasonality, along ACF and PACF plots. - **Text analysis**: most common categories (uppercase, lowercase, separator), scripts (Latin, Cyrillic) and blocks (ASCII, Cyrilic) - **File and Image analysis**: file sizes, creation dates, dimensions, indication of truncated images and existence of EXIF metadata - **Compare datasets**: one-line solution to enable a fast and complete report on the comparison of datasets - **Flexible output formats**: all analysis can be exported to an HTML report that can be easily shared with different parties, as JSON for an easy integration in automated systems and as a widget in a Jupyter Notebook. The report contains three additional sections: - **Overview**: mostly global details about the dataset (number of records, number of variables, overall missigness and duplicates, memory footprint) - **Alerts**: a comprehensive and automatic list of potential data quality issues (high correlation, skewness, uniformity, zeros, missing values, constant values, between others) - **Reproduction**: technical details about the analysis (time, version and configuration) ### 🎁 Latest features - Want to scale? Check the latest release with ⭐⚡[Spark support](https://ydata-profiling.ydata.ai/docs/master/pages/integrations/pypspark.html)! - Looking for how you can do an EDA for Time-Series 🕛 ? Check [this blogpost](https://towardsdatascience.com/how-to-do-an-eda-for-time-series-cbb92b3b1913). - You want to compare 2 datasets and get a report? Check [this blogpost](https://medium.com/towards-artificial-intelligence/how-to-compare-2-dataset-with-pandas-profiling-2ae3a9d7695e) ### ✨ Spark Spark support has been released, but we are always looking for an extra pair of hands 👐. [Check current work in progress!](https://github.com/Data-Centric-AI-Community/fg-data-profiling/projects/3). ## 📝 Use cases fg-data-profiling can be used to deliver a variety of different use-case. The documentation includes guides, tips and tricks for tackling them: | Use case | Description | |----------|---------------------------------------------------------------------------------------------| | [Comparing datasets](https://docs.profiling.ydata.ai/latest/features/comparing_datasets) | Comparing multiple version of the same dataset | | [Profiling a Time-Series dataset](https://docs.profiling.ydata.ai/latest/features/time_series_datasets) | Generating a report for a time-series dataset with a single line of code | |[Profiling large datasets](https://docs.profiling.ydata.ai/latest/features/big_data) | Tips on how to prepare data and configure `fg-data-profiling` for working with large datasets | | [Handling sensitive data](https://docs.profiling.ydata.ai/latest/features/sensitive_data) | Generating reports which are mindful about sensitive data in the input dataset | | [Dataset metadata and data dictionaries](https://docs.profiling.ydata.ai/latest/features/metadata) | Complementing the report with dataset details and column-specific data dictionaries | | [Customizing the report's appearance](https://docs.profiling.ydata.ai/latest/features/custom_reports) | Changing the appearance of the report's page and of the contained visualizations | ### Using inside Jupyter Notebooks There are two interfaces to consume the report inside a Jupyter notebook: through widgets and through an embedded HTML report.
The above is achieved by simply displaying the report as a set of widgets. In a Jupyter Notebook, run:
```python
profile.to_widgets()
```
The HTML report can be directly embedded in a cell in a similar fashion:
```python
profile.to_notebook_iframe()
```
### Exporting the report to a file
To generate a HTML report file, save the `ProfileReport` to an object and use the `to_file()` function:
```python
profile.to_file("your_report.html")
```
Alternatively, the report's data can be obtained as a JSON file:
```python
# As a JSON string
json_data = profile.to_json()
# As a file
profile.to_file("your_report.json")
```
### Using in the command line
For standard formatted CSV files (which can be read directly by pandas without additional settings), the `data_profiling` executable can be used in the command line. The example below generates a report named *Example Profiling Report*, using a configuration file called `default.yaml`, in the file `report.html` by processing a `data.csv` dataset.
```sh
data_profiling --title "Example Profiling Report" --config_file default.yaml data.csv report.html
```
Additional details on the CLI are available [on the documentation](https://ydata-profiling.ydata.ai/docs/master/pages/getting_started/quickstart.html#command-line-usage).
## 👀 Examples
The following example reports showcase the potentialities of the package across a wide range of dataset and data types:
* [Census Income](https://ydata-profiling.ydata.ai/examples/master/census/census_report.html) (US Adult Census data relating income with other demographic properties)
* [NASA Meteorites](https://ydata-profiling.ydata.ai/examples/master/meteorites/meteorites_report.html) (comprehensive set of meteorite landing - object properties and locations) [](https://colab.research.google.com/github/ydataai/pandas-profiling/blob/master/examples/meteorites/meteorites_cloud.ipynb) [](https://mybinder.org/v2/gh/ydataai/pandas-profiling/master?filepath=examples%2Fmeteorites%2Fmeteorites%5Fcloud.ipynb)
* [Titanic](https://ydata-profiling.ydata.ai/examples/master/titanic/titanic_report.html) (the "Wonderwall" of datasets) [](https://colab.research.google.com/github/ydataai/pandas-profiling/blob/master/examples/titanic/titanic_cloud.ipynb) [](https://mybinder.org/v2/gh/ydataai/pandas-profiling/master?filepath=examples%2Ftitanic%2Ftitanic%5Fcloud.ipynb)
* [NZA](https://ydata-profiling.ydata.ai/examples/master/nza/nza_report.html) (open data from the Dutch Healthcare Authority)
* [Stata Auto](https://ydata-profiling.ydata.ai/examples/master/stata_auto/stata_auto_report.html) (1978 Automobile data)
* [Colors](https://ydata-profiling.ydata.ai/examples/master/colors/colors_report.html) (a simple colors dataset)
* [Vektis](https://ydata-profiling.ydata.ai/examples/master/vektis/vektis_report.html) (Vektis Dutch Healthcare data)
* [UCI Bank Dataset](https://ydata-profiling.ydata.ai/examples/master/bank_marketing_data/uci_bank_marketing_report.html) (marketing dataset from a bank)
* [Russian Vocabulary](https://ydata-profiling.ydata.ai/examples/master/features/russian_vocabulary.html) (100 most common Russian words, showcasing unicode text analysis)
* [Website Inaccessibility](https://ydata-profiling.ydata.ai/examples/master/features/website_inaccessibility_report.html) (website accessibility analysis, showcasing support for URL data)
* [Orange prices](https://ydata-profiling.ydata.ai/examples/master/features/united_report.html) and
* [Coal prices](https://ydata-profiling.ydata.ai/examples/master/features/flatly_report.html) (simple pricing evolution datasets, showcasing the theming options)
* [USA Air Quality](https://github.com/Data-Centric-AI-Community/fg-data-profiling/tree/master/examples/usaairquality) (Time-series air quality dataset EDA example)
* [HCC](https://github.com/Data-Centric-AI-Community/fg-data-profiling/tree/master/examples/hcc) (Open dataset from healthcare, showcasing compare between two sets of data, before and after preprocessing)
## 🛠️ Installation
Additional details, including information about widget support, are available [on the documentation](https://ydata-profiling.ydata.ai/docs/master/pages/getting_started/installation.html).
### Using pip
[](https://pepy.tech/project/fg-data-profiling)
[](https://pepy.tech/project/fg-data-profiling/month)
[](https://pypi.org/project/fg-data-profiling/)
You can install using the `pip` package manager by running:
```sh
pip install -U fg-data-profiling
```
#### Extras
The package declares "extras", sets of additional dependencies.
* `[notebook]`: support for rendering the report in Jupyter notebook widgets.
* `[unicode]`: support for more detailed Unicode analysis, at the expense of additional disk space.
* `[spark]`: support for pyspark for big dataset analysis
Install these with e.g.
```sh
pip install -U "fg-data-profiling[notebook,unicode,spark]"
```
### Using conda
[](https://anaconda.org/conda-forge/pandas-profiling)
[](https://anaconda.org/conda-forge/pandas-profiling)
You can install using the `conda` package manager by running:
```sh
conda install -c conda-forge fg-data-profiling
```
### From source (development)
Download the source code by cloning the repository or click on [Download ZIP](https://github.com/ydataai/pandas-profiling/archive/master.zip) to download the latest stable version.
Install it by navigating to the proper directory and running:
```sh
pip install -e .
```
The profiling report is written in HTML and CSS, which means a modern browser is required.
You need [Python 3](https://python3statement.github.io/) to run the package. Other dependencies can be found in the requirements files:
| Filename | Requirements|
|----------|-------------|
| [requirements.txt](https://github.com/ydataai/pandas-profiling/blob/master/requirements.txt) | Package requirements|
| [requirements-dev.txt](https://github.com/ydataai/pandas-profiling/blob/master/requirements-dev.txt) | Requirements for development|
| [requirements-test.txt](https://github.com/ydataai/pandas-profiling/blob/master/requirements-test.txt) | Requirements for testing|
| [setup.py](https://github.com/ydataai/pandas-profiling/blob/master/setup.py) | Requirements for widgets etc. |
## 🔗 Integrations
To maximize its usefulness in real world contexts, `fg-data-profiling` has a set of implicit and explicit integrations with a variety of other actors in the Data Science ecosystem:
| Integration type | Description |
|---|---|
| [Other DataFrame libraries](https://docs.profiling.ydata.ai/latest/integrations/other_dataframe_libraries) | How to compute the profiling of data stored in libraries other than pandas |
| [Great Expectations](https://ydata-profiling.ydata.ai/docs/master/pages/integrations/great_expectations.html) | Generating [Great Expectations](https://greatexpectations.io) expectations suites directly from a profiling report |
| [Interactive applications](https://docs.profiling.ydata.ai/latest/integrations/interactive_applications) | Embedding profiling reports in [Streamlit](http://streamlit.io), [Dash](http://dash.plotly.com) or [Panel](https://panel.holoviz.org) applications |
| [Pipelines](https://ydata-profiling.ydata.ai/docs/master/pages/integrations/pipelines.html) | Integration with DAG workflow execution tools like [Airflow](https://airflow.apache.org) or [Kedro](https://kedro.org) |
| [Cloud services](https://ydata-profiling.ydata.ai/docs/master/pages/integrations/cloud_services.html) | Using `fg-data-profiling` in hosted computation services like [Lambda](https://lambdalabs.com), [Google Cloud](https://github.com/GoogleCloudPlatform/analytics-componentized-patterns/blob/master/retail/propensity-model/bqml/bqml_kfp_retail_propensity_to_purchase.ipynb) or [Kaggle](https://www.kaggle.com/code) |
| [IDEs](https://ydata-profiling.ydata.ai/docs/master/pages/integrations/ides.html) | Using `fg-data-profiling` directly from integrated development environments such as [PyCharm](https://www.jetbrains.com/pycharm/) |
## 🙋 Support
Need help? Want to share a perspective? Report a bug? Ideas for collaborations? Reach out via the following channels:
- [Stack Overflow](https://stackoverflow.com/questions/tagged/pandas-profiling+or+data-profiling): ideal for asking questions on how to use the package
- [GitHub Issues](https://github.com/Data-Centric-AI-Community/fg-data-profiling/issues): bugs, proposals for changes, feature requests
- [Discord](https://tiny.ydata.ai/dcai-data-profiling): ideal for projects discussions, ask questions, collaborations, general chat
> **Need Help?**
Data quality profiling and exploratory data analysis are crucial steps in the process of Data Science and Machine Learning development.
Data-profiling is a leading tool in the data understanding step of the data science workflow as a pioneering Python package.
`data-profiling` is a leading package for data profiling, that automates and standardizes the generation of detailed reports,
complete with statistics and visualizations. The significance of the package lies in how it streamlines the process of
understanding and preparing data for analysis in a single line of code! If you're ready to get started see the [quickstart](getting-started/quickstart.md)!
## Why use data-profiling?
`data-profiling` is a valuable tool for data scientists and analysts because it streamlines EDA, provides comprehensive insights, enhances data quality,
and promotes data science best practices.
- **Simple to user**: It is so **simple to use** - a single line of code is what you need to get you started. *Do you really need more to convince you?* 😛
```python linenums="1"
import pandas as pd
from data_profiling import ProfileReport
df = pd.read_csv('data.csv')
profile = ProfileReport(df, title="Profiling Report")
```
- **Comprehensive insights in a report**: a report including a wide range of statistics and visualizations,
providing a holistic view of your data. The report is shareable as a html file or while integrate as a widget in a Jupyter Notebook.
- **Data quality assessment**: excel at the identification of missing data, duplicate entries and outliers. These insights are essential
for data cleaning and preparation, ensuring the reliability of your analysis and leading to early problems' identification.
- **Ease of integration with other flows**: all metrics of the data profiling can be consumed in a standard JSON format.
- **Data exploration for large datasets**: even with dataset with a large number of rows, `data-profiling` will be able to help you
as it supports both Pandas Dataframes and [Spark Dataframes](integrations/pyspark.md).
To learn more about the package check out [concepts overview](getting-started/concepts.md).
## 📝 Features, functionalities & integrations
Data-profiling can be used to deliver a variety of different applications. The documentation includes guides, tips and tricks for tackling them:
| Features & functionalities | Description |
|----------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------|
| [Comparing datasets](features/comparing_datasets.md) | Comparing multiple version of the same dataset |
| [Profiling a Time-Series dataset](features/time_series_datasets.md) | Generating a report for a time-series dataset with a single line of code |
| [Profiling large datasets](features/big_data.md) | Tips on how to prepare data and configure `data-profiling` for working with large datasets |
| [Handling sensitive data](features/sensitive_data.md) | Generating reports which are mindful about sensitive data in the input dataset |
| [Dataset metadata and data dictionaries](features/metadata.md) | Complementing the report with dataset details and column-specific data dictionaries |
| [Customizing the report's appearance](features/custom_reports.md ) | Changing the appearance of the report's page and of the contained visualizations |
### Tutorials
Looking for how to use certain features or how to integrate `data-profiling` in your currect stack and workflows,
check our step-by-step tutorials.
- **How to master exploratory data analysis with data-profiling?** Check this [step-by-step tutorial](https://medium.com/ydata-ai/auditing-data-quality-with-pandas-profiling-b1bf1919f856).
- **Looking on how to do exploratory data analysis for Time-series 🕛?** Check how to in this
[blogpost](https://towardsdatascience.com/how-to-do-an-eda-for-time-series-cbb92b3b1913).
To learn more about this feature [check the documentation](features/time_series_datasets.md).
- **How to compare 2 datasets? We got you covered with this [step-by-step tutorial](https://medium.com/towards-artificial-intelligence/how-to-compare-2-dataset-with-pandas-profiling-2ae3a9d7695e)**
To learn more about this feature [check the documentation](features/comparing_datasets.md).
- **Want to scale for larger datasets?** Check the information about release with ⭐⚡[Spark support](https://ydata-profiling.ydata.ai/docs/master/pages/integrations/pypspark.html)!
For more information about spark integration [check the documentation](integrations/pyspark.md)
## 🙋 Support
Need help? Want to share a perspective? Report a bug? Ideas for collaborations? Reach out via the following channels:
- [Stack Overflow](https://stackoverflow.com/questions/tagged/pandas-profiling+or+ydata-profiling): ideal for asking questions on how to use the package
- [GitHub Issues](https://github.com/Data-Centric-AI-Community/data-profiling/issues): bugs, proposals for changes, feature requests
- [Discord](https://tiny.ydata.ai/dcai-ydata-profiling): ideal for projects discussions, ask questions, collaborations, general chat
!!! tip "Help us prioritizing - before reporting, double check, it is always better to upvote!"
Before reporting an issue on GitHub, check out [Common Issues](https://docs.profiling.ydata.ai/latest/support-contribution/common_issues/).
If you want to validate if your request was prioritized check the [project pipeline details]()
## 🤝🏽 Contributing
Learn how to get involved in the [Contribution Guide](support-contribution/contribution_guidelines.md).
A low-threshold place to ask questions or start contributing is the [Data Centric AI Community's Discord](https://tiny.ydata.ai/dcai-ydata-profiling).
A big thank you to all our amazing contributors!
### ⚡ We need your help - Spark!
Spark support has been released, but we are always looking for an extra pair of hands 👐.
[Check current work in progress!](https://github.com/Data-Centric-AI-Community/data-profiling/projects/3).
---
### Getting Started Quickstart (`docs/getting-started/quickstart.md`)
# Quickstart
Start by loading your pandas `DataFrame` as you normally would, e.g. by
using:
``` python linenums="1"
import numpy as np
import pandas as pd
from data_profiling import ProfileReport
df = pd.DataFrame(np.random.rand(100, 5), columns=["a", "b", "c", "d", "e"])
```
To generate the standard profiling report, merely run:
``` python
profile = ProfileReport(df, title="YData Profiling Report")
```
## Using inside Jupyter Notebooks
There are two interfaces to consume the report inside a Jupyter notebook
(see animations below): through widgets and through an embedded HTML
report.
This is achieved by simply displaying the report as a set of widgets. In
a Jupyter Notebook, run:
``` python
profile.to_widgets()
```
The HTML report can be directly embedded in a cell in a similar fashion:
``` python
profile.to_notebook_iframe()
```
## Exporting the report to a file
To generate a HTML report file, save the `ProfileReport` to an object
and use the `to_file()` function:
``` python
profile.to_file("your_report.html")
```
Alternatively, the report's data can be obtained as a JSON file:
``` python linenums="1" title="Save your profile report as a JSON file"
# As a JSON string
json_data = profile.to_json()
# As a file
profile.to_file("your_report.json")
```
## Command line usage
For standard formatted CSV files (which can be read directly by pandas
without additional settings), the `data_profiling` executable can be
used in the command line. The example below generates a report named
*Example Profiling Report*, using a configuration file called
`default.yaml`, in the file `report.html` by processing a `data.csv`
dataset.
``` bash
data_profiling --title "Example Profiling Report" --config_file default.yaml data.csv report.html
```
Information about all available options and arguments can be viewed
through the command below. The CLI allows defining input and output
filenames, setting a custom report title, specifying
`a configuration file for custom behaviour <../advanced_usage/changing_settings>`{.interpreted-text
role="doc"} and control other advanced aspects of the experience.
``` bash
data_profiling -h
```
---
### Advanced Settings Analytics (`docs/advanced_settings/analytics.md`)
# Analytics & Telemetry
## Overview
`data-profiling` is a powerful library designed to generate profile reports from pandas and Spark Dataframe objects.
As part of our ongoing efforts to improve user experience and functionality, `data-profiling`
includes a telemetry feature. This feature collects anonymous usage data, helping us understand how the
library is used and identify areas for improvement.
The primary goal of collecting telemetry data is to:
- Enhance the functionality and performance of the data-profiling library
- Prioritize new features based on user engagement
- Identify common issues and bugs to improve overall user experience
### Data Collected
The telemetry system collects non-personal, anonymous information such as:
- Python version
- `data-profiling` version
- Frequency of use of `data-profiling` features
- Errors or exceptions thrown within the library
## Disabling usage analytics
We respect your choice to not participate in our telemetry collection. If you prefer to disable telemetry, you can do so
by setting an environment variable on your system. Disabling telemetry will not affect the functionality
of the data-profiling library, except for the ability to contribute to its usage analytics.
### Set an Environment Variable
Open your terminal or command prompt and set the YDATA_PROFILING_NO_ANALYTICS environment variable to false.
````python
import os
os.environ['YDATA_PROFILING_NO_ANALYTICS'] = 'True'
````
---
### Advanced Settings Available Settings (`docs/advanced_settings/available_settings.md`)
# Available Settings
A set of options is available in order to customize the behaviour of ``data-profiling`` and the appearance of the generated report. The depth of customization allows the creation of behaviours highly targeted at the specific dataset being analysed. The available settings are listed below. To learn how to change them, check :doc:`changing_settings`.
## General settings
Global report settings:
{{ read_csv('./tables/config_general.csv') }}
## Variable summary settings
Settings related with the information displayed for each variable.
{{ read_csv('./tables/config_variables.csv') }}
```python linenums="1" title="Configuration example"
profile = df.profile_report(
sort="ascending",
vars={
"num": {"low_categorical_threshold": 0},
"cat": {
"length": True,
"characters": False,
"words": False,
"n_obs": 5,
},
},
)
profile.config.variables.descriptions = {
"files": "Files in the filesystem",
"datec": "Creation date",
"datem": "Modification date",
}
profile.to_file("report.html")
```
## Setting dataset schema type
Configure the schema type for a given dataset.
```python linenums="1" title='Set the variable type schema to Generate the profile report'
import json
import pandas as pd
from data_profiling import ProfileReport
from data_profiling.utils.cache import cache_file
file_name = cache_file(
"titanic.csv",
"https://raw.githubusercontent.com/datasciencedojo/datasets/master/titanic.csv",
)
df = pd.read_csv(file_name)
type_schema = {"Survived": "categorical", "Embarked": "categorical"}
# We can set the type_schema only for the variables that we are certain of their types.
# All the other will be automatically inferred.
report = ProfileReport(df, title="Titanic EDA", type_schema=type_schema)
report.to_file("report.html")
```
## Missing data overview plots
Settings related with the missing data section and the visualizations it can include.
{{ read_csv('./tables/config_missing.csv') }}
```python title="Configuration example: disable heatmap for large datasets" linenums="1"
profile = df.profile_report(
missing_diagrams={
"heatmap": False,
}
)
profile.to_file("report.html")
```
## Correlations
Settings regarding correlation metrics and thresholds.
The default value is `auto`, but the following correlation matrices are available:
{{ read_csv('./tables/corr_matrices.csv') }}
For each correlation matrix you can use the following configurations:
{{ read_csv('./tables/config_correlations.csv') }}
For instance, to disable all correlation computations (might be relevant for large datasets):
````python linenums="1" title="Disabling all correlation matrices"
profile = df.profile_report(
title="Report without correlations",
correlations={
"auto": {"calculate": False},
"pearson": {"calculate": False},
"spearman": {"calculate": False},
"kendall": {"calculate": False},
"phi_k": {"calculate": False},
"cramers": {"calculate": False},
},
)
# or using a shorthand that is available for correlations
profile = df.profile_report(
title="Report without correlations",
correlations=None,
)
````
## Interactions
Settings related with the interactions section.
{{ read_csv('./tables/config_interactions.csv') }}
## Report's appearance
Settings related with the appearance and style of the report.
{{ read_csv('./tables/config_html.csv') }}
---
### Advanced Settings Caching (`docs/advanced_settings/caching.md`)
# Caching results throughout multiple runs
The ``ProfileReport`` object caches intermediary results for improved performance, which exported HTML and JSON files will reuse.
If you modify the configuration in-between runs, either a new ``ProfileReport`` object should be created or relevant cached values should be invalidated through ``report.invalidate_cache()``. The specific set of caches to reset can be passed as an argument to the ``ìnvalidate_cache()`` method:
- *rendering* to invalidate previously rendered reports (HTML, JSON or widgets)
- *report* to remove the caching of the report's structure
- *None* (default) to invalidate all caches
---
### Advanced Settings Changing Settings (`docs/advanced_settings/changing_settings.md`)
# Changing settings
There are three ways to change the settings listed in
`available_settings`{.interpreted-text role="doc"}:
- through code
- through a custom configuration file
- through environment variables
## Through code
```python linenums="1" title="Configuration example"
# Create the ProfileReport object without specifying a DataFrame
profile = df.profile_report(title="Profiling Report", pool_size=1)
# Change the configuration as desired
profile.config.html.minify_html = False
# Specify a DataFrame and trigger the report's computation
profile.to_file("output.html")
```
Some related settings are grouped together in configuration shorthands,
making it easy to selectively enable or disable certain report sections
or functionality:
- `samples`: control whether the dataset preview is shown.
- `correlations`: control whether correlation computations are
executed.
- `missing_diagrams`: control whether missing value analysis is
executed.
- `duplicates`: control whether duplicate rows are previewed.
- `interactions`: control whether interactions are computed.
```python linenums="1" title="Disable samples, correlations, missing diagrams and duplicates at once"
r = ProfileReport(
samples=None,
correlations=None,
missing_diagrams=None,
duplicates=None,
interactions=None,
)
```
## Through a custom configuration file
To control `data-profiling` through a custom file, you can start with
one of the sample configuration files below:
- [default configuration
file](https://github.com/Data-Centric-AI-Community/data-profiling/blob/master/src/data_profiling/config_default.yaml)
(default)
- [minimal configuration
file](https://github.com/Data-Centric-AI-Community/data-profiling/blob/master/src/data_profiling/config_minimal.yaml)
(minimal computation, optimized for performance)
Change the configuration to your liking and point towards that
configuration file when computing the report:
``` python linenums="1" title="Custom configuration file"
from data_profiling import ProfileReport
profile = ProfileReport(df, config_file="your_config.yml")
profile.to_file("report.html")
```
## Through environment variables
Any configuration setting can also be read from environment variables.
For example:
```python linenums="1" title="Setting title for the report with parameters"
from data_profiling import ProfileReport
profile = ProfileReport(df, title="My Custom Profiling Report")
```
is equivalent to setting the title as an environment variable
```python linenums="1" title="Set title through environment variables"
import os
from data_profiling import ProfileReport
os.environ["PROFILE_TITLE"] = "My Custom Profiling Report"
profile = ProfileReport(df)
```
---
### Features Big Data (`docs/features/big_data.md`)
# Profiling large datasets
By default, `data-profiling` comprehensively summarizes the input
dataset in a way that gives the most insights for data analysis. For
small datasets, these computations can be performed in *quasi*
real-time. For larger datasets, deciding upfront which calculations to
make might be required. Whether a computation scales to a large datasets
not only depends on the exact size of the dataset, but also on its
complexity and on whether fast computations are available. If the
computation time of the profiling becomes a bottleneck,
`data-profiling` offers several alternatives to overcome it.
## Pyspark
[Spark](https://spark.apache.org/)
!!! Note "Minimal mode"
This mode was introduced in version v4.0.0
`data-profiling` now supports Spark Dataframes profiling. You can find
an example of the integration
[here](https://github.com/Data-Centric-AI-Community/data-profiling/blob/master/examples/features/spark_example.py).
**Features supported:** - Univariate variables' analysis - Head and Tail
dataset sample - Correlation matrices: Pearson and Spearman
*Coming soon* - Missing values analysis - Interactions - Improved
histogram computation
Keep an eye on the
[GitHub](https://github.com/Data-Centric-AI-Community/data-profiling/issues) page to
follow the updates on the implementation of [Pyspark Dataframes
support](https://github.com/orgs/ydataai/projects/16/views/2).
## Minimal mode
!!! Note "**Minimal mode**"
This mode was introduced in version v2.4.0
`data-profiling` includes a minimal configuration file where the most
expensive computations are turned off by default. This is the
recommended starting point for larger datasets.
``` python
profile = ProfileReport(large_dataset, minimal=True)
profile.to_file("output.html")
```
This configuration file can be found here:
[config_minimal.yaml](https://github.com/Data-Centric-AI-Community/data-profiling/blob/master/src/data_profiling/config_minimal.yaml).
More details on settings and configuration are available in
`../advanced_usage/available_settings`{.interpreted-text role="doc"}.
## Sample the dataset
An alternative way to handle really large datasets is to use a portion
of it to generate the profiling report. Several users report this is a
good way to scale back the computation time while maintaining
representativity.
```python linenums="1" title="Sampling a large dataset" # Sample 10.000 rows sample = large_dataset.sample(10000) profile = ProfileReport(sample, minimal=True) profile.to_file("output.html") ``` The reader of the report might want to know that the profile is generated using a sample from the data. This can be done by adding a description to the report (see `metadata`{.interpreted-text role="doc"} for details). ``` python linenums="1" title="Sample 5% of your dataset" description = "Disclaimer: this profiling report was generated using a sample of 5% of the original dataset." sample = large_dataset.sample(frac=0.05) profile = sample.profile_report(dataset={"description": description}, minimal=True) profile.to_file("output.html") ``` ## Disable expensive computations To decrease the computational burden in particularly large datasets but still maintain some information of interest that may stem from them, some computations can be filtered only for certain columns. Particularly, a list of targets can be provided to **Interactions**, so that only the interactions with these variables in specific are computed. ``` python linenums="1" title="Disable expensive computations" from data_profiling import ProfileReport import pandas as pd # Reading the data data = pd.read_csv( "https://raw.githubusercontent.com/datasciencedojo/datasets/master/titanic.csv" ) # Creating the profile without specifying the data source, to allow editing the configuration profile = ProfileReport() profile.config.interactions.targets = ["Name", "Sex", "Age"] # Assigning a DataFrame and exporting to a file, triggering computation profile.df = data profile.to_file("report.html") ``` The setting controlling this, `ìnteractions.targets`, can be changed via multiple interfaces (configuration files or environment variables). For details, see `../advanced_usage/changing_settings`{.interpreted-text role="doc"}. # Concurrency `data-profiling` is a project under active development. One of the highly desired features is the addition of a scalable backend such as [Modin](https://github.com/modin-project/modin) or [Dask](https://dask.org/). Keep an eye on the [GitHub](https://github.com/Data-Centric-AI-Community/data-profiling/issues) page to follow the updates on the implementation of a concurrent and highly scalable backend. Specifically, development of a Spark backend is [currently underway](https://github.com/Data-Centric-AI-Community/data-profiling/projects/3). --- ### Features Comparing Datasets (`docs/features/comparing_datasets.md`) # Dataset Comparisonpandas-profiling is a nifty tool to compute descriptive statistics on a dataframe and issue warnings on columns with many missing values, high skew, high cardinality categorical values, high correlation, etc: https://t.co/57IezPW1nI demo notebook: https://t.co/JpqTO9FK1p
— Olivier Grisel (@ogrisel) January 11, 2018
!!! note "Dataframes compare support"
Profiling compare is supported from
data-profiling version 3.5.0 onwards.
Profiling compare is not *(yet!)* available for Spark Dataframes
`data-profiling` can be used to compare multiple version of the same
dataset. This is useful when comparing data from multiple time periods,
such as two years. Another common scenario is to view the dataset
profile for training, validation and test sets in machine learning.
The following syntax can be used to compare two datasets:
``` python linenums="1" title="Comparing 2 datasets"
from data_profiling import ProfileReport
train_df = pd.read_csv("train.csv")
train_report = ProfileReport(train_df, title="Train")
test_df = pd.read_csv("test.csv")
test_report = ProfileReport(test_df, title="Test")
comparison_report = train_report.compare(test_report)
comparison_report.to_file("comparison.html")
```
The comparison report uses the `title` attribute out of `Settings` as a
label throughout. The colors are configured in
`settings.html.style.primary_colors`. The numeric precision parameter
`settings.report.precision` can be played with to obtain some additional
space in reports.
In order to compare more than two reports, the following syntax can be
used:
``` python linenums="1" title="Comparing more than 2 datasets"
from data_profiling import ProfileReport, compare
comparison_report = compare([train_report, validation_report, test_report])
# Obtain merged statistics
statistics = comparison_report.get_description()
# Save report to file
comparison_report.to_file("comparison.html")
```
!!! note
This functionality only ensures the support report comparison
of two datasets. It is possible to obtain the statistics - the report
may have formatting issues. One of the settings that can be changed is
`settings.report.precision`. As a rule of thumb, the value 10 can be
used for a single report and 8 for comparing two reports.
---
### Features Custom Reports (`docs/features/custom_reports.md`)
# Customizing the report
In some situations, a user might want to customize the appearance
of the report to match personal preferences or a corporate brand.
``data-profiling`` offers two major customization dimensions:
the **styling of the HTML report** and the **styling of the visualizations
and plots** contained within.
## Customizing the report's theme
Several aspects of the report can be customized. The table below shows the available settings:
{{ read_csv('./tables/config_html.csv') }}
See the [available changing settings](../advanced_settings/changing_settings.md)
to see how to change and apply these settings.
## Customizing the visualizations
### Plot rendering options
A way how to pass arguments to the underlying ``matplotlib`` visualization
engine is to use the ``plot`` argument when computing the profile.
It is possible to change the default format of images to png (default is SVG)
using the key-pair ``image_format: "png"`` and also the resolution of the images using ``dpi: 800``.
An example would be:
```python linenums="1" title="Customize plots rendering"
profile = ProfileReport(
planets,
title="YData Profiling Report",
explorative=True,
plot={"dpi": 200, "image_format": "png"},
)
```
### Pie charts
Frequency plots display the frequency of categories in categorical
(or boolean) features. By default, they use bars and are shown for features
with no more than 10 distinct values. To use pie charts, set
``plot.cat_freq.type`` to ``"pie"``. The display threshold can be
configured with the ``plot.cat_freq.max_unique`` setting; it does not
change the inferred variable type.
``` python linenums="1" title="Control pie charts frequency"
profile = ProfileReport(pd.DataFrame(["a", "b", "c"]))
profile.config.plot.cat_freq.type = "pie"
# Hide the frequency plot when there are more than 2 distinct values
profile.config.plot.cat_freq.max_unique = 2
```
If the feature exceeds this threshold, the frequency plot will not be
displayed. All frequency plots can therefore be removed by setting: ``plot.cat_freq.max_unique = 0``.
The pie chart's colors can be configured to any [recognised matplotlib colour](https://matplotlib.org/stable/tutorials/colors/colors.html) using the `plot.cat_freq.colors` setting.
``` python linenums="1" title="Control pie charts colors"
profile = ProfileReport(pd.DataFrame([1, 2, 3]))
profile.config.plot.cat_freq.type = "pie"
profile.config.plot.cat_freq.colors = ["gold", "b", "#FF796C"]
```
### Colour palettes
The palettes used in visualizations like correlation matrices and missing
values overview can also be customized via the ``plot`` argument. To customize
the palette used by the correlation matrix, use the ``correlation`` key:
``` python linenums="1" title="Changing visualizations color palettes"
from data_profiling import ProfileReport
profile = ProfileReport(
df,
title="YData Profiling Report",
explorative=True,
plot={"correlation": {"cmap": "RdBu_r", "bad": "#000000"}},
)
```
Similarly, the palette for *Missing values* can be changed using ``missing`` argument:
``` python linenums="1" python
from data_profiling import ProfileReport
profile = ProfileReport(
df,
title="YData Profiling Report",
explorative=True,
plot={"missing": {"cmap": "RdBu_r"}},
)
```
``data-profiling`` accepts all ``cmap`` values (colormaps) accepted by ``matplotlib``.
The list of available colour maps can [be accessed here](https://matplotlib.org/stable/tutorials/colors/colormaps.html>).
Alternatively, it is possible to create [custom palettes](https://matplotlib.org/stable/gallery/color/custom_cmap.html>).
---
### Features Metadata (`docs/features/metadata.md`)
# Dataset description & Metadata
## Dataset description
When sharing reports with coworkers or publishing online, it might be
important to include metadata of the dataset, such as author, copyright
holder or descriptions. `data-profiling` allows complementing a report
with that information. Inspired by [schema.org\'s
Dataset](https://schema.org/Dataset), the currently supported properties
are *description*, *creator*, *author*, *url*, *copyright_year* and
*copyright_holder*.
The following example shows how to generate a report with a
*description*, *copyright_holder* *copyright_year*, *creator* and *url*.
In the generated report, these properties are found in the *Overview*,
under *About*.
``` python linenums="1" title="Add profile report description"
report = df.profile_report(
title="Masked data",
dataset={
"description": "This profiling report was generated using a sample of 5% of the original dataset.",
"copyright_holder": "StataCorp LLC",
"copyright_year": 2020,
"url": "http://www.stata-press.com/data/r15/auto2.dta",
},
)
report.to_file(Path("stata_auto_report.html"))
```
## Column descriptions
In addition to providing dataset details, often users want to include
column-specific descriptions when sharing reports with team members and
stakeholders. `data-profiling` supports creating these descriptions, so
that the report includes a built-in data dictionary. By default, the
descriptions are presented in the *Overview* section of the report, next
to each variable.
``` python linenums="1" title="Generate a report with per-variable descriptions"
profile = df.profile_report(
variables={
"descriptions": {
"files": "Files in the filesystem, # variable name: variable description",
"datec": "Creation date",
"datem": "Modification date",
}
}
)
profile.to_file("report.html")
```
Alternatively, column descriptions can be loaded from a JSON file:
``` python linenums="1" title="dataset_column_definition.json"
{
"column name 1": "column 1 definition",
"column name 2": "column 2 definition"
}
```
``` python linenums="1" title="Generate a report with descriptions per variable from a JSON definitions file"
import json
import pandas as pd
import data_profiling
definition_file = "dataset_column_definition.json"
# Read the variable descriptions
with open(definition_file, "r") as f:
definitions = json.load(f)
# By default, the descriptions are presented in the Overview section, next to each variable
report = df.profile_report(variables={"descriptions": definitions})
# We can disable showing the descriptions next to each variable
report = df.profile_report(
variables={"descriptions": definitions}, show_variable_description=False
)
report.to_file("report.html")
```
## Dataset schema
In addition to providing dataset details, users often want to include
set type schemas. This is particularly important when integrating
`data-profiling` generation with the information already in a data
catalog. When using `data-profiling` ProfileReport, users can set the
type_schema property to control the generated profiling data types. Columns
not listed in `type_schema` get the type inferred from the data (see
[Data types](../getting-started/concepts.md#data-types)).
``` python linenums="1" title="Set the variable type schema to Generate the profile report"
import json
import pandas as pd
from data_profiling import ProfileReport
from data_profiling.utils.cache import cache_file
file_name = cache_file(
"titanic.csv",
"https://raw.githubusercontent.com/datasciencedojo/datasets/master/titanic.csv",
)
df = pd.read_csv(file_name)
type_schema = {"Survived": "categorical", "Embarked": "categorical"}
# We can set the type_schema only for the variables that we are certain of their types. All the other will be automatically inferred.
report = ProfileReport(df, title="Titanic EDA", type_schema=type_schema)
report.to_file("report.html")
```
---
### Features Profile Values (`docs/features/profile_values.md`)
# Accessing profile files
## Json output structure
## Univariate variables statistics through description_set
## Correlation matrices through description_set
---
### Features Sensitive Data (`docs/features/sensitive_data.md`)
# Handling sensitive data
In certain data-sensitive contexts (for instance, private health
records), sharing a report that includes a sample would violate privacy
constraints. The following configuration shorthand groups together
various options so that only aggregate information is provided in the
report and no individual records are shown:
``` python linenums="1"
report = df.profile_report(sensitive=True)
```
Additionally, `data-profiling` does not send data to external
services, making it suitable for private data.
## Sample and duplicates
Explicitly showing a dataset\'as sample and duplicate rows can be
disabled, to guarantee the report does not directly leak any data:
``` python linenums="1"
report = df.profile_report(duplicates=None, samples=None)
```
Alternatively, it is possible to still show a sample but The following
snippet demonstrates how to generate the report but using mock/synthetic
data in the dataset sample sections. Note that the `name` and `caption`
keys are optional.
``` python linenums="1" title="Generate profiling with sensitive data: mocked sample"
# Replace with the sample you'd like to present in the report (can be from a mock or synthetic data generator)
sample_custom_data = pd.DataFrame()
sample_description = "Disclaimer: the following sample consists of synthetic data following the format of the underlying dataset."
report = df.profile_report(
sample={
"name": "Mock data sample",
"data": sample_custom_data,
"caption": sample_description,
}
)
```
!!! warning
Be aware when using `pandas.read_csv` with sensitive data such as phone
numbers. pandas' type guessing will by default coerce phone numbers
such as `0612345678` to numeric. This leads to information leakage
through aggregates (min, max, quantiles). To prevent this from
happening, keep the string representation.
``` python linenums="1"
pd.read_csv("filename.csv", dtype={"phone": str})
```
Keeping the string dtype is not enough on its own: type inference
reads numeric strings as numbers too. Set the column's type explicitly
so it is profiled as text:
``` python linenums="1"
report = ProfileReport(df, type_schema={"phone": "text"})
```
---
### Features Time Series Datasets (`docs/features/time_series_datasets.md`)
# Time-Series data
`data-profiling` can be used for a quick Exploratory Data Analysis on
time-series data. This is useful for a quick understanding on the
behaviour of time dependent variables regarding behaviours such as time
plots, seasonality, trends, stationary and data gaps.
Combined with the profiling reports compare, you're able to compare the
evolution and data behaviour through time, in terms of time-series
specific statistics such as PACF and ACF plots. It also provides the
identification of gaps in the time series, caused either by missing
values or by entries missing in the time index.
!!! example "Time series EDA tutorial"
Do you want to learn how to interpret the time-series profiling
check out [blog content here](https://ydata.ai/resources/how-to-do-an-eda-for-time-series).
You can find the a [otebook with the
[full code in our examples folder](https://github.com/Data-Centric-AI-Community/data-profiling/tree/develop/examples/usaairquality).
---
### Getting Started Concepts (`docs/getting-started/concepts.md`)
# Concepts
!!! question "Text/corpus data - your input is needed!"
`data-profiling` team is considering the support of a new set of features for corpus data
and we want to hear from you! We're particularly interested in understanding why you think
these features would be useful, and your input will help us prioritize and refine this development.
👉 Upvote [add here link for the request form]
## Data Structures supported
The profiling offers comprehensive insights into various types of data, including tabular, time-series text and image data.
- **Tabular data:** when dealing with tabular data, such as spreadsheets or databases, the profiling provides valuable statistics on data distribution, central tendencies, and categorical variable frequencies.
It identifies multivariate relations such as correlations and interactions in a visual manner. It also identifies missing data.
- **Time-series data:** when dealing with data with temporal dimensions, the profiling extends its capabilities to capture trends, seasonality, cyclic patterns and missing data gaps.
It can reveal information about data volatility, periodicity, and anomalies, facilitating a deeper understanding of time-dependent trends.
- **Text:** when it comes to text data, such as strings or documents, the profiling offers insightful statistics on the distribution of word frequencies, common phrases, and unique words.
## Data types
Types that go beyond the storage dtype (integer, float, object) decide which analysis each column gets.
A column of strings such as `"1.5"`, `"2.0"` is profiled as a number, and a column of `"yes"`/`"no"` as a boolean.
``data-profiling`` recognizes the following types:
- Boolean
- Numeric
- DateTime (dates and datetimes)
- Categorical
- Text
- TimeSeries (with `tsmode=True`)
- URL, Path, File and Image: detected only when enabled in the config (all off by
default), e.g. `ProfileReport(df, vars={"url": {"active": True}})`. File requires
Path, and Image requires File.
- Unsupported: columns that match no other type (lists, dicts, mixed values,
all-missing columns). They get the generic statistics only: missing values,
distinct and unique counts, and memory size.
Each column starts as Unsupported and moves to the most specific type it satisfies. Some columns satisfy
more than one: `["1", "2", ..., "10"]` repeated over many rows can be read both as numbers and as categories.
In that case a fixed precedence decides, so the same column always gets the same type:
1. Boolean (strings listed in `vars.bool.mappings`)
2. Numeric
3. DateTime
4. URL, Path, File, Image
5. Categorical (at most `vars.num.low_categorical_threshold` distinct numbers, or at most
`vars.cat.cardinality_threshold` distinct strings that are less than `vars.cat.percentage_cat_threshold` of the rows)
6. TimeSeries
7. Text
Two string patterns that a date parser accepts are not read as dates: values without a digit (month or weekday
names such as `"Jan"`) and values that are plain numbers (years such as `"2024"`).
When inference gets a column wrong, set its type with `type_schema` (see [Dataset schema](../features/metadata.md#dataset-schema)).
To see why a column got its type, ask the typeset:
``` python
report = ProfileReport(df)
report.typeset.explain_type(df["zip"])
# [{'from': 'Unsupported', 'matched': ['Text'], 'chosen': 'Text'},
# {'from': 'Text', 'matched': ['Numeric', 'Categorical'], 'chosen': 'Numeric'}]
```
A step with more than one entry in `matched` is a column where the precedence decided the type.
Custom types and summaries are supported: subclass `ProfilingTypeSet` or build a typeset from
`data_profiling.model.typeset_base`, where the order of the types is the precedence. See the
[default typeset](https://github.com/Data-Centric-AI-Community/fg-data-profiling/blob/develop/src/data_profiling/model/typeset.py)
for reference.
## Data quality alerts
---
### Getting Started Examples (`docs/getting-started/examples.md`)
# Examples
The following example reports showcase the potentialities of the package
across a wide range of dataset and data types:
- [Census
Income](../examples/census/census_report.html)
(US Adult Census data relating income with other demographic
properties)
- [NASA
Meteorites](../examples/meteorites/meteorites_report.html)
(comprehensive set of meteorite landing - object properties and
locations)
[](https://mybinder.org/v2/gh/Data-Centric-AI-Community/data-profiling/master?filepath=examples%2Fmeteorites%2Fmeteorites.ipynb)
[](https://colab.research.google.com/github/Data-Centric-AI-Community/data-profiling/blob/master/examples/meteorites/meteorites.ipynb)
- [Titanic](../examples/titanic/titanic_report.html)
(the \"Wonderwall\" of datasets)
[](https://mybinder.org/v2/gh/Data-Centric-AI-Community/data-profiling/master?filepath=examples%2Ftitanic%2Ftitanic.ipynb)
[](https://colab.research.google.com/github/Data-Centric-AI-Community/data-profiling/blob/master/examples/titanic/titanic.ipynb)
- [NZA](../examples/nza/nza_report.html)
(open data from the Dutch Healthcare Authority)
- [Stata
Auto](../examples/stata_auto/stata_auto_report.html)
(1978 Automobile data, Stata format)
- [Colors](../examples/colors/colors_report.html)
(a simple colors dataset)
- [Vektis](../examples/vektis/vektis_report.html)
(Vektis Dutch Healthcare data)
- [UCI Bank
Dataset](../examples/bank_marketing_data/uci_bank_marketing_report.html)
(marketing dataset from a bank)
- [Russian
Vocabulary](../examples/features/russian_vocabulary.html)
(100 most common Russian words, showcasing unicode text analysis)
- [Website
Inaccessibility](../examples/features/website_inaccessibility_report.html)
(website accessibility analysis, showcasing support for URL data)
- [Orange
prices](../examples/features/united_report.html)
and [Coal
prices](../examples/features/flatly_report.html)
(simple pricing evolution datasets, showcasing the theming options)
- [USA Air
Quality](https://github.com/Data-Centric-AI-Community/data-profiling/tree/master/examples/usaairquality)
(Time-series air quality dataset EDA example)
- [HCC](https://github.com/Data-Centric-AI-Community/data-profiling/tree/master/examples/hcc)
(Open dataset from healthcare, showcasing compare between two sets
of data, before and after preprocessing)
---
### Getting Started Installation (`docs/getting-started/installation.md`)
# Installation
## Using pip
You can install using the ``pip`` package manager by running:
```console
pip install -U fg-data-profiling
```
If you are in a notebook (locally, LambdaLabs, Google Colab or Kaggle), you can run:
```python linenums="1"
import sys
!{sys.executable} -m pip install -U "fg-data-profiling[notebook]"
!pip install jupyter-contrib-nbextensions
```
Afterwards you can run the following command
```python linenums="1"
!jupyter nbextension enable --py widgetsnbextension
```
You may have to restart the kernel or runtime for the package to work.
## Using conda
[fg-data-profiling through Conda](https://anaconda.org/conda-forge/fg-data-profiling)
A new conda environment containing the module can be created via:
```console
conda create -n data-profiling
conda activate data-profiling
conda install -c conda-forge fg-data-profiling
```
!!! tip
Don't forget to specify the ``conda-forge`` channel.
Omitting it **will not** lead to an error, as an outdated package lives on the ``main`` channel and will be installed. See :doc:`../support_contrib/common_issues` for details.
## Widgets in Jupyter Notebook/Lab
For the Jupyter widgets extension (used for progress bars and the interactive widget-based report) to work, you might need to install and activate the corresponding extensions.
This can be done via ``pip``:
```console
pip install "fg-data-profiling[notebook]"
jupyter nbextension enable --py widgetsnbextension
```
Or via ``conda``:
```console
conda install -c conda-forge ipywidgets
```
In most cases, this will also automatically configure Jupyter Notebook and Jupyter Lab (``>=3.0``). For older versions of both or in more complex
environment configurations, refer to [the official ipywidgets documentation](https://ipywidgets.readthedocs.io/en/stable/user_install.html).
## From source
Download the source code by cloning the repository or by clicking on [Download ZIP](https://github.com/Data-Centric-AI-Community/fg-data-profiling/archive/master.zip).
Install it by navigating to the uncompressed directory and running:
```console
python setup.py install
```
This can also be done via the following one-liner:
```console
pip install https://github.com/Data-Centric-AI-Community/fg-data-profiling/archive/master.zip
```
## Extras
The package declares some "extras", sets of additional dependencies.
* ``[notebook]``: support for rendering the report in Jupyter notebook widgets.
* ``[unicode]``: support for more detailed Unicode analysis, at the expense of additional disk space.
* ``[spark]``: support for pyspark engine to run the profile on big datasets
* ``[arrow]``: profile Arrow-compatible inputs directly (pyarrow tables, Hugging Face ``datasets``, Polars, DuckDB) and compute text statistics with Arrow kernels. See [other DataFrame libraries](../integrations/other_dataframe_libraries.md).
Install these with e.g.
```console
pip install -U "fg-data-profiling[notebook,unicode,spark,arrow]"
```
---
### Integrations Bytewax (`docs/integrations/bytewax.md`)
# Profiling for streaming data
## About Bytewax
[Bytewax](https://github.com/bytewax/bytewax) is an OSS stream
processing framework designed specifically for Python developers.
It allows users to build streaming data pipelines and real-time
applications with capabilities similar to Flink, Spark, and Kafka
Streams, while providing a friendly and familiar interface and 100%
compatibility with the Python ecosystem.
## Stream processing with Bytewax and data-profiling
Data Profiling is key to a successful start of any machine learning
task, and refers to the step of [thoroughly understanding our data](https://ydata.ai/resources/advanced-eda-made-simple-using-pandas-profiling): its structure, behavior, and quality.
In a nutshell, data profiling involves analyzing aspects related to
the data's format and basic descriptors (e.g., number of samples,
number/types of features, duplicate values), its intrinsic
characteristics (such as the presence of missing data or imbalanced
features), and other complicating factors that may arise during data
collection or processing (e.g., erroneous values or inconsistent
features).
!!! note "Package versions"
The integration with bytewax is available for data-profiling with
any version >=3.0.0
### Simulating a streaming
The below code serves to mimic a stream of data. This not require when
streaming data sources are available.
``` python linenums="1" title="Imports"
from datetime import datetime, timedelta, timezone
from bytewax.dataflow import Dataflow
from bytewax.connectors.stdio import StdOutput
from bytewax.connectors.files import CSVInput
from bytewax.testing import run_main
```
Then, we define our dataflow object. Afterwards, we will use a stateless
map method where we pass in a function to convert the string to a
datetime object and restructure the data to the format (device_id,
data). The map method will make the change to each data point in a
stateless way. The reason we have modified the shape of our data is so
that we can easily group the data in the next steps to profile data for
each device separately rather than for all the devices
simultaneously.
``` python linenums="1" title="Setup a data stream"
flow = Dataflow()
flow.input("simulated_stream", CSVInput("/content/iot_telemetry_data_1000"))
# parse timestamp
def parse_time(reading_data):
reading_data["ts"] = datetime.fromtimestamp(float(reading_data["ts"]), timezone.utc)
return reading_data
flow.map(parse_time)
# remap format to tuple (device_id, reading_data)
flow.map(lambda reading_data: (reading_data["device"], reading_data))
```
Now we will take advantage of the stateful capabilities of bytewax to
gather data for each device over a duration of time that we have
defined. data-profiling expects a snapshot of the data over time, which
makes the window operator the perfect method to use to do this.
In data-profiling, we are able to produce summarizing statistics for a
dataframe which is specified for a particular context. For instance, in
this example, we can produce snapshots of data referring to each IoT
device or to particular time frames:
### Profile streaming snapshots
``` python linenums="1" title="Profiling the different data snapshots"
from bytewax.window import EventClockConfig, TumblingWindow
# This is the accumulator function, and outputs a list of readings
def acc_values(acc, reading):
acc.append(reading)
return acc
# This function instructs the event clock on how to retrieve the
# event's datetime from the input.
def get_time(reading):
return reading["ts"]
# Configure the `fold_window` operator to use the event time.
cc = EventClockConfig(get_time, wait_for_system_duration=timedelta(seconds=30))
# And a tumbling window
align_to = datetime(2020, 1, 1, tzinfo=timezone.utc)
wc = TumblingWindow(align_to=align_to, length=timedelta(hours=1))
flow.fold_window("running_average", cc, wc, list, acc_values)
flow.inspect(print)
```
After the snapshots are defined, leveraging data-profiling is as simple
as calling the ProfileReport for each of the dataframes we would like to
analyze:
``` python
import pandas as pd
from data_profiling import ProfileReport
def profile(device_id__readings):
print(device_id__readings)
device_id, readings = device_id__readings
start_time = (
readings[0]["ts"]
.replace(minute=0, second=0, microsecond=0)
.strftime("%Y-%m-%d %H:%M:%S")
)
df = pd.DataFrame(readings)
profile = ProfileReport(
df, tsmode=True, sortby="ts", title=f"Sensor Readings - device: {device_id}"
)
profile.to_file(f"Ts_Profile_{device_id}-{start_time}.html")
return f"device {device_id} profiled at hour {start_time}"
flow.map(profile)
```
In this example we are writing the images out to local files as part of
a function in a map method. These could be reported out via a messaging
tool, or we could save them to some remote storage in the future. Once
the profile is complete, the dataflow expects some output so we can use
the built-in [StdOutput]{.title-ref} to print the device that was
profiled and the time it was profiled at that was passed out of the
profile function in the map step:
``` python linenums="1"
flow.output("out", StdOutput())
```
There are multiple ways to execute Bytewax dataflows. In this example,
we use the same local machine, but Bytewax can also run on multiple
Python processes, across multiple hosts, in a [Docker
container](https://bytewax.io/docs/deployment/container), using a
[Kubernetes cluster](https://bytewax.io/docs/deployment/k8s-ecosystem),
and
[more](https://bytewax.io/docs/getting-started/execution#multiple-workers-manual-cluster).
In this example, we\'ll continue with a local setup, but we encourage
you to check [waxctl](https://bytewax.io/docs/deployment/waxctl) which
manages Kubernetes dataflow deployments once your pipeline is ready to
transition to production.
Assuming we are in the same directory as the file with the dataflow
definition, we can run it using:
``` linenums="1"
python -m bytewax.run data-profiling-streaming:flow
```
We can then use the profiling reports to validate the data quality,
check for changes in schemas or data formats, and compare the data
characteristics between different devices or time windows.
We can further leverage the [comparison report
functionality](https://ydata-profiling.ydata.ai/docs/master/pages/use_cases/comparing_datasets.html)
that highlights the differences between two data profiles in a
straightforward manner, making it easier for us to detect important
patterns that need to be investigated or issues that have to be
addressed:
``` python linenums="1" linenums="1" title="Comparing different streams"
#Generate the profile for each stream
snapshot_a_report = ProfileReport(df_a, title="Snapshot A")
snapshot_b_report = ProfileReport(df_b, title="Snapshot B")
#Compare the generated profiles
comparison_report = snapshot_a_report.compare(snapshot_b_report)
comparison_report.to_file("comparison_report.html")
```
Now you're all set to start exploring your data streams! Bytewax takes
care of all the processes necessary to handle and structure data streams
into snapshots, which can then be summarized and compared with
data-profiling through a comprehensive report of data characteristics.
---
### Integrations Great Expectations (`docs/integrations/great_expectations.md`)
# Great Expectations
!!! note "Package versions"
Great expectations integration is no longer supported.
You can recreate the integration with the following
packages versions:
- data-profiling==2.1.0
- great-expectations==0.13.4
[Great Expectations](https://www.greatexpectations.io) is a Python-based
open-source library for validating, documenting, and profiling your
data. It helps you to maintain data quality and improve communication
about data between teams. With Great Expectations, you can assert what
you expect from the data you load and transform, and catch data issues
quickly -- Expectations are basically *unit tests for your data*.
`data-profiling` features a method to create a suite of Expectations
based on the results of your `ProfileReport`!
## About Great Expectations
*Expectations* are assertions about your data. In Great Expectations,
those assertions are expressed in a declarative language in the form of
simple, human-readable Python methods. For example, in order to assert
that you want values in a column `passenger_count` in your dataset to be
integers between 1 and 6, you can say:
> `expect_column_values_to_be_between(column="passenger_count", min_value=1, max_value=6)`
Great Expectations then uses this statement to validate whether the
column `passenger_count` in a given table is indeed between 1 and 6, and
returns a success or failure result. The library currently provides
[several dozen highly expressive built-in
Expectations](https://docs.greatexpectations.io/en/latest/reference/glossary_of_expectations.html),
and allows you to write custom Expectations.
Great Expectations renders Expectations to clean, human-readable
documentation called *Data Docs*. These HTML docs contain both your
Expectation Suites as well as your data validation results each time
validation is run -- think of it as a continuously updated data quality
report.
For more information about Great Expectations, check out the [Great
Expectations
documentation](https://docs.greatexpectations.io/en/latest/) and join
the [Great Expectations Slack
channel](https://www.greatexpectations.io/slack) for help.
## Creating Expectation Suites with adata-profiling
An *Expectation Suite* is simply a set of Expectations. You can create
Expectation Suites by writing out individual statements, such as the one
above, or by automatically generating them based on profiler results.
`data-profiling` provides a simple `to_expectation_suite()` method that
returns a Great Expectations `ExpectationSuite` object which contains a
set of Expectations.
**Pre-requisites**: In order to run the `to_expectation_suite()` method,
you will need to install Great Expectations with
`pip install great_expectations`
If you would like to use the additional features such as saving the
Suite and building Data Docs, you will also need to configure a Great
Expectations Data Context by running `great_expectations init` in your
project\'s directory.
``` python linenums="1" title="Get your set of expectations"
import pandas as pd
from data_profiling import ProfileReport
df = pd.read_csv("titanic.csv")
profile = ProfileReport(df, title="Data Profiling Report", explorative=True)
# Obtain an Expectation Suite with a set of default Expectations
# By default, this also profiles the dataset, saves the suite, runs validation, and builds Data Docs
suite = profile.to_expectation_suite()
```
This assumes that the `great_expectations` Data Context directory is in
the *same path* where you run the script. In order to specify the
location of your Data Context, pass it in as an argument:
``` python linenums="1" title="Generate a suite of expectations"
import great_expectations as ge
data_context = ge.data_context.DataContext(
context_root_dir="/Users/panda/code/my_ge_project/"
)
suite = profile.to_expectation_suite(data_context=data_context)
```
You can also configure each feature individually in the function call:
``` python linenums="1" title="Configure features"
suite = profile.to_expectation_suite(
suite_name="titanic_expectations",
data_context=data_context,
save_suite=False,
run_validation=False,
build_data_docs=False,
handler=handler,
)
```
See [the Great Expectations
Examples](https://github.com/Data-Centric-AI-Community/data-profiling/blob/master/examples/integrations/great_expectations/great_expectations_example.py)
for complete examples.
---
### Integrations Ides (`docs/integrations/ides.md`)
# IDEs
The package can be directly consumed in some Integrated Development
Environments, such as [PyCharm](https://www.jetbrains.com/pycharm/).
## PyCharm
1. Install `data-profiling` via
`../getting_started/installation`{.interpreted-text role="doc"}
2. Locate your `data-profiling` executable.
On macOS / Linux / BSD:
``` console
$ which data_profiling
(example) /usr/local/bin/data_profiling
```
On Windows:
``` console
$ where data_profiling
(example) C:\ProgramData\Anaconda3\Scripts\data_profiling.exe
```
3. In PyCharm, go to *Settings* (or *Preferences* on macOS) \ *Tools*
\ *External tools*
4. Click the **+** icon to add a new external tool
5. Insert the following values
- Name: `Data Profiling`
- Program: *The location obtained in step 2*
- Arguments:
`"$FilePath$" "$FileDir$/$FileNameWithoutAllExtensions$_report.html"`
- Working Directory: `$ProjectFileDir$`
{.align-center
width="400px"}
To use the PyCharm Integration, right click on any dataset file and
*External Tools* \ *Data Profiling*.
---
### Integrations Interactive Applications (`docs/integrations/interactive_applications.md`)
# Interactive applications
The `data-profiling` report, through several of its interfaces, can be
integrated in interactive data applications such as those developed with
[Streamlit](https://streamlit.io) or [Panel](https://panel.holoviz.org).
## Streamlit
[Streamlit](https://www.streamlit.io) is an open-source Python library
made to build web-apps for machine learning and data science.
!!! note
This feature is only available for versions previous to data-profiling
(<=3.6.2).
``` python linenums="1" title="Creating a simple Streamlit app with data-profiling"
import pandas as pd
import data_profiling
import streamlit as st
from streamlit_pandas_profiling import st_profile_report
df = pd.read_csv(
"https://raw.githubusercontent.com/datasciencedojo/datasets/master/titanic.csv"
)
pr = df.profile_report()
st.title("Profiling in Streamlit")
st.write(df)
st_profile_report(pr)
```
You can install the [data-profiling
component](https://github.com/Ghasel/streamlit-pandas-profiling) for
Streamlit with pip.
``` console
pip install streamlit-pandas-profiling
```
## Dash
[Dash](hhttps://github.com/plotly/dash) is a Python framework for
building machine learning & data science web apps, built on top of
Plotly.js, React and Flask. It is commonly used for interactive data
exploration, precisely where `data-profiling` also focuses. Inline
access to the insights provided by `data-profiling` can help guide the
exploratory work allowed by Dash. To integrate a Profiling Report inside
a Dash app, two options exist:
### Load HTML version of report as an asset
Assuming the HTML version of the report is in `report.html`, move it to
a folder called `assets`. The snippet below shows a simple Dash app,
`app.py`, embedding this report:
``` python linenums="1" title="Create a Dash dashboard with data-profiling integrated"
import dash
from dash import html
app = dash.Dash(__name__)
app.layout = html.Div(
children=[
html.Iframe(
src="assets/census_report.html", # must be under assets/ to be properly served
style={"height": "1080px", "width": "100%"},
)
]
)
if __name__ == "__main__":
app.run_server(debug=True)
```
When running `python app.py`, a Dash app with the report embedded will
be available on
---
### Integrations Other Dataframe Libraries (`docs/integrations/other_dataframe_libraries.md`)
# Other DataFrame libraries
`data-profiling` is built on `pandas` and `numpy`. Pandas supports a
wide range of data formats including CSV, XLSX, SQL, JSON, HDF5, SAS,
BigQuery and Stata. Read more on [supported formats by
Pandas](https://pandas.pydata.org/docs/user_guide/io.html).
## Arrow-compatible inputs
With the `arrow` extra, `ProfileReport` accepts Arrow-compatible objects
directly, with no manual conversion:
```bash
pip install "fg-data-profiling[arrow]"
```
| Input | Notes |
|-------|-------|
| `pyarrow.Table`, `RecordBatch`, `RecordBatchReader`, `pyarrow.dataset.Dataset` | |
| Hugging Face `datasets.Dataset` | Read through its indices mapping (see below) |
| Polars `DataFrame` | `LazyFrame` must be collected first |
| DuckDB relation | |
| Any object implementing `__arrow_c_stream__` | Arrow PyCapsule interface |
```python linenums="1" title="Profiling a Hugging Face dataset"
from datasets import load_dataset
from data_profiling import ProfileReport
dataset = load_dataset("fancyzhx/ag_news", split="train")
report = ProfileReport(dataset, title="AG News")
report.to_file("ag_news.html")
```
```python linenums="1" title="Profiling a DuckDB query"
import duckdb
from data_profiling import ProfileReport
relation = duckdb.sql("SELECT * FROM 'reviews.parquet' WHERE stars <= 2")
ProfileReport(relation).to_file("low_star_reviews.html")
```
The input is converted to a pandas DataFrame whose string columns stay in
Arrow memory (`pd.ArrowDtype`): text is not copied into Python objects, and
text statistics (lengths, characters, words) are computed with Arrow kernels.
Numeric, boolean, date and categorical columns are profiled exactly as for a
pandas input. The `vars.text.engine` setting controls this: `auto` (default)
uses Arrow kernels for Arrow-backed columns only, `arrow` uses them for every
text column, and `pandas` never does.
!!! note "Hugging Face datasets after `filter`, `shuffle` or `select`"
These operations keep the original table and add an indices mapping.
The report always profiles the rows the dataset exposes, which copies
the selected rows into memory once. An INFO log line reports how many.
Streaming (`IterableDataset`) Hugging Face datasets are not supported:
materialize a sample first, e.g. `Dataset.from_list(list(ds.take(100_000)))`.
## Other libraries
For other frameworks of the Python Data ecosystem, convert to a pandas
`DataFrame` first. Large datasets might require
sampling (as seen in our documentation on [how to profile large datasets](../features/big_data.md)).
```python linenums="1" title="Dask to Pandas"
# Convert dask DataFrame to a pandas DataFrame
df = df.compute()
```
```python linenums="1" title="Vaex to Pandas"
# Convert vaex DataFrame to a pandas DataFrame
df = df.to_pandas_df()
```
!!! note "Modin interface"
This is not part of the API as pandas.DataFrame, naturally, does not posses such a method.
You can use the private method DataFrame._to_pandas() to do this conversion.
If you would like to do this through the official API you can always save the Modin DataFrame to
storage (csv, hdf, sql, ect) and then read it back using Pandas. This will probably be the safer
way when working big DataFrames, to avoid out of memory issues."
Source: https://github.com/modin-project/modin/issues/896
```python linenums="1" title="Modin to Pandas"
# Convert modin DataFrame to pandas DataFrame
df = df._to_pandas()
```
---
### Integrations Pipelines (`docs/integrations/pipelines.md`)
# Pipelines
With Python, command-line and Jupyter interfaces, `data-profiling`
integrates seamlessly with DAG execution tools like Airflow, Dagster,
Kedro and Prefect, allowing it to easily becomes a building block of
data ingestion and analysis pipelines. Integration with
[Dagster](https://github.com/dagster-io/dagster) or
[Prefect](https://github.com/prefecthq/prefect) can be achieved in a
similar way as with Airflow.
## Airflow
Integration with Airflow can be easily achieved through the
[BashOperator](https://airflow.apache.org/docs/stable/_api/airflow/operators/bash_operator/index.html)
or the
[PythonOperator](https://airflow.apache.org/docs/stable/_api/airflow/operators/python_operator/index.html#airflow.operators.python_operator.PythonOperator).
``` python linenums="1" title="data-profiling with Airflow"
# Using the command line interface
profiling_task = BashOperator(
task_id="Profile Data",
bash_command="pandas_profiling dataset.csv report.html",
dag=dag,
)
```
``` python linenums="1" title="data-profiling with Airflow"
# Using the Python interface
import data_profiling
def profile_data(file_name, report_file):
df = pd.read_csv(file_name)
report = data_profiling.ProfileReport(df, title="Profiling Report in Airflow")
report.to_file(report_file)
return "Report generated at {}".format(report_file)
profiling_task2 = PythonOperator(
task_id="Profile Data",
op_kwargs={"file_name": "dataset.csv", "report_file": "report.html"},
python_callable=profile_data,
dag=dag,
)
```
## Kedro
There is a community created [Kedro
plugin](https://github.com/BrickFrog/kedro-pandas-profiling) available.
---
### Integrations Pyspark (`docs/integrations/pyspark.md`)
# ⚡ Pyspark
!!! note ""Spark support"
**Spark dataframes support** - Spark Dataframes profiling is available
from data-profiling version 4.0.0 onwards
Data Profiling is a core step in the process of developing AI solutions.
For small datasets, the data can be loaded into memory and easily
accessed with Python and pandas dataframes. However, for larger datasets
what can be done?
Big data engines, that distribute the workload through different
machines, are the answer. Particularly, Spark rose as one of the most
used and adopted engines by the data community. `data-profiling`
provides an ease-to-use interface to generate complete and comprehensive
data profiling out of your Spark dataframes with a single line of code.
Getting started
## Installing Pyspark for Linux and Windows
!!! tip
Ensure that you first install the system requirements (spark and java).
: - Go to [Download Java
JDK](https://www.oracle.com/java/technologies/javase-jdk13-downloads.html)
and download the Java Development Kit (JDK).
- Download and install a [Spark version bigger than
3.3](https://spark.apache.org/downloads.html)
- Set your environment variables
``` console
export SPARK_VERSION=3.3.0
export SPARK_DIRECTORY=/opt/spark
export HADOOP_VERSION=2.7
mkdir -p ${SPARK_DIRECTORY}
sudo apt-get update
sudo apt-get -y install openjdk-8-jdk
curl https://archive.apache.org/dist/spark/spark-${SPARK_VERSION}/spark-${SPARK_VERSION}-bin-hadoop${HADOOP_VERSION}.tgz \
--output ${SPARK_DIRECTORY}/spark.tgz
cd ${SPARK_DIRECTORY} && tar -xvzf spark.tgz && mv spark-${SPARK_VERSION}-bin-hadoop${HADOOP_VERSION} sparkenv
```
A more detailed tutorial for the installation can be found
[here](https://www.datacamp.com/tutorial/installation-of-pyspark).
## Installing Pyspark for MacOS
Use `Homebrew` to ensure that the system requirements are
installed (java and scala (optional))
```console
console brew
install 
---
### Reference History (`docs/reference/history.md`)
# History & community
The `data-profiling` project became what it is today due to the work of
the creators to make it successful. This page aims to highlight a bit
of the development history. For the full picture, have a look at the
[contributor
history](https://github.com/Data-Centric-AI-Community/data-profiling/graphs/contributors).
**[YData](https://ydata.ai/)** (acquired by KPMG in 2025) took over as the main maintainer
of the package in 2022, and is responsible for releases such as the support for
time-series, compare datasets and spark support.
## Thank you to our amazing contributors
A big thank you to all our amazing contributors!
---
### Reference Resources (`docs/reference/resources.md`)
# Resources
This page contains a series of community-contributed resources that can
be found throughout the web. Found, written or recorded a contribution?
Feel free to contribute it via a pull request on GitHub.
## Notebooks
- Google Cloud Platform: [Building a propensity model for financial
services on Google
Cloud](https://cloud.google.com/solutions/building-a-propensity-model-for-financial-services-on-gcp)
- Kaggle: [Notebooks using
ydata-profiling (previously cally pandas-profiling)](https://www.kaggle.com/search?q=ydata-profiling)
(100+ notebooks)
- Pycaret: [Intermediate Level Tutorials include
pandas-profiling](https://github.com/pycaret/pycaret)
- Google BigQuery integration Notebook: [Building a propensity model for financial services on Google Cloud](https://github.com/GoogleCloudPlatform/analytics-componentized-patterns/blob/master/retail/propensity-model/bqml/bqml_kfp_retail_propensity_to_purchase.ipynb)
## Articles
- [Pandas-profiling now supports spark](https://www.databricks.com/blog/2023/04/03/pandas-profiling-now-supports-apache-spark.html)
(Fabiana, Miriam and Corey, Apr 3, 2023)
- [How to an EDA on Time-series data](https://ydata.ai/resources/how-to-do-an-eda-for-time-series)
(Fabiana Clemente, Oct 21, 2022)
- [Evaluation of freely available data profiling tools for health data
research application: a functional evaluation
review](https://bmjopen.bmj.com/content/12/5/e054186.abstract/) (Ben
Gordon, May 9, 2022)
- [Generate Reports Using Pandas Profiling, Deploy Using
Streamlit](https://www.analyticsvidhya.com/blog/2021/06/generate-reports-using-pandas-profiling-deploy-using-streamlit/)
(Kaustubh Gupta, June 25, 2021)
- [Bringing Customization to Pandas
Profiling](https://medium.com/@ianeaves/customizing-pandas-profiling-summaries-b16714d0dac9)
(Ian Eaves, March 5, 2021)
- [Beginner Friendly Data Science Projects Accepting
Contributions](https://towardsdatascience.com/beginner-friendly-data-science-projects-accepting-contributions-3b8e26f7e88e)
(Adam Ross Nelson, January 18, 2021)
- [Pandas profiling and exploratory data analysis with line one of
code!](https://towardsdatascience.com/pandas-profiling-and-exploratory-data-analysis-with-line-one-of-code-423111991e58)
(Magdalena Konkiewicz, Jun 10, 2020)
- [The Covid 19 health
issue](https://concillier.squarespace.com/datasets/covid-19)
(Concillier Kitungulu, April 20, 2020)
- [Accelerate Your Exploratory Data Analysis With
Pandas-Profiling](https://towardsdatascience.com/accelerate-your-exploratory-data-analysis-with-pandas-profiling-4eca0cb770d1)
(Sukanta Roy, April 19, 2020)
- [Pandas-Profiling, explore your data faster in
Python](https://www.datacourses.com/pandas-1150/) (Gabriel Fuhr, May
31, 2020)
- [Pandas One Line Magical Code for EDA: Pandas Profile
Report](https://dzone.com/articles/pandas-one-line-magical-code-for-eda-pandas-profil)
(Sunil Kappal, July 8, 2019)
- [Speed Up Your Exploratory Data Analysis With
Pandas-Profiling](https://towardsdatascience.com/speed-up-your-exploratory-data-analysis-with-pandas-profiling-88b33dc53625)
(Lukas Frei, April 26, 2019)
- [Pandas Exploratory Data Analysis: Data Profiling with one single
command](https://kanoki.org/2019/01/15/pandas-exploratory-data-analysis-data-profiling/)
(Vinay Babu, January 15, 2019)
- [Data Profiling with
pandas-profiling](https://www.leehonan.com/data-profiling-with-pandas-profiling/)
(Lee Honan, October 28, 2018)
- [3 Must-have tools if you\'re serious about machine
learning](https://fizzylogic.nl/2018/08/21/5-must-have-tools-if-youre-serious-about-machine-learning/)
(Willem Meints, August 21, 2018)
- [Exploratory Data Analysis of Craft Beers: Data
Profiling](https://www.datacamp.com/community/tutorials/python-data-profiling)
(Jean-Nicholas Hould, April 13, 2017)
## Videos
- [How to install data-profiling with conda and Python 3.11](https://www.youtube.com/watch?v=fvXZcpTwbtA)
(Fabiana Clemente, June 2023, 2023)
- [Installing Anaconda, Creating a virtual environment and installing
pandas-profiling in it](https://www.youtube.com/watch?v=q2E8RLsznaA)
(Abhiram R - EverythingPython, Jan 25, 2022)
- [How to Install and Use Pandas Profiling on Google
Colab](https://www.youtube.com/watch?v=pLxgt20kKWU) (Chanin
Nantasenamat, Apr 25, 2020)
- [Data Exploration in Python with Just one line of code \| Python
Tutorial](https://www.youtube.com/watch?v=oSTOcqcI9n0) (Abhishek
Agarrwal, Nov 7, 2019)
- [Pandas Profiling Correlations \| pythonbeginner
2019](https://www.youtube.com/watch?v=0BzIY4_Ftlo) (August 5, 2019)
- [Automated Data Profiling using Python Pandas
(pandas-profiling)](https://www.youtube.com/watch?v=vsL8osE_0HM)
(Kunaal Naik, Oct 14, 2019)
- [Data Science Tools - Data Profiling with Pandas (Simple Exploratory
Data Analysis)](https://www.youtube.com/watch?v=C7cmeEvUDfo)
(Jesse E. Agbe, Jun 20, 2019)
- [Exploratory data analysis in python - PyCon
2017](https://www.youtube.com/watch?v=W5WE9Db2RLU) (Chloe Mawer,
Jonathan Whitmore, May 18, 2017)
## Books
- [Machine Learning Pocket
Reference](https://www.oreilly.com/library/view/machine-learning-pocket/9781492047537/)
by Matt Harrison, August 2019, ISBN: 9781492047544
- [Python Data Science
Essentials](https://www.packtpub.com/product/python-data-science-essentials-third-edition/9781789537864)
by Alberto Boschetti and Luca Massaron, September 2018, ISBN:
9781789537864
## Miscellaneous
- [Penn Machine Learning Benchmarks
(PMLB)](https://epistasislab.github.io/pmlb/)
([description](https://arxiv.org/ftp/arxiv/papers/2012/2012.00058.pdf))
- [dabl package](https://github.com/dabl/dabl)
---
### Support Contribution Common Issues (`docs/support-contribution/common_issues.md`)
# Common issues
## TypeError: \_plot_histogram() got an unexpected keyword argument \'title\'
This error occurs when using outdated versions of the package.
Ensure that you are using the latest version, and when in a notebook,
ensure that you\'ve restarted the kernel when needed. Also make sure
that you install in the right Python environment (please use
`!{sys.executable} -m pip install -U fg-data-profiling`!). More
information on installing Python packages directly from a notebook:
[\'Installing Python Packages from a Jupyter
Notebook\'](https://jakevdp.github.io/blog/2017/12/05/installing-python-packages-from-jupyter/).
Related GitHub issues:
- [\[950\]](https://github.com/Data-Centric-AI-Community/data-profiling/issues/950)
- [\[939\]](https://github.com/Data-Centric-AI-Community/data-profiling/issues/939)
- [\[528\]](https://github.com/Data-Centric-AI-Community/data-profiling/issues/528)
- [\[485\]](https://github.com/Data-Centric-AI-Community/data-profiling/issues/485)
- [\[396\]](https://github.com/Data-Centric-AI-Community/data-profiling/issues/396)
## Jupyter \"IntSlider(value=0)\"
When in a Jupyter environment, if only text such as `IntSlider(value=0)`
or
`(children=(IntSlider(value=0, description='x', max=1), Output()), _dom_classes=('widget-interact',))`
is shown, then the Jupyter Widgets are not activated. The
`../getting_started/installation`{.interpreted-text role="doc"} page
contains instructions on how to resolve this problem.
## MemoryError: Unable to allocate\... when profiling datasets with very large values
A memory error that comes up when profiling datasets with large outliers
(even if the dataset itself is small), which is due to an underlying bug
in `numpy`, used to build a histogram. Although some [workarounds are
suggested on numpy\'s
GitHub](https://github.com/numpy/numpy/issues/10297), the bug is not yet
fixed. One workaround is to filter out large outliers prior to report
computation.
Related StackOverflow questions:
- [MemoryError when using data_profiling
profile_report](https://stackoverflow.com/questions/67342168/memoryerror-when-using-pandas-profiling-profile-report)
---
### Support Contribution Contribution Guidelines (`docs/support-contribution/contribution_guidelines.md`)
# Contribution Guidelines
## Contributing a new feature
- Open a new GitHub pull request with the patch.
- Ensure the PR description clearly describes the problem and
solution. Include the relevant issue number if applicable.
## Development tooling
To ease the development cycle, some tools are available. These can be
called from the root directory with the `make` command.
The following commands are supported:
``` console
make lint
make install
make examples
make docs
make test
make clean
```
## Contribution quality standards
To guarantee a high quality of the contributed code, the project
workflow validates the added modification as well as the introduced
commit messages. The same mechanisms are used locally to find and solve
existing issues before submitting a pull request.
To activate the local mechanisms (created using pre-commit hooks), run the
following commands:
``` console
pip install ".[dev]"
pre-commit install --hook-type commit-msg --hook-type pre-commit
```
## Git workflow
The Git workflow used in this project is based on [this blog
post](https://nvie.com/posts/a-successful-git-branching-model/). Using
this workflow allows for better collaboration between contributors and
automation of repetitive tasks.
In addition to the workflow described in the blog post, Github Actions
lints the code automatically on the release branches and builds
documentation from each push to the master branch. For now, we don\'t
use hotfix branches.
**Branch naming:**
- develop: development branch
- master: master branch
- feat/\[FEATURE NAME\]: feature branches
- docs/\[branch name\]: documentation and examples
{width="60.0%"}
## Contributor License Agreement (CLA)
This package does not have a Contributor License Agreement (CLA), as the
GitHub Terms of Service provides a sensible [explicit
default](https://help.github.com/en/github/site-policy/github-terms-of-service#6-contributions-under-repository-license):
> *Whenever you make a contribution to a repository containing notice of
> a license, you license your contribution under the same terms, and you
> agree that you have the right to license your contribution under those
> terms.*
Read Github\'s [open source legal
guide](https://opensource.guide/legal/#does-my-project-need-an-additional-contributor-agreement)
for further details.
## :fontawesome-brands-discord: Discord Data-centric community
The community is low-threshold place to ask questions regarding
development and contributing. [Join the Discord
community](https://discord.com/invite/mw7xjJ7b7s).
## More information
Read more on getting involved in the [Contribution Guide available on
GitHub](https://github.com/Data-Centric-AI-Community/data-profiling/blob/master/CONTRIBUTING.md).
---
### Support Contribution Help Troubleshoot (`docs/support-contribution/help_troubleshoot.md`)
# Help & Troubleshooting
## Troubleshooting
To start troubleshooting, we need to trace the issue to a bug in the
code or to something else (such as your local environment). The first
step is to create a new environment with a fresh installation (see
[Installation guide](../getting-started/installation.md) for
instructions). In many cases, the problem will be resolved by this step.
If the problem can be replicated in the new environment, then it likely
is a software bug. Before proceeding, check
`common_issues`{.interpreted-text role="doc"} to check whether it is a
previously identified common issue.
## Reporting a bug
To ensure the bug was not already reported by searching on Github under
[Issues](https://github.com/Data-Centric-AI-Community/data-profiling/issues). If you\'re
unable to find an open issue addressing the problem, [open a new
one](https://github.com/Data-Centric-AI-Community/data-profiling/issues/new/choose). If
possible, use the relevant bug report templates to create the issue.
You should provide the **minimal information to reproduce this bug**.
[This
guide](http://matthewrocklin.com/blog/work/2018/02/28/minimal-bug-reports)
can help in crafting a minimal bug report. Please include:
- The minimal code you are using to generate the report
- Version information is essential in reproducing and resolving bugs.
Include relevant environment details such as:
> - operating system (e.g. Windows, Linux, Mac)
> - Python version (e.g. `3.7`)
> - Interface: Jupyter notebook (or cloud services like Google
> Colab, Kaggle Kernels, etc), console or IDE (such as
> PyCharm,VS Code,etc)
> - package manager (e.g. `pip --version` or `conda info`)
> - packages (`pip freeze > packages.txt` or `conda list`). Please
> make sure this is contained in a collapsed section
> (instructions below)
- a sample of the dataset (`df.sample()` or `df.head()`). If the
dataset is confidential, for example when it contains
company-sensitive information, provide us with a synthetic or open
dataset that produces the same error. You can anonymize the column
names if necessary.
- a description of the dataset and its structure, for example by
reporting the DataFrame\'s structure through the output of
`df.info()`.
## Issue formatting
To craft helpful and easily readable issues, two formatting tricks are
recommended:
- Code highlighting: wrap all code and error messages in fenced
blocks, and in particular add the language identifier. Check the
[Github docs on highlighting code
blocks](https://docs.github.com/en/get-started/writing-on-github/working-with-advanced-formatting/creating-and-highlighting-code-blocks)
for details.
- Collapsed sections: organize long error messages and requirement
listings in collapsed sections. The [Github docs on collapsed
sections](https://docs.github.com/en/get-started/writing-on-github/working-with-advanced-formatting/organizing-information-with-collapsed-sections)
provide detailed information.
## Using Stack Overflow
Users with a request for help on how to use `data-profiling` should
consider asking their question on Stack Overflow, under the dedicated
`data-profiling` tag:
[](https://stackoverflow.com/questions/tagged/data-profiling) or,
[](https://stackoverflow.com/questions/tagged/pandas-profiling)
for questions about `data-profiling` older versions.
## :fontawesome-brands-discord: Discord community
[Join the Discord community](https://discord.com/invite/mw7xjJ7b7s) to
connect with both other users and developers that might be able to
answer your questions. The **#data-profiling** and **#need-help**
channels are recommended for questions and issues.
---
--- METRICS ---
- Files Extracted: 30
- Estimated Token Budget: ~30821 tokens
- Canonical Reference: https://codewiki.google/github.com/Data-Centric-AI-Community/data-profiling