NBprocessing

KO | EN

gitlite — search

#javascript #python #hacktoberfest #react #ai #typescript #llm #go #golang #android #machine-learning #rust #deep-learning #linux

★ 20 Open GitHub ↗

Pre-processing database using pre-written functions

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# NBprocessing

**Pre-processing database using pre-written functions**

**Written by**: Nir Barazida

**The [Medium blog-post](https://medium.com/@nirbarazida/the-ultimate-python-package-to-pre-process-data-for-machin-learning-c87bcc39fa66)**

**The full package usage is elaborated in this [Jupyter Notebook](https://github.com/nirbarazida/NBprocessing/blob/master/documentation/NBprocessing%20Package%20-%20Showcase.ipynb)**

### Usage Incentives

How many times have you received raw database and conduct the same action to pre-process it?
 - Check for missing values and fill them if necessary.
 - Combine low appearance categories under one umbrella category. 
 - Plot your data distribution and check for outliers.
 - Drop outliers by different methods.
 
 If you said yes to all of the above you have reached to right place!
 
The main purpose of the '**NBprocessing**' package is to make our Data Scientist life **easier**, or better yet - more efficient.\
The '**NBprocessing**' package stores most of the generics functions that we all use on a daily basic, such remove outliers, fill missing values etc.

## install and usage

Run from your command line prompt:
 
`pip install NBprocessing`
 
 It will also install all the dependent packages such pandas, numpy, seaborn etc.

## General

### package libraries
- Categorical - contains functions that are relevant to categorical features:

- `remove_categories(database, column_name, categories_to_drop)`
    - `fill_na_by_ratio(database, column_name)`
    - `combine_categories(database, column_name, category_name="other", threshold=0.01)`
    - `categories_not_in_common(train, test, column_name)`
    - `category_ratio(database, columns_to_check=None, num_categories=5)`
    - `label_encoder_features(database, features_to_encode)`
    - `OHE(database, features_list=None)` 
    
- Continuous - contains functions that are relevant to continuous features:

- `remove_outliers_by_boundaries(database, column_name, bot_qu, top_qu)` 
    - `fill_na_timedate(database, column_name)`
    - `get_num_outliers_by_value(database, filter_dict_up, filter_dict_down)`
    - `remove_outliers_by_value(database, filter_dict_up, filter_dict_down)`
    
- General - contains general functions:

- `missing_values(database)`
    - `split_and_check(database, column_name, test_size=0.3)`
    
- Plot - contains plots functions:

- `plot_missing_value_heatmap(database)`
    - `plot_corr_heat_map(database)`
    - `count_plot(database, column_list=None)`
    - `distribution_plot(database, column_list=None)`
    - `world_map_plot(database, locations_column, feature, title=None, color_bar_title=None)`

### import
- `from NBprocessing import NBcategorical`
- `from NBprocessing import NBcontinuous`
- `from NBprocessing import NBplot`
- `from NBprocessing import NBgeneral`

## Usage

### All usage of the package functions are reviewed very specifically in this [jupyter Notebook](https://github.com/nirbarazida/NBprocessing/blob/master/documentation/README_Notebook.ipynb)

### Basic Examples

- Categorical:
    - **Fill missing values in a categorical feature by the ratio of the categories**:\
    `fill_na_by_ratio(database, column_name)`
    
        Fill all missing values in the given column by the ratio of the categories in the column.\
        Because the ratio sum is not a perfect one - the extra missing values will be filled with the most common category in the column.
        
       - First, we would like to sum all missing values in every categorical feature.
       
            ![pic categorical 1](https://github.com/nirbarazida/NBprocessing/blob/master/documentation/readme_figures/categorical_1.png) 
       
       - Second, Lets explore the ratio of evey category in the feature 'fuel' with and without missing values
       
            ![pic categorical 1](https://github.com/nirbarazida/NBprocessing/blob/master/documentation/readme_figures/categorical_2.png) 
       
       - Last, we would like to fill the missing values and to keep the ratio of the features without them.\
       To do so we will use the `fill_na_by_ratio` function.
       
            ![pic categorical 3](https://github.com/nirbarazida/NBprocessing/blob/master/documentation/readme_figures/categorical_3.png)
             
       As we can see from the above, all the missing values were filled and we manged to keep the ratio of the categories.
      
    - **Combine low appearance categories under one category**:\
     `combine_categories(database, column_name, category_name="other", threshold=0.01)`
     
        Receives a threshold that is the minimum relative part of the category within the column.\
        All categories that are less than this threshold will be combined under the same category
        under the name 'category_name'.\
        The method will return a list with all the name of categories that were combined under 'category_name'.\
        With this list the user will be able to make the same action on the test set (assuming that the data
        was already splitted to train and test sets)

- First, we will check the ratio of appearance per etch category in the feature.
       
            ![pic categorical 4](https://github.com/nirbarazida/NBprocessing/blob/master/documentation/readme_figures/categorical_4.png)
            
       - Second, we would like to combine all the low appearance categories under one category and save the redacted features to a list.
       
            ![pic categorical 5](https://github.com/nirbarazida/NBprocessing/blob/master/documentation/readme_figures/categorical_5.png)
            
       - Last, we will make the same changes on our test data-set using the list that we've created:
       
            ![pic categorical 6](https://github.com/nirbarazida/NBprocessing/blob/master/documentation/readme_figures/categorical_6.png)
                
- Continuous:
    - **Remove outliers by top and bottom percentage of data boundaries**: 
     `remove_outliers_by_boundaries(database, column_name, bot_qu, top_qu)`
    
        The theory behind it:\
            _the number of outliers  will follow a binomial distribution with parameter p, which can generally be
            well-approximated by the Poisson distribution with λ = pn. Thus if one takes a normal distribution with
            cutoff 3 standard deviations from the mean, p is approximately 0.3%, and thus for 1000 trials one can
            approximate the number of samples whose deviation exceeds 3 sigmas by a Poisson distribution with λ = 3.
            3 times of standard deviation as my main data and out of this range would be the outlier._
            
         ![Image of Yaktocat](https://i.stack.imgur.com/AxYue.png)
    
        Thus, in a normal distribution the top and bottom boundaries should contain 99.7% of the data.
        However, not all data has Normal distribution thus the user is able to change the top and bottom boundaries
    
        Before removing the indexes will print a message to the user with the number of indexes that
        will be remove and the percent of the database that will be lost.
        the user will input 'y'(yes) to pressed and 'n'(no) to cancel the action.
        If the user choose 'yes' the method will continue to drop the indexes and will
        plot the new database shape.
        
        Let's see a live example:
    
    ![Continuous 1](https://github.com/nirbarazida/NBprocessing/blob/master/documentation/readme_figures/continuous_1.png)
        
- Plot:
    - Plot heat map of missing values
     `plot_missing_value_heatmap(database)`
     
        Plots a heat-map with all columns filled with green color.\
        The missing values indexes will be shown by a line in a gray color in their position.
        
        ![plot_1](https://github.com/nirbarazida/NBprocessing/blob/master/documentation/readme_figures/plot_1.png)

NBprocessing

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README