mlarena

KO | EN

gitlite — search

#javascript #python #hacktoberfest #ai #android #docker #nodejs #macos #windows #kubernetes #awesome-list #security #ios #openai

(★ 38)

Algorithm-agnostic machine learning toolkit for strealined model training, diagnostics, and optimization.

mlarena Latest Version Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# MLArena

[![Python Version](https://img.shields.io/badge/python-3.10%20%7C%203.11%20%7C%203.12%20%7C%203.13-blue.svg)](https://www.python.org/downloads/)
[![PyPI](https://img.shields.io/pypi/v/mlarena.svg)](https://pypi.org/project/mlarena/)
[![Downloads](https://img.shields.io/pepy/dt/mlarena)](https://pepy.tech/project/mlarena)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
[![Tests](https://img.shields.io/badge/tests-pytest-blue.svg)](https://pytest.org/)
[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
[![Imports: isort](https://img.shields.io/badge/%20imports-isort-%231674b1?style=flat&labelColor=ef8336)](https://pycqa.github.io/isort/)
[![CI/CD](https://github.com/MenaWANG/mlarena/actions/workflows/mlarena.yml/badge.svg)](https://github.com/MenaWANG/mlarena/actions/workflows/mlarena.yml)

`mlarena` is an algorithm-agnostic machine learning toolkit for streamlined model training, diagnostics, and optimization. Implemented as a custom `mlflow.pyfunc` model, it ensures seamless integration with the MLflow ecosystem for robust experiment tracking, model versioning, and framework-agnostic deployment.

It blends smart automation that embeds ML best practices with comprehensive tools for expert-level customization and diagnostics. This unique combination fills the gap between manual ML development and fully automated AutoML platforms. Moreover, it comes with a suite of practical utilities for data analysis and visualizations - see our [comparison with AutoML platforms](https://github.com/MenaWANG/mlarena/blob/master/docs/comparision-autoML.md) to determine which approach best fits your needs.

## Publications

For a quick guide over the package

- [Build Algorithm-Agnostic ML Pipelines in a Breeze](https://towardsdatascience.com/build-algorithm-agnostic-ml-pipelines-in-a-breeze/) - Published in Towards Data Science

> This article discussed some key challenges in algorithm-agnostic ML Pipeline building and demonstractes how MLarena can help to address them. Although more functionalities have been added after the publication of the article on 7 July 2025, it is nonetheless a good overview of MLarena's core functionalies and a good quick guide for starting with the package.

On specific functionalities

- [Help Your Model Learn the True Signal](https://towardsdatascience.com/help-your-model-learn-the-true-signal/) - Published in Towards Data Science

> This article explores the challenges of investigating data points that disproportionately disrupt a model's ability to learn the dominant signal. By leveraging an algorithm-agnostic approach inspired by Cook's Distance, it provides a method to effectively identify and diagnose these disruptive data points, ensuring that models capture stable, generalizable patterns. This technique is implemented as a helper function `calculate_cooks_d_like_influence` in the MLarena package, and is compatiable with any sklearn style ML algorithms.

- [When a Difference Actually Makes a Difference](https://towardsdatascience.com/when-a-difference-actually-makes-a-difference/) - Published in Towards Data Science

> This article offers actional steps for business decision makers to analyze group differences and avoid costly data misinterpretations. It demonstrates how three distinct business realities can hide behind a single bar chart, and offers a method to reveal the full story for more informed decision-making. This technique, including the visualization and statistical testing, is implemented as a helper function `plot_box_scatter` in the MLarena package.

The concepts and methodologies behind MLArena:

- [Algorithm-Agnostic Model Building with MLflow](https://medium.com/data-science/algorithm-agnostic-model-building-with-mlflow-b106a5a29535?source=friends_link&sk=a43a376c46116970cd983420cfd68afe) - Published in Towards Data Science
   > A foundational guide demonstrating how to build algorithm-agnostic ML pipelines using mlflow.pyfunc. The article explores creating generic model wrappers, encapsulating preprocessing logic, and leveraging MLflow's unified model representation for seamless algorithm transitions.

- [Explainable Generic ML Pipeline with MLflow](https://medium.com/data-science/explainable-generic-ml-pipeline-with-mlflow-2494ca1b3f96?source=friends_link&sk=ebe917c37719516a5b5170efc1bd0b32) - Published in Towards Data Science
   > An advanced implementation guide that extends the generic ML pipeline with more sophisticated preprocessing and SHAP-based model explanations. The article demonstrates how to build a production-ready pipeline that supports both classification and regression tasks, handles feature preprocessing, and provides interpretable model insights while maintaining algorithm agnosticism.

## Installation

The package is undergoing rapid development at the moment (pls see [CHANGELOG](https://github.com/MenaWANG/mlarena/blob/master/CHANGELOG.md) for details), it is therefore highly recommended to install with specific versions. For example

```bash
%pip install mlarena==0.5.2
```

If you are using the package in [Databricks ML Cluster with DBR runtime >= 16.0](https://learn.microsoft.com/en-us/azure/databricks/release-notes/runtime/16.0ml), you can install without dependencies like below:

```bash
%pip install mlarena==0.5.2 --no-deps
```
If you are using earlier DBR runtimes, simply install `optuna` in addition like below. Note: As of 2025-04-26, `optuna` is recommended by Databricks, while `hyperopt` will be [removed from Databricks ML Runtime](https://docs.databricks.com/aws/en/machine-learning/automl-hyperparam-tuning/).

```bash
%pip install mlarena==0.5.2 --no-deps
%pip install optuna==3.6.1
```

## Usage Example

* For quick start with a basic example, see [1.basic_usage.ipynb](https://github.com/MenaWANG/mlarena/blob/master/examples/1.basic_usage.ipynb).   
* For more advanced examples on model optimization, see [2.advanced_usage.ipynb](https://github.com/MenaWANG/mlarena/blob/master/examples/2.advanced_usage.ipynb).   
* For visualization utilities, see [3.utils_plot.ipynb](https://github.com/MenaWANG/mlarena/blob/master/examples/3.utils_plot.ipynb).
* For data cleaning and manipulation utilities, see [3.utils_data.ipynb](https://github.com/MenaWANG/mlarena/blob/master/examples/3.utils_data.ipynb).
* For statistical analysis utilities, see [3.utils_stats.ipynb](https://github.com/MenaWANG/mlarena/blob/master/examples/3.utils_stats.ipynb).
* For input/output utilities, see [3.utils_io.ipynb](https://github.com/MenaWANG/mlarena/blob/master/examples/3.utils_io.ipynb)
* For handling common challenges in machine learning, see [4.ml_discussions.ipynb](https://github.com/MenaWANG/mlarena/blob/master/examples/4.ml_discussions.ipynb).

## Visual Examples:

### Quick Start
Train and evaluate models quickly with `mlarena`'s default preprocessing pipeline, comprehensive reporting, and model explainability. The framework handles the complexities behind the scenes, allowing you to focus on insights rather than boilerplate code. See [1.basic_usage.ipynb](https://github.com/MenaWANG/mlarena/blob/master/examples/1.basic_usage.ipynb) for complete examples.

| Category | Classification Metrics & Plots | Regression Metrics & Plots |
|-------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Metrics | **Evaluation Parameters** • Threshold (classification cutoff) • Beta (F-beta weight parameter) **Core Performance Metrics** • Accuracy (overall correct predictions) • Precision (true positives / predicted positives) • Recall (true positives / actual positives) • F1 Score (harmonic mean of Precision & Recall) • Fβ Score (weighted harmonic mean, if β ≠ 1) • AUC (ranking quality) • MCC (Matthews Correlation Coefficient) • Log Loss (confidence-weighted error) • Brier Score (probability calibration quality) **Prediction Distribution** • Positive Rate (fraction of positive predictions) • Base Rate (actual positive class rate) | **Error Metrics** • RMSE (Root Mean Squared Error) • MAE (Mean Absolute Error) • Median Absolute Error • NRMSE (Normalized RMSE as % of) <ul style="margin-top:0;margin-bottom:0;"><li>mean</li><li>std</li><li>IQR</li></ul>• MAPE (Mean Absolute Percentage Error, excl. zeros) • SMAPE (Symmetric Mean Absolute Percentage Error) **Goodness of Fit** • R² (Coefficient of Determination) • Adjusted R² **Improvement over Baseline** • RMSE Improvement over Mean Baseline (%) • RMSE Improvement over Median Baseline (%) |
| Plots | • Metrics vs Threshold (Precision, Recall, Fβ, with vertical threshold line) • ROC Curve • Confusion Matrix (with colored overlays) | • Residual analysis (residuals vs predicted, with 95% prediction interval) • Prediction error plot (actual vs predicted, with perfect prediction line and error bands) |

#### Classification Models

![Classification Model Performance](https://raw.githubusercontent.com/MenaWANG/mlarena/master/docs/images/model_performance_classification.png)

#### Regression Models

![Regression Model Performance](https://raw.githubusercontent.com/MenaWANG/mlarena/master/docs/images/model_performance_regression.png)

### Explainable ML
One liner to create global and local explanation based on SHAP that will work across various classification and regression algorithms.

![Global Explanation](https://raw.githubusercontent.com/MenaWANG/mlarena/master/docs/images/global_explanation.png)

![Local Explanation](https://raw.githubusercontent.com/MenaWANG/mlarena/master/docs/images/local_explanation.png)

![Dependence Plot](https://raw.githubusercontent.com/MenaWANG/mlarena/master/docs/images/dependence_plot.png)

### Hyperparameter Optimization
`mlarena` offers iterative hyperparameter tuning with cross-validation for robust results and parallel coordinates visualization for search space diagnostics. See [2.advanced_usage.ipynb](https://github.com/MenaWANG/mlarena/blob/master/examples/2.advanced_usage.ipynb) for more.

![Hyperparameter Search Space](https://raw.githubusercontent.com/MenaWANG/mlarena/master/docs/images/parallel_coordinates.png)

### Plotting Utility Functions

`mlarena` offers handy utility visualizations for data exploration. Pls see [3.utils_plot.ipynb](https://github.com/MenaWANG/mlarena/blob/master/examples/3.utils_plot.ipynb) for more.

#### `plot_box_scatter` for comparing numerical distributions across categories with optional statistical testing
![plot_box_scatter](https://raw.githubusercontent.com/MenaWANG/mlarena/master/docs/images/plot_box_scatter.png)

#### `plot_distribution_over_time` for comparing numerical distributions over time
![plot_distribution_over_time](https://raw.githubusercontent.com/MenaWANG/mlarena/master/docs/images/plot_distribution_over_time.png)

#### `plot_distribution_over_time` color points using `point_hue`
![plot_distribution_over_time_point_hue](https://raw.githubusercontent.com/MenaWANG/mlarena/master/docs/images/plot_distribution_over_time_point_hue.png)

#### `plot_stacked_bar_over_time` for comparing categorical distributions over time
![plot_stacked_bar_over_time](https://raw.githubusercontent.com/MenaWANG/mlarena/master/docs/images/plot_stacked_bar_over_time.png)

#### `plot_metric_event_over_time` for timeseries trends and events
![plot_metric_event_over_time](https://raw.githubusercontent.com/MenaWANG/mlarena/master/docs/images/plot_metric_event_over_time.png)

### Data Utilities

Some handy utilities for data validation, cleaning and manipulations. Pls see [3.utils_data.ipynb](https://github.com/MenaWANG/mlarena/blob/master/examples/3.utils_data.ipynb) for more.

#### `is_primary_key`
![is_primary_key](https://raw.githubusercontent.com/MenaWANG/mlarena/master/docs/images/is_primary_key_demo.png)

## Features

**Algorithm Agnostic ML Pipeline**
- Unified interface for any scikit-learn compatible model
- Consistent workflow across classification and regression tasks
- Automated report generation with comprehensive metrics and visuals
- Built on `mlflow.pyfunc` for seamless MLOps integration
- Automated experiment tracking of parameters, metrics, and models
- Simplified handoff from experimentation to production via the MLflow framework

**Intelligent Preprocessing**
- Streamlined feature preprocessing with smart defaults and minimal code
- Automatic feature analysis with data-driven encoding recommendations 
- Integrated target encoding with visualization for optimal smoothing selection
- Feature filtering based on information theory metrics (mutual information)
- Intelligent feature name sanitization to prevent pipeline failure
- Handles the full preprocessing pipeline from missing values to feature encoding
- Seamless integration with scikit-learn and MLflow for production deployment

**Model Optimization**
- Efficient hyperparameter tuning with Optuna's TPE sampler
- Smart early stopping with patient pruning to save computation resources
  - Configurable early stopping parameter
  - Startup trials before pruning begins
  - Warmup steps per trial
- Cross-validation with variance penalty to prevent overfitting
- Parallel coordinates visualization for search history tracking and parameter space diagnostics
- Automated threshold optimization with business-focused F-beta scoring
  - Cross-validation or bootstrap methods
  - Configurable beta parameter for precision/recall trade-off
  - Confidence intervals for bootstrap method
- Flexible metric selection for optimization
  - Classification: AUC (default), F1, accuracy, log_loss, MCC
  - Regression: RMSE (default), MAE, median_ae, SMAPE, NRMSE (mean/std/IQR)

**Performance Analysis**
- Comprehensive metric tracking
  - Classification: AUC, F1, Fbeta, precision, recall, accuracy, log_loss, MCC, positive_rate, base_rate
  - Regression: RMSE, MAE, median_ae, R2, adjusted R2, MAPE, SMAPE, NRMSE (mean/std/IQR), improvement over mean/median baselines
- Performance visualization
  - Classification: 
    - Metrics vs Threshold plot (precision, recall, F-beta)
    - ROC curve with AUC score
    - Confusion matrix with color-coded cells
  - Regression:
    - Residual analysis with 95% prediction intervals
    - Prediction error plot with perfect prediction line and error bands
- Model interpretability
  - Global feature importance
  - Local prediction explanations

**Utils**
- Advanced plotting utilities
  - Distribution analysis
    - `plot_box_scatter`: Box plots with scatter overlay and statistical testing
      - Optional point coloring by category
      - Built-in statistical tests (ANOVA, Welch, Kruskal)
      - Customizable point jittering and transparency
  - Time series visualization
    - `plot_metric_event_over_time`: Track metrics with event annotations
      - Support for dual y-axes
      - Automatic date formatting based on time range
      - Customizable event markers and labels
      - Min/max value annotations
    - `plot_distribution_over_time`: Track distributions over time
      - Box plots with scatter overlay for each time period
      - Optional point coloring by category
      - Flexible time aggregation (hourly/daily/monthly/yearly)
      - Summary statistics for each period
    - `plot_stacked_bar_over_time`: Track categorical distributions
      - Percentage or count-based stacking
      - Flexible time aggregation
      - Custom category labeling
      - Automatic legend and color management
  - Categorical analysis
    - `plot_stacked_bar`: Stacked bar charts with statistical testing
      - Built-in chi-square and G-tests
      - Percentage or count-based visualization
      - Custom category labeling
- Statistical analysis utilities
  - Group comparison tools
    - `compare_groups`: Statistical testing across groups with visualization
    - `add_stratified_groups`: Create balanced groups with stratification
    - `optimize_stratification_strategy`: Find optimal stratification columns
  - Power analysis and sample size calculation
    - `power_analysis_numeric`: Power calculation for t-tests
    - `power_analysis_proportion`: Power calculation for proportion tests
    - `sample_size_numeric`: Required sample size for numeric outcomes
    - `sample_size_proportion`: Required sample size for proportion tests
    - `numeric_effectsize`: Cohen's d effect size calculation
- Data manipulation tools
  - Standardized dollar amount cleaning for financial analysis
  - Value counts with percentage calculation for categorical analysis
  - Smart date column transformation with flexible format handling
  - Schema and data quality utilities
    - Primary key validation with detailed diagnostics
    - Alphabetically sorted schema display
    - Safe column selection with case sensitivity options
    - Automatic removal of fully null columns
  - Complete Duplicate Management Workflow: "Discover → Investigate → Resolve":
    - Use `is_primary_key` to discover the existance of duplication issues
    - Use `find_duplicates` to analyze duplicate patterns
    - Use `deduplicate_by_rank` to intelligently resolve duplicates with business logic
- I/O utilities
  - `save_object`: Store Python objects to disk with customizable options
    - Support for pickle and joblib backends
    - Optional date stamping in filenames
    - Compression support for joblib backend
  - `load_object`: Retrieve Python objects with automatic backend detection
    - Seamless loading regardless of storage format
    - Direct compatibility with paths returned by `save_object`

## Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

## License

This project is licensed under the MIT License - see the LICENSE file for details.

mlarena

mlarena Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

mlarena

mlarena Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Download README