SynEval

KO | EN

gitlite — search

#python #ai #typescript #go #machine-learning #rust #nodejs #java #awesome #kubernetes #vue #security #pytorch #ios

(★ 11)

Comprehensive evaluation tools for synthetic data.

SynEval 최신버젼 다운로드

최종 버전 다운로드 (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# SynEval: Synthetic Data Evaluation Framework

SynEval is a comprehensive evaluation framework for assessing the quality of synthetic data generated by Large Language Models (LLMs). The framework provides quantitative scoring across four key dimensions:

- **Fidelity**: Measures how well the synthetic data preserves the statistical properties and patterns of the original data
- **Utility**: Evaluates the usefulness of synthetic data for downstream tasks
- **Diversity**: Assesses the variety and uniqueness of the generated data
- **Privacy**: Analyzes the privacy protection level of the synthetic data

## Installation

1. Clone the repository:

```bash
git clone https://github.com/privacy-enhancing-technologies/SynEval.git
cd SynEval
```

2. Create and activate a conda virtual environment:

```bash
conda create -n syneval python=3.10
conda activate syneval
```

3. Install dependencies:

```bash
pip install -r requirements.txt
```

4. Download NLTK data (required for text processing):

```bash
python -m nltk.downloader punkt punkt_tab stopwords
```

**Note**: You may see dependency conflict warnings during installation. This is normal in environments like Google Colab or when other packages are already installed. For a clean installation without conflicts, consider using a virtual environment.

## Large Data Files

Due to GitHub file size limits, the following datasets are hosted on Google Drive:

**[Download Data from Google Drive](https://drive.google.com/drive/folders/1xayL66wHnpbuMdzn0-XV2eUVeiCyEEqr)**

The Drive contains:

| File / Directory | Description | Size |
|---|---|---|
| `PST Code Submission/Data/Amazon/Amazon_Fashion.csv` | Amazon Fashion reviews dataset | 580 MB |
| `PST Code Submission/Data/Kiva Crowdfunding/kiva_loans.csv` | Kiva crowdfunding loans dataset | 187 MB |
| `PST Code Submission/Data/Fake Job Postings.csv` | Fake job postings dataset | 48 MB |
| `PST Code Submission/Data/Kiva Crowdfunding/loan_theme_ids.csv` | Kiva loan theme IDs | 30 MB |
| `PST Code Submission/Data/Amazon/real_100k.csv` | Amazon reviews sample (100k rows) | 23 MB |
| `PST Code Submission/SynGen/experiments/` | Experiment outputs and synthetic data generated by SynGen | ~450 MB |

After downloading, place the files in the corresponding paths under `PST Code Submission/`.

## Quick Start

### Command Line Usage

After installation, you can use SynEval from the command line. The main entry point for the framework is `run.py`. This script allows you to evaluate synthetic data against original data using various metrics.

The general command format is:

```bash
python run.py --synthetic <synthetic_data.csv> --original <original_data.csv> --metadata <metadata.json> [evaluation_flags] [--output <results.json>]
```

#### Required Arguments

- `--synthetic`: Path to the synthetic data CSV file
- `--original`: Path to the original data CSV file
- `--metadata`: Path to the metadata JSON file

#### Evaluation Flags

You can select one or more evaluation dimensions to run:

- `--fidelity`: Run fidelity evaluation
- `--utility`: Run utility evaluation
- `--diversity`: Run diversity evaluation
- `--privacy`: Run privacy evaluation

#### Optional Arguments

- `--output`: Path to save evaluation results in JSON format. If not specified, results will be printed to stdout. (Default: `artifacts/reports/evaluation_results.json`)
- `--plot`: Generate plots for all evaluation metrics and save them to the `artifacts/plots` directory. Plots visualize key metrics from fidelity, utility, diversity, and privacy evaluations.
- `--html`: Generate an HTML dashboard summarizing each evaluation dimension (saved to `artifacts/html/syneval_dashboard.html` by default).

#### Device Selection Arguments

- `--device`: Device to use for computation (`auto`, `cpu`, `cuda`). Default: `auto` (automatically detect best available device)
- `--force-cpu`: Force CPU usage even if GPU is available (overrides `--device`)
- `--gpu-memory-fraction`: Fraction of GPU memory to use (0.0-1.0, default: 0.8)

**Example:**

```bash
python run.py \
    --synthetic data/gpt4.1_synthetic.csv \
    --original data/real_10k.csv \
    --metadata data/metadata.json \
    --dimensions fidelity utility diversity privacy \
    --utility-input text \
    --utility-output rating \
    --output artifacts/reports/results.json \
    --plot \
    --html \
    --device auto
```

**For GPU acceleration (if available):**

**For CPU-only processing:**

All artifacts generated by `run.py` are stored under the `artifacts/` directory:

- `artifacts/cache`: cached intermediate computations
- `artifacts/plots`: matplotlib/seaborn plots
- `artifacts/html`: HTML dashboards (including privacy visualizations)
- `artifacts/reports`: JSON summaries and other text outputs

### Metadata Format

The metadata file should be a JSON file that describes the structure of your data. It should include column names, types, dataset name, and primary key information. See `data/metadata.json` for a concrete example.

## Evaluation Dimensions

Detailed documentation for each evaluation module is located under `evaluation/descriptions/`:

- [Fidelity](evaluation/descriptions/fidelity.md)
- [Utility](evaluation/descriptions/utility.md)
- [Diversity](evaluation/descriptions/diversity.md)
- [Privacy](evaluation/descriptions/privacy.md)

## Additional Tools

The `use_cases/` directory contains scenario-focused extensions that demonstrate how SynEval can be adapted to solve concrete problems beyond the core evaluation CLI (e.g., NER analysis, differential privacy dashboards).

## Contributing

As we implement more evaluation metrics, this README will be updated with additional documentation for each component.

## License

This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.

SynEval

SynEval 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

SynEval

SynEval 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

Download README