BioEval

KO | EN

gitlite — search

#javascript #hacktoberfest #react #typescript #machine-learning #rust #deep-learning #linux #windows #cli #kubernetes #awesome-list #vue #open-source

(★ 11)

A self-hosted web platform for benchmarking LLMs on medical NLP tasks.

BioEval Latest Version Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

<h1>BioEval</h1>

<p>Benchmark LLMs on medical NLP — from setup to statistical comparison, in one platform.</p>

</div>

<br />

Evaluating LLMs on medical tasks usually means writing one-off scripts, wrangling provider APIs, and losing track of results across spreadsheets. BioEval is a self-hosted web platform that handles prompt engineering, inference, postprocessing & evaluation, head-to-head comparison, and statistical analysis in one place.

### 📊 Built-in Benchmarks
12 curated medical datasets covering QA, NER, relation extraction, classification, and summarization — including PubMedQA, MedQA, BC5CDR, and ChemProt.

### 🔀 One Workspace for the Entire Workflow
Configure prompts, run inference across Azure OpenAI, Google Gemini, Anthropic, and local GPU models, auto-evaluate with 10+ metrics, and compare results side by side with statistical significance tests — all without leaving the browser.

### 📁 Bring Your Own Benchmark
Upload custom datasets and evaluate them with the same pipeline and metrics.

<div align="center">
  <h4>Benchmark Dashboard</h4>
  <img src="docs/screenshot-inferences.png" width="800" alt="Benchmark Dashboard" />
  <br /><br />
  <h4>Head-to-Head Comparison & Statistical Analysis</h4>
  <img src="docs/screenshot-evaluation.png" width="800" alt="Head-to-Head Comparison and Statistical Analysis" />
</div>

---

## Prerequisites

| Requirement | Notes |
|---|---|
| [Docker Desktop](https://www.docker.com/products/docker-desktop/) (macOS) or [Docker Engine](https://docs.docker.com/engine/install/) + [Compose plugin](https://docs.docker.com/compose/install/) (Linux) | Required for the supported platforms below |
| NVIDIA GPU + [NVIDIA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html) | Required only for local NVIDIA GPU inference (**Linux only**). Cloud providers (Azure OpenAI, Google Gemini, Anthropic) work without a GPU. |

---

## Quick Start

```bash
git clone https://github.com/Yale-BIDS-Chen-Lab/BioEval.git
cd BioEval/docker-files
cp .env.example .env
docker compose up --build          # first build takes a few minutes; add -d to run in background
```

Open **http://localhost:3000**, create an account, and add an integration (Azure OpenAI / Google Gemini / Anthropic API key) under **Settings**.

`.env.example` ships working `localhost` defaults, so the command above boots as-is for local use. It runs the **production build** (compiled, no hot reload).

**Before any shared or remote deployment**, edit `.env` and replace the secrets — at minimum `BETTER_AUTH_SECRET`. The backend **refuses to start** if it is missing, shorter than 16 characters, or left as the old dev default, so if you reused an existing `.env` and the backend won't come up, set a fresh secret:

```bash
openssl rand -base64 32            # paste the output into BETTER_AUTH_SECRET in .env
```

Also set real `POSTGRES_PASSWORD` / `RABBITMQ_PASSWORD` / `MINIO_ROOT_PASSWORD`, and point `FRONTEND_URL` / `VITE_BACKEND_URL` at your real origin. See [docs/PRODUCTION.md](docs/PRODUCTION.md) for the full checklist.

---

## Setup by Platform

- [macOS (Cloud API Only)](#macos-cloud-api-only)
- [macOS with Local Inference on Apple GPU (MPS)](#macos-with-local-inference-on-apple-gpu-mps)
- [Linux (Cloud API Only)](#linux-cloud-api-only)
- [Linux with Local Inference on NVIDIA GPU](#linux-with-local-inference-on-nvidia-gpu)

> All paths run the production build. `.env.example` boots locally as-is; set a real `BETTER_AUTH_SECRET` in `.env` (`openssl rand -base64 32`) before any shared or remote deployment.

### macOS (Cloud API Only)

1. Install and **open** [Docker Desktop for Mac](https://docs.docker.com/desktop/install/mac-install/) (the Docker daemon must be running before proceeding).
2. This setup uses cloud providers (Azure OpenAI, Google Gemini, Anthropic) for inference — no GPU required. If you want local inference on Apple GPU instead, use the **macOS with Local Inference on Apple GPU (MPS)** path below.
3. Open **Terminal**:
   ```bash
   cd docker-files
   cp .env.example .env
   docker compose up --build
   ```
4. Open **http://localhost:3000** in your browser.

### macOS with Local Inference on Apple GPU (MPS)

Use this path if you want BioEval to keep the shared stack in Docker but run `inference-service` natively on your Mac so local models can use the Apple GPU via `mps`.

Important:
- Choose **either** the default all-Docker path **or** this MPS path. Do **not** run `docker compose up --build` first and then start the host-native worker, or you will end up with both the Docker `inference` service and the macOS `inference-service` consuming the same RabbitMQ queues.
- If you already started the full Docker stack, stop it before switching to this path:
  ```bash
  cd docker-files
  docker compose down
  ```

1. Copy the Docker env file:
   ```bash
   cd docker-files
   cp .env.example .env
   ```
2. Start the shared services without the Docker `inference` container:
   ```bash
   cd docker-files
   ./start-macos-host-stack.sh
   ```
   This script uses `docker-compose.host-native.yml` so the Docker `backend`
   talks to the macOS host `inference-service` at `http://host.docker.internal:8000`
   instead of the Docker `inference` service.
3. In a second terminal, start `inference-service` on macOS:
   ```bash
   cd inference-service
   ./scripts/run-macos-host.sh
   ```
4. Open **http://localhost:3000** in your browser.

### Linux (Cloud API Only)

1. Install [Docker Engine](https://docs.docker.com/engine/install/) and the [Compose plugin](https://docs.docker.com/compose/install/linux/).
2. Run:
   ```bash
   cd docker-files
   cp .env.example .env
   docker compose up --build
   ```
3. Open **http://localhost:3000** in your browser.

### Linux with Local Inference on NVIDIA GPU

1. Install [Docker Engine](https://docs.docker.com/engine/install/) and the [Compose plugin](https://docs.docker.com/compose/install/linux/).
2. Install the [NVIDIA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html).
3. Run:
   ```bash
   cd docker-files
   cp .env.example .env
   docker compose -f docker-compose.yml -f docker-compose-gpu.yml up --build
   ```
4. Open **http://localhost:3000** in your browser.

---

## Managing the Stack

Run these from the `docker-files/` directory:

```bash
docker compose ps              # service status and health
docker compose logs -f backend # follow a service's logs (backend / frontend / inference / migrate)
docker compose up -d --build   # rebuild and (re)start after pulling changes
docker compose down            # stop all services
docker compose down -v         # stop and delete all data (database, model outputs, datasets)
```

## Support

We welcome bug reports, feature requests, and questions — please [open a GitHub issue](https://github.com/Yale-BIDS-Chen-Lab/BioEval/issues). Including your OS, Docker version, and relevant logs helps us respond faster.

For help extending BioEval with new models, metrics, or benchmark workflows, feel free to email xuguang.ai@outlook.com.

---

## Contributing

See [CONTRIBUTING.md](CONTRIBUTING.md).

---

## License

This project is licensed under the **MIT License**.
See [LICENSE.md](LICENSE.md).

BioEval

BioEval Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

BioEval

BioEval Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Download README