Clinical-AI-Evaluation-Framework

KO | EN

gitlite — search

#python #ai #llm #golang #android #rust #deep-learning #linux #docker #windows #kubernetes #awesome-list #security #openai

Clinical-AI-Evaluation-Framework

(★ 5)

This repository contains an implementation for an Automated Clinical AI Evaluation Pipeline.

Clinical-AI-Evaluation-Framework 최신버젼 다운로드

최종 버전 다운로드 (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Clinical Reliability Framework

> **Automated Reliability Layer for Clinical AI Pipelines**  
> *Deterministic, Reproducible, Transparent Evaluation*

---

## Project Overview
The extraction of structured data from medical records (OCR → NLP) is prone to errors that can have serious clinical implications. This project implements a **Reliability Auditing Layer** designed to evaluate, flag, and score the trustworthiness of extracted clinical entities.

Unlike "black-box" evaluations, this framework uses **deterministic heuristic rules** to verify consistency between the source text and the extracted JSON data. It audits four key dimensions:
- **Entity Type Consistency**: Is "Diabetes" correctly labeled as a `PROBLEM` or incorrectly as a `PROCEDURE`?
- **Assertion**: Does the text say "Patient denies chest pain" but the extraction says `POSITIVE`?
- **Temporality**: Is a "History of MI" incorrectly flagged as `CURRENT`?
- **Subject**: Is the `FAMILY_MEMBER`'s history wrongly attributed to the `PATIENT`?

---

## Key Features

- **Offline-First & Privacy-Centric**: No data leaves the local environment. Evaluation is performed using rule-based logic (`evaluator/rule_based.py`), ensuring GDPR/HIPAA compliance during the audit.
- **Explainable Scores**: Every error flag comes with a traceable reason (e.g., "Found negation keyword 'denies' in text span").
- **Scalable Architecture**: Processed 30+ charts in seconds using optimized Python generators.
- **Standardized Reporting**: Produces generic JSON reports compatible with downstream analytics dashboards.

---

## Repository Structure

```text
repo_name/
├── evaluator/
│   ├── rule_based.py     # The Brain: Core deterministic scoring logic
│   ├── schema.py         # Pydantic models for strict data validation
│   └── ...               # (Optional: Legacy LLM prototypes)
│
├── output/               # Generated Audit Reports (JSON)
├── test_data/            # Input Clinical Charts (Anonymized) (Ignored in version control)
├── test.py               # Main entry point (CLI)
└── report.md             # Final Executive Summary & Findings
```

---

##  Quick Start

### 1. Installation
Ensure you have Python 3.9+ installed.
```bash
pip install -r requirements.txt
```

### 2. Run Single File Audit
To evaluate a specific chart:
```bash
python test.py test_data/folder_name/chart.json output/chart_audit.json
```

### 3. Run Full Batch Evaluation
To audit the entire dataset and generate the summary report:
```bash
# Process all files
python test.py test_data output

# Generate Summary Markdown
python report_generator.py

# Verify Submission Integrity
python submission_check.py
```

---

## Scoring Logic (Under the Hood)

The framework applies a waterfall of strict checks for each extracted entity:

1.  **Textual Evidence Check**: Does the entity text actually exist in the source note?
2.  **Administrative Filter**: Suppress headers, footers, and page markers (common sources of hallucinations).
3.  **Contextual Inference**:
    - *Negation Detection*: Scans for "no", "denies", "ruled out".
    - *Temporal Logic*: Scans for "history of", "status post", "2015".
    - *Family History*: Scans section headers for "Family History" context to prevent subject attribution errors.

The result is a set of **Error Rates (0.0 - 1.0)** that allows engineers to pinpoint exactly *where* the model is failing.

---
*Submitted for the HiLabs Clinical AI Workshop*

Clinical-AI-Evaluation-Framework

Clinical-AI-Evaluation-Framework 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

Clinical-AI-Evaluation-Framework

Clinical-AI-Evaluation-Framework 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

Download README