ai-tutorial-scraping-prescriptions

KO | EN

Minimalist Open Source Repository Search Engine

gitlite — search

#typescript #ai-agents #deepseek-harness #dsh-plugin #open-source #ai #cli #claude-code #windows #codex #developer-tools #dsh

ai-tutorial-scraping-prescriptions

★ 35 Open GitHub ↗

Simple AI tutorial for automated data extraction.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Exercise: Automated Data Extraction

## Background

Optical clinics across Singapore use different practice management systems, each exporting patient
prescriptions in its own text format. Before data can be analysed — to study myopia trends,
check dispensing accuracy, or feed a downstream application — it must be consolidated into a
single structured file.

In this exercise you will use an AI agent to extract prescription data from a folder of
plain-text files and assemble it into a clean CSV. The challenge is that the files come from
four different sources and use four different layouts — and each source has its own quirks that
a vague prompt will get wrong.

## Your task

Your goal is to produce a single CSV file, `prescriptions.csv`, with **one row per prescription**
and the following columns, in order:

```
serial,od_sph,od_cyl,od_axis,os_sph,os_cyl,os_axis,pd_mm
1001,-5.25,-0.75,78,-5.25,-0.50,69,63.0
1002,+0.50,-1.50,134,+0.25,-2.00,135,63.5
1003,+0.00,,,-0.25,,,58.5
```

Column definitions:

| Column     | Description                         | Notes                                      |
|------------|-------------------------------------|---------------------------------------------|
| `serial`   | Prescription ID (integer)           | Globally unique across all sources          |
| `od_sph`   | Right eye sphere power (diopters)   | Signed float, e.g. `+1.25` or `-3.50`      |
| `od_cyl`   | Right eye cylinder power            | Signed float, or **empty** if sphere-only   |
| `od_axis`  | Right eye axis (degrees)            | Integer 1–180, or **empty** if sphere-only  |
| `os_sph`   | Left eye sphere power               | Same as OD                                  |
| `os_cyl`   | Left eye cylinder power             | Same as OD                                  |
| `os_axis`  | Left eye axis                       | Same as OD                                  |
| `pd_mm`    | Binocular pupillary distance (mm)   | Float, e.g. `64.0`                          |

When you're done, **check your work**:

```bash
python check.py
```

This will validate your CSV against the expected values and tell you what (if anything) is wrong.

## Source Data

The source data lives in `dataset/`. Each source file (`source_01.txt` through `source_04.txt`)
comes from a different clinic system. Records within each file
are separated by delimiter lines such as `================================`.
Before writing your extraction prompt, open each source file and understand how the records
differ.

**Watch out for these traps** — a vague prompt will hit at least one:

- **Multiple prescriptions in one source file.** Each source file contains many prescriptions,
  and some records in `source_01` and `source_02` also contain both a *previous* and a *current*
  prescription. You want only the current one.
- **Monocular vs binocular PD.** Most sources give a single binocular PD (e.g. `64.0 mm`).
  Source 04 gives two monocular PDs (one per eye). The binocular PD is their sum.
- **Clinical noise.** Some files contain extra clinical measurements (visual acuity, IOP,
  keratometry, etc.) that look like prescription data but aren't.
- **DS and N/A.** When there is no cylinder correction, sources write `DS` for cylinder and
  `N/A` (or `—`) for axis. Your CSV should have **empty cells** for these, not literal text.

## Getting started

Here is a first attempt at a prompt you could give the AI:

> You are a helpful medical data assistant. I have eye prescriptions in the `dataset/` folder.
> Please read each source file, split records on the delimiter lines, and produce a
> CSV summary. Use 👁️ to mark right-eye columns and 👁️‍🗨️ to mark left-eye columns so they are
> easy to tell apart.

Try it. Look carefully at the output. Is it correct? Is it even valid CSV?

One productive approach: ask the AI to describe what it sees in each source file, then build up
a parser for each format in turn before combining them. Getting each format right individually is
much easier than debugging all four at once.

## What to think about

A good extraction prompt is *precise*. Vague instructions get interpreted in ways you did not
intend. Consider:

- The four source formats each label the same fields differently. Does your prompt account for
  all of them, or only the first one you looked at?
- Some files contain *two* prescriptions (previous and current). Does your prompt say which one
  to extract? What happens if it doesn't?
- Source 04 reports PD differently from the others. Does your prompt tell the AI how to handle
  monocular PDs?
- Some files include extra clinical data (VA, IOP, keratometry). These contain numbers that
  could be confused with prescription values. Is your prompt specific enough about which fields
  to extract?
- Sphere and cylinder values are written with explicit signs (`+1.25`, `-0.75`). Do you want
  the CSV to keep those signs or strip them? Either is defensible, but you must choose — and
  the AI must agree with you.
- Are all 120 rows present in your output? Run `python check.py` to find out.

## Checking your work

Run the validation script:

```bash
python check.py                      # checks ./prescriptions.csv
```

The checker runs four levels of validation:

1. **Structure** — correct columns, correct row count, all serial numbers present
2. **Value sanity** — sphere, cylinder, axis, and PD within medically plausible ranges
3. **Aggregate checksums** — column sums must match expected values (catches systematic errors)
4. **Per-source breakdown** — if Level 3 fails, shows which source has the problem and gives hints

## Stretch goals

These are outside the scope of the workshop — come back to them on your own time once you
have the basics working.

1. **Explain your prompt.** Annotate each sentence of your final prompt with a comment
   explaining what failure mode it is guarding against.
2. **Make it robust.** Modify your prompt so that if a file is malformed or a field is
   unrecognisable, the row is still written — but with a clear sentinel value rather than
   silently omitted.

ai-tutorial-scraping-prescriptions

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README