pyatlas - GitLite

KO | EN

gitlite — search

#ai #llm #go #golang #android #machine-learning #deep-learning #nodejs #awesome-list #vue #security #pytorch #mcp #open-source

pyatlas

(★ 93)

Browse the top 10,000 packages on PyPI with the help of vector embeddings

파일 탐색기

최종 버전 다운로드 (.zip)

- - deploy-gh-pages.yml
  - main.yml
- .gitignore
- - screenshot.png
  - vite.svg
- - App.tsx
  - index.css
  - main.tsx
  - vite-env.d.ts
- .gitignore
- .prettierignore
- eslint.config.js
- index.html
- package-lock.json
- package.json
- README.md
- tailwind.config.js
- tsconfig.app.json
- tsconfig.json
- tsconfig.node.json
- vite.config.ts
- .gitignore
- inspect.py
- plot.py
- architecture.png
- screenshot.png
- - clustering.py
  - constellations.py
  - coordinates.py
  - labeling.py
  - metadata.py
  - plot_with_labels.py
  - plot_without_labels.py
- - raw_data_reader.py
- - embeddings_creator.py
- - calculate_cluster_metadata.py
  - calculate_constellations.py
  - create_vector_embeddings.py
  - download_raw_dataset.py
  - generate_cluster_labels.py
  - generate_clusters.py
  - generate_json_outputs.py
  - process_raw_dataset.py
  - setup.py
- - logging.py
- __init__.py
- config.py
- test_clustering.py
- test_constellations.py
- test_content_type_detector.py
- test_description_cleaner.py
- test_description_parser.py
.editorconfig
.env.template
.gitignore
.pre-commit-config.yaml
.prettierignore
DEVELOPMENT.md
LICENSE
Makefile
pypi_bigquery.sql
pyproject.toml
README.md
uv.lock

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# PyAtlas

<p align="center">
  <a href="https://fpgmaas.github.io/pyatlas/">
    <img src="public/screenshot.png" alt="Screenshot of PyAtlas" width="699">
  </a>
</p>

<p align="center">
  <strong>👉 <a href="https://fpgmaas.github.io/pyatlas/">Explore the map</a></strong>
</p>

## What does this do?

[PyAtlas](https://fpgmaas.github.io/pyatlas/) is an interactive map of the top 10,000 Python packages on [PyPI](https://pypi.org/). Packages with similar functionality are positioned close together, making it easy to discover alternatives or related tools. It’s mainly just a fun tech demo and hobby project, but it can also be a handy way to discover useful packages. For example, if you want to find packages similar to `matplotlib`, you can locate it on the map and explore the packages clustered around it.

## How does this work?

The project collects descriptions for the most popular packages on PyPI. These descriptions are converted into vector embeddings using [Sentence Transformers](https://www.sbert.net/). The high-dimensional embeddings are then reduced to 10 dimensions using [UMAP](https://umap-learn.readthedocs.io/), and packages are grouped into clusters using [HDBSCAN](https://hdbscan.readthedocs.io/). A second UMAP reduction creates the final 2D coordinates for visualization, using the cluster labels to keep similar packages together. Finally, cluster labels are generated using [OpenAI's](https://openai.com/) `gpt-5-mini` to describe each group.

</br>
<p align="center">
  <img src="public/architecture.png" width="500">
</p>

## Stack

The project uses the following technologies:

1. **[React](https://react.dev/)** with **[Three.js](https://threejs.org/)** for the interactive visualization
2. **[Sentence Transformers](https://www.sbert.net/)** for vector embeddings
3. **[UMAP](https://umap-learn.readthedocs.io/)** for dimensionality reduction
4. **[HDBSCAN](https://hdbscan.readthedocs.io/)** for clustering
5. **[OpenAI](https://openai.com/)** for generating cluster labels

## Development

See [development.md](./DEVELOPMENT.md)

## Data

The dataset for this project is created using the [PyPI dataset on Google BigQuery](https://console.cloud.google.com/marketplace/product/gcp-public-data-pypi/pypi). The SQL query used can be found in [pypi_bigquery.sql](./pypi_bigquery.sql).