GrokkedTransformer

(β˜… 241)

Code for NeurIPS'24 paper 'Grokked Transformers are Implicit Reasoners: A Mechanistic Journey to the Edge of Generalization'

File Explorer

Download Latest Version (.zip)

Showing a partial file list β€” download the zip above to see everything.

  • .gitignore
  • causal_tracing_comparison.py
  • causal_tracing_composition.py
  • comparison.ipynb
  • complex_reasoning.ipynb
  • composition.ipynb
  • eval_qa.py
  • LICENSE
  • LLM.ipynb
  • main.py
  • README.md

# Installation Guide

1. Get the code
git clone https://github.com/OSU-NLP-Group/GrokkedTransformer

Downloads the entire project code from GitHub to your computer.

cd GrokkedTransformer

Moves into the project folder you just downloaded.

2. Official Install Script

Easy Recommended
Prerequisites
  • Python 3 Python is required to use pip.
pip install torch==1.13.1+cu116 torchvision==0.14.1+cu116 torchaudio==0.13.1 --extra-index-url https://download.pytorch.org/whl/cu116

Installs the package published on PyPI directly β€” no need to clone the source.

βœ… After installing, open a new terminal and run the program's version command (e.g. --version) to confirm it worked.

Pulled directly from this repo's README.

3. Docker

Easy
Prerequisites
  • Git Needed to download the project code from GitHub.
  • Docker Desktop Needed to build and run containers. Install it and keep it running in the background.
⚠️ This is a large repository, so this method may point to an internal sub-package rather than the actual core product. Check the full README as well.
docker build -f transformers/docker/transformers-all-latest-gpu/Dockerfile -t grokkedtransformer .

Builds a runnable image based on the Dockerfile.

docker run -p 8080:80 grokkedtransformer

Runs the built image as an actual container.

βœ… Run docker compose ps to check the containers are Up. If the README mentions a port, open http://localhost:PORT in your browser.

4. Python

Easy
Prerequisites
  • Git Needed to download the project code from GitHub.
  • Python 3 On Windows, be sure to check 'Add Python to PATH' during installation.
pip install torch==1.13.1+cu116 torchvision==0.14.1+cu116 torchaudio==0.13.1 --extra-index-url https://download.pytorch.org/whl/cu116

Installs the package published on PyPI directly β€” no need to clone the source.

pip install -e .

Installs the Python libraries listed in requirements.txt (or similar).

βœ… If it runs without errors and prints output in the terminal, it worked.

Pulled directly from this repo's README.

5. Make

Medium
Prerequisites
  • Git Needed to download the project code from GitHub.
  • Make Usually pre-installed on Linux/macOS. On Windows, install separately (e.g. via MSYS2 or WSL).
cd simpletransformers

This project's files live in a subfolder, so move into it first.

make

Compiles the code based on the generated build configuration to produce an executable.

βœ… If it finishes without errors, it worked. Try running the generated executable directly.

Today's picks

caveman β˜… 96,051

πŸͺ¨ why use many token when few token do trick β€” Claude Code skill that cuts 65% of tokens b

JavaScript
gpt4all β˜… 77,396

GPT4All: Run Local LLMs on Any Device. Open-source and available for commercial use.

C++
create-react-app β˜… 103,290

Set up a modern web app by running one command.

JavaScript
vscode β˜… 189,119

Visual Studio Code

TypeScript
Best-websites-a-programmer-should-visit β˜… 76,276

:link: Some useful websites for programmers.
odysseus β˜… 85,872

Self-hosted AI workspace.

Python
// repository documentation