TinyStories

KO | EN

gitlite — search

#javascript #python #hacktoberfest #react #ai #typescript #llm #go #golang #android #machine-learning #rust #deep-learning #linux

(★ 41)

code to train a gpt-2 model to train it on tiny stories dataset according to the TinyStories paper

TinyStories 최신버젼 다운로드

최종 버전 다운로드 (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

Repository contains code to config a 28M param GPT-2 model to train it on tinystories dataset like the TinyStories paper. In the paper, they used several variations of **GPT-2** and **GPT-Neo** model, but **GPT-2** was the spotlight version. They tried to show you can create a SLM (Small Language Model) upto which small size params until it makes sense and compete against LLMs.

I am only providing the code to config a 28M model as transformers currently having a problem to install my favourite version `transformers==4.2.2` and new transformers requires accelerate if you're using pytorch and requires `partial state` I am not sure how you resolve the partial state error at least now. But, I wanted the method to config a 28M model asap! Which is why, I am only providing the config code. Will later update the repo to add training code.

Upcoming updates:
* Providing training script
* ~~providing TinyStories dataset in `.txt` format~~

Current Updates:
* Torchscale library method added for coding LLM of any amount params at ease. notebook: https://github.com/sleepingcat4/TinyStories/blob/master/alright!_calculated_param.ipynb

Dataset:
TinyStories dataset had two part ```1. GPT-3.5 Turbo generated dataset`` and ``2.GPT-4 generated dataset`` Including both will take a hue amount of space that's why, I am giving only **GPT-3.5 Turbo** dataset. [Google Drive Link](https://drive.google.com/drive/folders/12ZeAVOzuk3W_R0pBr-fXQkbTP_WRJUEx?usp=sharing)

**In the Google Drive link provided in the datasets section, you find both GPT-3.5 (Turbo) and GPT-4 datasets used by the paper's autors'**

Please, star the repository if you find it helpful and help others to find it. [Paper Link](https://arxiv.org/abs/2305.07759)

TinyStories

TinyStories 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

TinyStories

TinyStories 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

Download README