gbswt5

KO | EN

gitlite — search

#javascript #python #hacktoberfest #react #llm #golang #docker #nodejs #windows #kubernetes #vue #security #openai #ios

(★ 19)

CharFormer(Tay et al., 2022; Gradient-based Subword Tokenizer + T5) model implementation for Huggingface Transformers

gbswt5 Latest Version Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# GBSWT5

Huggingface Transformers-compatible GBST+T5 implementation(as CharFormer(Tay et al., 2022)) for GBST-KEByT5 Model.

Supports following pretrained checkpoints:
  * [etri-lirs/gbst-kebyt5-base-preview](https://huggingface.co/etri-lirs/gbst-kebyt5-base-preview)
  * [etri-lirs/gbst-kebyt5-large-preview](https://huggingface.co/etri-lirs/gbst-kebyt5-large-preview)

## How To Use
Install with pip.
```
pip install git+https://github.com/etri-crossmodal/gbswt5.git
```

### How to load a model
```
import gbswt5
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

tokenizer = AutoTokenizer.from_pretrained("etri-lirs/gbst-kebyt5-base-preview")
model = AutoModelForSeq2SeqLM.from_pretrained("etri-lirs/gbst-kebyt5-base-preview")
```

## Dependency
 * pytorch>=1.8.0
 * transformers>=4.27.0
 * einops>=0.6.0

## Citation
Written in korean only.
```
@article{shin2023tflm,
  title={Towards Korean-centric Token-free Pretrained Language Model},
  author={Shin, Hur and Ryu, Lee and Seo, Seong and Lim.},
  journal={Proceedings of the 35th Annual Conference on Human and Cognitive Language Technology, pp. 711-715.},
  year={2023}
}
```

## Acknowledgement

* This software was supported by the Institute of Information & communication Technology Planning & Evaluation (IITP) grant funded by the Korea government(MSIT). (No. RS-2022-00187238, Development of Large Korean Language Model Technology for Efficient Pre-training)
 * This software includes lucidrains/charformer-pytorch GitHub project for GBST implementation, which distributed under MIT License. Copyright (c) 2021 Phil Wang. all rights reserved. (Original Code URL: https://github.com/lucidrains/charformer-pytorch)
 * This software includes HuggingFace transformers's T5 implementation for GBST-enabled T5 model, which distributed under Apache 2.0 License. Copyright 2018- The Huggingface team. All rights reserved.
 
We are grateful for their excellent works.

gbswt5

gbswt5 Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

gbswt5

gbswt5 Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Download README