stif-indonesia - GitLite

KO | EN

gitlite — search

#javascript #hacktoberfest #ai #llm #android #rust #deep-learning #linux #java #kubernetes #vue #pytorch #ios #openai

stif-indonesia

(★ 123)

Implementation of "Semi-Supervised Low-Resource Style Transfer of Indonesian Informal to Formal Language with Iterative Forward-Translation".

파일 탐색기

최종 버전 다운로드 (.zip)

- - dev.for
  - dev.inf
  - test.for
  - test.inf
  - train.for
  - train.inf
- - batch0.inf
  - batch1.inf
  - batch10.inf
  - batch11.inf
  - batch2.inf
  - batch3.inf
  - batch4.inf
  - batch5.inf
  - batch6.inf
  - batch7.inf
  - batch8.inf
  - batch9.inf
- 00001_default_supervised_config.json
- 00002_default_semi_supervised_config.json
- meme_stif.PNG
- paper.PNG
- download_moses.sh
- __init__.py
- __main__.py
- logging.py
- main_moses.py
- util.py
.gitignore
LICENSE
README.md
requirements.txt

# 설치 가이드

1. 코드 내려받기

git clone https://github.com/haryoa/stif-indonesia

깃허브에서 프로젝트 코드 전체를 내 컴퓨터로 내려받습니다.

cd stif-indonesia

방금 내려받은 프로젝트 폴더 안으로 이동합니다.

2. Python

쉬움 추천

사전 준비물

Git GitHub에서 프로젝트 코드를 내려받으려면 필요합니다.
Python 3 설치 시 'Add Python to PATH' 옵션을 꼭 체크하세요 (Windows).

pip install -r requirements.txt

requirements.txt 등에 명시된 파이썬 라이브러리를 설치합니다.

python <실행할 파일명>.py # README에서 정확한 실행 파일명을 확인하세요

파이썬 스크립트(또는 모듈)를 실행합니다.

✅ 에러 메시지 없이 실행되고 터미널에 안내 문구가 출력되면 정상입니다.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# STIF-Indonesia

![Paper](https://raw.githubusercontent.com/haryoa/stif-indonesia/main/imgs/meme_stif.PNG)

An implementation of ["Semi-Supervised Low-Resource Style Transfer of Indonesian Informal to Formal Language with Iterative Forward-Translation"](https://arxiv.org/abs/2011.03286v1).

You can also find Indonesian informal-formal parallel corpus in this repository.

## Description

We were researching transforming a sentence from informal to its formal form. Our work addresses a style-transfer from informal to formal  Indonesian as a low-resource **machine translation**  problem. We benchmark several strategies to perform the style transfer.

In this repository, we provide the Phrase-Based Statistical Machine Translation, which has the highest result in our experiment. Note that, our data is extremely low-resource and domain-specific (Customer Service domain). Therefore, the system might not be robust towards out-of-domain input. Our future work includes exploring more robust style transfer. Stay tuned!

## Paper

![Paper](https://raw.githubusercontent.com/haryoa/stif-indonesia/main/imgs/paper.PNG)

You can access our paper below:

[Semi-Supervised Low-Resource Style Transfer of Indonesian Informal to Formal Language with Iterative Forward-Translation (IALP 2020)](https://arxiv.org/abs/2011.03286v1)

Medium Article: [Mengubah Bahasa Indonesia Informal Menjadi Baku Menggunakan Kecerdasan Buatan](https://medium.com/kata-engineering/mengubah-bahasa-indonesia-informal-menjadi-baku-menggunakan-kecerdasan-buatan-4c6317b00ea5) (In Indonesian)

## Requirements

We use the RELEASE 4.0 Ubuntu 17.04+ version which only works on the specified OS.

We haven't tested it on other OS (e.g.: OS X and Windows). If you want to run the source code, use Ubuntu 17.04+. If you use windows, we advise you to use the WSL-2 to run the code.

In this experiment, we wrap the MOSES code by using Python's `subprocess`. So Python installation is necessary. The system is tested on Python 3.9. We recommend it to install with `miniconda`. You can install it by following this link: https://docs.conda.io/en/latest/miniconda.html

## How To Run

First, clone the repository

```bash
git clone https://github.com/haryoa/stif-indonesia.git
```

Then run the MOSES downloader. We use .sh, so use a CLI application that can execute it. On the root project folder directory, do:

```bash
bash scripts/download_moses.sh
```

The script will download the Moses toolkit and extract it by itself.

### Install required python packages

Before running the program you have to install some prerequisites packages:

```bash
pip install -r requirements.txt
```

Alternatively, if you prefer to use `pipenv` instead you can run:

```bash
pipenv install
```

__NOTE:__ If you prefer to use `pipenv` you should preceed the command with `pipenv run`. E.g: `pipenv run python -m stif_indonesia --exp-scenario supervised`

### Run Supervised Experiments

To run the supervised one, do:

```bash
python -m stif_indonesia --exp-scenario supervised
```

It will read the experiment config in `experiment-config/00001_default_supervised_config.json`

### Run Semi-Supervised Experiments

To run the semi-supervised one, do:

```bash
python -m stif_indonesia --exp-scenario semi-supervised
```

It will read the experiment config in `experiment-config/00002_default_semi_supervised_config.json`

## Output

1. The training process will output the log of the experiment in `log.log`
2. The output of the model will be produced in the `output` folder

### Supervised output

It will output `evaluation`, `lm` , and `train`. `evaluation` is the result of  prediction on the test set, `lm` is the output of the trained LM, and `train` is the produced model by the Moses toolkit

### Semi-supervised output

It will output `agg_data`, `best_model_dir`, and `produced_tgt_data`. `agg_data` is the result of the forward-iteration data synthesis. `best_model_dir` is the best model produced by the training process, and `produced_tgt_data` is the prediction output of the test set.

### Score

Please check the `log.log` file which is the output of the process.

## Additional Information

If you want to replicate the dictionary-based method, you can use any informal - formal or slang dictionary on the internet.

For example, you can use [this dictionary](https://github.com/louisowen6/NLP_bahasa_resources/blob/master/combined_slang_words.txt).

If you want to replicate our GPT-2 experiment, you can use a pre-trained Indonesian GPT-2 such as [this one](https://huggingface.co/cahya/gpt2-large-indonesian-522M), or train it by yourself by using [Oscar Corpus](https://oscar-corpus.com/). After that, you can finetune it with the dataset that we have provided here. You should follow the paper on how to transform the data when you do the finetuning.

Note: You can also download oscar-corpus from [Huggingface's datasets](https://huggingface.co/datasets).

We use [Huggingface](https://github.com/huggingface/transformers)'s off-the-shelf implementation to train the model.

## Team

1. Haryo Akbarianto Wibowo @ Kata.ai
2. Tatag Aziz Prawiro @ Universitas Indonesia
3. Muhammad Ihsan @ Bina Nusantara
4. Alham Fikri Aji @ Kata.ai
5. Radityo Eko Prasojo @ Kata.ai
6. Rahmad Mahendra @ Universitas Indonesia