KO | EN

GitLite

Minimalist Open Source Repository Search Engine

gitlite — search

#typescript #ai-agents #cli #claude-code #codex #developer-tools #dsh-plugin #deepseek-harness #windows #open-source #agent-skills #ai

KoreanWordVectors

★ 107 Open GitHub ↗

Subword-level Word Vector Representations for Korean (ACL 2018)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Word Vector Representation for Korean
**Subword-level Word Vector Representations for Korean**<br/>
*Sungjoon Park, Jeongmin Byun, Sion Baek, Yongseok Cho, Alice Oh*<br/>
*Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (ACL 2018)*<br/>

## Abstract
Research on distributed word representations is focused on widely-used languages such as English. Although the same methods can be used for other languages, language-specific knowledge can enhance the accuracy and richness of word vector representations. In this paper, we look at improving distributed word representations for Korean using knowledge about the unique linguistic structure of Korean. Specifically, we decompose Korean words into the jamo-level, beyond the character-level, allowing a systematic use of subword information. To evaluate the vectors, we develop Korean test sets for word similarity and analogy and make them publicly available. The results show that our simple method outperforms word2vec and character-level Skip-Grams on semantic and syntactic similarity and analogy tasks and contributes positively toward down-stream NLP tasks such as sentiment analysis.

## Dataset
We open our evaluation dataset for Korean word vectors. Details are described below. We plan to develop more evaluation sets for Korean NLP communities, so any comments for theses sets or collaboration for constructing other sets are welcome!

## 1. WS-353 for word similarity (Korean)
+  2 graduate students translated original (English) set.
+ 14 native Korean speakers participated in evaluation of the set.
+ Excluded the minimum and maximum scores and compute the mean of the rest of the scores.
+ .82 correlation with original English set.
+ Some of words are replaced by more familiar words to Korean. ( e.g.) Arafat -> 안중근 )

## 2. Word Analogies (Korean)
+ 10,000 items. 5,000 for semantic and 5,000 syntactic items.
+ 5 categories for semantic and syntactic features.
+ Each category contains 1,000 items.
+ **Syntactic Features (with an example)** :
    + Case : 자동차 자동차를 인터넷 인터넷을
    + Tense  가다 갔다 공부하다 공부했다
    + Voice  갈다 갈리다 거래하다 거래되다
	+ Verb form  가다 가고 놓다 놓고
	+ Honorific  가다 가시다 공부하다 공부하시다
+	**Semantic Features (with an example)**:
	+ Capital-countries 아테네 그리스 바그다드 이라크
	+ male-female 남자 여자 아버지 어머니
	+ name-nationality 간디 인도 나폴레옹 프랑스
	+ country-language 아르헨티나 스페인어 미국 영어
	+ misc  개 강아지 소 송아지

## Korean word vector representation learning
### Building fastText for Korean
Before you start training Korean word vectors, you should build the source of subword-level Korean word vectors (a.k.a., Korean FastText) by using `make`.
```
$ cd src
$ make
```
This will produce object files for all the classes as well as the main binary `fasttext`.

### 1. Parse Korean documents.
First, you should parse a Korean document with `decompose_letters.py` This file will decompose original Korean letters in the document, generating a parsed document. The parsed document will be used as training data of the vectors. An example use case is as follows:
```
python decompose_letters.py [input_file_name] [parsed_file_name]
```

### 2. Train Korean word vectors.
Then, you can train subword-level word vectors for Korean. The source code depends on the implementation of `FastText`. Thus you can execute the complied source as like the original `FastText`. Note that the source code will accept the output file `[parsed_file_name]`  generated by `decompose_letters.py`. An example use case is as follows:
```
[fastText_executable_path] skipgram -input [parsed_file_name] -output [output_file_name] -minCount 10 -minjn 3 -maxjn 5 -minn 1 -maxn 4 -dim 300 -ws 5 -epoch 5 -neg 5 -loss ns -thread 16
```
The full list of parameters are given below.
```
-minCount : minimal number of word occurences [5]
-bucket : number of buckets [10000000]
-minn : min length of char ngram [1]
-maxn : max length of char ngram [4]
-minjn : min length of jamo ngram [3]
-maxjn : max length of jamo ngram [5]
-emptyjschar : empty jongsung symbol ["e"]
-t : sampling threshold [1e-4]
-lr : learning rate [0.05]
-dim : size of word vectors [100]
-ws : size of the context window [5]
-loss : loss function {ns, hs, softmax} [softmax]
-neg : number of negatives sampled [5]
-epoch : number of epochs [5]
-thread : number of threads [12]
-verbose : verbosity level [2]
```
As written in the paper, the default number of character-level n-grams is set to 1-4, and the number of jamo-level n-grams is set to 3-5. As the number of n-grams increases, you should adjust the number of maximum unique n-grams (bucket), otherwise some n-grams will be overridden. We recommend 10,000,000 for approximately 3GB of (parsed) Korean corpus.

### Constructing Korean OOV word vectors
The trained output file `[output_file_name].bin` can be used to compute word vectors for OOVs. Provided you have a text file `queries.txt` containing Korean decomposed words for which you want to compute vectors, use the following command:
```
$ [fastText_executable_path] print-word-vectors model.bin < queries.txt
```
Note that  `queries.txt` should contain decomposed Korean words, such as ㄱㅏㅇㅇㅏeㅈㅣe for 강아지. You can also use `jamo_split` method in `decompose_letters.py` to obtain decomposed Korean words.

### Reference
Please cite the followings if using this code for learning word representations for Korean or evaluating word vectors using the evaluation sets.

```
@inproceedings{park-etal-2018-subword,
    title = "Subword-level Word Vector Representations for {K}orean",
    author = "Park, Sungjoon and
      Byun, Jeongmin and
      Baek, Sion and
      Cho, Yongseok and
      Oh, Alice",
    booktitle = "Proceedings of the 56th Annual Meeting of the ACL",
    year = "2018",
    pages = "2429--2438"
}
```

## Change Log
01-11-19 : Add implementations. version 1.0 
05-04-18 : Initial upload of datasets. version 1.0

KoreanWordVectors

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README