convseg

KO | EN

gitlite — search

#javascript #python #hacktoberfest #react #ai #typescript #llm #go #golang #android #machine-learning #rust #deep-learning #linux

★ 146 Open GitHub ↗

Convolutional neural network and word embeddings for Chinese word segmentation

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# CONV-SEG
Convolutional neural network for Chinese word segmentation (CWS).
The corresponding paper: [**Convolutional Neural Network with Word Embeddings for Chinese Word Segmentation**](https://arxiv.org/pdf/1711.04411.pdf)

## Author
Chunqi Wang

## Dependencies
* [Python 2.7](https://www.python.org/)
* [Tensorflow 1.0](https://www.tensorflow.org/)

It is better to use a nvidia GPU to accelerate the training procedure.

## Data
Downlaod `data.zip` from [here](https://drive.google.com/file/d/0B-f0oKMQIe6sQVNxeE9JeUJfQ0k/view?usp=sharing&resourcekey=0-pqrbXYjENu6AJxR6mq8zJQ) (Note that the SIGHAN datasets should only be used for research purposes). Extract `data.zip` to this directory. So the file tree would be:

## How to use
First, give execute permission to scripts:

chmod +x train_cws.sh train_cws_wemb.sh

Train a preliminary model (CONV-SEG):

./train_cws.sh WHICH_DATASET WHICH_GPU
	
Train a model with word embeddings (WE-CONV-SEG):

./train_cws_wemb.sh WHICH_DATASET WHICH_GPU
	
We have two optional datasets: `pku` and `msr`. If you run the program in CPU environment, just leave the second argument empty.

For example, if you want to train the model CONV-SEG on the pku dataset and on gpu0, you should run:

./train_cws.sh pku 0
	
More arguments can be set in `train.py`.

## Test Score
| Model | PKU(dev) | PKU(test) | MSR(dev) | MSR(test) |
|:------|:---------|:----------|:---------|:----------|
| CONV-SEG | 96.8 | 95.7 | 97.2 | 97.3	|
| WE-CONV-SEG | 97.5 |	96.5	| 98.1 |	98.0 |

convseg

Today's picks

Was this content helpful?

Recent Feedback

convseg

Today's picks

Was this content helpful?

Recent Feedback

Download README