nseg - GitLite

KO | EN

gitlite — search

#javascript #python #react #ai #android #machine-learning #rust #deep-learning #windows #awesome #cli #awesome-list #ios #openai

nseg

(★ 96)

Node.js Version of MMSG for Chinese Word Segmentation

파일 탐색기

최종 버전 다운로드 (.zip)

- nseg
- dict.js
- dict.txt
- freq.csv
- freq.js
- evented.js
- normal.js
- sina-at.js
- sina-smily.js
- sina-tag.js
- - arabic-numerals.js
  - chinese-numerals.js
  - cyrillic.js
  - date.js
  - datetime.js
  - greek.js
  - index.js
  - latin.js
  - puncature.js
  - quantity.js
  - time.js
  - uri.js
- - reusable-list.js
  - string-stream.js
- evented.js
- filter.js
- frequency.js
- lexer.js
- matcher.js
- mmseg.js
- perftest.js
- qiuye.txt
- file-handler.js
- freq-builder.js
- hash-builder.js
- inspector.js
- runner.js
- normal-tests.js
.gitignore
.npmignore
.travis.yml
index.js
LICENSE
package.json
README.md

# 설치 가이드

1. 코드 내려받기

git clone https://github.com/mountain/nseg

깃허브에서 프로젝트 코드 전체를 내 컴퓨터로 내려받습니다.

cd nseg

방금 내려받은 프로젝트 폴더 안으로 이동합니다.

2. Node.js

쉬움 추천

사전 준비물

Git GitHub에서 프로젝트 코드를 내려받으려면 필요합니다.
Node.js npm 명령어를 쓰려면 Node.js가 설치되어 있어야 합니다. LTS(장기지원) 버전을 추천합니다.

$ npm install nseg

package.json에 명시된 라이브러리들을 내려받아 설치합니다.

$ npm install -g nseg

package.json에 명시된 라이브러리들을 내려받아 설치합니다.

✅ 명령어 실행 후 터미널에 나타나는 주소(보통 http://localhost:3000 형태)를 브라우저에서 열어보세요.

이 레포의 README에 적힌 실제 명령어를 그대로 가져왔습니다.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

Node.js Version of MMSG for Chinese Word Segmentation
======================================================
[![Build Status](https://secure.travis-ci.org/mountain/nseg.png)](http://travis-ci.org/mountain/nseg)

MMSG originally invented by Chih-Hao Tsai is a very popular Chinese word
segmentation algorithm. Many implementations are available on different
platforms including Python, Java, etc.

This package provide Node.js version of MMSG algorithm. The API is async and evented sytle.

So far this package is still in developing, but the basic functionalities are ready.

Install
=======

Use nseg in your own package

$ npm install nseg

Or if you want to use `nseg` command

$ npm install -g nseg

Command line
============

After intsalling globally, you can use `nseg` command to:

help

$ nseg help

segment text using default dictionary

$ nseg segf -i ~/project/text/shi.txt -o ~/project/output/shi.txt
    $ nseg segd -i ~/project/text -o ~/project/output

build user dictionary for loading aftermath

$ nseg dict ~/project/data/dict.js ~/dict/dict1.txt ~/dict/dict2.txt
    $ nseg dict ~/project/data/dict.js ~/dict

build character-frequecy map for loading aftermath

$ nseg freq ~/project/data/freq.js ~/freq/data1.csv ~/freq/data2.csv
    $ nseg freq ~/project/data/freq.js ~/freq

segment text using customized settings

$ nseg segd -d ~/project/data/dict.js -f ~/project/data/freq.js -i ~/project/text -o ~/project/output
    $ nseg segd -l ~/project/lex/ -i ~/project/text -o ~/project/output

check the existence of a word

$ nseg check "石狮"
    $ nseg check -d ~/project/data/dict.js "石狮"

Using nseg in program
======================

Preparation
-----------

- (Optional) build your own dictionay and freqency map
- (Optional) create your own lexical handler for special text pattern

Examples
--------

Stream-pipe style

````javascript
var dict  = require('../data/dict'),
    freq  = require('../data/freq'),
    date  = require('../lex/datetime'),
    sina  = require('../lex/sina');

var opts  = {
        dict: dict,
        freq: freq,
        lexers: [date, sina],
    };

var nseg = require('nseg').evented(opts);

var strmOut = fs.createWriteStream(target, {flags: 'w+', encoding: 'utf-8'}),
    strmIn  = fs.createReadStream(input);

var pipe = nseg(strmIn, strmOut);
pipe.on('error', function (err) {
    console.log('error', err);
});

pipe.start();

````

Normal callback style (buggy)

````javascript
var dict  = require('../data/dict'),
    freq  = require('../data/freq'),
    date  = require('../lex/datetime'),
    sina  = require('../lex/sina');

var opts  = {
        dict: dict,
        freq: freq,
        lexers: [date, sina],
    };

var nseg = require('nseg').normal(opts);

nseg('研究生源计划', function (result) {
    console.log(result);
});

````

Lexical handler customization
=============================

Lexical handlers support definitions by acceptor functions.

An acceptor function is a function with signature of

function accept(curchar, undecidedprefix, nextchar)

And return value should be one value from -1, 0, 1 on case of:

* -1: we can decide a negitive result for the current character.
* 0 : we should read more characters.
* 1 : we can decide a negitive result for the current character.

License
=======

MIT License