KO
|
EN
gitlite — search
Search
#python
#java
#python3
#arduino
#golang
#machine-learning
#rust
#html
#flask
#javascript
#seismology
#nodejs
dataProcessor
★ 12
Open GitHub ↗
中英文语料数据清洗及分布式分句分词预处理工作
Download README (.md)
Explore Similar Repositories
private_out_send
:
Script to collect ETH or ERC20 from a list of private key to 1 address
MultiColorText
:
Allows to draw text with multiple colors with just one function in Garry's Mod
observability
:
Project to apply concepts of the observability
slash
:
PHP library for functional programming
dejavu
:
DejaVu fonts for Go
// repository documentation
Was this content helpful?
★ 0
(0 ratings)
Select Rating:
★
★
★
★
★
Submit Feedback
Recent Feedback
×
Download README
Do you want to download the
README.md
file for
dataProcessor
?
Download (.md)
## 代码主要是对较大型语料(约14G的语料)进行分布式清洗和分句分词操作 #### 代码包括: ##### 如何批量读取文件夹及子文件夹下的数据 ##### 如何将批量整合文件夹及子文件下的数据 ##### 匹配中英文里可能出现的所有特殊字符 ##### 匹配各类网址及网页标签 ##### 匹配希腊字母汉语拼音及繁体字等 ##### 利用*PyLTP*模块进行分句 ##### 去掉文本空行函数 ##### 计时装饰器以及代码进度条 ##### 添加分布式分词处理类文件 **欢迎star和fork**