KO
|
EN
gitlite — search
Search
#javascript
#python
#hacktoberfest
#react
#ai
#typescript
#llm
#go
#golang
#android
#machine-learning
#rust
#deep-learning
#linux
go-tokenizer
★ 10
Open GitHub ↗
A Text Tokenizer library for Golang
Download README (.md)
Explore Similar Repositories
Pi-Radio-mp3-
:
Raspberry pi internet radio and mp3 player with bluetooth.
Cooking_Ingenieros
:
No description available.
GSoC-Proposal
:
GSoC proposal for GNU Radio
LogicMine
:
An evolved implementation of the mediator pattern, helping you build highly maintainable systems
toutiao
:
仿照今日头条toutiao.com做的一个Java web项目，是一个信息流+社交的web应用。使用SpringBoot+Mybatis+velocity开发。数据库使用了redis和mysql，同时加入了异步消息等进阶功能。
// repository documentation
Was this content helpful?
★ 0
(0 ratings)
Select Rating:
★
★
★
★
★
Submit Feedback
Recent Feedback
×
Download README
Do you want to download the
README.md
file for
go-tokenizer
?
Download (.md)
# Text Tokenizer for Golang  [](https://goreportcard.com/report/github.com/euskadi31/go-tokenizer) | Branch | Status | Coverage | | ------ | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- | | master | [](https://github.com/euskadi31/go-tokenizer/actions/workflows/go.yml) | [](https://coveralls.io/github/euskadi31/go-tokenizer?branch=master) | ```sh go get -u github.com/euskadi31/go-tokenizer/v3 ``` ```go import ( "fmt" "github.com/euskadi31/go-tokenizer/v3" ) func main() { t := tokenizer.New() tokens := t.Tokenize("I believe life is an intelligent thing: that things aren't random.") fmt.Print(tokens) // []string{"I", "believe", "life", "is", "an", "intelligent", "thing", "that", "things", "aren't", "random"} } ``` ## License go-tokenizer is licensed under [the MIT license](LICENSE.md).
