Go 语言高效分词, 支持英文、中文、日文等
该版本主要增加: AlphaNum 拉丁字母和数字分词设置, RemoveToken, gse.New(), 分词 ToLower 设置; Tirm 清除标点符号和空格, TextFreq 设置默认加载 frenquency, cutDAGNoHMM 和最短路径加动态规划配置, 支持 ", " 分割字典文件等功能.
增加更多示例、测试和 benchmark 代码, 修复 bug.
项目地址: https://github.com/go-ego/gse
add alphanum split words support use variable to control default frequency
add gse.New() function and update test code, add New() function AlphaNum set support
add ToLower set support and update code
add remove token support and test code, add addToken and removeToken benchmark test, update benchmark, example.go and godoc add more test code
add English example code, Add English token test code and Update README.md
update segment test code update segment test code, add more test and remove unless code
add Trim punct and space function support, add Trim test code, add Trim benchmark test code, add Trim example code
update Trim function not exclude "include other char string"
update DictPaths() allow use ", " to separated dictionary files
add cutDAGNoHMM support and test code, add Cut() cutDAGNoHMM option support, add test and benchmark code
add python and node binding
update gse code export splitTextToWords function export lookupTokens function
update example code update data re base code
add Find token example code update English example code
update example code use mixed text update dictionary example code
update code fixed example print update hmm test and example code
update gse server code update server code add hmm support, update jquery.min.js to v3.5.0 update server.go listen log print
update code and README.md Update README.md, add binding...
update godoc update load model godoc update godoc and var name
update go mod pkg and version remove unless go.sum hash update go mod and remove vendor
add github workflow support update go.yml update and fmt circle.yml, travis.yml add CI go1.13x support remove go1.11.x
rename utils.go to seg_utils.go
add more benchmark support add hmm benchmark test support add segmenter benchmark test code
add idf and pos files update issue and pull request template
update segment test code use expect update alpha num test code
1
cy476571989 2020-05-26 14:08:16 +08:00
赞👍,这个项目挺有用的。我看到目前只有一个简单的 readme, 还没有具体的文档,后面如果需要翻译文档的话,可以试试:www.breword.com ,能够让项目作者非常轻松的维护译文文档。
|
2
vway OP @cy476571989 ??
|
3
whyw 2020-05-26 22:48:59 +08:00
👍, 支持
|