V2EX = way to explore
V2EX 是一个关于分享和探索的地方
现在注册
已注册用户请  登录
The Go Programming Language
http://golang.org/
Go Playground
Go Projects
Revel Web Framework
vway
V2EX  ›  Go 编程语言

Gse v0.50.0 发布了, Go 高性能分词, 大幅度更新

  •  
  •   vway ·
    vcaesar · 2020-05-26 13:25:25 +08:00 · 2967 次点击
    这是一个创建于 1642 天前的主题,其中的信息可能已经有所发展或是发生改变。

    Go 语言高效分词, 支持英文、中文、日文等

    该版本主要增加: AlphaNum 拉丁字母和数字分词设置, RemoveToken, gse.New(), 分词 ToLower 设置; Tirm 清除标点符号和空格, TextFreq 设置默认加载 frenquency, cutDAGNoHMM 和最短路径加动态规划配置, 支持 ", " 分割字典文件等功能.

    增加更多示例、测试和 benchmark 代码, 修复 bug.

    项目地址: https://github.com/go-ego/gse

    Nisqually Glacier

    Add

    add alphanum split words support use variable to control default frequency

    add gse.New() function and update test code, add New() function AlphaNum set support

    add ToLower set support and update code

    add remove token support and test code, add addToken and removeToken benchmark test, update benchmark, example.go and godoc add more test code

    add English example code, Add English token test code and Update README.md

    update segment test code update segment test code, add more test and remove unless code

    add Trim punct and space function support, add Trim test code, add Trim benchmark test code, add Trim example code

    update Trim function not exclude "include other char string"

    update DictPaths() allow use ", " to separated dictionary files

    add cutDAGNoHMM support and test code, add Cut() cutDAGNoHMM option support, add test and benchmark code

    add python and node binding

    Updtae

    update gse code export splitTextToWords function export lookupTokens function

    update example code update data re base code

    add Find token example code update English example code

    update example code use mixed text update dictionary example code

    update code fixed example print update hmm test and example code

    update gse server code update server code add hmm support, update jquery.min.js to v3.5.0 update server.go listen log print

    update code and README.md Update README.md, add binding...

    update godoc update load model godoc update godoc and var name

    update go mod pkg and version remove unless go.sum hash update go mod and remove vendor

    add github workflow support update go.yml update and fmt circle.yml, travis.yml add CI go1.13x support remove go1.11.x

    rename utils.go to seg_utils.go

    add more benchmark support add hmm benchmark test support add segmenter benchmark test code

    add idf and pos files update issue and pull request template

    update segment test code use expect update alpha num test code

    4 条回复    2020-05-27 12:45:05 +08:00
    cy476571989
        1
    cy476571989  
       2020-05-26 14:08:16 +08:00
    赞👍,这个项目挺有用的。我看到目前只有一个简单的 readme, 还没有具体的文档,后面如果需要翻译文档的话,可以试试:www.breword.com ,能够让项目作者非常轻松的维护译文文档。
    vway
        2
    vway  
    OP
       2020-05-26 15:29:28 +08:00
    whyw
        3
    whyw  
       2020-05-26 22:48:59 +08:00
    👍, 支持
    vway
        4
    vway  
    OP
       2020-05-27 12:45:05 +08:00
    @whyw 🤝
    关于   ·   帮助文档   ·   博客   ·   API   ·   FAQ   ·   实用小工具   ·   2638 人在线   最高记录 6679   ·     Select Language
    创意工作者们的社区
    World is powered by solitude
    VERSION: 3.9.8.5 · 24ms · UTC 02:46 · PVG 10:46 · LAX 18:46 · JFK 21:46
    Developed with CodeLauncher
    ♥ Do have faith in what you're doing.