Package Usage: go: github.com/blevesearch/segment
Package segment is a library for performing Unicode Text Segmentation
as described in Unicode Standard Annex #29 http://www.unicode.org/reports/tr29/
Currently only segmentation at Word Boundaries is supported.
The functionality is exposed in two ways:
1. You can use a bufio.Scanner with the SplitWords implementation of SplitFunc.
The SplitWords function will identify the appropriate word boundaries in the input
text and the Scanner will return tokens at the appropriate place.
2. Sometimes you would also like information returned about the type of token.
To do this we have introduce a new type named Segmenter. It works just like Scanner
but additionally a token type is returned.
3 versions
Latest release: over 2 years ago
557 dependent packages
View more package details: https://packages.ecosystem.code.gouv.fr/registries/proxy.golang.org/packages/github.com/blevesearch/segment