You cannot select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
8 lines
313 B
Go
8 lines
313 B
Go
5 years ago
|
// Package tokenizer implements file tokenization used by the enry content
|
||
|
// classifier. This package is an implementation detail of enry and should not
|
||
|
// be imported by other packages.
|
||
|
package tokenizer
|
||
|
|
||
|
// ByteLimit defines the maximum prefix of an input text that will be tokenized.
|
||
|
const ByteLimit = 100000
|