tokenizers - Fast, Consistent Tokenization of Natural Language Text
Convert natural language text into tokens. Includes tokenizers for shingled n-grams, skip n-grams, words, word stems, sentences, paragraphs, characters, shingled characters, lines, Penn Treebank, regular expressions, as well as functions for counting characters, words, and sentences, and a function for splitting longer texts into separate documents, each with the same number of words. The tokenizers have a consistent interface, and the package is built on the 'stringi' and 'Rcpp' packages for fast yet correct tokenization in 'UTF-8'.
Last updated
nlppeer-reviewedtext-miningtokenizercpp
13.52 score 188 stars 81 dependents 1.3k scripts 46k downloadslinevis - Interactive Time Series Visualizations
Create interactive time series visualizations. 'linevis' includes an extensive API to manipulate time series after creation, and supports getting data out of the visualization. Based on the 'timevis' package and the 'vis.js' Timeline 'JavaScript' library <https://visjs.github.io/vis-timeline/docs/graph2d/>.
Last updated
3.70 score 3 scripts 228 downloads