-
Notifications
You must be signed in to change notification settings - Fork 1
/
Copy pathREADME.Rmd
74 lines (53 loc) · 1.8 KB
/
README.Rmd
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
---
output: github_document
---
<!-- README.md is generated from README.Rmd. Please edit that file -->
```{r, include = FALSE}
knitr::opts_chunk$set(
collapse = TRUE,
comment = "#>",
fig.path = "man/figures/README-",
out.width = "100%"
)
```
# wordpiece
<!-- badges: start -->
[![Lifecycle: experimental](https://img.shields.io/badge/lifecycle-experimental-orange.svg)](https://lifecycle.r-lib.org/articles/stages.html#experimental)
<!-- badges: end -->
The goal of wordpiece is to allow for easy text tokenization using a wordpiece vocabulary.
## Installation
You can install the released version of wordpiece from [CRAN](https://CRAN.R-project.org) with:
``` r
install.packages("wordpiece")
```
And the development version from [GitHub](https://github.com/) with:
``` r
# install.packages("devtools")
devtools::install_github("macmillancontentscience/wordpiece")
```
# Examples
This package can be used to tokenize text for modeling.
A common usecase would be to tokenize all text in a data.frame or other tibble.
```{r tokenize_df}
library(wordpiece)
library(dplyr, warn.conflicts = FALSE)
df_tokenized <- tibble(
text = c(
"I like tacos.",
"I like apples with cheese.",
"The unaffable coder wrote incorrect examples."
)
) %>%
mutate(
tokens = wordpiece_tokenize(text)
)
df_tokenized
df_tokenized$tokens[[1]]
```
## Code of Conduct
Please note that the wordpiece project is released with a [Contributor Code of Conduct](https://contributor-covenant.org/version/2/0/CODE_OF_CONDUCT.html).
By contributing to this project, you agree to abide by its terms.
## Disclaimer
This is not an officially supported Macmillan Learning product.
## Contact information
Questions or comments should be directed to Jonathan Bratt ([email protected]) and Jon Harmon ([email protected]).