Fix newline and apostrophe handling for BPE #574
Conversation
sayanshaw24
changed the title
| @@ -3,6 +3,7 @@ | |||
| import unittest | |||
|
|
|||
| import numpy as np | |||
| import ftfy | |||
There was a problem hiding this comment.
why do you need import it here? I think it was used by HF tokenizers.
There was a problem hiding this comment.
yeah but they did not check in that PR so HF natively does not import it so it fails without that import locally - might still pass on the ci but it will cause issues for locally testing in the future so i think it is better to have it in the code.
There was a problem hiding this comment.
not quite follow here, If you want other be aware that ftfy should be installed to to avoid the test failure. You may check ftfy installation here and raise a warning it is missed.
| @@ -3,6 +3,7 @@ | |||
| import unittest | |||
|
|
|||
| import numpy as np | |||
| import ftfy | |||
There was a problem hiding this comment.
not quite follow here, If you want other be aware that ftfy should be installed to to avoid the test failure. You may check ftfy installation here and raise a warning it is missed.
|
sounds good, will add the ftfy warning in a separate PR. |
fixes newline and apostrophe handling for issues discovered by long text testing.
Update: CLIPTokenizer tokenizes apostrophes separately (['you', ''', 're']) but CLIPTokenizerFast tokenizes together (['you', ''re']), unless ftfy is installed: huggingface/transformers#22166.