Add more tokenizer tests (#3742)

Browse Source

* Add more tokenizer tests

* Add starcoder

* Update test vocab files

* Restrict bpe tokenizer tests to unicode planes

* Update comment

* Comment cosmetics

* Remove bloom vocab/test

...

This commit is contained in:

Galunid authored and GitHub committed 2023-10-24 09:17:17 +02:00

1 parent 469c9addef

commit daab3d7f45

6 files changed

+16 -3

Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL

Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

models/ggml-vocab-baichuan.gguf

BIN

Binary file not shown.

models/ggml-vocab-gpt-neox.gguf

BIN

Binary file not shown.

models/ggml-vocab-refact.gguf

BIN

Binary file not shown.

models/ggml-vocab-starcoder.gguf

BIN

Binary file not shown.