llama : move vocab, grammar and sampling into separate files (#8508)

Browse Source

* llama : move sampling code into llama-sampling

ggml-ci

* llama : move grammar code into llama-grammar

ggml-ci

* cont

ggml-ci

* cont : pre-fetch rules

* cont

ggml-ci

* llama : deprecate llama_sample_grammar

* llama : move tokenizers into llama-vocab

ggml-ci

* make : update llama.cpp deps [no ci]

* llama : redirect external API to internal APIs

ggml-ci

* llama : suffix the internal APIs with "_impl"

ggml-ci

* llama : clean-up

...

This commit is contained in:

Georgi Gerganov authored and GitHub committed 2024-07-23 13:10:17 +03:00

1 parent 751fcfc6c3

commit 938943cdbf

18 files changed

+3521 -2968

Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL

Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

src/llama-vocab.cpp

+1721

File diff suppressed because it is too large. Load diff