common : only load backends when required (#22290)

* common : only load backends when required

Signed-off-by: Adrien Gallouët <angt@huggingface.co>

* llama : call ggml_backend_load_all() directly from llama_backend_init()

Signed-off-by: Adrien Gallouët <angt@huggingface.co>

* Add ggml_backend_load_all() where llama_backend_init() is not used

Signed-off-by: Adrien Gallouët <angt@huggingface.co>

---------

Signed-off-by: Adrien Gallouët <angt@huggingface.co>

tests/test-state-restore-fragmented.cpp

@@ -29,6 +29,9 @@ int main(int argc, char ** argv) {
     }
 
     // init
+
+    ggml_backend_load_all();
+
     common_init_result_ptr llama_init = common_init_from_params(params);
 
     llama_model * model = llama_init->model();