我使用以下代码尝试通过LlamaIndex库从LlamaCPP获取响应。我的模型存储在本地gguf文件中。由于我的VRAM有限，我尝试在CPU上进行推理。我的程序会打印出初始化代码（如下所示），但随后会无限期挂起且不产生任何响应。
import jsonfrom llama_index.llms.llama_cpp import LlamaCPPMODEL_URL = "https://huggingface.co/TheBloke/Llama-2-13B-chat-GGUF/resolve/main/llama-2-13b-chat.Q4_0.gguf"MODEL_PATH = Nonewith open("./paths.json", "r") as f: paths = json.load(f) if "llama-2-13b-chat" in paths: MODEL_URL = None MODEL_PATH = paths["llama-2-13b-chat"]llm = LlamaCPP( model_url=MODEL_URL, model_path=MODEL_PATH, temperature=0.1, max_new_tokens=256, context_window=3900, model_kwargs={"n_gpu_layers": 0}, # 使用CPU进行推理 verbose=True,)response = llm.complete("Hello, how are you?")print(str(response))输出：初始化后，无限期挂起。我期望的输出是打印出详细的初始化信息，然后是LLM的响应，最后终止程序。
llama_model_loader: loaded meta data with 19 key-value pairs and 363 tensors from ../models/llama-2-13b-chat.Q4_0.gguf (version GGUF V2)llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.llama_model_loader: - kv 0: general.architecture str = llamallama_model_loader: - kv 1: general.name str = LLaMA v2llama_model_loader: - kv 2: llama.context_length u32 = 4096llama_model_loader: - kv 3: llama.embedding_length u32 = 5120llama_model_loader: - kv 4: llama.block_count u32 = 40llama_model_loader: - kv 5: llama.feed_forward_length u32 = 13824llama_model_loader: - kv 6: llama.rope.dimension_count u32 = 128llama_model_loader: - kv 7: llama.attention.head_count u32 = 40llama_model_loader: - kv 8: llama.attention.head_count_kv u32 = 40llama_model_loader: - kv 9: llama.attention.layer_norm_rms_epsilon f32 = 0.000010llama_model_loader: - kv 10: general.file_type u32 = 2llama_model_loader: - kv 11: tokenizer.ggml.model str = llamallama_model_loader: - kv 12: tokenizer.ggml.tokens arr[str,32000] = ["<unk>", "<s>", "</s>", "<0x00>", "<...llama_model_loader: - kv 13: tokenizer.ggml.scores arr[f32,32000] = [0.000000, 0.000000, 0.000000, 0.0000...llama_model_loader: - kv 14: tokenizer.ggml.token_type arr[i32,32000] = [2, 3, 3, 6, 6, 6, 6, 6, 6, 6, 6, 6, ...llama_model_loader: - kv 15: tokenizer.ggml.bos_token_id u32 = 1llama_model_loader: - kv 16: tokenizer.ggml.eos_token_id u32 = 2llama_model_loader: - kv 17: tokenizer.ggml.unknown_token_id u32 = 0llama_model_loader: - kv 18: general.quantization_version u32 = 2llama_model_loader: - type f32: 81 tensorsllama_model_loader: - type q4_0: 281 tensorsllama_model_loader: - type q6_K: 1 tensorsllm_load_vocab: special tokens definition check successful ( 259/32000 ).llm_load_print_meta: format = GGUF V2llm_load_print_meta: arch = llamallm_load_print_meta: vocab type = SPMllm_load_print_meta: n_vocab = 32000llm_load_print_meta: n_merges = 0llm_load_print_meta: n_ctx_train = 4096llm_load_print_meta: n_embd = 5120llm_load_print_meta: n_head = 40llm_load_print_meta: n_head_kv = 40llm_load_print_meta: n_layer = 40llm_load_print_meta: n_rot = 128llm_load_print_meta: n_embd_head_k = 128llm_load_print_meta: n_embd_head_v = 128llm_load_print_meta: n_gqa = 1llm_load_print_meta: n_embd_k_gqa = 5120llm_load_print_meta: n_embd_v_gqa = 5120llm_load_print_meta: f_norm_eps = 0.0e+00llm_load_print_meta: f_norm_rms_eps = 1.0e-05llm_load_print_meta: f_clamp_kqv = 0.0e+00llm_load_print_meta: f_max_alibi_bias = 0.0e+00llm_load_print_meta: f_logit_scale = 0.0e+00llm_load_print_meta: n_ff = 13824llm_load_print_meta: n_expert = 0llm_load_print_meta: n_expert_used = 0llm_load_print_meta: causal attn = 1llm_load_print_meta: pooling type = 0llm_load_print_meta: rope type = 0llm_load_print_meta: rope scaling = linearllm_load_print_meta: freq_base_train = 10000.0llm_load_print_meta: freq_scale_train = 1llm_load_print_meta: n_yarn_orig_ctx = 4096llm_load_print_meta: rope_finetuned = unknownllm_load_print_meta: ssm_d_conv = 0llm_load_print_meta: ssm_d_inner = 0llm_load_print_meta: ssm_d_state = 0llm_load_print_meta: ssm_dt_rank = 0llm_load_print_meta: model type = 13Bllm_load_print_meta: model ftype = Q4_0llm_load_print_meta: model params = 13.02 Bllm_load_print_meta: model size = 6.86 GiB (4.53 BPW) llm_load_print_meta: general.name = LLaMA v2llm_load_print_meta: BOS token = 1 '<s>'llm_load_print_meta: EOS token = 2 '</s>'llm_load_print_meta: UNK token = 0 '<unk>'llm_load_print_meta: LF token = 13 '<0x0A>'llm_load_tensors: ggml ctx size = 0.18 MiBllm_load_tensors: CPU buffer size = 7023.90 MiB...................................................................................................llama_new_context_with_model: n_ctx = 4096llama_new_context_with_model: n_batch = 512llama_new_context_with_model: n_ubatch = 512llama_new_context_with_model: flash_attn = 0llama_new_context_with_model: freq_base = 10000.0llama_new_context_with_model: freq_scale = 1llama_kv_cache_init: CPU KV buffer size = 3200.00 MiBllama_new_context_with_model: KV self size = 3200.00 MiB, K (f16): 1600.00 MiB, V (f16): 1600.00 MiBllama_new_context_with_model: CPU output buffer size = 0.12 MiBllama_new_context_with_model: CPU compute buffer size = 368.01 MiBllama_new_context_with_model: graph nodes = 1286llama_new_context_with_model: graph splits = 1AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 | Model metadata: {'tokenizer.ggml.unknown_token_id': '0', 'tokenizer.ggml.eos_token_id': '2', 'general.architecture': 'llama', 'llama.context_length': '4096', 'general.name': 'LLaMA v2', 'llama.embedding_length': '5120', 'llama.feed_forward_length': '13824', 'llama.attention.layer_norm_rms_epsilon': '0.000010', 'llama.rope.dimension_count': '128', 'llama.attention.head_count': '40', 'tokenizer.ggml.bos_token_id': '1', 'llama.block_count': '40', 'llama.attention.head_count_kv': '40', 'general.quantization_version': '2', 'tokenizer.ggml.model': 'llama', 'general.file_type': '2'}Using fallback chat format: llama-2我的RAM使用量最终达到9.5GB/16，我的CPU使用率约为50%。如果有任何关于为什么会发生这种情况的见解，将不胜感激。
回答：
尝试使用流式输出。模型正在生成响应，但没有GPU的情况下速度非常慢。总的来说，13B模型相当大，如果它们占用超过10GB的RAM也是正常的。
response_iter = llm.stream_complete("Can you write me a poem about fast cars?")for response in response_iter: print(response.delta, end="", flush=True)还可以考虑使用更小的模型来加快输出速度：
MODEL_URL = "https://huggingface.co/TheBloke/Llama-2-7B-Chat-GGUF/resolve/main/llama-2-7b-chat.Q4_K_M.gguf"