Discussions

• 1 You must be logged in to vote

  🙏

  llava-16-gguf-conversion failed

  lyp-liu asked Dec 10, 2024 in Q&A · Closed · Unanswered

  

  

  1
• 1 You must be logged in to vote

  🙏

  According to pre-built binaries of Ubuntu, since which version is built for Ubuntu 24.04?

  HuangShumin asked Dec 10, 2024 in Q&A · Unanswered

  

  

  0
• 1 You must be logged in to vote

  💬

  Support for H100 "Transformer Engine"

  simusid started Dec 9, 2024 in General

  

  

  0
• 1 You must be logged in to vote

  🙏

  Challenges in Quantizing llama.cpp Models on Windows

  jasonsu123 asked Dec 9, 2024 in Q&A · Unanswered

  

  

  4
• 1 You must be logged in to vote

  🙏

  how to merge gguf files with serval sub gguf files?

  cqray1990 asked Dec 4, 2024 in Q&A · Unanswered

  

  

  1
• 1 You must be logged in to vote

  🙏

  What is the main function of ggml_backend_graph_compute() in clip_image_batch_encode?

  perp asked Dec 9, 2024 in Q&A · Unanswered

  

  

  0
• 1 You must be logged in to vote

  🙏

  error LNK2019

  lightrattle asked Dec 9, 2024 in Q&A · Unanswered

  

  

  0
• 1 You must be logged in to vote

  🙏

  Another > log question

  sendyomonay asked Dec 8, 2024 in Q&A · Unanswered

  

  

  0
• 2 You must be logged in to vote

  💡

  Speculative decoding potential for running big LLMs on consumer grade GPUs efficiently

  steampunque started Nov 23, 2024 in Ideas

  

  

  

  

  19
• 2 You must be logged in to vote

  💬

  Yarn rope settings only appear in deepseek and phi3 code, any reason?

  bartowski1182 started Nov 13, 2024 in General

  

  

  

  

  5
• 1 You must be logged in to vote

  🙏

  Could Someone Help me to Interpret Speed for Speculative Decoding?

  chigkim asked Dec 8, 2024 in Q&A · Unanswered

  

  

  0
• 1 You must be logged in to vote

  💬

  Status of Qwen*ForCausalLm ? (re: Support for MiniCPM changes)

  robbiemu started Dec 4, 2024 in General

  

  

  4
• 26 You must be logged in to vote

  🙌

  An OpenAI Compatible Web Server for llama.cpp

  abetlen started Apr 5, 2023 in Show and tell

  

  

  

  

  24
• 1 You must be logged in to vote

  🙌

  cyllama: a thin cython wrapper around llama.cpp

  shakfu started Dec 4, 2024 in Show and tell

  

  

  0
• 1 You must be logged in to vote

  🙏

  How to convert a model with a pooler + dense structure into GGUF?

  viosay asked Dec 7, 2024 in Q&A · Unanswered

  

  

  0
• 1 You must be logged in to vote

  🙏

  Token Latency decreases from 6s/token to 130ms/token with -ngl set to 1 on Mac M1 Metal GPU.

  Lizonghang asked Dec 3, 2024 in Q&A · Closed · Unanswered

  

  

  1
• 1 You must be logged in to vote

  🙏

  How can I extract the raw weights of each layer and also split them up into sub components? (like q_proj, input RMS normalization etc.)

  dawidlouisbekker asked Dec 6, 2024 in Q&A · Answered

  

  

  2
• 1 You must be logged in to vote

  🙏

  Loving the simplified cmake files! Just a quick Q on switching CPU & GPU backends

  freebiesoft asked Dec 6, 2024 in Q&A · Answered

  

  

  1
• 1 You must be logged in to vote

  🙏

  perf : CUDA FA is slower with Gemma models. Is this expected?

  question Further information is requested

  ggerganov asked Dec 6, 2024 in Q&A · Unanswered

  

  

  3
• 1 You must be logged in to vote

  🙏

  Can't use ggml-cpu

  lightrattle asked Dec 6, 2024 in Q&A · Closed · Unanswered

  

  

  0
• 3 You must be logged in to vote

  💡

  SAGE ATTENTION: 2.1 TIMES FASTER THAN FLASH ATTENTION 2 AND 2.7 TIMES FASTER THAN XFORMERS

  joseph777111 started Oct 15, 2024 in Ideas

  

  

  

  

  4
• 1 You must be logged in to vote

  🙏

  How to further quantize GGUF to Q4 format using llama.cpp?

  jasonsu123 asked Dec 5, 2024 in Q&A · Unanswered

  

  

  0
• 1 You must be logged in to vote

  🙏

  Difference in how embeddings are calculated vs Ollama

  eugeniosegala asked Dec 4, 2024 in Q&A · Unanswered

  

  

  0
• 1 You must be logged in to vote

  🙏

  Getting started - Trying to chat with Llama-3.2-3B on a M1 Macbook Air (CPU without ANY GPUs)

  jasonkaplan79 asked Dec 3, 2024 in Q&A · Unanswered

  

  

  0
• 1 You must be logged in to vote

  🙏

  Output shows <|endoftext|> tokens and

  UltraWelfare asked Nov 30, 2024 in Q&A · Unanswered

  

  

  

  

  3