In my tests I already figured out that —n-cpu-moe 18 is a good setting to have 2 parallel endpoints with a context size of about 54k. But still the prompt processing is slow. So, I wanted to try to increase the performance a bit.

Check the best value for -b and -ub.

adrian@bigdelli:~$ CUDA_VISIBLE_DEVICES=0 ~/llama.cpp/build/bin/llama-bench -hf bartowski/google_gemma-4-26B-A4B-it-GGUF:Q5_K_S —n-cpu-moe 18 -fa 1 -n 128 -b 512,1024,2048 -ub 128,256,512,1024 ggml_cuda_init: found 1 CUDA devices (Total VRAM: 11833 MiB): Device 0: NVIDIA GeForce RTX 2060, compute capability 7.5, VMM: yes, VRAM: 11833 MiB

model	size	params	backend	ngl	n_cpu_moe	n_batch	n_ubatch	fa	test	t/s
———————————————	————-:	————-:	—————	—:	————-:	———:	———-:	-:	———————:	—————————-:
gemma4 ?B Q5_K – Small	16.87 GiB	25.23 B	CUDA	99	18	512	128	1	pp512	69.90 ± 4.47
gemma4 ?B Q5_K – Small	16.87 GiB	25.23 B	CUDA	99	18	512	128	1	tg128	24.14 ± 0.15
gemma4 ?B Q5_K – Small	16.87 GiB	25.23 B	CUDA	99	18	512	256	1	pp512	119.58 ± 2.77
gemma4 ?B Q5_K – Small	16.87 GiB	25.23 B	CUDA	99	18	512	256	1	tg128	24.17 ± 0.21
gemma4 ?B Q5_K – Small	16.87 GiB	25.23 B	CUDA	99	18	512	512	1	pp512	208.32 ± 5.94
gemma4 ?B Q5_K – Small	16.87 GiB	25.23 B	CUDA	99	18	512	512	1	tg128	24.24 ± 0.25
gemma4 ?B Q5_K – Small	16.87 GiB	25.23 B	CUDA	99	18	512	1024	1	pp512	204.08 ± 6.59
gemma4 ?B Q5_K – Small	16.87 GiB	25.23 B	CUDA	99	18	512	1024	1	tg128	24.31 ± 0.30
gemma4 ?B Q5_K – Small	16.87 GiB	25.23 B	CUDA	99	18	1024	128	1	pp512	70.91 ± 2.03
gemma4 ?B Q5_K – Small	16.87 GiB	25.23 B	CUDA	99	18	1024	128	1	tg128	24.23 ± 0.28
gemma4 ?B Q5_K – Small	16.87 GiB	25.23 B	CUDA	99	18	1024	256	1	pp512	118.23 ± 4.00
gemma4 ?B Q5_K – Small	16.87 GiB	25.23 B	CUDA	99	18	1024	256	1	tg128	24.34 ± 0.24
gemma4 ?B Q5_K – Small	16.87 GiB	25.23 B	CUDA	99	18	1024	512	1	pp512	203.31 ± 3.21
gemma4 ?B Q5_K – Small	16.87 GiB	25.23 B	CUDA	99	18	1024	512	1	tg128	24.21 ± 0.15
gemma4 ?B Q5_K – Small	16.87 GiB	25.23 B	CUDA	99	18	1024	1024	1	pp512	205.94 ± 9.56
gemma4 ?B Q5_K – Small	16.87 GiB	25.23 B	CUDA	99	18	1024	1024	1	tg128	24.24 ± 0.12
gemma4 ?B Q5_K – Small	16.87 GiB	25.23 B	CUDA	99	18	2048	128	1	pp512	71.78 ± 2.65
gemma4 ?B Q5_K – Small	16.87 GiB	25.23 B	CUDA	99	18	2048	128	1	tg128	24.23 ± 0.26
gemma4 ?B Q5_K – Small	16.87 GiB	25.23 B	CUDA	99	18	2048	256	1	pp512	124.82 ± 4.47
gemma4 ?B Q5_K – Small	16.87 GiB	25.23 B	CUDA	99	18	2048	256	1	tg128	24.22 ± 0.20
gemma4 ?B Q5_K – Small	16.87 GiB	25.23 B	CUDA	99	18	2048	512	1	pp512	205.93 ± 6.00
gemma4 ?B Q5_K – Small	16.87 GiB	25.23 B	CUDA	99	18	2048	512	1	tg128	24.27 ± 0.41
gemma4 ?B Q5_K – Small	16.87 GiB	25.23 B	CUDA	99	18	2048	1024	1	pp512	204.90 ± 4.04
gemma4 ?B Q5_K – Small	16.87 GiB	25.23 B	CUDA	99	18	2048	1024	1	tg128	24.31 ± 0.27

Recheck the speeds with different amounts of MOE layers on the CPU.

adrian@bigdelli:~$ CUDA_VISIBLE_DEVICES=0 ~/llama.cpp/build/bin/llama-bench -hf bartowski/google_gemma-4-26B-A4B-it-GGUF:Q5_K_S —n-cpu-moe 14,15,16,17,18,19,20,21 -fa 1,0 -n 128 -b 1024 -ub 1024 ggml_cuda_init: found 1 CUDA devices (Total VRAM: 11833 MiB): Device 0: NVIDIA GeForce RTX 2060, compute capability 7.5, VMM: yes, VRAM: 11833 MiB

model	size	params	backend	ngl	n_cpu_moe	n_batch	n_ubatch	fa	test	t/s
———————————————	————-:	————-:	—————	—:	————-:	———:	———-:	-:	———————:	—————————-:
gemma4 ?B Q5_K – Small	16.87 GiB	25.23 B	CUDA	99	14	1024	1024	1	pp512	223.06 ± 9.02
gemma4 ?B Q5_K – Small	16.87 GiB	25.23 B	CUDA	99	14	1024	1024	1	tg128	29.18 ± 0.41
gemma4 ?B Q5_K – Small	16.87 GiB	25.23 B	CUDA	99	14	1024	1024	0	pp512	222.99 ± 4.69
gemma4 ?B Q5_K – Small	16.87 GiB	25.23 B	CUDA	99	14	1024	1024	0	tg128	28.79 ± 0.29
gemma4 ?B Q5_K – Small	16.87 GiB	25.23 B	CUDA	99	15	1024	1024	1	pp512	224.38 ± 6.31
gemma4 ?B Q5_K – Small	16.87 GiB	25.23 B	CUDA	99	15	1024	1024	1	tg128	27.55 ± 0.27
gemma4 ?B Q5_K – Small	16.87 GiB	25.23 B	CUDA	99	15	1024	1024	0	pp512	219.00 ± 5.54
gemma4 ?B Q5_K – Small	16.87 GiB	25.23 B	CUDA	99	15	1024	1024	0	tg128	27.57 ± 0.38
gemma4 ?B Q5_K – Small	16.87 GiB	25.23 B	CUDA	99	16	1024	1024	1	pp512	212.68 ± 3.87
gemma4 ?B Q5_K – Small	16.87 GiB	25.23 B	CUDA	99	16	1024	1024	1	tg128	26.23 ± 0.24
gemma4 ?B Q5_K – Small	16.87 GiB	25.23 B	CUDA	99	16	1024	1024	0	pp512	209.31 ± 7.52
gemma4 ?B Q5_K – Small	16.87 GiB	25.23 B	CUDA	99	16	1024	1024	0	tg128	26.15 ± 0.27
gemma4 ?B Q5_K – Small	16.87 GiB	25.23 B	CUDA	99	17	1024	1024	1	pp512	208.96 ± 3.50
gemma4 ?B Q5_K – Small	16.87 GiB	25.23 B	CUDA	99	17	1024	1024	1	tg128	25.07 ± 0.25
gemma4 ?B Q5_K – Small	16.87 GiB	25.23 B	CUDA	99	17	1024	1024	0	pp512	209.88 ± 9.27
gemma4 ?B Q5_K – Small	16.87 GiB	25.23 B	CUDA	99	17	1024	1024	0	tg128	24.79 ± 0.33
gemma4 ?B Q5_K – Small	16.87 GiB	25.23 B	CUDA	99	18	1024	1024	1	pp512	207.52 ± 6.04
gemma4 ?B Q5_K – Small	16.87 GiB	25.23 B	CUDA	99	18	1024	1024	1	tg128	24.18 ± 0.20
gemma4 ?B Q5_K – Small	16.87 GiB	25.23 B	CUDA	99	18	1024	1024	0	pp512	209.03 ± 9.32
gemma4 ?B Q5_K – Small	16.87 GiB	25.23 B	CUDA	99	18	1024	1024	0	tg128	23.76 ± 0.22
gemma4 ?B Q5_K – Small	16.87 GiB	25.23 B	CUDA	99	19	1024	1024	1	pp512	199.83 ± 5.87
gemma4 ?B Q5_K – Small	16.87 GiB	25.23 B	CUDA	99	19	1024	1024	1	tg128	23.45 ± 0.24
gemma4 ?B Q5_K – Small	16.87 GiB	25.23 B	CUDA	99	19	1024	1024	0	pp512	197.79 ± 3.33
gemma4 ?B Q5_K – Small	16.87 GiB	25.23 B	CUDA	99	19	1024	1024	0	tg128	22.94 ± 0.20
gemma4 ?B Q5_K – Small	16.87 GiB	25.23 B	CUDA	99	20	1024	1024	1	pp512	190.75 ± 3.30
gemma4 ?B Q5_K – Small	16.87 GiB	25.23 B	CUDA	99	20	1024	1024	1	tg128	22.70 ± 0.22
gemma4 ?B Q5_K – Small	16.87 GiB	25.23 B	CUDA	99	20	1024	1024	0	pp512	194.97 ± 6.65
gemma4 ?B Q5_K – Small	16.87 GiB	25.23 B	CUDA	99	20	1024	1024	0	tg128	22.22 ± 0.15
gemma4 ?B Q5_K – Small	16.87 GiB	25.23 B	CUDA	99	21	1024	1024	1	pp512	188.35 ± 3.29
gemma4 ?B Q5_K – Small	16.87 GiB	25.23 B	CUDA	99	21	1024	1024	1	tg128	21.98 ± 0.14
gemma4 ?B Q5_K – Small	16.87 GiB	25.23 B	CUDA	99	21	1024	1024	0	pp512	189.21 ± 2.23
gemma4 ?B Q5_K – Small	16.87 GiB	25.23 B	CUDA	99	21	1024	1024	0	tg128	21.71 ± 0.17

Conclusion

After these tests and some real life tests I now stick with the following settings:

llama-server -hf bartowski/google_gemma-4-26B-A4B-it-GGUF:Q5_K_S --device CUDA0 \
  --jinja --no-mmproj-offload --reasoning off \
  -fa 1 -b 1024 -ub 1024 -ctk q8_0 -ctv q8_0 --no-mmap \
  --host 0.0.0.0 --port 8002 -ngl 99 --n-cpu-moe 18 --parallel 2 --ctx-size 108000

With each additional layer, I lose context, and because I am using hermes-agent, I should stick with those 2 connections since the agent sometimes uses sub-agents for different tasks. However, I still see some ReadTimeouts when it comes to context compaction. But this is another problem.

Such models are great, and the MOE architecture makes it good if you don’t have enough VRAM to pack it onto your GPU entirely.
But there are still some tweaks possible. So, I played with the configuration a bit.

Not everything is possible with llama-bench. Especially settings like:
—no-mmap -> will keep all CPU experts in RAM instead of loading it from the hard drive if necessary. This may give you a small performance boost but will eat your RAM

—mlock -> ensures that the kernel is not allowed to move the things into swap. In my opinion, if you will use —no-mmap then you should have enough RAM; otherwise, let llama.cpp do the memory stuff.

—no-mmproj-offload -> move mmproj file, used for image reading, to RAM instead of VRAM. I do this constantly because I rarely use pictures and need the memory for the context.

My final setup:

llama-server -m ~/models/cache/bartowski_Qwen_Qwen3.5-35B-A3B-GGUF_Qwen_Qwen3.5-35B-A3B-Q5_K_S.gguf —device CUDA0 —jinja —no-mmproj-offload —reasoning off —parallel 2 —n-cpu-moe 28 -sm layer -ngl 99 -fa 1 -ub 512 -b 512 -ctk q8_0 -ctv q8_0 —no-mmap —mmproj ~/models/cache/bartowski_Qwen_Qwen3.5-35B-A3B-GGUF_mmproj-Qwen_Qwen3.5-35B-A3B-f16.gguf —host 0.0.0.0 —port 8002

It gives me a shared context of roughly 65k over 2 connections. The speed cuts usually in half when both are used, but this is still faster than having only one. My hermes-agent is now running a bit faster and more reliably. I get up to 28 tks tg per second. This surprised me, as with other model providers, I rarely came over 22, and now I am working with Q5 quantization, which works much better than Q4.

The speed decreases with bigger context, which is expectable, but still works well.

During implementation of a website. About 45632 token in context:

prompt eval time = 81623.01 ms / 6535 tokens ( 12.49 ms per token, 80.06 tokens per second)
eval time = 9194.38 ms / 99 tokens ( 92.87 ms per token, 10.77 tokens per second)
total time = 90817.39 ms / 6634 tokens

Single access via web with 16 tokens in context:

prompt eval time = 410.25 ms / 16 tokens ( 25.64 ms per token, 39.00 tokens per second)
eval time = 2506.55 ms / 64 tokens ( 39.16 ms per token, 25.53 tokens per second)
total time = 2916.80 ms / 80 tokens

Find the sweet spot in my Precision 7520 with eGPU RTX 2060 12GB VRAM via M.2 to OcuLink connection.

I used llama.cpp and played around with the -ncmoe flag to get my sweet spot of tokens per second. For speed measurement, I this time just used the web page provided by the server and asked for 20 prime numbers each time. The goal was to get the best speed but still a huge context window.

test	cmoe	tk\s	ctx
1	50	16.5	262144
2	40	19.1	147456
3	30-35	failed	failed
4	36	oom	4096
5	37	20.2	4096
6	38	19.3	40448
7	39	19.1	93952

Test 1
llama_memory_breakdown_print: | memory breakdown [MiB] | total free self model context compute unaccounted |
llama_memory_breakdown_print: | – CUDA0 (RTX 2060) | 11833 = 5843 + ( 5757 = 1358 + 3339 + 1060) + 231 |
llama_memory_breakdown_print: | – Host | 40969 = 40449 + 0 + 520 |

Test 2
llama_memory_breakdown_print: | memory breakdown [MiB] | total free self model context compute unaccounted |
llama_memory_breakdown_print: | – CUDA0 (RTX 2060) | 11833 = 1191 + (10417 = 7886 + 1911 + 620) + 223 |
llama_memory_breakdown_print: | – Host | 34031 = 33735 + 0 + 296

Test 3 – 4 failed

Test 5
llama_memory_breakdown_print: | memory breakdown [MiB] | total free self model context compute unaccounted |
llama_memory_breakdown_print: | – CUDA0 (RTX 2060) | 11833 = 801 + (10802 = 10334 + 126 + 342) + 228 |
llama_memory_breakdown_print: | – Host | 31229 = 31213 + 0 + 16 |

Test 6
llama_memory_breakdown_print: | memory breakdown [MiB] | total free self model context compute unaccounted |
llama_memory_breakdown_print: | – CUDA0 (RTX 2060) | 11833 = 1095 + (10508 = 9518 + 579 + 411) + 229 |
llama_memory_breakdown_print: | – Host | 32140 = 32053 + 0 + 87 |

Test 7
llama_memory_breakdown_print: | memory breakdown [MiB] | total free self model context compute unaccounted |
llama_memory_breakdown_print: | – CUDA0 (RTX 2060) | 11833 = 1147 + (10463 = 8702 + 1245 + 515) + 222 |
llama_memory_breakdown_print: | – Host | 33085 = 32894 + 0 + 191 |

llama-bench

llama-bench -m ~/models/cache/unsloth_Qwen3-Coder-Next-GGUF_Qwen3-Coder-Next-IQ4_XS.gguf —device CUDA0 -ncmoe 40 ggml_cuda_init: found 2 CUDA devices (Total VRAM: 15868 MiB): Device 0: NVIDIA GeForce RTX 2060, compute capability 7.5, VMM: yes, VRAM: 11833 MiB Device 1: Quadro M1200, compute capability 5.0, VMM: yes, VRAM: 4035 MiB

model	size	params	backend	ngl	dev	test	t/s
———————————————	————-:	————-:	—————	—:	——————	———————:	—————————-:
qwen3next 80B.A3B IQ4_XS – 4.25 bpw	39.74 GiB	79.67 B	CUDA	99	CUDA0	pp512	74.88 ± 0.82
qwen3next 80B.A3B IQ4_XS – 4.25 bpw	39.74 GiB	79.67 B	CUDA	99	CUDA0	tg128	20.02 ± 0.18

llama-bench -m ~/models/cache/unsloth_Qwen3-Coder-Next-GGUF_Qwen3-Coder-Next-IQ4_XS.gguf —device CUDA0 -ncmoe 39 ggml_cuda_init: found 2 CUDA devices (Total VRAM: 15868 MiB): Device 0: NVIDIA GeForce RTX 2060, compute capability 7.5, VMM: yes, VRAM: 11833 MiB Device 1: Quadro M1200, compute capability 5.0, VMM: yes, VRAM: 4035 MiB

model	size	params	backend	ngl	dev	test	t/s
———————————————	————-:	————-:	—————	—:	——————	———————:	—————————-:
qwen3next 80B.A3B IQ4_XS – 4.25 bpw	39.74 GiB	79.67 B	CUDA	99	CUDA0	pp512	76.09 ± 0.88
qwen3next 80B.A3B IQ4_XS – 4.25 bpw	39.74 GiB	79.67 B	CUDA	99	CUDA0	tg128	20.32 ± 0.17

Once upon a time I had the idea to use an old Dell Precision to learn something about AI. It worked, but I wanted to have more. More speed, more context, since it was still not enough. Therefore, I thought it was a good idea to use an old GTX 970 to speed it up. And I finally got this to work. The results were good, but the VRAM was still too small for bigger context.
I decided to buy an RTX 2060 card with 12 GB VRAM via eBay Kleinanzeigen for a small amount of money. And the drama began. Nothing worked, nothing. But I did not want to give up, so I started my journey to fix it.

During the journey, the card was sometimes visible in the PCIe tree (lspci -tv) and sometimes even with nvidia-smi. But after using nvidia-smi once, it fell off the bus.

The connection

The EXP GDC OCuLink came with the dock, an OCuLink cable, and an M.2 adapter. This combination did not work well, even with my GTX 970. I think it was because the connector of the adapter was very fragile and the signaling was not as stable as I wished. However, at the time I tried, I didn’t know. So, I bought another dock with another cable and another adapter and started to replace it one by one. Yet nothing seemed to work; the other equipment also did not work well, and still the card was not visible or had fallen off the bus. Then I bought another adapter and tinkered a bit with the system, so I was not sure if the new adapter was the reason or my tinkering, which at least brought the RTX to visibility. But at least with the new adapter (GINTOOYUN PCI-E4.0 M.2 NVME zu Oculink SFF-8611/8612), it was easier to handle the cable, even though it makes the cable longer, which is not the best if you have connection issues, but the card was visible in the lspci tree.

The performance

It is usual that the faster your communication is, the more it is affected by the environment. At the time I was thinking that my old GTX card did not support PCIE 3.0, since my old PC only supported 2.0. However, I was wrong, but that didn’t matter, because it brought me on the right track. I did want to use it for AI, so a fast connection had no relevance, but stability did. But later more.

The driver

During the research I found a GIT issue comment. He mentioned something interesting. The card falls off the bus only when CUDA is used. This was super important. I checked my configuration and tried to reproduce it. At the first attempts I had no success, but then I could see it. I forgot to disable the Llama service, which already worked and was assigned to the GTX; therefore, it also started directly on the RTX. This was the reason why the card fell off the bus some time after the startup of the system. Now, I was able to reproduce it. As long as I did not touch nvidia-smi or any other CUDA call, the card stayed still, and dmesg did not show this message. But when I called nvidia-smi, the card fell off. So, I started to blacklist the drivers at system start and to load them one by one. And it worked, sometimes, but it was more stable than before. Even nvidia-smi started working. But CUDA still not. When I loaded nvidia_uvm and called nvidia-smi then the connection died again.

The power management trap

From my experience in IT, I know that power management is not your friend, and often when it comes to connection issues, it is your worst enemy. If it works, fine, but if you want to disable it, good luck. Back to the story.
One of the first solutions was to disable ASPM. Yes, power management. So, I did. I wrote pcie_aspm=off into my grub file. I already did it before, because the GTX also didn’t react that well in this setup. And since it helped with the GTX, at least to get rid of the AER errors, I was expecting it to work. Guess what? It did only part of the job. During hassling around with the drivers and the performance and the testing and, and, and I saw it in the lspci tree (3d:00.0 is the address of the RTX), wtf. Can you see it?

root@bigdelli:/home/adrian# lspci -vv -s 3d:00.0 | grep -iE ‘LnkCap|LnkCtl|LnkSta|LnkCtl2|LnkSta2’ LnkCap: Port #0, Speed 8GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <512ns, L1 <4us LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+ LnkSta: Speed 2.5GT/s (downgraded), Width x4 (downgraded) LnkCap2: Supported Link Speeds: 2.5-8GT/s, Crosslink- Retimer- 2Retimers- DRS- LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis- LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+ EqualizationPhase1+ LnkCtl3: LnkEquIntrruptEn- PerformEqu-

Wait, I will show you.

LnkCtl:	ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+

It was still on.

After some research, I was unable to get it disabled. But I remembered from previous problems that

setpci

might be helpful in this topic. So, even though ChatGPT has not really been helpful so far (as it is good if the solution is already in the training data but bad if it comes to unresolved problems), I wanted to give it a try, and it came back with a correct command:

setpci -s 3d:00.0 CAP_EXP+10.w=0140

CAP_EXP are the PCIe capabilities
+10 -> move to the ASPM area
0140 -> set bits 6 and 8 enabled but removes bit 1: 0000 0001 0100 0000

It basically disables ASPM. 0140 is 0×0140.

My initial state was 0×0142, so 0000 0001 0100 0010. That means ASPM L1 enabled, Common Clock enabled, and Clock Power Management enabled.
I got it with this command:

sudo setpci -s 3d:00.0 CAP_EXP+10.w

I put that into a systemd service and restarted everything. And it worked!

The happy end

I did much more than I described; it was a back and forth the whole time. And on some points I was thinking that the card was broken, but it worked in my gaming PC. However, I am still removing some other steps that I tried to get the root issue visible.
It turned out that the GTX also supported PCIe 3, so even though this guided me in a good direction, I removed the manual speed downgrade, and it is still stable so far.

I am super happy that it now works very stably, and the speed is impressive. Depending on the model, the range is between 20 tk/s for bigger models and over 200 tk/s for smaller models.

My current grub line is:

GRUB_CMDLINE_LINUX="consoleblank=30 pcie_aspm=off pci=realloc pcie_port_pm=off systemd.unit=multi-user.target"

Currently, I disabled the following option:

adrian@bigdelli:~$ cat /etc/modprobe.d/nvidia-graphics-drivers-kms.conf
# Nvidia modesetting support. Set to 0 or comment to disable kernel modesetting
# and framebuffer console support. This must be disabled in case of Mosaic or SLI.
#Das war default
#options nvidia-drm modeset=1

The service that disables the ASPM (Please, remember that the PCIe address (3d:00.0) is most likely different from mine):

cat /etc/systemd/system/egpu-link-aspm-off.service
[Unit]
Description=Disable ASPM on eGPU root port and device
DefaultDependencies=no
After=local-fs.target
Before=multi-user.target
[Service]
Type=oneshot
ExecStart=/bin/sh -c '/usr/bin/setpci -s 3d:00.0 CAP_EXP+10.w=0140; sleep 1'
RemainAfterExit=yes
[Install]
WantedBy=multi-user.target

The machine’s specs:
Intel® Core™ i7-6920HQ CPU at 2.90GHz
48 GB RAM at 2400 MHz
Nvidia GTX 970 eGPU with 4GB vRAM
eGPU -> Oculink -> M.2 Adapter.

Prepare

adrian@bigdelli:~$ llama-bench --list-devices
ggml_cuda_init: found 2 CUDA devices:
  Device 0: NVIDIA GeForce GTX 970, compute capability 5.2, VMM: yes
  Device 1: Quadro M1200, compute capability 5.0, VMM: yes
Available devices:
  CUDA0: NVIDIA GeForce GTX 970 (4030 MiB, 3966 MiB free)
  CUDA1: Quadro M1200 (4035 MiB, 4001 MiB free)

Models that fit into the vRAM

llama-bench -ngl 99 -fa 1 -m models/granite/granite-4.0-micro-Q6_K.gguf --device CUDA0

NVIDIA GeForce GTX 970, compute capability 5.2, VMM: yes

model	size	params	backend	ngl	fa	test	t/s
granite-4.0-micro-Q6_K.gguf
granite 3B Q6_K	2.60 GiB	3.40 B	CUDA	99	1	pp512	389.80 ± 1.11
granite 3B Q6_K	2.60 GiB	3.40 B	CUDA	99	1	tg128	24.97 ± 0.04
LFM2-2.6B-Exp-Q6_K.gguf
lfm2 2.6B Q6_K	2.07 GiB	2.70 B	CUDA	99	1	pp512	546.39 ± 1.52
lfm2 2.6B Q6_K	2.07 GiB	2.70 B	CUDA	99	1	tg128	32.72 ± 0.15
LFM2.5-1.2B-Thinking-Q6_K.gguf
lfm2 1.2B Q6_K	915.96 MiB	1.17 B	CUDA	99	1	pp512	1281.22 ± 10.11
lfm2 1.2B Q6_K	915.96 MiB	1.17 B	CUDA	99	1	tg128	69.75 ± 0.35
Qwen2.5-3B-Instruct-Q4_K_M.gguf
qwen2 3B Q4_K – Medium	1.79 GiB	3.09 B	CUDA	99	1	pp512	441.62 ± 1.16
qwen2 3B Q4_K – Medium	1.79 GiB	3.09 B	CUDA	99	1	tg128	33.44 ± 0.02
Qwen2.5-3B-Instruct-Q6_K_L.gguf
qwen2 3B Q6_K	2.43 GiB	3.09 B	CUDA	99	1	pp512	459.30 ± 1.93
qwen2 3B Q6_K	2.43 GiB	3.09 B	CUDA	99	1	tg128	27.27 ± 0.05
Qwen2.5-3B-Instruct-Q8_0.gguf
qwen2 3B Q8_0	3.05 GiB	3.09 B	CUDA	99	1	pp512	143.61 ± 0.21
qwen2 3B Q8_0	3.05 GiB	3.09 B	CUDA	99	1	tg128	34.15 ± 0.01
Qwen3-4B-Instruct-2507-Q6_K.gguf
qwen3 4B Q6_K	3.07 GiB	4.02 B	CUDA	99	1	pp512	100.84 ± 0.31
qwen3 4B Q6_K	3.07 GiB	4.02 B	CUDA	99	1	tg128	21.97 ± 0.02
qwen2.5-1.5b-q8_0.gguf
qwen2 1.5B Q8_0	1.53 GiB	1.54 B	CUDA	99	1	pp512	961.78 ± 5.83
qwen2 1.5B Q8_0	1.53 GiB	1.54 B	CUDA	99	1	tg128	57.89 ± 0.03

Models that don’t fit into the vRAM

model	size	params	backend	ngl	fa	dev	test	t/s
deepseek2 30B.A3B Q4_K – Medium	16.88 GiB	29.94 B	CUDA	6	1	CUDA0	pp512	65.14 ± 0.26
deepseek2 30B.A3B Q4_K – Medium	16.88 GiB	29.94 B	CUDA	6	1	CUDA0	tg128	11.85 ± 0.02

Example via the llama-server provided website

Question: Please explain “bit shifting” to me as if I were 5 years old.
The answer metrics for the:

Fully loaded on GPU
LFM2.5-1.2B-Thinking-Q6_K.gguf	1,487 tokens	22s	65.85 t/s
LFM2-2.6B-Exp-Q6_K.gguf	1,174 tokens	37s	31.51 t/s
granite-4.0-micro-Q6_K.gguf	215 tokens	8.8s	24.34 t/s