Have you ever thought “I wish I could find the document where I’ve noted some information but I can’t even remember if it was a local notebook or a gdoc or what even was the filename” would be nice to…
The following is the leader board of all GPUs I’ve tested with llama-bench. For the model I’m using Q8_0 and Q4_K_M quantized Llama-3.1-8B-Instruct from the following repo: DevQuasar The reasoning behind the model selection is to use a generic widely…
Hardware Setup If you’re like me and using server-grade AMD Instinct passive-cooled AI accelerators, you should ensure you can provide adequate cooling. For this, I rely on a 120mm blower PWM fan. I’m particularly using the following I’ve also…
Just tried a less popular AI accelerator option, the AMD Radeon Instinct MI50, and it has performed surprisingly well in inference. But before looking at the results, what is this card? The card was launched in November 2018 by AMD.…
I’ve just fixed the power supply for one of my vintage computer a Videoton TVC, and though I’ll wriite a small blogpost about it. This computer shares the origin with me we’re both from Hungary. A Brief History of the…
I came across a few unusual prompts and wanted to dig deeper, understand the motivations behind them, and test their effectiveness. Some of these techniques are not new, and some might not work, but let’s explore how humans (and possibly…
I’ve rerun the Speculative Decoding experiment with some larger models where I’ve paired a Llama 3 70B primary model with the 8B draft model, to see if a larger model Llama 3 70B can benefit more from a draft model…
And there I’ve crunched the numbers on GPU. The configuration is a 3GPU system 1x RTX4080 + 2x RTX3090. Baseline has been set by the following llama.cpp generation command (same generation config and prompts has been used in the Speculative…
Here’s a followup on the Speculative decoding on CPU tradeoffs now performing the same test on a Threadripper 3970x, and finally I see some benefit. Acceptance rate Generation speed Summary The results seems very similar to the Xeon E5 test.…
That’s probably the trash can Mac Pro from 2013. In December 2013, Apple released a new cylindrical Mac Pro (colloquially called the “trash can Mac Pro). Apple said it offered twice the overall performance of the first generation while taking…
