I’ve decided to upgrade my ML workstation, I was especially interested to improve the CPU core count. Unfortunately the original Asus ROG Zenith Extreme only supports Threadripper Gen2 processors and as 2990WX is more expensive than a 3970X which is beating it’s performance significantly. I’ve ended up with a new (used) TRX40 motherboard and that had a cascading effect on the build.
I’ve get a used Zenith Extreme II Alpha which supports Gen3 Threadripper processors, this was relatively cheap $300 on ebay for a nice one with box and all accessories. The 3970X was a refurbished /used processor from amazon for almost $1k. These are still pretty expensive processors.
And here’s the cascading effect started. As the motherboard supports double the RAM I’ve added 256GB DDR4. As I wasn’t sure what’s gonna happen with the 3 NVME drives in the donor system, I wanted to have a backup so I’ve added a 20TB Seagate Iron Wolf Pro spinning disk to do the backup and later use in the new system. The 3x2TB was just very small for keeping some of the large models locally (for quantizing, training, or just have a bunch of GGUFs for local run).
I love the look of the existing Be Quiet! EATX case, but having this much components in the build and also keeping it air cooled was challenging. So I’ve also opted in for the ThermalTake Core W100 chassis which is a monster.
Regarding the CPU cooling, I’ve debated about an AIO. As the silicon layout changes and pushed the cores to the sides, a generic AIO that doesn’t cover the whole surface is not providing the best performance. So wanted something that covers the whole heat dissipation surface. The only AIO I’m aware that build for Threadripper in Enermax’s LIQTECH TR4 but the cooler has serious design flaws (easy to find videos about it on youtube). And I’m not in the business to build custom watercooling.
There are a bunch of Air Cooler options for Threadripper they even offer offset the cooler to be able to access the upper most PCIe slot, but that is back to square one if I’m worry about covering the whole heat dissipation area. I found only one cooler that solve this problem. The Ice Giant ProSiphon Elite. It provides plenty of room for the first PCIe slot, hangs above the RAM. Though it’s blocking the DIMM.2 slot, but more on that later.
One I’ve did a dryrun and found the main components working I’ve started to assemble the XL ATX compatibe ThermalTake chassis. Yes assemble it because it’s packaged IKEA style and you have to build it for yourself.
The chassis has plenty of room to install fans. I’ve added 10x140mm fans (front, back, bottom and top) that provide decent airflow. If someone wants to build a watercooled system this chassis allows you to install 3-4 radiators easily.
It’s came along pretty nice, but few things are bothering me:
- I don’t prefer the “new” PCI slot layout and 2nd and 3rd are right next to each other. This pack the GPUs even more tightly than before. Hope the better airflow and more vents will make up for it.
- The DIMM slot for additional Nvme drives is blocked by the cooler. The motherboard itself has 3 m.2 slots, but if all used the the last (4th) PCIe is restricted to Gen4 4x (instead of 8x)
- The chassis has a vertical GPU option. I’ve hoped I can install one of the RTX 3090s vertically so the 3 GPUs won’t be so tightly packed, but the RTX 4080 is much wider so it’s blocking the vertical mount. If I can drop it to the last PCIe slot it might work physically, but than I’d restrict the most powerful GPU in the system to Gen4 4x (instead of Gen4 16x).
- I had to 3D plan and print a console to the fan header. It’s not the end of the world but gave me some extra work.
The positives (and even surprises):
- As the new chassis much wider it gave me plenty of room for cable management and to install the HDD at the back side (you actually can install like 6 drives there).
- Everything has worked, though I’ve needed to figure out some motherboard configuration
- to use the 3rd m.2 slot required to set the 4th PCIe slot to 4x
- to have enough PCIe resources to drive all the 3 GPUs I’ve had to set the coding for the PCIe lanes to 64bit. Otherwise the motherboard has posted D4 errorcode for VGA.
- And the most surprising: both the Debian 12 and the Windows 10 OS has survived inplantation to the new system. Even the GRUB loader stayed in working condition.
Overall I’m satisfied, everything works. The thermal load on the tightly packed GPU yet to be tested with some heavy load. But during Cinebench 2024 the RTX 3090s seemingly not thermal throttled. Since I’ve installed the lated Nvidia Drivers the two RTX 3090s not showing up and used in Cinebench 2024 but appears on nvidia-smi and used in LMstudio. How to use all the three again if a problem for another day.
Here’s the latest benchmark with only the RTX 4080 (All 3 GPUs achieved 54xxx score – no screenshot though 🙂 )