Since getting the Orange Pi 5 SBC, I wanted to try connecting a GPU via the M.2 M key NVMe port to create a compact inference server. I expected this wouldn’t be easy, but a post and YouTube video initially gave me hope.
https://www.jeffgeerling.com/blog/2023/external-gpus-working-on-raspberry-pi-5
The components I’ve used (first)
- Orange Pi5 plus SBC
- JMT M.2 M Key NVMe External Graphics Card Stand Bracket
- Tesla K80 Accelerator ($32 from ebay 🙂 )
- printed shroud https://cults3d.com/en/3d-model/gadget/92mm-tesla-k80-fan-shroud
- be Quiet 92mm fan (I’ll need to replace with with a high static pressure fan)
- ATX PSU
- Orange Pi’s own Debian image
An interesting fact about the K80: It’s a dual GPU setup with two GPUs integrated on one PCB. Most comparison sites incorrectly state the CUDA core count as 2496, but that’s per GPU. The K80 actually has 2×2496 CUDA cores and 12GB of VRAM for each GPU, totaling 24GB. The two GPUs are clearly visible from the bottom, along with the 2x12x1GB VRAM modules around them.
I’ve connected everything together: the NVMe connection nicely controlling the PSU, and the card showing up in the output of the lspci command. After this, everything went south. Even though Nvidia has drivers for ARM64 platforms, nothing worked; all installations and compilations failed. Later, I read on a Reddit post that someone managed to partially install a custom-compiled driver. If I recall correctly, it said there are missing instructions from the chip because the chip manufacturer did not pay for certain licenses. It was deep in some subreddit that I can’t find anymore. Every other attempt seems unsuccessful.
https://www.reddit.com/r/OrangePI/comments/10vgbs8/orange_pi_5_4080_gpu/
https://www.reddit.com/r/OrangePI/comments/1b7d9nr/orange_pi_with_egpu/
I’s actually accepted the one advice I found there: “Best to stick with the hardware supported by Nvidia: x86.” So I’ve got an x86 SBC – youyeetoo X1 – N5105 x86.
You might think it would be straightforward from here, but not exactly. Nvidia’s Kepler GPUs (like my K80) support CUDA up to version 11.4. First, I tried to set up the drivers and CUDA on Debian 12. The 470.X Tesla drivers worked, but I ran into issues with the CUDA toolkit. When I installed the latest version, it updated the driver and broke everything. I could not compile the CUDA toolkit I downloaded from Nvidia. The situation was similar with Debian 11, which even suggested the wrong driver when I ran the nvidia-detect tool.
I finally found success with Ubuntu 20 LTS by following the instructions from this writeup: https://github.com/ashutoshIITK/install_cuda_cudnn_ubuntu_20
At this point the the 2 GPUs showed up with the nvidia-smi command, and nvcc --version also output the installed 11.4 CUDA toolkit version.
To make this work I’m restricted to the older torch==2.0.1 version that works with CUDA 11.4. And it all works now:
As a summary I would say any accelerator that doesn’t support CUDA 12 (or at least 11.8) makes very little to no sense. Like this other $12 K20 😂Â
But it’s fun to play with these not-so-old yet highly outdated accelerators, as the AI history has computed on these devices.