Need recommendation for new AM4 motherboard.

[michael_thm]

If you need self-hosted LLMs, sticking to Nvidia cards provide better software support on Linux. So it will be a better choice imo, even for 2x GPUs setup. Windows operating system generally has better support on AMD for AI. Choosing Linux over Windows for such use case are generally preferred. Though there will be some learning curve if you are not sufficiently "Linux savvy". Another setup option could be the newer AMD AI pro mini-PCs if you want to run really large models which requires lots of Vram, but are not going to come cheap too.

Personally, I am running my own self-hosted LLMs using the 3x RTX4000 SFF Ada Edition with 20GB VRAM which runs on 3 different hosts (each nodes on 5950X with 128GB ECC memory), loaded with 3 different LLMs models on Linux Mint 22.2 Zara (based on Ubuntu 24.04) virtual machines. I am also running a hyper-converged system with storage on 3x TrueNAS storage servers with a mixture of enterprise HDD and SSDs, inter-connected on a 2x 10GbE network backbone infrastructure (with MLAGG redundancy setup) which I use to cluster the hosts for bigger models when needed, though the latency will still be high (200G fiber network with RDMA preferred) but suitable for once in a while testing purposes.

Setting up your very own local LLMs or AI servers are really fun and opens up lots of possibilities. You can integrate with your smart home assistant over the Home Assistant platform, runs self monitoring 3D-printing farms, AI video/picture generator or create useful AI automations using n8n, just naming a few. Have fun!​

Haha. wait I strike toto 1st prize then I can go it your way ha.

On the topic of OS... I just happened to be thinking about deleting the ubuntu installation on SSD #2 to delegate it to loading models. Linux applications can live on WSL 2 I reckon? linux is annoying(for me)... even use chrome also it will demand password. Been using ubuntu on and off for many years but I won't dare to say I am savvy, just capable of following instructions nia. Windows is easier. Like today, I tried to learn how to use github to keep repositories up to date and I figured it out in 30mins, on linux I think it would take me a few hours with all the commandline entries? But still keeping the ubuntu for the rare use cases where windows just won't cut it... at least until I have time to try out WSL 2. In fact I bought my 6x8TB HDDs to try out ZFS. Overkill for my use case.

 

[sumako]

Take note the driver for that is also different.
What you are using the Geforce is for gaming not for AI modeling.
Nvidia also have AI cards but that is Telsa range.

PS the link below is just info sharing not my card.
You can buy one and try than you will know the difference.

Telsa cards are design for compute power. Take note this one have no display out.
So your AI model have to assign to this GPU.

Nvidia Tesla V100 PCIe https://carousell.app.link/sdY5qgm99Yb

Nvidia Tesla V100 is next level

can it be use on our consumer pc?

 

[ragnarok95]

Nvidia Tesla V100 is next level

can it be use on our consumer pc?

Can use. But he will tell u, geforce is for gaming/consumer and tesla for the work.

 

[TanKianW]

Haha. wait I strike toto 1st prize then I can go it your way ha.

On the topic of OS... I just happened to be thinking about deleting the ubuntu installation on SSD #2 to delegate it to loading models. Linux applications can live on WSL 2 I reckon? linux is annoying(for me)... even use chrome also it will demand password. Been using ubuntu on and off for many years but I won't dare to say I am savvy, just capable of following instructions nia. Windows is easier. Like today, I tried to learn how to use github to keep repositories up to date and I figured it out in 30mins, on linux I think it would take me a few hours with all the commandline entries? But still keeping the ubuntu for the rare use cases where windows just won't cut it... at least until I have time to try out WSL 2. In fact I bought my 6x8TB HDDs to try out ZFS. Overkill for my use case.

If there is a choice, I will go full Linux instead of WSL, but that is due to the fact that I daily drive ArchLinux or Debian, rarely Windows in recent years. As you said, I think WSL is fine too if you are used to Windows. Worth a try.

On the ZFS front, you might wanna read more on ZLOG and "Special Metadata" if you want to crank up the performance with SSDs. Storage pools on pure spinning HDDs might be too slow for most use cases. There is a (tutorial) thread on Level1tech forum on their use cases and benefits.

If you are setting them up on TrueNAS, feel free to check out my thread here:
https://forums.hardwarezone.com.sg/threads/starting-truenas-diy-nas-for-new-users.6480129/​