“Localized” AI on personal or office pc?

[NintendoSwitch]

If you have an existing PC, the setup cost is actually less than < $2K.

Just buy a used RTX3090 ( around $1.5 - 2k). Run wireguard / tailscale at home.

From your laptop connects to tailscale / wireguard.
Point your harness: Claude code / Deepseek to where your inference provider located.

that's it.

No PC. Laptop with iGPU nia.

 

[gld998]

To setup your own Ai, you need a powerful GPU. Otherwise, your comp or laptop will be very lag and like hang. That's why a local system just for AI cost from $8999 onwards, AMD from $6000 plus onwards.

Look up on running llm on your android phone. Use that... train it use your data.

 

[Lao_Tiko]

Minimum is 24gb vram GPU. 12gb is snail speed (ok, maybe closer to typewriter speed - can use lah but your models have to fit otherwise it will spill over into system ram which is very very slow to be useable.)

If you have a spare desktop you can try running Proxmox 9 on it. Create LXC (more efficient then VM) and passthrough the GPU. Run the ollama, LiteLLM and OpenWeb UI nested dockers in the LXC. You can wire up your APIs .... Actually I have no IT background so I used Claude to help me. Use AI to build AI ....then you can use AI to defeat AI .... 以毒攻毒

 

[gld998]

IMHO.
I hope to start small, using my 5800H with 32GB and see what I can do with it.

I using my 5660u igpu for my micro llm to run on llm studio. haha... performance is similar..

 

[enimen]

How does local AI work? U need to install the AI app is it?

 

[keenklee]

I using my 5660u igpu for my micro llm to run on llm studio. haha... performance is similar..

IMHO.
Can't wait to try it out.

 

[KakiMeow]

GCC all info is stays in SG and its separated from commercial cloud.

so mabbe gotch

simpler solution for ts

which also store local but

utilize powerful model

 

[gld998]

IMHO.
Can't wait to try it out.

You can use smaller model then responce rate can improve.

 

[gld998]

so mabbe gotch

simpler solution for ts

which also store local but

utilize powerful model

TS run road already... anyway there is Govtech AI called AI platformer... can run agentic stuff... dunno can do action yet or not.. last I heard was using rabbideploy to deploy vibe coding apps...

 

[KakiMeow]

TS run road already... anyway there is Govtech AI called AI platformer... can run agentic stuff... dunno can do action yet or not.. last I heard was using rabbideploy to deploy vibe coding apps...

nab.ehz

edmw

sleep tiger hide n seek dragon sia

 

[gld998]

Minimum is 24gb vram GPU. 12gb is snail speed (ok, maybe closer to typewriter speed - can use lah but your models have to fit otherwise it will spill over into system ram which is very very slow to be useable.)

If you have a spare desktop you can try running Proxmox 9 on it. Create LXC (more efficient then VM) and passthrough the GPU. Run the ollama, LiteLLM and OpenWeb UI nested dockers in the LXC. You can wire up your APIs .... Actually I have no IT background so I used Claude to help me. Use AI to build AI ....then you can use AI to defeat AI .... 以毒攻毒

Yes. It will be slow.. This explains it very clearly why its slow...

 

[Elnoxv]

How does local AI work? U need to install the AI app is it?

The most straightforward and simple way to do it is to use these app:
1. LM studio
2. Ollama
3. Unsloth

on Mac:
1. oMlx.

but do requires capable hardware in order to run.

 

[Pearce]

The most straightforward and simple way to do it is to use these app:
1. LM studio
2. Ollama
3. Unsloth

on Mac:
1. oMlx.

but do requires capable hardware in order to run.

I use ollama on my pc and it hangs

 

[gld998]

I use ollama on my pc and it hangs

ollama is not suitable for windows. try lmstudio.

 

[KnyghtRyder]

If your work has those kinds of restrictions then how would you even install the agent.

 

[kreja]

None of the current local llms can match frontier models from claude or chatgpt.

Also I dunno what is your specific use case so idk how everyone can so easily recommend solutions for you.

My suggestion is install lm studio and just try the popular local llms that your current hardware can handle and see if it can handle your usecase.

 

[samferguson]

TS run road already... anyway there is Govtech AI called AI platformer... can run agentic stuff... dunno can do action yet or not.. last I heard was using rabbideploy to deploy vibe coding apps...

Safe to use?

 

[gld998]

Safe to use?

Its govtech tool... you ask govtech ok...

 

[samferguson]

ollama is not suitable for windows. try lmstudio.

Ollama is good for me, however it's a pain to set the context window. Dunno why it always default to 4k even when I have enough VRAM. Don't like lmstudio, dunno why always stutters.

 

[gld998]

None of the current local llms can match frontier models from claude or chatgpt.

Also I dunno what is your specific use case so idk how everyone can so easily recommend solutions for you.

My suggestion is install lm studio and just try the popular local llms that your current hardware can handle and see if it can handle your usecase.

No local LLM can beat subscription based in term of performance. Just good enough will do.