“Localized” AI on personal or office pc?

[samferguson]

Its govtech tool... you ask govtech ok...

don't want help them train their model nor snoop around.

 

[gld998]

Ollama is good for me, however it's a pain to set the context window. Dunno why it always default to 4k even when I have enough VRAM. Don't like lmstudio, dunno why always stutters.

Well.. turn on logs and see what is actually happening. Use chat to troubleshoot.

 

[gld998]

don't want help them train their model nor snoop around.

then run locally... on your personal hardware

 

[Elnoxv]

I use ollama on my pc and it hangs

on what gpu and ram size? what model do you use ?

 

[Meteor84]

its relatively easy with ollama. just install, select the model u like and just use. as good as plug and play. but the performance is crap unless you have a beast of a PC. its fun if u r trying to learn and what not... but if u r used to chatgpt, gemini that kind of response, u cannot tahan how slow and stupid your localised AI can be. i much rather use incognito chat on gemini/chatgpt than rely on my LLM

 

[gld998]

its relatively easy with ollama. just install, select the model u like and just use. as good as plug and play. but the performance is crap unless you have a beast of a PC. its fun if u r trying to learn and what not... but if u r used to chatgpt, gemini that kind of response, u cannot tahan how slow and stupid your localised AI can be. i much rather use incognito chat on gemini/chatgpt than rely on my LLM

Use as hybrid bah.. i using my subscription to run heavy task and local ai for simple task. I have serious hardware as well with run local for my small startup.

 

[trento]

If the worry is confidential docs leaving the company, 3 options:

(1) run a small model on your own PC with Ollama or LM Studio: free and fine for summarising short docs, but noticeably weaker than ChatGPT;
(2) use the business/enterprise plans of ChatGPT or Claude, which don't train on your data by default. Check with your IT first;
(3) self-host an open model on the company's own server, so data never leaves.

For most offices (2) with IT approval is the practical answer; (1) is good for personal use.

Personal Claude also can opt out of training

 

[trento]

Minimum is 24gb vram GPU. 12gb is snail speed (ok, maybe closer to typewriter speed - can use lah but your models have to fit otherwise it will spill over into system ram which is very very slow to be useable.)

If you have a spare desktop you can try running Proxmox 9 on it. Create LXC (more efficient then VM) and passthrough the GPU. Run the ollama, LiteLLM and OpenWeb UI nested dockers in the LXC. You can wire up your APIs .... Actually I have no IT background so I used Claude to help me. Use AI to build AI ....then you can use AI to defeat AI .... 以毒攻毒

For small models 12gb vram is ok. Quite fast

 

[Lao_Tiko]

For small models 12gb vram is ok. Quite fast

Qwen 2.5? Haven't tried many locally run models myself. Claude is good but every 5 hours have to take a break

 

[BlackWing1977]

Let’s say I don’t want to use ChatGPT or Claude Ai due to nature of work (confidential data) that is best to stay local than to upload up to their server ….

Which AI sw is gd to be our “assistant co-colleague” leh?
Need to handle task so I can do my job
I asked CHAPGPT n Claude. Both on server … not as safe

Thanks!

I trying using localized AI like MUSE to generate image.. quality not that good and need to wait a long time.. so end up I stop using it...

 

[trento]

Qwen 2.5? Haven't tried many locally run models myself. Claude is good but every 5 hours have to take a break

If heavy tasks, then need claude pro.

Yea got qwen and others. 12gb vram is mighty fast even for 8-9b, summarising generate text no issues.

Last time i play game and like to see gpu performance. Now is play AI. Gpu temps like no big issue despite the high vram usage, unlike gaming where temps always shoot up. Im new to this so really need to ask AI what gpu is needed

 

[trento]

I trying using localized AI like MUSE to generate image.. quality not that good and need to wait a long time.. so end up I stop using it...

My take is as long as privacy not needed, cloud AI is always better, esp those image and video

 

[kreja]

Personal Claude also can opt out of training

"Trust me bro"

 

[trento]

"Trust me bro"

If don't trust them, then even Microsoft onedrive cannot trust liao

 

[Pearce]

on what gpu and ram size? what model do you use ?

hahahaha, I am using onboard gpu. Hope can have to money for higher end gpu

 

[808180579]

is 50+ token/s on a 125 billion parameters mixture of experts model or 170+ token/s on a 12B dense model considered snail speed on 12gb vram? that is text generation speed, prompt processing speed is in the thousands iirc

 

[Elnoxv]

hahahaha, I am using onboard gpu. Hope can have to money for higher end gpu

Oh gonna be hard. Smaller model may work though. Gotta give it a try to know.

 

[Elnoxv]

is 50+ token/s on a 125 billion parameters mixture of experts model or 170+ token/s on a 12B dense model considered snail speed on 12gb vram? that is text generation speed, prompt processing speed is in the thousands iirc

50t/s is great for 125b MOE. The intelligence level will be much higher compared to 12B.

and you should use the MOE model instead, it will be able to tackle more complex problems. However, if your task is simple e.g. summarisation. and presumably, nothing complex, you can prioritise on speed.

 

[808180579]

50t/s is great for 125b MOE. The intelligence level will be much higher compared to 12B.

and you should use the MOE model instead, it will be able to tackle more complex problems. However, if your task is simple e.g. summarisation. and presumably, nothing complex, you can prioritise on speed.

tbf its with quantization and other lossy optimizations in comparison to full weights. in comparison a dense model gemma4 26B A4B quantized at Q4 gives about the same rate.

speed is just one part of the equation though. i watched some YT vids that showed local AI completing actually hard tasks like reverse engineering modems and monitor firmware, that was the moment i realized local AI is actually mature enough on average consumer hardware now, and it is time to get on the train. the speed at which claude code is being used to accelerate github projects is the other trigger.

right now is the time the ship is leaving the harbor and last chance to not be left behind. the progress being made in AI and with AI this few weeks is really very scary in terms of the impact they will have on job replacement and the future of human society and economy.

 

[leonaheidern]

is transcription between multiple languages considered complex? there is 4 languages and i realized whisper drops all except the English even though i prompted to transcribe and translate all of them explicitly


Read HWZ Forum Rules!