“Localized” AI on personal or office pc?

[Elnoxv]

tbf its with quantization and other lossy optimizations in comparison to full weights. in comparison a dense model gemma4 26B A4B quantized at Q4 gives about the same rate.

speed is just one part of the equation though. i watched some YT vids that showed local AI completing actually hard tasks like reverse engineering modems and monitor firmware, that was the moment i realized local AI is actually mature enough on average consumer hardware now, and it is time to get on the train. the speed at which claude code is being used to accelerate github projects is the other trigger.

right now is the time the ship is leaving the harbor and last chance to not be left behind. the progress being made in AI and with AI this few weeks is really very scary in terms of the impact they will have on job replacement and the future of human society and economy.

Yah, Qwen3.8 27b is the first model that feels so good and not pale in comparison with the frontier model. However, the efficiency in problem solving speed is still bad. ( What took Opus 5.5, 5 minutes to solve would take Qwen3.8 27b 3-5 hours to solve. Though recent patches from the community do help slightly.)

Other bigger models like Deep-seek flash v4.1 and GLM 5.3 are much closer to frontier models, both in efficiency and intelligence but do require bigger VRAM.

However, VRAM is crazy expensive now. 15K for Mac Studio 256gb, 20-30K for RTX6000 96gb, 8-10K for 128b DGX spark.

At that price, it would simply be better to buy sub from Claude or OpenAI. There is little to gain, tbh. Unless you have special use cases or are already earning money from these investments.

and no doubt, these will greatly impact jobs and livelihood severely. Current retrenchments and hiring freezes are only the start, the worst isn’t over yet. Even bill gates himself warned us about the impact of AI and there will come to a point where AI is crossing a threshold where it can handle tasks at higher accuracy and will companies still need humans ?

not trying to scare anyone but these were said by bill gates himself so just sharing what i know.

The next few years are going to be difficult.

 

[testerjp]

This points out how far we are from robots that are useful. The head must be server size.

 

[Laneige]

I trying using localized AI like MUSE to generate image.. quality not that good and need to wait a long time.. so end up I stop using it...

Images I think v high consumption of memory

 

[Laneige]

If don't trust them, then even Microsoft onedrive cannot trust liao

Really or not
Claude trusting than Ms?

 

[kreja]

If don't trust them, then even Microsoft onedrive cannot trust liao

Never use before. Setup truenas on 10yr old low power laptop can liao. Atas abit buy synology nas for added security and convenience. Linux is final end goal.

As AI become more ubiquitous, I expect more data leaks and hacking to happen because companies undervalue humans over AI. Good to learn how to protect yourself.

 

[trento]

Really or not
Claude trusting than Ms?

Dunno. Unless got data to prove. Also got others like Google

 

[trento]

Never use before. Setup truenas on 10yr old low power laptop can liao. Atas abit buy synology nas for added security and convenience. Linux is final end goal.

As AI become more ubiquitous, I expect more data leaks and hacking to happen because companies undervalue humans over AI. Good to learn how to protect yourself.

U never use Microsoft before? Ok anyway, enterprise level is usually Microsoft or Google. Will be interesting if got Sg companies don't rely on either

 

[kreja]

U never use Microsoft before? Ok anyway, enterprise level is usually Microsoft or Google. Will be interesting if got Sg companies don't rely on either

Whatever is mandated in your company, any data loss arises, as long as you follow company SOP you are legally covered. The data loss is your company data, not any personal or private data that will affect?

Why are you mixing the two?

 

[trento]

Whatever is mandated in your company, any data loss arises, as long as you follow company SOP you are legally covered. The data loss is your company data, not any personal or private data that will affect?

Why are you mixing the two?

Not mixing. Yr company can sub to Claude, Microsoft etc and u can just use those AI services, and obviously yr company and yrself trust them

 

[kreja]

Not mixing. Yr company can sub to Claude, Microsoft etc and u can just use those AI services, and obviously yr company and yrself trust them

You are entitled to what you choose to believe, even your own AI delusions. I believe I have already clarified sufficiently for other readers to understand. It is common sense to separate work data and personal data. I do not trust AI with my personal life

 

[trento]

You are entitled to what you choose to believe, even your own AI delusions. I believe I have already clarified sufficiently for other readers to understand. It is common sense to separate work data and personal data. I do not trust AI with my personal life

You are also entitled to believe your own AI delusions. Of course we separate company and personal data. I never say we mix them. Either way, it's based on trust.

 

[twodragon]

If don't trust them, then even Microsoft onedrive cannot trust liao

thats why moi use mac mini as server

 

[gld998]

 

[Pearce]

What about this?