“Localized” AI on personal or office pc?

[Laneige]

Let’s say I don’t want to use ChatGPT or Claude Ai due to nature of work (confidential data) that is best to stay local than to upload up to their server ….

Which AI sw is gd to be our “assistant co-colleague” leh?
Need to handle task so I can do my job
I asked CHAPGPT n Claude. Both on server … not as safe

Thanks!

 

[Dracky]

Pose your question to Deepseek instead of EDMW. You may get a better answer.

 

[articland05]

from what i see in youtube, the setup take up huge amount of resources and still produce sluggish replies...no point

 

[Laneige]

from what i see in youtube, the setup take up huge amount of resources and still produce sluggish replies...no point

Maybe just simple two page documents leh
Co work if for eg sensitive documents means data will be used to upload n compare
I think ChatGPT “more honest” abt it whereas Claude will wana try to appear as more trustworthy n as long as u choose don’t use for sharing, u are “safe” whereas ChatGPT more upfront abt it

Just liked I tried Claude do nice nice until like take short cut n chut pattern .. that’s when skills are taught hor

 

[articland05]

Maybe just simple two page documents leh
Co work if for eg sensitive documents means data will be used to upload n compare
I think ChatGPT “more honest” abt it whereas Claude will wana try to appear as more trustworthy n as long as u choose don’t use for sharing, u are “safe” whereas ChatGPT more upfront abt it

Just liked I tried Claude do nice nice until like take short cut n chut pattern .. that’s when skills are taught hor

these days I use Gemini
so generous with free tokens no need pay money can do a lot

 

[gld998]

Let’s say I don’t want to use ChatGPT or Claude Ai due to nature of work (confidential data) that is best to stay local than to upload up to their server ….

Which AI sw is gd to be our “assistant co-colleague” leh?
Need to handle task so I can do my job
I asked CHAPGPT n Claude. Both on server … not as safe

Thanks!

Late boomer spotted... yes can run. but just mini llm can run. Agentic AI not allowed to run locally in you are working in Goverment but they are trying to change that.

 

[gld998]

@Laneige

You can try this if your laptop is Macbook on 32GB memory

 

[Money no have]

For a start is too expensive liao.

especially more so ram and graphics card sshortage caused by data centre.

you are looking at $5k minimum for a “normal “ ai

 

[runicz]

Let’s say I don’t want to use ChatGPT or Claude Ai due to nature of work (confidential data) that is best to stay local than to upload up to their server ….

Which AI sw is gd to be our “assistant co-colleague” leh?
Need to handle task so I can do my job
I asked CHAPGPT n Claude. Both on server … not as safe

Thanks!

If you want an enterprise solution for all your employees,(more than 10) you may need to hire proper SI provider to deploy your own on-prem server (typicall powered by Qwen/Deepsek/Mistral china models)
Ensure your SI provider has real AI Solution Architects. Most in this field of work are fake ordinary Software Architects with upgraded fancy titles like AI Architects for marketing. (Remember 15 years software development/engineering experience for windows XP systems means nothing as compared to a real AI enthusiast who have been doing AI since the early research days for 3-5 years)

If your company is small, as in you and your partner are the only employee, you should buy a 1 or 2 DGX spark to deploy your own LLM call api + chat assistant (aka as per your own words AI sw). The server will be encaged in your bedroom, unless someone crawl into your bedroom to steal the physical computer, your data will be fine. Do not buy the consumer level PC to host LLM as you will be constrained to weaker model, nowhere near frontier capabilities. You may also hire me as freelance to help if you want. I do this kind of work for enterprise.

Alternatively, Claude AI also allow you to buy enterprise subscription where by they will (verbally) promise you that any information you keyed into claude will not be used for training future frontier models and will be handled as client's enterprise data belonging to you.

 

[gld998]

If you want an enterprise solution for all your employees,(more than 10) you may need to hire proper SI provider to deploy your own on-prem server (typicall powered by Qwen/Deepsek/Mistral china models)
Ensure your SI provider has real AI Solution Architects. Most in this field of work are fake ordinary Software Architects with upgraded fancy titles like AI Architects for marketing. (Remember 15 years software development/engineering experience for windows XP systems means nothing as compared to a real AI enthusiast who have been doing AI since the early research days for 3-5 years)

If your company is small, as in you and your partner are the only employee, you should buy a 1 or 2 DGX spark to deploy your own LLM call api + chat assistant (aka as per your own words AI sw). The server will be encaged in your bedroom, unless someone crawl into your bedroom to steal the physical computers, your data will be fine. Do not buy the consumer level PC to host LLM as you will be constrained to weaker model, nowhere near frontier capabilities. You may also hire me as freelance to help if you want. I do this kind of work for enterprise.

Alternatively, Claude AI also allow you to buy enterprise subscription where by they will (verbally) promise you that any information you keyed into claude will not be used for training future frontier models and will be handled as client's enterprise data belonging to you.

Wrong place to advertise your services. This is discussion forum and not your personal advert for your services. Another bugger trying his luck. No need professional services, AI Itself can guide us by asking the right prompt.

 

[runicz]

Wrong place to advertise your services. This is discussion forum and not your personal advert for your services. Another bugger trying his luck. No need professional services, AI Itself can guide us by asking the right prompt.

How about you, you are advertising for a shity rpioff pc that cant host any proper LLM. How about I bring it to you you are perhaps properly just another laptop vendor?

Anyway, go ahead and ask AI then. BTW that Kimi k3 video of yours is a marketing fluff to cheat consumer. if theres a cheat advertiser out there, yours properly more fluffy ad than mine. Go ask AI why and dont ask me. lol

here is just a random passerby comment sincere to help.

if im an advertiser i will not be using a 2008 account.

Met a client not long ago, who took youtube videos srsly and kept arguing how a normal pc can run powerful llm for enterprise solution and my solution is fraud as well. well if thats the case, go build your own stuff and show it in deployment then, youtube video with fancy ppt slides means nothing.

 

[gld998]

How about you, you are advertising for a shity rpioff pc that cant host any proper LLM. How about I bring it to you you are perhaps properly just another laptop vendor?

Anyway, go ahead and ask AI then. BTW that Kimi k3 video of yours is a marketing fluff. Go ask AI why and dont ask me. lol

here is just a random passerby comment sincere to help.

if im an advertiser i will not be using a 2008 account.

Lol... PM la bugger... this is discussion forum not your personal advertisement space. You must be very desperate to advertise here. Enterprise experience? You sure?

 

[runicz]

Lol... PM la bugger... this is discussion forum not your personal advertisement space. You must be very desperate to advertise here. Enterprise experience? You sure?

How about your macbook? Companies burned billions on GPU server, you think ur freaking 5k macbook can replace it. You sure??

 

[BigMonkey]

depends on how many db you want to run.

 

[testerjp]

Iirc the most efficient one is the deepseek released a few days ago. Or wait for Kimi 3 distilled version.

 

[buttbERry]

just use gemini lor

 

[runicz]

@Laneige

You can try this if your laptop is Macbook on 32GB memory

I hope TS dont have silly idea because of some random kid on EDMW acting like professor of AI.

One look base on your choice of words I know you are not of AI research background.

P.S TS, I actually have the physical macbook in the youtube video in my room as dedicated LLM server for research purposes. I can also let you play with it if you want. You want to trust him or me up to you.

Also, theres a couple guys here who seems knowledgeable and knows what they are talking and also plenty just talking crap to sounds really smart and relevant for the modern world. They know who they are.

Challenges of world of AI lies in how you navigate around the world over-filled with good and bad information and data via filtering out the noise.

 

[keenklee]

Let’s say I don’t want to use ChatGPT or Claude Ai due to nature of work (confidential data) that is best to stay local than to upload up to their server ….

Which AI sw is gd to be our “assistant co-colleague” leh?
Need to handle task so I can do my job
I asked CHAPGPT n Claude. Both on server … not as safe

Thanks!

IMHO.
Give AI your requirement and ask on the hardware requirement, then move from there.

 

[Lss]

Llama? Or got better alternatives

 

[naked123]

Ollama? LM Studio?

I ran Gemma4 26B on my 9060 XT 16GB and got about 55 tokens per second.
Needs 32GB of system ram cos 16GB vram GPU is not enough and will spill over to system memory.
Context length (10K) is also significantly lower than others due to the lack of memory.

Qwen3.5 9B can fit entirely in VRAM but not as precise/detailed as Gemma4. Also roughly 58 tokens per second with roughly 200K context length.

GPT-OSS 20B can also fit entirely in VRAM but training cut off period is roughly 2023. So quite outdated. Roughly 58 tokens per second with 100K context length.