“Localized” AI on personal or office pc?

[gld998]

Ollama? LM Studio?

I ran Gemma4 26B on my 9060 XT 16GB and got about 55 tokens per second.
Needs 32GB of system ram cos 16GB vram GPU is not enough and will spill over to system memory.
Context length (10K) is also significantly lower than others due to the lack of memory.

Qwen3.5 9B can fit entirely in VRAM but not as precise/detailed as Gemma4. Also roughly 58 tokens per second with roughly 200K context length.

GPT-OSS 20B can also fit entirely in VRAM but training cut off period is roughly 2023. So quite outdated. Roughly 58 tokens per second with 100K context length.

As long if fits the purpose its good enough.. just ensure you enable 'self learn' My main setup is running on Genma 4 as its most stable. Ollama recently there is an update to improve stability. No more half way stop running issue etc.

 

[One Piece]

Depends on how "good" you want the AI model to be. For simple tasks like summarizing documents and text classifications, organize documents , generate simple document templates those 14B parameters and below small language models are sufficient and can they run on normal consumer specs computers. Any tasks more complex and sophisticated you need to run 20+ billions or up to hundreds of billions of parameters large models which require better cpu, gpu and memory hardware.

 

[gld998]

Maybe just simple two page documents leh
Co work if for eg sensitive documents means data will be used to upload n compare
I think ChatGPT “more honest” abt it whereas Claude will wana try to appear as more trustworthy n as long as u choose don’t use for sharing, u are “safe” whereas ChatGPT more upfront abt it

Just liked I tried Claude do nice nice until like take short cut n chut pattern .. that’s when skills are taught hor

Run Gemini notebook from Google. Its free. Great to compare documents. I always use this.
Sanitized and upload. Should be fine.

Guide on how to use

 

[mirrorlakestudio]

If the worry is confidential docs leaving the company, 3 options:

(1) run a small model on your own PC with Ollama or LM Studio: free and fine for summarising short docs, but noticeably weaker than ChatGPT;
(2) use the business/enterprise plans of ChatGPT or Claude, which don't train on your data by default. Check with your IT first;
(3) self-host an open model on the company's own server, so data never leaves.

For most offices (2) with IT approval is the practical answer; (1) is good for personal use.

 

[samferguson]

Don't forget to consider electricity bills also.

 

[NintendoSwitch]

For a start is too expensive liao.

especially more so ram and graphics card sshortage caused by data centre.

you are looking at $5k minimum for a “normal “ ai

Prices for GPUs were ridiculously high even before AI since Jensen fueled the crypto boom. Prices never came back down. Local AI setup is good if you have unlimited money when working in government. Not feasible for most other people anymore.

 

[NintendoSwitch]

Ollama? LM Studio?

I ran Gemma4 26B on my 9060 XT 16GB and got about 55 tokens per second.
Needs 32GB of system ram cos 16GB vram GPU is not enough and will spill over to system memory.
Context length (10K) is also significantly lower than others due to the lack of memory.

Qwen3.5 9B can fit entirely in VRAM but not as precise/detailed as Gemma4. Also roughly 58 tokens per second with roughly 200K context length.

GPT-OSS 20B can also fit entirely in VRAM but training cut off period is roughly 2023. So quite outdated. Roughly 58 tokens per second with 100K context length.

Bro naked123, can explain what you meant by 55 tokens per second to noob like moi. I tot tokens is consumed by AI when it processes the user's input. Did you mean that the setup you listed can process up to 55 tokens of processing per second?

 

[Elnoxv]

Let’s say I don’t want to use ChatGPT or Claude Ai due to nature of work (confidential data) that is best to stay local than to upload up to their server ….

Which AI sw is gd to be our “assistant co-colleague” leh?
Need to handle task so I can do my job
I asked CHAPGPT n Claude. Both on server … not as safe

Thanks!

You will need an RTX3090, use Qwen3.8 27B that's the most reliable to use right now for agentic work. Of course if you have budget, go for Mac Studio Ultra 128gbB / 256gb for the Qwen3.8 Flash Next model.

Download Deepseek harness and configure with it. You can do co-work / codex work.. install skills, MCP etc.

Other models won't work as reliably but do note: They are slower..

 

[Elnoxv]

Bro naked123, can explain what you meant by 55 tokens per second to noob like moi. I tot tokens is consumed by AI when it processes the user's input. Did you mean that the setup you listed can process up to 55 tokens of processing per second?

There are two important performance metrics when running LLMs locally: TTFT (Time to First Token) and decode speed.

TTFT measures how long it takes before the model produces its first output token. A major part of this is prefill, where the model processes your input prompt.

Decode speed measures how quickly the model generates subsequent tokens, typically in tokens per second (tokens/s).

Both matter, but their importance depends on your workload. Long prompts require fast prefill, while long responses benefit from fast decoding.

Apple Silicon Macs generally have slower prefill and decoding performance than high-end NVIDIA GPUs when running the same model entirely in GPU memory.

However, Macs have an advantage in unified memory capacity, allowing them to run larger models that might not fit into a single consumer NVIDIA GPU. MOE model will be the best for this hardware.

For serious local LLM use, particularly coding agents and workloads where speed matters, I'd generally recommend an NVIDIA GPU such as the RTX 3090, 4090, or 5090.

The RTX 5080 is also capable, but its 16GB VRAM can be restrictive compared to the 24GB available on the 3090 and 4090.

 

[KakiMeow]

how confidential siol

nowadays some gahmen projects also on Amazon Cloud

---

The Singapore government runs several major cloud and artificial intelligence (AI) projects on Amazon Web Services (AWS) under the Whole-of-Government (WOG) Government Commercial Cloud (GCC 2.0) framework: [1, 2, 3]

Key Singapore Government Projects & Initiatives on AWS

• • GovTech MAESTRO & Analytics.gov
    • MAESTRO (Machine Learning & AI Enterprise-level Secure Tool-Suite): Developed jointly by GovTech and the AWS Generative AI Innovation Center using Amazon Bedrock and Amazon SageMaker. It cuts machine learning deployment time and has been adopted by 20 public organizations and 45+ teams.
    • Analytics.gov: A cloud-native data analytics platform running on AWS that enables public sector agencies to process and analyze large datasets securely. [1, 2]
  • Ministry of Manpower (MOM) Document Processing & SSOC Autocoder
    • Processed over 1 million documents using AWS-powered AI tools, cutting sensemaking time by 50% and saving 2,000+ hours.
    • Implemented the SSOC Autocoder to classify 10 million job postings with a 92% accuracy rate. [1]
  • Central Provident Fund Board (CPFB) Contact Center
    • Deployed Amazon Connect integrated with Salesforce for AI-powered speech transcription, summarization, and evaluation across English, Mandarin, Malay, and Tamil.
    • Achieved a 90% first-call resolution rate, lowered per-call costs by up to 15%, and saved 20,000 agent hours annually. [1]
  • SkillsFuture Singapore (SSG) Digital Portal
    • Powers SSG's national digital training and skills-insight portal using AWS to deliver lifelong learning resources and the Skills Demand for the Future Economy data to Singaporeans. [1]
  • National Library Board (NLB) – StoryGen
    • Built using AWS generative AI services to dynamically generate localized and interactive storytelling experiences for library users. [1]
  • Maritime and Port Authority of Singapore (MPA)
    • Operates the Maritime AI-ML Digital Hub on AWS to boost port efficiency, vessel tracking, and traffic management using cloud-based high-performance computing. [1]

---

Infrastructure & Policy Background

• • Government Commercial Cloud (GCC 2.0): AWS became available on GCC 2.0, featuring Policy-as-Code for continuous security compliance and streamlined agency onboarding via TechPass and SEED. [1]
  • AI Spring Singapore: A joint program between the Singapore government and AWS under the National AI Strategy 2.0 (NAIS 2.0) to accelerate public-sector and enterprise AI adoption. [1, 2]

 

[carlongerdidu93]

dunno wat business jus do a search for ai small businesses
can set up ai to ans and reply emails , take orders into spreadsheet
if got money transactions shld be own self hosted , set up own server dont need state of art

 

[NintendoSwitch]

There are two important performance metrics when running LLMs locally: TTFT (Time to First Token) and decode speed.

TTFT measures how long it takes before the model produces its first output token. A major part of this is prefill, where the model processes your input prompt.

Decode speed measures how quickly the model generates subsequent tokens, typically in tokens per second (tokens/s).

Both matter, but their importance depends on your workload. Long prompts require fast prefill, while long responses benefit from fast decoding.

Apple Silicon Macs generally have slower prefill and decoding performance than high-end NVIDIA GPUs when running the same model entirely in GPU memory.

However, Macs have an advantage in unified memory capacity, allowing them to run larger models that might not fit into a single consumer NVIDIA GPU. MOE model will be the best for this hardware.

For serious local LLM use, particularly coding agents and workloads where speed matters, I'd generally recommend an NVIDIA GPU such as the RTX 3090, 4090, or 5090.

The RTX 5080 is also capable, but its 16GB VRAM can be restrictive compared to the 24GB available on the 3090 and 4090.

What about laptop GPUs? What would you recommend when desktops are not an option on the table?

 

[Elnoxv]

What about laptop GPUs? What would you recommend when desktops are not an option on the table?

Few options:
1. Laptop with mobile rtx gpu. min 24gb vram
2. Normal Laptop with e-gpu
3. Macbook Pro M5 Max 128gb
4. Laptop with AMD Strix Halo 128gb.

Speed is in order from 1 being the fastest to 4 slowest.

Or alternatively:
1. Setup TailScale / Wireguard at home. (Allow LAN connection)
2. Buy Mac Studio / Rtx gpu and run it 24/7
3. Connect to Mac Studio / linux At home via WireGuard or TailScale.

 

[Thoreldan]

The CISO/Chief Data officer should be driving the AI Initiatives focusing on security and data privacy within the enterprise. The strategy and SOP should be in place, not let employees decide what they want to run. Can you imagine the chaos when everyone uses their favorite LLM tools ?

 

[NintendoSwitch]

Few options:
1. Laptop with mobile rtx gpu. min 24gb vram
2. Normal Laptop with e-gpu
3. Macbook Pro M5 Max 128gb
4. Laptop with AMD Strix Halo 128gb.

Speed is in order from 1 being the fastest to 4 slowest.

Or alternatively:
1. Setup TailScale / Wireguard at home. (Allow LAN connection)
2. Buy Mac Studio / Rtx gpu and run it 24/7
3. Connect to Mac Studio / linux At home via WireGuard or TailScale.

Thanks Bro Elnoxv. Unfortunately all 7 options are out of my current budget.

 

[Elnoxv]

Thanks Bro Elnoxv. Unfortunately all 7 options are out of my current budget.

If you have an existing PC, the setup cost is actually less than < $2K.

Just buy a used RTX3090 ( around $1.5 - 2k). Run wireguard / tailscale at home.

From your laptop connects to tailscale / wireguard.
Point your harness: Claude code / Deepseek to where your inference provider located.

that's it.

 

[gld998]

from what i see in youtube, the setup take up huge amount of resources and still produce sluggish replies...no point

use a small or micro llm.. its powerful enough to do jobs locally. I m using now...

 

[Pearce]

To setup your own Ai, you need a powerful GPU. Otherwise, your comp or laptop will be very lag and like hang. That's why a local system just for AI cost from $8999 onwards, AMD from $6000 plus onwards.

 

[gld998]

how confidential siol

nowadays some gahmen projects also on Amazon Cloud

---

The Singapore government runs several major cloud and artificial intelligence (AI) projects on Amazon Web Services (AWS) under the Whole-of-Government (WOG) Government Commercial Cloud (GCC 2.0) framework: [1, 2, 3]

Key Singapore Government Projects & Initiatives on AWS

• • GovTech MAESTRO & Analytics.gov
    • MAESTRO (Machine Learning & AI Enterprise-level Secure Tool-Suite): Developed jointly by GovTech and the AWS Generative AI Innovation Center using Amazon Bedrock and Amazon SageMaker. It cuts machine learning deployment time and has been adopted by 20 public organizations and 45+ teams.
    • Analytics.gov: A cloud-native data analytics platform running on AWS that enables public sector agencies to process and analyze large datasets securely. [1, 2]
  • Ministry of Manpower (MOM) Document Processing & SSOC Autocoder
    • Processed over 1 million documents using AWS-powered AI tools, cutting sensemaking time by 50% and saving 2,000+ hours.
    • Implemented the SSOC Autocoder to classify 10 million job postings with a 92% accuracy rate. [1]
  • Central Provident Fund Board (CPFB) Contact Center
    • Deployed Amazon Connect integrated with Salesforce for AI-powered speech transcription, summarization, and evaluation across English, Mandarin, Malay, and Tamil.
    • Achieved a 90% first-call resolution rate, lowered per-call costs by up to 15%, and saved 20,000 agent hours annually. [1]
  • SkillsFuture Singapore (SSG) Digital Portal
    • Powers SSG's national digital training and skills-insight portal using AWS to deliver lifelong learning resources and the Skills Demand for the Future Economy data to Singaporeans. [1]
  • National Library Board (NLB) – StoryGen
    • Built using AWS generative AI services to dynamically generate localized and interactive storytelling experiences for library users. [1]
  • Maritime and Port Authority of Singapore (MPA)
    • Operates the Maritime AI-ML Digital Hub on AWS to boost port efficiency, vessel tracking, and traffic management using cloud-based high-performance computing. [1]

---

Infrastructure & Policy Background

• • Government Commercial Cloud (GCC 2.0): AWS became available on GCC 2.0, featuring Policy-as-Code for continuous security compliance and streamlined agency onboarding via TechPass and SEED. [1]
  • AI Spring Singapore: A joint program between the Singapore government and AWS under the National AI Strategy 2.0 (NAIS 2.0) to accelerate public-sector and enterprise AI adoption. [1, 2]

GCC all info is stays in SG and its separated from commercial cloud.

 

[keenklee]

IMHO.
I hope to start small, using my 5800H with 32GB and see what I can do with it.