Local models can already handle part of a company's workload. Documents, internal search, catalogs, and preliminary data processing may run on a company server, workstation, or user device.
Cloud models still have a clear role. The right choice depends on quality, privacy, latency, cost, and the team's ability to operate the infrastructure.
What changed with Gemma 3
Google describes Gemma 3 as a family of open models with 1B, 4B, 12B, and 27B parameters, built with technology from Gemini 2.0. Its stated capabilities include a 128K context window, multimodality in the larger versions, support for more than 140 languages, function calling, and structured output.
The announcement highlights the context size:
“Gemma 3 offers a 128k-token context window…” — Google Blog
Source: Google Blog, Introducing Gemma 3
A 128K window can hold long documents, instructions, task history, and knowledge-base fragments in one request. Window size alone does not guarantee an accurate answer, so the source material still needs selection and structure.
Why a company might need a local boundary
Cloud models are quick to adopt and often produce stronger results. They also send data to a provider, charge according to usage, and depend on regional availability, pricing, and service limits.
A local model can run inside a protected network. The company controls data location, access, logs, the model version, and its update policy.
Local deployment is not necessarily cheaper at the start. It requires hardware, inference setup, monitoring, and someone responsible for operations.
Why single-GPU deployment matters
Google DeepMind positions Gemma 3 as a family that can run on a single GPU or TPU:
“Gemma 3 is the most capable model that can run on a single GPU or TPU.” — Google DeepMind
Source: Google DeepMind, Gemma 3
The claim does not mean that the 27B version performs equally well on every laptop. Memory use, speed, and quality depend on model size, quantization, and the inference implementation.
A pilot may no longer require a server rack, however. A team can build a limited environment, measure quality on its own data, and decide whether further infrastructure is justified.
How quantization reduces memory requirements
Large model weights consume substantial memory. Quantization stores numbers at lower precision and reduces the capacity needed to run them.
Google released QAT variants of Gemma 3 that account for future low-precision operation during training. Its publication gives a specific example:
“This enables you to run powerful models like Gemma 3 27B locally on consumer-grade GPUs like the NVIDIA RTX 3090.” — Google Developers Blog
Source: Google Developers Blog, Gemma 3 QAT models
The deployment still needs testing on the actual configuration. A model that fits into memory may remain too slow or inaccurate for the intended process.
Multimodal use cases
Larger Gemma 3 versions can process images. Combined with OCR, internal databases, and tools, that capability expands the set of local workflows.
E-commerce. An assistant matches a customer's request against the catalog, product attributes, and inventory. A label photo can identify an item, while a separate tool checks current stock.
Manufacturing. An operator photographs an error code on a machine panel. The system finds the matching manual and suggests diagnostic steps, with engineer approval retained for important actions.
Documents. An internal assistant processes PDFs, contracts, invoices, and tables before preparing a summary and risk list. Source files remain inside company infrastructure.
Personal agents. Email, tasks, notes, and calendars can receive some local processing. Selected requests that need a stronger model may then go to the cloud.
When the cloud is more practical
A cloud model is usually the practical option when maximum reasoning quality is required, requests are rare, or launch speed has priority. The same applies to fresh web research and demanding multimodal work when a local stack cannot meet the requirements.
A hybrid design keeps sensitive and repeatable work inside the company. Rare, difficult requests can go to cloud services under explicit routing rules.
How to run a useful comparison
- Collect 20 to 30 real documents or conversations.
- Define the expected result and quality criteria.
- Run a local model, a cloud model, and a low-cost API model on the same set.
- Compare accuracy, speed, cost, privacy, and integration effort.
- Count the errors that a human reviewer must correct.
An agent also depends on tools, memory, permissions, logs, and action approval. The model alone does not determine system quality.
Local AI can now handle part of repeatable work close to the data. Its value lies in choosing the right boundary for each task, rather than trying to replace the cloud completely.