Posts promising a "$0 AI stack" circulate constantly. Download an open-weight model, run it with a local model server, store your documents in an open-source vector database, wire it together with a free agent framework, and host the front end on a free tier. No invoice, no vendor, no data leaving your laptop.
Most of that is true. It is possible to build a working AI assistant this way, and doing it is one of the best ways to learn how these systems work. The trouble starts when the demo impresses someone and the question becomes "can we just use this?" At that point the zero-rand architecture has not changed, but the obligations around it have. Costs that were invisible on a laptop appear as soon as other people depend on the system.
CloudNala's view is that a zero-cost AI architecture is a learning environment, not a production plan. Free components move cost from the invoice to someone's time and to the organisation's risk. That trade is excellent for learning and usually poor for production. The honest comparison is total cost of operation, not licence cost.
The running example
Nyala Media is fictional, and its systems and figures are illustrative. A developer on Nyala's digital team built a research assistant for the newsroom over two weekends. It runs an open-weight model on a workstation with a consumer GPU, indexes Nyala's published archive in an open-source vector store, and answers journalists' questions about past coverage with links to the source articles. It cost nothing beyond the workstation the developer already had.
Twelve journalists started using it within a week. The news editor asked whether it could be rolled out to the whole newsroom of about ninety people and connected to the unpublished drafts as well.
What free actually means
Each free component carries an obligation that is easy to miss on a laptop.
An open-weight model is free to download, but not free to run. It needs hardware with enough memory, and serving it to ninety people at once needs more than one workstation. Open-weight models also come with licence terms, which vary between model families and sometimes restrict commercial use or scale. Someone has to read them.
A local model server is software that must be patched, kept running and protected. On a laptop, access control is the laptop's login. On a shared server, it is a design decision.
An open-source vector store holds a copy of your content. Once that content includes unpublished drafts, the store contains some of the most sensitive material a newsroom has. It needs backups, access rules that match the source system's rules, and a retention policy.
An agent framework moves fast. Releases can change interfaces between versions, and a shared system needs pinned versions and a tested upgrade path rather than "latest".
Free hosting tiers have limits, apps that sleep when idle, and terms that can change with short notice.
A laptop is not a production environment. It has one owner, no redundancy, and it goes home at night.
None of these is a reason not to use open-source components. Many production systems are built on them. They are a reason to count what they cost to operate.
Cost appears as responsibility grows
The architecture of Nyala's assistant barely needed to change between the developer's laptop and a newsroom service. What changed at each step was who depended on it.
Learning. The developer's own time was the only cost. If it broke, nobody else noticed.
Team demo. Twelve journalists using it meant a shared host, which meant access control, and someone to restart it when it stopped.
Internal pilot. Ninety users and unpublished drafts meant support during working hours, an evaluation set to check answers were grounded in the right sources, and rules about what content may be indexed and who may see answers drawn from it.
Production. A service the newsroom relies on for deadlines means an agreed availability level, security review, someone on call, audit records and a plan for when the model or framework needs upgrading.
By the production step, the dominant costs were people's time and the risk of getting the sensitive-content rules wrong. The licence cost was still zero. It had stopped being the number that mattered.
Comparing options honestly
Nyala's technology lead put three options side by side, using total cost of operation over a year rather than licence fees. All figures were Nyala's own estimates and are illustrative.
| Cost element | Self-hosted open stack | Managed cloud AI services | Hybrid |
|---|---|---|---|
| Licences and API usage | Zero licence; hardware or cloud GPU | Pay per use | Pay per use for generation |
| Hardware or hosting | GPU server, power, space | None directly | Small, for the index |
| Patching and upgrades | Nyala's team | Mostly provider | Shared |
| Security and access control | Built by Nyala | Configured by Nyala | Configured by Nyala |
| Data location | Fully in Nyala's control | Depends on service and region | Index local, prompts to provider |
| Skills needed | Model serving, GPU operations | Cloud configuration | Both, at smaller scale |
Nyala chose the hybrid. The archive index stayed under Nyala's control, generation used a managed model service, and unpublished drafts were excluded until access rules matching the editorial system were in place. The developer's laptop version was kept as a sandbox for trying new models.
The choice will not be the same for everyone. An organisation with a strong platform team, GPU capacity already in place, or data that must not leave its own infrastructure may rationally choose the self-hosted route. The point is to make that decision with the operating costs visible. Our article Hosted or self-hosted: choosing a model path from South Africa covers that decision in more depth, and Why AI is an operating cost, not a once-off purchase covers the budgeting side.
Keep the free stack, for the right job
None of this argues against building on free and open components. The zero-rand stack has real strengths:
- It is the best way for engineers to learn how models, retrieval and agents behave, failures included.
- It lets a team test an idea with real content before asking for budget.
- It keeps sensitive experimentation entirely under the organisation's control.
- It avoids early lock-in while the use case is still uncertain.
The mistake is treating a successful experiment as a finished architecture. When other people start to depend on it, ask who will run it, who will secure it, who will answer for its outputs, and what that costs per year. If those questions have good answers, open components may well be the right production choice. If they do not, the zero-rand assistant has done its job: it proved the idea is worth paying to run properly.
If you are learning, our Build Your Way into AI Engineering series is a good companion to a home-built stack, and Your first AI project should include failure and cost explains why counting cost early is part of the learning.
This is the last part of Cloud in Plain English. Return to Part 1, AWS and Azure use different names for many of the same ideas, or see all parts on the series page.
How CloudNala can help
We help teams take a promising AI prototype and decide, with operating costs visible, whether it should become a self-hosted service, a managed one or a hybrid. That includes access rules for sensitive content, evaluation, support and a realistic annual cost, so the decision is made before the prototype quietly becomes production.
Work with CloudNala
CloudNala helps organisations move from technology ambition to practical execution across cloud, AI, data, platform engineering and digital services.
Whether you are exploring AI, modernising your cloud environment, building a public-sector digital service, or turning an idea into a working MVP, we can help you shape the roadmap and deliver the next step.
Request a Cloud Review or write to us at consult@cloudnala.co.za