The AI architect capability matrix, and a 90-day lab

Last year, by the end of the design phase, the lead cloud architect at Outeniqua Assurance, a fictional South African short-term insurer, had extended her design review and her architecture template for the claims assistant. Outeniqua's claims-assistance design is illustrative. Her manager then asked the question every architect in this position eventually hears: "How do we know you are now an AI architect, and how do we grow two more?"

A course list would not answer it. Neither would a certificate. Both measure exposure to material, not the ability to design and defend an AI system. What she needed was a way to say which capabilities an AI systems architect needs, to what depth, and how to show the depth with evidence from real work. This final part sets out both: a capability matrix and a 90-day lab.

Three depths, not one bar

Part 1 introduced three depths. They matter because most of the anxiety about AI careers comes from assuming every skill must be mastered.

• Awareness. You can explain the technique, recognise when it applies and ask a specialist the right question. You cannot yet design it alone.
• Working proficiency. You can design it, review someone else's design and challenge a weak one. You have done it at least once on realistic material.
• Specialist depth. You can build it, tune it and diagnose it when it fails in production. You are the person others ask.

An AI systems architect is defined by a profile across these depths, not by one score. Specialist depth everywhere is neither possible nor useful.

The capability matrix

The matrix lists thirteen capability areas. The first six are cloud disciplines that change shape in AI systems, as Part 2 described. Tool permissions, which Part 1 listed as new, sit in the first row because the discipline behind them is identity. The remaining seven are new or largely new, and behaviour drift is part of evaluation design. For each, it sets the depth CloudNala expects of an AI systems architect on three tracks.

The tracks describe the work, not seniority. Foundation is for an architect who reviews AI designs and must not miss the obvious risks. Production is for an architect who is accountable for an AI system going live. Advanced is for an architect who sets standards for others and owns the hardest design decisions.

Three choices in the table are deliberate.

First, model training and fine-tuning stay at awareness on every track. An advanced-track architect in an organisation that fine-tunes its own models may choose to go further, but the matrix does not require it. Most enterprise AI systems use models trained by someone else. An architect should know when fine-tuning is the right answer, and RAG, fine-tuning and knowledge graphs solve different problems is a good starting point, but designing training pipelines is specialist work that most architects will commission rather than perform.

Second, identity, reliability, cost per outcome and human authority need working proficiency from the start. These are where AI systems most often fail in regulated organisations, and they are also where an experienced architect's existing judgement helps most.

Third, evaluation design is the area where the jump to specialist depth matters most. The advanced track asks for specialist depth in four areas, but an architect who sets standards for evaluation must have built and maintained evaluation sets, not only read about them, and that experience is the hardest to acquire second-hand.

This is CloudNala's working model, built from our design reviews. It is not an industry standard. Adapt the areas and depths to your organisation, but keep the principle: name the depth for each area, and be honest about which ones are awareness.

How the matrix differs from a role profile

Organisation-wide learning is a different problem. Learning paths by role, not by tool describes how to group a whole workforce by the verbs their roles perform, and Four levels of AI engineering maturity describes how an organisation's practice matures. The matrix here is narrower. It is for one experienced architect, or a small architecture team, making a deliberate transition, and it is tied to evidence from one real system.

The 90-day lab

The lab is how the matrix becomes evidence. It is a small, safe system that an architect builds, breaks and documents over 90 days, alongside normal work. Outeniqua's architect built a reduced claims assistant using synthetic claims, invented policy wording and no production data. Every artefact she produced was reviewed by someone who does that work for real.

Days 1 to 30: a grounded assistant that can be evaluated

• Write a synthetic corpus: two invented policy wordings with exclusions, twenty synthetic claims, photographs replaced by text descriptions.
• Build grounded retrieval over the policy wording, with source references in every answer.
• Write forty evaluation cases with expected answers, including cases where the right answer is "the wording does not say".
• Record the first architecture decision record: why retrieval and not fine-tuning.

Evidence at day 30: the evaluation set, a first scored run, and the decision record. Reviewed by an engineer who has built evaluation sets before.

A zero-rand AI architecture is for learning is a fair description of this stage. It is a learning system, deliberately free of production promises.

Days 31 to 60: production concerns

• Add one tool: a read-only lookup of a synthetic claim record, called under the identity of the person asking.
• Add a typed output for the claim summary and validate it on return.
• Add tracing that records retrieved passages, model version and tool calls, and decide what must not be stored.
• Calculate cost per assisted claim at three volumes.
• Define a degraded mode and test it by switching off the model endpoint.
• Write the authority statement: what the assistant may summarise, flag and request, and what remains with a claims official.

Evidence at day 60: a threat model for the tool call, the cost model, a trace from a failed run, and the authority statement. Reviewed by a security architect and a claims operations manager.

Days 61 to 90: change and judgement

• Swap the model for a smaller one and rerun the evaluation set. Decide, with evidence, whether the change can ship.
• Change one paragraph of the policy wording and show which evaluation cases now fail and why.
• Introduce one deliberately hostile document (an instruction hidden in a repair quote) and show that it cannot change the outcome, because the assistant has no authority to change it, and that the summary and requests are unaffected or the document is flagged for a person.
• Present the design to a mixed panel of an architect, an engineer, a risk specialist and a business owner, using the extended template from Part 2.

Evidence at day 90: a change-impact report, the hostile-input test, and the panel's written questions and the answers. The panel decides whether the evidence supports production-track depth in each area.

An evaluation lab you can build in a week covers the tooling for the evaluation part if you want to start faster.

What the lab showed at Outeniqua

The lab exposed two gaps that reading had not. The architect's first evaluation set contained almost no cases where the correct answer was "I cannot tell from the wording", so the assistant's habit of guessing went unnoticed until the panel asked about it. And her first cost model assumed one model call per claim, while the traces showed between three and seven calls once retries and long quotes were included. Both findings changed the production design.

The lab also settled the original question. At day 90 the panel rated her at working proficiency in ten of the thirteen areas, specialist in human authority and identity, and awareness in model training, which met or exceeded the matrix's production track. Two colleagues then started the same lab with her as a reviewer. That is how a capability spreads: through reviewed evidence, not attendance records.

Making it work in your organisation

• Use synthetic data. The lab must be safe to break, share and discuss. No production claims, no real customer documents, no real policy numbers.
• Name reviewers before day 1. A lab without reviewers becomes a hobby project.
• Protect the time. Outeniqua set aside about half a day a week. Less than that and the lab stalls in the second month.
• Keep artefacts, not slides. The decision records, evaluation runs, traces and threat models are the evidence. They also become templates for real projects.
• Treat certificates as optional. Where they fit is covered in Learning paths by role, not by tool; if you hire against this matrix, Hire on evidence explains how to assess it fairly.

Key points

• Describe an AI systems architect as a profile across three depths: awareness, working proficiency and specialist depth.
• Use a capability matrix with explicit depths per track; keep model training at awareness for most architects.
• Identity, reliability, cost per outcome and human authority need working proficiency from the foundation track onwards.
• A 90-day lab on synthetic data, with named reviewers, turns the matrix into evidence: evaluation sets, cost models, traces, threat models and a panel review.
• Capability spreads through reviewed artefacts and peer review, not course completions or certificate counts.

This is the final part of From Cloud Architect to AI Systems Architect.

How CloudNala can help

CloudNala runs AI architecture reviews and capability labs for architecture teams moving into AI systems. We adapt the capability matrix to your estate and regulators, set up a synthetic lab based on one of your own planned use cases, act as an independent reviewer at each 30-day gate, and run the closing panel review, so your architects finish with evidence your organisation can rely on.

Work with CloudNala

CloudNala helps organisations move from technology ambition to practical execution across cloud, AI, data, platform engineering and digital services.

Whether you are exploring AI, modernising your cloud environment, building a public-sector digital service, or turning an idea into a working MVP, we can help you shape the roadmap and deliver the next step.

Book an AI Architecture Review or write to us at consult@cloudnala.co.za