When you actually need an ML specialist

When Sabie's head of engineering decoded the company's eleven AI job adverts in Part 1, one finding was left unresolved. Two adverts carried the title "AI engineer" and described different jobs. One was about connecting hosted models to products. The other was about fine-tuning and testing a classification model on Sabie's own data, which is machine learning engineering under another name. Sabie merged the first with the "agent engineer" advert into a single integration role. The second did not become one of the five roles Sabie kept. It was parked, because nobody could say whether the company actually needed that person.

Six months later the question came back with a deadline. A logistics client asked Sabie to add delivery-time prediction to the dispatch platform Sabie builds for it, and a second client wanted its support tickets classified automatically. The sales lead assumed both needed "an ML person". The head of engineering suspected only one did, and wanted a way to decide that would survive the next request too.

Sabie Systems is fictional, and its systems and figures are illustrative. Sabie is a Pretoria software and data services company of about 140 people that builds line-of-business systems and data platforms for mid-sized clients. Two of Sabie's own products now include AI assistants built on hosted language models.

Most AI features no longer need one

For most of the last decade, any feature that learned from data needed someone who could train a model. That is no longer true for a large share of AI work. Sabie's two assistants call a hosted language model, retrieve the client's documents and return structured answers. Nobody at Sabie trained anything. The skills those features needed were integration, evaluation and operation, which Part 2 showed already sit in the developer, tester and operations roles.

That shift makes the ML specialist easy to over-hire and easy to under-hire. Over-hire, and a specialist spends a year building infrastructure for models the company did not need, or fine-tuning a model that a hosted one with a good prompt would have matched. Under-hire, and an application team ships a predictive model with leakage in the training data, no calibration and no plan for retraining, and the problem surfaces only when a client notices that predictions have drifted.

Google's Rules of Machine Learning, written by Martin Zinkevich, opens with a rule that has aged well: "Don't be afraid to launch a product without machine learning." Its third rule is the counterweight: once you have data and a clear goal, choose machine learning over a complex heuristic. The question is not whether ML is good. It is whether this product, now, needs someone whose core verb is to train and validate models.

Five signals you need an ML specialist

Using the verbs from Part 1, an ML specialist is someone whose core work is to build and evaluate models themselves, rather than to integrate and operate models built by someone else. Sabie's head of engineering wrote down five signals that this work is real.

1. You must train or adapt a model on your own data. The prediction depends on patterns in data only you hold, such as delivery times on a particular client's routes, and no hosted model can know them. Fine-tuning a hosted model also counts if you will own the resulting model and its behaviour.
2. The model is a long-lived asset that needs a lifecycle. It will need retraining as the world changes, monitoring for drift, and versioning of data and features. The paper Hidden Technical Debt in Machine Learning Systems (Sculley and colleagues, NeurIPS 2015) catalogued how much of that ongoing cost sits outside the model code itself: data dependencies, feedback loops and configuration.
3. Errors need statistical diagnosis. When predictions are wrong, the cause may be class imbalance, target leakage, sampling bias or poor calibration. An application engineer can see that accuracy dropped. Diagnosing why, and proving the fix, is specialist work.
4. Serving is your problem. If you host models yourself, for cost, latency or data-location reasons, someone has to optimise inference, choose hardware and manage model artefacts. With a hosted API, the provider does most of this.
5. Someone outside the team will validate the model. A client, auditor or regulator expects documented model development, validation and monitoring. Where that expectation is real, you need someone who can produce the evidence, not only the model.

One signal on its own may justify a short engagement. Three or more usually justify a specialist role.

Signals that look like a need, but are not

Several common reasons for hiring an ML specialist do not survive inspection.

• "We have a lot of data." Data volume is not a use case. Without a decision the prediction would change and a way to measure whether it did, the data stays a cost.
• "We need AI for text." Classification, extraction and summarisation of text are now usually done with hosted models plus evaluation. Choose the smallest model that passes your own evaluation before assuming training is needed.
• "We should fine-tune to make it know our business." If the real problem is that the model lacks your current documents and facts, retrieval is usually the better first tool, and fine-tuning is better reserved for behaviour and format, as RAG, fine-tuning and knowledge graphs solve different problems explains.
• "The client asked for machine learning." Clients ask for outcomes in the vocabulary they know. Translate the request into a decision and a measure before choosing the technique.

The two requests at Sabie

The ticket classification request failed the signals test. The client wanted each incoming support ticket labelled with one of fourteen categories and an urgency level. Sabie's application team built a version using a hosted model with a structured output contract, evaluated it on 600 historical tickets labelled by the client's support leads, and compared it with a keyword-rule baseline. The hosted model agreed with the support leads on most tickets and clearly outperformed the rules. No training, no model to retrain, no serving to manage. The evaluation, run by Sabie's tester, was the skilled work. That is the evaluation verb, not the ML specialist role.

The delivery-time request passed four of the five signals. The prediction depended on the client's own route, depot and time-of-day patterns. It would drift with roadworks, new depots and seasonal volume, so it needed retraining. Early experiments showed predictions that looked accurate on average but were badly wrong for rural routes, which are rarer in the data. And the client's operations director wanted to know how the model was validated before letting dispatchers rely on it. Only serving was not an issue, because a small tabular model is cheap to host.

The same company, the same quarter, two "AI" requests: one needed an evaluator and a good application engineer, the other needed a specialist.

A readiness test before you hire

Hiring an ML specialist into an organisation that cannot use one is a common and expensive mistake. The specialist arrives, finds no labelled data, no route to production and nobody to review their work, and spends months building scaffolding. Before hiring, Sabie checks six conditions.

The delivery-time work passed five of six. It had no reviewer: nobody at Sabie could check a specialist's validation. That shaped how Sabie got the skill. Four signals would normally point to a specialist role, but the readiness test overrode the count: a permanent hire with no reviewer and only one confirmed use case would have been a lone specialist.

Four ways to get the skill

Hiring is one option among four, and not always the first.

• Hire when the signals are persistent: several products need trained models, and the lifecycle work will not end. A permanent specialist also builds organisational knowledge that contractors take with them.
• Contract for a bounded piece of work with a defined hand-over, such as building and validating a first model, with documentation and a retraining procedure an engineer can follow. This suits a single use case with an uncertain future.
• Partner with a firm or academic group when the work needs depth Sabie will rarely need again, or when independent validation is itself the requirement.
• Grow an existing data engineer or analyst into the role, with mentoring from a contracted specialist. This is slower and works best where the data and domain knowledge already sit with that person.

Sabie chose a combination: a six-month contract with an experienced ML engineer to build the delivery-time model, with one of Sabie's data engineers paired on every step. The contractor reviews the data engineer's work, and a university statistics group, engaged separately as a partner, validates the model before dispatchers rely on it, so that nobody marks their own homework. That validation is the reviewer the readiness test had found missing. If a second client needs a trained model within that period, Sabie will open a permanent role, interviewing against a work sample as described in Part 4.

Writing the role, if you hire

If the decision is to hire, the role profile should follow the same pattern as the others in this series: core verbs, work products and a named reviewer. For Sabie's draft:

• Core verbs: build and evaluate, with operate shared with the platform team.
• Work products: a documented training dataset with its known limitations; a model card or equivalent describing intended use and performance by segment; a validation report; a monitoring and retraining procedure.
• Reviewer: an external specialist at first, and the head of data once that capability exists in-house.
• Not in scope: building chatbots, writing prompts for hosted models or acting as the company's general AI adviser. Those are other roles, and loading them onto the specialist is the fastest way to waste one.

Key points

• Most AI features now call hosted models and need integration, evaluation and operation skills, not an ML specialist.
• Five signals point to a real need: training on your own data, a model lifecycle, statistical diagnosis, self-managed serving and external validation.
• Data volume, text tasks, a wish to fine-tune for knowledge and a client's vocabulary are not, on their own, reasons to hire.
• Run a readiness test before hiring: decision, data, baseline, route to production, reviewer and owner.
• Hire, contract, partner or grow according to how persistent the need is, and always give the specialist a reviewer.

This is the final part of AI Roles Decoded. Start again with Part 1, AI roles decoded: read the verbs, not the title, or see how evaluation gates real work for the skill both of Sabie's requests depended on.

How CloudNala can help

CloudNala helps technology leaders decide whether an AI request needs machine learning specialists, application engineers or neither. We test the request against the signals and the readiness conditions, size the work, and recommend whether to hire, contract, partner or grow the capability, with a role profile and a hand-over plan if specialist work is needed.

Work with CloudNala

CloudNala helps organisations move from technology ambition to practical execution across cloud, AI, data, platform engineering and digital services.

Whether you are exploring AI, modernising your cloud environment, building a public-sector digital service, or turning an idea into a working MVP, we can help you shape the roadmap and deliver the next step.

Book an AI Capability Planning Session or write to us at consult@cloudnala.co.za