Hiring managers in South African data teams describe the same experience. A candidate's portfolio lists a lakehouse, a streaming pipeline, an orchestration tool and an AI document pipeline. The interview reveals that each project ran once, on a sample dataset, and was never asked to survive a bad load, a schema change or a monthly cloud bill.
That is not the candidate's fault. Most learning material teaches tools, and tools are easy to demonstrate. What is hard to demonstrate, and what teams actually need, is the ability to run data that other people depend on. The earlier parts of this series described what that requires: ownership, contracts, checks, service levels and the right technique for each question.
This part turns those ideas into a practical portfolio. CloudNala's position is that a credible data engineering portfolio shows failure and recovery, not just construction, and that ten modest projects done properly prove more than one ambitious project done once. It is written for two readers: people building their skills, and the leaders who hire or develop them.
Gariep Grain Co-operative is fictional, and its systems and figures are illustrative. The projects below use a Gariep-style domain, members, weighbridge tickets, grading samples, silo bins and payments, because it is rich enough to be realistic and simple enough to generate synthetic data for. Any domain with a few related entities and a daily flow will do.
Three tiers, in order
The ten projects fall into three tiers. Each assumes the one below it.
Tier one: move data reliably
1. Incremental batch load. Load weighbridge tickets from an operational database into a warehouse, picking up only new and changed rows each night. Business outcome: a daily deliveries table that is complete by a stated time. Evidence: the load is rerun after a deliberate mid-run failure and produces no duplicates or gaps. Show the watermark logic and what happens on a late-arriving ticket.
2. Change data capture. Capture inserts, updates and deletes from the member registry as they happen, rather than by nightly comparison. Business outcome: member changes reach downstream systems within minutes. Evidence: a member's bank details are changed twice in an hour and both changes, in order, appear downstream with their timestamps. Explain how deletes are handled, because that is where most CDC designs are weakest.
3. Orchestration with retries and backfills. Schedule projects 1 and 2 with dependencies, retries and alerting, and run a backfill for a week of missing data. Business outcome: failures are retried or escalated without someone watching. Evidence: a run history showing a failure, an automatic retry, an alert, and a clean backfill. Show how the backfill avoids double-loading.
Tier two: make it trustworthy
4. Tests and a contract. Write a data contract for the deliveries table, as described in Part 1, and implement checks for schema, volume, ranges and references. Business outcome: bad loads are stopped before consumers see them. Evidence: a planted bad load, for example a silo exporting kilograms instead of tonnes, is caught and blocked. A test suite that has never failed proves nothing.
5. Dimensional model and semantic definitions. Model deliveries, members, silos and dates as facts and dimensions, and define the business measures once. Business outcome: "delivered tonnage" means the same thing in every report. Evidence: the definition of each measure, written down, with the two or three alternative definitions you deliberately did not use and why, which is the ownership question from Part 2.
6. Lineage and a catalogue. Capture lineage from source to report and publish the datasets in a catalogue. OpenLineage, a graduated project of the LF AI and Data Foundation, defines an open standard for lineage events that many orchestration and processing tools can emit. Business outcome: anyone can see where a number came from and what breaks if a source changes. Evidence: for one dashboard figure, a lineage view back to the weighbridge, and an impact list for a proposed column change.
7. Personal information controls. Classify the personal information in the member data, mask or tokenise it for analytical use, and restrict access by role. Business outcome: analysts can work with delivery patterns without seeing identity numbers or bank details. Evidence: the classification, the masking rule per column, and a demonstration that a restricted role cannot see the raw values. Note which POPIA questions you would refer to a privacy officer rather than decide yourself.
Tier three: run it for real
8. An event stream. Stream weighbridge events from silos as they happen, with a design for sites that lose connectivity. Business outcome: intake is visible during the day, not only the next morning. Evidence: a simulated four-hour outage at one site, after which buffered events arrive in order and are not duplicated. In South Africa, intermittent connectivity and power at rural sites are design inputs, not edge cases.
9. A cost-aware pipeline. Run the whole stack within a stated monthly budget in rand, and show what you changed to stay inside it. Business outcome: a pipeline the organisation can afford to keep running. Evidence: a monthly cost breakdown, the two or three biggest drivers, and the design changes that reduced them: partitioning, smaller compute, scheduled shutdowns, avoiding unnecessary data movement. Cloud services are mostly priced in US dollars, so a rand budget also carries exchange-rate risk worth noting.
10. A document pipeline for AI. Ingest policies and notices for retrieval, with versioning, metadata and permissions. Business outcome: an assistant always answers from the current version of a policy and only shows documents the user may see. Evidence: a policy is replaced and the old version stops being retrieved; a restricted document is not returned to an unauthorised user. The RAG That Survives Production series, especially the retrieval pipeline, end to end, covers what to test.
What reviewers actually look for
The evidence column is the point of the list. Two portfolios can contain identical tools and say very different things about the person who built them.
For each project, a short write-up beats a long one. A useful format is: the business problem in two sentences; the design and the one or two decisions that mattered; what failed and how you found out; what it costs to run; and what you would change. Hiring managers read the failure section first.
For leaders: use the list for development, not just hiring
The same ten projects work as a development path inside a data team. Pair each tier with a real dataset the team already owns. A junior engineer who has taken the deliveries table through projects 1, 3 and 4 with a senior reviewer has learnt more than a training course provides, and the organisation has a better deliveries table at the end.
Two adjustments help. First, require the evidence, not just the build: the planted failure, the backfill, the cost breakdown. Second, rotate people through the tier-two projects in particular. Contracts, definitions, lineage and privacy controls are where data engineering meets the business, and they are the skills most teams are short of.
The trade-off
Ten projects done properly take months, not weekends. Some candidates and some teams will reasonably choose fewer. If you must choose, projects 1, 3, 4 and 9 cover reliability, recovery, quality and cost, which are the four things most often missing. Streaming and AI pipelines are valuable, but they are tier three for a reason: built on a weak foundation, they fail in more interesting and more expensive ways.
The decision
If you are building a portfolio, pick one domain, generate synthetic data for it, and work up the tiers in order, writing the evidence as you go. If you lead a team, choose one dataset your organisation depends on and use the first tier as the next quarter's development plan.
Next: Part 6, Data quality and observability are operational disciplines. All parts are on the series page.
How CloudNala can help
We help data teams turn their most important datasets into working examples of reliable, tested and affordable pipelines, and use that work to develop their people. That can mean architecture and review support for a team's own build, or a short engagement to set up the first tier properly so the patterns can be repeated.
Work with CloudNala
CloudNala helps organisations move from technology ambition to practical execution across cloud, AI, data, platform engineering and digital services.
Whether you are exploring AI, modernising your cloud environment, building a public-sector digital service, or turning an idea into a working MVP, we can help you shape the roadmap and deliver the next step.
Book a Data Readiness Review or write to us at consult@cloudnala.co.za