Two reports, two answers, one question
The same metric is calculated differently in different systems because definitions were never agreed and no source is authoritative. Meetings spend their time reconciling rather than deciding.
Data engineering and AI readiness consulting
Most stalled AI initiatives are data problems wearing a different label. Before a model is worth building, the organisation needs data it can locate, trust, join and explain. We are engaged to build that foundation and to be candid about what it makes possible.
Business challenges
This work is commissioned when data has become a constraint on decisions rather than a by-product of systems.
Pipelines break silently
Extractions fail or arrive partially and nobody is alerted, so downstream reports are wrong until a person notices. Trust in the data falls faster than it can be rebuilt.
Analysts spend their time on plumbing
Most analytical effort goes into locating, cleaning and joining data rather than into analysis, because that work is repeated per question instead of solved once.
AI initiatives stall at the data step
A model was scoped before anyone established whether the required data exists, is labelled, is legally usable and can be served at the latency the use case needs.
Nobody can explain where a number came from
Lineage is undocumented, so an auditor's question or a disputed figure requires a manual investigation each time it is asked.
The warehouse cost curve is outpacing its value
Storage and compute grow with retention and query patterns that nobody owns, and no allocation exists to attribute the spend to a team.
Business outcomes
What changes when data is engineered as a product rather than extracted on request.
- Metrics defined once with an authoritative source, so reconciliation stops consuming decision time
- Pipeline failures detected and alerted rather than discovered downstream
- Analytical effort spent on analysis because access and joining are solved once
- An honest assessment of which AI use cases the current data foundation can support
- Lineage that answers where a number came from without a manual investigation
Why data engineering before AI
The constraint is almost never the model.
Organisations frequently arrive with an AI objective and discover the work is upstream: the data is spread across systems with different definitions, quality is unmeasured, lineage is undocumented, and the legal basis for using it in a model has not been established.
None of that is solved by selecting a model or a vendor. It is solved by engineering the data foundation — which also produces value on its own, because the same foundation improves reporting and operational decisions whether or not a model is ever deployed.
We will tell you when an AI use case is not currently supportable, and what would have to be true for it to become so. That is more useful than a proof of concept that cannot be productionised.
Our consulting approach
Four commitments that shape every data engagement.
Definitions before pipelines
A metric with two definitions has no authoritative value regardless of how well the pipeline is engineered. Semantics are agreed with the business before the technical work starts.
Data as a product
Each significant dataset has an owner, a documented contract, quality expectations and consumers who can rely on it. Data without an owner degrades by default.
Quality measured, not asserted
Completeness, freshness and validity are tested continuously and alerted on. Quality claimed without measurement is discovered to be false by a consumer, usually publicly.
Honest about AI readiness
We assess whether the data required by a use case exists, is usable and can be served at the necessary latency — and say so plainly when it cannot.
Enterprise data strategy
Data strategy answers a small number of questions that constrain everything downstream: which domains own which data, what is mastered where, which decisions the data is meant to support, and what the organisation is obliged to retain, protect or delete.
Without those answers, platform work produces a technically capable system that still cannot settle a disagreement about a number.
Data architecture
Architecture decides where data lands, how it is modelled and how consumers reach it. The choices are consequential and difficult to revisit once consumers depend on them.
We design layered storage with defined responsibilities, model for the access patterns that actually exist, and are explicit about which consumers get raw data and which get curated products.
- Layered storage with defined responsibility per layer, not a single dumping ground
- Modelling driven by real access patterns rather than by a preferred methodology
- Batch and streaming applied where each genuinely fits the latency requirement
- A stated position on which system masters which entity
Data platform modernisation
Modernisation is warranted when the existing platform constrains the questions that can be asked, or when its cost has decoupled from its value. It is not warranted merely because the technology is older.
We sequence migration so reporting continues throughout, usually by running the new platform alongside and cutting consumers over per domain rather than in a single event.
Pipeline and integration engineering
Pipelines are production systems and fail like production systems: partially, intermittently, and at inconvenient times. Treating them as scripts is why silent failure is so common.
We design for idempotent reruns, explicit late and out-of-order data handling, backfill as a routine operation, and alerting on absence — because the failure that matters most is data that never arrived.
- Idempotent processing so a rerun is safe rather than a decision
- Late and out-of-order arrival handled by design, not by assumption
- Backfill treated as routine, with the cost of a full replay known
- Alerting on absence and freshness, not only on job failure
Data quality engineering
Data quality is a measurement discipline. Completeness, validity, uniqueness, freshness and consistency are testable properties, and the tests belong in the pipeline where they can stop bad data propagating.
The important design decision is what happens on failure: quarantine, alert and continue, or halt. That is a business decision about which is worse — late data or wrong data — and it differs per dataset.
- Quality dimensions tested in-pipeline rather than reported after the fact
- Failure behaviour agreed per dataset: quarantine, warn, or halt
- Reconciliation against source systems where correctness is contractual
- Quality made visible to consumers rather than known only to the data team
Data governance and lineage
Governance succeeds when it makes data easier to use correctly. It fails when it becomes an approval queue, at which point teams build private copies and governance loses visibility entirely.
We implement catalogue and lineage so consumers can find data and trace a figure to its source, classification so sensitivity is known, and ownership so every significant dataset has someone accountable.
- Catalogue and lineage that answer where a number came from
- Classification of sensitivity and retention obligation per dataset
- Named ownership, because unowned data degrades by default
- Access model based on classification rather than on individual request history
Data security and privacy
Data platforms concentrate sensitive information, which makes them disproportionately valuable targets and disproportionately consequential when access control is loose.
We design access on classification rather than on request history, minimise personal data in analytical environments, and handle residency and retention as architectural constraints. Broader control design is security architecture work.
- Access derived from classification and role rather than accumulated requests
- Masking, tokenisation or synthesis for personal data in non-production
- Retention and deletion implemented rather than documented as policy
- Residency constraints treated as architecture, not as configuration
Analytics and reporting foundations
Reporting disputes are usually definition disputes. A semantic layer that defines each metric once, with its calculation and its owner, removes most of them.
We define the metrics that actually drive decisions, establish where each is calculated, and make self-service viable by giving analysts curated products rather than raw tables and an invitation.
AI readiness assessment
An AI readiness assessment answers, per use case, whether the required data exists at sufficient volume and quality, whether it is labelled, whether there is a lawful basis for its use, and whether it can be served at the latency the use case needs.
It also asks whether the problem needs a model at all. A meaningful proportion of proposed use cases are better solved by better reporting or by a rule, and saying so early saves considerable expense.
- Data availability, volume and quality assessed per use case rather than in general
- Lawful basis and consent position established before development
- Latency and serving requirements stated, because they constrain the architecture
- An explicit judgement on whether a model is the right instrument at all
Machine learning platform foundations
Models that reach production need the same engineering discipline as other software, plus concerns of their own: reproducible training, versioned datasets and features, and monitoring for drift between the data a model was trained on and the data it now sees.
We design those foundations. We are explicit that we build the platform and the data foundation rather than presenting ourselves as a research organisation — capability we can evidence, described accurately.
- Reproducible training with versioned data and parameters
- Feature definitions shared between training and serving to prevent skew
- Drift monitoring on inputs and outputs, with an action threshold
- A defined path to retrain, evaluate and roll back a model
Practices we hold data platforms to
A short list separates data platforms that are trusted from those that are worked around. Every significant dataset has an owner. Metrics are defined once. Quality is tested in the pipeline rather than reported afterwards. Reruns are safe. Lineage answers questions without investigation. And cost is allocated, because unallocated platform cost grows without resistance.
None is novel. Holding to them while delivery pressure argues for a shortcut is the difficult part, which is why they are written down and enforced rather than assumed.
Platform cost and efficiency
Data platform cost is driven by storage class, retention, query patterns and how often pipelines reprocess data that has not changed.
We put allocation in place first so consumption has an owner, then address the architectural drivers — partitioning, incremental processing and retention tiering — which usually matter more than instance sizing.
What you receive
Design artefacts and working reference implementations, not a tooling recommendation alone.
- Data strategy covering domain ownership, mastering and retention obligations
- Target data architecture with layer responsibilities and modelling approach
- Pipeline design with idempotency, late-arrival handling and backfill strategy
- Data quality framework with tested dimensions and agreed failure behaviour
- Governance model: catalogue, lineage, classification and named ownership
- AI readiness assessment per candidate use case, with the blocking conditions named
- Cost allocation model and the architectural drivers of platform spend
Engagement models
Initial project design
Engaged before significant code exists, when structural decisions are still cheap to change. Produces the target-state architecture, the technology selection rationale, and a sequencing plan the delivery team can execute.
Mid-project assistance
Engaged when delivery has slowed, costs are rising, or a design that fitted the original problem no longer fits the current one. Diagnostic first: we establish what is actually causing the drag before recommending change.
Ongoing support
A named architect available as the system evolves, covering design review, technology decisions and course correction. Delivered under a subscription rather than a fixed project scope.
Enterprise delivery
How a data engagement runs.
Free first consultation
An overview of your data estate and objectives, and an honest view of whether the constraint is data, tooling or definition.
Assessment
Sources, flows, quality, ownership and current cost, established from the systems rather than from documentation.
Architecture and semantics
Target architecture, metric definitions and ownership model, with the alternatives considered recorded.
Reference implementation
One domain taken end to end — ingestion, quality, curation and consumption — so the design is proven rather than described.
Domain-by-domain rollout
Remaining domains onboarded in sequence, with reporting continuing throughout.
Handover
Runbooks, decision records and ownership assignments handed to the team that will operate the platform.
Technologies
Chosen against data volume, latency requirement and the team that will operate them.
Relational and analytical databases
PostgreSQL, MySQL and SQL Server for operational data; columnar stores where analytical scan patterns dominate.
Data warehousing and lakehouse
Layered storage with defined responsibilities, and table formats supporting schema evolution.
Batch and stream processing
Scheduled transformation where latency permits, and stream processing where it genuinely does not.
Kafka and event streaming
Change data capture and event ingestion, with ordering and delivery semantics stated per stream.
Orchestration
Dependency-aware scheduling with retry, backfill and alerting on absence rather than only on failure.
Transformation and modelling
Version-controlled, tested transformations with lineage derived from the code rather than maintained separately.
Catalogue and lineage tooling
Discovery, classification and the ability to trace a figure to its source.
Analytics and semantic layers
Metric definitions held once, so reports agree by construction.
ML platform components
Experiment tracking, dataset and feature versioning, model registry and drift monitoring.
Cloud data services
AWS, Azure and Google Cloud data services, selected against volume, latency and the team that will operate them.
Industries served
Sector determines the data sensitivity, retention obligation and lineage requirements the platform must satisfy.
Industry
Healthcare
Clinical data sensitivity, consent, and interoperability between record systems.
IndustryBanking & Financial Services
Regulatory reporting with lineage that must withstand audit.
IndustryInsurance
Claims and policy data spread across platforms of different generations.
IndustryRetail & E-commerce
Customer, catalogue and inventory data reconciled across channels.
IndustryManufacturing
Sensor and production telemetry at volume, joined to enterprise systems.
IndustryTelecommunications
Very high-volume event data with tight retention economics.
IndustryGovernment & Public Sector
Residency, transparency obligations and long retention duties.
IndustryEducation
Student records with strict access control and reporting obligations.
IndustryEnergy & Utilities
Time-series asset telemetry and regulated reporting.
IndustryLogistics
Tracking and telemetry where freshness has direct operational value.
Why NG&NR
Foundations before models
We will tell you when an AI use case is not currently supportable and what would have to change. That is more useful than a proof of concept that cannot be productionised.
Accurate about our own scope
We build data foundations and ML platform engineering. We do not present ourselves as a research organisation, because that is not the capability we can evidence.
Independent of platform vendors
We resell nothing and hold no commissions, so a recommendation to use less platform costs us nothing.
Named accountability
A named architect owns the technical outcome and remains the point of contact for the engagement.
Frequently asked questions
With a specific decision or process you want to improve, then an assessment of whether the data that use case requires exists, is usable and can be served at the needed latency. Starting from the model is how initiatives stall at the data step.
Per use case rather than in general. We assess data availability, volume and quality, whether labels exist, whether there is a lawful basis for use, and what latency serving requires. Readiness is not a property of an organisation; it is a property of a use case.
We build the data foundations and the platform engineering around models — reproducible training, versioned datasets and features, drift monitoring and a path to retrain. We do not present ourselves as a research organisation, because that is not the capability we can evidence.
Almost never. It is usually a definition problem: the same metric calculated differently in different places because semantics were never agreed and no source is authoritative. New tooling reproduces the disagreement faster.
The distinction matters less than the layering. What matters is that each layer has a defined responsibility, that consumers know which layer they should use, and that the modelling reflects real access patterns. We choose the implementation against volume, latency and the team that will operate it.
By minimising it. Analytical environments rarely need identifiable data, so we apply masking, tokenisation or synthesis, derive access from classification rather than accumulated requests, and implement retention and deletion rather than documenting them as policy.
Catalogue so data can be found, lineage so a figure can be traced, classification so sensitivity is known, and named ownership so datasets have someone accountable. Governance that becomes an approval queue fails, because teams build private copies and visibility is lost entirely.
By alerting on absence and freshness rather than only on job failure — the failure that matters most is data that never arrived, which a job-status check does not detect. Combined with in-pipeline quality tests and agreed failure behaviour per dataset.
Usually yes, and usually we recommend it. Platform migration is expensive and rarely the actual constraint. The gains are typically in semantics, quality, ownership and pipeline engineering, all of which are achievable on the platform you already run.
Allocation first, so consumption has an owner. Then the architectural drivers: partitioning, incremental processing rather than full reprocessing, retention tiering and query patterns. Those matter considerably more than instance sizing.
Where the constraint is the wider estate rather than the data platform, that is enterprise architecture. Where it is the cloud platform beneath it, that is cloud consulting. Data engagements frequently run alongside one or both.
Typically a fixed-scope assessment, then optional ongoing support through a subscription as domains are onboarded. Pricing is transparent and includes GST. The first consultation is free.
Next step
Request a data and AI readiness assessment
A free first consultation covering your data estate, the decisions you want it to support, and an honest view of which AI use cases it can currently sustain.
No cost, no obligation. Pricing is transparent and includes GST.