The short definition
Tapstate is an open-source unified operational data engine.
It captures committed changes from operational databases, transforms those changes incrementally, and maintains fresh, queryable state for applications, APIs, automation, and AI agents. Instead of asking teams to assemble and operate separate products for capture, messaging, processing, state maintenance, and serving, Tapstate treats that complete path as one product responsibility.
Three related terms describe different parts of this idea:
- Operational State is the continuously maintained representation of the current business reality needed by operational consumers.
- The Operational State Layer is the shared architectural capability that constructs, maintains, and serves that state between systems of record and operational consumers.
- Tapstate is the engine a team deploys to build and maintain the Layer.
The Layer is what the enterprise establishes. The engine is what the team deploys. The maintained Operational State is what applications and workflows consume.
The problem is not moving data. It is owning the complete path.
Most operational-data projects do not begin with a blank architecture diagram. They begin with a practical request:
- make current order state available to an application;
- expose account and entitlement data through an API;
- offload repeated reads from production systems;
- provide an automation process or AI agent with current business context.
Delivering that outcome commonly requires several separately managed components: a CDC connector, an event broker, a stream processor, a cache or serving database, and the monitoring, retry, recovery, and schema-handling logic that connects them. Each component may be effective on its own. The engineering burden appears in making them behave as one reliable source-to-serve system.
That assembled path creates more than additional infrastructure. It divides responsibility.
Capture may know which source positions have been read. A processing job may know which transformations have completed. A serving store may know which records are currently queryable. When something becomes stale, duplicated, incomplete, or inconsistent, the team must reconstruct what happened across several operating surfaces.
Tapstate exists because the desired result is not a collection of independently functioning components. It is one continuously maintained operational state that downstream systems can query and reuse.
Operational State, the Layer, and the engine
Operational systems each hold authoritative local facts.
An order system may know whether an order was accepted. Billing may know whether payment succeeded or a partial refund has already been issued. Logistics may know where a shipment is. Inventory may know whether a replacement item is still available. CRM and support systems may hold customer status and active cases.
A real operational decision often needs several of those facts together.
Operational State is the maintained representation of that combined situation. It is oriented around what a consuming application, workflow, employee, or agent needs to understand and act on now—not around the schema of any one source system.
Useful Operational State is:
- current rather than periodically reconstructed;
- incrementally maintained as sources change;
- consolidated where the business situation spans systems;
- shaped around consumer-oriented entities and workflows;
- queryable and reusable;
- explicit about concerns such as identity, ordering, completeness, provenance, and freshness.
The Operational State Layer turns the repeated work of constructing that state into a shared architectural capability. Tapstate is the product form that performs the work.
One engine, three continuous responsibilities
Tapstate organizes the complete path around three responsibilities:
Capture. Transform. Serve.
These are not three unrelated products or optional feature categories. They describe one continuous operating path from committed source change to maintained, queryable state.
Capture: establish state and follow change
Capture begins by establishing the initial usable state and then following committed changes as operational systems continue to evolve.
Depending on the source and connector, this responsibility may include:
- initial loading;
- continuous change capture;
- source-position and checkpoint management;
- preservation of useful ordering and transaction context;
- recovery after interruption;
- continuity between backfill and ongoing streaming;
- management of relevant source-schema changes;
- minimizing avoidable impact on production workloads.
Tapstate treats Connector engineering as a source-specific knowledge problem, not simply an adapter-development task.
Database transaction logs were built primarily for database recovery and replication—not as universal integration interfaces. Their meaning varies across database engines, versions, configurations, workloads, and unusual transaction behavior. A mature connector therefore represents accumulated knowledge encoded into implementation, tests, operating guidance, and failure handling.
Tapstate builds on engineering lessons, tests, and issue knowledge accumulated through Tapdata’s production history. That continuity matters, but it does not remove the need to validate each current connector against its supported source versions, topologies, schema behavior, and recovery conditions. Historical knowledge informs the implementation; it is not a substitute for current evidence.
Schema change belongs inside this responsibility as well. A long-running operational path must expect source schemas to evolve while data is moving. The goal is to make relevant schema changes visible and manageable within the path rather than allowing them to become silent downstream drift. Exact handling depends on the source, change type, connector, transformation, and serving boundary.
Transform: reshape change into useful state
Raw source changes are rarely the final form an operational consumer needs.
A source may emit:
- an order-row update;
- a new payment record;
- a shipment-status change;
- a revised customer tier;
- a newly opened support case.
Transform incrementally converts those source-oriented changes into consumer-oriented state.
Within the Tapstate path, transformation may include:
- filtering and routing;
- normalization;
- enrichment;
- cross-system joins;
- denormalization;
- field and entity reshaping;
- maintaining derived state as new changes arrive.
The important distinction is that transformation does not have to wait for a separate batch process after data lands. It occurs as part of the same continuous operating path used to capture and maintain the state.
Tapstate is not intended to replace every general-purpose stream-processing workload. Highly customized event-time processing, complex timer-based computation, or a broad event-distribution topology may still belong in systems designed specifically for those purposes.
Tapstate focuses transformation on the work required to turn changing operational records into state that applications and workflows can use directly.
Serve: keep state current, queryable, and reusable
Serving is not simply publishing another stream of changes.
Tapstate maintains the resulting operational entities so they can be queried repeatedly without every consumer rebuilding the same source access, joins, transformations, and serving model.
The current product path emphasizes MongoDB-compatible access to application-ready state. Standard consumers can work with maintained entities such as current accounts, orders, inventory positions, customer situations, or entitlements rather than raw source events.
That state can support:
- operational applications;
- internal and external APIs;
- workflow automation;
- employee-facing tools;
- AI agents;
- selected downstream event or analytical paths.
The architectural principle is simple:
Build the operational state once, then reuse it across the consumers that need the same current business context.
From one source update to reusable operational state
Consider an order that changes from pending to paid.
At the source, that may be only one committed row update. But the consumer-facing order situation may also require:
- the associated payment status;
- current shipment progress;
- available replacement inventory;
- customer tier;
- any active refund;
- an open support case.
Tapstate captures the relevant source changes as they occur. Transform incrementally combines and reshapes them into a current order entity. Serve maintains that entity in a queryable form.
An application can then retrieve the current order situation without querying every production system individually. An API can expose the same maintained entity. An automation workflow can act on it. An agent can evaluate the order, payment, stock, shipment, and customer context together before taking an action.
The value is not merely that the data arrived quickly. It is that the complete operating path is responsible for turning several source changes into a maintained state that consumers can understand and reuse.
Reliability is part of the product path
For operational data infrastructure, production quality cannot be reduced to an assertion that the software has no bugs.
The more useful standard is whether the system can continuously:
- observe progress;
- detect discrepancies;
- validate recent changes;
- identify where an incorrect value entered the path;
- recover from interruption;
- remediate understood problems where safe;
- build operator confidence in the state being served.
Northstar describes this as a second control loop alongside replication: changes move through the system, while the system also validates the resulting state and looks for inconsistencies.
Tapstate’s architecture is designed to keep visibility, checkpointing, validation, and recovery connected across Capture, Transform, and Serve instead of treating them as independent concerns owned by unrelated systems. Public architecture messaging similarly places consistency, visibility, and recovery inside the Layer’s operating model.
The intended direction includes record- or event-level lineage that helps operators trace how a source change became served state, together with incremental validation and remediation where the discrepancy is understood and safely recoverable.
This should not be interpreted as a promise of universal self-healing.
Not every conflict has one automatically correct resolution. Not every source discrepancy can be repaired safely. The supported remediation classes and escalation boundaries remain part of the evolving product design.
The stable responsibility is broader:
Reliability should be continuously observed and engineered across the product path, not assumed because each component appears healthy in isolation.
How Tapstate maintains and exposes state
Different forms of state have different operational purposes.
Tapstate’s architecture is being designed around distinct roles for active processing state, application-ready serving state, and potential archival state.
Active processing state
Some state exists to support incremental computation, coordination, checkpoints, and low-latency processing while changes move through the system.
The current architecture direction includes distributed in-memory state for workloads where very low-latency active state is required. The precise implementation and product boundary may evolve as the runtime matures.
Application-ready serving state
Warm serving state is the maintained, queryable representation operational consumers use directly.
MongoDB is the current primary serving-store direction, exposed through MongoDB-compatible access patterns so applications can query operational entities through a familiar model rather than a proprietary consumer SDK.
Archive and time-oriented state
Some regulated, auditable, or agent-accountability scenarios may require more than the current state. They may need to reconstruct what information was available at an earlier decision point.
Northstar records an archive or cold-state direction, with Paimon currently identified as a tentative candidate. The exact cold-tier design remains unresolved because audit and time-travel value must be balanced against the product goal of a smaller operational footprint.
This is an evolving architecture direction, not a permanent storage contract.
The stable product outcome remains current, queryable Operational State. Historical reconstruction and long-term archival are additional responsibilities being evaluated where they create enough operational value to justify their complexity.
Why agents change the serving contract
Tapstate is not an AI platform and does not build agents.
Its role is to provide applications and agents with current operational context that is coherent enough to support action. An agent calling five source-system tools still has to determine whether the records describe the same entity, whether the sources are equally current, whether an action has already occurred, and which source owns each fact. Tool access alone does not resolve those questions.
Tapstate’s Agent-native direction includes exposing capabilities through the Model Context Protocol on both sides of the product:
- the control plane, where agents may help build or operate data paths;
- the data plane, where agents query maintained operational state.
MCP is a confirmed product direction in Northstar. Several related ideas are still evolving:
- freshness as an explicit contract;
- entity-centric query shapes;
- self-describing semantics;
- decision-time snapshots showing what state an agent saw when it acted.
These ideas point toward a stronger serving contract than simply returning a record.
An operational consumer may need to know:
- how current the state is;
- which systems contributed to it;
- whether important context is missing;
- what entity the data represents;
- what the state looked like when a previous decision was made.
Not all of these capabilities should be treated as universally available today. They represent the direction in which operational serving needs to evolve as software moves from reading data to acting on it.
A focused operational engine, not a replacement for the data estate
Understanding Tapstate also requires understanding its boundaries.
It includes CDC, but it is not merely CDC
CDC acquires source changes. Tapstate also transforms those changes, maintains current entities, and serves the resulting state.
It is not a warehouse replacement
Warehouses and lakehouses preserve history and support analytical workloads. Tapstate focuses on present operational context for action-oriented consumers. The two can coexist.
It does not replace every Kafka or Flink workload
Kafka remains well suited to broad event distribution and strategic event-backbone architectures. General-purpose stream processors remain suitable for complex event-time computation and highly customized topologies.
Tapstate is more opinionated: it focuses on the continuous path required to maintain and serve Operational State.
It is not an AI platform
Tapstate supplies current operational context to agents. It does not provide the agent runtime, reasoning model, orchestration framework, or business decision policy.
It does not replace systems of record
Source systems remain authoritative for local transactions and business rules. Tapstate derives and maintains operational state from committed source changes; it does not take ownership of the original transactions.
When Tapstate is useful—and when it is unnecessary
Tapstate is most relevant when several of the following conditions are present:
- the required state spans multiple operational systems;
- consumers need continuously updated context;
- direct querying of production systems is undesirable or restricted;
- several applications or teams need the same maintained entities;
- teams are repeatedly rebuilding similar source-to-serve paths;
- Capture, transformation, state maintenance, serving, visibility, and recovery need to operate as one lifecycle;
- stale or fragmented state can lead to an incorrect operational action.
Tapstate is not required for every data problem.
A simpler solution may be sufficient when:
- one source-system API already provides the complete required state;
- a database query or View satisfies the workload;
- the requirement is only to replicate changes into an existing destination;
- a mature CDC connector already solves the complete problem;
- a streaming database already owns the necessary ingestion, transformation, state, and serving responsibilities;
- batch freshness is acceptable;
- only one consumer needs the result and there is no meaningful reuse pressure.
Introducing a new architectural layer without a corresponding responsibility to own creates complexity rather than removing it.
The relevant question is not whether every enterprise should deploy Tapstate. It is whether the organization is already carrying the operational responsibility Tapstate is designed to consolidate.
What is established—and what is still evolving
Tapstate’s stable product identity is clear:
- it is an open-source unified operational data engine;
- it establishes and maintains an Operational State Layer;
- Capture, Transform, and Serve are continuous product responsibilities;
- its outcome is current, queryable, reusable Operational State;
- applications, APIs, automation, and agents are primary consumers;
- source-specific Connector engineering and operational continuity are core parts of the product thesis.
The current architecture direction also includes:
- MongoDB-compatible application serving;
- distributed active state;
- incremental validation;
- end-to-end lineage and observability;
- MCP exposure for control- and data-plane use.
Several boundaries are still evolving:
- the final cold and archival tier;
- which discrepancy classes can be remediated automatically;
- the exact vocabulary and guarantees behind freshness contracts;
- decision-time state reconstruction;
- the complete Agent-native surface;
- final support and maturity levels for individual Connectors;
- the precise boundary between Tapstate transformation and general-purpose stream processing.
Open design work does not change the core product definition. It clarifies how broadly and deeply the engine will fulfill the responsibilities it has chosen to own.
Conclusion
Tapstate is not another name for a data pipeline.
It is an open-source unified operational data engine responsible for turning committed operational change into maintained, queryable state.
The enterprise establishes an Operational State Layer. A team deploys Tapstate to build and maintain it. Applications, APIs, automation, and agents consume the resulting Operational State.
Capture acquires change. Transform converts it into useful entities. Serve keeps those entities current and reusable. Validation, visibility, recovery, and operational knowledge make the complete path dependable enough to support action.
That complete lifecycle—not any one Connector, transformation, or database—is the product.
