Why We Built Tapstate: From Moving Data to Maintaining Operational State

Tapstate did not begin with a new category name.

It began with years of working on a more practical problem: how to move changing data safely out of the operational systems that enterprises depend on.

That work led us deep into database transaction logs, CDC connectors, heterogeneous schemas, production incidents, recovery procedures, and the long tail of behaviors that appear only after infrastructure has been running in real customer environments.

Over time, we reached a conclusion that changed how we thought about the product:

Moving operational data is only the beginning. The harder responsibility is maintaining a current state that applications and agents can safely act on.

That conclusion is why we built tapstate.

tapstate is a unified operational data engine that captures committed database changes, transforms them incrementally, and serves live, queryable operational state to applications, automation, and AI agents. Its purpose is to establish and maintain an Operational State Layer between fragmented systems of record and the operational consumers that need to understand what is true now.

This article explains how years of production experience led us from data movement to Operational State—and why that shift required more than renaming an existing product.

We had already spent years moving enterprise data

Enterprise data integration often looks straightforward from a distance.

A source system contains data. Another system needs it. A pipeline connects the two.

In production, the work is rarely that simple.

Operational databases differ in how they expose transaction logs, identify committed changes, represent data types, handle long transactions, and behave across versions and configurations. Even where a database provides a documented log-reading mechanism, documentation alone rarely describes every condition an external CDC connector will encounter.

A connector must learn from:

• database internals;
• vendor-specific behavior;
• unusual transaction patterns;
• legacy application designs;
• schema changes;
• recovery scenarios;
• production incidents;
• customer-specific configurations.

That knowledge accumulates over time.

Northstar describes a mature connector not merely as old code, but as operational knowledge encoded in software, tests, procedures, and failure handling. Production behavior—not the happy path—is the real specification.

This experience shaped tapstate from the beginning.

But it also revealed that reliable Capture, while essential, did not finish the job.

The destination was not the outcome

Many integration projects are described by their endpoints:

• Oracle → MongoDB
• Database → Kafka
• Core system → Cloud
• Source A → Destination B

Those descriptions are useful for defining movement, but they do not necessarily describe the business outcome.

A record arriving at a destination does not mean an application now has the state it needs.

The destination may still contain:

• source-oriented schemas;
• separate records for the same business entity;
• inconsistent identifiers;
• raw change events rather than current state;
• fields that require enrichment from another system;
• data that is difficult for applications to query directly;
• no shared model for freshness, completeness, or provenance.

Teams then add another layer of work.

They normalize schemas, correlate identifiers, join streams, denormalize records, maintain current views, build serving APIs, add caches, and create monitoring and recovery logic around the entire path.

The real architecture often becomes:

Database logs → CDC → Messaging → Stream processing → State maintenance → Serving store → API or application

Each component may work correctly on its own. But the team still owns the assembly project: making several separately operated systems behave like one dependable source-to-serve path.

We came to believe that the desired outcome was not replicated records.

It was maintained, usable Operational State.

Production changed how we ranked product value

Infrastructure products are often evaluated through visible quantities:

• number of connectors;
• number of destinations;
• feature breadth;
• benchmark throughput;
• supported transformation nodes;
• configuration flexibility.

Those measures can matter. But prolonged production operation changes the order of importance.

Customers living with operational data infrastructure every day are eventually asking more fundamental questions:

• Is the state correct?
• Can we see where it came from?
• Can we detect when something has gone wrong?
• Can we recover without rebuilding everything manually?
• Can the system remain understandable months after deployment?
• Can operators trust it when business-critical applications depend on it?

Northstar records the value hierarchy behind the tapstate architecture as:

1. data consistency;
2. observability;
3. peace-of-mind maintenance after production.

That hierarchy changed what we believed the product should optimize for.

Production quality could no longer mean only that a pipeline had not reported an error.

A pipeline can remain technically “running” while:

• a source position stops advancing;
• one record is skipped;
• a transformation produces an incorrect entity;
• a schema change alters downstream meaning;
• a serving record becomes stale;
• different stages disagree about progress.

For operational infrastructure, confidence has to be built continuously.

Northstar defines production quality as the ability to detect, validate, recover, and build confidence in data correctness. It describes validation and observability as a second control loop alongside replication itself.

That principle became central to tapstate.

Scope became a product decision

One way to grow a data product is to keep adding scope.

More sources. More destinations. More workloads. More migration patterns. More transformation types. More exceptions for individual projects.

That breadth can be commercially useful, but it also creates a cost.

Every new target and adjacent workload introduces:

• new semantics;
• more compatibility combinations;
• additional failure modes;
• broader testing requirements;
• more operational surfaces;
• less clarity about what the product is fundamentally responsible for.

The predecessor product, Tapdata, addressed a broad range of real-time integration and operational-data requirements. That experience created substantial engineering knowledge, but it also taught us that product breadth and product responsibility are not the same thing.

tapstate is deliberately more focused.

The current architecture keeps source-side Capture strategically important, particularly the ability to understand difficult enterprise databases. At the same time, it narrows the destination and workload scope instead of attempting to remain a universal replication and migration platform. Northstar explicitly records database migration projects, broad big-data replication, and long-tail destinations as areas tapstate does not intend to pursue as core product scope.

This is not a reduction in ambition.

It is a decision to own one outcome more completely:

From committed source change to maintained, queryable Operational State.

Focus means saying no to work that does not strengthen that responsibility.

From moving changes to maintaining state

Operational State is the continuously maintained representation of the current business reality needed by an application, employee, automation process, or agent.

It does not necessarily exist in one database.

A current order situation may depend on:

• the order-management system;
• the payment processor;
• inventory;
• fulfillment;
• shipping;
• CRM;
• support.

A current customer situation may span commerce, loyalty, billing, service, and account systems.

Each source contains locally authoritative facts. The consuming workflow often requires a logical state assembled across several of them.

That state must be more than current.

It also needs to be:

• continuously updated;
• consolidated where necessary;
• shaped around the consuming workflow;
• explicit about identity and relationships;
• queryable;
• reusable;
• observable enough to trust.

Moving records does not automatically create this result. Operational State may require Capture, synchronization, identity resolution, ordering, transformation, joins, enrichment, incremental maintenance, and serving.

This is why tapstate is organized around three continuous responsibilities:

Capture. Transform. Serve.

Capture follows committed changes from source systems.

Transform reshapes those changes into consumer-oriented entities.

Serve maintains the resulting state in a form that applications and agents can query and reuse.

These are not separate product stories. Together, they describe the responsibility tapstate has chosen to own.

Why the Operational State Layer became the center

The concept of an Operational State Layer gave us a clearer way to describe the architectural outcome.

The Layer sits between systems of record and the consumers that need current operational context.

Systems of Record
Authoritative local transactions and business rules

↓

Operational State Layer
Continuously constructs, maintains, and serves current operational state

↓

Applications · APIs · Automation · Agents
Operational consumers that read and act on current state

The Layer does not replace the systems of record. Those systems remain authoritative for their local transactions and business rules.

It does not replace the analytical estate. Warehouses and lakehouses remain valuable for historical storage, reporting, aggregation, and analysis.

It also does not make every event-processing or messaging system unnecessary. Kafka, stream processors, databases, and analytical platforms can continue to play important roles.

The Layer has a more specific purpose:

Continuously construct, maintain, and serve the current operational reality that downstream systems need to act.

The distinction between the Layer and tapstate is important.

The Operational State Layer is the architectural capability an enterprise establishes.

The unified operational data engine is the product a team deploys to build and maintain it.

The Operational State is the maintained business state that consumers use.

This hierarchy gave the product a sharper identity than “another data platform.”

Why applications and agents make the problem more urgent

Operational State was necessary before the recent rise of AI agents.

Customer applications, digital services, internal workflows, APIs, and automation have long needed current information from systems they could not safely or efficiently query directly.

Agents amplify the need because they shorten the distance between data and action.

A person reviewing incomplete information may notice something is missing. They may open another system, ask a colleague, delay the decision, or apply judgment.

An agent may act immediately.

It may:

• approve a refund;
• update an account;
• route a case;
• reserve inventory;
• change an entitlement;
• personalize an offer;
• initiate a workflow.

When the context is stale or fragmented, the result is no longer merely an inaccurate report. It may be an incorrect operational action.

Northstar frames AI as an urgency multiplier and a new consumer class—not as a reason to redefine tapstate as an AI platform. tapstate does not build agents; it supplies the current enterprise context applications and agents need to act correctly.

This distinction matters.

Connecting an agent to several source-system tools gives it access. It does not automatically give it a coherent understanding of:

• which records describe the same entity;
• which facts are current;
• whether an action already occurred;
• which source owns a value;
• whether important context is missing.

Operational State addresses the context problem before the decision is made.

Why a rewrite, not a rename

tapstate is not a new label placed on the previous product.

Northstar describes it as a complete rewrite shaped by three lessons:

• product focus had been diluted by expanding scope;
• customer value needed to be reordered around consistency, observability, and maintainability;
• applications and agentic consumers were increasing the importance of current operational state.

A rewrite creates the opportunity to reconsider the product from first principles:

• What responsibility should belong inside one engine?
• Which workloads strengthen the core product?
• Which workloads should remain outside it?
• What must be observable across the entire path?
• What knowledge should survive even when implementation changes?
• How should state be served to operational consumers?

It also creates an obligation to be precise about evidence.

Tapdata’s production history informs tapstate. It contributes engineering lessons, tests, issue knowledge, and experience with database-specific behavior.

But previous Tapdata deployments should not be presented as proof that every rewritten tapstate subsystem or connector has already reached the same level of availability or maturity.

Northstar explicitly distinguishes the predecessor product from tapstate and identifies the external continuity story as something that must be handled carefully.

The continuity is in the knowledge—not in an assumption that old and new implementations are identical.

What we chose to carry forward

A rewrite does not mean beginning without history.

The most important asset accumulated through years of connector and production work is not a specific codebase. It is the body of knowledge surrounding real database behavior.

That knowledge exists in:

• connector tests;
• bug histories;
• customer issues;
• recovery scenarios;
• compatibility findings;
• operational procedures;
• edge cases that were not documented by vendors;
• lessons from systems running under real workloads.

Northstar describes this principle directly: implementation can be replaced, while the knowledge encoded in tests and issue history is the durable asset.

tapstate carries that knowledge forward while rebuilding the product around a more disciplined boundary.

We are not trying to preserve every historical accommodation.

We are trying to preserve what helps the new engine perform its chosen responsibility more reliably.

What we are choosing to build

tapstate is being built around several durable product commitments.

Own the path, not only one stage

Capture, Transform, and Serve must operate as parts of one continuous responsibility.

The product should not stop at emitting changes and leave every consumer to construct, maintain, and serve state independently.

Treat transformation as part of the operational path

Operational consumers rarely need raw source schemas.

Filtering, normalization, joins, enrichment, and entity shaping should happen incrementally as part of maintaining state, rather than only through a separate batch process after data lands.

Make reliability visible

Checkpointing, validation, lineage, recovery, and observability should help operators understand the state being served—not only whether individual processes remain alive.

Preserve source-specific engineering depth

Enterprise Capture cannot be reduced to a connector-count claim.

Each source represents a body of database-specific behavior that must be tested and matured across versions and production conditions.

Serve state for reuse

Applications, APIs, workflows, automation, and agents should be able to consume maintained entities without repeatedly rebuilding the same consolidation logic.

Remain focused

tapstate is not intended to become:

• a generic AI platform;
• a replacement for every Kafka or Flink workload;
• a historical warehouse;
• a universal database-migration product;
• a substitute for systems of record.

Those boundaries are part of the product—not limitations to hide.

Some parts are established. Others are still evolving.

The stable product identity is clear:

• tapstate is a unified operational data engine;
• it builds and maintains an Operational State Layer;
• Capture, Transform, and Serve are one product path;
• the result is live, queryable Operational State;
• operational applications and agents are primary consumers;
• consistency, observability, and maintainability are central product values.

Some implementation boundaries continue to evolve.

Northstar still records open work around:

• the exact archival and time-travel design;
• supported self-remediation boundaries;
• freshness contracts;
• decision-time state reconstruction;
• the full agent-facing surface;
• per-connector rewrite status and maturity;
• the final boundary between tapstate transformation and general-purpose stream processing.

We believe it is important to say this plainly.

A company story should not turn product direction into a claim of universal current availability.

Clarity about what is established and what remains under development is part of building trust—especially for infrastructure intended to support operational decisions.

A narrower scope and a larger responsibility

We built tapstate because we no longer believed that “moving data” adequately described the problem customers were trying to solve.

The difficult part begins after the first change is captured.

The system must understand source behavior, preserve continuity, transform records into useful entities, maintain current state, expose it to consumers, and help operators establish confidence that the state is correct enough to act on.

That is a larger responsibility than a connector or pipeline.

It also demands a narrower product boundary.

tapstate is the result of that trade-off:

• less interest in being every kind of data platform;
• more commitment to owning the source-to-serve operational path;
• less emphasis on breadth for its own sake;
• more emphasis on consistency, visibility, and life after production;
• less focus on where records are copied;
• more focus on the state applications and agents ultimately need.

tapstate is the product of a narrower scope and a larger responsibility.

That is why we built it.