Capture

Establish the baseline. Keep up with change.

Tapstate reads the data that already exists and, where the connector supports it, continues as new source activity arrives.

Each source keeps its capture modes, preparation requirements, progress, and recovery boundaries explicit.

Browse Connectors Read Capture Modes

Source system

Existing records

Rows and documents already present

New source activity

Committed changes from the source

Tapstate Capture

MODE

Full load + CDC

POSITION

Recorded and retained

SCOPE

Selected tables

example configuration

connector config

connector: mysql
mode: cdc
tables:
  - orders

Capture Modes

Choose how each source begins and continues.

The selected mode defines what Tapstate reads first and whether the source continues after that initial read. Available modes depend on the connector.

MODE

WHAT IT READS

CONTINUES?

Full load

The selected data that exists when the run begins

Full load + CDC

Existing data, followed by supported inserts, updates, and deletes

Yes

Stream

Messages from a broker-defined starting position

Yes

API

Records exposed through a service API

Connector-specific

Note: Use only the modes declared by the connector. The existence of a database log does not by itself mean CDC is supported.

Continuous Capture

From initial copy to ongoing change.

A continuous source first establishes the selected baseline, retains new committed activity from the recorded position, catches up with those retained changes, and then continues from current source progress.

Start

Recorded source position

Baseline track

Initial copy

Read the selected rows or documents that already exist.

Baseline ready

Changes during copy

Retained new commits

Keep supported changes that occur while the initial copy is running.

Runs in parallel with the initial copy, from the recorded position onward.

Catch up

Process the retained changes after the baseline is ready.

Continue

Follow new committed activity from current source progress.

Continuity note: Continuity depends on connector capabilities and source retention. A saved position cannot be reused after the required log, offset, or change history has expired.

Declarative Source

Define the source before the pipeline runs.

A Tapstate source resource makes the connector, capture mode, connection settings, and selected data scope explicit in a reviewable .tapstate.yml file.

Validate the resource contract first, then verify connectivity, permissions, and data behavior in the deployment environment.

source/orders-source.tapstate.yml

version: tapstate/v1

kind: source

01id: orders-source

02connector: mysql

03mode: cdc

04config:

host: db.internal

database: orders

username: ${MYSQL_USER}

password: ${MYSQL_PASSWORD}

05tables:

- orders

- /customers_.*/

Field Annotations

Stable resource ID

The pipeline references this source by a stable name rather than repeating its connection settings.

Connector contract

The connector determines the recognized fields, available roles, declared capture modes, and external-system requirements.

Capture mode

mode: cdc represents Full load + CDC for this source. Only modes declared by the connector are valid.

Connection settings

Connector-specific endpoint and access settings stay explicit, while secrets remain outside the committed file.

Selected data scope

Use literal names for fixed objects or patterns for dynamic matching when the connector supports discovery.

Validate the contract. Then verify the environment.

Resource validation checks the file structure, references, connector, mode, and recognized field shapes. It does not confirm that the deployment environment is reachable or that source data behaves as expected.

Resource validation checks

Resource version and structure
IDs and references
Declared connector and mode
Recognized fields and value shapes

Deployment verification checks

Network access and authentication
Source permissions and preparation
Snapshot and CDC behavior
Stop, resume, and representative data changes

Connector Contracts

Each connector defines how its source is captured.

Tapstate gives every source the same reviewable Capture model. The connector still determines how changes are read, what the source must expose, how progress can be retained, and what may limit recovery.

Shared Tapstate Capture Contract

Declared Mode

The selected capture behavior is explicit.

Selected Scope

The source objects remain reviewable.

Recorded Progress

The data path retains the position needed to continue.

Validation and Diagnostics

Configuration and runtime state remain inspectable.

Source

Change Mechanism

Source Preparation

Continuity Boundary

MySQL

Binary log

Row-level change events

Replication access
ROW format
FULL row image

Binlog retention must cover the interruption window.

PostgreSQL

WAL

Connector-selected decoder

Replication access
Slot and publication
Update and delete identity

WAL and slot state must remain available.

Decoder, topology, and DDL behavior remain deployment-specific.

Oracle

Redo logs

LogMiner baseline

ARCHIVELOG
Supplemental logging
Mining access

Required archived redo must remain available.

RAC, long transactions, and recovery can introduce replay behavior.

Connector support is defined by the connector, database version, capture mode, topology, and known limitations—not by the database name alone.

Browse Connectors & Compatibility →

Source Continuity

Keep the recorded position inside the recovery window.

Tapstate records source progress so a data path can continue after an interruption.

Resume remains possible only while the required log, offset, or change history is still available for that connector.

Source recovery window

How long the recorded position remains usable

Binlog · WAL · Redo · Oplog · Broker history

PastCurrent source

Expired history

Available source history

Recorded position

Retention boundary

Position available

Recorded position

Retention boundary

Interruption

Resume attempt

Current source

The recorded position remains inside retained source history. The connector can attempt to continue according to its declared recovery behavior.

Position expired

Recorded position

Retention boundary

Required history expired

Interruption

Resume attempt

Current source

The source no longer retains the history required by the recorded position. Normal resume is no longer possible and the data path may need a connector-specific recovery path or a new baseline.

Plan and verify before production

Set retention

Cover the longest realistic interruption, maintenance window, and recovery time.

Observe progress

Track source lag, long transactions, cleanup activity, and whether the recorded position remains usable.

Test interruption

Stop and resume a staging data path with representative inserts, updates, and deletes.

A recorded position does not create universal recovery semantics. Resume, replay, and redelivery remain connector-specific.

Read Troubleshooting Guidance →

Production Readiness

Verify the source path before production.

A valid source resource is only the starting point. Production Capture also depends on network reachability, least-privilege access, source-side change prerequisites, retention, topology, and observed behavior under representative conditions.

Prepare the external system with its connector guide, then test the complete Capture path in staging before using production data.

The path that must work

Tapstate Runtime

Deployment identity
Runtime environment

Network Path

Routing
Allowlist
TLS

External Source System

Source Endpoint

Credentials
Object scope
Database topology

Source Change Path

Log · WAL · Redo · Oplog
Broker or API mechanism
Retention where applicable

Tapstate Runtime

Deployment identity
Runtime environment

Network Path

Routing
Allowlist
TLS

External Source System

Source Endpoint

Credentials
Object scope
Database topology

Source Change Path

Log · WAL · Redo · Oplog
Broker or API mechanism
Retention where applicable

Readiness Area

Prepare

Verify

Network and identity

Prepare

A reachable network path, appropriate TLS or allowlist controls, and a dedicated account scoped to the selected source objects.

Verify

Connect from the Tapstate runtime, authenticate with the intended account, and discover the selected source objects.

Change prerequisites

Prepare

The connector-declared change mechanism, required keys or row identity, and sufficient log, slot, archive, or broker retention.

Verify

Confirm that the source exposes the required change records and that retained history covers the planned interruption window.

Operating conditions

Prepare

Expected write activity, long transactions, cleanup behavior, replication lag, and planned topology or failover operations.

Verify

Observe source lag and resource use under representative activity, and test topology changes where the connector and deployment support them.

Staging evidence

Prepare

Representative schemas, keys, data types, nulls, large values, and source change patterns.

Verify

Exercise the initial read, inserts, updates, deletes, stop, resume, and source-to-destination comparison.

Include schema changes only where the connector declares support.

Resource validation checks the declared contract. Production use still requires evidence from the actual network, source system, and staging data path.

Open Connector Preparation Guides →

Capture Boundary

Separate the Capture contract from the source environment.

Tapstate makes the source definition, selected scope, and capture progress explicit.

The source environment still controls access, change-history availability, topology, retention, and vendor-specific behavior. Transform and Serve take over after Capture has produced the initial records and supported later changes.

01External Source Environment

02Tapstate Capture

03Next in the Data Path

01External Source Environment

02Tapstate Capture

03Next in the Data Path

Access Path

Network reachability
Credentials and permissions
Source endpoint availability

Change History

Log, WAL, redo, Oplog, stream, or API history
Retention and cleanup behavior

Source Semantics

Keys and row identity
Database version and topology
Vendor-specific transaction behavior

Capture Owns

Source Definition

Connector
Mode
Selected scope

Capture Progress

Initial-read state
Recorded source position
Recovery context

Contract Visibility

Resource validation
Runtime status
Diagnostics

Transform

Shape, combine, enrich, and route captured records.

Serve

Write or materialize current state for downstream consumers.

Access Path

Network reachability
Credentials and permissions
Source endpoint availability

Change History

Log, WAL, redo, Oplog, stream, or API history
Retention and cleanup behavior

Source Semantics

Keys and row identity
Database version and topology
Vendor-specific transaction behavior

Capture Owns

Source Definition

Connector
Mode
Selected scope

Capture Progress

Initial-read state
Recorded source position
Recovery context

Contract Visibility

Resource validation
Runtime status
Diagnostics

Transform

Shape, combine, enrich, and route captured records.

Serve

Write or materialize current state for downstream consumers.

Capture output

Initial records
Supported inserts, updates, and deletes

Consumed by

Transform and Serve

Capture makes the source contract and its progress explicit. It does not extend the source's retained history or replace the downstream responsibilities of Transform and Serve.

Review the full architecture →

Start with the source you need to capture.

Review its available modes, preparation steps, compatibility, and known limitations before defining the data path.

Browse Connectors Read Capture Modes