Changelog

All notable changes to ElasticRelay are documented here.

v1.4.4 - March 11, 2026

🔧 PostgreSQL Snapshot-to-CDC Handoff & CDC Stability Fixes

This release fixes three classes of PostgreSQL synchronization issues: snapshot-to-CDC handoff, sustained CDC catch-up stability, and replication slot management.

🐛 Bug Fixes

1. PostgreSQL CDC Start Checkpoint Reuse

• Fixed PostgreSQL CDC starting from nil checkpoint after snapshot
• Reused snapshot checkpoint for PostgreSQL jobs when switching to CDC mode

2. PostgreSQL Snapshot Checkpoint Field Mapping

• Fixed PostgreSQL snapshot events being written into MySQL checkpoint fields
• Added connector-type specific checkpoint mapping for correct LSN fields

3. Snapshot Chunk Consistency LSN Propagation

• Fixed snapshot chunks using drifting WAL positions instead of one consistency point
• Captured one consistencyLSN and attached the same value to every emitted chunk

4. PostgreSQL CDC Async Batch Decoupling

• Fixed replication reader being blocked by downstream batch processing
• Added asynchronous event queue and dedicated batch worker for MultiJob

5. PostgreSQL CDC Event LSN Propagation

• Fixed CDC events being emitted without real WAL checkpoint position
• Tracked current WAL LSN and wrote it into CDC event checkpoints

6. PostgreSQL Job-Scoped Replication Slot Reuse

• Fixed snapshot and CDC running against different replication slots
• Changed to stable jobId-scoped slot naming with reuse during CDC startup

7. Replication Slot Pre-Creation Before Snapshot

• Pre-created publication and job-scoped slot during BeginSnapshot
• Snapshot and CDC now share the same slot context

8. PostgreSQL Logical Message Reassembly

• Fixed CDC dropping rows when pgoutput messages arrived fragmented or packed
• Added buffered logical message reassembly for complete payload parsing

9. PostgreSQL CDC Primary Key Extraction Fix

• Fixed CDC events using wrong or unstable primary keys
• Loaded actual PostgreSQL primary key columns from schema metadata

10. PostgreSQL Replication Connection Concurrency Fix

• Fixed protocol desynchronization during CDC streaming
• Refactored WAL processing loop into single-threaded receive/send model

11. Force-Initial-Sync Cleanup and CDC Restart Safety

• Added automatic cleanup of inactive PostgreSQL replication slots for force_initial_sync
• Cleared in-memory CDC batch/queue state before restarting the CDC loop

12. Empty PostgreSQL Table Snapshot Guard

• Skipped chunk creation for tables with zero estimated rows

13. Checkpoint Commit Noise Reduction

• Removed unused checkpoint DB pool creation
• Throttled checkpoint persistence to at most once per commit interval

Download v1.4.4 | GitHub Release

v1.4.3 - March 10, 2026

🔧 Snapshot Primary Key Extraction Fix

🐛 Bug Fixes

1. Multi-Source Snapshot PK Resolution

• Fixed snapshot path using hardcoded primary key field names (_id and id only)
• Added automatic primary key discovery for MySQL, PostgreSQL, and MongoDB snapshot sources
• Tables using info_id, user_code, order_no, or composite keys now generate correct Elasticsearch document IDs

2. Parallel Snapshot PK Propagation

• Fixed parallel snapshot workers still extracting document IDs from data["id"]
• Propagated discovered primary key column through TableTask and ChunkTask

3. Non-Auto-Increment PK Snapshot Fallback

• Added full-table-scan fallback chunk for tables with non-numeric primary keys
• Maintains correct sync for string-based primary keys while using optimized chunking for auto-increment

Download v1.4.3 | GitHub Release

v1.4.2 - January 30, 2026

🔧 Transform Rule Matching Fix

🐛 Bug Fixes

1. Source ID Matching Fix

• Fixed _source_id not being passed to Transform Engine
• Embedded _source_id field directly into event data JSON, bypassing unreliable gRPC Checkpoint transmission

2. Transform Engine Source ID Extraction

• Added extractSourceID() method to extract _source_id from event data JSON with fallback to Checkpoint.SourceType

3. ES Sink Cleanup

• Added _source_id to metadata cleanup list before Elasticsearch storage

Download v1.4.2 | GitHub Release

v1.4.1 - January 30, 2026

🔧 Transform Engine Integration & Data Masking Fixes

🐛 Bug Fixes

1. Transform Service Integration

• Fixed transformEvents method being a pass-through placeholder — now calls Transform Service via ApplyRules gRPC stream
• Fixed missing _table field in snapshot events preventing rule matching
• Fixed missing SourceType in Checkpoint for source_id matching

2. Type Converter Enhancement

• Added object type support for JSON fields (MySQL JSON columns)

3. Data Masking Engine

• Fixed large numeric values (phone numbers, ID cards) being converted to scientific notation
• Added valueToString() function for proper numeric formatting

4. MySQL Connector

• Fixed long numeric strings (phone, ID card, bank card) being incorrectly parsed as int64
• Only convert strings to numbers if length ≤ 10 digits

Download v1.4.1 | GitHub Release

v1.4.0 - January 17, 2026

🎉 Major Release: Transform Engine Complete Implementation

This release marks a major milestone in Phase 3 development — the complete implementation of the Transform Engine, providing enterprise-grade data transformation capabilities.

🚀 New Features

1. Transform Engine Core

• Rule matching with table pattern wildcards and priority-based processing
• Multi-rule support with statistics tracking
• Pass-through mode when no rules configured

2. Field Mapper

• Field rename, copy, move operations
• Nested path support using dot notation (user.profile.name)

3. Type Converter

• 10+ type conversions: string, int, float, bool, date, timestamp, keyword, text, object

4. Filter Engine

• 10 operators: eq, ne, gt, gte, lt, lte, in, nin, regex, exists
• Actions: include, exclude, route

5. Data Masking Engine

• 4 strategies: mask, hash, token, regex
• 5 preset templates: phone, ID card, email, bank card, name

6. Expression Engine

• 16 built-in functions: string (concat, substr, upper, lower, trim, replace, length), math (round, abs, floor, ceil, min, max), date (now, formatDate, parseDate), conditional (ifNull, ifEmpty, coalesce)
• Ternary expressions and field arithmetic

7. Configuration Loading

• New -transform-config command-line parameter
• Integration with start.sh startup script

⚡ Performance

🧪 Testing

• 38 unit test cases covering all components
• Benchmark tests for performance validation

✅ Phase 3 Milestone

Download v1.4.0 | GitHub Release

v1.3.1 - December 15, 2025

🐛 Bug Fixes

1. MongoDB CDC Index Routing Fix

• Fixed MongoDB CDC events incorrectly routing to elasticrelay_mongo-default index
• Updated extractTableName() to check both _table (MySQL/PostgreSQL) and _collection (MongoDB) fields
• cleanDataForES() now properly removes _collection, _database, and _id metadata fields

2. MongoDB Authentication Fix

• Added authSource=admin to MongoDB connection URI for proper authentication

3. MongoDB Snapshot Primary Key Extraction

• Fixed beginStandardSnapshot() to use _table field for consistency with ES sink

4. Primary Key Detection Enhancement

• Updated processSnapshotChunk() to check for _id field first, then fall back to id

🔧 Improvements

1. Elasticsearch Timeout Configuration

• Increased timeout from 3s to 30s for ensureIndexExists() and createDefaultIndex() operations
• Improved reliability for remote Elasticsearch servers with higher latency

2. Docker Compose MongoDB Replica Set

• Enhanced MongoDB container with keyFile authentication for replica set
• Improved replica set status detection and conditional initialization

3. Documentation Updates

• Updated README.md with MongoDB setup instructions
• Added MongoDB-specific scripts reference

4. Code Formatting

• Fixed indentation issues in cmd/elasticrelay/main.go
• Added MongoDB connector case to connector server creation switch statement

📁 Files Changed

• internal/sink/es/es.go - MongoDB collection name support in ES sink
• internal/connectors/mongodb/mongodb.go - Auth source fix and metadata field consistency
• internal/orchestrator/multi_orchestrator.go - MongoDB connector integration and _id support
• cmd/elasticrelay/main.go - MongoDB connector server creation
• docker-compose.yml - MongoDB replica set keyFile configuration
• README.md - MongoDB setup documentation

Download v1.3.1 | GitHub Release

v1.3.0 - December 7, 2025

🎉 Major Release: MongoDB Connector Complete Implementation

This release marks the completion of MongoDB Connector development, achieving 100% coverage of the three major database sources (MySQL, PostgreSQL, MongoDB) for the ElasticRelay CDC platform.

🚀 New Features

1. MongoDB Change Streams CDC Implementation

• Full implementation of MongoDB Change Streams for real-time CDC
• Automatic detection of Standalone, ReplicaSet, and Sharded Cluster deployments
• Complete resume token encoding/decoding for checkpoint persistence
• Support for INSERT, UPDATE, REPLACE, and DELETE operations
• Configurable options for flexible configuration

2. BSON Type Converter System

Complete BSON to JSON-friendly type conversion supporting:

• Basic Types: ObjectID → string (hex), DateTime → RFC3339, Timestamp → map
• Binary Types: Binary → base64 encoded map with subtype
• Numeric Types: Decimal128 → string (precision preserved), int32 → int64 normalization
• Special Types: Regex → map, JavaScript → string, CodeWithScope → map
• MongoDB-Specific: MinKey, MaxKey, DBPointer, Symbol, Undefined, Null
• Nested Structures: Recursive document and array conversion
• Document Flattening: Configurable max depth for Elasticsearch compatibility

3. Sharded Cluster Support

• Dedicated connector for sharded cluster monitoring via mongos
• Multi-shard watching for aggregated change events across shards
• Migration awareness for consistency during chunk migrations
• Chunk distribution monitoring for data distribution across shards

4. Parallel Snapshot Manager Integration

• MongoDBSnapshotAdapter implementing parallel snapshot interface
• Collection info retrieval with document count and field schema detection
• Multiple chunking strategies:
  • ObjectID-based chunking for standard collections
  • Numeric ID-based chunking for integer primary keys
  • Skip/Limit fallback for complex ID types
• Parallel processing for coordinated parallel snapshots

5. Checkpoint Manager Enhancement

• Job-specific checkpoint with resume token, cluster time, and event count
• Thread-safe operations with mutex protection
• Event counting for monitoring
• JSON file-based checkpoint persistence

🧪 Testing

Unit Tests

• Complete test coverage for BSON type conversion
• URI construction and pipeline building tests
• Operation type mapping and structure validation
• Full CRUD, concurrency, and persistence tests
• Benchmark tests for performance validation

Integration Tests

• Basic Change Stream functionality
• Resume token persistence and recovery
• UPDATE and DELETE operation handling
• Real MongoDB data type conversion
• Database-wide change monitoring
• Full connector integration

📊 Performance Characteristics

• Change Streams Latency: < 1s for real-time CDC events
• Type Conversion: Handles all MongoDB BSON types with 100% accuracy
• Parallel Snapshot: Configurable chunk size (default: 100,000 documents)
• Memory Efficient: Streaming processing with configurable batch sizes

✅ Milestone Achievement

Phase 2 Progress:

• MySQL Connector: ✅ Complete
• PostgreSQL Connector: ✅ Complete
• MongoDB Connector: ✅ Complete (this release)
• Multi-source CDC coverage: 100% 🎉

Download v1.3.0 | GitHub Release

v1.2.6 - November 25, 2025

🚀 Features & Improvements

Global Log Level Control System

• Centralized logging with configurable levels (debug, info, warn, error)
• Thread-safe implementation with mutex protection
• Runtime level changes supported
• Compatible with existing log.Printf calls

🐛 Bug Fixes

• Fixed hardcoded debug logs in PostgreSQL connector
• Replaced 34+ log.Printf("[DEBUG]...") with logger.Debug(...)
• Improved production log output clarity

📝 Configuration Changes

• Updated default log level to “info” for production readiness
• Added logger import to PostgreSQL connector

Download v1.2.6 | GitHub Release

v1.2.5 - November 25, 2025

🐛 Bug Fixes

MySQL Date/Time Format Issues Fixed

• Implemented missing tryParseDateTime function
• Fixed timezone handling inconsistency
• Unified datetime format to UTC RFC3339Nano
• Eliminated document_parsing_exception errors in Elasticsearch

Download v1.2.5 | GitHub Release

v1.2.4 - November 25, 2025

🐛 Bug Fixes

force_initial_sync Configuration Fixed

• force_initial_sync now checked before checkpoint validation
• Allows forcing fresh initial synchronization when needed
• Useful for development, testing, and data consistency recovery

Download v1.2.4 | GitHub Release

v1.2.3 - November 24, 2025

🎉 Major Features

PostgreSQL CDC Fully Fixed and Operational

• Fixed “conn busy” error in WAL replication
• Fixed RELATION message parsing
• Optimized replication slot management
• Enhanced message processing and error handling
• Improved logging for better debugging

🐛 Bug Fixes

• Replication connection management improved
• Message type handling corrected
• Error recovery enhanced

Download v1.2.3 | GitHub Release

v1.0.1 - October 12, 2025

🐛 Bug Fixes

MySQL CDC Permissions Fix

• Fixed MySQL CDC permissions issues
• Improved error messages for permission-related problems
• Updated documentation with correct MySQL user setup

Download v1.0.1 | GitHub Release

v1.0.0 - October 12, 2025

🎉 Initial Release

Core Features

• MySQL CDC support
  • Binlog-based change data capture
  • Initial snapshot synchronization
  • Real-time streaming
• PostgreSQL CDC support
  • Logical replication-based CDC
  • WAL-based streaming
  • Initial snapshot support
• Elasticsearch sink
  • Bulk indexing
  • Document transformation
  • Index management
• Dead Letter Queue (DLQ)
  • Failed message handling
  • Retry mechanism
  • Error tracking
• Checkpoint system
  • Position tracking
  • Recovery support
  • State persistence
• Configuration
  • JSON-based configuration
  • Multi-source support
  • Flexible mapping

Documentation

• Installation guide
• Quick start tutorial
• Configuration reference
• Troubleshooting guide

Download v1.0.0 | GitHub Release

Legend

• 🚀 Features & Improvements
• 🐛 Bug Fixes
• ⚠️ Breaking Changes
• 📝 Documentation
• 🔒 Security
• ⚡ Performance

Stay Updated

• GitHub Releases
• GitHub Discussions
• RSS Feed

For detailed technical changes, see the full changelog on GitHub.