Changelog

All notable changes to ElasticRelay are documented here.

v1.3.1 - December 15, 2025

🐛 Bug Fixes

1. MongoDB CDC Index Routing Fix

• Fixed MongoDB CDC events incorrectly routing to elasticrelay_mongo-default index
• Updated extractTableName() to check both _table (MySQL/PostgreSQL) and _collection (MongoDB) fields
• cleanDataForES() now properly removes _collection, _database, and _id metadata fields

2. MongoDB Authentication Fix

• Added authSource=admin to MongoDB connection URI for proper authentication

3. MongoDB Snapshot Primary Key Extraction

• Fixed beginStandardSnapshot() to use _table field for consistency with ES sink

4. Primary Key Detection Enhancement

• Updated processSnapshotChunk() to check for _id field first, then fall back to id

🔧 Improvements

1. Elasticsearch Timeout Configuration

• Increased timeout from 3s to 30s for ensureIndexExists() and createDefaultIndex() operations
• Improved reliability for remote Elasticsearch servers with higher latency

2. Docker Compose MongoDB Replica Set

• Enhanced MongoDB container with keyFile authentication for replica set
• Improved replica set status detection and conditional initialization

3. Documentation Updates

• Updated README.md with MongoDB setup instructions
• Added MongoDB-specific scripts reference

4. Code Formatting

• Fixed indentation issues in cmd/elasticrelay/main.go
• Added MongoDB connector case to connector server creation switch statement

📁 Files Changed

• internal/sink/es/es.go - MongoDB collection name support in ES sink
• internal/connectors/mongodb/mongodb.go - Auth source fix and metadata field consistency
• internal/orchestrator/multi_orchestrator.go - MongoDB connector integration and _id support
• cmd/elasticrelay/main.go - MongoDB connector server creation
• docker-compose.yml - MongoDB replica set keyFile configuration
• README.md - MongoDB setup documentation

Download v1.3.1 | GitHub Release

v1.3.0 - December 7, 2025

🎉 Major Release: MongoDB Connector Complete Implementation

This release marks the completion of MongoDB Connector development, achieving 100% coverage of the three major database sources (MySQL, PostgreSQL, MongoDB) for the ElasticRelay CDC platform.

🚀 New Features

1. MongoDB Change Streams CDC Implementation

• Full implementation of MongoDB Change Streams for real-time CDC
• Automatic detection of Standalone, ReplicaSet, and Sharded Cluster deployments
• Complete resume token encoding/decoding for checkpoint persistence
• Support for INSERT, UPDATE, REPLACE, and DELETE operations
• Configurable options for flexible configuration

2. BSON Type Converter System

Complete BSON to JSON-friendly type conversion supporting:

• Basic Types: ObjectID → string (hex), DateTime → RFC3339, Timestamp → map
• Binary Types: Binary → base64 encoded map with subtype
• Numeric Types: Decimal128 → string (precision preserved), int32 → int64 normalization
• Special Types: Regex → map, JavaScript → string, CodeWithScope → map
• MongoDB-Specific: MinKey, MaxKey, DBPointer, Symbol, Undefined, Null
• Nested Structures: Recursive document and array conversion
• Document Flattening: Configurable max depth for Elasticsearch compatibility

3. Sharded Cluster Support

• Dedicated connector for sharded cluster monitoring via mongos
• Multi-shard watching for aggregated change events across shards
• Migration awareness for consistency during chunk migrations
• Chunk distribution monitoring for data distribution across shards

4. Parallel Snapshot Manager Integration

• MongoDBSnapshotAdapter implementing parallel snapshot interface
• Collection info retrieval with document count and field schema detection
• Multiple chunking strategies:
  • ObjectID-based chunking for standard collections
  • Numeric ID-based chunking for integer primary keys
  • Skip/Limit fallback for complex ID types
• Parallel processing for coordinated parallel snapshots

5. Checkpoint Manager Enhancement

• Job-specific checkpoint with resume token, cluster time, and event count
• Thread-safe operations with mutex protection
• Event counting for monitoring
• JSON file-based checkpoint persistence

🧪 Testing

Unit Tests

• Complete test coverage for BSON type conversion
• URI construction and pipeline building tests
• Operation type mapping and structure validation
• Full CRUD, concurrency, and persistence tests
• Benchmark tests for performance validation

Integration Tests

• Basic Change Stream functionality
• Resume token persistence and recovery
• UPDATE and DELETE operation handling
• Real MongoDB data type conversion
• Database-wide change monitoring
• Full connector integration

📊 Performance Characteristics

• Change Streams Latency: < 1s for real-time CDC events
• Type Conversion: Handles all MongoDB BSON types with 100% accuracy
• Parallel Snapshot: Configurable chunk size (default: 100,000 documents)
• Memory Efficient: Streaming processing with configurable batch sizes

✅ Milestone Achievement

Phase 2 Progress:

• MySQL Connector: ✅ Complete
• PostgreSQL Connector: ✅ Complete
• MongoDB Connector: ✅ Complete (this release)
• Multi-source CDC coverage: 100% 🎉

Download v1.3.0 | GitHub Release

v1.2.6 - November 25, 2025

🚀 Features & Improvements

Global Log Level Control System

• Centralized logging with configurable levels (debug, info, warn, error)
• Thread-safe implementation with mutex protection
• Runtime level changes supported
• Compatible with existing log.Printf calls

🐛 Bug Fixes

• Fixed hardcoded debug logs in PostgreSQL connector
• Replaced 34+ log.Printf("[DEBUG]...") with logger.Debug(...)
• Improved production log output clarity

📝 Configuration Changes

• Updated default log level to “info” for production readiness
• Added logger import to PostgreSQL connector

Download v1.2.6 | GitHub Release

v1.2.5 - November 25, 2025

🐛 Bug Fixes

MySQL Date/Time Format Issues Fixed

• Implemented missing tryParseDateTime function
• Fixed timezone handling inconsistency
• Unified datetime format to UTC RFC3339Nano
• Eliminated document_parsing_exception errors in Elasticsearch

Download v1.2.5 | GitHub Release

v1.2.4 - November 25, 2025

🐛 Bug Fixes

force_initial_sync Configuration Fixed

• force_initial_sync now checked before checkpoint validation
• Allows forcing fresh initial synchronization when needed
• Useful for development, testing, and data consistency recovery

Download v1.2.4 | GitHub Release

v1.2.3 - November 24, 2025

🎉 Major Features

PostgreSQL CDC Fully Fixed and Operational

• Fixed “conn busy” error in WAL replication
• Fixed RELATION message parsing
• Optimized replication slot management
• Enhanced message processing and error handling
• Improved logging for better debugging

🐛 Bug Fixes

• Replication connection management improved
• Message type handling corrected
• Error recovery enhanced

Download v1.2.3 | GitHub Release

v1.0.1 - October 12, 2025

🐛 Bug Fixes

MySQL CDC Permissions Fix

• Fixed MySQL CDC permissions issues
• Improved error messages for permission-related problems
• Updated documentation with correct MySQL user setup

Download v1.0.1 | GitHub Release

v1.0.0 - October 12, 2025

🎉 Initial Release

Core Features

• MySQL CDC support
  • Binlog-based change data capture
  • Initial snapshot synchronization
  • Real-time streaming
• PostgreSQL CDC support
  • Logical replication-based CDC
  • WAL-based streaming
  • Initial snapshot support
• Elasticsearch sink
  • Bulk indexing
  • Document transformation
  • Index management
• Dead Letter Queue (DLQ)
  • Failed message handling
  • Retry mechanism
  • Error tracking
• Checkpoint system
  • Position tracking
  • Recovery support
  • State persistence
• Configuration
  • JSON-based configuration
  • Multi-source support
  • Flexible mapping

Documentation

• Installation guide
• Quick start tutorial
• Configuration reference
• Troubleshooting guide

Download v1.0.0 | GitHub Release

Legend

• 🚀 Features & Improvements
• 🐛 Bug Fixes
• ⚠️ Breaking Changes
• 📝 Documentation
• 🔒 Security
• ⚡ Performance

Stay Updated

• GitHub Releases
• GitHub Discussions
• RSS Feed

For detailed technical changes, see the full changelog on GitHub.