ElasticRelay - Multi-Source CDC Gateway
Real-time data synchronization from MySQL, PostgreSQL, MongoDB to Elasticsearch. Simple, reliable, and production-ready.
Key Features
Zero-Code Configuration
Wizard-style configuration with JSON-based setup
Multi-Database Support
MySQL, PostgreSQL, and MongoDB CDC with advanced features
Built-in Reliability
DLQ system, checkpointing, and automatic recovery
High Performance
Parallel processing, connection pooling, optimized throughput
Real-time Monitoring
Track sync status, metrics, and performance in real-time
Parallel Processing
Multi-threaded snapshot sync with chunking strategies
Why ElasticRelay?
- Zero-Code Configuration: Get started with simple JSON configuration
- Multi-Database Support: MySQL 5.7+, PostgreSQL 10+, and MongoDB 4.0+ CDC
- Data Transform Engine: Built-in field mapping, type conversion, data masking, expression evaluation, and conditional filtering — all configurable without code
- Built-in Reliability: DLQ system, automatic checkpointing, and recovery
- High Performance: Parallel processing and optimized throughput
- Production-Ready: Battle-tested in real-world scenarios
Architecture
┌─────────────┐
│ MySQL │───┐
└─────────────┘ │
│
┌─────────────┐ │ ┌──────────────┐ ┌───────────┐ ┌──────────────┐
│ PostgreSQL │───┼───▶│ ElasticRelay │───▶│ Transform │───▶│Elasticsearch │
└─────────────┘ │ └──────────────┘ └───────────┘ └──────────────┘
│
┌─────────────┐ │
│ MongoDB │───┘
└─────────────┘
Quick Start
MySQL Example
# 1. Download
wget https://github.com/yogoosoft/ElasticRelay/releases/latest/download/elasticrelay-linux-amd64
# 2. Make executable
chmod +x elasticrelay-linux-amd64
mv elasticrelay-linux-amd64 elasticrelay
# 3. Configure
cat > config.json <<EOF
{
"version": "3.0",
"data_sources": [{
"id": "mysql-main",
"type": "mysql",
"host": "localhost",
"port": 3306,
"user": "root",
"password": "password",
"database": "mydb"
}],
"elasticsearch": {
"addresses": ["http://localhost:9200"]
}
}
EOF
# 4. Run
./elasticrelay --config config.json
With Transform Engine
# Create transform configuration
cat > transform.json <<EOF
{
"transform_rules": [{
"name": "user-data-transform",
"table_pattern": "users",
"field_mappings": [
{"source": "user_name", "target": "username", "operation": "rename"}
],
"masking_rules": [
{"field": "phone", "strategy": "mask", "preset": "phone"},
{"field": "email", "strategy": "mask", "preset": "email"}
],
"filters": [
{"field": "status", "operator": "ne", "value": "deleted", "action": "exclude"}
]
}]
}
EOF
# Run with Transform Engine
./elasticrelay --config config.json -transform-config transform.json
MongoDB Example
# Configure for MongoDB
cat > config.json <<EOF
{
"version": "3.0",
"data_sources": [{
"id": "mongodb-main",
"type": "mongodb",
"connection_string": "mongodb://localhost:27017",
"database": "mydb"
}],
"elasticsearch": {
"addresses": ["http://localhost:9200"]
}
}
EOF
# Run
./elasticrelay --config config.json
Architecture
┌──────────────┐ ┌──────────────┐ ┌──────────────┐
│ │ │ │ │ │
│ MySQL │─────▶│ │─────▶│ │
│ │ │ │ │ │
└──────────────┘ │ │ │ Elasticsearch│
│ ElasticRelay │ │ │
┌──────────────┐ │ │ │ │
│ │─────▶│ │─────▶│ │
│ PostgreSQL │ │ │ │ │
│ │ │ │ └──────────────┘
└──────────────┘ │ │
│ │
┌──────────────┐ │ │
│ │─────▶│ │
│ MongoDB │ │ │
│ │ └──────┬───────┘
└──────────────┘ │
▼
┌─────────────────┐
│ DLQ System │
│ Checkpoints │
└─────────────────┘
Latest Version
v1.4.4
GitHub Stars
234
License
Apache 2.0