Introduction

What is ElasticRelay and why use it?

Introduction

ElasticRelay is a high-performance, production-ready Change Data Capture (CDC) gateway that enables real-time data synchronization from multiple database sources to Elasticsearch.

What is ElasticRelay?

ElasticRelay captures database changes in real-time and synchronizes them to Elasticsearch, making your data instantly searchable. It supports:

  • MySQL 5.7+ - Using binlog-based CDC
  • PostgreSQL 10+ - Using logical replication
  • MongoDB 4.0+ - Using change streams

Key Features

Zero-Code Configuration

Get started with simple JSON configuration files. No programming required.

Multi-Database Support

Connect to multiple databases simultaneously and sync them to Elasticsearch with a single ElasticRelay instance.

Data Transform Engine

Built-in enterprise-grade data transformation pipeline that processes data in-flight between your databases and Elasticsearch:

  • Field Mapping: Rename, copy, and move fields with nested path support (user.profile.name)
  • Type Conversion: 10+ type conversions including string, int, float, bool, date, timestamp
  • Data Masking: 4 masking strategies (mask, hash, token, regex) with 5 preset templates for phone, ID card, email, bank card, and name
  • Expression Engine: 16 built-in functions for string, math, date, and conditional operations
  • Conditional Filtering: 10 operators (eq, ne, gt, lt, in, regex, exists, etc.) with include/exclude/route actions
  • Performance: ~800,000 ops/sec full pipeline throughput

Built-in Reliability

  • Automatic Checkpointing: Never lose data with automatic progress tracking
  • Dead Letter Queue (DLQ): Failed events are safely stored for retry
  • Automatic Recovery: Seamlessly resume from where you left off after restarts

High Performance

  • Parallel Processing: Multi-threaded snapshot synchronization for large tables
  • Optimized Throughput: Efficient batching and bulk operations
  • Memory Efficient: Smart memory management for handling large datasets

Production-Ready

  • Battle-Tested: Used in production environments processing millions of events
  • Monitoring: Built-in metrics and health checks
  • Flexible Deployment: Run as a standalone binary, Docker container, or Kubernetes pod

Use Cases

Make your database data instantly searchable by syncing to Elasticsearch in real-time.

Analytics & Reporting

Build real-time dashboards and analytics on top of your operational data.

Data Lake Integration

Stream database changes to Elasticsearch as part of your data lake architecture.

Microservices Data Sync

Keep search indexes in sync with your microservices databases.

How It Works

  1. Connect: ElasticRelay connects to your database and starts listening for changes
  2. Capture: Database changes (INSERT, UPDATE, DELETE) are captured in real-time
  3. Transform: Data is transformed through the Transform Engine — field mapping, type conversion, masking, filtering, and expression evaluation are applied based on your configuration
  4. Sync: Changes are efficiently synced to Elasticsearch with automatic retry
  5. Monitor: Track progress with built-in checkpointing and metrics

Architecture Overview

┌─────────────┐
│   MySQL     │───┐
└─────────────┘   │
                  │
┌─────────────┐   │    ┌──────────────┐    ┌───────────┐    ┌──────────────┐
│ PostgreSQL  │───┼───▶│ ElasticRelay │───▶│ Transform │───▶│Elasticsearch │
└─────────────┘   │    └──────────────┘    └───────────┘    └──────────────┘
                  │
┌─────────────┐   │
│  MongoDB    │───┘
└─────────────┘

The Transform Engine sits between the CDC capture layer and the Elasticsearch sink, enabling data transformation without external dependencies.

What’s Next?