Introduction

What is ElasticRelay and why use it?

Introduction

ElasticRelay is a high-performance, production-ready Change Data Capture (CDC) gateway that enables real-time data synchronization from multiple database sources to Elasticsearch.

What is ElasticRelay?

ElasticRelay captures database changes in real-time and synchronizes them to Elasticsearch, making your data instantly searchable. It supports:

  • MySQL 5.7+ - Using binlog-based CDC
  • PostgreSQL 10+ - Using logical replication
  • MongoDB 4.0+ - Using change streams

Key Features

Zero-Code Configuration

Get started with simple JSON configuration files. No programming required.

Multi-Database Support

Connect to multiple databases simultaneously and sync them to Elasticsearch with a single ElasticRelay instance.

Built-in Reliability

  • Automatic Checkpointing: Never lose data with automatic progress tracking
  • Dead Letter Queue (DLQ): Failed events are safely stored for retry
  • Automatic Recovery: Seamlessly resume from where you left off after restarts

High Performance

  • Parallel Processing: Multi-threaded snapshot synchronization for large tables
  • Optimized Throughput: Efficient batching and bulk operations
  • Memory Efficient: Smart memory management for handling large datasets

Production-Ready

  • Battle-Tested: Used in production environments processing millions of events
  • Monitoring: Built-in metrics and health checks
  • Flexible Deployment: Run as a standalone binary, Docker container, or Kubernetes pod

Use Cases

Make your database data instantly searchable by syncing to Elasticsearch in real-time.

Analytics & Reporting

Build real-time dashboards and analytics on top of your operational data.

Data Lake Integration

Stream database changes to Elasticsearch as part of your data lake architecture.

Microservices Data Sync

Keep search indexes in sync with your microservices databases.

How It Works

  1. Connect: ElasticRelay connects to your database and starts listening for changes
  2. Capture: Database changes (INSERT, UPDATE, DELETE) are captured in real-time
  3. Transform: Data is transformed to match Elasticsearch document format
  4. Sync: Changes are efficiently synced to Elasticsearch with automatic retry
  5. Monitor: Track progress with built-in checkpointing and metrics

Architecture Overview

┌─────────────┐
│   MySQL     │───┐
└─────────────┘   │
                  │
┌─────────────┐   │    ┌──────────────┐    ┌──────────────┐
│ PostgreSQL  │───┼───▶│ ElasticRelay │───▶│Elasticsearch │
└─────────────┘   │    └──────────────┘    └──────────────┘
                  │
┌─────────────┐   │
│  MongoDB    │───┘
└─────────────┘

What’s Next?