fsync/docs/RABBITMQ.md

RabbitMQ Integration Guide

Fsync integrates with RabbitMQ for distributed file synchronization across multiple hosts.

Architecture

Host A                    RabbitMQ             Host B
┌──────────────┐         ┌──────────┐         ┌──────────────┐
│ Monitor      │         │Exchange  │         │ Monitor      │
│ (Publish)    │────────>│fsync.*   │────────>│ (Subscribe)  │
│              │         │          │         │              │
│ Detects:     │         └──────────┘         │ Receives:    │
│ - File mods  │              │               │ - Updates    │
│ - Creates    │         ┌─────────────┐      │ - Adds queue │
│ - Deletes    │         │Queue        │      │ - Fetches    │
└──────────────┘         │fsync.files  │      │ - Verifies   │
                         └─────────────┘      └──────────────┘

How File Sync Works

1. Local File Change Detection (Host A)

• File modified on Host A
• Fanotify detects change immediately
• Calculate SHA256 checksum
• Store event in Redis
• Publish to RabbitMQ with hostname, path, checksum

2. Remote Host Receives Update (Host B)

• Consume message from RabbitMQ queue
• Parse JSON with file details
• Add to priority queue (oldest/smallest first)
• Record in Redis

3. File Fetch and Sync (Host B)

• Process queue every 5 seconds
• Get next highest priority file
• Fetch via SCP from Host A (hostname:path)
• Verify checksum matches
• Store locally in cache
• Update Redis with status

Message Format

Events published to RabbitMQ as JSON:

{
  "hostname": "server1",
  "path": "/var/www/app.js",
  "event": "MODIFY",
  "checksum": "a1b2c3d4e5f6...",
  "timestamp": 1684423051,
  "size": 1024
}

Queue Priority Algorithm

Files fetched in this order:

1. By Age (oldest first) - files detected earliest get priority
2. By Size (smallest first) - smaller files processed faster

Example queue:

Order | Host  | File           | Age   | Size    | Status
------|-------|----------------|-------|---------|--------
1     | host2 | /etc/config    | 2h    | 2KB     | Fetch now
2     | host1 | /app/db.sql    | 90m   | 50MB    | Next
3     | host2 | /cache.tmp     | 30m   | 100MB   | Waiting
4     | host1 | /data/doc.pdf  | 20m   | 500KB   | Waiting

Configuration

Edit src/config.h:

/* RabbitMQ Server */
#define AMQP_HOST "127.0.0.1"
#define AMQP_PORT 5672
#define AMQP_USER "guest"
#define AMQP_PASSWORD "guest"
#define AMQP_VHOST "/"

/* Exchange and Queue */
#define AMQP_EXCHANGE "fsync.events"
#define AMQP_QUEUE_UPDATES "fsync.file.updates"
#define AMQP_ROUTING_KEY "fsync.file.*"

/* File Cache */
#define CACHE_DIR "/tmp/fsync-cache"
#define MAX_QUEUE_SIZE 1000

Quick Start

1. Start Services

docker-compose up -d
make build

2. Run Monitor on Multiple Hosts

Host A:

sudo ./build/fsync-monitor

Host B:

sudo ./build/fsync-monitor

3. Test Cross-Host Sync

On Host A, modify a file:

echo "test data" >> /var/www/index.html

Host B will:

• Receive notification via RabbitMQ
• Add /var/www/index.html to fetch queue
• Fetch file from Host A via SCP
• Verify checksum
• Cache locally

4. Monitor Progress

Check queue status:

redis-cli LLEN fsync:events:list

View cached files:

ls -lh /tmp/fsync-cache/

Monitor RabbitMQ: Open http://localhost:15672 (guest/guest)

Operational Scenarios

Scenario 1: Config File Update

Host1: /etc/app.conf modified
  ├─> Checksum: a1b2c3d4
  └─> Publish to RabbitMQ
       │
Host2: Receives notification
  ├─> Add to queue (priority 1)
  ├─> Fetch /etc/app.conf from Host1
  ├─> Verify checksum matches
  └─> Cache at /tmp/fsync-cache/host1_app.conf

Scenario 2: Bulk File Changes

1000 files modified on Host1
  └─> 1000 messages to RabbitMQ

Host2 receives 1000 updates
  └─> Add all to priority queue
       │
Processing:
  ├─> Fetch oldest files first
  ├─> Then by size (KB before MB)
  ├─> One at a time sequentially
  └─> Verify each checksum

Scenario 3: Multiple Source Hosts

Host1 changes: /var/www/app.js
  └─> Publish to RabbitMQ

Host2 changes: /etc/nginx.conf
  └─> Publish to RabbitMQ

Host3 receives both notifications
  └─> Queue both files
  └─> Fetch oldest first
  └─> Then by size

File Transfer Mechanism

SCP-Based Transfer

Current implementation uses scp:

scp -o ConnectTimeout=30 hostname:path local_path

Requirements:

• SSH key authentication configured
• SSH access between hosts
• Read permissions on source file

Checksum Verification

After transfer:

1. Calculate SHA256 of fetched file
2. Compare with checksum from notification
3. If mismatch: retry (max 3 attempts)
4. If success: cache and record in Redis

Redis Storage

Published Events

Key: fsync:events:{timestamp}:{hostname}:{pid}

Example:
fsync:events:2026-05-18T13:30:00Z:server1:1234
  ├─ hostname: "server1"
  ├─ path: "/var/www/app.js"
  ├─ event_type: "MODIFY"
  ├─ checksum: "a1b2c3d4..."
  └─ timestamp: "2026-05-18T13:30:00Z"

Fetch Records

Key: fsync:fetches:success:{hostname}:{path}:{timestamp}
Key: fsync:fetches:failed:{hostname}:{path}:{timestamp}

Performance

Throughput

• Event detection: <1ms
• Publishing: ~5-10ms per event
• File fetch: Depends on size/network
• Verification: ~100-500ms per file

Queue Processing

• Check interval: 1 second
• Process 1 file per 5 seconds (configurable)
• Network-bound for file transfer

Memory Usage

• Queue: ~1-2KB per pending file
• Cache: Limited by disk space
• Connections: ~5MB per RabbitMQ connection

Troubleshooting

Queue Not Draining

[QUEUE] Added: host2:/etc/config (queue size: 100)
[FETCHER] Failed to fetch: host2:/etc/config

Causes:

1. SSH key not configured
2. File permissions on source
3. Network connectivity
4. Host not reachable

Debug:

# Test SSH access
ssh user@host2 'ls -la /etc/config'

# Check firewall
nc -zv host2 22

Checksum Mismatch

[FETCHER] Checksum mismatch: /tmp/fsync-cache/host2_config

Causes:

1. File changed during transfer
2. Network corruption
3. Filesystem error

Action: Automatic retry (up to 3 times)

RabbitMQ Connection Lost

Failed to open AMQP socket

Fix:

docker-compose logs rabbitmq
docker-compose restart rabbitmq

File Transfer Timeout

[FETCHER] Failed to fetch after 3 attempts

Solutions:

1. Increase FETCH_TIMEOUT_SEC in config
2. Check network bandwidth
3. Check host load

Advanced Configuration

Custom Cache Directory

#define CACHE_DIR "/var/lib/fsync-cache"

Persist fetched files across restarts.

Parallel Fetching (Future)

Currently processes 1 file at a time. Can be extended to:

• Multi-threaded fetcher
• Batch SCP transfers
• Parallel checksums

Routing Filters

Modify routing key to sync specific paths:

#define AMQP_ROUTING_KEY "fsync.file.var.www.*"  /* Only /var/www */

Security Considerations

SSH Authentication

• Configure SSH keys for password-less access
• Use key-based authentication
• Restrict SSH to internal network

RabbitMQ

• Change default guest/guest credentials
• Use TLS for AMQP connections
• Restrict network access

File Permissions

• Monitor has read access to synced files
• Cache directory writable by monitor
• Source files readable by SSH user

Monitoring and Metrics

Queue Depth

redis-cli LLEN fsync:events:list

Fetch Success Rate

redis-cli KEYS "fsync:fetches:*" | wc -l

Cache Size

du -sh /tmp/fsync-cache/

Roadmap

• Basic file sync via RabbitMQ
• Priority queue (age + size)
• Checksum verification
• Multi-threaded fetching
• Rsync support (resume capability)
• Compression for transfers
• Bandwidth throttling
• Web UI for monitoring
• Metrics export (Prometheus)