9.1 KiB
9.1 KiB
Fsync - File System Monitor with Redis and RabbitMQ
Distributed file synchronization daemon that monitors file changes across multiple Linux hosts using fanotify, publishes notifications via RabbitMQ, and automatically fetches updated files between hosts.
Key Features
✅ Cross-Host File Sync - Automatically copy files between hosts when changes detected ✅ Real-time Monitoring - fanotify API captures changes instantly ✅ Distributed Architecture - Monitor multiple hosts in a cluster ✅ Priority Queue - Fetch oldest/smallest files first ✅ Checksum Verification - SHA256 integrity checks after transfer ✅ RabbitMQ Integration - Publish/subscribe file updates ✅ Redis Logging - Persistent event and transfer records ✅ Automatic Retries - Recovers from transient failures
How It Works
Host A (Source) → Host B (Destination)
1. File modified on Host A
├─> Detect via fanotify
├─> Calculate checksum
└─> Publish to RabbitMQ
2. Host B receives notification
├─> Parse message
├─> Add to priority queue
└─> Schedule fetch
3. Fetch file from Host A
├─> SCP: host-a:/path/file → /tmp/fsync-cache/
├─> Verify checksum
└─> Record in Redis
Use Cases
- Configuration Management - Sync config files across servers
- Log Aggregation - Copy logs from multiple hosts
- Backup Automation - Continuous sync of important files
- Development - Keep code in sync across team machines
- Database Sync - Mirror database files between replicas
Quick Start
1. Start Services
docker-compose up -d
This starts:
- Redis (port 6379)
- RabbitMQ (port 5672, UI: 15672)
- Redis Commander (port 8081)
2. Build and Run
make build
sudo ./build/fsync-monitor
3. Test Cross-Host Sync
On Host A:
echo "updated content" >> /var/www/app.js
On Host B:
# File automatically fetched and cached
ls -la /tmp/fsync-cache/host-a_app.js
Architecture
┌─────────────────────────────────────────────────────────────┐
│ Cluster of Hosts │
├─────────────────────────────────────────────────────────────┤
│ │
│ Host A RabbitMQ Host B Host C │
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐│
│ │Monitor │──>│Exchange │───>│Monitor │ │Monitor ││
│ │Publish │ │fsync.* │ │Subscribe │ │Subscribe ││
│ └──────────┘ └──────────┘ └──────────┘ └──────────┘│
│ │ │ │ │ │
│ └──────────────┴────────────────┴──────────────┘ │
│ │ │
│ ┌──────────┐ │
│ │ Redis │ │
│ │ Storage │ │
│ └──────────┘ │
└─────────────────────────────────────────────────────────────┘
File Sync Details
Detection → Publication → Subscription → Fetch
| Step | Component | Action |
|---|---|---|
| 1 | fanotify | Detect file change |
| 2 | Checksum | Calculate SHA256 |
| 3 | Redis | Store event |
| 4 | RabbitMQ | Publish to exchange |
| 5 | RabbitMQ | Route to all subscribers |
| 6 | Queue Mgr | Add to priority queue |
| 7 | File Fetcher | Fetch via SCP |
| 8 | Verification | Verify checksum |
| 9 | Cache | Store locally |
| 10 | Redis | Log transfer |
Priority Algorithm
Files fetched in order of:
- Age - older changes processed first
- Size - smaller files before larger ones
Queue Order:
1st: 2-hour-old 2KB file
2nd: 90-minute-old 50MB file
3rd: 30-minute-old 100MB file
4th: 20-minute-old 500KB file
Configuration
Basic Setup
Edit src/config.h:
#define REDIS_HOST "127.0.0.1"
#define REDIS_PORT 6379
#define AMQP_HOST "127.0.0.1"
#define AMQP_PORT 5672
#define AMQP_USER "guest"
#define AMQP_PASSWORD "guest"
#define CACHE_DIR "/tmp/fsync-cache"
#define MAX_QUEUE_SIZE 1000
For Production
#define REDIS_HOST "redis.example.com"
#define AMQP_HOST "rabbitmq.example.com"
#define CACHE_DIR "/var/lib/fsync-cache"
#define MAX_QUEUE_SIZE 5000
Requirements
- Linux 3.8+ (fanotify)
- Redis 5.0+
- RabbitMQ 3.8+
- SSH configured between hosts
- GCC/Clang compiler
- Libraries:
- libhiredis
- libssl/libcrypto
- librabbitmq
- libcjson
Install Dependencies
Ubuntu/Debian:
sudo apt-get install -y \
build-essential \
libhiredis-dev \
libssl-dev \
librabbitmq-dev \
libcjson-dev
RHEL/CentOS:
sudo yum install -y \
gcc make \
libhiredis-devel \
openssl-devel \
librabbitmq-devel
Project Structure
fsync/
├── src/
│ ├── main.c # Event loop + RabbitMQ integration
│ ├── monitor.c/h # fanotify monitoring
│ ├── redis_client.c/h # Redis operations
│ ├── amqp_client.c/h # RabbitMQ client (pub/sub)
│ ├── queue_manager.c/h # Priority queue for file updates
│ ├── file_fetcher.c/h # Remote file fetching via SCP
│ ├── checksum.c/h # SHA256 verification
│ └── config.h # Configuration constants
├── docs/
│ ├── API.md # Redis schema reference
│ └── RABBITMQ.md # RabbitMQ setup guide
├── config/fsync.conf # Runtime config template
├── Makefile # Build system
├── docker-compose.yml # Redis + RabbitMQ stack
└── README.md # This file
Command Line
# Build
make build
# Run (requires root)
sudo ./build/fsync-monitor
# Run with custom hosts
sudo ./build/fsync-monitor redis.local rabbitmq.local
# View logs
sudo ./build/fsync-monitor 2>&1 | grep -i error
# Clean build
make clean
# Install system-wide
sudo make install
Monitoring
RabbitMQ Management UI
- Default: guest/guest
- View queues, exchanges, bindings
- Monitor message flow
Redis CLI
# Check events
redis-cli LRANGE fsync:events:list 0 -1
# View specific event
redis-cli HGETALL fsync:events:*
# Monitor transfers
redis-cli KEYS "fsync:fetches:*"
# Watch in real-time
redis-cli MONITOR
Local Cache
# View cached files
ls -lh /tmp/fsync-cache/
# Check cache size
du -sh /tmp/fsync-cache/
# Clear old files (older than 7 days)
find /tmp/fsync-cache/ -mtime +7 -delete
Example Workflow
Scenario: Config File Update Across Cluster
# On web-server-1
echo "updated_setting=true" >> /etc/app.conf
# Automatically happens:
# 1. web-server-1 detects change
# 2. Calculates checksum
# 3. Publishes to RabbitMQ
# 4. web-server-2 receives notification
# 5. Adds /etc/app.conf to fetch queue
# 6. Fetches from web-server-1 via SCP
# 7. Verifies checksum
# 8. Caches at /tmp/fsync-cache/web-server-1_app.conf
# Verify on web-server-2
ls -la /tmp/fsync-cache/ | grep app.conf
Troubleshooting
Connection Issues
# Test Redis
redis-cli PING
# Test RabbitMQ
sudo docker logs fsync-rabbitmq
File Not Syncing
-
Check SSH access:
ssh user@remote-host 'cat /path/to/file' -
Check file permissions:
ls -la /path/to/file -
Check queue:
redis-cli LLEN fsync:events:list
Build Errors
Install missing development packages:
# Ubuntu
sudo apt-get install libhiredis-dev libssl-dev librabbitmq-dev
# RHEL
sudo yum install libhiredis-devel openssl-devel librabbitmq-devel
Documentation
- Redis API Schema - Event structure and queries
- RabbitMQ Integration - Messaging and sync details
Performance
- Event Processing: <1ms (fanotify)
- Checksum Calculation: 100-500ms per file
- File Transfer: Network-dependent
- Queue Size: Supports 1000+ pending items
- Memory: ~1-2KB per queued file
Security
- SSH: Key-based authentication required
- RabbitMQ: Configure credentials in production
- Redis: Add
requirepassauthentication - Network: Run on isolated/private networks
License
MIT