Files
fsync/docs/RABBITMQ.md
2026-05-18 14:05:06 -04:00

401 lines
8.3 KiB
Markdown

# RabbitMQ Integration Guide
Fsync integrates with RabbitMQ for distributed file synchronization across multiple hosts.
## Architecture
```
Host A RabbitMQ Host B
┌──────────────┐ ┌──────────┐ ┌──────────────┐
│ Monitor │ │Exchange │ │ Monitor │
│ (Publish) │────────>│fsync.* │────────>│ (Subscribe) │
│ │ │ │ │ │
│ Detects: │ └──────────┘ │ Receives: │
│ - File mods │ │ │ - Updates │
│ - Creates │ ┌─────────────┐ │ - Adds queue │
│ - Deletes │ │Queue │ │ - Fetches │
└──────────────┘ │fsync.files │ │ - Verifies │
└─────────────┘ └──────────────┘
```
## How File Sync Works
### 1. Local File Change Detection (Host A)
- File modified on Host A
- Fanotify detects change immediately
- Calculate SHA256 checksum
- Store event in Redis
- **Publish to RabbitMQ** with hostname, path, checksum
### 2. Remote Host Receives Update (Host B)
- Consume message from RabbitMQ queue
- Parse JSON with file details
- **Add to priority queue** (oldest/smallest first)
- Record in Redis
### 3. File Fetch and Sync (Host B)
- Process queue every 5 seconds
- Get next highest priority file
- **Fetch via SCP** from Host A (hostname:path)
- Verify checksum matches
- Store locally in cache
- Update Redis with status
## Message Format
Events published to RabbitMQ as JSON:
```json
{
"hostname": "server1",
"path": "/var/www/app.js",
"event": "MODIFY",
"checksum": "a1b2c3d4e5f6...",
"timestamp": 1684423051,
"size": 1024
}
```
## Queue Priority Algorithm
Files fetched in this order:
1. **By Age** (oldest first) - files detected earliest get priority
2. **By Size** (smallest first) - smaller files processed faster
Example queue:
```
Order | Host | File | Age | Size | Status
------|-------|----------------|-------|---------|--------
1 | host2 | /etc/config | 2h | 2KB | Fetch now
2 | host1 | /app/db.sql | 90m | 50MB | Next
3 | host2 | /cache.tmp | 30m | 100MB | Waiting
4 | host1 | /data/doc.pdf | 20m | 500KB | Waiting
```
## Configuration
Edit `src/config.h`:
```c
/* RabbitMQ Server */
#define AMQP_HOST "127.0.0.1"
#define AMQP_PORT 5672
#define AMQP_USER "guest"
#define AMQP_PASSWORD "guest"
#define AMQP_VHOST "/"
/* Exchange and Queue */
#define AMQP_EXCHANGE "fsync.events"
#define AMQP_QUEUE_UPDATES "fsync.file.updates"
#define AMQP_ROUTING_KEY "fsync.file.*"
/* File Cache */
#define CACHE_DIR "/tmp/fsync-cache"
#define MAX_QUEUE_SIZE 1000
```
## Quick Start
### 1. Start Services
```bash
docker-compose up -d
make build
```
### 2. Run Monitor on Multiple Hosts
**Host A:**
```bash
sudo ./build/fsync-monitor
```
**Host B:**
```bash
sudo ./build/fsync-monitor
```
### 3. Test Cross-Host Sync
On Host A, modify a file:
```bash
echo "test data" >> /var/www/index.html
```
Host B will:
- Receive notification via RabbitMQ
- Add `/var/www/index.html` to fetch queue
- Fetch file from Host A via SCP
- Verify checksum
- Cache locally
### 4. Monitor Progress
**Check queue status:**
```bash
redis-cli LLEN fsync:events:list
```
**View cached files:**
```bash
ls -lh /tmp/fsync-cache/
```
**Monitor RabbitMQ:**
Open http://localhost:15672 (guest/guest)
## Operational Scenarios
### Scenario 1: Config File Update
```
Host1: /etc/app.conf modified
├─> Checksum: a1b2c3d4
└─> Publish to RabbitMQ
│
Host2: Receives notification
├─> Add to queue (priority 1)
├─> Fetch /etc/app.conf from Host1
├─> Verify checksum matches
└─> Cache at /tmp/fsync-cache/host1_app.conf
```
### Scenario 2: Bulk File Changes
```
1000 files modified on Host1
└─> 1000 messages to RabbitMQ
Host2 receives 1000 updates
└─> Add all to priority queue
│
Processing:
├─> Fetch oldest files first
├─> Then by size (KB before MB)
├─> One at a time sequentially
└─> Verify each checksum
```
### Scenario 3: Multiple Source Hosts
```
Host1 changes: /var/www/app.js
└─> Publish to RabbitMQ
Host2 changes: /etc/nginx.conf
└─> Publish to RabbitMQ
Host3 receives both notifications
└─> Queue both files
└─> Fetch oldest first
└─> Then by size
```
## File Transfer Mechanism
### SCP-Based Transfer
Current implementation uses `scp`:
```bash
scp -o ConnectTimeout=30 hostname:path local_path
```
**Requirements:**
- SSH key authentication configured
- SSH access between hosts
- Read permissions on source file
### Checksum Verification
After transfer:
1. Calculate SHA256 of fetched file
2. Compare with checksum from notification
3. If mismatch: retry (max 3 attempts)
4. If success: cache and record in Redis
## Redis Storage
### Published Events
```
Key: fsync:events:{timestamp}:{hostname}:{pid}
Example:
fsync:events:2026-05-18T13:30:00Z:server1:1234
├─ hostname: "server1"
├─ path: "/var/www/app.js"
├─ event_type: "MODIFY"
├─ checksum: "a1b2c3d4..."
└─ timestamp: "2026-05-18T13:30:00Z"
```
### Fetch Records
```
Key: fsync:fetches:success:{hostname}:{path}:{timestamp}
Key: fsync:fetches:failed:{hostname}:{path}:{timestamp}
```
## Performance
### Throughput
- **Event detection:** <1ms
- **Publishing:** ~5-10ms per event
- **File fetch:** Depends on size/network
- **Verification:** ~100-500ms per file
### Queue Processing
- Check interval: 1 second
- Process 1 file per 5 seconds (configurable)
- Network-bound for file transfer
### Memory Usage
- Queue: ~1-2KB per pending file
- Cache: Limited by disk space
- Connections: ~5MB per RabbitMQ connection
## Troubleshooting
### Queue Not Draining
```
[QUEUE] Added: host2:/etc/config (queue size: 100)
[FETCHER] Failed to fetch: host2:/etc/config
```
**Causes:**
1. SSH key not configured
2. File permissions on source
3. Network connectivity
4. Host not reachable
**Debug:**
```bash
# Test SSH access
ssh user@host2 'ls -la /etc/config'
# Check firewall
nc -zv host2 22
```
### Checksum Mismatch
```
[FETCHER] Checksum mismatch: /tmp/fsync-cache/host2_config
```
**Causes:**
1. File changed during transfer
2. Network corruption
3. Filesystem error
**Action:** Automatic retry (up to 3 times)
### RabbitMQ Connection Lost
```
Failed to open AMQP socket
```
**Fix:**
```bash
docker-compose logs rabbitmq
docker-compose restart rabbitmq
```
### File Transfer Timeout
```
[FETCHER] Failed to fetch after 3 attempts
```
**Solutions:**
1. Increase `FETCH_TIMEOUT_SEC` in config
2. Check network bandwidth
3. Check host load
## Advanced Configuration
### Custom Cache Directory
```c
#define CACHE_DIR "/var/lib/fsync-cache"
```
Persist fetched files across restarts.
### Parallel Fetching (Future)
Currently processes 1 file at a time. Can be extended to:
- Multi-threaded fetcher
- Batch SCP transfers
- Parallel checksums
### Routing Filters
Modify routing key to sync specific paths:
```c
#define AMQP_ROUTING_KEY "fsync.file.var.www.*" /* Only /var/www */
```
## Security Considerations
### SSH Authentication
- Configure SSH keys for password-less access
- Use key-based authentication
- Restrict SSH to internal network
### RabbitMQ
- Change default guest/guest credentials
- Use TLS for AMQP connections
- Restrict network access
### File Permissions
- Monitor has read access to synced files
- Cache directory writable by monitor
- Source files readable by SSH user
## Monitoring and Metrics
### Queue Depth
```bash
redis-cli LLEN fsync:events:list
```
### Fetch Success Rate
```bash
redis-cli KEYS "fsync:fetches:*" | wc -l
```
### Cache Size
```bash
du -sh /tmp/fsync-cache/
```
## Roadmap
- [x] Basic file sync via RabbitMQ
- [x] Priority queue (age + size)
- [x] Checksum verification
- [ ] Multi-threaded fetching
- [ ] Rsync support (resume capability)
- [ ] Compression for transfers
- [ ] Bandwidth throttling
- [ ] Web UI for monitoring
- [ ] Metrics export (Prometheus)