401 lines
8.3 KiB
Markdown
401 lines
8.3 KiB
Markdown
# RabbitMQ Integration Guide
|
|
|
|
Fsync integrates with RabbitMQ for distributed file synchronization across multiple hosts.
|
|
|
|
## Architecture
|
|
|
|
```
|
|
Host A RabbitMQ Host B
|
|
┌──────────────┐ ┌──────────┐ ┌──────────────┐
|
|
│ Monitor │ │Exchange │ │ Monitor │
|
|
│ (Publish) │────────>│fsync.* │────────>│ (Subscribe) │
|
|
│ │ │ │ │ │
|
|
│ Detects: │ └──────────┘ │ Receives: │
|
|
│ - File mods │ │ │ - Updates │
|
|
│ - Creates │ ┌─────────────┐ │ - Adds queue │
|
|
│ - Deletes │ │Queue │ │ - Fetches │
|
|
└──────────────┘ │fsync.files │ │ - Verifies │
|
|
└─────────────┘ └──────────────┘
|
|
```
|
|
|
|
## How File Sync Works
|
|
|
|
### 1. Local File Change Detection (Host A)
|
|
|
|
- File modified on Host A
|
|
- Fanotify detects change immediately
|
|
- Calculate SHA256 checksum
|
|
- Store event in Redis
|
|
- **Publish to RabbitMQ** with hostname, path, checksum
|
|
|
|
### 2. Remote Host Receives Update (Host B)
|
|
|
|
- Consume message from RabbitMQ queue
|
|
- Parse JSON with file details
|
|
- **Add to priority queue** (oldest/smallest first)
|
|
- Record in Redis
|
|
|
|
### 3. File Fetch and Sync (Host B)
|
|
|
|
- Process queue every 5 seconds
|
|
- Get next highest priority file
|
|
- **Fetch via SCP** from Host A (hostname:path)
|
|
- Verify checksum matches
|
|
- Store locally in cache
|
|
- Update Redis with status
|
|
|
|
## Message Format
|
|
|
|
Events published to RabbitMQ as JSON:
|
|
|
|
```json
|
|
{
|
|
"hostname": "server1",
|
|
"path": "/var/www/app.js",
|
|
"event": "MODIFY",
|
|
"checksum": "a1b2c3d4e5f6...",
|
|
"timestamp": 1684423051,
|
|
"size": 1024
|
|
}
|
|
```
|
|
|
|
## Queue Priority Algorithm
|
|
|
|
Files fetched in this order:
|
|
|
|
1. **By Age** (oldest first) - files detected earliest get priority
|
|
2. **By Size** (smallest first) - smaller files processed faster
|
|
|
|
Example queue:
|
|
|
|
```
|
|
Order | Host | File | Age | Size | Status
|
|
------|-------|----------------|-------|---------|--------
|
|
1 | host2 | /etc/config | 2h | 2KB | Fetch now
|
|
2 | host1 | /app/db.sql | 90m | 50MB | Next
|
|
3 | host2 | /cache.tmp | 30m | 100MB | Waiting
|
|
4 | host1 | /data/doc.pdf | 20m | 500KB | Waiting
|
|
```
|
|
|
|
## Configuration
|
|
|
|
Edit `src/config.h`:
|
|
|
|
```c
|
|
/* RabbitMQ Server */
|
|
#define AMQP_HOST "127.0.0.1"
|
|
#define AMQP_PORT 5672
|
|
#define AMQP_USER "guest"
|
|
#define AMQP_PASSWORD "guest"
|
|
#define AMQP_VHOST "/"
|
|
|
|
/* Exchange and Queue */
|
|
#define AMQP_EXCHANGE "fsync.events"
|
|
#define AMQP_QUEUE_UPDATES "fsync.file.updates"
|
|
#define AMQP_ROUTING_KEY "fsync.file.*"
|
|
|
|
/* File Cache */
|
|
#define CACHE_DIR "/tmp/fsync-cache"
|
|
#define MAX_QUEUE_SIZE 1000
|
|
```
|
|
|
|
## Quick Start
|
|
|
|
### 1. Start Services
|
|
|
|
```bash
|
|
docker-compose up -d
|
|
make build
|
|
```
|
|
|
|
### 2. Run Monitor on Multiple Hosts
|
|
|
|
**Host A:**
|
|
```bash
|
|
sudo ./build/fsync-monitor
|
|
```
|
|
|
|
**Host B:**
|
|
```bash
|
|
sudo ./build/fsync-monitor
|
|
```
|
|
|
|
### 3. Test Cross-Host Sync
|
|
|
|
On Host A, modify a file:
|
|
```bash
|
|
echo "test data" >> /var/www/index.html
|
|
```
|
|
|
|
Host B will:
|
|
- Receive notification via RabbitMQ
|
|
- Add `/var/www/index.html` to fetch queue
|
|
- Fetch file from Host A via SCP
|
|
- Verify checksum
|
|
- Cache locally
|
|
|
|
### 4. Monitor Progress
|
|
|
|
**Check queue status:**
|
|
```bash
|
|
redis-cli LLEN fsync:events:list
|
|
```
|
|
|
|
**View cached files:**
|
|
```bash
|
|
ls -lh /tmp/fsync-cache/
|
|
```
|
|
|
|
**Monitor RabbitMQ:**
|
|
Open http://localhost:15672 (guest/guest)
|
|
|
|
## Operational Scenarios
|
|
|
|
### Scenario 1: Config File Update
|
|
|
|
```
|
|
Host1: /etc/app.conf modified
|
|
├─> Checksum: a1b2c3d4
|
|
└─> Publish to RabbitMQ
|
|
│
|
|
Host2: Receives notification
|
|
├─> Add to queue (priority 1)
|
|
├─> Fetch /etc/app.conf from Host1
|
|
├─> Verify checksum matches
|
|
└─> Cache at /tmp/fsync-cache/host1_app.conf
|
|
```
|
|
|
|
### Scenario 2: Bulk File Changes
|
|
|
|
```
|
|
1000 files modified on Host1
|
|
└─> 1000 messages to RabbitMQ
|
|
|
|
Host2 receives 1000 updates
|
|
└─> Add all to priority queue
|
|
│
|
|
Processing:
|
|
├─> Fetch oldest files first
|
|
├─> Then by size (KB before MB)
|
|
├─> One at a time sequentially
|
|
└─> Verify each checksum
|
|
```
|
|
|
|
### Scenario 3: Multiple Source Hosts
|
|
|
|
```
|
|
Host1 changes: /var/www/app.js
|
|
└─> Publish to RabbitMQ
|
|
|
|
Host2 changes: /etc/nginx.conf
|
|
└─> Publish to RabbitMQ
|
|
|
|
Host3 receives both notifications
|
|
└─> Queue both files
|
|
└─> Fetch oldest first
|
|
└─> Then by size
|
|
```
|
|
|
|
## File Transfer Mechanism
|
|
|
|
### SCP-Based Transfer
|
|
|
|
Current implementation uses `scp`:
|
|
|
|
```bash
|
|
scp -o ConnectTimeout=30 hostname:path local_path
|
|
```
|
|
|
|
**Requirements:**
|
|
- SSH key authentication configured
|
|
- SSH access between hosts
|
|
- Read permissions on source file
|
|
|
|
### Checksum Verification
|
|
|
|
After transfer:
|
|
1. Calculate SHA256 of fetched file
|
|
2. Compare with checksum from notification
|
|
3. If mismatch: retry (max 3 attempts)
|
|
4. If success: cache and record in Redis
|
|
|
|
## Redis Storage
|
|
|
|
### Published Events
|
|
|
|
```
|
|
Key: fsync:events:{timestamp}:{hostname}:{pid}
|
|
|
|
Example:
|
|
fsync:events:2026-05-18T13:30:00Z:server1:1234
|
|
├─ hostname: "server1"
|
|
├─ path: "/var/www/app.js"
|
|
├─ event_type: "MODIFY"
|
|
├─ checksum: "a1b2c3d4..."
|
|
└─ timestamp: "2026-05-18T13:30:00Z"
|
|
```
|
|
|
|
### Fetch Records
|
|
|
|
```
|
|
Key: fsync:fetches:success:{hostname}:{path}:{timestamp}
|
|
Key: fsync:fetches:failed:{hostname}:{path}:{timestamp}
|
|
```
|
|
|
|
## Performance
|
|
|
|
### Throughput
|
|
|
|
- **Event detection:** <1ms
|
|
- **Publishing:** ~5-10ms per event
|
|
- **File fetch:** Depends on size/network
|
|
- **Verification:** ~100-500ms per file
|
|
|
|
### Queue Processing
|
|
|
|
- Check interval: 1 second
|
|
- Process 1 file per 5 seconds (configurable)
|
|
- Network-bound for file transfer
|
|
|
|
### Memory Usage
|
|
|
|
- Queue: ~1-2KB per pending file
|
|
- Cache: Limited by disk space
|
|
- Connections: ~5MB per RabbitMQ connection
|
|
|
|
## Troubleshooting
|
|
|
|
### Queue Not Draining
|
|
|
|
```
|
|
[QUEUE] Added: host2:/etc/config (queue size: 100)
|
|
[FETCHER] Failed to fetch: host2:/etc/config
|
|
```
|
|
|
|
**Causes:**
|
|
1. SSH key not configured
|
|
2. File permissions on source
|
|
3. Network connectivity
|
|
4. Host not reachable
|
|
|
|
**Debug:**
|
|
```bash
|
|
# Test SSH access
|
|
ssh user@host2 'ls -la /etc/config'
|
|
|
|
# Check firewall
|
|
nc -zv host2 22
|
|
```
|
|
|
|
### Checksum Mismatch
|
|
|
|
```
|
|
[FETCHER] Checksum mismatch: /tmp/fsync-cache/host2_config
|
|
```
|
|
|
|
**Causes:**
|
|
1. File changed during transfer
|
|
2. Network corruption
|
|
3. Filesystem error
|
|
|
|
**Action:** Automatic retry (up to 3 times)
|
|
|
|
### RabbitMQ Connection Lost
|
|
|
|
```
|
|
Failed to open AMQP socket
|
|
```
|
|
|
|
**Fix:**
|
|
```bash
|
|
docker-compose logs rabbitmq
|
|
docker-compose restart rabbitmq
|
|
```
|
|
|
|
### File Transfer Timeout
|
|
|
|
```
|
|
[FETCHER] Failed to fetch after 3 attempts
|
|
```
|
|
|
|
**Solutions:**
|
|
1. Increase `FETCH_TIMEOUT_SEC` in config
|
|
2. Check network bandwidth
|
|
3. Check host load
|
|
|
|
## Advanced Configuration
|
|
|
|
### Custom Cache Directory
|
|
|
|
```c
|
|
#define CACHE_DIR "/var/lib/fsync-cache"
|
|
```
|
|
|
|
Persist fetched files across restarts.
|
|
|
|
### Parallel Fetching (Future)
|
|
|
|
Currently processes 1 file at a time. Can be extended to:
|
|
- Multi-threaded fetcher
|
|
- Batch SCP transfers
|
|
- Parallel checksums
|
|
|
|
### Routing Filters
|
|
|
|
Modify routing key to sync specific paths:
|
|
|
|
```c
|
|
#define AMQP_ROUTING_KEY "fsync.file.var.www.*" /* Only /var/www */
|
|
```
|
|
|
|
## Security Considerations
|
|
|
|
### SSH Authentication
|
|
|
|
- Configure SSH keys for password-less access
|
|
- Use key-based authentication
|
|
- Restrict SSH to internal network
|
|
|
|
### RabbitMQ
|
|
|
|
- Change default guest/guest credentials
|
|
- Use TLS for AMQP connections
|
|
- Restrict network access
|
|
|
|
### File Permissions
|
|
|
|
- Monitor has read access to synced files
|
|
- Cache directory writable by monitor
|
|
- Source files readable by SSH user
|
|
|
|
## Monitoring and Metrics
|
|
|
|
### Queue Depth
|
|
|
|
```bash
|
|
redis-cli LLEN fsync:events:list
|
|
```
|
|
|
|
### Fetch Success Rate
|
|
|
|
```bash
|
|
redis-cli KEYS "fsync:fetches:*" | wc -l
|
|
```
|
|
|
|
### Cache Size
|
|
|
|
```bash
|
|
du -sh /tmp/fsync-cache/
|
|
```
|
|
|
|
## Roadmap
|
|
|
|
- [x] Basic file sync via RabbitMQ
|
|
- [x] Priority queue (age + size)
|
|
- [x] Checksum verification
|
|
- [ ] Multi-threaded fetching
|
|
- [ ] Rsync support (resume capability)
|
|
- [ ] Compression for transfers
|
|
- [ ] Bandwidth throttling
|
|
- [ ] Web UI for monitoring
|
|
- [ ] Metrics export (Prometheus) |