first commit

docs/API.md

@@ -0,0 +1,239 @@
+# Fsync Redis Schema Documentation
+
+This document describes the Redis data structures used by the Fsync file system monitor.
+
+## Overview
+
+Fsync stores file system events in Redis with the following information:
+- **Hostname**: The server where the change occurred
+- **File Path**: The relative path of the changed file
+- **Event Type**: The type of file system event
+- **Checksum**: SHA256 checksum of file contents (for modify events)
+- **Metadata**: File size, permissions, timestamp, PID
+
+## Data Structures
+
+### Event Hash Keys
+
+Each file system event is stored as a Redis Hash with the following key format:
+
+```
+fsync:events:{timestamp}:{hostname}:{pid}
+```
+
+**Example:**
+```
+fsync:events:2026-05-18T13:28:51Z:server1:12345
+```
+
+### Event Hash Fields
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `hostname` | string | The hostname where the event occurred |
+| `path` | string | Absolute file path |
+| `event_type` | string | Type of event (OPEN, CLOSE_WRITE, MODIFY, CREATE, DELETE, MOVED_FROM, MOVED_TO, ATTRIB) |
+| `checksum` | string | SHA256 checksum of file contents (64 hex chars) |
+| `timestamp` | string | ISO8601 timestamp (2026-05-18T13:28:51Z) |
+| `file_size` | number | Size of file in bytes |
+| `permissions` | string | Octal permissions (e.g., "0644") |
+| `process_id` | number | PID of process that triggered the event |
+
+### Example Event
+
+```
+Key: fsync:events:2026-05-18T13:28:51Z:server1:1234
+Value:
+  hostname        "server1"
+  path            "/var/www/app.js"
+  event_type      "MODIFY"
+  checksum        "a1b2c3d4e5f6..."
+  timestamp       "2026-05-18T13:28:51Z"
+  file_size       "1024"
+  permissions     "0644"
+  process_id      "1234"
+```
+
+### Events List
+
+All event keys are added to a list for quick retrieval:
+
+```
+Key: fsync:events:list
+Type: List (LPUSH)
+```
+
+This allows efficient range queries of all recorded events.
+
+## Common Queries
+
+### Get All Events
+
+```bash
+redis-cli LRANGE fsync:events:list 0 -1
+```
+
+### Get Event Details
+
+```bash
+redis-cli HGETALL fsync:events:2026-05-18T13:28:51Z:server1:1234
+```
+
+### Get Events by Hostname
+
+```bash
+# Get list of all events
+redis-cli LRANGE fsync:events:list 0 -1
+
+# Filter by hostname using pattern
+redis-cli --pattern 'fsync:events:*:server1:*'
+```
+
+### Get Events by Path
+
+```bash
+# Scan and filter by path field
+redis-cli --scan --pattern 'fsync:events:*' | xargs redis-cli HGETALL
+```
+
+### Get Recent Events (Last 100)
+
+```bash
+redis-cli LRANGE fsync:events:list 0 99
+```
+
+### Get Events Modified in Last Hour
+
+```bash
+# Using Lua script or application-level filtering
+redis-cli LRANGE fsync:events:list 0 -1 | \
+  xargs -I {} redis-cli HGET {} timestamp | \
+  grep -E '(2026-05-18T1[2-3]:[0-5][0-9]:[0-5][0-9]Z)'
+```
+
+## Redis CLI Examples
+
+### Monitor Events in Real-time
+
+```bash
+redis-cli MONITOR | grep fsync:events
+```
+
+### Get Event Count
+
+```bash
+redis-cli LLEN fsync:events:list
+```
+
+### Delete Old Events
+
+```bash
+redis-cli EVAL "
+  local keys = redis.call('LRANGE', 'fsync:events:list', 0, -1)
+  for _, key in ipairs(keys) do
+    redis.call('DEL', key)
+  end
+" 0
+```
+
+### Export Events to JSON (with redis-cli)
+
+```bash
+redis-cli --json LRANGE fsync:events:list 0 -1
+```
+
+## Event Types
+
+| Event | Description |
+|-------|-------------|
+| `OPEN` | File was opened |
+| `CLOSE_WRITE` | File was closed after write |
+| `CLOSE_NOWRITE` | File was closed without write |
+| `MODIFY` | File contents were modified |
+| `CREATE` | File was created |
+| `DELETE` | File was deleted |
+| `MOVED_FROM` | File was moved (source) |
+| `MOVED_TO` | File was moved (destination) |
+| `ATTRIB` | File attributes changed (permissions, owner, etc.) |
+
+## Checksum Calculation
+
+SHA256 checksums are calculated for:
+- `MODIFY` events: Capture content changes
+- `CLOSE_WRITE` events: Verify final content
+- Other events: Empty hash or error indicator
+
+**Special Cases:**
+- Files > 100MB: Checksum = `0000000000000000000000000000000000000000000000000000000000000000`
+- Files that can't be read: Checksum = `error`
+- Directories/symlinks: No checksum (empty string)
+
+## Data Expiration
+
+Events are automatically expired after 30 days using Redis `EXPIRE` command.
+
+```
+EXPIRE fsync:events:{key} 2592000  # 30 days in seconds
+```
+
+## Performance Considerations
+
+1. **List Growth**: The `fsync:events:list` can grow large. Consider periodic cleanup or archival.
+2. **Hash Storage**: Each event stores ~1-2KB of data in Redis memory.
+3. **Network Events**: High-frequency events (millions/day) may benefit from batching.
+
+## Development Usage
+
+### Python Example
+
+```python
+import redis
+import json
+
+r = redis.Redis(host='localhost', port=6379, decode_responses=True)
+
+# Get all event keys
+event_keys = r.lrange('fsync:events:list', 0, -1)
+
+# Get event details
+for key in event_keys:
+    event = r.hgetall(key)
+    print(json.dumps(event, indent=2))
+```
+
+### Node.js Example
+
+```javascript
+const redis = require('redis');
+const client = redis.createClient();
+
+client.lrange('fsync:events:list', 0, -1, (err, keys) => {
+  keys.forEach(key => {
+    client.hgetall(key, (err, event) => {
+      console.log(event);
+    });
+  });
+});
+```
+
+## Troubleshooting
+
+### No Events Showing Up
+
+1. Verify Redis connection: `redis-cli PING` → `PONG`
+2. Check fsync-monitor is running: `ps aux | grep fsync-monitor`
+3. Verify file system activity: `touch /tmp/test.txt`
+4. Check Redis logs: `redis-cli INFO stats`
+
+### Memory Growing Too Large
+
+1. Check event count: `redis-cli LLEN fsync:events:list`
+2. Manually cleanup old events
+3. Consider archiving to external storage
+4. Reduce monitored paths to high-priority directories
+
+### Checksum Errors
+
+1. Verify file permissions: `ls -la /path/to/file`
+2. Check OpenSSL installation: `which openssl`
+3. Monitor disk space: `df -h`