fsync/docs/API.md

# Fsync Redis Schema Documentation

This document describes the Redis data structures used by the Fsync file system monitor.

## Overview

Fsync stores file system events in Redis with the following information:
- **Hostname**: The server where the change occurred
- **File Path**: The relative path of the changed file
- **Event Type**: The type of file system event
- **Checksum**: SHA256 checksum of file contents (for modify events)
- **Metadata**: File size, permissions, timestamp, PID

## Data Structures

### Event Hash Keys

Each file system event is stored as a Redis Hash with the following key format:

```
fsync:events:{timestamp}:{hostname}:{pid}
```

**Example:**
```
fsync:events:2026-05-18T13:28:51Z:server1:12345
```

### Event Hash Fields

| Field | Type | Description |
|-------|------|-------------|
| `hostname` | string | The hostname where the event occurred |
| `path` | string | Absolute file path |
| `event_type` | string | Type of event (OPEN, CLOSE_WRITE, MODIFY, CREATE, DELETE, MOVED_FROM, MOVED_TO, ATTRIB) |
| `checksum` | string | SHA256 checksum of file contents (64 hex chars) |
| `timestamp` | string | ISO8601 timestamp (2026-05-18T13:28:51Z) |
| `file_size` | number | Size of file in bytes |
| `permissions` | string | Octal permissions (e.g., "0644") |
| `process_id` | number | PID of process that triggered the event |

### Example Event

```
Key: fsync:events:2026-05-18T13:28:51Z:server1:1234
Value:
  hostname        "server1"
  path            "/var/www/app.js"
  event_type      "MODIFY"
  checksum        "a1b2c3d4e5f6..."
  timestamp       "2026-05-18T13:28:51Z"
  file_size       "1024"
  permissions     "0644"
  process_id      "1234"
```

### Events List

All event keys are added to a list for quick retrieval:

```
Key: fsync:events:list
Type: List (LPUSH)
```

This allows efficient range queries of all recorded events.

## Common Queries

### Get All Events

```bash
redis-cli LRANGE fsync:events:list 0 -1
```

### Get Event Details

```bash
redis-cli HGETALL fsync:events:2026-05-18T13:28:51Z:server1:1234
```

### Get Events by Hostname

```bash
# Get list of all events
redis-cli LRANGE fsync:events:list 0 -1

# Filter by hostname using pattern
redis-cli --pattern 'fsync:events:*:server1:*'
```

### Get Events by Path

```bash
# Scan and filter by path field
redis-cli --scan --pattern 'fsync:events:*' | xargs redis-cli HGETALL
```

### Get Recent Events (Last 100)

```bash
redis-cli LRANGE fsync:events:list 0 99
```

### Get Events Modified in Last Hour

```bash
# Using Lua script or application-level filtering
redis-cli LRANGE fsync:events:list 0 -1 | \
  xargs -I {} redis-cli HGET {} timestamp | \
  grep -E '(2026-05-18T1[2-3]:[0-5][0-9]:[0-5][0-9]Z)'
```

## Redis CLI Examples

### Monitor Events in Real-time

```bash
redis-cli MONITOR | grep fsync:events
```

### Get Event Count

```bash
redis-cli LLEN fsync:events:list
```

### Delete Old Events

```bash
redis-cli EVAL "
  local keys = redis.call('LRANGE', 'fsync:events:list', 0, -1)
  for _, key in ipairs(keys) do
    redis.call('DEL', key)
  end
" 0
```

### Export Events to JSON (with redis-cli)

```bash
redis-cli --json LRANGE fsync:events:list 0 -1
```

## Event Types

| Event | Description |
|-------|-------------|
| `OPEN` | File was opened |
| `CLOSE_WRITE` | File was closed after write |
| `CLOSE_NOWRITE` | File was closed without write |
| `MODIFY` | File contents were modified |
| `CREATE` | File was created |
| `DELETE` | File was deleted |
| `MOVED_FROM` | File was moved (source) |
| `MOVED_TO` | File was moved (destination) |
| `ATTRIB` | File attributes changed (permissions, owner, etc.) |

## Checksum Calculation

SHA256 checksums are calculated for:
- `MODIFY` events: Capture content changes
- `CLOSE_WRITE` events: Verify final content
- Other events: Empty hash or error indicator

**Special Cases:**
- Files > 100MB: Checksum = `0000000000000000000000000000000000000000000000000000000000000000`
- Files that can't be read: Checksum = `error`
- Directories/symlinks: No checksum (empty string)

## Data Expiration

Events are automatically expired after 30 days using Redis `EXPIRE` command.

```
EXPIRE fsync:events:{key} 2592000  # 30 days in seconds
```

## Performance Considerations

1. **List Growth**: The `fsync:events:list` can grow large. Consider periodic cleanup or archival.
2. **Hash Storage**: Each event stores ~1-2KB of data in Redis memory.
3. **Network Events**: High-frequency events (millions/day) may benefit from batching.

## Development Usage

### Python Example

```python
import redis
import json

r = redis.Redis(host='localhost', port=6379, decode_responses=True)

# Get all event keys
event_keys = r.lrange('fsync:events:list', 0, -1)

# Get event details
for key in event_keys:
    event = r.hgetall(key)
    print(json.dumps(event, indent=2))
```

### Node.js Example

```javascript
const redis = require('redis');
const client = redis.createClient();

client.lrange('fsync:events:list', 0, -1, (err, keys) => {
  keys.forEach(key => {
    client.hgetall(key, (err, event) => {
      console.log(event);
    });
  });
});
```

## Troubleshooting

### No Events Showing Up

1. Verify Redis connection: `redis-cli PING` → `PONG`
2. Check fsync-monitor is running: `ps aux | grep fsync-monitor`
3. Verify file system activity: `touch /tmp/test.txt`
4. Check Redis logs: `redis-cli INFO stats`

### Memory Growing Too Large

1. Check event count: `redis-cli LLEN fsync:events:list`
2. Manually cleanup old events
3. Consider archiving to external storage
4. Reduce monitored paths to high-priority directories

### Checksum Errors

1. Verify file permissions: `ls -la /path/to/file`
2. Check OpenSSL installation: `which openssl`
3. Monitor disk space: `df -h`