fsync/docs/API.md

Fsync Redis Schema Documentation

This document describes the Redis data structures used by the Fsync file system monitor.

Overview

Fsync stores file system events in Redis with the following information:

• Hostname: The server where the change occurred
• File Path: The relative path of the changed file
• Event Type: The type of file system event
• Checksum: SHA256 checksum of file contents (for modify events)
• Metadata: File size, permissions, timestamp, PID

Data Structures

Event Hash Keys

Each file system event is stored as a Redis Hash with the following key format:

fsync:events:{timestamp}:{hostname}:{pid}

Example:

fsync:events:2026-05-18T13:28:51Z:server1:12345

Event Hash Fields

Field	Type	Description
hostname	string	The hostname where the event occurred
path	string	Absolute file path
event_type	string	Type of event (OPEN, CLOSE_WRITE, MODIFY, CREATE, DELETE, MOVED_FROM, MOVED_TO, ATTRIB)
checksum	string	SHA256 checksum of file contents (64 hex chars)
timestamp	string	ISO8601 timestamp (2026-05-18T13:28:51Z)
file_size	number	Size of file in bytes
permissions	string	Octal permissions (e.g., "0644")
process_id	number	PID of process that triggered the event

Example Event

Key: fsync:events:2026-05-18T13:28:51Z:server1:1234
Value:
  hostname        "server1"
  path            "/var/www/app.js"
  event_type      "MODIFY"
  checksum        "a1b2c3d4e5f6..."
  timestamp       "2026-05-18T13:28:51Z"
  file_size       "1024"
  permissions     "0644"
  process_id      "1234"

Events List

All event keys are added to a list for quick retrieval:

Key: fsync:events:list
Type: List (LPUSH)

This allows efficient range queries of all recorded events.

Common Queries

Get All Events

redis-cli LRANGE fsync:events:list 0 -1

Get Event Details

redis-cli HGETALL fsync:events:2026-05-18T13:28:51Z:server1:1234

Get Events by Hostname

# Get list of all events
redis-cli LRANGE fsync:events:list 0 -1

# Filter by hostname using pattern
redis-cli --pattern 'fsync:events:*:server1:*'

Get Events by Path

# Scan and filter by path field
redis-cli --scan --pattern 'fsync:events:*' | xargs redis-cli HGETALL

Get Recent Events (Last 100)

redis-cli LRANGE fsync:events:list 0 99

Get Events Modified in Last Hour

# Using Lua script or application-level filtering
redis-cli LRANGE fsync:events:list 0 -1 | \
  xargs -I {} redis-cli HGET {} timestamp | \
  grep -E '(2026-05-18T1[2-3]:[0-5][0-9]:[0-5][0-9]Z)'

Redis CLI Examples

Monitor Events in Real-time

redis-cli MONITOR | grep fsync:events

Get Event Count

redis-cli LLEN fsync:events:list

Delete Old Events

redis-cli EVAL "
  local keys = redis.call('LRANGE', 'fsync:events:list', 0, -1)
  for _, key in ipairs(keys) do
    redis.call('DEL', key)
  end
" 0

Export Events to JSON (with redis-cli)

redis-cli --json LRANGE fsync:events:list 0 -1

Event Types

Event	Description
OPEN	File was opened
CLOSE_WRITE	File was closed after write
CLOSE_NOWRITE	File was closed without write
MODIFY	File contents were modified
CREATE	File was created
DELETE	File was deleted
MOVED_FROM	File was moved (source)
MOVED_TO	File was moved (destination)
ATTRIB	File attributes changed (permissions, owner, etc.)

Checksum Calculation

SHA256 checksums are calculated for:

• MODIFY events: Capture content changes
• CLOSE_WRITE events: Verify final content
• Other events: Empty hash or error indicator

Special Cases:

• Files > 100MB: Checksum = 0000000000000000000000000000000000000000000000000000000000000000
• Files that can't be read: Checksum = error
• Directories/symlinks: No checksum (empty string)

Data Expiration

Events are automatically expired after 30 days using Redis EXPIRE command.

EXPIRE fsync:events:{key} 2592000  # 30 days in seconds

Performance Considerations

1. List Growth: The fsync:events:list can grow large. Consider periodic cleanup or archival.
2. Hash Storage: Each event stores ~1-2KB of data in Redis memory.
3. Network Events: High-frequency events (millions/day) may benefit from batching.

Development Usage

Python Example

import redis
import json

r = redis.Redis(host='localhost', port=6379, decode_responses=True)

# Get all event keys
event_keys = r.lrange('fsync:events:list', 0, -1)

# Get event details
for key in event_keys:
    event = r.hgetall(key)
    print(json.dumps(event, indent=2))

Node.js Example

const redis = require('redis');
const client = redis.createClient();

client.lrange('fsync:events:list', 0, -1, (err, keys) => {
  keys.forEach(key => {
    client.hgetall(key, (err, event) => {
      console.log(event);
    });
  });
});

Troubleshooting

No Events Showing Up

1. Verify Redis connection: redis-cli PING → PONG
2. Check fsync-monitor is running: ps aux | grep fsync-monitor
3. Verify file system activity: touch /tmp/test.txt
4. Check Redis logs: redis-cli INFO stats

Memory Growing Too Large

1. Check event count: redis-cli LLEN fsync:events:list
2. Manually cleanup old events
3. Consider archiving to external storage
4. Reduce monitored paths to high-priority directories

Checksum Errors

1. Verify file permissions: ls -la /path/to/file
2. Check OpenSSL installation: which openssl
3. Monitor disk space: df -h