Overview
The file watcher system provides real-time visibility into all file system changes, enabling the Human-AI feedback loop through complete transparency of AI work.Copy
+==============================================================================+
| |
| FILE WATCHER SYSTEM OVERVIEW |
| |
+==============================================================================+
| |
| |
| +------------------+ |
| | FILE SYSTEM | |
| | (All Projects) | |
| +--------+---------+ |
| | |
| | File Events (create, modify, delete) |
| v |
| +------------------+ +-----------------+ |
| | File Watcher |----->| SQLite DB | |
| | (Rust Daemon) | | (Events Storage)| |
| | ~2.6MB binary | +-----------------+ |
| +--------+---------+ | |
| | | File Events Table |
| | Push Notifications | |
| v v |
| +------------------+ +-----------------+ |
| | Monolex App |----->| Work-Wiki | |
| | (Tauri) | | Transparency | |
| +------------------+ +-----------------+ |
| | | |
| | | AI Work Visible |
| v v |
| +--------------------------------------------------+ |
| | | |
| | HUMAN <> AI FEEDBACK LOOP | |
| | | |
| | AI Work --> Transparency --> Human Sees | |
| | ^ | | |
| | | v | |
| | +------- Prompt <---- Human Responds | |
| | | |
| +--------------------------------------------------+ |
| |
+==============================================================================+
Core Architecture
The file watcher is a standalone Rust daemon that operates independently of the main application. This design ensures reliability and performance.Copy
+==============================================================================+
| |
| COMPONENT ARCHITECTURE |
| |
+==============================================================================+
| |
| File Watcher Daemon (Rust) |
| | |
| +-- Entry Point Entry point, daemon lifecycle |
| | +-- Initializes logging |
| | +-- Gets platform paths |
| | +-- Checks daemon status (running/stale/none) |
| | +-- Writes lock files |
| | +-- Initializes database |
| | +-- Creates NotificationManager |
| | +-- Starts ControlServer |
| | |
| +-- File Watching File watching + ignore logic |
| | +-- 5-level priority filter |
| | +-- Event processing |
| | +-- ~200 ignore patterns |
| | |
| +-- Database Operations SQLite operations |
| | +-- Creates tables/indexes |
| | +-- Inserts events |
| | +-- WAL mode for concurrency |
| | |
| +-- Control Server Unix socket control server |
| | +-- Command processing |
| | +-- Status reporting |
| | +-- Client subscriptions |
| | |
| +-- Notifications Push notification system |
| | +-- Client management |
| | +-- Event broadcasting |
| | +-- Real-time delivery |
| | |
| +-- Platform Support Cross-platform path handling |
| | +-- macOS: ~/Library/Application Support/Monolex |
| | +-- Windows: %APPDATA%/Monolex |
| | +-- Linux: $XDG_CONFIG_HOME/monolex |
| | |
| TOTAL: ~2,060 lines of Rust |
| |
+==============================================================================+
Daemon Lifecycle
The daemon follows a strict initialization sequence to ensure reliability.Copy
+==============================================================================+
| |
| DAEMON LIFECYCLE STATE MACHINE |
| |
+==============================================================================+
| |
| |
| +------------+ |
| | START | |
| +-----+------+ |
| | |
| v |
| +------------------+ |
| | Initialize | |
| | Logging | |
| +--------+---------+ |
| | |
| v |
| +------------------+ |
| | Get Platform | |
| | Paths | |
| +--------+---------+ |
| | |
| v |
| +------------------+ +-------------+ |
| | Check PID |------->| DaemonStatus| |
| | File | +------+------+ |
| +------------------+ | |
| | |
| +----------------------------+---------------------------+ |
| | | | |
| v v v |
| +-----------+ +---------------+ +-------------+ |
| | None | | Running | | Stale | |
| | (proceed) | | (exit: 1) | | (cleanup) | |
| +-----+-----+ +---------------+ +------+------+ |
| | | |
| v | |
| +-----------+ +------------------------------------+ | |
| | Write Lock|<-------+ Cleanup Lock Files +---+ |
| | Files | |
| +-----+-----+ |
| | |
| v |
| +-----------+ |
| | Init DB | Initialize database |
| +-----+-----+ |
| | |
| v |
| +-----------+ |
| | Create | Notification manager |
| | NotifMgr | |
| +-----+-----+ |
| | |
| v |
| +-----------+ |
| | Ensure | Add default watch paths |
| | Paths | |
| +-----+-----+ |
| | |
| v |
| +-----------+ |
| | Start | Start watchers for each path |
| | Watchers | |
| +-----+-----+ |
| | |
| v |
| +-----------+ |
| | Control | Control socket server |
| | Socket | |
| | Server | |
| +-----+-----+ |
| | |
| v |
| +-----------+ |
| | RUNNING | Blocks until shutdown |
| +-----------+ |
| | |
| | Shutdown command received |
| v |
| +-----------+ |
| | CLEANUP | Remove lock files, close connections |
| +-----------+ |
| | |
| v |
| +-----------+ |
| | EXIT | |
| +-----------+ |
| |
+==============================================================================+
5-Level Priority Filter
The filtering system is crucial for reducing noise while capturing meaningful changes.Copy
+==============================================================================+
| |
| 5-LEVEL PRIORITY-BASED IGNORE SYSTEM |
| |
+==============================================================================+
| |
| File Change Event |
| | |
| v |
| +====================================================================+ |
| || || |
| || LEVEL 1: FILE EXTENSIONS (Highest Priority) || |
| || || |
| || Database: .db, .db-shm, .db-wal, .sqlite || |
| || Temp: .log, .tmp, .swp, .swo || |
| || Binary: .exe, .dll, .so, .dylib || |
| || Python: .pyc, .pyo || |
| || Java: .class, .jar || |
| || Rust: .rmeta, .rlib || |
| || Build: .tsbuildinfo, .eslintcache || |
| || || |
| || CRITICAL: Check compound extensions FIRST || |
| || Result: IGNORE if matched || |
| || || |
| +====================================================================+ |
| | pass |
| v |
| +====================================================================+ |
| || || |
| || LEVEL 2: APP BUNDLE & SYSTEM PATHS || |
| || || |
| || App Bundles: /Applications/*.app/ || |
| || Electron: electron.app, *.asar || |
| || System Caches: ~/Library/Caches/ || |
| || Temp Folders: /tmp/, /var/folders/ || |
| || Browsers: Firefox, Chrome, Safari, Arc || |
| || Cloud Sync: FileProvider, Dropbox, OneDrive || |
| || || |
| || Result: IGNORE if matched || |
| || || |
| +====================================================================+ |
| | pass |
| v |
| +====================================================================+ |
| || || |
| || LEVEL 3: BUILD/DEPENDENCY FOLDERS (~120 patterns) || |
| || || |
| || Node.js: node_modules/, dist/, build/, .next/ || |
| || Monorepo: .turbo/, .nx/, .pnpm-store/ || |
| || Rust: target/, debug/ || |
| || Python: __pycache__/, .venv/, venv/ || |
| || Go: vendor/ || |
| || Java: .gradle/, target/classes/ || |
| || .NET: bin/, obj/, packages/ || |
| || iOS: DerivedData/, Pods/ || |
| || IDE: .idea/, .vscode/, .vs/ || |
| || || |
| || Result: IGNORE if matched || |
| || || |
| +====================================================================+ |
| | pass |
| v |
| +====================================================================+ |
| || || |
| || LEVEL 4: DOTFILES/DOTFOLDERS (with Whitelist) || |
| || || |
| || Default: IGNORE all .* (dotfiles/dotfolders) || |
| || || |
| || +---------------------------------------------------------+ || |
| || | WHITELIST (Always Watch - AI Work Folders): | || |
| || | | || |
| || | /.niia/ - NIIA configuration | || |
| || | /.claude/ - Claude AI settings | || |
| || | /.agent/ - AI agent config | || |
| || | /.gemini/ - Gemini AI config | || |
| || | /.monolex/ - Monolex settings | || |
| || | | || |
| || | These folders tracked for Human <> AI visibility | || |
| || +---------------------------------------------------------+ || |
| || || |
| || Result: IGNORE unless whitelisted || |
| || || |
| +====================================================================+ |
| | pass |
| v |
| +====================================================================+ |
| || || |
| || LEVEL 5: USER-DEFINED EXCLUSIONS (from Database) || |
| || || |
| || Types: || |
| || folder: Pattern matches folder name || |
| || extension: Pattern matches file extension || |
| || || |
| || Result: IGNORE if matched || |
| || || |
| +====================================================================+ |
| | pass |
| v |
| +====================================================================+ |
| || || |
| || WATCH - Process the event || |
| || || |
| || +-- Write to SQLite (file_events table) || |
| || +-- Push notification to clients || |
| || +-- Update daily_stats || |
| || || |
| +====================================================================+ |
| |
+==============================================================================+
Database Schema
The database stores all file events for transparency and analysis.Copy
+==============================================================================+
| |
| FILE WATCHER DATABASE SCHEMA |
| |
+==============================================================================+
| |
| file_events (Core: stores all file change events) |
| Columns: id, ts, event_type, abs_path, rel_path, basename, |
| file_size, file_hash, is_directory, old_path, |
| correlation_id, created_at |
| |
| watch_paths (Which directories to monitor) |
| Columns: id, path, depth, active, ignore_patterns, |
| is_auto_added, created_at |
| |
| worker_status (Daemon health monitoring) |
| Columns: pid, start_time, last_ping, queue_size, |
| watcher_count, memory_usage, status, error_message |
| |
| daily_stats (Aggregated statistics) |
| Columns: date, total_events, create_count, change_count, |
| unlink_count, rename_count |
| |
| user_exclusions (Custom ignore patterns) |
| Columns: id, exclusion_type, pattern, created_at |
| |
| INDEXES: |
| +-- idx_file_events_ts (ts DESC) -- Time queries |
| +-- idx_file_events_path (abs_path) -- Path queries |
| +-- idx_file_events_type (event_type) -- Type filter |
| +-- idx_file_events_basename (basename) -- Filename |
| |
| PRAGMAS: |
| +-- journal_mode=WAL (Write-Ahead Logging for concurrency) |
| +-- busy_timeout=5000 (Wait 5s for locks) |
| |
+==============================================================================+
Push Notification System
Real-time event delivery enables immediate transparency.Copy
+==============================================================================+
| |
| PUSH NOTIFICATION FLOW |
| |
+==============================================================================+
| |
| File System |
| | |
| | Platform-specific event notification |
| v |
| +------------------+ |
| | Cross-platform | |
| | Watcher | |
| | (500ms debounce) | |
| +--------+---------+ |
| | |
| v |
| +------------------+ |
| | Event Handler | |
| +--------+---------+ |
| | |
| +----+----+ |
| | | |
| v v |
| +--------+ +---------------------+ |
| | SQLite | | NotificationManager | |
| | write | | | |
| +--------+ +----------+----------+ |
| | |
| v |
| +---------+---------+ |
| | clients: | |
| | Vec<Stream> | |
| | | |
| | [0]: Tauri App | |
| | [1]: CLI Tool | |
| | [2]: ... | |
| +---------+---------+ |
| | |
| +-----------------+------------------+ |
| | | | |
| v v v |
| +-------+ +-------+ +-------+ |
| |Client0| |Client1| |ClientN| |
| |(Tauri)| |(CLI) | |(...) | |
| +-------+ +-------+ +-------+ |
| |
| Message Format (JSON over Unix socket): |
| +----------------------------------------------------------------+ |
| | { | |
| | "type": "file-events-added", | |
| | "count": 3, | |
| | "events": [ | |
| | { | |
| | "ts": 1705500000000, | |
| | "event_type": "create", | |
| | "abs_path": "/project/src/main.rs", | |
| | "rel_path": "src/main.rs", | |
| | "size": 1024 | |
| | } | |
| | ] | |
| | } | |
| +----------------------------------------------------------------+ |
| |
| CRITICAL: stream.flush() after write_all() |
| Without flush, data may buffer and never reach client. |
| |
+==============================================================================+
Control Socket Protocol
The control socket provides daemon management and client subscriptions.Copy
+==============================================================================+
| |
| CONTROL SOCKET PROTOCOL |
| |
+==============================================================================+
| |
| Socket Path: |
| Unix: ~/.monolex/runtime/sockets/niia/watcher-control.sock |
| Windows: \\.\pipe\monolex-watcher-control |
| |
| COMMANDS: |
| |
| +====================================================================+ |
| | 1. shutdown | |
| |--------------------------------------------------------------------| |
| | Request: { "type": "shutdown" } | |
| | Response: { "success": true, "message": "Shutting down" } | |
| | Effect: Process exits with code 0 | |
| | Use case: Graceful daemon termination | |
| +====================================================================+ |
| |
| +====================================================================+ |
| | 2. status | |
| |--------------------------------------------------------------------| |
| | Request: { "type": "status" } | |
| | Response: { | |
| | "success": true, | |
| | "pid": 12345, | |
| | "uptime": 3600.0, | |
| | "watchers": 5 | |
| | } | |
| | Effect: Returns daemon statistics | |
| | Use case: Health check, monitoring | |
| +====================================================================+ |
| |
| +====================================================================+ |
| | 3. reload-paths | |
| |--------------------------------------------------------------------| |
| | Request: { "type": "reload-paths" } | |
| | Response: { "success": true, "message": "Reload triggered"} | |
| | Effect: Reloads watch_paths from database | |
| | Use case: After adding/removing watch paths via UI | |
| +====================================================================+ |
| |
| +====================================================================+ |
| | 4. subscribe | |
| |--------------------------------------------------------------------| |
| | Request: { "type": "subscribe" } | |
| | Response: { "success": true, "message": "Subscribed"} | |
| | Effect: Socket stays open for push notifications | |
| | Client receives file-events-added messages | |
| | Use case: Real-time file event streaming | |
| | | |
| | NOTE: After subscribe, socket ownership transfers to | |
| | NotificationManager. Client should read continuously. | |
| +====================================================================+ |
| |
+==============================================================================+
Cross-Platform Support
Platform-specific paths ensure consistent operation across operating systems.Copy
+==============================================================================+
| |
| PLATFORM-SPECIFIC PATHS |
| |
+==============================================================================+
| |
| +--------------------------------------------------------------------+ |
| | MACOS | |
| +--------------------------------------------------------------------+ |
| | | |
| | user_data: ~/Library/Application Support/Monolex | |
| | protocols: ~/Library/Application Support/Monolex/protocols | |
| | database: .../protocols/niia/database | |
| | runtime: .../protocols/niia/runtime | |
| | sockets: ~/.monolex/runtime/sockets/niia | |
| | | |
| +--------------------------------------------------------------------+ |
| |
| +--------------------------------------------------------------------+ |
| | WINDOWS | |
| +--------------------------------------------------------------------+ |
| | | |
| | user_data: %APPDATA%\Monolex | |
| | protocols: %APPDATA%\Monolex\protocols | |
| | database: %APPDATA%\Monolex\protocols\niia\database | |
| | runtime: %APPDATA%\Monolex\protocols\niia\runtime | |
| | sockets: %USERPROFILE%\.monolex\runtime\sockets\niia | |
| | control: \\.\pipe\monolex-watcher-control | |
| | | |
| +--------------------------------------------------------------------+ |
| |
| +--------------------------------------------------------------------+ |
| | LINUX | |
| +--------------------------------------------------------------------+ |
| | | |
| | user_data: $XDG_CONFIG_HOME/monolex (or ~/.config/monolex) | |
| | protocols: $XDG_CONFIG_HOME/monolex/protocols | |
| | database: .../protocols/niia/database | |
| | runtime: .../protocols/niia/runtime | |
| | sockets: ~/.monolex/runtime/sockets/niia | |
| | | |
| +--------------------------------------------------------------------+ |
| |
| WHY SHORT SOCKET PATH? |
| ===================== |
| |
| Unix sockets have path length limits (~104 chars). |
| macOS "~/Library/Application Support/..." is too long! |
| |
| Solution: Use ~/.monolex/runtime/sockets/ for all platforms. |
| This keeps paths short and consistent. |
| |
+==============================================================================+
THE CENTER: Transparency
The file watcher provides TRANSPARENCY - the foundation of the Human-AI feedback loop.Copy
+==============================================================================+
| |
| HOW FILE WATCHER SERVES THE HUMAN <> AI FEEDBACK LOOP |
| |
+==============================================================================+
| |
| THE CENTER: "Building an environment where Human <> AI |
| feedback loop can operate" |
| |
| File Watcher provides TRANSPARENCY - first element of the loop: |
| |
| +----------------------------------------------------------------+ |
| | | |
| | 1. TRANSPARENCY (File Watcher's Primary Role) | |
| | +-- Every file created by AI is captured in real-time | |
| | +-- Every modification is logged with timestamp | |
| | +-- Every deletion is recorded | |
| | +-- AI work becomes VISIBLE | |
| | | |
| | 2. ENABLING VISUALIZATION (Work-Wiki consumes this data) | |
| | +-- Events table provides the raw material | |
| | +-- Timestamps enable timeline reconstruction | |
| | +-- Path information enables project context | |
| | | |
| | 3. ENABLING FEEDBACK (Human can respond) | |
| | +-- Real-time push notifications alert humans | |
| | +-- Humans can see what AI modified immediately | |
| | +-- Humans can respond via prompt to correct/direct | |
| | | |
| +----------------------------------------------------------------+ |
| |
| WITHOUT FILE WATCHER: |
| +-- AI work happens invisibly |
| +-- Human cannot see what changed |
| +-- Feedback loop is blind |
| +-- Unity is impossible |
| |
| WITH FILE WATCHER: |
| +-- Every AI action is recorded |
| +-- Human can see, understand, respond |
| +-- Feedback loop operates |
| +-- Unity becomes possible |
| |
| File Watcher is the foundation of this loop. |
| Without visibility, there is no unity. |
| |
+==============================================================================+
Key Technical Achievements
- 95% less CPU, 91% less memory compared to Node.js version
- 5-level priority-based ignore system filtering ~200 patterns
- Direct SQLite writes eliminating IPC overhead
- Push notification system for real-time event delivery
- Cross-platform support (macOS, Windows, Linux)