## 1. Project Overview & Quickstart (dwarvesf/hidden) # Technical Documentation: dwarvesf/hidden > โ„น๏ธ **Provenance:** Multi-Tier Ingestion ยท [CodeWiki Reference](https://codewiki.google/github.com/dwarvesf/hidden) ### Overview (`README.md`)

download platform systemrequirements

## Hidden Bar Hidden Bar lets you hide menu bar items to give your Mac a cleaner look.

## ๐Ÿš€ Install ### ๏ฃฟ App Store [](https://itunes.apple.com/app/hidden-bar/id1452453066) ### Others The Hidden Bar is notarized before distributed out side App Store. It's safe to use ๐Ÿ‘ #### Using Homebrew ``` brew install --cask hiddenbar ``` #### Manual download - [Download latest version](https://github.com/dwarvesf/hidden/releases/latest) - Open and drag the app to the Applications folder. - Launch Hidden and drag the icon in your menu bar (hold CMD) to the right so it is between some other icons. ## ๐Ÿ•น Usage * `โŒ˜` + drag to move the Hidden icons around in the menu bar. * Click the Arrow icon to hide menu bar items.

## ๐Ÿ“š Documentation - [Manual](docs/MANUAL.md): every setting, the hidden Terminal-only options, and troubleshooting. - [Architecture](docs/ARCHITECTURE.md): how the hiding trick works, topology, and known limits. - [Maintainer runbook](docs/RUNBOOK.md): build, behavioral verification, and release process. ## โœจContributors This project exists thanks to all the people who contribute. Thank you guys so much ๐Ÿ‘ [](https://sourcerer.io/fame/phucledien/dwarvesf/hidden/links/0)[](https://sourcerer.io/fame/phucledien/dwarvesf/hidden/links/1)[](https://sourcerer.io/fame/phucledien/dwarvesf/hidden/links/2)[](https://sourcerer.io/fame/phucledien/dwarvesf/hidden/links/3)[](https://sourcerer.io/fame/phucledien/dwarvesf/hidden/links/4)[](https://sourcerer.io/fame/phucledien/dwarvesf/hidden/links/5)[](https://sourcerer.io/fame/phucledien/dwarvesf/hidden/links/6)[](https://sourcerer.io/fame/phucledien/dwarvesf/hidden/links/7) Please read [this](CONTRIBUTING.md) before you make a contribution. ## Requirements macOS version >= 13.0 (Ventura) Running an older macOS? The last release supporting macOS 10.13 High Sierra through 12 Monterey is [v1.10](https://github.com/dwarvesf/hidden/releases/tag/v1.10). Later versions require macOS 13 because autostart moved to the `SMAppService` API introduced there. ## You may also like - [Blurred](https://github.com/dwarvesf/Blurred) - A macOS utility that helps reduce distraction by dimming your inactive noise - [Micro Sniff](https://github.com/dwarvesf/micro-sniff) - An ultra-light macOS utility that notify whenever your micro-device is being used - [VimMotion](https://github.com/dwarvesf/VimMotionPublic) Vim style shortcut for MacOS ## License MIT © [Dwarves Foundation](https://github.com/dwarvesf) --- ### ARCHITECTURE (`docs/ARCHITECTURE.md`) # Architecture Hidden Bar is a single-process, sandboxed AppKit menubar utility (~1.5k lines of Swift, one dependency: [HotKey](https://github.com/soffes/HotKey)). There is no helper app, no daemon, no network. Everything happens inside three `NSStatusItem`s and one window. ## The core trick macOS offers no API to hide other apps' menubar icons. Hidden Bar fakes it with geometry: a separator status item whose **length is inflated to roughly the width of the widest attached screen**, shoving every icon to its left off-screen. Collapsing and expanding is just flipping that length between ~20pt and the inflated value. ```mermaid flowchart LR subgraph menubar [menu bar, right to left] direction RL ARROW["arrow item\n(toggle, variable length)"] SEP["separator item\n20pt expanded / ~2x screen width collapsed"] HIDDEN["other apps' icons\npushed off-screen when collapsed"] ALWAYS["always-hidden separator\n(optional)"] end ARROW --- SEP --- HIDDEN --- ALWAYS ``` Key consequences of this design: - The menubar replicates on every display, so the collapse length derives from the **widest** screen (`NSScreen.screens`), never `NSScreen.main`, and is re-applied to the live item on `didChangeScreenParametersNotification` (display hot-plug). - The length is bounded: `max(500, min(widestFrameWidth * 2, 10_000))`. macOS enforces a hard 10,000pt maximum on `NSStatusItem.length`. - `isCollapsed` is derived state: `separator.length > 20`, deliberately not an equality check, so it survives the length being recomputed while collapsed. - Icons macOS inserts to the LEFT of the separator (where new status items appear) are in the hidden zone by default; that is inherent to the trick. ## Topology ```mermaid flowchart TD AD[AppDelegate] -->|owns| SBC[StatusBarController] AD -->|registers| SM["SMAppService.mainApp\n(login item, macOS 13+)"] AD -->|global hotkey| HK[HotKey lib] SBC -->|3 status items| NSB[NSStatusBar.system] SBC -->|auto-hide| T["one-shot Timer\n(re-arms while pointer in menubar)"] SBC -->|opt-in| HM["global mouseMoved monitor\n(hover-to-expand, only if pref on)"] PREFS[Preferences facade] -->|UserDefaults| UD[(UserDefaults)] PREFS -->|posts| NC{{NotificationCenter\n.prefsChanged / .alwayHideToggle}} NC --> SBC PVC[PreferencesViewController] --> PREFS SBC -->|context menu| PVC ``` - **`AppDelegate`** (entry): registers default prefs, sets up the global hotkey, runs the one-shot legacy login-item migration, owns the `StatusBarController`. - **`StatusBarController`** (the product, ~370 lines): the three status items, when to collapse/expand, auto-hide timer, interaction-awareness, hover-to-expand, self-restore of dragged-off items. - **`MenuBarEngine`** (`Features/StatusBar/Engine/`): how hiding is achieved. The controller calls `collapse` / `expand` / `updateAlwaysHiddenSection` / `invalidateLayout` and never writes a status-item length itself. `MenuBarEngineFactory` is the one place that picks an engine for the running OS; today that is always `LegacyLengthEngine` (the length trick below). - **`Preferences`** (facade enum): typed accessors over `UserDefaults`; setters post `NotificationCenter` notifications that the controller and prefs window observe. There is no other state store. - **`PreferencesViewController` / `PreferencesWindowController`**: the only window (storyboard-based), shown on demand from the context menu. ## Behavior layers on the core trick | Layer | Mechanism | Cost when unused | |---|---|---| | Auto-hide | one-shot `Timer` after expand; at fire, if the pointer sits in any screen's menubar band (`visibleFrame.maxY ... frame.maxY`), it re-arms instead of collapsing | none (single point-in-rect check at fire) | | Hover-to-expand (opt-in) | global `.mouseMoved` monitor + 0.5s dwell timer; installed only when the `hoverToExpand` default is true at launch | zero: monitor not installed | | Self-restore | `isVisible = true` forced on our items at launch; Cmd-dragging them off otherwise bricks the app (its only UI is those items) | none | | Always-hidden section | a second separator item; its own length games, gated by `alwaysHiddenSectionEnabled` | item not created | ## Autostart macOS 13+ `SMAppService.mainApp`: the app registers itself; the login item is visible and revocable in System Settings > General > Login Items. On first launch after upgrade, a one-shot migration deauthorizes the legacy `com.dwarvesv.LauncherApplication` helper registration (the BTM database never garbage-collects those; see Apple TN3111). The helper app itself is gone. ## Security posture Sandboxed (`com.apple.security.app-sandbox`), hardened runtime, no network entitlement, no file I/O, no IPC surface, no shell or subprocess use. The only dependency is HotKey (a small Carbon `RegisterEventHotKey` wrapper) locked by the committed `Package.resolved`. The opt-in hover monitor observes pointer position only and discards event payloads. About-window links are hardcoded. A full-tree audit (2026-06) scored 9/10 with hygiene-level findings only. ## Known architectural limits - **The notch**: hidden icons sit "under" the notch area on notched Macs; the trick cannot reveal them there. The real fix is a spillover/second-bar design (tracked in issues #357/#341/#148; candidate implementations in PRs #350/#358). - **macOS 27**: the menu bar re-architecture in macOS 27 betas (`NSMenuBarNavigationSceneExtension`) breaks length-inflation hiding entirely (issue #360). A different mechanism may be required. - **Other apps' open menus**: interaction-awareness is pointer-position-based; a pointer deep inside another app's open dropdown is below the menubar band, so the collapse can still fire there. --- ### BACKLOG (`docs/BACKLOG.md`) # Backlog Durable, committed index of open work. The detailed planning layer (`_meta/megagoals/issue-list-clearing/` and `.claude/`) is local-only and gitignored, so this file is the version that survives a machine switch and a fresh clone. Keep it short: one line per item, pointing at the issue, SPEC, or file that holds the detail. Source of the current state: the v1.11 issue-clearing pass (2026-06-12), branch `fix/v1-11-batch` / draft PR #365, and SPEC-003. Core-model changes (separator length math, collapse state machine) are HIGH RISK and require a mandatory review-team pass. ## macOS 27 (#360) - **Direct build ships native hiding** (`NativeVisibilityEngine`, Debug-Direct / Release-Direct configurations). Uses private `MenuBarClientCore`; not App Store safe. - **App Store build still broken on 27.** The sandbox denies `mach-lookup com.apple.axserver`, so the sections cannot be read even with Accessibility granted (measured 2026-09-18). Options: a user-picked list of visible apps instead of โŒ˜-drag (no Accessibility, still private API), or a public mechanism if Apple ships one. - **Direct build bundle id.** Tested as `com.dwarvesv.minimalbar.direct`; decide whether release keeps `com.dwarvesv.minimalbar` (shared prefs, cannot coexist with the App Store copy) or ships separately. - **Tutorial text** still describes the `|` separator; the direct build on 27 uses the arrow as the boundary. Needs a per-build string. - **Adaptive-width experiment** (`fix/macos27-adaptive-engine`, local only): calibrating the separator width does not hide on a 1920pt non-notched display. Kept for reference. ## Blocked on external-display hardware - **#351 external-monitor visible-bar (26.4) verification.** The widest-attached-screen fix is code-only; the 26.4 + external-display reproduction was never run. Confirm no full-width bar leak on a real second monitor. ## Actionable now (no special hardware) - **UAT + merge draft PR #365.** Per the local `UAT.md` (one row per fix, click-through steps). Merge order matters (stacked dependency). Merge is Han's action, not the agent's. - **Cut the v1.11 release.** Version + CHANGELOG staged in #365. Needs a Developer ID for notarization + App Store submission. Shipping answers the "is this still maintained?" issues and unblocks the round-2 close sweep. - **Upgrade-path BTM verification before any signed release.** Install a pre-v1.11 build, update to v1.11, confirm Login Items shows no leftover `LauncherApplication` row (the one-shot `SMLoginItemSetEnabled(..., false)` deauth was added but never run on hardware). - **AXPress accessibility defect.** VoiceOver users cannot toggle the bar: the arrow's `AXPress` handler reads `NSApp.currentEvent`, which is nil under assistive synthesis. - **`hoverToExpand` Preferences checkbox.** Shipped as a Terminal-only `defaults write`; add a proper checkbox in `PreferencesViewController`. - **Surface the `SMAppService` error contract in the prefs UI.** On `register()` failure (unsigned build, or user denies in System Settings), the checkbox stays on while the system says off; the error is swallowed into NSLog. Fix when the prefs UI is next touched. `Common/Util.swift:29-31`. - **Round-2 issue close sweep (~35 issues).** Obsolete-OS + meta/support candidates, deferred until v1.11 ships so the closures carry the strongest answer. - **24h memory dogfood (#361).** Instruments stress-cycling found no leak; run v1.11 as the daily driver for 24h on the Air to close the open report. - **Old branch decision.** `feature/menubarDetection` (PR #115) and `feature/ghost-mode` (PR #57) were kept per never-delete. Review or discard, Han's call. ## v1.12 standalone wins (no Option D needed) - **#207 show clock/date when collapsed.** Loudest single feature ask (18+ comments). - **#355** prefs window layout overlap. **#324** single-instance guard. **#276** Cmd+W closes the prefs window. Appearance bundle via community PR #194. ## Architectural epic (Option D, #366) - **Managed-overflow / second-bar redesign.** The real fix that gates ~30 issues across four clusters: icon-drift (#28, #156, #181, #230, #231, #239, #252, #254, #275, #283, #321, #334), always-hidden (#171, #224, #242, #288), notch (#206, #225, #228, #245, #267, #269, #280, #292, #330), and macOS 27 (#360). Replaces length-inflation with a managed overflow bar, likely via Accessibility. Needs a design decision from Han + macOS 27 hardware. Design + folded M1 (pin icons, persist order) / M2 (decouple always-hidden from `areSeparatorsHidden`, recover stuck items) in SPEC-003. - **Security + behavior review of community PRs #358 and #350 first.** #358 (second bar, +1160 lines) and #350 (notch overflow, +429 lines) are the existing starting points. Do not merge on description alone; #358 especially needs a real review. - **#242 permanent icon-loss repro.** Needs a throwaway defaults profile (live repro risks losing real menu-bar icons). Required before the always-hidden decouple lands. --- ### MANUAL (`docs/MANUAL.md`) # Manual Everything Hidden Bar can do, including the parts with no UI. ## Basics - **Left-click the arrow** (`<` / `>`): expand or collapse the hidden section. - **Right-click the arrow** or **left-click the separator** (`|`): context menu (Preferences, Toggle Auto Collapse, Quit). - **โŒ˜-drag** icons in the menu bar to move them across the separator: icons to the separator's left are hidden when collapsed. - **Option-click the arrow**: show/hide the separators and the always-hidden area without expanding. ### macOS 27, direct download build The direct (non-App Store) build hides icons natively on macOS 27, and the **arrow itself is the boundary**: there is no `|` separator. โŒ˜-drag icons to the arrow's left to hide them when collapsed, to its right to keep them visible. The always-hidden separator still appears (while expanded) when that section is enabled. - Needs **Accessibility** permission (System Settings โ€บ Privacy & Security โ€บ Accessibility) to see where your icons are. Without it the bar stays expanded. - Hiding is per app: an app with several icons hides or shows them together. - macOS's own items (clock, Wi-Fi, Sound, Control Center) always stay visible. - Run Hidden Bar from `/Applications`; macOS only recognizes that copy as Hidden Bar, and a copy elsewhere hides its own arrow. ## Preferences window | Setting | What it does | |---|---| | Start Hidden Bar when I log in | Login item via System Settings (macOS 13+ `SMAppService`); revocable in System Settings > General > Login Items | | Show preferences on launch | Open this window at app start | | Auto collapse | Re-hide automatically after the chosen delay | | Global shortcut | System-wide expand/collapse hotkey (F-keys display as F18, not Fn18) | | Enable always hidden section | A second zone whose icons stay hidden even when expanded; revealed by option-clicking the arrow | | Use full menu bar on expanding | App becomes briefly "regular" while expanded (helps on tight menubars) | > **Always-hidden section, current behavior:** items in the always-hidden zone > are reliably pushed off-screen only when "hide separators" is also on > (option-click the arrow). With the separators visible, always-hidden items can > still appear after expanding. This coupling is a known limitation being > reworked alongside the menu-bar redesign; for now, option-click to hide the > separators if always-hidden items keep showing. Avoid placing critical icons in > the always-hidden zone until the rework lands, since a stuck off-screen item has > to be recovered by โŒ˜-dragging it back (macOS persists its position per app). ## Behaviors you get for free - **It won't collapse mid-use**: while your pointer is anywhere in the menu bar, the auto-collapse countdown defers and restarts; it resumes when you leave. - **Self-repair**: if the arrow or separator was โŒ˜-dragged off the bar (which used to make the app unreachable forever), they come back on next launch. - **Display changes**: plugging in or removing monitors re-sizes the hidden zone for the widest attached screen automatically. ## Hidden settings (Terminal) All via `defaults`; quit and relaunch the app after changing them. ```sh # expand by hovering the menu bar for ~0.5s (off by default) defaults write com.dwarvesv.minimalbar hoverToExpand -bool true # auto-collapse delay in seconds (the UI offers a fixed list; any value works) defaults write com.dwarvesv.minimalbar numberOfSecondForAutoHide -float 5 # force the app language regardless of system order (issue #287) defaults write com.dwarvesv.minimalbar AppleLanguages '(en)' ``` To undo any of them: `defaults delete com.dwarvesv.minimalbar `. ## Troubleshooting | Symptom | Fix | |---|---| | Icons I want visible got hidden after an update | โŒ˜-drag them to the right of the separator | | Login item missing after denying it once | System Settings > General > Login Items: re-enable Hidden Bar, then toggle the pref off/on | | A ghost "LauncherApplication" login item from old versions | Launch the current version once; it deauthorizes the legacy item automatically | | App language stuck | See the `AppleLanguages` command above, or System Settings > General > Language & Region > Applications | | Nothing hides on a macOS 27 beta | Known (issue #360); the menu bar re-architecture broke the hiding mechanism, fix under investigation | | A new or just-updated app's icon shows up already hidden | Expected, see "Why new icons start hidden" below; โŒ˜-drag it to the right of the separator once | ### Why new icons start hidden Hidden Bar hides icons by widening its separator so everything to the *left* of it slides off-screen. macOS always inserts a brand-new menu-bar icon at the far-left slot, which is inside that hidden zone, so a freshly launched or updated app can appear "swallowed". This is macOS positioning behavior, not Hidden Bar moving your icon: there is no way for one app to reposition another app's menu-bar icon. The one-time fix is to โŒ˜-drag the icon to the right of the separator; macOS remembers that placement per app. A built-in way to keep chosen icons pinned is being explored as part of the larger menu-bar redesign. ## Requirements macOS 13 Ventura or later. Pre-Ventura (10.13 - 12.x): use [v1.10](https://github.com/dwarvesf/hidden/releases/tag/v1.10), the last release on the old autostart mechanism. --- ### RUNBOOK (`docs/RUNBOOK.md`) # Maintainer runbook Build, verify, and release Hidden Bar. Assumes current Xcode on macOS 13+. ## Build ```sh # CI-style build, no signing required xcodebuild -project 'Hidden Bar.xcodeproj' -scheme 'Hidden Bar' \ -configuration Debug CODE_SIGNING_ALLOWED=NO build # runnable local build (uses your Apple Development identity) xcodebuild -project 'Hidden Bar.xcodeproj' -scheme 'Hidden Bar' \ -configuration Debug build ``` Both must exit 0 with **no** `MACOSX_DEPLOYMENT_TARGET` override; the floor is 13.0 in the project file. The single expected warning is the deliberately deprecated `SMLoginItemSetEnabled(false)` call that cleans up legacy installs. ## Behavioral verification (no test target exists; this is the methodology) The repo has no unit-test infrastructure; behavior is verified against the real menu bar. Two building blocks make that scriptable: 1. **Truth signal**: the separator's AX size. `osascript -e 'tell application "System Events" to tell process "Hidden Bar" to get size of menu bar item 2 of menu bar 2'` reads ~20pt expanded vs ~2x-screen-width collapsed. Item 1 is the arrow. 2. **Real clicks, not AXPress**: `AXPress` on the arrow is a no-op because the action handler reads `NSApp.currentEvent` (nil under assistive synthesis; known accessibility defect). Post real `CGEvent` mouse clicks at the arrow's AX-reported coordinates instead. Standard checks before any release: - expand/collapse toggle flips the separator size both ways; - with `numberOfSecondForAutoHide` set to 3, an expanded bar survives 2x the window while the pointer is parked in the menubar band, collapses within the window once the pointer leaves, and collapses normally if the pointer never entered; - `hoverToExpand` off: no monitor log line, hover does nothing; on: dwell expands (synthesize a `mouseMoved` stream: cursor warping alone fires no events); - autostart pref on -> `AutoStart: SMAppService.mainApp.status = 1` on stderr; off -> `= 0` (run the binary directly to capture stderr); - localization tables stay parseable: `plutil -lint hidden/*.lproj/*.strings`. When testing on a machine that runs Hidden Bar daily: export the prefs domain first (`defaults export com.dwarvesv.minimalbar backup.plist`), quit the installed app, test the dev build, then re-import and relaunch. ## Release 1. Verify the stack: every PR reviewed, builds green, behavioral checks above run. 2. Bump `MARKETING_VERSION` (all four Hidden Bar configurations) and `CURRENT_PROJECT_VERSION` in the project file. 3. Archive the GitHub build with the **Hidden Bar** scheme (`Release-Direct`), then notarize (Developer ID), staple, zip, and publish a GitHub release with notes listing closed issues. The **Hidden Bar App Store** scheme archives `Release`, the sandboxed App Store build, which cannot hide on macOS 27. v1.11 shipped that build to GitHub by mistake. Before zipping, confirm the app is unsandboxed: ```sh codesign -d --entitlements - 'Hidden Bar.app' | grep app-sandbox # must print nothing ``` 4. App Store (separate lane, **Hidden Bar App Store** scheme): the MAS listing has lagged GitHub since v1.8 (issues #281/#202); decide deliberately whether a release goes there too. 5. Homebrew cask (`brew install --cask hiddenbar`) follows the GitHub release artifact; notarization matters (issue #219). 6. Before any public release after the SMAppService migration: one upgrade-path run from a launcher-era build (v1.9 or older) verifying System Settings shows no ghost LauncherApplication login item. ## Stacked-PR hygiene Feature stacks land as chained PRs (`develop <- A <- B`). Merge the base PR first, then retarget the next PR's base to `develop` BEFORE merging it; deleting a merged branch auto-closes dependents otherwise. ## Issue triage map The live clusters and their code areas are documented at the end of [ARCHITECTURE.md](ARCHITECTURE.md#known-architectural-limits): notch, macOS 27 mechanism break, and pointer-vs-open-menu limits. Memory reports (#361 et al.) have so far not reproduced as leaks (constraint-leak fix landed; `leaks` clean over toggle stress); re-check with a 24h+ uptime `footprint` sample before chasing further. --- --- METRICS --- - Files Extracted: 5 - Estimated Token Budget: ~6062 tokens - Canonical Reference: https://codewiki.google/github.com/dwarvesf/hidden ## 2. Official Technical Reference & Guides (dwarvesf/dwarvesf.github.io) ## File: docs/changelog/0001-refactor-static-page-generation-and-paths.md ## Refactor: Static Page Generation and Path Resolution This update introduces a significant refactoring of the system that handles static page generation, aliases, and redirects. The primary goal was to simplify logic, improve maintainability, and ensure consistent path resolution across the application. **Key Changes & Improvements:** - **Unified Path Resolution:** Introduced a new centralized function, `getStaticJSONPaths` (in `src/lib/content/paths.ts`). This function now serves as the single source of truth for mapping browser-accessible URLs to their corresponding canonical content file paths. It intelligently combines direct markdown file paths, reversed alias mappings, and filtered redirect rules. - **Simplified Page Generation Logic:** The `getStaticPaths` and `getStaticProps` functions within `src/pages/[...slug].tsx` have been substantially simplified. They now rely entirely on `getStaticJSONPaths` to determine valid static paths and to resolve requested URLs to the correct content files. This reduces complexity and potential inconsistencies. - **Removal of Unused Static Pages:** As a direct result of the more precise path resolution provided by `getStaticJSONPaths`, pages that are not explicitly defined or resolvable through this new system are no longer generated. This helps in keeping the build lean and focused on relevant content. - **Robust Redirect and Alias Handling:** The logic for generating Nginx redirects (`getNginxRedirects`) and other redirect mappings (`getRedirectsNotToAliases` in `scripts/common.ts`) has been refactored. This ensures that redirects are handled more robustly, especially in relation to aliases, preventing conflicts and ensuring predictable behavior. - **Updated Changelog URL:** The navigation link for the Changelog in the sidebar (`src/components/layout/Sidebar.tsx`) has been updated from `/updates/changelog` to `/changelog` for a cleaner URL structure. --- ## File: docs/changelog/0002-unify-url-path-resolution-and-redirects.md ## Unified URL Path Resolution and Redirect Handling This update introduces a comprehensive system for managing URL path resolution and redirects throughout the application, ensuring consistency and improving user experience. Key changes include: - **Unified Redirect System**: Implemented a centralized mechanism to handle URL redirects. This ensures that users and search engines are consistently directed to the canonical versions of pages, improving SEO and reducing confusion from multiple URLs pointing to the same content. - **Client-Side Redirects**: A map of all known redirects is now injected into the `window` object on page load. This allows the client-side router to immediately redirect to the correct path without a server roundtrip, speeding up navigation for aliased or moved content. - **Server-Side Path Normalization**: Enhanced server-side logic to correctly resolve and normalize paths, especially for markdown content. This includes better handling of relative links within markdown files, ensuring they point to the correct, possibly redirected, final URL. - **Backlink Accuracy**: Improved the backlink generation process to account for redirects. Backlinks will now correctly point to the canonical URL of a page, even if the link in the source document uses an alias or an old path. - **Path Utilities**: Introduced new utility functions for normalizing paths (e.g., ensuring consistent trailing slashes, handling special characters) and for resolving paths on both client and server sides. These changes affect several parts of the application: - **Command Palette**: Navigation from the command palette now uses the client-side redirect logic to ensure users land on the correct page. - **Markdown Link Processing**: Links within markdown content are now resolved considering the unified redirect map, ensuring they are always up-to-date. - **Static Page Generation (`[...slug].tsx`)**: Page generation logic now correctly identifies canonical paths and incorporates redirect information for folder structures and backlinks. - **Application Entry (`_document.tsx`)**: Modified to inject the redirect map for client-side use. Overall, this refactor streamlines how URLs are handled, making the application more robust and user-friendly. --- ## File: docs/changelog/0003-fix-readme-redirects-nginx.md # Fix: README.md Redirects Missing from Nginx Configuration ## Problem For example: - `/research/topics/pm/readme.md` should redirect to `/research/topics/pm` - `/playbook/design/readme.md` should redirect to `/playbook/design` ## Root Cause The issue was in the `filterRedirectsLogic` function in `scripts/common.ts`. This function filters out redirects that match alias patterns to avoid conflicts. However, it was too aggressive and filtered out legitimate README.md redirects when their target paths matched alias patterns. The problematic logic checked if: 1. The redirect target matched any alias key/value 2. The redirect source matched any alias pattern ## Solution Modified the `filterRedirectsLogic` function in `scripts/common.ts` to include a special case for README.md redirects: 1. Added detection for README.md redirects using `normalizedRedirectKey.toLowerCase().endsWith('/readme.md')` 2. Modified the alias matching logic to skip filtering for README.md redirects 3. Ensured README.md redirects are always processed, regardless of alias conflicts ## Code Changes ```typescript // Special case: Always include README.md redirects, regardless of alias conflicts const isReadmeRedirect = normalizedRedirectKey .toLowerCase() .endsWith('/readme.md'); const isMatchedAlias = !isReadmeRedirect && aliasesEntries.some(([aliasKey, aliasVal]) => { // ... existing alias matching logic }); if (!isMatchedAlias || isReadmeRedirect) { processRedirect(/* ... */); } ``` ## Result - All README.md redirects are now properly included in the nginx configuration - Over 70+ README.md redirects are now working correctly - No conflicts with existing alias system - README.md files can be accessed directly and will redirect to their parent directory URLs ## Testing Verified the fix by: 1. Running `pnpm run generate-nginx-conf` 2. Confirming README.md redirects appear in `nginx_redirect_map.conf` 3. Cross-referencing with entries in `redirects.json` --- ## File: docs/changelog/0004-contributor-profile-promise-memoization.md # Fix: Improved Contributor Profile Memoization with Promise Caching ## Problem Solved The previous contributor profile memoization implementation had a critical race condition issue. When multiple asynchronous requests were made for the same contributor simultaneously, they could all bypass the cache and hit the API before any completed, defeating the purpose of memoization. **Race Condition Example:** ``` Time 1: fetchContributorProfile("john") starts - cache miss, API call begins Time 2: fetchContributorProfile("john") starts - cache still empty, second API call begins Time 3: Both API calls complete, both cache results separately ``` This led to: - Multiple API calls for the same contributor - Potential API overload during builds - Ineffective rate limiting protection ## Solution Implemented Switched from **result memoization** to **promise memoization**: ### Before (Result Memoization) ```typescript // Cache only stored resolved values const cache = new Map(); // Race condition: multiple calls could bypass empty cache if (cache.has(key)) return cache.get(key); const result = await fetchFromAPI(); cache.set(key, result); ``` ### After (Promise Memoization) ```typescript // Cache stores both promises and resolved values const cache = new Map< string, Promise | MochiUserProfile | string >(); // Immediate promise caching prevents race conditions const cached = cache.get(key); if (cached) return cached instanceof Promise ? await cached : cached; const promise = fetchFromAPI(); cache.set(key, promise); // Cached immediately! ``` ## Key Improvements 1. **Race Condition Eliminated**: Promise is cached immediately, ensuring concurrent requests share the same API call 2. **Dual Caching Strategy**: - Initially caches the Promise (prevents races) - Later replaces with resolved value (optimizes subsequent calls) 3. **Error Handling**: Failed promises are removed from cache to allow retries 4. **API Protection**: Guarantees only one API call per unique contributor during build ## Technical Details The implementation uses a hybrid approach: - **Phase 1**: Cache the Promise immediately when first requested - **Phase 2**: Replace Promise with resolved value after completion - **Concurrent Access**: Multiple callers await the same cached Promise - **Error Recovery**: Failed requests are removed from cache for retry capability ## Impact - **Reduced API Load**: Eliminates duplicate concurrent requests - **Better Build Reliability**: Prevents API overload during contributor page builds - **Improved Performance**: Faster resolution for concurrent requests - **Maintained Flexibility**: Still allows retries on failed requests This fix ensures the contributor profile fetching is both efficient and reliable during static site generation. --- ## File: docs/changelog/0005-refactor-link-utils-duplication.md # Refactor Link Utils: Eliminate Code Duplication and Improve Maintainability ## Changes Made ### Code Duplication Elimination - **Removed duplicated conditional logic** from three functions: `convert_links_with_alt_text`, `convert_links_without_alt_text`, and `convert_markdown_links` - **Created new helper function** `build_link/4` that consolidates the shared link building logic - **Unified file validation logic** by enhancing `file_valid?/3` to accept an optional `check_frontmatter` parameter ### Function Consolidation - **Enhanced `file_valid?/3`** to handle both file existence checking and frontmatter validation - **Removed `file_exists?/2`** function as its logic is now integrated into the enhanced `file_valid?/3` - **Updated `convert_embedded_images/3`** to use the unified `file_valid?/3` function ### Technical Benefits - **Reduced code duplication** by ~30 lines of repeated conditional logic - **Improved maintainability** - changes to link building logic now only need to be made in one place - **Enhanced consistency** - all link conversion functions now use the same validation approach - **Better parameter control** - frontmatter checking can now be optionally disabled per function call ### API Changes - `file_valid?/3` now accepts a third parameter `check_frontmatter` (defaults to `true` for backward compatibility) - `file_exists?/2` function has been removed (functionality moved to `file_valid?/3`) - New private function `build_link/4` handles the unified link building logic ### Usage Examples ```elixir # Check file validity with frontmatter validation (default behavior) file_valid?(link, current_file) file_valid?(link, current_file, true) # Check file existence only (no frontmatter validation) file_valid?(link, current_file, false) ``` This refactoring maintains all existing functionality while making the codebase more maintainable and reducing the potential for inconsistencies between different link conversion functions. --- ## File: docs/changelog/0006-git-submodule-hook-manager.md # Git Submodule Hook Manager ## Overview Introduced a new Git Submodule Hook Manager to streamline the management and enforcement of Git hooks across submodules within the repository. This feature significantly improves consistency and code quality by centralizing hook logic and automating its deployment. ## Key Features - **Centralized Hook Logic**: Git hooks (pre-commit, pre-push, post-commit) are now managed from a single, version-controlled remote script, ensuring all developers use the latest and most consistent checks. - **Dynamic Installation**: A new CLI tool (`scripts/git-shell-hook.ts`) automates the installation of lightweight shell scripts in each submodule's `.git/hooks/` directory. These shell scripts dynamically fetch and execute the latest remote hook logic. - **Submodule Filtering**: The system intelligently identifies and applies hooks only to `dwarvesf` organization submodules, preventing unintended interference with external repositories. - **Simplified Management**: Provides commands for easy setup, removal, and status checking of hooks across all or specific submodules. - **Comprehensive Documentation**: Each configured submodule now includes a dedicated `README.md` file (`_HOOK_README.md`) detailing how the hooks work, setup instructions, troubleshooting, and security considerations. ## How to Use ### Initial Setup (from the root of the main repository) To set up pre-commit hooks for all `dwarvesf` submodules, run: ```bash npx -y tsx scripts/git-shell-hook.ts setup --script-url --hook-type pre-commit ``` Replace `` with the raw URL of your desired hook script (e.g., a script hosted on GitHub). ### Managing Hooks in a Specific Submodule After initial setup, you can manage hooks directly within each submodule: **Install/Reinstall Hook:** `bash cd && ./pre-commit-hook.sh install ` **Remove Hook:** `bash cd && ./pre-commit-hook.sh remove ` **Check Hook Status:** `bash cd && ./pre-commit-hook.sh status ` _(Note: Replace `pre-commit-hook.sh` with `pre-push-hook.sh` or `post-commit-hook.sh` as appropriate for the hook type.)_ ## Impact This feature significantly enhances our development workflow by: - **Enforcing Standards**: Ensures consistent application of linting, formatting, and other quality checks across all relevant codebases. - **Reducing Manual Overhead**: Automates a previously manual and error-prone process of hook distribution. - **Improving Collaboration**: Provides a standardized environment for all contributors, reducing "it works on my machine" issues related to Git hooks. ## Security Note Be aware that the installed hooks dynamically fetch and execute scripts from a remote URL. Ensure that the `scriptUrl` points to a trusted source to mitigate security risks. --- ## File: docs/changelog/0006-memo-nft-report-system.md # Memo NFT Report System Implementation ## Summary Implemented a comprehensive NFT reporting system that provides dual-section analytics for memo.d.foundation's NFT ecosystem, tracking both content minting pipeline and blockchain collection activities. ## What's New ### ๐ŸŽจ Memo NFT Report Script (`scripts/memo-nft-report.ts`) - **Dual Data Source Integration**: Unified DuckDB interface for both remote parquet files and PostgreSQL database - **Section 1: Minted Memo Events**: Analyzes content preparation and minting pipeline status using `https://memo.d.foundation/db/vault.parquet` - **Section 2: Collected Memo Events**: Tracks actual NFT collection activities from `memo_nft.memo_minted_events` PostgreSQL table - **Rich Discord Notifications**: Comprehensive embed reports with health status indicators and key metrics - **Error Handling**: Graceful degradation with detailed logging and recovery mechanisms ### ๐Ÿ“Š Analytics & Metrics **Minting Pipeline Analysis:** - Content eligibility and completion rates - Author contribution tracking - Popular tag analysis - Content quality metrics **Collection Activity Analysis:** - Total collection events and volumes - Collector behavior patterns - Token popularity rankings - Transaction frequency analysis ### ๐Ÿค– GitHub Actions Automation (`.github/workflows/memo-nft-report.yml`) - **Scheduled Execution**: Daily reports at 9 AM UTC - **Manual Dispatch**: Configurable parameters for on-demand reporting - **Health Checks**: Comprehensive environment and connectivity validation - **Multiple Execution Modes**: Scheduled, manual, and development testing - **Error Recovery**: Failure log collection and artifact management ### ๐Ÿ”ง Technical Implementation - **DuckDB with PostgreSQL Extension**: Single database engine for heterogeneous data sources - **TypeScript**: Type-safe implementation with comprehensive interfaces - **Environment Configuration**: Secure credential management via GitHub secrets - **Performance Optimization**: Parallel data collection and efficient query execution ## Usage ### Environment Setup ```bash # Required environment variables MEMO_NFT_DB_CONNECTION_STRING="host=hostname port=5432 dbname=memo_nft user=username password=password" DISCORD_WEBHOOK_URL="https://discord.com/api/webhooks/..." ``` ### Local Development ```bash # Verbose output for development tsx scripts/memo-nft-report.ts --verbose # Send report to Discord tsx scripts/memo-nft-report.ts --send ``` ### Production Deployment - Configure GitHub repository secrets for database and Discord access - Workflow runs automatically daily or can be triggered manually - Monitor Discord channel for regular reports and health status ## Data Sources ### Parquet File Analysis - **Source**: `https://memo.d.foundation/db/vault.parquet` - **Purpose**: Minting pipeline analysis and content preparation tracking - **Key Fields**: `should_mint`, `minted_at`, `token_id`, `authors`, `tags`, `estimated_tokens` ### PostgreSQL Database Analysis - **Source**: `memo_nft.memo_minted_events` table - **Purpose**: Blockchain collection activity tracking - **Schema**: `id`, `to` (ETH address), `amount`, `token_id`, `timestamp` ## Report Features ### Discord Report Sections 1. **Minted Memo Events Section** - Pipeline completion statistics - Top contributing authors - Popular content categories - Recent minting activity trends 2. **Collected Memo Events Section** - Collection event summaries - Top collector rankings - Most popular token analysis - Transaction activity metrics ### Health Status Indicators - ๐ŸŸข **Healthy**: High minting efficiency (โ‰ฅ70%) with active collections - ๐ŸŸก **Warning**: Low efficiency or minimal collection activity - ๐Ÿ”ด **Critical**: System errors or data source unavailability ## Benefits ### For Content Creators - **Author Analytics**: Track individual contribution impact and minting success rates - **Content Performance**: Understand which topics and tags perform best in the NFT ecosystem - **Pipeline Visibility**: Monitor content progression from creation to NFT minting ### For Collectors & Community - **Collection Insights**: Identify trending tokens and popular content - **Community Activity**: Track overall ecosystem health and engagement levels - **Market Intelligence**: Understand collection patterns and collector behavior ### For Operations & Development - **System Monitoring**: Automated health checks and performance tracking - **Data Transparency**: Comprehensive visibility into the NFT pipeline - **Trend Analysis**: Historical data for strategic decision making ## Integration ### Existing Infrastructure Compatibility - **Similar Pattern**: Follows established monitoring system architecture like `monitor-vault-parquet.ts` - **DuckDB Ecosystem**: Leverages existing parquet processing infrastructure - **Discord Integration**: Uses established webhook notification patterns - **GitHub Actions**: Consistent with existing workflow automation ### Future Expansion Opportunities - **Cross-Source Analytics**: Potential for JOIN queries between data sources - **Historical Trending**: Time-series analysis for growth patterns - **Alert Systems**: Threshold-based notifications for anomaly detection - **Dashboard Integration**: Web-based visualization interfaces ## Technical Notes ### Security & Access Control - **Read-Only Database Access**: Prevents accidental data modification - **Environment Variable Security**: Encrypted credential storage via GitHub secrets - **Address Privacy**: Truncated display of Ethereum addresses for privacy - **HTTPS-Only Communications**: Secure data transmission protocols ### Performance Characteristics - **Memory Efficient**: ~500MB typical usage for DuckDB operations - **Network Optimized**: Efficient remote parquet file access - **Concurrent Processing**: Parallel data collection from multiple sources - **Query Optimization**: Indexed and filtered database access patterns This implementation establishes a robust foundation for NFT ecosystem monitoring and provides actionable insights for content creators, collectors, and platform operators while maintaining high standards for security, performance, and reliability. --- ## File: docs/changelog/0007-modular-markdown-linting-system.md # Modular Markdown Linting System ## Overview The project's Markdown linting system (`scripts/formatter/note-lint.js`) has undergone a significant refactor to introduce a modular, extensible, and configurable architecture. This enhancement improves the maintainability and effectiveness of our documentation quality checks, allowing for easier addition of new rules and automated fixes. ## Key Features - **Modular Rules**: Each linting rule (e.g., `frontmatter`, `no-heading1`, `relative-link-exists`) is now an independent module, making the system highly extensible. New rules can be added by simply creating a new file in `scripts/formatter/rules/` and exporting it via `rules/index.js`. - **Configurable Severity**: Rules can now be configured with different severity levels (`error` or `warn`), providing flexibility in how violations are enforced. - **Auto-Fixing Capabilities**: Many rules now support automatic correction of issues, reducing manual effort. For example, the `no-heading1` rule can automatically convert H1 headings to H2. - **Improved Reporting**: The linter provides clearer, color-coded output, summarizing errors, warnings, and fixable issues for better developer experience. - **ESLint-Inspired Architecture**: The internal design is inspired by ESLint, providing a `Linter` class and a `Rule` interface that offers a `context` object to rules for rich interaction with the file content and reporting mechanisms. ## How to Use ### Running the Linter From the project root, you can run the linter with a target directory: ```bash node scripts/formatter/note-lint.js ``` Example: ```bash node scripts/formatter/note-lint.js vault/ ``` ### Applying Fixes To automatically fix issues where rules support it, use the `--fix` flag: ```bash node scripts/formatter/note-lint.js vault/ --fix ``` ### Customizing Rules (Future) While a default configuration is provided, the new architecture supports loading custom configurations. This will allow teams to enable/disable specific rules or change their severity based on project needs. ## Impact This refactor significantly enhances our ability to maintain high-quality Markdown documentation. It streamlines the process of enforcing writing conventions, reduces manual correction time through auto-fixing, and provides a robust foundation for future expansion of linting capabilities. This leads to more consistent, readable, and reliable project documentation. --- ## File: docs/changelog/0008-fix-git-timestamp-based-incremental-processing.md # Fix: Git Timestamp-Based Incremental Processing ## Overview Fixed a critical issue in the memo export system where files fetched from Git repositories were not being processed correctly due to incorrect timestamp comparison. The system now uses Git commit timestamps instead of filesystem modification times for more accurate incremental processing. ## Problem Solved ### Previous Behavior - The system used filesystem modification time (mtime) to determine which files had changed since the last processing - When files were fetched from Git repositories (including submodules), the local mtime reflected when files were written to disk locally, not when they were actually modified in the repository - This caused: - **False negatives**: Recently updated remote files being skipped - **False positives**: Unchanged files being reprocessed unnecessarily - **Data inconsistency**: Export data becoming stale after Git operations ### New Behavior - The system now uses Git commit timestamps to accurately determine when files were last modified in the repository - File processing decisions are based on actual repository history, not local filesystem timestamps - Works correctly across all Git operations: `git pull`, `git submodule update`, repository cloning, etc. ## Technical Changes ### Core Improvement - **Replaced**: `File.stat(file_path).mtime` comparison - **With**: `git log -1 --format=%cI` commit timestamp comparison ### New Functions - `get_file_last_commit_timestamp/1`: Retrieves the last Git commit timestamp for a file - `find_git_root_for_file/1`: Dynamically finds the correct Git repository root (handles submodules) ### Enhanced Error Handling - Graceful fallback: If Git timestamp cannot be determined, files are included for processing (safer approach) - Detailed error logging for debugging timestamp issues - Fallback to first commit timestamp if latest commit lookup fails ## Impact on Users ### Immediate Benefits 1. **Accurate Processing**: Files are now correctly identified as changed/unchanged based on actual Git history 2. **Reliable Exports**: Export data will be consistent and up-to-date after Git operations 3. **Submodule Support**: Proper handling of files in Git submodules 4. **Reduced Redundancy**: Eliminates unnecessary reprocessing of unchanged files ### Use Cases Fixed - **Content Teams**: Markdown files updated in remote repositories are now properly detected and processed - **Automated Workflows**: CI/CD pipelines that fetch content updates will see accurate processing - **Submodule Users**: Projects using Git submodules for content management now work correctly - **Multi-Repository Setups**: Systems pulling from multiple Git sources maintain data consistency ## Migration Notes ### No Action Required - **Zero Downtime**: Change is completely transparent to existing users - **No Database Changes**: Existing database schema and data remain unchanged - **Backward Compatible**: All existing functionality continues to work as expected ### Requirements - **Git Availability**: System must have Git installed and properly configured (already a requirement for existing Git operations) - **Repository Access**: Git repositories must be accessible for timestamp queries ## Example Scenarios ### Before the Fix ```bash # User fetches updates git pull origin main git submodule update --remote # Run export mix memo.export # Result: Recently updated files skipped, stale data in export ``` ### After the Fix ```bash # User fetches updates git pull origin main git submodule update --remote # Run export mix memo.export # Result: All updated files correctly processed, export data current ``` ### Submodule Handling ``` vault/ โ”œโ”€โ”€ main-content.md # Uses main repo Git timestamps โ””โ”€โ”€ external-content/ # Git submodule โ””โ”€โ”€ submodule-file.md # Uses submodule Git timestamps ``` ## Performance Considerations - **Minimal Overhead**: Git timestamp queries are fast and cached by Git - **Batch Processing**: Timestamp checks are done efficiently during file filtering - **No Impact on Processing**: Core file processing performance unchanged ## Troubleshooting ### Common Issues 1. **"Git log failed"** errors: Ensure Git is installed and repositories are properly initialized 2. **"Could not determine git root"** warnings: Check that files are within Git repositories 3. **Timestamp parsing errors**: Verify Git version supports ISO8601 format output ### Debug Information - Enhanced logging shows Git timestamp retrieval process - Clear error messages indicate specific Git operation failures - Fallback behavior ensures processing continues even with Git issues ## Related Documentation - **ADR**: Architecture Decision Record documenting the technical rationale - **Git Integration**: How the system handles Git repositories and submodules - **Incremental Processing**: Overview of the timestamp-based filtering system --- ## File: docs/changelog/0009-per-file-processing-metadata-tracking.md # Enhancement: Per-File Processing Metadata Tracking ## Overview Significantly enhanced the memo export system's processing metadata tracking by implementing per-file tracking instead of a single global timestamp. The `processing_metadata.parquet` file now contains detailed information for each processed file, enabling better tracking, filtering, and analysis. ## What Changed ### Before: Global Timestamp Approach ```sql -- Old schema CRETE TABLE processing_metadata ( id INTEGER PRIMARY KEY DEFAULT 1, last_processed_at TIMESTAMP ); ``` - Single row with global timestamp - Limited visibility into individual file processing - Difficult to debug or analyze processing patterns - Coarse-grained incremental processing ### After: Per-File Metadata Tracking ```sql -- New schema CREATE TABLE processing_metadata ( file_path TEXT PRIMARY KEY, last_processed_at TIMESTAMP, git_commit_timestamp TIMESTAMP, processing_status VARCHAR DEFAULT 'processed', created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP, updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP ); ``` - One row per file with comprehensive metadata - Rich tracking and analysis capabilities - Git integration for accurate timestamping - Status tracking for operational visibility ## New Capabilities ### Enhanced Processing Metadata Export The `processing_metadata.parquet` file now contains: | Field | Type | Description | | ---------------------- | --------- | ------------------------------------------- | | `file_path` | TEXT | Relative path of the processed file | | `last_processed_at` | TIMESTAMP | When the file was last processed | | `git_commit_timestamp` | TIMESTAMP | Git commit timestamp of the file | | `processing_status` | VARCHAR | Processing status (processed, failed, etc.) | | `created_at` | TIMESTAMP | When tracking started for this file | | `updated_at` | TIMESTAMP | When metadata was last updated | ### Rich Analysis Queries Users can now perform detailed analysis: ```sql -- Files processed in the last hour SELECT file_path, last_processed_at FROM processing_metadata WHERE last_processed_at > NOW() - INTERVAL 1 HOUR; -- Files needing reprocessing (Git newer than processing) SELECT file_path, git_commit_timestamp, last_processed_at FROM processing_metadata WHERE git_commit_timestamp > last_processed_at; -- Processing frequency analysis SELECT file_path, DATEDIFF('day', created_at, updated_at) as days_tracked, last_processed_at FROM processing_metadata ORDER BY updated_at DESC; -- Correlate with vault data SELECT v.title, v.tags, pm.last_processed_at, pm.processing_status FROM vault v JOIN processing_metadata pm ON v.file_path = pm.file_path WHERE pm.processing_status = 'processed' ORDER BY pm.last_processed_at DESC; ``` ## Use Cases Enabled ### 1. Processing Analytics - **Track processing patterns**: Identify which files are processed most frequently - **Performance monitoring**: Monitor processing times and identify bottlenecks - **Status reporting**: Generate reports on processing success rates ### 2. Operational Debugging - **Failed processing identification**: Quickly find files that failed to process - **Incremental processing verification**: Verify which files were included in each run - **Git sync analysis**: Compare Git timestamps with processing timestamps ### 3. Content Management - **Stale content detection**: Find files that haven't been processed recently - **Update tracking**: Monitor when specific files were last updated - **Audit trails**: Maintain complete processing history per file ### 4. Export Analysis - **Filtering by processing time**: Filter exports by when files were processed - **Cross-reference data**: Join vault data with processing metadata - **Time-based analysis**: Analyze content changes over time ## Example Workflows ### Monitoring Recent Changes ```sql -- Files processed today with their content SELECT v.file_path, v.title, v.tags, pm.last_processed_at, pm.git_commit_timestamp FROM vault v JOIN processing_metadata pm ON v.file_path = pm.file_path WHERE pm.last_processed_at >= CURRENT_DATE ORDER BY pm.last_processed_at DESC; ``` ### Finding Outdated Content ```sql -- Files that haven't been processed in a week SELECT file_path, last_processed_at, processing_status, DATEDIFF('day', last_processed_at, NOW()) as days_since_processing FROM processing_metadata WHERE last_processed_at < NOW() - INTERVAL 7 DAY ORDER BY last_processed_at ASC; ``` ### Git Sync Verification ```sql -- Files where Git is newer than processing (need reprocessing) SELECT file_path, git_commit_timestamp, last_processed_at, DATEDIFF('hour', last_processed_at, git_commit_timestamp) as hours_behind FROM processing_metadata WHERE git_commit_timestamp > last_processed_at ORDER BY hours_behind DESC; ``` ## Migration and Compatibility ### Automatic Migration - **Zero downtime**: Existing installations automatically migrate to the new schema - **Backward compatibility**: All existing functionality continues to work - **Preserved behavior**: Incremental processing behavior remains unchanged ### Compatibility Layer - The system maintains backward compatibility by using `MIN(last_processed_at)` as a fallback global timestamp - Existing scripts and workflows continue to function - New capabilities are additive, not breaking ## Performance Improvements ### Optimized Database Operations - **Indexed queries**: Added index on `last_processed_at` for fast temporal queries - **Batch processing**: Metadata updates processed in batches of 50 files - **Efficient upserts**: Atomic `ON CONFLICT` operations for reliability ### Scalability Enhancements - **O(1) lookups**: Primary key on `file_path` ensures constant-time access - **Minimal overhead**: Per-file tracking adds negligible processing time - **Efficient storage**: Compact schema design minimizes storage impact ## Integration with Git Timestamps This enhancement builds on the Git timestamp integration (ADR-0013) to provide: - **Accurate change detection**: Git commit timestamps stored alongside processing times - **Cross-validation**: Compare processing timestamps with actual Git history - **Sync verification**: Identify files that need reprocessing based on Git changes ## Developer Benefits ### Enhanced Debugging ```sql -- Debug processing issues SELECT file_path, processing_status, last_processed_at, updated_at FROM processing_metadata WHERE processing_status != 'processed' ORDER BY updated_at DESC; ``` ### Processing Statistics ```sql -- Processing success rate SELECT processing_status, COUNT(*) as count, ROUND(COUNT(*) * 100.0 / SUM(COUNT(*)) OVER(), 2) as percentage FROM processing_metadata GROUP BY processing_status; ``` ## Future Enhancements The new schema enables future capabilities: 1. **Error Tracking**: Store specific error messages and failure reasons 2. **Processing Duration**: Track how long each file takes to process 3. **Retry Logic**: Implement intelligent retry mechanisms based on processing history 4. **Alerting**: Notify when files haven't been processed for extended periods 5. **Analytics Dashboard**: Rich visualizations of processing patterns and trends ## Impact on Exports ### Enhanced Parquet Files - **processing_metadata.parquet**: Now contains actionable per-file data instead of single global timestamp - **Rich analysis**: Enables sophisticated analysis of processing patterns - **Cross-referencing**: Easy to correlate with vault.parquet data using file_path ### Query Examples for Exported Data ```python # Python example using pandas import pandas as pd # Load the parquet files vault_df = pd.read_parquet('vault.parquet') processing_df = pd.read_parquet('processing_metadata.parquet') # Join vault data with processing metadata merged_df = vault_df.merge(processing_df, on='file_path', how='left') # Analyze recent changes recent_changes = merged_df[ merged_df['last_processed_at'] > pd.Timestamp.now() - pd.Timedelta(days=1) ] # Find files that need reprocessing needs_reprocessing = merged_df[ merged_df['git_commit_timestamp'] > merged_df['last_processed_at'] ] ``` ## Summary This enhancement transforms the processing metadata from a simple global timestamp into a comprehensive per-file tracking system. Users can now: - **Track individual files**: Know exactly when each file was processed - **Analyze patterns**: Understand processing behavior and identify issues - **Reference in exports**: Use file_path to correlate data across parquet files - **Monitor system health**: Get detailed visibility into processing status - **Debug efficiently**: Quickly identify and resolve processing issues The change is fully backward compatible and adds significant value for users who need to understand and analyze their content processing workflows. --- ## File: docs/changelog/added-limited-feeds.md ### Added: Support for Limited RSS/Atom Feeds Users can now request RSS and Atom feeds with a specific number of recent items by adding a `?limit=N` query parameter to the feed URL, where `N` is a positive integer. For example: - `/rss.xml?limit=20` will return the 20 most recent items in RSS format. - `/atom.xml?limit=50` will return the 50 most recent items in Atom format. - `/feed/index.xml?limit=10` will return the 10 most recent items for that specific feed path. If the `limit` parameter is not provided or is invalid (e.g., not a positive integer), the feed will default to showing 20 items. If a pre-generated limited feed file for the exact user-specified limit (or the default 20) is not available, the full, unlimited feed will be served as before. This feature enhances flexibility for feed readers and can help reduce bandwidth usage. ### Enhanced: Stricter Date Validation for Feed Content (2025-05-31) - The RSS/Atom feed generation script now enforces stricter validation for `date` and `lastmod` fields in content frontmatter. - Posts **must** have a valid `date` in their frontmatter (e.g., `date: 'YYYY-MM-DD'`) to be included in any RSS/Atom feed. Posts with invalid or missing `date` fields will be excluded. - The `lastmod` field, if invalid or missing, will correctly fall back to the `date` field's value. - This change ensures greater accuracy and reliability of publication dates in our feeds. Content authors should verify their articles have valid `date` frontmatter if they are intended for syndication. --- ## File: docs/changelog/aliases-and-shortlinks.md # Documentation for Global Folder-Level Aliases and Short Links This document explains how to use global folder-level aliases and short links in your Markdown files and configuration files. ## Global Folder-Level Aliases Global folder-level aliases allow you to create alternative paths for accessing content within a specific folder. This is useful for providing shorter or more memorable links to frequently accessed directories. To create a global folder-level alias for a folder, you need to create or edit the `.config.yaml` or `.config.yml` file within that folder in the `vault` directory. Add an `aliases` key with a list of alias paths (starting with a `/`) that should point to this folder. **Example:** To create an alias `/culture-alias` that points to the `/vault/culture` folder, you would create or edit `vault/culture/config.yaml` and add the following: ```yaml aliases: - /culture-alias ``` After adding the alias and running the export process, you can access any file within the `/vault/culture` folder using the `/culture-alias` path. For example, if you have a file at `/vault/culture/about.md`, you can access it via `/culture-alias/about`. **Important:** - Alias paths must start with a `/`. - The `config.yaml` file should be placed in the root of the folder you want to alias. - You need to run the export process after modifying `config.yaml` for the changes to take effect. ## Short Links (Redirects) Short links allow you to create shorter or alternative URLs that generate static files for SEO purposes, referring to a different, usually longer, URL. This is useful for creating clean and easy-to-share links while improving search engine visibility. To create a short link for a specific Markdown file, you need to add a `aliases` key to the frontmatter of that Markdown file in the `vault` directory. The `aliases` key should have a list of short link paths (starting with a `/`) that should refer to this file. **Example:** To create a short link `/share/ai-travel` that refers to the Markdown file at `/vault/updates/build-log/ai-ruby-travel-assistant-chatbot.md`, you would add the following to the frontmatter of `vault/updates/build-log/ai-ruby-travel-assistant-chatbot.md`: ```yaml --- title: 'AI-powered Ruby travel assistant' description: 'A case study exploring how we built an AI-powered travel assistant using Ruby and AWS Bedrock, demonstrating how choosing the right tools over popular choices led to a more robust and maintainable solution. This study examines our approach to integrating AI capabilities within existing Ruby infrastructure while maintaining enterprise security standards.' date: '2024-11-21' authors: - 'monotykamary' tags: - 'agents' - 'ai' aliases: - /share/ai-travel --- > "When the easy path isn't the right path, true engineering shines in adapting the right tools for the job." ## The challenge: Beyond the obvious choice ... ``` After adding the short link and running the export process, a static file will be generated at `/share/ai-travel` that points to the `/updates/build-log/ai-ruby-travel-assistant-chatbot` page for SEO purposes. **Important:** - Short link paths must start with a `/`. - The `aliases` key should be in the frontmatter of the target Markdown file. - You can define multiple short links for a single file by adding more entries under the `aliases` key. - You need to run the export process after modifying the Markdown file for the changes to take effect.