Skip to content

feat(drivers/alias): add optional cache and index-assisted resolution#2608

Open
syscc wants to merge 3 commits into
OpenListTeam:mainfrom
syscc:feat/alias-cache-index
Open

feat(drivers/alias): add optional cache and index-assisted resolution#2608
syscc wants to merge 3 commits into
OpenListTeam:mainfrom
syscc:feat/alias-cache-index

Conversation

@syscc

@syscc syscc commented Jun 13, 2026

Copy link
Copy Markdown
Contributor

Summary / 摘要

Adds opt-in caching for the Alias driver to reduce repeated backend polling when listing aggregated directories and resolving virtual Alias paths to backend paths.

Latest updates in this PR:

  • Adds alias_cache_enabled, alias_cache_expiration, alias_cache_max_entries, and alias_list_concurrency Alias options.

  • Caches aggregated Alias directory listings and resolved backend paths when enabled.

  • Treats missing Alias backend roots / future placeholder mounts as cacheable misses, so unavailable configured paths do not prevent caching valid results from existing backends.

  • Uses backend directory paths as the list cache key so internal op.List(..., model.ListArgs{}) calls without ReqPath do not collide.

  • Avoids caching partial list results when any backend returns a non-not-found error.

  • Clears the Alias cache on write-like operations and invalidates refreshed list/resolved entries when refresh is requested.

  • Adds index-assisted Alias path resolution: when the search index is enabled, complete, and not currently building, Alias uses indexed parent/name entries to prioritize likely backend paths, then verifies the chosen backend path with fs.Get before caching.

  • Narrows cached resolved child backend paths to paths that actually exist in listed backend directories, reducing later scans for large Alias mounts.

  • Adds background parent-directory prewarming after direct file resolution. Directly opening /videos/.../Season 1/E01 can prewarm /videos/.../Season 1, so later sibling episode lookups benefit from the same Alias list/resolved cache.

  • Keeps the feature disabled by default. alias_cache_max_entries=0 means unlimited entries.

  • This PR has breaking changes.
    / 此 PR 包含破坏性变更。

  • This PR changes public API, config, storage format, or migration behavior.
    / 此 PR 修改了公开 API、配置、存储格式或迁移行为。

  • This PR requires corresponding changes in related repositories.
    / 此 PR 需要关联仓库同步修改。

Related repository PRs / 关联仓库 PR:

  • OpenList-Frontend: N/A. The existing driver metadata form renders these options dynamically.
  • OpenList-Docs: N/A.

Related Issues / 关联 Issue

N/A

Testing / 测试

  • go test ./...
    • Not run locally; validation focused on the Alias package and source-mode runtime checks.
  • go test ./drivers/alias
  • Manual test / 手动测试:
    • Source-mode run: go run . --data data --config /tmp/openlist-config-with-s3.json --log-std server
    • Formatting: gofmt -w drivers/alias/driver.go drivers/alias/cache.go drivers/alias/index.go drivers/alias/meta.go drivers/alias/util.go internal/search/search.go
    • Package resolution: go list ./drivers/alias ./internal/search ./internal/search/db ./internal/search/db_non_full_text ./internal/search/meilisearch
    • Diff check: git diff --check

Missing backend roots / placeholder mounts

NAS Alias mount /videos, alias_list_concurrency=8, cache enabled. The Alias config contains many future placeholder roots that currently do not exist, for example /ODS/mov20/电影 and /ODS/tv65/电视剧.

Request Mode Result Time
/videos/电影/外语电影 refresh=true 200, 561 items 17.349308s
/videos/电影/外语电影 warm request 1 200, 561 items 0.027864s
/videos/电影/外语电影 warm request 2 200, 561 items 0.024428s
/videos/电影 refresh=true 200, 3 items 5.221206s
/videos/电影 warm request 1 200, 3 items 0.006914s
/videos/电影 warm request 2 200, 3 items 0.007506s
/videos/电视剧/欧美剧 refresh=true 200, 220 items 59.402720s
/videos/电视剧/欧美剧 warm request 1 200, 220 items 0.021879s
/videos/电视剧/欧美剧 warm request 2 200, 220 items 0.017238s

Result: missing configured backend roots no longer prevent caching valid aggregated Alias list results.

Third-level directory cache behavior

After listing parent directories, cached child resolution only keeps backend paths that actually exist for that child name.

Request Run Result Time
/videos/电视剧 first 200, 9 items 9.942850s
/videos/电视剧 warm 200, 9 items 0.000719s
/videos/电视剧/国产剧 first after parent 200, 458 items 4.264880s
/videos/电视剧/国产剧 warm 200, 458 items 0.001811s
/videos/电视剧/日韩剧 first after parent 200, 110 items 2.721708s
/videos/电视剧/日韩剧 warm 200, 110 items 0.000742s
/videos/电影/华语电影 first after parent 200, 393 items 1.649434s
/videos/电影/华语电影 warm 200, 393 items 0.001235s

Direct file playback parent prewarm

Validated on beta image from this PR branch:

beta (Commit: ef75563) - Frontend: rolling - Build at: 2026-06-15 23:26:53

Directly resolving the first episode prewarms the parent season directory in the background. After the prewarm delay, the parent list and sibling episode lookups are fast.

Scenario Result Time
Direct get /videos/电视剧/日韩剧/21世纪大君夫人 (2026)/Season 1/...S01E01.mp4 200 6.442576s
Parent list /videos/电视剧/日韩剧/21世纪大君夫人 (2026)/Season 1 after prewarm 200, 40 items 0.010086s
Direct get S01E02 first request 200 0.273630s
Direct get S01E02 repeat 200 0.009657s
Direct get S01E03 first request 200 0.288162s
Direct get S01E03 repeat 200 0.010404s
Scenario Result Time
Direct get /videos/电视剧/国产剧/边水往事 (2024)/Season 1/...S01E01.mkv 200 7.416666s
Parent list /videos/电视剧/国产剧/边水往事 (2024)/Season 1 after prewarm 200, 85 items 0.018576s
Direct get S01E02 first request 200 0.253483s
Direct get S01E02 repeat 200 0.009704s
Direct get S01E03 first request 200 0.229722s
Direct get S01E03 repeat 200 0.007850s

Earlier source-mode validation

Directory list cache behavior:

Request Mode Result Time
/videos/电视剧 refresh=true 200, 9 items 10.485506s
/videos/电视剧 warm request 1 200, 9 items 0.001217s
/videos/电视剧 warm request 2 200, 9 items 0.001104s
/videos/电视剧/欧美剧 refresh=true 200, 218 items 4.052087s
/videos/电视剧/欧美剧 warm request 1 200, 218 items 0.002297s
/videos/电视剧/欧美剧/3体 (2024) refresh=true 200, 12 items 0.581298s
/videos/电视剧/欧美剧/3体 (2024) warm request 1 200, 12 items 0.001222s

Search-index-assisted resolution validation:

Check Result Time
/api/admin/index/progress 200, is_done=true 17.183401s
/api/fs/search for /ODS/tv17/.../S01E01 200, target files found 0.038614s
cold /api/fs/get /videos/.../S01E01.mkv 200 9.572376s
repeat /api/fs/get /videos/.../S01E01.mkv 200 0.001666s
/api/fs/get /ODS/tv17/.../S01E01.mkv 200 0.000991s

Refresh behavior with Alias cache enabled:

Request Result Time
/api/fs/list /videos/.../Season 1, refresh=false 200, 47 items 0.002924s
/api/fs/list /videos/.../Season 1, refresh=true 200, 47 items 0.741430s
/api/fs/list /videos/.../Season 1, refresh=false 200, 47 items 0.002813s

Longer memory / latency comparison

Dataset: recursive list paths discovered under /videos: 4074 directories.

Cache-on measurement used a temporary 180-minute TTL during prewarm so entries would not expire before the 30-minute measurement. Local config was restored to 30 minutes after the test.

Mode Stabilize RSS avg Measure RSS avg Measure RSS max Requests / 30 min Avg list latency Max list latency Errors
Alias cache on 1362.0 MB 1258.4 MB 1725.9 MB 3224 0.0009s 0.0172s 0
Alias cache off 77.1 MB 106.2 MB 124.9 MB 1360 1.2767s 34.8874s 0

Notes:

  • RSS was sampled from the OpenList child process created by go run, not the parent go run process.
  • macOS RSS fluctuated due to runtime/OS memory return behavior.
  • The memory cost is why the feature is opt-in and has an optional alias_cache_max_entries limit.

Checklist / 检查清单

  • I have read CONTRIBUTING.
    / 我已阅读 CONTRIBUTING
  • I confirm this contribution follows the repository license, contribution policy, and code of conduct.
    / 我确认此贡献符合仓库许可证、贡献规范和行为准则。
  • I have formatted the changed code with gofmt, go fmt, or prettier where applicable.
    / 我已按适用情况使用 gofmtgo fmtprettier 格式化变更代码。
  • I have requested review from relevant maintainers or code owners where applicable.
    / 我已在适用情况下请求相关维护者或代码所有者审查。

AI Disclosure / AI 使用声明

  • This PR includes AI-assisted content.
    / 此 PR 包含 AI 辅助内容。

Tools used / 使用工具:

  • ChatGPT
  • Codex
  • GitHub Copilot
  • Claude
  • Gemini
  • Other (please specify) / 其他(请注明):

Usage scope / 使用范围:

  • Code generation / 代码生成

  • Refactoring / 重构

  • Documentation / 文档

  • Tests / 测试

  • Translation / 翻译

  • Review assistance / 审查辅助

  • I have reviewed and validated all AI-assisted content included in this PR.
    / 我已审核并验证此 PR 中的所有 AI 辅助内容。

  • I have ensured that all AI-assisted commits include Co-Authored-By attribution.
    / 我已确保所有 AI 辅助提交都包含 Co-Authored-By 归属信息。

  • I can reproduce all AI-assisted content included in this PR without any AI tools.
    / 我可以在没有任何 AI 工具的情况下重现此 PR 中包含的所有 AI 辅助内容。

Co-authored-by: OpenAI Codex <codex@openai.com>
@jyxjjj

jyxjjj commented Jun 13, 2026

Copy link
Copy Markdown
Member

Just a note: GitHub has Draft PRs for cases where the work isn’t ready for review yet, so you don’t need to wait until everything is review-ready before opening a PR.

@syscc

syscc commented Jun 13, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for the note. I’ll use Draft PRs for work-in-progress changes next time.

@syscc syscc marked this pull request as draft June 15, 2026 14:12
@syscc syscc force-pushed the feat/alias-cache-index branch from fdb3ae7 to de0b9ed Compare June 15, 2026 15:01
@syscc

syscc commented Jun 16, 2026

Copy link
Copy Markdown
Contributor Author

已补充本次 Alias 缓存与父目录预热更新,测试通过。麻烦管理审核一下,没问题的话帮忙合并。

@syscc syscc marked this pull request as ready for review June 16, 2026 04:36
@xrgzs xrgzs changed the title feat(alias): add optional cache and index-assisted resolution feat(drivers/alias): add optional cache and index-assisted resolution Jun 17, 2026
@xrgzs xrgzs added the Module: Driver Driver-Related Issue/PR label Jun 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Module: Driver Driver-Related Issue/PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants