Fix memory search fallback when embeddings are unavailable

[wayyoungboy]

Summary

• fall back to text search when query embeddings cannot be generated
• add retrieval mode and fusion controls across memory search APIs
• preserve vector-only and OceanBase backend compatibility while keeping FTS threshold behavior distinct from result quality
• keep metadata filters consistent across vector, FTS, and hybrid retrieval paths
• rebase onto the latest main and preserve metadata filters whose names collide with payload fields such as hash and data
• preserve explicit no-embedding mode behavior: default search stays empty when embeddings are intentionally disabled, while transient embedding failures can still fall back to FTS

Tests

• after activating the project virtual environment, python -m pytest --ignore=tests/regression -m "not e2e and not e2e_config"
• after activating the project virtual environment, python -m pytest tests/unit/test_sqlite_fts.py tests/unit/test_memory.py tests/unit/server/test_search_score_threshold.py tests/unit/test_list_memory_filters.py tests/integration/test_sqlite_fts_integration.py
• after activating the project virtual environment, python -m pytest tests/unit/test_intelligence_forget_soft_mark.py tests/integration/test_noop_embedding_mode.py tests/unit/test_noop_embedding.py -q
• public Memory API E2E with SQLite in-memory: forced embedding generation failure, then verified FTS fallback returns the single result matching collision-prone metadata filters; also verified explicitly disabled embeddings keep default search empty
• public Memory API E2E with a remote OceanBase test database: forced embedding generation failure, then verified OceanBase FTS fallback returns the expected metadata-filtered result; also verified explicitly disabled embeddings keep default search empty while explicit FTS retrieval still works

Fixes #1046

[wayyoungboy]

Review result: not ready to merge yet.

Findings:

• Blocker: the branch no longer merges cleanly with the current main branch. The conflicts are in src/powermem/storage/adapter.py and src/server/models/request.py, so this needs a rebase or merge conflict resolution before final review.
• Compatibility risk: the new top-level filter key handling in StorageAdapter changes plain filter semantics for user metadata keys that collide with promoted payload fields such as hash, data, created_at, updated_at, and fulltext_content. Existing callers using plain filters for metadata keys with those names can miss unless they use an explicit metadata.* filter. Please either preserve the previous metadata-filter behavior or document and test the intended syntax.

Validation:

• Targeted unit/integration tests passed in a local Python 3.11 environment: python -m pytest tests/unit/test_sqlite_fts.py tests/unit/test_memory.py tests/unit/server/test_search_score_threshold.py tests/unit/test_list_memory_filters.py tests/integration/test_sqlite_fts_integration.py
• E2E check passed through the public Memory API using SQLite storage: added two memories, forced search embedding generation to fail, and verified FTS fallback returned only the matching user-scoped result.

No LGTM because this PR is authored from the wayyoungboy fork and still has merge conflicts.

[wayyoungboy]

Updated this PR after review:

• rebased the branch onto the latest main, which resolves the merge conflicts
• preserved upstream observation request models while keeping the new search retrieval controls
• fixed SQLite/PGVector metadata filter compatibility for collision-prone keys such as hash and data; explicit payload filtering is still available with payload.<field>
• added regression coverage for those collision-prone metadata filters

Validation:

• after activating the project virtual environment, python -m pytest tests/unit/test_sqlite_fts.py tests/unit/test_memory.py tests/unit/server/test_search_score_threshold.py tests/unit/test_list_memory_filters.py tests/integration/test_sqlite_fts_integration.py passed
• public Memory API E2E with SQLite in-memory passed: embedding generation was forced to fail, and search returned the single FTS fallback hit matching hash/data metadata filters
• two independent code reviews found no blocking or actionable issues

[wayyoungboy]

Updated this PR after review:

• rebased the branch onto the latest main, which resolves the merge conflicts
• preserved upstream observation request models while keeping the new search retrieval controls
• fixed SQLite/PGVector metadata filter compatibility for collision-prone keys such as hash and data; explicit payload filtering is still available with payload.<field>
• kept explicit no-embedding mode backward compatible: default search remains empty when embeddings are intentionally disabled, while transient embedding failures can still fall back to FTS on supported stores
• added regression coverage for those collision-prone metadata filters

Validation:

• after activating the project virtual environment, python -m pytest --ignore=tests/regression -m "not e2e and not e2e_config" passed
• after activating the project virtual environment, python -m pytest tests/unit/test_sqlite_fts.py tests/unit/test_memory.py tests/unit/server/test_search_score_threshold.py tests/unit/test_list_memory_filters.py tests/unit/test_noop_embedding.py tests/integration/test_sqlite_fts_integration.py tests/integration/test_noop_embedding_mode.py passed
• public Memory API E2E with SQLite in-memory passed: embedding generation was forced to fail, and search returned the single FTS fallback hit matching hash/data metadata filters; explicit no-embedding mode also kept default search empty
• two independent code reviews found no blocking or actionable issues

[wayyoungboy]

Added the remaining OceanBase E2E validation for this PR:

• used the public Memory API against a remote OceanBase test database with a unique temporary collection
• forced query embedding generation to fail and verified search fell back to OceanBase full-text retrieval
• verified the fallback result respected a metadata filter on a non-conflicting metadata key
• verified explicitly disabled embeddings still keep default search empty
• verified explicit retrieval_mode="fts" still returns the OceanBase full-text hit when embeddings are disabled
• dropped the temporary collection after the test

Latest PR checks are green; release-only jobs that are not expected for this PR are skipped by workflow rules.

[wayyoungboy]

Self-review status for the current head: the branch is mergeable and current checks are green. The latest version keeps explicit no-embedding mode backward compatible: default search remains empty when embeddings are intentionally disabled, while transient embedding failures can still fall back to FTS on supported stores; explicit retrieval_mode="fts" remains available for no-embedding full-text lookup.

I do not see a new blocker in this static review. This should stay aligned with #1091 before either path lands, because #1091 currently changes the default no-embedding search behavior in the opposite direction. I am not marking this LGTM because this PR is authored from the wayyoungboy branch and still needs independent maintainer review/acceptance.

[lightzt99]

@wayyoungboy — \_quality_score is inconsistent across backends.

The FTS path assigns _quality_score with different semantics on the two backends:

SQLite (src/powermem/storage/sqlite/sqlite_vector_store.py, _fulltext_search)

payload['_fts_score'] = float(fts_score)
payload['_quality_score'] = 1.0   # hardcoded

OceanBase (src/powermem/storage/oceanbase/oceanbase.py, _fulltext_search)

metadata['_fts_score'] = fts_score
metadata['_quality_score'] = fts_score   # raw BM25 score (magnitude ~1e-6)

Memory.search filters by _quality_score:

# core/memory.py
quality_score = result.get("_quality_score")
if quality_score is None:
    quality_score = metadata.get("_quality_score")
if quality_score is not None and quality_score < threshold:
    continue

With threshold=0.5, behavior diverges:

Backend	FTS hits filtered by threshold
SQLite	All pass (1.0 ≥ 0.5) — threshold is effectively a no-op for FTS
OceanBase	Almost all filtered out (BM25 ~1e-6 ≪ 0.5) — threshold is overly aggressive for FTS

Same code, same threshold parameter, opposite behavior on the two backends. Users switching storage backends hit a silent contract split.

Note: _rrf_fusion uses rank position (1/(k+rank)), not _quality_score, so this inconsistency only affects threshold filtering on the pure-FTS path, not hybrid ranking.

Side note: please also confirm Memory.search and AsyncMemory.search stay aligned — the patch doesn't show the full async search body, so it's worth verifying both sides handle _quality_score identically.

---

For coordination: PR #1091 will be revised to drop its automatic FTS-fallback contract (which conflicted with this PR's "explicit no-embedding mode stays empty by default" stance) and will wait for #1052 to land before adding complementary MCP-layer retrieval_mode exposure. The two PRs should stay complementary rather than competing.