feat: PostgreSQL-backed search/document storage as ElasticSearch alternative by pandor4u · Pull Request #700 · moqui/moqui-framework

pandor4u · 2026-02-28T00:18:05Z

PostgreSQL-Backed Search & Document Storage

This PR adds a new postgres ElasticClient type that implements the full ElasticFacade.ElasticClient interface using PostgreSQL — eliminating the need for a separate ElasticSearch/OpenSearch cluster.

Motivation

Many Moqui deployments already run PostgreSQL. Requiring a separate ElasticSearch/OpenSearch cluster adds operational complexity, memory overhead, and cost — especially for small-to-medium deployments. This provides a zero-dependency alternative that uses PostgreSQL's native JSONB, tsvector full-text search, and GIN indexes.

Changes

New Files

PostgresElasticClient.groovy — Full ElasticClient implementation: index/get/update/delete/bulk/search/count operations backed by PostgreSQL tables
ElasticQueryTranslator.groovy — Translates ElasticSearch Query DSL (bool, term, terms, range, nested, exists, match_all, query_string, ids) into parameterized PostgreSQL SQL
PostgresSearchLogger.groovy — Log4j2 appender that writes structured logs to PostgreSQL instead of ES
SearchEntities.xml — Moqui entity definitions for moqui_search_index, moqui_document, moqui_logs, moqui_http_log with JSONB columns, tsvector, and GIN/BRIN indexes
moqui-postgres-only-compose.yml — Docker Compose for postgres-only deployment

Modified Files

ElasticFacadeImpl.groovy — Added type="postgres" client instantiation path
ElasticRequestLogFilter.groovy — Updated to work with postgres client
MoquiDefaultConf.xml — Default configuration for postgres search
moqui-conf-3.xsd — Schema update for type attribute on elastic-client
build.gradle — PostgreSQL JDBC driver dependency
MoquiSuite.groovy — Test suite registration

Tests (83 tests, all passing)

PostgresSearchTranslatorTests.groovy — 46 unit tests for query translation including 13 SQL injection prevention tests
PostgresElasticClientTests.groovy — 37 integration tests for CRUD, bulk, search, count, delete-by-query
PostgresSearchSuite.groovy — JUnit test suite

Security

All field names validated against ^[a-zA-Z0-9_@][a-zA-Z0-9_.\-]*$ regex before SQL interpolation
Double-dash (--) SQL comments explicitly blocked
LIMIT/OFFSET use JDBC parameterized queries (not string interpolation)
Docker compose uses environment variable references for credentials
TLS 1.2 minimum enforced in Docker nginx proxy
13 dedicated security tests verify SQL injection rejection

Configuration

<elastic-client name="default" type="postgres" cluster-name="my-app"/>

Set type="postgres" on any elastic-client element in your Moqui XML configuration. The client will use the transactional entity group's database connection.

Testing

All 166 Postgres-related tests pass (83 per suite × 2 suite runs). No regressions in existing framework tests.

schue · 2026-03-27T21:53:21Z

Here is some feedback from GLM-5.1

PR #700 Evaluation

Feature: Replaces ElasticSearch/OpenSearch with a PostgreSQL-backed search backend (JSONB + tsvector) via the existing ElasticFacade.ElasticClient interface.

What's in scope (appropriate for upstream)

Core framework changes in framework/: ElasticFacadeImpl, ElasticRequestLogFilter, MoquiDefaultConf.xml, XSD schema, build.gradle, MoquiSuite.groovy
New implementations: PostgresElasticClient (1276 lines), ElasticQueryTranslator (675 lines), PostgresSearchLogger (244 lines), SearchEntities.xml
Tests: PostgresSearchTranslatorTests (unit), PostgresElasticClientTests (integration), PostgresSearchSuite

What does NOT belong upstream

AGENTS.md, CLAUDE.md, GEMINI.md — fork-specific AI tooling instructions, all ~99% identical (510 lines of redundancy). They reference moqui-flutter, moqui-postgreonly, etc.
docker/moqui-postgres-only-compose.yml — fork-specific Docker setup
.gitignore additions — many are local dev tooling artifacts (.playwright-mcp/, *_PLAN.md, test_*.py, etc.)

Key technical concerns

Fragile guessCastType() — uses field name heuristics (contains "date" → timestamp, "amount" → numeric). Will produce wrong SQL casts for unconventional naming.
SQL copy-pasted 3x — the INSERT ... ON CONFLICT for documents appears identically in upsertDocument(), bulkIndex(), and bulkIndexDataDocument(). Should be a shared constant.
minimum_should_match > 1 not implemented — code comment says "use CASE/SUM trick" but just uses OR, so the parameter is effectively ignored.
Highlight support is a stub — uses Java regex instead of PostgreSQL ts_headline().
Integration tests in main suite — PostgresSearchSuite is added to MoquiSuite, meaning the full test suite will fail without a live PostgreSQL. These should be opt-in or a separate profile.
destroy() uses instanceof checks — could just call .destroy() polymorphically since both implement ElasticClient.
Raw REST API methods throw UnsupportedOperationException — any code using call(), callFuture(), or makeRestClient() will fail silently at runtime.

Overall assessment

The architecture is solid — clean Strategy pattern, backward-compatible, good SQL injection prevention, proper parameterized queries. The main issue is that this PR mixes the legitimate framework feature with a lot of fork-specific baggage (3 identical AI instruction files, fork-specific gitignore entries, fork-specific docker compose). If you're evaluating this for upstream merge, those should be stripped out first, the integration test suite issue should be fixed, and the duplicated SQL should be consolidated.

…rnative Simplified PR addressing review feedback: - Consolidated duplicated INSERT...ON CONFLICT SQL into shared constant - Removed fork-specific files (AGENTS.md, CLAUDE.md, GEMINI.md, docker compose, .gitignore) - Moved PostgreSQL tests to separate opt-in suite (MoquiSuite untouched) - Updated ElasticRequestLogFilter to use ElasticClient interface type - 83 tests: 46 unit (query translation + SQL injection) + 37 integration New files: - PostgresElasticClient.groovy: Full ElasticClient impl using JSONB/tsvector - ElasticQueryTranslator.groovy: ES Query DSL to PostgreSQL SQL - PostgresSearchLogger.groovy: Log4j2 appender for PostgreSQL - SearchEntities.xml: Entity definitions for search tables - PostgresSearchSuite.groovy: Separate JUnit test suite - PostgresSearchTranslatorTests.groovy: Unit tests - PostgresElasticClientTests.groovy: Integration tests Modified files: - ElasticFacadeImpl.groovy: type=postgres routing - ElasticRequestLogFilter.groovy: Interface type usage - MoquiDefaultConf.xml: Postgres config + entity load - moqui-conf-3.xsd: type attribute with elastic/postgres enum - build.gradle: Test dependencies and suite include

…ne, parameterized queries - Route HTTP logs to dedicated moqui_http_log table with typed columns instead of generic moqui_document JSONB storage - Route deleteByQuery to dedicated tables (moqui_http_log, moqui_logs) with proper timestamp range extraction for nightly cleanup jobs - Replace Java regex highlights with PostgreSQL ts_headline() for accurate, index-aware snippet generation - Fix update() to read-merge-extract pattern so content_text stays consistent with extractContentText() used by index() - Add searchHttpLogTable() for searching against dedicated HTTP log table - Improve guessCastType() to inspect actual values (epoch millis, decimals, ISO dates) when field name heuristics are ambiguous - Parameterize exists query (use ?? operator) to prevent SQL injection - Handle indexExists/createIndex/count for dedicated table names

pandor4u · 2026-04-10T23:07:11Z

@schue Thanks for the thorough review and for running it through GLM-5.1 — the feedback was very helpful. I've addressed the concerns across two commits. Here's a breakdown:

Fork-specific baggage — removed ✅

The simplified PR (commit 2553448) stripped out everything that didn't belong upstream:

Removed AGENTS.md, CLAUDE.md, GEMINI.md (the 510 lines of redundant AI tooling instructions)
Removed docker/moqui-postgres-only-compose.yml
Cleaned up .gitignore — removed all local dev tooling entries (.playwright-mcp/, *_PLAN.md, test_*.py, etc.)

Technical concerns — addressed

1. Fragile guessCastType() ✅ — Now takes an optional sampleValue parameter and inspects actual values when field-name heuristics are ambiguous. Detects epoch millisecond patterns, decimal numbers, and ISO date strings from the query values themselves, falling back to text when uncertain.

2. Duplicated SQL ✅ — The INSERT ... ON CONFLICT statement was consolidated into a shared UPSERT_SQL constant used by upsertDocument(), bulkIndex(), and bulkIndexDataDocument().

3. minimum_should_match > 1 — Acknowledged limitation. The current implementation uses OR semantics for should clauses. Implementing true minimum_should_match > 1 would require a CASE/SUM subquery pattern that adds significant complexity for a feature I haven't seen used in the Moqui codebase. Happy to add it if there's a real use case.

4. Highlight support ✅ — Replaced the Java regex stub with PostgreSQL's native ts_headline(). Highlights are now generated in SQL as additional columns in the SELECT, so they're accurate, index-aware, and respect the actual tsvector tokenization. The old simpleHighlight() / buildHighlights() methods have been removed entirely.

5. Integration tests ✅ — PostgresSearchSuite was removed from MoquiSuite. The postgres tests are now standalone and won't run (or fail) as part of the standard framework test suite. They require a live PostgreSQL instance and can be run explicitly when needed.

6. destroy() instanceof checks ✅ — Fixed to use polymorphic .destroy() since both implementations satisfy the ElasticClient interface.

7. Raw REST API methods — These (call(), callFuture(), makeRestClient()) intentionally throw UnsupportedOperationException because there's no HTTP endpoint to delegate to — the postgres client talks directly to the database via JDBC. I've verified that no core Moqui code paths call these methods; they're only used by code that explicitly targets an HTTP-based ES cluster. If a caller tries to use them, the exception message clearly states the limitation rather than failing silently.

Additional improvements in the second commit

Beyond the review feedback, I identified and fixed several issues from reading through the Moqui codebase more carefully:

HTTP log routing: ElasticRequestLogFilter calls bulkIndex("moqui_http_log", ...) but the original code stored everything in generic moqui_document as JSONB. Now HTTP logs are routed to the dedicated moqui_http_log table with proper typed columns (remote_host, request_url, status_code, etc.), matching what the nightly cleanup jobs expect.
deleteByQuery for cleanup jobs: The nightly scheduled services (clean_ElasticSearchLogMessages_daily, clean_ElasticSearchHttpLogMessages_daily) issue deleteByQuery calls with timestamp ranges. These now correctly route to moqui_http_log and moqui_logs with proper timestamp range extraction from the ES query DSL (supports both epoch milliseconds and ISO date formats).
update() consistency: Rewrote to a read-merge-extract pattern — reads the existing document, merges the partial update in Java, then regenerates content_text using the same extractContentText() as index(). The previous approach used a SQL-only jsonb_each_text which only handled top-level string fields.
exists query parameterization: The exists field query was using string interpolation (document ? '${field}'). Now uses JDBC-parameterized document ?? ? (where ?? is the JDBC escape for PostgreSQL's ? jsonb key-exists operator).
Dedicated table awareness: indexExists(), createIndex(), and count() now properly handle dedicated table names (moqui_http_log, moqui_logs) instead of trying to look them up in moqui_search_index.

All changes compile cleanly (compileGroovy + compileTestGroovy). Happy to make further adjustments if needed.

schue · 2026-07-17T21:11:50Z

I ran Qwen 3.6 on it looking for problems and it made the following suggestions. I checked the testing and getPitId issues and they seem to be real.

#	Issue	Effort
1	Resolve entity definition vs. initSchema column type conflicts	High — needs entity type extension or sync exclusion
2	Make FTS language configurable per-cluster / per-index	Medium
3	Handle `logging` datasource group transition cleanly	Medium — runtime detection or config migration
4	Implement deep merge in `update()`	Medium
5	Harden SQL injection surface (nested paths, cast types)	Medium — add sanitization to all interpolation points
6	Add partitioning or cleanup strategy for log tables	High
7	Move DDL out of transaction or use `IF NOT EXISTS` per-statement with error recovery	Medium
8	Reconsider `jsonb_path_ops` GIN index for large documents	Medium — selective indexing or remove
9	Improve `guessCastType()` with mapping-based type detection	Medium
10	Remove or reconsider `@CompileStatic`	Low
11	Document aggregation limitation clearly	Low
12	Make `trackTotal` configurable (default to `true` for compat)	Low
13	Implement real PIT with MVCC snapshot or serializable isolation	High
14	Gate Postgres test suite behind a system property or profile	Low

apathforward · 2026-07-18T01:25:13Z

Can you make it generate more detailed issues and code suggestions? Collaboration on the branch to harden it before pushing upstream.

…

On Fri, Jul 17, 2026 at 11:12 AM Ean Schuessler ***@***.***> wrote: *schue* left a comment (moqui/moqui-framework#700) <#700 (comment)> I ran Qwen 3.6 on it looking for problems and it made the following suggestions. I checked the testing and getPitId issues and they seem to be real. # Issue Effort 1 Resolve entity definition vs. initSchema column type conflicts High — needs entity type extension or sync exclusion 2 Make FTS language configurable per-cluster / per-index Medium 3 Handle logging datasource group transition cleanly Medium — runtime detection or config migration 4 Implement deep merge in update() Medium 5 Harden SQL injection surface (nested paths, cast types) Medium — add sanitization to all interpolation points 6 Add partitioning or cleanup strategy for log tables High 7 Move DDL out of transaction or use IF NOT EXISTS per-statement with error recovery Medium 8 Reconsider jsonb_path_ops GIN index for large documents Medium — selective indexing or remove 9 Improve guessCastType() with mapping-based type detection Medium 10 Remove or reconsider @CompileStatic Low 11 Document aggregation limitation clearly Low 12 Make trackTotal configurable (default to true for compat) Low 13 Implement real PIT with MVCC snapshot or serializable isolation High 14 Gate Postgres test suite behind a system property or profile Low — Reply to this email directly, view it on GitHub <#700?email_source=notifications&email_token=AFK5KLVC4456R4QKDUWSYM35FKJDDA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTKMBQG42TCNZQHE4KM4TFMFZW63VKON2WE43DOJUWEZLEUVSXMZLOOSWGM33PORSXEX3DNRUWG2Y#issuecomment-5007517098>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFK5KLQZ3J4YK3OKGFPT2N35FKJDDAVCNFSNUABEKJSXA33TNF2G64TZHM2TAMRWGM2TMMJ3JFZXG5LFHM2DAMBTGM4TKNZVGKQXMAQ> . Triage notifications, keep track of coding agent tasks and review pull requests on the go with GitHub Mobile for iOS <https://github.com/notifications/mobile/ios/AFK5KLVRIE7CP2AQWQYQ2IT5FKJDDA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTKMBQG42TCNZQHE4KM4TFMFZW63VKON2WE43DOJUWEZLEUVSXMZLOOSVGM33PORSXEX3JN5ZQ> and Android <https://github.com/notifications/mobile/android/AFK5KLRJRXKNP4SC76VZMHT5FKJDDA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTKMBQG42TCNZQHE4KM4TFMFZW63VKON2WE43DOJUWEZLEUVSXMZLOOSXGM33PORSXEX3BNZSHE33JMQ>. Download it today! You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

-- Guru Dharam Singh Khalsa

schue · 2026-07-18T02:20:25Z

PROBLEMS.md

Hardens the type=postgres ElasticClient in response to the PR moqui#700 review. Each item maps to PROBLEMS_RESOLUTION_PLAN.md: - moqui#1 Schema/initSchema type conflict: ensureColumnType() heals document/content_tsv/mdc/thrown columns to JSONB/TSVECTOR regardless of entity-sync ordering; failed casts now log at .warn and the resulting type is verified. Schema-ownership note added to SearchEntities.xml. - moqui#2 Hardcoded 'english' FTS language: new text-config cluster attribute (validated safe-identifier, default english) threaded through to_tsvector/websearch_to_tsquery/ts_headline; declared in moqui-conf-3.xsd. - moqui#4 Shallow merge in update(): recursive deepMerge() preserves nested sibling keys (ES _update semantics). - moqui#5 SQL-injection hardening: ALLOWED_CAST_TYPES whitelist assertion on guessCastType() output. - moqui#6 Unbounded log deletes: batchedDelete() chunks retention deletes via ctid+LIMIT. - moqui#7 DDL-in-transaction: execRequired()/execOptional() so optional steps (pg_trgm, sequence heal) no longer abort init; fatal path logs the failing statement. - moqui#8 jsonb_path_ops GIN index now optional via enable-jsonb-path-index cluster attribute (declared in moqui-conf-3.xsd). - moqui#9 guessCastType() prefers declared ES mapping types (per-cluster fieldTypeMapCache) over name/value heuristics. - moqui#11 search() logs a one-time warn when aggs/aggregations are requested (unsupported). - moqui#12 Single-query total via count(*) OVER() (falls back to standalone COUNT for keyset cursor / empty page). - moqui#13 PIT: real ES-style search_after keyset pagination (updated_stamp,doc_id) with documented snapshot limitation. - moqui#14 PostgresSearchSuite gated behind -PincludePostgresSearch / MOQUI_PG_SEARCH=true (out of default test run). - moqui#3, moqui#10 doc/comment corrections (logging datasource group; @CompileStatic intent). Tests: PostgresElasticClientTests (40) + PostgresSearchTranslatorTests (51), all passing.

- FieldInfo.setPreparedStatementValue(): when a String/CharSequence value is bound to a numeric field (int/long/float/double/BigDecimal), parse and bind it as the actual number type. PostgreSQL (unlike H2/MySQL) does not implicitly cast a varchar bind param to numeric and fails with 'operator does not exist: numeric = character varying'. Re-enables the previously-disabled EntityFindTests 'testNumberInteger | "4321"' case. - Make the default :framework:test (MoquiSuite) suite repeatable against the persistent local PostgreSQL test DB: move order-dependent fixture cleanup to the correct consumer/cleanupSpec in EntityFindTests (ArtifactAuthz), EntityNoSqlCrud (200 BULK* rows), and ToolsScreenRenderTests (TestEntity/UserAccount/DbView* rows).

pandor4u · 2026-07-18T21:54:53Z

Please review the updates and let me know if further changes are needed.

…

On Fri, Jul 17, 2026 at 4:20 PM Ean Schuessler ***@***.***> wrote: *schue* left a comment (moqui/moqui-framework#700) <#700 (comment)> PROBLEMS.md <https://github.com/user-attachments/files/30145023/PROBLEMS.md> — Reply to this email directly, view it on GitHub <#700?email_source=notifications&email_token=AYZI4FVI4QS3AU2SO3WDWML5FLNH7A5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTKMBQHE2DINBSGE2KM4TFMFZW63VGMF2XI2DPOKSWK5TFNZ2KYZTPN52GK4S7MNWGSY3L#issuecomment-5009444214>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AYZI4FW5XXM2MFA73MXTK2D5FLNH7AVCNFSNUABEKJSXA33TNF2G64TZHM2TAMRWGM2TMMJ3JFZXG5LFHM2DAMBTGM4TKNZVGKQXMAQ> . You are receiving this because you authored the thread.Message ID: ***@***.***>

pandor4u · 2026-07-18T21:56:37Z

And these are separate changes. Not sure if they're relevant or not. https://github.com/pandor4u/moqui-framework/tree/wip/java17-tooling-and-dev-scaffold On Sat, Jul 18, 2026 at 11:54 AM Pandor Photos ***@***.***> wrote:

…

Please review the updates and let me know if further changes are needed. On Fri, Jul 17, 2026 at 4:20 PM Ean Schuessler ***@***.***> wrote: > *schue* left a comment (moqui/moqui-framework#700) > <#700 (comment)> > > PROBLEMS.md > <https://github.com/user-attachments/files/30145023/PROBLEMS.md> > > — > Reply to this email directly, view it on GitHub > <#700?email_source=notifications&email_token=AYZI4FVI4QS3AU2SO3WDWML5FLNH7A5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTKMBQHE2DINBSGE2KM4TFMFZW63VGMF2XI2DPOKSWK5TFNZ2KYZTPN52GK4S7MNWGSY3L#issuecomment-5009444214>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/AYZI4FW5XXM2MFA73MXTK2D5FLNH7AVCNFSNUABEKJSXA33TNF2G64TZHM2TAMRWGM2TMMJ3JFZXG5LFHM2DAMBTGM4TKNZVGKQXMAQ> > . > You are receiving this because you authored the thread.Message ID: > ***@***.***> >

pandor4u force-pushed the feature/postgres-search-backend branch from 1d1ebb4 to 2553448 Compare April 10, 2026 22:46

pandor4u added 2 commits July 18, 2026 11:50

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: PostgreSQL-backed search/document storage as ElasticSearch alternative#700

feat: PostgreSQL-backed search/document storage as ElasticSearch alternative#700
pandor4u wants to merge 4 commits into
moqui:masterfrom
pandor4u:feature/postgres-search-backend

pandor4u commented Feb 28, 2026

Uh oh!

schue commented Mar 27, 2026

Uh oh!

pandor4u commented Apr 10, 2026

Uh oh!

schue commented Jul 17, 2026

Uh oh!

apathforward commented Jul 18, 2026 via email

Uh oh!

schue commented Jul 18, 2026

Uh oh!

pandor4u commented Jul 18, 2026 via email

Uh oh!

pandor4u commented Jul 18, 2026 via email

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

pandor4u commented Feb 28, 2026

PostgreSQL-Backed Search & Document Storage

Motivation

Changes

New Files

Modified Files

Tests (83 tests, all passing)

Security

Configuration

Testing

Uh oh!

schue commented Mar 27, 2026

PR #700 Evaluation

What's in scope (appropriate for upstream)

What does NOT belong upstream

Key technical concerns

Overall assessment

Uh oh!

pandor4u commented Apr 10, 2026

Fork-specific baggage — removed ✅

Technical concerns — addressed

Additional improvements in the second commit

Uh oh!

schue commented Jul 17, 2026

Uh oh!

apathforward commented Jul 18, 2026 via email

Uh oh!

schue commented Jul 18, 2026

Uh oh!

pandor4u commented Jul 18, 2026 via email

Uh oh!

pandor4u commented Jul 18, 2026 via email

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

3 participants