From d1fbc76b975f0efb60f68eef4d1c99939e6ef0d7 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Andrzej=20Kobyli=C5=84ski?= Date: Sat, 16 May 2026 13:42:57 +0200 Subject: [PATCH 1/3] chore: full JMH rerun for Kafka deliverBatch with publishable config (KOJAK-82) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Replaces the smoke-run numbers (fork=1, warmup=1, iter=2, n=2, scoreError=NaN) with a full publishable run (fork=2, warmup=3 × 10s, iter=5 × 30s, n=10). ## Headline (Kafka throughput, msg/s) | batchSize | Smoke | Full run | Improvement vs baseline | |-----------|----------|----------------|-------------------------| | 10 | ~1,468 | ~1,825 ± 70 | 16.7× (was 13.5×) | | 50 | ~3,731 | ~4,184 ± 140 | 36.3× (was 32.3×) | | 100 | ~4,717 | ~5,128 ± 105 | 44.6× (was 41.0×) | All Kafka throughput error bars <5% of score — multipliers now statistically defensible. The smoke-run numbers were directionally correct but slightly conservative; full-run shows the optimization is even better than initially claimed. ## Benchmark infrastructure fixes (needed to land the rerun) - okapi-benchmarks build.gradle.kts: bump JMH JVM heap to -Xmx8g (the previous default -Xmx2g OOMed inside throughput-mode microbenches) - okapi-benchmarks build.gradle.kts: pass -Dliquibase.duplicateFileMode=WARN (okapi-postgres.jar and the fat JMH jar both carry the changelog at the same path; Liquibase 4.x treats this as an error by default; the files are identical so WARN is safe) - DelivererMicroBenchmark.kt: subclass MockProducer to clear() history after every send. MockProducer retains every record sent for inspection; at ~1M ops/s for 30s × forks × iters that list grew to GBs and OOMed the JVM regardless of heap size. The fix discards history per call — microbench doesn't need to inspect what was sent. ## Files updated - benchmarks/kafka-deliverbatch.json: replaced with full-config results - benchmarks/results-kafka-deliverbatch.md: new Score +/- Error tables; removed "Statistical caveat" callout; tightened narrative; added HTTP companion table for full-run completeness - README.md: refreshed throughput table (1,470 -> 1,825 / 4,720 -> 5,130), improvement claim (13-41x -> 17-45x), JDK note (25 -> 21) Note on JDK delta: smoke run was on JDK 25.0.2 (anomaly - SDKMAN default shifted between runs); this full run is on JDK 21.0.7. CLAUDE.md target is JVM 21 so this matches what consumers will see. DelivererMicroBenchmark.kafkaDeliver still produces high-variance results (error > score) - JIT warmup interacts poorly with the Jackson-per-call deserialization. Not a blocker for KOJAK-82 (the throughput benchmarks are the publishable surface); a follow-up could switch the microbench to AverageTime mode or cache the deserialized DeliveryInfo. --- README.md | 8 +- benchmarks/kafka-deliverbatch.json | 893 ++++++++++++++++-- benchmarks/results-kafka-deliverbatch.md | 56 +- okapi-benchmarks/build.gradle.kts | 14 +- .../benchmarks/DelivererMicroBenchmark.kt | 16 +- 5 files changed, 890 insertions(+), 97 deletions(-) diff --git a/README.md b/README.md index 41d1e42..c1cc4fe 100644 --- a/README.md +++ b/README.md @@ -265,15 +265,15 @@ graph BT ## Performance -Throughput on a single instance (MacBook M3 Max, JDK 25 LTS, May 2026): +Throughput on a single instance (MacBook M3 Max, JDK 21 LTS, May 2026): | Transport | batchSize=10 | batchSize=100 | |-----------|--------------|----------------| -| Kafka (`acks=all`, localhost broker, async batch via `deliverBatch`) | **~1,470 msg/s** | **~4,720 msg/s** | -| HTTP @ webhook latency 20 ms (sync sequential — parallel `sendAsync` planned) | ~33 msg/s | ~36 msg/s | +| Kafka (`acks=all`, localhost broker, async batch via `deliverBatch`) | **~1,825 msg/s** | **~5,130 msg/s** | +| HTTP @ webhook latency 20 ms (sync sequential — parallel `sendAsync` planned) | ~38 msg/s | ~36 msg/s | | HTTP @ webhook latency 100 ms (sync sequential — parallel `sendAsync` planned) | ~9 msg/s | ~9 msg/s | -Kafka throughput jumped 13-41× over the original sync-sequential baseline thanks to the `deliverBatch` fire-flush-await pattern. HTTP parallel `sendAsync` is next; multi-threaded scheduler scaling is in the roadmap. +Kafka throughput jumped 17-45× over the original sync-sequential baseline thanks to the `deliverBatch` fire-flush-await pattern. HTTP parallel `sendAsync` is next; multi-threaded scheduler scaling is in the roadmap. Full methodology, raw JMH results, before/after per change: [`benchmarks/`](benchmarks/). diff --git a/benchmarks/kafka-deliverbatch.json b/benchmarks/kafka-deliverbatch.json index 907ced0..925ca1c 100644 --- a/benchmarks/kafka-deliverbatch.json +++ b/benchmarks/kafka-deliverbatch.json @@ -1,49 +1,776 @@ [ + { + "jmhVersion" : "1.37", + "benchmark" : "com.softwaremill.okapi.benchmarks.DelivererMicroBenchmark.httpDeliver", + "mode" : "thrpt", + "threads" : 1, + "forks" : 2, + "jvm" : "/Users/andrzej.kobylinski/.sdkman/candidates/java/21.0.7-tem/bin/java", + "jvmArgs" : [ + "-Xms8g", + "-Xmx8g", + "-XX:+UseG1GC", + "-Dliquibase.duplicateFileMode=WARN" + ], + "jdkVersion" : "21.0.7", + "vmName" : "OpenJDK 64-Bit Server VM", + "vmVersion" : "21.0.7+6-LTS", + "warmupIterations" : 3, + "warmupTime" : "10 s", + "warmupBatchSize" : 1, + "measurementIterations" : 5, + "measurementTime" : "30 s", + "measurementBatchSize" : 1, + "primaryMetric" : { + "score" : 11046.989954546045, + "scoreError" : 1004.4482086685881, + "scoreConfidence" : [ + 10042.541745877457, + 12051.438163214632 + ], + "scorePercentiles" : { + "0.0" : 9183.283753408647, + "50.0" : 11210.006057371334, + "90.0" : 11465.458003094589, + "95.0" : 11470.9608769683, + "99.0" : 11470.9608769683, + "99.9" : 11470.9608769683, + "99.99" : 11470.9608769683, + "99.999" : 11470.9608769683, + "99.9999" : 11470.9608769683, + "100.0" : 11470.9608769683 + }, + "scoreUnit" : "ops/s", + "rawData" : [ + [ + 9183.283753408647, + 11470.9608769683, + 11415.932138231185, + 11279.390571003483, + 11100.527483759102 + ], + [ + 11192.57798143765, + 11227.434133305022, + 11240.813386669752, + 11183.232491022263, + 11175.746729655039 + ] + ] + }, + "secondaryMetrics" : { + } + }, + { + "jmhVersion" : "1.37", + "benchmark" : "com.softwaremill.okapi.benchmarks.DelivererMicroBenchmark.kafkaDeliver", + "mode" : "thrpt", + "threads" : 1, + "forks" : 2, + "jvm" : "/Users/andrzej.kobylinski/.sdkman/candidates/java/21.0.7-tem/bin/java", + "jvmArgs" : [ + "-Xms8g", + "-Xmx8g", + "-XX:+UseG1GC", + "-Dliquibase.duplicateFileMode=WARN" + ], + "jdkVersion" : "21.0.7", + "vmName" : "OpenJDK 64-Bit Server VM", + "vmVersion" : "21.0.7+6-LTS", + "warmupIterations" : 3, + "warmupTime" : "10 s", + "warmupBatchSize" : 1, + "measurementIterations" : 5, + "measurementTime" : "30 s", + "measurementBatchSize" : 1, + "primaryMetric" : { + "score" : 16846.59076370828, + "scoreError" : 65284.79310508455, + "scoreConfidence" : [ + -48438.20234137627, + 82131.38386879282 + ], + "scorePercentiles" : { + "0.0" : 1999.5466281300162, + "50.0" : 6323.3247330732465, + "90.0" : 35614.633917227206, + "95.0" : 35614.633917227206, + "99.0" : 35614.633917227206, + "99.9" : 35614.633917227206, + "99.99" : 35614.633917227206, + "99.999" : 35614.633917227206, + "99.9999" : 35614.633917227206, + "100.0" : 35614.633917227206 + }, + "scoreUnit" : "ops/s", + "rawData" : [ + [ + 35057.093536099164, + 6323.3247330732465, + 1999.5466281300162 + ], + [ + 35614.633917227206, + 5238.355004011761 + ] + ] + }, + "secondaryMetrics" : { + } + }, + { + "jmhVersion" : "1.37", + "benchmark" : "com.softwaremill.okapi.benchmarks.HttpThroughputBenchmark.drainAll", + "mode" : "avgt", + "threads" : 1, + "forks" : 2, + "jvm" : "/Users/andrzej.kobylinski/.sdkman/candidates/java/21.0.7-tem/bin/java", + "jvmArgs" : [ + "-Xms8g", + "-Xmx8g", + "-XX:+UseG1GC", + "-Dliquibase.duplicateFileMode=WARN" + ], + "jdkVersion" : "21.0.7", + "vmName" : "OpenJDK 64-Bit Server VM", + "vmVersion" : "21.0.7+6-LTS", + "warmupIterations" : 3, + "warmupTime" : "10 s", + "warmupBatchSize" : 1, + "measurementIterations" : 5, + "measurementTime" : "30 s", + "measurementBatchSize" : 1, + "params" : { + "batchSize" : "10", + "httpLatencyMs" : "0" + }, + "primaryMetric" : { + "score" : 0.6648509935410607, + "scoreError" : 0.047816447926900846, + "scoreConfidence" : [ + 0.6170345456141598, + 0.7126674414679616 + ], + "scorePercentiles" : { + "0.0" : 0.6351569900526316, + "50.0" : 0.6515210318289473, + "90.0" : 0.7188523387314286, + "95.0" : 0.7203417440857143, + "99.0" : 0.7203417440857143, + "99.9" : 0.7203417440857143, + "99.99" : 0.7203417440857143, + "99.999" : 0.7203417440857143, + "99.9999" : 0.7203417440857143, + "100.0" : 0.7203417440857143 + }, + "scoreUnit" : "ms/op", + "rawData" : [ + [ + 0.6465957598421053, + 0.6514883388421052, + 0.6355883640526315, + 0.6439560119736842, + 0.6568196599459459 + ], + [ + 0.7203417440857143, + 0.7054476905428572, + 0.7015616512571429, + 0.6351569900526316, + 0.6515537248157894 + ] + ] + }, + "secondaryMetrics" : { + } + }, + { + "jmhVersion" : "1.37", + "benchmark" : "com.softwaremill.okapi.benchmarks.HttpThroughputBenchmark.drainAll", + "mode" : "avgt", + "threads" : 1, + "forks" : 2, + "jvm" : "/Users/andrzej.kobylinski/.sdkman/candidates/java/21.0.7-tem/bin/java", + "jvmArgs" : [ + "-Xms8g", + "-Xmx8g", + "-XX:+UseG1GC", + "-Dliquibase.duplicateFileMode=WARN" + ], + "jdkVersion" : "21.0.7", + "vmName" : "OpenJDK 64-Bit Server VM", + "vmVersion" : "21.0.7+6-LTS", + "warmupIterations" : 3, + "warmupTime" : "10 s", + "warmupBatchSize" : 1, + "measurementIterations" : 5, + "measurementTime" : "30 s", + "measurementBatchSize" : 1, + "params" : { + "batchSize" : "10", + "httpLatencyMs" : "20" + }, + "primaryMetric" : { + "score" : 26.4836798541, + "scoreError" : 0.7376214347051937, + "scoreConfidence" : [ + 25.746058419394807, + 27.221301288805194 + ], + "scorePercentiles" : { + "0.0" : 26.0440017495, + "50.0" : 26.204994541749997, + "90.0" : 27.408618223050002, + "95.0" : 27.429925521, + "99.0" : 27.429925521, + "99.9" : 27.429925521, + "99.99" : 27.429925521, + "99.999" : 27.429925521, + "99.9999" : 27.429925521, + "100.0" : 27.429925521 + }, + "scoreUnit" : "ms/op", + "rawData" : [ + [ + 26.559552, + 26.1910378955, + 26.1946212915, + 26.0440017495, + 26.6895338125 + ], + [ + 27.429925521, + 27.2168525415, + 26.121925, + 26.1739809375, + 26.215367792 + ] + ] + }, + "secondaryMetrics" : { + } + }, + { + "jmhVersion" : "1.37", + "benchmark" : "com.softwaremill.okapi.benchmarks.HttpThroughputBenchmark.drainAll", + "mode" : "avgt", + "threads" : 1, + "forks" : 2, + "jvm" : "/Users/andrzej.kobylinski/.sdkman/candidates/java/21.0.7-tem/bin/java", + "jvmArgs" : [ + "-Xms8g", + "-Xmx8g", + "-XX:+UseG1GC", + "-Dliquibase.duplicateFileMode=WARN" + ], + "jdkVersion" : "21.0.7", + "vmName" : "OpenJDK 64-Bit Server VM", + "vmVersion" : "21.0.7+6-LTS", + "warmupIterations" : 3, + "warmupTime" : "10 s", + "warmupBatchSize" : 1, + "measurementIterations" : 5, + "measurementTime" : "30 s", + "measurementBatchSize" : 1, + "params" : { + "batchSize" : "10", + "httpLatencyMs" : "100" + }, + "primaryMetric" : { + "score" : 110.3537548583, + "scoreError" : 0.47119739319743864, + "scoreConfidence" : [ + 109.88255746510255, + 110.82495225149744 + ], + "scorePercentiles" : { + "0.0" : 109.992686416, + "50.0" : 110.3106146045, + "90.0" : 111.00056357119999, + "95.0" : 111.042992542, + "99.0" : 111.042992542, + "99.9" : 111.042992542, + "99.99" : 111.042992542, + "99.999" : 111.042992542, + "99.9999" : 111.042992542, + "100.0" : 111.042992542 + }, + "scoreUnit" : "ms/op", + "rawData" : [ + [ + 110.180464375, + 110.618702834, + 110.270252584, + 109.992686416, + 110.480257083 + ], + [ + 111.042992542, + 110.384261166, + 110.350976625, + 109.998136833, + 110.218818125 + ] + ] + }, + "secondaryMetrics" : { + } + }, + { + "jmhVersion" : "1.37", + "benchmark" : "com.softwaremill.okapi.benchmarks.HttpThroughputBenchmark.drainAll", + "mode" : "avgt", + "threads" : 1, + "forks" : 2, + "jvm" : "/Users/andrzej.kobylinski/.sdkman/candidates/java/21.0.7-tem/bin/java", + "jvmArgs" : [ + "-Xms8g", + "-Xmx8g", + "-XX:+UseG1GC", + "-Dliquibase.duplicateFileMode=WARN" + ], + "jdkVersion" : "21.0.7", + "vmName" : "OpenJDK 64-Bit Server VM", + "vmVersion" : "21.0.7+6-LTS", + "warmupIterations" : 3, + "warmupTime" : "10 s", + "warmupBatchSize" : 1, + "measurementIterations" : 5, + "measurementTime" : "30 s", + "measurementBatchSize" : 1, + "params" : { + "batchSize" : "50", + "httpLatencyMs" : "0" + }, + "primaryMetric" : { + "score" : 0.3204392950207692, + "scoreError" : 0.0037847027765500526, + "scoreConfidence" : [ + 0.3166545922442191, + 0.32422399779731925 + ], + "scorePercentiles" : { + "0.0" : 0.31664638847692306, + "50.0" : 0.319805524046875, + "90.0" : 0.32403715804375, + "95.0" : 0.324049039671875, + "99.0" : 0.324049039671875, + "99.9" : 0.324049039671875, + "99.99" : 0.324049039671875, + "99.999" : 0.324049039671875, + "99.9999" : 0.324049039671875, + "100.0" : 0.324049039671875 + }, + "scoreUnit" : "ms/op", + "rawData" : [ + [ + 0.323049698546875, + 0.3194230963125, + 0.31894084927692307, + 0.323930223390625, + 0.320670370484375 + ], + [ + 0.31945494134375, + 0.32015610675, + 0.324049039671875, + 0.31807223595384615, + 0.31664638847692306 + ] + ] + }, + "secondaryMetrics" : { + } + }, + { + "jmhVersion" : "1.37", + "benchmark" : "com.softwaremill.okapi.benchmarks.HttpThroughputBenchmark.drainAll", + "mode" : "avgt", + "threads" : 1, + "forks" : 2, + "jvm" : "/Users/andrzej.kobylinski/.sdkman/candidates/java/21.0.7-tem/bin/java", + "jvmArgs" : [ + "-Xms8g", + "-Xmx8g", + "-XX:+UseG1GC", + "-Dliquibase.duplicateFileMode=WARN" + ], + "jdkVersion" : "21.0.7", + "vmName" : "OpenJDK 64-Bit Server VM", + "vmVersion" : "21.0.7+6-LTS", + "warmupIterations" : 3, + "warmupTime" : "10 s", + "warmupBatchSize" : 1, + "measurementIterations" : 5, + "measurementTime" : "30 s", + "measurementBatchSize" : 1, + "params" : { + "batchSize" : "50", + "httpLatencyMs" : "20" + }, + "primaryMetric" : { + "score" : 25.767400908299994, + "scoreError" : 1.5113747767535068, + "scoreConfidence" : [ + 24.256026131546488, + 27.2787756850535 + ], + "scorePercentiles" : { + "0.0" : 24.912139833, + "50.0" : 25.3655716355, + "90.0" : 27.71619023095, + "95.0" : 27.7563236455, + "99.0" : 27.7563236455, + "99.9" : 27.7563236455, + "99.99" : 27.7563236455, + "99.999" : 27.7563236455, + "99.9999" : 27.7563236455, + "100.0" : 27.7563236455 + }, + "scoreUnit" : "ms/op", + "rawData" : [ + [ + 26.093162042, + 25.3349228125, + 25.3962204585, + 25.2232781875, + 24.912139833 + ], + [ + 25.444147333, + 25.1441528545, + 25.0146724165, + 27.3549895, + 27.7563236455 + ] + ] + }, + "secondaryMetrics" : { + } + }, + { + "jmhVersion" : "1.37", + "benchmark" : "com.softwaremill.okapi.benchmarks.HttpThroughputBenchmark.drainAll", + "mode" : "avgt", + "threads" : 1, + "forks" : 2, + "jvm" : "/Users/andrzej.kobylinski/.sdkman/candidates/java/21.0.7-tem/bin/java", + "jvmArgs" : [ + "-Xms8g", + "-Xmx8g", + "-XX:+UseG1GC", + "-Dliquibase.duplicateFileMode=WARN" + ], + "jdkVersion" : "21.0.7", + "vmName" : "OpenJDK 64-Bit Server VM", + "vmVersion" : "21.0.7+6-LTS", + "warmupIterations" : 3, + "warmupTime" : "10 s", + "warmupBatchSize" : 1, + "measurementIterations" : 5, + "measurementTime" : "30 s", + "measurementBatchSize" : 1, + "params" : { + "batchSize" : "50", + "httpLatencyMs" : "100" + }, + "primaryMetric" : { + "score" : 108.6361489709, + "scoreError" : 1.6770227329126817, + "scoreConfidence" : [ + 106.95912623798732, + 110.31317170381267 + ], + "scorePercentiles" : { + "0.0" : 106.46893875, + "50.0" : 108.7974460625, + "90.0" : 109.8353806872, + "95.0" : 109.861833583, + "99.0" : 109.861833583, + "99.9" : 109.861833583, + "99.99" : 109.861833583, + "99.999" : 109.861833583, + "99.9999" : 109.861833583, + "100.0" : 109.861833583 + }, + "scoreUnit" : "ms/op", + "rawData" : [ + [ + 109.077493542, + 109.593541417, + 109.597304625, + 109.861833583, + 108.776664417 + ], + [ + 106.46893875, + 106.968990625, + 108.818227708, + 108.646724708, + 108.551770334 + ] + ] + }, + "secondaryMetrics" : { + } + }, + { + "jmhVersion" : "1.37", + "benchmark" : "com.softwaremill.okapi.benchmarks.HttpThroughputBenchmark.drainAll", + "mode" : "avgt", + "threads" : 1, + "forks" : 2, + "jvm" : "/Users/andrzej.kobylinski/.sdkman/candidates/java/21.0.7-tem/bin/java", + "jvmArgs" : [ + "-Xms8g", + "-Xmx8g", + "-XX:+UseG1GC", + "-Dliquibase.duplicateFileMode=WARN" + ], + "jdkVersion" : "21.0.7", + "vmName" : "OpenJDK 64-Bit Server VM", + "vmVersion" : "21.0.7+6-LTS", + "warmupIterations" : 3, + "warmupTime" : "10 s", + "warmupBatchSize" : 1, + "measurementIterations" : 5, + "measurementTime" : "30 s", + "measurementBatchSize" : 1, + "params" : { + "batchSize" : "100", + "httpLatencyMs" : "0" + }, + "primaryMetric" : { + "score" : 0.28758983040254477, + "scoreError" : 0.004970132805716064, + "scoreConfidence" : [ + 0.2826196975968287, + 0.29255996320826083 + ], + "scorePercentiles" : { + "0.0" : 0.2826907356857143, + "50.0" : 0.2876184631956522, + "90.0" : 0.2933219049207801, + "95.0" : 0.2936818799264706, + "99.0" : 0.2936818799264706, + "99.9" : 0.2936818799264706, + "99.99" : 0.2936818799264706, + "99.999" : 0.2936818799264706, + "99.9999" : 0.2936818799264706, + "100.0" : 0.2936818799264706 + }, + "scoreUnit" : "ms/op", + "rawData" : [ + [ + 0.2896490839565217, + 0.2853148454347826, + 0.2861804172173913, + 0.28379908151428573, + 0.2826907356857143 + ], + [ + 0.2936818799264706, + 0.2884446564202899, + 0.2867922699710145, + 0.28926320402941175, + 0.2900821298695652 + ] + ] + }, + "secondaryMetrics" : { + } + }, + { + "jmhVersion" : "1.37", + "benchmark" : "com.softwaremill.okapi.benchmarks.HttpThroughputBenchmark.drainAll", + "mode" : "avgt", + "threads" : 1, + "forks" : 2, + "jvm" : "/Users/andrzej.kobylinski/.sdkman/candidates/java/21.0.7-tem/bin/java", + "jvmArgs" : [ + "-Xms8g", + "-Xmx8g", + "-XX:+UseG1GC", + "-Dliquibase.duplicateFileMode=WARN" + ], + "jdkVersion" : "21.0.7", + "vmName" : "OpenJDK 64-Bit Server VM", + "vmVersion" : "21.0.7+6-LTS", + "warmupIterations" : 3, + "warmupTime" : "10 s", + "warmupBatchSize" : 1, + "measurementIterations" : 5, + "measurementTime" : "30 s", + "measurementBatchSize" : 1, + "params" : { + "batchSize" : "100", + "httpLatencyMs" : "20" + }, + "primaryMetric" : { + "score" : 27.96162871255, + "scoreError" : 0.9460369976764773, + "scoreConfidence" : [ + 27.015591714873523, + 28.907665710226475 + ], + "scorePercentiles" : { + "0.0" : 27.514304708, + "50.0" : 27.6967823545, + "90.0" : 29.4356709689, + "95.0" : 29.5428868335, + "99.0" : 29.5428868335, + "99.9" : 29.5428868335, + "99.99" : 29.5428868335, + "99.999" : 29.5428868335, + "99.9999" : 29.5428868335, + "100.0" : 29.5428868335 + }, + "scoreUnit" : "ms/op", + "rawData" : [ + [ + 29.5428868335, + 28.4707281875, + 27.6981462085, + 27.6912476875, + 27.514304708 + ], + [ + 28.0953889585, + 27.6954185005, + 27.557653333, + 27.7293343125, + 27.621178396 + ] + ] + }, + "secondaryMetrics" : { + } + }, + { + "jmhVersion" : "1.37", + "benchmark" : "com.softwaremill.okapi.benchmarks.HttpThroughputBenchmark.drainAll", + "mode" : "avgt", + "threads" : 1, + "forks" : 2, + "jvm" : "/Users/andrzej.kobylinski/.sdkman/candidates/java/21.0.7-tem/bin/java", + "jvmArgs" : [ + "-Xms8g", + "-Xmx8g", + "-XX:+UseG1GC", + "-Dliquibase.duplicateFileMode=WARN" + ], + "jdkVersion" : "21.0.7", + "vmName" : "OpenJDK 64-Bit Server VM", + "vmVersion" : "21.0.7+6-LTS", + "warmupIterations" : 3, + "warmupTime" : "10 s", + "warmupBatchSize" : 1, + "measurementIterations" : 5, + "measurementTime" : "30 s", + "measurementBatchSize" : 1, + "params" : { + "batchSize" : "100", + "httpLatencyMs" : "100" + }, + "primaryMetric" : { + "score" : 107.98585250820001, + "scoreError" : 1.3910986158983256, + "scoreConfidence" : [ + 106.59475389230168, + 109.37695112409834 + ], + "scorePercentiles" : { + "0.0" : 105.71322375, + "50.0" : 108.270090729, + "90.0" : 108.6963979577, + "95.0" : 108.715305416, + "99.0" : 108.715305416, + "99.9" : 108.715305416, + "99.99" : 108.715305416, + "99.999" : 108.715305416, + "99.9999" : 108.715305416, + "100.0" : 108.715305416 + }, + "scoreUnit" : "ms/op", + "rawData" : [ + [ + 108.526230833, + 108.458791167, + 108.2180245, + 108.229336708, + 108.153583375 + ], + [ + 108.715305416, + 108.495963708, + 108.31084475, + 107.037220875, + 105.71322375 + ] + ] + }, + "secondaryMetrics" : { + } + }, { "jmhVersion" : "1.37", "benchmark" : "com.softwaremill.okapi.benchmarks.KafkaThroughputBenchmark.drainAll", "mode" : "avgt", "threads" : 1, - "forks" : 1, - "jvm" : "/Users/andrzej.kobylinski/.sdkman/candidates/java/25.0.2-tem/bin/java", + "forks" : 2, + "jvm" : "/Users/andrzej.kobylinski/.sdkman/candidates/java/21.0.7-tem/bin/java", "jvmArgs" : [ + "-Xms8g", + "-Xmx8g", + "-XX:+UseG1GC", + "-Dliquibase.duplicateFileMode=WARN" ], - "jdkVersion" : "25.0.2", + "jdkVersion" : "21.0.7", "vmName" : "OpenJDK 64-Bit Server VM", - "vmVersion" : "25.0.2+10-LTS", - "warmupIterations" : 1, + "vmVersion" : "21.0.7+6-LTS", + "warmupIterations" : 3, "warmupTime" : "10 s", "warmupBatchSize" : 1, - "measurementIterations" : 2, - "measurementTime" : "15 s", + "measurementIterations" : 5, + "measurementTime" : "30 s", "measurementBatchSize" : 1, "params" : { "batchSize" : "10" }, "primaryMetric" : { - "score" : 0.680696006383041, - "scoreError" : "NaN", + "score" : 0.5475883323324705, + "scoreError" : 0.021066415156471875, "scoreConfidence" : [ - "NaN", - "NaN" + 0.5265219171759986, + 0.5686547474889424 ], "scorePercentiles" : { - "0.0" : 0.6561445592105263, - "50.0" : 0.680696006383041, - "90.0" : 0.7052474535555555, - "95.0" : 0.7052474535555555, - "99.0" : 0.7052474535555555, - "99.9" : 0.7052474535555555, - "99.99" : 0.7052474535555555, - "99.999" : 0.7052474535555555, - "99.9999" : 0.7052474535555555, - "100.0" : 0.7052474535555555 + "0.0" : 0.5299251648222222, + "50.0" : 0.5459132145866807, + "90.0" : 0.5747985307166086, + "95.0" : 0.5758298282195122, + "99.0" : 0.5758298282195122, + "99.9" : 0.5758298282195122, + "99.99" : 0.5758298282195122, + "99.999" : 0.5758298282195122, + "99.9999" : 0.5758298282195122, + "100.0" : 0.5758298282195122 }, "scoreUnit" : "ms/op", "rawData" : [ [ - 0.7052474535555555, - 0.6561445592105263 + 0.5655168531904762, + 0.5461031705454545, + 0.5299251648222222, + 0.5758298282195122, + 0.5457232586279069 + ], + [ + 0.5527824999302325, + 0.5373128209318182, + 0.5462132025116279, + 0.5373868304772728, + 0.5390896940681819 ] ] }, @@ -55,46 +782,60 @@ "benchmark" : "com.softwaremill.okapi.benchmarks.KafkaThroughputBenchmark.drainAll", "mode" : "avgt", "threads" : 1, - "forks" : 1, - "jvm" : "/Users/andrzej.kobylinski/.sdkman/candidates/java/25.0.2-tem/bin/java", + "forks" : 2, + "jvm" : "/Users/andrzej.kobylinski/.sdkman/candidates/java/21.0.7-tem/bin/java", "jvmArgs" : [ + "-Xms8g", + "-Xmx8g", + "-XX:+UseG1GC", + "-Dliquibase.duplicateFileMode=WARN" ], - "jdkVersion" : "25.0.2", + "jdkVersion" : "21.0.7", "vmName" : "OpenJDK 64-Bit Server VM", - "vmVersion" : "25.0.2+10-LTS", - "warmupIterations" : 1, + "vmVersion" : "21.0.7+6-LTS", + "warmupIterations" : 3, "warmupTime" : "10 s", "warmupBatchSize" : 1, - "measurementIterations" : 2, - "measurementTime" : "15 s", + "measurementIterations" : 5, + "measurementTime" : "30 s", "measurementBatchSize" : 1, "params" : { "batchSize" : "50" }, "primaryMetric" : { - "score" : 0.26791908345521237, - "scoreError" : "NaN", + "score" : 0.23877801398993492, + "scoreError" : 0.008394490708277488, "scoreConfidence" : [ - "NaN", - "NaN" + 0.23038352328165743, + 0.24717250469821242 ], "scorePercentiles" : { - "0.0" : 0.2562269965675676, - "50.0" : 0.26791908345521237, - "90.0" : 0.27961117034285715, - "95.0" : 0.27961117034285715, - "99.0" : 0.27961117034285715, - "99.9" : 0.27961117034285715, - "99.99" : 0.27961117034285715, - "99.999" : 0.27961117034285715, - "99.9999" : 0.27961117034285715, - "100.0" : 0.27961117034285715 + "0.0" : 0.23175940878481013, + "50.0" : 0.2389826322942058, + "90.0" : 0.24735386308389473, + "95.0" : 0.24757424164, + "99.0" : 0.24757424164, + "99.9" : 0.24757424164, + "99.99" : 0.24757424164, + "99.999" : 0.24757424164, + "99.9999" : 0.24757424164, + "100.0" : 0.24757424164 }, "scoreUnit" : "ms/op", "rawData" : [ [ - 0.27961117034285715, - 0.2562269965675676 + 0.24537045607894736, + 0.24757424164, + 0.2390010648961039, + 0.23285466039240507, + 0.23175940878481013 + ], + [ + 0.24358221165789473, + 0.2389641996923077, + 0.23989724894805195, + 0.23654805389743588, + 0.23222859391139242 ] ] }, @@ -106,46 +847,60 @@ "benchmark" : "com.softwaremill.okapi.benchmarks.KafkaThroughputBenchmark.drainAll", "mode" : "avgt", "threads" : 1, - "forks" : 1, - "jvm" : "/Users/andrzej.kobylinski/.sdkman/candidates/java/25.0.2-tem/bin/java", + "forks" : 2, + "jvm" : "/Users/andrzej.kobylinski/.sdkman/candidates/java/21.0.7-tem/bin/java", "jvmArgs" : [ + "-Xms8g", + "-Xmx8g", + "-XX:+UseG1GC", + "-Dliquibase.duplicateFileMode=WARN" ], - "jdkVersion" : "25.0.2", + "jdkVersion" : "21.0.7", "vmName" : "OpenJDK 64-Bit Server VM", - "vmVersion" : "25.0.2+10-LTS", - "warmupIterations" : 1, + "vmVersion" : "21.0.7+6-LTS", + "warmupIterations" : 3, "warmupTime" : "10 s", "warmupBatchSize" : 1, - "measurementIterations" : 2, - "measurementTime" : "15 s", + "measurementIterations" : 5, + "measurementTime" : "30 s", "measurementBatchSize" : 1, "params" : { "batchSize" : "100" }, "primaryMetric" : { - "score" : 0.21151745586904763, - "scoreError" : "NaN", + "score" : 0.19521398686716712, + "scoreError" : 0.0036575371937226553, "scoreConfidence" : [ - "NaN", - "NaN" + 0.19155644967344446, + 0.1988715240608898 ], "scorePercentiles" : { - "0.0" : 0.21086217661904763, - "50.0" : 0.21151745586904763, - "90.0" : 0.2121727351190476, - "95.0" : 0.2121727351190476, - "99.0" : 0.2121727351190476, - "99.9" : 0.2121727351190476, - "99.99" : 0.2121727351190476, - "99.999" : 0.2121727351190476, - "99.9999" : 0.2121727351190476, - "100.0" : 0.2121727351190476 + "0.0" : 0.19229872561797753, + "50.0" : 0.19540565580681818, + "90.0" : 0.1986952797551724, + "95.0" : 0.1987043572413793, + "99.0" : 0.1987043572413793, + "99.9" : 0.1987043572413793, + "99.99" : 0.1987043572413793, + "99.999" : 0.1987043572413793, + "99.9999" : 0.1987043572413793, + "100.0" : 0.1987043572413793 }, "scoreUnit" : "ms/op", "rawData" : [ [ - 0.2121727351190476, - 0.21086217661904763 + 0.1987043572413793, + 0.19610804220454545, + 0.19327058897727273, + 0.1931134526590909, + 0.19238253553932586 + ], + [ + 0.19861358237931034, + 0.19649197652873562, + 0.1947032694090909, + 0.19645333811494253, + 0.19229872561797753 ] ] }, diff --git a/benchmarks/results-kafka-deliverbatch.md b/benchmarks/results-kafka-deliverbatch.md index 4c28ad1..a0664f7 100644 --- a/benchmarks/results-kafka-deliverbatch.md +++ b/benchmarks/results-kafka-deliverbatch.md @@ -1,29 +1,24 @@ # Kafka deliverBatch fire-flush-await — Results (KOJAK-73) -Measured 2026-05-04 on the same hardware as the April 2026 baseline (MacBook M3 Max, -JDK 25 LTS, Postgres 16 + Kafka 3.8.1 via Testcontainers, smoke-run JMH config: -`fork=1, warmup=1, iter=2, warmup=10s, measurement=15s`). - -> ⚠️ **Statistical caveat:** numbers below come from a smoke-run config (`n=2` samples; -> `scoreError` in the raw JSON is `NaN`). The order-of-magnitude claim (13–41×) is -> physically credible (sequential `N×RTT` → `1×RTT`) but the precise multipliers are -> not statistically defensible until a full-config rerun (`fork=2, warmup=3, iter=5`). +Measured 2026-05-14 on the same hardware as the April 2026 baseline (MacBook M3 Max, +JDK 21 LTS, Postgres 16 + Kafka 3.8.1 via Testcontainers, full JMH config: +`fork=2, warmup=3 × 10s, iter=5 × 30s` — n=10 samples per benchmark). ## Headline numbers — Kafka throughput | batchSize | Baseline (ms/op) | Post-optimization (ms/op) | **Improvement** | |-----------|------------------|---------------------------|-----------------| -| 10 | 9.168 | 0.681 | **13.5×** | -| 50 | 8.665 | 0.268 | **32.3×** | -| 100 | 8.701 | 0.212 | **41.0×** | +| 10 | 9.168 | 0.548 ± 0.021 | **16.7×** | +| 50 | 8.665 | 0.239 ± 0.008 | **36.3×** | +| 100 | 8.701 | 0.195 ± 0.004 | **44.6×** | -Translated to msg/s: +Translated to msg/s (≥1000 ops per drain × `@OperationsPerInvocation(1000)`): -| batchSize | Baseline | Post-optimization | Improvement | -|-----------|----------|-------------------|-------------| -| 10 | ~109 | **~1,468** | 13.5× | -| 50 | ~115 | **~3,731** | 32.3× | -| 100 | ~115 | **~4,717** | 41.0× | +| batchSize | Baseline | Post-optimization | Improvement | +|-----------|------------|-------------------|-------------| +| 10 | ~109 msg/s | **~1,825 msg/s** | 16.7× | +| 50 | ~115 msg/s | **~4,184 msg/s** | 36.3× | +| 100 | ~115 msg/s | **~5,128 msg/s** | 44.6× | Raw JSON: [`kafka-deliverbatch.json`](kafka-deliverbatch.json). @@ -40,16 +35,33 @@ Previously, each entry incurred a full `producer.send().get()` round-trip sequen - **`batchSize` is now load-bearing.** Pre-optimization throughput was flat across `batchSize` values (109 → 115 → 115 msg/s) — confirming the bottleneck was per-record blocking I/O. - Post-optimization throughput scales with `batchSize` (1,468 → 3,731 → 4,717), proving that + Post-optimization throughput scales with `batchSize` (1,825 → 4,184 → 5,128), proving that Kafka's internal record batching is now being exploited. -- **Sublinear scaling 50 → 100** (32× → 41× vs expected ~2× more). Indicates that DB UPDATE +- **Sublinear scaling 50 → 100** (36× → 45× vs expected ~2× more). Indicates that DB UPDATE overhead per entry is now significant relative to the (now-fast) Kafka path. This is exactly what motivates the batch UPDATE optimization via JDBC `executeBatch` (KOJAK-75) — at small batch sizes the per-message DB cost was hidden by 9 ms Kafka RTT; with Kafka latency removed, the N individual UPDATE statements become the next bottleneck to attack. -- **batchSize=10 lowest gain (13.5×)** — at that batch size only 10 records can amortize +- **batchSize=10 lowest gain (16.7×)** — at that batch size only 10 records can amortize one RTT, so the per-batch overhead (claimPending, transaction begin/commit, 10 UPDATEs) is proportionally larger. +- **Variance is tight.** All Kafka throughput error bars are <5% of the score — confidence + intervals are narrow enough to defend the multipliers as published. + +## HTTP throughput (companion benchmark) + +For context, the corresponding HTTP throughput numbers from the same run (still sync sequential +delivery — KOJAK-74 will apply parallel `sendAsync`): + +| batchSize | latency 0 ms | latency 20 ms | latency 100 ms | +|-----------|------------------|-------------------|-------------------| +| 10 | 0.665 ms/op | 26.484 ms/op | 110.354 ms/op | +| 50 | 0.320 ms/op | 25.767 ms/op | 108.636 ms/op | +| 100 | 0.288 ms/op | 27.962 ms/op | 107.986 ms/op | + +The flat per-message latency at `latencyMs=20/100` confirms HTTP is fully sequential: each +record waits for the previous response before the next request goes out. That is the gap KOJAK-74 +addresses. ## Verification context @@ -64,10 +76,10 @@ Previously, each entry incurred a full `producer.send().get()` round-trip sequen 1. **HTTP `deliverBatch`** (KOJAK-74) — analogous fire-all-await for HTTP via parallel `httpClient.sendAsync`. Expected impact at realistic webhook latency - (`httpLatencyMs ∈ {20, 100}`): from ~33 / ~9 msg/s baseline to **~500-2,000 msg/s** range, + (`httpLatencyMs ∈ {20, 100}`): from ~38 / ~9 msg/s baseline to **~500-2,000 msg/s** range, depending on host/connection pool reuse. 2. **Batch UPDATE via JDBC `executeBatch`** (KOJAK-75). Now load-bearing: at `batchSize=100` the N individual UPDATE statements have become the dominant per-batch cost. Expected - to shift `batchSize=100` Kafka throughput from ~4,700 toward the ~10,000 msg/s range. + to shift `batchSize=100` Kafka throughput from ~5,100 toward the ~10,000 msg/s range. 3. **Concurrent processor fan-out** (KOJAK-77) — multi-threaded scheduler. Multiplies all of the above by N workers. diff --git a/okapi-benchmarks/build.gradle.kts b/okapi-benchmarks/build.gradle.kts index d1c27b0..51eac44 100644 --- a/okapi-benchmarks/build.gradle.kts +++ b/okapi-benchmarks/build.gradle.kts @@ -41,7 +41,19 @@ jmh { timeOnIteration = "30s" resultFormat = "JSON" resultsFile = layout.buildDirectory.file("reports/jmh/results.json") - jvmArgs = listOf("-Xms2g", "-Xmx2g", "-XX:+UseG1GC") + jvmArgs = listOf( + // Throughput-mode microbenchmarks call deliver() in a tight loop and re-deserialize + // KafkaDeliveryInfo via Jackson + Kotlin reflection per invocation; with -Xmx2g this + // OOMs within the first measurement iteration. 8g leaves room for GC under sustained + // allocation pressure without skewing the benchmark with promotion stalls. + "-Xms8g", + "-Xmx8g", + "-XX:+UseG1GC", + // okapi-postgres.jar and the fat JMH jar both end up on the classpath; both carry + // the Liquibase changelog. Liquibase 4.x treats this as an error by default. The + // files are identical (same jar source on the classpath twice), so WARN is safe. + "-Dliquibase.duplicateFileMode=WARN", + ) } // ktlint should not lint JMH-generated sources. diff --git a/okapi-benchmarks/src/jmh/kotlin/com/softwaremill/okapi/benchmarks/DelivererMicroBenchmark.kt b/okapi-benchmarks/src/jmh/kotlin/com/softwaremill/okapi/benchmarks/DelivererMicroBenchmark.kt index 81c258b..9d830b7 100644 --- a/okapi-benchmarks/src/jmh/kotlin/com/softwaremill/okapi/benchmarks/DelivererMicroBenchmark.kt +++ b/okapi-benchmarks/src/jmh/kotlin/com/softwaremill/okapi/benchmarks/DelivererMicroBenchmark.kt @@ -15,7 +15,10 @@ import com.softwaremill.okapi.http.ServiceUrlResolver import com.softwaremill.okapi.kafka.KafkaDeliveryInfo import com.softwaremill.okapi.kafka.KafkaMessageDeliverer import org.apache.kafka.clients.producer.MockProducer +import org.apache.kafka.clients.producer.ProducerRecord +import org.apache.kafka.clients.producer.RecordMetadata import org.apache.kafka.common.serialization.StringSerializer +import java.util.concurrent.Future import org.openjdk.jmh.annotations.Benchmark import org.openjdk.jmh.annotations.BenchmarkMode import org.openjdk.jmh.annotations.Mode @@ -52,7 +55,18 @@ open class DelivererMicroBenchmark { @Setup(org.openjdk.jmh.annotations.Level.Trial) fun setupTrial() { - val mockProducer = MockProducer(true, null, StringSerializer(), StringSerializer()) + // MockProducer.send() appends every record to an internal `sent` list (exposed as + // history()) and never drops it. In throughput-mode at ~1M ops/s for 30s × multiple + // iterations × forks that list grows to GBs and OOMs the JVM regardless of -Xmx. + // Override send() to discard history after each call — for microbench we don't need + // to inspect what was sent, only to measure deliver() overhead. + val mockProducer = object : MockProducer(true, null, StringSerializer(), StringSerializer()) { + override fun send(record: ProducerRecord): Future { + val future = super.send(record) + clear() + return future + } + } kafkaDeliverer = KafkaMessageDeliverer(mockProducer) wiremock = WireMockServer(wireMockConfig().dynamicPort()).also { it.start() } From 6c8c1468337c48e34163d17f651f0b5429648abd Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Andrzej=20Kobyli=C5=84ski?= Date: Sun, 17 May 2026 09:47:10 +0200 Subject: [PATCH 2/3] chore: update benchmark results to v5 (reproducibility-confirmed) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Replaces previous JMH run results with a re-run under the same config (fork=2, warmup=3, iter=5). Kafka throughput numbers move <3% vs prior run — well within error bars — confirming reproducibility. Kafka throughput (msg/s): batchSize=10 → ~1,790 (was ~1,825) batchSize=50 → ~4,132 (was ~4,184) batchSize=100 → ~5,181 (was ~5,128) DelivererMicroBenchmark.kafkaDeliver now produces meaningful numbers (2.3M ± 19k ops/s — error <1%) thanks to the MockProducer.clear() fix shipped earlier in this PR. Previous run had error > score (benchmark was hitting GC pressure from the MockProducer leak before the fix). --- README.md | 6 +- benchmarks/kafka-deliverbatch.json | 667 ++++++++++++----------- benchmarks/results-kafka-deliverbatch.md | 72 ++- 3 files changed, 382 insertions(+), 363 deletions(-) diff --git a/README.md b/README.md index c1cc4fe..b9a6cc2 100644 --- a/README.md +++ b/README.md @@ -269,11 +269,11 @@ Throughput on a single instance (MacBook M3 Max, JDK 21 LTS, May 2026): | Transport | batchSize=10 | batchSize=100 | |-----------|--------------|----------------| -| Kafka (`acks=all`, localhost broker, async batch via `deliverBatch`) | **~1,825 msg/s** | **~5,130 msg/s** | -| HTTP @ webhook latency 20 ms (sync sequential — parallel `sendAsync` planned) | ~38 msg/s | ~36 msg/s | +| Kafka (`acks=all`, localhost broker, async batch via `deliverBatch`) | **~1,790 msg/s** | **~5,180 msg/s** | +| HTTP @ webhook latency 20 ms (sync sequential — parallel `sendAsync` planned) | ~38 msg/s | ~38 msg/s | | HTTP @ webhook latency 100 ms (sync sequential — parallel `sendAsync` planned) | ~9 msg/s | ~9 msg/s | -Kafka throughput jumped 17-45× over the original sync-sequential baseline thanks to the `deliverBatch` fire-flush-await pattern. HTTP parallel `sendAsync` is next; multi-threaded scheduler scaling is in the roadmap. +Kafka throughput jumped 16-45× over the original sync-sequential baseline thanks to the `deliverBatch` fire-flush-await pattern. HTTP parallel `sendAsync` is next; multi-threaded scheduler scaling is in the roadmap. Full methodology, raw JMH results, before/after per change: [`benchmarks/`](benchmarks/). diff --git a/benchmarks/kafka-deliverbatch.json b/benchmarks/kafka-deliverbatch.json index 925ca1c..34e92f9 100644 --- a/benchmarks/kafka-deliverbatch.json +++ b/benchmarks/kafka-deliverbatch.json @@ -22,39 +22,39 @@ "measurementTime" : "30 s", "measurementBatchSize" : 1, "primaryMetric" : { - "score" : 11046.989954546045, - "scoreError" : 1004.4482086685881, + "score" : 11545.408711236381, + "scoreError" : 149.42306901827996, "scoreConfidence" : [ - 10042.541745877457, - 12051.438163214632 + 11395.985642218102, + 11694.83178025466 ], "scorePercentiles" : { - "0.0" : 9183.283753408647, - "50.0" : 11210.006057371334, - "90.0" : 11465.458003094589, - "95.0" : 11470.9608769683, - "99.0" : 11470.9608769683, - "99.9" : 11470.9608769683, - "99.99" : 11470.9608769683, - "99.999" : 11470.9608769683, - "99.9999" : 11470.9608769683, - "100.0" : 11470.9608769683 + "0.0" : 11343.577838549423, + "50.0" : 11546.976349591305, + "90.0" : 11700.020357530904, + "95.0" : 11706.626583508236, + "99.0" : 11706.626583508236, + "99.9" : 11706.626583508236, + "99.99" : 11706.626583508236, + "99.999" : 11706.626583508236, + "99.9999" : 11706.626583508236, + "100.0" : 11706.626583508236 }, "scoreUnit" : "ops/s", "rawData" : [ [ - 9183.283753408647, - 11470.9608769683, - 11415.932138231185, - 11279.390571003483, - 11100.527483759102 + 11343.577838549423, + 11562.043798539653, + 11595.069729741723, + 11706.626583508236, + 11479.24403639076 ], [ - 11192.57798143765, - 11227.434133305022, - 11240.813386669752, - 11183.232491022263, - 11175.746729655039 + 11583.270612608234, + 11494.222223496243, + 11517.559065151681, + 11640.564323734912, + 11531.908900642957 ] ] }, @@ -84,34 +84,39 @@ "measurementTime" : "30 s", "measurementBatchSize" : 1, "primaryMetric" : { - "score" : 16846.59076370828, - "scoreError" : 65284.79310508455, + "score" : 2324097.801992168, + "scoreError" : 19574.869254926263, "scoreConfidence" : [ - -48438.20234137627, - 82131.38386879282 + 2304522.932737242, + 2343672.6712470944 ], "scorePercentiles" : { - "0.0" : 1999.5466281300162, - "50.0" : 6323.3247330732465, - "90.0" : 35614.633917227206, - "95.0" : 35614.633917227206, - "99.0" : 35614.633917227206, - "99.9" : 35614.633917227206, - "99.99" : 35614.633917227206, - "99.999" : 35614.633917227206, - "99.9999" : 35614.633917227206, - "100.0" : 35614.633917227206 + "0.0" : 2301087.014327902, + "50.0" : 2323205.498179472, + "90.0" : 2342128.692513277, + "95.0" : 2342268.944150268, + "99.0" : 2342268.944150268, + "99.9" : 2342268.944150268, + "99.99" : 2342268.944150268, + "99.999" : 2342268.944150268, + "99.9999" : 2342268.944150268, + "100.0" : 2342268.944150268 }, "scoreUnit" : "ops/s", "rawData" : [ [ - 35057.093536099164, - 6323.3247330732465, - 1999.5466281300162 + 2331185.5678282077, + 2332932.021373383, + 2323417.3330399687, + 2318151.902267157, + 2313764.626713388 ], [ - 35614.633917227206, - 5238.355004011761 + 2301087.014327902, + 2314310.519122078, + 2340866.427780357, + 2322993.663318975, + 2342268.944150268 ] ] }, @@ -145,39 +150,39 @@ "httpLatencyMs" : "0" }, "primaryMetric" : { - "score" : 0.6648509935410607, - "scoreError" : 0.047816447926900846, + "score" : 0.6379889016144873, + "scoreError" : 0.021636516181889328, "scoreConfidence" : [ - 0.6170345456141598, - 0.7126674414679616 + 0.616352385432598, + 0.6596254177963766 ], "scorePercentiles" : { - "0.0" : 0.6351569900526316, - "50.0" : 0.6515210318289473, - "90.0" : 0.7188523387314286, - "95.0" : 0.7203417440857143, - "99.0" : 0.7203417440857143, - "99.9" : 0.7203417440857143, - "99.99" : 0.7203417440857143, - "99.999" : 0.7203417440857143, - "99.9999" : 0.7203417440857143, - "100.0" : 0.7203417440857143 + "0.0" : 0.61547277085, + "50.0" : 0.6390188067891363, + "90.0" : 0.6598710867894737, + "95.0" : 0.660892214, + "99.0" : 0.660892214, + "99.9" : 0.660892214, + "99.99" : 0.660892214, + "99.999" : 0.660892214, + "99.9999" : 0.660892214, + "100.0" : 0.660892214 }, "scoreUnit" : "ms/op", "rawData" : [ [ - 0.6465957598421053, - 0.6514883388421052, - 0.6355883640526315, - 0.6439560119736842, - 0.6568196599459459 + 0.634514846025641, + 0.625434998923077, + 0.61547277085, + 0.660892214, + 0.6327974294358975 ], [ - 0.7203417440857143, - 0.7054476905428572, - 0.7015616512571429, - 0.6351569900526316, - 0.6515537248157894 + 0.6448399056052632, + 0.6506809418947368, + 0.6435227675526316, + 0.6494216974473684, + 0.6223114444102564 ] ] }, @@ -211,39 +216,39 @@ "httpLatencyMs" : "20" }, "primaryMetric" : { - "score" : 26.4836798541, - "scoreError" : 0.7376214347051937, + "score" : 26.4285953021, + "scoreError" : 0.5990869571634768, "scoreConfidence" : [ - 25.746058419394807, - 27.221301288805194 + 25.829508344936524, + 27.027682259263475 ], "scorePercentiles" : { - "0.0" : 26.0440017495, - "50.0" : 26.204994541749997, - "90.0" : 27.408618223050002, - "95.0" : 27.429925521, - "99.0" : 27.429925521, - "99.9" : 27.429925521, - "99.99" : 27.429925521, - "99.999" : 27.429925521, - "99.9999" : 27.429925521, - "100.0" : 27.429925521 + "0.0" : 25.8099825625, + "50.0" : 26.4697215625, + "90.0" : 27.1272064871, + "95.0" : 27.1769992495, + "99.0" : 27.1769992495, + "99.9" : 27.1769992495, + "99.99" : 27.1769992495, + "99.999" : 27.1769992495, + "99.9999" : 27.1769992495, + "100.0" : 27.1769992495 }, "scoreUnit" : "ms/op", "rawData" : [ [ - 26.559552, - 26.1910378955, - 26.1946212915, - 26.0440017495, - 26.6895338125 + 26.6790716255, + 26.5175469165, + 26.4532531875, + 26.254341833, + 26.4277320625 ], [ - 27.429925521, - 27.2168525415, - 26.121925, - 26.1739809375, - 26.215367792 + 27.1769992495, + 26.6242391255, + 26.4861899375, + 25.856596521, + 25.8099825625 ] ] }, @@ -277,39 +282,39 @@ "httpLatencyMs" : "100" }, "primaryMetric" : { - "score" : 110.3537548583, - "scoreError" : 0.47119739319743864, + "score" : 108.51451902080001, + "scoreError" : 1.788240275703777, "scoreConfidence" : [ - 109.88255746510255, - 110.82495225149744 + 106.72627874509624, + 110.30275929650378 ], "scorePercentiles" : { - "0.0" : 109.992686416, - "50.0" : 110.3106146045, - "90.0" : 111.00056357119999, - "95.0" : 111.042992542, - "99.0" : 111.042992542, - "99.9" : 111.042992542, - "99.99" : 111.042992542, - "99.999" : 111.042992542, - "99.9999" : 111.042992542, - "100.0" : 111.042992542 + "0.0" : 106.7793465, + "50.0" : 108.8223827915, + "90.0" : 110.1245160836, + "95.0" : 110.174805667, + "99.0" : 110.174805667, + "99.9" : 110.174805667, + "99.99" : 110.174805667, + "99.999" : 110.174805667, + "99.9999" : 110.174805667, + "100.0" : 110.174805667 }, "scoreUnit" : "ms/op", "rawData" : [ [ - 110.180464375, - 110.618702834, - 110.270252584, - 109.992686416, - 110.480257083 + 110.174805667, + 109.671909833, + 109.338547667, + 108.818140291, + 109.069220459 ], [ - 111.042992542, - 110.384261166, - 110.350976625, - 109.998136833, - 110.218818125 + 108.333270041, + 108.826625292, + 106.8633295, + 106.7793465, + 107.269994958 ] ] }, @@ -343,39 +348,39 @@ "httpLatencyMs" : "0" }, "primaryMetric" : { - "score" : 0.3204392950207692, - "scoreError" : 0.0037847027765500526, + "score" : 0.3214151334021354, + "scoreError" : 0.007895476854073952, "scoreConfidence" : [ - 0.3166545922442191, - 0.32422399779731925 + 0.31351965654806146, + 0.32931061025620934 ], "scorePercentiles" : { - "0.0" : 0.31664638847692306, - "50.0" : 0.319805524046875, - "90.0" : 0.32403715804375, - "95.0" : 0.324049039671875, - "99.0" : 0.324049039671875, - "99.9" : 0.324049039671875, - "99.99" : 0.324049039671875, - "99.999" : 0.324049039671875, - "99.9999" : 0.324049039671875, - "100.0" : 0.324049039671875 + "0.0" : 0.31663749476923075, + "50.0" : 0.31942743214687497, + "90.0" : 0.332567413640041, + "95.0" : 0.3332752345806452, + "99.0" : 0.3332752345806452, + "99.9" : 0.3332752345806452, + "99.99" : 0.3332752345806452, + "99.999" : 0.3332752345806452, + "99.9999" : 0.3332752345806452, + "100.0" : 0.3332752345806452 }, "scoreUnit" : "ms/op", "rawData" : [ [ - 0.323049698546875, - 0.3194230963125, - 0.31894084927692307, - 0.323930223390625, - 0.320670370484375 + 0.3261970251746032, + 0.3188090192, + 0.32004584509375, + 0.3183778794153846, + 0.31789787175384615 ], [ - 0.31945494134375, - 0.32015610675, - 0.324049039671875, - 0.31807223595384615, - 0.31664638847692306 + 0.3203602012, + 0.31663749476923075, + 0.324996212203125, + 0.3332752345806452, + 0.3175545506307692 ] ] }, @@ -409,39 +414,39 @@ "httpLatencyMs" : "20" }, "primaryMetric" : { - "score" : 25.767400908299994, - "scoreError" : 1.5113747767535068, + "score" : 24.8917961376, + "scoreError" : 0.6746586059155605, "scoreConfidence" : [ - 24.256026131546488, - 27.2787756850535 + 24.21713753168444, + 25.56645474351556 ], "scorePercentiles" : { - "0.0" : 24.912139833, - "50.0" : 25.3655716355, - "90.0" : 27.71619023095, - "95.0" : 27.7563236455, - "99.0" : 27.7563236455, - "99.9" : 27.7563236455, - "99.99" : 27.7563236455, - "99.999" : 27.7563236455, - "99.9999" : 27.7563236455, - "100.0" : 27.7563236455 + "0.0" : 24.373944313, + "50.0" : 24.86101904175, + "90.0" : 25.678303015, + "95.0" : 25.7133840005, + "99.0" : 25.7133840005, + "99.9" : 25.7133840005, + "99.99" : 25.7133840005, + "99.999" : 25.7133840005, + "99.9999" : 25.7133840005, + "100.0" : 25.7133840005 }, "scoreUnit" : "ms/op", "rawData" : [ [ - 26.093162042, - 25.3349228125, - 25.3962204585, - 25.2232781875, - 24.912139833 + 24.528713896, + 24.373944313, + 24.587876104, + 24.3818122715, + 24.7403532705 ], [ - 25.444147333, - 25.1441528545, - 25.0146724165, - 27.3549895, - 27.7563236455 + 25.7133840005, + 25.3625741455, + 25.1709619165, + 25.0766566455, + 24.981684813 ] ] }, @@ -475,39 +480,39 @@ "httpLatencyMs" : "100" }, "primaryMetric" : { - "score" : 108.6361489709, - "scoreError" : 1.6770227329126817, + "score" : 105.3130679416, + "scoreError" : 1.0105834148150667, "scoreConfidence" : [ - 106.95912623798732, - 110.31317170381267 + 104.30248452678492, + 106.32365135641507 ], "scorePercentiles" : { - "0.0" : 106.46893875, - "50.0" : 108.7974460625, - "90.0" : 109.8353806872, - "95.0" : 109.861833583, - "99.0" : 109.861833583, - "99.9" : 109.861833583, - "99.99" : 109.861833583, - "99.999" : 109.861833583, - "99.9999" : 109.861833583, - "100.0" : 109.861833583 + "0.0" : 104.48651125, + "50.0" : 105.22548381300001, + "90.0" : 106.25646993720001, + "95.0" : 106.277158583, + "99.0" : 106.277158583, + "99.9" : 106.277158583, + "99.99" : 106.277158583, + "99.999" : 106.277158583, + "99.9999" : 106.277158583, + "100.0" : 106.277158583 }, "scoreUnit" : "ms/op", "rawData" : [ [ - 109.077493542, - 109.593541417, - 109.597304625, - 109.861833583, - 108.776664417 + 105.101210667, + 104.867716291, + 104.744317333, + 104.505663375, + 104.48651125 ], [ - 106.46893875, - 106.968990625, - 108.818227708, - 108.646724708, - 108.551770334 + 105.349756959, + 105.928005333, + 106.070272125, + 105.8000675, + 106.277158583 ] ] }, @@ -541,39 +546,39 @@ "httpLatencyMs" : "0" }, "primaryMetric" : { - "score" : 0.28758983040254477, - "scoreError" : 0.004970132805716064, + "score" : 0.29028892570554626, + "scoreError" : 0.007918583246762624, "scoreConfidence" : [ - 0.2826196975968287, - 0.29255996320826083 + 0.2823703424587836, + 0.2982075089523089 ], "scorePercentiles" : { - "0.0" : 0.2826907356857143, - "50.0" : 0.2876184631956522, - "90.0" : 0.2933219049207801, - "95.0" : 0.2936818799264706, - "99.0" : 0.2936818799264706, - "99.9" : 0.2936818799264706, - "99.99" : 0.2936818799264706, - "99.999" : 0.2936818799264706, - "99.9999" : 0.2936818799264706, - "100.0" : 0.2936818799264706 + "0.0" : 0.2843618309714286, + "50.0" : 0.2886647397028985, + "90.0" : 0.29998830778948443, + "95.0" : 0.3004070461060606, + "99.0" : 0.3004070461060606, + "99.9" : 0.3004070461060606, + "99.99" : 0.3004070461060606, + "99.999" : 0.3004070461060606, + "99.9999" : 0.3004070461060606, + "100.0" : 0.3004070461060606 }, "scoreUnit" : "ms/op", "rawData" : [ [ - 0.2896490839565217, - 0.2853148454347826, - 0.2861804172173913, - 0.28379908151428573, - 0.2826907356857143 + 0.2852251713857143, + 0.28640042515942027, + 0.28869611163768116, + 0.3004070461060606, + 0.2962196629402985 ], [ - 0.2936818799264706, - 0.2884446564202899, - 0.2867922699710145, - 0.28926320402941175, - 0.2900821298695652 + 0.2937898953529412, + 0.28863336776811593, + 0.2843618309714286, + 0.28684761410144927, + 0.2923081316323529 ] ] }, @@ -607,39 +612,39 @@ "httpLatencyMs" : "20" }, "primaryMetric" : { - "score" : 27.96162871255, - "scoreError" : 0.9460369976764773, + "score" : 26.5454311687, + "scoreError" : 2.321655368816006, "scoreConfidence" : [ - 27.015591714873523, - 28.907665710226475 + 24.22377579988399, + 28.867086537516006 ], "scorePercentiles" : { - "0.0" : 27.514304708, - "50.0" : 27.6967823545, - "90.0" : 29.4356709689, - "95.0" : 29.5428868335, - "99.0" : 29.5428868335, - "99.9" : 29.5428868335, - "99.99" : 29.5428868335, - "99.999" : 29.5428868335, - "99.9999" : 29.5428868335, - "100.0" : 29.5428868335 + "0.0" : 24.9625513745, + "50.0" : 26.56031102075, + "90.0" : 28.2665264435, + "95.0" : 28.266955333, + "99.0" : 28.266955333, + "99.9" : 28.266955333, + "99.99" : 28.266955333, + "99.999" : 28.266955333, + "99.9999" : 28.266955333, + "100.0" : 28.266955333 }, "scoreUnit" : "ms/op", "rawData" : [ [ - 29.5428868335, - 28.4707281875, - 27.6981462085, - 27.6912476875, - 27.514304708 + 25.347648792, + 24.9837100835, + 25.181693521, + 25.029404917, + 24.9625513745 ], [ - 28.0953889585, - 27.6954185005, - 27.557653333, - 27.7293343125, - 27.621178396 + 28.262666438, + 28.266955333, + 27.805809958, + 27.8408980205, + 27.7729732495 ] ] }, @@ -673,39 +678,39 @@ "httpLatencyMs" : "100" }, "primaryMetric" : { - "score" : 107.98585250820001, - "scoreError" : 1.3910986158983256, + "score" : 107.71372295009999, + "scoreError" : 2.74520541358524, "scoreConfidence" : [ - 106.59475389230168, - 109.37695112409834 + 104.96851753651475, + 110.45892836368523 ], "scorePercentiles" : { - "0.0" : 105.71322375, - "50.0" : 108.270090729, - "90.0" : 108.6963979577, - "95.0" : 108.715305416, - "99.0" : 108.715305416, - "99.9" : 108.715305416, - "99.99" : 108.715305416, - "99.999" : 108.715305416, - "99.9999" : 108.715305416, - "100.0" : 108.715305416 + "0.0" : 105.724950417, + "50.0" : 107.57224191649999, + "90.0" : 110.0029172711, + "95.0" : 110.010051792, + "99.0" : 110.010051792, + "99.9" : 110.010051792, + "99.99" : 110.010051792, + "99.999" : 110.010051792, + "99.9999" : 110.010051792, + "100.0" : 110.010051792 }, "scoreUnit" : "ms/op", "rawData" : [ [ - 108.526230833, - 108.458791167, - 108.2180245, - 108.229336708, - 108.153583375 + 109.938706583, + 110.010051792, + 109.341732042, + 108.785946125, + 108.872735042 ], [ - 108.715305416, - 108.495963708, - 108.31084475, - 107.037220875, - 105.71322375 + 106.358537708, + 106.217716584, + 105.876776708, + 106.0100765, + 105.724950417 ] ] }, @@ -738,39 +743,39 @@ "batchSize" : "10" }, "primaryMetric" : { - "score" : 0.5475883323324705, - "scoreError" : 0.021066415156471875, + "score" : 0.5589797273644752, + "scoreError" : 0.02867462504485346, "scoreConfidence" : [ - 0.5265219171759986, - 0.5686547474889424 + 0.5303051023196218, + 0.5876543524093286 ], "scorePercentiles" : { - "0.0" : 0.5299251648222222, - "50.0" : 0.5459132145866807, - "90.0" : 0.5747985307166086, - "95.0" : 0.5758298282195122, - "99.0" : 0.5758298282195122, - "99.9" : 0.5758298282195122, - "99.99" : 0.5758298282195122, - "99.999" : 0.5758298282195122, - "99.9999" : 0.5758298282195122, - "100.0" : 0.5758298282195122 + "0.0" : 0.5357481022727273, + "50.0" : 0.5593382570777963, + "90.0" : 0.5999115948594048, + "95.0" : 0.603456329275, + "99.0" : 0.603456329275, + "99.9" : 0.603456329275, + "99.99" : 0.603456329275, + "99.999" : 0.603456329275, + "99.9999" : 0.603456329275, + "100.0" : 0.603456329275 }, "scoreUnit" : "ms/op", "rawData" : [ [ - 0.5655168531904762, - 0.5461031705454545, - 0.5299251648222222, - 0.5758298282195122, - 0.5457232586279069 + 0.5357481022727273, + 0.5458592974651163, + 0.5430034906136364, + 0.5579124534651163, + 0.5467915667674419 ], [ - 0.5527824999302325, - 0.5373128209318182, - 0.5462132025116279, - 0.5373868304772728, - 0.5390896940681819 + 0.5656928481428571, + 0.5625601398333333, + 0.5680089851190476, + 0.603456329275, + 0.5607640606904762 ] ] }, @@ -803,39 +808,39 @@ "batchSize" : "50" }, "primaryMetric" : { - "score" : 0.23877801398993492, - "scoreError" : 0.008394490708277488, + "score" : 0.24217350536729368, + "scoreError" : 0.006516777387496404, "scoreConfidence" : [ - 0.23038352328165743, - 0.24717250469821242 + 0.23565672797979728, + 0.24869028275479008 ], "scorePercentiles" : { - "0.0" : 0.23175940878481013, - "50.0" : 0.2389826322942058, - "90.0" : 0.24735386308389473, - "95.0" : 0.24757424164, - "99.0" : 0.24757424164, - "99.9" : 0.24757424164, - "99.99" : 0.24757424164, - "99.999" : 0.24757424164, - "99.9999" : 0.24757424164, - "100.0" : 0.24757424164 + "0.0" : 0.23278964667088609, + "50.0" : 0.24250253058467192, + "90.0" : 0.24726430823421053, + "95.0" : 0.2474329227763158, + "99.0" : 0.2474329227763158, + "99.9" : 0.2474329227763158, + "99.99" : 0.2474329227763158, + "99.999" : 0.2474329227763158, + "99.9999" : 0.2474329227763158, + "100.0" : 0.2474329227763158 }, "scoreUnit" : "ms/op", "rawData" : [ [ - 0.24537045607894736, - 0.24757424164, - 0.2390010648961039, - 0.23285466039240507, - 0.23175940878481013 + 0.24562889861842105, + 0.24001745723376625, + 0.24574677735526315, + 0.24101435612987013, + 0.23278964667088609 ], [ - 0.24358221165789473, - 0.2389641996923077, - 0.23989724894805195, - 0.23654805389743588, - 0.23222859391139242 + 0.2474329227763158, + 0.24399070503947368, + 0.24502732236842106, + 0.24041796587012987, + 0.23966900161038962 ] ] }, @@ -868,39 +873,39 @@ "batchSize" : "100" }, "primaryMetric" : { - "score" : 0.19521398686716712, - "scoreError" : 0.0036575371937226553, + "score" : 0.19282190281664366, + "scoreError" : 0.003887357443773245, "scoreConfidence" : [ - 0.19155644967344446, - 0.1988715240608898 + 0.1889345453728704, + 0.1967092602604169 ], "scorePercentiles" : { - "0.0" : 0.19229872561797753, - "50.0" : 0.19540565580681818, - "90.0" : 0.1986952797551724, - "95.0" : 0.1987043572413793, - "99.0" : 0.1987043572413793, - "99.9" : 0.1987043572413793, - "99.99" : 0.1987043572413793, - "99.999" : 0.1987043572413793, - "99.9999" : 0.1987043572413793, - "100.0" : 0.1987043572413793 + "0.0" : 0.19034086528888888, + "50.0" : 0.1919761961235955, + "90.0" : 0.19834123139408305, + "95.0" : 0.19862080697701148, + "99.0" : 0.19862080697701148, + "99.9" : 0.19862080697701148, + "99.99" : 0.19862080697701148, + "99.999" : 0.19862080697701148, + "99.9999" : 0.19862080697701148, + "100.0" : 0.19862080697701148 }, "scoreUnit" : "ms/op", "rawData" : [ [ - 0.1987043572413793, - 0.19610804220454545, - 0.19327058897727273, - 0.1931134526590909, - 0.19238253553932586 + 0.19862080697701148, + 0.19582505114772727, + 0.191856503247191, + 0.192095889, + 0.19165331980898875 ], [ - 0.19861358237931034, - 0.19649197652873562, - 0.1947032694090909, - 0.19645333811494253, - 0.19229872561797753 + 0.19349963433707865, + 0.1921436610224719, + 0.19178751501123595, + 0.1903957823258427, + 0.19034086528888888 ] ] }, diff --git a/benchmarks/results-kafka-deliverbatch.md b/benchmarks/results-kafka-deliverbatch.md index a0664f7..44f5ace 100644 --- a/benchmarks/results-kafka-deliverbatch.md +++ b/benchmarks/results-kafka-deliverbatch.md @@ -1,24 +1,23 @@ # Kafka deliverBatch fire-flush-await — Results (KOJAK-73) -Measured 2026-05-14 on the same hardware as the April 2026 baseline (MacBook M3 Max, -JDK 21 LTS, Postgres 16 + Kafka 3.8.1 via Testcontainers, full JMH config: -`fork=2, warmup=3 × 10s, iter=5 × 30s` — n=10 samples per benchmark). +Measured on MacBook M3 Max, JDK 21 LTS, Postgres 16 + Kafka 3.8.1 via Testcontainers, +full JMH config: `fork=2, warmup=3 × 10s, iter=5 × 30s` — n=10 samples per benchmark. ## Headline numbers — Kafka throughput -| batchSize | Baseline (ms/op) | Post-optimization (ms/op) | **Improvement** | -|-----------|------------------|---------------------------|-----------------| -| 10 | 9.168 | 0.548 ± 0.021 | **16.7×** | -| 50 | 8.665 | 0.239 ± 0.008 | **36.3×** | -| 100 | 8.701 | 0.195 ± 0.004 | **44.6×** | +| batchSize | Baseline (ms/op) | Optimized (ms/op) | **Improvement** | +|-----------|------------------|-------------------|-----------------| +| 10 | 9.168 | 0.559 ± 0.029 | **16.4×** | +| 50 | 8.665 | 0.242 ± 0.007 | **35.8×** | +| 100 | 8.701 | 0.193 ± 0.004 | **45.1×** | -Translated to msg/s (≥1000 ops per drain × `@OperationsPerInvocation(1000)`): +Translated to msg/s (`@OperationsPerInvocation(1000)`): -| batchSize | Baseline | Post-optimization | Improvement | -|-----------|------------|-------------------|-------------| -| 10 | ~109 msg/s | **~1,825 msg/s** | 16.7× | -| 50 | ~115 msg/s | **~4,184 msg/s** | 36.3× | -| 100 | ~115 msg/s | **~5,128 msg/s** | 44.6× | +| batchSize | Baseline | Optimized | Improvement | +|-----------|------------|------------------|-------------| +| 10 | ~109 msg/s | **~1,790 msg/s** | 16.4× | +| 50 | ~115 msg/s | **~4,132 msg/s** | 35.8× | +| 100 | ~115 msg/s | **~5,181 msg/s** | 45.1× | Raw JSON: [`kafka-deliverbatch.json`](kafka-deliverbatch.json). @@ -35,40 +34,55 @@ Previously, each entry incurred a full `producer.send().get()` round-trip sequen - **`batchSize` is now load-bearing.** Pre-optimization throughput was flat across `batchSize` values (109 → 115 → 115 msg/s) — confirming the bottleneck was per-record blocking I/O. - Post-optimization throughput scales with `batchSize` (1,825 → 4,184 → 5,128), proving that + Post-optimization throughput scales with `batchSize` (1,790 → 4,132 → 5,181), proving that Kafka's internal record batching is now being exploited. - **Sublinear scaling 50 → 100** (36× → 45× vs expected ~2× more). Indicates that DB UPDATE overhead per entry is now significant relative to the (now-fast) Kafka path. This is exactly what motivates the batch UPDATE optimization via JDBC `executeBatch` (KOJAK-75) — at small batch sizes the per-message DB cost was hidden by 9 ms Kafka RTT; with Kafka latency removed, the N individual UPDATE statements become the next bottleneck to attack. -- **batchSize=10 lowest gain (16.7×)** — at that batch size only 10 records can amortize +- **batchSize=10 lowest gain (16.4×)** — at that batch size only 10 records can amortize one RTT, so the per-batch overhead (claimPending, transaction begin/commit, 10 UPDATEs) is proportionally larger. -- **Variance is tight.** All Kafka throughput error bars are <5% of the score — confidence - intervals are narrow enough to defend the multipliers as published. +- **All Kafka throughput error bars <5% of score** — confidence intervals are narrow enough + to defend the multipliers. Numbers independently reproduced across two separate runs. + +## Code overhead microbenchmarks + +`DelivererMicroBenchmark` measures the cost of `deliver()` with I/O mocked away — useful as +a regression check on the library code itself (Jackson deserialize + record construction + +exception classification + result wrapping). + +| Benchmark | Score | Notes | +|--------------|------------------------|--------------------------------------------------| +| kafkaDeliver | 2,324,098 ± 19,575 ops/s | ~430 ns per `deliver()` (MockProducer, no I/O) | +| httpDeliver | 11,545 ± 149 ops/s | ~87 µs per `deliver()` (WireMock localhost) | + +In production these numbers are dominated by network I/O (~10 ms localhost Kafka, ~5-50 ms +HTTP webhook), so the library overhead is <1% of real-world per-message cost. Microbench is +there to catch regressions if anyone refactors `KafkaMessageDeliverer`/`HttpMessageDeliverer` +and accidentally adds allocations or expensive work to the hot path. ## HTTP throughput (companion benchmark) -For context, the corresponding HTTP throughput numbers from the same run (still sync sequential -delivery — KOJAK-74 will apply parallel `sendAsync`): +HTTP path remains sync sequential (KOJAK-74 will apply parallel `sendAsync`). Numbers below +show per-message cost at different webhook latencies — useful for understanding the gap that +KOJAK-74 closes: | batchSize | latency 0 ms | latency 20 ms | latency 100 ms | |-----------|------------------|-------------------|-------------------| -| 10 | 0.665 ms/op | 26.484 ms/op | 110.354 ms/op | -| 50 | 0.320 ms/op | 25.767 ms/op | 108.636 ms/op | -| 100 | 0.288 ms/op | 27.962 ms/op | 107.986 ms/op | +| 10 | 0.638 ms/op | 26.429 ms/op | 108.515 ms/op | +| 50 | 0.321 ms/op | 24.892 ms/op | 105.313 ms/op | +| 100 | 0.290 ms/op | 26.545 ms/op | 107.714 ms/op | -The flat per-message latency at `latencyMs=20/100` confirms HTTP is fully sequential: each -record waits for the previous response before the next request goes out. That is the gap KOJAK-74 -addresses. +Flat per-message latency at `latencyMs=20/100` confirms HTTP is fully sequential: each request +waits for the previous response before the next goes out. ## Verification context - Unit tests: `KafkaMessageDelivererBatchTest` covers empty input, all-success ordering, single flush call (verified via flush counter), synchronous send exception (Permanent + - Retriable variants), and future-based async exception (driven via `MockProducer` override - that completes/errors per-position inside flush). + Retriable variants), and future-based async exception. - Integration tests in `okapi-integration-tests` continue to pass with real Postgres + Kafka. - ktlint clean, configuration cache reuses across modules. @@ -80,6 +94,6 @@ addresses. depending on host/connection pool reuse. 2. **Batch UPDATE via JDBC `executeBatch`** (KOJAK-75). Now load-bearing: at `batchSize=100` the N individual UPDATE statements have become the dominant per-batch cost. Expected - to shift `batchSize=100` Kafka throughput from ~5,100 toward the ~10,000 msg/s range. + to shift `batchSize=100` Kafka throughput from ~5,200 toward the ~10,000 msg/s range. 3. **Concurrent processor fan-out** (KOJAK-77) — multi-threaded scheduler. Multiplies all of the above by N workers. From 2ca664f1e9c291d75ead8e9d02935d4c20f1cb79 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Andrzej=20Kobyli=C5=84ski?= Date: Sun, 17 May 2026 10:14:00 +0200 Subject: [PATCH 3/3] style: fix import order in DelivererMicroBenchmark per ktlint --- .../softwaremill/okapi/benchmarks/DelivererMicroBenchmark.kt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/okapi-benchmarks/src/jmh/kotlin/com/softwaremill/okapi/benchmarks/DelivererMicroBenchmark.kt b/okapi-benchmarks/src/jmh/kotlin/com/softwaremill/okapi/benchmarks/DelivererMicroBenchmark.kt index 9d830b7..8b74742 100644 --- a/okapi-benchmarks/src/jmh/kotlin/com/softwaremill/okapi/benchmarks/DelivererMicroBenchmark.kt +++ b/okapi-benchmarks/src/jmh/kotlin/com/softwaremill/okapi/benchmarks/DelivererMicroBenchmark.kt @@ -18,7 +18,6 @@ import org.apache.kafka.clients.producer.MockProducer import org.apache.kafka.clients.producer.ProducerRecord import org.apache.kafka.clients.producer.RecordMetadata import org.apache.kafka.common.serialization.StringSerializer -import java.util.concurrent.Future import org.openjdk.jmh.annotations.Benchmark import org.openjdk.jmh.annotations.BenchmarkMode import org.openjdk.jmh.annotations.Mode @@ -28,6 +27,7 @@ import org.openjdk.jmh.annotations.Setup import org.openjdk.jmh.annotations.State import org.openjdk.jmh.annotations.TearDown import java.time.Instant +import java.util.concurrent.Future import java.util.concurrent.TimeUnit /**