fix(connector): real-time teardown, NODELAY, and multi-peer POSIX by Thecave3 · Pull Request #36 · wineslab/libe3

Thecave3 · 2026-07-04T14:34:33Z

Stacked on #35. Seven commits fixing defects surfaced by the paper's full baseline benchmark sweep:

Teardown / SIGTERM

ZMQ linger: every socket now sets ZMQ_LINGER=0. The default (-1) made zmq_ctx_destroy block forever flushing undelivered PUB messages, so a RAN ignored SIGTERM (observed: benchmark agents wedged in poll() for days).
POSIX shutdown: shutdown() now also unblocks the outbound listener's accept().

Real-time data path

TCP_NODELAY / SCTP_NODELAY on every accepted and connected data socket. Without it SCTP throughput capped at ~214 msg/s regardless of offered rate (one Nagle-style RTT stall per framed message). Verified after: SCTP tracks a 1 kHz offered rate 1:1.
One syscall per message: send_in_chunks coalesces the 4-byte length prefix and payload into a single writev().
Compensated pacing in simple_agent: sleep_until deadline scheduler, so offered rate equals 1/period (the old sleep_for capped 10 kHz configs at ~6.7 kHz, bounding every benchmark by the load generator).

Multi-peer POSIX

The RAN's POSIX data channels each accepted exactly one connection, independently, so two dApps could cross-wire (subscription channel won by A, indication channel by B) and everyone received nothing. Both listeners now accept for the connector's lifetime; receive() polls all subscriber sockets; send() broadcasts framed messages to all indication sockets, mirroring the ZMQ PUB/SUB semantics (RAN broadcasts, dApps filter by dApp id). No wire-format change; plain connect-and-read clients (Python dApp framework) keep working. Verified: two concurrent dApps over posix/tcp each receive the full indication stream. The multi-peer dispatch integration test now runs over both ZMQ and POSIX.

Benchmarks

bench_full_loop_latency emits fractional microseconds (sub-µs phases were truncated to 0).
bench_encoding_size: documented the info_bytes accounting and why APER packs below it (range-constrained integers, no per-field tags; verified against the E3AP ASN.1 module).

All 21 unit/integration tests pass (build-bench, ASN.1+JSON+protobuf, sanitizers off).

🤖 Generated with Claude Code

Set ZMQ_LINGER=0 on every socket (RAN REP/SUB/PUB, dApp REQ/SUB/PUB, and the reset path). The default linger of -1 makes zmq_ctx_destroy block until queued messages are flushed, so a RAN with undelivered indications on its PUB socket hung forever in dispose() and the process effectively ignored SIGTERM (the handler ran, but stop() never returned). Observed as benchmark agents wedged in poll() for days. Assisted-by: Claude:claude-fable-5

shutdown() interrupted the setup and inbound listeners but never the outbound one, so a RAN stopped before any dApp connected hung forever in the outbound thread's blocking accept() and stop() never returned. Assisted-by: Claude:claude-fable-5

Set TCP_NODELAY / SCTP_NODELAY on every accepted and connected TCP/SCTP socket (setup, inbound, outbound, and the dApp client paths). Without it each framed message (small length prefix + payload) stalls ~1 RTT waiting for the previous segment's ACK, measured as a hard ~214 msg/s throughput ceiling on SCTP regardless of offered rate. Set explicitly on accepted fds since Linux does not reliably inherit the option from the listener; no-op for UNIX-domain sockets. Assisted-by: Claude:claude-fable-5

send_in_chunks used a separate send() for the 4-byte length prefix and another for the payload; coalesce them into a single writev() (falling back to chunked send() for the remainder of oversized payloads). Halves the data-path syscall count and removes the tiny per-message segment that Nagle-style bundling was stalling on. Assisted-by: Claude:claude-fable-5

The RAN's POSIX data channels each accepted exactly one connection, once, and independently: with two dApps the subscription channel and the indication channel could be won by different peers (cross-wiring), so the subscribed dApp received nothing while extra clients sat in the listen backlog forever believing they were connected. Both listeners now accept for the connector's lifetime. receive() polls the listener plus every accepted subscriber socket (raw framing per peer preserved for the Python dApp framework); send() drains the indication listener and broadcasts each framed message to all peers, dropping dead ones, which mirrors the ZMQ PUB/SUB semantics the rest of libe3 assumes (RAN broadcasts, each dApp filters by its own dApp identifier; no peer attached = message dropped like PUB with no subscriber). Peers can attach and re-attach at any time; no wire format change and no identification handshake needed. The multi-peer dispatch integration test now runs the same no-crosstalk scenario over both ZMQ and POSIX. Assisted-by: Claude:claude-fable-5

Replace sleep_for(period) with a monotonic sleep_until deadline that advances by one period per cycle, so the offered rate equals 1/period independent of per-cycle work. The uncompensated sleep made each cycle cost period + work, capping the offered rate at ~6.7 kHz when 10 kHz was configured, which bounded every throughput benchmark by the load generator instead of the library. Falls back to re-anchoring (no catch-up bursts) when the loop is behind schedule. Assisted-by: Claude:claude-fable-5

bench_full_loop_latency computed phase deltas with integer ns->us division, truncating the six sub-microsecond phases (collect, encode, decode, process) to 0 in the report; emit fractional microseconds with two decimals instead (table structure unchanged for downstream parsers). bench_encoding_size: document precisely how info_bytes is computed (analytical fixed-width accounting of the sample PDU's field values, not an encoder measurement) and why ASN.1 APER can legitimately encode below that baseline (range-constrained integers in minimal bit width, no per-field tags; verified against the E3AP ASN.1 module, e.g. SubscriptionDelete 5 wire bytes vs 12 info bytes). Assisted-by: Claude:claude-fable-5

sm_data_thread_, its sm_data_handler_loop() polling stub, and the SM_POLL_INTERVAL constant were declared but never started or wired. A Service Model produces indications on the application's own thread and emits them through the outbound-emitter callback into the outbound queue; the RAN role runs setup/inbound/outbound plus the report worker. Remove the dead declaration, the empty poll loop, the always-false join, and the stale comments. No functional change; ctest 22/22. Assisted-by: Claude:claude-opus-4-8

github-actions · 2026-07-10T15:31:46Z

⏱️ Full-loop Latency Benchmark (commit `b46af4e`)

Full-loop latency benchmark (N=1012 after 50 warmup)

All values in microseconds (us). Link: zmq, transport: ipc, encoding: ASN.1 APER.

Phase	mean	p50	p99	max
1. Collect indication data	0.09	0.07	0.21	17.00
2. Create & encode indication	0.70	0.62	1.20	13.62
3. Deliver indication (RAN -> dApp)	88.69	80.92	165.39	176.55
4. Decode indication	0.55	0.52	0.89	17.73
5. Process data	0.03	0.03	0.04	0.13
6. Create & encode control	0.28	0.26	0.49	10.69
7. Deliver control (dApp -> RAN)	119.67	119.66	177.58	193.09
8. Decode & handle control	0.50	0.47	0.91	11.23
Total round-trip	210.52	196.74	326.11	334.71

Benchmarked on ubuntu-latest, Release build, ZMQ + IPC, ASN.1 APER.

github-actions · 2026-07-10T15:32:26Z

🔄 E2E dApp Integration Results (commit `b46af4e`)

posix/ipc

dApp exit: 0
Indications received: 7

posix/tcp

dApp exit: 0
Indications received: 7

zmq/ipc

dApp exit: 0
Indications received: 7

zmq/tcp

dApp exit: 0
Indications received: 7

example_simple_agent + example_simple_dapp on ubuntu-latest, Release build

github-actions · 2026-07-10T15:32:48Z

🔀 E2E Topologies — multi-dApp / multi-RAN (commit `b46af4e`)

zmq/ipc

✅ 1 RAN - 1 dApp: indications=5
- dapp peer=t11 ran=ran-solo sub=1 indications=5 seq=[0..4] dropped=0 (0%) age_ms(avg=1 max=1 @seq=0) hist[<=1:5 2-5:0 6-10:0 >10:0]
✅ 1 RAN - 2 dApps: dApp#1 ind=5 sub=1, dApp#2 ind=6 sub=2, RAN saw 2 dApps
- dapp1 peer=t12 ran=ran-shared sub=1 indications=5 seq=[0..4] dropped=0 (0%) age_ms(avg=0 max=0 @seq=0) hist[<=1:5 2-5:0 6-10:0 >10:0]
- dapp2 peer=t12 ran=ran-shared sub=2 indications=6 seq=[0..5] dropped=0 (0%) age_ms(avg=0 max=0 @seq=0) hist[<=1:6 2-5:0 6-10:0 >10:0]
✅ 2 RANs - 1 dApp: from ran-a ind=5, from ran-b ind=5
- dapp peer=t2a ran=ran-a sub=1 indications=5 seq=[0..4] dropped=0 (0%) age_ms(avg=0 max=0 @seq=0) hist[<=1:5 2-5:0 6-10:0 >10:0]
- dapp peer=t2b ran=ran-b sub=1 indications=5 seq=[0..4] dropped=0 (0%) age_ms(avg=0 max=0 @seq=0) hist[<=1:5 2-5:0 6-10:0 >10:0]

zmq/tcp

✅ 1 RAN - 1 dApp: indications=5
- dapp peer=default ran=ran-solo sub=1 indications=5 seq=[0..4] dropped=0 (0%) age_ms(avg=0.2 max=1 @seq=0) hist[<=1:5 2-5:0 6-10:0 >10:0]
✅ 1 RAN - 2 dApps: dApp#1 ind=5 sub=1, dApp#2 ind=6 sub=2, RAN saw 2 dApps
- dapp1 peer=default ran=ran-shared sub=1 indications=5 seq=[0..4] dropped=0 (0%) age_ms(avg=0.6 max=1 @seq=1) hist[<=1:5 2-5:0 6-10:0 >10:0]
- dapp2 peer=default ran=ran-shared sub=2 indications=6 seq=[0..5] dropped=0 (0%) age_ms(avg=0.5 max=1 @seq=1) hist[<=1:6 2-5:0 6-10:0 >10:0]
✅ 2 RANs - 1 dApp: from ran-a ind=5, from ran-b ind=5
- dapp peer=default ran=ran-a sub=1 indications=5 seq=[0..4] dropped=0 (0%) age_ms(avg=0.2 max=1 @seq=3) hist[<=1:5 2-5:0 6-10:0 >10:0]
- dapp peer=off100 ran=ran-b sub=1 indications=5 seq=[0..4] dropped=0 (0%) age_ms(avg=1 max=1 @seq=0) hist[<=1:5 2-5:0 6-10:0 >10:0]

example_simple_agent + example_simple_dapp on ubuntu-latest, Release. Indication age is report-only.