Efficient Trace Payload Protocol (v1 protocol)#10801
Efficient Trace Payload Protocol (v1 protocol)#10801gh-worker-dd-mergequeue-cf854d[bot] merged 89 commits intomasterfrom
Conversation
BenchmarksStartupParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 59 metrics, 12 unstable metrics. Startup time reports for petclinicgantt
title petclinic - global startup overhead: candidate=1.61.0-SNAPSHOT~c1d9b0a4df, baseline=1.62.0-SNAPSHOT~8569434beb
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.065 s) : 0, 1065131
Total [baseline] (11.085 s) : 0, 11085139
Agent [candidate] (1.06 s) : 0, 1060003
Total [candidate] (11.036 s) : 0, 11035566
section appsec
Agent [baseline] (1.27 s) : 0, 1269552
Total [baseline] (11.009 s) : 0, 11008648
Agent [candidate] (1.26 s) : 0, 1259920
Total [candidate] (11.029 s) : 0, 11028532
section iast
Agent [baseline] (1.231 s) : 0, 1230698
Total [baseline] (11.372 s) : 0, 11371663
Agent [candidate] (1.239 s) : 0, 1239369
Total [candidate] (11.337 s) : 0, 11337107
section profiling
Agent [baseline] (1.191 s) : 0, 1190752
Total [baseline] (11.087 s) : 0, 11087424
Agent [candidate] (1.183 s) : 0, 1183136
Total [candidate] (11.086 s) : 0, 11086317
gantt
title petclinic - break down per module: candidate=1.61.0-SNAPSHOT~c1d9b0a4df, baseline=1.62.0-SNAPSHOT~8569434beb
dateFormat X
axisFormat %s
section tracing
crashtracking [baseline] (1.238 ms) : 0, 1238
crashtracking [candidate] (1.216 ms) : 0, 1216
BytebuddyAgent [baseline] (637.209 ms) : 0, 637209
BytebuddyAgent [candidate] (634.387 ms) : 0, 634387
AgentMeter [baseline] (29.8 ms) : 0, 29800
AgentMeter [candidate] (29.488 ms) : 0, 29488
GlobalTracer [baseline] (251.431 ms) : 0, 251431
GlobalTracer [candidate] (248.841 ms) : 0, 248841
AppSec [baseline] (32.586 ms) : 0, 32586
AppSec [candidate] (32.302 ms) : 0, 32302
Debugger [baseline] (60.436 ms) : 0, 60436
Debugger [candidate] (60.605 ms) : 0, 60605
Remote Config [baseline] (611.95 µs) : 0, 612
Remote Config [candidate] (595.256 µs) : 0, 595
Telemetry [baseline] (8.091 ms) : 0, 8091
Telemetry [candidate] (8.847 ms) : 0, 8847
Flare Poller [baseline] (7.406 ms) : 0, 7406
Flare Poller [candidate] (7.515 ms) : 0, 7515
section appsec
crashtracking [baseline] (1.232 ms) : 0, 1232
crashtracking [candidate] (1.231 ms) : 0, 1231
BytebuddyAgent [baseline] (679.443 ms) : 0, 679443
BytebuddyAgent [candidate] (673.361 ms) : 0, 673361
AgentMeter [baseline] (12.207 ms) : 0, 12207
AgentMeter [candidate] (12.163 ms) : 0, 12163
GlobalTracer [baseline] (250.595 ms) : 0, 250595
GlobalTracer [candidate] (248.988 ms) : 0, 248988
IAST [baseline] (24.468 ms) : 0, 24468
IAST [candidate] (24.158 ms) : 0, 24158
AppSec [baseline] (186.051 ms) : 0, 186051
AppSec [candidate] (186.972 ms) : 0, 186972
Debugger [baseline] (66.927 ms) : 0, 66927
Debugger [candidate] (64.841 ms) : 0, 64841
Remote Config [baseline] (578.11 µs) : 0, 578
Remote Config [candidate] (568.578 µs) : 0, 569
Telemetry [baseline] (7.973 ms) : 0, 7973
Telemetry [candidate] (7.79 ms) : 0, 7790
Flare Poller [baseline] (3.467 ms) : 0, 3467
Flare Poller [candidate] (3.445 ms) : 0, 3445
section iast
crashtracking [baseline] (1.221 ms) : 0, 1221
crashtracking [candidate] (1.23 ms) : 0, 1230
BytebuddyAgent [baseline] (808.93 ms) : 0, 808930
BytebuddyAgent [candidate] (814.796 ms) : 0, 814796
AgentMeter [baseline] (11.462 ms) : 0, 11462
AgentMeter [candidate] (11.516 ms) : 0, 11516
GlobalTracer [baseline] (239.081 ms) : 0, 239081
GlobalTracer [candidate] (240.298 ms) : 0, 240298
IAST [baseline] (30.626 ms) : 0, 30626
IAST [candidate] (29.944 ms) : 0, 29944
AppSec [baseline] (28.265 ms) : 0, 28265
AppSec [candidate] (28.649 ms) : 0, 28649
Debugger [baseline] (63.43 ms) : 0, 63430
Debugger [candidate] (65.007 ms) : 0, 65007
Remote Config [baseline] (539.07 µs) : 0, 539
Remote Config [candidate] (541.654 µs) : 0, 542
Telemetry [baseline] (7.689 ms) : 0, 7689
Telemetry [candidate] (7.789 ms) : 0, 7789
Flare Poller [baseline] (3.363 ms) : 0, 3363
Flare Poller [candidate] (3.46 ms) : 0, 3460
section profiling
crashtracking [baseline] (1.197 ms) : 0, 1197
crashtracking [candidate] (1.18 ms) : 0, 1180
BytebuddyAgent [baseline] (695.205 ms) : 0, 695205
BytebuddyAgent [candidate] (691.17 ms) : 0, 691170
AgentMeter [baseline] (9.022 ms) : 0, 9022
AgentMeter [candidate] (8.984 ms) : 0, 8984
GlobalTracer [baseline] (208.975 ms) : 0, 208975
GlobalTracer [candidate] (207.248 ms) : 0, 207248
AppSec [baseline] (32.801 ms) : 0, 32801
AppSec [candidate] (32.572 ms) : 0, 32572
Debugger [baseline] (66.011 ms) : 0, 66011
Debugger [candidate] (65.276 ms) : 0, 65276
Remote Config [baseline] (592.756 µs) : 0, 593
Remote Config [candidate] (567.485 µs) : 0, 567
Telemetry [baseline] (7.848 ms) : 0, 7848
Telemetry [candidate] (7.801 ms) : 0, 7801
Flare Poller [baseline] (3.491 ms) : 0, 3491
Flare Poller [candidate] (3.461 ms) : 0, 3461
ProfilingAgent [baseline] (93.829 ms) : 0, 93829
ProfilingAgent [candidate] (93.399 ms) : 0, 93399
Profiling [baseline] (94.398 ms) : 0, 94398
Profiling [candidate] (93.961 ms) : 0, 93961
Startup time reports for insecure-bankgantt
title insecure-bank - global startup overhead: candidate=1.61.0-SNAPSHOT~c1d9b0a4df, baseline=1.62.0-SNAPSHOT~8569434beb
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.059 s) : 0, 1059412
Total [baseline] (8.883 s) : 0, 8882515
Agent [candidate] (1.056 s) : 0, 1056185
Total [candidate] (8.837 s) : 0, 8836871
section iast
Agent [baseline] (1.233 s) : 0, 1232916
Total [baseline] (9.534 s) : 0, 9533677
Agent [candidate] (1.237 s) : 0, 1237393
Total [candidate] (9.608 s) : 0, 9608234
gantt
title insecure-bank - break down per module: candidate=1.61.0-SNAPSHOT~c1d9b0a4df, baseline=1.62.0-SNAPSHOT~8569434beb
dateFormat X
axisFormat %s
section tracing
crashtracking [baseline] (1.228 ms) : 0, 1228
crashtracking [candidate] (1.23 ms) : 0, 1230
BytebuddyAgent [baseline] (633.829 ms) : 0, 633829
BytebuddyAgent [candidate] (632.845 ms) : 0, 632845
AgentMeter [baseline] (29.515 ms) : 0, 29515
AgentMeter [candidate] (29.224 ms) : 0, 29224
GlobalTracer [baseline] (250.365 ms) : 0, 250365
GlobalTracer [candidate] (248.328 ms) : 0, 248328
AppSec [baseline] (32.625 ms) : 0, 32625
AppSec [candidate] (32.307 ms) : 0, 32307
Debugger [baseline] (59.432 ms) : 0, 59432
Debugger [candidate] (59.405 ms) : 0, 59405
Remote Config [baseline] (600.127 µs) : 0, 600
Remote Config [candidate] (597.113 µs) : 0, 597
Telemetry [baseline] (8.125 ms) : 0, 8125
Telemetry [candidate] (7.984 ms) : 0, 7984
Flare Poller [baseline] (7.477 ms) : 0, 7477
Flare Poller [candidate] (8.085 ms) : 0, 8085
section iast
crashtracking [baseline] (1.236 ms) : 0, 1236
crashtracking [candidate] (1.237 ms) : 0, 1237
BytebuddyAgent [baseline] (810.213 ms) : 0, 810213
BytebuddyAgent [candidate] (814.266 ms) : 0, 814266
AgentMeter [baseline] (11.432 ms) : 0, 11432
AgentMeter [candidate] (11.479 ms) : 0, 11479
GlobalTracer [baseline] (239.199 ms) : 0, 239199
GlobalTracer [candidate] (239.731 ms) : 0, 239731
IAST [baseline] (30.207 ms) : 0, 30207
IAST [candidate] (30.065 ms) : 0, 30065
AppSec [baseline] (30.331 ms) : 0, 30331
AppSec [candidate] (26.662 ms) : 0, 26662
Debugger [baseline] (62.367 ms) : 0, 62367
Debugger [candidate] (65.17 ms) : 0, 65170
Remote Config [baseline] (532.495 µs) : 0, 532
Remote Config [candidate] (542.236 µs) : 0, 542
Telemetry [baseline] (7.758 ms) : 0, 7758
Telemetry [candidate] (7.847 ms) : 0, 7847
Flare Poller [baseline] (3.449 ms) : 0, 3449
Flare Poller [candidate] (3.417 ms) : 0, 3417
LoadParameters
See matching parameters
SummaryFound 1 performance improvements and 0 performance regressions! Performance is the same for 20 metrics, 15 unstable metrics.
Request duration reports for insecure-bankgantt
title insecure-bank - request duration [CI 0.99] : candidate=1.61.0-SNAPSHOT~c1d9b0a4df, baseline=1.62.0-SNAPSHOT~8569434beb
dateFormat X
axisFormat %s
section baseline
no_agent (1.261 ms) : 1249, 1274
. : milestone, 1261,
iast (3.296 ms) : 3246, 3346
. : milestone, 3296,
iast_FULL (6.056 ms) : 5993, 6118
. : milestone, 6056,
iast_GLOBAL (3.819 ms) : 3757, 3881
. : milestone, 3819,
profiling (2.348 ms) : 2325, 2372
. : milestone, 2348,
tracing (1.893 ms) : 1877, 1909
. : milestone, 1893,
section candidate
no_agent (1.248 ms) : 1236, 1261
. : milestone, 1248,
iast (3.362 ms) : 3313, 3411
. : milestone, 3362,
iast_FULL (6.092 ms) : 6030, 6154
. : milestone, 6092,
iast_GLOBAL (3.639 ms) : 3587, 3692
. : milestone, 3639,
profiling (2.179 ms) : 2157, 2201
. : milestone, 2179,
tracing (1.943 ms) : 1926, 1961
. : milestone, 1943,
Request duration reports for petclinicgantt
title petclinic - request duration [CI 0.99] : candidate=1.61.0-SNAPSHOT~c1d9b0a4df, baseline=1.62.0-SNAPSHOT~8569434beb
dateFormat X
axisFormat %s
section baseline
no_agent (18.435 ms) : 18249, 18620
. : milestone, 18435,
appsec (18.884 ms) : 18693, 19074
. : milestone, 18884,
code_origins (18.731 ms) : 18547, 18916
. : milestone, 18731,
iast (17.815 ms) : 17639, 17991
. : milestone, 17815,
profiling (18.308 ms) : 18124, 18493
. : milestone, 18308,
tracing (18.648 ms) : 18461, 18835
. : milestone, 18648,
section candidate
no_agent (19.178 ms) : 18984, 19373
. : milestone, 19178,
appsec (18.39 ms) : 18205, 18576
. : milestone, 18390,
code_origins (17.682 ms) : 17505, 17858
. : milestone, 17682,
iast (18.066 ms) : 17889, 18243
. : milestone, 18066,
profiling (18.376 ms) : 18189, 18563
. : milestone, 18376,
tracing (18.034 ms) : 17855, 18213
. : milestone, 18034,
DacapoParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 11 metrics, 1 unstable metrics. Execution time for biojavagantt
title biojava - execution time [CI 0.99] : candidate=1.61.0-SNAPSHOT~c1d9b0a4df, baseline=1.62.0-SNAPSHOT~8569434beb
dateFormat X
axisFormat %s
section baseline
no_agent (14.778 s) : 14778000, 14778000
. : milestone, 14778000,
appsec (14.769 s) : 14769000, 14769000
. : milestone, 14769000,
iast (18.376 s) : 18376000, 18376000
. : milestone, 18376000,
iast_GLOBAL (18.006 s) : 18006000, 18006000
. : milestone, 18006000,
profiling (14.79 s) : 14790000, 14790000
. : milestone, 14790000,
tracing (14.673 s) : 14673000, 14673000
. : milestone, 14673000,
section candidate
no_agent (15.239 s) : 15239000, 15239000
. : milestone, 15239000,
appsec (14.823 s) : 14823000, 14823000
. : milestone, 14823000,
iast (18.979 s) : 18979000, 18979000
. : milestone, 18979000,
iast_GLOBAL (18.155 s) : 18155000, 18155000
. : milestone, 18155000,
profiling (15.276 s) : 15276000, 15276000
. : milestone, 15276000,
tracing (15.022 s) : 15022000, 15022000
. : milestone, 15022000,
Execution time for tomcatgantt
title tomcat - execution time [CI 0.99] : candidate=1.61.0-SNAPSHOT~c1d9b0a4df, baseline=1.62.0-SNAPSHOT~8569434beb
dateFormat X
axisFormat %s
section baseline
no_agent (1.485 ms) : 1473, 1496
. : milestone, 1485,
appsec (3.759 ms) : 3542, 3977
. : milestone, 3759,
iast (2.27 ms) : 2199, 2340
. : milestone, 2270,
iast_GLOBAL (2.318 ms) : 2248, 2389
. : milestone, 2318,
profiling (2.098 ms) : 2043, 2153
. : milestone, 2098,
tracing (2.081 ms) : 2027, 2135
. : milestone, 2081,
section candidate
no_agent (1.484 ms) : 1473, 1496
. : milestone, 1484,
appsec (3.836 ms) : 3612, 4060
. : milestone, 3836,
iast (2.275 ms) : 2204, 2345
. : milestone, 2275,
iast_GLOBAL (2.322 ms) : 2251, 2393
. : milestone, 2322,
profiling (2.098 ms) : 2042, 2153
. : milestone, 2098,
tracing (2.079 ms) : 2025, 2133
. : milestone, 2079,
|
dougqh
left a comment
There was a problem hiding this comment.
Looks good to me.
I added a couple comments based on asking Claude to do a perf review.
I do think optimizing might the ID to byte conversion would be good.
I suspect that is where some (most?) of the extra 2GiB of byte[] were coming from.
But I'm fine with doing that in a separate pull request
# Conflicts: # dd-trace-core/src/main/java/datadog/trace/core/DDSpanContext.java
mcculls
left a comment
There was a problem hiding this comment.
LGTM, thanks for improving the protocol selection - and the test coverage
|
/merge |
|
View all feedbacks in Devflow UI.
The expected merge time in
|
What Does This Do
v1.0(v1.0/tracesendpoint) in the Java tracer.TraceMapperV1, a new serializer that encodes traces in thev1 msgpackformat with a flat attribute model, string table deduplication, and typed attribute values.ProtocolVersionenum to cleanly representv0.4,v0.5, andv1.0protocols including their fallback endpoint chains.DDAgentFeaturesDiscoveryto probe endpoints based on the configured protocol version.Motivation
Protocol v1 is a more efficient wire format: it deduplicates repeated strings via a string table, uses typed attribute values (string/bool/float/int/bytes/array). This reduces payload size and provides a cleaner foundation for future span model evolution. The implementation falls back gracefully to
v0.4when the agent does not advertisev1support.Additional Notes
Protocol v1 spec
System test passed both on GitHub and GitLab.