Skip to content

Add unified profiling feature gate config keys#11090

Open
jbachorik wants to merge 9 commits intomasterfrom
jb/prof-10633-unified-config
Open

Add unified profiling feature gate config keys#11090
jbachorik wants to merge 9 commits intomasterfrom
jb/prof-10633-unified-config

Conversation

@jbachorik
Copy link
Copy Markdown
Contributor

@jbachorik jbachorik commented Apr 13, 2026

What Does This Do

Adds unified profiling.<feature>.enabled configuration keys for all GA profiling features, following the pattern established by profiling.heap.enabled:

  • profiling.cpu.enabled (default: true) — gates CPU profiling for both JFR (jdk.ExecutionSample, jdk.NativeMethodSample, jdk.CPUTimeSample) and ddprof
  • profiling.walltime.enabled (default: true) — gates wall-clock profiling for ddprof (JFR timeline events are intentionally excluded — they serve queueing-time and blocking-event purposes beyond wall-clock profiling)
  • profiling.exception.enabled (default: true) — gates JFR exception profiling (datadog.ExceptionSample, datadog.ExceptionCount)
  • profiling.io.enabled (default: true) — gates JFR I/O profiling (jdk.FileRead/Write/Force, jdk.SocketRead/Write)
  • profiling.lock.enabled (default: true) — gates JFR lock profiling (jdk.JavaMonitorEnter, jdk.BiasedLock*)
  • profiling.thread.enabled (default: true) — gates JFR thread lifecycle profiling (jdk.ThreadStart/End)
  • profiling.direct.memory.enabled (default: false) — unified replacement for deprecated profiling.directallocation.enabled; old key still works as fallback

Also:

  • Adds PROFILING_ALLOCATION_ENABLED_DEFAULT = true constant (was dynamic-only before)
  • Updates DD_PROFILING_HEAP_ENABLED and DD_PROFILING_ALLOCATION_ENABLED metadata defaults to "true"
  • Deprecates PROFILING_HEAP_ENABLED_DEFAULT (dynamic default) and PROFILING_DIRECT_ALLOCATION_ENABLED constants
  • Adds === Feature Gates === section to profiler flare report

Motivation

PROF-10633 — Revisit and unify profiling settings.

Users had no consistent, feature-level way to disable individual profiling modes without reaching into backend-specific knobs (profiling.ddprof.cpu.enabled, profiling.ddprof.wall.enabled, etc.). The new unified keys provide a single toggle per feature that applies to both JFR and ddprof backends.

Configuration Design

Two-tier gate hierarchy

Each profiling feature is gated at two levels. The unified key is checked first; if it passes, backend-specific keys apply their own inner overrides.

flowchart TD
    PROF["dd.profiling.enabled = true"]

    PROF --> CPU
    PROF --> WALL
    PROF --> EXC
    PROF --> IO
    PROF --> LOCK
    PROF --> THR
    PROF --> DM

    subgraph CPU["profiling.cpu.enabled - default: true"]
        CPU_JFR["JFR: ExecutionSample, NativeMethodSample,<br>CPUTimeSample, CPUTimeSamplesLost"]
        CPU_DDPROF["ddprof: profiling.ddprof.cpu.enabled"]
    end

    subgraph WALL["profiling.walltime.enabled - default: true"]
        WALL_DDPROF["ddprof: profiling.ddprof.wall.enabled"]
        WALL_NOTE["Note: JFR timeline events are NOT gated here -<br>they serve queueing-time purposes independently"]
    end

    subgraph EXC["profiling.exception.enabled - default: true"]
        EXC_JFR["JFR: ExceptionSample, ExceptionCount"]
    end

    subgraph IO["profiling.io.enabled - default: true"]
        IO_JFR["JFR: FileRead, FileWrite, FileForce,<br>SocketRead, SocketWrite"]
    end

    subgraph LOCK["profiling.lock.enabled - default: true"]
        LOCK_JFR["JFR: JavaMonitorEnter, BiasedLock"]
    end

    subgraph THR["profiling.thread.enabled - default: true"]
        THR_JFR["JFR: ThreadStart, ThreadEnd"]
    end

    subgraph DM["profiling.direct.memory.enabled - default: false"]
        DM_INST["ByteBuffer.allocateDirect,<br>DirectByteBuffer ctor,<br>FileChannelImpl.map"]
    end
Loading

Config resolution for dd.profiling.direct.memory.enabled

This key replaces the deprecated dd.profiling.directallocation.enabled. The old key is honoured as an alias so existing deployments are unaffected.

Additional Notes

  • All new keys default to true so existing deployments are unaffected
  • profiling.direct.memory.enabled defaults to false to preserve backward compatibility with profiling.directallocation.enabled
  • The backend-specific keys (profiling.ddprof.cpu.enabled, etc.) remain as inner overrides for advanced use
  • Gates are evaluated once at recording start; profiling does not use Remote Config, so a disabled feature stays disabled for the recording lifetime

Contributor Checklist

Jira ticket: PROF-10633

Note: Once your PR is ready to merge, add it to the merge queue by commenting /merge. /merge -c cancels the queue request. /merge -f --reason "reason" skips all merge queue checks; please use this judiciously, as some checks do not run at the PR-level. For more information, see this doc.

@jbachorik jbachorik added tag: ai generated Largely based on code generated by an AI or LLM comp: profiling Profiling type: enhancement Enhancements and improvements labels Apr 13, 2026
@pr-commenter
Copy link
Copy Markdown

pr-commenter Bot commented Apr 13, 2026

Benchmarks

Startup

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master jb/prof-10633-unified-config
git_commit_date 1776861586 1776860941
git_commit_sha f89a0b26cc aaf76eb
release_version 1.62.0-SNAPSHOT~9f89a0b26cc 1.62.0-SNAPSHOT~aaf76eb4fe
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1776862877 1776862877
ci_job_id 1619335923 1619335923
ci_pipeline_id 109021655 109021655
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version Linux runner-zfyrx7zua-project-304-concurrent-1-cbc2m7tr 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Linux runner-zfyrx7zua-project-304-concurrent-1-cbc2m7tr 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
module Agent Agent
parent None None

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 59 metrics, 12 unstable metrics.

Startup time reports for insecure-bank
gantt
    title insecure-bank - global startup overhead: candidate=1.62.0-SNAPSHOT~aaf76eb4fe, baseline=1.62.0-SNAPSHOT~9f89a0b26cc

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.056 s) : 0, 1055782
Total [baseline] (8.856 s) : 0, 8855571
Agent [candidate] (1.061 s) : 0, 1060776
Total [candidate] (8.825 s) : 0, 8825082
section iast
Agent [baseline] (1.227 s) : 0, 1227321
Total [baseline] (9.583 s) : 0, 9583329
Agent [candidate] (1.219 s) : 0, 1219345
Total [candidate] (9.524 s) : 0, 9524232
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.056 s -
Agent iast 1.227 s 171.539 ms (16.2%)
Total tracing 8.856 s -
Total iast 9.583 s 727.758 ms (8.2%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.061 s -
Agent iast 1.219 s 158.569 ms (14.9%)
Total tracing 8.825 s -
Total iast 9.524 s 699.15 ms (7.9%)
gantt
    title insecure-bank - break down per module: candidate=1.62.0-SNAPSHOT~aaf76eb4fe, baseline=1.62.0-SNAPSHOT~9f89a0b26cc

    dateFormat X
    axisFormat %s
section tracing
crashtracking [baseline] (1.246 ms) : 0, 1246
crashtracking [candidate] (1.233 ms) : 0, 1233
BytebuddyAgent [baseline] (632.558 ms) : 0, 632558
BytebuddyAgent [candidate] (635.226 ms) : 0, 635226
AgentMeter [baseline] (29.395 ms) : 0, 29395
AgentMeter [candidate] (29.594 ms) : 0, 29594
GlobalTracer [baseline] (248.241 ms) : 0, 248241
GlobalTracer [candidate] (249.844 ms) : 0, 249844
AppSec [baseline] (32.019 ms) : 0, 32019
AppSec [candidate] (32.17 ms) : 0, 32170
Debugger [baseline] (59.157 ms) : 0, 59157
Debugger [candidate] (59.59 ms) : 0, 59590
Remote Config [baseline] (599.004 µs) : 0, 599
Remote Config [candidate] (603.892 µs) : 0, 604
Telemetry [baseline] (8.051 ms) : 0, 8051
Telemetry [candidate] (8.044 ms) : 0, 8044
Flare Poller [baseline] (8.289 ms) : 0, 8289
Flare Poller [candidate] (8.228 ms) : 0, 8228
section iast
crashtracking [baseline] (1.273 ms) : 0, 1273
crashtracking [candidate] (1.226 ms) : 0, 1226
BytebuddyAgent [baseline] (802.063 ms) : 0, 802063
BytebuddyAgent [candidate] (798.332 ms) : 0, 798332
AgentMeter [baseline] (11.482 ms) : 0, 11482
AgentMeter [candidate] (11.321 ms) : 0, 11321
GlobalTracer [baseline] (240.618 ms) : 0, 240618
GlobalTracer [candidate] (238.529 ms) : 0, 238529
AppSec [baseline] (28.817 ms) : 0, 28817
AppSec [candidate] (31.771 ms) : 0, 31771
Debugger [baseline] (63.22 ms) : 0, 63220
Debugger [candidate] (60.132 ms) : 0, 60132
Remote Config [baseline] (1.745 ms) : 0, 1745
Remote Config [candidate] (557.483 µs) : 0, 557
Telemetry [baseline] (11.619 ms) : 0, 11619
Telemetry [candidate] (11.838 ms) : 0, 11838
Flare Poller [baseline] (3.523 ms) : 0, 3523
Flare Poller [candidate] (3.513 ms) : 0, 3513
IAST [baseline] (25.977 ms) : 0, 25977
IAST [candidate] (25.776 ms) : 0, 25776
Loading
Startup time reports for petclinic
gantt
    title petclinic - global startup overhead: candidate=1.62.0-SNAPSHOT~aaf76eb4fe, baseline=1.62.0-SNAPSHOT~9f89a0b26cc

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.062 s) : 0, 1061884
Total [baseline] (11.064 s) : 0, 11064090
Agent [candidate] (1.056 s) : 0, 1055845
Total [candidate] (11.033 s) : 0, 11032671
section appsec
Agent [baseline] (1.247 s) : 0, 1247307
Total [baseline] (11.131 s) : 0, 11130684
Agent [candidate] (1.245 s) : 0, 1245227
Total [candidate] (11.093 s) : 0, 11092890
section iast
Agent [baseline] (1.225 s) : 0, 1224509
Total [baseline] (11.305 s) : 0, 11305013
Agent [candidate] (1.221 s) : 0, 1221051
Total [candidate] (11.311 s) : 0, 11311308
section profiling
Agent [baseline] (1.183 s) : 0, 1183103
Total [baseline] (11.045 s) : 0, 11044945
Agent [candidate] (1.183 s) : 0, 1182606
Total [candidate] (11.016 s) : 0, 11015927
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.062 s -
Agent appsec 1.247 s 185.424 ms (17.5%)
Agent iast 1.225 s 162.625 ms (15.3%)
Agent profiling 1.183 s 121.219 ms (11.4%)
Total tracing 11.064 s -
Total appsec 11.131 s 66.593 ms (0.6%)
Total iast 11.305 s 240.923 ms (2.2%)
Total profiling 11.045 s -19.146 ms (-0.2%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.056 s -
Agent appsec 1.245 s 189.382 ms (17.9%)
Agent iast 1.221 s 165.206 ms (15.6%)
Agent profiling 1.183 s 126.761 ms (12.0%)
Total tracing 11.033 s -
Total appsec 11.093 s 60.219 ms (0.5%)
Total iast 11.311 s 278.637 ms (2.5%)
Total profiling 11.016 s -16.744 ms (-0.2%)
gantt
    title petclinic - break down per module: candidate=1.62.0-SNAPSHOT~aaf76eb4fe, baseline=1.62.0-SNAPSHOT~9f89a0b26cc

    dateFormat X
    axisFormat %s
section tracing
crashtracking [baseline] (1.246 ms) : 0, 1246
crashtracking [candidate] (1.218 ms) : 0, 1218
BytebuddyAgent [baseline] (636.029 ms) : 0, 636029
BytebuddyAgent [candidate] (630.267 ms) : 0, 630267
AgentMeter [baseline] (29.4 ms) : 0, 29400
AgentMeter [candidate] (29.373 ms) : 0, 29373
GlobalTracer [baseline] (250.129 ms) : 0, 250129
GlobalTracer [candidate] (248.559 ms) : 0, 248559
AppSec [baseline] (32.177 ms) : 0, 32177
AppSec [candidate] (32.189 ms) : 0, 32189
Debugger [baseline] (60.288 ms) : 0, 60288
Debugger [candidate] (60.138 ms) : 0, 60138
Remote Config [baseline] (598.728 µs) : 0, 599
Remote Config [candidate] (595.428 µs) : 0, 595
Telemetry [baseline] (8.156 ms) : 0, 8156
Telemetry [candidate] (8.174 ms) : 0, 8174
Flare Poller [baseline] (7.597 ms) : 0, 7597
Flare Poller [candidate] (9.175 ms) : 0, 9175
section appsec
crashtracking [baseline] (1.214 ms) : 0, 1214
crashtracking [candidate] (1.221 ms) : 0, 1221
BytebuddyAgent [baseline] (660.227 ms) : 0, 660227
BytebuddyAgent [candidate] (660.72 ms) : 0, 660720
AgentMeter [baseline] (12.064 ms) : 0, 12064
AgentMeter [candidate] (12.05 ms) : 0, 12050
GlobalTracer [baseline] (249.109 ms) : 0, 249109
GlobalTracer [candidate] (248.675 ms) : 0, 248675
IAST [baseline] (24.686 ms) : 0, 24686
IAST [candidate] (24.449 ms) : 0, 24449
AppSec [baseline] (184.824 ms) : 0, 184824
AppSec [candidate] (183.921 ms) : 0, 183921
Debugger [baseline] (66.06 ms) : 0, 66060
Debugger [candidate] (65.158 ms) : 0, 65158
Remote Config [baseline] (613.819 µs) : 0, 614
Remote Config [candidate] (622.543 µs) : 0, 623
Telemetry [baseline] (8.613 ms) : 0, 8613
Telemetry [candidate] (8.536 ms) : 0, 8536
Flare Poller [baseline] (3.618 ms) : 0, 3618
Flare Poller [candidate] (3.491 ms) : 0, 3491
section iast
crashtracking [baseline] (1.227 ms) : 0, 1227
crashtracking [candidate] (1.217 ms) : 0, 1217
BytebuddyAgent [baseline] (800.773 ms) : 0, 800773
BytebuddyAgent [candidate] (798.95 ms) : 0, 798950
AgentMeter [baseline] (11.365 ms) : 0, 11365
AgentMeter [candidate] (11.363 ms) : 0, 11363
GlobalTracer [baseline] (240.174 ms) : 0, 240174
GlobalTracer [candidate] (238.688 ms) : 0, 238688
IAST [baseline] (26.735 ms) : 0, 26735
IAST [candidate] (25.672 ms) : 0, 25672
AppSec [baseline] (31.779 ms) : 0, 31779
AppSec [candidate] (32.445 ms) : 0, 32445
Debugger [baseline] (58.225 ms) : 0, 58225
Debugger [candidate] (59.106 ms) : 0, 59106
Remote Config [baseline] (525.056 µs) : 0, 525
Remote Config [candidate] (532.172 µs) : 0, 532
Telemetry [baseline] (14.029 ms) : 0, 14029
Telemetry [candidate] (13.358 ms) : 0, 13358
Flare Poller [baseline] (3.5 ms) : 0, 3500
Flare Poller [candidate] (3.493 ms) : 0, 3493
section profiling
crashtracking [baseline] (1.188 ms) : 0, 1188
crashtracking [candidate] (1.178 ms) : 0, 1178
BytebuddyAgent [baseline] (690.448 ms) : 0, 690448
BytebuddyAgent [candidate] (690.04 ms) : 0, 690040
AgentMeter [baseline] (9.161 ms) : 0, 9161
AgentMeter [candidate] (9.119 ms) : 0, 9119
GlobalTracer [baseline] (207.01 ms) : 0, 207010
GlobalTracer [candidate] (206.952 ms) : 0, 206952
AppSec [baseline] (32.572 ms) : 0, 32572
AppSec [candidate] (32.495 ms) : 0, 32495
Debugger [baseline] (65.565 ms) : 0, 65565
Debugger [candidate] (65.689 ms) : 0, 65689
Remote Config [baseline] (569.442 µs) : 0, 569
Remote Config [candidate] (571.037 µs) : 0, 571
Telemetry [baseline] (7.789 ms) : 0, 7789
Telemetry [candidate] (7.87 ms) : 0, 7870
Flare Poller [baseline] (3.541 ms) : 0, 3541
Flare Poller [candidate] (3.571 ms) : 0, 3571
ProfilingAgent [baseline] (94.036 ms) : 0, 94036
ProfilingAgent [candidate] (93.807 ms) : 0, 93807
Profiling [baseline] (94.599 ms) : 0, 94599
Profiling [candidate] (94.382 ms) : 0, 94382
Loading

Load

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master jb/prof-10633-unified-config
git_commit_date 1776861584 1776860941
git_commit_sha f89a0b26cc aaf76eb
release_version 1.62.0-SNAPSHOT~9f89a0b26cc 1.62.0-SNAPSHOT~aaf76eb4fe
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1776863356 1776863356
ci_job_id 1619335927 1619335927
ci_pipeline_id 109021655 109021655
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version Linux runner-zfyrx7zua-project-304-concurrent-1-g1q2ny7k 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Linux runner-zfyrx7zua-project-304-concurrent-1-g1q2ny7k 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 1 performance improvements and 2 performance regressions! Performance is the same for 17 metrics, 16 unstable metrics.

scenario Δ mean agg_http_req_duration_p50 Δ mean agg_http_req_duration_p95 Δ mean throughput candidate mean agg_http_req_duration_p50 candidate mean agg_http_req_duration_p95 candidate mean throughput baseline mean agg_http_req_duration_p50 baseline mean agg_http_req_duration_p95 baseline mean throughput
scenario:load:insecure-bank:no_agent:high_load better
[-115.585µs; -43.882µs] or [-10.594%; -4.022%]
unstable
[-1166.947µs; -295.686µs] or [-32.699%; -8.285%]
unstable
[+9.146op/s; +823.729op/s] or [+0.291%; +26.172%]
1.011ms 2.837ms 3563.812op/s 1.091ms 3.569ms 3147.375op/s
scenario:load:insecure-bank:iast_FULL:high_load worse
[+106.837µs; +303.301µs] or [+2.037%; +5.784%]
unsure
[+177.715µs; +773.254µs] or [+1.433%; +6.235%]
unstable
[-101.394op/s; +50.582op/s] or [-12.920%; +6.445%]
5.449ms 12.878ms 759.375op/s 5.244ms 12.403ms 784.781op/s
scenario:load:petclinic:code_origins:high_load worse
[+522.021µs; +1373.176µs] or [+2.960%; +7.786%]
unsure
[+185.859µs; +1504.105µs] or [+0.636%; +5.149%]
unstable
[-41.156op/s; +15.843op/s] or [-15.808%; +6.085%]
18.585ms 30.059ms 247.688op/s 17.638ms 29.214ms 260.344op/s
Request duration reports for insecure-bank
gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.62.0-SNAPSHOT~aaf76eb4fe, baseline=1.62.0-SNAPSHOT~9f89a0b26cc
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.413 ms) : 1396, 1430
.   : milestone, 1413,
iast (3.359 ms) : 3311, 3407
.   : milestone, 3359,
iast_FULL (5.889 ms) : 5830, 5948
.   : milestone, 5889,
iast_GLOBAL (3.697 ms) : 3637, 3758
.   : milestone, 3697,
profiling (2.095 ms) : 2077, 2114
.   : milestone, 2095,
tracing (1.917 ms) : 1901, 1933
.   : milestone, 1917,
section candidate
no_agent (1.239 ms) : 1227, 1251
.   : milestone, 1239,
iast (3.291 ms) : 3242, 3340
.   : milestone, 3291,
iast_FULL (6.093 ms) : 6031, 6154
.   : milestone, 6093,
iast_GLOBAL (3.588 ms) : 3528, 3647
.   : milestone, 3588,
profiling (2.047 ms) : 2029, 2065
.   : milestone, 2047,
tracing (1.887 ms) : 1872, 1903
.   : milestone, 1887,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.413 ms [1.396 ms, 1.43 ms] -
iast 3.359 ms [3.311 ms, 3.407 ms] 1.946 ms (137.7%)
iast_FULL 5.889 ms [5.83 ms, 5.948 ms] 4.476 ms (316.8%)
iast_GLOBAL 3.697 ms [3.637 ms, 3.758 ms] 2.284 ms (161.7%)
profiling 2.095 ms [2.077 ms, 2.114 ms] 682.172 µs (48.3%)
tracing 1.917 ms [1.901 ms, 1.933 ms] 504.151 µs (35.7%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.239 ms [1.227 ms, 1.251 ms] -
iast 3.291 ms [3.242 ms, 3.34 ms] 2.052 ms (165.6%)
iast_FULL 6.093 ms [6.031 ms, 6.154 ms] 4.853 ms (391.7%)
iast_GLOBAL 3.588 ms [3.528 ms, 3.647 ms] 2.348 ms (189.5%)
profiling 2.047 ms [2.029 ms, 2.065 ms] 807.992 µs (65.2%)
tracing 1.887 ms [1.872 ms, 1.903 ms] 648.162 µs (52.3%)
Request duration reports for petclinic
gantt
    title petclinic - request duration [CI 0.99] : candidate=1.62.0-SNAPSHOT~aaf76eb4fe, baseline=1.62.0-SNAPSHOT~9f89a0b26cc
    dateFormat X
    axisFormat %s
section baseline
no_agent (18.377 ms) : 18186, 18568
.   : milestone, 18377,
appsec (18.58 ms) : 18394, 18766
.   : milestone, 18580,
code_origins (17.924 ms) : 17742, 18107
.   : milestone, 17924,
iast (18.144 ms) : 17961, 18326
.   : milestone, 18144,
profiling (18.458 ms) : 18271, 18645
.   : milestone, 18458,
tracing (17.844 ms) : 17668, 18020
.   : milestone, 17844,
section candidate
no_agent (19.105 ms) : 18910, 19299
.   : milestone, 19105,
appsec (18.519 ms) : 18330, 18708
.   : milestone, 18519,
code_origins (18.844 ms) : 18656, 19032
.   : milestone, 18844,
iast (18.487 ms) : 18302, 18672
.   : milestone, 18487,
profiling (18.372 ms) : 18192, 18552
.   : milestone, 18372,
tracing (18.165 ms) : 17986, 18344
.   : milestone, 18165,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 18.377 ms [18.186 ms, 18.568 ms] -
appsec 18.58 ms [18.394 ms, 18.766 ms] 202.999 µs (1.1%)
code_origins 17.924 ms [17.742 ms, 18.107 ms] -452.498 µs (-2.5%)
iast 18.144 ms [17.961 ms, 18.326 ms] -233.357 µs (-1.3%)
profiling 18.458 ms [18.271 ms, 18.645 ms] 81.237 µs (0.4%)
tracing 17.844 ms [17.668 ms, 18.02 ms] -532.594 µs (-2.9%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 19.105 ms [18.91 ms, 19.299 ms] -
appsec 18.519 ms [18.33 ms, 18.708 ms] -585.759 µs (-3.1%)
code_origins 18.844 ms [18.656 ms, 19.032 ms] -260.225 µs (-1.4%)
iast 18.487 ms [18.302 ms, 18.672 ms] -617.523 µs (-3.2%)
profiling 18.372 ms [18.192 ms, 18.552 ms] -732.461 µs (-3.8%)
tracing 18.165 ms [17.986 ms, 18.344 ms] -939.889 µs (-4.9%)

Dacapo

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master jb/prof-10633-unified-config
git_commit_date 1776861586 1776860941
git_commit_sha f89a0b26cc aaf76eb
release_version 1.62.0-SNAPSHOT~9f89a0b26cc 1.62.0-SNAPSHOT~aaf76eb4fe
See matching parameters
Baseline Candidate
application biojava biojava
ci_job_date 1776863098 1776863098
ci_job_id 1619335932 1619335932
ci_pipeline_id 109021655 109021655
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version Linux runner-zfyrx7zua-project-304-concurrent-1-eljip7oj 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Linux runner-zfyrx7zua-project-304-concurrent-1-eljip7oj 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 10 metrics, 2 unstable metrics.

Execution time for biojava
gantt
    title biojava - execution time [CI 0.99] : candidate=1.62.0-SNAPSHOT~aaf76eb4fe, baseline=1.62.0-SNAPSHOT~9f89a0b26cc
    dateFormat X
    axisFormat %s
section baseline
no_agent (14.853 s) : 14853000, 14853000
.   : milestone, 14853000,
appsec (14.978 s) : 14978000, 14978000
.   : milestone, 14978000,
iast (18.34 s) : 18340000, 18340000
.   : milestone, 18340000,
iast_GLOBAL (17.764 s) : 17764000, 17764000
.   : milestone, 17764000,
profiling (14.96 s) : 14960000, 14960000
.   : milestone, 14960000,
tracing (14.738 s) : 14738000, 14738000
.   : milestone, 14738000,
section candidate
no_agent (15.154 s) : 15154000, 15154000
.   : milestone, 15154000,
appsec (14.73 s) : 14730000, 14730000
.   : milestone, 14730000,
iast (18.336 s) : 18336000, 18336000
.   : milestone, 18336000,
iast_GLOBAL (17.724 s) : 17724000, 17724000
.   : milestone, 17724000,
profiling (15.609 s) : 15609000, 15609000
.   : milestone, 15609000,
tracing (14.858 s) : 14858000, 14858000
.   : milestone, 14858000,
Loading
  • baseline results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 14.853 s [14.853 s, 14.853 s] -
appsec 14.978 s [14.978 s, 14.978 s] 125.0 ms (0.8%)
iast 18.34 s [18.34 s, 18.34 s] 3.487 s (23.5%)
iast_GLOBAL 17.764 s [17.764 s, 17.764 s] 2.911 s (19.6%)
profiling 14.96 s [14.96 s, 14.96 s] 107.0 ms (0.7%)
tracing 14.738 s [14.738 s, 14.738 s] -115.0 ms (-0.8%)
  • candidate results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 15.154 s [15.154 s, 15.154 s] -
appsec 14.73 s [14.73 s, 14.73 s] -424.0 ms (-2.8%)
iast 18.336 s [18.336 s, 18.336 s] 3.182 s (21.0%)
iast_GLOBAL 17.724 s [17.724 s, 17.724 s] 2.57 s (17.0%)
profiling 15.609 s [15.609 s, 15.609 s] 455.0 ms (3.0%)
tracing 14.858 s [14.858 s, 14.858 s] -296.0 ms (-2.0%)
Execution time for tomcat
gantt
    title tomcat - execution time [CI 0.99] : candidate=1.62.0-SNAPSHOT~aaf76eb4fe, baseline=1.62.0-SNAPSHOT~9f89a0b26cc
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.494 ms) : 1483, 1506
.   : milestone, 1494,
appsec (3.836 ms) : 3615, 4057
.   : milestone, 3836,
iast (2.288 ms) : 2218, 2358
.   : milestone, 2288,
iast_GLOBAL (2.329 ms) : 2259, 2398
.   : milestone, 2329,
profiling (2.529 ms) : 2307, 2751
.   : milestone, 2529,
tracing (2.093 ms) : 2039, 2147
.   : milestone, 2093,
section candidate
no_agent (1.492 ms) : 1480, 1503
.   : milestone, 1492,
appsec (3.779 ms) : 3562, 3997
.   : milestone, 3779,
iast (2.277 ms) : 2207, 2346
.   : milestone, 2277,
iast_GLOBAL (2.328 ms) : 2258, 2399
.   : milestone, 2328,
profiling (2.107 ms) : 2052, 2162
.   : milestone, 2107,
tracing (2.083 ms) : 2029, 2137
.   : milestone, 2083,
Loading
  • baseline results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 1.494 ms [1.483 ms, 1.506 ms] -
appsec 3.836 ms [3.615 ms, 4.057 ms] 2.342 ms (156.7%)
iast 2.288 ms [2.218 ms, 2.358 ms] 793.47 µs (53.1%)
iast_GLOBAL 2.329 ms [2.259 ms, 2.398 ms] 834.373 µs (55.8%)
profiling 2.529 ms [2.307 ms, 2.751 ms] 1.035 ms (69.2%)
tracing 2.093 ms [2.039 ms, 2.147 ms] 598.715 µs (40.1%)
  • candidate results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 1.492 ms [1.48 ms, 1.503 ms] -
appsec 3.779 ms [3.562 ms, 3.997 ms] 2.288 ms (153.4%)
iast 2.277 ms [2.207 ms, 2.346 ms] 784.974 µs (52.6%)
iast_GLOBAL 2.328 ms [2.258 ms, 2.399 ms] 836.686 µs (56.1%)
profiling 2.107 ms [2.052 ms, 2.162 ms] 615.135 µs (41.2%)
tracing 2.083 ms [2.029 ms, 2.137 ms] 591.475 µs (39.7%)

jbachorik and others added 3 commits April 13, 2026 19:07
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Deprecate misleading PROFILING_ALLOCATION_ENABLED_DEFAULT constant
- Update PROFILING_HEAP_ENABLED_DEFAULT javadoc to explain dynamic default
- Fix PROFILING_DIRECT_ALLOCATION_ENABLED missing @deprecated javadoc
- Drop JavaMonitorWait/ThreadPark/ThreadSleep from wall gate (timeline events)
- Add startup-only/no-RC comment to feature gate block in OpenJdkController
- Add BiasedLock noop comment for JDK 18+ awareness
- Add lock sampler note to DatadogProfilerConfig
- Extract DirectMemoryProfilingHelper shared utility
- Add gate tests: CPU, exception, I/O, lock, thread, wall-timeline
- Set metadata defaults to false (safe fallback for dynamic defaults)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@jbachorik jbachorik force-pushed the jb/prof-10633-unified-config branch from a020726 to d19b09b Compare April 13, 2026 17:08
@jbachorik jbachorik marked this pull request as ready for review April 14, 2026 13:46
@jbachorik jbachorik requested review from a team as code owners April 14, 2026 13:46
@jbachorik jbachorik requested review from dougqh and removed request for a team April 14, 2026 13:46
@jbachorik jbachorik requested a review from anandreshmi46 April 21, 2026 17:38
"aliases": []
}
],
"DD_PROFILING_DIRECTALLOCATION_ENABLED": [
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like this is a duplicate entry? A new entry with alias is already present


public static final String PROFILING_CPU_ENABLED = "profiling.cpu.enabled";
public static final boolean PROFILING_CPU_ENABLED_DEFAULT = true;
public static final String PROFILING_WALL_ENABLED = "profiling.walltime.enabled";
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All other constant mirrors config keys, should the constant here also should be renamed to PROFILING_WALLTIME_ENABLED? In supported-configurations.json its PROFILING_WALLTIME_ENABLED

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch! Should be WALLTIME

"aliases": []
}
],
"DD_PROFILING_DIRECTALLOCATION_ENABLED": [
Copy link
Copy Markdown

@anandreshmi46 anandreshmi46 Apr 22, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like the alias direction is inverted , the rest of this file always makes the new key primary and the old key the alias. Since PROFILING_DIRECT_MEMORY_ENABLED is the new replacement and PROFILING_DIRECT_ALLOCATION_ENABLED is Deprecated, I think this should be:
"DD_PROFILING_DIRECT_MEMORY_ENABLED": [{ "aliases": ["DD_PROFILING_DIRECTALLOCATION_ENABLED"] }]

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. I agree, this looks odd. However, I was not able to pass the validation of the config flags, IIRC.
I can try again, but I thing the config telemetry was not happy about removing the top level DD_PROFILING_DIRECTALLOCATION_ENABLED and keeping it as an alias or something.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am trying something to make this less confusing; let's see what system tests will tell us

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, I know. I was not able to add a new key with alias to an old key in the feature parity dashboard.
I tried the change but the validation fails - https://gitlab.ddbuild.io/datadog/apm-reliability/dd-trace-java/builds/1619017677

I will revert the debug commit.

@jbachorik jbachorik force-pushed the jb/prof-10633-unified-config branch from 13a2570 to 3ead772 Compare April 22, 2026 12:16
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR introduces unified, feature-level profiling gates (profiling.<feature>.enabled) to consistently enable/disable GA profiling features across JFR and ddprof backends, plus exposes these gates in profiler diagnostics.

Changes:

  • Add new unified config keys/constants for CPU, walltime, exception, I/O, lock, thread, and direct-memory profiling gates.
  • Wire the new gates into ddprof config evaluation, OpenJDK JFR event settings, and direct-memory instrumentation enablement (with fallback to deprecated direct-allocation flag).
  • Extend profiler flare reporting and OpenJDK controller tests to cover the new gating behavior.

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 6 comments.

Show a summary per file
File Description
metadata/supported-configurations.json Adds metadata entries for the new unified gate env vars (but contains a duplicate key + alias issues).
dd-trace-api/src/main/java/datadog/trace/api/config/ProfilingConfig.java Adds unified gate constants and deprecations for old/dynamic defaults (but has broken Javadoc links).
dd-java-agent/instrumentation/java/java-nio-1.8/.../FileChannelImplInstrumentation.java Switches enablement check to unified direct-memory gate with deprecated-key fallback.
dd-java-agent/instrumentation/java/java-nio-1.8/.../DirectByteBufferInstrumentation.java Switches enablement check to unified direct-memory gate with deprecated-key fallback.
dd-java-agent/instrumentation/java/java-nio-1.8/.../ByteBufferInstrumentation.java Switches enablement check to unified direct-memory gate with deprecated-key fallback.
dd-java-agent/agent-profiling/profiling-ddprof/.../DatadogProfilerConfig.java Applies unified CPU + walltime gates before backend-specific ddprof flags.
dd-java-agent/agent-profiling/profiling-controller/.../ProfilerSettingsSupport.java Captures unified gate values into controller settings for diagnostics/toString.
dd-java-agent/agent-profiling/profiling-controller/.../ProfilerFlareReporter.java Adds a “Feature Gates” section reporting unified gate state (with direct-memory fallback logic).
dd-java-agent/agent-profiling/profiling-controller-openjdk/src/test/.../OpenJdkControllerTest.java Adds tests asserting unified gates disable the corresponding JFR events (and walltime does not disable timeline events).
dd-java-agent/agent-profiling/profiling-controller-openjdk/src/main/.../OpenJdkController.java Disables relevant JFR events when unified gates are turned off.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines +2924 to +2931
"DD_PROFILING_DIRECTALLOCATION_ENABLED": [
{
"version": "A",
"type": "boolean",
"default": "false",
"aliases": ["DD_PROFILING_DIRECT_MEMORY_ENABLED"]
}
],
Copy link

Copilot AI Apr 23, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

metadata/supported-configurations.json now defines DD_PROFILING_DIRECTALLOCATION_ENABLED twice (one earlier with aliases: [], and again here with an alias). JSON duplicate keys are ambiguous (most parsers keep only the last) and will likely break the metadata tooling. Also, the alias direction looks reversed for a deprecation: if DD_PROFILING_DIRECT_MEMORY_ENABLED is the new canonical key, it should be its own entry with DD_PROFILING_DIRECTALLOCATION_ENABLED as an alias, rather than adding the new key as an alias of the deprecated one.

Copilot uses AI. Check for mistakes.
Comment on lines +2676 to +2683
"DD_PROFILING_CPU_ENABLED": [
{
"version": "A",
"type": "boolean",
"default": "true",
"aliases": []
}
],
Copy link

Copilot AI Apr 23, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR description says the metadata defaults for DD_PROFILING_HEAP_ENABLED and DD_PROFILING_ALLOCATION_ENABLED were updated to "true", but in metadata/supported-configurations.json they still read as "false" (e.g., DD_PROFILING_ALLOCATION_ENABLED default is false and DD_PROFILING_HEAP_ENABLED default is false). Either update those defaults as described, or adjust the PR description to match the actual behavior/metadata output.

Copilot uses AI. Check for mistakes.
Comment on lines 73 to +76
public static boolean isCpuProfilerEnabled(ConfigProvider configProvider) {
if (!configProvider.getBoolean(PROFILING_CPU_ENABLED, PROFILING_CPU_ENABLED_DEFAULT)) {
return false;
}
Copy link

Copilot AI Apr 23, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The new unified gates in isCpuProfilerEnabled / isWallClockProfilerEnabled change ddprof enablement behavior, but there are no unit tests covering that profiling.cpu.enabled=false or profiling.walltime.enabled=false forces these methods to return false even when the backend-specific profiling.ddprof.* flags are enabled. Since this module already has DatadogProfilerConfigTest, add targeted test cases to prevent regressions.

Copilot uses AI. Check for mistakes.
}

// jdk.JavaMonitorWait, jdk.ThreadPark, and jdk.ThreadSleep are intentionally NOT gated by
// PROFILING_WALL_ENABLED: they serve purposes beyond wall-clock profile construction
Copy link

Copilot AI Apr 23, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment refers to PROFILING_WALL_ENABLED, but the unified gate constant/key is PROFILING_WALLTIME_ENABLED (profiling.walltime.enabled). Updating the comment avoids confusion when searching for the controlling flag.

Suggested change
// PROFILING_WALL_ENABLED: they serve purposes beyond wall-clock profile construction
// PROFILING_WALLTIME_ENABLED: they serve purposes beyond wall-clock profile construction

Copilot uses AI. Check for mistakes.
Comment on lines +281 to +286
if (!configProvider.getBoolean(PROFILING_CPU_ENABLED, PROFILING_CPU_ENABLED_DEFAULT)) {
disableEvent(recordingSettings, "jdk.ExecutionSample", "CPU profiling is disabled");
disableEvent(recordingSettings, "jdk.NativeMethodSample", "CPU profiling is disabled");
disableEvent(recordingSettings, "jdk.CPUTimeSample", "CPU profiling is disabled");
disableEvent(recordingSettings, "jdk.CPUTimeSamplesLost", "CPU profiling is disabled");
}
Copy link

Copilot AI Apr 23, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

disableEvent logs "Disabling JFR event ... because it is {}"; passing reasons like "CPU profiling is disabled" produces awkward log messages ("because it is CPU profiling is disabled"). Consider using adjective/phrase-style reasons consistent with existing constants (e.g., "disabled by CPU profiling gate").

Copilot uses AI. Check for mistakes.
Comment on lines +14 to +26
/**
* @deprecated Not used. The actual default is computed dynamically by {@link
* datadog.trace.api.profiling.ProfilingSupport#isObjectAllocationSampleAvailable()}.
*/
@Deprecated public static final boolean PROFILING_ALLOCATION_ENABLED_DEFAULT = true;

public static final String PROFILING_HEAP_ENABLED = "profiling.heap.enabled";
public static final boolean PROFILING_HEAP_ENABLED_DEFAULT = false;

/**
* @deprecated The old value was {@code false}. The default is now computed dynamically via {@link
* datadog.trace.api.profiling.ProfilingSupport#isLiveHeapProfilingSafe()}.
*/
@Deprecated public static final boolean PROFILING_HEAP_ENABLED_DEFAULT = false;
Copy link

Copilot AI Apr 23, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Javadoc {@link datadog.trace.api.profiling.ProfilingSupport#...} references a type that doesn't exist in this module/repo (the runtime helper is com.datadog.profiling.controller.ProfilingSupport). As written, these links are broken/misleading; consider removing the {@link} or switching to plain text/{@code ...} so the API Javadoc doesn’t point to a non-existent class.

Copilot uses AI. Check for mistakes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp: profiling Profiling tag: ai generated Largely based on code generated by an AI or LLM type: enhancement Enhancements and improvements

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants