Add benchmark test and optimize byte slice reuse in network_reader

[dlinsley]

Problem

During performance testing of the gorouter with the udp-forwarder configured, we observed CPU consumption spikes in the udp-forwarder when the gorouter exceeds 1,500 requests per second. Profiling a benchmark revealed that the garbage collector consumes the majority of the CPU time because NetworkReader allocates a new byte slice for every single read.

Solution

This PR updates NetworkReader to use a sync.Pool for byte slice allocation, allowing the slices to be reused rather than constantly reallocated.

Impact

This change flattens memory usage and prevents the garbage collector from being overwhelmed when a large number of UDP messages are sent to the forwarder, resolving the CPU spikes under high load.

Benchmark comparison

benchstat old.bench new.bench
goos: linux
goarch: amd64
pkg: code.cloudfoundry.org/loggregator-agent-release/src/pkg/ingress/v1
cpu: 11th Gen Intel(R) Core(TM) i7-11850H @ 2.50GHz
                      │  old.bench  │              new.bench              │
                      │   sec/op    │    sec/op     vs base               │
NetworkReader_8KB-16    33.25µ ± 5%   24.27µ ± 11%  -26.98% (p=0.002 n=6)
NetworkReader_16KB-16   33.34µ ± 3%   22.15µ ± 22%  -33.55% (p=0.002 n=6)
NetworkReader_32KB-16   42.19µ ± 3%   23.77µ ±  9%  -43.66% (p=0.002 n=6)
NetworkReader_48KB-16   53.80µ ± 3%   28.38µ ±  5%  -47.26% (p=0.002 n=6)
NetworkReader_64KB-16   60.46µ ± 4%   30.75µ ± 21%  -49.13% (p=0.002 n=6)
geomean                 43.30µ        25.67µ        -40.70%

                      │  old.bench   │             new.bench             │
                      │     B/op     │    B/op     vs base               │
NetworkReader_8KB-16     8284.0 ± 0%   125.0 ± 3%  -98.49% (p=0.002 n=6)
NetworkReader_16KB-16   16476.0 ± 0%   125.5 ± 3%  -99.24% (p=0.002 n=6)
NetworkReader_32KB-16   32860.0 ± 0%   124.5 ± 4%  -99.62% (p=0.002 n=6)
NetworkReader_48KB-16   49245.0 ± 0%   122.0 ± 2%  -99.75% (p=0.002 n=6)
NetworkReader_64KB-16   65629.0 ± 0%   123.5 ± 4%  -99.81% (p=0.002 n=6)
geomean                 26.42Ki        124.1       -99.54%

                      │ old.bench  │             new.bench              │
                      │ allocs/op  │ allocs/op   vs base                │
NetworkReader_8KB-16    5.000 ± 0%   5.000 ± 0%       ~ (p=1.000 n=6) ¹
NetworkReader_16KB-16   5.000 ± 0%   5.000 ± 0%       ~ (p=1.000 n=6) ¹
NetworkReader_32KB-16   5.000 ± 0%   5.000 ± 0%       ~ (p=1.000 n=6) ¹
NetworkReader_48KB-16   5.000 ± 0%   5.000 ± 0%       ~ (p=1.000 n=6) ¹
NetworkReader_64KB-16   5.000 ± 0%   5.000 ± 0%       ~ (p=1.000 n=6) ¹
geomean                 5.000        5.000       +0.00%
¹ all samples are equal

Type of change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• This change requires a documentation update

Testing performed?

• Unit tests
• Integration tests
• Acceptance tests

Checklist:

• This PR is being made against the main branch, or relevant version branch
• I have made corresponding changes to the documentation
• I have added testing for my changes

If you have any questions, or want to get attention for a PR or issue please reach out on the #logging-and-metrics channel in the cloudfoundry slack

[chombium]

Thanks for the PR @dlinsley. The change looks pretty nice. I will fo few more test and then merge it.

[dlinsley]

@cloudfoundry/wg-app-runtime-platform-logging-and-metrics-approvers please let me know if there are any concerns or changes you would like to see here.

[chombium]

Hi @dlinsley,

Thank you very much for your patience and for the fix. The performance improvements look impressive.
I have one small objection about the buffer size definition, apart from that LGTM :)

[chombium]

Great catch ;)

[chombium]

Thanks for the quick fix @dlinsley

LGTM!