Replace rest-client with a persistent HTTP client for worker->web calls

[brianlball]

Problem

Every worker RestClient call in RunSimulateDataPoint opens a fresh TCP connection and leaves a TIME_WAIT socket behind; each closed connection also holds an nf_conntrack entry for ~120s. A datapoint job makes ~8-20 such calls (delete result_files, datapoint/analysis JSON gets, 5-15 report uploads). At high worker counts on the Kubernetes deployment path this churn exhausts nf_conntrack_max on the nodes hosting the web tier.

Fix

Route the five call sites through a per-process persistent client (OsHttp, backed by net-http-persistent, already in the bundle): ~8-20 connections per job drop to ~1-2. Resque forks a child per job and the client is built lazily, so each child opens its own connection on first use; DJ workers are long-lived and reuse the connection across jobs.

Based on a patch by Dan, with the following behavior-parity fixes:

• Multipart part Content-Type: Net::HTTP#set_form defaults file parts to application/octet-stream; download_result_file stores the part type and serves only text/html/application/json/text/plain inline, so HTML/JSON reports would force-download. The client now mime-maps by extension exactly as rest-client's MIME::Types guess did (part opts must use Symbol keys - String keys are silently ignored).
• Non-2xx raises OsHttp::Error (a StandardError), .code is an Integer, JSON.parse(response) works via to_str - existing retry loops behave unchanged.
• Client idle_timeout (15s) below Puma's 20s persistent timeout so idle connections are reopened instead of racing server-side closes on POSTs.
• Singleton rebuilds if os_server_host_url changes (the run_simulation feature specs repoint it at a per-process Capybara server; rest-client read APP_CONFIG per call).
• rest-client moves to the test group: the live-stack specs still drive the API with it; specs that relied on Bundler auto-require now require it explicitly.

Testing

• New socket-level specs (server/spec/lib/os_http_spec.rb, wired into run-server-tests.sh) assert the actual claim: sequential requests share one accepted TCP connection, reconnect after server close, error/response parity, multipart wire format.
• Re-enabled dj_run_simulation_data_point_spec and resque_run_simulation_data_point_spec in the docker CI job - they are the job-level integration tests for this path but had not run in CI since 2020. Verified green (7/7 and 2/2, two clean rounds) against the full docker-compose test stack, which required two bitrot/infra fixes:
  • openstudio-workflow no longer logs Completed the EnergyPlus simulation; assert on EnergyPlus returned '0' instead.
  • Fixed a cross-spec deletion race: Analysis#before_destroy enqueues DjJobs::DeleteAnalysis (an rm_rf of the analysis dir). Cleanup hooks outside the foreground around-wrapper enqueued it for the web-background container to run later, and because the spec formulations reuse a fixed analysis uuid, the deferred rm_rf deleted the shared analysis dir mid-run of the next spec. New destroy_projects_inline helper forces the deletion inline (also applied to the Enhance InitializeAnalysis resilience for corrupt seed.zip files #841 spec hooks that race them), plus a mkdir_p guard so the initialize-worker error path cannot ENOENT and mask the real failure.
• Stored result-file content types verified live from the stack run: eplustbl.html -> text/html, objectives.json -> application/json (inline disposition preserved).

Note: fix/run-simulate-data-point-hardening touches the same job file in different hunks; happy to coordinate merge order.

🤖 Generated with Claude Code