Increase size limit for JSON uploads to storage

[joehan]

Description

The Storage emulator was applying an artificially low limit on upload size when Content-Type: application/json was set. Fixes #8355

[aalej]

Tried testing this using https://github.com/aalej/issues-8355. Increasing the size limit removes the PayloadTooLargeError: request entity too large error, however the upload never seems to finish:

$ firebase emulators:exec --log-verbosity DEBUG --only storage -- 'curl "localhost:9199/upload/storage/v1/b/bucket/o?name=test&uploadType=media" -d "@large.json" -H "Content-Type: application/json" -o /dev/null -w "%{http_code}"'
i  emulators: Starting emulators: storage
i  Running script: curl "localhost:9199/upload/storage/v1/b/bucket/o?name=test&uploadType=media" -d "@large.json" -H "Content-Type: application/json" -o /dev/null -w "%{http_code}"
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  126k    0     0  100  126k      0    115  0:18:50  0:18:42  0:00:08     0^C
i  emulators: Received SIGINT (Ctrl-C) for the first time. Starting a clean shutdown.
i  emulators: Please wait for a clean shutdown or send the SIGINT (Ctrl-C) signal again to stop right now.
i  emulators: Shutting down emulators.
i  storage: Stopping Storage Emulator
⚠  Script exited with signal: SIGINT
i  logging: Stopping Logging Emulator
Using Google Analytics in DEBUG mode. Emulators (+ UI) events will be shown in GA Debug View only.
i  emulators: Shutting down emulators.

[joehan]

irebase emulators:exec --log-verbosity DEBUG --only storage -- 'curl "localhost:9199/upload/storage/v1/b/bucket/o?name=test&uploadType=media" -d "@large.json" -H "Content-Type: application/json" -o /dev/null -w "%{http_code}"

Good catch, i just repro'd this too. Investigating before merging - also need to double check that this is actually new behavior

EDIT: Just tested and this occurs even with an extremely small file - pretty sure we're doing something wrong in the curl, but I'm gonna poke around until I understand exactly whats going on

[mgraczyk]

@joehan FWIW there is a longstanding, separate bug where the emulator hangs forever when I upload certain files and set the Content-Type header.

This has been an issue for at least a year but I just rerun my tests automatically when this happens, and it never happens with prod GCS so I ignored it.

[mgraczyk]

@joehan I just looked, the problem is that the handler for the upload POST endpoint tries to read the body here, but it has already been read by express.json here

One fix would be to check for json in here:

As written, it will hang forever in reqBodyToBuffer whenever the body has already been read:

firebase-tools/src/emulator/shared/request.ts

Line 7 in 6c7245f

}

Since express.json writes an empty object {} for non-json Conten-Type headers, you can't just check for typeof request.body === "object", and may need to check the headers before the call to reqBodyToBuffer or something like that.

[andersonaddo]

@joehan what's the status on this? This is still a problem, as far as I can see

[joehan]

Hey @andersonaddo - this PR had been abandoned because it turned out the issue was a bit more complex than I originally thought. Closing it in favor of #10800, which I beleive actually fixes the issue.