.env-based secret param binding for Functions#10714
Conversation
…e params right now
Wiz Scan Summary
To detect these findings earlier in the dev lifecycle, try using Wiz Code VS Code Extension. |
14c6bec to
d479b2d
Compare
| "^" + // start of string | ||
| "csm:\/\/" + // scheme csm:// | ||
| "([a-z](?:[-a-z0-9]{0,61}[a-z0-9])?)" + // secret ID is a GCP resource ID | ||
| "(?:\/([-a-z0-9]+))?" + // capture an optional GCP label for version |
There was a problem hiding this comment.
It is either a number or "latest"
There was a problem hiding this comment.
Discussed over chat: GCP implemented a new feature where users could create arbitrary aliases for specific versions.
| foo: "bar", | ||
| FIREBASE_SECRET_REF_baz: "quux", | ||
| } as Record<string, string>; | ||
| const { userEnvs: userEnvs, secretRefs: secretRefs } = prepare.partitionUserEnvs(input); |
There was a problem hiding this comment.
FYI: functional.ts already has a partition function. It may be worth just making that support partitioning objects not just arrays.
There was a problem hiding this comment.
Actually, I stand corrected; functional.ts already has partitionRecord which seems like it should be used instead of a custom function.
There was a problem hiding this comment.
It's almost there but the fact that partitionUserEnvs modifies the keys in the process of partitioning (to strip the special key prefix) makes it kind of annoying to use directly. You'd have to take a second pass with something that basically does a mapObject but mutates keys instead of values. Nothing I tried looked any more readable than just having a custom function, unfortunately.
| default: false, | ||
| public: true, | ||
| }, | ||
| secretEnvParams: { |
There was a problem hiding this comment.
What is the rationale for keeping this experiment guarded? This won't affect any existing behavior because the key was previously disallowed
There was a problem hiding this comment.
Uh at first this was written before API review so I had to guard it in case it was merged before formal approval. I actually do like the idea of guarding it until the params implementation is done though, since anyone who tries to use this without param prompting is in for some undefined and unpleasant behavior.
…y for .env secrets
| export const SECRET_REF_PREFIX = "FIREBASE_SECRET_REF_"; | ||
| // prettier-ignore | ||
| const GCP_RESOURCE_ID_PATTERN = "([a-z](?:[-a-z0-9]{0,61}[a-z0-9])?)" | ||
| export const SECRET_REF_SHORT_RE = new RegExp( |
There was a problem hiding this comment.
Wow... took me so long to find something canonical but https://docs.cloud.google.com/secret-manager/docs/reference/rest/v1/projects.secrets/create#query-parameters does say that upper case letters are allowed. Also, there's no mention that it must start with an alpha
There was a problem hiding this comment.
Good find. The validation predicator on the internal proto also matches that.
| const FUNCTIONS_EMULATOR_DOTENV = ".env.local"; | ||
|
|
||
| const RESERVED_PREFIXES = ["X_GOOGLE_", "FIREBASE_", "EXT_"]; | ||
| const RESERVED_KEY_PREFIXES = ["FIREBASE_SECRET_REF"]; |
There was a problem hiding this comment.
Nit: isn't a trailing _ part of the prefix too? Also, I wonder if there's a better name for this that makes it clearer.
There was a problem hiding this comment.
Honestly, everything I thought of sounded clunky. RESERVED_PREFIX_ALLOWLIST, maybe?
| .filter((e) => e.codebase === codebase || e.codebase === undefined); | ||
| await resolveDefaultRegionsForBuild(wantBuild, backend.of(...relevantEndpoints)); | ||
|
|
||
| if (experiments.isEnabled("secretEnvParams")) { |
There was a problem hiding this comment.
I think I was wrong regarding whether this is good to have as an experiment because it can change existing behavior, but not here. I would assume that the best place to secret-guard is when we prompt for a secret value in resolving the build, but if they already have a codebase that uses secret refs, just move forwards with it. This prevents weird errors where we're collaborating on a codebase and it works on my computer but not yours.
There was a problem hiding this comment.
Yeah, I can see that. Didn't consider the collaboration angle.
e9833dd to
32ec1ee
Compare
| ); | ||
| } | ||
| let notFound = true; | ||
| endpoint.secretEnvironmentVariables?.forEach((envVar) => { |
There was a problem hiding this comment.
style guide says to use for ... of last I checked.
| * @return An object populated with project id, secret id, and version, with a field undefined if not provided. | ||
| */ | ||
| export function parseSecretRef(ref: string): ParsedSecretRef { | ||
| const shortMatch = SECRET_REF_SHORT_RE.exec(ref); |
There was a problem hiding this comment.
nit: Can JS REs do named captures? Would that be a better simplification here?
There was a problem hiding this comment.
I considered it, but the way i have the last capture group for the version jankily split up between multiple lines of the composed regex (so I can deal with the #/@ special cases) would make it kind of awkward to keep consistent here.
| .filter((e) => e.codebase === codebase || e.codebase === undefined); | ||
| await resolveDefaultRegionsForBuild(wantBuild, backend.of(...relevantEndpoints)); | ||
|
|
||
| const parsedSecretRefs = Object.fromEntries( |
There was a problem hiding this comment.
This is already in functional.ts as mapObject
| foo: "bar", | ||
| FIREBASE_SECRET_REF_baz: "quux", | ||
| } as Record<string, string>; | ||
| const { userEnvs: userEnvs, secretRefs: secretRefs } = prepare.partitionUserEnvs(input); |
There was a problem hiding this comment.
Actually, I stand corrected; functional.ts already has partitionRecord which seems like it should be used instead of a custom function.
| @@ -10,6 +10,7 @@ import { logBullet, logWarning } from "../utils"; | |||
| const FUNCTIONS_EMULATOR_DOTENV = ".env.local"; | |||
|
|
|||
| const RESERVED_PREFIXES = ["X_GOOGLE_", "FIREBASE_", "EXT_"]; | |||
There was a problem hiding this comment.
Maybe it would be good to add KIT_ to this now and see if anyone yells before we start depending on it being a reserved prefix.
There was a problem hiding this comment.
Oooh, that's a good idea.
Pulls secrets from the Functions .env files and propagates them into the Functions Build's SecretEnvVars (and from there, into the resolved Backend, and from there, into the JSON uploaded to Cloud Functions/Cloud Run).
Params support not included in this PR, to reduce the size of the review. This is obviously going to be a terrible user experience before params are supported (hence the experiment flag) but this should be enough to test the actual binding behavior.
Quick guide to file change intent:
environments.ts - Adds an experiment flag which by default prevents any changes from being made to the Build. If the experiment is not enabled, FIREBASE_SECRET_REF_ envs will still be accepted by the parser, but they'll do nothing (and will not appear in process.env)
env.ts - Modifies the key validation logic to allow keys that begin with FIREBASE_SECRET_REF_ even though FIREBASE_ is reserved. Otherwise knows nothing about this feature.
prepare.ts - Partitions the key-value pairs passed from env.ts into the normal entries and the secrets, which have FIREBASE_SECRET_REF_ stripped from the key at this point. If feature flag enabled, calls build.applyEnvSecretOverrides()
build.ts - Implements applyEnvSecretOverrides, which mutates a Build's state to upsert .env-defined secrets into SecretEnvVars before the Build is compiled into a Functions Backend. These secrets contain an
allowVersionPinningsentinel value if the version is specified, which will be relevant real soon now.backend.ts - Adds the
allowVersionPinningto a Backend's version of a SecretEnvVar so it can propagate to...validate.ts - Bypasses the logic that forces secret versions to always resolve to the current
latestifallowVersionPinningis set.