[BlackboxBenchmarking] Index the Fuzzer.builtin field#5263
Merged
Conversation
dylanjew
changed the title
dylanjew
force-pushed
the
dylanj/builtin-index
branch
from
865a705 to
e58449a
Compare
Collaborator
Author
|
Fixed lint error for the unused import |
ViniciustCosta
pushed a commit
that referenced
this pull request
May 11, 2026
…5265) Adds cron job to aggregate fuzzer stats into a daily bigquery table `fuzzer_stats.daily_stats`. ## Context We will use this to benchmark our blackbox fuzzers, previously we couldn't easily join the fuzzing hours from BigQuery with the bugs filed by clusterfuzz in our dashboards. We need a separate aggregated table because the `fuzzer_stats` `JobRun` tables are all in separate datasets per fuzzer, and we can't simply query across all of those datasets in BigQuery or Plx. The cron job defaults to yesterdays stats so we can run it after the stats are loaded into bigquery, but takes a date flag so we can backfill days as necessary. ### Idempotency Whenever a date is inserted, the schema uses `WRITE_TRUNCATE` with a date partition to overwrite all of the rows for that date. So if the job runs multiple times for the same day, it will not add additional rows but overwrite any previous rows for that date. This simplifies edge cases where the job fails or runs multiple times. We can just make sure the last run of the job succeeds and the data will be correct. It will just pull in the latest data on the JobRun tables for the fuzzers. #### Example query: ``` select fuzzer_name, SUM(fuzzing_duration) as fuzzing_duration, SUM(testcases_executed) as testcases_executed, from `your-project.fuzzer_stats.daily_stats` group by fuzzer_name order by fuzzing_duration desc limit 1000; ``` The remaining work here is to set up the cron job configuration. This PR only adds the logic for the job. [crbug.com/501066151](https://crbug.com/501066151) ### Related PRs: These migrate the bigquery and datastore schemas to support the new fields #5264 #5263 ### Testing Ran this against the dev data and verified that the fuzzer stats bigquery table is populated. Logs from dev: https://paste.googleplex.com/4884361662038016 After the job inserted the aggregated rows into BigQuery, I was able to compare the aggregated testcase stats and fuzzing hours between fuzzers for a given date range.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
We use this field to determine whether a fuzzer is blackbox or an engine guided fuzzer. In order to aggregate stats for blackbox fuzzers, we need to be able to query by the
builtinfield, which requires it to be indexed.https://docs.cloud.google.com/datastore/docs/concepts/indexes#unindexed_properties
The migration to index the field for existing fuzzers just requires rewriting the Fuzzers in the datastore without actually making any changes. Then the index will be created under the hood.
Testing
I ran this in dev and verified I can filter by
builtin == False