Skip over bad entries in UAL while parsing data#59
Open
JazzCore wants to merge 1 commit into
Open
Conversation
Schamper
requested changes
Jun 10, 2026
Some entries in UAL ese database can contain bad data - tombstoned records without data (no defined columns), or have mismatch between UAL table schema and actual number of columns in entry. Such records previously caused exceptions while parsing them, so instead parsing now we skip them.
Schamper
requested changes
Jun 16, 2026
| """Return whether the record is a tombstone or empty (has no defined column data).""" | ||
| return self._last_fixed_id == 0 and self._last_variable_id == 0 and self._tagged_data_count == 0 | ||
|
|
||
| def matches_schema(self) -> bool: |
Member
There was a problem hiding this comment.
Probably this can be a property too?
Comment on lines
+214
to
+221
| num_fixed, num_variable, num_tagged = self.table.column_counts | ||
| return all( | ||
| ( | ||
| self._last_fixed_id <= num_fixed, | ||
| (self._last_variable_id - 127) <= num_variable, | ||
| self._tagged_data_count <= num_tagged, | ||
| ) | ||
| ) |
Member
There was a problem hiding this comment.
Perhaps until we have other places to use it, it can be fine to just inline the logic from column_counts in here. So we don't need the change to the Table class.
| self._get_tag_field = lru_cache(4096)(self._get_tag_field) | ||
| self._find_tag_field_idx = lru_cache(4096)(self._find_tag_field_idx) | ||
|
|
||
| @property |
Member
There was a problem hiding this comment.
Perhaps add a is_valid property that combines both is_empty and matches_schema?
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
While running
ual.clients_accesson one machine got this exception:ual.clients_seengave me same exception, but now inDNSUAL table.For
CLIENTStable this was caused by entry without any data in it - all columns for it were not defined, soNonewas passed inwintimestamp, which gave exception.For
DNStable - faulty entry had more defined columns than DNS schema expected, which resulted in regular string passed towintimestamp.First i went with a fix to
wintimestampto use try-except and just return None on bad timestamps, but IMO it does not cover all cases, so instead I ended up adding verification for UAL entries to check if they are empty or matches table schema. For schema matching it just checks that entry does not have more columns than in schema (a lot of them have lower values, for ese it just means that value in that column is empty, as I understood).is_emptyandmatches_schemacan be used in anotheresecode, but since it will skip over data that can be carved in some cases (which can be useful), I limited it to just UAL.Test for it was a real pain because of double checksumming with ECC in
ese. Used existing test and added to it empty and schema-mismatch records to give same exceptions as in my case. DB is correct and can be parsed with Eric Zimmerman'sSumECmd(but it requires patching to remove hardcoded requirement forSystemIdentity.mdb). I verified this test (and my faulty real UAL) against it to ensure that results matches.