Skip to content

Skip over bad entries in UAL while parsing data#59

Open
JazzCore wants to merge 1 commit into
fox-it:mainfrom
JazzCore:ual-fixes
Open

Skip over bad entries in UAL while parsing data#59
JazzCore wants to merge 1 commit into
fox-it:mainfrom
JazzCore:ual-fixes

Conversation

@JazzCore

Copy link
Copy Markdown

While running ual.clients_access on one machine got this exception:

Traceback (most recent call last):
  File "/home/user/venv/bin/target-query", line 6, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/user/venv/dissect/dissect.target/dissect/target/tools/utils/cli.py", line 475, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/venv/dissect/dissect.target/dissect/target/tools/query.py", line 260, in main
    for record in record_generator:
                  ^^^^^^^^^^^^^^^^
  File "/home/user/venv/dissect/dissect.target/dissect/target/plugins/os/windows/ual.py", line 226, in client_access
    for path, client_record in self.read_table_records("CLIENTS"):
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/venv/dissect/dissect.target/dissect/target/plugins/os/windows/ual.py", line 219, in read_table_records
    for table_record in parser.get_table_records(table_name):
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/venv/dissect/dissect.database/dissect/database/ese/tools/ual.py", line 61, in get_table_records
    value = wintimestamp(value)
            ^^^^^^^^^^^^^^^^^^^
  File "/home/user/venv/lib/python3.12/site-packages/dissect/util/ts.py", line 187, in wintimestamp
    return _calculate_timestamp(float(ts) * 1e-7 - 11_644_473_600)  # Thanks FireEye
                                ^^^^^^^^^
TypeError: float() argument must be a string or a real number, not 'NoneType'

ual.clients_seen gave me same exception, but now in DNS UAL table.

For CLIENTS table this was caused by entry without any data in it - all columns for it were not defined, so None was passed in wintimestamp, which gave exception.
For DNS table - faulty entry had more defined columns than DNS schema expected, which resulted in regular string passed to wintimestamp.

First i went with a fix to wintimestamp to use try-except and just return None on bad timestamps, but IMO it does not cover all cases, so instead I ended up adding verification for UAL entries to check if they are empty or matches table schema. For schema matching it just checks that entry does not have more columns than in schema (a lot of them have lower values, for ese it just means that value in that column is empty, as I understood).

is_empty and matches_schema can be used in another ese code, but since it will skip over data that can be carved in some cases (which can be useful), I limited it to just UAL.

Test for it was a real pain because of double checksumming with ECC in ese. Used existing test and added to it empty and schema-mismatch records to give same exceptions as in my case. DB is correct and can be parsed with Eric Zimmerman's SumECmd (but it requires patching to remove hardcoded requirement for SystemIdentity.mdb). I verified this test (and my faulty real UAL) against it to ensure that results matches.

Comment thread dissect/database/ese/record.py Outdated
Comment thread dissect/database/ese/record.py Outdated
Some entries in UAL ese database can contain bad data - tombstoned
records without data (no defined columns), or have mismatch
between UAL table schema and actual number of columns in entry.
Such records previously caused exceptions while parsing them, so
instead parsing now we skip them.
"""Return whether the record is a tombstone or empty (has no defined column data)."""
return self._last_fixed_id == 0 and self._last_variable_id == 0 and self._tagged_data_count == 0

def matches_schema(self) -> bool:

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably this can be a property too?

Comment on lines +214 to +221
num_fixed, num_variable, num_tagged = self.table.column_counts
return all(
(
self._last_fixed_id <= num_fixed,
(self._last_variable_id - 127) <= num_variable,
self._tagged_data_count <= num_tagged,
)
)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps until we have other places to use it, it can be fine to just inline the logic from column_counts in here. So we don't need the change to the Table class.

self._get_tag_field = lru_cache(4096)(self._get_tag_field)
self._find_tag_field_idx = lru_cache(4096)(self._find_tag_field_idx)

@property

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps add a is_valid property that combines both is_empty and matches_schema?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants