feat: create raw data loader and setup of repo#4
Open
arjunsridhar12345 wants to merge 23 commits into
Open
Conversation
arjunsridhar12345
requested review from
arielleleon,
dougollerenshaw and
seanmcculloch
seanmcculloch
approved these changes
May 28, 2026
seanmcculloch
left a comment
seanmcculloch
left a comment
There was a problem hiding this comment.
looks good to me, my comments were all addressed
arielleleon
requested changes
May 29, 2026
Collaborator
arielleleon
left a comment
arielleleon
left a comment
There was a problem hiding this comment.
Awesome start! Just a few small things
| @@ -0,0 +1,90 @@ | |||
| """Raw data loader for dynamic foraging datasets.""" | |||
Collaborator
There was a problem hiding this comment.
Can you rename this to loader.py? The names are too repetitive from dynamic_foraging_processing.raw_data_loader.raw_data_loader import RawDataLoader. The import at __init__ glosses over this but I would argue it needs to be renamed
| @@ -0,0 +1,139 @@ | |||
| """Tests for ``dynamic_foraging_processing.raw_data_loader.raw_data_loader``.""" | |||
Collaborator
There was a problem hiding this comment.
One you change raw_data_loader.py -> loader.py, this should become test_loader.py
|
|
||
| def __init__( | ||
| self, | ||
| path: os.PathLike, |
Collaborator
There was a problem hiding this comment.
Can you make this either t.Union[ str | os.Pathlike ] or just str | os.Pathlike. You already cast it as a Path in the make_dataset
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR attempts to address #3 by creating a module for loading the raw data using the data contract for the Dynamic Foraging experiment defined here: https://github.com/AllenNeuralDynamics/Aind.Behavior.DynamicForaging/blob/main/src/aind_behavior_dynamic_foraging/data_contract/_dataset.py.
Using this diagram as guidance: https://github.com/AllenNeuralDynamics/aind-software-docs/blob/main/diagrams/dynamic_foraging/low_level/dynamic-foraging-low-level-processing.svg
An initial example notebook has been made to try and show how to access the raw data:
https://github.com/AllenNeuralDynamics/dynamic-foraging-processing/blob/3-create-raw-data-loader/examples/raw_data_loader_example.ipynb
NOTE: The temp dataset file exists ONLY because test data has been acquired, and the test dataset I had does not have all of the streams captured that are defined in the data contract. This file will be removed once a full dataset is acquired and validated
In addition, the PR does some setup for the repo so it tries to align with the coding standards that were recently merged by a group of engineers from SIPE and Sci Comp as the current package template has not been updated yet. There may be things I missed or maybe messed up but at a high level is the following. Looking back, this might have been better off in a separate PR but these changes are relatively small and I think can be lumped in to the first PR for this repo:
Small note to finish: This PR targets the dev branch in accordance with the above coding standards
Another small note: I can't add branch protection rules, @dougollerenshaw could you add them or make me an admin on the repo?