[Fix] Full update of data_dir in DatasetInfo with update_data_dir()

[youliangtan]

Description

The previous DatasetInfo.update_data_dir() only updates the datadir in each split, but not its DatasetIdentity, making the update incomplete. The datadir property doesn't have a setter too, thus, this adds a single liner to update the identity when the update_data_dir() is called.

[google-cla]

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

[fineguy]

Thanks @youliangtan for your interest in TFDS!

Do you have an example when the update is incomplete causing issues?

We use this function only once in:

datasets/tensorflow_datasets/core/dataset_builder.py

Lines 702 to 703 in 47f7a2e

	# The generated DatasetInfo contains references to `tmp_data_dir`
	self.info.update_data_dir(self.data_dir)

to fix data_dir in splits, but the identity should already have the right data_dir after dataset builder was initialized:

datasets/tensorflow_datasets/core/dataset_builder.py

Lines 281 to 284 in 47f7a2e

	if self.data_path.exists():
	self.info.read_from_directory(self._data_dir)
	else: # Use the code version (do not restore data)
	self.info.initialize_from_bucket()

[youliangtan]

Thanks for the reply, I'm using this method to update a new data_dir, after DatasetInfo is constructed

The code example is here. This script is mainly to merge or reshard the rlds data, maybe there's better way of doing this too. 🤷‍♂️

[fineguy]

I think DatasetInfo.update_data_dir should be a private method and only used internally.

For your code I suggest to consider these options :

• Use other tools (for example https://github.com/google/seqio) to combine multiple datasets.
• Dynamically define a dataset builder that gets examples from multiple datasets in _generate_examples().
• Construct your dataset_info using

  datasets/tensorflow_datasets/core/dataset_info.py

  Line 101 in 47f7a2e

  class DatasetIdentity:

[youliangtan]

Thanks for the suggestion. I think I will keep the current code as light as it is for now without depending on external lib.

Also, could the update_data_dir() be a public method? else it should better to be named as _update_data_dir 🤔

[fineguy]

I agree, that would be a nice change.

[tomvdw]

Have you looked into builder_from_directories? (see

datasets/tensorflow_datasets/core/read_only_builder.py

Line 152 in 303814e

def builder_from_directories(

) This merges multiple datasets.