Program hangs forever when adding two items at same path

[benoit74]

Given the following program:

from zimscraperlib.zim import Creator
from pathlib import Path
from zimscraperlib.zim import metadata

creator = Creator(Path("tests.zim"), "index.html").config_metadata(
    std_metadata=metadata.DEFAULT_DEV_ZIM_METADATA
)

creator.start()

creator.add_item_for("index.html", "Main Page", content="Main", is_front=True )

for i in range(10):
    creator.add_item_for(f"content{i}", content="P" * 1000000)

creator.add_item_for("conflict", content="Foo")
creator.add_item_for("conflict", content="Bar")

I get following logs:

Traceback (most recent call last):
  File "/home/benoit/Tools/zimcreateconflict.py", line 17, in <module>
    creator.add_item_for("conflict", content="Bar")
    ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<@beartype(zimscraperlib.zim.creator.Creator.add_item_for) at 0x7b7c6f51f5e0>", line 248, in add_item_for
  File "/home/benoit/Tools/venv3.14/lib/python3.14/site-packages/zimscraperlib/zim/creator.py", line 365, in add_item_for
    self.add_item(
    ~~~~~~~~~~~~~^
        StaticItem(
        ^^^^^^^^^^^
    ...<10 lines>...
        duplicate_ok=duplicate_ok,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "<@beartype(zimscraperlib.zim.creator.Creator.add_item) at 0x7b7c6f51f740>", line 83, in add_item
  File "/home/benoit/Tools/venv3.14/lib/python3.14/site-packages/zimscraperlib/zim/creator.py", line 410, in add_item
    raise exc
  File "/home/benoit/Tools/venv3.14/lib/python3.14/site-packages/zimscraperlib/zim/creator.py", line 407, in add_item
    super().add_item(item)
    ~~~~~~~~~~~~~~~~^^^^^^
  File "libzim/libzim.pyx", line 517, in libzim._Creator.add_item
RuntimeError: Impossible to add C/conflict
  dirent's title to add is : conflict
  existing dirent's title is : conflict

Which is normal. Scraper should not be doing this kind of silly thing.

What is abnormal is that the program hangs forever, waiting for something to complete. This should not happen (like the program should stop almost immediately).

I don't know it this is a pure libzim issue problem or something related to python-libzim (or python-scraperlib) so I'm asking for help to have confirmation regarding if you achieve to reproduce the problem in pure C++.

What seems to be key to reproduce the issue is to fill at least one cluster ; the more cluster we fill the hardest it is stop stop (with only range(10) in code above, python is still responding to Ctrl-C input. With range(100), I had to kill the program. With Without this initial content filling a cluster, everything works as expected (program terminates).

This has been discovered in openzim/maps#110

[benoit74]

Edit: fixed wrong word in last paragraph

[veloman-yunkan]

What is abnormal is that the program hangs forever,

I don't know anyone except Chuck Norris who is entitled to making such a claim. How long did you wait? 😉

[rgaudin]

I think we should first try to rule-out a scraperlib issue since there is a mechanism in place around this

See https://github.com/openzim/python-scraperlib/blob/main/src/zimscraperlib/zim/creator.py#L405

The printed stack trace indicates the exception was raised while it shouldn't: it matches the expected pattern

[veloman-yunkan]

I can't use the latest version of zimscraperlib since I am on python 3.10. Trying to reproduce it via the equivalentish libzim only script was not successful:

import libzim
from libzim.writer import Creator, StringProvider, Hint

DEFAULT_DEV_ZIM_METADATA = {
    "Name": "Test Name",
    "Title": "Test Title",
    "Creator": "Test Creator",
    "Publisher": "Test Publisher",
    "Date": "2023-01-01",
    "Description": "Test Description",
    "Language": "fra",
}

FRONT_ARTICLE_HINTS = {Hint.FRONT_ARTICLE : True}

class StaticItem(libzim.writer.Item):
    def __init__(self, **kwargs):
        super().__init__()
        for k, v in kwargs.items():
            setattr(self, k, v)

    def get_path(self) -> str:
        return getattr(self, "path", "")

    def get_title(self) -> str:
        return getattr(self, "title", "")

    def get_mimetype(self) -> str:
        return getattr(self, "mimetype", "text/html")

    def get_contentprovider(self) -> libzim.writer.ContentProvider:
        return StringProvider(content=getattr(self, "content", ""))

    def get_hints(self) -> dict[Hint, int]:
        return getattr(self, "hints", {})

creator = Creator("tests.zim")
creator.set_mainpath("index.html")
#creator.config_indexing(False, "eng")
with creator:
    for name, value in DEFAULT_DEV_ZIM_METADATA.items():
        creator.add_metadata(name, value)

    creator.add_item(StaticItem(path="index.html",
                                title="Main Page",
                                content="Main",
                                hints=FRONT_ARTICLE_HINTS))

    for i in range(10):
        creator.add_item(StaticItem(path=f"content{i}", content="P" * 1000000))

    creator.add_item(StaticItem(path="conflict", content="Foo"))
    creator.add_item(StaticItem(path="conflict", content="Bar"))

BTW, what is your version of libzim? I tried 9.4.1, 9.5.1, 9.6.0 and current main.

[benoit74]

9.5.1 ATM

[benoit74]

Thank you. Transferring to scraperlib for the time being.