Skip to content

Mysterious segfault/pipeline failures #20

Description

@feefladder

As discussed with @leandroleal, there are mysterious segfaults every now and then, only when it runs in a loop.

The good news:

  • Bug fixed wrt. read_rasters_cpp where it'd abort on inserting a single int
  • we now have so many tests that they cause erratic behaviour!
    the bad news:
  • erratic behaviour

example:


tests/data/test_toy.py .F                                                                                                                                                            [  3%]
tests/io/test_base.py ...                                                                                                                                                            [  9%]
tests/io/test_read_data.py .                                                                                                                                                         [ 11%]
tests/io/test_read_rasters.py .Deleting
F...                                                                                                                                                  [ 21%]
tests/io/test_read_rasters_compare.py .F.                                                                                                                                            [ 27%]
tests/io/test_read_rasters_cpp.py ......x..                                                                                                                                          [ 45%]
tests/parallel/test_utils.py .                                                                                                                                                       [ 47%]
tests/test_catalog.py ........                                                                                                                                                       [ 62%]
tests/test_io.py .......                                                                                                                                                             [ 76%]
tests/test_misc.py ........                                                                                                                                                          [ 92%]
tests/test_overlay.py ....                                                                                                                                                           [100%]

========================================================================================= FAILURES =========================================================================================
_____________________________________________________________________________________ test_ndvi_rdata ______________________________________________________________________________________

    def test_ndvi_rdata() -> None:
>       rdata: RasterData = toy.ndvi_rdata()

tests/data/test_toy.py:10:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
skmap/data/toy.py:52: in ndvi_rdata
    .read()
skmap/io/base.py:1310: in read
    self.array = read_rasters(
skmap/io/base.py:665: in read_rasters
    for array, raster_idx, data_exists in parallel.job(
skmap/parallel/utils.py:240: in job
    for worker_result in parallel(delayed(worker)(*args) for args in worker_args):
.venv/lib/python3.10/site-packages/joblib/parallel.py:1682: in _get_outputs
    yield from self._retrieve()
.venv/lib/python3.10/site-packages/joblib/parallel.py:1784: in _retrieve
    self._raise_error_fast()
.venv/lib/python3.10/site-packages/joblib/parallel.py:1859: in _raise_error_fast
    error_job.get_result(self.timeout)
.venv/lib/python3.10/site-packages/joblib/parallel.py:758: in get_result
    return self._return_or_raise()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <joblib.parallel.BatchCompletionCallBack object at 0x7fc585f94af0>

    def _return_or_raise(self):
        try:
            if self.status == TASK_ERROR:
>               raise self._result
E               joblib.externals.loky.process_executor.TerminatedWorkerError: A worker process managed by the executor was unexpectedly terminated. This could be caused by a segmentation fault while calling the function or by an excessive memory usage causing the Operating System to kill the worker.
E
E               The exit codes of the workers are {SIGINT(-2)}
E               Detailed tracebacks of the workers should have been printed to stderr in the executor process if faulthandler was not disabled.

.venv/lib/python3.10/site-packages/joblib/parallel.py:773: TerminatedWorkerError

enBLAS blas_thread_init: pthread_create failed for thread 63 of 64: Resource temporarily unavailable
OpenBLAS blas_thread_init: ensure that your address space and process count limits are big enough (ulimit -a)
OpenBLAS blas_thread_init: or set a smaller OPENBLAS_NUM_THREADS to fit into what you have available
OpenBLAS blas_thread_init: RLIMIT_NPROC 513308 current, 513308 max

Traceback (most recent call last):
  File "/var/home/fee/.local/share/uv/python/cpython-3.10.19-linux-x86_64-gnu/lib/python3.10/runpy.py", line 187, in _run_module_as_main
    mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
  File "/var/home/fee/.local/share/uv/python/cpython-3.10.19-linux-x86_64-gnu/lib/python3.10/runpy.py", line 110, in _get_module_details
    __import__(pkg_name)
  File "/var/home/fee/git/scikit-map/.venv/lib/python3.10/site-packages/joblib/__init__.py", line 115, in <module>
    from ._parallel_backends import ParallelBackendBase
  File "/var/home/fee/git/scikit-map/.venv/lib/python3.10/site-packages/joblib/_parallel_backends.py", line 21, in <module>
    from .executor import get_memmapping_executor
  File "/var/home/fee/git/scikit-map/.venv/lib/python3.10/site-packages/joblib/executor.py", line 11, in <module>
    from ._memmapping_reducer import TemporaryResourcesManager, get_memmapping_reducers
  File "/var/home/fee/git/scikit-map/.venv/lib/python3.10/site-packages/joblib/_memmapping_reducer.py", line 28, in <module>
    import numpy as np
  File "/var/home/fee/git/scikit-map/.venv/lib/python3.10/site-packages/numpy/__init__.py", line 114, in <module>
    from numpy.__config__ import show_config
  File "/var/home/fee/git/scikit-map/.venv/lib/python3.10/site-packages/numpy/__config__.py", line 4, in <module>
    from numpy._core._multiarray_umath import (
  File "/var/home/fee/git/scikit-map/.venv/lib/python3.10/site-packages/numpy/_core/__init__.py", line 23, in <module>
    from . import multiarray
  File "/var/home/fee/git/scikit-map/.venv/lib/python3.10/site-packages/numpy/_core/multiarray.py", line 10, in <module>
    from . import overrides
  File "/var/home/fee/git/scikit-map/.venv/lib/python3.10/site-packages/numpy/_core/overrides.py", line 7, in <module>
    from numpy._core._multiarray_umath import (
  File "<frozen importlib._bootstrap>", line 1022, in _find_and_load
KeyboardInterrupt

(I did not keyboardinterrupt)

or sometimes it just stops the testing process

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions