Syncing from upstream OCA/storage (16.0)#374
Merged
Merged
Conversation
``FsFileGC._gc_files_unsafe`` loaded the entire backlog of orphan files into a single Python list (via ``array_agg(store_fname)`` grouped by storage) and iterated ``fs.rm`` over all of them in one shot. With the Azure Blob backend (``adlfs``, same class of issue on ``s3fs`` and other fsspec-based clients) and tens of thousands of queued orphans, each HEAD+DELETE pair held onto response buffers and connection-pool state inside the SDK that was only released when the worker exited. In production (Odoo.sh, 30k+ orphans, Azure Blob backend) every autovacuum run hit Odoo's ``limit_memory_hard`` and received ``SIGKILL`` at around minute 17 — 60k+ blob requests per run, zero ``DELETE`` committed, queue never drained, next worker re-ran the same failing loop. 14 kills observed in a single 24 h window. Fix: paginate both the ``SELECT`` and the ``fs.rm`` loop per storage, in batches of ``_GC_BATCH_SIZE = 500``, with an explicit ``gc.collect()`` between batches to reclaim the SDK's buffered state. The caller ``_gc_files`` still holds the ``SHARE`` lock on ``fs_file_gc`` / ``ir_attachment`` and performs the final commit, so consistency guarantees and transactional semantics are unchanged. Signed-off-by: TecnologiaIG <tecnologia@intensegroupgt.com>
…lected This method can then be used by other extensions to customize how files are deleted based on the specific backend
More code shared with base implementation to list files
s3fs uses aiobotocore as s3 client, which is asynchronous, but Odoo does not support async calls, therefore the call to delete objects from S3 was not working. The fsspec library provides a way to run async calls in a synchronous way. The fsspec.asyn.sync function is now used to run the async call to delete objects from S3, in a synchronous way, in order to make it work in Odoo.
Updated by "Update PO files to match POT (msgmerge)" hook in Weblate. Translation: storage-16.0/storage-16.0-fs_attachment_s3 Translate-URL: https://translation.odoo-community.org/projects/storage-16-0/storage-16-0-fs_attachment_s3/
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## 16.0 #374 +/- ##
==========================================
- Coverage 90.24% 90.21% -0.03%
==========================================
Files 144 146 +2
Lines 4560 4659 +99
Branches 406 413 +7
==========================================
+ Hits 4115 4203 +88
- Misses 314 322 +8
- Partials 131 134 +3 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
bt_gitbot