File Manager: Add Compress & Extract archive support#2630
Conversation
- Add Compress option to folder and file context menus - Add Extract option to archive files (zip, tar, gz, etc.) - Create compress/extract dialog templates in Templates.php - Implement doAction cases 16 (compress) and 17 (extract) - Add auto-detection for archive format based on path (zstd for appdata, zip for shares) - Add updateArchiveName() helper to sync archive name with format selection - Add validation for compress/extract actions in Start button handler
…lementation - Control.php: Add format and archive_name parameters for compress action - file_manager: Implement case 16 (compress) with support for zip, tar, tar.gz, tar.zst - file_manager: Implement case 17 (extract) with auto-detection of archive type - Support for multiple archive formats: zip, tar, tar.gz, tar.bz2, tar.xz, tar.zst, 7z, rar - Add progress tracking via tee to status file - Add warning texts for compress/extract actions in Browse.page
…button, add debug logging
…cross extract calls, add debug logging
- Implement parse_zip_progress() to parse structured log format - Create zip_wrapper script for real-time zip progress parsing - Integrate zip_wrapper into compress case 16 - Parse DOT|timestamp entries to calculate speed and ETA - Display progress: Completed: X%, Archive: YMB, Speed: ZMB/s, ETA: H:MM:SS - Keep tar formats using simple status display for now
- Remove unused $format parameter from zip_wrapper - Fix shellcheck redirect warning (use : > instead of >) - Use pipe-safe format: FILE|size|filename (filename at end) - Add proper PHPDoc comment for parse_zip_progress - Update PHP parser to read filename from third field
- Change from pipeline to direct redirect - Avoids PID mismatch (pipeline returns wrong PID) - zip_wrapper logs to /var/tmp/zip.progress independently
…ip -> TEST-2.zip)
…add debug logging to archive name selection
…ps for speed calculation, fix regex to match end of line
…acter-by-character dot streaming
…may break PID tracking temporarily)
… needs AJAX or server-side implementation)
…for line end (dots come on same line)
…plate, integrated overwrite sub-tests, simplified sentinel detection
…checks file_manager: - add FM_DEBUG_FORCE_COPY_DELETE constant: forces rsync copy-delete path when /var/tmp/file.manager.force-copy-delete.debug exists (requires FM_DEBUG_MODE); used by test to exercise both move code paths - fix fm_delete_file(): only rename to .debug in debug mode, otherwise call delete_file() as before (was always renaming unconditionally) - apply FM_DEBUG_FORCE_COPY_DELETE override in move action (case 4/9) test-file-manager.sh: - add dir_fingerprint(): find-based metadata fingerprint (type, perms, mtime, size, symlink target); optional arg includes link count (%n) for hardlink verification - add fingerprint_match(): convenience wrapper for fingerprint comparison - add test_copy() / test_copy_file() / test_copy_folder(): tests copy actions 8 (file) and 3 (folder); checks source survives, destination exists, fingerprint matches - add test_move() / test_move_file() / test_move_folder(): tests move actions 9 (file) and 4 (folder) for both rsync-rename and copy-delete paths; pre-move fingerprint compared to post-move destination; hardlink count included in fingerprint only for rsync-rename folder moves (inodes preserved); last folder-move restores src_path - add dedicated source files for copy and move tests - add copy test calls (special name file + folder) - add move test calls (file rsync-rename, file copy-delete, folder rsync-rename, folder copy-delete)
rsync --backup-dir creates target directories with mkdir, which sets their mtime to the current time instead of preserving the original. Fix: record all dir mtimes (nanosecond precision via find -printf '%T@') into a bash associative array before the rsync-rename, then restore each dir mtime in the target afterwards using touch -d "@mtime". Also: copy empty dirs step now runs as step 1 (not affected by mtime issue since step 2 would overwrite them anyway). Also: add test case for special chars with backslash-asterisk in filename.
Backslash must be escaped: a filename like \* needs pattern \\* (escaped-backslash + escaped-asterisk) so rsync matches it literally. Without escaping \, the pattern \* would silently consume the backslash.
Also: check link count (%n) for ALL folder moves (action 4), not just rsync-rename - copy-delete must also preserve hardlinks.
…les and byte-for-byte diff - replace tmp file snapshots with variable capture: pre=$(find_metadata ...) - pipe diff output through head -20 | sed 's/^/ /' for readable output - use PIPESTATUS[0] to capture diff rc through the pipeline - replace all byte-for-byte diff -r checks with find_metadata comparisons - add zip-aware metadata fmt (omit %n since zip does not preserve hardlinks) - find_metadata accepts custom fmt as second arg (default includes %n) - fix run_action tail loop to also break on done:2 (used by search action)
file_manager:
- fix zip extract: restore symlink mtimes via ZipArchive +
touch -h after unzip (upstream bug since 2007: unzip never calls
lutimes() for symlinks; identify via S_IFLNK=0xA000 in external attrs)
- add fm_debug_log() helper; gate all 28 unconditional logger calls,
file_put_contents debug writes and temp file copies behind FM_DEBUG_MODE
test-file-manager.sh:
- rename JOB_TIMEOUT to TIMEOUT; add MULTI/SINGLE ENV overrides for
archive format arrays; move arrays from bottom to settings section
- add help block with usage, filter docs, ENV overrides, examples
- forward TIMEOUT/MULTI/SINGLE to remote host via SSH (printf %q)
- rewrite should_run(): accept multiple keywords; add word-boundary
matching for numeric IDs to prevent e.g. 1 matching 16
- expand all should_run() calls with full keyword sets
- find_metadata(): move %P first for stable sort before sed zeroing;
add extra predicate passthrough via ${@:3}
- test_extract(): single-file formats compare type+size only (no mtime);
zip: single-pass diff with DOS-time rounding via sed; fix symlink
mtime check now possible since file_manager restores it
- remove | head -20 from diff output in rename/copy/move checks
- test_chown(): action 11 test; verifies owner via stat -c '%U:%G' - test_search(): action 15 test; verifies results array in nchan done output; checks expected_match in results[].path if given - add fixtures: chown test file - add test sections: chown x2 (nobody:users/root:root), search x3 (exact name, wildcard *.txt, no-match pattern) - update help block with chown/search filter keywords
…hing normalize non-keyword chars (commas, semicolons, etc.) to spaces so both 'arc search mv' and 'arc,search,mv' work as filter input; prevents partial matches like 'arc' hitting 'search'
- count expected results with find -print0 | tr -cd '\0' | wc -c so filenames with newlines are counted correctly (1 per file) - compare against results | length: detects when file_manager search splits newline filenames into multiple results - check all result paths start with search_dir: catches split filenames that appear without directory prefix (location '---') - remove expected_match parameter: count + path checks are sufficient - fix: use [[ $done_line ]] instead of arithmetic expansion on JSON string
find outputs paths line-by-line; filenames containing newlines were
split across two lines, appearing as two separate search results.
fixes:
- find: add -print0 (null-terminated output)
- cat(): read with explode('\0') instead of file() to parse
null-terminated paths
- progress count: use substr_count(..., '\0') instead of wc -l
jq -r outputs paths with embedded newlines as multi-line text; grep then sees the second part as a separate string without the directory prefix, causing a false FAIL. Fix: validate paths entirely within jq using startswith() so embedded newlines are handled transparently. Use @JSON encoding for the diagnostic output on failure.
leaves /mnt/disk1/fm_test/dst intact when tests fail so created files can be inspected without having to comment out cleanup
- all source file setup is now in create_source_files(); idempotent - hardlink check: verify both files share the same inode; if not, delete hardlink2 and re-create it to restore the hardlink - after each folder-move test: silently move src back if needed, then call create_source_files so each test starts with clean src
FM_DEBUG_FORCE_COPY_DELETE is a PHP define() evaluated at process start - touching the force file while file_manager is already running had no effect. restart file_manager after creating the force file so the new process picks up the flag.
rename: rename, rn, 2, 7 (actions 2, 7) move: move, mv, 4, 9 (actions 4, 9)
…urce_files - separate filters: compress/cmp/arc/16 vs extract/arc/17 - src/files/ for all source files (was src/) - src/archives/ for pre-built archives used by extract tests - create_source_files builds all archives idempotently - extract tests no longer depend on compress tests having run first
- add build_archive(): native tools (tar/zip/gz/bz2/xz/zst/lz4), atomic creation via .tmp + rename, tar --sparse/--xattrs/--acls matching file_manager; tar.zst/tar.lz4 via pipe to avoid unrecognized flags - replace run_action archive building with build_archive() in create_source_files(); keeps simple [[ ! -f ]] guard - fix test_extract: default source_dir was $src_path, now $files_path - fix inode comparison: quote $inode2 (shellcheck SC2053) - normalize all mtimes to 202001010000 before/after archive builds so extract diffs are reproducible even after delete/rename tests recreate source files; touch -h to handle symlinks without following targets - add create_source_files after delete and rename test sections to repair source files before subsequent tests - fix move restore logic: $dst_path/src -> $dst_path/files - replace vague 'special name' labels with 'special chars' (files) and 'files dir' (folder tests on $files_path)
- remove parse_compress_progress() - fully superseded by parse_pv_progress() - remove PROGRESS_SCRIPT - defined but never used - remove TAR_RECORD_SIZE and TAR_CHECKPOINT - only used to define each other, never referenced elsewhere - remove two reset calls to parse_compress_progress() in case 16 and 17
|
@coderabbitai resume |
✅ Actions performedReviews resumed. |
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
.github/scripts/test-file-manager.sh (1)
497-497: 💤 Low valueAdd
localdeclaration forextracted_subdir.This variable is assigned without
local, causing it to leak into global scope. While this won't affect test correctness, it's inconsistent with other local variables in this function.📝 Proposed fix
- extracted_subdir=$(find "$dest" -mindepth 1 -maxdepth 1 -type d -printf '%f\n' -quit) + local extracted_subdir + extracted_subdir=$(find "$dest" -mindepth 1 -maxdepth 1 -type d -printf '%f\n' -quit)🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In @.github/scripts/test-file-manager.sh at line 497, The variable extracted_subdir is assigned without a local declaration and should be made local to avoid leaking into the global scope; update the function that contains the find assignment (where other local variables are declared) to change the assignment to declare extracted_subdir as local (i.e., add a local declaration for extracted_subdir before or on the same line as the find assignment) so it follows the same scoping pattern as the other local variables in that function.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@emhttp/plugins/dynamix/nchan/file_manager`:
- Around line 1178-1207: The docker-path detection (parsing
/boot/config/docker.cfg into $docker_subpaths and computing $is_docker_dest via
preg_match and str_starts_with against $dest) must be moved/duplicated so it
runs before the code paths that handle single-file archive extraction into an
existing directory; currently it's only inside the if (!is_dir($dest)) block
causing $is_docker_dest to be undefined. Extract the detection into a small
pre-check executed unconditionally (or initialize $is_docker_dest = false before
use), reuse the same logic that reads DOCKER_APP_CONFIG_PATH/DOCKER_IMAGE_FILE
with parse_ini_file and compares against $dest, and then use that computed
$is_docker_dest when calling exec("mkdir -pm0...") and exec("chown -R...") (and
apply the same fix to the similar block that handles compress/case 16). Ensure
you reference and reuse the existing helpers/variables ($dest, $docker_cfg,
$docker_subpaths, $is_docker_dest, preg_match, str_starts_with, quoted) so
behavior stays identical.
---
Nitpick comments:
In @.github/scripts/test-file-manager.sh:
- Line 497: The variable extracted_subdir is assigned without a local
declaration and should be made local to avoid leaking into the global scope;
update the function that contains the find assignment (where other local
variables are declared) to change the assignment to declare extracted_subdir as
local (i.e., add a local declaration for extracted_subdir before or on the same
line as the find assignment) so it follows the same scoping pattern as the other
local variables in that function.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI
Review profile: CHILL
Plan: Pro
Run ID: ccfae75d-fb20-4252-82d4-4f7302d86c35
📒 Files selected for processing (8)
.github/scripts/test-file-manager.shemhttp/plugins/dynamix/Browse.pageemhttp/plugins/dynamix/BrowseButton.pageemhttp/plugins/dynamix/include/Control.phpemhttp/plugins/dynamix/include/Templates.phpemhttp/plugins/dynamix/nchan/file_manageremhttp/plugins/dynamix/scripts/pvzipemhttp/plugins/dynamix/sheets/BrowseButton.css
🚧 Files skipped from review as they are similar to previous changes (4)
- emhttp/plugins/dynamix/sheets/BrowseButton.css
- emhttp/plugins/dynamix/include/Control.php
- emhttp/plugins/dynamix/include/Templates.php
- emhttp/plugins/dynamix/Browse.page
| // Create dest with proper permissions if it doesn't exist. | ||
| // Docker paths get root:root 755, all other shares get nobody:users 777. | ||
| if (!is_dir($dest)) { | ||
| $docker_cfg = @parse_ini_file('/boot/config/docker.cfg') ?: []; | ||
| $docker_subpaths = []; | ||
| foreach (['DOCKER_APP_CONFIG_PATH', 'DOCKER_IMAGE_FILE'] as $key) { | ||
| $val = rtrim(trim($docker_cfg[$key] ?? ''), '/'); | ||
| if ($val && preg_match('#^/mnt/[^/]+/(.+)$#', $val, $m)) { | ||
| $docker_subpaths[] = $m[1]; | ||
| } | ||
| } | ||
| if (empty($docker_subpaths)) $docker_subpaths = ['appdata', 'docker']; | ||
| $is_docker_dest = false; | ||
| if (preg_match('#^/mnt/[^/]+/(.+)$#', $dest, $m)) { | ||
| foreach ($docker_subpaths as $dp) { | ||
| if ($m[1] === $dp || str_starts_with($m[1], $dp.'/')) { | ||
| $is_docker_dest = true; | ||
| break; | ||
| } | ||
| } | ||
| } | ||
| $new_root = $dest; | ||
| $check = $dest; | ||
| while ($check !== '/' && !is_dir($check)) { | ||
| $new_root = $check; | ||
| $check = dirname($check); | ||
| } | ||
| exec("mkdir -pm0".($is_docker_dest ? '755' : '777')." ".quoted($dest)); | ||
| if (!$is_docker_dest) exec("chown -R nobody:users ".quoted($new_root)); | ||
| } |
There was a problem hiding this comment.
$is_docker_dest undefined when extracting single-file archives to an existing directory.
The docker-path detection at lines 1180-1207 only runs inside if (!is_dir($dest)). When extracting a single-file compressed archive (gz/bz2/xz/zst/lz4) to an existing directory, $is_docker_dest is never set, causing a PHP warning and incorrect permissions (always defaults to nobody:users 666 even for docker paths).
Compare to compress (case 16) at lines 921-930 which correctly computes $is_docker_path unconditionally before using it.
🐛 Suggested fix: Move docker detection before the mkdir block
// Pre-escape user-controlled paths for safe interpolation
$esc_archive = escapeshellarg($archive);
$esc_dest = escapeshellarg($dest);
fm_debug_log('extract start: archive=' . $archive . ', dest=' . $dest);
fm_debug_log('extract start: quoted_archive=' . quoted($archive) . ', quoted_dest=' . quoted($dest));
- // Create dest with proper permissions if it doesn't exist.
- // Docker paths get root:root 755, all other shares get nobody:users 777.
- if (!is_dir($dest)) {
- $docker_cfg = `@parse_ini_file`('/boot/config/docker.cfg') ?: [];
- $docker_subpaths = [];
- foreach (['DOCKER_APP_CONFIG_PATH', 'DOCKER_IMAGE_FILE'] as $key) {
- $val = rtrim(trim($docker_cfg[$key] ?? ''), '/');
- if ($val && preg_match('#^/mnt/[^/]+/(.+)$#', $val, $m)) {
- $docker_subpaths[] = $m[1];
- }
+ // Detect docker path for permission handling (needed even if dest exists)
+ $docker_cfg = `@parse_ini_file`('/boot/config/docker.cfg') ?: [];
+ $docker_subpaths = [];
+ foreach (['DOCKER_APP_CONFIG_PATH', 'DOCKER_IMAGE_FILE'] as $key) {
+ $val = rtrim(trim($docker_cfg[$key] ?? ''), '/');
+ if ($val && preg_match('#^/mnt/[^/]+/(.+)$#', $val, $m)) {
+ $docker_subpaths[] = $m[1];
}
- if (empty($docker_subpaths)) $docker_subpaths = ['appdata', 'docker'];
- $is_docker_dest = false;
- if (preg_match('#^/mnt/[^/]+/(.+)$#', $dest, $m)) {
- foreach ($docker_subpaths as $dp) {
- if ($m[1] === $dp || str_starts_with($m[1], $dp.'/')) {
- $is_docker_dest = true;
- break;
- }
+ }
+ if (empty($docker_subpaths)) $docker_subpaths = ['appdata', 'docker'];
+ $is_docker_dest = false;
+ if (preg_match('#^/mnt/[^/]+/(.+)$#', $dest, $m)) {
+ foreach ($docker_subpaths as $dp) {
+ if ($m[1] === $dp || str_starts_with($m[1], $dp.'/')) {
+ $is_docker_dest = true;
+ break;
}
}
+ }
+
+ // Create dest with proper permissions if it doesn't exist.
+ // Docker paths get root:root 755, all other shares get nobody:users 777.
+ if (!is_dir($dest)) {
$new_root = $dest;
$check = $dest;Also applies to: 1243-1264
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@emhttp/plugins/dynamix/nchan/file_manager` around lines 1178 - 1207, The
docker-path detection (parsing /boot/config/docker.cfg into $docker_subpaths and
computing $is_docker_dest via preg_match and str_starts_with against $dest) must
be moved/duplicated so it runs before the code paths that handle single-file
archive extraction into an existing directory; currently it's only inside the if
(!is_dir($dest)) block causing $is_docker_dest to be undefined. Extract the
detection into a small pre-check executed unconditionally (or initialize
$is_docker_dest = false before use), reuse the same logic that reads
DOCKER_APP_CONFIG_PATH/DOCKER_IMAGE_FILE with parse_ini_file and compares
against $dest, and then use that computed $is_docker_dest when calling
exec("mkdir -pm0...") and exec("chown -R...") (and apply the same fix to the
similar block that handles compress/case 16). Ensure you reference and reuse the
existing helpers/variables ($dest, $docker_cfg, $docker_subpaths,
$is_docker_dest, preg_match, str_starts_with, quoted) so behavior stays
identical.
Closes #2511
Summary
Adds compress and extract functionality to the Unraid File Manager.
What's new
Compress
.gz,.xz,.bz2,.zst,.lz4when exactly one file (no folder) is selected/boot/config/docker.cfgto detect the user's appdata/docker paths - selects TAR.ZST for those paths (ideal for container backups), ZIP for everything else.gzetc.) show "Running..." only — no byte-level progress (tool writes directly to output file)Extract
.zip,.tar,.tar.gz/.tgz,.tar.bz2/.tbz2,.tar.xz/.txz,.tar.zst,.tar.lz4,.gz,.bz2,.xz,.zst,.lz4-n/-o, TAR uses--skip-old-files --warning=no-existing-file; single-file formats abort with "File already exists" error when overwrite is off.gzetc.) show "Running..." only — no byte-level progressPermissions
Created archives and new target directories get the following permissions:
chown nobody:users,chmod 666- accessible via SMBchown root:root,chmod 600- prevents accidental exposure via container webservers777 nobody:usersfor shares,755 root:rootfor docker paths/mnt/<disk>/prefix so it correctly matches on any disk or pool (e.g./mnt/user/appdata,/mnt/disk1/appdata,/mnt/cache/appdataall match)Progress implementation
All progress parsing is handled by a dedicated
progressscript (scripts/progress) that is piped directly from the compress/extract command. It detects the running tool (zip/tar/unzip) by inspecting/proc, parses its stdout, and emits a structured log to the status file:The PHP worker reads this file incrementally every 250 ms and derives %, speed, and ETA from it. CHUNK lines carry sub-second timestamps (
$EPOCHREALTIME) for accurate speed calculation - integer-second timestamps would cause speed to snap to discrete multiples (e.g. 100 MB/1s, 100 MB/2s).ZIP compress:
zip -r -du -dd -ds 100memits a(size)suffix per file followed by one dot per 100 MB. The progress script parses stdout char-by-char: detects the(size)suffix to emit aFILEline, then counts dots to emitCHUNKlines.TOTALis pre-calculated withdu -sbfrom the source file list in the zip command.ZIP extract:
unziponly prints a filename after each file is fully written, so byte-level progress cannot come from stdout. Instead, the progress script reads the manifest fromunzip -lupfront, then polls each output file withstatevery 0.2s to emitCHUNKlines proportional to growing file size. The script reads unzip stdout char-by-char with a 100ms read timeout: on timeout (file is being written) it starts polling; on newline (small file finished before timeout) it emits a FILE line directly.TOTALis the sum of all file sizes from the manifest.TAR compress: uses
--checkpoint=N --checkpoint-action=exec="printf 'CHUNK\n'"(1 checkpoint = 100 MB). The progress script parses tar's verbose filename output forFILElines and theCHUNKsentinel lines for progress.TOTALis pre-calculated withdu -sbfrom the source file list.TAR extract: same checkpoint mechanism for
CHUNKlines.TOTALis obtained by runningtar -tvfwith a 5-second timeout before the main extract starts - instant for uncompressed archives, usually finishes within the timeout for compressed ones. If the timeout is hit, progress runs without a total (shows bytes transferred + speed only, switches to % automatically once TOTAL arrives).Single-file formats (
.gz,.bz2,.xz,.zst,.lz4): no stdout-based progress available - the decompressor writes directly to the output file. The UI shows "Running..." for the duration and "Done" on completion.Security
archive_nameis validated server-side (basename()+ empty/dot check) to prevent writing outside the target directory/mnt,/boot) server-side - same as all other file manager operationsescapeshellarg(); heredoc interpolation was replaced with pre-escaped variablesOther
filemanager.json) - same as copy/move.tmpfile reliablyuniqid-based name with collision-check loop, stored in the active job JSON so cancel can locate it even after a page reloadLANG=en_US.UTF-8for both compress and extract - Unraid runs with POSIX locale which mangles non-ASCII filenames (e.g. Chinese, Arabic) as#Uxxxxescape sequences without this-yto store symlinks as symlinks rather than skipping them (broken symlinks and cross-device symlinks were silently dropped without this)--xattrs --xattrs-include='*.*' --acls --sparse: preserves extended attributes (all namespaces:user.*,security.*,trusted.*), POSIX ACLs, and sparse-file holes (important for VM images, container overlay filesystems, and databases)--numeric-owner: restores exact uid/gid numbers instead of doing a name-based lookup (critical when restoring to a system where numeric IDs differ from the archive)Implementation status
^Jnewline in filenames handled; overwrite flag-n/-o)-y→ stored as symlinks (broken + cross-device symlinks preserved); TAR stores symlinks by default$, newlines - tested with real-world dataLANG=en_US.UTF-8on zip/unzip prevents#Uxxxxmangling--xattrs --acls; tested withuser.author,user.comment)-rincludes directory entries → empty dirs preserved; verified withdiff -rq(no differences)uniqid('', true)with a collision-check loop as an additional safeguardchown nobody:users,chmod 666- accessible via SMBchown root:root,chmod 600- prevents accidental exposure via container webservers777 nobody:usersfor shares,755 root:rootfor docker paths/mnt/<disk>/prefix so it matches correctly on any disk or pool (same logic as the format auto-detection in Browse.page).tmpfile/mnt/disk4/...paths could silently overwrite data on a different disk if the disk layout changed); relative paths (the default tar/zip behavior) are always safe and sufficient.gz,.xz,.bz2,.zst,.lz4) — only shown in the format dropdown when exactly one file (no folder) is selected; no progress % yet (compressor writes directly to output file, no stdout-based progress)pvor stat-polling (to be implemented)Summary by CodeRabbit
Release Notes
New Features
Improvements