Inject the template's download state in the secondary storage selectors

[winterhazel]

Description

This PR adds the following preset variables to the secondary storage selectors of TEMPLATE resources:

Attribute	Description
template.isPublic	Whether the template is marked as public
template.downloadDetails	List containing information about the template's state in each secondary storage that it is associated to
template.downloadDetails[i].dataStoreId	UUID of the secondary storage
template.downloadDetails[i].downloadState	Download state of template in secondary storage

Using them, it is possible to define heuristics to avoid the redownload/copy of public templates, for instance:

for (i = 0; i < template.downloadDetails.length; i++) {
  details = template.downloadDetails[i];
  if (template.isPublic && ['NOT_DOWNLOADED', 'DOWNLOAD_IN_PROGRESS', 'DOWNLOADED'].indexOf(details.downloadState) !== -1) {
    details.dataStoreId;
    break;
  }
}

The selector above will direct the template to a secondary storage that the template is already present when it is marked as public and downloaded/downloading. Hence, it will not be duplicated to other secondary storages.

Types of changes

• Breaking change (fix or feature that would cause existing functionality to change)
• New feature (non-breaking change which adds functionality)
• Bug fix (non-breaking change which fixes an issue)
• Enhancement (improves an existing feature and functionality)
• Cleanup (Code refactoring and cleanup, that may add test cases)
• build/CI
• test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

• Major
• Minor

Screenshots (if appropriate):

How Has This Been Tested?

In an environment with a single secondary storage, sec-nfs-1:

1. I created a secondary storage selector for templates with the heuristic rule available above.
2. I registered some public templates and waited until they finished downloading to sec-nfs-1.
3. I added another secondary storage, sec-nfs-2, and verified that the existing public templates were not copied/downloaded to sec-nfs-2 because the secondary storage selector was directing them to sec-nfs-1.

[winterhazel]

@blueorangutan package

[blueorangutan]

@winterhazel a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

[codecov]

Codecov Report

❌ Patch coverage is 35.00000% with 26 lines in your changes missing coverage. Please review.
✅ Project coverage is 16.26%. Comparing base (8eb162c) to head (e0d9f3b).
⚠️ Report is 7 commits behind head on 4.20.

Files with missing lines	Patch %	Lines
...ge/heuristics/presetvariables/DownloadDetails.java	0.00%	13 Missing ⚠️
...dstack/storage/heuristics/HeuristicRuleHelper.java	53.33%	6 Missing and 1 partial ⚠️
...k/storage/heuristics/presetvariables/Template.java	50.00%	6 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff            @@
##               4.20   #13025   +/-   ##
=========================================
  Coverage     16.26%   16.26%           
- Complexity    13433    13437    +4     
=========================================
  Files          5665     5666    +1     
  Lines        500530   500610   +80     
  Branches      60787    60793    +6     
=========================================
+ Hits          81411    81432   +21     
- Misses       410027   410068   +41     
- Partials       9092     9110   +18     

Flag	Coverage Δ
uitests	4.15% <ø> (ø)
unittests	17.12% <35.00%> (+<0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[blueorangutan]

Packaging result [SF]: ✖️ el8 ✖️ el9 ✖️ debian ✖️ suse15. SL-JID 17492

[winterhazel]

@blueorangutan package

[blueorangutan]

@winterhazel a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

[blueorangutan]

Packaging result [SF]: ✖️ el8 ✖️ el9 ✔️ debian ✖️ suse15. SL-JID 17947