SF-3777 Retrieve confidence data and calculate book level scores

[pmachapman]

This PR adds a card to the Serval Builds tab with Quality Estimation, allowing downloading of book and chapter data in a format very similar to SILNLP:

This change is 

[codecov]

Codecov Report

❌ Patch coverage is 80.65693% with 53 lines in your changes missing coverage. Please review.
✅ Project coverage is 80.87%. Comparing base (badab4f) to head (36a38c2).
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
...administration/build-confidences-export.service.ts	3.33%	29 Missing ⚠️
...p/serval-administration/serval-builds.component.ts	23.52%	12 Missing and 1 partial ⚠️
.../build-confidences/display-confidence.component.ts	70.00%	1 Missing and 2 partials ⚠️
...c/app/serval-administration/base-export-service.ts	71.42%	0 Missing and 2 partials ⚠️
...aAccess/SFDataAccessServiceCollectionExtensions.cs	33.33%	2 Missing ⚠️
src/SIL.XForge.Scripture/Models/VerseRefData.cs	81.81%	1 Missing and 1 partial ⚠️
...orge.Scripture/Controllers/MachineApiController.cs	95.83%	0 Missing and 1 partial ⚠️
...SIL.XForge.Scripture/Services/MachineApiService.cs	98.83%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #3876      +/-   ##
==========================================
- Coverage   80.88%   80.87%   -0.02%     
==========================================
  Files         634      639       +5     
  Lines       41017    41239     +222     
  Branches     6654     6686      +32     
==========================================
+ Hits        33177    33352     +175     
+ Misses       6802     6793       -9     
- Partials     1038     1094      +56     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

[github-actions]

📸 Screenshot diff deployed! (4 changes)

View the visual diff at: https://pr-3876--sf-screenshot-diffs.netlify.app

[RaymondLuong3]

It is exciting to be one step closer to introducing book and chapter level quality estimates to users!

@RaymondLuong3 reviewed 33 files and all commit messages, and made 7 comments.
Reviewable status: all files reviewed, 6 unresolved discussions (waiting on pmachapman).

---

src/SIL.XForge.Scripture/ClientApp/src/app/serval-administration/serval-builds.component.ts line 269 at r3 (raw file):

    } else {
      this.buildConfidencesExportService.exportCsv(rows, this.dateRange$.value);
    }

I think this should be exportRsv.

Code quote:

    } else {
      this.buildConfidencesExportService.exportCsv(rows, this.dateRange$.value);
    }

---

src/SIL.XForge.Scripture/ClientApp/src/app/serval-administration/serval-builds.component.ts line 278 at r3 (raw file):

    } else {
      this.buildConfidencesExportService.exportCsv(rows, this.dateRange$.value);
    }

I think this should be exportTsv

Code quote:

    } else {
      this.buildConfidencesExportService.exportCsv(rows, this.dateRange$.value);
    }

---

src/SIL.XForge.Scripture/ClientApp/src/app/translate/draft-generation/build-confidences/display-confidence.component.html line 11 at r2 (raw file):

    <div class="confidence red"><mat-icon>warning</mat-icon> Likely poor quality</div>
  }
}

Looks like this is likely to be shown to the user and needs to be localized. Oh yes, I see below you mention that this is only visible from the serval admin tab right now.

Code quote:

@switch (usabilityLabel) {
  @case ("Green") {
    <div class="confidence green"><mat-icon>check</mat-icon> Likely good quality</div>
  }
  @case ("Yellow") {
    <div class="confidence yellow"><mat-icon class="filled">circle</mat-icon> Likely moderate quality</div>
  }
  @case ("Red") {
    <div class="confidence red"><mat-icon>warning</mat-icon> Likely poor quality</div>
  }
}

---

src/SIL.XForge.Scripture/Services/MachineApiService.cs line 3100 at r3 (raw file):

                List<ScriptureSegmentUsability> _, // We do not require segment level confidence values
                List<ScriptureChapterUsability> usabilityChapters,
                List<ScriptureBookUsability> usabilityBooks

I know the name is already in Machine so I understand why the variables are named this way, but I have a hard time understanding the meaning of usabilityChapters and usabilityBooks. However, I'm not sure I have a good suggestion

Code quote:

                List<ScriptureSegmentUsability> _, // We do not require segment level confidence values
                List<ScriptureChapterUsability> usabilityChapters,
                List<ScriptureBookUsability> usabilityBooks

---

src/SIL.XForge.Scripture/Services/PreTranslationService.cs line 68 at r3 (raw file):

            // Get the references - old builds will only have Refs specified,
            // newer builds will have TargetRefs with Refs containing the same values
            List<string> references = [.. preTranslation.TargetRefs];

It seems strange that old builds use Refs, but this code is changing it from using TargetRefs to TargetRefs or Refs. This seems to me that we are introducing Refs as a new fallback. What am I missing?

Code quote:

            // Get the references - old builds will only have Refs specified,
            // newer builds will have TargetRefs with Refs containing the same values
            List<string> references = [.. preTranslation.TargetRefs];

---

test/SIL.XForge.Scripture.Tests/Services/MachineApiServiceTests.cs line 2335 at r3 (raw file):

        );

        using (Assert.EnterMultipleScope())

This is neat. I haven't encountered this before, but nice to see a way to identify multiple failures within a block.

Code quote:

       using (Assert.EnterMultipleScope())

[pmachapman]

@pmachapman made 5 comments and resolved 1 discussion.
Reviewable status: all files reviewed, 5 unresolved discussions (waiting on RaymondLuong3).

---

src/SIL.XForge.Scripture/ClientApp/src/app/serval-administration/serval-builds.component.ts line 269 at r3 (raw file):

Previously, RaymondLuong3 (Raymond Luong) wrote…

I think this should be exportRsv.

Done. Thank you!

---

src/SIL.XForge.Scripture/ClientApp/src/app/serval-administration/serval-builds.component.ts line 278 at r3 (raw file):

Previously, RaymondLuong3 (Raymond Luong) wrote…

I think this should be exportTsv

Done. Thank you!

---

src/SIL.XForge.Scripture/ClientApp/src/app/translate/draft-generation/build-confidences/display-confidence.component.html line 11 at r2 (raw file):

Previously, RaymondLuong3 (Raymond Luong) wrote…

Looks like this is likely to be shown to the user and needs to be localized. Oh yes, I see below you mention that this is only visible from the serval admin tab right now.

This will be localized when the wording is finalized. I thought it might be a bit of a waste of effort to localize it at this early stage.

---

src/SIL.XForge.Scripture/Services/MachineApiService.cs line 3100 at r3 (raw file):

Previously, RaymondLuong3 (Raymond Luong) wrote…

I know the name is already in Machine so I understand why the variables are named this way, but I have a hard time understanding the meaning of usabilityChapters and usabilityBooks. However, I'm not sure I have a good suggestion

I agree - the names are from the names used by machine, which are consistent with silnlp. I'm not sure what else to call them, without breaking that consistency.

---

src/SIL.XForge.Scripture/Services/PreTranslationService.cs line 68 at r3 (raw file):

Previously, RaymondLuong3 (Raymond Luong) wrote…

It seems strange that old builds use Refs, but this code is changing it from using TargetRefs to TargetRefs or Refs. This seems to me that we are introducing Refs as a new fallback. What am I missing?

Refs are used by the old corpora format, which does not have TargetRefs specified (this code fixes a bug I noticed). The new corpora format uses TargetRefs/SourceRefs. SF should use SourceRefs, but these are currently blank in Serval due to a bug which is being fixed in the next release. When that is done, I will need to implement support for SourceRefs for drafts made with that version of Serval onwards.

[RaymondLuong3]

@RaymondLuong3 reviewed 4 files and all commit messages, made 1 comment, and resolved 5 discussions.
Reviewable status: complete! all files reviewed, all discussions resolved (waiting on pmachapman).

---

src/SIL.XForge.Scripture/Services/PreTranslationService.cs line 68 at r3 (raw file):

Previously, pmachapman (Peter Chapman) wrote…

Refs are used by the old corpora format, which does not have TargetRefs specified (this code fixes a bug I noticed). The new corpora format uses TargetRefs/SourceRefs. SF should use SourceRefs, but these are currently blank in Serval due to a bug which is being fixed in the next release. When that is done, I will need to implement support for SourceRefs for drafts made with that version of Serval onwards.

Ok, got it. Thanks for the extra details.

[Nateowami]

@Nateowami resolved 4 discussions.
Reviewable status: 9 of 50 files reviewed, all discussions resolved (waiting on RaymondLuong3).

Requested changes mostly addressed

[Nateowami]

@Nateowami made 1 comment.
Reviewable status: 9 of 50 files reviewed, 1 unresolved discussion (waiting on pmachapman and RaymondLuong3).

---

src/SIL.XForge.Scripture/ClientApp/src/app/serval-administration/build-confidences-export.service.ts line 82 at r6 (raw file):

  private createRsvRows(rows: (BookConfidence | ChapterConfidence)[]): string[][] {
    if (this.isChapterConfidenceArray(rows)) {
      const headers: string[] = ['Book', 'Chapter', 'Projected chrF3', 'Usability', 'Label'];

Is it intentional that confidence isn't included here?

[pmachapman]

@pmachapman made 5 comments.
Reviewable status: 9 of 50 files reviewed, 1 unresolved discussion (waiting on Nateowami and RaymondLuong3).

---

src/SIL.XForge.Scripture/ClientApp/src/app/serval-administration/build-confidences-export.service.ts line 82 at r5 (raw file):

Previously, Nateowami wrote…

Sorry about PreTranslationService.cs getting tacked on to my comment. I accidentally pasted it. And I can't find where Reviewable's "preview publish" option went, so even though I tried to preview what I was publishing, I couldn't view it until after I hit publish. Do you know where it went?"

Done. No, sorry.

---

src/SIL.XForge.Scripture/ClientApp/src/app/serval-administration/build-confidences-export.service.ts line 123 at r5 (raw file):

Previously, Nateowami wrote…

Either I can't understand the intent, or this is a bug. It seems filenamePrefix is thrown away.

Did you mean:

filenamePrefix ?? (this.isChapterConfidenceArray(rows) ? 'usability_chapters' : 'usability_books')

Done. I got the brackets messed up...

---

src/SIL.XForge.Scripture/ClientApp/src/app/serval-administration/build-confidences-export.service.ts line 82 at r6 (raw file):

Previously, Nateowami wrote…

Is it intentional that confidence isn't included here?

Yes - I matched the columns in silnlp (see https://github.com/sillsdev/silnlp/blob/master/silnlp/nmt/quality_estimation.py#L393)

---

src/SIL.XForge.Scripture/ClientApp/src/app/translate/draft-generation/build-confidences/display-confidence.component.html line 3 at r5 (raw file):

Previously, Nateowami wrote…

If you put:

UsabilityLabel = UsabilityLabel

in the component, then in the template you can do

@case (UsabilityLabel.Green)

Otherwise it kind of defeats the purpose of the enum.

I think I am missing something obvious, but I get a runtime error Error: Cannot read properties of undefined (reading 'Green' when I do that. I notice all of our other enums have strings hard-coded.

---

src/SIL.XForge.Scripture/Controllers/MachineApiController.cs line 348 at r5 (raw file):

Previously, Nateowami wrote…

The docs for NoContent() say:

A StatusCodeResult that when executed will produce a 204 No Content response.

The doc for this method says it will return 404 confidence data is not available.

Done. Fixed documentation.

[RaymondLuong3]

Thanks for your work on this. And sorry for ripping out the PretranslationService on your the first time.

@RaymondLuong3 reviewed 41 files and all commit messages, and made 3 comments.
Reviewable status: all files reviewed, 3 unresolved discussions (waiting on Nateowami and pmachapman).

---

src/SIL.XForge.Scripture/Services/PreTranslationService.cs line 78 at r6 (raw file):

            }

            // Add the confidence value. HHowever, if the confidence value already exists, see if this pre-translation

Typo

Code quote:

Add the confidence value. HHowever, 

---

src/SIL.XForge.Scripture/Services/SFProjectService.cs line 1265 at r6 (raw file):

                    {
                        DateUpdated = DateTime.UtcNow,
                    }

Interesting. I have not come across this syntax. I looked it up.

Code quote:

                    qualityEstimationConfig with
                    {
                        DateUpdated = DateTime.UtcNow,
                    }

[Nateowami]

@Nateowami made 2 comments.
Reviewable status: all files reviewed, 3 unresolved discussions (waiting on pmachapman).

---

src/SIL.XForge.Scripture/ClientApp/src/app/serval-administration/build-confidences-export.service.ts line 82 at r6 (raw file):

Previously, pmachapman (Peter Chapman) wrote…

Yes - I matched the columns in silnlp (see https://github.com/sillsdev/silnlp/blob/master/silnlp/nmt/quality_estimation.py#L393)

Maybe I'm looking at it wrong, but getSpreadsheetRows seems to include Confidence

---

src/SIL.XForge.Scripture/ClientApp/src/app/translate/draft-generation/build-confidences/display-confidence.component.html line 3 at r5 (raw file):

Previously, pmachapman (Peter Chapman) wrote…

I think I am missing something obvious, but I get a runtime error Error: Cannot read properties of undefined (reading 'Green' when I do that. I notice all of our other enums have strings hard-coded.

I wonder if you forgot to save the component file where you put UsabilityLabel = UsabilityLabel on the class, because it's working fine for me. There's no reason it should be undefined.

Here's my git diff:

diff --git a/src/SIL.XForge.Scripture/ClientApp/src/app/translate/draft-generation/build-confidences/display-confidence.component.html b/src/SIL.XForge.Scripture/ClientApp/src/app/translate/draft-generation/build-confidences/display-confidence.component.html
index 45752297c..24e55ebd8 100644
--- a/src/SIL.XForge.Scripture/ClientApp/src/app/translate/draft-generation/build-confidences/display-confidence.component.html
+++ b/src/SIL.XForge.Scripture/ClientApp/src/app/translate/draft-generation/build-confidences/display-confidence.component.html
@@ -1,13 +1,13 @@
 @switch (usabilityLabel) {
-  @case ("Green") {
+  @case (UsabilityLabel.Green) {
     <div class="confidence green">
       <mat-icon [matTooltip]="usabilityTooltip">check_circle</mat-icon>{{ usabilityText }}
     </div>
   }
-  @case ("Yellow") {
+  @case (UsabilityLabel.Yellow) {
     <div class="confidence yellow"><mat-icon [matTooltip]="usabilityTooltip">circle</mat-icon>{{ usabilityText }}</div>
   }
-  @case ("Red") {
+  @case (UsabilityLabel.Red) {
     <div class="confidence red"><mat-icon [matTooltip]="usabilityTooltip">error</mat-icon>{{ usabilityText }}</div>
   }
   @case (undefined) {
diff --git a/src/SIL.XForge.Scripture/ClientApp/src/app/translate/draft-generation/build-confidences/display-confidence.component.ts b/src/SIL.XForge.Scripture/ClientApp/src/app/translate/draft-generation/build-confidences/display-confidence.component.ts
index a69028f6d..ef9a0536b 100644
--- a/src/SIL.XForge.Scripture/ClientApp/src/app/translate/draft-generation/build-confidences/display-confidence.component.ts
+++ b/src/SIL.XForge.Scripture/ClientApp/src/app/translate/draft-generation/build-confidences/display-confidence.component.ts
@@ -18,6 +18,8 @@ export class DisplayConfidenceComponent {
   @Input() confidence: Confidence | undefined;
   @Input() showText: boolean | undefined;
 
+  UsabilityLabel = UsabilityLabel;
+
   get usabilityLabel(): UsabilityLabel | undefined {
     return this.confidence?.label;

This is super minor and doesn't have to be changed, but I'm putting this here because it should work.

[pmachapman]

And sorry for ripping out the PretranslationService on your the first time.

Not a problem - the new API was always just around the corner!

@pmachapman made 4 comments and resolved 3 discussions.
Reviewable status: complete! all files reviewed, all discussions resolved (waiting on pmachapman).

---

src/SIL.XForge.Scripture/ClientApp/src/app/serval-administration/build-confidences-export.service.ts line 82 at r6 (raw file):

Previously, Nateowami wrote…

Maybe I'm looking at it wrong, but getSpreadsheetRows seems to include Confidence

Done. Gotcha - sorry I get it now. Added!

---

src/SIL.XForge.Scripture/ClientApp/src/app/translate/draft-generation/build-confidences/display-confidence.component.html line 3 at r5 (raw file):

Previously, Nateowami wrote…

I wonder if you forgot to save the component file where you put UsabilityLabel = UsabilityLabel on the class, because it's working fine for me. There's no reason it should be undefined.

Here's my git diff:

diff --git a/src/SIL.XForge.Scripture/ClientApp/src/app/translate/draft-generation/build-confidences/display-confidence.component.html b/src/SIL.XForge.Scripture/ClientApp/src/app/translate/draft-generation/build-confidences/display-confidence.component.html
index 45752297c..24e55ebd8 100644
--- a/src/SIL.XForge.Scripture/ClientApp/src/app/translate/draft-generation/build-confidences/display-confidence.component.html
+++ b/src/SIL.XForge.Scripture/ClientApp/src/app/translate/draft-generation/build-confidences/display-confidence.component.html
@@ -1,13 +1,13 @@
 @switch (usabilityLabel) {
-  @case ("Green") {
+  @case (UsabilityLabel.Green) {
     <div class="confidence green">
       <mat-icon [matTooltip]="usabilityTooltip">check_circle</mat-icon>{{ usabilityText }}
     </div>
   }
-  @case ("Yellow") {
+  @case (UsabilityLabel.Yellow) {
     <div class="confidence yellow"><mat-icon [matTooltip]="usabilityTooltip">circle</mat-icon>{{ usabilityText }}</div>
   }
-  @case ("Red") {
+  @case (UsabilityLabel.Red) {
     <div class="confidence red"><mat-icon [matTooltip]="usabilityTooltip">error</mat-icon>{{ usabilityText }}</div>
   }
   @case (undefined) {
diff --git a/src/SIL.XForge.Scripture/ClientApp/src/app/translate/draft-generation/build-confidences/display-confidence.component.ts b/src/SIL.XForge.Scripture/ClientApp/src/app/translate/draft-generation/build-confidences/display-confidence.component.ts
index a69028f6d..ef9a0536b 100644
--- a/src/SIL.XForge.Scripture/ClientApp/src/app/translate/draft-generation/build-confidences/display-confidence.component.ts
+++ b/src/SIL.XForge.Scripture/ClientApp/src/app/translate/draft-generation/build-confidences/display-confidence.component.ts
@@ -18,6 +18,8 @@ export class DisplayConfidenceComponent {
   @Input() confidence: Confidence | undefined;
   @Input() showText: boolean | undefined;
 
+  UsabilityLabel = UsabilityLabel;
+
   get usabilityLabel(): UsabilityLabel | undefined {
     return this.confidence?.label;

This is super minor and doesn't have to be changed, but I'm putting this here because it should work.

Done. Clearly user error on my part. Thank you!

---

src/SIL.XForge.Scripture/Services/PreTranslationService.cs line 78 at r6 (raw file):

Previously, RaymondLuong3 (Raymond Luong) wrote…

Typo

Done.