Skip to content

Add basic POSIX mirror tree support#1017

Open
AlCutter wants to merge 1 commit into
transparency-dev:mainfrom
AlCutter:posix_mirror_tree_support
Open

Add basic POSIX mirror tree support#1017
AlCutter wants to merge 1 commit into
transparency-dev:mainfrom
AlCutter:posix_mirror_tree_support

Conversation

@AlCutter

@AlCutter AlCutter commented Jun 23, 2026

Copy link
Copy Markdown
Collaborator

This PR adds a first cut at fleshing out the POSIX storage support for MTC mirrors.

Towards #945

@AlCutter AlCutter requested a review from roger2hk June 23, 2026 11:20
@AlCutter AlCutter force-pushed the posix_mirror_tree_support branch 3 times, most recently from 49e43f1 to 692034b Compare June 24, 2026 11:01
@AlCutter AlCutter force-pushed the posix_mirror_tree_support branch from 692034b to c21f4be Compare June 24, 2026 11:56
@AlCutter AlCutter changed the title Add POSIX mirror tree support Add basic POSIX mirror tree support Jun 24, 2026
@AlCutter AlCutter marked this pull request as ready for review June 24, 2026 12:16
@AlCutter AlCutter requested a review from a team as a code owner June 24, 2026 12:16
Comment thread storage/posix/files.go
if err := unlock(); err != nil {
panic(err)
}
m.s.mu.Unlock()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same comment as the previous pull request. Immediately defer the mutex unlock after it is locked.

Comment thread storage/posix/files.go

lh := make([][]byte, 0, maxBundles)
n := 0
for ri := range layout.Range(from, to, sourceSize) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
for ri := range layout.Range(from, to, sourceSize) {
for ri := range layout.Range(from, to - from, sourceSize) {

// Range returns an iterator over a list of RangeInfo structs which describe the bundles/tiles
// necessary to cover the specified range of individual entries/hashes `[from, min(from+N, treeSize) )`.
//
// If from >= treeSize or N == 0, the returned iterator will yield no elements.
func Range(from, N, treeSize uint64) iter.Seq[RangeInfo] {

Comment thread storage/posix/files.go
return 0, nil, errors.New("unimplemented")
targetSize := bundleIdx * layout.EntryBundleWidth
for b := range bundles {
p := len(b.Entries)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the number of entries is 0, it will be treated as a full tile writing empty bundle data to the full tile file.

Comment thread storage/posix/files.go
// marshalTlogEntryBundle returns a tlog-tiles compatible serialization of the provided entrybundle.
func marshalTlogEntryBundle(b api.EntryBundle) ([]byte, error) {
// Max size we could possibly write out.
data := make([]byte, 0, len(b.Entries)*(2+1<<16))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that this preallocation would consume around 16 MiB for a full bundle.

Comment thread storage/posix/files.go
func (m *MirrorWriter) fetchLeafHashes(ctx context.Context, from, to, sourceSize uint64) ([][]byte, error) {
const maxBundles = 300

lh := make([][]byte, 0, maxBundles)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are 256 hashes in one bundle. Should this be maxBundles * layout.EntryBundleWidth instead of maxBundles?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants