Skip to content

HDDS-10489. Use CRC tables to speed up galoisFieldMultiply in CrcUtil.#10155

Merged
szetszwo merged 2 commits intoapache:masterfrom
szetszwo:HDDS-10489
Apr 29, 2026
Merged

HDDS-10489. Use CRC tables to speed up galoisFieldMultiply in CrcUtil.#10155
szetszwo merged 2 commits intoapache:masterfrom
szetszwo:HDDS-10489

Conversation

@szetszwo
Copy link
Copy Markdown
Contributor

@szetszwo szetszwo commented Apr 28, 2026

What changes were proposed in this pull request?

Similar HADOOP-19012, we may use CRC tables to speed up multiplication over a Galois field in CRC computations.

What is the link to the Apache JIRA

HDDS-10489

How was this patch tested?

Copied tests and the benchmark program from HADOOP-19012.

adoroszlai
adoroszlai reviewed Apr 29, 2026
View reviewed changes
Copy link
Copy Markdown
Contributor

@adoroszlai adoroszlai left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @szetszwo for the patch.

Comment thread hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/client/checksum/CrcComposer.java Outdated
private static final Logger LOG = LoggerFactory.getLogger(CrcComposer.class);

private final int crcPolynomial;
private final ToIntFunction<Long> mod;
Copy link
Copy Markdown
Contributor

@adoroszlai adoroszlai Apr 29, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can use LongToIntFunction.

Russole
Russole reviewed Apr 29, 2026
View reviewed changes
Copy link
Copy Markdown
Contributor

@Russole Russole left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @szetszwo for the patch. Overall, LGTM.
I agree with the existing comments. LongToIntFunction looks like a better fit here.

@szetszwo
Copy link
Copy Markdown
Contributor Author

szetszwo commented Apr 29, 2026

@adoroszlai , @Russole , thanks for reviewing this!

We can use LongToIntFunction.

It is a great idea! It makes the code run much faster:

  • before
Result) 10 x 100000000 computations:
galoisFieldMultiply: 18.146s (55108563.87 ops)
tableMultiply      : 8.660s (115473441.11 ops)
tableMultiply is 109.54% faster
  • after
Result) 10 x 100000000 computations:
galoisFieldMultiply: 18.607s (53743214.92 ops)
tableMultiply      : 6.553s (152601861.74 ops)
tableMultiply is 183.95% faster

@szetszwo
Copy link
Copy Markdown
Contributor Author

szetszwo commented Apr 29, 2026

The failure of TestStreamRead is no related to this, StreamRead and Ozone in general. It is a bug in Ratis; filed RATIS-2510

@szetszwo szetszwo merged commit b508364 into apache:master Apr 29, 2026
90 of 91 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Russole Russole Russole left review comments

@adoroszlai adoroszlai adoroszlai approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@szetszwo @adoroszlai @Russole