TEZ-4723: TestMapProcessor is flaky by abstractdog · Pull Request #504 · apache/tez

abstractdog · 2026-06-02T09:54:14Z

No description provided.

ayushtkn · 2026-06-02T11:08:26Z

+      output.collect(key, value);
+      if (counter.incrementAndGet() % N == 0) {
+        try {
+          Thread.sleep(1);


This will not fix it I believe, it will increase the probability of it passing I believe. Can we wait in the Test's monitoring thread for some X condition instead? Like maybe using GenericTestUtils.waitFor()

agreed, the current solution is just a sophisticated Thread.sleep :) let me improve it

tez-yetus · 2026-06-02T11:13:39Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	5m 38s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ master Compile Tests _
+1 💚	mvninstall	3m 12s		master passed
+1 💚	compile	2m 25s		master passed
+1 💚	checkstyle	0m 20s		master passed
+1 💚	javadoc	0m 22s		master passed
+0 🆗	spotbugs	0m 35s		tez-mapreduce in master has 97 extant spotbugs warnings.
			_ Patch Compile Tests _
+1 💚	mvninstall	2m 8s		the patch passed
+1 💚	codespell	0m 53s		No new issues.
+1 💚	compile	2m 24s		the patch passed
+1 💚	javac	2m 24s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 14s		the patch passed
+1 💚	javadoc	0m 18s		the patch passed
+1 💚	spotbugs	0m 39s		the patch passed
			_ Other Tests _
+1 💚	unit	58m 27s		root in the patch passed.
+1 💚	asflicense	0m 20s		The patch does not generate ASF License warnings.
		78m 56s

Subsystem	Report/Notes
Docker	ClientAPI=1.54 ServerAPI=1.54 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-504/1/artifact/out/Dockerfile
Optional Tests	dupname compile unit asflicense javac javadoc spotbugs checkstyle codespell detsecrets
uname	Linux 0518a0fadd0b 5.15.0-173-generic #183-Ubuntu SMP Fri Mar 6 13:29:34 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	tez-personality.sh
git revision	master / `330fdc8`
Default Java	Eclipse Adoptium-21.0.11+10-LTS
Test Results	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-504/1/testReport/
Max. process+thread count	1344 (vs. ulimit of 5500)
modules	C: tez-mapreduce U: tez-mapreduce
Console output	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-504/1/console
versions	git=2.43.0 maven=3.9.15 spotbugs=4.9.3 codespell=2.4.1
Powered by	Apache Yetus 0.15.1 https://yetus.apache.org

This message was automatically generated.

abstractdog · 2026-06-03T11:43:27Z

@ayushtkn : created LatchMapper, please let me know if it's fine

tez-yetus · 2026-06-03T12:57:16Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	2m 20s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ master Compile Tests _
+1 💚	mvninstall	3m 3s		master passed
+1 💚	compile	2m 27s		master passed
+1 💚	checkstyle	0m 23s		master passed
+1 💚	javadoc	0m 24s		master passed
+0 🆗	spotbugs	0m 35s		tez-mapreduce in master has 97 extant spotbugs warnings.
			_ Patch Compile Tests _
+1 💚	mvninstall	2m 11s		the patch passed
+1 💚	codespell	0m 48s		No new issues.
+1 💚	compile	2m 31s		the patch passed
+1 💚	javac	2m 31s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 18s		tez-mapreduce: The patch generated 0 new + 2 unchanged - 2 fixed = 2 total (was 4)
+1 💚	javadoc	0m 22s		the patch passed
+1 💚	spotbugs	0m 43s		the patch passed
			_ Other Tests _
+1 💚	unit	58m 48s		root in the patch passed.
+1 💚	asflicense	0m 21s		The patch does not generate ASF License warnings.
		76m 17s

Subsystem	Report/Notes
Docker	ClientAPI=1.54 ServerAPI=1.54 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-504/5/artifact/out/Dockerfile
Optional Tests	dupname compile unit asflicense javac javadoc spotbugs checkstyle codespell detsecrets
uname	Linux 15ea1b7fa1d1 5.15.0-173-generic #183-Ubuntu SMP Fri Mar 6 13:29:34 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	tez-personality.sh
git revision	master / `330fdc8`
Default Java	Eclipse Adoptium-21.0.11+10-LTS
Test Results	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-504/5/testReport/
Max. process+thread count	2127 (vs. ulimit of 5500)
modules	C: tez-mapreduce U: tez-mapreduce
Console output	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-504/5/console
versions	git=2.43.0 maven=3.9.15 spotbugs=4.9.3 codespell=2.4.1
Powered by	Apache Yetus 0.15.1 https://yetus.apache.org

This message was automatically generated.

ayushtkn

LGTM. Minor suggestions, feel free to commit once you handle it

ayushtkn · 2026-06-08T06:10:12Z

+    scheduler.shutdownNow();
    task.close();
    sharedExecutor.shutdownNow();


maybe this could be in finally block, if the above assertion fails, resources would get leaked

ack, fixed in 384c066
also made TezSharedExecutor AutoCloseable

tez-yetus · 2026-06-08T10:07:09Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	3m 2s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ master Compile Tests _
+0 🆗	mvndep	0m 29s		Maven dependency ordering for branch
+1 💚	mvninstall	2m 49s		master passed
+1 💚	compile	2m 30s		master passed
+1 💚	checkstyle	0m 38s		master passed
+1 💚	javadoc	0m 40s		master passed
+0 🆗	spotbugs	0m 33s		tez-common in master has 13 extant spotbugs warnings.
+0 🆗	spotbugs	0m 36s		tez-mapreduce in master has 97 extant spotbugs warnings.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 6s		Maven dependency ordering for patch
+1 💚	mvninstall	2m 13s		the patch passed
+1 💚	codespell	0m 49s		No new issues.
+1 💚	compile	2m 32s		the patch passed
+1 💚	javac	2m 32s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	0m 19s	/results-checkstyle-tez-mapreduce.txt	tez-mapreduce: The patch generated 1 new + 1 unchanged - 3 fixed = 2 total (was 4)
+1 💚	javadoc	0m 38s		the patch passed
+1 💚	spotbugs	1m 24s		the patch passed
			_ Other Tests _
+1 💚	unit	59m 25s		root in the patch passed.
+1 💚	asflicense	0m 40s		The patch does not generate ASF License warnings.
		80m 53s

Subsystem	Report/Notes
Docker	ClientAPI=1.54 ServerAPI=1.54 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-504/6/artifact/out/Dockerfile
Optional Tests	dupname compile unit asflicense javac javadoc spotbugs checkstyle codespell detsecrets
uname	Linux e4b05018e27e 5.15.0-181-generic #191-Ubuntu SMP Fri May 22 19:09:02 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	tez-personality.sh
git revision	master / `330fdc8`
Default Java	Eclipse Adoptium-21.0.11+10-LTS
Test Results	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-504/6/testReport/
Max. process+thread count	1392 (vs. ulimit of 5500)
modules	C: tez-common tez-mapreduce U: .
Console output	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-504/6/console
versions	git=2.43.0 maven=3.9.15 spotbugs=4.9.3 codespell=2.4.1
Powered by	Apache Yetus 0.15.1 https://yetus.apache.org

This message was automatically generated.

tez-yetus · 2026-06-08T11:27:11Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 8s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ master Compile Tests _
+0 🆗	mvndep	0m 29s		Maven dependency ordering for branch
+1 💚	mvninstall	2m 49s		master passed
+1 💚	compile	2m 27s		master passed
+1 💚	checkstyle	0m 38s		master passed
+1 💚	javadoc	0m 40s		master passed
+0 🆗	spotbugs	0m 35s		tez-common in master has 13 extant spotbugs warnings.
+0 🆗	spotbugs	0m 37s		tez-mapreduce in master has 97 extant spotbugs warnings.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 6s		Maven dependency ordering for patch
+1 💚	mvninstall	2m 10s		the patch passed
+1 💚	codespell	0m 48s		No new issues.
+1 💚	compile	2m 29s		the patch passed
+1 💚	javac	2m 29s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 17s		The patch passed checkstyle in tez-common
+1 💚	checkstyle	0m 18s		tez-mapreduce: The patch generated 0 new + 1 unchanged - 3 fixed = 1 total (was 4)
+1 💚	javadoc	0m 40s		the patch passed
+1 💚	spotbugs	1m 17s		the patch passed
			_ Other Tests _
-1 ❌	unit	58m 58s	/patch-unit-root.txt	root in the patch passed.
+1 💚	asflicense	0m 39s		The patch does not generate ASF License warnings.
		77m 22s

Reason	Tests
Failed junit tests	tez.test.TestAMRecoveryAggregationBroadcast

Subsystem	Report/Notes
Docker	ClientAPI=1.54 ServerAPI=1.54 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-504/7/artifact/out/Dockerfile
Optional Tests	dupname compile unit asflicense javac javadoc spotbugs checkstyle codespell detsecrets
uname	Linux b63fdbbbcd47 5.15.0-181-generic #191-Ubuntu SMP Fri May 22 19:09:02 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	tez-personality.sh
git revision	master / `330fdc8`
Default Java	Eclipse Adoptium-21.0.11+10-LTS
Test Results	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-504/7/testReport/
Max. process+thread count	2129 (vs. ulimit of 5500)
modules	C: tez-common tez-mapreduce U: .
Console output	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-504/7/console
versions	git=2.43.0 maven=3.9.15 spotbugs=4.9.3 codespell=2.4.1
Powered by	Apache Yetus 0.15.1 https://yetus.apache.org

This message was automatically generated.

tez-yetus · 2026-06-08T12:49:55Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 7s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ master Compile Tests _
+0 🆗	mvndep	0m 30s		Maven dependency ordering for branch
+1 💚	mvninstall	2m 45s		master passed
+1 💚	compile	2m 26s		master passed
+1 💚	checkstyle	0m 38s		master passed
+1 💚	javadoc	0m 40s		master passed
+0 🆗	spotbugs	0m 34s		tez-common in master has 13 extant spotbugs warnings.
+0 🆗	spotbugs	0m 35s		tez-mapreduce in master has 97 extant spotbugs warnings.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 6s		Maven dependency ordering for patch
+1 💚	mvninstall	2m 10s		the patch passed
+1 💚	codespell	0m 48s		No new issues.
+1 💚	compile	2m 28s		the patch passed
+1 💚	javac	2m 28s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 16s		The patch passed checkstyle in tez-common
+1 💚	checkstyle	0m 17s		tez-mapreduce: The patch generated 0 new + 1 unchanged - 3 fixed = 1 total (was 4)
+1 💚	javadoc	0m 39s		the patch passed
+1 💚	spotbugs	1m 22s		the patch passed
			_ Other Tests _
+1 💚	unit	60m 34s		root in the patch passed.
+1 💚	asflicense	0m 40s		The patch does not generate ASF License warnings.
		78m 55s

Subsystem	Report/Notes
Docker	ClientAPI=1.54 ServerAPI=1.54 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-504/8/artifact/out/Dockerfile
Optional Tests	dupname compile unit asflicense javac javadoc spotbugs checkstyle codespell detsecrets
uname	Linux 278c82cff98d 5.15.0-181-generic #191-Ubuntu SMP Fri May 22 19:09:02 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	tez-personality.sh
git revision	master / `330fdc8`
Default Java	Eclipse Adoptium-21.0.11+10-LTS
Test Results	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-504/8/testReport/
Max. process+thread count	1452 (vs. ulimit of 5500)
modules	C: tez-common tez-mapreduce U: .
Console output	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-504/8/console
versions	git=2.43.0 maven=3.9.15 spotbugs=4.9.3 codespell=2.4.1
Powered by	Apache Yetus 0.15.1 https://yetus.apache.org

This message was automatically generated.

TEZ-4723: TestMapProcessor is flaky

28759a7

abstractdog requested a review from ayushtkn June 2, 2026 09:54

ayushtkn reviewed Jun 2, 2026

View reviewed changes

abstractdog force-pushed the TEZ-4723 branch 2 times, most recently from 5a07af8 to 8521d5d Compare June 3, 2026 11:39

LatchMapper

2b9f69d

abstractdog force-pushed the TEZ-4723 branch from 8521d5d to 2b9f69d Compare June 3, 2026 11:40

ayushtkn approved these changes Jun 8, 2026

View reviewed changes

improved resource closing

384c066

checkstyle

3028e3a

abstractdog force-pushed the TEZ-4723 branch from 3d1d431 to 3028e3a Compare June 8, 2026 11:30

abstractdog merged commit cba5689 into apache:master Jun 8, 2026
4 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

TEZ-4723: TestMapProcessor is flaky#504

TEZ-4723: TestMapProcessor is flaky#504
abstractdog merged 4 commits into
apache:masterfrom
abstractdog:TEZ-4723

abstractdog commented Jun 2, 2026

Uh oh!

ayushtkn Jun 2, 2026

Uh oh!

abstractdog Jun 3, 2026

Uh oh!

tez-yetus commented Jun 2, 2026

Uh oh!

abstractdog commented Jun 3, 2026

Uh oh!

tez-yetus commented Jun 3, 2026

Uh oh!

ayushtkn left a comment

Uh oh!

ayushtkn Jun 8, 2026

Uh oh!

abstractdog Jun 8, 2026

Uh oh!

tez-yetus commented Jun 8, 2026

Uh oh!

tez-yetus commented Jun 8, 2026

Uh oh!

tez-yetus commented Jun 8, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

abstractdog commented Jun 2, 2026

Uh oh!

ayushtkn Jun 2, 2026

Choose a reason for hiding this comment

Uh oh!

abstractdog Jun 3, 2026

Choose a reason for hiding this comment

Uh oh!

tez-yetus commented Jun 2, 2026

Uh oh!

abstractdog commented Jun 3, 2026

Uh oh!

tez-yetus commented Jun 3, 2026

Uh oh!

ayushtkn left a comment

Choose a reason for hiding this comment

Uh oh!

ayushtkn Jun 8, 2026

Choose a reason for hiding this comment

Uh oh!

abstractdog Jun 8, 2026

Choose a reason for hiding this comment

Uh oh!

tez-yetus commented Jun 8, 2026

Uh oh!

tez-yetus commented Jun 8, 2026

Uh oh!

tez-yetus commented Jun 8, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants