TEZ-4688 TEZ-4648: Tez upgrade to Hadoop 3.5.0 and jersey 2.x#474
Open
maheshrajus wants to merge 9 commits into
Open
TEZ-4688 TEZ-4648: Tez upgrade to Hadoop 3.5.0 and jersey 2.x#474maheshrajus wants to merge 9 commits into
maheshrajus wants to merge 9 commits into
Conversation
Contributor
|
FYI, |
This comment was marked as outdated.
This comment was marked as outdated.
maheshrajus
changed the title
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
maheshrajus
changed the title
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
Contributor
Author
|
@abstractdog @ayushtkn Can you please review and provide comments if any. Thanks ! |
Contributor
|
@maheshrajus , can you please rebase this PR and resolve merge conflicts? |
Contributor
Author
|
@Aggarwal-Raghav Hi, i rebased and pushed the commit just now. Could you please take a look at this? Thanks ! |
Contributor
|
The explicit dependency of |
This comment was marked as outdated.
This comment was marked as outdated.
Contributor
Author
@Aggarwal-Raghav Better this we will handle separately. thanks ! |
|
💔 -1 overall
This message was automatically generated. |
Contributor
Author
|
@abstractdog @ayushtkn @Aggarwal-Raghav |
abstractdog
requested review from
abstractdog
and removed request for
Aggarwal-Raghav
Contributor
Author
|
@Aggarwal-Raghav Can you please check and confirm your opinion on above points which we discussed ? Thanks ! |
Contributor
|
@maheshrajus , I'll check this by EOD or tomorrow. |
|
🎊 +1 overall
This message was automatically generated. |
Contributor
Author
|
@Aggarwal-Raghav Thanks for your inputs on this PR. Let me check and confirm the above discussed points. |
|
💔 -1 overall
This message was automatically generated. |
Contributor
Author
|
@Aggarwal-Raghav Fixed your review comments. Can you please check and confirm at your free time. Thanks ! |
Contributor
|
the latest job (https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-474/9/console) failed with: which is strange as my job is currently successfully passing this point (https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-504/6/) @Aggarwal-Raghav: you had similar failure recently, can you recall if it was a flaky thing or if it was resolved with some magic, rebase, etc.? |
Contributor
my last run also passed (https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-507/1/console), @maheshrajus , please try to force push again. |
Contributor
Author
|
@abstractdog @Aggarwal-Raghav After force push the CI pipeline is running fine. Thanks ! |
|
💔 -1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
Contributor
Author
|
@abstractdog @Aggarwal-Raghav I addressed all your review comments. Could you please take a look and approve it when you have a moment? |
abstractdog
requested changes
Jun 9, 2026
abstractdog
left a comment
abstractdog
left a comment
Contributor
There was a problem hiding this comment.
thanks @maheshrajus for the updated PR, left some comments, this is getting close!
| public HttpURLConnection getConnection(URL url) throws IOException { | ||
| try { | ||
| AuthenticatedURL authenticatedURL= ReflectionUtils.createClazzInstance( | ||
| Object authenticatedURL = ReflectionUtils.createClazzInstance( |
Contributor
There was a problem hiding this comment.
I believe this should work with AuthenticatedURL
| connConfigurator | ||
| }); | ||
| return ReflectionUtils.invokeMethod(authenticatedURL, | ||
| return (HttpURLConnection) ReflectionUtils.invokeMethod(authenticatedURL, |
Contributor
There was a problem hiding this comment.
cast is not needed here
| String inputUrl = "http://host:8080/path"; | ||
| String expectedUrl = inputUrl + "?user.name=" + UserGroupInformation.getCurrentUser().getShortUserName(); | ||
| HttpURLConnection httpURLConnection = connectionFactory.getHttpURLConnection(new URL(inputUrl)); | ||
| HttpURLConnection httpURLConnection = connectionFactory.getConnection(URI.create(inputUrl).toURL()); |
Contributor
There was a problem hiding this comment.
this is not related to the hadoop upgrade I think, but instead new URL becoming deprecated from Java20
if so, please create a separate ticket for handling the same in the whole tez codebase:
grep -iRH "new URL" --include="*.java"
Comment on lines
+258
to
+259
| userName = subEntity.getString(Constants.ENTITY_ID); | ||
| break; |
Contributor
There was a problem hiding this comment.
this seems unnecessary, we indent 2 spaces
it's strange that checkstyle didn't warn about this, can you please create a dummy PR to validate the same and a Jira if needed?
| ? jsonObject.getJSONObject("domain") | ||
| : jsonObject; | ||
| } else if (clazz == TimelineEntity.class) { | ||
| // TimelineEntity is annotated @XmlRootElement(name="entity"), Jersey 2/Jackson wrap it |
| </dependencies> | ||
| </dependencyManagement> | ||
|
|
||
| <dependencies> |
Contributor
There was a problem hiding this comment.
why <dependencies> appeared here? afaik, we only handle dependencyManagement in the root pom.xml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this PR?
This PR upgrades Tez to support:
As part of the upgrade, several ATS Timeline API response format differences were observed between older Hadoop versions and Hadoop 3.5.0. This PR adds compatibility handling for those schema variations.
Main updates include:
Handled ATS compatibility differences including:
otherinfo↔otherInfoentity↔entityIdentitytype↔entityTypedomain/id↔domainIdAdditional fixes:
How was this patch tested?
jersey upgrade changes taken from PR : #429 by @abstractdog as they need for hadoop 3.5.0 upgrade.