HADOOP-18073. Upgrade S3A in 3.3 branch to AWS SDK V2.(#5707) #5858

steveloughran · 2023-07-19T17:38:52Z

Description of PR

rebase of feature-HADOOP-18073-s3a-sdk-upgrade-3.3-rebase onto branch-3.3

How was this patch tested?

For code changes:

Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

mukund-thakur · 2023-07-19T17:56:53Z

Shouldn't there be one for trunk first?

hadoop-yetus · 2023-07-19T21:52:21Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	12m 54s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 3s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	xmllint	0m 0s		xmllint was not available.
+0 🆗	markdownlint	0m 0s		markdownlint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 79 new or modified test files.
			_ branch-3.3 Compile Tests _
+0 🆗	mvndep	15m 38s		Maven dependency ordering for branch
+1 💚	mvninstall	40m 40s		branch-3.3 passed
+1 💚	compile	20m 12s		branch-3.3 passed
+1 💚	checkstyle	3m 10s		branch-3.3 passed
+1 💚	mvnsite	3m 14s		branch-3.3 passed
+1 💚	javadoc	2m 3s		branch-3.3 passed
+0 🆗	spotbugs	0m 41s		branch/hadoop-project no spotbugs output file (spotbugsXml.xml)
+1 💚	shadedclient	41m 38s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 58s		Maven dependency ordering for patch
+1 💚	mvninstall	1m 59s		the patch passed
+1 💚	compile	20m 12s		the patch passed
-1 ❌	javac	20m 12s	/results-compile-javac-root.txt	root generated 17 new + 1862 unchanged - 15 fixed = 1879 total (was 1877)
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	3m 5s	/results-checkstyle-root.txt	root: The patch generated 17 new + 73 unchanged - 7 fixed = 90 total (was 80)
+1 💚	mvnsite	3m 10s		the patch passed
-1 ❌	javadoc	0m 41s	/results-javadoc-javadoc-hadoop-tools_hadoop-aws.txt	hadoop-tools_hadoop-aws generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
+0 🆗	spotbugs	0m 31s		hadoop-project has no data from spotbugs
-1 ❌	spotbugs	1m 35s	/new-spotbugs-hadoop-tools_hadoop-aws.html	hadoop-tools/hadoop-aws generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
+1 💚	shadedclient	41m 53s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	0m 29s		hadoop-project in the patch passed.
+1 💚	unit	18m 52s		hadoop-common in the patch passed.
+1 💚	unit	2m 54s		hadoop-aws in the patch passed.
+1 💚	asflicense	0m 58s		The patch does not generate ASF License warnings.
		252m 18s

Reason	Tests
SpotBugs	module:hadoop-tools/hadoop-aws
	Inconsistent synchronization of org.apache.hadoop.fs.s3a.S3AFileSystem.s3AsyncClient; locked 60% of time Unsynchronized access at S3AFileSystem.java:60% of time Unsynchronized access at S3AFileSystem.java:[line 1764]

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5858/1/artifact/out/Dockerfile
GITHUB PR	#5858
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint markdownlint
uname	Linux 90f724bc2e2e 4.15.0-212-generic #223-Ubuntu SMP Tue May 23 13:09:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	branch-3.3 / `aba4183`
Default Java	Private Build-1.8.0_362-8u372-ga~~us1-0ubuntu1~~18.04-b09
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5858/1/testReport/
Max. process+thread count	3144 (vs. ulimit of 5500)
modules	C: hadoop-project hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5858/1/console
versions	git=2.17.1 maven=3.6.0 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

steveloughran · 2023-07-20T11:27:54Z

@mukund-thakur this is the rebase for the feature branch-3.3; I do have one for trunk too, #5857

i want to make sure everyone is happy with this not complicating any open PRs, and I want to rebase atop branches with #5821 in as well.

Contributed by Ahmar Suhail

steveloughran · 2023-07-20T18:58:38Z

will manually merge if yetus is happy

hadoop-yetus · 2023-07-20T22:52:30Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 40s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 4s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+0 🆗	xmllint	0m 1s		xmllint was not available.
+0 🆗	markdownlint	0m 1s		markdownlint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 79 new or modified test files.
			_ branch-3.3 Compile Tests _
+0 🆗	mvndep	15m 26s		Maven dependency ordering for branch
+1 💚	mvninstall	40m 18s		branch-3.3 passed
+1 💚	compile	19m 22s		branch-3.3 passed
+1 💚	checkstyle	2m 59s		branch-3.3 passed
+1 💚	mvnsite	3m 44s		branch-3.3 passed
+1 💚	javadoc	2m 31s		branch-3.3 passed
+0 🆗	spotbugs	0m 52s		branch/hadoop-project no spotbugs output file (spotbugsXml.xml)
+1 💚	shadedclient	38m 11s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	1m 3s		Maven dependency ordering for patch
+1 💚	mvninstall	2m 2s		the patch passed
+1 💚	compile	18m 35s		the patch passed
-1 ❌	javac	18m 35s	/results-compile-javac-root.txt	root generated 17 new + 1862 unchanged - 15 fixed = 1879 total (was 1877)
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	2m 53s	/results-checkstyle-root.txt	root: The patch generated 17 new + 73 unchanged - 7 fixed = 90 total (was 80)
+1 💚	mvnsite	3m 43s		the patch passed
-1 ❌	javadoc	0m 55s	/results-javadoc-javadoc-hadoop-tools_hadoop-aws.txt	hadoop-tools_hadoop-aws generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
+0 🆗	spotbugs	0m 44s		hadoop-project has no data from spotbugs
-1 ❌	spotbugs	1m 43s	/new-spotbugs-hadoop-tools_hadoop-aws.html	hadoop-tools/hadoop-aws generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
+1 💚	shadedclient	39m 29s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	0m 43s		hadoop-project in the patch passed.
+1 💚	unit	18m 22s		hadoop-common in the patch passed.
+1 💚	unit	3m 8s		hadoop-aws in the patch passed.
+1 💚	asflicense	1m 9s		The patch does not generate ASF License warnings.
		234m 28s

Reason	Tests
SpotBugs	module:hadoop-tools/hadoop-aws
	Inconsistent synchronization of org.apache.hadoop.fs.s3a.S3AFileSystem.s3AsyncClient; locked 60% of time Unsynchronized access at S3AFileSystem.java:60% of time Unsynchronized access at S3AFileSystem.java:[line 1764]

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5858/2/artifact/out/Dockerfile
GITHUB PR	#5858
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint markdownlint
uname	Linux 2babfb97867e 4.15.0-212-generic #223-Ubuntu SMP Tue May 23 13:09:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	branch-3.3 / `d4deb9b`
Default Java	Private Build-1.8.0_362-8u372-ga~~us1-0ubuntu1~~18.04-b09
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5858/2/testReport/
Max. process+thread count	1302 (vs. ulimit of 5500)
modules	C: hadoop-project hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5858/2/console
versions	git=2.17.1 maven=3.6.0 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

mukund-thakur · 2023-07-24T16:07:24Z

hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AUtils.java

-   * @throws IllegalArgumentException if misconfigured
-   * @throws IOException problem getting username/secret from password source.
-   */
-  public static void initProxySupport(Configuration conf,


@ahmarsuhail why this has been removed?
CC @steveloughran

All config stuff (including this) has been moved to imp/AWSClientConfig. Proxy config can be found is AWSClientConfig.createProxyConfiguration()

steveloughran · 2023-07-26T14:10:05Z

ok, done the forced push of this; closing

HADOOP-18073. Upgrade S3A in 3.3 branch to AWS SDK V2.(apache#5707)

d4deb9b

Contributed by Ahmar Suhail

steveloughran force-pushed the s3/feature-HADOOP-18073-s3a-sdk-upgrade-3.3-rebase branch from aba4183 to d4deb9b Compare July 20, 2023 18:56

steveloughran marked this pull request as draft July 20, 2023 18:58

mukund-thakur reviewed Jul 24, 2023

View reviewed changes

steveloughran closed this Jul 26, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HADOOP-18073. Upgrade S3A in 3.3 branch to AWS SDK V2.(#5707) #5858

HADOOP-18073. Upgrade S3A in 3.3 branch to AWS SDK V2.(#5707) #5858

steveloughran commented Jul 19, 2023

mukund-thakur commented Jul 19, 2023

hadoop-yetus commented Jul 19, 2023

steveloughran commented Jul 20, 2023

steveloughran commented Jul 20, 2023

hadoop-yetus commented Jul 20, 2023

mukund-thakur Jul 24, 2023

ahmarsuhail Jul 24, 2023 •

edited

Loading

mukund-thakur Jul 24, 2023

steveloughran commented Jul 26, 2023

HADOOP-18073. Upgrade S3A in 3.3 branch to AWS SDK V2.(#5707) #5858

HADOOP-18073. Upgrade S3A in 3.3 branch to AWS SDK V2.(#5707) #5858

Conversation

steveloughran commented Jul 19, 2023

Description of PR

How was this patch tested?

For code changes:

mukund-thakur commented Jul 19, 2023

hadoop-yetus commented Jul 19, 2023

steveloughran commented Jul 20, 2023

steveloughran commented Jul 20, 2023

hadoop-yetus commented Jul 20, 2023

mukund-thakur Jul 24, 2023

Choose a reason for hiding this comment

ahmarsuhail Jul 24, 2023 • edited Loading

Choose a reason for hiding this comment

mukund-thakur Jul 24, 2023

Choose a reason for hiding this comment

steveloughran commented Jul 26, 2023

ahmarsuhail Jul 24, 2023 •

edited

Loading