HADOOP-18073. Upgrade AWS SDK to v2 in S3A #5981
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
See aws_sdk_v2_changelog.md for details. Co-authored-by: Ahmar Suhail <ahmarsu@amazon.co.uk> Co-authored-by: Alessandro Passaro <alexpax@amazon.co.uk> HADOOP-18073. Address review comments. (apache#31) addresses review comments + yetus errors Co-authored-by: Ahmar Suhail <ahmarsu@amazon.co.uk> Move MultiObjectDeleteException to impl Reinstate old constants Move TransferManager initialization to ClientFactory Add unit tests for BlockingEnumeration Add unit tests for SelectEventStreamPublisher updates new providers in TestS3AAWSCredentialsProvider to V2 update GET range referrer header logic to V2 adds in unit check for bytes HADOOP-18565. Complete outstanding items for the AWS SDK V2 upgrade. (apache#5421) Changes include * use bundled transfer manager * adds transfer listener to upload * adds support for custom signers * don't set default endpoint * removes v1 sdk bundle, only use core package * implements region caching + many more Note: spotbugs is warning about inconsistent synchronization in accessing a new s3a FS field. This will be fixed in a follow-up patch. Contributed by Ahmar Suhail
This removes the AWS V1 SDK as a hadoop-aws runtime dependency. It is still used at compile time so as to build a wrapper class V1ToV2AwsCredentialProviderAdapter which allows v1 credential provider to be used for authentication. All well known credential providers have their classname remapped from v1 to v2 classes prior to instantiation; this wrapper is not needed for them. There is no support for migrating other SDK plugin points (signing, handlers) Access to the v2 S3Client class used by an S3A FileSystem instance is now via a new interface org.apache.hadoop.fs.s3a.S3AInternals; other low-level operations (getObjectMetadata(Path)) have moved. Contributed by Steve Loughran
Upgrades the AWS sdk v2 version to 2.20.28 This * adds multipart COPY/rename in the java async client * removes the aws-crt JAR dependency Contributed by Ahmar Suhail
|
💔 -1 overall
This message was automatically generated. |
|
"interesting" |
|
@mukund-thakur's TTL patch broke the rebase; fixing by moving the configuration code and cutting the test -as with the move of the setting to the http client builders there's no getter to validate it was passed in. |
|
moving work to #5981 because the merge has got more complex with rebase/build problems. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description of PR
Aggregate PR of commits needed to move hadoop trunk to the AWS v2 SDK.
will merge as a chain of commits, just pushing through yetus as a single large PR to see how it reacts
How was this patch tested?
s3 london -Dprefetch -Dscale
Found one regression, HADOOP-18853, which affects one test and can be fixed later.
For code changes:
LICENSE,LICENSE-binary,NOTICE-binaryfiles?