Transfer full s3 from one account to another | Moving Large S3 Files using dataSync Across Account

If you're Transferring large S3 buckets across AWS accounts you can do that in multiple ways. In this video, I will explain how you can do it using dataSync aws which is 10x faster than normal transfer.
===================== Source bucket policy =====================
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": [
"arn:aws:iam::DIST_ACCOUNT_ID:role/datasync-role",
"arn:aws:iam::DIST_ACCOUNT_ID:user/distention_account_logged_in_user"
]
},
"Action": [
"s3:GetBucketLocation",
"s3:ListBucket",
"s3:ListBucketMultipartUploads"
],
"Resource": "arn:aws:s3:::rource_bucket"
},
{
"Effect": "Allow",
"Principal": {
"AWS": [
"arn:aws:iam::DIST_ACCOUNT_ID:role/datasync-role",
"arn:aws:iam::DIST_ACCOUNT_ID:user/distention_account_logged_in_user"
]
},
"Action": [
"s3:AbortMultipartUpload",
"s3:DeleteObject",
"s3:GetObject",
"s3:ListMultipartUploadParts",
"s3:PutObjectTagging",
"s3:GetObjectTagging",
"s3:PutObject"
],
"Resource": "arn:aws:s3:::source_bucket/*"
}
]
}
=================Distention account role policy==================
{
"Version": "2012-10-17",
"Statement": [
{
"Action": [
"s3:GetBucketLocation",
"s3:ListBucket",
"s3:ListBucketMultipartUploads"
],
"Effect": "Allow",
"Resource": "arn:aws:s3:::source_bucket"
},
{
"Action": [
"s3:AbortMultipartUpload",
"s3:DeleteObject",
"s3:GetObject",
"s3:ListMultipartUploadParts",
"s3:PutObject",
"s3:GetObjectTagging",
"s3:ListBucket",
"s3:PutObjectTagging"
],
"Effect": "Allow",
"Resource": "arn:aws:s3:::source_bucket/*"
}
]
}
=============== Distention Bucket policy ==================
{
"Version": "2008-10-17",
"Statement": [
{
"Sid": "DataSyncCreateS3LocationAndTaskAccess",
"Effect": "Allow",
"Principal": {
"AWS": [
"arn:aws:iam::DIST_ACCOUNT_ID:role/datasync-role",
"arn:aws:iam::DIST_ACCOUNT_ID:user/distention_account_logged_in_user"
]
},
"Action": [
"s3:GetBucketLocation",
"s3:ListBucket",
"s3:ListBucketMultipartUploads",
"s3:AbortMultipartUpload",
"s3:DeleteObject",
"s3:GetObject",
"s3:ListMultipartUploadParts",
"s3:PutObject",
"s3:GetObjectTagging",
"s3:PutObjectTagging"
],
"Resource": [
"arn:aws:s3:::dist_bucket",
"arn:aws:s3:::dist_bucket/*"
]
}
]
}
===================== AWS CLI command =====================
aws datasync create-location-s3 --s3-bucket-arn arn:aws:s3:::s3-source-01 --s3-storage-class STANDARD --s3-config BucketAccessRoleArn="arn:aws:iam::1234567890:role/datasync-role" --region us-east-1

[@WilliamVirkin]

this video explain WAY BETTER than the official documentation in AWS and also that documentation has a lot of inconsistency.

[@fahadsayed6933]

works like a charm! .. thank you very much. It saved me a lot of time.

[@InternalAmbition]

This video is great! Thank you

[@marxito0]

excellent thank

[@civil5109]

Thanks jdk develoment its is working now

[@jdkDevelopmentOfficial]

Pleasue

[@jogurt5661]

Great work, thx

[@BeSavage177]

Thanks for this!! Does this support accounts across different organizations so data can be transferred to various clients if needed?

[@jdkDevelopmentOfficial]

Yes it does!

[@syednawaz819]

It helped. Thanks a lot.

[@oblivster]

🔥🔥🔥

[@ravitejanimma3831]

I have followed your steps, and faced same error. "DataSync location access test failed: could not perform s3:HeadBucket on bucket Access denied. Ensure bucket access role has s3:ListBucket permission."

[@user-wj5qx7di9i]

not the best approach, but instead of using IAM role/ IAM user i used root user and instead of this string "arn:aws:iam::DIST_ACCOUNT_ID:user/distention_account_logged_in_user" try this: "arn:aws:iam::DIST_ACCOUNT_ID:root", this worked for me

[@tharunyarramreddy4183]

@@user-wj5qx7di9i if we dont have root user what is the alternative approach

[@AasthaKasera]

@@user-wj5qx7di9i Thanks a lot, this worked for me.

[@moforel]

Great Video! Is it possible to use datasync to transfer only objects with a specific tag? If so how?

[@gigadanelia8339]

Thanks!

[@bakhodiryakubov3981]

Can you move from Acccount1 West region to Account2 East region as well?

[@user-wp5ci6tg1o]

great video! one question, is object ownership disabled in the source account?

[@jdkDevelopmentOfficial]

yes

[@user-zf8jt4vj6l]

How is the EFS to EFS procedure between different accounts?

[@arsaphone]

Hi sir, this policy wont do anything harm to source s3 right? for example the source bucket cannot be accessible bcs the policy

[@jdkDevelopmentOfficial]

No there's nothing harm this policy is not for public It's for specific entity.

[@munchback9006]

Hi! can you please help i am getting this error while creating task : last step of the video Failed to create task Request ID: Action datasync:CreateTask Status code 400 API response DataSync location access test failed: could not perform s3:HeadBucket on bucket careem-pubweb-staging. Access denied. Ensure bucket access role has s3:ListBucket permission.

[@munchback9006]

i need to allow full access on datasync of source account to my cli user ?

[@jdkDevelopmentOfficial]

No need for full access but I think you're missing S3:ListBucket permission.

[@anubhavbiswas4142]

Does this feature involves cost? If I want to move millions of objects to another AWS account, will this incur any cost?

[@jdkDevelopmentOfficial]

Yes, It's costing but not too much you can find more details in their pricing module.

[@anubhavbiswas4142]

@@jdkDevelopmentOfficial thank you.. Yes i went through the pricing.. Thank you for this informative video

[@durantapatra443]

how to transfer 1GB single file ?

[@jdkDevelopmentOfficial]

You can transfer 1GB file using"aws sync" using aws cli.

[@siriusleto3758]

If you use MFA it will be more complicated. They never explain it in the tutorial.

[@user-jl6cj3sj6r]

Hi @jdkDevelopmentOfficial I followed your steps, but I got an error DataSync location access test failed: could not perform s3:GetObject in bucket source11. Access denied. Ensure bucket access role has s3:GetObject permission.

[@jawadali7728]

Make sure you have get files permission in source account.

i put "s3:GetObject", in the firts statement on the source role and works