40 questions
0
votes
3
answers
117
views
Copying S3 Server Logs to a Centralized AWS Account
As a part of centralized logging into a different AWS account, I will need to send the S3 Server Logs to a different AWS account that is used for Centralized Logging for all the AWS accounts in our ...
- 8,105
0
votes
1
answer
300
views
AWS DataSync S3 transfer failed with Cannot allocate memory
I am using AWS DataSync to copy an S3 bucket from one region to another in the same account. For a few of my buckets, the DataSync task execution is failing with the error "Cannot allocate memory&...
- 1,748
0
votes
1
answer
2k
views
AWS DataSync some errors in your task execution. No affected files in CloudWatch logs
I'm getting this error when I run a DataSync task:
Transfer and verification completed. There were some errors in your task execution. See a list of affected files in CloudWatch logs. If no files are ...
- 357
0
votes
1
answer
559
views
Which permissions for a cross-accounts transfert across two s3 with aws-datasync
I'm currently following the AWS official documentation tutorial for transferring a large volume of data between two S3 buckets, which can be found at this link: https://docs.aws.amazon.com/datasync/...
- 189
0
votes
1
answer
1k
views
Execute AWS Datasync task using step function
We have a requirement to create a state machine which includes to datasync tasks followed by each other. These datasync tasks are already created and now I want to execute it using step functions one ...
- 57
0
votes
0
answers
405
views
Task timed out when creating DataSync agent through boto3 in aws lambda
I'm trying to create a DataSync agent. When I use the console with the same parameters, it works fine and very quickly (a few seconds). However, if I want to create it by using the lambda function, ...
- 75
1
vote
1
answer
671
views
aws datasync cause double trigger for lambda event
I use AWS DataSync service to sync two buckets A and B
The DataSync defined to check integrity during transfer and copy only new data to another bucket (B).
I have lambda function that defined to ...
- 417
-1
votes
1
answer
286
views
Sync S3 Objects Across Bucket with Intelligent Tiering
We want to sync all data from a us-east-1 bucket to us-west-2 bucket. However, we have intelligent tiering enabled. The us-east-1 bucket has thousands of objects where a large portion of them can be ...
- 192
0
votes
1
answer
1k
views
AWS DataSync agent ran into an error connecting to AWS
I'm trying to sync local S3 object storage to AWS S3 using AWS DataSync.
I configured agent and it shows as online in AWS DataSync dashboard under Agents tab.
This is screen of local datasync agent: ...
0
votes
1
answer
4k
views
S3 to S3 Cross account data replication
I have a S3 bucket in Account A and i want to replicate it to S3 bucket in Account B. I need the whole process along with security measures explained in detail. Starting from how to setup the whole ...
- 37
0
votes
1
answer
1k
views
How can i access error individual attributes in golang
!! i Am new to go !!
I Am using the datasync API to start a task execution , that works with no issues. I Am struggling with the returned error struct, i want to acces individual elements but i can't ...
- 348
1
vote
0
answers
118
views
How can I install AWS Data sync agent on Hadoop cluster?
I have my hadoop sandbox set up on a Virtual Machine on Azure Cloud. I would like to install aws data sync agent on my hadoop sandbox so that I can create a task for data copy from my Hadoop cluster ...
- 23
1
vote
1
answer
4k
views
AWS DataSync: Unable to connect to S3 endpoint
I am trying to sync two S3 buckets in different accounts. I have successfully configured the locations and created a task. However, when I run the task I get a Unable to connect to S3 endpoint error. ...
0
votes
1
answer
1k
views
Invalid parameter: ARN account () must match authenticated user.",
My Python version is 3.9 and I am writing a AWS lambda function using boto3
In addition to assigning all admin access and s3full and datasync full roles, I also created a trust relationship but still ...
- 11.5k
1
vote
1
answer
2k
views
How to sync RDS database across regions in AWS
I have two MYSQL RDS Instance in Ohio and Mumbai region. Both RDS instance should sync each other. Does AWS have any services to enable sync between RDS Database? else should we configure to achieve? ...
2
votes
1
answer
3k
views
Describing ENI error when executing DataSync task in Fargate/Lambda
I am experiencing the error "InvalidRequestException: Invalid request made for describing ENI" when executing AWS DataSync task from Fargate/Lambda. What could be the cause of this?
- 1,462
0
votes
1
answer
1k
views
AWS Glue sync data from RDS (need to sync 4 table from all schema) to S3 (apache parque format)
We are using a Postgres RDS instance (db.t3.2xlarge with around 2TB data). We have a multi-tenancy application so for all organizations who sign up in our product, we are creating a separate schema ...
- 141
2
votes
1
answer
5k
views
AWS Autosync : moving data from FTP server to S3
I have a workflow which moves 700gb in files from an ftp server to an on-prem server for python script processing.
i would like to migrate this process to a AWS s3 bucket for lambda to process.
i ...
- 14k
1
vote
2
answers
901
views
Migrate existing data from Ali Cloud OSS to Amazon S3 Bucket
I am wondering if there is any AWS service to migrate existing data from ali cloud OSS to Aws S3 bucket.
Here is a service in Ali Cloud which migrates from S3 to OSS. I need opposite
https://www....
0
votes
1
answer
369
views
Ensure data is not lost during AWS S3 sync
I need to copy all contents of an S3 bucket to another S3 bucket. Planning to use s3 sync.
aws s3 sync s3://sourcebucket s3://destinationbucket
After this process, is there any way to verify if all ...
- 3,041