50,970 questions
-2
votes
1
answer
19
views
How do I find S3 Trigger conflicts in AWS Lambda?
I've built a Lambda function in the AWS Console, and it seems fine, but when I add an S3 Trigger, it gives me a message "Confugrations overlap. Configurations on the same bucket cannot share a ...
- 4,119
0
votes
0
answers
47
views
Best practice for backing up large SQL Server databases to S3 [closed]
We are currently backing up all of our databases to a series S3-Compatibable storageGRID buckets hosted by our data center provider. Some of our larger databases backups run quite long (upwards of 8 ...
0
votes
1
answer
77
views
Error when calling dbms_cloud.export_data
I'm trying to do my first CSV export into an existing AWS S3 bucket. Calling with type CSV:
call dbms_cloud.export_data(credential_name => 'cred', file_uri_list => 'https://s3.us-east-2....
- 1
0
votes
0
answers
28
views
CloudFront + S3 static site: same HTML/CSS renders differently than when served locally — is my Terraform CloudFront config wrong? [closed]
I’m hosting a static website on Amazon S3 behind CloudFront, with the entire setup created and deployed via Terraform. The odd behavior I’m running into is that the exact same HTML and CSS files ...
1
vote
2
answers
79
views
AWS CloudFormation deploy template stored in an S3 bucket [closed]
Trying to deploy a CloudFormation template that is stored in an S3 bucket as the template size is larger than 51kb so cant be done direct (from what I am led to believe and have seen when trying)
...
- 1,091
-4
votes
1
answer
63
views
AWS Lambda Python script calling SageMaker: AlgorithmError: Framework Error
I am using lambda and Python and S3.
# lambda_bootstrap_train.py
import boto3
import time
import json
import os
sm = boto3.client('sagemaker', region_name='us-east-2')
s3 = boto3.client('s3', ...
- 688
-1
votes
1
answer
64
views
I tried to make an aws IAM user to see S3 buckets with certain tags
Here is the IAM policy (Mostly by chatgpt) :
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "AllowListAllBucketsForConsole&...
- 11
1
vote
1
answer
85
views
Awswrangler: Parquet read into multiple of expected space
In a Lambda, I'm using AWS Wrangler to read data out of a date partitioned set of parquets and concatenate them together. I am doing this by calling wr.s3.read_parquet in a loop, compiling the loaded ...
- 80
0
votes
1
answer
48
views
AWS S3 SCP Policy to Prevent Disabling Public Access Block but Allow Re-enabling
I need to create an AWS IAM policy that prevents users from disabling the "Block all public access" configuration on S3 buckets, but still allows them to re-enable it if it was already ...
- 1,146
0
votes
1
answer
54
views
Spark Structured streaming job failed in mid of day due to delta file not found in S3 checkpoint folder
I'm using spark version 3.4.4 and running spark structured streaming application.
Data pipeline reads data from kafka continuously and do some aggregations on stream data and stores the result into ...
- 1,171
2
votes
1
answer
55
views
What is the best way to check the file size in S3 multipart upload?
I would like to set a file size limit in uploads through multipart upload in my application, but my problem is, using pre-signed-url, I can't check the file size of each part and all the options I ...
- 21
1
vote
0
answers
35
views
NiFi ListS3 Processor SdkClientException: Name or service not known for S3 bucket
I'm setting up an Apache NiFi flow using the ListS3 processor to scan an AWS S3 bucket, but I keep getting errors, and the processor isn't listing any files from the bucket.
NiFi Processor ...
- 61
1
vote
0
answers
40
views
Logging username that generates GET/PUT pre-signed url for audit in s3 boto
trying to log the username that generates a presigned url for 'GET' method for auditing.
Need the info to be available server-side.
What I've tried:
Concat the username to the url '&', this wont ...
- 21
0
votes
0
answers
127
views
Why am I getting "Invalid initialization vector" when calling my signed URL upload API in Node.js?
I’m working on a Node.js backend that generates a signed URL for uploading files to AWS S3 via CloudFront.
However, when I hit the signed URL endpoint from curl, I always get the following response:
{&...
0
votes
0
answers
66
views
Telegram Bot sendDocument fails with Amazon S3 URL
I am using PHP with Guzzle to send a document via Telegram Bot API using sendDocument.
The file is hosted on Amazon S3 with a URL like:
https://telegramfile.xxx.amazonaws.com/file/preview/chat/...
- 1
1
vote
2
answers
532
views
How to handle Seahorse::Client::NetworkingError with certificate verify failed error
I'm running a Rails 8 app with a Option model that configures active storage like so:
has_one_attached :picture do |attachable|
attachable.variant :small, resize_to_limit: [75, 75]
end
config/...
- 4,637
1
vote
0
answers
30
views
Databricks s3 mounted folder - same files but in two different folders, one works the other throws rate limit error?
On data bricks I have mounted two s3 folders with virtually identical folder and file structure:
s3a://mybucket/folder1 mounted to /mnt/original_folder
s3a://mybucket/folder2 mounted to /mnt/kj/test2
...
- 1,931
0
votes
1
answer
51
views
DownloadWithContext retrive incomplete file
I have a problem with using S3 API and Go.
To read an object and immediately transfer it to Minio i use Pipes:
pr, pw := io.Pipe()
For example, I will create an archive object using pipe:
zipWriter :=...
- 345
0
votes
0
answers
48
views
Using createUploadDestinationForResource SP API Uploads API call in process of uploading listings images
I'm using createUploadDestinationForResource SP API Uploads API call to get an upload destination for a listing image to be uploaded. It works, but when I'm using returned destination upload URL to ...
- 1,654
-1
votes
2
answers
134
views
How to connect to S3 without the large AWS SDK v2 bundle?
I'm trying to read some file from S3 with PySpark 4.0.1 and the S3AFileSystem.
The standard configuration using hadoop-aws 3.4.1 works, but it requires the AWS SDK Bundle. This single dependency is ...
- 5,704
0
votes
0
answers
43
views
Apple's app-site-association CDN returning 403 for S3 in EU (eu-west-1) region, but works in other regions
We are hosting apple-app-site-association files across multiple domains, in the US AWS regions it works fine, but for our EU-based domains (eu-west-1) it does not work.
Setup:
The apple-app-site-...
- 2,447
0
votes
0
answers
93
views
How to automatically delete malware-infected files in S3 using AWS GuardDuty?
I'm implementing a file upload workflow using Amazon S3 and want to integrate AWS GuardDuty for malware protection. The goal is to automatically scan uploaded files and delete any that are flagged as ...
- 11
0
votes
3
answers
110
views
403 Response for Presigned S3 PutObject URL
I'm getting a 403 response when trying to use a presigned S3 put object URL in a Javascript fetch call. Here's what I've verified:
The IAM role that generates the presigned URL can upload an object ...
- 729
0
votes
1
answer
85
views
Setup s3 log for airflow 3.0.5
I am setting remote_log for airflow to s3 minio. My airflow version is 3.0.5. Here is my configuration:
AIRFLOW__LOGGING__REMOTE_TASK_HANDLER_KWARGS: '{"delete_local_copy": true}'
...
0
votes
0
answers
38
views
Getting error when using s3FileSystem glob method to read parquet files
I am trying to read all the parquet files from a dated folder in S3 using S3FileSystem glob method:
def read_parquet_files_from_s3(self, table, schema, start_date, tenant_id):
bucket_name = '...
- 59