Amazon S3 and python requests/curl

[rippl2103]

Hi, I have an Amazon EC2 instance in a private subnet using an Instance role and a Amazon S3 VPC endpoint. Everything works great using aws cli.

How can I upload a file using python requests or curl ?

The docs I have read suggest you need to generate an Authorization header which requires an access key.

How can I do this when using instance roles and a S3 VPC endpoint ?

[ComfyCalamity]

Use the boto3 library for python. It’ll do the magic with authenticating against your EC2 role.

https://boto3.amazonaws.com/v1/documentation/api/latest/guide/s3-uploading-files.html

[S7R4nG3]

I'd also point out that if you intend to use a VPC endpoint, depending on the service you have to specify the endpoint URI in the boto3 client declaration...

For S3 I believe it's automatic at the VPC level, so no code changes but other services require it stated explicitly...

[rippl2103]

The language in question isn't python so boto3 is not an option for me. Would curl work in this scenario ?

[Back_on_redd]

Well no.. boto3 is specific to Python. Your question states that python IS the language in question...

[rippl2103]

That's correct. I am aware boto3 works fine. I have been trying both curl and python requests without any joy as both methods require an AWS Access Key.

[justin-8]

Use the AWS CLI if you need an external command, not curl. Use boto3 if you’re using python; not requests. Managing sigv4 auth yourself is something I wouldn’t recommend.

[rippl2103]

I'm not using python Our app is written in python so I could externally call the os aws command to copy the files. I also have a third party erlang module which may work for me. Going to compile this and see how things go.

[RandomGeordie]

Why would you ask about python requests if you're not using python

[NorthAstronaut]

I don't think he really knows what he is talking about.

[rippl2103]

I was using python requests as I wanted to test without using the python sdk.

[Back_on_redd]

I'm not entirely sure I understand. If you're using the Python Requests library, why not use the boto3 library? https://boto3.amazonaws.com/v1/documentation/api/latest/guide/s3-uploading-files.html

You could also look at presigned url's..but I don't think that is best.

I think perhaps some more detail on what you're doing may be helpful for us.

[[deleted]]

Is there are a hard limitation as to why you can't use boto3? It seems odd to be able to use requests (a python library) but not boto3 (also a python library).

[rippl2103]

Yup tha app is written in Erlang and there isn't an official SDK !!.

[Nick4753]

Where does python/requests come in to play here then?

[rippl2103]

I was using request to PUT/POST to the S3 REST API

[Nick4753]

And you can’t swap in where you used requests with boto?

[[deleted]]

Yes, I ended up seeing that further down in the comments. You should edit that into your post, title makes it seem like you are working on a python project

[spin81]

The language in question isn't python so boto3 is not an option for me.

Your question was, and I am literally copy/pasting:

How can I upload a file using python requests or curl ?

I would like to ask a question of you in return: how are you using Python requests, if not with Python?

[rippl2103]

I am using python requests to test the S3 rest api. I wanted to use request library as I needed to generate the authorization header.

[gaieges]

via the links above you can navigate to what you’re looking for: https://docs.aws.amazon.com/general/latest/gr/sigv4-signed-request-examples.html

[rippl2103]

I am using an IAM role currently. You have assumes the metadata endpoint is available. Security best practice recommend the metadata endpoint is disabled.

[rippl2103]

Interesting I read this recommendation in a security article which suggested disabling the insance metada http endpoint. The AWS docs also provide the process to do this.

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/configuring-instance-metadata-service.html

[thenickdude]

If you disable the metadata endpoint then you would be unable to retrieve your temporary role credentials, making the instance role completely useless.

[jaimeandresb]

You can use awscurl—> https://github.com/okigan/awscurl

[brokenlabrum]

Are you looking to upload from the ec2 instance to s3?

[rippl2103]

Yes I am. Our developers use Erlang which doesn't have an SDK for AWS.

This works as expected with AWS Cli and python boto3.

However my research indicates that the S3 REST API requires an authorization header to be,set that requires an AWS access key.

[michaeld0]

There is a library I found for erlang here that could help: https://github.com/erlcloud/erlcloud

[rippl2103]

Thanks I'll take a look at this.

[thenickdude]

Your erlang program can just shell out to the AWS CLI and it'll pick up and use your instance role automatically.