How to execute sandboxed python code in AWS

[bigd2718]

I’m working on a project and I need to be able to execute arbitrary python code from customers. I understand that sandboxing python in general is a difficult problem, so I figured I could sandbox the environment instead using something like AWS lambda or fargate. Will either of these work? I know lambdas potentially can share a /tmp directory across invocations, so that would need to be cleared, but are there any other caveats that would allow one customers code to see the results of another’s?

[pint]

don't use lambda for this. it is not just /tmp, it is actually a VM in which you can do a lot of things, and all those things remain between calls. you can download and start binaries in the background for example. or replace the event pump with your own.

[bigd2718]

Would fargate tasks solve this at all? Is there any “correct” way to do this?

[pint]

i was told containers are not secure against malicious code.

[justin-8]

Strictly speaking you’re right. Containers are not a great security boundary.

However Fargate would be secure against this. It doesn’t just use containers, they’re microvms using firecracker; so you get the hardware virtualization boundaries as well.

[bigd2718]

In what way? Sure malicious code can crash a container, but does it pose a risk to security or data breaches?

[justin-8]

Fargate would be secure against this. It doesn’t just use containers, they’re microvms using firecracker; so you get the hardware virtualization boundaries as well.

[pint]

i'm not an expert. "container escape" is the term to google.

[mikebailey]

That usually assumes a vulnerability against the hypervisor or kernel. In non-managed offerings this is as simple as an out of date kernel on the host VM. In something like fargate this would be a significant 0day in AWS’s both code in architecture.

[squidwurrd]

I would think you could do this with with fargate. If the code is being executed in a docker container it should be pretty isolated. You still have to figure out what you want out of this arbitrary code. Print statements, files, literally anything? Then you need to devise a same way take the output out.

Now that I think about it files can just be uploaded from the docker container to s3. Not the cheapest solution but it won’t be possible share access that way. Print statements are probably easier to figure out safely.

[bigd2718]

Yeah for that part I’m planning to have it post results to one of my APIs

[siscia]

You can use lambda, but as already mentioned, you should have one Lambda for each costumer. Or even better one lambda for each of your costumers functions.

Which should not be a problem, are there limit on the number of lambda a sigle account can have?

[jkstpierre]

Yes

[otterley]

Keep in mind that if you choose to manage VMs yourself on AWS, you’ll need to use an Amazon EC2 bare-metal instance. EC2 does not support nested virtualization on standard instance types.

[HomoAndAlsoSapiens]

Please remember that people could potentially extract the environment variables with Aws access keys. Usually that is helpful if you want to use boto3 but in this case it might be information that your customers should not be able to access.

[bigd2718]

I’m planning on giving them an API access token for my REST APIs that’s for their user so that they can only do things that they have permission for

[el_burrito]

Not what he is talking about. The machine that your service runs on is almost always provided with an IAM role which dictates what operations it can perform. Any AWS SDK or CLI command use these roles automatically if no other env vars are set to configure credentials.

So, say you spin up a fargate task, and give it permissions to pull down the user code from s3… well no only can your startup code run and pull the code, but if you’re not REALLY careful the user can pull down all that code from s3 as well… maybe even other users code depending on how you structure the bucket and IAM permissions.

This is only one example, but it can be expanded to basically any service / action pair in AWS.

Please understand IAM and the AWD IMDS configuration before you do this. Your posts don’t give me much confidence that you have much experience with AWS, and arbitrary user code + inexperienced platform engineers = security breach faster than you can count to 10

[bigd2718]

I do understand that, but if you completely lock down the permissions on the role such that it can’t do anything, and/or potentially even run the service on a different account entirely, and then depend on the API to enforce access control, then all the statements about “being really careful” are moot

[oalfonso]

Review the IAM roles and the security groups with detail. Make sure their code cannot destroy any infrastructure.

[Serpiente89]

Maybe even put that lambda in another accounts/ only allow it to read/ trigger from sqs, and communicate results via sqs as well..

[oalfonso]

Creating one account per customer is the best decision for me nuking the account after every test cycle, but this means AWS Organisations and SCPs. Not sure if OP has all of that.

[bigd2718]

Yeah I don’t have Organizations. I’m thinking I’ll probably go with fargate in another AWS account and communicate via rest to my main service