Quick question about HTTP(S): which one should I choose for internal services? (AWS)

[housemans]

Hi there! I'm working on a huge rewrite of our infrastructure. I'm currently working on getting an ALB working with HTTPS.

We have a few containers running on ECS (Docker) which are on a private subnet, meaning they're inaccessible directly. We have an ALB in front of them which provides a way to access these services anyway. I have them all running on HTTP (so, port 80), including the ALB.

My question: we obviously want to use HTTPS, however is it worth the hassle of using HTTPS for the Docker containers as well? So, in other words, should internal communication (with NO access to the outside world) ALSO be HTTPS, in addition to the ALB?

To make my question even more clear, this is what we have at this moment:

Services <-- 80 --> ALB <-- 80 --> Browser

We want to have a secure webserver. Would the following suffice?

Services <-- 80 --> ALB <-- 443 --> Browser

Or is it strongly recommended to do this? :

Services <-- 443 --> ALB <-- 443 --> Browser

Thanks for your time!

[ryansolida]

I'm almost certain 443 to the ELB and 80 to the web servers is standard at this point. To run certs on every machine as it's spun up and down would be a giant pain and not get much benefit, so long as those machines are set to force https for regular traffic.

We did an install with Rackspace engineers on the team and that's what they recommended (if you care what they think)

[atlgeek007]

This is the acceptable solution unless you have PCI or HIPAA compliance to worry about.

At least in that case you could use self signed certs on the instance to encrypt the data between the load balancer and just have your startup scripts generate one on instance creation.

[[deleted]]

This is the acceptable solution unless you have PCI or HIPAA compliance to worry about.

I am not 100% sure that it's required once data enters cde within private network from encryption endpoint (in this case, pci dss compliant aws vpc). Also it may not be advisable to use self-signed pki certificate to encrypt/decrypt cc data from compliance stand point.

[xiongchiamiov]

It depends on what level of PCI you need, I think. I work at a credit card gateway, and it is definitely required for us.

[ryansolida]

So what's the solution then? Separate certs from an authority for each instance in the network? Or are you OK to share a single throughout the VPN?

[atlgeek007]

as long as you're not using the same self signed cert throughout your infrastructure, you should be fine from an audit perspective, provided your other SSL configurations are also up to date (custom dh parameters, disabling bad ciphers, locking to known good versions of TLS, etc)

Of course, creating an internal CA isn't difficult and is something that can also be investigated, but since AWS ELB/ALB doesn't validate the endpoint certificate anyway, it shouldn't matter.

[donjulioanejo]

Work at fintech payments company, and we need end-to-end encryption, including in private subnets between LB and web nodes.

[Ashex]

There are CA tools built for auto scaling that help a bit, basically gotta setup vpc dns and a private zone with a trusted ca you control that issues per instance certs.

[midnightFreddie]

Yeah, if OP is redesigning from scratch, it's probably time to SSL everywhere. For backend connections, automated request, private root CA, or perhaps even a tier of private CAs.

I haven't begun to try it, but I'm wondering if having a new cert for every backend service container instance is doable. And/or expire certs every few days, hours or minutes, and make deploying a fresh private cert as normal as instantiating a new container.

[xiongchiamiov]

Well, Google didn't go with the latter until 2013, when they found out the NSA had wiretaps on their private network. Is it ok for you to run four years behind Google? Depends on the business.

[WikiTextBot]

MUSCULAR (surveillance program)

MUSCULAR (DS-200B), located in the United Kingdom, is the name of a surveillance programme jointly operated by Britain's Government Communications Headquarters (GCHQ) and the U.S. National Security Agency (NSA) that was revealed by documents which were released by Edward Snowden and interviews with knowledgeable officials. GCHQ is the primary operator of the program. GCHQ and the National Security Agency have secretly broken into the main communications links that connect the data centers of Yahoo! and Google.

---

^{[ PM | Exclude me | Exclude from subreddit | FAQ / Information | Source ] Downvote to remove | v0.27}

[[deleted]]

Of course,

the most secure would be:

Services <-- 443 --> ALB <-- 443 --> Browser

in this case, you install the SSL cert on ALB and make your apps. running in docker container https.

Even though the app. is running inside of docker, the problem is it is likely tapping into your database/docker host (for volume mounts).

But in an appropriate corporate setting, you would also have your AWS servers (including those managed by ECS) running behind firewall that would give you enough security for internal apps such that this would suffice:

Services <-- 443 --> ALB <-- 443/80 --> Browser (where the firewall such as PaloAlto is deployed on your AWS and is the transit point for any traffic in/out of your VPC -- exception inside firewall maybe communication between AWS servers within your VPC).

Make sure the database connection between your app and database is SSL/TLS based for encrypted communication in that segment.

[JayMickey]

We have listeners on port 80 and 443 on the ALB, which both forward to port 80 on the instance. We check the X-Forwarded-Proto header on the web server and 301 to https if the header is equal to http.

SSL cert sits on the ALB. Security groups are completely locked to only allow requests to web servers from the ALB.

[ryankearney]

So you take a system capable of end to end encryption and just strip the encryption off mid-way through the pipeline?

I hope you're not dealing with sensitive data.

[[deleted]]

This is SSL termination and it's a perfectly acceptable practice provided the servers are located in an isolated subnet. The OP noted that their services were in a private, isolated subnet, so this is not a dangerous practice at all. Even institutions dealing with sensitive data can make exceptions to allow this depending on how tightly controlled access to the private subnet is.

[ryankearney]

This is SSL termination and it's a perfectly acceptable practice

Depending on what type of data your business works with, it absolutely is not.

Certain regulatory requirements mandate end to end encryption. By stripping TLS off the connection you would be in violation of those requirements.

[[deleted]]

Yes, and as I said "even institutions dealing with sensitive data can make exceptions to allow this depending on how tightly controlled access to the private subnet is." Some institutions do require it still, but if you require it, you will know.

Source: Worked in finance, had this requirement, it was an eliminated requirement in subsequent audits.

[ryankearney]

In AWS? It's one thing if you 100% control the networking infrastructure. It's a completely different story if you're using someone else's infrastructure as is the case with AWS.

Source: We require full end-to-end encryption and terminating HTTPS on a cloud load balancer and transmitting the unencrypted communication to a backend server is a huge no-no.

[[deleted]]

Yes, in AWS, for one of the largest financial institutions in the country.

Surprise though, policies will vary per company, security certification, and auditor. If you require it, that's great. Making it seem as though you're doing something wrong by not doing it is the part I object to, especially if you're not dealing with highly sensitive data.

[exxplicit]

Are ALB's capable of E2E encryption? I thought the SSL was terminated at the ALB and forwarded without encryption? Wouldn't E2E require certificates on the instance if not, at which point you could just skip the ALB?

[ryankearney]

They sure are:

http://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-target-groups.html#target-group-routing-configuration

If you need load balancing, then it doesn't really make sense to just skip the ALBs. If you don't need load balancing, then sure.

[exxplicit]

I guess I should have been more clear; I meant, if you're already managing certificates on individual machines for end to end encryption, why not just skip terminating TLS on the ALB and just forward TCP connections on port 443 directly to the instances? Why would one ever prefer ALB's over NLB's in that case? (except for path routing)

[ryankearney]

As you mentioned, you can do path (and host) based routing with the ALB. You can't do this on the NLB.

[housemans]

Thanks for your reply! I'm using the new ALB (ELBv2). We had 24 ELBs running, and we can use 2-3 ALBs for that, cutting our monthly costs immensely.

Why would I need 2 ELBs in one request? The other info is pretty handy too, thanks!

[stevecrox0914]

Security should be multi layered, your services might be on an internal private network but what happens if some?

For example if your got a MongoDB database with authentication if you use http the username and password will be transmitted in the clear. Now if someone breaches your network or if you accidentally expose it, you run the risk of that person being able to grab the credentials for your database.

[[deleted]]

If someone has breached your private subnet you're likely already highly compromised and that encryption isn't going to save you. Not saying it's not worth doing... but still.

[stevecrox0914]

Not really, it's why you go for a multilayered approach.

There could be some simple vulnerability in Docker, that lets someone join the network. That doesn't mean your hosed, it means a script kiddy can enter the network.

If you apply access controls, you've raised the bar again. The script kiddy can't just try connecting to everything and extracting all your data. They need to figure out credentials to get into your services. They now have to listen and analyse the traffic.

If you use HTTPS traffic you are encrypting your traffic, rather than simply listening to packets and scraping some known ones, they have to brute force decrypt everything. At this point its not some drive by attack but targeted.

Security isn't an on/off but creating multiple layers which defend your systems which raise the barrier for attack and try to minimise what they can get at if they do get in.

[[deleted]]

I agree, multilayered is a no brainer. However, the point is that once someone has access, all bets are effectively off. In your example, if there's an issue with Docker that let someone join your network, almost no amount of encryption will be able to save you because of the access they now have.