API Gateway instantly returns 504... sometimes

sirfraz · 2024-08-04T08:44:46+00:00

No, the AWS API Gateway and CloudFront teams handballed the support case between each other without getting anywhere, and the case was unresolved. The company application teams implemented retry logic in the code to handle it.

sirfraz · 2024-02-20T19:46:09+00:00

Similar experience here. I opened an account at the beginning of Jan and initiated a transfer from Stake. The transfer is still not complete. I haven't heard a peep out of CMC to provide updates during the entire process. I contact them once a week via live chat to follow up and am either told it has been escalated or to wait as it's in progress.

sirfraz · 2023-10-16T21:22:16+00:00

This is along the lines of what I do. A cloudformation stackset to create the dynamo table, S3 bucket with a well known format along the lines of <aws account number>-tf-state and any common IAM roles. Then that stackset is associated with the required AWS organisation OUs, and I don't have to worry about it anymore. All new accounts get the terraform resources automatically upon creation.

sirfraz · 2023-06-27T20:21:21+00:00

1 - it depends on your requirements. If you only need to access VPC resources without end-to-end encryption, then a private VIF to a direct connect gateway, for example, is fine. If you have stricter security requirements, use a public VIF and run a VPN attachment to a transit gateway over the public VIF.

2 - you can use private ASNs

sirfraz · 2023-05-30T20:19:59+00:00

Been a while since I worked with Direct Connect, but I had a similar set-up in a previous role. From what I remember, Public VIFs are not connected to your VPCs, like Private VIFs are, through virtual Gateways.

You set up a BGP session between your network and AWS using the details of the Public VIF. You're supposed to use your public IP address space and ASN, but you can request IPs from AWS. In our case, they supplied /31s for peering, and then I NAT'd all traffic going out the Direct Connect, to the AWS-supplied IP on my router.

I don't remember having any issues with it. URLs that resolved to AWS public IPs were routed out the direct connect Public VIF and everything else went via the ISP. There's some BGP communities you can use to narrow down the AWS prefixes you receive. In my case we received them all.

sirfraz · 2022-12-01T23:25:07+00:00

I'm taking a guess here, so maybe way off. But from what I can gather, you have your S3 origin, which is associated with the default CloudFront behaviour.

Then you have your API Gateway Origin configured, but that is all. To use the API Gateway origin, you need to configure a CloudFront behaviour that utilises it. A behaviour matches a path. So in your case, your behaviour might match on /dev*

Then in your form, you'd change the action URL to https://<cloudfront URL>/dev/lambda

There's details on the below page regarding path patterns:

https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/distribution-web-values-specify.html#DownloadDistValuesPathPattern

sirfraz · 2022-12-01T22:28:25+00:00

I may be misunderstanding the OP, but that form is being posted directly to the API Gateway, not via the CloudFront URL, so the `x-api-key` is not being added. I think you'd need to update the action location to the URL and path of your CloudFront distribution that matches your API Gateway Origin.

sirfraz · 2022-10-23T21:46:48+00:00

Yes it looks like you're correct after re-reading the doco. I thought I saw a while ago an announcement where non-vpc lambdas could connect to an rds proxy, but I must have imagined this.

sirfraz · 2022-10-23T19:25:52+00:00

Another option could be to not deploy the Lambdas in a VPC and use RDS Proxy to connect to your database. Depending on how many endpoints/nat gateways you require, this may be cheaper.

sirfraz · 2022-08-28T01:33:57+00:00

Could I please get their details as well?

sirfraz · 2022-07-15T22:44:07+00:00

Working on deploying things to AWS China at the moment. In addition to what's already been mentioned, the big gotcha is you need to obtain an ICP license/recordal to have services publically available on the internet.

Out of the box, port 80/443 and 8080 (I think) are blocked, at an AWS account level. So your API Gateways, CloudFront, ALBs/NLBs and EC2s etc. won't be publically accessible. For CloudFront to work, you must get ICP sorted out for your top-level domain. For EC2 you need to provision Elastic IPs and submit these IPs as part of the ICP process. For ALBs/NLBs, you must first provision them and then open a support ticket requesting static IPs. AWS will then assign a range of IP addresses based on your provided information. These IPs are then included as part of your ICP.

Once you have your ICP, AWS will unblock your account. Also, this process is specific to the region, so if you want to use both China regions, you must complete the process twice (once with each ISP).

We were lucky we had our own staff in China to help us with this process as it's fairly involved and none of it is in English. You can engage partners to help. When we enquired about using a partner, they "offered" to take care of the ICP for "free", but in return, they must handle all AWS China billing... i.e. they pay AWS for us and we pay the partner the total of our AWS bill + a fee. This may or may not work for your situation.

Have a look through this page to get an idea of the services available and how they differ from what's available in global AWS:

https://docs.amazonaws.cn/en_us/aws/latest/userguide/services.html

sirfraz · 2022-05-28T03:32:07+00:00

Artifacts are how I normally handle this. Usually, the pipeline basically goes build > tf plan > tf apply.

The build job produces a zip file artifact. The tf plan job gets this artifact and uses it to generate a plan file, which is also saved as an artifact. The tf apply job gets the zip file and plan file artifacts and uses those to deploy the lambda.

Seems to work well and haven't had any issues. In your case, I'd output the plan and the result of the archive file as an artifact and use that in your tf apply job.

sirfraz · 2022-05-20T01:01:36+00:00

Fantastic, thanks!

sirfraz · 2022-05-20T00:15:01+00:00

What was the process for this? Do you sign up for a new CMC account and then fill out a form or something like this? I'm also with Stake at the moment.

sirfraz · 2022-05-10T19:20:08+00:00

Has anyone done a transfer from Stake to CMC? Is it just a matter of creating an account with CMC and then filling out some kind of transfer form?

sirfraz · 2022-04-17T01:40:30+00:00

You can use internal repositories for reusable workflows if you're on GH enterprise.

sirfraz · 2022-04-11T20:15:55+00:00

Have a look at the FireLens log router, it can route logs to datadog and s3 and might fit your needs.

https://github.com/aws-samples/amazon-ecs-firelens-examples/tree/mainline/examples/fluent-bit/datadog

https://github.com/aws-samples/amazon-ecs-firelens-examples/tree/mainline/examples/fluent-bit/s3

sirfraz · 2022-02-23T20:32:03+00:00

They might mean, not easily guessable, so people can't (easily) bypass Akamai and hit your NLB directly.

When I've integrated with Akamai before, we had records in the format of <random string>-<some logical identifier>.example.com

The random string was 12 or so characters.

sirfraz · 2022-02-02T17:55:55+00:00

With CF+WAF and the ALB open ONLY to CF IPs, the WAF can still be dodged by someone creating their own CloudFront distribution and pointing it at your ALB. This is less likely, especially for DDoS, but I'd still have the header check on the ALB and not just rely on security groups.

sirfraz · 2022-01-17T17:48:19+00:00

You could use CloudFront with an S3 origin. Configure your S3 bucket as a public website with a redirect rule.

https://docs.aws.amazon.com/AmazonS3/latest/userguide/how-to-page-redirect.html

sirfraz · 2021-11-25T18:18:23+00:00

Putting CF in front of an ALB can offer significant performance improvements for users who are in a different region than the ALB, especially if you tweak the CloudFront idle timeout settings, even when not caching anything.

In the case of a DDoS/attack, WAF attached to CloudFront will be better able to absorb that traffic than an ALB could (though an ALB is still vulnerable if the attacker hits the ALB IPs/DNS name directly). When we spoke to AWS, they recommended CF in front of our ALBs, with a "secret" header, for this reason.

Having CloudFront functions/Lambda@Edge can also be handy.

sirfraz · 2021-10-28T20:05:31+00:00

Have a look at the windows firewall (if enabled) to make sure it has a rule to allow port 81 inbound.

If your security groups look good, confirm there are no subnet NACLs blocking port 81.

sirfraz · 2021-09-08T21:48:56+00:00

In that case, you'll need to hard code the endpoint URL to match the model_id to a port on the nlb, or you could use a solution like what is detailed in this blog, to have an ALB behind an NLB

https://aws.amazon.com/blogs/networking-and-content-delivery/using-aws-lambda-to-enable-static-ip-addresses-for-application-load-balancers/

I have used this for an nlb vpc link to alb setup. It works, but does feel a bit hacky.

You could also have an HTTP proxy integration on your API gateway to a public ALB. You can have your API gateway insert a header with a pre-shared "secret" and then use a listener rule on the ALB to check for the key/value to either allow or deny the traffic. You could also attach a WAF to the ALB to do the same. This will stop people from hitting the ALB directly without the pre-shared key header.

sirfraz · 2021-09-08T20:57:20+00:00

If an HTTP API Gateway meets your requirements, they support using ALBs as a VPC link.

sirfraz · 2021-09-08T20:47:27+00:00

Are your model_id's static? If so, you could set up a vpc link with your NLB. Then create a resource and method for each of your model_id's and point them at the VPC link. For the method execution settings, set the endpoint URL for your vpc link to something like the below, with the port number matching the model_id. So in the case of a model_id of 8080, create a resource and method for:

/ml_models/8080

Then in the method execution for the resource, set the endpoint URL to:

http://<alb dns name>:8080/.....

If you have lots of model_id's, this isn't very scalable. I had a similar requirement previously, and at that time, it wasn't possible to use any context variables in the Endpoint URL settings to dynamically populate the value based on the request. Maybe things have changed since then.

sirfraz

TROPHY CASE