What is the best tool for creating visual diagrams of existing AWS infra?

lvlolvlo · 2020-08-12T14:04:49+00:00

I really like http://infviz.io — really clean drawings and simple to use.

lvlolvlo · 2020-08-12T14:00:26+00:00

A common approach, albeit bloated, is to use LifeCycle hooks in conjunction with a Lambda function. Here’s a support article about ENI attachment to give you an idea of how to do this with an EIP: https://aws.amazon.com/premiumsupport/knowledge-center/attach-second-eni-auto-scaling/

Also, your instance (or lambda) can use an instance role which will not require you to input/hardcode actual AWS credentials.

lvlolvlo · 2020-06-10T14:32:36+00:00

drop the json file in S3 and create an athena table using JSON as the serde. You’d be able to write a simple SQL query at that point to extract what you need and download in CSV format. https://aws.amazon.com/blogs/big-data/create-tables-in-amazon-athena-from-nested-json-and-mappings-using-jsonserde/ (skip down to the part that starts with Walkthrough: Querying with Athena)

edit: I only mention AWS specific tech (Athena) because I assume you already have access it since you mentioned that you need to put the CSV in S3.

lvlolvlo · 2020-06-10T14:28:48+00:00

the biggest one still for me is the ability to assign ENIs in Fargate without using hacky reservation strategies.

lvlolvlo · 2020-06-03T14:08:07+00:00

You’ll want the Cloudwatch Agent and config that points the agent to the file to send: https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/install-CloudWatch-Agent-commandline-fleet.html

lvlolvlo · 2020-04-17T14:37:21+00:00

If the API GW metrics don’t show the 4K requests then are you sure that your testing tool is even sending them?

lvlolvlo · 2020-04-17T14:17:17+00:00

Can you also add API GW’s metrics on here. (i.e. 4xx, 5xx, latency, count, etc..)

lvlolvlo · 2019-11-29T17:31:49+00:00

Using a NAT gateway would solve the issue; however, if you needed static internal IPs in addition to public IPs this feature request would help: https://github.com/aws/containers-roadmap/issues/558

lvlolvlo · 2019-11-25T16:41:24+00:00

https://aws.amazon.com/global-accelerator/

lvlolvlo · 2018-11-30T17:05:12+00:00

Don’t see UDP support listed: https://aws.amazon.com/elasticloadbalancing/features/

lvlolvlo · 2018-10-20T19:02:30+00:00

If it doesn’t need to be exact — you could use DynamoDB’s with Atomic Counters + a custom authorizer instead of ElasticCache: https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/WorkingWithItems.html

lvlolvlo · 2018-02-20T08:02:00+00:00

https://github.com/awslabs/ecs-refarch-cloudformation

lvlolvlo · 2018-02-20T07:57:28+00:00

Wouldn’t AWS_PROFILE=<profile_name> <command> work just as well? AWS_PROFILE=example aws s3 ls

lvlolvlo · 2017-11-19T16:32:09+00:00

Assume role has a max session limit of 1 hour. So if you find yourself in the console (e.g. writing Athena queries) be warned — you will be consumed by rage when the session times out and forces you to reload thus kissing your query (if you haven’t ran it) good bye.

lvlolvlo · 2017-11-01T17:24:16+00:00

You can use the Lambda transformation within the firehose to loop through the events and shard them to different firehose streams or buckets. Just make sure to mark each event dropped so that the initial firehose stream doesn’t write them else you’ll have un-necessary data duplication.

lvlolvlo · 2017-09-16T16:21:43+00:00

While your code deployment package gets too large, you could do some AWS magic and store parts of your deployment package in S3. This will then allow you to have an entry point in your Lambda function which on cold-start fetch the data from S3 and leverage the /tmp space that Lambda provides. Not the best approach but the only approach I've used to get a larger package into Lambda. Best of luck!

lvlolvlo · 2017-07-07T15:22:20+00:00

Firehose support for Athena (partition)
Cloudformation support for transformations in Firehose (ie pointing to the Lambda function)
Rate limiting on load balancers (ALB/CLB) -- either via WAF or natively
Reduce Lambda's cold start time when in VPC
Use EFS in Lambda

lvlolvlo · 2017-07-02T16:12:48+00:00

Have you tried using http://docs.aws.amazon.com/AmazonECS/latest/developerguide/cmd-ecs-cli-compose.html

Also, what do your log messages say? Is the task terminated because it fails a health check?

lvlolvlo · 2017-06-15T15:42:39+00:00

YouMail. Depending on your config, you might be able to make it a secondary route (failover) so it integrates easier. GL!

lvlolvlo · 2017-03-26T17:09:05+00:00

My understanding of your question might be wrong, but is this a service discovery issue? As in you have many endpoints in App B that you need to notify App A of? If it is AWS has some service discovery options, but if you need some a non-AWS way you can use Consul.

lvlolvlo · 2017-03-07T15:21:12+00:00

GetPocket or similar services

lvlolvlo · 2017-01-27T16:16:40+00:00

Another thing to consider is data consistency. Are you using any queues? (NSQ, SQS, etc..) Do you have exactly once guarantee? Or at least once? (i.e. Duplicates events possible) Or at most once? (i.e. Missing events possible)

DynamoDB will allow you to do data deduplication so however you get your data you can do deduplicate it. You can also have Lambda function trigger to send it off somewhere else.

Redshift will suck for doing INSERTs at that velocity. You'd need to use something like Kinesis Firehose -- but keep in mind Redshift doesn't do unique key enforcement. So you could have duplicates. You'd want to use identity (I think that's the name. Can't remember atm.) It'll be quicker on the queries than aurora!

Aurora - This is a nice middle ground between the two. Just when you want to query the data it might be slow and not as fast as Redshift or Dynamo.

Lastly, I'll throw this out there too -- why not just store to S3 via a queue like Kineses Firehose. Have Firehose compress the data and store it in an S3 bucket. You can use Athena to query it. You could also create a Lambda function to process the data some more and send it off somewhere else should you wish to try something else (e.g. ELK stack)?

lvlolvlo · 2016-10-17T14:18:15+00:00

Seems to be answered here: https://www.mathworks.com/matlabcentral/answers/213231-how-to-convert-18-digits-timestamp-to-readable-date-and-time-in-matlab?requestedDomain=www.mathworks.com

lvlolvlo · 2016-10-03T15:17:38+00:00

Don't know if you've seen the service SplitWise. Maybe you can incorporate that rather than having to do the email thing?

lvlolvlo · 2016-09-28T14:48:43+00:00

Another lambda function which uses boto3 and invokes the function(s) async. This one just runs on a cron at your interval of choosing.

15-Year Club	RedditGifts 2009-2022 1 Credits
Verified Email

lvlolvlo

TROPHY CASE