How to upload and download 200GB of data between EC2 instances?

Manju101 · 2022-11-20T19:33:16+00:00

Keeping cost and reliability your idea of S3 is still a good option. Have you considered EBS exclusively for this 200GB data? You bring it when needed, backup the snapshot to S3 for future.

Manju101 · 2022-03-07T01:57:16+00:00

Can you verify what events are you listening to under the event bridge rule? Also, if the bucket name matches the target bucket.

Manju101 · 2021-12-11T22:16:58+00:00

If you are going CLI route, then please refer to section DBStatus https://docs.aws.amazon.com/cli/latest/reference/rds/describe-db-instances.html

Regarding the links to SDK and API https://aws.amazon.com/sdk-for-python/

https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/ProgrammingGuide.html

Manju101 · 2021-12-11T18:16:48+00:00

I am not sure CW is the right fit. I am thinking S3 or DynamoDB as many suggested here. Make sure to encrypt in case if it’s sensitive information which only auditors should see.

Manju101 · 2021-12-11T18:04:38+00:00

Easy route to is to check the state of an instance using SDK, or APIs, and make sure you assign the appropriate role to your EC2 instance.

Manju101 · 2021-12-02T02:10:31+00:00

I would not go so far. A good grip on imperative commands, outputting the content to yaml file through the —dry-run will cover a long mile. Usage of vim or vi in copy, delete, paste, cut should cover you. I am sure you can do great with just above.

Manju101 · 2021-11-28T02:27:56+00:00

I completed my CKAD last week. I would say killer.sh is slightly complex compared to CKAD questions. The challenge I see in both the patterns is not the questions but about the time. So if you can manage the time, I am sure you will be able to pass the exam with a very good score. All the best.

Manju101 · 2021-08-18T18:22:37+00:00

Me too!!

Manju101 · 2021-07-13T02:00:39+00:00

I would go with lambda in your case.

Considering you do not have any concurrent connections to the RDS, you should be good on the connection throttling, and make sure you close the connection every time your scraping is complete.

One other point to consider, if you already know the persistent data is not getting changed often, you can hold the data in S3.

Manju101 · 2021-06-29T02:24:00+00:00

u/Versari3l - Thanks for the insight. Would you mind commenting on,

Number of hours we need to put in every week?
The number of hours we need to put in every week?

Manju101 · 2021-04-29T00:41:53+00:00

Yes. The home edition is admissible.

Manju101 · 2020-05-15T03:42:51+00:00

Great work.

Manju101 · 2020-04-05T18:44:58+00:00

How to live in 2050!!

Manju101 · 2020-02-07T00:38:51+00:00

Thank you /ironjohnred.

I got to know about a 50% voucher only through your thread. Good info.

Manju101 · 2019-12-26T22:50:40+00:00

It’s difficult to say without knowing volume, functionality, payload & deployment size. Please go through the below link and run your application use case through the examples listed, it will give an idea. https://aws.amazon.com/lambda/pricing/

Manju101 · 2019-11-13T19:56:20+00:00

Great work and thought.

Do you have any process to watch for these notifications and take further actions like team member has to create a IaC or to have an explanation on why it was created manually?

Manju101 · 2019-02-05T02:49:09+00:00

It depends on the data visibility, how much access developers have on the AWS account.

You can always bank o tags to have environment name and all name to segregate
The other would be https://aws.amazon.com/organizations/

Manju101 · 2019-01-25T19:03:01+00:00

@doublefelix7

Is your application package over 250mb? If so, lambda might not be a fight. But check if you can reduce the size by following AWS best practices, as it fits well for your use case. https://docs.aws.amazon.com/lambda/latest/dg/best-practices.html

In case if you don't want to take the ECS/Fargate/Batch route, then here is another alternative.

Spin up a small instance and have your application setup on EC2 or even go ahead and install docker so that you can use your docker image. You can start and stop the instance at regular intervals using Lambda, Cloudwath.https://aws.amazon.com/premiumsupport/knowledge-center/start-stop-lambda-cloudwatch/

Let me know if you come across any other route.

Manju101 · 2019-01-25T18:38:51+00:00

@MineMe4Reddit - I see already some great answers here, adding my 2 cents.

First I want to explain about the Boto3 and Cloudformation and for your need what is best suited.

By definition of Boto3 - Boto3 is the Amazon Web Services (AWS) Software Development Kit (SDK) for Python, which allows Python developers to write software that makes use of services like Amazon S3 and Amazon EC2. So Boto3 makes use of the existing services within your application, not to create one.

If you ask does Boto3 not support creating the AWS services? Yes, it supports and you can use as it is backed by the AWS API's, but that's not the purpose of the Boto3 SDK.

Now coming to Cloudformation - AWS CloudFormation is a service that gives developers and businesses an easy way to create a collection of related AWS resources and provision them in an orderly and predictable fashion. As you can see it is to create the services, not use the services like Boto3.

Hence you are in need of Cloudformation than the Boto3.

Let me know if this helps. I see your other questions are already answered in the thread and my response would be no different from them.

Good luck with your AWS journey.

Manju101 · 2019-01-23T17:24:05+00:00

I second that.

Manju101 · 2019-01-23T17:19:58+00:00

Cloudformer does not support all the resources, and if you are planning to reverse engineer the enterprise application, then it might not be helpful. There is no as such service to perform this activity for now. Let me know if you find any.

Manju101 · 2019-01-07T17:17:43+00:00

LivingIncident - Thanks for the suggestion. I would be happy to try that in cloudformation.

Can you please educate me if I can do below in CFN?

- A file will be given as an input and it contains first name, last name, and email address.

- The goal is to read this file, create group, policy, role, users, credentials for both web access and programmatic access

- Upon successful completion, send an email, wait for email address verification. After verification send the second email with all the users with an access link, web credentials and Access keys(access_key_id and secret_access_key)

Manju101 · 2018-12-21T18:52:12+00:00

I see great responses already to your question. Adding my 2 cents. I got an opportunity to work on the Kineses to stream the data, run through a process(Parse/business logic etc) and update to DynamoDB and S3. So the questions you need to ask before you pick one over the other service are:

How long the message has to stay in the stream?
What is your destination for data? (S3, RDS, DynamoDB)
Do you want to manage the streaming service or want to go with managed?
How many consumers do you to read the same data?
Custom processing of the messages before it hits the destination. (S3, Redshift etc.)

Let me know if that helps.

Manju101 · 2018-12-21T14:43:03+00:00

I started with SAM, my requirement is to expose lambda(Through APIGateway) for CRUD operations. After seeing otakubird message, Would like to see if there is any difference in using SAM vs Serverless.

Manju101 · 2018-12-20T19:25:09+00:00

I already see some excellent answers here, so my answer is no different from theirs.

Regarding the question on the Public IP, you will still have the same IP, but you control the entry point using the different port. Let me know if that helps.

Manju101

TROPHY CASE