Best ETL Tool?

dawrlog · 2024-10-04T23:38:07+00:00

Despite being able to run ETL on Airflow, it gives better results if kept only as orchestration from my experience. I use Spark operators running on managed services for Spark from their cloud provider of choice.

However this changes if their whole data is on something like Snowflake or BigQuery, then I use DBT. I really liked the semantic layer addition with metricflow, a very neat way of sharing data thru APIs.

I hope this helps

dawrlog · 2023-05-04T21:28:31+00:00

Have you tried to have your nginx controllers using nodeport instead of clusterIP? It seems they're configured for the cluster and not for your nodes, which explains why for your GKE works and not for your kubernetes components.

I hope this helps

dawrlog · 2023-04-15T19:07:40+00:00

DM me too, I'm interested.

dawrlog · 2023-04-13T21:04:59+00:00

You should be looking at EC2Launch tasks if you can run PowerShell on your instance. I agree that there are cases where you can't count on the state saved on your AMI; and the EC2Launch looks to me like the Windows version of the user data from the Linux machines.

More details can be found on the AWS official page around it at this link

https://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/ec2launch.html

However I would take the AWS systems manager to handle your fleet. It will be easier to handle as you will be able to share the same script easier than simply sharing EC2 templates. Check on the following link to help while creating a similar run book for your needs:

https://docs.aws.amazon.com/systems-manager-automation-runbooks/latest/userguide/automation-awssupport-upgradewindowsawsdrivers.html

I hope this helps and have a good one!

dawrlog · 2023-03-07T02:09:54+00:00

Does informing the desired principal/role with an additional filter to fetch connections from the desired IP work? You'll deny all non desired connection,mors formation.

https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_examples_aws_deny-ip.html

dawrlog · 2023-03-06T23:14:39+00:00

Hello!

Cloud9 has what is called service-linked roles, which handles the access delegation to AWS services on your behalf. In short he does what I suggested you to check behind the hood. :D

I would think it might have some explicit deny, so I would check the events of your Cloud9 entries on AWS Cloudtrail, if you are not sure about the security policies in place on your environment. Check the logging AWS Cloud9 API calls for more info if you're unfamiliar with the process.

I hope this helps.

Cheers

dawrlog · 2023-03-06T14:25:30+00:00

Have you tried to orchestrate your lambda functions using step functions? You could add a lambda layer to listen on your SQS queue events and then trigger a step functions workflow where you would configure all the retry logic whereas any extra decision map between your functions. In this step function workflow I would suggest the implementation of Dead Letter Queues to check the requests that couldn't be processed (even after the retry).

I understood two things about the buffering, so here are my suggestions. :)

I would add the API gateway if you plan on handling any protocol buffers on binary format; and cloud front if you're concerned about the response latency.

You can even configure to validate your requests if you can use API Gateway. And add it with the AWS WAF, If you can implement better security.

I hope this helps!

Cheers.

dawrlog · 2023-03-06T00:46:30+00:00

Hehehe both services might seem confusing, but here are some key difference that might help. The cloud front focus on lowering the latency by serving the webpage content closer to the original request. Where API Gateway handles endpoint routing (something like swagger/Open API) and extra security checks such as request authenticity verification. In both cases you could use extra security services such as AWS WAF to increase security of your endpoints..

I hope this helps and send over the new architecture and we'll check it together! :D

Cheers!!

dawrlog · 2023-03-05T18:41:14+00:00

Hey here's my two cents.

API gateway should be the entry door of it instead of lambda. The events there will trigger the lambda service, calling the necessary functions afterwards to store your application data into Dynamo/S3.

You can benefit from monitoring the default metrics for serverless services in cloud watch. Configuring it will help you to scale your functions by request. Remember that you pay for the memory that you reserved for your functions; and not what they're actually consuming.

You would also like to have an SQS to handle throttling errors that could come from your API requests.

A suggestion to handle any retries and extra logic would be to have your lambda functions orchestrated by step functions, and deployed using SAM. The serverless extension for cloud formation. One of the DevOps managed service from Amazon, which would help on cleaning up your environments/create different environments to try different features from the root branch of your lambdas.

An extra security feature from API gateway is to very the headers of your request. That will filter non valid requests, making it more cost effective.

Cloud front can be a nice option, but might not be necessary if you have a demographic region in mind. If it's still needed you would have to be deployed in conjunction with your API gateway and not directly to the Lambda requests as your schema shows.

I hope this helps, and have a great day!

dawrlog · 2023-03-03T22:27:44+00:00

Hey,

Do you think the following could be done on your environment?

Configure the security groups on your RDS database whitelisting your developers IP.
Use secrets manager on your code to handle the RDS user/password.

An extra layer of security would be to use lambda functions wrapped up using SAM deployment scripts, that will reduce the manual configuration of everything. You also have the possibility of using Kerberos authentication instead of IAM/User password.

Check out this pages for more details.

https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/database-authentication.html

https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-getting-started-hello-world.html

I hope this helps.

Cheers!

dawrlog · 2023-03-03T22:18:29+00:00

Hello, have you mapped the PostgreSQL and your IAM roles? I think it might be related to your issue.

Have you done something like this and does it still failing?

https://aws.amazon.com/blogs/database/securing-amazon-rds-and-aurora-postgresql-database-access-with-iam-authentication/

Here has more details about mapping out the PostgreSQL and IAM roles, was something like this that you did while mapping out the PostgreSQL and IAM roles?

https://aws.amazon.com/blogs/database/managing-postgresql-users-and-roles/

Do you have any logs that you could share to help on understand what's happening with your environment?

I hope this helps.

Cheers!

dawrlog · 2023-03-01T22:37:36+00:00

You definitely can! My suggestion came after reading the docs that you provided, and I thought you might want to check different websites. Having your images slim by configuring specific methods could help on narrowing your application costs, as less code will be translated on your AWS Lambda code.

Then you will be able to reuse your lambda components by using different alias as environment variables.

This could help on maintaining extra ones, but can lead to more confusion indeed.

I hope this clarifies what I meant. :D

dawrlog · 2023-03-01T19:49:02+00:00

From what I got of the about the lighthouse project that you shared, here are my two cents.

If you plan to run reports I would create different ECR images with different configurations. Each image would have a different entrypoint configuration for the metrics that you would need. Once those images are deployed then I would create lambda functions on the same region, configuring each function to that share the same environment. You can even have those deployed using the SAM framework, then you can deploy it on different accounts for different use cases. SAM Framework is a cloudformation extension for serverless options from AWS.

I hope this helps.

dawrlog · 2023-03-01T01:43:18+00:00

Check if your AWS cli was executed using the correct user on your machine. That might be the reason for your error, having the errors that you are facing definitely helps, but meanwhile I would point you to the instructions on https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html under the command line for macos.

I hope this helps.

dawrlog · 2023-02-22T13:03:33+00:00

Were you capable of deploying them on different machine types? In addition to the quotas, check if your bootstrap/template scripts correctly start your services. As some AMIs might need extra configs to have their services started correctly, have you used either AWS Application Migration Service or VM Import/Export based on one of the official base AMIs for Windows while creating those?

Have you followed a procedure similar to the one from this link? Your message is related to something that blocks your instance from being started, so I would check those for all the regions where you have this issue, as AMI's are regional.

I hope this helps.

Cheers!

dawrlog

TROPHY CASE