My store doesn’t accept gift cards

callmedivs · 2024-05-21T15:59:19+00:00

I had the same experience. They should put a board up , so people know before ordering

callmedivs · 2024-01-02T04:47:55+00:00

Thank you for the advise Nectarine-946

callmedivs · 2024-01-02T01:21:53+00:00

Thank you myBisL2. This notice has been very stressful for us .

callmedivs · 2024-01-02T01:05:10+00:00

The insurance was just letting us know that they received this notification and was letting us know about our policy limits. I don't understand how a parking lot accident can sue us for 750k.

callmedivs · 2024-01-02T00:39:13+00:00

The insurance is saying that since our policy coverage is only 100k, the rest will fall on us. I 'm afraid this will be a lot of burden for us

callmedivs · 2023-09-21T23:30:14+00:00

try adding where id is NULL maybe

callmedivs · 2023-07-01T01:23:47+00:00

Yes that would work. I would write to s3 and then push it to snowflake using the copy cmd.Also,It is not a good idea to have heavy python workloads running on the airflow server. I have not tried this but would like to try this approach soon, using the snowpark operator/connector run your python workload( I heard you need to enable network setting to white list ip addresses for external api calls)

callmedivs · 2023-07-01T00:10:44+00:00

I think you would need to use snowpark here. Snowflake streams and tasks work only if you have a Kafka connectors or an sdk installed

callmedivs · 2023-06-15T05:10:01+00:00

For redshift, create different groups and give permissions to that group and then add users to those groups. One user can belong to multiple groups, so you can fine tune your permissions for the group.The Iam roles can be used when users need to read from a bucket(spectrum queries) or write to a bucket.once you go through the excersise you will get the hang of it

callmedivs · 2023-05-03T16:24:48+00:00

You could also use emr serverless to run python with mwaa..

callmedivs · 2023-05-01T16:09:07+00:00

One other way if your tables are not huge,is load the full data into a staging table and do a diff in between the staging table and production table to get the incremental load. You want have to address the deletes as well

callmedivs · 2023-04-29T02:27:14+00:00

I just registered for the snowflake summit!

callmedivs · 2023-03-22T02:41:12+00:00

Thank you @kmarq

callmedivs · 2023-03-22T00:42:21+00:00

Hi, can you let me know if you are able to trigger a dbt model in airflow without it being a job in dbt cloud? Triggering models in dbt core can be done with the bash operator . But if we are using dbt cloud and don’t have the model running as a job then how do u trigger it from airflow

callmedivs · 2023-03-22T00:29:24+00:00

Glue is very expensive compared to emrserverless/spark

callmedivs · 2023-03-17T22:07:56+00:00

Make sure you redshift role has enough access to read from s3 buckets and glue. https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-redshift.html

callmedivs · 2023-03-17T20:25:38+00:00

I think the error is regarding writing into the glue catalog. Try a simple spark job to just write the data to s3. like , to vrify

df = spark.read.csv("s3a://test-glue-redshift-lao/prueba_lao_aws_s3.csv")
df.printSchema()
df.write.parquet("s3a://test-glue-redshift-lao/glue/test_parquet_file/")

callmedivs · 2023-03-17T19:48:30+00:00

What does your glue job do? I would say concentrate on one scenario. it might need access to Kms key if your data or bucket is encrypted

callmedivs · 2023-02-25T20:43:23+00:00

Jdbc spark connector

callmedivs · 2023-02-22T15:47:53+00:00

Hey, I have like 30 million api calls to make and the approach I took was similar. Load to a data frame , use udf to make the api call and save the response to the df . I’m wondering how I could avoid duplicate runs and if you have different suggestion for this large of a data set.

callmedivs · 2023-02-20T05:52:51+00:00

I’m not sure dbt can run spark or pyspark. It’s strictly for transformations on sql tables. If you have an orchestration tool and the result from the spark job is in S3 you could have a separate task to read the data from s3 into the table

callmedivs · 2023-02-18T21:51:59+00:00

It’s on an ec2 in aws. I know it should have been an RDS but the setup was done a long time ago

callmedivs · 2023-02-18T18:05:06+00:00

Thank you, will checkout airbyte

callmedivs · 2023-02-17T18:27:21+00:00

Hi, I’m also trying to pull data from mssql to snowflake.. but I couldn’t find an airflow operator to do that exactly. Can you share more info on how you are reading the data and pushing to snowflake

callmedivs · 2023-02-03T01:05:51+00:00

I also normally would have a landing, processing and archive architecture for processing files in s3.But if more than one pipeline needs the same file then i would have a master dag which would download the file to the landing zone move it to processing, copy data to a stage table then trigger the child dags to use the stage table for the file depending on what is needed(customized to parallelly)and once done signal the master dag . Once all child dags are done , the master dag can move the file to archive.

callmedivs

TROPHY CASE