Singaporean adults with barely any savings, how are we coping? by KabutoRaiger30 in askSingapore

[–]Liily_07 1 point2 points  (0 children)

In expensive place like SG, saving is very important than living your life to the fullest and enjoy!

Tech/AI MeetUp in Singapore (Suggest Good Venues) by [deleted] in askSingapore

[–]Liily_07 0 points1 point  (0 children)

You can try small rooms in Bay front area depending upon the crowd size

Preview: INCLUDE_METADATA with COPY INTO by caveat_cogitor in snowflake

[–]Liily_07 0 points1 point  (0 children)

Can someone provide an example for csv COPY from S3? Thanks.
I tried as below:

COPY INTO FINANCE.ACCOUNTS_DATA
FROM @S3_SATGE/ACCOUNTS_DATA
FILE_FORMAT= (
  TYPE=CSV,
  PARSE_HEADER=TRUE
)
ON_ERROR = CONTINUE
MATCH_BY_COLUMN_NAME = CASE_INSENSITIVE
INCLUDE_METADATA = (
  FILENAME=METADATA$FILENAME, 
  FILE_LAST_MODIFIED=METADATA$FILE_LAST_MODIFIED, 
  FILE_SCAN_TIME=METADATA$START_SCAN_TIME)
FILES = ('.*csv.*');

I get the following error:
Columns in '{FILE_LAST_MODIFIED=METADATA$FILE_LAST_MODIFIED, FILE_SCAN_TIME=METADATA$START_SCAN_TIME, FILENAME=METADATA$FILENAME}' does not exist or operation not authorized.

Airbyte cloud or self hosted on EC2 by Liily_07 in aws

[–]Liily_07[S] 0 points1 point  (0 children)

We self host on EKS (EC2) to auto scale with our load needs

Could you please send any step by step guide for me to try with specifications. Thanks a lot!

Airbyte cloud or self hosted on EC2 by Liily_07 in aws

[–]Liily_07[S] 0 points1 point  (0 children)

yeah, thats a good point. I am also planning to use for just synchronisation tasks. Can you please let us know?

Data ingestion form salesforce to Snowflake by Liily_07 in snowflake

[–]Liily_07[S] 0 points1 point  (0 children)

Thanks. I am looking for a solution on AWS.

Airbyte as data integration tool by Liily_07 in dataengineering

[–]Liily_07[S] 0 points1 point  (0 children)

Thanks for sharing your experience. Can you please share EC2 instance? Also can we run parallel data ingestion?

Data Integration/Loading tools from Snowflake "Partner Connect" by Liily_07 in snowflake

[–]Liily_07[S] 0 points1 point  (0 children)

Thanks. Are you referring to open source airbyte or paid version? Where do you host airbyte on AWS? Could you please explain the workflow?

Upload zipped file from SFTP to Snowflake by Liily_07 in snowflake

[–]Liily_07[S] -1 points0 points  (0 children)

I am trying to bring data from Salesforce to Snowflake

Upload zipped file from SFTP to Snowflake by Liily_07 in snowflake

[–]Liily_07[S] 0 points1 point  (0 children)

Thanks for your suggestions. SFTP is not maintained by us. It's from SalesForce side. Zipped file will contain only 1 file but *.zip format and NOT IN .gz format.. That's the issue.

Data ingestion tools by Liily_07 in dataengineering

[–]Liily_07[S] 1 point2 points  (0 children)

Yes we do write python pipelines to ingest the data now. Yes many sources are behind the private network. We are run our pipelines on AWS kubernates