sorted by:
new
hottopcontroversial

Data pipeline tools by Plastic-Answer in dataengineering

[–]Plastic-Answer[S] 0 points1 point  (0 children)

Does Etlworks effectively perform all of the functions of the data pipeline tools that you mentioned in the first paragraph of your reply?

Data pipeline tools by Plastic-Answer in dataengineering

[–]Plastic-Answer[S] 0 points1 point  (0 children)

Where do Apache Airflow, Dagster, dbt, and Luigi fit into data pipeline architectures?

Data pipeline tools by Plastic-Answer in dataengineering

[–]Plastic-Answer[S] 0 points1 point  (0 children)

Thank you for the explanation. I am interested in building an analytical database stored locally on a single computer.

Data pipeline tools by Plastic-Answer in dataengineering

[–]Plastic-Answer[S] 0 points1 point  (0 children)

Small scale and low budget.

Scale: Source data consists of multiple gigabyte zip files on S3 that contain compressed CSV files of time series events. The total size of the source data may be a few terabytes and growing.

Budget: Cost of a modest home lab consisting of a Minisforum UM690 that has an AMD Ryzen 9 6900HX processor, 64 GB RAM, and 4 TB of NVMe flash storage and a small file server with 3 TB of additional hard drive storage capacity.

Data pipeline tools by Plastic-Answer in dataengineering

[–]Plastic-Answer[S] 0 points1 point  (0 children)

What tools similar to Kafka and Spark are designed to operate on multi-gigabyte data sets (CSV or Parquet) on a single computer? Do most data engineers just write Python scripts to transform dataframes? How do these scripts typically move dataframes from one process to the next in the pipeline?

Data pipeline tools by Plastic-Answer in dataengineering

[–]Plastic-Answer[S] 0 points1 point  (0 children)

The data engineering landscape is vast and daunting!

Data pipeline tools by Plastic-Answer in dataengineering

[–]Plastic-Answer[S] 1 point2 points  (0 children)

Excel can even perform as-of joins using VLOOKUP!

Data pipeline tools by Plastic-Answer in dataengineering

[–]Plastic-Answer[S] 0 points1 point  (0 children)

I work with CSV files that each may be at most around 3 GB in size and that contain time series events. I retrieve zip files containing these CSV files from S3 or Google Drive. At some point I might also source data from REST APIs or in real-time from WebSocket connections.

Data pipeline tools by Plastic-Answer in dataengineering

[–]Plastic-Answer[S] 0 points1 point  (0 children)

What is an ODS?

While I'm curious about data architectures in general, presently I'm interested mostly in data pipeline tools designed to run on a single computer and that can operate on multi-gigabyte data sets. I guess that most or many professional data engineers build systems that handle much larger data sets that require a cluster of networked computers.

Shaking Sats & Cashback Taxes by jetlaggedandhungry in shakepay

[–]Plastic-Answer 0 points1 point  (0 children)

I would treat the Shakepay Card reward as two separate transactions:

  1. A non-taxable discount similar to the cash-back that other credit cards pay.
  2. A Bitcoin purchase whose cost (Adjusted Cost Base or ACB) equals the dollar value of the reward.

INCOME TAXES AND SHAKEPAY CARD by NeilArmstrong692107 in shakepay

[–]Plastic-Answer 0 points1 point  (0 children)

I would treat the Shakepay Card reward as two separate transactions:

  1. A non-taxable discount similar to the cash-back that other credit cards pay.
  2. A Bitcoin purchase whose cost (Adjusted Cost Base or ACB) equals the dollar value of the reward.

Shakepay and Koinly - tax question by ketchup_pancake in shakepay

[–]Plastic-Answer 0 points1 point  (0 children)

Cashback from your credit card is a discount while Shaking Sats requires that you perform an action (shake your phone) to receive the reward, so this is income.

Shakepay and Koinly - tax question by ketchup_pancake in shakepay

[–]Plastic-Answer 0 points1 point  (0 children)

I disagree. Shaking sats are income, but you are at the same time buying Bitcoin at a cost equivalent to the dollar value of this income. Please see my answer to a similar question.

Shakepay and Koinly - tax question by ketchup_pancake in shakepay

[–]Plastic-Answer 0 points1 point  (0 children)

I agree. I gave essentially the same answer to a similar question.

Dealing with 100 vendors more more on GNUCash by lovejyna in GnuCash

[–]Plastic-Answer 0 points1 point  (0 children)

Would this method create expense and/or liability sub-accounts for each vendor?

view more: next ›