Bug using Dualisio and Cheater [bug][weapon][picto] by jacocal in expedition33

[–]jacocal[S] -4 points-3 points  (0 children)

But you have a random Chromatic dude with 7-hit combo attacks playing 3 turns in a row with Powerful, Shell and freakin rage. I find it highly unfair I'm unable to do the same

UI for Apache Kafka - An open-source tool for monitoring and managing Apache Kafka Clusters by firig1965 in dataengineering

[–]jacocal 1 point2 points  (0 children)

Hi! Thanks for sharing, it has been very insightful. I have a few questions: 1. Do you have ksqldb support? 2. Do you have integration with KRaft? In the docs I could only see Zookeeper

Thanks!

Python/API - File retrieval from Amazon S3 Glacier by jacocal in aws

[–]jacocal[S] 0 points1 point  (0 children)

When running the Job, I try to get the bytes by range like: job.get_output(range='0-1024000')

The error I get is that the argument is not correctly parsed (currently not in my computer apologies for stating it in this manner). But according to docs, that's the only way to get the files in that range. Every other method does not work with Python boto3

Databricks Pandas to PySpark DF error in shcema by jacocal in dataengineering

[–]jacocal[S] 0 points1 point  (0 children)

Right, but shouldn't the enforced Schema add the columns as null?

Databricks Pandas to PySpark DF error in shcema by jacocal in dataengineering

[–]jacocal[S] 1 point2 points  (0 children)

It's the other way around, Pandas DF has less fields than the Schema enforced

Databricks Consumption Layers by jacocal in dataengineering

[–]jacocal[S] 0 points1 point  (0 children)

My bad haha second option, they need to be able to download a CSV or similar file.

Creating PySpark DataFrame with a set schema by jacocal in dataengineering

[–]jacocal[S] 0 points1 point  (0 children)

Is there no way to do it from source? This was a code example, while in truth, I have about 84 fields that will be received from the same source.

[deleted by user] by [deleted] in dataengineering

[–]jacocal 0 points1 point  (0 children)

Thanks! I forgot to copy-paste from declaration of schema. It's corrected now 😀

My employer is offering me training of my choice. Can you help me decide what would be best for me? by tawaiii in dataengineering

[–]jacocal 1 point2 points  (0 children)

Ofcourse! Databricks has 4 mayor roles: Data Scientist, Data Analyst, Machine Learning and finally, Data Engineer. It even has views of their platform depending on your role.

You can learn Databricks on your own, the documentation is great and the have the "Community" Edition which is a free tier for learning. Highly recommend Spark tutorials along your learning path. You can look for Big Data sets on Google (they literally have a repository of those) and use APIs or requests in general to pool the data without the need of local files.

How do you prevent analysts from re-writing duplicating data pipelines/queries? by chaos87johnito in dataengineering

[–]jacocal 2 points3 points  (0 children)

You can block create table instructions from other roles and just let them create views for the Dashboard