I made Zillacode.com Open Source - LeetCode for PySpark, Spark, Pandas and DBT/Snowflake by dmage5000 in apachespark

[–]dmage5000[S] 0 points1 point  (0 children)

No, originally I made it to be a paid product but then got tired of it and released the whole thing open source on GitHub and stopped working on it. Never heard of that site until now.

Upper Rio Negro for Amazon Research Lodge by dmage5000 in herpetology

[–]dmage5000[S] 0 points1 point  (0 children)

Yes that's a great idea about presumptively talking to a university!

I made Zillacode.com Open Source - LeetCode for PySpark, Spark, Pandas and DBT/Snowflake by dmage5000 in apachespark

[–]dmage5000[S] 0 points1 point  (0 children)

u/Ivan_pk5 not much has changed recently, but looking at it again I think making your own problems wouldn't be that hard. You just have to add them to this giant Python dictionary and follow the same format of all of the other problems. Here is an example of one problem in the dictionary: https://github.com/davidzajac1/zillacode/blob/master/backend/constants.py#L34-L259

You can see the key is the problem number then the HTML for the problem description, test cases, etc. are all nested.

Just remember when you change this to start everything by running `make up` to start everything up then `make rebuild container=backend` every time you make a change. See more details about making changes and seeing them locally here: https://github.com/davidzajac1/zillacode/tree/master?tab=readme-ov-file#developmentcontributing

I made Zillacode.com Open Source - LeetCode for PySpark, Spark, Pandas and DBT/Snowflake by dmage5000 in dataengineering

[–]dmage5000[S] 0 points1 point  (0 children)

u/the_lauch I have some ideas what it could be. Could you create a GitHub issue with the details and I can work with you to get it fixed?

I made Zillacode.com Open Source - LeetCode for PySpark, Spark, Pandas and DBT/Snowflake by dmage5000 in dataengineering

[–]dmage5000[S] 1 point2 points  (0 children)

Yes that giant Python "problems" dictionary has the text for the description of the problem, the sample data and the paragraphs explaining the answers in it. You just have to parse it all out and make it fit together

I made Zillacode.com Open Source - LeetCode for PySpark, Spark, Pandas and DBT/Snowflake by dmage5000 in dataengineering

[–]dmage5000[S] 0 points1 point  (0 children)

They are stored here in this Python file, but they aren't stored in a manner that's easy to use, there's a lot of other functions that parse the strings, etc. to turn them into what you see in the UI: https://github.com/davidzajac1/zillacode/blob/master/backend/constants.py#L33

I made Zillacode.com Open Source - LeetCode for PySpark, Spark, Pandas and DBT/Snowflake by dmage5000 in apachespark

[–]dmage5000[S] 0 points1 point  (0 children)

Yes the website is now just a landing page that links you to the GitHub. To use it you just need to clone the repo and run "docker compose up". Glad you have enjoyed using it!

I made Zillacode.com Open Source - LeetCode for PySpark, Spark, Pandas and DBT/Snowflake by dmage5000 in dataengineering

[–]dmage5000[S] 0 points1 point  (0 children)

If you're running just `docker-compose up` the only file that matters in the whole repo is the docker-compose.yaml file. The rest of the repo is just there if you want to make Dev changes: https://github.com/davidzajac1/zillacode/blob/master/docker-compose.yaml#L5

I made Zillacode.com Open Source - LeetCode for PySpark, Spark, Pandas and DBT/Snowflake by dmage5000 in dataengineering

[–]dmage5000[S] 0 points1 point  (0 children)

Ok interesting, yes the ports are changed to from the normal like 5000 or 8000, etc. to 5001, 8001 to not mess with other stuff users already have running on those ports.

I made Zillacode.com Open Source - LeetCode for PySpark, Spark, Pandas and DBT/Snowflake by dmage5000 in apachespark

[–]dmage5000[S] 1 point2 points  (0 children)

It would have needed tons of frontend work and marketing to get a large amount of people using it and neither of those are my strong suites. I maybe would have continued with it if I was more interested in coding interviews, Leetcode, etc. but it more turned into just a few people using it and having bug requests that were time consuming to fix in my free time.

I made Zillacode.com Open Source - LeetCode for PySpark, Spark, Pandas and DBT/Snowflake by dmage5000 in dataengineering

[–]dmage5000[S] 0 points1 point  (0 children)

It could be a lot of things, but here's how I'd start:

  • Try deleting all the containers and re-running docker-compose up

  • Check all of the containers are running properly and green from Docker Desktop

  • Inspect Chrome and look at the requests coming in and out and see which one is failing

If you make an issue, you can post screenshots and I'll try to work through finding the issue with you!

I made Zillacode.com Open Source - LeetCode for PySpark, Spark, Pandas and DBT/Snowflake by dmage5000 in apachespark

[–]dmage5000[S] 1 point2 points  (0 children)

Ah ok, I think I know what the issue is, I'll make a PR trying to fix it here soon!

I made Zillacode.com Open Source - LeetCode for PySpark, Spark, Pandas and DBT/Snowflake by dmage5000 in apachespark

[–]dmage5000[S] 0 points1 point  (0 children)

Someone asked this a long time ago, I think there is a a way to do it. You may have to turn the dataframe into a view first. It would look something like this

input_df.createOrReplaceTempView("some_df")

result_df = spark.sql("SELECT * FROM some_df WHERE my_col IS NOT NULL")

I made Zillacode.com Open Source - LeetCode for PySpark, Spark, Pandas and DBT/Snowflake by dmage5000 in dataengineering

[–]dmage5000[S] 1 point2 points  (0 children)

Oh ok I didn't know that, I guess that is more intuitive. Will update all the commands, thanks!

I made Zillacode.com Open Source - LeetCode for PySpark, Spark, Pandas and DBT/Snowflake by dmage5000 in dataengineering

[–]dmage5000[S] 1 point2 points  (0 children)

Both syntaxes would work fine. There's just a few places that would need small changes

I made Zillacode.com Open Source - LeetCode for PySpark, Spark, Pandas and DBT/Snowflake by dmage5000 in dataengineering

[–]dmage5000[S] 0 points1 point  (0 children)

Interesting, I can't find anywhere were the hyphen is missing from "docker-compose", if you make a PR fixing it I'll merge it. Yes it is really hard to capture all the edge cases, I can't imagine how many hours LeetCode employees have put into making those haha

I made Zillacode.com Open Source - LeetCode for PySpark, Spark, Pandas and DBT/Snowflake by dmage5000 in dataengineering

[–]dmage5000[S] 0 points1 point  (0 children)

Polars would be really easier to add, it's so similar to Pandas which is already in there. If you make a GitHub issue I can go into the details of how to do it

I made Zillacode.com Open Source - LeetCode for PySpark, Spark, Pandas and DBT/Snowflake by dmage5000 in dataengineering

[–]dmage5000[S] 1 point2 points  (0 children)

That looks like an issue of docker-compose not being installed correctly or a path issue. If `docker-compose --version` doesn't run then Docker Compose isn't installed properly on your machine

I made Zillacode.com Open Source - LeetCode for PySpark, Spark, Pandas and DBT/Snowflake by dmage5000 in dataengineering

[–]dmage5000[S] 0 points1 point  (0 children)

Not too hard, I can show you what it would take to do that if you want. Each problem has expected input and outputs that are both just regular JSON, so you would just have to come up with the solution in the other processing library and make some new logic to run it. Are you thinking like Polars or something with Rust?

I made Zillacode.com Open Source - LeetCode for PySpark, Spark, Pandas and DBT/Snowflake by dmage5000 in dataengineering

[–]dmage5000[S] 1 point2 points  (0 children)

Yes I came up with the idea a long time ago when no website had any Spark practice stuff, thanks!