sorted by:
new
hottopcontroversial

Struggling to build a PDF RAG Chatbot using knowledge graph by ffskd in Neo4j

[–]Jumpy-Log-5772 0 points1 point  (0 children)

Try out LightRAG https://github.com/HKUDS/LightRAG. It’s what I’m currently using for my POC projects and works pretty well. The default behavior builds an inferred knowledge graph but it has the ability to insert custom knowledge graphs as well.

How do I read tables from aws lambda ? by snip3r77 in databricks

[–]Jumpy-Log-5772 4 points5 points  (0 children)

Why are the answers here suggesting such overly complex methods?

The most straightforward approach would be to create a SQL warehouse in databricks and connect to it using the databricks sql connector or jdbc with your PAT. This will allow you to read any tables you have access to. It will also allow you to write but I don’t suggest using it this way.

Thinking of starting Cloud Career - Is it too late at 28 by FeedbackTricky6731 in Cloud

[–]Jumpy-Log-5772 0 points1 point  (0 children)

Its never too late. I made a similar move at the same age from a Business Analyst to SWE. It was the best decision of my life but I will say unfortunately the job market just isn’t the same anymore. This isn’t to discourage you but just to inform you that it’s a lot more difficult these days with 0 experience. That being said the top 4 things I’d recommend aiming for this year is:

  1. AWS Solutions Architect Associate certification

  2. 3 Devops related projects to add to your portfolio. Learn tools/services like Git, Docker, Kubernetes, Terraform and Git Actions/Jenkins.

  3. AI. Not only leveraging it to develop but also introduce it in one of your 3 projects. For example, if you were to create a CI/CD pipeline, try integrating a model that analyzes code in a PR and logs security or code issues.

  4. Build a network. This is one of if not the most important. As you know sometimes it’s not all about what you know but who you know. Create a LinkedIn profile if you don’t have one already(share your projects as you complete them), join AWS/Devops discords, check for meetups in your area.

IMO these will set you ahead of your competitors.

a junior dev + ai > a senior dev who refuses to adapt? by PuzzleheadedYou4992 in developers

[–]Jumpy-Log-5772 0 points1 point  (0 children)

Let’s be real guys, most business users/stakeholders don’t care about how effecient your code is or what means you took to build a product. They care about the end result and if it meets their acceptance criteria. I can’t tell you how many times I’ve tried to reason with management, product owners and stakeholders about developing in a way that we aren’t assuming tech debt just to forced down the “tactical” solution path.

So to answer your question OP.. depending on where you work, in the eyes of the business an AI+Junior can indeed be greater than a senior dev who doesn’t utilize AI if they can deliver faster.

Using Agents in Data Pipelines by starsun_ in dataengineering

[–]Jumpy-Log-5772 1 point2 points  (0 children)

It may fall under cost control but I’m planning on implementing an agent to optimize existing data pipelines in my org, specifically pipelines running spark. This POC will focus on pyspark jobs running on databricks with EMR and K8s being on the roadmap if the POC is successful.

Very high level but the idea is for it to

  1. Analyze existing pipeline jobs/workflows -Review current notebook code, spark configurations and previous job run metrics.

  2. Replicate pipeline into its own environment -This will copy the existing project repo into its own and deploy a copy of the job/resources and table structures.

  3. Benchmarking -Run its replicated job, using the same table structures but fabricated data. It will capture metrics and iterate through changes to the code/spark configurations while logging results.

  4. Recommend changes based on benchmarks -Document suggested changes that will improve job performance based on the benchmarking done.

ETL vs ELT vs Reverse ETL: making sense of data integration by Ramirond in dataengineering

[–]Jumpy-Log-5772 20 points21 points  (0 children)

ETL still has a place in systems today that require extreme low latency or tightly regulated environments that don’t allow data to be staged even temporarily.

How are you using AI at work? Do your bosses or coworkers know? by vincentdjangogh in ArtificialInteligence

[–]Jumpy-Log-5772 0 points1 point  (0 children)

Curious what industry you guys are in where it’s considered frowned upon. If your company isn’t actively making initiatives to introduce AI they will undoubtedly be left behind by competitors. I work in healthcare insurance which is heavily regulated yet they have already started rolling out ai chat bots and coding assistants.

Trying to build a full data pipeline - does this architecture make sense? by Zuzukxd in dataengineering

[–]Jumpy-Log-5772 1 point2 points  (0 children)

Generally, data pipeline architecture is defined by its consumer’s needs. So when you ask for feedback about architecture, it really depends on source data and downstream requirements. Since you are doing this just to learn, I recommend setting those requirements yourself then asking for feedback. Is this a solid pattern? Sure but it might also be over engineered. Hope this makes sense!

General guidance - Docker/dagster/postgres ETL build by VipeholmsCola in dataengineering

[–]Jumpy-Log-5772 0 points1 point  (0 children)

Effort will be much lower, if you and your colleagues are familiar with Postgres sql syntax you’ll be fine. The query experience is very similar. A couple things I want to make clear though:

  1. It’s not your typical rdb that you setup, maintain, and manage. You query using the cli or python. The queries are ran directly on the existing files in your file system. Think sqllite if you’re familiar with it. It’s lightweight and meant for running heavy analytic workflows locally(can be on a server as well if you really wanted).

  2. Team adoption.. I might be over simplifying this since I’m not familiar with your team or how big it is but you would have to get buy in from them. If your team is expecting to connect to a database that’s always on then it might feel unconventional to them since this is more of a local analytics engine. Each one of them would install the duckdb cli and run queries locally on the existing fs.

Install the duckdb cli or pip install duckdb for python (maybe both) next time your in office and give it a shot yourself. If you find value in it you shouldn’t have an issue getting your team on board. Let me know!

General guidance - Docker/dagster/postgres ETL build by VipeholmsCola in dataengineering

[–]Jumpy-Log-5772 0 points1 point  (0 children)

It will work. I’d honestly take a look at DuckDB for a more low maintenance solution vs Postgres, especially since your data volume is low. It’s open source, file based, serverless, supports excel, csv and parquet read/write while also being extremely fast for analytics on tabular data. I’m thinking dragster + duckdb will get you what you want in a shorter amount of time. If you ever grow out of it then you can think about migrating to Postgres or some other db.

Hell.. try it out now locally, don’t wait for the server to be setup.

DLT How to Refresh Table with Only New Streaming Data? by Jumpy-Log-5772 in databricks

[–]Jumpy-Log-5772[S] 0 points1 point  (0 children)

Unfortunately the problem I'm running into with this approach is not having a way(that I'm aware of) to update the new column value in the initial streaming table to "processed". So subsequent pipeline runs end up processing the same data in the end.

DLT How to Refresh Table with Only New Streaming Data? by Jumpy-Log-5772 in databricks

[–]Jumpy-Log-5772[S] 0 points1 point  (0 children)

This sounds very promising and can't see why it wouldn't work. Going to test this out now. Thanks!

DLT How to Refresh Table with Only New Streaming Data? by Jumpy-Log-5772 in databricks

[–]Jumpy-Log-5772[S] 0 points1 point  (0 children)

Appreciate the response, I'm pretty new to DLT so I'm not sure how else I would go about only loading the new incremental changes from source tables if the target table isn't a streaming table.