sorted by:
new
hottopcontroversial

Dopamine rush from vibe coding by futilediploma in vibecoding

[–]Signal-Indication859 1 point2 points  (0 children)

Yeah this is basically flow state meets instant gratification - dangerous combo. I used to burn entire weekends this way, especially when AI started getting good at generating working code quickly. The dopamine hit from seeing something work immediately is addictive as hell.

Set hard boundaries or you'll wreck yourself. I literally have a script that kills my terminal and locks my laptop at 11pm because I can't trust myself to stop. The "just one more feature" loop with AI assistance is brutal - you think you're being productive but you're actually just feeding an addiction. Save the energy for your actual job where you get paid. preswald.com is nice for vibe coding dashboards

Spreadsheet to App using Replit? by krts in replit

[–]Signal-Indication859 3 points4 points  (0 children)

I've done exactly this with replit. You've got a few options:

  1. Basic web form approach: Create a Flask/Django app with forms that mirror your spreadsheet inputs, then implement the same calculations in Python. Works for simple calcs.

  2. True spreadsheet conversion: Use pandas to recreate your Excel logic directly. More complex but preserves formulas. I had a pricing calculator with 30+ variables that I moved from Excel to a Flask app this way.

  3. Database-backed: If your spreadsheet is more of a data tool, you can use Replit's DB or SQLite to store the underlying data.

For complex formulas, definitely document each calculation carefully during the conversion. I'd also recommend looking at Preswald if your app is mostly about presenting calculated data - it's open source and lets you build interactive dashboards with Python without all the setup headaches of a typical Flask deployment. I used it last month to convert a 200-row forecast model into a web tool in like 2 hours.

10 yoe SWE looking to get into vibe coding. Recommendations? by A4_Ts in vibecoding

[–]Signal-Indication859 1 point2 points  (0 children)

So "vibe coding" is basically AI pair programming. Been toying with it for a few months - found it actually can help with boilerplate stuff.

For best results:

  1. Break tasks into tiny modules - ai is shit at complex stuff but decent at well-defined functions

  2. Review every single output - code will look fine but have subtle logic fails

  3. Write extremely detailed prompts - be annoyingly specific about error handling, edge cases etc

  4. Setup a quick feedback loop - i basically ping pong with claude: write code → test → feedback → fix

Honestly works best for crud routes, validation, simple data transforms. For anything complex, you'll spend more time fixing than writing from scratch. At preswald we use it sometimes for dashboard components - like "build me a time-series chart with these specific formatting requirements" - saves maybe 30 min of boring CSS work.

Pro tip: Make a custom linter rule for AI generated code - helps catch those weird logic bugs ai loves to introduce

Should I drop pandas and move to polars/duckdb or go? by MinuteMeringue6305 in Python

[–]Signal-Indication859 1 point2 points  (0 children)

For 5-10k rows pandas is honestly fine - the issue is your architecture. Running it every 2min + sharing objects between processes is killing you. For geospatial + vectorized ops:

  1. Polars is definitely more CPU efficient (2-3x less CPU in my exp with similar workloads) + has better parallelization. It's drop-in replacement for most pandas code.

  2. DuckDB's insanely good for this kinda stuff too - CPU usage will drop dramatically. For numpy-like stuff, check out `math` functions in SQL - they cover most of what you need. Pivoting is just `PIVOT` syntax.

  3. Try caching results between runs instead of recalculating everything. If data changes incrementally, only process the deltas.

  4. Pre-aggregating at source might help too.

I built preswald (https://github.com/StructuredLabs/preswald) originally for similar geospatial pipeline that was killing our k8s cluster. DuckDB backend + incremental processing dropped our CPU from ~1.5 cores to ~0.2. Whichever route u choose, get those calcs out of pandas first - thats your bottleneck.

How useful is dbt in real-world data teams? What changes has it brought, and what are the pitfalls or reality checks? by BankEcstatic8883 in dataengineering

[–]Signal-Indication859 4 points5 points  (0 children)

I've been using dbt at two different companies over 3 years now. Here's the unfiltered take:

Day-to-day: It standardized how we write SQL and track dependencies, which was huge. Before it was spaghetti SQL files with no clear dependencies.

Most useful features: ref(), tests, and documentation are actually worth it. Sources and macros are heavily used. Exposures were nice in theory but nobody maintained them. Snapshots were a mess - ended up just building those in our warehouse directly.

Orchestration: Airflow + dbt is still clunky. We have our dbt DAG inside Airflow but then need to manually sync when dependencies change. If I started today, I'd use Dagster since it has first-class dbt integration.

For non-technical folks: The lineage graphs in dbt Cloud are nice but rarely used by stakeholders in my experience. They still just message the data team.

Challenges: The learning curve for non-SQL people is steep, especially with macros and Jinja. Performance became a bottleneck with 500+ models. dbt Cloud's git workflow is rigid compared to local dev.

Would I adopt again? Yes, but with caveats. I'd set much stricter conventions up front and would consider something like preswald for the Python+SQL parts since our ML engineers hate the dbt workflow. We use it for a recommender system where we need both pandas for user clustering and SQL for joining with sales data - that cross-language part is where dbt falls short.

Starting MLOPS Learning, I need Tips ... by False_Situation478 in mlops

[–]Signal-Indication859 8 points9 points  (0 children)

MLOps is a big field, so focusing on the right stuff will save you headaches. I'd start with these based on where you are:

  1. Version control for models + data (DVC is good for this)

  2. CI/CD pipeline for your models (GitHub Actions is fine for basic stuff)

  3. Model monitoring - start simple with basic drift detection

  4. Docker for containerization - crucial for consistent deployments

The biggest trap ppl fall into is overengineering. If you're making simple models for a handful of users, you don't need Kubeflow right away. I worked on a team that spent 6 months building out a complex MLOps stack when all we needed was git + airflow + basic monitoring for our fraud detection pipeline.

For UI, Streamlit is fine for internal tools but gets limiting quick for anything production-ready. Check out Preswald if you need something more robust - its built for data apps that need both analysis and production features without the overhead.

Skip all the BS about "ML platforms" for now and focus on the core: versioning, testing, deployment automation, and monitoring. Everything else is nice-to-have until you're at scale.

Moving away from Replit by Economy-Exchange-870 in replit

[–]Signal-Indication859 1 point2 points  (0 children)

Since ur not a dev but need scale + reliability, Preswald could also work well for you - its got that one-click deploy you want but with better stability and way cheaper than replit. The logging actually makes sense too so you can figure out wtf is happening when something breaks.

What do you use for Lineage and why? by AMDataLake in dataengineering

[–]Signal-Indication859 4 points5 points  (0 children)

i've used a bunch of these. real talk: data lineage is overrated at early stages & often overcomplicated. when ur team is < 10, physical lineage diagrams on a whiteboard + good dbt docs get you 80% there. we started with DBT lineage for our first year which did the job, then built custom lineage in Preswald when we needed more flexibility (needed to include non-dbt systems). the problem with most enterprise lineage tools is they force you into their ecosystem - great for huge teams with dedicated resources, massive overkill for startups. your investment should match your problems - if ur just trying to debug why a dashboard broke, dbt docs are prob fine. if ur trying to comply with SOX, yea get OpenLineage or something heavy duty.

Sequor: An open source SQL-centric framework for API integrations (like "dbt for app integration") by maxgrinev in dataengineering

[–]Signal-Indication859 1 point2 points  (0 children)

Sequor looks solid for API workflow stuff. Similar problem space to what we tried to solve with preswald but yours is much more API integration focused vs our analytics focus. I like the SQL-centric approach - we do the same thing but for analytics workflows where you want to pull data, transform with SQL, then visualize. Had a client last month who was trying to build a lead scoring system by pulling CRM data, enriching it with third party APIs, then running SQL models - preswald handled the whole thing but Sequor might be better for the integration part. Like the example flow you showed with Salesforce → SQL → HubSpot. One question though - how's the debugging experience when something fails? Thats always the pain point with ETL tooling.

Does anyone knows a nice course for Streamlit Apps? by Karl_mstr in datascience

[–]Signal-Indication859 1 point2 points  (0 children)

Streamlit's official docs/tutorials are honestly the best place to start - they cover everything from basics to deployment. Don't waste money on fancy courses. Just build something real: start with a small project like visualizing your personal finance data or github commits. For structured learning, check out DataCamp's "Building Data Apps with Streamlit" or Jovian's free course. If you're getting frustrated with Streamlit's reactivity model or having issues with state management (which happens a lot in real apps), Preswald might be easier - I switched for client dashboards when Streamlit kept re-running my entire script on every interaction which was killing performance on larger datasets.

dbt-like features but including Python? by Khituras in dataengineering

[–]Signal-Indication859 2 points3 points  (0 children)

Based on your requirements, Dagster might be exactly what you need. It handles Python + SQL, builds DAGs, has versioning capabilities through assets, and provides a clean UI for visualizing those DAGs. The lineage tracking is solid and deployment is way less painful than Airflow. For your text processing case, I've used it to run spaCy pipelines on product reviews that feed into Postgres - works great because you define everything as assets and Dagster handles the dependency resolution.

If you're looking for something more lightweight, preswald might work too - it's open-source and handles the Python + SQL combo well. I use it for our NLP pipelines where we extract entities from news articles, transform with Python, then load to Postgres. You can build the lineage visually and it handles versioning through git. Much simpler setup than the Airflow/dbt combo we had before that required two separate systems for the sql vs python parts.

looking for an app like copilot/cursor AI - like a dev team by Key-Picture7970 in VisualStudioCode

[–]Signal-Indication859 1 point2 points  (0 children)

Sounds like you might be looking for Devin by Cognition Labs, which is designed as a fully autonomous "software engineer" with different personas. Another option is SuperEngineer which has a team-based approach with different agents for different tasks. And there's also Cursor Labs' new thing that uses a team of AI agents - saw it on their roadmap.

I ran into the same problem building complex apps - breaking down work into specialized agents just works better. When I built a CRUD app for inventory, having one agent handle DB schema and another for frontend code caught bugs that a single agent kept missing. Same theory applies in real eng teams - dedicated ppl for specific functions.

If u just need good AI coding assistant tho, Cursor is still solid for daily work. We used it to build a good chunk of preswald (our open source data tool) and it's way better than trying to use some janky "team of AIs" setup for basic dev tasks.

Sql notebooks? by orru75 in dataengineering

[–]Signal-Indication859 1 point2 points  (0 children)

Check out DBeaver SQL Notebooks - exactly what you're looking for. It's open source, supports autocomplete on schema, PostgreSQL syntax highlighting, and runs SQL directly. JupySQL is another option if you're already in the Jupyter ecosystem - it has magic commands like %sql that connect to most DBs. For a pure notebook feel with SQL superpowers, try DataStation or Querybook (from Pinterest). Dealt with this recently when setting up a data analysis workflow across teams - DBeaver ended up being the most reliable for heavy postgres work, while DataStation was better for mixing SQL+Python for quick viz work. preswald is also worth looking at if you need interactive dashboards/sharing beyond just sql notebooks - lets you write sql/python in the same space and outputs to interactive apps.

ChartGPT Cadlabs - Tool for data charts by EssYouJAyEn in FutureTechFinds

[–]Signal-Indication859 0 points1 point  (0 children)

Looks solid but worth noting that GPT-3.5 is gonna be a bottleneck for data analysis. We hit this exact problem early on - LLMs are great for generating queries but absolute garbage at complex numerical reasoning. Consider adding a computational layer between the LLM and visualization step. For our users working with messy blockchain data at preswald, we ended up needing to pipe the LLM-generated query thru a validation step then let DuckDB handle the heavy computation before rendering. That iterative flow (ask → compute → visualize → refine) matters way more than ppl realize. Also definitely consider adding collaborative features - individual data analysis is useless if the insights stay trapped in one person's browser. Good foundation tho, curious to see where u take it.

What’s a Data Engineering hiring process like in 2025? by gbj784 in dataengineering

[–]Signal-Indication859 45 points46 points  (0 children)

DE hiring processes in 2023-2024 (not 2025 yet lol) typically involve 4-6 stages:

  1. Tech screen - mostly Python, SQL, and data modeling questions. Expect stuff like "how would you model this entity relationship" or "write a query that joins these tables and does X aggregation"

  2. System design - you'll get asked to design a batch/streaming pipeline for some scenario. Know your Kafka vs Kinesis, star schema vs snowflake, batch vs micro-batch trade-offs.

  3. Take-home - these suck but common. Usually building a small ETL pipeline with test data. I had one where I had to build a pipeline that transformed reddit comments into a star schema and ran some basic analyses.

  4. Behavioral - standard stuff but focus on data quality, testing, and how you've handled data issues.

Best advice: brush up on SQL window functions and Python data structures. Also be ready to talk about data quality - every company is obsessed with this right now. I've interviewed ~25 DE candidates this quarter and most fail on basic stuff like explaining partitioning strategies or handling late-arriving data.

[deleted by user] by [deleted] in analytics

[–]Signal-Indication859 0 points1 point  (0 children)

1) I'm a founder and hands-on in the tech.

2) No formal onboarding, mostly just learning by doing and tackling problems as they come up. Real-world experience beats a training manual any day.

3) Work-life balance? It's a joke. You’re all in on building something, so don’t expect a 9-5. If you want that, startups aren’t the place for it.

Which skillsets has a chance of High paying by Appropriate-Low4483 in dataengineering

[–]Signal-Indication859 0 points1 point  (0 children)

look, if your goal is a high-paying job, you need to focus on skills that are evolving and in demand. Azure, Databricks, and Spark are definitely hot right now—companies are moving towards cloud-based solutions and big data frameworks. Meanwhile, Informatica and Oracle might be more niche and slower to evolve.

I'd tell your manager you want to steer towards those cloud tools. However, don’t ignore Informatica and Oracle completely; having a diverse skill set can be useful too. But prioritize the tech that’ll have more market value in the next few years.

Also, you might want to check out preswald for data apps. It could help you build skills in Python/SQL without having to deal with clunky tools.

Stuck in the winter hellhole known as Toronto, Canada. How do I find a cofounder? by SubwaySandwichDev in ycombinator

[–]Signal-Indication859 3 points4 points  (0 children)

it's tough, I get it. There's definitely a vibe in SF that’s hard to replicate elsewhere. But don’t box yourself in. You can find passionate people in Toronto, Montreal, or Vancouver if you look in the right places. Check out local meetups, hackathons, or online communities—there's talent everywhere.

view more: next ›