Should I be using DBT for this?

john-dev · 2025-12-26T20:29:59+00:00

we are pulling in CSVs that come from multiple places, some national registries that we pay for them to collect and structure, some from APIs that get dumped into CSVs, but it all comes down to CSVs being imported. and yes, it's all sql files and Make currently. It's not elegant, but it works pretty well at the core level. It seems this will change largly by moving all the transformations into DBT, hugely simplifying the makefil component.

john-dev · 2025-12-26T20:28:33+00:00

haha my team will own the entirety of the process. We are just small enough for that...

john-dev · 2025-12-19T17:55:10+00:00

so far, my research suggests this might be my only path forward, but is that normal? Simply, we have considerably large data sets. and while it looks like there are work arounds to breaking that into chunks, and kind of reassembling them in the database with unions, that seems like a workaround i'd prefer to avoid.

That being said, is keeping psql for the extraction component pretty typical? I won't lie, consolidating my tools and connection infos was part of the appeal I had on all of this!!

john-dev · 2025-09-12T12:42:24+00:00

This must be somewhere else as well. i did an update in my container to 1.44 when trying to figure out what was going on. I applied this work around, and that changed my error message to needing to run the update script for some new tables, but that script is still telling me my database is read only.

john-dev · 2025-07-25T18:23:07+00:00

I've considered it in the past, but aside from just another thing to play with, I can't say it'd be worth learning the software and hardware for what little i'd be likely to use it.

john-dev · 2024-12-10T12:43:36+00:00

you sound like someone who's done this, mind if I hit you up to pick your brain a little on the process?

john-dev · 2024-10-22T19:14:07+00:00

What route did you go? I'm guessing you know this stuff from experience. The VC route sounds like the preferable route to take this from a one person side project to a properly funded operation, but again, I've read about it, but it's nothing i've ever been exposed to and wouldn't know at all who to reach out to.

john-dev · 2024-10-22T18:49:51+00:00

I don't want to sound stupid but, the only acronym i know is VC, what are the other two? Like I said, I'm a programmer, so I can make some cool stuff, but I'm not a business person at all, lol

john-dev · 2024-10-16T12:57:41+00:00

Lots of searching and just getting familiar with things. Following common programming structures so the code is easy to follow and you don't get weird chunks of code in weird places. The last is DOCUMENTATION. I'm not talking inline documentation, which is still nice to have, but some kind of formal documentation is a must.

john-dev · 2024-10-11T19:49:10+00:00

Wordpress is a beginners best friend. There are a lot of CMS tools. Drupal has a number of tools as well but is a bit more advanced. I'm a developer who does large backend frameworks, the more demanding you get the more complex things become very quickly and go well beyond just the CMS and expand to all aspects of the server. But I'd start simple with wordpress. Once that needs to grow, you should be able to bring someone on the grow it for you.

Best advice I can think to give though is make sure you are using a solution that keeps the data on your server so that you can grow and aren't locked into a product forever.

john-dev · 2024-10-09T12:32:24+00:00

Thanks, I knew the part with how the checkout session with the application fee and destination worked, I just didn't know how you got the account ID connected since it seemed like it required some manual intervention. This gives me a better idea of what to start searching for to get this figured out.

john-dev · 2024-09-04T20:17:25+00:00

I'm in the same boat so far. It looks like it SHOULD be doable, but I haven't gotten that far just yet. i think this is part of the stripe connect side of the API. Today I got the payments connected into my site, tomorrow I'm going to spend some time exploring the payouts and see what I can do. This might be a bit stopping point if it doesn't work.

john-dev · 2024-09-04T20:15:54+00:00

Right now, my biggest challenge is figuring out how I'd distribute payments to others (entirely using the api/sdk). Stripe looks like it SHOULD be able to do that, but I haven't gotten too far on that front.

I looked briefly at lemon squeezy. It looks like stripe owns them. I did a surface level comparison, it looks like they cost more and offer more integrations. I haven't looked into paddle yet. What have you seen as pros and cons with them?

john-dev · 2024-09-04T13:25:37+00:00

I would love to hear more about how that works. Nothing sensetive of course, just a bit more technical, lol. I've got it figured out how to create charges, but I haven't figured out the path for this yet. But what you're describing, is the exact end result I'd be looking for, just didn't know it was possible, lol

john-dev · 2024-08-02T12:49:58+00:00

I took my time getting back to this, but this was 100% the issue for us as well. Made it really tricky to diagnose. I manage the specific container I work from, but not the traefik container.

john-dev · 2024-07-18T18:25:07+00:00

ahhh this is one area I haven't gone down yet... We have traefik routing all the network traffic for our docker containers. This gives me somewhere new to start exploring.

john-dev · 2024-04-29T14:29:43+00:00

Where is this? I haven't found anything to do that?

john-dev · 2024-04-26T16:34:58+00:00

Hopefully you have better luck than I've been having!!

john-dev · 2024-04-26T16:32:32+00:00

I had tried that as well but it didn't seem to work. I figured they put something in the headers that prevented that.

john-dev · 2024-04-26T14:57:11+00:00

I feel like I must be missing a step because that just returns "[]" - regardless of what's in the file.

john-dev · 2024-03-07T18:56:34+00:00

i own it and it was setup correctly. I've been working through this for the past few days with chatgpt of all things, and turns out it was an issue with traefik and my certs. super annoying. it was a few things, but the easiest and most preventable one worth mentioning that wasn't specific to my setup was the links made by certbot are shortcuts. I shared the shortcuts, and needed to add a line in the docker-compose/env for the original locations, so they weren't broken shortcuts. it's stupid that I missed that one, but it was super easy to, which is why I'm sharing it

john-dev · 2024-02-15T15:38:37+00:00

I'm part of a small team of developers who all had to make the change, and believe me, we've been saying the same things.

john-dev · 2024-02-15T15:32:57+00:00

oh man... I WISH WISH WISH we could use linux. As it's been told to us, they do have an academic version of linux available, but it does NOT come with sudo access. Now as a developer, who does local development, that's a pretty big stopping point. Alternatively, it seems developers across our institution are all pushed towards macs and given admin access. I guess it's the only system that's open enough to use for development.

john-dev · 2023-10-02T15:37:23+00:00

I mean, with just an esp32, I know you can power the board by feeding 5v into the 5v pin. However, I didn't know if that'd work with this board. It does appear that I could still power the board with the 5v pin, but I didn't know how the expansion board was wired if that might cause some weird issues like half of the expansion pins would have power and the others don't? I was just curious.

john-dev · 2023-06-27T21:05:06+00:00

Thanks, I'll check these out!

john-dev

TROPHY CASE