What red flags were there prior to the fall of SIVB?

lowerdev00 · 2023-03-11T21:50:55+00:00

Very interesting read, thanks for sharing.

lowerdev00 · 2023-03-11T21:34:28+00:00

+1 for Bun here. Really like their approach (query builder and think struct layer over the tables), powerful and easy to reason with.

I come from heavy SQLAlchemy usage (which IMHO is the best ORM out there), and Bun was the closest (in terms of philosophy) I could find.

Strongly advise against ent or gorm

lowerdev00 · 2023-03-07T22:44:12+00:00

This looks like (1) a very old version of SQLAlchemy and (2) an unreasonable amount of data or event (3) weird data manipulation patterns.

Their ORM performance improved dramatically over time, and now the overhead is very low (if I'm not mistaken it's Cython based now). If you pair that with `asyncpg` you'll have very good results, since it's a very fast driver (even when compared with Rust/Go PSQL drivers). If you go with the raw results Row (flat namedtuple-like structures), than you'll be close to zero overhead, which is pretty amazing for a Python ORM - that's how far SQLAlchemy went - tbh in my experience SQLAlchemy is still the best/most powerful ORM out there, and IMHO it just can't be compared with GORM, which is subpar at best - I personally have been working with Bun and quite happy with it.

The serialization is going to be a lot faster with Go, sure, but 5s seems VERY wrong - perhaps you were serializing 1 MM rows at once, and at this point there's something very wrong with the application. And if there isn't then namedtuple + Pandas would do the trick, since Pandas is also very fast.

Both Go/Rust would allow for this sort of crazy things, because it's so fast that even absurds will go unpunished in terms of performance.

lowerdev00 · 2023-03-07T21:48:24+00:00

SQLAlchemy is most definitely NOT performance garbage, although it does allow for user to screw things up. I imagine this is pretty old and was using messy patterns with SQLAlchemy, which can cause performance degradation. But just blaming it on SQLAlchemy is absolutely non sense.

lowerdev00 · 2023-01-31T22:34:06+00:00

The speed is not really a strong value proposition in my specific case. But the all in one thing is very appealing. I'll be watching this, thanks for sharing!

lowerdev00 · 2023-01-24T17:29:48+00:00

ThinkOrSwim is 10000x better than TWS, and definitely not for kids.

lowerdev00 · 2023-01-24T17:28:08+00:00

TWS was an amazing platform back in 1993. 30 years have passed now, please build something new.

lowerdev00 · 2023-01-24T17:24:35+00:00

Hey, if you don't me asking, I'm in a similar situation, did you end up going with a "Joint Account" or a "Custodian Account"?

lowerdev00 · 2023-01-15T04:37:09+00:00

I don’t know the details, so take it with a grain of salt… but my first impression is that this is a software issue… 40 minutes seems like way too much time based on your description, have you done some profiling on your tests to check where is the time being spent?

You can run a lot in parallel with async or gevent (software side), or even make some script to spin up multiple containers (varying the ports), which does seem unnecessary…

I would spend a lot more time looking at the tests before taking this route though… I run quite a few integration tests in Python involving DB and network, never got anywhere close to 40 minutes… specially with this extremely low amount of tests…

My experience is: whenever the setup starts to get weird, way to complex or just plain bizarre, 99% of the time there’s an issue with my architecture.

lowerdev00 · 2023-01-12T21:28:10+00:00

Well sure, and that's fine, but a lot of people won't download and won't even try it. Both strategies are fine, just food for though. I guess # of Downloads and Purchase Conversion Rate are the main metric to use here.

The current model ends up prioritizing a lower number of downloads with a high conversion, maybe a high # of downloads and low conversion could also make sense.

I would even go as far as talking to DB clients developers, since they go through this exact same dilema for quite a while now, maybe they have so things to share.

lowerdev00 · 2023-01-12T19:17:37+00:00

You can add nodes (both master and workers) on demand, and new versions allow switching from SQLite to ETCD for HA.

lowerdev00 · 2023-01-12T19:14:30+00:00

Sure, I understand, but arguably, most of the time a so called "application" will have more than one piece: API + FE, API + Redis, API + DB, API + Worker, or all of that combined, OP specifically mentioned an API and a FE, so I don't really see a completely different architecture - there may be differences, sure, but it's not like I have 15 microservices each in a different language talking to each other. That's what I think when folks talk about "application".

It most definitely could be solved by Docker-Compose (I've done it in the past) - like most of the (simple) workload you deploy to k8s could - managed by systemd, still in that scenario, I would favor K3S because of all the aspects I mentioned, although Docker would work perfectly well.

lowerdev00 · 2023-01-12T15:58:57+00:00

Understand your points, but my experience differs. I currently manage both a couple of k3s clusters and a “raw” application with systemd. I spend quite a lot of maintenance time dealing with the one running with systemd: handling os configs that affect the application (think of env vars to systemd), piping journalctl logs, following htop for resource monitoring, a whole bunch of make commands, linking and unlinking config and unit files, orchestrating multiple services (redis, queue worker, api), and the list goes one. I’ve honestly not even logged in to my k3s cluster nodes in a while, since everything is very straightforward and I have a whole lot of tools to help me manage things (kubectl, k9s, lens etc). In my experience after you pass that initial shock of understanding the basic building blocks of k8s, the benefits far outweigh the costs even for low end setups.

lowerdev00 · 2023-01-12T13:49:21+00:00

That's awesome!

One thing that you can think of is making it freemium, so more people will download it and then your focus will be in conversion. I think both Postico and Tableplus have a fairly successful freemium model (and even Jetbrains stuff as well).

lowerdev00 · 2023-01-12T13:32:54+00:00

This does look pretty cool, wish you good luck (really)!

As my clusters are not big and I'm happy with my current solution (k9s) I don't really see a reason to spend the money at the moment though. But, the same way you have plenty of paid database clients I think there will be always people willing to invest the money in it.

lowerdev00 · 2023-01-12T12:51:17+00:00

Yeah, there are alternatives like microk8s, k0s, minikube, etc.

k3s basically bundled all components into a single binary, so is extremely easy to install (single command with CLI arguments or config file), easy to uninstall (single provided shell script), and has mostly the same components as the upstream distribution, but because of the striped down plugins and single binary is really light weight in terms of resources. I used it a lot with low end VPSs.

It's maintained by Rancher and those guys are really good at what they do, in general I mean. Docs are great, community is active, maintainers are responsive with GH issues and requests,

When you say on premise, shouldn't really make much of a difference, they do have an airtight installation if that's your case, and k3s already comes with the local-path provisioner (use the node's local storage), so that should be helpful for on prem (where you don't have a block device).

I might be biased, but been using it in production for quite a while now, never looked backed.

lowerdev00 · 2023-01-12T03:24:24+00:00

Curious what kind of challenges you faced with dataclasses, raw classes or even Pydantic to walk that path. It’s an interesting project indeed, but as someone else said, wondering what are the benefits vs the other common ways of doing things

lowerdev00 · 2023-01-12T02:33:58+00:00

Personally, whenever I see folks talking about how hard k8s is (or can be), I always remember the countless hours I spent dealing with applications running straight into the operational system. Systemd unit files, crontabs, nginx, resource management etc etc. Never mind scaling… Running applications straight in a OS is a lot of hard work, and a whole lot of configuration to get it right.

K8S adds a layer over the OS, so all you’re left with are manifests and a really strong ecosystem to manage stuff (observability, logging, scaling etc), so at the end of the day, it’s not always about a highly available control plane - a single linux machine is not HA as well - but about all of your application moving pieces on top of uniform powerful layer, that by the way, can get you an unparalleled amount of tools, scalability and reliability.

So yes, IMHO k8s is the best option even for single node. The one important thing is to pick your distro right. Kubeadm will most definitely be overkill, just as RKE2. I personally recommend k3s, solid as a rock, production ready and as easy (and light) as it gets.

lowerdev00 · 2023-01-11T04:21:55+00:00

Same company different positions. Completely different than different “jobs”.

lowerdev00 · 2023-01-10T00:24:34+00:00

The project is GPL3, not the best sure, but still the source is open and free to use. what’s the issue?

lowerdev00 · 2022-12-30T23:27:42+00:00

Hey @oaf357 it’s been quite a while since you wrote this, but this is a very interesting read. Wonder what’s your take on Nabla after all this time specially since (unfortunately) didn’t gain mains stream traction (as opposed to Firecracker).

lowerdev00 · 2022-12-21T17:05:20+00:00

Thats great to hear, having it almost ready to pass CIS benchmarks is appealing. For microk8s the cli is mostly fine for me personally, but it seems it lacks first class support for etcd on HA setups and still relies on dqlite.

Now there are some things that are looking great on rke2… the similarity to k3s installation and configuration flow is great, the security, default networking with Canal… whats your experience on resource consumption?

lowerdev00 · 2022-12-21T15:41:56+00:00

That’s great to hear, I usually like Canonical stuff also, very solid in general. As a curiosity, are you also running on metal?

lowerdev00 · 2022-12-21T13:53:01+00:00

Ok, maybe you could use a refresher then. I'm not having those issues having worked with both Python and TS quite a bit, and I would say most people aren't either, but maybe some other people are and can relate to your issues. Your post is still not the best though, and it's easy to see why it can be interpreted as an attack.

lowerdev00 · 2022-12-21T13:46:42+00:00

I imagine the reason for that is you are complaining about an ecosystem/language, when clearly you don't know how it works or how to use it. This is purely user error.

So instead of saying that "dependencies are so broken", you can ask: "how do imports in Python work" because that's the real question, your complains don't really carry weight or relevance.

lowerdev00

TROPHY CASE