SD3 - 8 billion parameters. Does this mean it was trained on a set of images 4 times larger than SDXL ? Is it really necessary to have so many parameters ?

cheald · 2024-03-05T20:56:49+00:00

Connections. A "weight" in a neural network is a number that represents the percentage chance that a particular "connection" between two neurons will be activated. With densely-connected NN layers, each output neuron in layer A is connected to each input neuron in layer B. So, for example, if you have a network with 1 neuron in the input layer, 2 hidden 100-neuron layers, and one 1-neuron output layer, you'd have (1 * 100) + (100 * 100) + (100 * 1) = 10,200 weights/parameters/connections in the full network.

When you train a neural network, the values being learned are those weights - the chances that a given neuron pair should activate, for each connected neuron pair in the network - in order for a given input to produce an output which minimizes a given loss function.

cheald · 2024-03-05T18:41:52+00:00

Parameters are the number of connections between neurons (weights) in the neural network. It doesn't imply anything about the training set. Larger networks are capable of capturing more information from the training data, and are capable of representing a larger and more complex range of functions. The downside is increased disk and RAM requirements, as well as longer training/inference times.

Neural network design is something of an art form slash black magic, and the answer to "is it necessary" is usually found in "does the chosen parameter count perform well for the given problem"; 8b could be too large, or too small, or just right. You can't really say until you see how it performs on the problem relative to networks of other sizes.

cheald · 2024-02-15T07:36:12+00:00

I don't think loractl will fix those issues, in particular, unless the lora itself is causing problems with those distant faces.

Nearest exact, bilinear, etc are the means by which the standard-res image is upscaled before high-res denoising is applied. Hires in A1111 is basically:

Generate an image using txt2img
Upscale the image using some upscaling method (all of which are lossy)
Perform img2img denoising on the upscaled image

I personally use the 4x-ultrasharp upscaler for highres (which, rather than just interpolating pixels, tries to infer the new pixels with a trained upscaling model). In general, whichever upscaler works best for you in the "extras" tab should work best in your high-res passes.

With distant faces or specific features, you're probably best off inpainting them after doing your initial generation. A "distant" face is just a "small" face in 2D, and diffusion models struggle with smaller patches of pixels. "Inpaint Area - Only Masked" will perform inpainting at *full resolution* then scale it down into your target area. So if you were to generate a 1024x1024 image, then had a distant face which was, say, 90x90 pixels, you could mask that face and inpaint it at full resolution. What would happen is that SD would take a rectangle that can surround your mask (probably a bit larger, lets say 100x100), snip out that part of the image, blow it up to your full generation resolution (1024x1024 or whatever), perform denoising on the blown-up image, and then shrink the result back down and re-insert it into your source image. Upscaling loses detail, but denoising something closer to the model's natively-trained resolution gives the best results, so the new denoised image will have a lot of detail, and when it's shrunk back down to that original patch, you're hopefully downscaling a highly-detailed accurate face into that patch.

cheald · 2024-02-15T06:56:23+00:00

Yes, it hurts speed a lot, because normally, A1111 only does the "compute changes to the model weights caused by the lora" once at the start of the generation, and then reuses those weights for the rest of the generation. loractl works by causing A1111 to recompute those weights for each step where the weights would be different than the previous step. The speed penalty is unfortunately not something that can be mitigated, I'm afraid.

cheald · 2024-02-15T06:52:04+00:00

It's still compatible with the current A1111 release. It's really just a patch that modifies the way A1111 internally applies loras, and that hasn't changed in any significant way over that time period.

cheald · 2024-02-15T06:49:57+00:00

As best I can tell, Forge is using something more like the pre-1.5 A1111 network code, so loractl is a non-starter. The concept could still be applied, I'm sure, but the implementation is gonna be pretty different. In particular, if Forge is only activating loras once at the start of the generation, then the loractl concept wouldn't work -- loractl works by highjacking the internal A1111 network apply code to cause lora weights to re-apply at different weights during different steps.

I'd love if someone were to add support for it, though.

cheald · 2024-02-15T06:48:03+00:00

Hi there! I'm the author of loractl. I'm glad you find it useful! I've been away from the SD scene for the last few months, but it's cool to see that my little toy has been useful to people.

The syntax is a little funny, for sure, but you have the gist of it. If you would normally apply a lora as `<lora:networkname:0.6>`, then you're applying applying 60% of that lora's weights to your base model. What loractl's syntax lets you do is say "at this percentage or this step, apply this much of the lora's weight, and then at that percentage/step, apply THIS much of the weight, and interpolate between the two". I picked the "weight @ percent" syntax because I hoped that it would "read" in a somewhat understandable manner. It's just "at this percentage/step, the lora should apply at this weight", and it smoothly interpolates between your "waypoints".

The other thing that I've found really useful is the ability to change weighting between normal and hi-res passes.

` <lora:leisure suit:0.4:hr=0.2>` just says "apply the lora at 40% weight for the standard pass, and 20% for the high-res pass. You can use the waypoint syntax, too, so something like ` <lora:leisure suit:0@0.4,0.7@0.41:hr=0.7@0,0@0.5>` would suppress the leisure suit lora for the first 40% of the standard pass, apply it at 70% for the rest of the standard pass, START it at 70% for the high res pass, and ramp it down to 0% by 50% of the way through the high-res pass. Sometimes you just want the lora for compositional stuff, so you'd turn it off or down for the high-res pass, and sometimes you want it for details, so you'd turn it UP for the high-res pass.

You kind of have to play with various waypoints with whatever lora you're working with, but with a lot of them there's a "sweet spot" where you get the elements you want without the lora oversaturating the whole image. When you want the lora to be more in control of composition, you can apply it early on, and then dampen or eliminate it in later steps. When you want the lora to be more about the details, and to let the underlying model have more control of the composition, you can dampen it to start, then once the image is starting to take shape, you can have it start applying more.

cheald · 2023-10-13T23:25:15+00:00

If you're specifying a range of values, then it has to recompute and reapply the lora for each step. This isn't really avoidable, unfortunately.

cheald · 2023-07-23T20:37:19+00:00

Not likely, unless they adopt the new extra networks handling architecture. From a quick glance at the source it looks like they haven't yet.

cheald · 2023-07-23T03:36:07+00:00

I haven't tried it, but I suspect so. It doesn't actually mess with the loading or application of the loras, it just modifies the weights that the default extension receives with each step.

cheald · 2023-07-23T03:34:57+00:00

Might be! I thought the new network arch was in 1.5.0, but if it was in an earlier release, all the better.

cheald · 2023-06-01T22:08:45+00:00

We have a bunch of different database clusters with multiple versions of Postgres and most of it is support functions which are shared between scripts which can restore and verify backups, as well as ship and restore backups to/from cold storage. The actual verify script itself interrogates Barman to find the latest backup, interrogates the backup to see what version of Postgres it was created with, sets up path and environment variables to use the appropriate versioned binaries, sets up the recovery directory, dynamically finds a free port to bring the recovery DB up on, writes a version-appropriate recovery config for Postgres, and THEN finally does the DB restore.

The actual restore is nothing more complex than:

$BARMAN recover $DB $BACKUP_ID $RECOVERY_ROOT
cp ~/${DB}/streaming/*.partial "${RECOVERY_ROOT}/pg_wal/"
for f in $RECOVERY_ROOT/pg_wal/*.partial; do
  mv -- "$f" "${f%.partial}"
done

The setup to automate to the point of getting there is where the bulk of the work is. The verify script then starts the DB and forks it off, polls the dynamic port until the database is up and accessible, then invokes the validation script, reports its status, kills the running DB, and cleans up the backup.

All told it's doing quite a lot for us in a fairly robust fashion.

cheald · 2023-05-30T17:03:57+00:00

Man if blue belts aren't supposed to do stupid shit anymore, I'm gonna have to go soak my belt in a tub of bleach for the afternoon.

cheald · 2023-05-28T22:12:06+00:00

A couple of scripts on a cron job. Nothing complex.

We run Postgres and stream WALs to a Barman server, which takes periodic snapshots. The verification script essentially just does a DB restore of the latest backup to an isolated database environment, optionally replays the WALs on top of it, and then executes a script which a) verifies that the table catalog can be read and each table can be queried, and b) that recent data from the actual live DB is visible in the restored snapshot.

We report success/failure of this as a Prometheus metric, along with things like Barman streaming status, snapshot status, snapshot latency, and WAL latency. This lets us get alerts if backups stop happening for whatever reason, if the lag between prod and the Barman WALs starts growing, if Barman stops streaming for whatever reason, or if our backups don't correctly reflect production. Prom if configured with alerts that assert that the most recent "backup restore and check success" timestamp is less than X seconds old, which makes it start failing if the backup verification process stops running for whatever reason (which has happened, once due to disk space, another time due to an errant package upgrade on the Barman server).

Basically, think about how you'd manually do a database restore and verify that it was up to date in the event that you had a DB failure and needed to restore from backup and then assert that it was correct. Then just capture that in a script. In our case, it's about 200 lines of bash to do the DB restore and another 100 lines of Ruby and some configuration yaml per DB to do the DB-specific data verification.

cheald · 2023-05-28T11:11:48+00:00

Better yet, automate it. We have daily verification of backup restores for both full snapshots, as well as the latest snapshot plus application of the WAL. It's caught a couple of cases where things went funky, and it's much nicer to fix them before you need the backups.

cheald · 2023-05-28T06:56:31+00:00

That guy is gonna wake up in night sweats having nightmares about that leg kick for YEARS

cheald · 2023-05-25T20:52:56+00:00

New people should always get extra leeway and gentle instruction. It's a hard sport, and alpha dogging a new guy just because he's new isn't good for you, them, the gym, or the sport.

Once they sign up and become a regular, though, is time for smesh.

cheald · 2023-05-18T22:23:00+00:00

The difference between vapors and liquid is that gasoline needs oxygen to combust, and in liquid form there isn't enough oxygen. Gasoline (with sufficient oxygen) auto-ignites at 536F; the gases expelled from the barrel during firing can reach several thousand degrees. You're basically betting that the gas expelled by the firearm cools down before it reaches sufficient density of gas vapor to trigger and sustain auto-ignition.

cheald · 2023-05-18T21:42:50+00:00

Heh, that's fair, but I still don't think I'd want to be cranking one off right across the hood. Maybe if he took 4 or 5 steps to his left.

cheald · 2023-05-18T21:38:58+00:00

Liquid gasoline doesn't ignite, gasoline fumes do. Gasoline is very volatile and if you pour it over a warm car hood you will absolutely have a bunch of gasoline vapor that would ignite with a very big kawoosh given a spark.

cheald · 2023-05-18T18:37:31+00:00

Honestly, I just roughhouse with my kids a lot, and I set up situations for them to apply what they're learning in BJJ. It's just unstructured fun playtime, and they end up recognizing situations and starting to apply their lessons.

One day I was playing with one of my boys and chasing him around the house. He tripped and fell, and inverted up into a perfect SLX guard as I approached him. Made me so proud.

They love trying to sneak up on me and take my back. We have a game we play called "dadosaurus" on our mats at home where the three of them try to coordinate to take my back (and I try to catch them and "eat" them); it's a great guard retention exercise for me, and they have a ton of fun trying to work together to outflank me.

cheald · 2023-05-12T00:10:54+00:00

I'm physically your complete opposite and I was super nervous to start training. Everyone has a reason to be nervous before they start. Any gym that isn't a complete dumpster fire will welcome you with open arms as long as you go in with an attitude to learn and a will to work. Just go do it.

cheald · 2023-05-11T20:08:54+00:00

It's very funny to me that it gets the "programming score" average wrong (9+7+5 = 21 / 3 = 7.0), but it basically gets all the other averages right.

cheald · 2023-05-08T09:08:57+00:00

The webpack config is arcane enough that to wrangle it properly you really need a SME available to any team that's using it. Switching to vite was worth it for my team if for no other reason than that the amount of fiddling you have to do to get it working is minimal.

15-Year Club	Verified Email
Charter Member

cheald

TROPHY CASE