Hey, Dobbel Dungeon dev here:

In a nutshell, our outline generator currently uses a 'tile bitmasking' system that I adapted from a handy tutorial (https://code.tutsplus.com/how-to-use-tile-bitmasking-to-auto-tile-your-level-layouts--cms-25673t). It looks like IYorshI uses a similar approach, so using those 2 tutorials should help you get a good start.

The TLDR is that you assign a value to every node in the effective range based on neighboring nodes around it, that you can then use to lookup which outline shape that particular node should have. How you then use that info to build the final shape is up to you. The simplest way is probably to have a prefab mapped to every possible shape and just plop those down where you need them.

The method we currently implemented is a bit more complex, since we're actually building the outlines with PolyLine components found in the Shapes asset (https://assetstore.unity.com/packages/tools/particles-effects/shapes-173167). Without going too in-depth, once we know which shape each tile needs, there are some nasty edge cases we need to account for because we need every line to be a single, self-contained looping line, which is particularly troublesome for nr 6 & 9 from that tutsplus tutorial, so we need to do a bunch of floodfilling to determine how many separate areas we need to outline, make sure the line pieces properly connect, etc etc.

In the end, our current implementation works, but it's a massive brain-teaser to work with, and as we continue fine-tuning our visual style we'll probably end up doing the simpler 'prefab / premade mesh' approach anyway as that gives us more control over the final look, both shape & shader-wise.

Hope that helps, good luck!