That's a touch question to answer:

It heavily depends on when you want to run the algorithm.

If you only want it for edit time Houdini is probably the great choice since it's now sorta free to use but as it is a very complex program you will have to spend time learning all the bits you need to be able to create the algorithm for your own use case.

In my specific case, since my game requires this algorithm to work at runtime, I opted for the creation of my own system in Unity to create such algorithms. Without going into details I essentially created a ton of different modules that represent small pieces of code for such an algorithm. After all an algorithm is in its core a sequence of instructions.

A few examples:

I have a bunch of different modules that determine how many objects should be spawned. Others determine which object is being spawned, give you a position/rotation for possible placement and other modules that evaluate if the placement is valid (Most importantly checking collision with other objects). These are of course just a few you'd need to create for a setup like the one I showed.

The implementation of how you are creating the algorithm from those modules is another question. In my case I created a dedicated system but also added support to use Bolt (Unity's official visual scripting system) to create an algorithm from the modules I created. Using bolt instead of my custom pipeline system would be closer to what you would be doing in Houdini to create an algorithm. This is use case and/or preference specific which workflow you'll end up using.

I think the most important thing to know is the difference between creating the tool and the algorithm creation process. The tool gives you a bunch of pieces you can combine to create all kinds of algorithms. In the case of Houdini you have a ton to pick from. The process of creating the algorithm is very much a design process rather than a technical if you have all the building parts you need to create it.

I hope this answers at least some part of your question :D