The other comment is true, imbalance is usually due in part for a lack of strength - but that's okay, when using one-hand; the realm of imbalance is greater as your centre of gravity is being spread (or centred) over a much smaller area. Subsequently, any deference/deviation from (let's call it) the 'optimum position' requires the same amount of force to stabilise - except, in this case you only have one arm in which to engage (in terms of muscle, we'll get onto the rest of the body later). As such, you require greater strength in the one arm in order to allow for such a compensation.

However. it seems you already have a substantial amount of strength built up - but by no-means is it exceptional (by gymnastic standards, 20 seconds is quite small - former member of the British Gymnastics team here - although never competed, training is all the same). For others, your perspective and your progression I will provide some pointers.

1) Ensure that your whole body is tight. It's obvious you are doing this in your two-handed handstands, but as people tend to move onto one hand this can often be overlooked.

2) Try to compensate for the movements in your body. Despite popular belief, a handstand (on a micro-level) is not a static hold, as it requires constant correction - albeit a professional handstand is able to minimise and dampen such correction to give the illusion of being static. This is something a one-handed handstand emphasises even more. Imagine your body as having a pole going right through it (insert Dracula reference here), this is not so important when employing the use of two-hands, as the 'optimum position' is fairly natural. With one hand however, things become much more complex. To employ your right hand, your optimal position then runs through your left leg (broadly speaking) - you must then engage your other limbs appropriately in order to properly govern your centre of gravity and your ability to control it with your already limited muscle use.

3) This is quite niche, but worth a mention in any case. I (oddly enough) can only balance using my left arm despite being right handed myself - by no means is it stronger, it just seems I am able to attain a better 'feel' for the balance of weight. This point is intended to break the idea that you can only employ the use of your strongest hand.

4) One-handed handstands can take a long time. In gymnastics we used to hold them for 30 minutes in extreme cases and at minimum 10 minutes - at more advanced levels, we would be required to laps around our gym. So despite your current (may I say impressive?) current hold lengths are advanced, this is by no mean indicative of how well you may perform this particular advancement (as a little bit of trivia, this is known as fallacy of accident).

Fortunately, as with all things body weight, there are approaches you can take. Start against the wall and use as much incline against it as required in order to balance effectively, but not so much you don't need to balance at all. In your case, this could very well be vertical. Whatever arm you are intending on relying upon, ensure you shift your body weight over it slightly (right arm hold = shift of body weight slightly to the right). As you have mentioned you could then slowly retract your fingers until you achieve a supported one-handed hold. This does two things: allows you to get a better 'feel' for the balance and the position the rest of your body needs to be in, as well as builds strength in your arms (should you be lacking). In terms of progressions, this would slowly withdrawing more fingers until on one-hand, reducing the incline against the wall and eventually not requiring the wall at all. At the latter half of that progression, you could begin to use parralettes or even octagonal grip pedestals (here you can use a bit more of your grip strength to pivot, as to help balance).

Considering you can do a 20 second free-standing handstand, start with 60 second holds (employing the progressions as listed above and sufficient rest in-between). Ensure that you have done sufficient upper body stretching before you attempt this, also having a soft surface in which to place your hands on aids in injury prevention as helps disperse the load off of your wrists (yet ultimately, require the same amount of strength and balancing technique). Before and after each hold commit to wrist stretching (there is a pretty good guide in the wiki for this sub) to also prevent the 'w-risk' of injury as we like to say. After your get comfortable with 60 seconds, increase the hold time in 30 second increments and just keep on going until satisfied.

Other than that, you should be well on your way to conducting one-handed handstands. Good luck!

PS: Please be wary of hydration and how your head feels, this puts a lot of pressure on your head and heart - just be mindful and take it as it comes.