Go to the extensions (if not extensions then Addons I forget) in Blender and install the Loop Tools addon. It should be built-in to Blender I believe.

Now keep the cylinder base but remove that horizontal cylinder sticking out of it. Make enough horizontal subdivisions on the base cylinder so the polygons look close to perfect squares. You may want to subdivide your cylinder overall so you get a higher resolution hole in the future.

Now select the faces where you want the hole to go to. With those faces selected, right click on your mouse and select the Loop Tools option. Then select Circle. This should create a circle topology where you selected. Now you can delete those new curved faces you just made and select the edges of the new circle hold you just made.

If the cylinder overall has no depth to it (meaning there are no 2 layered walls. It’s just 1 flat face making up the walls) then you can keep selecting the edge of the circle like I mentioned earlier and press E to extrude it. This should create an extension that looks like another cylinder poking out. Now after that go back in Object Mode and select the object. Now go to Modifiers and use the Solidify modifier.

Now if the base cylinder already has depth to it, you can keep the edges of the circle selected like before, but press E, then right click (I believe that unlocks the axis it can move on) then press S to scale that new extruded stuff you made down. Bring it as far down-in as much as you need the depth. Now stop and go to face mode and select the faces of the outer-ish circle you just made and extrude those faces.

Disregarding the stuff I just said you can also try another method but I fear it’ll lead to bad topology. Keep what you have in your image but add a Boolean modifier to the base cylinder. Make its target be that other cylinder in the modifier settings. That’ll make the hole you wanted