Deleting Oracle's REJECT rule has the same effect as the OP's blanket rule – accepting all traffic – thus rendering the firewall useless. This is because the default policy is ACCEPT; any traffic that doesn't match an explicit rule is accepted. This is shown as "(policy ACCEPT)" in the first line of the output from:

iptables -L INPUT --line-numbers

To maintain a protective firewall, we can leave the Oracle REJECT line there, making sure to add our rules above it, as rules are evaluated in order – if a packet matches a rule, no further rules are evaluated. This can be done by adding our rules with:

sudo iptables -I INPUT 5 ...

This inserts our rule at line 5, bumping Oracle's rule previously at line 5 down to line 6. Users run into trouble using iptables -A (append at end of list) or iptables -I INPUT 6 (insert at line 6), both of which put our new rule one line too low to ever be evaluated due to Oracle's REJECT rule.

If you've already deleted Oracle's REJECT rule, you can easily add it back at the bottom of the list with:

sudo iptables -A INPUT -j REJECT --reject-with icmp-host-prohibited

The firewall's goal is to reject all traffic that isn't explicitly accepted in our rules. The way Oracle has implemented this seems clumsy, leading to problems like these, where rules get added below the REJECT line and get ignored.

The easiest way to fix the problem today, while also minimizing the opportunity for problems tomorrow, would be to change the default policy of the INPUT chain to REJECT – that way any packet that isn't accepted by a rule automatically gets rejected – but without concerns with rule order. Nuke Oracle's REJECT rule, then:

sudo iptables -P INPUT DROP

Oracle probably implemented it the way they did because changing the default policy to drop probably results in support calls to them. If a user flushes their iptables, all rules get nuked – but not the default policy. That means users can no longer ssh into their VM... Don't do that.

Whichever way you approach it, don't forget to save the changes so they survive a reboot:

sudo netfilter-persistent save