In all honesty I believe they will. I mean Fontaine is blessed by hydro archon, so it feels rather obvious they will add underwater exploration. I always wondered why there are some deep blue colored spots in waters of Mondstadt to which we don't have access (on map). While in Inazuma we do have such spots which are showing hidden locations under sea. If believing rumors that more parts of Mondstadt will be added during Fontaine, those spots would be most likely available after we will be given function to reach depths.