How to most effectively explore underwater?
I have found a few entrances to mineshafts and caverns at the ocean bottom, so I'd say it is certainly worth exploring any interesting features you come across, like weak sources of light (from lava or torches) or areas of oddly sunken sand. With the Update Aquatic update there are even more interesting things to find there, like Shipwrecks and Ocean ruins.
Here's a list of things you'll want for effective underwater exploration:
Potions of Water Breathing: Added in Java Edition 1.7.2, Potions of Water Breathing can give you either 3 or 8 minutes during which your oxygen bar won't decrease. It also slightly increases visibility under water. To brew them, you'll need to catch some Pufferfish.
A good helmet: First and foremost, you should get yourself a helmet enchanted with Respiration, ideally the max level of III. This lets your air last for much longer underwater. It also helps to have Aqua Affinity on the helmet too, which allows you to mine faster when underwater. If you don't get both of these when enchanting a single helmet, remember that the new anvil tool let's you combine two enchanted items so that you can consolidate and improve enchantments without having to keep rolling the dice at the enchantment table.
Waterproof lighting: You need to be able to see deep under water, so you have to bring waterproof sources of light. Torches are out, as water will knock them off, but jack-o-lanterns and glowstone blocks work great.
Making air pockets: Respiration-enchanted helmets can only do so much, especially if you want to reach really deep depths under water. You'll want to bring items that you can place on the ocean floor which will displace the water source blocks and give you an air pocket where you can catch your breath. With recent updates, objects that previously displaced water may not anymore, instead becoming Waterlogged. For example, ladders and signs used to work, but don't appear to anymore in Java Edition 1.13 onward. Doors still displace water as of Java Edition 1.15.2.
Use Optifine.
It has a clear water setting that will allow you to easily see through water, allowing you to explore without needing to dive to the bottom.
It's a client mod, so it'll work with all servers.
Example:
Clearwater Off
Clearwater On
You can use a bucket as a portable air source. When you are close to being out of air, right click with the empty bucket and you will have air. Remember to empty the bucket so you can repeat the trick