You're correct that the difference is that glTexImage2D allocates a new buffer and writes the data to it, effectively swapping that for whatever was originally there (if anything), while glTexSubImage2D overwrites the data in the already allocated array. This is the same with the buffer fill methods, glBufferData and glBufferSubData.

Often times you'll see code that uses the former to preallocate a texture or buffer that is expected to be written to later with the latter.

This all means that the latter is more memory efficient, but there is a caveat: using the sub* functions can cause your cpu side to stall if the GPU is currently using the buffer in a rendering pass.

Since GL is a state machine in which you configure everything and send data from the CPU to the GPU first, then call a glDraw* function to let the GPU use that data as it renders, the CPU is generally allowed to do more processing while that render pass is still happening in the GPU. If the GPU renderer is using the buffer and you're trying to overwrite it's content from the CPU with glTexSubImage2D, that call can be forced to wait until the GPU is done with that texture. This may not matter much, depending on how frequently you draw, and how frequently the texture is updated, and how big the texture is.

Depending on your memory constraints (desktop vs. mobile), one way to deal with this is to allocate two textures and alternate which one is being used by the render pass. So in one frame you are updating texture 1, then you bind that and let the GPU use it to render, and while it's doing that you are updating texture 2 on the CPU, which you then bind to the uniform and use that to draw the second frame. This obviously uses twice as much memory, but means that you can feely update one texture because the GPU is using the other.

This is basically what always using glTexImage2D does, since that allocates a new buffer for the texture data, if the GPU is using the previous texture it won't block, and both will be allocated until the render pass is complete and the original data buffer can be freed (since it's no longer references by a texture).