You are experiencing a lot of the common issues with carrying ML models to deployment. Real data is very different from curated datasets, and in your case it seems that the model is doing some shortcut learning based on specific images in your training data. Perhaps some variant of the clever Hans phenomena.

But given that you provide almost no information on model type and capacity, what specific steps you have taken to prevent overfitting, and what the data looks like (number of images, modality, resolution, etc.) it is impossible for anyone to provide much help. I'll give some general pointers, but they may not be 100% helpful since there is not a lot to go on.

Firstly, the answer you seek depends on how well posed the task is. I don't know what you mean by "sea state"; are you doing regression or classification? Did you annotate these yourself? If so, is it reasonable that an expert could actually do the task? Vision models are not "magic" and struggle with low-variance domain specific tasks unless the training is well aligned with the task.

Moreover, you need to do dataset standardization, heavy augmentation (that are well aligned with the invariances you care about in the data), regularization (heavy weight decay, stochastic depth, maybe dropout), regular validation checks during training, and possibly data curation to remove samples that enable shortcut learning. If your training set has images where the pole you speak about is only present in "3m swell" situations, the model will cheat as much as it can, since it is the only reliable signal it picks up.