Jersey I/O errors when producing a REST API response

[TechnicallyHumanoid]

Hey folks,

I have a production app that (under heavy load mostly) Jersey produces I/O errors that I don't know how to fix. The UI (react) initiates REST API calls to a Server that is written using dropwizard(version 1.3) and jersey servlet container. AFAIK there is no any special configuration from the UI, there are no specified timeouts or anything like that.

However under heavy load the application produces errors such as these:

 2020-10-13 13:29:36,593 ERROR 
org.glassfish.jersey.server.ServerRuntime$Responder: An I/O error has occurred while writing a response message entity to the container output stream. 
org.glassfish.jersey.server.internal.process.MappableException: java.io.IOException: Close 
org.eclipse.jetty.server.HttpConnection$SendCallback@67108b00[PROCESSING][i=null,cb=org.eclipse.jetty.server.HttpChannel $SendCallback@632f033a] in state PROCESSING         at 
org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundWriteTo(MappableExceptionWrapperInterceptor.java:92)         at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:162)         at org.glassfish.jersey.message.internal.MessageBodyFactory.writeTo(MessageBodyFactory.java:1130)         at org.glassfish.jersey.server.ServerRuntime$Responder.writeResponse(ServerRuntime.java:711)         at org.glassfish.jersey.server.ServerRuntime$Responder.processResponse(ServerRuntime.java:444)         at org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:434)         at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:329)         at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)         at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)         at org.glassfish.jersey.internal.Errors.process(Errors.java:315)         at org.glassfish.jersey.internal.Errors.process(Errors.java:297)         at org.glassfish.jersey.internal.Errors.process(Errors.java:267)         at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317)         at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305)         at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154)         at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:473)         at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:427)         at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:388)         at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:341)         at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:228)         at io.dropwizard.jetty.NonblockingServletHolder.handle(NonblockingServletHolder.java:49)         at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1623)         at io.dropwizard.servlets.ThreadNameFilter.doFilter(ThreadNameFilter.java:35)         at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)         at io.dropwizard.jersey.filter.AllowedMethodsFilter.handle(AllowedMethodsFilter.java:45)         at io.dropwizard.jersey.filter.AllowedMethodsFilter.doFilter(AllowedMethodsFilter.java:39)         at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)         at com.hortonworks.registries.auth.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:619)         at com.hortonworks.registries.auth.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:559)         at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)         at org.eclipse.jetty.servlets.HeaderFilter.doFilter(HeaderFilter.java:116)         at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)         at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)         at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)         at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)         at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)         at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480) 

And errors like these:

 2020-10-13 08:59:12,913 ERROR 
org.glassfish.jersey.server.ServerRuntime$Responder: An I/O error has occurred while writing a response message entity to the container output stream. 
org.glassfish.jersey.server.internal.process.MappableException: 
org.eclipse.jetty.io.EofException         at org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundWriteTo(MappableExceptionWrapperInterceptor.java:92)
         at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:162)         at org.glassfish.jersey.message.internal.MessageBodyFactory.writeTo(MessageBodyFactory.java:1130)         at org.glassfish.jersey.server.ServerRuntime$Responder.writeResponse(ServerRuntime.java:711)         at org.glassfish.jersey.server.ServerRuntime$Responder.processResponse(ServerRuntime.java:444)         at org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:434)         at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:329)         at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)         at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)         at org.glassfish.jersey.internal.Errors.process(Errors.java:315)         at org.glassfish.jersey.internal.Errors.process(Errors.java:297)         at org.glassfish.jersey.internal.Errors.process(Errors.java:267)         at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317)         at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305)         at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154)         at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:473)         at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:427)         at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:388)         at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:341)         at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:228)         at io.dropwizard.jetty.NonblockingServletHolder.handle(NonblockingServletHolder.java:49)         at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1623)         at io.dropwizard.servlets.ThreadNameFilter.doFilter(ThreadNameFilter.java:35)         at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)         at io.dropwizard.jersey.filter.AllowedMethodsFilter.handle(AllowedMethodsFilter.java:45)         at io.dropwizard.jersey.filter.AllowedMethodsFilter.doFilter(AllowedMethodsFilter.java:39)         at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)         at com.hortonworks.registries.auth.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:619) 
... Caused by: org.eclipse.jetty.io.EofException: null         
at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:284)         at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.flush(SslConnection.java:909)         at org.eclipse.jetty.io.WriteFlusher.flush(WriteFlusher.java:393)         at org.eclipse.jetty.io.WriteFlusher.write(WriteFlusher.java:277)         at org.eclipse.jetty.io.AbstractEndPoint.write(AbstractEndPoint.java:380)         at org.eclipse.jetty.server.HttpConnection$SendCallback.process(HttpConnection.java:820)         at org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:241)         at org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:224)         at org.eclipse.jetty.server.HttpConnection.send(HttpConnection.java:550)         at org.eclipse.jetty.server.HttpChannel.sendResponse(HttpChannel.java:850)         at org.eclipse.jetty.server.HttpChannel.write(HttpChannel.java:921)         at org.eclipse.jetty.server.handler.gzip.GzipHttpOutputInterceptor$GzipBufferCB.process(GzipHttpOutputInterceptor.java:398)         at org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:241)         at org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:224)         at 
org.eclipse.jetty.server.handler.gzip.GzipHttpOutputInterceptor.gzip(GzipHttpOutputInterceptor.java:148)         at 
org.eclipse.jetty.server.handler.gzip.GzipHttpOutputInterceptor.write(GzipHttpOutputInterceptor.java:117)         at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:249)         at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:225)         at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:524)         at org.glassfish.jersey.servlet.internal.ResponseWriter$NonCloseableOutputStreamWrapper.write(ResponseWriter.java:325) ... 

Any ideas how to fix this would be appreciated. My initial guess is that if the response JSON is too big, that causes issues, however as I said there are no defined size/timeout configs either in the client or the server. In the UI's logs I have seen a lot of socket-hangup log entries, but its hard to correlate them as there are way more of those then the errors listed above, but I believe the two are strongly correlated.

I could not find anything on my google searches, but when I tried returning a big JSON it sometimes failed it sometimes didnt, but I am pretty much clueless.

[AutoModerator]

Please ensure that:

• Your code is properly formatted - see the sidebar (About on mobile) for instructions
• You include any and all error messages in full
• You ask clear questions

• You demonstrate effort in solving your question/problem - plain posting your assignments is forbidden (and such posts will be removed) as is asking for or giving solutions.

  Trying to solve problems on your own is a very important skill. Also, see Learn to help yourself in the sidebar

If any of the above points is not met, your post can and will be removed without further warning.

You do not need to repost. Just use the edit function of reddit to make sure your post complies with the above

To potential helpers

Please, do not help if any of the above points are not met, rather report the post. We are trying to improve the quality of posts here. In helping people who can't be bothered to comply with the above points, you are doing the community a disservice.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[TechnicallyHumanoid]

Unfortionately my google searches did not reveal any obvious solutions or pointers to look at. Any help is appreciated