Intermittent 504 Clone Failures on Build in OpenShift

We are running Gogs Version: as an application inside OpenShift 3.11 cluster. We are performing s2i builds inside of OpenShift using internal gogs repos as source. Gogs frontend service is exposed via a route. We are generally successful when performing these s2i builds with internal gogs repo as source for the builds.

It seems that a couple of the repos with the most activity - hundreds of commits / pushes, a few hundred builds, begin exhibiting 504 failures during the git clone process of the OpenShift / Jenkins build. We can retry the build 10-20 times before it will succeed with the git clone, and begin the build process. We can also point the build at a different repo with identical contents (that we duplicated via git clone --mirror) and the build will be successful.

Any ideas on what would cause this behavior, and how to fix the root cause?

We are seeing this error:

2020/06/05 18:11:02 [ERROR] [.../routes/repo/http.go:284 serviceRPC()] HTTP.serviceRPC: fail to serve RPC 'upload-pack': exit status 141 - {error: git-upload-pack died of signal 13
 %!s(int=0)  %!s(bytes.readOp=0)}