Intermittent 504 Clone Failures on Build in OpenShift

We are running Gogs Version: 0.11.34.1122 as an application inside OpenShift 3.11 cluster. We are performing s2i builds inside of OpenShift using internal gogs repos as source. Gogs frontend service is exposed via a route. We are generally successful when performing these s2i builds with internal gogs repo as source for the builds.

It seems that a couple of the repos with the most activity - hundreds of commits / pushes, a few hundred builds, begin exhibiting 504 failures during the git clone process of the OpenShift / Jenkins build. We can retry the build 10-20 times before it will succeed with the git clone, and begin the build process. We can also point the build at a different repo with identical contents (that we duplicated via git clone --mirror) and the build will be successful.

Any ideas on what would cause this behavior, and how to fix the root cause?

We are seeing this error:

2020/06/05 18:11:02 [ERROR] [.../routes/repo/http.go:284 serviceRPC()] HTTP.serviceRPC: fail to serve RPC 'upload-pack': exit status 141 - {error: git-upload-pack died of signal 13
 %!s(int=0)  %!s(bytes.readOp=0)}

Hi, Gogs itself does not care/control how long an operation could be, it’s likely that the reverse proxy in place halted process in the middle of executing Git command while serving Git repository to clients.

0.11.34 is fairly an old version, any chance you could upgrade to at least 0.11.91 see if problem still exists?

Thanks for the reply! I’m not sure we could update, but I’ll look into that. We are seeing the issue occur again, months later, with the same repo that was giving us problems before.

What’s strange, is that I git clone -mirror to bring it local, then git push --mirror to a new repo, and the build is now completing when pointing to the new repo with identical content, but again consistently failing when pointing to the original repo.

Can I attach the log in this thread somewhere?

Yes, definitely. Attach a log file or as code block both fine.

[Macaron] 2020-10-27 19:28:05: Completed /PhoneProvisioning/MSProvisioningPhoneProvisioningHandler.git/git-upload-pack 500 Internal Server Error in 49.511641228s

^ this is the line related to the repo in question. It times out when trying to clone to it from the builder pod.

2020/10/27 19:25:20 [TRACE] Custom path: /etc/gogs
2020/10/27 19:25:20 [TRACE] Log path: /opt/gogs/log
2020/10/27 19:25:20 [TRACE] Log Mode: Console (Trace)
2020/10/27 19:25:20 [ INFO] Gogs 0.11.34.1122
2020/10/27 19:25:20 [ INFO] Cache Service Enabled
2020/10/27 19:25:20 [ INFO] Session Service Enabled
2020/10/27 19:25:21 [ INFO] Git Version: 1.8.3.1
2020/10/27 19:25:21 [ INFO] Git config user.name set to Gogs
2020/10/27 19:25:21 [ INFO] Git config user.email set to [email protected]
2020/10/27 19:25:43 [ INFO] Run Mode: Production
2020/10/27 19:25:43 [TRACE] Doing: CheckRepoStats
2020/10/27 19:25:43 [ INFO] Listen: http://0.0.0.0:3000
[Macaron] 2020-10-27 19:26:34: Started GET / for 10.159.2.1
2020/10/27 19:26:34 [TRACE] Session ID: a782f184aad22f90
2020/10/27 19:26:34 [TRACE] CSRF Token: feGIcvkmYglHhkc0mA49XYvp8Fo6MTYwMzgyNjc5NDU0NDQzNDIwOQ==
2020/10/27 19:26:34 [TRACE] Template: home
[Macaron] 2020-10-27 19:26:34: Completed / 200 OK in 2.95188ms
[Macaron] 2020-10-27 19:26:54: Started GET / for 10.159.2.1
2020/10/27 19:26:54 [TRACE] Session ID: 720f08b73668a3d8
2020/10/27 19:26:54 [TRACE] CSRF Token: ZAB77hxG7yIded48UlXaux8eG-06MTYwMzgyNjgxNDU0MzgyNDkxNw==
2020/10/27 19:26:54 [TRACE] Template: home
[Macaron] 2020-10-27 19:26:54: Completed / 200 OK in 1.39089ms
[Macaron] 2020-10-27 19:26:56: Started GET / for 10.159.2.1
2020/10/27 19:26:56 [TRACE] Session ID: 2bd95571f1ec2758
2020/10/27 19:26:56 [TRACE] CSRF Token: eSMVTXpAMz5i2SvCRkGCesbyTvU6MTYwMzgyNjgxNjUyMjY4Njk4MQ==
2020/10/27 19:26:56 [TRACE] Template: home
[Macaron] 2020-10-27 19:26:56: Completed / 200 OK in 1.692888ms
[Macaron] 2020-10-27 19:27:01: Started GET /PhoneProvisioning/surgeMail.git/info/refs?service=git-upload-pack for 10.155.235.4
2020/10/27 19:27:01 [TRACE] Session ID: e10576688e18c8bb
2020/10/27 19:27:01 [TRACE] CSRF Token: -OyMRr5AarEyyOFCIKYYhkep97U6MTYwMzgyNjgyMTQ3ODg2OTIzMA==
[Macaron] 2020-10-27 19:27:01: Completed /PhoneProvisioning/surgeMail.git/info/refs?service=git-upload-pack 200 OK in 179.954978ms
[Macaron] 2020-10-27 19:27:06: Started GET / for 10.159.2.1
2020/10/27 19:27:06 [TRACE] Session ID: 6afb7014a4531cd6
2020/10/27 19:27:06 [TRACE] CSRF Token: FjyLd6oqE_cGowJfCXHi6qeCcF06MTYwMzgyNjgyNjUyMjUxMTk4NQ==
2020/10/27 19:27:06 [TRACE] Template: home
[Macaron] 2020-10-27 19:27:06: Completed / 200 OK in 1.373291ms
[Macaron] 2020-10-27 19:27:07: Started GET /PhoneProvisioning/serviceability.git/info/refs?service=git-upload-pack for 10.155.235.4
2020/10/27 19:27:07 [TRACE] Session ID: b4241460bbbcbc65
2020/10/27 19:27:07 [TRACE] CSRF Token: oSEm5lWNtlEeloSzA0HRPnGO1eU6MTYwMzgyNjgyNzEyNDM1MzI5OQ==
2020/10/27 19:27:07 [TRACE] Template: status/404
[Macaron] 2020-10-27 19:27:07: Completed /PhoneProvisioning/serviceability.git/info/refs?service=git-upload-pack 404 Not Found in 39.257534ms
[Macaron] 2020-10-27 19:27:07: Started GET /PhoneProvisioning for 10.155.235.4
2020/10/27 19:27:07 [TRACE] Session ID: 2bfaac9daaa4ffbf
2020/10/27 19:27:07 [TRACE] CSRF Token: AoWuCP-lqvitdGxdBX124Mq-peU6MTYwMzgyNjgyNzQ0NTQ2NDAxOA==
2020/10/27 19:27:07 [TRACE] Template: org/home
[Macaron] 2020-10-27 19:27:07: Completed /PhoneProvisioning 200 OK in 105.547383ms
[Macaron] 2020-10-27 19:27:07: Started GET /img/favicon.png for 10.155.235.4
[Macaron] [Static] Serving /img/favicon.png
[Macaron] 2020-10-27 19:27:07: Completed /img/favicon.png 200 OK in 918.594µs
[Macaron] 2020-10-27 19:27:14: Started GET / for 10.159.2.1
2020/10/27 19:27:14 [TRACE] Session ID: 2f0230a2ced8c533
2020/10/27 19:27:14 [TRACE] CSRF Token: cBfXDnhuwG-GWi60_C289IUO5Ag6MTYwMzgyNjgzNDU0MzI0MzcyNg==
2020/10/27 19:27:14 [TRACE] Template: home
[Macaron] 2020-10-27 19:27:14: Completed / 200 OK in 3.690375ms
[Macaron] 2020-10-27 19:27:14: Started GET /PhoneProvisioning/MSProvisioningPhoneProvisioningHandler.git/info/refs?service=git-upload-pack for 10.155.235.4
2020/10/27 19:27:14 [TRACE] Session ID: 74de77ade3df1f91
2020/10/27 19:27:14 [TRACE] CSRF Token: ayjTeuqm1zuDp0k7pMSX59dGt7M6MTYwMzgyNjgzNDkzMDQxODI5Nw==
[Macaron] 2020-10-27 19:27:15: Completed /PhoneProvisioning/MSProvisioningPhoneProvisioningHandler.git/info/refs?service=git-upload-pack 200 OK in 219.44701ms
[Macaron] 2020-10-27 19:27:15: Started GET /PhoneProvisioning/MSProvisioningPhoneProvisioningHandler.git/info/refs?service=git-upload-pack for 10.155.235.4
2020/10/27 19:27:15 [TRACE] Session ID: 9d2f5fb2bb35b973
2020/10/27 19:27:15 [TRACE] CSRF Token: km4yXpV2zrWOxcOmFe8rgC8I5dU6MTYwMzgyNjgzNTMzOTQ5NzQyMA==
[Macaron] 2020-10-27 19:27:15: Completed /PhoneProvisioning/MSProvisioningPhoneProvisioningHandler.git/info/refs?service=git-upload-pack 200 OK in 98.261233ms
[Macaron] 2020-10-27 19:27:15: Started POST /PhoneProvisioning/MSProvisioningPhoneProvisioningHandler.git/git-upload-pack for 10.155.235.4
2020/10/27 19:27:15 [TRACE] Session ID: 77fc7e17b940caf0
2020/10/27 19:27:15 [TRACE] CSRF Token: VCe4uUSCJKUTZZgMWPv2DfhoODc6MTYwMzgyNjgzNTUwMTczOTQxOA==
[Macaron] 2020-10-27 19:27:16: Started GET / for 10.159.2.1
2020/10/27 19:27:16 [TRACE] Session ID: 5be5ec6f421ee55c
2020/10/27 19:27:16 [TRACE] CSRF Token: DWPuCQL42P21uj8aWKKFZ_9EohI6MTYwMzgyNjgzNjUyMzE2ODY4Mw==
2020/10/27 19:27:16 [TRACE] Template: home
[Macaron] 2020-10-27 19:27:16: Completed / 200 OK in 3.663376ms
[Macaron] 2020-10-27 19:27:26: Started GET / for 10.159.2.1
2020/10/27 19:27:26 [TRACE] Session ID: 171acc6c68f7d876
2020/10/27 19:27:26 [TRACE] CSRF Token: W67tD_2gOaOQpwj5Uzhq2hbkmTY6MTYwMzgyNjg0NjUyMjYwMTg4OQ==
2020/10/27 19:27:26 [TRACE] Template: home
[Macaron] 2020-10-27 19:27:26: Completed / 200 OK in 2.246585ms
[Macaron] 2020-10-27 19:27:34: Started GET / for 10.159.2.1
2020/10/27 19:27:34 [TRACE] Session ID: b19d39c85db3396e
2020/10/27 19:27:34 [TRACE] CSRF Token: 2VZflgWDWxdDyf3ua1GUbXPXCuI6MTYwMzgyNjg1NDU0MzQ2NDUyOQ==
2020/10/27 19:27:34 [TRACE] Template: home
[Macaron] 2020-10-27 19:27:34: Completed / 200 OK in 2.427184ms
[Macaron] 2020-10-27 19:27:36: Started GET / for 10.159.2.1
2020/10/27 19:27:36 [TRACE] Session ID: 058a43d98f6d9955
2020/10/27 19:27:36 [TRACE] CSRF Token: DhRNC9MkundvWWPYpHEzlo966YY6MTYwMzgyNjg1NjUyMjQ2Njg5Mg==
2020/10/27 19:27:36 [TRACE] Template: home
[Macaron] 2020-10-27 19:27:36: Completed / 200 OK in 1.703389ms
[Macaron] 2020-10-27 19:27:41: Started GET /user/login?redirect_to=%2fPhoneProvisioning for 10.155.235.4
2020/10/27 19:27:41 [TRACE] Session ID: 2bfaac9daaa4ffbf
2020/10/27 19:27:41 [TRACE] CSRF Token: AoWuCP-lqvitdGxdBX124Mq-peU6MTYwMzgyNjgyNzQ0NTQ2NDAxOA==
2020/10/27 19:27:41 [TRACE] Template: user/auth/login
[Macaron] 2020-10-27 19:27:41: Completed /user/login?redirect_to=%2fPhoneProvisioning 200 OK in 2.347484ms
[Macaron] 2020-10-27 19:27:43: Started POST /user/login for 10.155.235.4
2020/10/27 19:27:43 [TRACE] Session ID: 2bfaac9daaa4ffbf
2020/10/27 19:27:43 [TRACE] CSRF Token: AoWuCP-lqvitdGxdBX124Mq-peU6MTYwMzgyNjgyNzQ0NTQ2NDAxOA==
[Macaron] 2020-10-27 19:27:43: Completed /user/login 302 Found in 55.92832ms
[Macaron] 2020-10-27 19:27:43: Started GET /PhoneProvisioning for 10.155.235.4
2020/10/27 19:27:43 [TRACE] Session ID: 2bfaac9daaa4ffbf
2020/10/27 19:27:43 [TRACE] CSRF Token: 9sbVtHQN8w0lKYen7bkeKQ-6SBQ6MTYwMzgyNjg2MzIzNDEyODcyMg==
2020/10/27 19:27:43 [TRACE] Template: org/home
[Macaron] 2020-10-27 19:27:43: Completed /PhoneProvisioning 200 OK in 533.738376ms
[Macaron] 2020-10-27 19:27:46: Started GET / for 10.159.2.1
2020/10/27 19:27:46 [TRACE] Session ID: 1067906e7702f4ad
2020/10/27 19:27:46 [TRACE] CSRF Token: XTwZXMTDR3CejtBPGAafFNr0zBg6MTYwMzgyNjg2NjUyMjYxNjA5NA==
2020/10/27 19:27:46 [TRACE] Template: home
[Macaron] 2020-10-27 19:27:46: Completed / 200 OK in 2.356084ms
[Macaron] 2020-10-27 19:27:54: Started GET / for 10.159.2.1
2020/10/27 19:27:54 [TRACE] Session ID: 72a7cfce1a8a9804
2020/10/27 19:27:54 [TRACE] CSRF Token: 2eRVLoLBME46ey5R2pMpkCR4Jfo6MTYwMzgyNjg3NDU0MzY3ODMzMw==
2020/10/27 19:27:54 [TRACE] Template: home
[Macaron] 2020-10-27 19:27:54: Completed / 200 OK in 1.51379ms
[Macaron] 2020-10-27 19:27:56: Started GET / for 10.159.2.1
2020/10/27 19:27:56 [TRACE] Session ID: 657a856d47b98847
2020/10/27 19:27:56 [TRACE] CSRF Token: rX4gMi0Mmz7W44XZAf_kF-fa-p86MTYwMzgyNjg3NjUyMjY2Nzg5Ng==
2020/10/27 19:27:56 [TRACE] Template: home
[Macaron] 2020-10-27 19:27:56: Completed / 200 OK in 1.50179ms
[Macaron] 2020-10-27 19:28:02: Started GET /PhoneProvisioning/serviceability.git/info/refs?service=git-upload-pack for 10.155.235.4
2020/10/27 19:28:02 [TRACE] Session ID: d16a10134c023f2f
2020/10/27 19:28:02 [TRACE] CSRF Token: AUjrHoQYEhP-F8l9EUzRQ21yTLY6MTYwMzgyNjg4MjE0MzU4NzMzMQ==
2020/10/27 19:28:02 [TRACE] Template: status/404
[Macaron] 2020-10-27 19:28:02: Completed /PhoneProvisioning/serviceability.git/info/refs?service=git-upload-pack 404 Not Found in 19.988864ms
2020/10/27 19:28:05 http: multiple response.WriteHeader calls
2020/10/27 19:28:05 [ERROR] […/routes/repo/http.go:284 serviceRPC()] HTTP.serviceRPC: fail to serve RPC ‘upload-pack’: exit status 141 - {error: git-upload-pack died of signal 13
%!s(int=0) %!s(bytes.readOp=0)}
[Macaron] 2020-10-27 19:28:05: Completed /PhoneProvisioning/MSProvisioningPhoneProvisioningHandler.git/git-upload-pack 500 Internal Server Error in 49.511641228s
[Macaron] 2020-10-27 19:28:06: Started GET /PhoneProvisioning/surgeMail.git/info/refs?service=git-upload-pack for 10.155.235.4
2020/10/27 19:28:06 [TRACE] Session ID: 3a795e90f1025c44
2020/10/27 19:28:06 [TRACE] CSRF Token: 1ESJdLYvvfajkZZCDKBxMj3yO0U6MTYwMzgyNjg4NjUxNDUyMDM1Mw==
[Macaron] 2020-10-27 19:28:06: Started GET / for 10.159.2.1
2020/10/27 19:28:06 [TRACE] Session ID: 1e462d68b0f666cb
2020/10/27 19:28:06 [TRACE] CSRF Token: rahYDIZ6EuQVcPe2je0yaUpgtl06MTYwMzgyNjg4NjUyMjQ1MTI5OQ==
2020/10/27 19:28:06 [TRACE] Template: home
[Macaron] 2020-10-27 19:28:06: Completed / 200 OK in 1.552389ms
[Macaron] 2020-10-27 19:28:06: Completed /PhoneProvisioning/surgeMail.git/info/refs?service=git-upload-pack 200 OK in 118.831993ms
[Macaron] 2020-10-27 19:28:14: Started GET / for 10.159.2.1
2020/10/27 19:28:14 [TRACE] Session ID: f898458a4754185c
2020/10/27 19:28:14 [TRACE] CSRF Token: vIuMKQOaoCSt61a3BPkOn9jvF5c6MTYwMzgyNjg5NDU0MzQyNTEzOQ==
2020/10/27 19:28:14 [TRACE] Template: home
[Macaron] 2020-10-27 19:28:14: Completed / 200 OK in 1.811387ms
[Macaron] 2020-10-27 19:28:16: Started GET / for 10.159.2.1
2020/10/27 19:28:16 [TRACE] Session ID: ebe14fe1dfc4b3e3
2020/10/27 19:28:16 [TRACE] CSRF Token: TAJhbJZtx09LyBr_A5JSOoSsj7w6MTYwMzgyNjg5NjUyMjY5NjgwMA==
2020/10/27 19:28:16 [TRACE] Template: home
[Macaron] 2020-10-27 19:28:16: Completed / 200 OK in 1.626989ms
[Macaron] 2020-10-27 19:28:26: Started GET / for 10.159.2.1
2020/10/27 19:28:26 [TRACE] Session ID: afd348cf35970d59
2020/10/27 19:28:26 [TRACE] CSRF Token: 2-FYyWZc1GBmKKtr0Z3zol0tQ3Y6MTYwMzgyNjkwNjUyMjY2MTMwMw==
2020/10/27 19:28:26 [TRACE] Template: home
[Macaron] 2020-10-27 19:28:26: Completed / 200 OK in 1.681388ms