Gogs: Internal error (out of memory, realloc failed) during clone


#1

Using Gogs 0.11.66 and then Gogs 0.11.86 on a Debian 9.7 server. with sqlite3
I have not tried to reproduce this at try.gogs.io because it involves a very large repo - the Linux kernel.

I have been using Gogs successfully for LInux Kernel work for years now (thank you SO much!)
But when I checked in the latest kernel update from upstream (4.19.21 as it happens) and then tried to clone that … disaster.:

remote: fatal: Out of memory, realloc failed
remote: aborting due to possible repository corruption on the remote side.
fatal: early EOF
fatal: index-pack failed
error: git upload-pack: git-pack-objects died with error.
fatal: git upload-pack: aborting due to possible repository corruption on the remote side.
Gogs: Internal error

The repo is ok on disk (I can check out locally on the server, from the gogs-repostories folder, not using Gogs)
All other repos on my Gogs installation are still working fine
This is completely repeatable. I’ve upgraded Gogs, restarted gogs, restarted the server, everything I can think of, but I always see this failure.
According to the Gogs dashboard, it is only using a small amount of RAM:

Current Memory Usage 18 MB
Total Memory Allocated 56 MB
Memory Obtained 64 MB

I can’t see anywhere where any memory limit for Gogs would be set, so I can’t tweak anything.
I see no errors or other issues in the Gogs “System Notices” log
I am really puzzled by this problem … is there anything else I can try to debug this?


#2

There may be something odd about this repo after all. Becuase I was stuck with this problem I needed to solve quickly, and because I’d been able to check out “locally” on the server (directly accessing the repo on disk) I created a new repo via gogs and pushed the checked-out branch to that repo – and it’s fine!

I’d still rather like to understand why Gogs is throwing an internal error for the original repo, but at least this is a sort of way around it that works for me just now.


#3

You should check log in path_to_gogs/logs/hook/* for exact error message.


#4

There are only 0-byte files in the hooks folder, except post-receive.log which has a seemingly irrelevant line:

2019/02/14 14:07:47 [ERROR] […ogs/models/action.go:496 CommitRepoAction()] UpdateIssuesCommit: invalid issue reference [ref: net/mlx5]

and the higher level gogs.log just shows a single line related to the error:

2019/02/15 22:54:08 [ERROR] […/routes/repo/http.go:260 serviceRPC()] HTTP.serviceRPC: fail to serve RPC ‘upload-pack’: exit status 128 - error: git upload-pack: git-pack-objects died with error.

Perhaps I am missing some logging options?


#5

The closest answer I found is this: https://confluence.atlassian.com/bitbucketserverkb/git-operations-fail-due-to-git-pack-objects-died-with-error-849169949.html


#6

Oh wow. Sorry. I assumed it was a Gogs error due to the “Gogs: Internal error” line and the fact I could check out directly without a problem. But yes… git fsck finds faults AND git repack dies! The repo really does have some screwups after all. I’ll go away and dig in to this independently of Gogs. Thanks for your help! :slight_smile:

ian@moschops $ git fsck
Checking object directories: 100% (256/256), done.
warning in tag 5dc01c595e6c6ec9ccda4f6f69c131c0dd945f8c: missingTaggerEntry: invalid format - expected ‘tagger’ line
warning in tag 26791a8bcf0e6d33f43aef7682bdb555236d56de: missingTaggerEntry: invalid format - expected ‘tagger’ line
warning in tag 9e734775f7c22d2f89943ad6c745571f1930105f: missingTaggerEntry: invalid format - expected ‘tagger’ line
warning in tag 0397236d43e48e821cce5bbe6a80a1a56bb7cc3a: missingTaggerEntry: invalid format - expected ‘tagger’ line
warning in tag ebb5573ea8beaf000d4833735f3e53acb9af844c: missingTaggerEntry: invalid format - expected ‘tagger’ line
warning in tag 06f6d9e2f140466eeb41e494e14167f90210f89d: missingTaggerEntry: invalid format - expected ‘tagger’ line
warning in tag 701d7ecec3e0c6b4ab9bb824fd2b34be4da63b7e: missingTaggerEntry: invalid format - expected ‘tagger’ line
warning in tag 254cc2c3d0745835f73f5c9a76482b691668a738: missingTaggerEntry: invalid format - expected ‘tagger’ line
warning in tag 279c674f31885b18fd312149e7938850f7bd13bb: missingTaggerEntry: invalid format - expected ‘tagger’ line
warning in tag ea5a02080269424e629de1e7f1f385ba9be337ba: missingTaggerEntry: invalid format - expected ‘tagger’ line
warning in tag 733ad933f62e82ebc92fed988c7f0795e64dea62: missingTaggerEntry: invalid format - expected ‘tagger’ line
warning in tag c521cb0f10ef2bf28a18e1cc8adf378ccbbe5a19: missingTaggerEntry: invalid format - expected ‘tagger’ line
warning in tag a339981ec18d304f9efeb9ccf01b1f04302edf32: missingTaggerEntry: invalid format - expected ‘tagger’ line
Checking objects: 100% (7123884/7123884), done.
Checking connectivity: 7049060, done.
dangling commit df6b09f210cd52f5ef8a01b151be61c45239eced
dangling commit 0bc249b3aab6144b2055b1b698059cd68ec69646
dangling commit e58f4cd30d9c100d0d1b728e6c21ffb2a20606f6
dangling commit 034d690a920956925e9bc7e906b61a196936ad5d
dangling commit 214b9588327b014cdb2c7aba632d013ef2eefb4a
dangling commit f39e97820f8d8a076f6b5611f38dc033fdb4d786
dangling commit ad7babbfd69cef023fed9a4b6f57adcc2b902042
dangling commit 4fc5b63ac475837e5629a87777f47357be2e0105
dangling commit 4f60df1be2826dd08f7bf58fd551547f1478a10b
dangling commit 311ff7dbbf67ce4df247d0dc16424d30171011bc

ian@moschops $ git repack -adf --window=200 --depth=200
fatal: Out of memory, realloc failed
ian@moschops $


#7

Backup before kernel update! lol.