How to backup, restore and migrate


#21

My gogs version is :

Gogs version 0.11.29.0727

(the version number is the same as the latest).
I unziped the backup file,the sessions dir are included.
image

Thank you very much for your help and patience.


#22

Thanks! I think you can file an issue on GitHub as a record, because from the code backup should only include “attachments” and “avatars” subdirs under “data” directory.


#23

Hello @all.

I’m new to Gogs, also to git. So my question may be somehow strange. But I’ve got a curious problem. I’ve setup Gogs on Win 7 Pro 64 bit. It’s running as a service on this Windows box. Then I’ve wrote a simple batch file to get a backup of Gogs configs and also the repositories. The problem: The batch works fine when I’m logged on to the system. But running exactly the same batch from task scheduler, it looks like that the command gogs.exe backup will not create the backup in non-interactive mode.

First I thought it may caused by the user. But I’ve testet this with several users with the same result…

Does anybody here got the same experience? Or does anybody know any hints to solve the problem?

Regards

ThirtySomething

PS: The batch looks like this one

@echo off
d:
cd \gogs
d:\gogs\gogs.exe backup
xcopy gogs-backup-*.zip \\server\development\backup\gogs\ /c /v /y /j
del /f /q gogs-backup-*.zip
echo.
cd \

#24

Sorry, forgot most important information: I’m running Gogs in version 0.11.19.0609.


#25

还原时,提示:
[root@switch-subversion /home/git]# su - git -c "/opt/gogs/gogs restore --from=gogs-backup-20170919104250.zip"
2017/09/19 10:56:33 [ INFO] Restore backup from: gogs-backup-20170919104250.zip
2017/09/19 10:56:40 [FATAL] Fail to import ‘custom’: rename /tmp/gogs-backup/custom /opt/gogs/custom: invalid cross-device link

/opt和/tmp确实是在不同的磁盘上

报错:
invalid cross-device link
不可用的硬链接,没办法,我们采用了不同的磁盘
解决:
不过可以指定临时目录–tempdir=/opt/gogs/tmp
mkdir /opt/gogs/tmp
chown git:git /opt/gogs/tmp

权限不对:
[FATAL] Fail to extract ‘repositories.zip’: open 省略: permission denied
解决:
chmod u+w -R /opt/gogs-repositories/

成功还原的的过程:
chown git:git /opt/gogs
mkdir /opt/gogs/tmp
chown git:git /opt/gogs/tmp
mv /opt/gogs/custom.bak /root/recycle_bin/
chmod u+w -R /opt/gogs-repositories/
su - git -c “/opt/gogs/gogs restore --config=/opt/gogs/custom/conf/app.ini --tempdir=/opt/gogs/tmp --from=gogs-backup-20170919104250.zip”

还原之后,代码确实恢复了,但发现“版本发布”即已推送的标签tag还在


#27

备份详细帮助:
su - git -c "/opt/gogs/gogs backup -h"
NAME:
gogs backup - Backup files and database

USAGE:
gogs backup [command options] [arguments…]

DESCRIPTION:
Backup dumps and compresses all related files and database into zip file,
which can be used for migrating Gogs to another server. The output format is meant to be
portable among all supported database engines.

OPTIONS:
–config value, -c value Custom configuration file path (default: “custom/conf/app.ini”)
–verbose, -v Show process details
–tempdir value, -t value Temporary directory path (default: “/tmp”)
–target value Target directory path to save backup archive (default: “./”)
–archive-name value Name of backup archive (default: “gogs-backup-20170919114931.zip”)
–database-only Only dump database
–exclude-repos Exclude repositories

可以指定备份目标目录:
su - git -c “/opt/gogs/gogs backup --tempdir /opt/gogs/tmp --target /opt/gogs_bak/”

还原的详细帮助:
su - git -c "/opt/gogs/gogs restore -h"
NAME:
gogs restore - Restore files and database from backup

USAGE:
gogs restore [command options] [arguments…]

DESCRIPTION:
Restore imports all related files and database from a backup archive.
The backup version must lower or equal to current Gogs version. You can also import
backup from other database engines, which is useful for database migrating.

If corresponding files or database tables are not presented in the archive, they will
be skipped and remain unchanged.

OPTIONS:
–config value, -c value Custom configuration file path (default: “custom/conf/app.ini”)
–verbose, -v Show process details
–tempdir value, -t value Temporary directory path (default: “/tmp”)
–from value Path to backup archive
–database-only Only import database
–exclude-repos Exclude repositories


#28

I’ve done a successful backup, but now trying to restore it and getting this.

2 things:

  • I’m performing the backup on one instance, and restoring it on another (new, empty) instance
  • both instances are running in Docker containers.

a25ff52a1650:/app/gogs$ ./gogs restore --from /data/gogs
gogs/ gogs-backup-20170919192203.zip
a25ff52a1650:/app/gogs$ ./gogs restore --from /data/gogs-backup-20170919192203.zip
2017/09/19 19:37:01 [ INFO] Restore backup from: /data/gogs-backup-20170919192203.zip
2017/09/19 19:37:27 [FATAL] Fail to import database: fail to insert strcut: database disk image is malformed


#29

On Ubuntu 16.04 using the packager.io build, I had to do a few tweaks, some undocumented, to be able to perform a backup. Here is the command line I’ve had to use:

sudo -u gogs HOME=/var/www/gogs GOGS_CUSTOM=/etc/gogs /opt/gogs/gogs backup -v --exclude-repos

  • Have to sudo, because gogs is not allowed to login.
  • Have to set HOME because com.HomeDir() returns the root user’s home directory, but allows overriding it that way
  • Have to set GOGS_CUSTOM as explained on issue #2924 because packager.io sets the custom directory to /etc/gogs in its /etc/gogs/conf.d/other file.

At this point, the backup looks good for the DB and conf, attachments and avatars (and skips sessions).

When doing it in crontab, I have to wrap it in a script, and add USER=gogs so that the script looks like:

#!/bin/bash

export APP_DIR=/opt/gogs
export USER=gogs
export HOME=/var/www/gogs
export GOGS_CUSTOM=/etc/gogs
export TARGET=$HOME/backups

cd $APP_DIR
/opt/gogs/gogs backup -v --exclude-repos --target $TARGET
ls -l $TARGET
  • The APP_DIR vs TARGET difference was needed because when gogs backup is not run from the gogs directory, the data directory is skipped

  • The USER was needed, because without it, when running from the gogs user crontab, gogs backup fails with an error like:

    2018/04/25 12:35:01 [FATAL] […g/setting/setting.go:571 NewContext()] Expect user ‘gogs’ but current user is:

(yes current user is empty)


#30

is can migrate another OS?

ex) Backup from Windows Migrate to Linux(Cent OS)


#31

Yes, you should be able to.


#32

I tried to migrate mysql -> sqlite3.

I ran the backup command, .zip was created.

I then edited the custom conf app.ini, added PATH and changed db type to sqlite3.

The restore command, unfortunately, ended with segmentation fault in xorm (the sqlite db file was not created at the place where it should have been, according to PATH).

Should it work this way? Or do I need reinstall gogs from scratch with sqlite option and only then restore?


#33

What is the actual error message? And your Gogs version? (normally people only migrate away from SQLite3…)


#34
┌(storage1.foobar.io)─(✓)─(05:19 PM Mon Sep 10)
└─(/opt/gogs)─(28 files, 27Mb)─> uname -a
Linux storage1.foobar.io 3.13.0-86-generic #131-Ubuntu SMP Thu May 12 23:33:13 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

┌(storage1.foobar.io)─(✓)─(05:19 PM Mon Sep 10)
└─(/opt/gogs)─(28 files, 27Mb)─> id
uid=0(root) gid=0(root) groups=0(root)

┌(storage1.foobar.io)─(✓)─(05:19 PM Mon Sep 10)
└─(/opt/gogs)─(28 files, 27Mb)─> ./gogs -version
Gogs version 0.11.48.0426

┌(storage1.foobar.io)─(✓)─(05:19 PM Mon Sep 10)
└─(/opt/gogs)─(28 files, 27Mb)─> cat /etc/gogs/conf.d/other
export USER="gogs"
export GOGS_CUSTOM="/etc/gogs"

┌(storage1.foobar.io)─(✓)─(05:19 PM Mon Sep 10)
└─(/opt/gogs)─(28 files, 27Mb)─> source /etc/gogs/conf.d/other

┌(storage1.foobar.io)─(✓)─(05:20 PM Mon Sep 10)
└─(/opt/gogs)─(28 files, 27Mb)─> cat /etc/gogs/conf/app.ini
APP_NAME = FOOBAR Git Service
RUN_USER = gogs
RUN_MODE = prod

[repository]
ROOT                      = /home/gogs/gogs-repositories
SCRIPT_TYPE               = bash
FORCE_PRIVATE             = true

[server]
PROTOCOL               = http
DOMAIN                 = git.foobar.io
ROOT_URL               = https://git.foobar.io/
HTTP_ADDR              =
HTTP_PORT              = 3000

[database]
; Either "mysql", "postgres" or "sqlite3", it's your choice
DB_TYPE  = mysql
HOST     = 127.0.0.1:3306
NAME     = gogs
USER     = root
PASSWD   = this-is-not-disclosed

[admin]

[security]
INSTALL_LOCK                      = true
SECRET_KEY                        = this-is-not-disclosed

[service]
REGISTER_EMAIL_CONFIRM                 = true
DISABLE_REGISTRATION                   = true
REQUIRE_SIGNIN_VIEW                    = false
ENABLE_NOTIFY_MAIL                     = true
ENABLE_CAPTCHA                         = true

[mailer]
ENABLED         = true
; Mail server
; Gmail: smtp.gmail.com:587
; QQ: smtp.qq.com:25
; Note, if the port ends with "465", SMTPS will be used. Using STARTTLS on port 587 is recommended per RFC 6409. If the server supports STARTTLS it will always be used.
HOST            = localhost:25
; Mail from address, RFC 5322. This can be just an email address, or the `"Name" <email@example.com>` format
FROM            = FOOBAR Git Service <no-reply@foobar.io>
USER            = no-reply@foobar.io
SKIP_VERIFY     = true

[session]
PROVIDER          = file

[log]
ROOT_PATH  = /opt/gogs/log
MODE       = file
BUFFER_LEN = 10000
LEVEL      = Info

┌(storage1.foobar.io)─(✓)─(05:20 PM Mon Sep 10)
└─(/opt/gogs)─(28 files, 27Mb)─> ./gogs backup --config=/etc/gogs/conf/app.ini --database-only
2018/09/10 17:20:30 [ INFO] Backup root directory: /tmp/gogs-backup-213955758
2018/09/10 17:20:30 [ INFO] Packing backup files to: gogs-backup-20180910172030.zip
2018/09/10 17:20:31 [ INFO] Backup succeed! Archive is located at: gogs-backup-20180910172030.zip

┌(storage1.foobar.io)─(✓)─(05:20 PM Mon Sep 10)
└─(/opt/gogs)─(29 files, 28Mb)─> nano /etc/gogs/conf/app.ini

┌(storage1.foobar.io)─(✓)─(05:21 PM Mon Sep 10)
└─(/opt/gogs)─(29 files, 28Mb)─> cat /etc/gogs/conf/app.ini
APP_NAME = FOOBAR Git Service
RUN_USER = gogs
RUN_MODE = prod

[repository]
ROOT                      = /home/gogs/gogs-repositories
SCRIPT_TYPE               = bash
FORCE_PRIVATE             = true

[server]
PROTOCOL               = http
DOMAIN                 = git.foobar.io
ROOT_URL               = https://git.foobar.io/
HTTP_ADDR              =
HTTP_PORT              = 3000

[database]
; Either "mysql", "postgres" or "sqlite3", it's your choice
DB_TYPE  = sqlite3
HOST     = 127.0.0.1:3306
NAME     = gogs
USER     = root
PASSWD   = this-is-not-disclosed
PATH     = /home/gogs/gogs.db

[admin]

[security]
INSTALL_LOCK                      = true
SECRET_KEY                        = this-is-not-disclosed

[service]
REGISTER_EMAIL_CONFIRM                 = true
DISABLE_REGISTRATION                   = true
REQUIRE_SIGNIN_VIEW                    = false
ENABLE_NOTIFY_MAIL                     = true
ENABLE_CAPTCHA                         = true

[mailer]
ENABLED         = true
; Mail server
; Gmail: smtp.gmail.com:587
; QQ: smtp.qq.com:25
; Note, if the port ends with "465", SMTPS will be used. Using STARTTLS on port 587 is recommended per RFC 6409. If the server supports STARTTLS it will always be used.
HOST            = localhost:25
; Mail from address, RFC 5322. This can be just an email address, or the `"Name" <email@example.com>` format
FROM            = FOOBAR Git Service <no-reply@foobar.io>
USER            = no-reply@foobar.io
SKIP_VERIFY     = true

[session]
PROVIDER          = file

[log]
ROOT_PATH  = /opt/gogs/log
MODE       = file
BUFFER_LEN = 10000
LEVEL      = Info

┌(storage1.foobar.io)─(✓)─(05:21 PM Mon Sep 10)
└─(/opt/gogs)─(29 files, 28Mb)─> ls -la /home/gogs/
total 36
drwxr-xr-x  7 gogs gogs 4096 Sep 10 17:18 .
drwxr-xr-x  5 root root 4096 Oct  4  2016 ..
-rw-------  1 gogs gogs  258 Jun 13  2017 .bash_history
drwx------  3 gogs gogs 4096 Aug 30  2016 .cache
drwx------  3 gogs gogs 4096 Aug 30  2016 .config
-rw-r--r--  1 gogs gogs   71 Sep 10 17:18 .gitconfig
drwxr-xr-x 21 gogs gogs 4096 Jul  9 06:25 gogs-repositories
drwx------  3 gogs gogs 4096 Aug 30  2016 .local
-rw-rw-r--  1 gogs gogs    0 Aug 30  2016 .selected_editor
drwx------  2 gogs gogs 4096 Jul 12 10:28 .ssh

┌(storage1.foobar.io)─(✓)─(05:21 PM Mon Sep 10)
└─(/opt/gogs)─(29 files, 28Mb)─> ./gogs restore --config=/etc/gogs/conf/app.ini --database-only --from=gogs-backup-20180910172030.zip
2018/09/10 17:22:12 [ INFO] Restore backup from: gogs-backup-20180910172030.zip
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x80c29d]

goroutine 1 [running]:
github.com/gogits/gogs/vendor/github.com/go-xorm/xorm.(*Session).Begin(0xc420166280, 0xf106b8, 0xc420166280)
        /tmp/tmp.hyZgqVfLyu/.go/src/github.com/gogits/gogs/vendor/github.com/go-xorm/xorm/session_tx.go:10 +0x10d
github.com/gogits/gogs/vendor/github.com/go-xorm/xorm.(*Engine).DropTables(0x0, 0xc42008f3e8, 0x1, 0x1, 0x0, 0x0)
        /tmp/tmp.hyZgqVfLyu/.go/src/github.com/gogits/gogs/vendor/github.com/go-xorm/xorm/engine.go:1349 +0x84
github.com/gogits/gogs/models.ImportDatabase(0xc4205c0660, 0x13, 0x0, 0xc4205c0600, 0x13)
        /tmp/tmp.hyZgqVfLyu/.go/src/github.com/gogits/gogs/models/models.go:316 +0x372
github.com/gogits/gogs/cmd.runRestore(0xc42043c140, 0x10100, 0xc42043c140)
        /tmp/tmp.hyZgqVfLyu/.go/src/github.com/gogits/gogs/cmd/restore.go:86 +0x7cf
github.com/gogits/gogs/vendor/github.com/urfave/cli.HandleAction(0xdab080, 0xf0ef00, 0xc42043c140, 0xc420426400, 0x0)
        /tmp/tmp.hyZgqVfLyu/.go/src/github.com/gogits/gogs/vendor/github.com/urfave/cli/app.go:483 +0xb9
github.com/gogits/gogs/vendor/github.com/urfave/cli.Command.Run(0xecc5d4, 0x7, 0x0, 0x0, 0x0, 0x0, 0x0, 0xef73b1, 0x26, 0x0, ...)
        /tmp/tmp.hyZgqVfLyu/.go/src/github.com/gogits/gogs/vendor/github.com/urfave/cli/command.go:193 +0xb72
github.com/gogits/gogs/vendor/github.com/urfave/cli.(*App).Run(0xc4200eed00, 0xc4200100a0, 0x5, 0x5, 0x0, 0x0)
        /tmp/tmp.hyZgqVfLyu/.go/src/github.com/gogits/gogs/vendor/github.com/urfave/cli/app.go:250 +0x7d0
main.main()
        /tmp/tmp.hyZgqVfLyu/.go/src/github.com/gogits/gogs/gogs.go:41 +0x3ea

┌(storage1.foobar.io)─(✗)─(05:22 PM Mon Sep 10)
└─(/opt/gogs)─(29 files, 28Mb)─> ls -la /home/gogs/
total 36
drwxr-xr-x  7 gogs gogs 4096 Sep 10 17:18 .
drwxr-xr-x  5 root root 4096 Oct  4  2016 ..
-rw-------  1 gogs gogs  258 Jun 13  2017 .bash_history
drwx------  3 gogs gogs 4096 Aug 30  2016 .cache
drwx------  3 gogs gogs 4096 Aug 30  2016 .config
-rw-r--r--  1 gogs gogs   71 Sep 10 17:18 .gitconfig
drwxr-xr-x 21 gogs gogs 4096 Jul  9 06:25 gogs-repositories
drwx------  3 gogs gogs 4096 Aug 30  2016 .local
-rw-rw-r--  1 gogs gogs    0 Aug 30  2016 .selected_editor
drwx------  2 gogs gogs 4096 Jul 12 10:28 .ssh

┌(storage1.foobar.io)─(✓)─(05:22 PM Mon Sep 10)
└─(/opt/gogs)─(29 files, 28Mb)─>

#35

Thanks, looks like indeed a problem within XORM. You may want to try restore again to SQLite3 after next binary release. (or you can upgrade by source from master right now)