blog/2021-09-13-problems-and-solutions-with-gitea-1-15.md at fdcb902bd2766df6e4f6bbab99837abb8ed20f73 - blog

23 KiB

Raw Blame History

title	tags	updated	description
Problems and solutions with Gitea 1.15	[tutorial gitea mariadb postgresql]	2021-10-29 13:49:10	This blog is now self-hosted

My Gitea instance at software.franco.net.eu.org has been on SQLite since its inception in October 2018, up until some weeks ago. I never had any problems concerning database updates. Then I decided to migrate to something more scalable: MariaDB.

Migration of Gitea < 1.15.x from SQLite to MariaDB
Update to 1.15.2
- Revert
- PostgreSQL
  - Steps
Extras
Updates
- GitHub issues

Migration of Gitea < 1.15.x from SQLite to MariaDB

Migration was made possible using the provided dump tool. Here is what I did (these instructions have been written after some trials and errors):

create a MySQL user

mysql -u root

CREATE USER 'gitea' IDENTIFIED BY '${DB_PASSWORD}';

create a MySQL database

CREATE DATABASE giteadb CHARACTER SET 'utf8mb4' COLLATE 'utf8mb4_unicode_ci';
GRANT ALL PRIVILEGES ON giteadb.* TO 'gitea';
FLUSH PRIVILEGES;
exit

Go into the site administration and delete all authentication sources. This issue also happened to me.
Stop Gitea
```
systemctl stop gitea
```

disable all cron jobs at gitea boot. Gitea didn't start while cron jobs were enabled at boot. Edit /etc/gitea/app.ini:

[cron]
; Enable running cron tasks periodically.
ENABLED      = true
; Run cron tasks when Gitea starts.
RUN_AT_START = false

start and stop Gitea to have a clean restart without cron jobs running:
```
systemctl start gitea
sleep 60
systemctl stop gitea
```

create a database dump translated from sqlite to mysql. This command creates a zip file called gitea-dump-${id}.zip:

sudo -i -u gitea
HOME=/var/lib/gitea GITEA_WORK_DIR=/var/lib/gitea /usr/bin/gitea dump --skip-repository --skip-log --skip-custom-dir --skip-lfs-data --skip-attachment-data --database mysql -c /etc/gitea/app.ini

change the database type in /etc/gitea/app.ini. Comment out the old section and add a new mysql section like this:

[database]
; Either "mysql", "postgres", "mssql" or "sqlite3", it's your choice
DB_TYPE             = mysql
HOST                = 127.0.0.1:3306
NAME                = giteadb
USER                = gitea
; Use PASSWD = `your password` for quoting if you use special characters in the password.
PASSWD              = `${DB_PASSWORD}`

restore the database:

unzip gitea-dump-${id}.zip
mysql --default-character-set=utf8mb4 -ugitea -p${DB_PASSWORD} giteadb <gitea-db.sql

systemctl start gitea
re-add the authentication sources

This worked although some warnings remained when I lauched Gitea's doctor command.

Update to 1.15.2

After updating the packages, Gitea did not start. There were a bunch of errors (for example this)! I was able to recover the database by hand by having a look at the errors from the log.

After some time, however, I noticed that the mirror feature did not work and the repositories on the web UI did not update after pushing from git.

Jump to the solution

Revert

I tried reverting back to a previous database version but once you migrate you cannot go back easily.

PostgreSQL

I tried with a fresh PostgreSQL 13 database and Gitea 1.15.2. The repostitory and mirror problem was still there so at least I knew it wasn't a problem with the database.

What I did in the end was to re-install Gitea 1.14.7 and to switch to another new PostgreSQL database instead. The problem now was that I needed to recover most "issues", mirrors, users, etc... from the original MariaDB database. Once that was done after hard work I made a backup and proceeded to the migration of this new database to Gitea 1.15.0.

Steps

These steps work on Debian GNU/Linux 10 which is now oldstable.

1. setup the database

install PostgreSQL 13 from the official website
Follow the database preparations instructions

use the UNIX socket instead of TCP. This configuration might improve performance a little bit. Set this in /etc/gitea/app.ini

[database]
DB_TYPE             = postgres
HOST                = /var/run/postgresql
LOG_SQL             = false

2. recover the repositories

Create the users and push or "adopt" the repositories through the admin/repos URL. You will see the Unadopted Repository button there.

3. Adminer

Adminer is similar to phpMyAdmin. You can install it with:

apt-get install adminer/buster-backports

The backports version seem to be fully compatible with PostgreSQL 13.

Apache configuration

if you use PHP-FPM you can use this snippet in a VirtualHost directive:

Alias /dbadmin /usr/share/adminer/adminer/
<Directory "/usr/share/adminer/adminer/">
    Require ip 127.0.0.1
    Require ip 192.168.0.
    Options FollowSymlinks
</Directory>
Include conf-enabled/php7.3-fpm.conf

add this to the php.ini file of Apache, in /etc/php/7.3/apache2/php.ini:

[HOST=my.host]
open_basedir = /tmp/:/usr/share/adminer:/usr/share/php:/var/log/adminer

finally:
```
systemctl restart apache2
```
connect to http://my.host/dbadmin

4. set the packages on hold

Set these packages on hold so when you update the system they don't get updated by accident.

apt-mark hold gitea linux-image-amd64 postgresql-13 postgresql-client-13 postgresql-client-common postgresql-common

This is what you should get when you run apt-mark showhold:

gitea
linux-image-amd64
postgresql-13
postgresql-client-13
postgresql-client-common
postgresql-common

5. restore issues and comments

open the original database with adminer
get the repository id. Use the repository table to get it. Let's say it's 38
go to the issue table
click on the search filter on the top. Set repo_id = 38. Then click Select
on the web-ui with the new database create new issues within the repository. In our example we have 9 issues so we need to create 9 dummy issues. Comment content and the user creating them is irrelevant
now take note of the issues ids. In our case they are [1-9]
open the comment table and do a new query, filtering by issue id. Let's start with 1 In this case we only have 1 comment so we must create a dummy comment in the issue on web UI
for issue 2 we have 2 comments so we need to create 2 dummy comments
do this for issues [1-9]
once we have all the dummy data we can open the new database on adminer. What we have to do now is just to manually copy-paste all the data from one database to the other. Check the UI once in a while if everything is working

{% include image.html file="adminer_gitea_issue_table.png" alt="Gitea issue table on Adminer" caption="Gitea issue table on Adminer" %}

{% include image.html file="adminer_gitea_comment_table.png" alt="Gitea comment table on Adminer" caption="Gitea comment table on Adminer" %}

{% include image.html file="adminer_gitea_comment_table_2.png" alt="Gitea comment table for issue 2 on Adminer" caption="Gitea comment table for issue 2 on Adminer" %}

6. restore mirrors from bare repositories

once you have the repositories go into the repositories table
select the to-be mirrors repositories and set the is_mirror variable to true
go into the mirror table and create a new element. Use the correct repo_id.

{% include image.html file="adminer_gitea_repository_table_mirror.png" alt="Mirror variable in a repository" caption="Mirror variable in a repository" %}

{% include image.html file="adminer_gitea_mirror_table.png" alt="An example row of the mirror table" caption="An example row of the mirror table" %}

7. restore users

To restore users simply create new users using Gitea's admin interface and then copy-paste the data in the rows.

8. Database backups

I now use 15 minute separated backups using borgmatic. See also https://docs.franco.net.eu.org/automated-tasks/scripts.html#borgmatic-hooks-py

This is just an example for reference:

#
# borgmatic.iron_postgresql_giteadb.yaml
#
# Copyright (C) 2014-2020 Dan Helfman <https://torsion.org/borgmatic/docs/reference/config.yaml>
#               2021 Franco Masotti
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program.  If not, see <https://www.gnu.org/licenses/>.

location:
    source_directories: []
    repositories:
        - user@remotepc:backups/postgresql_giteadb.borg
storage:
    checkpoint_interval: 900
    lock_wait: 120
retention:
    keep_within: 1w
    keep_monthly: 1
consistency:
    checks:
        - archives
output:
    color: false
hooks:
    after_everything:
        - /home/jobs/scripts/by-user/root/borgmatic_hooks.py /home/jobs/scripts/by-user/root/borgmatic_hooks.iron_postgresql_giteadb.yaml 'finish' "{configuration_filename}" "{repository}" "{output}" "{error}"
    on_error:
        - /home/jobs/scripts/by-user/root/borgmatic_hooks.py /home/jobs/scripts/by-user/root/borgmatic_hooks.iron_postgresql_giteadb.yaml 'error' "{configuration_filename}" "{repository}" "{output}" "{error}"

    postgresql_databases:
        - name: giteadb
          # Use unix sockets instead of TCP.
          # See
          # https://torsion.org/borgmatic/docs/reference/configuration/
          #
          # hostname: 127.0.0.1
          # port: 5432
          username: gitea
          password: ${DB_PASS}
          format: tar
          options: "--verbose"

Extras

Torification

This is useful if you need to clone repositories, or do mirroring via TOR. I did this before switching to PostgreSQL:

Install TOR and configure it
```
apt-get install tor
```

run systemctl edit gitea.service and add this content:

[Unit]
Description=Gitea (Git with a cup of tea)
After=syslog.target
After=network.target
After=mysqld.service
After=postgresql.service
After=memcached.service
After=redis.service

# Comment or change these.
Requires=network.target
Requires=postgresql.service
Requires=redis.service

# Comment this if you don't need it.
# See
# https://docs.franco.net.eu.org/automated-tasks/scripts.html#notify-unit-status-py
OnFailure=notify-unit-status@%n.service

[Service]
ExecStart=
ExecStart=/usr/bin/torsocks --isolate /usr/bin/gitea web -c /etc/gitea/app.ini

User=gitea
Group=gitea
Type=simple
WorkingDirectory=~
RuntimeDirectory=gitea
LogsDirectory=gitea
StateDirectory=gitea
Environment=USER=gitea HOME=/var/lib/gitea GITEA_WORK_DIR=/var/lib/gitea
Restart=always
RestartSec=2s
CapabilityBoundingSet=
NoNewPrivileges=false
#SecureBits=noroot-locked
ProtectSystem=strict
ProtectHome=true
ReadWritePaths=/etc/gitea/app.ini
PrivateTmp=true
PrivateDevices=false
PrivateUsers=false
ProtectHostname=false
ProtectClock=false
ProtectKernelTunables=false
ProtectKernelModules=false
ProtectKernelLogs=false
ProtectControlGroups=true
LockPersonality=false
MemoryDenyWriteExecute=false
RestrictRealtime=false
RestrictSUIDSGID=false
SystemCallArchitectures=
SystemCallFilter=
SystemCallErrorNumber=EPERM
ReadWriteDirectories=/var/spool/postfix/maildrop

restart gitea
```
systemctl restart gitea.servvice
```
Test by cloning a repository from The Tor Project. Mirror this repository for example:

http://eweiibe6tdjsdprb4px6rqrzzcsi22m4koia44kc5pcjr7nec2rlxyad.onion/tpo/anti-censorship/bridgedb.git

HTTP2

enable HTTP2 on Apache
```
a2enmod http2
```
add this to your Apache configuration file. This setting can be global or per-virtualhost:
```
Protocols h2 h2c http/1.1
```
if you have problems with HTTP2 add this to /etc/gitea/app.ini:
```
[ui.notification]
EVENT_SOURCE_UPDATE_TIME=-1
```
See also https://github.com/go-gitea/gitea/issues/11978

Caching

Solution

The repository/mirror problem was still there. After lots of trials and errors I found a solution.

Redis

install Redis
```
apt-get install redis-server
```

setup Redis to listen on a UNIX socket exclusively. Edit /etc/redis/redis.conf:

unixsocket /var/run/redis/redis-server.sock
unixsocketperm 770
bind 127.0.0.1 ::1

# This disables listening on TCP.
port 0

# Avoid saving on disk
save ""

# Remove authentication.
requirepass ""

add the gitea user to the redis group so Gitea can have access to the socket.
```
usermod -aG redis gitea
```

restart Redis and Gitea:

systemctl restart redis-server.service gitea.service

check the logs at /var/log/redis/redis-server.log. If you have a warning about transparent hugepages. Add this service to Systemd.

# See
# https://unix.stackexchange.com/a/363887
# https://stackoverflow.com/a/64945381
# CC BY-SA 3.0
# (c) 2017 nelaaro

[Unit]
Description=madvise Transparent Huge Pages
Before=redis-server.service
Before=apache2.service
Before=gitea.service
Before=postgresql.service

[Service]
Type=oneshot
ExecStart=/bin/sh -c "/usr/bin/echo "madvise" | tee /sys/kernel/mm/transparent_hugepage/enabled"
ExecStart=/bin/sh -c "/usr/bin/echo "madvise" | tee /sys/kernel/mm/transparent_hugepage/defrag"

[Install]
WantedBy=multi-user.target

have a look at the Redis ArchWiki page.

Gitea configuration

To be able to use Redis in Gitea edit these sections in /etc/gitea/app.ini:

[cache]
ADAPTER  = redis
HOST     =  network=unix,addr=/var/run/redis/redis-server.sock,db=0,pool_size=100,idle_timeout=180s
ITEM_TTL = 24h

[session]
PROVIDER          = redis
PROVIDER_CONFIG  =  network=unix,addr=/var/run/redis/redis-server.sock,db=1,pool_size=100,idle_timeout=180s

[queue]
TYPE = redis
CONN_STR = network=unix,addr=/var/run/redis/redis-server.sock,db=2,pool_size=100,idle_timeout=180s

[queue.task]
QUEUE_TYPE = redis
QUEUE_CONN_STR = network=unix,addr=/var/run/redis/redis-server.sock,db=2,pool_size=100,idle_timeout=180s

[task]
QUEUE_TYPE = redis
QUEUE_CONN_STR = network=unix,addr=/var/run/redis/redis-server.sock,db=2,pool_size=100,idle_timeout=180s

Note: using Redis for queues and tasks seems to have solved the original problem.

Updating gitea next time

temporarly disable the service

systemctl stop gitea.service
systemctl mask gitea.service

copy the original database into a new one

sudo -i -u postgres
psql

CREATE DATABASE backup_giteadb_${major}_${minor}_${patch} WITH TEMPLATE giteadb;

where major, minor and patch correspond to the current Gitea version.

run the doctor on the original database

exit
exit
sudo -i -u gitea
HOME=/var/lib/gitea GITEA_WORK_DIR=/var/lib/gitea /usr/bin/gitea -c /etc/gitea/app.ini doctor --all
HOME=/var/lib/gitea GITEA_WORK_DIR=/var/lib/gitea /usr/bin/gitea -c /etc/gitea/app.ini doctor --all --fix

update Gitea

exit
sudo -i
apt-mark unhold gitea
apt-get update
apt-get dist-upgrade

migrate the database

exit
sudo -i -u gitea
HOME=/var/lib/gitea GITEA_WORK_DIR=/var/lib/gitea /usr/bin/gitea -c /etc/gitea/app.ini migrate

run the doctor again

HOME=/var/lib/gitea GITEA_WORK_DIR=/var/lib/gitea /usr/bin/gitea -c /etc/gitea/app.ini doctor --all
HOME=/var/lib/gitea GITEA_WORK_DIR=/var/lib/gitea /usr/bin/gitea -c /etc/gitea/app.ini doctor --all --fix

re-enable the service

exit
sudo -i
systemctl unmask gitea.service
systemctl start gitea.service
apt-mark hold gitea

Apache2 reverse proxy for Gitea

modify Gitea configuration (/etc/gitea/app.ini):

[server]
PROTOCOL                   = unix
DOMAIN                     = localhost
HTTP_ADDR                   = /var/run/gitea/gitea.sock
UNIX_SOCKET_PERMISSION      = 770

; Do not set this variable if PROTOCOL is set to 'unix'.
# LOCAL_ROOT_URL             = %(PROTOCOL)s://%(HTTP_ADDR)s:%(HTTP_PORT)s/

Add the www-data user to gitea's group so it can access the socket.
```
usermod -aG gitea www-data
```

Restart gitea and apache

systemctl restart gitea.service apache2.service

clearnet configuration in /etc/apache2/apache2.conf or a virtual host file:

variables (shell style):

ONION_ADDRESS: the TOR address where Gitea is exposed
TCP_PORT: the TCP/IP port where gitea should be listenting. Even if we are using a UNIX socket this is applicable. You can use Gitea's default port.
SERVER_NAME: the FQDN

#########
# Gitea #
#########
<IfModule mod_ssl.c>
<VirtualHost *:443>
    UseCanonicalName on

    ProxyPreserveHost On

    Keepalive On
    RewriteEngine on
    AllowEncodedSlashes NoDecode

    ServerName ${SERVER_NAME}

    ProxyBadHeader Ignore

    # TOR.
    # Remove this if you don't want Gitea being avaliable on TOR.
    ServerAlias ${ONION_ADDRESS}
    Header set Onion-Location "http://${ONION_ADDRESS}%{REQUEST_URI}s"

    SSLCompression      off

    ProxyPass  / unix:/var/run/gitea/gitea.sock|http://127.0.0.1:${TCP_PORT}/ nocanon
    ProxyPassReverse  / unix:/var/run/gitea/gitea.sock|http://127.0.0.1:${TCP_PORT}/

    Include /etc/letsencrypt/options-ssl-apache.conf
    SSLCertificateFile /etc/letsencrypt/live/${SERVER_NAME}/fullchain.pem
    SSLCertificateKeyFile /etc/letsencrypt/live/${SERVER_NAME}/privkey.pem
</VirtualHost>

if you use TOR add this configuration in /etc/apache2/apache2.conf or a virtual host file: