Added new post.

dev
Franco Masotti 2021-09-13 16:01:01 +02:00
parent 1b2bf116b6
commit e6fe1ddcf3
Signed by: frnmst
GPG Key ID: 24116ED85666780A
6 changed files with 720 additions and 0 deletions

Binary file not shown.

After

Width:  |  Height:  |  Size: 52 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 66 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 119 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 32 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 14 KiB

View File

@ -0,0 +1,720 @@
---
title: Problems and solutions with Gitea 1.15
tags: [tutorial, gitea, mariadb, postgresql]
updated: 2021-09-13 14:00:00
description: This blog is now self-hosted
---
My Gitea instance at [software.franco.net.eu.org](https://software.franco.net.eu.org)
has been on SQLite since its inception in October 2018, up until some weeks ago.
I never had any problems concerning database updates. Then I decided to migrate
to something more scalable: [MariaDB](https://mariadb.org/).
<!--TOC-->
- [Migration of Gitea < 1.15.x from SQLite to MariaDB](#migration-of-gitea--115x-from-sqlite-to-mariadb)
- [Update to 1.15.2](#update-to-1152)
- [Revert](#revert)
- [PostgreSQL](#postgresql)
- [Steps](#steps)
- [1. setup the database](#1-setup-the-database)
- [2. recover the repositories](#2-recover-the-repositories)
- [3. Adminer](#3-adminer)
- [Apache configuration](#apache-configuration)
- [4. set the packages on hold](#4-set-the-packages-on-hold)
- [5. restore issues and comments](#5-restore-issues-and-comments)
- [6. restore mirrors from bare repositories](#6-restore-mirrors-from-bare-repositories)
- [7. restore users](#7-restore-users)
- [8. Database backups](#8-database-backups)
- [Extras](#extras)
- [Torification](#torification)
- [HTTP2](#http2)
- [Caching](#caching)
- [Solution](#solution)
- [Redis](#redis)
- [Gitea configuration](#gitea-configuration)
- [Updating gitea next time](#updating-gitea-next-time)
- [Apache2 reverse proxy for Gitea](#apache2-reverse-proxy-for-gitea)
- [GitHub issue](#github-issue)
<!--TOC-->
## Migration of Gitea < 1.15.x from SQLite to MariaDB
Migration was made possible using the provided dump tool.
Here is what I did (these instructions have been written
after some trials and errors):
1. create a MySQL user
```shell
mysql -u root
```
```sql
CREATE USER 'gitea' IDENTIFIED BY '${DB_PASSWORD}';
```
2. create a MySQL database
```sql
CREATE DATABASE giteadb CHARACTER SET 'utf8mb4' COLLATE 'utf8mb4_unicode_ci';
GRANT ALL PRIVILEGES ON giteadb.* TO 'gitea';
FLUSH PRIVILEGES;
exit
```
3. Go into the site administration and delete all authentication sources.
[This issue](https://github.com/go-gitea/gitea/issues/16831)
also happened to me.
4. Stop Gitea
```shell
systemctl stop gitea
```
5. disable all cron jobs at gitea boot. Gitea didn't start while cron jobs
were enabled at boot. Edit `/etc/gitea/app.ini`:
```ini
[cron]
; Enable running cron tasks periodically.
ENABLED = true
; Run cron tasks when Gitea starts.
RUN_AT_START = false
```
6. start and stop Gitea to have a clean restart without cron jobs running:
```shell
systemctl start gitea
sleep 60
systemctl stop gitea
```
7. create a database dump translated from sqlite to mysql. This command creates
a zip file called `gitea-dump-${id}.zip`:
```shell
sudo -i -u gitea
HOME=/var/lib/gitea GITEA_WORK_DIR=/var/lib/gitea /usr/bin/gitea dump --skip-repository --skip-log --skip-custom-dir --skip-lfs-data --skip-attachment-data --database mysql -c /etc/gitea/app.ini
```
8. change the database type in `/etc/gitea/app.ini`. Comment out the old section and
add a new mysql section like this:
```ini
[database]
; Either "mysql", "postgres", "mssql" or "sqlite3", it's your choice
DB_TYPE = mysql
HOST = 127.0.0.1:3306
NAME = giteadb
USER = gitea
; Use PASSWD = `your password` for quoting if you use special characters in the password.
PASSWD = `${DB_PASSWORD}`
```
9. restore the database:
```shell
unzip gitea-dump-${id}.zip
mysql --default-character-set=utf8mb4 -ugitea -p${DB_PASSWORD} giteadb <gitea-db.sql
```
10. `systemctl start gitea`
11. re-add the authentication sources
This worked although some warnings remained when I lauched Gitea's doctor command.
## Update to 1.15.2
After updating the packages, Gitea did not start.
There were a bunch of errors (for example
[this](https://github.com/go-gitea/gitea/issues/16961))! I was able to recover the database
by hand by having a look at the errors from the log.
After some time, however, I noticed that the mirror feature did not work
and the repositories on the web UI did not update after pushing from git.
[Jump to the solution](#solution)
### Revert
I tried reverting back to a previous database version but once you migrate
you cannot go back easily.
### PostgreSQL
I tried with a fresh PostgreSQL 13 database and Gitea 1.15.2.
The repostitory and mirror problem was still there so at least I knew it
wasn't a problem with the database.
What I did in the end was to re-install Gitea 1.14.7 and to switch to
another new PostgreSQL database instead. The problem now was that
I needed to recover most "issues", mirrors, users, etc... from the original
MariaDB database.
Once that was done after hard work I made a backup and proceeded
to the migration of this new database to Gitea 1.15.0.
#### Steps
These steps work on Debian GNU/Linux 10 which is now *oldstable*.
#### 1. setup the database
1. install PostgreSQL 13 from the [official website](https://www.postgresql.org/download/linux/debian/)
2. Follow the [database preparations instructions](https://docs.gitea.io/en-us/database-prep/#postgresql)
3. use the UNIX socket instead of TCP. This configuration might improve performance a little bit. Set this in `/etc/gitea/app.ini`
```ini
[database]
DB_TYPE = postgres
HOST = /var/run/postgresql
LOG_SQL = false
```
#### 2. recover the repositories
Create the users and push or "adopt" the repositories through the `admin/repos` URL. You will see
the `Unadopted Repository` button there.
#### 3. Adminer
[Adminer](https://www.adminer.org/) is similar to phpMyAdmin. You can install it with:
```shell
apt-get install adminer/buster-backports
```
The backports version seem to be fully compatible with PostgreSQL 13.
##### Apache configuration
1. if you use [PHP-FPM](https://www.php.net/manual/en/install.fpm.php)
you can use this snippet in a `VirtualHost` directive:
```apache
Alias /dbadmin /usr/share/adminer/adminer/
<Directory "/usr/share/adminer/adminer/">
Require ip 127.0.0.1
Require ip 192.168.0.
Options FollowSymlinks
</Directory>
Include conf-enabled/php7.3-fpm.conf
```
2. add this to the php.ini file of Apache, in `/etc/php/7.3/apache2/php.ini`:
```ini
[HOST=my.host]
open_basedir = /tmp/:/usr/share/adminer:/usr/share/php:/var/log/adminer
```
3. finally:
```shell
systemctl restart apache2
```
4. connect to `http://my.host/dbadmin`
#### 4. set the packages on hold
Set these packages on hold so when you update the system they don't get
updated by accident.
```shell
apt-mark hold gitea linux-image-amd64 postgresql-13 postgresql-client-13 postgresql-client-common postgresql-common
```
This is what you should get when you run `apt-mark showhold`:
```shell
gitea
linux-image-amd64
postgresql-13
postgresql-client-13
postgresql-client-common
postgresql-common
```
#### 5. restore issues and comments
1. open the original database with adminer
1. get the repository id. Use the `repository` table to get it. Let's say it's `38`
2. go to the `issue` table
3. click on the search filter on the top. Set `repo_id` `=` `38`. Then click `Select`
4. on the web-ui with the new database create new issues within the repository. In our example we have 9
issues so we need to create 9 dummy issues. Comment content and the user creating them is irrelevant
5. now take note of the issues ids. In our case they are [`1`-`9`]
6. open the comment table and do a new query, filtering by issue id. Let's start with `1`
In this case we only have 1 comment so we must create a dummy comment in the issue on web UI
7. for issue `2` we have 2 comments so we need to create 2 dummy comments
8. do this for issues [`1`-`9`]
9. once we have all the dummy data we can open the new database on adminer.
What we have to do now is just to manually copy-paste all the data from
one database to the other. Check the UI once in a while if everything is working
{% include image.html file="adminer_gitea_issue_table.png" alt="Gitea issue table on Adminer" caption="Gitea issue table on Adminer" %}
{% include image.html file="adminer_gitea_comment_table.png" alt="Gitea comment table on Adminer" caption="Gitea comment table on Adminer" %}
{% include image.html file="adminer_gitea_comment_table_2.png" alt="Gitea comment table for issue 2 on Adminer" caption="Gitea comment table for issue 2 on Adminer" %}
#### 6. restore mirrors from bare repositories
1. once you have the repositories go into the `repositories` table
2. select the to-be mirrors repositories and set the `is_mirror` variable to `true`
2. go into the `mirror` table and create a new element. Use the correct `repo_id`.
{% include image.html file="adminer_gitea_repository_table_mirror.png" alt="Mirror variable in a repository" caption="Mirror variable in a repository" %}
{% include image.html file="adminer_gitea_mirror_table.png" alt="An example row of the mirror table" caption="An example row of the mirror table" %}
#### 7. restore users
To restore users simply create new users using Gitea's admin interface and then copy-paste the data
in the rows.
#### 8. Database backups
I now use 15 minute separated backups using borgmatic. See also
[https://docs.franco.net.eu.org/automated-tasks/scripts.html#borgmatic-hooks-py](https://docs.franco.net.eu.org/automated-tasks/scripts.html#borgmatic-hooks-py)
This is just an example for reference:
```yaml
#
# borgmatic.iron_postgresql_giteadb.yaml
#
# Copyright (C) 2014-2020 Dan Helfman <https://torsion.org/borgmatic/docs/reference/config.yaml>
# 2021 Franco Masotti
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <https://www.gnu.org/licenses/>.
location:
source_directories: []
repositories:
- user@remotepc:backups/postgresql_giteadb.borg
storage:
checkpoint_interval: 900
lock_wait: 120
retention:
keep_within: 1w
keep_monthly: 1
consistency:
checks:
- archives
output:
color: false
hooks:
after_everything:
- /home/jobs/scripts/by-user/root/borgmatic_hooks.py /home/jobs/scripts/by-user/root/borgmatic_hooks.iron_postgresql_giteadb.yaml 'finish' "{configuration_filename}" "{repository}" "{output}" "{error}"
on_error:
- /home/jobs/scripts/by-user/root/borgmatic_hooks.py /home/jobs/scripts/by-user/root/borgmatic_hooks.iron_postgresql_giteadb.yaml 'error' "{configuration_filename}" "{repository}" "{output}" "{error}"
postgresql_databases:
- name: giteadb
# Use unix sockets instead of TCP.
# See
# https://torsion.org/borgmatic/docs/reference/configuration/
#
# hostname: 127.0.0.1
# port: 5432
username: gitea
password: ${DB_PASS}
format: tar
options: "--verbose"
```
## Extras
### Torification
This is useful if you need to clone repositories, or do mirroring via TOR.
I did this before switching to PostgreSQL:
1. Install TOR and configure it
```shell
apt-get install tor
```
2. run `systemctl edit gitea.service` and add this content:
```systemd
[Unit]
Description=Gitea (Git with a cup of tea)
After=syslog.target
After=network.target
After=mysqld.service
After=postgresql.service
After=memcached.service
After=redis.service
# Comment or change these.
Requires=network.target
Requires=postgresql.service
Requires=redis.service
# Comment this if you don't need it.
# See
# https://docs.franco.net.eu.org/automated-tasks/scripts.html#notify-unit-status-py
OnFailure=notify-unit-status@%n.service
[Service]
ExecStart=
ExecStart=/usr/bin/torsocks --isolate /usr/bin/gitea web -c /etc/gitea/app.ini
User=gitea
Group=gitea
Type=simple
WorkingDirectory=~
RuntimeDirectory=gitea
LogsDirectory=gitea
StateDirectory=gitea
Environment=USER=gitea HOME=/var/lib/gitea GITEA_WORK_DIR=/var/lib/gitea
Restart=always
RestartSec=2s
CapabilityBoundingSet=
NoNewPrivileges=false
#SecureBits=noroot-locked
ProtectSystem=strict
ProtectHome=true
ReadWritePaths=/etc/gitea/app.ini
PrivateTmp=true
PrivateDevices=false
PrivateUsers=false
ProtectHostname=false
ProtectClock=false
ProtectKernelTunables=false
ProtectKernelModules=false
ProtectKernelLogs=false
ProtectControlGroups=true
LockPersonality=false
MemoryDenyWriteExecute=false
RestrictRealtime=false
RestrictSUIDSGID=false
SystemCallArchitectures=
SystemCallFilter=
SystemCallErrorNumber=EPERM
ReadWriteDirectories=/var/spool/postfix/maildrop
```
3. restart gitea
```shell
systemctl restart gitea.servvice
```
4. Test by cloning a repository from *The Tor Project*. Mirror this
repository for example:
[http://eweiibe6tdjsdprb4px6rqrzzcsi22m4koia44kc5pcjr7nec2rlxyad.onion/tpo/anti-censorship/bridgedb.git](http://eweiibe6tdjsdprb4px6rqrzzcsi22m4koia44kc5pcjr7nec2rlxyad.onion/tpo/anti-censorship/bridgedb.git)
### HTTP2
1. enable HTTP2 on Apache
```shell
a2enmod http2
```
2. add this to your Apache configuration file. This setting can be global or
per-virtualhost:
```apache
Protocols h2 h2c http/1.1
```
3. if you have problems with HTTP2 add this to `/etc/gitea/app.ini`:
```ini
[ui.notification]
EVENT_SOURCE_UPDATE_TIME=-1
```
See also [https://github.com/go-gitea/gitea/issues/11978](https://github.com/go-gitea/gitea/issues/11978)
### Caching
#### Solution
The repository/mirror problem was still there. After lots of trials and errors I found
a solution.
##### Redis
1. install Redis
```shell
apt-get install redis-server
```
2. setup Redis to listen on a UNIX socket exclusively. Edit `/etc/redis/redis.conf`:
```conf
unixsocket /var/run/redis/redis-server.sock
unixsocketperm 770
bind 127.0.0.1 ::1
# This disables listening on TCP.
port 0
# Avoid saving on disk
save ""
# Remove authentication.
requirepass ""
```
3. add the `gitea` user to the `redis` group so Gitea can have
access to the socket.
```
usermod -aG redis gitea
```
4. restart Redis and Gitea:
```shell
systemctl restart redis-server.service gitea.service
```
5. check the logs at `/var/log/redis/redis-server.log`. If you have a warning
about transparent hugepages. Add this service to Systemd.
```systemd
# See
# https://unix.stackexchange.com/a/363887
# https://stackoverflow.com/a/64945381
# CC BY-SA 3.0
# (c) 2017 nelaaro
[Unit]
Description=madvise Transparent Huge Pages
Before=redis-server.service
Before=apache2.service
Before=gitea.service
Before=postgresql.service
[Service]
Type=oneshot
ExecStart=/bin/sh -c "/usr/bin/echo "madvise" | tee /sys/kernel/mm/transparent_hugepage/enabled"
ExecStart=/bin/sh -c "/usr/bin/echo "madvise" | tee /sys/kernel/mm/transparent_hugepage/defrag"
[Install]
WantedBy=multi-user.target
```
6. have a look at the [Redis ArchWiki page](https://wiki.archlinux.org/title/Redis).
##### Gitea configuration
To be able to use Redis in Gitea edit these sections in `/etc/gitea/app.ini`:
```ini
[cache]
ADAPTER = redis
HOST = network=unix,addr=/var/run/redis/redis-server.sock,db=0,pool_size=100,idle_timeout=180s
ITEM_TTL = 24h
[session]
PROVIDER = redis
PROVIDER_CONFIG = network=unix,addr=/var/run/redis/redis-server.sock,db=1,pool_size=100,idle_timeout=180s
[queue]
TYPE = redis
CONN_STR = network=unix,addr=/var/run/redis/redis-server.sock,db=2,pool_size=100,idle_timeout=180s
[queue.task]
QUEUE_TYPE = redis
QUEUE_CONN_STR = network=unix,addr=/var/run/redis/redis-server.sock,db=2,pool_size=100,idle_timeout=180s
[task]
QUEUE_TYPE = redis
QUEUE_CONN_STR = network=unix,addr=/var/run/redis/redis-server.sock,db=2,pool_size=100,idle_timeout=180s
```
**Note: using Redis for queues and tasks seems to have solved the original problem.**
### Updating gitea next time
1. temporarly disable the service
```
systemctl stop gitea.service
systemctl mask gitea.service
```
2. copy the original database into a new one
```sql
CREATE DATABASE backup_giteadb_1.15.2 WITH TEMPLATE giteadb;
```
3. run the doctor on the original database
```shell
HOME=/var/lib/gitea GITEA_WORK_DIR=/var/lib/gitea /usr/bin/gitea -c /etc/gitea/app.ini doctor --all
HOME=/var/lib/gitea GITEA_WORK_DIR=/var/lib/gitea /usr/bin/gitea -c /etc/gitea/app.ini doctor --all --fix
```
4. update Gitea
```shell
apt-mark unhold gitea
apt-get update
apt-get dist-upgrade
```
5. migrate the database
```bash
HOME=/var/lib/gitea GITEA_WORK_DIR=/var/lib/gitea /usr/bin/gitea -c /etc/gitea/app.ini migrate
```
3. run the doctor again
```shell
HOME=/var/lib/gitea GITEA_WORK_DIR=/var/lib/gitea /usr/bin/gitea -c /etc/gitea/app.ini doctor --all
HOME=/var/lib/gitea GITEA_WORK_DIR=/var/lib/gitea /usr/bin/gitea -c /etc/gitea/app.ini doctor --all --fix
```
2. re-enable the service
```shell
systemctl unmask gitea.service
systemctl start gitea.service
```
### Apache2 reverse proxy for Gitea
1. clearnet configuration in `/etc/apache2/apache2.conf` or a virtual host file:
variables (shell style):
- `ONION_ADDRESS`: the TOR address where Gitea is exposed
- `TCP_PORT`: the TCP/IP port where gitea should be listenting.
Even if we are using a UNIX socket this is applicable. You can use Gitea's default port.
- `SERVER_NAME`: the FQDN
```apache
#########
# Gitea #
#########
<IfModule mod_ssl.c>
<VirtualHost *:443>
UseCanonicalName on
ProxyPreserveHost On
Keepalive On
RewriteEngine on
AllowEncodedSlashes NoDecode
ServerName ${SERVER_NAME}
ProxyBadHeader Ignore
# TOR.
# Remove this if you don't want Gitea being avaliable on TOR.
ServerAlias ${ONION_ADDRESS}
Header set Onion-Location "http://${ONION_ADDRESS}%{REQUEST_URI}s"
SSLCompression off
ProxyPass / unix:/var/run/gitea/gitea.sock|http://127.0.0.1:${TCP_PORT}/ nocanon
ProxyPassReverse / unix:/var/run/gitea/gitea.sock|http://127.0.0.1:${TCP_PORT}/
Include /etc/letsencrypt/options-ssl-apache.conf
SSLCertificateFile /etc/letsencrypt/live/${SERVER_NAME}/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/${SERVER_NAME}/privkey.pem
</VirtualHost>
```
2. if you use TOR add this configuration in `/etc/apache2/apache2.conf` or a virtual host file:
variables (shell style):
- `TCP_PORT`: the TCP/IP port where Gitea should be listenting. Even if we are using a UNIX socket this is applicable. You can use Gitea's default port
- `APACHE_TCP_PORT`: the TCP/IP port where Apache is listening
- `SERVER_NAME`: TOR's FQDN
```apache
#########
# Gitea #
#########
<IfModule mod_ssl.c>
<VirtualHost *:${APACHE_TCP_PORT}>
UseCanonicalName on
ProxyPreserveHost On
ProxyRequests off
AllowEncodedSlashes NoDecode
Keepalive On
RewriteEngine on
ServerName ${SERVER_NAME}
SSLCompression off
# Disable HTTP push and cloning for TOR.
# TODO: enable only cloning but not push.
RewriteRule ^(.*)/info/refs /errors
<Location "/errors">
Deny from all
</Location>
# Redirect api and login to black holes.
# Specific rules first, generic rules last.
ProxyPass /user/login unix:/var/run/gitea/gitea.sock|http://127.0.0.1:${TCP_PORT}/404 nocanon
ProxyPassReverse /user/login/ unix:/var/run/gitea/gitea.sock|http://127.0.0.1:${TCP_PORT}/404
ProxyPass /api unix:/var/run/gitea/gitea.sock|http://127.0.0.1:${TCP_PORT}/404 nocanon
ProxyPassReverse /api unix:/var/run/gitea/gitea.sock|http://127.0.0.1:${TCP_PORT}/404
ProxyPass / unix:/var/run/gitea/gitea.sock|http://127.0.0.1:${TCP_PORT}/ nocanon
ProxyPassReverse / unix:/var/run/gitea/gitea.sock|http://127.0.0.1:${TCP_PORT}/
RequestHeader set X-Forwarded-Proto "http"
RequestHeader set X-Forwarded-Port "${APACHE_TCP_PORT}"
RequestHeader set X-Forwarded-Host "${SERVER_NAME}"
# Distinguish normal traffic from tor's.
RequestHeader set X-Forwarded-For "tor"
</VirtualHost>
</IfModule>
```
3. if you use TOR, add this to `/etc/tor/torrc`
variables (shell style):
- `APACHE_TCP_PORT`: the TCP/IP port where Apache is listening
```conf
HiddenServiceDir /var/lib/tor/gitea/
HiddenServicePort 80 127.0.0.1:${APACHE_TCP_PORT}
```
Follow [these](https://2019.www.torproject.org/docs/tor-onion-service) instructions.
See these links for the UNIX socket explanation:
- [https://httpd.apache.org/docs/trunk/mod/mod_proxy.html#proxypass](https://httpd.apache.org/docs/trunk/mod/mod_proxy.html#proxypass)
- [https://wiki.archlinux.org/title/Talk:Gitea#Apache_Reverse_Proxy_Over_Unix_Socket](https://wiki.archlinux.org/title/Talk:Gitea#Apache_Reverse_Proxy_Over_Unix_Socket)
### GitHub issue
Follow it [here](https://github.com/go-gitea/gitea/issues/16992)