forgejo-docs/docs/developer/infrastructure.md

---
title: Hardware infrastructure
license: 'CC-BY-SA-4.0'
---

## Codeberg

Codeberg provides a LXC container with 48GB RAM, 24 threads and SSD drive to be used for the CI. A Forgejo Runner is installed in `/opt/runner` and registered with a token obtained from https://codeberg.org/forgejo. It does not allow running privileged containers or LXC containers for security reasons. The runner is intended to be used for pull requests, for instance in https://codeberg.org/forgejo/forgejo.

## Octopuce

[Octopuce provides hardware](https://codeberg.org/forgejo/sustainability) managed by [the devops team](https://codeberg.org/forgejo/governance/src/branch/main/TEAMS.md#devops). It can be accessed via a VPN which provides a DNS for the `octopuce.forgejo.org` internal domain.

The VPN is deployed and upgraded using the following [Enough command line](https://enough-community.readthedocs.io):

```shell
$ mkdir -p ~/.enough
$ git clone https://forgejo.octopuce.forgejo.org/forgejo/enough-octopuce ~/.enough/octopuce.forgejo.org
$ enough --domain octopuce.forgejo.org service create openvpn
```

## Hetzner

All hardware is running Debian GNU/linux bookworm.

### hetzner01

https://hetzner01.forgejo.org runs on an [EX101](https://www.hetzner.com/dedicated-rootserver/ex101) Hetzner hardware.

There is no backup, no redundancy and is dedicated to Forgejo runner instances.
If the hardware reboots, the runners do not restart automatically, they have to be restarted manually.

It hosts LXC containers setup with [lxc-helpers](https://code.forgejo.org/forgejo/lxc-helpers/):

- `forgejo-runners`

  Dedicated to Forgejo runners for the https://codeberg.org/forgejo organization.

  ```sh
  lxc-helpers.sh lxc_container_run forgejo-runners -- sudo --user debian bash
  cd codeberg.org/forgejo/
  forgejo-runner-3.2.0 --config config.yml daemon >& runner.log &
  ```

- `runner01-lxc`

  Dedicated to Forgejo runners for the https://code.forgejo.org
  organization with two labels: **docker** and **self-hosted**.

  - https://code.forgejo.org/forgejo
  - https://code.forgejo.org/actions
  - https://code.forgejo.org/forgejo-integration
  - https://code.forgejo.org/forgejo-contrib

  ```sh
  lxc-helpers.sh lxc_container_run runner01-lxc -- sudo --user debian bash
  cd code.forgejo.org
  for runner in forgejo-contrib forgejo forgejo-integration actions ; do ( cd $runner ; HOME=/srv/$runner forgejo-runner-3.2.0 --config config.yml daemon >&runner.log & ) ; done
  ```

The runners are installed with something like:

```sh
sudo wget -O /usr/local/bin/forgejo-runner-3.2.0 https://code.forgejo.org/forgejo/runner/releases/download/v3.2.0/forgejo-runner-3.2.0-linux-amd64
sudo chmod +x /usr/local/bin/forgejo-runner-3.2.0
```

### hetzner{02,03}

https://hetzner02.forgejo.org & https://hetzner03.forgejo.org run on [EX44](https://www.hetzner.com/dedicated-rootserver/ex44) Hetzner hardware.

A vSwitch is assigned via the Robot console on both servers
and [configured](https://docs.hetzner.com/robot/dedicated-server/network/vswitch#example-debian-configuration)
in /etc/network/interfaces for each of them with something like:

```
auto enp5s0.4000
iface enp5s0.4000 inet static
  address 10.53.100.2
  netmask 255.255.255.0
  vlan-raw-device enp5s0
  mtu 1400
```

#### DRBD

DRBD is configured with hetzner02 as the primary and hetzner03 as the secondary:

```
resource r0 {
    net {
        # A : write completion is determined when data is written to the local disk and the local TCP transmission buffer
        # B : write completion is determined when data is written to the local disk and remote buffer cache
        # C : write completion is determined when data is written to both the local disk and the remote disk
        protocol C;
        cram-hmac-alg sha1;
        # any secret key for authentication among nodes
        shared-secret "***";
    }
    disk {
        resync-rate 1000M;
    }
    on hetzner02 {
        address 10.53.100.2:7788;
        volume 0 {
            # device name
            device /dev/drbd0;
            # specify disk to be used for devide above
            disk /dev/nvme0n1p5;
            # where to create metadata
            # specify the block device name when using a different disk
            meta-disk internal;
        }
    }
    on hetzner03 {
        address 10.53.100.3:7788;
        volume 0 {
            device /dev/drbd0;
            disk /dev/nvme1n1p5;
            meta-disk internal;
        }
    }
}
```

The DRBD device is mounted on `/var/lib/lxc`.

In `/etc/fstab` there is a noauto line:

```
/dev/drbd0 /var/lib/lxc ext4 noauto,defaults 0 0
```

To prevent split brain situations a manual step is required at boot
time, on the machine that is going to be the primary, which is
hetzner02 in a normal situation.

```sh
sudo drbdsetup status
sudo drbdadm primary r0
sudo mount /var/lib/lxc
sudo lxc-autostart start
sudo lxc-ls -f
sudo drbdsetup status
```

#### Fast storage on /srv

The second disk on each node is mounted on /srv and can be used when
fast storage is needed and there is no need for backups, such as Forgejo runners.

#### LXC

LXC is setup with [lxc-helpers](https://code.forgejo.org/forgejo/lxc-helpers/).

The `/etc/default/lxc-net` file is the same on both machines:

```
USE_LXC_BRIDGE="true"
LXC_ADDR="10.6.83.1"
LXC_NETMASK="255.255.255.0"
LXC_NETWORK="10.6.83.0/24"
LXC_DHCP_RANGE="10.6.83.2,10.6.83.254"
LXC_DHCP_MAX="253"
LXC_IPV6_ADDR="fc16::216:3eff:fe00:1"
LXC_IPV6_MASK="64"
LXC_IPV6_NETWORK="fc16::/64"
LXC_IPV6_NAT="true"
```

#### Public IP addresses

The public IP addresses attached to the hosts are not failover IPs that can be moved from one host to the next.
The DNS entry needs to be updated if the primary hosts changes.

#### Containers

It hosts LXC containers setup with [lxc-helpers](https://code.forgejo.org/forgejo/lxc-helpers/).

- `code` on hetzner02

  Dedicated to https://code.forgejo.org

- `runner-forgejo-helm` on hetzner03

  Dedicated to https://codeberg.org/forgejo-contrib/forgejo-helm and running from an ephemral disk

## Uberspace

The website https://forgejo.org is hosted at
https://uberspace.de/. The https://codeberg.org/forgejo/website/ CI
has credentials to push HTML pages there.

## Installing Forgejo runners

### Preparing the LXC hypervisor

```shell
git clone https://code.forgejo.org/forgejo/lxc-helpers/

lxc-helpers.sh lxc_prepare_environment
sudo lxc-helpers.sh lxc_install_lxc_inside 10.120.13
```

### Creating an LXC container

```shell
lxc-helpers.sh lxc_container_create forgejo-runners
lxc-helpers.sh lxc_container_start forgejo-runners
lxc-helpers.sh lxc_container_user_install forgejo-runners $(id -u) $USER
lxc-helpers.sh lxc_container_run forgejo-runners -- sudo --user debian bash
sudo apt-get update
sudo apt-get install -y wget docker.io emacs-nox
sudo usermod -aG docker $USER # exit & enter again for the group to be active
lxc-helpers.sh lxc_prepare_environment
sudo wget -O /usr/local/bin/forgejo-runner https://code.forgejo.org/forgejo/runner/releases/download/v2.0.4/forgejo-runner-amd64
sudo chmod +x /usr/local/bin/forgejo-runner
echo 'export TERM=vt100' >> .bashrc
```

### Creating a runner

Multiple runners can co-exist on the same machine. To keep things
organized they are located in a directtory that is the same as the url
from which the token is obtained. For instance
DIR=codeberg.org/forgejo-integration means that the token was obtained from the
https://codeberg.org/forgejo-integration organization.

If a runner only provides unprivileged docker containers, the labels
should be
`LABELS=docker:docker://node:16-bullseye,ubuntu-latest:docker://node:16-bullseye`.

If a runner provides LXC containers and unprivileged docker
containers, the labels should be
`LABELS=docker:docker://node:16-bullseye,self-hosted`.

```shell
mkdir -p $DIR ; cd $DIR
forgejo-runner generate-config > config.yml
## edit config.yml
## Obtain a $TOKEN from https://$DIR
forgejo-runner register --no-interactive --token $TOKEN --name runner --instance https://codeberg.org --labels $LABELS
forgejo-runner --config config.yml daemon |& cat -v > runner.log &
```

#### codeberg.org config.yml

- `fetch_timeout: 30s` # because it can be slow at times
- `fetch_interval: 60s` # because there is throttling and 429 replies will mess up the runner
- cache `enabled: false` # because codeberg.org is still v1.19