0
0
Fork 0
mirror of https://codeberg.org/forgejo/docs.git synced 2024-11-29 18:52:48 -05:00
forgejo-docs/docs/admin/storage.md
Earl Warren 852b447f91
admin: storage: updates and improve S3
The original documentation was written with a focus on local storage
and missed a few important aspects of how storage types can be mixed
together in some subsystems.

* unify the susbsystem paths to be "directory" instead of using "base
  path". It is not technically a directory in S3 but looks like one
  and the approximation is unlikely to cause confusion.
* use S3 instead of minio where possible to emphasize the storage type
  is not MinIO specific and add a section for the compatibility with
  garage
* change the document structure to have two separate parts:
  * [storage] the global sections
  * [xxxx] the subsystem specific section
  and within each of them local and S3 are documented separately
  because they share nothing
* [storage] is documented to be a fallback if [xxxx] does not exist
  for a given subsystem. There is no notion of inheritance because
  it behaves in ways that are not tested and for which consistency is
  not guaranteed
* backward compatibility with Gitea is documented to be the reason why
  there are no safeguards against undocumented features
2023-08-31 12:35:32 +02:00

7.9 KiB

title license
Storage settings CC-BY-SA-4.0

The storage for each subsystem is defined in app.ini. It can either be on disk (local which is the default) or using a S3 compatible server (minio). In both cases each subsystem stores all files (or objects in the S3 parlance) in a dedicated directory as shown in the table below:

subsystem directory app.ini sections
Attachments attachments/ [attachment]
LFS lfs/ [lfs]
Avatars avatars/ [avatar]
Repository avatars repo-avatars/ [repo-avatar]
Repository archives repo-archive/ [repo-archive]
Packages packages/ [packages]
Actions logs actions_log/ [storage.actions_log]
Actions Artifacts actions_artifacts/ [actions.artifacts]

For instance:

  • if the STORAGE_TYPE is local and APP_DATA_PATH is /appdata, the default directory to store attachments is /appdata/attachments
  • if the STORAGE_TYPE is minio, the default directory to store attachments within the MINIO_BUCKET bucket will be attachments/.

Changing the storage for all subsystems

The [storage] section can be used to modify the storage of all subsystems. The default is to use local storage under APP_DATA_PATH and is equivalent to writing the following in app.ini:

[server]
APP_DATA_PATH = /forgejo/data

[storage]
STORAGE_TYPE = local
PATH = /forgejo/data

Using local

For local storage, the [storage] section can only be used to change the path under which all subsystems directories will be created, using the PATH settings with an absolute pathname.

For instance:

[storage]
STORAGE_TYPE = local
PATH = /mystorage

will change the default for storing attachments to /mystorage/attachments, LFS to /mystorage/lfs etc.

Using minio

The [storage] section can be used to change the default storage type used by all subsystems to minio.

For instance:

[storage]
STORAGE_TYPE = minio

MINIO_ENDPOINT = 127.0.0.1:9000
MINIO_ACCESS_KEY_ID = [redacted]
MINIO_SECRET_ACCESS_KEY = [redacted]
MINIO_BUCKET = forgejo
MINIO_LOCATION = us-east-1

will change the default for storing attachments to attachments/ in the forgejo bucket, LFS to lfs/ in the forgejo bucket etc.

NOTE: MINIO_BASE_PATH must not be set in the [storage] section.

Storage settings for a single subsystem

It is possible to configure some subsystems to use S3 storage and others to use local storage by adding settings to their respective sections. For instance:

[attachment]
PATH = /otherstorage/attachments

[lfs]
STORAGE_TYPE = minio
MINIO_BASE_PATH = lfs/

MINIO_ENDPOINT = 127.0.0.1:9000
MINIO_ACCESS_KEY_ID = [redacted]
MINIO_SECRET_ACCESS_KEY = [redacted]
MINIO_BUCKET = forgejo
MINIO_LOCATION = us-east-1

will store attachments in the local directory /otherstorage/attachments while lfs files will be stored in the S3 server within the lfs/ directory of the forgejo bucket.

Storage settings

The value of STORAGE_TYPE can be local (the default) for file system directories or minio for S3 servers. Each storage type has its own settings, as explained below.

Using local

There is just one setting when the STORAGE_TYPE is set to local: PATH. It must be an absolute path and is interpreted as follows.

In the [storage] section, PATH is the path under which the directories of each subsystem will be created instead of APP_DATA_PATH. For instance, if APP_DATA_PATH equals /appdata

[storage]
STORAGE_TYPE = local
PATH = /mystorage

will create attachments in /mystorage/attachments instead of /appdata/attachments, LFS files in /mystorage/lfs instead of /appdata/lfs, etc.

In the section dedicated to a subsystem (see the table in the introduction), PATH is the base path under which all files will be stored. For instance:

[storage]
STORAGE_TYPE = local
PATH = /mystorage

[attachment]
STORAGE_TYPE = local
PATH = /otherstorage/attachments

will store attachments in /otherstorage/attachments while lfs files will be stored in /mystorage/lfs.

Using minio

When the STORAGE_TYPE is set to minio, the settings are used to to connect to a S3 compatible server:

  • SERVE_DIRECT: false: Allows the storage driver to redirect to authenticated URLs to serve files directly. Only supported via signed URLs.
  • MINIO_ENDPOINT: localhost:9000: S3 endpoint to connect.
  • MINIO_ACCESS_KEY_ID: S3 accessKeyID to connect.
  • MINIO_SECRET_ACCESS_KEY: S3 secretAccessKey to connect.
  • MINIO_BUCKET: forgejo: S3 bucket to store the data.
  • MINIO_LOCATION: us-east-1: S3 location to create bucket.
  • MINIO_USE_SSL: false: S3 enabled ssl.
  • MINIO_INSECURE_SKIP_VERIFY: false: S3 skip SSL verification.

When used in the [storage] section they apply to all subsystems. When used in the section specific to a subsystem (see the table in the introduction), they are only used for objects that belong to this subsystem. Here is a example:

[storage]
STORAGE_TYPE = minio

SERVE_DIRECT = false
MINIO_ENDPOINT = garage:9000
MINIO_ACCESS_KEY_ID = [redacted]
MINIO_SECRET_ACCESS_KEY = [redacted]
MINIO_BUCKET = forgejo
MINIO_LOCATION = us-east-1
MINIO_USE_SSL = false
MINIO_INSECURE_SKIP_VERIFY = false

[lfs]
STORAGE_TYPE = minio
MINIO_BASE_PATH = nonstandardlfs/

SERVE_DIRECT = false
MINIO_ENDPOINT = minio:9000
MINIO_ACCESS_KEY_ID = [redacted]
MINIO_SECRET_ACCESS_KEY = [redacted]
MINIO_BUCKET = forgejo
MINIO_LOCATION = us-east-1
MINIO_USE_SSL = false
MINIO_INSECURE_SKIP_VERIFY = false
  • MINIO_BASE_PATH: only valid in the specific subsystem section (see the table in the introduction) overrides the default directory in which objects are stored in the MINIO_BUCKET bucket.

For all subsystems that use the minio storage type found in the [storage] section, the directory in which the objects are stored is determined using the table in the introduction. For instance LFS files will be stored in the lfs/ directory within the forgejo bucket.

When the minio storage is set in a section specific to a subsystem, the MINIO_BASE_PATH setting can be used to override the default directory. In the example above, MINIO_BASE_PATH = nonstandardlfs/ means LFS objects will be stored in the nonstandardlfs/ directory within the forgejo bucket instead of the lfs/ directory

S3 servers compatibility

Although the S3 storage type is named minio it does not rely on any MinIO specific features. The S3 storage type is tested to be compatible with:

Undocumented features

It is strongly recommended to avoid using undocumented features - such as [storage.attachments] as an alternative to [attachment] for instance (the plural is not a typo, it is a unification problem) - because their behavior is not thoroughly tested and may lead to unexpected results.

There are no safeguards preventing the use of undocumented features because it may not be backward compatible when upgrading from Gitea to Forgejo.

Legacy settings

Some settings are deprecated but still supported in the interest of backward compatibility. They should be replaced as follows:

  • [server].LFS_CONTENT_PATH is replaced with [lfs].PATH
  • [picture].AVATAR_UPLOAD_PATH is replaced with [avatar].PATH
  • [picture].REPOSITORY_AVATAR_UPLOAD_PATH is replaced with [repo-avatar].PATH

Legacy settings have a lower priority and will be overridden by their replacement if both are present. For instance:

[picture]
AVATAR_UPLOAD_PATH = /legacy_path

[avatar]
PATH = /avatar_path

will store avatar files in /avatar_path.