foster/forgejo-docs
0
0
Fork 0
mirror of https://codeberg.org/forgejo/docs.git synced 2024-11-21 17:36:59 -05:00
Code Issues Activity

admin: storage: updates and improve S3

Browse source 
The original documentation was written with a focus on local storage
and missed a few important aspects of how storage types can be mixed
together in some subsystems.

* unify the susbsystem paths to be "directory" instead of using "base
  path". It is not technically a directory in S3 but looks like one
  and the approximation is unlikely to cause confusion.
* use S3 instead of minio where possible to emphasize the storage type
  is not MinIO specific and add a section for the compatibility with
  garage
* change the document structure to have two separate parts:
  * [storage] the global sections
  * [xxxx] the subsystem specific section
  and within each of them local and S3 are documented separately
  because they share nothing
* [storage] is documented to be a fallback if [xxxx] does not exist
  for a given subsystem. There is no notion of inheritance because
  it behaves in ways that are not tested and for which consistency is
  not guaranteed
* backward compatibility with Gitea is documented to be the reason why
  there are no safeguards against undocumented features
This commit is contained in:
Earl Warren 2023-08-31 11:41:25 +02:00
parent 693f849196
commit 852b447f91
No known key found for this signature in database
GPG key ID: 0579CB2928A78A00
1 changed files with 184 additions and 74 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

258
docs/admin/storage.md
View file

@ -3,11 +3,13 @@ title: 'Storage settings'
license: 'CC-BY-SA-4.0' license: 'CC-BY-SA-4.0'
--- ---
The storage for each subsystem is defined in `app.ini`. It can either be on disk The storage for each subsystem is defined in `app.ini`. It can either
(`local`) or using a MinIO server (`minio`). The default is `local` be on disk (`local` which is the default) or using a S3 compatible
storage, using the following hierarchy under the `APP_DATA_PATH` directory: server (`minio`). In both cases each subsystem stores all files (or
objects in the S3 parlance) in a dedicated directory as shown in the
table below:
| subsystem | default base path | app.ini sections | | subsystem | directory | app.ini sections |
| ------------------- | ------------------ | --------------------- | | ------------------- | ------------------ | --------------------- |
| Attachments | attachments/ | [attachment] | | Attachments | attachments/ | [attachment] |
| LFS | lfs/ | [lfs] | | LFS | lfs/ | [lfs] |
@ -18,55 +20,37 @@ storage, using the following hierarchy under the `APP_DATA_PATH` directory:
| Actions logs | actions_log/ | [storage.actions_log] | | Actions logs | actions_log/ | [storage.actions_log] |
| Actions Artifacts | actions_artifacts/ | [actions.artifacts] | | Actions Artifacts | actions_artifacts/ | [actions.artifacts] |
For instance if `APP_DATA_PATH` was `/appdata`, the default directory to For instance:
store attachments will be `/appdata/attachments`.
## Overriding the defaults - if the `STORAGE_TYPE` is `local` and `APP_DATA_PATH` is
`/appdata`, the default directory to store attachments is
`/appdata/attachments`
- if the `STORAGE_TYPE` is `minio`, the
default directory to store attachments within the `MINIO_BUCKET`
bucket will be `attachments/`.
These defaults can be modified for all subsystems in the `[storage]` ## Changing the storage for all subsystems
section. For instance setting:
The `[storage]` section can be used to modify the storage of all
subsystems. The default is to use `local` storage under
`APP_DATA_PATH` and is equivalent to writing the following in `app.ini`:
``` ```
[server]
APP_DATA_PATH = /forgejo/data
[storage] [storage]
PATH = /mystorage STORAGE_TYPE = local
PATH = /forgejo/data
``` ```
will change the default for storing attachments to ### Using local
`/mystorage/attachments`. It is also possible to change these settings
for each subsystem in their dedicated section. For instance:
``` For `local` storage, the `[storage]` section can only be used to
[storage] change the path under which all subsystems directories will be
PATH = /mystorage created, using the `PATH` settings with an absolute pathname.
[attachment] For instance:
PATH = /otherstorage/attachments
```
will store attachments in `/otherstorage/attachments` while `lfs`
files will be stored in `/mystorage/lfs`.
## Storage type
The value of `STORAGE_TYPE` can be `local` (the default) or `minio`. For instance:
```
[storage]
STORAGE_TYPE = minio
```
will use `minio` for all subsystems (Attachments, LFS, etc.)
instead of storing them on disk. Each storage type has its own
settings, as explained below.
## `local` storage
There is just one setting when the `STORAGE_TYPE` is set to `local`,
`PATH`. It must be an absolute path and is interpreted as follows.
In the `[storage]` section, `PATH` is the directory under which the default
base path of each subsystem will be created instead of
`APP_DATA_PATH`. For instance, if `APP_DATA_PATH` equals `/appdata`:
``` ```
[storage] [storage]
@ -74,11 +58,82 @@ STORAGE_TYPE = local
PATH = /mystorage PATH = /mystorage
``` ```
Will create attachments in `/mystorage/attachments` instead of will change the default for storing attachments to
`/mystorage/attachments`, LFS to `/mystorage/lfs` etc.
### Using minio
The `[storage]` section can be used to change the default storage type
used by all subsystems to `minio`.
For instance:
```
[storage]
STORAGE_TYPE = minio
MINIO_ENDPOINT = 127.0.0.1:9000
MINIO_ACCESS_KEY_ID = [redacted]
MINIO_SECRET_ACCESS_KEY = [redacted]
MINIO_BUCKET = forgejo
MINIO_LOCATION = us-east-1
```
will change the default for storing attachments to `attachments/` in
the `forgejo` bucket, LFS to `lfs/` in the `forgejo` bucket etc.
> **NOTE:** `MINIO_BASE_PATH` must not be set in the `[storage]` section.
## Storage settings for a single subsystem
It is possible to configure some subsystems to use S3 storage and others to use local
storage by adding settings to their respective sections. For instance:
```
[attachment]
PATH = /otherstorage/attachments
[lfs]
STORAGE_TYPE = minio
MINIO_BASE_PATH = lfs/
MINIO_ENDPOINT = 127.0.0.1:9000
MINIO_ACCESS_KEY_ID = [redacted]
MINIO_SECRET_ACCESS_KEY = [redacted]
MINIO_BUCKET = forgejo
MINIO_LOCATION = us-east-1
```
will store attachments in the local directory
`/otherstorage/attachments` while `lfs` files will be stored in the S3
server within the `lfs/` directory of the `forgejo` bucket.
## Storage settings
The value of `STORAGE_TYPE` can be `local` (the default) for file
system directories or `minio` for S3 servers. Each storage type has
its own settings, as explained below.
### Using local
There is just one setting when the `STORAGE_TYPE` is set to `local`:
`PATH`. It must be an absolute path and is interpreted as follows.
In the `[storage]` section, `PATH` is the path under which the
directories of each subsystem will be created instead of
`APP_DATA_PATH`. For instance, if `APP_DATA_PATH` equals `/appdata`
```
[storage]
STORAGE_TYPE = local
PATH = /mystorage
```
will create attachments in `/mystorage/attachments` instead of
`/appdata/attachments`, LFS files in `/mystorage/lfs` instead of `/appdata/attachments`, LFS files in `/mystorage/lfs` instead of
`/appdata/lfs`, etc. `/appdata/lfs`, etc.
In the section dedicated to a subsystem (see the table above), `PATH` In the section dedicated to a subsystem (see the table in the introduction), `PATH`
is the base path under which all files will be stored. For instance: is the base path under which all files will be stored. For instance:
``` ```
@ -94,44 +149,73 @@ PATH = /otherstorage/attachments
will store attachments in `/otherstorage/attachments` while `lfs` will store attachments in `/otherstorage/attachments` while `lfs`
files will be stored in `/mystorage/lfs`. files will be stored in `/mystorage/lfs`.
## `minio` storage ### Using minio
When the `STORAGE_TYPE` is set to `minio`, the settings available in When the `STORAGE_TYPE` is set to `minio`, the settings are used to to
all sections (`[storage]` and `[XXXX]`) are: connect to a S3 compatible server:
- `SERVE_DIRECT`: **false**: Allows the storage driver to redirect to authenticated URLs to serve files directly. Only supported via signed URLs. - `SERVE_DIRECT`: **false**: Allows the storage driver to redirect to authenticated URLs to serve files directly. Only supported via signed URLs.
- `MINIO_ENDPOINT`: **localhost:9000**: Minio endpoint to connect. - `MINIO_ENDPOINT`: **localhost:9000**: S3 endpoint to connect.
- `MINIO_ACCESS_KEY_ID`: Minio accessKeyID to connect. - `MINIO_ACCESS_KEY_ID`: S3 accessKeyID to connect.
- `MINIO_SECRET_ACCESS_KEY`: Minio secretAccessKey to connect. - `MINIO_SECRET_ACCESS_KEY`: S3 secretAccessKey to connect.
- `MINIO_BUCKET`: **gitea**: Minio bucket to store the data. - `MINIO_BUCKET`: **forgejo**: S3 bucket to store the data.
- `MINIO_LOCATION`: **us-east-1**: Minio location to create bucket. - `MINIO_LOCATION`: **us-east-1**: S3 location to create bucket.
- `MINIO_USE_SSL`: **false**: Minio enabled ssl. - `MINIO_USE_SSL`: **false**: S3 enabled ssl.
- `MINIO_INSECURE_SKIP_VERIFY`: **false**: Minio skip SSL verification. - `MINIO_INSECURE_SKIP_VERIFY`: **false**: S3 skip SSL verification.
One setting is only available in the `[XXXX]` sections: When used in the `[storage]` section they apply to all
subsystems. When used in the section specific to a subsystem (see the table in the introduction), they
- `MINIO_BASE_PATH`: defaults to the `default base path` of the `XXXX` are only used for objects that belong to this subsystem. Here is a example:
subsystem (see the table above) and is a relative path within the
MinIO bucket defined by `MINIO_BUCKET`.
## Sections precedence
The sections in which a setting is found have the following priority:
- [XXXX] has precedence
- [storage] is the default
For instance:
``` ```
[storage] [storage]
PATH = /default STORAGE_TYPE = minio
[attachment] SERVE_DIRECT = false
PATH = /first MINIO_ENDPOINT = garage:9000
MINIO_ACCESS_KEY_ID = [redacted]
MINIO_SECRET_ACCESS_KEY = [redacted]
MINIO_BUCKET = forgejo
MINIO_LOCATION = us-east-1
MINIO_USE_SSL = false
MINIO_INSECURE_SKIP_VERIFY = false
[lfs]
STORAGE_TYPE = minio
MINIO_BASE_PATH = nonstandardlfs/
SERVE_DIRECT = false
MINIO_ENDPOINT = minio:9000
MINIO_ACCESS_KEY_ID = [redacted]
MINIO_SECRET_ACCESS_KEY = [redacted]
MINIO_BUCKET = forgejo
MINIO_LOCATION = us-east-1
MINIO_USE_SSL = false
MINIO_INSECURE_SKIP_VERIFY = false
``` ```
Will set the value of `PATH` for attachments to `/first`. - `MINIO_BASE_PATH`: **only valid in the specific subsystem section (see the table in the introduction)**
overrides the default directory in which objects are stored in the `MINIO_BUCKET` bucket.
For all subsystems that use the `minio` storage type found in the
`[storage]` section, the directory in which the objects are stored is
determined using the table in the introduction. For instance LFS files will be
stored in the `lfs/` directory within the `forgejo` bucket.
When the `minio` storage is set in a section specific to a subsystem,
the `MINIO_BASE_PATH` setting can be used to override the default
directory. In the example above, `MINIO_BASE_PATH = nonstandardlfs/`
means LFS objects will be stored in the `nonstandardlfs/` directory
within the `forgejo` bucket instead of the `lfs/` directory
## S3 servers compatibility
Although the S3 storage type is named `minio` it does not rely on any
[MinIO](https://min.io/) specific features. The S3 storage type is
[tested](https://codeberg.org/forgejo/forgejo/src/branch/forgejo/.forgejo/upgrades/test-upgrade.sh) to be compatible with:
- [MinIO](https://min.io/) 2021.3.17 and 2023-08-23
- [garage](https://garagehq.deuxfleurs.fr/) v0.8.2
## Undocumented features ## Undocumented features
@ -140,3 +224,29 @@ such as `[storage.attachments]` as an alternative to `[attachment]`
for instance (the plural is not a typo, it is a unification problem) - for instance (the plural is not a typo, it is a unification problem) -
because their behavior is not thoroughly tested and may lead to because their behavior is not thoroughly tested and may lead to
unexpected results. unexpected results.
There are no safeguards preventing the use of undocumented features
because it may not be backward compatible when upgrading from Gitea to
Forgejo.
## Legacy settings
Some settings are deprecated but still supported in the interest of
backward compatibility. They should be replaced as follows:
- `[server].LFS_CONTENT_PATH` is replaced with `[lfs].PATH`
- `[picture].AVATAR_UPLOAD_PATH` is replaced with `[avatar].PATH`
- `[picture].REPOSITORY_AVATAR_UPLOAD_PATH` is replaced with `[repo-avatar].PATH`
Legacy settings have a lower priority and will be overridden by their
replacement if both are present. For instance:
```
[picture]
AVATAR_UPLOAD_PATH = /legacy_path
[avatar]
PATH = /avatar_path
```
will store avatar files in `/avatar_path`.