From d1b6f00b45995e8e5383f464f93719daa5d198a5 Mon Sep 17 00:00:00 2001 From: Earl Warren Date: Tue, 8 Aug 2023 00:03:00 +0200 Subject: [PATCH] docs: admin: storage: create a new section --- admin/config-cheat-sheet.md | 148 ++++-------------------------------- admin/index.md | 1 + admin/storage.md | 129 +++++++++++++++++++++++++++++++ 3 files changed, 143 insertions(+), 135 deletions(-) create mode 100644 admin/storage.md diff --git a/admin/config-cheat-sheet.md b/admin/config-cheat-sheet.md index e1f4850e..0cfce0e0 100644 --- a/admin/config-cheat-sheet.md +++ b/admin/config-cheat-sheet.md @@ -474,8 +474,8 @@ Configuration at `[queue]` will set defaults for queues with overrides for indiv - `LENGTH`: **100**: Maximal queue size before channel queues block - `BATCH_LENGTH`: **20**: Batch data before passing to the handler - `CONN_STR`: **redis://127.0.0.1:6379/0**: Connection string for the redis queue type. For `redis-cluster` use `redis+cluster://127.0.0.1:6379/0`. Options can be set using query params. Similarly, LevelDB options can also be set using: **leveldb://relative/path?option=value** or **leveldb:///absolute/path?option=value**, and will override `DATADIR` -- `QUEUE_NAME`: **_queue**: The suffix for default redis and disk queue name. Individual queues will default to **`name`**`QUEUE_NAME` but can be overridden in the specific `queue.name` section. -- `SET_NAME`: **_unique**: The suffix that will be added to the default redis and disk queue `set` name for unique queues. Individual queues will default to **`name`**`QUEUE_NAME`_`SET_NAME`_ but can be overridden in the specific `queue.name` section. +- `QUEUE_NAME`: **\_queue**: The suffix for default redis and disk queue name. Individual queues will default to **`name`**`QUEUE_NAME` but can be overridden in the specific `queue.name` section. +- `SET_NAME`: **\_unique**: The suffix that will be added to the default redis and disk queue `set` name for unique queues. Individual queues will default to **`name`**`QUEUE_NAME`_`SET_NAME`_ but can be overridden in the specific `queue.name` section. - `MAX_WORKERS`: **(dynamic)**: Maximum number of worker go-routines for the queue. Default value is "CpuNum/2" clipped to between 1 and 10. Forgejo creates the following non-unique queues: @@ -800,18 +800,6 @@ Default templates for project boards: - `ALLOWED_TYPES`: **.csv,.docx,.fodg,.fodp,.fods,.fodt,.gif,.gz,.jpeg,.jpg,.log,.md,.mov,.mp4,.odf,.odg,.odp,.ods,.odt,.patch,.pdf,.png,.pptx,.svg,.tgz,.txt,.webm,.xls,.xlsx,.zip**: Comma-separated list of allowed file extensions (`.zip`), mime types (`text/plain`) or wildcard type (`image/*`, `audio/*`, `video/*`). Empty value or `*/*` allows all types. - `MAX_SIZE`: **4**: Maximum size (MB). - `MAX_FILES`: **5**: Maximum number of attachments that can be uploaded at once. -- `STORAGE_TYPE`: **local**: Storage type for attachments, `local` for local disk or `minio` for s3 compatible object storage service, default is `local` or other name defined with `[storage.xxx]` -- `SERVE_DIRECT`: **false**: Allows the storage driver to redirect to authenticated URLs to serve files directly. Currently, only Minio/S3 is supported via signed URLs, local does nothing. -- `PATH`: **data/attachments**: Path to store attachments only available when STORAGE_TYPE is `local` -- `MINIO_ENDPOINT`: **localhost:9000**: Minio endpoint to connect only available when STORAGE_TYPE is `minio` -- `MINIO_ACCESS_KEY_ID`: Minio accessKeyID to connect only available when STORAGE_TYPE is `minio` -- `MINIO_SECRET_ACCESS_KEY`: Minio secretAccessKey to connect only available when STORAGE_TYPE is `minio` -- `MINIO_BUCKET`: **gitea**: Minio bucket to store the attachments only available when STORAGE_TYPE is `minio` -- `MINIO_LOCATION`: **us-east-1**: Minio location to create bucket only available when STORAGE_TYPE is `minio` -- `MINIO_BASE_PATH`: **attachments/**: Minio base path on the bucket only available when STORAGE_TYPE is `minio` -- `MINIO_USE_SSL`: **false**: Minio enabled ssl only available when STORAGE_TYPE is `minio` -- `MINIO_INSECURE_SKIP_VERIFY`: **false**: Minio skip SSL verification available when STORAGE_TYPE is `minio` -- `MINIO_CHECKSUM_ALGORITHM`: **default**: Minio checksum algorithm: `default` (for MinIO or AWS S3) or `md5` (for Cloudflare or Backblaze) ## Log (`log`) @@ -1218,128 +1206,20 @@ WARNING: Changing the settings below can break federation. - `DEFAULT_INTERVAL`: **8h**: Default interval between each check - `MIN_INTERVAL`: **10m**: Minimum interval for checking. (Must be >1m). -## LFS (`lfs`) - -Storage configuration for lfs data. It will be derived from default `[storage]` or -`[storage.xxx]` when set `STORAGE_TYPE` to `xxx`. When derived, the default of `PATH` -is `data/lfs` and the default of `MINIO_BASE_PATH` is `lfs/`. - -- `STORAGE_TYPE`: **local**: Storage type for lfs, `local` for local disk or `minio` for s3 compatible object storage service or other name defined with `[storage.xxx]` -- `SERVE_DIRECT`: **false**: Allows the storage driver to redirect to authenticated URLs to serve files directly. Currently, only Minio/S3 is supported via signed URLs, local does nothing. -- `PATH`: **./data/lfs**: Where to store LFS files, only available when `STORAGE_TYPE` is `local`. If not set it fall back to deprecated LFS_CONTENT_PATH value in [server] section. -- `MINIO_ENDPOINT`: **localhost:9000**: Minio endpoint to connect only available when `STORAGE_TYPE` is `minio` -- `MINIO_ACCESS_KEY_ID`: Minio accessKeyID to connect only available when `STORAGE_TYPE` is `minio` -- `MINIO_SECRET_ACCESS_KEY`: Minio secretAccessKey to connect only available when `STORAGE_TYPE is` `minio` -- `MINIO_BUCKET`: **gitea**: Minio bucket to store the lfs only available when `STORAGE_TYPE` is `minio` -- `MINIO_LOCATION`: **us-east-1**: Minio location to create bucket only available when `STORAGE_TYPE` is `minio` -- `MINIO_BASE_PATH`: **lfs/**: Minio base path on the bucket only available when `STORAGE_TYPE` is `minio` -- `MINIO_USE_SSL`: **false**: Minio enabled ssl only available when `STORAGE_TYPE` is `minio` -- `MINIO_INSECURE_SKIP_VERIFY`: **false**: Minio skip SSL verification available when STORAGE_TYPE is `minio` - ## Storage (`storage`) -Default storage configuration for attachments, lfs, avatars, repo-avatars, repo-archive, packages, actions_log, actions_artifact. +| storage | default base path | app.ini sections | +| ----------------- | ------------------ | -------------------------------------------------- | +| attachments | attachments/ | [attachments] or [storage.attachements] | +| lfs | lfs/ | [lfs] or [storage.lfs] | +| avatars | avatars/ | [avatars] or [storage.avatars] | +| repo-avatars | repo-avatars/ | [repo-avatars] or [storage.repo-avatars] | +| repo-archive | repo-archive/ | [repo-archive] or [storage.repo-archive] | +| packages | packages/ | [packages] or [storage.packages] | +| actions_log | actions_log/ | [actions_log] or [storage.actions_log] | +| actions_artifacts | actions_artifacts/ | [actions_artifacts] or [storage.actions_artifacts] | -- `STORAGE_TYPE`: **local**: Storage type, `local` for local disk or `minio` for s3 compatible object storage service. -- `SERVE_DIRECT`: **false**: Allows the storage driver to redirect to authenticated URLs to serve files directly. Currently, only Minio/S3 is supported via signed URLs, local does nothing. -- `MINIO_ENDPOINT`: **localhost:9000**: Minio endpoint to connect only available when `STORAGE_TYPE` is `minio` -- `MINIO_ACCESS_KEY_ID`: Minio accessKeyID to connect only available when `STORAGE_TYPE` is `minio` -- `MINIO_SECRET_ACCESS_KEY`: Minio secretAccessKey to connect only available when `STORAGE_TYPE is` `minio` -- `MINIO_BUCKET`: **gitea**: Minio bucket to store the data only available when `STORAGE_TYPE` is `minio` -- `MINIO_LOCATION`: **us-east-1**: Minio location to create bucket only available when `STORAGE_TYPE` is `minio` -- `MINIO_USE_SSL`: **false**: Minio enabled ssl only available when `STORAGE_TYPE` is `minio` -- `MINIO_INSECURE_SKIP_VERIFY`: **false**: Minio skip SSL verification available when STORAGE_TYPE is `minio` - -The recommanded storage configuration for minio like below: - -```ini -[storage] -STORAGE_TYPE = minio -; Minio endpoint to connect only available when STORAGE_TYPE is `minio` -MINIO_ENDPOINT = localhost:9000 -; Minio accessKeyID to connect only available when STORAGE_TYPE is `minio` -MINIO_ACCESS_KEY_ID = -; Minio secretAccessKey to connect only available when STORAGE_TYPE is `minio` -MINIO_SECRET_ACCESS_KEY = -; Minio bucket to store the attachments only available when STORAGE_TYPE is `minio` -MINIO_BUCKET = gitea -; Minio location to create bucket only available when STORAGE_TYPE is `minio` -MINIO_LOCATION = us-east-1 -; Minio enabled ssl only available when STORAGE_TYPE is `minio` -MINIO_USE_SSL = false -; Minio skip SSL verification available when STORAGE_TYPE is `minio` -MINIO_INSECURE_SKIP_VERIFY = false -SERVE_DIRECT = true -``` - -Defaultly every storage has their default base path like below - -| storage | default base path | -| ----------------- | ------------------ | -| attachments | attachments/ | -| lfs | lfs/ | -| avatars | avatars/ | -| repo-avatars | repo-avatars/ | -| repo-archive | repo-archive/ | -| packages | packages/ | -| actions_log | actions_log/ | -| actions_artifacts | actions_artifacts/ | - -And bucket, basepath or `SERVE_DIRECT` could be special or overrided, if you want to use a different you can: - -```ini -[storage.actions_log] -MINIO_BUCKET = forgejo_actions_log -SERVE_DIRECT = true -MINIO_BASE_PATH = my_actions_log/ ; default is actions_log/ if blank -``` - -If you want to customerize a different storage for `lfs` if above default storage defined - -```ini -[lfs] -STORAGE_TYPE = my_minio - -[storage.my_minio] -STORAGE_TYPE = minio -; Minio endpoint to connect only available when STORAGE_TYPE is `minio` -MINIO_ENDPOINT = localhost:9000 -; Minio accessKeyID to connect only available when STORAGE_TYPE is `minio` -MINIO_ACCESS_KEY_ID = -; Minio secretAccessKey to connect only available when STORAGE_TYPE is `minio` -MINIO_SECRET_ACCESS_KEY = -; Minio bucket to store the attachments only available when STORAGE_TYPE is `minio` -MINIO_BUCKET = forgejo -; Minio location to create bucket only available when STORAGE_TYPE is `minio` -MINIO_LOCATION = us-east-1 -; Minio enabled ssl only available when STORAGE_TYPE is `minio` -MINIO_USE_SSL = false -; Minio skip SSL verification available when STORAGE_TYPE is `minio` -MINIO_INSECURE_SKIP_VERIFY = false -``` - -## Repository Archive Storage (`storage.repo-archive`) - -Configuration for repository archive storage. It will inherit from default `[storage]` or -`[storage.xxx]` when set `STORAGE_TYPE` to `xxx`. The default of `PATH` -is `data/repo-archive` and the default of `MINIO_BASE_PATH` is `repo-archive/`. - -- `STORAGE_TYPE`: **local**: Storage type for repo archive, `local` for local disk or `minio` for s3 compatible object storage service or other name defined with `[storage.xxx]` -- `SERVE_DIRECT`: **false**: Allows the storage driver to redirect to authenticated URLs to serve files directly. Currently, only Minio/S3 is supported via signed URLs, local does nothing. -- `PATH`: **./data/repo-archive**: Where to store archive files, only available when `STORAGE_TYPE` is `local`. -- `MINIO_ENDPOINT`: **localhost:9000**: Minio endpoint to connect only available when `STORAGE_TYPE` is `minio` -- `MINIO_ACCESS_KEY_ID`: Minio accessKeyID to connect only available when `STORAGE_TYPE` is `minio` -- `MINIO_SECRET_ACCESS_KEY`: Minio secretAccessKey to connect only available when `STORAGE_TYPE is` `minio` -- `MINIO_BUCKET`: **gitea**: Minio bucket to store the lfs only available when `STORAGE_TYPE` is `minio` -- `MINIO_LOCATION`: **us-east-1**: Minio location to create bucket only available when `STORAGE_TYPE` is `minio` -- `MINIO_BASE_PATH`: **repo-archive/**: Minio base path on the bucket only available when `STORAGE_TYPE` is `minio` -- `MINIO_USE_SSL`: **false**: Minio enabled ssl only available when `STORAGE_TYPE` is `minio` -- `MINIO_INSECURE_SKIP_VERIFY`: **false**: Minio skip SSL verification available when STORAGE_TYPE is `minio` - -## Repository Archives (`repo-archive`) - -- `STORAGE_TYPE`: **local**: Storage type for actions logs, `local` for local disk or `minio` for s3 compatible object storage service, default is `local` or other name defined with `[storage.xxx]` -- `MINIO_BASE_PATH`: **repo-archive/**: Minio base path on the bucket only available when STORAGE_TYPE is `minio` +The storage configuration for attachments, lfs, avatars, repo-avatars, repo-archive, packages, actions_log, actions_artifact is [explained in detail in a dedicated section](../storage/). ## Proxy (`proxy`) @@ -1359,8 +1239,6 @@ PROXY_HOSTS = *.github.com - `ENABLED`: **false**: Enable/Disable actions - `DEFAULT_ACTIONS_URL`: **https://code.forgejo.org**: Default address to get action plugins, e.g. the default value means downloading from "https://code.forgejo.org/actions/checkout" for "uses: actions/checkout@v3" -- `STORAGE_TYPE`: **local**: Storage type for actions logs, `local` for local disk or `minio` for s3 compatible object storage service, default is `local` or other name defined with `[storage.xxx]` -- `MINIO_BASE_PATH`: **actions_log/**: Minio base path on the bucket only available when STORAGE_TYPE is `minio` ## Other (`other`) diff --git a/admin/index.md b/admin/index.md index 02c9f4fc..935ddfe6 100644 --- a/admin/index.md +++ b/admin/index.md @@ -8,6 +8,7 @@ These documents are targeted to people who run Forgejo on their machines. - [Installation](./installation/) - [Database Preparation](./database-preparation/) - [Configuration Cheat Sheet](./config-cheat-sheet/) +- [Storage settings](./storage/) - [Upgrade guide](./upgrade/) - [Reverse Proxy](./reverse-proxy/) - [Email setup](./email-setup/) diff --git a/admin/storage.md b/admin/storage.md new file mode 100644 index 00000000..d0bdf45e --- /dev/null +++ b/admin/storage.md @@ -0,0 +1,129 @@ +--- +title: 'Storage settings' +license: 'CC-BY-SA-4.0' +--- + +The storage for each subsystem (`attachments`, `lfs`, `avatars`, +`repo-avatars`, `repo-archive`, `packages`, `actions_log`, +`actions_artifact`) is defined in `app.ini`. It can either be on disk +(`local`) or using a MinIO server (`minio`). The default is `local` +storage, using the following hierarchy under the `WORK_PATH` directory: + +| storage | default base path | app.ini sections | +| ----------------- | ------------------ | -------------------------------------------------- | +| attachments | attachments/ | [attachments] or [storage.attachements] | +| lfs | lfs/ | [lfs] or [storage.lfs] | +| avatars | avatars/ | [avatars] or [storage.avatars] | +| repo-avatars | repo-avatars/ | [repo-avatars] or [storage.repo-avatars] | +| repo-archive | repo-archive/ | [repo-archive] or [storage.repo-archive] | +| packages | packages/ | [packages] or [storage.packages] | +| actions_log | actions_log/ | [actions_log] or [storage.actions_log] | +| actions_artifacts | actions_artifacts/ | [actions_artifacts] or [storage.actions_artifacts] | + +For instance if `WORK_PATH` was `/appdata`, the default directory to +store attachments would be `/appdata/attachements`. + +## Overriding the defaults + +These defaults can be modified for all subsystems in the `[storage]` +section. For instance setting: + +``` +[storage] +PATH = /mystorage +``` + +would change the default for storing attachements to +`/mystorage/attachments`. It is also possible to change these settings +for each subsystem in a `[storage.XXXX]` section. For instance setting: + +``` +[storage] +PATH = /mystorage + +[storage.attachments] +PATH = /otherstorage/attachements +``` + +would store attachments in `/otherstorage/attachements` while `lfs` +files would be stored in `/mystorage/lfs`. + +Finally, instead of using `[storage.XXXX]` it is also possible to use +`[XXXX]` as a shorthand. + +## Storage type + +The value of `STORAGE_TYPE` can be `local` (the default) or `minio`. For instance: + +``` +[storage] +STORAGE_TYPE = minio +``` + +Will use `minio` for all subsystems (`attachements`, `lfs`, etc.) +instead of storing them on disk. Each storage type has its own +settings, as explained below. + +## `local` storage + +There is just one setting when the `STORAGE_TYPE` is set to `local`, +`PATH`. For instance: + +``` +[storage] +STORAGE_TYPE = local +PATH = /mystorage +``` + +If the value of PATH for the `XXXX` subsystem is relative, it is +constructed as follows: + +- The default base path is `WORK_PATH` (for instance `/appdata`) +- If `[storage].PATH` is relative (for instance `storage`), the default base path becomes `WORK_PATH`/`[storage].PATH` (for instance `/appdata/storage`) +- If `[storage.XXXX].PATH` is relative, the path becomes `WORK_PATH`/`[storage].PATH`/`[storage.XXXX].PATH` (for instance`/appdata/storage/lfs`) + +It is recommended to always set the `PATH` values to an absolute path +name because it is easier to understand and maintain. + +## `minio` storage + +When the `STORAGE_TYPE` is set to `minio`, the settings available in +all sections (`[storage]`, `[storage.XXXX]` and `[XXXX]`) are: + +- `SERVE_DIRECT`: **false**: Allows the storage driver to redirect to authenticated URLs to serve files directly. Only supported via signed URLs. +- `MINIO_ENDPOINT`: **localhost:9000**: Minio endpoint to connect. +- `MINIO_ACCESS_KEY_ID`: Minio accessKeyID to connect. +- `MINIO_SECRET_ACCESS_KEY`: Minio secretAccessKey to connect. +- `MINIO_BUCKET`: **gitea**: Minio bucket to store the data. +- `MINIO_LOCATION`: **us-east-1**: Minio location to create bucket. +- `MINIO_USE_SSL`: **false**: Minio enabled ssl. +- `MINIO_INSECURE_SKIP_VERIFY`: **false**: Minio skip SSL verification. + +One setting is only available in the `[storage.XXXX]` or `[XXXX]` sections: + +- `MINIO_BASE_PATH`: defaults to the `default base path` of the `XXXX` + subsystem (see the table above) and is a relative path within the + MinIO bucket defined by `MINIO_BUCKET`. + +## Sections precedence + +The sections in which a setting is found have the following priority: + +- [XXXX] is first +- [storage.XXXX] is second +- [storage] is last + +For instance: + +``` +[storage] +PATH = /last + +[storage.attachments] +PATH = /second + +[attachments] +PATH = /first +``` + +Will set the value of `PATH` for attachements to `/first`.