For #4082.
~~Per the discussion in the issue, the current plan will likely involve duplicating the redis library calling code once for each cacher, as neither garnet nor redict guarantee continued compatibility with redis.~~
See discussion below for details.
## Tasklist
- [x] Write workflow to run cache-specific unit test(s) only (cache, session, queue, nosql) for each cacher
- [x] Check whether garnet and redict pass unit tests with no code modification (gauge required work)
- both passed, but that is because there were very few tests that test the remote cache store
### Out of scope for this PR
- Improve test coverage
- `modules/cache` against a server
- `modules/session` against a server (also needs tests in general)
- _(?) Duplicate implementation for each cacher_
- _Restructure redis usage in `modules/cache` and `modules/settings/cache`_
- _Restructure `modules/session` and its settings_
- _Restructure `modules/queue` and its settings_
- _Restructure `modules/nosql` and its settings_
Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/4138
Reviewed-by: Earl Warren <earl-warren@noreply.codeberg.org>
Co-authored-by: Elias Elwyn <a@jthv.ai>
Co-committed-by: Elias Elwyn <a@jthv.ai>
Uses `gopls check <files>` as a linter. Tested locally and brings up 149
errors currently for me. I don't think I want to fix them in this PR,
but I would like at least to get this analysis running on CI.
List of errors:
```
modules/indexer/code/indexer.go:181:11: impossible condition: nil != nil
routers/private/hook_post_receive.go:120:15: tautological condition: nil == nil
services/auth/source/oauth2/providers.go:185:9: tautological condition: nil == nil
services/convert/issue.go:216:11: tautological condition: non-nil != nil
tests/integration/git_test.go:332:9: impossible condition: nil != nil
services/migrations/migrate.go:179:24-43: unused parameter: ctx
services/repository/transfer.go:288:48-69: unused parameter: doer
tests/integration/api_repo_tags_test.go:75:41-61: unused parameter: session
tests/integration/git_test.go:696:64-74: unused parameter: baseBranch
tests/integration/gpg_git_test.go:265:27-39: unused parameter: t
tests/integration/gpg_git_test.go:284:23-29: unused parameter: tmpDir
tests/integration/gpg_git_test.go:284:31-35: unused parameter: name
tests/integration/gpg_git_test.go:284:37-42: unused parameter: email
```
(cherry picked from commit 816222243af523316041692622be6f48ef068693)
Conflicts:
Makefile
trivial context conflict and also ask renovate to watch over it
do not include it in lint-backend because the errors are not fixed
We're stuck on an old version of golang deadcode. Renovate is confused by it's rename in https://codeberg.org/forgejo/forgejo/pulls/4043.
I don't remember how to use this tool to test it, let's see how the CI reacts to this PR.
Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/4048
Reviewed-by: Victoria <efertone@noreply.codeberg.org>
Reviewed-by: Earl Warren <earl-warren@noreply.codeberg.org>
Co-authored-by: 0ko <0ko@noreply.codeberg.org>
Co-committed-by: 0ko <0ko@noreply.codeberg.org>
Summary:
- Move existing test under a `testify` Suite as `baseRedisWithServerTestSuite`
- Those tests require real redis server.
- Add `go.uber.org/mock/mockgen@latest` as dependency
- as a tool (Makefile).
- in the `go.mod` file.
- Mock redis client lives under a `mock` directory under the queue module.
- That mock module has an extra hand-written mock in-memory redis-like struct.
- Add tests using the mock redis client.
- Changed the logic around queue provider creation.
- Now the `getNewQueue` returns a Queue provider directly, not an init
function to create it.
The whole Queue module is close to impossible to test properly because
everything is private, everything goes through a struct route. Because
of that, we can't test for example what keys are used for given queue.
To overcome this, as a first step I removed one step from that hard
route by allowing custom calls to create new queue provider. To achieve
this, I moved the creation logic into the `getNewQueue` (previously it
was `getNewQueueFn`). That changes nothing on that side, everything goes
as before, except the `newXXX` call happens directly in that function
and not outside that.
That made it possible to add extra provider specific parameters to those
function (`newXXX`). For example a client on redis. Calling it through
the `getNewQueue` function, it gets `nil`.
- If the provided client is not `nil`, it will use that instead of the
connection string.
- If it's `nil` (default behaviour), it creates a new redis client as it
did before, no changes to that.
The rest of the provider code is unchanged. All these changes were
required to make it possible to generate mock clients for providers and
use them.
For the tests, the existing two test cases are good with redis server,
and they need some extra helpers, for example to start a new redis
server if required, or waiting on a redis server to be ready to use.
These helpers are only required for test cases using real redis server.
For better isolation, moved existing test under a testify Suite, and
moved them into a new test file called `base_redis_with_server_test.go`
because, well they test the code with server. These tests do exactly the
same as before, calling the same sub-tests the same way as before, the
only change is the structure of the test (remove repetition, scope
server related helper functions).
Finally, we can create unit tests without redis server. The main focus of
this group of tests are higher level overview of operations. With the
mock redis client we can set up expectations about used queue names,
received values, return value to simulate faulty state.
These new unit test functions don't test all functionality, at least
it's not aimed for it now. It's more about the possibility of doing that
and add extra tests around parts we couldn't test before, for example
key.
What extra features can test the new unit test group:
- What is the received key for given queue? For example using `prefix`,
or if all the `SXxx` calls are expected to use `queue_unique` if
it's a unique queue.
- If it's not a unique queue, no `SXxx` functions are called, because
those sets are used only to check if a value is unique or not.
- `HasItem` return `false` always if it's a non-unique queue.
- All functions are called exactly `N` times, and we don't have any
unexpected calls to redis from the code.
Signed-off-by: Victoria Nadasdi <victoria@efertone.me>
More about codespell: https://github.com/codespell-project/codespell .
I personally introduced it to dozens if not hundreds of projects already and so far only positive feedback.
```
❯ grep lint-spell Makefile
@echo " - lint-spell lint spelling"
@echo " - lint-spell-fix lint spelling and fix issues"
lint: lint-frontend lint-backend lint-spell
lint-fix: lint-frontend-fix lint-backend-fix lint-spell-fix
.PHONY: lint-spell
lint-spell: lint-codespell
.PHONY: lint-spell-fix
lint-spell-fix: lint-codespell-fix
❯ git grep lint- -- .forgejo/
.forgejo/workflows/testing.yml: - run: make --always-make -j$(nproc) lint-backend checks-backend # ensure the "go-licenses" make target runs
.forgejo/workflows/testing.yml: - run: make lint-frontend
```
so how would you like me to invoke `lint-codespell` on CI? (without that would be IMHO very suboptimal and let typos sneak in)
Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/3270
Reviewed-by: Earl Warren <earl-warren@noreply.codeberg.org>
Co-authored-by: Yaroslav Halchenko <debian@onerussian.com>
Co-committed-by: Yaroslav Halchenko <debian@onerussian.com>
there are no tests but since Gitea uses @v1 since last month and Gitea
maintainers rely on make watch, it is safe to assume that upgrading is
not broken. Switching to v1 would require less scrutiny on the
upgrades. Even if there is breakage, it can be fixed with minimal
impact on the developer workflow.
When you cross-compile Gitea and you specify one of the envrionment
variables related to C flags, cgo will fail to build the generator
programs (e.g. generate-bindata) because GOOS and GOARCH are unset, but
those additional flags variables are not unset together with those.
To solve this issue, the simplest way that I've found is to disable cgo
in the `go generate` command as it's not really used there.
For example, I've had this problem with cross-compiling Gitea on FreeBSD
x86_64 to ARMv7 where it's necessary to pass `--target` to `clang` via
`CGO_CFLAGS`:
```
GOOS=freebsd \
GOARCH=arm \
GGOARM=7 \
CGO_ENABLED=1 \
SYSROOT=/usr/local/freebsd-sysroot/armv7 \
CC=clang \
CGO_CFLAGS="--target=armv7-unknown-freebsd13.2-gnueabihf" \
TAGS="bindata sqlite sqlite_unlock_notify" \
make SHELL='sh -x' build
```
```
Running go generate...
# runtime/cgo
In file included from gcc_freebsd_amd64.c:9:
In file included from /usr/include/signal.h:42:
/usr/include/sys/_ucontext.h:44:2: error: unknown type name 'mcontext_t'
modules/migration/schemas_bindata.go:8: running "go": exit status 1
# runtime/cgo
In file included from gcc_freebsd_amd64.c:9:
In file included from /usr/include/signal.h:42:
/usr/include/sys/_ucontext.h:44:2: error: unknown type name 'mcontext_t'
modules/options/options_bindata.go:8: running "go": exit status 1
# runtime/cgo
In file included from gcc_freebsd_amd64.c:9:
In file included from /usr/include/signal.h:42:
/usr/include/sys/_ucontext.h:44:2: error: unknown type name 'mcontext_t'
modules/public/public_bindata.go:8: running "go": exit status 1
# runtime/cgo
In file included from gcc_freebsd_amd64.c:9:
In file included from /usr/include/signal.h:42:
/usr/include/sys/_ucontext.h:44:2: error: unknown type name 'mcontext_t'
modules/templates/templates_bindata.go:8: running "go": exit status 1
gmake[1]: *** [Makefile:781: generate-go] Error 1
*** Error code 2
Stop.
```
But with this fix Gitea compiles successfully.
(cherry picked from commit d11133b83652238023b52576e0d3e57a4f4b21c9)
1. Set
[`BROWSERSLIST_IGNORE_OLD_DATA`](c6ddf7b387/node.js (L400))
to avoid warning on outdated browserslist data which the end user can
likely not do anything about and which is currently visible in the v1.21
branch.
2. Suppress all command echoing and add a "Running webpack..." message
in place.
Warning in question was this:
```
Browserslist: caniuse-lite is outdated. Please run:
npx update-browserslist-db@latest
Why you should do it regularly: https://github.com/browserslist/update-db#readme
```
(cherry picked from commit dcc3c17e5c41ad446b71215b095617e066a2e8e1)
Files in root were not linted, add them. No new violations.
(cherry picked from commit 354705450a410329d253023d2c66ef6d68ecc046)
Conflicts:
- CHANGELOG.md
Gitea specific, removed.
- Makefile
Adjusted SPELLCHECK_FILES: we don't need to filter the
CHANGELOG.md out. The conflict itself was resolved by manually
applying the change.
Fixes: https://github.com/go-gitea/gitea/issues/30512
I think this does mean those tools would run on a potential `vendor`
directory, but I'm not sure we really support vendoring of dependencies
anymore.
`release` has a `vendor` prerequisite so likely the source tarballs
contain vendor files?
(cherry picked from commit 8e12ef911a1d10dedb03e3127c42ca76f9850aca)
Conflicts:
- Makefile
Manually adjusted the changes.
Since https://github.com/go-gitea/gitea/pull/25686, a few `interface{}`
have sneaked into the codebase. Add this replacement to `make fmt` to
prevent this from happening again.
Ideally a linter would do this, but I haven't found any suitable.
(cherry picked from commit c77e8140bc2ac6521dbebfb77613dce2648bfcb8)
Conflicts:
- .gitattributes
Trivial conflict resolved by picking our choice of language
for `*.tmpl` files.
Get rid of one more jQuery dependant and have a nicer color picker as
well.
Now there is only a single global color picker init because that is all
that's necessary because the elements are present on the page when the
init code runs. The init is slightly weird because the module only takes
a selector instead of DOM elements directly.
The label modals now also perform form validation because previously it
was possible to trigger a 500 error `Color cannot be empty.` by clearing
out the color value on labels.
<img width="867" alt="Screenshot 2024-03-25 at 00 21 05"
src="https://github.com/go-gitea/gitea/assets/115237/71215c39-abb1-4881-b5c1-9954b4a89adb">
<img width="860" alt="Screenshot 2024-03-25 at 00 20 48"
src="https://github.com/go-gitea/gitea/assets/115237/a12cb68f-c38b-4433-ba05-53bbb4b1023e">
(cherry picked from commit dd8dde2be89921b2b1497c6cc5eafdde213429cb)
Fixes https://github.com/go-gitea/gitea/issues/30082.
Adds a new linter that searches for non-existant SVG images in
templates. Output before the fix was:
```
$ make lint-templates
SVG "octicon-warning" not found, used in templates/devtest/flex-list.tmpl
SVG "octicon-warning" not found, used in templates/devtest/flex-list.tmpl
SVG "octicon-markup" not found, used in templates/repo/diff/comment_form.tmpl
make: *** [Makefile:438: lint-templates] Error 1
```
<img width="306" alt="Screenshot 2024-03-25 at 23 31 05"
src="https://github.com/go-gitea/gitea/assets/115237/1052d1a9-bfec-4d5a-9cae-f895f78f7c93">
(cherry picked from commit 2ab5f05f40d93224f73e211e84de50a88a6ecf03)
Conflicts:
.github/workflows/files-changed.yml
.github/workflows/pull-compliance.yml
do not exist in Forgejo and the Forgejo workflows
already contain the changes
Instead of failing or do things that could be damaging, this script
prints a deprecation notice. It fixes the unnecessary breaking
change introduced in `[I18n] tooling and process`.
https://github.com/NixOS/nixpkgs/blob/master/pkgs/applications/version-management/forgejo/default.nix#L80
$ make merge-locales
NOT NEEDED: THIS IS A NOOP AS OF Forgejo 7.0 BUT KEPT FOR BACKWARD COMPATIBILITY
$ go run build/merge-forgejo-locales.go
NOT NEEDED: THIS IS A NOOP AS OF Forgejo 7.0 BUT KEPT FOR BACKWARD COMPATIBILITY
Also remove the build/crowdin-to-weblate.sh script that was never
needed.
Existing Forgejo packages may rely on setting GITEA_VERSION to specify
the version to build if:
* they do not build from the git repository with the proper tag
* they build from a source tarbal that does not have a VERSION file
With 7.0 the logic of setting the version was modified in the
`[RELEASE] Gitea version is for interoperability only` commit and
ignores this variable which creates an unecessary breaking change.
If GITEA_VERSION is set, the versions will be set on 7.0 exactly as
they would have with version before and included 1.21.
* If GITEA_VERSION is not set, all versions are the same
* If GITEA_VERSION is set, there is a distinction between the version
set in the binary are returned by the Gitea API and the
version returned by the Forgejo API which includes metadata.
Before:
$ make GITEA_VERSION=7.0.0 show-version-full
7.0.0-dev-1809-cd6fa771ab+gitea-1.22.0
$ make GITEA_VERSION=7.0.0 show-version-api
7.0.0-dev-1809-cd6fa771ab+gitea-1.22.0
After:
$ make GITEA_VERSION=7.0.0 show-version-full
7.0.0
$ make GITEA_VERSION=7.0.0 show-version-api
7.0.0+gitea-1.22.0
We don't need these modifications anymore since all tool
dependencies run via `go run`.
(cherry picked from commit ef33dcf946cc9754b51c955975d67f871702b958)
- Move some scripts from `build` to new `tools` dir. Eventually i would
like to move all but let's do it step-by-step.
- Add dir to eslint and move the files into vars.
- Update docs accordingly.
- While updating docs I noticed we were incorrectly having `public/img`
path still in a few places. Replace those with the current
`public/assets/img`.
---------
Co-authored-by: Nanguan Lin <nanguanlin6@gmail.com>
Co-authored-by: Giteabot <teabot@gitea.io>
(cherry picked from commit 095fdd691dd1a7d7748372cc73e7708278c80933)
1. Make fomantic build use [our
browserslist](e3524c63d6/package.json (L99)).
I found no other way than to sed-replace into it's js, the normal
browserlist config files do not work. The effect of this change is the
removal of some uneeded CSS vendor prefixes.
2. Regenerate `web_src/fomantic/package-lock.json`, this might shut up
some security scanners.
---------
Co-authored-by: Giteabot <teabot@gitea.io>
(cherry picked from commit da3b7f5039158faae4b617ca878061f8a4f3e489)
Now that Forgejo has its own release number, use the Makefile as a
reference.
Also document and improve support for debugging this
pull_request_target workflow by using a branch in the repository.
Gitea versions are
* A.B.C
* A.B.C+rc-0
* A.B.C+dev-5-g4fb9056
If Forgejo versions are:
* X.Y.Z+A.B.C
* X.Y.Z-rc0+A.B.C
* X.Y.Z-dev-1232-g4fb905687+A.B.C
It is non trivial for a client trying to sort out if the server
responding to `/api/v1/version` is Forgejo or Gitea. The Forgejo
version changes to be:
* X.Y.Z+gitea-A.B.C
* X.Y.Z-rc0+gitea-A.B.C
* X.Y.Z-dev-1232-g4fb905687+gitea-A.B.C
and a client can now:
* Split the version with +
* If the second part (the metadata) exists and contains "gitea", the
answer server is Forgejo
* Otherwise it is Gitea
1. `playwright/test` is already installed as part of `deps-frontend` on
CI which runs before, so it's better to not install it again (on a
potentially different version), and just use the version from
package.json and add the `deps-frontend` dependency.
2. `PLAYWRIGHT_DIR` is a undefined variable, so I removed it
```bash
$ git show c8ded77680 | grep PLAYWRIGHT_DIR
+playwright: $(PLAYWRIGHT_DIR)
```
(cherry picked from commit c236e64aca42b9ab0743431bc505033a0cb78b93)
The release number displayed by the API and the CLI becomes:
7.0.0+1.22.0
instead of
1.22.0
It would otherwise be inconsistent to have different version number depending on the interface. With the current implementation `/api/forgejo/v1/version` would return `7.0.0+1.22.0` while `/api/v1/version` would return `1.22.0`. The release would be announced as `7.0.0+1.22.0` but the web API would display `1.22.0`.
It may cause some tools that are Gitea specific to not behave as they should in the future if they expect a Gitea version number and activate some features depending on what it is. They would need to be patched to strip the leading Forgejo version number before asserting the Gitea version.
Refs: https://codeberg.org/forgejo/forgejo/issues/2425
Also ignore the *-test tags when figuring out the Forgejo version,
they exist in the integration repository and experimental repository
for daily releases.
This adds the ability to run `make test-e2e-debugserver` to start a
forgejo server pupulated with the test data from `models/fixtures`. This
is particularly useful for debugging the e2e tests with a external tool,
such as the Playwright extension for VSCode [1].
[1] https://open-vsx.org/extension/ms-playwright/playwright
Follow #29165.
* Introduce JSONTemplate to help to render JSON templates
* Introduce JSEscapeSafe for templates. Now only use `{{ ... |
JSEscape}}` instead of `{{ ... | JSEscape | Safe}}`
* Simplify "UserLocationMapURL" useage
(cherry picked from commit 31bb9f3247388b993c61a10190cfd512408ce57e)
Ref:
https://www.gnu.org/software/make/manual/html_node/Parallel-Disable.html
> If the .NOTPARALLEL special target with no prerequisites is specified
anywhere then the entire instance of make will be run serially,
regardless of the parallel setting
(cherry picked from commit 69ed1a4afbc9604cabe83041de31752dd5d101ee)
Conflicts:
README.md
The release name, as provided by FORGEJO_RELEASE, is used to build OCI
images and binary files. Although it can be the same as the Forgejo
version, it is not a requirement.
When the FORGEJO_RELEASE environment variable is set, use it as a
default for naming the binary file instead of FORGEJO_VERSION. For
instance, when building from the forgejo branch here is what is desired:
FORGEJO_VERSION=7.0.0-g2343
GITEA_VERSION=1.22.0
VERSION=vforgejo-test
The name of the release is also displayed with forgejo --version
for sanity check purposes.
Before:
FORGEJO_VERSION is the computed version
GITEA_VERSION is set manually
VERSION defaults to FORGEJO_VERSION
forgejo --help does not display VERSION
After:
FORGEJO_VERSION is the computed version
GITEA_VERSION is set manually
RELEASE_VERSION defaults to FORGEJO_VERSION
VERSION defaults to RELEASE_VERSION
forgejo --help displays VERSION
Forgejo now has its own lifecycle and its version is derived from the
tag. The Gitea tags are no longer found in the Forgejo codebase and
can no longer be used for that purpose.
When a Forgejo release is published, for interoperability with the
existing tools in the ecosystem, it advertises the supported Gitea
version via /api/v1/version. It is set in the Makefile manually and
cannot be automatically set.
https://codeberg.org/forgejo-contrib/delightful-forgejo#packaging
Existing Forgejo packages rely on the Makefile to build and the change
must be done in a way that tries to not break their assumptions. From
the point of view of a Forgejo package build, the following will
happen on the next major release:
- The package version will bump from v1.21.x to v7.0.0
- /api/v1/version will bump from v1.21.x to v1.22.0
- /api/forgejo/v1/version will bump from v6.x to v7.0.0
The Makefile uses the following variables:
GITEA_VERSION is returned by /api/v1/version
FORGEJO_VERSION is returned by /api/forgejo/v1/version
VERSION is used in the name the binary file and the source archive
Before:
GITEA_VERSION is the computed version
FORGEJO_VERSION is set manually
VERSION defaults to GITEA_VERSION
After:
FORGEJO_VERSION is the computed version
GITEA_VERSION is set manually
VERSION defaults to FORGEJO_VERSION
When the version is computed, it comes from:
- The content of the VERSION file if it exists. It is inserted in
the source archive because it does not contain a git repository
- Otherwise the output of `git describe`
- Use maintained fork https://github.com/golangci/misspell
- Rename `mispell-check` to `lint-spell`, add `lint-spell-fix`
- Run `lint-spell` in separate actions step
- Lint more files, fix discovered issues
- Remove inaccurate and outdated info in docs (we do not need GOPATH for
tools anymore)
Maybe later we can add more spellchecking tools, but I have not found
any good ones yet.
(cherry picked from commit 9c39f8515fa88d644736c6773d7a05d070a02e82)
Conflicts:
.github/workflows/pull-compliance.yml
Makefile
The gitea target is kept as is, for the sake of compatibility with
build systems that rely on it. The forgejo target creates a hard link
with forgejo.
(cherry picked from commit ce156c7cb4)
(cherry picked from commit b2cb232825)
(cherry picked from commit 85543c22bd)
(cherry picked from commit 302955a20d)
(cherry picked from commit 4b1be0858e)
(cherry picked from commit afed1a6e20)
(cherry picked from commit e81304bcd1)
(cherry picked from commit 479cd51a03)
(cherry picked from commit 8baab7c0c1)
(cherry picked from commit 5d25ad9f09)
(cherry picked from commit f36942789f)
(cherry picked from commit 9abfbae8a0)
(cherry picked from commit 079e0efb01)
(cherry picked from commit 0f1b83005a)
(cherry picked from commit 3cc227aaff)
(cherry picked from commit 245310c682)
(cherry picked from commit 3f83e30c93)
(cherry picked from commit e2bdd6af3d)
(cherry picked from commit fb3ac3a20b)
(cherry picked from commit 0ffbd46032)
(cherry picked from commit 1ba9e954e2)
(cherry picked from commit d4bc10d1e7)
(cherry picked from commit e54dc7293e)
(cherry picked from commit f50e5132d4)
(cherry picked from commit 335abddaa0)
(cherry picked from commit 61e80ffb1f)
(cherry picked from commit 4832167235)
(cherry picked from commit 1bbe7816b2)
- Add the experimental
[deacode](https://pkg.go.dev/golang.org/x/tools/internal/cmd/deadcode)
linter to Forgejo.
- To deal with false positives that can happen due to build tags or with code
that's currently only referenced by test code, the output of the tool is
compared against a known-good output.
- This commit doesn't make any attempt to remove any deadcode.
(cherry picked from commit ac462279e9)
(cherry picked from commit b5ea6e85ac)
(cherry picked from commit 5915f3643c)
[CLEANUP] Remove deadcode
- This is deadcode since https://codeberg.org/forgejo/forgejo/pulls/1802
removed the usage of it.
(cherry picked from commit d840b9923e)
(cherry picked from commit 9442bab626)
(cherry picked from commit 0de9d18863)
(cherry picked from commit 26abf78374)
(cherry picked from commit 05d3a143c3)
(cherry picked from commit 4b3d38d5e1)
(cherry picked from commit a726e71986)
(cherry picked from commit cb62ae5b98)
(cherry picked from commit 8195ba06d5)
(cherry picked from commit 4570fb591a)
(cherry picked from commit 1f4d33de2b)
