1
0
Fork 0
mirror of https://codeberg.org/forgejo/forgejo.git synced 2024-12-04 10:30:19 -05:00
Commit graph

45 commits

Author SHA1 Message Date
Jason Song
2bc47e81a3
Ensure GetCSRF doesn't return an empty token (#32130)
Since page templates keep changing, some pages that contained forms with
CSRF token no longer have them.

It leads to some calls of `GetCSRF` returning an empty string, which
fails the tests. Like

3269b04d61/tests/integration/attachment_test.go (L62-L63)

The test did try to get the CSRF token and provided it, but it was
empty.

(cherry picked from commit 13283873e9d523d5a5557f55d64f702c1a9f76ec)

Conflicts:
	tests/integration/integration_test.go
  trivial context conflict
2024-10-06 07:29:45 +02:00
Gusted
f78e397dd6
[TESTS] Move CreateDeclarativeRepo to more accessible location
- This allows `CreateDeclarativeRepo` to be used by other testing
packages such as E2EE testing.
- Removes unused function in `services/webhook/sourcehut/builds_test.go`.
2024-08-25 02:54:43 +02:00
Gergely Nagy
a414703c09
tests: Add an IsTemplate option to DeclarativeRepoOptions
This lets us use `CreateDeclarativeRepoWithOptions` to create template
repositories.

Signed-off-by: Gergely Nagy <forgejo@gergo.csillger.hu>
2024-08-02 11:10:34 +02:00
TheFox0x7
4de909747b Add testifylint to lint checks (#4535)
go-require lint is ignored for now

Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/4535
Reviewed-by: Gusted <gusted@noreply.codeberg.org>
Co-authored-by: TheFox0x7 <thefox0x7@gmail.com>
Co-committed-by: TheFox0x7 <thefox0x7@gmail.com>
2024-07-30 19:41:10 +00:00
Gusted
45401e044f
[CHORE] Update jsonschema library to v6
- Update the `github.com/santhosh-tekuri/jsonschema` library from v5 to
v6.
- Update the migration loader function to a type, which is now required
in V6.
- `github.com/santhosh-tekuri/jsonschema/v6` was already used by gof3,
so removing the v5 library and using the v6 library reduces the binary
size of Forgejo.
  - Before: 95912040 bytes
  - After: 95706152 bytes
2024-07-15 17:20:50 +02:00
oliverpool
985939c145 test: pkce only for OpenID Connect 2024-06-10 11:24:32 +02:00
Gergely Nagy
8cc5d5dc78
tests: Support creating a declarative repo without AutoInit
To be able to easily test cases where the repository does not have any
code, where the git repo itself is completely uninitialized, lets
support a case where the `AutoInit` property is false.

For the sake of backwards compatibility, if the option is not set either
way, it will default to `true`.

Signed-off-by: Gergely Nagy <forgejo@gergo.csillger.hu>
2024-05-01 11:06:38 +02:00
Earl Warren
7cabc5670d
Implement remote user login source and promotion to regular user
A remote user (UserTypeRemoteUser) is a placeholder that can be
promoted to a regular user (UserTypeIndividual). It represents users
that exist somewhere else. Although the UserTypeRemoteUser already
exists in Forgejo, it is neither used or documented.

A new login type / source (Remote) is introduced and set to be the login type
of remote users.

Type        UserTypeRemoteUser
LogingType  Remote

The association between a remote user and its counterpart in another
environment (for instance another forge) is via the OAuth2 login
source:

LoginName   set to the unique identifier relative to the login source
LoginSource set to the identifier of the remote source

For instance when migrating from GitLab.com, a user can be created as
if it was authenticated using GitLab.com as an OAuth2 authentication
source.

When a user authenticates to Forejo from the same authentication
source and the identifier match, the remote user is promoted to a
regular user. For instance if 43 is the ID of the GitLab.com OAuth2
login source, 88 is the ID of the Remote loging source, and 48323
is the identifier of the foo user:

Type        UserTypeRemoteUser
LogingType  Remote
LoginName   48323
LoginSource 88
Email       (empty)
Name        foo

Will be promoted to the following when the user foo authenticates to
the Forgejo instance using GitLab.com as an OAuth2 provider. All users
with a LoginType of Remote and a LoginName of 48323 are examined. If
the LoginSource has a provider name that matches the provider name of
GitLab.com (usually just "gitlab"), it is a match and can be promoted.

The email is obtained via the OAuth2 provider and the user set to:

Type        UserTypeIndividual
LogingType  OAuth2
LoginName   48323
LoginSource 43
Email       foo@example.com
Name        foo

Note: the Remote login source is an indirection to the actual login
source, i.e. the provider string my be set to a login source that does
not exist yet.
2024-04-25 13:03:49 +02:00
Gergely Nagy
6d2f645363
tests: Let CreateDeclarativeRepoWithOptions create a Wiki too
Add a new member to `DeclarativeRepoOptions`: `WikiBranch`. If
specified, create a Wiki with the given branch, and a single "Home"
page.

This will be used by an upcoming test.

Signed-off-by: Gergely Nagy <forgejo@gergo.csillger.hu>
2024-04-25 00:36:58 +02:00
Gergely Nagy
e7fcf3f189
tests: Refactor CreateDeclarativeRepo
Lets introduce a new helper function,
`CreateDeclarativeRepoWithOptions`! This is almost the same as the
existing `CreateDeclarativeRepo` helper, but instead of taking a list of
random parameters the author thought of at the time of its introduction,
it takes a `DeclarativeRepoOptions` struct, with optional members.

This makes it easier to extend the function, as new members can be added
without breaking or having to update existing callsites, as long as the
newly added members default to compatible values.

`CreateDeclarativeRepo` is then reimplemented on top of the new
function. Callsites aren't updated yet, we can do that organically,
whenever touching code that uses the older function.

No new functionality is introduced just yet, this is merely a refactor.

Signed-off-by: Gergely Nagy <forgejo@gergo.csillger.hu>
2024-04-24 23:39:47 +02:00
Gergely Nagy
5bf959a3a5
Convert the remaining interface{}s to any
Signed-off-by: Gergely Nagy <forgejo@gergo.csillger.hu>
2024-04-15 20:01:36 +02:00
Earl Warren
fd62033b98
[TESTS] disable test failure on log.Error for now (part 2)
Fixes: https://codeberg.org/forgejo/forgejo/issues/3153
2024-04-11 07:36:59 +02:00
Earl Warren
385bcca788 Merge pull request 'Data size unit localization' (#2528) from 0ko/forgejo:sizelocalize into forgejo
Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/2528
Reviewed-by: oliverpool <oliverpool@noreply.codeberg.org>
Reviewed-by: Gusted <gusted@noreply.codeberg.org>
2024-04-01 19:31:26 +00:00
0ko
bba1884582 Add integration test for TrSize 2024-03-30 21:10:47 +05:00
oliverpool
5785ae72c7 [TESTS] prevent overriding testlogger when calling mainApp 2024-03-27 22:54:13 +01:00
Earl Warren
a117dfd2b1
[ACTIONS] port scheduled actions from Gitea
Refs: https://codeberg.org/forgejo/forgejo/issues/2798
2024-03-26 16:51:56 +01:00
oliverpool
e43c6cd9df fail test when log.Error is called
except for some specific messages (to make the test pass for now)
2024-03-24 07:11:18 +01:00
Lunny Xiao
894d9b2836
Move context from modules to services (#29440)
Since `modules/context` has to depend on `models` and many other
packages, it should be moved from `modules/context` to
`services/context` according to design principles. There is no logic
code change on this PR, only move packages.

- Move `code.gitea.io/gitea/modules/context` to
`code.gitea.io/gitea/services/context`
- Move `code.gitea.io/gitea/modules/contexttest` to
`code.gitea.io/gitea/services/contexttest` because of depending on
context
- Move `code.gitea.io/gitea/modules/upload` to
`code.gitea.io/gitea/services/context/upload` because of depending on
context

(cherry picked from commit 29f149bd9f517225a3c9f1ca3fb0a7b5325af696)

Conflicts:
	routers/api/packages/alpine/alpine.go
	routers/api/v1/repo/issue_reaction.go
	routers/install/install.go
	routers/web/admin/config.go
	routers/web/passkey.go
	routers/web/repo/search.go
	routers/web/repo/setting/default_branch.go
	routers/web/user/home.go
	routers/web/user/profile.go
	tests/integration/editor_test.go
	tests/integration/integration_test.go
	tests/integration/mirror_push_test.go
	trivial context conflicts
	also modified all other occurrences in Forgejo specific files
2024-03-06 12:10:43 +08:00
oliverpool
76c2f5bf92
[GITEA] add test showing bug on resolving invalidated review comment
(cherry picked from commit 5fb3b927f1)
2024-02-05 16:57:56 +01:00
Gergely Nagy
48d752c630
[TESTS] Lift out CreateDeclarativeRepo()
There are a number of tests which require creating a repository on the
fly, and they each do it their own way. To reduce code duplication, lift
out this common pattern into a helper called `CreateDeclarativeRepo()`,
which lets us create a repository, set up enabled and disabled repo
units, and even add, delete, or update files.

Also convert a number of users of this pattern to the new helper - those
users that I introduced, and are in code introduced by Forgejo in the
first place.

Signed-off-by: Gergely Nagy <forgejo@gergo.csillger.hu>
(cherry picked from commit 342b7bae3c)
(cherry picked from commit 2ece8764e9)
2024-02-05 16:54:44 +01:00
Gusted
662c8ee341
[GITEA] Use existing jsonschema library
- Use the 'existing' jsonschema library for the nodeinfo integration test.

(cherry picked from commit 73864840f2)
(cherry picked from commit da36df306b)

Conflicts:
	go.mod
	https://codeberg.org/forgejo/forgejo/pulls/1581
(cherry picked from commit 2b4ab46d8e)

Conflicts:
	go.mod
	https://codeberg.org/forgejo/forgejo/pulls/1617
(cherry picked from commit 8064130344)
(cherry picked from commit 0ccefc633e)
(cherry picked from commit 19e647b531)
(cherry picked from commit 2bcc04889d)
(cherry picked from commit 2fd1932699)
(cherry picked from commit b9a3e1e525)
(cherry picked from commit 92d932d23f)
(cherry picked from commit c125217fea)
(cherry picked from commit f9801ba57b)

Conflicts:
	go.mod
	https://codeberg.org/forgejo/forgejo/pulls/2034
(cherry picked from commit 2558a8a764)
(cherry picked from commit f53b2d3112)
(cherry picked from commit c098055f0a)
(cherry picked from commit 0e1591554a)
(cherry picked from commit 876d9d5c6f)

Conflicts:
	go.mod
	https://codeberg.org/forgejo/forgejo/pulls/2249
(cherry picked from commit 7110bb6a78)
2024-02-05 16:09:40 +01:00
Gusted
23d32eb493
[GITEA] Improve HTML title on repositories
- The `<title>` element that lives inside the `<head>` element is an important element that gives browsers and search engine crawlers the title of the webpage, hence the element name. It's therefor important that this title is accurate.
- Currently there are three issues with titles on repositories. It doesn't use the `FullName` and instead only uses the repository name, this doesn't distinguish which user or organisation the repository is on. It doesn't show the full treepath in the title when visiting an file inside a directory and instead only uses the latest path in treepath. It can show the repository name twice if the `.Title` variable also included the repository name such as on the repository homepage.
- Use the repository's fullname (which include which user the repository is on) instead of just their name.
- Display the repository's fullname if it isn't already in `.Title`.
- Use the full treepath in the repository code view instead of just the
last path.
- Adds integration tests.
- Adds a new repository (`repo59`) that has 3 depths for folders, which
wasn't in any other fixture repository yet, so the full treepath for
could be properly tested.
- Resolves https://codeberg.org/forgejo/forgejo/issues/1276

(cherry picked from commit ff9a6a2cda)
(cherry picked from commit 76dffc8621)
(cherry picked from commit ff0615b9d0)
(cherry picked from commit 8712eaa394)
(cherry picked from commit 0c11587582)
(cherry picked from commit 3cbd9fb792)

Conflicts:
	tests/integration/repo_test.go
	https://codeberg.org/forgejo/forgejo/pulls/1512
(cherry picked from commit fbfdba8ae9)

Conflicts:
	models/fixtures/release.yml
	https://codeberg.org/forgejo/forgejo/pulls/1550
(cherry picked from commit 8b2bf0534c)
(cherry picked from commit d706d9e222)
(cherry picked from commit 6d46261a3f)
(cherry picked from commit f864d18ad3)
(cherry picked from commit 80f8620d0d)

[GITEA] Improve HTML title on repositories (squash) do not double escape

(cherry picked from commit 22882fe25c)
(cherry picked from commit 63e99df3d1)
(cherry picked from commit b65d777bc7)
(cherry picked from commit 2961f4f632)
(cherry picked from commit f7f723628c)
(cherry picked from commit 9ed7915826)
(cherry picked from commit 8b9ead4608)
(cherry picked from commit 50eeaf1fbc)
(cherry picked from commit ee6f32820e)
(cherry picked from commit bf337bed35)
(cherry picked from commit 6be9501ec0)
(cherry picked from commit b39860570d)
(cherry picked from commit 3f30f486d5)
(cherry picked from commit 5680ecdbe9)
(cherry picked from commit da6a19ad16)
(cherry picked from commit 5462493a77)
(cherry picked from commit 530fe57dde)
(cherry picked from commit f174f35644)

Conflicts:
	models/fixtures/repository.yml
	https://codeberg.org/forgejo/forgejo/pulls/2214
(cherry picked from commit 75212b3a59)
(cherry picked from commit 6e3c0be555)
2024-02-05 16:05:50 +01:00
Earl Warren
8cbd00592d
[TESTS] createUser via the user model helper for integration tests (squash)
See https://codeberg.org/forgejo/forgejo/pulls/2295
2024-02-05 15:06:51 +01:00
Gusted
c6a53c3172
[SECURITY] Rework long-term authentication
- This is a 'front-port' of the already existing patch on v1.21 and
v1.20, but applied on top of what Gitea has done to rework the LTA
mechanism. Forgejo will stick with the reworked mechanism by the Forgejo
Security team for the time being. The removal of legacy code (AES-GCM) has been
left out.
- The current architecture is inherently insecure, because you can
construct the 'secret' cookie value with values that are available in
the database. Thus provides zero protection when a database is
dumped/leaked.
- This patch implements a new architecture that's inspired from: [Paragonie Initiative](https://paragonie.com/blog/2015/04/secure-authentication-php-with-long-term-persistence#secure-remember-me-cookies).
- Integration testing is added to ensure the new mechanism works.
- Removes a setting, because it's not used anymore.

(cherry picked from commit e3d6622a63)
(cherry picked from commit fef1a6dac5)
(cherry picked from commit b0c5165145)
(cherry picked from commit 7ad51b9f8d)
(cherry picked from commit 64f053f383)
(cherry picked from commit f5e78e4c20)

Conflicts:
	services/auth/auth_token_test.go
	https://codeberg.org/forgejo/forgejo/pulls/2069
(cherry picked from commit f69fc23d4b)
(cherry picked from commit d955ab3ab0)
(cherry picked from commit 9220088f90)
(cherry picked from commit c73ac63696)
(cherry picked from commit 747a176048)

Conflicts:
	models/user/user.go
	routers/web/user/setting/account.go
	https://codeberg.org/forgejo/forgejo/pulls/2295
2024-02-05 15:06:15 +01:00
Earl Warren
345ceeb98f
[TESTS] coverage for SignInOAuthCallback
(cherry picked from commit f8e1619b99)
(cherry picked from commit 46d8bc9bdf)
(cherry picked from commit e0c7b7055f)
(cherry picked from commit faab747f8e)
(cherry picked from commit 46acb6a9a7)
(cherry picked from commit 22d964e744)
(cherry picked from commit 4c8a6031ac)
(cherry picked from commit 032e8c7a9a)
(cherry picked from commit 7a17a3b0fb)
(cherry picked from commit 8ea71c2a31)
(cherry picked from commit 4b027e2d37)
(cherry picked from commit d787089a5d)
(cherry picked from commit 7b9999357a)
(cherry picked from commit 80eb531c38)
(cherry picked from commit 373b198bfb)
(cherry picked from commit 15781eedf7)
(cherry picked from commit 46bdb17a2f)
(cherry picked from commit 22ec6c11ee)
(cherry picked from commit 3f94b9a111)
(cherry picked from commit a4194c29ff)
(cherry picked from commit aa80ba2ed1)
(cherry picked from commit d349f3e80e)
(cherry picked from commit ccb073f71a)
(cherry picked from commit d8a996a9c1)
(cherry picked from commit af12965737)
(cherry picked from commit 3867b17a48)
(cherry picked from commit 0c48072b2e)
(cherry picked from commit 9c5d675ded)
(cherry picked from commit 665119370f)
(cherry picked from commit 658417e7ad)
2024-02-05 14:44:32 +01:00
Earl Warren
5455605342
[TESTS] oauth2: add integration test helpers
(cherry picked from commit e11dcc60f2)

use backticks to avoid backslash

(cherry picked from commit 34212791ee)
(cherry picked from commit bde9473c69)
(cherry picked from commit d4deb43084)
(cherry picked from commit 08e91649b0)
(cherry picked from commit 2b988e5415)

[TESTS] auth LinkAccount test coverage (squash)

(cherry picked from commit a2b2e3066b)
(cherry picked from commit 841d1b5073)
(cherry picked from commit 35da630ad8)
(cherry picked from commit caf2dc4fa7)
(cherry picked from commit 6eb81e67ba)
(cherry picked from commit d59757239f)
(cherry picked from commit 38a121b688)
(cherry picked from commit 20613874ee)
(cherry picked from commit 6d2705e108)
(cherry picked from commit f177b72814)
(cherry picked from commit 75e1fc4c83)
(cherry picked from commit ba64fa9867)
(cherry picked from commit 0b8ab0893e)
(cherry picked from commit 1419d11435)
(cherry picked from commit 38766847e0)
(cherry picked from commit 6f23426a6a)
(cherry picked from commit 9e0ff9ca54)
(cherry picked from commit 353f3601c3)
(cherry picked from commit 6e4ae401d8)
(cherry picked from commit 1a7afe4153)
(cherry picked from commit f9f3e0cc02)
(cherry picked from commit 22fd0337f3)
(cherry picked from commit ee57e138d1)
(cherry picked from commit 21f9b7e73d)
(cherry picked from commit 17c548c092)
(cherry picked from commit 02d3186517)
(cherry picked from commit f02a040fa2)
(cherry picked from commit 3cf9f82b28)
(cherry picked from commit aa9d06dbac)
(cherry picked from commit 6894213154)
2024-02-05 14:44:32 +01:00
Earl Warren
34a5fe68a7
[TESTS] createUser via the user model helper for integration tests
(cherry picked from commit c1d14c5fff)
(cherry picked from commit e0e8aabc98)
(cherry picked from commit 392a415070)
(cherry picked from commit c7cf1307ca)
(cherry picked from commit 93b13d092b)

[TESTS] createUser via the user model helper for integration tests (squash)

(cherry picked from commit 6ff2383952)
(cherry picked from commit de2a6fe8c3)
(cherry picked from commit 398a6ab072)
(cherry picked from commit 16abc89780)
(cherry picked from commit 312a3ec5d9)
(cherry picked from commit 85c6d8e290)
(cherry picked from commit 79150d30a4)
(cherry picked from commit 436137962d)
(cherry picked from commit e4eb8d471e)
(cherry picked from commit a7257052f4)
(cherry picked from commit d5eba9a6dd)
(cherry picked from commit d89ef2ffa9)
(cherry picked from commit f1d25aa307)
(cherry picked from commit 60c7c07353)
(cherry picked from commit cfeff3afdb)
(cherry picked from commit 608ac6bd68)
(cherry picked from commit c64e530a13)
(cherry picked from commit 513db02971)
(cherry picked from commit 43eaaa5a61)
(cherry picked from commit 678eb49440)
(cherry picked from commit f7458dabda)
(cherry picked from commit 32568e43b4)
(cherry picked from commit 3fcbb2043d)
(cherry picked from commit 95f0b8eefa)
(cherry picked from commit f285b9469d)
(cherry picked from commit 4b8a02ad08)
(cherry picked from commit 70eb98025b)
(cherry picked from commit dc2d448705)
(cherry picked from commit e1402ca319)
(cherry picked from commit 807d587285)
2024-02-05 14:44:32 +01:00
KN4CK3R
838db2f891
Convert to url auth to header auth in tests (#28484)
Related #28390
2023-12-21 23:59:59 +00:00
silverwind
ce83609ff6
Upgrade to golangci-lint@v1.55.0 (#27756)
https://github.com/golangci/golangci-lint/releases/tag/v1.55.0
2023-10-24 02:54:59 +00:00
wxiaoguang
c28e29fd94
Refactor tests (#26464)
1. Give the global variable clear names
2. Use generic parameter for `onGiteaRun`
2023-08-12 16:30:16 +00:00
silverwind
88f835192d
Replace interface{} with any (#25686)
Result of running `perl -p -i -e 's#interface\{\}#any#g' **/*` and `make fmt`.

Basically the same [as golang did](2580d0e08d).
2023-07-04 18:36:08 +00:00
wxiaoguang
4e2f1ee58d
Refactor web package and context package (#25298)
1. The "web" package shouldn't depends on "modules/context" package,
instead, let each "web context" register themselves to the "web"
package.
2. The old Init/Free doesn't make sense, so simplify it
* The ctx in "Init(ctx)" is never used, and shouldn't be used that way
* The "Free" is never called and shouldn't be called because the SSPI
instance is shared

---------

Co-authored-by: Giteabot <teabot@gitea.io>
2023-06-18 09:59:09 +02:00
wxiaoguang
4647660776
Rewrite logger system (#24726)
## ⚠️ Breaking

The `log.<mode>.<logger>` style config has been dropped. If you used it,
please check the new config manual & app.example.ini to make your
instance output logs as expected.

Although many legacy options still work, it's encouraged to upgrade to
the new options.

The SMTP logger is deleted because SMTP is not suitable to collect logs.

If you have manually configured Gitea log options, please confirm the
logger system works as expected after upgrading.

## Description

Close #12082 and maybe more log-related issues, resolve some related
FIXMEs in old code (which seems unfixable before)

Just like rewriting queue #24505 : make code maintainable, clear legacy
bugs, and add the ability to support more writers (eg: JSON, structured
log)

There is a new document (with examples): `logging-config.en-us.md`

This PR is safer than the queue rewriting, because it's just for
logging, it won't break other logic.

## The old problems

The logging system is quite old and difficult to maintain:
* Unclear concepts: Logger, NamedLogger, MultiChannelledLogger,
SubLogger, EventLogger, WriterLogger etc
* Some code is diffuclt to konw whether it is right:
`log.DelNamedLogger("console")` vs `log.DelNamedLogger(log.DEFAULT)` vs
`log.DelLogger("console")`
* The old system heavily depends on ini config system, it's difficult to
create new logger for different purpose, and it's very fragile.
* The "color" trick is difficult to use and read, many colors are
unnecessary, and in the future structured log could help
* It's difficult to add other log formats, eg: JSON format
* The log outputer doesn't have full control of its goroutine, it's
difficult to make outputer have advanced behaviors
* The logs could be lost in some cases: eg: no Fatal error when using
CLI.
* Config options are passed by JSON, which is quite fragile.
* INI package makes the KEY in `[log]` section visible in `[log.sub1]`
and `[log.sub1.subA]`, this behavior is quite fragile and would cause
more unclear problems, and there is no strong requirement to support
`log.<mode>.<logger>` syntax.


## The new design

See `logger.go` for documents.


## Screenshot

<details>


![image](https://github.com/go-gitea/gitea/assets/2114189/4462d713-ba39-41f5-bb08-de912e67e1ff)


![image](https://github.com/go-gitea/gitea/assets/2114189/b188035e-f691-428b-8b2d-ff7b2199b2f9)


![image](https://github.com/go-gitea/gitea/assets/2114189/132e9745-1c3b-4e00-9e0d-15eaea495dee)

</details>

## TODO

* [x] add some new tests
* [x] fix some tests
* [x] test some sub-commands (manually ....)

---------

Co-authored-by: Jason Song <i@wolfogre.com>
Co-authored-by: delvh <dev.lh@web.de>
Co-authored-by: Giteabot <teabot@gitea.io>
2023-05-21 22:35:11 +00:00
wxiaoguang
6f9c278559
Rewrite queue (#24505)
# ⚠️ Breaking

Many deprecated queue config options are removed (actually, they should
have been removed in 1.18/1.19).

If you see the fatal message when starting Gitea: "Please update your
app.ini to remove deprecated config options", please follow the error
messages to remove these options from your app.ini.

Example:

```
2023/05/06 19:39:22 [E] Removed queue option: `[indexer].ISSUE_INDEXER_QUEUE_TYPE`. Use new options in `[queue.issue_indexer]`
2023/05/06 19:39:22 [E] Removed queue option: `[indexer].UPDATE_BUFFER_LEN`. Use new options in `[queue.issue_indexer]`
2023/05/06 19:39:22 [F] Please update your app.ini to remove deprecated config options
```

Many options in `[queue]` are are dropped, including:
`WRAP_IF_NECESSARY`, `MAX_ATTEMPTS`, `TIMEOUT`, `WORKERS`,
`BLOCK_TIMEOUT`, `BOOST_TIMEOUT`, `BOOST_WORKERS`, they can be removed
from app.ini.

# The problem

The old queue package has some legacy problems:

* complexity: I doubt few people could tell how it works.
* maintainability: Too many channels and mutex/cond are mixed together,
too many different structs/interfaces depends each other.
* stability: due to the complexity & maintainability, sometimes there
are strange bugs and difficult to debug, and some code doesn't have test
(indeed some code is difficult to test because a lot of things are mixed
together).
* general applicability: although it is called "queue", its behavior is
not a well-known queue.
* scalability: it doesn't seem easy to make it work with a cluster
without breaking its behaviors.

It came from some very old code to "avoid breaking", however, its
technical debt is too heavy now. It's a good time to introduce a better
"queue" package.

# The new queue package

It keeps using old config and concept as much as possible.

* It only contains two major kinds of concepts:
    * The "base queue": channel, levelqueue, redis
* They have the same abstraction, the same interface, and they are
tested by the same testing code.
* The "WokerPoolQueue", it uses the "base queue" to provide "worker
pool" function, calls the "handler" to process the data in the base
queue.
* The new code doesn't do "PushBack"
* Think about a queue with many workers, the "PushBack" can't guarantee
the order for re-queued unhandled items, so in new code it just does
"normal push"
* The new code doesn't do "pause/resume"
* The "pause/resume" was designed to handle some handler's failure: eg:
document indexer (elasticsearch) is down
* If a queue is paused for long time, either the producers blocks or the
new items are dropped.
* The new code doesn't do such "pause/resume" trick, it's not a common
queue's behavior and it doesn't help much.
* If there are unhandled items, the "push" function just blocks for a
few seconds and then re-queue them and retry.
* The new code doesn't do "worker booster"
* Gitea's queue's handlers are light functions, the cost is only the
go-routine, so it doesn't make sense to "boost" them.
* The new code only use "max worker number" to limit the concurrent
workers.
* The new "Push" never blocks forever
* Instead of creating more and more blocking goroutines, return an error
is more friendly to the server and to the end user.

There are more details in code comments: eg: the "Flush" problem, the
strange "code.index" hanging problem, the "immediate" queue problem.

Almost ready for review.

TODO:

* [x] add some necessary comments during review
* [x] add some more tests if necessary
* [x] update documents and config options
* [x] test max worker / active worker
* [x] re-run the CI tasks to see whether any test is flaky
* [x] improve the `handleOldLengthConfiguration` to provide more
friendly messages
* [x] fine tune default config values (eg: length?)

## Code coverage:

![image](https://user-images.githubusercontent.com/2114189/236620635-55576955-f95d-4810-b12f-879026a3afdf.png)
2023-05-08 19:49:59 +08:00
silverwind
47748df9b3
Enable forbidigo linter (#24278)
Enable [forbidigo](https://github.com/ashanbrown/forbidigo) linter which
forbids print statements. Will check how to integrate this with the
smallest impact possible, so a few `nolint` comments will likely be
required. Plan is to just go through the issues and either:

- Remove the print if it is nonsensical
- Add a `//nolint` directive if it makes sense

I don't plan on investigating the individual issues any further.

<details>
<summary>Initial Lint Results</summary>

```
modules/log/event.go:348:6: use of `fmt.Println` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

					fmt.Println(err)

					^

modules/log/event.go:382:6: use of `fmt.Println` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

					fmt.Println(err)

					^

modules/queue/unique_queue_disk_channel_test.go:20:2: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

	fmt.Printf("TempDir %s\n", tmpDir)

	^

contrib/backport/backport.go:168:2: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

	fmt.Printf("* Backporting %s to %s as %s\n", pr, localReleaseBranch, backportBranch)

	^

contrib/backport/backport.go:216:4: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

			fmt.Printf("* Navigate to %s to open PR\n", url)

			^

contrib/backport/backport.go:223:2: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

	fmt.Printf("* `xdg-open %s`\n", url)

	^

contrib/backport/backport.go:233:2: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

	fmt.Printf("* `git push -u %s %s`\n", remote, backportBranch)

	^

contrib/backport/backport.go:243:2: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

	fmt.Printf("* Amending commit to prepend `Backport #%s` to body\n", pr)

	^

contrib/backport/backport.go:272:3: use of `fmt.Println` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

		fmt.Println("* Attempting git cherry-pick --continue")

		^

contrib/backport/backport.go:281:2: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

	fmt.Printf("* Attempting git cherry-pick %s\n", sha)

	^

contrib/backport/backport.go:297:2: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

	fmt.Printf("* Current branch is %s\n", currentBranch)

	^

contrib/backport/backport.go:299:3: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

		fmt.Printf("* Current branch is %s - not checking out\n", currentBranch)

		^

contrib/backport/backport.go:304:3: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

		fmt.Printf("* Branch %s already exists. Checking it out...\n", backportBranch)

		^

contrib/backport/backport.go:308:2: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

	fmt.Printf("* `git checkout -b %s %s`\n", backportBranch, releaseBranch)

	^

contrib/backport/backport.go:313:2: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

	fmt.Printf("* `git fetch %s main`\n", remote)

	^

contrib/backport/backport.go:316:3: use of `fmt.Println` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

		fmt.Println(string(out))

		^

contrib/backport/backport.go:319:2: use of `fmt.Println` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

	fmt.Println(string(out))

	^

contrib/backport/backport.go:321:2: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

	fmt.Printf("* `git fetch %s %s`\n", remote, releaseBranch)

	^

contrib/backport/backport.go:324:3: use of `fmt.Println` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

		fmt.Println(string(out))

		^

contrib/backport/backport.go:327:2: use of `fmt.Println` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

	fmt.Println(string(out))

	^

models/unittest/fixtures.go:50:3: use of `fmt.Println` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

		fmt.Println("Unsupported RDBMS for integration tests")

		^

models/unittest/fixtures.go:89:3: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

		fmt.Printf("LoadFixtures failed after retries: %v\n", err)

		^

models/unittest/fixtures.go:110:4: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

			fmt.Printf("Failed to generate sequence update: %v\n", err)

			^

models/unittest/fixtures.go:117:6: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

					fmt.Printf("Failed to update sequence: %s Error: %v\n", value, err)

					^

models/migrations/base/tests.go:118:3: use of `fmt.Println` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

		fmt.Println("Environment variable $GITEA_ROOT not set")

		^

models/migrations/base/tests.go:127:3: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

		fmt.Printf("Could not find gitea binary at %s\n", setting.AppPath)

		^

models/migrations/base/tests.go:134:3: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

		fmt.Printf("Environment variable $GITEA_CONF not set - defaulting to %s\n", giteaConf)

		^

models/migrations/base/tests.go:145:3: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

		fmt.Printf("Unable to create temporary data path %v\n", err)

		^

models/migrations/base/tests.go:154:3: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

		fmt.Printf("Unable to InitFull: %v\n", err)

		^

models/migrations/v1_11/v112.go:34:5: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

				fmt.Printf("Error: %v", err)

				^

contrib/fixtures/fixture_generation.go:36:3: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

		fmt.Printf("CreateTestEngine: %+v", err)

		^

contrib/fixtures/fixture_generation.go:40:3: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

		fmt.Printf("PrepareTestDatabase: %+v\n", err)

		^

contrib/fixtures/fixture_generation.go:46:5: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

				fmt.Printf("generate '%s': %+v\n", r, err)

				^

contrib/fixtures/fixture_generation.go:53:5: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

				fmt.Printf("generate '%s': %+v\n", g.name, err)

				^

contrib/fixtures/fixture_generation.go:71:4: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

			fmt.Printf("%s created.\n", path)

			^

services/gitdiff/gitdiff_test.go:543:2: use of `println` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

	println(result)

	^

services/gitdiff/gitdiff_test.go:560:2: use of `println` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

	println(result)

	^

services/gitdiff/gitdiff_test.go:577:2: use of `println` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

	println(result)

	^

modules/web/routing/logger_manager.go:34:2: use of `print` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

	print Printer

	^

modules/doctor/paths.go:109:3: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

		fmt.Printf("Warning: can't remove temporary file: '%s'\n", tmpFile.Name())

		^

tests/test_utils.go:33:2: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

	fmt.Printf(format+"\n", args...)

	^

tests/test_utils.go:61:3: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

		fmt.Printf("Environment variable $GITEA_CONF not set, use default: %s\n", giteaConf)

		^

cmd/actions.go:54:9: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

	_, _ = fmt.Printf("%s\n", respText)

	       ^

cmd/admin_user_change_password.go:74:2: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

	fmt.Printf("%s's password has been successfully updated!\n", user.Name)

	^

cmd/admin_user_create.go:109:3: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

		fmt.Printf("generated random password is '%s'\n", password)

		^

cmd/admin_user_create.go:164:3: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

		fmt.Printf("Access token was successfully created... %s\n", t.Token)

		^

cmd/admin_user_create.go:167:2: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

	fmt.Printf("New user '%s' has been successfully created!\n", username)

	^

cmd/admin_user_generate_access_token.go:74:3: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

		fmt.Printf("%s\n", t.Token)

		^

cmd/admin_user_generate_access_token.go:76:3: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

		fmt.Printf("Access token was successfully created: %s\n", t.Token)

		^

cmd/admin_user_must_change_password.go:56:2: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

	fmt.Printf("Updated %d users setting MustChangePassword to %t\n", n, mustChangePassword)

	^

cmd/convert.go:44:3: use of `fmt.Println` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

		fmt.Println("Converted successfully, please confirm your database's character set is now utf8mb4")

		^

cmd/convert.go:50:3: use of `fmt.Println` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

		fmt.Println("Converted successfully, please confirm your database's all columns character is NVARCHAR now")

		^

cmd/convert.go:52:3: use of `fmt.Println` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

		fmt.Println("This command can only be used with a MySQL or MSSQL database")

		^

cmd/doctor.go:104:3: use of `fmt.Println` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

		fmt.Println(err)

		^

cmd/doctor.go:105:3: use of `fmt.Println` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

		fmt.Println("Check if you are using the right config file. You can use a --config directive to specify one.")

		^

cmd/doctor.go:243:3: use of `fmt.Println` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

		fmt.Println(err)

		^

cmd/embedded.go:154:3: use of `fmt.Println` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

		fmt.Println(a.path)

		^

cmd/embedded.go:198:3: use of `fmt.Println` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

		fmt.Println("Using app.ini at", setting.CustomConf)

		^

cmd/embedded.go:217:2: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

	fmt.Printf("Extracting to %s:\n", destdir)

	^

cmd/embedded.go:253:3: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

		fmt.Printf("%s already exists; skipped.\n", dest)

		^

cmd/embedded.go:275:2: use of `fmt.Println` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

	fmt.Println(dest)

	^

cmd/generate.go:63:2: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

	fmt.Printf("%s", internalToken)

	^

cmd/generate.go:66:3: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

		fmt.Printf("\n")

		^

cmd/generate.go:78:2: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

	fmt.Printf("%s", JWTSecretBase64)

	^

cmd/generate.go:81:3: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

		fmt.Printf("\n")

		^

cmd/generate.go:93:2: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

	fmt.Printf("%s", secretKey)

	^

cmd/generate.go:96:3: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

		fmt.Printf("\n")

		^

cmd/keys.go:74:2: use of `fmt.Println` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

	fmt.Println(strings.TrimSpace(authorizedString))

	^

cmd/mailer.go:32:4: use of `fmt.Print` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

			fmt.Print("warning: Content is empty")

			^

cmd/mailer.go:35:3: use of `fmt.Print` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

		fmt.Print("Proceed with sending email? [Y/n] ")

		^

cmd/mailer.go:40:4: use of `fmt.Println` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

			fmt.Println("The mail was not sent")

			^

cmd/mailer.go:49:9: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

	_, _ = fmt.Printf("Sent %s email(s) to all users\n", respText)

	       ^

cmd/serv.go:147:3: use of `println` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

		println("Gitea: SSH has been disabled")

		^

cmd/serv.go:153:4: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

			fmt.Printf("error showing subcommand help: %v\n", err)

			^

cmd/serv.go:175:4: use of `println` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

			println("Hi there! You've successfully authenticated with the deploy key named " + key.Name + ", but Gitea does not provide shell access.")

			^

cmd/serv.go:177:4: use of `println` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

			println("Hi there! You've successfully authenticated with the principal " + key.Content + ", but Gitea does not provide shell access.")

			^

cmd/serv.go:179:4: use of `println` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

			println("Hi there, " + user.Name + "! You've successfully authenticated with the key named " + key.Name + ", but Gitea does not provide shell access.")

			^

cmd/serv.go:181:3: use of `println` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

		println("If this is unexpected, please log in with password and setup Gitea under another user.")

		^

cmd/serv.go:196:5: use of `fmt.Print` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

				fmt.Print(`{"type":"gitea","version":1}`)

				^

tests/e2e/e2e_test.go:54:3: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

		fmt.Printf("Error initializing test database: %v\n", err)

		^

tests/e2e/e2e_test.go:63:3: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

		fmt.Printf("util.RemoveAll: %v\n", err)

		^

tests/e2e/e2e_test.go:67:3: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

		fmt.Printf("Unable to remove repo indexer: %v\n", err)

		^

tests/e2e/e2e_test.go:109:6: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

					fmt.Printf("%v", stdout.String())

					^

tests/e2e/e2e_test.go:110:6: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

					fmt.Printf("%v", stderr.String())

					^

tests/e2e/e2e_test.go:113:6: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

					fmt.Printf("%v", stdout.String())

					^

tests/integration/integration_test.go:124:3: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

		fmt.Printf("Error initializing test database: %v\n", err)

		^

tests/integration/integration_test.go:135:3: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

		fmt.Printf("util.RemoveAll: %v\n", err)

		^

tests/integration/integration_test.go:139:3: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

		fmt.Printf("Unable to remove repo indexer: %v\n", err)

		^

tests/integration/repo_test.go:357:4: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)

			fmt.Printf("%s", resp.Body)

			^
```

</details>

---------

Co-authored-by: Giteabot <teabot@gitea.io>
2023-04-24 05:50:58 -04:00
wxiaoguang
e422342eeb
Allow adding new files to an empty repo (#24164)
![image](https://user-images.githubusercontent.com/2114189/232561612-2bfcfd0a-fc04-47ba-965f-5d0bcea46c54.png)
2023-04-19 21:40:42 +08:00
wxiaoguang
5b9557aef5
Refactor cookie (#24107)
Close #24062

At the beginning, I just wanted to fix the warning mentioned by #24062

But, the cookie code really doesn't look good to me, so clean up them.

Complete the TODO on `SetCookie`: 

> TODO: Copied from gitea.com/macaron/macaron and should be improved
after macaron removed.
2023-04-13 15:45:33 -04:00
Lunny Xiao
c53ad052d8
Refactor the setting to make unit test easier (#22405)
Some bugs caused by less unit tests in fundamental packages. This PR
refactor `setting` package so that create a unit test will be easier
than before.

- All `LoadFromXXX` files has been splited as two functions, one is
`InitProviderFromXXX` and `LoadCommonSettings`. The first functions will
only include the code to create or new a ini file. The second function
will load common settings.
- It also renames all functions in setting from `newXXXService` to
`loadXXXSetting` or `loadXXXFrom` to make the function name less
confusing.
- Move `XORMLog` to `SQLLog` because it's a better name for that.

Maybe we should finally move these `loadXXXSetting` into the `XXXInit`
function? Any idea?

---------

Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: delvh <dev.lh@web.de>
2023-02-20 00:12:01 +08:00
Chongyi Zheng
de484e86bc
Support scoped access tokens (#20908)
This PR adds the support for scopes of access tokens, mimicking the
design of GitHub OAuth scopes.

The changes of the core logic are in `models/auth` that `AccessToken`
struct will have a `Scope` field. The normalized (no duplication of
scope), comma-separated scope string will be stored in `access_token`
table in the database.
In `services/auth`, the scope will be stored in context, which will be
used by `reqToken` middleware in API calls. Only OAuth2 tokens will have
granular token scopes, while others like BasicAuth will default to scope
`all`.
A large amount of work happens in `routers/api/v1/api.go` and the
corresponding `tests/integration` tests, that is adding necessary scopes
to each of the API calls as they fit.


- [x] Add `Scope` field to `AccessToken`
- [x] Add access control to all API endpoints
- [x] Update frontend & backend for when creating tokens
- [x] Add a database migration for `scope` column (enable 'all' access
to past tokens)

I'm aiming to complete it before Gitea 1.19 release.

Fixes #4300
2023-01-17 15:46:03 -06:00
Lunny Xiao
8c1bb77437
Remove test session cache to reduce possible concurrent problem (#22199) 2022-12-22 21:09:35 +08:00
zeripath
fe6608f72b
Attempt to fix TestExportUserGPGKeys (#22159)
There are repeated failures with this test which appear related to
failures in getTokenForLoggedInUser. It is difficult to further evaluate
the cause of these failures as we do not get given further information.

This PR will attempt to fix this.

First it adds some extra logging and it uses the csrf cookie primarily
for the csrf value.

If the problem does not occur again with those changes we could merge,
assume that it is fixed and hope that if it occurs in future the
additional logging will be helpful.

If not I will add more changes in attempt to fix.

Fix #22105

Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Co-authored-by: John Olheiser <john.olheiser@gmail.com>
Co-authored-by: techknowlogick <matti@mdranta.net>
Co-authored-by: delvh <dev.lh@web.de>
2022-12-21 09:46:16 +08:00
Meisam
f3370eeaee
verify nodeinfo response by schema (#22137)
... using
[github.com/xeipuuv/gojsonschema](https://github.com/xeipuuv/gojsonschema)

Co-authored-by: techknowlogick <techknowlogick@gitea.io>
2022-12-17 01:22:34 -05:00
flynnnnnnnnnn
e81ccc406b
Implement FSFE REUSE for golang files (#21840)
Change all license headers to comply with REUSE specification.

Fix #16132

Co-authored-by: flynnnnnnnnnn <flynnnnnnnnnn@github>
Co-authored-by: John Olheiser <john.olheiser@gmail.com>
2022-11-27 18:20:29 +00:00
Vladimir Yakovlev
ffa4f4b570
Check for valid user token in integration tests (#21520)
Added checks for logged user token.

Some builds fail at unrelated tests, due to missing token.

Example:
https://drone.gitea.io/go-gitea/gitea/62011/2/14

Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
2022-10-20 19:20:01 +01:00
Kyle D
c8ded77680
Kd/ci playwright go test (#20123)
* Add initial playwright config

* Simplify Makefile

* Simplify Makefile

* Use correct config files

* Update playwright settings

* Fix package-lock file

* Don't use test logger for e2e tests

* fix frontend lint

* Allow passing TEST_LOGGER variable

* Init postgres database

* use standard gitea env variables

* Update playwright

* update drone

* Move empty env var to commands

* Cleanup

* Move integrations to subfolder

* tests integrations to tests integraton

* Run e2e tests with go test

* Fix linting

* install CI deps

* Add files to ESlint

* Fix drone typo

* Don't log to console in CI

* Use go test http server

* Add build step before tests

* Move shared init function to common package

* fix drone

* Clean up tests

* Fix linting

* Better mocking for page + version string

* Cleanup test generation

* Remove dependency on gitea binary

* Fix linting

* add initial support for running specific tests

* Add ACCEPT_VISUAL variable

* don't require git-lfs

* Add initial documentation

* Review feedback

* Add logged in session test

* Attempt fixing drone race

* Cleanup and bump version

* Bump deps

* Review feedback

* simplify installation

* Fix ci

* Update install docs
2022-09-02 15:18:23 -04:00
Renamed from integrations/integration_test.go (Browse further)