The previous commit laid out the foundation of the quota engine, this
one builds on top of it, and implements the actual enforcement.
Enforcement happens at the route decoration level, whenever possible. In
case of the API, when over quota, a 413 error is returned, with an
appropriate JSON payload. In case of web routes, a 413 HTML page is
rendered with similar information.
This implementation is for a **soft quota**: quota usage is checked
before an operation is to be performed, and the operation is *only*
denied if the user is already over quota. This makes it possible to go
over quota, but has the significant advantage of being practically
implementable within the current Forgejo architecture.
The goal of enforcement is to deny actions that can make the user go
over quota, and allow the rest. As such, deleting things should - in
almost all cases - be possible. A prime exemption is deleting files via
the web ui: that creates a new commit, which in turn increases repo
size, thus, is denied if the user is over quota.
Limitations
-----------
Because we generally work at a route decorator level, and rarely
look *into* the operation itself, `size:repos:public` and
`size:repos:private` are not enforced at this level, the engine enforces
against `size:repos:all`. This will be improved in the future.
AGit does not play very well with this system, because AGit PRs count
toward the repo they're opened against, while in the GitHub-style fork +
pull model, it counts against the fork. This too, can be improved in the
future.
There's very little done on the UI side to guard against going over
quota. What this patch implements, is enforcement, not prevention. The
UI will still let you *try* operations that *will* result in a denial.
Signed-off-by: Gergely Nagy <forgejo@gergo.csillger.hu>
If a repository has
git config --add push.pushOption submit=".sourcehut/*.yml"
it failed when pushed because of the unknown submit push
option. It will be ignored instead.
Filtering out the push options is done in an earlier stage, when the
hook command runs, before it submits the options map to the private
endpoint.
* move all the push options logic to modules/git/pushoptions
* add 100% test coverage for modules/git/pushoptions
Test coverage for the code paths from which code was moved to the
modules/git/pushoptions package:
* cmd/hook.go:runHookPreReceive
* routers/private/hook_pre_receive.go:validatePushOptions
tests/integration/git_push_test.go:TestOptionsGitPush runs through
both. The test verifying the option is rejected was removed and, if
added again, will fail because the option is now ignored instead of
being rejected.
* cmd/hook.go:runHookProcReceive
* services/agit/agit.go:ProcReceive
tests/integration/git_test.go: doCreateAgitFlowPull runs through
both. It uses variations of AGit related push options.
* cmd/hook.go:runHookPostReceive
* routers/private/hook_post_receive.go:HookPostReceive
tests/integration/git_test.go:doPushCreate called by TestGit/HTTP/sha1/PushCreate
runs through both.
Note that although it provides coverage for this code path it does not use push options.
Fixes: https://codeberg.org/forgejo/forgejo/issues/3651
Enable [unparam](https://github.com/mvdan/unparam) linter.
Often I could not tell the intention why param is unused, so I put
`//nolint` for those cases like webhook request creation functions never
using `ctx`.
---------
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Co-authored-by: delvh <dev.lh@web.de>
(cherry picked from commit fc2d75f86d77b022ece848acf2581c14ef21d43b)
Conflicts:
modules/setting/config_env.go
modules/storage/azureblob.go
services/webhook/dingtalk.go
services/webhook/discord.go
services/webhook/feishu.go
services/webhook/matrix.go
services/webhook/msteams.go
services/webhook/packagist.go
services/webhook/slack.go
services/webhook/telegram.go
services/webhook/wechatwork.go
run make lint-go and fix Forgejo specific warnings
The right to force merge is uses the wrong predicate and
applies to instance admins:
ctx.user.IsAdmin
It must apply to repository admins and use the following predicate:
ctx.userPerm.IsAdmin()
This regression is from the ApplyToAdmins implementation in
79b7089360.
Fixes: https://codeberg.org/forgejo/forgejo/issues/3780
This exception existed for both instance admins and repo admins
before ApplyToAdmins was introduced in
79b7089360.
It should have been kept for instance admins only because they are not
subject to permission checks.
More about codespell: https://github.com/codespell-project/codespell .
I personally introduced it to dozens if not hundreds of projects already and so far only positive feedback.
```
❯ grep lint-spell Makefile
@echo " - lint-spell lint spelling"
@echo " - lint-spell-fix lint spelling and fix issues"
lint: lint-frontend lint-backend lint-spell
lint-fix: lint-frontend-fix lint-backend-fix lint-spell-fix
.PHONY: lint-spell
lint-spell: lint-codespell
.PHONY: lint-spell-fix
lint-spell-fix: lint-codespell-fix
❯ git grep lint- -- .forgejo/
.forgejo/workflows/testing.yml: - run: make --always-make -j$(nproc) lint-backend checks-backend # ensure the "go-licenses" make target runs
.forgejo/workflows/testing.yml: - run: make lint-frontend
```
so how would you like me to invoke `lint-codespell` on CI? (without that would be IMHO very suboptimal and let typos sneak in)
Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/3270
Reviewed-by: Earl Warren <earl-warren@noreply.codeberg.org>
Co-authored-by: Yaroslav Halchenko <debian@onerussian.com>
Co-committed-by: Yaroslav Halchenko <debian@onerussian.com>
Noteable additions:
- `redefines-builtin-id` forbid variable names that shadow go builtins
- `empty-lines` remove unnecessary empty lines that `gofumpt` does not
remove for some reason
- `superfluous-else` eliminate more superfluous `else` branches
Rules are also sorted alphabetically and I cleaned up various parts of
`.golangci.yml`.
(cherry picked from commit 74f0c84fa4245a20ce6fb87dac1faf2aeeded2a2)
Conflicts:
.golangci.yml
apply the linter recommendations to Forgejo code as well
It is possible to change some repo settings (its visibility, and
template status) via `git push` options: `-o repo.private=true`, `-o
repo.template=true`.
Previously, there weren't sufficient permission checks on these, and
anyone who could `git push` to a repository - including via an AGit
workflow! - was able to change either of these settings. To guard
against this, the pre-receive hook will now check if either of these
options are present, and if so, will perform additional permission
checks to ensure that these can only be set by a repository owner or
an administrator. Additionally, changing these settings is disabled for
forks, even for the fork's owner.
There's still a case where the owner of a repository can change the
visibility of it, and it will not propagate to forks (it propagates to
forks when changing the visibility via the API), but that's an
inconsistency, not a security issue.
Signed-off-by: Gergely Nagy <forgejo@gergo.csillger.hu>
Signed-off-by: Earl Warren <contact@earl-warren.org>
- Currently protected branch rules do not apply to admins, however in
some cases (like in the case of Forgejo project) you might also want to
apply these rules to admins to avoid accidental merges.
- Add new option to configure this on a per-rule basis.
- Adds integration tests.
- Resolves #65
Since `modules/context` has to depend on `models` and many other
packages, it should be moved from `modules/context` to
`services/context` according to design principles. There is no logic
code change on this PR, only move packages.
- Move `code.gitea.io/gitea/modules/context` to
`code.gitea.io/gitea/services/context`
- Move `code.gitea.io/gitea/modules/contexttest` to
`code.gitea.io/gitea/services/contexttest` because of depending on
context
- Move `code.gitea.io/gitea/modules/upload` to
`code.gitea.io/gitea/services/context/upload` because of depending on
context
(cherry picked from commit 29f149bd9f517225a3c9f1ca3fb0a7b5325af696)
Conflicts:
routers/api/packages/alpine/alpine.go
routers/api/v1/repo/issue_reaction.go
routers/install/install.go
routers/web/admin/config.go
routers/web/passkey.go
routers/web/repo/search.go
routers/web/repo/setting/default_branch.go
routers/web/user/home.go
routers/web/user/profile.go
tests/integration/editor_test.go
tests/integration/integration_test.go
tests/integration/mirror_push_test.go
trivial context conflicts
also modified all other occurrences in Forgejo specific files
Now we can get object format name from git command line or from the
database repository table. Assume the column is right, we don't need to
read from git command line every time.
This also fixed a possible bug that the object format is wrong when
migrating a sha256 repository from external.
<img width="658" alt="image"
src="https://github.com/go-gitea/gitea/assets/81045/6e9a9dcf-13bf-4267-928b-6bf2c2560423">
(cherry picked from commit b79c30435f439af8243ee281310258cdf141e27b)
Conflicts:
routers/web/repo/blame.go
services/agit/agit.go
context
- Remove `ObjectFormatID`
- Remove function `ObjectFormatFromID`.
- Use `Sha1ObjectFormat` directly but not a pointer because it's an
empty struct.
- Store `ObjectFormatName` in `repository` struct
Refactor Hash interfaces and centralize hash function. This will allow
easier introduction of different hash function later on.
This forms the "no-op" part of the SHA256 enablement patch.
To avoid deadlock problem, almost database related functions should be
have ctx as the first parameter.
This PR do a refactor for some of these functions.
This PR replaces all string refName as a type `git.RefName` to make the
code more maintainable.
Fix #15367
Replaces #23070
It also fixed a bug that tags are not sync because `git remote --prune
origin` will not remove local tags if remote removed.
We in fact should use `git fetch --prune --tags origin` but not `git
remote update origin` to do the sync.
Some answer from ChatGPT as ref.
> If the git fetch --prune --tags command is not working as expected,
there could be a few reasons why. Here are a few things to check:
>
>Make sure that you have the latest version of Git installed on your
system. You can check the version by running git --version in your
terminal. If you have an outdated version, try updating Git and see if
that resolves the issue.
>
>Check that your Git repository is properly configured to track the
remote repository's tags. You can check this by running git config
--get-all remote.origin.fetch and verifying that it includes
+refs/tags/*:refs/tags/*. If it does not, you can add it by running git
config --add remote.origin.fetch "+refs/tags/*:refs/tags/*".
>
>Verify that the tags you are trying to prune actually exist on the
remote repository. You can do this by running git ls-remote --tags
origin to list all the tags on the remote repository.
>
>Check if any local tags have been created that match the names of tags
on the remote repository. If so, these local tags may be preventing the
git fetch --prune --tags command from working properly. You can delete
local tags using the git tag -d command.
---------
Co-authored-by: delvh <dev.lh@web.de>
# Why this PR comes
At first, I'd like to help users like #23636 (there are a lot)
The unclear "Internal Server Error" is quite anonying, scare users,
frustrate contributors, nobody knows what happens.
So, it's always good to provide meaningful messages to end users (of
course, do not leak sensitive information).
When I started working on the "response message to end users", I found
that the related code has a lot of technical debt. A lot of copy&paste
code, unclear fields and usages.
So I think it's good to make everything clear.
# Tech Backgrounds
Gitea has many sub-commands, some are used by admins, some are used by
SSH servers or Git Hooks. Many sub-commands use "internal API" to
communicate with Gitea web server.
Before, Gitea server always use `StatusCode + Json "err" field` to
return messages.
* The CLI sub-commands: they expect to show all error related messages
to site admin
* The Serv/Hook sub-commands (for git clients): they could only show
safe messages to end users, the error log could only be recorded by
"SSHLog" to Gitea web server.
In the old design, it assumes that:
* If the StatusCode is 500 (in some functions), then the "err" field is
error log, shouldn't be exposed to git client.
* If the StatusCode is 40x, then the "err" field could be exposed. And
some functions always read the "err" no matter what the StatusCode is.
The old code is not strict, and it's difficult to distinguish the
messages clearly and then output them correctly.
# This PR
To help to remove duplicate code and make everything clear, this PR
introduces `ResponseExtra` and `requestJSONResp`.
* `ResponseExtra` is a struct which contains "extra" information of a
internal API response, including StatusCode, UserMsg, Error
* `requestJSONResp` is a generic function which can be used for all
cases to help to simplify the calls.
* Remove all `map["err"]`, always use `private.Response{Err}` to
construct error messages.
* User messages and error messages are separated clearly, the `fail` and
`handleCliResponseExtra` will output correct messages.
* Replace all `Internal Server Error` messages with meaningful (still
safe) messages.
This PR saves more than 300 lines, while makes the git client messages
more clear.
Many gitea-serv/git-hook related essential functions are covered by
tests.
---------
Co-authored-by: delvh <dev.lh@web.de>
This PR introduce glob match for protected branch name. The separator is
`/` and you can use `*` matching non-separator chars and use `**` across
separator.
It also supports input an exist or non-exist branch name as matching
condition and branch name condition has high priority than glob rule.
Should fix #2529 and #15705
screenshots
<img width="1160" alt="image"
src="https://user-images.githubusercontent.com/81045/205651179-ebb5492a-4ade-4bb4-a13c-965e8c927063.png">
Co-authored-by: zeripath <art27@cantab.net>
After #22362, we can feel free to use transactions without
`db.DefaultContext`.
And there are still lots of models using `db.DefaultContext`, I think we
should refactor them carefully and one by one.
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Change all license headers to comply with REUSE specification.
Fix #16132
Co-authored-by: flynnnnnnnnnn <flynnnnnnnnnn@github>
Co-authored-by: John Olheiser <john.olheiser@gmail.com>
This PR adds a context parameter to a bunch of methods. Some helper
`xxxCtx()` methods got replaced with the normal name now.
Co-authored-by: delvh <dev.lh@web.de>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
* Move access and repo permission to models/perm/access
* fix test
* fix git test
* Move functions sequence
* Some improvements per @KN4CK3R and @delvh
* Move issues related code to models/issues
* Move some issues related sub package
* Merge
* Fix test
* Fix test
* Fix test
* Fix test
* Rename some files
* Move access and repo permission to models/perm/access
* fix test
* Move some git related files into sub package models/git
* Fix build
* fix git test
* move lfs to sub package
* move more git related functions to models/git
* Move functions sequence
* Some improvements per @KN4CK3R and @delvh
- dont overwrite err with nil unintentionaly
- rename CheckPRReadyToMerge to CheckPullBranchProtections
- rename prQueue to prPatchCheckerQueue
from #9307
Co-authored-by: delvh <dev.lh@web.de>
Adds a feature [like GitHub has](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/creating-a-pull-request-from-a-fork) (step 7).
If you create a new PR from a forked repo, you can select (and change later, but only if you are the PR creator/poster) the "Allow edits from maintainers" option.
Then users with write access to the base branch get more permissions on this branch:
* use the update pull request button
* push directly from the command line (`git push`)
* edit/delete/upload files via web UI
* use related API endpoints
You can't merge PRs to this branch with this enabled, you'll need "full" code write permissions.
This feature has a pretty big impact on the permission system. I might forgot changing some things or didn't find security vulnerabilities. In this case, please leave a review or comment on this PR.
Closes #17728
Co-authored-by: 6543 <6543@obermui.de>
Follows #19266, #8553, Close #18553, now there are only three `Run..(&RunOpts{})` functions.
* before: `stdout, err := RunInDir(path)`
* now: `stdout, _, err := RunStdString(&git.RunOpts{Dir:path})`
This make checks in one single place so they dont differ and maintainer can not forget a check in one place while adding it to the other .... ( as it's atm )
Fix:
* The API does ignore issue dependencies where Web does not
* The API checks if "IsSignedIfRequired" where Web does not - UI probably do but nothing will some to craft custom requests
* Default merge message is crafted a bit different between API and Web if not set on specific cases ...
This PR continues the work in #17125 by progressively ensuring that git
commands run within the request context.
This now means that the if there is a git repo already open in the context it will be used instead of reopening it.
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Some refactors related repository model
* Move more methods out of repository
* Move repository into models/repo
* Fix test
* Fix test
* some improvements
* Remove unnecessary function
* Clean-up HookPreReceive and restore functionality for pushing non-standard refs
There was an inadvertent breaking change in #15629 meaning that notes refs and other
git extension refs will be automatically rejected.
Further following #14295 and #15629 the pre-recieve hook code is untenably long and
too complex.
This PR refactors the hook code and removes the incorrect forced rejection of
non-standard refs.
Fix #16688
Signed-off-by: Andrew Thornton <art27@cantab.net>
