- Databases are one of the most important parts of Forgejo, every
interaction with Forgejo uses the database in one way or another.
Therefore, it is important to maintain the database and recognize when
Forgejo is not doing well with the database. Forgejo already has the
option to log *every* SQL query along with its execution time, but
monitoring becomes impractical for larger instances and takes up
unnecessary storage in the logs.
- Add a QoL enhancement that allows instance administrators to specify a
threshold value beyond which query execution time is logged as a warning
in the xorm logger. The default value is a conservative five seconds to
avoid this becoming a source of spam in the logs.
- The use case for this patch is that with an instance the size of Codeberg, monitoring SQL logs is not very fruitful and most of them are uninteresting. Recently, in the context of persistent deadlock issues (https://codeberg.org/forgejo/forgejo/issues/220), I have noticed that certain queries hold locks on tables like comment and issue for several seconds. This patch helps to identify which queries these are and when they happen.
- Added unit test.
(cherry picked from commit 24bbe7886f)
(cherry picked from commit 6e29145b3c)
(cherry picked from commit 63731e3071)
(cherry picked from commit 3ce1a09736)
(cherry picked from commit a64426907d)
This field adds the possibility to set the update date when modifying
an issue through the API.
A 'NoAutoDate' in-memory field is added in the Issue struct.
If the update_at field is set, NoAutoDate is set to true and the
Issue's UpdatedUnix field is filled.
That information is passed down to the functions that actually updates
the database, which have been modified to not auto update dates if
requested.
A guard is added to the 'EditIssue' API call, to checks that the
udpate_at date is between the issue's creation date and the current
date (to avoid 'malicious' changes). It also limits the new feature
to project's owners and admins.
(cherry picked from commit c524d33402)
Add a SetIssueUpdateDate() function in services/issue.go
That function is used by some API calls to set the NoAutoDate and
UpdatedUnix fields of an Issue if an updated_at date is provided.
(cherry picked from commit f061caa655)
Add an updated_at field to the API calls related to Issue's Labels.
The update date is applied to the issue's comment created to inform
about the modification of the issue's labels.
(cherry picked from commit ea36cf80f5)
Add an updated_at field to the API call for issue's attachment creation
The update date is applied to the issue's comment created to inform
about the modification of the issue's content, and is set as the
asset creation date.
(cherry picked from commit 96150971ca)
Checking Issue changes, with and without providing an updated_at date
Those unit tests are added:
- TestAPIEditIssueWithAutoDate
- TestAPIEditIssueWithNoAutoDate
- TestAPIAddIssueLabelsWithAutoDate
- TestAPIAddIssueLabelsWithNoAutoDate
- TestAPICreateIssueAttachmentWithAutoDate
- TestAPICreateIssueAttachmentWithNoAutoDate
(cherry picked from commit 4926a5d7a2)
Add an updated_at field to the API call for issue's comment creation
The update date is used as the comment creation date, and is applied to
the issue as the update creation date.
(cherry picked from commit 76c8faecdc)
Add an updated_at field to the API call for issue's comment edition
The update date is used as the comment update date, and is applied to
the issue as an update date.
(cherry picked from commit cf787ad7fd)
Add an updated_at field to the API call for comment's attachment creation
The update date is applied to the comment, and is set as the asset
creation date.
(cherry picked from commit 1e4ff424d3)
Checking Comment changes, with and without providing an updated_at date
Those unit tests are added:
- TestAPICreateCommentWithAutoDate
- TestAPICreateCommentWithNoAutoDate
- TestAPIEditCommentWithAutoDate
- TestAPIEditCommentWithNoAutoDate
- TestAPICreateCommentAttachmentWithAutoDate
- TestAPICreateCommentAttachmentWithNoAutoDate
(cherry picked from commit da932152f1)
Pettier code to set the update time of comments
Now uses sess.AllCols().NoAutoToime().SetExpr("updated_unix", ...)
XORM is smart enough to compose one single SQL UPDATE which all
columns + updated_unix.
(cherry picked from commit 1f6a42808d)
Issue edition: Keep the max of the milestone and issue update dates.
When editing an issue via the API, an updated_at date can be provided.
If the EditIssue call changes the issue's milestone, the milestone's
update date is to be changed accordingly, but only with a greater
value.
This ensures that a milestone's update date is the max of all issue's
update dates.
(cherry picked from commit 8f22ea182e)
Rewrite the 'AutoDate' tests using subtests
Also add a test to check the permissions to set a date, and a test
to check update dates on milestones.
The tests related to 'AutoDate' are:
- TestAPIEditIssueAutoDate
- TestAPIAddIssueLabelsAutoDate
- TestAPIEditIssueMilestoneAutoDate
- TestAPICreateIssueAttachmentAutoDate
- TestAPICreateCommentAutoDate
- TestAPIEditCommentWithDate
- TestAPICreateCommentAttachmentAutoDate
(cherry picked from commit 961fd13c55)
(cherry picked from commit d52f4eea44)
(cherry picked from commit 3540ea2a43)
Conflicts:
services/issue/issue.go
https://codeberg.org/forgejo/forgejo/pulls/1415
(cherry picked from commit 56720ade00)
Conflicts:
routers/api/v1/repo/issue_label.go
https://codeberg.org/forgejo/forgejo/pulls/1462
(cherry picked from commit 57e597bf7e)
(cherry picked from commit 643a2b0e81)
(cherry picked from commit f10faffb4f)
(cherry picked from commit b440c5767e)
[TESTS] verify facts for the admin storage documentation (squash)
(cherry picked from commit d83d8ce57b)
(cherry picked from commit d8855ef27c)
(cherry picked from commit 11230466ec)
Follow-up to d58c542579 for Forgejo.
By default, Gitea does not select any map service that can be used
to introduce a 'Show this place on a map' button in the location
field of a user profile. Before I tried upstreaming this change to
Gitea, this was the case in Forgejo. This patch essentially recovers
this functionality, which is nice for public-facing instances and
communities.
Links to original PRs:
- https://codeberg.org/forgejo/forgejo/pulls/1076
- https://github.com/go-gitea/gitea/pull/26214
(cherry picked from commit bb187d5f61)
(cherry picked from commit ce02ef9078)
(cherry picked from commit 6b75c40e25)
(cherry picked from commit 6bc8e9f573)
(cherry picked from commit 063f8afdf7)
(cherry picked from commit c5cc736b72)
(cherry picked from commit 7b1bb4bedc)
(cherry picked from commit f9d75d4705)
(cherry picked from commit 64f76f4ab2)
(cherry picked from commit 5d02454155)
[CI] Forgejo Actions workflows
(cherry picked from commit 3ff59b5379)
(cherry picked from commit 8af826a6f7)
(cherry picked from commit d7c09d9cc8)
[CI] use the docker label instead of ubuntu-latest
(cherry picked from commit b6a6470db6)
[CI] all tests need compliance before proceeding
(cherry picked from commit b35c496f2c)
(cherry picked from commit 36a4148a8e)
(cherry picked from commit 7ffcffa653)
(cherry picked from commit 8a246d296e)
(cherry picked from commit dd0b6e1826)
[CI] Forgejo Actions based release process (squash) MySQL optimization
Refs: https://codeberg.org/forgejo/forgejo/issues/976
(cherry picked from commit b4b8c489e6)
(cherry picked from commit 1e861db4af)
(cherry picked from commit a6c0e00330)
(cherry picked from commit f97b336465)
(cherry picked from commit 6d65d5f0d6f798556a8f9e547896be03a5ee2f87)
(cherry picked from commit 79bfbadbed)
(cherry picked from commit e86c40a34a)
(cherry picked from commit da0c454adb)
(cherry picked from commit b49d892cda)
[CI] enable minio tests
(cherry picked from commit 4d8f438031)
(cherry picked from commit c4eeb0a61e)
[CI] Forgejo Actions based CI for PR & branches (squash) cleanup
(cherry picked from commit 80eb20e842)
(cherry picked from commit d2ff589858)
(cherry picked from commit f6eedecb67)
(cherry picked from commit 8b11cab677)
(cherry picked from commit be59270696)
(cherry picked from commit e068f8b191)
(cherry picked from commit 7855bb0c60)
(cherry picked from commit 45c4c8f443)
(cherry picked from commit 89520d67ff)
(cherry picked from commit 15eeb417a4)
(cherry picked from commit 6db53a2643)
(cherry picked from commit 2f689b321f)
(cherry picked from commit 04dc478314c3b4927cca78c354ca46ee217f035a)
(cherry picked from commit a554624f40)
(cherry picked from commit abca05f0d1)
(cherry picked from commit dc13e7eb22)
(cherry picked from commit a161c5740e)
(cherry picked from commit 06d33e2773)
(cherry picked from commit f536275161)
(cherry picked from commit a1381d9146fba42cb97d72d38525fa3e721bfb03)
(cherry picked from commit 74714e0246)
(cherry picked from commit 7749dbfe66)
(cherry picked from commit 4379249711)
(cherry picked from commit a69f55bebf)
(cherry picked from commit 24dd5fbfdb)
(cherry picked from commit dda856d6b8)
(cherry picked from commit e1bbfa3619)
(cherry picked from commit 91245ca917)
(cherry picked from commit 705d0558be)
(cherry picked from commit 9247594970)
(cherry picked from commit 9db1158a48)
(cherry picked from commit 3b36b77d87)
(cherry picked from commit 162fa1d8ae)
(cherry picked from commit d03d0afbb5)
(cherry picked from commit 7b8f92f787)
(cherry picked from commit 035abca969)
(cherry picked from commit a8fbf6bb56)
(cherry picked from commit 3be681d037b07880236cae1aa70245e5eb4d1497)
(cherry picked from commit 7e5d471c83)
(cherry picked from commit 323801d935)
(cherry picked from commit 3fdfe4bfea)
(cherry picked from commit 58a07421a4)
(cherry picked from commit dbb71a4c85)
(cherry picked from commit d442113520)
- Refactor lfs request code
- The original code uses `performRequest` function to create the
request, uses a callback to modify the request, and then send the
request.
- Now it's replaced with `createRequest` that only creates request and
`performRequest` that only sends the request.
- Reuse `createRequest` and `performRequest` in `http_client.go` and
`transferadapter.go`
---------
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
Closes #26329
This PR adds the ability to ignore revisions specified in the
`.git-blame-ignore-revs` file in the root of the repository.
The banner is displayed in this case. I intentionally did not add a UI
way to bypass the ignore file (same behaviour as Github) but you can add
`?bypass-blame-ignore=true` to the url manually.
---------
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
This PR adds a new field `RemoteAddress` to both mirror types which
contains the sanitized remote address for easier (database) access to
that information. Will be used in the audit PR if merged.
As described in the title.
Some points:
1. Why need those tests?
Because `buildIssueOverview` is not well tested, there are several
continuous bugs in the issue overview webpage.
2. Why in indexer_test.go?
It's hard to put those tests in `./modules/indexer/issue/db/db_test.go`
because those tests need 'real' data in db mocked by fixtures instead of
random data in `./modules/indexer/issue/internal/tests`. When using
'real' data(`unittest.PrepareTestDatabase`), `InitIssueIndexer` and the
package `init()` function of `indexer` are required to init indexer.
3. Why only db?
The other three indexer engines are well tested by random data and it's
okay to also test them with 'real' data in db mocked by fixtures. Any
follow-up PR is welcome.
4. Those tests are really basic, any more complicated tests are welcome.
5. I think it's also necessary to add tests in `TestAPISearchIssues`
in`api_test_issue.go` and `TestIssues` in `home_test.go`
Part of #27065
This reduces the usage of `db.DefaultContext`. I think I've got enough
files for the first PR. When this is merged, I will continue working on
this.
Considering how many files this PR affect, I hope it won't take to long
to merge, so I don't end up in the merge conflict hell.
---------
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
Since the issue indexer has been refactored, the issue overview webpage
is built by the `buildIssueOverview` function and underlying
`indexer.Search` function and `GetIssueStats` instead of
`GetUserIssueStats`. So the function is no longer used.
I moved the relevant tests to `indexer_test.go` and since the search
option changed from `IssueOptions` to `SearchOptions`, most of the tests
are useless now.
We need more tests about the db indexer because those tests are highly
connected with the issue overview webpage and now this page has several
bugs.
Any advice about those test cases is appreciated.
---------
Co-authored-by: CaiCandong <50507092+CaiCandong@users.noreply.github.com>
- In org mode you can specify an description for media via the following
syntax `[[description][media link]]`. The description is then used as
title or alt.
- This patch fixes the rendering of the description by seperating the
description and non-description cases and using `org.String()`.
- Added unit tests.
- Inspired by
6eb20dbda9/org/html_writer.go (L406-L427)
- Resolves https://codeberg.org/Codeberg/Community/issues/848
(cherry picked from commit 8b8aab8311)
Co-authored-by: Gusted <postmaster@gusted.xyz>
Fix #26723
Add `ChangeDefaultBranch` to the `notifier` interface and implement it
in `indexerNotifier`. So when changing the default branch,
`indexerNotifier` sends a message to the `indexer queue` to update the
index.
---------
Co-authored-by: techknowlogick <matti@mdranta.net>
Should BucketExists (HeadBucket) fail because of an error related to
the connection rather than the existence of the bucket, no information
is available and the admin is left guessing.
https://docs.aws.amazon.com/AmazonS3/latest/API/API_HeadBucket.html
> This action is useful to determine if a bucket exists and you have
> permission to access it. The action returns a 200 OK if the bucket
> exists and you have permission to access it.
>
> If the bucket does not exist or you do not have permission to access
> it, the HEAD request returns a generic 400 Bad Request, 403
> Forbidden or 404 Not Found code. A message body is not included, so
> you cannot determine the exception beyond these error codes.
GetBucketVersioning is used instead and exclusively dedicated to
asserting if using the connection does not return a BadRequest.
If it does the NewMinioStorage logs an error and returns. Otherwise
it keeps going knowing that BucketExists is not going to fail for
reasons unrelated to the existence of the bucket and the permissions
to access it.
(cherry picked from commit d1df4b3bc62e5e61893a923f1c4b58f084eb03af)
Refs: https://codeberg.org/forgejo/forgejo/issues/1338
Unfortunately, when a system setting hasn't been stored in the database,
it cannot be cached.
Meanwhile, this PR also uses context cache for push email avatar display
which should avoid to read user table via email address again and again.
According to my local test, this should reduce dashboard elapsed time
from 150ms -> 80ms .
If the AppURL(ROOT_URL) is an HTTPS URL, then the COOKIE_SECURE's
default value should be true.
And, if a user visits an "http" site with "https" AppURL, they won't be
able to login, and they should have been warned. The only problem is
that the "language" can't be set either in such case, while I think it
is not a serious problem, and it could be fixed easily if needed.
A set of terminology, along with a broader description, can help more
people engage with the Gitea queue system, providing insights and
ensuring its correct use.
---------
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
This feature was removed by #22219 to avoid possible CSRF attack.
This PR takes reverseproxy auth for API back but with default disabled.
To prevent possbile CSRF attack, the responsibility will be the
reverseproxy but not Gitea itself.
For those want to enable this `ENABLE_REVERSE_PROXY_AUTHENTICATION_API`,
they should know what they are doing.
---------
Co-authored-by: Giteabot <teabot@gitea.io>
Currently, Artifact does not have an expiration and automatic cleanup
mechanism, and this feature needs to be added. It contains the following
key points:
- [x] add global artifact retention days option in config file. Default
value is 90 days.
- [x] add cron task to clean up expired artifacts. It should run once a
day.
- [x] support custom retention period from `retention-days: 5` in
`upload-artifact@v3`.
- [x] artifacts link in actions view should be non-clickable text when
expired.
1. The `og:description` should be "a one to two sentence description of
your object"
* It shouldn't output all the user inputted content -- it would be
pretty huge.
* Maybe it only needs at most 300 bytes.
2. Do not render commit message as HTML
- While doing some sanity checks over OpenSSH's code for how they handle
certificates authentication. I stumbled on an condition that checks the
certificate type is really an user certificate on the server-side
authentication. This checks seems to be a formality and just for the
sake of good domain seperation, because an user and host certificate
don't differ in their generation, verification or flags that can be
included.
- Add this check to the builtin SSH server to stay close to the
unwritten SSH specification.
- This is an breaking change for setups where the builtin SSH server is
being used and for some reason host certificates were being used for
authentication.
-
(cherry picked from commit de35b141b7)
Refs: https://codeberg.org/forgejo/forgejo/pulls/1172
## ⚠️ BREAKING ⚠️
Like OpenSSH, the built-in SSH server will now only accept SSH user
certificates, not server certificates.
Co-authored-by: Gusted <postmaster@gusted.xyz>
Co-authored-by: Giteabot <teabot@gitea.io>
1. The old `prepareQueryArg` did double-unescaping of form value.
2. By the way, remove the unnecessary `ctx.Flash = ...` in
`MockContext`.
Co-authored-by: Giteabot <teabot@gitea.io>
Just like `models/unittest`, the testing helper functions should be in a
separate package: `contexttest`
And complete the TODO:
> // TODO: move this function to other packages, because it depends on
"models" package
