Backport #26745
Fixes #26548
This PR refactors the rendering of markup links. The old code uses
`strings.Replace` to change some urls while the new code uses more
context to decide which link should be generated.
The added tests should ensure the same output for the old and new
behaviour (besides the bug).
We may need to refactor the rendering a bit more to make it clear how
the different helper methods render the input string. There are lots of
options (resolve links / images / mentions / git hashes / emojis / ...)
but you don't really know what helper uses which options. For example,
we currently support images in the user description which should not be
allowed I think:
<details>
<summary>Profile</summary>
https://try.gitea.io/KN4CK3R
</details>
(cherry picked from commit 022552d5b6)
Sends email with information on the new user (time of creation and time of last sign-in) and a link to manage the new user from the admin panel
closes: https://codeberg.org/forgejo/forgejo/issues/480
Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/1371
Co-authored-by: Aravinth Manivannan <realaravinth@batsense.net>
Co-committed-by: Aravinth Manivannan <realaravinth@batsense.net>
(cherry picked from commit c721aa828b)
(cherry picked from commit 6487efcb9d)
Conflicts:
modules/notification/base/notifier.go
modules/notification/base/null.go
modules/notification/notification.go
https://codeberg.org/forgejo/forgejo/pulls/1422
(cherry picked from commit 7ea66ee1c5)
Conflicts:
services/notify/notifier.go
services/notify/notify.go
services/notify/null.go
https://codeberg.org/forgejo/forgejo/pulls/1469
(cherry picked from commit 7d2d997011)
(cherry picked from commit 435a54f140)
(cherry picked from commit 8ec7b3e448)
[GITEA] notifies admins on new user registration (squash) performance bottleneck
Refs: https://codeberg.org/forgejo/forgejo/issues/1479
(cherry picked from commit 97ac9147ff)
(cherry picked from commit 19f295c16b)
(cherry picked from commit 3367dcb2cf)
[GITEA] notifies admins on new user registration (squash) cosmetic changes
Co-authored-by: delvh <dev.lh@web.de>
(cherry picked from commit 9f1670e040)
(cherry picked from commit de5bb2a224)
(cherry picked from commit 8f8e52f31a)
(cherry picked from commit e0d5130312)
(cherry picked from commit f1288d6d9b)
(cherry picked from commit f664f41658)
(cherry picked from commit e44e6c7e47)
(cherry picked from commit c0d958cc4c)
(cherry picked from commit a88baa5e48)
[GITEA] notifies admins on new user registration (squash) ctx.Locale
(cherry picked from commit 2f6329f693)
As the docs of codeberg refer to the strings printed by the Forgejo
ssh servers, this is user-facing and is nice to update to the new
product name.
(cherry picked from commit 103991d73f)
(cherry picked from commit 2a0d3f85f1)
(cherry picked from commit eb2b4ce388)
(cherry picked from commit 0998b51716)
[BRANDING] forgejo log message
(cherry picked from commit d51a046ebe)
(cherry picked from commit d66e1c7b6e)
(cherry picked from commit b5bffe4ce8)
(cherry picked from commit 3fa776d856)
(cherry picked from commit 18d064f472)
(cherry picked from commit c95094e355)
(cherry picked from commit 5784290bc4)
(cherry picked from commit aee336886b)
(cherry picked from commit ec2f60b516)
(cherry picked from commit 7af742a284)
(cherry picked from commit f279e2a264)
(cherry picked from commit fd38cfb14e)
(cherry picked from commit 64c8226618)
(cherry picked from commit b546fb2304)
(cherry picked from commit ad10202177)
(cherry picked from commit c89cab9c2b)
(cherry picked from commit 9579322ec2)
(cherry picked from commit 16b44ad18d)
(cherry picked from commit 2571ff703b)
(cherry picked from commit ad61d9ce9b)
(cherry picked from commit 9b2c45d4d3)
(cherry picked from commit ed01b79a59)
(cherry picked from commit d040b66427)
(cherry picked from commit ffe0bbea48)
(cherry picked from commit 4c1b2c409b)
(cherry picked from commit 3d8338ed10)
(cherry picked from commit a92f044ea9)
[BRANDING] link to forgejo.org/docs instead of docs.gitea.io
(cherry picked from commit 3efafd0e08)
(cherry picked from commit 148185e34b)
(cherry picked from commit 834e264698)
(cherry picked from commit e72fa6eb1e)
[BRANDING] link to forgejo.org/docs instead of docs.gitea.io
Fix the link that was 404.
(cherry picked from commit ae515d7258)
(cherry picked from commit facc2367f0)
(cherry picked from commit 25784b9f21)
(cherry picked from commit 2efc6138d9)
(cherry picked from commit b9d0871631)
(cherry picked from commit f0446e51b9)
(cherry picked from commit 1638aa67fb)
(cherry picked from commit 290db6a018)
(cherry picked from commit 89b87cf542)
(cherry picked from commit 656ed94962)
(cherry picked from commit 036f879f96)
(cherry picked from commit 69eea35f81)
(cherry picked from commit b72e3f4a92)
(cherry picked from commit af606b8574)
(cherry picked from commit 7e47f8135c)
(cherry picked from commit 0e5218cc53)
(cherry picked from commit 7c2a20a528)
(cherry picked from commit 4e94006363)
(cherry picked from commit e47cdfc43f)
(cherry picked from commit 1dcb3e1da4)
(cherry picked from commit 67367c4e0f)
(cherry picked from commit 252087d1ff)
(cherry picked from commit f5977a43e5)
Conflicts:
templates/base/head_navbar.tmpl
https://codeberg.org/forgejo/forgejo/pulls/1351
(cherry picked from commit 594938eb15)
(cherry picked from commit 0257d038a7)
(cherry picked from commit 72821dd140)
[BRANDING] s/gitea/forgejo/ in HTML placeholders
Replaced Gitea branding with Forgejo for input placeholders
Closes: #686
Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/752
(cherry picked from commit 6160d37ca9)
(cherry picked from commit df61138c7e)
(cherry picked from commit 1f30566c3f)
(cherry picked from commit 539bb825f5)
(cherry picked from commit bee0f66c86)
(cherry picked from commit 60ad005c95)
(cherry picked from commit 282e26222e)
(cherry picked from commit f9ca551f3d)
(cherry picked from commit b2e04b04c3)
(cherry picked from commit c8f395a03c)
(cherry picked from commit 0d58ce49ae)
(cherry picked from commit c602ddf91e)
(cherry picked from commit 029e37271e)
(cherry picked from commit fdaa96b3cc)
(cherry picked from commit 515d99e27d)
(cherry picked from commit da73274ba1)
(cherry picked from commit ce90b696a0)
(cherry picked from commit b6bf98763b)
(cherry picked from commit 5b380d22d7)
[BRANDING] How to start a runner: URL to Actions admin documentation
(cherry picked from commit da91799e6f)
(cherry picked from commit 28231663b6)
(cherry picked from commit 533a90345b)
(cherry picked from commit 6a0e4e55dd)
(cherry picked from commit f47cd611c6)
(cherry picked from commit 001264b784)
(cherry picked from commit e4099e9bb9)
(cherry picked from commit 3a1885649f)
(cherry picked from commit c42802c710)
(cherry picked from commit a611ce8d6d)
(cherry picked from commit a3d7d10a80)
(cherry picked from commit 52adde671f)
(cherry picked from commit c9a3820fef)
(cherry picked from commit dce40997c9)
(cherry picked from commit 312a6b92f3)
[BRANDING] package templates & links
- Change Gitea to Forgejo where necessary.
- Point all documentation to Forgejo's documentation.
- Resolves #992
(cherry picked from commit d0b78a6ede)
(cherry picked from commit e2382f30ba)
(cherry picked from commit c41cf05a33)
(cherry picked from commit 797e598ae7)
(cherry picked from commit 970031a1c2)
(cherry picked from commit 0c1180e2e1)
Conflicts:
templates/package/content/alpine.tmpl
templates/package/content/cargo.tmpl
templates/package/content/chef.tmpl
templates/package/content/composer.tmpl
templates/package/content/conan.tmpl
templates/package/content/conda.tmpl
templates/package/content/container.tmpl
templates/package/content/cran.tmpl
templates/package/content/debian.tmpl
templates/package/content/generic.tmpl
templates/package/content/go.tmpl
templates/package/content/helm.tmpl
templates/package/content/maven.tmpl
templates/package/content/npm.tmpl
templates/package/content/nuget.tmpl
templates/package/content/pub.tmpl
templates/package/content/pypi.tmpl
templates/package/content/rpm.tmpl
templates/package/content/rubygems.tmpl
templates/package/content/swift.tmpl
templates/package/content/vagrant.tmpl
https://codeberg.org/forgejo/forgejo/pulls/1351
(cherry picked from commit 42ac9ff2ab)
(cherry picked from commit e390000bce)
(cherry picked from commit 56a437b29b)
Conflicts:
templates/package/content/cargo.tmpl
https://codeberg.org/forgejo/forgejo/pulls/1466
[BRANDING] s/Gitea/Forgejo/ in user visible help & comments
- Modify the README of the docker directory to point to the relevant
docker files and documentation for Forgejo.
(cherry picked from commit aca6371215)
(cherry picked from commit 0ba96b1bc4)
(cherry picked from commit 5c8e6b53f1)
Conflicts:
docker/README.md
https://codeberg.org/forgejo/forgejo/pulls/1351
(cherry picked from commit b3121c8004)
(cherry picked from commit 607f870416)
(cherry picked from commit 191d96afe4)
[BRANDING] healthcheck/check.go
(cherry picked from commit d703a236ce)
(cherry picked from commit d84ce3ff20)
(cherry picked from commit 2dbb844606)
(cherry picked from commit 14d3ae7e3a)
[BRANDING] s/Gitea/Forgejo/g in CLI output
(cherry picked from commit 7543c126bb)
(cherry picked from commit b66f422fc3)
(cherry picked from commit a81e4e46f3)
[BRANDING] Gitea->Forgejo in mailer code
(cherry picked from commit b91afea4ff)
(cherry picked from commit 5d7428167c)
(cherry picked from commit ed8101ba6c)
[BRANDING] use 'Forgejo' for Discord, Packagist, and Slack webhooks
Refs: https://codeberg.org/forgejo/forgejo/issues/1387
(cherry picked from commit 7dc3a05f5b)
(cherry picked from commit 133f2fc6cc)
[BRANDING] cmd/manager.go
(cherry picked from commit d1dba2c79d)
[BRANDING] pyproject.toml
(cherry picked from commit 7e8c868db2)
(cherry picked from commit 2395995c8b)
(cherry picked from commit dd6fbbf332)
Conflicts:
templates/package/content/cargo.tmpl
https://codeberg.org/forgejo/forgejo/pulls/1548
(cherry picked from commit 6f9a5d5cab)
(cherry picked from commit d0635c4a07)
(cherry picked from commit 5d3b4594df)
(cherry picked from commit 6da3b43eff)
(cherry picked from commit b60dfaba10)
(cherry picked from commit bcb9bb4dee)
(cherry picked from commit d5301b6a24)
Backport #27856
The only conflict is `ThemeName` in `500.tmpl`, it has been resolved
manually by keeping using old
`{{.SignedUser.Theme}}{{else}}{{DefaultTheme}}`
Before there was a "graceful function": RunWithShutdownFns, it's mainly
for some modules which doesn't support context.
The old queue system doesn't work well with context, so the old queues
need it.
After the queue refactoring, the new queue works with context well, so,
use Golang context as much as possible, the `RunWithShutdownFns` could
be removed (replaced by RunWithCancel for context cancel mechanism), the
related code could be simplified.
This PR also fixes some legacy queue-init problems, eg:
* typo : archiver: "unable to create codes indexer queue" => "unable to
create repo-archive queue"
* no nil check for failed queues, which causes unfriendly panic
After this PR, many goroutines could have better display name:
# ⚠️ Breaking
Many deprecated queue config options are removed (actually, they should
have been removed in 1.18/1.19).
If you see the fatal message when starting Gitea: "Please update your
app.ini to remove deprecated config options", please follow the error
messages to remove these options from your app.ini.
Example:
```
2023/05/06 19:39:22 [E] Removed queue option: `[indexer].ISSUE_INDEXER_QUEUE_TYPE`. Use new options in `[queue.issue_indexer]`
2023/05/06 19:39:22 [E] Removed queue option: `[indexer].UPDATE_BUFFER_LEN`. Use new options in `[queue.issue_indexer]`
2023/05/06 19:39:22 [F] Please update your app.ini to remove deprecated config options
```
Many options in `[queue]` are are dropped, including:
`WRAP_IF_NECESSARY`, `MAX_ATTEMPTS`, `TIMEOUT`, `WORKERS`,
`BLOCK_TIMEOUT`, `BOOST_TIMEOUT`, `BOOST_WORKERS`, they can be removed
from app.ini.
# The problem
The old queue package has some legacy problems:
* complexity: I doubt few people could tell how it works.
* maintainability: Too many channels and mutex/cond are mixed together,
too many different structs/interfaces depends each other.
* stability: due to the complexity & maintainability, sometimes there
are strange bugs and difficult to debug, and some code doesn't have test
(indeed some code is difficult to test because a lot of things are mixed
together).
* general applicability: although it is called "queue", its behavior is
not a well-known queue.
* scalability: it doesn't seem easy to make it work with a cluster
without breaking its behaviors.
It came from some very old code to "avoid breaking", however, its
technical debt is too heavy now. It's a good time to introduce a better
"queue" package.
# The new queue package
It keeps using old config and concept as much as possible.
* It only contains two major kinds of concepts:
* The "base queue": channel, levelqueue, redis
* They have the same abstraction, the same interface, and they are
tested by the same testing code.
* The "WokerPoolQueue", it uses the "base queue" to provide "worker
pool" function, calls the "handler" to process the data in the base
queue.
* The new code doesn't do "PushBack"
* Think about a queue with many workers, the "PushBack" can't guarantee
the order for re-queued unhandled items, so in new code it just does
"normal push"
* The new code doesn't do "pause/resume"
* The "pause/resume" was designed to handle some handler's failure: eg:
document indexer (elasticsearch) is down
* If a queue is paused for long time, either the producers blocks or the
new items are dropped.
* The new code doesn't do such "pause/resume" trick, it's not a common
queue's behavior and it doesn't help much.
* If there are unhandled items, the "push" function just blocks for a
few seconds and then re-queue them and retry.
* The new code doesn't do "worker booster"
* Gitea's queue's handlers are light functions, the cost is only the
go-routine, so it doesn't make sense to "boost" them.
* The new code only use "max worker number" to limit the concurrent
workers.
* The new "Push" never blocks forever
* Instead of creating more and more blocking goroutines, return an error
is more friendly to the server and to the end user.
There are more details in code comments: eg: the "Flush" problem, the
strange "code.index" hanging problem, the "immediate" queue problem.
Almost ready for review.
TODO:
* [x] add some necessary comments during review
* [x] add some more tests if necessary
* [x] update documents and config options
* [x] test max worker / active worker
* [x] re-run the CI tasks to see whether any test is flaky
* [x] improve the `handleOldLengthConfiguration` to provide more
friendly messages
* [x] fine tune default config values (eg: length?)
## Code coverage:
Add ntlm authentication support for mail
use "github.com/Azure/go-ntlmssp"
---------
Co-authored-by: yangtan_win <YangTan@Fitsco.com.cn>
Co-authored-by: silverwind <me@silverwind.io>
When creating attachments (issue, release, repo) the file size (being
part of the multipart file header) is passed through the chain of
creating an attachment to ensure the MinIO client can stream the file
directly instead of having to read it to memory completely at first.
Fixes #23393
Co-authored-by: KN4CK3R <admin@oldschoolhack.me>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
Close #23241
Before: press Ctrl+Enter in the Code Review Form, a single comment will
be added.
After: press Ctrl+Enter in the Code Review Form, start the review with
pending comments.
The old name `is_review` is not clear, so the new code use
`pending_review` as the new name.
Co-authored-by: delvh <leon@kske.dev>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
minio/sha256-simd provides additional acceleration for SHA256 using
AVX512, SHA Extensions for x86 and ARM64 for ARM.
It provides a drop-in replacement for crypto/sha256 and if the
extensions are not available it falls back to standard crypto/sha256.
---------
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: John Olheiser <john.olheiser@gmail.com>
Change the mailer interface to prevent leaking of possible hidden email
addresses when sending to multiple recipients.
Co-authored-by: Gusted <williamzijl7@hotmail.com>
Addition to #22056
This PR adds a hint to mail text if replies are supported.
I can't tell if the text structure is supported in every language. Maybe
we need to put the whole line in the translation file and use
parameters.
closes #13585
fixes #9067
fixes #2386
ref #6226
ref #6219
fixes #745
This PR adds support to process incoming emails to perform actions.
Currently I added handling of replies and unsubscribing from
issues/pulls. In contrast to #13585 the IMAP IDLE command is used
instead of polling which results (in my opinion 😉) in cleaner code.
Procedure:
- When sending an issue/pull reply email, a token is generated which is
present in the Reply-To and References header.
- IMAP IDLE waits until a new email arrives
- The token tells which action should be performed
A possible signature and/or reply gets stripped from the content.
I added a new service to the drone pipeline to test the receiving of
incoming mails. If we keep this in, we may test our outgoing emails too
in future.
Co-authored-by: silverwind <me@silverwind.io>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Change all license headers to comply with REUSE specification.
Fix #16132
Co-authored-by: flynnnnnnnnnn <flynnnnnnnnnn@github>
Co-authored-by: John Olheiser <john.olheiser@gmail.com>
Unfortunately the fallback configuration code for [mailer] that were
added in #18982 are incorrect. When you read a value from an ini section
that key is added. This leads to a failure of the fallback mechanism.
Further there is also a spelling mistake in the startTLS configuration.
This PR restructures the mailer code to first map the deprecated
settings on to the new ones - and then use ini.MapTo to map those on to
the struct with additional validation as necessary.
Ref #21744
Signed-off-by: Andrew Thornton <art27@cantab.net>
This PR adds a context parameter to a bunch of methods. Some helper
`xxxCtx()` methods got replaced with the normal name now.
Co-authored-by: delvh <dev.lh@web.de>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
I found myself wondering whether a PR I scheduled for automerge was
actually merged. It was, but I didn't receive a mail notification for it
- that makes sense considering I am the doer and usually don't want to
receive such notifications. But ideally I want to receive a notification
when a PR was merged because I scheduled it for automerge.
This PR implements exactly that.
The implementation works, but I wonder if there's a way to avoid passing
the "This PR was automerged" state down so much. I tried solving this
via the database (checking if there's an automerge scheduled for this PR
when sending the notification) but that did not work reliably, probably
because sending the notification happens async and the entry might have
already been deleted. My implementation might be the most
straightforward but maybe not the most elegant.
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: Lauris BH <lauris@nix.lv>
Co-authored-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
The recovery, API, Web and package frameworks all create their own HTML
Renderers. This increases the memory requirements of Gitea
unnecessarily with duplicate templates being kept in memory.
Further the reloading framework in dev mode for these involves locking
and recompiling all of the templates on each load. This will potentially
hide concurrency issues and it is inefficient.
This PR stores the templates renderer in the context and stores this
context in the NormalRoutes, it then creates a fsnotify.Watcher
framework to watch files.
The watching framework is then extended to the mailer templates which
were previously not being reloaded in dev.
Then the locales are simplified to a similar structure.
Fix #20210
Fix #20211
Fix #20217
Signed-off-by: Andrew Thornton <art27@cantab.net>
This bug affects tests which are sending emails (#20307). Some tests reinitialise the web routes (like `TestNodeinfo`) which messed up the mail templates. There is no reason why the templates should be loaded in the routes method.
* `PROTOCOL`: can be smtp, smtps, smtp+startls, smtp+unix, sendmail, dummy
* `SMTP_ADDR`: domain for SMTP, or path to unix socket
* `SMTP_PORT`: port for SMTP; defaults to 25 for `smtp`, 465 for `smtps`, and 587 for `smtp+startls`
* `ENABLE_HELO`, `HELO_HOSTNAME`: reverse `DISABLE_HELO` to `ENABLE_HELO`; default to false + system hostname
* `FORCE_TRUST_SERVER_CERT`: replace the unclear `SKIP_VERIFY`
* `CLIENT_CERT_FILE`, `CLIENT_KEY_FILE`, `USE_CLIENT_CERT`: clarify client certificates here
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Sometimes users want to receive email notifications of messages they create or reply to,
Added an option to personal preferences to allow users to choose
Closes #20149
* Refactor `i18n` to `locale`
- Currently we're using the `i18n` variable naming for the `locale`
struct. This contains locale's specific information and cannot be used
for general i18n purpose, therefore refactoring it to `locale` makes
more sense.
- Ref: https://github.com/go-gitea/gitea/pull/20096#discussion_r906699200
* Update routers/install/install.go
* Prototyping
* Start work on creating offsets
* Modify tests
* Start prototyping with actual MPH
* Twiddle around
* Twiddle around comments
* Convert templates
* Fix external languages
* Fix latest translation
* Fix some test
* Tidy up code
* Use simple map
* go mod tidy
* Move back to data structure
- Uses less memory by creating for each language a map.
* Apply suggestions from code review
Co-authored-by: delvh <dev.lh@web.de>
* Add some comments
* Fix tests
* Try to fix tests
* Use en-US as defacto fallback
* Use correct slices
* refactor (#4)
* Remove TryTr, add log for missing translation key
* Refactor i18n
- Separate dev and production locale stores.
- Allow for live-reloading in dev mode.
Co-authored-by: zeripath <art27@cantab.net>
* Fix live-reloading & check for errors
* Make linter happy
* live-reload with periodic check (#5)
* Fix tests
Co-authored-by: delvh <dev.lh@web.de>
Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
Co-authored-by: zeripath <art27@cantab.net>
* Move access and repo permission to models/perm/access
* fix test
* fix git test
* Move functions sequence
* Some improvements per @KN4CK3R and @delvh
* Move issues related code to models/issues
* Move some issues related sub package
* Merge
* Fix test
* Fix test
* Fix test
* Fix test
* Rename some files
* Move some repository related code into sub package
* Move more repository functions out of models
* Fix lint
* Some performance optimization for webhooks and others
* some refactors
* Fix lint
* Fix
* Update modules/repository/delete.go
Co-authored-by: delvh <dev.lh@web.de>
* Fix test
* Merge
* Fix test
* Fix test
* Fix test
* Fix test
Co-authored-by: delvh <dev.lh@web.de>
When Gitea is running as PID 1 git will occassionally orphan child processes leading
to (defunct) processes. This PR simply sets Setpgid to true on these child processes
meaning that these defunct processes will also be correctly reaped.
Fix #19077
Signed-off-by: Andrew Thornton <art27@cantab.net>
