- Configure `setting.CacheService.TTL` which will force the code to go
trough the caching mechanism.
- Remove the TODO and uncomment the test code.
(cherry picked from commit a201f2f189)
Refs: https://codeberg.org/forgejo/forgejo/pulls/974
---------
Co-authored-by: Gusted <postmaster@gusted.xyz>
- The `NoBetterThan` function can only handle comparisons between
"pending," "success," "error," and "failure." For any other comparison,
we directly return false. This prevents logic errors like the one in
#26121.
- The callers of the `NoBetterThan` function should also avoid making
incomparable calls.
---------
Co-authored-by: yp05327 <576951401@qq.com>
Co-authored-by: puni9869 <80308335+puni9869@users.noreply.github.com>
The JSONRedirect/JSONOK/JSONError functions were put into "Base" context
incorrectly, it would cause abuse.
Actually, they are for "web context" only, so, move them to the correct
place.
And by the way, use them to simplify old code: +75 -196
This PR includes #26007 's changes but have a UI to prompt administrator
about the deprecated settings as well as the log or console warning.
Then users will have enough time to notice the problem and don't have
surprise like before.
<img width="1293" alt="图片"
src="https://github.com/go-gitea/gitea/assets/81045/c33355f0-1ea7-4fb3-ad43-cd23cd15391d">
---------
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
This is a simple PR which moves the `GetListener` function to a
`DefaultGetListener` function, and changes `GetListener` to be a
variable which by default points to the `DefaultGetListener` function.
This allows people who may exist quasi-downstream of Gitea to create
alternate "GetListener" functions, with identical signatures, which
return different implementations of the `net.Listener` interface. This
approach is expressly intended to be non-invasive and have the least
possible impact on the gitea codebase. A previous version of this idea
was rejected before: https://github.com/go-gitea/gitea/issues/15544 but
because of issues like: https://github.com/go-gitea/gitea/issues/22335 I
**really** think that recommending people configure proxies by hand is
exactly the wrong way to do things(This is why there is a Tor Browser.).
This tiny change lets me put proper hidden service configuration into
single `i2p.go` file which lives in `modules/graceful/` and which never
has to be checked in to your codebase or affect your dependencies or
bloat your project in any way, it can live on a branch in my fork and
I'll fast-forward every release and never the twain shall meet.
The main use-case for this is to listen on Peer-to-Peer networks and
Hidden Services directly without error-prone and cumbersome
port-forwarding configuration. For instance, I might implement an
"I2PGetListener" as follows:
```Go
// adapted from i2p.go which is unchecked-in in my modules/graceful/ directory
import "github.com/eyedeekay/onramp"
var garlic = &onramp.Garlic{}
func I2PGetListener(network, address string) (net.Listener, error) {
// Add a deferral to say that we've tried to grab a listener
defer GetManager().InformCleanup()
switch network {
case "tcp", "tcp4", "tcp6", "i2p", "i2pt":
return garlic.Listen()
case "unix", "unixpacket":
// I2P isn't really a replacement for the stuff you use Unix sockets for and it's also not an anonymity risk, so treat them normally
unixAddr, err := net.ResolveUnixAddr(network, address)
if err != nil {
return nil, err
}
return GetListenerUnix(network, unixAddr)
default:
return nil, net.UnknownNetworkError(network)
}
}
```
I could then substitute that GetListener function and be 50% of the way
to having a fully-functioning gitea-over-hidden-services instance
without any additional configuration(The other 50% doesn't require any
code-changes on gitea's part).
There are 2 advantages here, one being convenience, first this turns
hidden services into a zero-configuration option for self-hosting gitea,
and second safety, these Go libraries are passing around
hidden-service-only versions of the net.Addr struct, they're using
hidden-service-only versions of the sockets, which are both expressly
designed to never require access to any information outside the hidden
service network, manipulating the application so it reveals information
about the host becomes much more difficult, and some attacks become
nearly impossible. It also opens up TLS-over-Hidden Services support
which is niche right now, of course, but in a future where gitea
instances federate if hidden services want to be part of the federation
they're probably going to need TLS certificates. They don't need to be
painful to set up.
This doesn't fix an open issue, but it might affect:
- https://github.com/go-gitea/gitea/issues/22335 - my `i2p.go` file
actually has a mod that fixes this but it requires adding a handful of
new dependencies to gitea and isn't compatible with the normal way you
guys recommend using a proxy so I don't think it's ready to send to you
as a PR, but if I can find a non-invasive way to fix it I will.
- https://github.com/go-gitea/gitea/issues/18240
I hereby agree to the Code of Conduct published here:
8b89563bf1/CODE_OF_CONDUCT.md
I have read and understood the recommendations published here:
8b89563bf1/CONTRIBUTING.md
Thank you for your consideration.
---------
Co-authored-by: eyedeekay <idk@mulder>
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
If the log file can't be opened, what should it do? panic/exit? ignore
logs? fallback to stderr?
It seems that "fallback to stderr" is slightly better than others ....
To avoid deadlock problem, almost database related functions should be
have ctx as the first parameter.
This PR do a refactor for some of these functions.
Replace #25892
Close #21942
Close #25464
Major changes:
1. Serve "robots.txt" and ".well-known/security.txt" in the "public"
custom path
* All files in "public/.well-known" can be served, just like
"public/assets"
3. Add a test for ".well-known/security.txt"
4. Simplify the "FileHandlerFunc" logic, now the paths are consistent so
the code can be simpler
5. Add CORS header for ".well-known" endpoints
6. Add logs to tell users they should move some of their legacy custom
public files
```
2023/07/19 13:00:37 cmd/web.go:178:serveInstalled() [E] Found legacy public asset "img" in CustomPath. Please move it to /work/gitea/custom/public/assets/img
2023/07/19 13:00:37 cmd/web.go:182:serveInstalled() [E] Found legacy public asset "robots.txt" in CustomPath. Please move it to /work/gitea/custom/public/robots.txt
```
This PR is not breaking.
---------
Co-authored-by: silverwind <me@silverwind.io>
Co-authored-by: Giteabot <teabot@gitea.io>
Fix #25776. Close #25826.
In the discussion of #25776, @wolfogre's suggestion was to remove the
commit status of `running` and `warning` to keep it consistent with
github.
references:
-
https://docs.github.com/en/rest/commits/statuses?apiVersion=2022-11-28#about-commit-statuses
## ⚠️ BREAKING ⚠️
So the commit status of Gitea will be consistent with GitHub, only
`pending`, `success`, `error` and `failure`, while `warning` and
`running` are not supported anymore.
---------
Co-authored-by: Jason Song <i@wolfogre.com>
The package `github.com/nfnt/resize` is deprecated and archived by the
author. `github.com/oliamb/cutter` is not maintained since 2018. We
could use `golang.org/x/image/draw` instead.
Bumping `github.com/golang-jwt/jwt` from v4 to v5.
`github.com/golang-jwt/jwt` v5 is bringing some breaking changes:
- standard `Valid()` method on claims is removed. It's replaced by
`ClaimsValidator` interface implementing `Validator()` method instead,
which is called after standard validation. Gitea doesn't seem to be
using this logic.
- `jwt.Token` has a field `Valid`, so it's checked in `ParseToken`
function in `services/auth/source/oauth2/token.go`
---------
Co-authored-by: Giteabot <teabot@gitea.io>
Update WorkPath/WORK_PATH related documents, remove out-dated
information.
Remove "StaticRootPath" on the admin config display page, because few
end user really need it, it only causes misconfiguration.
Co-authored-by: Giteabot <teabot@gitea.io>
Before: the concept "Content string" is used everywhere. It has some
problems:
1. Sometimes it means "base64 encoded content", sometimes it means "raw
binary content"
2. It doesn't work with large files, eg: uploading a 1G LFS file would
make Gitea process OOM
This PR does the refactoring: use "ContentReader" / "ContentBase64"
instead of "Content"
This PR is not breaking because the key in API JSON is still "content":
`` ContentBase64 string `json:"content"` ``
Move `public/*` to `public/assets/*`
Some old PRs (like #15219) introduced inconsistent directory system.
For example: why the local directory "public" is accessed by
`http://site/assets`? How to serve the ".well-known" files properly in
the public directory?
For convention rules, the "public" directory is widely used for the
website's root directory. It shouldn't be an exception for Gitea.
So, this PR makes the things consistent:
* `http://site/assets/foo` means `{CustomPath}/public/assets/foo`.
* `{CustomPath}/public/.well-known` and `{CustomPath}/public/robots.txt`
can be used in the future.
This PR is also a prerequisite for a clear solution for:
* #21942
* #25892
* discourse.gitea.io: [.well-known path serving custom files behind
proxy?](https://discourse.gitea.io/t/well-known-path-serving-custom-files-behind-proxy/5445/1)
This PR is breaking for users who have custom "public" files (CSS/JS).
After getting approvals, I will update the documents.
----
## ⚠️ BREAKING ⚠️
If you have files in your "custom/public/" folder, please move them to
"custom/public/assets/".
---------
Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: Giteabot <teabot@gitea.io>
Regression: https://github.com/go-gitea/gitea/pull/24805
Closes: #25945
- Disallow `javascript`, `vbscript` and `data` (data uri images still
work) url schemes even if all other schemes are allowed
- Fixed older `cbthunderlink` tests
---------
Co-authored-by: delvh <dev.lh@web.de>
The "creating dir/file during load config" is a longstanding and complex
problem.
This PR only does a quick patch, it still needs more refactorings in the
future.
Fix #25938
Before, in dev mode, there might be some error logs like:
```
2023/07/17 13:54:51 ...s/assetfs/layered.go:221:WatchLocalChanges() [E] Unable to watch directory .: lstat /data/work/gitea/custom/templates: no such file or directory
```
Because there is no "custom/templates" directory.
After: ignore such error, no such error message anymore.
Got the same problem as #25915 when updating an instance. The
`log.Fatal` should have been marked as breaking in #23911.
This PR adds a notice that the system is shutting down because of the
deprecated setting.
The `FileBlame` function looks strange, it has `revision` as argument
but doesn't use it.
Since the function never be used, I think we could just remove it.
If anyone thinks it should be kept, please help fix `revision`.
Co-authored-by: Giteabot <teabot@gitea.io>
Replace #25580
Fix #19453
The problem was: when users set "GITEA__XXX__YYY" , the "install page"
doesn't respect it.
So, to make the result consistent and avoid surprising end users, now
the "install page" also writes the environment variables to the config
file.
And, to make things clear, there are enough messages on the UI to tell
users what will happen.
There are some necessary/related changes to `environment-to-ini.go`:
* The "--clear" flag is removed and it was incorrectly written there.
The "clear" operation should be done if INSTALL_LOCK=true
* The "--prefix" flag is removed because it's never used, never
documented and it only causes inconsistent behavior.
Fixes (?) #25538
Fixes https://codeberg.org/forgejo/forgejo/issues/972
Regression #23879#23879 introduced a change which prevents read access to packages if a
user is not a member of an organization.
That PR also contained a change which disallows package access if the
team unit is configured with "no access" for packages. I don't think
this change makes sense (at the moment). It may be relevant for private
orgs. But for public or limited orgs that's useless because an
unauthorized user would have more access rights than the team member.
This PR restores the old behaviour "If a user has read access for an
owner, they can read packages".
---------
Co-authored-by: Giteabot <teabot@gitea.io>
To mock a handler:
```go
web.RouteMock(web.MockAfterMiddlewares, func(ctx *context.Context) {
// ...
})
defer web.RouteMockReset()
```
It helps:
* Test the middleware's behavior (assert the ctx.Data, etc)
* Mock the middleware's behavior (prepare some context data for handler)
* Mock the handler's response for some test cases, especially for some
integration tests and e2e tests.
Follow #25229
At present, when the trigger event is `pull_request_target`, the `ref`
and `sha` of `ActionRun` are set according to the base branch of the
pull request. This makes it impossible for us to find the head branch of
the `ActionRun` directly. In this PR, the `ref` and `sha` will always be
set to the head branch and they will be changed to the base branch when
generating the task context.
A couple of notes:
* Future changes should refactor arguments into a struct
* This filtering only is supported by meilisearch right now
* Issue index number is bumped which will cause a re-index
Fix #25736
Caused by #24048
Right now we only check the activity type for `pull_request` event when
`types` is specified or there are no `types` and filter. If a workflow
only specifies filters but no `types` like this:
```
on:
pull_request:
branches: [main]
```
the workflow will be triggered even if the activity type is not one of
`[opened, reopened, sync]`. We need to check the activity type in this
case.
Co-authored-by: Giteabot <teabot@gitea.io>
Remove unnecessary `if opts.Logger != nil` checks.
* For "CLI doctor" mode, output to the console's "logger.Info".
* For "Web Task" mode, output to the default "logger.Debug", to avoid
flooding the server's log in a busy production instance.
Co-authored-by: Giteabot <teabot@gitea.io>
To record which command is slow, this PR adds a debug log for slow git
operations.
---------
Co-authored-by: Lauris BH <lauris@nix.lv>
Co-authored-by: delvh <dev.lh@web.de>
Fix regression of #5363 (so long ago).
The old code definded a document mapping for `issueIndexerDocType`, and
assigned it to `BleveIndexerData` as its type. (`BleveIndexerData` has
been renamed to `IndexerData` in #25174, but nothing more.) But the old
code never used `BleveIndexerData`, it wrote the index with an anonymous
struct type. Nonetheless, bleve would use the default auto-mapping for
struct it didn't know, so the indexer still worked. This means the
custom document mapping was always dead code.
The custom document mapping is not useless, it can reduce index storage,
this PR brings it back and disable default mapping to prevent it from
happening again. Since `IndexerData`(`BleveIndexerData`) has JSON tags,
and bleve uses them first, so we should use `repo_id` as the field name
instead of `RepoID`.
I did a test to compare the storage size before and after this, with
about 3k real comments that were migrated from some public repos.
Before:
```text
[ 160] .
├── [ 42] index_meta.json
├── [ 13] rupture_meta.json
└── [ 128] store
├── [6.9M] 00000000005d.zap
└── [256K] root.bolt
```
After:
```text
[ 160] .
├── [ 42] index_meta.json
├── [ 13] rupture_meta.json
└── [ 128] store
├── [3.5M] 000000000065.zap
└── [256K] root.bolt
```
It saves about half the storage space.
---------
Co-authored-by: Giteabot <teabot@gitea.io>
Fixes #24723
Direct serving of content aka HTTP redirect is not mentioned in any of
the package registry specs but lots of official registries do that so it
should be supported by the usual clients.
This prevents the disk from overflowing with auth keys file
Fixes #17117
## ⚠️ BREAKING
This changes the default option for creating a backup of the authorized
key file when an update is made to default to false.
When branch's commit CommitMessage is too long, the column maybe too
short.(TEXT 16K for mysql).
This PR will fix it to only store the summary because these message will
only show on branch list or possible future search?
Resolve #24789
## ⚠️ BREAKING ⚠️
Before this, `DEFAULT_ACTIONS_URL` cound be set to any custom URLs like
`https://gitea.com` or `http://your-git-server,https://gitea.com`, and
the default value was `https://gitea.com`.
But now, `DEFAULT_ACTIONS_URL` supports only
`github`(`https://github.com`) or `self`(the root url of current Gitea
instance), and the default value is `github`.
If it has configured with a URL, an error log will be displayed and it
will fallback to `github`.
Actually, what we really want to do is always make it
`https://github.com`, however, this may not be acceptable for some
instances of internal use, so there's extra support for `self`, but no
more, even `https://gitea.com`.
Please note that `uses: https://xxx/yyy/zzz` always works and it does
exactly what it is supposed to do.
Although it's breaking, I belive it should be backported to `v1.20` due
to some security issues.
Follow-up on the runner side:
- https://gitea.com/gitea/act_runner/pulls/262
- https://gitea.com/gitea/act/pulls/70
This adds an API for uploading and Deleting Avatars for of Users, Repos
and Organisations. I'm not sure, if this should also be added to the
Admin API.
Resolves #25344
---------
Co-authored-by: silverwind <me@silverwind.io>
Co-authored-by: Giteabot <teabot@gitea.io>
Related #14180
Related #25233
Related #22639
Close #19786
Related #12763
This PR will change all the branches retrieve method from reading git
data to read database to reduce git read operations.
- [x] Sync git branches information into database when push git data
- [x] Create a new table `Branch`, merge some columns of `DeletedBranch`
into `Branch` table and drop the table `DeletedBranch`.
- [x] Read `Branch` table when visit `code` -> `branch` page
- [x] Read `Branch` table when list branch names in `code` page dropdown
- [x] Read `Branch` table when list git ref compare page
- [x] Provide a button in admin page to manually sync all branches.
- [x] Sync branches if repository is not empty but database branches are
empty when visiting pages with branches list
- [x] Use `commit_time desc` as the default FindBranch order by to keep
consistent as before and deleted branches will be always at the end.
---------
Co-authored-by: Jason Song <i@wolfogre.com>
releated to #21820
- Split `Size` in repository table as two new colunms, one is `GitSize`
for git size, the other is `LFSSize` for lfs data. still store full size
in `Size` colunm.
- Show full size on ui, but show each of them by a `title`; example:
- Return full size in api response.
---------
Signed-off-by: a1012112796 <1012112796@qq.com>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Co-authored-by: silverwind <me@silverwind.io>
Co-authored-by: DmitryFrolovTri <23313323+DmitryFrolovTri@users.noreply.github.com>
Co-authored-by: Giteabot <teabot@gitea.io>
Fix #25451.
Bugfixes:
- When stopping the zombie or endless tasks, set `LogInStorage` to true
after transferring the file to storage. It was missing, it could write
to a nonexistent file in DBFS because `LogInStorage` was false.
- Always update `ActionTask.Updated` when there's a new state reported
by the runner, even if there's no change. This is to avoid the task
being judged as a zombie task.
Enhancement:
- Support `Stat()` for DBFS file.
- `WriteLogs` refuses to write if it could result in content holes.
---------
Co-authored-by: Giteabot <teabot@gitea.io>
More fix for #24981
* #24981
Close #22361
* #22361
There were many patches for Gitea's sub-commands to satisfy the facts:
* Some sub-commands shouldn't output any log, otherwise the git protocol
would be broken
* Sometimes the users want to see "verbose" or "quiet" outputs
That's a longstanding problem, and very fragile. This PR is only a quick
patch for the problem.
In the future, the sub-command system should be refactored to a clear
solution.
----
Other changes:
* Use `ReplaceAllWriters` to replace
`RemoveAllWriters().AddWriters(writer)`, then it's an atomic operation.
* Remove unnecessary `syncLevelInternal` calls, because
`AddWriters/addWritersInternal` already calls it.
Co-authored-by: Giteabot <teabot@gitea.io>
Fix #25088
This PR adds the support for
[`pull_request_target`](https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#pull_request_target)
workflow trigger. `pull_request_target` is similar to `pull_request`,
but the workflow triggered by the `pull_request_target` event runs in
the context of the base branch of the pull request rather than the head
branch. Since the workflow from the base is considered trusted, it can
access the secrets and doesn't need approvals to run.
Fix #25481
The `InitWorkPathAndCommonConfig` calls `LoadCommonSettings` which does
many checks like "current user is root or not".
Some commands like "environment-to-ini" shouldn't do such check, because
it might be run with "root" user at the moment (eg: the docker's setup
script)
ps: in the future, the docker's setup script should be improved to avoid
Gitea's command running with "root"
Fix regression of #25174.
The `Init` of the db indexer should return true to indicate that the
index was opened/existed, or the indexer will try to populate the index
(not really populate, just fill the queue, `Index` method of the db
indexer is a dummy).
Refactor `modules/indexer` to make it more maintainable. And it can be
easier to support more features. I'm trying to solve some of issue
searching, this is a precursor to making functional changes.
Current supported engines and the index versions:
| engines | issues | code |
| - | - | - |
| db | Just a wrapper for database queries, doesn't need version | - |
| bleve | The version of index is **2** | The version of index is **6**
|
| elasticsearch | The old index has no version, will be treated as
version **0** in this PR | The version of index is **1** |
| meilisearch | The old index has no version, will be treated as version
**0** in this PR | - |
## Changes
### Split
Splited it into mutiple packages
```text
indexer
├── internal
│ ├── bleve
│ ├── db
│ ├── elasticsearch
│ └── meilisearch
├── code
│ ├── bleve
│ ├── elasticsearch
│ └── internal
└── issues
├── bleve
├── db
├── elasticsearch
├── internal
└── meilisearch
```
- `indexer/interanal`: Internal shared package for indexer.
- `indexer/interanal/[engine]`: Internal shared package for each engine
(bleve/db/elasticsearch/meilisearch).
- `indexer/code`: Implementations for code indexer.
- `indexer/code/internal`: Internal shared package for code indexer.
- `indexer/code/[engine]`: Implementation via each engine for code
indexer.
- `indexer/issues`: Implementations for issues indexer.
### Deduplication
- Combine `Init/Ping/Close` for code indexer and issues indexer.
- ~Combine `issues.indexerHolder` and `code.wrappedIndexer` to
`internal.IndexHolder`.~ Remove it, use dummy indexer instead when the
indexer is not ready.
- Duplicate two copies of creating ES clients.
- Duplicate two copies of `indexerID()`.
### Enhancement
- [x] Support index version for elasticsearch issues indexer, the old
index without version will be treated as version 0.
- [x] Fix spell of `elastic_search/ElasticSearch`, it should be
`Elasticsearch`.
- [x] Improve versioning of ES index. We don't need `Aliases`:
- Gitea does't need aliases for "Zero Downtime" because it never delete
old indexes.
- The old code of issues indexer uses the orignal name to create issue
index, so it's tricky to convert it to an alias.
- [x] Support index version for meilisearch issues indexer, the old
index without version will be treated as version 0.
- [x] Do "ping" only when `Ping` has been called, don't ping
periodically and cache the status.
- [x] Support the context parameter whenever possible.
- [x] Fix outdated example config.
- [x] Give up the requeue logic of issues indexer: When indexing fails,
call Ping to check if it was caused by the engine being unavailable, and
only requeue the task if the engine is unavailable.
- It is fragile and tricky, could cause data losing (It did happen when
I was doing some tests for this PR). And it works for ES only.
- Just always requeue the failed task, if it caused by bad data, it's a
bug of Gitea which should be fixed.
---------
Co-authored-by: Giteabot <teabot@gitea.io>
this will allow us to fully localize it later
PS: we can not migrate back as the old value was a one-way conversion
prepare for #25213
---
*Sponsored by Kithara Software GmbH*
In modern days, there is no reason to make users set "charset" anymore.
Close #25378
## ⚠️ BREAKING
The key `[database].CHARSET` was removed completely as every newer
(>10years) MySQL database supports `utf8mb4` already.
There is a (deliberately) undocumented new fallback option if anyone
still needs to use it, but we don't recommend using it as it simply
causes problems.
# The problem
There were many "path tricks":
* By default, Gitea uses its program directory as its work path
* Gitea tries to use the "work path" to guess its "custom path" and
"custom conf (app.ini)"
* Users might want to use other directories as work path
* The non-default work path should be passed to Gitea by GITEA_WORK_DIR
or "--work-path"
* But some Gitea processes are started without these values
* The "serv" process started by OpenSSH server
* The CLI sub-commands started by site admin
* The paths are guessed by SetCustomPathAndConf again and again
* The default values of "work path / custom path / custom conf" can be
changed when compiling
# The solution
* Use `InitWorkPathAndCommonConfig` to handle these path tricks, and use
test code to cover its behaviors.
* When Gitea's web server runs, write the WORK_PATH to "app.ini", this
value must be the most correct one, because if this value is not right,
users would find that the web UI doesn't work and then they should be
able to fix it.
* Then all other sub-commands can use the WORK_PATH in app.ini to
initialize their paths.
* By the way, when Gitea starts for git protocol, it shouldn't output
any log, otherwise the git protocol gets broken and client blocks
forever.
The "work path" priority is: WORK_PATH in app.ini > cmd arg --work-path
> env var GITEA_WORK_DIR > builtin default
The "app.ini" searching order is: cmd arg --config > cmd arg "work path
/ custom path" > env var "work path / custom path" > builtin default
## ⚠️ BREAKING
If your instance's "work path / custom path / custom conf" doesn't meet
the requirements (eg: work path must be absolute), Gitea will report a
fatal error and exit. You need to set these values according to the
error log.
----
Close #24818
Close #24222
Close #21606
Close #21498
Close #25107
Close #24981
Maybe close #24503
Replace #23301
Replace #22754
And maybe more
That's a longstanding INI package problem: the "MustXxx" calls change
the option values, and the following "Save" will save a lot of garbage
options into the user's config file.
Ideally we should refactor the INI package to a clear solution, but it's
a huge work.
A clear workaround is what this PR does: when "Save", load a clear INI
instance and save it.
Partially fix #25377, the "install" page needs more fine tunes.
### Summary
Extend the template variable substitution to replace file paths. This
can be helpful for setting up log files & directories that should match
the repository name.
### PR Changes
- Move files matching glob pattern when setting up repos from template
- For security, added ~escaping~ sanitization for cross-platform support
and to prevent directory traversal (thanks @silverwind for the
reference)
- Added unit testing for escaping function
- Fixed the integration tests for repo template generation by passing
the repo_template_id
- Updated the integration testfiles to add some variable substitution &
assert the outputs
I had to fix the existing repo template integration test and extend it
to add a check for variable substitutions.
Example:
1. The "web" package shouldn't depends on "modules/context" package,
instead, let each "web context" register themselves to the "web"
package.
2. The old Init/Free doesn't make sense, so simplify it
* The ctx in "Init(ctx)" is never used, and shouldn't be used that way
* The "Free" is never called and shouldn't be called because the SSPI
instance is shared
---------
Co-authored-by: Giteabot <teabot@gitea.io>
Follow up #22405
Fix #20703
This PR rewrites storage configuration read sequences with some breaks
and tests. It becomes more strict than before and also fixed some
inherit problems.
- Move storage's MinioConfig struct into setting, so after the
configuration loading, the values will be stored into the struct but not
still on some section.
- All storages configurations should be stored on one section,
configuration items cannot be overrided by multiple sections. The
prioioty of configuration is `[attachment]` > `[storage.attachments]` |
`[storage.customized]` > `[storage]` > `default`
- For extra override configuration items, currently are `SERVE_DIRECT`,
`MINIO_BASE_PATH`, `MINIO_BUCKET`, which could be configured in another
section. The prioioty of the override configuration is `[attachment]` >
`[storage.attachments]` > `default`.
- Add more tests for storages configurations.
- Update the storage documentations.
---------
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
The plan is that all built-in auth providers use inline SVG for more
flexibility in styling and to get the GitHub icon to follow
`currentcolor`. This only removes the `public/img/auth` directory and
adds the missing svgs to our svg build.
It should map the built-in providers to these SVGs and render them. If
the user has set a Icon URL, it should render that as an `img` tag
instead.
```
gitea-azure-ad
gitea-bitbucket
gitea-discord
gitea-dropbox
gitea-facebook
gitea-gitea
gitea-gitlab
gitea-google
gitea-mastodon
gitea-microsoftonline
gitea-nextcloud
gitea-twitter
gitea-yandex
octicon-mark-github
```
GitHub logo is now white again on dark theme:
<img width="431" alt="Screenshot 2023-06-12 at 21 45 34"
src="https://github.com/go-gitea/gitea/assets/115237/27a43504-d60a-4132-a502-336b25883e4d">
---------
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
Fixes #25160.
`data-source-position` of checkboxes in a task list was incorrect
whenever there was YAML front matter. This would result in issue content
or PR descriptions getting corrupted with random `x` or space characters
when a user checked or unchecked a task.
If redirect_to parameter has set value starting with `\\example.com`
redirect will be created with header `Location: /\\example.com` that
will redirect to example.com domain.
Fixes some issues with the swagger documentation for the new multiple
files API endpoint (#24887) which were overlooked when submitting the
original PR:
1. add some missing parameter descriptions
2. set correct `required` option for required parameters
3. change endpoint description to match it full functionality (every
kind of file modification is supported, not just creating and updating)
## Changes
- Adds the following high level access scopes, each with `read` and
`write` levels:
- `activitypub`
- `admin` (hidden if user is not a site admin)
- `misc`
- `notification`
- `organization`
- `package`
- `issue`
- `repository`
- `user`
- Adds new middleware function `tokenRequiresScopes()` in addition to
`reqToken()`
- `tokenRequiresScopes()` is used for each high-level api section
- _if_ a scoped token is present, checks that the required scope is
included based on the section and HTTP method
- `reqToken()` is used for individual routes
- checks that required authentication is present (but does not check
scope levels as this will already have been handled by
`tokenRequiresScopes()`
- Adds migration to convert old scoped access tokens to the new set of
scopes
- Updates the user interface for scope selection
### User interface example
<img width="903" alt="Screen Shot 2023-05-31 at 1 56 55 PM"
src="https://github.com/go-gitea/gitea/assets/23248839/654766ec-2143-4f59-9037-3b51600e32f3">
<img width="917" alt="Screen Shot 2023-05-31 at 1 56 43 PM"
src="https://github.com/go-gitea/gitea/assets/23248839/1ad64081-012c-4a73-b393-66b30352654c">
## tokenRequiresScopes Design Decision
- `tokenRequiresScopes()` was added to more reliably cover api routes.
For an incoming request, this function uses the given scope category
(say `AccessTokenScopeCategoryOrganization`) and the HTTP method (say
`DELETE`) and verifies that any scoped tokens in use include
`delete:organization`.
- `reqToken()` is used to enforce auth for individual routes that
require it. If a scoped token is not present for a request,
`tokenRequiresScopes()` will not return an error
## TODO
- [x] Alphabetize scope categories
- [x] Change 'public repos only' to a radio button (private vs public).
Also expand this to organizations
- [X] Disable token creation if no scopes selected. Alternatively, show
warning
- [x] `reqToken()` is missing from many `POST/DELETE` routes in the api.
`tokenRequiresScopes()` only checks that a given token has the correct
scope, `reqToken()` must be used to check that a token (or some other
auth) is present.
- _This should be addressed in this PR_
- [x] The migration should be reviewed very carefully in order to
minimize access changes to existing user tokens.
- _This should be addressed in this PR_
- [x] Link to api to swagger documentation, clarify what
read/write/delete levels correspond to
- [x] Review cases where more than one scope is needed as this directly
deviates from the api definition.
- _This should be addressed in this PR_
- For example:
```go
m.Group("/users/{username}/orgs", func() {
m.Get("", reqToken(), org.ListUserOrgs)
m.Get("/{org}/permissions", reqToken(), org.GetUserOrgsPermissions)
}, tokenRequiresScopes(auth_model.AccessTokenScopeCategoryUser,
auth_model.AccessTokenScopeCategoryOrganization),
context_service.UserAssignmentAPI())
```
## Future improvements
- [ ] Add required scopes to swagger documentation
- [ ] Redesign `reqToken()` to be opt-out rather than opt-in
- [ ] Subdivide scopes like `repository`
- [ ] Once a token is created, if it has no scopes, we should display
text instead of an empty bullet point
- [ ] If the 'public repos only' option is selected, should read
categories be selected by default
Closes #24501
Closes #24799
Co-authored-by: Jonathan Tran <jon@allspice.io>
Co-authored-by: Kyle D <kdumontnu@gmail.com>
Co-authored-by: silverwind <me@silverwind.io>
Before, Gitea shows the database table stats on the `admin dashboard`
page.
It has some problems:
* `count(*)` is quite heavy. If tables have many records, this blocks
loading the admin page blocks for a long time
* Some users had even reported issues that they can't visit their admin
page because this page causes blocking or `50x error (reverse proxy
timeout)`
* The `actions` stat is not useful. The table is simply too large. Does
it really matter if it contains 1,000,000 rows or 9,999,999 rows?
* The translation `admin.dashboard.statistic_info` is difficult to
maintain.
So, this PR uses a separate page to show the stats and removes the
`actions` stat.
## ⚠️ BREAKING
The `actions` Prometheus metrics collector has been removed for the
reasons mentioned beforehand.
Please do not rely on its output anymore.
The INI package has many bugs and quirks, and in fact it is
unmaintained.
This PR is the first step for the INI package refactoring:
* Use Gitea's "config_provider" to provide INI access
* Deprecate the INI package by golangci.yml rule
It's been disabled by default since 1.17
(https://github.com/go-gitea/gitea/pull/18914), and it never really
delivered any benefit except being another cache layer that has its own
unsolved invalidation issues. HTTP cache works, we don't need two cache
layers at the browser for assets.
## ⚠️ BREAKING
You can remove the config `[ui].USE_SERVICE_WORKER` from your `app.ini`
now.
The admin config page has been broken for many many times, a little
refactoring would make this page panic.
So, add a test for it, and add another test to cover the 500 error page.
Co-authored-by: Giteabot <teabot@gitea.io>
This PR creates an API endpoint for creating/updating/deleting multiple
files in one API call similar to the solution provided by
[GitLab](https://docs.gitlab.com/ee/api/commits.html#create-a-commit-with-multiple-files-and-actions).
To archive this, the CreateOrUpdateRepoFile and DeleteRepoFIle functions
in files service are unified into one function supporting multiple files
and actions.
Resolves #14619
gitea.com experienced the corrupted LevelQueue bug again.
I think the problem is clear now: if the keys in LevelDB went
out-of-sync, the LevelQueue itself doesn't have the ability to recover,
eg:
* LevelQueue.Len() reports 100
* LevelQueue.LPop() reports ErrNotFound = errors.New("no key found")
So it needs to dive into the LevelDB to remove all keys to recover the
corrupted LevelQueue.
More comments are in TestCorruptedLevelQueue.
As discussed in
https://github.com/go-gitea/gitea/pull/24953#issuecomment-1565630156.
## ⚠️ BREAKING ⚠️
1. The `ui.THEME_COLOR_META_TAG` setting has been removed. If you still
need to set the `theme-color` meta tag, add it via
`$GITEA_CUSTOM/templates/custom/header.tmpl` instead.
2. The non-standard `default-theme` meta-tag added in
https://github.com/go-gitea/gitea/pull/13809 has been removed. Third
party code that needs to obtain the currently loaded theme should use
the `theme-<name>` class on the `<html>` node instead, which reflect the
currently active theme.
The old EventWriter's Run does:
```go
for {
handlePause()
select {
case event <- Queue:
write the log event ...
}
}
```
So, if an event writer is started before the logger is paused, there is
a chance that the logger isn't paused for the first message.
The new logic is:
```go
for {
select {
case event <- Queue:
handlePause()
write the log event ...
}
}
```
Then the event writer can be correctly paused
The old process manager's `Trace` function by default calls `log.Printf`
to output "trace" level logs. That's not ideal because by default the
trace level logs should not be outputted. In history it didn't cause
problems because there was no other call to the process manager before
the logger system's initialization.
But if there is any package using the process manager before the "Trace"
function gets assigned to the logger system's trace function, the
process manager will outputs unexpected verbose messages, this behavior
is not expected in most cases.
Now, the logger system also uses process manager to manage its goroutine
contexts, so it's the time to fix the old "trace" behavior: by default,
do not output the trace level messages. Fix #24951
Before there was a "graceful function": RunWithShutdownFns, it's mainly
for some modules which doesn't support context.
The old queue system doesn't work well with context, so the old queues
need it.
After the queue refactoring, the new queue works with context well, so,
use Golang context as much as possible, the `RunWithShutdownFns` could
be removed (replaced by RunWithCancel for context cancel mechanism), the
related code could be simplified.
This PR also fixes some legacy queue-init problems, eg:
* typo : archiver: "unable to create codes indexer queue" => "unable to
create repo-archive queue"
* no nil check for failed queues, which causes unfriendly panic
After this PR, many goroutines could have better display name:
This PR replaces all string refName as a type `git.RefName` to make the
code more maintainable.
Fix #15367
Replaces #23070
It also fixed a bug that tags are not sync because `git remote --prune
origin` will not remove local tags if remote removed.
We in fact should use `git fetch --prune --tags origin` but not `git
remote update origin` to do the sync.
Some answer from ChatGPT as ref.
> If the git fetch --prune --tags command is not working as expected,
there could be a few reasons why. Here are a few things to check:
>
>Make sure that you have the latest version of Git installed on your
system. You can check the version by running git --version in your
terminal. If you have an outdated version, try updating Git and see if
that resolves the issue.
>
>Check that your Git repository is properly configured to track the
remote repository's tags. You can check this by running git config
--get-all remote.origin.fetch and verifying that it includes
+refs/tags/*:refs/tags/*. If it does not, you can add it by running git
config --add remote.origin.fetch "+refs/tags/*:refs/tags/*".
>
>Verify that the tags you are trying to prune actually exist on the
remote repository. You can do this by running git ls-remote --tags
origin to list all the tags on the remote repository.
>
>Check if any local tags have been created that match the names of tags
on the remote repository. If so, these local tags may be preventing the
git fetch --prune --tags command from working properly. You can delete
local tags using the git tag -d command.
---------
Co-authored-by: delvh <dev.lh@web.de>
This adds the ability to pin important Issues and Pull Requests. You can
also move pinned Issues around to change their Position. Resolves #2175.
## Screenshots
The Design was mostly copied from the Projects Board.
## Implementation
This uses a new `pin_order` Column in the `issue` table. If the value is
set to 0, the Issue is not pinned. If it's set to a bigger value, the
value is the Position. 1 means it's the first pinned Issue, 2 means it's
the second one etc. This is dived into Issues and Pull requests for each
Repo.
## TODO
- [x] You can currently pin as many Issues as you want. Maybe we should
add a Limit, which is configurable. GitHub uses 3, but I prefer 6, as
this is better for bigger Projects, but I'm open for suggestions.
- [x] Pin and Unpin events need to be added to the Issue history.
- [x] Tests
- [x] Migration
**The feature itself is currently fully working, so tester who may find
weird edge cases are very welcome!**
---------
Co-authored-by: silverwind <me@silverwind.io>
Co-authored-by: Giteabot <teabot@gitea.io>
1. non-dev mode is treated as prod mode, to protect users from
accidentally running in dev mode if there is a typo in this value.
2. in dev mode, do not need to really exit if there are template errors,
because the template errors could be fixed by developer soon and the
templates get reloaded, help:
* https://github.com/go-gitea/gitea/issues/24845#issuecomment-1557615382
3. Fine tune the mail template loading message.
close https://github.com/go-gitea/gitea/issues/16321
Provided a webhook trigger for requesting someone to review the Pull
Request.
Some modifications have been made to the returned `PullRequestPayload`
based on the GitHub webhook settings, including:
- add a description of the current reviewer object as
`RequestedReviewer` .
- setting the action to either **review_requested** or
**review_request_removed** based on the operation.
- adding the `RequestedReviewers` field to the issues_model.PullRequest.
This field will be loaded into the PullRequest through
`LoadRequestedReviewers()` when `ToAPIPullRequest` is called.
After the Pull Request is merged, I will supplement the relevant
documentation.
Fix #24896
If users set different languages by `linguist-language`, the `stats` map
could be: `java: 100, Java: 200`.
Language stats are stored as case-insensitive in database and there is a
unique key.
So, the different language names should be merged to one unique name:
`Java: 300`
Replace #19857
Close #19856
Close #10311
Close #10123
Major changes:
1. Move a lot of code from `environment-to-ini.go` to `config_env.go` to
make them testable.
2. Add `__FILE` support
3. Update documents
4. Add tests
Close #13454 , Close #23255, Close #14697 (and maybe more related
issues)
Many users have the requirement to customize the git config. This PR
introduces an easy way: put the options in Gitea's app.ini
`[git.config]`, then the config options will be applied to git config.
And it can support more flexible default config values, eg: now
`diff.algorithm=histogram` by default. According to:
https://stackoverflow.com/a/32367597/4754037 , `histogram diff` is
efficient and doesn't like to cause server-side problems.
---------
Co-authored-by: silverwind <me@silverwind.io>
Co-authored-by: KN4CK3R <admin@oldschoolhack.me>
Co-authored-by: Giteabot <teabot@gitea.io>
Fix #24882
The goroutines are all asynchronized. So it needs a little "sleep" to
make sure the writer's goroutine has been paused before sending messages
to it.
Co-authored-by: Giteabot <teabot@gitea.io>
Fix #24856
Rename "context.contextKey" to "context.WebContextKey", this context is
for web context only. But the Context itself is not renamed, otherwise
it would cause a lot of changes (if we really want to rename it, there
could be a separate PR).
The old test code doesn't really test, the "install page" gets broken
not only one time, so use new test code to make sure the "install page"
could work.
Replace #20257 (which is stale and incomplete)
Close #20255
Major changes:
* Deprecate the "WHITELIST", use "ALLOWLIST"
* Add wildcard support for EMAIL_DOMAIN_ALLOWLIST/EMAIL_DOMAIN_BLOCKLIST
* Update example config file and document
* Improve tests
## ⚠️ Breaking
The `log.<mode>.<logger>` style config has been dropped. If you used it,
please check the new config manual & app.example.ini to make your
instance output logs as expected.
Although many legacy options still work, it's encouraged to upgrade to
the new options.
The SMTP logger is deleted because SMTP is not suitable to collect logs.
If you have manually configured Gitea log options, please confirm the
logger system works as expected after upgrading.
## Description
Close #12082 and maybe more log-related issues, resolve some related
FIXMEs in old code (which seems unfixable before)
Just like rewriting queue #24505 : make code maintainable, clear legacy
bugs, and add the ability to support more writers (eg: JSON, structured
log)
There is a new document (with examples): `logging-config.en-us.md`
This PR is safer than the queue rewriting, because it's just for
logging, it won't break other logic.
## The old problems
The logging system is quite old and difficult to maintain:
* Unclear concepts: Logger, NamedLogger, MultiChannelledLogger,
SubLogger, EventLogger, WriterLogger etc
* Some code is diffuclt to konw whether it is right:
`log.DelNamedLogger("console")` vs `log.DelNamedLogger(log.DEFAULT)` vs
`log.DelLogger("console")`
* The old system heavily depends on ini config system, it's difficult to
create new logger for different purpose, and it's very fragile.
* The "color" trick is difficult to use and read, many colors are
unnecessary, and in the future structured log could help
* It's difficult to add other log formats, eg: JSON format
* The log outputer doesn't have full control of its goroutine, it's
difficult to make outputer have advanced behaviors
* The logs could be lost in some cases: eg: no Fatal error when using
CLI.
* Config options are passed by JSON, which is quite fragile.
* INI package makes the KEY in `[log]` section visible in `[log.sub1]`
and `[log.sub1.subA]`, this behavior is quite fragile and would cause
more unclear problems, and there is no strong requirement to support
`log.<mode>.<logger>` syntax.
## The new design
See `logger.go` for documents.
## Screenshot
<details>
</details>
## TODO
* [x] add some new tests
* [x] fix some tests
* [x] test some sub-commands (manually ....)
---------
Co-authored-by: Jason Song <i@wolfogre.com>
Co-authored-by: delvh <dev.lh@web.de>
Co-authored-by: Giteabot <teabot@gitea.io>
This change makes the CSS for `<video>` in markup match that of `<img>`,
and also allows additional attributes to be used. This way the width,
padding, alignment should work equally well for both.
Replace #16455
Close #21803
Mixing different Gitea contexts together causes some problems:
1. Unable to respond proper content when error occurs, eg: Web should
respond HTML while API should respond JSON
2. Unclear dependency, eg: it's unclear when Context is used in
APIContext, which fields should be initialized, which methods are
necessary.
To make things clear, this PR introduces a Base context, it only
provides basic Req/Resp/Data features.
This PR mainly moves code. There are still many legacy problems and
TODOs in code, leave unrelated changes to future PRs.
This PR
- [x] Move some functions from `issues.go` to `issue_stats.go` and
`issue_label.go`
- [x] Remove duplicated issue options `UserIssueStatsOption` to keep
only one `IssuesOptions`
Regression of #24536. If the user doesn't explicitly disable Actions, it
will be enabled.
1. Gitea will call `loadRepositoryFrom` before `loadActionsFrom`.
25d4f95df2/modules/setting/setting.go (L234-L237)
2. In `loadRepositoryFrom`,
`rootCfg.Section("actions").Key("ENABLED").MustBool(true)` will set
`actions.ENABLED` with `true`.
25d4f95df2/modules/setting/repository.go (L313-L315)
3. In `loadActionsFrom`, `rootCfg.Section("actions")` will get a section
with Actions enabled.
25d4f95df2/modules/setting/actions.go (L23-L26)
Although the cause of the problem was using `true` by copy-paste
mistake, it also surprised me that
**`rootCfg.Section("actions").Key("ENABLED").MustBool(true)` doesn't
only read, but also write.**
Make mailer SMTP check have timed context
Otherwise Gitea may block for long time if the DNS request blocks.
---------
Co-authored-by: Giteabot <teabot@gitea.io>
This PR adds support for the systemd notify protocol. Several status
messagess are provided. We should likely add a common notify/status
message for graceful.
Replaces #21140
Signed-off-by: Andrew Thornton <art27@cantab.net>
---------
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: ltdk <usr@ltdk.xyz>
Co-authored-by: Giteabot <teabot@gitea.io>
minio storage iterator shows different behavior with local fs iterator.
in local fs storage:
``` go
s.IterateObjects("prefix", func(path,obj)
println(path) // show "prefix/xxx.file"
})
```
in minio storage:
```go
s.IterateObjects("prefix", func(path,obj)
println(path) // show "xxx.file"
})
```
I think local fs is correct, minio use wrong `basePath` to trim storage
path prefix.
---------
Co-authored-by: Giteabot <teabot@gitea.io>
`http.ServeFile/ServeContent` handles `If-xxx`, `Content-Length`,
`Range` and `Etag` correctly
After this PR, storage files (eg: avatar) could be responded with
correct Content-Length.
Although some features are mixed together in this PR, this PR is not
that large, and these features are all related.
Actually there are more than 70 lines are for a toy "test queue", so
this PR is quite simple.
Major features:
1. Allow site admin to clear a queue (remove all items in a queue)
* Because there is no transaction, the "unique queue" could be corrupted
in rare cases, that's unfixable.
* eg: the item is in the "set" but not in the "list", so the item would
never be able to be pushed into the queue.
* Now site admin could simply clear the queue, then everything becomes
correct, the lost items could be re-pushed into queue by future
operations.
3. Split the "admin/monitor" to separate pages
4. Allow to download diagnosis report
* In history, there were many users reporting that Gitea queue gets
stuck, or Gitea's CPU is 100%
* With diagnosis report, maintainers could know what happens clearly
The diagnosis report sample:
[gitea-diagnosis-20230510-192913.zip](https://github.com/go-gitea/gitea/files/11441346/gitea-diagnosis-20230510-192913.zip)
, use "go tool pprof profile.dat" to view the report.
Screenshots:
---------
Co-authored-by: Jason Song <i@wolfogre.com>
Co-authored-by: Giteabot <teabot@gitea.io>
Replaces #24641
Currently, unit tests fail when run locally (unless users have minio
instance running). This PR only requires redis unit tests if in CI.
- Only run redis unit tests when `CI` env variable is set
- Add minio as a service in unit tests actions
According to the discussion with DanielGibson, the default "20m" seems
too short. It would make LFS fail if the file is large / network is
slow.
I think relaxing this timeout doesn't have side affect. So change the
default value to 24h, IMO that should be long enough.
## ⚠️ BREAKING
If admins want the previous timeout, they should set the setting
`[server].LFS_HTTP_AUTH_EXPIRY`.
#### Added
- API: Create a branch directly from commit on the create branch API
- Added `old_ref_name` parameter to allow creating a new branch from a
specific commit, tag, or branch.
- Deprecated `old_branch_name` parameter in favor of the new
`old_ref_name` parameter.
---------
Co-authored-by: silverwind <me@silverwind.io>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
1. Remove unused fields/methods in web context.
2. Make callers call target function directly instead of the light
wrapper like "IsUserRepoReaderSpecific"
3. The "issue template" code shouldn't be put in the "modules/context"
package, so move them to the service package.
---------
Co-authored-by: Giteabot <teabot@gitea.io>
# ⚠️ Breaking
Many deprecated queue config options are removed (actually, they should
have been removed in 1.18/1.19).
If you see the fatal message when starting Gitea: "Please update your
app.ini to remove deprecated config options", please follow the error
messages to remove these options from your app.ini.
Example:
```
2023/05/06 19:39:22 [E] Removed queue option: `[indexer].ISSUE_INDEXER_QUEUE_TYPE`. Use new options in `[queue.issue_indexer]`
2023/05/06 19:39:22 [E] Removed queue option: `[indexer].UPDATE_BUFFER_LEN`. Use new options in `[queue.issue_indexer]`
2023/05/06 19:39:22 [F] Please update your app.ini to remove deprecated config options
```
Many options in `[queue]` are are dropped, including:
`WRAP_IF_NECESSARY`, `MAX_ATTEMPTS`, `TIMEOUT`, `WORKERS`,
`BLOCK_TIMEOUT`, `BOOST_TIMEOUT`, `BOOST_WORKERS`, they can be removed
from app.ini.
# The problem
The old queue package has some legacy problems:
* complexity: I doubt few people could tell how it works.
* maintainability: Too many channels and mutex/cond are mixed together,
too many different structs/interfaces depends each other.
* stability: due to the complexity & maintainability, sometimes there
are strange bugs and difficult to debug, and some code doesn't have test
(indeed some code is difficult to test because a lot of things are mixed
together).
* general applicability: although it is called "queue", its behavior is
not a well-known queue.
* scalability: it doesn't seem easy to make it work with a cluster
without breaking its behaviors.
It came from some very old code to "avoid breaking", however, its
technical debt is too heavy now. It's a good time to introduce a better
"queue" package.
# The new queue package
It keeps using old config and concept as much as possible.
* It only contains two major kinds of concepts:
* The "base queue": channel, levelqueue, redis
* They have the same abstraction, the same interface, and they are
tested by the same testing code.
* The "WokerPoolQueue", it uses the "base queue" to provide "worker
pool" function, calls the "handler" to process the data in the base
queue.
* The new code doesn't do "PushBack"
* Think about a queue with many workers, the "PushBack" can't guarantee
the order for re-queued unhandled items, so in new code it just does
"normal push"
* The new code doesn't do "pause/resume"
* The "pause/resume" was designed to handle some handler's failure: eg:
document indexer (elasticsearch) is down
* If a queue is paused for long time, either the producers blocks or the
new items are dropped.
* The new code doesn't do such "pause/resume" trick, it's not a common
queue's behavior and it doesn't help much.
* If there are unhandled items, the "push" function just blocks for a
few seconds and then re-queue them and retry.
* The new code doesn't do "worker booster"
* Gitea's queue's handlers are light functions, the cost is only the
go-routine, so it doesn't make sense to "boost" them.
* The new code only use "max worker number" to limit the concurrent
workers.
* The new "Push" never blocks forever
* Instead of creating more and more blocking goroutines, return an error
is more friendly to the server and to the end user.
There are more details in code comments: eg: the "Flush" problem, the
strange "code.index" hanging problem, the "immediate" queue problem.
Almost ready for review.
TODO:
* [x] add some necessary comments during review
* [x] add some more tests if necessary
* [x] update documents and config options
* [x] test max worker / active worker
* [x] re-run the CI tasks to see whether any test is flaky
* [x] improve the `handleOldLengthConfiguration` to provide more
friendly messages
* [x] fine tune default config values (eg: length?)
## Code coverage:
The "modules/context.go" is too large to maintain.
This PR splits it to separate files, eg: context_request.go,
context_response.go, context_serve.go
This PR will help:
1. The future refactoring for Gitea's web context (eg: simplify the context)
2. Introduce proper "range request" support
3. Introduce context function
This PR only moves code, doesn't change any logic.
Due to #24409 , we can now specify '--not' when getting all commits from
a repo to exclude commits from a different branch.
When I wrote that PR, I forgot to also update the code that counts the
number of commits in the repo. So now, if the --not option is used, it
may return too many commits, which can indicate that another page of
data is available when it is not.
This PR passes --not to the commands that count the number of commits in
a repo
Help some users like #16832#1851
There are many users reporting similar problem: if the SECRET_KEY
mismatches, some operations (like 2FA login) only reports unclear 500
error and unclear "base64 decode error" log （some maintainers ever spent
a lot of time on debugging such problem)
The SECRET_KEY was not well-designed and it is also a kind of technical
debt. Since it couldn't be fixed easily, it's good to add clearer error
messages, then at least users could know what the real problem is.
---------
Co-authored-by: delvh <dev.lh@web.de>
This gives more "freshness" to the explore page. So it's not just the
same X users on the explore page by default, now it matches the same
sort as the repos on the explore page.
---------
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Replace #22117. Implement it in a more maintainable way.
Some licenses have placeholders e.g. the BSD licenses start with this
line:
```
Copyright (c) <year> <owner>.
```
This PR replaces the placeholders with the correct value when initialize
a new repo.
### FAQ
- Why not use a regex?
It will be a pretty complicated regex which could be hard to maintain.
- There're still missing placeholders.
There are over 500 licenses, it's impossible for anyone to inspect all
of them alone. Please help to add them if you find any, and it is also
OK to leave them for the future.
---------
Co-authored-by: Giteabot <teabot@gitea.io>
close #24449
The unit of `Actions` should be contorlled not only by
`repository.DISABLED_REPO_UNITS` but also by `actions.ENABLED`
in the `app.ini`.
Previously, the permission of the team's `Actions` unit was not
controlled by `actions.Enabled`. So, even if the user sets
`actions.Enabled` to false, he can still select the permission of the
`Actions` unit for the team.
This PR makes the permissions of the team's `Actions` unit also
controlled by `actions.Enabled`. Just append`TypeActions` into
`DisabledRepoUnits` slice when initializing if `actions.Enabled` is
false.
### Changes:
If `Actions` is set disbaled in `app.ini`, like below:
```yaml
[actions]
ENABLED = false
```
1. If user try to create/edit a team, will prompt user that `Actions` is disabled.
2. `actions` is not displayed in the sidebar on the team details page
Partially for #24457
Major changes:
1. The old `signedUserNameStringPointerKey` is quite hacky, use
`ctx.Data[SignedUser]` instead
2. Move duplicate code from `Contexter` to `CommonTemplateContextData`
3. Remove incorrect copying&pasting code `ctx.Data["Err_Password"] =
true` in API handlers
4. Use one unique `RenderPanicErrorPage` for panic error page rendering
5. Move `stripSlashesMiddleware` to be the first middleware
6. Install global panic recovery handler, it works for both `install`
and `web`
7. Make `500.tmpl` only depend minimal template functions/variables,
avoid triggering new panics
Screenshot:
<details>
</details>
The ordering of the final token causing a close of the queue in this
test may be out of sync due to concurrency. Instead just use ensure that
the queue is closed when everything expected is done.
Fixes: https://github.com/go-gitea/gitea/issues/23608
Fixes: https://github.com/go-gitea/gitea/issues/23977
---------
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: KN4CK3R <admin@oldschoolhack.me>
The old `HTMLRender` is not ideal.
1. It shouldn't be initialized multiple times, it consumes a lot of
memory and is slow.
2. It shouldn't depend on short-lived requests, the `WatchLocalChanges`
needs a long-running context.
3. It doesn't make sense to use FuncsMap slice.
HTMLRender was designed to only work for GItea's specialized 400+
templates, so it's good to make it a global shared instance.
For my specific use case, I'd like to get all commits that are on one
branch but NOT on the other branch.
For instance, I'd like to get all the commits on `Branch1` that are not
also on `master` (I.e. all commits that were made after `Branch1` was
created).
This PR adds a `not` query param that gets passed down to the `git log`
command to allow the user to exclude items from `GetAllCommits`.
See [git
documentation](https://git-scm.com/docs/git-log#Documentation/git-log.txt---not)
---------
Co-authored-by: Giteabot <teabot@gitea.io>
Co-authored-by: @awkwardbunny
This PR adds a Debian package registry. You can follow [this
tutorial](https://www.baeldung.com/linux/create-debian-package) to build
a *.deb package for testing. Source packages are not supported at the
moment and I did not find documentation of the architecture "all" and
how these packages should be treated.
---------
Co-authored-by: Brian Hong <brian@hongs.me>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
There was only one `IsRepositoryExist` function, it did: `has && isDir`
However it's not right, and it would cause 500 error when creating a new
repository if the dir exists.
Then, it was changed to `has || isDir`, it is still incorrect, it
affects the "adopt repo" logic.
To make the logic clear:
* IsRepositoryModelOrDirExist
* IsRepositoryModelExist
> The scoped token PR just checked all API routes but in fact, some web
routes like `LFS`, git `HTTP`, container, and attachments supports basic
auth. This PR added scoped token check for them.
---------
Signed-off-by: jolheiser <john.olheiser@gmail.com>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
## Needs and benefits
[Livebook](https://livebook.dev/) notebooks are used for code
documentation and for deep dives and note-taking in the elixir
ecosystem. Rendering these in these as Markdown on frogejo has many
benefits, since livemd is a subset of markdown. Some of the benefits
are:
- New users of elixir and livebook are scared by unformated .livemd
files, but are shown what they expect
- Sharing a notebook is as easy as sharing a link, no need to install
the software in order to see the results.
[goldmark-meraid ](https://github.com/abhinav/goldmark-mermaid) is a
mermaid-js parser already included in gitea. This makes the .livemd
rendering integration feature complete. With this PR class diagrams, ER
Diagrams, flow charts and much more will be rendered perfectly.
With the additional functionality gitea will be an ideal tool for
sharing resources with fellow software engineers working in the elixir
ecosystem. Allowing the git forge to be used without needing to install
any software.
## Feature Description
This issue requests the .livemd extension to be added as a Markdown
language extension.
- `.livemd` is the extension of Livebook which is an Elixir version of
Jupyter Notebook.
- `.livemd` is` a subset of Markdown.
This would require the .livemd to be recognized as a markdown file. The
Goldmark the markdown parser should handle the parsing and rendering
automatically.
Here is the corresponding commit for GitHub linguist:
https://github.com/github/linguist/pull/5672
Here is a sample page of a livemd file:
https://github.com/github/linguist/blob/master/samples/Markdown/livebook.livemd
## Screenshots
The first screenshot shows how github shows the sample .livemd in the
browser.
The second screenshot shows how mermaid js, renders my development
notebook and its corresponding ER Diagram. The source code can be found
here:
79615f7428/termiNotes.livemd
## Testing
I just changed the file extension from `.livemd`to `.md`and the document
already renders perfectly on codeberg. Check you can it out
[here](https://codeberg.org/lgh/Termi/src/branch/livemd2md/termiNotes.md)
---------
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
After #24317 this function is only used in one place where it is not
needed. I confirmed the timestamp still renders correctly
Signed-off-by: Yarden Shoham <git@yardenshoham.com>
This adds a API for getting License templates. This tries to be as close
to the [GitHub
API](https://docs.github.com/en/rest/licenses?apiVersion=2022-11-28) as
possible, but Gitea does not support all features that GitHub has. I
think they should been added, but this out f the scope of this PR. You
should merge #23006 before this PR for security reasons.
Enable [forbidigo](https://github.com/ashanbrown/forbidigo) linter which
forbids print statements. Will check how to integrate this with the
smallest impact possible, so a few `nolint` comments will likely be
required. Plan is to just go through the issues and either:
- Remove the print if it is nonsensical
- Add a `//nolint` directive if it makes sense
I don't plan on investigating the individual issues any further.
<details>
<summary>Initial Lint Results</summary>
```
modules/log/event.go:348:6: use of `fmt.Println` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)
fmt.Println(err)
^
modules/log/event.go:382:6: use of `fmt.Println` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)
fmt.Println(err)
^
modules/queue/unique_queue_disk_channel_test.go:20:2: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)
fmt.Printf("TempDir %s\n", tmpDir)
^
contrib/backport/backport.go:168:2: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)
fmt.Printf("* Backporting %s to %s as %s\n", pr, localReleaseBranch, backportBranch)
^
contrib/backport/backport.go:216:4: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)
fmt.Printf("* Navigate to %s to open PR\n", url)
^
contrib/backport/backport.go:223:2: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)
fmt.Printf("* `xdg-open %s`\n", url)
^
contrib/backport/backport.go:233:2: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)
fmt.Printf("* `git push -u %s %s`\n", remote, backportBranch)
^
contrib/backport/backport.go:243:2: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)
fmt.Printf("* Amending commit to prepend `Backport #%s` to body\n", pr)
^
contrib/backport/backport.go:272:3: use of `fmt.Println` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)
fmt.Println("* Attempting git cherry-pick --continue")
^
contrib/backport/backport.go:281:2: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)
fmt.Printf("* Attempting git cherry-pick %s\n", sha)
^
contrib/backport/backport.go:297:2: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)
fmt.Printf("* Current branch is %s\n", currentBranch)
^
contrib/backport/backport.go:299:3: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)
fmt.Printf("* Current branch is %s - not checking out\n", currentBranch)
^
contrib/backport/backport.go:304:3: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)
fmt.Printf("* Branch %s already exists. Checking it out...\n", backportBranch)
^
contrib/backport/backport.go:308:2: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)
fmt.Printf("* `git checkout -b %s %s`\n", backportBranch, releaseBranch)
^
contrib/backport/backport.go:313:2: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)
fmt.Printf("* `git fetch %s main`\n", remote)
^
contrib/backport/backport.go:316:3: use of `fmt.Println` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)
fmt.Println(string(out))
^
contrib/backport/backport.go:319:2: use of `fmt.Println` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)
fmt.Println(string(out))
^
contrib/backport/backport.go:321:2: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)
fmt.Printf("* `git fetch %s %s`\n", remote, releaseBranch)
^
contrib/backport/backport.go:324:3: use of `fmt.Println` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)
fmt.Println(string(out))
^
contrib/backport/backport.go:327:2: use of `fmt.Println` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)
fmt.Println(string(out))
^
models/unittest/fixtures.go:50:3: use of `fmt.Println` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)
fmt.Println("Unsupported RDBMS for integration tests")
^
models/unittest/fixtures.go:89:3: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)
fmt.Printf("LoadFixtures failed after retries: %v\n", err)
^
models/unittest/fixtures.go:110:4: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)
fmt.Printf("Failed to generate sequence update: %v\n", err)
^
models/unittest/fixtures.go:117:6: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)
fmt.Printf("Failed to update sequence: %s Error: %v\n", value, err)
^
models/migrations/base/tests.go:118:3: use of `fmt.Println` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)
fmt.Println("Environment variable $GITEA_ROOT not set")
^
models/migrations/base/tests.go:127:3: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)
fmt.Printf("Could not find gitea binary at %s\n", setting.AppPath)
^
models/migrations/base/tests.go:134:3: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)
fmt.Printf("Environment variable $GITEA_CONF not set - defaulting to %s\n", giteaConf)
^
models/migrations/base/tests.go:145:3: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)
fmt.Printf("Unable to create temporary data path %v\n", err)
^
models/migrations/base/tests.go:154:3: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)
fmt.Printf("Unable to InitFull: %v\n", err)
^
models/migrations/v1_11/v112.go:34:5: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)
fmt.Printf("Error: %v", err)
^
contrib/fixtures/fixture_generation.go:36:3: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)
fmt.Printf("CreateTestEngine: %+v", err)
^
contrib/fixtures/fixture_generation.go:40:3: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)
fmt.Printf("PrepareTestDatabase: %+v\n", err)
^
contrib/fixtures/fixture_generation.go:46:5: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)
fmt.Printf("generate '%s': %+v\n", r, err)
^
contrib/fixtures/fixture_generation.go:53:5: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)
fmt.Printf("generate '%s': %+v\n", g.name, err)
^
contrib/fixtures/fixture_generation.go:71:4: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)
fmt.Printf("%s created.\n", path)
^
services/gitdiff/gitdiff_test.go:543:2: use of `println` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)
println(result)
^
services/gitdiff/gitdiff_test.go:560:2: use of `println` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)
println(result)
^
services/gitdiff/gitdiff_test.go:577:2: use of `println` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)
println(result)
^
modules/web/routing/logger_manager.go:34:2: use of `print` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)
print Printer
^
modules/doctor/paths.go:109:3: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)
fmt.Printf("Warning: can't remove temporary file: '%s'\n", tmpFile.Name())
^
tests/test_utils.go:33:2: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)
fmt.Printf(format+"\n", args...)
^
tests/test_utils.go:61:3: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)
fmt.Printf("Environment variable $GITEA_CONF not set, use default: %s\n", giteaConf)
^
cmd/actions.go:54:9: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)
_, _ = fmt.Printf("%s\n", respText)
^
cmd/admin_user_change_password.go:74:2: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)
fmt.Printf("%s's password has been successfully updated!\n", user.Name)
^
cmd/admin_user_create.go:109:3: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)
fmt.Printf("generated random password is '%s'\n", password)
^
cmd/admin_user_create.go:164:3: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)
fmt.Printf("Access token was successfully created... %s\n", t.Token)
^
cmd/admin_user_create.go:167:2: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)
fmt.Printf("New user '%s' has been successfully created!\n", username)
^
cmd/admin_user_generate_access_token.go:74:3: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)
fmt.Printf("%s\n", t.Token)
^
cmd/admin_user_generate_access_token.go:76:3: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)
fmt.Printf("Access token was successfully created: %s\n", t.Token)
^
cmd/admin_user_must_change_password.go:56:2: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)
fmt.Printf("Updated %d users setting MustChangePassword to %t\n", n, mustChangePassword)
^
cmd/convert.go:44:3: use of `fmt.Println` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)
fmt.Println("Converted successfully, please confirm your database's character set is now utf8mb4")
^
cmd/convert.go:50:3: use of `fmt.Println` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)
fmt.Println("Converted successfully, please confirm your database's all columns character is NVARCHAR now")
^
cmd/convert.go:52:3: use of `fmt.Println` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)
fmt.Println("This command can only be used with a MySQL or MSSQL database")
^
cmd/doctor.go:104:3: use of `fmt.Println` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)
fmt.Println(err)
^
cmd/doctor.go:105:3: use of `fmt.Println` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)
fmt.Println("Check if you are using the right config file. You can use a --config directive to specify one.")
^
cmd/doctor.go:243:3: use of `fmt.Println` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)
fmt.Println(err)
^
cmd/embedded.go:154:3: use of `fmt.Println` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)
fmt.Println(a.path)
^
cmd/embedded.go:198:3: use of `fmt.Println` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)
fmt.Println("Using app.ini at", setting.CustomConf)
^
cmd/embedded.go:217:2: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)
fmt.Printf("Extracting to %s:\n", destdir)
^
cmd/embedded.go:253:3: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)
fmt.Printf("%s already exists; skipped.\n", dest)
^
cmd/embedded.go:275:2: use of `fmt.Println` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)
fmt.Println(dest)
^
cmd/generate.go:63:2: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)
fmt.Printf("%s", internalToken)
^
cmd/generate.go:66:3: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)
fmt.Printf("\n")
^
cmd/generate.go:78:2: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)
fmt.Printf("%s", JWTSecretBase64)
^
cmd/generate.go:81:3: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)
fmt.Printf("\n")
^
cmd/generate.go:93:2: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)
fmt.Printf("%s", secretKey)
^
cmd/generate.go:96:3: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)
fmt.Printf("\n")
^
cmd/keys.go:74:2: use of `fmt.Println` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)
fmt.Println(strings.TrimSpace(authorizedString))
^
cmd/mailer.go:32:4: use of `fmt.Print` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)
fmt.Print("warning: Content is empty")
^
cmd/mailer.go:35:3: use of `fmt.Print` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)
fmt.Print("Proceed with sending email? [Y/n] ")
^
cmd/mailer.go:40:4: use of `fmt.Println` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)
fmt.Println("The mail was not sent")
^
cmd/mailer.go:49:9: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)
_, _ = fmt.Printf("Sent %s email(s) to all users\n", respText)
^
cmd/serv.go:147:3: use of `println` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)
println("Gitea: SSH has been disabled")
^
cmd/serv.go:153:4: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)
fmt.Printf("error showing subcommand help: %v\n", err)
^
cmd/serv.go:175:4: use of `println` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)
println("Hi there! You've successfully authenticated with the deploy key named " + key.Name + ", but Gitea does not provide shell access.")
^
cmd/serv.go:177:4: use of `println` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)
println("Hi there! You've successfully authenticated with the principal " + key.Content + ", but Gitea does not provide shell access.")
^
cmd/serv.go:179:4: use of `println` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)
println("Hi there, " + user.Name + "! You've successfully authenticated with the key named " + key.Name + ", but Gitea does not provide shell access.")
^
cmd/serv.go:181:3: use of `println` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)
println("If this is unexpected, please log in with password and setup Gitea under another user.")
^
cmd/serv.go:196:5: use of `fmt.Print` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)
fmt.Print(`{"type":"gitea","version":1}`)
^
tests/e2e/e2e_test.go:54:3: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)
fmt.Printf("Error initializing test database: %v\n", err)
^
tests/e2e/e2e_test.go:63:3: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)
fmt.Printf("util.RemoveAll: %v\n", err)
^
tests/e2e/e2e_test.go:67:3: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)
fmt.Printf("Unable to remove repo indexer: %v\n", err)
^
tests/e2e/e2e_test.go:109:6: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)
fmt.Printf("%v", stdout.String())
^
tests/e2e/e2e_test.go:110:6: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)
fmt.Printf("%v", stderr.String())
^
tests/e2e/e2e_test.go:113:6: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)
fmt.Printf("%v", stdout.String())
^
tests/integration/integration_test.go:124:3: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)
fmt.Printf("Error initializing test database: %v\n", err)
^
tests/integration/integration_test.go:135:3: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)
fmt.Printf("util.RemoveAll: %v\n", err)
^
tests/integration/integration_test.go:139:3: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)
fmt.Printf("Unable to remove repo indexer: %v\n", err)
^
tests/integration/repo_test.go:357:4: use of `fmt.Printf` forbidden by pattern `^(fmt\.Print(|f|ln)|print|println)$` (forbidigo)
fmt.Printf("%s", resp.Body)
^
```
</details>
---------
Co-authored-by: Giteabot <teabot@gitea.io>
This refactors the `shared/datetime/short|long|full` templates into a
template helper function, which allows us to render absolute date times
within translatable phrases.
- Follows #23988
- The first attempt was in #24055
- This should help #22664
Changes:
1. Added the `DateTime` template helper that replaces the
`shared/datetime/short|long|full` templates
2. Used find-and-replace with varying regexes to replace the templates
from step 1 (for example, `\{\{template "shared/datetime/(\S+) \(dict
"Datetime" ([^"]+) "Fallback" ([^\)]+\)?) ?\)?\}\}` -> `{{DateTime "$1
$2 $3}}`)
3. Used the new `DateTime` helper in the issue due date timestamp
rendering
# Before
# After
---------
Signed-off-by: Yarden Shoham <git@yardenshoham.com>
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
The `SHOW_FOOTER_BRANDING` came from year 2015, and it seems nobody ever
uses it. It only shows an GitHub icon which seems unrelated to Gitea, it
doesn't do what document says. So, remove it.
## ⚠️ Breaking
Users can now remove the key `[other].SHOW_FOOTER_BRANDING` from their
app.ini.
- Update all tool dependencies to latest tag
- Remove unused errcheck, it is part of golangci-lint
- Include main.go in air
- Enable wastedassign again now that it's
[generics-compatible](https://github.com/golangci/golangci-lint/pull/3689)
- Restructured lint targets to new `lint-*` namespace
Follow #23328
The improvements:
1. The `contains` functions are covered by tests
2. The inconsistent behavior of `containGeneric` is replaced by
`StringUtils.Contains` and `SliceUtils.Contains`
3. In the future we can move more help functions into XxxUtils to
simplify the `helper.go` and reduce unnecessary global functions.
FAQ:
1. Why it's called `StringUtils.Contains` but not `strings.Contains`
like Golang?
Because our `StringUtils` is not Golang's `strings` package. There will
be our own string functions.
---------
Co-authored-by: silverwind <me@silverwind.io>
Before, there was a `log/buffer.go`, but that design is not general, and
it introduces a lot of irrelevant `Content() (string, error) ` and
`return "", fmt.Errorf("not supported")` .
And the old `log/buffer.go` is difficult to use, developers have to
write a lot of `Contains` and `Sleep` code.
The new `LogChecker` is designed to be a general approach to help to
assert some messages appearing or not appearing in logs.
Users can now upload `webp` images.
Browsers supporting webp images then display this as the avatar of this
user (every major browser except IE).
---------
Co-authored-by: silverwind <me@silverwind.io>
The old code is unnecessarily complex, and has many misuses.
Old code "wraps" a lot, wrap wrap wrap, it's difficult to understand
which kind of handler is used.
The new code uses a general approach, we do not need to write all kinds
of handlers into the "wrapper", do not need to wrap them again and
again.
New code, there are only 2 concepts:
1. HandlerProvider: `func (h any) (handlerProvider func (next)
http.Handler)`, it can be used as middleware
2. Use HandlerProvider to get the final HandlerFunc, and use it for
`r.Get()`
And we can decouple the route package from context package (see the
TODO).
# FAQ
## Is `reflect` safe?
Yes, all handlers are checked during startup, see the `preCheckHandler`
comment. If any handler is wrong, developers could know it in the first
time.
## Does `reflect` affect performance?
No. https://github.com/go-gitea/gitea/pull/24080#discussion_r1164825901
1. This reflect code only runs for each web handler call, handler is far
more slower: 10ms-50ms
2. The reflect is pretty fast (comparing to other code): 0.000265ms
3. XORM has more reflect operations already
# Background
Golang template is not friendly for large projects, and Golang template
team is quite slow, related:
* `https://github.com/golang/go/issues/54450`
Without upstream support, we can also have our solution to make HTML
template functions support context.
It helps a lot, the above Golang template issue `#54450` explains a lot:
1. It makes `{{Locale.Tr}}` could be used in any template, without
passing unclear `(dict "root" . )` anymore.
2. More and more functions need `context`, like `avatar`, etc, we do not
need to do `(dict "Context" $.Context)` anymore.
3. Many request-related functions could be shared by parent&children
templates, like "user setting" / "system setting"
See the test `TestScopedTemplateSetFuncMap`, one template set, two
`Execute` calls with different `CtxFunc`.
# The Solution
Instead of waiting for upstream, this PR re-uses the escaped HTML
template trees, use `AddParseTree` to add related templates/trees to a
new template instance, then the new template instance can have its own
FuncMap , the function calls in the template trees will always use the
new template's FuncMap.
`template.New` / `template.AddParseTree` / `adding-FuncMap` are all
quite fast, so the performance is not affected.
The details:
1. Make a new `html/template/Template` for `all` templates
2. Add template code to the `all` template
3. Freeze the `all` template, reset its exec func map, it shouldn't
execute any template.
4. When a router wants to render a template by its `name`
1. Find the `name` in `all`
2. Find all its related sub templates
3. Escape all related templates (just like what the html template
package does)
4. Add the escaped parse-trees of related templates into a new (scoped)
`text/template/Template`
5. Add context-related func map into the new (scoped) text template
6. Execute the new (scoped) text template
7. To improve performance, the escaped templates are cached to `template
sets`
# FAQ
## There is a `unsafe` call, is this PR unsafe?
This PR is safe. Golang has strict language definition, it's safe to do
so: https://pkg.go.dev/unsafe#Pointer (1) Conversion of a *T1 to Pointer
to *T2
## What if Golang template supports such feature in the future?
The public structs/interfaces/functions introduced by this PR is quite
simple, the code of `HTMLRender` is not changed too much. It's very easy
to switch to the official mechanism if there would be one.
## Does this PR change the template execution behavior?
No, see the tests (welcome to design more tests if it's necessary)
---------
Co-authored-by: silverwind <me@silverwind.io>
Co-authored-by: Jason Song <i@wolfogre.com>
Co-authored-by: Giteabot <teabot@gitea.io>
Close #7570
1. Clearly define the wiki path behaviors, see
`services/wiki/wiki_path.go` and tests
2. Keep compatibility with old contents
3. Allow to use dashes in titles, eg: "2000-01-02 Meeting record"
4. Add a "Pages" link in the dropdown, otherwise users can't go to the
Pages page easily.
5. Add a "View original git file" link in the Pages list, even if some
file names are broken, users still have a chance to edit or remove it,
without cloning the wiki repo to local.
6. Fix 500 error when the name contains prefix spaces.
This PR also introduces the ability to support sub-directories, but it
can't be done at the moment due to there are a lot of legacy wiki data,
which use "%2F" in file names.
Co-authored-by: Giteabot <teabot@gitea.io>
The old code has a lot of technical debts, eg: `repo/wiki/view.tmpl` /
`Iterate`
This PR improves the Wiki TOC display and improves the code.
---------
Co-authored-by: delvh <dev.lh@web.de>
Close #24062
At the beginning, I just wanted to fix the warning mentioned by #24062
But, the cookie code really doesn't look good to me, so clean up them.
Complete the TODO on `SetCookie`:
> TODO: Copied from gitea.com/macaron/macaron and should be improved
after macaron removed.
The default access log format has been unnecessarily escaped, leading to
spurious backslashes appearing in log lines.
Additionally, the `RemoteAddr` field includes the port, which breaks
most log parsers attempting to process it. I've added a call to
`net.SplitHostPort()` attempting to isolate the address alone, with a
fallback to the original address if it errs.
Signed-off-by: Gary Moon <gary@garymoon.net>
The idea is to use a Layered Asset File-system (modules/assetfs/layered.go)
For example: when there are 2 layers: "custom", "builtin", when access
to asset "my/page.tmpl", the Layered Asset File-system will first try to
use "custom" assets, if not found, then use "builtin" assets.
This approach will hugely simplify a lot of code, make them testable.
Other changes:
* Simplify the AssetsHandlerFunc code
* Simplify the `gitea embedded` sub-command code
---------
Co-authored-by: Jason Song <i@wolfogre.com>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Without this patch, the setting SSH.StartBuiltinServer decides whether
the native (Go) implementation is used rather than calling 'ssh-keygen'.
It's possible for 'using ssh-keygen' and 'using the built-in server' to
be independent.
In fact, the gitea rootless container doesn't ship ssh-keygen and can be
configured to use the host's SSH server - which will cause the public
key parsing mechanism to break.
This commit changes the decision to be based on SSH.KeygenPath instead.
Any existing configurations with a custom KeygenPath set will continue
to function. The new default value of '' selects the native version. The
downside of this approach is that anyone who has relying on plain
'ssh-keygen' to have special properties will now be using the native
version instead.
I assume the exec-variant is only there because /x/crypto/ssh didn't
support ssh-ed25519 until 2016. I don't see any other reason for using
it so it might be an acceptable risk.
Fixes #23363
EDIT: this message was garbled when I tried to get the commit
description back in.. Trying to reconstruct it:
## ⚠️ BREAKING ⚠️ Users who don't have SSH.KeygenPath
explicitly set and rely on the ssh-keygen binary need to set
SSH.KeygenPath to 'ssh-keygen' in order to be able to continue using it
for public key parsing.
There was something else but I can't remember at the moment.
EDIT2: It was about `make test` and `make lint`. Can't get them to run.
To reproduce the issue, I installed `golang` in `docker.io/node:16` and
got:
```
...
go: mvdan.cc/xurls/v2@v2.4.0: unknown revision mvdan.cc/xurls/v2.4.0
go: gotest.tools/v3@v3.4.0: unknown revision gotest.tools/v3.4.0
...
go: gotest.tools/v3@v3.0.3: unknown revision gotest.tools/v3.0.3
...
go: error loading module requirements
```
Signed-off-by: Leon M. Busch-George <leon@georgemail.eu>
Close #24002
Two problems before:
1. The `log.Fatal` is missing after these `wrapFatal` calls, so the
error is not shown to users.
2. `GetTemplateAssetNames` has different behaviors for local files and
builtin assets, for builtin assets, it also returns directories, so we
need to check the extension again.
I have tested with `TAGS="bindata sqlite sqlite_unlock_notify" make
build && ./gitea` , it works well now. Before, the server responds
internal server error (because it doesn't complete the template parsing)
None of the features of `unrolled/render` package is used.
The Golang builtin "html/template" just works well. Then we can improve
our HTML render to resolve the "$.root.locale.Tr" problem as much as
possible.
Next step: we can have a template render pool (by Clone), then we can
inject global functions with dynamic context to every `Execute` calls.
Then we can use `{{Locale.Tr ....}}` directly in all templates , no need
to pass the `$.root.locale` again and again.
One of the steps in #23328
Before there were 3 different but similar functions: dict/Dict/mergeinto
The code was just copied & pasted, no test.
This PR defines a new stable `dict` function, it covers all the 3 old
functions behaviors, only +160 -171
Future developers do not need to think about or guess the different dict
functions, just use one: `dict`
Why use `dict` but not `Dict`? Because there are far more `dict` than
`Dict` in code already ......
One of the proposals in #23328
This PR introduces a simple expression calculator
(templates/eval/eval.go), it can do basic expression calculations.
Many untested template helper functions like `Mul` `Add` can be replaced
by this new approach.
Then these `Add` / `Mul` / `percentage` / `Subtract` / `DiffStatsWidth`
could all use this `Eval`.
And it provides enhancements for Golang templates, and improves
readability.
Some examples:
----
* Before: `{{Add (Mul $glyph.Row 12) 12}}`
* After: `{{Eval $glyph.Row "*" 12 "+" 12}}`
----
* Before: `{{if lt (Add $i 1) (len $.Topics)}}`
* After: `{{if Eval $i "+" 1 "<" (len $.Topics)}}`
## FAQ
### Why not use an existing expression package?
We need a highly customized expression engine:
* do the calculation on the fly, without pre-compiling
* deal with int/int64/float64 types, to make the result could be used in
Golang template.
* make the syntax could be used in the Golang template directly
* do not introduce too much complex or strange syntax, we just need a
simple calculator.
* it needs to strictly follow Golang template's behavior, for example,
Golang template treats all non-zero values as truth, but many 3rd
packages don't do so.
### What's the benefit?
* Developers don't need to add more `Add`/`Mul`/`Sub`-like functions,
they were getting more and more.
Now, only one `Eval` is enough for all cases.
* The new code reads better than old `{{Add (Mul $glyph.Row 12) 12}}`,
the old one isn't familiar to most procedural programming developers
(eg, the Golang expression syntax).
* The `Eval` is fully covered by tests, many old `Add`/`Mul`-like
functions were never tested.
### The performance?
It doesn't use `reflect`, it doesn't need to parse or compile when used
in Golang template, the performance is as fast as native Go template.
### Is it too complex? Could it be unstable?
The expression calculator program is a common homework for computer
science students, and it's widely used as a teaching and practicing
purpose for developers. The algorithm is pretty well-known.
The behavior can be clearly defined, it is stable.
The first step of #23328
This PR cleans:
1. Dead function like `SubStr`
2. Unnecessary function like `UseHTTPS`, it should be provided by
`window.origin`
3. Duplicate function like `IsShowFullName`, there was already a
`DeafultShowFullName`
I have searched these removed functions globally, no use in code.
Remove the misbehaving function and call
Repository.GetFilesChangedBetween instead.
Fixes #23919
---
~~_TODO_ test this~~ `Repository.getFilesChanged` seems to be only used
by Gitea Actions, but a similar function already exists
**Update** I tested this change and the issue is gone.
The _graceful_ should fail less when the `.editorconfig` file isn't
properly written, e.g. boolean values from YAML or unparseable numbers
(when a number is expected). As is... information is lost as the
_warning_ (a go-multierror.Error) is ignored. If anybody knows how to
send them to the UI as warning; any help is appreciated.
Closes #20694
Signed-off-by: Yoan Blanc <yoan@dosimple.ch>
At first, we have one unified team unit permission which is called
`Team.Authorize` in DB.
But since https://github.com/go-gitea/gitea/pull/17811, we allowed
different units to have different permission.
The old code is only designed for the old version. So after #17811, if
org users have write permission of other units, but have no permission
of packages, they can also get write permission of packages.
Co-authored-by: delvh <dev.lh@web.de>
Follow #23633 and #23240
Close #23814
Now we almost have a complete test set for Gitea's LocalStore.
This PR is still a quick fix for the legacy locale system (see the
TODOs), to resolve the problems fundamentally, it needs more work in the
future.
