Backport #23911 by @lunny
Follow up #22405
Fix #20703
This PR rewrites storage configuration read sequences with some breaks
and tests. It becomes more strict than before and also fixed some
inherit problems.
- Move storage's MinioConfig struct into setting, so after the
configuration loading, the values will be stored into the struct but not
still on some section.
- All storages configurations should be stored on one section,
configuration items cannot be overrided by multiple sections. The
prioioty of configuration is `[attachment]` > `[storage.attachments]` |
`[storage.customized]` > `[storage]` > `default`
- For extra override configuration items, currently are `SERVE_DIRECT`,
`MINIO_BASE_PATH`, `MINIO_BUCKET`, which could be configured in another
section. The prioioty of the override configuration is `[attachment]` >
`[storage.attachments]` > `default`.
- Add more tests for storages configurations.
- Update the storage documentations.
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
Before, Gitea shows the database table stats on the `admin dashboard`
page.
It has some problems:
* `count(*)` is quite heavy. If tables have many records, this blocks
loading the admin page blocks for a long time
* Some users had even reported issues that they can't visit their admin
page because this page causes blocking or `50x error (reverse proxy
timeout)`
* The `actions` stat is not useful. The table is simply too large. Does
it really matter if it contains 1,000,000 rows or 9,999,999 rows?
* The translation `admin.dashboard.statistic_info` is difficult to
maintain.
So, this PR uses a separate page to show the stats and removes the
`actions` stat.
## ⚠️ BREAKING
The `actions` Prometheus metrics collector has been removed for the
reasons mentioned beforehand.
Please do not rely on its output anymore.
The admin config page has been broken for many many times, a little
refactoring would make this page panic.
So, add a test for it, and add another test to cover the 500 error page.
Co-authored-by: Giteabot <teabot@gitea.io>
Don't really know a better name for this. I've gone through some Forms
and added missing HTML attributes (mostly `maxlength`). I tried to fill
the Forms with dummy Data and see if Gitea throws a Error (e.g. maximum
length). If yes, I added the missing HTML attribute.
While working on this, I discovered that the Form to add OAuth2 Apps
just silently fails when filled with invalid data, so I fixed that too.
## ⚠️ Breaking
The `log.<mode>.<logger>` style config has been dropped. If you used it,
please check the new config manual & app.example.ini to make your
instance output logs as expected.
Although many legacy options still work, it's encouraged to upgrade to
the new options.
The SMTP logger is deleted because SMTP is not suitable to collect logs.
If you have manually configured Gitea log options, please confirm the
logger system works as expected after upgrading.
## Description
Close #12082 and maybe more log-related issues, resolve some related
FIXMEs in old code (which seems unfixable before)
Just like rewriting queue #24505 : make code maintainable, clear legacy
bugs, and add the ability to support more writers (eg: JSON, structured
log)
There is a new document (with examples): `logging-config.en-us.md`
This PR is safer than the queue rewriting, because it's just for
logging, it won't break other logic.
## The old problems
The logging system is quite old and difficult to maintain:
* Unclear concepts: Logger, NamedLogger, MultiChannelledLogger,
SubLogger, EventLogger, WriterLogger etc
* Some code is diffuclt to konw whether it is right:
`log.DelNamedLogger("console")` vs `log.DelNamedLogger(log.DEFAULT)` vs
`log.DelLogger("console")`
* The old system heavily depends on ini config system, it's difficult to
create new logger for different purpose, and it's very fragile.
* The "color" trick is difficult to use and read, many colors are
unnecessary, and in the future structured log could help
* It's difficult to add other log formats, eg: JSON format
* The log outputer doesn't have full control of its goroutine, it's
difficult to make outputer have advanced behaviors
* The logs could be lost in some cases: eg: no Fatal error when using
CLI.
* Config options are passed by JSON, which is quite fragile.
* INI package makes the KEY in `[log]` section visible in `[log.sub1]`
and `[log.sub1.subA]`, this behavior is quite fragile and would cause
more unclear problems, and there is no strong requirement to support
`log.<mode>.<logger>` syntax.
## The new design
See `logger.go` for documents.
## Screenshot
<details>
</details>
## TODO
* [x] add some new tests
* [x] fix some tests
* [x] test some sub-commands (manually ....)
---------
Co-authored-by: Jason Song <i@wolfogre.com>
Co-authored-by: delvh <dev.lh@web.de>
Co-authored-by: Giteabot <teabot@gitea.io>
Although some features are mixed together in this PR, this PR is not
that large, and these features are all related.
Actually there are more than 70 lines are for a toy "test queue", so
this PR is quite simple.
Major features:
1. Allow site admin to clear a queue (remove all items in a queue)
* Because there is no transaction, the "unique queue" could be corrupted
in rare cases, that's unfixable.
* eg: the item is in the "set" but not in the "list", so the item would
never be able to be pushed into the queue.
* Now site admin could simply clear the queue, then everything becomes
correct, the lost items could be re-pushed into queue by future
operations.
3. Split the "admin/monitor" to separate pages
4. Allow to download diagnosis report
* In history, there were many users reporting that Gitea queue gets
stuck, or Gitea's CPU is 100%
* With diagnosis report, maintainers could know what happens clearly
The diagnosis report sample:
[gitea-diagnosis-20230510-192913.zip](https://github.com/go-gitea/gitea/files/11441346/gitea-diagnosis-20230510-192913.zip)
, use "go tool pprof profile.dat" to view the report.
Screenshots:
---------
Co-authored-by: Jason Song <i@wolfogre.com>
Co-authored-by: Giteabot <teabot@gitea.io>
# ⚠️ Breaking
Many deprecated queue config options are removed (actually, they should
have been removed in 1.18/1.19).
If you see the fatal message when starting Gitea: "Please update your
app.ini to remove deprecated config options", please follow the error
messages to remove these options from your app.ini.
Example:
```
2023/05/06 19:39:22 [E] Removed queue option: `[indexer].ISSUE_INDEXER_QUEUE_TYPE`. Use new options in `[queue.issue_indexer]`
2023/05/06 19:39:22 [E] Removed queue option: `[indexer].UPDATE_BUFFER_LEN`. Use new options in `[queue.issue_indexer]`
2023/05/06 19:39:22 [F] Please update your app.ini to remove deprecated config options
```
Many options in `[queue]` are are dropped, including:
`WRAP_IF_NECESSARY`, `MAX_ATTEMPTS`, `TIMEOUT`, `WORKERS`,
`BLOCK_TIMEOUT`, `BOOST_TIMEOUT`, `BOOST_WORKERS`, they can be removed
from app.ini.
# The problem
The old queue package has some legacy problems:
* complexity: I doubt few people could tell how it works.
* maintainability: Too many channels and mutex/cond are mixed together,
too many different structs/interfaces depends each other.
* stability: due to the complexity & maintainability, sometimes there
are strange bugs and difficult to debug, and some code doesn't have test
(indeed some code is difficult to test because a lot of things are mixed
together).
* general applicability: although it is called "queue", its behavior is
not a well-known queue.
* scalability: it doesn't seem easy to make it work with a cluster
without breaking its behaviors.
It came from some very old code to "avoid breaking", however, its
technical debt is too heavy now. It's a good time to introduce a better
"queue" package.
# The new queue package
It keeps using old config and concept as much as possible.
* It only contains two major kinds of concepts:
* The "base queue": channel, levelqueue, redis
* They have the same abstraction, the same interface, and they are
tested by the same testing code.
* The "WokerPoolQueue", it uses the "base queue" to provide "worker
pool" function, calls the "handler" to process the data in the base
queue.
* The new code doesn't do "PushBack"
* Think about a queue with many workers, the "PushBack" can't guarantee
the order for re-queued unhandled items, so in new code it just does
"normal push"
* The new code doesn't do "pause/resume"
* The "pause/resume" was designed to handle some handler's failure: eg:
document indexer (elasticsearch) is down
* If a queue is paused for long time, either the producers blocks or the
new items are dropped.
* The new code doesn't do such "pause/resume" trick, it's not a common
queue's behavior and it doesn't help much.
* If there are unhandled items, the "push" function just blocks for a
few seconds and then re-queue them and retry.
* The new code doesn't do "worker booster"
* Gitea's queue's handlers are light functions, the cost is only the
go-routine, so it doesn't make sense to "boost" them.
* The new code only use "max worker number" to limit the concurrent
workers.
* The new "Push" never blocks forever
* Instead of creating more and more blocking goroutines, return an error
is more friendly to the server and to the end user.
There are more details in code comments: eg: the "Flush" problem, the
strange "code.index" hanging problem, the "immediate" queue problem.
Almost ready for review.
TODO:
* [x] add some necessary comments during review
* [x] add some more tests if necessary
* [x] update documents and config options
* [x] test max worker / active worker
* [x] re-run the CI tasks to see whether any test is flaky
* [x] improve the `handleOldLengthConfiguration` to provide more
friendly messages
* [x] fine tune default config values (eg: length?)
## Code coverage:
Since the login form label for user_name unconditionally displays
`Username or Email Address` for the `user_name` field, bring matching
LDAP filters to more prominence in the documentation/placeholders.
Signed-off-by: Gary Moon <gary@garymoon.net>
Close #24302
Part of #24229, Follows #24246
This PR focused on CSS style fine-tune, main changes:
1. Give `.ui.ui.ui.container` a width of `1280px` with a max-width of
`calc(100vw - 64px)`, so the main contents looks better on large
devices.
2. Share styles for table elements in all levels settings pages to fix
overflow of runners table on mobile and for consistency (The headers on
mobile can be further improved, but haven't found a proper way yet).
3. Use [stackable
grid](https://fomantic-ui.com/collections/grid.html#stackable) and
[device column width](https://fomantic-ui.com/examples/responsive.html)
for responsiveness for some pages (repo/org collaborators settings
pages, org teams related page)
4. Fixed #24302 by sharing label related CSS in reporg.css
5. Fine tune repo tags settings page
---------
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
After #24317 this function is only used in one place where it is not
needed. I confirmed the timestamp still renders correctly
Signed-off-by: Yarden Shoham <git@yardenshoham.com>
This refactors the `shared/datetime/short|long|full` templates into a
template helper function, which allows us to render absolute date times
within translatable phrases.
- Follows #23988
- The first attempt was in #24055
- This should help #22664
Changes:
1. Added the `DateTime` template helper that replaces the
`shared/datetime/short|long|full` templates
2. Used find-and-replace with varying regexes to replace the templates
from step 1 (for example, `\{\{template "shared/datetime/(\S+) \(dict
"Datetime" ([^"]+) "Fallback" ([^\)]+\)?) ?\)?\}\}` -> `{{DateTime "$1
$2 $3}}`)
3. Used the new `DateTime` helper in the issue due date timestamp
rendering
# Before
# After
---------
Signed-off-by: Yarden Shoham <git@yardenshoham.com>
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
The first step of #23328
This PR cleans:
1. Dead function like `SubStr`
2. Unnecessary function like `UseHTTPS`, it should be provided by
`window.origin`
3. Duplicate function like `IsShowFullName`, there was already a
`DeafultShowFullName`
I have searched these removed functions globally, no use in code.
Follow:
* #23574
* Remove all ".tooltip[data-content=...]"
Major changes:
* Remove "tooltip" class, use "[data-tooltip-content=...]" instead of
".tooltip[data-content=...]"
* Remove legacy `data-position`, it's dead code since last Fomantic
Tooltip -> Tippy Tooltip refactoring
* Rename reaction attribute from `data-content` to
`data-reaction-content`
* Add comments for some `data-content`: `{{/* used by the form */}}`
* Remove empty "ui" class
* Use "text color" for SVG icons (a few)
Caught by @wxiaoguang in
https://github.com/go-gitea/gitea/pull/23337#issuecomment-1467317742.
Additionally, there were three instances that have the same content as `templates/base/deletion_modal_actions.tmpl` but that are not intended to delete something.
Instead of renaming the template above, these instances were simply re-hard-coded again.
Renaming/improving the template above is left for future PRs.
This improves a lot of accessibility shortcomings.
Every possible instance of `<div class="button">` matching the command
`ag '<[^ab].*?class=.*?[" ]button[ "]' templates/ | grep -v 'dropdown'`
has been converted when possible.
divs with the `dropdown` class and their children were omitted as
1. more analysis must be conducted whether the dropdowns still work as
intended when they are a `button` instead of a `div`.
2. most dropdowns have `div`s as children. The HTML standard disallows
`div`s inside `button`s.
3. When a dropdown child that's part of the displayed text content is
converted to a `button`, the dropdown can be focused twice
Further changes include that all "gitea-managed" buttons with JS code
received an `e.preventDefault()` so that they don't accidentally submit
an underlying form, which would execute instead of cancel the action.
Lastly, some minor issues were fixed as well during the refactoring.
## Future improvements
As mentioned in
https://github.com/go-gitea/gitea/pull/23337#discussion_r1127277391,
`<a>`s without `href` attribute are not focusable.
They should later on be converted to `<button>`s.
---------
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
Co-authored-by: silverwind <me@silverwind.io>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
This PR adds support for reflogs on all repositories. It does this by
adding a global configuration entry.
Implements #14865
---------
Signed-off-by: Philip Peterson <philip.c.peterson@gmail.com>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
When there is an error creating a new openIDConnect authentication
source try to handle the error a little better.
Close #23283
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
https://github.com/go-gitea/gitea/pull/22950 removed `hide` class, and
use `gt-hidden`
But there are some missed `hide`....
---------
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
Close #22847
This PR:
* introduce Gitea's own `showElem` and related functions
* remove jQuery show/hide
* remove .hide class
* remove inline style=display:none
From now on:
do not use:
* "[hidden]" attribute: it's too weak, can not be applied to an element
with "display: flex"
* ".hidden" class: it has been polluted by Fomantic UI in many cases
* inline style="display: none": it's difficult to tweak
* jQuery's show/hide/toggle: it can not show/hide elements with
"display: xxx !important"
only use:
* this ".gt-hidden" class
* showElem/hideElem/toggleElem functions in "utils/dom.js"
cc: @silverwind , this is the all-in-one PR
To avoid duplicated load of the same data in an HTTP request, we can set
a context cache to do that. i.e. Some pages may load a user from a
database with the same id in different areas on the same page. But the
code is hidden in two different deep logic. How should we share the
user? As a result of this PR, now if both entry functions accept
`context.Context` as the first parameter and we just need to refactor
`GetUserByID` to reuse the user from the context cache. Then it will not
be loaded twice on an HTTP request.
But of course, sometimes we would like to reload an object from the
database, that's why `RemoveContextData` is also exposed.
The core context cache is here. It defines a new context
```go
type cacheContext struct {
ctx context.Context
data map[any]map[any]any
lock sync.RWMutex
}
var cacheContextKey = struct{}{}
func WithCacheContext(ctx context.Context) context.Context {
return context.WithValue(ctx, cacheContextKey, &cacheContext{
ctx: ctx,
data: make(map[any]map[any]any),
})
}
```
Then you can use the below 4 methods to read/write/del the data within
the same context.
```go
func GetContextData(ctx context.Context, tp, key any) any
func SetContextData(ctx context.Context, tp, key, value any)
func RemoveContextData(ctx context.Context, tp, key any)
func GetWithContextCache[T any](ctx context.Context, cacheGroupKey string, cacheTargetID any, f func() (T, error)) (T, error)
```
Then let's take a look at how `system.GetString` implement it.
```go
func GetSetting(ctx context.Context, key string) (string, error) {
return cache.GetWithContextCache(ctx, contextCacheKey, key, func() (string, error) {
return cache.GetString(genSettingCacheKey(key), func() (string, error) {
res, err := GetSettingNoCache(ctx, key)
if err != nil {
return "", err
}
return res.SettingValue, nil
})
})
}
```
First, it will check if context data include the setting object with the
key. If not, it will query from the global cache which may be memory or
a Redis cache. If not, it will get the object from the database. In the
end, if the object gets from the global cache or database, it will be
set into the context cache.
An object stored in the context cache will only be destroyed after the
context disappeared.
As discussed in #22847 the helpers in helpers.less need to have a
separate prefix as they are causing conflicts with fomantic styles
This will allow us to have the `.gt-hidden { display:none !important; }`
style that is needed to for the reverted PR.
Of note in doing this I have noticed that there was already a conflict
with at least one chroma style which this PR now avoids.
I've also added in the `gt-hidden` style that matches the tailwind one
and switched the code that needed it to use that.
Signed-off-by: Andrew Thornton <art27@cantab.net>
---------
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
Fixes #19555
Test-Instructions:
https://github.com/go-gitea/gitea/pull/21441#issuecomment-1419438000
This PR implements the mapping of user groups provided by OIDC providers
to orgs teams in Gitea. The main part is a refactoring of the existing
LDAP code to make it usable from different providers.
Refactorings:
- Moved the router auth code from module to service because of import
cycles
- Changed some model methods to take a `Context` parameter
- Moved the mapping code from LDAP to a common location
I've tested it with Keycloak but other providers should work too. The
JSON mapping format is the same as for LDAP.
---------
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
EDIT: The main change of this PR was resolved by #22599. This
complements that PR for some cases without label and complicated layout
to be added.
NOTE: Contributed by @Forgejo.
👋 Hey
I'm new around here, so I may have done some mistakes, sorry!
---
## Context
On a fresh Gitea install, when I go to the [config admin
page](http://localhost:3000/admin/config) I had a 500 error page.
The logs:
```
2022/12/10 20:08:47 ...s/context/context.go:232:HTML() [E] [6394d93f] Render failed: template: admin/config:180:22: executing "admin/config" at <.Service.DefaultAllowOnlyContributorsToTrackTime>: DefaultAllowOnlyContributorsToTrackTime has arguments but cannot be invoked as function
2022/12/10 20:08:47 [6394d93f] router: completed GET /admin/config for [::1]:43800, 500 Internal Server Error in 5.1ms @ admin/config.go:99(admin.Config)
```
## The fix
I removed the `$.Context` on the
`.Service.DefaultAllowOnlyContributorsToTrackTime` to fix the 500 error
page happening. It could be a mistake, and I don't fully understand what
I've done!
Signed-off-by: Restray <contact@restray.org>
