- Databases are one of the most important parts of Forgejo, every
interaction with Forgejo uses the database in one way or another.
Therefore, it is important to maintain the database and recognize when
Forgejo is not doing well with the database. Forgejo already has the
option to log *every* SQL query along with its execution time, but
monitoring becomes impractical for larger instances and takes up
unnecessary storage in the logs.
- Add a QoL enhancement that allows instance administrators to specify a
threshold value beyond which query execution time is logged as a warning
in the xorm logger. The default value is a conservative five seconds to
avoid this becoming a source of spam in the logs.
- The use case for this patch is that with an instance the size of Codeberg, monitoring SQL logs is not very fruitful and most of them are uninteresting. Recently, in the context of persistent deadlock issues (https://codeberg.org/forgejo/forgejo/issues/220), I have noticed that certain queries hold locks on tables like comment and issue for several seconds. This patch helps to identify which queries these are and when they happen.
- Added unit test.
(cherry picked from commit 24bbe7886f)
(cherry picked from commit 6e29145b3c)
(cherry picked from commit 63731e3071)
(cherry picked from commit 3ce1a09736)
(cherry picked from commit a64426907d)
(cherry picked from commit 4b19215691)
(cherry picked from commit e635674435)
(cherry picked from commit 9cf501f1af)
(cherry picked from commit 0d6b934eba)
(cherry picked from commit 4b6c273879)
(cherry picked from commit 89b1315338)
(cherry picked from commit edd8e66ce9)
[GITEA] Add slow SQL query warning (squash) document the setting
(cherry picked from commit ce38599c51)
(cherry picked from commit 794aa67c68)
(cherry picked from commit a4c2c6b004)
(cherry picked from commit 97912752bc)
(cherry picked from commit 00b5327c97)
(cherry picked from commit 1069c860e7)
(cherry picked from commit 84241f42c8)
(cherry picked from commit e4bda0e845)
(cherry picked from commit 7357fb91bf)
(cherry picked from commit a8dd7f6da2)
(cherry picked from commit e636e9f4be)
(cherry picked from commit bf04ae8603)
(cherry picked from commit 93b19e3568)
(cherry picked from commit 83f91363ad)
(cherry picked from commit e34a05bc73)
Mainly for MySQL/MSSQL.
It is important for Gitea to use case-sensitive database charset
collation. If the database is using a case-insensitive collation, Gitea
will show startup error/warning messages, and show the errors/warnings
on the admin panel's Self-Check page.
Make `gitea doctor convert` work for MySQL to convert the collations of
database & tables & columns.
* Fix #28131
## ⚠️ BREAKING ⚠️
It is not quite breaking, but it's highly recommended to convert the
database&table&column to a consistent and case-sensitive collation.
This patchset changes the connection string builder to use net.URL and
the host/port parser to use the stdlib function for splitting host from
port. It also adds a footnote about a potentially required portnumber
for postgres UNIX sockets.
Fixes: #24552
In modern days, there is no reason to make users set "charset" anymore.
Close #25378
## ⚠️ BREAKING
The key `[database].CHARSET` was removed completely as every newer
(>10years) MySQL database supports `utf8mb4` already.
There is a (deliberately) undocumented new fallback option if anyone
still needs to use it, but we don't recommend using it as it simply
causes problems.
## ⚠️ Breaking
The `log.<mode>.<logger>` style config has been dropped. If you used it,
please check the new config manual & app.example.ini to make your
instance output logs as expected.
Although many legacy options still work, it's encouraged to upgrade to
the new options.
The SMTP logger is deleted because SMTP is not suitable to collect logs.
If you have manually configured Gitea log options, please confirm the
logger system works as expected after upgrading.
## Description
Close #12082 and maybe more log-related issues, resolve some related
FIXMEs in old code (which seems unfixable before)
Just like rewriting queue #24505 : make code maintainable, clear legacy
bugs, and add the ability to support more writers (eg: JSON, structured
log)
There is a new document (with examples): `logging-config.en-us.md`
This PR is safer than the queue rewriting, because it's just for
logging, it won't break other logic.
## The old problems
The logging system is quite old and difficult to maintain:
* Unclear concepts: Logger, NamedLogger, MultiChannelledLogger,
SubLogger, EventLogger, WriterLogger etc
* Some code is diffuclt to konw whether it is right:
`log.DelNamedLogger("console")` vs `log.DelNamedLogger(log.DEFAULT)` vs
`log.DelLogger("console")`
* The old system heavily depends on ini config system, it's difficult to
create new logger for different purpose, and it's very fragile.
* The "color" trick is difficult to use and read, many colors are
unnecessary, and in the future structured log could help
* It's difficult to add other log formats, eg: JSON format
* The log outputer doesn't have full control of its goroutine, it's
difficult to make outputer have advanced behaviors
* The logs could be lost in some cases: eg: no Fatal error when using
CLI.
* Config options are passed by JSON, which is quite fragile.
* INI package makes the KEY in `[log]` section visible in `[log.sub1]`
and `[log.sub1.subA]`, this behavior is quite fragile and would cause
more unclear problems, and there is no strong requirement to support
`log.<mode>.<logger>` syntax.
## The new design
See `logger.go` for documents.
## Screenshot
<details>
</details>
## TODO
* [x] add some new tests
* [x] fix some tests
* [x] test some sub-commands (manually ....)
---------
Co-authored-by: Jason Song <i@wolfogre.com>
Co-authored-by: delvh <dev.lh@web.de>
Co-authored-by: Giteabot <teabot@gitea.io>
Creating pid file should not belong to setting package and only web
command needs that. So this PR moves pidfile creation from setting
package to web command package to keep setting package more readable.
I marked this as `break` because the PIDFile path moved. For those who
have used the pid build argument, it has to be changed.
---------
Co-authored-by: KN4CK3R <admin@oldschoolhack.me>
Replace #23350.
Refactor `setting.Database.UseMySQL` to
`setting.Database.Type.IsMySQL()`.
To avoid mismatching between `Type` and `UseXXX`.
This refactor can fix the bug mentioned in #23350, so it should be
backported.
Some bugs caused by less unit tests in fundamental packages. This PR
refactor `setting` package so that create a unit test will be easier
than before.
- All `LoadFromXXX` files has been splited as two functions, one is
`InitProviderFromXXX` and `LoadCommonSettings`. The first functions will
only include the code to create or new a ini file. The second function
will load common settings.
- It also renames all functions in setting from `newXXXService` to
`loadXXXSetting` or `loadXXXFrom` to make the function name less
confusing.
- Move `XORMLog` to `SQLLog` because it's a better name for that.
Maybe we should finally move these `loadXXXSetting` into the `XXXInit`
function? Any idea?
---------
Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: delvh <dev.lh@web.de>
Gitea will migrate the database model version automatically, but it
should be able to be disabled and keep Gitea shutdown if the version is
not matched.
Change all license headers to comply with REUSE specification.
Fix #16132
Co-authored-by: flynnnnnnnnnn <flynnnnnnnnnn@github>
Co-authored-by: John Olheiser <john.olheiser@gmail.com>
* Use default values when provided values are empty
- When provided values are empty like `:3000` would imply that host is
empty, use the default value.
- Resolves #20316
* Update database.go
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
* Refactor install page (db type)
* set correct default DB HOST for different DB TYPE
* remove legacy TiDB from documents
* unify the usage of DB TYPE, in code we only use "mysql". "MySQL" is only shown to users for friendly name.
* Gitea can use TiDB via MySQL protocol
Co-authored-by: zeripath <art27@cantab.net>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
One perennial issue is users running the install page,
changing the database dialect and then suffering with issues
This PR simply resets all of the database.Use flags on
initDBConfig. This should prevent this issue from occuring.
Fix #13788
Fix #5480
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
* Avatar support minio
* Support repo avatar minio storage
* Add missing migration
* Fix bug
* Fix test
* Add test for minio store type on avatars and repo avatars; Add documents
* Fix bug
* Fix bug
* Add back missed avatar link method
* refactor codes
* Simplify the codes
* Code improvements
* Fix lint
* Fix test mysql
* Fix test mysql
* Fix test mysql
* Fix settings
* Fix test
* fix test
* Fix bug
MySQL in its infinite wisdom determines that UTF8 does not
mean UTF8. Our install scripts know about this and will set
CHARSET to utf8mb4 if we users choose this but... users who
do not explicitly set this variable will default to utf8mb3
without knowing it.
This PR changes the unset CHARSET value to utf8mb4 if users
choose to use mysql.
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Add support for database schema
* Require setting search_path for the db user
* Add schema setting to admin/config.tmpl
* Use a schema different from default for psql tests
* Update postgres scripts to use custom schema
* Update to xorm/core 0.7.3 and xorm/xorm c37aff9b3a
* Fix migration test
Co-authored-by: Antoine GIRARD <sapk@users.noreply.github.com>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
* Expose db.SetMaxOpenConns and allow other dbs to set their connection params
* Add note about port exhaustion
Co-Authored-By: guillep2k <18600385+guillep2k@users.noreply.github.com>
