* CutDiffAroundLine makes the incorrect assumption that `---` and `+++` always represent part of the header of a diff.
This PR adds a flag to its parsing to prevent this problem and adds a streaming parsing technique to CutDiffAroundLine using an io.pipe instead of just sending data to an unbounded buffer.
Fix #14711
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Handle unquoted comment patch files
When making comment patches unfortunately the patch does not always quote the filename
This makes the diff --git header ambiguous again.
This PR finally adds handling for ambiguity in to parse patch
Fix #14812
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Add in testing for no error
There is no way currently for CutDiffAroundLine in this test to cause an
error however, it should still be tested.
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Added MirrorInterval to the API
* Remove MirrorInterval from CreateRepository
* Removed Duplicate UpdateMirror Function
* Updated Error Logging
* Update Log Message for is not Mirror
Co-authored-by: 6543 <6543@obermui.de>
* Delete Debug Statement that snuck in
Co-authored-by: zeripath <art27@cantab.net>
* Add Check for If Interval is too small
* Output to API Call
* Add Error Object when time is Less than Min Interval
* Frequency Error Message
Co-authored-by: zeripath <art27@cantab.net>
* Allow Zero Mirror Interval
Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: zeripath <art27@cantab.net>
* Add Ref to api.Issue
This PR adds Ref to api.Issue and adds the backend parts to the migrator.
Fix #13918
Signed-off-by: Andrew Thornton <art27@cantab.net>
* add black list and white list support for migrating repositories
* fix fmt
* fix lint
* fix vendor
* fix modules.txt
* clean diff
* specify log message
* use blocklist/allowlist
* allways use lowercase to match url
* Apply allow/block
* Settings: use existing "migrations" section
* convert domains lower case
* dont store unused value
* Block private addresses for migration by default
* fix lint
* use proposed-upstream func to detect private IP addr
* a nit
* add own error for blocked migration, add tests, imprufe api
* fix test
* fix-if-localhost-is-ipv4
* rename error & error message
* rename setting options
* Apply suggestions from code review
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Co-authored-by: zeripath <art27@cantab.net>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
An extension to #13444 - where we now ensure that comment reaction failures do not cause migrations failure
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Migrating reactions is just not that important
A failure during migrating reactions should not cause failure of
migration.
Signed-off-by: Andrew Thornton <art27@cantab.net>
* When checking issue reactions check the correct permission
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
* first draft
* update gitea sdk to 9e280adb4da
* adapt feat of updated sdk
* releases now works
* break the Reactions loop
* use convertGiteaLabel
* fix endless loop because paggination is not supported there !!!
* rename gitea local uploader files
* pagination can bite you in the ass
* Version Checks
* lint
* docs
* rename gitea sdk import to miss future conficts
* go-swagger: dont scan the sdk structs
* make sure gitea can shutdown gracefully
* make GetPullRequests and GetIssues similar
* rm useles
* Add Test: started ...
* ... add tests ...
* Add tests and Fixing things
* Workaround missing SHA
* Adapt: Ensure that all migration requests are cancellable
(714ab71ddc)
* LINT: fix misspells in test set
* adapt ListMergeRequestAwardEmoji
* update sdk
* Return error when creating giteadownloader failed
* update sdk
* adapt new sdk
* adopt new features
* check version before err
* adapt: 'migrate service type switch page'
* optimize
* Fix DefaultBranch
* impruve
* handle subPath
* fix test
* Fix ReviewCommentPosition
* test GetReviews
* add DefaultBranch int test set
* rm unused
* Update SDK to v0.13.0
* addopt sdk changes
* found better link
* format template
* Update Docs
* Update Gitea SDK (v0.13.1)
* use different structs for MigrateRepoOptions on UI and API
* Fix TokenAuth and rename UID to an understandable Name
* fix swagger doc
* simplify & mk redable
* R E F A C T O R:
migration has now internal 3 structs to store its options:
* the Options for WebUI: modules/auth/repo_form.go
* the Options for API: modules/structs/repo.go
* the option struct with after validation for internal prossessing: modules/migrations/base/options.go
* Copyright Header
* Deprecate UID - add RepoOwner
* adopt repo.go -> migrate.go
* add comment about each struct purpose
* lint
* fix migrations: gitlab_test
* more stable test
* Get Reactions to Pulls
* Reactions for Issues
* Comments are still hard
Co-authored-by: Lauris BH <lauris@nix.lv>
* Ensure that all migration requests are cancellable
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Use WithContext as RequestWithContext is go 1.14
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
* Add a storage layer for attachments
* Fix some bug
* fix test
* Fix copyright head and lint
* Fix bug
* Add setting for minio and flags for migrate-storage
* Add documents
* fix lint
* Add test for minio store type on attachments
* fix test
* fix test
* Apply suggestions from code review
Co-authored-by: guillep2k <18600385+guillep2k@users.noreply.github.com>
* Add warning when storage migrated successfully
* Fix drone
* fix test
* rebase
* Fix test
* display the error on console
* Move minio test to amd64 since minio docker don't support arm64
* refactor the codes
* add trace
* Fix test
* remove log on xorm
* Fi download bug
* Add a storage layer for attachments
* Add setting for minio and flags for migrate-storage
* fix lint
* Add test for minio store type on attachments
* Apply suggestions from code review
Co-authored-by: guillep2k <18600385+guillep2k@users.noreply.github.com>
* Fix drone
* fix test
* Fix test
* display the error on console
* Move minio test to amd64 since minio docker don't support arm64
* refactor the codes
* add trace
* Fix test
* Add URL function to serve attachments directly from S3/Minio
* Add ability to enable/disable redirection in attachment configuration
* Fix typo
* Add a storage layer for attachments
* Add setting for minio and flags for migrate-storage
* fix lint
* Add test for minio store type on attachments
* Apply suggestions from code review
Co-authored-by: guillep2k <18600385+guillep2k@users.noreply.github.com>
* Fix drone
* fix test
* Fix test
* display the error on console
* Move minio test to amd64 since minio docker don't support arm64
* don't change unrelated files
* Fix lint
* Fix build
* update go.mod and go.sum
* Use github.com/minio/minio-go/v6
* Remove unused function
* Upgrade minio to v7 and some other improvements
* fix lint
* Fix go mod
Co-authored-by: guillep2k <18600385+guillep2k@users.noreply.github.com>
Co-authored-by: Tyler <tystuyfzand@gmail.com>
* Refactor Cron and merge dashboard tasks
* Merge Cron and Dashboard tasks
* Make every cron task report a system notice on completion
* Refactor the creation of these tasks
* Ensure that execution counts of tasks is correct
* Allow cron tasks to be started from the cron page
* golangci-lint fixes
* Enforce that only one task with the same name can be registered
Signed-off-by: Andrew Thornton <art27@cantab.net>
* fix name check
Signed-off-by: Andrew Thornton <art27@cantab.net>
* as per @guillep2k
* as per @lafriks
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Add git.CommandContext variants
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: Lauris BH <lauris@nix.lv>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
There is a potential memory leak in `Workerpool` due to the intricacies of
`time.Timer` stopping.
Whenever a `time.Timer` is `Stop`ped its channel must be cleared using a
`select` if the result of the `Stop()` is `false`.
Unfortunately in `Workerpool` these were checked the wrong way round.
However, there were a few other places that were not being checked.
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
* First stab at a Gitlab migrations interface.
* Modify JS to show migration for Gitlab
* Properly strip out #gitlab tag from repo name
* Working Gitlab migrations!
Still need to figure out how to hide tokens/etc from showing up in opts.CloneAddr
* Try #2 at trying to hide credentials.
CloneAddr was being used as OriginalURL.
Now passing OriginalURL through from the form and saving it.
* Add go-gitlab dependency
* Vendor go-gitlab
* Use gitlab.BasicAuthClient
Correct CloneURL.
This should be functioning!
Previous commits fixed "Migrated from"
from including the migration credentials.
* Replaced repoPath with repoID globally.
RepoID is grabbed in NewGitlabDownloader
* Logging touchup
* Properly set private repo status.
Properly set milestone deadline time.
Consistently use Gitlab username for 'Name'.
* Add go-gitlab vendor cache
* Fix PR migrations:
- Count of issues is kept to set a non-conflicting PR.ID
- Bool is used to tell whether to fetch Issue or PR comments
* Ensure merged PRs are closed and set with the proper time
* Remove copyright and some commented code
* Rip out '#gitlab' based self-hosted Gitlab support
* Hide given credentials for migrated repos.
CloneAddr was being saved as OriginalURL.
Now passing OriginalURL through from the form and
saving it in it's place
* Use asset.URL directly, no point in parsing.
Opened PRs should fall through to false.
* Fix importing Milestones.
Allow importing using Personal Tokens or anonymous access.
* Fix Gitlab Milestone migration if DueDate isn't set
* Empty Milestone due dates properly return nil, not zero time
* Add GITLAB_READ_TOKEN to drone unit-test step
* Add working gitlab_test.go.
A Personal Access Token, given in env variable GITLAB_READ_TOKEN
is required to run the test.
* Fix linting issues
* Add modified JS files
* Remove pre-build JS files
* Only merged PRs are marged as merged/closed
* Test topics
* Skip test if gitlab is inaccessible
* Grab personal token from username, not password.
Matches Github migration implementation
* Add SetContext() to GitlabDownloader.
* Checking Updated field in Issues.
* Actually fetch Issue Updated time from Gitlab
* Add Gitlab migration GetReviews() stub
* Fix Patch and Clone URLs
* check Updated too
* fix mod
* make vendor with go1.14
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: Lauris BH <lauris@nix.lv>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
* Fix InsertReleases Nil Insert on Attachments
* FIX "No element on slice when insert" & smal refactor
* again
* impruve
* rm useles newline
* Apply suggestions from code review
Co-Authored-By: zeripath <art27@cantab.net>
* process insert as a whole
Co-authored-by: zeripath <art27@cantab.net>
* Some refactor on git diff and ignore getting commit information failed on migrating pull request review comments
* fix test
* fix lint
* Change error log to warn
* fix typo
* Migrate reviews when migrating repository from github
* fix lint
* Added test and migration when external user login
* fix test
* fix commented state
* Some improvements
* fix bug when get pull request and ref original author on code comments
* Fix migrated line; Added comment for review
* Don't load all pull requests attributes
* Fix typo
* wrong change copy head
* fix tests
* fix reactions
* Fix test
* fix fmt
* fix review comment reactions
* ci: configure remaining github limmit
* prepend with github since package is common to all migrations
* add RefreshRate
* Update github.go
* add missing space
* go fmt
* Read env variable GITHUB_READ_TOKEN for token
* Update .drone.yml
* Fix missing updated time on migrated issues and comments
* Fix testing and missing updated on migrating pullrequest
Co-authored-by: Antoine GIRARD <sapk@users.noreply.github.com>
