KN4CK3R
440039c0cc
Add push to remote mirror repository ( #15157 )
...
* Added push mirror model.
* Integrated push mirror into queue.
* Moved methods into own file.
* Added basic implementation.
* Mirror wiki too.
* Removed duplicated method.
* Get url for different remotes.
* Added migration.
* Unified remote url access.
* Add/Remove push mirror remotes.
* Prevent hangs with missing credentials.
* Moved code between files.
* Changed sanitizer interface.
* Added push mirror backend methods.
* Only update the mirror remote.
* Limit refs on push.
* Added UI part.
* Added missing table.
* Delete mirror if repository gets removed.
* Changed signature. Handle object errors.
* Added upload method.
* Added "upload" unit tests.
* Added transfer adapter unit tests.
* Send correct headers.
* Added pushing of LFS objects.
* Added more logging.
* Simpler body handling.
* Process files in batches to reduce HTTP calls.
* Added created timestamp.
* Fixed invalid column name.
* Changed name to prevent xorm auto setting.
* Remove table header im empty.
* Strip exit code from error message.
* Added docs page about mirroring.
* Fixed date.
* Fixed merge errors.
* Moved test to integrations.
* Added push mirror test.
* Added test.
2021-06-14 19:20:43 +02:00
Norwin
cb940c4312
Encrypt migration credentials at rest ( #15895 )
...
* encrypt migration credentials in task persistence
Not sure this is the best approach, we could encrypt the entire
`PayloadContent` instead. Also instead of clearing individual fields in
payload content, we could just delete the task once it has
(successfully) finished..?
* remove credentials of past migrations
* only run DB migration for completed tasks
* fix binding
* add omitempty
* never serialize unencrypted credentials
* fix import order
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
Co-authored-by: zeripath <art27@cantab.net>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2021-05-31 04:25:47 -04:00
zeripath
f0e15250b9
Migrate to use jsoniter instead of encoding/json ( #14841 )
...
* Migrate to use jsoniter
* fix tests
* update gitea.com/go-chi/binding
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: 6543 <6543@obermui.de>
2021-03-01 22:08:10 +01:00
6543
2b4a08e962
Migrations: Use Process Manager to create own Context ( #13792 )
2020-12-02 18:36:06 +00:00
zeripath
f40a2a4404
Store task errors following migrations and display them ( #13246 )
...
* Store task errors following migrations and display them
When migrate tasks fail store the error in the task table
and ensure that they show on the status page.
Fix #13242
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Update web_src/js/index.js
* Hide the failed first
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
2020-10-23 19:46:35 -04:00
zeripath
7edf7eb46c
Log the underlying panic in runMigrateTask ( #13096 )
...
If there is a panic during runMigrateTask we should capture and log the underlying
panic error.
This PR ensures that the panic is logged and captured as part of the task message.
Fix #13095
Signed-off-by: Andrew Thornton <art27@cantab.net>
2020-10-11 14:51:13 -04:00
6543
fd60ebfe14
[API] Migration: Change ServiceType String ( #12672 )
...
* use different structs for MigrateRepoOptions on UI and API
* Fix TokenAuth and rename UID to an understandable Name
* fix swagger doc
* simplify & mk redable
* R E F A C T O R:
migration has now internal 3 structs to store its options:
* the Options for WebUI: modules/auth/repo_form.go
* the Options for API: modules/structs/repo.go
* the option struct with after validation for internal prossessing: modules/migrations/base/options.go
* Copyright Header
* Deprecate UID - add RepoOwner
* adopt repo.go -> migrate.go
* add comment about each struct purpose
* lint
2020-09-10 23:29:19 +01:00
Lunny Xiao
b465d0d787
Move create/fork repository from models to modules/repository ( #9489 )
...
* Move create/fork repository from models to modules/repository
* fix wrong reference
* fix test
* fix test
* fix lint
* Fix DBContext
* remove duplicated TestMain
* fix lint
* fix conflicts
2020-01-12 20:11:17 +08:00
zeripath
62eb1b0f25
Graceful Queues: Issue Indexing and Tasks ( #9363 )
...
* Queue: Add generic graceful queues with settings
* Queue & Setting: Add worker pool implementation
* Queue: Add worker settings
* Queue: Make resizing worker pools
* Queue: Add name variable to queues
* Queue: Add monitoring
* Queue: Improve logging
* Issues: Gracefulise the issues indexer
Remove the old now unused specific queues
* Task: Move to generic queue and gracefulise
* Issues: Standardise the issues indexer queue settings
* Fix test
* Queue: Allow Redis to connect to unix
* Prevent deadlock during early shutdown of issue indexer
* Add MaxWorker settings to queues
* Merge branch 'master' into graceful-queues
* Update modules/indexer/issues/indexer.go
Co-Authored-By: guillep2k <18600385+guillep2k@users.noreply.github.com>
* Update modules/indexer/issues/indexer.go
Co-Authored-By: guillep2k <18600385+guillep2k@users.noreply.github.com>
* Update modules/queue/queue_channel.go
Co-Authored-By: guillep2k <18600385+guillep2k@users.noreply.github.com>
* Update modules/queue/queue_disk.go
* Update modules/queue/queue_disk_channel.go
Co-Authored-By: guillep2k <18600385+guillep2k@users.noreply.github.com>
* Rename queue.Description to queue.ManagedQueue as per @guillep2k
* Cancel pool workers when removed
* Remove dependency on queue from setting
* Update modules/queue/queue_redis.go
Co-Authored-By: guillep2k <18600385+guillep2k@users.noreply.github.com>
* As per @guillep2k add mutex locks on shutdown/terminate
* move unlocking out of setInternal
* Add warning if number of workers < 0
* Small changes as per @guillep2k
* No redis host specified not found
* Clean up documentation for queues
* Update docs/content/doc/advanced/config-cheat-sheet.en-us.md
* Update modules/indexer/issues/indexer_test.go
* Ensure that persistable channel queue is added to manager
* Rename QUEUE_NAME REDIS_QUEUE_NAME
* Revert "Rename QUEUE_NAME REDIS_QUEUE_NAME"
This reverts commit 1f83b4fc9b
2020-01-07 12:23:09 +01:00
Lunny Xiao
ffc904b1e0
Sleep longer if request speed is over github limitation ( #9335 )
...
* Sleep longer if request speed is over github limitation
* improve code
* remove unused code
* fix lint
* Use github's rate limit remain value to determine how long to sleep
* Save reset time when finished github api request
* fix bug
* fix lint
* Add context.Context for sleep
* fix test
* improve code
* fix bug and lint
* fix import order
2019-12-16 23:16:54 -05:00
Lunny Xiao
f02138a148
Fix bug when migrate from API ( #8631 )
...
* fix bug when migrate from API
* fix test
* fix test
* improve
* fix error message
2019-11-08 17:21:00 -05:00
Lunny Xiao
f2a3abc683
Move migrating repository from frontend to backend ( #6200 )
...
* move migrating to backend
* add loading image when migrating and fix tests
* fix format
* fix lint
* add redis task queue support and improve docs
* add redis vendor
* fix vet
* add database migrations and fix app.ini sample
* add comments for task section on app.ini.sample
* Update models/migrations/v84.go
Co-Authored-By: lunny <xiaolunwen@gmail.com>
* Update models/repo.go
Co-Authored-By: lunny <xiaolunwen@gmail.com>
* move migrating to backend
* add loading image when migrating and fix tests
* fix fmt
* add redis task queue support and improve docs
* fix fixtures
* fix fixtures
* fix duplicate function on index.js
* fix tests
* rename repository statuses
* check if repository is being create when SSH request
* fix lint
* fix template
* some improvements
* fix template
* unified migrate options
* fix lint
* fix loading page
* refactor
* When gitea restart, don't restart the running tasks because we may have servel gitea instances, that may break the migration
* fix js
* Update models/repo.go
Co-Authored-By: guillep2k <18600385+guillep2k@users.noreply.github.com>
* Update docs/content/doc/advanced/config-cheat-sheet.en-us.md
Co-Authored-By: guillep2k <18600385+guillep2k@users.noreply.github.com>
* fix tests
* rename ErrTaskIsNotExist to ErrTaskDoesNotExist
* delete release after add one on tests to make it run happy
* fix tests
* fix tests
* improve codes
* fix lint
* fix lint
* fix migrations
2019-10-13 21:23:14 +08:00
