mirror of
https://codeberg.org/forgejo/forgejo.git
synced 2024-11-14 14:49:32 +01:00
e1fe3bbdc0
This is an implementation of a quota engine, and the API routes to manage its settings. This does *not* contain any enforcement code: this is just the bedrock, the engine itself. The goal of the engine is to be flexible and future proof: to be nimble enough to build on it further, without having to rewrite large parts of it. It might feel a little more complicated than necessary, because the goal was to be able to support scenarios only very few Forgejo instances need, scenarios the vast majority of mostly smaller instances simply do not care about. The goal is to support both big and small, and for that, we need a solid, flexible foundation. There are thee big parts to the engine: counting quota use, setting limits, and evaluating whether the usage is within the limits. Sounds simple on paper, less so in practice! Quota counting ============== Quota is counted based on repo ownership, whenever possible, because repo owners are in ultimate control over the resources they use: they can delete repos, attachments, everything, even if they don't *own* those themselves. They can clean up, and will always have the permission and access required to do so. Would we count quota based on the owning user, that could lead to situations where a user is unable to free up space, because they uploaded a big attachment to a repo that has been taken private since. It's both more fair, and much safer to count quota against repo owners. This means that if user A uploads an attachment to an issue opened against organization O, that will count towards the quota of organization O, rather than user A. One's quota usage stats can be queried using the `/user/quota` API endpoint. To figure out what's eating into it, the `/user/repos?order_by=size`, `/user/quota/attachments`, `/user/quota/artifacts`, and `/user/quota/packages` endpoints should be consulted. There's also `/user/quota/check?subject=<...>` to check whether the signed-in user is within a particular quota limit. Quotas are counted based on sizes stored in the database. Setting quota limits ==================== There are different "subjects" one can limit usage for. At this time, only size-based limits are implemented, which are: - `size:all`: As the name would imply, the total size of everything Forgejo tracks. - `size:repos:all`: The total size of all repositories (not including LFS). - `size:repos:public`: The total size of all public repositories (not including LFS). - `size:repos:private`: The total size of all private repositories (not including LFS). - `sizeall`: The total size of all git data (including all repositories, and LFS). - `sizelfs`: The size of all git LFS data (either in private or public repos). - `size:assets:all`: The size of all assets tracked by Forgejo. - `size:assets:attachments:all`: The size of all kinds of attachments tracked by Forgejo. - `size:assets:attachments:issues`: Size of all attachments attached to issues, including issue comments. - `size:assets:attachments:releases`: Size of all attachments attached to releases. This does *not* include automatically generated archives. - `size:assets:artifacts`: Size of all Action artifacts. - `size:assets:packages:all`: Size of all Packages. - `size:wiki`: Wiki size Wiki size is currently not tracked, and the engine will always deem it within quota. These subjects are built into Rules, which set a limit on *all* subjects within a rule. Thus, we can create a rule that says: "1Gb limit on all release assets, all packages, and git LFS, combined". For a rule to stand, the total sum of all subjects must be below the rule's limit. Rules are in turn collected into groups. A group is just a name, and a list of rules. For a group to stand, all of its rules must stand. Thus, if we have a group with two rules, one that sets a combined 1Gb limit on release assets, all packages, and git LFS, and another rule that sets a 256Mb limit on packages, if the user has 512Mb of packages, the group will not stand, because the second rule deems it over quota. Similarly, if the user has only 128Mb of packages, but 900Mb of release assets, the group will not stand, because the combined size of packages and release assets is over the 1Gb limit of the first rule. Groups themselves are collected into Group Lists. A group list stands when *any* of the groups within stand. This allows an administrator to set conservative defaults, but then place select users into additional groups that increase some aspect of their limits. To top it off, it is possible to set the default quota groups a user belongs to in `app.ini`. If there's no explicit assignment, the engine will use the default groups. This makes it possible to avoid having to assign each and every user a list of quota groups, and only those need to be explicitly assigned who need a different set of groups than the defaults. If a user has any quota groups assigned to them, the default list will not be considered for them. The management APIs =================== This commit contains the engine itself, its unit tests, and the quota management APIs. It does not contain any enforcement. The APIs are documented in-code, and in the swagger docs, and the integration tests can serve as an example on how to use them. Signed-off-by: Gergely Nagy <forgejo@gergo.csillger.hu>
236 lines
7.9 KiB
Go
236 lines
7.9 KiB
Go
// Copyright 2014 The Gogs Authors. All rights reserved.
|
|
// Copyright 2017 The Gitea Authors. All rights reserved.
|
|
// SPDX-License-Identifier: MIT
|
|
|
|
package setting
|
|
|
|
import (
|
|
"fmt"
|
|
"os"
|
|
"runtime"
|
|
"strings"
|
|
"time"
|
|
|
|
"code.gitea.io/gitea/modules/log"
|
|
"code.gitea.io/gitea/modules/user"
|
|
"code.gitea.io/gitea/modules/util"
|
|
)
|
|
|
|
var ForgejoVersion = "1.0.0"
|
|
|
|
// settings
|
|
var (
|
|
// AppVer is the version of the current build of Gitea. It is set in main.go from main.Version.
|
|
AppVer string
|
|
// AppBuiltWith represents a human-readable version go runtime build version and build tags. (See main.go formatBuiltWith().)
|
|
AppBuiltWith string
|
|
// AppStartTime store time gitea has started
|
|
AppStartTime time.Time
|
|
|
|
// Other global setting objects
|
|
|
|
CfgProvider ConfigProvider
|
|
RunMode string
|
|
RunUser string
|
|
IsProd bool
|
|
IsWindows bool
|
|
|
|
// IsInTesting indicates whether the testing is running. A lot of unreliable code causes a lot of nonsense error logs during testing
|
|
// TODO: this is only a temporary solution, we should make the test code more reliable
|
|
IsInTesting = false
|
|
)
|
|
|
|
func init() {
|
|
IsWindows = runtime.GOOS == "windows"
|
|
if AppVer == "" {
|
|
AppVer = "dev"
|
|
}
|
|
|
|
// We can rely on log.CanColorStdout being set properly because modules/log/console_windows.go comes before modules/setting/setting.go lexicographically
|
|
// By default set this logger at Info - we'll change it later, but we need to start with something.
|
|
log.SetConsoleLogger(log.DEFAULT, "console", log.INFO)
|
|
}
|
|
|
|
// IsRunUserMatchCurrentUser returns false if configured run user does not match
|
|
// actual user that runs the app. The first return value is the actual user name.
|
|
// This check is ignored under Windows since SSH remote login is not the main
|
|
// method to login on Windows.
|
|
func IsRunUserMatchCurrentUser(runUser string) (string, bool) {
|
|
if IsWindows || SSH.StartBuiltinServer {
|
|
return "", true
|
|
}
|
|
|
|
currentUser := user.CurrentUsername()
|
|
return currentUser, runUser == currentUser
|
|
}
|
|
|
|
// PrepareAppDataPath creates app data directory if necessary
|
|
func PrepareAppDataPath() error {
|
|
// FIXME: There are too many calls to MkdirAll in old code. It is incorrect.
|
|
// For example, if someDir=/mnt/vol1/gitea-home/data, if the mount point /mnt/vol1 is not mounted when Forgejo runs,
|
|
// then Forgejo will make new empty directories in /mnt/vol1, all are stored in the root filesystem.
|
|
// The correct behavior should be: creating parent directories is end users' duty. We only create sub-directories in existing parent directories.
|
|
// For quickstart, the parent directories should be created automatically for first startup (eg: a flag or a check of INSTALL_LOCK).
|
|
// Now we can take the first step to do correctly (using Mkdir) in other packages, and prepare the AppDataPath here, then make a refactor in future.
|
|
|
|
st, err := os.Stat(AppDataPath)
|
|
if os.IsNotExist(err) {
|
|
err = os.MkdirAll(AppDataPath, os.ModePerm)
|
|
if err != nil {
|
|
return fmt.Errorf("unable to create the APP_DATA_PATH directory: %q, Error: %w", AppDataPath, err)
|
|
}
|
|
return nil
|
|
}
|
|
|
|
if err != nil {
|
|
return fmt.Errorf("unable to use APP_DATA_PATH %q. Error: %w", AppDataPath, err)
|
|
}
|
|
|
|
if !st.IsDir() /* also works for symlink */ {
|
|
return fmt.Errorf("the APP_DATA_PATH %q is not a directory (or symlink to a directory) and can't be used", AppDataPath)
|
|
}
|
|
|
|
return nil
|
|
}
|
|
|
|
func InitCfgProvider(file string) {
|
|
var err error
|
|
if CfgProvider, err = NewConfigProviderFromFile(file); err != nil {
|
|
log.Fatal("Unable to init config provider from %q: %v", file, err)
|
|
}
|
|
CfgProvider.DisableSaving() // do not allow saving the CfgProvider into file, it will be polluted by the "MustXxx" calls
|
|
}
|
|
|
|
func MustInstalled() {
|
|
if !InstallLock {
|
|
log.Fatal(`Unable to load config file for a installed Forgejo instance, you should either use "--config" to set your config file (app.ini), or run "forgejo web" command to install Forgejo.`)
|
|
}
|
|
}
|
|
|
|
func LoadCommonSettings() {
|
|
if err := loadCommonSettingsFrom(CfgProvider); err != nil {
|
|
log.Fatal("Unable to load settings from config: %v", err)
|
|
}
|
|
}
|
|
|
|
// loadCommonSettingsFrom loads common configurations from a configuration provider.
|
|
func loadCommonSettingsFrom(cfg ConfigProvider) error {
|
|
// WARNING: don't change the sequence except you know what you are doing.
|
|
loadRunModeFrom(cfg)
|
|
loadLogGlobalFrom(cfg)
|
|
loadServerFrom(cfg)
|
|
loadSSHFrom(cfg)
|
|
|
|
mustCurrentRunUserMatch(cfg) // it depends on the SSH config, only non-builtin SSH server requires this check
|
|
|
|
loadOAuth2From(cfg)
|
|
loadSecurityFrom(cfg)
|
|
if err := loadAttachmentFrom(cfg); err != nil {
|
|
return err
|
|
}
|
|
if err := loadLFSFrom(cfg); err != nil {
|
|
return err
|
|
}
|
|
loadTimeFrom(cfg)
|
|
loadRepositoryFrom(cfg)
|
|
if err := loadAvatarsFrom(cfg); err != nil {
|
|
return err
|
|
}
|
|
if err := loadRepoAvatarFrom(cfg); err != nil {
|
|
return err
|
|
}
|
|
if err := loadPackagesFrom(cfg); err != nil {
|
|
return err
|
|
}
|
|
if err := loadActionsFrom(cfg); err != nil {
|
|
return err
|
|
}
|
|
loadUIFrom(cfg)
|
|
loadAdminFrom(cfg)
|
|
loadAPIFrom(cfg)
|
|
loadBadgesFrom(cfg)
|
|
loadMetricsFrom(cfg)
|
|
loadCamoFrom(cfg)
|
|
loadI18nFrom(cfg)
|
|
loadGitFrom(cfg)
|
|
loadMirrorFrom(cfg)
|
|
loadMarkupFrom(cfg)
|
|
loadQuotaFrom(cfg)
|
|
loadOtherFrom(cfg)
|
|
return nil
|
|
}
|
|
|
|
func loadRunModeFrom(rootCfg ConfigProvider) {
|
|
rootSec := rootCfg.Section("")
|
|
RunUser = rootSec.Key("RUN_USER").MustString(user.CurrentUsername())
|
|
|
|
// The following is a purposefully undocumented option. Please do not run Forgejo as root. It will only cause future headaches.
|
|
// Please don't use root as a bandaid to "fix" something that is broken, instead the broken thing should instead be fixed properly.
|
|
unsafeAllowRunAsRoot := ConfigSectionKeyBool(rootSec, "I_AM_BEING_UNSAFE_RUNNING_AS_ROOT")
|
|
unsafeAllowRunAsRoot = unsafeAllowRunAsRoot || util.OptionalBoolParse(os.Getenv("GITEA_I_AM_BEING_UNSAFE_RUNNING_AS_ROOT")).Value()
|
|
RunMode = os.Getenv("GITEA_RUN_MODE")
|
|
if RunMode == "" {
|
|
RunMode = rootSec.Key("RUN_MODE").MustString("prod")
|
|
}
|
|
|
|
// non-dev mode is treated as prod mode, to protect users from accidentally running in dev mode if there is a typo in this value.
|
|
RunMode = strings.ToLower(RunMode)
|
|
if RunMode != "dev" {
|
|
RunMode = "prod"
|
|
}
|
|
IsProd = RunMode != "dev"
|
|
|
|
// check if we run as root
|
|
if os.Getuid() == 0 {
|
|
if !unsafeAllowRunAsRoot {
|
|
// Special thanks to VLC which inspired the wording of this messaging.
|
|
log.Fatal("Forgejo is not supposed to be run as root. Sorry. If you need to use privileged TCP ports please instead use setcap and the `cap_net_bind_service` permission")
|
|
}
|
|
log.Critical("You are running Forgejo using the root user, and have purposely chosen to skip built-in protections around this. You have been warned against this.")
|
|
}
|
|
}
|
|
|
|
// HasInstallLock checks the install-lock in ConfigProvider directly, because sometimes the config file is not loaded into setting variables yet.
|
|
func HasInstallLock(rootCfg ConfigProvider) bool {
|
|
return rootCfg.Section("security").Key("INSTALL_LOCK").MustBool(false)
|
|
}
|
|
|
|
func mustCurrentRunUserMatch(rootCfg ConfigProvider) {
|
|
// Does not check run user when the "InstallLock" is off.
|
|
if HasInstallLock(rootCfg) {
|
|
currentUser, match := IsRunUserMatchCurrentUser(RunUser)
|
|
if !match {
|
|
log.Fatal("Expect user '%s' but current user is: %s", RunUser, currentUser)
|
|
}
|
|
}
|
|
}
|
|
|
|
// LoadSettings initializes the settings for normal start up
|
|
func LoadSettings() {
|
|
initAllLoggers()
|
|
|
|
loadDBSetting(CfgProvider)
|
|
loadServiceFrom(CfgProvider)
|
|
loadOAuth2ClientFrom(CfgProvider)
|
|
loadCacheFrom(CfgProvider)
|
|
loadSessionFrom(CfgProvider)
|
|
loadCorsFrom(CfgProvider)
|
|
loadMailsFrom(CfgProvider)
|
|
loadProxyFrom(CfgProvider)
|
|
loadWebhookFrom(CfgProvider)
|
|
loadMigrationsFrom(CfgProvider)
|
|
loadIndexerFrom(CfgProvider)
|
|
loadTaskFrom(CfgProvider)
|
|
LoadQueueSettings()
|
|
loadProjectFrom(CfgProvider)
|
|
loadMimeTypeMapFrom(CfgProvider)
|
|
loadFederationFrom(CfgProvider)
|
|
loadF3From(CfgProvider)
|
|
}
|
|
|
|
// LoadSettingsForInstall initializes the settings for install
|
|
func LoadSettingsForInstall() {
|
|
loadDBSetting(CfgProvider)
|
|
loadServiceFrom(CfgProvider)
|
|
loadMailerFrom(CfgProvider)
|
|
}
|