meissa/c4k-forgejo
6
0
Fork 0
Code Issues Pull requests Projects Releases 27 Packages Wiki Activity

Merge branch 'main' of ssh://repo.prod.meissa.de:2222/meissa/c4k-forgejo

Browse source
This commit is contained in:
Michael Jerger 2024-08-05 08:58:47 +02:00
commit 2b8de6b907
27 changed files with 481 additions and 162 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

13
.gitlab-ci.yml
View file

@ -123,9 +123,10 @@ forgejo-backup-image-publish:
script:
- cd infrastructure/backup && pyb image publish
forgejo-federated-image-publish:
<<: *img
<<: *tag_only
stage: image
script:
- cd infrastructure/federated && pyb image publish
# This is currently not needed
#forgejo-federated-image-publish:
# <<: *img
# <<: *tag_only
# stage: image
# script:
# - cd infrastructure/federated && pyb image publish

5
README.md
View file

@ -35,6 +35,11 @@ After having deployed the yaml-file generated by the c4k-forgejo module you need
* The SSH-URL for a repo has the format: "ssh://git@domain:2222/[username]/[repo].git
Example: "git clone ssh://git@repo.test.meissa.de:2222/myuser/c4k-forgejo.git"
### Add Impressum
In order to customize the UI e.g. for adding an Impressum, see the [Forgejo Docs](https://forgejo.org/docs/latest/developer/customization/#adding-links-and-tabs).
The individually needed files have to be added by hand into the directory `/data/gitea/templates/custom/` in the forgejo Pod. Since a PV is mounted under `/data`, these ui customizations are persisted.
## Development & mirrors
Development happens at: https://repo.prod.meissa.de/meissa/c4k-forgejo

115
doc/Runbook_UpgradeFrom1.19To7.0.5.md Normal file
View file

@ -0,0 +1,115 @@
# Playbook Upgrade from 1.19 to 7.0.5
## Info: Relevant Breaking Changes:
* 1.19.3:Current version
* 1.20.1-0: Breaking https://codeberg.org/forgejo/forgejo/src/branch/forgejo/RELEASE-NOTES.md#1-20-1-0
* 1.21.1-0: https://codeberg.org/forgejo/forgejo/src/branch/forgejo/RELEASE-NOTES.md#1-21-1-0
* 7.0.0: https://codeberg.org/forgejo/forgejo/src/branch/forgejo/RELEASE-NOTES.md#7-0-0
* 8.0.0: https://codeberg.org/forgejo/forgejo/src/branch/forgejo/RELEASE-NOTES.md#8-0-0
## Preparations
1. Stop Forgejo Prod: `k scale -n forgejo deployment forgejo --replicas=0`
1. Disable Backup Cron: `k patch -n forgejo cronjobs forgejo-backup -p '{"spec" : {"suspend" : true }}'`
1. Scale up Backup-Restore Deployment: `kubectl scale -n forgejo deployment backup-restore --replicas=1`
1. Execute Manual Backup: `kubectl exec -n forgejo -it backup-restore-... -- /usr/local/bin/backup.sh`
### Create 2nd Repo Prod Server
1. Terraform Preparations for 2nd Server: TODO
1. Install c4k-forgejo Version `3.5.0`!
with config `"forgejo-image-version-overwrite": "1.19.3-0"` (in server-setup)
1. Stop Forgejo Deployment: `k scale -n forgejo deployment forgejo --replicas=0`
1. Disable Backup Cron: `k patch -n forgejo cronjobs forgejo-backup -p '{"spec" : {"suspend" : true }}'`
1. Scale up Backup-Restore Deployment: `kubectl scale -n forgejo deployment backup-restore --replicas=1`
1. Restore Forgejo Backup: See [BackupAndRestore.md](BackupAndRestore.md)
1. Check for `..._INSTALL_LOCK: true` in ConfigMap `forgejo-env`
1. Scale up Forgejo Deployment and check for (startup) problems: `k scale -n forgejo deployment forgejo --replicas=1`
## Upgrade to 1.20.1-0
1. Scale down Forgejo Deployment: `k scale -n forgejo deployment forgejo --replicas=0`
1. Adjust configmap: `k edit -n forgejo cm forgejo-env`
1. Remove `FORGEJO__database__CHARSET: utf8` (This was a misconfiguration, since this option only had effect for mysql dbs)
1. Change `FORGEJO__mailer__MAILER_TYPE: smtp+startls` TO `FORGEJO__mailer__PROTOCOL: smtp+starttls` (Missed deprecation from 1.19)
1. Change `FORGEJO__service__EMAIL_DOMAIN_WHITELIST: repo.test.meissa.de` TO `FORGEJO__service__EMAIL_DOMAIN_ALLOWLIST: repo.test.meissa.de` (Fallback deprecation in 1.21)
1. Delete app.ini: `k exec -n forgejo -it backup-restore-... -- rm /var/backups/gitea/conf/app.ini`
1. Set version to `1.20.1-0` with `k edit -n forgejo deployment forgejo`
1. Scale up Forgejo Deployment: `k scale -n forgejo deployment forgejo --replicas=1`
1. Check for errors: `k logs -n forgejo forgejo-...`
## Upgrade to 1.21.1-0
1. Scale down Forgejo Deployment: `k scale -n forgejo deployment forgejo --replicas=0`
1. Delete app.ini: `k exec -n forgejo -it backup-restore-... -- rm /var/backups/gitea/conf/app.ini`
1. Set version to `1.21.1-0` with `k edit -n forgejo deployment forgejo`
1. Scale up Forgejo Deployment: `k scale -n forgejo deployment forgejo --replicas=1`
1. Check for errors: `k logs -n forgejo forgejo-...`
1. After upgrading, login as an admin, go to the `/admin` page and click run `Sync missed branches from git data to databases` (`Fehlende Branches aus den Git-Daten in die Datenbank synchronisieren`). If this is not done there will be messages such as `LoadBranches: branch does not exist in the logs`.
## Upgrade to 7.0.0
1. Scale down Forgejo Deployment: `k scale -n forgejo deployment forgejo --replicas=0`
1. Adjust configmap: `k edit -n forgejo cm forgejo-env`
1. Change `FORGEJO__oauth2__ENABLE: "true"` TO `FORGEJO__oauth2__ENABLED: "true"`
1. Delete app.ini: `k exec -n forgejo -it backup-restore-... -- rm /var/backups/gitea/conf/app.ini`
1. Set version to `7.0.0` with `k edit -n forgejo deployment forgejo`
1. Scale up Forgejo Deployment: `k scale -n forgejo deployment forgejo --replicas=1`
1. Check for errors: `k logs -n forgejo forgejo-...`
## Upgrade to 8.0.0 (no relevant breaking changes)
1. Scale down Forgejo Deployment: `k scale -n forgejo deployment forgejo --replicas=0`
1. Delete app.ini: `k exec -n forgejo -it backup-restore-... -- rm /var/backups/gitea/conf/app.ini`
1. Set version to `8.0.0` with `k edit -n forgejo deployment forgejo`
1. Scale up Forgejo Deployment: `k scale -n forgejo deployment forgejo --replicas=1`
1. Check for errors: `k logs -n forgejo forgejo-...`
## Enable Federation
1. Scale down Forgejo Deployment: `k scale -n forgejo deployment forgejo --replicas=0`
1. Adjust configmap: `k edit -n forgejo cm forgejo-env`
1. Change `FORGEJO__federation__ENABLED: "false"` TO `FORGEJO__federation__ENABLED: "true"`
1. Delete app.ini: `k exec -n forgejo -it backup-restore-... -- rm /var/backups/gitea/conf/app.ini`
1. Scale up Forgejo Deployment: `k scale -n forgejo deployment forgejo --replicas=1`
1. Check for errors: `k logs -n forgejo forgejo-...`
## Post Work
1. Switch DNS to new server
1. Reenable Backup Cron on new server: `k patch -n forgejo cronjobs forgejo-backup -p '{"spec" : {"suspend" : false }}'`
1. Execute manual Backup on new server: `kubectl exec -n forgejo -it backup-restore-... -- /usr/local/bin/backup.sh`
1. Scale down Backup-Restore Deployment: `kubectl scale -n forgejo deployment backup-restore --replicas=1`
1. The scope of all access tokens might (invisibly) have changed (in v1.20). Thus, rotate all tokens!
1. Users should check their ssh keys: if they use rsa keys the minimum length should be 3072 bits! However, shorter keys should still work.
## Known Errors
### Error in v1.20.1-0
In the logs the following error can be found. This will be resolved automatically with the next upgrade (v1.21).
```
2024/07/08 08:31:30 ...g/config_provider.go:321:deprecatedSetting() [E] Deprecated fallback `[log]` `ROUTER` present. Use `[log]` `logger.router.MODE` instead. This fallback will be/has been removed in 1.21
```
# Add Shynet Analytics
1. Log into shynet & create new Service
1. Copy the generated html snippet and save it somewhere you remember
1. SSH into prod server
1. Make the necessary folders and files in forgejo data dir:
1. `kubectl exec -n forgejo -it forgejo-... -- bash`
1. `mkdir -p /data/gitea/templates/custom`
1. `touch /data/gitea/templates/custom/footer.tmpl`
1. Open the `footer.tmpl` and paste the saved snippet
1. Restart the pod
1. `k scale -n forgejo deployment forgejo --replicas=0`
1. `k scale -n forgejo deployment forgejo --replicas=1`
1. Add Information about analytics: Clone Datenschutz Repo
1. `git clone ssh://git@repo.prod.meissa.de:2222/meissa/Datenschutz.git`
1. Merge forgejo-upgrade into main
1. `git merge forgejo-upgrade`
1. Push to origin
1. `git push`

176
doc/Upgrading.md
View file

@ -17,5 +17,179 @@ kubectl scale deployment forgejo --replicas=1
```
Logging into the admin account should now show the new version.
You may want to update your c4k-forgejo resources to reflect the changes made on the cluster.
## Upgrading from 1.19
### Config related issues with c4k-forgejo v3.2.2
These errors show in the log, when just upgrading to forgejo v7.0.4 from 1.19 without changing the config.
The related config options are listed below the errors.
- Oauth2: ENABLED instead of ENABLE
- `FORGEJO__oauth2__ENABLED: "true"`
- [E] Deprecated config option `[log]` `ROUTER` present. Use `[log]` `logger.router.MODE` instead.
- `FORGEJO__log_0x2E_logger_0x2E_router__MODE: console, file`
- [E] Deprecated config option `[service]` `EMAIL_DOMAIN_WHITELIST` present. Use `[service]
` `EMAIL_DOMAIN_ALLOWLIST` instead.
- `FORGEJO__service__EMAIL_DOMAIN_ALLOWLIST: YOUR_ALLOW_LIST`
- [E] Deprecated config option `[mailer]` `MAILER_TYPE` present. Use `[mailer]` `PROTOCOL`
instead.
- [E] Deprecated fallback `[mailer]` `PROTOCOL = smtp+startls` present. Use `[mailer]` `PROTOCOL = smtp+starttls`` instead.
- `FORGEJO__mailer__PROTOCOL: smtp+starttls`
### Breaking Changes since 1.19
#### 1.19.3 & 1.19.4: Version installed by c4k-forgejo v3.2.2
#### 1.20.1-0: Breaking https://codeberg.org/forgejo/forgejo/src/branch/forgejo/RELEASE-NOTES.md#1-20-1-0
##### app.ini
- Check [queue] section - n/e
- Check [repository.editor] - n/e
- Check [storage] - n/e
- Check ssh_keygen_path in app.ini - n/e
- Is WORK_PATH set? Or app.ini writeable by forgejo server user?
- 1. No
- 2. Yes
- If not, it shows in the logs starting with: `Unable to update WORK_PATH`
- Also ssh pushing will likely fail
- *test ssh*
- Set logger.router.mode as described in environment-to-ini
- See: https://codeberg.org/forgejo/forgejo/src/branch/forgejo/contrib/environment-to-ini
- Check [git.reflog] and maybe move to [git.config] - n/e
- Check [indexer], [mailer], [repository] - n/e
##### tokens
- Scoped and personal access tokens were refactored
- Scope may change, if we have tokens they should be rotated
#### 1.21.1-0: https://codeberg.org/forgejo/forgejo/src/branch/forgejo/RELEASE-NOTES.md#1-21-1-0
##### custom themes
- Move to `custom/public/assets/`
##### git branches
- `/admin` page and click run Sync missed branches from git data to databases.
##### db - mysql
- c4k uses postgres
##### ssh server
- We don't use host cert used for auth
##### ssh keys
- All users need to check their key length, now 3072
##### tokens
- Finer restrictions might now return 404 errors on users tokens in certain teams with certain restrictions
#### 7.0.0: https://codeberg.org/forgejo/forgejo/src/branch/forgejo/RELEASE-NOTES.md#7-0-0
##### webhooks
- Do we use webhooks?
##### db
- Psql min ver is 12, c4k-common uses 14+
##### api
- [/repos/{owner}/{repo}/releases](https://code.forgejo.org/api/swagger/#/repository/repoListReleases)
- [/repos/{owner}/{repo}/push_mirrors](https://code.forgejo.org/api/swagger/#/repository/repoListPushMirrors)
- Application profiling
##### repos
- Do we have repo descriptions?
- https://codeberg.org/forgejo/forgejo/commit/1075ff74b5050f671c5f9824ae39390230b3c85d
##### app.ini
- Check [ui] - n/e
### Vor dem Upgrade
- Host cert used for auth? - no
- Do we use webhooks? - no
- Do we use:
- [/repos/{owner}/{repo}/releases - repoListReleases](https://code.forgejo.org/api/swagger/#/repository/repoListReleases) - no
- In the ListReleases, the `per_page` parameter has been decoupled from the `limit` parameter, we do not use the repoListReleases endpoint
- In the `ArtifactDeploymentApi` in dda-devops-build we only use the `POST` method
- The respective endpoint is [repoCreateRelease](https://code.forgejo.org/api/swagger/#/repository/repoCreateRelease)
- [`/repos/{owner}/{repo}/push_mirrors`](https://code.forgejo.org/api/swagger/#/repository/repoListPushMirrors) - no
- Application profiling - no
- Do we have repo descriptions? - yes
- There is now a sanitizer that only allows links, emphasis, code and emojis
- See: https://codeberg.org/forgejo/forgejo/commit/1075ff74b5050f671c5f9824ae39390230b3c85d
- Our repository descriptions are mostly plaintext and links
### Upgrade plan
TEST indicates actions that only apply to the test server and are ignored in PROD.
PROD indicates actions that only apply to the prod server and are ignored in TEST.
See also the overview for upgrading: https://forgejo.org/docs/latest/admin/upgrade/
- Set up Forgejo server with c4k-forgejo v3.2.2
- Has Forgejo v1.19
- TEST
- Delete old remote ids
- `ssh-keygen -f "/home/${USER}/.ssh/known_hosts" -R "repo.test.meissa.de"`
- Ssh to server
- Forgejo pod downscale
- `k scale deployment forgejo --replicas=0`
- Install lock off
- `k edit cm forgejo-env`
- Set to `FORGEJO__security__INSTALL_LOCK: "false"`
- Forgejo pod upscale
- `k scale deployment forgejo --replicas=1`
- Create admin test or prod admin and install forgejo
- `gopass show server/meissa/forgejo-test` bzw `-prod`
- Forgejo pod downscale
- Install lock on
- Set to `FORGEJO__security__INSTALL_LOCK: "true"`
- TEST
- Forgejo pod upscale
- Log in
- Make Ssh keys
- ed_xyz
- rsa mit 2048
- rsa mit 4096
- Create repos
- Forgejo pod downscale
- PROD
- Backup pod upscale
- `k scale deployment backup-restore --replicas=1`
- Restore backups
- Delete or rename app.ini's in the pod
- Backup pod downscale
- `k scale deployment backup-restore --replicas=0`
- Set image version to 7.0.4 in forgejo deployment
- `k edit deployment.apps forgejo`
- Update configmap:
- Double check install lock enabled
- `FORGEJO__oauth2__ENABLED: "true"`
- `FORGEJO__log_0x2E_logger_0x2E_router__MODE: console, file`
- `FORGEJO__service__EMAIL_DOMAIN_ALLOWLIST:`
- `FORGEJO__mailer__PROTOCOL: smtp+starttls`
- `FORGEJO__federation__ENABLED: true`
- TEST
- Backup pod upscale
- Delete or rename app.ini's in the pod
- Backup pod downscale
- Forgejo pod upscale
- Migrations happen automatically
- `/admin` page and click run Sync missed branches from git data to databases
- and **Sync missed tags ...*
- Rsa keys with size 2048 can not be added anymore. However, it seems they still can be used if they are on the server
- Team members having app tokens need to recreate them with proper scopes
- Add analytics: https://forgejo.org/docs/latest/admin/customization/

2
infrastructure/backup/build.py
View file

@ -6,7 +6,7 @@ from ddadevops import *
name = "c4k-forgejo"
MODULE = "backup"
PROJECT_ROOT_PATH = "../.."
version = "3.2.3-dev"
version = "3.5.1-dev"
@init

2
infrastructure/federated/build.py
View file

@ -6,7 +6,7 @@ from ddadevops import *
name = 'c4k-forgejo'
MODULE = 'federated'
PROJECT_ROOT_PATH = '../..'
version = "3.2.3-dev"
version = "3.5.1-dev"
@init
def initialize(project):

2
package.json
View file

@ -2,7 +2,7 @@
"name": "c4k-forgejo",
"description": "Generate c4k yaml for a forgejo deployment.",
"author": "meissa GmbH",
"version": "3.2.3-SNAPSHOT",
"version": "3.5.1-SNAPSHOT",
"homepage": "https://gitlab.com/domaindrivenarchitecture/c4k-forgejo#readme",
"repository": "https://www.npmjs.com/package/c4k-forgejo",
"license": "APACHE2",

15
project.clj
View file

@ -1,16 +1,17 @@
(defproject org.domaindrivenarchitecture/c4k-forgejo "3.2.3-SNAPSHOT"
(defproject org.domaindrivenarchitecture/c4k-forgejo "3.5.1-SNAPSHOT"
:description "forgejo c4k-installation package"
:url "https://domaindrivenarchitecture.org"
:license {:name "Apache License, Version 2.0"
:url "https://www.apache.org/licenses/LICENSE-2.0.html"}
:dependencies [[org.clojure/clojure "1.11.2" :scope "provided"]
[org.clojure/tools.reader "1.4.1"]
[org.domaindrivenarchitecture/c4k-common-clj "6.2.3"]
:dependencies [[org.clojure/clojure "1.11.3" :scope "provided"]
[org.clojure/tools.reader "1.4.2"]
[org.domaindrivenarchitecture/c4k-common-clj "7.0.0"]
[hickory "0.7.1" :exclusions [viebel/codox-klipse-theme]]]
:target-path "target/%s/"
:source-paths ["src/main/cljc"
"src/main/clj"]
:resource-paths ["src/main/resources"]
:resource-paths ["src/main/resources"
"project.clj"]
:repositories [["snapshots" :clojars]
["releases" :clojars]]
:deploy-repositories [["snapshots" {:sign-releases false :url "https://clojars.org/repo"}]
@ -23,9 +24,9 @@
:main dda.c4k-forgejo.uberjar
:uberjar-name "c4k-forgejo-standalone.jar"
:dependencies [[org.clojure/tools.cli "1.1.230"]
[ch.qos.logback/logback-classic "1.5.3"
[ch.qos.logback/logback-classic "1.5.6"
:exclusions [com.sun.mail/javax.mail]]
[org.slf4j/jcl-over-slf4j "2.0.12"]
[org.slf4j/jcl-over-slf4j "2.0.13"]
[com.github.clj-easy/graal-build-time "1.0.5"]]}}
:release-tasks [["test"]
["vcs" "assert-committed"]

2
shadow-cljs.edn
View file

@ -4,7 +4,7 @@
"src/test/cljc"
"src/test/cljs"
"src/test/resources"]
:dependencies [[org.domaindrivenarchitecture/c4k-common-cljs "6.1.3"]
:dependencies [[org.domaindrivenarchitecture/c4k-common-cljs "7.0.0"]
[hickory "0.7.1"]]
:builds {:frontend {:target :browser
:modules {:main {:init-fn dda.c4k-forgejo.browser/init}}

5
src/main/clj/dda/c4k_forgejo/uberjar.clj
View file

@ -7,10 +7,11 @@
(set! *warn-on-reflection* true)
(defn -main [& cmd-args]
(uberjar/main-common
(uberjar/main-cm
"c4k-forgejo"
core/config?
core/auth?
core/config-defaults
core/k8s-objects
core/config-objects
core/auth-objects
cmd-args))

9
src/main/cljc/dda/c4k_forgejo/backup.cljc
View file

@ -4,12 +4,13 @@
[dda.c4k-common.yaml :as yaml]
[dda.c4k-common.base64 :as b64]
[dda.c4k-common.common :as cm]
[dda.c4k-common.predicate :as p]
#?(:cljs [dda.c4k-common.macros :refer-macros [inline-resources]])))
(s/def ::aws-access-key-id cm/bash-env-string?)
(s/def ::aws-secret-access-key cm/bash-env-string?)
(s/def ::restic-password cm/bash-env-string?)
(s/def ::restic-repository cm/bash-env-string?)
(s/def ::aws-access-key-id p/bash-env-string?)
(s/def ::aws-secret-access-key p/bash-env-string?)
(s/def ::restic-password p/bash-env-string?)
(s/def ::restic-repository p/bash-env-string?)
#?(:cljs
(defmethod yaml/load-resource :backup [resource-name]

64
src/main/cljc/dda/c4k_forgejo/core.cljc
View file

@ -1,14 +1,23 @@
(ns dda.c4k-forgejo.core
(:require
[clojure.spec.alpha :as s]
[dda.c4k-common.yaml :as yaml]
[dda.c4k-common.common :as cm]
[dda.c4k-common.monitoring :as mon]
[dda.c4k-forgejo.forgejo :as forgejo]
[dda.c4k-forgejo.backup :as backup]
[dda.c4k-common.postgres :as postgres]))
(:require
[clojure.spec.alpha :as s]
[dda.c4k-common.yaml :as yaml]
[dda.c4k-common.common :as cm]
[dda.c4k-common.monitoring :as mon]
[dda.c4k-forgejo.forgejo :as forgejo]
[dda.c4k-forgejo.backup :as backup]
[dda.c4k-common.postgres :as postgres]
[dda.c4k-common.namespace :as ns]))
(def config-defaults {:issuer "staging", :deploy-federated "false"})
(def config-defaults {:namespace "forgejo"
:issuer "staging"
:deploy-federated "false"
:federation-enabled "false"
:db-name "forgejo"
:pv-storage-size-gb 5
:pvc-storage-class-name ""
:postgres-image "postgres:14"
:postgres-size :2gb})
(def rate-limit-defaults {:max-rate 10, :max-concurrent-requests 5})
(def config? (s/keys :req-un [::forgejo/fqdn
@ -18,8 +27,10 @@
::forgejo/service-noreply-address]
:opt-un [::forgejo/issuer
::forgejo/deploy-federated
::forgejo/federation-enabled
::forgejo/default-app-name
::forgejo/service-domain-whitelist
::forgejo/forgejo-image-version-overwrite
::backup/restic-repository
::mon/mon-cfg]))
@ -31,32 +42,39 @@
(def vol? (s/keys :req-un [::forgejo/volume-total-storage-size]))
(defn k8s-objects [config auth] ; ToDo: ADR for generate functions - vector or no vector?
(defn config-objects [config] ; ToDo: ADR for generate functions - vector or no vector?
(let [storage-class (if (contains? config :postgres-data-volume-path) :manual :local-path)]
(map yaml/to-string
(filter #(not (nil? %))
(cm/concat-vec
[(postgres/generate-config {:postgres-size :2gb :db-name "forgejo"})
(postgres/generate-secret auth)
(ns/generate config)
[(postgres/generate-configmap config)
(when (contains? config :postgres-data-volume-path)
(postgres/generate-persistent-volume (select-keys config [:postgres-data-volume-path :pv-storage-size-gb])))
(postgres/generate-pvc {:pv-storage-size-gb 5
:pvc-storage-class-name storage-class})
(postgres/generate-deployment {:postgres-image "postgres:14"
:postgres-size :2gb})
(postgres/generate-pvc (merge config {:pvc-storage-class-name storage-class}))
(postgres/generate-deployment config)
(postgres/generate-service config)
(forgejo/generate-deployment config)
(forgejo/generate-service)
(forgejo/generate-service-ssh)
(forgejo/generate-data-volume config)
(forgejo/generate-appini-env config)
(forgejo/generate-secrets auth)
(forgejo/generate-rate-limit-middleware rate-limit-defaults)] ; this does not have a vector as output
(forgejo/generate-rate-limit-ingress-and-cert config) ; this function has a vector as output
(forgejo/generate-appini-env config)]
(forgejo/generate-ratelimit-ingress-and-cert config) ; this function has a vector as output
(when (contains? config :restic-repository)
[(backup/generate-config config)
(backup/generate-secret auth)
(backup/generate-cron)
(backup/generate-backup-restore-deployment config)])
(when (:contains? config :mon-cfg)
(mon/generate (:mon-cfg config) (:mon-auth auth))))))))
(when (contains? config :mon-cfg)
(mon/generate-config)))))))
(defn auth-objects [config auth]
(map yaml/to-string
(filter #(not (nil? %))
(cm/concat-vec
(ns/generate config)
[(postgres/generate-secret config auth)
(forgejo/generate-secrets auth)]
(when (contains? config :restic-repository)
[(backup/generate-secret auth)])
(when (contains? config :mon-cfg)
(mon/generate-auth (:mon-cfg config) (:mon-auth auth)))))))

81
src/main/cljc/dda/c4k_forgejo/forgejo.cljc
View file

@ -33,11 +33,13 @@
(s/def ::default-app-name string?)
(s/def ::fqdn pred/fqdn-string?)
(s/def ::deploy-federated boolean-string?)
(s/def ::federation-enabled boolean-string?)
(s/def ::mailer-from pred/bash-env-string?)
(s/def ::mailer-host pred/bash-env-string?)
(s/def ::mailer-port pred/bash-env-string?)
(s/def ::service-domain-whitelist domain-list?)
(s/def ::service-noreply-address string?)
(s/def ::forgejo-image-version-overwrite string?)
(s/def ::mailer-user pred/bash-env-string?)
(s/def ::mailer-pw pred/bash-env-string?)
(s/def ::issuer pred/letsencrypt-issuer?)
@ -52,8 +54,10 @@
::service-noreply-address]
:opt-un [::issuer
::deploy-federated
::federation-enabled
::default-app-name
::service-domain-whitelist]))
::service-domain-whitelist
::forgejo-image-version-overwrite]))
(def rate-limit-config? (s/keys :req-un [::max-rate
::max-concurrent-requests]))
@ -66,8 +70,18 @@
[total]
total)
(def federated-image-name "domaindrivenarchitecture/c4k-forgejo-federated:latest")
(def non-federated-image-name "codeberg.org/forgejo/forgejo:1.19")
(def federated-image-name "domaindrivenarchitecture/c4k-forgejo-federated")
(def federated-image-version "latest")
(def non-federated-image-name "codeberg.org/forgejo/forgejo")
(def non-federated-image-version "8.0")
(defn-spec generate-image-str string?
[config config?]
(let [{:keys [deploy-federated forgejo-image-version-overwrite]} config
deploy-federated-bool (boolean-from-string deploy-federated)]
(if deploy-federated-bool
(str federated-image-name ":" (or forgejo-image-version-overwrite federated-image-version))
(str non-federated-image-name ":" (or forgejo-image-version-overwrite non-federated-image-version)))))
#?(:cljs
(defmethod yaml/load-resource :forgejo [resource-name]
@ -76,7 +90,7 @@
(defn generate-appini-env
[config]
(let [{:keys [default-app-name
deploy-federated
federation-enabled
fqdn
mailer-from
mailer-host
@ -85,7 +99,7 @@
service-noreply-address]
:or {default-app-name "forgejo instance"
service-domain-whitelist fqdn}} config
deploy-federated-bool (boolean-from-string deploy-federated)]
federation-enabled-bool (boolean-from-string federation-enabled)]
(->
(yaml/load-as-edn "forgejo/appini-env-configmap.yaml")
(cm/replace-all-matching-values-by-new-value "APPNAME" default-app-name)
@ -97,7 +111,7 @@
(cm/replace-all-matching-values-by-new-value "WHITELISTDOMAINS" service-domain-whitelist)
(cm/replace-all-matching-values-by-new-value "NOREPLY" service-noreply-address)
(cm/replace-all-matching-values-by-new-value "IS_FEDERATED"
(if deploy-federated-bool
(if federation-enabled-bool
"true"
"false")))))
@ -109,40 +123,22 @@
mailer-pw]} auth]
(->
(yaml/load-as-edn "forgejo/secrets.yaml")
(cm/replace-all-matching-values-by-new-value "DBUSER" (b64/encode postgres-db-user))
(cm/replace-all-matching-values-by-new-value "DBPW" (b64/encode postgres-db-password))
(cm/replace-all-matching-values-by-new-value "MAILERUSER" (b64/encode mailer-user))
(cm/replace-all-matching-values-by-new-value "MAILERPW" (b64/encode mailer-pw)))))
(cm/replace-all-matching "DBUSER" (b64/encode postgres-db-user))
(cm/replace-all-matching "DBPW" (b64/encode postgres-db-password))
(cm/replace-all-matching "MAILERUSER" (b64/encode mailer-user))
(cm/replace-all-matching "MAILERPW" (b64/encode mailer-pw)))))
(defn generate-ingress-and-cert
[config]
(let [{:keys [fqdn]} config]
(ing/generate-ingress-and-cert
(merge
{:service-name "forgejo-service"
:service-port 3000
:fqdns [fqdn]}
config))))
(defn-spec generate-rate-limit-ingress-and-cert pred/map-or-seq?
(defn-spec generate-ratelimit-ingress-and-cert seq?
[config config?]
(->
(generate-ingress-and-cert config) ; returns a vector
(#(assoc-in % ; Attention: heavily relying on the output order of ing/generate-ingress-and-cert
[1 :metadata :annotations :traefik.ingress.kubernetes.io/router.middlewares]
(str
(-> (second %) :metadata :annotations :traefik.ingress.kubernetes.io/router.middlewares)
", default-ratelimit@kubernetescrd")))))
; using :average and :burst seems sensible, :period may be interesting for fine tuning later on
(defn-spec generate-rate-limit-middleware pred/map-or-seq?
[config rate-limit-config?]
(let [{:keys [max-rate max-concurrent-requests]} config]
(->
(yaml/load-as-edn "forgejo/middleware-ratelimit.yaml")
(cm/replace-key-value :average max-rate)
(cm/replace-key-value :burst max-concurrent-requests))))
(let [{:keys [fqdn max-rate max-concurrent-requests namespace]} config]
(ing/generate-simple-ingress (merge
{:service-name "forgejo-service"
:service-port 3000
:fqdns [fqdn]
:average-rate max-rate
:burst-rate max-concurrent-requests
:namespace namespace}
config))))
(defn-spec generate-data-volume pred/map-or-seq?
[config vol?]
@ -150,18 +146,13 @@
data-storage-size (data-storage-by-volume-size volume-total-storage-size)]
(->
(yaml/load-as-edn "forgejo/datavolume.yaml")
(cm/replace-all-matching-values-by-new-value "DATASTORAGESIZE" (str (str data-storage-size) "Gi")))))
(cm/replace-all-matching "DATASTORAGESIZE" (str (str data-storage-size) "Gi")))))
(defn-spec generate-deployment pred/map-or-seq?
[config config?]
(let [{:keys [deploy-federated]} config
deploy-federated-bool (boolean-from-string deploy-federated)]
(->
(yaml/load-as-edn "forgejo/deployment.yaml")
(cm/replace-all-matching-values-by-new-value "IMAGE_NAME"
(if deploy-federated-bool
federated-image-name
non-federated-image-name)))))
(cm/replace-all-matching "IMAGE_NAME" (generate-image-str config))))
(defn generate-service
[]

28
src/main/cljs/dda/c4k_forgejo/browser.cljs
View file

@ -79,8 +79,7 @@
(when (not (st/blank? app-name))
{:default-app-name app-name})
(when (not (st/blank? domain-whitelist))
{:service-domain-whitelist domain-whitelist})
)))
{:service-domain-whitelist domain-whitelist}))))
(defn validate-all! []
(br/validate! "fqdn" ::forgejo/fqdn)
@ -103,16 +102,21 @@
(defn init []
(br/append-hickory (generate-content-div))
(-> js/document
(.getElementById "generate-button")
(.addEventListener "click"
#(do (validate-all!)
(-> (cm/generate-common
(config-from-document)
(br/get-content-from-element "auth" :deserializer edn/read-string)
core/config-defaults
core/k8s-objects)
(br/set-output!)))))
(let [config-only false
auth-only false]
(-> js/document
(.getElementById "generate-button")
(.addEventListener "click"
#(do (validate-all!)
(-> (cm/generate-cm
(config-from-document)
(br/get-content-from-element "auth" :deserializer edn/read-string)
core/config-defaults
core/config-objects
core/auth-objects
config-only
auth-only)
(br/set-output!))))))
(add-validate-listener "fqdn")
(add-validate-listener "deploy-federated")
(add-validate-listener "mailer-from")

1
src/main/resources/backup/backup-restore-deployment.yaml
View file

@ -2,6 +2,7 @@ apiVersion: apps/v1
kind: Deployment
metadata:
name: backup-restore
namespace: forgejo
spec:
replicas: 0
selector:

1
src/main/resources/backup/config.yaml
View file

@ -2,6 +2,7 @@ apiVersion: v1
kind: ConfigMap
metadata:
name: backup-config
namespace: forgejo
labels:
app.kubernetes.io/name: backup
app.kubernetes.io/part-of: forgejo

1
src/main/resources/backup/cron.yaml
View file

@ -2,6 +2,7 @@ apiVersion: batch/v1
kind: CronJob
metadata:
name: forgejo-backup
namespace: forgejo
labels:
app.kubernetes.part-of: forgejo
spec:

1
src/main/resources/backup/secret.yaml
View file

@ -2,6 +2,7 @@ apiVersion: v1
kind: Secret
metadata:
name: backup-secret
namespace: forgejo
type: Opaque
data:
aws-access-key-id: aws-access-key-id

9
src/main/resources/forgejo/appini-env-configmap.yaml
View file

@ -2,7 +2,7 @@ apiVersion: v1
kind: ConfigMap
metadata:
name: forgejo-env
namespace: default
namespace: forgejo
data:
#[admin]
FORGEJO__admin__DEFAULT_EMAIL_NOTIFICATIONS: "enabled" # Default configuration for email notifications for users (user configurable). Options: enabled, onmention, disabled
@ -16,7 +16,6 @@ data:
FORGEJO__database__NAME: forgejo
FORGEJO__database__LOG_SQL: "false"
FORGEJO__database__SSL_MODE: disable
FORGEJO__database__CHARSET: utf8
#[DEFAULT]
APP_NAME: APPNAME
@ -37,12 +36,12 @@ data:
#[mailer]
FORGEJO__mailer__ENABLED: "true"
FORGEJO__mailer__FROM: FROM
FORGEJO__mailer__MAILER_TYPE: smtp+startls
FORGEJO__mailer__PROTOCOL: smtp+starttls
FORGEJO__mailer__SMTP_ADDR: MAILERHOST
FORGEJO__mailer__SMTP_PORT: MAILERPORT
#[oauth2]
FORGEJO__oauth2__ENABLE: "true"
FORGEJO__oauth2__ENABLED: "true"
#[openid]
FORGEJO__openid__ENABLE_OPENID: "true"
@ -76,7 +75,7 @@ data:
FORGEJO__service__REQUIRE_SIGNIN_VIEW: "false"
FORGEJO__service__REGISTER_EMAIL_CONFIRM: "true"
FORGEJO__service__ENABLE_NOTIFY_MAIL: "true"
FORGEJO__service__EMAIL_DOMAIN_WHITELIST: WHITELISTDOMAINS
FORGEJO__service__EMAIL_DOMAIN_ALLOWLIST: WHITELISTDOMAINS
FORGEJO__service__ALLOW_ONLY_EXTERNAL_REGISTRATION: "false"
FORGEJO__service__ENABLE_BASIC_AUTHENTICATION: "true"
FORGEJO__service__ENABLE_CAPTCHA: "false"

2
src/main/resources/forgejo/datavolume.yaml
View file

@ -2,7 +2,7 @@ apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: forgejo-data-pvc
namespace: default
namespace: forgejo
labels:
app: forgejo
spec:

2
src/main/resources/forgejo/deployment.yaml
View file

@ -2,7 +2,7 @@ apiVersion: apps/v1
kind: Deployment
metadata:
name: forgejo
namespace: default
namespace: forgejo
labels:
app: forgejo
spec:

8
src/main/resources/forgejo/middleware-ratelimit.yaml
View file

@ -1,8 +0,0 @@
apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
name: ratelimit
spec:
rateLimit: # Config options for rate limiting: https://doc.traefik.io/traefik/middlewares/http/ratelimit/
average: AVG
burst: BRS

1
src/main/resources/forgejo/secrets.yaml
View file

@ -2,6 +2,7 @@ apiVersion: v1
kind: Secret
metadata:
name: forgejo-secrets
namespace: forgejo
data:
FORGEJO__database__USER: DBUSER
FORGEJO__database__PASSWD: DBPW

2
src/main/resources/forgejo/service-ssh.yaml
View file

@ -2,7 +2,7 @@ kind: Service
apiVersion: v1
metadata:
name: forgejo-ssh-service
namespace: default
namespace: forgejo
annotations:
metallb.universe.tf/allow-shared-ip: "shared-ip-service-group"
metallb.universe.tf/address-pool: public

2
src/main/resources/forgejo/service.yaml
View file

@ -2,7 +2,7 @@ kind: Service
apiVersion: v1
metadata:
name: forgejo-service
namespace: default
namespace: forgejo
spec:
selector:
app: forgejo

2
src/test/cljc/dda/c4k_forgejo/backup_test.cljc
View file

@ -13,6 +13,7 @@
:kind "ConfigMap",
:metadata
{:name "backup-config",
:namespace "forgejo",
:labels
#:app.kubernetes.io{:name "backup", :part-of "forgejo"}},
:data {:restic-repository "s3:s3.amazonaws.com/backup/federated-repo"}}
@ -23,6 +24,7 @@
:kind "ConfigMap",
:metadata
{:name "backup-config",
:namespace "forgejo",
:labels
#:app.kubernetes.io{:name "backup", :part-of "forgejo"}},
:data {:restic-repository "s3:s3.amazonaws.com/backup/repo"}}

74
src/test/cljc/dda/c4k_forgejo/forgejo_test.cljc
View file

@ -12,6 +12,40 @@
(st/instrument `cut/generate-ingress)
(st/instrument `cut/generate-secrets)
(deftest should-generate-image-str
(testing "non-federated-image"
(is (= "codeberg.org/forgejo/forgejo:8.0"
(cut/generate-image-str {:fqdn "test.de"
:mailer-from ""
:mailer-host "m.t.de"
:mailer-port "123"
:service-noreply-address ""
:deploy-federated "false"})))
(is (= "codeberg.org/forgejo/forgejo:1.19.3-0"
(cut/generate-image-str {:fqdn "test.de"
:mailer-from ""
:mailer-host "m.t.de"
:mailer-port "123"
:service-noreply-address ""
:deploy-federated "false"
:forgejo-image-version-overwrite "1.19.3-0"}))))
(testing "federated-image"
(is (= "domaindrivenarchitecture/c4k-forgejo-federated:latest"
(cut/generate-image-str {:fqdn "test.de"
:mailer-from ""
:mailer-host "m.t.de"
:mailer-port "123"
:service-noreply-address ""
:deploy-federated "true"})))
(is (= "domaindrivenarchitecture/c4k-forgejo-federated:3.2.0"
(cut/generate-image-str {:fqdn "test.de"
:mailer-from ""
:mailer-host "m.t.de"
:mailer-port "123"
:service-noreply-address ""
:deploy-federated "true"
:forgejo-image-version-overwrite "3.2.0"})))))
(deftest should-generate-appini-env
(is (= {:APP_NAME-c1 "",
:APP_NAME-c2 "test forgejo",
@ -29,21 +63,20 @@
:FORGEJO__server__ROOT_URL-c2 "https://test.com",
:FORGEJO__server__SSH_DOMAIN-c1 "test.de",
:FORGEJO__server__SSH_DOMAIN-c2 "test.com",
:FORGEJO__service__EMAIL_DOMAIN_WHITELIST-c1 "adb.de",
:FORGEJO__service__EMAIL_DOMAIN_WHITELIST-c2 "test.com,test.net",
:FORGEJO__service__EMAIL_DOMAIN_ALLOWLIST-c1 "adb.de",
:FORGEJO__service__EMAIL_DOMAIN_ALLOWLIST-c2 "test.com,test.net",
:FORGEJO__service__NO_REPLY_ADDRESS-c1 "",
:FORGEJO__service__NO_REPLY_ADDRESS-c2 "noreply@test.com"}
(th/map-diff (cut/generate-appini-env {:default-app-name ""
:deploy-federated "false"
:federation-enabled "false"
:fqdn "test.de"
:mailer-from ""
:mailer-host "m.t.de"
:mailer-port "123"
:service-domain-whitelist "adb.de"
:service-noreply-address ""
})
:service-noreply-address ""})
(cut/generate-appini-env {:default-app-name "test forgejo"
:deploy-federated "true"
:federation-enabled "true"
:fqdn "test.com"
:mailer-from "test@test.com"
:mailer-host "mail.test.com"
@ -55,7 +88,7 @@
(testing "non-federated"
(is (= {:apiVersion "apps/v1",
:kind "Deployment",
:metadata {:name "forgejo", :namespace "default", :labels {:app "forgejo"}},
:metadata {:name "forgejo", :namespace "forgejo", :labels {:app "forgejo"}},
:spec
{:replicas 1,
:selector {:matchLabels {:app "forgejo"}},
@ -64,7 +97,7 @@
:spec
{:containers
[{:name "forgejo",
:image "codeberg.org/forgejo/forgejo:1.19",
:image "codeberg.org/forgejo/forgejo:8.0",
:imagePullPolicy "IfNotPresent",
:envFrom [{:configMapRef {:name "forgejo-env"}} {:secretRef {:name "forgejo-secrets"}}],
:volumeMounts [{:name "forgejo-data-volume", :mountPath "/data"}],
@ -82,7 +115,7 @@
(testing "federated-deployment"
(is (= {:apiVersion "apps/v1",
:kind "Deployment",
:metadata {:name "forgejo", :namespace "default", :labels {:app "forgejo"}},
:metadata {:name "forgejo", :namespace "forgejo", :labels {:app "forgejo"}},
:spec
{:replicas 1,
:selector {:matchLabels {:app "forgejo"}},
@ -130,26 +163,3 @@
:storage-c2 "15Gi"}
(th/map-diff (cut/generate-data-volume {:volume-total-storage-size 1})
(cut/generate-data-volume {:volume-total-storage-size 15})))))
(deftest should-generate-middleware-ratelimit
(is (= {:apiVersion "traefik.containo.us/v1alpha1",
:kind "Middleware",
:metadata {:name "ratelimit"},
:spec {:rateLimit {:average 10, :burst 5}}}
(cut/generate-rate-limit-middleware {:max-rate 10, :max-concurrent-requests 5}))))
(deftest should-generate-middleware-ratelimit-ingress-and-cert
(is (= {:traefik.ingress.kubernetes.io/router.entrypoints "web, websecure",
:traefik.ingress.kubernetes.io/router.middlewares
"default-redirect-https@kubernetescrd, default-ratelimit@kubernetescrd",
:metallb.universe.tf/address-pool "public"}
(-> (second
(cut/generate-rate-limit-ingress-and-cert
{:fqdn "test.de"
:mailer-from ""
:mailer-host "m.t.de"
:mailer-port "123"
:service-noreply-address ""
:average 10
:burst 5}))
:metadata :annotations))))