Merge branch 'main' of ssh://repo.prod.meissa.de:2222/meissa/c4k-forgejo

.gitlab-ci.yml

@@ -123,9 +123,10 @@ forgejo-backup-image-publish:
   script:
     - cd infrastructure/backup && pyb image publish
 
-forgejo-federated-image-publish:
+# This is currently not needed
-  <<: *img
+#forgejo-federated-image-publish:
-  <<: *tag_only
+#  <<: *img
-  stage: image
+#  <<: *tag_only
-  script:
+#  stage: image
-    - cd infrastructure/federated && pyb image publish
+#  script:
+#    - cd infrastructure/federated && pyb image publish

README.md

@@ -35,6 +35,11 @@ After having deployed the yaml-file generated by the c4k-forgejo module you need
 * The SSH-URL for a repo has the format: "ssh://git@domain:2222/[username]/[repo].git
   Example: "git clone ssh://git@repo.test.meissa.de:2222/myuser/c4k-forgejo.git"
 
+### Add Impressum
+
+In order to customize the UI e.g. for adding an Impressum, see the [Forgejo Docs](https://forgejo.org/docs/latest/developer/customization/#adding-links-and-tabs).
+The individually needed files have to be added by hand into the directory `/data/gitea/templates/custom/` in the forgejo Pod. Since a PV is mounted under `/data`, these ui customizations are persisted.
+
 ## Development & mirrors
 
 Development happens at: https://repo.prod.meissa.de/meissa/c4k-forgejo

doc/Runbook_UpgradeFrom1.19To7.0.5.md

@@ -0,0 +1,87 @@
+# Playbook Upgrade from 1.19 to 7.0.5
+
+## Info: Relevant Breaking Changes:
+
+* 1.19.3:Current version
+* 1.20.1-0: Breaking https://codeberg.org/forgejo/forgejo/src/branch/forgejo/RELEASE-NOTES.md#1-20-1-0
+* 1.21.1-0: https://codeberg.org/forgejo/forgejo/src/branch/forgejo/RELEASE-NOTES.md#1-21-1-0
+* 7.0.0: https://codeberg.org/forgejo/forgejo/src/branch/forgejo/RELEASE-NOTES.md#7-0-0
+
+## Preparations
+
+1. Stop Forgejo Prod: `k scale deployment forgejo --replicas=0`
+1. Disable Backup Cron: `k patch cronjobs forgejo-backup -p '{"spec" : {"suspend" : true }}'`
+1. Scale up Backup-Restore Deployment: `kubectl scale deployment backup-restore --replicas=1`
+1. Execute Manual Backup: `kubectl exec -it backup-restore-... -- /usr/local/bin/backup.sh`
+
+### Create 2nd Repo Prod Server
+
+1. Terraform Preparations for 2nd Server: TODO
+1. Install c4k-forgejo Version TODO   
+   with config `"forgejo-image-version-overwrite": "1.19.3-0"`
+1. Stop Forgejo Deployment: `k scale deployment forgejo --replicas=0`
+1. Disable Backup Cron: `k patch cronjobs forgejo-backup -p '{"spec" : {"suspend" : true }}'`
+1. Scale up Backup-Restore Deployment: `kubectl scale deployment backup-restore --replicas=1`
+1. Restore Forgejo Backup: See [BackupAndRestore.md](BackupAndRestore.md)
+1. Check for `..._INSTALL_LOCK: true` in ConfigMap `forgejo-env`
+1. Scale up Forgejo Deployment and check for (startup) problems: `k scale deployment forgejo --replicas=1`
+
+## Upgrade to 1.20.1-0
+
+1. Scale down Forgejo Deployment: `k scale deployment forgejo --replicas=0`
+1. Adjust configmap: `k edit cm forgejo-env`
+    1. Remove `FORGEJO__database__CHARSET: utf8` (This was a misconfiguration, since this option only had effect for mysql dbs)
+    1. Change `FORGEJO__mailer__MAILER_TYPE: smtp+startls` TO `FORGEJO__mailer__PROTOCOL: smtp+starttls` (Missed deprecation from 1.19)
+    1. Change `FORGEJO__service__EMAIL_DOMAIN_WHITELIST: repo.test.meissa.de` TO `FORGEJO__service__EMAIL_DOMAIN_ALLOWLIST: repo.test.meissa.de` (Fallback deprecation in 1.21)
+1. Delete app.ini: `k exec -it backup-restore-... -- rm /var/backups/gitea/conf/app.ini`
+1. Set version to `1.20.1-0` with `k edit deployment forgejo`
+1. Scale up Forgejo Deployment: `k scale deployment forgejo --replicas=1`
+1. Check for errors
+
+## Upgrade to 1.21.1-0
+
+1. Scale down Forgejo Deployment: `k scale deployment forgejo --replicas=0`
+1. Delete app.ini: `k exec -it backup-restore-... -- rm /var/backups/gitea/conf/app.ini`
+1. Set version to `1.21.1-0` with `k edit deployment forgejo`
+1. Scale up Forgejo Deployment: `k scale deployment forgejo --replicas=1`
+1. Check for errors
+1. After upgrading, login as an admin, go to the `/admin` page and click run `Sync missed branches from git data to databases` (`Fehlende Branches aus den Git-Daten in die Datenbank synchronisieren`). If this is not done there will be messages such as `LoadBranches: branch does not exist in the logs`.
+
+## Upgrade to 7.0.0
+
+1. Scale down Forgejo Deployment: `k scale deployment forgejo --replicas=0`
+1. Adjust configmap: `k edit cm forgejo-env`
+    1. Change `FORGEJO__oauth2__ENABLE: "true"` TO `FORGEJO__oauth2__ENABLED: "true"`
+1. Delete app.ini: `k exec -it backup-restore-... -- rm /var/backups/gitea/conf/app.ini`
+1. Set version to `7.0.0` with `k edit deployment forgejo`
+1. Scale up Forgejo Deployment: `k scale deployment forgejo --replicas=1`
+1. Check for errors
+
+## Upgrade to 7.0.5 (no breaking changes)
+
+TODO: Upgrade to 8.0.0 instead after Release!
+
+1. Scale down Forgejo Deployment: `k scale deployment forgejo --replicas=0`
+1. Delete app.ini: `k exec -it backup-restore-... -- rm /var/backups/gitea/conf/app.ini`
+1. Set version to `7.0.5` with `k edit deployment forgejo`
+1. Scale up Forgejo Deployment: `k scale deployment forgejo --replicas=1`
+1. Check for errors
+
+## Post Work
+
+1. Switch DNS to new server
+1. Reenable Backup Cron on new server: `k patch cronjobs forgejo-backup -p '{"spec" : {"suspend" : false }}'`
+1. Execute manual Backup on new server: `kubectl exec -it backup-restore-... -- /usr/local/bin/backup.sh`
+1. Scale down Backup-Restore Deployment: `kubectl scale deployment backup-restore --replicas=1`
+1. The scope of all access tokens might (invisibly) have changed (in v1.20). Thus, rotate all tokens!
+1. Users should check their ssh keys: if they use rsa keys the minimum length should be 3072 bits! However, shorter keys should still work.
+
+## Known Errors
+
+### Error in v1.20.1-0
+
+In the logs the following error can be found. This will be resolved automatically with the next upgrade (v1.21).
+
+```
+2024/07/08 08:31:30 ...g/config_provider.go:321:deprecatedSetting() [E] Deprecated fallback `[log]` `ROUTER` present. Use `[log]` `logger.router.MODE` instead. This fallback will be/has been removed in 1.21
+```

infrastructure/backup/build.py

@@ -6,7 +6,7 @@ from ddadevops import *
 name = "c4k-forgejo"
 MODULE = "backup"
 PROJECT_ROOT_PATH = "../.."
-version = "3.2.3-dev"
+version = "3.3.2-dev"
 
 
 @init

infrastructure/federated/build.py

@@ -6,7 +6,7 @@ from ddadevops import *
 name = 'c4k-forgejo'
 MODULE = 'federated'
 PROJECT_ROOT_PATH = '../..'
-version = "3.2.3-dev"
+version = "3.3.2-dev"
 
 @init
 def initialize(project):

package.json

@@ -2,7 +2,7 @@
     "name": "c4k-forgejo",
     "description": "Generate c4k yaml for a forgejo deployment.",
     "author": "meissa GmbH",
-    "version": "3.2.3-SNAPSHOT",
+    "version": "3.3.2-SNAPSHOT",
     "homepage": "https://gitlab.com/domaindrivenarchitecture/c4k-forgejo#readme",
     "repository": "https://www.npmjs.com/package/c4k-forgejo",
     "license": "APACHE2",

project.clj

@@ -1,11 +1,11 @@
-(defproject org.domaindrivenarchitecture/c4k-forgejo "3.2.3-SNAPSHOT"
+(defproject org.domaindrivenarchitecture/c4k-forgejo "3.3.2-SNAPSHOT"
   :description "forgejo c4k-installation package"
   :url "https://domaindrivenarchitecture.org"
   :license {:name "Apache License, Version 2.0"
             :url "https://www.apache.org/licenses/LICENSE-2.0.html"}
-  :dependencies [[org.clojure/clojure "1.11.2" :scope "provided"]
+  :dependencies [[org.clojure/clojure "1.11.3" :scope "provided"]
-                 [org.clojure/tools.reader "1.4.1"]
+                 [org.clojure/tools.reader "1.4.2"]
-                 [org.domaindrivenarchitecture/c4k-common-clj "6.2.3"]
+                 [org.domaindrivenarchitecture/c4k-common-clj "6.4.1"]
                  [hickory "0.7.1" :exclusions [viebel/codox-klipse-theme]]]
   :target-path "target/%s/"
   :source-paths ["src/main/cljc"
@@ -23,9 +23,9 @@
                        :main dda.c4k-forgejo.uberjar
                        :uberjar-name "c4k-forgejo-standalone.jar"
                        :dependencies [[org.clojure/tools.cli "1.1.230"]
-                                      [ch.qos.logback/logback-classic "1.5.3"
+                                      [ch.qos.logback/logback-classic "1.5.6"
                                        :exclusions [com.sun.mail/javax.mail]]
-                                      [org.slf4j/jcl-over-slf4j "2.0.12"]
+                                      [org.slf4j/jcl-over-slf4j "2.0.13"]
                                       [com.github.clj-easy/graal-build-time "1.0.5"]]}}
   :release-tasks [["test"]
                   ["vcs" "assert-committed"]

src/main/cljc/dda/c4k_forgejo/backup.cljc

@@ -4,12 +4,13 @@
   [dda.c4k-common.yaml :as yaml]
   [dda.c4k-common.base64 :as b64]
   [dda.c4k-common.common :as cm]
+  [dda.c4k-common.predicate :as p]
   #?(:cljs [dda.c4k-common.macros :refer-macros [inline-resources]])))
 
-(s/def ::aws-access-key-id cm/bash-env-string?)
+(s/def ::aws-access-key-id p/bash-env-string?)
-(s/def ::aws-secret-access-key cm/bash-env-string?)
+(s/def ::aws-secret-access-key p/bash-env-string?)
-(s/def ::restic-password cm/bash-env-string?)
+(s/def ::restic-password p/bash-env-string?)
-(s/def ::restic-repository cm/bash-env-string?)
+(s/def ::restic-repository p/bash-env-string?)
 
 #?(:cljs
    (defmethod yaml/load-resource :backup [resource-name]

src/main/cljc/dda/c4k_forgejo/core.cljc

@@ -21,6 +21,7 @@
                               ::forgejo/deploy-federated
                               ::forgejo/default-app-name 
                               ::forgejo/service-domain-whitelist
+                              ::forgejo/forgejo-image-version-overwrite
                               ::backup/restic-repository
                               ::mon/mon-cfg]))
 

src/main/cljc/dda/c4k_forgejo/forgejo.cljc

@@ -38,6 +38,7 @@
 (s/def ::mailer-port pred/bash-env-string?)
 (s/def ::service-domain-whitelist domain-list?)
 (s/def ::service-noreply-address string?)
+(s/def ::forgejo-image-version-overwrite string?)
 (s/def ::mailer-user pred/bash-env-string?)
 (s/def ::mailer-pw pred/bash-env-string?)
 (s/def ::issuer pred/letsencrypt-issuer?)
@@ -53,7 +54,8 @@
                      :opt-un [::issuer
                               ::deploy-federated
                               ::default-app-name
-                              ::service-domain-whitelist]))
+                              ::service-domain-whitelist
+                              ::forgejo-image-version-overwrite]))
 
 (def rate-limit-config? (s/keys :req-un [::max-rate
                                          ::max-concurrent-requests]))
@@ -66,8 +68,18 @@
   [total]
   total)
 
-(def federated-image-name "domaindrivenarchitecture/c4k-forgejo-federated:latest")
+(def federated-image-name "domaindrivenarchitecture/c4k-forgejo-federated")
-(def non-federated-image-name "codeberg.org/forgejo/forgejo:1.19")
+(def federated-image-version "latest")
+(def non-federated-image-name "codeberg.org/forgejo/forgejo")
+(def non-federated-image-version "1.19")
+
+(defn-spec generate-image-str string?
+  [config config?]
+  (let [{:keys [deploy-federated forgejo-image-version-overwrite]} config
+        deploy-federated-bool (boolean-from-string deploy-federated)]
+    (if deploy-federated-bool
+      (str federated-image-name ":" (or forgejo-image-version-overwrite federated-image-version))
+      (str non-federated-image-name ":" (or forgejo-image-version-overwrite non-federated-image-version)))))
 
 #?(:cljs
    (defmethod yaml/load-resource :forgejo [resource-name]
@@ -158,10 +170,7 @@
         deploy-federated-bool (boolean-from-string deploy-federated)]
     (->
      (yaml/load-as-edn "forgejo/deployment.yaml")
-     (cm/replace-all-matching-values-by-new-value "IMAGE_NAME" 
+     (cm/replace-all-matching-values-by-new-value "IMAGE_NAME" (generate-image-str config)))))
-                                                  (if deploy-federated-bool
-                                                    federated-image-name
-                                                    non-federated-image-name)))))
 
 (defn generate-service
   []

src/test/cljc/dda/c4k_forgejo/forgejo_test.cljc

@@ -12,6 +12,40 @@
 (st/instrument `cut/generate-ingress)
 (st/instrument `cut/generate-secrets)
 
+(deftest should-generate-image-str
+  (testing "non-federated-image"
+    (is (= "codeberg.org/forgejo/forgejo:1.19"
+           (cut/generate-image-str {:fqdn "test.de"
+                                    :mailer-from ""
+                                    :mailer-host "m.t.de"
+                                    :mailer-port "123"
+                                    :service-noreply-address ""
+                                    :deploy-federated "false"})))
+    (is (= "codeberg.org/forgejo/forgejo:1.19.3-0"
+           (cut/generate-image-str {:fqdn "test.de"
+                                    :mailer-from ""
+                                    :mailer-host "m.t.de"
+                                    :mailer-port "123"
+                                    :service-noreply-address ""
+                                    :deploy-federated "false"
+                                    :forgejo-image-version-overwrite "1.19.3-0"}))))
+  (testing "federated-image"
+    (is (= "domaindrivenarchitecture/c4k-forgejo-federated:latest"
+           (cut/generate-image-str {:fqdn "test.de"
+                                    :mailer-from ""
+                                    :mailer-host "m.t.de"
+                                    :mailer-port "123"
+                                    :service-noreply-address ""
+                                    :deploy-federated "true"})))
+    (is (= "domaindrivenarchitecture/c4k-forgejo-federated:3.2.0"
+           (cut/generate-image-str {:fqdn "test.de"
+                                    :mailer-from ""
+                                    :mailer-host "m.t.de"
+                                    :mailer-port "123"
+                                    :service-noreply-address ""
+                                    :deploy-federated "true"
+                                    :forgejo-image-version-overwrite "3.2.0"})))))
+
 (deftest should-generate-appini-env
   (is (= {:APP_NAME-c1 "",
           :APP_NAME-c2 "test forgejo",
@@ -40,8 +74,7 @@
                                                 :mailer-host "m.t.de"
                                                 :mailer-port "123"
                                                 :service-domain-whitelist "adb.de"
-                                                :service-noreply-address ""
+                                                :service-noreply-address ""})
-                                                })
                       (cut/generate-appini-env {:default-app-name "test forgejo"
                                                 :deploy-federated "true"
                                                 :fqdn "test.com"