You cannot select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
forgejo/integrations
Anthony Wang e86f18a05a
User keypairs and HTTP signatures for ActivityPub federation using go-ap (#19133)
* go.mod: add go-fed/{httpsig,activity/pub,activity/streams} dependency

go get github.com/go-fed/activity/streams@master
go get github.com/go-fed/activity/pub@master
go get github.com/go-fed/httpsig@master

* activitypub: implement /api/v1/activitypub/user/{username} (#14186)

Return informations regarding a Person (as defined in ActivityStreams
https://www.w3.org/TR/activitystreams-vocabulary/#dfn-person).

Refs: https://github.com/go-gitea/gitea/issues/14186

Signed-off-by: Loïc Dachary <loic@dachary.org>

* activitypub: add the public key to Person (#14186)

Refs: https://github.com/go-gitea/gitea/issues/14186

Signed-off-by: Loïc Dachary <loic@dachary.org>

* activitypub: go-fed conformant Clock instance

Signed-off-by: Loïc Dachary <loic@dachary.org>

* activitypub: signing http client

Signed-off-by: Loïc Dachary <loic@dachary.org>

* activitypub: implement the ReqSignature middleware

Signed-off-by: Loïc Dachary <loic@dachary.org>

* activitypub: hack_16834

Signed-off-by: Loïc Dachary <loic@dachary.org>

* Fix CI checks-backend errors with go mod tidy

Signed-off-by: Anthony Wang <ta180m@pm.me>

* Change 2021 to 2022, properly format package imports

Signed-off-by: Anthony Wang <ta180m@pm.me>

* Run make fmt and make generate-swagger

Signed-off-by: Anthony Wang <ta180m@pm.me>

* Use Gitea JSON library, add assert for pkp

Signed-off-by: Anthony Wang <ta180m@pm.me>

* Run make fmt again, fix err var redeclaration

Signed-off-by: Anthony Wang <ta180m@pm.me>

* Remove LogSQL from ActivityPub person test

Signed-off-by: Anthony Wang <ta180m@pm.me>

* Assert if json.Unmarshal succeeds

Signed-off-by: Anthony Wang <ta180m@pm.me>

* Cleanup, handle invalid usernames for ActivityPub person GET request

Signed-off-by: Anthony Wang <ta180m@pm.me>

* Rename hack_16834 to user_settings

Signed-off-by: Anthony Wang <ta180m@pm.me>

* Use the httplib module instead of http for GET requests

* Clean up whitespace with make fmt

* Use time.RFC1123 and make the http.Client proxy-aware

* Check if digest algo is supported in setting module

* Clean up some variable declarations

* Remove unneeded copy

* Use system timezone instead of setting.DefaultUILocation

* Use named constant for httpsigExpirationTime

* Make pubKey IRI #main-key instead of /#main-key

* Move /#main-key to #main-key in tests

* Implemented Webfinger endpoint.

* Add visible check.

* Add user profile as alias.

* Add actor IRI and remote interaction URL to WebFinger response

* fmt

* Fix lint errors

* Use go-ap instead of go-fed

* Run go mod tidy to fix missing modules in go.mod and go.sum

* make fmt

* Convert remaining code to go-ap

* Clean up go.sum

* Fix JSON unmarshall error

* Fix CI errors by adding @context to Person() and making sure types match

* Correctly decode JSON in api_activitypub_person_test.go

* Force CI rerun

* Fix TestActivityPubPersonInbox segfault

* Fix lint error

* Use @mariusor's suggestions for idiomatic go-ap usage

* Correctly add inbox/outbox IRIs to person

* Code cleanup

* Remove another LogSQL from ActivityPub person test

* Move httpsig algos slice to an init() function

* Add actor IRI and remote interaction URL to WebFinger response

* Update TestWebFinger to check for ActivityPub IRI in aliases

* make fmt

* Force CI rerun

* WebFinger: Add CORS header and fix Href -> Template for remote interactions

The CORS header is needed due to https://datatracker.ietf.org/doc/html/rfc7033#section-5 and fixes some Peertube <-> Gitea federation issues

* make lint-backend

* Make sure Person endpoint has Content-Type application/activity+json and includes PreferredUsername, URL, and Icon

Setting the correct Content-Type is essential for federating with Mastodon

* Use UTC instead of GMT

* Rename pkey to pubKey

* Make sure HTTP request Date in GMT

* make fmt

* dont drop err

* Make sure API responses always refer to username in original case

Copied from what I wrote on #19133 discussion: Handling username case is a very tricky issue and I've already encountered a Mastodon <-> Gitea federation bug due to Gitea considering Ta180m and ta180m to be the same user while Mastodon thinks they are two different users. I think the best way forward is for Gitea to only use the original case version of the username for federation so other AP software don't get confused.

* Move httpsig algs constant slice to modules/setting/federation.go

* Add new federation settings to app.example.ini and config-cheat-sheet

* Return if marshalling error

* Make sure Person IRIs are generated correctly

This commit ensures that if the setting.AppURL is something like "http://127.0.0.1:42567" (like in the integration tests), a trailing slash will be added after that URL.

* If httpsig verification fails, fix Host header and try again

This fixes a very rare bug when Gitea and another AP server (confirmed to happen with Mastodon) are running on the same machine, Gitea fails to verify incoming HTTP signatures. This is because the other AP server creates the sig with the public Gitea domain as the Host. However, when Gitea receives the request, the Host header is instead localhost, so the signature verification fails. Manually changing the host header to the correct value and trying the veification again fixes the bug.


* Revert "If httpsig verification fails, fix Host header and try again"

This reverts commit f53e46c721a037c55facb9200106a6b491bf834c.

The bug was actually caused by nginx messing up the Host header when reverse-proxying since I didn't have the line `proxy_set_header Host $host;` in my nginx config for Gitea.

* Go back to using ap.IRI to generate inbox and outbox IRIs

* use const for key values

* Update routers/web/webfinger.go

* Use ctx.JSON in Person response to make code cleaner

* Revert "Use ctx.JSON in Person response to make code cleaner"

This doesn't work because the ctx.JSON() function already sends the response out and it's too late to edit the headers.

This reverts commit 95aad988975be3393c76094864ed6ba962157e0c.

* Use activitypub.ActivityStreamsContentType for Person response Content Type

* Limit maximum ActivityPub request and response sizes to a configurable setting

* Move setting key constants to models/user/setting_keys.go

* Fix failing ActivityPubPerson integration test by checking the correct field for username

* Add a warning about changing settings that can break federation

* Add better comments

* Don't multiply Federation.MaxSize by 1<<20 twice

* Add more better comments

* Fix failing ActivityPubMissingPerson test

We now use ctx.ContextUser so the message printed out when a user does not exist is slightly different

* make generate-swagger

For some reason I didn't realize that /templates/swagger/v1_json.tmpl was machine-generated by make generate-swagger... I've been editing it by hand for three months! 🤦

* Move getting the RFC 2616 time to a separate function

* More code cleanup

* Update go-ap to fix empty liked collection and removed unneeded HTTP headers

* go mod tidy

* Add ed25519 to httpsig algorithms

* Use go-ap/jsonld to add @context and marshal JSON

* Change Gitea user agent from the default to Gitea/Version

* Use ctx.ServerError and remove all remote interaction code from webfinger.go
2 years ago
..
gitea-repositories-meta rm .sample hooks which aren't used (#19101) 2 years ago
migration-test Remove legacy git code (ver < 2.0), fine tune markup tests (#19930) 2 years ago
README.md Fix various typos of software names (#18083) 3 years ago
README_ZH.md upgrade postgres refrence to 14 (#19416) 2 years ago
admin_user_test.go Update HTTP status codes to modern codes (#18063) 2 years ago
api_activitypub_person_test.go User keypairs and HTTP signatures for ActivityPub federation using go-ap (#19133) 2 years ago
api_admin_org_test.go format with gofumpt (#18184) 2 years ago
api_admin_test.go Fix order by parameter (#19849) 2 years ago
api_branch_test.go Update HTTP status codes to modern codes (#18063) 2 years ago
api_comment_test.go Move issues related files into models/issues (#19931) 2 years ago
api_fork_test.go Fix "data race" in testlogger (#9159) 5 years ago
api_gpg_keys_test.go format with gofumpt (#18184) 2 years ago
api_helper_for_declarative_test.go Auto merge pull requests when all checks succeeded via API (#9307) 2 years ago
api_httpsig_test.go Implement http signatures support for the API (#17565) 2 years ago
api_issue_label_test.go Move issues related files into models/issues (#19931) 2 years ago
api_issue_milestone_test.go Move milestone to models/issues/ (#19278) 2 years ago
api_issue_reaction_test.go Move issues related files into models/issues (#19931) 2 years ago
api_issue_stopwatch_test.go Move issues related files into models/issues (#19931) 2 years ago
api_issue_subscription_test.go Move issues related files into models/issues (#19931) 2 years ago
api_issue_test.go Move issues related files into models/issues (#19931) 2 years ago
api_issue_tracked_time_test.go Move issues related files into models/issues (#19931) 2 years ago
api_keys_test.go Fix order by parameter (#19849) 2 years ago
api_nodeinfo_test.go Fix issue overview for teams (#19652) 2 years ago
api_notification_test.go format with gofumpt (#18184) 2 years ago
api_oauth2_apps_test.go Use `CryptoRandomBytes` instead of `CryptoRandomString` (#18439) 2 years ago
api_org_test.go Remove dependent on session auth for api/v1 routers (#19321) 2 years ago
api_packages_composer_test.go Add Package Registry (#16510) 2 years ago
api_packages_conan_test.go Add Package Registry (#16510) 2 years ago
api_packages_container_test.go ContainerRegistry - removed Basic Auth header (#19735) 2 years ago
api_packages_generic_test.go Add Package Registry (#16510) 2 years ago
api_packages_helm_test.go Add Helm Chart registry (#19406) 2 years ago
api_packages_maven_test.go Add Package Registry (#16510) 2 years ago
api_packages_npm_test.go Add Package Registry (#16510) 2 years ago
api_packages_nuget_test.go Add Package Registry (#16510) 2 years ago
api_packages_pypi_test.go Add Package Registry (#16510) 2 years ago
api_packages_rubygems_test.go Add Package Registry (#16510) 2 years ago
api_packages_test.go Hide private repositories in packages (#19584) 2 years ago
api_private_serv_test.go Fix the bug: deploy key with write access can not push (#19010) 2 years ago
api_pull_commits_test.go Move issues related files into models/issues (#19931) 2 years ago
api_pull_review_test.go Move issues related files into models/issues (#19931) 2 years ago
api_pull_test.go Move issues related files into models/issues (#19931) 2 years ago
api_releases_test.go Fix order by parameter (#19849) 2 years ago
api_repo_archive_test.go Move repository model into models/repo (#17933) 3 years ago
api_repo_collaborator_test.go Add API to query collaborators permission for a repository (#18761) 2 years ago
api_repo_edit_test.go format with gofumpt (#18184) 2 years ago
api_repo_file_create_test.go Make git.OpenRepository accept Context (#19260) 2 years ago
api_repo_file_delete_test.go Move repository model into models/repo (#17933) 3 years ago
api_repo_file_get_test.go Add API to serve blob or LFS file content (#19689) 2 years ago
api_repo_file_helpers.go Propagate context and ensure git commands run in request context (#17868) 2 years ago
api_repo_file_update_test.go Make git.OpenRepository accept Context (#19260) 2 years ago
api_repo_get_contents_list_test.go Make git.OpenRepository accept Context (#19260) 2 years ago
api_repo_get_contents_test.go Make git.OpenRepository accept Context (#19260) 2 years ago
api_repo_git_blobs_test.go Move repository model into models/repo (#17933) 3 years ago
api_repo_git_commits_test.go format with gofumpt (#18184) 2 years ago
api_repo_git_hook_test.go Move repository model into models/repo (#17933) 3 years ago
api_repo_git_notes_test.go Move user related model into models/user (#17781) 3 years ago
api_repo_git_ref_test.go Move user related model into models/user (#17781) 3 years ago
api_repo_git_tags_test.go Remove `git.Command.Run` and `git.Command.RunInDir*` (#19280) 2 years ago
api_repo_git_trees_test.go Move repository model into models/repo (#17933) 3 years ago
api_repo_languages_test.go Update HTTP status codes to modern codes (#18063) 2 years ago
api_repo_lfs_locks_test.go format with gofumpt (#18184) 2 years ago
api_repo_lfs_migrate_test.go Move user related model into models/user (#17781) 3 years ago
api_repo_lfs_test.go Move some code into models/git (#19879) 2 years ago
api_repo_raw_test.go Move user related model into models/user (#17781) 3 years ago
api_repo_tags_test.go Simplify parameter types (#18006) 3 years ago
api_repo_teams_test.go Move organization related structs into sub package (#18518) 2 years ago
api_repo_test.go Fix order by parameter (#19849) 2 years ago
api_repo_topic_test.go Remove dependent on session auth for api/v1 routers (#19321) 2 years ago
api_settings_test.go Make mirror feature more configurable (#16957) 3 years ago
api_team_test.go [Refactor] convert team(s) to apiTeam(s) (#13745) 2 years ago
api_team_user_test.go Move user related model into models/user (#17781) 3 years ago
api_token_test.go Move user related model into models/user (#17781) 3 years ago
api_user_email_test.go Don't panic on `ErrEmailInvalid` (#19441) 2 years ago
api_user_heatmap_test.go Remove dependent on session auth for api/v1 routers (#19321) 2 years ago
api_user_org_perm_test.go Add `ContextUser` to http request context (#18798) 2 years ago
api_user_orgs_test.go format with gofumpt (#18184) 2 years ago
api_user_search_test.go Move user related model into models/user (#17781) 3 years ago
api_wiki_test.go Add API to get/edit wiki (#17278) 3 years ago
attachment_test.go Update HTTP status codes to modern codes (#18063) 2 years ago
auth_ldap_test.go Move almost all functions' parameter db.Engine to context.Context (#19748) 2 years ago
benchmarks_test.go format with gofumpt (#18184) 2 years ago
branches_test.go Remove legacy unmaintained packages, refactor to support change default locale (#19308) 2 years ago
change_default_branch_test.go Update HTTP status codes to modern codes (#18063) 2 years ago
cmd_keys_test.go format with gofumpt (#18184) 2 years ago
compare_test.go Allow default branch to be inferred on compare page (#17908) 3 years ago
cors_test.go Fix "data race" in testlogger (#9159) 5 years ago
create_no_session_test.go Update HTTP status codes to modern codes (#18063) 2 years ago
csrf_test.go Refactor CSRF protection modules, make sure CSRF tokens can be up-to-date. (#19337) 2 years ago
delete_user_test.go Move issues related files into models/issues (#19931) 2 years ago
download_test.go A better go code formatter, and now `make fmt` can run in Windows (#17684) 3 years ago
dump_restore_test.go Add Index to comment for migrations and mirroring (#18806) 2 years ago
editor_test.go Update HTTP status codes to modern codes (#18063) 2 years ago
empty_repo_test.go Move repository model into models/repo (#17933) 3 years ago
eventsource_test.go format with gofumpt (#18184) 2 years ago
explore_repos_test.go Fix "data race" in testlogger (#9159) 5 years ago
git_clone_wiki_test.go Unexport git.GlobalCommandArgs (#18376) 2 years ago
git_helper_for_declarative_test.go Remove `git.Command.Run` and `git.Command.RunInDir*` (#19280) 2 years ago
git_smart_http_test.go Update HTTP status codes to modern codes (#18063) 2 years ago
git_test.go Move issues related files into models/issues (#19931) 2 years ago
goget_test.go Refactor legacy `unknwon/com` package, improve golangci lint (#19284) 2 years ago
gpg_git_test.go format with gofumpt (#18184) 2 years ago
html_helper.go Frontend refactor, PascalCase to camelCase, remove unused code (#17365) 3 years ago
integration_test.go Remove legacy git code (ver < 2.0), fine tune markup tests (#19930) 2 years ago
issue_test.go Move issues related files into models/issues (#19931) 2 years ago
lfs_getobject_test.go Move some code into models/git (#19879) 2 years ago
lfs_local_endpoint_test.go format with gofumpt (#18184) 2 years ago
links_test.go Remove tab/TabName usage where it's not needed (#19973) 2 years ago
migrate_test.go Move user related model into models/user (#17781) 3 years ago
mirror_pull_test.go Move almost all functions' parameter db.Engine to context.Context (#19748) 2 years ago
mirror_push_test.go Make git.OpenRepository accept Context (#19260) 2 years ago
mssql.ini.tmpl Implement http signatures support for the API (#17565) 2 years ago
mysql.ini.tmpl Implement http signatures support for the API (#17565) 2 years ago
mysql8.ini.tmpl Implement http signatures support for the API (#17565) 2 years ago
nonascii_branches_test.go Update HTTP status codes to modern codes (#18063) 2 years ago
oauth_test.go Update HTTP status codes to modern codes (#18063) 2 years ago
org_count_test.go Move organization related structs into sub package (#18518) 2 years ago
org_test.go [Refactor] convert team(s) to apiTeam(s) (#13745) 2 years ago
pgsql.ini.tmpl Implement http signatures support for the API (#17565) 2 years ago
private-testing.key Use non-expiring key. (#17984) 3 years ago
privateactivity_test.go Update HTTP status codes to modern codes (#18063) 2 years ago
pull_compare_test.go Fix "data race" in testlogger (#9159) 5 years ago
pull_create_test.go Update HTTP status codes to modern codes (#18063) 2 years ago
pull_merge_test.go Move issues related files into models/issues (#19931) 2 years ago
pull_review_test.go Fix "data race" in testlogger (#9159) 5 years ago
pull_status_test.go Auto merge pull requests when all checks succeeded via API (#9307) 2 years ago
pull_update_test.go Move issues related files into models/issues (#19931) 2 years ago
release_test.go Remove legacy unmaintained packages, refactor to support change default locale (#19308) 2 years ago
rename_branch_test.go Update HTTP status codes to modern codes (#18063) 2 years ago
repo_activity_test.go format with gofumpt (#18184) 2 years ago
repo_branch_test.go Remove legacy unmaintained packages, refactor to support change default locale (#19308) 2 years ago
repo_commits_search_test.go Make TestCreateBranch and TestRepoCommitsSearch less noisy (#13471) 4 years ago
repo_commits_test.go Auto merge pull requests when all checks succeeded via API (#9307) 2 years ago
repo_fork_test.go Update HTTP status codes to modern codes (#18063) 2 years ago
repo_generate_test.go Update HTTP status codes to modern codes (#18063) 2 years ago
repo_migrate_test.go Update HTTP status codes to modern codes (#18063) 2 years ago
repo_search_test.go Automatically pause queue if index service is unavailable (#15066) 2 years ago
repo_tag_test.go Move some code into models/git (#19879) 2 years ago
repo_test.go Update JS dependencies (#19281) 2 years ago
repo_topic_test.go Refactor CSRF protection modules, make sure CSRF tokens can be up-to-date. (#19337) 2 years ago
repo_watch_test.go Some repository refactors (#17950) 3 years ago
repofiles_delete_test.go Renamed ctx.User to ctx.Doer. (#19161) 2 years ago
repofiles_update_test.go Make git.OpenRepository accept Context (#19260) 2 years ago
setting_test.go Update HTTP status codes to modern codes (#18063) 2 years ago
signin_test.go Remove legacy unmaintained packages, refactor to support change default locale (#19308) 2 years ago
signout_test.go Update HTTP status codes to modern codes (#18063) 2 years ago
signup_test.go Remove legacy unmaintained packages, refactor to support change default locale (#19308) 2 years ago
sqlite.ini.tmpl Implement http signatures support for the API (#17565) 2 years ago
ssh_key_test.go format with gofumpt (#18184) 2 years ago
testlogger.go Fix typo (#19889) 2 years ago
timetracking_test.go format with gofumpt (#18184) 2 years ago
user_avatar_test.go Update HTTP status codes to modern codes (#18063) 2 years ago
user_test.go Move issues related files into models/issues (#19931) 2 years ago
version_test.go Fix "data race" in testlogger (#9159) 5 years ago
view_test.go Display SVG files as images instead of text (#14101) 4 years ago
webfinger_test.go User keypairs and HTTP signatures for ActivityPub federation using go-ap (#19133) 2 years ago
xss_test.go Update HTTP status codes to modern codes (#18063) 2 years ago

README.md

Integrations tests

Integration tests can be run with make commands for the appropriate backends, namely:

make test-mysql
make test-pgsql
make test-sqlite

Make sure to perform a clean build before running tests:

make clean build

Run all tests via local drone

drone exec --local --build-event "pull_request"

Run sqlite integrations tests

Start tests

make test-sqlite

Run MySQL integrations tests

Setup a MySQL database inside docker

docker run -e "MYSQL_DATABASE=test" -e "MYSQL_ALLOW_EMPTY_PASSWORD=yes" -p 3306:3306 --rm --name mysql mysql:latest #(just ctrl-c to stop db and clean the container)
docker run -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" --rm --name elasticsearch elasticsearch:7.6.0 #(in a second terminal, just ctrl-c to stop db and clean the container)

Start tests based on the database container

TEST_MYSQL_HOST=localhost:3306 TEST_MYSQL_DBNAME=test TEST_MYSQL_USERNAME=root TEST_MYSQL_PASSWORD='' make test-mysql

Run pgsql integrations tests

Setup a pgsql database inside docker

docker run -e "POSTGRES_DB=test" -p 5432:5432 --rm --name pgsql postgres:latest #(just ctrl-c to stop db and clean the container)

Start tests based on the database container

TEST_PGSQL_HOST=localhost:5432 TEST_PGSQL_DBNAME=test TEST_PGSQL_USERNAME=postgres TEST_PGSQL_PASSWORD=postgres make test-pgsql

Run mssql integrations tests

Setup a mssql database inside docker

docker run -e "ACCEPT_EULA=Y" -e "MSSQL_PID=Standard" -e "SA_PASSWORD=MwantsaSecurePassword1" -p 1433:1433 --rm --name mssql microsoft/mssql-server-linux:latest #(just ctrl-c to stop db and clean the container)

Start tests based on the database container

TEST_MSSQL_HOST=localhost:1433 TEST_MSSQL_DBNAME=gitea_test TEST_MSSQL_USERNAME=sa TEST_MSSQL_PASSWORD=MwantsaSecurePassword1 make test-mssql

Running individual tests

Example command to run GPG test:

For SQLite:

make test-sqlite#GPG

For other databases(replace mssql to mysql, mysql8 or pgsql):

TEST_MSSQL_HOST=localhost:1433 TEST_MSSQL_DBNAME=test TEST_MSSQL_USERNAME=sa TEST_MSSQL_PASSWORD=MwantsaSecurePassword1 make test-mssql#GPG

Setting timeouts for declaring long-tests and long-flushes

We appreciate that some testing machines may not be very powerful and the default timeouts for declaring a slow test or a slow clean-up flush may not be appropriate.

You can either:

  • Within the test ini file set the following section:
[integration-tests]
SLOW_TEST = 10s ; 10s is the default value
SLOW_FLUSH = 5S ; 5s is the default value
  • Set the following environment variables:
GITEA_SLOW_TEST_TIME="10s" GITEA_SLOW_FLUSH_TIME="5s" make test-sqlite