You cannot select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
e28cc79c92
This addresses https://github.com/go-gitea/gitea/issues/18352 It aims to improve performance (and resource use) of the `SyncReleasesWithTags` operation for pull-mirrors. For large repositories with many tags, `SyncReleasesWithTags` can be a costly operation (taking several minutes to complete). The reason is two-fold: 1. on sync, every upstream repo tag is compared (for changes) against existing local entries in the release table to ensure that they are up-to-date. 2. the procedure for getting _each tag_ involves a series of git operations ```bash git show-ref --tags -- v8.2.4477 git cat-file -t 29ab6ce9f36660cffaad3c8789e71162e5db5d2f git cat-file -p 29ab6ce9f36660cffaad3c8789e71162e5db5d2f git rev-list --count 29ab6ce9f36660cffaad3c8789e71162e5db5d2f ``` of which the `git rev-list --count` can be particularly heavy. This PR optimizes performance for pull-mirrors. We utilize the fact that a pull-mirror is always identical to its upstream and rebuild the entire release table on every sync and use a batch `git for-each-ref .. refs/tags` call to retrieve all tags in one go. For large mirror repos, with hundreds of annotated tags, this brings down the duration of the sync operation from several minutes to a few seconds. A few unscientific examples run on my local machine: - https://github.com/spring-projects/spring-boot (223 tags) - before: `0m28,673s` - after: `0m2,244s` - https://github.com/kubernetes/kubernetes (890 tags) - before: `8m00s` - after: `0m8,520s` - https://github.com/vim/vim (13954 tags) - before: `14m20,383s` - after: `0m35,467s` I added a `foreachref` package which contains a flexible way of specifying which reference fields are of interest (`git-for-each-ref(1)`) and to produce a parser for the expected output. These could be reused in other places where `for-each-ref` is used. I'll add unit tests for those if the overall PR looks promising. |
2 years ago | |
|---|---|---|
| .. | ||
| activitypub | 2 years ago | |
| analyze | 3 years ago | |
| appstate | 2 years ago | |
| auth | 2 years ago | |
| avatar | 2 years ago | |
| base | 2 years ago | |
| cache | 2 years ago | |
| charset | 2 years ago | |
| container | 2 years ago | |
| context | 2 years ago | |
| convert | 2 years ago | |
| csv | 2 years ago | |
| doctor | 2 years ago | |
| emoji | 2 years ago | |
| eventsource | 3 years ago | |
| generate | 2 years ago | |
| git | 2 years ago | |
| gitgraph | 2 years ago | |
| graceful | 2 years ago | |
| hcaptcha | 4 years ago | |
| highlight | 2 years ago | |
| hostmatcher | 2 years ago | |
| httpcache | 2 years ago | |
| httplib | 2 years ago | |
| indexer | 2 years ago | |
| json | 2 years ago | |
| lfs | 2 years ago | |
| log | 2 years ago | |
| markup | 2 years ago | |
| metrics | 2 years ago | |
| migration | 2 years ago | |
| nosql | 2 years ago | |
| notification | 2 years ago | |
| options | 2 years ago | |
| packages | 2 years ago | |
| password | 3 years ago | |
| pprof | 3 years ago | |
| private | 2 years ago | |
| process | 2 years ago | |
| proxy | 3 years ago | |
| public | 2 years ago | |
| queue | 2 years ago | |
| recaptcha | 3 years ago | |
| references | 2 years ago | |
| repository | 2 years ago | |
| secret | 2 years ago | |
| session | 2 years ago | |
| setting | 2 years ago | |
| ssh | 2 years ago | |
| storage | 2 years ago | |
| structs | 2 years ago | |
| svg | 3 years ago | |
| sync | 5 years ago | |
| templates | 2 years ago | |
| test | 2 years ago | |
| timeutil | 2 years ago | |
| translation | 2 years ago | |
| typesniffer | 2 years ago | |
| updatechecker | 2 years ago | |
| upload | 3 years ago | |
| uri | 3 years ago | |
| user | 4 years ago | |
| util | 2 years ago | |
| validation | 2 years ago | |
| web | 2 years ago | |