Manage nuclei-templates with version control system (with git or something similar)

[dogancanbakir]

See the initial discussion: #4690

Notes from internal discussion @Mzack9999 and I:

During template updates, instead of rename, we create a new template, leaving outdated ones behind
The default nuclei-templates folder should be like a git repository, this way renames and deletions would be tracked correctly. Actually I guess we only unzip directly
Somehow we need to identify with template history, which ones are dupes based on content similarity and implement an auto-fix mechanism like to delete them if they were not altered and have only been renamed elsewhere, but I guess it's not super trivial and might take some time

We should consider introducing git like version control for nuclei templates to manage them since there can be lots of changes or modifications by mirroring for nuclei-templates since GH rate limit can cause issues.

[dwisiswant0]

Wouldn’t it be cleaner to just wipe everything (or move it to somewhere like /tmp for rollback purpose) before downloading the new version?

Using Git here just bloats things because we’re dragging in every commit, tree, and blob.

$ git count-objects -vH
count: 231
size: 2.04 MiB
in-pack: 972004
packs: 19
size-pack: 330.19 MiB
prune-packable: 26
garbage: 0
size-garbage: 0 bytes
$ git rev-list --objects main | git cat-file --batch-check='%(objectsize) %(rest)' | awk '{total += $1} END {print total}'
5.83704e+09

What’s the Git flow design supposed to look like here? Shallow clone makes sense on the first init, but what about after that? ex. tracking new/previous releases, comparing annotated commits, etc. IMO that’s just piling on complexity. (Also, why was this discussion closed already?)

[dwisiswant0]

[...] this way renames and deletions would be tracked correctly. Actually I guess we only unzip directly

AFAIK renames & deletions are only used for stats after a template upgrade, and because we don’t prune the old dir (we just unzip over it like you said), we end up with the same template IDs across multiple template files, which makes the stats error-prone and miscalculated.

We can get stats (properly) with GitHub API and compare 2 refs, /repos/projectdiscovery/nuclei-templates/compare/{base}...{head}.

[dogancanbakir]

The main pain point we’re trying to address is that the current unzip-over approach leads to duplication, ID collisions, and misleading stats because old templates aren’t being pruned or tracked properly. Wiping the directory before each sync would solve part of that, but it also removes any context/history that could help with things like template lineage or verifying whether a local modification was preserved.

Tracked at #6467

[dwisiswant0]

[...] but it also removes any context/history that could help with things like template lineage or verifying whether a local modification was preserved.

AFAIK the current stats calc uses checksums. We can still rely on templates-checksum.txt to preserve local mods or flag them as corrupt (or something else - if modified).

How that would look if we swapped to VCS (Git in this case), since for pulls, the Git state can’t be dirty.