Numerous Packages in Fedora fail `git-fsck`

sgallagh commented

2023-12-04 17:51:03 +00:00

Member

Describe the issue

Many packages (mostly Java) in Fedora dist-git fail to validate with git-fsck. This prevents importing those packages' history into forges that run this check on push for security reasons. In particular, this is preventing us from importing those packages into CentOS Stream 10, which uses Gitlab as the host for its dist-git.

The packages specifically needed for CentOS Stream 10 are:

antlr
apache-commons-cli
apache-commons-codec
apache-commons-exec
apache-commons-io
apache-commons-parent
beust-jcommander
bsf
jdom
jsch
log4j
maven-antrun-plugin
maven-archiver
maven-assembly-plugin
maven-shade-plugin
mojo-parent

The rest of the list is attached below. All of the affected packages have the same root issue: a packager many years ago had an extra < character in their author/committer field, which causes a (harmless) validation error.

What we propose:

Archive the current head of the rawhide branch with a new branch/tag identifier so that it will remain in the repository.
Rewrite history from the affected commit forwards, correcting the author/committer field so that the history on the rawhide branch will validate successfully. This means that rawhide will move HEAD and will need to be announced that anyone with a checkout will need to re-pull from it.
Import the history from the fixed Rawhide branch into CentOS Stream 10.

This will resolve the issue for these sixteen packages, but it's also worth noting that Fedora will still need to deal with this issue on other branches at some point if we move away from Pagure. We probably also want to bring Pagure in line with Gitlab here and make it run this check on pushes to ensure we don't reintroduce similar issues.

When do you need this? (YYYY/MM/DD)

Immediately

When is this no longer needed or useful? (YYYY/MM/DD)

This should be a one-time event.

If we cannot complete your request, what is the impact?

We will be unable to import the affected packages directly to CentOS Stream 10 and will need to find another workaround, probably making trivial merges/cherry-picks from Fedora unusable.

Appendix

The complete list of Fedora packages that fail git-fsck as of 2023-12-04.

ant-contrib
antlr
apache-commons-cli
apache-commons-codec
apache-commons-dbutils
apache-commons-digester
apache-commons-discovery
apache-commons-exec
apache-commons-io
apache-commons-parent
apache-commons-pool
apache-mime4j
beust-jcommander
bsf
cal10n
castor
checkstyle
cssparser
decentxml
derby
eclipse
eclipse-cmakeed
eclipse-egit
eclipse-jgit
eclipse-manpage
eclipse-pydev
eclipse-rpm-editor
eclipse-rpmstubby
eclipse-shelled
felix-osgi-foundation
geronimo-jaxrpc
geronimo-osgi-support
geronimo-saaj
gstreamer-java
httpunit
icu4j
javamail
jaxen
jboss-parent
jdom
jline
joda-time
jsch
kxml
log4j
maven-ant-plugin
maven-antrun-plugin
maven-archiver
maven-assembly-plugin
maven-checkstyle-plugin
maven-doxia-tools
maven-eclipse-plugin
maven-help-plugin
maven-idea-plugin
maven-javadoc-plugin
maven-plugin-exec
maven-pmd-plugin
maven-release
maven-repository-plugin
maven-shade-plugin
maven-skins
mojo-parent
msv
mx4j
nekohtml
plexus-active-collections
plexus-interactivity
rpmorphan
sqljet
xdoclet
xmltool

## Describe the issue Many packages (mostly Java) in Fedora dist-git fail to validate with `git-fsck`. This prevents importing those packages' history into forges that run this check on push for security reasons. In particular, this is preventing us from importing those packages into CentOS Stream 10, which uses Gitlab as the host for its dist-git. The packages specifically needed for CentOS Stream 10 are: ``` antlr apache-commons-cli apache-commons-codec apache-commons-exec apache-commons-io apache-commons-parent beust-jcommander bsf jdom jsch log4j maven-antrun-plugin maven-archiver maven-assembly-plugin maven-shade-plugin mojo-parent ``` The rest of the list is attached below. All of the affected packages have the same root issue: a packager many years ago had an extra `<` character in their author/committer field, which causes a (harmless) validation error. What we propose: 1. Archive the current head of the `rawhide` branch with a new branch/tag identifier so that it will remain in the repository. 2. Rewrite history from the affected commit forwards, correcting the author/committer field so that the history on the `rawhide` branch will validate successfully. This means that `rawhide` will move HEAD and will need to be announced that anyone with a checkout will need to re-pull from it. 3. Import the history from the fixed Rawhide branch into CentOS Stream 10. This will resolve the issue for these sixteen packages, but it's also worth noting that Fedora will still need to deal with this issue on other branches at some point if we move away from Pagure. We probably also want to bring Pagure in line with Gitlab here and make it run this check on pushes to ensure we don't reintroduce similar issues. ## When do you need this? (YYYY/MM/DD) Immediately ## When is this no longer needed or useful? (YYYY/MM/DD) This should be a one-time event. ## If we cannot complete your request, what is the impact? We will be unable to import the affected packages directly to CentOS Stream 10 and will need to find another workaround, probably making trivial merges/cherry-picks from Fedora unusable. ## Appendix The complete list of Fedora packages that fail `git-fsck` as of 2023-12-04. ``` ant-contrib antlr apache-commons-cli apache-commons-codec apache-commons-dbutils apache-commons-digester apache-commons-discovery apache-commons-exec apache-commons-io apache-commons-parent apache-commons-pool apache-mime4j beust-jcommander bsf cal10n castor checkstyle cssparser decentxml derby eclipse eclipse-cmakeed eclipse-egit eclipse-jgit eclipse-manpage eclipse-pydev eclipse-rpm-editor eclipse-rpmstubby eclipse-shelled felix-osgi-foundation geronimo-jaxrpc geronimo-osgi-support geronimo-saaj gstreamer-java httpunit icu4j javamail jaxen jboss-parent jdom jline joda-time jsch kxml log4j maven-ant-plugin maven-antrun-plugin maven-archiver maven-assembly-plugin maven-checkstyle-plugin maven-doxia-tools maven-eclipse-plugin maven-help-plugin maven-idea-plugin maven-javadoc-plugin maven-plugin-exec maven-pmd-plugin maven-release maven-repository-plugin maven-shade-plugin maven-skins mojo-parent msv mx4j nekohtml plexus-active-collections plexus-interactivity rpmorphan sqljet xdoclet xmltool ```

ngompa commented

2023-12-04 17:55:06 +00:00

Member

We should fix this if for nothing else because Pagure's remote pull request feature is effectively useless on these packages if you can't push to some other git server because it fails git-fsck.

kevin commented

2023-12-04 18:23:23 +00:00

Owner

I would want this to get FESCo approval and also get a lot of noise in the community before we do it... since all those repos checked out by maintainers would be useless/broken after we re-write history. ;(

phsmoura commented

2023-12-04 19:17:15 +00:00

Metadata Update from @phsmoura:

Issue tagged with: high-gain, high-trouble, ops

**Metadata Update from @phsmoura**: - Issue tagged with: high-gain, high-trouble, ops

sgallagh commented

2023-12-04 19:19:01 +00:00

Author

Member

OK, with the time it would take to get approved by FESCo and then communicate it effectively to the community as a whole, I think we (CentOS Stream 10) need to proceed with our backup plan of just breaking the inheritance and importing these with revised history. I'll leave this ticket open because we do still need to come up with a longer-term solution.

ngompa commented

2023-12-04 19:54:20 +00:00

Member

Well, we still have a month, so a FESCo ticket about this now (to be discussed/voted on by Thursday) and then an announcement shortly after can give us the time we need.

mjg commented

2023-12-05 10:56:00 +00:00

Please be diligent here.
"Fedora rewrites dist-git history to make RHEL's choice of non-opensource gitlab.com work" would be a very bad headline for all of us. And it's a predictable headline on you_name_it.
The issue of fsck-protected pushes is too technical to make it even in the byline on those sites ...

There are good reasons for both the non-rewrite policy as well as the fsck-protection. They are in conflict here. If we were in control of both forges the obvious short-term solution would be to check the fsck warning carefully and then override it. We cannot do that because gitlab.com does not let us do that. And it's really disappointing, given git has fsck.skipList specifically for that purpose, with fine-grained control.

I'm wondering, though - where does CentOS Stream 9 have its package sources, and why hasn't the problem surfaced there?

Does gitlab.com offer shallow clones? This would allow us to get the history from 2012 onwards (I checked bsf only), which should suffice for now - unwritten and fsck'able!

.mailmap does not help fsck, btw (only log). Neither does git replace.

Please be diligent here. "Fedora rewrites dist-git history to make RHEL's choice of non-opensource gitlab.com work" would be a very bad headline for all of us. And it's a predictable headline on you_name_it. The issue of fsck-protected pushes is too technical to make it even in the byline on those sites ... There are good reasons for both the non-rewrite policy as well as the fsck-protection. They are in conflict here. If we were in control of both forges the obvious short-term solution would be to check the fsck warning carefully and then override it. We cannot do that because gitlab.com does not let us do that. And it's really disappointing, given git has `fsck.skipList` specifically for that purpose, with fine-grained control. I'm wondering, though - where does CentOS Stream 9 have its package sources, and why hasn't the problem surfaced there? Does gitlab.com offer shallow clones? This would allow us to get the history from 2012 onwards (I checked bsf only), which should suffice for now - unwritten and fsck'able! `.mailmap` does not help `fsck`, btw (only `log`). Neither does `git replace`.

ngompa commented

2023-12-05 15:55:46 +00:00

Member

I'm wondering, though - where does CentOS Stream 9 have its package sources, and why hasn't the problem surfaced there?

CentOS Stream 9 also has it on gitlab.com, but it doesn't surface there because they didn't import Git history when they forked from Fedora Linux 34. They dropped all the history of the package and created new repos. However, this messes with attribution, especially for packages that use RPMAutoSpec. Back then, RPMAutoSpec was new and not broadly adopted, but now it is used in many core packages. Thus, the community pushed for preserving the Git history for packages when forking from Fedora the next time. That time is now.

> I'm wondering, though - where does CentOS Stream 9 have its package sources, and why hasn't the problem surfaced there? CentOS Stream 9 also has it on gitlab.com, but it doesn't surface there because they didn't import Git history when they forked from Fedora Linux 34. They dropped all the history of the package and created new repos. However, this messes with attribution, especially for packages that use RPMAutoSpec. Back then, RPMAutoSpec was new and not broadly adopted, but now it is used in many core packages. Thus, the community pushed for preserving the Git history for packages when forking from Fedora the next time. That time is now.

sgallagh commented

2023-12-05 17:07:53 +00:00

Author

Member

For the record, I've imported the sixteen CentOS Stream 10 branches by running the following magic incantation over them:

git filter-repo --force --email-callback 'return email.replace(b" <akurtako@redhat.com", b"akurtako@redhat.com")'

All of the issues were the result of an extra pair of characters in the email field for a short period around 2008 (likely in CVS and pulled over when we switched to git): < <akurtako@redhat.com> instead of <akurtako@redhat.com>

If FESCo approves the fixup, I can pull these 16 branches (and their updated history) back over to Fedora easily. I went ahead with the CentOS Stream 10 import right away because they are blocking our switch over to using CS10 as its own buildroot (rather than relying on ELN for the buildroot).

For the record, I've imported the sixteen CentOS Stream 10 branches by running the following magic incantation over them: ``` git filter-repo --force --email-callback 'return email.replace(b" <akurtako@redhat.com", b"akurtako@redhat.com")' ``` All of the issues were the result of an extra pair of characters in the email field for a short period around 2008 (likely in CVS and pulled over when we switched to git): `< <akurtako@redhat.com>` instead of `<akurtako@redhat.com>` If FESCo approves the fixup, I can pull these 16 branches (and their updated history) back over to Fedora easily. I went ahead with the CentOS Stream 10 import right away because they are blocking our switch over to using CS10 as its own buildroot (rather than relying on ELN for the buildroot).

mizdebsk commented

2023-12-18 14:15:53 +00:00

Contributor

For reference, the corresponding FESCo ticket is https://pagure.io/fesco/issue/3119

yselkowitz commented

2025-11-28 20:55:41 +00:00

Member

@mizdebsk wrote in #11822 (comment):

For reference, the corresponding FESCo ticket is https://pagure.io/fesco/issue/3119

The result being:

FESCo approves the rewriting the history in those repositories using
git filter-repo, with the old branches saved to refs/archive/
namespace.
APPROVED (+4, 0, 0)

But I don't think this was ever implemented?

@mizdebsk wrote in https://forge.fedoraproject.org/releng/tickets/issues/11822#issuecomment-193231: > For reference, the corresponding FESCo ticket is https://pagure.io/fesco/issue/3119 The result being: FESCo approves the rewriting the history in those repositories using git filter-repo, with the old branches saved to refs/archive/ namespace. APPROVED (+4, 0, 0) But I don't think this was ever implemented?

yselkowitz commented

2026-01-05 21:18:21 +00:00

Member

post-recharge ping

yselkowitz commented

2026-03-05 05:09:45 +00:00

Member

Can we move this forward?

kevin commented

2026-03-05 19:41:58 +00:00

Owner

So, who is this action on? @sgallagh or releng?

I guess releng since you can't force push...

perhaps we could make a one off script to do this and verify it then releng can run it?

So, who is this action on? @sgallagh or releng? I guess releng since you can't force push... perhaps we could make a one off script to do this and verify it then releng can run it?

sgallagh commented

2026-03-05 21:10:47 +00:00

Author

Member

Here's a new wrinkle we need to test (I haven't had a chance yet): whether the refs/archive content can be pushed to Forgejo. I assume that there's a plan for us to sunset src.fedoraproject.org at some point in favor of dist-git on Forgejo?

The filter-branch solution is acceptable for what we need out of the sync from Fedora to CentOS Stream, but we may just be delaying how we have to deal with it in Fedora.

Here's a new wrinkle we need to test (I haven't had a chance yet): whether the `refs/archive` content can be pushed to Forgejo. I assume that there's a plan for us to sunset src.fedoraproject.org at some point in favor of dist-git on Forgejo? The filter-branch solution is acceptable for what we need out of the sync from Fedora to CentOS Stream, but we may just be delaying how we have to deal with it in Fedora.

yselkowitz commented

2026-05-04 03:10:44 +00:00

Member

What is needed to move this forward?

kevin commented

2026-05-04 17:59:15 +00:00

Owner

@sgallagh wrote in #11822 (comment):

Here's a new wrinkle we need to test (I haven't had a chance yet): whether the refs/archive content can be pushed to Forgejo. I assume that there's a plan for us to sunset src.fedoraproject.org at some point in favor of dist-git on Forgejo?

I would think that would work fine... it's just a git repo ref?
Yes, we will be moving off src at some point, but timeline and setup is not set (at least that I know of)

The filter-branch solution is acceptable for what we need out of the sync from Fedora to CentOS Stream, but we may just be delaying how we have to deal with it in Fedora.

Perhaps, but we can deal with it then...

@yselkowitz wrote in #11822 (comment):

What is needed to move this forward?

Well, ideally, a script that has the invalid emails/whatever for each repo and can run the filter-branch?
if someone could make that I would be happy to run it.

@sgallagh wrote in https://forge.fedoraproject.org/releng/tickets/issues/11822#issuecomment-574230: > Here's a new wrinkle we need to test (I haven't had a chance yet): whether the `refs/archive` content can be pushed to Forgejo. I assume that there's a plan for us to sunset src.fedoraproject.org at some point in favor of dist-git on Forgejo? I would think that would work fine... it's just a git repo ref? Yes, we will be moving off src at some point, but timeline and setup is not set (at least that I know of) > The filter-branch solution is acceptable for what we need out of the sync from Fedora to CentOS Stream, but we may just be delaying how we have to deal with it in Fedora. Perhaps, but we can deal with it then... @yselkowitz wrote in https://forge.fedoraproject.org/releng/tickets/issues/11822#issuecomment-677775: > What is needed to move this forward? Well, ideally, a script that has the invalid emails/whatever for each repo and can run the filter-branch? if someone could make that I would be happy to run it.

sgallagh commented

2026-05-04 18:04:45 +00:00

Author

Member

@kevin wrote in #11822 (comment):

@sgallagh wrote in #11822 (comment):

Here's a new wrinkle we need to test (I haven't had a chance yet): whether the refs/archive content can be pushed to Forgejo. I assume that there's a plan for us to sunset src.fedoraproject.org at some point in favor of dist-git on Forgejo?

I would think that would work fine... it's just a git repo ref? Yes, we will be moving off src at some point, but timeline and setup is not set (at least that I know of)

I'm not so sure. If Forgejo behaves like every other forge out there today, it will run git fsck as part of it's pre-receive hook and reject the refs that fail it.

The filter-branch solution is acceptable for what we need out of the sync from Fedora to CentOS Stream, but we may just be delaying how we have to deal with it in Fedora.

Perhaps, but we can deal with it then...

@yselkowitz wrote in #11822 (comment):

What is needed to move this forward?

Well, ideally, a script that has the invalid emails/whatever for each repo and can run the filter-branch? if someone could make that I would be happy to run it.

for i in <the list above>; do
    git filter-repo --force --email-callback 'return email.replace(b" <akurtako@redhat.com", b"akurtako@redhat.com")'
done

As far as I know, that's all that is required. It was only ever this one specific email address that had the extra leading < causing the parsing error.

@kevin wrote in https://forge.fedoraproject.org/releng/tickets/issues/11822#issuecomment-692933: > @sgallagh wrote in #11822 (comment): > > > Here's a new wrinkle we need to test (I haven't had a chance yet): whether the `refs/archive` content can be pushed to Forgejo. I assume that there's a plan for us to sunset src.fedoraproject.org at some point in favor of dist-git on Forgejo? > > I would think that would work fine... it's just a git repo ref? Yes, we will be moving off src at some point, but timeline and setup is not set (at least that I know of) I'm not so sure. If Forgejo behaves like every other forge out there today, it will run `git fsck` as part of it's pre-receive hook and reject the refs that fail it. > > The filter-branch solution is acceptable for what we need out of the sync from Fedora to CentOS Stream, but we may just be delaying how we have to deal with it in Fedora. > > Perhaps, but we can deal with it then... > > @yselkowitz wrote in #11822 (comment): > > > What is needed to move this forward? > > Well, ideally, a script that has the invalid emails/whatever for each repo and can run the filter-branch? if someone could make that I would be happy to run it. ```bash for i in <the list above>; do git filter-repo --force --email-callback 'return email.replace(b" <akurtako@redhat.com", b"akurtako@redhat.com")' done ``` As far as I know, that's all that is required. It was only ever this one specific email address that had the extra leading `<` causing the parsing error.

yselkowitz commented

2026-05-04 18:06:33 +00:00

Member

But we need to archive the current rawhide branch too, right?

kevin self-assigned this

2026-05-04 18:17:52 +00:00

kevin commented

2026-05-04 18:18:06 +00:00

Owner

Ah, I misunderstood. I thought there was a collection of emails... I can do that.

yes, we should archive them before making any changes.

I can look at doing this soon...

Ah, I misunderstood. I thought there was a collection of emails... I can do that. yes, we should archive them before making any changes. I can look at doing this soon...

kevin added this to the Sprint ( 2026-05-04 to 2026-05-17 ) project

2026-05-04 18:18:14 +00:00

kevin added the

points

03

day-to-day

labels

2026-05-04 18:18:31 +00:00

kevin referenced this issue from a commit

2026-05-08 17:56:55 +00:00

pkgs: allow access to archive of git repos fixed for fsck issues

kevin commented

2026-05-08 18:04:20 +00:00

Owner

ok, so looking at this again today.

yeah, I guess fesco wanted the orig branches to be copied under archive. I'll have to look at how to do that. I guess thats 'all branches' ?

And... there's not any 'git filter' available on the pkgs01 machine and it's rhel8, so there's no epel8 package for it either.

I suppose we could do the work remotely and rsync it back. ;(

ok, so looking at this again today. yeah, I guess fesco wanted the orig branches to be copied under archive. I'll have to look at how to do that. I guess thats 'all branches' ? And... there's not any 'git filter' available on the pkgs01 machine and it's rhel8, so there's no epel8 package for it either. I suppose we could do the work remotely and rsync it back. ;(

yselkowitz commented

2026-05-08 20:19:36 +00:00

Member

For the purposes of ELN (CS/RHEL bootstrapping), fixing the rawhide branch is sufficient.

sgallagh commented

2026-05-20 16:13:55 +00:00

Author

Member

Okay, this took a little more effort than I had expected to get right, but I've now got a script prepared at https://github.com/sgallagher/fedora-git-filter-repo-bademail that will take a list of packages and perform the necessary changes to clean up the git history, as well as retaining the prior set of commits under refs/archive/bademail. Full details are in the README.md.

This does not make changes directly to the dist-git repos. It copies them locally and makes a bunch of destructive changes and then outputs the bare repos which should be examined by the rel-eng engineer before copying them over to dist-git.

I'd appreciate greatly if I could get a couple sets of eyes on the logic as well. I've tried to be as careful as possible here, but I don't like any situation where git history gets rewritten.

Okay, this took a little more effort than I had expected to get right, but I've now got a script prepared at https://github.com/sgallagher/fedora-git-filter-repo-bademail that will take a list of packages and perform the necessary changes to clean up the git history, as well as retaining the prior set of commits under `refs/archive/bademail`. Full details are in the README.md. This does *not* make changes directly to the dist-git repos. It copies them locally and makes a bunch of __destructive__ changes and then outputs the bare repos which should be examined by the rel-eng engineer before copying them over to dist-git. I'd appreciate greatly if I could get a couple sets of eyes on the logic as well. I've tried to be as careful as possible here, but I don't like any situation where git history gets rewritten.

jnsamyak modified the project from Sprint ( 2026-05-04 to 2026-05-17 ) to Sprint ( 2026-06-01 to 2026-06-14 )

2026-06-01 09:55:34 +00:00

kevin commented

2026-06-03 21:31:26 +00:00

Owner

Sorry for the delay here. Finally looked into this.

Thanks a ton! This is super clear and helpful. I really appreciate it. :)

I ran it here and looked...and... I still see 'git fsck' failing?

git fsck                         
Checking ref database: 100% (1/1), done.
Checking object directories: 100% (256/256), done.
error in commit cf3b89778c4cd30d175dfdca948b2ed0ecccff58: badEmail: invalid author/committer line - bad email

Sorry for the delay here. Finally looked into this. Thanks a ton! This is super clear and helpful. I really appreciate it. :) I ran it here and looked...and... I still see 'git fsck' failing? ``` git fsck Checking ref database: 100% (1/1), done. Checking object directories: 100% (256/256), done. error in commit cf3b89778c4cd30d175dfdca948b2ed0ecccff58: badEmail: invalid author/committer line - bad email ```

mjg commented

2026-06-04 07:45:37 +00:00

@kevin wrote in #11822 (comment):

Sorry for the delay here. Finally looked into this.

Thanks a ton! This is super clear and helpful. I really appreciate it. :)

I ran it here and looked...and... I still see 'git fsck' failing?
git fsck                         
Checking ref database: 100% (1/1), done.
Checking object directories: 100% (256/256), done.
error in commit cf3b89778c4cd30d175dfdca948b2ed0ecccff58: badEmail: invalid author/committer line - bad email

Which repo did you run this in? The rewritten repo still contains the unwritten (including bad) commits for reference. The repo which has only the rewritten commits should fsck fine.

@kevin wrote in https://forge.fedoraproject.org/releng/tickets/issues/11822#issuecomment-795195: > Sorry for the delay here. Finally looked into this. > > Thanks a ton! This is super clear and helpful. I really appreciate it. :) > > I ran it here and looked...and... I still see 'git fsck' failing? > > ```text > git fsck > Checking ref database: 100% (1/1), done. > Checking object directories: 100% (256/256), done. > error in commit cf3b89778c4cd30d175dfdca948b2ed0ecccff58: badEmail: invalid author/committer line - bad email > ``` Which repo did you run this in? The rewritten repo still contains the unwritten (including bad) commits for reference. The repo which has only the rewritten commits should fsck fine.

sgallagh commented

2026-06-04 15:22:22 +00:00

Author

Member

@kevin wrote in #11822 (comment):

Sorry for the delay here. Finally looked into this.

Thanks a ton! This is super clear and helpful. I really appreciate it. :)

I ran it here and looked...and... I still see 'git fsck' failing?
git fsck                         
Checking ref database: 100% (1/1), done.
Checking object directories: 100% (256/256), done.
error in commit cf3b89778c4cd30d175dfdca948b2ed0ecccff58: badEmail: invalid author/committer line - bad email

Yes, the resulting repo will still fail git-fsck, because those commits are still present in the refs/archive/bademail path. That was very intentional, so we don't lose those commits that have been used to build packages.

However, the actual branches under refs/heads will all be clean. So if we want to push those to GitLab (in the case of CentOS Stream 11) or Forgejo at some point, we can exclude the refs/archive/bademail contents.

For an experiment, I just did this:

mkdir antlr-clean; cd antlr-clean
git init .
git remote add dist-git ../output/bare/antlr
git pull dist-git main
git fsck

Which is entirely clean.

Effectively, what this patch does is "hide" the old commits in the refs/archive/bademail path so they don't show up in git branch or git refs list on a clone, but they remain in the repo if you need to reference them directly or you know where to look.

@kevin wrote in https://forge.fedoraproject.org/releng/tickets/issues/11822#issuecomment-795195: > Sorry for the delay here. Finally looked into this. > > Thanks a ton! This is super clear and helpful. I really appreciate it. :) > > I ran it here and looked...and... I still see 'git fsck' failing? > > ```text > git fsck > Checking ref database: 100% (1/1), done. > Checking object directories: 100% (256/256), done. > error in commit cf3b89778c4cd30d175dfdca948b2ed0ecccff58: badEmail: invalid author/committer line - bad email > ``` Yes, the resulting **repo** will still fail git-fsck, because those commits are still present in the `refs/archive/bademail` path. That was very intentional, so we don't lose those commits that have been used to build packages. However, the actual branches under `refs/heads` will all be clean. So if we want to push those to GitLab (in the case of CentOS Stream 11) or Forgejo at some point, we can exclude the `refs/archive/bademail` contents. For an experiment, I just did this: ``` mkdir antlr-clean; cd antlr-clean git init . git remote add dist-git ../output/bare/antlr git pull dist-git main git fsck ``` Which is entirely clean. Effectively, what this patch does is "hide" the old commits in the `refs/archive/bademail` path so they don't show up in `git branch` or `git refs list` on a clone, but they remain in the repo if you need to reference them directly or you know where to look.

kevin commented

2026-06-04 17:10:49 +00:00

Owner

Yeah, I thought that might be the case, but I could not find a way to see what branch/ref git-fsck found that commit object... I am sure there is a way, I just couldn't find it.

Anyhow, I guess that looks good to me, shall I save off the old repos and sync these in place?

Or I suppose we could push them to staging first and confirm there's no weird pagure problems with the repos?

Yeah, I thought that might be the case, but I could not find a way to see what branch/ref git-fsck found that commit object... I am sure there is a way, I just couldn't find it. Anyhow, I guess that looks good to me, shall I save off the old repos and sync these in place? Or I suppose we could push them to staging first and confirm there's no weird pagure problems with the repos?

sgallagh commented

2026-06-04 17:22:38 +00:00

Author

Member

@kevin wrote in #11822 (comment):

Or I suppose we could push them to staging first and confirm there's no weird pagure problems with the repos?

That's probably the safest approach, yeah.

@kevin wrote in https://forge.fedoraproject.org/releng/tickets/issues/11822#issuecomment-817797: > Or I suppose we could push them to staging first and confirm there's no weird pagure problems with the repos? That's probably the safest approach, yeah.

kevin commented

2026-06-09 00:00:15 +00:00

Owner

ok. I have re-run the script then synced all those repos to src.stg.fedoraproject.org.

Can you do a check and see if they look correct to you?

ok. I have re-run the script then synced all those repos to src.stg.fedoraproject.org. Can you do a check and see if they look correct to you?

sgallagh commented

2026-06-09 12:16:19 +00:00

Author

Member

@kevin wrote in #11822 (comment):

ok. I have re-run the script then synced all those repos to src.stg.fedoraproject.org.

Can you do a check and see if they look correct to you?

I don't have access to that server. Could you tar-zip up a couple random repos and attach them here for inspection?

@kevin wrote in https://forge.fedoraproject.org/releng/tickets/issues/11822#issuecomment-818788: > ok. I have re-run the script then synced all those repos to src.stg.fedoraproject.org. > > Can you do a check and see if they look correct to you? I don't have access to that server. Could you tar-zip up a couple random repos and attach them here for inspection?

kevin commented

2026-06-09 18:23:47 +00:00

Owner

You have the same access you have to the prod one? ;)

ie, https://src.stg.fedoraproject.org/rpms/whatever

You should be able to git clone from them, or even push to them if you have commits on that package.

You have the same access you have to the prod one? ;) ie, https://src.stg.fedoraproject.org/rpms/whatever You should be able to git clone from them, or even push to them if you have commits on that package.

sgallagh commented

2026-06-09 19:37:39 +00:00

Author

Member

Sorry. I misunderstood.

A spot check of several affected (and non-affected) packages looks correct to me.

In the future, if we move to Forgejo for dist-git, we will want to just archive these repos somewhere and then push only the refs/heads content into the new hosting site, so we don't carry the broken commits forward. If Forgejo does any sanity checking on push the way that GitLab does, we wouldn't be able to import these repos anyway.

I think we are in good shape here until then. Thanks!

Sorry. I misunderstood. A spot check of several affected (and non-affected) packages looks correct to me. In the future, if we move to Forgejo for dist-git, we will want to just archive these repos somewhere and then push only the `refs/heads` content into the new hosting site, so we don't carry the broken commits forward. If Forgejo does any sanity checking on push the way that GitLab does, we wouldn't be able to import these repos anyway. I think we are in good shape here until then. Thanks!

jnsamyak modified the project from Sprint ( 2026-06-01 to 2026-06-14 ) to Backlog

2026-06-15 14:10:29 +00:00

sgallagh commented

2026-06-22 15:00:59 +00:00

Author

Member

OK, so now that this is verified, can we get this taken care of for PROD as well? The CentOS Stream team is in the process of doing the initial import over for RHEL 11 (they'll continue to sync automatically until the Fedora 46 branch) and these packages are still failing that import.

EDIT: For clarity, we import by pulling and pushing the individual branch, not the entire repository, so as long as this fix is in place, the CS 11 import will be unblocked.

OK, so now that this is verified, can we get this taken care of for PROD as well? The CentOS Stream team is in the process of doing the initial import over for RHEL 11 (they'll continue to sync automatically until the Fedora 46 branch) and these packages are still failing that import. EDIT: For clarity, we import by pulling and pushing the individual branch, not the entire repository, so as long as this fix is in place, the CS 11 import will be unblocked.

kevin commented

2026-06-22 20:12:53 +00:00

Owner

Yes, it's on my list. no need to remind me. ;)

I will try and go do it now and get it over with...

Yes, it's on my list. no need to remind me. ;) I will try and go do it now and get it over with...

kevin commented

2026-06-22 20:30:15 +00:00

Owner

ok. Should be synced now. Can you check some and confirm? Then we can close this if all looks well (after 2 years).

Should we send an announcement about it so maintainers know what happened?

ok. Should be synced now. Can you check some and confirm? Then we can close this if all looks well (after 2 years). Should we send an announcement about it so maintainers know what happened?

sgallagh commented

2026-06-22 20:40:12 +00:00

Author

Member

@kevin wrote in #11822 (comment):

ok. Should be synced now. Can you check some and confirm? Then we can close this if all looks well (after 2 years).

I just pulled antlr and it still appears to be the old repository. I don't see the new archive refs either. Are you sure it synced to the public space?

Should we send an announcement about it so maintainers know what happened?

Yeah, we probably should. I don't know if it needs to go to devel or devel-announce; maybe just to the maintainer lists?

@kevin wrote in https://forge.fedoraproject.org/releng/tickets/issues/11822#issuecomment-826119: > ok. Should be synced now. Can you check some and confirm? Then we can close this if all looks well (after 2 years). I just pulled `antlr` and it still appears to be the old repository. I don't see the new archive refs either. Are you sure it synced to the public space? > Should we send an announcement about it so maintainers know what happened? Yeah, we probably should. I don't know if it needs to go to devel or devel-announce; maybe just to the maintainer lists?

yselkowitz commented

2026-06-22 20:40:24 +00:00

Member

Doesn't seem synced yet, is there some caching in place?

kevin commented

2026-06-22 21:01:41 +00:00

Owner

No, no caching, something didn't sync right it seems, trying to figure out what. Theres serveral hops here...

script on my machine -> rsync output/ dir to batcave01 -> rsync output/ dir to pkgs01 -> for loop to rsync each bare/ repo to the bare/git ones.

trying to see where I went wrong.

No, no caching, something didn't sync right it seems, trying to figure out what. Theres serveral hops here... script on my machine -> rsync output/ dir to batcave01 -> rsync output/ dir to pkgs01 -> for loop to rsync each bare/ repo to the bare/git ones. trying to see where I went wrong.

kevin commented

2026-06-22 21:13:52 +00:00

Owner

Might have been a permissions issue? Does it look any better now?

yselkowitz commented

2026-06-22 21:15:48 +00:00

Member

Still not afaics.

kevin commented

2026-06-22 21:33:25 +00:00

Owner

I am confused here.

% git clone https://src.fedoraproject.org/rpms/antlr.git                       
Cloning into 'antlr'...                                                                           
remote: Enumerating objects: 710, done.                                                           
remote: Counting objects: 100% (710/710), done.                                                   
remote: Compressing objects: 100% (271/271), done.                                                
remote: Total 710 (delta 417), reused 710 (delta 417), pack-reused 0                              Receiving objects: 100% (710/710), 108.46 KiB | 593.00 KiB/s, done.           
Resolving deltas: 100% (417/417), done.            
 %                      git ls-remote                                                              
From https://src.fedoraproject.org/rpms/antlr.git                                    
c24bacbe22e8ffd4c51a631a76e1bbcb89996b2d        HEAD                             
c24bacbe22e8ffd4c51a631a76e1bbcb89996b2d        refs/archive/bademail/HEAD         
8686dc24bfe10c71f688d7d523baba129cafd6f6        refs/archive/bademail/f10        
376eb65e76a093bb6def322ff43ccde6bfba833d        refs/archive/bademail/f11          
49afa50a192ab6c331d9b7738cc6b2ef43e71abc        refs/archive/bademail/f12        
fa51edcbc989c2d1896666d48a9200a00238ee21        refs/archive/bademail/f13 
...

The archive refs are there?

I mean I rsynced the exact dirs from output/bare/ over them... do I need to run with --delete? but I didn't in stg.

I am confused here. ``` % git clone https://src.fedoraproject.org/rpms/antlr.git Cloning into 'antlr'... remote: Enumerating objects: 710, done. remote: Counting objects: 100% (710/710), done. remote: Compressing objects: 100% (271/271), done. remote: Total 710 (delta 417), reused 710 (delta 417), pack-reused 0 Receiving objects: 100% (710/710), 108.46 KiB | 593.00 KiB/s, done. Resolving deltas: 100% (417/417), done. % git ls-remote From https://src.fedoraproject.org/rpms/antlr.git c24bacbe22e8ffd4c51a631a76e1bbcb89996b2d HEAD c24bacbe22e8ffd4c51a631a76e1bbcb89996b2d refs/archive/bademail/HEAD 8686dc24bfe10c71f688d7d523baba129cafd6f6 refs/archive/bademail/f10 376eb65e76a093bb6def322ff43ccde6bfba833d refs/archive/bademail/f11 49afa50a192ab6c331d9b7738cc6b2ef43e71abc refs/archive/bademail/f12 fa51edcbc989c2d1896666d48a9200a00238ee21 refs/archive/bademail/f13 ... ``` The archive refs are there? I mean I rsynced the exact dirs from output/bare/ over them... do I need to run with --delete? but I didn't in stg.

sgallagh commented

2026-06-22 22:11:08 +00:00

Author

Member

The archive refs are supposed to be there (for now). It's how we ensure that the repo has retained all of the commits that have been used to create builds up to this point.

When we eventually move to a new forge, we will want to ONLY carry forward the new refs (probably by pulling and pushing individual branches rather than cloning the entire repository or else by doing a git filter-repo again and purging the archive refs).

That being said... something definitely went wrong here:

c24bacbe22e8ffd4c51a631a76e1bbcb89996b2d        refs/archive/bademail/rawhide
...
c24bacbe22e8ffd4c51a631a76e1bbcb89996b2d        refs/heads/rawhide

Those should absolutely not be the same commit.

Let me see if I can figure out what went wrong with my script, or if the issue is on your end...

The archive refs are supposed to be there (for now). It's how we ensure that the repo has retained all of the commits that have been used to create builds up to this point. When we eventually move to a new forge, we will want to ONLY carry forward the new refs (probably by pulling and pushing individual branches rather than cloning the entire repository or else by doing a `git filter-repo` again and purging the archive refs). That being said... something definitely went wrong here: ``` c24bacbe22e8ffd4c51a631a76e1bbcb89996b2d refs/archive/bademail/rawhide ... c24bacbe22e8ffd4c51a631a76e1bbcb89996b2d refs/heads/rawhide ``` Those should absolutely not be the same commit. Let me see if I can figure out what went wrong with my script, or if the issue is on your end...

sgallagh commented

2026-06-22 22:27:19 +00:00

Author

Member

I just ran my script against antlr:

$ cat pkgs
antlr
$ ./filter-email.sh --debug pkgs
+++ dirname ./filter-email.sh
++ cd .
++ pwd
+ SCRIPT_DIR=/home/sgallagh/workspace/fedora-git-filter-repo-bademail
+ OUTPUT_DIR=/home/sgallagh/workspace/fedora-git-filter-repo-bademail/output
+ mkdir -p /home/sgallagh/workspace/fedora-git-filter-repo-bademail/output/full /home/sgallagh/workspace/fedora-git-filter-repo-bademail/output/bare
+ git filter-repo --version
+ packages=()
+ read -r package
+ packages+=("$package")
+ read -r package
+ pushd /home/sgallagh/workspace/fedora-git-filter-repo-bademail/output/full
~/workspace/fedora-git-filter-repo-bademail/output/full ~/workspace/fedora-git-filter-repo-bademail
+ for package in "${packages[@]}"
+ '[' -z antlr ']'
+ process_package antlr
+ local package=antlr
+ echo 'Processing package: antlr'
Processing package: antlr
+ dist_git_url=https://src.fedoraproject.org/rpms/antlr.git
+ git clone --origin origin https://src.fedoraproject.org/rpms/antlr.git
Cloning into 'antlr'...
remote: Enumerating objects: 710, done.
remote: Counting objects: 100% (710/710), done.
remote: Compressing objects: 100% (271/271), done.
remote: Total 710 (delta 417), reused 710 (delta 417), pack-reused 0
Receiving objects: 100% (710/710), 108.46 KiB | 1.90 MiB/s, done.
Resolving deltas: 100% (417/417), done.
+ pushd /home/sgallagh/workspace/fedora-git-filter-repo-bademail/output/full/antlr
~/workspace/fedora-git-filter-repo-bademail/output/full/antlr ~/workspace/fedora-git-filter-repo-bademail/output/full ~/workspace/fedora-git-filter-repo-bademail
+ git filter-repo --email-callback 'return email.replace(b" <akurtako@redhat.com", b"akurtako@redhat.com")'
NOTICE: Removing 'origin' remote; see 'Why is my origin removed?'
        in the manual if you want to push back there.
        (was https://src.fedoraproject.org/rpms/antlr.git)
Parsed 245 commits
New history written in 0.04 seconds; now repacking/cleaning...
Repacking your repo and cleaning out old unneeded objects
HEAD is now at a7f5c65 Rebuilt for https://fedoraproject.org/wiki/Fedora_44_Mass_Rebuild
Enumerating objects: 710, done.
Counting objects: 100% (710/710), done.
Delta compression using up to 32 threads
Compressing objects: 100% (331/331), done.
Writing objects: 100% (710/710), done.
Total 710 (delta 357), reused 710 (delta 357), pack-reused 0 (from 0)
Completely finished after 0.33 seconds.
+ re_add_remote_refs https://src.fedoraproject.org/rpms/antlr.git
+ local dist_git_url=https://src.fedoraproject.org/rpms/antlr.git
+ git remote add origin https://src.fedoraproject.org/rpms/antlr.git
+ git remote update origin
Fetching origin
remote: Enumerating objects: 358, done.
remote: Counting objects: 100% (358/358), done.
remote: Compressing objects: 100% (112/112), done.
remote: Total 352 (delta 236), reused 348 (delta 232), pack-reused 0
Receiving objects: 100% (352/352), 50.93 KiB | 2.04 MiB/s, done.
Resolving deltas: 100% (236/236), completed with 4 local objects.
From https://src.fedoraproject.org/rpms/antlr
 * [new branch]      f10                              -> origin/f10
[...]
 * [new branch]      javapackages                     -> origin/javapackages
 * [new branch]      main                             -> origin/main
 * [new branch]      rawhide                          -> origin/rawhide
 * [new branch]      stream-ant-1.10                  -> origin/stream-ant-1.10
 * [new branch]      stream-javapackages-tools-201901 -> origin/stream-javapackages-tools-201901
 * [new branch]      stream-javapackages-tools-201902 -> origin/stream-javapackages-tools-201902
+ git for-each-ref '--format=%(refname:lstrip=3)' refs/remotes/origin/
+ read -r ref
+ git update-ref -m 'Archiving remote ref: HEAD' refs/archive/bademail/HEAD refs/remotes/origin/HEAD
+ read -r ref
+ git update-ref -m 'Archiving remote ref: f10' refs/archive/bademail/f10 refs/remotes/origin/f10
[...]
+ git update-ref -m 'Archiving remote ref: stream-javapackages-tools-201901' refs/archive/bademail/stream-javapackages-tools-201901 refs/remotes/origin/stream-javapackages-tools-201901
+ read -r ref
+ git update-ref -m 'Archiving remote ref: stream-javapackages-tools-201902' refs/archive/bademail/stream-javapackages-tools-201902 refs/remotes/origin/stream-javapackages-tools-201902
+ read -r ref
+ git remote remove origin
+ popd
~/workspace/fedora-git-filter-repo-bademail/output/full ~/workspace/fedora-git-filter-repo-bademail
+ git clone --mirror /home/sgallagh/workspace/fedora-git-filter-repo-bademail/output/full/antlr /home/sgallagh/workspace/fedora-git-filter-repo-bademail/output/bare/antlr
Cloning into bare repository '/home/sgallagh/workspace/fedora-git-filter-repo-bademail/output/bare/antlr'...
done.
+ popd
~/workspace/fedora-git-filter-repo-bademail

I then cloned the bare repo and inspected it:

$ mkdir output/cloned; cd output/cloned
$ git clone ../bare/antlr; cd antlr
Cloning into 'antlr'...
done.
$ git ls-remote
From /home/sgallagh/workspace/fedora-git-filter-repo-bademail/output/cloned/../bare/antlr
a7f5c65c2acb20951144f1e48e0bf1a3c7560ea0        HEAD
c24bacbe22e8ffd4c51a631a76e1bbcb89996b2d        refs/archive/bademail/HEAD
8686dc24bfe10c71f688d7d523baba129cafd6f6        refs/archive/bademail/f10
[...]
c24bacbe22e8ffd4c51a631a76e1bbcb89996b2d        refs/archive/bademail/main
c24bacbe22e8ffd4c51a631a76e1bbcb89996b2d        refs/archive/bademail/rawhide
[...]
a7f5c65c2acb20951144f1e48e0bf1a3c7560ea0        refs/heads/main
a7f5c65c2acb20951144f1e48e0bf1a3c7560ea0        refs/heads/rawhide
[...]

I also note that git refs list does not show refs/archive/bademail/*, as expected.

I therefore have to conclude that the copy over to the final location was performed in error.

I just ran my script against `antlr`: ``` $ cat pkgs antlr $ ./filter-email.sh --debug pkgs +++ dirname ./filter-email.sh ++ cd . ++ pwd + SCRIPT_DIR=/home/sgallagh/workspace/fedora-git-filter-repo-bademail + OUTPUT_DIR=/home/sgallagh/workspace/fedora-git-filter-repo-bademail/output + mkdir -p /home/sgallagh/workspace/fedora-git-filter-repo-bademail/output/full /home/sgallagh/workspace/fedora-git-filter-repo-bademail/output/bare + git filter-repo --version + packages=() + read -r package + packages+=("$package") + read -r package + pushd /home/sgallagh/workspace/fedora-git-filter-repo-bademail/output/full ~/workspace/fedora-git-filter-repo-bademail/output/full ~/workspace/fedora-git-filter-repo-bademail + for package in "${packages[@]}" + '[' -z antlr ']' + process_package antlr + local package=antlr + echo 'Processing package: antlr' Processing package: antlr + dist_git_url=https://src.fedoraproject.org/rpms/antlr.git + git clone --origin origin https://src.fedoraproject.org/rpms/antlr.git Cloning into 'antlr'... remote: Enumerating objects: 710, done. remote: Counting objects: 100% (710/710), done. remote: Compressing objects: 100% (271/271), done. remote: Total 710 (delta 417), reused 710 (delta 417), pack-reused 0 Receiving objects: 100% (710/710), 108.46 KiB | 1.90 MiB/s, done. Resolving deltas: 100% (417/417), done. + pushd /home/sgallagh/workspace/fedora-git-filter-repo-bademail/output/full/antlr ~/workspace/fedora-git-filter-repo-bademail/output/full/antlr ~/workspace/fedora-git-filter-repo-bademail/output/full ~/workspace/fedora-git-filter-repo-bademail + git filter-repo --email-callback 'return email.replace(b" <akurtako@redhat.com", b"akurtako@redhat.com")' NOTICE: Removing 'origin' remote; see 'Why is my origin removed?' in the manual if you want to push back there. (was https://src.fedoraproject.org/rpms/antlr.git) Parsed 245 commits New history written in 0.04 seconds; now repacking/cleaning... Repacking your repo and cleaning out old unneeded objects HEAD is now at a7f5c65 Rebuilt for https://fedoraproject.org/wiki/Fedora_44_Mass_Rebuild Enumerating objects: 710, done. Counting objects: 100% (710/710), done. Delta compression using up to 32 threads Compressing objects: 100% (331/331), done. Writing objects: 100% (710/710), done. Total 710 (delta 357), reused 710 (delta 357), pack-reused 0 (from 0) Completely finished after 0.33 seconds. + re_add_remote_refs https://src.fedoraproject.org/rpms/antlr.git + local dist_git_url=https://src.fedoraproject.org/rpms/antlr.git + git remote add origin https://src.fedoraproject.org/rpms/antlr.git + git remote update origin Fetching origin remote: Enumerating objects: 358, done. remote: Counting objects: 100% (358/358), done. remote: Compressing objects: 100% (112/112), done. remote: Total 352 (delta 236), reused 348 (delta 232), pack-reused 0 Receiving objects: 100% (352/352), 50.93 KiB | 2.04 MiB/s, done. Resolving deltas: 100% (236/236), completed with 4 local objects. From https://src.fedoraproject.org/rpms/antlr * [new branch] f10 -> origin/f10 [...] * [new branch] javapackages -> origin/javapackages * [new branch] main -> origin/main * [new branch] rawhide -> origin/rawhide * [new branch] stream-ant-1.10 -> origin/stream-ant-1.10 * [new branch] stream-javapackages-tools-201901 -> origin/stream-javapackages-tools-201901 * [new branch] stream-javapackages-tools-201902 -> origin/stream-javapackages-tools-201902 + git for-each-ref '--format=%(refname:lstrip=3)' refs/remotes/origin/ + read -r ref + git update-ref -m 'Archiving remote ref: HEAD' refs/archive/bademail/HEAD refs/remotes/origin/HEAD + read -r ref + git update-ref -m 'Archiving remote ref: f10' refs/archive/bademail/f10 refs/remotes/origin/f10 [...] + git update-ref -m 'Archiving remote ref: stream-javapackages-tools-201901' refs/archive/bademail/stream-javapackages-tools-201901 refs/remotes/origin/stream-javapackages-tools-201901 + read -r ref + git update-ref -m 'Archiving remote ref: stream-javapackages-tools-201902' refs/archive/bademail/stream-javapackages-tools-201902 refs/remotes/origin/stream-javapackages-tools-201902 + read -r ref + git remote remove origin + popd ~/workspace/fedora-git-filter-repo-bademail/output/full ~/workspace/fedora-git-filter-repo-bademail + git clone --mirror /home/sgallagh/workspace/fedora-git-filter-repo-bademail/output/full/antlr /home/sgallagh/workspace/fedora-git-filter-repo-bademail/output/bare/antlr Cloning into bare repository '/home/sgallagh/workspace/fedora-git-filter-repo-bademail/output/bare/antlr'... done. + popd ~/workspace/fedora-git-filter-repo-bademail ``` I then cloned the bare repo and inspected it: ``` $ mkdir output/cloned; cd output/cloned $ git clone ../bare/antlr; cd antlr Cloning into 'antlr'... done. $ git ls-remote From /home/sgallagh/workspace/fedora-git-filter-repo-bademail/output/cloned/../bare/antlr a7f5c65c2acb20951144f1e48e0bf1a3c7560ea0 HEAD c24bacbe22e8ffd4c51a631a76e1bbcb89996b2d refs/archive/bademail/HEAD 8686dc24bfe10c71f688d7d523baba129cafd6f6 refs/archive/bademail/f10 [...] c24bacbe22e8ffd4c51a631a76e1bbcb89996b2d refs/archive/bademail/main c24bacbe22e8ffd4c51a631a76e1bbcb89996b2d refs/archive/bademail/rawhide [...] a7f5c65c2acb20951144f1e48e0bf1a3c7560ea0 refs/heads/main a7f5c65c2acb20951144f1e48e0bf1a3c7560ea0 refs/heads/rawhide [...] ``` I also note that `git refs list` does not show `refs/archive/bademail/*`, as expected. I therefore have to conclude that the copy over to the final location was performed in error.

kevin commented

2026-06-22 23:51:37 +00:00

Owner

It's not right on my laptop. I am not sure why... digging...

I just ran the script. I wonder if something changed in rawhide that broke things...

It's not right on my laptop. I am not sure why... digging... I just ran the script. I wonder if something changed in rawhide that broke things...

kevin commented

2026-06-23 00:03:08 +00:00

Owner

ok. I think I got it. It was the rsync.

Can you all look now?

ok. I think I got it. It was the rsync. Can you all look now?