infra/tickets
15
4
Fork 0
Code Issues 88 Projects Wiki Activity Actions

vmhost-p09-copr ipv6 woes #13085

New issue
Closed
opened 2026-01-26 17:42:21 +00:00 by kevin · 3 comments
kevin commented 2026-01-26 17:42:21 +00:00
Owner
Copy link

Description of request

The vmhost-p09-copr01/02/03/04 machines in rdu3 have a ipv6 network setup to allow ssh in from the copr backend to manage them and get build artifacts off them.

Connectivity to the actual hypervisors seems to be working just fine.
However, vm's on those builders come up and sporadically become unreachable via ipv6.

It may be some bridge/bond issue, but everything is setup exactly like many of our other machines.
Or it could be some ipv6 / firewall bug.
Need to gather more info/debugging.

### Description of request The vmhost-p09-copr01/02/03/04 machines in rdu3 have a ipv6 network setup to allow ssh in from the copr backend to manage them and get build artifacts off them. Connectivity to the actual hypervisors seems to be working just fine. However, vm's on those builders come up and sporadically become unreachable via ipv6. It may be some bridge/bond issue, but everything is setup exactly like many of our other machines. Or it could be some ipv6 / firewall bug. Need to gather more info/debugging.
kevin commented 2026-01-28 22:45:49 +00:00
Author
Owner
Copy link

I updated all the vmhost-p09-copr* machines to the latest today and rebooted them.

The problem still seems to be there.

pinging a builder from my home network I notice:

22:39:07.153445 IP6 2620:52:6:1161:dead:beef:cafe:c110.llmnr > vmhost-p09-copr02.rdu3.fedoraproject.org.57324: Flags [S.], seq 3245253773, ack 1703004485, win 64260, options [mss 1440,sackOK,TS val 249750629 ecr 3268020716,nop,wscale 10], length 0
22:39:07.154235 IP6 2620:52:6:1161::2 > 2620:52:6:1161:dead:beef:cafe:c110: ICMP6, time exceeded in-transit for vmhost-p09-copr02.rdu3.fedoraproject.org, length 88

Will see if networking has any ideas there...

I updated all the vmhost-p09-copr* machines to the latest today and rebooted them. The problem still seems to be there. pinging a builder from my home network I notice: ``` 22:39:07.153445 IP6 2620:52:6:1161:dead:beef:cafe:c110.llmnr > vmhost-p09-copr02.rdu3.fedoraproject.org.57324: Flags [S.], seq 3245253773, ack 1703004485, win 64260, options [mss 1440,sackOK,TS val 249750629 ecr 3268020716,nop,wscale 10], length 0 22:39:07.154235 IP6 2620:52:6:1161::2 > 2620:52:6:1161:dead:beef:cafe:c110: ICMP6, time exceeded in-transit for vmhost-p09-copr02.rdu3.fedoraproject.org, length 88 ``` Will see if networking has any ideas there...
kevin commented 2026-01-31 18:24:42 +00:00
Author
Owner
Copy link

They said that they found the same thing and it's like theres a loop on one port where it bounces back and forth and hits tome exceeded.

They want me to get tcpdumps from both interfaces on the server and also from inside the guest builder.
We will need copr folks to get tcpdump available in the guest for us to do that part.

They said that they found the same thing and it's like theres a loop on one port where it bounces back and forth and hits tome exceeded. They want me to get tcpdumps from both interfaces on the server and also from inside the guest builder. We will need copr folks to get tcpdump available in the guest for us to do that part.
kevin commented 2026-02-13 17:27:07 +00:00
Author
Owner
Copy link

Thanks to a bunch of work from @praiskup and puk from the networking team, this was tracked down to mac caching in the switching layer. ;(

It may now be fixed, but if it comes up again we should know the bug and can report it upstream.

Great work!

Thanks to a bunch of work from @praiskup and puk from the networking team, this was tracked down to mac caching in the switching layer. ;( It may now be fixed, but if it comes up again we should know the bug and can report it upstream. Great work!
kevin closed this issue 2026-02-13 17:27:07 +00:00
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
No results found.
Labels
Clear labels   
announcement

   
anubis

Issues related to anubis

  
authentication

Authentication related issues

  
automate

   
aws

Issues related to Amazon Web Services

  
backlog

Tasks that will take longer then ~4 hours to do

  
blocked

Issue is blocked

  
bodhi

Issues with Bodhi

  
ci

Continuous Integration

  
Closed As
Duplicate
Closed with the reason of Duplicate

  
Closed As
Fixed
Closed with the reason of Fixed

  
Closed As
Fixed with Explanation
Closed with the reason of Fixed with Explanation

  
Closed As
Initiative Worthy
Closed with the reason of Initiative Worthy

  
Closed As
Insufficient data
Closed with the reason of Insufficient data

  
Closed As
Invalid
Closed with the reason of Invalid

  
Closed As
Spam
Closed with the reason of Spam

  
Closed As
Upstream
Closed with the reason of Upstream

  
Closed As/Will Not
Can Not fix
Closed with the reason of Will Not/Can Not fix

  
cloud

Items in FedoraInfraCloud or COmmunishift

  
communishift

issues/questions about os.edorainfracloud.org ( Community OpenShift)

  
copr

COPR system (run by other team)

  
database

Issues related to databases

  
day-to-day

These are small self contained tasks that are part of normal operations, but are async from sprint planning.

  
dc-move

Items related to datacenter move

  
deprecated

Tools which Infrastructure may not be able to fix

  
dev

Tasks requiring some dev work

  
discourse

outsourced tool

  
dns

Changes or problems with DNS

  
downloads

rsync,dl.fp.o,AWS,torrents

  
easyfix

   
epel

Issues relating to epel

  
factory2

Generic issues for the factory2 team

  
firmitas

   
forgejo_migration

Request migration to the new forge

  
Gain
High
task that gets us a lot / is important / makes many people happy

  
Gain
Low
tasks that gets us little / isn't very important / makes only 1 person or few happy

  
Gain
Medium
task that gets us some / is somewhat important / makes some people happy

  
gitlab

   
greenwave

Issues with greenwave

  
hardware

Problem is with Fedora Infrastructure Owned/Leased Hardware

  
help wanted

   
high-trouble

task that is complex/long/difficult

  
iad2

Something to do with the iad2 datacenter

  
koji

Issues with the build system

  
koschei

Items needing Koschei team

  
lists

issues with lists

  
low-trouble

task that is easy/short

  
mbs

Issues with mbs.fedoraproject.org

  
medium-trouble

task that is medium complex / difficult

  
mini-initiative

Something that is bigger than one person but not a full initiative

  
mirrorlists

Way to tell systems where downloads are

  
monitoring

   
Needs investigation

   
notifier

   
odcs

On Demand Compose Service issues

  
OpenShift

Issues with OpenShift

  
ops

ops problem now...

  
OSBS

OpenShift Build System issues in Fedora

  
outage

   
packager_workflow_blocker

Blocks the packager workflow

  
pagure

Issues with pagure.io

  
permissions

Items which need permissions set in outside tools

  
Priority
Needs Review
Priority of Needs Review

  
Priority
Next Meeting
Priority of Next Meeting

  
Priority
🔥 URGENT 🔥
Priority of 🔥 URGENT 🔥

  
Priority
Waiting on Assignee
Priority of Waiting on Assignee

  
Priority
Waiting on External
Priority of Waiting on External

  
Priority
Waiting on Reporter
Priority of Waiting on Reporter

  
rabbitmq

issues with the message broker

  
rdu-cc

Items doing with RDU Community Cage

  
release-monitoring

release-monitoring.org (Anitya)

  
releng

For problems which should be in the releng ticket system

  
repoSpanner

Issues with repoSpanner

  
request-for-resources

   
s390x

The secondary architecture hardware and problems related with it

  
security

   
SMTP

Email issues outside of lists (DMARC, SPF, aliases)

  
sprint-0

   
sprint-1

   
src.fp.o

Issues related to src.fedoraproject.org

  
staging

Issues in staging

  
taiga

outsourced tool

  
unfreeze

Patches to be applied after freeze is lifted

  
waiverdb

Issues with waiverdb

  
websites-general

General items with a website focus

  
wiki

Issues with the Wiki

  
Backlog Status
Needs Review
Work items that have been created or updated but need review, refinement, or approval before they can be considered ready for sprint planning.

  
Backlog Status
Ready
Work items that have been reviewed, refined, and are ready to be committed to a sprint. These items have clear requirements and can be worked on immediately when capacity is available.

  
chore

Work that needs to be done but doesn't add new functionality or fix bugs. Maintenance tasks that keep the system running smoothly and the codebase clean.

  
documentation

Work focused on creating, updating, or improving documentation for users, developers, or system administrators.

  
points
01
A task that is simple, small, and has very little uncertainty. It should be easy enough for any team member to pick up and complete quickly, without a lot of discussion or research.

  
points
02
This is a task that is slightly more complex or larger than a "1." It might involve two or three simple steps, but the outcome is still very clear and there's little uncertainty.

  
points
03
Medium-sized task with some noticeable complexity. It might require a bit more thought or collaboration. The steps are well-defined, but it's not a trivial change.

  
points
05
A significant piece of work. It has more complexity and a higher degree of uncertainty than smaller tasks. It might require some research, a design discussion, or collaboration across multiple teams.

  
points
08
Complex task with a lot of uncertainty. It's often the largest size that a team will consider for a single sprint. It likely requires a detailed breakdown and multiple team members to complete.

  
points
13
Any task that feels like it's a "13" or more is considered an Epic. These are too large and uncertain to be worked on in a single sprint and should be broken down.

  
Priority
High
This is for work that is critical and needs to be addressed in the next sprint. It's often for urgent bugs, important features, or a major piece of technical debt that is blocking other work.

  
Priority
Low
This is for work that is good to do but not urgent. It's a candidate for being worked on when there's extra time or when it rises in importance. It can stay in the backlog for a longer period.

  
Priority
Medium
This is for important work that should be done soon, but isn't as urgent as a high-priority item. It will likely be considered for the next 1-2 sprints.

  
Sprint Status
Blocked
Work items that cannot progress due to external dependencies, blockers, or issues that need resolution.

  
Sprint Status
Done
Work items that have been completed, reviewed, and integrated. No further work is needed within the current sprint.

  
Sprint Status
In Progress
Work items currently being worked on by a team member. Active development, research, or implementation is happening.

  
Sprint Status
Review
Work items that are complete but need review, testing, or approval before being marked as done.

  
Sprint Status
To Do
Work items that are committed to the sprint but haven't been started yet. These items are ready to be picked up by team members.

  
Technical Debt

Work that addresses shortcuts, workarounds, or suboptimal implementations that were done previously to meet deadlines. These items improve code quality, maintainability, or performance without adding new features.

  
Work Item
Bug
Issues where the system is not working as intended or designed. Bugs represent deviations from expected behavior.

  
Work Item
Epic
Large initiatives that are broken down into multiple smaller work items (user stories, tasks, bugs). Epics represent significant features or projects that span multiple sprints.

  
Work Item
Spike
Time-boxed research, investigation, or proof-of-concept work to reduce uncertainty or explore solutions. Spikes help inform future development decisions.

  
Work Item
Task
Used to break down larger user stories into smaller, manageable implementation pieces. They focus on the specific technical steps needed to deliver the user story's value.

  
Work Item
User Story
User-facing features or functionality that can be framed from a user perspective, describing what someone wants to accomplish and why.

No labels
announcement
anubis
authentication
automate
aws
backlog
blocked
bodhi
ci
Closed As
Duplicate
Closed As
Fixed
Closed As
Fixed with Explanation
Closed As
Initiative Worthy
Closed As
Insufficient data
Closed As
Invalid
Closed As
Spam
Closed As
Upstream
Closed As/Will Not
Can Not fix
cloud
communishift
copr
database
day-to-day
dc-move
deprecated
dev
discourse
dns
downloads
easyfix
epel
factory2
firmitas
forgejo_migration
Gain
High
Gain
Low
Gain
Medium
gitlab
greenwave
hardware
help wanted
high-trouble
iad2
koji
koschei
lists
low-trouble
mbs
medium-trouble
mini-initiative
mirrorlists
monitoring
Needs investigation
notifier
odcs
OpenShift
ops
OSBS
outage
packager_workflow_blocker
pagure
permissions
Priority
Needs Review
Priority
Next Meeting
Priority
🔥 URGENT 🔥
Priority
Waiting on Assignee
Priority
Waiting on External
Priority
Waiting on Reporter
rabbitmq
rdu-cc
release-monitoring
releng
repoSpanner
request-for-resources
s390x
security
SMTP
sprint-0
sprint-1
src.fp.o
staging
taiga
unfreeze
waiverdb
websites-general
wiki
Backlog Status
Needs Review
Backlog Status
Ready
chore
documentation
points
01
points
02
points
03
points
05
points
08
points
13
Priority
High
Priority
Low
Priority
Medium
Sprint Status
Blocked
Sprint Status
Done
Sprint Status
In Progress
Sprint Status
Review
Sprint Status
To Do
Technical Debt
Work Item
Bug
Work Item
Epic
Work Item
Spike
Work Item
Task
Work Item
User Story
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Current Sprint ( 2026-02-09 to 2026-02-22 )
Assignees
Clear assignees
No assignees
kevin
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
infra/tickets#13085
No description provided.