| @mnasiadka:matrix.org | Ok, had some fiddling with ipv6 on mirror03 in VexxHost's ca-ymq-1 region, and what we're getting is some... rogue ipv6 ND with router bit set? Although it comes from MAC addresses that are not Neutron ports (I checked both projects in VexxHost): | 11:26 |
|---|---|---|
| ``` | ||
| # tcpdump -i ens3 -nn -tttt -e 'icmp6 and ip6[40] == 136 and (ip6[44] & 0x80 != 0)' -vvv | ||
| tcpdump: listening on ens3, link-type EN10MB (Ethernet), snapshot length 262144 bytes | ||
| 2026-09-22 11:25:52.405053 fa:16:3e:b0:bb:6e > 33:33:00:00:00:01, ethertype IPv6 (0x86dd), length 86: (hlim 255, next-header ICMPv6 (58) payload length: 32) 2604:e100:1:0:f816:3eff:feb0:bb6e > ff02::1: [icmp6 sum ok] ICMP6, neighbor advertisement, length 32, tgt is 2604:e100:1:0:f816:3eff:feb0:bb6e, Flags [router, override] | ||
| destination link-address option (2), length 8 (1): fa:16:3e:b0:bb:6e | ||
| 0x0000: fa16 3eb0 bb6e | ||
| ``` | ||
| @mnasiadka:matrix.org | * Ok, had some fiddling with ipv6 on mirror03 in VexxHost's ca-ymq-1 region, and what we're getting is some... rogue ipv6 ND with router bit set? Although it comes from MAC addresses that are not Neutron ports (I checked both projects in VexxHost): | 11:27 |
| ``` | ||
| # tcpdump -i ens3 -nn -tttt -e 'icmp6 and ip6[40] == 136 and (ip6[44] & 0x80 != 0)' -vvv | ||
| tcpdump: listening on ens3, link-type EN10MB (Ethernet), snapshot length 262144 bytes | ||
| 2026-09-22 11:25:52.405053 fa:16:3e:b0:bb:6e > 33:33:00:00:00:01, ethertype IPv6 (0x86dd), length 86: (hlim 255, next-header ICMPv6 (58) payload length: 32) 2604:e100:1:0:f816:3eff:feb0:bb6e > ff02::1: [icmp6 sum ok] ICMP6, | ||
| neighbor advertisement, length 32, tgt is 2604:e100:1:0:f816:3eff:feb0:bb6e, Flags [router, override] | ||
| destination link-address option (2), length 8 (1): fa:16:3e:b0:bb:6e | ||
| 0x0000: fa16 3eb0 bb6e | ||
| ``` | ||
| @mnasiadka:matrix.org | * Ok, had some fiddling with ipv6 on mirror03 in VexxHost's ca-ymq-1 region, and what we're getting is some... rogue ipv6 ND with router bit set? Although it comes from MAC addresses that are not Neutron ports (I checked both projects in VexxHost): | 11:27 |
| ``` | ||
| root@mirror03:~# tcpdump -i ens3 -nn -tttt -e 'icmp6 and ip6[40] == 136 and (ip6[44] & 0x80 != 0)' -vvv | ||
| tcpdump: listening on ens3, link-type EN10MB (Ethernet), snapshot length 262144 bytes | ||
| 2026-09-22 11:25:52.405053 fa:16:3e:b0:bb:6e > 33:33:00:00:00:01, ethertype IPv6 (0x86dd), length 86: (hlim 255, next-header ICMPv6 (58) payload length: 32) 2604:e100:1:0:f816:3eff:feb0:bb6e > ff02::1: [icmp6 sum ok] ICMP6, neighbor advertisement, length 32, tgt is 2604:e100:1:0:f816:3eff:feb0:bb6e, Flags [router, override] | ||
| destination link-address option (2), length 8 (1): fa:16:3e:b0:bb:6e | ||
| 0x0000: fa16 3eb0 bb6e | ||
| ``` | ||
| @mnasiadka:matrix.org | * Ok, had some fiddling with ipv6 on mirror03 in VexxHost's ca-ymq-1 region, and what we're getting is some... | 11:36 |
| rogue ipv6 ND with router bit set? | ||
| Although it comes from MAC addresses that are not Neutron ports (I checked both projects in VexxHost): | ||
| ``` | ||
| root@mirror03:~# tcpdump -i ens3 -nn -tttt -e 'icmp6 and ip6[40] == 136 and (ip6[44] & 0x80 != 0)' -vvv | ||
| tcpdump: listening on ens3, link-type EN10MB (Ethernet), snapshot length 262144 bytes | ||
| 2026-09-22 11:25:52.405053 fa:16:3e:b0:bb:6e > 33:33:00:00:00:01, ethertype IPv6 (0x86dd), length 86: (hlim 255, next-header ICMPv6 (58) payload length: 32) 2604:e100:1:0:f816:3eff:feb0:bb6e > ff02::1: [icmp6 sum ok] ICMP6, neighbor advertisement, length 32, tgt is 2604:e100:1:0:f816:3eff:feb0:bb6e, Flags [router, override] | ||
| destination link-address option (2), length 8 (1): fa:16:3e:b0:bb:6e | ||
| 0x0000: fa16 3eb0 bb6e | ||
| ``` | ||
| @mnasiadka:matrix.org | mnaser: Any idea what can cause this? ^^ | 11:37 |
| @mnasiadka:matrix.org | Basically two mac addresses are sending those packets: | 11:39 |
| - fa:16:3e:fa:c5:a8 | ||
| - fa:16:3e:b0:bb:6e | ||
| @mnasiadka:matrix.org | Ok, opened a ticket in VexxHost | 11:46 |
| @fungicide:matrix.org | thanks mnasiadka! hopefully that's enough for them to track down through bridge tables/ports to find where they originate | 12:58 |
| @mnasiadka:matrix.org | Yes, they replied they are on it, so I assume there should be an answer soon :-) | 12:58 |
| @mnasiadka:matrix.org | Should I use some shared mailbox for such tickets in future? | 12:59 |
| @fungicide:matrix.org | fa:16:3e is our infra-root at openstack.org inbox is what we usually use | 13:01 |
| @fungicide:matrix.org | er, i spliced two comments there, sorry | 13:01 |
| @fungicide:matrix.org | fa:16:3e is not showing up in my old ieee oui tables | 13:01 |
| @fungicide:matrix.org | looks likely to be locally-administered | 13:03 |
| @mnasiadka:matrix.org | Isn't fa:16:3e what Neutron usually uses? | 13:10 |
| @fungicide:matrix.org | no idea, since (amusingly) i don't really manage neutron with any regularity | 13:13 |
| @fungicide:matrix.org | but if it is, then it could either be something leaking from another part of the cloud infrastructure, or from neutron running in a devstack for one of our test nodes (which would still be concerning as it's in a different tenant) | 13:15 |
| @mnasiadka:matrix.org | the mac addresses are not visible as neutron ports from any of two tenants that we own, so I'd assume it's a third tenant or something like that? a bit weird | 13:46 |
| @mnasiadka:matrix.org | maybe that public network is shared amongst all tenants | 13:49 |
| @jim:acmegating.com | quick update on the blob-store related zuul issue: i don't see any errors today, so i'm going to examine the theory that either the upgrade or restart caused the issues as a one-time event. | 13:54 |
| @jim:acmegating.com | i think i have found the issue with the blob store; i have monkeypatched the production schedulers with the fix, though, due to the nature of the error, there could be some lingering issues; they should clear up on their own eventually. | 18:41 |
| @clarkb:matrix.org | looks like zuul got a fix proposed too | 18:42 |
| @jim:acmegating.com | yep; would be good to restart with that once it merges | 18:43 |
| @fungicide:matrix.org | it looks like i'll need to adjust the compacting logic in the prune-borg-backups script, i missed that we're writing a heredoc into sudo so the flag envvar we set in the outer portion of the script doesn't make it into that inner script | 20:12 |
| @fungicide:matrix.org | #status log Pruned /opt/backups-202605 on backup03.ca-ymq-1.vexxhost reducing utilization from 97% to 47% | 20:49 |
| @status:opendev.org | @fungicide:matrix.org: finished logging | 20:49 |
| @fungicide:matrix.org | that was after i commented out the conditional to force it to compact on that server | 20:50 |
| @fungicide:matrix.org | i've added myself a reminder to figure out how to correct the script so the var gets passed in properly | 20:50 |
| -@gerrit:opendev.org- Clark Boylan proposed: [opendev/system-config] 1006843: Update Gerrit images to 3.13.9 and 3.14.3 https://review.opendev.org/c/opendev/system-config/+/1006843 | 21:40 | |
| -@gerrit:opendev.org- Clark Boylan proposed: [opendev/system-config] 1006843: Update Gerrit images to 3.13.9 and 3.14.3 https://review.opendev.org/c/opendev/system-config/+/1006843 | 21:51 | |
| -@gerrit:opendev.org- Zuul merged on behalf of Clark Boylan: [opendev/system-config] 1006843: Update Gerrit images to 3.13.9 and 3.14.3 https://review.opendev.org/c/opendev/system-config/+/1006843 | 22:50 | |
| @fungicide:matrix.org | #status notice The Gerrit service on review.opendev.org will be offline momentarily while we upgrade to a new patch release, but should return within a few minutes | 23:00 |
| @status:opendev.org | @fungicide:matrix.org: sending notice | 23:00 |
| -@status:opendev.org- NOTICE: The Gerrit service on review.opendev.org will be offline momentarily while we upgrade to a new patch release, but should return within a few minutes | 23:03 | |
| @status:opendev.org | @fungicide:matrix.org: finished sending notice | 23:03 |
| @jim:acmegating.com | #status log restarted zuul schedulers with blobstore fix | 23:03 |
| @status:opendev.org | @jim:acmegating.com: finished logging | 23:03 |
| -@gerrit:opendev.org- Clark Boylan proposed: [opendev/bindep] 1006865: DNM push check for gerrit https://review.opendev.org/c/opendev/bindep/+/1006865 | 23:06 | |
Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!