Friday, 2020-10-30

*** jamesmcarthur has joined #openstack-infra00:09
*** zbr has quit IRC00:11
*** zbr has joined #openstack-infra00:11
*** zxiiro has joined #openstack-infra00:15
*** tosky has quit IRC00:27
*** jamesmcarthur has quit IRC00:47
*** zzzeek has quit IRC01:00
*** zzzeek has joined #openstack-infra01:01
*** dklyle has quit IRC01:10
*** dklyle has joined #openstack-infra01:12
*** jamesmcarthur has joined #openstack-infra01:13
*** jamesmcarthur has quit IRC01:29
*** jamesmcarthur has joined #openstack-infra01:29
*** jamesmcarthur has quit IRC01:34
*** jamesmcarthur has joined #openstack-infra01:37
*** howell has quit IRC01:49
*** rpioso has quit IRC01:50
*** PrinzElvis has quit IRC01:50
*** srwilkers has quit IRC01:50
*** csatari has quit IRC01:50
*** ildikov has quit IRC01:50
*** cjohnston has quit IRC01:50
*** dviroel has quit IRC01:50
*** howell has joined #openstack-infra01:51
*** srwilkers has joined #openstack-infra01:51
*** PrinzElvis has joined #openstack-infra01:51
*** rpioso has joined #openstack-infra01:51
*** ildikov has joined #openstack-infra01:51
*** csatari has joined #openstack-infra01:51
*** dviroel has joined #openstack-infra01:51
*** cjohnston has joined #openstack-infra01:51
*** knikolla has quit IRC01:52
*** knikolla has joined #openstack-infra01:53
*** gyee has quit IRC02:10
*** dave-mccowan has joined #openstack-infra02:29
*** dklyle has quit IRC02:32
*** ramishra has joined #openstack-infra02:41
*** rcernin has quit IRC02:43
*** rcernin has joined #openstack-infra03:12
*** psachin has joined #openstack-infra03:19
*** jamesmcarthur has quit IRC03:25
*** jamesmcarthur has joined #openstack-infra03:28
*** rcernin has quit IRC03:37
*** rcernin has joined #openstack-infra04:01
openstackgerritIan Wienand proposed openstack/project-config master: Use internal mirror for RAX IAD/DFW  https://review.opendev.org/76049504:09
*** rcernin has quit IRC04:10
*** rcernin has joined #openstack-infra04:10
*** ysandeep|away is now known as ysandeep|ruck04:16
*** matbu has quit IRC04:22
*** matbu has joined #openstack-infra04:24
*** irclogbot_2 has quit IRC04:27
*** irclogbot_1 has joined #openstack-infra04:32
*** jamesmcarthur has quit IRC04:47
*** jamesmcarthur has joined #openstack-infra04:48
*** jamesmcarthur has quit IRC04:52
*** ykarel has joined #openstack-infra05:10
*** akahat is now known as akahat|PTO05:20
*** matt_kosut has joined #openstack-infra05:30
*** evrardjp has quit IRC05:33
*** evrardjp has joined #openstack-infra05:33
*** jamesmcarthur has joined #openstack-infra05:33
*** eharney has quit IRC05:49
*** vishalmanchanda has joined #openstack-infra05:51
*** jamesmcarthur has quit IRC05:53
*** eharney has joined #openstack-infra06:01
*** ysandeep|ruck is now known as ysandeep|afk06:33
*** rcernin has quit IRC06:39
*** lmiccini has joined #openstack-infra06:41
*** dmsimard has quit IRC06:48
*** jamesmcarthur has joined #openstack-infra06:58
*** jamesmcarthur has quit IRC07:11
*** eolivare has joined #openstack-infra07:24
*** lpetrut has joined #openstack-infra07:27
*** ralonsoh has joined #openstack-infra07:39
*** jcapitao has joined #openstack-infra07:58
*** zxiiro has quit IRC07:58
*** slaweq has joined #openstack-infra08:00
*** ysandeep|afk is now known as ysandeep08:02
*** ysandeep is now known as ysandeep|ruck08:06
*** dychen has quit IRC08:12
*** hberaud has joined #openstack-infra08:13
*** rpittau|afk is now known as rpittau08:15
*** yamamoto has quit IRC08:19
*** dhill has quit IRC08:24
*** rcernin has joined #openstack-infra08:27
*** dhill has joined #openstack-infra08:28
*** rcernin has quit IRC08:45
*** tosky has joined #openstack-infra08:50
*** yamamoto has joined #openstack-infra08:57
*** jpena|off is now known as jpena08:58
*** ociuhandu has joined #openstack-infra09:00
*** rcernin has joined #openstack-infra09:02
*** sshnaidm|rover is now known as sshnaidm|roff09:06
*** sshnaidm|roff is now known as sshnaidm|off09:06
*** dtantsur|afk is now known as dtantsur09:08
*** yamamoto has quit IRC09:10
*** ysandeep|ruck is now known as ysandeep|lunch09:11
*** lucasagomes has joined #openstack-infra09:14
*** rcernin has quit IRC09:17
*** gfidente has joined #openstack-infra09:28
*** yamamoto has joined #openstack-infra09:29
*** derekh has joined #openstack-infra09:30
*** yamamoto has quit IRC09:48
*** yamamoto has joined #openstack-infra09:52
*** yamamoto has quit IRC09:57
*** yamamoto has joined #openstack-infra09:59
*** yamamoto has quit IRC10:02
*** yamamoto has joined #openstack-infra10:05
*** dtantsur is now known as dtantsur|afk10:06
*** dtantsur|afk is now known as dtantsur|brb10:06
*** jamesmcarthur has joined #openstack-infra10:11
*** hashar has joined #openstack-infra10:16
*** jamesmcarthur has quit IRC10:16
*** ysandeep|lunch is now known as ysandeep|ruck10:27
*** Ng has quit IRC10:50
*** Ng has joined #openstack-infra10:50
*** iurygregory has quit IRC10:50
*** iurygregory has joined #openstack-infra10:51
*** ykarel has quit IRC10:52
*** ykarel has joined #openstack-infra10:53
*** yamamoto has quit IRC10:59
*** dtantsur|brb is now known as dtantsur11:01
*** yamamoto has joined #openstack-infra11:04
*** ldenny has quit IRC11:10
*** dmsimard has joined #openstack-infra11:11
*** zzzeek has quit IRC11:15
*** zzzeek has joined #openstack-infra11:16
*** hberaud has quit IRC11:25
*** ysandeep|ruck is now known as ysandeep|brb11:32
*** ociuhandu_ has joined #openstack-infra11:47
*** hberaud has joined #openstack-infra11:48
*** ysandeep|brb is now known as ysandeep|ruck11:49
*** ociuhandu has quit IRC11:50
*** jcapitao is now known as jcapitao_lunch11:51
*** ociuhandu_ has quit IRC11:57
*** ociuhandu has joined #openstack-infra11:58
*** rledisez has quit IRC12:00
*** rledisez has joined #openstack-infra12:04
*** rlandy has joined #openstack-infra12:07
*** hashar is now known as hasharLunch12:12
*** thiagop has joined #openstack-infra12:15
*** jpena is now known as jpena|lunch12:34
dtantsurhi folks! anything wrong with log servers? I see some number of jobs failing with RETRY_LIMIT and no logs.12:36
*** rfolco has joined #openstack-infra12:41
*** jamesmcarthur has joined #openstack-infra12:46
*** jcapitao_lunch is now known as jcapitao13:04
fungii'll see if i can spot one. we did just turn inap-mtl01 back on at 23:20 so it's possible there are still some rogue instances getting into an arp fight there or something along those lines13:10
*** derekh has quit IRC13:13
fungithe first two i've checked ran at least their final try in inap-mtl0113:15
fungiWARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!13:16
fungiyeah, this looks like a classic arp fight13:16
fungiData could not be sent to remote host \"198.72.124.27\". Make sure this host can be reached over ssh: ssh: connect to host 198.72.124.27 port 22: Connection timed out13:19
fungithat was another one in inap-mtl0113:19
*** Lewis has joined #openstack-infra13:23
*** Lewis is now known as Guest9879013:24
*** jpena|lunch is now known as jpena13:30
*** thiagop has quit IRC13:33
*** hasharLunch is now known as hashar13:33
*** yamamoto has quit IRC13:33
*** thiagop has joined #openstack-infra13:37
openstackgerritThierry Carrez proposed openstack/ptgbot master: Fix bot crash when 'none' URL provided  https://review.opendev.org/76055513:37
openstackgerritThierry Carrez proposed openstack/ptgbot master: Fix potential crash in etherpad command code  https://review.opendev.org/76055713:44
*** bdodd has quit IRC13:45
*** jamesdenton has joined #openstack-infra13:46
openstackgerritJeremy Stanley proposed openstack/project-config master: Revert "Revert "Temporarily stop booting nodes in inap-mtl01""  https://review.opendev.org/76055813:48
*** jamesmcarthur has quit IRC13:49
*** david-lyle has joined #openstack-infra13:53
*** zxiiro has joined #openstack-infra13:53
openstackgerritMerged openstack/project-config master: Revert "Revert "Temporarily stop booting nodes in inap-mtl01""  https://review.opendev.org/76055814:14
*** d34dh0r53 has quit IRC14:28
*** ociuhandu_ has joined #openstack-infra14:30
*** ociuhandu has quit IRC14:33
*** dave-mccowan has quit IRC14:33
*** dave-mccowan has joined #openstack-infra14:37
*** d34dh0r53 has joined #openstack-infra14:38
*** yamamoto has joined #openstack-infra14:40
rpittauhi all! We've noticed a weird thing when running some ipv6 jobs on focal, suddenly the loop at https://opendev.org/openstack/devstack/src/branch/master/lib/neutron_plugins/services/l3#L404 is not working anymore, causing a failure of the entire job right after because of paste.openstack.org/show/799534/14:45
rpittauthe interface is not recognized therefore not accepting router advertisements anymore14:46
rpittauan example of failure in focal -> https://ef77a3b273cb529607b4-a219f47d4102f914b241b8f5223ad891.ssl.cf5.rackcdn.com/759245/1/check/ironic-tempest-ipxe-ipv6/652a80c/job-output.txt14:46
rpittaua success in bionic -> https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_d59/760119/1/check/ironic-tempest-ipxe-ipv6/d59a1c9/job-output.txt14:47
clarkbrpittau: does https://ef77a3b273cb529607b4-a219f47d4102f914b241b8f5223ad891.ssl.cf5.rackcdn.com/759245/1/check/ironic-tempest-ipxe-ipv6/652a80c/controller/logs/worlddump-latest.txt help at all?14:48
clarkbI expect that to a linux difference not something we control14:48
*** ysandeep|ruck is now known as ysandeep|brb14:49
clarkbrpittau: note the line of code you mention as failing is different than the line of code in your log paste14:50
clarkbare we sure that ens3 is no longer accepting RAs?14:50
rpittauclarkb: I actually noticed a difference on the routing tables between bionic and focal in the logs14:50
rpittauclarkb: I know the line is different14:51
rpittauone is the cause, the other the ffect14:51
rpittaubut actually also the line tha tI pointed is failing acsue it's empty14:51
clarkbrpittau: right can you explain that beacuse looking at the world dump ens3 is still getting RAs (it has a valid global ipv6 address)14:51
rpittauit doesn't execute the loop14:51
rpittauclarkb: if you check the ip -6 route command, you'll see that there's no default route14:52
rpittautherefore the for loop is not doing anything14:52
clarkbok but not running there doesn't necessarily mean that is the problem. I guess I'm looking for a stronger connection to better undersatnd what is going on particularly since worlddump indicates that ipv6 is still working on ens314:52
rpittauso no RA14:52
*** yamamoto has quit IRC14:52
fungior could something be clearing the route?14:52
rpittaumaybe14:52
clarkbrpittau: there is a default route, but it routes via the link scope addr not the global addr so ya something is amiss there14:53
fungii would expect those routes to have a long lifetime, much longer than these nodes stick around anyway14:53
rpittauok, so I talked in the neutron channel already but I wanted to check with infra as the script mentions explicitly Infra :)14:53
clarkbrpittau: my memory for that code is that the problem was neutron would explicitly disable RAs all all interfaces beacuse neutron wants to be the router14:54
fungihmm, i'll have to look at the example but just to be clear, a default route with a linklocal next hop is entirely valid global routing for ipv6 (even recommended as it avoids wasting global addresses on router serials)14:54
clarkbrpittau: this is a problem if you are hosting neutron on test nodes that need to accept RAs to function14:54
clarkbfungi: rpittau default via fe80::f816:3eff:fe77:b05c dev ens3 proto ra metric 1024 expires 280sec hoplimit 64 pref medium14:54
clarkbthat is the route from the world dump14:54
fungithat looks fine to me as long as fe80::f816:3eff:fe77:b05c is reachable on ens314:54
fungiwould need to look at the neighbor table to confirm14:55
clarkbyes, I'm still not sure I'm convinced that br-ex route mangling is failing due to the linked code14:55
fungithe fact that fe80::f816:3eff:fe77:b05c is linklocal shouldn't present a problem as long as that's a real serial on the next hop router14:56
fungithough if you have a test which is trying to ping it or something, you'll definitely need to interface-scope the address like fe80::f816:3eff:fe77:b05c%ens314:57
fungibut that's still quite standard for a v6 global routing topology14:57
clarkbrpittau: basically that code is ensuring we don't stop accepting RAs from the hosting cloud. Its possible that is currently broken for some reason and we're ok because the lifetime on the RA is long enough that (most) jobs don't notice. However, at the same time I'm not sure that your example correlates to not accepting RAs since we seem to have ipv6 configured in the worlddump on ens314:58
fungihere's what my workstation's default route looks like, for comparison (it also has a global v6 address on the interface): default via fe80::20d:b9ff:fe37:18c5 dev enp3s0 proto ra metric 1024 expires 1769sec pref medium14:58
clarkbrpittau: unless maybe we need to explicitly allow RAs on br-ex and that was happening in that loop previously but not anymore?14:59
rpittauclarkb: ok, then I'm back to square 1 :)14:59
rpittauthe only difference that I've noticed is that it was working fine on bionic but not anymore on focal14:59
fungithe "proto ra" on both of those suggests it received the route via an ra at some point14:59
rpittauand then I saw that the RA accepting loop doesn't report results anymore15:00
rpittaubut yeah, I can see that in one case it is actually working15:01
rpittauI mean, the loop15:01
rpittaustill no route to host in the end15:01
rpittauso it seems not related15:01
rpittauok, I have no clue at this point15:02
clarkbI would probably look at linux bridge and ovs differences15:02
clarkbin most jobs that is an ovs bridge right?15:02
rpittauyes15:02
rpittauso the only clue is the distro difference15:02
rpittauand well, the kernel15:02
*** jamesmcarthur has joined #openstack-infra15:03
clarkbbr-ex: <BROADCAST,MULTICAST> mtu 1400 qdisc noop state DOWN group default qlen 1000 notice the interface is DOWN is that perhaps related?15:03
clarkbcan you not set a default route on a DOWN interface?15:03
clarkber I guess that isn't a new default route15:03
clarkbjust a normal route15:03
rpittaunot sure there is worlddump in the bionic jobs, or it's in a different location ?15:04
clarkbit only runs on failure15:04
rpittauah ok15:04
fungiyou could get one by forcing a failure in a dnm change, as an easy way15:10
rpittaufungi: yep, what I was thinking to do, thanks!15:10
*** derekh has joined #openstack-infra15:16
*** david-lyle has quit IRC15:19
*** jamesmcarthur has quit IRC15:19
*** dklyle has joined #openstack-infra15:19
*** jamesmcarthur has joined #openstack-infra15:19
*** portdirect has quit IRC15:20
*** portdirect has joined #openstack-infra15:20
*** ysandeep|brb is now known as ysandeep15:21
*** ysandeep is now known as ysandeep|away15:22
*** ociuhandu_ has quit IRC15:22
*** ociuhandu has joined #openstack-infra15:23
openstackgerritMerged openstack/project-config master: Replace old Victoria cycle signing key with Wallaby  https://review.opendev.org/76036415:23
*** icey has quit IRC15:27
*** icey has joined #openstack-infra15:27
*** thedaveking1 has joined #openstack-infra15:31
*** thedaveking has quit IRC15:32
*** thedaveking1 has quit IRC15:36
*** jamesmcarthur has quit IRC15:37
*** vishalmanchanda has quit IRC15:40
*** hamalq has joined #openstack-infra15:42
*** bdodd has joined #openstack-infra15:43
*** rpittau is now known as rpittau|afk15:54
*** jamesmcarthur has joined #openstack-infra15:55
*** jamesmcarthur has quit IRC15:56
*** jamesmcarthur has joined #openstack-infra15:57
*** ykarel is now known as ykarel|away15:58
*** tdasilva has quit IRC16:20
*** tdasilva has joined #openstack-infra16:20
*** hamalq has quit IRC16:21
*** hamalq has joined #openstack-infra16:22
*** ykarel|away has quit IRC16:22
*** tdasilva has quit IRC16:23
*** tdasilva_ has joined #openstack-infra16:23
*** thiago__ has joined #openstack-infra16:26
*** tdasilva_ has quit IRC16:29
*** lpetrut has quit IRC16:31
*** jcapitao has quit IRC16:42
*** jamesmcarthur has quit IRC16:43
*** eolivare has quit IRC16:47
*** jamesmcarthur has joined #openstack-infra16:50
*** lucasagomes has quit IRC16:51
*** yamamoto has joined #openstack-infra16:52
*** yamamoto has quit IRC16:56
*** thedaveking has joined #openstack-infra16:57
*** jamesmcarthur has quit IRC17:01
*** jamesmcarthur has joined #openstack-infra17:01
*** ociuhandu_ has joined #openstack-infra17:03
*** jamesmcarthur has quit IRC17:05
*** jamesmcarthur has joined #openstack-infra17:05
*** ociuhandu has quit IRC17:06
*** ociuhandu_ has quit IRC17:08
*** ociuhandu has joined #openstack-infra17:11
*** ociuhandu has quit IRC17:16
*** hberaud has quit IRC17:18
*** thedaveking has quit IRC17:30
*** thedaveking has joined #openstack-infra17:46
*** ralonsoh has quit IRC17:55
*** jpena is now known as jpena|off17:57
*** zxiiro has quit IRC17:58
*** lmiccini has quit IRC17:59
*** jamesmcarthur has quit IRC18:01
*** jamesmcarthur has joined #openstack-infra18:05
*** derekh has quit IRC18:05
*** jamesmcarthur has quit IRC18:13
*** dtantsur is now known as dtantsur|afk18:15
*** thedaveking has quit IRC18:18
*** psachin has quit IRC18:23
*** jamesmcarthur has joined #openstack-infra18:33
*** jamesmcarthur has quit IRC18:35
*** gfidente is now known as gfidente|afk18:38
*** hashar has quit IRC18:43
*** jamesmcarthur has joined #openstack-infra18:44
*** jamesmcarthur has quit IRC18:58
*** ociuhandu has joined #openstack-infra19:50
*** ociuhandu has quit IRC19:52
*** jamesmcarthur has joined #openstack-infra19:54
*** jamesmcarthur has quit IRC20:05
*** jamesmcarthur has joined #openstack-infra20:22
*** yolanda has quit IRC20:24
*** yolanda has joined #openstack-infra20:25
*** dklyle has quit IRC20:26
*** armax has joined #openstack-infra20:26
*** dklyle has joined #openstack-infra20:34
*** slaweq has quit IRC20:38
*** ade_lee has joined #openstack-infra20:50
ade_leehey all - quick question if anyone knows -- I'm trying to create a devstack job that enables fips on the system.  To do that, I have an ansible script that sets up fips and then has to reboot the node.20:53
ade_leehttps://review.opendev.org/#/c/760665/120:53
ade_leelooks like the node reboots but never comes back -- is there a something special I need to do to do this?20:54
clarkbyou have to restart the zuul console log process. There is a rolw for that20:57
clarkbotherwise it should come back for ssh just fine and ansible should run against it20:57
ade_leeclarkb, which role is that?20:59
*** gfidente|afk has quit IRC21:00
ade_leeclarkb, gotta head out - but I'll touch base early next week - it looks like it didn't come back for ssh though -- anyways the job is retrying21:02
ade_leehttps://zuul.opendev.org/t/openstack/stream/14545ce85c14403193dae317346ad128?logfile=console.log21:02
clarkbhttps://opendev.org/zuul/zuul-jobs/src/branch/master/roles/start-zuul-console that one I think21:02
clarkbdo you have a pasue?21:03
clarkbyou have to tell anaible the hostis going away too and to wait for it21:03
ade_leeclarkb, ah, cool21:03
ade_leeclarkb, yeah - I don't have a pause21:04
*** jamesdenton has quit IRC21:10
*** matt_kosut has quit IRC21:21
*** rlandy has quit IRC21:35
*** yamamoto has joined #openstack-infra22:00
*** jamesmcarthur has quit IRC22:01
*** jamesmcarthur has joined #openstack-infra22:02
*** jamesmcarthur has quit IRC22:10
*** ociuhandu has joined #openstack-infra22:15
*** jamesmcarthur has joined #openstack-infra22:16
*** hberaud has joined #openstack-infra22:18
*** ociuhandu has quit IRC22:20
*** rfolco is now known as rfolco|bbl22:21
*** dave-mccowan has quit IRC22:26
*** yamamoto has quit IRC22:32
*** jamesmcarthur has quit IRC22:33
*** jamesmcarthur has joined #openstack-infra22:33
*** jamesmcarthur has quit IRC22:41
*** jamesmcarthur has joined #openstack-infra22:42
*** jamesmcarthur has quit IRC22:46
*** jamesmcarthur has joined #openstack-infra22:46
*** armax has quit IRC23:06
*** eharney has quit IRC23:11
*** hberaud has quit IRC23:14
*** jamesmcarthur has quit IRC23:22
*** TheJulia is now known as TheJulia_on_PTO23:24
*** yolanda has quit IRC23:33
*** jamesdenton has joined #openstack-infra23:39
*** jamesdenton has quit IRC23:40
*** hashar has joined #openstack-infra23:45
*** hashar has quit IRC23:45
*** hashar has joined #openstack-infra23:48

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!