| *** slaweq has quit IRC | 00:01 | |
| *** jamesmcarthur has joined #openstack-infra | 00:14 | |
| *** markvoelker has joined #openstack-infra | 00:19 | |
| *** jamesmcarthur has quit IRC | 00:21 | |
| *** jamesmcarthur has joined #openstack-infra | 00:26 | |
| *** markvoelker has quit IRC | 00:30 | |
| *** diablo_rojo has joined #openstack-infra | 00:41 | |
| *** goldyfruit has joined #openstack-infra | 00:43 | |
| *** hongbin has joined #openstack-infra | 00:52 | |
| *** jamesmcarthur has quit IRC | 00:53 | |
| *** jamesmcarthur has joined #openstack-infra | 00:54 | |
| *** dychen has quit IRC | 00:58 | |
| *** jamesmcarthur has quit IRC | 01:01 | |
| *** hwoarang has quit IRC | 01:03 | |
| *** slaweq has joined #openstack-infra | 01:11 | |
| *** yamamoto has quit IRC | 01:12 | |
| *** hwoarang has joined #openstack-infra | 01:12 | |
| *** jamesmcarthur has joined #openstack-infra | 01:15 | |
| *** slaweq has quit IRC | 01:15 | |
| *** yamamoto has joined #openstack-infra | 01:17 | |
| *** jamesmcarthur has quit IRC | 01:28 | |
| *** markvoelker has joined #openstack-infra | 01:33 | |
| *** michael-beaver has quit IRC | 01:46 | |
| *** jamesmcarthur has joined #openstack-infra | 02:01 | |
| *** apetrich has quit IRC | 02:09 | |
| *** threestrands has joined #openstack-infra | 02:13 | |
| *** jamesmcarthur has quit IRC | 02:30 | |
| *** jamesmcarthur_ has joined #openstack-infra | 02:30 | |
| *** roman_g has quit IRC | 02:33 | |
| *** markvoelker has quit IRC | 02:40 | |
| *** jamesmcarthur_ has quit IRC | 02:45 | |
| *** ricolin has joined #openstack-infra | 02:52 | |
| *** ramishra has joined #openstack-infra | 02:53 | |
| *** larainema has joined #openstack-infra | 02:53 | |
| *** jamesmcarthur has joined #openstack-infra | 02:53 | |
| *** xinranwang has joined #openstack-infra | 02:57 | |
| *** slaweq has joined #openstack-infra | 03:11 | |
| *** rh-jelabarre has joined #openstack-infra | 03:11 | |
| *** yamamoto has quit IRC | 03:14 | |
| *** yamamoto has joined #openstack-infra | 03:15 | |
| *** slaweq has quit IRC | 03:15 | |
| *** diablo_rojo has quit IRC | 03:18 | |
| *** diablo_rojo has joined #openstack-infra | 03:19 | |
| *** yamamoto has quit IRC | 03:20 | |
| *** hongbin has quit IRC | 03:23 | |
| *** dchen has quit IRC | 03:24 | |
| *** dchen has joined #openstack-infra | 03:24 | |
| *** psachin has joined #openstack-infra | 03:34 | |
| *** goldyfruit has quit IRC | 03:48 | |
| *** hongbin has joined #openstack-infra | 03:49 | |
| *** udesale has joined #openstack-infra | 04:00 | |
| *** ramishra has quit IRC | 04:00 | |
| *** whoami-rajat has joined #openstack-infra | 04:02 | |
| *** jamesmcarthur has quit IRC | 04:04 | |
| *** slaweq has joined #openstack-infra | 04:11 | |
| *** ykarel has joined #openstack-infra | 04:13 | |
| *** rh-jelabarre has quit IRC | 04:14 | |
| *** slaweq has quit IRC | 04:16 | |
| *** ramishra has joined #openstack-infra | 04:17 | |
| *** yamamoto has joined #openstack-infra | 04:21 | |
| *** ociuhandu has joined #openstack-infra | 04:30 | |
| *** ociuhandu has quit IRC | 04:35 | |
| *** kjackal has joined #openstack-infra | 04:37 | |
| *** hwoarang has quit IRC | 04:37 | |
| *** hwoarang has joined #openstack-infra | 04:38 | |
| *** hongbin has quit IRC | 04:41 | |
| *** markvoelker has joined #openstack-infra | 04:41 | |
| *** markvoelker has quit IRC | 04:46 | |
| *** xenos76 has joined #openstack-infra | 04:48 | |
| *** dave-mccowan has quit IRC | 04:51 | |
| *** rakhmerov has joined #openstack-infra | 04:52 | |
| *** xenos76 has quit IRC | 05:00 | |
| *** pcaruana has joined #openstack-infra | 05:07 | |
| *** xenos76 has joined #openstack-infra | 05:09 | |
| *** slaweq has joined #openstack-infra | 05:11 | |
| *** slaweq has quit IRC | 05:15 | |
| *** yamamoto has quit IRC | 05:16 | |
| *** odicha has joined #openstack-infra | 05:16 | |
| ianw | tristanC / clarkb / donnyd : dropped a comment in https://review.opendev.org/#/c/686749/11 but after testing today, I'm fairly convinced this is a NM issue where it times out waiting for a permanent link-local address and then gives up trying to configure ipv6 | 05:16 |
|---|---|---|
| ianw | I have filed : https://bugzilla.redhat.com/show_bug.cgi?id=1760179 | 05:16 |
| openstack | bugzilla.redhat.com bug 1760179 in NetworkManager "IPv6 address never assigned, possibly "linklocal6: waiting for link-local addresses failed due to timeout"" [Unspecified,New] - Assigned to lkundrak | 05:16 |
| ianw | i'm out for today, but I think that we might have luck making glean just wait a bit to make sure DAD has happened and the link-local address is permanent before starting networkmanager | 05:17 |
| *** yamamoto has joined #openstack-infra | 05:17 | |
| openstackgerrit | Simon Westphahl proposed zuul/nodepool master: Sort waiting static nodes by creation time https://review.opendev.org/687271 | 05:19 |
| openstackgerrit | Merged zuul/zuul master: Include session expired reason in API fetch error message. https://review.opendev.org/686976 | 05:24 |
| openstackgerrit | Merged zuul/zuul master: Ensure tenant web_root url has a trailing slash https://review.opendev.org/676826 | 05:28 |
| *** pcaruana has quit IRC | 05:35 | |
| *** kjackal has quit IRC | 05:36 | |
| *** jtomasek has quit IRC | 05:50 | |
| *** jaosorior has joined #openstack-infra | 06:04 | |
| *** roman_g has joined #openstack-infra | 06:10 | |
| *** pgaxatte has joined #openstack-infra | 06:20 | |
| *** surpatil has joined #openstack-infra | 06:21 | |
| *** yamamoto has quit IRC | 06:22 | |
| *** yamamoto has joined #openstack-infra | 06:25 | |
| *** kjackal has joined #openstack-infra | 06:34 | |
| *** threestrands has quit IRC | 06:36 | |
| *** threestrands has joined #openstack-infra | 06:36 | |
| *** iurygregory has joined #openstack-infra | 06:38 | |
| *** threestrands has quit IRC | 06:41 | |
| *** hwoarang has quit IRC | 06:49 | |
| *** pcaruana has joined #openstack-infra | 06:51 | |
| *** yamamoto has quit IRC | 06:53 | |
| openstackgerrit | Simon Westphahl proposed zuul/zuul master: Spec for allowing circular dependencies https://review.opendev.org/643309 | 06:54 |
| openstackgerrit | Simon Westphahl proposed zuul/zuul master: Add optional support for circular dependencies https://review.opendev.org/685354 | 06:55 |
| *** jaosorior has quit IRC | 06:57 | |
| *** yamamoto has joined #openstack-infra | 06:58 | |
| *** zhangfei has joined #openstack-infra | 06:58 | |
| *** tesseract has joined #openstack-infra | 06:59 | |
| *** hwoarang has joined #openstack-infra | 07:01 | |
| *** kopecmartin|off is now known as kopecmartin | 07:02 | |
| *** slaweq has joined #openstack-infra | 07:02 | |
| *** gfidente has joined #openstack-infra | 07:02 | |
| *** ykarel is now known as ykarel|lunch | 07:04 | |
| *** xinranwang has quit IRC | 07:07 | |
| *** rcernin has quit IRC | 07:07 | |
| *** ccamacho has joined #openstack-infra | 07:08 | |
| *** ccamacho has quit IRC | 07:09 | |
| *** ccamacho has joined #openstack-infra | 07:09 | |
| *** ricolin has quit IRC | 07:10 | |
| *** tosky has joined #openstack-infra | 07:12 | |
| *** jpena|off is now known as jpena | 07:13 | |
| openstackgerrit | Simon Westphahl proposed zuul/nodepool master: Sort waiting static nodes by creation time https://review.opendev.org/687271 | 07:19 |
| openstackgerrit | Simon Westphahl proposed zuul/nodepool master: Don't touch static nodes that are allocated https://review.opendev.org/687806 | 07:19 |
| *** pkopec has joined #openstack-infra | 07:23 | |
| *** FlorianFa has quit IRC | 07:24 | |
| *** Florian has joined #openstack-infra | 07:25 | |
| *** yamamoto has quit IRC | 07:25 | |
| *** eernst has joined #openstack-infra | 07:28 | |
| *** yamamoto has joined #openstack-infra | 07:33 | |
| *** zbr has joined #openstack-infra | 07:33 | |
| *** elod has quit IRC | 07:36 | |
| *** apetrich has joined #openstack-infra | 07:43 | |
| *** elod has joined #openstack-infra | 07:44 | |
| *** zbr has quit IRC | 07:44 | |
| *** eernst has quit IRC | 07:56 | |
| *** trident has quit IRC | 07:58 | |
| *** trident has joined #openstack-infra | 08:01 | |
| *** zbr has joined #openstack-infra | 08:02 | |
| *** ralonsoh has joined #openstack-infra | 08:02 | |
| *** ociuhandu has joined #openstack-infra | 08:03 | |
| openstackgerrit | Simon Westphahl proposed zuul/nodepool master: Don't touch static nodes that are allocated https://review.opendev.org/687806 | 08:03 |
| openstackgerrit | Simon Westphahl proposed zuul/nodepool master: Sort waiting static nodes by creation time https://review.opendev.org/687271 | 08:03 |
| *** lucasagomes has joined #openstack-infra | 08:04 | |
| *** ociuhandu has quit IRC | 08:08 | |
| *** tkajinam has quit IRC | 08:10 | |
| *** kjackal_v2 has joined #openstack-infra | 08:10 | |
| *** kjackal has quit IRC | 08:11 | |
| *** arxcruz|rover is now known as arxcruz | 08:11 | |
| *** rpittau|afk is now known as rpittau | 08:13 | |
| frickler | prometheanfire: can you take a look at https://review.opendev.org/682635 please? gentoo dib tests are failing for some time now | 08:25 |
| frickler | see also https://review.opendev.org/682639 | 08:26 |
| *** yamamoto has quit IRC | 08:29 | |
| *** derekh has joined #openstack-infra | 08:31 | |
| *** yamamoto has joined #openstack-infra | 08:31 | |
| *** jtomasek has joined #openstack-infra | 08:33 | |
| *** dchen has quit IRC | 08:36 | |
| *** markvoelker has joined #openstack-infra | 08:44 | |
| *** yamamoto has quit IRC | 08:48 | |
| *** diablo_rojo has quit IRC | 08:49 | |
| *** markvoelker has quit IRC | 08:50 | |
| *** lennyb has quit IRC | 08:57 | |
| *** lennyb has joined #openstack-infra | 08:58 | |
| *** dtantsur|afk is now known as dtantsur | 09:01 | |
| *** diablo_rojo has joined #openstack-infra | 09:05 | |
| *** e0ne has joined #openstack-infra | 09:06 | |
| *** Florian has quit IRC | 09:19 | |
| *** FlorianFa has joined #openstack-infra | 09:19 | |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: Zuul Web: add /api/user/authorizations endpoint https://review.opendev.org/641099 | 09:23 |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: Reduce sleep to avoid race conditions https://review.opendev.org/684726 | 09:24 |
| *** ykarel|lunch is now known as ykarel | 09:26 | |
| *** yamamoto has joined #openstack-infra | 09:29 | |
| *** ricolin has joined #openstack-infra | 09:30 | |
| *** gfidente has quit IRC | 09:33 | |
| *** gfidente has joined #openstack-infra | 09:42 | |
| *** diablo_rojo has quit IRC | 09:47 | |
| *** udesale has quit IRC | 09:52 | |
| *** udesale has joined #openstack-infra | 09:53 | |
| *** ykarel is now known as ykarel|afk | 09:54 | |
| *** udesale has quit IRC | 10:00 | |
| *** udesale has joined #openstack-infra | 10:01 | |
| *** ociuhandu has joined #openstack-infra | 10:03 | |
| *** ociuhandu has quit IRC | 10:19 | |
| *** ociuhandu has joined #openstack-infra | 10:28 | |
| openstackgerrit | Thierry Carrez proposed opendev/puppet-ptgbot master: Deploy the etherpads.html file https://review.opendev.org/687850 | 10:31 |
| openstackgerrit | Thierry Carrez proposed opendev/puppet-ptgbot master: Deploy glyphicon font files https://review.opendev.org/687851 | 10:31 |
| *** xek_ has joined #openstack-infra | 10:34 | |
| openstackgerrit | Merged opendev/irc-meetings master: Update StoryBoard meeting day/time https://review.opendev.org/687644 | 10:43 |
| *** zhangfei has quit IRC | 10:51 | |
| *** slaweq_ has joined #openstack-infra | 10:57 | |
| *** ociuhandu has quit IRC | 10:59 | |
| *** slaweq has quit IRC | 10:59 | |
| openstackgerrit | Merged zuul/nodepool master: Add port-cleanup-interval config option https://review.opendev.org/687024 | 11:00 |
| *** jpena is now known as jpena|lunch | 11:04 | |
| *** udesale has quit IRC | 11:09 | |
| *** yamamoto has quit IRC | 11:24 | |
| *** jhesketh has quit IRC | 11:29 | |
| *** ykarel|afk is now known as ykarel | 11:32 | |
| *** larainema has quit IRC | 11:34 | |
| *** ociuhandu has joined #openstack-infra | 11:37 | |
| *** ociuhandu has quit IRC | 11:42 | |
| *** jhesketh has joined #openstack-infra | 11:43 | |
| *** weshay|ruck is now known as weshay | 11:48 | |
| *** ociuhandu has joined #openstack-infra | 11:48 | |
| *** yamamoto has joined #openstack-infra | 11:50 | |
| *** jpena|lunch is now known as jpena | 11:58 | |
| *** rh-jelabarre has joined #openstack-infra | 12:02 | |
| *** goldyfruit has joined #openstack-infra | 12:03 | |
| *** rfolco has joined #openstack-infra | 12:05 | |
| *** rfolco is now known as rfolco|ruck | 12:07 | |
| openstackgerrit | Simon Westphahl proposed zuul/nodepool master: Don't touch static nodes that are allocated https://review.opendev.org/687806 | 12:10 |
| openstackgerrit | Simon Westphahl proposed zuul/nodepool master: Sort waiting static nodes by creation time https://review.opendev.org/687271 | 12:10 |
| *** AJaeger has quit IRC | 12:12 | |
| *** goldyfruit has quit IRC | 12:14 | |
| *** rlandy has joined #openstack-infra | 12:15 | |
| *** AJaeger has joined #openstack-infra | 12:17 | |
| *** markvoelker has joined #openstack-infra | 12:21 | |
| *** Goneri has joined #openstack-infra | 12:24 | |
| *** derekh has quit IRC | 12:24 | |
| *** tmorin has joined #openstack-infra | 12:28 | |
| tmorin | hi folks (infra-root) | 12:32 |
| tmorin | I have a change that's +1+W but not being merged (stuck in "ready to submit" state, the one change it depends on has been merged months ago) | 12:32 |
| tmorin | even after rechecks (quite a few) and trying to go again through +1+W from the initial state... | 12:32 |
| tmorin | I'm hoping someone could perhaps check what is happening ? https://review.opendev.org/#/c/636422 | 12:33 |
| tmorin | ^^ slaweq_ | 12:34 |
| *** jaosorior has joined #openstack-infra | 12:34 | |
| *** tmorin has quit IRC | 12:34 | |
| *** tmorin has joined #openstack-infra | 12:34 | |
| frickler | tmorin: I think you need to rebase that change, it is on top of https://review.opendev.org/#/c/636962/1 while PS3 of that has merged | 12:36 |
| slaweq_ | tmorin: I just rebased https://review.opendev.org/#/c/636422 | 12:45 |
| tmorin | thanks frickler | 12:45 |
| tmorin | thanks slaweq_ , I just saw that | 12:45 |
| tmorin | frickler: aren't there cases (most ?) where gerrit can be smart enough to rebase on its own ? | 12:46 |
| *** mriedem has joined #openstack-infra | 12:50 | |
| *** markvoelker has quit IRC | 12:51 | |
| openstackgerrit | Jeremy Stanley proposed opendev/puppet-openstack_infra_spec_helper master: Block minitest 5.12.1 https://review.opendev.org/687884 | 12:51 |
| fungi | tmorin: not when the explicit parent of the change is an outdated patchset | 12:52 |
| fungi | because that parent will never appear in the git history | 12:53 |
| fungi | gerrit doesn't rebase changes, it only merges them | 12:53 |
| fungi | and it can't merge a change which has a parent that isn't in the repository | 12:53 |
| fungi | or at least isn't in that branch | 12:53 |
| *** udesale has joined #openstack-infra | 12:55 | |
| *** aaronsheffield has joined #openstack-infra | 12:56 | |
| *** dtantsur is now known as dtantsur|afk | 12:57 | |
| *** ihti has quit IRC | 12:58 | |
| *** anteaya has quit IRC | 13:00 | |
| *** ihti has joined #openstack-infra | 13:01 | |
| openstackgerrit | Jeremy Stanley proposed opendev/puppet-ptgbot master: Deploy the etherpads.html file https://review.opendev.org/687850 | 13:05 |
| openstackgerrit | Jeremy Stanley proposed opendev/puppet-ptgbot master: Deploy glyphicon font files https://review.opendev.org/687851 | 13:06 |
| *** dpawlik has joined #openstack-infra | 13:13 | |
| *** priteau has joined #openstack-infra | 13:13 | |
| *** trident has quit IRC | 13:14 | |
| *** trident has joined #openstack-infra | 13:15 | |
| *** psachin has quit IRC | 13:15 | |
| *** michael-beaver has joined #openstack-infra | 13:18 | |
| *** dpawlik has quit IRC | 13:18 | |
| tmorin | thanks for the explanation frickler, fungi! | 13:22 |
| *** dpawlik has joined #openstack-infra | 13:22 | |
| *** goldyfruit has joined #openstack-infra | 13:23 | |
| openstackgerrit | Monty Taylor proposed zuul/zuul-registry master: HEAD object after PUT https://review.opendev.org/687681 | 13:30 |
| fungi | infra-puppet-core: can i get an expedited approval on a gem pin in https://review.opendev.org/687884 to fix our centos-7 puppet jobs? | 13:31 |
| pabelanger | +2 | 13:31 |
| fungi | job results on latest patchsets of 687850 and 687851 show it's working | 13:32 |
| *** david-lyle is now known as dklyle | 13:33 | |
| *** goldyfruit has quit IRC | 13:35 | |
| *** goldyfruit has joined #openstack-infra | 13:37 | |
| openstackgerrit | Monty Taylor proposed zuul/zuul-registry master: HEAD object after PUT https://review.opendev.org/687681 | 13:40 |
| fungi | thanks pabelanger! | 13:43 |
| fungi | i went ahead and self-approved so we don't block puppet module changes | 13:43 |
| *** tmorin has left #openstack-infra | 13:43 | |
| *** dave-mccowan has joined #openstack-infra | 13:43 | |
| *** eharney has joined #openstack-infra | 13:47 | |
| *** rkukura_ has joined #openstack-infra | 13:52 | |
| *** tosky has quit IRC | 13:55 | |
| *** rkukura has quit IRC | 13:55 | |
| *** rkukura_ is now known as rkukura | 13:55 | |
| *** yamamoto has quit IRC | 13:56 | |
| *** ccamacho has quit IRC | 13:56 | |
| *** ccamacho has joined #openstack-infra | 13:56 | |
| openstackgerrit | Merged opendev/puppet-openstack_infra_spec_helper master: Block minitest 5.12.1 https://review.opendev.org/687884 | 13:56 |
| *** spsurya has joined #openstack-infra | 13:57 | |
| *** diablo_rojo has joined #openstack-infra | 14:00 | |
| *** surpatil has quit IRC | 14:00 | |
| *** dklyle has quit IRC | 14:02 | |
| *** dklyle has joined #openstack-infra | 14:04 | |
| *** mriedem has quit IRC | 14:04 | |
| *** mriedem has joined #openstack-infra | 14:05 | |
| *** georgk has quit IRC | 14:06 | |
| *** fdegir has quit IRC | 14:06 | |
| openstackgerrit | Merged opendev/puppet-ptgbot master: Deploy the etherpads.html file https://review.opendev.org/687850 | 14:06 |
| openstackgerrit | Merged opendev/puppet-ptgbot master: Deploy glyphicon font files https://review.opendev.org/687851 | 14:06 |
| *** georgk has joined #openstack-infra | 14:07 | |
| *** fdegir has joined #openstack-infra | 14:07 | |
| *** odicha has quit IRC | 14:10 | |
| *** sreejithp has joined #openstack-infra | 14:13 | |
| *** ociuhandu has quit IRC | 14:14 | |
| *** markvoelker has joined #openstack-infra | 14:15 | |
| *** adriant has quit IRC | 14:29 | |
| *** iokiwi has quit IRC | 14:29 | |
| *** adriant has joined #openstack-infra | 14:31 | |
| *** iokiwi has joined #openstack-infra | 14:31 | |
| *** yamamoto has joined #openstack-infra | 14:36 | |
| *** jpena is now known as jpena|off | 14:39 | |
| *** pcaruana has quit IRC | 14:39 | |
| *** yamamoto has quit IRC | 14:41 | |
| *** dave-mccowan has quit IRC | 14:41 | |
| *** chandankumar is now known as raukadah | 14:43 | |
| *** pgaxatte has quit IRC | 14:43 | |
| *** xenos76 has quit IRC | 14:44 | |
| *** xenos76 has joined #openstack-infra | 14:45 | |
| *** ociuhandu has joined #openstack-infra | 14:48 | |
| *** jamesmcarthur has joined #openstack-infra | 14:52 | |
| *** ociuhandu has quit IRC | 14:53 | |
| *** yamamoto has joined #openstack-infra | 14:55 | |
| *** ociuhandu has joined #openstack-infra | 14:58 | |
| *** xenos76 has quit IRC | 15:00 | |
| openstackgerrit | Frode Nordahl proposed openstack/project-config master: Add OVN charms https://review.opendev.org/687925 | 15:00 |
| *** pcaruana has joined #openstack-infra | 15:01 | |
| *** xenos76 has joined #openstack-infra | 15:01 | |
| AJaeger | config-core, please review ianw's CentOS 8 stack starting at https://review.opendev.org/#/c/687445 | 15:06 |
| openstackgerrit | Sean McGinnis proposed openstack/project-config master: Add stable notifications to openstack-glance https://review.opendev.org/687931 | 15:10 |
| *** ykarel is now known as ykarel|afk | 15:11 | |
| *** ociuhandu has quit IRC | 15:12 | |
| openstackgerrit | Frode Nordahl proposed openstack/project-config master: Add OVN charms https://review.opendev.org/687925 | 15:17 |
| *** ociuhandu has joined #openstack-infra | 15:18 | |
| AJaeger | thanks, mnaser ! | 15:21 |
| mnaser | infra-root: i think it would be good if someone +2'd this and watched it -- https://review.opendev.org/#/c/687453/2 | 15:22 |
| mnaser | np AJaeger | 15:22 |
| *** gyee has joined #openstack-infra | 15:22 | |
| *** pcaruana has quit IRC | 15:26 | |
| fungi | approved, i'll set a reminder to check the image build log once that's deployed | 15:27 |
| AJaeger | thanks, fungi! I'm sure ianw will check as well once he's awake ;) | 15:29 |
| *** eernst has joined #openstack-infra | 15:29 | |
| *** zbr has quit IRC | 15:29 | |
| *** ociuhandu has quit IRC | 15:30 | |
| *** zbr has joined #openstack-infra | 15:31 | |
| openstackgerrit | Merged openstack/project-config master: infra-pkg-needs: Update pkg-maps for CentOS 8, select chronyd https://review.opendev.org/687445 | 15:32 |
| openstackgerrit | Merged openstack/project-config master: zuul-worker: no selinux python2 libs on CentOS 8 https://review.opendev.org/687446 | 15:32 |
| openstackgerrit | Merged openstack/project-config master: infra-package-needs: fix haveged install for all CentOS releases https://review.opendev.org/687447 | 15:32 |
| openstackgerrit | Merged openstack/project-config master: nodepool/elements : use abstracted commands https://review.opendev.org/686524 | 15:33 |
| openstackgerrit | Merged openstack/project-config master: Remove explicit set of DIB_SIMPLE_INIT_NETWORKMANAGER https://review.opendev.org/687452 | 15:33 |
| *** yamamoto has quit IRC | 15:34 | |
| *** slaweq_ is now known as slaweq | 15:34 | |
| *** ociuhandu has joined #openstack-infra | 15:35 | |
| prometheanfire | fungi: yep, looks good | 15:37 |
| openstackgerrit | Merged openstack/project-config master: CentOS 8 initial deployment https://review.opendev.org/687453 | 15:40 |
| corvus | fungi, mordred, clarkb: the gerrit maintainers would like us to take a lok at https://review.opendev.org/685533 | 15:41 |
| *** jtomasek has quit IRC | 15:41 | |
| *** ykarel|afk is now known as ykarel | 15:41 | |
| *** dave-mccowan has joined #openstack-infra | 15:42 | |
| clarkb | corvus: it is avalid feature in many gerrit installs, wouldnt itbe better to accept the flag and fail if the gerrit cant support it rather than remove it entirely? | 15:43 |
| *** lucasagomes has quit IRC | 15:45 | |
| *** ociuhandu has quit IRC | 15:45 | |
| fungi | seems like it's already basically deprecated in gerrit 2.15, so suggesting that folks who need to use that feature on an older gerrit deployment should avoid upgrading git-review could make sense | 15:46 |
| *** rpittau is now known as rpittau|afk | 15:47 | |
| *** ociuhandu has joined #openstack-infra | 15:48 | |
| mordred | I'm torn - I like supporting older things - but even in older gerrits it's a feature that doesn't exactly do what people think it does | 15:48 |
| clarkb | ya, but we disable it in our gerrit and return an error to git review | 15:49 |
| clarkb | we didnt rm it from git review | 15:49 |
| mordred | ya | 15:49 |
| *** Goneri has quit IRC | 15:51 | |
| *** kmalloc has left #openstack-infra | 15:52 | |
| corvus | other options: keeping it around until 3.1 is the oldest supported release? emitting a warning that it's deprecated and will be removed? | 15:53 |
| fungi | we don't seem to test it, so not even sure if that feature is actually working | 15:53 |
| fungi | at a minimum it deserves a release note, but sure a deprecation warning, and then removing at the following release would be gentler | 15:54 |
| *** roman_g has quit IRC | 15:54 | |
| *** jaosorior has quit IRC | 15:55 | |
| *** ociuhandu has quit IRC | 15:56 | |
| *** roman_g has joined #openstack-infra | 16:01 | |
| *** eernst has quit IRC | 16:05 | |
| *** vkmc has joined #openstack-infra | 16:06 | |
| *** yamamoto has joined #openstack-infra | 16:08 | |
| *** mriedem is now known as mriedem_lunch | 16:13 | |
| *** yamamoto has quit IRC | 16:15 | |
| *** udesale has quit IRC | 16:19 | |
| *** jpena|off is now known as jpena | 16:19 | |
| *** igordc has joined #openstack-infra | 16:20 | |
| *** Goneri has joined #openstack-infra | 16:22 | |
| *** dklyle has quit IRC | 16:30 | |
| *** david-lyle has joined #openstack-infra | 16:30 | |
| *** kopecmartin is now known as kopecmartin|off | 16:31 | |
| *** david-lyle is now known as dklyle | 16:31 | |
| *** ociuhandu has joined #openstack-infra | 16:33 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-registry master: Add docker buildset test https://review.opendev.org/687953 | 16:34 |
| *** Goneri has quit IRC | 16:34 | |
| *** ociuhandu has quit IRC | 16:38 | |
| *** ccamacho has quit IRC | 16:39 | |
| *** pcaruana has joined #openstack-infra | 16:40 | |
| *** dpawlik has quit IRC | 16:48 | |
| fungi | https://nb01.openstack.org/centos-8-0000000001.log | 16:53 |
| fungi | ianw: we have centos-8 images, it looks like | 16:53 |
| openstackgerrit | James E. Blair proposed zuul/zuul-registry master: Run docker and podman push/pull tests https://review.opendev.org/687692 | 16:54 |
| openstackgerrit | James E. Blair proposed zuul/zuul-registry master: Add docker buildset test https://review.opendev.org/687953 | 16:55 |
| pabelanger | fungi: ianw: Ooooh, nice! | 16:55 |
| fungi | still uploading in all providers, but i'll see if we get any nodes building once they populate | 16:55 |
| pabelanger | Yah, that would be cool. If works out of box, we'll totally at it to zuul.a.c to test too | 16:56 |
| *** e0ne has quit IRC | 16:56 | |
| *** jamesmcarthur has quit IRC | 17:03 | |
| *** jamesmcarthur_ has joined #openstack-infra | 17:04 | |
| *** gfidente has quit IRC | 17:07 | |
| *** rlandy is now known as rlandy|brb | 17:08 | |
| *** ociuhandu has joined #openstack-infra | 17:10 | |
| *** ykarel is now known as ykarel|away | 17:11 | |
| corvus | fungi, mordred, pabelanger: the example ansible facts in the documentation looks familiar: https://docs.ansible.com/ansible/latest/user_guide/playbooks_variables.html#variables-discovered-from-systems-facts | 17:15 |
| pabelanger | indeed | 17:15 |
| corvus | that's a great idea -- and they could have redacted way less info :) | 17:16 |
| corvus | go ahead and throw in those ssh host keys | 17:16 |
| clarkb | ha | 17:17 |
| corvus | anyway, i was going to go look up how to get the uid of the user zuul was running as, and i end up getting the actual value in the docs! that's some spot-on documentation | 17:17 |
| openstackgerrit | James E. Blair proposed zuul/zuul-registry master: Run docker and podman push/pull tests https://review.opendev.org/687692 | 17:21 |
| openstackgerrit | James E. Blair proposed zuul/zuul-registry master: Add docker buildset test https://review.opendev.org/687953 | 17:21 |
| openstackgerrit | Adam Coldrick proposed opendev/storyboard-webclient master: Adds Migration Docs to Dashboard https://review.opendev.org/680235 | 17:22 |
| *** priteau has quit IRC | 17:23 | |
| *** dpawlik has joined #openstack-infra | 17:25 | |
| *** Goneri has joined #openstack-infra | 17:27 | |
| fungi | centos-8 images have gone to a ready state in rax-dfw and rax-ord so far | 17:29 |
| *** dpawlik has quit IRC | 17:29 | |
| clarkb | do we expect them to have the same NM problems on FN and limestone? | 17:29 |
| clarkb | also any idea if further debugging was done there? | 17:29 |
| fungi | the first min-ready node is building in ord now | 17:30 |
| clarkb | I'm about to page all that back in and look at booting some upstream images for comparison | 17:30 |
| fungi | clarkb: not sure | 17:30 |
| fungi | clarkb: see overnight scrollback from ianw though, he opened an upstream bug i think | 17:30 |
| clarkb | ooh this is excellent reading | 17:31 |
| *** mriedem_lunch is now known as mriedem | 17:34 | |
| *** ykarel|away has quit IRC | 17:35 | |
| fungi | i think the centos-8 image isn't booting successfully in rax-ord | 17:37 |
| clarkb | I wonder if the solicitation delay affects things with that timeout in ianw's bug | 17:37 |
| clarkb | I'm going to build an image without that delay being updated | 17:37 |
| fungi | false alarm. may have been an nova cache update delay. this time it went ready! 104.130.211.12 2001:4801:7827:102:be76:4eff:fe10:6c90 | 17:38 |
| *** ociuhandu has quit IRC | 17:38 | |
| *** ociuhandu has joined #openstack-infra | 17:39 | |
| *** ricolin has quit IRC | 17:39 | |
| fungi | i'm timing out ssh'ing into it via ipv6 though | 17:39 |
| fungi | and ipv4 for that matter | 17:39 |
| fungi | can't establish a socket on 22/tcp | 17:40 |
| fungi | and no replies to icmp echo request | 17:41 |
| clarkb | that could be the NM issue | 17:41 |
| clarkb | because current glean can't configure ipv6 on rax on centos | 17:41 |
| fungi | oh, it got deleted | 17:41 |
| clarkb | and ipv4 is what breaks with current glean | 17:41 |
| *** Goneri has quit IRC | 17:41 | |
| fungi | and now building again | 17:41 |
| fungi | why would it have gone ready? | 17:41 |
| clarkb | that I do not knlow | 17:42 |
| fungi | seems like the launcher shouldn't have listed it as ready if it was just going to delete it | 17:42 |
| fungi | the next build went straight to deleting | 17:43 |
| fungi | | 0012251618 | rax-iad | centos-8 | 54abb4e5-c42e-41c8-a3aa-3174392c8a84 | 104.130.4.224 | 2001:4802:7802:104:be76:4eff:fe20:e39 | deleting | 00:00:00:04 | unlocked | | 17:43 |
| clarkb | is it timing out against ssh? | 17:43 |
| fungi | so we probably need to get a console log | 17:43 |
| clarkb | ya probably boot one by hand and check the console | 17:43 |
| *** jamesmcarthur_ has quit IRC | 17:44 | |
| fungi | no, strangely the launcher log just says it's deleting an unused node, after doing the full dance to collect the host key | 17:44 |
| fungi | so nodepool thinks it should boot the node, but also thinks it should delete it | 17:44 |
| fungi | ?!? | 17:44 |
| *** rlandy|brb is now known as rlandy | 17:45 | |
| fungi | http://paste.openstack.org/show/782735/ | 17:45 |
| pabelanger | you can have nodepool collection console log, via api, if cloud supports it | 17:45 |
| clarkb | pabelanger: rax does not support it | 17:46 |
| fungi | yeah, i actually suspect there's nothing wrong with the node it booted though | 17:46 |
| pabelanger | clarkb: ah, only rax is failing? | 17:46 |
| clarkb | fungi: nodepool only boots based on min ready and demand | 17:46 |
| fungi | pabelanger: only rax has tried to boot it so far | 17:46 |
| clarkb | fungi: maybe min ready weirdness? | 17:46 |
| clarkb | fungi: ya I think rax is where we satisfy min ready by default so that makes sense | 17:46 |
| fungi | min-ready is set to 1, or it presumably wouldn't be booting any at all | 17:46 |
| *** ramishra has quit IRC | 17:51 | |
| clarkb | ok removing the RA delay sysctl setting 5/5 instances get working ipv6 on centos on fn | 17:53 |
| clarkb | now checking to see if they all got ipv4 configured too | 17:53 |
| clarkb | yup they all have ipv4. I'm going to test fedora next | 17:55 |
| clarkb | I think this may be what causes us to tickle the bug that ianw filed | 17:55 |
| clarkb | and I guess having explicit config for ipv6 in NM causes it to not ignore interfaces as we had hoped | 17:55 |
| clarkb | if that is the case I think we update glean and dib and tag them together at roughly the same time | 17:55 |
| *** jpena is now known as jpena|off | 17:57 | |
| clarkb | I'll need to test these centos and fedora images on all the clouds too probably | 17:57 |
| clarkb | since they all use a slightly different varient of glean behavior :( | 17:57 |
| openstackgerrit | David Shrewsbury proposed zuul/nodepool master: WIP: experimenting with using ZK for fake driver https://review.opendev.org/687150 | 17:59 |
| *** ociuhandu has quit IRC | 17:59 | |
| *** jamesmcarthur has joined #openstack-infra | 18:00 | |
| *** ccamacho has joined #openstack-infra | 18:02 | |
| clarkb | ok 5/5 ipv6 setups work on fedora29 too without solicitation delay. only 3/5 ipv4 setups work | 18:13 |
| clarkb | it feels like we can have ipv6 or ipv4 but if you'd like to have both then you need to look in another castle | 18:13 |
| clarkb | ianw: ^ to tl;dr removing the router solicitation delay seems to fix ipv6 configuration, but we go back to having problems with ipv4 in some cases | 18:15 |
| *** efried is now known as efried_pto | 18:16 | |
| *** igordc has quit IRC | 18:19 | |
| *** ykarel|away has joined #openstack-infra | 18:23 | |
| pabelanger | looking at https://launchpad.net/~openstack-ci-core/+archive/ubuntu/vhd-util we don't have bionic packages, which is needed for DIB / and rackspace. Could we try to rebuild xenial dpkg for bionic? | 18:28 |
| fungi | pabelanger: and there's not one included directly in bionic/universe now? | 18:30 |
| pabelanger | fungi: no, I think we carry an out of tree patch, IIRC | 18:30 |
| fungi | ahh | 18:30 |
| pabelanger | when I last tried to use vhd-util directly for vhd, I don't believe it worked | 18:31 |
| clarkb | ya its an out of tree patch :/ | 18:33 |
| clarkb | upstream fedora-29 image takes forever to bring up networking to the point where I thought it had failed | 18:33 |
| clarkb | however it does bring up both ipv4 and ipv6 with cloud init (at least on a single attempt I need to boot a bunch more tests since fail rate seems to be ~40%) | 18:33 |
| clarkb | I notice that it does not explicitly configure ipv6 in sysconfig and the only ipv4 option we don't use is the one for persistent dhcp | 18:34 |
| clarkb | it also doesn't set NM_CONTROLLED=yes but nmcli implies it is actually NM controlled | 18:34 |
| clarkb | it is possible that PERSISTENT_DHCLIENT is the behavior change we need for ipv4 so I will be testing that after lunch | 18:35 |
| *** yamamoto has joined #openstack-infra | 18:35 | |
| *** e0ne has joined #openstack-infra | 18:37 | |
| *** yamamoto has quit IRC | 18:40 | |
| *** e0ne has quit IRC | 18:41 | |
| *** Goneri has joined #openstack-infra | 18:44 | |
| openstackgerrit | David Shrewsbury proposed zuul/nodepool master: Fix builder shutdown race in tests https://review.opendev.org/687965 | 18:49 |
| openstackgerrit | Merged opendev/storyboard-webclient master: Adds Migration Docs to Dashboard https://review.opendev.org/680235 | 18:53 |
| openstackgerrit | Merged opendev/storyboard master: Link development.rst to contributing.rst https://review.opendev.org/645960 | 18:56 |
| *** prometheanfire has quit IRC | 18:57 | |
| *** prometheanfire has joined #openstack-infra | 18:58 | |
| openstackgerrit | Frode Nordahl proposed openstack/project-config master: Add OVN charms https://review.opendev.org/687925 | 18:59 |
| *** ykarel|away has quit IRC | 19:00 | |
| fungi | after moving logs to swift (i think) the build-javascript-content job result for opendev/storyboard-webclient has stopped being usable for anything involving interactions with the storybaord-dev.o.o api or authenticating with openid: https://99957bd7ffedb79bb17e-02cf1f4ef0de29ab49209009be295d1d.ssl.cf2.rackcdn.com/680235/2/gate/build-javascript-content/4fb6c68/npm/html/ | 19:02 |
| fungi | we did something similar to solve those sorts of problems for the zuul dashboard preview builds, right? | 19:03 |
| clarkb | there is the zuul proxy thing but that mostly has to do with rooting the uris at / | 19:04 |
| openstackgerrit | Frode Nordahl proposed openstack/project-config master: Add OVN charms https://review.opendev.org/687925 | 19:07 |
| *** yamamoto has joined #openstack-infra | 19:07 | |
| *** kjackal_v2 has quit IRC | 19:08 | |
| openstackgerrit | Frode Nordahl proposed openstack/project-config master: Add OVN charms https://review.opendev.org/687925 | 19:09 |
| *** kjackal has joined #openstack-infra | 19:11 | |
| *** whoami-rajat has quit IRC | 19:12 | |
| *** yamamoto has quit IRC | 19:12 | |
| openstackgerrit | Merged zuul/nodepool master: Don't touch static nodes that are allocated https://review.opendev.org/687806 | 19:26 |
| *** ociuhandu has joined #openstack-infra | 19:26 | |
| *** bnemec has quit IRC | 19:29 | |
| *** pkopec has quit IRC | 19:29 | |
| *** bnemec has joined #openstack-infra | 19:30 | |
| openstackgerrit | David Shrewsbury proposed zuul/nodepool master: Fix builder shutdown race in tests https://review.opendev.org/687965 | 19:32 |
| fungi | oh, yeah hrm... | 19:34 |
| fungi | in this case it's more a problem of cors permission i think | 19:34 |
| openstackgerrit | Merged zuul/nodepool master: Sort waiting static nodes by creation time https://review.opendev.org/687271 | 19:38 |
| *** Goneri has quit IRC | 19:40 | |
| clarkb | Ok I didn't end up finding lunch and just went ahead and tested adding PERSISTENT_DHCLIENT. It seems to have been more reliable. My first 6 boots of fedora nd centos each worked (12 total boots) | 19:41 |
| clarkb | then I wrote a script that would boot fedora, ssh in via ipv6 and check ipv4 in a loop and that caught a failure almost immediately | 19:42 |
| clarkb | The difference between upstream images and ours must be in boot timing/races or some other network manager config | 19:42 |
| *** yamamoto has joined #openstack-infra | 19:43 | |
| clarkb | I think the next step is to enable NM debug logging and then reproduce, but I'm running out of steam on this | 19:43 |
| clarkb | if it is still a valid option I Think we should consider not using NM | 19:44 |
| fungi | yeah, it doesn't seem well-suited to this use case | 19:45 |
| clarkb | if NM is required (I think that was the concern that newer fedora/rhel/centos would require it) then we need to probably have a heart to heart with upstream | 19:46 |
| clarkb | the docs are really bad ( like really bad ), the behavior is unexpected and not logged (when it decides to ignore an interface you've explcitly told it to not ignore via NM_MANAGED=yes and similar config) | 19:47 |
| *** yamamoto has quit IRC | 19:48 | |
| clarkb | curiously I've recently started having similar problems on my local desktop | 19:48 |
| fungi | and also it seems to be just plain unreliable due to timing races | 19:48 |
| clarkb | remember all those reboots I did for apparomor? | 19:49 |
| clarkb | well now NM comes up and doesn't configure any interfaces until I restart it | 19:49 |
| clarkb | thankfully (heh not really) I'd already run into this behavior with glean and know that restarting it likely fixes it | 19:49 |
| clarkb | I expect my problems on the desktop are also timing races | 19:50 |
| clarkb | if anyone is wondering where to find teh docs for RH's sysconfig + NM configuration it is in gnome | 19:51 |
| clarkb | not in the RH docs as far as I can tell | 19:51 |
| clarkb | https://developer.gnome.org/NetworkManager/stable/nm-settings-ifcfg-rh.html I Guess because the rh nm settings plugin is actually an upstraem NM plugin | 19:52 |
| *** ociuhandu has quit IRC | 19:54 | |
| *** igordc has joined #openstack-infra | 20:03 | |
| EmilienM | hey folks | 20:03 |
| EmilienM | ERROR Ansible plugin dir /var/lib/zuul/builds/228ffd2f4c70427bb4cb895178dd67a7/ansible/pre_playbook_1/role_0/tripleo-ansible/roles/tripleo-container-manage/filter_plugins found adjacent to playbook /var/lib/zuul/builds/228ffd2f4c70427bb4cb895178dd67a7/ansible/pre_playbook_1/role_0/tripleo-ansible/roles/tripleo-container-manage in non-trusted repo. | 20:03 |
| EmilienM | it seems like it doesn't like my customer filter plugin | 20:04 |
| pabelanger | EmilienM: yah, zuul won't load top-level plugins for security reasons | 20:04 |
| pabelanger | since they would run on executor side | 20:05 |
| EmilienM | what should I do? | 20:05 |
| pabelanger | to work in untrusted, you'd need to move them | 20:05 |
| paladox | fyi if you use gerrit.wikimedia.org we have upcoming maintenance https://lists.wikimedia.org/pipermail/wikitech-l/2019-October/092664.html :) | 20:05 |
| pabelanger | EmilienM: or see how to move them to trusted context | 20:05 |
| pabelanger | EmilienM: in this case, you likely can used them with nested ansible | 20:06 |
| EmilienM | pabelanger: I need to go afk a little, if you can comment on https://review.opendev.org/#/c/686196/ please | 20:07 |
| pabelanger | sure, can look in a bit | 20:07 |
| EmilienM | thx | 20:07 |
| *** michael-beaver has quit IRC | 20:08 | |
| *** jamesmcarthur has quit IRC | 20:08 | |
| *** jamesmcarthur has joined #openstack-infra | 20:09 | |
| *** eharney_ has joined #openstack-infra | 20:12 | |
| *** eharney has quit IRC | 20:12 | |
| *** eharney_ is now known as eharney | 20:13 | |
| *** jamesmcarthur has quit IRC | 20:14 | |
| *** yamamoto has joined #openstack-infra | 20:14 | |
| *** pcaruana has quit IRC | 20:15 | |
| *** Goneri has joined #openstack-infra | 20:19 | |
| *** yamamoto has quit IRC | 20:19 | |
| *** jamesmcarthur has joined #openstack-infra | 20:25 | |
| *** jamesmcarthur has quit IRC | 20:27 | |
| *** e0ne has joined #openstack-infra | 20:28 | |
| *** spsurya has quit IRC | 20:28 | |
| *** jamesmcarthur has joined #openstack-infra | 20:30 | |
| ianw | pabelanger: i'll look at the bionic packages | 20:32 |
| clarkb | ianw: I wrote a lot above doing further NM fiddling . Ithink our RA solicit delay may be the cause of the delay that cause NM to timeout | 20:32 |
| clarkb | removing the solicit delay fixes that problem but brings back the "ipv4 doesn't work becase NM won't configure the interface now" | 20:33 |
| *** kjackal has quit IRC | 20:35 | |
| ianw | clarkb: yeah, just looking ... how long was the ra delay? | 20:36 |
| clarkb | I then tried the one difference I Found on an upstream image compard with ours and it is the PERSISTENT_DHCLIENT=yes setting. Setting that doesn't fix the NM refuses to configure interface beacuse something else got there first | 20:36 |
| clarkb | ianw: I think we are at 30 second now | 20:36 |
| clarkb | we started at 10 | 20:36 |
| clarkb | I undid the changeto update it and kernel seems to default to 1 second | 20:36 |
| ianw | hrm, i guess that would explain hitting that timeout | 20:38 |
| clarkb | it is really odd to me that the NM_MANAGED flag seems to be ignored | 20:38 |
| ianw | i never saw the ipv4 failures ... do you now have an image that replicates that? | 20:38 |
| openstackgerrit | James E. Blair proposed zuul/zuul-registry master: Add docker buildset test https://review.opendev.org/687953 | 20:39 |
| clarkb | ianw: ya clarkb-test-glean-fedora3 and clarkb-test-glean-centos3 do it on fn | 20:39 |
| clarkb | it isn't 100% failure though | 20:40 |
| clarkb | its the same problem that we put the solicit delay in place to fix | 20:40 |
| clarkb | `nmcl c show` shows two ens3/eth0's | 20:40 |
| clarkb | one that is a stand in because kernel managed the itnerface and the other for the interface we asked NM to configure that it refuses to configure | 20:40 |
| *** factor has quit IRC | 20:42 | |
| clarkb | clarkb-test-glean-fedora exhibits this issue in fn | 20:42 |
| clarkb | I think we've got two different bugs playing off of each other. Effectively forcing us to have working ipv4 or working ipv6 but not both consistently | 20:43 |
| ianw | clarkb: interesting, because i was booting clarkb-test-glean-fedora yesterday to debug that timeout, and didn't see this | 20:43 |
| clarkb | ianw: ya those older images had the solicit delay and would get ipv4 reliably but not ipv6 | 20:43 |
| ianw | oh, ok, so the image is updated? | 20:43 |
| clarkb | I built new images without the solicit delay thinking it might explain the bug you filed | 20:43 |
| clarkb | ianw: so that is an instance name | 20:43 |
| *** jamesmcarthur has quit IRC | 20:43 | |
| clarkb | clarkb-test-glean-fedora3 is the image that instance is built on | 20:44 |
| clarkb | (name collisions across resource types) | 20:44 |
| *** jamesmcarthur has joined #openstack-infra | 20:44 | |
| *** eharney has quit IRC | 20:45 | |
| ianw | ok trying fedora3 image now | 20:45 |
| clarkb | you can ssh into clarkb-test-glean-fedora via ipv6 then ifconfig and nmcli c show and nmcli d show to see what is going on there | 20:45 |
| clarkb | its basically the same problem as before we added the solicit delay. | 20:46 |
| clarkb | ianw: note it isn't a 100% failure so you may have to loop a few times to catch one | 20:46 |
| *** roman_g has quit IRC | 20:48 | |
| *** e0ne has quit IRC | 20:48 | |
| ianw | ok, trying now | 20:49 |
| *** factor has joined #openstack-infra | 20:49 | |
| *** FlorianFa has quit IRC | 20:49 | |
| clarkb | oh wait that host may use a different ssh key beause it was part of my test loop on bridge. If you su to me on bridge you'll be able to ssh from the key I generated there for this task | 20:49 |
| *** jamesmcarthur has quit IRC | 20:50 | |
| clarkb | sorry I had been using the infra root keys previosuly but wanted to do an automated loop check to see if we were still catching this problem and set up a new ssh key forthat on bridge | 20:50 |
| *** jamesmcarthur has joined #openstack-infra | 20:50 | |
| *** jamesmcarthur has quit IRC | 20:52 | |
| ianw | well i am able to log into a host iwth that image | 20:53 |
| *** tesseract has quit IRC | 20:53 | |
| ianw | i've rebooted 15 times now and not seen an issue :/ | 20:53 |
| clarkb | ianw: ipv6 works | 20:53 |
| clarkb | only ipv4 fails and only sometimes (it was my 13th boot that it failed) | 20:54 |
| clarkb | 13th new instance boot, not reboots of the same host | 20:54 |
| ianw | ahh, ok ... i am clearing the glean file. i wonder if there's other persistent state | 20:55 |
| ianw | clarkb: we really need to capture it with debugging on. is the .qcow2 somewhere i can guestfish into it and update the config file? | 20:57 |
| clarkb | ianw: yes nb01.openstack.org:~clarkb/something-fedora.qcow2 | 20:58 |
| clarkb | it should be the file with the latest timestmap | 20:58 |
| clarkb | (sorry I killed my ssh agent and haven't dug out the physical media to reload keys yet) | 20:58 |
| ianw | ok, will play now | 20:59 |
| openstackgerrit | James E. Blair proposed zuul/zuul-registry master: Add podman buildset test https://review.opendev.org/687986 | 20:59 |
| *** xenos76 has quit IRC | 20:59 | |
| clarkb | ianw: that image includes the PERSISTENT_DHCLIENT change to the ifcfg files and the removal of changing the RA solicit delay | 21:00 |
| clarkb | otherwise it should be the same as the images I had built previously | 21:00 |
| *** slaweq has quit IRC | 21:01 | |
| *** FlorianFa has joined #openstack-infra | 21:01 | |
| ianw | test-glean-nm-updates-fedora.qcow2 | 21:01 |
| ianw | at least the centos-8 build went ok | 21:04 |
| *** jamesmcarthur has joined #openstack-infra | 21:07 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-registry master: Add podman buildset test https://review.opendev.org/687986 | 21:08 |
| *** FlorianFa has quit IRC | 21:08 | |
| *** markvoelker has quit IRC | 21:09 | |
| *** sreejithp has quit IRC | 21:11 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-registry master: Add podman buildset test https://review.opendev.org/687986 | 21:11 |
| *** slaweq has joined #openstack-infra | 21:11 | |
| *** rfolco|ruck has quit IRC | 21:13 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-registry master: Add podman buildset test https://review.opendev.org/687986 | 21:14 |
| *** slaweq has quit IRC | 21:17 | |
| *** FlorianFa has joined #openstack-infra | 21:21 | |
| *** xek_ has quit IRC | 21:22 | |
| *** benj- has quit IRC | 21:28 | |
| *** benj has joined #openstack-infra | 21:31 | |
| *** benj is now known as Guest69423 | 21:31 | |
| *** e0ne has joined #openstack-infra | 21:34 | |
| *** jamesmcarthur has quit IRC | 21:46 | |
| *** ociuhandu has joined #openstack-infra | 21:55 | |
| *** ociuhandu has quit IRC | 21:59 | |
| *** jbadiapa has quit IRC | 22:02 | |
| *** yamamoto has joined #openstack-infra | 22:03 | |
| *** trident has quit IRC | 22:03 | |
| *** mriedem has quit IRC | 22:04 | |
| *** trident has joined #openstack-infra | 22:05 | |
| *** yamamoto has quit IRC | 22:08 | |
| *** ralonsoh has quit IRC | 22:12 | |
| ianw | clarkb: replicated, with debug logs | 22:18 |
| clarkb | progress | 22:19 |
| *** rlandy is now known as rlandy|bbl | 22:23 | |
| *** e0ne has quit IRC | 22:24 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-registry master: Remove unused file from functional test https://review.opendev.org/687998 | 22:28 |
| *** yamamoto has joined #openstack-infra | 22:30 | |
| *** igordc has quit IRC | 22:36 | |
| clarkb | ianw: are you able to share the logs (I'm mostly curious to see what they look like with debugging enabled) | 22:40 |
| ianw | clarkb: yep, one tick | 22:40 |
| ianw | clarkb: have you ever maanged to boot & get into a upstream fedora image? | 22:40 |
| clarkb | ianw: yes, I did that using the fedora29 image on FN with ssh keys metadata set and config drive enabled | 22:41 |
| ianw | hrm, i'm trying that and no joy, but with my own image modified with nm | 22:42 |
| ianw | https://people.redhat.com/~iwienand/bad.txt | 22:42 |
| clarkb | ianw: the image I used is uploaded in FN and called upstream-fedora29 or similar | 22:42 |
| ianw | no, i tell a lie, it's up now ... it just took a while | 22:42 |
| ianw | platform: signal: link added: 2: eth0 <DOWN;broadcast,multicast> mtu 1450 arp 1 ethernet? not-init addrgenmode eui64 addr FA:16:3E:03:BE:62 driver virtio_net rx:0,0 tx:0,0 | 22:44 |
| ianw | on the upstream image, eth0 is starting in "DOWN" state ... | 22:44 |
| clarkb | oh ya it does take a long time | 22:45 |
| clarkb | I thought it had failed too then tried again a few minutes later and it worked | 22:45 |
| clarkb | Oct 10 21:51:57 ianw-test-glean-debug NetworkManager[854]: <debug> [1570744317.8035] Connection 'ens3' differs from candidate 'System ens3' in ipv4.method | 22:45 |
| clarkb | I think ^ is sort of the first clue as to why this is happening on the failed case | 22:45 |
| ianw | yep, that's the key message i think, where it decides "can't touch this" | 22:45 |
| *** rcernin has joined #openstack-infra | 22:46 | |
| ianw | i wonder if cloud-init is clearing an RA addresses and downing then interface? | 22:46 |
| clarkb | System ens3 has ipv4.method set to auto (this comes from our config) and ens3 has it set to disabled | 22:46 |
| clarkb | (thinking out loud here, messages like that should be well above debug level imo) | 22:47 |
| ianw | once again, "eth0 <DOWN;broadcast,multicast>" on upstream image | 22:48 |
| clarkb | downing the interface so that NM sees it as fresh and new? That could be | 22:49 |
| clarkb | could probably make an image with modified glean that does that without too much trouble | 22:50 |
| clarkb | though that may still race? | 22:50 |
| clarkb | since the interface could be UP'd between glean.sh running and NM starting | 22:50 |
| ianw | what would do that though? | 22:51 |
| clarkb | I think it would have to be cloud init or a udev rule? | 22:52 |
| ianw | maybe that's it ... something in udev? | 22:53 |
| ianw | we set our own udev rules right? | 22:54 |
| clarkb | ya I think we use udev rules to trigger glean.sh against specific interfaces? | 22:55 |
| * clarkb looks | 22:55 | |
| clarkb | ya glean/init/glean-udev.rules | 22:56 |
| clarkb | has a one liner that appears to be systemd specific saying "when you udev add a network interface add a systemd wants rule for glean.sh to run against taht interface" | 22:56 |
| clarkb | ianw: we could potentially add a udev rule that down's the interface on add from the start | 22:57 |
| clarkb | then NM would bring it up | 22:57 |
| clarkb | (and that should avoid the stray RAs? | 22:58 |
| ianw | it seems to be a pretty big difference here ... i wonder what brings it up | 22:59 |
| ianw | i wonder if it's the predicable ntework naming ... cloud-init is still using eth0 | 23:00 |
| clarkb | we get a different set of udev rules when changing name schemes right? | 23:01 |
| clarkb | I suppose that could be it | 23:02 |
| *** aaronsheffield has quit IRC | 23:05 | |
| *** tkajinam has joined #openstack-infra | 23:06 | |
| ianw | Oct 10 21:51:55 localhost kernel: virtio_net virtio0 ens3: renamed from eth0 | 23:06 |
| ianw | perhaps that brings it up? | 23:06 |
| clarkb | that could be | 23:07 |
| clarkb | SUBSYSTEM=="net", ACTION=="add", ATTR{addr_assign_type}=="0", RUN+="ip link set $name down" | 23:07 |
| clarkb | a rule like ^ might do what we want? | 23:07 |
| *** ccamacho has quit IRC | 23:10 | |
| *** markvoelker has joined #openstack-infra | 23:10 | |
| clarkb | the fedora default is to devbiosname | 23:10 |
| clarkb | the upstream image might be setting biosdevname=0 on the kernel command line? | 23:11 |
| *** slaweq has joined #openstack-infra | 23:11 | |
| clarkb | I know we've set kernel parameters with dib in the past, we should be able to set biosdevname=0 and test with that | 23:11 |
| *** diablo_rojo has quit IRC | 23:11 | |
| clarkb | ianw: actually centos doesn't biosdevname | 23:12 |
| ianw | net.ifnames=0 | 23:13 |
| clarkb | clarkb-test-glean-centos3 is the equivalent image but for centos7 (and it is the centos image on nb01 in my homedir if you want to modify it) | 23:13 |
| clarkb | I did 6 boots of centos on that image without problems | 23:14 |
| clarkb | maybe we see if centos 7 has the problem at all and if so that should rule this out? | 23:14 |
| *** markvoelker has quit IRC | 23:15 | |
| *** slaweq has quit IRC | 23:16 | |
| ianw | just seeing if net.ifnames makes any difference to initial interface state | 23:16 |
| ianw | 2001:470:e045:8000:f816:3eff:fe03:be62 port 22: Connection refused ... seems to have locked me out :/ | 23:17 |
| clarkb | oops | 23:18 |
| ianw | clarkb: do you have a host can try sshing to 192.168.48.151 ? | 23:19 |
| ianw | wait, i'm in how | 23:20 |
| ianw | now | 23:20 |
| clarkb | fwiw I should have a test node I booted previously I could bounce through | 23:20 |
| openstackgerrit | James E. Blair proposed zuul/zuul-registry master: Add podman buildset test https://review.opendev.org/687986 | 23:25 |
| donnyd | clarkb: so this is a little strange http://grafana.openstack.org/d/3Bwpi5SZk/nodepool-fortnebula?orgId=1&from=now-24h&to=now | 23:27 |
| donnyd | I have been watching this issue for the last week or so | 23:27 |
| donnyd | there are huge chunks of time where nodepool is reporting the vm | 23:27 |
| donnyd | as deleting | 23:27 |
| donnyd | but they are deleted in a few seconds from FN | 23:28 |
| *** diablo_rojo has joined #openstack-infra | 23:28 | |
| donnyd | so I am not quite sure what to do from my end... | 23:29 |
| donnyd | but its missing a lot of CI cycle time | 23:29 |
| clarkb | donnyd: nodepool will actually poll nova to check that the delete succeeded | 23:29 |
| clarkb | is it possible that nova isn't actually reporting those deletes as completed via the api? | 23:30 |
| *** vesper11 has quit IRC | 23:30 | |
| donnyd | when I do an openstack server list during one of these events I don't see any not reporting as ACTIVE | 23:30 |
| *** vesper11 has joined #openstack-infra | 23:31 | |
| *** goldyfruit has quit IRC | 23:31 | |
| clarkb | donnyd: ya so nodepool may have asked nova to delete them and the state didn't change | 23:31 |
| clarkb | (nodepool will retry) | 23:32 |
| donnyd | but it seems to take quite a while, and it just started doing it like a week or so ago | 23:32 |
| donnyd | I will keep an eye on it | 23:32 |
| donnyd | nothing is busted.. just want the community to get the most FN can give | 23:32 |
| clarkb | donnyd: in those cases it might be helpful to check the api logs for incoming delete requests and see if nova failed to handle them | 23:33 |
| ianw | clarkb: still seems to rename them, even with out the cmdline option | 23:33 |
| donnyd | kk | 23:33 |
| *** dchen has joined #openstack-infra | 23:34 | |
| ianw | "KVM guests exclusively using virtio-net type interfaces can safely set net.ifnames=0" | 23:35 |
| ianw | maybe we should be setting it anyway | 23:35 |
| ianw | subprocess.check_call(['ip', 'link', 'set', 'dev', iface, 'up']) | 23:39 |
| ianw | https://opendev.org/opendev/glean/src/branch/master/glean/cmd.py#L1134 might be our smoking gun here | 23:40 |
| clarkb | hrm it does seem odd that we would do that when we write the config after the fact and then rely on the init system to up the network with the correct config | 23:44 |
| clarkb | I think this is an optimization to only configure interfaces with an active carrier link | 23:44 |
| clarkb | but maybe that isn't worth doing | 23:44 |
| clarkb | also maybe we can check that without UPing the interfaces | 23:45 |
| ianw | yeah, this fits almost exactly ... interface comes up ... sometimes gets the RA ... networkmanager doesn't touch it by design because it thinks it's configured by something else | 23:45 |
| clarkb | ianw: I would argue this is also a bug in NM because we've explicitly told NM you manage this interface | 23:46 |
| clarkb | via the NM_MANAGED flag | 23:46 |
| clarkb | if we weren't setting that then ok fine the behavior kind of makes sense | 23:46 |
| clarkb | ip link show foo gives me a NO-CARRIER attribute on an unplugged rj45 jack | 23:47 |
| clarkb | the interface is up though | 23:47 |
| * clarkb downs it to see if that changes | 23:47 | |
| ianw | yeah, i mean according to -> https://bugs.debian.org/cgi-bin/bugreport.cgi?att=1;bug=755202;filename=irc-log.txt;msg=156 that's basically, as they say "intended but sub-optimal behaviour" | 23:47 |
| clarkb | hrm maybe this interface was down | 23:49 |
| clarkb | I need a more interesting network setup on this machine to be able to compare between interfaces | 23:49 |
| openstackgerrit | James E. Blair proposed zuul/zuul-registry master: Add podman buildset test https://review.opendev.org/687986 | 23:49 |
| clarkb | ianw: the sysfs carrier attribute check should actually be sufficient | 23:52 |
| clarkb | ianw: there is a good chance we can just delete that ip link set foo up command | 23:52 |
| clarkb | I bet if I git blame this we'll actually get some commit about how this was added to fix baremetal use cases | 23:53 |
| *** gagehugo has quit IRC | 23:54 | |
| clarkb | wow that code actually comes from disk image builder says the commit that mordred wrote | 23:55 |
| ianw | yeah, was just looking at cloud-init which uses "sys/net/devname/carrier is 1" | 23:55 |
| clarkb | ianw: also since we are predominantly systemd and udev driven now we should only be touching interfaces that exist and are expected to do a thing | 23:55 |
| clarkb | but even when we aren't I don't know what that wait is supposed to accomplish. I guess it gives time for "hardware" to establish that l1 connection | 23:56 |
| clarkb | (wouldn't your pre linux boot stuff do that though?) | 23:57 |
| clarkb | I think my vote is to remove the ip link up then we can do some exhaustive boot tests across all the things (ugh) and if they work just roll with it | 23:57 |
| ianw | yeah, just playing with that on my test host now | 23:58 |
| clarkb | TheJulia actually updated that exec call to be compat with older python at some point. Makes me think that we probably only hit that code path on baremetal | 23:59 |
| clarkb | (we would've seen issues with it on our VMs otherwise) | 23:59 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!