| *** jamesmcarthur has joined #openstack-infra | 00:01 | |
| *** cgoncalves has joined #openstack-infra | 00:09 | |
| *** rcernin has joined #openstack-infra | 00:13 | |
| *** jamesmcarthur has quit IRC | 00:23 | |
| *** trident has joined #openstack-infra | 00:33 | |
| *** jamesmcarthur has joined #openstack-infra | 00:47 | |
| *** jamesmcarthur has quit IRC | 01:00 | |
| *** jamesmcarthur has joined #openstack-infra | 01:02 | |
| *** imacdonn has quit IRC | 01:13 | |
| *** imacdonn has joined #openstack-infra | 01:13 | |
| *** slaweq has joined #openstack-infra | 01:14 | |
| *** jamesmcarthur has quit IRC | 01:17 | |
| *** slaweq has quit IRC | 01:24 | |
| *** altlogbot_2 has quit IRC | 01:28 | |
| *** altlogbot_0 has joined #openstack-infra | 01:29 | |
| *** jamesmcarthur has joined #openstack-infra | 01:45 | |
| *** jamesmcarthur has quit IRC | 01:51 | |
| *** apetrich has quit IRC | 01:58 | |
| *** jamesmcarthur has joined #openstack-infra | 02:05 | |
| *** armax has quit IRC | 02:05 | |
| *** rfolco has quit IRC | 02:09 | |
| *** bobh has joined #openstack-infra | 02:12 | |
| *** yamamoto has joined #openstack-infra | 02:16 | |
| *** slaweq has joined #openstack-infra | 02:16 | |
| *** bobh has quit IRC | 02:23 | |
| *** slaweq has quit IRC | 02:24 | |
| *** bhavikdbavishi has joined #openstack-infra | 02:25 | |
| *** bobh has joined #openstack-infra | 02:26 | |
| *** bhavikdbavishi1 has joined #openstack-infra | 02:28 | |
| *** bhavikdbavishi has quit IRC | 02:30 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 02:30 | |
| *** hwoarang has quit IRC | 02:36 | |
| *** hwoarang has joined #openstack-infra | 02:38 | |
| *** rkukura has joined #openstack-infra | 02:40 | |
| *** bhavikdbavishi has quit IRC | 02:59 | |
| *** slaweq has joined #openstack-infra | 03:15 | |
| *** yamamoto has quit IRC | 03:18 | |
| *** slaweq has quit IRC | 03:24 | |
| *** jamesmcarthur has quit IRC | 03:25 | |
| *** psachin has joined #openstack-infra | 03:27 | |
| *** rh-jelabarre has joined #openstack-infra | 03:48 | |
| *** zhurong has quit IRC | 03:49 | |
| *** ykarel has joined #openstack-infra | 03:49 | |
| *** bobh has quit IRC | 03:53 | |
| *** bobh has joined #openstack-infra | 03:58 | |
| *** bobh has quit IRC | 04:02 | |
| *** udesale has joined #openstack-infra | 04:09 | |
| *** bhavikdbavishi has joined #openstack-infra | 04:10 | |
| *** slaweq has joined #openstack-infra | 04:11 | |
| *** factor has quit IRC | 04:13 | |
| *** factor has joined #openstack-infra | 04:14 | |
| *** SotK has quit IRC | 04:14 | |
| *** SotK has joined #openstack-infra | 04:15 | |
| *** Teolupus has joined #openstack-infra | 04:16 | |
| *** slaweq has quit IRC | 04:24 | |
| *** rh-jelabarre has quit IRC | 04:29 | |
| Teolupus | Out of curiosity, why on Neutron changes like this one https://review.opendev.org/#/c/162088/ I can see the third party vendors' CI checks, whereas in more recent changes like https://review.opendev.org/#/c/414251/ I only see their CI systems in the reviewers list? Are third party vendors still testing every code change to Neutron on their non-voting CIs and feeding that back upstream? | 04:33 |
|---|---|---|
| *** ykarel has quit IRC | 04:53 | |
| *** bobh has joined #openstack-infra | 05:09 | |
| *** slaweq has joined #openstack-infra | 05:11 | |
| *** bobh has quit IRC | 05:13 | |
| AJaeger_ | Teolupus: I see in 414251 the CI checks as well... | 05:17 |
| *** AJaeger_ is now known as AJaeger | 05:17 | |
| *** ykarel has joined #openstack-infra | 05:17 | |
| AJaeger | Teolupus: Those are still reporting. Btw. 414251 is one year old, look at newer ones. To show up in the list of reviews, the format needs to follow what we parse, if a CI system invents a new format, bad luck ;( | 05:19 |
| *** ociuhandu has joined #openstack-infra | 05:20 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Add letsencrypt documentation https://review.opendev.org/669602 | 05:21 |
| *** slaweq has quit IRC | 05:25 | |
| *** ociuhandu has quit IRC | 05:27 | |
| *** ociuhandu_ has joined #openstack-infra | 05:27 | |
| *** whoami-rajat has joined #openstack-infra | 05:30 | |
| *** ociuhandu_ has quit IRC | 05:33 | |
| *** Tengu has quit IRC | 05:38 | |
| *** Tengu has joined #openstack-infra | 05:43 | |
| *** raukadah is now known as chandankumar | 05:51 | |
| AJaeger | config-core, could you review the stack at https://review.opendev.org/#/c/668955 for testing of zuul-jobs, please? | 05:52 |
| *** ccamacho has joined #openstack-infra | 06:00 | |
| *** slaweq has joined #openstack-infra | 06:11 | |
| *** pcaruana has joined #openstack-infra | 06:12 | |
| *** jbadiapa has joined #openstack-infra | 06:19 | |
| *** slaweq has quit IRC | 06:26 | |
| *** xek_ has joined #openstack-infra | 06:30 | |
| *** slaweq has joined #openstack-infra | 06:39 | |
| *** ginopc has joined #openstack-infra | 06:55 | |
| *** Teolupus has quit IRC | 07:01 | |
| *** iurygregory has joined #openstack-infra | 07:07 | |
| *** rcernin has quit IRC | 07:12 | |
| *** pgaxatte has joined #openstack-infra | 07:14 | |
| *** ricolin has joined #openstack-infra | 07:18 | |
| *** psachin has quit IRC | 07:19 | |
| *** sshnaidm|ruck is now known as sshnaidm|afk | 07:21 | |
| *** witek has joined #openstack-infra | 07:23 | |
| *** apetrich has joined #openstack-infra | 07:32 | |
| *** kjackal has joined #openstack-infra | 07:33 | |
| *** iokiwi has quit IRC | 07:38 | |
| *** adriant has quit IRC | 07:38 | |
| *** lucasagomes has joined #openstack-infra | 07:39 | |
| *** ykarel is now known as ykarel|lunch | 07:44 | |
| *** ociuhandu has joined #openstack-infra | 07:45 | |
| *** ralonsoh has joined #openstack-infra | 07:55 | |
| *** dchen has quit IRC | 07:57 | |
| *** iurygregory has quit IRC | 07:57 | |
| *** bobh has joined #openstack-infra | 08:02 | |
| *** pkopec has joined #openstack-infra | 08:03 | |
| *** rpittau|afk is now known as rpittau | 08:03 | |
| *** bobh has quit IRC | 08:06 | |
| openstackgerrit | Merged zuul/zuul master: Fix multi-tenant caching of extra config files https://review.opendev.org/669008 | 08:06 |
| openstackgerrit | Merged opendev/irc-meetings master: Add Image Encryption Popup-Team Meeting https://review.opendev.org/669151 | 08:08 |
| *** tosky has joined #openstack-infra | 08:11 | |
| *** iokiwi has joined #openstack-infra | 08:20 | |
| *** iurygregory has joined #openstack-infra | 08:24 | |
| *** AJaeger has quit IRC | 08:26 | |
| *** psachin has joined #openstack-infra | 08:33 | |
| *** sshnaidm|afk is now known as sshnaidm|ruck | 08:38 | |
| *** ykarel|lunch is now known as ykarel | 08:42 | |
| *** iurygregory has quit IRC | 08:51 | |
| *** derekh has joined #openstack-infra | 08:54 | |
| *** iurygregory has joined #openstack-infra | 08:56 | |
| *** lpetrut has joined #openstack-infra | 09:00 | |
| *** dayou_ has quit IRC | 09:01 | |
| *** dayou_ has joined #openstack-infra | 09:01 | |
| *** Lucas_Gray has joined #openstack-infra | 09:05 | |
| *** priteau has joined #openstack-infra | 09:08 | |
| *** ociuhandu has quit IRC | 09:11 | |
| *** bobh has joined #openstack-infra | 09:17 | |
| *** bobh has quit IRC | 09:21 | |
| *** ykarel is now known as ykarel|meeting | 09:31 | |
| *** pkopec has quit IRC | 09:34 | |
| *** AJaeger has joined #openstack-infra | 09:40 | |
| *** panda is now known as panda|bbl | 09:48 | |
| *** ociuhandu has joined #openstack-infra | 09:48 | |
| *** electrofelix has joined #openstack-infra | 10:29 | |
| *** pkopec has joined #openstack-infra | 10:29 | |
| *** ykarel_ has joined #openstack-infra | 10:31 | |
| *** pkopec has quit IRC | 10:31 | |
| *** panda|bbl has quit IRC | 10:33 | |
| icey | is it possible to get zuul to land a change on a retired project? It doesn't seem to be showing up in gate jobs? https://review.opendev.org/#/c/669618/ | 10:33 |
| *** ykarel|meeting has quit IRC | 10:33 | |
| *** panda has joined #openstack-infra | 10:37 | |
| *** bhavikdbavishi has quit IRC | 10:38 | |
| AJaeger | icey: without jobs configured, it is not possible - and we have ACLs as readonly, so nobody has access to +1/+2A | 10:38 |
| AJaeger | icey: so, add noop-jobs in project-config... | 10:39 |
| AJaeger | icey: and add charm-ceph to zuul/main.yaml and update ACLs - and then revert back... | 10:39 |
| icey | AJaeger: that's a bit heavy handed seeming to clean up the old, retired project :-P | 10:40 |
| openstackgerrit | Chris MacNaughton (icey) proposed openstack/project-config master: Add charm-ceph to allow cleaning up retired project https://review.opendev.org/669632 | 10:42 |
| *** priteau has quit IRC | 10:42 | |
| openstackgerrit | Chris MacNaughton (icey) proposed openstack/project-config master: Remove charm-ceph as it's a retired project https://review.opendev.org/669633 | 10:43 |
| icey | AJaeger: something like that, but with a `recheck` on the retired project between landing? | 10:43 |
| openstackgerrit | Chris MacNaughton (icey) proposed openstack/project-config master: Remove charm-ceph as it's a retired project https://review.opendev.org/669633 | 10:44 |
| icey | (updated with a depends on to the desired change) | 10:44 |
| AJaeger | icey: yes, recheck will be needed... | 10:45 |
| AJaeger | icey: not enough, let me comment... | 10:45 |
| icey | thanks AJaeger - it's not super high priority for me, but I thought it'd be nice to get it "properly" retired, give the ML thread | 10:45 |
| AJaeger | icey: I'll stop reviewing that insanity for a single .gitgnore ;( Let me send a reply on the mailing list | 10:48 |
| icey | thanks for the comment AJaeger - in the meantime, I'll abandon these changes then | 10:52 |
| *** pkopec has joined #openstack-infra | 11:00 | |
| AJaeger | icey: ok - let's see where the discussion goes... | 11:00 |
| *** udesale has quit IRC | 11:00 | |
| icey | AJaeger: indeed! | 11:00 |
| *** ykarel_ is now known as ykarel|meeting | 11:01 | |
| *** priteau has joined #openstack-infra | 11:08 | |
| *** priteau has quit IRC | 11:16 | |
| *** tesseract has joined #openstack-infra | 11:17 | |
| *** yamamoto has joined #openstack-infra | 11:18 | |
| *** tesseract has quit IRC | 11:20 | |
| *** adriancz has joined #openstack-infra | 11:20 | |
| *** tesseract has joined #openstack-infra | 11:21 | |
| *** rh-jelabarre has joined #openstack-infra | 11:22 | |
| *** yamamoto has quit IRC | 11:22 | |
| *** ricolin has quit IRC | 11:26 | |
| openstackgerrit | Chandan Kumar (raukadah) proposed openstack/diskimage-builder master: [RHEL] Set _clear_old_files=0 in install-pip element https://review.opendev.org/669641 | 11:32 |
| *** Lucas_Gray has quit IRC | 11:37 | |
| *** happyhemant has joined #openstack-infra | 11:37 | |
| *** rfolco has joined #openstack-infra | 11:38 | |
| *** Lucas_Gray has joined #openstack-infra | 11:40 | |
| *** bhavikdbavishi has joined #openstack-infra | 11:43 | |
| *** Lucas_Gray has quit IRC | 11:48 | |
| *** ykarel_ has joined #openstack-infra | 11:49 | |
| *** ykarel|meeting has quit IRC | 11:52 | |
| *** Lucas_Gray has joined #openstack-infra | 11:52 | |
| *** bobh has joined #openstack-infra | 11:53 | |
| *** iurygregory has quit IRC | 12:04 | |
| *** bobh has quit IRC | 12:06 | |
| *** ykarel_ is now known as ykarel | 12:06 | |
| *** smarcet has joined #openstack-infra | 12:06 | |
| *** ykarel is now known as ykarel|afk | 12:07 | |
| *** ykarel|afk has quit IRC | 12:11 | |
| *** goldyfruit has quit IRC | 12:22 | |
| *** rlandy has joined #openstack-infra | 12:29 | |
| *** ijw has joined #openstack-infra | 12:30 | |
| *** ijw has quit IRC | 12:35 | |
| *** ykarel|afk has joined #openstack-infra | 12:40 | |
| *** ykarel|afk is now known as ykarel | 12:43 | |
| *** ekultails has joined #openstack-infra | 12:54 | |
| *** rfarr__ has joined #openstack-infra | 12:54 | |
| *** iurygregory has joined #openstack-infra | 13:07 | |
| *** bhavikdbavishi has quit IRC | 13:08 | |
| *** bhavikdbavishi has joined #openstack-infra | 13:09 | |
| *** goldyfruit has joined #openstack-infra | 13:10 | |
| *** tesseract has quit IRC | 13:14 | |
| *** yolanda has quit IRC | 13:14 | |
| *** tesseract has joined #openstack-infra | 13:16 | |
| *** lseki has joined #openstack-infra | 13:20 | |
| *** mriedem has joined #openstack-infra | 13:23 | |
| *** aaronsheffield has joined #openstack-infra | 13:32 | |
| *** sreejithp has joined #openstack-infra | 13:35 | |
| AJaeger | infra-root, seems our retirement ACLs are not working - https://review.opendev.org/#/admin/projects/openstack/charm-ceph,access lists the old ACLs, not the retired ones. But project-config/gerrit/projects.yaml points to the retired ones... | 13:41 |
| AJaeger | this is beneficial, if we need to merge https://review.opendev.org/#/c/669618/ ;) But points out a problem... | 13:42 |
| *** haleyb|away is now known as haleyb | 13:44 | |
| *** electrofelix has quit IRC | 13:45 | |
| AJaeger | same problem here : https://review.opendev.org/#/admin/projects/opendev/puppet-releasestatus,access . | 13:45 |
| AJaeger | Or do I misunderstand the ACLs process? | 13:45 |
| clarkb | AJaeger: that is a known issue with jeepyb. I have a patch up somewhere for it but we need to test it and manually apply it once merged | 13:46 |
| AJaeger | clarkb: Ok, thanks | 13:47 |
| *** bobh has joined #openstack-infra | 13:47 | |
| *** jamesmcarthur has joined #openstack-infra | 13:47 | |
| AJaeger | clarkb: but since repo is read-only, people won't be able to merge anything, correct? | 13:47 |
| clarkb | The readonly change is part of the acl update I think so itmay not flip to read only eother | 13:48 |
| *** eharney has joined #openstack-infra | 13:49 | |
| AJaeger | oops | 13:50 |
| AJaeger | but since we remove it from zuul, zuul will not merge - so, leaves the loophole that an infra-root could send a +2 | 13:50 |
| AJaeger | no real problem I guess ;) | 13:50 |
| *** bobh has quit IRC | 13:52 | |
| *** jamesmcarthur has quit IRC | 13:52 | |
| *** jcoufal has joined #openstack-infra | 13:53 | |
| *** ykarel is now known as ykarel|afk | 13:53 | |
| *** michael-beaver has joined #openstack-infra | 13:53 | |
| corvus | i'm going to restart zuul to pick up the config caching fix | 13:57 |
| *** sgw has quit IRC | 13:57 | |
| AJaeger | \o/ | 13:57 |
| AJaeger | corvus: good morning! I updated your zuul-jobs stack, could you check later whether it's fine? Everything passes now :) https://review.opendev.org/#/c/668955 | 13:58 |
| corvus | AJaeger: thanks, will do | 13:58 |
| johnsom | FYI, https://zuul.openstack.org/status all I get is "Something went wrong." | 13:59 |
| *** smarcet has quit IRC | 14:00 | |
| AJaeger | johnsom: corvus is restarting zuul... | 14:00 |
| AJaeger | johnsom: grab a coffee/tee/whatever and try again, please ;) | 14:00 |
| corvus | AJaeger: we have jobs which use "fedora-latest" nodesets; are you saying they are broken? | 14:01 |
| johnsom | lol, just did grab my coffee. | 14:01 |
| AJaeger | corvus: notesets are fine - but not as LABEL | 14:01 |
| *** priteau has joined #openstack-infra | 14:02 | |
| corvus | AJaeger: oh, i see... | 14:02 |
| *** smarcet has joined #openstack-infra | 14:02 | |
| *** sthussey has joined #openstack-infra | 14:04 | |
| *** FlorianFa has quit IRC | 14:05 | |
| *** smarcet has quit IRC | 14:06 | |
| corvus | #status log restarted all of zuul on commit 5b851c14f2bd73039748fca71b5db3b05b697f7f | 14:07 |
| openstackstatus | corvus: finished logging | 14:07 |
| *** rfarr__ has quit IRC | 14:08 | |
| *** rfarr has joined #openstack-infra | 14:08 | |
| *** dmellado has quit IRC | 14:09 | |
| *** stevebaker has quit IRC | 14:09 | |
| *** dmellado has joined #openstack-infra | 14:11 | |
| *** Lucas_Gray has quit IRC | 14:12 | |
| *** stevebaker has joined #openstack-infra | 14:14 | |
| *** Lucas_Gray has joined #openstack-infra | 14:14 | |
| *** yolanda has joined #openstack-infra | 14:15 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Don't use -latest nodesets in integration jobs https://review.opendev.org/669675 | 14:17 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Change the repo used in some base-roles tests https://review.opendev.org/669676 | 14:18 |
| corvus | AJaeger: your updates look good, thanks; those are the small changes i'd like to make ^ | 14:18 |
| elod | hi, i have a question regarding a patch, which was merged in but gerrit showed it as open and now after a recheck it got (2nd time?) merged, but disappeared: https://review.opendev.org/667721 | 14:18 |
| elod | should/can I do anything with it? | 14:19 |
| openstackgerrit | Chandan Kumar (raukadah) proposed openstack/diskimage-builder master: [RHEL-8] Set _clear_old_files=0 in install-pip element https://review.opendev.org/669641 | 14:20 |
| AJaeger | corvus: will review later - thanks | 14:22 |
| *** ykarel|afk is now known as ykarel | 14:23 | |
| corvus | elod: if the change appears in the git repo and the review history isn't critical, then i wouldn't worry about it. i'll look into it to see if there's something systemically wrong, but if it's not super important, i may not try to recover it. | 14:23 |
| *** dpawlik has quit IRC | 14:24 | |
| *** armax has joined #openstack-infra | 14:24 | |
| elod | corvus: i don't think it's superimportant :) just looked weird and i didn't know what to do with it :) thanks for the answer! | 14:25 |
| corvus | this is the db record for that change: http://paste.openstack.org/show/754160/ | 14:26 |
| corvus | elod: ^ that look right? | 14:26 |
| elod | corvus: the change_key and change_id looks ok, i mean that's the one | 14:28 |
| corvus | elod: project and branch right? | 14:28 |
| elod | corvus: manila and stable/queens, yes | 14:29 |
| corvus | elod: thanks | 14:29 |
| elod | corvus: thanks, too! | 14:30 |
| *** stevebaker has quit IRC | 14:30 | |
| *** dmellado has quit IRC | 14:30 | |
| *** dmellado has joined #openstack-infra | 14:32 | |
| *** chandankumar is now known as raukadah | 14:35 | |
| *** ricolin has joined #openstack-infra | 14:35 | |
| corvus | https://review.opendev.org/#/c/667721/4 works, patchsets 5 and 6 are in the database and the refs are on disk, their commits are present. they have the same sha, which is curious. i believe ps6 is the merge from zuul, but i don't understand why gerrit would create a new ps with the same sha for a merge. | 14:36 |
| *** stevebaker has joined #openstack-infra | 14:37 | |
| *** bdodd has joined #openstack-infra | 14:38 | |
| corvus | none of the patchsets are drafts | 14:38 |
| corvus | i'm going to stop here and see if fungi or clarkb have any ideas about whether we need to dig further | 14:40 |
| elod | corvus: patch 5 was merged on July 4, but I saw that it's still open in gerrit, so I commented a recheck, which resulted this strange issue :S | 14:40 |
| elod | corvus: ok, thanks again! | 14:40 |
| clarkb | looking at gitea it was only merged once | 14:42 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Don't use -latest nodesets in integration jobs https://review.opendev.org/669675 | 14:45 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Change the repo used in some base-roles tests https://review.opendev.org/669676 | 14:45 |
| *** smarcet has joined #openstack-infra | 14:46 | |
| *** gouthamr has quit IRC | 14:47 | |
| *** josephrsandoval has joined #openstack-infra | 14:48 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Add test job for ensure-sphinx https://review.opendev.org/668958 | 14:49 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Update the sphinx python package sets https://review.opendev.org/668954 | 14:49 |
| *** stevebaker has quit IRC | 14:49 | |
| corvus | cloudnull: ^ we needed a recheck anyway, so i rebased your change on the one that adds the test | 14:49 |
| *** dmellado has quit IRC | 14:49 | |
| *** smarcet has quit IRC | 14:50 | |
| clarkb | infra-root can I get a second review on https://review.opendev.org/#/c/669007/ to finish up the fortnebula mirror work. Currently working with donnyd to diagnose an image upload problem then hopefully we are in a spot to turn on some test nodes | 14:51 |
| *** dmellado has joined #openstack-infra | 14:53 | |
| *** jamesmcarthur has joined #openstack-infra | 14:54 | |
| cloudnull | thanks corvus | 14:55 |
| *** gouthamr has joined #openstack-infra | 14:56 | |
| *** iurygregory_ has joined #openstack-infra | 14:57 | |
| *** iurygregory has quit IRC | 14:57 | |
| *** stevebaker has joined #openstack-infra | 15:01 | |
| *** josephrsandoval has quit IRC | 15:07 | |
| *** dklyle has joined #openstack-infra | 15:13 | |
| *** sgw has joined #openstack-infra | 15:15 | |
| *** bobh has joined #openstack-infra | 15:15 | |
| openstackgerrit | Merged opendev/system-config master: Add apache restart handler for fortnebula LE setup https://review.opendev.org/669007 | 15:17 |
| *** smarcet has joined #openstack-infra | 15:17 | |
| *** gyee has joined #openstack-infra | 15:17 | |
| openstackgerrit | Merged openstack/cookiecutter master: Add a testenv for lower-constraints https://review.opendev.org/660732 | 15:19 |
| *** bobh has quit IRC | 15:20 | |
| *** apetrich has quit IRC | 15:24 | |
| *** dklyle has quit IRC | 15:28 | |
| *** dklyle has joined #openstack-infra | 15:28 | |
| *** iurygregory has joined #openstack-infra | 15:30 | |
| *** rlandy has quit IRC | 15:31 | |
| *** ykarel is now known as ykarel|away | 15:32 | |
| fungi | okay, back from errands, catching up again | 15:32 |
| *** iurygregory_ has quit IRC | 15:32 | |
| openstackgerrit | Clark Boylan proposed openstack/project-config master: Add test nodes in fortnebula cloud https://review.opendev.org/669705 | 15:34 |
| clarkb | images are uploading. If we don't want to approve ^ until images are uploaded I'll happily approve that change then if yall want to review it | 15:35 |
| fungi | corvus: so it looks like we did not see the scheduler memory leak last week while i was away... maybe i'm the cause? ;) | 15:35 |
| AJaeger | fungi, clarkb, could you review the zuul-jobs self-testing stack starting at https://review.opendev.org/#/c/668955 , please? That is work by corvus and updated by me - so, I'm not voting... | 15:35 |
| clarkb | Also I checked and ssh keys and security groups seem to have updated so fixes for cloud launcher seem to have worked | 15:35 |
| *** rlandy_ has joined #openstack-infra | 15:36 | |
| fungi | corvus: as for repl, we can enable it on demand now using the rpc client right? and don't need to do that until we're ready to connect and inspect? | 15:36 |
| *** rlandy_ is now known as rlandy | 15:37 | |
| clarkb | donnyd: https://review.opendev.org/669705 if you want to give that a look over too | 15:37 |
| *** rlandy has quit IRC | 15:38 | |
| corvus | AJaeger: i think you should vote on that stack | 15:39 |
| corvus | fungi: yes | 15:39 |
| fungi | awesome, thanks! | 15:40 |
| donnyd | clarkb: will nodepool just figure out what image to use. All the images that are loaded have what looks like a date attached to the name, but the play only has a generic name | 15:42 |
| AJaeger | corvus: with +2? or just +1? | 15:42 |
| clarkb | donnyd: ya there is a lookup table in a zookeeper database it maintains | 15:42 |
| clarkb | donnyd: so we use logical names in the yaml there but it finds the current actual image for that logical name in the db | 15:43 |
| donnyd | makes perfect sense to me | 15:43 |
| *** rlandy has joined #openstack-infra | 15:44 | |
| *** rlandy is now known as rlandy_ | 15:45 | |
| *** rlandy has joined #openstack-infra | 15:45 | |
| *** tesseract has quit IRC | 15:46 | |
| AJaeger | corvus: want to +2 on https://review.opendev.org/#/c/669147 ? | 15:47 |
| *** tesseract has joined #openstack-infra | 15:48 | |
| *** smarcet_ has joined #openstack-infra | 15:48 | |
| *** smarcet has quit IRC | 15:48 | |
| *** smarcet_ is now known as smarcet | 15:48 | |
| *** lpetrut has quit IRC | 15:50 | |
| *** pgaxatte has quit IRC | 15:53 | |
| *** smarcet has quit IRC | 15:54 | |
| *** kjackal has quit IRC | 15:55 | |
| *** ginopc has quit IRC | 15:57 | |
| clarkb | down to two images remaining to upload for https://review.opendev.org/669705 corvus fungi maybe you can take a look at that change? | 16:04 |
| openstackgerrit | Stephen Finucane proposed openstack/project-config master: Initiate retirement of openstack/docs-specs https://review.opendev.org/668853 | 16:05 |
| openstackgerrit | Stephen Finucane proposed openstack/project-config master: Retire docs-specs https://review.opendev.org/668854 | 16:05 |
| corvus | clarkb: any other fn changes i should look at? | 16:05 |
| clarkb | corvus: I think that is it right now | 16:06 |
| clarkb | the fix for LE apache handler got in already and that was the other one I had in the queue | 16:06 |
| *** iurygregory has quit IRC | 16:07 | |
| *** lucasagomes has quit IRC | 16:09 | |
| *** mattw4 has joined #openstack-infra | 16:09 | |
| *** smarcet has joined #openstack-infra | 16:09 | |
| clarkb | corvus: reading the change AJaeger linked above, https://review.opendev.org/#/c/668955, is the tr cleanup something we have a problem with or just boilerplate from elsewhere? tryign to figure out what the double newline thing is for | 16:10 |
| *** whoami-rajat has quit IRC | 16:10 | |
| fungi | elod: corvus: the situation with 667721 is strange indeed. looks like ps5 was the result of a modification through the webui uncommenting a job name in .zuul.yaml and removing some associated todo comment lines. the timing of that edit is around when our daily database backups happen i think? so this may have been gerrit timing out an update query while waiting on the lock to be released | 16:10 |
| fungi | we probably still have logs from that date, so going to see what i can find in them | 16:11 |
| fungi | oh, this is fun... | 16:12 |
| fungi | [2019-07-04 00:16:08,506] [HTTP-83] ERROR com.google.gerrit.httpd.restapi.RestApiServlet : Error in POST /changes/667721/edit:publish | 16:13 |
| fungi | as usual, the associated java backtrace is a mile long | 16:13 |
| corvus | clarkb: boilerplate from project-config -- ruamel has the entire file too much indented. | 16:14 |
| fungi | but seems like it's bubbling up from in com.google.gwtorm.jdbc.JdbcAccess.update | 16:14 |
| fungi | Caused by: com.google.gwtorm.server.OrmConcurrencyException: Concurrent modification detected | 16:14 |
| fungi | so that does sound like it could indeed be related to db backups | 16:14 |
| corvus | fungi: oh good catch | 16:15 |
| *** whoami-rajat has joined #openstack-infra | 16:15 | |
| openstackgerrit | Merged openstack/project-config master: Add test nodes in fortnebula cloud https://review.opendev.org/669705 | 16:15 |
| clarkb | All images have uploaded so ^ should be all set | 16:16 |
| *** smarcet has quit IRC | 16:16 | |
| elod | fungi: thanks for the investigation! the result sounds interesting :-o | 16:17 |
| *** bhavikdbavishi has quit IRC | 16:17 | |
| donnyd | is that all that we need? just those 13 or so images | 16:17 |
| *** tesseract has quit IRC | 16:18 | |
| *** Lucas_Gray has quit IRC | 16:18 | |
| *** smarcet has joined #openstack-infra | 16:24 | |
| corvus | clarkb: is there a grafana dashboard? | 16:24 |
| clarkb | corvus: not yet | 16:26 |
| clarkb | donnyd: ya that is all of them | 16:26 |
| corvus | clarkb: want me to write that? | 16:26 |
| *** jamesmcarthur has quit IRC | 16:27 | |
| clarkb | corvus: that would be great, thanks | 16:27 |
| clarkb | I'm going to work through this zuul-jobs stack then do a bike ride. Will keep an eye on fortnebula instances too | 16:28 |
| donnyd | enjoy clarkb | 16:29 |
| donnyd | I have my eyes peeled | 16:29 |
| openstackgerrit | James E. Blair proposed openstack/project-config master: grafana: add FortNebula dashboard https://review.opendev.org/669710 | 16:30 |
| *** jamesmcarthur has joined #openstack-infra | 16:30 | |
| *** dtantsur is now known as dtantsur|afk | 16:34 | |
| *** jamesmcarthur has quit IRC | 16:35 | |
| *** igordc has joined #openstack-infra | 16:37 | |
| mnaser | AJaeger: is https://review.opendev.org/#/c/668853/ okay to land now? you had a -1 earlier | 16:45 |
| mnaser | trying to move a stack in openstack/governance which depends on the retirement of this | 16:45 |
| *** rpittau is now known as rpittau|afk | 16:50 | |
| openstackgerrit | James E. Blair proposed zuul/zuul master: WIP: Run jobs when their own config changes https://review.opendev.org/669722 | 16:53 |
| openstackgerrit | Merged zuul/zuul-jobs master: Add a script to make platform-specific versions of jobs https://review.opendev.org/668955 | 16:54 |
| AJaeger | mnaser: that one is fine finally | 16:55 |
| mnaser | AJaeger: yay great | 16:55 |
| mnaser | AJaeger: i' | 16:55 |
| *** psachin has quit IRC | 16:55 | |
| mnaser | i have +2 and i will let you +A then i will recheck to get the +1 noop on the retirement patch | 16:55 |
| mnaser | and chase down some reviews :) | 16:55 |
| *** jamesmcarthur has joined #openstack-infra | 16:56 | |
| AJaeger | mnaser: done... | 16:57 |
| AJaeger | fungi, thanks for reviews - please review https://review.opendev.org/#/c/669147 as final change... | 16:58 |
| fungi | yep, i'm on that one now | 16:59 |
| *** derekh has quit IRC | 17:00 | |
| AJaeger | thanks, fungi | 17:00 |
| fungi | i was just trying to check the dnm triggering change for it first | 17:00 |
| AJaeger | ah | 17:00 |
| fungi | but gertty was crashing on me so took a moment to pull it up | 17:00 |
| AJaeger | fungi: it will merge conflict - I'll rebase once the existing stack is in... | 17:02 |
| openstackgerrit | Merged openstack/project-config master: grafana: add FortNebula dashboard https://review.opendev.org/669710 | 17:03 |
| *** diablo_rojo has joined #openstack-infra | 17:04 | |
| openstackgerrit | Merged openstack/project-config master: Initiate retirement of openstack/docs-specs https://review.opendev.org/668853 | 17:05 |
| openstackgerrit | Andreas Jaeger proposed zuul/zuul-jobs master: Add Gentoo integration tests https://review.opendev.org/669147 | 17:06 |
| AJaeger | fungi, corvus, could you review again, please? ^ | 17:06 |
| * AJaeger had to rebase | 17:06 | |
| *** ykarel|away has quit IRC | 17:06 | |
| *** igordc has quit IRC | 17:10 | |
| openstackgerrit | Merged zuul/zuul-jobs master: Add base role integration jobs https://review.opendev.org/668061 | 17:11 |
| openstackgerrit | Alex Schultz proposed openstack/project-config master: Retire tempest-tripleo-ui https://review.opendev.org/667949 | 17:14 |
| AJaeger | config-core, could you review https://review.opendev.org/#/c/668958 for zuul-jobs testing of sphinx role, please? | 17:14 |
| AJaeger | mnaser: want to review 667949 , please? | 17:14 |
| openstackgerrit | Merged zuul/zuul-jobs master: Add multi-node integration jobs https://review.opendev.org/668767 | 17:15 |
| openstackgerrit | Merged zuul/zuul-jobs master: Don't use -latest nodesets in integration jobs https://review.opendev.org/669675 | 17:15 |
| *** ralonsoh has quit IRC | 17:19 | |
| fungi | corvus: repeatable crash in gertty by passing an integer topic to the search: http://paste.openstack.org/show/754170/ | 17:21 |
| *** priteau has quit IRC | 17:21 | |
| fungi | not sure if that's already a known issue | 17:21 |
| corvus | fungi: neat, didn't know that. | 17:23 |
| fungi | there are a couple newer commits in master than the version i've got installed, but none which look like they would address that | 17:26 |
| *** vankalice has joined #openstack-infra | 17:28 | |
| corvus | fungi: yeah, it repros locally and i'm running master | 17:28 |
| corvus | AJaeger: it turns out the only role left in ozj is mirror-info, and we moved that to opendev/base-jobs, so the real version of that role is elsewhere (and untested, since it's a config-project). | 17:30 |
| corvus | it's the role that creates the /etc/ci/mirror_info.sh script. is that something we want to keep doing in opendev? or is that a legacy-openstack-only thing? | 17:31 |
| fungi | grey area... it's legacy for sure, whether it's openstack-only i'm less certain | 17:32 |
| AJaeger | It looks worthwhile for opendev as well | 17:32 |
| AJaeger | but it's something that not every Zuul user might want | 17:32 |
| corvus | i see 3 options for that: 1) keep doing it for all of opendev, in which case we should move it to an untrusted project in the opendev tenant to regain the testing we had in ozj. 2) make it openstack-only -- in which case we delete it from opendev base jobs, and make a new openstack base job which includes it from ozj; 3) generalize it and support it in zuul-jobs for everyone. | 17:32 |
| *** jamesmcarthur has quit IRC | 17:34 | |
| fungi | i think with the bindep-fallback and fetch-zuul-cloner changes recently we treated "legacy" zuul v2 transitional bits as openstackisms | 17:34 |
| fungi | so i'm leaning toward option 2 there | 17:35 |
| donnyd | clarkb: I see some nodes spinning up fyi | 17:35 |
| corvus | donnyd, clarkb: http://grafana.openstack.org/d/3Bwpi5SZk/nodepool-fortnebula?orgId=1 | 17:35 |
| AJaeger | fungi, corvus, I see mirror support as important - but /etc/ci/mirror_info.sh sounds legacy... | 17:36 |
| AJaeger | sorry, can't think this through completely right now ;/ | 17:36 |
| corvus | yeah, pulling on this thread is going to unravel a bit | 17:37 |
| *** jamesmcarthur has joined #openstack-infra | 17:37 | |
| corvus | donnyd: the graph is showing errors; i'll track down the launcher and see what they are | 17:38 |
| fungi | yes, we used to have nodepool install that file in the ready script i think? and then added the ansible role during the v3 transition to support the auto-converted jobs which were using it | 17:38 |
| corvus | yeah. i *think* in the long run, we can expect jobs to do whatever configuration they need via ansible variables, so i *think* deprecating it will be viable | 17:39 |
| *** ijw has joined #openstack-infra | 17:39 | |
| *** ijw_ has joined #openstack-infra | 17:40 | |
| donnyd | it looks like all the instances are available | 17:41 |
| *** igordc has joined #openstack-infra | 17:41 | |
| corvus | donnyd, clarkb: http://paste.openstack.org/show/754171/ | 17:41 |
| openstackgerrit | Merged zuul/zuul-jobs master: Change the repo used in some base-roles tests https://review.opendev.org/669676 | 17:42 |
| openstackgerrit | Merged zuul/zuul-jobs master: Add Gentoo integration tests https://review.opendev.org/669147 | 17:42 |
| fungi | http://codesearch.openstack.org/?q=mirror_info.sh shows it being used by tripleo (quickstart and ci), dib, devstack (optional i think?), openstack-ansible, devstack-gate and some d-g hooks in other projects (legacy i guess), helm's loci image builder, also puppet-openstack-integration and windmill look like they optionally use it | 17:42 |
| *** igordc has quit IRC | 17:43 | |
| *** weifan has joined #openstack-infra | 17:44 | |
| *** ijw has quit IRC | 17:44 | |
| *** roman_g has quit IRC | 17:45 | |
| donnyd | corvus: I am running it down | 17:46 |
| *** SpamapS has quit IRC | 17:46 | |
| *** igordc has joined #openstack-infra | 17:46 | |
| openstackgerrit | James E. Blair proposed openstack/openstack-zuul-jobs master: Remove base role integration testing https://review.opendev.org/669727 | 17:47 |
| openstackgerrit | James E. Blair proposed openstack/openstack-zuul-jobs master: Remove base role integration testing https://review.opendev.org/669727 | 17:48 |
| donnyd | corvus: clarkb looks like the centos image is not bringing up ipv6 | 17:48 |
| donnyd | I am using SLAAC ATM | 17:48 |
| corvus | donnyd: ah. it does look like nodes for other images are going into ready / in-use states, which further points the finger at something centos-7 specific | 17:49 |
| donnyd | well the fedora image is doing the same. I can swap over ipv6 advertisement methods to see if the images will work another way | 17:50 |
| corvus | (also, i just discovered to much amusement that "gentos" is a dvorak typo for "centos") | 17:50 |
| *** jamesmcarthur has quit IRC | 17:52 | |
| *** SpamapS has joined #openstack-infra | 17:55 | |
| corvus | donnyd: in case you're not aware, we use glean on our images: https://opendev.org/opendev/glean#glean | 17:55 |
| corvus | as long as there's no network info on a config drive, it should get out of the way | 17:55 |
| corvus | but otherwise, it could be a factor | 17:56 |
| clarkb | catching up | 17:58 |
| *** jamesmcarthur has joined #openstack-infra | 17:58 | |
| clarkb | I believe that fedora and centos images rely on router advertisements for ipv6 | 17:58 |
| clarkb | beacuse glean doesn't know how to configure them statically (eg like on rax) | 17:58 |
| *** jamesmcarthur has quit IRC | 17:59 | |
| clarkb | I can boot a !centos test node, ssh into that then ssh into a centos test node via private ipv4 to see what is going on | 17:59 |
| donnyd | I can swap the networking over to dhcpv6-stateless | 18:01 |
| *** ociuhandu_ has joined #openstack-infra | 18:01 | |
| clarkb | logan-: ^ is that what we've got in limestone? | 18:01 |
| logan- | checking | 18:03 |
| logan- | yep https://opendev.org/limestone/ci-cloud-config/src/branch/master/network_bootstrap/bootstrap-neutron.yml#L87-L88 | 18:03 |
| *** ociuhandu has quit IRC | 18:03 | |
| donnyd | That seems like an easier fix to me | 18:04 |
| donnyd | and It would be better if fn was setup like the others best I can | 18:04 |
| *** ociuhandu_ has quit IRC | 18:06 | |
| clarkb | ok I see a problem (my fault). fix on its way | 18:11 |
| clarkb | unrelated to ipv6 | 18:11 |
| fungi | was it stateful dhcp6 in fn previously? | 18:13 |
| openstackgerrit | Clark Boylan proposed openstack/project-config master: Use opendev mirror in fortnebula cloud https://review.opendev.org/669733 | 18:13 |
| donnyd | Well I swapped it out anyways | 18:14 |
| *** jamesmcarthur has joined #openstack-infra | 18:14 | |
| clarkb | fungi: we didn't use it previously | 18:14 |
| donnyd | fungi: no it was SLAAC | 18:14 |
| clarkb | fungi: and ubuntu handle all the cases with glean just fine iirc | 18:14 |
| *** jamesmcarthur has quit IRC | 18:14 | |
| clarkb | infra-root https://review.opendev.org/669733 is needed to make jobs work in fn (a thing I missed with the mirror setup in zuul) | 18:14 |
| donnyd | its dhcpv6-stateless now | 18:14 |
| AJaeger | clarkb, fungi, could either of your review https://review.opendev.org/668958 , please? | 18:14 |
| fungi | ahh | 18:14 |
| *** jamesmcarthur has joined #openstack-infra | 18:15 | |
| clarkb | AJaeger: looking | 18:15 |
| fungi | well, stateless dhcp6 still relies on slaac | 18:15 |
| AJaeger | thanks, clarkb. btw. +2 on 733 | 18:15 |
| fungi | it's just a means of handling dns servers and the like | 18:15 |
| donnyd | fungi: understood, but its what limestone is using. | 18:16 |
| donnyd | so might as well be consistent | 18:16 |
| fungi | sure, not disagreeing | 18:16 |
| clarkb | fungi: donnyd ya neutron's terms for things don't always necessarily map to what you expect them either | 18:16 |
| fungi | just curious what makes that problematic for centos/fedora | 18:16 |
| clarkb | so it could be that dhcpv6 stateless does something unexecptedly different | 18:16 |
| fungi | oh, right, i forgot neutron has its own definitions for those terms ;) | 18:17 |
| clarkb | fungi: the difference on our images is that glean does not configure ipv6 via config drive data | 18:17 |
| AJaeger | config-core, if anybody wants to review, please look at https://review.opendev.org/667949 https://review.opendev.org/665910 and https://review.opendev.org/668708 | 18:17 |
| clarkb | fungi: when on red hat distros | 18:17 |
| clarkb | fungi: its possible that slaac comes with config drive data and dhcpv6 stateless does not | 18:17 |
| clarkb | because that data is expected from dhcp in that config | 18:17 |
| fungi | got it | 18:17 |
| logan- | originally limestone was set up with dhcpv6-stateful and it was causing problems in the nodes https://opendev.org/limestone/ci-cloud-config/commit/d59ba2a74d4661d0868ca43d9f6744b91c8d06ad -- i don't remember the details though | 18:17 |
| *** witek has quit IRC | 18:20 | |
| *** whoami-rajat has quit IRC | 18:20 | |
| yoctozepto | gerrit masters, could you take a look at why url in this commit message displays weirdly in gerrit summary: https://review.opendev.org/#/c/669730/1 | 18:21 |
| clarkb | logan-: ya I think it came down to different neutron behaviors interacting with glean | 18:21 |
| *** priteau has joined #openstack-infra | 18:22 | |
| *** tdasilva has joined #openstack-infra | 18:23 | |
| *** vankalice has quit IRC | 18:23 | |
| donnyd | looks like the gentoo image is having issues on ipv6 as well | 18:29 |
| clarkb | yoctozepto: our gerrit config injects url links for changeids | 18:29 |
| openstackgerrit | Merged zuul/zuul-jobs master: Add test job for ensure-sphinx https://review.opendev.org/668958 | 18:30 |
| clarkb | yoctozepto: if you replace that gerrit url with just the cahnge id or a url to a change number and not id the weird render should go away | 18:30 |
| yoctozepto | clarkb: thanks for explanation! | 18:30 |
| *** weifan has quit IRC | 18:32 | |
| *** weifan has joined #openstack-infra | 18:32 | |
| *** weifan has quit IRC | 18:32 | |
| *** weifan has joined #openstack-infra | 18:32 | |
| *** weifan has quit IRC | 18:34 | |
| *** weifan has joined #openstack-infra | 18:34 | |
| openstackgerrit | Merged zuul/zuul-jobs master: Update the sphinx python package sets https://review.opendev.org/668954 | 18:34 |
| openstackgerrit | Merged openstack/project-config master: Use opendev mirror in fortnebula cloud https://review.opendev.org/669733 | 18:34 |
| *** weifan has quit IRC | 18:35 | |
| *** weifan has joined #openstack-infra | 18:35 | |
| *** weifan has quit IRC | 18:36 | |
| *** weifan has joined #openstack-infra | 18:36 | |
| *** weifan has quit IRC | 18:36 | |
| *** weifan has joined #openstack-infra | 18:37 | |
| *** weifan has quit IRC | 18:37 | |
| *** weifan has joined #openstack-infra | 18:37 | |
| *** weifan has quit IRC | 18:38 | |
| *** ijw_ has quit IRC | 18:43 | |
| *** ijw has joined #openstack-infra | 18:45 | |
| *** sshnaidm|ruck is now known as sshnaidm|afk | 18:46 | |
| *** ociuhandu has joined #openstack-infra | 18:47 | |
| *** weifan has joined #openstack-infra | 18:49 | |
| donnyd | seems like fedora is still not working on dhcpv6-stateless | 18:51 |
| *** ijw has quit IRC | 18:52 | |
| *** whoami-rajat has joined #openstack-infra | 18:57 | |
| clarkb | I'll boot a couple test nodes (ubuntu to ipv6 in and cetnos to examine?) | 18:58 |
| clarkb | donnyd: Quota exceeded for cores: Requested 8, but already used 64 of 64 cores (HTTP 403) (Request-ID: req-6e9ed64a-c33d-41ba-843a-b62c51611aa3) | 19:00 |
| clarkb | donnyd: we'll need to bump that up. cpus, ram, instances, ports and disk | 19:00 |
| donnyd | clarkb: done | 19:01 |
| donnyd | i removed quota for everything that isn't cpu's, instances and memory | 19:02 |
| clarkb | now hitting the core limit of 8 (I need 10 * 8 for nodepool + 2 * 8 for my test nodes = 96) | 19:04 |
| clarkb | alright two test nodes building. Lets see how they do | 19:05 |
| *** weifan has quit IRC | 19:11 | |
| openstackgerrit | Jeff Liu proposed zuul/zuul-operator master: Create tasks that spin up kubernetes and run the operator https://review.opendev.org/668029 | 19:12 |
| donnyd | hrm... seems like the bionic instance is stuck | 19:15 |
| clarkb | donnyd: should I delete it and try again? | 19:15 |
| clarkb | or maybe leave it and try again (so that it can be debugged)? | 19:16 |
| donnyd | seems like something a little deeper is going on | 19:16 |
| openstackgerrit | Brian Haley proposed opendev/irc-meetings master: Create a meeting for Networking OVN project https://review.opendev.org/668013 | 19:19 |
| *** ricolin has quit IRC | 19:21 | |
| *** yamamoto has joined #openstack-infra | 19:22 | |
| *** cloudnull is now known as cloudkiller | 19:25 | |
| *** yamamoto has quit IRC | 19:26 | |
| clarkb | donnyd: I'm gonna pop out for a few but will leave the test ndoe taht is stuck as is so you can poke at it | 19:29 |
| donnyd | ok thanks | 19:29 |
| clarkb | let me know if you think I should boot a second test node or if I should just wait | 19:29 |
| *** factor has quit IRC | 19:30 | |
| donnyd | You can just leave it | 19:30 |
| *** factor has joined #openstack-infra | 19:30 | |
| *** jtomasek has joined #openstack-infra | 19:31 | |
| openstackgerrit | Jeff Liu proposed zuul/zuul-operator master: Create tasks that spin up kubernetes and run the operator https://review.opendev.org/668029 | 19:33 |
| *** icarusfactor has joined #openstack-infra | 19:45 | |
| *** eharney has quit IRC | 19:46 | |
| *** factor has quit IRC | 19:47 | |
| *** pkopec_ has joined #openstack-infra | 19:49 | |
| *** weifan has joined #openstack-infra | 19:50 | |
| *** pkopec has quit IRC | 19:51 | |
| *** ijw has joined #openstack-infra | 19:56 | |
| donnyd | clarkb: seems like the only thing that is working via ipv6 is bionic and xenial | 19:56 |
| fungi | not even debian? | 19:57 |
| donnyd | doesn't look like it | 19:57 |
| fungi | strange | 19:57 |
| fungi | bionic and buster should be fairly similar in the parts that would matter to boot and glean | 19:58 |
| clarkb | ya I've got another test instance booting so that I can poke at the centos node | 19:58 |
| clarkb | I think we'll just ahve to see what is going on | 19:58 |
| fungi | yeah, does seem like this is deeper than just glean and ipv6 | 19:59 |
| *** factor has joined #openstack-infra | 20:01 | |
| *** ociuhandu has quit IRC | 20:02 | |
| *** icarusfactor has quit IRC | 20:03 | |
| donnyd | lmk what you find out clarkb | 20:03 |
| clarkb | donnyd: fungi 2001:470:e045:1:a0ac:2bb:a371:7c3 seems to be the addr it wants to use (generated from mac addr with your prefix I think) | 20:05 |
| clarkb | that is what ifconfig reports at least | 20:05 |
| clarkb | now to check itnerface configs | 20:05 |
| clarkb | er rather the addr in neutron is based on the mac addr with prefix I think | 20:07 |
| clarkb | the address above does not seem to be? | 20:07 |
| *** jcoufal has quit IRC | 20:08 | |
| donnyd | what mac is it showing? | 20:10 |
| openstackgerrit | Jeff Liu proposed zuul/zuul-operator master: Create tasks that spin up kubernetes and run the operator https://review.opendev.org/668029 | 20:11 |
| clarkb | donnyd: ether fa:16:3e:f0:ba:85 | 20:12 |
| clarkb | inet6 2001:4800:7817:103:be76:4eff:fe04:5a1d prefixlen 64 scopeid 0x0<global> | 20:12 |
| clarkb | oh interesting that address keeps changing | 20:13 |
| donnyd | that is the link-local address isn | 20:13 |
| clarkb | wait | 20:13 |
| donnyd | isn't it | 20:13 |
| clarkb | wrong terminal and host | 20:13 |
| clarkb | inet6 2001:470:e045:1:a0ac:2bb:a371:7c3 prefixlen 64 scopeid 0x0<global> | 20:13 |
| clarkb | there | 20:13 |
| clarkb | the mac addr matches the prefix + mac based address from neutron | 20:13 |
| donnyd | can you give me the output from ``` ip a``` | 20:14 |
| clarkb | hrm paste is not working? | 20:16 |
| clarkb | donnyd: https://gist.github.com/cboylan/ebaf1104fce9ae666266871558b6eff9 | 20:17 |
| clarkb | I'm digging through logs now to see if I can find what assigned the address it has | 20:17 |
| donnyd | 2001:470:e045:1:f816:3eff:fef0:ba85 | 20:17 |
| donnyd | no idea where this little fella is coming from 2001:470:e045:1:a0ac:2bb:a371:7c3/64 | 20:18 |
| clarkb | I'm going to try restarting networking to see if that changes anything | 20:20 |
| donnyd | https://www.irccloud.com/pastebin/hz2mbZRq/ | 20:21 |
| donnyd | https://www.irccloud.com/pastebin/a01IAmg7/ | 20:21 |
| clarkb | donnyd: this is running with network manager so I'm not sure the dhcpv6 config helps? | 20:21 |
| donnyd | is NM set to yes or no? | 20:21 |
| clarkb | yes | 20:22 |
| donnyd | hrm | 20:24 |
| *** icarusfactor has joined #openstack-infra | 20:25 | |
| fungi | that looks like a discrepancy between traditional eui-64 slaac addressing and "privacy extensions" addressing | 20:25 |
| fungi | ff:fe as the middle bytes of the host half of the address indicate eui-64 | 20:26 |
| clarkb | fungi: thats the wrong address (I got it off bridge accidentally) | 20:26 |
| fungi | ahh | 20:26 |
| *** factor has quit IRC | 20:27 | |
| clarkb | restarting the network caused it to come up with the correct network address | 20:28 |
| donnyd | did you put the params in from above? | 20:30 |
| clarkb | donnyd: no | 20:30 |
| donnyd | or just restart the network | 20:30 |
| clarkb | just systemctl restart network | 20:30 |
| clarkb | possibly a race with glean and network manager then | 20:30 |
| donnyd | well that restarts network, not NetworkManager | 20:30 |
| clarkb | it should restart network manager too doesn't it? | 20:31 |
| clarkb | since network manager provides network? | 20:31 |
| donnyd | the network service and NetworkManager are not the same, but I could be mistaken | 20:31 |
| donnyd | can you burn that instance and try doing just a network manager restart? | 20:32 |
| clarkb | yup | 20:32 |
| *** Lucas_Gray has joined #openstack-infra | 20:32 | |
| *** smarcet has quit IRC | 20:34 | |
| fungi | for centos i think the network service feeds the /etc/network-scripts/ifcfg-foo files into ifconfig (or more likely iproute2 these days) | 20:34 |
| clarkb | fungi: ya /etc/sysconfig/network-scripts/* | 20:35 |
| clarkb | whcih is where things were configured and I thought networkmanager also read that config | 20:35 |
| fungi | seems like network-manager knows how to read those files, but i agree restarting one doesn't necessarily notify the other | 20:35 |
| clarkb | we should be able to restart network manager easily enough in a few minutes | 20:35 |
| openstackgerrit | Jeff Liu proposed zuul/zuul-operator master: Create tasks that spin up kubernetes and run the operator https://review.opendev.org/668029 | 20:35 |
| clarkb | restarting network manager does not fix it | 20:38 |
| *** rlandy has quit IRC | 20:39 | |
| clarkb | This makes me wonder if network manager just doesn't work right with glean | 20:39 |
| *** rlandy_ is now known as rlandy | 20:39 | |
| fungi | i wouldn't be shocked to learn that's the case | 20:40 |
| *** ianychoi has joined #openstack-infra | 20:41 | |
| clarkb | do we need IPV6_AUTOCONF=yes ? | 20:42 |
| *** priteau has quit IRC | 20:44 | |
| fungi | i find references to setting IPV6_AUTOCONF=no for disabling slaac addressing | 20:45 |
| fungi | suggesting that it's likely defaulting to yes | 20:46 |
| donnyd | IPV6_AUTOCONF = Enable IPv6 autoconf configuration for this interface (an IPv6 address will be requested using Neighbor Discovery (ND) | 20:46 |
| clarkb | I added that config to the ifcfg-eth0 file and restarted network manager to no change. Then rebooted (I think restarting network wouldve been fine) | 20:46 |
| clarkb | and that updated the address | 20:46 |
| clarkb | I think network manager is just oblivious to ipv6 | 20:46 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: WIP: Run jobs when their own config changes https://review.opendev.org/669752 | 20:46 |
| clarkb | I'm not sure what the intended mode of operation is here :/ | 20:47 |
| clarkb | personally I've always used nm and /etc/network/interfaces type config separately | 20:48 |
| fungi | it may be that nm expects ipv6 to "just work" so ignores related settings | 20:48 |
| clarkb | and use only one or the other | 20:48 |
| donnyd | https://developer.gnome.org/NetworkManager/stable/settings-ipv6.html | 20:49 |
| clarkb | if I had to guess at a failure mode it is this: glean and network manager units are configured to happen in the right order but not glean and network. So if nm isn't managing ipv6 the glean written config which includes things like "here is your mac address" fail to configure stuff properly | 20:49 |
| *** priteau has joined #openstack-infra | 20:49 | |
| donnyd | NM_SETTING_IP6_CONFIG_ADDR_GEN_MODE_EUI64 (0) or NM_SETTING_IP6_CONFIG_ADDR_GEN_MODE_STABLE_PRIVACY (1) | 20:49 |
| donnyd | and the default is of course 1 | 20:49 |
| fungi | i honestly have no idea how neutron deals with privacy extensions | 20:50 |
| donnyd | Not sure what the difference is between stable privacy mode and EUI64 other than EUI64 looks to be what SLAAC uses | 20:50 |
| *** pcaruana has quit IRC | 20:50 | |
| donnyd | does this work in limestone? | 20:51 |
| clarkb | donnyd: I don't think we actually know. The assumption is that it worked, btu the NM changes in glean may have broken limestone + red hat distros and we didn't notice | 20:51 |
| slaweq | fungi: hi, if You are asking about IPv6 privacy extension, than neutron don't support it AFAICT | 20:52 |
| fungi | thanks for confirming, slaweq! | 20:52 |
| slaweq | yw fungi :) | 20:52 |
| clarkb | https://developer.gnome.org/NetworkManager/stable/nm-settings-ifcfg-rh.html is the relevant redhat config to nm stuff | 20:53 |
| fungi | yeah, the switch to nm in our images was a very recent one, i think? | 20:53 |
| clarkb | fungi: relatively recent yes | 20:53 |
| donnyd | ``` OpenStack currently doesn’t support the privacy extensions defined by RFC 4941. The interface identifier and DUID used must be directly derived from the MAC as described in RFC 2373. ``` | 20:54 |
| donnyd | is there a way to provide the config that is required to glean? | 20:54 |
| clarkb | donnyd: glean already has that data via config-drive. The larger issue is that no one has written the code that configures interfaces for ipv6 on red hat distros | 20:55 |
| donnyd | Yea, I was a little late on that. one | 20:56 |
| clarkb | so in general we rely on do the right thing behavior of distros with it just working on boot | 20:56 |
| donnyd | it looks to me like the only distro that works as we expect is bionic and xenial though | 20:57 |
| clarkb | reading these docs the default is to do what we want | 20:57 |
| clarkb | with network manager specifically | 20:57 |
| clarkb | donnyd: its possible the other distros have similarly broken | 20:58 |
| donnyd | I also saw the gentoo image does that same | 20:58 |
| clarkb | gentoo relies on systemd-network or whatever its called iirc | 20:59 |
| fungi | systemd-networkd | 20:59 |
| clarkb | xenial and bionic are /etc/network/interfaces (as should be stretch and buster though buster is brand new) | 21:00 |
| *** whoami-rajat has quit IRC | 21:00 | |
| donnyd | the debian image also exhibits the same behavior | 21:00 |
| clarkb | donnyd: is it buster or stretch or both? | 21:00 |
| clarkb | maybe thats the thing to do then. Start checking the other distros and see what the common thread is here /me boots a stretch image | 21:01 |
| *** mattw4 has quit IRC | 21:02 | |
| donnyd | yea, that may be worth while. If it's more common to not work oob, then maybe we should handle it a different way | 21:02 |
| *** mattw4 has joined #openstack-infra | 21:03 | |
| *** nicolasbock has joined #openstack-infra | 21:04 | |
| clarkb | also when ianw wakes he may be able to help debug as he added the NM support to glean | 21:08 |
| clarkb | stretch instance is one its way up | 21:09 |
| openstackgerrit | Brian Haley proposed opendev/irc-meetings master: Create a meeting for Networking OVN project https://review.opendev.org/668013 | 21:12 |
| *** goldyfruit has quit IRC | 21:14 | |
| donnyd | clarkb: I looks like its the same deal as centos | 21:14 |
| donnyd | it* | 21:14 |
| donnyd | also, how are you beating zuul.. I am apparently not fast enough to get an instance scheduled | 21:14 |
| clarkb | donnyd: I had to delete the centos node to make a slot | 21:15 |
| clarkb | which is fine because I already "corrupted" it by rebooting it | 21:16 |
| clarkb | still trying to get into the stretch instance | 21:16 |
| clarkb | it may not be configuring ipv4 there either | 21:16 |
| fungi | seems like we ought to just set max-servers there to 0 until this is figured out? | 21:16 |
| clarkb | fungi: I think the xenial and bionic nodes are working and nodepool won't give zuul a broken centos. So dependso n whether or not it is more valuable to have easier time booting test instances vs zuul instances | 21:17 |
| clarkb | I can go either way on that | 21:17 |
| *** jamesmcarthur has quit IRC | 21:17 | |
| *** smarcet has joined #openstack-infra | 21:21 | |
| clarkb | ok stretch does work | 21:22 |
| clarkb | its just super slow to boot? | 21:22 |
| clarkb | in any case ipv6 there configured properly it just took a while for network (v4 or v6) to come up | 21:23 |
| * clarkb tries another | 21:23 | |
| *** mattw4 has quit IRC | 21:26 | |
| *** priteau has quit IRC | 21:27 | |
| *** mattw4 has joined #openstack-infra | 21:27 | |
| *** priteau has joined #openstack-infra | 21:27 | |
| *** cloudkiller is now known as cloudnull | 21:27 | |
| fungi | that could be other services not finding sufficient entropy at boot | 21:28 |
| *** priteau has quit IRC | 21:29 | |
| clarkb | opensuse 15 image also works with ipv6 | 21:36 |
| *** jtomasek has quit IRC | 21:36 | |
| clarkb | double checking buster now | 21:38 |
| clarkb | then gentoo | 21:38 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Run jobs when their own config changes https://review.opendev.org/669752 | 21:39 |
| *** smarcet has quit IRC | 21:39 | |
| *** sreejithp has quit IRC | 21:41 | |
| clarkb | buster works | 21:42 |
| openstackgerrit | James E. Blair proposed opendev/system-config master: Remove .zuul.yaml file matchers https://review.opendev.org/669762 | 21:44 |
| openstackgerrit | James E. Blair proposed opendev/system-config master: Remove .zuul.yaml file matchers https://review.opendev.org/669762 | 21:48 |
| *** smarcet has joined #openstack-infra | 21:49 | |
| fungi | as slow to boot as stretch? | 21:53 |
| *** mriedem has quit IRC | 21:53 | |
| clarkb | no, but that may be due to image caching too? | 21:57 |
| clarkb | still waiting on gentoo to connect via ipv4 or v6 | 21:58 |
| clarkb | console log shows nothing | 21:58 |
| clarkb | (that may be a bug in our image) | 21:58 |
| fungi | oh, yeah if that image hadn't been booted on that hypervisor host yet | 21:59 |
| clarkb | I'll give gentoo another couple minutes before I move on | 22:04 |
| clarkb | still no connectivity over either ip version and no console log | 22:04 |
| ianw | hello, so basically ipv6 on centos not working with our images on fortnebula? | 22:06 |
| clarkb | ianw: ya | 22:07 |
| clarkb | ianw: and a reboot seems to fix it | 22:07 |
| *** slaweq has quit IRC | 22:07 | |
| clarkb | I'm doing a survey of our other images currently | 22:07 |
| ianw | hrm, yeah ordering is always a suspect ... | 22:08 |
| clarkb | on the off chance gentoo's problem is the image upload I have deleted it and will let nodepool reupload it | 22:09 |
| ianw | i feel deja-vu something pabelanger was looking at | 22:09 |
| ianw | i'm not sure we ever got to the bottom of it | 22:10 |
| *** tjgresha has joined #openstack-infra | 22:10 | |
| clarkb | Booting fedora-29 now | 22:10 |
| clarkb | lets see if it acts like centos | 22:10 |
| *** pkopec__ has joined #openstack-infra | 22:11 | |
| *** pkopec_ has quit IRC | 22:14 | |
| clarkb | ya fedora29 is the same | 22:18 |
| clarkb | comes up with a bad/wrong ipv6 address first boot but on second boot is correct | 22:18 |
| ianw | but a globally addressed bad address (not fe80:: etc?) | 22:19 |
| clarkb | correct | 22:19 |
| clarkb | bad: inet6 2001:470:e045:1:3a36:438:3e36:2e0d good: inet6 2001:470:e045:1:f816:3eff:feca:651 | 22:19 |
| clarkb | is it possible that the mac addr isn't the same? | 22:20 |
| clarkb | I mean the actual ports mac address is different than fa:16:3e:ca:06:51 and then we change it? | 22:20 |
| clarkb | maybe sysfs will tell me | 22:20 |
| *** slaweq has joined #openstack-infra | 22:23 | |
| fungi | the first (bad) one is a privacy extensions address, the second is a eui-64 slaac address | 22:27 |
| fungi | likely something is starting out with privacy extensions addressing turned on | 22:28 |
| *** slaweq has quit IRC | 22:28 | |
| clarkb | fungi: I thought you said the bridge.o.o address was the one that was privacy extensioned? (the one I mispasted) | 22:29 |
| clarkb | fungi: is the bad address above also a privacy extensioned addr? | 22:29 |
| fungi | yes | 22:29 |
| clarkb | ah ok | 22:29 |
| ianw | yeah if i put the "bad" one into http://silmor.de/ipaddrcalc.html#ip6 it doesn't come back with a MAC (::3a36:438:3e36:2e0d) | 22:29 |
| clarkb | in that case why would a reboot change the behavior? | 22:29 |
| fungi | 2001:470:e045:1:3a36:438:3e36:2e0d does not have a ff:fe in the middle of the host portion of the address | 22:29 |
| fungi | so is not eui-64 | 22:30 |
| clarkb | reading fedora bugs seems that NM doesn't have a default for privacy extensions and instead relies on sysctl | 22:31 |
| clarkb | which makes this weirder because we don't change sysctl with glean and a reboot shouldn't change that I don't think? | 22:31 |
| fungi | privacy extensions stable addressing is likely enabled in the kernel by default on those platforms, though not sure why the eui-64 addressing gets added on second boot | 22:31 |
| clarkb | I'm going to look at sysctl on my current working/fixed instance and then boot a new one that should be broken and check that one | 22:32 |
| ianw | afaik glean has never ventured in to this area | 22:33 |
| *** ekultails has quit IRC | 22:34 | |
| *** jamesmcarthur has joined #openstack-infra | 22:34 | |
| clarkb | net.ipv6.conf.ens3.use_tempaddr = 0 | 22:35 |
| fungi | privacy extensions temporary addressing and privacy extensions stable addressing likely use different sysctls | 22:35 |
| fungi | tempaddrs never really caught on because they make network management problematic | 22:36 |
| fungi | which is why stable addressing came about later as more or less a replacement | 22:36 |
| clarkb | fungi: I don't see any sysctls for it otherwise | 22:36 |
| clarkb | also I found a clue | 22:36 |
| clarkb | getting a link | 22:37 |
| *** Lucas_Gray has quit IRC | 22:37 | |
| *** Lucas_Gray has joined #openstack-infra | 22:37 | |
| clarkb | https://opendev.org/openstack/project-config/src/branch/master/nodepool/elements/nodepool-base/install.d/99-disable-rfc3041 | 22:37 |
| clarkb | that may explain why things potentially change after a reboot | 22:37 |
| auristor | privacy addresses should be transient and be flagged as temporary. the rule is that when present, new outgoing connections default to the temporary address, unless the socket is bound to the permanent address | 22:37 |
| clarkb | apparently those sysctls are applied after networking is bootstrapped | 22:38 |
| clarkb | now to boot a fresh node and compare | 22:38 |
| auristor | temporary addresses should not replace permanent addresses. | 22:38 |
| clarkb | auristor: ya I think what fungi is describing are stable permanent addrs not derived from mac addr | 22:39 |
| ianw | https://binblog.info/2017/09/21/ipv6-privacy-stable-addressing-roundup/ good link | 22:40 |
| ianw | Linux -> It's complicated | 22:40 |
| ianw | :) | 22:40 |
| fungi | yep, those | 22:40 |
| fungi | tempaddrs are still relevant for things like mobile devices | 22:41 |
| clarkb | NetworkManager defaults to using addr-gen-mode=stable-privacy in the [ipv6] section of /etc/NetworkManager/system-connections/<Connection>. that may be it then | 22:41 |
| fungi | for servers they're aggravating | 22:41 |
| clarkb | ok where is my link of settings for sysconfig to map into nm | 22:41 |
| clarkb | https://developer.gnome.org/NetworkManager/stable/nm-settings-ifcfg-rh.html | 22:42 |
| clarkb | that says the default is eui64 | 22:42 |
| ianw | clarkb: so the thought is that on first boot ipv6 has come up before nm or something (ordering?) ... then we configure it, then second boot it uses the stable address? | 22:42 |
| clarkb | ianw: ya | 22:43 |
| clarkb | I still don't know how or why though | 22:43 |
| clarkb | one thing I've noticed is that network manager starts before glean, then glean updates the sysconfig interface file whihc network manager notices then updates the interface config | 22:44 |
| clarkb | possible that NM isn't fully updating the interface config on that update (because not everything can be updated?) but then second boot the files are already in place and it does the correct thing then? | 22:45 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Run jobs when their own config changes https://review.opendev.org/669752 | 22:47 |
| clarkb | hrm the new fedora-29 I booted seems to actually have the correct IP | 22:49 |
| clarkb | so a race at boot seems likely | 22:49 |
| ianw | another link -> https://arusso.io/Disabling_IPv6_Privacy_Extensions/ ... same sort of issue. host uses SLAAC and if NM makes up an address it doesn't work | 22:49 |
| clarkb | it does seem like a really bad default for servers | 22:50 |
| ianw | i wonder if a "nmcli conn down/up" cycle as a hack works ... not very nice | 22:51 |
| *** bdodd has quit IRC | 22:51 | |
| auristor | there can be three types of permanent non-link-local addresses: SLAAC+RDNSS and SLAAC+Stateless DHCPv6, and Stateful DHCPv6. It is possible for a host to have more than one of them. | 22:51 |
| clarkb | we are using the second | 22:52 |
| fungi | tempaddrs are annoying for servers. cga (privacy stable) are a fine choice *if* your environment can support them since it hides your mac | 22:52 |
| clarkb | fungi: right but seems like none of these hosted envs support it | 22:53 |
| clarkb | fungi: that blog above isn't even openstack :) | 22:53 |
| fungi | neutron wants to be able to know what address the guest will come up with, so yeah needs something (pre)deterministic | 22:53 |
| *** jamesmcarthur has quit IRC | 22:53 | |
| auristor | you don't want temp addresses on servers unless you know the software that is running knows how to properly bind to permanent addresses and always reply using the same interface the request came in on. | 22:54 |
| fungi | also leaking the mac is less of a concern on virtual machines since they're fabricated and don't reflect the hardware manufacturer et cetera | 22:54 |
| auristor | AuriStorFS servers go to a lot of effort to ensure that we never use or advertise temporary addresses | 22:54 |
| fungi | auristor: in this case they're not tempaddrs, they're cga | 22:55 |
| fungi | so the main problem is that the virtual machine hosting platform can't guess in advance what cga will be generated by the guest on first boot | 22:55 |
| auristor | If SLACC+Stateless DHCPv6 is in use, any chance there is more than one DHCPv6 server ? | 22:55 |
| *** goldyfruit has joined #openstack-infra | 22:56 | |
| fungi | ought to be able to tell from the syslog, though i'm curious how stateless would impact addressing at all | 22:57 |
| clarkb | auristor: I don't think so its just generating the private addr instead of the mac based one | 22:58 |
| clarkb | looking at fedora29 syslog it seems that glean writes the ifcfg-ens3 file before NM processes that interface and it works | 22:58 |
| auristor | for CGA, where / when is the public key being generated? does that happen at after first boot? | 22:58 |
| clarkb | I need to boot fedora-29 instances until I catch a broken one then compare syslogs I guess | 22:58 |
| fungi | networkmanager handles the address generation as a bit of a black box. from what i gather it creates the address each time it sees an interface with a new mac and then stashes that for future use | 23:00 |
| *** jamesmcarthur has joined #openstack-infra | 23:00 | |
| fungi | (not that it embeds the mac in the address, just uses it for indexing and handling hotplug events) | 23:00 |
| *** tkajinam has quit IRC | 23:01 | |
| *** tkajinam has joined #openstack-infra | 23:01 | |
| ianw | from my notes -> http://paste.openstack.org/show/754182/ ... i think that was what i was thinking of with the "reboot fixes it" | 23:01 |
| fungi | anyway, the long and short of it is that the environment these images are booted in can't support cga at all, so we need to turn it off for playforms where it defaults to on | 23:01 |
| fungi | s/playforms/platforms/ | 23:02 |
| fungi | likely this came about with the switch to using networkmanager on those images | 23:02 |
| auristor | as I understand it the benefit of a cga address is that openssh client or server can use it and its possible to prove the client and server's public keys match that used to generate the cga | 23:02 |
| fungi | that was the design goal for that rfc, yes, though i don't know of any actual implementations that do it that way | 23:03 |
| fungi | instead they just generate a random seed and use that to create the addresses | 23:03 |
| fungi | it was supposed to tie in with send (secure neighbor discovery) | 23:03 |
| fungi | but it has mostly ended up being used as a replacement for eui-64 addressing where folks want autoconfiguration but don't want to expose their interface macs | 23:04 |
| fungi | and who want to make it harder to brute-force the address space too | 23:05 |
| auristor | the same chicken and egg problem exists. which come first after initial boot, bringing up the interface or generating the key material? | 23:05 |
| ianw | for reference; i would say that https://review.opendev.org/#/c/618964/9..17/glean/init/glean-nm%2540.service was the ordering change i referenced in the notes | 23:06 |
| *** tosky has quit IRC | 23:06 | |
| ianw | i'm sure i had a good reason for using local-fs in there, although not sure i can remember what it was :/ | 23:06 |
| clarkb | ianw: it has to write the config to disk | 23:07 |
| fungi | i agree it's a bit of a chicken-and-egg problem, but i also have never been concerned with exposing my servers interface macs or having brute-forcible address space in my networks | 23:07 |
| clarkb | and then glean seems to wait for the link to go active | 23:07 |
| clarkb | heh 2/2 working fedora-29 boots now | 23:08 |
| clarkb | donnyd: ^ did you change anything cloud side? or maybe this is just a really weird race and we are gonna hit it somtimes | 23:09 |
| * clarkb tries centos7 again | 23:09 | |
| donnyd | no | 23:09 |
| donnyd | just tinkering with the storage and the air handler | 23:09 |
| ianw | maybe this is also supposed to have a network-* target somewhere? https://opendev.org/opendev/glean/src/branch/master/glean/init/glean-nm@.service | 23:10 |
| ianw | discussing this is like groundhog day ... | 23:10 |
| *** dchen has joined #openstack-infra | 23:11 | |
| *** weifan has quit IRC | 23:12 | |
| ianw | as puppet seems to have shown too, computers may be good at serialising DAG's, but the humans configuring them can very quickly create very difficult to maintain things with them | 23:12 |
| lifeless | ianw: like k8s ? :P | 23:13 |
| ianw | i'm 99% certain that if that "Before=local-fs" in the glean nm .service file gets changed to "Before=network-pre" (which does seem logical) it won't boot in the CI tests; that was the original issue | 23:14 |
| *** andreaf has quit IRC | 23:15 | |
| *** andreaf has joined #openstack-infra | 23:15 | |
| clarkb | yup comparing working fedora-29 syslog to broken centos7 the glean ifcfg-$interface writes happen before NetworkManager edits that interface and it works. Happens after and it fails | 23:18 |
| clarkb | which would explain why a reboot fixes it (beacuse that ifcfg file is already in place on reboot) | 23:18 |
| *** lseki has quit IRC | 23:19 | |
| clarkb | we don't really change any settings in that file that I would expect to affect this though | 23:19 |
| *** goldyfruit has quit IRC | 23:19 | |
| clarkb | oh except that the file may not exist at all on very first boot so NM uses NM defaults and on second boot it uses the sysconfig plugin which depends on those defaults? | 23:20 |
| clarkb | maybe an easy fix here is to write out ifcfg files? (problem with that is biosdevname is a thing now) | 23:20 |
| clarkb | I'm going to bet that that is the difference here. The sysconfig plugin likely tries to be server sane and NM without it is trying to handle the laptop case | 23:21 |
| clarkb | ianw: can we do a Before NetworkManager ? | 23:22 |
| ianw | clarkb: i'm wondering if | 23:23 |
| ianw | Before=network-pre.target | 23:23 |
| ianw | After=local-fs.target | 23:23 |
| clarkb | ah | 23:23 |
| ianw | might be the spot? i'm not sure i tested that combo | 23:23 |
| clarkb | that looks good :) | 23:23 |
| ianw | i think i went straight from "network-pre.target" not working to "local-fs.target" working in -> https://review.opendev.org/#/c/618964/9..17/glean/init/glean-nm%2540.service | 23:23 |
| clarkb | I think I can test that by rm'ing the ifcfg file. update systemd config then reboot | 23:23 |
| * clarkb tries that | 23:24 | |
| ianw | Maybe Before=network-pre.target, Wants=network-pre.target, After=local-fs.target ? | 23:24 |
| clarkb | trying that | 23:24 |
| ianw | it's still got to have that DefaultDependencies=no as well | 23:25 |
| clarkb | I edited the unit in place (can just delete the VM and try again if necessary) | 23:26 |
| clarkb | so didn't change anything but Before and Wants | 23:26 |
| clarkb | ianw: I think that did it | 23:26 |
| clarkb | interface has correct ipv6 addr and I see in syslog that glean ran before the NM config of the interface | 23:27 |
| clarkb | ianw: want to propose that change and see if CI is equially happy with it? | 23:27 |
| clarkb | (it was your idea) | 23:27 |
| ianw | ... not sure i'd take credit for it :) | 23:28 |
| clarkb | I'm happy to push a change up and start some testing too | 23:29 |
| * clarkb does this | 23:29 | |
| openstackgerrit | Clark Boylan proposed opendev/glean master: Update NM unitfile to run after fs and before NM https://review.opendev.org/669772 | 23:31 |
| clarkb | something like that | 23:31 |
| clarkb | now booting gentoo again with new image upload to see if the issue there was a bad image | 23:33 |
| *** weifan has joined #openstack-infra | 23:35 | |
| *** rcernin has joined #openstack-infra | 23:35 | |
| *** rlandy has quit IRC | 23:36 | |
| openstackgerrit | Ian Wienand proposed opendev/glean master: network-manager: add network-pre dependencies https://review.opendev.org/669773 | 23:36 |
| ianw | oh heh, great minds :) | 23:36 |
| clarkb | your commit message is much better than mine (this is good though will get us twice as many test results) | 23:36 |
| *** jamesmcarthur has quit IRC | 23:37 | |
| ianw | yeah, let's let them run and make sure we get green | 23:37 |
| ianw | with hindsight, this ordering now seems logical. but i've thought the ordering was correct many times before :) | 23:38 |
| *** smarcet has quit IRC | 23:39 | |
| *** pkopec_ has joined #openstack-infra | 23:40 | |
| *** gyee has quit IRC | 23:42 | |
| *** pkopec__ has quit IRC | 23:42 | |
| clarkb | ok confirmed that gentoo doesn't work either. glean is not running for any interfaces there | 23:43 |
| clarkb | ah ok it may not run per interface like on other distros | 23:44 |
| clarkb | it may be racing the growfs | 23:45 |
| clarkb | growfs completes after glean (though not much after) | 23:45 |
| *** mattw4 has quit IRC | 23:45 | |
| clarkb | prometheanfire: ^ any ideas? the logging isn't great (I don't see any logs from glean other than mounting the config drive) and there is no net file in /etc/conf.d/net | 23:47 |
| clarkb | if the writes failed I would've expected tracebacks in journalctl -u glean though and don't have that | 23:47 |
| openstackgerrit | Nate Johnston proposed opendev/irc-meetings master: Capture artifacts from ical generation https://review.opendev.org/669775 | 23:50 |
| clarkb | I'll leave the gentoo isntance up in case prometheanfire is around and able to debug at some point | 23:50 |
| clarkb | but other than waiting for ci results I think this is good | 23:51 |
| *** michael-beaver has quit IRC | 23:53 | |
| prometheanfire | hi? | 23:54 |
| *** weifan has quit IRC | 23:54 | |
| *** sthussey has quit IRC | 23:54 | |
| prometheanfire | so glean is having a problem with growfs? | 23:54 |
| prometheanfire | also, gentoo systemd or basic system? | 23:55 |
| *** gyee has joined #openstack-infra | 23:57 | |
| *** jamesmcarthur has joined #openstack-infra | 23:57 | |
| clarkb | prometheanfire: gentoo systemd is not configuring networking | 23:57 |
| clarkb | with glean | 23:57 |
| clarkb | there is no net.ens3 file in /etc/conf.d | 23:58 |
| prometheanfire | it shouldn't be | 23:58 |
| donnyd | clarkb: so are we cooking with gas now? | 23:58 |
| clarkb | (so one idea I had is fs/growroot related) | 23:58 |
| prometheanfire | it should be setting things up in /etc/systemd/network | 23:58 |
| clarkb | prometheanfire: oh where is the config then? | 23:58 |
| prometheanfire | gentoo-systemd uses networkd | 23:58 |
| clarkb | ok I must've misread the glean code then | 23:58 |
| clarkb | oh I see | 23:58 |
| clarkb | two different paths | 23:58 |
| prometheanfire | yep | 23:59 |
| clarkb | donnyd: ya we think fix is up and getting tested now | 23:59 |
| clarkb | prometheanfire: ok I'll debug that later then. dinner prep now | 23:59 |
| prometheanfire | :D | 23:59 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!