| clarkb | or just rm the file and let the cron sweep through | 00:00 |
|---|---|---|
| clarkb | zm* and zuul01 are in similar situation (I think zuul01 was legit using that until we did the base.yaml split out and took zuul out of the git playbook) | 00:00 |
| *** ianychoi has joined #openstack-infra | 00:00 | |
| ianw | yeah, i guess at least if we see this again it won't be a total surprise | 00:01 |
| clarkb | http://mirror.dfw.rax.opendev.org/ubuntu/ exists now too | 00:02 |
| fungi | solace in small victories? ;) | 00:06 |
| fungi | i finished mowing the lawn after giving up halfway through yesterday. i'll take comfort in thaht | 00:06 |
| *** hwoarang has quit IRC | 00:09 | |
| *** hwoarang has joined #openstack-infra | 00:10 | |
| *** weifan has quit IRC | 00:11 | |
| *** weifan has joined #openstack-infra | 00:11 | |
| ianw | ok ... ansible -v -f 10 'z*' -m shell -a "rm /etc/facter/facts.d/ansible.json" look ok? | 00:14 |
| *** weifan has quit IRC | 00:15 | |
| *** lseki has quit IRC | 00:18 | |
| ianw | well, it's done ... so i think that should mean opendev mirror is full operational | 00:19 |
| ianw | proxy_80_access.log is probably the wrong name for that log file | 00:20 |
| fungi | if we grow the list of replacement opendev mirrors incrementally, mind the case sensitivity on the conditional check | 00:21 |
| fungi | i just corrected the case on the region name since it wasn't immediately obvious to me how to make that an case-insensitive comparison | 00:21 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: mirror: rename 80/443 log files https://review.opendev.org/662357 | 00:23 |
| ianw | fungi: yeah, that one had never occurred to me | 00:24 |
| *** rkukura has quit IRC | 00:32 | |
| *** rkukura has joined #openstack-infra | 00:32 | |
| *** hwoarang has quit IRC | 00:32 | |
| *** hwoarang has joined #openstack-infra | 00:33 | |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: executor: run cleanup playbook on stop https://review.opendev.org/661881 | 00:33 |
| *** Lucas_Gray has quit IRC | 00:34 | |
| *** rkukura has quit IRC | 00:42 | |
| *** hwoarang has quit IRC | 00:48 | |
| *** hwoarang has joined #openstack-infra | 00:49 | |
| openstackgerrit | Paul Belanger proposed zuul/zuul master: Discard GitHub PullReview if incomplete https://review.opendev.org/662347 | 00:55 |
| *** mriedem_away has quit IRC | 00:57 | |
| *** hwoarang has quit IRC | 01:15 | |
| *** yamamoto has joined #openstack-infra | 01:16 | |
| *** hwoarang has joined #openstack-infra | 01:23 | |
| ianw | AJaeger: so https://gerrit-dash-creator.readthedocs.io/en/latest/dashboards/index.html is back .. i had to go into the settings in RTD and flip it to use python3 | 01:25 |
| ianw | i'm not sure about the style though? the old page used the old openstack theme, now it's a generic rtd thing with an ad | 01:26 |
| ianw | oh doh, you referenced that in the CL ... all good | 01:31 |
| *** hwoarang has quit IRC | 01:35 | |
| ianw | AJager: so i guess the only thing for us to note is that we can flip the RTD python building version by adding a config file and "python:" to make sure we use 3.x -- https://docs.readthedocs.io/en/stable/config-file/v2.html | 01:35 |
| *** hwoarang has joined #openstack-infra | 01:35 | |
| ianw | or, someone with access can flip it on the settings page, but the config file seems better overall | 01:35 |
| *** michael-beaver has quit IRC | 01:40 | |
| *** gyee has quit IRC | 01:42 | |
| *** hwoarang has quit IRC | 01:57 | |
| *** hwoarang has joined #openstack-infra | 01:57 | |
| *** apetrich has quit IRC | 01:57 | |
| *** whoami-rajat has joined #openstack-infra | 01:57 | |
| *** igordc has quit IRC | 01:58 | |
| *** hwoarang has quit IRC | 02:08 | |
| *** hwoarang has joined #openstack-infra | 02:09 | |
| *** markvoelker has joined #openstack-infra | 02:10 | |
| *** markvoelker has quit IRC | 02:14 | |
| *** bnemec has quit IRC | 02:18 | |
| *** bnemec has joined #openstack-infra | 02:29 | |
| *** bobh has joined #openstack-infra | 02:34 | |
| *** gregoryo has joined #openstack-infra | 02:38 | |
| *** bobh has quit IRC | 02:39 | |
| *** hwoarang has quit IRC | 02:49 | |
| *** hwoarang has joined #openstack-infra | 02:49 | |
| *** ricolin has joined #openstack-infra | 02:54 | |
| *** hwoarang has quit IRC | 02:58 | |
| *** hwoarang has joined #openstack-infra | 02:59 | |
| *** nicolasbock has quit IRC | 02:59 | |
| *** hwoarang has quit IRC | 03:07 | |
| *** hwoarang has joined #openstack-infra | 03:07 | |
| *** dave-mccowan has quit IRC | 03:08 | |
| *** bnemec has quit IRC | 03:11 | |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: docs: add cleanup-run documentation https://review.opendev.org/662147 | 03:18 |
| *** hwoarang has quit IRC | 03:27 | |
| *** hwoarang has joined #openstack-infra | 03:28 | |
| *** bnemec has joined #openstack-infra | 03:29 | |
| *** ykarel has joined #openstack-infra | 03:52 | |
| *** Lucas_Gray has joined #openstack-infra | 04:09 | |
| *** Lucas_Gray has quit IRC | 04:16 | |
| *** Lucas_Gray has joined #openstack-infra | 04:21 | |
| *** toabctl has joined #openstack-infra | 04:41 | |
| openstackgerrit | Merged openstack/diskimage-builder master: allow the use of non-bzip compressed stages for building gentoo https://review.opendev.org/661098 | 04:42 |
| *** hwoarang has quit IRC | 04:45 | |
| *** hwoarang has joined #openstack-infra | 04:45 | |
| *** raukadah is now known as chandankumar | 04:51 | |
| *** raissa has quit IRC | 04:56 | |
| *** kjackal has quit IRC | 05:01 | |
| *** tkajinam has quit IRC | 05:03 | |
| *** pcaruana has joined #openstack-infra | 05:03 | |
| *** tkajinam has joined #openstack-infra | 05:03 | |
| *** tkajinam has quit IRC | 05:04 | |
| *** ricolin has quit IRC | 05:05 | |
| *** igordc has joined #openstack-infra | 05:06 | |
| *** tkajinam has joined #openstack-infra | 05:07 | |
| *** rcernin has quit IRC | 05:07 | |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: executor: run cleanup playbook on stop https://review.opendev.org/661881 | 05:19 |
| *** dpawlik has joined #openstack-infra | 05:25 | |
| *** hwoarang has quit IRC | 05:25 | |
| *** hwoarang has joined #openstack-infra | 05:26 | |
| *** Qiming has joined #openstack-infra | 05:26 | |
| *** ramishra_ has quit IRC | 05:26 | |
| *** quiquell has joined #openstack-infra | 05:31 | |
| *** tkajinam_ has joined #openstack-infra | 05:33 | |
| *** hwoarang has quit IRC | 05:34 | |
| *** hwoarang has joined #openstack-infra | 05:34 | |
| *** tkajinam has quit IRC | 05:36 | |
| *** ramishra has joined #openstack-infra | 05:42 | |
| *** kjackal has joined #openstack-infra | 05:42 | |
| *** Lucas_Gray has quit IRC | 05:44 | |
| *** hwoarang has quit IRC | 05:44 | |
| *** hwoarang has joined #openstack-infra | 05:44 | |
| *** ykarel is now known as ykarel|afk | 05:47 | |
| *** dtantsur|afk is now known as dtantsur | 05:52 | |
| *** igordc has quit IRC | 05:52 | |
| *** hwoarang has quit IRC | 05:58 | |
| *** hwoarang has joined #openstack-infra | 05:59 | |
| *** yamamoto has quit IRC | 06:01 | |
| *** lpetrut has joined #openstack-infra | 06:03 | |
| *** dpawlik has quit IRC | 06:08 | |
| *** dpawlik has joined #openstack-infra | 06:08 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: fail early when lates build information can not be fetched https://review.opendev.org/657892 | 06:09 |
| *** kopecmartin|off is now known as kopecmartin | 06:09 | |
| *** rcernin has joined #openstack-infra | 06:10 | |
| AJaeger | ianw: thanks for fixing https://gerrit-dash-creator.readthedocs.io/ ! | 06:12 |
| *** gregoryo has quit IRC | 06:20 | |
| AJaeger | config-core, could you review https://review.opendev.org/662156 to add upper-constraints directly to post docs jobs, please? | 06:21 |
| AJaeger | I saw clarkb's earlier note - let's be careful what to merge to not create conflicts with the rename changes... | 06:21 |
| openstackgerrit | Merged openstack/diskimage-builder master: bindep: add sudo https://review.opendev.org/635913 | 06:23 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Makes image caching more resilient https://review.opendev.org/656478 | 06:31 |
| *** xek_ has joined #openstack-infra | 06:34 | |
| *** yamamoto has joined #openstack-infra | 06:36 | |
| *** e0ne has joined #openstack-infra | 06:36 | |
| *** yamamoto has quit IRC | 06:41 | |
| *** yamamoto has joined #openstack-infra | 06:41 | |
| *** dpawlik has quit IRC | 06:43 | |
| *** bobh has joined #openstack-infra | 06:45 | |
| *** ramishra has quit IRC | 06:47 | |
| *** armax has quit IRC | 06:49 | |
| *** ramishra has joined #openstack-infra | 06:49 | |
| *** bobh has quit IRC | 06:50 | |
| *** yamamoto has quit IRC | 06:51 | |
| *** ykarel|afk has quit IRC | 06:52 | |
| *** ykarel has joined #openstack-infra | 06:54 | |
| *** apetrich has joined #openstack-infra | 06:57 | |
| *** yamamoto has joined #openstack-infra | 06:59 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: bindep: exclude zypper from debian-stretch https://review.opendev.org/635910 | 07:02 |
| *** ginopc has joined #openstack-infra | 07:07 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Increase size of EFI system partition (again) https://review.opendev.org/641702 | 07:09 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Increase size of EFI system partition (again) https://review.opendev.org/641702 | 07:10 |
| *** tesseract has joined #openstack-infra | 07:12 | |
| *** yamamoto has quit IRC | 07:18 | |
| *** ccamacho has quit IRC | 07:24 | |
| *** yamamoto has joined #openstack-infra | 07:24 | |
| *** ccamacho has joined #openstack-infra | 07:24 | |
| *** roman_g has joined #openstack-infra | 07:25 | |
| *** dchen has quit IRC | 07:26 | |
| *** tosky has joined #openstack-infra | 07:33 | |
| *** tkajinam_ has quit IRC | 07:42 | |
| *** dpawlik has joined #openstack-infra | 07:46 | |
| *** jpich has joined #openstack-infra | 07:48 | |
| *** iurygregory has joined #openstack-infra | 07:50 | |
| *** adriancz has joined #openstack-infra | 07:52 | |
| *** rcernin has quit IRC | 07:52 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Add version-less RHEL element for RHEL7 and RHEL8 https://review.opendev.org/643731 | 07:56 |
| openstackgerrit | Merged openstack/diskimage-builder master: Deprecate rhel7 in favor of rhel https://review.opendev.org/647710 | 08:01 |
| *** happyhemant has joined #openstack-infra | 08:05 | |
| *** kjackal has quit IRC | 08:06 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Add option to skip update packages https://review.opendev.org/636060 | 08:13 |
| *** pkopec has joined #openstack-infra | 08:15 | |
| *** lucasagomes has joined #openstack-infra | 08:19 | |
| *** Lucas_Gray has joined #openstack-infra | 08:20 | |
| openstackgerrit | Slawek Kaplonski proposed openstack/openstack-zuul-jobs master: Add neutron-tempest-plugin to bgpvpn-bagpipe job https://review.opendev.org/662402 | 08:33 |
| *** ginopc has quit IRC | 08:34 | |
| *** derekh has joined #openstack-infra | 08:40 | |
| *** janki has joined #openstack-infra | 08:42 | |
| *** kjackal has joined #openstack-infra | 08:44 | |
| openstackgerrit | Merged openstack/diskimage-builder master: fail early when lates build information can not be fetched https://review.opendev.org/657892 | 08:45 |
| *** ramishra has quit IRC | 08:46 | |
| *** ramishra has joined #openstack-infra | 08:46 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Makes image caching more resilient https://review.opendev.org/656478 | 08:54 |
| openstackgerrit | Merged openstack/diskimage-builder master: Increase size of EFI system partition (again) https://review.opendev.org/641702 | 09:05 |
| *** ykarel is now known as ykarel|lunch | 09:07 | |
| openstackgerrit | Merged openstack/diskimage-builder master: bindep: exclude zypper from debian-stretch https://review.opendev.org/635910 | 09:10 |
| openstackgerrit | Mark Meyer proposed zuul/zuul master: Extend event reporting https://review.opendev.org/662134 | 09:24 |
| *** panda is now known as panda|ruck | 09:24 | |
| openstackgerrit | Mark Meyer proposed zuul/zuul master: Extend event reporting https://review.opendev.org/662134 | 09:29 |
| *** ociuhandu has joined #openstack-infra | 09:33 | |
| *** yboaron_ has quit IRC | 09:36 | |
| *** stakeda has quit IRC | 09:37 | |
| *** yboaron_ has joined #openstack-infra | 09:37 | |
| *** electrofelix has joined #openstack-infra | 09:38 | |
| *** ykarel|lunch is now known as ykarel | 09:39 | |
| *** janki has quit IRC | 09:46 | |
| *** janki has joined #openstack-infra | 09:48 | |
| *** janki has quit IRC | 09:49 | |
| *** janki has joined #openstack-infra | 09:50 | |
| *** kjackal has quit IRC | 09:52 | |
| *** janki has quit IRC | 09:52 | |
| *** kjackal has joined #openstack-infra | 09:53 | |
| *** janki has joined #openstack-infra | 09:53 | |
| *** Lucas_Gray has quit IRC | 09:55 | |
| *** yamamoto has quit IRC | 09:58 | |
| *** bobh has joined #openstack-infra | 10:00 | |
| *** janki has quit IRC | 10:02 | |
| *** janki has joined #openstack-infra | 10:03 | |
| *** janki has quit IRC | 10:04 | |
| *** bobh has quit IRC | 10:04 | |
| *** yboaron_ has quit IRC | 10:07 | |
| openstackgerrit | Mark Meyer proposed zuul/zuul master: Extend event reporting https://review.opendev.org/662134 | 10:12 |
| openstackgerrit | Mark Meyer proposed zuul/zuul master: Extend event reporting https://review.opendev.org/662134 | 10:12 |
| *** rfarr__ has joined #openstack-infra | 10:23 | |
| *** rfarr_ has joined #openstack-infra | 10:25 | |
| *** rfarr__ has quit IRC | 10:28 | |
| *** yamamoto has joined #openstack-infra | 10:32 | |
| *** yboaron_ has joined #openstack-infra | 10:38 | |
| AJaeger | mordred: do you have a governance change up that correspondents to https://review.opendev.org/#/c/657277/ ? | 10:39 |
| *** ginopc has joined #openstack-infra | 10:39 | |
| *** yamamoto has quit IRC | 10:40 | |
| *** yboaron_ has quit IRC | 10:40 | |
| *** yboaron_ has joined #openstack-infra | 10:41 | |
| *** yboaron_ has quit IRC | 10:46 | |
| *** yboaron_ has joined #openstack-infra | 10:46 | |
| *** yboaron_ has quit IRC | 10:48 | |
| *** yboaron_ has joined #openstack-infra | 10:51 | |
| *** yboaron_ has quit IRC | 10:53 | |
| *** yboaron_ has joined #openstack-infra | 10:54 | |
| *** yboaron_ has quit IRC | 10:56 | |
| *** yboaron_ has joined #openstack-infra | 10:56 | |
| *** yboaron_ has quit IRC | 10:58 | |
| *** yboaron_ has joined #openstack-infra | 10:59 | |
| icey | per the comment on https://review.opendev.org/#/c/660908/ - how do we handle project renames for a project coming under governance from the 'x' namespace? | 11:02 |
| *** yboaron_ has quit IRC | 11:04 | |
| AJaeger | icey: you need to make a change like https://review.opendev.org/644584 - see https://docs.openstack.org/infra/manual/creators.html#project-renames for full details. | 11:04 |
| AJaeger | icey: I doubt we can squeeze you in, clarkb needs to answer that. | 11:05 |
| icey | AJaeger: even though the only rename is s/x/openstack? ie, the current path is `https://opendev.org/x/charm-vault`, but it needs to become 'https://opendev.org/openstack/charm-vault'? - on that assumption, I'll start on the patch | 11:06 |
| AJaeger | icey: but if we do, provide a change stacked on top of https://review.opendev.org/657277 - without touching the rest of the stack! | 11:06 |
| icey | AJaeger: shouldn't kill me to wait :) | 11:06 |
| AJaeger | icey: that rename needs a gerrit downtime, we will have one in 3 hours - but next one will come... | 11:06 |
| *** yboaron_ has joined #openstack-infra | 11:07 | |
| icey | AJaeger: is there an easy way to identify the last entry in that stack? :-P | 11:07 |
| AJaeger | icey: the one I gave you is the last entry ;) | 11:07 |
| icey | the related-changes list suggests it's the charm-lxd rename? | 11:07 |
| icey | ah, the "fix missed renames" one? | 11:07 |
| icey | (you've linked me two in that series ;-P ) | 11:08 |
| AJaeger | I said "on top of 657277" - and gave an easier example ;) | 11:08 |
| icey | thanks :) | 11:08 |
| *** yboaron_ has quit IRC | 11:09 | |
| *** yboaron_ has joined #openstack-infra | 11:10 | |
| *** yboaron_ has quit IRC | 11:12 | |
| icey | AJaeger: seems like I should have the Depends-On to the other change as well which will, I suspect, force us to wait until the next go-round | 11:14 |
| icey | (the governance change) | 11:14 |
| openstackgerrit | Chris MacNaughton (icey) proposed openstack/project-config master: Rename x/charm-vault to openstack/charm-vault https://review.opendev.org/662432 | 11:15 |
| *** yboaron_ has joined #openstack-infra | 11:15 | |
| *** yboaron_ has quit IRC | 11:16 | |
| *** yboaron_ has joined #openstack-infra | 11:17 | |
| *** yboaron_ has quit IRC | 11:22 | |
| *** dave-mccowan has joined #openstack-infra | 11:25 | |
| *** tjgresha has quit IRC | 11:30 | |
| *** tjgresha has joined #openstack-infra | 11:33 | |
| *** yboaron_ has joined #openstack-infra | 11:38 | |
| *** yboaron_ has quit IRC | 11:42 | |
| *** yboaron_ has joined #openstack-infra | 11:44 | |
| *** EmilienM is now known as EvilienM | 11:45 | |
| *** yboaron_ has quit IRC | 11:47 | |
| *** nicolasbock has joined #openstack-infra | 11:48 | |
| *** eharney has quit IRC | 11:50 | |
| *** lifeless has quit IRC | 11:51 | |
| *** lifeless has joined #openstack-infra | 11:52 | |
| *** kaiokmo has joined #openstack-infra | 11:56 | |
| *** rh-jelabarre has joined #openstack-infra | 12:06 | |
| *** iurygregory has quit IRC | 12:07 | |
| *** logan- has quit IRC | 12:07 | |
| *** logan- has joined #openstack-infra | 12:07 | |
| *** iurygregory has joined #openstack-infra | 12:08 | |
| *** hwoarang has quit IRC | 12:14 | |
| *** hwoarang has joined #openstack-infra | 12:14 | |
| *** openstack has joined #openstack-infra | 12:27 | |
| *** ChanServ sets mode: +o openstack | 12:27 | |
| *** bgmccollum has quit IRC | 12:28 | |
| *** bgmccollum has joined #openstack-infra | 12:31 | |
| *** yamamoto has joined #openstack-infra | 12:32 | |
| openstackgerrit | Lucian Petrut proposed openstack/pbr master: Fix Windows support https://review.opendev.org/662450 | 12:35 |
| *** yamamoto has quit IRC | 12:36 | |
| *** hwoarang has quit IRC | 12:39 | |
| *** hwoarang has joined #openstack-infra | 12:39 | |
| openstackgerrit | Chris Dent proposed opendev/irc-meetings master: Cancel placement/scheduler meeting https://review.opendev.org/662451 | 12:40 |
| *** rlandy has joined #openstack-infra | 12:41 | |
| *** sreejithp has quit IRC | 12:43 | |
| *** hjensas is now known as hjensas|afk | 12:44 | |
| *** hrw has joined #openstack-infra | 12:47 | |
| hrw | morning | 12:47 |
| *** Goneri has joined #openstack-infra | 12:49 | |
| hrw | In Kolla project I have a patch which adds aarch64 CI job (based on Debian, written by someone who moved on to other job). I would like to add CentOS based one. can someone tell me do I have to create some node in infra or sth? | 12:51 |
| *** adriancz has quit IRC | 12:51 | |
| hrw | nevermind. probably found out | 12:56 |
| *** ramishra has quit IRC | 12:59 | |
| *** ramishra has joined #openstack-infra | 12:59 | |
| *** mriedem has joined #openstack-infra | 13:05 | |
| *** yamamoto has joined #openstack-infra | 13:07 | |
| *** sreejithp has joined #openstack-infra | 13:07 | |
| *** sreejithp_ has joined #openstack-infra | 13:09 | |
| *** sreejithp_ has quit IRC | 13:09 | |
| *** sreejithp_ has joined #openstack-infra | 13:09 | |
| *** sreejithp_ has quit IRC | 13:09 | |
| *** eharney has joined #openstack-infra | 13:09 | |
| *** cloudnull has quit IRC | 13:11 | |
| *** d34dh0r53 has quit IRC | 13:11 | |
| *** sreejithp has quit IRC | 13:11 | |
| openstackgerrit | Chris MacNaughton (icey) proposed openstack/project-config master: Rename x/charm-vault to openstack/charm-vault https://review.opendev.org/662432 | 13:12 |
| *** d34dh0r53 has joined #openstack-infra | 13:13 | |
| *** d34dh0r53 has quit IRC | 13:15 | |
| *** cloudnull has joined #openstack-infra | 13:17 | |
| *** d34dh0r53 has joined #openstack-infra | 13:18 | |
| *** igordc has joined #openstack-infra | 13:19 | |
| *** aaronsheffield has joined #openstack-infra | 13:23 | |
| openstackgerrit | Mark Meyer proposed zuul/zuul master: Extend event reporting https://review.opendev.org/662134 | 13:26 |
| *** gfidente|afk is now known as gfidente | 13:28 | |
| *** kazsh has quit IRC | 13:32 | |
| *** kazsh has joined #openstack-infra | 13:33 | |
| openstackgerrit | Merged opendev/git-review master: Change StoryBoard URL to be a permalink https://review.opendev.org/661451 | 13:41 |
| *** electrofelix has quit IRC | 13:43 | |
| *** yamamoto has quit IRC | 13:50 | |
| *** jcoufal has joined #openstack-infra | 13:50 | |
| *** yamamoto has joined #openstack-infra | 13:51 | |
| *** yamamoto has quit IRC | 13:51 | |
| *** yamamoto has joined #openstack-infra | 13:51 | |
| *** yamamoto has quit IRC | 13:56 | |
| fungi | it's about time for me to stop the ansible cron on bridge.o.o in preparation for our rename maintenance in an hour | 13:59 |
| *** sreejithp has joined #openstack-infra | 13:59 | |
| fungi | i've commented out the run_all.sh and run_cloud_launcher.sh entries in its crontab now | 14:00 |
| *** yamamoto has joined #openstack-infra | 14:01 | |
| fungi | the previous run_all.sh pass started at 13:45z, so i'll keep an eye out and make sure it doesn't start again after that completes | 14:01 |
| AJaeger | good morning, fungi! | 14:03 |
| hrw | I had 662456/2 started on debian-stretch-arm64. job failed, I fixed patch, sent new one and it queued because debian-stretch-arm64 node is still marked as 'building' while it's console.log shows only 'END OF STREAM' message. is there something I did wrong or is it normal operation? | 14:04 |
| *** Lucas_Gray has joined #openstack-infra | 14:05 | |
| *** yamamoto has quit IRC | 14:05 | |
| hrw | http://zuul.openstack.org/nodes shows: 0006892801 debian-stretch-arm64 ssh 66b734b5-b8f3-4677-b33e-fd46a31a8624 linaro-london building 30 minutes ago | 14:05 |
| *** armax has joined #openstack-infra | 14:07 | |
| AJaeger | hrw: for aarch64 CentOS, you need to create an image first, see https://opendev.org/openstack/project-config/src/branch/master/nodepool/nl03.openstack.org.yaml | 14:07 |
| hrw | AJaeger: job generates centos images, can be run on debian node without problem | 14:08 |
| *** piotrowskim has quit IRC | 14:08 | |
| AJaeger | hrw: you're talking Container image or VM image? | 14:08 |
| hrw | AJaeger: kolla runs on host and generates container images using other/same os. so host=debian, container=centos is perfectly fine | 14:09 |
| hrw | AJaeger: and good to see that there are debian-buster-arm64 too. will switch | 14:09 |
| AJaeger | hrw: I see | 14:09 |
| AJaeger | you pushed a new patch for 66245, it's now 662456/3 - so the old job got killed | 14:10 |
| hrw | AJaeger: as I do all kolla builds on Debian it is easier for me to run CI on Debian host as I know that it works | 14:10 |
| hrw | AJaeger: ok | 14:10 |
| AJaeger | If you only want ot experiment with single job, here's a trick: Remove all jobs from zuul.yaml for testing - and only add the new one to check queue. Once it works, add it as you do now ;) | 14:11 |
| hrw | AJaeger: right! will do it right now | 14:11 |
| *** michael-beaver has joined #openstack-infra | 14:12 | |
| hrw | AJaeger: so 'for tests' remove all normal jobs, does not define own one as 'experimental' but as normal. so zuul will by default run only my job on arm64. and once I have it working properly, readd normal jobs. | 14:14 |
| AJaeger | hrw: yes, that is my suggestion to not to have "check experimental" all the time and rerun jobs that are not neededed for now | 14:15 |
| hrw | AJaeger: great! thanks a lot | 14:15 |
| *** chandankumar is now known as raukadah | 14:18 | |
| *** igordc has quit IRC | 14:18 | |
| hrw | 662456/4 in queue then | 14:19 |
| fungi | yeah, the experimental pipeline is more appropriate for jobs you expect to define once and then possibly run on multiple different changes over some extended period of time | 14:23 |
| fungi | once you have it working in the check pipeline of your wip change, you can always switch it to experimental before merging | 14:24 |
| fungi | but having it proposed to the check pipeline while you're working through it gives you the ability to see it run without having to merge it first | 14:25 |
| fungi | and as AJaeger says, while the change is wip you can remove all other jobs in that change if you want to avoid waiting on them to complete | 14:26 |
| AJaeger | fungi, do you have time for two short reviews for the infra-specs rename? https://review.opendev.org/662298 and https://review.opendev.org/662297 , please | 14:26 |
| *** dpawlik has quit IRC | 14:27 | |
| *** weifan has joined #openstack-infra | 14:27 | |
| fungi | #status notice The Gerrit service at https://review.opendev.org/ will be offline for briefly maintenance starting at 15:00 UTC (roughly 30 minutes from now); for details see http://lists.openstack.org/pipermail/openstack-discuss/2019-May/006684.html | 14:28 |
| *** roman_g has quit IRC | 14:28 | |
| fungi | hrm, typo in that, but also statusbot is mia | 14:29 |
| fungi | will restart | 14:29 |
| *** openstackstatus has joined #openstack-infra | 14:30 | |
| *** ChanServ sets mode: +v openstackstatus | 14:30 | |
| fungi | #status notice The Gerrit service at https://review.opendev.org/ will be offline briefly for maintenance starting at 15:00 UTC (roughly 30 minutes from now); for details see http://lists.openstack.org/pipermail/openstack-discuss/2019-May/006684.html | 14:30 |
| openstackstatus | fungi: sending notice | 14:30 |
| *** weifan has quit IRC | 14:31 | |
| *** ginopc has quit IRC | 14:32 | |
| -openstackstatus- NOTICE: The Gerrit service at https://review.opendev.org/ will be offline briefly for maintenance starting at 15:00 UTC (roughly 30 minutes from now); for details see http://lists.openstack.org/pipermail/openstack-discuss/2019-May/006684.html | 14:32 | |
| AJaeger | thanks, fungi! | 14:32 |
| *** ginopc has joined #openstack-infra | 14:32 | |
| AJaeger | fungi, I have to step out now for the next 30mins or so ;( Do you want to sneak in https://review.opendev.org/662432 for the rename - or is it too late and should wait for next time? If you support bringing it in, please check with clarkb. | 14:34 |
| openstackstatus | fungi: finished sending notice | 14:34 |
| * AJaeger fears it's too late but wanted to bring attention to it - so that we don't merge it by accident if not desired. | 14:34 | |
| AJaeger | So, please -2 if it's too late... | 14:34 |
| AJaeger | change itself LGTM | 14:34 |
| clarkb | Ya I'd rather not change things at this point | 14:35 |
| AJaeger | then I'll -2. | 14:35 |
| fungi | the governance addition does seem to have merged, so we could probably include it | 14:35 |
| *** ginopc has quit IRC | 14:35 | |
| clarkb | I already need to edit the rename playbook to deal with systemd | 14:35 |
| fungi | yeah, and the lack of passing tests | 14:35 |
| fungi | let's just plan to have another batch soon and this one can be at the top of the list | 14:35 |
| openstackgerrit | David Shrewsbury proposed zuul/zuul master: Add autohold-info CLI command https://review.opendev.org/662487 | 14:36 |
| corvus | there isn't really a problem with the tests is there? | 14:36 |
| AJaeger | icey: see above | 14:36 |
| fungi | oh, right, no lack of passing tests, just lack of testability | 14:36 |
| *** igordc has joined #openstack-infra | 14:36 | |
| fungi | we have to bypass zuul for these | 14:36 |
| AJaeger | corvus: it's based on a rename change that cannot be tested | 14:36 |
| * AJaeger will step out now | 14:36 | |
| corvus | right | 14:36 |
| corvus | i don't see a problem with doing it | 14:36 |
| corvus | it's only a change to our data file | 14:36 |
| clarkb | does it need to be rebased on the stack? | 14:37 |
| AJaeger | No, it's on top | 14:37 |
| * AJaeger removed his -2 - and really steps out now. | 14:37 | |
| fungi | if clarkb's okay with adding it to the input file, i'm fine including it | 14:37 |
| corvus | lgtm if clarkb's ok | 14:38 |
| clarkb | ok if you can review it for correctness now Im making tea and will prep a dir on bridge with edited playbook and repos.yaml (can include that rename if it looks good) | 14:38 |
| openstackgerrit | David Shrewsbury proposed zuul/zuul master: Store autohold requests in zookeeper https://review.opendev.org/661114 | 14:38 |
| openstackgerrit | David Shrewsbury proposed zuul/zuul master: Add autohold-info CLI command https://review.opendev.org/662487 | 14:38 |
| *** lseki has joined #openstack-infra | 14:38 | |
| corvus | to the best of my ability, it looks correct. i'll happily propose the fixup patch if it turns out that 'v' does not come after 't'. i'm really bad at alphabetizing. | 14:39 |
| *** setuid has joined #openstack-infra | 14:40 | |
| icey | corvus: I'm in the same boat, I did my ABCs twice on that patch ;-) | 14:40 |
| fungi | i rely heavily on the sort(1) utility | 14:40 |
| corvus | is there an etherpad or anything i should bring up? | 14:40 |
| fungi | i'm getting it now | 14:41 |
| fungi | corvus: https://etherpad.openstack.org/p/project-renames-2019-05-31 | 14:41 |
| clarkb | corvus: fungi icey remote: https://review.opendev.org/662097 Rename records for project renames on 20190531 updated with x/charm-vault rename | 14:42 |
| fungi | i'll fix teh topic on the charm-vault change | 14:42 |
| corvus | i just did that | 14:42 |
| corvus | topic | 14:42 |
| fungi | cool, you're ahead of me ;) | 14:42 |
| icey | my bad on the topic - thanks :) | 14:43 |
| fungi | we're not including 661803 in the maintenance, right? | 14:45 |
| fungi | looks like it doesn't actually involve a rename, it's just got that topic | 14:45 |
| corvus | correct, i don't think we should include it | 14:45 |
| clarkb | correct | 14:46 |
| clarkb | fungi: corvus /root/renames/20190531 on bridge has an updated rename playbook (I commented out the gerrit stop and everything after the start (inclusive with the start)) | 14:46 |
| fungi | cool | 14:46 |
| clarkb | I also have repos.yaml with a repos_first.yaml with a single project in it and a repos_second.yaml if we want to do it in two phases like we did with the great big migration | 14:47 |
| fungi | that sounds fine to me | 14:47 |
| fungi | less to clean up if things go sideways immediately | 14:47 |
| clarkb | I'm updating the etherpad with the commands to use that edited playbook now | 14:48 |
| fungi | i've added myself to project bootstrappers so i can merge the changes, unless someone else was planning to take care of that | 14:49 |
| *** npochet has joined #openstack-infra | 14:49 | |
| clarkb | corvus: one thing not yet in the etherpad is if we are going to stop zuul. Is that something you think we should do? | 14:52 |
| fungi | any objection to me removing the prophylactic wip votes from the rename changes now? | 14:52 |
| clarkb | fungi: nope | 14:53 |
| clarkb | (ansible puppet is disabled so should be safe) | 14:53 |
| *** eharney has quit IRC | 14:53 | |
| *** ykarel is now known as ykarel|away | 14:53 | |
| corvus | we should really get https://review.opendev.org/588610 merged :/ | 14:54 |
| corvus | clarkb: the only reason to stop zuul is to avoid gate resets | 14:55 |
| clarkb | corvus: another question, using the edited playbook like we did last time do you recall if there is anything special we have to do to make the role lookup stuff work? | 14:55 |
| corvus | if we think it'll be relatively quick, we could try it without stopping zuul. i'm game. | 14:55 |
| fungi | we've done it that way before | 14:55 |
| openstackgerrit | Mark Meyer proposed zuul/zuul master: Extend event reporting https://review.opendev.org/662134 | 14:55 |
| fungi | since the switch to gerrit online reindexing, the rename outages have been short enough that wasn't too disruptive for zuul | 14:56 |
| corvus | clarkb: what's role lookup stuff? | 14:56 |
| *** lpetrut has quit IRC | 14:56 | |
| *** Lucas_Gray has quit IRC | 14:56 | |
| clarkb | corvus: in this case looks like it is actually an include_tasks | 14:56 |
| clarkb | for gitea-rename-tasks.yaml | 14:56 |
| clarkb | and ya looking at the root homedir we have a copy in there which I assume we pulled in so that the edited rename playbook would work last time | 14:57 |
| clarkb | I'll make a copy of that out of system-config in my renames dir now | 14:57 |
| clarkb | and new version of that includes tasks out of a role | 14:58 |
| clarkb | maybe it would be easier to have a clone of system-config and then edit that in place | 14:58 |
| *** Lucas_Gray has joined #openstack-infra | 14:58 | |
| fungi | the first several rename changes in the stack seem to be able to run and pass jobs... why are those different? | 14:59 |
| fungi | and should we still bypass gating on them anyway? | 14:59 |
| clarkb | fungi: they don't edit the zuul config | 14:59 |
| clarkb | fungi: zuul does pre merge sanity checking on config changes even for trusted repos | 14:59 |
| clarkb | so if a project that isn't part of the main.yaml shows up it will be an error | 15:00 |
| fungi | they edit zuul/main.yaml but not zuul.d/projects.yaml, so i guess the latter is the problem | 15:00 |
| clarkb | yes | 15:00 |
| fungi | cool | 15:00 |
| fungi | so do we want them to merge normally, or just merge them all manually together? | 15:00 |
| corvus | all manually together i'd say | 15:01 |
| fungi | cool, i'm ready to do those once we have a backup of the zuul pipelines (assuming we still think that's warranted) | 15:02 |
| corvus | i just made such a backup | 15:02 |
| fungi | should we status alert? | 15:02 |
| clarkb | ya I can do the status alert | 15:02 |
| fungi | thanks | 15:02 |
| fungi | okay, so ready for me to merge the project-rename stack? | 15:02 |
| clarkb | how about #status alert Gerrit is now entering its maintenance window. Expect Gerrit outages in the near future. We will notify when it is back up and running. | 15:03 |
| fungi | wfm | 15:03 |
| corvus | clarkb: wfm | 15:03 |
| clarkb | also I made a clone of system-config in my renames dir so that I don't have to copy a bunch of included tasks | 15:03 |
| clarkb | and edited rename_repos.yaml in place there | 15:03 |
| corvus | clarkb: i'm ready for fungi to proceed; you? | 15:03 |
| clarkb | yup sending the alert now | 15:03 |
| clarkb | #status alert Gerrit is now entering its maintenance window. Expect Gerrit outages in the near future. We will notify when it is back up and running. | 15:04 |
| openstackstatus | clarkb: sending alert | 15:04 |
| fungi | yep, i'll go ahead with #4 | 15:04 |
| openstackgerrit | Fabien Boucher proposed zuul/zuul master: Disable gc in test_scheduler.TestExecutor https://review.opendev.org/661316 | 15:04 |
| clarkb | I'll start a screen session on bridge | 15:05 |
| clarkb | root screen running | 15:05 |
| openstackgerrit | Slawek Kaplonski proposed zuul/zuul-jobs master: Add role to fetch journal log from test node https://review.opendev.org/643733 | 15:05 |
| openstackgerrit | Merged openstack/project-config master: Rename charm-lxd to charm-nova-lxd https://review.opendev.org/644584 | 15:06 |
| openstackgerrit | Merged openstack/project-config master: Renaming of Public Cloud WG to Public Cloud SIG https://review.opendev.org/659489 | 15:06 |
| openstackgerrit | Merged openstack/project-config master: Rename request: airship-in-a-bottle https://review.opendev.org/656008 | 15:06 |
| openstackgerrit | Merged openstack/project-config master: Rename x/compute-hyperv to openstack/compute-hyperv https://review.opendev.org/661682 | 15:06 |
| -openstackstatus- NOTICE: Gerrit is now entering its maintenance window. Expect Gerrit outages in the near future. We will notify when it is back up and running. | 15:07 | |
| *** ChanServ changes topic to "Gerrit is now entering its maintenance window. Expect Gerrit outages in the near future. We will notify when it is back up and running." | 15:07 | |
| openstackgerrit | Merged openstack/project-config master: Rename x/pyeclib and x/liberasurecode to openstack/* https://review.opendev.org/657155 | 15:07 |
| openstackgerrit | Merged openstack/project-config master: Rename opendev projects we missed https://review.opendev.org/657277 | 15:07 |
| openstackgerrit | Merged openstack/project-config master: Rename x/charm-vault to openstack/charm-vault https://review.opendev.org/662432 | 15:07 |
| clarkb | now we wait for replication to complete | 15:07 |
| corvus | i think it's done? | 15:07 |
| fungi | gerrit's queue does seem to indicate a git-upload-pack for /openstack/project-config | 15:08 |
| corvus | show queue is empty and gitea is up to date | 15:08 |
| corvus | git-upload-pack means someone is fetching it | 15:08 |
| clarkb | ya I just checked gitea and it looks good from here | 15:08 |
| fungi | yep, i think we're set | 15:08 |
| clarkb | ok I'll try to stop gerrit the normal way now? | 15:09 |
| corvus | clarkb: ++ | 15:09 |
| clarkb | and we'll figure it out if that doesn't work (as expected) | 15:09 |
| clarkb | as expected did not work | 15:09 |
| clarkb | fungi: did you end up having to kill it previously? | 15:09 |
| fungi | yeah, systemd is smart enough to believe it's not running because it thinks it never successfully started... can call the initscript directly or just sigterm | 15:10 |
| clarkb | ok I'll run sudo /etc/init.d/gerrit stop | 15:10 |
| *** kjackal has quit IRC | 15:10 | |
| openstackgerrit | Fabien Boucher proposed zuul/zuul master: Set the DiskAccountant log as warning level https://review.opendev.org/660996 | 15:10 |
| openstackgerrit | David Shrewsbury proposed zuul/zuul master: Record held node IDs with authold request https://review.opendev.org/662498 | 15:10 |
| openstackstatus | clarkb: finished sending alert | 15:11 |
| clarkb | waiting for it to stop now | 15:11 |
| fungi | looks like the java process is gone | 15:11 |
| clarkb | and the command returned so we should be good | 15:11 |
| clarkb | I've got the git diff for my edited playbook up on the root screen now | 15:12 |
| clarkb | ready to proceed with running the repos_first.yaml playbook? | 15:12 |
| fungi | yeah | 15:12 |
| corvus | ++ | 15:12 |
| fungi | i've joined your screen session | 15:13 |
| clarkb | that command look good? I will hit enter if so | 15:13 |
| corvus | ++ | 15:13 |
| fungi | yeah, i think so | 15:13 |
| fungi | and we're off to the races | 15:13 |
| *** eharney has joined #openstack-infra | 15:14 | |
| fungi | and... oof | 15:14 |
| * AJaeger is back | 15:14 | |
| clarkb | I feel like we actually ran into this last time. My omnibus must've missed something | 15:14 |
| *** e0ne has quit IRC | 15:14 | |
| fungi | mind if i scroll up the buffer? | 15:15 |
| *** Lucas_Gray has quit IRC | 15:15 | |
| corvus | need - include_vars: "{{ repolist }}" | 15:15 |
| corvus | in that play | 15:15 |
| clarkb | oh right | 15:15 |
| clarkb | fungi: go for it but corvus has the fix. I can edit it in | 15:16 |
| fungi | no worries, i've made a very large xterm and am reattaching from that | 15:16 |
| corvus | clarkb: lgtm | 15:17 |
| fungi | wow, my massive terminal with tiny font is still not as wide as yours (by a few columns) | 15:17 |
| clarkb | fungi: I've pulled up my edits if you want to look before I run it again. Including commenting out the bits that did run successfully | 15:17 |
| clarkb | fungi: all the pixels :) sorry I use tiling wm that uses all available space and I give renames a dedicated workspace | 15:17 |
| fungi | ahh, yep, looks good | 15:17 |
| corvus | that window uses 1.25 of my monitors :) | 15:18 |
| clarkb | ready to rerun the command? | 15:18 |
| corvus | ++ | 15:18 |
| corvus | i do feel like we're running into the same problems as last time | 15:18 |
| fungi | well, this is on a 2560x1440 panel but it's rotated into portrait orientation so i only have 1440 pixels of width to play with (my window is several times taller than yours though) | 15:18 |
| fungi | and yeah, go for it | 15:19 |
| clarkb | corvus: is var supposed to be vars ? | 15:19 |
| corvus | that should just be "vars" | 15:19 |
| corvus | okay, this one doesn't seem familiar after all :) | 15:20 |
| fungi | ahh | 15:20 |
| clarkb | like that? | 15:20 |
| corvus | yep | 15:20 |
| clarkb | nice thing about using a git repo this time is we'll get a git diff :) | 15:20 |
| * fungi crosses fingers superstitiously | 15:20 | |
| clarkb | ready to rerun? I don't think I need to edit any tasks out? | 15:20 |
| fungi | yeah, ready | 15:21 |
| corvus | ++ | 15:21 |
| fungi | org not in gitea_orgs | 15:21 |
| *** jaosorior has quit IRC | 15:22 | |
| corvus | *this* one seems familiar | 15:23 |
| corvus | do we have a copy of what we ran during the great migration? | 15:23 |
| corvus | because istr this one is a more structural error | 15:23 |
| fungi | so gitea_orgs is a strictundefined type, which i guess means it's not set? | 15:23 |
| clarkb | corvus: ya should still be in /root/ | 15:23 |
| clarkb | fungi: ya I think that is it and I recall similar like corvus last time | 15:23 |
| clarkb | looks like we shortcutted last time and removed the check if we didn't need to rename any orgs | 15:25 |
| clarkb | in this case I don't think we are renaming any ogs. Shall we shortcut again and remove this section of the playbook? | 15:25 |
| corvus | ok | 15:25 |
| fungi | that's expedient for now | 15:25 |
| fungi | none of the current batch of changes is creating new orgs anyway, right? | 15:26 |
| clarkb | I don't think so but we should double check | 15:26 |
| clarkb | do the edits I've made look correct? If so I'll pull up the repos.yaml files next and we can confirm no new orgs | 15:27 |
| corvus | clarkb: just the one task, right? | 15:27 |
| fungi | yeah, you do seem to have commented out the failing task | 15:27 |
| clarkb | corvus: yup that does include_task setup-orgs | 15:27 |
| corvus | 1 sec | 15:28 |
| clarkb | k | 15:28 |
| corvus | lgtm | 15:29 |
| clarkb | alright pulling up the repos.yaml files next to check there are no new orgs | 15:29 |
| clarkb | openstack opendev airship x | 15:30 |
| clarkb | all are preexisting so we should be good | 15:30 |
| corvus | agree | 15:30 |
| clarkb | shall I rerun the ansible command again? | 15:30 |
| corvus | ++ | 15:31 |
| fungi | yep | 15:31 |
| AJaeger | fungi, nothing creates new orgs | 15:31 |
| clarkb | failure on gitea06 is "ok" | 15:31 |
| corvus | agree | 15:31 |
| fungi | aha, that's the one which scrolled by too fast for me to read | 15:31 |
| fungi | yeah, that looks far better | 15:32 |
| clarkb | ok I'll remove my comments around the gerrit sections in the rename playbook and we can run this on the second list | 15:32 |
| corvus | ++ | 15:32 |
| clarkb | that look good? | 15:33 |
| corvus | yep | 15:33 |
| fungi | lgtm | 15:33 |
| fungi | crank it up | 15:33 |
| clarkb | corvus: you ready too? | 15:33 |
| corvus | clarkb: ++ | 15:33 |
| fungi | woo! | 15:35 |
| clarkb | the only failure is expected on gitea06 | 15:35 |
| * clarkb pulls up commented out bits of rename playbook so that we can see what is next | 15:35 | |
| corvus | as an aside -- it seems zuul is behaving in a slightly different way than i would expect -- it's essentially frozen because it's throwing an exception about being unable to connect. i think that ultimately the change it just tried to merge will fail to merge and there will be a gate reset, but in the mean time, it's paused and there shouldn't be any additional fallout | 15:36 |
| clarkb | k | 15:36 |
| fungi | interesting behavior, and thanks for the update | 15:36 |
| corvus | (it is, however, writing log files as quickly as it can) | 15:36 |
| clarkb | so next step according to the playbook is to start gerrit, then rename groups and start reindexing | 15:36 |
| clarkb | should I start gerrit now? | 15:37 |
| fungi | seems right | 15:37 |
| corvus | ++ | 15:37 |
| fungi | and yeah, let's see if the timeout change has fixed it | 15:37 |
| clarkb | ok don't have a screen on review.o.o but I'll just go ahead and start it | 15:37 |
| corvus | for posterity -- here's the zuul traceback: http://paste.openstack.org/show/752367/ | 15:37 |
| clarkb | sudo systemctl start gerrit has been started | 15:37 |
| fungi | i confirm it seems the process is running | 15:38 |
| clarkb | not logging much though (not sure if that is normal or not) | 15:39 |
| fungi | this is fairly normal | 15:39 |
| clarkb | but we also know it is slow to start | 15:39 |
| fungi | give it another minute or so | 15:39 |
| clarkb | there it goes | 15:39 |
| clarkb | [2019-05-31 15:39:56,735] [main] INFO com.google.gerrit.pgm.Daemon : Gerrit Code Review 2.13.12-11-g1707fec ready | 15:40 |
| clarkb | systemd knows it is running too | 15:40 |
| clarkb | web ui still coming up though | 15:40 |
| fungi | my webui reload is spinning now instead of the proxy throwing a 500 | 15:40 |
| fungi | and now it's loading! | 15:40 |
| fungi | seems to be fully up | 15:41 |
| AJaeger | \o/ | 15:41 |
| *** Lucas_Gray has joined #openstack-infra | 15:41 | |
| clarkb | ok I'll sort out the group renames and reindexing now | 15:41 |
| corvus | zuul is receiving (replication) events | 15:41 |
| clarkb | do you want me to start a root screen on gerrit for that? | 15:41 |
| fungi | did you want to just use ansible to trigger those bits, or you want to run them locally by hand? | 15:41 |
| corvus | as expected, the top gate change did not report, there was a gate reset and #2 was promoted | 15:42 |
| clarkb | I was thinking of just running them by hand | 15:42 |
| clarkb | but I guess I can comment everything but that section out of the playbook and run it with repos_second.yaml? | 15:42 |
| clarkb | fungi: corvus ^ would you prefer I do it that way? | 15:42 |
| fungi | for some reason i thought gerrit group renames and triggering renidexing were done by the playbook | 15:42 |
| corvus | yeah, i think the group renames etc will be easiest to do with a one-off playbook | 15:43 |
| clarkb | fungi: they are, but they happen after the start which I commented out so I commented those tasks out too | 15:43 |
| corvus | shouldn't you run it for all the repos, not just second? | 15:43 |
| fungi | right, i was reacting more to the comment in the etherpad which was probably copied from a maintenance where we hadn't added those yet | 15:43 |
| clarkb | corvus: I think it doesn't matter beacuse we just need the group data but ya I can run it against the master repos.yaml list | 15:43 |
| clarkb | working on playbook now | 15:44 |
| corvus | ah, i didn't recall which yaml files have the groups, but it sounds like we're on the same page | 15:44 |
| fungi | oh, nevermind, the note in the etherpad is referring to the edited version of the playbook | 15:44 |
| fungi | got it | 15:44 |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Remove unused ACLs https://review.opendev.org/661913 | 15:45 |
| clarkb | fungi: corvus ok one off playbook in root screen. If you can read and check that I'll prep it for running next | 15:45 |
| corvus | clarkb: ++ | 15:46 |
| clarkb | ok I'm going to remove the pause wait thing in there | 15:46 |
| clarkb | since we have already confirmed gerrit is up | 15:46 |
| clarkb | editing done | 15:46 |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Use retired ACL everywhere https://review.opendev.org/662223 | 15:46 |
| clarkb | fungi: ^ that look correct to you? | 15:47 |
| fungi | checking the reindex command | 15:47 |
| fungi | looks like index accounts no longer exists | 15:48 |
| fungi | gerrit index --help says the actual parameters are activate, changes and start | 15:48 |
| openstackgerrit | David Shrewsbury proposed zuul/zuul master: Record held node IDs with autohold request https://review.opendev.org/662498 | 15:48 |
| fungi | oh, wait | 15:48 |
| clarkb | fungi: I think that is the db name | 15:48 |
| clarkb | and is an argument to the command | 15:49 |
| fungi | yeah, gerrit index start INDEX [--] [--force] [--help (-h)] | 15:49 |
| *** bnemec is now known as beekneemech | 15:49 | |
| fungi | so that should be correct | 15:49 |
| clarkb | the rest of the playbook look good? | 15:49 |
| fungi | in the past i've seen `gerrit index start --force accounts` start to index changes so just making sure | 15:50 |
| fungi | i think we moved --force to the end to fix that | 15:50 |
| fungi | all looks correct to me | 15:50 |
| clarkb | k | 15:50 |
| *** gyee has joined #openstack-infra | 15:50 | |
| openstackgerrit | David Shrewsbury proposed zuul/zuul master: Record held node IDs with autohold request https://review.opendev.org/662498 | 15:51 |
| clarkb | fungi: corvus command is queued up, ready to run that? | 15:51 |
| fungi | yep, go for it | 15:52 |
| *** quiquell has quit IRC | 15:52 | |
| clarkb | hostkey verification failed | 15:52 |
| fungi | host key verification | 15:52 |
| clarkb | bah | 15:52 |
| fungi | yeah | 15:52 |
| corvus | isn't that odd? | 15:53 |
| corvus | doesn't ansible run stuff on that host all the time? | 15:53 |
| fungi | different port, different ssh host key | 15:53 |
| clarkb | corvus: its ssh from gerrit2 to review.opendev.org port 29418 that is failing I think | 15:53 |
| fungi | this is the gerrit mina-sshd host key | 15:53 |
| corvus | gotcha | 15:53 |
| fungi | not the openssh host key for the server | 15:53 |
| clarkb | we probably have review.openstack.org accepted but not opendev.org | 15:53 |
| corvus | so we should log in as gerrit2 on review01, and make "gerrit ls-projects" work | 15:53 |
| clarkb | ++ | 15:54 |
| clarkb | I can do that | 15:54 |
| corvus | k | 15:54 |
| corvus | specifically, log in as gerrit2 and make "ssh -p 29418 -i /home/gerrit2/review_site/etc/ssh_project_rsa_key openstack-project-creator@review.opendev.org gerrit ls-projects" work | 15:54 |
| fungi | even just the gerrit base command without ls-projects should return a help message from the service | 15:54 |
| clarkb | corvus: yup done | 15:54 |
| clarkb | ready to rerun the command? | 15:55 |
| clarkb | ansible command in screen that is | 15:55 |
| fungi | yeah | 15:55 |
| clarkb | and done | 15:55 |
| fungi | much better | 15:55 |
| clarkb | that concludes that "automated" portion of this rename | 15:56 |
| fungi | 34716 tasks | 15:56 |
| fungi | reindexing will likely wait behind replication | 15:56 |
| clarkb | they run in different threads iirc | 15:56 |
| fungi | ahh, okay, perhaps not then | 15:56 |
| clarkb | ok so now we are mostly in the verification period | 15:56 |
| corvus | should we status ok now? | 15:56 |
| clarkb | corvus: ya I think we likely can if we are happy with how gerrit gitea and zuul are running | 15:57 |
| fungi | i think so | 15:57 |
| fungi | has zuul regained sanity? | 15:57 |
| corvus | yeah | 15:57 |
| clarkb | ok I'll send status ok now | 15:57 |
| fungi | did the change in question eventually fail to merge or did it go through once zuul reconnected? | 15:57 |
| corvus | it failed to merge | 15:57 |
| clarkb | I think corvus said it failed | 15:57 |
| clarkb | #status ok | 15:57 |
| openstackstatus | clarkb: sending ok | 15:57 |
| fungi | got it, i missed that in the rest of our troubleshooting. thanks! | 15:58 |
| corvus | i'm giving it a recheck now | 15:58 |
| clarkb | The last major step in the rename is to reenable our puppetry | 15:58 |
| corvus | clarkb: you know you could have said "#status ok Gerrit maintenance complete" | 15:59 |
| clarkb | in theory we can do that now since we force merged and replicated prior to the restart | 15:59 |
| clarkb | corvus: oh I did not know | 15:59 |
| corvus | (seeing as how we did say we would notify when it's up and running) | 15:59 |
| *** ChanServ changes topic to "Discussion of OpenStack Developer and Community Infrastructure | docs http://docs.openstack.org/infra/ | bugs https://storyboard.openstack.org/ | source https://git.openstack.org/cgit/openstack-infra/ | channel logs http://eavesdrop.openstack.org/irclogs/%23openstack-infra/" | 15:59 | |
| clarkb | ya I thought it gave a canned message for some reason | 15:59 |
| corvus | if you don't give it a message, it just silently changes the topic back ^ | 15:59 |
| clarkb | I can do a notice | 15:59 |
| clarkb | how about #status notice Gerrit is back up and running again. Thank you for your patience. | 16:00 |
| sreejithp | hello all, git clone is failing on https://review.opendev.org/airship/in-a-bottle with the error "stderr: fatal: remote error: Git repository not found | 16:00 |
| sreejithp | " | 16:00 |
| clarkb | haev to wait for status ok to complete first but otherwise ^ should work | 16:00 |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Retire opendev-website (step 3) https://review.opendev.org/661803 | 16:00 |
| corvus | clarkb: yeah | 16:00 |
| clarkb | sreejithp: yup it was renamed. THe redirects only work on the gitea opendev.org side of things. YOu'll have to update your gerrit remotes | 16:00 |
| AJaeger | sreejithp: we just renamed it to airship/airship-in-a-bottle | 16:00 |
| fungi | sreejithp: the airship maintainers requested to have it renamed to https://review.opendev.org/airship/airship-in-a-bottle | 16:00 |
| *** eharney has quit IRC | 16:01 | |
| corvus | since we merged the p-c changes first, i think we can continue with the etherpad | 16:01 |
| corvus | next is re-enable ansible cron? | 16:01 |
| fungi | on it | 16:02 |
| clarkb | yup. I'm gonna do a local project config git update just to sanity check for myself | 16:02 |
| clarkb | but should be safe | 16:02 |
| sreejithp | thanks all | 16:02 |
| AJaeger | zuul is still complaiing about the new repos - is that to be expected? | 16:02 |
| clarkb | changes I expect are there | 16:02 |
| *** rlandy is now known as rlandy|brb | 16:02 | |
| clarkb | AJaeger: ya we haven't updated zuul's config yet. That happens as side effect of next change | 16:02 |
| AJaeger | great! | 16:02 |
| clarkb | AJaeger: the step where fungi reenables ansible + puppet then puppet can update zuul config | 16:03 |
| fungi | i've uncommented the run_all.sh and run_cloud_launcher.sh entries in the bridge.o.o root crontab | 16:03 |
| fungi | should start back up at 16:15z | 16:03 |
| AJaeger | clarkb: I see... | 16:03 |
| fungi | AJaeger: last sentence under #8 on https://etherpad.openstack.org/p/project-renames-2019-05-31 | 16:03 |
| clarkb | still waiting on statusbot to finish sneding its ok | 16:04 |
| clarkb | looks like it took ~6 minutes last time | 16:04 |
| corvus | after we move the gitea project creation into python, we can write a test job that runs the rename playbook -- with the *actual* data for a rename. | 16:05 |
| clarkb | corvus: that would be so nice | 16:05 |
| clarkb | I guess I should work on pushing some of my fixes up now | 16:05 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: More repo rename playbook fixes https://review.opendev.org/662510 | 16:07 |
| clarkb | corvus: fungi ^ that captures the two fixes we needed but does not do anything with gitea orgs lists | 16:08 |
| fungi | i guess we're still not sure what the fix is for the org creation piece? | 16:08 |
| corvus | i can take a stab at that | 16:09 |
| *** Lucas_Gray has quit IRC | 16:10 | |
| *** kopecmartin is now known as kopecmartin|off | 16:10 | |
| clarkb | thansk | 16:11 |
| clarkb | I'm guessing we need to copy a taks out of our regular gitea management into gitea-rename-tasks.yaml | 16:11 |
| clarkb | (the one that gets a list of orgs already in the system) | 16:11 |
| *** Lucas_Gray has joined #openstack-infra | 16:12 | |
| openstackgerrit | James E. Blair proposed opendev/system-config master: Fix gitea rename playbook org creation https://review.opendev.org/662511 | 16:12 |
| corvus | yep | 16:12 |
| clarkb | heh we are really overloading user namespaces aren't we | 16:13 |
| *** mriedem is now known as mriedem_hangry | 16:14 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: WIP: registry test job https://review.opendev.org/661327 | 16:15 |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Fix gerrit/projects.yaml after rename https://review.opendev.org/662513 | 16:15 |
| AJaeger | infra-root, here's a hot fix ^ | 16:15 |
| corvus | we need to stop ansible | 16:15 |
| clarkb | corvus: go for it | 16:16 |
| *** Ng is now known as Ng_ | 16:17 | |
| clarkb | looks like someone killed run_all.sh | 16:17 |
| clarkb | there are ansible processes for base.yaml running | 16:17 |
| clarkb | but I think those are ok? | 16:17 |
| corvus | i think everything is dead now | 16:18 |
| clarkb | yup I don't see anything now | 16:18 |
| clarkb | so whats up? | 16:18 |
| *** jpich has quit IRC | 16:18 | |
| corvus | AJaeger's hotfix | 16:18 |
| corvus | we would have created x/ansible-role-cloud-launcher | 16:18 |
| clarkb | ah looking now | 16:18 |
| corvus | let's see if we did | 16:18 |
| clarkb | the manage projects log on review.o.o is still empty so I don't think it got as far as gerrit | 16:19 |
| corvus | and we did | 16:19 |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Sort zuul.d/projects.yaml again https://review.opendev.org/662514 | 16:19 |
| corvus | oh? | 16:19 |
| corvus | https://review.opendev.org/#/admin/projects/x/ansible-role-cloud-launcher exists | 16:19 |
| clarkb | that was its old name maybe we didn't rename it? | 16:19 |
| AJaeger | and 662514 is another hotfix (but not critical, just blocks us merging anything) | 16:19 |
| clarkb | ya I made repos.yaml out of projects.yaml diffs | 16:20 |
| AJaeger | clarkb: we renamed it elsewhere, e.g. ACLs are renamed | 16:20 |
| corvus | oh | 16:20 |
| corvus | we *did not* rename ansible-role-cloud-launcher | 16:20 |
| clarkb | so if it wasn't in there then it probably didn't get renamed the above url doesn't work if you replace x with opendev | 16:20 |
| clarkb | corvus: ya I don't think we did | 16:20 |
| clarkb | so the real fix we want is to undo the other rename stuff then pick this up next time | 16:20 |
| corvus | AJaeger: your change is wrong then -- it should move it back to x/ | 16:20 |
| AJaeger | corvus: can any of you take over - I need to step out for another 30mins now ;( | 16:21 |
| clarkb | I can help sort it out | 16:21 |
| *** eharney has joined #openstack-infra | 16:21 | |
| fungi | i killed the remaining ansible remote processes | 16:21 |
| AJaeger | I can sort it... | 16:21 |
| AJaeger | I'm off by 30mins, have time. | 16:21 |
| clarkb | AJaeger: ok | 16:21 |
| corvus | i'll re-enable the ansible cron then? | 16:21 |
| *** dtantsur is now known as dtantsur|afk | 16:22 | |
| clarkb | corvus: will manage projects just fail to run in that case because it won't have a acl file for that repo? | 16:22 |
| clarkb | if that is the behavior we expect then I think we might be ok with the cron running | 16:22 |
| corvus | fungi: thanks, though i'm a little curious about what processes you killed since my ps wasn't reporting any... | 16:22 |
| fungi | ssh: /root/.ansible/cp/7e582814a5 et cetera | 16:22 |
| clarkb | the ssh control persist processes | 16:22 |
| corvus | fungi: oh, those are harmless and die on their own. that's the controlpersist process | 16:23 |
| fungi | ahh | 16:23 |
| *** sgw1 has joined #openstack-infra | 16:23 | |
| fungi | i thought those were the remote ansible script runs | 16:23 |
| corvus | clarkb: hrm, i dunno. want to force-merge a fix? | 16:23 |
| clarkb | corvus: we'll still have to wait for replication to complete | 16:23 |
| corvus | fungi: nah, those look like: root 16757 0.0 0.0 47060 5452 ? S 16:17 0:00 ssh -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/3954b8345e 2001:4800:7818:103:be76:4eff:fe04:976a /bin/sh -c | 16:23 |
| corvus | '/usr/bin/python && sleep 0' | 16:23 |
| clarkb | so probably don't need to force merge and just reenable cron later today or on monday? | 16:24 |
| fungi | corvus: thanks, noted | 16:24 |
| clarkb | it would be cool if there was a jump the queue replication command | 16:24 |
| *** lucasagomes has quit IRC | 16:24 | |
| clarkb | down to 14k tasks now | 16:24 |
| clarkb | 2k are indexing tasks | 16:25 |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Fix ansible-role-cloud-launcher https://review.opendev.org/662513 | 16:25 |
| AJaeger | fixed both issues together ^ | 16:25 |
| clarkb | I guess we can do an out of band replicate like we did in the past for refs/* | 16:25 |
| corvus | unfortunately, i don't think we can merge any changes to project-config until ansible runs and updates the zuul config | 16:26 |
| clarkb | we might have to force merge anyway or we can update the zuul config via kick.sh? | 16:26 |
| clarkb | ya | 16:26 |
| corvus | let's force-merge it | 16:27 |
| clarkb | k | 16:27 |
| fungi | i'm still trying to digest what transpired... | 16:27 |
| corvus | fungi: you still a boostrapper? if so, when you're happy, you want to force-merge 662513 | 16:27 |
| fungi | so did we or did we not rename x/ansible-role-cloud-launcher? looks like it's still named that ni gerrit | 16:27 |
| fungi | and yes, i'm still in bootstrappers | 16:28 |
| clarkb | fungi: x/ansible-role-cloud-launcher did all the steps to rename except for updating gerrit/projecs.yaml. I generated my repos.yaml for the rename playbook from projects.yaml diffs so we did not rename that project | 16:28 |
| clarkb | fungi: however we did update the zuul config for it and the gerrit acl config | 16:28 |
| clarkb | AJaeger's change undoes the updates so everything is back to x/ansible-role-cloud-launcher again | 16:28 |
| fungi | got it | 16:28 |
| fungi | i was looking at the earlier iteration of that change | 16:28 |
| *** ramishra has quit IRC | 16:29 | |
| clarkb | and we'll have to rename that project with the next group of renames | 16:30 |
| clarkb | not a huge deal other than its making updating zuul's config awkward | 16:30 |
| clarkb | but we can kick.sh our way out of that if we need to | 16:30 |
| openstackgerrit | Merged openstack/project-config master: Fix ansible-role-cloud-launcher https://review.opendev.org/662513 | 16:30 |
| fungi | grr, gertty has decided it's offline again... just a sec | 16:30 |
| fungi | there it goes | 16:30 |
| clarkb | AJaeger: thank you for catching that | 16:31 |
| clarkb | 13k ish tasks now and just over 2k are indexing tasks | 16:31 |
| clarkb | we seem to be making reasonable progress | 16:31 |
| AJaeger | clarkb: my other change failed and looking at log files I noticed it. Sorry for missing it in review ;( | 16:32 |
| clarkb | I wonder if we could disable zuul's sanity checking of the config (force it to use last good config) so that jobs can run against updated data | 16:32 |
| clarkb | openstackstatus never send the finished sending alert message | 16:35 |
| corvus | when we have the zuul playbook finished, this won't be a problem since we'll be updating zuul's config much faster | 16:36 |
| clarkb | re statusbot the logs for it indicate it is still running I think | 16:37 |
| clarkb | should I try sending the notice now? | 16:37 |
| clarkb | I guess I should grep for tracebacks first | 16:37 |
| *** ociuhandu_ has joined #openstack-infra | 16:38 | |
| clarkb | nothing | 16:38 |
| clarkb | corvus: fungi ^ thoughts on that. Should I try the notice or keep waiting? | 16:40 |
| *** ociuhandu has quit IRC | 16:41 | |
| clarkb | I know in the past it would have a sad if trying to process two events at once | 16:41 |
| fungi | clarkb: oh, did it never confirm the ok? | 16:41 |
| clarkb | not that I can see in scrollback | 16:41 |
| clarkb | it confirmed the alert but not the ok | 16:41 |
| AJaeger | fungi, could you review https://review.opendev.org/#/c/662298/ to unbreak infra-manual building, please? | 16:42 |
| *** ociuhandu_ has quit IRC | 16:42 | |
| fungi | yeah, and it never parted/joined the channel in that time either | 16:42 |
| fungi | looks like it was processing them in alpha order and got as far as #zuul at 2019-05-31 16:01:29,599 | 16:43 |
| clarkb | so maybe it completed | 16:44 |
| fungi | (from the last line of the log) | 16:44 |
| fungi | i wonder if a #status ok with no message included doesn't generate a confirmation for some reason | 16:44 |
| fungi | yeah, i'd say go ahead with the notice | 16:44 |
| fungi | problems in the past had to do with it losing track of channel topic changes, which look to have already completed | 16:45 |
| fungi | also, i think that particular bug was eventually fixed | 16:45 |
| clarkb | #status notice Gerrit is back up and running again. Thank you for your patience and sorry for the delay in this notification (we thought the statusbot was still busy updating channel topics). | 16:45 |
| openstackstatus | clarkb: sending notice | 16:45 |
| *** rlandy|brb is now known as rlandy | 16:45 | |
| corvus | it should send a 'finished' message even with no ok message | 16:45 |
| clarkb | the notice is working fwiw | 16:46 |
| clarkb | also down to 9.7k replication tasks | 16:46 |
| -openstackstatus- NOTICE: Gerrit is back up and running again. Thank you for your patience and sorry for the delay in this notification (we thought the statusbot was still busy updating channel topics). | 16:46 | |
| clarkb | now just under 9.6k | 16:47 |
| clarkb | about an hour and a half away from completion? | 16:48 |
| clarkb | though that may depend on size of repos | 16:48 |
| *** hwoarang has quit IRC | 16:48 | |
| fungi | it will spend a lot more time on the larger repos, but those don't actually matter today | 16:48 |
| *** hwoarang has joined #openstack-infra | 16:48 | |
| clarkb | if that estimate is correct (or close) then we should have plenty of time today to turn on the cron today | 16:48 |
| openstackstatus | clarkb: finished sending notice | 16:49 |
| fungi | as long as the reindexing tasks for the renamed repos complete, i think we're fine | 16:49 |
| clarkb | fungi: we also need the ansible-role-cloud-launcher fix to replicate so we can turn on ansible cron | 16:49 |
| fungi | we didn't rename nova, neutron or openstack-manuals so when those are the only ones left (and typically they run for quite a while longer than the rest) we can presumably proceed | 16:50 |
| AJaeger | Now I need to step out... | 16:51 |
| clarkb | AJaeger: enjoy your evening and weekend. Thank you for all the help | 16:52 |
| clarkb | now down to 8.6k replication tasks | 16:54 |
| clarkb | moving faster than I mathed | 16:54 |
| fungi | so if there is no pending replication task for openstack/project-config we can go ahead i guess? | 16:56 |
| clarkb | still waiting on those | 16:56 |
| clarkb | but ya I guess so | 16:56 |
| clarkb | also gitea06 is struggling. We may want to remove it from the replication list (later) | 16:58 |
| clarkb | nova and neutron are the last indexing projects | 16:59 |
| fungi | `... gerrit show-queue -w|grep -n openstack/project-config` says there are 1298 tasks ahead of the last gitea push for project-config | 16:59 |
| *** derekh has quit IRC | 17:00 | |
| fungi | oh, though i guess we don't care about replication to gitea06 | 17:02 |
| fungi | so more accurately, 1188 waiting replication tasks ahead of the final gitea replication of project-config excepting the one for gitea06 (which is a but farther down the queue) | 17:03 |
| clarkb | correct its just 07 we are waiting for now | 17:03 |
| clarkb | https://gitea01.opendev.org:3000/openstack/project-config/commit/6312597c060cb93689197a60140e32c31fd06edd does exist though (and I think everywhere but 06 and 07) | 17:03 |
| clarkb | and github | 17:04 |
| fungi | another alternative would be to just take 07 out of haproxy temporarily? | 17:04 |
| clarkb | I think it should get there soon. It is in the osas now | 17:04 |
| fungi | if we don't want to wait | 17:04 |
| fungi | yeah | 17:04 |
| fungi | i've been told that i have a hard stop for lunch at 17:30, lest christine gnaw off one of my arms | 17:06 |
| clarkb | did we want to take a break now then reconvenve in ~1.5 hours or whatever to enable the cron? | 17:06 |
| clarkb | corvus: ^ don't know what your schedule is like | 17:06 |
| fungi | wfm, having to type one-handed for the remainder of the afternoon would significantly impact my productivity | 17:06 |
| openstackgerrit | Merged openstack/infra-manual master: Update infra-specs URL https://review.opendev.org/662298 | 17:08 |
| corvus | clarkb: sounds good | 17:08 |
| clarkb | I did a double check on 01-05 and 08 they all have the fix for ansible-role-cloud-launcher so just 06 and 07 | 17:08 |
| clarkb | fungi: go have lunch then :) | 17:08 |
| fungi | yeah, still 1127 replication tasks ahead of it | 17:09 |
| clarkb | another idea I had was for us to update our git remotes to be gerrit for things that drive stuff | 17:09 |
| fungi | will bbiaw | 17:09 |
| clarkb | because then we wouldn't ahve to wait for replication | 17:09 |
| fungi | clarkb: once we get a shared backend for gitea we only need to replicate once | 17:09 |
| clarkb | fungi: thats true | 17:09 |
| fungi | also switching openstack/ repos over to github replication via zuul job means we don't need to do those either | 17:10 |
| corvus | though it may not be much faster than this | 17:10 |
| fungi | fair ;) | 17:10 |
| *** xek_ has quit IRC | 17:12 | |
| fungi | #status log manually removed nova 2013.1 release from pypi | 17:14 |
| openstackstatus | fungi: finished logging | 17:14 |
| clarkb | fungi: hurry before you lose your arm :P | 17:15 |
| *** gfidente is now known as gfidente|afk | 17:16 | |
| clarkb | 5.3k total tasks now | 17:18 |
| clarkb | I think 07 may be slow due to disk io (it doesn't seem to be doing a ton of cpu) | 17:23 |
| clarkb | plenty of memory and disk available too | 17:24 |
| *** haleyb has joined #openstack-infra | 17:24 | |
| haleyb | have a question, maybe someone can point me in the right direction. i'm trying to debug a period job failure that's failing to install the proper kernel-devel package. Digging into it, the job is using 'fedora-latest', but the kernel booted in fc28, not fc29. Instead of continuing I'd figured i'd ask if that looked wrong, especially since fc28 is now EOL | 17:28 |
| clarkb | haleyb: a link to the job would be good, but there is a good chance that fedora-latest was never updated to be newer than 28 | 17:29 |
| clarkb | *link to job log | 17:29 |
| clarkb | hrm no my local copy of the nodeset def says it is fedora-29 so ya logs would be a big help | 17:30 |
| haleyb | clarkb: the log from my hacked job is http://logs.openstack.org/67/661067/2/check/networking-ovn-tempest-dsvm-ovs-master-fedora/8beacab/job-output.txt.gz | 17:30 |
| haleyb | search for kernel-devel | 17:30 |
| haleyb | i had to add the job as a standard one to get it to run on the patch | 17:30 |
| clarkb | that job ran on a fedora-28 node according to ansible facts http://logs.openstack.org/67/661067/2/check/networking-ovn-tempest-dsvm-ovs-master-fedora/8beacab/zuul-info/host-info.controller.yaml so we can at least confirm that | 17:31 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: WIP: registry test job https://review.opendev.org/661327 | 17:31 |
| *** Lucas_Gray has quit IRC | 17:31 | |
| haleyb | clarkb: i thought my backtrack through the zuul definitions showed 'fedora-latest' | 17:32 |
| corvus | http://zuul.openstack.org/job/networking-ovn-tempest-dsvm-ovs-master-fedora | 17:32 |
| clarkb | haleyb: https://opendev.org/openstack/devstack/src/branch/master/.zuul.yaml#L51-L59 its teh devstack fedora latest nodeset | 17:33 |
| clarkb | and that one hasn't updated | 17:33 |
| clarkb | so you can either upate it or use a different nodeset | 17:33 |
| clarkb | haleyb: https://opendev.org/openstack/openstack-zuul-jobs/src/branch/master/zuul.d/nodesets.yaml#L14-L18 is the generic one (note it doesn't ahve the controlloer/compute/etc labels that the devstack nodesets have) | 17:33 |
| haleyb | clarkb: thanks, i guess i asked in the right channel :) So is changing the devstack one to use fedora-29 a good start? | 17:35 |
| clarkb | probably | 17:36 |
| clarkb | one thing to check is if tripleo uses that nodeset | 17:36 |
| clarkb | they want to keep using 28 until centos 8 is a thing | 17:36 |
| clarkb | but I'd probably just notify them and move forward since as you say it is eol | 17:37 |
| haleyb | clarkb: getfedora.org says it is as of Wednesday | 17:37 |
| *** hwoarang has quit IRC | 17:37 | |
| clarkb | haleyb: ya they knew that was a risk when they did the python3 testing on 28 | 17:37 |
| *** hwoarang has joined #openstack-infra | 17:38 | |
| clarkb | and they may have their own nodesets already | 17:38 |
| openstackgerrit | Merged opendev/system-config master: More repo rename playbook fixes https://review.opendev.org/662510 | 17:39 |
| haleyb | looks like neutron, networking-ovn, barbican, devstack and novajoin use the devstack-single-node-fedora-latest nodeset | 17:39 |
| * redrobot hears barbican in the distance; pokes head in | 17:40 | |
| haleyb | redrobot: i was just looking in codesearch for repos using the devstack-single-node-fedora-latest nodeset | 17:40 |
| redrobot | haleyb, gotcha. Yeah we use in barbican to test the DogTag backend plugin, which is only available in Fedora IIUC | 17:41 |
| haleyb | redrobot: as that is using f28 which just went EOL i was going to propose updating to f29, which may or may not solve the problem i'm chasing installing packages in a job | 17:42 |
| *** gfidente|afk has quit IRC | 17:43 | |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Fix gerrit/projects.yaml sort order https://review.opendev.org/662527 | 17:43 |
| redrobot | haleyb, that would work for us. We don't really care about the Fedora version. | 17:43 |
| clarkb | AJaeger: thanks. I don't think we need that one before we start ansible cron again but will be good to have working jobs :) | 17:44 |
| clarkb | gitea07 is in the olso projects now | 17:44 |
| AJaeger | clarkb: indeed, it just unblocks | 17:44 |
| haleyb | redrobot: i'll propose it and try and get cross-repo test patches to see if things break, have no idea | 17:45 |
| redrobot | haleyb, cool. I can help fix barbican things if they break. | 17:46 |
| haleyb | redrobot: thanks, hopefully i can propose these today and check on them monday | 17:46 |
| *** iurygregory has quit IRC | 17:46 | |
| *** hwoarang has quit IRC | 17:48 | |
| *** hwoarang has joined #openstack-infra | 17:48 | |
| *** raissa has joined #openstack-infra | 17:50 | |
| corvus | clarkb: have you seed "dial tcp [2607:ff68:100:54:f816:3eff:fe6e:65ff]:5000: connect: permission denied" in a registry job yet? | 17:50 |
| clarkb | corvus: no | 17:50 |
| clarkb | the most recent issues I had seen were with the docker hub proxy on 5001 | 17:51 |
| clarkb | (causing it to fallback to docker hub then fail auth there) | 17:51 |
| clarkb | permission denied there is interesting that almost reads like you can't open a socket | 17:51 |
| clarkb | since when auth fails you get the your auth is bad message instead | 17:51 |
| corvus | yeah....that was trying to connect to a simulated intermediate registry -- (ie, it's a stand-in for insecure-ci-registry.opendev.org). it was in limestone over ipv6, but there's an /etc/hosts entry for it, so skopeo is using a hostname. | 17:52 |
| *** raissa has quit IRC | 17:53 | |
| clarkb | is it possible skopeo is doing that namespacing for image transfers again and that doesn't have an ipv6 stack? | 17:53 |
| *** raissa has joined #openstack-infra | 17:53 | |
| corvus | wow. i have run this job a lot, and it does not run on limestone often. | 17:55 |
| clarkb | openstack/project-config is replicating to gitea07 now | 17:56 |
| corvus | clarkb: i found a previous run on limestone, it hit the same error :/ | 17:57 |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Rename x/ansible-role-cloud-launcher -> opendev/ https://review.opendev.org/662530 | 17:58 |
| corvus | clarkb: does that mean we need to socat for the the transfer to the intermediate registry too? | 17:58 |
| *** panda|ruck has quit IRC | 17:59 | |
| clarkb | corvus: in production if this is the issue it may be using ipv4 (because there is a record for that in dns) | 18:00 |
| clarkb | in which case we probably don't absolutely need to | 18:00 |
| corvus | clarkb: this is for the test i'm making | 18:00 |
| clarkb | you might try it | 18:00 |
| clarkb | though in the test env we hsould have private ipv4 that we could use too | 18:00 |
| clarkb | (and put that in /etc/hosts instead) | 18:01 |
| *** pkopec_ has quit IRC | 18:01 | |
| *** panda has joined #openstack-infra | 18:01 | |
| corvus | clarkb: do you think using "nodepool.private_ipv4 || nodepool.public_ipv4" is the way to go? | 18:02 |
| clarkb | I think so. To me that is preferable to the socat hack (whcih we should only really use where necessary) | 18:02 |
| corvus | k | 18:02 |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Fix gerrit/projects.yaml sort order https://review.opendev.org/662527 | 18:02 |
| *** kjackal has joined #openstack-infra | 18:04 | |
| *** tesseract has quit IRC | 18:04 | |
| clarkb | project-config still replicating | 18:06 |
| *** tosky has quit IRC | 18:09 | |
| *** dave-mccowan has quit IRC | 18:09 | |
| corvus | oh looks like private_ipv4 is always defined | 18:10 |
| corvus | even if it's public | 18:10 |
| openstackgerrit | Brian Haley proposed zuul/zuul-jobs master: Update repo file for fedora-29 https://review.opendev.org/662536 | 18:11 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: WIP: registry test job https://review.opendev.org/661327 | 18:11 |
| clarkb | haleyb: is that required? | 18:12 |
| clarkb | I think pre 28 is when things changed | 18:12 |
| clarkb | so I think that will break 28 | 18:12 |
| haleyb | clarkb: i don't know, was just following what was done when 28 was added | 18:12 |
| clarkb | haleyb: 29 is added and running and fine. The only thing that should need to be chagned is that devstack nodeset definition I linked you | 18:12 |
| haleyb | clarkb: any idea how to overcome this zuul complaint in https://review.opendev.org/#/c/662529/ ? | 18:13 |
| clarkb | ah the problem is what the stable branches ahve the nodesets too and they must match | 18:14 |
| clarkb | I don't know that -latest nodesets will work in branched repos like devstack give that constraint | 18:14 |
| clarkb | corvus: ^ | 18:14 |
| clarkb | haleyb: zuul requires nodeset definitions to be the same acorss branches | 18:14 |
| clarkb | but that change to a single branch breaks that rule | 18:14 |
| clarkb | so you might need to add a -29 nodeset and switch to that | 18:14 |
| clarkb | then we can delete the -latest nodeset when it isn't used anymroe | 18:15 |
| *** ykarel|away has quit IRC | 18:15 | |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Check ACLs for all namespaces https://review.opendev.org/662537 | 18:15 |
| haleyb | from reading the commit message i thought the point was to avoid using the number, but i'm not going to fight it | 18:15 |
| clarkb | haleyb: yes I think that was pabelanger's intent when he added the -latest nodeset but when he added them to project-config which is branchless that was fine. The problems arise when it got cargo culted into other repos | 18:16 |
| clarkb | if those repos are branched | 18:16 |
| clarkb | another appraoch would be to move the devstack nodeset defs into openstack-zuul-jobs | 18:17 |
| corvus | the process is supposed to be to remove nodesets from branches | 18:20 |
| hrw | hi again | 18:21 |
| clarkb | corvus: ah so clean up stable/* | 18:21 |
| corvus | 1) create branch from master 2) remove nodesets 3) update nodeset in master branch 4) repeat | 18:21 |
| corvus | yeah | 18:21 |
| *** rkukura has joined #openstack-infra | 18:21 | |
| corvus | the idea is that nodesets aren't likely to change in the minutes after a branch creation, so if you do that, you won't hit an error. | 18:22 |
| haleyb | corvus: so these should move? or should be removed from stable/* ? | 18:22 |
| corvus | if you put it off, then you're in this situation | 18:22 |
| clarkb | hrw: hello | 18:22 |
| corvus | haleyb: removed from stable | 18:22 |
| hrw | in 662456/4 I selected 'debian-buster-arm64' label for nodeset as it is listed in https://opendev.org/openstack/project-config/src/branch/master/nodepool/nl03.openstack.org.yaml as being present in 'linaro-london' cloud. But zuul gives me NODE_FAILURE on check. | 18:22 |
| corvus | haleyb: so if you remove all the nodesets from stable/* branches, then have your change to update the -latest nodeset depends-on those, all should be good. | 18:23 |
| corvus | (the alternative would be to ignore nodesets from stable branches, which would be more convenient, except that it would also be *way* more confusing) | 18:23 |
| haleyb | corvus: let my try that, think i just might have to remove it from stable/stein or will that then fail, and down the rabbit hole | 18:23 |
| corvus | haleyb: it should be perfectly fine to remove it from all stable branches, it's basically a noop already. but you do need to remove it from every branch it appears in (i haven't checked too see which ones it does) | 18:24 |
| corvus | haleyb: to be clear, just the definition needs to be removed | 18:24 |
| clarkb | hrw: I don't see any buster images currently so something must be broken | 18:24 |
| hrw | http://zuul.openstack.org/nodes does not list 'debian-buster-arm64' (only 'ubuntu-xenial-arm64', 'ubuntu-bionic-arm64' and 'debian-stretch-arm64') | 18:24 |
| clarkb | stretch, bionic and xenial are present | 18:25 |
| haleyb | corvus: ack, i'll give it a try | 18:25 |
| hrw | clarkb: will switch back to stretch then | 18:25 |
| clarkb | hrw: we'll have to dig into why those buster images aren't being built. No build logs either https://nb03.openstack.org/ | 18:27 |
| clarkb | but right now I'm still pretty context switched into the project renaming so probably won't dig into that until later today or monday | 18:27 |
| clarkb | project-config is still replicating | 18:27 |
| hrw | clarkb: no rush. stretch is fine too | 18:28 |
| hrw | thanks for all the work and have a good weekend | 18:29 |
| corvus | clarkb: looks like we're waiting on gitea07 and it is replicating p-c as we speak | 18:29 |
| clarkb | corvus: yup | 18:29 |
| clarkb | its been going for a while | 18:29 |
| corvus | i wonder if there's a problem | 18:30 |
| clarkb | One thing that happened before was when we had gitea07 restart when we did the big replication last time it was around the openstack/puppet repos | 18:30 |
| clarkb | possible it didn't fully replicate refs/notes and refs/changes for project-config since that is alphabetically close | 18:30 |
| clarkb | (and now it is catching up?) | 18:30 |
| clarkb | nova is still reindexing btu is the only one left there | 18:31 |
| corvus | clarkb: yeah, work is happening on 07 with p-c | 18:31 |
| corvus | it's pushing old change refs | 18:31 |
| clarkb | ya so that explains it. It was probably caught in that restart along with the puppet repos | 18:32 |
| corvus | ps shows commands like: bash hooks/update refs/changes/02/559802/2 0000000000000000000000000000000000000000 a2f6d622089bfa34470b58ad76db01a3d6a3f59b | 18:32 |
| corvus | i don't think the refs are ordered, so it's hard to say it's progress | 18:32 |
| clarkb | disk usage looks ok (I checked that because if it was replicating a lot of stuff it might overrun the disk again too) | 18:33 |
| clarkb | we can check if master is updated | 18:34 |
| * clarkb looks | 18:34 | |
| clarkb | https://gitea07.opendev.org:3000/openstack/project-config/commit/6312597c060cb93689197a60140e32c31fd06edd exists and https://gitea07.opendev.org:3000/openstack/project-config/commits/branch/master shows it as head | 18:35 |
| clarkb | so we are probably ok to proceed | 18:35 |
| clarkb | oh its done | 18:36 |
| corvus | just when i was about to be able to quantify it :) | 18:36 |
| corvus | in case you were wondering, project-config has about 35k refs | 18:36 |
| clarkb | wow | 18:36 |
| clarkb | and we ended up with 8.6GB of disk free which GCing will clean up a bit from there | 18:37 |
| clarkb | so that all looks good | 18:37 |
| *** whoami-rajat has quit IRC | 18:37 | |
| clarkb | I think we can proceed whenever the humans are ready? | 18:37 |
| clarkb | did we want to wait for fungi to return? | 18:37 |
| * fungi returned a while back, sorry | 18:38 | |
| fungi | still catching up | 18:38 |
| clarkb | next cron run will be 18:45 UTC if we reenable before then | 18:38 |
| corvus | i think we're good | 18:38 |
| corvus | i will re-enable | 18:39 |
| clarkb | ya I can't think of anything else to check (I individually checked the giteas for the ref we need) | 18:39 |
| corvus | re-enabled | 18:39 |
| clarkb | `tail -f /var/log/manage_projects.log` if you want to watch jeepyb there | 18:40 |
| clarkb | fungi: re github archival, you are thinking maybe add jroll to the access list there? | 18:41 |
| *** yamamoto has joined #openstack-infra | 18:41 | |
| clarkb | my one concern with this is we were previously trying to reduce our account surface area in github (to avoid people accidenting github pushes/merges/etc) | 18:42 |
| clarkb | so we may need to have a think about whether or not we use a second shared account (not shared by opendev but for openstack specifically) or require second admin accounts or something | 18:42 |
| jroll | I would be terrified of screwing that up, but am willing to try if y'all prefer | 18:42 |
| jroll | can always revoke my access when complete | 18:42 |
| clarkb | personally I'm happy for openstack to self manage it, but I think the general please don't use your real account rules we were slowly moving towards were decent (I say this and I haven't removed my account yet so shame on me :) ) | 18:43 |
| fungi | clarkb: adding jroll wfm as long as it wfh | 18:43 |
| fungi | i think (at least) sdague was already given access to that org previously to aid in curation | 18:44 |
| *** mriedem_hangry is now known as mriedem | 18:44 | |
| jroll | I'm fine with doing it (and can create another account if you prefer), if one or two of you can help review my logic/list in the email I sent to the ML a bit ago | 18:44 |
| clarkb | jroll: really the concern is that if you are working on your github fork of say nova or ironic that you don't accidentally push your company fork to upstream nova in github | 18:45 |
| clarkb | jroll: and if you are aware of those concerns then its probably fine because you'll be careful :) | 18:45 |
| jroll | clarkb: for sure, understandable | 18:45 |
| clarkb | and ya we can help review the plan and details | 18:45 |
| jroll | excellent | 18:46 |
| *** yamamoto has quit IRC | 18:46 | |
| clarkb | ansible should've started | 18:46 |
| clarkb | yes seems to be starting base.yaml | 18:47 |
| fungi | yeah, watching the run_all log | 18:47 |
| fungi | looks... busy | 18:47 |
| * fungi can't read that fast | 18:47 | |
| clarkb | this will take like 15 minutes iirc | 18:49 |
| clarkb | (base.yaml) | 18:49 |
| clarkb | though we did clean it up a bit so maybe shorter now | 18:49 |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Check ACLs for all namespaces https://review.opendev.org/662537 | 18:50 |
| *** bobh has joined #openstack-infra | 18:51 | |
| clarkb | doing exim then snmpd then iptables then onto the service stuff and puppet stuff | 18:53 |
| *** eharney has quit IRC | 18:54 | |
| clarkb | base is done | 18:55 |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Use retired ACL everywhere https://review.opendev.org/662223 | 18:55 |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Remove unused ACLs https://review.opendev.org/661913 | 18:55 |
| clarkb | its doing dns servers now. which we should check after to make sure the renames didn't end up in unhappyness | 18:56 |
| clarkb | They should follow the redirects | 18:56 |
| clarkb | gitea running now | 18:58 |
| clarkb | project-config on bridge updated properly | 18:59 |
| clarkb | gitea looked good | 19:00 |
| *** openstackgerrit has quit IRC | 19:01 | |
| fungi | yeah, all seems fine so far | 19:01 |
| clarkb | jeepyb is running now and project-config on review was updated | 19:01 |
| clarkb | yup looking good | 19:02 |
| clarkb | review is done | 19:05 |
| clarkb | it is doing afs now | 19:05 |
| clarkb | then I think zuul is in the else block now | 19:06 |
| clarkb | nova is still indexing but we are now down to 8 tasks in the gerrit queue | 19:07 |
| clarkb | other than that indexing I think we are caught up | 19:07 |
| clarkb | zuul should get puppeted soon then we can recheck changes | 19:08 |
| clarkb | timburke: ^ we need to update the zuul main.yaml on the zuul scheduler server that should happen shortly then you can recheck your changes | 19:09 |
| *** openstackgerrit has joined #openstack-infra | 19:10 | |
| openstackgerrit | Slawek Kaplonski proposed zuul/zuul-jobs master: Add role to fetch journal log from test node https://review.opendev.org/643733 | 19:10 |
| fungi | i left a similar comment on 662303 | 19:11 |
| clarkb | puppet is just now starting to run on zuul01 | 19:12 |
| clarkb | and its done | 19:12 |
| fungi | we should be clear to recheck those changes now? i can do that | 19:12 |
| clarkb | https://review.opendev.org/#/c/662225/2 I just rechecked that one as a canary but y a I think you can recheck them all | 19:13 |
| fungi | 662303 was the only other i was aware of | 19:13 |
| clarkb | I think all of AJaeger's fixes above too? | 19:14 |
| AJaeger | fungi, could you review this infra-specs change so that all links work again, please? https://review.opendev.org/#/c/662297/ | 19:14 |
| clarkb | hrm we still have errors in the zuul web ui | 19:14 |
| clarkb | maybe it is loading its config? | 19:14 |
| AJaeger | clarkb: mine should be fine (with exception of system-config) - but I'll double check now | 19:14 |
| clarkb | there are a ton of merger jobs being logged by the scheduler so that may be it | 19:15 |
| AJaeger | clarkb: http://zuul.openstack.org/projects still shows x/pyeclib | 19:15 |
| clarkb | AJaeger: ya I think it hasn't finished reloading its new config yet | 19:15 |
| AJaeger | agreed | 19:16 |
| timburke | thanks for running that down clarkb and AJaeger -- fwiw, i'm not in a huge rush or anything; we don't have anything we're trying to push through on those projects right now | 19:16 |
| AJaeger | and "Last reconfigured" is not new yet... | 19:16 |
| clarkb | according ot the logs I think it is done | 19:17 |
| AJaeger | timburke: It's a broken zuul job that will show up as error on zuul's status page - so, I'd like to see it fixed ;) | 19:17 |
| timburke | fair enough :-) | 19:17 |
| clarkb | but web ui is not updated yet | 19:17 |
| corvus | i think it's still churning | 19:17 |
| *** imacdonn has quit IRC | 19:17 | |
| AJaeger | timburke: but no need for you to stay longer for it - can wait until Monday... | 19:18 |
| *** imacdonn has joined #openstack-infra | 19:18 | |
| clarkb | last reconfigured did just update | 19:18 |
| AJaeger | and we have http://zuul.openstack.org/project/openstack/pyeclib ;) | 19:18 |
| corvus | 2019-05-31 19:18:23,469 DEBUG zuul.ConfigLoader: Created layout id a255741dea884c67ae73f3fbfd12d7a1 | 19:18 |
| AJaeger | \o/ | 19:18 |
| AJaeger | still our 5 config errors ;( | 19:18 |
| corvus | (the delay is the re-enqueing of changes after loading the config) | 19:19 |
| clarkb | AJaeger: ya I think those might be cached? I'm not convinced yet that they are still a problem | 19:19 |
| clarkb | 662225 got far enough to start jobs at least | 19:20 |
| AJaeger | timburke: you can +2A the change if you like - if it passes, it will be fine ;) | 19:21 |
| clarkb | as did pyeclib | 19:21 |
| openstackgerrit | Merged openstack/project-config master: Fix gerrit/projects.yaml sort order https://review.opendev.org/662527 | 19:21 |
| timburke | 🎉 | 19:21 |
| corvus | that's one heck of a cache | 19:21 |
| fungi | "cache" | 19:21 |
| clarkb | oh I see its not complaining about projects not existing anymore | 19:21 |
| clarkb | but the other errors are probably actually errors | 19:21 |
| clarkb | so some of them did go away | 19:21 |
| clarkb | system-config-run-dns is not defined apparently ( as an example) | 19:22 |
| clarkb | and os-http isn't in zuul/main.yaml at all | 19:22 |
| clarkb | so a few things to clean up I guess | 19:22 |
| corvus | agree those look legit | 19:23 |
| clarkb | 662225 fixes system-config-run-dns | 19:24 |
| clarkb | (or should) | 19:24 |
| clarkb | and ya os-http is just removed from project-config/zuul/main.yaml is the reason for that we are retiring it? | 19:25 |
| AJaeger | os-http was moved - so that's wrong ;( | 19:26 |
| * AJaeger sends another hotfix | 19:26 | |
| clarkb | ya it was moved but maybe it is also being retired? Simple fix in either case | 19:26 |
| *** happyhemant has quit IRC | 19:27 | |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Readd x/os-http https://review.opendev.org/662549 | 19:27 |
| AJaeger | clarkb: it was only moved... | 19:28 |
| clarkb | AJaeger: ok so your fix is the correct thing | 19:29 |
| clarkb | it has my +2 | 19:29 |
| corvus | and +3 | 19:30 |
| AJaeger | thanks | 19:30 |
| corvus | AJaeger: thank you | 19:31 |
| AJaeger | could you also review clarkb's record of the renames? https://review.opendev.org/661845 and https://review.opendev.org/#/c/662097 ? | 19:31 |
| corvus | i also +3d the system-config change | 19:31 |
| * AJaeger can approve these | 19:32 | |
| clarkb | I updated the second one with the additional project this morning too | 19:33 |
| clarkb | (so it should be good to go) | 19:33 |
| AJaeger | clarkb: yeah - both approved now. | 19:33 |
| * AJaeger added a few reviews to https://etherpad.openstack.org/p/project-renames-2019-05-31 | 19:34 | |
| clarkb | looking | 19:35 |
| clarkb | AJaeger: do we need to set the var for https://review.opendev.org/#/c/662537/2 in tox.ini ? | 19:37 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: WIP: registry test job https://review.opendev.org/661327 | 19:37 |
| corvus | imma gonna get some lunch | 19:37 |
| clarkb | anyone else want to review https://review.opendev.org/#/c/662297/ whcih is a cleanup for urls in our infra specs? If not I'll just approve it | 19:37 |
| clarkb | also https://review.opendev.org/#/c/662293/1/docs-site/infra-index.html.tmpl for linking to the specs | 19:38 |
| clarkb | I think all of the more urgent once that fix zuul's running config have been approved though which is great | 19:38 |
| *** jcoufal has quit IRC | 19:38 | |
| fungi | hrm, yeah, i can't tell from http://logs.openstack.org/37/662537/2/check/project-config-gerrit/b16c388/job-output.txt.gz#_2019-05-31_18_54_31_175193 whether it actually did anything | 19:40 |
| clarkb | fungi: corvus: I'm going to detach from the screen now | 19:40 |
| AJaeger | clarkb: we could remove that var completely, looks like historic relict | 19:40 |
| clarkb | I'm out of the screen last one in should close it :) | 19:41 |
| AJaeger | fungi: it runs's here http://logs.openstack.org/37/662537/2/check/project-config-gerrit/b16c388/job-output.txt.gz#_2019-05-31_18_55_10_853800 | 19:41 |
| clarkb | AJaeger: I wonder if the intent was to force openstack to follow certain rules but not force stackforge | 19:42 |
| openstackgerrit | Merged openstack/infra-manual master: Follow rename of sandbox repos https://review.opendev.org/662226 | 19:42 |
| clarkb | if the rules are relaxed enough we can probably get away without that distinction and just check for sane things | 19:42 |
| AJaeger | clarkb: it passes everywhere, so we could test it everywhere | 19:42 |
| clarkb | k | 19:42 |
| clarkb | then ya maybe we just want to run it everywhere and if we run into a case where that isn't helpful anymore add in the separation again | 19:43 |
| fungi | AJaeger: yeah, i saw the "are valid" result, but there's no indication of whether it possibly checked 0 acls and all were valid | 19:43 |
| AJaeger | fungi: shall I add some logging? | 19:43 |
| fungi | i guess we can assume it didn't sit on its thumbs for 30 seconds, so i'm okay with it | 19:44 |
| clarkb | #status log Performed repo renames. Some were cleanup from opendev migration and others were normal reorgs. | 19:44 |
| openstackstatus | clarkb: finished logging | 19:44 |
| fungi | the gap suggests there was *something* going on anyway | 19:44 |
| * AJaeger will udpate | 19:45 | |
| fungi | AJaeger: even just a count of acls in the summary might be helpful, but i don't think it's urgent | 19:45 |
| clarkb | I'm going to go for a bike ride and find lunch as soon as the kids get home | 19:45 |
| clarkb | but I think things are looking good and the fixes we need are on their way in | 19:46 |
| fungi | mostly concerned that if the variable in that loop is undefined it will short-circuit silently | 19:46 |
| fungi | whereas at least the static list didn't run the same risk | 19:46 |
| clarkb | ya | 19:47 |
| *** slaweq has quit IRC | 19:48 | |
| clarkb | Thanks everyone for the help on this. Much smoother than last time :) | 19:48 |
| clarkb | and with the bug fixes we pushed up hopefully even better the next time | 19:48 |
| fungi | yes, huge thanks to everyone! | 19:48 |
| fungi | i need to go run some errands i skipped during lunch, but glad to see things are basically working now | 19:49 |
| clarkb | I'll be out for probably an hour or so in just a few minutes. When I get back will help ensure the rest of these fixups get in and if I have time take a quick look at why arm64 buster images aren't happening | 19:50 |
| openstackgerrit | Merged openstack/project-config master: Readd x/os-http https://review.opendev.org/662549 | 19:50 |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Check ACLs for all namespaces https://review.opendev.org/662537 | 19:53 |
| AJaeger | fungi: should be much better now ^ | 19:53 |
| *** slaweq has joined #openstack-infra | 19:54 | |
| *** hjensas|afk is now known as hjensas | 19:54 | |
| AJaeger | timburke: and pyeclib jobs work fine ;) - on https://review.opendev.org/662303 | 19:54 |
| *** dmsimard has quit IRC | 19:55 | |
| *** dmsimard has joined #openstack-infra | 19:55 | |
| *** dmsimard has quit IRC | 19:56 | |
| fungi | AJaeger: ooh, awesome thanks! | 19:56 |
| *** eharney has joined #openstack-infra | 19:57 | |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Retire opendev-website (step 3) https://review.opendev.org/661803 | 19:57 |
| AJaeger | and this retires last openstack-infra repo ^ | 19:57 |
| *** mattw4 has joined #openstack-infra | 19:58 | |
| *** slaweq has quit IRC | 19:58 | |
| *** dmsimard has joined #openstack-infra | 20:01 | |
| *** dmsimard has quit IRC | 20:01 | |
| *** Ng_ is now known as Ng | 20:03 | |
| fungi | okay, popping back out to run errands now | 20:04 |
| *** e0ne has joined #openstack-infra | 20:07 | |
| *** xek_ has joined #openstack-infra | 20:11 | |
| *** e0ne has quit IRC | 20:12 | |
| *** e0ne has joined #openstack-infra | 20:13 | |
| AJaeger | so, Zuul has been reconfigured and now knows x/os-http - one error less ;) | 20:14 |
| * AJaeger waves good night | 20:16 | |
| clarkb | good night. thanjs again | 20:19 |
| *** dmsimard has joined #openstack-infra | 20:21 | |
| *** dpawlik has joined #openstack-infra | 20:23 | |
| corvus | WOW! https://review.opendev.org/661327 works :) | 20:24 |
| corvus | that's one doozy of a job. note the 145 tasks in the main playbook | 20:24 |
| corvus | there's still one TODO in there, which is to figure out a way to verify the image has been pushed to the intermediate registry. | 20:27 |
| corvus | probably can do an ansible uri module call or something | 20:27 |
| corvus | however the push command did run: http://logs.openstack.org/27/661327/42/check/zuul-jobs-test-registry/e42ceca/ara-report/result/3441ba5e-5dd5-4226-b8ee-523588dd544f/ | 20:28 |
| *** bobh has quit IRC | 20:29 | |
| *** e0ne has quit IRC | 20:30 | |
| openstackgerrit | Merged opendev/system-config master: Follow opendev renames https://review.opendev.org/662225 | 20:30 |
| *** dpawlik has quit IRC | 20:31 | |
| corvus | looks like the only remaining errors are in pyeclib | 20:33 |
| *** rkukura has quit IRC | 20:45 | |
| *** rfolco has quit IRC | 20:45 | |
| *** kjackal has quit IRC | 20:47 | |
| *** dklyle has quit IRC | 20:58 | |
| clarkb | and pyeclib fixes ahve since merged. No more errors | 21:10 |
| clarkb | do we need to restart nodepool builders to pick up new images? I wonder if that is why we don't have debian buster for arm64 yer | 21:13 |
| corvus | clarkb: not under normal circumstances | 21:13 |
| corvus | it should still watch its config file | 21:14 |
| clarkb | hrm the config for it is complete as far as I can tell. We have a label, provider, and diskimage | 21:14 |
| clarkb | and that file is up to date on nb03.o.o | 21:14 |
| clarkb | hrm we have filled the disk on that builder though | 21:14 |
| clarkb | I'll start by cleaning that up | 21:15 |
| clarkb | ok I think part of the issue here is we have >2 ready iamges for each image on that builder I'm guessing because of the clouds that went away | 21:18 |
| clarkb | fungi: any idea where we are in cleaning those up? | 21:19 |
| clarkb | I think mordreds reorg of clouds.yamls may actually clean some of that up but probably not on the nodepool config side? | 21:19 |
| clarkb | this may end up being a next week thing as the thread is already getting long :) | 21:19 |
| clarkb | at this point I expect we'll have to sanitize the db manually since the cloud went poof | 21:21 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: WIP: registry test job https://review.opendev.org/661327 | 21:21 |
| corvus | clarkb: if you have a minute, can you take a quick look at that? ^ it should be working now (previous patchset passed, latest is cleanup and a bit more checking). i think the next thing is to find a permanent home for it, but it'd be good to get a second set of eyes on it now before i go too far down that road. | 21:23 |
| clarkb | can do | 21:23 |
| corvus | in particular, feel free to say "wtf add comments here" :) | 21:24 |
| *** rkukura has joined #openstack-infra | 21:25 | |
| clarkb | corvus: seems sane. There are a couple bits that I wonder if we need them to be test specific like setting up certs and user credentials. Don't the existing roles already do that? | 21:34 |
| *** jtomasek has quit IRC | 21:37 | |
| corvus | clarkb: not quite in exactly the way we need for the test. i would like to refactor them so we can reduce that duplication. we could either continue that work on this change, or after we create the job. | 21:39 |
| corvus | (i didn't want to do it before now because i wanted testing for that kind of refactoring :) | 21:40 |
| clarkb | that seems like a reasonable compromise | 21:40 |
| *** Lucas_Gray has joined #openstack-infra | 21:43 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: WIP: registry test job https://review.opendev.org/661327 | 21:48 |
| *** xek_ has quit IRC | 21:48 | |
| *** pcaruana has quit IRC | 21:54 | |
| *** sreejithp has quit IRC | 22:08 | |
| *** sreejithp has joined #openstack-infra | 22:09 | |
| clarkb | fungi: if you are still around https://review.opendev.org/#/c/662511/ would be a good one to get in before we forget | 22:10 |
| *** hwoarang has quit IRC | 22:15 | |
| *** hwoarang has joined #openstack-infra | 22:16 | |
| *** _erlon_ has quit IRC | 22:19 | |
| *** sreejithp has quit IRC | 22:20 | |
| *** rkukura has quit IRC | 22:31 | |
| *** rkukura has joined #openstack-infra | 22:31 | |
| *** rlandy has quit IRC | 22:32 | |
| *** rkukura has quit IRC | 22:34 | |
| *** yamamoto has joined #openstack-infra | 22:44 | |
| *** yamamoto has quit IRC | 22:48 | |
| *** michael-beaver has quit IRC | 22:52 | |
| fungi | i was not around but am back now and taking a look | 23:01 |
| *** rfarr_ has quit IRC | 23:08 | |
| clarkb | I'm doing another pass of the changes AJaeger put on the etherpad now | 23:09 |
| clarkb | make sure nothing needs a recheck or new review | 23:09 |
| clarkb | fungi: do you want to look at https://review.opendev.org/#/c/662297/ before I approve it? cleanup of infra-specs for new urls | 23:13 |
| *** mattw4 has quit IRC | 23:15 | |
| fungi | yeah, looking now | 23:17 |
| clarkb | I annotated the etherpad with the current state of things for all the changes there | 23:17 |
| corvus | i approved the remaining ones | 23:21 |
| clarkb | \o/ | 23:22 |
| clarkb | spec update now approved too | 23:22 |
| clarkb | hopefully that means AJaeger returns to work on monday with a slightly cleaner slate/smaller backlog | 23:23 |
| *** aaronsheffield has quit IRC | 23:23 | |
| *** raissa has quit IRC | 23:24 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: WIP: registry test job https://review.opendev.org/661327 | 23:26 |
| openstackgerrit | Merged opendev/infra-specs master: Update site URL https://review.opendev.org/662297 | 23:33 |
| *** panda has quit IRC | 23:37 | |
| openstackgerrit | Merged opendev/system-config master: Fix gitea rename playbook org creation https://review.opendev.org/662511 | 23:37 |
| clarkb | looking at the calendar I'll refresh certs week after next | 23:42 |
| openstackgerrit | Merged openstack/project-config master: Check ACLs for all namespaces https://review.opendev.org/662537 | 23:43 |
| openstackgerrit | Merged openstack/project-config master: Remove unused ACLs https://review.opendev.org/661913 | 23:43 |
| openstackgerrit | Merged openstack/project-config master: Use retired ACL everywhere https://review.opendev.org/662223 | 23:43 |
| openstackgerrit | Merged openstack/project-config master: Update infra-specs link https://review.opendev.org/662293 | 23:43 |
| *** gyee has quit IRC | 23:44 | |
| *** rh-jelabarre has quit IRC | 23:53 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!