Wednesday, 2020-11-18

clarkbgotcha00:01
openstackgerritClark Boylan proposed opendev/system-config master: Omnibus Gerrit 3.2 changes  https://review.opendev.org/76289500:05
openstackgerritClark Boylan proposed opendev/system-config master: Clean up old Gerrit html theming and commentlinks  https://review.opendev.org/75716100:05
openstackgerritClark Boylan proposed opendev/system-config master: Remove reviewdb config from Gerrit  https://review.opendev.org/75716200:05
clarkbfungi: ^ reabsed on top of your merged change now00:05
clarkband I've re WIP'd that stack again00:06
openstackgerritIan Wienand proposed opendev/system-config master: Migrate codesearch site to container  https://review.opendev.org/76296000:25
ianwclarkb/fungi: ^ i know it's probably a bit much to chew on now, but ^ should be ready for review00:25
clarkbya, that may need to be a morning activity00:26
openstackgerritIan Wienand proposed opendev/system-config master: Migrate codesearch site to container  https://review.opendev.org/76296000:39
*** hamalq has quit IRC00:41
openstackgerritIan Wienand proposed opendev/system-config master: Migrate codesearch site to container  https://review.opendev.org/76296000:44
*** mlavalle has quit IRC01:35
*** stevebaker has quit IRC01:59
kevinzianw: ping02:18
kevinzianw: Good noon! I saw that in Linaro UK we have an instance: mirror01.london.linaro-london.openstack.org, I wonder it is still needed or we can remove it?02:18
kevinzianw: And also, the nb03.openstack.org node02:19
ianwkevinz: hey, we can remove those, we're not using that region any more02:19
kevinzianw: Cool, thanks for clarify02:19
*** stevebaker has joined #opendev02:37
openstackgerritIan Wienand proposed opendev/system-config master: test if ansible-core install works  https://review.opendev.org/76309902:55
*** brinzhang has joined #opendev03:03
openstackgerritJeremy Stanley proposed zuul/zuul-jobs master: validate-host: Options to require v4 and v6 routes  https://review.opendev.org/76306503:08
openstackgerritIan Wienand proposed opendev/system-config master: test if ansible-core install works  https://review.opendev.org/76309903:17
*** stevebaker has quit IRC03:50
openstackgerritIan Wienand proposed opendev/system-config master: devel job: use ansible-core name  https://review.opendev.org/76309903:51
ianwdmsimard / fungi : ^ that should fix the devel job04:01
*** ykarel has joined #opendev04:03
*** fressi has joined #opendev05:02
*** roman_g has joined #opendev05:21
*** roman_g has quit IRC05:27
*** marios has joined #opendev06:14
*** rpittau|afk is now known as rpittau06:49
*** icey has quit IRC06:52
*** icey has joined #opendev06:54
*** fressi has quit IRC06:54
*** sboyron has joined #opendev07:02
*** DSpider has joined #opendev07:16
*** ralonsoh has joined #opendev07:31
*** clayg has quit IRC07:35
*** clayg has joined #opendev07:35
*** zbr4 has quit IRC07:38
*** roman_g has joined #opendev07:40
*** slaweq has joined #opendev07:43
*** eolivare has joined #opendev07:44
*** fressi has joined #opendev07:47
*** dtantsur|afk is now known as dtantsur07:51
*** hashar has joined #opendev07:58
ykarelcan https://review.opendev.org/#/c/763038/1 be cleared to avoid jobs hitting it08:01
*** stevebaker has joined #opendev08:01
*** marios is now known as marios|ruck08:08
*** andrewbonney has joined #opendev08:16
*** zbr has joined #opendev08:25
openstackgerritMerged openstack/project-config master: Disable limestone provider due to IPv4-less nodes  https://review.opendev.org/76303808:42
*** iurygregory has quit IRC08:44
*** tosky has joined #opendev08:45
*** iurygregory has joined #opendev08:50
*** ykarel_ has joined #opendev09:04
*** mgoddard has joined #opendev09:08
*** ykarel has quit IRC09:08
*** parallax has quit IRC09:09
*** parallax has joined #opendev09:09
*** slaweq has quit IRC09:17
*** slaweq has joined #opendev09:19
*** yoctozepto has quit IRC09:50
*** brinzhang_ has joined #opendev09:51
*** yoctozepto has joined #opendev09:51
*** brinzhang has quit IRC09:54
*** marios|ruck has quit IRC10:06
*** ykarel_ is now known as ykarel10:11
*** bolg has joined #opendev10:25
*** kevinz has quit IRC10:26
*** icey has quit IRC10:31
*** icey has joined #opendev10:32
*** hemanth_n has quit IRC11:04
*** icey has quit IRC11:05
*** marios has joined #opendev11:08
*** marios is now known as marios|ruck11:08
*** lpetrut has joined #opendev11:10
*** icey has joined #opendev11:26
*** slaweq has quit IRC11:39
*** icey has quit IRC11:41
*** auristor has quit IRC11:43
*** auristor has joined #opendev11:43
*** icey has joined #opendev11:43
*** slaweq has joined #opendev11:44
*** icey has quit IRC11:49
*** icey has joined #opendev11:55
*** icey has quit IRC12:23
*** icey has joined #opendev12:24
*** icey has quit IRC12:50
*** dtantsur is now known as dtantsur|brb13:06
*** ykarel has quit IRC13:17
*** ykarel has joined #opendev13:47
*** jaicaa has quit IRC14:14
*** dtantsur|brb is now known as dtantsur14:22
*** ralonsoh has left #opendev14:43
*** chandankumar is now known as raukadah14:45
*** fressi has quit IRC15:10
*** lpetrut has quit IRC15:21
*** mlavalle has joined #opendev15:25
openstackgerritJeremy Stanley proposed zuul/zuul-jobs master: validate-host: Options to require v4 and v6 routes  https://review.opendev.org/76306516:02
clarkbfungi: thinking out loud while drinking tea: if/when we restart gerrit to double check happyness with those config options we should ensure that gerrit doesn't rewrite the underlying file and remove our config options16:08
clarkbfungi: if it does do that beacuse it doesn't recognize those options then we need to be sure they are in place before we start the upgrade16:09
clarkb(I really wish programs didn't change their config files it is so unfriendly to operations)16:09
* clarkb scribbles a note on the etherpad to check the config updates are in place before we start16:11
*** ykarel has quit IRC16:35
fungithat's a great point16:48
clarkbfungi: there are a number of release-post things queued up according to zuul, should we coordinate with the openstack release team really quickly to find a good time for a quick gerrit restart?16:53
*** marios|ruck has quit IRC16:54
fungimaybe, i'll see if they've cleared their review backlog17:13
fungii've asked in #openstack-release17:15
openstackgerritThierry Carrez proposed opendev/irc-meetings master: Push back Large Scale SIG meeting by one hour  https://review.opendev.org/76322717:20
clarkbfungi: if it does clear out those options I think we can land https://review.opendev.org/#/c/758599/ and ansible will put them back for us17:22
*** mgoddard has quit IRC17:33
*** jaicaa has joined #opendev17:43
fungimgagne: any chance you're able to get folks to clean up rogue server instances in inap? we had to stop using it nearly a month ago, and when we tried to reenable it a couple of weeks back we had to immediately turn it off again because it was still unstable17:47
clarkbfungi: semi related were you still planning to add testing to the traceroute check chagne?17:49
clarkbI don't want to approve it unless it has been tested somehow but not sure if that iwll happen in change or via base-test17:50
fungiyeah, i'm not entirely clear on what testing we do or can do in zuul-jobs for that role. i'll take another look real quickly17:51
fungilooks like the extent of pre-merge testing for it is that we execute the role supplying a minimum number of required parameters and make sure it doesn't fail18:00
clarkbin that case the default is unlikely to have regressed :)18:01
fungiactually testing the "fail if missing ipv4/fail if missing ipb6" scenarios would require fiddling with the node's networking or mocking significant parts of the network stack18:01
clarkbcan probably do a child change that sets those parameters then inspect how it reacts without actually merging it?18:01
clarkbsince the outcomes will depend on which cloud you land on18:01
fungiyeah, i'll get that pushed up18:02
fungishould i test requiring v6 routes initially? we don't currently have any providers where we expect v4 routes to be missing18:07
*** hamalq has joined #opendev18:07
clarkbya18:07
clarkbcan you run it multiple times in the same play?18:07
clarkbif so I would do v4 required then v6 required then both18:08
clarkband depending on where we land we should get decent coverage?18:08
fungisure, i think i can just include it multiple times18:08
*** eolivare has quit IRC18:10
openstackgerritJeremy Stanley proposed opendev/base-jobs master: base-test: exercise validate-host IP family assert  https://review.opendev.org/76324118:13
clarkbhrm since that won't do premerge testing I'm not sure my previous that that we havne't regressed is correct18:14
*** dtantsur is now known as dtantsur|afk18:15
openstackgerritJeremy Stanley proposed opendev/bindep master: DNM: exercise base-test  https://review.opendev.org/76324318:16
fungiclarkb: so we need to first merge https://review.opendev.org/763065 and then that and after that i can recheck the dnm job?18:17
clarkbfungi: yes, except that 763065 will immediately apply to all jobs that start after that merge18:17
clarkbif we could check that we don't regress first then I'd be ok with that, but I don't think we can? Let me look closer18:18
fungiin theory we can rerun validate host in an untrusted job?18:19
clarkbfungi: ok I think the file you want to modify is zuul/zuul-jobs/test-playbooks/base-roles/validate-host.yaml18:19
fungiwith a depends-on18:19
clarkbif you modify ^ that is run pre merge18:19
fungiahh18:19
clarkbrather than modifying base-test18:19
clarkband the presence of that playbook does imply that the non regression check case is covered, but doing that testing in zuul-jobs (even if we don't merge teh chagne due to provider differences) is still simpler18:20
fungithough for the thorough test, i guess i could just rerun the role in the run phase of a job right?18:20
fungiwith a depends-on to the zuul-jobs change18:21
clarkbzuul/zuul-jobs/test-playbooks/base-roles/validate-host.yaml is poart of the run playbook for the job it runs in18:21
clarkb(I'm not sure I understand the question)18:21
clarkbI think both approaches will give us approximately the same test coverage but doing it in zuul-jobs will be simpler as it is pre merge18:21
fungiyeah, i guess i could make a dnm change to rerun validate-host with multiple options in zuul/zuul-jobs/test-playbooks/base-roles/validate-host.yaml18:21
fungiis what i was suggesting18:21
clarkbyup exactly that as a child of your change18:22
fungibut i misread your initial suggestion as still involving base-test in some way18:22
fungiokay, got it18:22
clarkbsometimes base-test is required, I wasn't sure in this case18:22
clarkbif it is required due to privileges or whatever then we have to do that dance but if it isnt' we should avoid it18:22
clarkb~3 minutes to release jobs being done I think18:23
openstackgerritJeremy Stanley proposed zuul/zuul-jobs master: DNM: Exercise validate-host IP family assert  https://review.opendev.org/76324518:25
fungirelease and release-post are both empty now18:29
fungiianw: looks like you have a couple of root screen sessions on review.o.o created a week ago with borg backup work going on in them. are you still using those or can they be closed out?18:31
fungistatus log The Gerrit service at review.opendev.org is being restarted quickly as a pre-upgrade sanity check, estimated downtime is less than 5 minutes.18:33
clarkblgtm18:33
*** fressi has joined #opendev18:33
fungier, make that status notice18:33
clarkboh ya we want people to see it18:33
clarkbnotice++18:33
fungido we want to do that now or wait for anything else?18:34
clarkbI can't think of anything else to wait on18:34
fungi#status notice The Gerrit service at review.opendev.org is being restarted quickly as a pre-upgrade sanity check, estimated downtime is less than 5 minutes.18:34
openstackstatusfungi: sending notice18:34
fungii'll down the container and then re-up it18:34
-openstackstatus- NOTICE: The Gerrit service at review.opendev.org is being restarted quickly as a pre-upgrade sanity check, estimated downtime is less than 5 minutes.18:34
clarkbk, I'm around if anything comes up18:34
fungiit's starting now18:35
fungilog says it's up18:36
clarkbstill waiting for apache to agree (thats normal to lag)18:37
fungii'm getting a webui now18:37
clarkbme too18:37
openstackstatusfungi: finished sending notice18:37
clarkband our config options appear to have remained in place \o/18:38
fungiexcellent18:38
fungialso i *think* tail -F is the solution to following the error_log before gerrit reopens it18:39
fungiat least it seemed to work for me this time18:39
*** rpittau is now known as rpittau|afk18:39
clarkboh neat18:40
clarkbI'll have to use -F now and get rid of that -f memory18:40
fungii'm not entirely convinced, may have merely been coincident timing on my part18:41
fungilooks like gerritbot needs a restart19:01
*** openstackgerrit has quit IRC19:02
fungiit's restarting now19:03
*** andrewbonney has quit IRC19:04
*** roman_g has quit IRC19:15
sean-k-mooneyclarkb: fungi did ye push an update to the zuul ui recently19:36
sean-k-mooneyit looks like its not rendering the logs properly anymore19:36
sean-k-mooneyhttps://zuul.opendev.org/t/openstack/build/00ff397ca5374a0b9441036d43ee3416/log/controller/logs/screen-n-cpu.txt#737119:36
sean-k-mooneythat should be the first line of a multiple output19:37
sean-k-mooneybasically the guest libvirt xml19:37
sean-k-mooneyits there in the raw view but truncated in the app view19:37
clarkbI havent but there has been ui work19:37
sean-k-mooneywell when i ment pushed i ment deployed a new version to production19:38
fungilooks like it's broken horizontal scrolling as well... i don't get a horizontal scroll bar unless i scroll to the end of the page first19:39
fungisean-k-mooney: it's continuously deployed19:39
sean-k-mooney yes that too19:39
sean-k-mooneyyou cant use keyborad arrows anymore19:39
clarkbbut not continuously restarted iirc19:39
sean-k-mooneyso if you search and it scrolls you have to go to the bottom and scoll back19:40
sean-k-mooneyits kind of annoying but i didnt know if i was the only one that did that19:40
clarkbBut I've been too busy too follow all the recent ui discussion19:40
clarkbmaybe ask in #zuul and we can traceback from there19:40
sean-k-mooneyi was just going to ask should i bring it up there instead19:41
fungii just mentioned it in there19:41
ianwfungi: any backups sessions can be cleared, thanks19:42
fungino worries, appreciate it19:43
*** sboyron has quit IRC19:52
ianwfungi / clarkb : if you have a little time for the codesearch container i can work on deployment https://review.opendev.org/#/c/762960/; otherwise i might make the current job non-voting for something to avoid the puppet failures19:55
clarkbYa I can review it after lunch19:56
*** sboyron has joined #opendev20:04
*** openstackgerrit has joined #opendev20:04
openstackgerritJeremy Stanley proposed openstack/project-config master: Require working IPv4 on all job nodes  https://review.opendev.org/76325320:04
openstackgerritJeremy Stanley proposed openstack/project-config master: Revert "Disable limestone provider due to IPv4-less nodes"  https://review.opendev.org/76325420:04
fungiinfra-root: ^ that stack will get us back a bit of node capacity... our available quota is rather strained at this point20:46
*** mgoddard has joined #opendev20:50
clarkbwere we able to confirm v8a testing the new stuff is happy?20:55
fungiyeah, i left some comments20:56
fungiseems to work as designed20:56
fungihowever it does highlight that we have almost no providers with ipv6-connected nodes at present20:56
*** slaweq has quit IRC20:58
fungi(10 out of 11 role tests in the dnm change failed due to lack of ipv6 connectivity)20:58
clarkbgreat, I'm double checking test results then will probably go through and approve things. corvus you had previously reviewed https://review.opendev.org/#/c/763065/ as well, want to give that another look?20:58
fungithey seemed to all succeed on the ipv4-required plays though20:59
fungiwhich is what we expect20:59
fungiamusingly, gentoo nodes seem to pick up working ipv6 addresses/routes in rackspace while other distros aren't21:01
fungithat may point to a shortcoming in glean21:01
clarkbI've +2'd the three non WIP/DNM changes. Maybe give corvus a few minutes to chime in on the zuul-jobs change again if interested then start landing them?21:02
fungii can't recall which network management method gentoo ends up using21:02
fungiprometheanfire: ^ ?21:02
clarkbyes it is glean issue since ipv6 is statically configured21:02
clarkbgentoo uses the systemd setup21:02
clarkbI think glean should statically configure ipv6 on debuntu and gentoo but none of the rpm distros21:03
fungiahh,systemd-networkd?21:03
clarkbya that thing21:03
fungiit's possible all the debuntu nodes from that buildset landed in ovh or something21:03
fungiyep, that was it exactly, all 4 debuntu builds landed in ovh21:05
fungiso those might have also succeeded if they'd run in rax21:05
clarkbianw: left comments on the code search change. The approach seems fine and most of the issues are minor21:15
clarkbianw: I think its mostly about cleaning things up then we're good21:15
*** DSpider has quit IRC21:20
ianwclarkb: thanks, will llook in on21:29
corvusclarkb, fungi: 763065 approved after having a fascinating conversation with myself.21:29
clarkbcorvus: do we need to do a meetpad call so that we can hear human voices again?21:30
corvusclarkb: we're human?21:30
*** slaweq has joined #opendev21:30
clarkbgood point. I think I'm turning into the human equivalent of a gollum21:31
clarkbespecially now with all the rain arriving the sun probably won't be seen again for months21:36
openstackgerritSlawek Kaplonski proposed zuul/zuul-jobs master: [multi-node-bridge] Add script to configure connectivity  https://review.opendev.org/76265021:37
*** mgoddard has quit IRC21:44
openstackgerritMerged zuul/zuul-jobs master: validate-host: Options to require v4 and v6 routes  https://review.opendev.org/76306521:45
clarkbfungi: ^ I think you can start to land the other changes now21:45
fungiwill changes to zuul/site-variables.yaml be picked up automatically once deployed?21:49
clarkbyes they are included in new asnible runes iirc21:49
fungitrying to gauge how long to wait before approving the limestone removal revert21:49
clarkbwow typing is difficult21:49
clarkbabsically zuul does ansible -e that file -e other stuff iirc21:49
clarkbso once that is in place on the executors we should see it for all the jobs21:50
* fungi consults the runes of asnible21:50
fungiahh, right21:52
openstackgerritMerged openstack/project-config master: Require working IPv4 on all job nodes  https://review.opendev.org/76325321:54
fungiso in theory i could approve both at the same time since one gets deployed to the executors and the other to a launcher and the problem is unlikely to recur in any window where nodes were booted by the launcher and then got handed off to a not-yet-updated executor to run a job21:54
fungibut probably still better to wait and make sure the first change gets deployed to all the executors before i approve the one for the launcher21:55
clarkbya and you can manually update the launcher to speed things up once we're sure the executor is happy?21:55
fungii guess, it's not a huge rush though, just trying to get back a bit of capacity (50 nodes)21:56
clarkb++21:56
fungicorvus: you have an autohold with the comment "debug multi-stage multi-arch" which has three nodes held for 131 days... are you still using those?22:01
corvusnope!  sorry22:02
fungino worries, just in there cleaning up, i can remove22:02
fungimnaser: you have an autohold with the comment "debug multi-arch containers" which has a node held for 117 days... are you still using that?22:02
fungiianw: you have an autohold for "fedora containerfile" (not sure how old) which doesn't happen to have caught any nodes. still needed?22:05
*** slaweq has quit IRC22:16
*** sboyron has quit IRC22:22
*** johnsom has quit IRC22:27
*** johnsom has joined #opendev22:29
*** hashar has quit IRC22:39
ianwfungi: oh, that can go.  i thought it was going to fail and it passed :)22:43
fungicool, i'll clear it out, thanks for confirming22:46
prometheanfirefungi: gentoo image uses networkd22:57
prometheanfirefor debian hosts I manage I use networkd too, I mean, if you are gonna use systemd anywayway...22:58
mordredif you're going to replace one thing with systemd, why not replace everything right?23:10
* fungi replaced systemd with systemd23:11
fungiprometheanfire: thanks, and yeah it turns out the buildset just split coincidentally between providers so all the debuntu builds ran somewhere without working ipv6 and so gentoo was the only one which passed23:12
ianwclarkb: thanks for the comments, i can work on that23:54
ianwfor supervisord, i added that.  the reason is so that we can restart houndd in the container when the config changes, without restarting the whole container23:54
ianwthe reason for that was to keep the update scripts inside the container23:54
clarkbgotcha23:55
clarkbhacky sighup I guess23:55
ianwyeah; they've got a inotify watcher under review for the config file anyway23:56
ianwso hopefully we can modify it to just update the config file and that's it23:56

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!