clarkb | gotcha | 00:01 |
---|---|---|
openstackgerrit | Clark Boylan proposed opendev/system-config master: Omnibus Gerrit 3.2 changes https://review.opendev.org/762895 | 00:05 |
openstackgerrit | Clark Boylan proposed opendev/system-config master: Clean up old Gerrit html theming and commentlinks https://review.opendev.org/757161 | 00:05 |
openstackgerrit | Clark Boylan proposed opendev/system-config master: Remove reviewdb config from Gerrit https://review.opendev.org/757162 | 00:05 |
clarkb | fungi: ^ reabsed on top of your merged change now | 00:05 |
clarkb | and I've re WIP'd that stack again | 00:06 |
openstackgerrit | Ian Wienand proposed opendev/system-config master: Migrate codesearch site to container https://review.opendev.org/762960 | 00:25 |
ianw | clarkb/fungi: ^ i know it's probably a bit much to chew on now, but ^ should be ready for review | 00:25 |
clarkb | ya, that may need to be a morning activity | 00:26 |
openstackgerrit | Ian Wienand proposed opendev/system-config master: Migrate codesearch site to container https://review.opendev.org/762960 | 00:39 |
*** hamalq has quit IRC | 00:41 | |
openstackgerrit | Ian Wienand proposed opendev/system-config master: Migrate codesearch site to container https://review.opendev.org/762960 | 00:44 |
*** mlavalle has quit IRC | 01:35 | |
*** stevebaker has quit IRC | 01:59 | |
kevinz | ianw: ping | 02:18 |
kevinz | ianw: Good noon! I saw that in Linaro UK we have an instance: mirror01.london.linaro-london.openstack.org, I wonder it is still needed or we can remove it? | 02:18 |
kevinz | ianw: And also, the nb03.openstack.org node | 02:19 |
ianw | kevinz: hey, we can remove those, we're not using that region any more | 02:19 |
kevinz | ianw: Cool, thanks for clarify | 02:19 |
*** stevebaker has joined #opendev | 02:37 | |
openstackgerrit | Ian Wienand proposed opendev/system-config master: test if ansible-core install works https://review.opendev.org/763099 | 02:55 |
*** brinzhang has joined #opendev | 03:03 | |
openstackgerrit | Jeremy Stanley proposed zuul/zuul-jobs master: validate-host: Options to require v4 and v6 routes https://review.opendev.org/763065 | 03:08 |
openstackgerrit | Ian Wienand proposed opendev/system-config master: test if ansible-core install works https://review.opendev.org/763099 | 03:17 |
*** stevebaker has quit IRC | 03:50 | |
openstackgerrit | Ian Wienand proposed opendev/system-config master: devel job: use ansible-core name https://review.opendev.org/763099 | 03:51 |
ianw | dmsimard / fungi : ^ that should fix the devel job | 04:01 |
*** ykarel has joined #opendev | 04:03 | |
*** fressi has joined #opendev | 05:02 | |
*** roman_g has joined #opendev | 05:21 | |
*** roman_g has quit IRC | 05:27 | |
*** marios has joined #opendev | 06:14 | |
*** rpittau|afk is now known as rpittau | 06:49 | |
*** icey has quit IRC | 06:52 | |
*** icey has joined #opendev | 06:54 | |
*** fressi has quit IRC | 06:54 | |
*** sboyron has joined #opendev | 07:02 | |
*** DSpider has joined #opendev | 07:16 | |
*** ralonsoh has joined #opendev | 07:31 | |
*** clayg has quit IRC | 07:35 | |
*** clayg has joined #opendev | 07:35 | |
*** zbr4 has quit IRC | 07:38 | |
*** roman_g has joined #opendev | 07:40 | |
*** slaweq has joined #opendev | 07:43 | |
*** eolivare has joined #opendev | 07:44 | |
*** fressi has joined #opendev | 07:47 | |
*** dtantsur|afk is now known as dtantsur | 07:51 | |
*** hashar has joined #opendev | 07:58 | |
ykarel | can https://review.opendev.org/#/c/763038/1 be cleared to avoid jobs hitting it | 08:01 |
*** stevebaker has joined #opendev | 08:01 | |
*** marios is now known as marios|ruck | 08:08 | |
*** andrewbonney has joined #opendev | 08:16 | |
*** zbr has joined #opendev | 08:25 | |
openstackgerrit | Merged openstack/project-config master: Disable limestone provider due to IPv4-less nodes https://review.opendev.org/763038 | 08:42 |
*** iurygregory has quit IRC | 08:44 | |
*** tosky has joined #opendev | 08:45 | |
*** iurygregory has joined #opendev | 08:50 | |
*** ykarel_ has joined #opendev | 09:04 | |
*** mgoddard has joined #opendev | 09:08 | |
*** ykarel has quit IRC | 09:08 | |
*** parallax has quit IRC | 09:09 | |
*** parallax has joined #opendev | 09:09 | |
*** slaweq has quit IRC | 09:17 | |
*** slaweq has joined #opendev | 09:19 | |
*** yoctozepto has quit IRC | 09:50 | |
*** brinzhang_ has joined #opendev | 09:51 | |
*** yoctozepto has joined #opendev | 09:51 | |
*** brinzhang has quit IRC | 09:54 | |
*** marios|ruck has quit IRC | 10:06 | |
*** ykarel_ is now known as ykarel | 10:11 | |
*** bolg has joined #opendev | 10:25 | |
*** kevinz has quit IRC | 10:26 | |
*** icey has quit IRC | 10:31 | |
*** icey has joined #opendev | 10:32 | |
*** hemanth_n has quit IRC | 11:04 | |
*** icey has quit IRC | 11:05 | |
*** marios has joined #opendev | 11:08 | |
*** marios is now known as marios|ruck | 11:08 | |
*** lpetrut has joined #opendev | 11:10 | |
*** icey has joined #opendev | 11:26 | |
*** slaweq has quit IRC | 11:39 | |
*** icey has quit IRC | 11:41 | |
*** auristor has quit IRC | 11:43 | |
*** auristor has joined #opendev | 11:43 | |
*** icey has joined #opendev | 11:43 | |
*** slaweq has joined #opendev | 11:44 | |
*** icey has quit IRC | 11:49 | |
*** icey has joined #opendev | 11:55 | |
*** icey has quit IRC | 12:23 | |
*** icey has joined #opendev | 12:24 | |
*** icey has quit IRC | 12:50 | |
*** dtantsur is now known as dtantsur|brb | 13:06 | |
*** ykarel has quit IRC | 13:17 | |
*** ykarel has joined #opendev | 13:47 | |
*** jaicaa has quit IRC | 14:14 | |
*** dtantsur|brb is now known as dtantsur | 14:22 | |
*** ralonsoh has left #opendev | 14:43 | |
*** chandankumar is now known as raukadah | 14:45 | |
*** fressi has quit IRC | 15:10 | |
*** lpetrut has quit IRC | 15:21 | |
*** mlavalle has joined #opendev | 15:25 | |
openstackgerrit | Jeremy Stanley proposed zuul/zuul-jobs master: validate-host: Options to require v4 and v6 routes https://review.opendev.org/763065 | 16:02 |
clarkb | fungi: thinking out loud while drinking tea: if/when we restart gerrit to double check happyness with those config options we should ensure that gerrit doesn't rewrite the underlying file and remove our config options | 16:08 |
clarkb | fungi: if it does do that beacuse it doesn't recognize those options then we need to be sure they are in place before we start the upgrade | 16:09 |
clarkb | (I really wish programs didn't change their config files it is so unfriendly to operations) | 16:09 |
* clarkb scribbles a note on the etherpad to check the config updates are in place before we start | 16:11 | |
*** ykarel has quit IRC | 16:35 | |
fungi | that's a great point | 16:48 |
clarkb | fungi: there are a number of release-post things queued up according to zuul, should we coordinate with the openstack release team really quickly to find a good time for a quick gerrit restart? | 16:53 |
*** marios|ruck has quit IRC | 16:54 | |
fungi | maybe, i'll see if they've cleared their review backlog | 17:13 |
fungi | i've asked in #openstack-release | 17:15 |
openstackgerrit | Thierry Carrez proposed opendev/irc-meetings master: Push back Large Scale SIG meeting by one hour https://review.opendev.org/763227 | 17:20 |
clarkb | fungi: if it does clear out those options I think we can land https://review.opendev.org/#/c/758599/ and ansible will put them back for us | 17:22 |
*** mgoddard has quit IRC | 17:33 | |
*** jaicaa has joined #opendev | 17:43 | |
fungi | mgagne: any chance you're able to get folks to clean up rogue server instances in inap? we had to stop using it nearly a month ago, and when we tried to reenable it a couple of weeks back we had to immediately turn it off again because it was still unstable | 17:47 |
clarkb | fungi: semi related were you still planning to add testing to the traceroute check chagne? | 17:49 |
clarkb | I don't want to approve it unless it has been tested somehow but not sure if that iwll happen in change or via base-test | 17:50 |
fungi | yeah, i'm not entirely clear on what testing we do or can do in zuul-jobs for that role. i'll take another look real quickly | 17:51 |
fungi | looks like the extent of pre-merge testing for it is that we execute the role supplying a minimum number of required parameters and make sure it doesn't fail | 18:00 |
clarkb | in that case the default is unlikely to have regressed :) | 18:01 |
fungi | actually testing the "fail if missing ipv4/fail if missing ipb6" scenarios would require fiddling with the node's networking or mocking significant parts of the network stack | 18:01 |
clarkb | can probably do a child change that sets those parameters then inspect how it reacts without actually merging it? | 18:01 |
clarkb | since the outcomes will depend on which cloud you land on | 18:01 |
fungi | yeah, i'll get that pushed up | 18:02 |
fungi | should i test requiring v6 routes initially? we don't currently have any providers where we expect v4 routes to be missing | 18:07 |
*** hamalq has joined #opendev | 18:07 | |
clarkb | ya | 18:07 |
clarkb | can you run it multiple times in the same play? | 18:07 |
clarkb | if so I would do v4 required then v6 required then both | 18:08 |
clarkb | and depending on where we land we should get decent coverage? | 18:08 |
fungi | sure, i think i can just include it multiple times | 18:08 |
*** eolivare has quit IRC | 18:10 | |
openstackgerrit | Jeremy Stanley proposed opendev/base-jobs master: base-test: exercise validate-host IP family assert https://review.opendev.org/763241 | 18:13 |
clarkb | hrm since that won't do premerge testing I'm not sure my previous that that we havne't regressed is correct | 18:14 |
*** dtantsur is now known as dtantsur|afk | 18:15 | |
openstackgerrit | Jeremy Stanley proposed opendev/bindep master: DNM: exercise base-test https://review.opendev.org/763243 | 18:16 |
fungi | clarkb: so we need to first merge https://review.opendev.org/763065 and then that and after that i can recheck the dnm job? | 18:17 |
clarkb | fungi: yes, except that 763065 will immediately apply to all jobs that start after that merge | 18:17 |
clarkb | if we could check that we don't regress first then I'd be ok with that, but I don't think we can? Let me look closer | 18:18 |
fungi | in theory we can rerun validate host in an untrusted job? | 18:19 |
clarkb | fungi: ok I think the file you want to modify is zuul/zuul-jobs/test-playbooks/base-roles/validate-host.yaml | 18:19 |
fungi | with a depends-on | 18:19 |
clarkb | if you modify ^ that is run pre merge | 18:19 |
fungi | ahh | 18:19 |
clarkb | rather than modifying base-test | 18:19 |
clarkb | and the presence of that playbook does imply that the non regression check case is covered, but doing that testing in zuul-jobs (even if we don't merge teh chagne due to provider differences) is still simpler | 18:20 |
fungi | though for the thorough test, i guess i could just rerun the role in the run phase of a job right? | 18:20 |
fungi | with a depends-on to the zuul-jobs change | 18:21 |
clarkb | zuul/zuul-jobs/test-playbooks/base-roles/validate-host.yaml is poart of the run playbook for the job it runs in | 18:21 |
clarkb | (I'm not sure I understand the question) | 18:21 |
clarkb | I think both approaches will give us approximately the same test coverage but doing it in zuul-jobs will be simpler as it is pre merge | 18:21 |
fungi | yeah, i guess i could make a dnm change to rerun validate-host with multiple options in zuul/zuul-jobs/test-playbooks/base-roles/validate-host.yaml | 18:21 |
fungi | is what i was suggesting | 18:21 |
clarkb | yup exactly that as a child of your change | 18:22 |
fungi | but i misread your initial suggestion as still involving base-test in some way | 18:22 |
fungi | okay, got it | 18:22 |
clarkb | sometimes base-test is required, I wasn't sure in this case | 18:22 |
clarkb | if it is required due to privileges or whatever then we have to do that dance but if it isnt' we should avoid it | 18:22 |
clarkb | ~3 minutes to release jobs being done I think | 18:23 |
openstackgerrit | Jeremy Stanley proposed zuul/zuul-jobs master: DNM: Exercise validate-host IP family assert https://review.opendev.org/763245 | 18:25 |
fungi | release and release-post are both empty now | 18:29 |
fungi | ianw: looks like you have a couple of root screen sessions on review.o.o created a week ago with borg backup work going on in them. are you still using those or can they be closed out? | 18:31 |
fungi | status log The Gerrit service at review.opendev.org is being restarted quickly as a pre-upgrade sanity check, estimated downtime is less than 5 minutes. | 18:33 |
clarkb | lgtm | 18:33 |
*** fressi has joined #opendev | 18:33 | |
fungi | er, make that status notice | 18:33 |
clarkb | oh ya we want people to see it | 18:33 |
clarkb | notice++ | 18:33 |
fungi | do we want to do that now or wait for anything else? | 18:34 |
clarkb | I can't think of anything else to wait on | 18:34 |
fungi | #status notice The Gerrit service at review.opendev.org is being restarted quickly as a pre-upgrade sanity check, estimated downtime is less than 5 minutes. | 18:34 |
openstackstatus | fungi: sending notice | 18:34 |
fungi | i'll down the container and then re-up it | 18:34 |
-openstackstatus- NOTICE: The Gerrit service at review.opendev.org is being restarted quickly as a pre-upgrade sanity check, estimated downtime is less than 5 minutes. | 18:34 | |
clarkb | k, I'm around if anything comes up | 18:34 |
fungi | it's starting now | 18:35 |
fungi | log says it's up | 18:36 |
clarkb | still waiting for apache to agree (thats normal to lag) | 18:37 |
fungi | i'm getting a webui now | 18:37 |
clarkb | me too | 18:37 |
openstackstatus | fungi: finished sending notice | 18:37 |
clarkb | and our config options appear to have remained in place \o/ | 18:38 |
fungi | excellent | 18:38 |
fungi | also i *think* tail -F is the solution to following the error_log before gerrit reopens it | 18:39 |
fungi | at least it seemed to work for me this time | 18:39 |
*** rpittau is now known as rpittau|afk | 18:39 | |
clarkb | oh neat | 18:40 |
clarkb | I'll have to use -F now and get rid of that -f memory | 18:40 |
fungi | i'm not entirely convinced, may have merely been coincident timing on my part | 18:41 |
fungi | looks like gerritbot needs a restart | 19:01 |
*** openstackgerrit has quit IRC | 19:02 | |
fungi | it's restarting now | 19:03 |
*** andrewbonney has quit IRC | 19:04 | |
*** roman_g has quit IRC | 19:15 | |
sean-k-mooney | clarkb: fungi did ye push an update to the zuul ui recently | 19:36 |
sean-k-mooney | it looks like its not rendering the logs properly anymore | 19:36 |
sean-k-mooney | https://zuul.opendev.org/t/openstack/build/00ff397ca5374a0b9441036d43ee3416/log/controller/logs/screen-n-cpu.txt#7371 | 19:36 |
sean-k-mooney | that should be the first line of a multiple output | 19:37 |
sean-k-mooney | basically the guest libvirt xml | 19:37 |
sean-k-mooney | its there in the raw view but truncated in the app view | 19:37 |
clarkb | I havent but there has been ui work | 19:37 |
sean-k-mooney | well when i ment pushed i ment deployed a new version to production | 19:38 |
fungi | looks like it's broken horizontal scrolling as well... i don't get a horizontal scroll bar unless i scroll to the end of the page first | 19:39 |
fungi | sean-k-mooney: it's continuously deployed | 19:39 |
sean-k-mooney | yes that too | 19:39 |
sean-k-mooney | you cant use keyborad arrows anymore | 19:39 |
clarkb | but not continuously restarted iirc | 19:39 |
sean-k-mooney | so if you search and it scrolls you have to go to the bottom and scoll back | 19:40 |
sean-k-mooney | its kind of annoying but i didnt know if i was the only one that did that | 19:40 |
clarkb | But I've been too busy too follow all the recent ui discussion | 19:40 |
clarkb | maybe ask in #zuul and we can traceback from there | 19:40 |
sean-k-mooney | i was just going to ask should i bring it up there instead | 19:41 |
fungi | i just mentioned it in there | 19:41 |
ianw | fungi: any backups sessions can be cleared, thanks | 19:42 |
fungi | no worries, appreciate it | 19:43 |
*** sboyron has quit IRC | 19:52 | |
ianw | fungi / clarkb : if you have a little time for the codesearch container i can work on deployment https://review.opendev.org/#/c/762960/; otherwise i might make the current job non-voting for something to avoid the puppet failures | 19:55 |
clarkb | Ya I can review it after lunch | 19:56 |
*** sboyron has joined #opendev | 20:04 | |
*** openstackgerrit has joined #opendev | 20:04 | |
openstackgerrit | Jeremy Stanley proposed openstack/project-config master: Require working IPv4 on all job nodes https://review.opendev.org/763253 | 20:04 |
openstackgerrit | Jeremy Stanley proposed openstack/project-config master: Revert "Disable limestone provider due to IPv4-less nodes" https://review.opendev.org/763254 | 20:04 |
fungi | infra-root: ^ that stack will get us back a bit of node capacity... our available quota is rather strained at this point | 20:46 |
*** mgoddard has joined #opendev | 20:50 | |
clarkb | were we able to confirm v8a testing the new stuff is happy? | 20:55 |
fungi | yeah, i left some comments | 20:56 |
fungi | seems to work as designed | 20:56 |
fungi | however it does highlight that we have almost no providers with ipv6-connected nodes at present | 20:56 |
*** slaweq has quit IRC | 20:58 | |
fungi | (10 out of 11 role tests in the dnm change failed due to lack of ipv6 connectivity) | 20:58 |
clarkb | great, I'm double checking test results then will probably go through and approve things. corvus you had previously reviewed https://review.opendev.org/#/c/763065/ as well, want to give that another look? | 20:58 |
fungi | they seemed to all succeed on the ipv4-required plays though | 20:59 |
fungi | which is what we expect | 20:59 |
fungi | amusingly, gentoo nodes seem to pick up working ipv6 addresses/routes in rackspace while other distros aren't | 21:01 |
fungi | that may point to a shortcoming in glean | 21:01 |
clarkb | I've +2'd the three non WIP/DNM changes. Maybe give corvus a few minutes to chime in on the zuul-jobs change again if interested then start landing them? | 21:02 |
fungi | i can't recall which network management method gentoo ends up using | 21:02 |
fungi | prometheanfire: ^ ? | 21:02 |
clarkb | yes it is glean issue since ipv6 is statically configured | 21:02 |
clarkb | gentoo uses the systemd setup | 21:02 |
clarkb | I think glean should statically configure ipv6 on debuntu and gentoo but none of the rpm distros | 21:03 |
fungi | ahh,systemd-networkd? | 21:03 |
clarkb | ya that thing | 21:03 |
fungi | it's possible all the debuntu nodes from that buildset landed in ovh or something | 21:03 |
fungi | yep, that was it exactly, all 4 debuntu builds landed in ovh | 21:05 |
fungi | so those might have also succeeded if they'd run in rax | 21:05 |
clarkb | ianw: left comments on the code search change. The approach seems fine and most of the issues are minor | 21:15 |
clarkb | ianw: I think its mostly about cleaning things up then we're good | 21:15 |
*** DSpider has quit IRC | 21:20 | |
ianw | clarkb: thanks, will llook in on | 21:29 |
corvus | clarkb, fungi: 763065 approved after having a fascinating conversation with myself. | 21:29 |
clarkb | corvus: do we need to do a meetpad call so that we can hear human voices again? | 21:30 |
corvus | clarkb: we're human? | 21:30 |
*** slaweq has joined #opendev | 21:30 | |
clarkb | good point. I think I'm turning into the human equivalent of a gollum | 21:31 |
clarkb | especially now with all the rain arriving the sun probably won't be seen again for months | 21:36 |
openstackgerrit | Slawek Kaplonski proposed zuul/zuul-jobs master: [multi-node-bridge] Add script to configure connectivity https://review.opendev.org/762650 | 21:37 |
*** mgoddard has quit IRC | 21:44 | |
openstackgerrit | Merged zuul/zuul-jobs master: validate-host: Options to require v4 and v6 routes https://review.opendev.org/763065 | 21:45 |
clarkb | fungi: ^ I think you can start to land the other changes now | 21:45 |
fungi | will changes to zuul/site-variables.yaml be picked up automatically once deployed? | 21:49 |
clarkb | yes they are included in new asnible runes iirc | 21:49 |
fungi | trying to gauge how long to wait before approving the limestone removal revert | 21:49 |
clarkb | wow typing is difficult | 21:49 |
clarkb | absically zuul does ansible -e that file -e other stuff iirc | 21:49 |
clarkb | so once that is in place on the executors we should see it for all the jobs | 21:50 |
* fungi consults the runes of asnible | 21:50 | |
fungi | ahh, right | 21:52 |
openstackgerrit | Merged openstack/project-config master: Require working IPv4 on all job nodes https://review.opendev.org/763253 | 21:54 |
fungi | so in theory i could approve both at the same time since one gets deployed to the executors and the other to a launcher and the problem is unlikely to recur in any window where nodes were booted by the launcher and then got handed off to a not-yet-updated executor to run a job | 21:54 |
fungi | but probably still better to wait and make sure the first change gets deployed to all the executors before i approve the one for the launcher | 21:55 |
clarkb | ya and you can manually update the launcher to speed things up once we're sure the executor is happy? | 21:55 |
fungi | i guess, it's not a huge rush though, just trying to get back a bit of capacity (50 nodes) | 21:56 |
clarkb | ++ | 21:56 |
fungi | corvus: you have an autohold with the comment "debug multi-stage multi-arch" which has three nodes held for 131 days... are you still using those? | 22:01 |
corvus | nope! sorry | 22:02 |
fungi | no worries, just in there cleaning up, i can remove | 22:02 |
fungi | mnaser: you have an autohold with the comment "debug multi-arch containers" which has a node held for 117 days... are you still using that? | 22:02 |
fungi | ianw: you have an autohold for "fedora containerfile" (not sure how old) which doesn't happen to have caught any nodes. still needed? | 22:05 |
*** slaweq has quit IRC | 22:16 | |
*** sboyron has quit IRC | 22:22 | |
*** johnsom has quit IRC | 22:27 | |
*** johnsom has joined #opendev | 22:29 | |
*** hashar has quit IRC | 22:39 | |
ianw | fungi: oh, that can go. i thought it was going to fail and it passed :) | 22:43 |
fungi | cool, i'll clear it out, thanks for confirming | 22:46 |
prometheanfire | fungi: gentoo image uses networkd | 22:57 |
prometheanfire | for debian hosts I manage I use networkd too, I mean, if you are gonna use systemd anywayway... | 22:58 |
mordred | if you're going to replace one thing with systemd, why not replace everything right? | 23:10 |
* fungi replaced systemd with systemd | 23:11 | |
fungi | prometheanfire: thanks, and yeah it turns out the buildset just split coincidentally between providers so all the debuntu builds ran somewhere without working ipv6 and so gentoo was the only one which passed | 23:12 |
ianw | clarkb: thanks for the comments, i can work on that | 23:54 |
ianw | for supervisord, i added that. the reason is so that we can restart houndd in the container when the config changes, without restarting the whole container | 23:54 |
ianw | the reason for that was to keep the update scripts inside the container | 23:54 |
clarkb | gotcha | 23:55 |
clarkb | hacky sighup I guess | 23:55 |
ianw | yeah; they've got a inotify watcher under review for the config file anyway | 23:56 |
ianw | so hopefully we can modify it to just update the config file and that's it | 23:56 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!