| *** diablo_rojo has quit IRC | 00:04 | |
| ianw | i have a feeling that the inbuilt openafs tests are failing during hte build of the openafs 1.8.5 packages :/ | 00:07 |
|---|---|---|
| *** tosky has quit IRC | 00:07 | |
| *** jamesmcarthur has joined #openstack-infra | 00:15 | |
| *** jamesmcarthur has quit IRC | 00:23 | |
| *** jamesmcarthur has joined #openstack-infra | 00:25 | |
| clarkb | ianw: tonyb https://github.com/go-gitea/gitea/blob/v1.9.6/CHANGELOG.md there is a 1.9.6 gitea we could potentially update to | 00:26 |
| clarkb | we are on 1.9.5. According to the changelog there is at least one gogit bugfix | 00:27 |
| clarkb | the original github issue that spawned that change had to do with chagnes of ownership of repos in gitea though | 00:28 |
| clarkb | I doubt that actually fixes the problem | 00:28 |
| clarkb | mordred: ^ fyi | 00:28 |
| *** rlandy|rover has quit IRC | 00:30 | |
| *** jamesmcarthur has quit IRC | 00:34 | |
| *** goldyfruit has joined #openstack-infra | 00:39 | |
| *** kjackal has quit IRC | 00:40 | |
| openstackgerrit | David Moreau Simard proposed zuul/zuul master: DNM: Test zuul-stream-functional with ara 1.2 https://review.opendev.org/694622 | 00:46 |
| *** weshay_ has joined #openstack-infra | 00:48 | |
| *** jamesmcarthur has joined #openstack-infra | 00:53 | |
| openstackgerrit | David Moreau Simard proposed zuul/zuul master: DNM: Test zuul-stream-functional with ara 1.2 https://review.opendev.org/694622 | 00:58 |
| ianw | clarkb: we should be able to test with https://launchpad.net/~openstack-ci-core/+archive/ubuntu/openafs-1.8.5-test/+packages ... if it builds | 01:00 |
| *** redrobot has quit IRC | 01:07 | |
| ianw | oh FFS, another thing wrong now | 01:08 |
| ianw | https://review.opendev.org/#/c/687954/ nova has dropped python2 so all the nodepool tests are borked | 01:09 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: WIP test if run-mirror job always times out https://review.opendev.org/694851 | 01:11 |
| *** slaweq has joined #openstack-infra | 01:11 | |
| ianw | tonyb: see comments in https://github.com/go-gitea/gitea/issues/9006 ... i guess your client actually hangs right? rather than closes the connection | 01:12 |
| tonyb | ianw: correct | 01:14 |
| tonyb | ianw: I left it for more than 24 hours ;P | 01:15 |
| *** slaweq has quit IRC | 01:15 | |
| tonyb | ianw: I guess in order to make debugging easier it might be worth pulling one server out of the rotation so that I'm the only one hitting it? | 01:16 |
| *** haleyb has joined #openstack-infra | 01:16 | |
| openstackgerrit | David Moreau Simard proposed zuul/zuul master: DNM: Test zuul-stream-functional with ara 1.2 https://review.opendev.org/694622 | 01:18 |
| *** jamesmcarthur has quit IRC | 01:19 | |
| *** gyee has quit IRC | 01:29 | |
| *** jamesmcarthur has joined #openstack-infra | 01:31 | |
| *** ociuhandu has joined #openstack-infra | 01:31 | |
| *** ociuhandu has quit IRC | 01:36 | |
| *** jamesmcarthur has quit IRC | 01:39 | |
| ianw | ianw@ubuntu-bionic-rax-iad-0012899150:~$ ls /afs/openstack.org/mirror | 01:40 |
| ianw | just hangs | 01:40 |
| *** jamesmcarthur has joined #openstack-infra | 01:42 | |
| ianw | how bizarre, it works for an existing client like mirror-update.opendev.org, but not for a new client | 01:42 |
| *** goldyfruit has quit IRC | 01:43 | |
| *** ociuhandu has joined #openstack-infra | 01:44 | |
| ianw | it's just mirror, not project/, docs/ etc | 01:46 |
| *** ociuhandu has quit IRC | 01:48 | |
| *** jamesmcarthur has quit IRC | 01:49 | |
| *** ociuhandu has joined #openstack-infra | 01:49 | |
| ianw | there's three stuck releases | 01:50 |
| ianw | does anyone else have an afs client they can point at /afs/openstack.org/mirror to see if it works? | 01:51 |
| *** jamesmcarthur has joined #openstack-infra | 01:52 | |
| *** ociuhandu has quit IRC | 01:57 | |
| *** rh-jelabarre has quit IRC | 02:02 | |
| *** factor has quit IRC | 02:03 | |
| *** ricolin has joined #openstack-infra | 02:04 | |
| tonyb | ianw: .... I'm happy to install something to test that but I don't know if that'd be helpful overall | 02:04 |
| tonyb | ianw: I don't know if I need creds etc | 02:04 |
| ianw | you don't but it can be a bit of a pain depending on distro | 02:05 |
| ianw | i'm rebuilding my kernel mods, hasn't kept up to date with upgrades locally | 02:06 |
| ianw | i'm thinking of restarting the servers. not exactly ideal, but i don't have any better ideas | 02:06 |
| *** jtomasek has quit IRC | 02:06 | |
| *** onovy has quit IRC | 02:09 | |
| *** slaweq has joined #openstack-infra | 02:11 | |
| *** onovy has joined #openstack-infra | 02:11 | |
| ianw | ok, locally "ls /afs/openstack.org/mirror" just goes off into la-la land for me too | 02:12 |
| ianw | i think that if you don't already have it cached (e.g. current mirrors, mirror-update) -- for example in the system-config test where we start a new node -- it's borked | 02:12 |
| *** jtomasek has joined #openstack-infra | 02:13 | |
| ianw | which probably means this is on borrowed time until it becomes gate breaking | 02:13 |
| *** slaweq has quit IRC | 02:16 | |
| *** jamesmcarthur has quit IRC | 02:21 | |
| *** igordc has quit IRC | 02:21 | |
| clarkb | we think it is a serverissue then? | 02:23 |
| *** jamesmcarthur has joined #openstack-infra | 02:30 | |
| *** roman_g has quit IRC | 02:33 | |
| tonyb | ianw: Ahh okay | 02:35 |
| ianw | clarkb: i can't see anything else it could be ... | 02:35 |
| tonyb | I can do the kmod thing but I don't know that that'll be any better than what you're doing | 02:35 |
| ianw | tonyb: fedora right? | 02:41 |
| *** goldyfruit has joined #openstack-infra | 02:41 | |
| ianw | if so, maybe you could install openafs-client from -> https://copr.fedorainfracloud.org/coprs/jsbillings/openafs/ | 02:42 |
| tonyb | ianw: on it | 02:42 |
| ianw | and then /usr/bin/time ls /afs/openstack.org/mirror/ | 02:42 |
| tonyb | ianw, clarkb: so how do we update a server to gitea 1.9.6? | 02:46 |
| tonyb | ianw: running the dkms-openafs scriptlet | 02:46 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: gitea: Use 1.9.6 https://review.opendev.org/694894 | 02:49 |
| ianw | tonyb: ^ something like that | 02:50 |
| tonyb | ianw: Ahh okay ;P | 02:52 |
| *** ociuhandu has joined #openstack-infra | 02:56 | |
| tonyb | ianw: I suspect thete is some AFS setup I'm missing | 02:58 |
| tonyb | ls: cannot access '/afs/openstack.org/mirror': No such file or directory | 02:59 |
| ianw | umm, /etc/openafs/ThisCell has "openstack.org" in it? | 03:01 |
| *** ociuhandu has quit IRC | 03:01 | |
| ianw | and then you might need to "service openafs-client start" | 03:01 |
| tonyb | ianw: approx 2seconds | 03:02 |
| ianw | well, ok ... | 03:03 |
| *** ociuhandu has joined #openstack-infra | 03:03 | |
| tonyb | sorry 5seconds | 03:03 |
| ianw | so i guess that nix's that idea | 03:03 |
| tonyb | is that good? | 03:03 |
| ianw | i guess it means the reasons why the mirror jobs are failing are still unresolved :/ | 03:04 |
| tonyb | Oh :( | 03:05 |
| *** ociuhandu has quit IRC | 03:07 | |
| *** apetrich has quit IRC | 03:08 | |
| *** ociuhandu has joined #openstack-infra | 03:10 | |
| ianw | i dunno. at this point, gitea is broken-ish, openafs is broken-ish, and all nodepool testing is broken thanks to nova dropping py2 support | 03:10 |
| ianw | too much broken-ness for me in one day :) | 03:10 |
| *** slaweq has joined #openstack-infra | 03:11 | |
| *** slaweq has quit IRC | 03:16 | |
| *** ociuhandu has quit IRC | 03:18 | |
| *** ociuhandu has joined #openstack-infra | 03:25 | |
| *** ociuhandu has quit IRC | 03:30 | |
| *** ociuhandu has joined #openstack-infra | 03:32 | |
| *** ociuhandu has quit IRC | 03:36 | |
| *** carl_cai has joined #openstack-infra | 03:46 | |
| clarkb | sorry I popped out, dinner happened. nodepool is probably the easiset to fix so mayve start tgere? | 03:51 |
| *** ykarel|away has joined #openstack-infra | 03:53 | |
| tonyb | ianw: That is indeed a sad summary | 03:58 |
| *** ykarel|away is now known as ykarel | 03:58 | |
| *** goldyfruit has quit IRC | 04:00 | |
| *** jamesmcarthur has quit IRC | 04:00 | |
| *** jamesmcarthur has joined #openstack-infra | 04:00 | |
| *** ykarel has joined #openstack-infra | 04:02 | |
| *** jamesmcarthur has quit IRC | 04:06 | |
| *** ociuhandu has joined #openstack-infra | 04:06 | |
| *** slaweq has joined #openstack-infra | 04:11 | |
| *** ociuhandu has quit IRC | 04:14 | |
| *** slaweq has quit IRC | 04:15 | |
| *** weshay has quit IRC | 04:18 | |
| *** weshay has joined #openstack-infra | 04:19 | |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: install-devstack: switch to Python 3 https://review.opendev.org/694898 | 04:22 |
| ianw | clarkb: ^ that's one way to fix nodepool testing, maybe | 04:22 |
| openstackgerrit | Ian Wienand proposed zuul/nodepool master: [dnm] test devstack python3 https://review.opendev.org/694899 | 04:26 |
| ianw | we shall see | 04:26 |
| *** jamesmcarthur has joined #openstack-infra | 04:31 | |
| ianw | we must have got past the AFS sanity check @ https://opendev.org/opendev/system-config/src/branch/master/playbooks/roles/mirror/tasks/main.yaml#L1 | 04:38 |
| *** jamesmcarthur has quit IRC | 04:44 | |
| *** jamesmcarthur has joined #openstack-infra | 04:47 | |
| *** kjackal has joined #openstack-infra | 04:47 | |
| *** ddurst has quit IRC | 05:01 | |
| *** ociuhandu has joined #openstack-infra | 05:18 | |
| *** jamesmcarthur has quit IRC | 05:20 | |
| ianw | ok, afs02.dfw.openstack.org is dead | 05:21 |
| *** ykarel_ has joined #openstack-infra | 05:21 | |
| *** ociuhandu has quit IRC | 05:22 | |
| *** ykarel has quit IRC | 05:24 | |
| ianw | https://imgur.com/a/LwgTftV | 05:24 |
| ianw | this must be the root cause | 05:24 |
| ianw | i'm rebooting it ... let's see if it comes up | 05:24 |
| *** elod is now known as elod_off | 05:28 | |
| ianw | well wouldn't you know it, the host that was stuck on ls /afs/openstack.org/mirror is now unstack | 05:28 |
| ianw | i guess this was is a half-dead state, enough to respond to connections but not enough to give back data | 05:29 |
| *** ociuhandu has joined #openstack-infra | 05:30 | |
| *** kjackal has quit IRC | 05:33 | |
| *** ociuhandu has quit IRC | 05:35 | |
| *** pcaruana has joined #openstack-infra | 05:43 | |
| *** jamesmcarthur has joined #openstack-infra | 05:51 | |
| *** lennyb has joined #openstack-infra | 05:53 | |
| *** jamesmcarthur has quit IRC | 05:55 | |
| *** carl_cai has quit IRC | 05:56 | |
| *** surpatil has joined #openstack-infra | 06:00 | |
| ianw | ok, that fixed the test job, calling it fixed | 06:06 |
| ianw | #status log rebooted afs02.dfw.openstack.org after it's console was full of I/O errors. very much like what we've seen before during host migrations that didn't go so well | 06:07 |
| ianw | statusbot is gone again!? | 06:07 |
| *** openstackstatus has joined #openstack-infra | 06:08 | |
| *** ChanServ sets mode: +v openstackstatus | 06:08 | |
| ianw | #status log rebooted afs02.dfw.openstack.org after it's console was full of I/O errors. very much like what we've seen before during host migrations that didn't go so well | 06:09 |
| openstackstatus | ianw: finished logging | 06:09 |
| *** slaweq has joined #openstack-infra | 06:11 | |
| *** soniya29 has joined #openstack-infra | 06:14 | |
| *** slaweq has quit IRC | 06:15 | |
| *** lpetrut has quit IRC | 06:25 | |
| *** Lucas_Gray has joined #openstack-infra | 06:34 | |
| *** pcaruana has quit IRC | 06:56 | |
| openstackgerrit | Merged zuul/zuul master: Change colors of various "negative" results in UI https://review.opendev.org/691828 | 06:56 |
| *** rcernin has quit IRC | 06:58 | |
| *** dpawlik has joined #openstack-infra | 07:03 | |
| *** udesale has joined #openstack-infra | 07:06 | |
| openstackgerrit | Merged zuul/zuul master: Refresh public OpenPGP key for Jeremy Stanley https://review.opendev.org/693441 | 07:09 |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Remove unused path variable from promote secrets https://review.opendev.org/694912 | 07:20 |
| AJaeger | clarkb: I addressed your comment in https://review.opendev.org/#/c/681582 with https://review.opendev.org/694912 | 07:21 |
| AJaeger | thanks for catching that obsolete variable | 07:21 |
| AJaeger | ianw: can you test https://review.opendev.org/#/c/694898 with a stable change that is using python2, please? | 07:25 |
| mnasiadka | morning | 07:26 |
| openstackgerrit | Merged zuul/zuul master: Enable starting executors in paused mode https://review.opendev.org/692812 | 07:27 |
| mnasiadka | we (kolla/kolla-ansible) would like to use elastic-recheck, but our ansible logs are being pushed to primary/logs/ansible/deploy.txt for example - what is the correct way of adding those files for submit-logstash-jobs so those land on logstash? override logstash_processor_config? | 07:27 |
| *** xek has quit IRC | 07:32 | |
| *** apetrich has joined #openstack-infra | 07:34 | |
| mnasiadka | actually it's not that simple to override logstash_processor_config I think | 07:34 |
| ianw | AJaeger: hrm, that role is only used in the zuul/nodepool jobs, where it is unbranched like that. i'd have to think about if it supports other branches | 07:40 |
| ianw | corvus may have an opinion ^ (https://review.opendev.org/#/c/694898) | 07:41 |
| ianw | tbh the best thing would be for nova to not break default devstack ... but anyway | 07:41 |
| *** dpawlik has quit IRC | 07:43 | |
| *** pgaxatte has joined #openstack-infra | 07:44 | |
| *** ykarel_ has quit IRC | 07:44 | |
| *** ykarel_ has joined #openstack-infra | 07:44 | |
| *** dpawlik has joined #openstack-infra | 07:44 | |
| *** pkopec has joined #openstack-infra | 07:45 | |
| *** ykarel_ is now known as ykarel | 07:48 | |
| *** ccamacho has joined #openstack-infra | 07:48 | |
| *** surpatil has quit IRC | 07:49 | |
| *** soniya29 has quit IRC | 07:50 | |
| *** surpatil has joined #openstack-infra | 07:57 | |
| *** slaweq has joined #openstack-infra | 08:00 | |
| *** tesseract has joined #openstack-infra | 08:18 | |
| *** dchen has quit IRC | 08:25 | |
| *** tosky has joined #openstack-infra | 08:26 | |
| *** pcaruana has joined #openstack-infra | 08:36 | |
| *** iurygregory has joined #openstack-infra | 08:39 | |
| *** jpena|off is now known as jpena | 08:42 | |
| *** ykarel is now known as ykarel|lunch | 08:48 | |
| *** ralonsoh has joined #openstack-infra | 08:50 | |
| *** priteau has joined #openstack-infra | 08:55 | |
| *** tkajinam has quit IRC | 08:56 | |
| openstackgerrit | Simon Westphahl proposed zuul/zuul master: Add optional support for circular dependencies https://review.opendev.org/685354 | 08:56 |
| AJaeger | ianw: oh fun... I had assumed the role would be used broader | 08:57 |
| *** FlorianFa has joined #openstack-infra | 08:58 | |
| *** ociuhandu has joined #openstack-infra | 09:00 | |
| *** lucasagomes has joined #openstack-infra | 09:02 | |
| *** rpittau|afk is now known as rpittau | 09:08 | |
| *** ykarel|lunch is now known as ykarel|pto | 09:09 | |
| *** xek has joined #openstack-infra | 09:26 | |
| kashyap | AJaeger: Morning, when you get a min: http://lists.openstack.org/pipermail/openstack-discuss/2019-November/010907.html | 09:28 |
| *** derekh has joined #openstack-infra | 09:38 | |
| *** ykarel|pto has quit IRC | 09:43 | |
| *** ociuhandu has quit IRC | 09:43 | |
| *** jamesmcarthur has joined #openstack-infra | 09:48 | |
| *** jamesmcarthur has quit IRC | 09:53 | |
| *** gfidente has joined #openstack-infra | 09:59 | |
| *** sshnaidm|afk is now known as sshnaidm|ruck | 10:10 | |
| AJaeger | kashyap: replied ;) | 10:14 |
| kashyap | AJaeger: Thank you | 10:14 |
| kashyap | Excellent, just read it. | 10:15 |
| *** Lucas_Gray has quit IRC | 10:19 | |
| *** Lucas_Gray has joined #openstack-infra | 10:21 | |
| *** ociuhandu has joined #openstack-infra | 10:31 | |
| *** lpetrut has joined #openstack-infra | 10:40 | |
| *** rfolco has joined #openstack-infra | 10:43 | |
| *** dpawlik has quit IRC | 10:45 | |
| *** jamesmcarthur has joined #openstack-infra | 10:50 | |
| *** jamesmcarthur has quit IRC | 10:54 | |
| *** florianf has joined #openstack-infra | 11:01 | |
| *** pgaxatte has quit IRC | 11:03 | |
| *** udesale has quit IRC | 11:17 | |
| *** dpawlik has joined #openstack-infra | 11:21 | |
| *** dpawlik has quit IRC | 11:26 | |
| *** dpawlik has joined #openstack-infra | 11:30 | |
| *** roman_g has joined #openstack-infra | 11:30 | |
| openstackgerrit | Jens Harbott (frickler) proposed openstack/project-config master: Drop broken legacy job from devstack https://review.opendev.org/694989 | 11:39 |
| *** gibi has joined #openstack-infra | 11:40 | |
| frickler | config-core: ^^ that one is needed to unblock devstack thanks to nova | 11:41 |
| AJaeger | frickler: did you see https://review.opendev.org/#/c/694898 ? | 11:43 |
| AJaeger | frickler: oh, a different change. | 11:44 |
| AJaeger | gmann: could you look at 694989, please? ^ | 11:44 |
| openstackgerrit | Slawek Kaplonski proposed opendev/irc-meetings master: Remove Networking OVN and ML2+OVS+DVR Convergence Team Meeting https://review.opendev.org/694991 | 11:46 |
| *** dtantsur|afk is now known as dtantsur | 11:49 | |
| *** ociuhandu has quit IRC | 11:52 | |
| *** ociuhandu has joined #openstack-infra | 11:53 | |
| *** roman_g has quit IRC | 11:53 | |
| *** ociuhandu has quit IRC | 11:53 | |
| *** roman_g has joined #openstack-infra | 11:53 | |
| *** ociuhandu has joined #openstack-infra | 11:54 | |
| *** ociuhandu has quit IRC | 11:59 | |
| *** goldyfruit has joined #openstack-infra | 12:18 | |
| *** jamesmcarthur has joined #openstack-infra | 12:19 | |
| *** dpawlik has quit IRC | 12:28 | |
| *** jpena is now known as jpena|lunch | 12:31 | |
| *** dpawlik has joined #openstack-infra | 12:31 | |
| *** soniya29 has joined #openstack-infra | 12:32 | |
| *** dave-mccowan has joined #openstack-infra | 12:34 | |
| *** icey has quit IRC | 12:34 | |
| *** icey has joined #openstack-infra | 12:35 | |
| *** ociuhandu has joined #openstack-infra | 12:36 | |
| *** roman_g has quit IRC | 12:37 | |
| *** roman_g has joined #openstack-infra | 12:39 | |
| *** Lucas_Gray has quit IRC | 12:43 | |
| *** icey has quit IRC | 12:46 | |
| *** icey has joined #openstack-infra | 12:46 | |
| *** florianf has left #openstack-infra | 12:51 | |
| *** rlandy has joined #openstack-infra | 12:54 | |
| *** rlandy is now known as rlandy|rover | 12:54 | |
| *** pgaxatte has joined #openstack-infra | 12:56 | |
| *** udesale has joined #openstack-infra | 12:58 | |
| *** kjackal has joined #openstack-infra | 12:58 | |
| *** gfidente has quit IRC | 12:59 | |
| *** surpatil has quit IRC | 12:59 | |
| *** ociuhandu has quit IRC | 13:00 | |
| *** ociuhandu has joined #openstack-infra | 13:01 | |
| *** iurygregory has quit IRC | 13:02 | |
| *** rh-jelabarre has joined #openstack-infra | 13:02 | |
| *** chandankumar is now known as raukadah | 13:05 | |
| *** goldyfruit has quit IRC | 13:08 | |
| *** jamesmcarthur has quit IRC | 13:12 | |
| *** jamesmcarthur has joined #openstack-infra | 13:12 | |
| *** ociuhandu has quit IRC | 13:17 | |
| *** ociuhandu has joined #openstack-infra | 13:18 | |
| *** hamzy has quit IRC | 13:19 | |
| corvus | ianw, AJaeger: why isn't devstack just changing the python3 setting? | 13:21 |
| corvus | frickler: ^ | 13:21 |
| *** ociuhandu_ has joined #openstack-infra | 13:22 | |
| corvus | and how did nova even land a change that broke devstack? | 13:23 |
| *** ociuhandu has quit IRC | 13:26 | |
| corvus | it seems like the nova jobs should not be adding options to devstack which are required for it to work. those should just be in devstack. | 13:29 |
| *** jpena|lunch is now known as jpena | 13:31 | |
| openstackgerrit | Tobias Henkel proposed zuul/nodepool master: Add ready endpoint to webapp https://review.opendev.org/695001 | 13:34 |
| frickler | corvus: nova seems to run only py3 jobs. devstack still has lots of soon-to-be-legacy py2 jobs, but nobody cleaned them up yet | 13:35 |
| frickler | eight of them are still voting on devstack https://review.opendev.org/#/c/694967/ | 13:35 |
| corvus | frickler: right, i think the issue is that nova is adding configuration options to its own jobs which are actually required by devstack to run | 13:35 |
| *** liuyulong has joined #openstack-infra | 13:36 | |
| frickler | corvus: well, it's not their own jobs, they simply base everything on tempest-py3 and grenade-py3 | 13:36 |
| corvus | frickler: no, take a look at what gets reverted in https://review.opendev.org/694891 | 13:36 |
| *** soniya29 has quit IRC | 13:36 | |
| corvus | the job "nova-tempest-v2-api" has "USE_PYTHON3: True" set locally | 13:37 |
| corvus | there's a bunch of other settings there too, one wonders if some of those should also be in devstack | 13:38 |
| *** eharney has quit IRC | 13:41 | |
| *** mriedem has joined #openstack-infra | 13:42 | |
| *** kashyap has left #openstack-infra | 13:47 | |
| *** ddurst has joined #openstack-infra | 13:50 | |
| *** ociuhandu_ has quit IRC | 13:52 | |
| *** jamesmcarthur has quit IRC | 13:52 | |
| *** ociuhandu has joined #openstack-infra | 13:53 | |
| *** tkajinam has joined #openstack-infra | 13:56 | |
| *** tkajinam has quit IRC | 13:56 | |
| *** tkajinam has joined #openstack-infra | 13:57 | |
| *** ociuhandu has quit IRC | 13:58 | |
| *** ociuhandu has joined #openstack-infra | 14:02 | |
| *** kjackal has quit IRC | 14:07 | |
| *** ociuhandu has quit IRC | 14:08 | |
| corvus | AJaeger, frickler: based on conversation in #openstack-nova, it does not appear that nova on devstack is going to be fixed immediately, therefore i think we should merge https://review.opendev.org/694898 (cc ianw) | 14:12 |
| *** gfidente has joined #openstack-infra | 14:14 | |
| *** pgaxatte has quit IRC | 14:15 | |
| fungi | makes sense, i've approved it now | 14:15 |
| tosky | can anyone please send an email to the list explaining what's going on? | 14:16 |
| tosky | oh, do I understand it correctly that https://review.opendev.org/#/c/694898/ only fix the zuul job? | 14:17 |
| corvus | tosky: yes, it is not even remotely a solution to the problem | 14:18 |
| tosky | oki | 14:18 |
| *** pgaxatte has joined #openstack-infra | 14:19 | |
| *** goldyfruit has joined #openstack-infra | 14:19 | |
| corvus | tosky: as long as the nova/devstack state remains as-is, then effectively devstack is broken without setting that. that role is therefore simply adopting to the new normal and setting that. it is (to my knowledge) only used by a few jobs which simply want to test against an openstack cloud as users. | 14:19 |
| *** aaronsheffield has joined #openstack-infra | 14:21 | |
| *** goldyfruit_ has joined #openstack-infra | 14:28 | |
| *** iurygregory has joined #openstack-infra | 14:29 | |
| *** goldyfruit has quit IRC | 14:30 | |
| fungi | right, it's a stop-gap while folks work out what openstack/devstack's behavior should be in the face of the python3 move | 14:30 |
| *** pcrews has joined #openstack-infra | 14:30 | |
| fungi | i expect devstack jobs running for most openstack projects to remain broken until a solution is chosen | 14:31 |
| *** kjackal has joined #openstack-infra | 14:32 | |
| fungi | i read the goal (still not sure why it was structured as a cycle goal) as leaf projects who intend to drop python2-specific jobs should do so by milestone 1, and then libraries which intend to drop python2-specific jobs should do so by milestone 2, and then everyone should stop ripping out jobs so the software can be polished with reasonably stable testing for the remainder of the cycle | 14:34 |
| frickler | fungi: ripping out jobs is one thing, changing a core project to hard fail with py2 immediately is something different | 14:35 |
| *** pgaxatte has quit IRC | 14:35 | |
| fungi | well, i don't think they intended to make it do that | 14:35 |
| frickler | intended or not, they did | 14:36 |
| fungi | but they didn't have any testing against default devstack deployments | 14:36 |
| fungi | so it was able to be merged and deadlock everyone who was running a default devstack deployment | 14:36 |
| fungi | it used to be that we had at least one generic devstack+tempest job that all projects were required to run | 14:37 |
| fungi | frankly though, i really appreciate the validation of our testing principles there. the moment nova stopped testing with python 2 they also managed to immediately break their ability to even be installed under python 2. if it's not tested it's (immediately, apparently!) broken ;) | 14:39 |
| frickler | one could argue that that would be tempest-full-py3 now. and that devstack just wasn't fast enough to move along. it's just that except ianw and some fraction of myself, nobody really cares about devstack anymore | 14:39 |
| frickler | fungi: yeah, I agree with that, it's just bad timing/priorization IMHO | 14:40 |
| openstackgerrit | Merged zuul/zuul-jobs master: install-devstack: switch to Python 3 https://review.opendev.org/694898 | 14:41 |
| frickler | fungi: corvus: it would still be good if you could look at https://review.opendev.org/694989 so that devstack can recover on its own, likely by temporarily making all py2 jobs nv | 14:42 |
| * frickler needs to go do some paid work now | 14:42 | |
| tosky | fungi, corvus: I understand your concerns and I've preferred a full revert, but in this case I prefer a good workaround which has an expiration date attached anyway (a few weeks) than spending times leaving the gates broken | 14:48 |
| AJaeger | frickler: happy to merge 694989 in general - I just wonder whether we loose coverage. If we merge it, could you do a followup to remove the job defintion from openstack-zuul-jobs, please? | 14:48 |
| *** jaosorior has joined #openstack-infra | 14:49 | |
| *** ociuhandu has joined #openstack-infra | 14:49 | |
| *** roman_g has quit IRC | 14:51 | |
| frickler | AJaeger: we (devstack) might want to create a devstack-updown job testing the unstack part, but I'm not sure if anyone cares enough. I'd be fine with marking unstack.sh as unsupported, too | 14:53 |
| frickler | AJaeger: and yeah, I'll do the followup tomorrow | 14:53 |
| AJaeger | frickler: thanks. Since up-down is only on devstack, let me +2... | 14:54 |
| *** eharney has joined #openstack-infra | 14:58 | |
| *** tkajinam has quit IRC | 15:00 | |
| *** pgaxatte has joined #openstack-infra | 15:03 | |
| jpena | hi! Looking at http://grafana.openstack.org/d/ACtl1JSmz/afs?orgId=1, it looks like the AFS mirrors have not been synced for a few days. Is this a known issue? | 15:04 |
| fungi | jpena: there was a problem with an afs server which wound up hanging and needing a reboot | 15:04 |
| fungi | we need to keep an eye on things though and make sure updates resume, the reboot happened at ~0600z | 15:05 |
| fungi | it's possible there's still a bit of cleanup which needs to be performed | 15:06 |
| fungi | i'll take a look in a moment | 15:06 |
| jpena | thanks fungi | 15:07 |
| *** michael-beaver has joined #openstack-infra | 15:08 | |
| *** dpawlik has quit IRC | 15:11 | |
| *** markmcclain has quit IRC | 15:21 | |
| *** markmcclain has joined #openstack-infra | 15:23 | |
| *** jamesmcarthur has joined #openstack-infra | 15:31 | |
| *** rlandy|rover is now known as rlandy|rover|mtg | 15:32 | |
| *** sshnaidm|ruck has quit IRC | 15:49 | |
| *** sshnaidm has joined #openstack-infra | 15:50 | |
| *** sshnaidm is now known as sshnaidm|ruck | 15:50 | |
| *** pcaruana has quit IRC | 15:50 | |
| *** pcaruana has joined #openstack-infra | 15:51 | |
| *** armax has quit IRC | 15:54 | |
| *** armax has joined #openstack-infra | 15:56 | |
| *** sreejithp has joined #openstack-infra | 15:59 | |
| openstackgerrit | Matt Riedemann proposed opendev/elastic-recheck master: Add query for nova install on py27 bug 1853166 https://review.opendev.org/695021 | 16:00 |
| openstack | bug 1853166 in OpenStack Compute (nova) "nova not installable on py2 environments which breaks the gate" [Critical,In progress] https://launchpad.net/bugs/1853166 - Assigned to Luigi Toscano (ltoscano) | 16:00 |
| *** udesale has quit IRC | 16:00 | |
| *** roman_g has joined #openstack-infra | 16:03 | |
| *** gyee has joined #openstack-infra | 16:05 | |
| *** lucasagomes has quit IRC | 16:11 | |
| *** lucasagomes has joined #openstack-infra | 16:13 | |
| openstackgerrit | Javier Peña proposed zuul/zuul-registry master: Do not overwrite the kept_manifests variable when pruning https://review.opendev.org/693410 | 16:14 |
| *** kjackal has quit IRC | 16:16 | |
| *** ociuhandu has quit IRC | 16:17 | |
| fungi | infra-root: i've confirmed the mirrors are all stuck calling vos release since around the utc start of 2019-11-15, so will likely need some care in cleanly terminating those. i'll dig deeper on how to do that in an hour or so, but need to go run some errands first | 16:21 |
| *** rlandy|rover|mtg is now known as rlandy|rover | 16:22 | |
| openstackgerrit | Merged opendev/elastic-recheck master: Add query for nova install on py27 bug 1853166 https://review.opendev.org/695021 | 16:22 |
| openstack | bug 1853166 in OpenStack Compute (nova) "nova not installable on py2 environments which breaks the gate" [Critical,In progress] https://launchpad.net/bugs/1853166 - Assigned to Luigi Toscano (ltoscano) | 16:22 |
| fungi | #status log all afs mirrors are stuck in the middle of vos release commands since early utc friday | 16:22 |
| openstackstatus | fungi: finished logging | 16:22 |
| fungi | okay, disappearing, should be back soon | 16:24 |
| *** iurygregory has quit IRC | 16:36 | |
| *** jpena is now known as jpena|brb | 16:46 | |
| *** lucasagomes has quit IRC | 16:47 | |
| *** jaosorior has quit IRC | 16:48 | |
| *** tesseract has quit IRC | 16:51 | |
| *** pgaxatte has quit IRC | 16:54 | |
| openstackgerrit | David Shrewsbury proposed zuul/zuul master: Fix zuul-stream-functional tests https://review.opendev.org/694619 | 16:56 |
| openstackgerrit | David Shrewsbury proposed zuul/zuul master: Fix zuul-stream-functional tests https://review.opendev.org/694619 | 16:57 |
| *** jaosorior has joined #openstack-infra | 17:02 | |
| *** jamesmcarthur has quit IRC | 17:05 | |
| *** rpittau is now known as rpittau|afk | 17:06 | |
| openstackgerrit | Merged opendev/irc-meetings master: Remove Networking OVN and ML2+OVS+DVR Convergence Team Meeting https://review.opendev.org/694991 | 17:14 |
| *** jamesmcarthur has joined #openstack-infra | 17:17 | |
| *** dtantsur is now known as dtantsur|afk | 17:20 | |
| *** ociuhandu has joined #openstack-infra | 17:22 | |
| *** ccamacho has quit IRC | 17:23 | |
| *** jpena|brb is now known as jpena | 17:26 | |
| *** ociuhandu has quit IRC | 17:27 | |
| *** michael-beaver has quit IRC | 17:27 | |
| *** roman_g has quit IRC | 17:36 | |
| *** roman_g has joined #openstack-infra | 17:36 | |
| *** gfidente has quit IRC | 17:36 | |
| openstackgerrit | Tristan Cacqueray proposed opendev/gerritbot master: Add change-created event type https://review.opendev.org/286366 | 17:42 |
| openstackgerrit | Tristan Cacqueray proposed opendev/gerritlib master: Read all Gerrit events from poll interruption https://review.opendev.org/412757 | 17:43 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-jobs master: DNM: negative test https://review.opendev.org/522438 | 17:43 |
| *** jamesmcarthur has quit IRC | 17:43 | |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: trigger: add job filter event https://review.opendev.org/639905 | 17:48 |
| *** munimeha1 has joined #openstack-infra | 17:50 | |
| *** igordc has joined #openstack-infra | 17:52 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: use-buildset-registry: Vendor pytoml and remarshal https://review.opendev.org/695050 | 17:56 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: WIP: use-buildset-registry: add podman support https://review.opendev.org/695051 | 17:56 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: docs: add default project configuration guide https://review.opendev.org/571994 | 17:57 |
| *** derekh has quit IRC | 18:01 | |
| *** ociuhandu has joined #openstack-infra | 18:04 | |
| *** ricolin has quit IRC | 18:04 | |
| *** ociuhandu has quit IRC | 18:08 | |
| *** armax has quit IRC | 18:11 | |
| fungi | yeesh, i keep forgetting quite how sisyphean it is trying to find a restaurant that's open for lunch on an off-season tuesday on a barrier island | 18:11 |
| fungi | time to start a website keeping track of which of our local establishments is open on what days | 18:12 |
| fungi | anyway, i'm back, and fed | 18:12 |
| fungi | if clarkb is still away come 19:00 i'll run the meeting | 18:12 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: config: blacklist pipeline names that can not be used in template https://review.opendev.org/693961 | 18:13 |
| tosky | fungi: you can add the opening times to openstreetmap | 18:14 |
| fungi | yep | 18:14 |
| fungi | also the roads | 18:15 |
| tosky | ... that would help too | 18:15 |
| * fungi should get into the osm cartography community | 18:15 | |
| tosky | it's fun! | 18:15 |
| fungi | the up-side is that we don't have that many roads and, being a linear island they mostly only run two directions | 18:16 |
| tosky | then you can start mapping the trees, the type of lands, every other object around... | 18:16 |
| fungi | so if something's not where you went, it's back the way you came | 18:16 |
| *** ociuhandu has joined #openstack-infra | 18:17 | |
| mordred | fungi: many years ago I worked on a restaurant menu/ordering system for a company in Bar Harbor, ME that was trying to help off-season restaurants be able to provide food to people in their offices (kind of like a local seamless/grubhub several years before those were a thing) ... largely because of the "fun" of running a restaurant in a place with an off season | 18:19 |
| fungi | yep, this area is an awful lot like bar harbor. if chris weren't so opposed to cold weather i'd probably live there instead | 18:21 |
| *** ociuhandu has quit IRC | 18:21 | |
| mordred | fungi: bar harbor is not a place one should live if one does not like cold weather | 18:21 |
| fungi | i grew up getting snowed in for weeks at a stretch, so it doesn't bother me | 18:22 |
| fungi | i just love that the island has a mount desert which is really neither a mountain nor a desert | 18:23 |
| fungi | (also a frenchman bay which has remarkably few frenchmen in it) | 18:25 |
| mordred | frenchmen street in new orleans also has remarkably few frenchmen in it | 18:26 |
| *** dpawlik has joined #openstack-infra | 18:30 | |
| *** jpena is now known as jpena|off | 18:32 | |
| *** pkopec has quit IRC | 18:33 | |
| openstackgerrit | Merged zuul/zuul master: Fix zuul-stream-functional tests https://review.opendev.org/694619 | 18:38 |
| *** armax has joined #openstack-infra | 18:51 | |
| auristor | Here is a list of the RO volume sites that are currently flagged as "do not use" which require a "vos release" https://paste.fedoraproject.org/paste/4dzQoqB5jUtU6jUSJpdKnQ | 18:59 |
| clarkb | fungi: ya Im about 20 minutes away right niw | 18:59 |
| fungi | thanks auristor! about to run a meeting but will be working out how to safely terminate the hung vos releases once that's done | 19:00 |
| auristor | There are no volume transactions in flight on any volserver at present | 19:01 |
| auristor | each of the volume entries is currently locked. "vos unlock <vol-name-or-id>" prior to the "vos release" | 19:03 |
| auristor | each "vos release" will be a full volume transfer | 19:03 |
| *** ralonsoh has quit IRC | 19:03 | |
| auristor | if there are "vos release" processes that are stuck they can be killed | 19:05 |
| clarkb | ok at computer now | 19:19 |
| ianw | fungi / auristor : I haven't been back through scroll back ... but yesterday afternoon (my time) i had to reboot afs02.dfw.openstack.org because it was completely dead (see status updates) | 19:20 |
| clarkb | ianw: ya I think that either the server going down or the reboot caused all the things to end up getting locked because releases weren't working | 19:22 |
| clarkb | now we need to unlock and release them (this will be full releases though so may need some care) | 19:22 |
| ianw | there were already locked vos releases at the time (i thinki mentioned them) | 19:22 |
| AJaeger | clarkb: https://review.opendev.org/694912 addresses your comment in https://review.opendev.org/681582 - could you put both on your stack for later today, please? | 19:30 |
| fungi | ianw: yep, vos release calls are all hung from circa 2019-11-15 | 19:35 |
| fungi | not terminating | 19:35 |
| fungi | sounds like we can unceremoniously kill and retry them | 19:35 |
| ianw | i think overall we need to consider https://review.opendev.org/691824 | 19:36 |
| *** ociuhandu has joined #openstack-infra | 19:37 | |
| openstackgerrit | Merged openstack/project-config master: Add base promote job for moving static.o.o https://review.opendev.org/681582 | 19:38 |
| *** rkukura has quit IRC | 19:39 | |
| *** dpawlik has quit IRC | 19:41 | |
| *** ociuhandu has quit IRC | 19:47 | |
| *** dpawlik has joined #openstack-infra | 19:48 | |
| *** goldyfruit___ has joined #openstack-infra | 19:55 | |
| *** gmann is now known as gmann_afk | 19:57 | |
| *** goldyfruit_ has quit IRC | 19:57 | |
| *** michael-beaver has joined #openstack-infra | 20:02 | |
| *** Qiming has quit IRC | 20:03 | |
| clarkb | infra-root would anyone else like to reivew https://review.opendev.org/#/c/694181/12 before I approve it to have gitea01 request and opendev.org cert from LE? | 20:03 |
| fungi | i haven't been following it as closely as i should, so happy with you forging ahead | 20:04 |
| fungi | i'm going to start trying to kill vos release processes | 20:04 |
| ianw | corvus: is there more to do other than bumping the version in https://review.opendev.org/#/c/694894/ for gitea update? | 20:05 |
| corvus | clarkb: the certs will simply remain on 02-06 right? | 20:05 |
| ianw | (first rodeo there) | 20:05 |
| *** Qiming has joined #openstack-infra | 20:05 | |
| clarkb | corvus: yes 02-08 should be unaffected. https://review.opendev.org/#/c/694181/12/inventory/groups.yaml is the bit of the change that control that | 20:05 |
| clarkb | corvus: there is a second change to flip 02-08 over once 01 is happy | 20:05 |
| corvus | ianw: i think mordred has performed diffs of the templates between versions to see if there are changes we might need to update/match in our local templates. also, considering our lack of ui testing of gitea, sometimes it's helpful to run a build locally. but with a micro version bump, both things seem pretty low risk, so may not be necessary. | 20:07 |
| clarkb | also maybe we should coordinate the LE change and the gitea upgrade | 20:07 |
| clarkb | to avoid one or the other causing problems for each other | 20:08 |
| ianw | clarkb: yes :) i would not want to put in the gitea update too close to US end of day | 20:08 |
| clarkb | ianw: you good with me approving LE change now? | 20:09 |
| ianw | clarkb: yep, i'm around all day in the small chance of issues | 20:09 |
| clarkb | ok approving now | 20:09 |
| ianw | corvus: ok, cool, i can do a diff and note it in the change | 20:10 |
| ianw | corvus: with the buildset registry jobs, is it expected to have to use "image: zuul-jobs.buildset-registry:5000/zuul/nodepool-builder" (i.e. the full path to the buildset registry) to pick up the built images | 20:18 |
| ianw | AFAICT, docker doesn't support changing the default registry | 20:18 |
| AJaeger | config-core, regarding py2/3, here's https://review.opendev.org/694989 to unblock devstack. Please also review https://review.opendev.org/694834 and https://review.opendev.org/694912 | 20:19 |
| *** dpawlik has quit IRC | 20:22 | |
| corvus | ianw: no... where is that happening? | 20:25 |
| corvus | ianw: speculative images should automatically be used | 20:26 |
| *** tosky has quit IRC | 20:27 | |
| openstackgerrit | Merged openstack/project-config master: Drop broken legacy job from devstack https://review.opendev.org/694989 | 20:28 |
| openstackgerrit | Merged openstack/project-config master: Add v2 branch for monitoring by IRC gerritbot https://review.opendev.org/694834 | 20:30 |
| ianw | corvus: so they were not in my nodepool-builder job ... let me dig out some old logs | 20:32 |
| fungi | #status log manually killed all vos release processes running since 2019-11-15 on mirror-update.openstack.org and mirror-update.opendev.org servers | 20:32 |
| openstackstatus | fungi: finished logging | 20:32 |
| ianw | corvus: so here it is setting up the registry https://zuul.opendev.org/t/zuul/build/85aff5d08e524985b99e6369018b4d0c/log/job-output.txt#495 | 20:33 |
| ianw | then the pulls happen later @ https://zuul.opendev.org/t/zuul/build/85aff5d08e524985b99e6369018b4d0c/log/job-output.txt#33498 | 20:34 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: prometheus: add options to start the server and process collector https://review.opendev.org/599209 | 20:34 |
| ianw | those are the upstream sha hash's from dockerhub | 20:34 |
| ianw | when i switched to job to use the registry hostname, it pulled the speculative built images correctly | 20:35 |
| *** openstackgerrit has quit IRC | 20:35 | |
| ianw | then i started reading https://github.com/moby/moby/issues/33069 which makes me think you can't actually override the default | 20:36 |
| *** ociuhandu has joined #openstack-infra | 20:38 | |
| *** openstackgerrit has joined #openstack-infra | 20:41 | |
| openstackgerrit | Merged opendev/system-config master: Manage opendev.org cert with LE https://review.opendev.org/694181 | 20:41 |
| *** ociuhandu has quit IRC | 20:42 | |
| *** ociuhandu has joined #openstack-infra | 20:43 | |
| *** dpawlik has joined #openstack-infra | 20:45 | |
| fungi | clarkb: ^ heads up | 20:45 |
| *** eharney has quit IRC | 20:48 | |
| *** igordc has quit IRC | 20:50 | |
| AJaeger | could I get a second +2 on https://review.opendev.org/694912 , please? | 20:52 |
| *** goldyfruit_ has joined #openstack-infra | 20:58 | |
| *** dpawlik has quit IRC | 20:58 | |
| *** ociuhandu has quit IRC | 20:58 | |
| corvus | ianw: use-docker-mirror must run before use-buildset-registry, but it's running after in that job. use-docker-mirror will overwrite the config created by use-buildset-registry that instructs docker to use the buildset registry. (use-buildset-registry will, in turn, remove the mirror config, but that's non-fatal, and something we can fix later; we can ignore that detail for now) | 20:59 |
| *** goldyfruit___ has quit IRC | 21:00 | |
| corvus | ianw: left comments on change | 21:02 |
| corvus | https://zuul.opendev.org/t/zuul/build/85aff5d08e524985b99e6369018b4d0c/console is helpful for seeing the sequence | 21:03 |
| corvus | looks like it's invoked from install-docker | 21:03 |
| clarkb | infra-root https://gitea01.opendev.org:3000/ says LE cert \o/ | 21:03 |
| *** gmann_afk is now known as gmann | 21:03 | |
| clarkb | and opendev.org is a subject alt name | 21:04 |
| clarkb | so should be workign through the proxy too | 21:04 |
| ianw | corvus: ahh, thank you! makes sense now | 21:04 |
| clarkb | I think that means we can approve https://review.opendev.org/#/c/694184/10 whenever others are comfortable with the LE update on 01 | 21:04 |
| corvus | clarkb, ianw: i'm really impressed by the gitops on display here. the LE cert was a pretty significant change all done through automation :) | 21:05 |
| fungi | yes, this has been amazing | 21:05 |
| clarkb | corvus: and tested too :) | 21:05 |
| fungi | and rolled out successfully on the first try! | 21:06 |
| fungi | slight typo in the commit message for 694184, but approved | 21:08 |
| clarkb | oh yup 08 not 07 in the commit message | 21:08 |
| fungi | i was initially confused as to what had happened with 08 until i dug into the diff and realized that was just a typo | 21:13 |
| *** igordc has joined #openstack-infra | 21:15 | |
| *** dpawlik has joined #openstack-infra | 21:17 | |
| *** dpawlik has quit IRC | 21:23 | |
| *** rkukura has joined #openstack-infra | 21:23 | |
| *** ociuhandu has joined #openstack-infra | 21:29 | |
| clarkb | two neat side effects of this cert change. First is you can hit the backends directly with ssl verification. Second is you can check the cert on the frontend to see which backend you hit | 21:30 |
| fungi | both useful properties | 21:30 |
| fungi | the second moreso than the first (you can always instruct your client to ignore cert mismatches, but with layer 4 distribution there are no http headers indicating the proxied source) | 21:31 |
| *** iokiwi has joined #openstack-infra | 21:32 | |
| paladox | https://groups.google.com/forum/#!topic/repo-discuss/54FJmlfyUIQ is definitely nice! | 21:34 |
| fungi | yeah, we've had requests for that, particularly from documentation teams, who want a webui for creating documentation which could involve creating new changes and adding new files, not just altering what's been pushed by others | 21:36 |
| openstackgerrit | Merged opendev/system-config master: Manage opendev.org with LE on all giteas https://review.opendev.org/694184 | 21:42 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: prometheus: add options to start the server and process collector https://review.opendev.org/599209 | 21:44 |
| clarkb | I believe the other giteas will update in about 12 minutes | 21:47 |
| *** ociuhandu has quit IRC | 21:48 | |
| *** mriedem has quit IRC | 21:48 | |
| *** eharney has joined #openstack-infra | 21:48 | |
| *** jamesmcarthur has joined #openstack-infra | 21:48 | |
| *** lpetrut has quit IRC | 21:50 | |
| *** mriedem has joined #openstack-infra | 21:51 | |
| ianw | seems likely one of the ze's streaming has stopped again -> https://zuul.opendev.org/t/zuul/stream/669609d18c39435e9919ea80be0ca4bc?logfile=console.log | 21:53 |
| ianw | i show no listening 7900 on 04, 08, 09 | 21:54 |
| *** rcernin has joined #openstack-infra | 21:59 | |
| fungi | if you dig, you'll likely see executor processes sacrificed by the oom-killer in dmesg since the last restart | 22:00 |
| *** rcernin has quit IRC | 22:01 | |
| *** rcernin has joined #openstack-infra | 22:01 | |
| *** rcernin has quit IRC | 22:01 | |
| *** rcernin has joined #openstack-infra | 22:02 | |
| clarkb | all giteas but 07 now have LE certs. 07 appears to still be waiting for its cert to be issued? | 22:02 |
| ianw | fungi: yep; any objections to a rolling restart of them? | 22:03 |
| clarkb | fatal: [gitea07.opendev.org]: FAILED! => {"changed": false, "module_stderr": "", "module_stdout": "", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": -13} | 22:04 |
| ianw | clarkb: interesting, there's stuff in /etc/letsencrypt-certs | 22:04 |
| clarkb | it failed to run the handler | 22:05 |
| clarkb | ianw: ya I think the acme stuff completed successfully then ansible had an internal error trying to run the handler | 22:06 |
| ianw | hrm, the handlers have always been a bit odd. https://opendev.org/opendev/system-config/src/branch/master/playbooks/roles/letsencrypt-create-certs/handlers/main.yaml#L8 i could never get "listen:" to work | 22:06 |
| clarkb | MODULE FAILURE implies to me it is an ansible error and not an error in what ansible was running if that makes sense | 22:06 |
| clarkb | pabelanger: dmsimard ^ any idea what may cause that? | 22:07 |
| *** jtomasek has quit IRC | 22:07 | |
| clarkb | infra-root any objection to me manually running the handler steps on 07 so that we can get the cert updated there too? | 22:07 |
| clarkb | I don't expect the next ansible run will try to run it | 22:07 |
| ianw | no, i think the next run acme will not renew the cert so nothing should happen | 22:08 |
| ianw | i mean, "i agree with you, i think ..." :) | 22:08 |
| clarkb | ya I'm going to go ahead and do the cert copy manually then restart gitea there | 22:09 |
| ianw | or, you could try deleting the certs and see if the next run works? | 22:09 |
| clarkb | ianw: we are only allowed 5 renewals per week, probably best to avoid eating into those for now? | 22:09 |
| ianw | i think that's failed validation? | 22:10 |
| clarkb | "Renewals are treated specially: they don’t count against your Certificates per Registered Domain limit, but they are subject to a Duplicate Certificate limit of 5 per week." | 22:10 |
| dmsimard | clarkb: without more output than that it's hard to say what the issue is | 22:10 |
| clarkb | dmsimard: that is all ansible gave us | 22:10 |
| clarkb | and the same handler ran 7 times on other host just fine this afternoon :/ | 22:11 |
| dmsimard | clarkb: where at ? increasing verbosity might provide the actual traceback | 22:11 |
| fungi | 87.5% success rate | 22:11 |
| *** jtomasek has joined #openstack-infra | 22:11 | |
| clarkb | dmsimard: on bridge.openstack.org against gitea07.opendev.org | 22:11 |
| * dmsimard looks | 22:12 | |
| clarkb | dmsimard: its a handler for LE cert issuance/renewal so we can only run it 5 times per week in that setup before hitting the rate limit if I read LE docs properly | 22:12 |
| clarkb | why is ansible so verbose about warnings you can't fix but won't show you what an error is when they happen | 22:12 |
| clarkb | I don't understand ansible's logging | 22:12 |
| dmsimard | you're telling me :) | 22:13 |
| dmsimard | it tends to eat tracebacks unless you go -vvvv | 22:13 |
| clarkb | that is the opposite of what I want. Can we get them to disable the warnings unless I -vvvv and always spit out tracebacks on fatal errors? | 22:14 |
| clarkb | considering that we can force renewal by deleting the data in /etc/letsencrypt I think we can keep that in our back pocket. For now I'm going to copy the cert data over to gitea data dir and restart gitea on 07 so that it matches its siblings | 22:15 |
| dmsimard | I'm not sure -- if there is, I'd like to know too | 22:15 |
| *** priteau has quit IRC | 22:15 | |
| dmsimard | I have this problem a bit often with ara :p | 22:15 |
| dmsimard | clarkb: I think the logging callback (to /var/log/ansible/ansible.log) could be eating the traceback | 22:16 |
| ianw | clarkb: SGTM. i don't have any other ideas. are we 2.7 or 2.8 on bridge now? | 22:16 |
| dmsimard | clarkb: is the raw/foreground output somewhere ? | 22:16 |
| ianw | 2.8.0 ... maybe we should update? | 22:17 |
| clarkb | seems reasonable. To the current 2.8x? | 22:18 |
| clarkb | dmsimard: I think that is included in that file | 22:18 |
| clarkb | dmsimard: bash /opt/system-config/run_all.sh -c >> /var/log/ansible/run_all_cron.log 2>&1 is the command | 22:18 |
| clarkb | dmsimard: so that should include stdout and stderr in that file. | 22:18 |
| clarkb | gitea07 is now like the others. Cert renewal complete (for the next 3 months at least) | 22:19 |
| dmsimard | clarkb: ok, I see it but it's not verbose enough to spit the traceback :( | 22:19 |
| fungi | we still have our sslcheck set up as a fallback, but maybe we should switch it to check the individual backends since they have individual certs? | 22:20 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: bridge.o.o: update to latest Ansible https://review.opendev.org/695099 | 22:20 |
| fungi | otherwise if, say, 07 fails to renew then we might not catch it by just checking the lb | 22:21 |
| clarkb | fungi: that is an excellent idea | 22:21 |
| clarkb | I'll write that change now | 22:21 |
| ianw | clarkb: we can try 2.9.1 ... the -devel job has been passing but actually updating will kick off all the testinfra based jobs | 22:23 |
| ianw | dmsimard: is ARA 2.9.1 compatible? | 22:23 |
| dmsimard | ianw: it is | 22:24 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Validate all gitea backend certs https://review.opendev.org/695101 | 22:24 |
| clarkb | fungi: ^ | 22:24 |
| fungi | reviewing, thanks! | 22:25 |
| ianw | dmsimard: cool, i'll be interested to see if all our jobs pass on https://review.opendev.org/695099 then | 22:26 |
| clarkb | ya testing will be a good sanity check of 2.9.1 for us. We actually have pretty good testing of a variety of things now through our system-config run jobs | 22:27 |
| dmsimard | ianw: I really need to catch up, would be happy to help upgrading the current playbooks to use the latest version of ara instead of <1.0 | 22:29 |
| ianw | dmsimard: well me too; the latest ara supports the html file output? | 22:31 |
| dmsimard | the biggest issue was probably the lack of built-in interface and static generation which both landed in 1.2 | 22:31 |
| dmsimard | ianw: I have this WIP patch up in zuul: https://review.opendev.org/#/c/694622/ | 22:31 |
| dmsimard | demo of the built-in UI: https://api.trunk.demo.recordsansible.org/?order=-started | 22:32 |
| ianw | dmsimard: i mean it might be as easy as bumping https://opendev.org/opendev/system-config/src/branch/master/playbooks/bridge.yaml#L22 | 22:33 |
| dmsimard | however the static version doesn't have some features (like sorting) | 22:33 |
| ianw | i certainly like it for debugging those system-config jobs with the nested runs | 22:33 |
| *** pcaruana has quit IRC | 22:33 | |
| clarkb | ianw: ya was helpful to sort out the afs related issues yetserday | 22:33 |
| clarkb | basically I saw that ansible was running up to a point further than what we got in console log, then looked in syslog for more and was able to narrow it down to the specific task | 22:34 |
| mnaser | now that https://review.opendev.org/#/c/681582/4 has landed, does this mean we need to get the afs stuff created to unblock https://review.opendev.org/#/c/681583/5 ? | 22:38 |
| *** slaweq has quit IRC | 22:39 | |
| clarkb | mnaser: yes, we are currently doing prework for that (removing volume locks and rereleasing them to recover from a fileserver outage) | 22:39 |
| dmsimard | ianw: looking, I saw that the nested report is generated through the ara-report role (neat) but the CLI command in 1.x is different -- at least for the time being | 22:39 |
| clarkb | mnaser: I expect once that has settled we'll get the new volumes created pretty quickly | 22:39 |
| mnaser | clarkb: ok cool, great, just wanted to make sure i stayed in the loop :) | 22:39 |
| dmsimard | ianw: instead of "ara generate html <path>" it's "ara-manage generate <path>" | 22:39 |
| dmsimard | there is some tweaking necessary in ara-report | 22:40 |
| *** ociuhandu has joined #openstack-infra | 22:41 | |
| *** aaronsheffield has quit IRC | 22:41 | |
| *** sreejithp has quit IRC | 22:41 | |
| *** sreejithp has joined #openstack-infra | 22:42 | |
| *** jamesmcarthur has quit IRC | 22:42 | |
| openstackgerrit | David Moreau Simard proposed zuul/zuul-jobs master: DNM: test ara-report role using ara>1.0 https://review.opendev.org/695107 | 22:43 |
| clarkb | ianw: fungi: let me know if I can help with the afs recovery now that gitea ssl stuff is mostly done | 22:44 |
| *** jaosorior has quit IRC | 22:45 | |
| *** ociuhandu has quit IRC | 22:46 | |
| openstackgerrit | David Moreau Simard proposed opendev/system-config master: DNM: Test ara 1.2 for bridge https://review.opendev.org/695108 | 22:46 |
| dmsimard | ianw: ^ quick stab at it | 22:47 |
| dmsimard | getting pulled for dinner, bbl | 22:47 |
| *** slaweq has joined #openstack-infra | 22:48 | |
| *** munimeha1 has quit IRC | 22:49 | |
| *** jaosorior has joined #openstack-infra | 22:49 | |
| fungi | clarkb: mostly just waiting to see if it resumes normally after i killed the stuck vos releases | 22:49 |
| fungi | some of them may exceed the timeouts and need running manually though | 22:50 |
| *** jaosorior has quit IRC | 22:52 | |
| *** slaweq has quit IRC | 22:53 | |
| ianw | fungi: yeah ... most of them? | 22:55 |
| ianw | fungi: do you want to think about https://review.opendev.org/#/c/691824/ and maybe we can try it for fedora, say? | 22:56 |
| ianw | otherwise i guess it will be screen sessions | 22:56 |
| fungi | sure, will take a look in a sec | 22:58 |
| *** slaweq has joined #openstack-infra | 23:03 | |
| openstackgerrit | Merged opendev/system-config master: Validate all gitea backend certs https://review.opendev.org/695101 | 23:06 |
| *** ociuhandu has joined #openstack-infra | 23:07 | |
| *** tkajinam has joined #openstack-infra | 23:07 | |
| *** ociuhandu has quit IRC | 23:09 | |
| *** ociuhandu has joined #openstack-infra | 23:11 | |
| *** armax has quit IRC | 23:13 | |
| *** armax has joined #openstack-infra | 23:14 | |
| *** sreejithp has quit IRC | 23:15 | |
| *** ociuhandu has quit IRC | 23:16 | |
| *** slaweq has quit IRC | 23:16 | |
| *** armax has quit IRC | 23:16 | |
| *** priteau has joined #openstack-infra | 23:17 | |
| *** tosky has joined #openstack-infra | 23:24 | |
| *** slaweq has joined #openstack-infra | 23:25 | |
| *** tosky has quit IRC | 23:26 | |
| *** jklare has quit IRC | 23:28 | |
| *** calbers has quit IRC | 23:28 | |
| *** slaweq has quit IRC | 23:30 | |
| openstackgerrit | Merged opendev/system-config master: AFS: Allow for remote vos release with localauth https://review.opendev.org/691824 | 23:30 |
| *** ociuhandu has joined #openstack-infra | 23:30 | |
| fungi | ianw: ^ | 23:31 |
| *** dchen has joined #openstack-infra | 23:31 | |
| ianw | fungi: cool, let's see if the next pulse deploys it as we expect and if it works? | 23:31 |
| * fungi nods | 23:32 | |
| *** jklare has joined #openstack-infra | 23:32 | |
| *** calbers has joined #openstack-infra | 23:35 | |
| *** ociuhandu has quit IRC | 23:40 | |
| *** slaweq has joined #openstack-infra | 23:42 | |
| ianw | fungi: i'm not sure if that script on the afs server side should implement a lock to avoid parallel releases | 23:43 |
| *** calbers has quit IRC | 23:45 | |
| *** jklare has quit IRC | 23:45 | |
| fungi | yeah, that was noted in the commit message too | 23:45 |
| fungi | presumably we'd just do a per-volume lock | 23:46 |
| fungi | though i think vos release has its own locking anyway? | 23:46 |
| *** slaweq has quit IRC | 23:47 | |
| *** calbers has joined #openstack-infra | 23:48 | |
| ianw | yeah, the existing lock on the update script side is the per-volume lock | 23:51 |
| ianw | i wouldn't want to hazard a guess about overall implications of multiple releases on different volumes | 23:52 |
| *** slaweq has joined #openstack-infra | 23:57 | |
| ianw | ze09 had actually completely dropped out; i think oom killed it | 23:58 |
| *** ociuhandu has joined #openstack-infra | 23:58 | |
| ianw | i'm going to reboot the host, just for sanity | 23:58 |
| *** goldyfruit_ has quit IRC | 23:59 | |
| ianw | #status log restarted ze04, ze08 & ze09 due to OOM kills of the streaming daemon. ze09 zuul processes were completely stopped so rebooted the host | 23:59 |
| openstackstatus | ianw: finished logging | 23:59 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!