| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: configure-mirrors: fix stripped newline https://review.opendev.org/c/zuul/zuul-jobs/+/831323 | 00:04 |
|---|---|---|
| clarkb | agenda sent | 00:10 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: configure-mirrors: fix stripped newline https://review.opendev.org/c/zuul/zuul-jobs/+/831323 | 00:10 |
| *** rlandy|ruck is now known as rlandy|out | 00:20 | |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: ensure-podman: add containernetworking-plugins https://review.opendev.org/c/zuul/zuul-jobs/+/831326 | 00:30 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: ensure-podman: add containernetworking-plugins https://review.opendev.org/c/zuul/zuul-jobs/+/831326 | 01:45 |
| opendevreview | Merged zuul/zuul-jobs master: configure-mirrors: fix stripped newline https://review.opendev.org/c/zuul/zuul-jobs/+/831323 | 03:55 |
| *** prometheanfire is now known as Guest844 | 05:52 | |
| *** Guest844 is now known as prometheanfire | 05:56 | |
| *** ysandeep is now known as ysandeep|afk | 06:29 | |
| opendevreview | wangxiyuan proposed zuul/zuul-jobs master: Add openEuler to iptalbe firewall persist https://review.opendev.org/c/zuul/zuul-jobs/+/830706 | 06:47 |
| *** ysandeep|afk is now known as ysandeep | 07:41 | |
| frickler | ianw: this first regressed in the periodic jobs, not sure if some change in buster or something in our builds maybe? https://zuul.opendev.org/t/zuul/builds?job_name=zuul-jobs-test-ensure-pip-debian-buster&project=zuul/zuul-jobs | 07:48 |
| frickler | ERROR: Package 'pluggy' requires a different Python: 2.7.16 not in '>=3.6' | 07:52 |
| frickler | which is weird, because pluggy-1.0.0 was released July 2021. | 07:53 |
| *** jpena|off is now known as jpena | 08:33 | |
| frickler | hmm, cannot reproduce on a local buster instance, will hold a node for checking | 09:45 |
| ianw | frickler: thanks, yeah on my todo list for tomorrow as i have a typo change out that hits that | 09:47 |
| frickler | ianw: that's where I found the issue, yes ;) | 09:48 |
| *** mgoddard- is now known as mgoddard | 10:04 | |
| frickler | ah, that's another case of our wheel mirror breaking things serving py2.py3 wheel that only work with py3 really | 10:15 |
| frickler | https://paste.opendev.org/show/bCyaxggbD2jDN9iKgBhr/ | 10:16 |
| frickler | so we broke that when we fixed wheel publishing. not sure of the best way forward. maybe disable those for py2 jobs? that's likely to have some other fallout due to wheels no longer being buildable, but maybe we want to have that fallout anyways | 10:18 |
| opendevreview | Dr. Jens Harbott proposed zuul/zuul-jobs master: Don't enable pip mirror for Debian Buster https://review.opendev.org/c/zuul/zuul-jobs/+/831354 | 10:29 |
| frickler | corvus: I'm seeing openstack items in the periodic queue at https://zuul.opendev.org/t/zuul/status , is that related to the bug you mentioned earlier? | 10:35 |
| *** rlandy|out is now known as rlandy|ruck | 11:18 | |
| *** bhagyashris is now known as bhagyashris|ruck | 11:30 | |
| frickler | another option for ^^ would be in the ensure-pip check to add a constraints file containing "pluggy<1;python_version=='2.7'", but that also seems quite hacky to me | 12:23 |
| fungi | frickler: i'm curious why we built a wheel for a version of pluggy which isn't installable on that platform in the first place, unless we used a pip that was too old to check the metadata in pypi | 12:27 |
| fungi | oh, i get it. we built that wheel with python3 but the python3 and python jobs use the same platform-specific pool of wheels | 12:30 |
| fungi | if they weren't shared between interpreters for that platform it wouldn't be an issue, but i'm not sure how we'd tell it which path to use | 12:32 |
| fungi | i guess if we fronted it with a pypi simple api and served metadata indicating which interpreter versions each version had been built with, that would be the non-hacky "make it work like pypi does" solution, but that seems like a lot of work | 12:34 |
| *** ysandeep is now known as ysandeep|break | 12:38 | |
| frickler | fungi: yes, the latter would be the correct full-size solution, but not sure how feasible it would be. also we can still try to sit this out until we really don't need py2 anymore | 13:03 |
| opendevreview | yatin proposed openstack/project-config master: Add Resource Usage Dashboard https://review.opendev.org/c/openstack/project-config/+/831374 | 13:05 |
| ykarel | clarkb, fungi can u check ^, this is for Resource Usage that we used to ask to be collected from executors in past | 13:05 |
| ykarel | and u suggested to collect from graphite | 13:05 |
| ykarel | i used % to get relative data across tenants/projects | 13:06 |
| ykarel | as with numbers i didn't got what scale they were from | 13:06 |
| ykarel | i just compared from previous collected output from ze01 and it looks similar to that | 13:08 |
| ykarel | https://paste.opendev.org/show/811516/ 11/30/2021 - 12/06/2021 | 13:08 |
| ykarel | https://paste.opendev.org/show/bYTlHXfbX84aESK6cLMM/ 11/08/2021 - 11/14/2021 | 13:08 |
| ykarel | also i am not good in dashboards, i just attempted to put something up | 13:09 |
| *** ysandeep|break is now known as ysandeep | 13:29 | |
| opendevreview | yatin proposed openstack/project-config master: Add Some periodic jobs to Neutron Dasboard https://review.opendev.org/c/openstack/project-config/+/831378 | 13:33 |
| opendevreview | yatin proposed openstack/project-config master: Add Some periodic jobs to Neutron Dashboard https://review.opendev.org/c/openstack/project-config/+/831378 | 13:35 |
| *** pojadhav is now known as pojadhav|afk | 14:19 | |
| *** pojadhav|afk is now known as pojadhav | 14:57 | |
| *** pojadhav is now known as pojadhav|afk | 15:18 | |
| *** ysandeep is now known as ysandeep|dinner | 15:34 | |
| *** ysandeep|dinner is now known as ysandeep | 15:46 | |
| *** Guest721 is now known as diablo_rojo_phone | 15:47 | |
| opendevreview | Clark Boylan proposed openstack/project-config master: Set airship-citycloud max-servers to 0 https://review.opendev.org/c/openstack/project-config/+/831397 | 15:57 |
| opendevreview | Clark Boylan proposed openstack/project-config master: Remove airship-citycloud nodepool provider https://review.opendev.org/c/openstack/project-config/+/831398 | 15:57 |
| clarkb | for ^ do I need to set the image list to an empty list first so that nodepool can delete the images from its zk db? | 15:57 |
| clarkb | Or will it handle that on its own? | 15:57 |
| yoctozepto | hi; I've asked frickler personally but he pointed out it's best if I ask publicly - I need a node held for kolla-ansible-debian-source-ironic job in https://review.opendev.org/c/openstack/kolla-ansible/+/644271 | 16:03 |
| yoctozepto | the debian's behaviour is weird and I need to see why :/ | 16:04 |
| frickler | yoctozepto: do you want to reduce the number of jobs running in that patch or in a patch stacked on top? assuming this might need multiple iterations that might be helpful | 16:05 |
| yoctozepto | frickler: yes, I was about to ask about that | 16:05 |
| yoctozepto | I will trim it in that patch | 16:05 |
| opendevreview | Merged zuul/zuul-jobs master: [ensure-python] Fix for CentOS/RHEL 9 packages https://review.opendev.org/c/zuul/zuul-jobs/+/829533 | 16:05 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Remove airship-citycloud resources https://review.opendev.org/c/opendev/system-config/+/831400 | 16:06 |
| yoctozepto | frickler: done | 16:08 |
| frickler | yoctozepto: o.k., the hold is set, once you submit the update, it should get triggered. | 16:08 |
| frickler | you can also watch it here with the new UI https://zuul.opendev.org/t/openstack/autohold/0000000031 | 16:09 |
| clarkb | infra-root I was notified overnight (and noticed when I sat down this morning) that the airship-citycloud resources are apparently going away "this evening" I think that is relative to CEST so nowish | 16:09 |
| fungi | clarkb: i'm not sure if nodepool has grown the smarts to notice when it should clean up uploaded images for a nonexistent provider | 16:09 |
| clarkb | fungi: ok let me respin this change a bit | 16:09 |
| fungi | so an empty image list may be prudent | 16:09 |
| yoctozepto | frickler: should I resubmit the patch again? | 16:09 |
| frickler | yoctozepto: no, the trigger happens at the end of the job, so we are good as is | 16:09 |
| yoctozepto | frickler: ack, thanks :-) | 16:10 |
| opendevreview | Clark Boylan proposed openstack/project-config master: Set airship-citycloud max-servers to 0 https://review.opendev.org/c/openstack/project-config/+/831397 | 16:11 |
| opendevreview | Clark Boylan proposed openstack/project-config master: Remove airship-citycloud nodepool provider https://review.opendev.org/c/openstack/project-config/+/831398 | 16:11 |
| clarkb | fungi: ^ I combo'd it with the max-server: 0 change | 16:11 |
| fungi | frickler: ^ | 16:12 |
| frickler | clarkb: I think we might want to keep the grafana page for sentimental reasons for some time? i.e. to be able to still see the history? | 16:14 |
| clarkb | frickler: sure I can keep that for now | 16:15 |
| clarkb | I'll respin the child change | 16:15 |
| fungi | also the system-config change doesn't really need to depend on the project-config change, does it? | 16:17 |
| fungi | seems like they could both gate in parallel | 16:17 |
| opendevreview | Clark Boylan proposed openstack/project-config master: Remove airship-citycloud nodepool provider https://review.opendev.org/c/openstack/project-config/+/831398 | 16:18 |
| clarkb | fungi: it does because it provides the nodepool credentials | 16:19 |
| clarkb | fungi: if they get out of order nodepool will have a sad | 16:19 |
| clarkb | frickler: ^ updated | 16:19 |
| fungi | clarkb: oh, yep got it, we don't want to merge it until the remaining nodes and images have drained | 16:20 |
| *** ysandeep is now known as ysandeep|out | 16:21 | |
| fungi | so should probably be wip since it's not safe to merge until that time | 16:21 |
| clarkb | ya exactly. I can WIP it | 16:21 |
| frickler | we'll also need dns updates after the mirror removal, not urgent, but easy to forget | 16:23 |
| clarkb | frickler: ++ I'll push a change for that now so it is on the todo list | 16:23 |
| opendevreview | Clark Boylan proposed opendev/zone-opendev.org master: Remove the airship citycloud mirror dns records https://review.opendev.org/c/opendev/zone-opendev.org/+/831405 | 16:27 |
| clarkb | frickler: ^ thank you for calling that out | 16:27 |
| opendevreview | Merged openstack/project-config master: Set airship-citycloud max-servers to 0 https://review.opendev.org/c/openstack/project-config/+/831397 | 16:35 |
| *** bhagyashris_ is now known as bhagyashris|ruck | 16:43 | |
| clarkb | that change ahs applied to the servers at this point and we should see it start to wind down | 16:56 |
| clarkb | I went ahead and added mirror01.kna1.airship-citycloud.opendev.org to the emergency.yaml hosts file to avoid problems if it gets removed out from under us | 16:57 |
| clarkb | since that will affect base and other playbooks iirc | 16:58 |
| yoctozepto | frickler: it's held, and my public ssh key is in https://api.github.com/users/yoctozepto/keys | 16:58 |
| clarkb | Side note: https://review.opendev.org/c/opendev/system-config/+/831400 shows a list of merge conflicts. I think that merge conflict checking must be working :) | 17:01 |
| clarkb | infra-root I'm hoping to get to deleting the subunit worker nodes and openstack health at some point this week. I think they were all pretty stateless servers but could do snapshots for them anyway. (the real data is in the databse which I'm not sure what the best way to handle that is given its size) | 17:05 |
| fungi | we can snapshot the trove instance before we delete it | 17:08 |
| fungi | i've done that with others | 17:08 |
| opendevreview | Merged openstack/project-config master: Add Resource Usage Dashboard https://review.opendev.org/c/openstack/project-config/+/831374 | 17:08 |
| yoctozepto | or could someone else let me in on https://zuul.opendev.org/t/openstack/autohold/0000000031 ? :-) | 17:08 |
| clarkb | fungi: this is our biggest instance by far though. But maybe that is fine | 17:08 |
| clarkb | yoctozepto: ya give me a sec | 17:09 |
| yoctozepto | clarkb: :-) | 17:09 |
| clarkb | yoctozepto: root@172.99.69.81 | 17:11 |
| yoctozepto | clarkb: thanks | 17:17 |
| *** jpena is now known as jpena|off | 17:38 | |
| clarkb | There is one node in the airship cloud that seems to be failing to delete. I suspect we may need to perform manual intervention on that one. | 17:54 |
| clarkb | corvus: ^ I don't think we've had to do that with the zk situation (or at least I don't recall doing it). I assume it is as simple as deleting the node records directly? | 17:56 |
| clarkb | I'll approve the full removal change once the in-use node is gone under that assumption | 17:56 |
| corvus | clarkb: if the launcher is unable to delete the node by removal time, i think there is a nodepool command to delete the zk node directly | 17:59 |
| clarkb | oh neat /me looks | 17:59 |
| clarkb | corvus: looks like `nodepool erase $provider` thanks! | 18:00 |
| clarkb | ok we are down to the one deleting node. I'm going to approve the second project-config chagne now | 18:13 |
| clarkb | then we can run the nodepool erase command since I believe it will be sad about that deleting node | 18:14 |
| opendevreview | Merged openstack/project-config master: Remove airship-citycloud nodepool provider https://review.opendev.org/c/openstack/project-config/+/831398 | 18:23 |
| clarkb | ok I need some food, then meeting and can followup on ^ and erase the provider from zk if necessary. Thank you for all the help with reviews | 18:26 |
| *** rlandy|ruck is now known as rlandy|ruck|biab | 18:35 | |
| *** rlandy|ruck|biab is now known as rlandy|ruck | 18:53 | |
| yoctozepto | I powered off the node held for https://zuul.opendev.org/t/openstack/autohold/0000000031 | 19:08 |
| yoctozepto | thank you | 19:08 |
| clarkb | yoctozepto: we're good to delete it then? | 19:08 |
| yoctozepto | clarkb: yes | 19:08 |
| yoctozepto | clarkb: and disable the autohold I presume | 19:09 |
| clarkb | yoctozepto: it auto disables. I'll take care of that after our meeting | 19:10 |
| yoctozepto | clarkb: ack | 19:10 |
| opendevreview | Merged opendev/system-config master: encrypt-logs: turn on for all prod playbooks https://review.opendev.org/c/opendev/system-config/+/830784 | 19:37 |
| clarkb | infra-root https://review.opendev.org/c/opendev/system-config/+/831400 is ready to go now. I ran nodepool erase airship-kna1 and the deleting node no longer shows up in listings | 19:37 |
| clarkb | I've just realized that we should keep the clouds.yaml on bridge for now to do any other additional cleanup that may be necessary | 19:39 |
| clarkb | let me respin really quickly | 19:39 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Remove airship-citycloud resources https://review.opendev.org/c/opendev/system-config/+/831400 | 19:40 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Remove airship-kna1 from bridge clouds.yaml https://review.opendev.org/c/opendev/system-config/+/831441 | 19:55 |
| opendevreview | Clark Boylan proposed openstack/project-config master: Remove airship-kna1 grafana dashboard https://review.opendev.org/c/openstack/project-config/+/831442 | 19:57 |
| clarkb | I've WIP'd those last two changes to ensure they don't land early. but 831400 should be reviewable and mergable. Please do that if you have time :) | 19:59 |
| * clarkb grabs lunch in the meantime | 20:01 | |
| frickler | looking at https://review.opendev.org/c/opendev/system-config/+/831400/2/inventory/service/host_vars/bridge.openstack.org.yaml , gerrit shows a short popup that files with lines > 500 chars will not be syntax highlighted. no bug, just a funny anecdote maybe | 20:03 |
| frickler | (it's the long rsa ssh keys if anyone wonders and doesn't want to check themselves) | 20:04 |
| opendevreview | Dr. Jens Harbott proposed zuul/zuul-jobs master: Fix ensure-pip test on Debian Buster https://review.opendev.org/c/zuul/zuul-jobs/+/831443 | 20:13 |
| *** dtantsur_ is now known as dtantsur | 20:24 | |
| clarkb | I've approved the system-config change to remove the airship citycloud resources | 20:59 |
| clarkb | I'll monitor that | 20:59 |
| opendevreview | Tristan Cacqueray proposed zuul/zuul-jobs master: DNM: generate ansi log for benchmark purpose https://review.opendev.org/c/zuul/zuul-jobs/+/763054 | 21:07 |
| clarkb | The airship-kna1 mirror removal change should land soon. I'll look into delting the mirror after the school run | 22:01 |
| opendevreview | Merged opendev/system-config master: Remove airship-citycloud resources https://review.opendev.org/c/opendev/system-config/+/831400 | 22:02 |
| clarkb | the kex handler fixup for gerrit merged | 22:11 |
| clarkb | frickler: ^ you may want to retest gerrit master soon just to be sure it works as expected for you (I tried testing it too and it seemed to work from here) | 22:11 |
| fungi | ooh! that'll be nice. is that slated for 3.5.x then, or not until 3.6? | 22:12 |
| Clark[m] | 3.6 as that is what got the updated MINA | 22:18 |
| Clark[m] | We could possibly vendor the mina changes in Gerrit on 3.5 but that seems like a lot of effort | 22:19 |
| fungi | yeah, too bad. for some reason i thought they had targeted the updated mina at a 3.5 patch release | 22:31 |
| clarkb | infra-root: 1ffefdcc-1a2e-4e22-aaba-2c9c5608c924 | mirror01.kna1.airship-citycloud.opendev.org <- I'll delete this server if that looks correct to you | 22:59 |
| ianw | ++; probably has a volume attached? | 22:59 |
| clarkb | no looks like it uses a 250GB flavor type instead | 23:00 |
| clarkb | I checked a volume listing and it was empty | 23:00 |
| ianw | ahh that could be right too :) | 23:01 |
| clarkb | ok it is gone | 23:03 |
| clarkb | #status log Deleted mirror01.kna1.airship-citycloud.opendev.org to finish up resource cleanup in that cloud provider now that it has been removed from Nodepool. | 23:03 |
| opendevstatus | clarkb: finished logging | 23:03 |
| fungi | thanks clarkb! | 23:10 |
| clarkb | ykarel isn't here anymore but seems https://grafana.opendev.org/d/94891e7b01/resource-usage-by-tenants-and-projects?orgId=1 is working | 23:10 |
| opendevreview | Merged opendev/zone-opendev.org master: Remove the airship citycloud mirror dns records https://review.opendev.org/c/opendev/zone-opendev.org/+/831405 | 23:13 |
| *** clarkb is now known as Guest985 | 23:32 | |
| *** Guest985 is now known as clarkb | 23:41 | |
| jentoio | question on the zull-registry container maintence task. I noticed that a zuul userid:groupid is already there in system-config/inventory/service/group_vars/zuul.yaml | 23:54 |
| jentoio | should I use this one of create a new one? I'm not sure how group_vars are inherited in playbooks yet | 23:54 |
| clarkb | jentoio: we should probably try to reuse that definition. Lets see if the registry is in that group though | 23:58 |
| clarkb | jentoio: https://opendev.org/opendev/system-config/src/branch/master/inventory/service/groups.yaml that file defines all of our groups based on hostname matching. In this case insecure-ci-registry is the zuul-registry host and it is part of the registry group and not the zuul group | 23:59 |
| clarkb | jentoio: in that case I think you can just update this group instead to contain the same content: https://opendev.org/opendev/system-config/src/branch/master/inventory/service/group_vars/registry.yaml | 23:59 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!