Tuesday, 2022-03-01

opendevreviewIan Wienand proposed zuul/zuul-jobs master: configure-mirrors: fix stripped newline  https://review.opendev.org/c/zuul/zuul-jobs/+/83132300:04
clarkbagenda sent00:10
opendevreviewIan Wienand proposed zuul/zuul-jobs master: configure-mirrors: fix stripped newline  https://review.opendev.org/c/zuul/zuul-jobs/+/83132300:10
*** rlandy|ruck is now known as rlandy|out00:20
opendevreviewIan Wienand proposed zuul/zuul-jobs master: ensure-podman: add containernetworking-plugins  https://review.opendev.org/c/zuul/zuul-jobs/+/83132600:30
opendevreviewIan Wienand proposed zuul/zuul-jobs master: ensure-podman: add containernetworking-plugins  https://review.opendev.org/c/zuul/zuul-jobs/+/83132601:45
opendevreviewMerged zuul/zuul-jobs master: configure-mirrors: fix stripped newline  https://review.opendev.org/c/zuul/zuul-jobs/+/83132303:55
*** prometheanfire is now known as Guest84405:52
*** Guest844 is now known as prometheanfire05:56
*** ysandeep is now known as ysandeep|afk06:29
opendevreviewwangxiyuan proposed zuul/zuul-jobs master: Add openEuler to iptalbe firewall persist  https://review.opendev.org/c/zuul/zuul-jobs/+/83070606:47
*** ysandeep|afk is now known as ysandeep07:41
fricklerianw: this first regressed in the periodic jobs, not sure if some change in buster or something in our builds maybe? https://zuul.opendev.org/t/zuul/builds?job_name=zuul-jobs-test-ensure-pip-debian-buster&project=zuul/zuul-jobs07:48
fricklerERROR: Package 'pluggy' requires a different Python: 2.7.16 not in '>=3.6'07:52
fricklerwhich is weird, because pluggy-1.0.0 was released July 2021.07:53
*** jpena|off is now known as jpena08:33
fricklerhmm, cannot reproduce on a local buster instance, will hold a node for checking09:45
ianwfrickler: thanks, yeah on my todo list for tomorrow as i have a typo change out that hits that09:47
fricklerianw: that's where I found the issue, yes ;)09:48
*** mgoddard- is now known as mgoddard10:04
fricklerah, that's another case of our wheel mirror breaking things serving py2.py3 wheel that only work with py3 really10:15
fricklerso we broke that when we fixed wheel publishing. not sure of the best way forward. maybe disable those for py2 jobs? that's likely to have some other fallout due to wheels no longer being buildable, but maybe we want to have that fallout anyways10:18
opendevreviewDr. Jens Harbott proposed zuul/zuul-jobs master: Don't enable pip mirror for Debian Buster  https://review.opendev.org/c/zuul/zuul-jobs/+/83135410:29
fricklercorvus: I'm seeing openstack items in the periodic queue at https://zuul.opendev.org/t/zuul/status , is that related to the bug you mentioned earlier?10:35
*** rlandy|out is now known as rlandy|ruck11:18
*** bhagyashris is now known as bhagyashris|ruck11:30
frickleranother option for ^^ would be in the ensure-pip check to add a constraints file containing "pluggy<1;python_version=='2.7'", but that also seems quite hacky to me12:23
fungifrickler: i'm curious why we built a wheel for a version of pluggy which isn't installable on that platform in the first place, unless we used a pip that was too old to check the metadata in pypi12:27
fungioh, i get it. we built that wheel with python3 but the python3 and python jobs use the same platform-specific pool of wheels12:30
fungiif they weren't shared between interpreters for that platform it wouldn't be an issue, but i'm not sure how we'd tell it which path to use12:32
fungii guess if we fronted it with a pypi simple api and served metadata indicating which interpreter versions each version had been built with, that would be the non-hacky "make it work like pypi does" solution, but that seems like a lot of work12:34
*** ysandeep is now known as ysandeep|break12:38
fricklerfungi: yes, the latter would be the correct full-size solution, but not sure how feasible it would be. also we can still try to sit this out until we really don't need py2 anymore13:03
opendevreviewyatin proposed openstack/project-config master: Add Resource Usage Dashboard  https://review.opendev.org/c/openstack/project-config/+/83137413:05
ykarelclarkb, fungi can u check ^, this is for Resource Usage that we used to ask to be collected from executors in past13:05
ykareland u suggested to collect from graphite13:05
ykareli used % to get relative data across tenants/projects13:06
ykarelas with numbers i didn't got what scale they were from13:06
ykareli just compared from previous collected output from ze01 and it looks similar to that13:08
ykarelhttps://paste.opendev.org/show/811516/ 11/30/2021 - 12/06/202113:08
ykarel    https://paste.opendev.org/show/bYTlHXfbX84aESK6cLMM/ 11/08/2021 - 11/14/202113:08
ykarelalso i am not good in dashboards, i just attempted to put something up13:09
*** ysandeep|break is now known as ysandeep13:29
opendevreviewyatin proposed openstack/project-config master: Add Some periodic jobs to Neutron Dasboard  https://review.opendev.org/c/openstack/project-config/+/83137813:33
opendevreviewyatin proposed openstack/project-config master: Add Some periodic jobs to Neutron Dashboard  https://review.opendev.org/c/openstack/project-config/+/83137813:35
*** pojadhav is now known as pojadhav|afk14:19
*** pojadhav|afk is now known as pojadhav14:57
*** pojadhav is now known as pojadhav|afk15:18
*** ysandeep is now known as ysandeep|dinner15:34
*** ysandeep|dinner is now known as ysandeep15:46
*** Guest721 is now known as diablo_rojo_phone15:47
opendevreviewClark Boylan proposed openstack/project-config master: Set airship-citycloud max-servers to 0  https://review.opendev.org/c/openstack/project-config/+/83139715:57
opendevreviewClark Boylan proposed openstack/project-config master: Remove airship-citycloud nodepool provider  https://review.opendev.org/c/openstack/project-config/+/83139815:57
clarkbfor ^ do I need to set the image list to an empty list first so that nodepool can delete the images from its zk db?15:57
clarkbOr will it handle that on its own?15:57
yoctozeptohi; I've asked frickler personally but he pointed out it's best if I ask publicly - I need a node held for kolla-ansible-debian-source-ironic job in https://review.opendev.org/c/openstack/kolla-ansible/+/64427116:03
yoctozeptothe debian's behaviour is weird and I need to see why :/16:04
frickleryoctozepto: do you want to reduce the number of jobs running in that patch or in a patch stacked on top? assuming this might need multiple iterations that might be helpful16:05
yoctozeptofrickler: yes, I was about to ask about that16:05
yoctozeptoI will trim it in that patch16:05
opendevreviewMerged zuul/zuul-jobs master: [ensure-python] Fix for CentOS/RHEL 9 packages  https://review.opendev.org/c/zuul/zuul-jobs/+/82953316:05
opendevreviewClark Boylan proposed opendev/system-config master: Remove airship-citycloud resources  https://review.opendev.org/c/opendev/system-config/+/83140016:06
yoctozeptofrickler: done16:08
frickleryoctozepto: o.k., the hold is set, once you submit the update, it should get triggered.16:08
frickleryou can also watch it here with the new UI https://zuul.opendev.org/t/openstack/autohold/000000003116:09
clarkbinfra-root I was notified overnight (and noticed when I sat down this morning) that the airship-citycloud resources are apparently going away "this evening" I think that is relative to CEST so nowish16:09
fungiclarkb: i'm not sure if nodepool has grown the smarts to notice when it should clean up uploaded images for a nonexistent provider16:09
clarkbfungi: ok let me respin this change a bit16:09
fungiso an empty image list may be prudent16:09
yoctozeptofrickler: should I resubmit the patch again?16:09
frickleryoctozepto: no, the trigger happens at the end of the job, so we are good as is16:09
yoctozeptofrickler: ack, thanks :-)16:10
opendevreviewClark Boylan proposed openstack/project-config master: Set airship-citycloud max-servers to 0  https://review.opendev.org/c/openstack/project-config/+/83139716:11
opendevreviewClark Boylan proposed openstack/project-config master: Remove airship-citycloud nodepool provider  https://review.opendev.org/c/openstack/project-config/+/83139816:11
clarkbfungi: ^ I combo'd it with the max-server: 0 change16:11
fungifrickler: ^16:12
fricklerclarkb: I think we might want to keep the grafana page for sentimental reasons for some time? i.e. to be able to still see the history?16:14
clarkbfrickler: sure I can keep that for now16:15
clarkbI'll respin the child change16:15
fungialso the system-config change doesn't really need to depend on the project-config change, does it?16:17
fungiseems like they could both gate in parallel16:17
opendevreviewClark Boylan proposed openstack/project-config master: Remove airship-citycloud nodepool provider  https://review.opendev.org/c/openstack/project-config/+/83139816:18
clarkbfungi: it does because it provides the nodepool credentials16:19
clarkbfungi: if they get out of order nodepool will have a sad16:19
clarkbfrickler: ^ updated16:19
fungiclarkb: oh, yep got it, we don't want to merge it until the remaining nodes and images have drained16:20
*** ysandeep is now known as ysandeep|out16:21
fungiso should probably be wip since it's not safe to merge until that time16:21
clarkbya exactly. I can WIP it16:21
fricklerwe'll also need dns updates after the mirror removal, not urgent, but easy to forget16:23
clarkbfrickler: ++ I'll push a change for that now so it is on the todo list16:23
opendevreviewClark Boylan proposed opendev/zone-opendev.org master: Remove the airship citycloud mirror dns records  https://review.opendev.org/c/opendev/zone-opendev.org/+/83140516:27
clarkbfrickler: ^ thank you for calling that out16:27
opendevreviewMerged openstack/project-config master: Set airship-citycloud max-servers to 0  https://review.opendev.org/c/openstack/project-config/+/83139716:35
*** bhagyashris_ is now known as bhagyashris|ruck16:43
clarkbthat change ahs applied to the servers at this point and we should see it start to wind down16:56
clarkbI went ahead and added mirror01.kna1.airship-citycloud.opendev.org to the emergency.yaml hosts file to avoid problems if it gets removed out from under us16:57
clarkbsince that will affect base and other playbooks iirc16:58
yoctozeptofrickler: it's held, and my public ssh key is in https://api.github.com/users/yoctozepto/keys16:58
clarkbSide note: https://review.opendev.org/c/opendev/system-config/+/831400 shows a list of merge conflicts. I think that merge conflict checking must be working :)17:01
clarkbinfra-root I'm hoping to get to deleting the subunit worker nodes and openstack health at some point this week. I think they were all pretty stateless servers but could do snapshots for them anyway. (the real data is in the databse which I'm not sure what the best way to handle that is given its size)17:05
fungiwe can snapshot the trove instance before we delete it17:08
fungii've done that with others17:08
opendevreviewMerged openstack/project-config master: Add Resource Usage Dashboard  https://review.opendev.org/c/openstack/project-config/+/83137417:08
yoctozeptoor could someone else let me in on https://zuul.opendev.org/t/openstack/autohold/0000000031 ? :-)17:08
clarkbfungi: this is our biggest instance by far though. But maybe that is fine17:08
clarkbyoctozepto: ya give me a sec17:09
yoctozeptoclarkb: :-)17:09
clarkbyoctozepto: root@
yoctozeptoclarkb: thanks17:17
*** jpena is now known as jpena|off17:38
clarkbThere is one node in the airship cloud that seems to be failing to delete. I suspect we may need to perform manual intervention on that one.17:54
clarkbcorvus: ^ I don't think we've had to do that with the zk situation (or at least I don't recall doing it). I assume it is as simple as deleting the node records directly?17:56
clarkbI'll approve the full removal change once the in-use node is gone under that assumption17:56
corvusclarkb: if the launcher is unable to delete the node by removal time, i think there is a nodepool command to delete the zk node directly17:59
clarkboh neat /me looks17:59
clarkbcorvus: looks like `nodepool erase $provider` thanks!18:00
clarkbok we are down to the one deleting node. I'm going to approve the second project-config chagne now18:13
clarkbthen we can run the nodepool erase command since I believe it will be sad about that deleting node18:14
opendevreviewMerged openstack/project-config master: Remove airship-citycloud nodepool provider  https://review.opendev.org/c/openstack/project-config/+/83139818:23
clarkbok I need some food, then meeting and can followup on ^ and erase the provider from zk if necessary. Thank you for all the help with reviews18:26
*** rlandy|ruck is now known as rlandy|ruck|biab18:35
*** rlandy|ruck|biab is now known as rlandy|ruck18:53
yoctozeptoI powered off the node held for https://zuul.opendev.org/t/openstack/autohold/000000003119:08
yoctozeptothank you19:08
clarkbyoctozepto: we're good to delete it then?19:08
yoctozeptoclarkb: yes19:08
yoctozeptoclarkb: and disable the autohold I presume19:09
clarkbyoctozepto: it auto disables. I'll take care of that after our meeting19:10
yoctozeptoclarkb: ack19:10
opendevreviewMerged opendev/system-config master: encrypt-logs: turn on for all prod playbooks  https://review.opendev.org/c/opendev/system-config/+/83078419:37
clarkbinfra-root https://review.opendev.org/c/opendev/system-config/+/831400 is ready to go now. I ran nodepool erase airship-kna1 and the deleting node no longer shows up in listings19:37
clarkbI've just realized that we should keep the clouds.yaml on bridge for now to do any other additional cleanup that may be necessary19:39
clarkblet me respin really quickly19:39
opendevreviewClark Boylan proposed opendev/system-config master: Remove airship-citycloud resources  https://review.opendev.org/c/opendev/system-config/+/83140019:40
opendevreviewClark Boylan proposed opendev/system-config master: Remove airship-kna1 from bridge clouds.yaml  https://review.opendev.org/c/opendev/system-config/+/83144119:55
opendevreviewClark Boylan proposed openstack/project-config master: Remove airship-kna1 grafana dashboard  https://review.opendev.org/c/openstack/project-config/+/83144219:57
clarkbI've WIP'd those last two changes to ensure they don't land early. but 831400 should be reviewable and mergable. Please do that if you have time :)19:59
* clarkb grabs lunch in the meantime20:01
fricklerlooking at https://review.opendev.org/c/opendev/system-config/+/831400/2/inventory/service/host_vars/bridge.openstack.org.yaml , gerrit shows a short popup that files with lines > 500 chars will not be syntax highlighted. no bug, just a funny anecdote maybe20:03
frickler(it's the long rsa ssh keys if anyone wonders and doesn't want to check themselves)20:04
opendevreviewDr. Jens Harbott proposed zuul/zuul-jobs master: Fix ensure-pip test on Debian Buster  https://review.opendev.org/c/zuul/zuul-jobs/+/83144320:13
*** dtantsur_ is now known as dtantsur20:24
clarkbI've approved the system-config change to remove the airship citycloud resources20:59
clarkbI'll monitor that20:59
opendevreviewTristan Cacqueray proposed zuul/zuul-jobs master: DNM: generate ansi log for benchmark purpose  https://review.opendev.org/c/zuul/zuul-jobs/+/76305421:07
clarkbThe airship-kna1 mirror removal change should land soon. I'll look into delting the mirror after the school run22:01
opendevreviewMerged opendev/system-config master: Remove airship-citycloud resources  https://review.opendev.org/c/opendev/system-config/+/83140022:02
clarkbthe kex handler fixup for gerrit merged22:11
clarkbfrickler: ^ you may want to retest gerrit master soon just to be sure it works as expected for you (I tried testing it too and it seemed to work from here)22:11
fungiooh! that'll be nice. is that slated for 3.5.x then, or not until 3.6?22:12
Clark[m]3.6 as that is what got the updated MINA22:18
Clark[m]We could possibly vendor the mina changes in Gerrit on 3.5 but that seems like a lot of effort22:19
fungiyeah, too bad. for some reason i thought they had targeted the updated mina at a 3.5 patch release22:31
clarkbinfra-root: 1ffefdcc-1a2e-4e22-aaba-2c9c5608c924 | mirror01.kna1.airship-citycloud.opendev.org <- I'll delete this server if that looks correct to you22:59
ianw++; probably has a volume attached?22:59
clarkbno looks like it uses a 250GB flavor type instead23:00
clarkbI checked a volume listing and it was empty23:00
ianwahh that could be right too :)23:01
clarkbok it is gone23:03
clarkb#status log Deleted mirror01.kna1.airship-citycloud.opendev.org to finish up resource cleanup in that cloud provider now that it has been removed from Nodepool.23:03
opendevstatusclarkb: finished logging23:03
fungithanks clarkb!23:10
clarkbykarel isn't here anymore but seems https://grafana.opendev.org/d/94891e7b01/resource-usage-by-tenants-and-projects?orgId=1 is working23:10
opendevreviewMerged opendev/zone-opendev.org master: Remove the airship citycloud mirror dns records  https://review.opendev.org/c/opendev/zone-opendev.org/+/83140523:13
*** clarkb is now known as Guest98523:32
*** Guest985 is now known as clarkb23:41
jentoioquestion on the zull-registry container maintence task. I noticed that a zuul userid:groupid is already there in system-config/inventory/service/group_vars/zuul.yaml 23:54
jentoioshould I use this one of create a new one? I'm not sure how group_vars are inherited in playbooks yet23:54
clarkbjentoio: we should probably try to reuse that definition. Lets see if the registry is in that group though23:58
clarkbjentoio: https://opendev.org/opendev/system-config/src/branch/master/inventory/service/groups.yaml that file defines all of our groups based on hostname matching. In this case insecure-ci-registry is the zuul-registry host and it is part of the registry group and not the zuul group23:59
clarkbjentoio: in that case I think you can just update this group instead to contain the same content: https://opendev.org/opendev/system-config/src/branch/master/inventory/service/group_vars/registry.yaml23:59

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!