Tuesday, 2023-09-12

opendevreviewGoutham Pacha Ravi proposed openstack/project-config master: Add manila-core to osc/sdk repo config  https://review.opendev.org/c/openstack/project-config/+/89460500:25
opendevreviewGoutham Pacha Ravi proposed openstack/project-config master: Add manila-core to osc/sdk repo config  https://review.opendev.org/c/openstack/project-config/+/89460500:26
opendevreviewBernhard Berg proposed zuul/zuul-jobs master: prepare-workspace-git: Add ability to define synced pojects  https://review.opendev.org/c/zuul/zuul-jobs/+/88791708:47
*** amoralej is now known as amoralej|lunch12:16
opendevreviewMerged openstack/project-config master: Add manila-core to osc/sdk repo config  https://review.opendev.org/c/openstack/project-config/+/89460512:55
*** amoralej|lunch is now known as amoralej12:58
opendevreviewdaniel.pawlik proposed zuul/zuul-jobs master: Add feature to set --vm-driver name for minikube  https://review.opendev.org/c/zuul/zuul-jobs/+/89475513:35
opendevreviewdaniel.pawlik proposed zuul/zuul-jobs master: Add feature to set --vm-driver name for minikube  https://review.opendev.org/c/zuul/zuul-jobs/+/89475513:38
opendevreviewdaniel.pawlik proposed zuul/zuul-jobs master: Add feature to set --vm-driver name for minikube  https://review.opendev.org/c/zuul/zuul-jobs/+/89475514:18
opendevreviewdaniel.pawlik proposed zuul/zuul-jobs master: Add feature to set --vm-driver name for minikube  https://review.opendev.org/c/zuul/zuul-jobs/+/89475514:23
clarkbfrickler: any luck with the inmotion image(s) and possibly leaked instances keeping the images around?14:46
fricklerclarkb: not yet, I just noticed that there are even more images in glance with status "queued" than pending delete in nodepool, but didn't check in more detail so far14:49
clarkbI think those may be leaks that we can delete as normal users. I can give that a try after breakfast14:50
clarkbthe queued ones specifically14:50
fricklerone general note, victoria is pretty old, is there any plan/provisions to update that cluster? or would that imply a redeployment (which might be even easier when we want to go to something recent)?14:52
fricklermaybe we could skip the cleanup that way, too14:53
clarkbfrickler: no hard plans but yes it should be updated. The path forward there is to redeploy14:54
clarkbfrickler: yuriys isn't in this channel right now but in vancouver said that our deployment was one of the first and we helped them work through a lot of the initial problems and learning. He thought it would be easier to redeploy at this point if we go that route14:55
clarkbI think for planning a redeployment we mostly need to determine if we can deploy a new cloud alongside the old one or if we need to stop the current one and replace it (for quota/resource reasons)14:56
clarkbthen make note of the general architecture. I think it is 3 converged control nodes + 4 compute nodes of varying size and we run instances on provider networking to avoid losing IPs to router interfaces14:56
clarkbone of the main things we worked with them on was ssl+certs (I think we are doing self signed so we have an explicitly trusted cert in our clouds.yaml config). I expect that may have improved sicne we did the deployment though14:58
fricklerceph seems to be running 1 nvme osd on each of the hosts14:58
fricklerI think after the openstack release we should be fine with dropping that capacity for a couple of days if needed15:00
clarkbagreed, there tends to be a lull in activity after openstack makes its release15:00
* clarkb scribbles a note to contact yuriys for any other inputs15:01
clarkbfungi: ^ you may be interested too. yuriys shares a timezone with you15:02
fricklerI'll try to look at the kolla configuration to see if I notice anything else noteworthy. but I think I'd prefer to focus on redeployment then and ignore the image cleanup until then15:02
fricklerceph seems to be at ~35% usage, so no lack of capacity there15:03
clarkbfrickler: our main limitation was cpu and memory iirc15:04
clarkbactually no just cpu15:04
clarkbwe tried overcommitting and it just didn't work for the CI workloads and so ended up on a 1:1 or close to 1:1 cpu ratio for vms15:04
jrosseris this for aarch64 capacity?15:09
clarkbjrosser: no that cloud is x86_6415:10
fricklermaximum 51 nodes, so roughly 10% of our total capacity15:15
fungiwill a redeploy preserve our mirror instance, or will we need to recreate it too?15:19
clarkbcorvus: looking at the list of github repos without apps I wonder if some would be willing to provide an install with fewer perms. Do you know what perms are necessary to do the advisory type testing we do? I think we have the app set up to request perms that could potentially be used for gating but we aren't doing that with these repos15:19
clarkbfungi: it will need to be redeployed too15:19
clarkbspecifically thinking projects like gnocchi and sqlalchemy/dogpile.cache might be open to that as we know their maintainers and can explain the use case15:21
clarkbI suspect it would be a hard sell to get any app installed on ansible or cryptography15:21
clarkbfungi: do you know who to get in touch with in the monasca project about removing the elastic/kibana zuul integratoin?15:23
clarkbthat project is no longer OSI compatible and I'd like to remove it from our zuul config (note I don't think CI use cases are in violation of the lciense, I just don't want to be in a situation where non open source software in our system is making our lives more difficult so it should be removed)15:24
fungiclarkb: i think it's probably a question for the openstack tc. looks like there's not even a ptl candidate for that team15:26
clarkbfungi: ack I'll bring it up there15:26
fricklerthere's a potential candidate https://review.opendev.org/c/openstack/election/+/89303315:26
fricklerjust noone with actual contributions15:26
fungithey have an irc channel but nobody using it other than bot notifications15:26
fricklerthe electorate for that project would have been elodilles and me, but just for cleanup patches15:27
fungiif there are no substantive changes being tested for their repos, then it's pretty easy to make the argument that they're not doing anything with that job integration anyway15:28
frickleryes, so just proposing a patch for cleanup and force-merging with TC approval if needed would sound fine to me15:31
clarkbcorvus: searching hound for pravega/zookeeper-operator shows the only instance of that string is in the zuul tenant config itself. I think that can be removed? Pinging you as I suspect that was in use by zuul operator testing at some point or was intended to be15:33
clarkbfrickler: the github.com/eventlet/eventlet inclusion was added I think for testing with neutron. Do you recall if that was the case? I can't find it in hound (so maybe it is on older branches?)15:35
clarkbbut I think eventlet/eventlet can just be removed too if it isn't actually used anywhere15:35
corvusclarkb: i don't think i have an exhaustive list, but my guess would be just "read" for "repo admin", "repo contents", "issues", and "prs".  maybe "statuses". from https://zuul-ci.org/docs/zuul/latest/drivers/github.html#configure-github15:35
fricklerclarkb: I'm not sure, multiple projects come to my mind that might do testing on eventlet latest15:36
clarkbcorvus: cool maybe we can work with someone like zzzeek to test that15:36
corvusclarkb: yeah, i think we can remove the zk operator15:36
clarkbfrickler: ok git logs can probably point me more directly at the people who started with it at least15:37
clarkbI need to pause for breakfast but I should be able to push changes up for some of this soon15:38
fungiskimming codesearch, i don't see any jobs setting eventlet/eventlet as a required project15:38
clarkbfungi: ya I couldnt' find any examples either15:39
clarkbgit logs say it was swift doing python3 in ~201915:39
fricklergit blame finds https://review.opendev.org/c/openstack/project-config/+/65688915:39
fricklertimburke: ^^ is that still being used somewhere?15:39
clarkblooks like in the related chagne the required-project was used in older patchsets then eventually removed before the change merged15:40
clarkbI suspect we can remove eventlet too15:41
clarkbI can get a change up for zookeeper-operator and eventlet together then another separate one for kibana15:41
fungilooks like there's going to be a python 3.12.0rc3 on monday with some more bug fixes... getting close!16:07
corvusclarkb: i think https://review.opendev.org/794688 and generating a personal access token for the account would help16:22
corvusi just tested my own personal access token and verified that i get the full 5k rate limit using that, and was able to load the opendev configuration as currently written.16:23
corvusso i think we just need a change like that to handle automatic app -> token fallback while still having app configured16:24
corvus(alternatively, we could drop the app config, set a token, and run without that patch today)16:24
fricklerdid something change about github api limits recently? I think I'm seeing more issues downstream, too16:38
fricklerseems we've had a spam message pass on service-discuss: https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/message/JLGRB7TNXJK2W3ELRXMOTAK3NH5TNYI3/16:53
clarkbcorvus: ok looks like there are some things that need addressing in the change. I suspect ianw may not have time for that now. I can try and push on it16:53
clarkbfrickler: I think fungi developed a reaction process to that when it happened to zuul.16:54
corvusclarkb: sounds good; i think they are fairly straightforward16:55
clarkbI still think removing unneeded projects is a good idea, but good to know we can improve this through zuul tool16:55
corvusyeah.  best outcome is we have the 5k ops available to projects that actually use them. and we don't have to have a high bar for adding projects.16:57
opendevreviewClark Boylan proposed openstack/project-config master: Remove two unnused github projects from tenant config  https://review.opendev.org/c/openstack/project-config/+/89481317:02
opendevreviewClark Boylan proposed openstack/project-config master: Remove github.com/elastic/kibana from Zuul config  https://review.opendev.org/c/openstack/project-config/+/89481417:02
clarkbI suspect that first one is good to go. For the second I'm hoping I can make use of the TC meeting to get an ack that this is also going to happen17:02
corvusdid i read that kibana is only used by a dormant project?17:03
clarkbcorvus: yes17:04
clarkbbut apparently part of the raeson it is dormant is that no one can get the CI working. Of course my response to that would be start over on the CI jobs but people seem scared of doing that17:04
clarkbthe number of branches they have makes the github branch viewer sad when I scroll through it too fast17:05
corvusmaybe they thought it was a kpi metric to optimize.  let's get the number of stars and branches super high!17:10
clarkbcorvus: if you remove a project from a pipeline config it shouldn't report zuul config errors via that pipeline's reporter correct?17:20
opendevreviewBernhard Berg proposed zuul/zuul-jobs master: prepare-workspace-git: Add ability to define synced pojects  https://review.opendev.org/c/zuul/zuul-jobs/+/88791717:27
JayFinfra-root: can someone please hold a failing node for ironic-tempest-bios-redfish-pxe on https://review.opendev.org/c/openstack/ironic/+/89446017:33
JayFYou can put the keys from https://github.com/jayofdoom.keys on it when it gets caught17:34
corvusclarkb: yes17:35
JayFlet me know when the hold is in place so I can recheck17:37
clarkbJayF: the hold is in place17:37
clarkblet us know when it fails and we can update ssh keys17:37
JayFclarkb: I think we got a winner, https://github.com/jayofdoom.keys https://zuul.opendev.org/t/openstack/build/2f3f200d942c48ac94bbef27d70cbb2519:08
clarkbJayF: I can add them after the infra team meeting19:08
JayFI am about to lunch so there is no real rush, just wanted to send the message so I wouldn't forget :)19:09
JayFthank you19:09
fungi#status log Deleted a stray spam post from the service-discuss list archives20:26
opendevstatusfungi: finished logging20:26
fungii guess if there's something to document, it's https://gitlab.com/mailman/hyperkitty/-/issues/26120:26
fungilog into the webui with superuser credentials and message/thread deletion is glaringly obvious20:26
clarkbJayF: root@
fungioh, i missed that there's recent progress on the message deletion issue in hyperkitty: https://gitlab.com/mailman/hyperkitty/-/merge_requests/53420:30
clarkbfungi: when you did the deletion you did so as the django superuser then?20:51
clarkbI'm going to pop out for a bike ride but when I return I'm planning to land the zuul tenant config cleanups and draft that email to yuriys20:53
fungiclarkb: yeah, i opened a browser tab in a new "container" so that it wouldn't be logged in as my list owner account, logged into mai21:08
fungilman with the admin account in our private hostvars on bridge, and then there were "delete post" and "delete thread" buttons next to the spam message in hyperkitty21:09
fungihopefully the author of that hyperkitty mr finishes up the work and gets it into an upcoming release21:10
fungibecause ideally the list owners or moderators should have permission to take those actions21:10
fungibut even moreso, i hope this just doesn't come up all that often21:12
clarkbI guess that doesn't sound too bad but ya less than ideal22:53
clarkbanyone want to review https://review.opendev.org/c/openstack/project-config/+/894813/1 and child before I proceed with approving them?22:53
fungiyeah, i can22:56
fungireviewed and approved22:56
clarkbhttps://etherpad.opendev.org/p/VvYeKbEBSpVo0TU2vGqe something like this for the inmotion cloud deployment?22:59
opendevreviewMerged openstack/project-config master: Remove two unnused github projects from tenant config  https://review.opendev.org/c/openstack/project-config/+/89481323:08
opendevreviewMerged openstack/project-config master: Remove github.com/elastic/kibana from Zuul config  https://review.opendev.org/c/openstack/project-config/+/89481423:08
fungiclarkb: draft e-mail lgtm. the only other thing that comes to mind is whether we should wait for kolla-ansible to release its 2023.2 version too and then deploy that23:19
fungiwhich ties into your "what versions are available" question23:20

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!