| cloudnull | hey all, just reading scrollback. Are the API issues at RAX all on legacy OSPC? | 00:01 |
|---|---|---|
| cloudnull | anything we can help with ? | 00:03 |
| clarkb | cloudnull: yes! I'm drafting an email that summarizes if you want to wait a few more minutes | 00:05 |
| cloudnull | ++ | 00:05 |
| clarkb | cloudnull: sent you should see it momentarily | 00:08 |
| cloudnull | 👍 ack | 00:09 |
| clarkb | cloudnull: and I'm happy to followup here once you have a chance to digest | 00:12 |
| * cloudnull replied. | 00:20 | |
| cloudnull | tldr IDK what's happening in OSPC, but folks are looking | 00:20 |
| * cloudnull looking at what we can do with your flex quota now | 00:21 | |
| clarkb | cloudnull: thanks! re the nova keyerror thing I'd have to hand you over to melwitt and sean-k-mooney (sean isn't in here but is in #openstack-infra) | 00:22 |
| cloudnull | o/ melwitt | 00:24 |
| melwitt | hey, was just reading the backscroll. do we have a log to look at with the keyerror in it? | 00:25 |
| cloudnull | clarkb SJC and DFW quotas have increased. | 00:27 |
| cloudnull | * SJC - instances 100, memory 512, FIP 60 | 00:27 |
| cloudnull | * DFW - instances 100, memory 768, FIP 75 | 00:27 |
| clarkb | cloudnull: melwitt in sjc3 today 75355359-40d7-4005-aae4-13de2efa0c0b was one of the instances I manually deleted to clear out the listing error | 00:29 |
| clarkb | I don't know if its only a subset that had the keyerror or not. I can record all of the uuids that were in an error state when I did my deletes | 00:29 |
| clarkb | cloudnull:l re quota thanks! | 00:29 |
| clarkb | I think the zuul launcher should autoamtically use the extra room | 00:30 |
| cloudnull | melwitt here's what it looks like server side https://gisty.link/603eee0e3c31d9a0dfd23e8fce753dc4b7adca85/raw | 00:32 |
| melwitt | thanks | 00:33 |
| melwitt | hrm, looks like this might have caused a regression https://review.opendev.org/c/openstack/nova/+/939658 | 00:34 |
| cloudnull | any idea on how we might mitigate that ? | 00:37 |
| cloudnull | BTW - we're still on 2024.1 | 00:37 |
| cloudnull | updates to 2025.1 planned - but not done yet. | 00:37 |
| melwitt | hm ok, it's only been backported back to 2024.2 so far and 2024.1 hasn't merged yet https://review.opendev.org/c/openstack/nova/+/955305 | 00:39 |
| cloudnull | clarkb once you all are logged into IAD and ready to put work there, let me know and I'll get that quota adjusted too. | 00:39 |
| melwitt | oh lol .. this is actually a fix for the KeyError, not the other way around | 00:39 |
| * melwitt can't read | 00:40 | |
| cloudnull | oh, cool. so fixed in 2025.1 - potential backport for 2024.1 | 00:40 |
| melwitt | yes, 2024.1 backport is approved but hasn't made it through the gate yet | 00:40 |
| melwitt | looks like the root cause is when a server in the returned server list does not have a matching record in the nova_api.request_specs database table | 00:42 |
| cloudnull | ++ added a comment to the review. hopefully we can appease the gate-keeper | 00:43 |
| melwitt | afaict the only way to get around this for now is to delete the server the KeyError is being raised for | 00:43 |
| cloudnull | ++ that's what we've been doing, when it happens. | 00:44 |
| cloudnull | its been a rare occurrence but good to know the fix is already in | 00:45 |
| clarkb | cloudnull: yup, not sure how quickly I'll get to that as we have to boot a new mirror node first, but will try to get to it sooner than later | 00:47 |
| melwitt | fingers crossed for the gate | 00:47 |
| opendevreview | James E. Blair proposed opendev/zuul-providers master: Set rax-flex instance limits https://review.opendev.org/c/opendev/zuul-providers/+/957478 | 01:07 |
| corvus | clarkb: cloudnull ^ we don't detect fip limits in zuul (yet), so that will set an instance limit equal to the fip limits you specified, since that's our effective upper bound on instances. | 01:08 |
| clarkb | corvus: thanks I've approved it | 01:10 |
| opendevreview | Merged opendev/zuul-providers master: Set rax-flex instance limits https://review.opendev.org/c/opendev/zuul-providers/+/957478 | 01:10 |
| opendevreview | Merged openstack/project-config master: Retire Monasca project https://review.opendev.org/c/openstack/project-config/+/957063 | 01:18 |
| corvus | i deleted 3 nodes leaked by the old nodepool system | 01:25 |
| corvus | remote: https://review.opendev.org/c/zuul/zuul/+/957479 Replace getQuotaUsed with cache [NEW] | 02:01 |
| corvus | remote: https://review.opendev.org/c/zuul/zuul/+/957480 Only log quota messages when acted upon [NEW] | 02:01 |
| corvus | two more improvements that can wait for tomorrow | 02:01 |
| opendevreview | OpenStack Proposal Bot proposed openstack/project-config master: Normalize projects.yaml https://review.opendev.org/c/openstack/project-config/+/957483 | 02:58 |
| *** mrunge_ is now known as mrunge | 05:41 | |
Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!