| opendevreview | Goutham Pacha Ravi proposed openstack/nova master: DNM: Add virtiofs share attachment CI jobs https://review.opendev.org/c/openstack/nova/+/999772 | 03:12 |
|---|---|---|
| opendevreview | Goutham Pacha Ravi proposed openstack/nova master: DNM: Add virtiofs share attachment CI jobs https://review.opendev.org/c/openstack/nova/+/999772 | 03:12 |
| opendevreview | Takashi Kajinami proposed openstack/nova master: Stub kernel support check of AMD-SEV in tests https://review.opendev.org/c/openstack/nova/+/999780 | 04:15 |
| opendevreview | Dr. Jens Harbott proposed openstack/nova master: DNM: Test unit tests on kvm https://review.opendev.org/c/openstack/nova/+/999729 | 04:16 |
| opendevreview | Takashi Kajinami proposed openstack/nova master: DNM: Test unit tests on kvm https://review.opendev.org/c/openstack/nova/+/999729 | 04:26 |
| opendevreview | Goutham Pacha Ravi proposed openstack/nova master: DNM: Add virtiofs share attachment CI jobs https://review.opendev.org/c/openstack/nova/+/999772 | 04:55 |
| opendevreview | Goutham Pacha Ravi proposed openstack/nova master: DNM: Add virtiofs share attachment CI jobs https://review.opendev.org/c/openstack/nova/+/999772 | 04:56 |
| opendevreview | Takashi Kajinami proposed openstack/nova master: Stub kernel support check of AMD-SEV in tests https://review.opendev.org/c/openstack/nova/+/999780 | 06:18 |
| opendevreview | Takashi Kajinami proposed openstack/nova master: DNM: Test unit tests on kvm https://review.opendev.org/c/openstack/nova/+/999729 | 06:18 |
| opendevreview | Takashi Kajinami proposed openstack/nova master: Stub kernel support check of AMD-SEV in tests https://review.opendev.org/c/openstack/nova/+/999780 | 07:26 |
| opendevreview | Takashi Kajinami proposed openstack/nova master: DNM: Test unit tests on kvm https://review.opendev.org/c/openstack/nova/+/999729 | 07:26 |
| gibi | gmaan: re threadpool, not yet, I will try to get to it today but there are higher prio things in front of it at the moment | 07:32 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: DNM: check tempest for instability https://review.opendev.org/c/openstack/nova/+/999378 | 07:40 |
| bauzas | gibi: may I gently ask you for a short +W ? https://review.opendev.org/c/openstack/nova/+/999607 | 07:43 |
| gibi | bauzas: done. Have you checked if a failing post copy live migration in this case leaves the instance on the source node in ACTIVE state and we leak nothing on the dest node? | 07:44 |
| bauzas | gibi: at least I saw that by the tempest whitebox test, the instance is back to the source | 07:45 |
| bauzas | https://review.opendev.org/c/openstack/whitebox-tempest-plugin/+/939498 | 07:45 |
| gibi | OK, thanks | 07:46 |
| bauzas | but I'll verify whether the instance is back ACTIVE for sure | 07:46 |
| opendevreview | Kamil Sambor proposed openstack/nova master: libvirt: retry get_guest() on InstanceNotFound during post-copy VIF reattach https://review.opendev.org/c/openstack/nova/+/997380 | 07:51 |
| mattcrees | Good morning! Would anyone be free to give me another review of my bugfix? https://review.opendev.org/c/openstack/nova/+/982660 | 08:07 |
| opendevreview | Takashi Kajinami proposed openstack/nova master: Stub kernel support check of AMD-SEV in tests https://review.opendev.org/c/openstack/nova/+/999780 | 09:07 |
| opendevreview | Takashi Kajinami proposed openstack/nova master: DNM: Test unit tests on kvm https://review.opendev.org/c/openstack/nova/+/999729 | 09:07 |
| opendevreview | Takashi Kajinami proposed openstack/nova master: Stub kernel support check of AMD-SEV in tests https://review.opendev.org/c/openstack/nova/+/999780 | 09:33 |
| opendevreview | Takashi Kajinami proposed openstack/nova master: DNM: Test unit tests on kvm https://review.opendev.org/c/openstack/nova/+/999729 | 09:33 |
| tkajinam | hmm I was able to fix unit tests but am struggling to fix functional tests. I don't know why that patching of update_provider_tree is ignored ... | 09:34 |
| opendevreview | Merged openstack/nova master: doc: Document postcopy incompatibility with vGPU live migration https://review.opendev.org/c/openstack/nova/+/999607 | 09:40 |
| opendevreview | Merged openstack/osc-placement master: tox: Fix outdated functional test environments https://review.opendev.org/c/openstack/osc-placement/+/998825 | 11:06 |
| opendevreview | Elod Illes proposed openstack/nova stable/2025.1: [stable-only] Set skip-level grenade as voting https://review.opendev.org/c/openstack/nova/+/999742 | 11:08 |
| opendevreview | Merged openstack/osc-placement master: [tox] Use constraints for requirements in docs job https://review.opendev.org/c/openstack/osc-placement/+/998816 | 11:14 |
| opendevreview | Merged openstack/osc-placement master: reno: Update master for unmaintained/2024.1 https://review.opendev.org/c/openstack/osc-placement/+/965495 | 11:16 |
| opendevreview | Merged openstack/osc-placement master: Exclude tests directory from coverage calculation https://review.opendev.org/c/openstack/osc-placement/+/906665 | 11:17 |
| opendevreview | Elod Illes proposed openstack/osc-placement stable/2026.1: [tox] Use constraints for requirements in docs job https://review.opendev.org/c/openstack/osc-placement/+/999813 | 12:19 |
| opendevreview | Elod Illes proposed openstack/osc-placement stable/2025.2: [tox] Use constraints for requirements in docs job https://review.opendev.org/c/openstack/osc-placement/+/999814 | 12:21 |
| opendevreview | Elod Illes proposed openstack/osc-placement stable/2025.1: [tox] Use constraints for requirements in docs job https://review.opendev.org/c/openstack/osc-placement/+/999815 | 12:23 |
| jangutter[m] | Hi folks, we ran into an issue in one of our OpenStack clusters where the autoincrement ID in the nova.instance_system_metadata table ran into the max for the column size https://opendev.org/openstack/nova/src/branch/master/nova/db/main/models.py#L993 | 12:42 |
| jangutter[m] | For now, we’re promoting that column to BigInteger, but I was wondering if there’s a process for reclaiming old ID’s and resetting the autoincrement? Our combination of massive amounts of instance properties plus churn in instance launches triggered this, but I was wondering if there’s a periodic maintenance step we’re missing. | 12:42 |
| sean-k-mooney | jangutter[m]: your not the first | 12:42 |
| sean-k-mooney | so we didnt do that to avoid the upgrade impact but we have considered doing that in the past | 12:43 |
| sean-k-mooney | everntully we will need to do the same upstream | 12:43 |
| sean-k-mooney | what we did for our old internal cloud | 12:43 |
| jangutter[m] | [sean-k-mooney](https://matrix.to/#/@sean-k-mooney:matrix.org) I was literally calculating how soon folks like CERN or telcos would run into this! | 12:43 |
| sean-k-mooney | was manually reindex the records in teh table | 12:43 |
| sean-k-mooney | ya so our internal cloud hit this about 2-3 years ago | 12:44 |
| jangutter[m] | We took down our DB and re-imported (since we could take the downtime) | 12:44 |
| sean-k-mooney | melwitt had the pleasure of helping them fix it | 12:44 |
| sean-k-mooney | i think we did somethign simialr we shuffled the row ids back to the start of the reange fillign the gaps and then reset the increment point | 12:45 |
| jangutter[m] | But yeah, migrating the primary index is not to be taken lightly. | 12:45 |
| sean-k-mooney | the way we woudl do that upstream woudl be to add a new column and index then later hava a contract migration to drop the old integer column | 12:45 |
| sean-k-mooney | so you woudl do the expand schema migration ot add the new colume then upgrade nova then we woudl drop the old primary index column | 12:46 |
| sean-k-mooney | jangutter[m]: i dont know how shareable or robust the scripting we did internaly was but i could see adding a nova-manage command to do this if it was safe to do so | 12:47 |
| jangutter[m] | You’d still run out of indexes, but hopefully other things would bother you first (like the rise of the Zombie people in 2045) | 12:48 |
| sean-k-mooney | but ya the instance system metadata table is the first to hit it because there are many entreis per vm | 12:48 |
| jangutter[m] | Thanks for confirming! | 12:49 |
| sean-k-mooney | im just checking if i have any refence ot the downstream issue its been a while | 12:50 |
| jangutter[m] | Yeah, strangely I didn’t find any launchpad issues | 12:50 |
| sean-k-mooney | we discused it in the ptg a few years ago but i dont know if we created a bug | 12:51 |
| sean-k-mooney | https://redhat.atlassian.net/browse/OSPRH-4216 | 12:52 |
| sean-k-mooney | ok so that was orgially imported form bugzill 2 jira instances ago | 12:53 |
| sean-k-mooney | https://bugzilla.redhat.com/show_bug.cgi?id=2164247 | 12:54 |
| sean-k-mooney | https://etherpad.opendev.org/p/nova-bobcat-ptg#L431 | 12:56 |
| sean-k-mooney | jangutter[m]: so it looks like we agreed new PK will be big ints and to ask operators what there prefence woudl be for upgrades and if they woudl prefer reindexing | 12:57 |
| sean-k-mooney | jangutter[m]: so if you have a prefence let us know :) | 12:57 |
| jangutter[m] | We’re planning our Epoxy migrations now (and looks like most of them will be offline), so we might not be a good example. | 12:58 |
| sean-k-mooney | i think most operator would just like sometihng that works :) | 13:00 |
| sean-k-mooney | where they dont have to think to hard about it | 13:00 |
| sean-k-mooney | ultimatly we need to change all our primary keys to bigints eventually | 13:01 |
| jangutter[m] | Yeah, and the fixes are different whether you’re having an outage or not, of course. | 13:01 |
| jangutter[m] | At least this isn’t as bad as the Y2038 problem. If you’ve been running a cluster long enough for this to affect you, chances are you know a bit about the internals of OpenStack. | 13:04 |
| sean-k-mooney | ya the internal culsterwas running since like osp7 and upgraded along the way so 6-8 years of primarlly a dev/ci workload with many many short lived vms | 13:08 |
| sean-k-mooney | anyeay if melwitt rememebr andy more or still has the script somewhere perhaps she can comment on how feasible it woudl be to have a nova-manage command to "heal" it when needed but that a workaround not a real fix | 13:09 |
| sean-k-mooney | jangutter[m]: have ye fixed this for your current cloud | 13:10 |
| sean-k-mooney | you mentioned you just promoted the column right | 13:10 |
| sean-k-mooney | i dont know that we ever join on that primary key so for that specici table droping that column might also be an option | 13:11 |
| sean-k-mooney | the unique constraitn is the instance id or uuid and the metadata key i think and we alwasy join on the instance id | 13:12 |
| sean-k-mooney | ya its on the instanfce uuid we actuly do not have a proper uniwue constrating on that table... | 13:14 |
| sean-k-mooney | https://github.com/openstack/nova/blob/master/nova/db/main/models.py#L987-L1004 | 13:14 |
| sean-k-mooney | i wonder if that would be the better fix in this case add a uniqeconstrat on instnace_uuid + key colum and drop id entirly | 13:15 |
| sean-k-mooney | the instance uuid is effectivly the real "primary key" based on our actul usage which is why its already got its own index | 13:16 |
| sean-k-mooney | ah this has the soft delete mixin | 13:21 |
| sean-k-mooney | so ya we cant actully change the colume name like that or delete the id colume and use a compistie primay key | 13:22 |
| sean-k-mooney | so the only options are reindex or promot to BigInt | 13:22 |
| sean-k-mooney | jangutter[m]: you will need to promote the deleted colume to the same type because of softdelete fyi | 13:25 |
| sean-k-mooney | otherwase you wont be abel to delete if the value exceeds int max | 13:25 |
| jangutter[m] | oh yes, we saw you need to do it in pairs. | 13:38 |
| jangutter[m] | We managed to speedrun it in 2-3 years with a combination of ~50 properties per instance and a tight create/delete loop. | 13:40 |
| sean-k-mooney | yep that will do it | 13:41 |
| sean-k-mooney | we did the reindex by sufflleing all the rows down to avoid needing to mess with the schema | 13:41 |
| sean-k-mooney | as noted we dont actully use the id values | 13:43 |
| sean-k-mooney | so the hack is basiclly | 13:43 |
| sean-k-mooney | SET @num := 0; | 13:43 |
| sean-k-mooney | UPDATE instance_system_metadata SET id = @num := (@num+1); | 13:43 |
| sean-k-mooney | ALTER TABLE instance_system_metadata AUTO_INCREMENT =1; | 13:43 |
| sean-k-mooney | i say basiclly becase if the deleted colum is non 0 you have to set delete=id as well | 13:44 |
| sean-k-mooney | to keep the soft delete state correct | 13:44 |
| sean-k-mooney | im sure you can express that as a single query but i think we didi it in python | 13:45 |
| jangutter[m] | That would be a handy hack to have in nova-manage. | 13:45 |
| jangutter[m] | but you have to be sure the id isn’t referenced elsewhere. | 13:46 |
| sean-k-mooney | ya and there are other tabels that we can or coudl do the same for | 13:51 |
| sean-k-mooney | basiclly a `nova-manage db compact` command | 13:52 |
| sean-k-mooney | i think to do that properly you need to do it with a full table lock and batch it to now block operations | 13:52 |
| sean-k-mooney | or do it offline but but if you do it 1000 rows at a time ectra you could do it online | 13:53 |
| MaxLamprecht[m] | Hey folks, I wanted to ask regarding some attachment fixes I proposed a while ago(we merged 1 out of 3 fixes a while ago). https://review.opendev.org/c/openstack/nova/+/974832 and https://review.opendev.org/c/openstack/nova/+/972045 are still open with valid reproducers/regression tests. Is there something I can do to get more attention on this topic? | 14:13 |
| opendevreview | minwoo seo proposed openstack/nova master: libvirt: Add volume migration bandwidth option to nova.conf https://review.opendev.org/c/openstack/nova/+/999841 | 14:24 |
| tkajinam | clif, hj could you try https://review.opendev.org/c/openstack/nova/+/999780 when you have time ? | 14:39 |
| tkajinam | there are a few functional tests still failing but CI shows at least unit tests are passing | 14:39 |
| clif | tkajinam: passes for me: py3: OK (113.29=setup[2.17]+cmd[99.77,7.69,3.66] seconds) | 14:42 |
| tkajinam | clif, that's good. could you also run functional tests there, too ? | 14:42 |
| clif | sure | 14:43 |
| tkajinam | I expect two reshape tests may fail | 14:43 |
| clif | tkajinam: I get two failures in functional: nova.tests.functional.libvirt.test_reshape.SevResphapeTests.test_create_servers_with_amd_sev and nova.tests.functional.libvirt.test_reshape.SevResphapeTests.test_create_servers_with_amd_sev_mixed | 14:49 |
| tkajinam | yeah | 14:49 |
| tkajinam | it's wired that update_available_resource is called there though it's mocked out | 14:49 |
| opendevreview | Clif Houck proposed openstack/nova master: perf(ironic): eliminate O(N²) ProviderTree deepcopy at startup (B6) https://review.opendev.org/c/openstack/nova/+/980676 | 15:03 |
| melwitt | MaxLamprecht[m]: channel ping is good, both have green CI, i will try to look soon | 15:06 |
| Uggla | Reminder upstream bug triage in ~20mn. | 15:08 |
| lajoskatona | sean-k-mooney: Hi, there is a patch ( https://review.opendev.org/c/openstack/nova/+/982085) for https://bugs.launchpad.net/nova/+bug/2051685 (After repeat of incomplete migration nova applies wrong (status=error) migration context in update_available_resource periodic job) | 15:12 |
| lajoskatona | sean-k-mooney: and a reproducer functional test under the commit. As I remember we had a short chat about it, if you have time can you please check it? Bence / rubasov started to work on it but He was moved to another project so I am kind of responsible for it now :) | 15:12 |
| opendevreview | Clif Houck proposed openstack/nova master: Parallelize per-node resource updates https://review.opendev.org/c/openstack/nova/+/980679 | 15:15 |
| opendevreview | minwoo seo proposed openstack/nova master: libvirt: Add volume migration bandwidth option to nova.conf https://review.opendev.org/c/openstack/nova/+/999841 | 15:28 |
| Uggla | Link reminder: meet.google.com/zjr-rxus-hzj (upstream triage) | 15:33 |
| opendevreview | Takashi Kajinami proposed openstack/nova master: Stub kernel support check of AMD-SEV in tests https://review.opendev.org/c/openstack/nova/+/999780 | 15:40 |
| opendevreview | Takashi Kajinami proposed openstack/nova master: DNM: Test unit tests on kvm https://review.opendev.org/c/openstack/nova/+/999729 | 15:40 |
| *** Unknown123 is now known as Mike-- | 16:28 | |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Add reproducer for bug/2162912 https://review.opendev.org/c/openstack/nova/+/999877 | 16:37 |
| tkajinam | sean-k-mooney dansmith https://review.opendev.org/c/openstack/nova/+/999780 fixes the unit/func tests in amd machine | 17:06 |
| tkajinam | verified in ci by https://review.opendev.org/c/openstack/nova/+/999729 | 17:07 |
| tkajinam | just fyi | 17:07 |
| tkajinam | (but appreciate your reviews, if you have time | 17:07 |
| sean-k-mooney | i looked breifly this morningin the only thing i was debtin is if you shoudl pass in the var when constucting the fixture as you are doign or just have the class var defiend in the fixture and use it direcly | 17:08 |
| sean-k-mooney | that would just avoid proxing it here https://review.opendev.org/c/openstack/nova/+/999780/6/nova/tests/functional/libvirt/base.py | 17:09 |
| sean-k-mooney | but i see why you did it this way | 17:19 |
| sean-k-mooney | +2 form me | 17:22 |
| sean-k-mooney | gmaan: https://review.opendev.org/c/openstack/nova/+/980679 this is the patch that i mentioned today related to usign an executor to Parallelize per-node resource updates | 17:46 |
| sean-k-mooney | its workign the exact same as the _sync_power_executor https://review.opendev.org/c/openstack/nova/+/980679/5/nova/compute/manager.py#669 | 17:46 |
| -opendevstatus- NOTICE: The Gerrit service on review.opendev.org will be offline momentarily at 20:00 UTC (an hour from now) while we restart for a patch upgrade, but should return within a few minutes. | 19:03 | |
| gmaan | sean-k-mooney: ack | 19:06 |
| -opendevstatus- NOTICE: The Gerrit service on review.opendev.org will not be restarted at 20:00 UTC after all, due to unanticipated image build regressions, and will be rescheduled to a later time/date. | 19:14 | |
| opendevreview | Bryan Fraschetti proposed openstack/nova master: Feat: Add Command for Forceful BDM Deletion https://review.opendev.org/c/openstack/nova/+/985717 | 19:20 |
| opendevreview | Bryan Fraschetti proposed openstack/nova master: add task state for attach volume and detach volume https://review.opendev.org/c/openstack/nova/+/985718 | 19:20 |
| opendevreview | Bryan Fraschetti proposed openstack/nova master: Reject to attach and detach volume while task_state is not None https://review.opendev.org/c/openstack/nova/+/758918 | 19:20 |
| opendevreview | Bryan Fraschetti proposed openstack/nova master: Reject Volume Attachment if task_state not Attaching https://review.opendev.org/c/openstack/nova/+/985719 | 19:20 |
| opendevreview | Merged openstack/placement stable/2026.1: Reproduce bug 2160721 https://review.opendev.org/c/openstack/placement/+/998873 | 20:07 |
| opendevreview | Merged openstack/placement stable/2026.1: Check overall capacity before generating candidates https://review.opendev.org/c/openstack/placement/+/998874 | 20:07 |
Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!