| opendevreview | Sylvain Bauza proposed openstack/nova master: Add mtty/mdpy support for testing fake mdevs https://review.opendev.org/c/openstack/nova/+/898100 | 08:34 |
|---|---|---|
| opendevreview | Sylvain Bauza proposed openstack/nova master: WIP : Add mtty support to nova-next https://review.opendev.org/c/openstack/nova/+/922140 | 08:34 |
| opendevreview | LiuWenYu proposed openstack/nova master: Implements: Fixed the issue where the code logic would throw an error when reading a variable after NUMA node 0’s CPUs were fully utilized. Related-Bug: #2109675 https://review.opendev.org/c/openstack/nova/+/948540 | 09:38 |
| gibi | sean-k-mooney: the generic env passing logic works nicely here https://review.opendev.org/c/openstack/nova/+/948450/2 with the devstack patch https://review.opendev.org/c/openstack/devstack/+/948436 So now we can selectively turn on threading mode per service from the zuul job config. | 09:47 |
| *** elodille1 is now known as elodilles | 10:11 | |
| elodilles | auniyal bauzas Uggla : hi, sorry for pinging you, but could you take a look at the 2023.2 Bobcat EOL patch of nova please: https://review.opendev.org/c/openstack/releases/+/948196 | 11:06 |
| elodilles | it would be nice to have an ACK from the team o:) | 11:07 |
| elodilles | i've checked and hashes are still all correct | 11:07 |
| sean-k-mooney | gibi: awsome that was also much smaller then i was expecting | 11:34 |
| sean-k-mooney | 'SYSTEMD_ENV_VARS["n-sch"]': OS_NOVA_DISABLE_EVENTLET_PATCHING=true | 11:35 |
| sean-k-mooney | a i see wll thats a little bit of a wart but i guess when your are rendering a local.conf via yaml | 11:36 |
| sean-k-mooney | via ansible | 11:36 |
| sean-k-mooney | that less bad then it could have been | 11:36 |
| sean-k-mooney | so in this version nova-next is the only job using the new code path | 11:37 |
| sean-k-mooney | i guess it make sense to use that as our canary job | 11:37 |
| gibi | yeah | 11:47 |
| gibi | I think I will duplicate nova-multi-cell to have a threading variant eventually as well | 11:48 |
| gibi | but for now nova-next is good enough | 11:48 |
| sean-k-mooney | zigo: i dont know if this is just a ubunut issue or also a debian issue but novnc is declared as requiring phty3-novnc which requires python3-oslo.confing. however as far as i can tell that depency was removed in 0.6.0 when they deleted there vendro copy of nova vnc proxy, novnc does not seam to have any python deps any more and is basiclly pure javascript | 12:48 |
| bauzas | gibi: sean-k-mooney: I had no time in between yesterday and today to look at the scheduler eventlet series, anything you want me to know ? | 12:48 |
| sean-k-mooney | bauzas: it now work well enough to pass ci | 12:49 |
| zigo | sean-k-mooney: I guess you mean 1.6.0, no? | 12:49 |
| sean-k-mooney | zigo: no i mean that dep was remove lik 10 years ago | 12:49 |
| bauzas | sean-k-mooney: scatter gather works fine ? | 12:49 |
| bauzas | what about performance ? | 12:49 |
| zigo | sean-k-mooney: Looks like the Ubuntu package is just the Debian one, so that one is on me indeed ! :) | 12:49 |
| sean-k-mooney | zigo: the last release to use oslo was v0.5.1 | 12:49 |
| zigo | sean-k-mooney: So, python3-novnc should just die, right? | 12:50 |
| sean-k-mooney | zigo: context is we install novnc form disto package by default, and neutron have a oslo tip job that install oslo mater content adn for reason the venv we use in devstack has --site-packages | 12:51 |
| sean-k-mooney | zigo: that also proably reasonable | 12:51 |
| sean-k-mooney | i have no idea where its defiend and what its provideing | 12:51 |
| sean-k-mooney | well i sort of do https://ubuntu.pkgs.org/24.04/ubuntu-universe-amd64/python3-novnc_1.3.0-2_all.deb.html | 12:52 |
| sean-k-mooney | zigo: it looks like its basicaly empty | 12:53 |
| zigo | Yeah, doing the work now. | 12:53 |
| zigo | Hopefully, this will reach Trixie in time for the release. | 12:53 |
| * zigo is supposed to be 2 weeks on holidays in China right now... :P | 12:54 | |
| sean-k-mooney | zigo: oh sorry about that | 12:54 |
| zigo | Well, nothing much to do, I'm at my familly in law place, booooorrrriiiinnnggg ... :) | 12:54 |
| zigo | Oh, btw, got all of Epoxy in Debian 13 already, all ... but Neutron, which should migrate in 8 days. | 12:55 |
| gibi | bauzas: in short the performance seems OK. I will have a blogpost out with more data in couple of hours. At the moment there is nothing to really review for landing as I need to clean up the patches | 12:55 |
| zigo | So, looking good. | 12:55 |
| zigo | Except Eventlet and Python 3.13 ! | 12:55 |
| zigo | I hope we can find a solution. | 12:55 |
| sean-k-mooney | zigo: i was going to ask | 12:55 |
| zigo | Not sure if Herve is still working on that. | 12:55 |
| sean-k-mooney | i tought 3.13 was supper broken | 12:56 |
| zigo | I haven't had a chance to test it yet. | 12:56 |
| sean-k-mooney | well maybe not super broken but the whoel thread id stuff was broken so... | 12:56 |
| zigo | That's on my plate for when I return from holidays. | 12:56 |
| zigo | I need all in Trixie to be able to test though. | 12:56 |
| sean-k-mooney | zigo: so as far as i know eventlet work well enough to pass our unit test and funcitonal | 12:57 |
| zigo | Oh, ok. | 12:57 |
| sean-k-mooney | but fails in reality when mokeypatching due to some changies in the threading module | 12:57 |
| bauzas | gibi: cool, thanks for the info, I'm now myself hardly working on mtty effort | 12:57 |
| zigo | I do unit tests for all packages at build time and autopkgtest, so I know that at least isn't broken. | 12:57 |
| sean-k-mooney | not sure if that is still a the case | 12:57 |
| zigo | What I'm missing is a full setup + tempest run. | 12:57 |
| sean-k-mooney | btu that is where it was at before the ptg | 12:57 |
| zigo | With my Bookworm backports setup, I have 1900+ tempest tests passing. | 12:58 |
| zigo | I'll see how far I can go with Trixie. | 12:58 |
| sean-k-mooney | zigo: ack. that all promising | 12:58 |
| sean-k-mooney | that remind me i need to go update our pre-commit versions for py3.13 | 12:58 |
| sean-k-mooney | zigo:anyway thanks for looking at this while on vacation but please dont spend too much time on it | 13:00 |
| zigo | No worries. | 13:00 |
| zigo | If I put the novnc package somewhere, can you test it? | 13:01 |
| zigo | I did test 1.6.0 already... | 13:01 |
| zigo | Worked, though had to fix some CSS. | 13:02 |
| sean-k-mooney | frickler: ^ would you be able to test it in the test env you had set up | 13:02 |
| frickler | I've now installed novnc from source already, better setup a new host I guess | 13:02 |
| sean-k-mooney | zigo: i could proably install it into a devstack too i just dont have one set up for this at the moment | 13:03 |
| zigo | http://shade.infomaniak.ch/novnc_1.6.0-2_all.deb | 13:03 |
| frickler | (that is a held node, not a local env) | 13:03 |
| zigo | It's ok, I'll just upload like that ... | 13:03 |
| zigo | uploaded | 13:05 |
| zigo | To get the fixed novnc in Ubuntu, you may have to ping some Canonical guys ... jamespage (in here) for example! :) | 13:06 |
| sean-k-mooney | ack, it would be nice to fix in 24.04 | 13:06 |
| zigo | jamespage: Could you get novnc 1:1.6.0-2 pushed to Ubuntu? | 13:06 |
| zigo | sean-k-mooney: Not sure they will push a new upstream release to an already released Ubuntu though, but maybe in their PPA. | 13:07 |
| sean-k-mooney | ya we have a workaround anywya | 13:07 |
| sean-k-mooney | which is just install form source | 13:07 |
| sean-k-mooney | and it only affects jobs | 13:07 |
| sean-k-mooney | that are using unreleased oslo | 13:07 |
| sean-k-mooney | which is a very small minority | 13:07 |
| sean-k-mooney | so we are likey ok if they cant | 13:07 |
| sean-k-mooney | we could also jsut swap to debian :) | 13:08 |
| gibi | bauzas: blogpost with more data https://gibizer.github.io/posts/Eventlet-Removal-Scheduler-First-Run/ | 13:11 |
| bauzas | ta a lot | 13:13 |
| * bauzas has some issues withe the libvirt nodedev for mtty but I'll fix it | 13:13 | |
| opendevreview | LiuWenYu proposed openstack/nova master: Implements: Fixed the issue where the code logic would throw an error when reading a variable after NUMA node 0’s CPUs were fully utilized. Related-Bug: #2109675 https://review.opendev.org/c/openstack/nova/+/948540 | 13:23 |
| zigo | sean-k-mooney: By all means, please get jobs running on Debian 13 and Py 3.13 ! :) | 13:24 |
| sean-k-mooney | zigo: we moved nova-hybrid-plug to debian 12 to test with spice | 13:25 |
| sean-k-mooney | zigo: so we will proably move that to debain 13 once thats avaiable provided spice is not compiled out | 13:26 |
| frickler | zigo: fyi jamespage is no longer with canonical | 13:33 |
| zigo | Oh, I didn't know. | 13:34 |
| frickler | that's why I mention it ;) happened pretty recently | 13:34 |
| zigo | Who's in charge of OpenStack stuff now at Canonical then? | 13:35 |
| frickler | I'm not sure, I think all the contacts I knew earlier are now gone | 13:36 |
| zigo | Only me stays ... forever ! :) | 13:40 |
| opendevreview | Sylvain Bauza proposed openstack/nova master: Add mtty/mdpy support for testing fake mdevs https://review.opendev.org/c/openstack/nova/+/898100 | 14:58 |
| opendevreview | Sylvain Bauza proposed openstack/nova master: WIP : Add mtty support to nova-next https://review.opendev.org/c/openstack/nova/+/922140 | 14:58 |
| sean-k-mooney | bauzas: so you can just install mdevctl on ubuntu | 16:20 |
| sean-k-mooney | its not there by default but it should be packaged | 16:20 |
| bauzas | melwitt: fun stuff, ubuntu noble doesn't have mdevctl https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_92e/openstack/92e4f13448c94210af9f36f145e4a58c/controller/logs/screen-n-cpu.txt | 16:20 |
| sean-k-mooney | https://manpages.ubuntu.com/manpages/focal/man8/mdevctl.8.html | 16:20 |
| sean-k-mooney | bauzas: it does its just not instlaled by defualt | 16:20 |
| bauzas | sean-k-mooney: yeah, we need to install it in nova-next | 16:20 |
| sean-k-mooney | you could add it to bindep in the test profile | 16:21 |
| bauzas | but honestly, I hate that silent exception | 16:21 |
| sean-k-mooney | to keep it optional | 16:21 |
| sean-k-mooney | bauzas: well the better fix would be to fall back to the old way of not using libvirt persitnace if mdevctl is not there | 16:21 |
| bauzas | sean-k-mooney: I honestly don't know how to do it, examples would be welcomed | 16:22 |
| bauzas | sean-k-mooney: yup, we should failback on a libvirt exception | 16:22 |
| sean-k-mooney | sure just add the package to https://github.com/openstack/nova/blob/master/bindep.txt | 16:22 |
| bauzas | I know what bindep is, I don't know how to make it optional | 16:22 |
| sean-k-mooney | [platform:dpkg test] | 16:22 |
| sean-k-mooney | install on debing but only for the test profile | 16:23 |
| sean-k-mooney | we use test in devstack | 16:23 |
| bauzas | k | 16:23 |
| sean-k-mooney | we could jsut list it as required | 16:23 |
| sean-k-mooney | i.e. without test | 16:23 |
| sean-k-mooney | if we want to always use the new persitance api | 16:23 |
| bauzas | that's what we do | 16:24 |
| bauzas | anyway, I'll quickly add it to bindep before I go off | 16:24 |
| opendevreview | Sylvain Bauza proposed openstack/nova master: Add mtty/mdpy support for testing fake mdevs https://review.opendev.org/c/openstack/nova/+/898100 | 16:26 |
| opendevreview | Sylvain Bauza proposed openstack/nova master: WIP : Add mtty support to nova-next https://review.opendev.org/c/openstack/nova/+/922140 | 16:26 |
| *** __ministry is now known as Guest14847 | 16:55 | |
| haleyb | zigo: regarding Canonical contact, i would start with Billy (wolsen) who isn't usually on irc, but you could try me as a proxy, i meet with them every Monday | 17:03 |
| EugenMayer440180 | after my instance stopped restart properly, i restarted the cluster. Now i see that the docker container nova-compute did not start (restarted the controller/computes entirely), with the message '3 RLock(s) were not greened, to fix this error make sure you run eventlet.monkey_patch() before importing any other modules.' | 17:38 |
| EugenMayer440180 | it seems like the controller shows all the VM 'active' and running, but they are not. Seems like the state cannot be updated | 17:41 |
| EugenMayer440180 | the actual issue seems to be 'My compute node 9946da6f-9a3d-45c7-a9e3-e7e8f99811fc has hypervisor_hostname compute1.cluster.kontextwork.net but virt driver reports it should be compute1. Possible rename detected, refusing to start!' | 17:49 |
| EugenMayer440180 | we recently had issues with 2024.2 with hostname != hostname -f and thus we changed that the computes nor longer have a FQDN like 'compute1.cluster.kontextwork.net' but rather compute1 | 17:54 |
| EugenMayer440180 | this seems to be an issue now. Is there any way to re-register the computes with the controller using the new hostname without loosing any data? | 17:55 |
| dansmith | EugenMayer440180: you cannot change the hostname of the computes, ever, for any reason | 18:13 |
| EugenMayer440180 | well i just did. | 18:24 |
| EugenMayer440180 | changed it in the database, restarted the computes and everything is up and running again | 18:29 |
| sean-k-mooney | EugenMayer440180: the problem is the hostname is also in teh cinder and neutron dbs in several tables | 18:30 |
| EugenMayer440180 | not using cinder, so this one i dodged. Neutron and the network agents already think about compute1 | 18:31 |
| sean-k-mooney | you would have to consitently change it in 10-20 tables | 18:31 |
| EugenMayer440180 | at least when quering the network agents and so forth, they all use the hostname already (or lets say the hostname which is the new FQDN) | 18:31 |
| sean-k-mooney | changing the hypervior_hostname will break all of the placement allcoations too | 18:31 |
| sean-k-mooney | neutron port bindign if they are out of sync can be fixed by detachign the ports and reattachign them or live migration/shleve/resize | 18:32 |
| sean-k-mooney | EugenMayer440180: the thing you hit preventign the agent form starting was likely https://specs.openstack.org/openstack/nova-specs/specs/2023.1/implemented/stable-compute-uuid.html | 18:33 |
| sean-k-mooney | after years fo saying renaming compute nodes is not supproted we finally started enforcing it | 18:34 |
| EugenMayer440180 | checked the allocations tables in nova/nova_api/nova_cell0, they are all empty | 18:34 |
| EugenMayer440180 | sean-k-mooney i did not choose to rename those | 18:34 |
| sean-k-mooney | that not where they are stored | 18:34 |
| sean-k-mooney | placement has its own db | 18:34 |
| sean-k-mooney | EugenMayer440180: has the value of hypervior_hostname changed in the db? if so that is the name used for the compute_node resouce provider in placement | 18:35 |
| EugenMayer440180 | found them there, as you expected. Will try to fix them there too. Thanks for the hint | 18:37 |
| EugenMayer440180 | sean-k-mooney i understand that i tapped into the internals. But there was no other choice for me here and i will clean up the mess that this created one by one until we migrate away i assume. | 18:41 |
| EugenMayer440180 | The reason i changed the FQDN was the migration to 2024.2 where nova stopped scheduling instances since in my case, the compute nodes always had the 'hostname' (compute1) while the hypervisors had the FQDN (hostname -f) ... it's nothing i have chosen and worked since not sure, 5 years - but then stopped here. Then i had to change the FQDN work | 18:43 |
| EugenMayer440180 | again, so i could e.g. restore instances from glance again. What i did not notice then is that this then broke the hypervisors / nova_libvirt (which happened now) | 18:43 |
| EugenMayer440180 | i was told then 'openstack does not like when 'hostname != hostname -f' and this can lead to issues ... which it did with 2024. Kind of suprising that something of this size cannot handle what is a usual setup for me since decades. | 18:44 |
| sean-k-mooney | we have not chnage how the logic in nova works | 18:45 |
| sean-k-mooney | you do not have to have host and hypervior_hostname match | 18:45 |
| sean-k-mooney | they can be differnt they jsut can change over time | 18:45 |
| sean-k-mooney | hypervior hostname comes form libvirt | 18:45 |
| sean-k-mooney | you can see what it returns with virish hostname | 18:45 |
| EugenMayer440180 | well not sure who broke it, might have been neutron, since it was all about the chassis name | 18:46 |
| sean-k-mooney | the host value comes form the config and defautl to socket.gethostname)( | 18:46 |
| EugenMayer440180 | 'due to no OVN chassis for host: compute3' | 18:46 |
| sean-k-mooney | neutron have ways to cofnigure that via data in the ovs db in the case of ovn or config options for ml2/ovs | 18:46 |
| EugenMayer440180 | with 2024.2 the chassis name (which was compute1 and so forth in my case) was no longer how nova expected (since what nova expected was compute1.cluster...) - this was the change that broke it | 18:47 |
| sean-k-mooney | the value we pass for nova to neutron in the port biding is the valuce form conf.host | 18:48 |
| sean-k-mooney | which default to socket.gethostname() | 18:48 |
| sean-k-mooney | so that implies either your dhcp server started handign out domians | 18:48 |
| EugenMayer440180 | well then it either changed with neitron or kolla does no longer properly configure those - cannot say. | 18:48 |
| sean-k-mooney | you /etc/host change teh fqdn to be the canonical hostname | 18:48 |
| sean-k-mooney | or /etc/hostname now has the fqdn | 18:48 |
| EugenMayer440180 | My DNS was always resolving the FQDN properly, yes | 18:48 |
| EugenMayer440180 | yes hosts where properly setup, as usual, and also the hostname. I mean, it's the 101 and it was not changed, and after it broke, i tripple checked it. | 18:49 |
| sean-k-mooney | ya so i know that the ovs-ctl script that is used to start ovs can also have a hostnaem parmater | 18:50 |
| sean-k-mooney | i would assume it has to be either a change in that, in ovs or in how ovn is configured by kolla | 18:50 |
| sean-k-mooney | EugenMayer440180: did you report an issue to the kolla team? | 18:50 |
| EugenMayer440180 | yes, kolla might be a source for that trouble too. | 18:51 |
| sean-k-mooney | im wondering if it could be a regresion on there end and if other will be impacted | 18:51 |
| EugenMayer440180 | Yes / no. It basically ended up to be noones 'topic'. Neutron was insisting on 'that is how chasis are', nova was 'well it must much, this is how nova always looked for the chases' and kolla was 'well we did not change anything, do not look here' | 18:51 |
| EugenMayer440180 | If i may ask, a different topic. The nova cli is deprecated as i have seen, but the openstack cli has no replacement for 'backup' - does this mean that it will be dropped in the (when?) near future? | 18:55 |
| EugenMayer440180 | is there an official EOL for the nova cli? | 18:55 |
| sean-k-mooney | yes and no. its been feature frozen for about 2 years | 18:57 |
| sean-k-mooney | the python bindign are also frozen | 18:57 |
| sean-k-mooney | we plan to remove it once heat/watcher/horizon stop usign it | 18:57 |
| sean-k-mooney | EugenMayer440180: https://docs.openstack.org/python-openstackclient/latest/cli/command-objects/server-backup.html | 18:58 |
| EugenMayer440180 | any time horizon here? years, months? 2025.x or 2026? | 18:58 |
| sean-k-mooney | my hope would be in the next 12-24 months but eventlet removal is a higher priority | 18:59 |
| EugenMayer440180 | i tried to onboard the above, but it always failed with 'More than one Image exists with the name' | 18:59 |
| sean-k-mooney | EugenMayer440180: osc has backup and we reached feature partiy with nova client a few years ago. teh only comamnd we did nto port are the host evacuate ones | 19:00 |
| EugenMayer440180 | last time i tried was Nov 2023 though | 19:00 |
| EugenMayer440180 | just tried again and: | 19:02 |
| EugenMayer440180 | openstack --insecure server backup create --os-compute-api-version 2.64 --name "backup-vpn" --type daily --rotate 5 2ba904fb-47e3-4ee8-92b4-66e41bcf2a96 | 19:02 |
| EugenMayer440180 | More than one Image exists with the name 'backup-vpn'. | 19:02 |
| sean-k-mooney | well you ment to provide unique names for each backup i belive | 19:02 |
| sean-k-mooney | when passing --name | 19:02 |
| EugenMayer440180 | well it creates a backup, it just fails | 19:03 |
| EugenMayer440180 | (and nova backup actually needs that name to rotate .. so i thing osc will need that too;v | 19:03 |
| EugenMayer440180 | i would love to use a name including a date, but this does not work for rotate - the name has to match the older backups. | 19:04 |
| sean-k-mooney | well osc is using the sdk | 19:04 |
| sean-k-mooney | but lo9oking at the api | 19:04 |
| sean-k-mooney | https://docs.openstack.org/api-ref/compute/#id57 | 19:04 |
| sean-k-mooney | the name is not optional | 19:04 |
| sean-k-mooney | EugenMayer440180: its not a bfv instance is it | 19:05 |
| EugenMayer440180 | my point is, nova cli forced the name to be 'always the same' to be able to group the rotation - osc dropts that issue but ends up beeing in need for that too anyway. so with osc you are basically in a no-go spot | 19:05 |
| EugenMayer440180 | bfv? | 19:05 |
| sean-k-mooney | boot form volume | 19:05 |
| sean-k-mooney | osc will default to using the server display name if you dont pass a name for the backup | 19:06 |
| sean-k-mooney | wich is reasonble but do you know if the error is coming form nova or form the client | 19:06 |
| EugenMayer440180 | if osc defaults to that, it will drop an error, since the server-name should be the same every time | 19:06 |
| EugenMayer440180 | oh you mean i should try that once | 19:07 |
| sean-k-mooney | the rotaion i belive is ment ot be server side | 19:07 |
| sean-k-mooney | but im wonderign if your geting a error form nova or if its coming form the client or sdk | 19:08 |
| EugenMayer440180 | the rotation is based on the name - that is no assumption. Tried and tested that, documented that | 19:08 |
| sean-k-mooney | it might be form here https://github.com/openstack/python-openstackclient/blob/master/openstackclient/compute/v2/server_backup.py#L102 | 19:09 |
| sean-k-mooney | so they should be saving the uuid that is returned form the backup call and then using that not the name to look upt the image | 19:09 |
| EugenMayer440180 | well is that new on osc? because nova does not do that, it used the name | 19:09 |
| sean-k-mooney | this is an osc bug | 19:10 |
| EugenMayer440180 | or, nova changed that recently, i'am not reading the commits | 19:10 |
| sean-k-mooney | its a regression form when they swaped form the novaclint to the sdk | 19:10 |
| sean-k-mooney | EugenMayer440180: this has not change in nova in many many years | 19:10 |
| EugenMayer440180 | well then i can assure you, the 'uuid lookup' is broken too. | 19:10 |
| sean-k-mooney | but the openstackclinet moved all the nova command to the openstacksdk | 19:11 |
| sean-k-mooney | EugenMayer440180: its not using uuid lookup at all | 19:11 |
| sean-k-mooney | its finding the backup image by name | 19:11 |
| EugenMayer440180 | ok you confused me / i got confused. If the name is required (you mean the server name or the backup name?) | 19:11 |
| sean-k-mooney | in novas api the name of the backup image is required | 19:12 |
| sean-k-mooney | osc is passing that to the backup_server functon | 19:12 |
| sean-k-mooney | but its alos doing image_client.find_image(backup_name, ignore_missing=False) | 19:12 |
| EugenMayer440180 | ok then we are actually on the same spot. The name is required and is used to find the older backups, the lookup group for the rotate, the PK or whatever | 19:12 |
| sean-k-mooney | i htink your error is happening here https://github.com/openstack/python-openstackclient/blob/master/openstackclient/compute/v2/server_backup.py#L102 | 19:13 |
| sean-k-mooney | i have an devstafck actully let me check | 19:13 |
| EugenMayer440180 | so changing the name for every backup would be wrong. Thus wht osc requires of me (using a unique name) is wrong (and this is what you pointed out to be a bug) | 19:13 |
| sean-k-mooney | ya it would | 19:15 |
| EugenMayer440180 | not sure how to take it, that the centralized, only way to backup an instance (openstack server backup) is broken for over 1.5 years and nobody has noticed that | 19:15 |
| sean-k-mooney | it will need to be sable | 19:15 |
| sean-k-mooney | EugenMayer440180: you realsise we condiered removing the backup api entirely | 19:15 |
| sean-k-mooney | instend of porting it | 19:15 |
| sean-k-mooney | it is not really used much | 19:16 |
| sean-k-mooney | most people just use snaphost directly instead of backup | 19:16 |
| sean-k-mooney | because backup has never workd form server booted form cinder volume | 19:16 |
| sean-k-mooney | where as the snapshot command do | 19:16 |
| sean-k-mooney | im not saying this should not be fix im saying its a rearly used api/command | 19:17 |
| EugenMayer440180 | yeah, this would explain that. Can you safely say that both 2025.x releases will not remove nova backup? or is this nothing that can be said | 19:18 |
| sean-k-mooney | more or less yes. i expect it to still be a thing in 2026.1 | 19:19 |
| sean-k-mooney | it might not exist in 2027.1 or 2027.2 if we mange to get everyone over to the sdk | 19:20 |
| EugenMayer440180 | well i will not get to 2026 anyway, so this is not what i would worry then. I plan to migrate away from openstack in Dec 2025 and want to be sure the backup strategy works as long as that. | 19:21 |
| sean-k-mooney | EugenMayer440180: https://paste.opendev.org/show/bZLdha8p7l5juf4jF8Dw/ | 19:21 |
| EugenMayer440180 | Again sean-k-mooney, as all the other times, you helped me so much. I would hand you just another medal for that :) | 19:21 |
| sean-k-mooney | so with a clean vm on master the backup commadn works | 19:21 |
| sean-k-mooney | at least once | 19:21 |
| sean-k-mooney | lets see what happens if i do it again | 19:22 |
| sean-k-mooney | sean@devstack-ui:~$ openstack --os-cloud devstack-admin server backup create --name test --type daily --rotate 5 6a314bdf-66ba-42d5-908a-c13e6f67b214 | 19:22 |
| sean-k-mooney | Could not load 'message_list': module 'zaqarclient.queues.v2.cli' has no attribute 'OldListMessages' | 19:22 |
| sean-k-mooney | Could not load 'message_post': module 'zaqarclient.queues.v2.cli' has no attribute 'OldPostMessages' | 19:22 |
| sean-k-mooney | More than one Image exists with the name 'test'. | 19:22 |
| sean-k-mooney | so ya my guess is we only test creating 1 backup | 19:22 |
| sean-k-mooney | and since that works we didnt notice | 19:22 |
| EugenMayer440180 | for a backup supporting rotation? thats elaborated :) | 19:23 |
| sean-k-mooney | https://paste.opendev.org/show/827657/ | 19:23 |
| sean-k-mooney | so this is a clinet bug | 19:24 |
| sean-k-mooney | nova responce fine and it fails exactly where i said it would | 19:24 |
| sean-k-mooney | sean@devstack-ui:~$ openstack --os-cloud devstack-admin image list | 19:25 |
| sean-k-mooney | Could not load 'message_list': module 'zaqarclient.queues.v2.cli' has no attribute 'OldListMessages' | 19:25 |
| sean-k-mooney | Could not load 'message_post': module 'zaqarclient.queues.v2.cli' has no attribute 'OldPostMessages' | 19:25 |
| sean-k-mooney | +--------------------------------------+--------------------------+--------+ | 19:25 |
| sean-k-mooney | | ID | Name | Status | | 19:25 |
| sean-k-mooney | +--------------------------------------+--------------------------+--------+ | 19:25 |
| sean-k-mooney | | 6a1617c4-7b38-47ce-a197-c24932f1f358 | cirros-0.6.3-x86_64-disk | active | | 19:25 |
| sean-k-mooney | | e7feb98c-9eca-463b-a8d1-54d90b265c76 | test | active | | 19:25 |
| sean-k-mooney | | 7631698f-aac7-4ba0-93fd-c4ac7ccdc46f | test | active | | 19:25 |
| sean-k-mooney | | 19aaf0a5-2978-4a8f-ac69-a812e4805750 | test | active | | 19:25 |
| sean-k-mooney | +--------------------------------------+--------------------------+--------+ | 19:25 |
| sean-k-mooney | its not happy with that as it expecgin thte name to be unique | 19:25 |
| sean-k-mooney | so we can see in the debug output on line 62 https://paste.opendev.org/show/827657/ | 19:27 |
| sean-k-mooney | RESP BODY: {"image_id": "e7feb98c-9eca-463b-a8d1-54d90b265c76"} | 19:28 |
| sean-k-mooney | so the fix should be to save that to var and use it when geting the iamge https://github.com/openstack/python-openstackclient/blob/master/openstackclient/compute/v2/server_backup.py#L94-L102 | 19:28 |
| EugenMayer440180 | good catch | 19:31 |
| sean-k-mooney | EugenMayer440180: so unfortunetly to fix this we need to fix the sdk first | 19:38 |
| sean-k-mooney | https://github.com/openstack/openstacksdk/blob/df1b9e4f8ca877a7caddcbb9afc66de225a8dd21/openstack/compute/v2/server.py#L635-L655 | 19:38 |
| sean-k-mooney | it should return the id like create_image does https://github.com/openstack/openstacksdk/blob/df1b9e4f8ca877a7caddcbb9afc66de225a8dd21/openstack/compute/v2/server.py#L508-L551 | 19:39 |
| opendevreview | melanie witt proposed openstack/nova master: DNM try to install mdevctl https://review.opendev.org/c/openstack/nova/+/948581 | 19:53 |
| opendevreview | melanie witt proposed openstack/nova master: DNM try to install mdevctl https://review.opendev.org/c/openstack/nova/+/948581 | 20:04 |
| sean-k-mooney | EugenMayer440180: i have not witten any test but i have a version working locally | 20:07 |
| sean-k-mooney | EugenMayer440180: https://review.opendev.org/c/openstack/python-openstackclient/+/948586 and https://review.opendev.org/c/openstack/openstacksdk/+/948584 are the fixes | 20:23 |
| sean-k-mooney | EugenMayer440180: but it would be nice if there was a bug report. it will take a while to fully fix it becasue we will need a new sdk relase and test and then the client patch can merge after | 20:24 |
| sean-k-mooney | EugenMayer440180: im not sure if/when ill have time to do that but your welcome to try out the fix | 20:25 |
| opendevreview | melanie witt proposed openstack/nova master: DNM try to install mdevctl https://review.opendev.org/c/openstack/nova/+/948581 | 20:39 |
| sean-k-mooney | melwitt: weried im not sure why bindep didnt work | 20:50 |
| EugenMayer440180 | sean-k-mooney thanks! | 20:51 |
| sean-k-mooney | melwitt: oh i see the issue "bindep_profiles=" | 20:53 |
| sean-k-mooney | i was wrgon we dont install the test profile in the devstack job we invoke it with no profile | 20:54 |
| sean-k-mooney | so the fix is just to remvoe test form https://review.opendev.org/c/openstack/nova/+/922140/20/bindep.txt | 20:54 |
| melwitt | sean-k-mooney: I dunno, I looked around for awhile in devstack and codesearch and it looked like it's pretty much just for tox envs like func and unit test jobs | 20:54 |
| sean-k-mooney | right | 20:54 |
| sean-k-mooney | i guess maybe we dont invoke it at all | 20:55 |
| melwitt | sean-k-mooney: oh really? I looked for an invocation in the job output and didn't find it | 20:55 |
| sean-k-mooney | well im not sure | 20:55 |
| sean-k-mooney | melwitt: we do this https://paste.opendev.org/show/bdsCVZoGiydogkQcKlh8/ | 20:56 |
| sean-k-mooney | so im seing 2 things local bindep=0 | 20:56 |
| sean-k-mooney | and local bindep_profiles= | 20:57 |
| melwitt | yeah IIUC that is looking for a "-bindep" option in the install package args and if it's not found it won't invoke | 20:57 |
| sean-k-mooney | so it looks like we install it without enabling bindep or any profile | 20:57 |
| sean-k-mooney | ok so the fix i guess is what your propsoing | 20:58 |
| sean-k-mooney | jsut have the nova devstack plugin install it | 20:58 |
| melwitt | I might be wrong, just wanted to try it out and see what happens | 20:58 |
| sean-k-mooney | oh your doing it with a preplaybook | 20:58 |
| sean-k-mooney | it would be nice to just do it in the devstack plug since its compiling the mtty module too | 20:58 |
| sean-k-mooney | it might as well ensure mdevctl is install that way we can use it locally too | 20:59 |
| melwitt | from what I could tell there is not a clean way to invoke bindep if it's not a tox job and then I thought well I could just run bindep and then I thought if bindep isn't already running then I don't need to install 50 packages just to get one package | 20:59 |
| sean-k-mooney | right we dont need to use bindep | 20:59 |
| sean-k-mooney | we can just do package_install mdevctl here https://github.com/openstack/nova/blob/master/devstack/plugin.sh#L9 | 20:59 |
| melwitt | yeah, that's a good point it would be good to have it for devstack dev | 20:59 |
| sean-k-mooney | oh its install_package not package install | 21:00 |
| sean-k-mooney | but ya "install_package mdevctl" woudl fix it or you could do that in devstack propper | 21:01 |
| sean-k-mooney | here https://github.com/openstack/devstack/blob/master/lib/nova_plugins/hypervisor-libvirt#L90-L125 | 21:02 |
| melwitt | yeah I was looking at that too earlier | 21:02 |
| sean-k-mooney | or here https://github.com/openstack/devstack/blob/master/lib/nova_plugins/functions-libvirt#L58-L106 | 21:02 |
| sean-k-mooney | i honestly dont really have a prefernce | 21:02 |
| sean-k-mooney | other then via devstack in some from | 21:03 |
| sean-k-mooney | the playbook works fine for ci too | 21:03 |
| melwitt | that's what I found first but didn't want to mess with it just to see nova-next run | 21:03 |
| melwitt | me neither. I am not that expert about devstack and all that | 21:03 |
| sean-k-mooney | the reson i sugggested here https://github.com/openstack/nova/blob/master/devstack/plugin.sh#L9 was it will only do it if we compile the mtty supprot | 21:04 |
| sean-k-mooney | so it would only be done in nova-next | 21:04 |
| melwitt | I think that makes sense | 21:04 |
| sean-k-mooney | but ya if it works i guess any of the options are fine | 21:04 |
| sean-k-mooney | on ubuntu its a recommended package but we dont install those by default | 21:05 |
| melwitt | yeah I noticed it in the "recommended packages" in the job output | 21:05 |
| sean-k-mooney | the recommened packages are not insane we could enable those. the suggesetd ones are more porblematic | 21:06 |
| melwitt | welp, my thing didn't work. zuul job already angry 😝 | 21:07 |
| sean-k-mooney | :) it was worth a try | 21:07 |
| sean-k-mooney | oh i see | 21:07 |
| melwitt | "/opt/stack/devstack/local.sh: line 18: install_package: command not found" yay | 21:07 |
| sean-k-mooney | ya that ran at the end but im not sure if its sourced | 21:08 |
| melwitt | I thought devstack functions would be available in there. I dunno | 21:08 |
| sean-k-mooney | if its just run in a subshell it wont have access to devtack functions | 21:08 |
| melwitt | that must be why then I guess | 21:09 |
| sean-k-mooney | its odd because i tought it was actully sourced but i may have imagined that | 21:09 |
| sean-k-mooney | ok ya its not sourced https://github.com/openstack/devstack/blob/master/stack.sh#L1489 | 21:10 |
| melwitt | a-ha | 21:10 |
| sean-k-mooney | if you wanted to be really hacky not that i would encurage that :) local.conf is sourced.... | 21:11 |
| sean-k-mooney | but ya i guess plugin and then maybe squash with sylvian or waith thill they get back | 21:12 |
| melwitt | me do hacks? I would never! | 21:12 |
| melwitt | 😂 | 21:12 |
| melwitt | yeah sylvain can add whatever way he prefers. I didn't want to mess up his patch | 21:13 |
| sean-k-mooney | are you on PTO tomorow? im not sure who is around and who is off | 21:14 |
| melwitt | no I'm not | 21:14 |
| sean-k-mooney | ah ok so may 1st isnt a day off in the US | 21:14 |
| melwitt | oh yeah it's a holiday in france looks like | 21:15 |
| sean-k-mooney | in ireland too kind of we move almost all our public holidays to the ned monday https://publicholidays.ie/ | 21:16 |
| melwitt | oh I see. that's better I think | 21:17 |
| sean-k-mooney | what confusitng is while we normally jsut call it the may bankholidy its somethime called labor day but that differnt in the us | 21:18 |
| sean-k-mooney | so i avoid that name | 21:18 |
| melwitt | may day but not mayday | 21:19 |
| sean-k-mooney | ya so may1st is tradtionaly the 1st day of summer in ireland | 21:20 |
| sean-k-mooney | well ikid of its a whole thing https://en.wikipedia.org/wiki/Irish_calendar the month of summer in teh irish language do not align to those in most english speaking contries | 21:23 |
| opendevreview | melanie witt proposed openstack/nova master: DNM try to install mdevctl https://review.opendev.org/c/openstack/nova/+/948581 | 21:24 |
| melwitt | cool I didn't know that | 21:26 |
| sean-k-mooney | its partly a genererational thing. while its still tought espically in irish speakign schools both calandars are actully used dependign on the context. | 21:27 |
| sean-k-mooney | melwitt: when i was young i was alwasy confused why americans have spring break in summer | 21:28 |
| sean-k-mooney | hum maybe ye dont | 21:29 |
| sean-k-mooney | https://www.voyista.com/blog/when-are-spring-break-dates-colleges-universities ? | 21:29 |
| JayF | Spring Break is usually in April, but it can depend on when Easter lands | 21:29 |
| sean-k-mooney | i tought it was in may for some reason | 21:29 |
| sean-k-mooney | ah ok | 21:30 |
| JayF | early may or late march is likely possible | 21:30 |
| JayF | plus it's not standard; in most places in the US school calendars are set at the district (city/county) level | 21:30 |
| JayF | although in practice they generally fall in close proximity | 21:30 |
| melwitt | I was going to say similar, that they usually put it near easter | 21:30 |
| sean-k-mooney | that must make thing harder to plan for | 21:30 |
| sean-k-mooney | ya its the same here we just call it midterm and it easter adjcent | 21:31 |
| JayF | I mean, really same thing for Summer break | 21:31 |
| JayF | my wife has worked schools that start early August, she has worked schools that start early September (with summer start moving along with the start date) | 21:32 |
| sean-k-mooney | not that the first sunday after the first full moon after the equanox relaly help nail it down | 21:32 |
| melwitt | lol | 21:33 |
| sean-k-mooney | we litrally get the modern Gregorian calendar form cristiantity and then dont even use it for easter... | 21:33 |
| sean-k-mooney | it moves because it tied to the 13 month lunar calandar and you have to translate | 21:33 |
| sean-k-mooney | an this is why you thank the maintianer so time zone lib for there hard work and hope you never have to maintian them | 21:34 |
| sean-k-mooney | that and 15min timezones that have non 1 hour DST adjustments | 21:35 |
| sean-k-mooney | https://youtu.be/-5wpm-gesOY?si=ufuellJXKZcZBjQ7 | 21:38 |
| melwitt | ok so installing it in plugin.sh made it install to the controller only, need to also install on the subnode | 22:05 |
| sean-k-mooney | well that implies the plugin is not runing on the compute | 22:06 |
| sean-k-mooney | we need to compile the module in both places | 22:06 |
| melwitt | yeah | 22:06 |
| sean-k-mooney | so the plugin is probly just not enabled on the subnode | 22:06 |
| sean-k-mooney | ya looking at the job the devstack_plugins section is not defiend on the subnode | 22:07 |
| sean-k-mooney | which is a prexisting bug i guess | 22:08 |
| sean-k-mooney | maybe also the libvirt section libvirt | 22:08 |
| melwitt | ah ok | 22:09 |
| sean-k-mooney | hum | 22:09 |
| sean-k-mooney | https://718714dcc35fdedf2036-89f33ee168e8c2014b2d25c632c1d92a.ssl.cf1.rackcdn.com/openstack/0dc5d5810e004f078a7dd5cf7a968e79/compute1/logs/local_conf.txt | 22:09 |
| sean-k-mooney | so no that not the problem | 22:09 |
| sean-k-mooney | the problem is we are not setting the vard to do the compile? | 22:10 |
| melwitt | oh huh | 22:10 |
| sean-k-mooney | ya NOVA_COMPILE_MDEV_SAMPLES is not defiend on the subnode | 22:11 |
| melwitt | ohh | 22:11 |
| opendevreview | melanie witt proposed openstack/nova master: DNM try to install mdevctl https://review.opendev.org/c/openstack/nova/+/948581 | 22:12 |
| melwitt | 5th times a charm | 22:14 |
| opendevreview | melanie witt proposed openstack/nova master: DNM try to install mdevctl https://review.opendev.org/c/openstack/nova/+/948581 | 22:43 |
| melwitt | artom_: I'm working on the vtpm live migration spec and I notice in the wip patches all of the config option names are "tpm" vs "vtpm" from the spec. for example, default_tpm_secret_security vs default_vtpm_secret_security. is this intentional? should I update the config option names in the spec while I am at it? | 23:41 |
Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!