| opendevreview | HanGuangyu proposed openstack/nova master: Unify ServersController._flavor_id_from_req_data param to server_dict https://review.opendev.org/c/openstack/nova/+/898315 | 11:27 |
|---|---|---|
| ioni | hi guys | 12:09 |
| ioni | i have an issue with deleting an instance after upgrading to Zed(I think). The error I'm having is: Requested operation is not valid: cannot undefine domain with nvram | 12:10 |
| ioni | https://paste.xinu.at/qvp4dm/ | 12:10 |
| ioni | the only thing that I changed is to add hw_machine_type=q35 to that specific image | 12:11 |
| ioni | and is kinda random | 12:11 |
| ioni | to delete the instance successfully i have to run virsh undefine domain --nvram first | 12:11 |
| ioni | the problems occurs randomly on instances that sometimes are newly built with that image and deleted after | 12:12 |
| ioni | i think i need to add to the image hw_firmware_type uefi for nova to understand to undefine nvram first before undefining the domain | 12:22 |
| opendevreview | ribaudr proposed openstack/nova-specs master: Re-propose "Allow Manila shares to be directly attached to an instance when using libvirt" for Caracal https://review.opendev.org/c/openstack/nova-specs/+/898319 | 13:17 |
| *** drannou_ is now known as drannou | 13:18 | |
| opendevreview | ribaudr proposed openstack/nova-specs master: Re-propose "Allow Manila shares to be directly attached to an instance when using libvirt" for Caracal https://review.opendev.org/c/openstack/nova-specs/+/898319 | 13:20 |
| opendevreview | ribaudr proposed openstack/nova-specs master: Re-propose "Allow local scaphandre directory to be mapped to an instance using virtiofs" for Caracal https://review.opendev.org/c/openstack/nova-specs/+/898320 | 13:24 |
| dansmith | we've got something going on with the live migration jobs and maybe others | 14:43 |
| dansmith | lots of post failures | 14:43 |
| dansmith | "msg": "The `lxml` module is not importable. Check the requirements." | 14:46 |
| dansmith | sean-k-mooney: do you know how we get ansible modules installed on the hosts? like the virt module, which has requirements for libvirt-python, etc | 14:52 |
| opendevreview | Tobias Urdin proposed openstack/nova master: [WIP] Handle scaling of cputune.shares https://review.opendev.org/c/openstack/nova/+/898326 | 14:56 |
| opendevreview | Tobias Urdin proposed openstack/nova master: [WIP] Handle scaling of cputune.shares https://review.opendev.org/c/openstack/nova/+/898326 | 14:59 |
| tobias-urdin | sean-k-mooney[m]: feedback on ^ – doesn't handle all cases yet but i dont want to spend time on it if it's just garbage then we'll just carry something similar downstream | 14:59 |
| dansmith | sean-k-mooney: https://6df0d90b08f38024b3b0-ddacc1cdacca3582cad1fbbc53a5babe.ssl.cf2.rackcdn.com/897674/1/check/nova-live-migration/beaa1e0/job-output.txt | 15:08 |
| sean-k-mooney | dansmith: so my guess is we need to set the interperter path for the remote node | 15:10 |
| dansmith | sean-k-mooney: to what? something inside our new venv? | 15:11 |
| sean-k-mooney | to the python3 binary in the venv | 15:11 |
| dansmith | still, something changed over the weekend and I don't know what that is.. maybe the ansible module got changed or libvirt-python changed to a soft req on lxml or something? | 15:11 |
| sean-k-mooney | ya did any changes happen in devstack we can look at this after the meeting | 15:12 |
| dansmith | not really seeing anything | 15:13 |
| opendevreview | Dan Smith proposed openstack/nova master: Install lxml before we need it in post-run https://review.opendev.org/c/openstack/nova/+/898435 | 15:33 |
| dansmith | sean-k-mooney: bauzas this ^ seems to be working, it already passed the live-migration-ceph job | 16:34 |
| sean-k-mooney | sweet i have it open but had not refreshed | 16:35 |
| sean-k-mooney | ah i assume your watching the live output | 16:35 |
| dansmith | yup | 16:35 |
| dansmith | just passed the main job as well | 16:36 |
| sean-k-mooney | added link to the passing job where it ran and left a +2 | 16:39 |
| sean-k-mooney | if we start seeing this type of dep issue often i woudl prefer to use bindep or similar to store the packages in a file or move the list of packagges to a role default | 16:39 |
| sean-k-mooney | but for now this is fine | 16:39 |
| dansmith | ack, well, I'd like to get the gate unblocked at least | 16:41 |
| dansmith | I know bindep is what we use for that stuff normally, but I figured it was more aimed at setup for running the services and not just for ansible stuff to work | 16:42 |
| dansmith | I suppose it's also usable | 16:42 |
| sean-k-mooney | bindep has tags | 16:42 |
| sean-k-mooney | so we can use [ansible] | 16:42 |
| sean-k-mooney | for example if we wanted to store ansible only deps or whatever | 16:43 |
| dansmith | ack | 16:43 |
| sean-k-mooney | we do use the "test" tag for functional/unit tests deps | 16:43 |
| dansmith | so maybe I should just slap this into bindep.txt instead of doing this in the ansible? | 16:43 |
| dansmith | or maybe just follow this up with that so we can unblock the gate | 16:43 |
| sean-k-mooney | well bindep is likely not aviable on the host outside of the venv where devstack installed it | 16:44 |
| sean-k-mooney | so if we want to go that way lets do it in a followup | 16:44 |
| sean-k-mooney | as first we need to decied how to make bindep avaialable | 16:44 |
| dansmith | oh, we install bindep in the venv to manage deps on the host? I figured that was installed and managed before we got to devstack at all | 16:45 |
| dansmith | yeah, let me queue up something behind this if for no other reason than to serve as a reminder | 16:45 |
| sean-k-mooney | for the devstack job i belive its in the venv for the unit test jobs its installed in the job | 16:46 |
| sean-k-mooney | before we invoke tox | 16:46 |
| sean-k-mooney | https://github.com/openstack/nova/blob/9dceb735f6926c71a8c2e2c6ef8b5e0a1363041d/bindep.txt#L24 | 16:46 |
| sean-k-mooney | lxml is a dep in bindep already for what its worth | 16:46 |
| sean-k-mooney | but its marked as a test dep | 16:47 |
| sean-k-mooney | so its not installed by default by devstack | 16:47 |
| sean-k-mooney | actully no | 16:47 |
| sean-k-mooney | thats not the python bindings | 16:47 |
| sean-k-mooney | that the clib that lxml uses | 16:48 |
| dansmith | yeah, neither is libvirt-python, but that's clearly installed which is weird that it works | 16:48 |
| dansmith | er, python3-libvirt | 16:48 |
| sean-k-mooney | i belive that will be install by devstack seperatly | 16:48 |
| sean-k-mooney | https://github.com/openstack/devstack/blob/ca4d5132e63752878620c4e4f374d98d433b3f52/lib/nova_plugins/functions-libvirt#L96 | 16:49 |
| sean-k-mooney | and here for ubuntu https://github.com/openstack/devstack/blob/ca4d5132e63752878620c4e4f374d98d433b3f52/lib/nova_plugins/functions-libvirt#L72C87-L72C102 | 16:49 |
| opendevreview | Dan Smith proposed openstack/nova master: Move lxml dep to bindep.txt https://review.opendev.org/c/openstack/nova/+/898439 | 16:49 |
| dansmith | sean-k-mooney: right but I would expect python3-libvirt to have a hard dep on python3-lxml, but perhaps not | 16:50 |
| sean-k-mooney | i dont think it does | 16:50 |
| sean-k-mooney | we need lxml to parse the xml | 16:50 |
| sean-k-mooney | but the xml is generated by libvirt | 16:50 |
| sean-k-mooney | which would use the c lib | 16:50 |
| sean-k-mooney | not the python bindings | 16:50 |
| dansmith | I guess I expected some of the python bindings to manipulate xml chunks themselves, but perhaps not | 16:51 |
| sean-k-mooney | so just confirmign that his would work devstack invokes bindep if teh bindep file exits for a project here in setup_package https://github.com/openstack/devstack/blob/ca4d5132e63752878620c4e4f374d98d433b3f52/inc/python#L465 | 16:52 |
| sean-k-mooney | that is called form _setup_package_with_constraints_edit which is called form setup_deploy here https://github.com/openstack/devstack/blob/ca4d5132e63752878620c4e4f374d98d433b3f52/inc/python#L359 | 16:55 |
| sean-k-mooney | and we call that for nova here https://github.com/openstack/devstack/blob/ca4d5132e63752878620c4e4f374d98d433b3f52/lib/nova#L971 | 16:56 |
| sean-k-mooney | so based on that call graph | 16:56 |
| sean-k-mooney | we are not specifying any profile | 16:56 |
| sean-k-mooney | i.e. it sthe same as just runing bindep in the nova root effectivly | 16:56 |
| sean-k-mooney | so we just need to confirm that https://review.opendev.org/c/openstack/nova/+/898439 make it included in that properly | 16:57 |
| sean-k-mooney | ah you added no tag so yes it will be | 16:57 |
| dansmith | bauzas: can you +W this to unblock the gate please? https://review.opendev.org/c/openstack/nova/+/898435 | 17:36 |
| bauzas | the quickiest I was able to | 17:36 |
| dansmith | thanks | 17:36 |
| dansmith | sean-k-mooney: the bindep approach doesn't leave us with lxml installed | 17:38 |
| sean-k-mooney | ack i wonder if we are removing it in devstack to aovid pip issues | 17:41 |
| sean-k-mooney | id dont see anything that would remove it but lets stick with teh simple solution for now | 17:42 |
| dansmith | idk, but glad we approved the first solution in the short term :) | 17:42 |
| sean-k-mooney | dansmith: for what its worth i dont see any packages being installed | 17:51 |
| dansmith | sean-k-mooney: bindep packages you mean? | 17:51 |
| dansmith | I didn't see it try lxml that's for sure | 17:51 |
| sean-k-mooney | ya bindep is installed bu ti dont see ay execution of setup_package that actully installs anything | 17:51 |
| * dansmith nods | 17:51 | |
| sean-k-mooney | so that might have silently been broken at some point | 17:52 |
| sean-k-mooney | im sing things like inc/python:_setup_package_with_constraints_edit:387 : [[ /opt/stack/neutron == -bindep* ]] | 17:52 |
| dansmith | I'd have expected more breakage in that case | 17:53 |
| sean-k-mooney | ah so reading the code | 17:54 |
| sean-k-mooney | https://github.com/openstack/devstack/blob/master/inc/python#L385 | 17:54 |
| sean-k-mooney | the fintion is checkign if it was invoked with -bindep | 17:54 |
| sean-k-mooney | as a flag | 17:54 |
| sean-k-mooney | and poping it | 17:54 |
| sean-k-mooney | and we are not doign that | 17:54 |
| sean-k-mooney | we do not pass -bindep when seting up nova | 17:55 |
| dansmith | probably need to get gmann to consult here I think | 17:55 |
| sean-k-mooney | https://github.com/openstack/devstack/blob/ca4d5132e63752878620c4e4f374d98d433b3f52/lib/nova#L971 | 17:55 |
| sean-k-mooney | i dont see any usage of this in tree so maybe this was added for out of tree plugins to call | 17:55 |
| sean-k-mooney | https://github.com/openstack/devstack/commit/58243f6203f3385039fe7124b037d5abf7e1d59e | 17:56 |
| sean-k-mooney | we can always follow up in #openstack-qa but my read on this is the infra is in place but not hooked up to be called | 17:58 |
| sean-k-mooney | this seams to be the only? usage https://github.com/openstack/devstack/commit/dd3731c86a13334125715e44880491042526227c | 17:59 |
| sean-k-mooney | so it be used if we use libs_from_git=nova i guess but that shoudl be the case in teh ci job | 18:00 |
| sean-k-mooney | am not quite its still only takign that path for specific lib | 18:01 |
| sean-k-mooney | not anything listed in libs form git so ya it is not hooked up for nova/service proejcts | 18:02 |
| sean-k-mooney | it shoudl be simple enough to hook up but we shoudl check with gmann et al first to see if there is a reason to only do this for libs | 18:02 |
| dansmith | yeah seems like there must be something missing | 18:03 |
| sean-k-mooney | i think to really do this properly and get the most use out of it we woudl want to modify lib/nova and the nova hypervisor plugsin to pass a diffetn tag (the virt driver) and on the nova side group driver deps by tag .e.g libvirt or ironic in the bindeps file | 18:05 |
| sean-k-mooney | it would be a nice enhancement but a lot of work to fully get right. | 18:06 |
| sean-k-mooney | im honestly fine with your first patch | 18:06 |
| sean-k-mooney | i would only look at hooking up bindep in devstack if we find ourselve needing to do this more hten once or twice a cycle | 18:06 |
| gmann | dansmith: sean-k-mooney hi | 18:08 |
| sean-k-mooney | o/ | 18:08 |
| gmann | bindeps can be installed two way, via flag while installing nova or have file in devstack/file/bindep.txt itself which will be indalled by default | 18:09 |
| gmann | installed | 18:09 |
| gmann | but nova needs to have that in-tree under devstack directory with plugin setup | 18:09 |
| sean-k-mooney | gmann: yep so in this case its a dep for an ansible module we are using in one specific ci job post playbook | 18:10 |
| gmann | I mean here - https://github.com/openstack/nova/tree/master/devstack | 18:10 |
| sean-k-mooney | the dep also happens to be a dep nova has | 18:10 |
| sean-k-mooney | we are assumign that until last week lxml was insallsed transitivly via a weak dep somewhere | 18:11 |
| sean-k-mooney | so the ci playbook worked | 18:11 |
| sean-k-mooney | and that weak dep got lost over the weekend | 18:11 |
| gmann | ohk so we want to have explicit in nova bindep right? | 18:11 |
| sean-k-mooney | well for now we jsut installed it in the job. but then i mentioned bindep to dansmith but that dint work becasue we dont install nova's bindep currently | 18:12 |
| sean-k-mooney | gmann: i was wondering if there was a reason we only do it for specific libs | 18:12 |
| sean-k-mooney | i.e. if we passed the flag for nova would it break things? | 18:12 |
| dansmith | gmann: not sure what changed between last week and today, but this must have been on the system before | 18:13 |
| dansmith | gmann: but yeah, it's really an ansible module requirement, so not sure it should be in nova's bindep to be honest | 18:13 |
| gmann | ohk | 18:14 |
| gmann | not sure if changed in opendev side, I cannot remember/see anything devstack removed on this | 18:16 |
| sean-k-mooney | looking at the commit history i dont see anything obvious on the devstack side | 18:16 |
| gmann | yeah | 18:16 |
| sean-k-mooney | perhaps a chagn ein tempet or shared job config but i think its not realted to the devstack repo | 18:16 |
| gmann | which one? | 18:17 |
| sean-k-mooney | the partent of the job that failed was tempest-multinode-full-py3 so i was wondering if it was in anyway related to https://github.com/openstack/tempest/commit/5c64e3913a1a5a061a6b04b212ce35aaa84f2184 | 18:20 |
| sean-k-mooney | but i dont think so | 18:20 |
| gmann | yeah, that is not related | 18:20 |
| gmann | it is just adding stable/2023.2 jobs | 18:21 |
| sean-k-mooney | there is nothing else really that lins up | 18:21 |
| sean-k-mooney | for now we have a solution just isntalling the dep in teh job | 18:21 |
| sean-k-mooney | so we are unblocked | 18:21 |
| sean-k-mooney | https://review.opendev.org/c/openstack/nova/+/898435 | 18:21 |
| sean-k-mooney | why it broke is less imporant | 18:22 |
| gmann | someone at door. will be back in 10-15 min | 18:22 |
| sean-k-mooney | am im going to drop in a few mins o/ | 18:22 |
| opendevreview | Alexey Stupnikov proposed openstack/nova master: Translate VF network capabilities to port binding https://review.opendev.org/c/openstack/nova/+/884439 | 21:47 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!