Tuesday, 2020-08-18

« Monday, 2020-08-17 Index Wednesday, 2020-08-19 »
*** tetsuro has joined #openstack-nova00:01
*** zhanglong has joined #openstack-nova00:04
*** tetsuro_ has joined #openstack-nova00:06
*** tetsuro has quit IRC00:10
*** tetsuro_ has quit IRC00:16
*** zhanglong has quit IRC00:18
*** zhanglong has joined #openstack-nova00:20
*** brinzhang_ has joined #openstack-nova00:24
*** brinzhang0 has quit IRC00:27
*** mvorwerk_ has joined #openstack-nova00:29
*** mvorwerk has quit IRC00:30
*** songwenping__ has joined #openstack-nova00:41
*** songwenping_ has quit IRC00:44
*** songwenping_ has joined #openstack-nova01:01
openstackgerritMerged openstack/nova master: Removes the delta file once image is extracted  https://review.opendev.org/73442101:01
openstackgerritMerged openstack/nova stable/rocky: Remove 'test_cold_migrate_with_physnet_fails' test  https://review.opendev.org/70311501:01
*** songwenping__ has quit IRC01:04
*** jangutter_ has joined #openstack-nova01:43
*** jangutter_ has quit IRC01:43
*** jangutter_ has joined #openstack-nova01:43
*** jangutter has quit IRC01:44
*** gyee has quit IRC01:49
*** sapd1 has joined #openstack-nova01:51
*** zhanglong has quit IRC01:59
*** zhanglong has joined #openstack-nova02:00
*** brinzhang0 has joined #openstack-nova02:14
*** songwenping__ has joined #openstack-nova02:14
*** brinzhang_ has quit IRC02:17
*** songwenping_ has quit IRC02:18
*** songwenping_ has joined #openstack-nova02:25
*** mkrai has joined #openstack-nova02:25
openstackgerritMerged openstack/nova stable/rocky: Block rebuild when NUMA topology changed  https://review.opendev.org/70311602:28
*** songwenping__ has quit IRC02:28
openstackgerritMerged openstack/nova stable/rocky: Disable NUMATopologyFilter on rebuild  https://review.opendev.org/70311702:30
openstackgerritMerged openstack/nova stable/rocky: FUP for in-place numa rebuild  https://review.opendev.org/70311802:30
openstackgerritMerged openstack/nova stable/pike: Fix os-simple-tenant-usage result order  https://review.opendev.org/70960002:36
*** songwenping__ has joined #openstack-nova02:44
*** zhanglong has quit IRC02:46
*** songwenping_ has quit IRC02:47
*** zhanglong has joined #openstack-nova02:51
*** sapd1 has quit IRC02:52
*** hongbin has joined #openstack-nova03:15
openstackgerritXinliang Liu proposed openstack/nova master: Always pass image source info to ironic  https://review.opendev.org/74662603:16
*** songwenping_ has joined #openstack-nova03:34
*** songwenping__ has quit IRC03:37
*** zhanglong has quit IRC03:37
*** psachin has joined #openstack-nova03:37
*** dave-mccowan has quit IRC03:47
*** mkrai has quit IRC03:47
*** mkrai has joined #openstack-nova03:55
*** sapd1 has joined #openstack-nova04:17
*** tbachman has quit IRC04:20
*** songwenping__ has joined #openstack-nova04:21
*** songwenping_ has quit IRC04:24
*** ratailor has joined #openstack-nova04:30
*** evrardjp has quit IRC04:33
*** evrardjp has joined #openstack-nova04:33
*** hongbin has quit IRC04:39
*** vishalmanchanda has joined #openstack-nova04:47
*** zhanglong has joined #openstack-nova04:50
*** tbachman has joined #openstack-nova05:18
*** links has joined #openstack-nova05:26
*** sapd1 has quit IRC05:28
*** ociuhandu has joined #openstack-nova05:43
*** songwenping_ has joined #openstack-nova05:44
*** songwenping__ has quit IRC05:47
*** ociuhandu has quit IRC05:47
*** belmoreira has joined #openstack-nova05:56
*** jsuchome has joined #openstack-nova06:24
*** songwenping__ has joined #openstack-nova06:27
*** happyhemant has joined #openstack-nova06:28
*** songwenping_ has quit IRC06:31
*** mvorwerk has joined #openstack-nova06:36
*** mvorwerk_ has quit IRC06:38
*** zhanglong has quit IRC06:43
*** zhanglong has joined #openstack-nova06:45
*** maciejjozefczyk has joined #openstack-nova06:46
*** tetsuro has joined #openstack-nova06:51
*** tetsuro has quit IRC07:01
*** rcernin has quit IRC07:02
*** sapd1 has joined #openstack-nova07:03
*** zhanglong has quit IRC07:03
*** rcernin has joined #openstack-nova07:03
*** songwenping_ has joined #openstack-nova07:04
*** maciejjozefczyk has quit IRC07:05
*** slaweq has joined #openstack-nova07:06
*** zhanglong has joined #openstack-nova07:07
*** songwenping__ has quit IRC07:07
*** tesseract has joined #openstack-nova07:07
*** slaweq has quit IRC07:11
*** kashyap has joined #openstack-nova07:11
*** dklyle has quit IRC07:13
*** zhanglong has quit IRC07:15
*** slaweq has joined #openstack-nova07:16
*** zhanglong has joined #openstack-nova07:18
*** lpetrut has joined #openstack-nova07:23
*** maciejjozefczyk has joined #openstack-nova07:24
*** sapd1 has quit IRC07:38
*** hongbin has joined #openstack-nova07:39
*** tosky has joined #openstack-nova07:40
*** tetsuro has joined #openstack-nova07:42
*** hongbin has quit IRC07:43
openstackgerritLuigi Toscano proposed openstack/python-novaclient master: zuul functional job: drop the custom playbooks  https://review.opendev.org/74664907:44
*** xek has joined #openstack-nova07:51
*** tetsuro has quit IRC08:00
*** songwenping__ has joined #openstack-nova08:01
*** tetsuro has joined #openstack-nova08:01
*** songwenping_ has quit IRC08:04
*** tetsuro has quit IRC08:06
*** ociuhandu has joined #openstack-nova08:06
*** ociuhandu has quit IRC08:11
*** sapd1 has joined #openstack-nova08:23
*** songwenping_ has joined #openstack-nova08:24
*** songwenping__ has quit IRC08:27
*** rcernin has quit IRC08:34
*** zhanglong has quit IRC08:34
*** zhanglong has joined #openstack-nova08:34
*** martinkennelly has joined #openstack-nova08:35
*** priteau has joined #openstack-nova08:36
*** ociuhandu has joined #openstack-nova08:37
*** zhanglong has quit IRC08:38
*** zhanglong has joined #openstack-nova08:40
*** zhanglong has quit IRC08:51
*** tetsuro has joined #openstack-nova08:52
*** k_mouza has joined #openstack-nova08:55
*** zhanglong has joined #openstack-nova08:55
*** tetsuro_ has joined #openstack-nova08:57
*** tetsuro has quit IRC08:59
*** tetsuro has joined #openstack-nova09:03
*** tetsuro_ has quit IRC09:06
*** sapd1 has quit IRC09:09
*** admin0 has joined #openstack-nova09:14
*** songwenping__ has joined #openstack-nova09:14
*** songwenping_ has quit IRC09:17
admin0hi all .. in one new setup with ceph, in the logs, i see the claim successful .. but the instance is just kind of stuck in the spawn/build state .. i don't see any errors in nova .. this is the log from the hypervisor ..  https://gist.github.com/a1git/c14bc9268f429f48169c6c050a605d0c09:20
admin0the instance stays in spawning state for hours like this . and is never created09:21
admin0system is all ubuntu 18.04 ..09:23
*** mvorwerk has quit IRC09:24
*** mvorwerk has joined #openstack-nova09:25
*** mkrai_ has joined #openstack-nova09:26
*** mkrai has quit IRC09:29
*** brinzhang_ has joined #openstack-nova09:34
*** songwenping_ has joined #openstack-nova09:34
lyarwoodadmin0: Aug 18 04:09:12 h3 nova-compute[75006]: 2020-08-18 04:09:12.789 75006 DEBUG oslo_concurrency.processutils [req-61d78385-6bb2-4af2-aea9-f0942ef497b7 1be712988728425ea0060d3bf7ed7a1e 594a25f5acc346518cd6340ec8c36945 - default default] Running cmd (subprocess): rbd import --pool vms /var/lib/nova/instances/_base/09:35
lyarwood85e55585a75fa020bfd46e25ab7aad9c7d360a41 fd14724b-8e52-49c9-b906-9f9a5ab5a651_disk --image-format=2 --id cinder --conf /etc/ceph/ceph.conf execute /openstack/venvs/nova-20.1.4/lib/python3.6/site-packages/oslo_concurrency/processutils.py:09:35
*** jangutter has joined #openstack-nova09:35
lyarwood^ that's where it appears to get stuck09:35
lyarwoodimporting the image into the local imagecache09:35
lyarwoodthe env isn't using rbd for images FWIW, I'm not sure if you assume it is09:36
lyarwoodah wait it is09:36
*** tetsuro has quit IRC09:36
lyarwoodso it's importing into rbd from the imagecache09:37
*** tetsuro has joined #openstack-nova09:37
*** brinzhang0 has quit IRC09:37
admin0and the files are present in base09:37
lyarwoodstupid question but how big is /var/lib/nova/instances/_base/85e55585a75fa020bfd46e25ab7aad9c7d360a4109:37
*** songwenping__ has quit IRC09:37
admin01.1G09:38
*** martinkennelly has quit IRC09:38
lyarwoodso nothing huge09:38
*** jangutter_ has quit IRC09:38
admin0nothing .. i even tried it with a cirros image . stuck in the same way09:38
*** tetsuro has quit IRC09:39
*** tetsuro has joined #openstack-nova09:39
lyarwoodadmin0: is glance using ceph as well?09:39
admin0it is09:39
admin0i have no issues uploading images to glance ..   which i validated09:39
*** songwenping__ has joined #openstack-nova09:39
lyarwoodadmin0: try converting this image or the cirros image to RAW from qcow209:39
admin0creating volumes is also having no issues09:39
admin0hmm..09:40
admin0ok .. let me try that way09:40
admin0i found a snippet : https://gist.github.com/winggundamth/7fe69a99ffcd1dfd14471101497b140709:40
lyarwoodyup so with RAW we can clone directly between rbd pools09:41
lyarwoodwith qcow2 in stable/train that you appear to be using we download the image to the compute and upload to the vms pool09:41
*** songwenping_ has quit IRC09:42
openstackgerritMamduh proposed openstack/os-vif master: Update linux_net - Refactor code of linux_net to more cleaner and increase performace  https://review.opendev.org/74667309:46
*** tetsuro has quit IRC09:48
*** Luzi has joined #openstack-nova09:53
admin0lyarwood, https://gist.githubusercontent.com/a1git/78adef462d99eb88829006f36b988f6b/raw/03ffff0d17a94200f6ce4d2594c76f9c2f130890/gistfile1.txt   -- this is using a 112mb cirros raw image09:54
admin0still in spawning state09:54
admin0i can confirm that from this server, if i copy a command like ceph df --format=json --id cinder --conf /etc/ceph/ceph.conf .. it works09:56
*** tkajinam has quit IRC09:58
lyarwoodadmin0: looks like the final thing it's doing again is a rbd import09:59
lyarwoodadmin0: Aug 18 05:47:41 h1 nova-compute[74659]: 2020-08-18 05:47:41.882 74659 DEBUG oslo_concurrency.processutils [req-f72a63da-329e-45c4-b4dc-7728bb8d2b93 1be712988728425ea0060d3bf7ed7a1e 594a25f5acc346518cd6340ec8c36945 - default default] Running cmd (subprocess): rbd import --pool vms /var/lib/nova/instances/9ea7d409:59
lyarwoodadmin0: I think that's the config drive10:00
lyarwoodhmmm odd10:00
admin0so removing config drive will fix this ?10:00
admin0i can give that a try also10:01
lyarwoodwell it will allow an instance to spawn but there's an underlying issue with rbd import to address10:01
lyarwoodbtw to keep the log snippets sane try grepping for the request-id listed in `openstack server event list $instance` next time10:03
lyarwoodit should limit the logs you need to look at to that instance and request to spawn etc10:04
lyarwoodbrb10:04
*** zhanglong has quit IRC10:05
admin0i only saw delete and create in the event10:07
admin0events: [{'event': 'compute__do_build_and_run_instance', 'start_time': '2020-08-18T09:47:36.000000', 'finish_time': None, 'result': None, 'traceback': None}]10:09
*** sapd1 has joined #openstack-nova10:14
*** martinkennelly has joined #openstack-nova10:15
*** ociuhandu has quit IRC10:17
*** ociuhandu has joined #openstack-nova10:18
admin0got it10:20
admin0how to grep using that specific instanceid10:20
lyarwoodadmin0: it's the request-id that you want10:28
lyarwoodopenstack server event list $instance | grep create | awk '{ print $2 }'10:28
*** songwenping_ has joined #openstack-nova10:34
*** songwenping__ has quit IRC10:37
admin0i am making change to remove the config drive to retry again10:42
admin0so config drives are not compatible with ceph ?10:42
lyarwoodadmin0: they are but there's something broken in your env with regards to importing disks into rbd10:43
admin0is there any manual command i can try to validate this ?10:43
lyarwoodI'm not sure how you've deployed your env but try using rbd --import as the same user as is running nova-compute10:44
admin0its done using openstack-ansible10:44
admin0so the standard osa install10:45
admin0which has quiet frankly worked well in all past deployments .. but in this new one, it suddenly does not work and i am unable to figure out why10:45
lyarwoodkk I've never actually used OSA but that would be my way forward here, run the command as the same user and in the same place as nova-compute is running it10:46
lyarwoodand then throw on some additional -v flags (or whatever rbd uses) to debug10:46
lyarwoodsean-k-mooney: ^ when you're about10:47
admin0so basically will be to change nova.conf debug = true, grep the subprocess to capture the commands and then run the same commands again ..10:47
admin0i will give it a shot10:47
lyarwoodadmin0: looks like you're already logging in DEBUG, but yeah.10:47
admin0its because it failed here :D10:48
lyarwoodadmin0: you just need to be sure you're running the commands under the same user and path (assuming OSA isn't deploying containers?)10:48
admin0it is not10:48
lyarwoodkk10:48
admin0the containers are only in the controllers . but as far as i have seen the api does its job and its the hypervisor where its stuck10:48
lyarwoodyeah it's just the rbd import that's borked10:48
admin0is this the file which is the source of all these commands? https://opendev.org/openstack/nova/src/branch/master/nova/virt/libvirt/storage/rbd_utils.py10:52
lyarwoodadmin0: https://opendev.org/openstack/nova/src/branch/stable/train/nova/virt/libvirt/storage/rbd_utils.py for the version you are running10:58
*** jangutter_ has joined #openstack-nova10:59
admin0i removed the configdrive option .. checking once more10:59
admin0and all images are raw11:00
admin0lyarwood, it worked11:02
admin0finally its active now11:02
admin0removing config drive did the trick11:02
*** jangutter has quit IRC11:02
admin0so if ceph is being used ( and cinder/glance/nova ) is in the same cluster, its recommended to use raw instead of qcow2 ?11:03
lyarwoodYeah RAW instead of qcow2 to avoid the download and import11:04
lyarwoodbut for config drives you can't avoid it, but they should be small and quick enough to import that it shouldn't be an issue11:04
*** mkrai_ has quit IRC11:05
lyarwoodmelwitt / johnthetubaguy / dansmith ; https://review.opendev.org/#/c/699291/ - thoughts on finally landing this from mriedem?11:07
jsuchomelyarwood: hi, me again ... any chance for that job testing direct-rbd-download?11:08
lyarwoodjsuchome: yup still on my list11:09
admin0so 2 observations .. if i use a qcow image, they do not boot at all .. if i change to raw images, they show active in the horizon, and i can see the console, but they are stuck on boot "booting from hard drive .. "11:18
lyarwoodadmin0: and these are cirros images?11:20
admin0yes11:20
admin0one is bionic, but the output is kind of similar11:20
admin0they are stuck on boot11:20
lyarwoodadmin0: that smells like there's something really wrong with your ceph env then11:21
lyarwoodjsuchome: I think I'm going to try to test your change locally first instead of insisting on a job to exercise it11:22
lyarwoodjsuchome: so I don't hold this up any longer11:22
jsuchomecool11:23
*** songwenping__ has joined #openstack-nova11:24
* lyarwood deploys devstack with ENABLE_CEPH_NOVA=false11:25
lyarwoodthat I expect to fail tbh11:25
lyarwoodas the rbd devstack plugin is held togther with tape and chewing gum11:25
*** songwenping_ has quit IRC11:27
openstackgerritLee Yarwood proposed openstack/nova master: [goal] Prepare for job migration to Ubuntu Focal (20.04)  https://review.opendev.org/73812611:28
*** brinzhang0 has joined #openstack-nova11:34
*** brinzhang_ has quit IRC11:36
*** maciejjozefczyk has quit IRC11:45
*** maciejjozefczyk has joined #openstack-nova11:46
*** raildo has joined #openstack-nova11:48
mnaserlyarwood: we actually use it quite often for CI and it's been running quite OK for us12:01
mnaserhappy to help clean it up if need be12:01
*** dave-mccowan has joined #openstack-nova12:01
lyarwoodmnaser: yeah apologies I didn't mean anything by that just that it has failed with a few non-standard deployment configs I've given it in the past :)12:03
lyarwoodthis however worked AFAICT12:03
mnaserOh yeah no i agree that is is held together by tape though12:04
mnaserI think it deploys Ceph 12.x12:04
mnaser3 major releases behind12:04
mnaserWhich isn’t ideal but eh, I don’t think anyone would like it if i made it implemented the latest release using cephadm which requires docker :p12:05
lyarwoodthere appears to be some logic for that already in the plugin iirc12:05
lyarwoodbut not for the latest and greatest release12:06
lyarwoodjust the container part12:06
* lyarwood needs to drop offline, back later \o12:06
*** songwenping_ has joined #openstack-nova12:14
*** songwenping__ has quit IRC12:18
*** zzzeek has quit IRC12:19
*** zzzeek has joined #openstack-nova12:23
*** songwenping__ has joined #openstack-nova12:39
*** songwenping_ has quit IRC12:42
*** psachin has quit IRC12:57
stephenfinsean-k-mooney: Have you any idea what sets this field during live migration? https://github.com/openstack/nova/blob/master/nova/objects/migrate_data.py#L4113:06
*** priteau has quit IRC13:06
kashyapHe should definitely know ... in the not-too-distant past I've seen Sean talk about VIF types.13:09
sean-k-mooneyoh hi one sec13:11
sean-k-mooneystephenfin: yes i responed in the code review13:11
stephenfincode review?13:11
sean-k-mooneyits set in the conducrot live migration task13:11
sean-k-mooneythere is a patch up related to a bug13:12
sean-k-mooneyi assume you were reviewing https://review.opendev.org/#/c/742180/13:12
sean-k-mooneyfor https://bugs.launchpad.net/nova/+bug/188839513:12
openstackLaunchpad bug 1888395 in OpenStack Compute (nova) "shared live migration of a vm with a vif is broken in train" [Undecided,In progress] - Assigned to Kirill Egorov (kegorov-progmaticlab)13:12
sean-k-mooneythe current patch makes incorrect assumtions13:12
stephenfinah, no, I'm trying to add functional tests for SR-IOV live migration13:12
sean-k-mooneyah ok13:12
sean-k-mooneystephenfin: https://github.com/openstack/nova/blob/d4c857dfcb1ccfa5410de55671e69c722bbc990e/nova/conductor/tasks/live_migrate.py#L377-L38313:13
*** brinzhang_ has joined #openstack-nova13:13
sean-k-mooneywe create new vifs that have the old vif object inside them then we populate the vif_type form the dest port binding13:13
sean-k-mooneystephenfin: so its set here https://github.com/openstack/nova/blob/d4c857dfcb1ccfa5410de55671e69c722bbc990e/nova/conductor/tasks/live_migrate.py#L433-L43613:14
sean-k-mooneybut since that uses setattr you would find it with grep13:14
stephenfinsetattr13:14
stephenfingfdi13:14
stephenfinsean-k-mooney++ Thanks. Try as I may, I couldn't find that /o\13:15
sean-k-mooneythree are only 4 fileds you could replace that with 4 lines13:15
stephenfinI think I might. setattr kills discoverability13:15
sean-k-mooneyif there were more filed i might argue for settattr but its just vnic_type, vif_type, profile and details13:16
*** brinzhang0 has quit IRC13:17
sean-k-mooneythats not going to change without a neutron api change and a new microversion which will require a nova code change to use anyway13:17
*** sapd1 has quit IRC13:20
sean-k-mooneystephenfin: if your going to refactor it might be nice to drop this too https://github.com/openstack/nova/blob/d4c857dfcb1ccfa5410de55671e69c722bbc990e/nova/conductor/tasks/live_migrate.py#L372-L37913:20
stephenfinagreed13:20
sean-k-mooneylyarwood: sorry you pinged me eairlier but i didnt have irc up on my monitor this morning. did you figure it out?13:26
sean-k-mooneyoh im not connected downstream either i must not have connect to the vpn13:26
*** mvorwerk has quit IRC13:27
*** mvorwerk has joined #openstack-nova13:29
*** Luzi has quit IRC13:34
*** hongbin has joined #openstack-nova13:35
*** nweinber has joined #openstack-nova13:39
sean-k-mooneystephenfin: https://review.opendev.org/#/c/728037/6 is in merge conflict so ill adress your nit and ping you to re review once done.13:41
sean-k-mooneyif im rebasing i might as well fix them now13:42
stephenfinsean-k-mooney: cool13:44
stephenfinsean-k-mooney: another question for you. The NeutronFixture isn't returning a vif_type key in the response to 'POST /v2.0/ports/{port_id}/bindings'13:45
stephenfinIs that neutron's responsibility? Feel free to tell me to RTFM (I don't have an SR-IOV deployment to hand to experiment :()13:45
sean-k-mooneyyes the vif type can technicaly change on a migration but the vif type is set by the ml2 driver when it binds the port13:46
sean-k-mooneystephenfin: can you show me the line13:46
stephenfinThis piece you linked earlier https://github.com/openstack/nova/blob/d4c857dfcb1ccfa5410de55671e69c722bbc990e/nova/conductor/tasks/live_migrate.py#L433-L43613:47
sean-k-mooneyi ment the line in the fixture13:47
sean-k-mooneyi wanted to see what you have acess too13:47
stephenfinOhh13:47
stephenfinSec while I figure that out13:47
sean-k-mooneyyou could copy it form the soruce vif13:47
sean-k-mooneyin the migrate vif objects13:48
sean-k-mooneyit can change but only if your migrating from like linux bridge hosts to ovs13:48
sean-k-mooneyfor sriov it should not change although its techincially possible13:48
stephenfinsean-k-mooney: https://github.com/openstack/nova/blob/master/nova/tests/fixtures.py#L1659-L167613:48
stephenfinThat's what the fixture is doing. It's rather dumb13:49
sean-k-mooneyif you went from say a hardware offloaded ovs to sriov host13:49
stephenfinOkay, I have the port_id so I can use the stored details13:49
sean-k-mooney ya it does jus tthe minium it needed too13:49
stephenfinwith a big note saying that stub doesn't allow you to switch backends13:49
stephenfinif that sounds reasonable13:49
sean-k-mooney self._port_bindings[port_id][host]13:49
*** tkajinam has joined #openstack-nova13:50
sean-k-mooneyso you you likely can just look it up  assuming [host] also has another entry for the souce node13:50
sean-k-mooneyso loop over self._port_bindings[port_id] and fine the one where the key is not euqal to host13:51
sean-k-mooneythen grab the info from that13:51
*** k_mouza has quit IRC13:51
stephenfinGood call13:51
* stephenfin tries that13:51
*** k_mouza has joined #openstack-nova13:53
sean-k-mooneythat wont always work but it should work in the migration case13:53
sean-k-mooneyso you will need to guard it with an if and only do it if the vif_type is not in the binding data i guess13:54
sean-k-mooneyactully it might work because reasons.13:56
sean-k-mooneyhttps://github.com/openstack/nova/blob/master/nova/tests/fixtures.py#L1573 that is where the vif_type is currently coming form in the fixture13:56
sean-k-mooneywell i guess htere are other places too13:57
sean-k-mooneylike here https://github.com/openstack/nova/blob/master/nova/tests/fixtures.py#L135713:57
sean-k-mooneywe build new ports form port 2 ^ https://github.com/openstack/nova/blob/master/nova/tests/fixtures.py#L1836-L183713:58
sean-k-mooneythe current sriov_port does not have the vif_type set13:59
sean-k-mooneyhttps://github.com/openstack/nova/blob/master/nova/tests/fixtures.py#L1448-L146913:59
*** mlavalle has joined #openstack-nova14:01
stephenfinturns out we don't currently set that self._port_bindings field for the first binding14:02
stephenfinbig TODOs from gibi14:02
* stephenfin goes to close off that TODO first14:02
sean-k-mooneyactully looking at the neutron code the VIF_TYPE for sriov depned on the vnic_type14:03
sean-k-mooneyhttps://github.com/openstack/neutron/blob/master/neutron/plugins/ml2/drivers/mech_sriov/mech_driver/mech_driver.py#L81-L8514:03
luyaostephenfin: I split the vpmem enhencement patch to three and polish them with Alex's help, I'll appreciate that you review it again if you have time, thanks.  https://review.opendev.org/#/c/714653/14:03
stephenfinluyao: Thanks for the reminder. I'll try get to that this evening or tomorrow :)14:04
luyaostephenfin: Thank you in advance. :)14:04
sean-k-mooneystephenfin: so its "hostdev_physical" for PFs and "hw_veb" for everyting else14:04
sean-k-mooneyVIF_TYPE_MACVTAP = 'macvtap' is not used for sriov its for the macvtap backend14:05
sean-k-mooneyso you can ignore that14:05
*** bbowen has joined #openstack-nova14:08
*** bbowen_ has quit IRC14:10
openstackgerritsean mooney proposed openstack/os-vif master: update tox envs and support pdf docs  https://review.opendev.org/72803714:13
*** brinzhang0 has joined #openstack-nova14:19
*** davee_ has joined #openstack-nova14:20
*** brinzhang_ has quit IRC14:21
*** davee_ has quit IRC14:22
*** songwenping_ has joined #openstack-nova14:24
*** songwenping__ has quit IRC14:27
*** bbowen_ has joined #openstack-nova14:28
*** bbowen has quit IRC14:29
*** mvorwerk has quit IRC14:33
*** mvorwerk has joined #openstack-nova14:34
iurygregoryhey nova folks =) in ironic we are trying to create specific flavors (according to the boot_option that is set on capabilities) and I'm facing some problems with No valid host since nova filters doesn't match https://zuul.opendev.org/t/openstack/build/65e4cb538b7149d48f0942e8639b9bb3/log/controller/logs/screen-n-sch.txt#864 any ideas on how I can solve this? =)14:35
iurygregoryI have the feeling I need to do something on tempest to make it work, https://review.opendev.org/741445 this is the patch I have14:35
iurygregoryand this one https://review.opendev.org/741886 tests the changes from tempest14:36
stephenfinsean-k-mooney: that's correct XML for an SR-IOV VIF, right? http://paste.openstack.org/show/796916/14:39
*** links has quit IRC14:42
*** dklyle has joined #openstack-nova14:44
*** songwenping__ has joined #openstack-nova14:44
*** ratailor has quit IRC14:45
sean-k-mooneystephenfin: for vnic_type=direct yes that is the minimal xml although for a flat network14:46
sean-k-mooneystephenfin: you can just create an instance of the config object and get it to render the xml if you want14:47
*** songwenping_ has quit IRC14:48
*** lpetrut has quit IRC14:48
stephenfinSo how is this supposed to work? https://github.com/openstack/nova/blob/master/nova/virt/libvirt/migration.py#L37814:48
stephenfininterface_dev is the lxml.etree representation of http://paste.openstack.org/show/796916/14:49
stephenfinso '.find('address') should read '.find('source/address')' (or whatever the correct incantation is), no?14:50
* stephenfin assumes not to say this works elsewhere14:50
sean-k-mooneystephenfin: let me check the libvirt docs14:50
sean-k-mooney <interface type='bridge'>14:51
sean-k-mooney    <source bridge='br1'/>14:51
sean-k-mooney    <target dev='vnet7'/>14:51
sean-k-mooney    <mac address="00:11:22:33:44:55"/>14:51
sean-k-mooney  </interface>14:51
sean-k-mooneyaddrss is in the mac eleemnt for linux bridge too14:51
sean-k-mooneyoh that is proably looking up the device by pci adress?14:53
stephenfinyeah, this is PCI address14:53
sean-k-mooneythe target pci adress?14:53
sean-k-mooneysomething like this  <address type='pci' domain='0' bus='0' slot='4' function='0' multifunction='on'/>14:54
stephenfinyup14:54
stephenfinThe comment says "Save off the hw address and MTU presented to the guest since that can't change during live migration."14:54
stephenfinbut the place it's getting the hw address from seems wrong :-\14:54
sean-k-mooneythe hardware adress normaly means mac address14:54
sean-k-mooneyinterface_nodes = xml_doc.findall('./devices/interface')14:55
sean-k-mooneyso ya that looking for an address element in the interface14:55
sean-k-mooneybut they could be saving the guest pci address14:55
sean-k-mooneythis is populated by libvirt14:55
sean-k-mooneylet me check a vm quickly14:56
stephenfinah, so there are multiple address. Okay, that makes sense.14:56
stephenfin*address elements14:56
sean-k-mooneyyes there is a top level one for the target adress to present the device in the guest14:56
sean-k-mooneyand then a sub elemen inside the source element which is the host address14:56
stephenfinand parsing code for it here https://github.com/openstack/nova/blob/master/nova/virt/libvirt/config.py#L1891-L189314:57
*** mkrai has joined #openstack-nova14:57
stephenfinso bad mocks again14:57
* stephenfin keeps digging14:57
sean-k-mooneyhttp://paste.openstack.org/show/796917/14:57
*** belmoreira has quit IRC14:57
sean-k-mooneythat is what an ovs interface looks like14:57
sean-k-mooneyif needed i can stack my sriov host but it will looke more or less the same14:58
sean-k-mooneyits getting the element as line 1114:58
stephenfinsean-k-mooney++ ta14:59
stephenfindefinitely bad mocks so14:59
sean-k-mooneystephenfin: the xml will be different depeing on what vnic type your dealing with by the way15:00
sean-k-mooneyvnic_type=direct use <interface type=hostdev> with a source address and top level address15:01
sean-k-mooneyvnic_type=macvtap use <interface type=direct> with a source netdev and top level address15:01
stephenfinthere's our incomplete mock https://github.com/openstack/nova/blob/master/nova/tests/unit/virt/libvirt/fakelibvirt.py#L1132-L114015:01
sean-k-mooneyvnic_type=direct-physical uses <hostdev>  with two addresses15:02
sean-k-mooneystephenfin: well its not incomplete15:02
sean-k-mooneystephenfin: we dont set the adress elements15:02
sean-k-mooneystephenfin: libvirt adds them15:02
sean-k-mooneyso for unit tests that actully correct15:03
stephenfinFair15:03
sean-k-mooneythat said you can poroably just add the address element and increment the pci address for each device15:03
sean-k-mooneyi dont think that matters too much15:04
openstackgerritLuigi Toscano proposed openstack/python-novaclient master: zuul functional job: drop the custom playbooks  https://review.opendev.org/74664915:04
*** priteau has joined #openstack-nova15:04
sean-k-mooneystephenfin: if it helps with the functial tests and its not too intrusive you might as well make it emulate the populated behvior we see when w read it back15:05
*** vishalmanchanda has quit IRC15:06
*** mkrai has quit IRC15:10
*** mkrai_ has joined #openstack-nova15:10
*** jangutter_ is now known as jangutter15:12
*** sapd1 has joined #openstack-nova15:12
*** gyee has joined #openstack-nova15:16
*** lpetrut has joined #openstack-nova15:23
*** mvorwerk has quit IRC15:24
mnaserlyarwood: i jinxed it, it's broken -- python3-rbd : Depends: librbd1 (>= 14.2.11-1bionic) but 12.2.13-0ubuntu0.18.04.2 is to be installed15:32
*** songwenping_ has joined #openstack-nova15:33
*** songwenping_ has quit IRC15:34
*** songwenping_ has joined #openstack-nova15:35
*** songwenping__ has quit IRC15:37
*** hongbin has quit IRC15:40
*** songwenping_ has quit IRC15:43
*** songwenping_ has joined #openstack-nova15:44
*** brinzhang_ has joined #openstack-nova15:48
*** songwenping__ has joined #openstack-nova15:48
*** songwenping_ has quit IRC15:51
*** brinzhang0 has quit IRC15:51
sean-k-mooneystephenfin: just passed zull https://review.opendev.org/#/c/728037/ when you have time to re+215:54
stephenfinsean-k-mooney: done16:14
*** tesseract has quit IRC16:19
sean-k-mooneystephenfin: thanks :)16:30
*** mkrai_ has quit IRC16:34
*** brinzhang0 has joined #openstack-nova16:39
*** brinzhang_ has quit IRC16:42
openstackgerritDouglas Mendizábal proposed openstack/nova master: Fix indentation nits  https://review.opendev.org/74676516:44
*** sapd1 has quit IRC16:54
*** ociuhandu has quit IRC17:03
*** k_mouza has quit IRC17:04
*** songwenping_ has joined #openstack-nova17:14
*** mkrai has joined #openstack-nova17:15
*** songwenping__ has quit IRC17:17
*** ociuhandu has joined #openstack-nova17:17
*** ociuhandu has quit IRC17:22
*** lpetrut has quit IRC17:29
*** songwenping__ has joined #openstack-nova17:34
*** songwenping_ has quit IRC17:38
*** mkrai has quit IRC17:54
openstackgerritMerged openstack/nova master: Remove six.PY2 and six.PY3  https://review.opendev.org/72651217:58
openstackgerritMerged openstack/nova master: Remove six.add_metaclass  https://review.opendev.org/72656517:58
openstackgerritMerged openstack/nova master: Remove six.reraise  https://review.opendev.org/72689817:58
*** mriedem has joined #openstack-nova18:09
*** _erlon_ has joined #openstack-nova18:17
*** songwenping_ has joined #openstack-nova18:24
*** songwenping__ has quit IRC18:27
*** songwenping__ has joined #openstack-nova18:34
*** songwenping_ has quit IRC18:37
*** luyao has quit IRC19:01
*** arne_wiebalck has quit IRC19:01
lyarwoodstephenfin: still about?19:03
lyarwoodstephenfin: https://review.opendev.org/#/c/746210/ can you yank that out of the gate and I'll address the style nits19:03
*** jsuchome has quit IRC19:04
lyarwoodhmm actually if I just push another PS that should do it19:05
*** luyao has joined #openstack-nova19:05
*** arne_wiebalck has joined #openstack-nova19:09
openstackgerritLee Yarwood proposed openstack/nova master: libvirt: Do not reference VIR_ERR_DEVICE_MISSING when libvirt is < v4.1.0  https://review.opendev.org/74621019:10
lyarwood^ melwitt / dansmith / stephenfin ; ^ dropped the newline stuff, something to talk about at the PTG anyway as we've landed lots of this from stephenfin already this cycle :)19:12
lyarwoodthere doesn't appear to be a pep8 rule for this btw so I'm not sure how we'd enforce this either way19:13
*** k_mouza has joined #openstack-nova19:14
*** k_mouza has quit IRC19:17
*** suryasingh has quit IRC19:26
*** mriedem has left #openstack-nova19:37
*** dustinc has joined #openstack-nova19:39
*** mvorwerk has joined #openstack-nova19:41
*** priteau has quit IRC19:44
openstackgerritLee Yarwood proposed openstack/nova master: DNM zuul: Enable [glance]/allowed_direct_url_schemes within nova-ceph-multistore  https://review.opendev.org/74322019:44
*** brinzhang_ has joined #openstack-nova19:44
*** brinzhang0 has quit IRC19:47
*** ociuhandu has joined #openstack-nova19:58
*** ociuhandu has quit IRC20:02
*** martinkennelly has quit IRC20:04
*** martinkennelly has joined #openstack-nova20:04
*** happyhemant has quit IRC20:11
iurygregoryhey nova folks =) in ironic we are trying to create specific flavors (according to the boot_option that is set on capabilities) and I'm facing some problems with No valid host since nova filters doesn't match https://zuul.opendev.org/t/openstack/build/65e4cb538b7149d48f0942e8639b9bb3/log/controller/logs/screen-n-sch.txt#864 any ideas on how I can solve this? =)20:22
iurygregoryI have the feeling I need to do something on tempest to make it work but I have no idea what it would be =) (metadata didn't work) , https://review.opendev.org/74144520:22
iurygregoryin this one https://review.opendev.org/741886 I'm testing the changes from tempest20:23
*** songwenping_ has joined #openstack-nova20:24
*** brinzhang0 has joined #openstack-nova20:25
*** songwenping__ has quit IRC20:26
*** brinzhang_ has quit IRC20:28
*** nweinber has quit IRC20:30
*** songwenping_ has quit IRC20:31
*** songwenping_ has joined #openstack-nova20:32
sean-k-mooneyiurygregory: what did the flavor look like20:51
sean-k-mooney"cpu_arch"="$ironic_node_arch"20:52
sean-k-mooneythat is proaby the issue20:52
sean-k-mooneythat has no prifix so the compute capablity filter will match on it20:52
sean-k-mooneyusing un namespaced extra specs is discuraged20:52
sean-k-mooneyit should not be done if usign either the compute capablity filter or aggreate instance extra specs filter20:53
sean-k-mooneyiurygregory: did you add the capablites to the host. can you point to them in any of the logs20:55
sean-k-mooneyit just looks like they dont match but there is very little to go on20:56
*** martinkennelly has quit IRC20:57
*** mvorwerk has quit IRC20:59
openstackgerritDouglas Mendizábal proposed openstack/nova stable/train: Allow TLS ciphers/protocols to be configurable for console proxies  https://review.opendev.org/74679821:06
iurygregoryshummm21:12
iurygregorylet me check here sean-k-mooney21:12
*** _erlon_ has quit IRC21:12
iurygregorywe are creating the flavors in https://review.opendev.org/#/c/741886/9/devstack/lib/ironic21:12
sean-k-mooneyyep but where are you setting the capablities on the compute nodes21:13
sean-k-mooneyi assume that is done via ironic21:13
iurygregorythe additional thing is that now our flavors will have the capabilities:boot_option=local/netboot21:13
iurygregorywe also set in the ironic node21:13
sean-k-mooneyright which will enable the computecablities filter21:13
iurygregoryso it would match21:13
sean-k-mooneycan you show in the logs that they are being set21:14
*** raildo has quit IRC21:14
sean-k-mooneyhttps://zuul.opendev.org/t/openstack/build/65e4cb538b7149d48f0942e8639b9bb3/log/controller/logs/devstacklog.txt#3084721:16
openstackgerritDouglas Mendizábal proposed openstack/nova stable/stein: Allow TLS ciphers/protocols to be configurable for console proxies  https://review.opendev.org/74680021:16
sean-k-mooneyiurygregory: uefi is beeing set21:16
iurygregorysean-k-mooney, this job is with uefi21:17
sean-k-mooneyiurygregory: but i dont see the boot_option21:17
iurygregoryoh O.o21:17
sean-k-mooneyhttps://zuul.opendev.org/t/openstack/build/65e4cb538b7149d48f0942e8639b9bb3/log/controller/logs/devstacklog.txt#3110121:18
*** rcernin has joined #openstack-nova21:18
sean-k-mooneythat is the only use of capabilities:boot_option=netboot21:18
iurygregoryoh it's because tempest was suposed to update ...21:19
iurygregorywith the patch https://review.opendev.org/#/c/741445/23/ironic_tempest_plugin/tests/scenario/baremetal_manager.py21:19
iurygregoryI thought this was working o.o21:19
sean-k-mooneythe ironic_tempest_plugin was installed form pip21:20
sean-k-mooneyhttps://zuul.opendev.org/t/openstack/build/65e4cb538b7149d48f0942e8639b9bb3/log/controller/logs/devstacklog.txt#32985-3298821:20
sean-k-mooneyi think21:20
sean-k-mooneyactully maybe not21:21
sean-k-mooneyiurygregory: what repo is this job defined in21:21
sean-k-mooneyironic?21:21
iurygregorysean-k-mooney, yeah21:21
sean-k-mooneyim wondiering it it listed as a require_project21:22
sean-k-mooneywithiout that depends-on does not work21:22
iurygregoryit does have Depends-On21:22
iurygregoryironic has Depends-On on the ironic-tempest-plugin change21:22
iurygregorytempest https://review.opendev.org/#/c/741445 ironic https://review.opendev.org/#/c/74188621:23
sean-k-mooneyyep but yyou need to add https://github.com/openstack/ironic/blob/master/zuul.d/ironic-jobs.yaml#L13 to the job or it does not work21:23
sean-k-mooneyin this case its there21:23
sean-k-mooneyso this plugin is being instealled in the tempet virtual env21:25
sean-k-mooneyhttps://zuul.opendev.org/t/openstack/build/65e4cb538b7149d48f0942e8639b9bb3/log/controller/logs/devstacklog.txt#32903-3292421:25
iurygregoryyeah, seems like tempest is not updating the node with the boot_option...21:26
sean-k-mooneyi think its correctly installing it  tox -evenv-tempest -- pip install -c /tmp/tempest_u_c_m.37WJbEB0sa /opt/stack/ironic-tempest-plugin21:26
sean-k-mooneythat using the version cloned by zuul21:26
*** xek has quit IRC21:26
sean-k-mooneyiurygregory: have you defiend the required tempest config options21:28
iurygregoryyeah the version installed is correct (at least the logs shows "23adec07f1d5998951fe841cd68989c6d85f7d98 Update basic_ops"21:28
iurygregoryyeah, I checked the config and it does look correct21:28
iurygregoryI double checked with the job-output.txt the flavors are correct21:30
sean-k-mooneyhttps://zuul.opendev.org/t/openstack/build/65e4cb538b7149d48f0942e8639b9bb3/log/controller/logs/tempest_conf.txt#141-14421:30
iurygregoryuuid etc21:30
sean-k-mooneyso ya it looks like they are set correctly21:30
sean-k-mooneyso this is what is ment to set the caps https://review.opendev.org/#/c/741445/23/ironic_tempest_plugin/tests/scenario/baremetal_manager.py@20521:31
sean-k-mooneyhttps://review.opendev.org/#/c/741445/23/ironic_tempest_plugin/tests/scenario/baremetal_manager.py@17421:32
sean-k-mooneyby default it does not set teh caps21:32
iurygregoryyeah, but the test sends True21:33
iurygregoryhttps://review.opendev.org/#/c/741445/23/ironic_tempest_plugin/tests/scenario/test_baremetal_basic_ops.py@20821:33
*** songwenping__ has joined #openstack-nova21:33
sean-k-mooneyyes https://review.opendev.org/#/c/741445/23/ironic_tempest_plugin/tests/scenario/test_baremetal_basic_ops.py@20821:33
iurygregoryat least it should send =)21:34
sean-k-mooneydoes the compute capablity filter requrie all cap to be requested21:35
*** rcernin has quit IRC21:35
iurygregorythis information I don't know =)21:35
sean-k-mooneyi was wonderinf it the default falvor was failing but no21:36
sean-k-mooneythe only failing tests are teh new ones21:36
*** songwenping_ has quit IRC21:36
sean-k-mooneywell actully21:36
sean-k-mooneyhttps://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_65e/741886/9/check/ironic-tempest-ipa-partition-uefi-pxe_ipmitool/65e4cb5/testr_results.html21:36
sean-k-mooneyonly 2 tests failed21:36
iurygregoryyeah21:37
iurygregorythe ones I did the changes and require the new flavor etc =)21:37
sean-k-mooneywell you added https://review.opendev.org/#/c/741445/23/ironic_tempest_plugin/tests/scenario/test_baremetal_basic_ops.py@32021:38
sean-k-mooneywhic passed21:38
sean-k-mooneythat looks like it can use the new code21:38
sean-k-mooneyoh that was skipped21:39
iurygregoryI'm not sure where did you see that test_baremetal_server_ops_wholedisk_image passed o.o21:39
iurygregoryoh =)21:40
sean-k-mooneyim guessing rescure is not enabled or something like that21:40
iurygregorybecause rescue wasn't enable so it will skip21:40
sean-k-mooneyyep21:40
iurygregorywe are trying to make our jobs test partition and wholedisk images at the same time, so we can remove some jobs21:41
sean-k-mooneyso this is wehre the caps are currently configured21:43
sean-k-mooneyhttps://github.com/openstack/ironic/blob/master/devstack/lib/ironic#L2293-L230721:43
sean-k-mooneystaticly in devstack21:43
sean-k-mooneywell it goes on longer21:43
sean-k-mooneyhttps://github.com/openstack/ironic/blob/master/devstack/lib/ironic#L2293-L234421:43
iurygregoryyeah21:45
iurygregorybut tempest should be able to just go and update21:45
iurygregory.-.21:45
sean-k-mooneywell21:45
iurygregoryor nope?21:45
sean-k-mooneyits updating them after it creates the instance21:45
sean-k-mooneyhttps://review.opendev.org/#/c/741445/23/ironic_tempest_plugin/tests/scenario/baremetal_manager.py@19021:46
sean-k-mooneyhttps://review.opendev.org/#/c/741445/23/ironic_tempest_plugin/tests/scenario/baremetal_manager.py@19721:46
sean-k-mooneythe node is not update until 20521:46
sean-k-mooneyso you need to do it before creating the nova instance21:47
sean-k-mooneyso move it to 18321:47
iurygregorywow21:47
sean-k-mooneyam i reading that right21:48
iurygregorythis would be strange21:48
iurygregorysince nova is doing the things with the ironic virt driver21:49
iurygregorythe node is only ready on line 22821:49
sean-k-mooneyhow do you mean21:49
iurygregorywhen nova is enable it uses the nova.virt.ironic https://opendev.org/openstack/nova/src/branch/master/nova/virt/ironic/21:50
sean-k-mooneywell right now everything that happens after 20021:50
sean-k-mooneyis waiting for ironcit to finish proviioning the servier21:50
sean-k-mooneybut you need to add the cabpiltiy before calling nova21:50
iurygregoryhummm21:51
sean-k-mooneyso the cap need to be set on one of the hosts before nova tries to find a node using the flavor21:51
iurygregoryso if I set on devstack/lib/ironic to grab one node and set it will work21:52
sean-k-mooneythis test also cant run in parallel with other tests since its cahanging capablities21:52
*** rcernin has joined #openstack-nova21:52
sean-k-mooneysure it proably will work21:52
sean-k-mooneyto test both case in 1 job you would need two ironic compute nodes21:53
sean-k-mooneythen hard code one to each boot mode21:53
*** brinzhang_ has joined #openstack-nova21:54
iurygregoryyeah21:55
iurygregorytks for the help sean-k-mooney =)21:55
*** brinzhang0 has quit IRC21:56
*** rcernin has quit IRC21:57
*** rcernin has joined #openstack-nova22:09
*** rcernin has quit IRC22:09
*** brinzhang0 has joined #openstack-nova22:14
*** brinzhang_ has quit IRC22:17
*** tosky has quit IRC22:18
*** songwenping_ has joined #openstack-nova22:24
*** songwenping__ has quit IRC22:28
*** rcernin has joined #openstack-nova22:33
*** mlavalle has quit IRC22:33
*** pas-ha has quit IRC22:42
*** csatari has quit IRC22:43
*** knikolla has quit IRC22:43
*** csatari has joined #openstack-nova22:55
*** pas-ha has joined #openstack-nova22:55
*** knikolla has joined #openstack-nova22:56
*** JamesBenson has joined #openstack-nova23:10
*** songwenping__ has joined #openstack-nova23:24
*** songwenping_ has quit IRC23:27
*** slaweq has quit IRC23:33
*** songwenping_ has joined #openstack-nova23:43
*** brinzhang_ has joined #openstack-nova23:44
*** songwenping__ has quit IRC23:46
*** brinzhang0 has quit IRC23:47
« Monday, 2020-08-17 Index Wednesday, 2020-08-19 »

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!