Thursday, 2022-03-03

*** akekane__ is now known as abhishekk05:26
gibidmitriis, sean-k-mooney: I made a fix for the func test pci device issue in https://review.opendev.org/c/openstack/nova/+/829248/4/nova/compute/resource_tracker.py let me know what you think. I'm happy to split that patch up if needed07:30
*** akekane__ is now known as abhishekk07:42
*** ricolin is now known as Guest111207:59
*** ricolin_ is now known as ricolin07:59
*** akekane_ is now known as abhishekk08:11
opendevreviewribaudr proposed openstack/python-novaclient master: Microversion 2.91: Support specifying destination host to unshelve  https://review.opendev.org/c/openstack/python-novaclient/+/83165110:25
opendevreviewOpenStack Release Bot proposed openstack/os-vif stable/yoga: Update .gitreview for stable/yoga  https://review.opendev.org/c/openstack/os-vif/+/83166410:50
opendevreviewOpenStack Release Bot proposed openstack/os-vif stable/yoga: Update TOX_CONSTRAINTS_FILE for stable/yoga  https://review.opendev.org/c/openstack/os-vif/+/83167010:51
opendevreviewOpenStack Release Bot proposed openstack/os-vif master: Update master for stable/yoga  https://review.opendev.org/c/openstack/os-vif/+/83167810:51
opendevreviewOpenStack Release Bot proposed openstack/osc-placement stable/yoga: Update .gitreview for stable/yoga  https://review.opendev.org/c/openstack/osc-placement/+/83169510:51
opendevreviewOpenStack Release Bot proposed openstack/osc-placement stable/yoga: Update TOX_CONSTRAINTS_FILE for stable/yoga  https://review.opendev.org/c/openstack/osc-placement/+/83169810:51
opendevreviewOpenStack Release Bot proposed openstack/osc-placement master: Update master for stable/yoga  https://review.opendev.org/c/openstack/osc-placement/+/83170110:51
opendevreviewOpenStack Release Bot proposed openstack/python-novaclient stable/yoga: Update .gitreview for stable/yoga  https://review.opendev.org/c/openstack/python-novaclient/+/83170410:51
opendevreviewOpenStack Release Bot proposed openstack/python-novaclient stable/yoga: Update TOX_CONSTRAINTS_FILE for stable/yoga  https://review.opendev.org/c/openstack/python-novaclient/+/83170510:51
opendevreviewOpenStack Release Bot proposed openstack/python-novaclient master: Update master for stable/yoga  https://review.opendev.org/c/openstack/python-novaclient/+/83170610:51
*** __ministry1 is now known as __ministry11:05
*** dasm|off is now known as dasm|rover12:08
opendevreviewMerged openstack/os-vif stable/yoga: Update .gitreview for stable/yoga  https://review.opendev.org/c/openstack/os-vif/+/83166412:55
opendevreviewMerged openstack/os-vif stable/yoga: Update TOX_CONSTRAINTS_FILE for stable/yoga  https://review.opendev.org/c/openstack/os-vif/+/83167012:57
opendevreviewribaudr proposed openstack/python-novaclient master: Microversion 2.91: Support specifying destination host to unshelve  https://review.opendev.org/c/openstack/python-novaclient/+/83165113:09
sean-k-mooneyby the way am i the only one who sees this when they run commands13:22
sean-k-mooney/usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead13:22
sean-k-mooney  from cryptography.utils import int_from_bytes13:22
sean-k-mooney/usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead13:22
sean-k-mooney  from cryptography.utils import int_from_bytes13:22
sean-k-mooneyim seing it for nova manage and also osc13:22
sean-k-mooneyi assume secretstorage is managed by uc13:25
sean-k-mooneyi wonder if that has beeen adressed there yet13:26
gibisean-k-mooney: I see that in a fairly recent devstack from master too so I think this is still happening13:54
sean-k-mooneygibi: ack i think this need to be updated in secrtstorage but i dont even know what we use that for14:12
sean-k-mooneylike sure it provides a way for securly storing password 14:13
sean-k-mooneyhttps://pypi.org/project/SecretStorage/14:13
sean-k-mooneybut why woudl we need dbus integratoin for nova-manage14:13
sean-k-mooneyliek we are not going to be pulling things form the gnome keyring14:13
*** ricolin_ is now known as ricolin14:13
sean-k-mooneyim guessing this is really from oslo14:14
sean-k-mooneyactully i dont see it as a direct dep in any openstack porject that woudl make sense14:15
sean-k-mooneyhttps://codesearch.opendev.org/?q=secretstorage&i=nope&literal=nope&files=&excludeFiles=&repos=14:15
sean-k-mooneybased on strace it looks like its coming form keyring14:19
sean-k-mooneyit might be a sideffect fo stevador and the entrypoint scan it does14:21
sean-k-mooneygibi: ok so this is from python-keystoneclient14:26
sean-k-mooneyhttps://opendev.org/openstack/python-keystoneclient/commit/5939541bc771e1205394b05e757d7b23b3aca86214:26
opendevreviewMerged openstack/python-novaclient master: Update master for stable/yoga  https://review.opendev.org/c/openstack/python-novaclient/+/83170614:32
tobias-urdinhow does nova track the available of vgpu resources through placement? Or maybe phrased like: how does nova update placement with availability of vgpu resources with multiple "flavors" when using custom traits?14:32
tobias-urdini've tried to dig through nova scheduler and resource tracker code to understand how it calculates where there is availability14:33
sean-k-mooneytobias-urdin: that has cahnged a littel in the last release or two14:34
sean-k-mooneybut basically you list the mdev type and the partent gpu pci adress in the config14:34
sean-k-mooneythne nova will look at the avaiable count and create a Resouce prover per card you listed with an inventory of VGPU for each rp/card14:35
sean-k-mooneywe recently added support for generic mdevs so now you can use a differnt resouce class if you prefer14:35
sean-k-mooneyso you could have CUSTOM_NVIDIA_LARGE instead of VGPU14:36
sean-k-mooneythe generic mdev support is intendd for things that are not GPUs too14:36
sean-k-mooneyso in terms of capsity if you are not useing any traits request14:37
sean-k-mooneythe compute agent as part of update_aviable_resouces periodic task(and init_host) will read the capasity info form sysfs and translate that into RPs that are child RPs of the comptue node RP with on RP per phsyical gpu card14:38
sean-k-mooneyif you just ask for resouces:CGPU=1 in the flavor 14:38
sean-k-mooneythen we will not filter on any mdev type or trait and will just select host with free VGPU inventory14:39
sean-k-mooneywhich is fine if all your gpus are the same14:39
sean-k-mooneyif you have differnt ones configured you shoudl use traits or the new generic mdev feature instead ot differenciate14:39
sean-k-mooneytobias-urdin: is there anything in particalr that you wanted to know beyond that overview14:40
tobias-urdinso let's say i have two cards 0000:3b:00.0 and 0000:af:00.0 that is NVIDIA A10 cards, I enable VFs on those and get like 30 each let's say on 0000:3b:x.x 0000:af:x.x and assign enabled_mdev_types=nvidia-1, nvidia-2 and [mdev_nvidia-1]/device_addresses=<all VF for 0000:3b:x.x> (and same for 0000:af card) - assign those CUSTOM_NVIDIA_1 and14:41
tobias-urdinCUSTOM_NVIDIA_2 traits, flavors with resources:VGPU=1 and trait:CUSTOM_NVIDIA_1=required etc14:41
tobias-urdinhow is placement populated for all those <computenode>_pci_0000_3b_x_x RPs to know how many it is left when it a 1:1 mapping in that RP per PCI dev addr14:42
* tobias-urdin reading through above14:42
ade_lee_sean-k-mooney, hey -- can you take a look at the failing experimental fips job on https://review.opendev.org/c/openstack/tempest/+/83160714:51
ade_lee_sean-k-mooney, this time its on centos-914:51
ade_lee_sean-k-mooney, dansmith : https://zuul.opendev.org/t/openstack/build/0a4f8346b89f4bbfa92135dbdbf811f914:52
sean-k-mooneyso rescue and temp url14:52
ade_lee_sean-k-mooney, ack - what does that point to?14:54
dansmithcheck the cinder logs?14:54
sean-k-mooneyrescure failde because fo this 14:55
sean-k-mooneyWaiting for libvirt event about the detach of device vdb with device alias virtio-disk1 from instance 7323f68a-b4dc-4630-b4fd-bd7a7f69d4f4 is timed out.14:55
sean-k-mooneyso that looks like the intermiting libvirt volume detach issue14:55
sean-k-mooneyya it is internal error: unable to execute QEMU command 'device_del': Device virtio-disk1 is already in the process of unplug.14:56
sean-k-mooneythat will be fixed by a new qemu soon we hope14:56
dansmithseems like a lot of these centos job fails are qemu/libvirt related14:56
dansmithwhich is pretty disturbing :/14:57
sean-k-mooneyif it makes you feel better those also fail on rhel downstream14:57
dansmithnot really :)14:57
* sean-k-mooney it make me feel worse but who know14:57
gibithe centos jobs are running with newer libvirt and qemu than the ubunut jobs so we see the new failure modes there first14:58
sean-k-mooneycurrntly yes although it prevoulsy was the other way around14:58
sean-k-mooneygibi: dansmith  what is more disturbing to me is this is using the pc machine type14:59
sean-k-mooneynot q3514:59
dansmithhmm14:59
sean-k-mooneygibi: so this is partly realted to the fact that even with the event based case we still retry15:00
sean-k-mooneybut fundimetnally qemu is taking a long time to detach 15:00
sean-k-mooneywhich it shoudl not, the wait for sshable/pingable tempest change might help15:01
sean-k-mooneyif this is because the os is not ready15:01
sean-k-mooneybut this happeing a lot lately15:01
sean-k-mooneywhat do the ObjectTempUrlTest test do 15:02
sean-k-mooneyare they swift related im not familar with them15:02
sean-k-mooney GET https://149.202.163.165:8080/v1/AUTH_ab7063290b7341eeb77f5198d9e09903/tempest-TestContainer-735377846/tempest-TestObject-176829762915:03
sean-k-mooneythat looks like possible swift to me15:03
sean-k-mooneyade_lee_: in anycase the rescue failure does not look fips related15:05
tobias-urdinsean-k-mooney: any input on above?15:07
ade_lee_sean-k-mooney, ack - I didn't think it was, but unfortunately , it is blocking the fips patches.  Do we have any workarounds/possible fixes ?  I'll ask the swift folks about the swift issues.15:09
ade_lee_sean-k-mooney, is there a BZ /launchpad to track these libvirt/qemu issues?15:10
gibisean-k-mooney: yeah, the base case retry is something we can remove when we switch to qemu 6.2 as a minimum15:10
gibi(or something around 6.2 Im not sure)15:10
sean-k-mooneywe proably could make it conditonal on the version before we raise our minium15:15
gibiyeah, good point15:15
sean-k-mooneytobias-urdin: oh i missed your follow ups15:15
sean-k-mooneyill read back one sec15:16
sean-k-mooneyade_lee_: why is it blocking?15:16
sean-k-mooneythe job is non voting right15:16
sean-k-mooneyi think we could proceed with this failure unless it reliably fails every time in the fips job?15:17
ade_lee_sean-k-mooney, fair enough -- maybe what we do then is change the job to be for centos-9 - and then merge it15:18
sean-k-mooneybasically what i woudl hope is when we fix this normally it would be fixed for fips15:19
*** efried1 is now known as efried15:24
ade_lee_sean-k-mooney, ack - I'll update to centos 9  - and then ping for reviews.  do we have any sense of when it will be fixed ? its showing up in glance reviews, cinder etc ..15:26
ade_lee_sean-k-mooney, a BZ will be super helpful so I can track things15:26
sean-k-mooneyso we kind fo do have one for qemu and there are a few cix issues15:30
sean-k-mooneywe dont have a singel one for nova for example because its not really a nova issue15:30
sean-k-mooneywe think that some of the recent bugfixes in qemu and libvirt will help15:30
opendevreviewribaudr proposed openstack/python-novaclient master: Microversion 2.91: Support specifying destination host to unshelve  https://review.opendev.org/c/openstack/python-novaclient/+/83165115:33
tobias-urdinsean-k-mooney: no hurry, let me know when you have a second :)16:19
sean-k-mooneytobias-urdin: sorry im in a meeting downstream which is why i did not respond16:19
opendevreviewMerged openstack/python-novaclient stable/yoga: Update .gitreview for stable/yoga  https://review.opendev.org/c/openstack/python-novaclient/+/83170416:34
opendevreviewMerged openstack/python-novaclient stable/yoga: Update TOX_CONSTRAINTS_FILE for stable/yoga  https://review.opendev.org/c/openstack/python-novaclient/+/83170516:34
* Uggla had a hard time with pep8 "E127 continuation line over-indented for visual indent". :)17:21
Ugglasean-k-mooney, fyi now the unshelve to host is complete with client part as well.17:28
opendevreviewMerged openstack/osc-placement stable/yoga: Update .gitreview for stable/yoga  https://review.opendev.org/c/openstack/osc-placement/+/83169517:56
opendevreviewMerged openstack/osc-placement stable/yoga: Update TOX_CONSTRAINTS_FILE for stable/yoga  https://review.opendev.org/c/openstack/osc-placement/+/83169817:57
sean-k-mooneyUggla: ack ill try and review what you have proably monday18:12
Ugglasean-k-mooney, no hurries I think.18:33
sean-k-mooneyare all patches in https://review.opendev.org/q/topic:bp%252Funshelve-to-host18:51
sean-k-mooneyif so ill add that to my review-list bookmark folder for monday18:51
opendevreviewAde Lee proposed openstack/nova master: Test setting the nova job to centos-9-stream  https://review.opendev.org/c/openstack/nova/+/83184420:13
*** dasm|rover is now known as dasm|off23:16

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!