Monday, 2022-02-07

*** amoralej|off is now known as amoralej07:02
opendevreviewAttila Fazekas proposed openstack/nova master: Note the deleyad address view  https://review.opendev.org/c/openstack/nova/+/82785607:20
*** hemna3 is now known as hemna07:37
yuvalHey is there any special keyword I can reply to my patch to make zuul re-test my patch?08:34
gibiyuval: say: recheck <reason, e.g. the bug number your patch hit>08:36
yuvalgibi: in this chat or on the gerrit gui?08:42
gibiyuval: in gerrit as a reply 08:42
yuvalthanks08:42
opendevreviewMerged openstack/nova master: docs: Follow-ups for cells v2, architecture docs  https://review.opendev.org/c/openstack/nova/+/82733610:27
*** ralonsoh_ is now known as ralonsoh10:51
opendevreviewTobias Urdin proposed openstack/nova master: Update announce self workaround opt description  https://review.opendev.org/c/openstack/nova/+/82682911:25
gibisean-k-mooney: hi! I'm +2 on the off-path networking series now11:34
sean-k-mooneyack that is nice to hear11:35
gibisean-k-mooney: also looked at your healthcheck wip patch and left some notes11:39
sean-k-mooneythanks i need to spend more time on that in general but once i have the basic infra working adding the checks should be fairly quick11:51
gibiyepp12:00
sean-k-mooneyim currenly thinking about what to keep and what to factor out into the resoucetracker class or whatever i end up calling it12:03
sean-k-mooneybut looking a the current manager ther eare clearly 2 types of functiosn and data12:04
gibiyes12:04
gibithat run method makes it an active component, but also it encapsulates some response data12:04
sean-k-mooneyya so run sleep and generating the responce feel like tehy should all be on one class12:05
sean-k-mooneyand the data storage and querying its state shoudl be in anohter12:05
sean-k-mooneygibi: i did not quite understand what you ment by https://review.opendev.org/c/openstack/nova/+/825015/3/nova/healthcheck/manager.py#3112:06
sean-k-mooneyi always split imports into 8 groups , standar lib, external lib, openstack libs, nova  and in each of those 4 i split into imports then from x import12:08
sean-k-mooneywere you saying that i have split it into more groups then you were expecting or something else?12:09
gibisean-k-mooney: I meant that in nova I see 3 groups: stdlib, 3pp lib, repo local import12:10
gibisean-k-mooney: but I have no real problem with more groups12:10
gibiso feel free to ignore that command12:11
gibi*comment12:11
sean-k-mooneygibi: i use the same groups in nova for my other work12:20
sean-k-mooneyhacking allows both i really hate when we mix from and import in the same group12:21
gibiOK, I can adapt to it per file :)12:21
sean-k-mooneyi used to go fix that but now i more or less deal with the impoort confution12:21
sean-k-mooneyi find it really diffuct to parse how  i should order things when you mix the import and from together12:22
sean-k-mooneyi have considered proposing we us automatic import sorting at one point but there are more importnat hills to die on12:22
sean-k-mooneygibi: but effectivly you were suggestign treating openstack libs and third party python libs the same12:24
gibisean-k-mooney: our current nova style do so. I have no hard feelings either way12:25
sean-k-mooneywe are not super consitent with that12:25
sean-k-mooneysometimes we include evently in the standardlib section :)12:25
sean-k-mooneyif people want me to change it i can but ill leave it for now until i have it working end to end12:27
gibileave it :)12:29
opendevreviewStephen Finucane proposed openstack/placement master: db: Update 'select()' calls  https://review.opendev.org/c/openstack/placement/+/80110312:30
opendevreviewStephen Finucane proposed openstack/placement master: db: Remove use of non-integer/slice indices  https://review.opendev.org/c/openstack/placement/+/80110412:30
opendevreviewStephen Finucane proposed openstack/placement master: db: Replace deprecated 'FromClause.select().whereclause' parameter  https://review.opendev.org/c/openstack/placement/+/80110512:30
opendevreviewStephen Finucane proposed openstack/placement master: db: Use explicit transactions  https://review.opendev.org/c/openstack/placement/+/80110612:30
opendevreviewStephen Finucane proposed openstack/placement master: db: Remove unnecessary use of '_mapping'  https://review.opendev.org/c/openstack/placement/+/80110712:30
opendevreviewStephen Finucane proposed openstack/placement master: tox: Enable SQLAlchemy 2.0 warnings  https://review.opendev.org/c/openstack/placement/+/80110812:30
opendevreviewStephen Finucane proposed openstack/placement master: tests: Restore - don't reset - warning filters  https://review.opendev.org/c/openstack/placement/+/82811912:30
sean-k-mooneyhehe speaking of placement ^ gibi  how is you any trait series coming12:31
stephenfinsean-k-mooney: I'd appreciate reviews on that, btw :) It's all very simple stuff but preps us for sqla 2.012:32
stephenfinNeed to find time to finish the equivalent nova series /o\12:32
sean-k-mooneystephenfin: ack i can take a look. im going to try an go thorugh the off path seires this morning maybe this afternoon12:33
*** amoralej is now known as amoralej|lunch12:56
gibisean-k-mooney: melwitt gave +2 on the next patch in the any-traits series 13:01
gibisean-k-mooney: you your eyese are appreciated there too :)13:01
* gibi go and look at the placement sqla series now13:02
bauzasfolks, I'm a bit on and off today, I think I'm impacted by the COVID13:05
* bauzas still awaits for the PCR test result13:06
elodillesohh, bauzas get well soon :S13:07
elodilles(i just wanted to ask for a stable review, but then i won't disturb you with that :S)13:08
gibibauzas: take care!13:08
gibibauzas: if you need I can cover you tomorrow on the weekly meeting13:09
bauzasgibi: no, unfortunately in France if you're impacted but you WFH, they don't give you some offtime13:09
gibithat sounds bad13:10
gibianyhow I'm not bound by the french law so I can still cover you if you want :)13:10
opendevreviewTobias Urdin proposed openstack/nova master: Cleanup old resize instances dir before resize  https://review.opendev.org/c/openstack/nova/+/82786513:16
bauzasfor the moment, I don't have a lot of issues, just some cough, a sore throat and nose blowing13:16
bauzaslike if it was a small rhinitis13:18
bauzasfwiw, my daughter got the same on Tuesday and we saw the positif test on Wed so I tested myself too on Wed morning with a selftest, then an antigenic test on Wed evening, and then a PCR test on Thursday13:19
bauzaseventually a last selftest on Friday13:19
bauzasall of them were negative13:19
bauzasbut yesterday evening, I selftested again (ie. after 5 days of being negative) and eventually it was positive :(13:20
bauzashence the PCR test today13:20
*** dasm|off is now known as dasm13:20
bauzasso, see, maybe testing yourself up to J+4 doesn't work 13:20
bauzasD+4 I mean13:21
gibiyeah13:22
gibiI heard similar stories13:22
bauzasfrom what I read, this is specific to Omicron13:25
bauzasfor Delta, this worked13:25
bauzasbut now, symptoms arrive before the viral charge13:25
bauzasor at the same time13:26
bauzasso, previously, testing yourself at D+3 was verifying your viral charge13:26
bauzaswhile now, your viral charge would only arrive around D+513:28
bauzasthe more people know, the better it will be13:28
sean-k-mooneydmitriis:  -1 for https://review.opendev.org/c/openstack/nova/+/824833/7/nova/network/neutron.py#152713:40
dmitriissean-k-mooney: looking13:40
sean-k-mooneytl;dr we cant assume a PF has a netdev13:41
sean-k-mooneyso you cant assume it has a mac13:41
sean-k-mooneyyour new fucntion will be called for all VFs not just remote managed ones13:42
sean-k-mooneyso you need to not raise13:42
sean-k-mooneyjust dont include the info if it cant be recived13:42
sean-k-mooney*retrived13:42
dmitriissean-k-mooney: yeah, a PF can be something else, much like a PF with a netdev can have non-netdev VFs13:43
dmitriisgood point13:43
sean-k-mooneywe broke this in the past13:44
sean-k-mooneyby always trying to get the netdev name for bandwith qos and i broke it differently in a different bugfix13:44
sean-k-mooneyso we have been bitten by it twice13:45
sean-k-mooneydmitriis: actully just to reinforce that you cant assume all VFs are nics13:46
gibiohh, I knew that we cannot assume that netdev exists but I did not realized that to get the MAC we need the netdev13:46
sean-k-mooneythat is what we broke with the init qos support we broke vf for qat13:47
dmitriissean-k-mooney: yeah, Bluefield2 devs have a management VF, for example, which is not a netdev13:47
sean-k-mooneyi can triple check how we do the lookup but in general the mac wont be in /sys if the netdev is not created13:47
sean-k-mooneygibi: we do the lookup like this https://github.com/openstack/nova/blob/master/nova/pci/utils.py#L173-L19113:50
sean-k-mooneywhich i woudl expect to fail sicne we are geting the netdev path13:50
gibiyeah you are correct13:50
sean-k-mooneydmitriis: anyway hopefully that will be a minor change jsut dont populate the filed if the data is not available13:51
sean-k-mooneyim going to continue reviewing the rest of the series in the mean time13:51
dmitriissean-k-mooney: Thinking about it now, I assumed that since we are in the neutron-related code path and got a PCI device allocated which is a netdev VF, I am safe to assume the PF is also a netdev. However, this may not be true all the time.13:51
dmitriissean-k-mooney: ack13:51
sean-k-mooneydmitriis: ya that was the assumtion that bit us for the calvim thunderx13:52
sean-k-mooneyi even hat that in a comment13:52
sean-k-mooneyhttps://review.opendev.org/c/openstack/nova/+/777679/3/nova/virt/libvirt/host.py13:52
sean-k-mooneyso no we cant assume that13:52
dmitriissean-k-mooney: yes, I recall fixing something for that comment but I definitely missed this part13:53
*** amoralej|lunch is now known as amoralej13:53
opendevreviewIlya Popov proposed openstack/nova master: Fix to implement 'pack' or 'spread' VM's NUMA cells  https://review.opendev.org/c/openstack/nova/+/80564913:54
gibimelwitt: fyi the placement perfload job (non voting) probably broken since the consumer types feature https://zuul.opendev.org/t/openstack/build/0d0e850ea9de47e8bf30ad1dbd1d3b91/log/job-output.txt#1163 14:05
gibimelwitt: I think we don't really look at that job so I'm not sure we want to fix it14:05
chateaulavgibi: for the hanging comment about the tempest testing, you mean in addition to an additional ci job, correct?14:08
chateaulavfor https://review.opendev.org/c/openstack/nova/+/822053, and thanks for the continued reviews!14:08
gibichateaulav: yeah so I think a separate job that test some emulation (maybe aarch64) with tempest. But I guess some of the existing tempest test will not work with emulation so we need a trimmed down test list14:08
chateaulavok, makes sense for the most part. first time messing with tempest, but ill get something together14:11
gibichateaulav: thanks14:13
sean-k-mooneygibi: chateaulav by the way have either of you got a working vm image for this testing. i think the centos9s cloud image will booth properly on arm (it did on my mac nativly)  but i could not get cirros to work14:17
sean-k-mooneythe centos image is huge in comparison at almost 700mb14:17
gibisean-k-mooney: I did not tired14:18
sean-k-mooneyok well that might be one of the chanllages with doing the arm testing via emulation but it is likely fine14:18
sean-k-mooneywe can have devstack download addtional images14:19
sean-k-mooneybut i was having trouble geting cirrus to boot with uefi in genreal and that is required for aarch6414:19
sean-k-mooneyso uefi on x86 and aarch64 did not seam to work with cirros14:19
sean-k-mooneyi got an error with the alingment or either the filesystme or uefi firmware in the console and  then no output14:20
sean-k-mooneychateaulav: what have you been using to test with qemu directly?14:20
chateaulavsean-k-mooney: yeah havent tried the cirros on aarch hardware, but on x86 works fine.14:20
chateaulavhave code running in a deployment ostack env14:20
sean-k-mooneydid you do an install form iso or something else14:21
sean-k-mooneychateaulav: ok so you did arch64 via qemu on x86 with the aarch64 cirros image14:21
chateaulavhttp://download.cirros-cloud.net/0.5.2/cirros-0.5.2-aarch64-disk.img14:21
chateaulavsean-k-mooney: qcow14:21
chateaulavyes14:21
sean-k-mooneyya i tried that and it woudl not boot on may ubuntu aarch64 vm on my macbook air14:22
sean-k-mooneyi was trying to see if i could use that as a arm dev env14:22
sean-k-mooneychateaulav: did you put anything special in the glacne metadtaa14:23
chateaulavhw_emulation_architecture='aarch64', hw_firmware_type='uefi', hw_machine_type='virt'14:23
sean-k-mooneyya ok that is what i was expecting14:23
sean-k-mooneythanks14:23
sean-k-mooneyi can give it a try again i pulled the image form github so maybe there is a delta14:24
sean-k-mooneyfrom here https://github.com/cirros-dev/cirros/releases/tag/0.5.214:24
chateaulavthe ubuntu one is kinda weird because of how they setup the video aspect but i havent had issues with other vendors. it still builds but i only ever have ssh14:24
dmitriissean-k-mooney: RE https://review.opendev.org/c/openstack/nova/+/824833/7/nova/network/neutron.py#1536 shouldn't we always be able to get a VF num for a valid VF PCI address? This would only raise if a device somehow got unbound just before this code ran and a symlink to physfn of a VF is not present to retrieve the VF num OR if the PCI address is14:25
dmitriisnot valid in the first place. Happy to remove this check and let operators to figure it out in case it happens but just curious.14:25
sean-k-mooneydmitriis: im not sure if all vendors always export the symilink to work that out14:25
sean-k-mooneyi would expect that the vf number should generally be avaialble but rather then raise in this funciton i think we shoudl just not include the info if not avaiable14:26
sean-k-mooneyas a general pattern14:26
opendevreviewElod Illes proposed openstack/nova stable/wallaby: workarounds: Add libvirt_disable_apic  https://review.opendev.org/c/openstack/nova/+/80562814:28
sean-k-mooneydmitriis: i say perfer as im open to being conviced otherwise but in general i woudl prefer not to reate retiving info as an error. its true it might mean that device cannot be remote managed but in that case the operator has incorrectly tagged it. on the neutron side the ml2/driver can check the profile and fail the binding if the info is not present tha it needs14:28
dmitriissean-k-mooney: I think it's the generic SR-IOV handling code in the kernel that creates the sysfs entry14:32
dmitriishttps://github.com/torvalds/linux/blob/v5.16/drivers/pci/iov.c#L144-L14714:32
dmitriishttps://github.com/torvalds/linux/blob/v5.16/drivers/pci/iov.c#L29514:32
dmitriisIn this case, the case where the info won't be retrievable will likely never happen so I have no issue in making it optional. Just trying to reason about what to put into a comment there.14:32
sean-k-mooneyjust say somthign like. this should be created by default on all modeern kernels but we make it optional to cater for expotic hardware or older kernel where this may not be ture14:34
dmitriissean-k-mooney: ack, will do14:34
sean-k-mooney dmitriis  looking at 4.14 for example im not sure it does the same https://github.com/torvalds/linux/blob/v4.14/drivers/pci/iov.c#L165-L17614:35
sean-k-mooneywell14:35
sean-k-mooneyit has the physfn symlink14:36
sean-k-mooneyi guess it has the virtfn symlink too14:36
sean-k-mooneyso i guess it would not break on centos 814:36
sean-k-mooneydmitriis: i know mellonox/nvida changed where the representor netdevs were create for example in a relitivly recent kernel14:37
sean-k-mooneyhttps://github.com/openstack/os-vif/commit/b37de19c58c877f5174d76d0a4ba5ab519f464e8#diff-087288ddacb7516a4762ef38c4361754c2d6bb07a52952b06aa6f607dc3811fe14:37
sean-k-mooney5.8 https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net.git/commit/?id=123f0f53dd64b67e34142485fe866a8a581f12f114:38
sean-k-mooneydmitriis: im trying to avoid failure in nova when thing like that ar changed14:39
dmitriissean-k-mooney: yeah, probably worth one VM failing to boot than the whole daemon dying because of the exception14:39
dmitriistotally agree14:39
sean-k-mooneyyep plus as i said in the neutron ml2 driver you can fail the port binding an log an explit error message if the required info is not present14:40
dmitriissean-k-mooney: also very true14:41
sean-k-mooneyyou could proably also add a check in nova at a differnt point if you wanted before that but i woudl just log the error form neutron personlaly14:41
dmitriissean-k-mooney: I'll log a warning in Nova in this case and simply return no data in the profile and let Neutron fail to bind14:42
sean-k-mooneywell you shoudl not return an empty profile14:42
sean-k-mooneythis code will be used for other VFs that may not be remote managed14:42
sean-k-mooneyand tehy do not need the vf number14:43
sean-k-mooneywe pass the vf pci adress14:43
sean-k-mooneythat is all they need in the general case14:43
dmitriisrephrasing: 1. if PF MAC cannot be retrieved, do not include vf_num or card_serial_number either14:43
dmitriis2. if vf_num cannot be retrieved, do not include pf_mac and card_serial_number14:43
sean-k-mooneyah well ya perhaps14:44
sean-k-mooneyi was thinking just include all the info you can always but i guess that works too14:44
dmitriisok, just need to fix the tests a little and I'll re-upload14:45
sean-k-mooneyack14:45
sean-k-mooneydmitriis: by the way have you tested this feature with a second ovs on the compute host14:46
*** Guest1862 is now known as dansmith14:47
sean-k-mooneywe had some internal question regarding can we have both offpath port and on path port on the same host14:47
sean-k-mooneythe reason for that was to use vm management port with vnic_type=normal and datapalne prots with vnic_type=smartnic14:48
dmitriissean-k-mooney: i.e. having a DPU and a regular SR-IOV card 14:48
sean-k-mooneywell dpu and another nic that can be used with ovs or linux bridge14:48
dmitriisright14:48
dmitriisI haven't tested that explicitly but just trying to think what would that entail 14:49
sean-k-mooneybasically it was raised that by using the dpu you are limiting the numer or port that can be created to the number of vfs14:49
sean-k-mooneydmitriis: i think it  should just work honestly14:49
sean-k-mooneyi was just wondering if you had considerd it14:49
dmitriissean-k-mooney: yes, the limitation on the number of ports have definitely come up in our conversations14:49
sean-k-mooneydmitriis: all it really requires is the ml2 dirver changes to not require you to designate a host as only dpu enabled14:50
dmitriisand nothing immediately jumps into mind regarding OVS on the host being a problem14:50
sean-k-mooneye.g. make the binding desiion per port based on vnic_type14:50
sean-k-mooneyack that is what my assement was too14:50
dmitriissean-k-mooney: yeah and IIRC it's just based on the vnic_type, we just added handling for vnic_type smartnic to the OVN mechanism driver14:51
dmitriisso it should just bind a normal port regularly14:51
sean-k-mooneyyep14:51
sean-k-mooneycool14:51
sean-k-mooneywe are still trying to assess what woudl be requried to eventually support this in our product14:51
dmitriissean-k-mooney: yeah, plus hardware is hard to come by14:52
sean-k-mooneythat too14:52
dmitriisI think this feature will be useful for FPGA-based SmartNICs as well. I've seen the ones with a separate CPU but with an FPGA instead of an ASIC14:52
dmitriisconsidering the DPDK dataplane usage at the SmartNIC's CPU side for that with rte_flow for offload14:53
sean-k-mooneyi do potentially have access to a BF2 that i coudl use but only 1 so i obviouly cant test multi node and in general an all in one deployment while useful is not fully reflective of how it would be deployed in a datacenter14:53
dmitriisthe Nova and Neutron bits would work the same though14:53
sean-k-mooneydmitriis: yes in princial it could be14:53
dmitriissean-k-mooney: I have an idea on how to test multi-node with just one BF214:53
sean-k-mooneythe only fpga based smartnic i have use used armstong os with only 2 arm cores and i think 4GB of ram14:54
sean-k-mooneydmitriis: run two copies of nova-compute with difernt host values in teh conf on the same host14:54
sean-k-mooneyand use differnt ports on the BF2?14:54
dmitriissean-k-mooney: 2 VMs with nested virt a the hypervisor, plus 2 VMs at the DPU side. Each VM uses 1 PF. VPD can be faked by bind mounting a file in the right sysfs location14:54
dmitriisBF2's ARM CPU is virt-capable btw xD14:55
sean-k-mooneydmitriis: that proably wotn work becaue wehn the PF is passed to the qemu instacle the pcie capablityies are not in the pci config space in teh guest14:55
sean-k-mooneylspci in the guest will only print the pci capablities not the extened pcie cabalities like sriov14:56
sean-k-mooneyin my previous experince14:56
sean-k-mooneyeven when using q3514:56
dmitriissean-k-mooney: I was about to ask about q35 :^)14:56
sean-k-mooneyif it does work let me know :)14:56
dmitriissean-k-mooney: there's another option14:56
dmitriisBF2 on one host + ConnectX on another14:56
dmitriisjust use the remote_managed feature with ovs/ovn-controller running locally on the connectx node14:57
sean-k-mooneyya i tought about that14:57
sean-k-mooneyjsut addign the serial to the chasis table for the hosts ovn14:58
dmitriisyes14:58
sean-k-mooneyin prinicapal that would owrk without BF214:58
dmitriisyes, technically, we just made Nova and Neutron realize that networking agents may run remotely but the trivial case is running locally14:58
sean-k-mooneyi did not really want to say that to our qe however :)14:58
dmitriisyeah, fair enough :^)14:59
sean-k-mooneybut ya in principal we can test most of the integration that way for any nic that supprot vpd14:59
dmitriisor even bind mount a file to the right sysfs location15:00
sean-k-mooneyi.e. if we can get the serial,mac and vf number we can test the end to end integration15:00
dmitriisI got used to crafting VPD blobs since I needed to unit test the libvirt change15:00
dmitriisbasically, I crafted them byte-by-byte15:00
dmitriissean-k-mooney: ^ yes15:00
sean-k-mooneyya thats doable15:01
sean-k-mooneyalthough i dont think it woudl be hard to have a python class that modeled it and just serialise it as a byte string15:01
sean-k-mooneyrather then do it by hand15:01
sean-k-mooneyunfortunetly i dont think we can use that to test in the upstream ci15:01
dmitriissean-k-mooney: yeah, it's not too hard. The only tricky part there is checksum calculation but it's a simple algorithm15:02
sean-k-mooneycrc32 or similar i assume15:02
sean-k-mooneyit would not be hard to look at the c code and ectra15:02
dmitriissean-k-mooney: just one's complement15:02
sean-k-mooneythat said im not sure libvirt cares about the crc15:02
sean-k-mooneyack15:03
dmitriissean-k-mooney: IIRC I validate the checksum for the read-only portion there15:03
sean-k-mooneyoh in the nova code?15:03
sean-k-mooneyi dont think i recall seeing that15:03
dmitriissean-k-mooney: no-no, in the Libvirt code that parses VPD15:04
sean-k-mooneyah15:04
sean-k-mooneyok15:04
dmitriissean-k-mooney: there is a chance that something may be added to the mellanox CI. I know CX5 hw is there based on the logs but no BF2 (yet)15:04
dmitriismaybe I'll have some info later this week about that15:04
dmitriisthere are automation challenges: i.e. we need to bring up devstack + also program the DPU15:05
dmitriisbut devices availability first I guess15:05
sean-k-mooneyi actully have been doing some experiment on the side lately and as part of my test infra i have a fake copy of part of /sys 15:05
sean-k-mooneyhttps://github.com/SeanMooney/arbiterd/tree/master/arbiterd_tests/test_data/sys15:05
sean-k-mooneythe idea being that in my func test it would use the fake copy of the file system15:05
sean-k-mooneyits too bad we cant create fake pf devices using kernel modules for testing15:06
sean-k-mooneyi have looked into it in the past and we can create the fake netdevs for sriov15:06
dmitriissean-k-mooney: there's netdevsim but it doesn't create PCI devices15:06
sean-k-mooneybut not the pci toploty15:06
sean-k-mooneyyep15:07
sean-k-mooneyexactly15:07
dmitriiscan't pass them to a VM, yep15:07
sean-k-mooneyi tried to use that for sriov testing in the upstream ci in the past15:07
sean-k-mooneyi also was lookign at the mdev equivlent 15:07
dmitriiswe utilize netdevsim for ovn-vif testing15:07
sean-k-mooneysame issue no pci endpoints15:07
dmitriisand Frode is working on some code to extend it in the kernel but it's only usable for the ovn-vif testing part15:08
sean-k-mooneyya that kind fo makes sense since it woudl not need the pci adress just the netdevs15:08
sean-k-mooneyif it ever gets extened to implement pci vfs/pfs 15:09
sean-k-mooneywe would be able to use it for basic pci pasthorugh testing15:09
sean-k-mooneywe woudl not need the networking to actully work15:09
sean-k-mooneyjust enough to pass it to qemu15:09
sean-k-mooneythat is a lot of work but would be super useful for us15:09
dmitriissean-k-mooney: I also looked at a possibility of using the rocker switch in QEMU but it's useful for switchdev testing but doesn't have SR-IOV emulation15:09
sean-k-mooneyya15:10
dmitriissean-k-mooney: yes, also Libvirt's testing of VF-related stuff is quite minimal. They have the same problem with emulating VFs presumably15:10
sean-k-mooneyso at some point we likely will want to supprot vdpa with subfunction rather then vfs as the parent15:10
sean-k-mooneythe rocker switch might be useful to test that eventually15:10
dmitriisyeah, SFs are something we also started looking into but there's more work to do at the ovn-vif side. Even for VFs we ran into a race at the BF2 side https://github.com/ovn-org/ovn-vif/pull/1 and SFs are even more dynamic15:12
dmitriisTL;DR: SR-IOV is enabled at the hypervisor and then BF2 realizes that it needs to create representors and the DPU kernel starts creating udev events for them and rename them15:13
sean-k-mooneyack in a similar vain i was hoping to eventualy start using https://github.com/torvalds/linux/blob/d4ec3d5535c784c3adbc41c2bbc5d17a00a4a898/samples/vfio-mdev/mtty.c and https://github.com/torvalds/linux/blob/d4ec3d5535c784c3adbc41c2bbc5d17a00a4a898/samples/vfio-mdev/mdpy.c or15:13
sean-k-mooneyhttps://github.com/torvalds/linux/blob/d4ec3d5535c784c3adbc41c2bbc5d17a00a4a898/samples/vfio-mdev/mbochs.c to test our generic mdev support15:13
dmitriismtty, interesting15:14
sean-k-mooneyi belive it just echos back what you send to it15:14
sean-k-mooneybut that woudl be super simple to test in tempest15:15
dmitriisyes, quite useful15:15
sean-k-mooneyour current mdev code i belive assumes each mdev has a pci device as a parent15:15
sean-k-mooneyso we would need to relax that15:15
sean-k-mooneyto use the driver but if we did we could get full ci coverage for it15:15
sean-k-mooneywhen people start using mdev for subfuction or non VF usecase we might need to do that anyway15:16
sean-k-mooneyat which point we can test it properly in ci15:16
dmitriisI heard that SFs were introduced to support container use-cases because of scalability issues, less so for VMs. Doesn't mean VMs cannot use them but still.15:17
dmitriisscalability ~ small number of ports15:17
sean-k-mooneydepend on who you ask. i know intel were interestin in there version for contaienr but also saw it as a way to adress the vm case in terms of number of ports15:18
sean-k-mooneydmitriis: really the use case is the same allow more then max_vf ports 15:18
sean-k-mooneythat said i think recent connectx/bf2 cards have started to get into the 1000+ vf range15:19
dmitriissean-k-mooney: yeah, there are some parts of the SR-IOV spec that can be used to overcome the 256 limit15:19
dmitriissome neat tricks with PCI address allocation15:19
sean-k-mooneyyes you multiple buses to card to work around that15:20
dmitriisyep15:20
sean-k-mooneycontaienr and vm cloud system really prefer to pertend that the port limit is infinity15:21
sean-k-mooneylike k8s and opnestack really dont treat ports as a finite reqouce outside fo sriov15:21
dmitriissean-k-mooney: there's another problem: accounting and quotas on certain hw limits. There's a finite number of flows that can be programmed and not a lot of data on hw limits per VF/SF15:22
dmitriislikewise, no APIs to query that AFAIK15:22
sean-k-mooneyin reality most customer proably wont hit that limit but they are woried fi they layer things like openshift running on top of openstack they might15:22
dmitriissean-k-mooney: yes, plus with layers on top of OpenStack people start running overlays on top of overlays15:23
sean-k-mooneyya there are both limits on the number of tables and the table row count and then seperat lmits on but the hardwar flows offload15:23
dmitriisand doing that using guest CPU15:23
sean-k-mooneyok i better get back to reviewing your changes before my next meeting :)15:24
dmitriissean-k-mooney: right, I'll get to re-submitting too :^)15:24
*** dansmith is now known as Guest210215:29
*** Guest2102 is now known as dansmith15:37
sean-k-mooneydmitriis: is there anyting in the port beyond the vnic_type that tells nova that we need a vf15:49
sean-k-mooneydmitriis: gibi  be might not be able to reuse vnic_type smartnic15:49
opendevreviewTobias Urdin proposed openstack/nova master: Cleanup old resize instances dir before resize  https://review.opendev.org/c/openstack/nova/+/82786515:50
sean-k-mooneyif we assume that all port that have vnic_type smartnic require a vf that would break ironics use of it yes?15:50
opendevreviewBalazs Gibizer proposed openstack/placement master: Fix perfload jobs after consumer_types  https://review.opendev.org/c/openstack/placement/+/82816715:50
gibisean-k-mooney: I don't know how ironic uses the smartnic vnic_type15:51
gibimelwitt: I think the perfload job is easy to fix https://review.opendev.org/c/openstack/placement/+/82816715:51
dmitriissean-k-mooney: (thinking)15:51
sean-k-mooneyhttps://specs.openstack.org/openstack/ironic-specs/specs/12.1/support-smart-nic.html15:51
sean-k-mooneygibi: they bascially wanted to supprot ovs running on the smartnic also15:54
sean-k-mooneybut using ml2/ovs15:54
sean-k-mooneywith the neutron agetn deployed on the smartnic15:55
dmitriissean-k-mooney: they have a special config option for the neutron-openvswitch-agent as well15:55
sean-k-mooneyyes15:55
sean-k-mooneydmitriis: the possible problem i am seing is when we go to scedule the vm we have not bound the port15:55
sean-k-mooneysince we have not selected a host yet15:56
sean-k-mooneyso i dont think we will be able to tell the difference between the ironic usage and the new usage15:56
sean-k-mooneyand we wont know if we shoudl request a VF or not15:56
sean-k-mooneyso we might need to use a differnt off-path vnic type15:57
sean-k-mooneyinstead15:57
dmitriissean-k-mooney: in Nova we decide if we want to request a remote_managed port or not15:57
dmitriisand add that to a InstancePCIRequest15:57
sean-k-mooneydmitriis: yes based on the vnic_type right15:57
dmitriisyes15:57
sean-k-mooneyright so if we only look at that we need a new vnic type15:58
dmitriisbut that's in Nova, trying to figure out how that affects ironic15:58
sean-k-mooneysince that decision need to be made before  we schdule and therefor cannot depend on the host or driver15:58
sean-k-mooneybooting ironic service via nova api15:58
sean-k-mooney*servers15:58
sean-k-mooneyin both cases the vnic_type would be the same15:59
sean-k-mooneythe falvor woudl be differnt but we would not know its an ironic flavor15:59
sean-k-mooneydmitriis: so i think we need to quickly add a new vnic in neutron-lib and update the nova and neutron code to use that16:00
opendevreviewTakashi Kajinami proposed openstack/nova master: Allow authorization by user_id for server resume action  https://review.opendev.org/c/openstack/nova/+/82816816:01
opendevreviewTakashi Kajinami proposed openstack/nova master: Allow authorization by user_id for server resume action  https://review.opendev.org/c/openstack/nova/+/82816816:01
dmitriissean-k-mooney: we could, trying to think if there's anything we've missed that could allow us to avoid that. We originally wanted to add a new VNIC type but then decided to reuse VNIC_TYPE_SMARTNIC after some reviews.16:03
sean-k-mooneyyep i proably suggested the reuse :) but i think its proably required16:03
sean-k-mooney*the new vnic16:03
sean-k-mooneyi dont think there is anything else on the port we can use to differenciate16:03
* gibi needs to jump on a call but will read back16:04
sean-k-mooneyoh same 16:05
opendevreviewTakashi Kajinami proposed openstack/nova master: Allow authorization by user_id for server resume action  https://review.opendev.org/c/openstack/nova/+/82816816:10
*** amoralej is now known as amoralej|off16:22
gibiso if we always translate the smartnic vnic_type to InstancePCIRequest but ironic support smartnic with ml2/ovs then we might have a problem16:38
gibido we require ironic to use some custom resource in the flavor or it is just an option?16:38
sean-k-mooneyam we require cpu, ram and disk to be overriden to resources:cpu=016:42
*** dansmith is now known as Guest210816:45
*** Guest2108 is now known as dansmith16:46
gibithat could be a clue16:46
sean-k-mooneyim not sure if we want to rely on that16:46
sean-k-mooneywe could16:46
gibiyeah it is hackins16:46
gibihackis16:46
gibiwhen we have the lib freeze?16:47
sean-k-mooneyi belive in a week or two16:47
sean-k-mooneyill check16:47
gibifeb 1816:47
sean-k-mooneyya 16:47
sean-k-mooneyactully proably 17th16:48
gibiyeah16:48
sean-k-mooneyso we still have time to add the value16:48
gibilets try the new vnic_type way then16:48
gibithat would be clean16:48
sean-k-mooneyso vnic_type=off-path16:49
sean-k-mooneyactully maybe vnic_type=off-path-direct16:49
sean-k-mooneyso that we can have vnic_type=off-path-vdpa or off-path-mev or whaterver later if needed16:49
sean-k-mooneybut keep the pattern16:49
gibiyes, off-path or remote-managed whathever floats the neturon team's boat16:50
sean-k-mooneyremote-managed might be better ya16:51
sean-k-mooneykeep the continuity with the config option16:51
dmitriissean-k-mooney, gibi: been trying to find something to use as a clue (such as image or flavor metadata)16:56
dmitriisbut, yes, it's tricky to rely on that16:56
sean-k-mooneydmitriis: i dont think there is really anything and really it shoudl be soemthing on the port anyway17:01
sean-k-mooneywe could look at the flaovr but that is messy17:01
dmitriissean-k-mooney: ovs hardware offload uses binding:profile to differentiate between legacy SR-IOV and offload cases https://docs.openstack.org/neutron/latest/admin/config-ovs-offload.html#validate-open-vswitch-hardware-offloading17:02
dmitriisbeen considering utilizing something like that as well17:02
sean-k-mooneydmitriis: the port is not bound at this point17:02
sean-k-mooneywe cant use anything that depend on the host/neutron backend 17:03
sean-k-mooneysince the host is only selected after the scheduling17:03
sean-k-mooney dmitriis  also --binding-profile '{"capabilities": ["switchdev"]}'  i dot think was ever actully implemented17:03
dmitriissean-k-mooney: yeah, that's the unfortunate thing - I don't know the driver before scheduling happens17:03
dmitriissean-k-mooney: ah, right we discussed that way back17:04
sean-k-mooneyit also would be admin only since binind-profile is admin only17:04
sean-k-mooneyyep17:04
dmitriissean-k-mooney: yes, that would be a shame if it was admin-only17:04
dmitriissean-k-mooney: ok, I am convinced we need a new VNIC type then17:04
dmitriissean-k-mooney: I'll propose a change and, I suppose, spec changes too17:05
dmitriishopefully I'll get quick turnaround from the Neutron team17:05
dmitriisfnordahl ^ FYI17:05
sean-k-mooneyyep im happy to review the spec change and approve once neutron agree17:05
dmitriissean-k-mooney: proposed a spec change for Neutron: https://review.opendev.org/c/openstack/neutron-specs/+/828173. Will upload a neutron-lib change and also raise a Nova spec update.17:22
sean-k-mooneydmitriis: ack.17:26
chateaulavsean-k-mooney: i think i found it, essentially have to tell nova to set `cpu = None` so that no topology is identified. https://usercontent.irccloud-cdn.com/file/EAK0qvLu/riscv17:38
sean-k-mooneytopology17:39
sean-k-mooneyoh pci topology17:39
sean-k-mooneyhum i dont know if None is a supported value or if that is stable17:39
chateaulavhttps://usercontent.irccloud-cdn.com/file/5U9wBUXt/image.png17:39
sean-k-mooneye.g. does that just mean "qemu you decide"17:39
sean-k-mooneyis that new code your addign to nova17:40
sean-k-mooneysettign the cpu element to None17:40
sean-k-mooneyfor risxv6417:40
sean-k-mooney]hum that might break other features17:41
sean-k-mooneyit will disable the topology but also numa affinity17:41
sean-k-mooneyhave you tried settign the sockets and thread to 117:41
chateaulavgonna do more testing, but that is locally currently. 17:42
sean-k-mooneyack. try hw:cpu_sockets=1 hw:cpu_threads=117:42
chateaulavyeah i did no 'cpu.model = None' first and it did not like that17:42
sean-k-mooneyi think your current code is just not generatign the cpu element entirly17:43
sean-k-mooneybut if its just a toplocy issue perhaps you can generate a toplogy it will like17:43
chateaulavyeah, i believe so.17:43
sean-k-mooneywhen you dont set the falvor extra specs technially the toplogy is driver defined17:44
sean-k-mooneyso you can modify it to work based on the requirments17:44
sean-k-mooneyfor legacy reason the libvirt driver genearte a toplogy of 1 socket per vcpu requested17:45
sean-k-mooneywhich is bad for a number of reasons17:45
sean-k-mooneyit used to improve perfromance but i stongly doubt it does and it cause issue for windows for example 17:45
sean-k-mooneychateaulav: so if you need to change the default toplogy for the emulation usecase i think that is ok17:46
chateaulavdefinetly, now that ive isolated the area, gonna tweak the topology settings17:46
opendevreviewTobias Urdin proposed openstack/nova master: Cleanup old resize instances dir before resize  https://review.opendev.org/c/openstack/nova/+/82786518:14
opendevreviewDmitrii Shcherbakov proposed openstack/nova-specs master: Late Amendments to the Off-path Backends Spec  https://review.opendev.org/c/openstack/nova-specs/+/82817718:22
dmitriissean-k-mooney: https://review.opendev.org/c/openstack/nova-specs/+/828177 - uploaded18:27
sean-k-mooneydmitriis: ack am im not really happy with adding allow_remote_managed_ports to the whitelist18:33
sean-k-mooneythat feels semanticly incorrect to me18:33
*** Uggla is now known as Uggla|afk18:33
sean-k-mooneydmitriis: the docs issue on the spec repo is not related to your patch by the way18:36
dmitriissean-k-mooney: the alternative approach I considered was to expose a property on the whitelist which would tell us about whether remote_managed is used or not18:36
sean-k-mooneydmitriis: that i think i coudl accpet yes18:36
sean-k-mooneyreally the whitelist shoudl not really have awareness of the off-path feature18:37
dmitriissean-k-mooney: hmm, I felt it was slightly backwards because I am exposing state for somebody else to check but maybe I am wrong18:37
sean-k-mooneya porperty i think woudl be ok18:37
dmitriisI'd have to pass it through both on the whitelist and PciDeviceSpec though18:37
dmitriisbecause I only know about the use of those tags in the spec object18:37
sean-k-mooneythe whitelist shoudl not be doign any validation of those tags18:38
sean-k-mooneyonly the pci tracker should18:38
sean-k-mooneythe whitelist is a POD object18:39
sean-k-mooneyit does some basic validation fo the imports18:39
sean-k-mooneybut it should not be doing any filtering 18:39
dmitriissean-k-mooney: there's some validation done here https://github.com/openstack/nova/blob/b6fe7521afa8d42febc68f5f79782f7bcc3b568f/nova/compute/manager.py#L143018:41
dmitriiswhich is also very early18:41
dmitriisand I have access to the whitelist here18:41
sean-k-mooneythat is in the compute manager18:41
sean-k-mooneyso its oke to do it there18:41
sean-k-mooneybut really it shoudl not be there either18:42
sean-k-mooneyit shoudl be in the resouce tracker or pci manager18:42
sean-k-mooneyin https://github.com/openstack/nova/blob/master/nova/pci/manager.py#L3818:42
dmitriissean-k-mooney: yeah, this should be set up relatively early as well18:43
sean-k-mooneyyou shoudl do any validation you need here https://github.com/openstack/nova/blob/master/nova/pci/manager.py#L78-L8318:43
sean-k-mooneyits called as part of _setup_pci_tracker in teh resouce traceker18:44
sean-k-mooneyhttps://github.com/openstack/nova/blob/14bfbaa50ed3c000a691a37afbd20d86f431b3fb/nova/compute/resource_tracker.py#L764-L77318:44
sean-k-mooneywhich is invoked as part of _init_compute_node18:45
dmitriissean-k-mooney: ack. And when it comes to exposing the property, I could populate _has_remote_managed after this line https://github.com/openstack/nova/blob/b6fe7521afa8d42febc68f5f79782f7bcc3b568f/nova/pci/whitelist.py#L51 and expose a property18:45
dmitriisand also expose a property on the spec objects themselves18:46
dmitriisI'd have to do a search over spec objects then and see whether any one of those has a property18:46
dmitriisif that's ok, I'll just do it18:46
sean-k-mooney  i think that shoudl be ok18:47
sean-k-mooneyit sound cleaner to me overall18:48
sean-k-mooneywhere/why are you using this by the way18:48
dmitriissean-k-mooney: this is to make sure remote_managed tags aren't usable on a host where libvirt is too old (doesn't have VPD handling code)18:49
sean-k-mooneyright so you dont need this on the spec object or whitelist object18:49
sean-k-mooneythat is a check you can do in the libvirt driver at start up18:49
sean-k-mooneyyou can just loop over the spec objects and check for the tag18:49
sean-k-mooneyand check the verisons18:50
sean-k-mooneythe device spec will already have the value in the PciDeviceSpec.tags field18:50
dmitriishttps://review.opendev.org/c/openstack/nova/+/827839/2/nova/virt/libvirt/driver.py#81618:50
dmitriis^ so it's a host-based capability18:51
dmitriishost-dependent *18:51
sean-k-mooneyyes18:51
sean-k-mooneythat does not need to you modify these data stuctures18:52
dmitriissean-k-mooney: I don't have access to the host object in those18:52
dmitriisbut maybe there's some other way to check the Libvirt version18:52
sean-k-mooneyyou should not be checkign for them in these18:52
sean-k-mooneydmitriis: you shoudl not be doing https://review.opendev.org/c/openstack/nova/+/827839/2/nova/pci/devspec.py#31918:53
sean-k-mooneythat is not the correct place to raise that excption18:53
sean-k-mooneyeither the device spec of whitelist shoudl raise that18:53
dmitriissean-k-mooney: ah, sorry, yes we're moving it elsewhere18:53
dmitriissean-k-mooney: so maybe in the dev tracker I have access to that info, checking18:54
sean-k-mooneyi dont think you do18:54
sean-k-mooneywe abstrackt the virt driver info away18:55
sean-k-mooneythe pci module is ment ot be virt dirver independent18:55
sean-k-mooneyyou shoudl be doing this check in the libvirt driver.py18:55
sean-k-mooneyyou coudl perhaps do it in update_devices_from_hypervisor_resources 18:56
sean-k-mooneyhttps://github.com/openstack/nova/blob/b6fe7521afa8d42febc68f5f79782f7bcc3b568f/nova/pci/manager.py#L11118:56
sean-k-mooneybut you would have to modify the device_json which im not sure is the right thing18:56
sean-k-mooneyam i need to call it a day18:57
dmitriissean-k-mooney: ack, I'll think about how to do it18:57
sean-k-mooneydmitriis: can you look at moving the check to the libvirt driver18:57
dmitriissean-k-mooney: sure18:58
*** hemna3 is now known as hemna19:20
opendevreviewDan Smith proposed openstack/nova master: Join quota exception family trees  https://review.opendev.org/c/openstack/nova/+/82818519:32
opendevreviewDan Smith proposed openstack/nova master: Move keypair quota error message into exception  https://review.opendev.org/c/openstack/nova/+/82818619:32
dansmithmelwitt: ^19:32
dansmithI have one other thing I'm going to stack on there, but it'll take me a few more minutes19:32
melwittdansmith: ack19:33
opendevreviewDan Smith proposed openstack/nova master: Move keypair quota error message into exception  https://review.opendev.org/c/openstack/nova/+/82818619:34
*** noonedeadpunk_ is now known as noonedeadpunk19:34
chateaulav      <nova:flavor name="r1.small">20:02
chateaulav        <nova:memory>4096</nova:memory>20:02
chateaulav        <nova:disk>16</nova:disk>20:02
chateaulav        <nova:swap>0</nova:swap>20:02
chateaulav        <nova:ephemeral>0</nova:ephemeral>20:02
chateaulav        <nova:vcpus>2</nova:vcpus>20:02
chateaulav      </nova:flavor>20:02
chateaulavsean-k-mooney: so setting any topology items doesnt work20:03
chateaulavthe above and below are what is set in regards to cpu20:03
chateaulavhttps://www.irccloud.com/pastebin/f9YIhb6z/20:03
opendevreviewDan Smith proposed openstack/nova master: Join quota exception family trees  https://review.opendev.org/c/openstack/nova/+/82818520:09
opendevreviewDan Smith proposed openstack/nova master: Move keypair quota error message into exception  https://review.opendev.org/c/openstack/nova/+/82818620:09
opendevreviewmelanie witt proposed openstack/nova master: Raise InstanceNotFound on fkey constraint fail saving info cache  https://review.opendev.org/c/openstack/nova/+/82694220:12
*** dasm is now known as dasm|off22:43

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!