Friday, 2023-04-21

« Thursday, 2023-04-20 Index Saturday, 2023-04-22 »
*** iurygregory is now known as iurygregory|holiday02:26
gibisean-k-mooney: +207:49
dvo-plv_sean-k-mooney: Hello08:41
dvo-plv_I would like to clarify our blueprint status. https://review.opendev.org/c/openstack/nova-specs/+/86837708:42
dvo-plv_Maybe community need something from us to start further activities08:44
sean-k-mooneyno just pinging me and others to reveiw. my time is currently quite limited for upstream work so my review bandwith has been reduced alot due to downstream work.09:48
sean-k-mooneyi think i was more or less ok with it the last time i looked so ill take a look again09:49
sean-k-mooneydvo-plv_: im happy with the spec as is. others might ask for more info but i think you have the main points.09:53
sean-k-mooneyi changed the topic in gerrit to bp/virtio-packedring-configuration-support09:54
sean-k-mooneycan you use the same topic for the code09:54
dvo-plv_okay, I will update topic for code. Currently it has https://review.opendev.org/q/topic:VirtIO_PackedRing10:19
dvo-plv_What does this topic means ?10:19
sean-k-mooneywe can use that on the spec10:29
sean-k-mooneythe topic is just wa way to group related patches together10:29
sean-k-mooneyso the code and spec should use the same one10:29
sean-k-mooneyby convention we use bp/<bluprint name> for features that are tracked by a bluepirnt or spec10:30
sean-k-mooneyand bug/<bug number> for bugs10:30
sean-k-mooneyotherwise its freeform10:30
sean-k-mooneyfor feature that require changes in multiple project we try to use the same topic across all of them to be able to see all reated patches quickly10:31
sean-k-mooneyso now that you updated it https://review.opendev.org/q/topic:bp%252Fvirtio-packedring-configuration-support10:31
sean-k-mooneywe can see the spec, nova and os-traits patches all in one view10:32
sean-k-mooneythat tells me you are missing a patch to glance to update the metadefs https://github.com/openstack/glance/blob/master/etc/metadefs/compute-libvirt.json10:33
sean-k-mooneydvo-plv_: ^ the metadefs are what horizon and heat use to generate the drop down for adding traits automatically10:33
sean-k-mooneysorry not triats extra_specs and image properties10:34
dvo-plv_okay, I will investigate and fix10:35
sean-k-mooneyyou basically jsut need to copy this https://github.com/openstack/glance/blob/master/etc/metadefs/compute-libvirt.json#L30-L3510:36
sean-k-mooneyupdatign mem_encyption to packed_ring10:36
sean-k-mooneythe namespce/prefix is automaticaly handeled by https://github.com/openstack/glance/blob/master/etc/metadefs/compute-libvirt.json#L7-L1610:37
dvo-plv_okay, thanks. I would liek to check it on the horizon ui too10:37
sean-k-mooneyits also good to update the userful image properies doc https://github.com/openstack/glance/commit/3a281b9bc62a1b8b0f1468bc641105a5662f8ecd is an example10:37
dvo-plv_sure10:38
dvo-plv_Could you also clarify me with your scheduler according to the upstream work? We would like to start work over multiqueue support for hardware offloading10:38
sean-k-mooneyother can review your code not jsut me. but currently i have around 20% of my time for upstream work less fo the last 3-4 weeks because i was busy with downstream planing activies due to the ptg and other issues10:39
sean-k-mooneyi have been trying to capture all the outcomes of the ptg in our downstream jira and plannign the work for our team there10:40
sean-k-mooneynormally i have more time for upstream review so that shoudl go back to normal soon10:41
sean-k-mooneyregarding multi queue 10:41
sean-k-mooneydo you mean https://review.opendev.org/c/openstack/nova-specs/+/85551410:42
dvo-plv_Whom I should ping regarding review process. I saw gibi on some review process10:42
dvo-plv_Yes, we would like to analyze it better and provide some vision, how we can improve this in the OpenStack for hardware offloading, expecially for our nic10:43
sean-k-mooneyso multi queue is only partly implemented for hardware offload10:43
sean-k-mooneyi dont think it actully works we cannot enable it for nics that use sriov10:44
sean-k-mooneyi.e. nics that present the vf directly to the guest10:44
sean-k-mooneybut it might eb possibel for macvtap or vdpa10:44
sean-k-mooneydvo-plv_: in terms of pings the active member of the nova core team10:45
sean-k-mooneydvo-plv_: so gibi, bauzas, melwitt, gmann and  dansmith  are you best bet in addtion to me. stephenfin is also around somethimes but they mainly work on non nova related thigns day to day10:46
dvo-plv_okay, thanks10:47
sean-k-mooneyso looping back to multi queue10:49
sean-k-mooneyimplementing this in the way you wanted is not going to be easy or really desireable form a nova point of view10:50
sean-k-mooneythere are a few parts to this problem10:51
sean-k-mooneyfirst we need to detach and recored the number of queues avaialbel in the vf10:51
sean-k-mooneysecond we need to be able to schudle based on that (either updatign the pci filter or recording this in placement)10:52
sean-k-mooneythird we need a way to request a device with a min number of queuse multiup queuse out side the falvoar/image (likely on the nutron port)10:53
sean-k-mooneyfinally we need to take the resouce request form teh port and include that in our scueduling reeust and wonce we find a host/device that meets that need we need to ensure that the qemu device is cofnigured correctly10:54
dvo-plv_regarding first question, we investigated it and the best option on our opinio is to parse other config. we configure queues like that -a 0000:65:00.0,representor=[4-6],portqueues=[4:2,5:2,6:2]. Yes it will work only for our nic10:54
sean-k-mooneywhich config 10:54
sean-k-mooneythe pci config space10:55
dvo-plv_ovs. other_config10:55
sean-k-mooneyyou can useuslly get this form sysfs i tought10:55
sean-k-mooneywe cant do that10:55
sean-k-mooneythe ovs port wont exist at that point10:55
dvo-plv_no we can not, this is untrivial task for our dpdk driver10:55
sean-k-mooneyoh right so honestly you cant start this work10:56
sean-k-mooneyuntil the basic work of supporting the dpdk represtors is done10:56
dvo-plv_ovs port not, but vf yes. We would like to parse this config and fill it to the device_spec10:56
sean-k-mooneythe ovs port will be created and added by os-vif10:56
sean-k-mooneyonly after we ahve selected a vf 10:57
sean-k-mooneyso i think we need https://review.opendev.org/c/openstack/nova-specs/+/859290 to be done before we can talk about multiqueue10:57
dvo-plv_I see, we thought we can start to find solution for all comments to the blueprint in the parallel at the moment10:59
sean-k-mooneywell we could but its going to be diffuctly to compelte jsut one of the 3-4 specs you have propsoed this cycle10:59
sean-k-mooneymaybe 210:59
sean-k-mooneyits very unlikely that all of them will land11:00
dvo-plv_You mentined that multiqueue functional will be possible for vdpa. So maybe it will be better to move from virtio-forwarder to the vdpa nvic type for future purposes11:00
sean-k-mooneywell there is work in dpdk to supprot vdpa11:01
sean-k-mooneyi was expecting to have a vdpa-user type at some point for that11:01
sean-k-mooneyhttps://doc.dpdk.org/guides/vdpadevs/features_overview.html11:02
sean-k-mooneyi have not looked into it much11:02
sean-k-mooneyi dont think that is supported by ovs-dpdk currenlty but i have not really been following it closely11:03
sean-k-mooneydvo-plv_: so for the basic enablement we are going to be trackign napatec VF which we will add to ovs as dpdk prots corret11:04
sean-k-mooneyand then those will be exposed to the guest as vhost-user ports11:04
sean-k-mooneyso for multi queue we would need to read the number of queues on the vf ideally 11:05
dvo-plv_This multiqueue functional with queue mq and vector is in the our ovs fork at the moment11:05
sean-k-mooneybecause we need that info for schduling11:05
sean-k-mooneyok so thats kind of a problem11:05
sean-k-mooneywe do not really allow enablment of forked functionality in nova11:05
dvo-plv_Yes, I remember that it requires for placement to handle scheduler with queues number11:06
sean-k-mooneyif we can do it generically we enabel it so i was hoppng we coudl do somehting liek read /sys/bus/pci/device/<address>/num_queus or somehting like that11:06
sean-k-mooneyideally vai libvirt nodedev api11:07
sean-k-mooneynot reading sys directly11:07
sean-k-mooneyso you can get the queue like this https://paste.opendev.org/show/bVSM5IDtJTRwhcIcuTFs/11:09
sean-k-mooneythat is a pf11:09
sean-k-mooneybut i belive the same is true for VFs11:09
sean-k-mooneyi done see the queues in libvirt https://paste.opendev.org/show/bJHNfZR8JNpxJVvLggCI/11:11
sean-k-mooneyso the first step woudl really be to add the ablityu to get the queus form libvirt to libvirt11:12
sean-k-mooneydvo-plv_: do you need the VFs to be bound to vfio-pci11:14
sean-k-mooneyi assume use so i geuss this infor will not be aviable via the vf since it wont have a netdev11:15
dvo-plv_we probe vfio-pci driver modprobe vfio-pci enable_sriov=111:16
dvo-plv_ane then allocate vf echo "$NUMVFS" > /sys/bus/pci/devices/0000:$BUS:00.0/sriov_numvfs11:16
sean-k-mooneyyep thats pretty standard for dpdk although the enable_sriov bit is relitvly recent11:17
dvo-plv_we don not have netdev devices for that. so this is hard to get queue at linux layer11:17
sean-k-mooneyya11:17
sean-k-mooneyso the probelm is the device spec is not intended for configuration11:17
sean-k-mooneyit was orgianly just for filtering11:17
sean-k-mooneywe have since added some metadta to it11:18
sean-k-mooneyim not sure hwo peopel would fell about adding the number of queues11:18
dvo-plv_yes, so this is why we firstly decided that user can fill device_spec with additional rapameter for filtering11:18
dvo-plv_queue_number11:18
sean-k-mooneydvo-plv_: right so that approch has been rejected in the past11:19
dvo-plv_yes11:19
sean-k-mooneyi mean before you propsoed it11:19
sean-k-mooneythere have been attempts to do this in teh past and it was rejected11:19
sean-k-mooneythat said we now have enough things like this that we might be ok with it11:20
sean-k-mooneywe now have things liek remote_managed and resouce class 11:20
dvo-plv_I see. now you would like to see that queue parameter gets automatically like metadata and placemnet filter nodes according to the required queue. But you would not like to increase resource provider11:20
sean-k-mooneyso addign queue_pairs=<count> might be ok11:20
sean-k-mooneyam no we can model this in placment but it would need use to have one RP per vf11:21
sean-k-mooneywhich is not soemthgn we wanted to do if we coudl avoid it11:22
sean-k-mooneywe would need to adress the placment scaling bug first11:22
sean-k-mooneydvo-plv_: https://review.opendev.org/c/openstack/nova/+/85588511:23
sean-k-mooneyso we could not track this in placment initally11:23
sean-k-mooneywe would have to track this in nova and use the pci filter to filter based on the queus 11:23
sean-k-mooneyeventully it could be done in placement but we also need to start trackign neutron consumabel pci devices in placemnet before that11:24
sean-k-mooneythe only workable solution i see in the next 6-12 months is to do this in nova11:25
sean-k-mooneyif we require all VFs in the same pool to have the same queue count11:26
sean-k-mooneythen we can add the queue_pair count to the extra_info on the pci_device in the nova db11:26
sean-k-mooneyand the pci_passhtough filter can use that11:26
dvo-plv_lets assume that we have already dealt with the automatic queue getting. Lets use traits config. if this node has vf with 2 and 3 queues. Placemnet will add traits 2_queus and 3_queues to the scheduler to fitler node by queue. and than, when node is choosen, nova and choose appropariate vf by queue number11:28
sean-k-mooneyno11:28
sean-k-mooneythsi is not a correct use of traits11:28
sean-k-mooneytraits cannot be used for Quantitative aspect of a resuce i.e the number of queuse or frequency of a cpu11:29
sean-k-mooneyHW_NIC_MULTIQUEUE is an accpaable trait which we already have https://github.com/openstack/os-traits/blob/master/os_traits/hw/nic/__init__.py#L1811:30
sean-k-mooneybut 2_queus is not11:30
sean-k-mooneyQuantitative aspects must eb tracked as inventories in a resouce provider11:31
sean-k-mooneydvo-plv_: we cannot currently track neutron consumable pci devices in placment by the way11:32
sean-k-mooneyvmaccel: has said they want to work on that this cycle11:32
sean-k-mooneyso for bobcat it would be risky to asume that work woudl be compelte in time for multi queu to be implemetned11:33
dvo-plv_does it will be part of this spec ? https://specs.openstack.org/openstack/nova-specs/specs/2023.1/implemented/pci-device-tracking-in-placement.html11:33
sean-k-mooneydvo-plv_: no that spec epxlictly dose not support any pci device that can be use via neutron11:33
dvo-plv_so, the main problem taht we can not filter specific node fro the pool with required vf queue number, right ?11:37
sean-k-mooneywe can solve that todya with the pci_passhtough_filter11:37
sean-k-mooneyas i said there are 3-4 peice that need to be done11:38
sean-k-mooney1 recored the number of quesue (add queue_pairs to devspec and store it in pci_divice.extra_info.network_caps.queu_pairs)11:39
sean-k-mooney2 add a new extention to neutron to queueu queue pairs11:39
sean-k-mooney3 modify the pci pasthough filter to use the neutron request to fine a vf that fullfiles the need11:40
dvo-plv_i will be positive and believe that we can solve all of it)11:40
sean-k-mooney4 update teh libvirt generation to use queues=min(vf.queues,flavor.vcpus)11:40
sean-k-mooneywe can but you need to file a spec for the neutorn api extention and implement that before we can do the nova part11:41
dvo-plv_I have a question regarding neutron extension11:42
sean-k-mooneysure11:42
sean-k-mooneywe might be able to use the neutron port tag extention by the way 11:43
sean-k-mooneybut we need a way to say either that this port need multi queue or it need multi queue with at least X queues11:43
sean-k-mooneywe could initally skip that i guess11:45
sean-k-mooneyand just use hw_vif_multiqueu11:45
sean-k-mooneybut i fear that woul break things so i think that would be a bad idea11:45
dvo-plv_maybe it will more logical to extend neutron port with some additional parameter openstack port create --network net10 < --queues=2 or --binding-profile queues=2 > ... port10 11:45
sean-k-mooneyno binding profie is not a user facing field11:45
sean-k-mooneyit is for nova to pass info to the network backend11:46
sean-k-mooneyit shoudl never be written by a human or neutron11:46
dvo-plv_use flavor or image is not a soklution, because we could have vm with 2 port and differnet queues number. or leave it like a limitation11:46
sean-k-mooneyya so this is whyi think we need a per port solution which means a neutron api extenion 11:47
sean-k-mooneyor reusing an exsiting one11:47
sean-k-mooney--binding-profile cant be used as makign that writabel is a security risk11:47
dvo-plv_sorry, Im not familiar with it. Do you means that https://docs.openstack.org/neutron/latest/contributor/internals/api_extensions.html11:48
sean-k-mooneykind of so https://github.com/openstack/neutron-lib/tree/master/neutron_lib/api/definitions are all the api extenstiosn that neutron supprots11:49
sean-k-mooneythe binding profile for example is part of the portbindings api extention https://github.com/openstack/neutron-lib/blob/master/neutron_lib/api/definitions/portbindings.py#L31-L3411:50
dvo-plv_okay, I will investiagate it. regarding this extension I should create rfe and talk with ralonsoh regarding t hat , right ?11:50
sean-k-mooneyyes11:51
sean-k-mooneyi see two possibale approches11:51
sean-k-mooneyeither model this as part fo the QOS extentions11:51
sean-k-mooneyor as a sepereate multiqueue extension11:51
sean-k-mooneyif we add a new one11:51
dvo-plv_Thank you, I have alot of work now11:55
sean-k-mooneyone exsiting api we might be able to use is https://docs.openstack.org/neutron/latest/contributor/internals/tag.html11:55
sean-k-mooneywe could use that for hw_vif_multiqueue=true|false for example on a per port basis11:56
sean-k-mooneythe issue is its currently a string field and we woudl really prefer it to be a key value field11:56
sean-k-mooneywell a dict of key values11:57
sean-k-mooneywe have 3 or 4 usecasue that woudl benifit form a v2 of this feature that wsa key value11:57
sean-k-mooneywe spoke to ralonsoh about that durign the ptg 11:58
sean-k-mooneyso tha might just be the best thign to do 11:58
sean-k-mooneythen you coudl do {nova_min_queues:2,nova_multiqueue:true} on a per port basis11:59
sean-k-mooneywe coudl also do use it for nova_delete_on_detach: true|false12:00
dvo-plv_but this parameter is relayted to the flavor and image12:01
ralonsohsorry, I'm having l;unch now12:09
ralonsohI'll read this channel later12:09
opendevreviewArtom Lifshitz proposed openstack/nova master: Fix pep8 errors with new hacking  https://review.opendev.org/c/openstack/nova/+/87451713:56
opendevreviewArtom Lifshitz proposed openstack/nova master: Fix pep8 errors with new hacking  https://review.opendev.org/c/openstack/nova/+/87451714:28
opendevreviewStephen Finucane proposed openstack/nova master: docs: Correct a typo, grammar in AZ doc  https://review.opendev.org/c/openstack/nova/+/88123515:20
« Thursday, 2023-04-20 Index Saturday, 2023-04-22 »

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!