Thursday, 2025-07-03

sean-k-mooney[m]i dont think we should supprot the dmi info at all01:45
sean-k-mooney[m]its a libvirt internal implation detail so im really not a fan of building on that for config drive or any other metadata mechanium01:46
sean-k-mooney[m]i certainly dont want to rush into adding supprot for this as a bug fix without propper discussion as this si an undocumjented internal detail01:47
sean-k-mooney[m]if your using virtio-scsi you can have 100s of disks if needed but iconfig drive does consume a pci/pcie sloct by default but iits not generally a problem01:51
mikalsean-k-mooney[m]: I finally have all the USB and sound changes passing CI, so that's nice.06:43
Ugglasean-k-mooney[m], fyi I did a first review pass on 953940: support iothread for nova vms | https://review.opendev.org/c/openstack/nova-specs/+/95394009:00
bauzassean-k-mooney[m]: honestly, I'd prefer to leave the current owner to provide a new revision from him09:25
mikalFor what its worth, https://review.opendev.org/q/topic:%22libvirt-vdi%22 is a series of four patches all with passing CI and a +2 on them if someone has a spare minute or two.09:54
sean-k-mooneybauzas: well i wanted to hevilaly revise there proposal, also i have propsoed thie in some form twice before so i wrote it form scratch based on my prior exploration of this in past cycles 10:01
opendevreviewAlexey Stupnikov proposed openstack/nova master: Don't create instance_extra entry for deleted instance  https://review.opendev.org/c/openstack/nova/+/41277110:38
opendevreviewRajesh Tailor proposed openstack/nova-specs master: Show finish_time field in instance action show  https://review.opendev.org/c/openstack/nova-specs/+/92978011:49
opendevreviewsean mooney proposed openstack/nova-specs master: support iothread for nova vms  https://review.opendev.org/c/openstack/nova-specs/+/95394012:10
opendevreviewBiser Milanov proposed openstack/nova master: Hardcode the use of iothreads for KVM.  https://review.opendev.org/c/openstack/nova/+/91866912:43
opendevreviewBiser Milanov proposed openstack/nova master: Hardcode the use of iothreads for KVM.  https://review.opendev.org/c/openstack/nova/+/91866912:43
opendevreviewBalazs Gibizer proposed openstack/nova master: Run nova-api and -metadata in threaded mode  https://review.opendev.org/c/openstack/nova/+/95195712:54
opendevreviewBalazs Gibizer proposed openstack/nova master: Warn on long task wait time for executor  https://review.opendev.org/c/openstack/nova/+/95266612:54
opendevreviewBalazs Gibizer proposed openstack/nova master: Allow to start unit test without eventlet  https://review.opendev.org/c/openstack/nova/+/95343612:54
opendevreviewBalazs Gibizer proposed openstack/nova master: Run unit test with threading mode  https://review.opendev.org/c/openstack/nova/+/95347512:54
opendevreviewBalazs Gibizer proposed openstack/nova master: [test]RPC using threading or eventlet selectively  https://review.opendev.org/c/openstack/nova/+/95381512:54
opendevreviewBalazs Gibizer proposed openstack/nova master: Do not yield in threading mode  https://review.opendev.org/c/openstack/nova/+/95099412:59
opendevreviewBalazs Gibizer proposed openstack/nova master: Run nova-api and -metadata in threaded mode  https://review.opendev.org/c/openstack/nova/+/95195712:59
opendevreviewBalazs Gibizer proposed openstack/nova master: Warn on long task wait time for executor  https://review.opendev.org/c/openstack/nova/+/95266612:59
opendevreviewBalazs Gibizer proposed openstack/nova master: Allow to start unit test without eventlet  https://review.opendev.org/c/openstack/nova/+/95343612:59
opendevreviewBalazs Gibizer proposed openstack/nova master: Run unit test with threading mode  https://review.opendev.org/c/openstack/nova/+/95347512:59
opendevreviewBalazs Gibizer proposed openstack/nova master: [test]RPC using threading or eventlet selectively  https://review.opendev.org/c/openstack/nova/+/95381512:59
sean-k-mooneybauzas: can you provide input on specifcly this comment https://review.opendev.org/c/openstack/nova-specs/+/953940/comment/c0db3aad_9199e6e7/ do you want me to move  hw:iothreads_per_disk to the alternitives or only update the iothread on reboot for this cycle.13:04
sean-k-mooneyill simplfy the spec based on that input.13:04
sean-k-mooneylong term we defintly shoudl supprot  hw:iothreads_per_disk btu we dont need to supprot it initally my expection is we will enhance the iothrad funciotnaltiy incremnetally over a few rleases as we need too.13:06
sean-k-mooneythe ohter thing i was undeceded on was shoudl it be  hw:iothreads_per_disk or  hw:iothreads_per_volume i.e. the number of addtional io treads to add per cinder volume or per any disk assocated with the guest i.e. swap or ephemreal13:08
sean-k-mooneyi think for simplictly to have a baseline this cycle we shoudl just simplfy and move all the per disk part to a diffent spec for next cycle like you were suggesting13:09
bauzassean-k-mooney: yeah my concern was about to say "given all the discussions we need to have for iothreads per disk, maybe we should just do that by another spec"13:15
bauzasand my main concern with that is the fact that we need to restart an instance for using other io threads if we modify the number when we add a new volume13:16
sean-k-mooneyso we dont actuuly need the restart as far as i can tell but we can just update the xml we need to call a sperat api to add or remove the iothread 13:18
sean-k-mooneyso ok ill move alll fo that to alterniivs and simplfy the spec to only static iothread set via the hw:iothreads extra spec13:19
bauzasoh, the libvirt API accepts a specific call to add a new iothread ?13:20
bauzasif so, tell that please :)13:20
sean-k-mooneyso you can do it on the cli to a live domain. i commented on that in my reply13:21
sean-k-mooneyso there is an api but i need to go find what virsh is calling13:21
sean-k-mooneybut we do need to track/assign the iothread id to use it13:21
sean-k-mooneyso we can use it but its more complexity then i would ike to do this cycle13:22
sean-k-mooneyim going to simplfy the cpu affintiy too.13:29
opendevreviewsean mooney proposed openstack/nova-specs master: support iothread for nova vms  https://review.opendev.org/c/openstack/nova-specs/+/95394013:51
opendevreviewRajesh Tailor proposed openstack/nova-specs master: Show finish_time field in instance action show  https://review.opendev.org/c/openstack/nova-specs/+/92978014:05
opendevreviewsean mooney proposed openstack/nova-specs master: support iothread for nova vms  https://review.opendev.org/c/openstack/nova-specs/+/95394014:30
opendevreviewsean mooney proposed openstack/nova-specs master: support iothread for nova vms  https://review.opendev.org/c/openstack/nova-specs/+/95394014:34
sean-k-mooneyUggla: bauzas ^ the only open quesiton left in that version is triat or compute service bump. a triat is simpler in my view and will be a better ux but i am not going to update the spec until folks provide feed back. i need to go test somethign else for a bit14:43
Ugglasean-k-mooney, I'm more in favor of trait, but I'm not sure I have rational behind that. I need to think about it.14:51
sean-k-mooneyi do it work for mixed clouds with diffent virt drivers14:51
sean-k-mooneyeither differnt version or vmware + libvirt or libvirt + ironic ectra14:52
sean-k-mooneywe can do it th eother way but to me its harder to implement correctly14:52
opendevreviewMasahito Muroi proposed openstack/nova-specs master: Add spec for virtio-blk multiqueue extra_spec  https://review.opendev.org/c/openstack/nova-specs/+/95163616:50
masahitoThanks for the quick reviews. I narrow down the spec scope only supporting multiqueue to enable the incremental supports and to simplify the discussion.16:55
sean-k-mooneymasahito: im sorry that we didnt really have tiem to work on this mroe closely earlier in the cycle16:58
sean-k-mooneymasahito: im not entrily sure what the right thing to do is. technial the approval deadline is end of buisse today16:59
sean-k-mooneyill try and take a look at your spec again shortly17:00
sean-k-mooneybtu im not sure if other will have time today17:00
masahitoit's ok. I understand the iothreads part needs extra discussion now because of the entire Nova consistency.17:09
opendevreviewMerged openstack/nova master: Use futurist for _get_default_green_pool()  https://review.opendev.org/c/openstack/nova/+/94807217:11
dansmithsean-k-mooney: my request is that instead of you proposing a competing spec, you propose a revision on top of masahito's for whatever you think needs to be different17:11
masahitoand if we keep talking the iothreads part we could merge it early timing of G development cycle even though the spec is slipped from F release.17:11
sean-k-mooneydansmith: they are not descibign very diffent things17:11
dansmithit will make it much easier to understand any differences and avoid the appearance that we're going to choose one or the other17:11
dansmithsean-k-mooney: cool, then should be easy17:11
sean-k-mooneydansmith: masahito spec if for block multi queue only and mine is only for iothreads17:12
sean-k-mooneyand the two compose together17:12
sean-k-mooneydansmith: masahito's orginally did both in a more adanced way the either not propose17:13
masahito^ iothreads part means mapping disk to iothread.17:13
dansmithsean-k-mooney: okay I only skimmed them but I thought yours was proposing both17:14
sean-k-mooneyno i was going to rely on the fact multi is enable by default by qemu for now and focus only on the iothrad part17:15
dansmithmulti-queue has the same concerns for blk as eth I assume, where you have to scale them appropriately without using up all the .. what is it, pci root ports or something?17:15
sean-k-mooneyno17:15
masahitoyup. sean's spec gives me clear view of road to goal.17:15
sean-k-mooneyso multi queue for blk is on by default but you get i think 1 queue per vcpu? masahito  is that correct17:15
sean-k-mooneydansmith: for networkign the problem was that on the host side all the quese were used by ovs17:16
sean-k-mooneydansmith: but you and to enable them manually in the guest driver17:16
sean-k-mooneyabout 2-3 year ago the virtio net driver in the guest was updated to enabel all the queue if the nic has more then one17:16
sean-k-mooneyso you dont actully need to do anyting to have netowrking work propelry with a moderen kernel17:17
masahitoright. 1 queue per vcpu is default from 5.217:18
dansmithokay so blk multiqueue on its own gets you more queues, but limited actual parallelism by the default of $vcpu threads to service them,17:19
dansmithand iothreads lets you increase that past that number if you want, say 16 queues, 2 vcpus, and 16 io threads to serve those queues?17:19
sean-k-mooneydansmith: no by default it create a virtual device with 1 queue pre vcpu but all the io is handel by the single emulator thread17:20
sean-k-mooneydansmith: so the only thing it gives you is lock free reade/write in the guest kernel17:20
sean-k-mooneyif you mape one queue to each core or somethign like that17:20
dansmithby default, I understand17:20
dansmither, well, okay,17:21
dansmithby default one queue per vcpu, all funneling through a single thread shared with emulator right?17:21
sean-k-mooneyyep17:21
dansmithso two devices, four vcpus means 8 queues for one thread shared with emulator to handle17:22
sean-k-mooneyyep17:22
dansmithso even more iothreads with the default queues will improve performance17:22
sean-k-mooneymasahito can correct me if im wrong but what they ahve observied is if you have too many queue it does not actully help with perfoamce17:22
dansmithI'm not sure why multiple queues per device would really help with performance if you don't increase threads, but maybe it's because block devices are scheduled and have barriers unlike nics?17:23
sean-k-mooneyso if you ahve a vm with 64vcpus and 4 qeues it can perfroae better the 64Qs in some cases17:23
masahitoright for disk performance, not network performance.17:23
sean-k-mooneydansmith: masahito  want to decouble the queu count form teh vcpu count manily to reduce the queu count on large vms and i gues increase it on really small ones17:24
dansmithah, okay I see that in the spec but I glossed over it before17:25
sean-k-mooneybut the reall perfromace uplift is expcted to come form the addtion of iothreads 17:25
masahito2 or 4 queue with 1 emulator thread performance is better than 1 queue. But in case of large CPU flavor, 16 or more, the number of queues seems to be too much for single threads.17:25
dansmithyeah okay, I had a different impression from reading these two things, but I see why we'd need both now17:25
masahitoThe IO performance doesn't increase even though the number of queue is increased. 1 queue and 1 emulator thread is better performance than 16 queues and 1 emulator thread in case of 16 cores17:26
sean-k-mooneyso multi queu for disks results in one kernel vhost thread per queue if i recall, at least with ovs-dpdk it allwos more dpdk cores to process packtest as it did a rount robbin of the quce across teh aviabel dpdk cores.17:28
sean-k-mooneybut for block queue we dont get that automcati implict scaling17:28
sean-k-mooneyi think that partly becasue there is no vhost offload for block devices 17:30
sean-k-mooneythe main benifit to havign a multi queu dis is it allwos the use of the multi quei disk io schduelr in the kernel to do io priotisitation17:31
sean-k-mooneyso you can use mq-deadline or bfq or kyber in the guest ectra17:32
dansmithyeah, that's what I meant above in terms of benefit for the scheduled block requests17:32
sean-k-mooneyack17:32
dansmithsean-k-mooney: can we not combine these, at least at the trait layer, to allow advertising and requesting these blk queues and threads?17:33
dansmithI know the threads are useful for more than just blk, but if we can assume that threads mean you can do the blk queue that might be nice17:33
sean-k-mooneywe can we can also combine the spec if we really want ot at this point17:33
dansmithjust hate the trait explosion we get17:34
sean-k-mooneyorgially masahito goal was much more advance then the combidnaiton of these two spec, we can still evlove to that goal but the intent was to reduce the scope to soemthign achivable this cycle17:34
sean-k-mooneydansmith: if we can do both this cycle i dont se why we would need 2 traits17:36
sean-k-mooneyso if we put the multi queue enhacement on top of the iotreads one when we are writign the code for this i think we can resue it to mean both17:37
dansmithyeah17:37
sean-k-mooneythe multi queu change will now be very small so im not worried about that missing17:37
dansmithI don't really see how we can get these both merged this cycle at this point17:37
dansmithI'm out for the next ten days starting tomorrow17:38
dansmithI dunno how others feel17:38
dansmithgibi: just noticed you rechecking for rabbit reasons again... any chance we're mucking with things that are causing us to stall a real thread or something?18:12
dansmithI mean, I realize I should know as well as you, but..18:12
dansmithare we seeing those same failures elsewhere?18:12
clarkbdansmith: have a link to an example failure?18:13
clarkb(I want to rule one thing out or in)18:13
dansmithclarkb: https://review.opendev.org/c/openstack/nova/+/948079?tab=change-view-tab-header-zuul-results-summary18:13
dansmithnot sure which of those fails is the one he rechecked for18:14
dansmith(man we have /got/ to squelch the verbose tracebacking from os-brick)18:15
clarkbeach of the three failures were multinode jobs and zuul-launcher (the new nodepool replacement) gave you mixed cloud provider nodes. Its only really supposed to do this when there is no other choice, but there was at least one bug causing it to do that too often that should be fixed now18:16
clarkbthose jobs ran ~5 hours ago so the fix for that particular problem should've been in place.18:16
dansmithah18:16
clarkbBut anyway if your multinode job can't handle that (due to ip addressing or latency or whatever) that could be a reason18:16
dansmithso, nova officially doesn't support cross-site MQ, even though lots of people do it18:19
dansmithredhat has strict latency requirements for when we allow people to do it18:19
dansmithgetting nodes from multiple providers would seem to make that kindof difficult18:19
clarkbI've brought up that the mixed nodesets are still happening in #opendev as we're trying to track zuul-launcher behaviors that are unexpected or cause problems there18:20
clarkbya its not the default18:20
dansmithack18:20
clarkband I think the vast majority of the time it isn't happening that way (opendev's own multinode jobs testing deployment of our software is also sensitive to it and I noticed immediately when it was doing it too often. Since I think I'e only seen one failure on our side btu we also run fewer jobs than nova and have fewer changes)18:21
dansmithin general, we're a lot more tolerant of it than I would have thought, so it's probably not really a problem in a lot of cases (and maybe good to be throwing some of it in there)18:24
dansmithI'm just saying, it's definitely not what we expect to happen18:24
dansmithlooks like this is the fail he was referring to: https://zuul.opendev.org/t/openstack/build/66557ea92bae4e809c09526eba619c07/log/compute1/logs/screen-n-cpu.txt18:25
dansmithwhich is just fully unable to talk to rabbit18:25
clarkbwhich may be an issue of ip routing18:25
clarkbdepending no how the ip addrs are configured. By default we'd allow the traffic to cross the internet and get from point a to point b but you have to use the public ips on both sides which may be floating ips etc18:26
dansmithand you think that our jobs might be misconfigured for that?18:26
dansmithI don't _think_ nova-next does a ton of weird config18:27
opendevreviewMasahito Muroi proposed openstack/nova-specs master: Add spec for the network id query in server list and server details  https://review.opendev.org/c/openstack/nova-specs/+/93882318:27
clarkbyes its possible they use private ips ratehr than public ips18:28
clarkbor listen only on the local ip address so connections for the floating ip are rejected18:28
dansmithtransport_url = rabbit://stackrabbit:secretrabbit@10.209.32.130:5672/18:28
clarkbthat'll do it18:28
dansmithokay so I don't think that's specific to nova-next here18:29
dansmithwell, it does make some choices about neutron so maybe it's wrapped up in that18:32
sean-k-mooneyits been a very long time but we used tc when i was at intel to injectup to 120ms of latency into the links between host18:51
sean-k-mooneyfor the most part that seam to still be fine provide the clocks were in sync18:51
dansmithyeah it's clearly not routable because of the private ip in this case18:51
dansmithuep18:51
sean-k-mooneyah its routing ya18:52
sean-k-mooneythat woudl make sense. i know we try and setbale a vxlan or gre mesh usign linux bridge in the multi node jobs18:52
sean-k-mooneyand hten use that for neutron tenant networking18:52
sean-k-mooneybut i dont know if we actuly route all traffic over that18:52
sean-k-mooneyi guess we are not or that  is not working properly if we are18:53
dansmithit's not that because this is just nova-compute unable to contact the mq on the controller node because it's not using its public ip18:54
dansmithnothing to do with guest networking18:54
sean-k-mooneywhat i ment was if we used the ip form that brdige network those might work18:55
sean-k-mooneybut ya in devstack we lookup the ip of the node by defualt18:55
sean-k-mooneyi dotn think we overried that in the jobs18:55
dansmithright18:55
sean-k-mooneywe expct the vms to be on the same neutron network in the josb since nodes in any given nodeset are nto allwos to comefrom diffent providers18:56
sean-k-mooneyor at least they were not allowed to do tha tin the past so the josb depend on that18:56
sean-k-mooneychangign that would be a breakign change18:56
sean-k-mooneynot nessiarly a bad one but not one that our jobs can handel so it woudl need ot be opt in18:56

Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!