Monday, 2021-03-08

« Sunday, 2021-03-07 Index Tuesday, 2021-03-09 »
*** zzzeek has quit IRC00:17
*** zzzeek has joined #openstack-nova00:18
*** LinPeiWen has joined #openstack-nova00:38
*** tosky has quit IRC00:42
*** ricolin has quit IRC00:46
*** ricolin has joined #openstack-nova00:50
*** benj_ has joined #openstack-nova00:54
*** benj_ has quit IRC01:01
*** songwenping_ has quit IRC01:01
*** benj_ has joined #openstack-nova01:02
*** rcernin has quit IRC01:12
*** rcernin_ has joined #openstack-nova01:12
*** benj_ has quit IRC01:17
*** benj_ has joined #openstack-nova01:17
*** swp20 has joined #openstack-nova01:34
*** benj_ has left #openstack-nova02:00
*** benj_ has joined #openstack-nova02:14
*** k_mouza has joined #openstack-nova02:15
*** spatel_ has joined #openstack-nova02:16
*** k_mouza has quit IRC02:19
*** LinPeiWen has quit IRC02:32
*** spatel_ has quit IRC02:43
*** zzzeek has quit IRC02:44
*** zzzeek has joined #openstack-nova02:47
*** LinPeiWen has joined #openstack-nova02:49
*** vishalmanchanda has joined #openstack-nova03:00
*** zzzeek has quit IRC03:10
*** zzzeek has joined #openstack-nova03:12
*** rcernin_ has quit IRC03:14
*** rcernin_ has joined #openstack-nova03:21
*** LinPeiWen has quit IRC03:28
*** psachin has joined #openstack-nova03:42
*** jamesdenton has quit IRC03:58
*** jamesden_ has joined #openstack-nova03:58
*** zzzeek has quit IRC04:02
*** zzzeek has joined #openstack-nova04:04
*** prometheanfire has quit IRC04:07
*** prometheanfire has joined #openstack-nova04:07
*** khomesh24 has joined #openstack-nova04:24
*** k_mouza has joined #openstack-nova04:50
*** k_mouza has quit IRC04:55
*** ratailor has joined #openstack-nova05:05
*** LinPeiWen has joined #openstack-nova05:24
*** zzzeek has quit IRC05:26
*** zzzeek has joined #openstack-nova05:29
*** k_mouza has joined #openstack-nova05:42
*** k_mouza has quit IRC05:47
*** zzzeek has quit IRC06:03
*** zzzeek has joined #openstack-nova06:05
*** LinPeiWen has quit IRC06:05
*** khomesh24 has quit IRC06:24
*** ratailor_ has joined #openstack-nova06:41
*** ratailor__ has joined #openstack-nova06:44
*** ratailor has quit IRC06:44
*** ratailor_ has quit IRC06:48
*** ratailor_ has joined #openstack-nova06:52
*** ratailor__ has quit IRC06:57
*** slaweq has joined #openstack-nova07:10
*** rcernin_ has quit IRC07:14
*** ralonsoh has joined #openstack-nova07:23
*** lpetrut has joined #openstack-nova07:24
*** songwenping_ has joined #openstack-nova07:34
*** swp20 has quit IRC07:38
*** fnordahl has quit IRC07:39
*** fnordahl has joined #openstack-nova07:39
*** k_mouza has joined #openstack-nova07:43
*** k_mouza has quit IRC07:48
*** tesseract has joined #openstack-nova07:53
*** LinPeiWen has joined #openstack-nova07:57
*** andrewbonney has joined #openstack-nova08:15
*** rpittau|afk is now known as rpittau08:23
*** whoami-rajat_ has joined #openstack-nova08:26
*** khomesh24 has joined #openstack-nova08:28
*** tosky has joined #openstack-nova08:34
*** brinzhang has joined #openstack-nova08:34
*** ircuser-1 has joined #openstack-nova08:35
*** brinzhang0 has quit IRC08:36
*** martinkennelly has joined #openstack-nova08:50
*** liuyulong has joined #openstack-nova08:51
*** lucasagomes has joined #openstack-nova08:57
openstackgerritLee Yarwood proposed openstack/nova master: nova-next: Start testing the q35 machine type  https://review.opendev.org/c/openstack/nova/+/70870108:59
lyarwoodstephenfin: thanks for asking for the release, I guess they sorted out their issues between each of us asking09:00
lyarwoodstephenfin: of Cirros that is, I've updated the nova-next change but it's going to depend on getting 0.5.2 cached first09:00
*** liuyulong has quit IRC09:03
*** nautik has joined #openstack-nova09:19
*** derekh has joined #openstack-nova09:22
*** links has joined #openstack-nova09:26
*** ociuhandu has joined #openstack-nova09:27
*** jraju__ has joined #openstack-nova09:30
*** Luzi has joined #openstack-nova09:31
*** links has quit IRC09:34
*** songwenping__ has joined #openstack-nova09:34
*** k_mouza has joined #openstack-nova09:37
*** songwenping_ has quit IRC09:37
* bauzas apologies for being on and off this morning but eventually I'm just getting an optic fiber today09:40
bauzas(FTTH soldering)09:40
*** k_mouza has quit IRC09:42
*** ociuhandu has quit IRC09:44
*** ociuhandu has joined #openstack-nova09:44
*** ociuhandu has quit IRC09:50
openstackgerritLee Yarwood proposed openstack/nova stable/train: compute: Lock by instance.uuid lock during swap_volume  https://review.opendev.org/c/openstack/nova/+/75873309:50
openstackgerritLee Yarwood proposed openstack/nova stable/stein: compute: Lock by instance.uuid lock during swap_volume  https://review.opendev.org/c/openstack/nova/+/75873409:51
*** ociuhandu has joined #openstack-nova09:51
openstackgerritLee Yarwood proposed openstack/nova stable/queens: compute: Lock by instance.uuid lock during swap_volume  https://review.opendev.org/c/openstack/nova/+/75873609:51
openstackgerritLee Yarwood proposed openstack/nova stable/rocky: compute: Lock by instance.uuid lock during swap_volume  https://review.opendev.org/c/openstack/nova/+/75873509:52
nautikHello guys, I have a quick question: I see keypair names only allow "_- " as special characters, and I have a lot of users trying to input a name with @ in it (default suggested by ssh-keygen I think)09:53
nautik(code is here https://opendev.org/openstack/nova/src/branch/master/nova/compute/api.py#L6343)09:53
nautikwould anyone know if there is a reason not to allow the '@' character? Or could this be suggested?09:53
nautikI found this commit introducing the check https://github.com/openstack/nova/commit/c8b0a9a3be7ca276d91d470a629fdd020981299309:54
nautikwhich fixes an issue about / (https://bugs.launchpad.net/nova/+bug/937408)09:54
openstackLaunchpad bug 937408 in OpenStack Compute (nova) "Can't delete keypair with a slash" [Medium,Fix released] - Assigned to Justin Shepherd (jshepher)09:54
gibibauzas: good for you :)09:55
nautikbut appearently users are commonly trying to put '@' in their key name09:56
gibinautik: based on the bug you linked the issue was forward slash in the name '/'. So we could try to see if we can make @ work09:57
gibinautik: if you have time to propose a patch that allows @ along with some functional test then I will be happy to review it09:57
nautikok cool, thank you09:58
*** k_mouza has joined #openstack-nova09:58
*** mugsie__ is now known as mugsie10:11
*** jangutter has quit IRC10:41
*** jangutter has joined #openstack-nova10:42
*** jangutter has quit IRC10:43
*** jangutter has joined #openstack-nova10:44
elodlyarwood: fyi, train and older branches are blocked until these aren't merged: https://review.opendev.org/q/Id62e91b1609db4b1d2fa425010bac1ce77e9fc5110:50
lyarwoodelod: ack was about to look at the failures of my earlier swap_volume instance.uuid backports10:50
lyarwoodelod: thanks for the heads up10:50
elodnp10:50
lyarwoodI guess it's yet another reverse order backport because of grenade?10:53
elodonly stein needs to be merged prior to train10:59
elodstein can be merged directly10:59
lyarwoodah of course, grenade doesn't vote prior to that11:00
*** ociuhandu has quit IRC11:04
*** ociuhandu has joined #openstack-nova11:06
*** jangutter has quit IRC11:07
*** jangutter has joined #openstack-nova11:07
*** ociuhandu has quit IRC11:10
*** khomesh24 has quit IRC11:24
*** derekh has quit IRC11:25
*** derekh has joined #openstack-nova11:27
*** jangutter has quit IRC11:29
*** jangutter has joined #openstack-nova11:30
*** ociuhandu has joined #openstack-nova11:36
admin0hi all .. what scheduler controls --property quota:disk_io_limit for a ceph backend in nova11:41
admin0so that io limits can be properly enforced11:41
*** jangutter has quit IRC11:42
*** jangutter has joined #openstack-nova11:42
*** ociuhandu has quit IRC11:47
gibiadmin0: based on the code it is only used by the vmware virt driver11:47
sean-k-mooneyadmin0: the io filter limits the scudling based on higlevel io ops11:49
sean-k-mooneylike building a vm or snapshots11:49
sean-k-mooneyit does not look at disk iops or bandwith11:50
sean-k-mooneyi assume that is what you wanted to enforce or loadblance on?11:50
*** mgariepy has quit IRC11:52
*** ociuhandu has joined #openstack-nova11:56
*** ociuhandu has quit IRC11:56
*** ociuhandu has joined #openstack-nova11:58
*** ociuhandu has quit IRC11:58
*** ociuhandu has joined #openstack-nova11:58
*** ociuhandu has quit IRC11:58
*** ociuhandu has joined #openstack-nova12:00
*** ociuhandu has quit IRC12:06
sean-k-mooneyadmin0: disk_read_iops_sec disk_write_iops_sec and disk_total_iops_sec are supported in the libvirt driver by the way12:09
sean-k-mooneyadmin0: e.g.  --property quota:disk_total_iops_sec=50012:09
sean-k-mooneyadmin0: there is no scheduling based on this and instead it enforece a limit on the amount of iops that a vm can do.12:10
admin0sean-k-mooney, https://gist.github.com/a1git/35020a03a3870741af463fc18e1e811c12:12
admin0i have the limit set at 80012:12
admin0but i am getting a lot and its not hitting any limit12:12
admin0or that i don't know how to test this using fio12:12
sean-k-mooneywhat virt driver are you using12:13
sean-k-mooneyalso you cant change that after the vm is booted12:13
sean-k-mooneyso after you do the flavor set you need to create a new vm with that flavor12:13
*** ociuhandu has joined #openstack-nova12:13
sean-k-mooney"openstack flavor set g6 --property quota:disk_total_iops_sec=800" would be what you shoudl set if you are using libvirt12:14
*** ociuhandu has quit IRC12:22
*** nightmare_unreal has joined #openstack-nova12:41
nightmare_unrealhello is there a way to get statistics of live-migration ? Like what was the avg disk transfer rate, how much time it took etc.12:42
sean-k-mooneyhistorical migrationg i dont think so but you can get stats form the ongoing migration at least form libvirt12:42
sean-k-mooneyposibly also form the api but im not sure how much detail we have in the migration status12:43
*** lpetrut has quit IRC12:44
*** artom has joined #openstack-nova12:45
*** iurygregory has quit IRC12:46
lyarwoodnightmare_unreal: https://docs.openstack.org/api-ref/compute/?expanded=show-migration-details-detail#show-migration-details12:46
lyarwoodnightmare_unreal: we don't list the average transfer rate etc12:47
sean-k-mooneyah we do have disk_remaining_bytes etctra in that responce good12:47
lyarwoodnightmare_unreal: iirc it is dumped in the logs12:47
nightmare_unrealah then i can calculate i think12:47
nightmare_unrealdisk remaining / total disk12:47
lyarwoodyup12:48
nightmare_unrealthanks :D12:48
sean-k-mooneyam not entirly12:48
sean-k-mooneythe byts migrated can be larger then total disk12:48
lyarwoodright and remaining can also move around12:49
sean-k-mooneydisk_processed_bytes can be larger then disk_total_bytes if the vm is wriging to the disk while its being migrated12:49
sean-k-mooneyyou can get a microsoft style progress bar where it sometiems gose backwared or time remaing goes up as the persentage completeion increases if your try to use disk_remaining_bytes/disk_total_bytes to caulate things12:51
sean-k-mooneythe data is not meaning less but unless the guest is idel it wont be linear12:51
lyarwoodlol12:51
*** ociuhandu has joined #openstack-nova12:52
nightmare_unrealahah I see12:53
nightmare_unreali will keep that in mind.12:53
*** ociuhandu has quit IRC12:56
*** bauzas has quit IRC12:59
sean-k-mooneygibi: i have a pe8 issue in my port numa policies patch so ill fix the doc issue when i repin it shortly13:06
gibisean-k-mooney: ack13:06
sean-k-mooneyby the way did you want me to add a new doc for this or just fix the comment13:07
sean-k-mooneyi could add a doc too but did you have a specific place in mind?13:07
gibisean-k-mooney: just add a doc to the function about the item in the returned tuple13:07
sean-k-mooneyah cool. i was wondering if you wanted a user facing doc. i could add one of those as a followup but porbly need to think where that should go. ill fix the comment thanks:)13:08
*** lpetrut has joined #openstack-nova13:09
*** psachin has quit IRC13:10
*** mgariepy has joined #openstack-nova13:10
*** __ministry1 has joined #openstack-nova13:26
gibisean-k-mooney: regarding the user facing doc, I hope neturon has at least an API doc for this port parameter13:29
gibi :)13:29
*** ociuhandu has joined #openstack-nova13:29
sean-k-mooneyi was thinking maybe we should add something to the networking guide13:30
gibiyeah that would be good to have ^^13:30
sean-k-mooneythey did not add the osc command but i submited a patch to do that13:30
gibithe api ref is not missing but pretty short13:31
sean-k-mooneyso ill submit a follow up patch to the networkign guide to use that13:31
gibi"The port NUMA affinity policy requested during the virtual machine scheduling. Values: None, requiered, preferred or legacy."13:31
sean-k-mooneyya i can proably extend that too and explain what the policies do13:31
gibihm and there is a typo in it13:31
sean-k-mooneywe have the explaniaton in nova but it better to not need to cross reference13:31
gibirequiered -> required13:31
sean-k-mooneyoh yes there is.13:32
sean-k-mooneyonly in the doc however13:32
gibialso when artom's series lands with the socket value we need to document that for the port attribute too13:32
sean-k-mooneyyep although that requires a neutron change to extend the db enum. its just a sting field but the model need to be extended13:33
*** zoharm has joined #openstack-nova13:33
gibiI see13:33
gibigood to know that13:33
*** jangutter has quit IRC13:34
gibiI assumed it is an unrestricted string in neutron13:34
gibibut the it is better13:34
*** ociuhandu has quit IRC13:34
*** jangutter has joined #openstack-nova13:34
sean-k-mooneyi think in terms fo the db coloum defiention yes but they are validating it in python13:34
*** jraju__ has quit IRC13:37
sean-k-mooneythe only thing that is slightly unfortunete is they enabled each policy with a seperate flag in osc13:37
sean-k-mooneyrather then a vaule that takes a sting paramter but its not a huge issue13:38
*** ociuhandu has joined #openstack-nova13:39
admin0sean-k-mooney, i am using kvm ..  first the flavor was created and the limit was set .. after that the instance was booted up13:42
sean-k-mooneyadmin0: yes but you use a vaule that is only supproted for vmware13:43
sean-k-mooneyadmin0: try the same with the one i pointed too13:43
*** jangutter has quit IRC13:44
admin0oh13:44
admin0i get it13:44
admin0thank you13:44
sean-k-mooneyopenstack flavor set g6 --property quota:disk_total_iops_sec=80013:44
admin0i will retry13:44
*** jangutter has joined #openstack-nova13:44
sean-k-mooneythat one and the others in the disk tuning section of https://docs.openstack.org/nova/latest/user/flavors.html work with libvirt13:45
openstackgerritMerged openstack/nova master: Replace md5 for fips  https://review.opendev.org/c/openstack/nova/+/77768613:46
openstackgerritMerged openstack/nova master: Centralize sqlite FK constraint enforcement  https://review.opendev.org/c/openstack/nova/+/77383313:47
openstackgerritsean mooney proposed openstack/nova master: support per port numa policies with sriov  https://review.opendev.org/c/openstack/nova/+/77379213:57
sean-k-mooneyalex_xu: stephenfin  would either of ye have time to review ^ gibi made the doc update and fixed the pep8 issue.13:58
gibisean-k-mooney: thanks +213:59
sean-k-mooneyjust lookign at your sqlite change that really is just refactoring it so that its cenratlised since we already had the dynamic logic. do we run with 3.7+ in the gate?14:00
sean-k-mooneyi assume we have no way to requrie a minium of 3.7 beyond moving to an os that provides it14:01
gibisean-k-mooney: I think you are referring to melwitt's change :)14:01
sean-k-mooneyhttps://review.opendev.org/c/openstack/nova/+/77383314:01
sean-k-mooneythe one that just merged14:01
gibiyepp14:02
gibithat one14:02
sean-k-mooneyoh did melwitt write that oh she did sorry still find part of the new gerrit confusing14:02
*** ratailor_ has quit IRC14:02
gibiwe only have https://github.com/openstack/requirements/blob/master/bindep.txt#L18 in bindep about sqlite14:02
gibiso I guess it depends on the host os14:03
*** mnaser has quit IRC14:03
*** mnaser has joined #openstack-nova14:03
*** mnaser has quit IRC14:03
*** mnaser has joined #openstack-nova14:03
sean-k-mooneyya which is fine i guess14:03
sean-k-mooneyaslong as it compatible with our version of sqlalchmy then it should not impact use much since its mainly for testing14:04
gibiyeah, it is just test dep14:04
sean-k-mooneyah yes i rememebr the downstream bug melwitt is fixing14:04
sean-k-mooneythe partal archiving of db rows was causing odd behavior for the simple tenant usage api on our internal cloud14:05
*** bauzas has joined #openstack-nova14:05
gibiyes I just read the fix today for that14:06
sean-k-mooneymelwitt: were you going to bring up the idea of using cacade delelets in  the ptg14:06
sean-k-mooneygibi: ya melwitt  is fixing it in a way that potentally backportable. an alternitive is so use a cascading delete trigger and let the db do it atomically14:07
sean-k-mooneybut there were reasons we dont use those in nova in the past14:08
sean-k-mooneyneutron does use them but i dont think thye have shadow tables14:08
gibibut we dont delete here we archive here14:08
sean-k-mooneythe issue downstream was beacuse we archive and purge at the same time14:09
*** amodi has joined #openstack-nova14:09
sean-k-mooneyso when the purge is happening with a limit on the rows it can cause half deleted recored until the next time the purge happens14:09
gibiwe we replace archiving with direct cascad delete that could work14:09
gibis/we we/if we/14:09
sean-k-mooneyya there has been some work done by ovh to do archiving externally with https://github.com/ovh/osarchiver14:10
sean-k-mooneythey are going to contbute that to either os-opstools or to a new openstack/ repo14:11
sean-k-mooneyif that was adopted instead it could allow us to remove archiving form the project entirly14:11
sean-k-mooneyproably somethign we shoudl discuss at the ptg or on the maining list14:12
sean-k-mooneythey developed it orginally for manila which dose not have shadow tabels and then started using it for all projects in production to have common way to do it14:13
openstackgerritLee Yarwood proposed openstack/nova master: WIP zuul: Replace grenade and nova-grenade-multinode with grenade-multinode  https://review.opendev.org/c/openstack/nova/+/77888514:15
openstackgerritLee Yarwood proposed openstack/nova master: zuul: Remove nova-dsvm-multinode-base  https://review.opendev.org/c/openstack/nova/+/77890814:15
openstackgerritLee Yarwood proposed openstack/nova master: zuul: Create and use nova-libvirt-irrelevant-files  https://review.opendev.org/c/openstack/nova/+/77890914:15
*** __ministry1 has quit IRC14:16
*** jangutter has quit IRC14:16
*** spatel_ has joined #openstack-nova14:17
*** jangutter has joined #openstack-nova14:17
lemkoHi, since I upgraded my openstack to Victoria (with upgrade of libvirt), my instances are randomly dying for no reason... "shutting down, reason=crashed" from qemu. Otherwise I can see this from nova-compute "Instance shutdown by itself. Calling the stop API. Current vm_state: active, current task_state: None, original DB power_state: 1, current VM14:20
lemkopower_state: 4". Any idea?14:20
sean-k-mooneylemko: did you check the qemu instance log14:21
lemkoYes.14:21
sean-k-mooneylemko: /var/lib/libvirt/qemu/instace*14:21
sean-k-mooneywere there any errors in the log14:21
lemko"shutting down, reason=crashed" is the only line interesting14:21
sean-k-mooneyok that looks like a qemu or kvm issue then not openstack14:22
sean-k-mooneydid you have any OOM erros in dmesg/journalctl14:22
sean-k-mooneyits possible the kernel is killing them if the host ran out of memory14:22
sean-k-mooneybut more likely you are hitting a qemu bug14:22
*** songwenping_ has joined #openstack-nova14:23
*** songwenping__ has quit IRC14:27
lemkoI do not see anything related to OOM.14:27
sean-k-mooneythere are some more detailed logging option you can enabel for libvirt/qemu to help debug kashyap has a write up14:28
kashyapWe can't tell yet what bug it is, without more information on the error / crash14:28
sean-k-mooneykashyap: do you have teh link to how to enable the debug logging for qemu instances14:29
kashyaplemko: Try this, if you can use 'virt-admin' tool -- https://kashyapc.fedorapeople.org/virt/virt-admin.txt14:29
kashyapsean-k-mooney: --^14:29
sean-k-mooneythanks :)14:29
kashyapsean-k-mooney: lemko: Another option, the manual way (if you don't have 'virt-admin' tool): https://kashyapc.fedorapeople.org/virt/openstack/request-nova-libvirt-qemu-debug-logs.txt14:29
*** links has joined #openstack-nova14:30
kashyaplemko: Just to eliminate one line of debugging: you're not using nested KVM, are you?  I.e. your compute node is baremetal, yes?14:31
kashyaplemko: If you're the second URL I posted, just go straight to the bottom, this section: "To manually enable libvirt log filters"14:32
*** songwenping__ has joined #openstack-nova14:37
admin0sean-k-mooney, thank you .. it works as required14:38
*** songwenping_ has quit IRC14:41
openstackgerritLee Yarwood proposed openstack/nova master: libvirt: Create qcow2 disks with the correct size without extending  https://review.opendev.org/c/openstack/nova/+/77927514:45
*** Luzi has quit IRC14:49
*** bauzas has quit IRC14:49
*** songwenping_ has joined #openstack-nova14:50
*** bauzas has joined #openstack-nova14:51
*** songwenping__ has quit IRC14:53
*** k-s-dean has joined #openstack-nova15:08
*** derekh has quit IRC15:08
*** bauzas has quit IRC15:11
*** links has quit IRC15:11
*** bauzas has joined #openstack-nova15:12
kashyapThe new Gerrit has been significantly slower for me; not sure if it's just me15:12
sean-k-mooneyits because of the db storage in git15:13
sean-k-mooneyit increase io on the server side15:13
*** songwenping_ has quit IRC15:14
sean-k-mooneyat least that is my understanding15:14
*** songwenping_ has joined #openstack-nova15:14
*** tesseract has quit IRC15:19
*** zzzeek has quit IRC15:19
*** tesseract has joined #openstack-nova15:19
*** zzzeek has joined #openstack-nova15:21
kashyapI see15:21
artomI need to clear my email filters and restart them from scratch15:26
artomIt's been incorrectly flagging actual people reviews as third party CIs, and marking them read15:27
artomLeading to me missing reviews15:27
*** lpetrut has quit IRC15:32
*** ociuhandu has quit IRC15:37
*** ociuhandu has joined #openstack-nova15:37
openstackgerritClaudiu Belu proposed openstack/nova master: live migration: Pass limits onto scheduler-chosen destination  https://review.opendev.org/c/openstack/nova/+/77928515:38
*** claudiub has joined #openstack-nova15:39
openstackgerritClaudiu Belu proposed openstack/nova master: live migration: Pass limits onto scheduler-chosen destination  https://review.opendev.org/c/openstack/nova/+/77928515:40
claudiub^ solved my NUMA node 0 overcommit due to live-migration issue. :)15:42
*** ociuhandu has quit IRC15:43
sean-k-mooneyim not sure how that would solve your issue15:44
sean-k-mooneyclaudiub: its still not going to result in the vms decrementing resouces in the host numa toplogy blob15:44
*** k-s-dean has quit IRC15:47
*** ociuhandu has joined #openstack-nova15:47
*** ociuhandu has quit IRC15:47
*** ociuhandu has joined #openstack-nova15:48
claudiubbasically, just passing those limits we get from the scheduler to nova-compute. Those limits were None previously, so the CPU and memory allocation ratios were never checked, so NUMA node 0 was always a valid candidate for live-migration, even though it shouldn't if it's over the limit already. Now, when live-migrating, if the NUMA node 0 is full, they will be placed in NUMA node 1.15:48
sean-k-mooneyif you just use hw:numa_nodes=1 it will never be full15:49
sean-k-mooneyand you should have got that full behavior before15:49
claudiubit wasn't only full, it was overscheduled.15:50
sean-k-mooneyno hw:numa_nodes does not claim/decrement that avaiable resouce on the host15:50
sean-k-mooneywithout hw:cpu_policy=dedicated or hw:mem_page_size set15:50
*** jangutter has quit IRC15:50
*** jangutter has joined #openstack-nova15:51
*** macz_ has joined #openstack-nova15:52
sean-k-mooneyclaudiub: at a minium you need to add func test for this here https://github.com/openstack/nova/blob/master/nova/tests/functional/libvirt/test_numa_servers.py15:53
claudiubsure, will do.15:53
sean-k-mooneyand proably in https://github.com/openstack/nova/blob/master/nova/tests/functional/libvirt/test_numa_live_migration.py15:53
*** ociuhandu has quit IRC15:53
*** ociuhandu has joined #openstack-nova15:54
sean-k-mooneybasicaly you will need to boot 2 vms with just hw:numa_nodes=1 and show that the first one fills numa0 and the second get soft pinned to numa 115:54
openstackgerritMerged openstack/nova master: libvirt: Add firmware auto-configuration support  https://review.opendev.org/c/openstack/nova/+/77667815:56
claudiubAnyways, that's what we've seen in our Train environment. For example, we had a node with 400+ instances on NUMA node 0 (max should have been ~320), and if we would do a live-migration and that node was chosen as the destination, the instance would be placed in NUMA node 0. After this fix, the instances were being placed in NUMA node 1 instead.15:56
*** k-s-dean has joined #openstack-nova15:57
sean-k-mooneyright but as i said as far as i am aware you fix should not actully fix it. it may be i doubt its a full fix for all edgecases15:57
claudiubit's a simple fix for a simple usecase that can also be easily be backported. Definitely, I haven't fixed everything about NUMA in nova, but I don't think it detracts anything if we add it.15:59
*** spatel__ has joined #openstack-nova15:59
sean-k-mooneyclaudiub: we will see ill try an play with it after code feeze, are you ok waiting until after that.16:00
sean-k-mooneyif this is backporaable it can be backported after FF anyway16:01
*** gmann is now known as gmann_afk16:01
*** dklyle has joined #openstack-nova16:01
claudiubsure. As I've said, we tried this in Train, and we seem to be more stable now when it comes to live-migrations.16:02
*** spatel_ has quit IRC16:03
*** spatel__ is now known as spatel16:04
melwittsean-k-mooney: no, not planning to talk about cascading deletes. that would require removing soft-deletion which from past attempts would be very involved and I don't have the bandwidth to get into it. it would also change behaviors like being able to view the instance action list of a deleted instance, there's also a literal soft delete instance API that we'd have to decide what to do with, and so on. I think it would likely need a16:14
melwitt deprecation cycle(s) to get rid of because of the changes in behavior if we were to make soft deletion no longer possible16:14
sean-k-mooneymelwitt: ack cool16:15
lemkokashyap, nope, I'm not using nested virtualizaiton16:17
*** links has joined #openstack-nova16:17
*** zoharm has quit IRC16:24
*** links has quit IRC16:32
*** khomesh24 has joined #openstack-nova16:33
melwittsean-k-mooney, gibi: the bug downstream will happen even if we don't purge at the same time as archive. the issue is only with archive. archive moves a record to the shadow table, so it deletes it from the main table and inserts it in the shadow table. we use the word "purge" to refer to deleting things from the shadow tables i.e. 'nova-manage db purge'16:34
gibimelwitt: thanks16:35
openstackgerritBalazs Gibizer proposed openstack/nova stable/train: Warn when starting services with older than N-1 computes  https://review.opendev.org/c/openstack/nova/+/77929316:42
openstackgerritBalazs Gibizer proposed openstack/nova stable/train: Add upgrade check about old computes  https://review.opendev.org/c/openstack/nova/+/77929416:42
*** mgariepy has quit IRC16:58
sean-k-mooneybauzas: https://review.opendev.org/c/openstack/nova/+/773792 is the pci port numa policy patch. it nit a volume detach issue thats unrelated so just recheked it there a while ago16:58
*** lucasagomes has quit IRC17:00
*** sapd1 has quit IRC17:01
*** sapd1 has joined #openstack-nova17:01
openstackgerritStephen Finucane proposed openstack/nova master: libvirt: Add tests for insecure UEFI configuration  https://review.opendev.org/c/openstack/nova/+/77736717:06
openstackgerritStephen Finucane proposed openstack/nova master: libvirt: Stop passing around virt_type, caps  https://review.opendev.org/c/openstack/nova/+/77568917:06
openstackgerritStephen Finucane proposed openstack/nova master: libvirt: Add missing type hints  https://review.opendev.org/c/openstack/nova/+/77568817:06
openstackgerritStephen Finucane proposed openstack/nova master: trivial: Clarify purpose of 'Host.supports_*' properties  https://review.opendev.org/c/openstack/nova/+/77873917:06
openstackgerritStephen Finucane proposed openstack/nova master: hardware: Start parsing 'os_secure_boot'  https://review.opendev.org/c/openstack/nova/+/68262817:06
openstackgerritStephen Finucane proposed openstack/nova master: compute: Report COMPUTE_SECURITY_UEFI_SECURE_BOOT  https://review.opendev.org/c/openstack/nova/+/77667917:06
openstackgerritStephen Finucane proposed openstack/nova master: scheduler: Translate secure boot requests to trait  https://review.opendev.org/c/openstack/nova/+/77668017:06
openstackgerritStephen Finucane proposed openstack/nova master: libvirt: Wire up 'os_secure_boot' property  https://review.opendev.org/c/openstack/nova/+/77668117:06
openstackgerritStephen Finucane proposed openstack/nova master: libvirt: Report secure boot support to scheduler  https://review.opendev.org/c/openstack/nova/+/77569017:06
openstackgerritStephen Finucane proposed openstack/nova master: tests: Add functional tests for UEFI, secure boot  https://review.opendev.org/c/openstack/nova/+/77668217:06
openstackgerritStephen Finucane proposed openstack/nova master: tests: Remove duplicated 'start_compute' helper  https://review.opendev.org/c/openstack/nova/+/77668317:06
openstackgerritStephen Finucane proposed openstack/nova master: docs: Document UEFI secure boot feature  https://review.opendev.org/c/openstack/nova/+/77668417:06
openstackgerritStephen Finucane proposed openstack/nova master: libvirt: Add parsing of firmware metadata files  https://review.opendev.org/c/openstack/nova/+/77930217:06
openstackgerritStephen Finucane proposed openstack/nova master: libvirt: Move some host arch checks to guest checks  https://review.opendev.org/c/openstack/nova/+/77930317:06
openstackgerritStephen Finucane proposed openstack/nova master: libvirt: Use firmware metadata files to configure instance  https://review.opendev.org/c/openstack/nova/+/77930417:06
stephenfinPhew17:07
*** tesseract has quit IRC17:10
*** tesseract has joined #openstack-nova17:12
lyarwoodstephenfin: I'll hit that again in the morning17:13
lyarwoodor maybe later tonight if I get time17:13
*** iurygregory has joined #openstack-nova17:17
*** khomesh24 has quit IRC17:20
*** tbachman_ has joined #openstack-nova17:22
*** tbachman has quit IRC17:25
*** tbachman_ is now known as tbachman17:25
*** jangutter has quit IRC17:26
*** jangutter has joined #openstack-nova17:26
*** hamalq has joined #openstack-nova17:29
*** mgariepy has joined #openstack-nova17:38
*** nightmare_unreal has quit IRC17:42
*** iurygregory has quit IRC17:49
*** andrewbonney has quit IRC18:05
*** rpittau is now known as rpittau|afk18:16
*** amodi has quit IRC18:16
*** dtantsur is now known as dtantsur|afk18:17
*** ociuhandu has quit IRC18:20
*** ociuhandu has joined #openstack-nova18:22
*** mlavalle has joined #openstack-nova18:23
*** ociuhandu has quit IRC18:27
*** amodi has joined #openstack-nova18:27
openstackgerritStephen Finucane proposed openstack/nova master: libvirt: Enable 'vmcoreinfo' feature by default  https://review.opendev.org/c/openstack/nova/+/75291218:30
openstackgerritStephen Finucane proposed openstack/nova master: WIP: libvirt: Make vmcoreinfo dependent on feature flag  https://review.opendev.org/c/openstack/nova/+/77934218:30
*** amodi has quit IRC18:34
stephenfinartom: ^18:35
stephenfintl;dr: I will take the feature flag approach but not now. Too many test failures, too little time18:36
artomstephenfin, \o/18:36
artomFair18:36
stephenfin(as noted in the patch itself, the secure boot series should clear up most of those failures since I've significantly improved the FakeLibvirtFixture)18:36
artomIt's always the tests...18:38
* artom wonders when fakelibvirt and real libvirt will just... converge18:38
artomAnd we'll have re-implemented libvirt in Python, only for qemu and kvm18:38
*** iurygregory has joined #openstack-nova18:46
*** tesseract has quit IRC18:51
*** ociuhandu has joined #openstack-nova18:51
*** zul has joined #openstack-nova18:54
*** ociuhandu has quit IRC18:56
*** k_mouza has quit IRC19:08
*** stand has quit IRC19:15
*** k-s-dean has quit IRC19:21
*** stand has joined #openstack-nova19:22
*** k-s-dean has joined #openstack-nova19:22
*** k-s-dean has quit IRC19:29
*** ociuhandu has joined #openstack-nova19:30
*** gmann_afk is now known as gmann19:32
artomgibi, still active?19:36
*** songwenping__ has joined #openstack-nova19:39
artomI guess not, but I think I'm confident enough in my thinking that I'll go ahead19:40
artomReplied in review, in any case19:40
*** amodi has joined #openstack-nova19:41
*** songwenping_ has quit IRC19:43
*** ociuhandu has quit IRC19:43
*** whoami-rajat_ is now known as whoami-rajat19:44
sean-k-mooneystephenfin: replied on the port policy spec. bauzas  suggested i check for the extenion to be extra safe so dont really want to remove it unless he is ok with it19:49
openstackgerritArtom Lifshitz proposed openstack/nova master: pci manager: replace node_id parameter with compute_node  https://review.opendev.org/c/openstack/nova/+/77874720:19
openstackgerritArtom Lifshitz proposed openstack/nova master: pci: track host NUMA topology in stats  https://review.opendev.org/c/openstack/nova/+/77414920:19
openstackgerritArtom Lifshitz proposed openstack/nova master: pci: implement the 'socket' NUMA affinity policy  https://review.opendev.org/c/openstack/nova/+/77277920:19
*** vishalmanchanda has quit IRC20:47
*** efried has quit IRC20:51
*** ociuhandu has joined #openstack-nova20:53
*** efried has joined #openstack-nova20:56
*** gyee has joined #openstack-nova20:56
melwittlyarwood: fyi https://review.opendev.org/c/openstack/nova/+/777218 and https://review.opendev.org/c/openstack/nova/+/777219 are ready for review on stable/ussuri. there are two more patches under them (sorry!) that I included bc (1) it's a bug fix I had forgotten to backport and (2) they eliminate merge conflicts20:57
*** ociuhandu has quit IRC21:01
*** k_mouza has joined #openstack-nova21:08
*** k_mouza has quit IRC21:13
openstackgerritsean mooney proposed openstack/nova master: add constants for vnic type vdpa  https://review.opendev.org/c/openstack/nova/+/77047421:30
openstackgerritsean mooney proposed openstack/nova master: add vdpa nodedev parsing and interface config gen  https://review.opendev.org/c/openstack/nova/+/77053221:30
openstackgerritsean mooney proposed openstack/nova master: extend pci dev_type for vdpa  https://review.opendev.org/c/openstack/nova/+/77748121:30
openstackgerritsean mooney proposed openstack/nova master: add vdpa device detection  https://review.opendev.org/c/openstack/nova/+/77053321:30
openstackgerritsean mooney proposed openstack/nova master: add hw:mlock extra spec  https://review.opendev.org/c/openstack/nova/+/77834721:30
openstackgerritsean mooney proposed openstack/nova master: add vdpa vnic to pci request mapping and filtering.  https://review.opendev.org/c/openstack/nova/+/77835021:30
openstackgerritsean mooney proposed openstack/nova master: add hw:mlock extra spec  https://review.opendev.org/c/openstack/nova/+/77834721:32
openstackgerritsean mooney proposed openstack/nova master: add vdpa nodedev parsing and interface config gen  https://review.opendev.org/c/openstack/nova/+/77053221:32
openstackgerritsean mooney proposed openstack/nova master: extend pci dev_type for vdpa  https://review.opendev.org/c/openstack/nova/+/77748121:32
openstackgerritsean mooney proposed openstack/nova master: add vdpa device detection  https://review.opendev.org/c/openstack/nova/+/77053321:32
openstackgerritsean mooney proposed openstack/nova master: add vdpa vnic to pci request mapping and filtering.  https://review.opendev.org/c/openstack/nova/+/77835021:32
openstackgerritGhanshyam proposed openstack/nova master: Change default policy for GET '/os-hypervisors' API  https://review.opendev.org/c/openstack/nova/+/76579821:49
*** tjgresha has joined #openstack-nova21:51
openstackgerritGhanshyam proposed openstack/nova master: Change default policy for GET '/os-hypervisors' API  https://review.opendev.org/c/openstack/nova/+/76579821:51
*** rcernin has joined #openstack-nova21:54
*** ociuhandu has joined #openstack-nova22:01
*** ociuhandu has quit IRC22:06
*** whoami-rajat has quit IRC22:06
*** slaweq has quit IRC22:15
openstackgerritMerged openstack/nova master: libvirt: Add tests for insecure UEFI configuration  https://review.opendev.org/c/openstack/nova/+/77736722:19
*** xek has quit IRC22:28
*** spatel has quit IRC22:29
*** tjgresha has quit IRC22:34
*** ociuhandu has joined #openstack-nova22:38
*** ociuhandu has quit IRC22:44
openstackgerritsean mooney proposed openstack/nova master: Support per port numa policies with SR-IOV  https://review.opendev.org/c/openstack/nova/+/77379223:05
openstackgerritsean mooney proposed openstack/nova master: Support per port numa policies with SR-IOV  https://review.opendev.org/c/openstack/nova/+/77379223:05
*** zzzeek has quit IRC23:08
*** songwenping_ has joined #openstack-nova23:08
*** songwenping__ has quit IRC23:12
*** zzzeek has joined #openstack-nova23:12
*** martinkennelly has quit IRC23:21
*** martinkennelly has joined #openstack-nova23:23
*** claudiub has quit IRC23:51
*** ociuhandu has joined #openstack-nova23:54
*** ociuhandu has quit IRC23:58
« Sunday, 2021-03-07 Index Tuesday, 2021-03-09 »

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!