Wednesday, 2021-05-19

« Tuesday, 2021-05-18 Index Thursday, 2021-05-20 »
*** mlavalle has quit IRC00:05
*** macz_ has quit IRC00:09
*** k_mouza has joined #openstack-nova00:14
*** k_mouza has quit IRC00:19
*** dustinc has quit IRC00:30
*** swp20 has joined #openstack-nova00:38
*** sapd1_x has quit IRC00:47
openstackgerritnorman shen proposed openstack/nova master: Saving security group to info_cache  https://review.opendev.org/c/openstack/nova/+/78634800:48
*** swp20 has quit IRC00:56
*** yangkai has joined #openstack-nova01:07
*** __ministry has joined #openstack-nova01:21
*** swp20 has joined #openstack-nova01:45
*** macz_ has joined #openstack-nova02:10
*** macz_ has quit IRC02:14
openstackgerritnorman shen proposed openstack/nova-specs master: Speed up server details  https://review.opendev.org/c/openstack/nova-specs/+/79162002:16
*** k_mouza has joined #openstack-nova02:34
*** k_mouza has quit IRC02:39
*** rcernin has quit IRC02:42
*** rcernin has joined #openstack-nova02:50
openstackgerritnorman shen proposed openstack/nova-specs master: Speed up server details  https://review.opendev.org/c/openstack/nova-specs/+/79162003:12
*** mkrai has joined #openstack-nova03:18
openstackgerritnorman shen proposed openstack/nova-specs master: Speed up server details  https://review.opendev.org/c/openstack/nova-specs/+/79162003:38
*** psachin has joined #openstack-nova03:39
*** ociuhandu has joined #openstack-nova03:40
*** ociuhandu has quit IRC03:44
*** macz_ has joined #openstack-nova03:46
*** macz_ has quit IRC03:51
*** sapd1_x has joined #openstack-nova04:29
*** ratailor has joined #openstack-nova05:00
*** ralonsoh has joined #openstack-nova05:36
*** sapd1_x has quit IRC05:38
*** macz_ has joined #openstack-nova05:47
*** macz_ has quit IRC05:52
*** supamatt has joined #openstack-nova05:58
*** swp20 has quit IRC06:19
*** slaweq has joined #openstack-nova06:33
*** k_mouza has joined #openstack-nova06:35
*** vishalmanchanda has joined #openstack-nova06:35
*** k_mouza has quit IRC06:39
*** nightmare_unreal has joined #openstack-nova06:45
*** dklyle has quit IRC06:47
*** mkrai has quit IRC07:06
*** jawad_axd has quit IRC07:07
*** jawad_axd has joined #openstack-nova07:07
*** sapd1_x has joined #openstack-nova07:09
*** jawad_axd has quit IRC07:11
*** ociuhandu has joined #openstack-nova07:15
*** lucasagomes has joined #openstack-nova07:16
*** ociuhandu has quit IRC07:16
*** andrewbonney has joined #openstack-nova07:17
*** rcernin has quit IRC07:20
*** ralonsoh has quit IRC07:24
*** ralonsoh has joined #openstack-nova07:24
*** mkrai has joined #openstack-nova07:27
*** rpittau|afk is now known as rpittau07:33
*** jawad_axd has joined #openstack-nova07:37
*** tosky has joined #openstack-nova07:38
*** rcernin has joined #openstack-nova07:47
*** macz_ has joined #openstack-nova07:48
*** ociuhandu has joined #openstack-nova07:52
*** macz_ has quit IRC07:52
*** rcernin has quit IRC07:54
*** ChanServ has quit IRC07:54
*** ChanServ has joined #openstack-nova07:54
*** services. sets mode: +o ChanServ07:54
*** derekh has joined #openstack-nova07:55
*** rcernin has joined #openstack-nova07:56
*** swp20 has joined #openstack-nova08:01
*** ociuhandu has quit IRC08:02
*** ociuhandu has joined #openstack-nova08:05
*** ociuhandu has quit IRC08:07
*** ociuhandu has joined #openstack-nova08:07
*** rcernin has quit IRC08:09
*** mkrai has quit IRC08:10
*** rcernin has joined #openstack-nova08:10
*** rcernin has quit IRC08:19
*** ociuhandu has quit IRC08:20
*** rcernin has joined #openstack-nova08:20
*** ociuhandu has joined #openstack-nova08:20
openstackgerritDaniel Bengtsson proposed openstack/nova master: Use the new type HostDomainOpt.  https://review.opendev.org/c/openstack/nova/+/78824008:23
*** rcernin has quit IRC08:25
*** k_mouza has joined #openstack-nova08:30
*** ociuhandu has quit IRC08:30
*** ociuhandu has joined #openstack-nova08:31
*** ociuhandu has quit IRC08:31
*** ociuhandu has joined #openstack-nova08:31
*** k_mouza has quit IRC08:34
*** rcernin has joined #openstack-nova08:38
*** rcernin has quit IRC08:44
*** ignaziocassano has joined #openstack-nova08:49
ignaziocassanoIgnazio Cassano <ignaziocassano@gmail.com>08:50
ignaziocassano10:48 (1 minuto fa)08:50
ignaziocassanoa openstack-discuss08:50
ignaziocassanoHello Guys,08:50
ignaziocassanoon train centos7 I am facing live migration issue only for some instances (not all).08:50
ignaziocassanoThe error reported is:08:50
ignaziocassano2021-05-19 08:45:57.096 142537 ERROR nova.compute.manager [-] [instance: b18450e8-b3db-4886-a737-c161d99c6a46] Live migration failed.: libvirtError: Unable to read from monitor: Connection reset by peer08:50
ignaziocassanosome instances migrate without errors. I tried to stop end restart libvirtd end nova-compute without solving08:51
ignaziocassanoOn instances where migration failed, If I stop them and I start on another node, If  migrate them on original node and migrate again, it works08:52
ignaziocassanoSorry, the version is stein08:55
*** lyarwood has joined #openstack-nova08:56
*** rcernin has joined #openstack-nova08:56
stephenfinignaziocassano: Have you looked into the libvirt logs directly or investigated syslog?08:58
stephenfinkashyap can correct me if I'm wrong, but that error usually implies the connection between QEMU and libvirt has died08:58
* kashyap blinks and looks08:59
kashyapstephenfin: Yes, you're right08:59
kashyapBut the underlying problem could be anywhere ... and needs more details to debug09:00
kashyapignaziocassano: Are you migrating including your storage?  (I.e. "block migration"?)09:00
ignaziocassanokashyap the storage is shared on netapp nfs09:01
ignaziocassanono errors on openvswitch agent09:01
*** rcernin has quit IRC09:01
*** rcernin has joined #openstack-nova09:02
ignaziocassanokashyap: I presume the connection between QEMU and libvirt ha died for some instances, but I do not know how I can verify it09:03
kashyapignaziocassano: Right; that's the reason.  So one way to debug this is to obtain libvirt debug log filters that track interactions b/n QEMU and libvirt09:04
*** k_mouza has joined #openstack-nova09:04
kashyapignaziocassano: You can get it this way:09:04
kashyapOn the relevant compute nodes (both source and dest):09:04
kashyap(1) Set the log output file:  $> virt-admin daemon-log-outputs "1:file:/var/log/libvirt/libvirtd.log"09:04
kashyap(2) Configure the dynamic filters:09:04
kashyap$> virt-admin daemon-log-filters \ "1:libvirt 1:qemu 1:conf 1:security 3:event 3:json 3:file 3:object 1:util"09:04
kashyap(Remove that "\")09:05
ignaziocassanokashyap: on messages it reports:  19:22:29 podvc-kvm04 libvirtd: 2021-05-16 17:22:29.562+0000: 166295: error : qemuMonitorIO:718 : internal error: End of file from qemu monitor09:05
kashyapAnd then re-run the migration of the affected instances09:05
kashyapignaziocassano: Yeah; that's also "normal" / "expected"; doesn't tell much still09:05
kashyapignaziocassano: BTW, can you quickly try this:09:06
kashyap    $> journalctl -u libvirtd -l --since=yesterday -p err09:06
ignaziocassano error : qemuDomainObjBeginJobInternal:6825 : Timed out during operation: cannot acquire state change lock (held by monitor=remo09:08
kashyapUgh, this one...09:08
ignaziocassanounfortunately I cannot try to move further vm because they are in production and I must inform my customers09:09
*** ociuhandu has quit IRC09:10
ignaziocassanoit us very strange because for some instances it works fine09:10
kashyapignaziocassano: So this is one of the most painful errors to debug from libvirt.  I recently wrote a response on a Red Hat bugzilla09:10
*** ociuhandu has joined #openstack-nova09:10
kashyapLet me get that for you (beware: no easy solution here :-()09:10
ignaziocassanokashyap, I read some red hat suggestions, but they include the instance restart :-(09:11
kashyapignaziocassano: Yes, you're not wrong, afraid.09:12
kashyapignaziocassano: These "cannot acquire state change lock" errors are notorious; and it could be due to QEMU getting hung, which in turn could be caused by stuck I/O09:12
*** ociuhandu has quit IRC09:13
*** ociuhandu has joined #openstack-nova09:13
ignaziocassanoI did not see any nfs stale on my compute node09:14
*** vishalmanchanda has quit IRC09:14
kashyapignaziocassano: Can you post the affected guest log from /var/log/libvir/qemu/instance-YYYY.log?09:15
kashyapIt _might_ sometimes havea  clue09:16
openstackgerritliuzhuangzhuang proposed openstack/nova master: Fix RBD timeout  https://review.opendev.org/c/openstack/nova/+/78658809:16
kashyapAnd also this (if the output is long, use a paste-bin): `journalctl -u libvirtd -r`09:16
*** mnasiadka has quit IRC09:17
*** coreycb has quit IRC09:19
ignaziocassanokashyap, unfortunately since the affected vm remained in pause on source and destination, I had to destroy them so the log files contains only:09:20
ignaziocassano021-05-19 08:11:19.800+0000: initiating migration09:20
ignaziocassano2021-05-19 08:12:30.446+0000: shutting down, reason=destroyed09:20
kashyapHm; that won't help us much, afraid09:21
ignaziocassanoI could try on destination host if there is something else09:21
admin0hi guys .. i have a strange issue:  openstack server show $uuid shows its host on hypervisor =>  h7 . on the hypervisor, hostname, hostname -f and virsh hostname returns h7 , but during migration of this instance ( ceph backed ) to say h9 or h10, it says h7 host not found ..  .. how can i address/solve this ?09:22
ignaziocassanoyes09:22
ignaziocassanoI post it on openstack pastebin09:22
ignaziocassano:q!09:22
admin0https://gist.github.com/a1git/c22ec0c17aaa9dcf95fd7485eb76af2f looks like my hostnames cycled from h7, h7. h7.openstack.local .. so now i am unable to migrate anything off of the hosts09:22
*** damien_r has joined #openstack-nova09:23
ignaziocassanokashyap: http://paste.openstack.org/show/805476/09:24
kashyapignaziocassano: Ha, see this:09:25
kashyap---09:25
kashyap2021-05-19T08:12:30.397606Z qemu-kvm: error while loading state for instance 0x0 of device '0000:00:08.0/virtio-blk'09:25
kashyap2021-05-19T08:12:30.399542Z qemu-kvm: load of migration failed: Input/output error09:25
kashyap2021-05-19 08:12:31.022+0000: shutting down, reason=crashed09:25
kashyap---09:26
kashyapI've pretty sure seen that 'virtio-blk' error before09:26
*** rcernin has quit IRC09:26
ignaziocassanokashyap: so do you think instanance not worked fine also before live migration ?09:27
*** mkrai has joined #openstack-nova09:27
kashyapignaziocassano: See this bug (comments are complex; browse it from the bottom): https://bugs.launchpad.net/nova/+bug/176179809:27
openstackLaunchpad bug 1761798 in OpenStack Compute (nova) "live migration intermittently fails in CI with "VQ 0 size 0x80 Guest index 0x12c inconsistent with Host index 0x134: delta 0xfff8"" [Medium,Confirmed]09:27
kashyapignaziocassano: No, not really; see this bit from the logs:09:28
kashyap[quote]09:28
kashyapwe get this "guest index inconsistent" error when the migrated RAM is inconsistent with the migrated 'virtio' device state. And a common case is where a 'virtio' device does an operation after the vCPU is stopped and after RAM has been transmitted.09:28
kashyap[/quote]09:28
kashyap(From my comment#11)09:29
ignaziocassanokashyap: my englush is poor but I did not read a conclusion.09:30
ignaziocassanokayshap: do you think post_copy and autoconverge can help ?09:31
kashyapignaziocassano: Your English is not poor; there's no conclusion yet, as the problem is complex.09:31
kashyapignaziocassano: They can help if you're guest is doing I/O faster than your migration can keep up09:31
*** mkrai has quit IRC09:33
*** whoami-rajat has joined #openstack-nova09:33
ignaziocassanoSo, the solution at this time is waiting for someone solves the bug, right ?09:33
kashyapignaziocassano: Not really; as upstream QEMU claims these problems should not occur with newer QEMU releases09:34
kashyapBut if you can generate a reproducer that'll help.  But usually reproducers in this case are difficult to come by.09:34
ignaziocassanokashyap: let me know if I am wrong: new qemu version come with centos 8, right ?09:35
*** coreycb has joined #openstack-nova09:35
kashyapignaziocassano: Yes.  Or even a potentially updated QEMU on CentOS7 (if you're running older ones; I haven't checked)09:35
ignaziocassanokashyap: I am running the last available for centos 7 repos. I cannot migrate to centos 8 before upgrading to train, but with train there is another bug for live migration because loss packages during it09:37
*** vishalmanchanda has joined #openstack-nova09:38
ignaziocassanoI stein I solved with Seean Mooney workaround to force port legacy binding. On train I have not found any workaround yet09:38
kashyapI see; I don't know about that problem, afraid09:39
*** mnasiadka has joined #openstack-nova09:39
gibisean-k-mooney: hi! If you have time then please check https://bugs.launchpad.net/nova/+bug/1928922 Is the observation correct? If yes then why neutron sends vif-plugged before nova even requested the plug?09:41
openstackLaunchpad bug 1928922 in OpenStack Compute (nova) "evacuation tests in nova-live-migration post hook fails with VirtualInterfaceCreateException due to vif-plugged event received before nova starts waiting for it." [Medium,New] - Assigned to Balazs Gibizer (balazs-gibizer)09:41
ignaziocassanokashyap: thanks, I will keep informed with the bug you mentioned09:42
*** ociuhandu has quit IRC09:45
*** ociuhandu has joined #openstack-nova09:47
*** macz_ has joined #openstack-nova09:49
*** ignaziocassano has quit IRC09:52
openstackgerritStephen Finucane proposed openstack/nova stable/train: Reproduce bug 1897528  https://review.opendev.org/c/openstack/nova/+/79211609:52
openstackbug 1897528 in OpenStack Compute (nova) "32bit pci domain number is not supported" [High,In progress] https://launchpad.net/bugs/1897528 - Assigned to Balazs Gibizer (balazs-gibizer)09:52
openstackgerritStephen Finucane proposed openstack/nova stable/train: Ignore PCI devices with 32bit domain  https://review.opendev.org/c/openstack/nova/+/79211709:52
*** ociuhandu has quit IRC09:53
*** macz_ has quit IRC09:54
*** k_mouza has quit IRC09:59
*** k_mouza has joined #openstack-nova10:05
*** mkrai has joined #openstack-nova10:12
*** dtantsur|afk is now known as dtantsur10:16
sean-k-mooneygibi: that should be fixed now10:19
sean-k-mooneygibi: i have not read it fully but there was a race with the neutron dhcp server10:20
sean-k-mooneygibi: what release was this10:20
gibisean-k-mooney: this is from master10:27
gibifairly recent master10:27
sean-k-mooneyi belive to stop the race you need to set a neutron config option10:28
sean-k-mooneygibi: i think https://review.opendev.org/c/openstack/nova/+/770745 should help with the issue10:30
sean-k-mooneybut you would still need to enable https://review.opendev.org/c/openstack/neutron/+/790702/1/neutron/conf/common.py10:30
sean-k-mooney[nova]/live_migration_events=true10:31
sean-k-mooneygibi: i have not fully looked at the bug yet so ill do that shortly and try an confirm if its the same thing10:31
sean-k-mooneygibi: but im pretty sure its https://bugzilla.redhat.com/show_bug.cgi?id=193043210:31
openstackbugzilla.redhat.com bug 1930432 in openstack-nova "Nova evacuate fails due to timeout waiting for a network-vif-plugged event for instance" [Medium,Verified] - Assigned to smooney10:31
gibidoes live_migration_event affects evacutaion?10:31
*** tesseract has joined #openstack-nova10:32
sean-k-mooneykind of the patch has 2 fixes. one it only send events form the l2 agent to nova and second if fixes the filtering in nueton to allow procing the port if it has any port binding for the current host. previously it only did it for active port bindign which was wrong10:33
sean-k-mooneygibi: are you able to repoduce this reliably10:34
*** ociuhandu has joined #openstack-nova10:34
gibisean-k-mooney: nope, it is random and seemingly infrequent10:34
sean-k-mooneyok i was going to suggest chanig the default of that config option since its ment to be removed in Y it shoudl default to true in Xena anyway10:35
sean-k-mooneyand then using a depends on patch to test10:35
*** tbachman has quit IRC10:35
sean-k-mooneybut if its infrequent we might not see a difference10:35
*** tbachman has joined #openstack-nova10:35
gibilooking at the live migration fix, I think we could have a similar race during evacuation causing the vent to arrive too early10:36
sean-k-mooneygibi: one casue fo this in the past was that during evac in the ci we were previously just stoping the nova compute agent not the neutron l2 agent so wehn we did the port update it would respond10:37
sean-k-mooneye.g. the souce agent woudl say yep its already wired10:37
gibisean-k-mooney: I confirmed that the q-agt is dead on the source host during this run10:38
sean-k-mooneyi think we fixed that in our job10:38
gibiyes, it is fixed in the test10:38
sean-k-mooneyok good10:38
sean-k-mooneyam did we also ensure the job is not swapted to ovn10:38
sean-k-mooneyi belvie we did10:38
gibichecking...10:38
sean-k-mooneyit looks like ml2/ovs10:41
gibiwe only held back nova-next on ovs https://review.opendev.org/c/openstack/nova/+/776944 but nova-live-migration moved to OVN as far as I see10:41
sean-k-mooneyreally the build you linked to has teh screen-q-agt.txt10:41
sean-k-mooneyi guess it has not run since the default change maybe?10:43
gibihm it run last week10:43
sean-k-mooneythis was from friday yes10:44
gibilet me check a more recent run...10:44
gibihm a todays run also has q-agt.txt10:44
gibibut I don't see in the zuul config where we set the Q_AGENT option to be openvswitch for this job10:45
sean-k-mooneyyep also looking at it10:45
sean-k-mooneyis this still zullv2?10:45
sean-k-mooneyno? https://github.com/openstack/nova/blob/master/.zuul.yaml#L53-L8710:46
sean-k-mooneythat looks like it v310:46
sean-k-mooneyok well that is a different mistery to look at after10:46
sean-k-mooneymaybe it has something to do with         neutron-trunk: true but that seams odd if it does10:48
gibiyeah, that is the only odd thing in the job config10:48
sean-k-mooneydo you think the previos live_migation_evetns patch would fix evacuate?10:48
sean-k-mooneywe were planning to backport that to train as the race exits basically since netuuron has been a thing10:49
*** yangkai has quit IRC10:49
gibiI don't think it fixes the evacuate as we see the error on master _after_ the live migration event patch merged10:50
gibibut the logic behind the failure can be the same10:50
sean-k-mooneygibi: well its disabled by default10:50
sean-k-mooneybecause it need the nova patch to be present to enable it10:51
gibialso it only changes _get_neutron_events_for_live_migration() which is fairly live migration specific10:51
gibiohh you talk about the neutron fixes10:51
sean-k-mooneyyes10:52
gibiwe can try that10:52
gibias you said we want that to default to true anyhow10:52
sean-k-mooneythe nova patch was just so that the neutron fix would not break live migraiton10:52
sean-k-mooneyand because we needed the nova fix the neutron one defaulted to false for last cycle10:52
sean-k-mooneybut now it can go to ture since the nova patch is present and backported10:53
gibilets switch it to true on master10:54
sean-k-mooneywould you like me to submit a patch to change the default and note the evacuation bug you filed as a related bug? or will i leave that to you10:54
gibiyou have a bit more context on that neutron flag so if you have time please propose the switch10:54
sean-k-mooneyok ill go do that shortly i have a docs meeting at the top of the hour so ill likely do it after that10:55
*** ociuhandu has quit IRC10:58
*** ociuhandu has joined #openstack-nova10:58
gibisean-k-mooney: thanks10:59
gibisean-k-mooney: it seems the devstack default OVN change has been reverted https://review.opendev.org/c/openstack/devstack/+/79110411:00
sean-k-mooneyoh ok that would explain why we are seeing ml2/ovs11:01
sean-k-mooneythings exploded?11:01
gibibased on the revert commit message yes there are extra things to fix11:01
sean-k-mooneyim really happy we did not make the default change during feature freeze then :)11:01
gibiyes :)11:02
gibigood judgement11:02
*** mkrai has quit IRC11:04
*** ociuhandu has quit IRC11:09
*** links has joined #openstack-nova11:21
*** sapd1_x has quit IRC11:21
*** f0o has joined #openstack-nova11:22
* bauzas goes afk for the afternoon if you look at me (getting a new electric car ;) 11:23
gibibauzas: congrats!11:24
gibiis it a tesla?11:24
bauzasgibi: nope, for my wife, a Peugeot 20811:24
bauzasI already have a PHEV (Skoda Superb iV) for me :)11:25
gibisounds cool :)11:26
*** lpetrut has joined #openstack-nova11:29
lyarwoodelod: https://review.opendev.org/c/openstack/nova/+/788720 - thoughts on landing this series?11:39
*** ociuhandu has joined #openstack-nova11:39
elodlyarwood: sorry, yes, I started to look it yesterday, but want to see the "whole picture" first o:)11:42
elodlyarwood: it is (again) a bit too big for my taste for stable, but it's more or less clean if I'm not mistaken....11:43
lyarwoodelod: yeah agreed it's pretty large but while it's clean I thought it would be useful11:46
lyarwoodelod: it's something we wanted downstream for Wallaby either way11:46
lyarwoodelod: so if we can squeeze it in early this cycle it would be great :)11:47
*** ociuhandu has quit IRC11:48
elodlyarwood: understood :) I'll try to go through the patches today11:48
*** macz_ has joined #openstack-nova11:50
lyarwoodmany thanks11:50
gibielod: as far as I remember it is a clean cherr-pick all the way. I did a mistake when I first tried to cherry-pick it as I used some old unmereged version of a patch11:52
*** macz_ has quit IRC11:54
sean-k-mooneybauzas: i have been debating about getting an eletric car for a while but 1 i like my mini and 2 i driver maybe 5000KMs per year so its har to justtify spending more then a few grand on a car12:04
elodgibi: that part is OK then :) just have to think whether the series is valid for backport or not o:)  (by looking at the code and patches yesterday it was a bit "featurish" for me, but probably there's no risk to backport... but haven't looked all of the patches yet)12:08
*** ociuhandu has joined #openstack-nova12:16
*** ociuhandu has quit IRC12:21
*** ratailor has quit IRC12:22
*** ociuhandu has joined #openstack-nova12:26
gibielod: it does not change any external interfaces except the two new config options, but those can be removed, if you wish, from the backport. There is also no externally visible behavior change except the fix of the failure12:32
gibiI guess you feel it as a feature becuase we started using a different mechanism to talk to libvirt regarding the attachment, we went from polling to waiting for events. And waiting for events needed extra preparation in the code12:33
sean-k-mooneyi dont think its a featue its just a refacotiong12:35
gibisean-k-mooney: yes, it needed a sizeable refactoring to properly fix that bug12:35
sean-k-mooneythe behaivior before and after modulo bugs is identical from blackbox perspctive12:35
gibiyepp12:35
*** Luzi has joined #openstack-nova12:40
elodyes... the refactor... o:) thanks for the details, it is useful for me to hear other opinions as well!12:43
lyarwoodhttps://review.opendev.org/c/openstack/nova/+/790660 - core reviews on this bugfix would be appreciated if anyone has time this week12:44
lyarwoodand the series below it sorry12:44
gibilyarwood: added to my queue12:44
lyarwoodthanks12:44
*** _erlon_ has joined #openstack-nova12:56
*** ricolin has joined #openstack-nova13:08
*** psachin has quit IRC13:10
openstackgerritBalazs Gibizer proposed openstack/nova master: Add same_subtree field to RequestLevelParams  https://review.opendev.org/c/openstack/nova/+/79150313:18
openstackgerritBalazs Gibizer proposed openstack/nova master: Bump min placement microversion to 1.36  https://review.opendev.org/c/openstack/nova/+/79150413:18
openstackgerritBalazs Gibizer proposed openstack/nova master: Support same_subtree in allocation_canadidate query  https://review.opendev.org/c/openstack/nova/+/79150513:18
openstackgerritBalazs Gibizer proposed openstack/nova master: Support the new port resource_request format  https://review.opendev.org/c/openstack/nova/+/78720813:18
openstackgerritBalazs Gibizer proposed openstack/nova master: Transfer RequestLevelParams from ports to scheduling  https://review.opendev.org/c/openstack/nova/+/79150613:18
*** dasp has quit IRC13:24
*** dasp has joined #openstack-nova13:25
sean-k-mooney...  https://twitter.com/freenodestaff https://fuchsnet.ch/freenode-resign-letter.txt13:28
*** ricolin has quit IRC13:29
*** macz_ has joined #openstack-nova13:30
*** ociuhandu has quit IRC13:31
*** ociuhandu has joined #openstack-nova13:32
sean-k-mooneyos maybe moving to irc.libera.chat i guess although we will have to see how the topic plays out in the mailing list13:33
*** macz_ has quit IRC13:35
kashyapYeah; this whole episode seems bizarre13:36
*** ociuhandu has quit IRC13:36
*** Luzi has quit IRC13:39
sean-k-mooneyits not the first time that this type of thing has happened. its similar to the whole mysql vs mariadb split after the sale to oracale. when legal entities assocaited with opensrouce comunities are sold and the aquiring company desides to put there stamp on comunity it can often result in a split13:41
sean-k-mooneykashyap: in this case it sound liek the new ownwer of freenode LTD wanted to moitise it in some way and as a result the people that actully ran it but were not employees rightly dont want to have there work or the comunity monitised13:42
sean-k-mooneyreading between the lines there is also privacy concerns and perhaps lawful intercept or us law eleemnt to this too13:44
kashyapI see.  I haven't read their story in full, but it's annoying13:49
*** tosky has quit IRC13:56
*** tosky has joined #openstack-nova14:00
*** sapd1_x has joined #openstack-nova14:00
*** raildo has joined #openstack-nova14:10
*** ociuhandu has joined #openstack-nova14:19
*** alexe9191 has joined #openstack-nova14:23
*** ociuhandu has quit IRC14:24
alexe9191Good day everyone:) . A while ago i had an issue with the nova scheduler in rocky and I was advised to use the placement api instead of filters. However, we are depending on the instance extra specs aggregate filter to match certain hosts properties. Like for instance, SSD.14:24
alexe9191I am wondering if such an option is available for placement? and if so, how to configure nova to use it? it's not clear in the documentation.14:24
sean-k-mooneyalexe9191: there is a replacemnt yes14:31
sean-k-mooneyhttps://docs.openstack.org/nova/latest/reference/isolate-aggregates.html14:31
sean-k-mooneyalexe9191: its not a direct replacment but it achive the same goal using trait in a slight better way14:31
sean-k-mooneyalexe9191: i say its not a direct replacment because you need update your flaovrs14:32
alexe9191I am assuming I also need to update the hosts one by one to add that trait to each resource class?14:33
alexe9191and also update the flavor to use trait instead of for instance hw:cpu_policy='dedicated' ?14:34
alexe9191or quota:disk_write? something like that14:34
sean-k-mooneyno you would have to create a new flavor with the trait  e.g. traits:CUSTOM_SSD=required as an extra spec14:36
sean-k-mooneythe resize the instance to use that trait14:36
*** belmoreira has joined #openstack-nova14:37
alexe9191but that trait need to be saved to the hypervisor provider, correct? or would it be inherited from the aggregate the host is sitting in?14:38
alexe9191Creating new flavors it not the issue. I am wondering if I have to update 800x hypervisors with the new CUSTOM_SSD trait and all other custom traits that i need?14:39
*** ociuhandu has joined #openstack-nova14:39
alexe9191ah I see. I think you can add the trait to the aggregate as well?14:40
sean-k-mooneyyou add it to the aggreate yes14:40
sean-k-mooneyyou do also have to add it too the compute node RP14:40
sean-k-mooneyalexe9191: traits live on the resouce provider not the invetories14:41
sean-k-mooneyso you woudl put CUSTOM_SSD on all the compute nodes with an SSD14:41
sean-k-mooneyalexe9191: you can continue to use the old filter by the way14:42
alexe9191ah the --property in the flavor you mean? that would be translated to a trait?14:42
sean-k-mooneyalexe9191: no14:42
sean-k-mooneyso there are two ways to do it sorry14:43
sean-k-mooneyyou can tag your flavor with a requried trait and tag each host rp with the custome trait14:43
sean-k-mooneyor you can use aggreates https://docs.openstack.org/nova/latest/reference/isolate-aggregates.html14:43
*** ociuhandu has quit IRC14:44
alexe9191I'd actually rather use the aggregates because that means i don't have to update each host and also update each new host. The documentation is however very confusing.14:44
sean-k-mooneyyou stil need to update each host14:44
*** ociuhandu has joined #openstack-nova14:44
sean-k-mooneyso there are 3 parts14:44
sean-k-mooneyfirst you update the hosts rescoue providers to have the custom tratis14:45
sean-k-mooneysecond you update the image or flavor to requrest the requred trait14:45
sean-k-mooneythird you enabel the placment prefilter and update the aggrate to enforace that only instance that request a given trait can land on that aggrate14:45
sean-k-mooneyalexe9191: when you add a tarit to a host you can land on that host if you request it or not14:46
sean-k-mooneywhen you add a trait to the image or flavor as required you will guarentte that you willl land on a host with that trait14:46
alexe9191I thought the placement prefilter was enabled by default? do I need to explicitly enable it?14:46
sean-k-mooneywhen you add the required trait to the aggreate you will guarentee you will only land in that aggreate if you asked for the requited trait14:47
sean-k-mooneyalexe9191: its disable by default so you have to enable it yes14:47
alexe9191Apologies for the question but it's not mentioned anywhere in nova.conf nor do I see a placement filter in the scheduler/filter source tree.14:49
alexe9191How would I enable it?14:49
sean-k-mooneyits covered in the doc14:49
sean-k-mooneyhttps://docs.openstack.org/nova/latest/reference/isolate-aggregates.html14:49
sean-k-mooneyyou enable https://docs.openstack.org/nova/latest/configuration/config.html#scheduler.enable_isolated_aggregate_filtering14:49
alexe9191aha. I though about placement as a whole not that specific pre-filter.14:50
sean-k-mooneyalexe9191: placemetn filter are what we call prefilter they are not implemented in scheduler/filters since they work differently14:50
sean-k-mooneypre-filter modify the quiery we pass to placment before the normal filters run to be more strict14:51
sean-k-mooneyalexe9191: they are implemented in https://github.com/openstack/nova/blob/master/nova/scheduler/request_filter.py14:51
alexe9191Thank you:)14:52
alexe9191That being said. I am wondering if there are any known bugs in scheduler in nova rocky?14:52
sean-k-mooneymost of them are disable by default although some are always on and we will be move more to on by default as we continue14:52
sean-k-mooneyin the schduler in general proably14:52
sean-k-mooneyin this code not that im direclty aware of that would affect your usecase14:53
alexe9191Allow me to be more specific.14:53
alexe9191We have availability zones and we have aggregates to group some hosts that has certain properties like I mentioned.14:53
alexe9191For some reason. the scheduling works just fine till it stops. And then the solution to that would be removing one host from the aggregates and readding it. And then scheduling would work again.14:54
sean-k-mooneythat is odd14:54
alexe9191It seems to fail mostly on the availability zone filter or the extra spec filter. The other day it said that property defined in the flavor does not match the metadata on the aggregate. But it does14:54
*** jawad_axd has quit IRC14:55
sean-k-mooneywell you can disable that if you enabel https://github.com/openstack/nova/blob/stable/rocky/nova/scheduler/request_filter.py#L6314:55
sean-k-mooneyalexe9191: you do not have the issolated aggrate feature in rocky that we were discussing14:55
sean-k-mooneybut you you have access to using placment for AZs14:55
sean-k-mooneyinstead of teh az filter14:55
alexe9191Indeed. This is why I am investigating moving what I can move to the placement. Also for performance reasons.14:56
sean-k-mooneyalexe9191: https://docs.openstack.org/nova/latest/admin/availability-zones.html#availability-zones-with-placement14:56
sean-k-mooneyalexe9191: that is still not enabled by default upstream but that is becasue we ment to do it in victoria and we got distrated14:57
sean-k-mooneyalexe9191: ill be enableing it by defualt this cycle and deprecating the AZ filter for removal in the Y release14:57
alexe9191may I ask what issues has been faced when it was enabled by default?14:57
sean-k-mooneynone that im aware of14:58
sean-k-mooneywe have not enabled it by default yet14:58
alexe9191ah I misread. Apologies.14:58
sean-k-mooneyi submited https://review.opendev.org/c/openstack/nova/+/74560514:59
alexe9191enabling this pre-filter will not have an effect on the extraspecs aggregate I assume? As I intend to slowly migrate the users to new flavors.14:59
sean-k-mooneyto do it last year but i got distracted14:59
sean-k-mooneyalexe9191: correct it will not14:59
sean-k-mooneywhat it will do is map all AZ to plamcent aggrate and then include the aggreate uuid in the query15:00
sean-k-mooneyso if you as for AZ X it will tell placment to only look at host that are in AZ x by mapping those host to a placment aggaet and stating the canidat hosts must be a member_of the AZ15:01
*** lpetrut has quit IRC15:01
*** nightmare_unreal has quit IRC15:01
alexe9191That is the desired behaviour.15:01
sean-k-mooneywe have been very carful to ensure that you can opt into the new plamcent feature gradulally15:02
sean-k-mooneyplacment should never break any of the exisitng filters15:02
*** rpittau is now known as rpittau|bbl15:03
sean-k-mooneybut it should narrow down thte host passed to it so the filters have to check less host by moveing more of the filtering to placment and sql instead of python15:03
*** dklyle has joined #openstack-nova15:03
alexe9191Excellent.15:04
alexe9191The mentioned bug about the scheduling, is it fixed in releases after rocky?15:05
*** jawad_axd has joined #openstack-nova15:05
alexe9191Does not seem to be: https://bugs.launchpad.net/nova/+bug/167721715:05
openstackLaunchpad bug 1677217 in OpenStack Compute (nova) " AggregateImagePropertiesIsolation filter return unwanted compute nodes" [Low,Triaged]15:05
*** macz_ has joined #openstack-nova15:06
sean-k-mooneyalexe9191: no its not really a bug15:06
sean-k-mooneyalexe9191: its a limitation fo the desgin fo filter15:06
alexe9191I see that there is a "workaround" sort to speak...15:07
sean-k-mooneythey cannot easily work the way they wanted it to which is why we built the placment feature to prevent it15:07
sean-k-mooneyyes you can set the key to something in all images15:07
sean-k-mooneythen it will see teh key and enforce the rules15:07
sean-k-mooneybut that is basically the best we can do in the exsitng filters15:08
*** mlavalle has joined #openstack-nova15:08
sean-k-mooneyalexe9191: there is an out of tree filter that kind fo fixes it https://opendev.org/x/nfv-filters/src/branch/master15:08
sean-k-mooneybut that is no longer maintined15:09
alexe9191Excellent. I am gonna try that on a subset of hosts and see what the results are. I wish there was a way to define the trait in the nova.conf file of the compute host though instead of adding it to each host one by one.15:09
alexe9191thanks for the tip:)15:09
sean-k-mooneyalexe9191: there is but not in rocky15:09
alexe9191aw?15:09
sean-k-mooneyin ussuri or wallably we added a provider.ymal file that allows you to add traits and resource provider via a file15:10
sean-k-mooneyalexe9191: https://docs.openstack.org/nova/latest/admin/managing-resource-providers.html15:10
sean-k-mooneyhttps://specs.openstack.org/openstack/nova-specs/specs/ussuri/approved/provider-config-file.html15:10
sean-k-mooneyso ya it was complted in victoria https://specs.openstack.org/openstack/nova-specs/specs/victoria/implemented/provider-config-file.html15:11
alexe9191This is good news!15:12
kashyapgibi: How did you unpack Sam Su's email?   For me in Mutt it only shows up as " Error: unable to create OpenSSL subprocess!"15:12
kashyapgibi: Thanks for quoting it in full!15:12
gibikashyap: I guess that it was sent from outlook so I forwarded it to my work email, and outlook was able to extract the content from the smime15:14
kashyapAh, I see15:14
kashyapThx :)15:14
gibiartom: hi! are you OK if we make the decision about the meeting schedule change tomorrow on the meeting?15:16
*** Alon_KS has quit IRC15:17
artomgibi, totally, it was my plan as well, and is in fact on the agenda :)15:17
gibiartom: awesome :)15:18
*** _erlon_ has quit IRC15:18
*** Alon_KS has joined #openstack-nova15:22
*** dave-mccowan has joined #openstack-nova15:41
*** jawad_axd has quit IRC15:41
*** ociuhandu has quit IRC15:45
*** lucasagomes has quit IRC15:59
*** k_mouza has quit IRC16:14
*** hamalq has joined #openstack-nova16:25
*** cgoncalves has quit IRC16:32
*** cgoncalves has joined #openstack-nova16:33
*** tesseract has quit IRC16:43
*** rpittau|bbl is now known as rpittau16:45
*** cgoncalves has quit IRC16:52
*** cgoncalves has joined #openstack-nova16:53
*** derekh has quit IRC17:02
*** ricolin_ has joined #openstack-nova17:03
*** ralonsoh has quit IRC17:11
*** links has quit IRC17:11
*** gyee has joined #openstack-nova17:12
*** dtantsur is now known as dtantsur|afk17:25
*** rpittau is now known as rpittau|afk17:37
*** ociuhandu has joined #openstack-nova17:46
*** ociuhandu has quit IRC17:52
*** k_mouza has joined #openstack-nova18:00
* stephenfin finishes for the evening o/18:01
*** k_mouza has quit IRC18:05
*** andrewbonney has quit IRC18:05
openstackgerritArtom Lifshitz proposed openstack/nova stable/wallaby: Test SRIOV port move operations with PCI conflicts  https://review.opendev.org/c/openstack/nova/+/79071018:17
openstackgerritArtom Lifshitz proposed openstack/nova stable/wallaby: Update SRIOV port pci_slot when unshelving  https://review.opendev.org/c/openstack/nova/+/79071118:17
openstackgerritArtom Lifshitz proposed openstack/nova stable/wallaby: Neutron fixture: don't clobber profile and vif_details if empty  https://review.opendev.org/c/openstack/nova/+/79223318:17
artomme: Damn, I'm getting KeyError on ['pci_slot'] in my func tests, clearly there's a patch that fixed that in the fixtures and I need to find it and backport it18:18
*** bbowen has quit IRC18:18
artomalso me: wrote the patch himself in master, forgot it even existed18:18
*** vishalmanchanda has quit IRC18:22
*** fyx has joined #openstack-nova18:30
sean-k-mooneygibi: by the way now that we have provider.yaml we could have bandwith or pps inventoreis chreated using it too right instead of having to modify nova18:34
sean-k-mooney*neutron18:34
sean-k-mooneyim not nessisarly saying we want to do that but i was just thinkink about the ovn case it might be nice to enabel gurenteed minium bandwith with ovn by having nova via provider.yaml report invetories of bandwithd18:36
*** belmoreira has quit IRC18:36
sean-k-mooneyas i replied on your spec though the way i would expect teh inventories to be configured woudl be useing pseudo agent binding by adding the config info to the external-ids column in the chassis table of the ovn-southdb for the given host18:37
sean-k-mooneyand have the ml2 driver report the inveotreis as normal18:37
sean-k-mooneyjust pointing out that since provider.yaml exsits taht might be another option18:37
*** sapd1_x has quit IRC18:56
*** whoami-rajat has quit IRC19:19
*** bbowen has joined #openstack-nova19:26
*** dklyle has quit IRC19:34
*** dklyle has joined #openstack-nova19:52
*** k_mouza has joined #openstack-nova20:15
*** MrClayPole_ has joined #openstack-nova20:15
« Tuesday, 2021-05-18 Index Thursday, 2021-05-20 »

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!