Thursday, 2022-12-08

opendevreviewyangzhipeng proposed openstack/nova master: when evacuate is performing, and restart compute node, if get instance info early, the instance state is not latest. this will reset instance task to error incorrectly, so refresh instance when modify instance state.  https://review.opendev.org/c/openstack/nova/+/86696006:46
opendevreviewyzp proposed openstack/nova master: Remove all tag if instance has beed hard deleted.  https://review.opendev.org/c/openstack/nova/+/86536206:53
opendevreviewyzp proposed openstack/nova master: Refresh instance when init instance in rebuilding  https://review.opendev.org/c/openstack/nova/+/86696007:54
opendevreviewyzp proposed openstack/nova master: Remove all tag if instance has beed hard deleted. Signed-off-by: yangzhipeng  <yangzhipeng@cmss.chinamobile.com>  https://review.opendev.org/c/openstack/nova/+/86536207:58
opendevreviewyzp proposed openstack/nova master: Remove all tag if instance has beed hard deleted. Signed-off-by: yangzhipeng  <yangzhipeng@cmss.chinamobile.com>  https://review.opendev.org/c/openstack/nova/+/86536208:00
opendevreviewyzp proposed openstack/nova master: Remove all tag if instance has beed hard deleted.  https://review.opendev.org/c/openstack/nova/+/86536208:03
opendevreviewnorman shen proposed openstack/nova master: Skip deleting instance info for same host migration  https://review.opendev.org/c/openstack/nova/+/86652108:04
opendevreviewyzp proposed openstack/nova master: Refresh instance when init instance in rebuilding  https://review.opendev.org/c/openstack/nova/+/86696009:48
opendevreviewyzp proposed openstack/nova master: Refresh instance when init instance.  https://review.opendev.org/c/openstack/nova/+/86696010:27
*** dasm|off is now known as dasm13:00
gibido we have a blocked nova gate? https://zuul.opendev.org/t/openstack/build/32c3c4a56eec4d7eac956a629abdee36213:24
gibiit looks we have constant failure in https://zuul.opendev.org/t/openstack/builds?job_name=nova-ceph-multistore&project=openstack/nova13:24
gibisince yesterday13:25
gibialso in https://zuul.opendev.org/t/openstack/builds?job_name=nova-grenade-multinode&project=openstack/nova13:25
gibi"venv: failed with tempest is not allowed, use allowlist_externals to allow it"13:26
sean-k-mooneythat sound like we are not installing tempet in the venv properly13:47
gibiyeah13:51
gibigrenade fails a bit differently but maybe related too13:51
gibithis seems related https://review.opendev.org/c/openstack/tempest/+/865314/1/tox.ini13:52
gibibut merged couple weeks ago13:53
sean-k-mooneyya so there was a mail about that 13:53
sean-k-mooneythe allowlist_external=* is problematic13:53
sean-k-mooneygibi: https://lists.openstack.org/pipermail/openstack-discuss/2022-November/031343.html13:57
sean-k-mooneyclarkb: ^ so that tempest fix may have broken grenade?13:59
sean-k-mooneygibi: my guess is the tempest venv is not being recreated14:02
sean-k-mooneyso on zed it would have had allowlist_external=* and used global pip 14:03
sean-k-mooneyas a result tempest was likely insalled outside of the venv?14:03
sean-k-mooneyand then with the master tox file it would have failed?14:03
gibiit fails in nova-ceph-multistore too so it is not grenade specific14:04
sean-k-mooneynot sure if that actully what is happening as i tough we used master tempetst regradels of the branch but i would guess its soemthign like that14:04
sean-k-mooneyoh ok14:04
sean-k-mooneyso may somethign related to our jobs14:04
frickleryes, that allowlist needs to include tempest14:05
* frickler can do a patch14:06
sean-k-mooneyfrickler: im not sure it should14:06
sean-k-mooneytempest shoudl be installed in teh venv14:06
sean-k-mooneywe shoudl not be using tempest form teh host right?14:06
sean-k-mooneywe install tempest in a venv because it is often not the same version of openstack14:06
gibithis is where we fail https://github.com/openstack/devstack/blob/master/lib/tempest#L73114:06
sean-k-mooneyand we do it that way to avoid the possibel depency conflcit that woudl create14:06
sean-k-mooneyso tempest should __not__ be in the allowlist14:07
sean-k-mooneyya so that shoudl be using tempst form the venv14:07
sean-k-mooneyshoudl it be tox -evenv-tempest  https://github.com/openstack/devstack/blob/master/lib/tempest#L70914:08
sean-k-mooneyinstead of venv14:08
sean-k-mooneytox -evenv-tempest seams to be what we use on the other lines14:08
gibihm, good point14:08
gibithat seems like the wrong env14:09
gibiI will push a patch14:09
frickleroh, cool, so this found an actual bug, nice14:09
ralonsohsean-k-mooney, thanks! I saw this issue in out CI too14:11
ralonsohour*14:11
gibidevstack fix is up https://review.opendev.org/c/openstack/devstack/+/86699714:14
opendevreviewKonrad Gube proposed openstack/nova-specs master: Use extend volume completion action  https://review.opendev.org/c/openstack/nova-specs/+/85549014:28
opendevreviewBence Romsics proposed openstack/nova master: doc: soft delete and shadow tables  https://review.opendev.org/c/openstack/nova/+/86700114:44
opendevreviewBence Romsics proposed openstack/nova master: doc: soft delete and shadow tables  https://review.opendev.org/c/openstack/nova/+/86700115:06
elendrysHello here, is this channel open to nova related questions or is it dedicated to developers ?15:19
fricklerelendrys: this is a bit of a grey area, #openstack is what is meant for general questions, but the chance to get an answer there is low, so you might as well try here directly15:45
fungialso the openstack-discuss@lists.openstack.org mailing list has a much higher chance of getting you an answer eventually15:47
elendrysOk thank you15:54
sean-k-mooneyelendrys: did you have a question in general15:54
sean-k-mooneywhile the channel is not for supprot we will try an point you in the right direction in genreal15:55
elendrysYes15:55
elendrysI have servers with cinder volumes, backed in PureStorage array over iscsi15:56
sean-k-mooneyok so the vm is using host mount iscsi block devices passthroug to the vms15:57
elendrysWhen i resize volumes attached to running servers, I can see in nova's log that it rescans the array to get the new size when cinder notifies15:57
elendrysBut in the vm, the size always have 1 notification late size15:57
elendrysI mean, a 10Gb vol expended to 20 stills show as 10, but if I extend to 30, it shows 2015:58
elendrysI can't get why (probably libvirt) it is missing something15:58
sean-k-mooneyhum interesting. 15:58
sean-k-mooneyam im not sure this is libvirt/qemu related15:59
elendrysIf I stop then start the server it shows 30 as intended15:59
sean-k-mooneyi woudl suppect this is more likely to be related to iscsid or simialr15:59
sean-k-mooneyelendrys: did you check on the host if the host kernel sees the correct volume size15:59
sean-k-mooneyyou should be able to find the dev path form the vm xml and check with lsblk or a similar too to see if it sees the correct current size16:00
sean-k-mooneyif the hosts sees the correct size then the issue is likely at the qemu level16:01
sean-k-mooneyelendrys: i dont think libvirt is invovled for voulme resize in this case16:01
elendrysIt does16:02
sean-k-mooneyim wondering is there anything you can do at the gues os level 16:05
elendrysIt is queued somewhere by nova16:05
elendrysI tried to force scsci bus scan but it doesn't change16:06
elendrysbut when I extend an extra time, the previous size is seen instantaneously by the guest16:06
elendrysit's just 1 extend late 16:07
*** ministry is now known as __ministry16:07
sean-k-mooneywell nova as far as i am aware will not need to notify qemu but perhaps we do16:17
clarkband those are sizes reported by lsblk?16:20
sean-k-mooneyunfortunetly i dont currently have the low level details loaded in my brain16:20
sean-k-mooneyclarkb: ya16:20
sean-k-mooneythe size of the block device repoted to the guest16:20
sean-k-mooneywhich should eventualy be show in lsblk on the host and guest16:21
clarkbya I'm wondering if the guest does an explicit check via lsblk if it sees correct info and the only problem is in propogating the not on demand info16:21
elendrysIf it is better I can open the discussion on the mailing list16:25
kgube_the size seems to come from os-brick LinuxSCSI which calls "blockdev --getsize64" after a rescan16:26
kgube_so maybe the rescan in LinuxSCSI.extend_volume is not working16:31
kgube_elendrys, have you tried it with debug logging?16:32
elendrysnope, but I can do that quickly16:33
kgube_there should be "Starting size: " and "volume size after scsi device rescan " in the log16:35
elendrysStarting size: 76235669504 extend_volume /usr/lib/python3/dist-packages/os_brick/initiator/linuxscsi.py:55516:36
elendrysvolume size after scsi device rescan 80530636800 extend_volume16:36
elendryshem16:39
elendrys- /dev/disk/by-id/dm-uuid-mpath-3624a9370cebbf98ad4534b81000386de has shown up. wait_for_path 16:39
elendrysmpath(/dev/disk/by-id/dm-uuid-mpath-3624a9370cebbf98ad4534b81000386de) current size 7623566950416:39
elendrysmpath(/dev/disk/by-id/dm-uuid-mpath-3624a9370cebbf98ad4534b81000386de) new size 7623566950416:39
elendrysExtend iSCSI Volume /dev/dm-28; new_size=76235669504 extend_volume16:40
elendrysResizing target device /dev/dm-28 to 76235669504 _resize_attached_volume16:40
elendrysI think that it rely on multipath but it is not refreshed quickly enough16:40
kgube_yeah that seems to come from the multipath code16:43
kgube_https://github.com/openstack/os-brick/blob/master/os_brick/initiator/linuxscsi.py#L63916:43
elendrysit explains why I see the previous size in the guest16:45
kgube_I don't know enough about scsi to have any idea what to do about this, though16:47
elendrysShould I fill an issue on the tracker or just ask the mailing list ?16:47
elendrysThank you for your time :)16:48
kgube_hm, I'm not sure. I'm very new to Nova dev16:53
elodillessean-k-mooney: sorry for pinging, but if you have some free minutes to review this old stable backport, that would be awesome: https://review.opendev.org/c/openstack/nova/+/76450416:55
elodilles(it has a lot of +1 on it)16:55
kgube_elendrys, maybe try the mailing list first there is also a lot of openstack users reading it16:57
kgube_elendrys, maybe someone has seen this before and found a workaround16:58
sean-k-mooneyoh ya that sure happy to see that merge17:08
sean-k-mooneyelodilles: give me 5 mins to finsih somehting and ill review it then17:09
elodillesthanks in advance o/17:12
sean-k-mooneyelodilles: once that merges im happy to continue moving this back to train so ill keep an eye out for that over then few days17:27
sean-k-mooneyelodilles: how are the stabel gates at the moment have they been affected by the tempest tox issue on master or the tox 4.0 issues 17:28
elodillessean-k-mooney: thanks, i'll try to keep an eye on them, too :) good question about the gate, I haven't checked that yet :S but i guess we will see :/17:32
*** dasm is now known as dasm|off21:50

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!