Tuesday, 2021-06-15

opendevreviewKeigo Noha proposed openstack/cinder master: Add libcgroup related packages in bindep.txt  https://review.opendev.org/c/openstack/cinder/+/79500900:55
opendevreviewZohar Mamedov proposed openstack/cinder-specs master: Implements: blueprint nvmeof-client-raid-healing-agent  https://review.opendev.org/c/openstack/cinder-specs/+/79636505:04
*** geguileo is now known as Guest219705:05
opendevreviewHui Jiang proposed openstack/cinder master: [DNM] just for test, trigger CI.  https://review.opendev.org/c/openstack/cinder/+/74134906:27
opendevreviewGirish Chilukuri proposed openstack/cinder master: [SVF]:Fix multiple lshost calls during attach.  https://review.opendev.org/c/openstack/cinder/+/77262306:43
opendevreviewGirish Chilukuri proposed openstack/cinder master: [SVF]:HyperSwap volume service status update  https://review.opendev.org/c/openstack/cinder/+/79128106:59
*** Guest2197 is now known as geguileo08:00
opendevreviewZohar Mamedov proposed openstack/cinder-specs master: NVMe-oF connection agent  https://review.opendev.org/c/openstack/cinder-specs/+/79636509:28
lxkongHi there, could anybody please help to look at this issue when attaching a volume to a VM? https://dpaste.com/D5E6Z5TQR#10:40
lxkongdefault devstack installation 10:40
lxkongcinder master branch as of today10:40
lxkongthere is no special cinder related config in my local.conf file10:41
opendevreviewHelen Walsh proposed openstack/cinder master: PowerMax Driver - Improve error handling around deletes  https://review.opendev.org/c/openstack/cinder/+/79628613:29
opendevreviewBrian Rosmaita proposed openstack/cinder master: DNM: cinderlib wallaby development check  https://review.opendev.org/c/openstack/cinder/+/79647113:31
opendevreviewEric Harney proposed openstack/cinder master: LVM: Use --readonly for lvdisplay in lv_has_snapshot  https://review.opendev.org/c/openstack/cinder/+/77212613:45
tbarroneharney: rosmaita: the extra-specs spec owed from PTG is posted here: https://review.opendev.org/c/openstack/cinder-specs/+/79616613:55
tbarroneharney: rosmaita: sorry it's a bit down to the wire w.r.t. deadlines.  13:55
rosmaitatbarron: just saw it, and you are almost a week early :)13:56
tbarroneharney: rosmaita: IMO it's ready except I want Matt to double-check my description of the SoS use case and13:56
tbarronas abishop says, we should enumerate the initial set of TENANT_VISIBLE_EXTRA_SPECS and make clear that it's hard-coded13:57
tbarroncorrelated to a microversion13:57
tbarronif you want to add more later, you can, with an api change13:57
tbarronrosmaita: That's a very charitable attitude.13:58
tbarronrosmaita: eharney: the basic idea is here: https://review.opendev.org/c/openstack/cinder/+/79604913:58
tbarronrosmaita: eharney: and here are a couple of cleanup patches from my code survey for this project:13:59
rosmaitatbarron: ty14:01
tbarronrosmaita: yw, but I think it's me that is on bended knee saying please and thank you14:02
*** ricolin_ is now known as ricolin16:26
*** ricolin_ is now known as ricolin17:32
jungleboyjrosmaita:  Have you guys been pinged about issues with gate performance?19:49
rosmaitajungleboyj: not that i'm aware of19:51
jungleboyjSo, in the TC meeting last week (I was out) Cinder is still being highlighted as failing the gate too often.19:55
toskywell, it's more of "I heard of"19:55
jungleboyjI have been behind on reviews so I haven't seen how often things have been failing.19:56
toskymay I suggest to have specific pointers and data before we go into this? 19:56
jungleboyjtosky: ++19:56
jungleboyjdansmith: Do you have specific pointers?19:56
toskyas I've mentioned, cinder-tempest-plugin-lvm-lio-barbican is failing due to barbican sqlalchemy issues (there are patches)19:58
toskyand that's going on for at least one week19:58
jungleboyjtosky:  Ok.  So that may be the issue that was being referred to.  Are there patches that need review to resolve that?19:59
toskyand the issue has been raised with the barbican people today during the weekly meeting20:00
dansmithjungleboyj: I don't have any notes since the last time I provided them, no20:00
dansmithjungleboyj: just the seat-of-the-pants feeling that most of my rechecks last week were volume test fails20:01
jungleboyjtosky:  So we are waiting on barbican to resolve?20:01
toskydansmith: if it's cinder-tempest-plugin-lvm-lio-barbican, it's the barbican issue20:01
toskyjust check the logs20:01
dansmithjungleboyj: I can try to take more notes, but that's more effort to try and keep that info organized20:01
rosmaitasorry, my laptop is sensitive to any criticism of cinder and required a reboot20:01
dansmithtosky: yeah, I haven't seen any of those keywords personally, but I haven't been digging into the fails lately20:01
toskydansmith: sure, please think about my (our) point of view: it's not nice to be flagged as the bad people just based on "maybe"20:02
jungleboyjrosmaita: Bwah ha ha20:02
jungleboyjDude, you need a new laptop.  I should have pinged you with the sale we had last week.20:03
dansmithtosky: I'm not the only one who has flagged the issue, and I think I've been constructive in my comments and previous attempts to collect data20:03
jungleboyjSo, currently, we have a known issue we are waiting to have fixed.20:03
toskydansmith: and the previous comments (when you collected the data) were accurate and people have worked to solve the issues20:04
toskydansmith: so it would be useful to continue in that same way20:04
toskyagain: https://zuul.openstack.org/builds?job_name=cinder-tempest-plugin-lvm-lio-barbican20:04
jungleboyjI think once that is resolved it would be fair to ask the team to do a review of failures we are seeing from Zuul and see if there is a pattern.  Could be an edge case like we had before where we were running out of disk space.20:04
toskyall jobs fails while deploying barbican20:04
rosmaitai just put a reminder in the barbican channel to review https://review.opendev.org/c/openstack/barbican/+/79628420:05
toskyand that's a known issue, with patches provided (even by cinder people /me looks at geguileo) in parallel to barbican people and we are waiting on those20:05
jungleboyjrosmaita:  ++  Thank you.20:05
dansmithtosky: AFAIK, none of the projects I contribute to even run that job, so I don't think that includes any of the failures I've seen lately20:06
toskyare we talking about volume tests failure in other jobs (like generic tempest-all &co) ?20:07
mnaseri think so20:07
rosmaitafair enough, but it would really help if we could get an elasticsearch query so we know what exact tests we are talking about here, and what the failures are20:07
dansmiththis is one I see a ton: https://7d817d44c5b67f01d22a-c45b8f440d62b7dd5b1adf370d99b8a4.ssl.cf1.rackcdn.com/788077/15/check/tempest-integrated-storage/d348bf1/testr_results.html20:07
rosmaitayou mean you see a failure in teardown a lot?20:08
dansmithyeah, it gets reported against different tests a lot of course20:09
dansmithbut that "failed to delete because error_deleting" state seems to happen quite a bit20:09
jungleboyjDreaded volumes in error_deleting 20:09
toskythat's useful, thanks20:10
dansmithtosky: fwiw, that was in the previous batch I reported, so when I continue to see similar patterns, I assume that the same things are likely still problems20:10
mnaserStderr: '  /dev/sda1: open failed: No such file or directory\n  /dev/sda15: stat failed: No such file or directory\n  Path /dev/sda15 no longer valid for device(8,15)\n  /dev/sda1: stat failed: No such file or directory\n  Path /dev/sda1 no longer valid for device(8,1)\n  /dev/sda15: stat failed: No such file or directory\n  Path /dev/sda15 no longer valid for device(8,15)\n  Device open /dev/sda 8:0 failed errno 2\n  Device open 20:10
mnaser/dev/sda 8:0 failed errno 2\n  WARNING: Scan ignoring device 8:0 with no paths.\n'20:10
dansmithI'm really really not trying to be un-constructive, FWIW, and I think I usually put my money where my mouth is on that20:11
mnaserlooks like rootwrap is calling `lvdisplay --noheading -C -o Attr stack-volumes-lvmdriver-1/volume-cf7f9fe4-57c3-4f21-833c-8ae1dd09061a` and it's actually failing20:11
mnaserthis ran on rax, i think the root device is vda there?20:12
mnaserwonder where teh sda1/sda15 came from20:12
dansmithif it's using iscsi, those might be the initiator devices?20:12
mnaserso indeed, rax systems uses /dev/xvdXX20:13
dansmithanyway, I gotta run an errand, bbl20:13
rosmaitawe can discuss this at tomorrow's cinder meeting20:15
mnasergetting systemd-journal-remote instaled locally20:15
jungleboyjrosmaita:  Yeah.  20:15
mnaserim getting the journald output to debug locallynow20:16
jungleboyjDumb question but why does it say /dev/sda1: open failed but then the rest of the error messages says /dev/sda15 ?20:16
mnaseri tink all of /dev/sda disappears20:16
mnaserok so20:17
mnasersda1 and sda15 are both part of sda and get mounted via iscsi20:17
mnaserJun 14 15:59:34 ubuntu-focal-rax-iad-0025106244 kernel: lvdisplay[151617]: segfault at 800 ip 00007fbcb95e0860 sp 00007ffef3fc7f28 error 4 in libc-2.31.so[7fbcb948c000+178000]20:22
mnaserJun 14 15:59:34 ubuntu-focal-rax-iad-0025106244 kernel: Code: 84 00 00 00 00 00 0f 1f 40 00 f3 0f 1e fa 48 89 f1 49 89 d0 48 89 fa 4d 85 c0 0f 84 ca 20 00 00 49 83 f8 08 0f 86 60 21 00 00 <80> 39 00 0f 84 c7 1c 00 00 80 79 01 00 0f 84 dd 1c 00 00 80 79 0220:22
toskyoh, right https://review.opendev.org/c/openstack/cinder/+/77212620:27
mnaserlooks like lyarwood is already on it, it looks like more commands need code to recover from these20:27
toskymnaser: see the review ^^20:27
jungleboyjCool.  So, need to get barbican fixed so we can merge that.  :-)20:29
lxkongHi there, could anybody please help to look at this issue when attaching a volume to a VM? https://dpaste.com/D5E6Z5TQR#20:56
opendevreviewSofia Enriquez proposed openstack/cinder-tempest-plugin master: [Test][DMT] Check tls-proxy support  https://review.opendev.org/c/openstack/cinder-tempest-plugin/+/79458021:12
jungleboyjlxkong  Have you verified that port 3260 isn't already in use?21:36
jungleboyjlxkong:  Could also be a firewall issue?21:36
eharneythat usually happens when port 3260 is in use by scsi-target-utils/tgtd21:38
lxkongjungleboyj, eharney, thanks for both of your replies, yeah, I've checked, 3260 was used by tgt service23:05
lxkongwhich I manually stopped and disabled23:05
lxkongI'm trying again23:05
lxkongit works now, thanks for the help from all of you guys23:09

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!