| opendevreview | Keigo Noha proposed openstack/cinder master: Add libcgroup related packages in bindep.txt https://review.opendev.org/c/openstack/cinder/+/795009 | 00:55 |
|---|---|---|
| opendevreview | Zohar Mamedov proposed openstack/cinder-specs master: Implements: blueprint nvmeof-client-raid-healing-agent https://review.opendev.org/c/openstack/cinder-specs/+/796365 | 05:04 |
| *** geguileo is now known as Guest2197 | 05:05 | |
| opendevreview | Hui Jiang proposed openstack/cinder master: [DNM] just for test, trigger CI. https://review.opendev.org/c/openstack/cinder/+/741349 | 06:27 |
| opendevreview | Girish Chilukuri proposed openstack/cinder master: [SVF]:Fix multiple lshost calls during attach. https://review.opendev.org/c/openstack/cinder/+/772623 | 06:43 |
| opendevreview | Girish Chilukuri proposed openstack/cinder master: [SVF]:HyperSwap volume service status update https://review.opendev.org/c/openstack/cinder/+/791281 | 06:59 |
| *** Guest2197 is now known as geguileo | 08:00 | |
| opendevreview | Zohar Mamedov proposed openstack/cinder-specs master: NVMe-oF connection agent https://review.opendev.org/c/openstack/cinder-specs/+/796365 | 09:28 |
| lxkong | Hi there, could anybody please help to look at this issue when attaching a volume to a VM? https://dpaste.com/D5E6Z5TQR# | 10:40 |
| lxkong | default devstack installation | 10:40 |
| lxkong | cinder master branch as of today | 10:40 |
| lxkong | there is no special cinder related config in my local.conf file | 10:41 |
| opendevreview | Helen Walsh proposed openstack/cinder master: PowerMax Driver - Improve error handling around deletes https://review.opendev.org/c/openstack/cinder/+/796286 | 13:29 |
| opendevreview | Brian Rosmaita proposed openstack/cinder master: DNM: cinderlib wallaby development check https://review.opendev.org/c/openstack/cinder/+/796471 | 13:31 |
| opendevreview | Eric Harney proposed openstack/cinder master: LVM: Use --readonly for lvdisplay in lv_has_snapshot https://review.opendev.org/c/openstack/cinder/+/772126 | 13:45 |
| tbarron | eharney: rosmaita: the extra-specs spec owed from PTG is posted here: https://review.opendev.org/c/openstack/cinder-specs/+/796166 | 13:55 |
| tbarron | eharney: rosmaita: sorry it's a bit down to the wire w.r.t. deadlines. | 13:55 |
| rosmaita | tbarron: just saw it, and you are almost a week early :) | 13:56 |
| tbarron | eharney: rosmaita: IMO it's ready except I want Matt to double-check my description of the SoS use case and | 13:56 |
| tbarron | as abishop says, we should enumerate the initial set of TENANT_VISIBLE_EXTRA_SPECS and make clear that it's hard-coded | 13:57 |
| tbarron | correlated to a microversion | 13:57 |
| tbarron | if you want to add more later, you can, with an api change | 13:57 |
| tbarron | rosmaita: That's a very charitable attitude. | 13:58 |
| tbarron | rosmaita: eharney: the basic idea is here: https://review.opendev.org/c/openstack/cinder/+/796049 | 13:58 |
| tbarron | rosmaita: eharney: and here are a couple of cleanup patches from my code survey for this project: | 13:59 |
| tbarron | https://review.opendev.org/c/openstack/cinder/+/796113 | 13:59 |
| tbarron | https://review.opendev.org/c/openstack/cinder/+/796114 | 14:00 |
| rosmaita | tbarron: ty | 14:01 |
| tbarron | rosmaita: yw, but I think it's me that is on bended knee saying please and thank you | 14:02 |
| *** ricolin_ is now known as ricolin | 16:26 | |
| *** ricolin_ is now known as ricolin | 17:32 | |
| jungleboyj | rosmaita: Have you guys been pinged about issues with gate performance? | 19:49 |
| jungleboyj | Again? | 19:49 |
| rosmaita | jungleboyj: not that i'm aware of | 19:51 |
| jungleboyj | So, in the TC meeting last week (I was out) Cinder is still being highlighted as failing the gate too often. | 19:55 |
| tosky | well, it's more of "I heard of" | 19:55 |
| jungleboyj | I have been behind on reviews so I haven't seen how often things have been failing. | 19:56 |
| tosky | may I suggest to have specific pointers and data before we go into this? | 19:56 |
| jungleboyj | tosky: ++ | 19:56 |
| jungleboyj | dansmith: Do you have specific pointers? | 19:56 |
| tosky | as I've mentioned, cinder-tempest-plugin-lvm-lio-barbican is failing due to barbican sqlalchemy issues (there are patches) | 19:58 |
| tosky | and that's going on for at least one week | 19:58 |
| jungleboyj | tosky: Ok. So that may be the issue that was being referred to. Are there patches that need review to resolve that? | 19:59 |
| tosky | yes | 19:59 |
| tosky | and the issue has been raised with the barbican people today during the weekly meeting | 20:00 |
| dansmith | jungleboyj: I don't have any notes since the last time I provided them, no | 20:00 |
| dansmith | jungleboyj: just the seat-of-the-pants feeling that most of my rechecks last week were volume test fails | 20:01 |
| jungleboyj | tosky: So we are waiting on barbican to resolve? | 20:01 |
| tosky | dansmith: if it's cinder-tempest-plugin-lvm-lio-barbican, it's the barbican issue | 20:01 |
| tosky | just check the logs | 20:01 |
| dansmith | jungleboyj: I can try to take more notes, but that's more effort to try and keep that info organized | 20:01 |
| rosmaita | sorry, my laptop is sensitive to any criticism of cinder and required a reboot | 20:01 |
| dansmith | tosky: yeah, I haven't seen any of those keywords personally, but I haven't been digging into the fails lately | 20:01 |
| tosky | dansmith: sure, please think about my (our) point of view: it's not nice to be flagged as the bad people just based on "maybe" | 20:02 |
| jungleboyj | rosmaita: Bwah ha ha | 20:02 |
| jungleboyj | Dude, you need a new laptop. I should have pinged you with the sale we had last week. | 20:03 |
| dansmith | tosky: I'm not the only one who has flagged the issue, and I think I've been constructive in my comments and previous attempts to collect data | 20:03 |
| jungleboyj | So, currently, we have a known issue we are waiting to have fixed. | 20:03 |
| tosky | dansmith: and the previous comments (when you collected the data) were accurate and people have worked to solve the issues | 20:04 |
| tosky | dansmith: so it would be useful to continue in that same way | 20:04 |
| tosky | again: https://zuul.openstack.org/builds?job_name=cinder-tempest-plugin-lvm-lio-barbican | 20:04 |
| jungleboyj | I think once that is resolved it would be fair to ask the team to do a review of failures we are seeing from Zuul and see if there is a pattern. Could be an edge case like we had before where we were running out of disk space. | 20:04 |
| tosky | all jobs fails while deploying barbican | 20:04 |
| rosmaita | i just put a reminder in the barbican channel to review https://review.opendev.org/c/openstack/barbican/+/796284 | 20:05 |
| tosky | and that's a known issue, with patches provided (even by cinder people /me looks at geguileo) in parallel to barbican people and we are waiting on those | 20:05 |
| jungleboyj | rosmaita: ++ Thank you. | 20:05 |
| dansmith | tosky: AFAIK, none of the projects I contribute to even run that job, so I don't think that includes any of the failures I've seen lately | 20:06 |
| tosky | are we talking about volume tests failure in other jobs (like generic tempest-all &co) ? | 20:07 |
| mnaser | i think so | 20:07 |
| rosmaita | fair enough, but it would really help if we could get an elasticsearch query so we know what exact tests we are talking about here, and what the failures are | 20:07 |
| dansmith | this is one I see a ton: https://7d817d44c5b67f01d22a-c45b8f440d62b7dd5b1adf370d99b8a4.ssl.cf1.rackcdn.com/788077/15/check/tempest-integrated-storage/d348bf1/testr_results.html | 20:07 |
| rosmaita | you mean you see a failure in teardown a lot? | 20:08 |
| dansmith | yeah, it gets reported against different tests a lot of course | 20:09 |
| dansmith | but that "failed to delete because error_deleting" state seems to happen quite a bit | 20:09 |
| jungleboyj | Dreaded volumes in error_deleting | 20:09 |
| tosky | that's useful, thanks | 20:10 |
| dansmith | tosky: fwiw, that was in the previous batch I reported, so when I continue to see similar patterns, I assume that the same things are likely still problems | 20:10 |
| mnaser | Stderr: ' /dev/sda1: open failed: No such file or directory\n /dev/sda15: stat failed: No such file or directory\n Path /dev/sda15 no longer valid for device(8,15)\n /dev/sda1: stat failed: No such file or directory\n Path /dev/sda1 no longer valid for device(8,1)\n /dev/sda15: stat failed: No such file or directory\n Path /dev/sda15 no longer valid for device(8,15)\n Device open /dev/sda 8:0 failed errno 2\n Device open | 20:10 |
| mnaser | /dev/sda 8:0 failed errno 2\n WARNING: Scan ignoring device 8:0 with no paths.\n' | 20:10 |
| dansmith | I'm really really not trying to be un-constructive, FWIW, and I think I usually put my money where my mouth is on that | 20:11 |
| mnaser | looks like rootwrap is calling `lvdisplay --noheading -C -o Attr stack-volumes-lvmdriver-1/volume-cf7f9fe4-57c3-4f21-833c-8ae1dd09061a` and it's actually failing | 20:11 |
| mnaser | this ran on rax, i think the root device is vda there? | 20:12 |
| mnaser | wonder where teh sda1/sda15 came from | 20:12 |
| dansmith | if it's using iscsi, those might be the initiator devices? | 20:12 |
| mnaser | https://7d817d44c5b67f01d22a-c45b8f440d62b7dd5b1adf370d99b8a4.ssl.cf1.rackcdn.com/788077/15/check/tempest-integrated-storage/d348bf1/controller/logs/df.txt | 20:13 |
| jungleboyj | Strange. | 20:13 |
| mnaser | so indeed, rax systems uses /dev/xvdXX | 20:13 |
| dansmith | anyway, I gotta run an errand, bbl | 20:13 |
| rosmaita | we can discuss this at tomorrow's cinder meeting | 20:15 |
| mnaser | getting systemd-journal-remote instaled locally | 20:15 |
| jungleboyj | rosmaita: Yeah. | 20:15 |
| mnaser | im getting the journald output to debug locallynow | 20:16 |
| jungleboyj | Dumb question but why does it say /dev/sda1: open failed but then the rest of the error messages says /dev/sda15 ? | 20:16 |
| mnaser | i tink all of /dev/sda disappears | 20:16 |
| mnaser | ok so | 20:17 |
| mnaser | sda1 and sda15 are both part of sda and get mounted via iscsi | 20:17 |
| jungleboyj | Ok. | 20:21 |
| mnaser | Jun 14 15:59:34 ubuntu-focal-rax-iad-0025106244 kernel: lvdisplay[151617]: segfault at 800 ip 00007fbcb95e0860 sp 00007ffef3fc7f28 error 4 in libc-2.31.so[7fbcb948c000+178000] | 20:22 |
| mnaser | Jun 14 15:59:34 ubuntu-focal-rax-iad-0025106244 kernel: Code: 84 00 00 00 00 00 0f 1f 40 00 f3 0f 1e fa 48 89 f1 49 89 d0 48 89 fa 4d 85 c0 0f 84 ca 20 00 00 49 83 f8 08 0f 86 60 21 00 00 <80> 39 00 0f 84 c7 1c 00 00 80 79 01 00 0f 84 dd 1c 00 00 80 79 02 | 20:22 |
| jungleboyj | Yuck. | 20:24 |
| mnaser | https://bugs.launchpad.net/cinder/+bug/1901783/comments/15 | 20:26 |
| tosky | oh, right https://review.opendev.org/c/openstack/cinder/+/772126 | 20:27 |
| mnaser | looks like lyarwood is already on it, it looks like more commands need code to recover from these | 20:27 |
| tosky | mnaser: see the review ^^ | 20:27 |
| jungleboyj | Cool. So, need to get barbican fixed so we can merge that. :-) | 20:29 |
| lxkong | Hi there, could anybody please help to look at this issue when attaching a volume to a VM? https://dpaste.com/D5E6Z5TQR# | 20:56 |
| opendevreview | Sofia Enriquez proposed openstack/cinder-tempest-plugin master: [Test][DMT] Check tls-proxy support https://review.opendev.org/c/openstack/cinder-tempest-plugin/+/794580 | 21:12 |
| jungleboyj | lxkong Have you verified that port 3260 isn't already in use? | 21:36 |
| jungleboyj | lxkong: Could also be a firewall issue? | 21:36 |
| eharney | that usually happens when port 3260 is in use by scsi-target-utils/tgtd | 21:38 |
| lxkong | jungleboyj, eharney, thanks for both of your replies, yeah, I've checked, 3260 was used by tgt service | 23:05 |
| lxkong | which I manually stopped and disabled | 23:05 |
| lxkong | I'm trying again | 23:05 |
| lxkong | it works now, thanks for the help from all of you guys | 23:09 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!