| KeithMnemonic | smcginnis: jungleboyj: thanks i think those ScaleIO are ready for +A | 00:08 |
|---|---|---|
| *** markvoelker has quit IRC | 00:17 | |
| *** markvoelker has joined #openstack-cinder | 00:17 | |
| *** markvoelker has quit IRC | 00:21 | |
| *** erlon has quit IRC | 00:35 | |
| *** raunak12 has quit IRC | 00:58 | |
| *** mhen has quit IRC | 01:05 | |
| *** mhen has joined #openstack-cinder | 01:07 | |
| *** ianychoi has quit IRC | 01:17 | |
| *** moshele has joined #openstack-cinder | 01:26 | |
| *** ianychoi has joined #openstack-cinder | 01:28 | |
| *** Dinesh_Bhor has joined #openstack-cinder | 01:31 | |
| *** Dinesh_Bhor has quit IRC | 01:38 | |
| *** ianychoi has quit IRC | 01:42 | |
| *** Dinesh_Bhor has joined #openstack-cinder | 01:47 | |
| openstackgerrit | Xianjin Shao proposed openstack/cinder master: This adds the missing 'is_public' parameter for volume type details https://review.openstack.org/607115 | 01:54 |
| *** dave-mccowan has joined #openstack-cinder | 01:56 | |
| *** markvoelker has joined #openstack-cinder | 02:18 | |
| *** ianychoi has joined #openstack-cinder | 02:19 | |
| *** moshele has quit IRC | 02:30 | |
| *** markvoelker has quit IRC | 02:51 | |
| *** Dinesh_Bhor has quit IRC | 03:00 | |
| *** Dinesh_Bhor has joined #openstack-cinder | 03:01 | |
| *** dave-mccowan has quit IRC | 03:08 | |
| *** raunak12 has joined #openstack-cinder | 03:27 | |
| openstackgerrit | Merged openstack/os-brick master: Remove meanless debug log https://review.openstack.org/605380 | 03:43 |
| *** markvoelker has joined #openstack-cinder | 03:48 | |
| *** penick has quit IRC | 03:52 | |
| *** Dinesh_Bhor has quit IRC | 04:01 | |
| *** Dinesh_Bhor has joined #openstack-cinder | 04:05 | |
| *** Dinesh_Bhor has quit IRC | 04:20 | |
| *** markvoelker has quit IRC | 04:21 | |
| openstackgerrit | Rajat Dhasmana proposed openstack/cinder master: Fix: storage_pools key in Huawei Driver https://review.openstack.org/607299 | 04:23 |
| *** gkadam has joined #openstack-cinder | 04:32 | |
| *** gkadam has quit IRC | 04:32 | |
| *** raunak12 has quit IRC | 04:49 | |
| *** Dinesh_Bhor has joined #openstack-cinder | 04:58 | |
| *** Bhujay has joined #openstack-cinder | 05:01 | |
| *** Bhujay has quit IRC | 05:01 | |
| *** Bhujay has joined #openstack-cinder | 05:02 | |
| *** markvoelker has joined #openstack-cinder | 05:18 | |
| *** markvoelker has quit IRC | 05:52 | |
| *** moshele has joined #openstack-cinder | 06:00 | |
| *** vivsoni has joined #openstack-cinder | 06:10 | |
| *** Dinesh_Bhor has quit IRC | 06:11 | |
| *** dims has quit IRC | 06:24 | |
| *** e0ne has joined #openstack-cinder | 06:24 | |
| *** dims has joined #openstack-cinder | 06:26 | |
| *** raghavendrat has joined #openstack-cinder | 06:29 | |
| raghavendrat | hello | 06:29 |
| *** dims has quit IRC | 06:34 | |
| *** Dinesh_Bhor has joined #openstack-cinder | 06:35 | |
| *** vivsoni has quit IRC | 06:35 | |
| *** dims has joined #openstack-cinder | 06:35 | |
| *** dpawlik has joined #openstack-cinder | 06:39 | |
| *** pcaruana has joined #openstack-cinder | 06:51 | |
| whoami-rajat | raghavendrat: Hi, any queries? | 06:53 |
| *** vivsoni has joined #openstack-cinder | 06:53 | |
| *** sdin has joined #openstack-cinder | 07:10 | |
| *** raghavendrat has left #openstack-cinder | 07:10 | |
| *** Dinesh_Bhor has quit IRC | 07:19 | |
| *** jroll has quit IRC | 07:19 | |
| *** logan- has quit IRC | 07:19 | |
| *** sdinescu has quit IRC | 07:19 | |
| *** pocketprotector has quit IRC | 07:19 | |
| *** odyssey4me has quit IRC | 07:19 | |
| *** amoralej|off is now known as amoralej | 07:27 | |
| *** jroll has joined #openstack-cinder | 07:32 | |
| *** logan- has joined #openstack-cinder | 07:32 | |
| *** odyssey4me has joined #openstack-cinder | 07:32 | |
| *** gkadam has joined #openstack-cinder | 07:44 | |
| *** rcernin has quit IRC | 07:55 | |
| *** vivsoni has quit IRC | 08:07 | |
| *** jroll has quit IRC | 08:13 | |
| *** logan- has quit IRC | 08:13 | |
| *** odyssey4me has quit IRC | 08:13 | |
| *** markvoelker has joined #openstack-cinder | 08:18 | |
| *** vivsoni has joined #openstack-cinder | 08:18 | |
| *** jroll has joined #openstack-cinder | 08:27 | |
| *** logan- has joined #openstack-cinder | 08:27 | |
| *** odyssey4me has joined #openstack-cinder | 08:27 | |
| *** Emine has joined #openstack-cinder | 08:41 | |
| openstackgerrit | Ivaylo Mitev proposed openstack/cinder master: Implement volume capacity stats for VMware https://review.openstack.org/605762 | 08:41 |
| *** jroll has quit IRC | 08:51 | |
| *** logan- has quit IRC | 08:51 | |
| *** odyssey4me has quit IRC | 08:51 | |
| *** markvoelker has quit IRC | 08:51 | |
| *** jroll has joined #openstack-cinder | 09:05 | |
| *** logan- has joined #openstack-cinder | 09:05 | |
| *** odyssey4me has joined #openstack-cinder | 09:05 | |
| *** Emine has quit IRC | 09:12 | |
| *** Emine has joined #openstack-cinder | 09:25 | |
| *** sdin has quit IRC | 09:52 | |
| *** sdin has joined #openstack-cinder | 09:52 | |
| *** sdin has quit IRC | 09:52 | |
| *** sdinescu has joined #openstack-cinder | 09:53 | |
| *** Emine has quit IRC | 10:07 | |
| *** kukacz has quit IRC | 10:12 | |
| *** kukacz has joined #openstack-cinder | 10:13 | |
| *** gkadam has quit IRC | 10:24 | |
| *** luizbag has joined #openstack-cinder | 10:31 | |
| *** luizbag has quit IRC | 10:35 | |
| *** luizbag has joined #openstack-cinder | 10:36 | |
| *** mvkr has quit IRC | 10:48 | |
| *** markvoelker has joined #openstack-cinder | 10:49 | |
| *** ganso has joined #openstack-cinder | 10:52 | |
| *** mvkr has joined #openstack-cinder | 11:02 | |
| *** erlon has joined #openstack-cinder | 11:03 | |
| *** moshele has quit IRC | 11:03 | |
| *** markvoelker has quit IRC | 11:22 | |
| openstackgerrit | Helen Walsh proposed openstack/cinder master: VMAX Driver - Failover Unisphere Support https://review.openstack.org/570401 | 11:23 |
| *** gary_perkins has joined #openstack-cinder | 11:30 | |
| openstackgerrit | Rajat Dhasmana proposed openstack/cinder master: API-REF:os-quota-sets v2 API reference has the wrong parameters https://review.openstack.org/607533 | 11:33 |
| *** lemko has joined #openstack-cinder | 11:35 | |
| lemko | Hi. I'm using queens and iscsi with multipath (IBM storwize is the backend). Creating volumes from images is extremely slow. Same as detaching volumes. Any idea what might be wrong? I'm talking about something like 10 mins to detach a volume from an instance. | 11:43 |
| *** abishop_ has joined #openstack-cinder | 11:45 | |
| lemko | Could it be due to old open-iscsi package? | 11:46 |
| *** e0ne has quit IRC | 12:04 | |
| *** Emine has joined #openstack-cinder | 12:14 | |
| *** moshele has joined #openstack-cinder | 12:15 | |
| *** e0ne has joined #openstack-cinder | 12:16 | |
| *** amoralej is now known as amoralej|lunch | 12:18 | |
| *** dave-mccowan has joined #openstack-cinder | 12:19 | |
| smcginnis | lemko: There should be something in the logs at least showing the timestamps of different operations. You might be able to tell from there where the delay is. | 12:36 |
| smcginnis | lemko: It definitely should not take that long. | 12:36 |
| *** rosmaita has joined #openstack-cinder | 12:38 | |
| *** rosmaita has quit IRC | 12:45 | |
| openstackgerrit | Sean McGinnis proposed openstack/cinder master: Add missing 'is_public' volume type parameter https://review.openstack.org/607115 | 12:46 |
| *** rosmaita has joined #openstack-cinder | 12:46 | |
| *** moshele has quit IRC | 12:53 | |
| openstackgerrit | Sean McGinnis proposed openstack/cinder master: Add missing 'is_public' volume type parameter https://review.openstack.org/607115 | 13:09 |
| *** mvkr has quit IRC | 13:23 | |
| *** eharney has joined #openstack-cinder | 13:23 | |
| *** dustins has joined #openstack-cinder | 13:32 | |
| *** amoralej|lunch is now known as aoralej | 13:32 | |
| *** mriedem has joined #openstack-cinder | 13:37 | |
| *** aoralej is now known as amoralej | 13:41 | |
| *** mvkr has joined #openstack-cinder | 13:50 | |
| openstackgerrit | Jay Rubenstein proposed openstack/cinder master: During volume extend scaling io by volume size did not occur https://review.openstack.org/606959 | 13:52 |
| *** ThomasWhite has quit IRC | 14:03 | |
| *** woojay has quit IRC | 14:04 | |
| jungleboyj | smcginnis: If you are bored do you want to take a look at this: https://review.openstack.org/607058 | 14:05 |
| jungleboyj | I am going to have rebase the second one I realized. Probably should have chained them. | 14:05 |
| jungleboyj | Once this goes in I can propose the third: https://review.openstack.org/607062 | 14:06 |
| *** Bhujay has quit IRC | 14:06 | |
| smcginnis | jungleboyj: Was there a follow up to my comments? | 14:06 |
| *** hoangcx has quit IRC | 14:06 | |
| *** Bhujay has joined #openstack-cinder | 14:07 | |
| jungleboyj | smcginnis: Doh, did I miss comments? | 14:07 |
| smcginnis | Apparently. :) | 14:07 |
| jungleboyj | :-p | 14:08 |
| jungleboyj | Sorry about that. Will address those now. | 14:08 |
| jungleboyj | I had been watching the patch and totally missed the -1. | 14:09 |
| smcginnis | Just minor things, then I think we should get those through right away. | 14:09 |
| openstackgerrit | Jay Bryant proposed openstack/cinder master: Remove the ITRI DISCO driver https://review.openstack.org/607058 | 14:11 |
| openstackgerrit | Jay Bryant proposed openstack/cinder master: Remove the HGST Flash Storage Driver https://review.openstack.org/607062 | 14:14 |
| jungleboyj | Cool. Thanks. | 14:14 |
| *** Bhujay has quit IRC | 14:16 | |
| lemko | thanks for your answer smcginnis | 14:18 |
| lemko | Here's the log : http://paste.openstack.org/show/731376/ | 14:18 |
| smcginnis | lemko: That doesn't appear to be a detach. Looks like it's creating a volume from image, and the image conversion and writing process is taking some time to complete. | 14:19 |
| lemko | Image download and conversion have a normal speed. But it took 11 minutes to do the rest. the log is about creating a volume from cirrus. | 14:20 |
| lemko | and with detach it's basically the same | 14:20 |
| lemko | it seems to take ages to detach the volume | 14:20 |
| lemko | cirros* :) | 14:21 |
| smcginnis | lemko: It would be interesting to take a look at what is happening with the detach, but for the create from image, after it converts the image it needs to actually write that data out to the new volume. That is somewhat dependent on your data path speed. | 14:21 |
| smcginnis | One thing that can be done to help mitigate this is to use image caching. | 14:21 |
| *** hoangcx has joined #openstack-cinder | 14:22 | |
| smcginnis | lemko: Might want to try setting this up - https://docs.openstack.org/cinder/latest/admin/blockstorage-image-volume-cache.html | 14:22 |
| lemko | I'm a but surprised that writing 11 MB image to a 1 or 2 GB volume would take that much time | 14:22 |
| lemko | btw I'm using openstack-helm that I tunnel to support iscsi storage. | 14:22 |
| smcginnis | Me too, that's a really long time for such a small amount of data. | 14:22 |
| smcginnis | Not sure I understand that last statement. | 14:22 |
| lemko | My openstack is containerized. But I think the container layer doesn't matter with iscsi | 14:23 |
| smcginnis | Depending how the networking is set up, it could cause that iSCSI traffic to have very poor performance. | 14:24 |
| smcginnis | But all of that should not affect detach, only that initial volume create. | 14:24 |
| lemko | I've done some tests like unmaping the volume from host with my storage interface after a sort time (1 min) and then checking the volume manually. | 14:24 |
| lemko | the volume is fine | 14:24 |
| lemko | So to me the volume has the good data | 14:25 |
| lemko | and this from the begining | 14:25 |
| lemko | the rest of the time is cinder-volume doing nothing | 14:25 |
| lemko | and finally unmapping it | 14:25 |
| lemko | Also I've tried mouting iscsi volumes in my container, and it's fine. | 14:26 |
| lemko | So I'm really wondering what I'm missing | 14:26 |
| smcginnis | Detach has nothing to do with your data on the volume. Other than whatever nova needs to do to prepare for the device to go away. | 14:26 |
| smcginnis | And mount/unmount usually isn't a very long operation. Are you seeing delays there too? | 14:26 |
| openstackgerrit | Jay Rubenstein proposed openstack/cinder master: SF: Handle qos values on extend volume https://review.openstack.org/606959 | 14:27 |
| *** dpawlik has quit IRC | 14:27 | |
| smcginnis | lemko: Anyway, I would first set up image caching (linked to above) and see if that improves things for you. Then if the initial create is faster, we can see what other delays are left. | 14:29 |
| lemko | Attachment of volume is fast | 14:31 |
| lemko | and my problem is typically the same when I attach a volume to an instance and then detach it. | 14:32 |
| lemko | I'll try to find some log | 14:32 |
| *** woojay has joined #openstack-cinder | 14:33 | |
| openstackgerrit | Miriam Yumi proposed openstack/cinder stable/rocky: NetApp SolidFire: Fix force_detach https://review.openstack.org/607595 | 14:35 |
| *** raunak12 has joined #openstack-cinder | 14:40 | |
| lemko | smcginnis http://paste.openstack.org/show/731378/ and then it took ten more minutes to have it detached | 14:45 |
| lemko | http://paste.openstack.org/show/731381/ | 14:47 |
| smcginnis | Weird it's double logging everything. | 14:48 |
| smcginnis | But that message is that it's starting the detach. Nova then needs to do several steps after that to remove the volume, flush IO, and tear down the local device path. | 14:49 |
| smcginnis | So some or all of those steps are taking awhile to complete. | 14:49 |
| lemko | Could it be linked to multipath.conf? | 14:50 |
| *** KeithMnemonic has quit IRC | 14:51 | |
| lemko | or iscsi package version? | 14:51 |
| smcginnis | lemko: Hmm, potentially. Multipath handling can cause some more work to happen. | 14:52 |
| smcginnis | lemko: What version of the services are you running? | 14:52 |
| lemko | open-iscsi and multipath? | 14:52 |
| lemko | or you mean openstack services? | 14:52 |
| smcginnis | Sorry, OpenStack. | 14:52 |
| lemko | Queens | 14:52 |
| smcginnis | I know there have been various improvements for multipath, device scanning, and other optimizations in the os-brick library. But I think most of those came in rocky. | 14:55 |
| lemko | because open-iscsi package for ubuntu is kinda old | 14:55 |
| smcginnis | lemko: You could look in syslog to see if there are any device removal or iscsi messages during that period after the nova message that it's detaching the volume and when you see things actually call into Cinder to do the detach. | 14:55 |
| smcginnis | lemko: Yeah, seeing if you can upgrade open-iscsi *shouldn't* hurt and might bring some improvements. | 14:56 |
| *** Emine has quit IRC | 14:58 | |
| *** eharney has quit IRC | 15:09 | |
| *** raunak12 has quit IRC | 15:10 | |
| *** dpawlik has joined #openstack-cinder | 15:19 | |
| openstackgerrit | Helen Walsh proposed openstack/cinder master: VMAX docs - Restructure of content https://review.openstack.org/607615 | 15:22 |
| openstackgerrit | Helen Walsh proposed openstack/cinder master: VMAX docs - Replace serial_number https://review.openstack.org/607615 | 15:24 |
| *** dpawlik has quit IRC | 15:24 | |
| *** eharney has joined #openstack-cinder | 15:24 | |
| *** pcaruana has quit IRC | 15:33 | |
| *** Luzi has joined #openstack-cinder | 15:47 | |
| smcginnis | imacdonn: I saw you pinged me last night. Need something yet? | 15:48 |
| *** raunak12 has joined #openstack-cinder | 15:48 | |
| openstackgerrit | Merged openstack/os-brick stable/rocky: Succeed on iSCSI detach when path just went down https://review.openstack.org/607041 | 15:50 |
| openstackgerrit | Merged openstack/python-cinderclient master: refactor the getid method base.py https://review.openstack.org/589692 | 15:51 |
| *** penick has joined #openstack-cinder | 15:56 | |
| *** _erlon_ has joined #openstack-cinder | 15:59 | |
| *** tpsilva has joined #openstack-cinder | 16:08 | |
| openstackgerrit | Sean McGinnis proposed openstack/cinder master: Update docs landing page to follow guideline https://review.openstack.org/607627 | 16:12 |
| *** hoangcx has quit IRC | 16:12 | |
| *** hoangcx has joined #openstack-cinder | 16:13 | |
| smcginnis | tbarron: Any interest/availability to participate in something like this if it comes together? https://etherpad.openstack.org/p/cinder-stein-mid-cycle-planning | 16:20 |
| smcginnis | tbarron: Mostly just making sure you are aware of it. Maybe a joint cinder/manila event? | 16:20 |
| tbarron | smcginnis: I'd likely participate and maybe some of the other local manila folks too. We have enough remotees in manila that we'll probabbly go on and do a virtual mid-cycle as well. | 16:22 |
| smcginnis | tbarron: Makes sense. | 16:22 |
| tbarron | smcginnis: you'll get eric to go if you have it in colorado or next to a pool hall | 16:24 |
| smcginnis | :) | 16:27 |
| openstackgerrit | Elise Gafford proposed openstack/os-brick stable/queens: Succeed on iSCSI detach when path just went down https://review.openstack.org/607632 | 16:28 |
| *** dims has quit IRC | 16:28 | |
| openstackgerrit | Merged openstack/python-brick-cinderclient-ext master: Removed older version of python added 3.5 https://review.openstack.org/606391 | 16:32 |
| *** dims_ has joined #openstack-cinder | 16:35 | |
| *** e0ne has quit IRC | 16:47 | |
| *** asbishop has joined #openstack-cinder | 16:55 | |
| *** abishop_ has quit IRC | 16:58 | |
| *** Luzi has quit IRC | 17:02 | |
| imacdonn | smcginnis: re. ping last night ... I've been looking into online data migrations ... I got past the immediate hurdle, but there are some issues here ... looks like cinder uses the same mechanism as nova, and has a couple of the same bugs | 17:02 |
| imacdonn | is there a designated database guru in the cinder team? | 17:04 |
| smcginnis | Hmm, not sure really. | 17:04 |
| imacdonn | k, well, I guess I'll just open bugs then ... one of the issues is a bit tricky, and it seems that nova hasn't quite figured out how to address it yet | 17:06 |
| imacdonn | issue is that, when a migration fails for some reason, the exception gets swallowed, a fairly useless warning is logged, and it moves one (user may or not even be aware) - https://review.openstack.org/#/c/600085/ | 17:08 |
| smcginnis | That's not good. | 17:11 |
| imacdonn | mriedem: any update on that from the nova side? | 17:14 |
| jungleboyj | smcginnis: I was just wondering about a DB guru too given the issues we were talking about during the meeting. | 17:17 |
| jungleboyj | imacdonn: You should sit in on our meetings if you can. | 17:17 |
| imacdonn | jungleboyj: I was in "fly on the wall mode" for the last couple ;) | 17:17 |
| jungleboyj | imacdonn: That is good. | 17:31 |
| openstackgerrit | Rajat Dhasmana proposed openstack/cinder master: API-REF:os-quota-sets v2 API reference has the wrong parameters https://review.openstack.org/607533 | 17:36 |
| *** mvkr has quit IRC | 17:39 | |
| openstackgerrit | Merged openstack/python-cinderclient master: Fix encoding of query parameters https://review.openstack.org/602486 | 17:54 |
| openstackgerrit | Merged openstack/os-brick stable/rocky: Ignore volume disconnect if it is not connected https://review.openstack.org/604578 | 17:55 |
| *** asbishop has quit IRC | 17:56 | |
| openstackgerrit | Erlon R. Cruz proposed openstack/cinder stable/queens: NetApp SolidFire: Fix CG snapshot deletion https://review.openstack.org/607662 | 17:58 |
| *** asbishop has joined #openstack-cinder | 17:58 | |
| *** e0ne has joined #openstack-cinder | 18:02 | |
| *** raunak12_ has joined #openstack-cinder | 18:05 | |
| *** raunak12 has quit IRC | 18:07 | |
| *** raunak12_ is now known as raunak12 | 18:07 | |
| openstackgerrit | Erlon R. Cruz proposed openstack/cinder stable/pike: NetApp SolidFire: Fix CG snapshot deletion https://review.openstack.org/607665 | 18:08 |
| mriedem | imacdonn: on my patch? no, i haven't updated it yet. | 18:18 |
| imacdonn | mriedem: On the approach, I guess ... I hacked a change into cinder's version to hopefully log the exception (but still move on) ... so at least I have a chance of being able to diagnose the failure | 18:20 |
| imacdonn | mriedem: but that's maybe only half of the problem ... other half is how to catch the situation that "something went a bit wrong" | 18:20 |
| *** imacdonn has quit IRC | 18:21 | |
| *** imacdonn has joined #openstack-cinder | 18:21 | |
| imacdonn | guh, http proxy ditched my connection ... did I miss anything ? | 18:22 |
| jungleboyj | imacdonn: Every cell phone in the US just simultaneously blew up with a Presidential Text. | 18:24 |
| *** pcaruana has joined #openstack-cinder | 18:24 | |
| *** itlinux has joined #openstack-cinder | 18:24 | |
| jungleboyj | #TimeToSolveRealProblems | 18:24 |
| imacdonn | jungleboyj: hah, yeah, I knew that was happening .. but I don't think it's related .. this stupid proxy dumps me a couple of times every day ... suspect they have some sort of max session lifetime, to try to manage load | 18:25 |
| jungleboyj | :-) I didn't think it was related. You asked what you missed though. ;-) | 18:25 |
| *** mvkr has joined #openstack-cinder | 18:34 | |
| *** e0ne has quit IRC | 18:45 | |
| *** e0ne has joined #openstack-cinder | 18:48 | |
| *** pcaruana has quit IRC | 18:50 | |
| *** lemko has quit IRC | 18:58 | |
| *** amoralej is now known as amoralej|off | 19:03 | |
| *** abishop_ has joined #openstack-cinder | 19:08 | |
| *** e0ne has quit IRC | 19:10 | |
| *** asbishop has quit IRC | 19:10 | |
| openstackgerrit | Merged openstack/cinder master: Add missing 'is_public' volume type parameter https://review.openstack.org/607115 | 19:20 |
| openstackgerrit | Merged openstack/cinder-tempest-plugin master: Fix consistency groups test credentials https://review.openstack.org/599720 | 19:20 |
| *** spartakos has joined #openstack-cinder | 19:24 | |
| imacdonn | question for anyone, in lieu of a DB guru :P .... is it expected/required for cinder services to be running when online_data_migrations are applied? That does not seem to be the case for nova, but I'm attempting to do it on cinder, and getting a MessagingTimeout (as if it's trying to talk to something) | 19:36 |
| smcginnis | I thought so. Hence the name "online_". | 19:37 |
| imacdonn | well, I was lead to interpret that as "these *CAN* be done online", not that they HAVE TO be | 19:37 |
| imacdonn | (for nova, at least) | 19:38 |
| imacdonn | this is where it seems to need services online: https://github.com/openstack/cinder/blob/master/cinder/cmd/manage.py#L103-L109 | 19:50 |
| imacdonn | now I wonder if it needs the cinder-volume services to be online .. it's going to be make life interesting, if so | 19:51 |
| smcginnis | Ah, right. There was this because of that - https://review.openstack.org/#/c/601684/ | 19:52 |
| imacdonn | aha! | 19:52 |
| imacdonn | so .. that attempts to prevent the problem in the future, but is there a fix for the one that already snuck in? | 19:54 |
| imacdonn | that is the one I tripped on, FWIW: ERROR cinder.cmd.manage [req-e335c38e-83ca-4e09-9318-4f9c9a7f93b9 - - - - -] Error attempting to run shared_targets_online_data_migration: MessagingTimeout: Timed out waiting for a reply to message ID fb8a5301e73d499bb9c0e706107dffbb | 19:56 |
| *** luizbag has quit IRC | 20:01 | |
| openstackgerrit | Sean McGinnis proposed openstack/cinder master: Update unit test debug instructions https://review.openstack.org/607700 | 20:07 |
| smcginnis | imacdonn: No, no fix that I am aware of. | 20:07 |
| smcginnis | imacdonn: But you're welcome to try to tackle that. ;) | 20:07 |
| imacdonn | yeah. hmmm. I either tackle it, or tackle restructuring my deployment/upgrade process :/ | 20:08 |
| smcginnis | Sounds like a win-win. | 20:09 |
| smcginnis | :) | 20:09 |
| imacdonn | hah | 20:09 |
| imacdonn | first I'd have to understand what the migration is trying to accomplish ... seems like an "after lunch" sort of thing | 20:09 |
| smcginnis | Or after morning coffee. | 20:10 |
| imacdonn | that may have a better chance of success | 20:10 |
| mriedem | i have a semi interesting question.... | 20:15 |
| imacdonn | uh oh | 20:15 |
| mriedem | let's say i have a volume attached to a server with an attachment record created for that volume, | 20:15 |
| jungleboyj | Go on. | 20:15 |
| mriedem | then i logically detach the volume from the server and replace it with a new volume, but the old attachment is still there, | 20:15 |
| mriedem | then when i update the attachment record with the host connector, that attachment record is for the old volume, not the new one | 20:16 |
| mriedem | that would be....bad and stuff yeah? | 20:16 |
| jungleboyj | What do you mean 'Logically detach' ? | 20:16 |
| *** spartakos has quit IRC | 20:17 | |
| mriedem | sec | 20:18 |
| mriedem | https://review.openstack.org/#/c/600628/5/specs/stein/approved/detach-boot-volume.rst@103 | 20:21 |
| mriedem | this is the spec to allow swapping root volume on a server while it's shelved offloaded | 20:21 |
| mriedem | when we shelve offload an instance, we create an empty attachment record for the volume(s) attached to the server so they remain 'reserved' while the instance is shelved, | 20:22 |
| mriedem | and then on unshelve, we spawn the guest on a new compute host and update all of those attachments with new host connectors | 20:22 |
| mriedem | in this spec, they propose being able to detach the root volume on the instance and then attach a new root volume - but we have to do something with those attachment records | 20:22 |
| smcginnis | In that case, there's no need to reserve the volume since you probably should just delete the original boot volume. Then when the new boot volume is created, just attach like normal. | 20:23 |
| mriedem | right i'm thinking if you detach the reserved root volume while the instance is shelved, we delete the attachment record for that volume as well, | 20:24 |
| mriedem | then if you attach a new root volume to the shelved instance, we create a new empty attachment to reserve the volume before you unshelve it | 20:24 |
| jungleboyj | mriedem: I am thinking the same thing. | 20:24 |
| smcginnis | mriedem: Yeah, that sounds logical. | 20:24 |
| jungleboyj | Yeah. That makes sense. | 20:24 |
| mriedem | thanks spock :) | 20:25 |
| smcginnis | mriedem: Live long and prosper. | 20:26 |
| jungleboyj | https://gph.is/1DzWYFR | 20:26 |
| jungleboyj | smcginnis: jinx | 20:26 |
| smcginnis | \/ | 20:26 |
| jungleboyj | This one is good too: https://gph.is/1DzWYFR | 20:27 |
| jungleboyj | Hmm, the other unsupported driver right now is the datacore and it looks like they have been running. | 20:29 |
| jungleboyj | Their FC job hasn't been successful still in like 225 days. Maybe need to e-mail them again. | 20:30 |
| smcginnis | Maybe remove the one and not the other? | 20:30 |
| jungleboyj | I suppose. | 20:30 |
| jungleboyj | Let me ping them first and see if there is a reason for it. | 20:31 |
| jungleboyj | Huawei hasn't run in 113 days. | 20:42 |
| jungleboyj | And looks like there might be something wrong with the Storewize jobs. | 20:43 |
| smcginnis | Unsupported! | 20:46 |
| jungleboyj | Sent an note to Isaac to see what he says. What is up with Huawei though? Should I mark that unsupported? | 20:47 |
| smcginnis | I've pinged the internal team a few times now. Might be about time to do that. They are definitely past our policy on CI. | 20:47 |
| mriedem | huawei is best way | 20:48 |
| *** _erlon_ has quit IRC | 20:48 | |
| smcginnis | The best ever. We're hooje. | 20:48 |
| jungleboyj | Ok. Will push up that patch and see what happens. | 20:49 |
| * jungleboyj giggles | 20:50 | |
| jungleboyj | jgriffith: No Solidfire CI in 95 days. | 20:50 |
| smcginnis | He don't care | 20:50 |
| jungleboyj | Oh, that is right. He is at RedHat now. | 20:51 |
| smcginnis | Hmm, how many days ago did jgriffith leave NetApp? 95? | 20:51 |
| jungleboyj | erlon: You have any visibility into that at NetApp for Solidfire? | 20:51 |
| smcginnis | Oh, there's a different CI running it now. I remember being confused until I noticed that. | 20:51 |
| jungleboyj | That sounds suspiciously right. | 20:51 |
| smcginnis | NetApp Solidfire CI or something. | 20:51 |
| gouthamr | I think they forgot to update the wiki :P | 20:51 |
| gouthamr | i reset the wiki to jgriffith's CI at some point? | 20:52 |
| smcginnis | jungleboyj: I've updated my report to include NetApp SolidFire CI | 20:53 |
| jungleboyj | smcginnis: Ok. So that is was the old data? | 20:53 |
| smcginnis | If it still just said "Solidfire CI" | 20:53 |
| erlon | jungleboyj, we actually have changed the CI name | 20:53 |
| jungleboyj | smcginnis: Yeah. It did. There are a few others in here that I think need to be removed. | 20:53 |
| smcginnis | Probably. I haven't touched it for awhile. | 20:54 |
| erlon | jungleboyj, solidfire CIN has been reporting consistently in the last 1/2 months | 20:54 |
| * jungleboyj sighs Both our FCZM CIs are in a bad state. | 20:54 | |
| jungleboyj | erlon: Ok. Sounds like I just need sean to update the report. | 20:54 |
| *** spartakos has joined #openstack-cinder | 20:54 | |
| *** eharney has quit IRC | 20:57 | |
| openstackgerrit | Merged openstack/cinder stable/queens: nimble storage: retype support https://review.openstack.org/607331 | 20:57 |
| openstackgerrit | Merged openstack/cinder master: VMAX docs - Replace serial_number https://review.openstack.org/607615 | 20:57 |
| jungleboyj | smcginnis: IBM GPFS CI, Infortrend CI, SolidFire CI | 20:58 |
| jungleboyj | Those looke like they should be removed from the report. | 20:58 |
| smcginnis | Cool, gone. | 20:58 |
| *** asbishop has joined #openstack-cinder | 20:58 | |
| jungleboyj | Going to send nasty grams to Cisco and Brocade again and then propose a few 'Unsupported' patches. | 20:58 |
| smcginnis | We should probably talk about what we want to do with FCZM in the event we no longer have any drivers for it. | 20:59 |
| * jungleboyj shudders | 21:00 | |
| jungleboyj | It is a notable selling point with a number of users I have talked to. | 21:00 |
| smcginnis | I love removing code, but that would suck to lose that. | 21:00 |
| jungleboyj | Wonder what a FC Switch costs on ebay? | 21:00 |
| jungleboyj | I have a server now. Just need a couple of storage backends and switches. | 21:01 |
| *** abishop_ has quit IRC | 21:01 | |
| * jungleboyj probably doesn't make enough to pay the electric bill. | 21:01 | |
| smcginnis | When I went through with Angela, there were only like 12 tests that actually needed to be run to test FC connectivity. Theoretically, it should be easy to set something up if you have a switch, HBAs, and cables. | 21:01 |
| jungleboyj | https://www.ebay.com/itm/Brocade-Silkworm-3850-16-Port-Fibre-Channel-Switch-works/253766054512?hash=item3b15a2b270:g:wBcAAOSwf0hbUlHp:sc:FedExHomeDelivery!55904!US!-1 | 21:03 |
| jungleboyj | Don't need storage behind it? | 21:03 |
| jungleboyj | https://www.ebay.com/itm/Cisco-Fibre-Channel-Multilayer-Fabric-Switch-with-24-SFPs/283092392212?hash=item41e99f1914:g:mg8AAOSwtbBbamAr | 21:04 |
| jungleboyj | Would have to put HBAs in the x3650 though. | 21:05 |
| * jungleboyj can't believe I am googling this. | 21:05 | |
| *** erlon has quit IRC | 21:06 | |
| jungleboyj | Hmm, the HBAs are pretty cheap. | 21:07 |
| * jungleboyj should be e-mailing Cisco and Brocade instead. | 21:07 | |
| *** spartakos has quit IRC | 21:08 | |
| jungleboyj | The topic is on the MidCycle agenda | 21:08 |
| openstackgerrit | Miriam Yumi proposed openstack/cinder stable/pike: NetApp SolidFire: Fix force_detach https://review.openstack.org/599821 | 21:13 |
| imacdonn | smcginnis jungleboyj: (now that it's after lunch) ... it seems like the implementation doesn't match the spec here: https://specs.openstack.org/openstack/cinder-specs/specs/queens/add-shared-targets-to-volume-ref.html | 21:25 |
| *** spartakos has joined #openstack-cinder | 21:26 | |
| jungleboyj | imacdonn: That wouldn't surprise me. We have that problem in a number of places. | 21:26 |
| smcginnis | Hmm, could be changes were made over the course of reviewing the implementation that didn't get put back into the spec once it was finalized. | 21:26 |
| imacdonn | either way, it seems bad | 21:27 |
| imacdonn | if the spec was implemented, we wouldn't need the RPC dependency in the migration | 21:27 |
| imacdonn | is jgriffith around these days ? | 21:32 |
| *** asbishop has quit IRC | 21:34 | |
| *** rosmaita has left #openstack-cinder | 21:35 | |
| jungleboyj | imacdonn: If you say his name 3 times he will show up. ;-) | 21:36 |
| * imacdonn is almost gullible enough | 21:37 | |
| jungleboyj | imacdonn: Only almost gullible? :-( | 21:42 |
| smcginnis | jgriffith, jgriffith, jgriffith | 21:43 |
| * imacdonn watches intently | 21:43 | |
| * jungleboyj holds my breath | 21:44 | |
| smcginnis | Lettin' me down man. | 21:44 |
| jgriffith | smcginnis: :) | 21:47 |
| imacdonn | ooooh | 21:47 |
| smcginnis | \o/ | 21:47 |
| jgriffith | I was composing myself :) | 21:47 |
| jungleboyj | @!j | 21:47 |
| smcginnis | :) | 21:48 |
| jungleboyj | (._.) ~ ︵ ┻━┻ | 21:48 |
| jungleboyj | imacdonn: See. You doubted me. | 21:48 |
| jgriffith | imacdonn: what specifiacally are you having issues with | 21:48 |
| imacdonn | jungleboyj: :) | 21:49 |
| * jungleboyj makes note that pinging jgriffith on Twitter helps. | 21:49 | |
| jgriffith | jungleboyj: twitter? | 21:49 |
| jgriffith | Oh.. lookie there | 21:49 |
| imacdonn | jgriffith: I'm trying to do an offline upgrade, and the shared_targets_online_data_migration is failing because the cinder-volume service is not running | 21:49 |
| jgriffith | geesh | 21:49 |
| imacdonn | jgriffith: per https://review.openstack.org/#/c/601684/ , that's a naughty migration | 21:49 |
| imacdonn | jgriffith: also, it doesn't match the spec | 21:49 |
| * smcginnis pictures jgriffith running away | 21:50 | |
| jgriffith | imacdonn: yeah... sigh; let me pull up the review comment that got me a -1 until I modified it | 21:50 |
| jgriffith | Oh, no need it's in that link you just sent | 21:50 |
| jungleboyj | jgriffith: OMG, so it really did work. Ha ha ha! | 21:51 |
| jungleboyj | k979250h | 21:51 |
| jungleboyj | Ahhhh! Please ignore that. | 21:51 |
| jgriffith | I glance at IRC every once in a while thank you very much :) | 21:51 |
| jgriffith | looks like a password to me | 21:52 |
| jungleboyj | Nothing to see there. | 21:52 |
| * imacdonn wonders what NSFW site it's for | 21:52 | |
| jungleboyj | Hey now! | 21:53 |
| jgriffith | imacdonn: so anyway... I'm not sure what I should do here on this | 21:53 |
| imacdonn | jgriffith: I'm not either ... was it determined that having the volume service handle it on startup (per the spec) was not suitable after all ? | 21:54 |
| jgriffith | Well, reviewers didn't seem to like that approach | 21:55 |
| jgriffith | I *should* have updated the spec, but that's kinda the bogus nature of our specs IMHO | 21:55 |
| smcginnis | What do reviewers know. | 21:55 |
| *** dustins has quit IRC | 21:55 | |
| jgriffith | You can try and design everything under the sun, but when it comes time to actually review and merge, things tend to change | 21:56 |
| *** dustins has joined #openstack-cinder | 21:56 | |
| jgriffith | imacdonn: so to make sure I understand completely, the issue is that you want to be able to perform this updgrade/migration when the services are "down" | 21:56 |
| imacdonn | jgriffith: yes... especially since it's the cinder-volume service, which I kindof think of like an agent, not a control service | 21:57 |
| jgriffith | I believe we should be able to handle "both" ways, online DM requiring the services and some way to stash it until the service is available | 21:57 |
| jgriffith | imacdonn: yeah, the problem is you have the other camp that says "never take anything down to upgrade" "we must have live upgrades" | 21:58 |
| jgriffith | Satisfying both isn't trivial | 21:58 |
| jgriffith | IMO at least | 21:58 |
| imacdonn | jgriffith: I think the key message should be "your migrations should not REQUIRE services to be down" | 21:58 |
| jgriffith | I guess we could add another migration that handles this offline? | 21:58 |
| jgriffith | wait, now I'm confused | 21:59 |
| jgriffith | imacdonn: ahh.. you mean that's the key, not one or the other | 21:59 |
| jgriffith | sigh | 21:59 |
| imacdonn | right :) | 21:59 |
| jgriffith | do you have an example of such a unicorn :) | 22:00 |
| jgriffith | I might be able to work on an update if I have something to work off of | 22:01 |
| jgriffith | but given until this migration we didn't have a live-migration at all I got to be first and make the first mistakes ) | 22:01 |
| imacdonn | any migration that only looks at the database should work whether the services are up or not | 22:02 |
| openstackgerrit | Jay Bryant proposed openstack/cinder master: Mark Veritas HyperScale Driver Unsupported https://review.openstack.org/607728 | 22:03 |
| imacdonn | problem here, as you said (I think), is that the DB doesn't record the info we need for this one | 22:03 |
| imacdonn | as a side-note; doesn't cinder support multiple backends per host now? How would it handle one drive that uses shared targets and another that doesn't ? | 22:04 |
| imacdonn | driver* | 22:04 |
| openstackgerrit | Merged openstack/cinder-tempest-plugin master: fix tox python3 overrides https://review.openstack.org/605706 | 22:05 |
| jgriffith | imacdonn: it's driver based, not node based | 22:06 |
| imacdonn | oh, I guess "host" includes the backend name | 22:07 |
| jgriffith | affirm | 22:07 |
| imacdonn | but wait ... how does RPC handle that ? | 22:07 |
| jgriffith | host@driver | 22:07 |
| jgriffith | we have some funky biz in there | 22:08 |
| imacdonn | heh | 22:08 |
| jgriffith | That's super dooper old by the way (multi-backend) | 22:08 |
| imacdonn | another side-note ... this migration is in cmd/manage.py ... all the others, and all of the nova ones, are under db, not cmd | 22:08 |
| jgriffith | Becaues it's "mine" and it's "special" :) | 22:09 |
| imacdonn | mmmkay | 22:09 |
| *** itlinux has quit IRC | 22:09 | |
| jgriffith | seriously though, I'm unclear on the issue there | 22:10 |
| jgriffith | online migrations seemed like a "manage" thing so that's where I stuffed it | 22:10 |
| jgriffith | Nova has the same pattern FWIW | 22:11 |
| imacdonn | I guess "cmd" is the user-interface (CLI) part | 22:11 |
| imacdonn | the nova migrations are under db | 22:11 |
| jgriffith | https://github.com/openstack/nova/blob/master/nova/cmd/manage.py#L387 | 22:11 |
| imacdonn | oh, I mean, the actual code ... the list is there, yeah | 22:12 |
| jgriffith | Ummmm so is the one in Cinder | 22:12 |
| jgriffith | https://github.com/openstack/cinder/blob/master/cinder/db/sqlalchemy/migrate_repo/versions/115_add_shared_targets_to_volumes.py | 22:13 |
| imacdonn | that's the schema migration | 22:13 |
| imacdonn | I'm talking about the online data migrations | 22:13 |
| jgriffith | imacdonn: I get the impression that regardless of what I say you'll have an unsatisfied response :) | 22:14 |
| imacdonn | Not trying to be argumentative ... just sharing my observations .... there does seem to be a lot of confusion around this whole online migrations topic, in nova too | 22:15 |
| jgriffith | imacdonn: yeah... objects, rpc-versions, online-migrations and now placement | 22:15 |
| jgriffith | it's sort of how new things grow | 22:15 |
| jgriffith | So I'm certainly not claiming (nor did I ever claim) to be an expert on online-migrations | 22:16 |
| smcginnis | I we did realize placement wouldn't work for our needs, so at least that's one less. For now. | 22:16 |
| jgriffith | smcginnis: for now :) | 22:16 |
| jgriffith | It'll be back next year :) | 22:16 |
| *** spartakos has quit IRC | 22:17 | |
| jungleboyj | jgriffith: I don't know. I think it is gone. At least for a while. | 22:17 |
| jgriffith | imacdonn: I see what you're saying WRT to the impl though | 22:20 |
| imacdonn | jgriffith: OK, so long as I'm right ... :-) | 22:20 |
| *** Zer0Byte_ has joined #openstack-cinder | 22:20 | |
| Zer0Byte_ | hi | 22:20 |
| Zer0Byte_ | i have a issue | 22:20 |
| Zer0Byte_ | only when i migrate | 22:20 |
| jgriffith | Zer0Byte_: me too, but it's all the time | 22:21 |
| Zer0Byte_ | tryting to migrate a volume from one backend storage to another | 22:21 |
| imacdonn | kinda strikes me that, if the volume service could handle this on startup, we wouldn't need a migration at all, since the column has a default value | 22:21 |
| jungleboyj | Zer0Byte_: Ok. | 22:21 |
| Zer0Byte_ | BadRequest: Expecting to find domain in project. The server could not comply with the request since it is either malformed or otherwise incorrect. The client is assumed to be in error. (HTTP 400) (Request-ID: req-ed02c1c3-8905-4c4f-bee1-4789596bf6a5) | 22:21 |
| * imacdonn has 99 issues | 22:21 | |
| Zer0Byte_ | i check the openrc file | 22:21 |
| jgriffith | imacdonn: TBF there are nova migrations implemented in cmd/manage.py as well.. but anyway | 22:21 |
| Zer0Byte_ | and is everything right i can do any another command on cinder | 22:22 |
| Zer0Byte_ | except migrate even from horizon fail | 22:22 |
| jgriffith | or at least not implemented under db.xxxx; but regardless. | 22:22 |
| jgriffith | I guess I should try to figure out a way to fix this | 22:22 |
| Zer0Byte_ | what do you mean with db.xxx? | 22:23 |
| imacdonn | jgriffith: OK .. that was really just a side-note ... but yeah, I think we need to figure something out about the RPC/service dependency | 22:23 |
| jgriffith | imacdonn: kind of a broader topic in general I think; might be worth raising at next weeks meeting, and perhaps even follow up and coordinate via an ML posting. | 22:24 |
| jgriffith | see if everybody is on the same page or not | 22:24 |
| Zer0Byte_ | https://pastebin.com/YHWd7v8a | 22:25 |
| Zer0Byte_ | this is the full stacktrace | 22:25 |
| *** rcernin has joined #openstack-cinder | 22:25 | |
| imacdonn | jgriffith: OK, yeah. Maybe jungleboyj can help facilitate the discussion | 22:26 |
| *** ganso has quit IRC | 22:27 | |
| jungleboyj | imacdonn: Sure. Can you add it to the meeting info? | 22:27 |
| imacdonn | jungleboyj: ok, will do | 22:28 |
| Zer0Byte_ | i guess i have a idea | 22:29 |
| Zer0Byte_ | if you see the client | 22:30 |
| Zer0Byte_ | sorry teh stacktrace you see is using novaclientv2 | 22:30 |
| jungleboyj | Something weird in your environment there. | 22:30 |
| Zer0Byte_ | i guess novaclient don't use domain_name right | 22:30 |
| Zer0Byte_ | ? | 22:30 |
| Zer0Byte_ | why? | 22:30 |
| jungleboyj | It more appears to be something wrong with communicating with keystone given where it is failing. | 22:31 |
| imacdonn | Zer0Byte_: do you have a keystone_authtoken section in your cinder.conf ? Does it have project_domain_name and user_domain_name in it ? | 22:34 |
| Zer0Byte_ | yeah have the default that juju set | 22:35 |
| Zer0Byte_ | all of them have setted up service domain | 22:35 |
| Zer0Byte_ | project_domain_name = service_domain | 22:35 |
| Zer0Byte_ | user_domain_name = service_domain | 22:35 |
| *** dustins has quit IRC | 22:36 | |
| openstackgerrit | Merged openstack/cinder master: Remove cinder-tox-compliance job https://review.openstack.org/593192 | 22:39 |
| Zer0Byte_ | keystone show this error only | 22:40 |
| Zer0Byte_ | (keystone.common.wsgi): 2018-10-03 22:39:41,268 WARNING Expecting to find domain in project. The server could not comply with the request since it is either malformed or otherwise incorrect. The client is assumed to be in error. | 22:40 |
| Zer0Byte_ | if the | 22:42 |
| Zer0Byte_ | volume is attached to nova instance send the error | 22:42 |
| Zer0Byte_ | volume unattached | 22:43 |
| Zer0Byte_ | works great | 22:43 |
| Zer0Byte_ | looks like something in the nova client | 22:51 |
| Zer0Byte_ | of cinder | 22:51 |
| *** spartakos has joined #openstack-cinder | 23:01 | |
| Zer0Byte_ | well guys | 23:03 |
| Zer0Byte_ | question | 23:03 |
| Zer0Byte_ | is that | 23:03 |
| Zer0Byte_ | when im trying touse nova api with keystone v2 doesn;t work throwing the same error | 23:04 |
| Zer0Byte_ | now the question is | 23:04 |
| Zer0Byte_ | exist a way if i have a domain to use keystone v2 | 23:04 |
| *** Zer0Byte_ has quit IRC | 23:37 | |
| *** itlinux has joined #openstack-cinder | 23:48 | |
| *** mchlumsky has quit IRC | 23:49 | |
| *** mriedem has quit IRC | 23:51 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!