| *** markvoelker has quit IRC | 00:02 | |
| *** whoami-rajat has joined #openstack-cinder | 00:27 | |
| *** lseki has quit IRC | 00:37 | |
| *** masayukig has joined #openstack-cinder | 00:46 | |
| *** whfnst has joined #openstack-cinder | 01:10 | |
| *** zhengMa has joined #openstack-cinder | 01:59 | |
| *** markvoelker has joined #openstack-cinder | 02:03 | |
| *** markvoelker has quit IRC | 02:08 | |
| zhengMa | rosmaita: smcginnis: We have updated the patch - Leverage hw accelerator in image compression #Link: https://review.opendev.org/668825. Expecting reviewing! Thanks a lot!:) | 02:09 |
|---|---|---|
| *** mmethot_ has quit IRC | 02:11 | |
| *** mmethot_ has joined #openstack-cinder | 02:12 | |
| *** swegener has quit IRC | 02:13 | |
| rosmaita | zhengMa: ty, will take a look | 02:15 |
| rosmaita | KeithMnemonic: geguileo is usually in this channel, he is in UTC+2 time zone | 02:17 |
| zhengMa | rosmaita: thanks!! | 02:19 |
| *** whoami-rajat has quit IRC | 02:29 | |
| *** rajinir has quit IRC | 02:50 | |
| *** rajinir has joined #openstack-cinder | 02:56 | |
| *** dave-mccowan has quit IRC | 03:00 | |
| *** baojg has quit IRC | 03:15 | |
| *** rajinir has quit IRC | 03:18 | |
| *** psachin has joined #openstack-cinder | 03:21 | |
| *** baojg has joined #openstack-cinder | 03:23 | |
| *** rajinir has joined #openstack-cinder | 03:24 | |
| *** Reepicheep has quit IRC | 03:35 | |
| *** Reepicheep has joined #openstack-cinder | 03:35 | |
| *** rmcallis has quit IRC | 03:59 | |
| *** baojg has quit IRC | 04:27 | |
| *** ociuhandu has joined #openstack-cinder | 04:31 | |
| *** ociuhandu has quit IRC | 04:35 | |
| *** baojg has joined #openstack-cinder | 04:54 | |
| *** udesale has joined #openstack-cinder | 05:01 | |
| *** pcaruana has joined #openstack-cinder | 05:16 | |
| *** Luzi has joined #openstack-cinder | 05:16 | |
| *** pcaruana has quit IRC | 05:29 | |
| *** boxiang has joined #openstack-cinder | 05:47 | |
| *** zhubx has quit IRC | 05:47 | |
| *** openstackgerrit has joined #openstack-cinder | 05:53 | |
| openstackgerrit | ZhengMa proposed openstack/cinder master: Leverage hw accelerator in image compression https://review.opendev.org/668825 | 05:53 |
| *** n-saito has quit IRC | 05:55 | |
| *** spsurya has joined #openstack-cinder | 05:55 | |
| *** sapd1_x has joined #openstack-cinder | 06:19 | |
| *** markvoelker has joined #openstack-cinder | 06:47 | |
| *** swegener has joined #openstack-cinder | 06:52 | |
| *** markvoelker has quit IRC | 06:52 | |
| *** trident has quit IRC | 07:22 | |
| *** tosky has joined #openstack-cinder | 07:27 | |
| *** sahid has joined #openstack-cinder | 07:29 | |
| *** brinzhang_ has joined #openstack-cinder | 07:29 | |
| *** trident has joined #openstack-cinder | 07:31 | |
| *** brinzhang has quit IRC | 07:32 | |
| *** brinzhang_ has quit IRC | 07:33 | |
| *** trident has quit IRC | 07:36 | |
| *** trident has joined #openstack-cinder | 07:46 | |
| *** rcernin has quit IRC | 07:47 | |
| *** brinzhang has joined #openstack-cinder | 07:49 | |
| *** tkajinam has quit IRC | 08:03 | |
| *** lpetrut has joined #openstack-cinder | 08:07 | |
| *** lpetrut has quit IRC | 08:08 | |
| *** e0ne has joined #openstack-cinder | 08:41 | |
| *** lpetrut has joined #openstack-cinder | 08:48 | |
| *** zhengMa has left #openstack-cinder | 08:50 | |
| *** sapd1_x has quit IRC | 08:53 | |
| *** pcaruana has joined #openstack-cinder | 09:19 | |
| *** uberjay has quit IRC | 09:28 | |
| *** uberjay has joined #openstack-cinder | 09:28 | |
| *** boxiang has quit IRC | 09:39 | |
| *** boxiang has joined #openstack-cinder | 09:40 | |
| *** ociuhandu has joined #openstack-cinder | 10:05 | |
| *** pcaruana has quit IRC | 10:07 | |
| *** Luzi has quit IRC | 10:13 | |
| *** sapd1_x has joined #openstack-cinder | 10:25 | |
| *** Luzi has joined #openstack-cinder | 10:29 | |
| *** udesale has quit IRC | 10:34 | |
| *** udesale has joined #openstack-cinder | 10:34 | |
| *** sahid has quit IRC | 10:36 | |
| *** Sneha has joined #openstack-cinder | 10:42 | |
| *** Sneha has quit IRC | 10:47 | |
| *** carloss has joined #openstack-cinder | 10:53 | |
| *** pcaruana has joined #openstack-cinder | 10:55 | |
| *** zhubx has joined #openstack-cinder | 10:58 | |
| *** boxiang has quit IRC | 11:00 | |
| *** boxiang has joined #openstack-cinder | 11:15 | |
| *** sapd1_x has quit IRC | 11:15 | |
| *** zhubx has quit IRC | 11:18 | |
| *** sahid has joined #openstack-cinder | 11:29 | |
| *** takamatsu has joined #openstack-cinder | 11:52 | |
| *** raghavendrat has joined #openstack-cinder | 11:54 | |
| *** markvoelker has joined #openstack-cinder | 12:11 | |
| *** enriquetaso has joined #openstack-cinder | 12:19 | |
| *** dave-mccowan has joined #openstack-cinder | 12:25 | |
| *** pcaruana has quit IRC | 12:27 | |
| *** openstackstatus has quit IRC | 12:28 | |
| *** openstack has joined #openstack-cinder | 12:33 | |
| *** ChanServ sets mode: +o openstack | 12:33 | |
| *** jmlowe has joined #openstack-cinder | 12:33 | |
| KeithMnemonic | geguileo are you around? | 12:57 |
| geguileo | KeithMnemonic: yes, about to go have lunch, but around :-) | 12:58 |
| *** pcaruana has joined #openstack-cinder | 12:59 | |
| geguileo | KeithMnemonic: I read you made progress on the Lenovo/os-brick issue | 12:59 |
| KeithMnemonic | the issue i am seeing with a Lenovo DS6200 is in this code, https://github.com/openstack/os-brick/blob/stable/pike/os_brick/initiator/connectors/iscsi.py#L799-L810 what happens is the migrate 6 instances in batch, the first 3 run ok, but then all of a sudden no disks are added to the dict for removal | 13:00 |
| geguileo | KeithMnemonic: where you able to check what was actually present on sysfs? | 13:01 |
| geguileo | KeithMnemonic: under /sys/class/scsi_host/.... | 13:02 |
| KeithMnemonic | so for exmaple, the last working one has " u'iqn.2002-09.com.lenovo:01.array.00c0ff3b2da0' ): (set([u'sdh']), set([ u'sdj',u'sdb', u'sdy' ] ) " | 13:02 |
| KeithMnemonic | i did not get back on the customer system yet, waiting on them | 13:02 |
| KeithMnemonic | and then after the last good one, i see | 13:02 |
| KeithMnemonic | " u'iqn.2002-09.com.lenovo:01.array.00c0ff3b2da0' ): (set([]), set([ u'sdj',u'sdb', u'sdy' ] ) " | 13:03 |
| KeithMnemonic | so instead of it picking sdj, sdb, or sdy , it is null | 13:03 |
| geguileo | KeithMnemonic: thatś because the LUN doesn't match | 13:04 |
| KeithMnemonic | ok so check sysfs to see the lun mapping? | 13:04 |
| KeithMnemonic | since these instances boot and run fine, other thna migration | 13:04 |
| KeithMnemonic | what would mess up the mapping | 13:05 |
| KeithMnemonic | and i did a similar test on a HPE 3par i have and it is fine | 13:05 |
| KeithMnemonic | i do not have a DS6200 | 13:05 |
| *** mriedem has joined #openstack-cinder | 13:05 | |
| geguileo | I can only think of the 3par backend changing the LUNs on its own | 13:06 |
| geguileo | after the first 3 LUNs have been removed | 13:06 |
| geguileo | but afaik that would break everything... | 13:06 |
| geguileo | so I doubt that's happening | 13:07 |
| KeithMnemonic | you mean the 6200? | 13:07 |
| KeithMnemonic | the HPE 3par , migrates 6 as expected | 13:08 |
| KeithMnemonic | the Lenovo DS6200 does not | 13:08 |
| KeithMnemonic | it only gets the first 3 | 13:08 |
| geguileo | sorry, I meant the 6200, not the 3par | 13:08 |
| KeithMnemonic | so i am also waiting for them to test doing these not in 1 batch but perhaps seperated by a 60 second wait or so | 13:09 |
| KeithMnemonic | to see if there is some concurrency | 13:09 |
| KeithMnemonic | i see something odd in the trace as well | 13:09 |
| geguileo | I would look into the state when the error happens | 13:09 |
| KeithMnemonic | so before the migrate, i see path_checker_state up 24, paths 24, | 13:10 |
| KeithMnemonic | after the first one | 13:10 |
| KeithMnemonic | down 4, up 20 paths 24 | 13:10 |
| KeithMnemonic | or maybe during the first instance to migrate | 13:10 |
| KeithMnemonic | so this seems ok | 13:10 |
| geguileo | doesn't sound right | 13:10 |
| geguileo | no path should be down if things are properly configured | 13:11 |
| KeithMnemonic | and it continues this 4x multiple until down 4 up 12, paths 16 | 13:11 |
| geguileo | it means that multipathing is probably not configured in OpenStack | 13:11 |
| KeithMnemonic | and this conicides with instance 3 migrationg correctly | 13:11 |
| geguileo | but if they are migrated correctly, why are the paths appearing as down? | 13:11 |
| KeithMnemonic | mpath is configured i see it in the config file and the virsh does a dm-X device | 13:11 |
| KeithMnemonic | that is before they get removed | 13:12 |
| geguileo | then os-brick would clean the devices correctly | 13:12 |
| geguileo | first flushing the multipath and then removing the devices | 13:12 |
| KeithMnemonic | i see the multipath -f | 13:12 |
| KeithMnemonic | in the trace | 13:12 |
| *** raghavendrat has left #openstack-cinder | 13:12 | |
| geguileo | is there a trace when doing the multipath -f? | 13:13 |
| KeithMnemonic | but than are that down 4 up 12, paths 16 , i see down 5 up 11, paths 12 | 13:13 |
| KeithMnemonic | so now it is not even adding up | 13:13 |
| KeithMnemonic | 5+11 != 12 | 13:13 |
| geguileo | paths should all be up | 13:13 |
| geguileo | if they are properly removed | 13:13 |
| geguileo | if you have the logs you should confirm that the first devices that where removed are the ones that should have been removed | 13:14 |
| geguileo | you check when the volume attaches the LUN and the individual devices that form the volume | 13:15 |
| geguileo | then when the detach is requested you confirm that os-brick is disconnecting the right device for the right LUN | 13:15 |
| geguileo | it should be OK, but just to make sure that os-brick and Nova are not doing anything crazy | 13:16 |
| geguileo | KeithMnemonic: I'm starving, going to have lunch, will be back in 30-40 minutes | 13:18 |
| KeithMnemonic | ok np | 13:18 |
| KeithMnemonic | ping you after you come back | 13:18 |
| *** eharney has joined #openstack-cinder | 13:24 | |
| *** pcaruana has quit IRC | 13:28 | |
| *** Luzi has quit IRC | 13:38 | |
| *** lseki has joined #openstack-cinder | 13:41 | |
| *** psachin has quit IRC | 13:43 | |
| geguileo | KeithMnemonic: I'm back now | 13:57 |
| KeithMnemonic | ok so here is what I did, i looked at virsh and got the dm-X for each of the 6 instances, then looked at multipath -ll to line up the disks | 13:58 |
| KeithMnemonic | and i can see if follows in the correct order | 13:58 |
| KeithMnemonic | until it goes astray | 13:58 |
| geguileo | It should also be possible to follow it in the logs | 13:59 |
| KeithMnemonic | i did | 13:59 |
| geguileo | And you can also look info sysfs | 13:59 |
| *** pcaruana has joined #openstack-cinder | 13:59 | |
| KeithMnemonic | and as i said for the first 3 all in working as expected | 14:00 |
| geguileo | but yeah, with that one should be able to see what happened when the detach failed | 14:00 |
| geguileo | see the LUN that was trying to detach and confirm what's in the stem | 14:00 |
| KeithMnemonic | it never even does the multipath -f for the failed ones | 14:00 |
| KeithMnemonic | so should _get_connection_devices return the device it need to disconnect/detach | 14:02 |
| KeithMnemonic | where does _get_connection_devices occur in the workflow? | 14:03 |
| *** whoami-rajat has joined #openstack-cinder | 14:12 | |
| *** kukacz has joined #openstack-cinder | 14:14 | |
| kukacz | hi, anybody familiar with solidfire driver around? I'm troubleshooting disabled image caching after upgrading cinder from Ocata to Queens | 14:17 |
| kukacz | seeing the sf_allow_template_caching parameter is now disabled by default, I wonder how to achieve similar behaviour of fast volume from image provisioning | 14:18 |
| geguileo | KeithMnemonic: it's the first step | 14:19 |
| geguileo | KeithMnemonic: before os-brick can do any detach it gathers the mapping of (target, portal) to volumes | 14:19 |
| geguileo | KeithMnemonic: which is the purpose of the _get_connection_devices method | 14:20 |
| geguileo | KeithMnemonic: And the volumes that are mapped are split in 2 | 14:22 |
| geguileo | KeithMnemonic: those that belong to this connection_properties, and those that don't | 14:22 |
| geguileo | then we group all the volumes that belong to this connection so we can disconnect them | 14:23 |
| KeithMnemonic | so that is where it lies, i am looking at sysfs now to line up paths/disk wit dm devices, give me a few to finish that | 14:24 |
| geguileo | KeithMnemonic: OK, here's is the disconnection code https://github.com/openstack/os-brick/blob/stable/pike/os_brick/initiator/connectors/iscsi.py#L864-L897 | 14:24 |
| KeithMnemonic | so i verifed all of the paths are present before the migration starts | 14:29 |
| KeithMnemonic | and the luns line up | 14:29 |
| KeithMnemonic | 3600c0ff0003b7d6c79615d5d01000000 dm-7 Lenovo,DS6200 | 14:29 |
| KeithMnemonic | Class Device path = "/sys/devices/platform/host5/session1/target5:0:0/5:0:0:2/block/sdh" | 14:29 |
| KeithMnemonic | Class Device path = "/sys/devices/platform/host6/session2/target6:0:0/6:0:0:2/block/sdg" | 14:29 |
| KeithMnemonic | Class Device path = "/sys/devices/platform/host8/session4/target8:0:0/8:0:0:2/block/sdi" | 14:29 |
| KeithMnemonic | Class Device path = "/sys/devices/platform/host7/session3/target7:0:0/7:0:0:2/block/sdf" | 14:29 |
| KeithMnemonic | 3600c0ff0003b7d6c7a615d5d01000000 dm-10 Lenovo,DS6200 | 14:29 |
| KeithMnemonic | Class Device path = "/sys/devices/platform/host5/session1/target5:0:0/5:0:0:3/block/sdj" | 14:29 |
| KeithMnemonic | Class Device path = "/sys/devices/platform/host6/session2/target6:0:0/6:0:0:3/block/sdm" | 14:29 |
| KeithMnemonic | Class Device path = "/sys/devices/platform/host7/session3/target7:0:0/7:0:0:3/block/sdl" | 14:29 |
| KeithMnemonic | Class Device path = "/sys/devices/platform/host8/session4/target8:0:0/8:0:0:3/block/sdk" | 14:29 |
| KeithMnemonic | so for exmaple dm-7 migrates correctly | 14:29 |
| smcginnis | paste is your (well, our) friend. | 14:30 |
| KeithMnemonic | dm-10 is one of the ones where no multipath -f is ever run by brick and _get_connection_devices returns empty for the "belongs" | 14:30 |
| geguileo | smcginnis: so true :-) | 14:30 |
| KeithMnemonic | yeag let me do that osrry | 14:30 |
| geguileo | KeithMnemonic: do you have the connection_information that is passed to os-brick's disconnect for dm-10? | 14:31 |
| KeithMnemonic | http://paste.openstack.org/show/777163/ | 14:31 |
| KeithMnemonic | let me get that in the next paste | 14:31 |
| KeithMnemonic | did you want that output when it starts the migrate for the instance attached to dm-10? | 14:32 |
| geguileo | KeithMnemonic: I'd like to see the call to os-brick for the disconnect of the volume | 14:33 |
| geguileo | KeithMnemonic: where one can see the connection information | 14:33 |
| geguileo | KeithMnemonic: it shows the dictionary with the tartget_iqn, target_portal, target_iqns, target_portals, etc | 14:33 |
| geguileo | KeithMnemonic: in that dict we can see the target_lun | 14:34 |
| KeithMnemonic | http://paste.openstack.org/show/777164/ | 14:37 |
| geguileo | KeithMnemonic: that doesn't look like the Nova call to os-brick's disconnect_volume | 14:39 |
| KeithMnemonic | http://paste.openstack.org/show/777165/ | 14:40 |
| KeithMnemonic | if you give me a keyword to search on | 14:40 |
| geguileo | that second one looks right | 14:41 |
| KeithMnemonic | ok | 14:41 |
| geguileo | KeithMnemonic: does that data dictionary have target_iqns or just target_iqn? | 14:42 |
| geguileo | (the pastebin doesn't have the complete "data" dictionary" so I don't know) | 14:42 |
| KeithMnemonic | target_iqn, http://paste.openstack.org/show/777166/ | 14:43 |
| KeithMnemonic | so let me check a working one | 14:44 |
| KeithMnemonic | working one also has target_iqn | 14:44 |
| *** FlorianFa has quit IRC | 14:44 | |
| geguileo | KeithMnemonic: something looks weird... | 14:44 |
| KeithMnemonic | also what is weird is on the 3par, i never see it do this iscsiadm -m discoverydb -o show -P 1 but i do on the lenovo | 14:45 |
| KeithMnemonic | so it is like the method to get stuff form the backend is different | 14:45 |
| KeithMnemonic | or the backends just respond differentlky | 14:46 |
| geguileo | KeithMnemonic: because the 3PAR driver returns all the target IQNs and portals | 14:46 |
| geguileo | KeithMnemonic: whereas the Lenovo relies on discovery | 14:46 |
| KeithMnemonic | ah | 14:46 |
| KeithMnemonic | so maybe something is not discovered properly? | 14:46 |
| KeithMnemonic | after some time. | 14:46 |
| KeithMnemonic | i wonder if this is some concurency issue | 14:46 |
| geguileo | let me have a look at the discovery code | 14:47 |
| KeithMnemonic | i.e it gets buys migrating 1-3, so then discovery on 4-6 does not work correctly | 14:47 |
| KeithMnemonic | s/buys/busy | 14:47 |
| KeithMnemonic | Be back in a few, thanks again geguileo for helping me dig into this | 14:48 |
| geguileo | KeithMnemonic: OK, when you get back I'd like to see the log entry on the request of the failed disconnect that says "Getting connected devices for (ips,iqns,luns)=' | 14:54 |
| *** pcaruana has quit IRC | 14:55 | |
| *** raghavendrat has joined #openstack-cinder | 14:58 | |
| raghavendrat | hi e0ne: eharney: hemna_: jungleboyj: smcginnis: whoami-rajat: this is regarding https://review.opendev.org/#/c/677945/ | 15:01 |
| raghavendrat | Zuul and HPE Storage CI [python 3.7] have passed. | 15:02 |
| *** udesale has quit IRC | 15:08 | |
| *** lpetrut has quit IRC | 15:12 | |
| *** sapd1 has quit IRC | 15:14 | |
| *** sapd1 has joined #openstack-cinder | 15:15 | |
| KeithMnemonic | ok back let me grab it and put it into paste | 15:25 |
| *** boxiang has quit IRC | 15:27 | |
| *** boxiang has joined #openstack-cinder | 15:28 | |
| KeithMnemonic | geguile so that is interesting, i see 6 entries, which should correspond to the 6 migrations the luns in those are 7,5,4,2,2,2 but those are not the luns listed under /sys | 15:30 |
| KeithMnemonic | http://paste.openstack.org/show/777169/ | 15:30 |
| KeithMnemonic | 7 is not even listed here http://paste.openstack.org/show/777163/ | 15:30 |
| geguileo | KeithMnemonic: now that's weird... | 15:31 |
| KeithMnemonic | so that would make sense why we saw the down in the path | 15:31 |
| KeithMnemonic | if 7 is not listed and it pulled that one first | 15:31 |
| geguileo | KeithMnemonic: could you check your system logs to see if there's any SCSI message about LUNs? | 15:32 |
| *** enriquetaso has quit IRC | 15:32 | |
| KeithMnemonic | like remapping? | 15:32 |
| KeithMnemonic | sure | 15:32 |
| KeithMnemonic | like this " Warning! Received an indication that the LUN assignments on this target h | 15:33 |
| KeithMnemonic | ave changed. The Linux SCSI layer does not automatical" ;-) | 15:33 |
| geguileo | KeithMnemonic: yes | 15:33 |
| KeithMnemonic | that occurs before the migrations start | 15:34 |
| KeithMnemonic | ok so that is why this all goes fubar | 15:34 |
| geguileo | KeithMnemonic: then that's a problem with the 6200 system | 15:34 |
| KeithMnemonic | question is why is the kernel detecting a change | 15:34 |
| *** CeeMac has joined #openstack-cinder | 15:35 | |
| KeithMnemonic | jungleboyj: Paging a 6200 person ;-) | 15:35 |
| geguileo | KeithMnemonic: because there's probably an AEN/AER message sent by the 6200 | 15:36 |
| *** carloss has quit IRC | 15:40 | |
| *** e0ne has quit IRC | 15:48 | |
| *** sahid has quit IRC | 15:49 | |
| hemna_ | mep | 15:51 |
| *** raghavendrat has left #openstack-cinder | 15:55 | |
| *** jmlowe has quit IRC | 15:55 | |
| hemna_ | so the 6200 is causing this ? | 15:56 |
| geguileo | hemna_: looks that way | 15:56 |
| geguileo | hemna_: it seems that once you remove the lower 3 LUNs it goes an reassigns the others | 15:57 |
| hemna_ | ok not much if anything we can do then | 15:57 |
| hemna_ | ouch | 15:57 |
| hemna_ | why the hell would they do that? | 15:57 |
| geguileo | hemna_: not really, because the reassigning is even breaking attached volumes | 15:57 |
| hemna_ | all of the existing paths would fail on the host | 15:57 |
| geguileo | hemna_: exactly | 15:57 |
| hemna_ | w i t f | 15:57 |
| geguileo | lol | 15:57 |
| hemna_ | that's bad mmmkay | 15:57 |
| hemna_ | drugs are bad mmmkay | 15:58 |
| geguileo | yeah, because Linux doesn't automatically change the LUNs even when it receives the message from the backend | 15:58 |
| hemna_ | changing luns are bad mmmkay | 15:58 |
| geguileo | This is what I said back at the beginning of this session: | 16:00 |
| geguileo | geguileo | I can only think of the 3par backend changing the LUNs on its own | 16:00 |
| geguileo | geguileo | after the first 3 LUNs have been removed | 16:00 |
| geguileo | geguileo | but afaik that would break everything... | 16:00 |
| geguileo | geguileo | so I doubt that's happening | 16:00 |
| geguileo | and it turns out that's precisely what's happening :-( | 16:01 |
| hemna_ | I don't think the 3par changes luns | 16:01 |
| hemna_ | the lun id is set at initialize_connection time and never changes for an existing vol export | 16:01 |
| geguileo | hemna_: 3PAR doesn't do it | 16:01 |
| geguileo | hemna_: I made the mistake of saying 3PAR instead of 6200 | 16:02 |
| hemna_ | once it has the lun id assigned, it's never changed | 16:02 |
| hemna_ | ok | 16:02 |
| geguileo | hemna_: yeah, I was referring to the 6200 that's giving trouble | 16:02 |
| *** mriedem is now known as mriedem_afk | 16:09 | |
| *** kukacz has quit IRC | 16:19 | |
| *** enriquetaso has joined #openstack-cinder | 16:40 | |
| *** carloss has joined #openstack-cinder | 16:42 | |
| openstackgerrit | Helen Walsh proposed openstack/cinder master: PowerMax Docs - Short host and port group name changes https://review.opendev.org/682696 | 16:47 |
| *** e0ne has joined #openstack-cinder | 16:55 | |
| *** pcaruana has joined #openstack-cinder | 16:55 | |
| *** zhubx has joined #openstack-cinder | 16:57 | |
| *** boxiang has quit IRC | 17:00 | |
| *** e0ne has quit IRC | 17:00 | |
| *** e0ne has joined #openstack-cinder | 17:01 | |
| KeithMnemonic | hemna: geguileo: thanks again, i am asking that customer to have lenovo examine their config/FW on the 6200 | 17:02 |
| hemna_ | ok good luck KeithMnemonic | 17:02 |
| geguileo | KeithMnemonic: no problem, yes, that would be best | 17:02 |
| *** e0ne has quit IRC | 17:05 | |
| *** e0ne has joined #openstack-cinder | 17:13 | |
| *** e0ne has quit IRC | 17:17 | |
| *** ociuhandu has quit IRC | 17:25 | |
| *** jmlowe has joined #openstack-cinder | 17:25 | |
| *** enriquetaso has quit IRC | 17:26 | |
| *** e0ne has joined #openstack-cinder | 17:33 | |
| *** e0ne has quit IRC | 17:43 | |
| *** ociuhandu has joined #openstack-cinder | 17:44 | |
| *** ociuhandu has quit IRC | 17:48 | |
| *** mmethot_ has quit IRC | 17:53 | |
| *** mmethot_ has joined #openstack-cinder | 17:53 | |
| *** mriedem_afk is now known as mriedem | 17:53 | |
| *** jmlowe has quit IRC | 18:02 | |
| *** whoami-rajat has quit IRC | 18:12 | |
| *** enriquetaso has joined #openstack-cinder | 18:19 | |
| *** gregwork has joined #openstack-cinder | 18:31 | |
| *** jmlowe has joined #openstack-cinder | 18:32 | |
| gregwork | what options are available when a volume snapshot is stuck in "Deleting" for a long time. | 18:32 |
| gregwork | i have an app (cloudforms) which is going through and performing smart state analysis (snapshot instnace volume / analyze contents / discard snapshot) on a bunch of tenants and for whatever reason that process failed. I have a pile of snapshots all sitting in "Deleting" now for a long time and im trying to clean them up. the back end storage is ceph | 18:34 |
| gregwork | the ones stuck in deleting have os-extended-snapshot-attributes:progress 0% | 18:34 |
| hemna_ | did you initiate the snapshot delete ? | 18:37 |
| gregwork | it did | 18:37 |
| *** henriqueof has joined #openstack-cinder | 18:38 | |
| gregwork | but it has been in this state for hours | 18:38 |
| hemna_ | I'd check the c-vol logs | 18:38 |
| gregwork | ls -la | 18:38 |
| hemna_ | at the time the delete was issued | 18:38 |
| gregwork | er wrong window sorry | 18:38 |
| gregwork | ther eappear to be a bunch of sqlalchemy errors and lost connectivity to the back end mysql | 18:39 |
| hemna_ | ceph driver complaints? | 18:40 |
| gregwork | nothing at present ceph -s shows HEALTH_OK | 18:40 |
| hemna_ | if the ceph driver just lost connectivity to the cluster, you can simple reset the state of the snaps, and try again | 18:40 |
| gregwork | cinder-shced says NoValidBackend: Cannot place snapshot on hostgroup@tripleo_ceph#tripleo_ceph | 18:42 |
| gregwork | thre is a bunch of that | 18:42 |
| gregwork | around the same time | 18:42 |
| hemna_ | anything else around that message? | 18:42 |
| hemna_ | probably a reason why it can't place it | 18:42 |
| *** hemna_ is now known as hemna_afk | 18:43 | |
| gregwork | volume service is down. host: hostgroup@tripleo_ceph | 18:43 |
| hemna_afk | there yah go | 18:43 |
| hemna_afk | if the host is back up, then just reset state on the snaps and delete em | 18:43 |
| hemna_afk | ok really afk | 18:43 |
| gregwork | how do i reset the state | 18:43 |
| hemna_afk | cinderclient | 18:43 |
| gregwork | alright ill look into it | 18:44 |
| gregwork | thanks | 18:44 |
| *** ociuhandu has joined #openstack-cinder | 18:52 | |
| *** ociuhandu has quit IRC | 18:56 | |
| *** e0ne has joined #openstack-cinder | 18:59 | |
| *** jmlowe has quit IRC | 19:08 | |
| mriedem | eharney: smcginnis: can we get https://review.opendev.org/#/c/668130/ into devstack-plugin-ceph now so we do'nt regress any shelve testing in queens? | 19:17 |
| mriedem | the nova change is merged | 19:17 |
| eharney | mriedem: looks good to me | 19:20 |
| mriedem | thanks | 19:21 |
| *** sahid has joined #openstack-cinder | 19:36 | |
| *** mmethot_ has quit IRC | 19:37 | |
| *** mmethot_ has joined #openstack-cinder | 19:37 | |
| *** e0ne has quit IRC | 19:38 | |
| *** mmethot_ has quit IRC | 19:40 | |
| *** mmethot_ has joined #openstack-cinder | 19:41 | |
| *** spsurya has quit IRC | 19:48 | |
| *** jmlowe has joined #openstack-cinder | 19:48 | |
| *** sahid has quit IRC | 19:50 | |
| *** e0ne has joined #openstack-cinder | 19:50 | |
| *** senrique_ has joined #openstack-cinder | 20:09 | |
| *** enriquetaso has quit IRC | 20:12 | |
| *** whfnst has quit IRC | 20:15 | |
| *** mriedem has quit IRC | 20:32 | |
| *** mriedem has joined #openstack-cinder | 20:32 | |
| *** zhubx has quit IRC | 20:41 | |
| *** boxiang has joined #openstack-cinder | 20:41 | |
| *** trident has quit IRC | 20:48 | |
| *** mmethot_ has quit IRC | 20:48 | |
| *** mmethot has joined #openstack-cinder | 20:49 | |
| *** ociuhandu has joined #openstack-cinder | 20:53 | |
| *** mmethot has quit IRC | 20:53 | |
| *** mmethot has joined #openstack-cinder | 20:54 | |
| *** mmethot has quit IRC | 20:55 | |
| *** eharney has quit IRC | 20:55 | |
| *** mmethot has joined #openstack-cinder | 20:55 | |
| *** pcaruana has quit IRC | 20:58 | |
| *** trident has joined #openstack-cinder | 21:00 | |
| *** idlemind has joined #openstack-cinder | 21:10 | |
| *** mmethot has quit IRC | 21:13 | |
| *** mmethot has joined #openstack-cinder | 21:14 | |
| *** e0ne has quit IRC | 21:19 | |
| *** e0ne has joined #openstack-cinder | 21:28 | |
| *** e0ne has quit IRC | 21:28 | |
| *** senrique_ has quit IRC | 21:29 | |
| *** ociuhandu has quit IRC | 21:33 | |
| *** markvoelker has quit IRC | 21:35 | |
| jungleboyj | rosmaita: Were you going to do the forum topic submissions? | 21:59 |
| rosmaita | jungleboyj: yes, i think they are due on friday? | 21:59 |
| jungleboyj | That sounds right. Ok, just wanted to make sure you were planning to do that so it didn't get missed. | 22:00 |
| rosmaita | Thanks! | 22:00 |
| jungleboyj | Thank you. | 22:02 |
| *** hemna has joined #openstack-cinder | 22:09 | |
| *** lseki has quit IRC | 22:45 | |
| *** mriedem has quit IRC | 22:52 | |
| *** hemna has quit IRC | 22:57 | |
| *** tkajinam has joined #openstack-cinder | 23:04 | |
| *** zhubx has joined #openstack-cinder | 23:09 | |
| *** boxiang has quit IRC | 23:10 | |
| *** tosky has quit IRC | 23:11 | |
| *** markvoelker has joined #openstack-cinder | 23:22 | |
| *** n-saito has joined #openstack-cinder | 23:36 | |
| *** rcernin has joined #openstack-cinder | 23:44 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!