| *** Yogi1 has joined #openstack-cinder | 00:00 | |
| *** markvoelker_ has quit IRC | 00:02 | |
| *** Yogi1 has quit IRC | 00:12 | |
| patrickeast | asselin: i wonder how hard it would be to set up something like logstash internally for a 3rd party ci system | 00:19 |
|---|---|---|
| asselin | patrickeast, shouldn't be too hard: http://ci.openstack.org/logstash.html | 00:20 |
| asselin | it's on my list to things to do eventually | 00:20 |
| *** drjones has quit IRC | 00:21 | |
| *** annashen has joined #openstack-cinder | 00:21 | |
| *** _cjones_ has joined #openstack-cinder | 00:21 | |
| *** Rockyg has quit IRC | 00:22 | |
| *** leeantho has quit IRC | 00:23 | |
| *** _cjones_ has quit IRC | 00:25 | |
| *** primechuck has joined #openstack-cinder | 00:33 | |
| *** ozialien has joined #openstack-cinder | 00:35 | |
| *** r-daneel has quit IRC | 00:37 | |
| *** dims_ has quit IRC | 00:40 | |
| *** garthb_ has quit IRC | 00:41 | |
| *** garthb has quit IRC | 00:41 | |
| *** mdbooth has quit IRC | 00:46 | |
| *** annashen has quit IRC | 00:52 | |
| *** mdbooth has joined #openstack-cinder | 00:52 | |
| *** esker has quit IRC | 00:54 | |
| thingee | mtreinish: there is definitely a delete happening before the detach. | 00:57 |
| *** aswadr has joined #openstack-cinder | 00:58 | |
| *** Lee1092 has joined #openstack-cinder | 00:59 | |
| *** jwcroppe has quit IRC | 01:02 | |
| *** heyun has joined #openstack-cinder | 01:02 | |
| *** primechuck has quit IRC | 01:04 | |
| *** ozialien has quit IRC | 01:06 | |
| *** markvoelker has joined #openstack-cinder | 01:07 | |
| *** mriedem has joined #openstack-cinder | 01:18 | |
| *** julim has joined #openstack-cinder | 01:18 | |
| *** Yogi1 has joined #openstack-cinder | 01:19 | |
| *** Mandell has quit IRC | 01:24 | |
| *** Mandell has joined #openstack-cinder | 01:25 | |
| *** Mandell has quit IRC | 01:26 | |
| *** Mandell has joined #openstack-cinder | 01:27 | |
| *** fanyaohong has quit IRC | 01:29 | |
| *** markvoelker has quit IRC | 01:30 | |
| *** Mandell has quit IRC | 01:32 | |
| *** _cjones_ has joined #openstack-cinder | 01:33 | |
| *** mtanino has quit IRC | 01:37 | |
| *** dims has joined #openstack-cinder | 01:40 | |
| *** Mandell has joined #openstack-cinder | 01:43 | |
| *** Mandell has quit IRC | 01:44 | |
| *** Yogi1 has quit IRC | 01:45 | |
| *** primechuck has joined #openstack-cinder | 01:45 | |
| thingee | mtreinish: http://paste.openstack.org/show/204050/ | 01:46 |
| thingee | mtreinish: 352125fa-5920-418d-aa7c-db64669c828c | 01:46 |
| thingee | notice how that volume is detached, exported, deleted, detached | 01:46 |
| *** dims has quit IRC | 01:47 | |
| *** annashen has joined #openstack-cinder | 01:53 | |
| *** kaisers1 has joined #openstack-cinder | 01:56 | |
| *** Yogi1 has joined #openstack-cinder | 01:58 | |
| *** kaisers has quit IRC | 01:59 | |
| *** winston-1_ has quit IRC | 02:04 | |
| *** harlowja is now known as harlowja_away | 02:05 | |
| *** strictlyb has quit IRC | 02:08 | |
| *** esker has joined #openstack-cinder | 02:08 | |
| *** thingee has quit IRC | 02:09 | |
| *** annashen has quit IRC | 02:17 | |
| *** patrickeast has quit IRC | 02:18 | |
| *** bkopilov has quit IRC | 02:18 | |
| *** Yogi1 has quit IRC | 02:20 | |
| *** Apoorva_ has joined #openstack-cinder | 02:28 | |
| *** Apoorva has quit IRC | 02:31 | |
| *** Apoorva_ has quit IRC | 02:32 | |
| *** primechuck has quit IRC | 02:32 | |
| *** liusheng has joined #openstack-cinder | 02:33 | |
| *** _cjones_ has quit IRC | 02:33 | |
| *** dannywilson has quit IRC | 02:36 | |
| *** davechen has joined #openstack-cinder | 02:44 | |
| *** dims has joined #openstack-cinder | 02:48 | |
| *** strictlyb has joined #openstack-cinder | 02:50 | |
| *** dims has quit IRC | 02:53 | |
| *** Longgeek has joined #openstack-cinder | 02:57 | |
| *** IanGovett has quit IRC | 03:06 | |
| *** jwcroppe has joined #openstack-cinder | 03:07 | |
| *** ho has quit IRC | 03:17 | |
| *** annashen has joined #openstack-cinder | 03:17 | |
| *** thingee has joined #openstack-cinder | 03:36 | |
| *** haomaiwang has quit IRC | 03:37 | |
| *** bkopilov has joined #openstack-cinder | 03:40 | |
| *** annashen has quit IRC | 03:43 | |
| *** avishay has joined #openstack-cinder | 03:55 | |
| *** strictlyb has quit IRC | 03:56 | |
| *** mriedem has quit IRC | 03:57 | |
| *** mriedem has joined #openstack-cinder | 03:59 | |
| *** _cjones_ has joined #openstack-cinder | 04:00 | |
| *** strictlyb has joined #openstack-cinder | 04:03 | |
| *** mriedem has quit IRC | 04:04 | |
| *** Longgeek_ has joined #openstack-cinder | 04:05 | |
| *** Longgeek has quit IRC | 04:06 | |
| *** rushiagr_away is now known as rushiagr | 04:10 | |
| *** Mandell has joined #openstack-cinder | 04:14 | |
| *** ankit_ag has joined #openstack-cinder | 04:25 | |
| *** ankit8188 has joined #openstack-cinder | 04:27 | |
| *** ankit_ag has quit IRC | 04:27 | |
| *** vilobhmm1 has joined #openstack-cinder | 04:39 | |
| *** _cjones_ has quit IRC | 04:43 | |
| *** annashen has joined #openstack-cinder | 04:43 | |
| *** afazekas has quit IRC | 04:51 | |
| *** rushiagr is now known as rushiagr_away | 05:02 | |
| *** vilobhmm1 has quit IRC | 05:02 | |
| *** vilobhmm1 has joined #openstack-cinder | 05:05 | |
| *** sks has joined #openstack-cinder | 05:09 | |
| *** rongze has joined #openstack-cinder | 05:11 | |
| *** annashen has quit IRC | 05:13 | |
| openstackgerrit | Mike Perez proposed openstack/cinder-specs: Introduce use case section https://review.openstack.org/149449 | 05:16 |
| openstackgerrit | Mike Perez proposed openstack/cinder-specs: Introduce use case section https://review.openstack.org/149449 | 05:21 |
| *** rongze_ has joined #openstack-cinder | 05:21 | |
| *** rongze has quit IRC | 05:21 | |
| *** zerda has joined #openstack-cinder | 05:22 | |
| openstackgerrit | Jeegn Chen proposed openstack/cinder: Remove redundant code from VNX Cinder Driver https://review.openstack.org/173183 | 05:22 |
| *** rongze has joined #openstack-cinder | 05:28 | |
| *** rongze_ has quit IRC | 05:29 | |
| *** rongze has quit IRC | 05:29 | |
| openstackgerrit | Jeegn Chen proposed openstack/cinder: Remove redundant code from VNX Cinder Driver https://review.openstack.org/173183 | 05:29 |
| *** vilobhmm1 has quit IRC | 05:29 | |
| openstackgerrit | Mike Perez proposed openstack/cinder-specs: Open Liberty for specs https://review.openstack.org/173675 | 05:30 |
| openstackgerrit | Mike Perez proposed openstack/cinder-specs: Open Liberty for specs https://review.openstack.org/173675 | 05:31 |
| *** zerda has quit IRC | 05:43 | |
| *** zerda has joined #openstack-cinder | 05:44 | |
| *** dims has joined #openstack-cinder | 05:45 | |
| *** dims has quit IRC | 05:50 | |
| *** rushiagr_away is now known as rushiagr | 05:51 | |
| *** BharatK has quit IRC | 05:55 | |
| *** nlevinki has joined #openstack-cinder | 06:02 | |
| *** xyang1 has quit IRC | 06:05 | |
| *** avishay_ has joined #openstack-cinder | 06:06 | |
| *** avishay has quit IRC | 06:06 | |
| *** alecv has joined #openstack-cinder | 06:07 | |
| *** annashen has joined #openstack-cinder | 06:11 | |
| *** lpetrut has joined #openstack-cinder | 06:16 | |
| *** BharatK has joined #openstack-cinder | 06:18 | |
| *** bnemec has quit IRC | 06:18 | |
| *** thingee has quit IRC | 06:19 | |
| *** _cjones_ has joined #openstack-cinder | 06:21 | |
| *** TobiasE has joined #openstack-cinder | 06:26 | |
| *** nshaikh has joined #openstack-cinder | 06:26 | |
| *** annashen has quit IRC | 06:44 | |
| *** tshefi has joined #openstack-cinder | 06:45 | |
| *** nikesh has joined #openstack-cinder | 06:46 | |
| *** nkrinner has joined #openstack-cinder | 06:50 | |
| openstackgerrit | Vincent Hou proposed openstack/cinder: Select one instance to swap the volume for migration https://review.openstack.org/174223 | 06:54 |
| *** gcivitella has joined #openstack-cinder | 06:59 | |
| *** bnemec has joined #openstack-cinder | 07:00 | |
| *** dansmith has quit IRC | 07:04 | |
| *** markus_z has joined #openstack-cinder | 07:05 | |
| *** sks has quit IRC | 07:06 | |
| *** nshaikh has quit IRC | 07:06 | |
| *** BharatK has quit IRC | 07:07 | |
| *** dansmith has joined #openstack-cinder | 07:08 | |
| *** dansmith is now known as Guest67672 | 07:08 | |
| *** _cjones_ has quit IRC | 07:13 | |
| *** gcivitella_ has joined #openstack-cinder | 07:17 | |
| *** gcivitella has quit IRC | 07:18 | |
| *** liusheng has quit IRC | 07:18 | |
| *** rongze has joined #openstack-cinder | 07:18 | |
| *** gcivitella__ has joined #openstack-cinder | 07:22 | |
| *** gcivitella has joined #openstack-cinder | 07:23 | |
| *** gcivitella_ has quit IRC | 07:23 | |
| *** Mandell has quit IRC | 07:25 | |
| *** BharatK has joined #openstack-cinder | 07:28 | |
| *** sks has joined #openstack-cinder | 07:28 | |
| *** nshaikh has joined #openstack-cinder | 07:28 | |
| *** esker has quit IRC | 07:30 | |
| *** ronis has joined #openstack-cinder | 07:37 | |
| *** pschaef has joined #openstack-cinder | 07:38 | |
| *** rongze has quit IRC | 07:38 | |
| *** rongze has joined #openstack-cinder | 07:38 | |
| openstackgerrit | Deepak C Shetty proposed openstack/cinder: WIP: Cinder part of volume snapshot improvements https://review.openstack.org/172808 | 07:39 |
| *** chlong has quit IRC | 07:40 | |
| *** jistr has joined #openstack-cinder | 07:43 | |
| *** annashen has joined #openstack-cinder | 07:43 | |
| *** Mandell has joined #openstack-cinder | 07:44 | |
| *** abhiram_moturi has quit IRC | 07:50 | |
| *** abhiram_moturi has joined #openstack-cinder | 07:50 | |
| *** rongze has quit IRC | 07:56 | |
| *** rongze has joined #openstack-cinder | 07:56 | |
| *** rongze has quit IRC | 08:01 | |
| *** Mandell has quit IRC | 08:01 | |
| *** esker has joined #openstack-cinder | 08:01 | |
| winston-d | jbernard, jdurgin: https://bugs.launchpad.net/cinder/+bug/1444855 | 08:02 |
| openstack | Launchpad bug 1444855 in Cinder "RBD driver doesn't support customized ceph cluster name" [Undecided,New] | 08:02 |
| winston-d | jbernard, jdurgin: rbd driver should allow user to specify cluster name. | 08:03 |
| *** dalgaaf has joined #openstack-cinder | 08:07 | |
| *** ociuhandu has joined #openstack-cinder | 08:08 | |
| openstackgerrit | Deepak C Shetty proposed openstack/cinder-specs: Assisted snapshot improvements https://review.openstack.org/172373 | 08:10 |
| *** sgotliv has joined #openstack-cinder | 08:10 | |
| openstackgerrit | Daisuke Fujita proposed openstack/cinder: Fix a wrong argument of delete_keys method https://review.openstack.org/174241 | 08:11 |
| *** jordanP has joined #openstack-cinder | 08:12 | |
| openstackgerrit | Fabien Boucher proposed openstack/cinder: Complete the doc/README.rst instructions to build docs https://review.openstack.org/165105 | 08:13 |
| *** annashen has quit IRC | 08:16 | |
| *** e0ne has joined #openstack-cinder | 08:24 | |
| *** c0m0 has joined #openstack-cinder | 08:31 | |
| *** ndipanov has joined #openstack-cinder | 08:33 | |
| *** rongze has joined #openstack-cinder | 08:34 | |
| *** rongze has quit IRC | 08:44 | |
| *** rongze has joined #openstack-cinder | 08:45 | |
| *** rongze_ has joined #openstack-cinder | 08:46 | |
| *** rongze has quit IRC | 08:46 | |
| *** e0ne has quit IRC | 08:48 | |
| *** esker has quit IRC | 08:52 | |
| *** skraynev has quit IRC | 08:55 | |
| *** rongze_ has quit IRC | 08:57 | |
| *** rongze has joined #openstack-cinder | 08:57 | |
| *** e0ne has joined #openstack-cinder | 09:09 | |
| *** jaypipes has quit IRC | 09:10 | |
| *** e0ne has quit IRC | 09:10 | |
| *** rongze_ has joined #openstack-cinder | 09:10 | |
| *** rongze has quit IRC | 09:11 | |
| *** rongze_ has quit IRC | 09:12 | |
| *** skraynev has joined #openstack-cinder | 09:12 | |
| *** avishay_ has quit IRC | 09:14 | |
| *** annashen has joined #openstack-cinder | 09:15 | |
| *** esker has joined #openstack-cinder | 09:17 | |
| *** avishay_ has joined #openstack-cinder | 09:18 | |
| *** dims has joined #openstack-cinder | 09:19 | |
| *** dims has quit IRC | 09:24 | |
| *** lpetrut has quit IRC | 09:25 | |
| *** deepakcs has joined #openstack-cinder | 09:25 | |
| *** geguileo has quit IRC | 09:35 | |
| *** sgotliv has quit IRC | 09:35 | |
| *** nshaikh has quit IRC | 09:36 | |
| *** aswadr has quit IRC | 09:36 | |
| *** nshaikh has joined #openstack-cinder | 09:36 | |
| *** geguileo has joined #openstack-cinder | 09:37 | |
| *** esker has quit IRC | 09:38 | |
| openstackgerrit | Rick Chen proposed openstack/cinder: Add patch for consistency group update in ProphetStor driver https://review.openstack.org/169653 | 09:43 |
| *** aix has joined #openstack-cinder | 09:44 | |
| *** Maike has joined #openstack-cinder | 09:45 | |
| *** ishant has joined #openstack-cinder | 09:45 | |
| *** afazekas has joined #openstack-cinder | 09:47 | |
| *** annashen has quit IRC | 09:49 | |
| *** davechen has quit IRC | 09:52 | |
| *** lpetrut has joined #openstack-cinder | 10:04 | |
| *** IanGovett has joined #openstack-cinder | 10:06 | |
| *** dims has joined #openstack-cinder | 10:06 | |
| *** divya has joined #openstack-cinder | 10:11 | |
| divya | hi | 10:11 |
| divya | creating volume getting error, the cinder scheduler has error "No valid host found" | 10:12 |
| divya | can anybody help | 10:12 |
| *** rongze has joined #openstack-cinder | 10:13 | |
| *** rongze has quit IRC | 10:17 | |
| *** cbader has quit IRC | 10:18 | |
| *** asselin has quit IRC | 10:19 | |
| *** deepakcs has quit IRC | 10:23 | |
| divya | hi | 10:24 |
| DuncanT | divya: Usually means your volume manager didn't start properly, grep ERROR in the cinder-volume log | 10:30 |
| openstackgerrit | Petrut Lucian proposed openstack/cinder: SMBFS: Fix retrieving total allocated size https://review.openstack.org/158713 | 10:34 |
| openstackgerrit | Petrut Lucian proposed openstack/cinder: Windows SMBFS: fix volume extend https://review.openstack.org/174290 | 10:34 |
| openstackgerrit | Petrut Lucian proposed openstack/cinder: Windows SMBFS: Fix image resize errors during volume creation https://review.openstack.org/174291 | 10:34 |
| openstackgerrit | Petrut Lucian proposed openstack/cinder: SMBFS: Add minimum qemu-img version requirement https://review.openstack.org/161299 | 10:35 |
| *** sgotliv has joined #openstack-cinder | 10:42 | |
| *** rongze has joined #openstack-cinder | 10:42 | |
| *** kmartin_ has quit IRC | 10:46 | |
| *** annashen has joined #openstack-cinder | 10:47 | |
| *** rongze has quit IRC | 10:47 | |
| *** buster_604 has joined #openstack-cinder | 10:50 | |
| openstackgerrit | Petrut Lucian proposed openstack/cinder: SMBFS: Lock on a per-volume basis https://review.openstack.org/169858 | 10:52 |
| *** geguileo has quit IRC | 11:00 | |
| *** erlon has joined #openstack-cinder | 11:08 | |
| *** e0ne has joined #openstack-cinder | 11:10 | |
| *** geguileo has joined #openstack-cinder | 11:12 | |
| *** marcusvrn1 has joined #openstack-cinder | 11:15 | |
| *** aix has quit IRC | 11:17 | |
| *** marcusvrn has quit IRC | 11:18 | |
| *** timcl1 has left #openstack-cinder | 11:18 | |
| *** esker has joined #openstack-cinder | 11:22 | |
| *** annashen has quit IRC | 11:23 | |
| *** timcl has joined #openstack-cinder | 11:25 | |
| *** esker has quit IRC | 11:27 | |
| *** peluse has joined #openstack-cinder | 11:29 | |
| divya | Hi DuncanT, how to check the volume manager started or not, i am using devstack standalone setup | 11:43 |
| *** rongze has joined #openstack-cinder | 11:43 | |
| DuncanT | divya: If you've not enabled persistent logs in your local.conf, then you can just look at the c-vol screen tab | 11:44 |
| *** aix has joined #openstack-cinder | 11:45 | |
| divya | DuncanT i dont find any errors related to hp3par storage in the c-volumes | 11:46 |
| *** jamielennox is now known as jamielennox|away | 11:47 | |
| divya | File "/usr/local/lib/python2.7/dist-packages/hp3parclient/http.py", line 208, in _time_request^M ^[[01;31m2015-04-15 04:57:21.546 TRACE cinder.volume.manager ^[[01;35m^[[00m resp, body = self.request(url, method, **kwargs)^M ^[[01;31m2015-04-15 04:57:21.546 TRACE cinder.volume.manager ^[[01;35m^[[00m File "/usr/local/lib/python2.7/dist-packages/hp3parclient/http.py", line 202, in request^M ^[[01;31m2015-04-15 04:57:21.54 | 11:48 |
| divya | this error is observed in cinder volumes | 11:48 |
| *** rongze has quit IRC | 11:48 | |
| DuncanT | That paste is not readable, can you try to put the entire error into paste.openstack.org ? | 11:49 |
| openstackgerrit | yogeshprasad proposed openstack/cinder: Add chap support to CloudByte cinder driver https://review.openstack.org/173733 | 11:54 |
| *** peluse has quit IRC | 11:54 | |
| *** ishant has quit IRC | 11:56 | |
| *** rongze has joined #openstack-cinder | 11:59 | |
| divya | DuncanT-> i have pasted in #204086 | 12:00 |
| divya | can you please check | 12:00 |
| *** Mandell has joined #openstack-cinder | 12:02 | |
| *** rongze_ has joined #openstack-cinder | 12:04 | |
| *** avishay__ has joined #openstack-cinder | 12:05 | |
| *** rongze has quit IRC | 12:05 | |
| *** Mandell has quit IRC | 12:06 | |
| openstackgerrit | yogeshprasad proposed openstack/cinder: Add chap support to CloudByte cinder driver https://review.openstack.org/173733 | 12:07 |
| *** avishay_ has quit IRC | 12:08 | |
| *** heyun has quit IRC | 12:12 | |
| *** heyun has joined #openstack-cinder | 12:12 | |
| *** dims has quit IRC | 12:14 | |
| divya | Hi DuncanT, please let me know the whether u r able to view the logs | 12:14 |
| *** dims__ has joined #openstack-cinder | 12:17 | |
| *** zhenguo has quit IRC | 12:17 | |
| *** e0ne has quit IRC | 12:20 | |
| *** julim has quit IRC | 12:22 | |
| *** sks has quit IRC | 12:22 | |
| *** esker has joined #openstack-cinder | 12:23 | |
| *** annashen has joined #openstack-cinder | 12:25 | |
| *** rzerda has joined #openstack-cinder | 12:26 | |
| *** bkopilov has quit IRC | 12:27 | |
| *** esker has quit IRC | 12:27 | |
| *** rongze_ has quit IRC | 12:28 | |
| *** zerda has quit IRC | 12:29 | |
| *** rzerda has quit IRC | 12:30 | |
| *** cbader has joined #openstack-cinder | 12:30 | |
| divya | DuncanT-Logs are there in http://paste.openstack.org/show/204086/ | 12:30 |
| divya | the logs show HP3par array is uninitialised | 12:31 |
| divya | can you please help | 12:31 |
| *** bswartz has quit IRC | 12:34 | |
| *** avishay has joined #openstack-cinder | 12:38 | |
| *** avishay__ has quit IRC | 12:38 | |
| *** Yogi1 has joined #openstack-cinder | 12:39 | |
| *** akerr has joined #openstack-cinder | 12:41 | |
| *** julim has joined #openstack-cinder | 12:47 | |
| *** akerr has quit IRC | 12:48 | |
| *** akerr has joined #openstack-cinder | 12:48 | |
| *** pschaef has quit IRC | 12:52 | |
| *** avishay has quit IRC | 12:53 | |
| *** mwichmann has joined #openstack-cinder | 12:55 | |
| *** mriedem has joined #openstack-cinder | 12:56 | |
| *** annashen has quit IRC | 12:57 | |
| *** avishay has joined #openstack-cinder | 12:58 | |
| *** eharney has quit IRC | 13:00 | |
| *** avishay has quit IRC | 13:03 | |
| *** mwichmann has left #openstack-cinder | 13:03 | |
| *** annegentle has joined #openstack-cinder | 13:04 | |
| *** avishay has joined #openstack-cinder | 13:11 | |
| dulek | DuncanT: Hi! I couldn't make it to the meeting yesterday, but I see guys mentioned me a few times. | 13:12 |
| DuncanT | divya: Sorry, got pulled out into an incident, back now, just looking at your log | 13:12 |
| dulek | DuncanT: I was looking into some issues and right now fellow engineer is writing a PoC of moving locks in c-vol to tooz, | 13:13 |
| DuncanT | divya: If you restart the volume manager (just ctrl+c it then repeat the last command in the shell history) you should be able to find the first error, which will tell you *why* the manager is uninitialised | 13:14 |
| dulek | DuncanT: This may solve issues with drivers locks. Only requirement - redis installed. | 13:14 |
| dulek | DuncanT: This is how Ceilometer is using tooz. :) | 13:14 |
| DuncanT | dulek: I've been doing some more analysis today. One issue that isn't solved by tooz or similar is how to get rid of stuck lock when a node crashes / reboots/etc | 13:15 |
| dulek | DuncanT: I don't think this issue is valid. Let me ask my colleague and get back to you. | 13:15 |
| DuncanT | dulek: We might be better using a state change in the API to prevent concurrent operations at source... | 13:16 |
| DuncanT | dulek: No locks needed then | 13:16 |
| *** bswartz has joined #openstack-cinder | 13:17 | |
| *** sgotliv_ has joined #openstack-cinder | 13:18 | |
| *** sgotliv has quit IRC | 13:19 | |
| *** nlevinki has quit IRC | 13:19 | |
| *** dustins has joined #openstack-cinder | 13:19 | |
| *** Longgeek_ has quit IRC | 13:20 | |
| dulek | DuncanT: I do think that this is better approach, just trying to explore others possibilities. | 13:20 |
| *** nlevinki has joined #openstack-cinder | 13:20 | |
| dulek | DuncanT: So tooz won't deadlock on service failure because it requires services to send also heartbeats. | 13:21 |
| dulek | DuncanT: Lock is released if there are no heartbeats send. | 13:21 |
| dulek | DuncanT: Also - lock is released if service creates a new coordinator. | 13:22 |
| dulek | DuncanT: I'll add that to Etherpad. And definitely I want to be a part of this effort in L to make c-vol A/A-ready. :) | 13:23 |
| geguileo | DuncanT: There is timeout on locks | 13:23 |
| geguileo | DuncanT: So if service dies lock is released | 13:23 |
| geguileo | dulek: DuncanT Like I mentioned yesterday Tooz with Redis doesn't solve all issues for atomic state changes | 13:24 |
| geguileo | dulek: DuncanT Although it reduces to almost none, there are still corner cases where errors may happen | 13:25 |
| dulek | geguileo: Can you describe an example? It will be better to be on the same page. :) | 13:26 |
| geguileo | dulek: Ok | 13:27 |
| openstackgerrit | Tina Tang proposed openstack/cinder: Attach/detach batch processing in VNX driver https://review.openstack.org/154350 | 13:28 |
| dulek | geguileo: And if some drivers *require* locking we can create a way to do that - just before setting the driver - configure tooz. :) | 13:28 |
| geguileo | dulek: On that part I agree :) | 13:28 |
| *** zhipeng has joined #openstack-cinder | 13:29 | |
| geguileo | dulek: Case is when server A gets lock, then B waits onlock, A changes from available to X in galera node 1 and releases the lock, B acquires lock and looks at galera node 2 where resource is still set as available because propagation hasn't happened | 13:29 |
| *** markvoelker has joined #openstack-cinder | 13:30 | |
| geguileo | dulek: Unless you lock the resource for the whole duration of the operation and not just the state change | 13:30 |
| geguileo | dulek: But then you have to be careful with the heartbeat | 13:30 |
| dulek | geguileo: Oh, right now Cinder is locking for the whole operation I think. | 13:31 |
| dulek | geguileo: And I was thinking of that approach. | 13:31 |
| geguileo | dulek: But if I'm not mistaken the state change happens elsewhere | 13:32 |
| dulek | geguileo: I was never thinking of using tooz to simulate check-and-set (or select for update). That would be silly. ;) | 13:32 |
| geguileo | dulek: Because locks are being used for example so that snapshot don't get deleted when creating volume from snapshot | 13:32 |
| geguileo | dulek: Ok, so you were looking at tooz for resource locking and not for state changing? | 13:33 |
| dulek | geguileo: Exactly. | 13:34 |
| dulek | geguileo: | 13:36 |
| dulek | geguileo: https://github.com/openstack/cinder/blob/master/cinder/volume/manager.py#L121 | 13:36 |
| dulek | geguileo: I was thinking of these. | 13:36 |
| *** primechuck has joined #openstack-cinder | 13:36 | |
| *** Guest67672 is now known as dansmith | 13:36 | |
| *** dansmith is now known as Guest14453 | 13:37 | |
| geguileo | dulek: Ok, but you know that state changes happen on API | 13:37 |
| geguileo | dulek: and then calls rpcapi | 13:38 |
| dulek | geguileo: Yes, the more you explain the more I think check-and-set is definitely required. | 13:38 |
| dulek | geguileo: In the first place. :) | 13:38 |
| geguileo | dulek: Exactly, that's the first issue to solve: atomic state changes | 13:38 |
| *** esker has joined #openstack-cinder | 13:39 | |
| geguileo | dulek: Then for operations that need to lock resources we could use Tooz | 13:39 |
| dulek | geguileo: Okay, I'm glad we agreed. :) | 13:39 |
| geguileo | dulek: Me too :) | 13:39 |
| geguileo | dulek: I've been playing a little bit with Tooz and it's very convenient :) | 13:40 |
| *** rushil has joined #openstack-cinder | 13:42 | |
| *** divya has quit IRC | 13:42 | |
| *** Guest14453 is now known as dansmith_ | 13:43 | |
| *** esker has quit IRC | 13:43 | |
| *** ganso_ has joined #openstack-cinder | 13:46 | |
| *** annegentle has quit IRC | 13:48 | |
| *** IanGovett has quit IRC | 13:49 | |
| *** IanGovett has joined #openstack-cinder | 13:51 | |
| *** xyang has joined #openstack-cinder | 13:52 | |
| *** heyun has quit IRC | 13:54 | |
| *** annashen has joined #openstack-cinder | 13:56 | |
| *** dansmith_ is now known as dansmith | 13:58 | |
| *** zhenguo has joined #openstack-cinder | 14:01 | |
| *** xyang1 has joined #openstack-cinder | 14:02 | |
| DuncanT | dulek: geguileo: Sorry I keep dropping out of the converstation. The approach I'm trying to take for this not is to list the problem before looking at a solution. An example problem is that some drivers (e.g. the fs based ones) need mutual exclusion between delete, snap, create from snap etc. This is currently provided by the lock decorators in the drivers | 14:04 |
| *** kaisers1 has quit IRC | 14:04 | |
| DuncanT | dulek: geguileo: the nova->cinder API has races, including tryi ng to attach the volume to two VMs ast once. The current protection against that is the reserve call which sets the attaching state, and a call to check the state - this is still racy | 14:05 |
| geguileo | DuncanT: Yes, I know we have multiple issues: state changes, locking using files, drivers that apparently are not ussing synchronization... | 14:05 |
| DuncanT | dulek: geguileo: There are some locks in cinder-volume manager. I'm not sure what they are there to protect against, I'd like to know | 14:06 |
| dulek | DuncanT: Is there any way to achieve driver needs without distributed locking? | 14:06 |
| DuncanT | Once we have alist of all of the issues we are looking to solve, we can look at the various solutions | 14:06 |
| geguileo | DuncanT: The problem about the state races is the one I'm looking to fix now, that's what the tests I'm doing are for | 14:06 |
| DuncanT | dulek: FS drivers need mutual excludion of certain operations | 14:06 |
| DuncanT | geguileo: So some of the state races go away entirely if you fix the nnova api | 14:07 |
| DuncanT | geguileo: That is an example of why I intend to nak any state patches for a few days until we clearly understand the problems | 14:07 |
| geguileo | geguileo: But wouldn't it make more sense to remove race conditions inhouse? | 14:07 |
| DuncanT | geguileo: With the current API, you *can't* fix the race in-house | 14:07 |
| geguileo | DuncanT: That was for you, I don't know why I said it to myself ;) | 14:07 |
| geguileo | DuncanT: Really? | 14:08 |
| *** bkopilov has joined #openstack-cinder | 14:08 | |
| geguileo | DuncanT: Why can't these be tackled in Cinder direcly?, I must be missing something | 14:08 |
| dulek | geguileo: This is because current nova->cinder interaction is broken | 14:09 |
| geguileo | dulek: But isn't Nova using Cinder API for the interaction? | 14:10 |
| dulek | geguileo: What I noticed in the meeting logs is that Nova checks internal Cinder states for volume. DuncanT, is that the problem? Or are there more? | 14:10 |
| geguileo | dulek: So if we fix race conditions in API, souldn't that remove the issue in Cinder? | 14:10 |
| geguileo | dulek: You mean it access directly to the DB to check the state? | 14:11 |
| *** haomaiwang has joined #openstack-cinder | 14:11 | |
| *** sgotliv_ has quit IRC | 14:12 | |
| *** Mandell has joined #openstack-cinder | 14:13 | |
| dulek | geguileo: It was in the Etherpad actually | 14:14 |
| dulek | https://github.com/openstack/nova/blob/master/nova/volume/cinder.py#L308 | 14:14 |
| DuncanT | dulek: Nova checking the state is a problem, yeah | 14:14 |
| dulek | geguileo: I guess this introduces races because state can change. But to be honest this one can probably be taken care in cinder | 14:14 |
| DuncanT | dulek: Nova expects once it has chacked the state that that state can't change under it | 14:15 |
| *** haomaiwang has quit IRC | 14:15 | |
| DuncanT | dulek: e.g. if nova does check then reserve, and the check says the reserve should work, then nova has no code patch to handle the reserve failing | 14:15 |
| DuncanT | dulek: The checks should be gotten rid of, and make reserve fail if the state isn't right, and make nova deal with that | 14:16 |
| *** markvoelker_ has joined #openstack-cinder | 14:16 | |
| geguileo | DuncanT: dulek But that's Nova's problem | 14:16 |
| geguileo | ;) | 14:16 |
| DuncanT | duleek: I'm not sure we even need a reserve, just have initialise connection do the work of reserve | 14:16 |
| DuncanT | geguileo: No, it's our problem - stuck volumes make cinder look bad, not nova | 14:16 |
| DuncanT | geguileo: The bugs get filed as cinder bugs | 14:17 |
| DuncanT | geguileo: etc | 14:17 |
| geguileo | DuncanT: I was joking | 14:17 |
| dulek | ifco | 14:17 |
| *** dulek has quit IRC | 14:17 | |
| geguileo | DuncanT: But that's a completely different problem from the one I was talking about... | 14:17 |
| *** dulek has joined #openstack-cinder | 14:18 | |
| *** annegentle has joined #openstack-cinder | 14:18 | |
| dulek | hah, wrong window | 14:18 |
| dulek | DuncanT: Okay, thanks, this gets a lot clearer now. | 14:19 |
| *** markvoelker has quit IRC | 14:19 | |
| *** e0ne has joined #openstack-cinder | 14:19 | |
| DuncanT | dulek: No problem. I just wanted to be clear why I was resisting merging any of the state change fixes now - it is unclear exactly which of the many 'state problems' and 'state races' we have the patches are trying to solve | 14:20 |
| dulek | DuncanT: To be honest I'm aware of two approaches - micro-states and work with removing the file locks and get status changes atomic. | 14:21 |
| dulek | DuncanT: Are there more? | 14:21 |
| *** avishay has quit IRC | 14:21 | |
| *** emagana has joined #openstack-cinder | 14:21 | |
| DuncanT | dulek: Micro-states is one. Great big states is two. DLM is a third. Changing the cinder model to be more tolerant of failure and eventual consitency is another | 14:22 |
| DuncanT | dulek: I'm still at the 'what are the bugs, in detail' stage | 14:23 |
| dulek | DuncanT: That's fine. What's DLM? | 14:23 |
| *** markvoelker has joined #openstack-cinder | 14:23 | |
| DuncanT | Distributed lock manager, e.g. tooz | 14:23 |
| *** sgotliv_ has joined #openstack-cinder | 14:25 | |
| *** markvoelker_ has quit IRC | 14:26 | |
| *** mtanino has joined #openstack-cinder | 14:27 | |
| *** fanyaohong has joined #openstack-cinder | 14:27 | |
| *** thangp has joined #openstack-cinder | 14:27 | |
| *** ankit8188 has quit IRC | 14:29 | |
| *** eharney has joined #openstack-cinder | 14:30 | |
| *** annashen has quit IRC | 14:32 | |
| *** markvoelker has quit IRC | 14:34 | |
| *** nlevinki has quit IRC | 14:37 | |
| sdague | is there anyone that might be up for helping look at the following bug from a cinder perspective - https://bugs.launchpad.net/nova/+bug/1445021 | 14:38 |
| openstack | Launchpad bug 1445021 in OpenStack Compute (nova) "nova-compute does not start after upgrade from juno->kilo if there are boot from volume servers running" [Critical,New] | 14:38 |
| sdague | to help make sure cinder vol isn't doing something odd on start | 14:38 |
| sdague | we're debugging in the #openstack-nova channel | 14:38 |
| *** jaypipes has joined #openstack-cinder | 14:39 | |
| jbernard | sdague: im on a call, but i can try to help in a few | 14:41 |
| sdague | jbernard: cool, thanks | 14:41 |
| *** e0ne has quit IRC | 14:46 | |
| *** IanGovett1 has joined #openstack-cinder | 14:47 | |
| mtanino | sdague: Seems there is a /dev/disk/by-path symbolic link but in fact, there is no iSCSI connection or failed to re-connect iSCSI target, so that blockdev can't read read block device and return an exception. | 14:48 |
| sdague | mtanino: ok, what would have torn that down? | 14:48 |
| *** hemnafk is now known as hemna | 14:48 | |
| mtanino | sdague: there may be some errors in the /var/log/messages? | 14:48 |
| sdague | mtanino: http://logs.openstack.org/91/173791/11/check/check-grenade-dsvm/fc725f5/logs/syslog.txt.gz is the syslog | 14:49 |
| *** IanGovett has quit IRC | 14:49 | |
| openstackgerrit | Huang Zhiteng proposed openstack/cinder: Add support for customized cluster name https://review.openstack.org/174415 | 14:52 |
| -mtanino- Apr 16 11:46:43 ubuntu kernel: [ 2591.042808] connection10:0: detected conn error (1020) | 14:52 | |
| -mtanino- Apr 16 11:46:44 ubuntu tgtd: accept_connection(217) 7 | 14:52 | |
| -mtanino- Apr 16 11:46:44 ubuntu tgtd: cmnd_exec_login(694) Login request (security negotiation): 0 | 14:52 | |
| -mtanino- Apr 16 11:46:44 ubuntu tgtd: iscsi_tcp_event_handler(287) connection closed 0x19f3c40 | 14:52 | |
| -mtanino- Apr 16 11:46:44 ubuntu tgtd: conn_close(103) connection closed, 0x19f3c40 1 | 14:52 | |
| sdague | gah | 14:52 |
| sdague | ok, this might be test harness issue | 14:53 |
| mtanino | sdague: There are many messages about iscsi connection fails | 14:53 |
| *** markstur has joined #openstack-cinder | 14:54 | |
| bswartz | thingee has been AFK a lot lately ... does anyone know if he's on the road or something? | 14:55 |
| *** nkrinner has quit IRC | 14:56 | |
| *** garysmith_ has joined #openstack-cinder | 14:57 | |
| *** lpabon has joined #openstack-cinder | 14:59 | |
| ameade | bswartz: I think he may be on vacay or something | 15:01 |
| hemna | bswartz, no idea. haven't seen him much | 15:01 |
| *** annegentle has quit IRC | 15:04 | |
| *** markvoelker has joined #openstack-cinder | 15:04 | |
| sdague | mtanino: so the thing that's weird about that | 15:05 |
| *** avishay has joined #openstack-cinder | 15:06 | |
| mtanino | sdague: ?? | 15:06 |
| sdague | there is a guest running with that as it's root device | 15:06 |
| sdague | and the guest still works | 15:06 |
| *** zhenguo has quit IRC | 15:06 | |
| sdague | including pulling a file out of the home directory that I put there before tgt went offline | 15:06 |
| mtanino | hmm odd status | 15:07 |
| *** peluse has joined #openstack-cinder | 15:07 | |
| *** ronis has quit IRC | 15:07 | |
| *** Maike has quit IRC | 15:08 | |
| *** dannywilson has joined #openstack-cinder | 15:09 | |
| mtanino | Can you dd command with odirect on the home directory without using filesystem cache? | 15:10 |
| mtanino | sdague: ^ | 15:10 |
| *** markvoelker has quit IRC | 15:11 | |
| mtanino | or "touch fileabc" and "sync" several times. | 15:11 |
| *** rushiagr is now known as rushiagr_away | 15:12 | |
| *** rushil has quit IRC | 15:12 | |
| *** lpetrut has left #openstack-cinder | 15:14 | |
| *** rushil has joined #openstack-cinder | 15:15 | |
| *** rushil has quit IRC | 15:16 | |
| sdague | mtanino: I can trigger a sync, you think we're just in the fscache? | 15:18 |
| abhiram_moturi | i am having tempest failures in thirdparty for volume tests. Has anyone had that issue before? they are related to setUpClass. | 15:18 |
| *** kmartin has joined #openstack-cinder | 15:18 | |
| mtanino | sdague: http://paste.openstack.org/show/204117/ | 15:19 |
| *** dannywilson has quit IRC | 15:19 | |
| jbernard | this is a plausible theory | 15:20 |
| mtanino | sdague: From the log the instance has sda and sdb, and iscsi connection of "sda" is failed. | 15:20 |
| sdague | mtanino: yeh, that's the only volume | 15:22 |
| mtanino | seems /dev/sdb is still alive. | 15:22 |
| *** alexpilotti has joined #openstack-cinder | 15:24 | |
| *** Mandell has quit IRC | 15:25 | |
| mtanino | Ah, that log is /var/log/messages on the host. | 15:25 |
| *** Mandell has joined #openstack-cinder | 15:26 | |
| sdague | mtanino: ok, yeh, if i trigger a sync it hangs | 15:27 |
| mtanino | sdague: so there is no iSCSI connection. | 15:28 |
| sdague | yep | 15:28 |
| sdague | and that will make the test do the right thing | 15:28 |
| sdague | good call | 15:28 |
| *** annashen has joined #openstack-cinder | 15:29 | |
| sdague | mtanino: thanks for the help | 15:30 |
| mtanino | sdague: sure. But I don't understand why the iSCSI session is suddenly disconnected. | 15:31 |
| sdague | mtanino: because the devstack code is wrong, and stop_cinder stops tgt as well | 15:31 |
| mtanino | sdague: oh, really. | 15:31 |
| *** HenryG has joined #openstack-cinder | 15:31 | |
| *** russellb has quit IRC | 15:32 | |
| *** ozialien has joined #openstack-cinder | 15:32 | |
| sdague | we just never noticed before cinder was probably brought up before nova | 15:32 |
| *** reed has quit IRC | 15:33 | |
| *** reed has joined #openstack-cinder | 15:33 | |
| *** tsekiyama has joined #openstack-cinder | 15:34 | |
| mtanino | hmm | 15:34 |
| *** russellb has joined #openstack-cinder | 15:35 | |
| *** ozialien has left #openstack-cinder | 15:38 | |
| *** esker has joined #openstack-cinder | 15:39 | |
| *** avishay has quit IRC | 15:41 | |
| *** TobiasE has quit IRC | 15:44 | |
| *** annegentle has joined #openstack-cinder | 15:46 | |
| openstackgerrit | Rajini Ram proposed openstack/cinder: Fixes the retries on Network Connection Error https://review.openstack.org/171736 | 15:52 |
| *** lpetrut has joined #openstack-cinder | 15:52 | |
| *** jungleboyj has joined #openstack-cinder | 15:54 | |
| *** _cjones_ has joined #openstack-cinder | 15:55 | |
| *** russellb has quit IRC | 15:57 | |
| *** nestorf has quit IRC | 16:01 | |
| *** russellb has joined #openstack-cinder | 16:01 | |
| *** thingee has joined #openstack-cinder | 16:02 | |
| thingee | mtreinish: there is definitely a delete happening before the detach. | 16:02 |
| thingee | mtreinish: http://paste.openstack.org/show/204050/ | 16:02 |
| thingee | mtreinish: 352125fa-5920-418d-aa7c-db64669c828c | 16:02 |
| thingee | notice how that volume is detached, exported, deleted, detached | 16:02 |
| bswartz | thingee: ping | 16:03 |
| thingee | bswartz: hi | 16:03 |
| *** rongze has joined #openstack-cinder | 16:04 | |
| *** emagana has quit IRC | 16:05 | |
| *** jungleboyj has quit IRC | 16:06 | |
| *** rongze has quit IRC | 16:06 | |
| *** annashen has quit IRC | 16:06 | |
| *** rongze has joined #openstack-cinder | 16:07 | |
| mtreinish | thingee: does it do a detach right before the delete? | 16:07 |
| mtreinish | although it does look like we do a second detach after the delete (which fails) | 16:07 |
| thingee | yes | 16:07 |
| thingee | detach, export, deleted, detach | 16:08 |
| *** markvoelker has joined #openstack-cinder | 16:08 | |
| thingee | I'm only running the volume boot pattern test | 16:08 |
| *** jistr has quit IRC | 16:08 | |
| *** ociuhandu has quit IRC | 16:08 | |
| *** rongze has quit IRC | 16:09 | |
| *** adurbin_ has joined #openstack-cinder | 16:10 | |
| *** afazekas has quit IRC | 16:11 | |
| *** emagana has joined #openstack-cinder | 16:13 | |
| *** nshaikh has quit IRC | 16:13 | |
| *** Liu has quit IRC | 16:13 | |
| jgriffith | xyang1: ping | 16:13 |
| *** Liu has joined #openstack-cinder | 16:13 | |
| mtreinish | thingee: so looking at the test again I don't actually see where it would be calling detach itself. The cleanup path just looks like it has volume create/deletes, snapshot create/deletes, and server create/deletes | 16:13 |
| *** markvoelker has quit IRC | 16:13 | |
| xyang1 | jgriffith: Hi | 16:14 |
| mtreinish | and everything is done boot from volume | 16:14 |
| jgriffith | xyang1: hiya | 16:14 |
| mtreinish | thingee: so would that detach be coming from nova? | 16:14 |
| jgriffith | xyang1: I'm looking at the batch attach/detach for VNX, are you familiar with it? | 16:14 |
| *** ociuhandu has joined #openstack-cinder | 16:14 | |
| xyang1 | jgriffith: Oh, that one. Sort of | 16:14 |
| jgriffith | xyang1: LOL :) | 16:14 |
| xyang1 | jgriffith: Any concerns? | 16:15 |
| jgriffith | xyang1: there's an awful lot going on there | 16:15 |
| jgriffith | xyang1: I'm trying to *find* the responses to the queued calls | 16:15 |
| *** rushiagr_away is now known as rushiagr | 16:15 | |
| jgriffith | xyang1: also wondering if she meant to remove the rev history on purpose? | 16:15 |
| xyang1 | jgriffith: Yes, the purpose is to improve performance | 16:15 |
| *** annegentle has quit IRC | 16:15 | |
| jgriffith | xyang1: right, and it should "fix" the Ericcson case no? | 16:15 |
| xyang1 | jgriffith: yes:) | 16:16 |
| mtreinish | thingee: although tbh stack tracing on a detach because the volume can't be found is weird | 16:16 |
| jgriffith | xyang1: so it looks like it just batches and processes the attach/detach commands in their own queue | 16:16 |
| xyang1 | jgriffith: They do lots of stress test | 16:16 |
| jgriffith | xyang1: so we still won't get false responses over the RPC layer | 16:16 |
| jgriffith | xyang1: in other words we don't change status until it's processed in the batch/queue | 16:16 |
| xyang1 | jgriffith: Right, this is to avoid time out | 16:17 |
| jgriffith | xyang1: That was my only concern, but I *think* I see how the code works | 16:17 |
| jgriffith | xyang1: wanted to check with you :) | 16:17 |
| jgriffith | xyang1: and the version history stuff? | 16:17 |
| xyang1 | jgriffith: Which version history? | 16:17 |
| jgriffith | xyang1: well... that's the thing I am kinda confused about | 16:17 |
| xyang1 | Not following | 16:17 |
| jgriffith | xyang1: oh... version history: https://review.openstack.org/#/c/154350/20/cinder/volume/drivers/emc/emc_cli_fc.py | 16:18 |
| *** asselin has joined #openstack-cinder | 16:18 | |
| xyang1 | jgriffith: Let me see | 16:18 |
| jgriffith | xyang1: Intentionally removing the old version history? | 16:18 |
| xyang1 | jgriffith: Hmm, that looks like a mistake | 16:18 |
| jgriffith | xyang1: That's what I thought | 16:19 |
| xyang1 | jgriffith: No reason to erase everything | 16:19 |
| jgriffith | xyang1: so back to the change itself :) | 16:19 |
| xyang1 | jgriffith: Looks like a rebase error | 16:19 |
| jgriffith | xyang1: so how does this "fix" the issue? | 16:19 |
| xyang1 | jgriffith: This only delays time out from happening | 16:20 |
| xyang1 | jgriffith: Because once time out happens, it can leave faulty device | 16:20 |
| jgriffith | xyang1: oh... so the idea is to continue processing the queue on the VNX even if one of them times out? | 16:20 |
| xyang1 | jgriffith: If another rescan happens in between | 16:21 |
| xyang1 | jgriffith: Right | 16:21 |
| jgriffith | xyang1: that makes sense | 16:22 |
| xyang1 | jgriffith: To reduce the change of getting timeout | 16:22 |
| xyang1 | Chance | 16:22 |
| jgriffith | xyang1: sorry... it's kinda hard to follow for me | 16:22 |
| xyang1 | jgriffith: It is confusing:) | 16:22 |
| *** cbader has quit IRC | 16:22 | |
| jgriffith | xyang1: and I got confused looking at all the new hlu methods | 16:23 |
| xyang1 | jgriffith: It is a very big patch | 16:23 |
| jgriffith | xyang1: :) | 16:24 |
| xyang1 | jgriffith: I'll need to take a look again to refresh my memory as well:) | 16:24 |
| jgriffith | xyang1: ok, so I gave a -1 just to clear up the quesiton on version history | 16:24 |
| xyang1 | jgriffith: Sure. Thanks for reviewing! | 16:25 |
| jgriffith | xyang1: I *think* the patch itself is "ok" withyour help in explaining it :) | 16:25 |
| xyang1 | jgriffith: Did I really explain it?:) | 16:25 |
| xyang1 | jgriffith: Glad I can help | 16:25 |
| jgriffith | xyang1: LOL | 16:25 |
| xyang1 | jgriffith: About replication V2 spec, did you see my comments on CG integration? Any thoughts? | 16:26 |
| xyang1 | jgriffith: Or is it out of control?:) | 16:28 |
| jgriffith | xyang1: I haven't seen your comments, lemme look | 16:28 |
| *** krtaylor has quit IRC | 16:28 | |
| xyang1 | jgriffith: Ok | 16:28 |
| *** annashen has joined #openstack-cinder | 16:28 | |
| *** leeantho has joined #openstack-cinder | 16:29 | |
| jgriffith | xyang1: interesting... yeah | 16:29 |
| jgriffith | xyang1: that is useful I think, but I hadn't thought of it | 16:29 |
| jgriffith | xyang1: but wouldn't that just be up to the driver? | 16:29 |
| thingee | mtreinish: yeah I'm thinking it's coming from nova | 16:29 |
| xyang1 | jgriffith: It seems that we need to add a few more new CG APIs to support both | 16:29 |
| jgriffith | xyang1: so you say "replicate volume 'foo'" | 16:30 |
| thingee | mtreinish: I've glanced through logs and didn't see anything new that would be obvious | 16:30 |
| *** sgotliv_ has quit IRC | 16:30 | |
| jgriffith | xyang1: driver looks and and says "oh... this is in a CG, so replicate all of them" | 16:30 |
| thingee | mtreinish: I'll file a bug and keep looking. | 16:30 |
| *** mriedem has quit IRC | 16:30 | |
| xyang1 | jgriffith: Then you need to put them in group earlier | 16:30 |
| thingee | against nova and tempest until I can verify what's doing it | 16:31 |
| xyang1 | jgriffith: Otherwise how do you tell | 16:31 |
| xyang1 | jgriffith: I mean when we create a CG, we choose 1 backend that supports all volume types, so this should apply to cg for replication? | 16:32 |
| *** annegentle has joined #openstack-cinder | 16:32 | |
| *** mriedem has joined #openstack-cinder | 16:32 | |
| *** mwichmann has joined #openstack-cinder | 16:33 | |
| marcusvrn1 | mtreinish: hi | 16:34 |
| *** dulek has quit IRC | 16:34 | |
| *** c0m0 has quit IRC | 16:35 | |
| *** alecv has quit IRC | 16:35 | |
| *** ronis has joined #openstack-cinder | 16:36 | |
| xyang1 | jgriffith: I think what you have proposed are still needed for volume. If we can also handle the case when a CG id is provided at volume creation time, then it shoud work for CG as well | 16:38 |
| *** markus_z has quit IRC | 16:38 | |
| *** lpabon has quit IRC | 16:39 | |
| *** EmilienM is now known as EmilienM|afk | 16:39 | |
| thingee | mtreinish: instead of stacktrace, are you saying it should just move along? | 16:44 |
| *** patrickeast has joined #openstack-cinder | 16:44 | |
| mtreinish | thingee: yeah, log that it wasn't found, but that's not something unexpected | 16:45 |
| mtreinish | it just should be caught because a not found is expected | 16:46 |
| thingee | mtreinish: wonder if that would have any bad side effects from the api that uses that raise for status code return. | 16:46 |
| thingee | mtreinish: but I agree | 16:46 |
| *** afazekas has joined #openstack-cinder | 16:47 | |
| mtreinish | thingee: well you can still use an exception but it should just be raised by the rpc response from c-vol in c-api, not unhandled in c-vol | 16:47 |
| mtreinish | marcusvrn1: hi | 16:47 |
| *** jordanP has quit IRC | 16:47 | |
| *** Apoorva has joined #openstack-cinder | 16:50 | |
| *** sks has joined #openstack-cinder | 16:50 | |
| marcusvrn1 | mtreinish: we were discussing a few days ago, with thingee, about ec2 tests on CI...and I saw that one tempest patch was posted to remove those tests...but I don't know why, sometimes our CI runs ec2 tests and sometimes it doesn't (e.g http://177.84.241.119:10000/44/174044/1/check/hitachi-hbsd-fc/ab4d79e/console.html vs http://177.84.241.119:10000/03/151303/4/check/hitachi-hbsd-fc/4c71029/console.html) | 16:52 |
| *** krtaylor has joined #openstack-cinder | 16:53 | |
| mtreinish | marcusvrn1: so the tempest config files are different | 16:53 |
| mtreinish | in the second one the config file says ec2 is disabled | 16:54 |
| mtreinish | that's because its running on a stable juno patch | 16:54 |
| mtreinish | we disable the ec2 tests on stable because it doesnt work | 16:54 |
| mtreinish | there was a fix with a new boto which never was backported | 16:54 |
| *** annashen has quit IRC | 16:54 | |
| mtreinish | so instead of worrying about it we just changed devstack to tell tempest that ec2 wasn't available | 16:54 |
| *** russellb has quit IRC | 16:55 | |
| *** crose has joined #openstack-cinder | 16:55 | |
| *** marcusvrn1 has quit IRC | 16:55 | |
| *** emagana has quit IRC | 16:56 | |
| *** Adriano_ has joined #openstack-cinder | 16:57 | |
| *** bswartz has quit IRC | 16:58 | |
| *** marcusvrn has joined #openstack-cinder | 16:58 | |
| marcusvrn | mtreinish: I want to disable it for all CI jobs... is it ok? | 16:58 |
| *** harlowja_away is now known as harlowja | 16:58 | |
| *** Yogi11 has joined #openstack-cinder | 16:58 | |
| mtreinish | marcusvrn: that's your call, I can't make a judgement there it's your system | 16:58 |
| mtreinish | (fwiw I really want to kill the ec2 tests in tempest) | 16:59 |
| mtreinish | thingee: about the detach if it is from nova, it could be that nova is in a loop trying detach multiple times after a timeout or something and understands a 404 as job done | 16:59 |
| *** mtanino has quit IRC | 16:59 | |
| mtreinish | I'd have to look at the nova code to figure it out though | 16:59 |
| patrickeast | we’ve run into some issues with intermittent failures on those boto tests, i’ve been thinking about disabling them too… not sure that it really buys us any particular benefit with regards to testing our driver compared with the other volume tests | 17:00 |
| *** russellb has joined #openstack-cinder | 17:00 | |
| *** IanGovett1 has quit IRC | 17:00 | |
| patrickeast | i’m looking to see if its just something we have poorly configured in oure devstacks though before turning them off | 17:00 |
| patrickeast | our* | 17:00 |
| *** annashen has joined #openstack-cinder | 17:01 | |
| marcusvrn | mtreinish: I see. thanks :) (I want to kill those ec2 tests too lol) | 17:01 |
| mtreinish | thingee: I'll start digging more after I get some food | 17:01 |
| *** Yogi1 has quit IRC | 17:02 | |
| mtreinish | patrickeast: yeah, you just have to be careful with just skipping things (even with intermittent fails) because it could be exposing a real problem with something | 17:02 |
| *** IanGovett has joined #openstack-cinder | 17:02 | |
| patrickeast | mtreinish: yea definitely | 17:02 |
| jgriffith | xyang1: sorry... got pulled away | 17:05 |
| jgriffith | xyang1: so yes, that would mean you have to know it's a CG prior to replication | 17:05 |
| jgriffith | xyang1: honestly it's been my opinion that requiring CG and Replication knowledge at create time is "ok" | 17:05 |
| jgriffith | xyang1: but I know that's not going to fly | 17:06 |
| *** emagana has joined #openstack-cinder | 17:06 | |
| *** dannywilson has joined #openstack-cinder | 17:06 | |
| jgriffith | xyang1: but keep in mind there's retype | 17:06 |
| jgriffith | xyang1: which solves at least "some" of that | 17:06 |
| xyang1 | jgriffith: You are saying only let it happen at volume create time? Not create a CG ahead of time? | 17:07 |
| *** dannywilson has quit IRC | 17:08 | |
| jgriffith | xyang1: I'm saying I'm not saying anything yet :) | 17:08 |
| xyang1 | jgriffith: Ok:) | 17:08 |
| jgriffith | xyang1: LOL | 17:08 |
| asselin | patrickeast, marcusvrn we disabled those tests...haven't gotten them to work yet | 17:08 |
| jgriffith | xyang1: IMO I think types work well for these sorts of things | 17:09 |
| jgriffith | xyang1: I'm not necessarily convinced that new API's and moving away from those was a good idea | 17:09 |
| *** dannywilson has joined #openstack-cinder | 17:09 | |
| *** annegentle has quit IRC | 17:09 | |
| jgriffith | xyang1: at the same time though, thingee has ran on the platform of putting more backend management in Cinder | 17:09 |
| *** dannywilson has quit IRC | 17:09 | |
| jgriffith | xyang1: so he obviously has very different vision than I do | 17:09 |
| *** dannywilson has joined #openstack-cinder | 17:10 | |
| jgriffith | xyang1: and I think you, hemna and others did as well | 17:10 |
| *** rongze has joined #openstack-cinder | 17:10 | |
| jgriffith | ^^ have a different view than me | 17:10 |
| *** dannywilson has quit IRC | 17:10 | |
| xyang1 | jgriffith: :) | 17:10 |
| *** gcivitella has quit IRC | 17:10 | |
| *** dannywilson has joined #openstack-cinder | 17:10 | |
| *** dannywilson has quit IRC | 17:10 | |
| jgriffith | xyang1: honestly, based on all the notes in the review I have up I'm not sure there's any way to make everybody happy here | 17:10 |
| jgriffith | xyang1: we've become a pretty divisive group | 17:11 |
| xyang1 | jgriffith: The spec looks good, I just want to make sure we think about cg as well | 17:11 |
| jgriffith | xyang1: Yeah, that's an important piece and you're right I didn't talk about it at all | 17:12 |
| jgriffith | xyang1: I'm just thinking..... | 17:12 |
| *** markvoelker has joined #openstack-cinder | 17:12 | |
| jgriffith | xyang1: Trying to think if doing something like retypes to get what you want would work | 17:12 |
| jgriffith | xyang1: Because of the whole knowing if it's a CG at creation or not etc | 17:13 |
| jgriffith | xyang1: I kinda like pushing the "how" back to the driver | 17:13 |
| xyang1 | jgriffith: You mean retype to make all volumes in one type? | 17:13 |
| *** tshefi has quit IRC | 17:13 | |
| jgriffith | xyang1: Like saying... if you say "replicate" this volume and it's a CG it's up to the driver to figure out how to do it | 17:13 |
| jgriffith | xyang1: "retype to make all volumes in one type" might be one approach | 17:14 |
| xyang1 | jgriffith: You are saying hiding that from cinder? | 17:14 |
| jgriffith | xyang1: so you have Type=CG with N-volumes | 17:14 |
| jgriffith | xyang1: and you want to replicate the CG... | 17:14 |
| jgriffith | xyang1: then you'd need to retype to "Type=CG-Replicated" | 17:14 |
| *** annegentle has joined #openstack-cinder | 17:14 | |
| jgriffith | xyang1: I'd like to hide more of it from Cinder than we do currently yes | 17:15 |
| jgriffith | xyang1: because the deltas between the platforms makes things a bit difficult IMO | 17:15 |
| jgriffith | xyang1: well... not "difficult" but kind of impossible | 17:15 |
| jgriffith | xyang1: and very difficult to maintain/read | 17:15 |
| xyang1 | jgriffith: Then it won't be integrated with the existing CG implementation | 17:15 |
| jgriffith | xyang1: That's the problem | 17:16 |
| jgriffith | xyang1: I'd like ot figure out how to keep the CG integration that's there and make this idea work with it | 17:16 |
| jgriffith | I just haven't come up with that yet :) | 17:16 |
| xyang1 | jgriffith: If we take cg id at create volume time, that should solve the problem | 17:17 |
| jgriffith | xyang1: Oh... duhhh, yeah | 17:17 |
| xyang1 | jgriffith: Then maybe we don't need cg level enable etc | 17:18 |
| thingee | eharney: you mind signing off? https://review.openstack.org/#/c/169858/13 | 17:18 |
| xyang1 | jgriffith: I can think about that | 17:18 |
| jgriffith | xyang1: so that would be really cool | 17:18 |
| xyang1 | jgriffith: If we take care of it at create volume time, may be depends on whether it is in group or not, we will do the right thing with one api | 17:19 |
| *** IanGovett1 has joined #openstack-cinder | 17:19 | |
| xyang1 | jgriffith: Volume level api to enable, disable, failover | 17:20 |
| xyang1 | jgriffith: Ok, let me think more about it | 17:20 |
| jgriffith | xyang1: that might work, and might be fairly clean | 17:20 |
| xyang1 | jgriffith: Sure | 17:20 |
| *** IanGovett has quit IRC | 17:21 | |
| *** IanGovett has joined #openstack-cinder | 17:23 | |
| *** IanGovett1 has quit IRC | 17:24 | |
| *** Maike has joined #openstack-cinder | 17:25 | |
| *** garysmith_ has quit IRC | 17:25 | |
| *** rongze has quit IRC | 17:26 | |
| *** fanyaohong has quit IRC | 17:29 | |
| *** bkopilov has quit IRC | 17:31 | |
| *** j_king_ is now known as j_king | 17:31 | |
| *** ronis_ has joined #openstack-cinder | 17:32 | |
| *** bkopilov has joined #openstack-cinder | 17:32 | |
| *** ronis has quit IRC | 17:35 | |
| eharney | thingee: looking | 17:35 |
| *** bkopilov has quit IRC | 17:37 | |
| *** bkopilov has joined #openstack-cinder | 17:40 | |
| *** markvoelker has quit IRC | 17:43 | |
| openstackgerrit | Walter A. Boring IV (hemna) proposed openstack/os-brick: Mask passwords with iscsiadm commands https://review.openstack.org/174484 | 17:43 |
| *** EmilienM|afk is now known as EmilienM | 17:43 | |
| openstackgerrit | Walter A. Boring IV (hemna) proposed openstack/cinder: Mask passwords with iscsiadm commands https://review.openstack.org/174485 | 17:44 |
| *** ociuhandu has quit IRC | 17:45 | |
| *** bkopilov has quit IRC | 17:45 | |
| *** btran has joined #openstack-cinder | 17:46 | |
| *** bkopilov has joined #openstack-cinder | 17:46 | |
| thingee | hemna: durr thanks | 17:48 |
| hemna | np | 17:48 |
| *** alexpilotti has quit IRC | 17:48 | |
| *** Yogi11 has quit IRC | 17:49 | |
| openstackgerrit | Walter A. Boring IV (hemna) proposed openstack/os-brick: Brick: Fix race in removing iSCSI device https://review.openstack.org/168506 | 17:52 |
| *** rongze has joined #openstack-cinder | 17:52 | |
| *** bkopilov has quit IRC | 17:54 | |
| *** rongze_ has joined #openstack-cinder | 17:54 | |
| *** rongze has quit IRC | 17:55 | |
| *** EmilienM is now known as EmilienM|afk | 17:55 | |
| *** aix has quit IRC | 17:57 | |
| *** afazekas has quit IRC | 17:59 | |
| *** rongze has joined #openstack-cinder | 18:00 | |
| *** Yogi1 has joined #openstack-cinder | 18:00 | |
| *** rongze_ has quit IRC | 18:00 | |
| *** jaypipes has quit IRC | 18:01 | |
| *** jaypipes has joined #openstack-cinder | 18:01 | |
| *** Maike has quit IRC | 18:03 | |
| *** Apoorva_ has joined #openstack-cinder | 18:06 | |
| *** Apoorva_ has quit IRC | 18:08 | |
| *** Apoorva has quit IRC | 18:09 | |
| *** Apoorva_ has joined #openstack-cinder | 18:09 | |
| *** rongze has quit IRC | 18:09 | |
| *** dulek has joined #openstack-cinder | 18:12 | |
| openstackgerrit | Doug Hellmann proposed openstack/python-cinderclient: Uncap library requirements for liberty https://review.openstack.org/174529 | 18:12 |
| *** lpabon has joined #openstack-cinder | 18:16 | |
| *** lpabon has quit IRC | 18:16 | |
| *** crose has quit IRC | 18:21 | |
| thingee | mtreinish, jgriffith: continuing http://paste.openstack.org/show/204050/ from yesterday ... so terminate connection is deleting the volume and then later the volumebootpattern is detaching the volume that was deleted. | 18:21 |
| *** Maike has joined #openstack-cinder | 18:22 | |
| hemna | new star wars trailer | 18:22 |
| hemna | so epic!! | 18:22 |
| mtreinish | thingee: I couldn't find anywhere in the test was calling attach or detach (there was a method to do detach, but it didn't look like it was ever called) | 18:22 |
| mtreinish | thingee: do you have the api logs and tempest logs showing tempest making the detach? | 18:25 |
| thingee | mtreinish: pretty sure it's on the nova side at this point | 18:27 |
| *** vilobhmm1 has joined #openstack-cinder | 18:27 | |
| *** vilobhmm11 has joined #openstack-cinder | 18:29 | |
| *** bswartz has joined #openstack-cinder | 18:30 | |
| vilobhmm11 | thingee, DuncanT : added previous seen issues and patches proposed that i know of to the etherpad please have a look https://etherpad.openstack.org/p/cinder-active-active-vol-service-issues | 18:32 |
| *** vilobhmm1 has quit IRC | 18:32 | |
| *** primechuck has quit IRC | 18:32 | |
| vilobhmm11 | as we discussed in yesterday's meeting ^^ | 18:32 |
| *** EmilienM|afk is now known as EmilienM | 18:33 | |
| openstackgerrit | Angela Smith proposed openstack/cinder-specs: Spec for Brocade Zone Driver Virtual Fabrics support https://review.openstack.org/144389 | 18:35 |
| *** primechuck has joined #openstack-cinder | 18:35 | |
| *** markvoelker has joined #openstack-cinder | 18:42 | |
| *** sks has quit IRC | 18:44 | |
| *** Liu has quit IRC | 18:46 | |
| *** Liu has joined #openstack-cinder | 18:46 | |
| openstackgerrit | Angela Smith proposed openstack/cinder-specs: Spec for Brocade Zone Driver Virtual Fabrics support https://review.openstack.org/144389 | 18:47 |
| openstackgerrit | Angela Smith proposed openstack/cinder-specs: Spec for Brocade Zone Driver Friendly Zone names https://review.openstack.org/144384 | 18:48 |
| *** Maike has quit IRC | 18:50 | |
| *** crose has joined #openstack-cinder | 18:50 | |
| mtreinish | jgriffith: so I'm going to bug you again how goes the cli test migration? | 18:51 |
| jgriffith | mtreinish: :) | 18:58 |
| jgriffith | mtreinish: thanks for the reminder | 18:58 |
| jgriffith | mtreinish: I need to get that back up on my list | 18:58 |
| jgriffith | mtreinish: I honestly haven't done anything more than submit the base change to cinderclient | 18:59 |
| mtreinish | well at least you're doing it. I cant get anyone from glance, heat, or swift to even start the process | 18:59 |
| jgriffith | mtreinish: and that has been sitting in review for *ever* | 18:59 |
| jgriffith | mtreinish: well... half of it | 18:59 |
| *** ozialien has joined #openstack-cinder | 18:59 | |
| *** jungleboyj has joined #openstack-cinder | 18:59 | |
| jgriffith | mtreinish: :( | 19:00 |
| jgriffith | mtreinish: so I'lll carve out some time here in the next week or so and take another hard run at it | 19:00 |
| jgriffith | I'd like to have some stuff in place for Vancouver | 19:00 |
| jgriffith | mtreinish: so we can chat in person :) | 19:00 |
| mtreinish | thingee: ^^^ can you put your elected powers to work and grease the the wheels? | 19:00 |
| mtreinish | jgriffith: cool thanks | 19:00 |
| mtreinish | thingee: lol, the review wheels, not the jgriffith wheel :) | 19:01 |
| mtreinish | jgriffith: I probably wouldn't snub you in Vancouver if we didn't get this in place :) | 19:01 |
| openstackgerrit | Angela Smith proposed openstack/cinder-specs: Spec for Brocade Zone Driver Friendly Zone names https://review.openstack.org/144384 | 19:02 |
| *** cbader has joined #openstack-cinder | 19:04 | |
| *** rongze has joined #openstack-cinder | 19:06 | |
| *** rushiagr is now known as rushiagr_away | 19:08 | |
| *** rongze has quit IRC | 19:10 | |
| *** markvoelker has quit IRC | 19:14 | |
| *** Mandell has quit IRC | 19:19 | |
| *** Yogi11 has joined #openstack-cinder | 19:23 | |
| *** jungleboyj has quit IRC | 19:24 | |
| *** Mandell has joined #openstack-cinder | 19:25 | |
| thingee | mtreinish: ok not delete on termination. Nova never issues the delete volume it seems. I definitely see tempest log with a delete request to cinder | 19:25 |
| *** Yogi1 has quit IRC | 19:26 | |
| mtreinish | thingee: yeah that's consistent with my reading of the tempest code. When tempest calls it's create_volume() it registers a cleanup for the volume delete | 19:27 |
| mtreinish | fwiw, it also does the same with server create, so it'll call server delete and wait for that before calling volume delete (because cleanup is a lifo) | 19:28 |
| mtreinish | which is why tempest doesnt ever issue a detach happens because the servers are all boot from volume for that test | 19:29 |
| thingee | pretty sure there is something out of order with tempest doing the delete and then a detach call happening :( | 19:30 |
| thingee | but I'm not sure why it's not happening in gate, but just to some drivers. | 19:30 |
| *** annashen has quit IRC | 19:32 | |
| tbarron | Is there a way to tell taskflow to only attempt a volume create one time when a single backend is configured? | 19:32 |
| tbarron | I'm not seeing a cinder.conf setting for this right off. | 19:33 |
| mtreinish | thingee: well how does nova handle a delete with a boot from volume server? | 19:33 |
| thingee | only if the block device mapping has a delete on terminate. I have a breakpoint on the nova.volume.cinder.delete and it never gets hit. also tempest is explicitly setting delete on terminate to false | 19:34 |
| thingee | I specifically see from tempest_lib a delete call to cinder | 19:34 |
| mtreinish | thingee: also is this on the first volume or the volume from the snapshot? | 19:35 |
| tbarron | I ask in part because we have automated tests that are apparently triggering a race condition between taskflow volume create and volume delete, where the latter is triggered by polling for the volume to got to error or available. | 19:35 |
| *** dustins has quit IRC | 19:35 | |
| mtreinish | thingee: ok, does nova not issue a detach before deleting the server from volume. (sorry I'm ignorant of that code path) | 19:35 |
| tbarron | taskflow appears to set volume state from creating to error to creating to error etc. on negative tests (where the crteate is supposed to fail). | 19:36 |
| mtreinish | thingee: i get that tempest is calling the volume delete after the server is deleted, I'm just trying to figure out where that detach is coming from | 19:36 |
| thingee | mtreinish: correct detach is after. | 19:36 |
| tbarron | our test cleanup code can see the error state and attempt to delete the volume while taskflow is still doing retries. | 19:36 |
| *** annegentle has quit IRC | 19:36 | |
| tbarron | http://paste.openstack.org/show/204136/ | 19:37 |
| mtreinish | thingee: so nova deletes the vm and then detaches the volume? That seems backwards to me | 19:37 |
| thingee | whoops sorry before | 19:38 |
| thingee | https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L2601 | 19:38 |
| tbarron | at 015-04-16 12:54:33.550 we see taskflow setting state to error | 19:38 |
| *** dustins has joined #openstack-cinder | 19:39 | |
| thingee | mtreinish: and that comes from https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L2661 | 19:39 |
| tbarron | At 2015-04-16 12:54:33.638 we see the manager trying to do a delete, before taskflow has kicked off the next create attempt | 19:39 |
| mtreinish | thingee: ah ok that makes sense. So the volume isn't attached when tempest goes to delete the volume. So I wonder where that later detach call is coming from? | 19:40 |
| mtreinish | thingee: do you have a req-id from the second detach (after the volume is created) that you can search for in the tempest logs? Or the timestamp to check nova's logs | 19:41 |
| mtreinish | (since nova doesn't log the req-id from cinder requests) | 19:41 |
| tbarron | The delete is running inside an oslo concurrency lock, but that seems to not exclude taskflow create from doing DB updates. | 19:42 |
| thingee | mtreinish: hang on... | 19:42 |
| tbarron | harlowja: ^^^^^^ taskflow cinder create questions above | 19:44 |
| harlowja | sup | 19:44 |
| thingee | mtreinish: pretty sure they're all coming from nova | 19:44 |
| thingee | detaches that is | 19:45 |
| *** timcl has quit IRC | 19:45 | |
| tbarron | harlowja: can you read the last 5 minutes of buffer here, questions about an apparent race between taskflow volume create and volume delete on the same volume? | 19:45 |
| -openstackstatus- NOTICE: gerrit has been restarted to clear a problem with its event stream. any gerrit changes updated or approved between 19:14 and 19:46 utc will need to be rechecked or have their approval reapplied for zuul to pick them up | 19:46 | |
| mtreinish | ok, that's what I was expecting. I guess now that just means we need to trace through nova to figure out why it's sending a second detach after the server and volume have already been deleted | 19:46 |
| mtreinish | and why we never hit that in the gate (or with other backends) | 19:46 |
| tbarron | harlowja: here's a better pastebin, the first one didn't show manager delete_volume gaining the oslo concurrency lock: | 19:47 |
| tbarron | http://paste.openstack.org/show/204139/ | 19:47 |
| thingee | mtreinish: maybe because the instance we're deleting has a bdm to that volume still? | 19:47 |
| thingee | that was deleted | 19:47 |
| *** julim has quit IRC | 19:48 | |
| thingee | mtreinish: just tracing through https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L4834 | 19:50 |
| harlowja | tbarron ok, so request comes in to delete, seems like it gets the lock 'lvo_inner2' create is still running, and reverting right , i don't see delete every entering the RUNNING state though, guess thats cause delete isn't using taskflow | 19:52 |
| tbarron | harlowja: right, delete isn't using taskflow. | 19:53 |
| tbarron | this delete is crapping out right away b/c it finds the volume in an unxpected state (no host string, but it | 19:53 |
| *** bkopilov has joined #openstack-cinder | 19:53 | |
| tbarron | it's trying to parse it). | 19:53 |
| mtreinish | thingee: yeah that should return a 404 right? because the volume was deleted | 19:54 |
| tbarron | I left that part out of the log and description because the more fundamental issue to me is that we allow the delete in while taskflow is still retrying | 19:54 |
| harlowja | tbarron what lock is the create getting, seems like it could get a few @ https://github.com/openstack/cinder/blob/master/cinder/volume/manager.py#L437 | 19:54 |
| mtreinish | thingee: err or just a NotFound | 19:54 |
| mtreinish | thingee: I think I'm missing something | 19:55 |
| harlowja | tbarron and locked_volume_operation seems to get some other kind of locks | 19:55 |
| harlowja | sooo how's that work :-/ | 19:55 |
| tbarron | not sure | 19:55 |
| harlowja | ha | 19:55 |
| harlowja | seems like they use different locks, so maybe thats the problem | 19:56 |
| tbarron | harlowja: does the create in taskflow hold the same lock through all three attempts to do the volume create? | 19:56 |
| harlowja | tbarron the lock isn't in the taskflow code, so yes | 19:57 |
| harlowja | https://github.com/openstack/cinder/blob/master/cinder/volume/manager.py#L458 | 19:57 |
| *** dustins has quit IRC | 19:57 | |
| tbarron | even though it is setting the DB state back to error at the end of the create? | 19:57 |
| harlowja | utils.synchronized(locked_action, external=True) is doing all that | 19:57 |
| harlowja | the lock is around the running of the taskflow 'flow' | 19:57 |
| harlowja | outside of taskflow | 19:57 |
| tbarron | harlowja: I'm a complete newbie relative to taskflow. | 19:58 |
| harlowja | np | 19:58 |
| harlowja | here to help! | 19:58 |
| harlowja | so that code is locking, then letting taskflow run the workflow, and then release, so some lock is held during that entire process | 19:59 |
| tbarron | harlowja: but you are saying if I understand you that we should be setting up mutual exclusion between create and delete using locks instead of, e.g. checks on DB state, through the entire 3-attempt create lifecycle. | 19:59 |
| harlowja | that seems to be intention, not sure the reality :) | 20:00 |
| tbarron | We appear instead - though I can't say I know every corner of the code - to rely on the volume state being available or error to see if it's safe to do a delete. | 20:00 |
| tbarron | And you set it back to error. | 20:01 |
| *** annegentle has joined #openstack-cinder | 20:01 | |
| harlowja | well it makes me wonder why delete and create are running at the same time (like that pastebin shows) | 20:02 |
| harlowja | they shouldn't be able to | 20:02 |
| tbarron | # NOTE(harlowja): this is awkward to be done here, shouldn't | 20:02 |
| tbarron | # this happen at the scheduler itself | 20:02 |
| tbarron | harlowja: you do note: ^^^ | 20:02 |
| harlowja | ya, thats the general rescheduling loop junk | 20:02 |
| * harlowja can't fix all the things, so i just leave comments about weirdness i saw when refactoring that, lol | 20:03 | |
| tbarron | harlowja: is there a way to configure taskflow to only do one attempt? That will give us a workaround for our automated testing while the more general locking/exclusion issue is sorted out. | 20:03 |
| harlowja | so taskflow isn't really involved in that either | 20:03 |
| harlowja | its the cinder rescheduling loop | 20:04 |
| harlowja | there is a way afaik | 20:04 |
| harlowja | https://github.com/openstack/cinder/blob/master/cinder/scheduler/filter_scheduler.py#L194 | 20:04 |
| harlowja | tbarron ^ is really what is doing this, taskflow isn't really involved (another weirdness that i can't change, haha) | 20:05 |
| tbarron | harlowja: I see! thanks. | 20:05 |
| harlowja | someone needs to write a blog post about that stuff, haha | 20:05 |
| harlowja | basically its a weird big loop (that i don't like...) | 20:05 |
| harlowja | i'd rather have the scheudler (or something else) manage that entire workflow | 20:06 |
| tbarron | harlowja: sorry to be lumping all these external-to-taskflow issues under the taskflow rubric | 20:06 |
| harlowja | instead of sending things around loops via the message-queue | 20:06 |
| harlowja | np | 20:06 |
| harlowja | tbarron its a common misunderstand (nobody seems to know that rescheduling magic loop) | 20:06 |
| harlowja | *misunderstanding | 20:06 |
| tbarron | gotta say though that there are so may log messages pertaining to this area that run from taskflow that it's an easy misstake to make. | 20:06 |
| harlowja | *magic loop is same in nova (and cinder cloned from nova...) | 20:07 |
| *** rongze has joined #openstack-cinder | 20:07 | |
| tbarron | harlowja: I hate magic. | 20:07 |
| tbarron | drives me crazy. I don't want to be fooled. | 20:07 |
| tbarron | harlowja: thanks for your help. I'm going to set CONF.max_scheduler_attempts = 1 for now. | 20:09 |
| tbarron | That said, I think I'm not the only one to have confusion about the mix of taskflow (micro) state transitions and cinder | 20:10 |
| tbarron | object state transitions, esp. exclusion relations when doing updates on the latter. | 20:10 |
| *** annegentle has quit IRC | 20:10 | |
| harlowja | i know | 20:11 |
| harlowja | tis life :-P | 20:11 |
| *** annegentle has joined #openstack-cinder | 20:11 | |
| harlowja | the magic loop though pre-exists me, but i underestand the misunderstanding | 20:11 |
| harlowja | its common | 20:11 |
| harlowja | bb | 20:11 |
| tbarron | ty | 20:11 |
| *** rongze has quit IRC | 20:11 | |
| *** markvoelker has joined #openstack-cinder | 20:12 | |
| openstackgerrit | xing-yang proposed openstack/cinder: Fixed issue with mismatched config in VMAX driver https://review.openstack.org/174044 | 20:14 |
| *** xyang has quit IRC | 20:14 | |
| *** jwcroppe has quit IRC | 20:20 | |
| *** annashen has joined #openstack-cinder | 20:23 | |
| *** esker has quit IRC | 20:23 | |
| *** emagana has quit IRC | 20:24 | |
| *** Apoorva has joined #openstack-cinder | 20:26 | |
| *** Apoorva_ has quit IRC | 20:29 | |
| *** esker has joined #openstack-cinder | 20:33 | |
| *** ozialien has quit IRC | 20:38 | |
| *** markvoelker has quit IRC | 20:44 | |
| *** akerr has quit IRC | 20:45 | |
| *** emagana has joined #openstack-cinder | 20:46 | |
| *** annegentle has quit IRC | 20:46 | |
| *** peluse has quit IRC | 20:52 | |
| *** thangp has quit IRC | 20:52 | |
| *** dannywilson has joined #openstack-cinder | 20:54 | |
| jgriffith | mtreinish: sureee... you say that now :) | 20:55 |
| mtreinish | jgriffith: heh, I did put "probably" in there :) | 20:56 |
| thingee | hemna: ping | 20:56 |
| *** cburgess_ has quit IRC | 20:58 | |
| *** cburgess has joined #openstack-cinder | 20:58 | |
| thingee | mtreinish: here's the trace of the last detach http://paste.openstack.org/show/204144/ | 21:00 |
| hemna | yough | 21:00 |
| thingee | hemna, mtreinish: so I think nova isn't expecting InvalidVolume to be raised from cinder in detach. | 21:01 |
| *** dannywil_ has joined #openstack-cinder | 21:01 | |
| thingee | hemna, mtreinish: this changed with the multi-attach stuff | 21:01 |
| mtreinish | thingee: oh, i could see that | 21:02 |
| *** patrickeast_ has joined #openstack-cinder | 21:02 | |
| *** annashen has quit IRC | 21:02 | |
| thingee | mtreinish: and I think nova used to just ignore the detail failing | 21:02 |
| *** patrickeast has quit IRC | 21:02 | |
| *** patrickeast_ is now known as patrickeast | 21:02 | |
| *** dannywilson has quit IRC | 21:03 | |
| mtreinish | thingee: so you think we should just catch that now in nova and just move on | 21:03 |
| thingee | mtreinish: https://github.com/openstack/cinder/commit/10d54216871fde27172fbcb1a3c9bfec59b824c2#diff-724c53b8cd580ae533b4fb301df7c3e9R822 | 21:03 |
| *** crose has quit IRC | 21:04 | |
| *** Yogi11 has quit IRC | 21:04 | |
| thingee | mtreinish, hemna https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L2601 | 21:05 |
| *** bill_az has joined #openstack-cinder | 21:05 | |
| thingee | mtreinish, hemna: at least according to this trace http://paste.openstack.org/show/204144/ | 21:05 |
| *** dulek has quit IRC | 21:05 | |
| *** ronis_ has quit IRC | 21:06 | |
| mtreinish | thingee: oh it does check for volume not found | 21:06 |
| hemna | that's a trace ? | 21:06 |
| mtreinish | thingee: what does InvalidVolume get converted into in cinderclient? | 21:06 |
| * hemna is confused | 21:06 | |
| thingee | 400 | 21:06 |
| hemna | is there a defect somewhere? | 21:07 |
| thingee | hemna: yes, multi-attach change is raising something nova isn't expecting | 21:07 |
| thingee | see my last few messages for the trace and code from nova/cinder sides | 21:07 |
| mtreinish | thingee: yeah that's definitely a behavior change then | 21:07 |
| *** rongze has joined #openstack-cinder | 21:08 | |
| mtreinish | it would be pretty easy to either change cinder back, or have nova understand a 400 in that except block | 21:08 |
| *** bswartz has quit IRC | 21:08 | |
| hemna | launchpad url ? | 21:09 |
| thingee | hemna: none yet, just discovered | 21:09 |
| hemna | ok well why are we getting an invalid volume? | 21:10 |
| hemna | is the uuid bogus being passed to cinder ? | 21:10 |
| thingee | hemna: dunno, I've been working most yesterday and today with mtreinish tracing down this issue. | 21:11 |
| hemna | logfile with the exception ? | 21:11 |
| *** harlowja is now known as harlowja_away | 21:11 | |
| *** emagana has quit IRC | 21:12 | |
| mtreinish | hemna: it's not bogus, it's just already been deleted | 21:12 |
| hemna | why ? | 21:12 |
| *** rongze has quit IRC | 21:13 | |
| hemna | why is nova calling cinder to detach a deleted volume ? | 21:13 |
| *** EmilienM is now known as EmilienM|afk | 21:13 | |
| hemna | and why is cinder allowing a volume to be deleted if it's attached ? | 21:13 |
| thingee | hemna: I think it's because nova has it in the bdm, but tempest is deleting it. | 21:13 |
| thingee | hemna: the point is the nova just isn't expecting invalidvolume to be raised, so it stacktraces. | 21:14 |
| hemna | not sure that makes sense to me | 21:14 |
| hemna | cinder shouldn't allow a volume to be deleted by anyone if it's attached | 21:14 |
| thingee | hemna: still trying to understand myself | 21:14 |
| mtreinish | hemna: tempest deletes the server waits for it to disappear, then issues the volume delete and waits for it to delete | 21:14 |
| mtreinish | since it's a boot from volume server, the detach is happening in nova | 21:15 |
| mtreinish | thingee: does cinderclients detach call wait for the detach to finish? (its async iirc) | 21:15 |
| hemna | ok, but that still doesn't explain the out of sequence events | 21:15 |
| hemna | re: a volume being deleted before detached | 21:16 |
| *** Liu has quit IRC | 21:16 | |
| thingee | hemna: again, tempest does the delete, nova does the detach. | 21:16 |
| thingee | completely separate | 21:16 |
| jgriffith | hemna: I'm completely confused becasue one of the pastebins I looked at earlier had the detach | 21:16 |
| jgriffith | so who knows | 21:16 |
| *** annegentle has joined #openstack-cinder | 21:16 | |
| mtreinish | I'll bbiab, got to head home before the qa meeting | 21:16 |
| jgriffith | "then" the delete | 21:16 |
| hemna | right, but if tempest calls cinder to delete a volume still attached, that should puke | 21:17 |
| * hemna is confused | 21:17 | |
| jgriffith | hemna: +1 | 21:17 |
| thingee | +1 | 21:17 |
| thingee | :) | 21:17 |
| jgriffith | thingee: is there any way to just get a complete set of logs posted? | 21:17 |
| * thingee still waiting for test | 21:17 | |
| hemna | hehe ok. I'm not totally crazy then | 21:17 |
| hemna | phew | 21:17 |
| thingee | jgriffith: yeah just a pain..as I mentioned I'm waiting on the test | 21:18 |
| jgriffith | thingee: cool | 21:18 |
| hemna | https://github.com/openstack/cinder/blob/master/cinder/volume/manager.py#L524-L526 | 21:19 |
| hemna | maybe the status == detaching ? | 21:19 |
| *** Liu has joined #openstack-cinder | 21:21 | |
| *** dims__ has quit IRC | 21:23 | |
| *** cbader has quit IRC | 21:25 | |
| *** Adriano_ has quit IRC | 21:26 | |
| thingee | jgriffith, hemna, mtreinish: ok I think that was from a result of my breakpoints. false alarm.. However, I think it may be possible for that to happen if things were running slow? | 21:27 |
| thingee | as hemna mentioned with the possibility of things in a detaching state. | 21:28 |
| openstackgerrit | Walter A. Boring IV (hemna) proposed openstack/os-brick: Brick: Fix race in removing iSCSI device https://review.openstack.org/168506 | 21:28 |
| *** bswartz has joined #openstack-cinder | 21:29 | |
| openstackgerrit | Angela Smith proposed openstack/cinder-specs: Spec for Brocade Zone Driver Virtual Fabrics support https://review.openstack.org/144389 | 21:29 |
| hemna | I'm just guessing at this point | 21:30 |
| *** eharney has quit IRC | 21:30 | |
| hemna | I'd hope that the API would prevent the call from getting to the manager if the volume is in an 'ing' state | 21:30 |
| hemna | but I don't think we are that safe | 21:30 |
| *** lpetrut has quit IRC | 21:32 | |
| *** tsekiyam_ has joined #openstack-cinder | 21:33 | |
| *** tsekiyama has quit IRC | 21:36 | |
| *** tsekiyam_ has quit IRC | 21:37 | |
| mtreinish | thingee: ah ok, I could see that causing an issue | 21:37 |
| *** emagana has joined #openstack-cinder | 21:38 | |
| *** jamielennox|away is now known as jamielennox | 21:39 | |
| *** markvoelker has joined #openstack-cinder | 21:42 | |
| *** annegentle has quit IRC | 21:44 | |
| *** annegentle has joined #openstack-cinder | 21:45 | |
| *** dannywil_ has quit IRC | 21:46 | |
| *** annegentle has quit IRC | 21:47 | |
| *** dannywilson has joined #openstack-cinder | 21:47 | |
| *** annegentle has joined #openstack-cinder | 21:48 | |
| anish | umm, how do I refer to a Bug in a spec ? what's the appropriate section ? | 21:50 |
| *** xyang has joined #openstack-cinder | 21:50 | |
| openstackgerrit | Anish Bhatt proposed openstack/cinder-specs: Add open-iscsi transport support to brick https://review.openstack.org/152740 | 21:51 |
| hemna | anish, references section maybe | 21:52 |
| anish | that's what I did right now | 21:52 |
| *** EmilienM|afk is now known as EmilienM | 21:55 | |
| thingee | mtreinish, jgriffith, hemna: http://stkci.daterainc.com/refs-changes-63-161363-6/index.html | 22:04 |
| thingee | ok, I take it back, this is the same issue i was hitting earlier in my ci | 22:04 |
| thingee | hemna: see timestamp 2015-04-15 20:01:50 in c-vol | 22:05 |
| *** esker has quit IRC | 22:05 | |
| thingee | and then n-cpu logs at 20:01:50 gets a 500 from cinderclient | 22:06 |
| *** dims__ has joined #openstack-cinder | 22:08 | |
| *** rongze has joined #openstack-cinder | 22:08 | |
| *** annegentle has quit IRC | 22:09 | |
| *** annegentle has joined #openstack-cinder | 22:10 | |
| hemna | ah yes | 22:12 |
| hemna | I have seen that one before | 22:13 |
| *** harlowja_away is now known as harlowja | 22:13 | |
| *** markvoelker has quit IRC | 22:13 | |
| hemna | jogo had enabled a tempest test, and the test was calling delete on the vm (boot from volume), and then the cleanup of the test called detach. | 22:13 |
| hemna | so the test was calling detach on a volume that had already been detached | 22:14 |
| hemna | so the exception was correct. | 22:14 |
| *** rongze has quit IRC | 22:14 | |
| hemna | this may be the case as wel | 22:14 |
| hemna | well | 22:14 |
| jgriffith | hemna: thingee 2015-04-15 20:01:50.739 10938 | 22:15 |
| *** annegentle has quit IRC | 22:15 | |
| thingee | jgriffith: which process? | 22:15 |
| jgriffith | thingee: c-vol | 22:15 |
| jgriffith | thingee: call do driver.detach_volume | 22:16 |
| jgriffith | s/do/to/ | 22:16 |
| * jgriffith doesn't know why the trace info doesn't seem to align though | 22:19 | |
| jgriffith | and grep doesn't seem to find that message which is "weird" | 22:20 |
| jgriffith | it *was* here: https://github.com/openstack/cinder/commit/bc7a284bd2fd27d0f29a8facd5a67abc90667f26 | 22:24 |
| jgriffith | but looks like it's been removed since | 22:24 |
| jgriffith | thingee: guess I could look at the patch ref you have ) | 22:25 |
| jgriffith | thingee: oh... arch 4! | 22:25 |
| jgriffith | March | 22:25 |
| jgriffith | thingee: that's pretty old | 22:26 |
| jgriffith | thingee: although upate April 15 | 22:26 |
| thingee | jgriffith: march 4th? | 22:27 |
| thingee | jgriffith: where did you see that? | 22:27 |
| jgriffith | thingee: that patch was submitted on march 4 | 22:27 |
| jgriffith | thingee: but updated version 6 on the 15'th of april | 22:27 |
| jgriffith | thingee: I was trying to figure out why the messages were so out of date with whats in Master right now | 22:28 |
| jgriffith | thingee: because a bunch of that appears to have changed | 22:28 |
| hemna | so for whatever reason nova is calling detach on that volume more than once | 22:28 |
| hemna | hence the exception | 22:28 |
| jgriffith | thingee: your logs are for: https://review.openstack.org/#/c/161363/ | 22:28 |
| hemna | most likely a tempest cleanup | 22:28 |
| jgriffith | hemna: which is fine | 22:28 |
| jgriffith | hemna: IMO that should just be handled/ignored by the driver | 22:29 |
| jgriffith | hemna: "Hey... stand up", "ok, I already am" | 22:29 |
| thingee | jgriffith: in san.py? | 22:29 |
| jgriffith | thingee: lemme pull the patch so I can look at it all | 22:30 |
| jgriffith | thingee: but you may want to try this on current master too | 22:30 |
| jgriffith | thingee: as it looks like that raise might have been removed anyway | 22:30 |
| *** Lee1092 has quit IRC | 22:30 | |
| *** krtaylor has quit IRC | 22:31 | |
| jgriffith | thingee: Yeah, that's been *fixed* already | 22:31 |
| *** jaypipes has quit IRC | 22:31 | |
| jgriffith | "if len(attachments) == 1: blah blabh " | 22:31 |
| *** annegentle has joined #openstack-cinder | 22:31 | |
| jgriffith | DOH!!! | 22:32 |
| jgriffith | hehe | 22:32 |
| jgriffith | Turns out *I* was the one with the outdated checkout :) | 22:33 |
| *** annegentle has quit IRC | 22:33 | |
| thingee | :) | 22:33 |
| hemna | heh | 22:33 |
| *** annegentle has joined #openstack-cinder | 22:33 | |
| jgriffith | thingee: hemna so IMO the problem is in fact the introduction of the else statement there | 22:33 |
| thingee | jgriffith: I was going to be very happy if that was the problem | 22:33 |
| jgriffith | thingee: hemna in manager.py line 877 | 22:34 |
| jgriffith | thingee: well... it kinda is :) | 22:34 |
| thingee | yeah that's was my earlier theory | 22:34 |
| hemna | well, it's a valid catch though | 22:35 |
| jgriffith | thingee: api:detach blindly sends the call over RPC | 22:35 |
| hemna | someone shouldn't be calling detach if there aren't any left. | 22:35 |
| jgriffith | hemna: sure, but it's harmless | 22:35 |
| jgriffith | hemna: no reason to barf | 22:35 |
| hemna | ok | 22:35 |
| jgriffith | hemna: know what I mean? | 22:35 |
| hemna | sure | 22:35 |
| jgriffith | hemna: similar to the "delete on a backend that doesn't have the volume" | 22:35 |
| jgriffith | Log a warning and move on | 22:35 |
| thingee | hemna, jgriffith: let me remove the exception and try again | 22:35 |
| hemna | I suppose that's a way to look at it :) | 22:35 |
| jgriffith | thingee: hemna not saying there isn't something else "wrong" | 22:36 |
| hemna | which I guess I'm ok with as well. | 22:36 |
| jgriffith | just saying we should puke | 22:36 |
| jgriffith | hemna: so at first I would've done the exact same thing and put the raise there like you did | 22:36 |
| hemna | just do a return then ? | 22:36 |
| hemna | I'm just curious if that might leave the volume in detaching then. | 22:36 |
| jgriffith | hemna: in reality a proper thing would be to check at volume.api:detach before firing off the rpc call | 22:36 |
| hemna | if we just bail at that point with a return. | 22:37 |
| jgriffith | hemna: that's the "real" bug IMHO | 22:37 |
| jgriffith | hemna: and leave your raise there | 22:37 |
| *** annegentle has quit IRC | 22:37 | |
| openstackgerrit | Walter A. Boring IV (hemna) proposed openstack/os-brick: Brick: Fix race in removing iSCSI device https://review.openstack.org/168506 | 22:37 |
| hemna | yah that makes sense. | 22:37 |
| jgriffith | hemna: oh... so I was saying don't raise, pass.. and update | 22:37 |
| jgriffith | hemna: but it's not that simple | 22:38 |
| jgriffith | hemna: there's a good deal of *stuff* after that now | 22:38 |
| jgriffith | hemna: so fix the api and that should be "good" | 22:38 |
| *** EmilienM is now known as EmilienM|afk | 22:40 | |
| *** mriedem has quit IRC | 22:42 | |
| jgriffith | hemna: so I'm curious... did something "change" that we don't use "begin detaching" anymore? | 22:43 |
| jgriffith | hemna: or that Nova doesn't use it I guess? | 22:43 |
| hemna | so that's part of the problem | 22:44 |
| hemna | Nova isn't using the correct order of Cinder APIs | 22:44 |
| hemna | at times | 22:44 |
| hemna | the rebuild bug that popped up was a case in point of that | 22:44 |
| hemna | nova was simply calling detach | 22:44 |
| hemna | without calling terminate_connection | 22:45 |
| hemna | :( | 22:45 |
| hemna | so it wouldn't surprise me if nova doesn't call reserve, begin, etc | 22:45 |
| * hemna has a sad | 22:45 | |
| openstackgerrit | John Griffith proposed openstack/cinder: Add volume status check to api detach method https://review.openstack.org/174634 | 22:45 |
| jgriffith | hemna: oh... yeah :) | 22:46 |
| jgriffith | hemna: although actually.. I think in this test Tempest is calling cinderclient directly itself | 22:46 |
| jgriffith | so it has nothing to do with Nova | 22:46 |
| hemna | yah you are probably right. | 22:46 |
| jgriffith | thingee: so that's your failure BTW I think | 22:46 |
| jgriffith | thingee: we didn't used to raise in that case... now we do | 22:46 |
| jgriffith | thingee: not quite sure why it wouldn't be seen *everywhere* but frankly I haven't looked :) | 22:47 |
| thingee | jgriffith: good, that's what I was saying earlier :) | 22:47 |
| jgriffith | thingee: by "everywhere" I mean other drivers | 22:47 |
| thingee | jgriffith: maybe my ci is slow | 22:47 |
| jgriffith | thingee: Oh... well if you already knew why did I just waste the last hour looking at this with you | 22:47 |
| thingee | jgriffith: I raised it earlier...and then I think you just started looking on your own to come to the same conclusion | 22:48 |
| *** markvoelker has joined #openstack-cinder | 22:48 | |
| jgriffith | thingee: oh... how stupid of me | 22:48 |
| jgriffith | thought i was being helpful | 22:48 |
| jgriffith | didn't realize you had it all figured out. | 22:48 |
| jgriffith | sorry... ok, just ignore me :) | 22:49 |
| thingee | well I wasn't sure...that's why I included you all. to see if I was crazy | 22:49 |
| *** markvoelker_ has joined #openstack-cinder | 22:49 | |
| *** dims__ has quit IRC | 22:50 | |
| *** markvoel_ has joined #openstack-cinder | 22:50 | |
| *** markvoe__ has joined #openstack-cinder | 22:51 | |
| *** markvoelker has quit IRC | 22:53 | |
| *** markvoelker_ has quit IRC | 22:54 | |
| *** Mandell has quit IRC | 22:54 | |
| *** markvoel_ has quit IRC | 22:55 | |
| *** Mandell has joined #openstack-cinder | 22:55 | |
| *** zhipeng has quit IRC | 22:56 | |
| *** ganso_ has quit IRC | 22:57 | |
| *** chlong has joined #openstack-cinder | 23:02 | |
| *** rongze has joined #openstack-cinder | 23:09 | |
| *** Mandell has quit IRC | 23:09 | |
| *** EmilienM|afk is now known as EmilienM | 23:10 | |
| *** rongze has quit IRC | 23:14 | |
| mtreinish | jgriffith: tempest doesn't call cinderclient at all (except in the cli tests which will be moved shortly :) ) | 23:14 |
| jgriffith | mtreinish: LOL... nice plug! :) | 23:17 |
| jgriffith | mtreinish: so I should clarify what I meant... meaning that the detach call is made during the cleanup by Tempest | 23:18 |
| mtreinish | jgriffith: it's not in test volume boot pattern which is what thingee was looking at | 23:18 |
| jgriffith | mtreinish: ahhh | 23:18 |
| mtreinish | it's all boot from volume | 23:19 |
| mtreinish | so there is no manual attach or detach | 23:19 |
| jgriffith | mtreinish: well regardless, apparantly he knew/knows what the problem is there | 23:19 |
| jgriffith | mtreinish: oh..... so it's using the crazy "nova boot" args that create the volume and boot all in once | 23:19 |
| jgriffith | mtreinish: suppose I should go look at the actual test :) | 23:21 |
| *** rongze has joined #openstack-cinder | 23:21 | |
| mtreinish | jgriffith: yeah pretty much: http://git.openstack.org/cgit/openstack/tempest/tree/tempest/scenario/test_volume_boot_pattern.py#n52 | 23:21 |
| jgriffith | mtreinish: yeah, so that's troublesome all on it's own I'm afraid | 23:22 |
| thingee | mtreinish: my tempest.log was showing a DELETE to cinder | 23:22 |
| *** rongze has quit IRC | 23:23 | |
| jgriffith | mtreinish: oh... no, we're talking different :) | 23:23 |
| *** rongze has joined #openstack-cinder | 23:23 | |
| *** fanyaohong has joined #openstack-cinder | 23:23 | |
| jgriffith | mtreinish: that sends the call to create_volume_from_image and then passes in that ID | 23:23 |
| jgriffith | mtreinish: that's cool | 23:23 |
| mtreinish | oh ok, cool | 23:23 |
| jgriffith | mtreinish: there's a crazy thing that Horizon does where it does it all in one shot | 23:23 |
| mtreinish | oh I didn't even realize there was something to do that | 23:24 |
| jgriffith | mtreinish: and it's stupid lame because for any "real" sized volume it will time out every time | 23:24 |
| jgriffith | or... real sized image | 23:24 |
| mtreinish | thingee: sure a delete, but no detach | 23:24 |
| *** rongze_ has joined #openstack-cinder | 23:24 | |
| *** rongze has quit IRC | 23:24 | |
| mtreinish | thingee: it deletes the volume it creates after it deletes the server | 23:24 |
| jgriffith | mtreinish: yeah, it's an awful thing IMHO | 23:24 |
| thingee | mtreinish: yea | 23:24 |
| *** rongze has joined #openstack-cinder | 23:25 | |
| *** rongze_ has quit IRC | 23:25 | |
| *** rongze_ has joined #openstack-cinder | 23:32 | |
| *** rongze has quit IRC | 23:35 | |
| *** Apoorva_ has joined #openstack-cinder | 23:37 | |
| *** Apoorva has quit IRC | 23:37 | |
| *** hemna is now known as hemnafk | 23:39 | |
| *** thingee has quit IRC | 23:41 | |
| *** emagana has quit IRC | 23:50 | |
| *** rongze has joined #openstack-cinder | 23:52 | |
| *** rongze has quit IRC | 23:53 | |
| *** rongze_ has quit IRC | 23:56 | |
| *** markvoe__ has quit IRC | 23:58 | |
| *** dannywilson has quit IRC | 23:59 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!