| *** mriedem has quit IRC | 00:01 | |
| *** julim has joined #openstack-cinder | 00:10 | |
| *** _cjones_ has quit IRC | 00:11 | |
| *** Yogi1 has joined #openstack-cinder | 00:14 | |
| *** IanGovett has quit IRC | 00:14 | |
| *** r-daneel has quit IRC | 00:24 | |
| *** jwcroppe has joined #openstack-cinder | 00:28 | |
| *** Yogi11 has joined #openstack-cinder | 00:33 | |
| *** dims_ has joined #openstack-cinder | 00:35 | |
| *** Yogi1 has quit IRC | 00:36 | |
| *** dims has quit IRC | 00:36 | |
| *** heyun has joined #openstack-cinder | 00:38 | |
| *** emagana has quit IRC | 00:40 | |
| *** emagana has joined #openstack-cinder | 00:41 | |
| *** IanGovett has joined #openstack-cinder | 00:42 | |
| *** leeantho has quit IRC | 00:46 | |
| *** emagana has quit IRC | 00:48 | |
| *** IanGovett has quit IRC | 00:49 | |
| *** annashen has joined #openstack-cinder | 00:49 | |
| *** kmartin has quit IRC | 00:51 | |
| *** annashen has quit IRC | 00:55 | |
| *** fanyaohong has quit IRC | 00:59 | |
| *** esker has joined #openstack-cinder | 01:24 | |
| *** davechen has joined #openstack-cinder | 01:27 | |
| *** jwcroppe has quit IRC | 01:32 | |
| *** davechen1 has joined #openstack-cinder | 01:34 | |
| *** davechen has quit IRC | 01:36 | |
| *** vilobhmm1 has quit IRC | 01:37 | |
| *** davechen has joined #openstack-cinder | 01:45 | |
| *** davechen1 has quit IRC | 01:46 | |
| *** davechen1 has joined #openstack-cinder | 01:55 | |
| *** xyang has quit IRC | 01:55 | |
| *** davechen has quit IRC | 01:55 | |
| openstackgerrit | xing-yang proposed openstack/python-cinderclient: Add support to incremental backups in cinder https://review.openstack.org/112101 | 01:55 |
|---|---|---|
| *** Lee1092 has joined #openstack-cinder | 01:56 | |
| *** kaisers has joined #openstack-cinder | 01:58 | |
| *** kaisers1 has quit IRC | 01:59 | |
| *** changbl has joined #openstack-cinder | 02:14 | |
| *** harlowja is now known as harlowja_away | 02:17 | |
| *** Yogi12 has joined #openstack-cinder | 02:20 | |
| *** dims_ has quit IRC | 02:21 | |
| *** Yogi11 has quit IRC | 02:24 | |
| openstackgerrit | Patrick East proposed openstack/cinder: Add locking to PureISCSIDriver around creating Purity Host objects. https://review.openstack.org/173653 | 02:24 |
| *** Apoorva has quit IRC | 02:26 | |
| *** vilobhmm1 has joined #openstack-cinder | 02:28 | |
| *** Yogi12 has quit IRC | 02:32 | |
| *** patrickeast has quit IRC | 02:40 | |
| *** julim has quit IRC | 02:43 | |
| *** julim has joined #openstack-cinder | 02:55 | |
| *** esker has quit IRC | 02:57 | |
| *** esker has joined #openstack-cinder | 02:57 | |
| *** mriedem has joined #openstack-cinder | 03:02 | |
| *** mriedem1 has quit IRC | 03:04 | |
| *** mriedem has quit IRC | 03:08 | |
| *** jwcroppe has joined #openstack-cinder | 03:14 | |
| *** Longgeek has joined #openstack-cinder | 03:18 | |
| *** aswadr has joined #openstack-cinder | 03:25 | |
| openstackgerrit | Jay Bryant proposed openstack/os-brick: Sync loopingcall from oslo-incubator for os-brick https://review.openstack.org/173667 | 03:37 |
| *** li_zhang has joined #openstack-cinder | 03:41 | |
| *** Longgeek has quit IRC | 03:41 | |
| openstackgerrit | Jay Bryant proposed openstack/os-brick: Use oslo_log instead of openstack.common.log https://review.openstack.org/173106 | 03:45 |
| thingee | jungleboyj: is this still going to happen? https://blueprints.launchpad.net/cinder/+spec/db2-database | 03:48 |
| jungleboyj | thingee: Technically that is happening. | 03:52 |
| jungleboyj | thingee: Don't don't have the unittests working yet. | 03:53 |
| *** aswadr has quit IRC | 03:54 | |
| thingee | jungleboyj: ok | 03:54 |
| thingee | so liberty? | 03:54 |
| jungleboyj | thingee: Yeah, I should make time to finish that up. | 03:57 |
| *** shyama has joined #openstack-cinder | 03:58 | |
| *** Mandell has quit IRC | 04:05 | |
| *** krtaylor has quit IRC | 04:07 | |
| *** Mandell has joined #openstack-cinder | 04:09 | |
| *** zhangli has joined #openstack-cinder | 04:09 | |
| *** li_zhang has quit IRC | 04:09 | |
| *** krtaylor has joined #openstack-cinder | 04:10 | |
| *** zhangli has quit IRC | 04:14 | |
| *** zhangli has joined #openstack-cinder | 04:15 | |
| *** patrickeast has joined #openstack-cinder | 04:16 | |
| patrickeast | thingee: winston-d: thanks for the super quick reviews on https://review.openstack.org/#/c/173653/ ! | 04:18 |
| thingee | patrickeast: np, just waiting on the pure ci | 04:19 |
| *** heyun has quit IRC | 04:20 | |
| *** btran has quit IRC | 04:21 | |
| *** annashen has joined #openstack-cinder | 04:21 | |
| patrickeast | thingee: hmm, it should have finished by now... | 04:22 |
| * patrickeast goes to check | 04:22 | |
| *** sks has joined #openstack-cinder | 04:23 | |
| patrickeast | thingee: oh derp | 04:24 |
| patrickeast | thingee: my zuul-merger was stuck, had like 3 things in the queue from the last couple of hours | 04:25 |
| patrickeast | thingee: ~45 min and it should be posted | 04:25 |
| patrickeast | thingee: although, this brings up an interesting point... this bug wouldn't actually have been found from the CI since we don't have CHAP enabled in the tests... I've already added a task to figure out which combinations of config variables we need to have setup to get better coverage | 04:26 |
| patrickeast | things like multipathing on/off, chap, etc | 04:26 |
| *** annashen has quit IRC | 04:26 | |
| *** ankit_ag has joined #openstack-cinder | 04:29 | |
| *** zhangli has quit IRC | 04:32 | |
| *** heyun has joined #openstack-cinder | 04:32 | |
| *** rushiagr_away is now known as rushiagr | 04:36 | |
| *** Longgeek has joined #openstack-cinder | 04:37 | |
| *** avishay has joined #openstack-cinder | 04:37 | |
| *** rushiagr is now known as rushiagr_away | 04:38 | |
| *** rushiagr_away is now known as rushiagr | 04:38 | |
| *** Longgeek has quit IRC | 04:44 | |
| *** xyang1 has quit IRC | 04:52 | |
| *** BharatK has joined #openstack-cinder | 04:53 | |
| *** yamada-h has quit IRC | 05:00 | |
| *** jwcroppe has quit IRC | 05:01 | |
| patrickeast | thingee: the ci just finished | 05:02 |
| *** Mandell has quit IRC | 05:02 | |
| *** yamada-h has joined #openstack-cinder | 05:02 | |
| thingee | patrickeast: thanks | 05:04 |
| thingee | patrickeast: +2/a | 05:05 |
| patrickeast | thingee: what would you say the odds are of being able to get that into kilo? | 05:05 |
| patrickeast | i'm still kicking myself that we didn't find it until today... went through like 4 people banging on this and no one noticed until today when i was looking at some new stuff for L >.< | 05:06 |
| openstackgerrit | Mike Perez proposed openstack/cinder-specs: Open Liberty for specs https://review.openstack.org/173675 | 05:07 |
| patrickeast | oo | 05:07 |
| thingee | patrickeast: We have to do an rc-2 no matter what, so I think good :) | 05:07 |
| *** deepakcs has joined #openstack-cinder | 05:08 | |
| patrickeast | thingee: awesome | 05:08 |
| thingee | I'll review the kilo propose patch | 05:08 |
| patrickeast | thingee: thanks! | 05:08 |
| thingee | patrickeast: need to get my voting rights on that branch adjusted. +1 for now | 05:09 |
| thingee | patrickeast: have a good night. I'm going to head to bed early for some much needed sleep. | 05:10 |
| patrickeast | thingee: night! thanks again for the reviews | 05:11 |
| openstackgerrit | Daisuke Fujita proposed openstack/cinder: Fix a wrong argument of create method https://review.openstack.org/173679 | 05:23 |
| *** nkrinner has joined #openstack-cinder | 05:23 | |
| *** Mandell has joined #openstack-cinder | 05:26 | |
| *** thingee has quit IRC | 05:29 | |
| *** ociuhandu has quit IRC | 05:29 | |
| *** resker has joined #openstack-cinder | 05:32 | |
| *** nikesh has joined #openstack-cinder | 05:32 | |
| *** nlevinki has joined #openstack-cinder | 05:33 | |
| *** lan has quit IRC | 05:35 | |
| *** lan has joined #openstack-cinder | 05:35 | |
| *** esker has quit IRC | 05:35 | |
| *** sgotliv has joined #openstack-cinder | 05:36 | |
| *** ociuhandu has joined #openstack-cinder | 05:38 | |
| *** ankit8188 has joined #openstack-cinder | 05:38 | |
| *** ankit8188 has quit IRC | 05:38 | |
| *** patrickeast has quit IRC | 05:40 | |
| nikesh | thingee: hi | 05:40 |
| *** yamada-h has quit IRC | 05:44 | |
| *** Longgeek has joined #openstack-cinder | 05:48 | |
| *** dulek has joined #openstack-cinder | 05:49 | |
| *** bkopilov has quit IRC | 05:53 | |
| *** ociuhandu has quit IRC | 05:56 | |
| *** bkopilov has joined #openstack-cinder | 05:59 | |
| *** TobiasE has joined #openstack-cinder | 06:03 | |
| *** TobiasE has quit IRC | 06:05 | |
| *** bkopilov has quit IRC | 06:24 | |
| *** yamada-h has joined #openstack-cinder | 06:30 | |
| *** yamada-h has quit IRC | 06:35 | |
| *** vilobhmm1 has quit IRC | 06:49 | |
| *** tshefi has joined #openstack-cinder | 06:50 | |
| *** bkopilov has joined #openstack-cinder | 06:50 | |
| *** sgotliv has quit IRC | 06:55 | |
| *** bkopilov has quit IRC | 06:56 | |
| *** ho has quit IRC | 07:00 | |
| *** nshaikh has joined #openstack-cinder | 07:00 | |
| *** yamada-h has joined #openstack-cinder | 07:01 | |
| *** yamada-h has quit IRC | 07:05 | |
| *** rwsu has quit IRC | 07:09 | |
| *** bkopilov has joined #openstack-cinder | 07:11 | |
| *** markus_z has joined #openstack-cinder | 07:12 | |
| *** jordanP has joined #openstack-cinder | 07:13 | |
| *** jcespedes_ has quit IRC | 07:19 | |
| *** jistr has joined #openstack-cinder | 07:24 | |
| *** annashen has joined #openstack-cinder | 07:24 | |
| *** annashen has quit IRC | 07:29 | |
| *** ankit_ag has quit IRC | 07:30 | |
| *** ankit_ag has joined #openstack-cinder | 07:33 | |
| *** chlong has quit IRC | 07:35 | |
| *** yamada-h has joined #openstack-cinder | 07:37 | |
| *** yamada-h has quit IRC | 07:37 | |
| *** c0m0 has joined #openstack-cinder | 07:41 | |
| *** bkopilov has quit IRC | 07:45 | |
| *** yamada-h has joined #openstack-cinder | 07:48 | |
| *** yamada-h has quit IRC | 07:48 | |
| *** bkopilov has joined #openstack-cinder | 07:49 | |
| *** rushiagr is now known as rushiagr_away | 07:49 | |
| *** abhiram_moturi has quit IRC | 07:50 | |
| *** abhiram_moturi has joined #openstack-cinder | 07:50 | |
| *** bkopilov has quit IRC | 07:54 | |
| *** bkopilov has joined #openstack-cinder | 07:54 | |
| *** ronis_ has joined #openstack-cinder | 07:56 | |
| *** rushiagr_away is now known as rushiagr | 08:01 | |
| *** e0ne has joined #openstack-cinder | 08:02 | |
| *** e0ne has quit IRC | 08:05 | |
| *** e0ne has joined #openstack-cinder | 08:08 | |
| *** Mandell has quit IRC | 08:08 | |
| *** ronis_ has quit IRC | 08:10 | |
| *** bkopilov has quit IRC | 08:10 | |
| *** shyama has quit IRC | 08:15 | |
| *** bkopilov has joined #openstack-cinder | 08:17 | |
| *** e0ne has quit IRC | 08:19 | |
| *** e0ne has joined #openstack-cinder | 08:23 | |
| *** ndipanov has joined #openstack-cinder | 08:27 | |
| *** e0ne has quit IRC | 08:29 | |
| *** ociuhandu has joined #openstack-cinder | 08:31 | |
| *** e0ne has joined #openstack-cinder | 08:33 | |
| *** alecv has joined #openstack-cinder | 08:35 | |
| *** alecv has quit IRC | 08:37 | |
| *** bkopilov has quit IRC | 08:39 | |
| *** zerda has joined #openstack-cinder | 08:40 | |
| *** e0ne has quit IRC | 08:41 | |
| *** bkopilov has joined #openstack-cinder | 08:45 | |
| *** davechen1 has left #openstack-cinder | 08:47 | |
| *** bkopilov has quit IRC | 08:51 | |
| *** bkopilov has joined #openstack-cinder | 08:52 | |
| *** krtaylor has quit IRC | 08:57 | |
| *** bkopilov has quit IRC | 08:57 | |
| *** alecv has joined #openstack-cinder | 09:03 | |
| *** shyama has joined #openstack-cinder | 09:03 | |
| *** krtaylor has joined #openstack-cinder | 09:04 | |
| *** sgotliv has joined #openstack-cinder | 09:11 | |
| zerda | hemna, hello, I wonder is there any plans to re-introduce HP MSA driver? | 09:11 |
| openstackgerrit | yogeshprasad proposed openstack/cinder: Add chap support to CloudByte cinder driver https://review.openstack.org/173733 | 09:19 |
| *** heyun has quit IRC | 09:20 | |
| *** nshaikh has left #openstack-cinder | 09:31 | |
| *** TobiasE has joined #openstack-cinder | 09:31 | |
| *** e0ne has joined #openstack-cinder | 09:34 | |
| *** e0ne is now known as e0ne_ | 09:49 | |
| *** liusheng has quit IRC | 09:51 | |
| *** jamielennox is now known as jamielennox|away | 09:52 | |
| *** nshaikh has joined #openstack-cinder | 09:59 | |
| *** e0ne_ has quit IRC | 10:01 | |
| *** Longgeek has quit IRC | 10:02 | |
| *** Longgeek_ has joined #openstack-cinder | 10:02 | |
| *** e0ne has joined #openstack-cinder | 10:03 | |
| *** theanalyst has quit IRC | 10:06 | |
| *** theanalyst has joined #openstack-cinder | 10:09 | |
| *** ronis has joined #openstack-cinder | 10:11 | |
| *** annashen has joined #openstack-cinder | 10:26 | |
| *** lpetrut has joined #openstack-cinder | 10:27 | |
| *** nlevinki_ has joined #openstack-cinder | 10:28 | |
| *** IanGovett has joined #openstack-cinder | 10:28 | |
| *** nlevinki has quit IRC | 10:30 | |
| *** annashen has quit IRC | 10:31 | |
| *** heyun has joined #openstack-cinder | 10:34 | |
| *** ociuhandu has quit IRC | 10:34 | |
| *** MIDENN_ has quit IRC | 10:35 | |
| DuncanT | zerda: Do you have a desire to use an MSA with Openstack? | 10:37 |
| *** e0ne is now known as e0ne_ | 10:39 | |
| zerda | DuncanT, we're considering this now | 10:40 |
| DuncanT | zerda: Thanks. Hemna is probably your best sorce of info, but he's west coast US so won't be on for a while. As far as I know there's no current plan simply due to lack of demand | 10:41 |
| *** e0ne_ is now known as e0ne | 10:42 | |
| zerda | DuncanT, OK, got it, thank you. I'm 13 hours away from him, guess it's will be better to write an email :) | 10:43 |
| *** deepakcs has quit IRC | 10:46 | |
| *** zhenguo has quit IRC | 10:48 | |
| *** asselin_ has quit IRC | 10:49 | |
| *** e0ne has quit IRC | 10:53 | |
| *** e0ne has joined #openstack-cinder | 10:54 | |
| openstackgerrit | Duncan Thomas proposed openstack/cinder: Add CA cert option to backups swift driver https://review.openstack.org/149618 | 10:55 |
| *** sgotliv has quit IRC | 11:00 | |
| *** heyun has quit IRC | 11:01 | |
| nikesh | zerda DuncanT: hi we are working with hp to introduce back hpmsa driver for liberty release | 11:06 |
| *** bkopilov has joined #openstack-cinder | 11:07 | |
| *** dims has joined #openstack-cinder | 11:08 | |
| *** BharatK has quit IRC | 11:09 | |
| DuncanT | I'm confused, can anybody see why the nimble CI failed on https://review.openstack.org/#/c/149618/ ? | 11:11 |
| zerda | nikesh, glad to hear that | 11:14 |
| *** sgotliv has joined #openstack-cinder | 11:15 | |
| *** e0ne is now known as e0ne_ | 11:19 | |
| *** nlevinki_ has quit IRC | 11:19 | |
| sdague | does anyone know if this statement is still true? | 11:20 |
| sdague | " Currently cinderclient needs you to specify the *volume api* version. " | 11:21 |
| sdague | because some code in openrc in devstack completely masks out openstack client having volumes support because of it | 11:21 |
| *** nlevinki has joined #openstack-cinder | 11:22 | |
| *** BharatK has joined #openstack-cinder | 11:23 | |
| *** e0ne_ has quit IRC | 11:29 | |
| *** Longgeek has joined #openstack-cinder | 11:29 | |
| *** Longgeek_ has quit IRC | 11:31 | |
| *** ociuhandu has joined #openstack-cinder | 11:37 | |
| *** e0ne has joined #openstack-cinder | 11:38 | |
| openstackgerrit | Rushi Agrawal proposed openstack/cinder-specs: Snapshot sharing https://review.openstack.org/173779 | 11:41 |
| *** aix has quit IRC | 11:50 | |
| *** Lee1092 has quit IRC | 11:51 | |
| *** akerr has joined #openstack-cinder | 12:01 | |
| *** julim has quit IRC | 12:01 | |
| *** Yogi11 has joined #openstack-cinder | 12:03 | |
| *** arif-ali has quit IRC | 12:05 | |
| *** Mandell has joined #openstack-cinder | 12:09 | |
| openstackgerrit | Rushi Agrawal proposed openstack/cinder-specs: Snapshot sharing https://review.openstack.org/173779 | 12:11 |
| *** arif-ali has joined #openstack-cinder | 12:12 | |
| *** Mandell has quit IRC | 12:13 | |
| *** e0ne is now known as e0ne_ | 12:14 | |
| *** bswartz has quit IRC | 12:22 | |
| *** Lee1092 has joined #openstack-cinder | 12:23 | |
| *** zerda has quit IRC | 12:24 | |
| *** e0ne_ has quit IRC | 12:26 | |
| *** aix has joined #openstack-cinder | 12:28 | |
| *** zhenguo has joined #openstack-cinder | 12:29 | |
| *** e0ne has joined #openstack-cinder | 12:30 | |
| *** timcl has joined #openstack-cinder | 12:32 | |
| *** eharney has joined #openstack-cinder | 12:34 | |
| *** nshaikh has left #openstack-cinder | 12:40 | |
| *** jaypipes has quit IRC | 12:42 | |
| *** julim has joined #openstack-cinder | 12:43 | |
| smcginnis | sdague: I know thingee has a patch out for automatic version discovery. | 12:44 |
| smcginnis | sdague: I don't know enough, but I thought it just defaulted to v1. | 12:44 |
| sdague | smcginnis: no idea, but in all devstack configs we force it to v2, which causes openstack client heartburn | 12:45 |
| smcginnis | sdague: AFIAK, v2 is where we want to be, but v1 is still the client default. | 12:45 |
| sdague | ok, so I feel like we should not be overriding that default in devstack | 12:49 |
| sdague | devstack tries to keep the defaults when possible of what the projects are doing | 12:50 |
| *** sgotliv has quit IRC | 12:50 | |
| smcginnis | sdague: That does seem like it would be safer. | 12:50 |
| sdague | it would be good to know what the cinder plan is here to change that default | 12:50 |
| *** annegentle has joined #openstack-cinder | 12:50 | |
| *** sgotliv has joined #openstack-cinder | 12:51 | |
| smcginnis | I believe the plan is to have the default v2, but I will let other more knowledgable folks confirm that. :) | 12:51 |
| *** sgotliv has quit IRC | 12:51 | |
| *** sgotliv has joined #openstack-cinder | 12:51 | |
| e0ne | sdague: i think we'll change default to v2 in L because we'll remove v1 soon :) | 12:53 |
| *** jaypipes has joined #openstack-cinder | 12:56 | |
| *** timcl1 has joined #openstack-cinder | 12:57 | |
| smcginnis | hemna: Preso looks nice. It's a great start. | 12:57 |
| smcginnis | hemna: I wish dropbox would allow shared editing. :\ | 12:58 |
| *** bswartz has joined #openstack-cinder | 12:58 | |
| smcginnis | hemna: I thought it did, but maybe not odp format? | 12:58 |
| *** resker has quit IRC | 12:58 | |
| *** timcl has quit IRC | 13:00 | |
| *** Yogi11 has quit IRC | 13:09 | |
| *** dims has quit IRC | 13:13 | |
| *** dims has joined #openstack-cinder | 13:15 | |
| *** crose has joined #openstack-cinder | 13:19 | |
| *** primechuck has joined #openstack-cinder | 13:20 | |
| *** david-lyle has quit IRC | 13:24 | |
| *** dustins has joined #openstack-cinder | 13:25 | |
| *** jungleboyj has quit IRC | 13:26 | |
| *** mriedem has joined #openstack-cinder | 13:27 | |
| *** cbader has joined #openstack-cinder | 13:27 | |
| *** annashen has joined #openstack-cinder | 13:29 | |
| *** r-daneel has joined #openstack-cinder | 13:31 | |
| *** annashen has quit IRC | 13:33 | |
| *** BharatK has quit IRC | 13:35 | |
| *** jwcroppe has joined #openstack-cinder | 13:38 | |
| *** jwcroppe has quit IRC | 13:38 | |
| *** jwcroppe has joined #openstack-cinder | 13:39 | |
| *** ozialien has joined #openstack-cinder | 13:39 | |
| *** marcusvrn has quit IRC | 13:41 | |
| *** Yogi1 has joined #openstack-cinder | 13:41 | |
| *** e0ne is now known as e0ne_ | 13:43 | |
| *** e0ne_ is now known as e0ne | 13:43 | |
| *** mtanino has joined #openstack-cinder | 13:48 | |
| *** markvoelker has joined #openstack-cinder | 13:48 | |
| *** marcusvrn has joined #openstack-cinder | 13:56 | |
| *** Yogi11 has joined #openstack-cinder | 13:58 | |
| e0ne | winston-d: hi. are you around? | 14:00 |
| *** dustins has quit IRC | 14:00 | |
| *** rushil has joined #openstack-cinder | 14:01 | |
| *** Yogi1 has quit IRC | 14:01 | |
| *** dustins has joined #openstack-cinder | 14:02 | |
| *** rongze has joined #openstack-cinder | 14:06 | |
| *** esker has joined #openstack-cinder | 14:06 | |
| *** nkrinner has quit IRC | 14:06 | |
| *** bswartz has quit IRC | 14:06 | |
| *** ronis has quit IRC | 14:07 | |
| *** markvoelker_ has joined #openstack-cinder | 14:13 | |
| *** markvoelker has quit IRC | 14:16 | |
| *** ozialien has quit IRC | 14:17 | |
| *** bswartz has joined #openstack-cinder | 14:17 | |
| *** ozialien has joined #openstack-cinder | 14:19 | |
| *** marcusvrn1 has joined #openstack-cinder | 14:22 | |
| *** marcusvrn has quit IRC | 14:24 | |
| *** ozialien has quit IRC | 14:26 | |
| *** ankit_ag has quit IRC | 14:32 | |
| *** rushiagr is now known as rushiagr_away | 14:33 | |
| *** emagana has joined #openstack-cinder | 14:36 | |
| *** primechuck has quit IRC | 14:36 | |
| *** markvoelker has joined #openstack-cinder | 14:36 | |
| *** markvoel_ has joined #openstack-cinder | 14:38 | |
| *** russellb has quit IRC | 14:38 | |
| jgriffith | _/away | 14:39 |
| *** markvoelker_ has quit IRC | 14:40 | |
| *** markvoel_ has quit IRC | 14:41 | |
| *** russellb has joined #openstack-cinder | 14:42 | |
| jgriffith | \/hmmm | 14:42 |
| *** markvoelker has quit IRC | 14:42 | |
| *** marcusvrn has joined #openstack-cinder | 14:45 | |
| smcginnis | jgriffith: No coffee yet? :P | 14:45 |
| *** sks has quit IRC | 14:45 | |
| jgriffith | smcginnis: hehe.. nope | 14:45 |
| jgriffith | smcginnis: It's brewing though :) | 14:46 |
| nikesh | thingee:hi,could you please approve bps https://blueprints.launchpad.net/cinder/+spec/dothill-iscsi-cinder-driver https://blueprints.launchpad.net/cinder/+spec/dothill-fc-cinder-driver and target them for liberty | 14:46 |
| *** marcusvrn1 has quit IRC | 14:46 | |
| nikesh | sooner i am planning to submit patch for reviews for that | 14:47 |
| smcginnis | nikesh: Probably better not to keep pinging him on that. He will get to it. | 14:47 |
| smcginnis | nikesh: Just post your code when ready. If the bp isn't approved by then it will be before merging. | 14:47 |
| *** marcusvrn1 has joined #openstack-cinder | 14:47 | |
| *** BharatK has joined #openstack-cinder | 14:49 | |
| *** marcusvrn has quit IRC | 14:49 | |
| nikesh | smcginnis: ok thanks | 14:50 |
| *** primechuck has joined #openstack-cinder | 14:52 | |
| *** asselin has joined #openstack-cinder | 14:53 | |
| *** annegentle has quit IRC | 14:54 | |
| *** timcl1 has quit IRC | 14:54 | |
| *** smoriya has joined #openstack-cinder | 14:54 | |
| *** jordanP has quit IRC | 14:55 | |
| *** patrickeast has joined #openstack-cinder | 14:55 | |
| *** kmartin has joined #openstack-cinder | 14:58 | |
| DuncanT | nikesh: You're aware you'll need working CI before we merge new drivers, right? | 14:58 |
| *** xyang has joined #openstack-cinder | 14:58 | |
| *** jungleboyj has joined #openstack-cinder | 14:58 | |
| *** mdenny has joined #openstack-cinder | 15:00 | |
| *** rwsu has joined #openstack-cinder | 15:00 | |
| DuncanT | Hmmm, nothing at all on the agenda yet | 15:01 |
| DuncanT | Shortest weekly meeting ever | 15:01 |
| smcginnis | DuncanT: Should be quick. | 15:01 |
| *** timcl has joined #openstack-cinder | 15:01 | |
| smcginnis | But I'm sure we'll find something to fill the time. | 15:01 |
| smcginnis | Should probably talk Summit. | 15:01 |
| *** Yogi11 has quit IRC | 15:02 | |
| *** Mandell has joined #openstack-cinder | 15:02 | |
| *** jkraj has joined #openstack-cinder | 15:02 | |
| Swanson | Ballpark timelines for liberty would be nice. | 15:02 |
| *** Yogi1 has joined #openstack-cinder | 15:03 | |
| jungleboyj | Swanson: There was a mailing list thread from ttx on that this week. | 15:03 |
| jungleboyj | At least from a project level. | 15:03 |
| Swanson | I need to stop sorting the mailing list into "cinder" and "other" folders. | 15:04 |
| *** nlevinki has quit IRC | 15:04 | |
| jungleboyj | Swanson: :-) The e-mail is a never ending challenge. | 15:04 |
| *** smoriya has quit IRC | 15:05 | |
| *** xyang1 has joined #openstack-cinder | 15:05 | |
| nikesh | DuncanT:yes | 15:06 |
| *** TobiasE has quit IRC | 15:07 | |
| *** xyang has quit IRC | 15:07 | |
| e0ne | https://github.com/openstack/openstack-specs/blob/master/specs/no-downward-sql-migration.rst - interesting.. i've missed it:( | 15:08 |
| e0ne | winston-d: short update about BlockDeviceDriver | 15:10 |
| *** shyama has quit IRC | 15:10 | |
| e0ne | winston-d: it was implemented for using in Sahara | 15:10 |
| *** shyama has joined #openstack-cinder | 15:11 | |
| *** luv has joined #openstack-cinder | 15:11 | |
| e0ne | winston-d: sahara uses it now and our sahara team has contacts for people who use it with sahara | 15:11 |
| *** rushiagr_away is now known as rushiagr | 15:11 | |
| e0ne | winston-d: feel free to ask it in #openstack-sahara | 15:11 |
| *** markvoelker has joined #openstack-cinder | 15:12 | |
| e0ne | winston-d: or ask me if you need some driver support/maitanence/fixing | 15:12 |
| e0ne | winston-d: i havn't any testing results for this driver :( | 15:12 |
| nikesh | jgriffith: how can we make sos-ci working before merging drivers | 15:12 |
| DuncanT | nikesh: Cherry pick your driver on top of cinder, there's a devstack config option to do so | 15:13 |
| luv | hi, Im writing a volume driver for cinder here (just messing around at this stage) and I would like to propagate an error all the way back to the user behind horizon (or cli). I see that raised exceptions in cinder are not propagated via rabbitmq (as opposed to nova?). | 15:13 |
| *** c0m0 has quit IRC | 15:13 | |
| *** li_zhang has joined #openstack-cinder | 15:14 | |
| DuncanT | luv: Most operations are async, so the user is gone before the exception is raised | 15:14 |
| *** rongze has quit IRC | 15:15 | |
| luv | DuncanT: that might not be the case when the user is using horizon | 15:16 |
| *** markvoelker has quit IRC | 15:16 | |
| DuncanT | luv: The 'user' from a cinder PoV is whatever is at the end of the socket | 15:17 |
| jgriffith | nikesh: modify the local.conf template to fetch your changes from your own github repo | 15:17 |
| luv | DuncanT: i see no issue with reporting errors via rabbitmq and horizon processing them and showing them to the user | 15:18 |
| luv | all async and clean | 15:18 |
| DuncanT | luv: I believe you can configure the cinder logger to send all ERROR or EXCEPTION to rabbit | 15:18 |
| *** bswartz has quit IRC | 15:19 | |
| luv | nova has a wrap_exception decorator for reporting exceptions via rabbitmq (or other supported messaging system, ofc) | 15:19 |
| *** crose has quit IRC | 15:19 | |
| *** c0m0 has joined #openstack-cinder | 15:20 | |
| e0ne | luv: did you see realated conversation in the openstack-dev ML? | 15:20 |
| *** crose has joined #openstack-cinder | 15:20 | |
| luv | DuncanT: that might something to look at, would you mind pointing me at the correct docs? | 15:20 |
| *** timcl has quit IRC | 15:20 | |
| luv | e0ne: nope ... sounds interesting tho - will check it out | 15:20 |
| e0ne | luv: https://www.mail-archive.com/openstack-dev@lists.openstack.org/msg50001.html | 15:21 |
| luv | e0ne: yup, looks exactly like my problem. will check it out tomorrow. afk now :) | 15:22 |
| *** timcl has joined #openstack-cinder | 15:22 | |
| *** Mandell has quit IRC | 15:22 | |
| *** ronis has joined #openstack-cinder | 15:24 | |
| *** ozialien has joined #openstack-cinder | 15:26 | |
| nikesh | jgriffith : so in sos-ci/sos-ci/ansible/tasks/install_devstack.yml,do we have to modify cinder_branch with our patch ref | 15:26 |
| nikesh | means in valid event we can catch only our patch ref | 15:27 |
| nikesh | and it will update cinder branch | 15:27 |
| e0ne | jbernard: hi. are you aroung? | 15:28 |
| e0ne | s/aroung/around | 15:28 |
| jbernard | e0ne: i am, how goes it? | 15:28 |
| *** jungleboyj has quit IRC | 15:28 | |
| e0ne | jbernard: i'm reading your comment for https://review.openstack.org/#/c/145678/. you didn't add proposal to design session, did you? | 15:29 |
| jbernard | i did | 15:29 |
| jgriffith | nikesh: so the idea is to just fetch/cherry pick your driver in | 15:29 |
| jbernard | e0ne: i added a placeholder in the etherpad | 15:29 |
| *** jdurgin1 has joined #openstack-cinder | 15:29 | |
| jbernard | e0ne: but i haven't had a chance to post any proposed contents yet | 15:30 |
| jgriffith | nikesh: check the wiki, thingee wrote up a deal on it under the FAQ | 15:30 |
| *** ronis has quit IRC | 15:30 | |
| *** harlowja_at_home has joined #openstack-cinder | 15:31 | |
| *** russellb has quit IRC | 15:32 | |
| DuncanT | luv: I don't know if it is documented, sorry | 15:33 |
| *** li_zhang has quit IRC | 15:34 | |
| *** _cjones_ has joined #openstack-cinder | 15:34 | |
| *** jungleboyj has joined #openstack-cinder | 15:35 | |
| e0ne | jbernard: fyi, i'm looking on this issue too. i hope, i'll could provide some fix for it later this week | 15:36 |
| *** russellb has joined #openstack-cinder | 15:36 | |
| jbernard | e0ne: great, ill remove the session proposal | 15:36 |
| e0ne | jbernard: wait, please | 15:37 |
| jbernard | e0ne: no problem | 15:37 |
| e0ne | jbernard: 'When eventlet isn't enough; alternatives for long-running tasks' is it yours about the current bug? | 15:37 |
| jbernard | that's the one i added, yes | 15:37 |
| *** li_zhang has joined #openstack-cinder | 15:38 | |
| *** _cjones_ has quit IRC | 15:38 | |
| *** _cjones_ has joined #openstack-cinder | 15:38 | |
| e0ne | i'm confused a bit. the problem not about long-running tasks. | 15:38 |
| e0ne | the problem is in eventlet-blocking tasks:( | 15:38 |
| jbernard | exactly | 15:39 |
| *** ronis has joined #openstack-cinder | 15:39 | |
| jbernard | volume delete can be a long running task | 15:39 |
| jbernard | which causes eventlet to not be able to schedule others tasks | 15:39 |
| e0ne | yes | 15:39 |
| jbernard | thereby blocking forward progress | 15:39 |
| e0ne | but in general, that's a problem for short time tasks too | 15:40 |
| jgriffith | eventlet doesn't block | 15:40 |
| e0ne | jgriffith: c lib blocks eventlet thread | 15:40 |
| jbernard | right, that's what we mean | 15:41 |
| *** rongze has joined #openstack-cinder | 15:41 | |
| jgriffith | jbernard: e0ne ok, not going down this rat hole again :) | 15:41 |
| jbernard | jgriffith: smart man :) | 15:41 |
| *** li_zhang has quit IRC | 15:41 | |
| e0ne | :) | 15:41 |
| * jgriffith should've saved the code snippet to demonstrate the eventlet stuff | 15:42 | |
| *** li_zhang has joined #openstack-cinder | 15:42 | |
| jgriffith | may be different than what you two are looking at, but probably still valuable | 15:42 |
| jbernard | jgriffith: yeah, post it if you find it | 15:42 |
| jbernard | the basic problem in rbd is that volume delete may take a long time | 15:43 |
| jbernard | spawning a subprocess for each delete operation is not good | 15:43 |
| *** fanyaohong has joined #openstack-cinder | 15:43 | |
| jgriffith | jbernard: kk... I'll see if I can find it | 15:43 |
| jbernard | but waiting for one to return causes havoc for the other evenelet threads | 15:43 |
| e0ne | jgriffith: just to clarify. eventlet doesn't monkey patch C libs | 15:43 |
| jgriffith | e0ne: jbernard so are you looking at internal usage of eventlet in RDB? | 15:44 |
| jbernard | internal, as in making librbd evenelet-aware? | 15:44 |
| *** marcusvrn has joined #openstack-cinder | 15:44 | |
| jgriffith | e0ne: jbernard Ahh.. got ya | 15:44 |
| *** lpetrut has quit IRC | 15:45 | |
| e0ne | jbernard: long-running ceph jobs is one part of issue | 15:45 |
| jbernard | e0ne: agreed | 15:45 |
| *** marcusvrn1 has quit IRC | 15:46 | |
| e0ne | jbernard: in my case, it's blocked when librados doesn't respose to calls and we disable connect timeout by default | 15:46 |
| DuncanT | e0ne: Is patching librbd to be eventlet aware the solution? | 15:46 |
| jbernard | DuncanT: unlikely | 15:47 |
| e0ne | DuncanT: i'm looking on it. it uses threads in a strange way | 15:47 |
| jbernard | DuncanT: i spoke with josh about this, i don't think they want it upstream | 15:47 |
| DuncanT | jbernard: Any reason given? | 15:47 |
| *** Apoorva has joined #openstack-cinder | 15:48 | |
| jbernard | DuncanT: i believe the thread model complicates things, but i need to go dig through my logs | 15:48 |
| DuncanT | jbernard: exec might be the only sensible answer in the short term then | 15:48 |
| jbernard | DuncanT: that would work, but then you have the problem with large volume numbers | 15:49 |
| DuncanT | jbernard: Why? It is only doing an operation on them one at a time | 15:49 |
| jbernard | DuncanT: what happens when a second volume is requested for deletion? | 15:50 |
| jbernard | DuncanT: will it queue? | 15:50 |
| DuncanT | jbernard: By default, no, you can put a lock in the driver if needed, but generally it will fork up to n_workers copies in parallel - about 50 I believe | 15:51 |
| *** zhithuang has joined #openstack-cinder | 15:51 | |
| *** zhithuang is now known as winston-d_ | 15:51 | |
| DuncanT | jbernard: You can add locking to make it queue, but the ceph cmd line client supports a fair amount of parallelism | 15:51 |
| *** timcl has quit IRC | 15:52 | |
| e0ne | DuncanT, jbernard: if we do a queue, we need to remember that several cinder-volume services could be available | 15:53 |
| *** annegentle has joined #openstack-cinder | 15:54 | |
| jbernard | fair points | 15:54 |
| jbernard | i think the major concern is that deleting large rbd volumes can be CPU intensive, so system load can overtake the cinder node | 15:54 |
| DuncanT | e0ne: ceph seems quite happy with parallel operations AFAICT... the only reason to queue them in the driver qould be as a crude rate limit | 15:54 |
| e0ne | DuncanT: agree | 15:55 |
| *** thingee has joined #openstack-cinder | 15:55 | |
| e0ne | i hope, simple thread could resolve this issue, but i don't test it yet | 15:56 |
| *** jwcroppe has quit IRC | 15:56 | |
| *** deepakcs has joined #openstack-cinder | 15:56 | |
| *** jwcroppe has joined #openstack-cinder | 15:57 | |
| *** xyang has joined #openstack-cinder | 15:57 | |
| *** rongze has quit IRC | 15:59 | |
| *** bswartz has joined #openstack-cinder | 15:59 | |
| smcginnis | join #openstack-meeting | 15:59 |
| smcginnis | Oops. :) | 15:59 |
| winston-d_ | smcginnis: yes sir | 15:59 |
| *** vilobhmm1 has joined #openstack-cinder | 16:00 | |
| *** jwcroppe has quit IRC | 16:01 | |
| *** vilobhmm11 has joined #openstack-cinder | 16:02 | |
| openstackgerrit | Walter A. Boring IV (hemna) proposed openstack/os-brick: Brick: Fix race in removing iSCSI device https://review.openstack.org/168506 | 16:02 |
| *** emagana has quit IRC | 16:03 | |
| *** jistr has quit IRC | 16:03 | |
| *** jistr has joined #openstack-cinder | 16:04 | |
| *** vilobhmm1 has quit IRC | 16:04 | |
| *** marcusvrn1 has joined #openstack-cinder | 16:07 | |
| *** jistr has quit IRC | 16:09 | |
| *** marcusvrn has quit IRC | 16:09 | |
| *** emagana has joined #openstack-cinder | 16:10 | |
| *** harlowja_at_home has quit IRC | 16:10 | |
| *** jwcroppe has joined #openstack-cinder | 16:11 | |
| *** markvoelker has joined #openstack-cinder | 16:13 | |
| *** rushil has quit IRC | 16:13 | |
| *** dosaboy has quit IRC | 16:13 | |
| *** markus_z has quit IRC | 16:14 | |
| *** dosaboy has joined #openstack-cinder | 16:14 | |
| *** jistr has joined #openstack-cinder | 16:15 | |
| *** annegentle has quit IRC | 16:16 | |
| *** Yogi1 has quit IRC | 16:17 | |
| *** jistr has quit IRC | 16:17 | |
| *** timcl has joined #openstack-cinder | 16:18 | |
| *** _cjones_ has quit IRC | 16:19 | |
| *** markvoelker has quit IRC | 16:19 | |
| *** lpetrut has joined #openstack-cinder | 16:19 | |
| *** Yogi1 has joined #openstack-cinder | 16:21 | |
| *** _cjones_ has joined #openstack-cinder | 16:21 | |
| *** emagana has quit IRC | 16:22 | |
| *** sgotliv has quit IRC | 16:25 | |
| *** dustins has quit IRC | 16:26 | |
| *** marcusvrn has joined #openstack-cinder | 16:27 | |
| *** ociuhandu has quit IRC | 16:28 | |
| *** dannywilson has joined #openstack-cinder | 16:28 | |
| *** marcusvrn1 has quit IRC | 16:29 | |
| e0ne | winston-d_: did you see my messages about block device driver? ^^ | 16:29 |
| winston-d_ | e0ne: sorry, i'm on a different computer, I need to logon to the other one to see the history | 16:30 |
| e0ne | winston-d_: np. anyway, you can find it in irc logs too | 16:31 |
| winston-d_ | e0ne: sure, yeah, we have logs now | 16:31 |
| *** emagana has joined #openstack-cinder | 16:32 | |
| *** annashen has joined #openstack-cinder | 16:32 | |
| *** jungleboyj has quit IRC | 16:32 | |
| *** leeantho has joined #openstack-cinder | 16:32 | |
| *** li_zhang has quit IRC | 16:35 | |
| *** mwichmann has left #openstack-cinder | 16:35 | |
| *** li_zhang has joined #openstack-cinder | 16:35 | |
| *** thingee has quit IRC | 16:37 | |
| *** annashen has quit IRC | 16:37 | |
| *** annashen has joined #openstack-cinder | 16:37 | |
| *** jdurgin1 has quit IRC | 16:38 | |
| smcginnis | Zuuls looking pretty backed up again. | 16:38 |
| *** EmilienM is now known as EmilienM|afk | 16:38 | |
| *** jungleboyj has joined #openstack-cinder | 16:39 | |
| *** li_zhang has quit IRC | 16:40 | |
| *** c0m0 has quit IRC | 16:40 | |
| *** patrickeast has quit IRC | 16:41 | |
| *** asselin has quit IRC | 16:42 | |
| *** alecv has quit IRC | 16:43 | |
| *** dannywilson has quit IRC | 16:43 | |
| *** shyama has quit IRC | 16:43 | |
| *** markvoelker has joined #openstack-cinder | 16:44 | |
| *** jungleboyj has quit IRC | 16:44 | |
| *** rasoto_ has joined #openstack-cinder | 16:44 | |
| *** afazekas has joined #openstack-cinder | 16:47 | |
| rasoto_ | hello all, I am trying to find information on how can I create Cinder Volume Schedules to take snapshots , can someone point me in the right direction? | 16:48 |
| bswartz | thingee: you still around? | 16:50 |
| *** markvoelker has quit IRC | 16:50 | |
| *** Yogi11 has joined #openstack-cinder | 16:50 | |
| bswartz | ...looks like he bailed | 16:50 |
| smcginnis | rasoto_: Cron job? | 16:51 |
| e0ne | smcginnis: +1 :) | 16:51 |
| rasoto_ | there is no way to have a user/tenant create the schedules via the dashboard? | 16:52 |
| *** Yogi1 has quit IRC | 16:53 | |
| smcginnis | rasoto_: There's nothing in horizon that I'm aware of. | 16:53 |
| rasoto_ | smcginnis: and I am assuming there is nothing on cinder either, right? | 16:54 |
| smcginnis | rasoto_: Right, there's no snapshot scheduling service in cinder. | 16:55 |
| rasoto_ | I see | 16:55 |
| e0ne | rasoto_: you're right. cinder doesn't care on it | 16:55 |
| rasoto_ | well thank you everyone for the help :) | 16:55 |
| smcginnis | rasoto_: np. Doesn't sound like the answer you were looking for, but at least it's an answer. :) | 16:55 |
| *** e0ne has quit IRC | 16:56 | |
| rasoto_ | smcginnis: yeah, I wonder if there is something on the NetApp side since that is the service/driver I am using | 16:56 |
| smcginnis | rasoto_: I think most arrays have some sort of scheduled snapshot functionality. | 16:57 |
| *** dustins has joined #openstack-cinder | 16:57 | |
| *** jungleboyj has joined #openstack-cinder | 16:57 | |
| smcginnis | rasoto_: The problem is it's usually not integrated on the host side, so no guarantee the data is in a good state. | 16:57 |
| smcginnis | tbarron: You around? ^^ | 16:58 |
| tbarron | smcginnis: yessir | 16:58 |
| smcginnis | tbarron: Sorry, you're my go to NetApp guy. :) | 16:58 |
| *** avishay has quit IRC | 16:58 | |
| tbarron | smcginnis: wazzup? | 16:58 |
| *** bswartz has quit IRC | 16:59 | |
| smcginnis | tbarron: rasoto_ was looking for info on snapshot scheduling. Wondering if NetApp does anything special. | 16:59 |
| *** ozialien has quit IRC | 17:00 | |
| rasoto_ | tbarron: I can move this to the openstack-netapp channel if you want | 17:01 |
| tbarron | rasoto_: you can certainly set up a DOT snapshot schedule for NetApp backends. | 17:02 |
| tbarron | This is external to Cinder. | 17:02 |
| tbarron | An administrator might want to do that. | 17:03 |
| akerr | smcginnis: I believe we have special extra-specs to define a snapshot policy in Manila, but not currently in Cinder. But even then its a backend snapshot and not a manila facing snapshot | 17:03 |
| tbarron | Might be a good idea. | 17:03 |
| vilobhmm11 | DuncanT : just saw your msg | 17:03 |
| smcginnis | Yeah, might be worth thinking through for a bit. | 17:04 |
| rasoto_ | tbarron: can a FlexVol have multiple schedules? and can a user choose which one? | 17:04 |
| tbarron | It provides administrative backup as opposed to cinder backup, which is self-service backup for tenants. | 17:04 |
| akerr | might be best to move to #openstack-netapp so we don't spam all of cinder about our netapp specific implmentations | 17:04 |
| rasoto_ | akerr: sure | 17:04 |
| tbarron | akerr: +1 | 17:04 |
| smcginnis | xyang has an interesting proposal for snapshot used for backup. | 17:05 |
| smcginnis | I hope we can do something there. | 17:05 |
| *** Mandell has joined #openstack-cinder | 17:05 | |
| *** bswartz has joined #openstack-cinder | 17:06 | |
| tbarron | smcginnis: yes, that's an interesting proposal. I kinda like it. Cinder snapshots are per-volume, NetApp backend snapshots are per-flexvol, so it can be confusing ... | 17:06 |
| DuncanT | Anybody interested in adding scheduled snapshots / scheduled backups? It's a clean use-case AFAICT | 17:07 |
| smcginnis | tbarron: Yea. :) | 17:07 |
| smcginnis | DuncanT: Does it belong in cinder is my question. | 17:07 |
| smcginnis | DuncanT: But agree there is a use case for having it. | 17:07 |
| smcginnis | akerr: Thanks for pointing that out? Do you know where in the code that's handled? I'd like to take a look. | 17:09 |
| smcginnis | Not sure why I swapped punctuation on that. :) | 17:10 |
| DuncanT | smcginnis: Not sure if it belongs or not. Lots of arguments both ways | 17:11 |
| smcginnis | DuncanT: Yeah, that's how I feel. | 17:11 |
| xyang | DuncanT, smcginnis: that is an interesting and valid use case | 17:11 |
| DuncanT | smcginnis: Saying 'use cron' is not a good argument, since you need somewhere reliable to host the cron | 17:12 |
| smcginnis | DuncanT: I know from experience with other vendors solutions, they usually provide the means for snapping, but not scheduling. | 17:12 |
| smcginnis | DuncanT: At least it's not a very good end user experience. | 17:12 |
| xyang | smcginnis: backup product usually has scheduling | 17:12 |
| DuncanT | smcginnis: A time-based scheduling service for openstack might be a valid argument, but seems unlikely to get much traction | 17:12 |
| smcginnis | xyang: Yeah, that's why I'm wondering if it really belongs. Seems kind of like it should be handled by something else like a real backup solution. | 17:13 |
| akerr | smcginnis: https://github.com/openstack/manila/blob/79d3a38b195c7a73a2448ca2df7c9506fbe501a3/manila/share/drivers/netapp/dataontap/client/client_cmode.py#L773 | 17:13 |
| smcginnis | DuncanT: Agreed. | 17:13 |
| smcginnis | akerr: Thank you!! | 17:13 |
| xyang | smcginnis: right, it is debatable. cinder has backup service | 17:14 |
| *** ndipanov has quit IRC | 17:15 | |
| smcginnis | akerr: Do you know, is that an option specific to that driver, or is that common for any driver that chooses to implement it. | 17:15 |
| smcginnis | akerr: Not assuming you are the expert though. :) | 17:15 |
| akerr | smcginnis: I'm pretty sure its specific to the netapp driver, but I'll let bswartz correct me | 17:15 |
| xyang | akerr: snapshot created by using the policy are not exposed to Manila, correct? | 17:15 |
| akerr | xyang: correct | 17:15 |
| xyang | akerr: thanks | 17:16 |
| DuncanT | If we add it to cinder, we definitely want it to produce real cinder snapshots, not some hidden thing | 17:16 |
| smcginnis | So we could conceivable have an extra spec that sets up a snapshot schedule on the array for cinder. | 17:16 |
| tbarron | DuncanT: I can see use cases here, but all of them require getting the backup service to scale better as a prerequisite to fancier stuff. | 17:16 |
| DuncanT | Same behaviour as a cron job would have | 17:16 |
| smcginnis | DuncanT: +1 | 17:16 |
| akerr | DuncanT: +1 | 17:16 |
| smcginnis | DuncanT: Any requests for this in your public cloud? | 17:16 |
| DuncanT | tbarron: Improving the speed/throughput of backup or something else? | 17:17 |
| akerr | smcginnis: I've requested it from Rackspace cloud :) | 17:17 |
| DuncanT | smcginnis: A few people asked if it is supported, but didn't seem that desparate for it | 17:17 |
| tbarron | DuncanT: that, and probably also not having to load all the volume backends on the backup cinder node | 17:17 |
| *** Mandell has quit IRC | 17:17 | |
| smcginnis | akerr, DuncanT: Thanks | 17:17 |
| smcginnis | bbl, lunch train is leaving. :) | 17:17 |
| tbarron | DuncanT: should the cron-like service be part of Cinder or should it be from some OpenStack automation external to Cinder itself? | 17:18 |
| DuncanT | Any chance of a core who isn't Mike taking a quick look at https://review.openstack.org/#/c/149618/ please, I try not to +A things from my team | 17:19 |
| DuncanT | tbarron: I'm not sure there'll be much buy-in for an openstack-cron-aaS - what are the usecases? Nova instance snapshots maybe? cinder & manilla snaps, cinder backups, anything else? | 17:20 |
| DuncanT | tbarron: It is arguably useful just for those I guess... be a fun thing to design, not too big | 17:20 |
| tbarron | DuncanT: I'd be happy with doing it in cinder, but in a way that it could be pulled out and generalized. | 17:21 |
| DuncanT | Within the cinder umbrella for sure is a good way to bootstrap... I'd start with a clean git tree, unique endpoint etc from day one if we are going to support more than just cinder though | 17:22 |
| *** aix has quit IRC | 17:22 | |
| tbarron | DuncanT: I suspect there are other use cases that would emerge if there were a general service, but I certainly don't want to be another advocate of a service looking for problems to solve. | 17:22 |
| *** asselin has joined #openstack-cinder | 17:23 | |
| DuncanT | tbarron: If the list of verbs that can be scheduled is extensible, then I'd say there's enough in the above to justify the existance | 17:23 |
| *** Mandell has joined #openstack-cinder | 17:24 | |
| hemna | back | 17:25 |
| *** bkopilov has quit IRC | 17:26 | |
| *** Mandell has quit IRC | 17:26 | |
| hemna | DuncanT, isn't the scheduled snapshots accomplished by using Mistral? | 17:27 |
| hemna | scheduling workflows, etc. | 17:27 |
| hemna | not that I've used mistral | 17:27 |
| DuncanT | Never heard of it! | 17:27 |
| hemna | but still | 17:27 |
| hemna | https://wiki.openstack.org/wiki/Mistral | 17:27 |
| *** Mandell has joined #openstack-cinder | 17:27 | |
| DuncanT | hemna: Looks like a superset of the thing I was suggesting we design. Thanks for the pointer! | 17:28 |
| hemna | np | 17:28 |
| DuncanT | rasoto_: Are you still about? | 17:28 |
| *** Mandell has quit IRC | 17:29 | |
| vilobhmm11 | DuncanT : mistral , taskflow can do the same thing…mistral can be the service undelying using taskflow lib or something like thatå | 17:30 |
| *** Mandell has joined #openstack-cinder | 17:30 | |
| *** bkopilov has joined #openstack-cinder | 17:30 | |
| hemna | taskflow isn't for scheduling repeating jobs in the future | 17:31 |
| hemna | or even 1 time jobs | 17:31 |
| vilobhmm11 | DuncanT : tooz can also be used for distributed locking have you ever thought of using it for the problem(A-A) we discussed in the meeting today.. | 17:31 |
| vilobhmm11 | hemna : ^^ | 17:31 |
| geguileo | vilobhmm11: I've had a look into Tooz | 17:32 |
| vilobhmm11 | ok what are you findings geguileo ? | 17:32 |
| DuncanT | vilobhmm11: The A-A problem has many, many facets, and I really want to avoid jumping into a solution before we have a good grasp of at least most of the problems | 17:32 |
| geguileo | vilobhmm11: When looking into solutions for the atomic state changes | 17:32 |
| hemna | DuncanT, +1 | 17:32 |
| DuncanT | vilobhmm11: State is far from the only problem | 17:32 |
| vilobhmm11 | DuncanT : agree | 17:32 |
| geguileo | DuncanT: But it's one of them ;) | 17:32 |
| DuncanT | vilobhmm11: We can start with a big, clunky state lock once we fix nova | 17:33 |
| DuncanT | geguileo: It might be possible to make it not a problem though, by fixing other thigns | 17:33 |
| vilobhmm11 | DuncanT : my 2 cents for it | 17:33 |
| DuncanT | *things | 17:33 |
| vilobhmm11 | agree | 17:33 |
| vilobhmm11 | +1 | 17:33 |
| geguileo | DuncanT: You sure? | 17:33 |
| geguileo | DuncanT: From what I could gather it is a problem | 17:33 |
| DuncanT | geguileo: No, I'm not sure, but I know some parts of it can definitely go away | 17:34 |
| geguileo | DuncanT: But you still have the problem with transitions | 17:34 |
| DuncanT | geguileo: Once those parts have gone away, maybe the number of states and changes is small enough that 'update state=$new where id=$id and state=$state' will be enough | 17:34 |
| DuncanT | Just look at the count returned from that... if it is 1 then you just did a (low cost) atomic check-and-set transition | 17:35 |
| DuncanT | Nothing external needed | 17:35 |
| geguileo | DuncanT: That plus a retry on deadlock works | 17:35 |
| geguileo | DuncanT: Although a select for update plus retry on deadlocks also works | 17:35 |
| DuncanT | \select for update breaks in galera, and is slower | 17:36 |
| geguileo | DuncanT: It doesn't break in Galera, I have tested it :) | 17:36 |
| geguileo | DuncanT: It's not supported | 17:36 |
| *** harlowja_away is now known as harlowja | 17:36 | |
| DuncanT | Not supported == broken | 17:36 |
| geguileo | DuncanT: In the sense that it doesn't lock row on all nodes | 17:36 |
| hemna | tooz looks interesting | 17:36 |
| geguileo | Tooz doesn't solve the underlying problem | 17:36 |
| *** deepakcs has quit IRC | 17:37 | |
| hemna | for locks it might work | 17:37 |
| *** e0ne has joined #openstack-cinder | 17:37 | |
| geguileo | DuncanT: It's not broken, you would get the lock in one of the DBs and for the others collissions would be controlled by the retry on deadlock | 17:37 |
| hemna | would have to play with it though | 17:37 |
| hemna | tooz does locking in a db | 17:37 |
| geguileo | hemna: I have played with the 3 solutions | 17:37 |
| hemna | and has drivers for several db's, it looks like | 17:37 |
| geguileo | hemna: Yes, but it doesn't take into account the galera situation | 17:38 |
| hemna | which is ? | 17:38 |
| geguileo | hemna: It only locks in one DB, not in all the DB nodes | 17:38 |
| geguileo | hemna: So you need to whatchout for deadlocks | 17:38 |
| *** bkopilov has quit IRC | 17:39 | |
| *** bkopilov has joined #openstack-cinder | 17:39 | |
| hemna | is that really a problem Cinder needs to deal with though | 17:39 |
| hemna | I really don't think that's a use case that Cinder has to worry about | 17:40 |
| geguileo | I have tested 5 workes doing 100 state transitions (available-deleting-available) simultateoulsy against an HAProxy, round robind sending requests to 3 different nodes (each new request is sent to the next server) | 17:40 |
| hemna | I dunno. | 17:40 |
| geguileo | hemna: I think it is | 17:40 |
| geguileo | hemna: Otherwise you could have race conditions | 17:40 |
| hemna | frankly, I think using something like tooz would be far better off than we are today. | 17:40 |
| geguileo | hemna: Tooz alone doesn't solve it, I have checked it | 17:41 |
| hemna | tooz will move the locks from local to central | 17:41 |
| geguileo | It reduces to almost 0 the number of DB deadlocks, but these happen from time to time | 17:41 |
| vilobhmm11 | hemna : +1 thats why i brought that up :) | 17:41 |
| geguileo | hemna: Yes, and that could be fine for locks on the creation workflow (I still have to confirm) | 17:42 |
| vilobhmm11 | but one thing to be cautious is only locking is supported by the tooz db driver and other concepts like group membership etc are not supported | 17:42 |
| geguileo | vilobhmm11: I have tested tooz with 3 different backends and it's still not enough to avoid issues | 17:42 |
| DuncanT | hemna: It absolutely is a problem cinder has to deal with IMO, as is deploying against galera requires some haproxy madness to get a working install. We can fix those (quota and something else) but introducing more is madness | 17:42 |
| vilobhmm11 | hemna, DuncanT details : http://docs.openstack.org/developer/tooz/compatibility.html | 17:42 |
| *** cbits has joined #openstack-cinder | 17:43 | |
| geguileo | I have done the locking with Redis | 17:43 |
| geguileo | as backend | 17:43 |
| vilobhmm11 | geguileo : I would suggest you to please file bug against tooz if you are consistently seeing these issues. | 17:43 |
| geguileo | vilobhmm11: What issues? | 17:43 |
| DuncanT | tooz might well be a good solution, but I want to have the problem(s) spelt out very clearly first | 17:44 |
| vilobhmm11 | the issues you are dicussing about various tooz drivers you tested against | 17:44 |
| vilobhmm11 | DuncanT : +2 | 17:44 |
| geguileo | vilobhmm11: You mean that DB would not assure us a solution with galera configuration? | 17:44 |
| vilobhmm11 | I ahree | 17:44 |
| vilobhmm11 | agree DuncanT : ^^ | 17:44 |
| *** bkopilov has quit IRC | 17:44 | |
| geguileo | vilobhmm11: The issues are not specific to the driver, but the Galera cluster environment | 17:45 |
| geguileo | vilobhmm11: Where you have replication delays | 17:45 |
| DuncanT | At the moment there's no clear definition of what the state issue(s) are. My idea of what the state machine work was solving is definitely different to others | 17:45 |
| *** leeantho has quit IRC | 17:45 | |
| hemna | I'd just hate to see us give up because it doesn't solve ever single <1% use case | 17:45 |
| *** leeantho has joined #openstack-cinder | 17:45 | |
| geguileo | hemna: Solution proposed by DuncanT plus deadlock retry solves 100% of cases | 17:46 |
| geguileo | hemna: So does select for update + deadlock retry | 17:46 |
| DuncanT | I don't know how well my solution scales | 17:46 |
| geguileo | hemna: So does Tooz + any of above 2 solutions | 17:46 |
| vilobhmm11 | DuncanT : since i was the one working on the state machine work would like to answer questions if you have any ? But agree with hemna i would also hate to give up on this | 17:46 |
| DuncanT | 'select for update' does *not* work with galera. You don | 17:46 |
| DuncanT | t 't get atomicity | 17:46 |
| hemna | yah, but that doesn't work against every db. which is why tooz is an attempt at db locking which is db agnostic | 17:46 |
| geguileo | DuncanT: 5 workers fighting for the same row with 100 transitions (available-deleting-available) worked fine | 17:47 |
| hemna | so, maybe an option is to talk to the tooz dev and ping them about it? | 17:47 |
| *** markvoelker has joined #openstack-cinder | 17:47 | |
| DuncanT | geguileo: We need to see 500 workers fighting for the same row, or 5x100 workers fighting for the same 5 rows | 17:47 |
| geguileo | DuncanT: You don't get atomicity in all cluster, but when there's a collision you get a Deadlock situation | 17:47 |
| hemna | see what they think if they even support the galera deployment | 17:47 |
| vilobhmm11 | hemna : +1 ; | 17:47 |
| hemna | I'm not trying to say that tooz is the panacea, but it's worth looking into | 17:48 |
| geguileo | hemna: I agree, that's why I've been looking into it :) | 17:48 |
| geguileo | DuncanT: What do you mean 5x100 workers? | 17:48 |
| DuncanT | Definitely worth investigating. I want to start by documenting what situation(s) the locks are there to prevent, though | 17:48 |
| *** leeantho has quit IRC | 17:49 | |
| vilobhmm11 | hemna : agree…DuncanT : +1 | 17:49 |
| DuncanT | geguileo: 100 workers fighting for row 1, 100 fighting for row 2, .... row 5 | 17:49 |
| geguileo | DuncanT: You mean locks for state changes or the other local locks we currently have? | 17:49 |
| geguileo | DuncanT: I can test that | 17:50 |
| DuncanT | geguileo: Both | 17:50 |
| geguileo | DuncanT: What numbers would you like to see regarding that test? | 17:50 |
| *** e0ne is now known as e0ne_ | 17:50 | |
| *** erlon has joined #openstack-cinder | 17:50 | |
| DuncanT | geguileo: That test would be great. We should an order of magnitude bigger than the biggest system we expect, so 100 workers per row, 5 rows should be good | 17:50 |
| *** Mandell has quit IRC | 17:51 | |
| geguileo | DuncanT: I'll do that, but I want to make sure I get the right information so we can review it | 17:51 |
| *** Longgeek has quit IRC | 17:51 | |
| DuncanT | Some missing info on tooz docs, like do we need to clean locks up or is that done on release? | 17:51 |
| *** markvoelker has quit IRC | 17:52 | |
| geguileo | DuncanT: As far as I could see locks don't get cleaned up | 17:52 |
| geguileo | DuncanT: For example if you use a file, the file stays there | 17:52 |
| DuncanT | geguileo: First thing is does it fall over in a heap, second thing is how long does log acquire take (min, max, mean, stddev) | 17:52 |
| DuncanT | geguileo: There isn't an API to clean them up AFAICT.... that is bad | 17:53 |
| hemna | on a heavily loaded DB no less | 17:53 |
| hemna | :P | 17:53 |
| hemna | the time when you least want locks to fail, but are most likely to fail, is under heavy load | 17:53 |
| DuncanT | third thing is how much do other db accesses slow down while the test is running | 17:53 |
| geguileo | hemna: Mmmm, my tests are with no load on the DB since I deployed the HAProxy + galera proxy just for this test | 17:53 |
| *** Yogi11 has quit IRC | 17:53 | |
| *** Apoorva has quit IRC | 17:54 | |
| geguileo | What kind of additional load would you suggest? | 17:54 |
| DuncanT | geguileo: Can you write a really simple load generator to run at the same time? Just tight loop some selects and inserts in an unrelated table | 17:54 |
| DuncanT | Say 10 selects and 5 updates a second | 17:54 |
| DuncanT | Of an arbitary table | 17:54 |
| geguileo | DuncanT: Cool | 17:55 |
| geguileo | DuncanT: I can do that :) | 17:55 |
| geguileo | DuncanT: How many workers of that load? | 17:55 |
| DuncanT | geguileo: I dunno, 50 to start with? That represents a fairly hefty load for cinder, but might be realistic for a shared openstack db | 17:56 |
| *** Apoorva has joined #openstack-cinder | 17:56 | |
| geguileo | DuncanT: Ok | 17:56 |
| DuncanT | geguileo: If you can put the test script(s) on github or something then other people can examine them if they want to | 17:56 |
| *** annashen has quit IRC | 17:57 | |
| geguileo | DuncanT: I'll start modifying my environmet and code to accomodate those changes 3 changes (number of workers, numbers to report, load generator) | 17:57 |
| rasoto_ | DuncanT: sorry step away from the computer, I am here now | 17:58 |
| *** timcl1 has joined #openstack-cinder | 17:58 | |
| geguileo | DuncanT: Ok, once I've made the changes I'll upload the code so other can review it with the results | 17:58 |
| harlowja | hemna tooz dev here :-P | 17:58 |
| * harlowja is tooz dev, haha | 17:58 | |
| *** openstackstatus has quit IRC | 17:58 | |
| harlowja | u can't escape me! | 17:58 |
| harlowja | lol | 17:58 |
| *** openstackstatus has joined #openstack-cinder | 17:59 | |
| *** ChanServ sets mode: +v openstackstatus | 17:59 | |
| *** fanyaohong has quit IRC | 17:59 | |
| DuncanT | harlowja: Any feedback on lock cleanup? | 17:59 |
| harlowja | unsure | 18:00 |
| harlowja | lol | 18:00 |
| harlowja | what means :-P | 18:00 |
| *** timcl has quit IRC | 18:00 | |
| DuncanT | harlowja: If I create a lock called '1'. lock it, unlock it, create a lock called '2', etc, for a billion locks, does that leave a billion db rows/files/etc lying around? | 18:00 |
| hemna | harlowja, Tooz! | 18:01 |
| hemna | heh awesome. | 18:01 |
| harlowja | u can run but u can't hide | 18:01 |
| harlowja | the josh monster eat u | 18:01 |
| harlowja | lol | 18:01 |
| hemna | hehe | 18:01 |
| harlowja | DuncanT so this is assuming a DB lock? | 18:01 |
| harlowja | the tooz one is just using inbuilt mysql/postgres locks, not its own table | 18:01 |
| harlowja | *at least currently | 18:01 |
| harlowja | bb, meeting | 18:02 |
| -openstackstatus- NOTICE: Gerrit has stopped emitting events so Zuul is not alerted to changes. We will restart Gerrit shortly to correct the problem. | 18:02 | |
| *** ChanServ changes topic to "Gerrit has stopped emitting events so Zuul is not alerted to changes. We will restart Gerrit shortly to correct the problem." | 18:02 | |
| *** ozialien has joined #openstack-cinder | 18:02 | |
| geguileo | Yes, it's using SELECT GET_LOCK | 18:02 |
| *** annashen has joined #openstack-cinder | 18:02 | |
| geguileo | That's why it's not "Galera safe" | 18:02 |
| DuncanT | Ah, ok, thanks | 18:03 |
| geguileo | Redis is probably the best backend | 18:03 |
| *** cbits has quit IRC | 18:03 | |
| *** patrickeast has joined #openstack-cinder | 18:03 | |
| *** annegentle has joined #openstack-cinder | 18:04 | |
| DuncanT | We can't expect people to deploy redis though :-( | 18:04 |
| geguileo | That's what Ceilometer is using | 18:04 |
| geguileo | Tooz + Redis | 18:04 |
| DuncanT | I've got to head off now, sorry, cab home just turned up | 18:04 |
| geguileo | DuncanT: See ya | 18:05 |
| *** e0ne_ is now known as e0ne | 18:05 | |
| DuncanT | Lots of people aren't interested in ceiliometer | 18:05 |
| smcginnis | rasoto_: I think the ping was to point out the mistral project. | 18:07 |
| *** bswartz has quit IRC | 18:07 | |
| smcginnis | rasoto_: Might be an answer to scheduling snapshots. | 18:07 |
| smcginnis | Just catching up after getting back from lunch though... | 18:07 |
| hemna | doh | 18:07 |
| hemna | is gerrit down ? | 18:07 |
| hemna | oh | 18:07 |
| vilobhmm11 | yes | 18:07 |
| hemna | hehe...I guess I can't read. nm | 18:07 |
| vilobhmm11 | got notification | 18:07 |
| rasoto_ | smcginnis: ok, ty :) I will take a look a it | 18:08 |
| vilobhmm11 | ChanServ has changed the topic to: Gerrit has stopped emitting events so Zuul is not alerted to changes. We will restart Gerrit shortly to correct the problem. | 18:08 |
| *** Mandell has joined #openstack-cinder | 18:13 | |
| *** dosaboy has quit IRC | 18:13 | |
| *** dosaboy has joined #openstack-cinder | 18:14 | |
| *** dannywilson has joined #openstack-cinder | 18:15 | |
| *** dosaboy has quit IRC | 18:20 | |
| *** dosaboy has joined #openstack-cinder | 18:21 | |
| *** EmilienM|afk is now known as EmilienM | 18:22 | |
| *** wormwood has joined #openstack-cinder | 18:22 | |
| *** dosaboy has quit IRC | 18:22 | |
| *** dosaboy has joined #openstack-cinder | 18:24 | |
| wormwood | greetings all, I'm writing a simple script that uses the cinderclient lib and was wondering how can I can force a new volume object to reload it's stale data. Horizon is showing me that it attached to the instance, but new_volume.status still shows it as available | 18:24 |
| *** dosaboy has quit IRC | 18:24 | |
| *** dosaboy has joined #openstack-cinder | 18:24 | |
| *** ChanServ changes topic to "The OpenStack Block Storage Project Cinder | The New Kids On the Block | https://wiki.openstack.org/wiki/Cinder" | 18:24 | |
| -openstackstatus- NOTICE: Gerrit has been restarted. New patches, approvals, and rechecks between 17:30 and 18:20 UTC may have been missed by Zuul and will need rechecks or new approvals added. | 18:24 | |
| *** bswartz has joined #openstack-cinder | 18:28 | |
| *** e0ne has quit IRC | 18:30 | |
| patrickeast | asselin: hey, i wanted to follow up with you about having multiple nic’s on nodepool nodes for iscsi networks | 18:31 |
| patrickeast | asselin: what issues did you run into with it? | 18:31 |
| *** angela-s has joined #openstack-cinder | 18:31 | |
| *** winston-d_ has quit IRC | 18:34 | |
| *** thingee has joined #openstack-cinder | 18:34 | |
| *** tshefi has quit IRC | 18:34 | |
| *** annegentle has quit IRC | 18:35 | |
| *** annegentle has joined #openstack-cinder | 18:35 | |
| *** eharney has quit IRC | 18:36 | |
| *** Yogi1 has joined #openstack-cinder | 18:39 | |
| smcginnis | Jenkins appears to be chugging along again. | 18:40 |
| *** bswartz has quit IRC | 18:44 | |
| *** bswartz has joined #openstack-cinder | 18:44 | |
| *** Mandell has quit IRC | 18:45 | |
| *** rushil has joined #openstack-cinder | 18:46 | |
| harlowja | DuncanT geguileo is correct, without tooz having a db migration/schema (and its own db...) the other drivers are better suited for distributed locking | 18:46 |
| harlowja | having its own db in a library is not exactly common (taskflow does it, but its not super-optimial there either) | 18:47 |
| asselin | patrickeast, hi, the issue is on the openstack side. how to configure it | 18:51 |
| asselin | patrickeast, provider side | 18:51 |
| patrickeast | asselin: gotcha | 18:51 |
| asselin | what I was trying to do (and failed) was to have a local.conf local.sh devstack able to preconfigure it to 'just work' | 18:52 |
| *** ho has joined #openstack-cinder | 18:53 | |
| patrickeast | asselin: so we ended up doing 3 flat networks, public_mgmt, public_iscsi1, public_iscsi2, each with a bridge to a physical adapter on an initiator and a range of ip’s to use | 18:53 |
| patrickeast | asselin: then a private_mgmt network which uses floating ip’s for internet access (from the public_mgmt net) | 18:54 |
| patrickeast | asselin: and we directly attach nodes to the public_iscsi ones | 18:54 |
| patrickeast | where we let neutron do dhcp | 18:54 |
| asselin | patrickeast, do you have scripts to setup? or manual? | 18:55 |
| patrickeast | asselin: for the provider setup its manual (we have a small number of big hosts), but once the networking is set up all we did was add them to the nodepool image setup as additonal networks and everything just worked | 18:55 |
| asselin | patrickeast, we're still using nova-network. I tried neutron but gave up there too.... | 18:56 |
| rhe00 | patrickeast: that's the set up I have as well | 18:56 |
| *** Mandell has joined #openstack-cinder | 18:57 | |
| asselin | patrickeast, I'm not so familar with networking so the manual setup has been a struggle :/ | 18:57 |
| patrickeast | heh yea, we used RDO to setup our nodes and i was excited thinking the networking would be taken care of | 18:58 |
| patrickeast | the first step on our wiki of how to set up another provider is ‘delete all the existing network configuration' | 18:58 |
| asselin | lol | 18:58 |
| patrickeast | took us a while to figure out how to make it work | 18:58 |
| asselin | patrickeast, would be great if there was a devstack plugin to automate it.... | 19:00 |
| patrickeast | yea | 19:00 |
| asselin | btw, we're using devstack to setup the providers | 19:00 |
| *** bswartz has quit IRC | 19:00 | |
| asselin | I figured maybe when neutron become default in devstack we can switch and go from there | 19:01 |
| patrickeast | ah yea, thats a good idea | 19:01 |
| patrickeast | i assume you can do this same setup with nova-net, but i don’t know how | 19:01 |
| asselin | my original one worked on nova-net | 19:01 |
| asselin | back in icehouse | 19:02 |
| *** bswartz has joined #openstack-cinder | 19:03 | |
| patrickeast | heres some copy-pasta from our internal wiki that describes how to set it up http://paste.openstack.org/show/204027/ | 19:03 |
| *** crose has quit IRC | 19:03 | |
| * asselin looks | 19:03 | |
| patrickeast | formatting didn’t carry over well, but it shows the gist of how the networking got initialized | 19:03 |
| *** annashen has quit IRC | 19:04 | |
| *** eharney has joined #openstack-cinder | 19:04 | |
| * patrickeast notices a few typos in the configs | 19:05 | |
| asselin | patrickeast, this is really great. thank you for sharing | 19:07 |
| openstackgerrit | Jay Bryant proposed openstack/os-brick: Use oslo_log instead of openstack.common.log https://review.openstack.org/173106 | 19:07 |
| jgriffith | patrickeast: asselin if you're ever interested... might be useful: http://goo.gl/dp5Btc | 19:07 |
| asselin | patrickeast, when I get the courage to try again, I will follow your tips :) | 19:07 |
| patrickeast | asselin: np, figure others can benefit from my and dannywilson banging our heads on the wall for a week or so | 19:07 |
| jgriffith | but if you're forging bravely into neutron you're golden :) | 19:07 |
| patrickeast | :o nova network looks so much easier to use | 19:08 |
| jgriffith | I don't use DHCP on the second nic because it got kinda wonky in my ansible stuff | 19:08 |
| jgriffith | patrickeast: yeah... my big complaint about neutron | 19:08 |
| asselin | jgriffith, thanks will look at that too | 19:09 |
| jgriffith | patrickeast: it's just "harder" | 19:09 |
| jgriffith | asselin: you bet | 19:09 |
| * dannywilson remembers crying into my beer trying to get those steps finalized. :) | 19:09 | |
| patrickeast | lol | 19:09 |
| openstackgerrit | Jay Bryant proposed openstack/os-brick: Use oslo_log instead of openstack.common.log https://review.openstack.org/173106 | 19:10 |
| *** Yogi1 has quit IRC | 19:17 | |
| *** jaypipes has quit IRC | 19:18 | |
| *** jaypipes has joined #openstack-cinder | 19:21 | |
| *** jwcroppe has quit IRC | 19:21 | |
| *** jwcroppe has joined #openstack-cinder | 19:22 | |
| *** sgotliv has joined #openstack-cinder | 19:24 | |
| *** ronis has quit IRC | 19:26 | |
| *** jwcroppe has quit IRC | 19:26 | |
| *** xyang has quit IRC | 19:27 | |
| *** david-lyle has joined #openstack-cinder | 19:30 | |
| *** winston-d_ has joined #openstack-cinder | 19:30 | |
| *** e0ne has joined #openstack-cinder | 19:30 | |
| *** Mandell has quit IRC | 19:35 | |
| *** winston-d_ has quit IRC | 19:35 | |
| *** Mandell has joined #openstack-cinder | 19:38 | |
| openstackgerrit | xing-yang proposed openstack/cinder: Fixed issue with mismatched config in VMAX driver https://review.openstack.org/174044 | 19:40 |
| *** e0ne has quit IRC | 19:41 | |
| *** timcl1 has quit IRC | 19:42 | |
| *** markvoelker has joined #openstack-cinder | 19:50 | |
| *** ozialien has quit IRC | 19:50 | |
| *** kmartin has quit IRC | 19:51 | |
| *** jwcroppe has joined #openstack-cinder | 19:51 | |
| *** Rockyg has joined #openstack-cinder | 19:51 | |
| *** rushil has quit IRC | 19:52 | |
| *** jwcroppe has quit IRC | 19:53 | |
| *** bkopilov has joined #openstack-cinder | 19:54 | |
| *** markvoelker has quit IRC | 19:55 | |
| *** sgotliv has quit IRC | 19:56 | |
| thingee | mtreinish: so volume boot pattern issue still exists. I guess we have to skip? | 19:59 |
| thingee | mtreinish: just been having people with third party ci asking | 19:59 |
| *** dustins has quit IRC | 20:00 | |
| mtreinish | thingee: it works fine in the gate, and ceph fixed the bug they were having with it | 20:00 |
| *** Yogi1 has joined #openstack-cinder | 20:00 | |
| mtreinish | I still contend people are hitting issues with their ci configs or backends | 20:00 |
| mtreinish | because it's the first test many of them have run which actually logins into a guest and does any verification of the volume | 20:01 |
| thingee | yeah I'm even hitting issues. was working fine since april 7ish | 20:02 |
| thingee | mtreinish: ^ | 20:02 |
| mtreinish | thingee: hmm, I don't think we merged anything new then which would have effected that test | 20:03 |
| mtreinish | (well except for a temporary ceph skip which the revert is up for) | 20:03 |
| *** nkrinner has joined #openstack-cinder | 20:03 | |
| thingee | mriedem: I thought when we spoke at the cinder meeting there was a general issue with volumebootpattern, not just ceph? | 20:04 |
| *** annashen has joined #openstack-cinder | 20:04 | |
| *** Yogi1 has quit IRC | 20:05 | |
| mtreinish | thingee: so the only thing I've seen about that was: https://review.openstack.org/#/c/171569/ | 20:05 |
| mtreinish | which jgriffith had some issue with, which is why I've held off on it | 20:05 |
| mtreinish | thingee: oh and gluster complained that it's impossible for the test to work with them | 20:06 |
| *** rushiagr is now known as rushiagr_away | 20:06 | |
| mtreinish | but that's different :) | 20:06 |
| mriedem | thingee: there is, with glusterfs and gpfs also | 20:06 |
| mriedem | thingee: mtreinish: https://review.openstack.org/#/c/173408/ | 20:06 |
| mriedem | we also need this to stop running the ceph gate job on stable https://review.openstack.org/#/c/173319/ | 20:07 |
| mriedem | since it won't work on stable icehouse/juno | 20:07 |
| *** ronis has joined #openstack-cinder | 20:07 | |
| mriedem | mtreinish: you want to pull some infra strings there? :) | 20:07 |
| mtreinish | well that regex is wrong | 20:07 |
| mtreinish | oh the stable thing sure we can poke people on that | 20:07 |
| *** jwcroppe has joined #openstack-cinder | 20:07 | |
| *** annashen has quit IRC | 20:08 | |
| *** jwcroppe has quit IRC | 20:08 | |
| mriedem | yeah i already took a karma hit yesterday for an infra review request | 20:08 |
| *** annashen has joined #openstack-cinder | 20:08 | |
| mriedem | save throw failed, etc | 20:08 |
| *** dustins has joined #openstack-cinder | 20:09 | |
| *** ronis has quit IRC | 20:11 | |
| mtreinish | thingee: if that patch (171569) I linked before seems valid to you feel free to leave your feedback on it | 20:12 |
| mtreinish | I was just defering to jgriffith | 20:12 |
| *** timcl has joined #openstack-cinder | 20:14 | |
| thingee | mtreinish, mriedem https://objects.dreamhost.com/datera/console.txt | 20:14 |
| thingee | I fail both volumebootpattern and volumebootpatternv2 | 20:16 |
| mtreinish | thingee: oh fun failure to schedule the server boot | 20:16 |
| *** rushil has joined #openstack-cinder | 20:16 | |
| thingee | mtreinish: bb sorry | 20:16 |
| mtreinish | thingee: is there a log file index I can look at? | 20:16 |
| mtreinish | I need to see the nova logs to figure out why it didn't think it could boot the server | 20:16 |
| thingee | mtreinish: I can upload it when I get back, sorry. | 20:17 |
| thingee | mtreinish: and thank you | 20:17 |
| mtreinish | thingee: no worries, thanks | 20:18 |
| *** rushiagr_away is now known as rushiagr | 20:24 | |
| *** kmartin has joined #openstack-cinder | 20:25 | |
| *** Yogi1 has joined #openstack-cinder | 20:26 | |
| rmstar | hi guys. just a quick question about cinder create...when it is creating the volume from an image, is there a way to see the percentage / progress? ie. how far along is the "download" ? | 20:28 |
| *** bswartz has quit IRC | 20:28 | |
| hemna | jungleboyj, ping | 20:31 |
| ameade | Hey folks, I have two really simple driver patches i want to speed up merging. Been up since Feb | 20:31 |
| ameade | https://review.openstack.org/#/c/157439/ | 20:32 |
| ameade | https://review.openstack.org/#/c/170580/ | 20:32 |
| ameade | if anyone has a chance, should be an easy review but the churn is starting to get in the way WRT rebases | 20:32 |
| ameade | thanks in advance | 20:32 |
| jungleboyj | hemna: What up? | 20:33 |
| hemna | jungleboyj, https://review.openstack.org/#/c/173106/ | 20:33 |
| hemna | question for you in there I think | 20:33 |
| jungleboyj | hemna: :-) Thank you. | 20:35 |
| jungleboyj | Swamped today. | 20:35 |
| hemna | ah crap sorry | 20:35 |
| hemna | I'm trying to get theses brick patches done | 20:35 |
| jungleboyj | hemna: Don't apologize. | 20:35 |
| jgriffith | rmstar: nope | 20:36 |
| jgriffith | mtreinish: hey... so I don't necessarily have an "issue" with it per say | 20:36 |
| rmstar | thanks jgriffith | 20:36 |
| jgriffith | mtreinish: It's proceduraly correct I think | 20:36 |
| jgriffith | mtreinish: I was just bothered by the fact that something else seems to "handle" it for the majority of the drivers | 20:37 |
| jgriffith | mtreinish: LVM, SolidFire etc | 20:37 |
| jgriffith | mtreinish: and now with the jbernard 's fix, RBD | 20:37 |
| jgriffith | mtreinish: that being said, I don't see any reason NOT to merge it either. | 20:37 |
| jungleboyj | hemna: Updated. | 20:38 |
| vilobhmm11 | jungelboyj : thanks | 20:39 |
| vilobhmm11 | +1 | 20:39 |
| rmstar | jgriffith: just out of curiousity. Can anyone just register a wishlist? or do i have to go through you guys? | 20:40 |
| vilobhmm11 | jungelboyj : out of curiosity, so should every service start using oslo_log than using the logging module ? | 20:41 |
| jgriffith | rmstar: You are more than welcome to register one | 20:41 |
| vilobhmm11 | jungleboyj : ^^ | 20:43 |
| mtreinish | jgriffith: ok fair enough, yeah I couldnt figure out why it seems to work for most cases either | 20:45 |
| mtreinish | jgriffith: I think I was probably confusing that discussion with the one around the gluster stuff | 20:45 |
| mtreinish | they did happen at the same time... | 20:45 |
| jungleboyj | vilobhmm11: Yes, all everything outside of oslo-incubator should be using oslo_log . | 20:45 |
| jgriffith | mtreinish: yeah, that was the one I objected to skipping the test :) | 20:46 |
| jgriffith | mtreinish: which given another 24 hours got fixed in the driver as it should have :) | 20:46 |
| hemna | xyang1, ping | 20:47 |
| *** rasoto_ has left #openstack-cinder | 20:47 | |
| mtreinish | yeah, we definitely agreed on that :) | 20:47 |
| *** dustins has quit IRC | 20:48 | |
| anteaya | thingee: are you about? | 20:53 |
| anteaya | thingee: I'm reading the logs from last week's cinder meeting regarding ceph and want to hear from you what your current understanding is | 20:53 |
| anteaya | I want to make sure I understand what you are thinking | 20:54 |
| openstackgerrit | Jay Bryant proposed openstack/os-brick: Sync latest _i18n module for os_brick https://review.openstack.org/174096 | 20:54 |
| jungleboyj | Captain Oslo .... away! | 20:54 |
| hemna | strikes again! | 20:55 |
| hemna | :) | 20:55 |
| jungleboyj | :-) | 20:57 |
| *** rushil has quit IRC | 20:57 | |
| *** jungleboyj has quit IRC | 21:04 | |
| *** Yogi1 has quit IRC | 21:05 | |
| *** akerr has quit IRC | 21:05 | |
| *** Mandell has quit IRC | 21:06 | |
| *** Mandell has joined #openstack-cinder | 21:08 | |
| *** jwcroppe has joined #openstack-cinder | 21:10 | |
| *** jwcroppe has quit IRC | 21:10 | |
| *** Lee1092 has quit IRC | 21:10 | |
| *** esker has quit IRC | 21:12 | |
| *** esker has joined #openstack-cinder | 21:15 | |
| *** jkraj has quit IRC | 21:16 | |
| *** rushiagr is now known as rushiagr_away | 21:24 | |
| xyang1 | hemna: Hi | 21:24 |
| hemna | xyang1, https://bugs.launchpad.net/nova/+bug/1382440 | 21:26 |
| openstack | Launchpad bug 1382440 in OpenStack Compute (nova) "Detaching multipath volume doesn't work properly when using different targets with same portal for each multipath device" [High,Fix committed] - Assigned to Hiroyuki Eguchi (h-eguchi) | 21:26 |
| hemna | do you remember that one ? | 21:26 |
| xyang1 | Ya, is it a backport? | 21:27 |
| xyang1 | I have to fix it for vnx multipath to work | 21:27 |
| xyang1 | What is the question? | 21:28 |
| *** nkrinner has quit IRC | 21:28 | |
| xyang1 | That fix is also in brick | 21:29 |
| *** lpetrut has quit IRC | 21:29 | |
| xyang1 | hemna: I'll have take a closer look later. | 21:32 |
| thingee | mtreinish: hey back | 21:33 |
| thingee | mtreinish: http://stkci.daterainc.com/refs-changes-63-161363-6/index.html | 21:33 |
| hemna | xyang1, so I'm looking through the libvirt volume code for iSCSI | 21:33 |
| thingee | mtreinish: I need to disappear again | 21:33 |
| hemna | and trying to reconcile the differences with os-brick | 21:33 |
| thingee | anteaya: I'll get back to you. have a meeting. | 21:33 |
| xyang1 | hemna: If they only test with LVM, that will break other drivers | 21:33 |
| anteaya | thingee: thanks | 21:33 |
| *** jwcroppe has joined #openstack-cinder | 21:36 | |
| thingee | mtreinish: back | 21:41 |
| thingee | mtreinish: let me know if you need anymore information. | 21:41 |
| thingee | anteaya: just continuing the convo in #openstack-infra | 21:42 |
| anteaya | thingee: yup thanks | 21:42 |
| anteaya | thingee: also if you can look at this thread: http://lists.openstack.org/pipermail/openstack-infra/2015-April/002647.html | 21:42 |
| anteaya | thingee: do you need anything from infra at this point? I'm not sure who this person is and how much time you have already spent on/with them | 21:43 |
| *** Mandell has quit IRC | 21:43 | |
| mtreinish | thingee: so cinder is returning 500 during test volume boot pattern, check the n-cpu log around: 2015-04-15 20:01:50.797 | 21:45 |
| mtreinish | thingee: I'll check the cinder logs next to see why | 21:45 |
| *** dims_ has joined #openstack-cinder | 21:46 | |
| *** jamielennox|away is now known as jamielennox | 21:46 | |
| mtreinish | thingee: oh actually there were failures before that in the log | 21:46 |
| *** Rockyg has quit IRC | 21:46 | |
| mtreinish | (for that test) | 21:46 |
| mtreinish | thingee: like Stderr: u'iscsiadm: No session found.\n' | 21:46 |
| thingee | mtreinish: right | 21:47 |
| mtreinish | yeah that looks like the cause of the failure to boot (which results in the tempest error) | 21:47 |
| thingee | anteaya: I don't think we've discussed this case yet. We wrote this for people that are trying to test a driver not in master yet https://wiki.openstack.org/wiki/Cinder/tested-3rdParty-drivers#How_do_I_run_my_CI_to_test_all_cinder_patches_with_my_driver_not_yet_merged.3F | 21:47 |
| *** annegentle has quit IRC | 21:47 | |
| thingee | mtreinish: yea | 21:48 |
| *** dims has quit IRC | 21:48 | |
| thingee | mtreinish: so I'm not sure what changed there | 21:49 |
| anteaya | thingee: thanks I didn't know about that portion of the wikipage | 21:49 |
| hemna | crap | 21:51 |
| mtreinish | thingee: yeah dunno. My gut feeling is something happened to the the tgtd config or something | 21:51 |
| mtreinish | but I don't think we've made devstack changes or anythign there lately | 21:52 |
| *** annegentle has joined #openstack-cinder | 21:52 | |
| hemna | thingee, when you are free | 21:52 |
| *** vilobhmm11 has quit IRC | 21:53 | |
| mtreinish | thingee: I doubt you guys would have landed a cinder change to break that considering the point in the cycle | 21:53 |
| thingee | mtreinish: yeah | 21:53 |
| thingee | ok, I'll do a little more looking and let you know | 21:53 |
| *** markvoelker has joined #openstack-cinder | 21:53 | |
| *** nikesh has quit IRC | 21:54 | |
| thingee | bummer I really thought this was related to the other issue ceph was hitting and was hoping it would disappear | 21:54 |
| thingee | I've been having the ci disabled since | 21:54 |
| xyang1 | hemna: Hi, the fix is probably fine. It preserved my change for the case when each iqn is associated with a portal | 21:55 |
| asselin | thingee, which issue? | 21:55 |
| hemna | xyang1, so I have 2 bugs that have been 'fixed' in nova's libvirt volume drivers that aren't in cinder's brick (and obviously os-brick) | 21:55 |
| hemna | I'm wondering if I should file new bugs against os-brick for this ? | 21:55 |
| hemna | https://bugs.launchpad.net/cinder/+bug/1367189 | 21:55 |
| openstack | Launchpad bug 1367189 in Cinder "multipath not working with Storwize backend if CHAP enabled" [High,Fix released] - Assigned to TaoBai (baitao2020) | 21:55 |
| hemna | and | 21:55 |
| hemna | https://bugs.launchpad.net/nova/+bug/1382440 | 21:55 |
| openstack | Launchpad bug 1382440 in OpenStack Compute (nova) "Detaching multipath volume doesn't work properly when using different targets with same portal for each multipath device" [High,Fix committed] - Assigned to Hiroyuki Eguchi (h-eguchi) | 21:56 |
| hemna | neither of those made it into cinder/brick | 21:56 |
| hemna | I realized it after I started working on a nova patch to use os-brick | 21:56 |
| xyang1 | hemna: ok, sure, need a bug for them | 21:56 |
| thingee | asselin: I started noticing all the volumebootpatterns failing for me around the 7th http://stkci.daterainc.com/refs-changes-63-161363-6/index.html | 21:56 |
| * asselin looks | 21:57 | |
| *** leeantho has joined #openstack-cinder | 21:57 | |
| asselin | thingee, consistently or intermittently? | 21:57 |
| mtreinish | thingee: digging more through the logs it looks like cinder isn't having any issue talking to that target like 2 sec before nova fails | 21:57 |
| asselin | thingee, we're seeing that issue recently intermittently | 21:58 |
| asselin | "Details: {u'code': 400, u'message': u'Security group is still in use' | 21:58 |
| hemna | xyang1, it doesn't look like os-brick is a project in launchpad | 21:58 |
| thingee | asselin: consistent...I've tried by hand a few times | 21:58 |
| *** markvoelker has quit IRC | 21:58 | |
| thingee | asselin: can do another couple of runs to verify | 21:58 |
| hemna | anteaya, ping | 21:59 |
| *** Mandell has joined #openstack-cinder | 21:59 | |
| *** primechuck has quit IRC | 21:59 | |
| anteaya | hello hemna | 22:00 |
| anteaya | how can I help? | 22:00 |
| hemna | so I'm not sure if you are the right person to ask | 22:00 |
| anteaya | let's find out | 22:00 |
| hemna | but it seems that the os-brick (Subproject of cinder) isn't an official launchpad project | 22:01 |
| hemna | so I can't assign bugs to it | 22:01 |
| hemna | python-cinderclient is a project in launchpad | 22:01 |
| thingee | hemna: hi | 22:01 |
| hemna | I think os-brick should be a project in launchpad, so we can assign bugs to it | 22:01 |
| anteaya | hemna: http://docs.openstack.org/infra/manual/creators.html#set-up-launchpad | 22:01 |
| hemna | I'm not sure how that happens | 22:01 |
| anteaya | take a look at that part of the infra manual | 22:01 |
| hemna | ok thanks | 22:02 |
| anteaya | if you are still unhappy ping again | 22:02 |
| anteaya | welcome | 22:02 |
| hemna | thingee, do you agree ? | 22:02 |
| *** esker has quit IRC | 22:02 | |
| thingee | hemna: yup no problem with that | 22:02 |
| hemna | (I have 2 bugs that I need to append to mark them as bugs in os-brick) | 22:02 |
| hemna | the 2 I mentioned above | 22:02 |
| *** annegentle has quit IRC | 22:03 | |
| anteaya | you see that cinder has a group already: http://git.openstack.org/cgit/openstack-infra/project-config/tree/gerrit/projects.yaml#n740 | 22:03 |
| anteaya | do you want os-brick to have its own bugs page or should they be folded into cinder bugs? | 22:03 |
| anteaya | most folks reporting a bug won't be able to descern the difference and will put them in the wrong place anyhow | 22:04 |
| jgriffith | thingee: what.. you don't like my ci stuff any more :) | 22:04 |
| *** patrickeast has quit IRC | 22:04 | |
| hemna | I think it should be the same as however python-cinderclient works | 22:04 |
| hemna | https://launchpad.net/python-cinderclient | 22:04 |
| anteaya | http://git.openstack.org/cgit/openstack-infra/project-config/tree/gerrit/projects.yaml#n1006 | 22:04 |
| anteaya | looks like they have their own group | 22:05 |
| thingee | jgriffith: no, this was just something I forgot to do after discussion at the midcycle meetup. | 22:07 |
| thingee | jgriffith: if you don't remember, I can probably find the notes from the discussion. You were there. | 22:07 |
| jgriffith | thingee: I'm sure you could, and that's really not necessary at all | 22:08 |
| thingee | jgriffith: but I didn't mean for it to come across like that. | 22:08 |
| jgriffith | thingee: I really don't care, so we're even :) | 22:08 |
| thingee | jgriffith: ok... | 22:08 |
| thingee | jgriffith: by the way have you seen this volumeboot pattern issue? http://stkci.daterainc.com/refs-changes-63-161363-6/console.log.txt | 22:09 |
| anteaya | hemna: you are on your way then, I have to step afk for a bit | 22:09 |
| hemna | ok thanks, I'm walking through creating it | 22:09 |
| anteaya | okay great | 22:09 |
| thingee | hemna: thanks | 22:10 |
| hemna | np | 22:10 |
| jgriffith | thingee: nope, do you have the rest of the logs? | 22:10 |
| thingee | jgriffith: http://stkci.daterainc.com/refs-changes-63-161363-6/index.html | 22:11 |
| hemna | thingee, https://launchpad.net/os-brick | 22:11 |
| thingee | jgriffith: don't worry about it, I'll be diving into it when I get a chance | 22:11 |
| hemna | Doens't look like it's grouped under OpenStack though fwiw | 22:11 |
| hemna | and I can't edit it now, since I'm not the mx of it :P | 22:12 |
| jgriffith | thingee: your iscsi connection failed | 22:13 |
| jgriffith | thingee: look in n-cpu.log; 2015-04-15 19:58:49.971 | 22:13 |
| thingee | jgriffith: yeah so as mtreinish mentioned, the connection is known to cinder one minute and then fails with nova later. | 22:13 |
| jgriffith | thingee: ? | 22:14 |
| mtreinish | jgriffith: oh, that time stamp is around the same time when cinder was doing iscsi "things" | 22:15 |
| mtreinish | the tb I noticed was right below that at like 20:01 iirc | 22:15 |
| jgriffith | mtreinish: it's when nova tries to make the iscsi connection | 22:15 |
| asselin | thingee, we see that issue in icehouse too | 22:16 |
| jgriffith | mtreinish: but there's no target exported from Datera | 22:16 |
| thingee | asselin, jgriffith: whew..ok..well I just started noticing this around april 7th | 22:16 |
| asselin | which client was released april 7th? | 22:17 |
| thingee | same storage backend I've been using all the other times my ci has been successful. | 22:17 |
| hemna | ok both of those bugs have been added to os-brick | 22:17 |
| jgriffith | why do you guys think it's related to a client? | 22:18 |
| asselin | jgriffith, b/c clients releases have a reputation of breaking the world | 22:19 |
| jgriffith | asselin: well... seems like a stretch to say "client release" broke iscsiadm session | 22:20 |
| jgriffith | asselin: but could be | 22:20 |
| jgriffith | I dunno | 22:20 |
| *** annegentle has joined #openstack-cinder | 22:20 | |
| *** patrickeast has joined #openstack-cinder | 22:21 | |
| asselin | jgriffith, you're right...I should say some external dependency. | 22:21 |
| thingee | jgriffith: I haven't blamed anyone. Just been trying to figure out why things suddenly stopped working. | 22:21 |
| jgriffith | thingee: well.. not sure. Can't trace it too easily with your iqn change | 22:22 |
| jgriffith | thingee: is there a direct way to tie in the volume-id with that now? | 22:22 |
| *** julim has quit IRC | 22:22 | |
| jgriffith | thingee: ahh... I get it | 22:22 |
| jgriffith | first segment | 22:22 |
| *** annegentle has quit IRC | 22:24 | |
| *** _cjones_ has quit IRC | 22:24 | |
| jgriffith | thingee: asselin you guys using nova-net or neutron in your deployments? | 22:25 |
| asselin | jgriffith, neutron | 22:25 |
| thingee | nova net | 22:25 |
| jgriffith | hehe.. that eliminates those two :) | 22:25 |
| thingee | :) | 22:25 |
| jgriffith | so I see the init connection in the manager | 22:26 |
| jgriffith | thingee: but... | 22:27 |
| jgriffith | thingee: you vomit later when they try the detach | 22:27 |
| thingee | jgriffith: yeah i might just try pause things in the detach part and verify things myself. | 22:29 |
| jgriffith | thingee: well... | 22:29 |
| jgriffith | thingee: the problem is that the open iscsi session query is what's failing | 22:30 |
| jgriffith | thingee: so it never actually opened a session and attached | 22:30 |
| *** annashen has quit IRC | 22:30 | |
| jgriffith | thingee: in other words, that response to the detach isn't surprising IMHO | 22:30 |
| jgriffith | thingee: the trick is, why nova couldn't get an iscsi session to the iqn you provided it | 22:30 |
| jgriffith | asselin: curious what your failure looks like? | 22:31 |
| hemna | anish, ping | 22:31 |
| jgriffith | asselin: is it the same iscsi session connect failure? | 22:31 |
| *** annashen has joined #openstack-cinder | 22:31 | |
| asselin | jgriffith, http://paste.openstack.org/show/204044/ | 22:32 |
| asselin | jgriffith, no...same error message, but seems to be caused for a different reason | 22:32 |
| jgriffith | asselin: meh... tempest dump is useful; but what's in nova-compute log for the attach? | 22:32 |
| jgriffith | asselin: so it's tricky when comparing tempest output because of the cascade | 22:33 |
| anish | hemna: sup | 22:33 |
| hemna | anish, heyas | 22:33 |
| asselin | jgriffith, yeah...realize that "Security group is sitll in use" is just likely cleanup failing... | 22:34 |
| hemna | anish, so I'm trying to work on getting os-brick ready for Nova | 22:34 |
| anish | transport changes ? | 22:34 |
| hemna | anish, and I noticed 3 issues that would prevent nova from using it currently. | 22:34 |
| hemna | anish, one of them is an issue you had fixed in libvirtISCSIVolumeDriver, but not in cinder/brick or os-brick | 22:34 |
| hemna | anish, https://bugs.launchpad.net/os-brick/+bug/1370226 | 22:34 |
| openstack | Launchpad bug 1370226 in os-brick "LibvirtISCSIVolumeDriver cannot find volumes that include pci-* in the /dev/disk/by-path device" [Undecided,New] | 22:34 |
| hemna | anish, any chance you can take a look at that, since you are most familiar with it. | 22:35 |
| anish | yes | 22:35 |
| asselin | jgriffith, so in our icehouse case..seem to be related to: http://paste.openstack.org/show/204045/ | 22:35 |
| hemna | anish, that would be awesome. | 22:35 |
| hemna | anish, I can try but I'm not as familiar | 22:35 |
| anish | no worries, I was planning to take a look at it anyways | 22:35 |
| anish | there's also the iser component with the same issue | 22:35 |
| hemna | anish, ok great | 22:35 |
| hemna | anish, can I assign the os-brick portion to you then ? | 22:36 |
| jgriffith | asselin: hehe... that's a whole different can of worms | 22:36 |
| anish | yeah | 22:36 |
| *** annegentle has joined #openstack-cinder | 22:36 | |
| jgriffith | asselin: bad stuff... the 500 coming back from the client is not cool | 22:36 |
| jgriffith | asselin: given it's icehouse makes me suspect of API versions that you're using | 22:36 |
| jgriffith | but that would be a 100% thing I think | 22:37 |
| hemna | anish, thanks a bunch! really appreciate it | 22:37 |
| hemna | anish, if you need any help, let me know | 22:37 |
| *** _cjones_ has joined #openstack-cinder | 22:37 | |
| anish | hemna: is there any documentation on how to use os-brick with nova ? | 22:37 |
| asselin | jgriffith, and c-api shows http://paste.openstack.org/show/204046/ | 22:38 |
| *** mriedem has quit IRC | 22:38 | |
| asselin | and I've been noticing some amq issues...that's oslo | 22:39 |
| hemna | anish, hehe no. | 22:39 |
| hemna | anish, I'm working on a patch to get nova to use os-brick now | 22:39 |
| hemna | anish, which is what lead me to see the differences | 22:40 |
| asselin | jgriffith, oslo versions: http://paste.openstack.org/show/204047/ | 22:40 |
| hemna | anish, I have a WIP for cinder to use os-brick | 22:40 |
| anish | I have something slighty related https://review.openstack.org/#/c/152740/ | 22:40 |
| hemna | anish, and you can get the os-brick code working with cinder upload to image | 22:40 |
| anish | spec that addresses that bug | 22:40 |
| hemna | anish, https://review.openstack.org/#/c/155552/ | 22:40 |
| anish | I should update that spec to include the bug | 22:40 |
| hemna | anish, yah that'd be a good idea | 22:41 |
| *** annegentle has quit IRC | 22:41 | |
| *** bswartz has joined #openstack-cinder | 22:42 | |
| patrickeast | jgriffith: asselin: dunno if its related to what you guys are seeing, but we were recently seeing some failures on the volume_boot_pattern tests, they show up way more frequently with higher concurrency in the testing, and from our (small) sample set it failed 100% of the time when the V2 and V1 versions of the tests were running at the same time.. we had planned to dig into it more in our next sprint | 22:43 |
| hemna | so, at this point it, I think it might make sense to do a drop of os-brick now | 22:45 |
| hemna | get cinder using it | 22:45 |
| hemna | and then get the 3 os-brick bugs in, then another drop and then nova can be based off of the 2nd drop | 22:45 |
| hemna | or we'll miss our L window with nova at this point. | 22:45 |
| hemna | os-brick as it is now, is just like cinder/brick | 22:45 |
| hemna | so, once the 2nd drop happens, cinder will get the benefit of the 3 fixed bugs | 22:46 |
| asselin | jgriffith, patrickeast thingee I'm seen other rabbit mq issues: http://15.126.198.151/98/172198/3/check/3par-fc-driver-master-client-pip-eos10-dsvm/c22be0d/logs/screen-c-vol.txt.gz?level=ERROR | 22:46 |
| hemna | in the mean time we need a strategy for testing Cinder against os-brick as well | 22:46 |
| *** Liu has quit IRC | 22:46 | |
| *** Yogi1 has joined #openstack-cinder | 22:46 | |
| * asselin steps away for a bit | 22:47 | |
| *** jwcroppe has quit IRC | 22:48 | |
| *** patrickeast has quit IRC | 22:48 | |
| *** Liu has joined #openstack-cinder | 22:52 | |
| *** markvoelker_ has joined #openstack-cinder | 22:53 | |
| hemna | anish, so I can try and just ignore the differences in libvirt's iscsi volume driver and put up a WIP that uses os-brick | 22:58 |
| hemna | it'll be missing the 3 bugs/features, but should function | 22:58 |
| hemna | anish, I can try and do that tomorrow | 22:59 |
| anish | I can take it up from there, should not be an issue | 22:59 |
| hemna | for now the cinder WIP should provide you with a mechanism for testing | 23:00 |
| anish | sounds good, looking at the review patch | 23:00 |
| jgriffith | asselin: that's no good! | 23:01 |
| *** erlon has quit IRC | 23:01 | |
| *** Rockyg has joined #openstack-cinder | 23:01 | |
| *** winston-d_ has joined #openstack-cinder | 23:03 | |
| *** hemna is now known as hemnafk | 23:05 | |
| *** winston-1_ has joined #openstack-cinder | 23:08 | |
| *** jaypipes has quit IRC | 23:08 | |
| *** winston-d_ has quit IRC | 23:08 | |
| *** chlong has joined #openstack-cinder | 23:11 | |
| *** Yogi1 has quit IRC | 23:15 | |
| *** Yogi1 has joined #openstack-cinder | 23:16 | |
| *** patrickeast has joined #openstack-cinder | 23:16 | |
| *** fanyaohong has joined #openstack-cinder | 23:18 | |
| *** annashen has quit IRC | 23:21 | |
| *** jaypipes has joined #openstack-cinder | 23:22 | |
| *** jwcroppe has joined #openstack-cinder | 23:30 | |
| *** drjones has joined #openstack-cinder | 23:31 | |
| *** kmartin_ has joined #openstack-cinder | 23:34 | |
| *** j_king_ has joined #openstack-cinder | 23:34 | |
| *** Mandell has quit IRC | 23:37 | |
| *** Mandell has joined #openstack-cinder | 23:37 | |
| *** luv has quit IRC | 23:38 | |
| *** _cjones_ has quit IRC | 23:38 | |
| *** mdbooth has quit IRC | 23:38 | |
| *** j_king has quit IRC | 23:38 | |
| *** timcl has quit IRC | 23:38 | |
| *** kmartin has quit IRC | 23:38 | |
| *** bkopilov has quit IRC | 23:38 | |
| *** timcl1 has joined #openstack-cinder | 23:38 | |
| *** Tross has joined #openstack-cinder | 23:38 | |
| *** luv_ has joined #openstack-cinder | 23:38 | |
| *** aarefiev_ has joined #openstack-cinder | 23:38 | |
| *** mdbooth_ has joined #openstack-cinder | 23:38 | |
| *** luv_ is now known as luv | 23:38 | |
| *** bkopilov has joined #openstack-cinder | 23:38 | |
| *** lan_ has joined #openstack-cinder | 23:38 | |
| *** reed_ has joined #openstack-cinder | 23:38 | |
| *** cburgess_ has joined #openstack-cinder | 23:38 | |
| *** mdbooth_ is now known as mdbooth | 23:38 | |
| *** markvoelker has joined #openstack-cinder | 23:39 | |
| *** Tross1 has quit IRC | 23:39 | |
| *** lan has quit IRC | 23:39 | |
| *** aarefiev has quit IRC | 23:39 | |
| *** reed has quit IRC | 23:39 | |
| *** cburgess has quit IRC | 23:39 | |
| *** garthb has joined #openstack-cinder | 23:39 | |
| *** garthb_ has joined #openstack-cinder | 23:39 | |
| *** markvoelker_ has quit IRC | 23:40 | |
| *** reed_ is now known as reed | 23:40 | |
| *** markvoelker has quit IRC | 23:40 | |
| *** markvoelker has joined #openstack-cinder | 23:40 | |
| *** markvoelker_ has joined #openstack-cinder | 23:41 | |
| *** Yogi1 has quit IRC | 23:42 | |
| *** emagana has quit IRC | 23:43 | |
| *** boichev has quit IRC | 23:43 | |
| *** esker has joined #openstack-cinder | 23:44 | |
| *** markvoelker has quit IRC | 23:45 | |
| thingee | mtreinish: yeah can definitely reproduce it with running one of those tests alone | 23:51 |
| *** nestorf has joined #openstack-cinder | 23:55 | |
| asselin | jgriffith, no....i wish I had logstash internally.... | 23:58 |
| asselin | another occurence: http://15.126.198.151/18/149618/5/check/3par-fc-driver-master-client-pip-eos10-dsvm/2caaf4d/logs/screen-c-vol.txt.gz?level=TRACE#_2015-04-15_14_31_43_520 | 23:59 |
| *** wolsen_ is now known as wolsen | 23:59 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!