| openstackgerrit | Sean McGinnis proposed openstack/cinder master: Remove Tegile volume driver https://review.openstack.org/501333 | 00:06 |
|---|---|---|
| *** wanghao has joined #openstack-cinder | 00:19 | |
| *** dave-mccowan has joined #openstack-cinder | 00:35 | |
| *** gouthamr has quit IRC | 00:47 | |
| *** zhenguo has joined #openstack-cinder | 01:06 | |
| *** markstur has joined #openstack-cinder | 01:15 | |
| *** psachin has joined #openstack-cinder | 01:19 | |
| *** markstur has quit IRC | 01:20 | |
| *** edmondsw has joined #openstack-cinder | 01:24 | |
| *** edmondsw has quit IRC | 01:29 | |
| *** Shunli has joined #openstack-cinder | 01:40 | |
| *** dave-mccowan has quit IRC | 01:49 | |
| *** markvoelker has quit IRC | 01:54 | |
| *** Shatadru has joined #openstack-cinder | 02:19 | |
| *** Shatadru is now known as Shatadru|coffee| | 02:33 | |
| *** markstur has joined #openstack-cinder | 02:36 | |
| *** chenying has joined #openstack-cinder | 02:40 | |
| *** lhx__ has joined #openstack-cinder | 02:56 | |
| *** Shatadru|coffee| is now known as Shatadru | 03:05 | |
| openstackgerrit | Merged openstack/cinder master: Brocade: Add HTTP connection cleanup https://review.openstack.org/509292 | 03:12 |
| *** gkadam has joined #openstack-cinder | 03:35 | |
| *** links has joined #openstack-cinder | 03:50 | |
| *** crushil has joined #openstack-cinder | 03:50 | |
| *** markvoelker has joined #openstack-cinder | 03:55 | |
| *** gouthamr has joined #openstack-cinder | 03:58 | |
| *** dikonoor has joined #openstack-cinder | 04:00 | |
| *** Shatadru is now known as Shatadru|afk | 04:04 | |
| *** gnufied has joined #openstack-cinder | 04:05 | |
| *** felipemonteiro_ has joined #openstack-cinder | 04:12 | |
| *** felipemonteiro__ has joined #openstack-cinder | 04:13 | |
| *** felipemonteiro_ has quit IRC | 04:17 | |
| *** bhagyashri_s has quit IRC | 04:19 | |
| *** zsli_ has joined #openstack-cinder | 04:27 | |
| *** felipemonteiro__ has quit IRC | 04:28 | |
| *** markvoelker has quit IRC | 04:28 | |
| *** felipemonteiro__ has joined #openstack-cinder | 04:29 | |
| *** bhagyashris has joined #openstack-cinder | 04:29 | |
| *** markstur has quit IRC | 04:29 | |
| *** Shunli has quit IRC | 04:29 | |
| *** Shatadru|afk is now known as Shatadru | 04:34 | |
| *** crushil has quit IRC | 04:34 | |
| *** felipemonteiro__ has quit IRC | 04:38 | |
| *** jwcroppe has joined #openstack-cinder | 04:46 | |
| *** ntpttr_laptop has joined #openstack-cinder | 04:48 | |
| *** Dinesh_Bhor has joined #openstack-cinder | 04:48 | |
| *** markstur has joined #openstack-cinder | 04:50 | |
| *** ntpttr_laptop has quit IRC | 04:54 | |
| *** markstur has quit IRC | 04:54 | |
| *** edmondsw has joined #openstack-cinder | 05:00 | |
| *** mikal has quit IRC | 05:04 | |
| *** edmondsw has quit IRC | 05:05 | |
| *** wanghao has quit IRC | 05:16 | |
| *** wanghao has joined #openstack-cinder | 05:16 | |
| *** markvoelker has joined #openstack-cinder | 05:26 | |
| *** gouthamr has quit IRC | 05:27 | |
| *** kwathore has joined #openstack-cinder | 05:35 | |
| *** kwathore_ has quit IRC | 05:36 | |
| *** spectr has joined #openstack-cinder | 05:54 | |
| *** markvoelker has quit IRC | 05:58 | |
| *** ChubYann has quit IRC | 06:01 | |
| *** jwcroppe has quit IRC | 06:05 | |
| *** jwcroppe has joined #openstack-cinder | 06:08 | |
| *** belmoreira has joined #openstack-cinder | 06:18 | |
| *** cheneydc has joined #openstack-cinder | 06:20 | |
| *** josecastroleon has joined #openstack-cinder | 06:25 | |
| *** isaacb has joined #openstack-cinder | 06:26 | |
| *** rcernin has joined #openstack-cinder | 06:40 | |
| *** gcb has joined #openstack-cinder | 06:55 | |
| *** markvoelker has joined #openstack-cinder | 06:55 | |
| *** pcaruana has joined #openstack-cinder | 07:08 | |
| *** jdillaman has quit IRC | 07:09 | |
| *** edmondsw has joined #openstack-cinder | 07:18 | |
| *** takedakn has joined #openstack-cinder | 07:19 | |
| *** edmondsw has quit IRC | 07:22 | |
| *** jdillaman has joined #openstack-cinder | 07:23 | |
| *** alexchadin has joined #openstack-cinder | 07:25 | |
| *** edisonxiang has joined #openstack-cinder | 07:28 | |
| *** markvoelker has quit IRC | 07:29 | |
| *** takedakn has quit IRC | 07:31 | |
| *** hoonetorg has quit IRC | 07:33 | |
| *** Shatadru is now known as Shatadru|brb | 07:34 | |
| *** jwcroppe has quit IRC | 07:42 | |
| *** hoonetorg has joined #openstack-cinder | 07:47 | |
| *** jdillaman has quit IRC | 07:54 | |
| *** alexchadin has quit IRC | 08:14 | |
| *** alexchadin has joined #openstack-cinder | 08:14 | |
| *** gcb has quit IRC | 08:17 | |
| *** gcb has joined #openstack-cinder | 08:17 | |
| *** alexchadin has quit IRC | 08:18 | |
| *** alexchadin has joined #openstack-cinder | 08:21 | |
| *** markvoelker has joined #openstack-cinder | 08:26 | |
| *** markstur has joined #openstack-cinder | 08:27 | |
| *** markstur has quit IRC | 08:32 | |
| *** e0ne has joined #openstack-cinder | 08:35 | |
| *** Shatadru|brb is now known as Shatadru | 08:39 | |
| *** edmondsw has joined #openstack-cinder | 08:39 | |
| openstackgerrit | junbo.li proposed openstack/cinder master: Switch base to the latest in doc link address https://review.openstack.org/502249 | 08:40 |
| *** wN has quit IRC | 08:40 | |
| *** edmondsw has quit IRC | 08:43 | |
| *** wN has joined #openstack-cinder | 08:46 | |
| *** wN has quit IRC | 08:46 | |
| *** wN has joined #openstack-cinder | 08:46 | |
| *** ArchiFleKs has quit IRC | 08:50 | |
| *** finucannot is now known as stephenfin | 08:54 | |
| *** markvoelker has quit IRC | 08:59 | |
| openstackgerrit | junbo.li proposed openstack/cinder master: Api-ref: Add missing api response for volume transfer https://review.openstack.org/504023 | 09:01 |
| *** markstur has joined #openstack-cinder | 09:03 | |
| *** gcb has quit IRC | 09:06 | |
| *** kaisers1 has quit IRC | 09:06 | |
| *** markstur has quit IRC | 09:08 | |
| *** ircuser-1 has joined #openstack-cinder | 09:10 | |
| *** gcb has joined #openstack-cinder | 09:19 | |
| *** psachin has quit IRC | 09:20 | |
| *** psachin has joined #openstack-cinder | 09:22 | |
| *** kaisers has joined #openstack-cinder | 09:24 | |
| swamireddy | hi | 09:26 |
| swamireddy | I have 2 volume types named as "hdd" and "ssd"...here how can I list all volumes of "hdd" type only | 09:27 |
| swamireddy | cinder list --type=hdd | 09:27 |
| *** zsli_ has quit IRC | 09:29 | |
| *** ociuhandu has joined #openstack-cinder | 09:35 | |
| openstackgerrit | Silvan Kaiser proposed openstack/cinder master: [WIP] Adds Overlay Volumes Created from Snapshots to Quobyte https://review.openstack.org/507050 | 09:38 |
| *** markstur has joined #openstack-cinder | 09:39 | |
| *** markstur has quit IRC | 09:44 | |
| *** wanghao_ has joined #openstack-cinder | 09:52 | |
| *** wanghao__ has joined #openstack-cinder | 09:53 | |
| *** wanghao has quit IRC | 09:53 | |
| *** ociuhandu has quit IRC | 09:55 | |
| *** wanghao_ has quit IRC | 09:56 | |
| *** markvoelker has joined #openstack-cinder | 09:56 | |
| *** cheneydc has quit IRC | 09:58 | |
| *** gcb has quit IRC | 09:59 | |
| *** mvk has quit IRC | 10:00 | |
| *** psachin has quit IRC | 10:05 | |
| *** wanghao__ has quit IRC | 10:08 | |
| *** mvk has joined #openstack-cinder | 10:11 | |
| *** psachin has joined #openstack-cinder | 10:20 | |
| *** edmondsw has joined #openstack-cinder | 10:28 | |
| *** markvoelker has quit IRC | 10:30 | |
| *** edmondsw has quit IRC | 10:32 | |
| *** sdague has joined #openstack-cinder | 10:41 | |
| *** lpetrut has joined #openstack-cinder | 10:43 | |
| openstackgerrit | Sorin Sbarnea proposed openstack/cinder master: bump pylint to recent version https://review.openstack.org/509755 | 10:43 |
| *** Shatadru is now known as Shatadru|Gone | 10:56 | |
| *** lhx__ has quit IRC | 10:56 | |
| *** lhx_ has joined #openstack-cinder | 10:56 | |
| *** alexchadin has quit IRC | 10:57 | |
| *** nicolasbock has joined #openstack-cinder | 11:01 | |
| openstackgerrit | xiaoqin proposed openstack/cinder master: Add Storwize replication group support https://review.openstack.org/469394 | 11:02 |
| *** edmondsw has joined #openstack-cinder | 11:10 | |
| *** edmondsw has quit IRC | 11:10 | |
| *** nicolasbock_ has joined #openstack-cinder | 11:13 | |
| *** nicolasbock has quit IRC | 11:16 | |
| *** markvoelker has joined #openstack-cinder | 11:27 | |
| *** vivsoni_ has quit IRC | 11:33 | |
| *** yongwc has joined #openstack-cinder | 11:36 | |
| *** vivsoni has joined #openstack-cinder | 11:48 | |
| *** tpsilva has joined #openstack-cinder | 11:52 | |
| *** dave-mccowan has joined #openstack-cinder | 11:53 | |
| *** dave-mcc_ has joined #openstack-cinder | 11:55 | |
| *** josecastroleon has quit IRC | 11:58 | |
| *** abishop has joined #openstack-cinder | 11:58 | |
| *** dave-mccowan has quit IRC | 11:58 | |
| *** markvoelker has quit IRC | 12:00 | |
| *** gouthamr has joined #openstack-cinder | 12:01 | |
| *** edmondsw has joined #openstack-cinder | 12:05 | |
| *** vivsoni_ has joined #openstack-cinder | 12:09 | |
| *** vivsoni has quit IRC | 12:09 | |
| *** yongwc has quit IRC | 12:13 | |
| *** vivsoni_ has quit IRC | 12:13 | |
| *** vivsoni has joined #openstack-cinder | 12:13 | |
| *** liverpooler has joined #openstack-cinder | 12:18 | |
| *** alexchadin has joined #openstack-cinder | 12:21 | |
| *** markvoelker has joined #openstack-cinder | 12:21 | |
| *** hoonetorg has quit IRC | 12:23 | |
| openstackgerrit | TommyLike proposed openstack/cinder master: [policy in code] Add support for group, g-snapshot resources https://review.openstack.org/507812 | 12:23 |
| openstackgerrit | wang yong proposed openstack/cinder master: Inspur Cinder FC/iSCSI driver https://review.openstack.org/502455 | 12:29 |
| *** ganso has joined #openstack-cinder | 12:29 | |
| *** lhx_ has quit IRC | 12:29 | |
| *** jwcroppe_ has joined #openstack-cinder | 12:32 | |
| *** gkadam has quit IRC | 12:35 | |
| *** chlong has quit IRC | 12:37 | |
| *** gouthamr has quit IRC | 12:38 | |
| *** mriedem has joined #openstack-cinder | 12:40 | |
| mriedem | jgriffith_: sorry for the confusion on https://review.openstack.org/#/c/510201/ - nullable=True in the db column is good, but shouldn't do that in the object | 12:41 |
| *** hoonetorg has joined #openstack-cinder | 12:43 | |
| *** catintheroof has joined #openstack-cinder | 12:43 | |
| openstackgerrit | TommyLike proposed openstack/cinder master: [policy in code] Add support for snapshot resource https://review.openstack.org/507721 | 12:44 |
| openstackgerrit | Unana Okpoyo proposed openstack/cinder master: VMAX driver - Deprecate backend xml configuration https://review.openstack.org/508843 | 12:51 |
| openstackgerrit | TommyLike proposed openstack/cinder master: [policy in code] Add support for backup resource https://review.openstack.org/507015 | 12:52 |
| *** jwcroppe_ has quit IRC | 12:59 | |
| *** wanghao has joined #openstack-cinder | 13:05 | |
| *** dustins has joined #openstack-cinder | 13:09 | |
| *** chenying_ has joined #openstack-cinder | 13:12 | |
| *** chenying has quit IRC | 13:15 | |
| openstackgerrit | Ciara Stacke proposed openstack/cinder master: VMAX driver - Add async replication support https://review.openstack.org/509222 | 13:16 |
| *** gcb has joined #openstack-cinder | 13:17 | |
| *** eharney has joined #openstack-cinder | 13:19 | |
| *** jdwidari has joined #openstack-cinder | 13:22 | |
| *** jdwidari has quit IRC | 13:23 | |
| *** lbragstad has joined #openstack-cinder | 13:26 | |
| *** belmoreira has quit IRC | 13:30 | |
| *** belmoreira has joined #openstack-cinder | 13:32 | |
| *** haobing has joined #openstack-cinder | 13:37 | |
| *** cinderuser378 has joined #openstack-cinder | 13:39 | |
| *** KeithMnemonic has joined #openstack-cinder | 13:39 | |
| *** dgonzalez has left #openstack-cinder | 13:39 | |
| *** gouthamr has joined #openstack-cinder | 13:40 | |
| *** jdwidari has joined #openstack-cinder | 13:40 | |
| *** gouthamr has quit IRC | 13:41 | |
| cinderuser378 | Hi | 13:41 |
| cinderuser378 | cinder-manage is depricated in pike | 13:41 |
| cinderuser378 | I could not find alternative for | 13:42 |
| cinderuser378 | "cinder-manage volume update_host --currenthost <current host> --newhost <new host>" | 13:42 |
| cinderuser378 | Also I am looking for an option where we can change the host for a particular volume | 13:42 |
| *** kwathore_ has joined #openstack-cinder | 13:42 | |
| smcginnis | cinderuser378: cinder-manage isn't deprecated. At least as far as I know. | 13:43 |
| smcginnis | cinderuser378: And you can use migrate or retype with migration to move a volume to a different host. | 13:43 |
| cinderuser378 | migrate has a limitation that if snapshots exists then migrate does not work | 13:45 |
| *** kwathore has quit IRC | 13:45 | |
| smcginnis | cinderuser378: Well, snapshots are children of a volume, so yes, it would. | 13:45 |
| smcginnis | cinderuser378: To break that dependency you may be able to clone the volume, then migrate the clone. | 13:46 |
| *** psachin has quit IRC | 13:48 | |
| *** gouthamr has joined #openstack-cinder | 13:49 | |
| cinderuser378 | in our scenario, we have a volume and its snapshots on 2 hosts. we maintain volume and snapshot replication on 2 hosts. one is primary and one is secondary. | 13:52 |
| cinderuser378 | As long as primary is up, it serves the data requests. If it goes down, we simply want to change volume ownership in cinder.volumes table | 13:53 |
| *** crushil has joined #openstack-cinder | 13:53 | |
| cinderuser378 | so I am looking for something which is as simple as ""cinder-manage volume update_host" | 13:53 |
| cinderuser378 | which only changes host in cinder.volumes | 13:54 |
| cinderuser378 | but our concern does not stop here. we have more than 2 cinder nodes. each volume can have different primary and secondary | 13:54 |
| cinderuser378 | so if one primary fails, we want to change the host per volume | 13:55 |
| cinderuser378 | this is not possible using ""cinder-manage volume update_host --currenthost <current host> --newhost <new host>" | 13:55 |
| smcginnis | cinderuser378: If you're doing something outside of cinder like that then you would need to manually update the database probably. | 13:55 |
| cinderuser378 | at present we are doing the same :) | 13:56 |
| cinderuser378 | but we were checking if such feature is available through "openstack volume" cli | 13:57 |
| *** isaacb has quit IRC | 13:58 | |
| smcginnis | cinderuser378: Oh, so using cinder's replication functionality? Sounded like you were doing something special specifically for these volumes. | 13:58 |
| cinderuser378 | at present we are not using cinder-replication, but we have future plan to move to that feature. | 13:59 |
| smcginnis | cinderuser378: OK, you've totally confused me now. :) | 13:59 |
| cinderuser378 | the earlier suggession "manually update the database", we are not sure, how much acceptable this approach is. | 14:00 |
| cinderuser378 | yeah, sorry for that. | 14:00 |
| cinderuser378 | we handle volume-replication by ourself | 14:01 |
| cinderuser378 | not through cinder | 14:01 |
| smcginnis | cinderuser378: Well, "acceptable" is what works for you in this case I'm afraid. | 14:02 |
| smcginnis | All I can say is, it should be mostly "safe". :) | 14:02 |
| openstackgerrit | Silvan Kaiser proposed openstack/cinder master: [WIP] Adds Overlay Volumes Created from Snapshots to Quobyte https://review.openstack.org/507050 | 14:02 |
| *** links has quit IRC | 14:03 | |
| *** felipemonteiro__ has joined #openstack-cinder | 14:03 | |
| *** felipemonteiro_ has joined #openstack-cinder | 14:04 | |
| cinderuser378 | @smcginnis thanks for the response | 14:04 |
| cinderuser378 | This improves our confidence in present approach :) | 14:05 |
| smcginnis | cinderuser378: No problem. Good luck! If you see ways we can shape the official replication support, feel free to jump in. Always great to get real user input on how things should work. | 14:06 |
| *** chlong has joined #openstack-cinder | 14:08 | |
| *** felipemonteiro__ has quit IRC | 14:08 | |
| *** david-lyle has joined #openstack-cinder | 14:08 | |
| *** cinderuser378 has quit IRC | 14:09 | |
| *** alexchadin has quit IRC | 14:10 | |
| *** jmlowe has joined #openstack-cinder | 14:10 | |
| *** chlong_ has joined #openstack-cinder | 14:21 | |
| *** gouthamr has quit IRC | 14:23 | |
| *** spectr has quit IRC | 14:30 | |
| *** wanghao has quit IRC | 14:31 | |
| *** belmoreira has quit IRC | 14:33 | |
| *** belmoreira has joined #openstack-cinder | 14:37 | |
| *** jdillaman has joined #openstack-cinder | 14:37 | |
| *** gouthamr has joined #openstack-cinder | 14:38 | |
| *** ntpttr_laptop has joined #openstack-cinder | 14:39 | |
| *** jdillaman has quit IRC | 14:42 | |
| *** ntpttr_laptop has quit IRC | 14:45 | |
| openstackgerrit | Ciara Stacke proposed openstack/cinder master: VMAX driver - Add async replication support https://review.openstack.org/509222 | 14:53 |
| *** belmoreira has quit IRC | 14:54 | |
| *** markstur has joined #openstack-cinder | 14:56 | |
| *** links has joined #openstack-cinder | 14:57 | |
| *** haobing has quit IRC | 14:59 | |
| *** jwcroppe has joined #openstack-cinder | 15:00 | |
| *** jdillaman has joined #openstack-cinder | 15:00 | |
| *** shvepsy_ has quit IRC | 15:06 | |
| *** shvepsy has joined #openstack-cinder | 15:06 | |
| jungleboyj | smcginnis: Interesting discussion on the replication. Would be nice to see some people actually using it and giving feedback. | 15:07 |
| smcginnis | jungleboyj: ++ | 15:18 |
| *** gcb has quit IRC | 15:20 | |
| *** chlong has quit IRC | 15:22 | |
| *** chlong_ has quit IRC | 15:22 | |
| *** mlakat has quit IRC | 15:24 | |
| mriedem | smcginnis: this is the error in syslog when that device-mapper lvchange failure happens | 15:25 |
| mriedem | Oct 09 12:08:37 ubuntu-xenial-citycloud-kna1-11286426 kernel: device-mapper: table: 252:7: thin: Unable to activate thin device while pool is suspended Oct 09 12:08:37 ubuntu-xenial-citycloud-kna1-11286426 kernel: device-mapper: ioctl: error adding target to table | 15:25 |
| mriedem | oh i guess you already knew that | 15:25 |
| smcginnis | mriedem: Yeah, their expected, right? | 15:26 |
| smcginnis | Even more so if we are doing retries. | 15:26 |
| smcginnis | mriedem: Have you seen job failures? | 15:26 |
| mriedem | dansmith said he was getting kicked out of the gate | 15:27 |
| smcginnis | mriedem: Do you have a link to the review? | 15:27 |
| dansmith | yeah, a bunch of things are failing in the gate right now | 15:27 |
| mriedem | although, | 15:27 |
| mriedem | if you restrict that query to 2 days, the error shows up but the jobs are passing | 15:28 |
| mriedem | so what dan hit could be something else | 15:28 |
| smcginnis | Yeah, I haven't seen failures since that merged. | 15:28 |
| smcginnis | dansmith: Do you have a link? | 15:28 |
| dansmith | yep just a sec | 15:28 |
| dansmith | smcginnis: http://logs.openstack.org/40/510140/6/gate/gate-tempest-dsvm-cells-ubuntu-xenial/5051445/logs/screen-c-vol.txt.gz?level=TRACE#_Oct_09_14_17_21_290811 | 15:29 |
| dansmith | this other one is a bit different, but: | 15:29 |
| smcginnis | dansmith, mriedem: That's something different. That's on trying to delete the volume. | 15:30 |
| dansmith | http://logs.openstack.org/50/498950/18/gate/gate-tempest-dsvm-neutron-full-ubuntu-xenial/e4982c0/logs/devstacklog.txt.gz#_2017-10-09_15_10_12_484 | 15:30 |
| dansmith | I've seen like four of that deadlock-to-reply issue (the first one) and a couple of the latter c-api failed to start one | 15:30 |
| mriedem | yeah i think the retry on the lvchange one is probably fixed, but the stacktrace is noise | 15:31 |
| mriedem | i can try and cleanup the unnecessary trace | 15:31 |
| dansmith | but that job failed right there | 15:31 |
| dansmith | due to that taking too long and never replying on rpc, it looked like | 15:32 |
| smcginnis | 'c-api did not start' looks like yet another different failure. | 15:32 |
| dansmith | smcginnis: yeah I looked at c-api.log and didn't see any reason for that, | 15:34 |
| dansmith | so I was thinking t might also be a deadlock to reply sort of thing that devstack's simple mind interpreted as no service running | 15:34 |
| mriedem | yeah c-api is definitely serving requests | 15:35 |
| mriedem | devstack should just be checking systemctl | 15:36 |
| dansmith | mriedem: you're saying that's what it's doing or should be doing? | 15:36 |
| dansmith | I thought it was poking it with curl | 15:37 |
| mriedem | yeah i guess it does hit the api with curl | 15:38 |
| mriedem | i thought it was using systemctl | 15:38 |
| smcginnis | rval=124 on the curl call. Maybe not ready yet? | 15:39 |
| *** tbarron is now known as tbarron|PTO | 15:39 | |
| dansmith | hitting it with curl is the right thing to do I think, because just because systemctl says it's running doesn't mean much | 15:39 |
| smcginnis | Yeah, that should be good. | 15:39 |
| smcginnis | Nothing in the c-api logs indicating an issue though. | 15:39 |
| smcginnis | Though with the start up logs, looks like it may have just tried to hit it too soon. Wasn't quite ready yet based on the timestamp. | 15:40 |
| dansmith | okay well, like I said, the timeout during the actual run is the more common and concerning one | 15:40 |
| smcginnis | Slow gate nodes? | 15:40 |
| dansmith | so probably not worth drilling on this before the other | 15:41 |
| jgriffith_ | mriedem thanks for the tip on that service object, still missing something in the fakes somewhere (service.get doesn't seem to be implemented) but that hint saved me some time so thanks | 15:41 |
| *** jgriffith_ is now known as jgriffith | 15:41 | |
| mriedem | looks like it took just over a minute for c-api to start | 15:41 |
| mriedem | Oct 09 15:10:14.032382 ubuntu-xenial-inap-mtl01-11290241 devstack@c-api.service[21361]: [pid: 21364|app: 0|req: 1/1] 198.72.124.210 () {60 vars in 949 bytes} [Mon Oct 9 15:10:14 2017] GET /volume/v3 => generated 114 bytes in 2 msecs (HTTP/1.1 401) 5 headers in 233 bytes (1 switches on core 0) | 15:41 |
| *** liverpooler has quit IRC | 15:41 | |
| mriedem | 2017-10-09 15:10:12.484 | + lib/cinder:start_cinder:506 : die 506 'c-api did not start' | 15:41 |
| mriedem | so missed it by 2 seconds | 15:41 |
| eharney | the last line of screen-c-api does say "ready in 62 seconds" | 15:42 |
| smcginnis | dansmith: So you've seen that first runtime error multiple times? | 15:42 |
| mriedem | eharney: ah yeah | 15:43 |
| mriedem | so maybe a slow node | 15:43 |
| smcginnis | Wonder if we need to give some time for thin snapshots to actually delete and this is just trying to delete the volume before things are complete. | 15:43 |
| mriedem | or maybe c-api is just really slow to start :) | 15:43 |
| smcginnis | Which could again point to a slow node. | 15:43 |
| mriedem | eharney: compare to n-api | 15:43 |
| mriedem | Oct 09 15:05:30.335892 ubuntu-xenial-inap-mtl01-11290241 devstack@n-api.service[11763]: WSGI app 0 (mountpoint='') ready in 8 seconds on interpreter 0xd23ae0 pid: 11766 (default app) | 15:43 |
| mriedem | 8 seconds | 15:43 |
| jungleboyj | smcginnis: There has also been a history of such actions with lvm being slow. | 15:43 |
| mriedem | what are you guys doing on c-api startup? | 15:43 |
| *** mvk has quit IRC | 15:45 | |
| smcginnis | mriedem: Making tea. | 15:46 |
| jgriffith | jungleboyj but LVM shouldn't impact c-api at all | 15:46 |
| smcginnis | jgriffith: Seems to be two symptoms of slow nodes. | 15:46 |
| jgriffith | and that's not completely accurate :). That was a mapper problem | 15:46 |
| jgriffith | smcginnis I'm not sure I buy the "slow nodes" thing in this case (c-api) | 15:47 |
| jungleboyj | jgriffith: Right. I was responding to smcginnis' comment about deleting snapshots possibly being slow. | 15:47 |
| jgriffith | jungleboyj ah | 15:47 |
| smcginnis | jgriffith: Well someone has the wrong timing then if c-api initialized but curl from devstack failed before it was ready. | 15:47 |
| jgriffith | smcginnis I hear ya... | 15:48 |
| jgriffith | or we're doing something we shouldn't be doing in c-api :) | 15:48 |
| *** catintheroof has quit IRC | 15:48 | |
| smcginnis | jgriffith: That's probably the case. | 15:48 |
| smcginnis | I would think starting up the api service should be pretty quick. | 15:48 |
| smcginnis | I have no baseline data though. | 15:48 |
| jgriffith | indeed | 15:48 |
| mriedem | https://bugs.launchpad.net/cinder/+bug/1722326 | 15:49 |
| openstack | Launchpad bug 1722326 in Cinder "c-api times out devstack waiting to start" [Undecided,New] | 15:49 |
| mriedem | "Initializing extension manager." happens 6 times | 15:49 |
| mriedem | i'm guessing 3 times per worker | 15:51 |
| mriedem | there are 2 workers | 15:51 |
| mriedem | so, | 15:53 |
| mriedem | it looks like the api does this "work_cleanup" blocking rpc call to the scheduler | 15:53 |
| jgriffith | mriedem V2 and V3 :( | 15:53 |
| mriedem | which is going to hit the db | 15:53 |
| *** ntpttr_laptop has joined #openstack-cinder | 15:53 | |
| mriedem | jgriffith: ah ok | 15:53 |
| mriedem | so this blocking work_cleanup thing looks bad | 15:53 |
| *** ntpttr_laptop has quit IRC | 15:53 | |
| jgriffith | Yeah, I was just looking at why c-api block on rpc/scheduler? | 15:55 |
| mriedem | do you guys cache the service version? | 15:55 |
| openstackgerrit | Unana Okpoyo proposed openstack/cinder master: VMAX driver - Enable CHAP authentication for vmax backend https://review.openstack.org/510614 | 15:57 |
| *** e0ne has quit IRC | 16:01 | |
| openstackgerrit | Matt Riedemann proposed openstack/cinder master: DNM: see what is calling work_cleanup on c-api start https://review.openstack.org/510615 | 16:01 |
| *** armax has joined #openstack-cinder | 16:02 | |
| jgriffith | smcginnis so could it be that something in c-api is hitting your retry loop in the lvm side for some reason? | 16:03 |
| jgriffith | I know that sounds crazy... but | 16:03 |
| smcginnis | jgriffith: I wouldn't think so. | 16:04 |
| mriedem | it's more likely that you have 3 things hitting this work_cleanup code for each API route, | 16:04 |
| mriedem | with 2 workers, | 16:04 |
| mriedem | and it's a blocking rpc call to the scheduler which also does a db lookup | 16:04 |
| mriedem | and the results aren't cached | 16:04 |
| smcginnis | The LV activation, the c-api initialization, and the LV delete all seem like symptoms of just everything taking longer to me. | 16:04 |
| mriedem | per workers | 16:04 |
| mriedem | https://review.openstack.org/510615 will sniff out what's calling work_cleanup on start | 16:05 |
| jgriffith | nice | 16:05 |
| smcginnis | Good opportunity for some optimization at any rate. | 16:05 |
| mriedem | we cache the service version lookups, sec | 16:06 |
| mriedem | https://review.openstack.org/#/c/265326/ | 16:06 |
| mriedem | ^ is old, we also cache in api and the scheduler now too | 16:07 |
| openstackgerrit | Unana Okpoyo proposed openstack/cinder master: VMAX driver - Enable CHAP authentication for vmax backend https://review.openstack.org/510614 | 16:08 |
| *** jmlowe has quit IRC | 16:10 | |
| *** dave-mcc_ is now known as dave-mccowan | 16:13 | |
| openstackgerrit | Jeremy Zhang proposed openstack/cinder master: Add v3 api-ref for deleting extra spec for a volume type https://review.openstack.org/504009 | 16:13 |
| *** chlong_ has joined #openstack-cinder | 16:16 | |
| *** chlong has joined #openstack-cinder | 16:16 | |
| mriedem | so i dont think anything is calling the work cleanup api on startup as that's in a microversion, i was confused by the log messages, | 16:20 |
| mriedem | it's an upgrading check on each service startup, | 16:20 |
| mriedem | so i'm working on removing that to see how much time is saved | 16:20 |
| *** eharney has quit IRC | 16:21 | |
| *** lpetrut has quit IRC | 16:23 | |
| *** mvk has joined #openstack-cinder | 16:26 | |
| *** cali_boxer has quit IRC | 16:28 | |
| *** cali_boxer has joined #openstack-cinder | 16:31 | |
| *** pcaruana has quit IRC | 16:32 | |
| *** liverpooler has joined #openstack-cinder | 16:48 | |
| *** rcernin has quit IRC | 16:49 | |
| *** Apoorva has joined #openstack-cinder | 16:58 | |
| *** links has quit IRC | 16:58 | |
| *** Apoorva has quit IRC | 16:58 | |
| *** Apoorva has joined #openstack-cinder | 16:58 | |
| *** harlowja has joined #openstack-cinder | 17:01 | |
| *** jdwidari has quit IRC | 17:03 | |
| *** arnewiebalck_ has joined #openstack-cinder | 17:03 | |
| openstackgerrit | Matt Riedemann proposed openstack/cinder master: Remove newton-compat upgrade code on service startup https://review.openstack.org/510615 | 17:05 |
| *** cali_boxer has quit IRC | 17:15 | |
| *** dikonoor has quit IRC | 17:15 | |
| *** cali_boxer has joined #openstack-cinder | 17:17 | |
| *** cali_boxer has quit IRC | 17:22 | |
| *** cali_boxer has joined #openstack-cinder | 17:23 | |
| *** lbragstad has quit IRC | 17:24 | |
| *** arnewiebalck_ has quit IRC | 17:28 | |
| *** lbragstad has joined #openstack-cinder | 17:42 | |
| *** jwcroppe has quit IRC | 17:43 | |
| *** e0ne has joined #openstack-cinder | 17:51 | |
| *** dhinesh has joined #openstack-cinder | 17:55 | |
| dhinesh | has anyone seen this error before "oslo_config.cfg.NoSuchOptError: no such option backend in group [key_manager]" | 17:55 |
| *** jmlowe has joined #openstack-cinder | 17:57 | |
| *** jwcroppe has joined #openstack-cinder | 17:57 | |
| *** mriedem1 has joined #openstack-cinder | 17:58 | |
| *** mriedem has quit IRC | 17:59 | |
| *** edmondsw has quit IRC | 18:00 | |
| smcginnis | dhinesh: You're oslo.config library is out of date I think. | 18:01 |
| *** edmondsw has joined #openstack-cinder | 18:02 | |
| smcginnis | *your | 18:05 |
| dhinesh | smcginnis: suggestion on upgrading it? | 18:05 |
| smcginnis | dhinesh: Well, depends on your environment. Pip, apt, or whatever that is that is used now instead of yum that isn't yum. | 18:06 |
| *** edmondsw has quit IRC | 18:06 | |
| dhinesh | smcginnis: ok thanks | 18:07 |
| *** Apoorva_ has joined #openstack-cinder | 18:08 | |
| *** Apoorva has quit IRC | 18:11 | |
| *** pcaruana has joined #openstack-cinder | 18:11 | |
| *** mriedem1 is now known as mriedem | 18:11 | |
| *** edmondsw has joined #openstack-cinder | 18:12 | |
| *** edmondsw has quit IRC | 18:17 | |
| mriedem | jgriffith: so i see why "Initializing extension manager" shows up 6 times in n-api startup; we have 2 workers, and v2 and v3 APIs have the router, so that's 4 - i was lost on the other 2, and it looks like that's due to the Versions router extending APIRouter and defining an ExtensionManager class - why the versions router needs to care about loading extensions though, i'm not sure | 18:25 |
| *** eharney has joined #openstack-cinder | 18:27 | |
| openstackgerrit | Matt Riedemann proposed openstack/cinder master: Do not load extendable routes for the Versions router https://review.openstack.org/510653 | 18:35 |
| *** e0ne has quit IRC | 18:39 | |
| openstackgerrit | Merged openstack/cinder master: Remove Tegile volume driver https://review.openstack.org/501333 | 18:50 |
| *** lpetrut has joined #openstack-cinder | 18:58 | |
| *** lpetrut has quit IRC | 19:00 | |
| *** lpetrut has joined #openstack-cinder | 19:01 | |
| *** dustins has quit IRC | 19:03 | |
| *** eharney has quit IRC | 19:08 | |
| *** edmondsw has joined #openstack-cinder | 19:09 | |
| thingee | jungleboyj: is anybody by chance keeping up with the market place vendor listing for cinder? ^ | 19:12 |
| jungleboyj | thingee: Good question. Probably not at the moment. | 19:13 |
| openstackgerrit | Dhinesh Balasubramaniam proposed openstack/cinder master: Hedvig Cinder driver implementation https://review.openstack.org/433341 | 19:14 |
| jungleboyj | thingee: Looking at the marketplace it looks out of date. | 19:17 |
| jungleboyj | Though, isn't that generated from something else? | 19:18 |
| jungleboyj | smcginnis: ^^ | 19:18 |
| *** diablo_rojo has joined #openstack-cinder | 19:21 | |
| thingee | jungleboyj: right it's the driver log | 19:23 |
| thingee | jungleboyj: nobody is set to update, but I was updating it for kilo and liberty | 19:24 |
| thingee | the support matrix would help with this by the way :) | 19:24 |
| jungleboyj | :-) | 19:25 |
| jungleboyj | thingee: So, the marketplace is not automatically updated then? | 19:26 |
| thingee | jungleboyj: it is, it pulls from driver log | 19:26 |
| thingee | but we're moving away from driver log | 19:26 |
| thingee | https://review.openstack.org/#/q/topic:support-matrix | 19:26 |
| thingee | jungleboyj: http://lists.openstack.org/pipermail/openstack-dev/2017-July/119365.html | 19:27 |
| thingee | I smell lunch, bbl | 19:27 |
| *** hemna_ has joined #openstack-cinder | 19:36 | |
| *** dustins has joined #openstack-cinder | 19:38 | |
| *** mikal has joined #openstack-cinder | 19:44 | |
| *** eharney has joined #openstack-cinder | 19:45 | |
| *** Apoorva_ has quit IRC | 19:47 | |
| *** Apoorva has joined #openstack-cinder | 19:57 | |
| *** ChubYann has joined #openstack-cinder | 20:02 | |
| *** crushil has quit IRC | 20:03 | |
| *** crushil has joined #openstack-cinder | 20:03 | |
| *** liverpooler has quit IRC | 20:13 | |
| *** pcaruana has quit IRC | 20:18 | |
| *** boris_42_ has joined #openstack-cinder | 20:18 | |
| *** jwcroppe has quit IRC | 20:23 | |
| *** jwcroppe has joined #openstack-cinder | 20:24 | |
| *** armax has quit IRC | 20:26 | |
| *** jwcroppe has quit IRC | 20:29 | |
| openstackgerrit | Sean McGinnis proposed openstack/cinder master: Remove newton-compat upgrade code on service startup https://review.openstack.org/510615 | 20:36 |
| *** felipemonteiro has joined #openstack-cinder | 20:45 | |
| *** felipemonteiro_ has quit IRC | 20:46 | |
| *** crushil has quit IRC | 20:47 | |
| *** felipemonteiro_ has joined #openstack-cinder | 20:52 | |
| openstackgerrit | Sean McGinnis proposed openstack/cinder master: Remove Hitachi volume drivers https://review.openstack.org/507709 | 20:55 |
| *** felipemonteiro has quit IRC | 20:55 | |
| *** erlon has joined #openstack-cinder | 20:57 | |
| ildikov | smcginnis: do you have a target number to hit on removing things? ^^ :) | 20:58 |
| smcginnis | ildikov: The more the better. ;) | 20:59 |
| ildikov | smcginnis: you could just remove the whole cinder folder then :) | 20:59 |
| *** crushil has joined #openstack-cinder | 21:00 | |
| smcginnis | ildikov: I'll leave that for jgriffith. I just want to slim it down as much as possible. | 21:00 |
| ildikov | smcginnis: I thought jgriffith is waiting for you to provide the clean slate :) | 21:01 |
| smcginnis | Disappointing that stackalytics doesn't seem to be working right for me. | 21:01 |
| smcginnis | Would actually like to see how many LOC. | 21:01 |
| smcginnis | http://stackalytics.com/?module=cinder-group&metric=commits&user_id=sean-mcginnis | 21:01 |
| smcginnis | Pretty sure I've changed more than 29 lines. | 21:02 |
| smcginnis | ildikov: Maybe, that could be. :) | 21:02 |
| ildikov | smcginnis: hmm, maybe it doesn't count the negative changes anymore? | 21:02 |
| ildikov | smcginnis: no idea | 21:03 |
| ildikov | smcginnis: but LOC is not a that good metric anyway :) | 21:03 |
| smcginnis | ildikov: Eh, not many of them are, but I was curious. | 21:04 |
| ildikov | smcginnis: :) | 21:05 |
| smcginnis | Now if someone would just approve my stestr patches. | 21:08 |
| smcginnis | *cough cough* https://review.openstack.org/#/q/status:open+branch:master+topic:stestr+owner:%22Sean+McGinnis+%253Csean.mcginnis%2540gmail.com%253E%22 | 21:08 |
| ildikov | smcginnis: the Pike numbers look good | 21:08 |
| smcginnis | ildikov: Wow, that's more than I expected. | 21:08 |
| ildikov | smcginnis: :) | 21:10 |
| *** lpetrut has quit IRC | 21:11 | |
| *** e0ne has joined #openstack-cinder | 21:11 | |
| ildikov | smcginnis: I'm way better than you though, I almost have as many commits as LOC changed in Cinder :D | 21:13 |
| openstackgerrit | Sean McGinnis proposed openstack/cinder master: Do not load extendable routes for the Versions router https://review.openstack.org/510653 | 21:13 |
| smcginnis | ildikov: Nice! | 21:13 |
| ildikov | smcginnis: I'm not saying I'm proud, but well, small changes can do big things :) | 21:14 |
| smcginnis | ildikov: Often more than the big changes. | 21:15 |
| ildikov | smcginnis: +1 :) | 21:15 |
| jungleboyj | smcginnis: I will look after I figure out why Glance can't put stuff in Swift. | 21:18 |
| jungleboyj | Where the heck has this day gone? | 21:18 |
| smcginnis | jungleboyj: Are you using WSGI? | 21:19 |
| smcginnis | jungleboyj: There are known issues for glance running behind Apache right now. | 21:19 |
| *** abishop has quit IRC | 21:19 | |
| jungleboyj | smcginnis: Hmmm, this is just a default TrippleO install at the moment. | 21:20 |
| jungleboyj | What is the problem we would see smcginnis | 21:20 |
| smcginnis | jungleboyj: Timeouts uploading images. Apache tries to cache everything up before passing it along, so glance never has a chance to process smaller chunks. | 21:21 |
| *** Apoorva has quit IRC | 21:21 | |
| *** armax has joined #openstack-cinder | 21:21 | |
| openstackgerrit | Matt Riedemann proposed openstack/cinder master: Remove newton-compat upgrade code on service startup https://review.openstack.org/510615 | 21:21 |
| openstackgerrit | Matt Riedemann proposed openstack/cinder master: Do not load extendable routes for the Versions router https://review.openstack.org/510653 | 21:21 |
| *** Apoorva has joined #openstack-cinder | 21:21 | |
| jungleboyj | smcginnis: The error I am seeing is: 2017-10-09 20:29:52.433 98685 INFO swiftclient [req-0ecdfc50-3a96-468b-ab1b-f6f28bab7980 38b38012d8564a8390e05e90bc327518 8dc1dd62ce8b4cfd8fb7e24b19aaad96 - default default] RESP BODY: <html><h1>Service Unavailable</h1><p>The server is currently unavailable. Please try again at a later time.</p></html> | 21:30 |
| smcginnis | Hmm, that's in the glance logs? | 21:38 |
| smcginnis | Actually looks like a credential or setting issue with the swift settings. | 21:38 |
| *** e0ne has quit IRC | 21:38 | |
| *** felipemonteiro_ has quit IRC | 21:43 | |
| *** gouthamr has quit IRC | 21:45 | |
| *** dustins has quit IRC | 21:57 | |
| *** chlong_ has quit IRC | 21:59 | |
| *** chlong has quit IRC | 21:59 | |
| *** jmlowe has quit IRC | 22:01 | |
| *** ganso has quit IRC | 22:06 | |
| *** armaan has joined #openstack-cinder | 22:23 | |
| *** boris_42_ has quit IRC | 22:25 | |
| *** lbragstad has quit IRC | 22:25 | |
| *** edmondsw has quit IRC | 22:27 | |
| armaan | hello folks, I am running tempest and two cinder tests are failing because of the difference between transport url i.e. reference = 'http' and actual = 'https'. Any idea how can i fix this? | 22:28 |
| *** catintheroof has joined #openstack-cinder | 22:47 | |
| *** sdague has quit IRC | 22:56 | |
| *** mriedem1 has joined #openstack-cinder | 23:11 | |
| *** mriedem has quit IRC | 23:13 | |
| *** crushil has quit IRC | 23:21 | |
| *** crushil has joined #openstack-cinder | 23:21 | |
| *** armaan has quit IRC | 23:23 | |
| *** armaan has joined #openstack-cinder | 23:24 | |
| *** savvas has joined #openstack-cinder | 23:37 | |
| openstackgerrit | Merged openstack/cinder master: Dell EMC PS: Fix extend volume creating unmanaged snapsots https://review.openstack.org/509006 | 23:38 |
| *** savvas has quit IRC | 23:47 | |
| *** tpsilva has quit IRC | 23:51 | |
| *** stakeda has joined #openstack-cinder | 23:55 | |
| *** edmondsw has joined #openstack-cinder | 23:59 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!