Wednesday, 2017-06-21

*** chlong has quit IRC00:03
*** yangyape_ has quit IRC00:17
*** yangyapeng has joined #openstack-cinder00:20
*** yangyapeng has quit IRC00:25
*** Julien-zte has joined #openstack-cinder00:36
*** wanghao_ has joined #openstack-cinder00:36
*** dave-mccowan has joined #openstack-cinder00:38
*** wanghao has quit IRC00:39
*** MVenesio has quit IRC00:47
openstackgerritTommyLike proposed openstack/cinder master: Cinder volume revert to snapshot  https://review.openstack.org/39173601:00
*** fredli__ has joined #openstack-cinder01:01
*** dillaman has quit IRC01:08
*** jdillaman has quit IRC01:08
*** zhangguoqing has joined #openstack-cinder01:13
*** edisonxiang_ is now known as edisonxiang01:18
*** yangyapeng has joined #openstack-cinder01:21
*** jdillaman has joined #openstack-cinder01:21
*** dillaman has joined #openstack-cinder01:21
*** yangyape_ has joined #openstack-cinder01:25
openstackgerritShunei Shiono proposed openstack/cinder master: NEC driver: Optimal path for non-disruptive backup  https://review.openstack.org/47485201:27
*** yangyapeng has quit IRC01:28
*** kaisers_ has quit IRC01:33
*** Julien-zte has quit IRC01:36
*** KeithMnemonic1 has quit IRC01:39
*** lucasxu has joined #openstack-cinder01:39
*** lucasxu has quit IRC01:39
*** markvoelker has quit IRC01:40
*** markvoelker has joined #openstack-cinder01:41
*** markvoelker has quit IRC01:45
*** lhx has joined #openstack-cinder01:46
*** lhx is now known as Guest2274201:47
*** gkadam__ has quit IRC01:47
openstackgerritjiamin proposed openstack/cinder master: DS8K driver: specify pool and lss in extra-specs  https://review.openstack.org/43585701:57
*** bkopilov has quit IRC01:59
*** fredli__ has quit IRC02:02
*** adisky__ has quit IRC02:03
*** Guest22742 is now known as lhx__02:05
*** dhinesh has quit IRC02:07
*** zhugaoxiao has joined #openstack-cinder02:13
*** mriedem has quit IRC02:24
openstackgerritTommyLike proposed openstack/cinder master: Cinder volume revert to snapshot  https://review.openstack.org/39173602:37
*** mikal_ is now known as mikal03:15
*** bkopilov has joined #openstack-cinder03:17
*** zhangguoqing has quit IRC03:25
*** zhangguoqing has joined #openstack-cinder03:28
*** markvoelker has joined #openstack-cinder03:29
*** zhangguoqing has quit IRC03:36
*** Julien-zte has joined #openstack-cinder03:43
*** nicolasbock has joined #openstack-cinder03:46
*** gkadam__ has joined #openstack-cinder03:52
openstackgerritTommyLike proposed openstack/cinder master: Add ability to extend 'in-use' volume  https://review.openstack.org/45428703:55
*** kaisers_ has joined #openstack-cinder04:00
*** jwcroppe has joined #openstack-cinder04:01
*** psachin has joined #openstack-cinder04:02
*** chhavi has joined #openstack-cinder04:03
*** links has joined #openstack-cinder04:07
*** dave-mccowan has quit IRC04:09
*** salv-orlando has joined #openstack-cinder04:12
*** salv-orlando has quit IRC04:13
*** salv-orlando has joined #openstack-cinder04:22
*** salv-orlando has quit IRC04:27
*** arnewiebalck_ has joined #openstack-cinder04:31
*** adisky__ has joined #openstack-cinder04:40
*** gouthamr has quit IRC04:43
*** arnewiebalck_ has quit IRC04:50
*** jerrygb has joined #openstack-cinder04:56
*** gkadam__ has quit IRC05:01
*** jerrygb has quit IRC05:01
*** salv-orlando has joined #openstack-cinder05:02
*** gkadam has joined #openstack-cinder05:05
*** links has quit IRC05:07
openstackgerritCheng Yong Zhao proposed openstack/cinder master: Add gmcv support in SVC driver  https://review.openstack.org/46119105:07
*** links has joined #openstack-cinder05:23
*** ChubYann has quit IRC05:27
*** salv-orlando has quit IRC05:39
*** salv-orlando has joined #openstack-cinder05:39
*** pewp has quit IRC05:40
*** e0ne has joined #openstack-cinder05:43
*** salv-orlando has quit IRC05:44
*** pewp has joined #openstack-cinder05:44
*** Julien-zte has quit IRC05:44
*** e0ne has quit IRC05:47
*** Julien-zte has joined #openstack-cinder05:48
*** Julien-zte has quit IRC05:50
*** Julien-zte has joined #openstack-cinder05:52
*** Julien-zte has quit IRC05:54
*** diablo_rojo has quit IRC05:57
*** Julien-zte has joined #openstack-cinder05:57
*** Julien-zte has quit IRC05:57
openstackgerritTommyLike proposed openstack/cinder master: Add ability to extend 'in-use' volume  https://review.openstack.org/45428706:04
*** bkopilov has quit IRC06:06
*** Poornima_K has joined #openstack-cinder06:07
*** wanghao_ has quit IRC06:07
*** Poornima_K has quit IRC06:19
*** Julien-zte has joined #openstack-cinder06:22
*** lkuchlan has joined #openstack-cinder06:23
*** pcaruana has joined #openstack-cinder06:31
*** isaacb has joined #openstack-cinder06:32
*** Shunli has joined #openstack-cinder06:32
*** pgadiya has joined #openstack-cinder06:35
*** jerrygb has joined #openstack-cinder06:36
*** wanghao has joined #openstack-cinder06:36
*** bkopilov has joined #openstack-cinder06:42
openstackgerritTommyLike proposed openstack/cinder master: [I18n] Don't convert exception.msg into string  https://review.openstack.org/47602206:51
*** rcernin has joined #openstack-cinder06:59
*** markvoelker has quit IRC07:00
*** markvoelker has joined #openstack-cinder07:00
*** markvoelker has quit IRC07:01
*** markvoelker has joined #openstack-cinder07:01
*** markvoelker has quit IRC07:01
*** edmondsw has joined #openstack-cinder07:03
*** markvoelker has joined #openstack-cinder07:07
*** edmondsw has quit IRC07:08
*** tesseract has joined #openstack-cinder07:15
openstackgerritTommyLike proposed openstack/cinder master: [i18n] Don't convert exception.msg into string  https://review.openstack.org/47602207:19
openstackgerritAshish Billore proposed openstack/cinder master: Include Usage for attachment-delete CLI  https://review.openstack.org/47602907:25
*** links has quit IRC07:31
*** jerrygb has quit IRC07:34
*** zhangguoqing has joined #openstack-cinder07:35
openstackgerritDeepak Mourya proposed openstack/cinder-specs master: Remove support for py34.  https://review.openstack.org/47444507:35
*** wanghao has quit IRC07:37
*** wanghao has joined #openstack-cinder07:39
*** links has joined #openstack-cinder07:44
*** arnewiebalck_ has joined #openstack-cinder07:44
openstackgerritTommyLike proposed openstack/python-cinderclient master: DNM. Test environment variables  https://review.openstack.org/47316507:45
*** salv-orlando has joined #openstack-cinder07:54
*** zzzeek has quit IRC08:00
*** zzzeek has joined #openstack-cinder08:02
*** hjensas has quit IRC08:02
*** zhangguoqing has quit IRC08:09
*** arnewiebalck_ has quit IRC08:09
*** zhangguoqing has joined #openstack-cinder08:10
*** e0ne has joined #openstack-cinder08:10
*** sticker has quit IRC08:12
*** zhangguoqing has quit IRC08:13
*** e0ne has quit IRC08:13
openstackgerritIsaac Beckman proposed openstack/cinder master: IBM Storage - Fix vol create from snapshot in CG  https://review.openstack.org/47579308:14
*** reveller has quit IRC08:15
openstackgerritjiamin proposed openstack/cinder master: DS8K: add replication consistency group support  https://review.openstack.org/46587708:17
*** lpetrut has joined #openstack-cinder08:25
*** e0ne has joined #openstack-cinder08:27
*** chhavi has quit IRC08:28
*** e0ne has quit IRC08:33
*** zhangguoqing has joined #openstack-cinder08:39
*** gkadam_ has joined #openstack-cinder08:43
*** rcernin has quit IRC08:45
*** Poornima_K has joined #openstack-cinder08:46
*** gkadam has quit IRC08:46
*** rcernin has joined #openstack-cinder08:50
*** edmondsw has joined #openstack-cinder08:51
*** wanghao has quit IRC08:53
*** wanghao has joined #openstack-cinder08:54
*** edmondsw has quit IRC08:56
*** Julien-zte has quit IRC08:56
*** Julien-zte has joined #openstack-cinder08:58
*** hjensas has joined #openstack-cinder08:58
*** aspiers has joined #openstack-cinder09:00
*** salv-orlando has quit IRC09:02
*** salv-orlando has joined #openstack-cinder09:03
*** salv-orlando has quit IRC09:03
*** salv-orlando has joined #openstack-cinder09:06
*** salv-orlando has quit IRC09:08
*** salv-orlando has joined #openstack-cinder09:09
*** Poornima_K has quit IRC09:10
*** stakeda has quit IRC09:16
*** e0ne has joined #openstack-cinder09:18
*** chhavi has joined #openstack-cinder09:21
*** wanghao has quit IRC09:31
*** Shunli has quit IRC09:31
*** Poornima_K has joined #openstack-cinder09:41
*** e0ne has quit IRC09:50
*** e0ne has joined #openstack-cinder09:52
*** Julien-zte has quit IRC09:55
openstackgerritHelen Walsh proposed openstack/cinder master: VMAX driver - bootable volume errors  https://review.openstack.org/47267409:57
*** gkadam_ has quit IRC10:06
*** gkadam_ has joined #openstack-cinder10:06
*** hjensas has quit IRC10:07
*** gkadam__ has joined #openstack-cinder10:07
openstackgerritravi edpuganti proposed openstack/cinder master: Added CG capability to volume group in CoprHD  https://review.openstack.org/45961010:09
*** gkadam_ has quit IRC10:11
*** salv-orl_ has joined #openstack-cinder10:12
*** salv-orlando has quit IRC10:14
openstackgerritIvan Kolodyazhny proposed openstack/cinder master: WIP. Use 'ceph df' to calculate provisioned space  https://review.openstack.org/47607910:15
Aurelgad1ohello guys, I have an issue with cinder volume. with cinder-manage service list I can see my scheduler and my volume being online. When I create a new volume, I can see logs in the scheduler about my cinder-volume host being chosen for the volume, but I get nothing in the cinder-volume logs, the volume never gets created and stays in the "creating" status10:21
*** gkadam__ is now known as gkadam10:21
Aurelgad1odo you have any pointers on where i should look at ?10:21
openstackgerritIsaac Beckman proposed openstack/cinder master: Ibm storage: Added replication group support  https://review.openstack.org/46873810:28
*** yangyape_ has quit IRC10:30
*** Poornima_K has quit IRC10:33
*** edmondsw has joined #openstack-cinder10:39
*** zhonghua has quit IRC10:40
*** zhonghua has joined #openstack-cinder10:41
*** edmondsw has quit IRC10:45
*** chhavi has quit IRC10:47
*** yangyapeng has joined #openstack-cinder10:51
*** e0ne has quit IRC10:52
*** mvk has quit IRC10:54
*** bkopilov has quit IRC10:56
*** ericyoung has joined #openstack-cinder10:57
*** lhx__ has quit IRC10:57
*** zhangguoqing has quit IRC10:58
*** yangyapeng has quit IRC10:59
*** lhx__ has joined #openstack-cinder11:00
*** BlackDex_ is now known as BlackDex11:00
*** lhx__ has quit IRC11:10
*** lhx__ has joined #openstack-cinder11:10
openstackgerritKushal Wathore proposed openstack/cinder master: HPE 3PAR: Adding driver capability  https://review.openstack.org/47412211:10
openstackgerritKushal Wathore proposed openstack/cinder master: HPE Lefthand: Fixed cloning operation  https://review.openstack.org/46262411:22
*** mvk has joined #openstack-cinder11:23
openstackgerritKushal Wathore proposed openstack/cinder master: 3PAR: Fixed image cache enabled capability.  https://review.openstack.org/46174411:25
openstackgerritMerged openstack/python-cinderclient master: Fix cmd options for updating a quota class  https://review.openstack.org/45313411:28
*** abishop has joined #openstack-cinder11:35
*** yangyapeng has joined #openstack-cinder11:36
*** yangyapeng has quit IRC11:41
*** yangyapeng has joined #openstack-cinder11:41
*** dave-mccowan has joined #openstack-cinder11:41
*** reveller has joined #openstack-cinder11:47
openstackgerritMerged openstack/cinder master: Fix NoneType has no attribute get error  https://review.openstack.org/47167211:49
openstackgerritMerged openstack/cinder master: Replace deprecated BaseException.message with six.text_type  https://review.openstack.org/47129911:51
*** isaacb_ has joined #openstack-cinder11:52
*** isaacb has quit IRC11:53
*** e0ne has joined #openstack-cinder11:56
*** zhangguoqing has joined #openstack-cinder12:00
*** zhangguoqing has quit IRC12:06
*** pgadiya has quit IRC12:14
danpawlikjbernard, jungleboyj, sdatko, smcginnis: I added release notes https://review.openstack.org/#/c/456672/ . I hope it should be good right now12:19
danpawlikcan you review :) ?12:19
danpawlikthanks!12:19
*** edmondsw has joined #openstack-cinder12:21
*** isaacb_ has quit IRC12:25
*** isaacb_ has joined #openstack-cinder12:27
*** zhangguoqing has joined #openstack-cinder12:28
*** bkopilov has joined #openstack-cinder12:32
openstackgerritEric Young proposed openstack/cinder master: ScaleIO Driver: Fix for using scheduler specified pool  https://review.openstack.org/47611612:39
*** isaacb__ has joined #openstack-cinder12:40
*** isaacb__ has quit IRC12:40
*** isaacb_ has quit IRC12:42
jungleboyjdanpawlik:  Reviewed.  Thanks for dealing with all my harassment.  :-)12:48
danpawlikjungleboyj: Its good that you look on the details. Its really important and a right way to have a good/clean code in repo :)12:51
*** eharney has joined #openstack-cinder12:51
jungleboyjdanpawlik:  You are right.  Appreciate that you understand that!12:53
*** salv-orl_ has quit IRC12:54
*** mriedem has joined #openstack-cinder12:56
*** mdovgal has quit IRC12:56
*** salv-orlando has joined #openstack-cinder12:56
*** liverpooler has joined #openstack-cinder12:57
*** salv-orlando has quit IRC12:57
*** pewp has quit IRC12:57
*** salv-orlando has joined #openstack-cinder12:58
*** dave-mccowan has quit IRC12:58
*** jerrygb has joined #openstack-cinder13:00
*** lhx__ has quit IRC13:03
*** jwcroppe has quit IRC13:03
*** jwcroppe has joined #openstack-cinder13:03
*** jwcroppe has quit IRC13:03
*** jwcroppe has joined #openstack-cinder13:04
*** pewp has joined #openstack-cinder13:04
*** jwcroppe has quit IRC13:06
*** gouthamr has joined #openstack-cinder13:07
*** lucasxu has joined #openstack-cinder13:07
*** salv-orlando has quit IRC13:08
*** salv-orlando has joined #openstack-cinder13:12
*** cknight has joined #openstack-cinder13:14
openstackgerritIvan Kolodyazhny proposed openstack/cinder master: WIP. Do not fail if RBD image is broken  https://review.openstack.org/47540013:17
*** cknight has quit IRC13:18
*** jwcroppe has joined #openstack-cinder13:30
*** eric_young has joined #openstack-cinder13:31
openstackgerritJeremy Zhang proposed openstack/cinder master: Fix assertTrue in functional tests  https://review.openstack.org/47566113:32
*** ericyoung has quit IRC13:33
*** lhx has joined #openstack-cinder13:38
*** lhx is now known as Guest7924213:38
*** xyang1 has joined #openstack-cinder13:38
openstackgerritJeremy Zhang proposed openstack/cinder master: Fix assertTrue in functional tests  https://review.openstack.org/47566113:39
openstackgerritTommyLike proposed openstack/python-cinderclient master: Enabled like filter support in client  https://review.openstack.org/46621413:40
*** dave-mccowan has joined #openstack-cinder13:41
*** reveller has quit IRC13:43
*** cknight has joined #openstack-cinder13:44
*** catintheroof has joined #openstack-cinder13:45
*** reveller has joined #openstack-cinder13:46
tommylikehuanyone who is familiar with i18n could take a look at this patch? https://review.openstack.org/#/c/474547/ thanks in advance :)13:47
*** diablo_rojo has joined #openstack-cinder13:47
*** zhangguoqing has quit IRC13:48
*** salv-orlando has quit IRC13:49
*** zhangguoqing has joined #openstack-cinder13:51
*** lhx_ has joined #openstack-cinder13:54
*** Guest79242 has quit IRC13:57
*** adisky__ has quit IRC13:59
*** jerrygb has quit IRC14:04
*** crushil has joined #openstack-cinder14:04
*** chlong has joined #openstack-cinder14:07
*** gkadam has quit IRC14:08
*** felipemonteiro has joined #openstack-cinder14:14
*** felipemonteiro_ has joined #openstack-cinder14:15
*** Julien-zte has joined #openstack-cinder14:17
*** felipemonteiro has quit IRC14:19
tommylikehuping geguileo14:22
tommylikehuwhat's the issue we have in the wsgi file at present? https://github.com/openstack/cinder/blob/master/cinder/api/openstack/wsgi.py14:23
*** dustins has joined #openstack-cinder14:30
*** Julien-zte has quit IRC14:32
*** Julien-z_ has joined #openstack-cinder14:34
*** reveller has quit IRC14:36
*** Julien-z_ has quit IRC14:37
*** gouthamr has quit IRC14:37
*** gouthamr has joined #openstack-cinder14:37
e0nejbernard, eharney: hi guys, are you around?14:39
jbernarde0ne: nope14:39
e0ne:(14:39
jbernarde0ne: jk, im here! XD14:39
e0neI would like to discuss https://bugs.launchpad.net/cinder/+bug/169878614:39
openstackLaunchpad bug 1698786 in Cinder "cinder-volume fails on start when rbd pool contains partially deleted images" [Undecided,In progress] - Assigned to Ivan Kolodyazhny (e0ne)14:39
* jbernard 's still got it14:39
jbernarde0ne: sure14:40
e0nethere are two proposed fixes: https://review.openstack.org/475400 and https://review.openstack.org/47607914:40
*** markvoelker has quit IRC14:40
e0nethe first one looks like workaround14:40
e0nebut the second one changes how we calculate capacity now14:40
jbernardgimme a sec, looking14:41
e0nesure14:41
*** reveller has joined #openstack-cinder14:42
jbernardi cannot remember why we iterate on volumes… that would have been a nice place for a comment14:43
jbernarde0ne: the results are identical?14:43
*** Julien-zte has joined #openstack-cinder14:43
e0nejbernard, nope. 'ceph df' shows real provisioned size14:44
e0nejbernard: the current method returns allocated space14:44
jbernardahh right, we want allocated, no?14:44
e0newe want, if I didn't miss anything14:45
jbernardelse we loose our ability to overprovision a thin backend14:45
e0nelet me find review request with current implementation. there were some useful comments14:45
jbernardor maybe we handle that elsewhere14:45
e0nehere is it https://review.openstack.org/#/c/178262/14:46
e0nejbernard: here is your comment https://review.openstack.org/#/c/178262/1/cinder/volume/drivers/rbd.py14:47
eharneyi don't think we want to shell out to "ceph df"14:47
e0neeharney: I agree with you, but I didn't find python bindings for it14:48
jbernardRados.get_cluster_stats()14:48
jbernardhttp://docs.ceph.com/docs/master/rados/api/python/14:48
jbernardthat may not offer stats per-pool though :/14:49
geguileotommylikehu: I realized during lunch that I was mistaken, was about to mention it in the review14:50
geguileolol14:50
jbernardbut there are two issues, 1) volumes shouldn't be able to be left in a partially deleted state14:50
jbernardand 2) what is _get_usage_info() expected to return?14:50
e0nejbernard: you're right. Rados.get_cluster_stats doesn't return per pool stats14:51
eharney_get_usage_info is populating the normal get_volume_stats info14:51
e0nejbernard: they shouldn't. but in real world something goes wrong:(14:53
e0nejbernard: probably, we need some periodic job to clean up such volumes14:53
jbernarde0ne: could we at least detect them and move them to lost+found (or something to that effect)14:54
e0nejbernard: I'm working on such patch14:54
e0nejbernard: but it's a bit complicated because we have to figure out what volumes are failed to delete and what are deleting now14:55
*** markvoelker has joined #openstack-cinder14:57
*** rcernin has quit IRC15:07
*** jwcroppe has quit IRC15:07
e0nejbernard, eharney: I'll be happy to get any feedback from you to get this issue fixed15:09
jbernarde0ne: personally i think skipping images that raise an exception is okay15:11
jbernarde0ne: unless we're not supposed to report allocated space15:11
jbernarde0ne: but i think we are15:11
jbernarda librados call for pool stats would be nice15:11
*** zhangguoqing has quit IRC15:12
e0nejbernard: do we have such api for librados?15:12
*** pcaruana has quit IRC15:12
jbernarde0ne: not that i know of, at least by the docs, i need to look closer15:13
*** psachin has quit IRC15:13
openstackgerritMerged openstack/cinder master: Fix assertTrue in functional tests  https://review.openstack.org/47566115:13
*** chhavi has joined #openstack-cinder15:18
*** dillaman has quit IRC15:19
*** dillaman has joined #openstack-cinder15:21
Aurelgad1oalso iirc ceph df shows the raw storage, it does not take into account the replica overhead15:28
Aurelgad1oI beg your pardon for reposting but now that north america is awake ...15:29
e0neAurelgad1o: how big is overhead for replica?15:29
Aurelgad1oI have an issue with cinder volume. with cinder-manage service list I can see my scheduler and my volume being online. When I create a new volume, I can see logs in the scheduler about my cinder-volume host being chosen for the volume, but I get nothing in the cinder-volume logs, the volume never gets created and stays in the "creating status"15:29
Aurelgad1oe0ne: if you have three OSDs of 1 TB, you'll see free space as 3TB. By default, ceph replicates data 3 times, but it depends on the pool's configuration15:30
*** sudipto has joined #openstack-cinder15:31
*** sudipto_ has joined #openstack-cinder15:31
Aurelgad1oa ceph admin can also set quotas on pools : http://docs.ceph.com/docs/kraken/rados/operations/pools/#set-pool-quotas15:32
e0neAurelgad1o: it could be an issue with messaging (oslo.messaging, rabbitmq) in your case15:32
Aurelgad1oe0ne: I don't see messages staying in the queues, but how can I be sure messages are posted in the queues ? the DEBUG verbosity does not seems to be enough for me to see that15:32
e0neAurelgad1o: do you have any errors in scheduler logs?15:33
Aurelgad1oe0ne: nope, I see the volumebeing scheduled to the host I want to and there is no error15:34
*** lpetrut has quit IRC15:36
*** felipemonteiro_ has quit IRC15:38
*** mchlumsky has joined #openstack-cinder15:39
openstackgerritHelen Walsh proposed openstack/cinder master: VMAX driver - Replication, replacing SMI-S with REST  https://review.openstack.org/46602715:41
*** danpawlik is now known as danpawlik_15:43
*** crushil has quit IRC15:44
mchlumskyHello. I have a question regarding the volume_clear* options in cinder.conf. Is the volume_clear functionality implemented in each driver?15:46
e0nemchlumsky: hi. no, it's implemented only fror lvm and unsupported bdd drivers15:51
*** wewe0901 has joined #openstack-cinder15:51
e0neAurelgad1o: :(. looks like something with messaging, but I'm not sure15:51
*** crushil has joined #openstack-cinder15:52
mchlumskye0ne, thanks!15:53
*** salv-orlando has joined #openstack-cinder15:55
*** salv-orlando has quit IRC15:56
*** salv-orlando has joined #openstack-cinder15:57
*** markvoelker has quit IRC15:58
*** markvoelker has joined #openstack-cinder15:59
*** diablo_rojo has quit IRC16:00
geguileojungleboyj: I broke disconnect for StorwizeSVCISCSIDriver and some other drivers with latest os-brick changes16:00
geguileojungleboyj: Any change you could test the fix?  https://review.openstack.org/#/c/475760/16:00
*** Apoorva has joined #openstack-cinder16:00
*** diablo_rojo has joined #openstack-cinder16:01
jungleboyjI don't have access to that hardware anymore.  hemna  ^16:01
*** gnarld_ is now known as cFouts16:01
*** tesseract has quit IRC16:03
*** aarefiev is now known as aarefiev_afk16:07
openstackgerritHelen Walsh proposed openstack/cinder master: VMAX driver - QoS, replacing SMI-S with REST  https://review.openstack.org/45675916:08
*** felipemonteiro has joined #openstack-cinder16:09
*** felipemonteiro_ has joined #openstack-cinder16:10
*** salv-orl_ has joined #openstack-cinder16:12
*** lkuchlan has quit IRC16:12
*** felipemonteiro has quit IRC16:14
openstackgerritHelen Walsh proposed openstack/cinder master: VMAX driver - Compression, replacing SMI-S with REST  https://review.openstack.org/46019116:14
*** salv-orlando has quit IRC16:15
*** Julien-zte has quit IRC16:21
*** Julien-zte has joined #openstack-cinder16:24
*** Julien-zte has quit IRC16:24
*** jwcroppe has joined #openstack-cinder16:35
openstackgerritHelen Walsh proposed openstack/cinder master: VMAX driver - Replication, replacing SMI-S with REST  https://review.openstack.org/46602716:40
openstackgerritHelen Walsh proposed openstack/cinder master: VMAX driver - Live Migration, replacing SMI-S with REST  https://review.openstack.org/47227916:43
*** jwcroppe has quit IRC16:44
openstackgerritTommyLike proposed openstack/cinder master: Fix i18n issues  https://review.openstack.org/47454716:44
*** mvk has quit IRC16:56
openstackgerritHelen Walsh proposed openstack/cinder master: VMAX driver - implement Generic Volume Group feature  https://review.openstack.org/47454616:59
*** Apoorva_ has joined #openstack-cinder17:02
tbarronwinston-d: was just wondering what the multinode SIGHUP issue is exactly17:03
jungleboyjPeople talk to much!17:04
eharneyi'm still lost on why there's a concern there about multinode and HA too17:04
tbarronwinston-d: clearly one has to change config on multiple nodes and deliver a SIGHUP on each17:04
jungleboyj@!t17:04
pewpjungleboyj (╯°□°)╯︵ ┻━┻17:04
tbarronwinston-d: and if you do that there will be windows when some nodes are running old config and some running nes config17:04
tbarronwinston-d: do we have options that cause a problem in those windows?17:05
jungleboyjtbarron: Right, so you are saying the only option is to shut everone down and then restart?17:05
tbarronjungleboyj: I'm not *saying* that, but I'm asking if that's true and why17:05
*** Apoorva has quit IRC17:05
*** lhx_ has quit IRC17:06
tbarronjungleboyj: there are clearly windows for inconsistent options, I'm just wondering what they are and do they cause trouble17:06
winston-dhaving to restart cinder to reload new cfg value, means you can trade a small down time with consistency.17:06
tbarronjungleboyj: winston-d for example an option to change logging will not cause a problem in those windows17:06
*** harlowja has joined #openstack-cinder17:07
winston-dnode 1/node 2 are HA pair, you change cfg for both nodes, you shutdown 1 and 2 and then start them one by one.17:07
tbarronjungleboyj: winston-d but if you were running active-active volume service and changed an option that affects actual backend behavior ...17:07
winston-dno inconsistent window17:07
tbarronwinston-d: "jinx"17:07
tbarronwinston-d: do you know of other problematic cases than active/active volume service?17:08
tbarronwinston-d: problems for multiple api, scheduler, or backup services?17:09
mchlumsky@eric_young, Hi! It looks like ScaleIO doesn't wipe the data after we delete a volume in OpenStack (unless it's something we need to configure in ScaleIO itself). We've seen newly created volumes with pre-existing filesystems on them. Can the volume_clear feature be implemented in the ScaleIO driver?17:09
tbarronwinston-d: or problems across services (seems unlikely b/c it's loosely coupled rpc stuff)17:09
openstackgerritHelen Walsh proposed openstack/cinder master: VMAX driver - bootable volume errors  https://review.openstack.org/47267417:10
*** lbragstad has joined #openstack-cinder17:10
winston-dtbarron: I don't, I'm trying to see if I can think of any.17:11
*** e0ne has quit IRC17:11
tommylikehuhey geguileo , does this what are you prefer? https://review.openstack.org/#/c/474547/17:11
pewphemna ( ´ ▽ ` )ノ17:11
*** chhavi has quit IRC17:12
lbragstaddiablo_rojo: fwiw - one thing that's helped keystone cut down on cruft in our configuration file is the use of resource-option17:12
openstackgerritMerged openstack/cinder master: VMAX driver - Retype, replacing SMI-S with REST  https://review.openstack.org/45675717:12
winston-dif inconsistent API leads to potentially inconsistent data plane change, it'd be considered as bad as inconsistent A/A c-vols17:13
lbragstaddiablo_rojo: we noticed it when we were ending up with whitelisted user-ids in our config file for various things17:13
lbragstaddiablo_rojo: so - we implemented https://github.com/openstack/keystone/commit/1896d1ba0d24d3780ce8c7652fa4c4378a02255d17:14
lbragstaddiablo_rojo: which allowed us to deprecate and remove some of our configuration cruft - https://github.com/openstack/keystone/commit/9844fa1e264fe52276d9bd804b26b518ef3aaa2417:14
tbarronwinston-d: so I have a theoretical feel for how there *could* be problems in this regard (maybe) but17:15
tbarronwinston-d: do you actually have any concrete known options that cause such problems?17:15
tbarronwinston-d: even with active-active volume service?17:15
*** links has quit IRC17:15
tbarronwinston-d: for the sake of argument we can assume active-active volume service for the same backend17:16
tbarronwinston-d: and SIGHUP one volume service, then wait ten minutes and SIGHUP the other17:16
tbarronin both cases changing from config A to config A'17:16
tbarronwinston-d: what options make this break things in that 10 minute window?17:16
diablo_rojolbragstad, ooOOoo good to know17:18
lbragstaddiablo_rojo: in case that helps at all17:18
openstackgerritMerged openstack/cinder master: Cinder volume revert to snapshot  https://review.openstack.org/39173617:19
winston-dtbarron: i probably need more than 10 mins to think of one. ;)17:19
tbarronwinston-d: fair enough17:20
tbarronwinston-d: but if it's just a *theoretical* possibility that there *could* be such options, then maybe17:20
tbarronwinston-d: it would be sufficient to make sure that new options don't have that kind of problem in code review17:21
*** jwcroppe has joined #openstack-cinder17:21
tbarronwinston-d: the reason I say that is that tightly coupling service instances on multiple nodes in that way seems to me a poor idea in the first place17:21
*** _alastor1 is now known as _alastor_17:21
hemnachanging the default volume type will lead to inconsistent behavior.17:22
eharneythe other question there is, if there are options that would break like that, do they break already if people start/stop/restart services in the right order?17:22
*** mvk has joined #openstack-cinder17:22
eharneyinconsistent behavior like a different default volume type seems like a reasonable thing to have happen in this circumstance, people need to know that if they change config, all of the services need to reload it17:23
eharneythat isn't any different from what happens with service configs today17:23
hemnathere are many things in the cinder.conf changes that would break things17:24
hemnadb url, transport_url17:24
hemnastuffs like that17:25
tbarronhemna: that would break things on multi-node today if we don't stop, edit, start in the right order?17:25
hemnayah17:25
hemnaif you are dumb enough to change that17:25
hemnaand not restart all nodes17:25
hemnado the cinder.conf quota settings do anything after the first start?17:25
tbarronhemna: do these give more than transitory errors if we stop and restart quickly?17:26
winston-dwell, human error is always there, i don't think it's fair to go by flawed design because human err is inevitable.17:27
hemnawhat concerns me about this is that we are all not sure what will happen at this point.17:27
hemnaand no way to test it17:27
tbarronhemna: so is using SIGHUP (delivering a storm of them in random order) to all the nodes running cinder services more dangerous than delivering a similarly random 'systemctl stop <service'; sed-cmd; 'systemctl start service' via ssh to all the relevant nodes?17:28
jgriffithso not to interrupt here...17:28
jgriffithbut I'm going to anyway17:28
jgriffith:)17:28
tbarronmy point is that it's not the SIGHUP that is the issue17:28
* tbarron pauses a sec17:29
jgriffithin a standard multi-node today there's not really a problem here because there's no shared state17:29
jgriffiththe example of the db node or rpc node is kinda silly17:29
tbarronpewp17:29
hemnayah it is, but it's still valid17:29
jgriffithbut in an A/A c-vol scenario there are a number of pit-falls that are valid17:30
*** sudipto_ has quit IRC17:30
*** sudipto has quit IRC17:30
jgriffithyour driver configs need to match17:30
jgriffithand the example of default volume-type is an interesting point IMO17:30
tbarroni'm looking for a concrete example of harm by mismatch17:31
tbarronnot saying it's not there, just think it would be useful17:31
jgriffithdefault volume type is a pretty good one17:31
jgriffithIMO17:31
jgriffithmisconfigured device option17:31
jgriffithsome devices not a big deal... but say Netapp for example that has 22 options or something like that17:31
tbarronwhat's exactly the harm if that is different for a brief period?17:31
winston-djgriffith: if someone changed the 'sf_vol_prefix' on one node for SF driver, and the other node only got updated with new value after 10 mins, would that be a problem?17:31
jgriffiththere's potential there for bad things17:32
jgriffithwinston-d yes17:32
jgriffithwinston-d that would mean that a/a wouldn't work17:32
winston-djgriffith: node1, 'uuid' as prefix, a vol created with that, node2, 'cinder' as prefix.17:32
jgriffithbecause you'd be looking for the wrong volume-name17:32
tbarronso you would have some "old" prefixes and some "new" prefixes anyways unless SF converts all the old to new17:32
jgriffithyou're screwed17:32
smcginnisdiablo_rojo, jgriffith: FYI, Doug is trying to get pieces in place to be able to generate config files. This is part of that effort: https://review.openstack.org/#/c/474307/17:33
jgriffithtbarron it doesn't work that way sadly (or happily actually)17:33
tbarronplaying devil's advocate, your driver should convert old to new17:33
winston-duser vol A got created by node1, and then he wanted to create a snapshot/or delete the vol, but that request got handled by node2.17:33
jgriffithtbarron ummm... no; actually it shouldn't17:33
eharneythe sf_vol_prefix case is one where you can't change it for a deployment that has volumes even on a single node deployment17:33
tbarronjgriffith: i expect you are right17:33
winston-dvol A's the status would become 'error_deleting' if request was a deleete?17:33
jgriffitheharney you're correct17:33
jgriffithat least kinda17:34
eharneythere are a handful of options in that category, i think17:34
diablo_rojosmcginnis, thanks for the heads up17:34
* diablo_rojo opens the link17:34
jgriffitheharney you're right.. but I do think there is at least some things in an A/A setup that should be considered a bit17:34
jgriffithand what I do think would be interesting is a 'shared-config'17:35
jgriffithso you introduce a config option that says "a/a pair with xyz" or whatever17:35
tbarronso volume prefix isn't safe, but it's never safe to change anyways, right?17:35
jgriffithand the config is shared17:35
tbarronso it's not the concrete example why SIGHUP is bad17:35
jgriffithtbarron well, it's somewhat different... it's not necessarily safe in any scenario unless you're doing it knowingly; but in A/A it has all sorts of weird effects17:36
jgriffithdepending on which c-vol service receives the request determines what the behavior/results will be17:36
jgriffithas opposed to a traditional deployment it's just consistent and relatively easy to trouble-shoot and figure out17:37
jgriffithI'm not arguing one side or the other here, just presenting some thoughts17:38
tbarronyeah, with A/A if a backend can't handle "old" config after it has seen "new" config then one should shut them all down, change config for all, and start17:38
jgriffithwhat about the shared-config idea?17:38
jgriffithanybody have thoughts on that?17:38
tbarronjgriffith: explain it a bit more17:38
jgriffithwell, if I'm saying this is an A/A c-vol setup; then I most likely want to use the same config on both nodes right?17:39
tbarronin manila I have some private config that I want to share between multiple ganesha servers on different nodes BTW17:39
tbarronwill probably put it (json) in a rados object17:39
jgriffiththere's nothing to prevent sharing a config across nodes via rpc I don't think17:40
jgriffithor...17:40
jgriffithadding the ability to perform a check/comparison17:40
jgriffithso as part of init you do a compare of the two and make sure they match17:40
jgriffithif they don't, you can log the info/deltas or maybe error out17:40
jgriffitherror out and fail to init may not be the way to go at first17:41
jgriffithbut logging error info stating "Hey!  These configs for the driver are different, and here's the things that don't match!!!  You should look at these and make sure this is ok"17:41
jgriffithIt's just a usability thing17:42
winston-dwell, the problem is not just config failed to be deployed to pair of nodes simultaneously. it's more about if pair of nodes can load new config into memory and start acting with new behavior.17:42
winston-dwith as minimum as possible drift/inconsistent window.17:42
jgriffithwinston-d you mean like the config on one node was flat out wrong/incopatible?17:42
jgriffithincompatable17:43
jgriffithwith the backend device?17:43
jgriffithso the drift window... IMO that circles back to the only solution being a shared config17:43
jgriffithI don't know of another way to eliminate that window completely17:44
*** jwcroppe has quit IRC17:45
winston-dtake sf_vol_prefix for example, 'uuid' is changed to 'cinder', the cinder.conf on both nod1 and node2 has been changed successfully at almost the same time, but node1 is doing somethign heavy with high load, it took it 30 secs before the new value is applied while node2 takes the new value alsmost instantly.17:45
winston-dmy point is exactly that there is no way to eliminate that window.17:46
jgriffithwinston-d yeah... there really isn't17:46
jgriffitheven my idea would still have a small window17:46
jgriffithanything already on the queue it's just too bad17:47
winston-dis 1 sec window small enough? in our peak hour, we might have 20~40 vol being created in 1 sec.17:47
jgriffithprobably not17:47
eric_youngmchlkumsky: That sounds odd to me, can you follow up with your Dell EMC Support channels. I would imagine that creating a new volume in SIO would NOT give you back an existing volume with a filesystem17:48
winston-dthe behavior of dynamic config reload is not predictable becuase of load or other factors.17:49
patrickeastit might just be something on a case-by-case basis for the config options... they aren't all equal, some you really might need to stop all requests or do some more complex workflow17:49
jgriffithpatrickeast +117:49
patrickeastimo adding a reload without restart is just a tool for advanced users... but a really dangerous one17:49
tbarronpatrickeast: yeah17:49
winston-dthe safest way is to trade down time for consistency.17:49
jgriffithI guess the only option there would be flush the queue, wait, restart17:49
winston-da maintenance windows and notification email save us all.17:50
jgriffithwinston-d seems reasonable to me17:50
tbarronbut it would be real handy for when you want to turn on debug17:50
tbarronand stuff like that17:50
tbarronmaybe documenting when it's safe and when not makes sense17:51
patrickeasttbarron: +117:51
patrickeastthings like that, or adjusting timeouts17:51
patrickeastit makes total sense17:51
winston-dtbarron: that i totally agree. you know how painful to restart a c-vol hosts a few thousands vols.17:51
smcginnistbarron: https://review.openstack.org/#/c/445885/17:51
jgriffithsmcginnis you beat me to it!17:51
smcginnisHah, and I'm not even really here. :P17:52
winston-dbut in production, we want consistent and predictable behavior to minize the risk.17:52
jgriffithsmcginnis either shows that you're "that good" or that I'm "that bad"17:52
jgriffithor both :)17:52
smcginnisLet's go with neither and that I'm just bad at focusing on training material. ;)17:52
jgriffithhaha17:53
jgriffithshhhh.. they might be listening!17:53
jgriffithso what I've gathered from this long discussion that I've admittedly only partially followed and understood everything about is; I don't think dynamic config is such a hot idea17:54
jgriffithand by that I mean, updating config without a restart17:54
smcginnisjgriffith: I think this is actually more useful in that we need to explicitly decide each option that can be changed: https://review.openstack.org/#/c/464028/17:55
winston-djgriffith: that's my take as well.17:55
jgriffithsmcginnis that's interesting... hadn't seen it17:55
jgriffithI was about to say that we could selectively choose things to "allow" and add it to geguileo 's patch for log level17:56
jgriffithbut the mutable config thing is interesting17:56
* jgriffith doesn't completely grok it yet17:56
smcginnisjgriffith: Yep, this would be the first step, then we can mark specific options that we would want to be able to change.17:57
jgriffithsmcginnis winston-d tbarron that seems like a reasonable compromise no?17:57
tbarronunderstanding which options are safe to change without stopping services on all nodes and updating all config before restarting would be worthwhile in itself17:57
winston-dmutable without restart?17:57
jgriffithwinston-d yes, but only white-listed options17:57
smcginnisFWIW, I don't think this oslo.config option has been really communicated as a new option folks should look at.17:58
tbarronjust not allowing SIGHUP doesn't by itself guarantee that ops will do the right thing anyways17:58
jgriffithtbarron well this does allow SIGHUP17:58
tbarronjgriffith: right17:58
tbarronjgriffith: I'm agreeing that "marking" the options as safe or needing care would be good17:59
jgriffithoh.. got ya17:59
tbarronand saying that SIGHUP is not the only way to screw up with the options that aren't safe17:59
winston-dour fingers are much fatter than we think18:00
tbarronwinston-d: :)18:00
tbarronwinston-d: is it 2am or 3am where you are?18:01
winston-djgriffith, smcginnis: so yeah, 'mutable' cfg is one good chnage to make if dynamic reloading is inevitable.18:02
winston-dtbarron: it's 2am. i shall get some sleep before my next meeting.18:03
smcginnisI should probably update that to include log levels so theirs at least an example using the mutable-ness.18:03
tbarronwinston-d: i just sleep during the meetings18:03
tbarronsmcginnis: +118:03
smcginnis*there's18:03
hemnacould we use a special restart/reload RPC message to cut down on the window to all of the nodes?18:04
winston-dtbarron: i need to learn sleep talking sooner. ;)18:04
tbarron:)18:04
winston-dok, signing off now. good day to you all.18:04
*** winston-d is now known as winston-d_zZZ18:05
hemnadunno, this seems like a lot of hand wavy to me and not enough thought/design behind it18:05
*** jwcroppe has joined #openstack-cinder18:07
*** dustins_ has joined #openstack-cinder18:07
*** pewp has quit IRC18:08
*** dustins has quit IRC18:08
*** Apoorva_ has quit IRC18:09
*** Apoorva has joined #openstack-cinder18:10
*** pewp has joined #openstack-cinder18:11
*** salv-orl_ has quit IRC18:17
openstackgerritEric Young proposed openstack/cinder master: ScaleIO: Logging cleanup  https://review.openstack.org/47622518:18
*** stvnoyes has left #openstack-cinder18:19
mchlumsky@eric_young, we disabled the Zero padding feature in SIO for performance concerns which means we can get data from previously deleted volumes. With the volume_clear feature we could quickly zero-out the first few megabytes and no longer have a filesystem.18:20
eric_youngmchlumsky: make sense but please follow up with the normal support channels so this enhancement can get prioritized. It's possible I can support something like that in Pike+18:24
mchlumsky@eric_young, will do. I'll open a bug report as well. thanks!18:25
*** jwcroppe has quit IRC18:50
*** wewe0901 has quit IRC19:05
*** nicolasbock has quit IRC19:07
*** arnewiebalck_ has joined #openstack-cinder19:25
*** lkuchlan has joined #openstack-cinder19:28
*** lkuchlan has quit IRC19:28
*** kaisers_ has quit IRC19:36
*** ChubYann has joined #openstack-cinder19:48
*** crushil has quit IRC19:50
*** jwcroppe has joined #openstack-cinder20:00
*** crushil has joined #openstack-cinder20:02
*** openstackgerrit has quit IRC20:03
*** salv-orlando has joined #openstack-cinder20:04
*** kaisers_ has joined #openstack-cinder20:05
*** gouthamr has quit IRC20:09
*** liverpooler has quit IRC20:12
*** jwcroppe_ has joined #openstack-cinder20:17
*** jwcroppe has quit IRC20:21
*** openstackgerrit has joined #openstack-cinder20:22
openstackgerritxing-yang proposed openstack/cinder master: Functional tests for group replication  https://review.openstack.org/47278420:22
openstackgerritxing-yang proposed openstack/python-cinderclient master: Tiramisu: replication group support  https://review.openstack.org/35222920:23
*** gouthamr has joined #openstack-cinder20:30
*** gouthamr has quit IRC20:35
openstackgerritChaynika Saikia proposed openstack/python-cinderclient master: Closes-Bug: 1696556  https://review.openstack.org/47626420:38
openstackbug 1696556 in python-cinderclient "UnboundLocalError on message-list" [Undecided,In progress] https://launchpad.net/bugs/1696556 - Assigned to Chaynika Saikia (csaikia)20:38
*** arnewiebalck_ has quit IRC20:40
*** gouthamr has joined #openstack-cinder20:48
*** crushil has quit IRC20:53
*** lucasxu has quit IRC20:54
*** mchlumsky has quit IRC21:08
*** crushil has joined #openstack-cinder21:09
openstackgerritChaynika Saikia proposed openstack/python-cinderclient master: UnboundLocalError on message-list  https://review.openstack.org/47626421:09
*** abishop has quit IRC21:13
*** chlong has quit IRC21:22
*** eric_young has quit IRC21:28
openstackgerritMerged openstack/python-cinderclient master: Enabled like filter support in client  https://review.openstack.org/46621421:37
*** cknight has quit IRC21:49
*** catintheroof has quit IRC21:51
*** markvoelker has quit IRC22:04
*** markvoelker_ has joined #openstack-cinder22:06
*** markvoelker_ has quit IRC22:11
*** felipemonteiro_ has quit IRC22:12
*** salv-orl_ has joined #openstack-cinder22:12
*** salv-orlando has quit IRC22:15
*** salv-orl_ has quit IRC22:15
*** Julien-zte has joined #openstack-cinder22:15
*** jwcroppe has joined #openstack-cinder22:15
*** salv-orlando has joined #openstack-cinder22:16
*** markvoelker has joined #openstack-cinder22:17
*** xyang1 has quit IRC22:18
*** jwcroppe_ has quit IRC22:19
*** jwcroppe has quit IRC22:20
*** salv-orlando has quit IRC22:21
*** dustins_ has quit IRC22:39
*** Julien-zte has quit IRC22:50
*** Julien-zte has joined #openstack-cinder22:51
*** Apoorva_ has joined #openstack-cinder23:01
*** Apoorva has quit IRC23:05
*** sdague has quit IRC23:11
*** Julien-zte has quit IRC23:32
*** diablo_rojo has quit IRC23:33
*** Apoorva_ has quit IRC23:33
*** Apoorva has joined #openstack-cinder23:34
*** hoonetorg has quit IRC23:39
*** flwang2 has quit IRC23:42
*** s-shiono has joined #openstack-cinder23:43
*** salv-orlando has joined #openstack-cinder23:47
*** salv-orlando has quit IRC23:52
*** KeithMnemonic has joined #openstack-cinder23:54
*** hoonetorg has joined #openstack-cinder23:56

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!