Monday, 2018-09-24

« Sunday, 2018-09-23 Index Tuesday, 2018-09-25 »
*** hoangcx has joined #openstack-cinder00:41
*** imacdonn has quit IRC01:17
*** imacdonn has joined #openstack-cinder01:17
*** mschuppert has quit IRC01:25
*** brinzhang has joined #openstack-cinder01:34
*** rcernin has quit IRC01:40
*** rcernin has joined #openstack-cinder01:40
*** Bhujay has joined #openstack-cinder02:04
*** Bhujay has quit IRC02:12
*** imacdonn has quit IRC02:35
*** psachin has joined #openstack-cinder02:40
*** imacdonn has joined #openstack-cinder02:50
*** sapd1 has joined #openstack-cinder03:05
*** whoami-rajat has joined #openstack-cinder03:06
*** vivsoni has joined #openstack-cinder03:25
vivsonihi team, please review03:32
vivsonihttps://review.openstack.org/#/c/603448/03:32
vivsonithanks03:32
*** psachin has quit IRC03:48
*** jdillaman1 has quit IRC03:55
*** icey has quit IRC03:56
*** icey has joined #openstack-cinder03:57
*** psachin has joined #openstack-cinder04:00
*** AJaeger has left #openstack-cinder04:06
*** vivsoni has quit IRC04:10
*** pooja_jadhav has joined #openstack-cinder04:14
*** vivsoni has joined #openstack-cinder04:30
*** vivsoni has quit IRC04:37
*** vivsoni has joined #openstack-cinder04:37
openstackgerritRajat Dhasmana proposed openstack/cinder master: Fix multiattach set to false after retype  https://review.openstack.org/60404004:38
*** vivsoni has quit IRC04:43
*** vivsoni has joined #openstack-cinder04:46
*** Bhujay has joined #openstack-cinder04:58
*** spsurya has joined #openstack-cinder05:02
*** dave-mccowan has joined #openstack-cinder05:36
openstackgerritDhinesh Balasubramaniam proposed openstack/cinder master: Hedvig Cinder driver implementation  https://review.openstack.org/43334105:45
*** jiapei has joined #openstack-cinder05:46
*** Bhujay has quit IRC05:53
*** Bhujay has joined #openstack-cinder05:59
*** dpawlik has joined #openstack-cinder06:03
*** pcaruana has joined #openstack-cinder06:05
openstackgerritRajat Dhasmana proposed openstack/cinder master: Fix DRBD volume driver creating a 2-volume resource  https://review.openstack.org/60468406:13
*** e0ne has joined #openstack-cinder06:23
*** Luzi has joined #openstack-cinder06:28
*** belmoreira has joined #openstack-cinder06:36
*** e0ne_ has joined #openstack-cinder06:41
*** e0ne has quit IRC06:44
*** belmoreira has quit IRC06:45
*** belmoreira has joined #openstack-cinder06:47
*** rcernin has quit IRC07:06
*** ircuser-1 has joined #openstack-cinder07:20
*** alexchadin has joined #openstack-cinder07:51
*** alexchadin has quit IRC08:14
*** nikeshm has quit IRC08:16
*** openstackgerrit has quit IRC08:22
*** Bhujay has quit IRC08:38
*** alexchadin has joined #openstack-cinder08:42
*** kukacz has quit IRC09:10
*** kukacz has joined #openstack-cinder09:12
*** ianychoi_ has joined #openstack-cinder09:30
*** Bhujay has joined #openstack-cinder09:32
*** ianychoi has quit IRC09:34
*** Bhujay has quit IRC09:37
*** e0ne_ has quit IRC09:38
*** e0ne has joined #openstack-cinder09:38
*** sapd1__ has quit IRC09:54
*** sapd1_ has joined #openstack-cinder09:59
*** Emine has joined #openstack-cinder10:01
*** e0ne has quit IRC10:08
*** brinzhang has quit IRC10:08
*** ganso has joined #openstack-cinder10:08
*** dcdamien has joined #openstack-cinder10:08
*** jiapei has quit IRC10:23
*** e0ne has joined #openstack-cinder10:27
*** e0ne has quit IRC10:40
*** Bhujay has joined #openstack-cinder10:40
*** e0ne has joined #openstack-cinder10:49
*** erlon has joined #openstack-cinder10:50
*** luizbag has joined #openstack-cinder10:54
*** sapd1 has quit IRC10:55
*** e0ne has quit IRC10:57
*** Bhujay has quit IRC11:02
*** Bhujay has joined #openstack-cinder11:05
*** Bhujay has quit IRC11:06
*** Bhujay has joined #openstack-cinder11:07
*** Bhujay has quit IRC11:08
*** Bhujay has joined #openstack-cinder11:08
*** Bhujay has quit IRC11:09
*** Bhujay has joined #openstack-cinder11:10
*** Bhujay has quit IRC11:11
*** Bhujay has joined #openstack-cinder11:11
*** mvkr has quit IRC11:17
*** alexchadin has quit IRC11:20
*** mvkr has joined #openstack-cinder11:29
*** Roamer` has quit IRC11:39
*** zul has joined #openstack-cinder11:52
*** alexchadin has joined #openstack-cinder11:59
*** alexchadin has quit IRC12:05
*** alexchadin has joined #openstack-cinder12:06
*** e0ne has joined #openstack-cinder12:10
*** psachin has quit IRC12:13
*** zul has quit IRC12:15
*** Bhujay has quit IRC12:15
*** tpsilva has joined #openstack-cinder12:16
*** Bhujay has joined #openstack-cinder12:16
*** mmethot has quit IRC12:18
*** mmethot has joined #openstack-cinder12:18
*** openstackgerrit has joined #openstack-cinder12:18
openstackgerritRajat Dhasmana proposed openstack/cinder master: Fix multiattach set to false after retype  https://review.openstack.org/60404012:18
vivsoniHi Team, Please review https://review.openstack.org/#/c/603448/12:28
*** zul has joined #openstack-cinder12:31
*** Bhujay has quit IRC12:39
*** Bhujay has joined #openstack-cinder12:40
*** Bhujay has quit IRC12:41
*** yumiriam has joined #openstack-cinder12:41
*** Bhujay has joined #openstack-cinder12:41
*** alexchadin has quit IRC12:43
*** jiapei has joined #openstack-cinder12:47
*** yumiriam has quit IRC12:52
*** jdillaman has joined #openstack-cinder12:53
*** alexchadin has joined #openstack-cinder12:53
*** icey3 has joined #openstack-cinder12:58
KeithMnemonicsmcginnis: another one ready when you have a minute https://review.openstack.org/60440812:59
KeithMnemonichope you had a nice weekend13:00
*** icey3 has quit IRC13:01
*** zul has quit IRC13:01
*** mriedem has joined #openstack-cinder13:09
*** yumiriam has joined #openstack-cinder13:16
*** lbragstad has joined #openstack-cinder13:17
*** zul has joined #openstack-cinder13:30
*** belmorei_ has joined #openstack-cinder13:30
*** belmoreira has quit IRC13:32
*** icey has quit IRC13:34
*** SteelyDan is now known as dansmith13:34
*** e0ne has quit IRC13:38
*** e0ne has joined #openstack-cinder13:47
*** edmondsw has joined #openstack-cinder13:48
*** woojay has joined #openstack-cinder13:51
*** dustins has joined #openstack-cinder13:55
mriedemso uh,13:56
mriedemhttps://developer.openstack.org/api-ref/block-storage/v3/#create-a-volume13:56
mriedemis the volume_type parameter the volume type id (uuid) or name?13:56
mriedemthe API reference isn't clear13:56
mriedemoh it's either?13:57
*** chilo has joined #openstack-cinder14:01
openstackgerritChuck Short proposed openstack/cinder master: Remove ploop from rootwrap  https://review.openstack.org/58137014:03
*** Luzi has quit IRC14:03
*** icey has joined #openstack-cinder14:09
*** belmorei_ has quit IRC14:19
*** chilo has quit IRC14:26
*** chilo has joined #openstack-cinder14:26
*** belmoreira has joined #openstack-cinder14:30
geguileomriedem: Should accept any of them14:32
geguileomriedem: we call objects.VolumeType.get_by_name_or_id(context, req_volume_type))14:33
mriedemyeah i see that now14:34
mriedemi'll update the API reference to be clear14:34
geguileomriedem: thanks14:34
*** dpawlik has quit IRC14:37
*** dpawlik has joined #openstack-cinder14:40
*** mchlumsky has joined #openstack-cinder14:40
*** dpawlik has quit IRC14:40
openstackgerritMatt Riedemann proposed openstack/cinder master: api-ref: mark name as optional in volume create API  https://review.openstack.org/60480314:43
*** mchlumsky has quit IRC14:45
*** mchlumsky has joined #openstack-cinder14:47
*** dave-mccowan has quit IRC14:50
*** gnufied has joined #openstack-cinder14:50
mriedemthe volume-type request parameter here is pretty weird https://developer.openstack.org/api-ref/block-storage/v3/#update-quota-classes-for-a-project14:51
mriedemthe volume type is really a namespace when updating quota class sets right?14:51
mriedemlike "volumes_lvmdriver-1"14:51
mriedemyeah this is more clear https://developer.openstack.org/api-ref/block-storage/v3/#update-quotas-for-a-project14:52
mriedemvolumes_{volume_type}14:52
geguileomriedem: iirc that needs to be the name, not the uuid14:52
* geguileo checks14:53
mriedemyeah should be using the "volumes_number_for_type" api ref parameter14:53
geguileojungleboyj: the deadline for the Forum topics is Wednesday, right?14:56
mriedemit is yes14:56
mriedemhttps://bugs.launchpad.net/cinder/+bug/179412014:56
openstackLaunchpad bug 1794120 in Cinder "os-quota-class-sets API reference is confusing for volume type usage" [Medium,Triaged]14:56
geguileojungleboyj: the etherpad feels a little lonely14:56
geguileomriedem: can topics be proposed on Wednesday throughout the day?14:57
jungleboyjgeguileo: Yeah, I noticed it is kind of quiet.14:57
jungleboyjI have a few follow up items from the PTG.  I can get through those today and can send another note to the ML.14:57
mriedemgeguileo: the ML email doesn't specify a time14:58
geguileojungleboyj: I remember there was at least 1 item to be added14:58
mriedem"The Forum Topic Submission session started September 12 and will run  through September 26th.  Now  is the time to wrangle the topics you gathered during your  Brainstorming Phase and start pushing forum topics through. Don't rely  only on a PTL to make the agenda...  step on up and place the items you consider important front and center."14:58
jungleboyjmriedem: Thank you.14:58
jungleboyjWe can make that a focus of discussion in our Wednesday meeting and get stuff submitted as well geguileo14:59
geguileomriedem: We'll assume it's the last day that they'll accept14:59
geguileojungleboyj: sounds good14:59
geguileojungleboyj: I wasn't sure if we had to summit them by tomorrow, or if we could wait to after the W meeting14:59
jungleboyjI interpret through the 26th to be inclusive.  I can ping Jimmy to be sure.14:59
mriedemha, the cinder v2 api-ref for os-quota-class-sets is a full copy of the request/response parameters from nova's api15:04
mriedem*os-quota-sets15:04
mriedemhttps://developer.openstack.org/api-ref/block-storage/v2/#quota-sets-extension-os-quota-sets15:06
*** _hemna has quit IRC15:07
*** Bhujay has quit IRC15:07
openstackgerritMatt Riedemann proposed openstack/cinder master: api-ref: fix req/resp params for v3 os-quota-class-sets  https://review.openstack.org/60480815:09
jungleboyjgeguileo:  Yeah, we are good through Wednesday.  So, I will send some notes today.15:11
*** dave-mccowan has joined #openstack-cinder15:12
openstackgerritMatt Riedemann proposed openstack/cinder master: api-ref: clarify volume_type param in volume create API  https://review.openstack.org/60481015:14
*** dave-mccowan has quit IRC15:17
mriedemdoes anyone know if this could be an intermittent/transient thing during volume extend of an attached volume? http://logs.openstack.org/69/595069/2/gate/tempest-full/169db43/controller/logs/screen-n-cpu.txt.gz#_Sep_24_10_55_26_75969115:22
mriedemSep 24 10:55:26.759691 ubuntu-xenial-rax-ord-0002235107 nova-compute[15157]: WARNING os_brick.initiator.connectors.iscsi [req-eb1112fc-2010-4878-b6ed-7bd6d5c65b82 req-118f3f55-90c0-47b4-98c9-9b8069625dce service nova] Couldn't find iscsi sessions because iscsiadm err: iscsiadm: could not read session targetname: 5 Sep 24 10:55:26.760040 ubuntu-xenial-rax-ord-0002235107 nova-compute[15157]: iscsiadm: could not find session i15:22
mriedemfor session2215:22
*** dave-mccowan has joined #openstack-cinder15:23
imacdonnSounds like a failure I had in a nova check job on Friday. Since my change had nothing remotely to do with extending volumes, I did a recheck (without analysing logs too deeply), and it worked the second time15:24
mriedemwell i know this is an intermittent gate failure http://status.openstack.org/elastic-recheck/#173219915:24
mriedemit's been around since we added support to cinder/nova for extending an in-use volume15:24
mriedemit's some kind of race,15:24
mriedemi'm just wondering if retrying on the nova side would help15:24
mriedemor os-brick15:25
imacdonnyeah, it seems like an os-brick thing (tag geguileo)15:25
imacdonnpondering what might cause iscsiadm to fail like that15:26
geguileoI'm in a meeting, will check as soon as I finish15:26
*** dustins has quit IRC15:26
mriedemcomments added to https://bugs.launchpad.net/nova/+bug/173219915:28
openstackLaunchpad bug 1732199 in OpenStack Compute (nova) "test_extend_attached_volume fails with Unexpected compute_extend_volume result 'Error'" [Medium,Confirmed]15:28
*** dustins has joined #openstack-cinder15:30
mriedemhmm, n-cpu might be passing stale connection_info to os-brick15:32
*** dpawlik has joined #openstack-cinder15:38
imacdonnit looks like os-brick is just running "iscsiadm -m session" (not trying to target anything in particular), and getting that error15:38
mriedemdoes volume extend in cinder create a new export?15:38
mriedemb/c nova-compute is passing connection_info to os-brick from when the volume was initially attached to the server15:38
mriedemoh i guess we don't even get that far yeah15:39
mriedemhttp://logs.openstack.org/69/595069/2/gate/tempest-full/169db43/controller/logs/syslog.txt.gz#_Sep_24_10_55_2415:42
mriedemSep 24 10:55:24 ubuntu-xenial-rax-ord-0002235107 kernel:  connection21:0: detected conn error (1020)15:42
*** dpawlik has quit IRC15:42
*** alexchadin has quit IRC15:47
geguileoI haven't looked into the os-brick extend_volume method before15:48
geguileoand it seems to be using a completely different code path than the one used when we do the attachments15:50
imacdonnlooking at that syslog, there appears to be some other lvm/tgtadm stuff going on at the time ... probably some sort of concurrency issue15:50
mriedemdoesn't c-vol have periodics that would hit that at the same time?15:51
mriedems/would/could/15:51
mriedemlvdisplay and such?15:52
imacdonnseems like it would likely pop up somewhere in cinder15:52
*** sapd1 has joined #openstack-cinder15:53
geguileoI am new to this bug, so I'm lacking too much context15:54
geguileothe failure appears on Nova after the volume has been extended?15:54
geguileoand when Nova call os-brick's volume_extend method?15:55
imacdonnlooks like it, yes16:00
imacdonnSep 24 10:55:26.743054 ubuntu-xenial-rax-ord-0002235107 nova-compute[15157]: DEBUG os_brick.initiator.connectors.iscsi [req-eb1112fc-2010-4878-b6ed-7bd6d5c65b82 req-118f3f55-90c0-47b4-98c9-9b8069625dce service nova] ==> extend_volume: call u"{'args': (<os_brick.initiator.connectors.iscsi.ISCSIConnector object at 0x7fccb3e5f4d0>, {u'device_path': u'/dev/sda', u'target_discovered': False, u'encrypted': False, u'qos_specs': None, u'target_iqn':16:00
imacdonnu'iqn.2010-10.org.openstack:volume-26727af1-a60f-4753-baf5-7e7012361915', u'target_portal': u'10.210.68.187:3260', u'volume_id': u'26727af1-a60f-4753-baf5-7e7012361915', u'auth_password': u'***', u'target_lun': 1, u'access_mode': u'rw', u'auth_username': u'AD9ppe4xymUAEPLvToKo', u'auth_method': u'CHAP'}), 'kwargs': {}}" {{(pid=15157) trace_logging_wrapper /usr/local/lib/python2.7/dist-packages/os_brick/utils.py:146}}16:00
*** e0ne has quit IRC16:01
*** lbragstad has quit IRC16:10
*** lbragstad has joined #openstack-cinder16:18
geguileoit looks like a retry would solve it16:19
geguileothough I'm not sure if this should be added to os-brick itself instead of Nova...16:20
geguileoand this could mean that we cannot do an extend while do any other iSCSI operations...16:20
geguileobecause any other operation could fail due to that iSCSI error message16:21
* geguileo checking more to confirm suspicions16:21
*** spotz is now known as spotz_16:25
*** spotz_ is now known as spotz16:25
*** icey has quit IRC16:29
*** icey has joined #openstack-cinder16:33
imacdonngeguileo: Do you think the extend is causing the problem? It looked like an innocent victim to me ....16:36
*** dave-mccowan has quit IRC16:38
openstackgerritKumar Prashant proposed openstack/cinder master: VMAX Driver - Fix for invalid device id length  https://review.openstack.org/60329416:38
jungleboyjgeguileo:  I have added some additional topics into the etherpad for the forum16:42
geguileojungleboyj: thanks, will look at it now16:43
geguileoimacdonn: I believe the problem is caused by the extend.16:43
geguileoI have updated the bug16:43
*** jiapei has quit IRC16:43
geguileojungleboyj: I like them!!!16:44
imacdonngeguileo: how did you determine that the extend is causing the connection failures ?16:44
geguileojungleboyj: Even if there are other sessions about Edge, it makes sense for us to have one specific for storage16:44
geguileoimacdonn: I believe there was nothing else happening on the node at the time (could be wrong though)16:45
jungleboyjgeguileo:  Cool.  Tried to look at the PTG and think about things that we had questions as to what the users might be doing.16:45
imacdonngeguileo: In the syslog, there is a lvm create at the time .. there is no lvm create for extend, right ?16:45
imacdonnSep 24 10:55:25 ubuntu-xenial-rax-ord-0002235107 sudo[25710]:    stack : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/local/bin/cinder-rootwrap /etc/cinder/rootwrap.conf env LC_ALL=C lvcreate -T -V 1g -n volume-939eec7d-e33e-410a-bb56-ec880cd8f1cf stack-volumes-lvmdriver-1/stack-volumes-lvmdriver-1-pool16:45
geguileoimacdonn: no, I believe there is a tgtadm update16:46
geguileosorry, tgt-admin16:46
* geguileo checks again16:46
imacdonngeguileo: there's a bunch of stuff happening around that time ... my point is that it appears that there's some other job running in parallel... so I'm not sure we can assume the extend is to blame16:47
imacdonn(although maybe it should handle the failure more elegantly)16:47
geguileoimacdonn: it could be a tgt issue as well16:47
geguileoimacdonn: does this issue happen on the LIO gate as well?16:48
imacdonnI suspect some other job is interacting with the target side while the extend job is trying to refresh its initiator side, and sometimes the timing conflicts16:48
imacdonndon't know about LIO16:48
geguileoimacdonn: the tgt should have different targets for each volume, not sharing the target16:49
*** ganso has quit IRC16:49
geguileoso it shouldn't matter16:49
imacdonn"shouldn't" :)16:49
geguileowhat you do in other target should not matter for the one we have connected16:49
geguileowell, if it's happening it's a tgt bug16:49
imacdonncould be16:50
imacdonnI wonder how the connection error is detected ... maybe the trigger was some time back16:51
imacdonnWhat does tgt-admin --update do? Notice that it's being run with the force option16:56
openstackgerritKumar Prashant proposed openstack/cinder master: VMAX Driver - Place volume in SG as part of unmanage volume  https://review.openstack.org/60328316:58
*** dpawlik has joined #openstack-cinder17:00
*** belmoreira has quit IRC17:04
openstackgerritMerged openstack/cinder master: Propose example volume protection tests  https://review.openstack.org/60248917:06
*** mvkr has quit IRC17:25
*** dave-mccowan has joined #openstack-cinder17:35
*** dpawlik has quit IRC17:51
KeithMnemonicjungleboyj: not sure if smcginnis is online so if  you get time can you please review this? https://review.openstack.org/60440817:54
jungleboyjKeithMnemonic:  Looking.  Yeah, I think he might be out today.  I haven't heard from him either.17:55
KeithMnemonicthanks!17:55
jungleboyjLooks ok.  jgriffith  Can you take a look too and merge it if possible?17:57
*** e0ne has joined #openstack-cinder18:06
*** mvkr has joined #openstack-cinder18:15
*** dpawlik has joined #openstack-cinder18:17
*** dpawlik_ has joined #openstack-cinder18:17
*** dpawlik has quit IRC18:18
imacdonnjungleboyj: Is there any policy on python style, beyond passing the zuul checks? nit-picking on https://review.openstack.org/604230 is trying my patience a bit...18:19
jungleboyjimacdonn:  Let me look.18:20
imacdonntnx18:20
jungleboyjYeah, that one does seem to be stuck in a nit-pick loop.18:24
jungleboyjLet me respond.18:24
imacdonnthanks ... the code that's being picked on is old - either moved around, or replicated ... if the style is wrong, the whole driver code needs to be revamped18:27
openstackgerritKumar Prashant proposed openstack/cinder master: VMAX Driver - Fix for manage volume if volume is part of SG  https://review.openstack.org/60430318:30
*** ianychoi_ is now known as ianychoi18:54
*** sapd1 has quit IRC19:15
*** pcaruana has quit IRC19:17
KeithMnemonicjgriffith: Thanks for any review you can do. https://review.openstack.org/60440819:42
*** pck has quit IRC19:45
*** pck has joined #openstack-cinder19:45
*** dpawlik_ has quit IRC19:51
*** dpawlik has joined #openstack-cinder19:55
*** dpawlik has quit IRC19:56
*** erlon has quit IRC20:05
*** dpawlik has joined #openstack-cinder20:13
*** dpawlik has quit IRC20:18
*** crazik has quit IRC20:18
*** luizbag has quit IRC20:28
*** e0ne has quit IRC20:37
*** dpawlik has joined #openstack-cinder20:40
*** amito has quit IRC21:10
*** amito has joined #openstack-cinder21:11
*** dpawlik has quit IRC21:12
openstackgerritMerged openstack/cinder master: ZFSSA handle manage nonexistent volume  https://review.openstack.org/60392521:13
jgriffithKeithMnemonic: you really should consider just using provider-id strings for that mapping21:40
jgriffithit would make your life significantly easier and you wouldn't need to call out to your api at all21:41
KeithMnemonicjgriffith: thanks, I was just pulling this change in from the code already in rocky. I can see if there are any more changes that address this in rocky that are not in pike21:55
jgriffithKeithMnemonic: yeah, I hear ya; that's why I just pointed it out.  Not something for backports, but might be a nice improvement for you this upcoming release :)21:56
*** mriedem is now known as mriedem_away21:57
*** dustins has quit IRC22:01
*** Emine has quit IRC22:03
*** luizbag has joined #openstack-cinder22:13
*** luizbag has quit IRC22:13
*** dpawlik has joined #openstack-cinder22:16
*** dpawlik has quit IRC22:20
*** rcernin has joined #openstack-cinder22:45
*** pooja-jadhav has joined #openstack-cinder22:51
*** kukacz_ has joined #openstack-cinder22:55
*** dims_ has joined #openstack-cinder22:59
*** Reepicheep_ has joined #openstack-cinder23:00
*** kukacz has quit IRC23:00
*** pooja_jadhav has quit IRC23:00
*** dims has quit IRC23:00
*** cburgess has quit IRC23:00
*** strigazi has quit IRC23:00
*** Reepicheep has quit IRC23:00
*** kambiz has quit IRC23:00
*** andreykurilin has quit IRC23:03
*** chilo has quit IRC23:04
*** andreykurilin has joined #openstack-cinder23:05
*** tpsilva has quit IRC23:43
*** erlon has joined #openstack-cinder23:46
« Sunday, 2018-09-23 Index Tuesday, 2018-09-25 »

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!