Wednesday, 2018-05-09

*** bkopilov__ has joined #openstack-cinder00:06
*** bkopilov_ has joined #openstack-cinder00:06
*** MasterofJOKers has quit IRC00:17
*** dgonzalez has quit IRC00:17
*** dgonzalez has joined #openstack-cinder00:17
*** MasterofJOKers has joined #openstack-cinder00:18 proposed openstack/cinder master: Fix i18n issue in volume_action and admin_action
*** lixiaoy1 has joined #openstack-cinder00:33
*** Nel1x has joined #openstack-cinder00:40
*** Dinesh_Bhor has joined #openstack-cinder00:42
*** mriedem has quit IRC00:44
*** harlowja has quit IRC00:51
*** andreas_s has joined #openstack-cinder00:54
*** nicolasbock has quit IRC00:57
*** andreas_s has quit IRC00:58
*** markstur has quit IRC01:04
*** markstur has joined #openstack-cinder01:05
*** felipemonteiro__ has joined #openstack-cinder01:08
*** markstur has quit IRC01:10
*** kaisers_ has joined #openstack-cinder01:12
*** kaisers has quit IRC01:15
*** armaan has quit IRC01:15
*** armaan has joined #openstack-cinder01:16
*** username_ has joined #openstack-cinder01:21
openstackgerritMerged openstack/cinder master: RBD snapshot needs to be protected/unprotected after being managed/unmanaged
openstackgerritMerged openstack/cinder master: Update the HP MSA and Lenovo driver documentation
*** username_ is now known as username__01:23
*** felipemonteiro__ has quit IRC01:27
*** zhaochao has joined #openstack-cinder01:27
openstackgerritMerged openstack/os-brick master: Fix bindep for multipath
*** dalgaaf has quit IRC01:42
openstackgerritMerged openstack/cinder master: Parent of RBD cloned volume marked as deleted should not be manageable
openstackgerritMerged openstack/cinder master: Collect resource id from RequestSpec parameter
openstackgerritMerged openstack/cinder master: [doc] Add the possible status for attachment
openstackgerritMerged openstack/cinder master: IBM XIV: enable FC zoning to all the ports
openstackgerritMerged openstack/cinder stable/ocata: RBD: Handle ImageNotFound exception in _get_usage_info correctly
openstackgerritMerged openstack/cinder master: RBD: add support for active/active replication
*** dalgaaf has joined #openstack-cinder01:45
*** psachin` has joined #openstack-cinder02:09
openstackgerritTommyLike proposed openstack/cinder master: Support availability-zone type
*** s-shiono has joined #openstack-cinder02:15
*** psachin` has quit IRC02:26
*** username__ has quit IRC02:35
*** psachin has joined #openstack-cinder02:41
*** rcernin has quit IRC03:14
*** vivsoni has quit IRC03:25
*** vivsoni has joined #openstack-cinder03:25
*** andreas_s has joined #openstack-cinder03:28
*** andreas_s has quit IRC03:32
*** threestrands has joined #openstack-cinder03:38
*** threestrands has quit IRC03:38
*** threestrands has joined #openstack-cinder03:38
*** links has joined #openstack-cinder03:43
*** gyan_ has joined #openstack-cinder03:44
*** dave-mccowan has quit IRC03:48
*** lkuchlan has joined #openstack-cinder03:52
*** Nel1x has quit IRC04:08
openstackgerritMerged openstack/cinder stable/pike: Improve ChunkedBackupDriver hashlib calls
openstackgerritMerged openstack/cinder stable/queens: [Optimize] Validate configured scheduler filters when initialize
*** cburgess_ has quit IRC04:16
*** markstur has joined #openstack-cinder04:21
*** lkuchlan has quit IRC04:25
*** markstur_ has joined #openstack-cinder04:26
*** cburgess has joined #openstack-cinder04:26
*** markstur has quit IRC04:29
*** abhishekk has joined #openstack-cinder04:55
*** Dinesh_Bhor has quit IRC05:00
*** Dinesh_Bhor has joined #openstack-cinder05:03
*** abhishekk has quit IRC05:31
*** hoonetorg has quit IRC05:33
*** moshele has joined #openstack-cinder05:34
*** kaisers_ has quit IRC05:34
*** pgadiya has joined #openstack-cinder05:37
*** pgadiya has quit IRC05:37
*** rajinir has quit IRC05:38
*** fnordahl has quit IRC05:40
*** fnordahl has joined #openstack-cinder05:42
*** hoonetorg has joined #openstack-cinder05:45
*** e0ne has joined #openstack-cinder05:46
*** evin has quit IRC05:47
*** e0ne has quit IRC05:50
*** kaisers has joined #openstack-cinder05:51
*** lkuchlan has joined #openstack-cinder05:51
*** threestrands has quit IRC05:54
*** pcaruana has joined #openstack-cinder05:57
*** armaan has quit IRC06:05
*** armaan has joined #openstack-cinder06:05
*** armaan_ has joined #openstack-cinder06:13
*** markstur_ has quit IRC06:14
*** evin has joined #openstack-cinder06:15
*** armaan has quit IRC06:16
*** Dinesh_Bhor has quit IRC06:16
*** Dinesh_Bhor has joined #openstack-cinder06:18
*** hamdyk has joined #openstack-cinder06:20
*** andreas_s has joined #openstack-cinder06:25
*** andreas_s has quit IRC06:25
*** andreas_s has joined #openstack-cinder06:25
*** markstur has joined #openstack-cinder06:28
*** markstur has quit IRC06:33
*** MasterofJOKers_ has joined #openstack-cinder06:33
*** armaan_ has quit IRC06:33
*** armaan has joined #openstack-cinder06:34
*** MasterofJOKers has quit IRC06:34
*** MasterofJOKers_ is now known as MasterofJOKers06:34
openstackgerritRyan Liang proposed openstack/cinder master: [Unity] Add support of removing empty host
*** alexchadin has joined #openstack-cinder07:16
openstackgerritYingzhe Zeng proposed openstack/cinder master: Huawei driver supports create_group_from_src
*** moshele has quit IRC07:20
*** tesseract has joined #openstack-cinder07:22
openstackgerritYu Zhang proposed openstack/cinder master: IBM XIV: Report backend state in service list
*** moshele has joined #openstack-cinder07:24
*** moshele has quit IRC07:25
openstackgerritYu Zhang proposed openstack/cinder stable/queens: IBM XIV: enable FC zoning to all the ports
openstackgerritliuyamin proposed openstack/python-cinderclient master: Switch from ostestr to stestr
*** masber has quit IRC07:36
*** abhishekk has joined #openstack-cinder07:40
*** alexchadin has quit IRC07:41
*** alexchadin has joined #openstack-cinder07:42
*** arnewiebalck_ has joined #openstack-cinder07:46
*** armaan has quit IRC07:50
*** armaan has joined #openstack-cinder07:51
*** Guest17 has joined #openstack-cinder07:59
*** namnh has joined #openstack-cinder08:02
*** Guest17 is now known as MikeStack08:03
MikeStackDoes anyone here work with Cinder volumes with a different driver than LVM?08:04
*** armaan has quit IRC08:05
*** armaan has joined #openstack-cinder08:06
*** liuzz_ has quit IRC08:07
*** liuzz has joined #openstack-cinder08:13
*** s-shiono has quit IRC08:18
*** lpetrut has joined #openstack-cinder08:33
*** Kensuke has joined #openstack-cinder08:34
*** edmondsw has joined #openstack-cinder08:35
*** bkopilov__ has quit IRC08:35
*** Kensuke has quit IRC08:36
*** bkopilov_ has quit IRC08:36
*** andreas_s has quit IRC08:38
*** liuzz_ has joined #openstack-cinder08:39
*** arnewiebalck_ has quit IRC08:39
*** edmondsw has quit IRC08:40
*** andreas_s has joined #openstack-cinder08:40
*** liuzz has quit IRC08:42
*** arnewiebalck_ has joined #openstack-cinder08:44
*** bkopilov has joined #openstack-cinder08:50
*** bkopilov_ has joined #openstack-cinder08:50
*** gyankum has joined #openstack-cinder08:55
*** alexchadin has quit IRC08:55
*** lixiaoy1 has quit IRC08:56
*** nicolasbock has joined #openstack-cinder08:56
*** arnewiebalck_ has quit IRC09:00
*** gkadam__ has joined #openstack-cinder09:06
*** Dinesh_Bhor has quit IRC09:32
*** lkuchlan has quit IRC09:45
vivsoniIn unmounting a volume, we are using os-brick's disconnect_volume() function09:45
vivsoniand this is failing here,
vivsonibasically, it fails in executing command  'multipath -l /dev/sdb'09:47
vivsonii am able to execute the same succesfully in terminal09:47
vivsoniraise exception.CommandExecutionFailed(cmd='multipath -l %s' % device)09:47
vivsoniany suspected root cause ?09:47
*** neha_alhat has quit IRC09:50
*** mszwed has quit IRC10:05
*** mszwed has joined #openstack-cinder10:06
*** namnh has quit IRC10:28
*** abhishekk has quit IRC10:36
*** erlon has joined #openstack-cinder10:51
*** ganso has joined #openstack-cinder10:51
*** andreas_s_ has joined #openstack-cinder10:56
*** psachin` has joined #openstack-cinder10:56
*** andreas_s has quit IRC10:57
*** psachin has quit IRC10:57
*** mchlumsky_ has joined #openstack-cinder10:59
*** gkadam_ has joined #openstack-cinder10:59
*** gkadam__ has quit IRC10:59
*** andreas_s has joined #openstack-cinder10:59
*** mchlumsky has quit IRC10:59
*** andreas_s_ has quit IRC11:01
*** lkuchlan has joined #openstack-cinder11:21
*** Nel1x has joined #openstack-cinder11:37
*** armaan has quit IRC11:41
*** armaan has joined #openstack-cinder11:41
*** hemna_ has quit IRC11:53
*** lkuchlan has quit IRC12:00
*** pchavva has joined #openstack-cinder12:06
*** edmondsw has joined #openstack-cinder12:10
*** armaan has quit IRC12:16
*** jmccarthy has joined #openstack-cinder12:17
*** tpsilva has joined #openstack-cinder12:18
*** lkuchlan has joined #openstack-cinder12:20
*** jmlowe has quit IRC12:20
*** pcaruana has quit IRC12:21
*** abishop has joined #openstack-cinder12:25
*** Nel1x has quit IRC12:28
*** armaan has joined #openstack-cinder12:30
jmccarthyI've got a question about this here:
jmccarthyI'm able to attach the unencrypted volume without issue, but the encrypted volume briefly says attaching, before going back to available (and fails to attach to instance of course)12:32
jmccarthyI'm not seeing anything obviously wrong in the configs or debug logs anything like that at the moment .. any suggestions to narrow down what the issue may be ?12:33
*** dave-mccowan has joined #openstack-cinder12:44
*** gyan_ has quit IRC12:51
*** gyankum has quit IRC12:51
*** dustins has joined #openstack-cinder12:52
*** lpetrut has quit IRC12:53
*** armaan has quit IRC12:54
*** arnewiebalck_ has joined #openstack-cinder12:55
*** arnewiebalck_ has quit IRC12:57
*** andreas_s has quit IRC13:00
*** andreas_s has joined #openstack-cinder13:00
*** arnewiebalck_ has joined #openstack-cinder13:02
*** arnewiebalck_ has quit IRC13:02
*** jmlowe has joined #openstack-cinder13:02
*** andreas_s has quit IRC13:03
*** andreas_s has joined #openstack-cinder13:04
*** psachin` has quit IRC13:04
*** sapd has quit IRC13:05
*** liuzz_ has quit IRC13:08
*** lpetrut has joined #openstack-cinder13:09
*** arnewiebalck_ has joined #openstack-cinder13:10
*** liverpooler has joined #openstack-cinder13:10
*** mriedem has joined #openstack-cinder13:13
*** andreas_s has quit IRC13:16
*** throwsb1 has joined #openstack-cinder13:19
jmccarthyI'm trying out, the unencrypted part is fine, but for the encrypted volume it goes wrong:
jmccarthyAfter this basically "cryptsetup isLuks --verbose /dev/disk/by-id/scsi-36001405fd9938e58a604414b0067aabc" returned: 1 "13:26
jmccarthyAny folks about who are familiar with this area ?13:27
*** andreas_s_ has joined #openstack-cinder13:28
*** andreas_s_ has quit IRC13:28
*** andreas_s_ has joined #openstack-cinder13:29
openstackgerritMerged openstack/cinder master: Replace cluster object's host with service_topic_queue
*** andreas_s_ has quit IRC13:31
*** andreas_s_ has joined #openstack-cinder13:31
*** crose has joined #openstack-cinder13:33
*** felipemonteiro__ has joined #openstack-cinder13:34
*** armaan has joined #openstack-cinder13:34
*** arnewiebalck_ has quit IRC13:36
smcginnistommylikehu, geguileo, e0ne (if you were here): Please take a look at
*** pcaruana has joined #openstack-cinder13:40
*** mriedem1 has joined #openstack-cinder13:43
*** mriedem has quit IRC13:46
*** mriedem1 is now known as mriedem13:46
*** andreas_s_ has quit IRC13:53
*** felipemonteiro_ has joined #openstack-cinder13:53
*** luizbag has joined #openstack-cinder13:53
*** andreas_s has joined #openstack-cinder13:53
tommylikehusmcginnis:  oh, it's comming :)13:54
*** evin has quit IRC13:55
smcginnisSeems crazy it's only two weeks away.13:55
tommylikehusmcginnis:  hope you can have a great discussion there and have a better solution for this13:56
*** felipemonteiro__ has quit IRC13:56
smcginnistommylikehu: Me too. Please add any thoughts or notes you can to the etherpad that you think might help with that.14:00
*** hamdyk has quit IRC14:00
*** hamdyk has joined #openstack-cinder14:00
tommylikehusmcginnis:  of course14:00
smcginnistommylikehu: Thanks! :)14:01
*** andreas_s has quit IRC14:02
*** lpetrut has quit IRC14:02
*** arnewiebalck_ has joined #openstack-cinder14:05
*** pcaruana has quit IRC14:05
*** andreas_s has joined #openstack-cinder14:07
*** andreas_s has quit IRC14:08
*** pcaruana has joined #openstack-cinder14:08
*** lpetrut has joined #openstack-cinder14:08
*** andreas_s has joined #openstack-cinder14:08
*** gnufied has joined #openstack-cinder14:16
*** lpetrut has quit IRC14:18
*** eharney has quit IRC14:24
*** pcaruana has quit IRC14:25
*** pcaruana has joined #openstack-cinder14:26
*** arnewiebalck_ has quit IRC14:28
*** andreas_s has quit IRC14:30
*** andreas_s has joined #openstack-cinder14:31
*** luizbag has quit IRC14:31
*** luizbag has joined #openstack-cinder14:32
*** andreas_s_ has joined #openstack-cinder14:34
*** felipemonteiro__ has joined #openstack-cinder14:34
*** andreas_s has quit IRC14:36
*** felipemonteiro_ has quit IRC14:37
*** r-daneel has joined #openstack-cinder14:37
*** evin has joined #openstack-cinder14:37
*** andreas_s_ has quit IRC14:38
MikeStacki have a basic question14:39
MikeStackmy cinder volume service in the horizon is in down however when i check it in the compute it is running14:39
*** _alastor_ has quit IRC14:39
MikeStacki looked in all of the logs and i can't find something clear14:40
*** r-daneel_ has joined #openstack-cinder14:41
smcginnisMikeStack: Have you done "cinder service-list" on the command line to see what that reports?14:42
*** r-daneel has quit IRC14:42
*** r-daneel_ is now known as r-daneel14:42
MikeStackthe scheduler is up and the cinder volume is enabled but down14:43
MikeStackalso it seems to be constantly updating since in the horizon i see "Last updated 0 sec"14:44
smcginnisOK, so horizon is out of the picture. There an actual issue with your cinder service.14:44
*** andreas_s has joined #openstack-cinder14:44
smcginnisMikeStack: Have you looked in the cinder logs?14:44
smcginnisMikeStack: There should be some error in their stating why the service is down.14:45
MikeStack ERROR cinder.service [-] Manager for service cinder-volume 3ovSdpdk@lvm is reporting problems, not sending heartbeat. Service will appear "down".14:46
MikeStackI get these messages14:46
smcginnisMikeStack: There should be messages from the actual cinder-volume service.14:48
*** jdillaman has quit IRC14:48
*** hamdyk has quit IRC14:48
*** andreas_s has quit IRC14:49
jungleboyjMikeStack:  May need to look further back in the /var/log/cinder/volume.log to see the first failure that has caused the volume service to be down.14:49
MikeStacki attached some of the log from cinder-volume maybe you could understand something14:50
smcginnisMikeStack: You might need to restart the service to get something. That looks like random runtime activity.14:51
*** srihas has quit IRC14:52
mriedemi'm debugging a gate failure where all the test is doing is resetting a group snapshot status from available to creating,
MikeStackOk thanks i will clear it and resend14:52
mriedemlooking at the API, all this does is update the record in the db; it returns a 202 though i'm not sure why as there is no async rpc cast or anything14:53
*** hemna_ has joined #openstack-cinder14:53
mriedemthe test times out waiting for the group snapshot status to go to creating and never does - again, not sure why, there are no errors in the c-api logs14:53
smcginnisI'm only seeing the create and then the delete for the volume group.14:55
MikeStacksmcginnis this is in debug mode so there is much more output in the log14:55
smcginnisMikeStack: You should be able to search for ERROR and WARNING messages.14:55
*** felipemonteiro__ has quit IRC14:55
*** felipemonteiro_ has joined #openstack-cinder14:55
mriedemthis is the entry in the c-api logs
jungleboyjsmcginnis:  Wonder if something with the API validation changes broke something?14:56
*** lkuchlan has quit IRC14:56
smcginnisjungleboyj: That's a possibility.14:57
MikeStacksmcginnis: these are the errors when i grep14:58
MikeStackfatal_exception_format_errors  = False log_opt_values /usr/lib/python2.7/dist-packages/oslo_config/
MikeStackpublish_errors                 = False log_opt_values /usr/lib/python2.7/dist-packages/oslo_config/
smcginnisDid you grep -i? It will be ERROR.14:59
*** _alastor_ has joined #openstack-cinder14:59
*** lkuchlan has joined #openstack-cinder15:00
mriedemjungleboyj: smcginnis: dno't think it's schema validation because that would be a 400 response if it were wrong15:00
MikeStackMySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION _check_effective_sql_mode /usr/lib/python2.7/dist-packages/oslo_db/sqlalchemy/
jungleboyjmriedem:  Ahhh .. true enough.15:00
mriedemlooks like something just caused this though
jungleboyjMikeStack:  That is a normal part of start up.15:01
*** andreas_s has joined #openstack-cinder15:01
MikeStackO Ok thanks15:01
openstackLaunchpad bug 1770179 in Cinder "GroupSnapshotsV319Test.test_reset_group_snapshot_status intermittently fails waiting to reset status to 'creating'" [Undecided,Confirmed]15:01
mriedemoh maybe not, i see a hit on 5/315:02
*** jdillaman has joined #openstack-cinder15:02
smcginnismriedem: 8 failes in the last week? Doesn't seem like anything too unusual.15:02
mriedemwhat's unusual is that a synchronous API call to update a record in the database is not updating the record, and not failing in the api15:03
*** markstur has joined #openstack-cinder15:03
*** markstur has quit IRC15:03
*** markstur has joined #openstack-cinder15:03
mriedemunless it's a bad set of tests that are making changes to the same group at the same time?15:03
*** edmondsw has quit IRC15:04
smcginnisAnd no sign other than in the API log. Network issues? Or that.15:04
mriedemthe test creates and deletes the group itself so i don't think it's tests sharing the same resource15:04
mriedemnothing in the api logs wrt failures15:04
mriedemgets the request, and returns 20215:05
*** lpetrut has joined #openstack-cinder15:05
mriedemMay 08 17:16:37.706719 ubuntu-xenial-inap-mtl01-0003921471 devstack@c-api.service[29070]: DEBUG cinder.api.v3.group_snapshots [None req-e73546a4-7365-4132-b289-0b1aee94e2ac tempest-GroupSnapshotsV319Test-838719435 tempest-GroupSnapshotsV319Test-838719435] Updating group 'e2d2b2bf-dde6-4357-9af1-79b57017690a' with 'creating' {{(pid=29072) _reset_status /opt/stack/new/cinder/cinder/api/v3/}}15:05
*** andreas_s has quit IRC15:06
smcginnisAnd all it does is call into the scheduler and update the object:
smcginnisI suppose we could throw a debug log in there, but doesn't seem like that should be necessary.15:07
*** gyankum has joined #openstack-cinder15:08
*** gyan_ has joined #openstack-cinder15:08
mriedemcall into the scheduler?15:09
mriedemthere is no rpc cast to another service here,15:09
mriedemit's just c-api -> db update15:10
smcginnisYeah, right.15:10
mriedemthat's why i was at first confused by the 202 response15:10
smcginnisYeah, this looks like it should all be synchronous and should have completed by the time it returns:
mriedemdug into the GroupSnapshot and DB API code for this, i don't see anything obvious that would like, ignore the update15:11
smcginnisSo not sure why we return a 202 there.15:11
mriedemwhoever added the api probably just copied the 202 response from another api15:11
mriedemwithout thinking about what 202 means15:11
mriedemanyway, that's tangetial15:11
smcginnisYeah, that was touched, but just to change 202 to a constant.15:12
*** felipemonteiro_ has quit IRC15:16
*** felipemonteiro_ has joined #openstack-cinder15:16
*** zhaochao has quit IRC15:21
mriedemdoes something update the group snapshot status in the backgroup?15:21
*** eharney has joined #openstack-cinder15:22
mriedemlike, could it be going from creating -> available (normal) -> creating (test calls reset_status) -> available (something updates it from the background)?15:22
*** eharney has quit IRC15:22
geguileojungleboyj: I have no updates on the HA front, though I'm really hoping to finish the CSI driver POC by next week and be able to get back to the HA A/A stuff  :-(15:23
jungleboyjgeguileo:  Ok.  Thanks for the update.  Glad it is on the roadmap.15:23
smcginnismriedem: Not that I know of. xyang, you around? Do you happen to remember?15:23
geguileojungleboyj: it's *always* in the roadmap, and it's killing me that I haven't been able to finish that   :-(15:24
jungleboyj:-)  Understood.  Having the same struggles here too.15:24
*** jmccarthy has quit IRC15:24
*** links has quit IRC15:25
*** armaan has quit IRC15:25
*** jmccarthy has joined #openstack-cinder15:25
*** armaan has joined #openstack-cinder15:26
*** rajinir has joined #openstack-cinder15:26
tommylikehumriedem, smcginnis are you saying why it's 202?15:31
smcginnistommylikehu: Yeah15:31
tommylikehulooks like it was introduced by my first feature commit15:32
*** lkuchlan has quit IRC15:32
smcginnisOh right, I see that now.15:32
tommylikehusmcginnis:  and this ?
smcginnistommylikehu: Was there a reason for 202, or just didn't realize that really should only be used for async?15:33
tommylikehusmcginnis: haha, I thought we always use ACCEPTED for this kind of things15:36
tommylikehusee our admin actions15:36
tommylikehushould be a mistake15:36
*** dpawlik_ has quit IRC15:37
smcginnistommylikehu: Yeah, that's probably what is used most of the time, but I think the majority of them run asynchronous so need ACCEPTED rather than SUCCESS (202 vs 200).15:38
tommylikehu@smcginnis  I mean our existing reset_status actions are already returning 202 when I prepared landing reset group and group snapshot status features15:41
smcginnistommylikehu: OK, probably a pre-existing mistake then.15:41
*** evin has quit IRC15:42
tommylikehusmcginnis:  we can fix all of them:)15:42
smcginnisYeah, that should be an easy one. And we've checked before with the API-WG that it's fine switching from one success code to a different success code if it's making it "more correct".15:43
*** dpawlik has joined #openstack-cinder15:44
*** erlon_ has joined #openstack-cinder15:53
*** markstur_ has joined #openstack-cinder16:12
*** markstur has quit IRC16:16
*** r-daneel has quit IRC16:18
openstackgerritJon Bernard proposed openstack/cinder master: Add support for promoting a failed over backend
*** lpetrut has quit IRC16:25
*** gyankum has quit IRC16:33
*** imacdonn has quit IRC16:34
*** imacdonn has joined #openstack-cinder16:34
*** gyan_ has quit IRC16:34
*** lpetrut has joined #openstack-cinder16:34
*** masber has joined #openstack-cinder16:40
jgriffith@g/ws 2016:40
*** r-daneel has joined #openstack-cinder16:54
*** armaan has quit IRC16:55
*** armaan has joined #openstack-cinder16:55
*** evin has joined #openstack-cinder16:58
*** jmccarthy has left #openstack-cinder16:59
*** lpetrut has quit IRC17:00
*** lpetrut has joined #openstack-cinder17:01
*** tesseract has quit IRC17:15
*** gouthamr has quit IRC17:27
*** mchlumsky_ has quit IRC17:35
*** mchlumsky has joined #openstack-cinder17:37
*** eharney has joined #openstack-cinder17:38
mnaseri'm trying to figure out a specific what-seems-to-be-a-bug between nova and cinder17:38
mnaserpike nova + queens cinder... volume fails to get provisioned (seems to instantly go into 'error' state)17:38
mnasernova doesnt give any more info, so i figured i'd push up a patch to get a bit more info, but it looks like cinder doesn't have some "failure reason" exposed anywhere in the api?17:39
mnaser(unless i'm missing it)17:39
smcginnismnaser: Something like this?
mnaserTIL about the messages17:41
*** mchlumsky has quit IRC17:41
smcginnismnaser: Otherwise there should be something in the lgos.17:41
mnaserthat's the strange thing, i don't see anything in the logs, i can't track the POST17:41
mnaserbut I can see the GET that nova-compute did (which it inevitebly saw the volume as error)17:41
smcginnismnaser: Weird. Do you at least see the POST in the api log?17:42
mnasersmcginnis: i can try and track down the POST but that'll be difficult, cause we don't log post data, but probably could be tracked by timestmap17:42
* mnaser looks17:42
*** mchlumsky has joined #openstack-cinder17:43
*** gkadam_ has quit IRC17:43
mnaserok so looks like this was boot from volume using cinder snapshot17:44
mnaser"Create volume request issued successfully."17:46
*** eharney has quit IRC17:48
mnaseroh hm17:48
mnaseri think i hit an osa bug17:48
openstackLaunchpad bug 1768231 in openstack-ansible "Cinder filter scheduler not enabled with multiple storage back ends" [Medium,Confirmed]17:49
mnaseris that needed when using *one* backend but with multiple cinder-volume agents?17:50
mnaserone backend is down, same exact behaviour17:51
*** dustins has quit IRC17:54
mnaserok so it looks like a ceph/rbd exception can cause a thread to fail to start17:54
mnaserError starting thread.: ImageNotFound: [errno 2] error opening image <snip> ..17:56
*** jmccarthy has joined #openstack-cinder17:57
*** armaan has quit IRC18:00
*** armaan has joined #openstack-cinder18:00
mnaser2018-05-08 22:23:39.864 128077 ERROR cinder.volume.drivers.rbd ImageNotFound: [errno 2] error opening image volume-32799ead-f32d-4831-8ba6-f9d320371537 at snapshot None18:01
mnaseri think we're hitting a race condition where stats collection is happening while a volume gets removed18:01
mnaserespecially with a large # of volumes as it is iterating18:01
*** bkopilov has quit IRC18:01
*** eharney has joined #openstack-cinder18:01
*** bkopilov_ has quit IRC18:02
mnaserwould ya look at that18:02
mnaserthanks e0ne and melwitt for fixing things once again18:03
*** r-daneel has quit IRC18:07
*** r-daneel has joined #openstack-cinder18:07
*** eharney has quit IRC18:13
*** bkopilov has joined #openstack-cinder18:14
*** armaan has quit IRC18:17
*** armaan has joined #openstack-cinder18:17
smcginnismnaser: Looks pretty relevant. :)18:18
*** armaan has quit IRC18:18
*** eharney has joined #openstack-cinder18:18
*** jmccarthy has left #openstack-cinder18:18
mnasersmcginnis: yup applying locally and going to have a look18:18
*** armaan has joined #openstack-cinder18:18
smcginnismnaser: Would be great to hear confirmation if that fixes your issue.18:19
*** dklyle has joined #openstack-cinder18:19
mnaserdigging logs18:20
*** mvenesio has quit IRC18:20
*** dklyle has quit IRC18:21
mnaserthings move quick enough to trigger this quickly18:21
*** mvenesio has joined #openstack-cinder18:21
*** gnufied has quit IRC18:22
mnaserhmm, i hate to say it but i'll double check more18:22
mnaseri think it might not be fixed :x18:22
*** gnufied has joined #openstack-cinder18:22
mnasergrep 'is not found' /var/log/cinder/cinder-volume.log gives nothing18:22
mnaserbut i still see some entires with imagenotfound exceptions18:23
mnaserthe stack trace is different now18:23
*** arnewiebalck_ has joined #openstack-cinder18:24
*** bkopilov has quit IRC18:24
*** dklyle has joined #openstack-cinder18:25
*** mvenesio has quit IRC18:25
openstackgerritVipin Balachandran proposed openstack/cinder master: Log backup_service.restore() exception
mnaserso that raises the exception there18:29
mnaserand reraises it again18:29
smcginnismnaser: So somewhere inside the RBD driver?18:29
mnaserexcept self.rbd.ImageNotFound:18:29
*** arnewiebalck_ has quit IRC18:29
mnaseri think that's not right18:29
*** bkopilov_ has joined #openstack-cinder18:29
mnaseror hmm, maybe it is18:30
mnaseris it still raising ImageNotFound18:30
smcginnisMaybe jbernard or eharney have ideas?18:32
eharneywas just reading over the log here18:32
eharneythis is a failure during stats collection?18:33
jbernardmnaser: the volume in the exception, what is it's current state?18:34
mnaserjbernard: it's gone18:35
mnaserjbernard: no error, just deleted18:35
jbernardvery recently?18:35
jbernardas in, racing with the periodic task?18:35
mnaserwe have a pretty big cloud so my guess as it is looping through the list of images, by the time it gets to it, that volume is deleted18:35
mnaser applied this locally18:35
mnaserbut it seems like the exception handling for ImageNotFound is somehow not happening18:35
mnaseralmsot as if it's another exception being raised18:36
jbernardmnaser: can you confirm _get_usage_info() is in the backtrace?18:36
mnaserjbernard: it is not in the backtrace, it used to be before applying that patch18:37
mnaserthe backtrace you see is the one i pasted18:37
*** bkopilov has joined #openstack-cinder18:38
mnaseroh you know what18:38
jbernardmnaser: there are a few places where RBDvolumeProxy.__init__ are called, you may be hitting a race we haven't seen18:38
mnaserjbernard: i think im not hitting the race18:38
mnaserit logs an exception regardless.18:39
mnaserso i think it's being handled above in the stack, but logged once in there18:39
jbernardyes, it will log but you should keep rolling18:39
melwittyeah, that's something I noticed too after the fix18:39
mnaserso i think there's no issue, it's just log noise18:39
melwittwas thinking maybe that should be adjusted to not log a trace since it's "normal" during a race18:39
mnasercause nothing is broken, but the noise is still there for logs18:39
mnaser(hence why i thought/felt that it might still be a problem)18:39
jbernardit might be good to change the nature of the log message to reduce that noise, it's going to scare anyone that sees it18:40
mnasermelwitt: i agree, if it's going to be something that is handled down the stack i guess18:40
jbernardill get something up soon18:40
mnaserit could also be something that is important to look at like.. failing to connect to the ceph cluster, so not sure how we'd deal with that18:40
melwittI think it should be logged, just not a traceback18:41
jbernardyou'll see other exceptions in that case18:41
mnasermaybe drop the 'LOG.exception("error opening rbd image %s", name)' out of the rbd volume proxy and let the caller take care of it18:42
mnaserbecause it reraises anyways18:42
mnaserso instead of seeing two tracebacks on failures, it'll be one (on unexpected failures) or nothing (if it's purposely handled/skipped)18:42
openstackgerritLuiz Benedito Aidar Gavioli proposed openstack/cinder master: Fix hostname validation regex.
*** arnewiebalck_ has joined #openstack-cinder18:44
mnaseralso on another note, it would be nice for cinder scheduler to log the volume that it's doing work for18:45
mnasergrepping logs shows nothing when using the volume id, but then digging in manually you can see the failure18:45
openstackgerritJon Bernard proposed openstack/cinder master: RBD: remove redundant exception log to reduce noise
jbernardmnaser: ^18:46
mnaserjbernard: thank you18:47
jbernardmnaser: thanks for the feedback18:47
mnasernp :)18:48
openstackgerritSean McGinnis proposed openstack/cinder master: Add descriptions for auth_strategy choices
*** dustins has joined #openstack-cinder18:49
*** mvenesio has joined #openstack-cinder18:51
mnaserok yay18:52
mnaserround #218:52
mnasercinder-volume[567641]: terminate called after throwing an instance of 'std::runtime_error'18:52
mnasercinder-volume[567641]: what():  random_device::random_device(const std::string&)18:52
smcginnisSorry, you only get one a day.18:52
mnaserill get back to digging18:52
mnaser(probably already fixed, i bet)18:52
smcginnisAny traceback or context around that error?18:52
mnasersmcginnis: no.. so what happens is "INFO oslo_service.service [req-459698ea-0f43-4056-8729-d39fef627862 - - - - -] Child 567692 killed by signal 6"18:53
mnaserand that is a cinder-volume child process for a specific backend18:53
mnaserINFO cinder.service [req-459698ea-0f43-4056-8729-d39fef627862 - - - - -] Starting cinder-volume node (version 12.0.2)18:53
mnaserwhich then starts up successfully again18:53
mnaserbut for a period of time that stays down18:54
mnaserthe time of it being killed corollates with that syslog entry18:54
smcginnisYou're just finding all the fun ones today. I haven't seen that one either.18:54
mnaserwhat is interesting the req that is tied into it18:55
mnaserdoes that mean a request crashed it18:55
mnaseri could probably help track it down18:55
*** dustins_ has joined #openstack-cinder18:56
*** dustins_ has quit IRC18:56
*** dustins_ has joined #openstack-cinder18:56
smcginnisHmm, maybe18:57
*** dustins has quit IRC18:58
mnaserseems tied up to periodic tasks18:58
luizbageharney, jungleboyj, could you review this patch:, please?18:58
*** dustins_ has quit IRC18:59
*** gouthamr has joined #openstack-cinder18:59
*** mvenesio has quit IRC19:00
*** dustins_ has joined #openstack-cinder19:00
*** arnewiebalck_ has quit IRC19:01
*** dustins_ has quit IRC19:03
*** dustins_ has joined #openstack-cinder19:03
*** pchavva has quit IRC19:03
*** dschoenb__ has joined #openstack-cinder19:06
*** dustins_ has quit IRC19:09
*** r-daneel_ has joined #openstack-cinder19:11
*** r-daneel has quit IRC19:11
*** r-daneel_ is now known as r-daneel19:11
*** felipemonteiro__ has joined #openstack-cinder19:11
*** mvk has quit IRC19:13
*** gouthamr has quit IRC19:13
*** felipemonteiro_ has quit IRC19:15
openstackgerritLuiz Benedito Aidar Gavioli proposed openstack/cinder master: Fix hostname validation regex.
*** crose has quit IRC19:16
*** ociuhandu has joined #openstack-cinder19:16
mnaserso i dug in and i think it might be the system running out of entropy, especially with the exception being raised in random_device()19:17
mnaserentropy was around 100-200 while i checked it, added haveged and going to watch it19:17
*** gouthamr has joined #openstack-cinder19:17
*** links has joined #openstack-cinder19:23
*** ociuhandu has quit IRC19:27
*** dklyle has quit IRC19:40
*** luizbag has quit IRC19:42
*** links has quit IRC19:45
*** dschoenb__ has quit IRC19:47
*** dschoenb__ has joined #openstack-cinder19:48
openstackgerritLucio Seki proposed openstack/cinder master: Fix hostname validation regex.
openstackgerritLucio Seki proposed openstack/cinder master: Fix hostname validation regex
*** edmondsw has joined #openstack-cinder19:54
*** jmlowe has quit IRC19:58
*** erlon has quit IRC20:06
*** erlon_ is now known as erlon20:06
*** MikeStack has quit IRC20:12
*** lpetrut has quit IRC20:14
*** dklyle has joined #openstack-cinder20:48
*** jmlowe has joined #openstack-cinder20:53
*** pcaruana has quit IRC20:59
*** throwsb1 has quit IRC21:07
*** mvenesio has joined #openstack-cinder21:10
*** dschoenb__ has quit IRC21:14
*** gouthamr has quit IRC21:16
*** dave-mccowan has quit IRC21:17
*** dave-mccowan has joined #openstack-cinder21:22
*** mvenesio has quit IRC21:24
*** jmlowe has quit IRC21:39
*** gouthamr has joined #openstack-cinder21:42
*** gouthamr has quit IRC21:59
openstackgerritMerged openstack/cinder master: Adding NVMET target for NVMeOF
*** erlon has quit IRC22:03
*** felipemonteiro__ has quit IRC22:06
*** zigo has quit IRC22:11
*** zigo has joined #openstack-cinder22:11
*** gouthamr has joined #openstack-cinder22:14
*** liverpooler has quit IRC22:15
*** markstur_ has quit IRC22:21
*** mvk has joined #openstack-cinder22:32
*** threestrands has joined #openstack-cinder22:33
*** threestrands has quit IRC22:33
*** threestrands has joined #openstack-cinder22:33
*** threestrands_ has joined #openstack-cinder22:36
*** threestrands has quit IRC22:38
*** dhinesh has joined #openstack-cinder22:42
*** r-daneel has quit IRC22:43
dhineshis there a list of mandatory operations(eg: create_cloned_volume ) that is required by the third party driver to qualify?22:45
*** gouthamr has quit IRC22:52
*** gouthamr has joined #openstack-cinder22:55
*** dklyle has quit IRC22:57
*** andreas_s has joined #openstack-cinder22:57
*** andreas_s has quit IRC23:01
*** mriedem has quit IRC23:02
*** _alastor_ has quit IRC23:04
*** edmondsw has quit IRC23:10
*** edmondsw has joined #openstack-cinder23:11
*** edmondsw has quit IRC23:15
smcginnisdhinesh: There's this:
smcginnisdhinesh: And the details in this:
*** ganso has quit IRC23:18
dhineshsmcginnis: cool. thanks23:20
openstackgerritSean McGinnis proposed openstack/cinder master: Add snapshots to minimum driver interface
openstackgerritDhinesh Balasubramaniam proposed openstack/cinder master: Hedvig Cinder driver implementation
*** dhinesh has quit IRC23:31
*** moshele has joined #openstack-cinder23:35
*** stakeda has joined #openstack-cinder23:45
*** lpetrut has joined #openstack-cinder23:47
*** lpetrut has quit IRC23:52

Generated by 2.15.3 by Marius Gedminas - find it at!