Thursday, 2018-09-06

*** brinzhang has joined #openstack-cinder00:18
*** spartakos has quit IRC00:30
*** rcernin has quit IRC00:50
*** rcernin has joined #openstack-cinder00:51
*** erlon has quit IRC00:57
*** tonyb has quit IRC01:07
*** tonyb has joined #openstack-cinder01:09
*** erlon has joined #openstack-cinder01:11
*** dominix has joined #openstack-cinder01:15
*** erlon has quit IRC01:17
openstackgerritwanghao proposed openstack/cinder master: Support to query volume filter by changes since/before
*** fnordahl has quit IRC01:19
*** fnordahl has joined #openstack-cinder01:20
*** zhaochao has joined #openstack-cinder01:26
*** tonyb has quit IRC01:27
*** tonyb has joined #openstack-cinder01:31
*** lbragstad has quit IRC01:43
*** lbragstad has joined #openstack-cinder01:45
*** sapd1_ has joined #openstack-cinder02:03
*** sapd1 has quit IRC02:07
*** _hemna has quit IRC02:08
*** lbragstad has quit IRC02:09
*** sapd1_ has quit IRC02:09
*** spartakos has joined #openstack-cinder02:09
*** _hemna has joined #openstack-cinder02:10
*** sapd1 has joined #openstack-cinder02:13
*** nicolasbock has quit IRC02:18
dominixhi there02:32
dominixI am wondering if Cinder has some feature able to move block from one kind of disk to another depending if its read often or not02:33
dominixlike multi tier dispatch02:33
dominixmoving block from SSD type to SAS and then SATA if less often accessed02:34
dominixor move these block from SATA to SAS to SSD if more often accessed.02:34
dominixhow will you name this feature ?02:35
*** psachin has joined #openstack-cinder02:45
openstackgerritwang yong proposed openstack/cinder master: Cinder volume driver for Inspur AS13000 series
*** gcb_ has quit IRC03:14
*** gcb_ has joined #openstack-cinder03:17
*** gkadam has joined #openstack-cinder03:31
*** spartakos has quit IRC03:35
*** nwonknu has quit IRC03:48
*** nwonknu has joined #openstack-cinder04:00
*** markvoelker has joined #openstack-cinder04:06
*** brinzhang has quit IRC04:10
*** markvoelker has quit IRC04:11
*** brinzhang has joined #openstack-cinder04:11
openstackgerritBoxiang Zhu proposed openstack/cinder master: Fix wrong filter of backups in db api
*** gcb_ has quit IRC04:17
whoami-rajatsmcginnis:  ok, thought the same, Thanks04:24
openstackgerritRaunak Kumar proposed openstack/cinder master: nimble storage: retype support
*** hoangcx has quit IRC04:40
*** hoangcx has joined #openstack-cinder04:40
*** stakeda has joined #openstack-cinder04:42
*** Bhujay has joined #openstack-cinder04:45
*** gcb_ has joined #openstack-cinder04:53
*** markvoelker has joined #openstack-cinder05:07
*** gkadam_ has joined #openstack-cinder05:41
*** gkadam has quit IRC05:43
*** gkadam__ has joined #openstack-cinder05:45
*** gkadam_ has quit IRC05:47
*** gcb_ has quit IRC05:53
*** Bhujay has quit IRC06:07
*** gcb_ has joined #openstack-cinder06:17
*** Bhujay has joined #openstack-cinder06:18
*** Luzi has joined #openstack-cinder06:18
*** pcaruana has joined #openstack-cinder06:25
*** dominix has quit IRC06:35
*** markvoelker has quit IRC06:45
*** alexchadin has joined #openstack-cinder06:54
*** lpetrut has joined #openstack-cinder06:57
*** e0ne has joined #openstack-cinder06:58
*** rcernin has quit IRC07:00
*** Emine has joined #openstack-cinder07:06
*** alexchadin has quit IRC07:10
*** alexchadin has joined #openstack-cinder07:27
*** hoonetorg has quit IRC07:34
*** pcaruana has quit IRC07:36
*** markvoelker has joined #openstack-cinder07:43
*** hoonetorg has joined #openstack-cinder07:47
*** pcaruana has joined #openstack-cinder07:48
openstackgerritMerged openstack/cinder master: fix misspelling of 'configuration'
*** gcb_ has quit IRC08:10
*** david-lyle has quit IRC08:14
*** dklyle has joined #openstack-cinder08:15
*** kaisers has quit IRC08:19
*** hoangcx has quit IRC08:19
*** hoangcx has joined #openstack-cinder08:19
*** kaisers has joined #openstack-cinder08:25
*** alexchadin has quit IRC08:58
*** alexchadin has joined #openstack-cinder09:14
*** erlon has joined #openstack-cinder09:16
*** alexchadin has quit IRC09:24
*** erlon has quit IRC09:32
whoami-rajatHi cores, the cinder api-version command is failing, please review the patch addressing it
*** erlon has joined #openstack-cinder10:10
*** luizbag has joined #openstack-cinder10:15
*** e0ne has quit IRC10:15
*** erlon has quit IRC10:20
*** erlon has joined #openstack-cinder10:26
*** nicolasbock has joined #openstack-cinder10:31
*** stakeda has quit IRC10:31
*** e0ne has joined #openstack-cinder10:39
*** dave-mccowan has joined #openstack-cinder10:46
*** ganso has joined #openstack-cinder10:49
erlongeguileo, hey, can you re+2 the CG patch?
erlonour internal submit system messed up the fix10:56
erlonshould be fine now10:56
geguileoerlon: lol10:58
erlongeguileo, thanks!11:00
*** dcdamien has quit IRC11:00
erlonneed to get this merge and backported asap, it blocking our RH Certification which will dealine in late this month o.O11:01
* erlon freaks!11:01
*** dcdamien has joined #openstack-cinder11:03
*** brinzhang has quit IRC11:07
*** brinzhang has joined #openstack-cinder11:08
*** erlon has quit IRC11:15
*** abishop has joined #openstack-cinder11:21
*** abishop has quit IRC11:21
*** Emine has quit IRC11:23
*** Emine has joined #openstack-cinder11:25
*** erlon has joined #openstack-cinder11:25
*** eharney has quit IRC11:39
*** psachin has quit IRC11:51
*** yumiriam has quit IRC11:53
*** abishop has joined #openstack-cinder11:54
*** alexchadin has joined #openstack-cinder12:11
*** markvoelker has quit IRC12:22
*** eharney has joined #openstack-cinder12:38
*** alexchadin has quit IRC12:41
*** e0ne has quit IRC12:41
*** alexchadin has joined #openstack-cinder12:49
*** lpetrut has quit IRC12:54
*** lbragstad has joined #openstack-cinder12:54
*** yumiriam has joined #openstack-cinder12:55
*** e0ne has joined #openstack-cinder12:59
*** mriedem has joined #openstack-cinder13:08
*** lbragstad has quit IRC13:09
openstackgerritSean McGinnis proposed openstack/cinder master: Tintri: Inherit tests from BaseDriverTestCase
*** lbragstad has joined #openstack-cinder13:15
*** brinzhang has quit IRC13:15
*** mchlumsky has joined #openstack-cinder13:32
whoami-rajatHi e0ne , jungleboyj , the patch has passed Nible CI tests and the logs are available now, , Please review for W+113:42
*** eharney has quit IRC13:43
e0newhoami-rajat: +2'ed13:47
*** finixlei has joined #openstack-cinder13:56
*** eharney has joined #openstack-cinder13:58
erloneharney, smcginnis, someone else in the channel was mentioning a failure on cinder_tempest consistency group tests13:59
erlonthis fixes it :
openstackgerritChuck Short proposed openstack/cinder master: Simplify running pylint
erlonbut the lvm-lio is failing14:00
eharneyerlon: great, can we tie this to ?14:00
openstackLaunchpad bug 1789944 in Cinder "cinder-tempest-plugin CG tests are broken" [Undecided,New]14:00
erloneharney, yep good idea14:02
*** finixlei has quit IRC14:06
*** finixlei has joined #openstack-cinder14:06
erloneharney, any idea why zuul is failing? who is the maintainer?14:07
eharneyerlon: lio-barbican job is failing regularly due to a bug in Nova that's being fixed currently14:07
eharneyerlon: i'm the maintainer of that job14:07
erloneharney, hmm, would adding a Depends-on to the patch make it pass?14:09
*** finixlei has quit IRC14:18
*** finixlei has joined #openstack-cinder14:18
eharneyerlon: the bug is -- looks like it's landed in Nova master now14:24
openstackLaunchpad bug 1789484 in OpenStack Compute (nova) rocky "n-cpu fails init on timeout calling n-cond-cell1" [Medium,In progress] - Assigned to Matt Riedemann (mriedem)14:24
*** finixlei has quit IRC14:24
erloneharney, it landed on Aug 30, so, may be theres' something else causing the failure14:27
mriedemwhich branch is failing?14:27
mriedemmaster is fixed but rocky is not yet14:27
*** finixlei has joined #openstack-cinder14:28
eharneyit's failing due to a different long standing issue with backup in that job14:29
eharneynot that bug above14:29
*** Bhujay has quit IRC14:34
*** Luzi has quit IRC14:47
*** itlinux has quit IRC14:56
*** r-daneel has quit IRC15:17
*** spartakos has joined #openstack-cinder15:25
*** alexchadin has quit IRC15:38
*** e0ne has quit IRC15:42
*** pcaruana has quit IRC15:42
*** r-daneel has joined #openstack-cinder15:44
*** r-daneel has quit IRC15:45
*** finixlei has quit IRC15:50
*** itlinux has joined #openstack-cinder15:54
*** pcaruana has joined #openstack-cinder16:01
*** itlinux_ has joined #openstack-cinder16:08
*** itlinux has quit IRC16:12
jungleboyjsmcginnis: eharney geguileo erlon ganso  I have created a Doodle Poll for the Placement Discussion:16:18
jungleboyjSending e-mail to the ML now.16:18
*** mriedem is now known as mriedem_away16:19
* smcginnis needs to figure out his schedule16:19
jungleboyjsmcginnis:  Yeah, I am starting to see that as well.16:21
smcginnisRight now it's just a big nebulous bunch of conflicting things.16:22
*** zhaochao has quit IRC16:22
*** avishay has joined #openstack-cinder16:23
jungleboyjsmcginnis: Stop being so popular.16:23
avishayHi all. I'm sometimes seeing that create_volume requests take several minutes to reach cinder-volume. I don't know where it's getting stuck. Running Ocata. Has anyone seen anything like this before?16:25
jungleboyjOh, much needed laugh.  Thank you.16:25
jungleboyjavishay: !?!  There is a name from the past!16:25
jungleboyjavishay:  I have not seen or heard of anything like that happening?16:25
avishayYes, it is I :)16:26
*** avishay has quit IRC16:31
*** avishay has joined #openstack-cinder16:32
avishaySorry, got disconnected16:32
avishayAny tips on how to find where/why the message is getting stuck would also be most appreciated.16:32
*** itlinux_ has quit IRC16:33
jungleboyjAre you seeing anything weird from RabbitMQ?16:34
avishaynothing in the logs16:35
jungleboyjHmmm.  Can you tell if the delay is in the API or scheduler?16:37
* jungleboyj just dug a camera and the Yeti out of the moving boxes and put them by my computer bag so I don't forget them this time.16:37
jungleboyjsmcginnis:  Do you have a webcam with a tripod mount?16:38
avishayunfortunately there are no logs in scheduler in the create volume flow, so I don't know when it gets there16:38
smcginnisjungleboyj: Nope16:38
avishayBTW is the lack of logging there on purpose?16:38
jungleboyjsmcginnis:  Bummer.  Ok.  I will have to MacGyver something together.16:38
jungleboyjavishay:  No, there should be logging in the scheduler.  Could you have something weird happening with multiple schedulers running?16:39
avishayNo, check the code... No logs16:40
avishayBe back in about an hour, need to drive16:41
whoami-rajatHi jungleboyj , smcginnis , tommylikehu , can you please review these patches.16:52
openstackgerritSean McGinnis proposed openstack/cinder master: Ensure VNX unit tests don't sleep
*** itlinux has joined #openstack-cinder17:03
*** imacdonn has quit IRC17:05
*** imacdonn has joined #openstack-cinder17:05
*** penick has joined #openstack-cinder17:06
*** gkadam__ has quit IRC17:09
*** spartakos has quit IRC17:11
*** r-daneel has joined #openstack-cinder17:11
*** ChanServ sets mode: -rf 17:18
*** swamireddy has joined #openstack-cinder17:18
*** netapp has joined #openstack-cinder17:18
*** psachin has joined #openstack-cinder17:18
*** raunak12 has joined #openstack-cinder17:24
*** psachin has quit IRC17:26
*** pcaruana has quit IRC17:27
*** e0ne has joined #openstack-cinder17:34
*** spartakos has joined #openstack-cinder17:35
*** links has joined #openstack-cinder17:38
*** links has quit IRC17:38
*** xyang has joined #openstack-cinder17:39
*** mvenesio has joined #openstack-cinder17:40
*** mriedem_away is now known as mriedem17:41
openstackgerritRajat Dhasmana proposed openstack/cinder master: Fujitsu DX: Add retries for busy error
*** eharney has quit IRC17:51
*** xyang is now known as xyang_17:56
*** xyang_ is now known as xyang17:59
*** e0ne has quit IRC18:01
*** pcaruana has joined #openstack-cinder18:02
*** links has joined #openstack-cinder18:02
openstackgerritSean McGinnis proposed openstack/cinder master: Ds8k: Mock evenlet.sleep calls
*** openstackgerrit has quit IRC18:05
*** avishay_ has joined #openstack-cinder18:06
*** e0ne has joined #openstack-cinder18:06
*** avishay has quit IRC18:06
*** avishay_ is now known as avishay18:06
*** e0ne has quit IRC18:08
*** mvenesio has quit IRC18:10
*** mvenesio has joined #openstack-cinder18:11
*** links has quit IRC18:11
*** spartakos has quit IRC18:15
*** melwitt is now known as melwitt_awaysick18:18
*** erlon has quit IRC18:22
imacdonncore team, please review - IMO, this should be backported to Rocky (at least), since disabling the v2 API is recommended18:31
smcginnisThat's twice now today that I've pulled up a patch and thought it looked really familiar. :)18:33
imacdonnyou're like "what fool coded this? Oh ..."18:35
smcginnisHaha, pretty much. :D18:35
jungleboyjOnce again, smcginnis  is the one man Cinder show.  ;-)18:44
imacdonn:) tnx jungleboyj18:44
smcginnis"scatterbrained" might be a better term.18:44
jungleboyjsmcginnis: Welcome to the club.18:47
raunak12core team, please review ?19:00
avishayHey everyone.  I have a system that's acting up.  Running Ocata.  Not sure why, but I have 500 volumes stuck in "deleting" state.  Force delete doesn't help.  How do I get rid of them?19:05
*** e0ne has joined #openstack-cinder19:05
*** eharney has joined #openstack-cinder19:06
*** openstackgerrit has joined #openstack-cinder19:06
openstackgerritMiriam Yumi proposed openstack/cinder-tempest-plugin master: Fix consistency groups test credentials
avishayDid it in SQL...19:08
avishayIn SQL I see many many workers with status="deleting"?19:19
crazikavishay: what is your backend?19:20
avishaycrazik, RBD19:20
jungleboyjavishay:  You can use reset state to change the state to something other than deleting and then try to delete them again.19:20
crazikavishay: any errors in logs?19:21
jungleboyjWonder if there is something weird going on with your DB causing the performance issues.19:21
crazikavishay: check cinder logs as well as ceph19:21
*** jdillaman has quit IRC19:22
openstackgerritKeith Berger proposed openstack/cinder stable/pike: VMware: Optimize volume creation from image
openstackgerritSean McGinnis proposed openstack/cinder master: Remove monkey patch examples from UTs
avishayholy crap... 20,000 messages in cinder-volume's queue...19:24
craziknot good.19:25
*** luizbag has quit IRC19:25
craziklooks like cinder-volume service has problems19:25
avishayyes... big time19:26
jungleboyjOooh ... I saw this before.19:26
jungleboyjHas one of your OSDs filled up?19:26
jungleboyjIf so, then RBD stops processing all requests and it looks like Cinder has lost its mind.19:27
jungleboyjReally, it is just that RBD has stopped and the only clue is a little message put into dmesg somewhere.19:27
jungleboyjavishay:  ^^^19:27
avishayRBD started rebalancing, don't know if it's related19:28
crazikceph -s ?19:29
crazikor ceph health status19:29
jungleboyjavishay:  I need to run for a while but my best guess is that it is due to Ceph no longer processing requests.  I had to go and delete a volume by hand or something to free up space and then it started going again.19:30
avishayjungleboyj, thanks!19:30
*** spartakos has joined #openstack-cinder19:30
crazikavishay: what ceph version?19:31
jungleboyjThere will be a message in one of the system logs, dmesg or /var/log/messages that indicates that a disk is full.19:31
jungleboyjHope that helps.  Let me know what you find out.19:31
avishaycrazik, luminous19:32
crazikdo you have ceph-mgr service?19:32
avishayjungleboyj, thanks, will do!19:32
crazikthen you have a simple web status for ceph cluster19:32
craziksorry, I mispelled last command, it should be: ceph health detail19:33
crazikbut ok19:33
crazik    usage:   1942 GB used, 14825 GB / 16767 GB avail19:34
avishaylooks like it's not in great shape19:34
crazikI don't have access to my clusters now19:35
*** pcaruana has quit IRC19:35
crazikcheck ceph pg stat19:35
avishay512 pgs: 4 active+undersized+degraded+remapped+backfilling, 477 active+clean, 31 active+undersized+degraded+remapped+backfill_wait; 960 GB data, 1946 GB used, 14820 GB / 16767 GB avail; 1878 kB/s rd, 857 kB/s wr, 901 op/s; 16088/770199 objects degraded (2.089%); 1549/770199 objects misplaced (0.201%); 500 kB/s, 0 objects/s recovering19:36
crazikI have never seend 'undersized'19:36
crazikbut it may be something like 'osd full'19:36
*** Emine has quit IRC19:37
crazikwell. no19:37
*** Emine has joined #openstack-cinder19:38
KeithMnemonicsmcginnis: jungleboyj I am back! missed you guys too much!19:38
smcginnisKeithMnemonic: Welcome back!19:38
KeithMnemonici have 10 vmware patches to land, 1 by 1, like Noahs Ark! ( i guess that was 2 by 2)19:39
KeithMnemonicor is there a express entrance19:39
smcginnisI saw the one and confirmed it was in later branches. Just waiting on zuul results now.19:39
KeithMnemonicthese are all in rocky19:39
KeithMnemonicjust need them in pike19:39
smcginniss/rocky/queens/ ?19:39
KeithMnemonicthey are in the rocky branch already, so i dont have to do queens or rocky19:40
KeithMnemonicjust put them in pike19:40
smcginnisThey need to be in queens to go to pike.19:40
KeithMnemonicthry are in queens19:40
KeithMnemonicand rocky19:40
smcginnisp, q, r19:40
smcginnisOK, good.19:40
KeithMnemonicjust not pike19:40
KeithMnemonicso much easier this time19:40
smcginnisAre they dependent on each other? You could just submit them all if they are not.19:41
KeithMnemonicsorry that is what i meant, just need to take them to one branch19:41
KeithMnemonici can check . if they are no dependencies, i can cherry-pick, review, cherty-pick, review,....19:42
KeithMnemonicor do i need to do something diff on my local branch19:42
smcginnisYou might want to create fresh branches locally for each cherry-pick so you don't make them dependent on each other, if there is no need to have them go in in order. Otherwise, yep, that's about it.19:43
KeithMnemonicok let me go get some through.the local branches are all from stable/pike not my previous working branch?19:44
smcginnisYep, that's right.19:44
smcginnisOnce it's submitted you could probably just do cherry-pick, git review, git reset --hard HEAD~1, cherry-pick, git review, etc..19:45
smcginnisOtherwise the commits get stacked on each other and gerrit will try to keep that ordering.19:45
KeithMnemonicok so cherry picks first on my working branch? wont that stack the first submit and the second?19:46
KeithMnemonicwhat i mean is did in need the reset now, or after my second review?19:47
smcginnisKeithMnemonic: You need to either check out stable/pike first between each, or reset to get rid of the new cherry-pick before doing the next one.19:47
KeithMnemonicrgiht so reset first from my curent state, thanks19:48
smcginnisYeah, since you now have your first backport on your local branch, reset before doing the next one unless you want the next one to be dependent on the first one.19:48
openstackgerritKeith Berger proposed openstack/cinder stable/pike: VMware: Add support for cloning attached volumes
smcginnisKeithMnemonic: Hmm, hate to tell you this, but that one adds new functionality to the driver, so that doesn't meet stable policy to allow backporting.19:50
KeithMnemonicno problem, let me see if i can get any of the other "fixes" in19:51
*** bswartz has quit IRC19:51
KeithMnemonicsmcginnis: if I look here  do "optimize" and improve" typically meet the guidelines?19:55
smcginnisKeithMnemonic: In general, yes.19:57
KeithMnemonicso if i try and cherry picks this  and it is not clean does that mean it is missing a depends on?19:58
smcginnisKeithMnemonic: Potentially. You would have to see what the conflicts are and if they are just something shifted around.19:59
*** johanssone has joined #openstack-cinder19:59
KeithMnemonicyes i did that at first and that i why i figured it would be easiest to just put them all in order20:00
KeithMnemonicisd this feature or fix ? maybe that is a new game show20:00
smcginnisLooks like that's just capturing some info to the backend, so that one is fine.20:01
*** yumiriam has quit IRC20:02
*** e0ne has quit IRC20:02
openstackgerritKeith Berger proposed openstack/cinder stable/pike: VMware: Add 'managedBy' info
*** avishay has quit IRC20:12
*** avishay has joined #openstack-cinder20:12
*** avishay has quit IRC20:14
*** amrith has joined #openstack-cinder20:17
KeithMnemonicsmcginnis: this one is a "no go" ?
*** nicolasbock has quit IRC20:34
*** savvas_ has joined #openstack-cinder20:48
*** savvas_ has quit IRC20:48
*** savvas has joined #openstack-cinder20:49
*** nicolasbock has joined #openstack-cinder20:54
*** dims has quit IRC20:55
*** dims has joined #openstack-cinder21:11
*** crazik has quit IRC21:11
*** crazik has joined #openstack-cinder21:18
*** crazik has joined #openstack-cinder21:18
*** mvenesio has quit IRC21:18
*** mvenesio has joined #openstack-cinder21:19
*** mvenesio has quit IRC21:25
*** dave-mccowan has quit IRC22:00
*** mriedem is now known as mriedem_away22:02
*** mriedem_away has quit IRC22:02
smcginnisKeithMnemonic: Kind of a grey area, but I think that could be argued as a bug fix since you can't snap an attached volume otherwise.22:06
KeithMnemonicok thanks!22:09
*** ganso has quit IRC22:17
*** dave-mccowan has joined #openstack-cinder22:23
*** rcernin has joined #openstack-cinder22:29
*** itlinux has quit IRC22:34
*** dave-mccowan has quit IRC23:01
*** tpsilva has quit IRC23:02
*** _alastor_ has quit IRC23:03
*** raunak12 has quit IRC23:08
*** _alastor_ has joined #openstack-cinder23:38
*** _alastor_ has quit IRC23:43
*** itlinux has joined #openstack-cinder23:45

Generated by 2.15.3 by Marius Gedminas - find it at!