Friday, 2019-08-02

*** rosmaita has quit IRC00:18
*** lixiaoy1 has quit IRC00:23
*** lixiaoy1 has joined #openstack-cinder00:23
*** rosmaita has joined #openstack-cinder00:31
*** ianychoi has joined #openstack-cinder00:34
*** bhagyashris has joined #openstack-cinder00:48
*** lseki has quit IRC00:53
*** jmlowe has joined #openstack-cinder00:58
*** senrique_ has joined #openstack-cinder01:02
*** dviroel has quit IRC01:10
*** jbernard_ is now known as jbernard01:44
*** e0ne has joined #openstack-cinder01:48
*** e0ne has quit IRC01:52
*** sapd1_ has quit IRC01:59
*** Kuirong has joined #openstack-cinder02:06
*** sapd1 has joined #openstack-cinder02:13
*** senrique_ has quit IRC02:16
openstackgerritSofia Enriquez proposed openstack/cinder master: Unable to create cinder incremental backup
*** minmin has quit IRC02:30
openstackgerritTejdeep Kautharam proposed openstack/cinder master: extend_volume method fails if volume size is float
openstackgerrithjy proposed openstack/cinder master: Fix comment feedback in patch sets 19 and 20
*** ruffian_sheep has joined #openstack-cinder02:44
*** n-saito has joined #openstack-cinder02:48
*** sapd1 has quit IRC02:54
*** ruffian_sheep has quit IRC02:55
openstackgerritBoxiang Zhu proposed openstack/cinder master: Fix issue of getting detail backups list info
*** dpawlik has quit IRC03:04
openstackgerritBoxiang Zhu proposed openstack/cinder master: Fix issue of getting detail backups list info
*** psachin has joined #openstack-cinder03:21
*** psachin has quit IRC03:36
*** carloss has quit IRC03:43
*** gkadam has joined #openstack-cinder03:49
*** boxiang has joined #openstack-cinder03:56
*** jojoda_ has joined #openstack-cinder04:09
*** e0ne has joined #openstack-cinder04:30
*** whoami-rajat has joined #openstack-cinder04:34
*** e0ne has quit IRC04:35
*** abhishekk has joined #openstack-cinder04:38
*** tkajinam has quit IRC05:00
*** sapd1 has joined #openstack-cinder05:01
*** tkajinam has joined #openstack-cinder05:02
*** Luzi has joined #openstack-cinder05:05
*** udesale has joined #openstack-cinder05:07
*** yaawang has quit IRC05:19
*** yaawang has joined #openstack-cinder05:28
*** yaawang has quit IRC05:33
*** markvoelker has joined #openstack-cinder05:34
*** markvoelker has quit IRC05:39
*** tkajinam_ has joined #openstack-cinder05:53
*** tkajinam has quit IRC05:55
*** e0ne has joined #openstack-cinder06:04
*** sahid has joined #openstack-cinder06:09
openstackgerritrenminmin proposed openstack/cinder master: Correct new_type type in VolumeManager retype method
*** jojoda_ has quit IRC06:17
*** e0ne has quit IRC06:42
*** e0ne has joined #openstack-cinder06:45
openstackgerritpengyuesheng proposed openstack/os-brick master: Blacklist sphinx 2.1.0 (autodoc bug)
*** rcernin has quit IRC06:58
openstackgerritpengyuesheng proposed openstack/os-brick master: Bump the openstackdocstheme extension to 1.20
*** redrobot has quit IRC07:17
*** tesseract has joined #openstack-cinder07:17
*** e0ne has quit IRC07:31
*** ruffian_sheep has joined #openstack-cinder07:42
whoami-rajatruffian_sheep: Hi07:43
ruffian_sheepwhoami-rajat:Is it correct to submit the patch like this? I tried to imitate other vendors to submit patches to fix bugs.07:44
ruffian_sheepwhoami-rajat:Also, I found this problem, it seems that it is not the file I wrote? Should I delete it? releasenotes/source/unreleased.rst:388:Inline interpreted text or phrase reference start-string without end-string.07:45
*** jcosmao has joined #openstack-cinder07:47
ruffian_sheepwhoami-rajat: After all of this has been processed, I just need to change this state to confirmed, right?07:48
openstackLaunchpad bug 1837920 in Cinder "MacroSAN driver has numerous issues that need to be addressed" [Undecided,New] - Assigned to MS-tester (macrosan-x)07:48
*** tosky has joined #openstack-cinder07:48
whoami-rajatruffian_sheep:  you don't require the release note here since the driver added and the fixes you're making are in the same release.07:51
whoami-rajatruffian_sheep: you can set it as confirmed now but I don't think it will matter, when your patch gets merged, your bug status will change to fix committed automatically07:53
ruffian_sheepwhoami-rajat:I see other vendors seem to have added a release note explaining which bug was fixed.So I should delete the yaml bug-1837920-0f07216ac5ef13a3.yaml to commit again,right?07:55
ruffian_sheepwhoami-rajat:Ok, then I won’t change its status.07:56
whoami-rajatruffian_sheep: release notes are a way to tell operators what all changes were made in that particular release. Since the vendor drivers were merged in previous releases and they made bug fixes in a different release, so they need to mention it. In your case, your driver was added in Train release is what IMO the users should know.07:58
whoami-rajatruffian_sheep: yes you can delete the release note here07:58
whoami-rajatThanks for the consistent work.07:59
openstackgerritNaoki Saito proposed openstack/cinder master: NEC Driver: allow more than 4 iSCSI portals
ruffian_sheepwhoami-rajat:Releasenotes/source/unreleased.rst But it seems that this file is still problematic, I will still report an error when I execute it.08:03
ruffian_sheepreleasenotes/source/unreleased.rst:388:Inline interpreted text or phrase reference start-string without end-string.08:04
jc_Hi, I am trying to add or mount all hard disks using ceph on ubuntu server 18.04 following the (mimic), but I am getting some errors while creating the OSDs. Can anyone help me in identifying the actual problem.08:05
whoami-rajatruffian_sheep: AFAIK the unreleased file is created from all the release notes of current release. There might be some issue with your current release note ,I think the error won't show if you remove it.08:09
*** ruffian_sheep has quit IRC08:17
*** ruffian_sheep has joined #openstack-cinder08:22
*** lemko has joined #openstack-cinder08:24
ruffian_sheepwhoami-rajat:I have seen other vendors make changes to the version of the driver after fixing the bug. For example, from 1.0.0 to 1.0.1. It is accompanied by the bug number that was fixed. Doesn't this require a release note?08:25
whoami-rajatruffian_sheep: So IIUC, the driver shouldn't have merged in the first place due to the review comments. And IMO fixing the review comments shouldn't cause a version bump. I might be wrong and will discuss the same with other members.08:32
ruffian_sheepwhoami-rajat:Yes, I re-downloaded the latest cinder driver. At present, this file is originally on the cinder main line. Not generated by my release notes.08:44
*** ruffian_sheep has quit IRC08:59
*** tkajinam_ has quit IRC09:01
*** e0ne has joined #openstack-cinder09:03
*** lixiaoy1 has quit IRC09:04
*** markvoelker has joined #openstack-cinder09:13
*** Conqueror has joined #openstack-cinder09:14
*** abishop has quit IRC09:15
*** abishop has joined #openstack-cinder09:15
*** ruffian_sheep has joined #openstack-cinder09:31
*** v12aml has joined #openstack-cinder09:32
*** markvoelker has quit IRC09:46
ruffian_sheepwhoami-rajat:So should I add a release note? I saw that they all added ;)09:47
whoami-rajatruffian_sheep: I don't think you should but I will discuss and let you know.09:48
*** v12aml has quit IRC09:52
*** ruffian_sheep has quit IRC09:53
*** ruffian_sheep has joined #openstack-cinder09:53
ruffian_sheepwhoami-rajat:So should I add a release note? I saw that they all added ;)09:54
ruffian_sheepwhoami-rajat:Also, I am curious to ask. What do IMO, AFAIK, and IIUC mean? I do not understand lollll09:54
*** ociuhandu has joined #openstack-cinder09:54
whoami-rajatruffian_sheep: hah, In my Opinion, As far as I know and if I understand correctly.09:55
*** jawad_axd has joined #openstack-cinder09:56
*** ruffian_sheep has quit IRC10:00
*** abhishekk has quit IRC10:12
*** Kuirong has quit IRC10:15
*** ociuhandu has quit IRC10:34
*** ociuhandu has joined #openstack-cinder10:34
*** ociuhandu has quit IRC10:36
*** ociuhandu has joined #openstack-cinder10:37
*** davidsha has joined #openstack-cinder10:49
*** brinzhang has quit IRC10:55
*** boxiang has quit IRC11:00
*** zhubx has joined #openstack-cinder11:00
*** usr2033 has joined #openstack-cinder11:06
usr2033Is there anyone that used "use_db_reconnect" parameter?11:07
*** bhagyashris has quit IRC11:26
*** carloss has joined #openstack-cinder11:38
*** takamatsu is now known as mauro|call11:44
*** dviroel has joined #openstack-cinder11:45
*** markvoelker has joined #openstack-cinder12:00
*** udesale has quit IRC12:02
*** udesale has joined #openstack-cinder12:03
*** markvoelker has quit IRC12:03
*** markvoelker has joined #openstack-cinder12:03
*** markvoelker has quit IRC12:04
*** redrobot has joined #openstack-cinder12:07
*** markvoelker has joined #openstack-cinder12:16
*** jawad_axd has quit IRC12:16
*** Luzi has quit IRC12:21
*** ociuhandu has quit IRC12:24
*** dpawlik has joined #openstack-cinder12:25
*** gkadam has quit IRC12:26
*** n-saito has quit IRC12:44
usr2033can i ping connection before mysql connection?12:45
*** markvoelker has quit IRC12:58
*** markvoelker has joined #openstack-cinder12:58
*** ociuhandu has joined #openstack-cinder13:11
*** eharney has joined #openstack-cinder13:19
*** mriedem has joined #openstack-cinder13:21
*** markvoelker has quit IRC13:25
*** markvoelker has joined #openstack-cinder13:25
*** lseki has joined #openstack-cinder13:36
davidshaHey, has anyone run devstack on Centos 7 recently?13:54
eharneyyes, i've been working in a centos env lately13:57
davidshahave you noticed an issue with mysql5.6 and cinders database migrations?14:02
eharneymine seems to be running mariadb 10.3.10?14:04
eharneybut, i haven't noticed anything w/ db migrations14:05
davidshakk, it's just me having wierd issues then, nvm!14:06
eharneywhat kind of errors are you seeing?14:09
davidshaI have since lost the exact error, but it was like it was trying to put a string too large into one of the tables, and as I was looking it up it seemed to be an issue that comes up when something is written for 5.7 but run on 5.614:14
*** enriquetaso has joined #openstack-cinder14:30
hemna_what's the good word?14:30
*** gkadam has joined #openstack-cinder14:32
*** davee_ has quit IRC14:35
*** gkadam has quit IRC14:38
openstackgerritSofia Enriquez proposed openstack/cinder master: Unable to create cinder incremental backup
*** dpawlik has quit IRC14:38
*** donnyd has joined #openstack-cinder14:42
donnydDoes anyone know if there is a way to expand the number of workers for the volume service?14:43
smcginnisdonnyd: There's backend_native_threads_pool_size, but not sure which operations that actually has an impact on.14:45
donnydThe reason I ask is I have an all nvme block storage server and it seems the volume service can sometimes have a hard time keeping up14:46
donnydIt works most of the time, but when it doesn't it blows up pretty fantastically14:47
smcginnisMaybe osapi_volume_workers for API requests or executor_thread_pool_size in [oslo.messaging]?14:48
donnydthe volume service is the only thing running on that server, so there is no shared load from anything else14:48
smcginnisOr wsgi_default_pool_size14:48
donnydWell its not the apis, i already fixed that one14:48
donnydIts the actual volume service itself14:48
smcginnisWhat is it taking a long time on? Like what operations?14:49
donnydWell I am supposed to get central logging up and running, and when I can stop putting out fires then I would be able to give you more precise info.14:50
donnydThis is the error I get, but only when the load is very high14:51
donnydStderr: 'Traceback (most recent call last):\n  File "/usr/local/bin/cinder-rtstool", line 10, in <module>\n    sys.exit(main())\n  File "/usr/local/lib/python3.7/site-packages/cinder/cmd/", line 288, in main\n    **optional_args)\n  File "/usr/local/lib/python3.7/site-packages/cinder/cmd/", line 56, in create\n    dev=backing_device)\n  File "/usr/lib/python3.7/site-packages/rtslib_fb/", line14:51
donnyd738, in __init__\n    self._configure(dev, wwn, readonly)\n  File "/usr/lib/python3.7/site-packages/rtslib_fb/", line 748, in _configure\n    raise RTSLibError("Device %s is not a TYPE_DISK block device" % dev)\nrtslib_fb.utils.RTSLibError: Device /dev/cinder-volumes/volume-899b9319-2092-4105-b336-82e1171aed6b is not a TYPE_DISK block device\n'14:51
donnydAnd there are some others too14:52
smcginnisHmm, that looks like some kind of timing issue with rtstool or something.14:52
donnydIts like it can't do the things its supposed to fast enough.. not 100% sure its a cinder issue14:54
donnydis there a way to throttle concurency?14:54
donnydIt takes a beating from zuul, so if I could slow things down just a little it would probably work fine too14:56
donnydMaybe limiting nova could also work14:56
*** usr2033 has quit IRC14:56
smcginnisI wonder if that driver actually needs to add come concurrency locking to make sure operations don't overlap and interfere with each other.14:57
smcginnisI don't really have any suggestions, but maybe someone else here might.14:57
smcginnisYou could also file a bug to capture any logs and debugging, but not sure if the driver maintainers are jumping on those or not.14:58
*** sahid has quit IRC15:05
donnydI think throttling nova is going to work for me15:06
donnydzuul will launch 100 instances at a time, which is fine for the big fellas... but my poor little cloud doesn't handle that well15:07
*** ociuhandu has quit IRC15:09
*** ociuhandu has joined #openstack-cinder15:09
*** markvoelker has quit IRC15:18
donnydYea, it did.. I was just beating my volume server too much. Thanks smcginnis15:19
*** markvoelker has joined #openstack-cinder15:20
*** vishalmanchanda has quit IRC15:21
*** eharney has quit IRC15:32
*** zhubx has quit IRC15:38
*** zhubx has joined #openstack-cinder15:38
*** zhubx has quit IRC15:40
*** zhubx has joined #openstack-cinder15:41
*** ociuhandu has quit IRC15:56
*** ociuhandu has joined #openstack-cinder16:00
*** davidsha has quit IRC16:03
openstackgerritLuigi Toscano proposed openstack/cinder-tempest-plugin master: WIP Update cinder-tempest-plugin-lvm-lio (revamp)
openstackgerritLuigi Toscano proposed openstack/cinder master: WIP Port several legacy tempest jobs to Zuul v3
*** udesale has quit IRC16:07
openstackgerritRafael Weingärtner proposed openstack/cinder master: Add mv to return volume type name and ID with volumes
*** pcaruana has quit IRC16:14
*** ociuhandu has quit IRC16:21
*** ociuhandu has joined #openstack-cinder16:22
*** ociuhandu has quit IRC16:26
*** ociuhandu has joined #openstack-cinder16:27
*** altlogbot_0 has quit IRC16:29
*** irclogbot_0 has quit IRC16:33
*** altlogbot_0 has joined #openstack-cinder16:36
*** irclogbot_0 has joined #openstack-cinder16:41
openstackgerritSofia Enriquez proposed openstack/cinder master: Fix ceph driver error during incremental backup creation
rosmaitajungleboyj: smcginnis: pots: see what you think of this:
rosmaitasmcginnis: i can't figure out how to get '[[local|localrc]]' to show up as literals (the wiki turns it into a link)16:52
smcginnisMaybe throw in some extra spaces?16:56
potsrosmaita: maybe this is the answer?
*** ociuhandu has quit IRC16:57
*** ociuhandu has joined #openstack-cinder16:57
smcginnisrosmaita: Thanks for getting that together. Maybe now that it is there, others will be willing to add their notes to help make it easier to run CIs.16:58
*** tesseract has quit IRC16:58
rosmaitathat would be nice16:58
smcginnisWe can hope...16:58
rosmaitai tried the extra spaces, i will give pots's suggestion a try16:58
rosmaitaif all else fails i will ask fungi16:59
fungiof course you will ;)17:00
rosmaitafungi: hello! got it to work by using &#91;&#91;17:01
rosmaitaso, nevermind17:01
fungiyou can wrap it in something like <nowiki></nowiki> too i think17:01
* fungi doesn't remember the exact syntax17:01
rosmaitadidn't think of that one, i tried <code></code> but no dice17:02
fungimediawiki docs should be easy to find for the nowiki tag17:02
fungiit basically just turns off markup for anything inside there17:02
*** ociuhandu has quit IRC17:02
rosmaitacool, i will have to keep that in mind17:02
*** jcosmao has left #openstack-cinder17:03
rosmaitauseful to know that the html entity codes work too in a pinch17:03
rosmaitai am taking the afternoon off, i wish everyone a good weekend17:03
*** rosmaita has left #openstack-cinder17:04
*** dpawlik has joined #openstack-cinder17:19
*** openstackgerrit has quit IRC17:22
*** ociuhandu has joined #openstack-cinder17:26
*** dpawlik has quit IRC17:34
*** eharney has joined #openstack-cinder17:41
potsjungleboyj: smcginnis: do we have an os-brick expert i can run something by before posting a patch?  os-brick support for iSCSI multipath seems to be broken, at least for my configs.17:49
*** ociuhandu has quit IRC17:49
smcginnispots: I think hemna_ is still the export, but IIRC geguileo did some work fixing some of the multipath issues.17:50
*** ociuhandu has joined #openstack-cinder17:50
*** ociuhandu has quit IRC17:50
potshi, i think there's an error in the os-brick where it does discovery of multipath devices.17:51
potsthe symptom is we see this command failing: "iscsiadm -m node -T ('',) -p --op update -n node.startup -v automatic"17:52
hemna_what's the error?17:52
hemna_do you have a stacktrace?  pastebin?17:52
potsit's due to this statement in _recover_node_startup_values at the bottom of initiator/connectors/               recover_connection['target_iqn'] = [iqn]17:54
potsthe extra level of indirection causes the command line syntax error you see there.  iscsiadm just reports "No records found"17:54
jungleboyjpots:  Yes, hemna_  is your friend.  Sorry for the slow response.17:59
hemna_ok just a sec18:00
potsjungleboyj: no sweat, just glad i think i found it before the weekend started.18:01
potsjungleboyj: this is keeping the multiattach tests from passing for the Lenovo driver.18:02
potsjungleboyj: just on iSCSI.18:02
jungleboyjpots: ah, I saw some failures there. Thanks for looking into it.18:03
potshemna_: here's a full set of logs if it helps:
hemna_so it looks like 'discovery_auth_method' isn't set in the connection_properties from the driver18:04
hemna_what version of open-iscsi ?18:07
hemna_iscsiadm is complaining that it can't set the node.session.scan mode to manual18:09
potsYes, I wasn't aware I was supposed to be setting that property.  It's open-iscsi 2.0.873+git0.3b4b4500-14ubuntu3.718:09
hemna_which means that I think it's old18:09
potsit's what comes with ubuntu 16.0418:10
hemna_I think that could be too old18:10
hemna_your /etc/iscsi dir is empty18:10
potsit just failed to upload due to permissions, are you looking for anything in particular?18:11
*** e0ne has quit IRC18:11
hemna_just making sure the env is ok18:11
potsi did try patching that statement and it seemed to fix the problem.18:13
potsbut you see how the iscsiadm command is called with a bogus command line arg, right?18:14
hemna_well for whatever reason iscsiadm doesn't think there is anyting exported from that target portal target combo18:14
potsbecause we're passing in a list of iqns instead of one iqn, so the list gets interpolated into the iscsiadm command line instead of a single iqn, creating a syntax error.18:16
potsthe test case seems to pass if I pass in "iqn" instead of "[iqn]"18:19
hemna_this is all geguileo's new discovery code18:22
*** lemko has quit IRC18:23
potsso would this need to be patched on pypi?18:25
donnydis there a known method to get volumes unstuck from the detaching state?18:32
hemna_donnyd: you need to find out why it didn't complete18:32
hemna_there is a way to force the volume state, but that's only if you know it's safe to do it18:33
donnydi know that its safe18:33
donnydYea, that was the first thing i tried18:33
donnydThen i flipped them back to in-use and back to deatching.18:34
donnydthey don't actaully move from the detaching state18:34
hemna_because something is failing18:34
hemna_you have to look at the logs to find out why18:34
donnydwell volume is doing nothing and the apis and schedulers throw no errors18:35
*** openstackgerrit has joined #openstack-cinder18:35
openstackgerritSofia Enriquez proposed openstack/cinder master: Fix ceph: only close rbd image after snapshot iteration is finished
hemna_nova ?18:35
donnydno tracebacks or errors I can see in nova either18:37
hemna_it's in there18:37
donnydI had a couple hundred volumes blow up, and there are just a few left that won't actaully detach18:37
hemna_pots:what version of iscsiadm ?18:38
hemna_pots:what's odd is that the iscsiadm error isn't a syntax error18:40
potsit accepts the parens on the command line?18:40
hemna_that's what the error should be18:40
hemna_but it's not18:41
hemna_Stderr: 'iscsiadm: No records found\n'18:41
hemna_that's what you are seeing18:41
hemna_I think that's simply a translation18:41
potshemna_: i think you're missing the quotes in your bash example18:42
hemna_I copy and pasted from the traceback output18:42
hemna_I think that's a red herring18:42
donnydso it shows the requests for state change, and no errors18:44
donnydnova api placement and compute logs show nothing18:44
donnydunless you want to see a bunch of haproxy checks and instance usage audits, there is nothing to see18:45
hemna_donnyd:that's just the cinder api18:45
hemna_the failure is in nova compute18:45
potshemna_: i put before and after logs for one test case here:
donnydhemna_: there is nothing in nova compute... there may have been an error thrown when the instance existed, but zuul promptly burned them18:46
donnydwell nodepool18:47
donnydwhen state is flipped from in-use to deatching, it should try to detach.. and it does not18:47
hemna_does cinder terminate_connection get called?18:47
hemna_and does nova-compute call os-brick to detach ?18:48
donnydnova-compute is not doing anything. I am assuming terminate_connection is in cinder-volume, but i don't know18:49
hemna_c-vol gets called to terminate_connection18:49
hemna_then nova issues commands to os-brick to detach18:49
hemna_if terminate_connection isn't returning, then there is a problem with the driver/backend18:50
donnydcinder-volume is just waiting there18:50
donnydservice says its up18:51
hemna_c-api gets the call to detach ?18:51
donnyd2019-08-02 14:42:58.701 38013 DEBUG cinder.api.openstack.wsgi [req-d6f76cb2-acc3-4851-b6de-c97e2e904149 6610181deb904e08ad71af296b769a78 e8fd161dc34c421a979a9e6421f823e9 - default default] Action: 'action', calling method: _reset_status, body: {"os-reset_status": {"status": "detaching"}} _process_stack /usr/lib/python2.7/site-packages/cinder/api/openstack/
hemna_that's a reset status18:52
hemna_not a call to detach18:52
donnydnever happens18:55
*** yaawang has joined #openstack-cinder18:57
donnydThat is not something I can see how to control. I can set the state from in-use to detaching, which should trigger a call to detach like it did on the other volumes... but it doesn't seem to do it and i see no logs18:57
potshemna_:  at the risk of beating a dead horse, here's the iscsiadm command run with the 2.9.1 code, with my patch, and one just to show that there is a matching record.
potshemna: and i think you're right about the open-iscsi being to old, but i'm not sure what to do about that.  Is Ubuntu 16.04 too old?  I tried to move to 18.04 but saw kernel crashes with nested virtualization.19:04
*** CeeMac has quit IRC19:04
*** e0ne has joined #openstack-cinder19:13
*** yaawang has quit IRC19:25
*** tosky has quit IRC19:32
*** dpawlik has joined #openstack-cinder19:37
*** rtjure has quit IRC19:38
*** irclogbot_0 has quit IRC19:39
*** irclogbot_2 has joined #openstack-cinder19:42
*** whoami-rajat has quit IRC20:04
hemna_pots:looking at other CI runs, lots of people are getting those complaings about trying to update the scan mode20:09
hemna_I'm guessing it was never looked at20:09
potsshall i submit a bug with the patch?20:09
hemna_trying to find other iscsi CI's that are running w/ multipath on20:10
potsi can post a patch too if it would make things faster.  do you expect geguileo to be around Monday am?20:10
*** e0ne has quit IRC20:11
potsi upgraded to py37 and turned on multipath and multiattach all at once.  what could go wrong?20:11
potsok, posted a bug:
openstackLaunchpad bug 1838820 in Cinder "Bad args to iscsiadm for iSCSI multipath target" [Undecided,New]20:11
*** dpawlik has quit IRC20:19
*** enriquetaso has quit IRC20:28
*** baojg has quit IRC20:31
*** rtjure has joined #openstack-cinder20:33
*** mriedem has quit IRC20:33
*** markvoelker has quit IRC20:42
donnydI was just trying to setup this (multipath) lvm/iscsi up for my cloud and couldn't get it working20:46
donnydIn my case I was only seeing one client > target connection, even though c-vol was populating the target with both portals20:47
*** markvoelker has joined #openstack-cinder20:52
*** markvoelker has quit IRC20:56
*** _erlon_ has joined #openstack-cinder21:15
*** jdillaman1 has quit IRC21:28
*** spatel has joined #openstack-cinder21:35
smcginnisdonnyd: Are these concurrent volume creation requests all unique images? Or creating multiple volumes from the same set of images?21:38
donnydI have caching turned on, so it's just a clone of the same 3 volumes21:39
smcginnisdonnyd: OK, just wanted to make sure you were aware of
smcginnisLVM is probably a little limited with that. I know other backends can create over 100 volumes at once with that turned on because for some of them it's just an update of some metadata.21:40
donnydYea, that helps quite a lot, but 70 requests at the same time doesn't work on just one lowly c-vol server21:41
smcginnisOh well, throttling it is I guess.21:41
donnydI would just use ceph, but it's too slow21:41
donnydIt seems to work, it's just building jobs for the openstack ci21:42
*** spatel has quit IRC21:42
donnydAnd at my scale (small), I have to make the best of everything21:42
donnydIt also lowers the load on neutron, so two birds one stone21:43
donnydNodepool will beat a cloud to death if not throttled21:43
smcginnisSeems like a good area where someone could spend some time doing some performance analysis to see if there's anyway to optimize things, but if it's between getting things a little slower versus having things fail, I guess it's a good solution.21:44
donnydAfter I flipped on the throttling, all of the issues disappeared21:44
donnydThat is what I have spent my nights and weekends doing for the last three weeks21:44
donnydI needed FN to put out as much as possible with as little as possible21:45
donnydNow you get 50K iops in each instance21:45
smcginnisNice, that's a good improvement from what you were seeing before, right?21:46
donnyd4k to 50k21:47
smcginnisNot bad!21:47
*** markvoelker has joined #openstack-cinder21:48
*** eharney has quit IRC22:20
*** markvoelker has quit IRC22:21
*** sapd1 has quit IRC22:26
*** sapd1 has joined #openstack-cinder22:26
*** sapd1 has quit IRC22:27
*** enriquetaso has joined #openstack-cinder22:42
*** dviroel has quit IRC22:53
*** N3l1x has quit IRC23:17
*** markvoelker has joined #openstack-cinder23:20
*** _erlon_ has quit IRC23:25
*** markvoelker has quit IRC23:53

Generated by 2.15.3 by Marius Gedminas - find it at!