| *** rosmaita has quit IRC | 00:18 | |
| *** lixiaoy1 has quit IRC | 00:23 | |
| *** lixiaoy1 has joined #openstack-cinder | 00:23 | |
| *** rosmaita has joined #openstack-cinder | 00:31 | |
| *** ianychoi has joined #openstack-cinder | 00:34 | |
| *** bhagyashris has joined #openstack-cinder | 00:48 | |
| *** lseki has quit IRC | 00:53 | |
| *** jmlowe has joined #openstack-cinder | 00:58 | |
| *** senrique_ has joined #openstack-cinder | 01:02 | |
| *** dviroel has quit IRC | 01:10 | |
| *** jbernard_ is now known as jbernard | 01:44 | |
| *** e0ne has joined #openstack-cinder | 01:48 | |
| *** e0ne has quit IRC | 01:52 | |
| *** sapd1_ has quit IRC | 01:59 | |
| *** Kuirong has joined #openstack-cinder | 02:06 | |
| *** sapd1 has joined #openstack-cinder | 02:13 | |
| *** senrique_ has quit IRC | 02:16 | |
| openstackgerrit | Sofia Enriquez proposed openstack/cinder master: Unable to create cinder incremental backup https://review.opendev.org/674152 | 02:17 |
|---|---|---|
| *** minmin has quit IRC | 02:30 | |
| openstackgerrit | Tejdeep Kautharam proposed openstack/cinder master: extend_volume method fails if volume size is float https://review.opendev.org/674154 | 02:35 |
| openstackgerrit | hjy proposed openstack/cinder master: Fix comment feedback in patch sets 19 and 20 https://review.opendev.org/674156 | 02:42 |
| *** ruffian_sheep has joined #openstack-cinder | 02:44 | |
| *** n-saito has joined #openstack-cinder | 02:48 | |
| *** sapd1 has quit IRC | 02:54 | |
| *** ruffian_sheep has quit IRC | 02:55 | |
| openstackgerrit | Boxiang Zhu proposed openstack/cinder master: Fix issue of getting detail backups list info https://review.opendev.org/672887 | 03:01 |
| *** dpawlik has quit IRC | 03:04 | |
| openstackgerrit | Boxiang Zhu proposed openstack/cinder master: Fix issue of getting detail backups list info https://review.opendev.org/672887 | 03:19 |
| *** psachin has joined #openstack-cinder | 03:21 | |
| *** psachin has quit IRC | 03:36 | |
| *** carloss has quit IRC | 03:43 | |
| *** gkadam has joined #openstack-cinder | 03:49 | |
| *** boxiang has joined #openstack-cinder | 03:56 | |
| *** jojoda_ has joined #openstack-cinder | 04:09 | |
| *** e0ne has joined #openstack-cinder | 04:30 | |
| *** whoami-rajat has joined #openstack-cinder | 04:34 | |
| *** e0ne has quit IRC | 04:35 | |
| *** abhishekk has joined #openstack-cinder | 04:38 | |
| *** tkajinam has quit IRC | 05:00 | |
| *** sapd1 has joined #openstack-cinder | 05:01 | |
| *** tkajinam has joined #openstack-cinder | 05:02 | |
| *** Luzi has joined #openstack-cinder | 05:05 | |
| *** udesale has joined #openstack-cinder | 05:07 | |
| *** yaawang has quit IRC | 05:19 | |
| *** yaawang has joined #openstack-cinder | 05:28 | |
| *** yaawang has quit IRC | 05:33 | |
| *** markvoelker has joined #openstack-cinder | 05:34 | |
| *** markvoelker has quit IRC | 05:39 | |
| *** tkajinam_ has joined #openstack-cinder | 05:53 | |
| *** tkajinam has quit IRC | 05:55 | |
| *** e0ne has joined #openstack-cinder | 06:04 | |
| *** sahid has joined #openstack-cinder | 06:09 | |
| openstackgerrit | renminmin proposed openstack/cinder master: Correct new_type type in VolumeManager retype method https://review.opendev.org/657985 | 06:11 |
| *** jojoda_ has quit IRC | 06:17 | |
| *** e0ne has quit IRC | 06:42 | |
| *** e0ne has joined #openstack-cinder | 06:45 | |
| openstackgerrit | pengyuesheng proposed openstack/os-brick master: Blacklist sphinx 2.1.0 (autodoc bug) https://review.opendev.org/674205 | 06:57 |
| *** rcernin has quit IRC | 06:58 | |
| openstackgerrit | pengyuesheng proposed openstack/os-brick master: Bump the openstackdocstheme extension to 1.20 https://review.opendev.org/674206 | 06:59 |
| *** redrobot has quit IRC | 07:17 | |
| *** tesseract has joined #openstack-cinder | 07:17 | |
| *** e0ne has quit IRC | 07:31 | |
| *** ruffian_sheep has joined #openstack-cinder | 07:42 | |
| ruffian_sheep | whoami-rajat:Hi | 07:42 |
| whoami-rajat | ruffian_sheep: Hi | 07:43 |
| ruffian_sheep | whoami-rajat:Is it correct to submit the patch like this? I tried to imitate other vendors to submit patches to fix bugs. | 07:44 |
| ruffian_sheep | https://review.opendev.org/#/c/674156/ | 07:44 |
| ruffian_sheep | whoami-rajat:Also, I found this problem, it seems that it is not the file I wrote? Should I delete it? releasenotes/source/unreleased.rst:388:Inline interpreted text or phrase reference start-string without end-string. | 07:45 |
| *** jcosmao has joined #openstack-cinder | 07:47 | |
| ruffian_sheep | whoami-rajat:https://bugs.launchpad.net/cinder/+bug/1837920 After all of this has been processed, I just need to change this state to confirmed, right? | 07:48 |
| openstack | Launchpad bug 1837920 in Cinder "MacroSAN driver has numerous issues that need to be addressed" [Undecided,New] - Assigned to MS-tester (macrosan-x) | 07:48 |
| *** tosky has joined #openstack-cinder | 07:48 | |
| whoami-rajat | ruffian_sheep: you don't require the release note here since the driver added and the fixes you're making are in the same release. | 07:51 |
| whoami-rajat | ruffian_sheep: you can set it as confirmed now but I don't think it will matter, when your patch gets merged, your bug status will change to fix committed automatically | 07:53 |
| ruffian_sheep | whoami-rajat:I see other vendors seem to have added a release note explaining which bug was fixed.So I should delete the yaml bug-1837920-0f07216ac5ef13a3.yaml to commit again,right? | 07:55 |
| ruffian_sheep | whoami-rajat:Ok, then I won’t change its status. | 07:56 |
| whoami-rajat | ruffian_sheep: release notes are a way to tell operators what all changes were made in that particular release. Since the vendor drivers were merged in previous releases and they made bug fixes in a different release, so they need to mention it. In your case, your driver was added in Train release is what IMO the users should know. | 07:58 |
| whoami-rajat | ruffian_sheep: yes you can delete the release note here | 07:58 |
| ruffian_sheep | ok | 07:59 |
| whoami-rajat | Thanks for the consistent work. | 07:59 |
| openstackgerrit | Naoki Saito proposed openstack/cinder master: NEC Driver: allow more than 4 iSCSI portals https://review.opendev.org/668088 | 07:59 |
| ruffian_sheep | whoami-rajat:Releasenotes/source/unreleased.rst But it seems that this file is still problematic, I will still report an error when I execute it. | 08:03 |
| ruffian_sheep | releasenotes/source/unreleased.rst:388:Inline interpreted text or phrase reference start-string without end-string. | 08:04 |
| jc_ | Hi, I am trying to add or mount all hard disks using ceph on ubuntu server 18.04 following the https://docs.ceph.com/docs/master/start/quick-ceph-deploy/ (mimic), but I am getting some errors while creating the OSDs. Can anyone help me in identifying the actual problem. | 08:05 |
| whoami-rajat | ruffian_sheep: AFAIK the unreleased file is created from all the release notes of current release. There might be some issue with your current release note ,I think the error won't show if you remove it. | 08:09 |
| *** ruffian_sheep has quit IRC | 08:17 | |
| *** ruffian_sheep has joined #openstack-cinder | 08:22 | |
| *** lemko has joined #openstack-cinder | 08:24 | |
| ruffian_sheep | whoami-rajat:I have seen other vendors make changes to the version of the driver after fixing the bug. For example, from 1.0.0 to 1.0.1. It is accompanied by the bug number that was fixed. Doesn't this require a release note? | 08:25 |
| whoami-rajat | ruffian_sheep: So IIUC, the driver shouldn't have merged in the first place due to the review comments. And IMO fixing the review comments shouldn't cause a version bump. I might be wrong and will discuss the same with other members. | 08:32 |
| ruffian_sheep | whoami-rajat:Yes, I re-downloaded the latest cinder driver. At present, this file is originally on the cinder main line. Not generated by my release notes. | 08:44 |
| *** ruffian_sheep has quit IRC | 08:59 | |
| *** tkajinam_ has quit IRC | 09:01 | |
| *** e0ne has joined #openstack-cinder | 09:03 | |
| *** lixiaoy1 has quit IRC | 09:04 | |
| *** markvoelker has joined #openstack-cinder | 09:13 | |
| *** Conqueror has joined #openstack-cinder | 09:14 | |
| *** abishop has quit IRC | 09:15 | |
| *** abishop has joined #openstack-cinder | 09:15 | |
| *** ruffian_sheep has joined #openstack-cinder | 09:31 | |
| *** v12aml has joined #openstack-cinder | 09:32 | |
| *** markvoelker has quit IRC | 09:46 | |
| ruffian_sheep | whoami-rajat:So should I add a release note? I saw that they all added ;) | 09:47 |
| whoami-rajat | ruffian_sheep: I don't think you should but I will discuss and let you know. | 09:48 |
| *** v12aml has quit IRC | 09:52 | |
| *** ruffian_sheep has quit IRC | 09:53 | |
| *** ruffian_sheep has joined #openstack-cinder | 09:53 | |
| ruffian_sheep | whoami-rajat:So should I add a release note? I saw that they all added ;) | 09:54 |
| ruffian_sheep | whoami-rajat:Also, I am curious to ask. What do IMO, AFAIK, and IIUC mean? I do not understand lollll | 09:54 |
| *** ociuhandu has joined #openstack-cinder | 09:54 | |
| whoami-rajat | ruffian_sheep: hah, In my Opinion, As far as I know and if I understand correctly. | 09:55 |
| *** jawad_axd has joined #openstack-cinder | 09:56 | |
| *** ruffian_sheep has quit IRC | 10:00 | |
| *** abhishekk has quit IRC | 10:12 | |
| *** Kuirong has quit IRC | 10:15 | |
| *** ociuhandu has quit IRC | 10:34 | |
| *** ociuhandu has joined #openstack-cinder | 10:34 | |
| *** ociuhandu has quit IRC | 10:36 | |
| *** ociuhandu has joined #openstack-cinder | 10:37 | |
| *** davidsha has joined #openstack-cinder | 10:49 | |
| *** brinzhang has quit IRC | 10:55 | |
| *** boxiang has quit IRC | 11:00 | |
| *** zhubx has joined #openstack-cinder | 11:00 | |
| *** usr2033 has joined #openstack-cinder | 11:06 | |
| usr2033 | Is there anyone that used "use_db_reconnect" parameter? | 11:07 |
| *** bhagyashris has quit IRC | 11:26 | |
| *** carloss has joined #openstack-cinder | 11:38 | |
| *** takamatsu is now known as mauro|call | 11:44 | |
| *** dviroel has joined #openstack-cinder | 11:45 | |
| *** markvoelker has joined #openstack-cinder | 12:00 | |
| *** udesale has quit IRC | 12:02 | |
| *** udesale has joined #openstack-cinder | 12:03 | |
| *** markvoelker has quit IRC | 12:03 | |
| *** markvoelker has joined #openstack-cinder | 12:03 | |
| *** markvoelker has quit IRC | 12:04 | |
| *** redrobot has joined #openstack-cinder | 12:07 | |
| *** markvoelker has joined #openstack-cinder | 12:16 | |
| *** jawad_axd has quit IRC | 12:16 | |
| *** Luzi has quit IRC | 12:21 | |
| *** ociuhandu has quit IRC | 12:24 | |
| *** dpawlik has joined #openstack-cinder | 12:25 | |
| *** gkadam has quit IRC | 12:26 | |
| *** n-saito has quit IRC | 12:44 | |
| usr2033 | can i ping connection before mysql connection? | 12:45 |
| *** markvoelker has quit IRC | 12:58 | |
| *** markvoelker has joined #openstack-cinder | 12:58 | |
| *** ociuhandu has joined #openstack-cinder | 13:11 | |
| *** eharney has joined #openstack-cinder | 13:19 | |
| *** mriedem has joined #openstack-cinder | 13:21 | |
| *** markvoelker has quit IRC | 13:25 | |
| *** markvoelker has joined #openstack-cinder | 13:25 | |
| *** lseki has joined #openstack-cinder | 13:36 | |
| davidsha | Hey, has anyone run devstack on Centos 7 recently? | 13:54 |
| eharney | yes, i've been working in a centos env lately | 13:57 |
| davidsha | have you noticed an issue with mysql5.6 and cinders database migrations? | 14:02 |
| eharney | mine seems to be running mariadb 10.3.10? | 14:04 |
| eharney | but, i haven't noticed anything w/ db migrations | 14:05 |
| davidsha | kk, it's just me having wierd issues then, nvm! | 14:06 |
| eharney | what kind of errors are you seeing? | 14:09 |
| davidsha | I have since lost the exact error, but it was like it was trying to put a string too large into one of the tables, and as I was looking it up it seemed to be an issue that comes up when something is written for 5.7 but run on 5.6 | 14:14 |
| hemna_ | mornin | 14:25 |
| davidsha | morning | 14:27 |
| jungleboyj | Morning. | 14:29 |
| *** enriquetaso has joined #openstack-cinder | 14:30 | |
| hemna_ | what's the good word? | 14:30 |
| *** gkadam has joined #openstack-cinder | 14:32 | |
| *** davee_ has quit IRC | 14:35 | |
| *** gkadam has quit IRC | 14:38 | |
| openstackgerrit | Sofia Enriquez proposed openstack/cinder master: Unable to create cinder incremental backup https://review.opendev.org/674152 | 14:38 |
| *** dpawlik has quit IRC | 14:38 | |
| *** donnyd has joined #openstack-cinder | 14:42 | |
| donnyd | Does anyone know if there is a way to expand the number of workers for the volume service? | 14:43 |
| smcginnis | donnyd: There's backend_native_threads_pool_size, but not sure which operations that actually has an impact on. | 14:45 |
| donnyd | The reason I ask is I have an all nvme block storage server and it seems the volume service can sometimes have a hard time keeping up | 14:46 |
| donnyd | It works most of the time, but when it doesn't it blows up pretty fantastically | 14:47 |
| smcginnis | Maybe osapi_volume_workers for API requests or executor_thread_pool_size in [oslo.messaging]? | 14:48 |
| donnyd | the volume service is the only thing running on that server, so there is no shared load from anything else | 14:48 |
| smcginnis | Or wsgi_default_pool_size | 14:48 |
| donnyd | Well its not the apis, i already fixed that one | 14:48 |
| donnyd | Its the actual volume service itself | 14:48 |
| smcginnis | What is it taking a long time on? Like what operations? | 14:49 |
| donnyd | Well I am supposed to get central logging up and running, and when I can stop putting out fires then I would be able to give you more precise info. | 14:50 |
| donnyd | This is the error I get, but only when the load is very high | 14:51 |
| donnyd | Stderr: 'Traceback (most recent call last):\n File "/usr/local/bin/cinder-rtstool", line 10, in <module>\n sys.exit(main())\n File "/usr/local/lib/python3.7/site-packages/cinder/cmd/rtstool.py", line 288, in main\n **optional_args)\n File "/usr/local/lib/python3.7/site-packages/cinder/cmd/rtstool.py", line 56, in create\n dev=backing_device)\n File "/usr/lib/python3.7/site-packages/rtslib_fb/tcm.py", line | 14:51 |
| donnyd | 738, in __init__\n self._configure(dev, wwn, readonly)\n File "/usr/lib/python3.7/site-packages/rtslib_fb/tcm.py", line 748, in _configure\n raise RTSLibError("Device %s is not a TYPE_DISK block device" % dev)\nrtslib_fb.utils.RTSLibError: Device /dev/cinder-volumes/volume-899b9319-2092-4105-b336-82e1171aed6b is not a TYPE_DISK block device\n' | 14:51 |
| donnyd | And there are some others too | 14:52 |
| smcginnis | Hmm, that looks like some kind of timing issue with rtstool or something. | 14:52 |
| donnyd | https://www.irccloud.com/pastebin/i0BLkoGx/ | 14:54 |
| donnyd | Its like it can't do the things its supposed to fast enough.. not 100% sure its a cinder issue | 14:54 |
| donnyd | is there a way to throttle concurency? | 14:54 |
| donnyd | It takes a beating from zuul, so if I could slow things down just a little it would probably work fine too | 14:56 |
| donnyd | Maybe limiting nova could also work | 14:56 |
| *** usr2033 has quit IRC | 14:56 | |
| smcginnis | I wonder if that driver actually needs to add come concurrency locking to make sure operations don't overlap and interfere with each other. | 14:57 |
| smcginnis | I don't really have any suggestions, but maybe someone else here might. | 14:57 |
| smcginnis | You could also file a bug to capture any logs and debugging, but not sure if the driver maintainers are jumping on those or not. | 14:58 |
| *** sahid has quit IRC | 15:05 | |
| donnyd | I think throttling nova is going to work for me | 15:06 |
| donnyd | zuul will launch 100 instances at a time, which is fine for the big fellas... but my poor little cloud doesn't handle that well | 15:07 |
| donnyd | LOL | 15:07 |
| donnyd | http://grafana.openstack.org/d/3Bwpi5SZk/nodepool-fortnebula?orgId=1&from=now-30m&to=now | 15:07 |
| *** ociuhandu has quit IRC | 15:09 | |
| *** ociuhandu has joined #openstack-cinder | 15:09 | |
| *** markvoelker has quit IRC | 15:18 | |
| donnyd | Yea, it did.. I was just beating my volume server too much. Thanks smcginnis | 15:19 |
| smcginnis | Cool! | 15:19 |
| *** markvoelker has joined #openstack-cinder | 15:20 | |
| *** vishalmanchanda has quit IRC | 15:21 | |
| *** eharney has quit IRC | 15:32 | |
| *** zhubx has quit IRC | 15:38 | |
| *** zhubx has joined #openstack-cinder | 15:38 | |
| *** zhubx has quit IRC | 15:40 | |
| *** zhubx has joined #openstack-cinder | 15:41 | |
| *** ociuhandu has quit IRC | 15:56 | |
| *** ociuhandu has joined #openstack-cinder | 16:00 | |
| *** davidsha has quit IRC | 16:03 | |
| openstackgerrit | Luigi Toscano proposed openstack/cinder-tempest-plugin master: WIP Update cinder-tempest-plugin-lvm-lio (revamp) https://review.opendev.org/672804 | 16:04 |
| openstackgerrit | Luigi Toscano proposed openstack/cinder master: WIP Port several legacy tempest jobs to Zuul v3 https://review.opendev.org/671945 | 16:05 |
| *** udesale has quit IRC | 16:07 | |
| openstackgerrit | Rafael Weingärtner proposed openstack/cinder master: Add mv to return volume type name and ID with volumes https://review.opendev.org/666886 | 16:10 |
| *** pcaruana has quit IRC | 16:14 | |
| *** ociuhandu has quit IRC | 16:21 | |
| *** ociuhandu has joined #openstack-cinder | 16:22 | |
| *** ociuhandu has quit IRC | 16:26 | |
| *** ociuhandu has joined #openstack-cinder | 16:27 | |
| *** altlogbot_0 has quit IRC | 16:29 | |
| *** irclogbot_0 has quit IRC | 16:33 | |
| *** altlogbot_0 has joined #openstack-cinder | 16:36 | |
| *** irclogbot_0 has joined #openstack-cinder | 16:41 | |
| openstackgerrit | Sofia Enriquez proposed openstack/cinder master: Fix ceph driver error during incremental backup creation https://review.opendev.org/674152 | 16:43 |
| rosmaita | jungleboyj: smcginnis: pots: see what you think of this: https://wiki.openstack.org/wiki/Cinder/3rdParty-drivers-py3-update | 16:51 |
| rosmaita | smcginnis: i can't figure out how to get '[[local|localrc]]' to show up as literals (the wiki turns it into a link) | 16:52 |
| smcginnis | Maybe throw in some extra spaces? | 16:56 |
| pots | rosmaita: maybe this is the answer? https://en.wikipedia.org/wiki/Template:Square_bracket_open | 16:56 |
| *** ociuhandu has quit IRC | 16:57 | |
| *** ociuhandu has joined #openstack-cinder | 16:57 | |
| smcginnis | rosmaita: Thanks for getting that together. Maybe now that it is there, others will be willing to add their notes to help make it easier to run CIs. | 16:58 |
| *** tesseract has quit IRC | 16:58 | |
| rosmaita | that would be nice | 16:58 |
| smcginnis | We can hope... | 16:58 |
| rosmaita | i tried the extra spaces, i will give pots's suggestion a try | 16:58 |
| rosmaita | if all else fails i will ask fungi | 16:59 |
| fungi | of course you will ;) | 17:00 |
| rosmaita | fungi: hello! got it to work by using [[ | 17:01 |
| rosmaita | so, nevermind | 17:01 |
| fungi | you can wrap it in something like <nowiki></nowiki> too i think | 17:01 |
| * fungi doesn't remember the exact syntax | 17:01 | |
| rosmaita | didn't think of that one, i tried <code></code> but no dice | 17:02 |
| fungi | mediawiki docs should be easy to find for the nowiki tag | 17:02 |
| fungi | it basically just turns off markup for anything inside there | 17:02 |
| *** ociuhandu has quit IRC | 17:02 | |
| rosmaita | cool, i will have to keep that in mind | 17:02 |
| *** jcosmao has left #openstack-cinder | 17:03 | |
| rosmaita | useful to know that the html entity codes work too in a pinch | 17:03 |
| fungi | yup | 17:03 |
| rosmaita | i am taking the afternoon off, i wish everyone a good weekend | 17:03 |
| *** rosmaita has left #openstack-cinder | 17:04 | |
| *** dpawlik has joined #openstack-cinder | 17:19 | |
| *** openstackgerrit has quit IRC | 17:22 | |
| *** ociuhandu has joined #openstack-cinder | 17:26 | |
| *** dpawlik has quit IRC | 17:34 | |
| *** eharney has joined #openstack-cinder | 17:41 | |
| pots | jungleboyj: smcginnis: do we have an os-brick expert i can run something by before posting a patch? os-brick support for iSCSI multipath seems to be broken, at least for my configs. | 17:49 |
| *** ociuhandu has quit IRC | 17:49 | |
| smcginnis | pots: I think hemna_ is still the export, but IIRC geguileo did some work fixing some of the multipath issues. | 17:50 |
| *** ociuhandu has joined #openstack-cinder | 17:50 | |
| hemna_ | sup | 17:50 |
| *** ociuhandu has quit IRC | 17:50 | |
| pots | hi, i think there's an error in the os-brick where it does discovery of multipath devices. | 17:51 |
| pots | the symptom is we see this command failing: "iscsiadm -m node -T ('iqn.2015-11.com.hpe:storage.msa2050.120114328c',) -p 10.135.193.109:3260 --op update -n node.startup -v automatic" | 17:52 |
| hemna_ | what's the error? | 17:52 |
| hemna_ | do you have a stacktrace? pastebin? | 17:52 |
| pots | it's due to this statement in _recover_node_startup_values at the bottom of initiator/connectors/iscsi.py: recover_connection['target_iqn'] = [iqn] | 17:54 |
| pots | the extra level of indirection causes the command line syntax error you see there. iscsiadm just reports "No records found" | 17:54 |
| pots | backtrace: https://paste.ubuntu.com/p/mYNk9QxVys/ | 17:56 |
| jungleboyj | pots: Yes, hemna_ is your friend. Sorry for the slow response. | 17:59 |
| hemna_ | ok just a sec | 18:00 |
| pots | jungleboyj: no sweat, just glad i think i found it before the weekend started. | 18:01 |
| pots | jungleboyj: this is keeping the multiattach tests from passing for the Lenovo driver. | 18:02 |
| pots | jungleboyj: just on iSCSI. | 18:02 |
| jungleboyj | pots: ah, I saw some failures there. Thanks for looking into it. | 18:03 |
| pots | hemna_: here's a full set of logs if it helps: http://os-logs.tristero.net/hpmsa-ci/674077-2/HPMSA-iSCSI/ | 18:04 |
| hemna_ | so it looks like 'discovery_auth_method' isn't set in the connection_properties from the driver | 18:04 |
| hemna_ | what version of open-iscsi ? | 18:07 |
| hemna_ | iscsiadm is complaining that it can't set the node.session.scan mode to manual | 18:09 |
| pots | Yes, I wasn't aware I was supposed to be setting that property. It's open-iscsi 2.0.873+git0.3b4b4500-14ubuntu3.7 | 18:09 |
| hemna_ | which means that I think it's old | 18:09 |
| pots | it's what comes with ubuntu 16.04 | 18:10 |
| hemna_ | hrmm | 18:10 |
| hemna_ | I think that could be too old | 18:10 |
| hemna_ | your /etc/iscsi dir is empty | 18:10 |
| hemna_ | http://os-logs.tristero.net/hpmsa-ci/674077-2/HPMSA-iSCSI/etc/iscsi/ | 18:10 |
| pots | it just failed to upload due to permissions, are you looking for anything in particular? | 18:11 |
| *** e0ne has quit IRC | 18:11 | |
| hemna_ | just making sure the env is ok | 18:11 |
| pots | i did try patching that statement and it seemed to fix the problem. | 18:13 |
| pots | but you see how the iscsiadm command is called with a bogus command line arg, right? | 18:14 |
| hemna_ | well for whatever reason iscsiadm doesn't think there is anyting exported from that target portal target combo | 18:14 |
| pots | because we're passing in a list of iqns instead of one iqn, so the list gets interpolated into the iscsiadm command line instead of a single iqn, creating a syntax error. | 18:16 |
| pots | the test case seems to pass if I pass in "iqn" instead of "[iqn]" | 18:19 |
| hemna_ | this is all geguileo's new discovery code | 18:22 |
| *** lemko has quit IRC | 18:23 | |
| pots | so would this need to be patched on pypi? | 18:25 |
| donnyd | is there a known method to get volumes unstuck from the detaching state? | 18:32 |
| hemna_ | donnyd: you need to find out why it didn't complete | 18:32 |
| hemna_ | there is a way to force the volume state, but that's only if you know it's safe to do it | 18:33 |
| donnyd | i know that its safe | 18:33 |
| hemna_ | reset-state | 18:33 |
| donnyd | Yea, that was the first thing i tried | 18:33 |
| donnyd | Then i flipped them back to in-use and back to deatching. | 18:34 |
| donnyd | they don't actaully move from the detaching state | 18:34 |
| hemna_ | because something is failing | 18:34 |
| hemna_ | you have to look at the logs to find out why | 18:34 |
| donnyd | well volume is doing nothing and the apis and schedulers throw no errors | 18:35 |
| *** openstackgerrit has joined #openstack-cinder | 18:35 | |
| openstackgerrit | Sofia Enriquez proposed openstack/cinder master: Fix ceph: only close rbd image after snapshot iteration is finished https://review.opendev.org/625396 | 18:35 |
| hemna_ | nova ? | 18:35 |
| donnyd | no tracebacks or errors I can see in nova either | 18:37 |
| hemna_ | it's in there | 18:37 |
| donnyd | I had a couple hundred volumes blow up, and there are just a few left that won't actaully detach | 18:37 |
| hemna_ | pots:what version of iscsiadm ? | 18:38 |
| pots | 2.0.873+git0.3b4b4500-14ubuntu3.7 | 18:39 |
| hemna_ | pots:what's odd is that the iscsiadm error isn't a syntax error | 18:40 |
| pots | it accepts the parens on the command line? | 18:40 |
| hemna_ | http://paste.openstack.org/show/755344/ | 18:40 |
| hemna_ | that's what the error should be | 18:40 |
| hemna_ | but it's not | 18:41 |
| hemna_ | Stderr: 'iscsiadm: No records found\n' | 18:41 |
| hemna_ | that's what you are seeing | 18:41 |
| hemna_ | oh | 18:41 |
| hemna_ | I think that's simply a translation | 18:41 |
| pots | hemna_: i think you're missing the quotes in your bash example | 18:42 |
| hemna_ | I copy and pasted from the traceback output | 18:42 |
| hemna_ | so... | 18:42 |
| hemna_ | https://paste.ubuntu.com/p/mYNk9QxVys/ | 18:42 |
| hemna_ | I think that's a red herring | 18:42 |
| hemna_ | possibly | 18:42 |
| donnyd | https://www.irccloud.com/pastebin/q447jYRo/ | 18:43 |
| donnyd | so it shows the requests for state change, and no errors | 18:44 |
| donnyd | nova api placement and compute logs show nothing | 18:44 |
| donnyd | unless you want to see a bunch of haproxy checks and instance usage audits, there is nothing to see | 18:45 |
| hemna_ | donnyd:that's just the cinder api | 18:45 |
| hemna_ | the failure is in nova compute | 18:45 |
| pots | hemna_: i put before and after logs for one test case here: http://os-logs.tristero.net/chris/ | 18:45 |
| donnyd | hemna_: there is nothing in nova compute... there may have been an error thrown when the instance existed, but zuul promptly burned them | 18:46 |
| donnyd | well nodepool | 18:47 |
| donnyd | when state is flipped from in-use to deatching, it should try to detach.. and it does not | 18:47 |
| hemna_ | does cinder terminate_connection get called? | 18:47 |
| hemna_ | and does nova-compute call os-brick to detach ? | 18:48 |
| donnyd | nova-compute is not doing anything. I am assuming terminate_connection is in cinder-volume, but i don't know | 18:49 |
| hemna_ | c-vol gets called to terminate_connection | 18:49 |
| hemna_ | then nova issues commands to os-brick to detach | 18:49 |
| hemna_ | if terminate_connection isn't returning, then there is a problem with the driver/backend | 18:50 |
| donnyd | https://www.irccloud.com/pastebin/Sd1WVORh/ | 18:50 |
| donnyd | cinder-volume is just waiting there | 18:50 |
| donnyd | service says its up | 18:51 |
| hemna_ | c-api gets the call to detach ? | 18:51 |
| donnyd | 2019-08-02 14:42:58.701 38013 DEBUG cinder.api.openstack.wsgi [req-d6f76cb2-acc3-4851-b6de-c97e2e904149 6610181deb904e08ad71af296b769a78 e8fd161dc34c421a979a9e6421f823e9 - default default] Action: 'action', calling method: _reset_status, body: {"os-reset_status": {"status": "detaching"}} _process_stack /usr/lib/python2.7/site-packages/cinder/api/openstack/wsgi.py:870 | 18:52 |
| hemna_ | that's a reset status | 18:52 |
| hemna_ | not a call to detach | 18:52 |
| donnyd | never happens | 18:55 |
| *** yaawang has joined #openstack-cinder | 18:57 | |
| donnyd | That is not something I can see how to control. I can set the state from in-use to detaching, which should trigger a call to detach like it did on the other volumes... but it doesn't seem to do it and i see no logs | 18:57 |
| pots | hemna_: at the risk of beating a dead horse, here's the iscsiadm command run with the 2.9.1 code, with my patch, and one just to show that there is a matching record. https://paste.ubuntu.com/p/VWtdhKHZWJ/ | 19:01 |
| pots | hemna: and i think you're right about the open-iscsi being to old, but i'm not sure what to do about that. Is Ubuntu 16.04 too old? I tried to move to 18.04 but saw kernel crashes with nested virtualization. | 19:04 |
| *** CeeMac has quit IRC | 19:04 | |
| *** e0ne has joined #openstack-cinder | 19:13 | |
| *** yaawang has quit IRC | 19:25 | |
| *** tosky has quit IRC | 19:32 | |
| *** dpawlik has joined #openstack-cinder | 19:37 | |
| *** rtjure has quit IRC | 19:38 | |
| *** irclogbot_0 has quit IRC | 19:39 | |
| *** irclogbot_2 has joined #openstack-cinder | 19:42 | |
| *** whoami-rajat has quit IRC | 20:04 | |
| hemna_ | pots:looking at other CI runs, lots of people are getting those complaings about trying to update the scan mode | 20:09 |
| hemna_ | I'm guessing it was never looked at | 20:09 |
| hemna_ | :( | 20:09 |
| pots | shall i submit a bug with the patch? | 20:09 |
| hemna_ | trying to find other iscsi CI's that are running w/ multipath on | 20:10 |
| pots | i can post a patch too if it would make things faster. do you expect geguileo to be around Monday am? | 20:10 |
| *** e0ne has quit IRC | 20:11 | |
| pots | i upgraded to py37 and turned on multipath and multiattach all at once. what could go wrong? | 20:11 |
| hemna_ | :) | 20:11 |
| pots | ok, posted a bug: https://bugs.launchpad.net/cinder/+bug/1838820 | 20:11 |
| openstack | Launchpad bug 1838820 in Cinder "Bad args to iscsiadm for iSCSI multipath target" [Undecided,New] | 20:11 |
| *** dpawlik has quit IRC | 20:19 | |
| *** enriquetaso has quit IRC | 20:28 | |
| *** baojg has quit IRC | 20:31 | |
| *** rtjure has joined #openstack-cinder | 20:33 | |
| *** mriedem has quit IRC | 20:33 | |
| *** markvoelker has quit IRC | 20:42 | |
| donnyd | I was just trying to setup this (multipath) lvm/iscsi up for my cloud and couldn't get it working | 20:46 |
| donnyd | In my case I was only seeing one client > target connection, even though c-vol was populating the target with both portals | 20:47 |
| *** markvoelker has joined #openstack-cinder | 20:52 | |
| *** markvoelker has quit IRC | 20:56 | |
| *** _erlon_ has joined #openstack-cinder | 21:15 | |
| *** jdillaman1 has quit IRC | 21:28 | |
| *** spatel has joined #openstack-cinder | 21:35 | |
| smcginnis | donnyd: Are these concurrent volume creation requests all unique images? Or creating multiple volumes from the same set of images? | 21:38 |
| donnyd | I have caching turned on, so it's just a clone of the same 3 volumes | 21:39 |
| smcginnis | donnyd: OK, just wanted to make sure you were aware of https://docs.openstack.org/cinder/latest/admin/blockstorage-image-volume-cache.html | 21:40 |
| smcginnis | LVM is probably a little limited with that. I know other backends can create over 100 volumes at once with that turned on because for some of them it's just an update of some metadata. | 21:40 |
| donnyd | Yea, that helps quite a lot, but 70 requests at the same time doesn't work on just one lowly c-vol server | 21:41 |
| smcginnis | Oh well, throttling it is I guess. | 21:41 |
| donnyd | I would just use ceph, but it's too slow | 21:41 |
| donnyd | It seems to work, it's just building jobs for the openstack ci | 21:42 |
| *** spatel has quit IRC | 21:42 | |
| donnyd | And at my scale (small), I have to make the best of everything | 21:42 |
| smcginnis | ;) | 21:42 |
| donnyd | It also lowers the load on neutron, so two birds one stone | 21:43 |
| donnyd | Nodepool will beat a cloud to death if not throttled | 21:43 |
| donnyd | Lol | 21:43 |
| smcginnis | Seems like a good area where someone could spend some time doing some performance analysis to see if there's anyway to optimize things, but if it's between getting things a little slower versus having things fail, I guess it's a good solution. | 21:44 |
| donnyd | After I flipped on the throttling, all of the issues disappeared | 21:44 |
| donnyd | That is what I have spent my nights and weekends doing for the last three weeks | 21:44 |
| donnyd | I needed FN to put out as much as possible with as little as possible | 21:45 |
| donnyd | Now you get 50K iops in each instance | 21:45 |
| smcginnis | Nice, that's a good improvement from what you were seeing before, right? | 21:46 |
| donnyd | 4k to 50k | 21:47 |
| smcginnis | Not bad! | 21:47 |
| *** markvoelker has joined #openstack-cinder | 21:48 | |
| *** eharney has quit IRC | 22:20 | |
| *** markvoelker has quit IRC | 22:21 | |
| *** sapd1 has quit IRC | 22:26 | |
| *** sapd1 has joined #openstack-cinder | 22:26 | |
| *** sapd1 has quit IRC | 22:27 | |
| *** enriquetaso has joined #openstack-cinder | 22:42 | |
| *** dviroel has quit IRC | 22:53 | |
| *** N3l1x has quit IRC | 23:17 | |
| *** markvoelker has joined #openstack-cinder | 23:20 | |
| *** _erlon_ has quit IRC | 23:25 | |
| *** markvoelker has quit IRC | 23:53 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!