| *** hemna has quit IRC | 00:10 | |
| *** whoami-rajat has quit IRC | 00:20 | |
| *** imacdonn has quit IRC | 00:58 | |
| *** dklyle has joined #openstack-cinder | 01:20 | |
| *** baojg has joined #openstack-cinder | 01:38 | |
| *** gmann has quit IRC | 01:43 | |
| *** dklyle has quit IRC | 02:02 | |
| *** hemna has joined #openstack-cinder | 02:06 | |
| openstackgerrit | zhufl proposed openstack/cinder master: [Trivial fix]Remove unnecessary slash https://review.opendev.org/657219 | 02:32 |
|---|---|---|
| *** hemna has quit IRC | 02:39 | |
| *** altlogbot_2 has quit IRC | 02:41 | |
| *** altlogbot_3 has joined #openstack-cinder | 02:42 | |
| *** hemna has joined #openstack-cinder | 03:08 | |
| *** psachin has joined #openstack-cinder | 03:18 | |
| *** whoami-rajat has joined #openstack-cinder | 03:24 | |
| *** brinzhang has joined #openstack-cinder | 03:27 | |
| *** altlogbot_3 has quit IRC | 03:31 | |
| *** altlogbot_2 has joined #openstack-cinder | 03:31 | |
| *** gkadam has joined #openstack-cinder | 03:42 | |
| *** gkadam has quit IRC | 03:42 | |
| *** hemna has quit IRC | 03:43 | |
| *** ruffian_sheep has joined #openstack-cinder | 03:53 | |
| *** udesale has joined #openstack-cinder | 03:53 | |
| *** sapd1_x has joined #openstack-cinder | 03:54 | |
| *** sapd1_x has quit IRC | 04:41 | |
| *** ruffian_sheep has quit IRC | 04:44 | |
| *** boxiang has quit IRC | 04:51 | |
| *** boxiang has joined #openstack-cinder | 04:51 | |
| *** davee_ has quit IRC | 05:03 | |
| *** davee_ has joined #openstack-cinder | 05:03 | |
| *** hemna has joined #openstack-cinder | 05:06 | |
| *** sapd1_x has joined #openstack-cinder | 05:25 | |
| *** hemna has quit IRC | 05:40 | |
| *** sapd1_x has quit IRC | 05:53 | |
| *** lpetrut has joined #openstack-cinder | 05:54 | |
| *** sapd1_x has joined #openstack-cinder | 05:55 | |
| openstackgerrit | zhufl proposed openstack/cinder master: [Trivial fix]Remove unnecessary slash https://review.opendev.org/657219 | 05:59 |
| openstackgerrit | Yury Kulazhenkov proposed openstack/cinder master: remove support for deprecated options in VxFlex OS driver https://review.opendev.org/638358 | 06:03 |
| openstackgerrit | Yury Kulazhenkov proposed openstack/cinder master: Add support for VxFlex OS 3.0 to VxFlex OS driver https://review.opendev.org/639963 | 06:03 |
| openstackgerrit | Yury Kulazhenkov proposed openstack/cinder master: rename ScaleIO driver to VxFlex OS https://review.opendev.org/634397 | 06:06 |
| openstackgerrit | Yury Kulazhenkov proposed openstack/cinder master: remove support for deprecated options in VxFlex OS driver https://review.opendev.org/638358 | 06:06 |
| openstackgerrit | Yury Kulazhenkov proposed openstack/cinder master: Add support for VxFlex OS 3.0 to VxFlex OS driver https://review.opendev.org/639963 | 06:06 |
| *** udesale has quit IRC | 06:23 | |
| *** udesale has joined #openstack-cinder | 06:24 | |
| openstackgerrit | Yachika_Ralhan proposed openstack/cinder master: Add new OSNEXUS Quantastor Cinder driver https://review.opendev.org/617503 | 06:43 |
| *** tesseract-RH has joined #openstack-cinder | 07:05 | |
| *** baojg has quit IRC | 07:07 | |
| *** altlogbot_2 has quit IRC | 07:13 | |
| *** altlogbot_0 has joined #openstack-cinder | 07:17 | |
| *** sahid has joined #openstack-cinder | 07:33 | |
| *** brinzh has joined #openstack-cinder | 07:39 | |
| *** brinzhang has quit IRC | 07:42 | |
| *** udesale has quit IRC | 07:55 | |
| *** udesale has joined #openstack-cinder | 07:55 | |
| openstackgerrit | tonybrad proposed openstack/cinderlib master: Change deps source https://review.opendev.org/657339 | 08:14 |
| *** pcaruana has joined #openstack-cinder | 08:24 | |
| *** sapd1_x has quit IRC | 08:40 | |
| openstackgerrit | Yachika_Ralhan proposed openstack/cinder master: Add new OSNEXUS Quantastor Cinder driver https://review.opendev.org/617503 | 08:56 |
| *** altlogbot_0 has quit IRC | 09:11 | |
| *** hemna has joined #openstack-cinder | 09:12 | |
| *** altlogbot_2 has joined #openstack-cinder | 09:14 | |
| *** altlogbot_2 has quit IRC | 09:15 | |
| *** altlogbot_1 has joined #openstack-cinder | 09:16 | |
| *** savihou has joined #openstack-cinder | 09:31 | |
| *** gmann has joined #openstack-cinder | 09:41 | |
| *** hemna has quit IRC | 09:46 | |
| *** gmann is now known as gmann_pto | 09:57 | |
| *** udesale has quit IRC | 10:34 | |
| *** udesale has joined #openstack-cinder | 10:35 | |
| *** takamatsu has joined #openstack-cinder | 10:35 | |
| *** anks2k has joined #openstack-cinder | 10:51 | |
| *** anks2k has left #openstack-cinder | 10:52 | |
| openstackgerrit | DingDong proposed openstack/cinder master: [Unity] Retype volume support https://review.opendev.org/656271 | 10:52 |
| openstackgerrit | DingDong proposed openstack/cinder master: [Unity] Retype volume support https://review.opendev.org/656271 | 10:56 |
| *** dpawlik has joined #openstack-cinder | 11:03 | |
| openstackgerrit | DingDong proposed openstack/cinder master: [Unity] Retype volume support https://review.opendev.org/656271 | 11:04 |
| openstackgerrit | DingDong proposed openstack/cinder master: [Unity] Retype volume support https://review.opendev.org/656271 | 11:05 |
| *** carloss has joined #openstack-cinder | 11:06 | |
| *** udesale has quit IRC | 11:17 | |
| *** altlogbot_1 has quit IRC | 11:19 | |
| *** altlogbot_3 has joined #openstack-cinder | 11:20 | |
| *** nikeshm has joined #openstack-cinder | 11:34 | |
| nikeshm | hi | 11:35 |
| *** hemna has joined #openstack-cinder | 11:43 | |
| *** henriqueof has joined #openstack-cinder | 11:44 | |
| *** m75abrams has joined #openstack-cinder | 12:05 | |
| *** abishop has joined #openstack-cinder | 12:08 | |
| *** _hemna has joined #openstack-cinder | 12:10 | |
| *** hemna has quit IRC | 12:12 | |
| *** altlogbot_3 has quit IRC | 12:14 | |
| *** _hemna has quit IRC | 12:16 | |
| *** udesale has joined #openstack-cinder | 12:19 | |
| *** altlogbot_0 has joined #openstack-cinder | 12:20 | |
| *** nikeshm has quit IRC | 12:20 | |
| *** udesale has quit IRC | 12:25 | |
| *** brinzh has quit IRC | 12:26 | |
| *** udesale has joined #openstack-cinder | 12:26 | |
| *** pcaruana has quit IRC | 12:29 | |
| *** hemna has joined #openstack-cinder | 12:43 | |
| *** zul has joined #openstack-cinder | 12:46 | |
| *** _hemna has joined #openstack-cinder | 12:48 | |
| *** mchlumsky has joined #openstack-cinder | 12:48 | |
| *** hemna has quit IRC | 13:01 | |
| *** mchlumsky has quit IRC | 13:05 | |
| *** mchlumsky has joined #openstack-cinder | 13:08 | |
| *** m75abrams has quit IRC | 13:11 | |
| *** altlogbot_0 has quit IRC | 13:15 | |
| *** altlogbot_2 has joined #openstack-cinder | 13:16 | |
| *** pcaruana has joined #openstack-cinder | 13:16 | |
| *** baojg has joined #openstack-cinder | 13:17 | |
| *** mmethot has joined #openstack-cinder | 13:19 | |
| *** hemna has joined #openstack-cinder | 13:19 | |
| *** mriedem has joined #openstack-cinder | 13:19 | |
| *** altlogbot_2 has quit IRC | 13:21 | |
| *** _hemna has quit IRC | 13:21 | |
| *** altlogbot_3 has joined #openstack-cinder | 13:24 | |
| *** altlogbot_3 has quit IRC | 13:25 | |
| *** altlogbot_1 has joined #openstack-cinder | 13:26 | |
| *** psachin has quit IRC | 13:28 | |
| *** udesale has quit IRC | 13:46 | |
| *** m75abrams has joined #openstack-cinder | 13:46 | |
| *** udesale has joined #openstack-cinder | 13:47 | |
| openstackgerrit | Walter A. Boring IV (hemna) proposed openstack/cinder master: Move NVMET driver exceptions to driver file https://review.opendev.org/656957 | 13:48 |
| *** ianychoi has quit IRC | 13:48 | |
| *** _hemna has joined #openstack-cinder | 13:52 | |
| *** ianychoi has joined #openstack-cinder | 13:52 | |
| openstackgerrit | Walter A. Boring IV (hemna) proposed openstack/cinder master: move NVMEOF exception to target file https://review.opendev.org/656976 | 13:56 |
| *** gnufied has joined #openstack-cinder | 14:01 | |
| openstackgerrit | Walter A. Boring IV (hemna) proposed openstack/cinder master: move Hyperscale exceptions to driver https://review.opendev.org/656994 | 14:02 |
| *** ianychoi has quit IRC | 14:04 | |
| *** udesale has quit IRC | 14:16 | |
| *** ianychoi has joined #openstack-cinder | 14:18 | |
| *** udesale has joined #openstack-cinder | 14:21 | |
| *** egonzalez has quit IRC | 14:22 | |
| *** egonzalez has joined #openstack-cinder | 14:24 | |
| *** _hemna has quit IRC | 14:26 | |
| *** m75abrams has quit IRC | 14:34 | |
| *** lpetrut has quit IRC | 14:36 | |
| *** baojg has quit IRC | 14:39 | |
| *** baojg has joined #openstack-cinder | 14:43 | |
| *** dklyle has joined #openstack-cinder | 14:48 | |
| *** dklyle has quit IRC | 14:49 | |
| *** david-lyle has joined #openstack-cinder | 14:49 | |
| *** imacdonn has joined #openstack-cinder | 15:11 | |
| *** david-lyle is now known as dklyle | 15:13 | |
| *** markvoelker has joined #openstack-cinder | 15:16 | |
| openstackgerrit | Jay Rubenstein proposed openstack/cinder master: SF: Handle qos values on extend volume https://review.opendev.org/606959 | 15:21 |
| hemna | smcginnis: thanks for the comment | 15:26 |
| hemna | trying to get all the right things in place | 15:26 |
| smcginnis | hemna: No problem. Hoping we can avoid the extra work. | 15:26 |
| *** qqmber has joined #openstack-cinder | 15:27 | |
| qqmber | hi there.. | 15:27 |
| qqmber | Everytime I send a lot of small files to a volume that is managed by cinder, I got this error: "blocked for more than 120 seconds" in the instance and the volume is remounted as read only. | 15:28 |
| qqmber | here is a bug report back from 2014: https://bugs.launchpad.net/nova/+bug/1373513 | 15:28 |
| openstack | Launchpad bug 1373513 in Cinder "Lvm hang during tempest tests" [Critical,Fix released] - Assigned to John Griffith (john-griffith) | 15:28 |
| smcginnis | qqmber: That would be an issue on the data path. Cinder doesn't do anything with that. | 15:29 |
| qqmber | smcginnis: the data path... the network? | 15:29 |
| smcginnis | qqmber: Check your networking for issues. | 15:29 |
| smcginnis | Yep | 15:29 |
| qqmber | :( | 15:29 |
| smcginnis | qqmber: Are you using LVM? | 15:29 |
| qqmber | yes, I'm | 15:29 |
| qqmber | let's see | 15:30 |
| smcginnis | qqmber: OK, so could maybe be something on that host that is serving up LVM via iSCSI. Maybe something of interest in the logs there too. | 15:30 |
| qqmber | let me see | 15:30 |
| qqmber | smcginnis: at least in the host that serves as storage node that has the volume in question, there is no errors in its ethernet interface | 15:33 |
| qqmber | what log shuld I see? messages? which one? | 15:34 |
| smcginnis | qqmber: You could also check CPU load. If that host is too busy, that could cause some slow responses and potentially retries. | 15:34 |
| *** baojg has quit IRC | 15:37 | |
| qqmber | smcginnis: the CPU load on the storage node reach 10. Being a host with 32 CPU, I don't think that would be a problem (AFAIK since is below 32 the load I'd be ok) | 15:39 |
| qqmber | this is the only instance that has traffic. It is a new Openstack installation (Rocky) | 15:40 |
| *** savihou has quit IRC | 15:45 | |
| jgriffith | qqmber: What are you using for your backing files in your LVM deployment? | 15:46 |
| jgriffith | Are they *real* disks? | 15:47 |
| jgriffith | and could you clarify what you mean by "sending lots of small files to a volume" | 15:47 |
| qqmber | jgriffith: real = /dev/sdf from a Dell server with Centos 7 installed... ? | 15:48 |
| jgriffith | qqmber: ok, that's good, eliminates most of the topic in that bug | 15:48 |
| openstackgerrit | Rajini Karthik proposed openstack/cinder master: Dell EMC SC: Handle the mappings of multiattached volume https://review.opendev.org/656835 | 15:49 |
| qqmber | Total: 120GB. Each file is 20KB | 15:49 |
| jgriffith | qqmber: in that case it leads back to what smcginnis mentioned regarding network drops. Could be iscsi queue depth or just dropped packets | 15:49 |
| jgriffith | qqmber: are you just trying to do something like `cp ` a ton of files to your cinder-volume (mounted and formatted)? | 15:50 |
| qqmber | jgriffith: at least there are not errors on the network interface.. What else should I look at? | 15:50 |
| qqmber | jgriffith: rsync | 15:51 |
| jgriffith | qqmber: You'd have to look in syslog then I think | 15:52 |
| jgriffith | But that's just going to tell you what you already know | 15:52 |
| *** henriqueof has quit IRC | 15:53 | |
| qqmber | jgriffith: the volume that "lives" on /dev/sdf from one physical host (called "iquique"), receives files from another place (Another server in the same rack, on the datacenter, passing thru one or two switches) thru the gateway host (Neutron, called "Arica"), and the volume is mounted on an intance that runs on another host (called "Antofagasta). Yes, it is mounted and formatted with ext4. | 15:53 |
| jgriffith | you'll have to figure out how to tune iscsi on your system to deal with your tansfer. I don't know if queue depth would help here or not | 15:53 |
| jgriffith | could also be a caching issues | 15:54 |
| jgriffith | Ohhh.. hmm, have you isolated it to the cinder volume by trying to rsync to the root fs on the nova instance? | 15:54 |
| qqmber | jgriffith: I'm not familiar with queue depth nor cache (in this scenario). Where can I read more about it? | 15:54 |
| openstackgerrit | Rajini Karthik proposed openstack/cinder master: Dell EMC SC: Handle the mappings of multiattached volume https://review.opendev.org/656835 | 15:54 |
| jgriffith | unless this is boot from volume then you're back in the same place :) | 15:54 |
| qqmber | jgriffith: It doesn't boot from the volume | 15:55 |
| qqmber | jgriffith: I can try to move some of this files to the root (100GB) | 15:55 |
| qqmber | the error occurs after 65GB approx | 15:55 |
| jgriffith | Then if it were me I would first check if it's even an issue with Cinder volumes or not | 15:55 |
| qqmber | I'll do that... but what if that tests works fine? Where would you look at? | 15:56 |
| jgriffith | If you want to read a little about queuedepth and burn the rest of your day you can start here: https://library.netapp.com/ecmdocs/ECMP1196793/html/GUID-A055B184-0876-4376-9C75-35FE8C9BE832.html | 15:57 |
| jgriffith | :)) | 15:57 |
| qqmber | jgriffith: what do you mean with issues on the cinder volumes? | 15:57 |
| jgriffith | qqmber: your problem may not have anything to do with Cinder/iSCSI | 15:57 |
| jgriffith | it may be something on your networking or on the guest OS itself | 15:57 |
| qqmber | the thing is that in my past deployment (Kilo) I had the same problem.. no erros like this until I did a DRP of a Elasticsearch node.. (same scenario, a lot of small files moving very fast) | 15:58 |
| jgriffith | qqmber: this might be a silly question (sorry if it is) but is the fs going readonly on the source or the destination? | 16:01 |
| qqmber | then the problem occurs? yes | 16:01 |
| qqmber | it remount as readonly | 16:01 |
| qqmber | jgriffith: kernel: INFO: task jbd2/vdb1-8:6796 blocked for more than 120 seconds. | 16:02 |
| qqmber | that's what I got in the instance | 16:02 |
| qqmber | when the problem occurs | 16:02 |
| qqmber | kernel: EXT4-fs (vdb1): Remounting filesystem read-only | 16:02 |
| jgriffith | So you can also look at things like this: https://www.linuxquestions.org/questions/linux-newbie-8/rsync-error-read-only-file-system-30-a-854252/ | 16:02 |
| jgriffith | Even though that's a USB drive there are some tips on things to look for, and an example of somebody hitting issues on disk as well | 16:03 |
| qqmber | what I think, is there is too much traffic/hits/io/etc. for the net_interface/volume/switch/OS/etc. and the volumes crashes or something, and remunt as RO for safety. | 16:05 |
| qqmber | the question is: Is the interface? is the switch? etc.. | 16:05 |
| qqmber | I'll read everything. Thanks! | 16:05 |
| qqmber | jgriffith: just before remount I got this on the instance: 11:15:37 ada-1 kernel: blk_update_request: I/O error, dev vdb, sector 1044710472 | 16:07 |
| qqmber | that's after the 120 sec timeout (6 minutes later..) | 16:08 |
| qqmber | or 8 minutes after the last "ok" | 16:08 |
| jgriffith | qqmber: I think you are on the right track (although it's a guess) | 16:11 |
| qqmber | I have good questions.. but no answers! hahaha | 16:11 |
| *** mvkr has quit IRC | 16:11 | |
| jgriffith | qqmber: you and me both :) | 16:18 |
| *** _hemna has joined #openstack-cinder | 16:22 | |
| *** sahid has quit IRC | 16:48 | |
| qqmber | jgriffith: do you think this could be related? https://bugzilla.redhat.com/show_bug.cgi?id=1439989 | 16:52 |
| openstack | bugzilla.redhat.com bug 1439989 in systemd "why so many sessions scope file didn't been deleted?" [Medium,New] - Assigned to dtardon | 16:52 |
| qqmber | smcginnis: or you? | 16:53 |
| qqmber | because I'm having this issue on the storage host.. it is running docker (I installed openstack with kolla-ansible) | 16:53 |
| qqmber | If I run "systemctl | grep 'of user'| grep 'abandoned'" I see a lot of abandoned sessions... and this in messages: "systemd: Failed to start Session c102725 of user root." every 30 seconds | 16:54 |
| *** gmann_pto has quit IRC | 16:56 | |
| *** henriqueof has joined #openstack-cinder | 17:07 | |
| *** tejdeep has joined #openstack-cinder | 17:14 | |
| *** whoami-rajat has quit IRC | 17:24 | |
| *** tesseract-RH has quit IRC | 17:24 | |
| *** udesale has quit IRC | 17:27 | |
| *** _alastor_ has joined #openstack-cinder | 17:29 | |
| *** tejdeep has quit IRC | 17:36 | |
| *** tejdeep has joined #openstack-cinder | 17:37 | |
| *** jrubenst has joined #openstack-cinder | 17:58 | |
| *** markvoelker has quit IRC | 18:10 | |
| *** _hemna has quit IRC | 18:15 | |
| hemna | qqmber: logs ? | 18:18 |
| qqmber | hemna: in /var/log/messages I see "systemd: Failed to start Session c102725 of user root." every 1 minute or so... | 18:19 |
| qqmber | actually is like 7 at once.. then, one minute later or so, seven more.. and so on | 18:19 |
| hemna | looks like you have too many user sessions | 18:21 |
| hemna | maybe cron jobs out of control? | 18:21 |
| hemna | this is inside a running container ? | 18:22 |
| qqmber | hemna: it seems that is a Docker problem | 18:22 |
| qqmber | no, in the hsot, not the container | 18:22 |
| qqmber | hsot = host | 18:22 |
| hemna | ok, so the docker host then | 18:22 |
| qqmber | yep | 18:22 |
| qqmber | I have my guess... I'll try to change something (reinstall and check then) | 18:23 |
| hemna | ok | 18:23 |
| qqmber | byt he way, I found this at the momento of the error: kernel: connection9:0: detected conn error (1021) | 18:26 |
| *** _hemna has joined #openstack-cinder | 18:33 | |
| *** KeithMnemonic has joined #openstack-cinder | 18:39 | |
| KeithMnemonic | I have two (stein & rocky) up when anyone has a chance to please help move them along https://review.opendev.org/656802 & https://review.opendev.org/656911 | 18:40 |
| KeithMnemonic | these are 3par fixes | 18:40 |
| *** _hemna has quit IRC | 18:40 | |
| hemna | KeithMnemonic: I +1'd those, we need folks with stable releases +2 | 18:43 |
| hemna | qqmber: ok that looks like a host kernel issue and/or a docker issue | 18:44 |
| KeithMnemonic | hemna: thanks that is why i posted ;-) | 18:44 |
| *** tejdeep has quit IRC | 18:51 | |
| *** tejdeep has joined #openstack-cinder | 18:52 | |
| openstackgerrit | Walter A. Boring IV (hemna) proposed openstack/cinder master: Remove unused exception https://review.opendev.org/657437 | 19:09 |
| *** jrubenst has quit IRC | 19:15 | |
| *** rchurch_ has joined #openstack-cinder | 19:32 | |
| *** carloss_ has joined #openstack-cinder | 19:33 | |
| *** rchurch has quit IRC | 19:34 | |
| *** _hemna has joined #openstack-cinder | 19:37 | |
| *** carloss has quit IRC | 19:39 | |
| *** hemna has quit IRC | 19:39 | |
| *** carloss_ is now known as carloss | 19:39 | |
| *** takamatsu has quit IRC | 19:51 | |
| *** nwonknu has quit IRC | 20:00 | |
| openstackgerrit | Walter A. Boring IV (hemna) proposed openstack/cinder master: move synology driver exceptions https://review.opendev.org/657446 | 20:02 |
| *** mchlumsky has quit IRC | 20:31 | |
| *** zul has quit IRC | 20:47 | |
| *** pcaruana has quit IRC | 20:51 | |
| *** nwonknu has joined #openstack-cinder | 21:01 | |
| *** dpawlik has quit IRC | 21:15 | |
| *** henriqueof has quit IRC | 21:16 | |
| *** abishop has quit IRC | 21:17 | |
| *** dpawlik has joined #openstack-cinder | 21:20 | |
| *** dpawlik has quit IRC | 21:24 | |
| *** jdillaman has quit IRC | 21:27 | |
| *** mvkr has joined #openstack-cinder | 22:00 | |
| *** carloss has quit IRC | 22:22 | |
| *** rcernin has joined #openstack-cinder | 22:48 | |
| *** tkajinam has joined #openstack-cinder | 23:03 | |
| *** qqmber has quit IRC | 23:07 | |
| *** tejdeep has quit IRC | 23:24 | |
| *** tejdeep has joined #openstack-cinder | 23:26 | |
| *** jsquare has quit IRC | 23:39 | |
| *** mmethot has quit IRC | 23:45 | |
| *** jsquare has joined #openstack-cinder | 23:45 | |
| *** david-lyle has joined #openstack-cinder | 23:55 | |
| *** dklyle has quit IRC | 23:55 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!