| *** tommylikehu_ has joined #openstack-manila | 01:10 | |
| openstackgerrit | Zhenyu Zheng proposed openstack/manila master: Add priority for access rule https://review.openstack.org/572283 | 02:26 |
|---|---|---|
| openstackgerrit | Zhenyu Zheng proposed openstack/manila master: Add priority for access rule https://review.openstack.org/572283 | 02:26 |
| *** markstur has joined #openstack-manila | 05:03 | |
| *** markstur has quit IRC | 05:07 | |
| *** markstur has joined #openstack-manila | 05:12 | |
| *** markstur has quit IRC | 06:18 | |
| *** markstur has joined #openstack-manila | 07:23 | |
| *** markstur has quit IRC | 07:28 | |
| *** e0ne has joined #openstack-manila | 07:38 | |
| *** pcaruana has joined #openstack-manila | 07:58 | |
| *** pcaruana is now known as muttley | 07:58 | |
| openstackgerrit | Maurice Schreiber proposed openstack/manila master: NetApp cDOT store port IDs and addresses at share server backend details https://review.openstack.org/619573 | 08:03 |
| *** markstur has joined #openstack-manila | 08:25 | |
| *** markstur has quit IRC | 08:30 | |
| *** tommylikehu_ has quit IRC | 08:40 | |
| *** ganso has joined #openstack-manila | 09:49 | |
| carlos_silva | gouthamr: i tried both | 09:57 |
| carlos_silva | tbarron: i'm using the 16.04 version | 09:58 |
| carlos_silva | gouthamr: i tried reaching the server manually after the error, it took a very long time for the ssh to go through, but it works | 09:59 |
| *** luizbag has joined #openstack-manila | 10:20 | |
| *** arne_wiebalck_ has joined #openstack-manila | 10:24 | |
| *** e0ne has quit IRC | 10:24 | |
| *** e0ne has joined #openstack-manila | 10:27 | |
| *** erlon has joined #openstack-manila | 10:31 | |
| *** markstur has joined #openstack-manila | 10:45 | |
| *** markstur has quit IRC | 10:50 | |
| *** arne_wiebalck_ has quit IRC | 11:39 | |
| *** luizbag has quit IRC | 12:24 | |
| *** luizbag has joined #openstack-manila | 12:28 | |
| *** erlon has quit IRC | 12:29 | |
| *** markstur has joined #openstack-manila | 12:36 | |
| *** markstur has quit IRC | 12:40 | |
| *** erlon has joined #openstack-manila | 12:43 | |
| *** luizbag has quit IRC | 12:44 | |
| *** tpsilva has joined #openstack-manila | 12:48 | |
| *** arne_wiebalck_ has joined #openstack-manila | 12:53 | |
| *** eharney has quit IRC | 12:54 | |
| *** arne_wiebalck_ has quit IRC | 12:58 | |
| *** arne_wiebalck_ has joined #openstack-manila | 13:03 | |
| tbarron | carlos_silva: I read that paramiko has a 15s timeout, wonder if it can be tweaked. | 13:04 |
| tbarron | carlos_silva: thanks for raising https://bugs.launchpad.net/manila/+bug/1807126 | 13:04 |
| openstack | Launchpad bug 1807126 in Manila "Cannot create shares using generic driver" [Undecided,New] | 13:04 |
| *** markstur has joined #openstack-manila | 13:07 | |
| *** muttley has quit IRC | 13:08 | |
| *** markstur has quit IRC | 13:12 | |
| *** muttley has joined #openstack-manila | 13:21 | |
| *** muttley has quit IRC | 13:25 | |
| *** muttley has joined #openstack-manila | 13:26 | |
| *** muttley has quit IRC | 13:29 | |
| *** vollman has joined #openstack-manila | 13:30 | |
| *** pcaruana has joined #openstack-manila | 13:34 | |
| *** pcaruana has quit IRC | 13:39 | |
| *** markstur has joined #openstack-manila | 13:39 | |
| *** pcaruana has joined #openstack-manila | 13:43 | |
| *** markstur has quit IRC | 13:44 | |
| *** pcaruana has quit IRC | 13:47 | |
| *** arne_wiebalck_ has quit IRC | 13:57 | |
| *** eharney has joined #openstack-manila | 14:01 | |
| *** markstur has joined #openstack-manila | 14:11 | |
| *** markstur has quit IRC | 14:16 | |
| *** markstur has joined #openstack-manila | 14:42 | |
| *** markstur has quit IRC | 14:47 | |
| *** jgrosso has joined #openstack-manila | 15:09 | |
| *** markstur has joined #openstack-manila | 15:14 | |
| *** markstur has quit IRC | 15:19 | |
| *** zul has joined #openstack-manila | 15:52 | |
| *** markstur has joined #openstack-manila | 15:58 | |
| *** markstur has quit IRC | 16:03 | |
| gouthamr | carlos_silva: if you're willing, can you try the code in this comment in your env, i am curious to see what paramiko's debug logs show - https://bugs.launchpad.net/manila/+bug/1786059/comments/7 | 16:03 |
| openstack | Launchpad bug 1786059 in Manila "Cannot Connect to Share Server when Creating Share with Generic Driver (DHSS=True)" [Undecided,Invalid] | 16:03 |
| tbarron | gouthamr: bswartz: ganso: carlos_silva: I guess 'Error reading SSH protocol headers" *could* be an auth issue but I was thinking that this | 16:03 |
| tbarron | https://stackoverflow.com/questions/25609153/paramiko-error-reading-ssh-protocol-banner | 16:03 |
| tbarron | not that ^^ | 16:03 |
| tbarron | I was thinking that this: | 16:03 |
| tbarron | https://stackoverflow.com/questions/25609153/paramiko-error-reading-ssh-protocol-banner | 16:04 |
| tbarron | yeah that | 16:04 |
| tbarron | fit more with carlos_silva seeing that ssh works manually but takes a real long time | 16:04 |
| gouthamr | hmmm, "banner_timeout" isn't configurable through manila | 16:04 |
| gouthamr | but if it works locally, no harm adding it | 16:05 |
| tbarron | but yeah, try gouthamr's suggestion | 16:05 |
| bswartz | tbarron: As I mentioned a few weeks ago, I'd recommend extreme caution when reading error messages from paramiko | 16:07 |
| tbarron | gouthamr: I already pointed to youe comment#7 yesterday and I think he's tried those suggestions :) | 16:07 |
| tbarron | bswartz: ack | 16:07 |
| bswartz | When paramiko works, it works great, but when it fails, it fails spectacularly and spews errors sometimes unrelated to the actual problem | 16:07 |
| gouthamr | tbarron: paramiko's logs are helpful - atleast were in understanding why the last failure occurred | 16:08 |
| bswartz | gouthamr: I'm not sure about that | 16:08 |
| bswartz | Is there a way to make paramiko log verbosely? | 16:09 |
| gouthamr | yes | 16:09 |
| gouthamr | paramiko.common.logging.basicConfig(level=paramiko.common.DEBUG) | 16:09 |
| gouthamr | code snippet with manila's utils method in https://bugs.launchpad.net/manila/+bug/1786059/comments/7 | 16:09 |
| openstack | Launchpad bug 1786059 in Manila "Cannot Connect to Share Server when Creating Share with Generic Driver (DHSS=True)" [Undecided,Invalid] | 16:09 |
| *** takamatsu has quit IRC | 16:10 | |
| bswartz | gouthamr: do we do this by default? | 16:12 |
| bswartz | or does it have to be turned on by the user? | 16:12 |
| gouthamr | bswartz: we don't, it's too noisy | 16:12 |
| gouthamr | bswartz: we could allow configuring it | 16:12 |
| bswartz | That's what I thought | 16:12 |
| bswartz | Perhaps debug logging would help find the problems | 16:12 |
| * gouthamr remembers kazoo's logging issues :s | 16:12 | |
| bswartz | Stepping through the code in a debugger definitely helps find the problems, and they're not the same as what the python exceptions say they are | 16:13 |
| gouthamr | carlos_silva: could you please share your devstack local.conf on the bug report? | 16:16 |
| carlos_silva | gouthamr: ok, i will share it | 16:19 |
| *** e0ne has quit IRC | 16:30 | |
| carlos_silva | This is the output after running the code present here: https://bugs.launchpad.net/manila/+bug/1786059/comments/7 https://www.irccloud.com/pastebin/a1TnKndJ/ | 16:31 |
| openstack | Launchpad bug 1786059 in Manila "Cannot Connect to Share Server when Creating Share with Generic Driver (DHSS=True)" [Undecided,Invalid] | 16:31 |
| carlos_silva | gouthamr: ^ | 16:31 |
| carlos_silva | gouthamr: i added a comment with the local.conf file | 16:35 |
| gouthamr | carlos_silva: thank you.. will take a look | 16:45 |
| gouthamr | the weirdness may be explained by this: https://github.com/paramiko/paramiko/issues/1192#issuecomment-382876285 | 16:56 |
| gouthamr | okay, so we're able to see that the SSH socket is open, but when we use paramiko it blows up | 17:47 |
| gouthamr | self._ssh_exec(server, ['sudo', 'exportfs']) | 17:47 |
| *** jmlowe has quit IRC | 17:49 | |
| *** eharney has quit IRC | 17:52 | |
| *** e0ne has joined #openstack-manila | 18:08 | |
| *** e0ne has quit IRC | 18:09 | |
| *** jmlowe has joined #openstack-manila | 18:17 | |
| *** jmlowe has quit IRC | 18:37 | |
| *** eharney has joined #openstack-manila | 18:42 | |
| gouthamr | tbarron: https://github.com/openstack/manila-image-elements/blob/master/elements/manila-ssh/install.d/fetch-public-ssh-keys.sh | 18:50 |
| *** jmlowe has joined #openstack-manila | 18:56 | |
| *** vollman has quit IRC | 19:04 | |
| gouthamr | tbarron: http://logs.openstack.org/82/556982/4/check/manila-tempest-dsvm-mysql-generic/4c2ca8d/logs/screen-m-shr.txt.gz?level=ERROR | 19:20 |
| tbarron | bswartz: ganso: for the record, gouthamr and I saw huge delay during key exchange for ssh to the service vm but no delay when we went from a service vm ssh client to itself. Maybe an ssh version issue? | 21:07 |
| tbarron | or some other key/crypto packages | 21:09 |
| bswartz | tbarron: maaaaaybe | 21:13 |
| bswartz | More likely networking issue, either routing or DNS | 21:13 |
| tbarron | bswartz: the delay is always at the key exchange | 21:14 |
| tbarron | same spot every time | 21:14 |
| bswartz | does it eventually succeed? | 21:14 |
| tbarron | yes | 21:14 |
| bswartz | what test method are you using | 21:14 |
| tbarron | just ssh from host to VM, and ssh from VM to VM at 127.0.0.1 | 21:15 |
| bswartz | Is this just ssh -vvv from the command line? | 21:15 |
| tbarron | the latter has no delay, the former about 60s at the key xchg | 21:15 |
| tbarron | yes, no paramiko involved, just cmd line ssh | 21:15 |
| *** jmlowe has quit IRC | 21:16 | |
| *** ganso has quit IRC | 21:18 | |
| bswartz | I can't explain that | 21:18 |
| bswartz | It's not something I've observed before | 21:18 |
| bswartz | Is the nova image still that pathetically old ubuntu image? | 21:18 |
| gouthamr | unsure how old it is, but its running Xenial | 21:20 |
| tbarron | i thought I'd make a bionic one since we're running bionic now for the VM | 21:25 |
| tbarron | it's a way to get the latest packages :) | 21:25 |
| tbarron | probably not a bad idea even if it doesn't fix the issue | 21:26 |
| gouthamr | tbarron: ++ it was part of the plan for stein, never tried this but the code suggests you set MANILA_IMG_OS_VER to the release name, in this case bionic | 21:26 |
| *** luizbag has joined #openstack-manila | 21:29 | |
| *** luizbag has quit IRC | 21:29 | |
| tbarron | gouthamr: yup, that works locally, I'm going to push the review | 21:32 |
| gouthamr | tbarron: yay, easy peasy | 21:33 |
| gouthamr | :P | 21:33 |
| openstackgerrit | Tom Barron proposed openstack/manila-image-elements master: Update manila-service-image default to bionic https://review.openstack.org/623330 | 21:34 |
| tbarron | gouthamr: bswartz ^^ | 21:35 |
| openstackgerrit | Goutham Pacha Ravi proposed openstack/manila master: Test bionic manila service image https://review.openstack.org/623332 | 21:36 |
| gouthamr | tbarron: ty, ^ should work with our hook | 21:36 |
| bswartz | #zorilla | 21:37 |
| *** jgrosso has quit IRC | 21:47 | |
| *** jmlowe has joined #openstack-manila | 21:55 | |
| tbarron | nice, but you are supposed to be studying gouthamr | 22:00 |
| gouthamr | :D i'm here 24x7 too :P | 22:00 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!