*** whoami-rajat has joined #openstack-manila | 01:18 | |
*** baojg has joined #openstack-manila | 01:34 | |
*** lseki has joined #openstack-manila | 02:58 | |
*** enriquetaso has joined #openstack-manila | 03:54 | |
*** enriquetaso has quit IRC | 03:59 | |
*** enriquetaso has joined #openstack-manila | 04:26 | |
*** lseki has quit IRC | 05:08 | |
*** rwadolowski has joined #openstack-manila | 05:14 | |
*** enriquetaso has quit IRC | 05:44 | |
*** kopecmartin|off is now known as kopecmartin | 06:26 | |
rwadolowski | tbarron, are you alive ? :) | 08:52 |
---|---|---|
tbarron | rwadolowski: barely, just had my first sip of morning coffee. 'sup? | 09:43 |
rwadolowski | I am looking into: https://bugs.launchpad.net/manila/+bug/1806576 | 09:52 |
openstack | Launchpad bug 1806576 in Manila "CEPHFS/NFS driver fails with error code 108" [Undecided,Triaged] - Assigned to Tom Barron (tpb) | 09:52 |
rwadolowski | it's occured in my env | 09:52 |
rwadolowski | so if you have some time, we can try to debug-fix it | 09:53 |
tbarron | rwadolowski: are you running two ganesha servers against the same ceph cluster? | 09:55 |
rwadolowski | yep, exactly | 09:55 |
rwadolowski | i changed cephfs_volume_path_prefix | 09:55 |
rwadolowski | still same | 09:55 |
tbarron | do they use the same CepFS file system and metadata and data pools or separate filesystems? | 09:56 |
rwadolowski | my setup is using manila-share in container (kolla) and nfs-ganesha on this same host, but outside container | 09:56 |
rwadolowski | this same | 09:56 |
tbarron | i saw in the backlog your discussion w/ gouthamr regarding the dbus socket | 09:58 |
tbarron | do goth ganesha instances use the same socket path? | 09:59 |
tbarron | both | 09:59 |
rwadolowski | yes, so i decided to go outside docker with ganesha | 09:59 |
rwadolowski | http://paste.openstack.org/show/751221/ this is config for backend. On each compute there is different ip and path | 10:00 |
tbarron | outside docker, inside docker are both fine but I would worry about using the same socket path | 10:00 |
* tbarron looks | 10:00 | |
rwadolowski | I have each ganesha on different controler | 10:00 |
tbarron | ah | 10:01 |
tbarron | and you are using different cephfs_volume_pat_prefix for each backend I guess? | 10:02 |
rwadolowski | exactly, hostname of each controller, example /{{controller_hostname}} | 10:03 |
tbarron | looks like we have an incidental logging issue in that traceback in the bug: Share instance information in exception can not be written to db because it contains {} and it is not a dictionary | 10:05 |
tbarron | do you have that error too? | 10:05 |
rwadolowski | yes, i will paste my error, sec | 10:06 |
tbarron | with the main issue being a stat failure? | 10:06 |
rwadolowski | http://paste.openstack.org/show/751222/ | 10:07 |
rwadolowski | yes | 10:08 |
*** carloss has joined #openstack-manila | 10:09 | |
tbarron | and this works with the first ganesha bacj end even when two are configured? | 10:09 |
tbarron | back | 10:10 |
* tbarron broke right hand so typing is awkward | 10:10 | |
rwadolowski | yes, only one manila(ganesha) can create shares, the other one are throwing errors | 10:12 |
tbarron | so as you see the error is from the ceph volume library which is part of ceph rather than manila itself (not even in the manila driver code) | 10:12 |
tbarron | not trying to dump the prob elsewhere by saying that btw | 10:13 |
tbarron | it'a a python libraru though | 10:13 |
tbarron | if you would take a minute to put those pastebins as well as that for the working back end in the bug that will be great | 10:15 |
tbarron | also manila, ganesha, and ceph versions you are using | 10:15 |
rwadolowski | yeah, no problem | 10:15 |
rwadolowski | i will try to debug it deeper, maybe ceph-devs will help me :) | 10:16 |
tbarron | rwadolowski: https://github.com/ceph/ceph/blob/master/src/pybind/ceph_volume_client.py#L625 | 10:17 |
tbarron | it's catching ObjectNotFound but not error code 108 mebbe ... | 10:20 |
rwadolowski | probably this error we're facing https://github.com/ceph/ceph/blob/ae4f94c868e87e7a60d35b7832db7988fd60ab53/src/common/freebsd_errno.cc#L100 | 10:21 |
openstackgerrit | melissaml proposed openstack/manila master: Replace git.openstack.org URLs with opendev.org URLs https://review.opendev.org/658325 | 10:21 |
tbarron | rwadolowski: if you fix this you'll get your name on the ceph t-shirt for the next release :) | 10:22 |
tbarron | batrick: ^^^^^^^ | 10:23 |
tbarron | rwadolowski: he's on the west coast US so won't be looking now but may know right off | 10:23 |
tbarron | also gregsfortytwo ^^^^ | 10:24 |
tbarron | rwadolowski: you can also ask in freenode #ceph or on <ceph-users@lists.ceph.com> | 10:26 |
tbarron | rwadolowski: btw does the working back end also have ceph_volume_path_prefix configured? or is it using the default? | 10:32 |
tbarron | cephfs_volume_path_prefix | 10:32 |
*** lpetrut has joined #openstack-manila | 10:35 | |
rwadolowski | configured, each of backends have different path | 10:35 |
tbarron | rwadolowski: k | 10:37 |
tbarron | rwadolowski: and at driver init time there aren't errors showing up in manila share service log around this code (for the second back end?) - | 10:38 |
tbarron | https://github.com/openstack/manila/blob/master/manila/share/drivers/cephfs/driver.py#L204 | 10:39 |
*** baojg has quit IRC | 10:41 | |
* tbarron wonders if it's an auth/caps/client keys issue | 10:44 | |
rwadolowski | i can generate seperate keypairs and try to do it | 10:44 |
tbarron | rwadolowski: just thinking out loud ..., the ceph guys will know better | 10:45 |
tbarron | 108 is pretty obscure | 10:46 |
rwadolowski | no errors at startup | 10:55 |
tbarron | rwadolowski: ack | 11:08 |
rwadolowski | tbarron, got it | 11:17 |
* tbarron is all ears | 11:18 | |
rwadolowski | caps | 11:18 |
tbarron | do you need a separate client key for each back end? | 11:19 |
rwadolowski | different caps for each and it is working like a charm | 11:19 |
rwadolowski | for each manila :) | 11:19 |
tbarron | rwadolowski: yeah i guess that makes sense | 11:20 |
tbarron | nice work! | 11:21 |
rwadolowski | new question is if we can change something on the ceph side | 11:22 |
rwadolowski | so probably tools like kolla should generate caps for each endpoint, but this sounds more like a workaround | 11:23 |
tbarron | it may be a security feature, how does the cluster know both subtrees are being managed by the same admin? | 11:25 |
tbarron | rwadolowski: but it's a reasonable question for the ceph folks | 11:25 |
rwadolowski | good point | 11:25 |
rwadolowski | i am thinking about this: https://github.com/openstack/manila/blob/2609950d1e8d0b57c01fead23101ee7b419dbfb7/manila/share/drivers/cephfs/driver.py#L208 | 11:29 |
rwadolowski | what are cons of disabling eviction at all | 11:29 |
tbarron | rwadolowski: that's a Q for the ceph devs but i'm thinking it's a cleanup stale client sessions thing | 11:35 |
tbarron | http://docs.ceph.com/docs/mimic/cephfs/eviction/ | 11:35 |
*** rwadolowski has quit IRC | 11:38 | |
*** enriquetaso has joined #openstack-manila | 12:22 | |
*** enriquetaso has quit IRC | 12:23 | |
*** enriquetaso has joined #openstack-manila | 12:23 | |
*** lseki has joined #openstack-manila | 12:57 | |
*** rwadolowski has joined #openstack-manila | 13:08 | |
*** ianychoi_ is now known as ianychoi | 14:00 | |
*** lpetrut has quit IRC | 14:43 | |
*** lpetrut has joined #openstack-manila | 14:47 | |
*** lpetrut has quit IRC | 14:57 | |
*** josecastroleon has quit IRC | 15:32 | |
*** lpetrut has joined #openstack-manila | 16:06 | |
*** s0ru has joined #openstack-manila | 16:12 | |
openstackgerrit | Soledad Kuczala proposed openstack/manila master: Adding documentation for User Messages in Manila Documentation https://review.opendev.org/645010 | 16:23 |
*** s0ru has quit IRC | 16:47 | |
*** enriquetaso has quit IRC | 16:50 | |
*** enriquetaso has joined #openstack-manila | 16:55 | |
*** whoami-rajat has quit IRC | 16:58 | |
*** kopecmartin is now known as kopecmartin|off | 17:04 | |
*** lpetrut has quit IRC | 17:17 | |
batrick | rwadolowski: are you using freebsd? | 17:45 |
batrick | on Linux that's just ESHUTDOWN | 17:45 |
batrick | if you look at src/include/rados.h that's also EBLACKLIST (ceph specific errno) | 17:46 |
batrick | so it's possible your ganesha got blacklisted somehow | 17:46 |
batrick | these errors are happening during setup or after using it for some time? | 17:46 |
*** whoami-rajat has joined #openstack-manila | 18:09 | |
*** dviroel has quit IRC | 21:52 | |
*** carloss has quit IRC | 21:57 | |
openstackgerrit | Goutham Pacha Ravi proposed openstack/manila-specs master: Move unimplemented specifications to separate folder https://review.opendev.org/657894 | 22:20 |
openstackgerrit | Goutham Pacha Ravi proposed openstack/manila-specs master: Move unimplemented specifications to separate folder https://review.opendev.org/657894 | 22:20 |
*** whoami-rajat has quit IRC | 22:39 | |
*** enriquetaso has quit IRC | 23:39 | |
*** enriquetaso has joined #openstack-manila | 23:41 | |
*** lseki has quit IRC | 23:46 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!