Monday, 2020-07-27

« Sunday, 2020-07-26 Index Tuesday, 2020-07-28 »
openstackgerritGhanshyam Mann proposed openstack/manila-tempest-plugin master: DNM: testing gate on ubuntu focal  https://review.opendev.org/74313400:38
openstackgerritGhanshyam Mann proposed openstack/manila-ui master: DNM: testing gate on ubuntu focal  https://review.opendev.org/74313500:38
openstackgerritGhanshyam Mann proposed openstack/manila-image-elements master: DNM: testing gate on ubuntu focal  https://review.opendev.org/74313600:38
openstackgerritGhanshyam Mann proposed openstack/manila master: DNM: testing gate on ubuntu focal  https://review.opendev.org/74313700:38
openstackgerritGhanshyam Mann proposed openstack/manila-specs master: DNM: testing gate on ubuntu focal  https://review.opendev.org/74313800:41
openstackgerritGhanshyam Mann proposed openstack/manila-test-image master: DNM: testing gate on ubuntu focal  https://review.opendev.org/74313900:41
openstackgerritGhanshyam Mann proposed openstack/python-manilaclient master: DNM: testing gate on ubuntu focal  https://review.opendev.org/74314000:41
*** baojg has quit IRC03:39
*** baojg has joined #openstack-manila03:55
*** tosky has joined #openstack-manila06:42
*** lpetrut has joined #openstack-manila07:10
*** tosky has quit IRC08:03
*** tosky has joined #openstack-manila09:33
*** tkajinam has quit IRC10:10
*** lpetrut has quit IRC10:17
*** baojg has quit IRC10:19
*** baojg has joined #openstack-manila10:21
*** raildo has joined #openstack-manila11:30
*** jungleboyj has joined #openstack-manila11:42
*** baojg has quit IRC12:10
*** baojg has joined #openstack-manila12:11
*** baojg has quit IRC12:26
*** baojg has joined #openstack-manila12:35
*** baojg has quit IRC12:48
*** baojg has joined #openstack-manila12:51
*** eharney has joined #openstack-manila12:57
*** lpetrut has joined #openstack-manila14:14
*** jv_ has joined #openstack-manila14:18
*** baojg has quit IRC14:34
*** hoonetorg has quit IRC14:35
*** baojg has joined #openstack-manila14:36
*** hoonetorg has joined #openstack-manila14:49
*** lpetrut has quit IRC14:49
*** lpetrut has joined #openstack-manila15:42
*** raildo has quit IRC16:28
*** raildo has joined #openstack-manila16:31
*** vhari has quit IRC16:55
*** vhari has joined #openstack-manila17:00
*** baojg has quit IRC17:32
*** yan0s has joined #openstack-manila17:33
yan0sHello all I'm using manila with ganesha over cephfs and I'm faceing the following issue17:34
yan0s when I have an export mounted on a client and I turn off the ganesha server17:34
*** baojg has joined #openstack-manila17:34
yan0sthen I cannot mount the share again, even after I restart the ganesha server17:34
yan0salso if I try to delete this share from manila, the share hangs in "deleting" status17:35
yan0sI can see that the ceph volume still exists17:35
*** jv_ has quit IRC17:36
dviroeltbarron:  :)18:06
tbarronyan0s: dviroel i'll be online here in half an hour or so, many fires today18:07
yan0stbarron, thanks I'll ping you again then18:08
gouthamryan0s: are you using ganesha with its rados store enabled? (config opt: ganesha_rados_store_enable)18:12
yan0sI have used an automated way with juju so I'm not sure18:13
yan0sI think I'm using cephfs instead of rados18:13
yan0swhere  can I check for this option?18:14
gouthamryan0s: okay, you can find out by checking the manila.conf file, which is typically in /etc/manila where manila's running18:14
yan0sganesha_rados_store_enable = True18:15
yan0sI did some investigation with: dbus-send --system --print-reply --dest=org.ganesha.nfsd /org/ganesha/nfsd/ExportMgr org.ganesha.nfsd.exportmgr.ShowExports18:16
gouthamryan0s: ack, can you check whether you see any errors recovering in the ganesha logs?18:16
gouthamrare exports available? ^18:17
yan0sno they are not18:18
yan0sif a add a new "manila allow-access" command I can see the new export18:18
yan0sif I try to delete the old rule with "manila allow-deny" it gets in error state18:19
yan0sI can see some errors when in the ganesha logs18:19
yan0sload_fsal :NFS STARTUP :CRIT :Could not dlopen module:/usr/lib/x86_64-linux-gnu/ganesha/libfsalvfs.so Error:/usr/lib/x86_64-linux-gnu/ganesha/libfsalvfs.so: cannot open shared object file: No such file or directory18:19
yan0soad_fsal :NFS STARTUP :MAJ :Failed to load module (/usr/lib/x86_64-linux-gnu/ganesha/libfsalvfs.so) because: Can not access a needed shared library18:20
yan0smain :NFS STARTUP :WARN :No export entries found in configuration file !!!18:20
yan0sconfig_errs_to_log :CONFIG :CRIT :Config File (/etc/ganesha/ganesha.conf:27): Failed to load FSAL (VFS) because: Can not access a needed shared library18:23
yan0sconfig_errs_to_log :CONFIG :CRIT :Config File (/etc/ganesha/ganesha.conf:27): 1 validation errors in block FSAL18:23
yan0sconfig_errs_to_log :CONFIG :CRIT :Config File (/etc/ganesha/ganesha.conf:27): Errors processing block (FSAL)18:23
yan0sconfig_errs_to_log :CONFIG :CRIT :Config File (/etc/ganesha/ganesha.conf:11): 1 validation errors in block EXPORT18:23
yan0sconfig_errs_to_log :CONFIG :CRIT :Config File (/etc/ganesha/ganesha.conf:11): Errors processing block (EXPORT)18:23
yan0snfs_Init_svc :DISP :CRIT :Cannot acquire credentials for principal nfs18:23
yan0snfs_Init_admin_thread :NFS CB :EVENT :Admin thread initialized18:23
yan0snfs_rpc_cb_init_ccache :NFS STARTUP :EVENT :Callback creds directory (/var/run/ganesha) already exists18:23
yan0snfs_rpc_cb_init_ccache :NFS STARTUP :WARN :gssd_refresh_krb5_machine_credential failed (-1765328160:0)18:23
yan0s(commands to manage share access were "manila access-allow" and "manila access-deny")18:25
gouthamryan0s: the missing libfsalvfs.so may be unrelated to cephfs, we'll keep that aside for a moment18:26
gouthamryan0s: the config opt ganesha_rados_store_pool_name18:27
gouthamryan0s: has the name of the rados pool that ganesha must use to store its exports18:28
gouthamryan0s: (ganesha_rados_store_pool_name is in /etc/manila/manila.conf as well)18:28
gouthamryan0s: can you check if the exports are there with "rados -p <ganesha_rados_store_pool_name> get ganesha-export-index -"18:29
yan0slet me check18:29
yan0sI got all the share ids in the form:18:30
yan0s%url rados://manila-ganesha/ganesha-export-share-2620b680-1d6c-4de0-b3be-5b5b3e9f27b318:30
*** lpetrut has quit IRC18:33
*** tosky has quit IRC18:33
gouthamryan0s: great, so the rados store's alright, adding newer exports works, but, ganesha is somehow not able to recover from this rados pool - can you now check the ganesha configuration file? there must be a section like this: http://paste.openstack.org/show/796346/18:33
yan0sganesha.conf -> https://pastebin.com/gBminJ4A18:36
yan0salso there is a ceph.conf in /etc/ganesha/18:36
gouthamryan0s: that seems like the problem, ganesha's config is using a VFS FSAL, when it should be using CephFS18:36
yan0shttps://pastebin.com/40duuDXM18:37
* tbarron sends gouthamr coffee for all his wake up and debug stuff this "morning"18:37
yan0sthe last pastebin was /etc/ganesha/ceph.conf18:37
gouthamryan0s: that seems to be the problem, (you have VFS FSAL enabled, with some config that is unrelated to cephfs) - but, you'll need ganesha to be configured with the rados store manila is writing to18:41
gouthamryan0s: here's a sample from my environment: http://paste.openstack.org/show/796347/18:41
gouthamryan0s: it was generated using the template from ceph-ansible: https://github.com/ceph/ceph-ansible/blob/master/roles/ceph-nfs/templates/ganesha.conf.j218:42
gouthamryan0s: this is probably something juju must do, but isn't?18:42
yan0sok I will try this, thank you very much!18:43
yan0swhat is the expected behavior for the nfs client when the nfs-ganesha server restarts?18:44
yan0sshould the share be unmounted and then mounted back?18:45
yan0sor is it possible to become accessible when the ganesha server comes back?18:45
yan0salso is there any tutorial suggested for HA NFS setup with manila-ganesha?18:46
gouthamryan0s: tbarron might know better, but i think a hard mount will keep attempting to connect to the server, so if ganesha is up and serving exports, it will reconnect18:47
tbarrongouthamr: +118:51
gouthamryan0s: regarding "HA NFS setup with manila-ganesha": red hat deploys an active-passive setup with corosync/pacemaker when using manila18:52
tbarronyan0s: and upstream ganesha is now starting to support active-active deployment so18:52
tbarronwith that you should be able to run multiple instances behind an ha-proxy or other load balancer w/o all the pacemaker stuff18:53
tbarronand put the exports in rados so they can be seen by multiple instances18:53
* tbarron mentions in case canonical wants to up the ante on ganesha HA18:53
gouthamr+1, unfortunately i don't have docs/tutorials handy... except this one for experimenting with active-active config: https://jtlayton.wordpress.com/2018/12/10/deploying-an-active-active-nfs-cluster-over-cephfs/18:54
tbarronfor the pacemaker stuff it's all open source tripleo, but not an easy read18:54
gouthamryes, glusterfs pioneered in this area, and has some good docs/blog posts that could probably be repurposed for the ganesha bits18:54
yan0sjuju supports the pacemaker way so I'll probably follow that for now18:56
yan0sthank you very much for all the help18:57
*** baojg has quit IRC18:59
*** baojg has joined #openstack-manila19:00
gouthamryou're welcome yan0s19:06
gouthamr dviroel tbarron: ready to merge https://review.opendev.org/742987/ ?19:06
dviroelgouthamr: I was expecting to see manila-tempest-dsvm-lvm-em-only voting19:08
gouthamrwait,19:08
gouthamrthat's very weird - lemme look19:08
gouthamrasking #openstack-infra, thanks for noticing dviroel tbarron19:13
gouthamrsilly mistakes galore19:16
openstackgerritGoutham Pacha Ravi proposed openstack/manila stable/queens: [extended-maintenance-branches-only] Replace LVM job  https://review.opendev.org/74298719:18
gouthamrworks now, dviroel ^19:20
gouthamrmerge it when it passes, i guess :) there're some changes i'm hoping to take downstream19:21
*** andrebeltrami has joined #openstack-manila19:23
*** jv_ has joined #openstack-manila19:34
dviroelsure gouthamr, will do19:39
openstackgerritGoutham Pacha Ravi proposed openstack/manila master: [ci] Remove explicit compression of log files  https://review.opendev.org/74332819:44
dviroeltbarron: much faster20:52
tbarrondviroel: sorry, wasn't looking here, popped up in my email20:53
dviroeltbarron: np at all, tks for reviewing it :)20:57
openstackgerritGoutham Pacha Ravi proposed openstack/manila master: Zuulv3 native grenade job  https://review.opendev.org/74172721:01
openstackgerritGoutham Pacha Ravi proposed openstack/manila stable/train: Update LVM volume extend  https://review.opendev.org/74228921:08
*** raildo has quit IRC21:14
*** samueldmq has joined #openstack-manila21:20
*** tosky has joined #openstack-manila21:25
openstackgerritMerged openstack/manila stable/queens: [extended-maintenance-branches-only] Replace LVM job  https://review.opendev.org/74298722:17
openstackgerritGoutham Pacha Ravi proposed openstack/manila stable/train: Monkey patch original current_thread _active  https://review.opendev.org/74343122:37
openstackgerritMerged openstack/manila stable/stein: [NetApp] update set_preferred_dc for ontapi 1.150  https://review.opendev.org/74208622:40
*** tkajinam has joined #openstack-manila22:52
*** hoonetorg has quit IRC23:06
*** hoonetorg has joined #openstack-manila23:19
*** tosky has quit IRC23:25
*** andrebeltrami has quit IRC23:33
« Sunday, 2020-07-26 Index Tuesday, 2020-07-28 »

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!