15:00:06 <bswartz> #startmeeting manila
15:00:07 <openstack> Meeting started Thu Sep 28 15:00:06 2017 UTC and is due to finish in 60 minutes.  The chair is bswartz. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:08 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:00:10 <openstack> The meeting name has been set to 'manila'
15:00:14 <bswartz> hello all
15:00:17 <dustins> \o
15:00:22 <toabctl> hey
15:00:24 <ganso> hello
15:00:25 <zhongjun> hi
15:00:46 <tbarron> hi
15:00:46 <vkmc> o/
15:01:20 <bswartz> vponomaryov gouthamr cknight xyang markstur: courtesy ping
15:01:29 <gouthamr> hello o/
15:01:40 <bswartz> #topic announcements
15:02:20 <bswartz> This week is R-22 -- milestone 1 is R-19 -- 3 weeks away
15:02:27 <bswartz> #link https://releases.openstack.org/queens/schedule.html
15:02:33 <bswartz> we have a bunch of specs that need reviewing
15:03:09 <bswartz> last week we held a virtual PTG, and it was well attended and we used the whole 2 days
15:03:28 <gouthamr> that's also our spec freeze deadline
15:03:29 <bswartz> we're becoming as bad as those cinder guys :-p
15:03:36 <gouthamr> ouch :(
15:03:52 <bswartz> I jest
15:04:03 <markstur> o/
15:04:21 <bswartz> #agenda https://wiki.openstack.org/wiki/Manila/Meetings
15:04:36 <bswartz> #topic Discuss and elicit Feedback for PTG
15:04:46 <bswartz> we decided to take notes right on the topics etherpad itself
15:04:51 <bswartz> #link https://etherpad.openstack.org/p/manila-ptg-queens
15:05:08 <bswartz> first of all, we wrote down a LOT of #action items on the etherpad
15:05:30 <bswartz> check for actions with your name on them and make sure to follow up
15:05:44 <tbarron> maybe cross them out when done
15:05:58 <bswartz> tbarron: that's a good idea
15:06:10 <bswartz> then I can look for uncrossed out action and hound people about them
15:06:53 <bswartz> some action don't have names so those will need owners if they didn't already get done
15:07:09 <bswartz> but I don't both people about those today -- I'm still working through my own list
15:07:35 <bswartz> I wanted to use this time to find out if there were any questions about stuff we covered last week
15:07:47 <bswartz> or any questions from those who couldn't attend
15:07:56 <bswartz> or feedback about the format or content of the PTG
15:08:23 <bswartz> on the subject of the format, we talked about the downsides of it being virtual at the end last thursday
15:08:29 <zhongjun> I have one feedback
15:08:41 <bswartz> zhongjun: go ahead
15:08:42 <dustins> I think that sitting on the phone for two days got a little old, but it's better than not participating at all
15:09:21 <zhongjun> bswartz: about reset log level: Talk to the Cinder guys to get more information about this in order to reach an agreement
15:09:34 <bswartz> dustins: yes I worry that as these events get longer and longer, virtual is a painful format
15:09:48 <toabctl> dustins, +1 (but I only attended half of the time due to personal schedule conflicts)
15:09:48 <zhongjun> bswartz: SIGHUP for Cinder will restart the whole service Which is a problem because we do some operations in the data path
15:09:57 <amito-infinidat> o/
15:10:19 <dustins> And there were several times where I wrote down "it would have been nice to go into a hallway and talk with $PROJECT person about $TOPIC"
15:10:20 <bswartz> zhongjun: I thought cinder had implemented handling for SIGHUP
15:10:53 <zhongjun> bswartz: And for SIGHUP you have to go node by node changing it and restarting those services with SIGHUP
15:10:53 <zhongjun> Whereas with the API you can just make the request from outside the cloud as long as you have admin privileges
15:11:12 <bswartz> yes for the Rocky PTG I think we need to seriously consider going back to the face-to-face format, even if it requires unnatural scheduling gymnastics
15:11:34 <zhongjun> bswartz: yeah, they think SIGHUP is not enough
15:11:57 <vkmc> yeah, I think these downsides were discussed when we voted on the formats
15:12:15 <ganso> zhongjun: we could not reach an agreement on what to do for dynamic log level
15:12:16 <zhongjun> bswartz: yeah, they think SIGHUP can not satisfy their requiement
15:12:28 <ganso> zhongjun: there were advantages and disadvantages for both approaches
15:12:34 <vkmc> face-to-face, although a bit harder to plan, feels less stressful
15:12:45 <bswartz> zhongjun: I know there's a discussion about the ease of use of the 2 approaches -- my point is that it's not hard to make SIGHUP do the right thing, and I thought that at least SIGHUP wasn't handled the same as SIGTERM
15:13:10 <tbarron> zhongjun: +1 for a convo with cinder and trying to drive the two projects to give a consistent user experience if we can
15:13:26 <bswartz> so the one thing I'll mention about face-to-face is that, although it's not accounced yet, it seems likely that Rocky PTG will be held in Dublin
15:13:37 <tbarron> face-to-beer
15:13:43 <amito-infinidat> tbarron: lol
15:13:51 <toabctl> hehe
15:13:51 <bswartz> so travel costs may be higher for US-based people than for the last 2 PTGs
15:13:56 <ganso> bswartz, vkmc, dustins: we had discussed internally the possibility of maybe having the manila PTG for the Tuesday and Friday of the PTG in order to have less conflict with cross projects and Cinder, although we are not sure if that will be possible
15:14:16 <bswartz> yeah it's something to be worked out
15:14:27 <zhongjun> bswartz:Such as：create volume from image uses the data path
15:14:28 <zhongjun> it would give it something like 60 seconds to complete for create vol from image it may be enough  but not for migrations that are using dd
15:14:32 <ganso> tbarron: lol
15:14:43 <dustins> ganso: Hmm, not a bad idea, no one ever said the days we chat have to be consecutive :)
15:14:44 <bswartz> I expressed my opinion that the original design summit + midcycle format was preferable to the current system, but few in the community share those feelings
15:15:13 <bswartz> so we should plan around the current conference+PTG split as the reality going forward
15:15:29 <zhongjun> bswartz:  There are feedback from cinder
15:15:55 <zhongjun> bswartz: SIGHUP do the some of the right thing, not all of right thing
15:15:57 <gouthamr> though, if you think like me that the PTG and Forum can be back-to-back, please email blast the good folks at the Foundation :D
15:16:11 <bswartz> zhongjun: yes that matches my understanding
15:16:14 <tbarron> we could probably schedule some joint manila-cinder meetings like they do nova-cinder for overlapping issues like the dynamic config thing
15:16:36 <bswartz> SIGHUP *can* do the right thing, but people find it insufficiently convenient to use
15:16:57 <zhongjun> tbarron : +1 manila-cinder meeting
15:16:59 <bswartz> tbarron: good point
15:17:10 <tbarron> bswartz: my point is that users should get the same thing from cinder and manila on this, not atm on one side of the fence or another
15:17:27 <dustins> tbarron: +1, there's a lot of overlap with what we do and what they do. It'd be nice to be on the same page
15:17:29 <bswartz> that would save time on the parts of hte infra and docs folks so they wouldn't need to say the same things twice
15:18:08 <zhongjun> bswartz: SIGHUP not just  insufficiently convenient
15:18:12 <bswartz> that's something to work out between jungleboyj and myself as the Rocky PTG planning gets underway
15:18:29 <amito-infinidat> tbarron: + 1
15:18:32 <jungleboyj> What?
15:18:44 <tbarron> oh, nm then :)
15:18:58 * tbarron yanks jungleboyj's chain
15:19:46 <bswartz> anyways everyone is now focused on Sydney and the next PTG is far away -- just wanted to mention that after attending both PTGs myself, I'm back to leaning in favor of face to face PTG next time, and we'll need to figure out the specifics later on during queens
15:20:06 <jungleboyj> It would be good to have a Cinder/Manila meeting.
15:20:24 <bswartz> since zhongjun has a specific concern, let's switch topics and try to resolve it here
15:20:32 <bswartz> #topic changing log level dynamically
15:21:00 <zhongjun> thanks
15:21:11 <bswartz> zhongjun: are you claiming the there's a technical impediment to changing log levels in Cinder using only SIGHUP?
15:21:19 <jungleboyj> Getting dynamic reconfig working would be good.  Doesn't work for Cinder right now, not sure how well it works with Manila
15:21:39 <jungleboyj> bswartz:  No, Gorka actually implemented another mechanism for it.
15:22:09 <bswartz> jungleboyj: yes we've heard -- I'm trying to figure out if the reason is technical or ergonomical
15:22:44 <bswartz> all of the arguments i've heard so far have been of the flavor: "it's too hard to actually modify config files and send signals to the cinder processes"
15:23:30 <bswartz> and I consider that to be an ergonomic argument
15:23:52 <jungleboyj> bswartz:  That was where things kept getting stuck.  So, that was why Gorka made the dynamic log level change to enable it without having to tackle making everything dynamic.
15:23:53 <zhongjun> bswartz: there is a technical  impediment to changing log levels in Cinder using only SIGHUP
15:24:12 <bswartz> my worry is the slippery slope of allowing everything to be reconfigured through APIs, even though the config file could be used
15:24:19 <zhongjun> bswartz:  It just not work well now
15:24:50 <bswartz> zhongjun: can you describe what doesn't work well, or why it doesn't work well?
15:24:51 <jungleboyj> bswartz:  Understood.
15:25:01 <jungleboyj> We made logging a special case for better or worse.
15:25:38 <tbarron> zhongjun said earlier that it would break data path operations, right?
15:25:53 <bswartz> tbarron: only if SIGHUP == SIGTERM
15:25:59 <zhongjun> bswartz: SIGHUP for Cinder will restart the whole service Which is a problem because we do some operations in the data path
15:26:17 <bswartz> that seems simply like poor handling of SIGHUP
15:26:19 <ganso> we must not restart the service
15:26:26 <tbarron> zhongjun so that was what I was going to ask, isn't that SIGTEMRM behavior?
15:26:30 <bswartz> signal handling in Unix is not hard to do right
15:26:32 <tbarron> SIGTERM
15:27:26 <zhongjun> tbarron: Just seen SIGHUP service_pid
15:27:27 <bswartz> the challenge we have is enumerating which options are safe to change without a restart, and then only updating those options when a SIGHUP is received
15:27:32 <zhongjun> tbarron: Just seed SIGHUP service_pid
15:27:40 <zhongjun> s/seed/send
15:28:05 <bswartz> zhongjun: by default, SIGHUP is treated like SIGTERM -- you have to write code to make a service do the right thing
15:28:15 <zhongjun> bswartz: it will restart in internal service
15:28:15 <bswartz> olso.service has that code AIUI
15:28:44 <bswartz> I don't know if cinder uses it yet
15:29:03 <bswartz> someone needs to investigate oslo service signal handling and whether it can be used to anything useful at all
15:29:18 <bswartz> I've been presuming that it would work for at least some subset of our options, and log levels seems like the most obvious application
15:30:36 <bswartz> it seems that we still don't have a complete picture on this topic
15:31:29 <bswartz> and it seems that cinder chose the path it did just to be expedient -- presumably because the feature was high priority for someone
15:32:30 <zhongjun> bswartz: How to make the request from outside the cloud
15:32:41 <zhongjun> bswartz:  use SIGHUP
15:33:00 <bswartz> that's a deployer concern
15:33:22 <tbarron> bswartz: well, all manila deployments have to be deployed
15:33:23 <bswartz> the same way you edit any config option from outside the cloud
15:33:37 <tbarron> bswartz: that's fairer
15:34:18 <bswartz> I consider logging to be a strictly deployer-oriented facility
15:34:47 <bswartz> if there are use cases for some log messages being available to people who can't access config files, then maybe we need a more elaborate logging facility
15:35:47 <bswartz> I could be convinced that logging is special somehow -- I just haven't heard a good reason why yet
15:36:09 <tbarron> it's probably worth thinking about real world use cases with support desks and such
15:36:18 <bswartz> tbarron: exactly
15:36:53 <bswartz> I'm curious to know who these persons are who are consuming log messages but who aren't empowered to change configurations
15:37:24 <tbarron> so in our case a standard thing would be 1) turn on debug; 2) repeat the steps to cause the problem; 3) generate sosreport; 4) turn off debug
15:37:36 <bswartz> perhaps the best way to service them would be to turn on debug logging always, but to direct logs to a log processing tool instead of the normal files, and to have the log processing tool filter debug logs normally
15:37:53 <zhongjun> bswartz: I put our real use cases in PTG before
15:39:55 <bswartz> zhongjun: so the argument you're making is that there's no technical reason that log levels couldn't be updated using a config file change and a SIGHUP -- but that there is a special use case around log consumption that requires non-administrators to be able to temporarily elevate the log level
15:40:25 <bswartz> the key here would be that there's something special about log level that doesn't apply to any other config option
15:40:40 <zhongjun> bswartz:  turn on debug logging always could consume many storage
15:41:00 <bswartz> because I would be upset if we implement a special API for changing log levels, and then someone wanted to do the same for some other config option next release
15:41:20 <bswartz> zhongjun: that's only if the logs are not being filtered
15:41:54 <bswartz> it's possible to filter logs before they get written to disk
15:42:21 <tbarron> I think any production cloud worth its salt should not get disk filled by debug ON in logging
15:42:31 <bswartz> my suggestion was to consider doing the filtering somewhere outside manila -- in which case the use case could be solved with Pike or earlier in additional to queens
15:42:36 <tbarron> a developer cloud is another thing
15:43:30 <zhongjun> bswartz: We usually use original openstack project,  it better to find a good way in openstack itself
15:43:41 <bswartz> I'm just imaging that 60 new APIs are added to 60 openstack projects to control dynamic logging in each project...
15:43:54 <bswartz> when this is really a common problem that should have a common soluition
15:44:07 <bswartz> solution*
15:44:10 <tbarron> bswartz: I think there are objective reasons not to allow REST interface to config in general.
15:44:24 <tbarron> There aren't the same real world use cases to justify it.
15:44:31 <tbarron> logging is special.
15:44:57 <tbarron> you asked about trace logging and that is a grey case but there aren't many grey cases I think.
15:45:31 <bswartz> yeah there are vendor specific log options (at least for NetApp) that the same logic could be applied to
15:45:32 <zhongjun> bswartz:  logging is really special, there isn't another one I could see .
15:45:45 <tbarron> if someone says they want a rest API to change back end connection details then I don't see the corresponding use case.
15:45:56 <vkmc> I have been involved in a discussion like this in some other project I worked on... the main issue is 1. security concerns 2. use cases 3. this doesn't fit the idea of "as a service", end users shouldn't have access to logs
15:45:57 <bswartz> the NetApp debugging procedure is to enable trace logging in our driver while reproducing strange problems
15:46:13 <tbarron> netapp has trace-flags, arguably like logging, but are there other grey cases?
15:46:23 <tbarron> And are any of them not back end specific?
15:46:35 <bswartz> tbarron: we're adding a new config option to make trace logging even more granular
15:46:41 * bswartz looks to gouthamr for the patch
15:46:54 <tbarron> That's a criterion one could apply right there: no back end specific config should have a REST api to toggle.
15:47:03 * gouthamr wakes up
15:47:05 <gouthamr> #LINK: https://review.openstack.org/#/c/505437/
15:47:08 <tbarron> I could get behind that rule.
15:47:19 <zhongjun> bswartz: It's really so hard to change log level in our real client envoriment (original project)
15:47:47 <tbarron> Then I'd ask: are there any other cases to worry about?
15:48:08 <bswartz> dunno if any of you have read the 12-factor app
15:48:17 <bswartz> #link https://12factor.net/
15:48:25 <bswartz> read number 11
15:48:26 <toabctl> was that topic discussed with the oslo.config guys maybe? if specific options need some sort of rest api, maybe there is a generic way which can be implemented there and then hooked into the different projects?
15:49:10 <bswartz> I don't consider 12factor.net gospel by any means, but there is wisdom in some of it, and I happen to agree with 11
15:50:14 <bswartz> anyways we're going to run out of time and there's another topic for today
15:50:27 <bswartz> so we'll need to table this discussion
15:50:35 <bswartz> #topic Bug Tsar/Czar
15:50:42 <bswartz> dustins: you still here?
15:50:47 <dustins> Yessir!
15:51:15 <dustins> So at the PTG there was talk of the need of a Bug Tsar
15:51:20 <bswartz> toabctl: your suggestion sounds more appealing than what cinder did
15:51:49 <dustins> And, to make a long story short, I'm volunteering my services to it
15:51:50 <bswartz> dustins: I thought it was czar
15:51:56 <dustins> Assuming that everyone's onboard
15:52:05 <dustins> bswartz: I think they're interchangable?
15:52:06 <bswartz> but if you want to job, you can spell it however you want!
15:52:11 * dustins shrugs
15:52:21 <bswartz> awesome, thanks dustins
15:52:23 <dustins> Bug...Person?
15:52:32 <gouthamr> BuggerKing
15:52:52 <ganso> lol
15:52:54 <dustins> bswartz: of course, I'll likely add a section to the Manila meetings to discuss bug status/bug folks that are assigned bugs
15:52:56 <bswartz> Bug Emperor
15:52:59 <dustins> gouthamr: no
15:53:01 <dustins> bswartz: yes
15:53:26 <tbarron> dustins: job comes with title but no salary bump
15:53:27 <gouthamr> :D
15:53:31 <bswartz> dustins: that would be good -- to review the open bug list weekly or something close to weekly
15:53:53 <gouthamr> lol
15:53:56 <gouthamr> dustins: you're awesome for doing it though
15:54:09 <dustins> bswartz: I plan on taking a glance daily and deep dive at least weekly and then reporting to the group here
15:54:34 <bswartz> dustins: do we need to schedule any kind of extra meetings for you to sync up with me or other community members about bugs?
15:54:47 <bswartz> or do you want to handle all the communications in this meeting?
15:55:07 <dustins> bswartz: Not yet, but I would like to have a list of maintainers so that driver-specific bugs I can assign to the proper owners
15:55:14 <dustins> If I need it, I'll ask for it
15:55:23 <bswartz> okay
15:55:30 <bswartz> dustins: I'll share my maintainer list with you
15:55:37 <dustins> bswartz: Awesome, thanks
15:55:44 <bswartz> it's most likely out of date, but since nothing official exists, I keep a spreadsheet
15:55:53 <gouthamr> dustins: there's probably an inaccurate driverlog for manila..
15:56:06 <dustins> Hmm, we should make that accurate...
15:56:13 <bswartz> driverlog might be a better data source
15:56:24 <bswartz> we could synchronize the 2
15:56:44 <bswartz> I hate having my own private information source -- I would like it to be public and maintained by other people
15:56:48 <gouthamr> yep, https://github.com/openstack/driverlog/blob/master/etc/default_data.json - pushing the NTAP update today
15:56:59 <dustins> Sounds like a job for...etherpad?
15:57:07 <bswartz> not etherpad
15:57:17 <bswartz> wiki or git
15:57:23 <dustins> +1
15:57:28 <bswartz> driverlog seems to prefer git
15:57:48 <tbarron> dustins: move quick, give netapp bugs to cknight
15:58:03 <dustins> tbarron: I'm sure he won't mind :)
15:58:19 <zhongjun> tbarron: :)
15:58:26 <bswartz> wow -- a 5000 line json file was the best idea to store this data?
15:58:38 <dustins> Said no one ever
15:58:45 <bswartz> smh
15:58:51 <tbarron> bswartz: we could have made a separate repo for each vendor :)
15:58:56 <bswartz> nooooooo
15:58:59 <tbarron> heh
15:59:04 <bswartz> how about directories with yaml files
15:59:13 <tbarron> happy medium
15:59:22 <gouthamr> timecheck, yaml vs json can go forever
15:59:25 <bswartz> oh wel
15:59:25 <gouthamr> #vote yaml
15:59:34 <bswartz> that's not true
15:59:38 <bswartz> yaml wins hands down
15:59:40 <tbarron> #vote database
15:59:53 <gouthamr> #vote manila
15:59:53 <bswartz> json is for computer consumption, not human consumption
16:00:01 <bswartz> but you're right about the time
16:00:14 <bswartz> thanks everyone
16:00:24 <dustins> thanks, bswartz!
16:00:26 <bswartz> and thanks dustins for volunteering for bug czar
16:00:32 <gouthamr> \o thanks dustins
16:00:35 <dustins> My pleasure
16:00:47 <bswartz> there is a wiki where you should probably make that official
16:00:55 <dustins> Somewhere, probably
16:00:55 <bswartz> I'll find it
16:01:01 <bswartz> #endmeeting