20:00:30 <johnsom> #startmeeting Octavia
20:00:31 <openstack> Meeting started Wed Jun 14 20:00:30 2017 UTC and is due to finish in 60 minutes.  The chair is johnsom. Information about MeetBot at http://wiki.debian.org/MeetBot.
20:00:32 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
20:00:35 <openstack> The meeting name has been set to 'octavia'
20:00:42 <johnsom> Hi folks
20:00:54 <rm_work> o/
20:00:55 <jniesz> Hi
20:01:12 <johnsom> #topic Announcements
20:01:20 <johnsom> We are heading towards feature freeze Pike-3 July 24th
20:01:27 <johnsom> #link https://releases.openstack.org/pike/schedule.html
20:01:27 <xgerman_> o/
20:01:50 <johnsom> Please keep that in mind.  We are in the last stretch of Pike.
20:02:14 <nmagnezi> p/
20:02:16 <nmagnezi> o/
20:02:25 <johnsom> Also of note, the TC ran an analysis on project activity
20:02:30 <johnsom> https://www.irccloud.com/pastebin/gy1jZ2Xg/
20:02:42 <johnsom> #link https://etherpad.openstack.org/p/9YSc50YgUb
20:02:47 <johnsom> The link is the full list
20:03:28 <johnsom> It does show a slow down in contributions to octavia.
20:03:39 <nmagnezi> OSIC.. :<
20:03:49 <johnsom> Though it's not as bad as some other projects that are in the 60% drop rang
20:04:10 <johnsom> Some OSIC, some other contributors moving off working on OpenStack
20:04:21 <johnsom> Just thought I would share the numbers
20:04:28 <xgerman_> yeah, I think OpenStack is really hurting for devs
20:04:30 <rm_work> really?
20:04:34 <rm_work> we are + on contributors
20:04:43 <johnsom> Yep
20:04:51 <johnsom> By one
20:04:55 <rm_work> and yeah only -8% commits merged, but
20:04:56 <rm_work> that's actually surprising
20:04:57 <johnsom> Y-o-Y
20:05:03 <rm_work> because we merged SOOOO much stuff
20:05:44 <johnsom> Yeah, the numbers are probably a bit skewed due to the point in time the project is at.  Early projects are going to have high commit rates
20:06:35 <johnsom> Also of note, we released 1.0 version of the openstack client plugin package for Octavia v2 API.  Nice job!
20:07:01 <johnsom> Stats, status, and quota are left to do there, but otherwise it should be pretty complete
20:07:20 <nmagnezi> JudeC, well done!
20:07:32 <johnsom> #link https://pypi.python.org/pypi/python-octaviaclient/1.0.0
20:07:43 <JudeC> :)
20:07:58 <johnsom> Yeah, super handy to have
20:08:10 <jniesz> nice
20:08:12 <johnsom> Any other announcements?
20:08:18 <xgerman_> +1
20:08:41 <johnsom> #topic Brief progress reports / bugs needing review
20:08:56 <johnsom> I want to highlight a couple of open specs:
20:09:02 <johnsom> l3-active-active spec
20:09:09 <johnsom> #link https://review.openstack.org/453005
20:09:15 <johnsom> flavors spec
20:09:21 <johnsom> #link https://review.openstack.org/392485
20:09:34 <xgerman_> reviewed both
20:09:42 <xgerman_> have some q in OpenQ
20:09:48 <johnsom> These both could use more reviews (credit to xgerman_ for his recent review comments)
20:10:02 <m-greene> reviewing flavors today
20:10:14 <jniesz> yes, thanks for the input
20:10:29 <johnsom> As for progress updates, I jumped into finishing the RBAC implementation for the Octavia v2 API
20:10:55 <rm_work> i've been making a personal push towards better work/life balance recently... so, i haven't been able to get to those big specs yet :(
20:11:04 <johnsom> I am making decent progress.  I do have a discussion item for later in the meeting on the policies.
20:11:44 <johnsom> I will also be getting back to finishing the api-ref.  I have the l7 apis and quotas to finish
20:12:04 <johnsom> #link https://developer.openstack.org/api-ref/load-balancer/
20:12:15 <johnsom> In case you have not seen the new api reference docs
20:12:33 <johnsom> Any other updates?
20:13:01 <johnsom> #topic Discuss RBAC policies
20:13:08 <xgerman_> I think we are in a spot where we can start reviewing the octavia-proxy in lbaasv2
20:13:32 <xgerman_> (if we load it on top jude’s filter stuff)
20:13:38 <johnsom> Ok, quick question for the team.  Most projects have simple "admin or owner" policies defined.
20:13:59 <johnsom> xgerman_ Cool, need to find time to check that out and review the filters again
20:14:14 <xgerman_> thx
20:14:21 <johnsom> Nova team has proposed making this a bit more granular
20:14:27 <johnsom> #link https://review.openstack.org/#/c/427872/19/specs/pike/approved/additional-default-policy-roles.rst
20:14:45 <johnsom> Interesting parts start around line 144
20:15:16 <johnsom> Basically instead of just Admin/not admin they are proposing a "observer" or read-only policy level
20:15:57 <johnsom> It would also mean the project would have project specific roles that could be loaded up into keystone.
20:16:18 <johnsom> "load-balancer_observer" for example.
20:16:25 <johnsom> Thoughts on this?
20:17:45 <johnsom> Or, should we just follow most other projects and have simple "admin_or_owner" style rules
20:17:47 <johnsom> ?
20:18:11 <johnsom> #link http://docs-draft.openstack.org/72/472872/7/check/gate-octavia-docs-ubuntu-xenial/68c5715//doc/build/html/main/policy.html
20:18:51 <johnsom> That is part of my WIP patch.  I have only done LBs so far
20:19:13 <rm_work> johnsom: i think the observer stuff is standard?
20:19:16 <rm_work> i have seen it a lot
20:19:24 <rm_work> at least in barbican
20:19:29 <rm_work> but maybe barbican's model is non-standard
20:19:48 <rm_work> https://github.com/openstack/barbican/blob/master/etc/barbican/policy.json
20:19:54 <johnsom> Right, this is OpenStack, standard has 50 meanings...  Grin
20:20:03 <rm_work> lol
20:21:15 <johnsom> No comments?
20:22:06 <xgerman_> I think I like what nova is doing
20:22:38 <johnsom> Cool, thanks for the input
20:22:46 <jniesz> I think having read only makes sense
20:22:58 <jniesz> if you want to only give visibility and not any other access
20:23:35 <johnsom> nmagnezi Any comments from a packager?
20:24:21 <nmagnezi> johnsom, mmm.. i'll have to look more about this before I can comment. sorry :\
20:24:27 <johnsom> Ok
20:25:15 <johnsom> Well, I might head down the path that nova is proposing because I kind of like it too.  If you have comments please put them on the patch (sooner would be nice, grin).
20:25:36 <johnsom> #link https://review.openstack.org/#/c/472872/
20:25:45 <johnsom> I think it gives more flexibility to the operator
20:26:05 <johnsom> Which I am always of fan of empowering the operators
20:26:23 <johnsom> #topic jsonschema blueprint
20:26:31 <johnsom> #link https://blueprints.launchpad.net/octavia/+spec/jsonschema-api-validation
20:26:43 <johnsom> We brought this up last week
20:26:55 <johnsom> It is a proposal for an alternative to WSME.
20:27:13 <johnsom> Any new comments?  Did people have time to investigate?
20:28:05 <rm_work> i still like voluptuous
20:28:22 <rm_work> but don't have any other comments besides "this is suuuuper low priority"
20:28:37 <johnsom> Can you comment to that blueprint you feelings?
20:28:50 <johnsom> Yeah, I am kind of in the "Is this necessary now" camp
20:28:54 <rm_work> there's definitely issues with the WSME stuff (especially that I ran into around single-create) but it's already been worked around, so
20:29:12 <rm_work> if someone wants to contribute that work, and would otherwise not work on octavia at all, then go for it :P
20:29:22 <rm_work> but if you're going to do SOMETHING on octavia, please don't do that
20:29:22 <johnsom> Yeah, it also has some strangeness with exception handling, but it seems to *work*
20:29:57 <johnsom> Ok, let's capture comments there and move on
20:30:12 <johnsom> #topic Proposal to move Octavia to storyboard
20:30:19 <johnsom> #link https://storyboard.openstack.org/
20:30:28 <johnsom> Another carry over from last week.
20:30:53 <johnsom> I looked at this a bit.  Folks that have used it seem to think there are still some issues to be worked out.
20:31:02 <nmagnezi> i asked a team mate who work on neutron about this
20:31:13 <nmagnezi> he was not too familiar with it
20:31:31 <nmagnezi> just replied that he does not understand the justification
20:31:48 <johnsom> I think we should delay this to Queens-1 at the earliest as I think we have more important things to work on and I don't want to disrupt the Pike release.
20:31:50 <nmagnezi> so.. not a lot of info there..
20:32:08 <nmagnezi> yeah, sounds like a good call
20:32:22 <johnsom> It's a migration, unknown gotchas, and learning a new tool, so....
20:32:38 <nmagnezi> and overall it would be nice to see feedback from projects who already migrated.. see how is it working for them
20:33:08 <johnsom> #link https://storyboard.openstack.org/#!/project/list
20:33:11 <johnsom> It's a short list
20:33:34 <johnsom> Oh, nevermind, the scroller is in the corner.
20:33:41 <johnsom> It must be all projects loaded
20:34:47 <johnsom> The one plus is I guess it doesn't go out to lunch like launchpad does with bugs that have a lot of people on them.
20:34:56 <johnsom> That is annoying, but rare
20:35:17 <nmagnezi> i'm always surprised to discover new names for projects in openstack.. seem like ppl start a new one every day :)
20:35:43 <johnsom> Yeah, or they are long forgotten projects...
20:36:02 <johnsom> Ok, I will reply that we aren't ready to make that move in Pike
20:36:07 <johnsom> #topic Open Discussion
20:36:07 <nmagnezi> ack.
20:36:13 <johnsom> Other items today?
20:36:42 <johnsom> the uwsgi stuff is causing me some headaches with devstack.
20:36:46 <xgerman_> oh yeah, I wanted to talk flavor
20:37:12 <johnsom> I built a fresh VM and the openrc was not configured right for the new keystone URL path.
20:37:13 <cpuga> o/
20:37:29 <johnsom> I also found that last night uwsgi was running 100% cpu
20:37:34 <johnsom> So, yeah, pains
20:37:42 <johnsom> xgerman_ Go for it
20:37:56 <xgerman_> https://review.openstack.org/#/c/392485/8/specs/version1/flavors.rst
20:38:22 <rm_work> oh, logging
20:38:24 <xgerman_> so it is propsoed to sye some complex database scheme to store keys and values which are set on flavors
20:38:27 <rm_work> i wanted to discuss logging
20:38:51 <cpuga> xgerman: thx for the review
20:38:53 <johnsom> rm_work ok, that will be next
20:38:54 <rm_work> also -- aren't most of these done: https://bugs.launchpad.net/octavia?field.searchtext=Align+octavia+api&search=Search&field.status%3Alist=NEW&field.status%3Alist=INCOMPLETE_WITH_RESPONSE&field.status%3Alist=INCOMPLETE_WITHOUT_RESPONSE&field.status%3Alist=CONFIRMED&field.status%3Alist=TRIAGED&field.status%3Alist=INPROGRESS&field.status%3Alist=FIXCOMMITTED&field.assignee=&field.bug_reporter=&field.omit_dupes=on&field.has_patch=&
20:38:56 <rm_work> field.has_no_package=
20:38:57 <rm_work> ugh
20:38:58 <rm_work> long link
20:39:01 <xgerman_> I am a bot worried that is overengineered
20:39:48 <cpuga> we could reduce the number of tables, it all depends on what experience we would like to provide to the operator who will be creating the flavors
20:39:52 <johnsom> rm_work Yeah, I need to go through those and see if we met the criteria to close out
20:40:27 <xgerman_> cpuga our original design was just some metadata text field to put the values in
20:41:00 <xgerman_> but maybe we should defer until everybody had. a look?
20:41:22 <m-greene> I would like to see flavors be a self-contained standalone container for key/values, and be potentially FK’ed from any primary lbaas resource, such as listener, pool, healthmonitor.  Not just the loadbalancer.
20:41:47 <m-greene> as long as the design doesn’t prohibit that as a future enhancement..
20:42:22 <cpuga> the flavor is a framework which could in theory be applied to any compoent
20:42:29 <xgerman_> +1
20:42:39 <m-greene> great
20:43:04 <johnsom> Hmm, there is a good point here though, with a more structured table we could expose some "discover-ability" about the flavor as opposed to making the operator document that outside of the implementation.
20:43:13 <m-greene> except for the lb_topology_name field, which seems to be lb specific..
20:43:38 <cpuga> I felt the concept of lb_topology would apply to all providers at a minimum with "single" topology support.
20:43:44 <cpuga> what do you guys think?
20:43:59 <m-greene> if the flavor was a json schema, it would provide the documentation, and allow ocatvia to perform generic schema validation
20:44:53 <m-greene> for best user-experience through CLI commands (instant error-checking, instead of delayed FAIL status whenever the provider driver gets around to ACK’ing the request)
20:44:58 <johnsom> I think to some degree it is up to the provider to interpret the contents of the flavor.
20:45:34 <m-greene> I read something in the new spec that made me think it shifted away from json to a more binary structure
20:45:45 <cpuga> json schema could be submitted as a value
20:45:45 <m-greene> if the vendor meta-data can still be json then I’m fine
20:46:11 <xgerman_> well, it lays it out as key value
20:46:21 <xgerman_> that would be overkill for just some json
20:46:26 <m-greene> cpuga: is lb_topology a new octavia-ism?
20:47:04 <johnsom> Yeah, the key, value, thing seems to be contrary to the json schema
20:47:25 <johnsom> m-greene We have had topology as a configuration setting since Mitaka
20:47:43 <cpuga> m-greene: lb_topology is an octavia table
20:47:54 <johnsom> It's for all LBs now, but should be configurable via the flavor instead
20:48:08 <m-greene> ok- that’s why I don’t recognize it.. our vendor driver is still only neutron_lbaas
20:48:46 <m-greene> so then would we care about it once a vendor runs under octavia?  or since it allows NULL we can simply ignore the field
20:49:10 <johnsom> Ok, I think I want to read through this and comment on the patch.  Any other discussion we need on it before we move on to logging?
20:49:21 <xgerman_> I am good
20:49:36 <xgerman_> rm_work logiing?
20:49:56 <rm_work> well, the spec is actually someone else's
20:49:59 <johnsom> m-greene It might be something that becomes legacy or yeah, is ignored by the providers.
20:50:03 <rm_work> or ... is it code actually, i forget
20:51:16 <rm_work> #link https://review.openstack.org/#/c/471682/
20:51:23 <rm_work> (or does johnsom have to #link)
20:51:40 <johnsom> Nope, you can too I think
20:51:49 <rm_work> the bug:
20:51:51 <rm_work> #link https://bugs.launchpad.net/octavia/+bug/1665069
20:51:52 <openstack> Launchpad bug 1665069 in octavia "Support of haproxy log aggregation capabilites" [Wishlist,In progress] - Assigned to Ganpat Agarwal (gans-developer)
20:52:29 <johnsom> Yeah, so this has some work to do.
20:52:55 <rm_work> I was going to recommend we make the logging situation be handled by syslog rather than haproxy, as it's a bit more standard and would allow for a customizable syslog config -- and that also allows for aggregating both log types separately
20:53:03 <rm_work> but wanted to see other people's opinions
20:53:03 <johnsom> First, we need to decide how we want to tag them with the LB ID and probably project_id?
20:53:22 <rm_work> basically, i think going straight to code here was maybe a bit hasty
20:53:28 <rm_work> though code always appreciated :P
20:53:41 <xgerman_> what can  people learn form the logs anyway?
20:53:42 <rm_work> just in this case, I don't know if I agree with the direction so I feel bad if there's effort wasted
20:54:15 <rm_work> xgerman_: to tell you the truth i don't really know, other than maybe debugging some stuff if your app isn't working right? :/
20:54:22 <rm_work> but I know that customers always seem to ask for it
20:54:37 <xgerman_> very true - those pesky customers
20:54:41 <johnsom> Right, they want the per-connection logs to debug messed up deployments
20:54:43 <jniesz> and auditing
20:55:21 <xgerman_> ok, I am with @rm_work and syslog
20:55:51 <johnsom> I guess I envision bouncing them through the local syslog to allow some limited spooling
20:55:53 <xgerman_> but everybody does logging differently and some might want to throw a fluentd on each amphora
20:55:56 <rm_work> yeah, and i am kinda envisioning a whole [logging] config section, a template path for custom syslog templates, etc
20:56:19 <johnsom> Yeah, we will need to setup the log format inside haproxy too
20:56:33 <xgerman_> I think this is our action-item
20:56:35 <rm_work> obviously people are welcome to do their own elements for their own completely custom solution
20:56:42 <johnsom> Right
20:56:47 <m-greene> philosophical discussion… is logging for developers to understand bugs coming from operators, or for operators to self-diagnose and use some reference manual to identify remediation?
20:56:55 <rm_work> yeah, we can expose the haproxy log format as config
20:56:56 <rm_work> with a reasonable default
20:57:14 <xgerman_> m-greene it’s for users debugging their apps —
20:57:17 <rm_work> m-greene: i think it's for developers to understand bugs coming from developers?
20:57:22 <johnsom> m-greene We are mostly talking about user visible per-connection logs
20:57:30 <rm_work> and also maybe statistics stuff for usage?
20:57:59 <m-greene> in my firewall days, logs are required by certain companies for audit purposes
20:57:59 <jniesz> in our case it would be for application teams to debug / statistics / analytics
20:58:08 <m-greene> HIPAA
20:58:11 <rm_work> ^^ yeah those are the use-cases i've heard
20:58:13 <xgerman_> so we should propose/template a format and maybe give a reference (syslog) and leave the rest to the operator
20:58:20 <rm_work> yep
20:58:30 <rm_work> though i think syslog will be generic enough to cover 90% of cases
20:58:54 <johnsom> One case I know this was useful in the past, some idiot used libcurl without configuring it.  They were downloading large files over small pipes.  By default libcurl closes the connection idle or not after 5 minutes.  This log would show the client closed the connection and the LB or server did not fail the connection.
20:58:54 <rm_work> syslog can send logs basically anywhere these days
20:59:19 <jaypipes> johnsom: I take great offense at being referred to as small.
20:59:29 <xgerman_> well, we also need to account for people wanting to ship certs on the a,mp to secure the communication, etc
20:59:40 <johnsom> Right, and proxies for syslog are available for the lb-mgmt-net
20:59:57 <johnsom> jaypipes Ha, no offence intended
21:00:04 <jaypipes> johnsom: :) j/k
21:00:07 <xgerman_> lol
21:00:14 <m-greene> so octavia needs loging for security audit purposes… also see emergency alerts, utility billin
21:00:24 <johnsom> Ok, out of time.  More discussion in the channel or on the patch.....
21:00:30 <johnsom> #endmeeting