20:00:15 <johnsom> #startmeeting Octavia
20:00:15 <openstack> Meeting started Wed Mar 21 20:00:15 2018 UTC and is due to finish in 60 minutes.  The chair is johnsom. Information about MeetBot at http://wiki.debian.org/MeetBot.
20:00:16 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
20:00:18 <openstack> The meeting name has been set to 'octavia'
20:00:25 <johnsom> Hi folks
20:00:40 <johnsom> #topic Announcements
20:01:05 <johnsom> "S" series naming poll closes today - check you e-mail for your voting link
20:01:22 <johnsom> TC elections are coming up in April
20:01:34 <xgerman_> o/
20:01:45 <johnsom> That is about what I have for announcements this week.
20:01:50 <johnsom> Any others?
20:02:12 <xgerman_> summit talks?
20:02:25 <xgerman_> also travel support is closing today
20:03:16 <johnsom> Sure, I have two talks scheduled for Vancouver: Octavia project update (40 minutes extended version) and I signed up for an on-boarding session for Octavia.
20:03:53 <nmagnezi_> o/
20:04:00 <rm_work> cool
20:04:02 <xgerman_> I have. a talk on Octavia and K8
20:04:06 <rm_work> do either of those actually require voting
20:04:13 <rm_work> or are they just ... set
20:04:16 <xgerman_> they are all confirmed
20:04:19 <rm_work> cool.
20:04:24 <rm_work> hey i'm doing octavia on k8s :)
20:04:51 <johnsom> Mine are set by the foundation. They are not yet on the schedule, they will add them later
20:05:44 <johnsom> Any other announcements?
20:06:00 <johnsom> #topic Brief progress reports / bugs needing review
20:06:45 <johnsom> Ugh, trying to remember what I worked on. A bunch of gate related issues.
20:07:01 <cgoncalves> did we all...
20:07:05 <cgoncalves> *didn't
20:07:14 <johnsom> Oh, helped xgerman_ out with the proxy plugin gate job.
20:07:46 <johnsom> Today i am starting work on the tempest plugin again to address the comments folks left and the discussion we had at the PTG.
20:07:58 <rm_work> i made a little review thing with mine and german's listed https://etherpad.openstack.org/p/octavia-priority-reviews
20:08:07 <rm_work> having something like that can be useful for us
20:08:09 <johnsom> Hope to wrap that up today/tomorrow so you all can shot more holes in it... grin
20:08:18 <rm_work> if you want something reviewed, add it and i can look :)
20:08:26 <rm_work> was hoping i could get johnsom to help prioritize and add stuff too
20:08:43 <johnsom> rm_work should I add that to the channel topic like we do at the end of the cycle?
20:08:45 <rm_work> we usually do one of these before each release and it seems to be quite helpful for velocity
20:08:49 <rm_work> I think that'd be good
20:09:13 <johnsom> Ok, I will work on that after the meeting
20:09:17 <rm_work> it's sometimes hard for me to tell what i should be reviewing, across so many projects
20:09:23 <rm_work> and with a lot of stuff WIP
20:09:36 <johnsom> EVERYTHING! hahahha
20:10:11 <johnsom> Yeah, it really helps me with my review dashboard if folks use the workflow -1 for WIP patches
20:11:13 <johnsom> Once the tempest patch is straightened out, I'm on to provider driver work
20:11:50 <johnsom> I did start a migration from neutron-lbaas to octavia script one evening. It is really just a start. I think rm_work is going to work on it some
20:12:25 <johnsom> Any other updates?
20:12:40 <johnsom> I think I saw Nir had some rally stuff in flight, so that is good too
20:13:00 <nmagnezi_> yup
20:13:03 <nmagnezi_> still in the works
20:13:08 <nmagnezi_> but will finalize this very soon
20:13:18 <nmagnezi_> basically two patches are already up
20:13:18 <johnsom> Nice
20:13:27 <nmagnezi_> 1. add octavia pythonclient support
20:13:38 <nmagnezi_> 2. port the existing scenario to use Octavia
20:13:44 <nmagnezi_> which is mostly done
20:14:05 <nmagnezi_> the 3rd patch will contain additional stuff (mostly CRUD for our LB resources)
20:14:42 <johnsom> I also know I'm behind on dashboard patch reviews. I hope to load that up and do some reviews on that stuff this week.  Lots of good work going on there
20:15:04 <nmagnezi_> oh and just if it interests anyone here, Rally is about to split to two code bases. to I ported my patches to rally-openstack
20:15:16 <nmagnezi_> johnsom, I can help with those
20:15:39 <johnsom> Ah, interesting, so there will be an OpenStack specific Rally and then a general use one?
20:16:25 <nmagnezi_> from what I understood from the Rally core team, the will have a generic base framework and additional code bases for plugins
20:16:39 <johnsom> That makes sense
20:16:52 <nmagnezi_> yup
20:16:59 <cgoncalves> openstack-octavia-rally project? :)
20:17:04 <nmagnezi_> haha
20:17:09 <nmagnezi_> maybe rally-k8s
20:17:12 <nmagnezi_> who knows :)
20:17:16 <johnsom> Yeah, if we need a repo for the plugin let me know
20:17:30 <nmagnezi_> but anyhow, the split is still WIP
20:17:37 <johnsom> Ok
20:17:41 <nmagnezi_> johnsom, I don't think we will, but will keep you posted
20:17:46 <johnsom> #topic Other OpenStack activities of note
20:17:54 <johnsom> OpenStack Common Healthcheck WSGI Middleware spec
20:18:00 <johnsom> #link https://review.openstack.org/#/c/531456
20:18:33 <johnsom> Our friend mugsie is proposing a common method to report service health
20:18:44 <johnsom> Interesting read, worth commenting on.
20:18:57 * mugsie will reply on that spec soon, I promise
20:19:21 <johnsom> I know our friends at F5 were interested in how we can expose the controller health in a useful way. This might be the answer
20:19:23 <rm_work> neat
20:19:57 <johnsom> Proposed "Extended Maintenance" policy for stable branches
20:20:05 <johnsom> #link https://review.openstack.org/#/c/548916/
20:20:32 <johnsom> Also of interest, proposals about how to handle extended maintenance for stable branches.
20:21:15 <johnsom> For those of you not running master with a few days lag....
20:21:46 <johnsom> Ok, on to the big topic....
20:21:55 <nmagnezi_> rm_work, i think he pointed at you :D
20:21:56 <johnsom> #topic Octavia deleted status vs. 404
20:22:08 <johnsom> #link https://review.openstack.org/#/c/545493/
20:22:27 <johnsom> In my work on the tempest plugin and the proxy plugin gate I noticed we have a problem
20:22:53 <johnsom> Most services in OpenStack return a 404 when a query comes in for a deleted item.
20:23:16 <johnsom> The current Octavia API does not, it returns a record with the provisioning_status marked DELETED.
20:23:32 <rm_work> yeah ... i also just noticed that nova somehow accepts a --deleted param to show old deleted stuff, otherwise 404s on gets
20:23:37 <johnsom> This morning I confirmed that neutron-lbaas also returns a 404
20:23:55 <johnsom> So, we have a backward compatibility issue.
20:23:56 <xgerman_> yep,  my proxy-gate chokes on the DEL:ETED
20:24:23 <cgoncalves> backport material?
20:24:27 <johnsom> So, I wrote up this patch, which switches it over to 404 and gives a path to having a --deleted flag.
20:24:40 <johnsom> Well, here is the part I need your input on....
20:25:03 <xgerman_> cgoncalves: we are in a funny spot. We released the API and documented the DELETED behavior
20:25:07 <johnsom> We have now released two versions with the API doing the "DELETED" bit, even though the api-ref does show the 404
20:25:09 <cgoncalves> backporting would fix and break API at the same time
20:25:16 <xgerman_> yes.
20:25:36 <xgerman_> Related: n-lbaas returns 404 instead of 403 (FORBIDDEN)
20:25:50 <johnsom> Actually, I don't think we documented the "DELETED" case. I haven't looked through the whole api-ref, but I know the 404's are there
20:26:13 <johnsom> xgerman_ I'm going to ignore the 403 thing for now. That is a neutron oddity
20:26:15 <xgerman_> mmh, we need to make sure we don’t *change* the API after the fact
20:26:39 <xgerman_> johnsom: well it breaks backward compatibility - but I doubt anyone was using that
20:27:01 <johnsom> xgeman_ Yeah, but let's focus on one topic at a time
20:27:29 <xgerman_> ok, we always told people to use octav ia API - we could change our recommendation to use proxy if 100$ compaitibility is needed and octavia-API if you are willing to fix the two or three variations
20:27:48 <johnsom> Yeah, ok, in the API-REF we list the DELETED as a possible provisioning_status, but not in any of the sections. Each section does list 404 however.
20:28:53 <johnsom> So, how do we want to handle this issue in Octavia API?
20:29:05 <xgerman_> mmh, so I am for consistency between servcies…
20:29:14 <johnsom> 1. Consider it an API bug, fix, backport, beg for forgiveness in the release notes.
20:29:31 <cgoncalves> it has to be fixed either now or later. I'd say fix it now and backport to queens and perhaps also pike (if taken as a critical issue). existing deployments could eventually start observing a different behavior, yes...
20:29:52 <johnsom> 2. Bump the API version. Likely a major bump as it's not necessarily backward compat
20:30:10 <rm_work> yeah...
20:30:13 <rm_work> I would say fix it now
20:30:14 <johnsom> 3. ???
20:30:25 <rm_work> the pain will be less
20:30:32 <rm_work> we're about to have people switching over en-mass soon
20:30:48 <johnsom> Yeah, I think we need to do it now, I'm just struggling with how...
20:30:48 <rm_work> my guess is relatively few have actually seen or would be affected by this
20:30:58 <xgerman_> let’s not do a 3.0 - people already are freaked out about 2.0
20:31:09 <nmagnezi_> lol
20:31:13 <nmagnezi_> xgerman_, good point
20:31:47 <johnsom> Yeah, I think the most pain would be with the libraries. I can of course fix openstacksdk, but what about gopher and openstack4j
20:32:20 <johnsom> Yeah, I really don't want to do 3.0 now. (though there are other result codes that neutron-lbaas used that are wrong IMO)
20:32:22 <nmagnezi_> johnsom,what usually justifies an API minor version bump? bug fixes or just new features
20:32:24 <nmagnezi_> ?
20:32:38 <rm_work> i think we just ... fix it and take the backlash
20:32:40 <rm_work> if there is any
20:33:00 <xgerman_> terraform etc. wait for DELETED
20:33:07 <johnsom> Right, A.B.C  A is breaking change to the API, B new features but compat, C is bug fixes
20:33:07 <rm_work> hmm
20:33:21 <rm_work> errr
20:33:26 <rm_work> can we... make it a deployer option?
20:33:38 <nmagnezi_> rm_work, please no :<
20:33:38 <rm_work> temporarily?
20:33:42 <rm_work> i mean
20:33:50 <johnsom> But we don't really have our API versioning story straight yet. Our discovery is broken
20:34:14 <rm_work> start it deprecated, but allow people time to flip it over
20:34:16 <xgerman_> yep, and gophercloud is rewriting anyway
20:34:18 <rm_work> "you can flip this now, soon it will be flipped for you"
20:34:21 <johnsom> LOL, I just had a thought.  It's bad, but a thought....   404 with the DELETED body....
20:34:26 <xgerman_> so if we sneak it in now they should be fine
20:34:28 <rm_work> LOL
20:34:31 <rm_work> ummmmmmmmmmm
20:34:33 <rm_work> yes?
20:34:37 <rm_work> I mean... why not?
20:34:45 <rm_work> though probably would still break the tools
20:34:53 <rm_work> because they'd see the status first is my guess
20:35:03 <johnsom> Yeah, I think it doesn't really help that much
20:35:09 <xgerman_> +1
20:35:20 <xgerman_> do we have the deletes=True option?
20:35:30 <xgerman_> so we show deleted ones on request?
20:35:31 <johnsom> So, frankly I think people will be happy that it becomes the same as the other services.
20:35:39 <rm_work> yes
20:35:44 <johnsom> Also, our client already handles lit like a 404
20:35:48 <rm_work> i think we may just need to cause some temporary breakage
20:35:56 <rm_work> to get to consistency
20:36:23 <xgerman_> we are playing with credibility here. People don’t like us breaking things IHMO
20:36:55 <johnsom> ha, people keep bringing up v1 so.... haters are going to hate
20:36:58 <rm_work> this would be us NOT breaking things IMO
20:37:08 <rm_work> because i don't think many people have switched yet
20:37:08 <johnsom> I am more about doing the right thing that people whining
20:37:12 <rm_work> to octavia
20:37:18 <xgerman_> well, I have an install which relies on that feature
20:37:25 <rm_work> does it?
20:37:38 <xgerman_> yep, bot terraform and k8 provider waut for DELETED
20:37:42 <rm_work> hmmm
20:37:48 <rm_work> can we fix those to work with BOTH
20:37:51 <rm_work> *first*
20:37:54 <rm_work> get patches in for them
20:37:57 <rm_work> and then do the switch
20:37:57 <xgerman_> but it’s Pike — so as long as we leave that alone I am +2
20:38:19 <rm_work> maybe once a patch lands to make them work both ways we can backport?
20:38:42 <johnsom> Oye, I feel like it should be backported all the way to Pike...
20:38:45 <nmagnezi_> rm_work, if that will add a config option that would be a problem
20:38:49 <nmagnezi_> to backport..
20:38:51 <cgoncalves> johnsom: +1
20:39:09 <rm_work> nmagnezi_: nah that was a different thought
20:39:14 <rm_work> yes
20:39:19 <rm_work> so what if we do it in Master
20:39:23 <nmagnezi_> rm_work, oh, alright
20:39:25 <rm_work> then make terraform work with either
20:39:32 <rm_work> then once that merges we backport to pike
20:39:34 <johnsom> xgerman_ do you have the places in the repos for teraform and k8s that we can go do these patches?
20:40:04 <xgerman_> I can find them — but my problem is that we are doing deploys and our version management isn’t that great
20:40:05 <johnsom> Yeah,  parallel effort this
20:40:59 <xgerman_> so I like this change come with a cahnge Pike->Queen
20:41:07 <xgerman_> just my 2ct
20:41:28 <johnsom> So let's go around the room and get your thoughts on the situation and how you think we should move forward.
20:42:04 <johnsom> It's a bad situation no matter what, we missed it while doing testing for pike
20:42:33 <johnsom> No one wants to go first?
20:43:03 <johnsom> Do we wait and talk about this again next week?
20:43:10 <rm_work> I mean
20:43:12 <rm_work> I said my bit
20:43:57 <rm_work> fix it to 404. fix terraform to work with either and wait for them to release the fix. backport all the way to pike.
20:44:17 <xgerman_> so did I it’s easier to communicate beginning X you need this new terraform thing
20:45:18 <johnsom> Personally, to me the API-REF is the spec. It lists 404 as a result for the calls. So this is a bug in my book. I would fix it, backport it, and be proactive fixing things we think might break.
20:45:26 <rm_work> yep
20:45:28 <rm_work> agree
20:45:43 <rm_work> if terraform starts breaking, people can look and see the changelog item and get the new release
20:46:00 <cgoncalves> johnsom: +1
20:46:09 <nmagnezi_> johnsom, +2
20:46:27 <cgoncalves> I will overflow soon with all my +1s
20:46:34 <xgerman_> I know people and it’s easier to tell them for Queens you need the new terraform then; oh, you did some stable Pike release and now everyhting is broken
20:46:44 <johnsom> Ok. So please review the patch on master. We seem to all agree we can land that.
20:47:24 <johnsom> xgerman_ please paste some pointers in the channel so we can be aggressive at adding a fix.  I will go check openstacksdk and fix if needed.
20:48:15 <johnsom> We should double check the OSC plugin too.  Someone want to volunteer for that?
20:48:25 <xgerman_> ok, will do
20:49:04 <johnsom> I am assuming the will-do is for the links and not the OSC plugin?
20:49:18 <xgerman_> I will do the terraform fixes
20:49:30 <xgerman_> and investigate the k8s ones
20:49:35 <johnsom> Thanks.
20:49:59 <johnsom> Ok, if no one has cycles for our client plugin I will take a look there too.
20:50:29 <johnsom> I will put an agenda item on the meeting for status and when we should start landing backports.
20:50:50 <johnsom> #topic Open Discussion
20:50:57 <johnsom> Other items today?
20:51:01 <nmagnezi_> I have a question
20:51:05 <nmagnezi_> about our tempest plugin
20:51:09 <johnsom> ok
20:51:12 <nmagnezi_> specifically about https://github.com/openstack/octavia-tempest-plugin/blob/master/octavia_tempest_plugin/config.py#L79-L81
20:51:26 <nmagnezi_> just wondering why did we adopt a specific name for role
20:51:29 <cgoncalves> I knew it!
20:51:38 <nmagnezi_> cgoncalves, :D
20:51:53 <nmagnezi_> I was struggeling with this today
20:51:56 <johnsom> So, first part, that code is going away
20:52:13 <johnsom> #link https://review.openstack.org/543030
20:52:23 <johnsom> replaced with: https://review.openstack.org/#/c/543034/
20:53:08 <johnsom> The reason for the role is the RBAC
20:53:10 <johnsom> #link https://docs.openstack.org/octavia/latest/configuration/policy.html
20:53:30 <johnsom> So OpenStack is moving towards a richer RBAC scheme.
20:53:43 <johnsom> Currently it's either "ADMIN" or "OWNER" by project
20:53:52 <nmagnezi_> aha
20:54:05 <openstackgerrit> German Eichberger proposed openstack/neutron-lbaas master: Fix proxy extension for neutron RBAC  https://review.openstack.org/554004
20:54:06 <johnsom> nova and octavia both implemented this new RBAC scheme
20:54:22 <nmagnezi_> alright so I guess tripleO should configure all the roles mentioned in https://docs.openstack.org/octavia/latest/configuration/policy.html regardless of tempest
20:54:31 <xgerman_> +1
20:54:36 <johnsom> Where you need to have a role "member" to access the load-balancer service (or nova)
20:55:05 <johnsom> This is what is in as "default", but we provide a policy.json that allows you to set it back to the old way
20:55:22 <johnsom> #link https://github.com/openstack/octavia/tree/master/etc/policy
20:55:48 <johnsom> That said, there is a proposal out to officially align these across the services.
20:56:04 <nmagnezi_> johnsom, thanks a lot. will surely read this.
20:56:31 <nmagnezi_> johnsom,  we came across this when we started to test tripleO based deployments with via the tempest plugin
20:56:34 <johnsom> #link https://review.openstack.org/#/c/523973/
20:56:40 <xgerman_> nmagnezi_: https://github.com/openstack/openstack-ansible-os_octavia/blob/25f3446fabd92a74322495bd536696074306d01f/tasks/octavia_policy.yml
20:57:02 <nmagnezi_> and since I was not aware of those RBAC related roles, I tried to understand where this is coming from
20:57:27 <johnsom> That docs page is the source of truth there...
20:57:33 <nmagnezi_> xgerman_, thanks!
20:57:43 <nmagnezi_> johnsom, it's not always is ;)
20:58:04 <johnsom> True
20:58:11 <johnsom> but we are getting better
20:58:15 <nmagnezi_> indeed
20:58:32 <cgoncalves> api-ref is also the source of truth but... DELETED... :P
20:58:42 <johnsom> The "default Octavia policies" section is built out of the code, so will stay accurate
20:59:09 <johnsom> Yeah, api-ref is the truth, our code lies
20:59:16 <johnsom> lol
20:59:20 <nmagnezi_> haha
20:59:38 <johnsom> One minute left.
20:59:43 <johnsom> Thanks folks.
20:59:58 <xgerman_> o/
21:00:00 <nmagnezi_> o/
21:00:02 <johnsom> If you have other questions I will be around
21:00:07 <johnsom> #endmeeting