16:02:03 <johnsom> #startmeeting Octavia
16:02:04 <openstack> Meeting started Wed Feb  5 16:02:03 2020 UTC and is due to finish in 60 minutes.  The chair is johnsom. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:02:05 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:02:07 <openstack> The meeting name has been set to 'octavia'
16:02:29 <ataraday_> hi
16:02:33 <gthiemonge> Hi
16:02:35 <cgoncalves> hi
16:02:43 <johnsom> I'm not sure if rm_work is around today
16:03:03 * johnsom thinks he will use his IRC nic as much as possible
16:03:28 <johnsom> #topic Announcements
16:03:35 <haleyb> hi
16:04:33 <johnsom> The only announcement I have today is that there are known issues on the stable branches at the moment. Most are triggered by projects removing python 2.7 support.
16:04:48 <johnsom> Thank you to everyone that is helping to track those down and get fixes in.
16:05:00 <johnsom> #link https://review.opendev.org/705806
16:05:11 <johnsom> For example, the one I did yesterday for diskimage-builder
16:05:16 <cgoncalves> thank you!
16:05:42 <johnsom> There is still an issue we have on stable branches, but I think gthiemonge will raise that topic a bit later in the meeting.
16:06:12 <johnsom> Any other announcements today?
16:07:36 <johnsom> #topic Brief progress reports / bugs needing review
16:07:44 <johnsom> Ok, moving on to updates
16:09:13 <johnsom> I continue to work on the failover flow. I have posted a very WIP patch, just to save my place, but there is still a bunch of work to do. Right now I am re-working the amphora failover part (LB is functional in the posted patch). I still have some retry work to do, test updates, docstrings, relnotes, etc.  Moving forward, but still not fully magical.
16:10:07 <johnsom> I have also been spending some time with our NDSU team. They have a pretty good idea what they got themselves into now. We will meet again today to continue the technical discussions.
16:10:32 <ataraday_> Highlight for jobboard change #link https://review.opendev.org/#/c/647406/ - all reviews appreciated
16:11:05 <johnsom> ataraday_ Thank you!
16:12:56 <cgoncalves> cool, the active-standby job ran and passed. spare pool did not, though. I'll check why is that
16:14:01 <johnsom> Nice. Any other updates today?
16:14:36 <cgoncalves> I posted patch for neutron-lbaas, yay!
16:15:06 <johnsom> Sigh, yeah. Is that all reviewed or does it still need eyes?
16:15:35 <cgoncalves> needs review. it should be straight forward. it's exact same fix we did for octavia
16:15:48 <johnsom> I should mention there was a governance discussion about neutron-lbaas and retired vs. deprecated. I think that is sorted out now. The issue was we still have stable branches.
16:15:57 <cgoncalves> #link https://review.opendev.org/#/q/I060826fab5067a816297d01d5de20457fd78a742
16:16:52 <johnsom> Ah, for dashboard. Ok
16:17:41 <ataraday_> Also I'm a bit confused with issue I describe in #link https://storyboard.openstack.org/#!/story/2007197 - I'm concered about amphora with healthy VMs marked stale in that case... Is there anything that could be done?
16:18:41 <johnsom> I saw your comment. I will need to look at your log again. In the case of a DB outage, the health check should pause for the full health interval to allow the amps to catch up. I need to see if that happened or not
16:19:09 <johnsom> As Carlos pointed out, this was previously fixed.
16:19:47 <cgoncalves> there may be edge cases, though
16:20:17 <johnsom> yeah, and pulling the DB out from under OpenStack (not just Octavia) "Causes bad things to happen"(tm)
16:20:28 <johnsom> It could be nova killed the instances too
16:21:07 <johnsom> I will try to spend some more time on the log today to see if I can see something
16:22:07 <johnsom> It is important to track down though. We should be "failing safe".
16:22:17 <ataraday_> In case which I was able to simulate Octavia started failover
16:22:30 <ataraday_> thanks a lot for looking into this!
16:23:36 <johnsom> Thank you for trying to reproduce it and gather logs. Super helpful
16:25:19 <johnsom> Any other updates today or should we move on to the amphora-agent issue?
16:25:46 <johnsom> #topic Amphora agent is pulling master upper-constraints.txt file
16:25:57 <johnsom> gthiemonge Do you want to introduce this issue?
16:26:44 <gthiemonge> yes, we have an issue on stable branches
16:27:11 <gthiemonge> when we build the amphora image, we pull upper-constraint from master
16:27:38 <gthiemonge> and we have now an issue with oslo.concurrency that cannot be installed
16:28:04 <gthiemonge> dulek and cgoncalves proposed commits to fix it
16:28:36 <gthiemonge> https://review.opendev.org/#/c/705713/ and https://review.opendev.org/#/c/705927/
16:29:19 <gthiemonge> but we need to find a good way to fix the issue
16:29:24 <johnsom> #link https://github.com/openstack/octavia/blob/stable/train/elements/amphora-agent/source-repository-amphora-agent#L3
16:29:29 <johnsom> It's that line I think
16:29:39 <gthiemonge> correct
16:29:44 <cgoncalves> +1
16:30:12 <gthiemonge> is using stable/XXX instead of master on this line on stable branches a good fix?
16:31:09 <johnsom> Yeah, that would fix it. But it also requires us to remember to bump that on each new stable branch.....
16:31:12 <cgoncalves> IMO it is and better than my proposed patch (it was just to share another solution, I still prefer changing the URL)
16:31:21 <cgoncalves> right
16:31:26 <gthiemonge> johnsom: this might be a concern
16:31:44 <haleyb> gthiemonge: the other alternative is something like https://review.opendev.org/#/c/703642/ that was done for neutron-lib, i can send one out for oslo.concurrency if it doesn't work
16:31:50 <johnsom> #link https://github.com/openstack/octavia/blob/stable/pike/elements/amphora-agent/source-repository-amphora-agent
16:32:06 <johnsom> Well, it had been done in pike, so we have proof it gets forgotten
16:32:08 <cgoncalves> so there's the openstack bot that automatically proposes changes to tox.ini when branches are created. could it be taught to make an additional change for octavia?
16:33:32 <johnsom> Yes there is. I'm not sure how the release folks would feel about having a project specific change in there, but we can try.....
16:33:37 <johnsom> One other idea:
16:34:37 <dulek> I understand that automation is awesome, but folks, just get yourself a list of stuff to do when stable branches get created and follow them. :P
16:34:41 <johnsom> We extend Adam's proposed patch to include a parameter to diskimage-create.sh that specifies a branch. Then we key off of that to automatically set the UC and amphora-agent branch settings, ONLY IF they are not allready set on the command line.
16:35:05 <cgoncalves> dulek, are you self-volunteering? :)
16:35:31 <johnsom> I had blocked Adam's patch as I had it break on me with a bug in the DIB "cache" code, where it only pulls the amphora-agent repo from the cache and will throw errors with "commit (maybe it was tag)" not found errors.
16:35:44 <dulek> cgoncalves: Oh, I already have one for kuryr-kubernetes. And frankly I don't want to add "update DIB_REPOREF_amphora_agent in zuul.d".
16:36:01 <johnsom> dulek lol, we would let you
16:36:02 <cgoncalves> the con of that approach is if one wants to build from a tag or a specific hash. it would used for requirements which would not work
16:36:41 <cgoncalves> dulek, it's just one simple thing. you already have a list and are an octavia user ;)
16:36:43 <johnsom> Yeah, the parameter would have to be "branch" only, not the "tag" option the environment variable would provide
16:37:16 <johnsom> Would be interested to see what rm_work thinks...  grin
16:38:32 <dulek> I don't think any solution would save you making a commit - either to devstack/settings (to make sure default DevStack on stable branches is sane) or to Zuul files (to make sure correct branch is passed to tests) or to the file I modified.
16:38:39 <johnsom> Open to anyone's opinion too, just wanted to ping him again for good measure
16:39:15 <dulek> If it could be automatic, then we wouldn't have a bot doing those changes in tox.ini on every branching.
16:39:38 <johnsom> I would like whatever solution we decide on to not just be focused on devstack/gates. We see users running into this issue too.
16:40:19 <dulek> johnsom: My patch satisfies both DevStack and Zuul. ;)
16:40:19 <johnsom> There is usually 2-3 a year that accidentally build a master image.
16:40:54 * dulek is talking only about upper-constraints.txt when installing amphora agent.
16:41:09 <cgoncalves> I see Adam's patch as something that could address the issue at hand but not at 100%
16:41:19 <johnsom> Right.
16:41:51 <cgoncalves> I asked in the infra channel if project specific tasks could be added to the bot on stable branch create
16:42:19 <johnsom> So maybe this is a layered solution. 1. Adam's patch, enhanced to be "branch" and fix UC at the same time. 2. Update the stable branch repos. 3. Propose a bot patch to automate this for us.
16:42:43 <johnsom> cgoncalves That bot is owned by the release team, so #openstack-release is the best channel
16:42:48 <cgoncalves> +1
16:42:59 <cgoncalves> oh, didn't know that. thanks
16:43:36 <johnsom> What do we think of my proposal?  If we agree with that, I will volunteer to take point.
16:43:37 <cgoncalves> asked in -release
16:44:47 <cgoncalves> should action 1 and 2 be swapped? action 2 is a quick fix
16:44:59 <johnsom> We need to update our README docs too. It's still not very clear on this whole issue.
16:45:21 * johnsom thinks he wasn't thinking order, but just a list of things to do
16:45:59 * johnsom maybe that was cgoncalves volunteering to do the stable patches part....
16:45:59 <cgoncalves> +1 in that case, with order being 2, 1, 3 if possible :)
16:46:16 <cgoncalves> sure
16:46:27 <cgoncalves> dulek already posted a patch for train
16:46:43 <cgoncalves> unless he wants to do for other branches, I can copy-cat
16:46:48 <johnsom> We should do both the amphora-agent and the UC IMO
16:47:09 <dulek> cgoncalves: Go for copy-cating. ;)
16:47:21 <dulek> cgoncalves: I'm just here to unblock Kuryr's stable/train..
16:47:53 <cgoncalves> johnsom, I'm lost, sorry. what's the fix required to UC?
16:48:24 <johnsom> cgoncalves Here: https://review.opendev.org/#/c/705927/1/elements/amphora-agent/source-repository-amphora-agent both the agent and the UC need to be pinned to the stable branch.
16:48:40 <cgoncalves> aaaah, yeah yeah
16:49:02 <cgoncalves> thought you were referring to u-c.txt
16:50:05 <johnsom> Ok, do we need a vote on this path forward? grin (shout out to the project founders)
16:50:27 <johnsom> Any other comments on this approach?
16:51:31 <haleyb> just that pinning U-C based on python version is also possible, if all else fails
16:51:40 <johnsom> Thank you gthiemonge for raising the topic!
16:52:24 <johnsom> haleyb Do you think that is a better option?
16:53:28 <haleyb> i don't know if it's better, but if it's totally related to py2 it could work
16:54:17 <johnsom> We would still have an issue with the agent I think, as octavia would already be checked out.
16:54:52 <haleyb> right
16:56:08 <johnsom> Ok, we have a few minutes for open discussion...
16:56:15 <johnsom> #topic Open Discussion
16:56:25 <johnsom> Just in case someone had another topic this week
16:58:36 <johnsom> lol, ok.
16:58:50 <johnsom> If there is nothing else this week I will close the meeting out.
17:00:00 <johnsom> #endmeeting