16:02:03 #startmeeting Octavia 16:02:04 Meeting started Wed Feb 5 16:02:03 2020 UTC and is due to finish in 60 minutes. The chair is johnsom. Information about MeetBot at http://wiki.debian.org/MeetBot. 16:02:05 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 16:02:07 The meeting name has been set to 'octavia' 16:02:29 hi 16:02:33 Hi 16:02:35 hi 16:02:43 I'm not sure if rm_work is around today 16:03:03 * johnsom thinks he will use his IRC nic as much as possible 16:03:28 #topic Announcements 16:03:35 hi 16:04:33 The only announcement I have today is that there are known issues on the stable branches at the moment. Most are triggered by projects removing python 2.7 support. 16:04:48 Thank you to everyone that is helping to track those down and get fixes in. 16:05:00 #link https://review.opendev.org/705806 16:05:11 For example, the one I did yesterday for diskimage-builder 16:05:16 thank you! 16:05:42 There is still an issue we have on stable branches, but I think gthiemonge will raise that topic a bit later in the meeting. 16:06:12 Any other announcements today? 16:07:36 #topic Brief progress reports / bugs needing review 16:07:44 Ok, moving on to updates 16:09:13 I continue to work on the failover flow. I have posted a very WIP patch, just to save my place, but there is still a bunch of work to do. Right now I am re-working the amphora failover part (LB is functional in the posted patch). I still have some retry work to do, test updates, docstrings, relnotes, etc. Moving forward, but still not fully magical. 16:10:07 I have also been spending some time with our NDSU team. They have a pretty good idea what they got themselves into now. We will meet again today to continue the technical discussions. 16:10:32 Highlight for jobboard change #link https://review.opendev.org/#/c/647406/ - all reviews appreciated 16:11:05 ataraday_ Thank you! 16:12:56 cool, the active-standby job ran and passed. spare pool did not, though. I'll check why is that 16:14:01 Nice. Any other updates today? 16:14:36 I posted patch for neutron-lbaas, yay! 16:15:06 Sigh, yeah. Is that all reviewed or does it still need eyes? 16:15:35 needs review. it should be straight forward. it's exact same fix we did for octavia 16:15:48 I should mention there was a governance discussion about neutron-lbaas and retired vs. deprecated. I think that is sorted out now. The issue was we still have stable branches. 16:15:57 #link https://review.opendev.org/#/q/I060826fab5067a816297d01d5de20457fd78a742 16:16:52 Ah, for dashboard. Ok 16:17:41 Also I'm a bit confused with issue I describe in #link https://storyboard.openstack.org/#!/story/2007197 - I'm concered about amphora with healthy VMs marked stale in that case... Is there anything that could be done? 16:18:41 I saw your comment. I will need to look at your log again. In the case of a DB outage, the health check should pause for the full health interval to allow the amps to catch up. I need to see if that happened or not 16:19:09 As Carlos pointed out, this was previously fixed. 16:19:47 there may be edge cases, though 16:20:17 yeah, and pulling the DB out from under OpenStack (not just Octavia) "Causes bad things to happen"(tm) 16:20:28 It could be nova killed the instances too 16:21:07 I will try to spend some more time on the log today to see if I can see something 16:22:07 It is important to track down though. We should be "failing safe". 16:22:17 In case which I was able to simulate Octavia started failover 16:22:30 thanks a lot for looking into this! 16:23:36 Thank you for trying to reproduce it and gather logs. Super helpful 16:25:19 Any other updates today or should we move on to the amphora-agent issue? 16:25:46 #topic Amphora agent is pulling master upper-constraints.txt file 16:25:57 gthiemonge Do you want to introduce this issue? 16:26:44 yes, we have an issue on stable branches 16:27:11 when we build the amphora image, we pull upper-constraint from master 16:27:38 and we have now an issue with oslo.concurrency that cannot be installed 16:28:04 dulek and cgoncalves proposed commits to fix it 16:28:36 https://review.opendev.org/#/c/705713/ and https://review.opendev.org/#/c/705927/ 16:29:19 but we need to find a good way to fix the issue 16:29:24 #link https://github.com/openstack/octavia/blob/stable/train/elements/amphora-agent/source-repository-amphora-agent#L3 16:29:29 It's that line I think 16:29:39 correct 16:29:44 +1 16:30:12 is using stable/XXX instead of master on this line on stable branches a good fix? 16:31:09 Yeah, that would fix it. But it also requires us to remember to bump that on each new stable branch..... 16:31:12 IMO it is and better than my proposed patch (it was just to share another solution, I still prefer changing the URL) 16:31:21 right 16:31:26 johnsom: this might be a concern 16:31:44 gthiemonge: the other alternative is something like https://review.opendev.org/#/c/703642/ that was done for neutron-lib, i can send one out for oslo.concurrency if it doesn't work 16:31:50 #link https://github.com/openstack/octavia/blob/stable/pike/elements/amphora-agent/source-repository-amphora-agent 16:32:06 Well, it had been done in pike, so we have proof it gets forgotten 16:32:08 so there's the openstack bot that automatically proposes changes to tox.ini when branches are created. could it be taught to make an additional change for octavia? 16:33:32 Yes there is. I'm not sure how the release folks would feel about having a project specific change in there, but we can try..... 16:33:37 One other idea: 16:34:37 I understand that automation is awesome, but folks, just get yourself a list of stuff to do when stable branches get created and follow them. :P 16:34:41 We extend Adam's proposed patch to include a parameter to diskimage-create.sh that specifies a branch. Then we key off of that to automatically set the UC and amphora-agent branch settings, ONLY IF they are not allready set on the command line. 16:35:05 dulek, are you self-volunteering? :) 16:35:31 I had blocked Adam's patch as I had it break on me with a bug in the DIB "cache" code, where it only pulls the amphora-agent repo from the cache and will throw errors with "commit (maybe it was tag)" not found errors. 16:35:44 cgoncalves: Oh, I already have one for kuryr-kubernetes. And frankly I don't want to add "update DIB_REPOREF_amphora_agent in zuul.d". 16:36:01 dulek lol, we would let you 16:36:02 the con of that approach is if one wants to build from a tag or a specific hash. it would used for requirements which would not work 16:36:41 dulek, it's just one simple thing. you already have a list and are an octavia user ;) 16:36:43 Yeah, the parameter would have to be "branch" only, not the "tag" option the environment variable would provide 16:37:16 Would be interested to see what rm_work thinks... grin 16:38:32 I don't think any solution would save you making a commit - either to devstack/settings (to make sure default DevStack on stable branches is sane) or to Zuul files (to make sure correct branch is passed to tests) or to the file I modified. 16:38:39 Open to anyone's opinion too, just wanted to ping him again for good measure 16:39:15 If it could be automatic, then we wouldn't have a bot doing those changes in tox.ini on every branching. 16:39:38 I would like whatever solution we decide on to not just be focused on devstack/gates. We see users running into this issue too. 16:40:19 johnsom: My patch satisfies both DevStack and Zuul. ;) 16:40:19 There is usually 2-3 a year that accidentally build a master image. 16:40:54 * dulek is talking only about upper-constraints.txt when installing amphora agent. 16:41:09 I see Adam's patch as something that could address the issue at hand but not at 100% 16:41:19 Right. 16:41:51 I asked in the infra channel if project specific tasks could be added to the bot on stable branch create 16:42:19 So maybe this is a layered solution. 1. Adam's patch, enhanced to be "branch" and fix UC at the same time. 2. Update the stable branch repos. 3. Propose a bot patch to automate this for us. 16:42:43 cgoncalves That bot is owned by the release team, so #openstack-release is the best channel 16:42:48 +1 16:42:59 oh, didn't know that. thanks 16:43:36 What do we think of my proposal? If we agree with that, I will volunteer to take point. 16:43:37 asked in -release 16:44:47 should action 1 and 2 be swapped? action 2 is a quick fix 16:44:59 We need to update our README docs too. It's still not very clear on this whole issue. 16:45:21 * johnsom thinks he wasn't thinking order, but just a list of things to do 16:45:59 * johnsom maybe that was cgoncalves volunteering to do the stable patches part.... 16:45:59 +1 in that case, with order being 2, 1, 3 if possible :) 16:46:16 sure 16:46:27 dulek already posted a patch for train 16:46:43 unless he wants to do for other branches, I can copy-cat 16:46:48 We should do both the amphora-agent and the UC IMO 16:47:09 cgoncalves: Go for copy-cating. ;) 16:47:21 cgoncalves: I'm just here to unblock Kuryr's stable/train.. 16:47:53 johnsom, I'm lost, sorry. what's the fix required to UC? 16:48:24 cgoncalves Here: https://review.opendev.org/#/c/705927/1/elements/amphora-agent/source-repository-amphora-agent both the agent and the UC need to be pinned to the stable branch. 16:48:40 aaaah, yeah yeah 16:49:02 thought you were referring to u-c.txt 16:50:05 Ok, do we need a vote on this path forward? grin (shout out to the project founders) 16:50:27 Any other comments on this approach? 16:51:31 just that pinning U-C based on python version is also possible, if all else fails 16:51:40 Thank you gthiemonge for raising the topic! 16:52:24 haleyb Do you think that is a better option? 16:53:28 i don't know if it's better, but if it's totally related to py2 it could work 16:54:17 We would still have an issue with the agent I think, as octavia would already be checked out. 16:54:52 right 16:56:08 Ok, we have a few minutes for open discussion... 16:56:15 #topic Open Discussion 16:56:25 Just in case someone had another topic this week 16:58:36 lol, ok. 16:58:50 If there is nothing else this week I will close the meeting out. 17:00:00 #endmeeting