Wednesday, 2020-02-05

*** armax has quit IRC00:29
*** rcernin has joined #openstack-lbaas01:06
*** armax has joined #openstack-lbaas01:36
*** vesper has joined #openstack-lbaas01:38
*** vesper11 has quit IRC01:39
*** TMM_ is now known as TMM01:56
*** mithilarun has quit IRC01:56
*** mithilarun has joined #openstack-lbaas01:57
*** mithilarun has quit IRC02:06
*** mithilarun has joined #openstack-lbaas02:07
*** vishalmanchanda has joined #openstack-lbaas02:10
*** mithilarun has quit IRC02:12
*** mithilarun has joined #openstack-lbaas02:15
*** armax has quit IRC02:16
*** armax has joined #openstack-lbaas02:16
*** armax has quit IRC02:17
*** mithilarun has quit IRC02:24
*** mithilarun has joined #openstack-lbaas02:30
*** mithilarun has quit IRC02:56
*** armax has joined #openstack-lbaas03:19
*** rcernin has quit IRC03:31
*** ramishra has joined #openstack-lbaas03:50
*** ramishra has quit IRC04:01
*** armax has quit IRC04:03
*** ramishra has joined #openstack-lbaas04:24
*** hongbin has joined #openstack-lbaas04:55
*** rcernin has joined #openstack-lbaas04:57
*** rcernin has quit IRC04:58
*** rcernin has joined #openstack-lbaas04:58
*** mithilarun has joined #openstack-lbaas05:28
*** hongbin has quit IRC05:28
*** mithilarun has quit IRC05:32
*** rcernin has quit IRC06:03
*** ccamposr has joined #openstack-lbaas06:39
*** maciejjozefczyk has joined #openstack-lbaas08:03
*** tkajinam has quit IRC08:12
*** tesseract has joined #openstack-lbaas08:16
dulekjohnsom: Wonderful, thank you!08:24
*** rcernin has joined #openstack-lbaas08:31
*** rpittau|afk is now known as rpittau08:34
*** rcernin has quit IRC08:48
*** rcernin has joined #openstack-lbaas08:53
*** rcernin has quit IRC09:07
*** rcernin has joined #openstack-lbaas09:16
*** rcernin has quit IRC09:20
*** rcernin has joined #openstack-lbaas09:26
*** gcheresh has joined #openstack-lbaas09:27
*** rcernin has quit IRC09:33
*** gcheresh_ has joined #openstack-lbaas09:38
*** gcheresh has quit IRC09:39
*** ramishra has quit IRC10:15
*** ramishra has joined #openstack-lbaas10:21
*** gcheresh_ has quit IRC10:35
*** gcheresh_ has joined #openstack-lbaas10:41
*** ataraday_ has joined #openstack-lbaas10:58
dulekjohnsom: Okay, this solves some of the issue but apparently stable/train scripts keep to use master requirements, so it now fails on a different thing…11:08
dulekjohnsom: I'll try to fix this one.11:08
*** rpittau is now known as rpittau|bbl11:10
*** ramishra has quit IRC11:18
dulekHa, worth to say that uncompatible build of oslo.concurrency was released 2 days ago. It's always everything at once!11:18
openstackgerritMichał Dulko proposed openstack/octavia stable/train: Use stable upper-constraints.txt in Amphora builds
*** ramishra has joined #openstack-lbaas11:22
*** psachin has joined #openstack-lbaas11:30
dulekThis thing may solve it… I think…?11:31
cgoncalvesdulek, yeah, we recently found an issue where master upper-constraints is being pulled11:35
dulekcgoncalves: Will my patch fix it? :D11:35
cgoncalvesright. so that is one possible solution we also thought about, yes11:36
dulekcgoncalves: Whoops, is it the correct one or is there a better one?11:36
cgoncalvesdulek, we have not reached a decision on the solution yet11:37
*** pcaruana has quit IRC11:37
dulekcgoncalves: I see… Okay, let's see if my patch works with Kuryr build. Meanwhile your stable/train branch is most likely busted too thanks to py3-only oslo.concurrency released 2 days ago.11:39
cgoncalvescorrect. same for all stable branches11:39
*** pcaruana has joined #openstack-lbaas11:50
*** rpittau|bbl is now known as rpittau13:15
*** rpittau is now known as rpittau|mtg14:03
*** pcaruana has quit IRC14:17
*** rpittau|mtg is now known as rpittau14:25
*** psachin has quit IRC14:25
*** jamesdenton has joined #openstack-lbaas14:34
*** TrevorV has joined #openstack-lbaas14:47
*** mithilarun has joined #openstack-lbaas14:48
*** mithilarun has quit IRC14:52
*** ramishra has quit IRC15:37
*** ianychoi_ is now known as ianychoi15:40
*** osmanlicilegi has quit IRC15:42
*** osmanlicilegi has joined #openstack-lbaas15:42
*** armax has joined #openstack-lbaas15:47
*** pcaruana has joined #openstack-lbaas15:49
openstackgerritMerged openstack/octavia master: Update the lb_id on an amp earlier if we know it
dulekcgoncalves, johnsom: Seems like that patch above helps with stable branches. It doesn't fix all the issues Kuryr has though. ;)15:53
*** gcheresh_ has quit IRC16:00
johnsom#startmeeting Octavia16:02
openstackMeeting started Wed Feb  5 16:02:03 2020 UTC and is due to finish in 60 minutes.  The chair is johnsom. Information about MeetBot at
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.16:02
*** openstack changes topic to " (Meeting topic: Octavia)"16:02
openstackThe meeting name has been set to 'octavia'16:02
johnsomI'm not sure if rm_work is around today16:02
* johnsom thinks he will use his IRC nic as much as possible16:03
*** Trevor_V has joined #openstack-lbaas16:03
johnsom#topic Announcements16:03
*** openstack changes topic to "Announcements (Meeting topic: Octavia)"16:03
johnsomThe only announcement I have today is that there are known issues on the stable branches at the moment. Most are triggered by projects removing python 2.7 support.16:04
johnsomThank you to everyone that is helping to track those down and get fixes in.16:04
johnsomFor example, the one I did yesterday for diskimage-builder16:05
cgoncalvesthank you!16:05
*** openstackstatus has joined #openstack-lbaas16:05
*** ChanServ sets mode: +v openstackstatus16:05
johnsomThere is still an issue we have on stable branches, but I think gthiemonge will raise that topic a bit later in the meeting.16:05
johnsomAny other announcements today?16:06
*** TrevorV has quit IRC16:07
johnsom#topic Brief progress reports / bugs needing review16:07
*** openstack changes topic to "Brief progress reports / bugs needing review (Meeting topic: Octavia)"16:07
johnsomOk, moving on to updates16:07
johnsomI continue to work on the failover flow. I have posted a very WIP patch, just to save my place, but there is still a bunch of work to do. Right now I am re-working the amphora failover part (LB is functional in the posted patch). I still have some retry work to do, test updates, docstrings, relnotes, etc.  Moving forward, but still not fully magical.16:09
johnsomI have also been spending some time with our NDSU team. They have a pretty good idea what they got themselves into now. We will meet again today to continue the technical discussions.16:10
ataraday_Highlight for jobboard change #link - all reviews appreciated16:10
johnsomataraday_ Thank you!16:11
cgoncalvescool, the active-standby job ran and passed. spare pool did not, though. I'll check why is that16:12
johnsomNice. Any other updates today?16:14
cgoncalvesI posted patch for neutron-lbaas, yay!16:14
johnsomSigh, yeah. Is that all reviewed or does it still need eyes?16:15
cgoncalvesneeds review. it should be straight forward. it's exact same fix we did for octavia16:15
johnsomI should mention there was a governance discussion about neutron-lbaas and retired vs. deprecated. I think that is sorted out now. The issue was we still have stable branches.16:15
johnsomAh, for dashboard. Ok16:16
*** ramishra has joined #openstack-lbaas16:16
ataraday_Also I'm a bit confused with issue I describe in #link!/story/2007197 - I'm concered about amphora with healthy VMs marked stale in that case... Is there anything that could be done?16:17
johnsomI saw your comment. I will need to look at your log again. In the case of a DB outage, the health check should pause for the full health interval to allow the amps to catch up. I need to see if that happened or not16:18
johnsomAs Carlos pointed out, this was previously fixed.16:19
cgoncalvesthere may be edge cases, though16:19
johnsomyeah, and pulling the DB out from under OpenStack (not just Octavia) "Causes bad things to happen"(tm)16:20
johnsomIt could be nova killed the instances too16:20
johnsomI will try to spend some more time on the log today to see if I can see something16:21
johnsomIt is important to track down though. We should be "failing safe".16:22
ataraday_In case which I was able to simulate Octavia started failover16:22
*** kklimonda has quit IRC16:22
ataraday_thanks a lot for looking into this!16:22
*** kklimonda has joined #openstack-lbaas16:22
johnsomThank you for trying to reproduce it and gather logs. Super helpful16:23
johnsomAny other updates today or should we move on to the amphora-agent issue?16:25
johnsom#topic Amphora agent is pulling master upper-constraints.txt file16:25
*** openstack changes topic to "Amphora agent is pulling master upper-constraints.txt file (Meeting topic: Octavia)"16:25
johnsomgthiemonge Do you want to introduce this issue?16:25
gthiemongeyes, we have an issue on stable branches16:26
gthiemongewhen we build the amphora image, we pull upper-constraint from master16:27
gthiemongeand we have now an issue with oslo.concurrency that cannot be installed16:27
gthiemongedulek and cgoncalves proposed commits to fix it16:28
gthiemonge and
gthiemongebut we need to find a good way to fix the issue16:29
johnsomIt's that line I think16:29
gthiemongeis using stable/XXX instead of master on this line on stable branches a good fix?16:30
*** mithilarun has joined #openstack-lbaas16:30
johnsomYeah, that would fix it. But it also requires us to remember to bump that on each new stable branch.....16:31
cgoncalvesIMO it is and better than my proposed patch (it was just to share another solution, I still prefer changing the URL)16:31
gthiemongejohnsom: this might be a concern16:31
haleybgthiemonge: the other alternative is something like that was done for neutron-lib, i can send one out for oslo.concurrency if it doesn't work16:31
johnsomWell, it had been done in pike, so we have proof it gets forgotten16:32
cgoncalvesso there's the openstack bot that automatically proposes changes to tox.ini when branches are created. could it be taught to make an additional change for octavia?16:32
johnsomYes there is. I'm not sure how the release folks would feel about having a project specific change in there, but we can try.....16:33
johnsomOne other idea:16:33
dulekI understand that automation is awesome, but folks, just get yourself a list of stuff to do when stable branches get created and follow them. :P16:34
johnsomWe extend Adam's proposed patch to include a parameter to that specifies a branch. Then we key off of that to automatically set the UC and amphora-agent branch settings, ONLY IF they are not allready set on the command line.16:34
cgoncalvesdulek, are you self-volunteering? :)16:35
johnsomI had blocked Adam's patch as I had it break on me with a bug in the DIB "cache" code, where it only pulls the amphora-agent repo from the cache and will throw errors with "commit (maybe it was tag)" not found errors.16:35
dulekcgoncalves: Oh, I already have one for kuryr-kubernetes. And frankly I don't want to add "update DIB_REPOREF_amphora_agent in zuul.d".16:35
johnsomdulek lol, we would let you16:36
cgoncalvesthe con of that approach is if one wants to build from a tag or a specific hash. it would used for requirements which would not work16:36
cgoncalvesdulek, it's just one simple thing. you already have a list and are an octavia user ;)16:36
johnsomYeah, the parameter would have to be "branch" only, not the "tag" option the environment variable would provide16:36
johnsomWould be interested to see what rm_work thinks...  grin16:37
*** psachin has joined #openstack-lbaas16:38
dulekI don't think any solution would save you making a commit - either to devstack/settings (to make sure default DevStack on stable branches is sane) or to Zuul files (to make sure correct branch is passed to tests) or to the file I modified.16:38
johnsomOpen to anyone's opinion too, just wanted to ping him again for good measure16:38
dulekIf it could be automatic, then we wouldn't have a bot doing those changes in tox.ini on every branching.16:39
johnsomI would like whatever solution we decide on to not just be focused on devstack/gates. We see users running into this issue too.16:39
dulekjohnsom: My patch satisfies both DevStack and Zuul. ;)16:40
johnsomThere is usually 2-3 a year that accidentally build a master image.16:40
* dulek is talking only about upper-constraints.txt when installing amphora agent.16:40
cgoncalvesI see Adam's patch as something that could address the issue at hand but not at 100%16:41
cgoncalvesI asked in the infra channel if project specific tasks could be added to the bot on stable branch create16:41
johnsomSo maybe this is a layered solution. 1. Adam's patch, enhanced to be "branch" and fix UC at the same time. 2. Update the stable branch repos. 3. Propose a bot patch to automate this for us.16:42
johnsomcgoncalves That bot is owned by the release team, so #openstack-release is the best channel16:42
cgoncalvesoh, didn't know that. thanks16:42
johnsomWhat do we think of my proposal?  If we agree with that, I will volunteer to take point.16:43
cgoncalvesasked in -release16:43
cgoncalvesshould action 1 and 2 be swapped? action 2 is a quick fix16:44
johnsomWe need to update our README docs too. It's still not very clear on this whole issue.16:44
* johnsom thinks he wasn't thinking order, but just a list of things to do16:45
* johnsom maybe that was cgoncalves volunteering to do the stable patches part....16:45
cgoncalves+1 in that case, with order being 2, 1, 3 if possible :)16:45
cgoncalvesdulek already posted a patch for train16:46
cgoncalvesunless he wants to do for other branches, I can copy-cat16:46
johnsomWe should do both the amphora-agent and the UC IMO16:46
dulekcgoncalves: Go for copy-cating. ;)16:47
dulekcgoncalves: I'm just here to unblock Kuryr's stable/train..16:47
cgoncalvesjohnsom, I'm lost, sorry. what's the fix required to UC?16:47
johnsomcgoncalves Here: both the agent and the UC need to be pinned to the stable branch.16:48
cgoncalvesaaaah, yeah yeah16:48
cgoncalvesthought you were referring to u-c.txt16:49
johnsomOk, do we need a vote on this path forward? grin (shout out to the project founders)16:50
johnsomAny other comments on this approach?16:50
haleybjust that pinning U-C based on python version is also possible, if all else fails16:51
johnsomThank you gthiemonge for raising the topic!16:51
johnsomhaleyb Do you think that is a better option?16:52
*** ccamposr has quit IRC16:52
haleybi don't know if it's better, but if it's totally related to py2 it could work16:53
johnsomWe would still have an issue with the agent I think, as octavia would already be checked out.16:54
*** maciejjozefczyk has quit IRC16:54
johnsomOk, we have a few minutes for open discussion...16:56
johnsom#topic Open Discussion16:56
*** openstack changes topic to "Open Discussion (Meeting topic: Octavia)"16:56
johnsomJust in case someone had another topic this week16:56
*** psachin has quit IRC16:58
johnsomlol, ok.16:58
johnsomIf there is nothing else this week I will close the meeting out.16:58
*** openstack changes topic to "Discussions for OpenStack Octavia | Priority bug review list:"17:00
openstackMeeting ended Wed Feb  5 17:00:00 2020 UTC.  Information about MeetBot at . (v 0.1.4)17:00
openstackMinutes (text):
johnsomThanks everyone17:00
gthiemongejohnsom: thank you!17:00
openstackgerritCarlos Goncalves proposed openstack/octavia stable/train: Use stable upper-constraints.txt in Amphora builds
*** mithilarun has quit IRC17:08
*** mithilarun has joined #openstack-lbaas17:08
*** mithilarun has quit IRC17:11
*** mithilarun has joined #openstack-lbaas17:12
cgoncalvesso, where was I before the hard-reboot?! lol17:12
johnsomUgh. Yeah, my bottom task bar in Fedora disappeared overnight, so I'm guessing I'm not far from a major reboot myself.17:13
openstackgerritCarlos Goncalves proposed openstack/octavia stable/stein: Use stable upper-constraints.txt in Amphora builds
*** mithilarun has quit IRC17:16
openstackgerritCarlos Goncalves proposed openstack/octavia stable/stein: Use stable upper-constraints.txt in Amphora builds
openstackgerritCarlos Goncalves proposed openstack/octavia stable/rocky: Use stable upper-constraints.txt in Amphora builds
openstackgerritCarlos Goncalves proposed openstack/octavia stable/queens: Use stable upper-constraints.txt in Amphora builds
*** NobodyCam has quit IRC17:50
*** NobodyCam has joined #openstack-lbaas17:51
*** ataraday_ has quit IRC17:56
*** rpittau is now known as rpittau|afk18:01
*** mithilarun has joined #openstack-lbaas18:02
*** johnsom has quit IRC18:03
*** johnsom has joined #openstack-lbaas18:03
cgoncalvessigh. so tempest is obviously pulling master u-c.txt and that fails because oslo.concurrency is not capped18:10
johnsomwhack-a-mole I tell you.....18:11
johnsomThe python 2.7 transition is going to keep popping up issues....18:11
*** tesseract has quit IRC18:11
*** gcheresh_ has joined #openstack-lbaas19:10
cgoncalvesgrenaaaaade! BOOM! :/19:13
cgoncalvesgrenade exploded in
rm_workok i see you realized why :D19:17
* rm_work looks for the queens patch19:17
* rm_work realizes there aren't any others19:18
cgoncalvesrm_work, queens patch isn't ready for merge yet19:18
cgoncalvesdevstack stable/queens has to be fixed first to fix tempest >=py3.6 to fix oslo.concurrency19:18
cgoncalvesfun, fun, fun19:19
rm_workok well while we're waiting...19:19
cgoncalvesdon't you even...19:19
* rm_work did19:19
*** larsks has left #openstack-lbaas19:28
openstackgerritMichael Johnson proposed openstack/octavia master: Add diskimage-create arg for Octavia Git branch
openstackgerritMichael Johnson proposed openstack/octavia master: Add diskimage-create arg for Octavia Git branch
johnsomOpps, missed one update in the Readme19:30
*** gcheresh_ has quit IRC19:32
openstackgerritMichael Johnson proposed openstack/octavia master: Add diskimage-create arg for Octavia Git branch
*** NobodyCam has quit IRC19:39
*** johnsom has quit IRC19:40
*** kklimonda has quit IRC19:40
openstackgerritMichael Johnson proposed openstack/octavia master: Add diskimage-create arg for Octavia Git branch
openstackgerritMichael Johnson proposed openstack/octavia master: Add diskimage-create arg for Octavia Git branch
*** gcheresh_ has joined #openstack-lbaas21:24
*** mithilarun has quit IRC21:25
*** mithilarun has joined #openstack-lbaas21:25
*** mithilarun has quit IRC21:32
*** mithilarun has joined #openstack-lbaas21:53
*** nicolasbock has joined #openstack-lbaas21:56
*** gcheresh_ has quit IRC21:58
*** maciejjozefczyk has joined #openstack-lbaas22:02
*** maciejjozefczyk has quit IRC22:18
*** nicolasbock has quit IRC22:23
*** mithilarun has quit IRC22:36
*** mithilarun has joined #openstack-lbaas22:37
*** Trevor_V has quit IRC22:40
*** mithilarun has quit IRC22:41
*** tkajinam has joined #openstack-lbaas22:48
*** kklimonda has joined #openstack-lbaas22:59
*** NobodyCam has joined #openstack-lbaas23:00
*** johnsom has joined #openstack-lbaas23:02
openstackgerritAdam Harwell proposed openstack/octavia master: Use routed network filter if it exists
*** mithilarun has joined #openstack-lbaas23:28
*** mithilarun has quit IRC23:33
*** mithilarun has joined #openstack-lbaas23:55

Generated by 2.15.3 by Marius Gedminas - find it at!