Wednesday, 2018-08-29

*** yamamoto has quit IRC01:03
*** yamamoto has joined #openstack-lbaas01:03
*** dosaboy has quit IRC01:19
*** zioproto has quit IRC01:19
*** irenab has quit IRC01:22
*** hongbin has joined #openstack-lbaas01:48
*** pck has quit IRC01:49
*** andreykurilin has quit IRC01:49
*** zigo has quit IRC01:49
*** rm_work has quit IRC01:49
*** pck has joined #openstack-lbaas01:49
*** andreykurilin has joined #openstack-lbaas01:49
*** rm_work has joined #openstack-lbaas01:55
*** sapd1 has joined #openstack-lbaas02:34
xgerman_^^ happy-dance02:34
*** openstack has joined #openstack-lbaas02:52
dayou^^ chicken-winner02:52
*** ChanServ sets mode: +o openstack02:52
*** openstackstatus has joined #openstack-lbaas03:00
*** ChanServ sets mode: +v openstackstatus03:00
rm_workhmm, johnsom it seems the housekeeping is a bit fragile still :/03:03
rm_workrecently i've seen the DB connection hiccup and then never fix itself, and so it just stops spares-pooling03:04
rm_work"Database connection was found disconnected; reconnecting"03:05
johnsomSome missing exception handling?03:05
rm_workand then from there on out only the cert-rotate kept going03:05
johnsomI know it retried for a while03:05
rm_workgonna look if i have time, trying to get a new version pushed sometime soon tho (our prod is painfully out of date)03:05
rm_workit's been like 3 months since I managed a deploy :303:06
rm_workhoping to get it out this week so i can be running Rocky by the time the PTG happens :)03:07
*** yboaron_ has joined #openstack-lbaas03:17
*** ramishra has joined #openstack-lbaas03:37
*** hongbin has quit IRC03:51
*** lxkong has joined #openstack-lbaas04:04
*** abaindur has quit IRC04:08
*** abaindur has joined #openstack-lbaas04:09
*** irenab has joined #openstack-lbaas04:41
*** yboaron_ has quit IRC04:44
*** abaindur has quit IRC05:32
*** abaindur has joined #openstack-lbaas05:33
*** abaindur has quit IRC05:38
*** reedipb has joined #openstack-lbaas05:44
numansreedipb, Hi. So we have wrongly implemented the batch member update in ovn driver then right ?05:50
reedipbnumans : some changes are required , its not 'wrong' :)05:51
numansreedipb, ok.05:51
reedipbI am working on it numans, maybe I will post an update in another hour or so :)05:51
reedipbbtw Good Morning :D05:51
numansreedipb, great. thanks.05:51
numansreedipb, good morning.. will be noon soon :)05:52
reedipbtill it aint noon , its a great morning :P05:52
*** abaindur has joined #openstack-lbaas06:09
*** pcaruana has joined #openstack-lbaas06:50
*** rcernin has quit IRC07:00
*** yboaron has joined #openstack-lbaas07:03
*** velizarx has joined #openstack-lbaas07:06
*** dolly has quit IRC07:16
*** takamatsu has quit IRC07:16
*** abaindur has quit IRC07:33
*** ataraday has joined #openstack-lbaas07:37
*** ataraday has quit IRC07:43
*** threestrands has quit IRC07:50
*** ktibi has joined #openstack-lbaas07:56
*** zigo has joined #openstack-lbaas07:57
*** celebdor has joined #openstack-lbaas07:57
*** Emine has joined #openstack-lbaas07:58
openstackgerritOpenStack Proposal Bot proposed openstack/octavia-dashboard master: Imported Translations from Zanata
reedipbjohnsom: around?08:13
reedipbjohnsom : I would like to ask you something about the batch_update function... Let me know when you are around :)08:19
*** reedipb has quit IRC08:26
*** yboaron_ has joined #openstack-lbaas10:34
*** yboaron has quit IRC10:36
*** yamamoto_ has joined #openstack-lbaas10:41
*** yamamoto has quit IRC10:45
*** celebdor has quit IRC10:47
*** velizarx has quit IRC11:03
*** velizarx has joined #openstack-lbaas11:18
*** reedipb has joined #openstack-lbaas11:19
*** yboaron_ has quit IRC11:57
*** yboaron_ has joined #openstack-lbaas11:58
*** amuller has joined #openstack-lbaas12:28
*** yamamoto has joined #openstack-lbaas12:33
*** yamamoto_ has quit IRC12:36
*** celebdor has joined #openstack-lbaas12:39
*** celebdor has quit IRC12:39
*** celebdor has joined #openstack-lbaas12:40
*** velizarx has quit IRC12:58
openstackgerritEvgeny Fedoruk proposed openstack/octavia master: Specification for extending provider driver library
*** pcaruana has quit IRC13:04
*** velizarx has joined #openstack-lbaas13:13
openstackgerritAnn Taraday proposed openstack/octavia master: Fix passphrase None errors
*** yboaron_ has quit IRC13:20
openstackgerritAnn Taraday proposed openstack/octavia master: Fix passphrase None errors
*** velizarx has quit IRC13:29
*** velizarx has joined #openstack-lbaas13:31
*** pcaruana has joined #openstack-lbaas14:50
cgoncalvesdayou, hi. is the LB delete cascade available via the dashboard?14:52
*** colin- has joined #openstack-lbaas14:57
*** fnaval has joined #openstack-lbaas15:03
*** Emine has quit IRC15:27
johnsomreedipb What is the question?15:32
reedipbjohnsom: I have thought of the solution from OVN, but want to still confirm the design for other proivider drivers15:33
reedipbjohnsom: currently with batch updates, we need to delete all Existing members which are not passed in the Request.15:33
reedipbjohnsom: with Amphora, its easy. DB is directly accessed with the session15:34
reedipbjohnsom: for other Provider Drivers who do not access Octavia DB for List/Show functions, they may not be able to find out the Existing Pool Members15:35
reedipb( unless they adopt something internally )15:35
johnsomRight, there are many parts of the API that assume the appliance or driver agent is tracking objects in some way. I know one was planning to tag all of the child objects with the ID, etc.15:36
reedipbok ok ... sounds good ..15:37
johnsomThere is also a spec coming that will add a "backup" API to the Octavia API that would dump a full LB config.15:38
cgoncalves^ this spec maybe?15:52
*** celebdor1 has joined #openstack-lbaas15:54
*** celebdor has quit IRC15:57
johnsomYeah, but I guess not exactly what I was expecting15:58
*** ktibi has quit IRC16:01
*** tobias-urdin has joined #openstack-lbaas16:14
*** ramishra has quit IRC16:15
*** celebdor1 has quit IRC16:17
openstackgerritCarlos Goncalves proposed openstack/octavia-tempest-plugin master: DNM: ML2/OVN + DVR enabled job
openstackgerritCarlos Goncalves proposed openstack/octavia-tempest-plugin master: DNM: ML2/OVN + DVR enabled job
*** colin- has left #openstack-lbaas17:00
*** colin- has joined #openstack-lbaas17:00
*** abaindur has joined #openstack-lbaas17:28
*** abaindur has quit IRC17:34
*** Emine has joined #openstack-lbaas17:44
*** velizarx has quit IRC18:20
*** abaindur has joined #openstack-lbaas18:21
*** amuller has quit IRC18:40
*** Emine has quit IRC19:08
*** pcaruana has quit IRC19:09
*** celebdor1 has joined #openstack-lbaas19:16
johnsom#startmeeting Octavia20:00
openstackMeeting started Wed Aug 29 20:00:09 2018 UTC and is due to finish in 60 minutes.  The chair is johnsom. Information about MeetBot at
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.20:00
*** openstack changes topic to " (Meeting topic: Octavia)"20:00
openstackThe meeting name has been set to 'octavia'20:00
johnsomHi folks20:00
johnsom#topic Announcements20:00
*** openstack changes topic to "Announcements (Meeting topic: Octavia)"20:00
johnsomSame reminder about the PTG etherpad:20:00
johnsomThe PTG is coming up fast, so I will try to put a rough schedule together soon20:01
johnsomAlso a note, the TC election nominations start tomorrow.20:01
johnsomIn case you are interested in running for the TC20:02
johnsomThe official Rocky release patch is up for review. I have already checked the SHAs and our stuff looks good.20:02
johnsomAny other announcements today?20:03
johnsomFinally found the Rocky release link...20:04
cgoncalvesOctavia Queens 2.0.2 tagging is blocked until stable team reviews a commit that caught that attention20:04
johnsomYeah, I saw that.20:04
johnsomFYI, I have already bumped the OpenStack Ansible SHA to use that 2.0.2 version20:05
johnsomI think it will be ok since it includes a default.20:05
johnsomThat is all I have so, moving on down the agenda20:06
johnsom#topic Brief progress reports / bugs needing review20:06
*** openstack changes topic to "Brief progress reports / bugs needing review (Meeting topic: Octavia)"20:06
johnsomI have wrapped up most of my internal work so should have a bit more time to work on things upstream.20:06
johnsomI recently wrote a tool to stress test the health manager process. It can also be used to populate a DB with load balancers.20:07
johnsomI will probably post that in my github space sometime in the near future.20:07
cgoncalvesnice, thanks!20:08
cgoncalvesdo share the link later :)20:08
johnsomTesting has shown that we have a bit of a performance regressing in the HM and I have a patch for that in the works.20:08
nmagnezijohnsom, if only we had such tools for nlbaas :D20:08
johnsomI also noticed the UDP code is not reporting listener health correctly and the work around code is in my way with the HM fix, so I will also be posting a fix for that soonish20:08
johnsomnmagnezi HA, well, you can run my existing API stress tool against neutron-lbaas....  But the results aren't pretty20:09
nmagneziyeah.. we know.. :)20:09
johnsomIt does pass well on Octavia however20:10
johnsomThat is mostly what I have been up to over the week.  Any other updates?20:10
johnsomOh, I forgot one item20:10
johnsomI have posted an alternate option for how we can handle API versioning in the tempest plugin.20:11
johnsomrm_work Also posted an idea for it.20:11
johnsomWe should try to review those and merge one soonish.20:11
johnsomWithout this, the tempest plugin fails when run against queens Octavia20:12
nmagneziYes, we saw that in-house20:12
nmagneziThanks so much for submitting that one20:12
johnsomIt tries to test features that were added in Rocky on the Queens cloud20:12
nmagneziYeah stuff like listener timeout and such20:12
johnsomThis patch tests that patch with stable/queens:20:13
johnsomReviews would be appreciated.20:13
cgoncalvesah, forgot to re-review. you made jobs voting, thanks20:14
johnsomAny other updates?20:14
nmagneziSpeaking of reviews, I have a question about other patch you posted20:14
cgoncalveslol, I already re-reviewed :)20:14
johnsomSure, what is the question?20:15
nmagneziI also posted this in gerrit, what would happen if we have a new controller (with this patch) with an older amp20:15
nmagneziAnd we try to ask for stats20:15
johnsomYou should get an answer as this doesn't connect to the AMP, it just pulls data we already have in the database in a different way.20:16
nmagneziOh, alright. didn't get to test it just yet sadly, just a thought that I had20:17
johnsomYeah, as of now, there is never a connection from the API process directly to an Amphora.  Only the other three processes20:18
johnsomThe background on this patch and the two in tempest-plugin is I needed a gate test that tests out VRRP failover for an internal request.20:19
johnsomThe best way I could come up with to figure out which amp is passing traffic was to look at the per-amphora stats.20:19
johnsomThe gate test works, but I still need to add some tempest API tests to the middle patch in that chain, so I have it WIP right now20:20
johnsomOk, if we don't have any more updates, I will move on.20:21
johnsom#topic Upgrade-checker community goal20:21
*** openstack changes topic to "Upgrade-checker community goal (Meeting topic: Octavia)"20:21
johnsomOne of the two community goals for Stein is to have an "upgrade-checker" script like nova has.20:22
johnsomFrankly I think this is low hanging fruit if someone is looking for a small project.20:22
johnsomThere are details here:20:22
johnsomJust wanted to raise awareness in case someone is looking for a little project.20:23
johnsomI don't think we have much that would need to go in there at the moment.20:23
johnsom#topic Open Discussion20:24
*** openstack changes topic to "Open Discussion (Meeting topic: Octavia)"20:24
johnsomAny other topics today?20:25
cgoncalveswe've received couple of reports these past days that octavia can bring down all loadbalancers due to a DB outage20:26
cgoncalvesI plan to work on this as soon as I can20:26
cgoncalvesa traceback can be read here: bug 1603138 in openstack-octavia "Controller replacement with Octavia- After controller replacement amphora is in ERROR state" [Urgent,Closed: duplicate] - Assigned to cgoncalves20:27
johnsomYeah, bummer.  I have added a comment to that story with an idea of how to approach it20:27
cgoncalvesexamples of operations that can trigger this issue are: node reboot, db service restart, cloud upgrade (causing DB downtime)20:28
cgoncalvesso, whatever causes DB downtime...20:28
johnsomWell, ideally you are running a DB cluster so minor things like a DB restart should not impact Octavia20:28
nmagnezia tmp network disruption can also block the connection to DB20:29
cgoncalvesyeah, I need to check why it still caused that on 3-controller HA deployments20:29
johnsomThere is built in DB retries in the oslo DB layer, so short network blips should not trigger it either.  Plus, with the default settings it would have to be unreachable for 60 seconds or more across all of the HMs20:30
johnsomMaybe there is a bug in galera (if that is what you are using for clustering) or oslo DB.  We should do some testing.20:31
johnsomEither way, I think we should improve how the HMs handle the situation20:32
cgoncalvesagreed. we should catch db_exc.DBConnectionError exceptions in health_check()20:32
xgerman_sorry, had some prod issue to attend...20:32
cgoncalvesxgerman_, let me guess. octavia brought down all LBs due to DB outage? xd20:33
xgerman_no, unrelated to Octavia20:33
nmagneziHe had to look around for zombies20:33
cgoncalvesha! :)20:33
*** luksky11 has joined #openstack-lbaas20:34
johnsomcgoncalves Feel free to ping me if you want to bounce ideas around....20:34
cgoncalvesjohnsom, I most certainly will, thanks :)20:35
johnsomOther topics for today?20:35
xgerman_PTG? Did you plug the etherpad?20:36
xgerman_ah saw it20:37
johnsomYeah, I posted the normal reminder at the start of the meeting20:38
johnsomxgerman_ lol, oh, we will be talking about Octavia.....  (topic 18 in the etherpad)20:38
cgoncalvesI like item "18. Octavia". let's discuss Octavia at the PTG xD20:38
johnsomOk, if we don't have any more topics, I will close out the meeting.20:39
*** abaindur_ has joined #openstack-lbaas20:40
johnsomOk, thanks folks!20:40
*** openstack changes topic to "Discussion of OpenStack Load Balancing (Octavia) |"20:40
openstackMeeting ended Wed Aug 29 20:40:34 2018 UTC.  Information about MeetBot at . (v 0.1.4)20:40
openstackMinutes (text):
*** abaindur has quit IRC20:41
*** yamamoto has quit IRC21:08
rm_workI think I like yours better johnsom, now that we have the version stuff to work with :P21:12
rm_work(that didn't exist when I proposed mine)21:13
johnsomCool. I think it's a good option too21:13
*** celebdor1 has quit IRC21:16
johnsomSurvey Monkey is going IPO...  wow21:19
*** rcernin has joined #openstack-lbaas21:49
*** yamamoto has joined #openstack-lbaas21:53
*** fnaval has quit IRC21:57
*** abaindur_ has quit IRC22:02
*** abaindur has joined #openstack-lbaas22:03
*** yamamoto has quit IRC22:03
*** luksky11 has quit IRC22:03
*** abaindur has quit IRC22:07
*** threestrands has joined #openstack-lbaas22:19
*** threestrands has quit IRC22:19
*** threestrands has joined #openstack-lbaas22:22

Generated by 2.15.3 by Marius Gedminas - find it at!