Wednesday, 2019-08-28

lxkongjohnsom, rm_work, hi guys, do you think it's useful to provide an admin API to reset the lb status? e.g. from pending to error?00:07
lxkongsometimes we have some lbs  in pending_create or update status that the users can't do anything for that00:08
lxkongas a result, we have to use sql api...00:08
johnsomlxkong: no, we decided to fix that a better way at the ptg00:37
johnsomlxkong: also, make sure you gracefully shutdown your controllers, no kill -9. That is pretty much the only way modern versions can have pending_* status when a controller isn’t working on it00:48
lxkongjohnsom: yes, we realised it's probably because the octavia-worker is restarted or killed when handling the lb creating or updating00:53
lxkongjohnsom: is there any news for the ' better way'?00:53
johnsomIt is WIP, Ann is lead on it. It is the jobboard work.00:58
johnsomGraceful shutdowns will not cause that btw, check you init scripts to make sure they only do graceful shutdowns01:00
*** sapd1 has joined #openstack-lbaas02:18
lxkongjohnsom: ok, thanks for the advise02:24
*** ricolin has joined #openstack-lbaas02:25
*** psachin has joined #openstack-lbaas03:10
*** ramishra has joined #openstack-lbaas03:17
*** ianychoi has quit IRC03:29
*** ianychoi has joined #openstack-lbaas03:30
*** gcheresh has joined #openstack-lbaas03:48
*** gcheresh has quit IRC04:04
*** sapd1 has quit IRC04:14
*** tkajinam_ has joined #openstack-lbaas04:37
*** tkajinam has quit IRC04:40
rm_workIt kills me a little bit though, but I'm trying to be strong lxkong -- I feel your pain, but I know fixing the short term pain by adding such an API would be negative long term04:59
*** ramishra has quit IRC05:49
*** ramishra has joined #openstack-lbaas05:53
*** mkuf has joined #openstack-lbaas06:48
*** trident has quit IRC07:08
*** rcernin has quit IRC07:12
*** trident has joined #openstack-lbaas07:17
*** ivve has joined #openstack-lbaas07:28
*** tkajinam_ has quit IRC08:02
*** sapd1 has joined #openstack-lbaas09:24
openstackgerritAnn Taraday proposed openstack/octavia master: Transition member flows to use dicts
openstackgerritAnn Taraday proposed openstack/octavia master: [WIP] Lb flows to dicts
openstackgerritAnn Taraday proposed openstack/octavia master: [WIP] Jobboard based controller
*** ricolin has quit IRC10:25
*** tesseract has joined #openstack-lbaas11:12
*** sapd1 has quit IRC11:14
*** boden has joined #openstack-lbaas11:42
*** Guest98786 is now known as v1k0d3n11:52
*** v1k0d3n is now known as Guest296011:53
*** Guest2960 has quit IRC11:56
openstackgerritAnn Taraday proposed openstack/octavia master: Convert listener flows to use provider models
*** ricolin has joined #openstack-lbaas12:50
*** boden has quit IRC12:58
openstackgerritAnn Taraday proposed openstack/octavia master: Convert pool flows to use dicts
openstackgerritAnn Taraday proposed openstack/octavia master: Convert pool flows to use dicts
*** boden has joined #openstack-lbaas13:20
*** Vorrtex has joined #openstack-lbaas14:05
*** openstackgerrit has quit IRC14:22
*** amotoki_ has quit IRC14:34
*** amotoki has joined #openstack-lbaas14:35
*** Vorrtex has quit IRC15:09
*** Vorrtex has joined #openstack-lbaas15:10
*** ivve has quit IRC15:24
*** ash2307 has joined #openstack-lbaas15:28
*** ash2307 has left #openstack-lbaas15:28
*** ataraday_ has joined #openstack-lbaas15:30
johnsom#startmeeting Octavia16:00
openstackMeeting started Wed Aug 28 16:00:08 2019 UTC and is due to finish in 60 minutes.  The chair is johnsom. Information about MeetBot at
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.16:00
*** openstack changes topic to " (Meeting topic: Octavia)"16:00
openstackThe meeting name has been set to 'octavia'16:00
johnsomHi folks16:00
johnsomIt might be a quiet week as I know a few people are out today.16:01
rm_workah hey, so my appointment was not the time i thought lol16:01
rm_workso i'm here :D16:01
johnsomAh, cool. Well, I put an agenda together. Do you want to run the meeting or should I continue16:01
rm_worknah, go ahead ;)16:02
* johnsom knows the answer but will ask anyway16:02
johnsom#topic Announcements16:02
*** openstack changes topic to "Announcements (Meeting topic: Octavia)"16:02
johnsomStarting off, it saddens me to say, but Nir has stepped down as an Octavia core.16:03
johnsomHe has been working on neutron-lbaas and octavia for some time, so it's sad that he is leaving the project.16:03
rm_workindeed :(16:03
*** ricolin has quit IRC16:04
johnsomHe is changing jobs, so needed to make the change.16:04
*** ricolin has joined #openstack-lbaas16:04
johnsomAlso, TC and PTL nominations for the "U" cycle are now open.16:04
rm_workwoo! one of ya'll can do this annoying job ;) lol16:05
johnsomLet me or Adam know if you have any questions about the process/role16:05
rm_workoooh, johnsom again? :D16:05
mugsiejohnsom again, definitely16:05
johnsommugsie I hear you are now available. Finally moving on to the fancy project?16:06
mugsieI am happy in my no PTL job world :P16:06
johnsommugsie for Octavia PTL!16:06
colin-lots of time now that he's not doing designate!16:07
rm_workwell so far nothing has caught fire under my watch, so maybe I can do it again, lol16:07
johnsomI am just waiting for mugsie to propose C# as an OpenStack supported language16:07
* mugsie writes that down for April 116:08
*** tesseract has quit IRC16:08
johnsomFinally, my weekly reminder that feature freeze is upon us.16:08
johnsomWe must have features merged by Sept 2nd for library changes and Sept. 9th for everything else.16:09
johnsomAny other announcements this week?16:09
johnsom#topic Brief progress reports / bugs needing review16:10
*** openstack changes topic to "Brief progress reports / bugs needing review (Meeting topic: Octavia)"16:10
johnsomI have been working on the failover flow task. It is coming along, but slow due to distractions.16:11
*** ajay33 has joined #openstack-lbaas16:11
johnsomSpeaking of, I also proposed a patch that changes the devstack plugin to use a dual certificate authority with an intermediate CA. This is intended to help us test more like a production deployment.16:12
*** ricolin has quit IRC16:12
johnsomNote, it does use weak security (1024 bit keys, aes128, etc.) to save gate entropy and CPU cycles. So no copy/paste please.  Refer to the certificates guide for how to do it better.16:13
ataraday_I rebased listener, pool, lb, amphora, member flows to dict and test against joboard - all work properly, so I need to create heathmonitor change, and check l7 policy changes - that this refactor work will be complite.16:13
johnsomNice! After I get through this failover fix I plan to focus back on the jobboard work for a bit.16:14
johnsomataraday_ How do you feel about merge timing for those? Are these things you really want to get into Train or can they wait for the start of "U"?16:14
ataraday_johnsom, I will probably ask you advise this week - we need to have housekeepling and healthmanager using worker v2 as well, how better to organize this...16:15
ataraday_johnsom, I guess this should wait to get proper reviews16:15
johnsomataraday_ Ok, happy to help as I can. The failover is a priority for me.16:16
ataraday_as none of my changes are merged for now16:16
johnsomataraday_ We can still do reviews and polish the patches. We would just potentially delay the merge to the start of U16:16
johnsomIn fact, I would highly support doing reviews on these16:17
ataraday_yes, reviews are really appreciated :)16:17
johnsomataraday_ Ok, thank you. Sorry I didn't get to spend as much time helping as I had hoped.16:17
johnsomAny other updates today?16:18
ataraday_johnsom, no problem - issues with redis took a lot of time unfortunatly16:18
johnsomYeah, I still want to look at adding etcd support. Maybe I can do that in "U"16:18
johnsomIf there are no more updates, I will move on to Open Discussion16:19
johnsom#topic Open Discussion16:20
*** openstack changes topic to "Open Discussion (Meeting topic: Octavia)"16:20
johnsomAlso please remember, we have a priority patch review list we are working against:16:20
ataraday_I'm afraid etc will have the same issue as redis as it is key-value storage after all, and zookeper has its own claim check - which makes it work easier16:20
johnsomWe have mad incredible progress, thank you!16:20
johnsomataraday_ So from your experience, zookeeper is currently the best backend for our jobboard work?16:21
johnsomAre there any other topics this week?16:23
colin-briefly want to echo lxkong's sentiment yesterday16:23
colin-he and others (myself included) have run into situations where resources become stuckin an immutable state and the database updates necessary to clean them are (for me, at least) sometimes scary16:23
ataraday_johnsom, I was testing only on devstack, but it seems better - as with redis we need to intoduce listerners, claims and retries on our side16:23
colin-fully appreciate that there is graceful shutdown logic designed to prevent these cases but also recognize that i'm encountering them16:24
colin-will try to do better about quantifying when and how in a story so that it's less anecdotal in the future16:24
ataraday_and with zookeper just set proper type of backend16:24
johnsomcolin- If you see that occurring outside of a non-graceful shutdown I would be interested to see a bug report and the logs16:24
johnsomcolin- Are you on queens or newer?16:25
johnsomataraday_ ok, maybe we should re-evaluate our Redis decision now that we have more information. At least for the development phase. I don't want us spending a bunch of extra time due to a backend selection.16:26
ataraday_Anyway I hope to make this configrable via conf which backend to use, so that it can be tested properly on scale, larger env, etc...16:26
johnsom+1 to that16:26
johnsomAny other topics today?16:29
johnsomOk, thanks folks! Have a great week.16:30
*** openstack changes topic to "Discussions for OpenStack Octavia | Priority bug review list:"16:30
openstackMeeting ended Wed Aug 28 16:30:36 2019 UTC.  Information about MeetBot at . (v 0.1.4)16:30
openstackMinutes (text):
colin-sorry i dropped off! i am on rocky at the moment16:36
colin-and fully understood about the bug reports and logs to substantiate them16:36
colin-wouldn't expect anything without those first :)16:37
johnsomAh, ok. Then yeah, I wouldn't expect issues aside from the force kill16:37
colin-and when you say that you're talking about manually stopping any of the octavia processes, right? API, worker, healthmanager, housekeeping, amphora-agent?16:38
johnsomWhat I am talking about is anything that is stopping any one of the controller processes via kill -9, power off, or killing a container, etc. If you instead do a kill -15 it will do a graceful shutdown and you will not have the issue.16:39
johnsomI.e. make sure you didn't create systemd units that kill -9....16:39
colin-understood, yeah that is not something we ever practice so i'm confident it's not happening in our case. thanks for the details16:43
colin-never have needed to kill any of them honestly16:43
colin-and will ensure the unit files don't either but restarts are infrequent too16:44
*** salmankhan has joined #openstack-lbaas16:51
*** salmankhan has quit IRC17:13
*** ramishra has quit IRC18:12
*** psachin has quit IRC18:13
*** ajay33 has quit IRC18:20
*** ivve has joined #openstack-lbaas19:20
*** ataraday_ has quit IRC19:22
*** openstackgerrit has joined #openstack-lbaas19:32
openstackgerritMichael Johnson proposed openstack/octavia master: Update the load balancing cookbook
rm_workdentist appointment, be back in a while, slightly dented19:38
johnsomQuestion is will you mumble while you type?19:38
rm_work*we'll have to find out19:39
gregworkjohnsom: do you expose load balancer logs to tenants ?19:44
gregworkif i stand up a lb in my tenant .. can i see what its doing for my traffic19:44
johnsomgregwork As of Train you can:
gregworkwas that backported into queens (osp13) ?19:45
johnsomThat was my new feature for the Train release19:45
johnsomNo, per the OpenStack stable release policy, we cannot backport features to stable releases.19:46
gregworkheh i was hoping it was going to be available in the LTS release19:46
johnsomWell, OpenStack doesn't really have LTS releases. It would be up to the distro if they decide to backport, etc.19:47
rm_workLTS for Octavia is "master" :D19:48
gregworkyeah i was referring to the distro i am using19:48
johnsomlol, well, that is kind of true...19:48
rm_workI'm planning to support the master branch indefinitely... ;)19:48
gregwork(being redhat openstack platform 13)19:48
johnsomYeah, I can't speak for distros (at least in this forum), but you certainly can request that of your distro.19:49
johnsomHere I have my community hat on... grin19:49
johnsomDifferent times, different forums I have my red hat on. lol19:51
*** atmark has quit IRC20:07
*** mloza has joined #openstack-lbaas20:08
*** boden has quit IRC21:08
*** Vorrtex has quit IRC21:16
*** dayou has quit IRC21:51
*** ivve has quit IRC22:03
*** dayou has joined #openstack-lbaas22:05
openstackgerritMichael Johnson proposed openstack/octavia master: Fix base (VRRP) port abandoned on revert
*** tkajinam has joined #openstack-lbaas22:51
*** rcernin has joined #openstack-lbaas23:21
openstackgerritMichael Johnson proposed openstack/octavia master: Fix base (VRRP) port abandoned on revert
rm_workthat sounds icky23:32
johnsomMore side effect of the network driver doing too much in one call23:32
johnsomThat is a simple fix until the would network driver interface gets a reboot.23:33
*** Swami has joined #openstack-lbaas23:44
*** Swami has quit IRC23:45

Generated by 2.15.3 by Marius Gedminas - find it at!