| lxkong | johnsom, rm_work, hi guys, do you think it's useful to provide an admin API to reset the lb status? e.g. from pending to error? | 00:07 |
|---|---|---|
| lxkong | sometimes we have some lbs in pending_create or update status that the users can't do anything for that | 00:08 |
| lxkong | as a result, we have to use sql api... | 00:08 |
| johnsom | lxkong: no, we decided to fix that a better way at the ptg | 00:37 |
| johnsom | lxkong: also, make sure you gracefully shutdown your controllers, no kill -9. That is pretty much the only way modern versions can have pending_* status when a controller isn’t working on it | 00:48 |
| lxkong | johnsom: yes, we realised it's probably because the octavia-worker is restarted or killed when handling the lb creating or updating | 00:53 |
| lxkong | johnsom: is there any news for the ' better way'? | 00:53 |
| johnsom | It is WIP, Ann is lead on it. It is the jobboard work. | 00:58 |
| johnsom | Graceful shutdowns will not cause that btw, check you init scripts to make sure they only do graceful shutdowns | 01:00 |
| *** sapd1 has joined #openstack-lbaas | 02:18 | |
| lxkong | johnsom: ok, thanks for the advise | 02:24 |
| *** ricolin has joined #openstack-lbaas | 02:25 | |
| *** psachin has joined #openstack-lbaas | 03:10 | |
| *** ramishra has joined #openstack-lbaas | 03:17 | |
| *** ianychoi has quit IRC | 03:29 | |
| *** ianychoi has joined #openstack-lbaas | 03:30 | |
| *** gcheresh has joined #openstack-lbaas | 03:48 | |
| *** gcheresh has quit IRC | 04:04 | |
| *** sapd1 has quit IRC | 04:14 | |
| *** tkajinam_ has joined #openstack-lbaas | 04:37 | |
| *** tkajinam has quit IRC | 04:40 | |
| rm_work | It kills me a little bit though, but I'm trying to be strong lxkong -- I feel your pain, but I know fixing the short term pain by adding such an API would be negative long term | 04:59 |
| *** ramishra has quit IRC | 05:49 | |
| *** ramishra has joined #openstack-lbaas | 05:53 | |
| *** mkuf has joined #openstack-lbaas | 06:48 | |
| *** trident has quit IRC | 07:08 | |
| *** rcernin has quit IRC | 07:12 | |
| *** trident has joined #openstack-lbaas | 07:17 | |
| *** ivve has joined #openstack-lbaas | 07:28 | |
| *** tkajinam_ has quit IRC | 08:02 | |
| *** sapd1 has joined #openstack-lbaas | 09:24 | |
| openstackgerrit | Ann Taraday proposed openstack/octavia master: Transition member flows to use dicts https://review.opendev.org/657842 | 10:07 |
| openstackgerrit | Ann Taraday proposed openstack/octavia master: [WIP] Lb flows to dicts https://review.opendev.org/671725 | 10:08 |
| openstackgerrit | Ann Taraday proposed openstack/octavia master: [WIP] Jobboard based controller https://review.opendev.org/647406 | 10:08 |
| *** ricolin has quit IRC | 10:25 | |
| *** tesseract has joined #openstack-lbaas | 11:12 | |
| *** sapd1 has quit IRC | 11:14 | |
| *** boden has joined #openstack-lbaas | 11:42 | |
| *** Guest98786 is now known as v1k0d3n | 11:52 | |
| *** v1k0d3n is now known as Guest2960 | 11:53 | |
| *** Guest2960 has quit IRC | 11:56 | |
| openstackgerrit | Ann Taraday proposed openstack/octavia master: Convert listener flows to use provider models https://review.opendev.org/660236 | 12:49 |
| *** ricolin has joined #openstack-lbaas | 12:50 | |
| *** boden has quit IRC | 12:58 | |
| openstackgerrit | Ann Taraday proposed openstack/octavia master: Convert pool flows to use dicts https://review.opendev.org/665381 | 12:59 |
| openstackgerrit | Ann Taraday proposed openstack/octavia master: Convert pool flows to use dicts https://review.opendev.org/665381 | 13:10 |
| *** boden has joined #openstack-lbaas | 13:20 | |
| *** Vorrtex has joined #openstack-lbaas | 14:05 | |
| *** openstackgerrit has quit IRC | 14:22 | |
| *** amotoki_ has quit IRC | 14:34 | |
| *** amotoki has joined #openstack-lbaas | 14:35 | |
| *** Vorrtex has quit IRC | 15:09 | |
| *** Vorrtex has joined #openstack-lbaas | 15:10 | |
| *** ivve has quit IRC | 15:24 | |
| *** ash2307 has joined #openstack-lbaas | 15:28 | |
| *** ash2307 has left #openstack-lbaas | 15:28 | |
| *** ataraday_ has joined #openstack-lbaas | 15:30 | |
| johnsom | #startmeeting Octavia | 16:00 |
| openstack | Meeting started Wed Aug 28 16:00:08 2019 UTC and is due to finish in 60 minutes. The chair is johnsom. Information about MeetBot at http://wiki.debian.org/MeetBot. | 16:00 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 16:00 |
| *** openstack changes topic to " (Meeting topic: Octavia)" | 16:00 | |
| openstack | The meeting name has been set to 'octavia' | 16:00 |
| johnsom | Hi folks | 16:00 |
| colin- | howdy | 16:00 |
| johnsom | It might be a quiet week as I know a few people are out today. | 16:01 |
| rm_work | ah hey, so my appointment was not the time i thought lol | 16:01 |
| rm_work | so i'm here :D | 16:01 |
| gthiemonge | Hi | 16:01 |
| johnsom | Ah, cool. Well, I put an agenda together. Do you want to run the meeting or should I continue | 16:01 |
| johnsom | ? | 16:01 |
| rm_work | nah, go ahead ;) | 16:02 |
| * johnsom knows the answer but will ask anyway | 16:02 | |
| rm_work | ^_^ | 16:02 |
| johnsom | #topic Announcements | 16:02 |
| *** openstack changes topic to "Announcements (Meeting topic: Octavia)" | 16:02 | |
| ataraday_ | hi | 16:02 |
| johnsom | Starting off, it saddens me to say, but Nir has stepped down as an Octavia core. | 16:03 |
| johnsom | #link http://lists.openstack.org/pipermail/openstack-discuss/2019-August/008785.html | 16:03 |
| johnsom | He has been working on neutron-lbaas and octavia for some time, so it's sad that he is leaving the project. | 16:03 |
| rm_work | indeed :( | 16:03 |
| *** ricolin has quit IRC | 16:04 | |
| johnsom | He is changing jobs, so needed to make the change. | 16:04 |
| *** ricolin has joined #openstack-lbaas | 16:04 | |
| johnsom | Also, TC and PTL nominations for the "U" cycle are now open. | 16:04 |
| johnsom | #link http://lists.openstack.org/pipermail/openstack-discuss/2019-August/008679.html | 16:04 |
| rm_work | woo! one of ya'll can do this annoying job ;) lol | 16:05 |
| johnsom | Let me or Adam know if you have any questions about the process/role | 16:05 |
| rm_work | oooh, johnsom again? :D | 16:05 |
| mugsie | johnsom again, definitely | 16:05 |
| rm_work | yisss | 16:06 |
| johnsom | mugsie I hear you are now available. Finally moving on to the fancy project? | 16:06 |
| mugsie | I am happy in my no PTL job world :P | 16:06 |
| johnsom | mugsie for Octavia PTL! | 16:06 |
| johnsom | lol | 16:06 |
| colin- | lots of time now that he's not doing designate! | 16:07 |
| rm_work | well so far nothing has caught fire under my watch, so maybe I can do it again, lol | 16:07 |
| johnsom | I am just waiting for mugsie to propose C# as an OpenStack supported language | 16:07 |
| johnsom | grin | 16:07 |
| * mugsie writes that down for April 1 | 16:08 | |
| *** tesseract has quit IRC | 16:08 | |
| johnsom | nice | 16:08 |
| johnsom | Finally, my weekly reminder that feature freeze is upon us. | 16:08 |
| johnsom | We must have features merged by Sept 2nd for library changes and Sept. 9th for everything else. | 16:09 |
| johnsom | #link https://releases.openstack.org/train/schedule.html | 16:09 |
| johnsom | Any other announcements this week? | 16:09 |
| johnsom | #topic Brief progress reports / bugs needing review | 16:10 |
| *** openstack changes topic to "Brief progress reports / bugs needing review (Meeting topic: Octavia)" | 16:10 | |
| johnsom | I have been working on the failover flow task. It is coming along, but slow due to distractions. | 16:11 |
| *** ajay33 has joined #openstack-lbaas | 16:11 | |
| johnsom | Speaking of, I also proposed a patch that changes the devstack plugin to use a dual certificate authority with an intermediate CA. This is intended to help us test more like a production deployment. | 16:12 |
| johnsom | #link https://review.opendev.org/#/c/678923/ | 16:12 |
| *** ricolin has quit IRC | 16:12 | |
| johnsom | Note, it does use weak security (1024 bit keys, aes128, etc.) to save gate entropy and CPU cycles. So no copy/paste please. Refer to the certificates guide for how to do it better. | 16:13 |
| ataraday_ | I rebased listener, pool, lb, amphora, member flows to dict and test against joboard - all work properly, so I need to create heathmonitor change, and check l7 policy changes - that this refactor work will be complite. | 16:13 |
| johnsom | Nice! After I get through this failover fix I plan to focus back on the jobboard work for a bit. | 16:14 |
| johnsom | ataraday_ How do you feel about merge timing for those? Are these things you really want to get into Train or can they wait for the start of "U"? | 16:14 |
| ataraday_ | johnsom, I will probably ask you advise this week - we need to have housekeepling and healthmanager using worker v2 as well, how better to organize this... | 16:15 |
| ataraday_ | johnsom, I guess this should wait to get proper reviews | 16:15 |
| johnsom | ataraday_ Ok, happy to help as I can. The failover is a priority for me. | 16:16 |
| ataraday_ | as none of my changes are merged for now | 16:16 |
| johnsom | ataraday_ We can still do reviews and polish the patches. We would just potentially delay the merge to the start of U | 16:16 |
| johnsom | In fact, I would highly support doing reviews on these | 16:17 |
| ataraday_ | yes, reviews are really appreciated :) | 16:17 |
| johnsom | ataraday_ Ok, thank you. Sorry I didn't get to spend as much time helping as I had hoped. | 16:17 |
| johnsom | Any other updates today? | 16:18 |
| ataraday_ | johnsom, no problem - issues with redis took a lot of time unfortunatly | 16:18 |
| johnsom | Yeah, I still want to look at adding etcd support. Maybe I can do that in "U" | 16:18 |
| johnsom | If there are no more updates, I will move on to Open Discussion | 16:19 |
| johnsom | #topic Open Discussion | 16:20 |
| *** openstack changes topic to "Open Discussion (Meeting topic: Octavia)" | 16:20 | |
| johnsom | Also please remember, we have a priority patch review list we are working against: | 16:20 |
| johnsom | #link https://etherpad.openstack.org/p/octavia-priority-reviews | 16:20 |
| ataraday_ | I'm afraid etc will have the same issue as redis as it is key-value storage after all, and zookeper has its own claim check - which makes it work easier | 16:20 |
| johnsom | We have mad incredible progress, thank you! | 16:20 |
| johnsom | ataraday_ So from your experience, zookeeper is currently the best backend for our jobboard work? | 16:21 |
| johnsom | Are there any other topics this week? | 16:23 |
| colin- | briefly want to echo lxkong's sentiment yesterday | 16:23 |
| colin- | he and others (myself included) have run into situations where resources become stuckin an immutable state and the database updates necessary to clean them are (for me, at least) sometimes scary | 16:23 |
| ataraday_ | johnsom, I was testing only on devstack, but it seems better - as with redis we need to intoduce listerners, claims and retries on our side | 16:23 |
| colin- | fully appreciate that there is graceful shutdown logic designed to prevent these cases but also recognize that i'm encountering them | 16:24 |
| colin- | will try to do better about quantifying when and how in a story so that it's less anecdotal in the future | 16:24 |
| ataraday_ | and with zookeper just set proper type of backend | 16:24 |
| johnsom | colin- If you see that occurring outside of a non-graceful shutdown I would be interested to see a bug report and the logs | 16:24 |
| johnsom | colin- Are you on queens or newer? | 16:25 |
| johnsom | ataraday_ ok, maybe we should re-evaluate our Redis decision now that we have more information. At least for the development phase. I don't want us spending a bunch of extra time due to a backend selection. | 16:26 |
| ataraday_ | Anyway I hope to make this configrable via conf which backend to use, so that it can be tested properly on scale, larger env, etc... | 16:26 |
| johnsom | +1 to that | 16:26 |
| johnsom | Any other topics today? | 16:29 |
| johnsom | Ok, thanks folks! Have a great week. | 16:30 |
| johnsom | #endmeeting | 16:30 |
| *** openstack changes topic to "Discussions for OpenStack Octavia | Priority bug review list: https://etherpad.openstack.org/p/octavia-priority-reviews" | 16:30 | |
| openstack | Meeting ended Wed Aug 28 16:30:36 2019 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 16:30 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/octavia/2019/octavia.2019-08-28-16.00.html | 16:30 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/octavia/2019/octavia.2019-08-28-16.00.txt | 16:30 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/octavia/2019/octavia.2019-08-28-16.00.log.html | 16:30 |
| ataraday_ | thanks! | 16:31 |
| colin- | sorry i dropped off! i am on rocky at the moment | 16:36 |
| colin- | and fully understood about the bug reports and logs to substantiate them | 16:36 |
| colin- | wouldn't expect anything without those first :) | 16:37 |
| johnsom | Ah, ok. Then yeah, I wouldn't expect issues aside from the force kill | 16:37 |
| colin- | and when you say that you're talking about manually stopping any of the octavia processes, right? API, worker, healthmanager, housekeeping, amphora-agent? | 16:38 |
| johnsom | What I am talking about is anything that is stopping any one of the controller processes via kill -9, power off, or killing a container, etc. If you instead do a kill -15 it will do a graceful shutdown and you will not have the issue. | 16:39 |
| johnsom | I.e. make sure you didn't create systemd units that kill -9.... | 16:39 |
| colin- | understood, yeah that is not something we ever practice so i'm confident it's not happening in our case. thanks for the details | 16:43 |
| colin- | never have needed to kill any of them honestly | 16:43 |
| colin- | and will ensure the unit files don't either but restarts are infrequent too | 16:44 |
| *** salmankhan has joined #openstack-lbaas | 16:51 | |
| *** salmankhan has quit IRC | 17:13 | |
| *** ramishra has quit IRC | 18:12 | |
| *** psachin has quit IRC | 18:13 | |
| *** ajay33 has quit IRC | 18:20 | |
| *** ivve has joined #openstack-lbaas | 19:20 | |
| *** ataraday_ has quit IRC | 19:22 | |
| *** openstackgerrit has joined #openstack-lbaas | 19:32 | |
| openstackgerrit | Michael Johnson proposed openstack/octavia master: Update the load balancing cookbook https://review.opendev.org/679128 | 19:32 |
| rm_work | dentist appointment, be back in a while, slightly dented | 19:38 |
| johnsom | o/ | 19:38 |
| johnsom | Question is will you mumble while you type? | 19:38 |
| rm_work | we | 19:39 |
| rm_work | *we'll have to find out | 19:39 |
| gregwork | johnsom: do you expose load balancer logs to tenants ? | 19:44 |
| gregwork | if i stand up a lb in my tenant .. can i see what its doing for my traffic | 19:44 |
| johnsom | gregwork As of Train you can: https://docs.openstack.org/octavia/latest/admin/log-offloading.html#tenant-flow-logs | 19:45 |
| gregwork | was that backported into queens (osp13) ? | 19:45 |
| johnsom | That was my new feature for the Train release | 19:45 |
| gregwork | o | 19:46 |
| johnsom | No, per the OpenStack stable release policy, we cannot backport features to stable releases. | 19:46 |
| gregwork | heh i was hoping it was going to be available in the LTS release | 19:46 |
| johnsom | Well, OpenStack doesn't really have LTS releases. It would be up to the distro if they decide to backport, etc. | 19:47 |
| johnsom | FYI, https://releases.openstack.org/ | 19:48 |
| rm_work | LTS for Octavia is "master" :D | 19:48 |
| gregwork | yeah i was referring to the distro i am using | 19:48 |
| johnsom | lol, well, that is kind of true... | 19:48 |
| rm_work | I'm planning to support the master branch indefinitely... ;) | 19:48 |
| gregwork | (being redhat openstack platform 13) | 19:48 |
| johnsom | Yeah, I can't speak for distros (at least in this forum), but you certainly can request that of your distro. | 19:49 |
| johnsom | Here I have my community hat on... grin | 19:49 |
| johnsom | Different times, different forums I have my red hat on. lol | 19:51 |
| *** atmark has quit IRC | 20:07 | |
| *** mloza has joined #openstack-lbaas | 20:08 | |
| *** boden has quit IRC | 21:08 | |
| *** Vorrtex has quit IRC | 21:16 | |
| *** dayou has quit IRC | 21:51 | |
| *** ivve has quit IRC | 22:03 | |
| *** dayou has joined #openstack-lbaas | 22:05 | |
| openstackgerrit | Michael Johnson proposed openstack/octavia master: Fix base (VRRP) port abandoned on revert https://review.opendev.org/679153 | 22:34 |
| *** tkajinam has joined #openstack-lbaas | 22:51 | |
| *** rcernin has joined #openstack-lbaas | 23:21 | |
| openstackgerrit | Michael Johnson proposed openstack/octavia master: Fix base (VRRP) port abandoned on revert https://review.opendev.org/679153 | 23:23 |
| rm_work | that sounds icky | 23:32 |
| johnsom | More side effect of the network driver doing too much in one call | 23:32 |
| johnsom | That is a simple fix until the would network driver interface gets a reboot. | 23:33 |
| *** Swami has joined #openstack-lbaas | 23:44 | |
| *** Swami has quit IRC | 23:45 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!