Wednesday, 2020-06-17

johnsomWe might want to make that stevedore a "pass the data to multiple drivers" model as opposed to a single driver.00:04
rm_workthat helps clarify things a bit00:12
rm_workwe can discuss the specific metrics we want to add another time, soon?00:13
rm_workbut we can start with the assumption that it'll be at LEAST cpu and ram00:13
johnsomSure. The only two I know of are CPU and RAM. We don't even need to necessarily store them or figure out what to do with them now, just that we start collecting them.00:14
johnsomIn Shanghai there was some discussion of pool level and member level stats like we have for load balancers and listeners, but I'm not sure I get the concept of those, plus I think it may be hard if not impossible to collect. Also UDP may not be able to provide them.00:16
rm_workwe're going to have to introduce "system" level stats00:17
johnsomI'm not sure if that was a "wouldn't it be neat" or "I have a use case for this" type of thing00:17
rm_workoutside of listeners/pools00:17
rm_workalso, system level stats will NOT be deltas00:17
rm_workat least, most of them won't00:18
johnsomYeah. Correct00:18
rm_workwhich means we may also have to provide a setting per stat in the structure to say whether it's actually a delta or not00:18
rm_workfor everything00:18
johnsomUmm, I don't want to fatten the heartbeat message too much. Really the version should determine how the message is parsed.00:19
rm_workright but....00:19
rm_workwell ok00:19
rm_worki guess we can just "know" on the driver side on the control-plane00:19
rm_workthat's probably fine00:19
johnsomYeah, heartbeat side is a bit tightly coupled and space constrained00:20
johnsomWhich is another argument for deltas00:20
rm_workadding metrics to the agenda for tomorrow00:25
*** armax has quit IRC00:33
*** yamamoto has quit IRC00:35
*** yamamoto has joined #openstack-lbaas00:35
*** wuchunyang has quit IRC00:38
*** wuchunyang has joined #openstack-lbaas00:39
*** wuchunyang has quit IRC00:51
*** wuchunyang has joined #openstack-lbaas01:20
*** spatel has joined #openstack-lbaas01:21
*** wuchunyang has quit IRC01:31
*** wuchunyang has joined #openstack-lbaas01:48
*** armax has joined #openstack-lbaas01:52
*** wuchunyang has quit IRC02:21
*** wuchunyang has joined #openstack-lbaas02:35
*** wuchunyang has quit IRC02:41
*** wuchunyang has joined #openstack-lbaas02:41
*** psachin has joined #openstack-lbaas02:48
*** rcernin has quit IRC02:49
*** rcernin has joined #openstack-lbaas02:54
*** wuchunyang has quit IRC03:00
*** rcernin has quit IRC03:08
*** wuchunyang has joined #openstack-lbaas03:16
*** wuchunyang has quit IRC03:19
*** psachin has quit IRC03:31
*** psachin has joined #openstack-lbaas03:33
*** armax has quit IRC03:40
*** rcernin has joined #openstack-lbaas03:45
*** rcernin has quit IRC03:45
*** rcernin has joined #openstack-lbaas03:50
*** yamamoto has quit IRC04:19
*** gcheresh has joined #openstack-lbaas04:20
*** yamamoto has joined #openstack-lbaas04:35
*** wuchunyang has joined #openstack-lbaas04:38
*** vishalmanchanda has joined #openstack-lbaas04:42
*** wuchunyang has quit IRC04:52
*** threestrands has joined #openstack-lbaas05:27
*** spatel has quit IRC05:29
*** wuchunyang has joined #openstack-lbaas05:31
*** threestrands has quit IRC05:33
*** wuchunyang has quit IRC05:35
*** wuchunyang has joined #openstack-lbaas06:12
*** rpittau|afk is now known as rpittau06:56
*** JayLiu has joined #openstack-lbaas07:07
*** psachin has quit IRC07:15
*** psachin has joined #openstack-lbaas07:17
*** wuchunyang has quit IRC07:33
*** rcernin has quit IRC07:47
*** ataraday_ has joined #openstack-lbaas07:53
*** maciejjozefczyk has joined #openstack-lbaas08:02
*** maciejjozefczyk has quit IRC08:03
*** maciejjozefczyk has joined #openstack-lbaas08:03
openstackgerritCarlos Goncalves proposed openstack/octavia master: add the verify for the session
*** ramishra has quit IRC08:17
*** JayLiu has quit IRC08:33
*** also_stingrayza is now known as stingrayza08:34
*** salmankhan has joined #openstack-lbaas08:52
*** salmankhan1 has joined #openstack-lbaas08:55
*** tkajinam has quit IRC08:57
*** salmankhan has quit IRC08:57
*** salmankhan1 is now known as salmankhan08:57
openstackgerritCarlos Goncalves proposed openstack/octavia master: Use uwsgi binary from path
openstackgerritCarlos Goncalves proposed openstack/octavia master: add the verify for the session
*** aannuusshhkkaa has quit IRC09:20
*** born2bake has joined #openstack-lbaas09:31
*** ramishra has joined #openstack-lbaas09:37
*** yamamoto has quit IRC10:01
*** yamamoto has joined #openstack-lbaas10:02
*** yamamoto has quit IRC10:02
openstackgerritAnn Taraday proposed openstack/octavia master: Preupgrade check for amphorav2 provider
cgoncalvesataraday_, hi! FYI, you may want to rebase ^ on top of (gate fix)10:08
*** rpittau is now known as rpittau|bbl10:18
*** psachin has quit IRC10:31
*** yamamoto has joined #openstack-lbaas10:38
*** yamamoto has quit IRC10:39
*** yamamoto has joined #openstack-lbaas10:40
*** TMM has quit IRC10:40
*** TMM has joined #openstack-lbaas10:40
ataraday_cgoncalves, Oh, thanks! I though fixes in devstack repo is enough :)10:43
openstackgerritAnn Taraday proposed openstack/octavia master: Preupgrade check for amphorav2 provider
*** yamamoto has quit IRC10:50
*** yamamoto has joined #openstack-lbaas11:00
*** psachin has joined #openstack-lbaas11:03
*** also_stingrayza has joined #openstack-lbaas11:34
*** stingrayza has quit IRC11:36
*** psachin has quit IRC11:58
*** psachin has joined #openstack-lbaas12:00
*** rpittau|bbl is now known as rpittau12:06
*** numans has quit IRC12:14
*** spatel has joined #openstack-lbaas12:31
*** spatel has quit IRC12:36
*** TrevorV has joined #openstack-lbaas13:37
*** rpittau is now known as rpittau|brb13:39
*** kberger_ has joined #openstack-lbaas13:45
*** kberger_ has quit IRC13:46
*** kberger_ has joined #openstack-lbaas13:46
*** KeithMnemonic has quit IRC13:49
*** rpittau|brb is now known as rpittau13:53
*** yamamoto has quit IRC14:06
*** TMM has quit IRC14:22
*** TMM has joined #openstack-lbaas14:23
*** yamamoto has joined #openstack-lbaas14:34
*** yamamoto has quit IRC14:34
*** yamamoto has joined #openstack-lbaas14:34
*** psachin has quit IRC14:34
*** yamamoto has quit IRC14:41
johnsomcgoncalves Thanks for catching that we were specifying a path for uwsgi. I have rage-merged it to unblock the gates14:50
cgoncalvesit was copy-pasta from devstack and confirmed by another designate patch14:52
johnsomYeah, I think some of that was template stuff from the community goal, so not surprised to see it in a few repos.14:55
*** gcheresh has quit IRC15:29
*** aannuusshhkkaa has joined #openstack-lbaas15:57
johnsom#startmeeting Octavia16:00
openstackMeeting started Wed Jun 17 16:00:33 2020 UTC and is due to finish in 60 minutes.  The chair is johnsom. Information about MeetBot at
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.16:00
*** openstack changes topic to " (Meeting topic: Octavia)"16:00
openstackThe meeting name has been set to 'octavia'16:00
johnsomHello everyone16:00
openstackgerritMerged openstack/octavia master: Use uwsgi binary from path
johnsom#topic Announcements16:02
*** openstack changes topic to "Announcements (Meeting topic: Octavia)"16:02
johnsomWell, that was one announcement ^^^^16:02
johnsomuwsgi was broken for devstack recently. That patch should resolve the master branch.16:03
johnsomDoes anyone have any other announcements this week?16:05
rm_workaannuusshhkkaa and shtepanie are joining us for the summer as dev interns at vzm!16:06
johnsomYay! Welcome16:06
cgoncalvesnice, welcome!16:06
shtepaniethank you!16:07
aannuusshhkkaaThank you!! :)16:07
rm_workIn the process of getting them up to speed, and we've got a topic later about what they'll be working on (metrics!)16:07
openstackgerritMerged openstack/octavia master: add the verify for the session
*** armax has joined #openstack-lbaas16:08
johnsomSounds good16:08
johnsom#topic Brief progress reports / bugs needing review16:08
*** openstack changes topic to "Brief progress reports / bugs needing review (Meeting topic: Octavia)"16:08
johnsomI have been focused on catching up on reviews, getting the stable branches - sigh - stable, and cutting some stable branch releases.16:09
ataraday_I was a bit off with internal processes. Now, want to highlight two changes, adding retry and preupgrade check for amphorav216:10
johnsomWe got Ussuri and Stein out of the gate. Train is still broken on grenade issues16:10
rm_workOh was it just this last week that we EOL'd two branches? Or was that already announced16:10
rm_workI'm losing track of time16:10
johnsomOh, yeah, in fact it was!16:11
* johnsom is living in a time warp as well16:11
johnsomWe have officially EOL'd the Ocata and Pike releases of Octavia.16:11
johnsomThanks rm_work for leading that effort and navigating the process waters16:12
cgoncalves+1, thank you16:12
rm_workYou mean breaking through the process wall like koolaid man16:12
johnsomYes, that exactly, lol16:12
rm_workWhich is my preferred style of political negotiation :D16:13
johnsomWell, it was a good thing as we should have truth in advertising and really no one was maintaining those branches anymore.16:13
johnsomI also spent some time looking at the centos amphora images to see if I could find any tricks to speed it up under qemu tcg. I achieved a huge improvement of 17 seconds.16:14
johnsomWhich means, it still takes four minutes to boot and is still a problem.16:15
rm_workWell that's something16:16
johnsomYeah, not worth the trouble really.16:16
johnsomOk, any other progress reports or updates?16:17
johnsomataraday_ Thanks for the patches!16:17
*** rpittau is now known as rpittau|afk16:17
cgoncalvesI spent some time working on diskimage-builder to add centos 8 stream support. centos stream is the rolling  pre-release of RHEL 8 and CentOS 8. there's a WIP patch in octavia side that builds an amphora and runs the tests, all successful16:17
cgoncalvesI also continued to review johnsom's monster patch aka failover refactor patch16:17
rm_workYeah we need to get that in :)16:18
johnsomYeah, I have done a few comment update spins on that16:18
rm_workWe've been running it in prod for over a month now? Multiple months maybe?16:18
rm_workIt's been good16:18
johnsomNice, that is good feedback.16:18
johnsomFor the most part, the comment have been minor issues. I think the biggest change was adding retry timeouts to the configuration file.16:19
johnsomBased on the PTG feedback16:20
johnsomOk, if there are no more updates, we can move on to "metrics"16:20
johnsom#topic metrics16:20
*** openstack changes topic to "metrics (Meeting topic: Octavia)"16:20
johnsomrm_work You have the conn16:20
rm_workSo, we're picking up this task!16:21
rm_workWe discussed it briefly last night, and it seems it's essentially three parts16:21
rm_work... and my irc window doesn't want to scroll back that far, apparently16:22
rm_workanyway, we think it's essentially:16:23
rm_work1) Add new metrics at the system level (for example, RAM usage, CPU usage)16:23
rm_work2) Transition to sending deltas instead of absolute totals, where it makes sense (for things like total connections and transfer bytes, but not for current active or the system stuff probably)16:24
njohnstontdd 916:25
rm_work3) Rework/improve the driver layer to allow running multiple metrics storage drivers at once, and probably add at least one new driver for shipping metrics somewhere like influxdb16:25
johnsom+1 That is the list I am aware of16:26
rm_workWe'll probably tackle 1 and 2 first16:27
rm_workThe discussion topic today though is basically -- can we brainstorm what we actually want for #1?16:27
johnsomYeah, those should go together nicely with a heartbeat protocol version bump16:27
johnsomyeah, that is a good question.16:28
rm_worki listed the two i can think of off the top of my head16:28
johnsomMy first thought is percentages. Simply because the agent would have the best information about the nova flavor of the instance.16:29
rm_workyeah, definitely thinking percentages16:29
johnsomAh, you meant which metrics. Yeah, RAM and CPU are on the top of my list. I personally don't have any others.16:29
rm_workwhich does mean those numbers would be absolute, not deltas16:29
rm_workyeah is there anything else useful we could collect?16:30
cgoncalvesdisk? local logs can pile up16:30
rm_workas an admin, having an at-a-glance of the disk might be useful in that specific situation16:31
johnsomPersonally I think we have other ways to address that (log offloading and hourly rotation), but we have seen one case where some other issue in the cloud filled the system log file with garbage.16:31
rm_workbut i don't know how generally useful that'd be in the 99% case for a user16:31
cgoncalvesok, it was just a thought. we can add later if we want to16:32
rm_worki guess we should clarify the goal16:32
rm_workI THINK what we're trying to do is add metrics that would allow one essential insight: how much "capacity" does my LB have left16:32
johnsomYeah, my goal for those is to get us a step closer to auto-scaling16:33
rm_workand really, I am considering formulas that we could use to turn that into one easily digestable number16:33
rm_workwhich is apparently what AWS does with ELB16:33
johnsomInitially I'm not sure we should even add the "system" metrics to the API. Simply because they have little to no meaning for other provider drivers16:34
rm_workYeah I think I agree16:35
johnsomAnd I don't want us to get in a strange situation when we enable active/active.16:35
rm_workWe should just collect at first16:35
rm_workwhich actually simplifies the task a bit :D16:36
rm_workthen we can handle what to do with those new metrics in step 3, when we ship them somewhere16:36
aannuusshhkkaaAWS offers read/write bandwidth, idle time, latency on EBS. Do we want to offer features like that?16:37
johnsomCorrect. Maybe, if we want to give some indication to the end user, we could consider adding a "HIGH LOAD" operating status, but I would consider that #4 or #20 on the list.16:37
rm_workI wonder if we can actually have any idea what percentage of read/write bandwidth is actually being used16:38
rm_workthat would require an operator config setting, possibly16:38
johnsomRight, that is a hard one given neutron can't usually come close to what we can handle.16:38
rm_workyeah, and even if we know which NIC is in a HV, we don't know what bandwidth is like on the rest of the VMs that live there16:39
johnsomWe do have bytes in/out and with deltas you could calculate the rate16:39
rm_workah, yeah... how do we do deltas, exactly? that is one of my major concerns16:39
rm_workthere's a few concerns there actually16:39
rm_workfirstly, HOW? do we *reset* haproxy's counters constantly?16:40
rm_workdo we keep an internal tracker in the agent?16:40
rm_workI suppose that's just going to be some research16:40
johnsomYeah, my expectation is the agent will keep the previous value in memory and calculate the delta16:40
rm_workalso, since we use UDP, do we just... hope all the packets get there, and possibly under-report?16:41
johnsomYes, this would be a "may under report in some cases" scenario16:41
rm_workwe have a sequence number so on the control-plane we could actually tell if we're missing packets and try to do some fill based on the points on either side... but that could be wrong too16:41
rm_workand better to under-report than over-report i guess16:41
rm_workalso don't want to hugely increase the workload on the heartbeat ingestion16:42
johnsomRight and complex. We do already have a sequence number in the heartbeat message. We just don't use it for more than a nonce16:42
johnsomFYI, here are the metrics haproxy can report:16:43
johnsomHowever, the LVS UDP side cannot support most of those16:43
johnsomSo until we can switch out the UDP engine, that may constrain what we report or we need to call out the limitations.16:44
johnsomI also want to make sure we are careful to not put in things that other drivers don't support. I.e. no haproxy specific metrics.16:44
rm_workah, hrsp_4xx hrsp_5xx etc was something mentioned16:45
rm_workbut I don't know if we want to try to ship those from haproxy, or allow those to be calculated by a user via log analysis16:45
johnsomereq and econ are also candidates16:45
rm_workI believe other things should be able to report those for HTTP type stuff16:45
rm_workwe already ship ereq :)16:46
johnsomAh, ok, so .... grin16:46
rm_workmaybe hanafail?16:46
rm_work"failed health checks details"16:46
rm_workthat is one other use-case16:46
rm_workbut also, can be handled by logs16:46
johnsomYeah, that is in the flow logs16:47
johnsomWe also need to keep in mind the heartbeat message size. I think it is limited to 64k at the moment. That includes both stats and status16:47
johnsomNot that we can't change that, but just a consideration16:48
rm_worklbtot for members would be interesting16:48
rm_work"total number of times a server was selected, either for new16:48
rm_work     sessions, or when re-dispatching"16:48
johnsomYeah, that is per-member hits16:49
rm_workbut anyway, I suppose we can move on, could be here all day :D16:49
rm_workand also, the user could get that info *from their members* :D16:49
johnsomLots of goodies, but we need to be conservative16:49
johnsomOr the flow logs, it's in there16:50
rm_workalright, last thing would be, does anyone ELSE want to work on #3? since that also could probably be done in parallel16:50
rm_work(updating to allow multiple drivers to be used at once, and adding one for influxdb or similar)16:50
johnsomThe hard work there is defining the interface really16:51
rm_workif not, we can look at handling that after we wrap up 1+216:51
rm_workI think the interface is already defined -- technically it's already a driver layer?16:51
rm_workand it takes "our health message" :D16:52
rm_workunless you are saying you want to rework that16:52
rm_workand actually do some level of pre-parsing first16:52
johnsomYeah, I was trying to remember what the content was. It's a de-wrapped heartbeat json isn't it?16:53
rm_workthat would require a decent refactor -- it'd basically mean shifting 90% of the current "update_db" code up above the driver layer16:53
rm_workwhich maybe should be done16:53
rm_workbecause that doesn't really make sense16:53
rm_workwe should have all that pre-parsing outside of the drivers16:53
rm_workand the "update_db" part should literally just be taking the final stats struct, and ... updating the DB16:53
johnsomYeah, we should be able to rev the message format version without requiring all of the drivers to respin IMO. If we can avoid it.16:54
rm_workit's actually pretty badly organized16:54
rm_workok so maybe we MOVE the driver layer there16:54
rm_workdo you think it'd be ok to break our existing plugin agreement there?16:54
rm_worki doubt anyone is using it?16:54
johnsomIt is not a published interface today. We don't document it.16:55
rm_workit's internal to the amphora-driver16:55
rm_workso we'll probably reorganize that first16:55
rm_workwhich i guess actually means parts of #3 will be #016:55
johnsomBut do keep in mind, we have a stats interface for the provider drivers too16:55
rm_workyeah but i believe it is already totally different from the interface i'm referring to16:56
johnsomYeah, it is a bit different16:56
*** maciejjozefczyk has quit IRC16:56
*** maciejjozefczyk has joined #openstack-lbaas16:57
rm_workI am referring to that one16:57
rm_workbecause currently 100% of the logic that parses the packet lives in the "update_db" driver16:58
johnsomYeah, that will need improvement16:58
rm_workwhich is ... not correct16:58
rm_workthat should all happen well before it passes to a driver, and what it should pass is a final structure with data16:58
johnsomI am talking about:16:59
rm_workyeah thats already closer to what an "update_db" driver SHOULD be tho17:00
rm_workso right inside there, we can actually share the driver layer and the struct we pass, I think17:00
johnsomOk, we are out of time today. Thanks for the great discussion and work on metrics!17:00
rm_workdriver layer should be here:
rm_worko/ thanks everyone17:01
*** openstack changes topic to "Discussions for OpenStack Octavia | Priority bug review list:"17:01
openstackMeeting ended Wed Jun 17 17:01:17 2020 UTC.  Information about MeetBot at . (v 0.1.4)17:01
openstackMinutes (text):
*** spatel has joined #openstack-lbaas17:16
*** kberger_ has quit IRC17:20
*** ataraday_ has quit IRC17:21
*** ccamposr has joined #openstack-lbaas17:21
*** ccamposr__ has quit IRC17:24
*** KeithMnemonic has joined #openstack-lbaas17:25
*** KeithMnemonic has quit IRC17:27
*** KeithMnemonic has joined #openstack-lbaas17:37
*** KeithMnemonic has quit IRC17:43
*** spatel has quit IRC18:37
*** spatel has joined #openstack-lbaas18:39
*** spatel has quit IRC19:08
*** vishalmanchanda has quit IRC19:38
*** spatel has joined #openstack-lbaas19:40
*** emccormick_ has joined #openstack-lbaas20:11
*** tobberydberg_ has joined #openstack-lbaas20:12
*** JasonF has joined #openstack-lbaas20:13
*** irclogbot_1 has quit IRC20:14
*** tobberydberg has quit IRC20:18
*** emccormick has quit IRC20:18
*** dtruong has quit IRC20:18
*** JayF has quit IRC20:18
*** emccormick_ is now known as emccormick20:18
*** dtruong has joined #openstack-lbaas20:19
*** irclogbot_1 has joined #openstack-lbaas20:21
*** gcheresh has joined #openstack-lbaas20:30
*** lxkong has quit IRC20:35
*** aannuusshhkkaa has quit IRC20:37
*** shtepanie has quit IRC20:38
*** hemanth_n has quit IRC20:40
*** emccormick has quit IRC20:41
*** emccormick has joined #openstack-lbaas20:45
*** emccormick has quit IRC20:51
*** shtepanie has joined #openstack-lbaas20:52
*** hemanth_n has joined #openstack-lbaas20:57
*** shtepanie has quit IRC20:57
*** maciejjozefczyk has quit IRC21:00
*** JasonF is now known as JayF21:06
*** lxkong has joined #openstack-lbaas21:09
*** spatel has quit IRC21:13
*** lxkong has quit IRC21:16
*** gcheresh has quit IRC21:17
*** salmankhan has quit IRC21:27
*** aannuusshhkkaa has joined #openstack-lbaas21:34
*** lxkong has joined #openstack-lbaas21:34
*** spatel has joined #openstack-lbaas21:38
*** emccormick has joined #openstack-lbaas21:40
*** shtepanie has joined #openstack-lbaas21:42
*** spatel has quit IRC21:43
*** spatel has joined #openstack-lbaas21:45
*** stingrayza has joined #openstack-lbaas21:54
*** also_stingrayza has quit IRC21:57
*** spatel has quit IRC22:02
*** born2bake has quit IRC22:22
*** spatel has joined #openstack-lbaas22:29
*** spatel has quit IRC22:32
*** TrevorV has quit IRC22:36
*** rcernin has joined #openstack-lbaas22:42
*** tkajinam has joined #openstack-lbaas22:54
*** shtepanie has quit IRC23:05
*** ccamposr__ has joined #openstack-lbaas23:15
*** ccamposr has quit IRC23:18
*** ccamposr has joined #openstack-lbaas23:41
*** ccamposr__ has quit IRC23:44

Generated by 2.17.2 by Marius Gedminas - find it at!