16:00:14 #startmeeting Octavia 16:00:15 Meeting started Wed Jul 1 16:00:14 2020 UTC and is due to finish in 60 minutes. The chair is johnsom. Information about MeetBot at http://wiki.debian.org/MeetBot. 16:00:16 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 16:00:18 The meeting name has been set to 'octavia' 16:00:25 o/ 16:00:26 Hi everyone 16:00:28 hi 16:00:29 hi 16:00:56 #topic Announcements 16:01:06 I don't really have any announcements this week. 16:01:35 We finally got a stable/train release out the door. I think it had been nine months since the last release. 16:01:40 * johnsom shames himself 16:01:56 Any other announcements this week? 16:02:56 #topic Brief progress reports / bugs needing review 16:03:43 I create patch with experimental job with amphorav2 https://review.opendev.org/#/c/737993/ and find out that several things is/got broken. 16:03:52 #link https://review.opendev.org/#/c/738609/ 16:04:03 Aside from reviews, rebases, and the occasional small bug fix, I have been focusing on failover for amphorav2. It's a bit slow going, but making progress. 16:04:42 there is also issue with barbican tls, still looking into it.. 16:04:54 I think after we have that landed there are going to be some patches to simplify and clean up stuff. I'm trying to not go crazy doing that now as I don't want another monster patch and want to move faster on this patch. 16:05:08 Nice, thank you. 16:05:12 johnsom, Thanks for proposing failover for amphorav2! 16:05:23 I'm also looking into why the IPv6 job is timing out. sigh 16:05:37 ataraday_ Still a lot to be done, but work in progress 16:06:36 I extended the amphora flavor capabilities to add amp_image_tag. this is useful for multi-architecture clouds and testing amphora images in staging environments, for example. as part of that work I also removed some deprecated options and create an image driver interface (noop and glance drivers) 16:07:05 Yeah, nice! I think I have reviewed about half of that now 16:07:19 I'm revisiting the failover threshold thingy 16:07:31 #link https://review.opendev.org/#/c/656811/ 16:07:45 Nice, also helpful 16:07:46 I successfully deployed via devstack Octavia in an arm64/aarch64 system, although the amphora agent is not coming up in Zuul CI. this is a side project, thus low priority for me 16:08:10 isn't TrevorV working on that stuff? aarch64 16:08:26 Trevor is working on power 16:08:32 ahh he's on ppc, ok 16:09:04 There was a mailing list post about the ppc work and Octavia I mentioned last week. I think I was the only one to reply 16:09:10 OpenDev has aarch64 nodepool nodes, thanks to Linaro! 16:09:16 what's a "mailing list" 16:09:21 lol 16:09:32 I forgot what email was after Google killed Inbox 16:09:33 RIP Inbox 16:09:34 You sign up for coupons with one 16:09:43 RIP email 16:09:44 IT'S A TRAP! 16:10:18 ooo nice, aarch64 testing resources is good :) do we also have ppc? 16:10:47 I found a clever way to run our CI jobs in nest-virt enabled nodes, cutting job time from close to 2 hours down to as low as 38 minutes 16:10:53 #link https://review.opendev.org/#/c/738246/ 16:10:58 I don't think so, not in zuul at least. Red Hat has some ppc stuffs 16:11:01 yeah I just +2'd that, seems workable 16:11:35 yeah, we have also ppc systems internally but, as you said, TrevorV has been working on that 16:12:07 Yeah, I don't have cycles to put into that at the moment 16:12:09 * rm_work is just enjoying causing TrevorV's IRC client to possibly ping 16:12:12 Woah woah woah! I haven't touched arm 16:12:18 haha there we go 16:12:26 What'd I miss now? 16:12:28 yes, johnsom pointed out it was ppc :D 16:12:34 Arm either, though I have run our code on a raspberry pi 4 with very good results. 16:12:38 I just remembered you were on some alternate arch 16:12:58 TrevorV, we were saying you've been working on ppc, not arm 16:13:52 ok so moving on 16:13:58 Yep 16:14:09 #topic Open Discussion 16:14:17 Someone seems anxious.... 16:14:44 Any other topics this week? 16:15:16 I asked just before the meeting started a question about the amphorav2. would now be a good time to discuss it? 16:16:01 Sure 16:16:13 ok, I'll paste the question again: 16:16:16 have we considered either renaming the amphorav2 provider to amphora or alias amphora->amphorav2 once we switch the default to amphorav2? 16:16:21 I think it is a logical choice to make that change at some point, but given the extra infrastructure I'm not sure aliasing amphorav2 to amphora is a good idea. 16:16:29 * johnsom pastes his answer again 16:16:50 ok, so you're not for aliasing. are you then for renaming or? 16:17:07 That said, I might look at if we can have the v2 path jobboard/extra requirements optional. It seems like it should be do-able 16:18:05 Interested in what people think.... 16:18:15 Yeah I had some concerns here 16:18:51 If we deprecate/remove the amphora (v1) driver, we are going to be explicitly choosing to make our default deployment more complex than previously (introducing a second required service) 16:19:11 if we DON'T, we will forever have two code-paths, and that sucks, so I don't think it's really an option T_T 16:19:51 Yeah, we have to deprecate the v1 path. It's just not workable long term IMO 16:19:53 SO going from that: if we deprecate the old version, and force people over to the new version, given the new complexity/requirements, they MUST explicitly switch over for it to work 16:20:49 so for people who do upgrades without reading release-notes, if they upgrade from say X->Y release and v1 becomes v2, it's just going to explode but not "cleanly"/"clearly" 16:21:19 rather than "the provider does not exist" it'll be some error nestled down inside the worker where it tries and fails to connect to unconfigured redis 16:22:18 We can put some tooling in to warn people or make it obvious early. I think Ann already has updated the upgrade check tool. 16:22:33 * johnsom notes, upgrade check tool nobody runs.... 16:22:35 ataraday_ has a pre-upgrade check for the amphorav2 provider that may be handy 16:22:37 #link https://review.opendev.org/#/c/735556/ 16:22:41 yeah 16:23:26 We can set it up to check on startup and fail to start if it's not present as well. 16:23:52 fail to start if a flavor is using a provider that doesn't exist? 16:24:38 Or the default is v2, yeah, basically. It's ugly, but possible 16:25:59 anyway, yeah, if we rename/alias automatically then it really hides stuff from the operator that I don't think we should be hiding 16:26:41 So I think the original question was about the name. I assume for upgrade reasons. Is there a need to rename it from amphorav2? 16:26:54 amphorav2 should be able manage amphorav1-created resources, correct? when we remove the amphorav1 code, we could either do a db migration to change the provider driver of existing LBs to amphorav2 or rename amphorav2 to amphora (with the bonus that we don't potentially break things for the user as the provider name would not change) 16:26:56 i mean, it does look kinda weird 16:27:52 I guess so, with the caveat that we prevent the service from starting if everything for amphorav2 isn't configured properly? 16:28:05 which kinda handles that in a more up-front way 16:28:13 my question comes from a deployment side as I started work to support the amphorav2 in tripleo 16:28:47 yeah ok i'm coming around, maybe we do alias it, and deal with my concerns via the service startup checking its config is valid 16:29:11 same as "can i connect to SQL" and "can I connect to RMQ", add "can I connect to Redis/Zookeeper" 16:29:15 I think we should also look at making the "jobboard" part of the v2 driver optional. 16:29:24 hmm that is the other posibility 16:29:38 I actually would prefer that but I didn't know if THAT was feasible, as it's pretty baked in 16:29:42 We should be able to run the new flows just like we have, without the need for redis, and have a bit-flip for jobboard 16:30:06 yeah ok I think that is my #1 preferred option -- and then yes, alias amphora to amphorav2 16:30:09 Really it's about how we launch the flows. 16:30:21 and keep the default to False for "use_jobboard" 16:31:00 without jobboard we won't be able to resume jobs 16:31:18 right 16:31:19 so why we will may need that? 16:31:27 Right, that would be the trade off of that setting 16:31:35 so Octavia is still possible to run without Redis 16:31:41 in this case we may leave amphora 16:31:45 and using no additional upgrade requirements 16:32:12 Pierre Riteau proposed openstack/octavia master: Add debootstrap installation instructions for CentOS https://review.opendev.org/738885 16:32:15 basically that allows for what I wanted as far as "keeping v1 and v2 around" except we don't ACTUALLY have to keep v1 around, and consolidate code paths 16:32:29 because the ONLY advantage of v1 was not requiring Redis 16:34:38 Yeah, I really don't want to keep the v1 code around. That is asking for mistakes to be made and doubles work effort (I'm feeling the pain now, lol) 16:35:16 yep 16:35:31 i absolutely do not want to keep it around, for the record 16:35:56 +1. amphorav2++ 16:35:59 I just hated the idea of having no way to run without jobboard (if you want a simpler install, with the possibility of stuff stuck in PENDING) 16:36:09 So, maybe in parallel someone can look at making that config setting? Or I could look at it after failover is done. 16:37:09 I think it's just making a method that runs the flows like the v1 driver does or like the v2 driver does, depending on the config setting. 16:38:03 I have little to none (more like none TBH, lol) understanding on the amphorav2/jobboard but I could maybe take a look 16:38:31 this may make code really twisted 16:38:46 cgoncalves Ok cool. I can give you a few pointers to what I'm thinking. 16:38:52 * cgoncalves l 16:38:56 oops! 16:39:24 yeah, my concern (and why I didn't ask for this originally) was that it might not even be possible or would make things incredibly more complex within v2 16:39:33 ataraday_ Yeah, I may be overlooking something, but I think we should give it a shot. 16:40:21 we needed feedback on metric selection for the amphoras.. can we take that up next? 16:40:28 ok, so decision was: yes to alias, and try to make v2 have a "jobboardless" option? 16:40:45 if feasible, I'd advocate for use_jobboard=true as default. devstack can set redis up and is our recommendation for production environments, right? 16:40:53 Or was it "alias if we can make the jobboard part optional"? 16:40:58 hmm maybe that 16:41:20 yes, let's plan to take that topic up next aannuusshhkkaa! 16:41:21 Yeah, we should push for it as the default 16:41:33 I am unsure I agree 16:42:12 but I guess it is as simple as "oh, my service won't start due to an error that says I don't have redis configured and might need to turn off use_jobboard" and then they do that 16:42:21 I think it is a question of timing. 16:42:24 rm_work, before this conversation, amphorav2 was already set to become the default in Victoria or later release so either way Redis would be required 16:43:21 yeah which I never liked 16:43:24 We need to make a call by the end of ms2 because we need to send an e-mail out about the need for Redis to the deployment tools have time to add it. 16:43:36 ok 16:43:38 you get the bonus now that there may be a chance jobboard/redis not to be mandatory :) 16:43:47 lets do some research on this and see if it's even possible to make it optional? 16:44:02 then revisit before ms2 16:44:05 Ok, yeah, I agree 16:44:12 MS2 is coming up fast though 16:44:33 kk 16:44:35 Week of July 27th 16:44:39 can we move on to metrics then? 16:44:44 yes! 16:44:47 #link https://releases.openstack.org/victoria/schedule.html 16:44:47 :D 16:45:09 metrics, that's why you were anxious earlier :D 16:45:14 I think so. What is up with metrics? I know there is a patch that needs some reviews 16:45:17 heh 16:45:22 here is the list of metrics we are thinking of implementing: 16:45:22 Must Haves: 16:45:22 CPU Usage (Current %) 16:45:22 Load Averages 16:45:22 RAM Usage (some combo of: total / free / available / cached / etc) 16:45:23 Nice to Haves: 16:45:23 Disk usage (used/free? or used%?) 16:45:24 Random extra HAProxy fields (taking suggestions) 16:45:25 are we good on the ones we have selected? are we missing something? have we included something that isn’t plausible? 16:45:26 yes, we have one patch up that changes the interfaces around slightly 16:45:53 ack, in the future you should paste multi-line stuff into something like http://paste.openstack.org/ 16:46:18 Do we need load averages? Personally I would like to keep the data minimal, so just %'s 16:47:01 I feel like it's generally more useful than "point in time" CPU usage 16:47:03 Yeah, you can get booted off the server to too much multi-line pastes. 16:47:22 rm_work gotcha 16:48:16 johnsom, haha okay! thanks for the correction.. 16:48:32 Ok, if you have a use for it. 16:48:48 Well, it's super easy for a single point in time CPU % to be totally off 16:48:53 aannuusshhkkaa The server can consider you a spam bot basically. 16:49:09 yeap that makes sense.. will keep that in mind.. 16:49:43 Yeah, but we will get samples every 10 seconds or so. 16:49:48 whereas load averages are very nice locally generated averages that we can keep collecting at a much slower interval and still have them be useful 16:50:09 yeah if we were collecting every second, doing our own averages might make more sense, but... 16:50:40 Yeah, maybe. You would have to then get the number of cores to calculate a percent 16:51:12 hmm yes that is true 16:51:30 possibly do THAT locally? and return load average %s 16:51:41 rather than have to ship that info up and calculate later 16:52:05 That could work. 16:52:17 anyway, we will look into options for that -- any comments about the others? RAM numbers that are actually useful? 16:52:26 Linux "free memory" is basically a useless metric 16:52:29 AFAICT 16:52:47 but I don't know exactly which combination of RAM metrics is *actually* most useful 16:53:02 Yeah, really I'm looking for % of available memory 16:53:13 available/total 16:53:28 cache and free, not so useful to me 16:55:16 okay.. what about disk usage? used %s? 16:56:20 I would assume used% is better possibly? though I also wonder if without context that could be not so useful if it says 50% and then you wonder "at what rate will that fill" 16:56:46 Rate is why you have time series 16:57:08 IMO 16:57:25 timeseries/deltas 16:58:12 A simple scaling driver could have just thresholds and ignore rate. A fancy driver could build a rate and make decisions. 16:58:59 Then at some point someone can add AI and ML, then build a model of what works and doesn't. Then we retire 16:59:14 lol 16:59:29 so that means... percentage? or used/total? lol 16:59:39 Oh, just about out of time for the meeting. We can continue after if you still have questions. 16:59:47 Or defer to next week. 16:59:52 yes we do.. 17:00:04 next week would probably be a little too late.. 17:00:10 #endmeeting