20:00:14 #startmeeting Octavia 20:00:15 Meeting started Wed Jun 29 20:00:14 2016 UTC and is due to finish in 60 minutes. The chair is johnsom. Information about MeetBot at http://wiki.debian.org/MeetBot. 20:00:16 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 20:00:18 The meeting name has been set to 'octavia' 20:00:25 o/ 20:00:25 Hi folks 20:00:25 Howdy, folks! 20:00:26 o/ 20:00:30 Hi 20:00:33 o/ 20:00:33 o/ 20:00:35 Hi 20:00:39 hi 20:00:44 meeting start? 20:00:51 hi 20:01:03 Yes, I started the meeting 20:01:08 #topic Announcements 20:01:21 Octavia / Neutron-LBaaS mid-cycle 20:01:32 #link https://etherpad.openstack.org/p/lbaas-octavia-newton-midcycle 20:02:17 Also, I have added lbaas-merge RFE bugs with the tag lbaas-merge in launchpad 20:02:17 hello everyone 20:02:24 Nice! 20:02:27 #link https://bugs.launchpad.net/octavia/+bugs?field.tag=lbaas-merge 20:02:40 So we can start tracking those. Please update/expand as needed 20:02:51 Sounds good! 20:03:15 Last announcement I have, I am planning to put in our usual LBaaS/Octavia talk for the summit. 20:03:27 o/ 20:03:34 I think we should cover merge topics (give awareness). 20:03:46 oh yes. 20:03:51 o/ 20:03:53 johnsom: I would like to be involved in that talk, eh. 20:04:00 If you have any other topics you would like to have some time during the talk, please send me topic ideas 20:04:09 Assuming you can stomach sharing a stage with me again. ;) 20:04:16 sbalukoff: heh, and next you're going to state that the world is round. 20:04:23 sbalukoff Are you guys going to be in a place to do a A/A demo? 20:04:56 johnsom: No idea at this point. :/ 20:05:15 going back to the midcycle announcement, please update that etherpad if you're going; there are a lot of tentatives still on the list. 20:05:30 Ok. Well, please send me an e-mail or IRC with your topics so I can get a list of participants/topics for the talk 20:05:37 Ok! 20:05:59 Any other announcements today? 20:06:33 #topic Brief progress reports / bugs needing review 20:06:58 I have been working on cleaning up our bugs list and getting the merge tasks in so we can track them. 20:07:15 I should be back in the coding game this week, so that is going to be nice. 20:07:16 i too have been working on bugs 20:07:27 sry, back, disconnected :-( 20:07:29 Been mostly focused on internal reviews. Will go in and update for my health monitor change in neutron lbaas shortly so we can get that merged soon 20:07:32 verifying and closing out where necesseary 20:07:34 Oh-- apparently there's a new IBM team out of China who are going to be working on LBaaS / Octavia. I couldn't tell from the e-mail I got whether they intend to do a lot of upstream work. Obviously, I'm pushing hard for that to be the case. 20:07:40 * blogan sneaks in late 20:08:02 reviews needed on these - should be quick 20:08:03 #link https://review.openstack.org/#/c/309635/ 20:08:09 #link https://review.openstack.org/#/c/308091/ 20:08:10 I'm also seeing a light at the end of the tunnel regarding the internal work I'm doing. In a week or two I expect to have more time to work on upstream again. 20:08:17 The functional job for lbaas is ready and passing, please review 20:08:23 #link https://review.openstack.org/320999 20:08:26 Though next week I will be out, as I'm taking some vacation after the 4th. 20:08:52 cool, thanks eezhova_! 20:08:58 reviews also needed on those two: 20:08:59 https://review.openstack.org/#/c/299998/ 20:08:59 also if anyone has any spare cycles and would like to take over the TempestPlugin for Neutron LbaaS 20:09:00 #link https://review.openstack.org/#/c/321087/ 20:09:04 And fnaval 20:09:07 https://review.openstack.org/#/c/327966/ 20:09:10 #link https://review.openstack.org/#/c/329507/ 20:09:22 #link https://review.openstack.org/#/c/332913/ 20:09:33 #link https://review.openstack.org/#/c/334374/ 20:09:56 Sweet. 20:10:07 Nice. I like the activity. 20:10:12 these three are quite ready to merge :) 20:10:24 We also have someone updating our DIB elements, so good stuff there too 20:10:38 Yeah, that's great! 20:11:15 Ok, on to our other topics today 20:11:29 #topic Amphora-agent supporting multiple Linux flavors (nmagnezi) 20:11:40 nmagnezi you have the floor 20:11:47 hi guys 20:11:51 #link http://lists.openstack.org/pipermail/openstack-dev/2016-June/098464.html 20:12:11 yup. so the issue is summed up in this mail 20:12:24 i basically have two possible solutions for this 20:12:50 i'm fine with the idea as long as someone on the team is willing to maintain it. adding work for another company, in a vacuum, makes no sense to me. 20:13:24 dougwig, by "adding work" what do you mean exactly 20:13:31 dougwig: +1 20:13:44 dougwig, i intend to work on the patch. can you please tell me what more is needed? 20:13:49 Yeah, I agree with dougwig, someone has to do the work and maintain it. That said, I think there is value in doing a systemd spin as it seems to be the path forward for a number of these distros 20:13:56 nmagnezi: your proposal adds code, potential bugs, maintenance, doubles test matrix, etc. it's not free after the code is submitted. 20:14:39 nmagnezi: i understand redhat doesn't want to ship an ubuntu image. makes sense. but it's not free, and we have to realize and account for that. 20:14:44 dougwig, agreed. and as johnsom said supporting systemd is valuable for more than just "vendor x" :) 20:14:59 its teh future! 20:15:07 dougwig, yup. i understand 20:15:09 i think xenial is systemd, so we're stuck with that steaming pile of overreach, regardless of distro. 20:15:17 nmagnezi: Right. well, the idea here is that "vendor X" should probably be maintaining it. ;) 20:15:29 sbalukoff, :) 20:15:31 dougwig: just wait until the next "better" one comes around 20:15:32 dougwig: Haha! 20:15:35 dougwig Tell us how you really feel about systemd 20:15:35 but wait 20:15:54 what do you guys think about the "do it like lbaas and l3 agent" approach? 20:16:20 It sounds like that ought to simplify start-up woes. 20:16:39 And make for less distro-dependent code. But I could be wrong, I suppose. 20:16:53 Well, I think we have value in the way upstart is setup today (not just because I had a part in that), where the processes can start and restart themselves in dependent 20:17:20 however it may make time to the vip passing traffic slower, as it'd go from native NIC and haproxy to needing the agent to start, and then settle, and then plug. 20:17:28 Yep, it's a lot more work to build these components into a daemon that upstart is already doing for us. 20:17:50 If we went down the agent path, we would need to implement process monitoring for the other parts (haproxy, keepalived, etc.) which upstart provides us today 20:18:01 Yep. 20:18:14 can we just have systemd be the load balancer, and respond to our REST requests? 20:18:19 johnsom, like in https://review.openstack.org/#/c/327966/ ?;) 20:18:28 dougwig, lol 20:19:02 dougwig, sure, anything is possible (as he shakes his head towards dougwig) 20:19:07 johnsom, imo the agent path will eventually be more future proof 20:19:22 i'm just keeping with the design principles of systemd, y'all. don't shoot the messenger. 20:19:30 Hahaha 20:19:40 johnsom, plus it has already proven to work well for two other agents 20:19:41 nmagnezi True. It also makes sequencing very clear. 20:20:00 nmagnezi: i'd like to see the boot time different between ifup/haproxy versus agent+same. 20:20:07 /different/difference/ 20:20:33 dougwig, that's fair. 20:20:53 ideally we'd have a container amphora that was responding to traffic in 2-3 seconds; add an agent on top is significant in that kind of scenario. 20:21:13 dougwig: +1 20:21:20 That's the future I want to see. 20:21:45 will the containers use the same python agent? 20:21:46 I agree, but we aren't dealing with the service VM amp at that point either 20:22:16 nmagnezi: Probably initially. 20:22:39 nmagnezi: Though, of course, especially interface manipulations don't always translate 1:1 between VMs and containers. 20:22:40 johnsom: like we won't just copy the framework over? c'mon, devs are lazy. i mean, efficient. 20:22:44 dougwig: i think emacs already has a systemd plugin to do that 20:22:53 Haha 20:23:01 hah 20:23:03 rm_work: HAHA! 20:23:14 emacs has everything you could possibly want except a good text editor. 20:23:17 Oh, emacs, you had to go there 20:23:26 yep, HAD to. 20:23:37 anyway, I like systemd better than upstart personally >_> 20:23:41 you might need three foot pedals and a head-banging stick to do all the combos quickly, but at least the arrow keys work without switching modes. 20:23:46 so i may be not one to talk 20:23:58 Ok, so I would propose going down the systemd path and not folding this all into the agent. 20:24:13 +1 from me, personally. 20:24:15 okay so I just wanted to know that if I attempt to implement "the agent path" it will be something you will be willing to eventually merge 20:24:21 Makes sense, especially since it's viable in xenial, too. 20:24:26 aye. glad to hear that 20:24:28 as proven by me submitting systemd fixes and having my CR blocked because i refuse to touch upstart >_> 20:24:38 btw that means changes to the way things are done with Ubuntu as well 20:24:52 aside from how the agent itself is started on boot 20:25:03 nmagnezi: i think you're reading johnsom's suggestion and the +1's backwards. :) 20:25:15 Eh... We'll probably want to move the 'default' amphora image beyond trusty at some point soon anyway. 20:25:15 nmagnezi Yes, that is fine as long as it works and meets the functionality 20:26:04 dougwig, i stand corrected. 20:26:14 sbalukoff Yes. I am slightly worried that DIB might do that TO us at some point. Versioning isn't clear in the elements IMHO 20:26:24 Yep. 20:27:03 johnsom, dougwig, to conclude you guys are against the agent path and want me to replicate the use of sysvinit with systemd? 20:27:05 That, and I don't want to have to back-port trusty fixes for the next three years. 20:27:11 based on how long it'll take infra to switch over, i expect we have until sometime in the fall. 20:27:27 Ok, I think we are aligning around the systemd approach. nmagnezi does that cover your topic? 20:27:52 johnsom, yes. 20:28:05 Excellent. thank you for the topic! 20:28:25 #topic Should amphorae be rebootable? 20:28:39 #link https://bugs.launchpad.net/octavia/+bug/1517290 20:28:40 Launchpad bug 1517290 in octavia "Not able to ssh to amphora or curl the vip after rebooting" [High,Opinion] 20:28:59 I did not get a chance to test this to confirm if it is still an issue. Anyone else? 20:29:18 Otherwise, you can slap my wrist and I will try again this coming week. 20:29:28 i still think the answer should be yes, but i believe this horse was beaten into glue. 20:29:37 I was almost entirely on internal stuff this last week. Didn't have time to review. 20:30:17 how far down the making the cattle into pets logic do we go? 20:30:24 I kind of a agree, rebootable is a good thing. I want to test if this ssh issue still exists or not. I think not, so we could kick that can 20:30:25 i like pet cows 20:30:36 they're delicious 20:30:39 Haha 20:30:41 Haha 20:30:58 #topic Load balancer upgrades (ihrachys) 20:31:01 my cat is 20. i can clearly never let go. 20:31:12 #link http://lists.openstack.org/pipermail/openstack-dev/2016-June/098471.html 20:31:19 dougwig: how many times have you rebooted your cat? 20:31:23 5 20:31:27 I know he can't make it to the meeting, but he said he will read the notes. 20:31:28 I heard him talk about it 20:31:52 yeah cattle may not be pets, but you still immunize them and put them inside during storms and such, right? >_> 20:32:03 because it'd be bad if your herd all froze and died 20:32:17 Oh, that's a long e-mail! 20:32:17 I think the one issue that comes to mind for me is the sticky table sync would be lost 20:32:18 to extend the metaphor a bit <_< 20:32:20 ihar's suggestion was to use fip's as a zero-downtime upgrade mechanism. it'd work, except for the many deployments that don't have fip's. 20:32:29 i got through half of ihar's email and then got pulled away 20:32:36 I still wonder about outage time with the FIP update as well 20:32:44 oh lol i'm on like a 2 minute delay because i'm scrolled up slightly >_< 20:32:44 oh but fips are defocre required soon! 20:33:10 defacto? 20:33:15 my cloud doesn't use fip's. it's uses provider nets. i don't mind an optional mechanism on top of FIPs, but i'd hate to see something requiring them (and thus requiring NAT as well.) 20:33:25 i think he meant defcore. 20:33:27 I hate FIPs. They're not IPv6 friendly. :/ 20:33:28 dougwig: your cloud won't be an openstack cloud then, sorry you lose 20:33:30 not defcore optional? wow, suck. 20:33:47 I am not a fan of FIPs either 20:33:55 it'll be a year until that becomes required 20:34:00 neutron's deployment advice is heading away from fip's. and now we're requiring them? when provider nets don't support them? what a mess. 20:34:12 blogan: So a year to fight and delay that for another year... 20:34:15 its apparently the only way defcore sees clouds should give out external IPs 20:34:34 Yeah, that's stupid. 20:34:42 FIPs should be available and used exceptionally, not the normal case. IMO. 20:34:57 Because, again, can we please join the 2000's and *think* about IPv6 already? 20:34:58 well i've successfully gotten us off topic, back to ihar's suggestion 20:35:16 sadly, whether we rely on FIPs goes to the heart of the suggestion. 20:35:30 yeah true 20:35:31 I need to read through that e-mail. 20:35:34 sbalukoff IPv6 is broken in octavia btw. 20:35:38 as i was reading it, i was wondering if a port/vif swap couldn't achieve something similar? 20:35:42 johnsom: Quiet, you! 20:35:42 sounds like a pluggable approach 20:36:02 It should be an easy fix 20:36:04 johnsom: Hopefully there's a bug open about this? 20:36:05 dougwig: i was thinking of the same thing, kind of what we do on failover 20:36:09 I don't think fips as the normal case is the right way either. After mostly-skimming the email 20:36:10 which is kind of what he was saying wouldnt' work 20:36:20 Ok, so let's take some time to read through and reply to the e-mail. 20:36:32 dougwig: I'd prefer the port/vif swap if that's viable. 20:36:47 Ok.... so create an action item? 20:36:58 Sure 20:37:03 sbalukoff: tahts what failover does currently 20:37:14 Everyone go to the mailing list and bitch about how FIPs suck. Also, feedback on Ihar's suggestion. 20:37:19 but yeah i guess everyone should read the email 20:37:27 #action everyone read over Ihar's e-mail about amp upgrades and reply to the e-mail 20:37:55 #topic Open Discussion 20:37:56 let's get systemd to assign the FIPs, and make an audio representation with pulseaudio. then we can all die in a vat of mediocrity. 20:38:07 Hi guys, I have a patch that somehow breaks Octavia driver here #link https://review.openstack.org/#/c/326018, could you please have a look? 20:38:10 dougwig: Haha 20:38:29 #link https://review.openstack.org/#/c/326018 20:38:40 evgenyf: I'll try to get to that this week. 20:38:51 evgenyf: should we be amused that your own CI failed on it? 20:38:58 Heh! 20:39:06 sbalukoff: thanks 20:39:49 Anything more on Ihar's suggestion? 20:39:55 dougwig: our CI is broken , had no chance to fix it, hope to do it soon 20:40:20 Oh wait, sorry, didn't see the topic change. 20:40:50 Er... anyone have anything else they'd like to discuss at this time? 20:40:52 Yep, open discussion time 20:41:07 haproxy (with lbaas agent) related patches up for review: 20:41:09 sbalukoff Any update on the docs work? 20:41:09 #link https://review.openstack.org/#/c/327966/ 20:41:16 #link https://review.openstack.org/#/c/299998/ 20:41:22 I think we only saw one patch so far 20:41:30 johnsom: No progress in the last couple weeks. Been on internal stuff (and so has the docs team) 20:41:32 Yes, some patch for neutron flavors plugin 20:41:35 oh btw i stand corrected about defcore requiring fips, it got pulled out 20:41:38 #link https://review.openstack.org/#/c/318153 20:41:44 got everyone all riled up for nothing 20:42:05 that's half the fun 20:42:14 blogan that figures.... 20:42:16 routers are still in though 20:42:20 johnsom: well, the good news is I'm getting a lot of experience on deploying Octavia in a non-devstack environment, meaning that doc at least ought to be fairly fleshed out. :) 20:42:26 * blogan hides in his hobbit hole 20:42:35 sbalukoff Awesome! 20:42:40 blogan you know damn well you're too big for one of those 20:42:53 blogan: did systemd get ripped out in favor of init, too? if so, i'll end today happy. 20:43:07 johnsom: But yes, once this internal stuff is done, I intend to spend some time on that, and then having docs people fix my broken English. 20:43:13 dougwig: in some distros they never went to systemd, so you can be happy 20:43:22 Hahaha. I will help too 20:43:23 And also herding that new IBM China team into doing a lot of upstream work, if I can. :) 20:43:41 TrevorV: my species is Hobbitus Gigantis 20:44:00 at some point there will be a big pile of work in taking the neutron-lbaas tests and running them against the octavia api. unless we've already duped them all. 20:44:12 We haven't. 20:44:20 Right 20:44:21 if you're looking for a big chunk of upstream that needs loooking at, sbalukoff 20:44:34 dougwig: Right! 20:44:36 dougwig: if we had the passthrough or proxy thing working then we'll buy some time on that 20:44:58 I'm fairly certain we'll need the passthrough proxy in any case. 20:45:09 Because LBaaSv2 and Octavia don't *exactly* line up. 20:45:13 blogan: you mean make octavia jobs that just use the neutron shim? yes, but then we'd have pretty limited coverage on the far half of the "just switch endpoints" message. 20:45:24 Though they are damned close. 20:45:29 sbalukoff: no, but one of the efforts is to get tehir APIs to line up 20:45:36 Yep, I know. 20:46:00 it'd be funny if we made the neutron api shim be haproxy... in an amp! 20:46:07 HAHA 20:46:12 #link https://bugs.launchpad.net/octavia/+bug/1596639 20:46:12 Launchpad bug 1596639 in octavia "Align Octavia API to Neutron-LBaaS APIs" [High,New] 20:46:39 dougwig: true, in that case, it'd just be a matter of switching the tests to use a different service in teh catalog 20:46:53 i have a question about octavia gates. how will a patch that changes the amphora agent get tested? isn't the amphora image deployed with octavia deployed from pypi? (meaning master code) 20:46:59 once octavia accepts true nlbaas calls 20:47:24 nmagnezi: no, devstack installs it from zuul, so via your git change. and the image is built on the fly. 20:47:24 nmagnezi: no, we build it -- though there is a bug i think still where it builds from master and not the patchset specified? 20:47:38 ^^ did we fix the "from master" issue? 20:47:52 i was looking into it at some point and had some code that did that... 20:47:55 rm_work I was just trying to remember if we got that fixed or not 20:47:56 rm_work: really? why aren't we using the octavia that's already *on disk* ? 20:48:05 dougwig: because we just aren't 20:48:06 sec 20:48:08 let me find my fix 20:48:35 there's a file in tree that determines what version of the octavia code the amp image should have 20:48:47 and since its in tree, its always pulling master 20:48:48 is there? 20:48:50 ah yes 20:48:56 https://review.openstack.org/#/c/317756/2/elements/amphora-agent-ubuntu/extra-data.d/50-copy_local_source_files 20:48:58 so 20:49:00 this "works" 20:49:08 but it was part of another effort that i abandoned 20:49:10 we could pull it out 20:49:25 https://github.com/openstack/octavia/blob/master/elements/amphora-agent/source-repository-amphora-agent 20:49:38 There is a DIB override that should be used 20:49:43 yeah see this CR and ignore like half of it: https://review.openstack.org/#/c/317756/2 20:49:47 blogan, that git url breaks infra's CI rules, btw. 20:50:09 dougwig: well thats good to know, a year later! 20:50:17 Heh! 20:50:23 Someone just put a patch for that 20:50:51 but short answer is -- i fixed it, but then dropped that effort instead of getting it merged :/ 20:51:14 rm_work: dropped it bc it wasn't working out well? or time? 20:51:25 it was just a tiny piece of the amp-minifying effort 20:51:28 and i got retasked 20:51:44 THAT small part worked fine 20:51:52 dougwig, rm_work, thanks for the answer guys. yet i must admit that I didn't get how exactly the patch that should be tested ends up in the amphora disk 20:51:54 i can pull it out and resubmit 20:52:17 nmagnezi: yeah so what ends up happening is the amp agent changes get tested the next time AFTER your patch merges ;p 20:52:22 so, basically hilarity ensues 20:52:23 nmagnezi: it doesn't, rm_work was dong a patch that would fix that issue 20:52:31 Sure. Probably worthwhile looking at the patch johnsom was just talking about, too. 20:52:50 oh, so this is why i didn't get it :D 20:53:04 Yeah, that's... less than ideal. 20:53:08 I don't understand much with the DIB, this might be of some help https://review.openstack.org/#/c/331392/ 20:53:13 nmagnezi: i think what some have done is update that file i linked above to their git revew and push up a test change to see veryting works fine, and then change it back and push up their review. an awful workflow 20:53:43 blogan, wow.. 20:53:51 Yeah. I can take this as an action too 20:54:00 i'll see if i can get this done ASAP 20:54:07 probably tomorrow, as i have internal fires today 20:54:08 Well, in our defense, there aren't that many projects that use VM images like this. :/ 20:54:19 As part of the service delivery, I mean. 20:54:21 since i already have working code for it 20:54:29 unless someone else wants to take it over 20:54:35 the ssh driver never had this problem :) 20:54:39 but, examples are all there :P 20:54:43 blogan: +2 20:54:47 blogan: +3 20:54:48 * blogan watches the sshexit 20:54:50 Haha 20:55:00 * rm_work votes Yes for Sshexit 20:55:12 wait, or is it the other way around 20:55:16 I don't even 20:55:19 blogan -5 20:55:56 I don't understand what Sshexit is, but it has a catchy name so i'll vote for it anyway 20:56:00 frito +100? 20:56:09 What else can we do to sabotage our own project? 20:56:16 HAHA 20:56:18 Ok, rm_work are you going to take this issue? 20:56:18 if i vote for sshexit, do we fork the project and dump all REST? 20:56:33 johnsom: yeah i'll try to tackle that today 20:56:36 or tomorrow 20:56:38 maybe today 20:56:39 the project is pluggable, why fork 20:56:39 Ok 20:56:43 amusingly i've been doing a lot of flask lately, and like it much more than pecan. 20:56:52 dougwig: yes flask is much better 20:56:53 but alas 20:56:55 Any other topics before dougwig proposes a move to ruby AGAIN? 20:57:01 dougwig: try falcon 20:57:05 note we did the agent in Flask :P 20:57:07 dougwig: Why have an API anyway, let's just spawn some kind of load balancer from a neutron agent. 20:57:08 blogan: why fork? because i don't trust this core team, of course. they're shifty. 20:57:08 dougwig: ^^ 20:57:28 yeah I don't trust those cores either. not one bit. 20:57:30 sbalukoff: let's just build the lb into DVR ports. 20:57:48 dougwig: Exactly! I mean, it's a network function anyway, right? 20:57:48 (welcome to Octavia Open Discussion) 20:57:51 have we digressed enough? 20:57:56 The thing I don't like about flask is the funky port stuff they do. If it restarts it abandons the TCP port in a bad way 20:57:59 blogan: can we vote on that? 20:58:10 johnsom: that's just if you're using their builtin server 20:58:18 johnsom: i run flask through gunicorn, it's just uswgi 20:58:24 *uwsgi 20:58:30 ugwsi 20:58:36 so... you're saying then... it's all.... unicorns? 20:58:37 uwigw 20:58:38 Ah, good, it's a mess with what we have now 20:58:46 ...and rainbows. 20:58:46 johnsom: yeah werkzeug is shit 20:58:48 Ok, two minutes 20:58:57 (gunicorn is a cheap knockoff of the ruby version. i'll just leave that out there.) 20:59:05 Ah, right it's werk 20:59:05 Haha 20:59:08 * blogan rolls eyes 20:59:20 Ok. thanks folks! 20:59:22 Ok, y'all! Thanks for the meeting, eh! 20:59:24 johnsom: actually that'd be a fix.... 20:59:31 johnsom: run gunicorn inside the amp lol 20:59:32 we're at time 20:59:35 thankfully 20:59:40 HAHA 20:59:41 lol 20:59:45 o/ 20:59:48 ruby fork ftw! 20:59:50 #endmeeting