| mnaser | johnsom: sorry you're right, this isnt backported to stable/queens | 00:16 |
|---|---|---|
| mnaser | https://review.openstack.org/#/c/548989/ this one | 00:17 |
| johnsom | Ok, I didn't *think* it was, but thought maybe I was crazy. | 00:17 |
| *** abaindur has quit IRC | 00:18 | |
| johnsom | You want to do the work or should I? | 00:18 |
| mnaser | johnsom: it seems to merge conflict when i complain, and im writing a long post mortem to an angry customer so :< | 00:18 |
| johnsom | Yeah, ok, I will get it. | 00:18 |
| *** abaindur has joined #openstack-lbaas | 00:19 | |
| *** yamamoto has quit IRC | 00:34 | |
| johnsom | mnaser Argh, there was a DB schema change before that patch.... | 01:01 |
| *** ramishra has quit IRC | 01:13 | |
| *** annp has joined #openstack-lbaas | 01:14 | |
| *** abaindur has quit IRC | 02:26 | |
| *** ramishra has joined #openstack-lbaas | 02:35 | |
| *** hongbin has joined #openstack-lbaas | 02:36 | |
| openstackgerrit | wangxiyuan proposed openstack/octavia master: Correct the base class for v2 l7rules type tests https://review.openstack.org/619397 | 02:42 |
| openstackgerrit | wangxiyuan proposed openstack/octavia master: Tags support for lb resources https://review.openstack.org/606006 | 02:42 |
| *** ramishra has quit IRC | 03:15 | |
| *** ramishra has joined #openstack-lbaas | 03:35 | |
| *** hongbin_ has joined #openstack-lbaas | 04:23 | |
| *** hongbin has quit IRC | 04:24 | |
| *** hongbin has joined #openstack-lbaas | 04:37 | |
| *** hongbin_ has quit IRC | 04:38 | |
| *** hongbin has quit IRC | 04:40 | |
| *** yamamoto has joined #openstack-lbaas | 04:58 | |
| *** pcaruana has joined #openstack-lbaas | 05:09 | |
| *** yboaron_ has joined #openstack-lbaas | 05:10 | |
| *** abaindur has joined #openstack-lbaas | 05:12 | |
| openstackgerrit | wangxiyuan proposed openstack/octavia master: Tags support for lb resources https://review.openstack.org/606006 | 06:15 |
| *** jarodwl has joined #openstack-lbaas | 06:35 | |
| *** ramishra has quit IRC | 06:48 | |
| *** rcernin has quit IRC | 06:57 | |
| *** ramishra has joined #openstack-lbaas | 07:05 | |
| *** velizarx has joined #openstack-lbaas | 07:15 | |
| *** ccamposr has joined #openstack-lbaas | 07:15 | |
| *** ramishra has quit IRC | 07:18 | |
| *** ramishra has joined #openstack-lbaas | 07:21 | |
| *** velizarx has quit IRC | 07:30 | |
| *** velizarx has joined #openstack-lbaas | 07:41 | |
| *** rpittau has joined #openstack-lbaas | 08:01 | |
| *** abaindur has quit IRC | 08:09 | |
| *** rpittau_ has joined #openstack-lbaas | 08:31 | |
| *** abaindur has joined #openstack-lbaas | 08:31 | |
| *** rpittau has quit IRC | 08:33 | |
| *** rpittau_ is now known as rpittau | 08:35 | |
| *** abaindur has quit IRC | 08:35 | |
| *** rpittau_ has joined #openstack-lbaas | 08:40 | |
| *** rpittau has quit IRC | 08:40 | |
| *** rpittau_ is now known as rpittau | 08:49 | |
| *** yamamoto has quit IRC | 08:56 | |
| *** yboaron_ has quit IRC | 09:04 | |
| *** yboaron_ has joined #openstack-lbaas | 09:04 | |
| *** celebdor has joined #openstack-lbaas | 09:07 | |
| *** yboaron_ has quit IRC | 09:13 | |
| *** velizarx has quit IRC | 09:17 | |
| *** velizarx has joined #openstack-lbaas | 09:18 | |
| *** yboaron_ has joined #openstack-lbaas | 09:25 | |
| *** rpittau has quit IRC | 09:34 | |
| *** yamamoto has joined #openstack-lbaas | 09:43 | |
| *** yamamoto has quit IRC | 10:02 | |
| *** yamamoto has joined #openstack-lbaas | 10:07 | |
| *** tobias-urdin has quit IRC | 10:10 | |
| *** tobias-urdin has joined #openstack-lbaas | 10:15 | |
| *** jarodwl has quit IRC | 10:33 | |
| *** jarodwl has joined #openstack-lbaas | 10:34 | |
| *** salmankhan has joined #openstack-lbaas | 10:35 | |
| *** salmankhan has quit IRC | 10:45 | |
| *** salmankhan has joined #openstack-lbaas | 10:53 | |
| *** yamamoto has quit IRC | 11:57 | |
| *** hvhaugwitz has quit IRC | 12:03 | |
| *** hvhaugwitz has joined #openstack-lbaas | 12:03 | |
| *** yamamoto has joined #openstack-lbaas | 12:11 | |
| *** witek has quit IRC | 12:23 | |
| *** witek has joined #openstack-lbaas | 12:23 | |
| *** celebdor has quit IRC | 12:31 | |
| *** celebdor has joined #openstack-lbaas | 12:35 | |
| *** takamatsu has quit IRC | 12:37 | |
| *** takamatsu has joined #openstack-lbaas | 12:43 | |
| *** celebdor has quit IRC | 13:03 | |
| *** celebdor has joined #openstack-lbaas | 13:04 | |
| *** phuoc has quit IRC | 13:15 | |
| *** phuoc has joined #openstack-lbaas | 13:16 | |
| *** takamatsu has quit IRC | 13:29 | |
| *** takamatsu has joined #openstack-lbaas | 13:30 | |
| *** salmankhan1 has joined #openstack-lbaas | 13:42 | |
| *** salmankhan has quit IRC | 13:43 | |
| *** salmankhan1 is now known as salmankhan | 13:43 | |
| *** takamatsu has quit IRC | 13:44 | |
| *** yamamoto has quit IRC | 13:45 | |
| *** salmankhan has quit IRC | 13:47 | |
| *** takamatsu has joined #openstack-lbaas | 13:48 | |
| *** jarodwl has quit IRC | 13:49 | |
| *** jarodwl has joined #openstack-lbaas | 13:50 | |
| *** yboaron_ has quit IRC | 13:54 | |
| *** salmankhan has joined #openstack-lbaas | 14:06 | |
| *** yamamoto has joined #openstack-lbaas | 14:07 | |
| *** ptoohill1 has quit IRC | 14:56 | |
| *** ivve has joined #openstack-lbaas | 15:21 | |
| *** ramishra has quit IRC | 15:46 | |
| *** rpittau has joined #openstack-lbaas | 16:28 | |
| *** celebdor has quit IRC | 16:52 | |
| *** rpittau has quit IRC | 16:52 | |
| *** pcaruana has quit IRC | 17:03 | |
| *** KeithMnemonic has quit IRC | 17:14 | |
| colin- | was reviewing this document and had a question for the channel related to the direction that flavors will take https://docs.openstack.org/octavia/latest/contributor/specs/version1.0/flavors.html | 17:16 |
| johnsom | Ok. I am working on that code now, so good timing | 17:16 |
| colin- | as i'm looking into nova-lxd and potentially using it with octavia it occurs to me that this direction doc does not address a scenario where we might have two ways to provision an amp (traditional kvm method, as well as nova-lxd) and want to be able to differentiate at lb creation | 17:17 |
| johnsom | Yeah, we have that a bit today with the different compute drivers. You could implement a nova-lxd via a new, or aliased compute driver. Though this may also simply be a configuration we add. | 17:19 |
| johnsom | Basically think of flavors as taking elements of the configuration file (with the config file being the default if not defined in a flavor) and making flavors that apply settings on a per-LB basis. | 17:20 |
| colin- | downstream of the amp agent basically | 17:20 |
| johnsom | Umm, no, upstream. Flavors can be applied/supported by all drivers. | 17:20 |
| johnsom | Not all drivers will implement all options that another driver might | 17:21 |
| colin- | probably chose "downstream" poorly, what i meant was the changes that falvors describe are largely focused on the haproxy configuration rather than the characteristics of the nova resource | 17:22 |
| johnsom | No, I don't think so. Things like filesystem size, number of vCPUs, nova flavor, will all be parameters that can be defined for the amphora driver. | 17:23 |
| *** ccamposr has quit IRC | 17:27 | |
| *** jackivanov has quit IRC | 17:28 | |
| colin- | ok great, that's a big relief as we're excited to be able to use different compute flavor types as well as any new drivers | 17:39 |
| *** kosa777777 has joined #openstack-lbaas | 17:52 | |
| *** rtjure has joined #openstack-lbaas | 18:13 | |
| *** salmankhan has quit IRC | 18:26 | |
| *** fnaval has joined #openstack-lbaas | 18:44 | |
| *** pck has quit IRC | 19:14 | |
| *** abaindur has joined #openstack-lbaas | 19:27 | |
| *** celebdor has joined #openstack-lbaas | 19:45 | |
| johnsom | #startmeeting Octavia | 20:00 |
| openstack | Meeting started Wed Nov 28 20:00:20 2018 UTC and is due to finish in 60 minutes. The chair is johnsom. Information about MeetBot at http://wiki.debian.org/MeetBot. | 20:00 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 20:00 |
| *** openstack changes topic to " (Meeting topic: Octavia)" | 20:00 | |
| openstack | The meeting name has been set to 'octavia' | 20:00 |
| johnsom | Hi folks | 20:00 |
| cgoncalves | ahoy! | 20:00 |
| johnsom | Seems quiet this week. | 20:01 |
| johnsom | #topic Announcements | 20:01 |
| *** openstack changes topic to "Announcements (Meeting topic: Octavia)" | 20:01 | |
| johnsom | Just a friendly reminder, we have a priority review list: | 20:01 |
| johnsom | #link https://etherpad.openstack.org/p/octavia-priority-reviews | 20:01 |
| johnsom | Lots of goodies looking for a review.... | 20:01 |
| johnsom | FYI, I am trying to update it at least once a week | 20:02 |
| johnsom | Otherwise I don't think I have any announcements today. Anyone else? | 20:02 |
| cgoncalves | we released octavia-tempest-plugin 0.2.0 | 20:02 |
| nmagnezi | o/ | 20:02 |
| johnsom | Ah yes, good stuff there. | 20:03 |
| *** celebdor has quit IRC | 20:03 | |
| nmagnezi | johnsom, on that note, I've built 0.2.0 RPMs today so there will get ship in the near future | 20:03 |
| johnsom | There are a few patches up for review on the stable/queens branch too. I hope we can get those in and cut a queens release soon | 20:03 |
| xgerman | o/ | 20:03 |
| johnsom | nmagnezi Nice! | 20:04 |
| johnsom | #topic Brief progress reports / bugs needing review | 20:04 |
| *** openstack changes topic to "Brief progress reports / bugs needing review (Meeting topic: Octavia)" | 20:04 | |
| johnsom | Other than taking Thanksgiving week off, I have been working on reviews and catching up on email, etc. | 20:04 |
| johnsom | I have started looking at the next step for the flavors work as well. | 20:05 |
| *** celebdor has joined #openstack-lbaas | 20:05 | |
| johnsom | Any other updates? | 20:05 |
| * cgoncalves has been fixing the usual suspect, again | 20:05 | |
| * johnsom grumbles "triple-opps" | 20:06 | |
| nmagnezi | lol | 20:06 |
| cgoncalves | I reviewed the tags patch. good stuff there. still need to vote though | 20:06 |
| johnsom | Ok cool, I pitched in my +2 this morning on that. It does look great and he fixed the code duplication issue I posted. | 20:07 |
| cgoncalves | yeah, the mixin. I wanted to leave that comment too | 20:07 |
| cgoncalves | tags was something a vendor also was interested in. hopefully we will be able to pass it down to the provider | 20:08 |
| johnsom | Nope! | 20:08 |
| cgoncalves | ok... | 20:08 |
| cgoncalves | you mean in this patch or never? | 20:08 |
| johnsom | That allows bad behaviors like using tags to configure drivers and such. They get overloaded and it becomes a mess. | 20:08 |
| johnsom | There really isn't a good reason a driver would need access to the tags. | 20:09 |
| johnsom | They should pass in driver config settings via flavors | 20:09 |
| johnsom | never IMO | 20:10 |
| johnsom | What vendor thought they wanted tags? | 20:10 |
| cgoncalves | ok, I don't know the use case they have in mind. via flavors sounds reasonable and likely doable for their needs | 20:10 |
| xgerman | Yep. | 20:12 |
| johnsom | Yeah, the right away to do driver configs is flavors, so it is at least validated, etc. | 20:12 |
| johnsom | It's part of the driver spec | 20:12 |
| xgerman | Indeed. No random stuff passed in | 20:12 |
| johnsom | And coming to a patch review near you soon! grin | 20:12 |
| johnsom | tenant: tag: "vCPUs": 255 | 20:13 |
| johnsom | "I need it fast!" | 20:13 |
| johnsom | grin | 20:13 |
| johnsom | Ok, other updates or on to some topics for this week? | 20:14 |
| johnsom | #topic Brief progress reports / bugs needing review | 20:14 |
| *** openstack changes topic to "Brief progress reports / bugs needing review (Meeting topic: Octavia)" | 20:14 | |
| johnsom | Dang, wrong topic | 20:14 |
| johnsom | #topic Amp failover when failover already occurred (Queens) | 20:14 |
| *** openstack changes topic to "Amp failover when failover already occurred (Queens) (Meeting topic: Octavia)" | 20:14 | |
| xgerman | You are repeating yourself | 20:15 |
| johnsom | #link https://review.openstack.org/#/c/548989/ | 20:15 |
| * johnsom thinks he should not be eating lunch at the same time | 20:15 | |
| johnsom | Back in Rocky we had this handy patch that fixes an issue with LBs that failover twice. | 20:15 |
| nmagnezi | johnsom, bon appétit | 20:15 |
| xgerman | Multi tasking | 20:16 |
| johnsom | Basically if the first attempt failed, say nova issue, then it tries again. The problem is in between housekeeping can get a bit aggressive and purge some useful records. | 20:16 |
| johnsom | I looked at backporting this to Queens (recently a vendor hit is as they were missing another patch), but as it is, this patch depends on some schema changes that were not part of Queens | 20:17 |
| johnsom | <sad face> | 20:17 |
| johnsom | However, I think I can improve the situation for Queens at least some with a partial backport of this. It wouldn't be a true cherry-pick because of the other schema dependencies. | 20:18 |
| johnsom | Before I dig into that I wanted to run it by you all for comment | 20:19 |
| cgoncalves | first, is there a way to easily assert when a patch is not backportable because of schema changes, in this case? what should we do better? | 20:20 |
| johnsom | Ok. So, I don't think it would be right/best to use the cherry pick string, etc. when the bulk of the fix won't be going back. Are we ok with a "new" patch against stable/queens that attempts to fix this? That patch would not go forward to Rocky/Stein as there is a more complete fix already there. | 20:21 |
| xgerman | I am ok with that | 20:21 |
| johnsom | Yeah, I manually cherry-picked and fixed the conflicts, the unit and functional tests blew up due to the missing schema | 20:22 |
| cgoncalves | ok, good | 20:22 |
| johnsom | So, yes, it was obvious there was a required schema change patch that was not on queens | 20:22 |
| cgoncalves | fine with a new patch | 20:22 |
| johnsom | tests=good | 20:23 |
| nmagnezi | Same here | 20:23 |
| johnsom | Ok cool. I will propose something later today that should improve that scenario. | 20:23 |
| johnsom | I will reference the Rocky patch and explain the situation in the commit message. | 20:23 |
| johnsom | I think it's the best answer as folks are using queens as an LTS and this should be fixed. | 20:24 |
| johnsom | #topic VIP ACLs/SGs - continued | 20:24 |
| *** openstack changes topic to "VIP ACLs/SGs - continued (Meeting topic: Octavia)" | 20:24 | |
| johnsom | We ran out of time on this last week, so I added it to this weeks agenda. | 20:25 |
| johnsom | Is there more discussion needed on this topic? | 20:25 |
| johnsom | Going once.... | 20:26 |
| johnsom | Going twice... (we can always discuss more at a future meeting) | 20:26 |
| johnsom | #topic Open Discussion | 20:26 |
| *** openstack changes topic to "Open Discussion (Meeting topic: Octavia)" | 20:26 | |
| johnsom | Other topics for today? | 20:27 |
| johnsom | Ok, then I will get busy on the queens patch and back to doing interesting jsonschema things for flavors. | 20:29 |
| johnsom | Have a good week! | 20:29 |
| johnsom | #endmeeting | 20:29 |
| *** openstack changes topic to "Discussions for Octavia | Stein priority review list: https://etherpad.openstack.org/p/octavia-priority-reviews" | 20:29 | |
| openstack | Meeting ended Wed Nov 28 20:29:56 2018 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 20:29 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/octavia/2018/octavia.2018-11-28-20.00.html | 20:29 |
| nmagnezi | O/ | 20:30 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/octavia/2018/octavia.2018-11-28-20.00.txt | 20:30 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/octavia/2018/octavia.2018-11-28-20.00.log.html | 20:30 |
| cgoncalves | I swear I read it as "johnsom's schema" things | 20:30 |
| johnsom | Well.... It might turn out that way | 20:30 |
| johnsom | I'm kind of excited about it as I think it will be a hugely powerful feature for operators | 20:31 |
| cgoncalves | johnsom's schema? for sure! | 20:33 |
| *** openstackgerrit has quit IRC | 20:36 | |
| xgerman | cgoncalves: did you upload our presentations? | 20:55 |
| cgoncalves | xgerman, I did, in PDF format | 20:55 |
| xgerman | +1 | 20:56 |
| *** celebdor has quit IRC | 20:59 | |
| *** rcernin has joined #openstack-lbaas | 21:50 | |
| *** ccamposr has joined #openstack-lbaas | 21:58 | |
| *** rcernin has quit IRC | 22:03 | |
| *** rcernin has joined #openstack-lbaas | 22:12 | |
| *** ccamposr has quit IRC | 22:24 | |
| colin- | sorry i missed the meeting earlier, wanted to ask if anyone has heard anything about the active/active work? saw ptg notes and reviews/docs it links to but hadn't heard anyone talking about it lately | 22:54 |
| johnsom | colin- I have started that work, but had to re-prioritized it down as we lost some people that were working on other key things like provider drivers and flavors. I will probably pick it up again in Train. Plus, we need Ryu for it and that project is transitioning into OpenStack neutron as project Ken. So, I also wanted the dust to settle there. | 23:01 |
| johnsom | https://github.com/openstack/os-ken | 23:03 |
| colin- | ah interesting, ok | 23:06 |
| colin- | goood to know | 23:06 |
| johnsom | mnaser Taking another look at your queens issue today. Can you check your CW and HM logs for "Failed to update amphora %(amp)s status to ERROR due to: " or "Reverting mark amphora deleted in DB for amp id %(amp)s and compute id %(comp)s" messages for your missing amp records? | 23:06 |
| johnsom | Those are error and warning level logs so you should have them. | 23:08 |
| mnaser | I will dig those once I get back in the office johnsom | 23:13 |
| johnsom | Thank you | 23:13 |
| lxkong | johnsom: hi i am wondering if it's true that we could only delete 'ACTIVE' lbs? for other status, deletion will fail, right? | 23:22 |
| johnsom | No, you can delete ERROR LBs/Objects as well | 23:22 |
| lxkong | johnsom: ah, right, so for other status will return some response(maybe 409) immediately, right? | 23:23 |
| xgerman | yep | 23:23 |
| lxkong | cool, thanks xgerman johnsom | 23:24 |
| johnsom | Correct, all of the PENDING_* statuses will return 409 immediately until they either succeed or the retries expire and they go to ERROR | 23:24 |
| xgerman | in other notes keystone claims in their docs not to return an email with the user GET but it does anyway | 23:25 |
| mnaser | grep 'Failed to update amphora' /var/log/octavia/* yield nothing | 23:44 |
| mnaser | i do see results for : grep 'Reverting mark amphora deleted in DB' /var/log/octavia/* | 23:45 |
| johnsom | mnaser Enough "Reverting" messages for the missing amps? | 23:46 |
| mnaser | johnsom: only for 3 | 23:47 |
| mnaser | and i remember there was 3 ERROR state LBs at the time | 23:47 |
| johnsom | Ok, so that is the 3 others we knew about. | 23:47 |
| johnsom | Any chance a CW or HM got kill -9'd? | 23:47 |
| mnaser | johnsom: maybe we attempted to restart the services to figure out whats going on | 23:51 |
| *** fnaval has quit IRC | 23:51 | |
| johnsom | I'm also wondering if there are any game over messages from CW or HM, but I don't know if those happen during normal operations or not. They come from taskflow. Trying to look for a string | 23:52 |
| johnsom | It is probably GAME_OVER in the logs | 23:53 |
| johnsom | Just seeing if taskflow had to abort out under us | 23:53 |
| johnsom | Might be in combination with "failed" | 23:54 |
| johnsom | Yeah, "FAILURE" might be a better search | 23:55 |
| *** blake has joined #openstack-lbaas | 23:58 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!