Wednesday, 2018-11-28

mnaserjohnsom: sorry you're right, this isnt backported to stable/queens00:16
mnaser this one00:17
johnsomOk, I didn't *think* it was, but thought maybe I was crazy.00:17
*** abaindur has quit IRC00:18
johnsomYou want to do the work or should I?00:18
mnaserjohnsom: it seems to merge conflict when i complain, and im writing a long post mortem to an angry customer so :<00:18
johnsomYeah, ok, I will get it.00:18
*** abaindur has joined #openstack-lbaas00:19
*** yamamoto has quit IRC00:34
johnsommnaser Argh, there was a DB schema change before that patch....01:01
*** ramishra has quit IRC01:13
*** annp has joined #openstack-lbaas01:14
*** abaindur has quit IRC02:26
*** ramishra has joined #openstack-lbaas02:35
*** hongbin has joined #openstack-lbaas02:36
openstackgerritwangxiyuan proposed openstack/octavia master: Correct the base class for v2 l7rules type tests
openstackgerritwangxiyuan proposed openstack/octavia master: Tags support for lb resources
*** ramishra has quit IRC03:15
*** ramishra has joined #openstack-lbaas03:35
*** hongbin_ has joined #openstack-lbaas04:23
*** hongbin has quit IRC04:24
*** hongbin has joined #openstack-lbaas04:37
*** hongbin_ has quit IRC04:38
*** hongbin has quit IRC04:40
*** yamamoto has joined #openstack-lbaas04:58
*** pcaruana has joined #openstack-lbaas05:09
*** yboaron_ has joined #openstack-lbaas05:10
*** abaindur has joined #openstack-lbaas05:12
openstackgerritwangxiyuan proposed openstack/octavia master: Tags support for lb resources
*** jarodwl has joined #openstack-lbaas06:35
*** ramishra has quit IRC06:48
*** rcernin has quit IRC06:57
*** ramishra has joined #openstack-lbaas07:05
*** velizarx has joined #openstack-lbaas07:15
*** ccamposr has joined #openstack-lbaas07:15
*** ramishra has quit IRC07:18
*** ramishra has joined #openstack-lbaas07:21
*** velizarx has quit IRC07:30
*** velizarx has joined #openstack-lbaas07:41
*** rpittau has joined #openstack-lbaas08:01
*** abaindur has quit IRC08:09
*** rpittau_ has joined #openstack-lbaas08:31
*** abaindur has joined #openstack-lbaas08:31
*** rpittau has quit IRC08:33
*** rpittau_ is now known as rpittau08:35
*** abaindur has quit IRC08:35
*** rpittau_ has joined #openstack-lbaas08:40
*** rpittau has quit IRC08:40
*** rpittau_ is now known as rpittau08:49
*** yamamoto has quit IRC08:56
*** yboaron_ has quit IRC09:04
*** yboaron_ has joined #openstack-lbaas09:04
*** celebdor has joined #openstack-lbaas09:07
*** yboaron_ has quit IRC09:13
*** velizarx has quit IRC09:17
*** velizarx has joined #openstack-lbaas09:18
*** yboaron_ has joined #openstack-lbaas09:25
*** rpittau has quit IRC09:34
*** yamamoto has joined #openstack-lbaas09:43
*** yamamoto has quit IRC10:02
*** yamamoto has joined #openstack-lbaas10:07
*** tobias-urdin has quit IRC10:10
*** tobias-urdin has joined #openstack-lbaas10:15
*** jarodwl has quit IRC10:33
*** jarodwl has joined #openstack-lbaas10:34
*** salmankhan has joined #openstack-lbaas10:35
*** salmankhan has quit IRC10:45
*** salmankhan has joined #openstack-lbaas10:53
*** yamamoto has quit IRC11:57
*** hvhaugwitz has quit IRC12:03
*** hvhaugwitz has joined #openstack-lbaas12:03
*** yamamoto has joined #openstack-lbaas12:11
*** witek has quit IRC12:23
*** witek has joined #openstack-lbaas12:23
*** celebdor has quit IRC12:31
*** celebdor has joined #openstack-lbaas12:35
*** takamatsu has quit IRC12:37
*** takamatsu has joined #openstack-lbaas12:43
*** celebdor has quit IRC13:03
*** celebdor has joined #openstack-lbaas13:04
*** phuoc has quit IRC13:15
*** phuoc has joined #openstack-lbaas13:16
*** takamatsu has quit IRC13:29
*** takamatsu has joined #openstack-lbaas13:30
*** salmankhan1 has joined #openstack-lbaas13:42
*** salmankhan has quit IRC13:43
*** salmankhan1 is now known as salmankhan13:43
*** takamatsu has quit IRC13:44
*** yamamoto has quit IRC13:45
*** salmankhan has quit IRC13:47
*** takamatsu has joined #openstack-lbaas13:48
*** jarodwl has quit IRC13:49
*** jarodwl has joined #openstack-lbaas13:50
*** yboaron_ has quit IRC13:54
*** salmankhan has joined #openstack-lbaas14:06
*** yamamoto has joined #openstack-lbaas14:07
*** ptoohill1 has quit IRC14:56
*** ivve has joined #openstack-lbaas15:21
*** ramishra has quit IRC15:46
*** rpittau has joined #openstack-lbaas16:28
*** celebdor has quit IRC16:52
*** rpittau has quit IRC16:52
*** pcaruana has quit IRC17:03
*** KeithMnemonic has quit IRC17:14
colin-was reviewing this document and had a question for the channel related to the direction that flavors will take
johnsomOk. I am working on that code now, so good timing17:16
colin-as i'm looking into nova-lxd and potentially using it with octavia it occurs to me that this direction doc does not address a scenario where we might have two ways to provision an amp (traditional kvm method, as well as nova-lxd) and want to be able to differentiate at lb creation17:17
johnsomYeah, we have that a bit today with the different compute drivers. You could implement a nova-lxd via a new, or aliased compute driver. Though this may also simply be a configuration we add.17:19
johnsomBasically think of flavors as taking elements of the configuration file (with the config file being the default if not defined in a flavor) and making flavors that apply settings on a per-LB basis.17:20
colin-downstream of the amp agent basically17:20
johnsomUmm, no, upstream. Flavors can be applied/supported by all drivers.17:20
johnsomNot all drivers will implement all options that another driver might17:21
colin-probably chose "downstream" poorly, what i meant was the changes that falvors describe are largely focused on the haproxy configuration rather than the characteristics of the nova resource17:22
johnsomNo, I don't think so. Things like filesystem size, number of vCPUs, nova flavor, will all be parameters that can be defined for the amphora driver.17:23
*** ccamposr has quit IRC17:27
*** jackivanov has quit IRC17:28
colin-ok great, that's a big relief as we're excited to be able to use different compute flavor types as well as any new drivers17:39
*** kosa777777 has joined #openstack-lbaas17:52
*** rtjure has joined #openstack-lbaas18:13
*** salmankhan has quit IRC18:26
*** fnaval has joined #openstack-lbaas18:44
*** pck has quit IRC19:14
*** abaindur has joined #openstack-lbaas19:27
*** celebdor has joined #openstack-lbaas19:45
johnsom#startmeeting Octavia20:00
openstackMeeting started Wed Nov 28 20:00:20 2018 UTC and is due to finish in 60 minutes.  The chair is johnsom. Information about MeetBot at
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.20:00
*** openstack changes topic to " (Meeting topic: Octavia)"20:00
openstackThe meeting name has been set to 'octavia'20:00
johnsomHi folks20:00
johnsomSeems quiet this week.20:01
johnsom#topic Announcements20:01
*** openstack changes topic to "Announcements (Meeting topic: Octavia)"20:01
johnsomJust a friendly reminder, we have a priority review list:20:01
johnsomLots of goodies looking for a review....20:01
johnsomFYI, I am trying to update it at least once a week20:02
johnsomOtherwise I don't think I have any announcements today. Anyone else?20:02
cgoncalveswe released octavia-tempest-plugin 0.2.020:02
johnsomAh yes, good stuff there.20:03
*** celebdor has quit IRC20:03
nmagnezijohnsom, on that note, I've built 0.2.0 RPMs today so there will get ship in the near future20:03
johnsomThere are a few patches up for review on the stable/queens branch too. I hope we can get those in and cut a queens release soon20:03
johnsomnmagnezi Nice!20:04
johnsom#topic Brief progress reports / bugs needing review20:04
*** openstack changes topic to "Brief progress reports / bugs needing review (Meeting topic: Octavia)"20:04
johnsomOther than taking Thanksgiving week off, I have been working on reviews and catching up on email, etc.20:04
johnsomI have started looking at the next step for the flavors work as well.20:05
*** celebdor has joined #openstack-lbaas20:05
johnsomAny other updates?20:05
* cgoncalves has been fixing the usual suspect, again20:05
* johnsom grumbles "triple-opps"20:06
cgoncalvesI reviewed the tags patch. good stuff there. still need to vote though20:06
johnsomOk cool, I pitched in my +2 this morning on that. It does look great and he fixed the code duplication issue I posted.20:07
cgoncalvesyeah, the mixin. I wanted to leave that comment too20:07
cgoncalvestags was something a vendor also was interested in. hopefully we will be able to pass it down to the provider20:08
cgoncalvesyou mean in this patch or never?20:08
johnsomThat allows bad behaviors like using tags to configure drivers and such. They get overloaded and it becomes a mess.20:08
johnsomThere really isn't a good reason a driver would need access to the tags.20:09
johnsomThey should pass in driver config settings via flavors20:09
johnsomnever IMO20:10
johnsomWhat vendor thought they wanted tags?20:10
cgoncalvesok, I don't know the use case they have in mind. via flavors sounds reasonable and likely doable for their needs20:10
johnsomYeah, the right away to do driver configs is flavors, so it is at least validated, etc.20:12
johnsomIt's part of the driver spec20:12
xgermanIndeed. No random stuff passed in20:12
johnsomAnd coming to a patch review near you soon!  grin20:12
johnsomtenant: tag: "vCPUs": 25520:13
johnsom"I need it fast!"20:13
johnsomOk, other updates or on to some topics for this week?20:14
johnsom#topic Brief progress reports / bugs needing review20:14
*** openstack changes topic to "Brief progress reports / bugs needing review (Meeting topic: Octavia)"20:14
johnsomDang, wrong topic20:14
johnsom#topic Amp failover when failover already occurred (Queens)20:14
*** openstack changes topic to "Amp failover when failover already occurred (Queens) (Meeting topic: Octavia)"20:14
xgermanYou are repeating yourself20:15
* johnsom thinks he should not be eating lunch at the same time20:15
johnsomBack in Rocky we had this handy patch that fixes an issue with LBs that failover twice.20:15
nmagnezijohnsom, bon app├ętit20:15
xgermanMulti tasking20:16
johnsomBasically if the first attempt failed, say nova issue, then it tries again. The problem is in between housekeeping can get a bit aggressive and purge some useful records.20:16
johnsomI looked at backporting this to Queens (recently a vendor hit is as they were missing another patch), but as it is, this patch depends on some schema changes that were not part of Queens20:17
johnsom<sad face>20:17
johnsomHowever, I think I can improve the situation for Queens at least some with a partial backport of this. It wouldn't be a true cherry-pick because of the other schema dependencies.20:18
johnsomBefore I dig into that I wanted to run it by you all for comment20:19
cgoncalvesfirst, is there a way to easily assert when a patch is not backportable because of schema changes, in this case? what should we do better?20:20
johnsomOk. So, I don't think it would be right/best to use the cherry pick string, etc. when the bulk of the fix won't be going back. Are we ok with a "new" patch against stable/queens that attempts to fix this? That patch would not go forward to Rocky/Stein as there is a more complete fix already there.20:21
xgermanI am ok with that20:21
johnsomYeah, I manually cherry-picked and fixed the conflicts, the unit and functional tests blew up due to the missing schema20:22
cgoncalvesok, good20:22
johnsomSo, yes, it was obvious there was a required schema change patch that was not on queens20:22
cgoncalvesfine with a new patch20:22
nmagneziSame here20:23
johnsomOk cool. I will propose something later today that should improve that scenario.20:23
johnsomI will reference the Rocky patch and explain the situation in the commit message.20:23
johnsomI think it's the best answer as folks are using queens as an LTS and this should be fixed.20:24
johnsom#topic VIP ACLs/SGs - continued20:24
*** openstack changes topic to "VIP ACLs/SGs - continued (Meeting topic: Octavia)"20:24
johnsomWe ran out of time on this last week, so I added it to this weeks agenda.20:25
johnsomIs there more discussion needed on this topic?20:25
johnsomGoing once....20:26
johnsomGoing twice...  (we can always discuss more at a future meeting)20:26
johnsom#topic Open Discussion20:26
*** openstack changes topic to "Open Discussion (Meeting topic: Octavia)"20:26
johnsomOther topics for today?20:27
johnsomOk, then I will get busy on the queens patch and back to doing interesting jsonschema things for flavors.20:29
johnsomHave a good week!20:29
*** openstack changes topic to "Discussions for Octavia | Stein priority review list:"20:29
openstackMeeting ended Wed Nov 28 20:29:56 2018 UTC.  Information about MeetBot at . (v 0.1.4)20:29
openstackMinutes (text):
cgoncalvesI swear I read it as "johnsom's schema" things20:30
johnsomWell.... It might turn out that way20:30
johnsomI'm kind of excited about it as I think it will be a hugely powerful feature for operators20:31
cgoncalvesjohnsom's schema? for sure!20:33
*** openstackgerrit has quit IRC20:36
xgermancgoncalves: did you upload our presentations?20:55
cgoncalvesxgerman, I did, in PDF format20:55
*** celebdor has quit IRC20:59
*** rcernin has joined #openstack-lbaas21:50
*** ccamposr has joined #openstack-lbaas21:58
*** rcernin has quit IRC22:03
*** rcernin has joined #openstack-lbaas22:12
*** ccamposr has quit IRC22:24
colin-sorry i missed the meeting earlier, wanted to ask if anyone has heard anything about the active/active work? saw ptg notes and reviews/docs it links to but hadn't heard anyone talking about it lately22:54
johnsomcolin- I have started that work, but had to re-prioritized it down as we lost some people that were working on other key things like provider drivers and flavors. I will probably pick it up again in Train. Plus, we need Ryu for it and that project is transitioning into OpenStack neutron as project Ken. So, I also wanted the dust to settle there.23:01
colin-ah interesting, ok23:06
colin-goood to know23:06
johnsommnaser Taking another look at your queens issue today.  Can you check your CW and HM logs for "Failed to update amphora %(amp)s status to ERROR due to: " or "Reverting mark amphora deleted in DB for amp id %(amp)s and compute id %(comp)s" messages for your missing amp records?23:06
johnsomThose are error and warning level logs so you should have them.23:08
mnaserI will dig those once I get back in the office johnsom23:13
johnsomThank you23:13
lxkongjohnsom: hi i am wondering if it's true that we could only delete 'ACTIVE' lbs? for other status, deletion will fail, right?23:22
johnsomNo, you can delete ERROR LBs/Objects as well23:22
lxkongjohnsom: ah, right, so for other status will return some response(maybe 409) immediately, right?23:23
lxkongcool, thanks xgerman johnsom23:24
johnsomCorrect, all of the PENDING_* statuses will return 409 immediately until they either succeed or the retries expire and they go to ERROR23:24
xgermanin other notes keystone claims in their docs  not to return an email with the user GET but it does anyway23:25
mnasergrep 'Failed to update amphora' /var/log/octavia/* yield nothing23:44
mnaseri do see results for : grep 'Reverting mark amphora deleted in DB' /var/log/octavia/*23:45
johnsommnaser Enough "Reverting" messages for the missing amps?23:46
mnaserjohnsom: only for 323:47
mnaserand i remember there was 3 ERROR state LBs at the time23:47
johnsomOk, so that is the 3 others we knew about.23:47
johnsomAny chance a CW or HM got kill -9'd?23:47
mnaserjohnsom: maybe we attempted to restart the services to figure out whats going on23:51
*** fnaval has quit IRC23:51
johnsomI'm also wondering if there are any game over messages from CW or HM, but I don't know if those happen during normal operations or not. They come from taskflow.  Trying to look for a string23:52
johnsomIt is probably GAME_OVER in the logs23:53
johnsomJust seeing if taskflow had to abort out under us23:53
johnsomMight be in combination with "failed"23:54
johnsomYeah, "FAILURE" might be a better search23:55
*** blake has joined #openstack-lbaas23:58

Generated by 2.15.3 by Marius Gedminas - find it at!