Tuesday, 2019-12-03

rm_workjohnsom: around>00:06
*** ivve has quit IRC00:16
rm_workjohnsom: https://review.opendev.org/#/c/696947/00:35
rm_workataraday_: ok, all +2'd from me, up to https://review.opendev.org/#/c/668898/ which I do have a concern on, with the passing of "dict_config" bool00:39
rm_workjohnsom: ^^ your court now00:39
ataraday_rm_work, https://review.opendev.org/#/c/668898/ working on resolving comments there - remove "dict_config" and add more constants00:41
rm_worksorry about ALL the constants stuff lol00:41
rm_worki actually was ignoring "id" i thought00:42
rm_workbecause ... yeah00:42
rm_workso many00:42
johnsomrm_work o/00:45
johnsomStill not 100% sure we couldn't use moves. We have used that for exceptions before....00:46
*** sorrison has joined #openstack-lbaas01:02
openstackgerritMerged openstack/python-octaviaclient master: Fix API breakage due to new exceptions module  https://review.opendev.org/69694701:08
rm_workjohnsom: what he said makes sense01:19
rm_workjohnsom: if you can figure it out, i'm down to replace that01:20
johnsomYeah, decided I had better things to do today01:20
rm_worklike merging the first 5 or so of ataraday_'s patches? :D01:20
johnsomWell, reviewing a bunch of stuff. Yeah, I plan to look at those next01:22
openstackgerritAnn Taraday proposed openstack/octavia master: Transition amphora flows to dicts  https://review.opendev.org/66889801:24
rm_workataraday_: thanks so much for doing all those constants! I really didn't expect you to do the other constants that weren't even part of your change T_T01:30
rm_workbut really awesome that you did :D01:30
ataraday_rm_work, when you start adding new constants it is hard to stop :D01:31
ataraday_althouh I think they all are used in changes I made, so this is all related01:33
*** sorrison has quit IRC01:38
*** sorrison has joined #openstack-lbaas01:38
*** sorrison has quit IRC01:40
rm_workcool, well, first 5 of your changes are +2 from me01:50
rm_worklet's get this chain shrunk a bit01:50
rm_worknone of this is going to break anything until we're entirely done and people switch to the v2 driver, so whatever, we can do final testing+cleanup after all of the chain merges01:51
rm_workor on the very last patch01:51
johnsomIt still touches shared code though, so we should be looking at the patches carefully01:52
*** sorrison has joined #openstack-lbaas01:55
*** sorrison has quit IRC01:58
rm_workYeah, I mean I am watching what's happening, but most of the meaty changes are in v2/ files02:01
rm_workand all the scenarios are still passing, so02:01
*** sorrison has joined #openstack-lbaas02:02
johnsomI am going to have to finish looking at that monster listener patch tomorrow.02:03
rm_workit's not THAT big02:04
johnsomIt was my mess in the first place, so clearly it needs *extra* attention. lol02:04
rm_workunless you're talking about a different one than i'm thinking02:04
rm_work+849? meh02:04
rm_work+2665, -107202:05
rm_workIt's not exactly a competition though, admittedly02:06
rm_work... and if it was, it'd be more like ... golf rules02:06
*** sorrison has quit IRC02:15
*** sorrison has joined #openstack-lbaas02:16
*** gthiemonge has quit IRC02:35
*** gthiemonge has joined #openstack-lbaas02:35
sorrisonrm_work: Had a look into that failure in my tempest job to do with missing yaml02:42
sorrisonIt also fails in your job too with the same error, it's just non-voting.02:43
sorrisonoctavia-v2-act-stdby-dsvm-scenario-two-node fails but the py2 one passes02:44
*** zaneb has left #openstack-lbaas02:59
openstackgerritAnn Taraday proposed openstack/octavia master: Convert Lb flows to use provider dicts  https://review.opendev.org/67172503:09
openstackgerritMerged openstack/octavia master: Use retry for AmphoraComputeConnectivityWait  https://review.opendev.org/66279103:25
openstackgerritAnn Taraday proposed openstack/octavia master: Convert health_monitor flows to use provider models  https://review.opendev.org/67977903:28
*** sorrison has quit IRC04:04
*** sorrison has joined #openstack-lbaas04:10
*** sorrison has quit IRC04:15
openstackgerritAnn Taraday proposed openstack/octavia master: Transition l7policy flows to dicts  https://review.opendev.org/66597704:19
openstackgerritAnn Taraday proposed openstack/octavia master: Transition l7rule flows to dicts  https://review.opendev.org/66817304:38
*** sorrison has joined #openstack-lbaas04:45
*** sorrison has quit IRC04:52
*** sorrison has joined #openstack-lbaas04:57
*** tkajinam has quit IRC05:02
openstackgerritSam Morrison proposed openstack/octavia-tempest-plugin master: Add new scenario test to create LB in specific AZ  https://review.opendev.org/69534905:18
openstackgerritAnn Taraday proposed openstack/octavia master: Jobboard based controller  https://review.opendev.org/64740605:19
*** tkajinam has joined #openstack-lbaas05:33
*** sapd1_x has joined #openstack-lbaas06:23
*** gcheresh has joined #openstack-lbaas06:26
*** ataraday_ has quit IRC06:39
*** sorrison has quit IRC07:14
*** tkajinam_ has joined #openstack-lbaas07:18
*** tkajinam_ has quit IRC07:19
*** tkajinam_ has joined #openstack-lbaas07:20
*** tkajinam has quit IRC07:21
*** sapd1_x has quit IRC07:23
*** sorrison has joined #openstack-lbaas07:25
*** CobHead has quit IRC07:50
*** trident has quit IRC07:51
*** trident has joined #openstack-lbaas07:51
*** sorrison has quit IRC07:53
*** sorrison has joined #openstack-lbaas07:54
*** maciejjozefczyk has joined #openstack-lbaas07:56
openstackgerritFederico Ressi proposed openstack/python-octaviaclient master: Add DevStack CI functional job  https://review.opendev.org/69696708:16
*** tesseract has joined #openstack-lbaas08:16
*** sorrison has quit IRC08:26
*** sorrison has joined #openstack-lbaas08:27
*** sorrison has quit IRC08:42
*** rpittau|afk is now known as rpittau08:43
*** sorrison has joined #openstack-lbaas08:43
*** sorrison has quit IRC08:48
*** ivve has joined #openstack-lbaas08:51
openstackgerritFederico Ressi proposed openstack/python-octaviaclient master: Add job for executing functional tests on CentOS 7  https://review.opendev.org/69703209:04
openstackgerritGregory Thiemonge proposed openstack/octavia-tempest-plugin master: Use VM as client in traffic operation scenario tests  https://review.opendev.org/68756209:05
*** gcheresh has quit IRC09:08
*** gcheresh has joined #openstack-lbaas09:08
openstackgerritFederico Ressi proposed openstack/python-octaviaclient master: Add DevStack CI functional job  https://review.opendev.org/69696709:14
openstackgerritFederico Ressi proposed openstack/python-octaviaclient master: Add DevStack CI functional job  https://review.opendev.org/69696709:15
openstackgerritFederico Ressi proposed openstack/python-octaviaclient master: Add DevStack CI functional jobs  https://review.opendev.org/69696709:15
openstackgerritFederico Ressi proposed openstack/python-octaviaclient master: Add DevStack CI functional jobs  https://review.opendev.org/69696709:18
*** tkajinam_ has quit IRC09:20
*** sorrison has joined #openstack-lbaas09:23
*** sorrison has quit IRC09:28
openstackgerritFederico Ressi proposed openstack/python-octaviaclient master: Add DevStack CI functional jobs  https://review.opendev.org/69696709:40
*** dasp has quit IRC09:42
*** dasp has joined #openstack-lbaas09:43
*** abaindur has quit IRC09:47
*** sorrison has joined #openstack-lbaas09:58
*** sorrison has quit IRC10:04
*** rcernin has quit IRC10:06
*** pcaruana has joined #openstack-lbaas10:08
*** dmellado has quit IRC10:33
*** dmellado has joined #openstack-lbaas10:35
*** sorrison has joined #openstack-lbaas10:40
*** sorrison has quit IRC10:48
*** sorrison has joined #openstack-lbaas10:49
*** sorrison has quit IRC10:53
*** sorrison has joined #openstack-lbaas10:54
*** sorrison has quit IRC10:59
*** sorrison has joined #openstack-lbaas11:00
*** sorrison has quit IRC11:05
*** sorrison has joined #openstack-lbaas11:09
*** sorrison has quit IRC11:13
openstackgerritMaciej Józefczyk proposed openstack/octavia-tempest-plugin master: Discover used LB algorithm used for test  https://review.opendev.org/67226411:16
*** rpittau is now known as rpittau|bbl11:18
openstackgerritMaciej Józefczyk proposed openstack/octavia-tempest-plugin master: Discover LB algorithm used for test  https://review.opendev.org/67226411:20
*** sorrison has joined #openstack-lbaas11:26
openstackgerritFederico Ressi proposed openstack/python-octaviaclient master: Add DevStack CI functional jobs  https://review.opendev.org/69696711:27
*** sorrison has quit IRC11:34
*** vishalmanchanda has joined #openstack-lbaas11:34
*** sorrison has joined #openstack-lbaas11:38
openstackgerritFederico Ressi proposed openstack/python-octaviaclient master: Add DevStack CI functional jobs  https://review.opendev.org/69696711:39
maciejjozefczykHey! cgoncalves ltomasbo johnsom :) I'm starting moving ovn provider driver code to separate repository. What do you think about naming it as ' octavia-ovn-provider-driver'? Or ' octavia-ovn-provider'?11:40
*** trident has quit IRC11:40
*** sorrison has quit IRC11:42
*** trident has joined #openstack-lbaas11:43
openstackgerritFederico Ressi proposed openstack/python-octaviaclient master: Add DevStack CI functional jobs  https://review.opendev.org/69696711:43
*** sorrison has joined #openstack-lbaas11:44
*** sorrison has quit IRC11:49
*** sorrison has joined #openstack-lbaas11:52
*** gcheresh has quit IRC11:52
cgoncalvesmaciejjozefczyk, I think either are good options.11:54
*** sorrison has quit IRC11:56
maciejjozefczykcgoncalves thanks!11:59
*** sorrison has joined #openstack-lbaas12:00
*** sorrison has quit IRC12:04
*** gcheresh has joined #openstack-lbaas12:04
*** sorrison has joined #openstack-lbaas12:13
*** sorrison has quit IRC12:18
*** sorrison has joined #openstack-lbaas12:24
*** sorrison has quit IRC12:28
*** servagem has joined #openstack-lbaas12:35
*** sorrison has joined #openstack-lbaas12:37
*** sorrison has quit IRC12:42
*** sorrison has joined #openstack-lbaas12:54
*** sorrison has quit IRC12:58
maciejjozefczykcgoncalves, https://review.opendev.org/#/c/697076/1 :)13:08
*** dasp has quit IRC13:12
*** dasp has joined #openstack-lbaas13:12
*** sorrison has joined #openstack-lbaas13:13
dulekHi! We see Kuryr gates are red, timeouting waiting for the LB to get created.13:17
dulekI'd blame one of 3 last commits: https://opendev.org/openstack/octavia/commits/branch/master13:17
dulekWe do run with Python 3 for a while now, so shouldn't be that.13:18
dulekA link to a failing run: https://zuul.opendev.org/t/openstack/build/524e34982ce14415864bcf716926ff0b13:20
dulekI don't see anything interesting in logs though.13:20
*** sorrison has quit IRC13:23
openstackgerritMichał Dulko proposed openstack/octavia master: DNM: Revert "Use 2048-bits keys for devstack certificates"  https://review.opendev.org/69708413:25
openstackgerritMichał Dulko proposed openstack/octavia master: DNM: Revert "Use retry for AmphoraComputeConnectivityWait"  https://review.opendev.org/69708613:27
dulekI'll try to figure out which one caused it using the above. ^13:27
*** sorrison has joined #openstack-lbaas13:35
*** rpittau|bbl is now known as rpittau13:46
*** sorrison has quit IRC13:46
*** sorrison has joined #openstack-lbaas13:53
*** sorrison has quit IRC13:58
*** sorrison has joined #openstack-lbaas14:08
*** sorrison has quit IRC14:12
*** strobert1 has quit IRC14:14
*** strobert1 has joined #openstack-lbaas14:17
*** sorrison has joined #openstack-lbaas14:23
*** trident has quit IRC14:24
*** trident has joined #openstack-lbaas14:25
*** sorrison has quit IRC14:30
*** pcaruana has quit IRC14:33
*** TrevorV has joined #openstack-lbaas14:35
*** sorrison has joined #openstack-lbaas14:44
*** sorrison has quit IRC14:48
dulekOkay, so it seems none of the above caused it. Any other clues?14:58
*** pcaruana has joined #openstack-lbaas15:04
*** sorrison has joined #openstack-lbaas15:04
cgoncalveswhy do you think it is octavia and not nova or neutron, for example?15:06
cgoncalvesah, you pasted a link to the job. looking15:06
*** sorrison has quit IRC15:09
dulekcgoncalves: You can stop, I'll try my theory first.15:11
dulekTo not waste your time.15:11
dulekAs it might be related to us actually.15:11
dulekYou've dropped Python 2, but I see we're using CentOS 7 amphora.15:12
dulekAnd as the build happened this morning: https://tarballs.openstack.org/octavia/test-images/15:12
cgoncalvesdulek, ok. I don't see anything relevant in octavia logs or nova15:12
dulekIt's possible CentOS 7 version from link above isn't working at all anymore.15:12
dulekSo I'm going to try with Ubuntu amp.15:13
cgoncalvesnit: kuryr-kubernetes-tempest-containerized is non-voting. remove it from your gate queue.15:13
*** sorrison has joined #openstack-lbaas15:14
dulekcgoncalves: Thanks for spotting this!15:16
*** sorrison has quit IRC15:20
dulekcgoncalves: octavia/cmd/agent.py is what runs on the amp as the amp server?15:23
*** sorrison has joined #openstack-lbaas15:23
cgoncalvesdulek, yes15:25
dulekAlright, seems like my hypothesis is correct.15:25
cgoncalvesFWIW, http://zuul.openstack.org/builds?job_name=octavia-v2-dsvm-py2-scenario-centos-715:26
cgoncalvesdulek, which is?15:26
dulekcgoncalves: The agent is no longer able to start on CentOS 7 Amp due to lack of Python 3.15:28
*** sorrison has quit IRC15:28
dulekcgoncalves: It's run against Python 2, so for example gunicorn is not installed there: https://opendev.org/openstack/octavia/src/branch/master/requirements.txt#L5815:28
cgoncalvesoh, shoot. yeah, we killed py2 yesterday15:33
dulekhttps://review.opendev.org/#/c/697114/ - let's see how this goes.15:34
dulekcgoncalves: Do you know where I can find jobs producing those: https://tarballs.openstack.org/octavia/test-images/ ?15:34
dulekIs it possible Ubuntu image there still uses Python 2 and won't work too?15:35
cgoncalvesalthough you're using image from RDO which is expected to install the amphora-agent RPM15:35
dulekThose are from RDO?15:35
cgoncalvesUbuntu is expected to use py315:35
cgoncalvesbleh, I need coffee. no, they are not. I had the impression you had linked us before to the RDO images15:35
cgoncalvesbut it was to https://tarballs.openstack.org/octavia/test-images/15:36
*** tesseract has quit IRC15:37
cgoncalveshaleyb, hey. were you planning to update the centos 7 job to use py3?15:39
cgoncalvesunless we switch it to py3, we should delete the periodic job publish-openstack-octavia-amphora-image-centos715:40
*** sorrison has joined #openstack-lbaas15:40
haleybcgoncalves: yes, was working on that today, should add a story15:46
*** tesseract has joined #openstack-lbaas15:48
*** TrevorV has quit IRC15:50
*** sorrison has quit IRC15:50
rm_workah so we'll have a py3 cent7? :D15:56
cgoncalvesrm_work, u no laike? :)15:57
*** sorrison has joined #openstack-lbaas15:57
cgoncalvescentos7-py3 controller + centos8-py3 amp. hmmm :)15:57
*** sorrison has quit IRC16:02
rm_workit'll be cool :D16:02
*** sorrison has joined #openstack-lbaas16:09
*** TrevorV has joined #openstack-lbaas16:12
openstackgerritCarlos Goncalves proposed openstack/octavia master: WIP: Add support to CentOS 8 amphora images  https://review.opendev.org/69712816:13
*** sorrison has quit IRC16:14
*** ivve has quit IRC16:15
openstackgerritCarlos Goncalves proposed openstack/octavia master: WIP: Add support to CentOS 8 amphora images  https://review.opendev.org/69712816:18
*** jamesdenton has quit IRC16:25
cgoncalvesactive-standby centos 8 based load balancer up and running ;)16:25
*** sorrison has joined #openstack-lbaas16:29
*** sorrison has quit IRC16:37
*** ataraday_ has joined #openstack-lbaas16:42
rm_workcgoncalves: so you gonna review the jobboard stuff today?16:43
rm_worki've been through like the first 6 patches and commented and ann fixed stuff yesterday, so i'm +2 now on a bunch16:43
rm_workwe could really get this stuff moving XD16:43
*** sorrison has joined #openstack-lbaas16:44
cgoncalvesrm_work, I started reviewing the AZ admin + client patches today16:47
*** sorrison has quit IRC16:48
johnsomI plan to continue no the jobboard reviews today16:52
rm_workcgoncalves: ok16:52
rm_workcgoncalves: i took your words to heart and started trying to get the jobboard stuff moving finally16:52
rm_workso, up to you16:53
rm_workbut a big part of that chain is +2, and i'm going through more today16:53
rm_workgranted, it's all just logical code review, until 100% of it is in, none of it is really going to work, I think16:53
rm_workso I haven't bothered testing it in reality16:53
rm_workso long as all our v1 stuff continues to work and pass tests, i don't see a problem merging these pieces, up to the very last patch where we'll want stuff to actually be provably working16:54
*** sorrison has joined #openstack-lbaas16:54
cgoncalvesrm_work, let me know what works best, merging AZ or jobboard first16:57
rm_workwell, i just had to go back and -1 a bunch of jobboard stuff actually, though not for anything major... just an extra pop() that i'm not understanding17:00
rm_workbut yes, I do really need AZ soon (yesterday)17:00
rm_workso if you can get that in, great17:00
johnsomI think there is a pressing timeline of need for AZ, so maybe splitting up the reviews across cores is a good approach. I have already done the AZ reviews and started the jobboard work.17:00
ataraday_rm_work, https://review.opendev.org/#/c/657842/31/octavia/api/drivers/amphora_driver/v2/driver.py - we need to pop ID as we have in original_member (old object dict) and it will be conflicting with the on in update dict17:01
johnsomrm_work I think we can cut the octavia-lib AZ needs right?17:01
ataraday_I put comment on healthmonitor change about that17:01
*** sorrison has quit IRC17:02
haleybjohnsom: i'll have to find a place for the ovn driver, guess it can just live on it's own under the neutron umbrella17:02
*** tesseract has quit IRC17:03
johnsomhaleyb Yeah, I wanted to make it clear that the Octavia team hasn't agreed to adopt that third party driver. You can still give Octavia core access in gerrit though, if you want.17:03
haleybjohnsom: yes, it's an odd bird, and I know the company that will do most of the early work ;-)  i'll figure something out regarding the name/location17:06
*** sorrison has joined #openstack-lbaas17:10
*** rpittau is now known as rpittau|afk17:12
*** sorrison has quit IRC17:15
rm_workataraday_: ah ok, so actually that really is intended17:16
rm_workjohnsom: what? you mean cut a release?17:17
rm_workataraday_: but is that true for the rest? because all of them do it -- pool / hm / l7policy / etc17:17
johnsomrm_work Don't you need octavia-lib released with the AZ changes?17:17
rm_worki don't think that ID should conflict?17:17
rm_workjohnsom: so, we're waiting on one more lib merge17:17
rm_workjohnsom: needs the first API change to merge before it can (it's a dep)17:18
rm_workjohnsom: THEN we could cut it, yes17:18
rm_workthat's part of why i'm trying to get the first couple patches moving17:18
rm_workbecause i know there's a few stages involved17:18
cgoncalvesrm_work, AZ admin API is in the gate now17:18
rm_workoh ok cool :D just noticed17:18
johnsomOk, I was just looking at the octavia-lib list which seemed ready17:19
rm_worktempest tests also?17:19
rm_workjohnsom: very shortly17:19
rm_workjohnsom: once https://review.opendev.org/#/c/694707/ lands17:19
rm_workcgoncalves: https://review.opendev.org/#/c/695291/17:20
rm_workthat goes with the change you +A'd17:20
cgoncalvesyeah, skimming it now17:20
ataraday_rm_work, I got errors with update command about conflicting passed ID, I can recheck if I don't miss somethings as it was couple of months ago :)17:20
rm_workataraday_: hmmmm ok, yeah so that's kinda what i was getting at ... maybe i am missing something, and it really does have to be gone17:21
*** sorrison has joined #openstack-lbaas17:21
rm_workataraday_: is it easy to test?17:21
rm_workataraday_: i noticed there was at least one patch (listeners?) that DIDN'T seem to do it17:21
ataraday_yeah, I will recheck, I've got devstack reinstalling so in an hour or something...17:23
rm_worki mean... also maybe it just doesn't matter <_<17:24
rm_workif it's not used17:24
rm_workI thought it might be used17:24
rm_workbut i'm not sure17:24
rm_workjohnsom: so yeah, once that patch merges, then the lib patch will go to the gate, and then once THAT merges, we can cut a lib release17:28
*** sorrison has quit IRC17:28
*** sorrison has joined #openstack-lbaas17:30
*** sorrison has quit IRC17:36
*** sorrison has joined #openstack-lbaas17:37
*** gcheresh has quit IRC17:41
*** sorrison has quit IRC17:41
*** sorrison has joined #openstack-lbaas17:52
*** sorrison has quit IRC17:56
ataraday_rm_work, I checked update loadbalancer without poping ID and got error http://paste.openstack.org/show/787057/ not conflicting, but it expectes ID not loadbalancer_id, I think it will be the same for rest of resources - will check listener now.17:58
rm_workahhhh hold on17:59
rm_workYES DUH sorry17:59
rm_workI get it17:59
rm_workthough ... i still have one comment17:59
rm_workcan you guess what it is? :D18:00
ataraday_how it is working now?18:01
ataraday_okay >_<18:02
rm_workhahaha no worries, we can do a pass at the end18:03
rm_workseriously it's fine18:03
rm_worki am gonna do like a global find, on all the common ones18:03
rm_workand fix it all18:03
rm_workataraday_: switching to +2s18:05
rm_workdon't touch18:05
rm_work*you'll need to rebase in a sec i think because of the merging AZ patch?18:07
rm_workthough I can also do that for you (since I said I would) :D18:07
ataraday_I can rebase no worries, if johnsom will have some comments I may fix constants as well :)18:08
johnsomID = 'id' lol, I have resisted this urge for a while.18:09
*** sorrison has joined #openstack-lbaas18:11
rm_workhahah yeah18:14
rm_worki specifically avoided commenting on those18:14
rm_workbut ataraday_ did it all anyway XD18:14
*** sorrison has quit IRC18:16
johnsomIt probably does save a small amount of RAM, lol18:16
johnsomHuge shout-out for all of the constants work BTW!18:17
*** maciejjozefczyk has quit IRC18:18
*** vishalmanchanda has quit IRC18:21
*** vishalmanchanda has joined #openstack-lbaas18:21
*** armax has quit IRC18:22
johnsomataraday_ Thoughts on this failure in the listener patch? https://zuul.opendev.org/t/openstack/build/01fe86a687a74a38b8aec472e7a8cfa1/log/controller/logs/screen-o-cw.txt.gz?severity=418:25
ataraday_johnsom, this is how retry works, the expection is thrown and then task is rertied18:26
johnsomHmm, I thought we had a better way. These ERROR logs are going to freak operators out. Our current WARNING is bad enough with people thinking it is failing.18:27
*** sorrison has joined #openstack-lbaas18:28
ataraday_johnsom, if we caught the expection the task won't retry18:29
johnsomYeah, I follow. Just need to think about this a bit.18:29
*** sorrison has quit IRC18:32
ataraday_Sure, I got only one idea - drop redis support - use only zookeper - no reties will be needed :D But this is serious reduction18:33
ataraday_and using zookeper I think is not an option as default for upstream gates I guess18:35
johnsomWell, I think there are two paths in addition, maybe more. 1. Figure out how to mask/block that traceback for these retry exceptions. 2. Switch to using the taskflow notify capability and retry in the code again.18:35
*** elico has joined #openstack-lbaas18:36
johnsomI am rechecking as one of the listener tests failed. It could be just a slow RAX host, but I would like to see it pass.18:36
*** abaindur has joined #openstack-lbaas18:37
*** sorrison has joined #openstack-lbaas18:40
ataraday_yeah, this is the one I wrote you about - see it failing from time to time18:41
*** abaindur has quit IRC18:42
*** sorrison has quit IRC18:44
rm_workyeah it seems to fail in a number of the runs18:46
rm_workvarious ones do :(18:46
rm_workwe'll have to figure it out, but IMO we should figure it out in the final patch in the chain18:47
openstackgerritCarlos Goncalves proposed openstack/octavia master: WIP: Add support to CentOS 8 amphora images  https://review.opendev.org/69712818:47
rm_workmy vote is that we merge the first 8 or so now and then really dive in on the last patch :)18:47
*** ccamposr has joined #openstack-lbaas18:47
*** sorrison has joined #openstack-lbaas18:49
*** ccamposr__ has quit IRC18:50
rm_workhmm in a little bit i'm gonna be grabbing lunch and a siesta, but then i'll be back around, so if those patches get rebased, I will +2 again today :)18:51
johnsomOk, so for the ERROR logging, I think I will either open stories or create an etherpad to track "known issues".18:51
*** abaindur has joined #openstack-lbaas18:51
rm_workyeah maybe we can catch, mangle the exception, and reraise?18:52
johnsomDo we have a preference?18:52
*** abaindur has quit IRC18:52
rm_worki prefer etherpad18:52
rm_workless overhead18:52
*** abaindur has joined #openstack-lbaas18:52
johnsomWell, it's either logging changes or we switch back to in-task retry with the "update_progress" called to keep the job board happy. Those are the two top-of-head thoughts18:53
johnsomI know we talked through this once, but I don't remember the whole decision tree we went down.18:53
rm_worki think we could just catch, update the stacktrace to be something more palatable, and reraise?18:53
*** sorrison has quit IRC18:54
ataraday_I think I posted comment with description of all think in the story18:56
johnsomAh, ok18:56
ataraday_Hm, no - it was just comment on https://review.opendev.org/#/c/647406/9 on the 9th patch set and then we discuss this on meeting18:59
*** gmann is now known as gmann_afk19:00
*** jamesdenton has joined #openstack-lbaas19:00
*** gcheresh has joined #openstack-lbaas19:01
*** armax has joined #openstack-lbaas19:04
johnsomI tried to capture that in the etherpad. Not sure etherpad is going to work well with all the details we need to capture, we can always switch later.19:10
openstackgerritMerged openstack/octavia master: Availability Zone admin API  https://review.opendev.org/69376519:14
*** gcheresh has quit IRC19:19
*** gcheresh has joined #openstack-lbaas19:23
*** ivve has joined #openstack-lbaas19:28
*** ramishra has quit IRC19:45
*** gcheresh has quit IRC20:21
*** gmann_afk is now known as gmann20:25
*** vesper11 has quit IRC20:49
*** vesper11 has joined #openstack-lbaas20:51
*** ccamposr__ has joined #openstack-lbaas20:58
*** sorrison has joined #openstack-lbaas20:59
*** ccamposr has quit IRC21:01
*** sorrison has quit IRC21:04
*** abaindur has quit IRC21:08
*** TrevorV has quit IRC21:10
*** gcheresh has joined #openstack-lbaas21:20
*** sorrison has joined #openstack-lbaas21:27
*** servagem has quit IRC21:33
*** gcheresh has quit IRC21:39
openstackgerritMichael Johnson proposed openstack/octavia master: Convert listener flows to use provider models  https://review.opendev.org/66023621:56
johnsomrebased after the AZ API patch21:56
openstackgerritMichael Johnson proposed openstack/octavia master: Convert pool flows to use dicts  https://review.opendev.org/66538122:00
openstackgerritMichael Johnson proposed openstack/octavia master: Transition member flows to use dicts  https://review.opendev.org/65784222:01
openstackgerritMichael Johnson proposed openstack/octavia master: Transition amphora flows to dicts  https://review.opendev.org/66889822:01
openstackgerritMichael Johnson proposed openstack/octavia master: Convert Lb flows to use provider dicts  https://review.opendev.org/67172522:01
johnsomrebasing a stack now that the az api patch merged.22:01
johnsomHmm, wondering if we don't have a problem in the jobboard chain, I see the pool patch also failed the v2 job with the same listener CRUD timeout. However this one was on inap....22:05
*** pcaruana has quit IRC22:06
sorrisonjohnson: I've also been looking into gate failures for octavia-v2-act-stdby-dsvm-scenario-two-node due to python-yaml not being installed. octavia-v2-act-stdby-dsvm-py2-scenario-two-node works22:06
johnsomWell, the rebase will fire off another run to test it22:06
ataraday_johnsom, thanks for rebase!22:07
johnsomsorrison Thank you for looking at the two-node jobs! They need help.22:07
johnsomsorrison However, I am looking at the worker v2 jobs: octavia-v2-dsvm-scenario-amphora-v2 This is a special test for the jobboard work.22:07
johnsomOnly slightly confusing lol22:07
johnsomataraday_ I'm going to go through the pool patch now22:08
sorrisonjohnson: yeah I'm still trying to figure out how the DIB all fits together. There is something that is run in the py2 version that isn't run in the py3 one22:08
johnsomsorrison I am the local (to Octavia) DIB expert, so let me know if I can help with something22:09
sorrisonI'll ping you when I get in deeper22:10
johnsomHappy to setup a screen sharing session too if it would help you. Just let me know22:10
sorrisonthanks. I'll have another stab at it first22:11
johnsomLast I left the two-node jobs, there was some intermittent network connectivity issue between the nodes. Especially with IPv6.22:15
* haleyb has not completed his two-node job triage22:17
*** armax has quit IRC22:31
*** armax has joined #openstack-lbaas22:32
lxkongsorrison, rm_work, i know you are working on AZ support for LB, but Is it possible to create amphorae for active/passive LB in different AZs?22:33
lxkongi mean, in the current feature design22:34
sorrisonlxkong: Not in the initial work but I think that could be added22:35
johnsomlxkong No, currently active/passive only does nova anti-affinity. There is a patch that does a cross AZ thing, but it requires a special network setup for the cloud and other special patches.22:35
johnsomlxkong Do you have shared L2 networks across AZs?22:35
*** elico has quit IRC22:36
lxkongjohnsom: We have a private cloud customer asking for this feature. They have three AZs deployed in a single region, and they configured very fast L2 network between AZs.22:37
johnsomlxkong Ok, so you do have L2 across AZs such that the VIP could move across AZs. Good to know. That makes the problem space much better. I still wish nova would allow us to do anti-affinity with AZs using server groups, but they don't have that feature.22:38
lxkongjohnsom: ok, so is it easy to implement in octavia side?22:39
johnsomlxkong Ha, no, not easy.....  Basically it's adding nova scheduling to Octavia so it can "decide" the AZs. So, not easy.22:39
cgoncalvesjohnsom, is there a launchpad for nova?22:39
johnsomlxkong I think rm_work has a patch that mostly gets you there, it would just need to be fixed to not require the special networking setup.22:40
lxkongjohnsom: cool, i willl find that patch and take a look22:40
sorrisoncgoncalves: https://bugs.launchpad.net/nova/22:41
johnsomlxkong rm_work will probably be back on in about three hours if you want to chat with him about it22:41
lxkongjohnsom: that would be great :-)22:41
cgoncalvessorrison, ah, lol, sorry. I meant to ask if there's a bug # open in nova requesting AZ anti-affinity22:41
johnsomcgoncalves I don't remember if there is or not22:41
sorrisonna I haven't heard that request in the nova circles22:42
johnsomYeah, the api-ref for server groups only lists "by host", no mention of AZ support: https://docs.openstack.org/api-ref/compute/#server-groups-os-server-groups22:44
sorrisonjohnsom: a bit stuck with this dib issue if you have a sec22:49
sorrisonI'm wondering if its an ordering issue22:49
sorrisonpython3 broken: https://storage.bhs1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_8cd/695291/12/check/octavia-v2-act-stdby-dsvm-scenario-two-node/8cdbc7d/controller/logs/dib-build/amphora-x64-haproxy.qcow2_log.txt.gz22:49
johnsomsorrison Sure, I can make time.22:49
sorrisonpython2 working: https://b649c19a68edc56feea8-49130948639ed40b079dd8450de896f5.ssl.cf2.rackcdn.com/695291/12/check/octavia-v2-act-stdby-dsvm-py2-scenario-two-node/c4d7087/controller/logs/dib-build/amphora-x64-haproxy.qcow2_log.txt.gz22:49
sorrisonthe svc-map needs python-yaml but it doesn't install it in the py3 job22:50
johnsomsorrison Hmm, well, we have PyYAML in our requirements.txt file: https://github.com/openstack/octavia/blob/master/diskimage-create/requirements.txt22:51
*** elico has joined #openstack-lbaas22:51
*** elico has quit IRC22:52
*** elico has joined #openstack-lbaas22:52
openstackgerritBrian Haley proposed openstack/octavia-tempest-plugin master: Create python3 Centos-7 scenario job  https://review.opendev.org/69719822:53
openstackgerritBrian Haley proposed openstack/octavia master: Add back the Centos-7 scenario job  https://review.opendev.org/69719922:53
*** tkajinam has joined #openstack-lbaas22:56
*** rcernin has joined #openstack-lbaas22:57
johnsomsorrison Ok, yes, this is a known issue with devstack.  This task in the devstack ansible installs it: https://zuul.opendev.org/t/openstack/build/8cdbc7d3ff584597a09a8753486406d7/log/job-output.txt#264622:58
johnsomsorrison But it only installs it in python2, not py322:58
johnsomThe real issue is probably that our gates are not installing that diskimage-create/requirements.txt list of packages22:59
johnsomsorrison https://github.com/openstack/octavia/blob/master/devstack/plugin.sh#L5723:02
johnsomSo in there, no where do we install the requirements.txt. That is likely the bug, we should use the devstack alias for python (to get the correct version) and do a pip install -r of our requirements.23:03
sorrisonok, I noticed that DIB_LOCAL_ELEMENTS: openstack-ci-mirrors wasn't set for the 2 node jobs either from what I can tell23:03
johnsomHmm, though come to think of it, DIB should be doing that already. Let me look at that error again.23:04
sorrisonThe logs are from https://review.opendev.org/#/c/695291/12 if you want the non-raw logs23:04
johnsomYeah, ok, the plot gets thicker: https://github.com/openstack/diskimage-builder/blob/master/diskimage_builder/elements/svc-map/package-installs.yaml23:07
johnsomThis is likely a mistake23:07
johnsomWell, maybe not if it installs svc-map inside the image.23:08
johnsomYeah, ok, this is a DIB bug23:08
johnsomso, extra-data.d runs outside the chroot, which means the package-install will not install the required yaml library for the element. It's missing an dependency and is invoking python incorrectly.23:10
cgoncalvesjohnsom is indeed the local DIB expert ;)23:11
sorrisonglad someone here is!23:18
sorrisonStill can't figure out what makes this issue a python3 one when it works in python223:20
*** elico has quit IRC23:23
*** armax has quit IRC23:23
johnsomI think the problem is the bang path in that script: #!/usr/bin/env python23:25
johnsomI think it's running py2 though it should be py3 as that is where the requirements are installed.23:25
sorrisonyeah ok, I see that commit changes it to #!/usr/local/bin/dib-python23:26
sorrisonDo we need to set DIB_PYTHON_VERSION=3 somewhere?23:28
sorrisonno scratch that, looks like that is set correctly23:29
*** abaindur has joined #openstack-lbaas23:40
johnsomYeah, dib-python is only present inside the chroot, so doesn't help here. I'm bouncing ideas off of Ian in the DIB channel.23:43
*** abaindur has quit IRC23:45
*** ivve has quit IRC23:47
*** abaindur has joined #openstack-lbaas23:52
*** abaindur has quit IRC23:53
johnsomsorrison I will take a pass at a DIB fix and will let you know.23:53
*** abaindur has joined #openstack-lbaas23:53

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!