Friday, 2018-03-16

« Thursday, 2018-03-15 Index Saturday, 2018-03-17 »
*** rcernin has joined #openstack-lbaas00:21
*** AlexeyAbashkin has joined #openstack-lbaas00:32
*** AlexeyAbashkin has quit IRC00:36
openstackgerritMerged openstack/octavia master: Properly test access to tls_refs in the API layer  https://review.openstack.org/55030300:38
openstackgerritMerged openstack/octavia master: Log output of subprocess check_output() errors  https://review.openstack.org/54509000:44
*** harlowja has quit IRC00:49
*** fnaval has quit IRC01:31
*** fnaval has joined #openstack-lbaas01:32
*** AlexeyAbashkin has joined #openstack-lbaas01:52
*** AlexeyAbashkin has quit IRC01:56
*** jaff_cheng has joined #openstack-lbaas02:08
xgerman_rm_work: make sure to look at https://review.openstack.org/#/c/549259/ - will turn it into periodic tomorrow again03:34
*** jaff_cheng has quit IRC03:38
*** jaff_cheng has joined #openstack-lbaas03:39
*** sapd has joined #openstack-lbaas04:03
*** links has joined #openstack-lbaas04:32
*** harlowja has joined #openstack-lbaas04:35
*** harlowja has quit IRC05:22
*** gcheresh has joined #openstack-lbaas05:58
*** gcheresh has quit IRC06:17
*** AlexeyAbashkin has joined #openstack-lbaas06:36
openstackgerritMerged openstack/octavia-dashboard master: Updated from global requirements  https://review.openstack.org/55154006:50
*** AlexeyAbashkin has quit IRC06:51
*** sticker has quit IRC07:05
openstackgerritMerged openstack/neutron-lbaas-dashboard master: Remove tox_install  https://review.openstack.org/55298507:10
*** rcernin has quit IRC07:20
*** kobis has joined #openstack-lbaas07:23
openstackgerritOpenStack Proposal Bot proposed openstack/neutron-lbaas master: Updated from global requirements  https://review.openstack.org/55277207:31
openstackgerritOpenStack Proposal Bot proposed openstack/neutron-lbaas-dashboard master: Updated from global requirements  https://review.openstack.org/55153407:31
openstackgerritOpenStack Proposal Bot proposed openstack/octavia master: Updated from global requirements  https://review.openstack.org/55321307:34
openstackgerritOpenStack Proposal Bot proposed openstack/octavia-dashboard master: Updated from global requirements  https://review.openstack.org/55370807:34
*** AlexeyAbashkin has joined #openstack-lbaas07:50
*** devfaz has quit IRC08:04
*** devfaz has joined #openstack-lbaas08:06
*** kobis has quit IRC08:19
*** kobis has joined #openstack-lbaas08:19
*** devfaz has quit IRC08:19
*** kobis has quit IRC08:20
*** kobis has joined #openstack-lbaas08:20
*** devfaz has joined #openstack-lbaas08:20
*** kobis has quit IRC08:20
*** kobis has joined #openstack-lbaas08:21
*** kobis has quit IRC08:21
*** kobis has joined #openstack-lbaas08:21
*** kobis has quit IRC08:22
rm_workxgerman_: will do ;)08:24
rm_workmmm working gates so good08:24
*** tesseract has joined #openstack-lbaas08:30
*** gcheresh has joined #openstack-lbaas08:38
*** chandankumar is now known as chkumar|off08:42
rm_workxgerman_: yep i see. you can switch it back and get it ready to go08:45
*** gcheresh has quit IRC09:04
*** gcheresh has joined #openstack-lbaas09:07
rm_workblegh, i may have been wrong about the centos image requiring 3g now :/ or else it was intermittent09:40
rm_workah it may just be VERY close, and it was when I installed an extra couple of packages for the haproxy build >_<09:41
rm_worki think actually we can bring it back down to 2gb for now T_T09:41
*** kobis has joined #openstack-lbaas09:47
*** annp has joined #openstack-lbaas09:47
*** kobis has quit IRC09:53
*** kobis has joined #openstack-lbaas09:53
*** kobis has quit IRC09:54
*** kobis has joined #openstack-lbaas09:54
*** kobis has quit IRC09:54
*** kobis has joined #openstack-lbaas09:55
*** kobis has quit IRC09:55
*** salmankhan has joined #openstack-lbaas10:09
*** sapd has quit IRC10:18
openstackgerritMerged openstack/octavia master: Split up extra init steps and start processes  https://review.openstack.org/55102110:27
*** jaff_cheng has quit IRC10:28
*** gcheresh has quit IRC10:43
*** salmankhan has quit IRC10:49
*** salmankhan has joined #openstack-lbaas10:54
*** kobis has joined #openstack-lbaas11:11
*** yamamoto has quit IRC11:24
mnaserDoes Octavia currently support SSL with SNI too?11:45
rm_workyes11:46
*** yamamoto has joined #openstack-lbaas11:58
*** rxcomm5UA66D has joined #openstack-lbaas12:20
*** annp has quit IRC12:24
*** yamamoto has quit IRC12:31
*** yamamoto has joined #openstack-lbaas12:36
mnaserrm_work: cool, thanks for the quick response12:58
rm_worknp :)13:04
xgerman_mnaser: I have been told your Octavia isn’t working well for hodegpodge13:09
xgerman_anyhting we can do to help?13:10
mnaserxgerman_: oh? he has mentioned something along those lines, i haven't had a chance to troubleshoot13:10
mnaserlet me check13:10
xgerman_thanks - that would be fantastic13:10
mnaseri see two in error state13:12
mnaserlet me dig up logs13:12
mnaseroh sigh13:13
mnaserscheduler13:13
mnaserwe used a specific flavor for it13:13
mnaserlet me change soemthing13:13
mnaserok, should be fixed, let me try again13:16
mnaserxgerman_: so the issue seems to have been the fact that we were using a custom private flavor, and we did some scheduler changes in our cloud which forgot that flavor so the instance could never be scheduled.  our amphora's now just use common public flavors and its working well13:21
mnaserhogepodge: ^ :)13:21
*** links has quit IRC13:56
*** chkumar|off is now known as chandankumar14:07
cgoncalvesrm_work: hey! could you please have a look at https://review.openstack.org/#/c/553414/ ?14:11
rm_workmomentarily14:11
cgoncalvesit's blocking octavia and neutron-lbaas backports to queens14:11
cgoncalvesthanks14:11
*** salmankhan has quit IRC14:19
rm_worki just quickly threw the backport over the wall, you don't have to wait on me to fix it if you want14:22
rm_workerr though14:23
rm_workwe ARE setting neutron-lib>=1.13.014:23
rm_workit's unchanged14:23
rm_worki don't see anything that needs to be changed there14:24
cgoncalvesaaah sorry! I checked queens g-r and read it as "neutron>=13.0.0"14:25
cgoncalveshttp://logs.openstack.org/14/553414/1/check/requirements-check/bb797b5/job-output.txt.gz#_2018-03-15_14_57_35_58280214:26
cgoncalvesuh-oh! how can we fix that now...14:26
cgoncalveswell, I guess we could ignore the change in requirements.txt.......?14:27
rm_workerrr14:34
cgoncalvessubmitted new patch set but I fear it will fail badly14:34
rm_workthat's bad14:34
rm_workso they will have to figure out how to deal with / allow this14:34
rm_workthis may be an infra problem now14:34
rm_workbecause that patch won't work14:35
rm_workneutron needs to be in there14:35
rm_worki guess we could add it directly in tox.ini14:35
rm_workah i think14:38
rm_worki can fix it maybe14:38
cgoncalvesplease :)14:38
rm_workinclude it explicitly and lock it to the queens neutron14:39
rm_workdone14:39
openstackgerritGerman Eichberger proposed openstack/octavia master: Periodic job to build + publish diskimage  https://review.openstack.org/54925914:40
rm_workneutron wasn't included in global-reqs at all in queens so i had to do the constraint myself there14:40
rm_workerg i should add a comment14:40
xgerman_rm_work: we don’t support any other protocol than HTTPS for terminated TLS14:41
cgoncalveswhy <13.0.0?14:41
cgoncalvesoh, stupid. nevermind lol14:42
rm_work13.0.0 would be the rocky release code14:42
rm_work12.x will be queens and any stable releases14:42
rm_workactually it's not a dumb question, that's why i'm making a comment there right now14:42
rm_workwhen we backport to pike, we will need to update that to >=11.0.0,<12.0.014:43
rm_workI believe that'll work but let me clear it with the infra folks14:44
*** salmankhan has joined #openstack-lbaas14:46
*** links has joined #openstack-lbaas14:48
*** salmankhan has quit IRC14:50
*** salmankhan has joined #openstack-lbaas14:51
*** chandankumar is now known as chkumar|off14:52
rm_workk i am 99% sure that is correct and should resolve the issue14:54
cgoncalves:D14:55
rm_workeugh there's so many gotchas with this single-create stuff15:00
rm_workit's pretty close but15:00
rm_workthe return structures are giving me headaches15:00
*** yamamoto has quit IRC15:26
*** links has quit IRC15:32
*** yamamoto has joined #openstack-lbaas15:36
openstackgerritwangqi proposed openstack/neutron-lbaas-dashboard master: Update the doc links to the newest ones  https://review.openstack.org/55383015:41
*** yamamoto has quit IRC15:41
*** AlexeyAbashkin has quit IRC15:48
*** yamamoto has joined #openstack-lbaas15:51
*** ianychoi_ has joined #openstack-lbaas15:53
*** yamamoto has quit IRC15:55
johnsomrm_work Have you seen a situation when a flow was in progress but a SIGTERM (15) aborts it mid-flow?15:57
*** vegarl_ has joined #openstack-lbaas15:58
*** ipsecguy_ has joined #openstack-lbaas15:59
*** andreykurilin_ has joined #openstack-lbaas16:01
*** ianychoi has quit IRC16:03
*** ipsecguy has quit IRC16:03
*** redondo-mk has quit IRC16:03
*** andreykurilin has quit IRC16:03
*** vegarl has quit IRC16:03
*** ptoohill has quit IRC16:03
*** ptoohill has joined #openstack-lbaas16:03
*** harlowja has joined #openstack-lbaas16:03
openstackgerritMerged openstack/octavia master: Updated from global requirements  https://review.openstack.org/55321316:03
rm_workjohnsom: not verifiably16:05
johnsomOk, I have my suspects this case is a systemd config that does SIGTERM and then gets impatient and SIGKILLS16:05
*** yamamoto has joined #openstack-lbaas16:06
*** kobis has quit IRC16:09
*** kobis has joined #openstack-lbaas16:09
*** kobis has quit IRC16:10
*** redondo-mk has joined #openstack-lbaas16:10
*** kobis has joined #openstack-lbaas16:10
*** kobis has quit IRC16:11
*** yamamoto has quit IRC16:11
*** kobis has joined #openstack-lbaas16:11
*** kobis has quit IRC16:11
*** kobis has joined #openstack-lbaas16:12
*** kobis has quit IRC16:12
*** kobis has joined #openstack-lbaas16:12
*** kobis has quit IRC16:13
*** kobis has joined #openstack-lbaas16:13
cgoncalvesrm_work: the updated u-c for stable/queens didn't help much https://review.openstack.org/#/c/553399/16:14
*** kobis has quit IRC16:14
cgoncalvesit did help, in fact, but still fails at the gate16:14
rm_workerr16:14
rm_workyeah it helped16:14
rm_workthis is a different problem i think16:14
cgoncalvesno route to host on the multinode one16:16
rm_workyeah this just needs a recheck16:17
cgoncalveslet's hope so16:18
*** yamamoto has joined #openstack-lbaas16:21
*** yamamoto has quit IRC16:25
*** kobis has joined #openstack-lbaas16:31
*** yamamoto has joined #openstack-lbaas16:36
*** yamamoto has quit IRC16:41
*** yamamoto has joined #openstack-lbaas16:51
*** yamamoto has quit IRC16:56
*** harlowja has quit IRC17:05
*** yamamoto has joined #openstack-lbaas17:06
*** yamamoto has quit IRC17:07
*** yamamoto has joined #openstack-lbaas17:07
*** yamamoto has quit IRC17:07
*** kobis has quit IRC17:13
*** AlexeyAbashkin has joined #openstack-lbaas17:17
*** AlexeyAbashkin has quit IRC17:22
*** ianychoi__ has joined #openstack-lbaas17:24
*** ianychoi_ has quit IRC17:27
rm_workwhelp. i've kinda hit a wall17:36
rm_workthis was complex but not too much so, until the part where i have to return the right type17:36
rm_worki can return a full response, or a regular response, but i can't choose depending on circumstance17:37
rm_workAFAICT17:37
rm_worki got to where i'm hacking through runtime frames17:37
rm_workso i think i need to take a step back17:38
*** kobis has joined #openstack-lbaas17:41
*** imacdonn has quit IRC17:44
*** imacdonn has joined #openstack-lbaas17:45
rm_workaugh i actually pulled off the frame/gc thing i think, but it still didn't work17:57
*** harlowja has joined #openstack-lbaas18:01
*** harlowja_ has joined #openstack-lbaas18:04
*** harlowja has quit IRC18:06
*** yamamoto has joined #openstack-lbaas18:08
*** yamamoto has quit IRC18:14
openstackgerritGerman Eichberger proposed openstack/octavia master: Periodic job to build + publish diskimage  https://review.openstack.org/54925918:26
xgerman_maybe rebasing appeases the agtes18:27
openstackgerritMichael Johnson proposed openstack/octavia master: Add a devstack variable for legacy RBAC  https://review.openstack.org/55387918:43
openstackgerritMichael Johnson proposed openstack/neutron-lbaas master: Gate API test for the lbaasv2-proxy plugin  https://review.openstack.org/53935018:45
openstackgerritMichael Johnson proposed openstack/neutron-lbaas master: Gate API test for the lbaasv2-proxy plugin  https://review.openstack.org/53935018:45
*** AlexeyAbashkin has joined #openstack-lbaas18:50
*** salmankhan has quit IRC18:54
*** AlexeyAbashkin has quit IRC18:54
*** tesseract has quit IRC19:00
*** gcheresh has joined #openstack-lbaas19:00
*** yamamoto has joined #openstack-lbaas19:10
*** yamamoto has quit IRC19:20
*** beagles is now known as beagles_biab19:26
*** AlexeyAbashkin has joined #openstack-lbaas19:50
*** gcheresh has quit IRC19:54
*** AlexeyAbashkin has quit IRC19:55
*** fnaval_ has joined #openstack-lbaas20:01
*** fnaval has quit IRC20:01
*** yamamoto has joined #openstack-lbaas20:17
*** yamamoto has quit IRC20:22
*** kobis has quit IRC20:31
*** kobis has joined #openstack-lbaas20:31
*** kobis has quit IRC20:32
*** kobis has joined #openstack-lbaas20:32
*** kobis has quit IRC20:33
*** kobis has joined #openstack-lbaas20:33
*** kobis has quit IRC20:33
*** kobis has joined #openstack-lbaas20:34
*** kobis has quit IRC20:34
rm_worki guess no one else is here today20:47
rm_workor, meetings20:47
rm_worklol20:47
johnsomJust got back from lunch20:47
rm_workthe limitations of this pecan-wsme stuff is killing me20:48
johnsomSomething I can help with?20:48
johnsomI think I'm pretty comfortable with it now.20:48
rm_workeh20:49
rm_workcan't dynamically change the return type20:49
rm_workwhich means i've got this all basically good ... except...20:49
rm_worki can either have it return the Full response, or not20:50
rm_workbut i can't choose dynamically at runtime20:50
rm_worki have to hardcode one20:50
rm_worki'll post what i have20:51
johnsomRight, that is interesting.  Do we need to return more than what we do now for single call?20:51
rm_worki'm starting with Listener20:53
rm_workso ...20:53
rm_workPOST to Listener right now returns the normal response20:53
rm_workbut if we do a listener with pools20:53
rm_workit needs to return a Full response20:53
rm_workbut if I always return a Full response... it'll drastically change what comes back in the normal "just create a listener" scenario20:54
openstackgerritAdam Harwell proposed openstack/octavia master: [WIP] Single-create from Listener object  https://review.openstack.org/55390121:08
rm_workjohnsom: ^^ you can take a look21:08
rm_workmost of the work wasn't too bad21:08
rm_workah i did find some bugs in single-create for LBs :/21:10
rm_workhttps://review.openstack.org/#/c/553901/1/octavia/api/v2/controllers/load_balancer.py@344 and @35521:10
rm_work:(21:10
rm_workbad copy/paste21:10
rm_workmay want to spin that out separately along with the rest of the changes in this file, and backport21:11
johnsomOk, german and I are looking at: http://logs.openstack.org/50/539350/14/check/neutron-lbaasv2-dsvm-api-proxy/6c906cb/controller/logs/screen-q-svc.txt.gz#_Mar_16_19_45_49_63964021:12
*** JudeC has joined #openstack-lbaas21:12
johnsomSlightly puzzled21:12
rm_workhmmmm21:13
rm_workah, proxy21:14
johnsomIt's like neutron is failing the policy check after the call to Octavia was successful21:14
xgerman_yes21:15
johnsomYeah, trying to get a gate running to test the proxy21:15
rm_workso it's possible ... that the tenant id is stored differently21:15
rm_workdashed / undashed21:15
rm_workjust throwing that out there21:15
rm_worki've seen it done both ways and i think octavia and neutron may do it differently21:15
johnsomWell, it may just not be present as we are project_id21:16
xgerman_but then why did the other stuff work?21:16
rm_workI recall that when mixing v1 and v2 in octavia, the ones sent from neutron with v1 have different tenant id formatting that the v2 stuff21:16
johnsomBut, I'm trying to figure out why it's checking after the successful call21:16
rm_workso the lookups were wonky21:16
rm_workyeah not related i guess if that's the issue ;P21:16
johnsomI mean the LB built just fine in Octavia, that part was not a problem21:16
rm_workyeah i guess it should be past the auth code21:17
johnsomAh, it looks like a followup get call21:18
*** yamamoto has joined #openstack-lbaas21:18
xgerman_yep, they do gets to look at quotas21:19
rm_workyeah21:19
rm_work"GET /v2.0/lbaas/loadbalancers/ab2305c5-b21e-4ae6-a6ca-ef6b06e96f06 HTTP/1.1" status: 50021:19
rm_workfails to actually print the tenantid tho21:19
johnsomStill shows a 200 from Octavia on that GET though....21:19
rm_workdespite the %(tenant_id)s21:19
johnsomI just traced the req-dc3c8406-8007-463f-b50e-71dc59d1247621:20
rm_workerr21:21
rm_workdid that id get reused?21:21
*** gcheresh has joined #openstack-lbaas21:21
rm_workit looks like a GET and ... something else21:21
rm_worknot sure how that's possible21:21
johnsomI only see the GET.  I have 10 hits in that log21:22
rm_workah yeah k21:22
rm_workwhat is "Unable to find ':' as separator in tenant_id."21:22
johnsomYeah, my guesstimate is the tenant_id is blank21:23
rm_workprolly21:23
rm_workbecause we don't use that anymore21:23
rm_workwe use project_id21:23
rm_workO_o21:23
*** yamamoto has quit IRC21:23
xgerman_https://github.com/openstack/neutron/blob/e81f4f68d0eb8def599f2addeac56656bd7f12b8/neutron/policy.py#L26321:24
xgerman_it seems that can be ignored21:24
rm_workwell maybe sometimes21:25
rm_workbut i feel like it might be a canary21:25
rm_workhmm ah though it prolly found `_`21:25
xgerman_yep21:25
johnsomWell, it's 500 and erroring out the tempest test so...21:25
johnsomwe can't ignore it21:26
rm_workyeah it's just that message may be a red herring21:26
rm_worknot that the test is ok21:26
xgerman_https://github.com/openstack/neutron/blob/e81f4f68d0eb8def599f2addeac56656bd7f12b8/neutron/policy.py#L43-L4621:26
xgerman_That  looks outright fishy21:26
rm_workit was added 2 months ago21:28
rm_workso, semi-recently in neutron time21:28
johnsomBut, the non-proxy tempest tests are still passing21:29
*** gcheresh has quit IRC21:29
*** AlexeyAbashkin has joined #openstack-lbaas21:29
johnsom::q21:29
xgerman_magic21:30
xgerman_or prob. because they have stuff in the DB21:30
rm_workhmm21:30
rm_workwhat are we even doing in this method with target="tenant" >_>21:32
xgerman_I doubt it’s us… prob. somethign the wrap around each plugin21:32
xgerman_they apparently check somwhow which fields are allowed to be shown?21:33
*** AlexeyAbashkin has quit IRC21:34
*** rm_mobile has joined #openstack-lbaas21:36
xgerman_does this get fixed through rechecks? http://logs.openstack.org/59/549259/28/check/octavia-v1-dsvm-py3x-scenario-multinode/e69be01/job-output.txt.gz#_2018-03-16_20_06_29_67761021:36
rm_workmaybe? probably? that's a new one tho21:37
johnsomYeah, I *think* this is due to new qemu slowness, it seems to be RAX hosts, it looks like the 600 second tempest test timeout is too short for how long it takes a VM to boot.21:38
johnsomxgerman_ I am wondering if you are right, if the extension stuff expects tenant_id in the result set...21:44
xgerman_potentially…21:44
johnsomAh, yep21:44
johnsomhttp://control-that-vm.blogspot.com/2014/03/resourceattributemap-in-neutron.html21:45
johnsomrequired_by_policy21:45
johnsomhttps://github.com/openstack/neutron-lbaas/blob/master/neutron_lbaas/extensions/loadbalancerv2.py#L17321:45
johnsomSo, the proxy needs to fake up that record in the result sets I think21:46
johnsomxgerman_ Do you want me to just add that in this patch?21:47
johnsomBummer, it means the proxy was broken in queens21:47
xgerman_sure, that should be a one liner21:47
johnsomProbably need to make a seperate patch that can be backported21:47
xgerman_mmh, yes21:47
rm_mobiletenant_id is a zombie21:55
xgerman_yep21:56
*** rm_mobile has quit IRC22:07
*** fnaval_ has quit IRC22:07
*** imacdonn has quit IRC22:07
*** mestery has quit IRC22:07
*** rm_mobile has joined #openstack-lbaas22:07
*** fnaval_ has joined #openstack-lbaas22:07
*** imacdonn has joined #openstack-lbaas22:07
*** mestery has joined #openstack-lbaas22:07
*** rm_mobile has quit IRC22:08
*** rm_mobile has joined #openstack-lbaas22:08
rm_mobileWhelp I got netsplit22:09
johnsomjoy, I'm running tox22:10
*** devfaz has quit IRC22:10
*** devfaz has joined #openstack-lbaas22:12
rm_mobileIsn't that pretty normal? :)22:12
rm_mobileI wish detox worked with our crazy tox.ini setup22:13
rm_mobileStuff breaks last time I checked :(22:13
johnsomYeah, just fyi more than anything else22:15
*** yamamoto has joined #openstack-lbaas22:20
*** yamamoto has quit IRC22:26
openstackgerritMichael Johnson proposed openstack/neutron-lbaas master: Gate API test for the lbaasv2-proxy plugin  https://review.openstack.org/53935023:08
johnsomok, I'm not convince I know the real issue, so running again with more debug logging23:08
xgerman_+123:08
*** yamamoto has joined #openstack-lbaas23:22
*** yamamoto has quit IRC23:27
openstackgerritMichael Johnson proposed openstack/neutron-lbaas master: Gate API test for the lbaasv2-proxy plugin  https://review.openstack.org/53935023:45
johnsomblah, friday23:45
xgerman_rm_work: there is an e-mail from somebody asking for an alternative network driver on the ML23:46
xgerman_I was about to point him to your driver but will leave that for you because I am not sure if it would solve his problems23:46
johnsomBeyond the one I responded to?23:46
xgerman_no, that one23:46
johnsomSomeone wanted a non-allowed address pairs driver, which is very different23:47
xgerman_yes, but rm_work L3 driver doesn’t need allowed address pair?23:47
xgerman_since he repoints the FLIP23:48
*** openstackgerrit has quit IRC23:48
« Thursday, 2018-03-15 Index Saturday, 2018-03-17 »

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!