Thursday, 2018-06-07

rm_workjohnsom: the idea is that now, other bits take care of this (because we KNOW it's a deleted amp)00:02
rm_worki thought i already had code in there for this00:02
rm_workbut maybe not?00:04
rm_workbut you should be able to just explicitly check if it's deleted00:04
rm_workno more beating around the bush00:04
johnsomrm_work I'm not following you.  This whole issue is where nova fails to delete the service vm after we delete an amp.  That VM would still be sending heartbeats even though we consider it deleted.00:04
rm_workbut the amp record will still exist00:04
rm_workso we can look it up00:04
rm_workand be like "oh, it's deleted"00:04
johnsomWell, it won't once the cleanup job comes around00:04
rm_workright but...00:05
rm_workwe should know pretty quick00:05
rm_workand the cleanup should be like00:05
rm_workhours to days00:05
rm_workbut i get your point...00:06
johnsomRight, so you have german's lab where nova goes out to lunch in "deleting", they sit there for weeks stuck in this state sending heartbeats.  We want to fail fast those and not try to fail them over00:07
johnsomSo I found a completely un-used task, nuked that, after that there are two places using the get lbs on amp (one is sketch), so I think it's safe to add a filter on deleted to that db repo method to exclude those00:08
rm_worki look forward to the next patchset :P00:15
*** SumitNaiksatam has quit IRC00:15
rm_workoh or did you already00:16
johnsomJust about to post00:16
johnsomrunning tests00:17
johnsomSo, grab a coffee or something.  grin00:17
*** threestrands has quit IRC00:25
*** threestrands has joined #openstack-lbaas00:28
openstackgerritMichael Johnson proposed openstack/octavia master: Improve the error logging for zombie amphora
*** longkb1 has joined #openstack-lbaas00:32
johnsomrm_work There you go00:32
rm_worknice file delete00:33
rm_workalso like the TODO00:33
rm_workthat subquery also seemed kinda dumb, wasn't sure the point exactly00:33
rm_workthis one
rm_workso the subquery gets...00:33
rm_workerr it'd be like00:34
rm_workSELECT load_balancer_id FROM amphora where id = $AMPID;00:34
rm_workwhich ... why does that even need to be a subquery00:34
rm_workthat's just a join00:34
rm_workthis whole thing makes no sense to me00:35
rm_workif you just deleted it and said00:35
rm_work"ok, we need to get the LB record for this amp"00:35
rm_workit would not look remotely like that00:35
johnsomRight it gets the list of lb id's (if there was more than one, then gets the whole LB objects for each00:35
rm_workok but like00:35
rm_worki know we said "multi-lb per amp blah blah"00:35
rm_workthat's physically impossible with this database schema00:36
johnsom(not defending, just clarifying)00:36
rm_workso unless we changed the DB schema...00:36
rm_workwhat are we doing00:36
rm_workcan we just agree to stop dancing around that multi-lb-amp thing00:36
rm_workand just acknowledge it's not a thing00:37
rm_workand start fixing it00:37
rm_workinstead of continuing to add more backflips-and-hoops logic00:37
rm_workjohnsom: ^^00:39
rm_workit simplifies a lot of our code00:39
rm_workwhich we continue to dodge around because "we might do this maybe kinda in the future but probably not"00:39
johnsomI kind of agree, I was just looking for the patch I thought we had from someone wanting to do this multi-lb on amp thing00:40
rm_worki mean ... make your own driver in that case00:40
* rm_work shrugs00:40
rm_workall of this is pluggable00:40
rm_workand this subquery BS is almost unreadable00:40
johnsomIt was this one:
rm_workand the logic it spawns is batshit crazy00:41
johnsomIt is true00:41
johnsomPersonally I think we are heading towards smaller not fatter amps00:41
rm_workreading the story now00:42
rm_worklike... i get it? but ... eugh00:42
johnsomYeah, I personally would support just calling this driver to be single lb per amp00:43
rm_workanyway, IMO, we should write the query to match the CURRENT SCHEMA, and then if we do this schema change, it can fix the query00:43
rm_workwell, i want to say that too, but I see now that this is a db schema issue that would apply to any amp driver00:43
rm_workso to be generic we would need to do this00:43
rm_workbut AT THAT POINT, when the schema changes, we could change this query to be different00:44
johnsomThat patch is so old it should be abandoned frankly00:44
rm_workand that would be a huge schema change, like, non-backwards-compat IMO?00:44
rm_workso i don't even know how we manage that00:44
rm_workdropping a column?!00:44
johnsomSo, are you going to ask me to re-write this method or are we applying the Spackle?00:46
rm_workreviewed with a -2 and some comments00:47
johnsomI meant my patch00:47
rm_workI think the spackle should be "eggshell" color00:47
rm_workyeah ummm00:47
rm_worki don't want to just force this whole change into that patch00:47
rm_workif you are rewriting that query ...00:47
rm_workmaybe now is the time to at least fix THAT one00:47
johnsomstarts purple then turns white as it dries00:47
rm_workbecause that query is like00:48
johnsomIt's really not too bad....00:48
rm_workyou think so? lol00:48
rm_workit's textbook "subquery that is so simple it should be a join" compounded by "SQLAlchemy is lulz to define certain things"00:49
johnsomThe join and single query will be long too00:49
rm_worklet me take a crack at it00:49
rm_worki'll pastebin what i think is right00:49
johnsomYeah, I mean, the subquery thing is efficient and allows the nice modeling of the list of LBs.  It's just not really needed as it won't be a list00:50
rm_workdoes this look right?
johnsomThat might work.  It's missing the not deleted for the amp though00:57
rm_workupdated to provide commentary00:58
rm_workoh i didn't see that on yours00:58
rm_workah whoops00:58
johnsomWhy not just switch it from returning all() and a list to returning one LB object?00:59
rm_workI WANT TO01:00
rm_workthen I'd have to go fix everywhere that uses the function too01:00
rm_workbecause it expects a list01:00
rm_workone sec, screw it01:00
johnsomThere are only two places01:02
rm_workrealized that01:02
johnsomYep, that should do it.01:03
johnsomAre you hacking my patch or should I?01:03
rm_workah, if you want, I have it all done tho01:06
rm_workwell, mostly01:06
johnsomI'll do it real quick01:10
johnsomKeep your review clean..  lol01:10
rm_worki mean i finished01:10
*** JudeC__ has quit IRC01:10
johnsomok, then just do it01:11
rm_worktests running01:14
rm_workthere were a LOT of test changes lol01:14
*** SumitNaiksatam has joined #openstack-lbaas01:14
*** hongbin has joined #openstack-lbaas01:16
openstackgerritAdam Harwell proposed openstack/octavia master: Improve the error logging for zombie amphora
rm_worktests take a long time now <_<01:19
rm_workmake sure that looks good to you01:19
johnsomcannot merge... hmnmm01:20
rm_workone sec let me pull and rebase01:20
rm_workmaybe now?01:20
openstackgerritAdam Harwell proposed openstack/octavia master: Improve the error logging for zombie amphora
rm_worknot sure what that was about01:21
rm_workobviously some change...01:21
rm_workparent changed, so i must have just been behind01:21
rm_workit wasn't based on the failover-failed one we merged01:22
rm_workwhich is where i said i added some skip-logic already01:22
rm_workwhich is not in THIS patch (that change already landed)01:23
johnsomYeah, that looks pretty good to me01:28
rm_worklook forward to seeing the scenario results01:29
rm_worki love the new scenario HM traffic test01:29
rm_workit really makes sure the HM is working properly01:29
rm_workand by HM, I mean ... both01:29
rm_workthe new HealthMonitor traffic scenario really verifies that the HealthManager process is working properly ;)01:29
*** threestrands has quit IRC01:35
*** threestrands has joined #openstack-lbaas01:35
*** threestrands has quit IRC01:36
*** threestrands has joined #openstack-lbaas01:37
*** threestrands has quit IRC01:37
*** threestrands has joined #openstack-lbaas01:37
*** kiennt26 has joined #openstack-lbaas01:37
*** threestrands has quit IRC01:38
openstackgerritAdam Harwell proposed openstack/octavia master: Experimental multi-az support
openstackgerritAdam Harwell proposed openstack/octavia master: WIP: AZ Evacuation resource
openstackgerritAdam Harwell proposed openstack/octavia master: WIP: Floating IP Network Driver (spans L3s)
*** threestrands has joined #openstack-lbaas01:38
*** threestrands has quit IRC01:38
*** threestrands has joined #openstack-lbaas01:38
*** threestrands has quit IRC01:39
*** threestrands has joined #openstack-lbaas01:40
*** threestrands has quit IRC01:40
*** threestrands has joined #openstack-lbaas01:40
*** threestrands has quit IRC01:41
*** blake has quit IRC01:56
rm_workjohnsom: augh, amp failover may be broken01:56
rm_worki mean sorry01:56
rm_work*not* failover explicitly01:56
rm_workand i said the wrong thing anyway01:56
rm_workthe "loadbalancer failover" API call might be broken01:57
rm_workif you run it on a LB that has already had a failover of any type (ie, run it more than one time)01:57
rm_workthe fix is trivial01:57
rm_workI want to update the tempest testing first though, to have it like... do a few tries in a row01:57
johnsomYeah, we always need to test multi failover01:58
johnsomApi or not01:59
openstackgerritAdam Harwell proposed openstack/octavia master: WIP: Floating IP Network Driver (spans L3s)
bzhao__johnsom: Thank you very much for!/story/2002165   and!/story/200216002:03
bzhao__But I still want to make sure that you listed task, for the client certificate authentication part, you suggest that 2 fields would be extended for linstener API, such  as "client_ca_tls_container_ref" for ca certificate, and a "option" for client ca certification(for this, maybe something can named it as "client_ca_cert_type", or some other things like this? :))02:03
bzhao__And the 3rd task "add header insertion" part, I found You suggest that support the ca certification on real service not LB itself. What I introduced is LB(haproxy) to check the client ca certificate to decide whether LB forward the arrived request. Then, I check the our api-ref, the "header insertion" field is insert something in HTTP header then forward the requests to backend server, so I'm confu02:03
bzhao__o I'm confused that whether we are following, haha. But I still think the 3rd task is very valuable for server-client authentication. Here, I just want to make sure that we are the same.02:04
*** threestrands has joined #openstack-lbaas02:11
*** threestrands has quit IRC02:11
*** threestrands has joined #openstack-lbaas02:11
johnsombzhao__: I am cooking dinner now, but might be able to chat in an hour or two.  I guess I was not clear on those as I had a different intent.  I will comment on the story too02:35
openstackgerritAdam Harwell proposed openstack/octavia master: WIP: Floating IP Network Driver (spans L3s)
bzhao__johnsom: Thank you. :) haha02:46
*** kiennt26 has quit IRC02:53
*** threestrands has quit IRC03:13
*** kiennt26 has joined #openstack-lbaas03:32
johnsombzhao__ Ok, can chat now.03:50
bzhao__johnsom: :)03:51
johnsomSo, for the first one, I think the API needs to be extended with two parts:03:51
johnsom1. The reference to the ca certificate used to validate the client certificate.03:51
johnsom2. An option for "No verification" or None, "Optional verification" meaning it will as for a cert, but still pass traffic if no cert is given or it fails validation, and finally "Required" meaning no connection if it isn't a valid cert.03:52
johnsomOptional is handy when you want to present one web site to non-cert holder, but a different if they present a valid cert.03:53
johnsomCome to think about it, there might be another task to add L7 support for valid cert connections.03:54
johnsombzhao__ Does that help clarify my thinking on that Task?03:54
bzhao__johnsom:  Thanks, I'm thinking. .03:55
*** blake has joined #openstack-lbaas03:57
johnsomThis is the equivalent haproxy config:
bzhao__johnsom: I agree. Thanks for this link. It's very helpful to understand. :)03:59
bzhao__johnsom:  Then , I follow u04:00
*** yamamoto has joined #openstack-lbaas04:00
johnsombzhao__ Great!  I have used this in the past to allow new users to sign up and get a cert, but users with certs get logged directly into the web application.04:01
bzhao__johnsom: sorry for the pool haproxy configuration knowledge. ha04:01
johnsombzhao__ No worries!  Ok, next is the "insert headers" task04:01
bzhao__johnsom:  So cool practice04:02
johnsomThis is about giving the backend server information about the client certificate authentication at the load balancer.04:02
*** blake has quit IRC04:02
johnsomWhen we validate a client certificate, we want options to insert an HTTP header that tells the backend server we validated the certificate and this user is "trusted".04:02
bzhao__johnsom: Yeah, so we must to make sure the backend server support HTTPS04:03
johnsomThe second header would extract the "common name" or CN or subject of the certificate and pass it to the backend server in the HTTP header04:03
bzhao__johnsom: Correct, but AFAIK, the HTTPS offloading will be processed in LB. Please feel free to correct. HA04:04
johnsomBoth of these do not require HTTPS on the backend04:04
bzhao__johnsom: OK04:04
johnsombzhao__ Correct, the client cert is validated and the TLS terminated at the load balancer.04:04
johnsomBut, because we are doing that, we need to give the backend web server information about that user.04:05
johnsomOptionally we would insert an HTTP header in the connection to the backend server, like we do today for "X-Forwarded-For", but one that tells the web server we successfully validated their client certificate and one that provides the "common name" that was contained in the certificate we validated.04:06
johnsomFor example, the common name can be the user's e-mail address04:06
bzhao__johnsom:  let me re-taste you said. Oh,04:06
bzhao__johnsom:  Thanks, thanks very much, michael. It make sence to me . It is true that is valuable to implement.04:07
johnsombzhao__ Great! yes, I have used this in the past. The first header lets me know they had a validated certificate, the second let's me know the user we validated.04:08
johnsomThese are powerful security tools04:09
*** blake has joined #openstack-lbaas04:09
bzhao__johnsom: Woow, you thought is very advanced, ha . I greatly admire. :)04:09
bzhao__johnsom: Yeah, our customers want this function.04:10
bzhao__johnsom: Thank you for direction.04:10
johnsombzhao__ Ok, sounds good!  Let me know if you have additional questions or concerns.  I think both RFE are good and should be done.04:11
rm_workhmmm what do i need to do to get bash autocomplete to work for the client? >_>04:11
johnsombzhao__ Also, we will get UDP in for Rocky. It's on my list to review again as soon as I can.  Looking forward to this04:11
rm_worklike openstack-cli in general, not just our plugin04:11
johnsomrm_work Log out and back in again.  It loads, but only in the next bash session.04:12
bzhao__johnsom: OK, the second RFE, is for health check for HTTP head host for dns04:12
johnsomIf that doesn't work, you are missing the autocomplete package04:12
bzhao__johnsom:  Thanks for UDP part.04:12
rm_workjohnsom: L7 for cert validation IS actually a use-case I need internally <_<04:12
rm_workso maybe could work on that04:12
johnsombzhao__ yes, I think the host header is good.04:13
johnsomrm_work We are talking about:!/story/200216504:13
rm_workjohnsom: i'm talking about on my mac, for the autocomplete04:13
rm_worki think i'm missing a package04:13
*** blake has quit IRC04:14
rm_workandyeah, i was just going through scrollback04:14
rm_workso referring to the first thing04:14
bzhao__johnsom:  For health check, you suggested that make it more common, I just want to told u that I will figure out that  common part for Header. HA, not any more questions now, so the fields could be not only the 2 fields I mentioned in the RFE.04:14
rm_workre: [20:54:09]  <johnsom>Come to think about it, there might be another task to add L7 support for valid cert connections.04:14
johnsomrm_work bash-completion I think04:15
bzhao__johnsom:  maybe something like "text" field for common usage04:15
bzhao__johnsom:  sorry, I will leave for lunch, happy to see you comment when my back. :) . Thank you Michael.04:16
rm_workhmmm bash-completion already installed via brew :/ sad04:16
rm_worki wonder if i need to get the openstack cli bash hooks in some special place04:16
johnsombzhao__ I'm not sure I am following. My comment was supporting it for HEAD and GET is the same work, so we should do it at the same time and not limit to HEAD.04:16
johnsombzhao__ I think there is two parameters, one is HTTP version, one is for the host string.04:17
johnsomrm_work yeah, it drops a file somewhere.  I think it actually comes from cliff package04:17
rm_workmaybe it just isn't in the right place for OSX/Brew bash-completion04:18
johnsomrm_work it is /etc/bash_completion.d for me and the file is osc.bash_completion04:18
rm_workyours works in OSX?04:18
johnsomNo, ubuntu04:18
rm_workyeah heh04:18
rm_workcan look for osc.bash_completion tho04:18
rm_worklocate: warning: database ‘/usr/local/var/locate/locatedb’ is more than 8 days old (actual age is 313.2 days)04:19
rm_worklittle bit old04:19
johnsomlol, yeah, a "bit" over 8 days04:20
johnsombzhao__ Are we good on the host header for health monitor?  Do we need to discuss more?04:24
rm_workhmmm seems i don't have osc.bash_completion anywhere :(04:25
johnsomHmm, oddness04:25
johnsomWant me to paste bin it for you?04:25
rm_workohh i think i need to generate it04:26
johnsomOh, that is a large file04:26
johnsomHmm, it is auto-magic on ubuntu04:26
*** hongbin has quit IRC04:26
rm_workheh and their requirements are bad04:27
rm_workneed to manually install oslo_log to use the command04:27
rm_workor it explodes04:27
*** links has joined #openstack-lbaas04:28
rm_workgot it! woo04:29
*** annp has joined #openstack-lbaas04:29
johnsomYeah, the completion is necessary04:30
rm_worki had just made a ton of aliases04:31
johnsomOk, I need to call it a night for IRC.  It's 9:30 PM here.  I will catch you all in the morning.04:35
* rm_work waves04:35
*** blake has joined #openstack-lbaas04:45
*** blake has quit IRC04:50
*** SumitNaiksatam_ has joined #openstack-lbaas04:52
*** SumitNaiksatam has quit IRC04:53
*** SumitNaiksatam_ is now known as SumitNaiksatam04:53
bzhao__johnsom:  Sorry, I'm back for lunch. I saw you comments,  I think I didn't get u at that time, but for now, I know what you want. Sure, we can support GET and HEAD. The extend fields are clear. Thanks.  For "Are we good on the host header for health monitor?  Do we need to discuss more?", I think we get a agreement for this. :) Thank you and good night.05:01
openstackgerritAdam Harwell proposed openstack/octavia master: WIP: Floating IP Network Driver (spans L3s)
*** JudeC_ has joined #openstack-lbaas05:10
*** JudeC_ has quit IRC06:05
*** kobis has joined #openstack-lbaas06:23
*** kobis has quit IRC06:23
*** kobis has joined #openstack-lbaas06:24
*** kobis has quit IRC06:24
*** pcaruana has joined #openstack-lbaas06:35
*** kobis has joined #openstack-lbaas06:51
*** AlexeyAbashkin has joined #openstack-lbaas07:04
*** nmanos has joined #openstack-lbaas07:07
*** kobis has quit IRC07:17
*** kobis has joined #openstack-lbaas07:24
*** JudeC_ has joined #openstack-lbaas07:32
*** Alexey_Abashkin has joined #openstack-lbaas07:34
*** AlexeyAbashkin has quit IRC07:35
*** Alexey_Abashkin is now known as AlexeyAbashkin07:35
*** AlexeyAbashkin has quit IRC07:46
*** AlexeyAbashkin has joined #openstack-lbaas07:56
*** JudeC_ has quit IRC08:01
*** AlexeyAbashkin has quit IRC08:23
*** AlexeyAbashkin has joined #openstack-lbaas08:26
*** AlexeyAbashkin has quit IRC08:33
*** AlexeyAbashkin has joined #openstack-lbaas08:38
openstackgerritAdam Harwell proposed openstack/octavia master: WIP: Floating IP Network Driver (spans L3s)
*** AlexeyAbashkin has quit IRC08:45
*** kiennt26 has quit IRC09:07
*** AlexeyAbashkin has joined #openstack-lbaas09:31
*** kobis has quit IRC09:42
*** links has quit IRC09:48
*** AlexeyAbashkin has quit IRC09:50
*** AlexeyAbashkin has joined #openstack-lbaas10:02
*** links has joined #openstack-lbaas10:05
*** AlexeyAbashkin has quit IRC10:13
*** Alexey_Abashkin has joined #openstack-lbaas10:13
*** Alexey_Abashkin is now known as AlexeyAbashkin10:16
*** longkb1 has quit IRC10:16
*** longkb has joined #openstack-lbaas10:17
*** kobis has joined #openstack-lbaas10:18
*** AlexeyAbashkin has quit IRC10:23
*** AlexeyAbashkin has joined #openstack-lbaas10:25
*** Alexey_Abashkin has joined #openstack-lbaas10:38
*** AlexeyAbashkin has quit IRC10:39
*** Alexey_Abashkin is now known as AlexeyAbashkin10:39
*** links has quit IRC10:39
*** links has joined #openstack-lbaas10:56
*** AlexeyAbashkin has quit IRC11:01
*** yamamoto has quit IRC11:14
*** AlexeyAbashkin has joined #openstack-lbaas11:51
*** amuller has joined #openstack-lbaas11:58
*** yamamoto has joined #openstack-lbaas12:10
*** yamamoto has quit IRC12:21
*** longkb has quit IRC12:27
*** fnaval has joined #openstack-lbaas12:31
*** longkb has joined #openstack-lbaas12:39
openstackgerritNir Magnezi proposed openstack/octavia master: DNM: testing v2 scenario gate
openstackgerritNir Magnezi proposed openstack/octavia master: DNM: testing v2 scenario gate
openstackgerritNir Magnezi proposed openstack/octavia master: DNM: Testing gate addition
*** yamamoto has joined #openstack-lbaas12:49
*** rcernin has quit IRC12:56
*** b_bezak has joined #openstack-lbaas12:57
*** rpietrzak has joined #openstack-lbaas12:59
*** longkb has quit IRC13:00
openstackgerritRafal Pietrzak proposed openstack/octavia master: Signed-off-by: Rafal Pietrzak <> Author: Rafal Pietrzak <>
*** AlexeyAbashkin has quit IRC13:08
*** Alexey_Abashkin has joined #openstack-lbaas13:08
*** Alexey_Abashkin is now known as AlexeyAbashkin13:10
openstackgerritRafal Pietrzak proposed openstack/octavia master: Adding support for the octavia listener X-Forwarded-Proto header insertion. Author: Rafal Pietrzak <>
openstackgerritRafal Pietrzak proposed openstack/octavia master: Adding support for the octavia listener X-Forwarded-Proto header insertion.
*** atoth has joined #openstack-lbaas13:19
*** openstackgerrit has quit IRC13:34
*** AlexeyAbashkin has quit IRC13:43
*** b_bezak has quit IRC13:49
*** b_bezak has joined #openstack-lbaas13:50
*** AlexeyAbashkin has joined #openstack-lbaas13:50
*** b_bezak has quit IRC13:54
*** Alexey_Abashkin has joined #openstack-lbaas14:02
*** AlexeyAbashkin has quit IRC14:02
*** Alexey_Abashkin is now known as AlexeyAbashkin14:02
*** links has quit IRC14:23
*** Alexey_Abashkin has joined #openstack-lbaas14:38
*** AlexeyAbashkin has quit IRC14:41
*** Alexey_Abashkin is now known as AlexeyAbashkin14:41
xgerman_rm_work: johnsom PSA:
xgerman_no impact for us but FYI14:46
xgerman_(and might hose lbaas_v2)14:47
*** rpietrzak has quit IRC14:57
johnsomI don't see that we use that field in either repo15:02
johnsomAt least not in our repo15:12
xgerman_probably worth to let the driver authors know so they can adjust (not that I think they will)15:13
*** Alexey_Abashkin has joined #openstack-lbaas15:20
*** AlexeyAbashkin has quit IRC15:22
*** Alexey_Abashkin is now known as AlexeyAbashkin15:22
*** AlexeyAbashkin has quit IRC15:22
*** AlexeyAbashkin has joined #openstack-lbaas15:23
-openstackstatus- NOTICE: Zuul update for Ansible 2.5 in progress. Scheduler crashed as unexpected side effect of pip upgrade. Will be back and running shortly.15:30
*** kobis has quit IRC15:31
*** Alexey_Abashkin has joined #openstack-lbaas15:37
*** AlexeyAbashkin has quit IRC15:37
*** Alexey_Abashkin is now known as AlexeyAbashkin15:37
*** AlexeyAbashkin has quit IRC15:53
*** kobis has joined #openstack-lbaas15:56
*** JudeC_ has joined #openstack-lbaas16:01
-openstackstatus- NOTICE: The zuul upgrade to ansible 2.5 is complete and zuul is running again. Changes uploaded or approved between 15:25 and 15:45 will need to be rechecked. Please report any problems in #openstack-infra16:10
*** yamamoto has quit IRC16:14
*** kobis1 has joined #openstack-lbaas16:17
*** kobis has quit IRC16:18
*** kobis1 has quit IRC16:19
*** kobis has joined #openstack-lbaas16:20
*** yamamoto has joined #openstack-lbaas16:21
*** SumitNaiksatam has quit IRC16:25
*** kobis has quit IRC16:25
*** yamamoto has quit IRC16:26
*** JudeC_ has quit IRC16:29
*** kobis has joined #openstack-lbaas16:42
*** kobis has quit IRC16:44
*** openstackgerrit has joined #openstack-lbaas16:46
*** SumitNaiksatam has joined #openstack-lbaas16:52
*** JudeC_ has joined #openstack-lbaas17:12
*** yamamoto has joined #openstack-lbaas17:23
*** sshank has joined #openstack-lbaas17:30
*** atoth has quit IRC17:30
*** yamamoto has quit IRC17:31
*** amuller has quit IRC17:31
*** openstackgerrit has quit IRC17:34
cgoncalveswhere/how can I see tracebacks in a devstack env? journalctl isn't showing any and I'm sure octavia is crashing17:35
johnsomThey should all go in the journal, "journalctl -u devstack@o-cw | less"17:36
xgerman_I see them all in journalctl — previously they were oin /opt/stack/logs17:36
cgoncalvesright, that's what I'm doing17:37
cgoncalvesjournalctl -a -f -u devstack@o-api to be precise but doesn't make any difference17:37
johnsomOtherwise, there have been a few times that a try/except block "eats" the exception and traceback17:37
johnsomLook around the code you are expecting to fail and see if there might be a try block eating it17:38
cgoncalvesno try block17:39
*** atoth has joined #openstack-lbaas17:42
cgoncalvesI'll keep looking. I must be missing something trivial17:44
*** atoth has quit IRC17:46
rm_workcgoncalves: debug = True ?17:48
rm_workdoes it hide tracebacks otherwise?17:49
rm_workactually I feel like it shouldn't... but17:49
johnsomYeah, they should come out at a level higher than debug17:49
johnsomI think they are all at ERROR level17:50
*** atoth has joined #openstack-lbaas17:50
cgoncalvesmeh, found it, stupid of me. the caller is indeed wrapping catching all exception and not printing the exception msg17:52
johnsomMy fun this morning is making our DELETED vs delete story consistent...17:55
johnsomThis kind of sums it up:17:55
cgoncalvesrm_work, the barbican acl patch actually doesn't work. when I ran it the other day I did as admin user :/17:55
rm_workcgoncalves: hah, ok yeah, i was honestly super surprised17:57
rm_workjohnsom: T_T17:58
xgerman_cgoncalves: you probably get a lol out of that. I need to write an Octavia installer for OSP12…18:01
rm_workI just had a case where the user didn't set a HM check method18:12
rm_workand what got into the haproxy config was:18:12
rm_workoption httpchk None /18:13
rm_workor maybe it was because he updated the method18:13
rm_workupdated another field18:14
*** rpietrzak has joined #openstack-lbaas18:16
*** sshank has quit IRC18:20
*** kobis has joined #openstack-lbaas18:22
*** SumitNaiksatam has quit IRC18:23
*** SumitNaiksatam has joined #openstack-lbaas18:29
rm_workjohnsom: yeah we allow nulling-out the http_method field on a HM on update18:29
rm_workoriginally we default it to GET (correctly)18:29
rm_workbut if you actually pass 'http_method': None18:29
rm_workduring an update18:29
rm_workit'll null it18:29
rm_workand then the template happily puts "None" into the haproxy config18:30
rm_workwhich breaks healthchecks18:30
rm_workIMO this should not be possible18:30
rm_workor the default should reassert?18:30
rm_workcould just drop it into the template...18:31
rm_workbut I think we should reassert the default somehow18:31
rm_workjust struggling to decide where/how18:31
rm_workit's basically impossible to test updates in our functional testing18:32
rm_workOH WAIT NO IT ISN'T NOW18:32
rm_workDB gets updated after the driver call!18:32
rm_workjohnsom: so we changed our "UPDATE" model18:33
rm_workwe persist those in the DB without relying on the controller-worker to do the update18:33
rm_workso our functional tests just became 100% more useful18:33
cgoncalvesxgerman_, well you could try backporting the octavia bits from tripleo queens (tripleo-heat-templates, tripleo-common, puppet-octavia)18:34
xgerman_yeah,  I probably will just write some shell scripts…18:36
rm_workxgerman_: T_T18:37
*** kobis has quit IRC18:37
rpietrzakHi, I just want thanks you Michael for reviewing my very, very short proposal regarding the X-Forwarded-Proto header;) I will follow you guidance, and ofcourse this test should be based on the listner type, as you have written. Thanks :)18:38
rm_workjohnsom: what do you think should happen if a user passes in "None" as the http_method on an update explicitly?18:40
rm_workshould we just ignore it18:40
rm_workor should we "reset to default"?18:40
rm_workI kinda want to have a "NOT NULL" on the DB for that field too <_<18:42
rm_workeh probably easier to backport without tho18:43
johnsomrpietrzak No problem. Let me know if we can help in any way.  Thanks for the contribution!18:43
johnsomrm_work Just a sec, let me look at this.18:44
johnsomrm_work So this isn't that it wasn't specified (Unset) it's that they actually passed in None?18:45
rm_workso we update the field in the DB to "None"18:45
rm_workor NULL rather18:45
rm_workand the models -> haproxy template don't have defaults18:46
rm_workso: "option httpchk None /"18:46
rm_workmaybe a fix in the haproxy template to default :/18:46
rm_workthat's where I am right now18:46
johnsomrm_work Umm, how????
rm_workEnums must not check against None?18:47
rm_workadd this to octavia/tests/functional/api/v2/test_health_monitor.py18:47
johnsomYeah looking at that now:
*** SumitNaiksatam has quit IRC18:50
*** SumitNaiksatam has joined #openstack-lbaas18:50
rm_workI just went into my DB and did a mass update18:51
rm_workUPDATE health_monitor SET http_method = 'GET' WHERE http_method IS NULL;18:51
rm_workto try to stave off this problem18:51
rm_workbecause of the way the UI here sends updates, like 90% of them were this way <_<18:51
johnsomrm_work Something doesn't add up.  That wsme check is pretty straight forward18:54
rm_worki'd think so yes18:54
rm_workwalking through my test now...18:55
rm_worknope, it's gotta be WSME18:56
rm_workliterally looking at what gets passed into the `put` function in the controller18:56
rm_workit's got `http_method = None`18:56
rm_worknot Unset18:56
*** kobis has joined #openstack-lbaas18:58
*** beisner-sick is now known as beisner19:00
rm_workat the very bottom of octavia/tests/unit/api/v2/types/test_health_monitors.py19:01
rm_workclearly shows WSME allowing None19:01
rm_workah it should be ValueError, whatever19:02
rm_workthe point is it doesn't even raise anything19:02
rm_workerrrr, it doesn't even run the validate19:03
johnsomThat is "not good" as the wsme comment code even calls out the None, so either it's a bug or we aren't doing something right19:03
*** rm_mobile has joined #openstack-lbaas19:08
rm_mobileAh nm19:08
rm_mobileWas just too early. It does validate the children...19:08
rm_mobileBut short circuits for both Unset *and* None19:08
rm_mobileLook at 36119:09
*** rm_mobile has quit IRC19:09
*** rm_mobile has joined #openstack-lbaas19:09
rm_worki think because theoretically you might want to allow a purposeful unsetting of an Enum19:16
rm_workof course, IMO you would just add None to the Enum <_<19:16
*** mugsie has quit IRC19:19
*** mugsie has joined #openstack-lbaas19:21
*** mugsie has quit IRC19:21
*** mugsie has joined #openstack-lbaas19:21
*** mugsie has quit IRC19:21
*** mugsie has joined #openstack-lbaas19:21
*** mugsie has quit IRC19:21
*** mugsie has joined #openstack-lbaas19:21
*** atoth has quit IRC19:22
*** mugsie has quit IRC19:22
*** Guest68045 has joined #openstack-lbaas19:24
*** Guest68045 has quit IRC19:24
*** mugsie_ has joined #openstack-lbaas19:24
*** mugsie_ has quit IRC19:27
*** mugsie_ has joined #openstack-lbaas19:27
*** mugsie_ is now known as _mugsie19:27
*** _mugsie is now known as mugsie_19:28
*** kobis has quit IRC19:51
*** rpietrzak has quit IRC20:09
rm_workand some other words20:20
rm_workthat are non-baked-goods related20:21
johnsomWell, that was double. One for wsme, one for a bug I just found in the tempest code I wrote20:21
johnsomI wish we could fix wsme, but I bet that changing that would break *someone*20:22
*** openstackgerrit has joined #openstack-lbaas20:31
openstackgerritMichael Johnson proposed openstack/octavia-tempest-plugin master: Fix a potential IP address conflict
johnsomSince the tests run in parallel I hit an issue where two LBs happened to pick the same random fixed IP at the same time.20:32
johnsomrm_work FYI, this passes the tempest tests local and live.20:35
rm_workLOL is that seriously a thing20:36
rm_workjohnsom: so i am in favor of updating the renderer to default the value it passes to the template20:36
rm_worklet the DB get nulled I guess20:36
rm_workassume the drivers can handle defaulting20:37
*** pcaruana has quit IRC20:39
johnsomSo, if the type of the HM is TCP it should be None right?20:40
johnsomFrankly, we should remove the defaults from the WSME types and validate/default in the controller code too.  It's lame to be setting url_path for TCP health monitors20:48
johnsomrm_work Should I spin a patch for HM?20:49
rm_workjohnsom: err, wait, let me see what it looks like20:50
rm_workyeah ok so you're right re: db20:51
rm_workso yeah i think the answer is the renderer20:52
rm_workjohnsom: if you've got it open and/or in-progress, sure20:52
openstackgerritRafal Pietrzak proposed openstack/octavia master: Adding support for the octavia listener X-Forwarded-Proto header insertion.
rm_workugh seriously how hard would it be to do TLS Reencryption >_>21:29
johnsomnot that bad21:29
rm_workyeah seems to just need a ca-file21:32
rm_workand ... that's it21:32
rm_worki might just do this21:32
rm_workit's so dumb that we still don't support something so trivial and yet so important21:32
johnsomI have open questions about CRLs21:32
rm_workCRLs are shitty21:32
johnsomBonus would be supporting a client cert too21:32
rm_workuse short-lived certs21:32
johnsomCRLs are not great, but we can't really do OCSP...21:33
rm_workAnchor had the right approach IMO <_<21:33
rm_worki know you weren't a fan, but....21:33
johnsomerrr, I'm not a believer in short lived certs21:33
johnsomBut this is philosophy beer talk21:34
johnsomWhy didn't we write a novel21:34
rm_workwe still can ;)21:34
johnsomJust trying to figure out how to fit it on a line without being too dumb21:34
rm_workpaste me the block21:35
johnsomGets me:21:37
johnsomThat is what I'm going with...21:39
rm_workwhy not just
johnsomDoesn't that whine that you have a hanging (?21:44
rm_workwe do that all over the place21:45
johnsomAh, yeah, I must be thinking of something else21:47
*** SumitNaiksatam has quit IRC22:23
*** rcernin has joined #openstack-lbaas22:24
johnsomUgh this enum thing.....22:52
johnsomhttp_method is not a valid option for health monitors of type None22:52
johnsomThat exception is my new code, but the fact that it took None for the "Mandatory" type field is ....22:52
rm_workso, literally was just gonna do this23:18
rm_worktwo line change23:18
johnsomYeah, it's bigger than that.... Plus I'm fixing the dumb defaults23:18
rm_worki mean... at the renderer23:18
rm_workobviously tests need updating, but23:18
rm_workleave it up to the driver23:19
rm_workit doesn't matter if we always pass something to http_method, because the template won't render the line at all if it's not of type HTTP23:19
rm_workyeah you're getting into the business of a whole validation layer on the front side of the API23:20
johnsomBut we should be validating the API input to match or API spec.  Right now that doesn't line up.23:20
rm_workI was thinking might lighter touch23:20
rm_work*much lighter23:20
rm_workyeah, I suppose so ;)23:20
*** fnaval has quit IRC23:57

Generated by 2.15.3 by Marius Gedminas - find it at!