| *** yamamoto has joined #openstack-lbaas | 00:23 | |
| *** yamamoto has quit IRC | 00:27 | |
| *** yamamoto has joined #openstack-lbaas | 00:47 | |
| *** AustinR has quit IRC | 01:30 | |
| *** AustinR has joined #openstack-lbaas | 01:30 | |
| *** yamamoto has quit IRC | 01:42 | |
| *** yamamoto has joined #openstack-lbaas | 02:13 | |
| *** yamamoto has quit IRC | 02:19 | |
| *** yamamoto has joined #openstack-lbaas | 02:20 | |
| *** ricolin has joined #openstack-lbaas | 02:27 | |
| *** goldyfruit_ has joined #openstack-lbaas | 05:26 | |
| *** ramishra has joined #openstack-lbaas | 05:47 | |
| lxkong | hi guys, I met with an issue with stable/stein, http://dpaste.com/1KGXYXT, anyone has seen the similar issue before? | 05:59 |
|---|---|---|
| lxkong | the amphora image OS is Ubuntu Bionic | 05:59 |
| rm_work | oh hey lxkong | 06:23 |
| lxkong | rm_work: hi | 06:23 |
| rm_work | hmmm, not sure | 06:23 |
| rm_work | when was amp image last built? | 06:23 |
| f0o | morning, rm_work looking at it now | 06:23 |
| rm_work | i didn't think the 1.0 API change made it to stein | 06:23 |
| rm_work | ah hey f0o | 06:24 |
| lxkong | rm_work: really? Give me 1 sec | 06:24 |
| rm_work | let me look | 06:26 |
| rm_work | i really thought we did that in train, but maybe time is just getting away from me, lol | 06:26 |
| lxkong | rm_work: image was built on 20190924 | 06:26 |
| lxkong | but i saw a few commits come in just now when i did 'git pull' | 06:27 |
| rm_work | yeah pretty sure it's train | 06:27 |
| rm_work | so image was built from ... ? | 06:27 |
| rm_work | stable/stein also? | 06:27 |
| lxkong | that's what i thought | 06:28 |
| rm_work | yeah hmm i guess we did backport it | 06:28 |
| rm_work | but only 14 days ago | 06:28 |
| lxkong | from stable/stein code, i can see something like `'1.0': AmphoraAPIClient1_0(),` | 06:28 |
| rm_work | err | 06:28 |
| f0o | rm_work: I got a few loadbalancers stuck in pending_create because I failed somewhere with the SSL CA settings, how can I force delete them? | 06:28 |
| rm_work | sorry wrong commit | 06:28 |
| rm_work | f0o: if you killed the worker, you're going to need to go into the DB and update the status on the loadbalancer object to ERROR | 06:29 |
| rm_work | then you can delete | 06:29 |
| f0o | ok | 06:29 |
| rm_work | hmm did we really backport this on June 27? >_> | 06:29 |
| rm_work | i guess it's possible | 06:29 |
| rm_work | ok, so maybe that isn't the issue | 06:29 |
| rm_work | ok no, merged August 30 | 06:30 |
| rm_work | but you should still have it in the amp image | 06:31 |
| lxkong | yes, i was using the latest stable/stein on Sep 24 | 06:31 |
| rm_work | hmmmm | 06:31 |
| lxkong | rm_work: do you think the 1.0 API is not stable for stable/stein? | 06:32 |
| rm_work | you could always try using train :D RC1 should be out, lol | 06:32 |
| rm_work | it SHOULD be | 06:32 |
| rm_work | but ... I haven't personally tested it after that backport | 06:32 |
| rm_work | the gates do though, and they passed | 06:32 |
| lxkong | rm_work: we are using stable/queens, so train is a little bit far away | 06:32 |
| rm_work | but you're on stein for Octavia? | 06:33 |
| rm_work | so what's one more? | 06:33 |
| rm_work | ;) | 06:33 |
| lxkong | we just upgraded from queens to stein | 06:33 |
| lxkong | in the testing stage | 06:33 |
| rm_work | ah | 06:33 |
| lxkong | then i saw the issue | 06:33 |
| rm_work | it's consistent? | 06:34 |
| lxkong | not sure it's related to 1.0 api or something else | 06:34 |
| lxkong | i think so | 06:34 |
| lxkong | when creating the pool | 06:34 |
| lxkong | lb/listener creation are both fine | 06:34 |
| rm_work | AH, if you got that far then it's definitely not that | 06:34 |
| lxkong | but failed when creating a pool | 06:34 |
| rm_work | hmm so the socket died on the amp side | 06:35 |
| rm_work | what do the logs on the CW side look like? | 06:35 |
| lxkong | timeout | 06:35 |
| rm_work | hmm, it's on the reload... interesting | 06:36 |
| rm_work | is it active/standby? and if so, can you get the keepalived journal from the amp? | 06:40 |
| f0o | is there a way to see which ca the worker has currently loaded? It doesnt seem to be the one from the config because openssl s_connect can validate the amphora:9443 connection just fine but the worker complains about unknown CA... | 06:40 |
| lxkong | yes, it's active/standby, the current lb was failed over, but failed, i need to create a new one | 06:41 |
| rm_work | f0o: not that i'm aware of, don't think we built in anything for that | 06:43 |
| rm_work | i suppose you could add a quick log statement to output it on failures | 06:43 |
| f0o | yeah that was my thinking now too | 06:44 |
| f0o | `openssl s_client -host 172.16.3.93 -port 9443 -showcerts -verify 99 -CAfile ca.crt < /dev/null` verifies it just fine, I'm sure I've just missed a config somewhere | 06:44 |
| rm_work | make sure you have followed the cert guide maybe? https://docs.openstack.org/octavia/latest/admin/guides/certificates.html#configuring-octavia | 06:50 |
| f0o | yeah I did | 06:51 |
| rm_work | then not sure :( | 06:51 |
| rm_work | I will admit it can be a little confusing | 06:51 |
| f0o | well it's using /etc/octavia/ca.crt which is what my openssl line is using too | 06:51 |
| f0o | so that is verified and valid | 06:52 |
| rm_work | in which section? | 06:52 |
| f0o | I just put a log.error("%(ca)s", {'ca':CONF.haproxy_amphora.server_ca}) in rest_api_driver.py#626 | 06:53 |
| rm_work | remember client_ca and server_ca are different, and [certificates]/ca_certificate is the same as [haproxy_amphora]/server_ca | 06:53 |
| f0o | that log line gives me /etc/octavia/ca.crt; `openssl s_client -host 172.16.3.228 -port 9443 -verify 1 -CAfile /etc/octavia/ca.crt < /dev/null | grep -i verification` returns `Verification: OK` | 06:54 |
| *** maciejjozefczyk has joined #openstack-lbaas | 06:55 | |
| rm_work | what is the actual error you get? | 06:55 |
| f0o | 2019-10-04 06:55:43.330 3264 WARNING octavia.amphorae.drivers.haproxy.rest_api_driver [-] Could not connect to instance. Retrying.: requests.exceptions.SSLError: HTTPSConnectionPool(host='172.16.3.228', port=9443): Max retries exceeded with url: /0.5/info (Caused by SSLError(SSLError(1, '[SSL: TLSV1_ALERT_UNKNOWN_CA] tlsv1 alert unknown ca (_ssl.c:852)'),)) | 06:55 |
| rm_work | ok so that's the cert on the amp side failing to be authed against what we think it should be (the one generated with local_cert_generator) | 06:56 |
| f0o | oooh | 06:56 |
| f0o | so that's phase2 ? | 06:56 |
| rm_work | so [certificates]/ca_certificate | 06:57 |
| rm_work | yeah uhh | 06:57 |
| rm_work | so i get pretty confused here too, and i just follow the guide completely lol | 06:57 |
| f0o | ca_certificate = /etc/octavia/ca.crt | 06:57 |
| f0o | :D | 06:57 |
| f0o | I'm using ECDSA certs (because we got CFSSL centrally deployed for PKI needs) - if it matters | 06:58 |
| rm_work | no idea | 07:01 |
| rm_work | shouldn't probably | 07:01 |
| f0o | so curl can validate it and only complains about subject name (since it's a UUID and not IP) - openssl can happily validate it too | 07:02 |
| rm_work | which cert is curl validating? | 07:02 |
| rm_work | is curl successfully connecting to the amp and the amp is accepting the cert you gave to curl -- AND curl is also validating the amp's cert successfully? | 07:03 |
| rm_work | if yes -- then i'm out of ideas | 07:03 |
| rm_work | other than check your system time :D | 07:03 |
| f0o | curl is only attempting to connect to :9443 without supplying a client-cert | 07:04 |
| *** AlexStaf has quit IRC | 07:04 | |
| f0o | if I add a temporary host to match the UUID it validates fully and returns a json | 07:05 |
| f0o | {"api_version":"1.0"} | 07:05 |
| rm_work | hmm | 07:05 |
| f0o | let me supply a client-cert | 07:06 |
| rm_work | yeah i am just wondering if it might be the client cert that the CW sends out isn't matching the CA we installed on the amp | 07:06 |
| *** ramishra has quit IRC | 07:06 | |
| rm_work | but the amp should be rejecting your connection if you don't send a client cert | 07:06 |
| f0o | ok the plot thickens | 07:07 |
| f0o | without the client-cert it validates fine | 07:07 |
| rm_work | O_o | 07:07 |
| f0o | with the client-cert I get the same error in curl | 07:07 |
| f0o | so the client-cert must be missing the CA | 07:07 |
| rm_work | it shouldn't allow you to communicate with the amp without a correct client cert | 07:08 |
| rm_work | like, the amp should just hang up | 07:08 |
| rm_work | that's a huge part of the security model, lol | 07:08 |
| rm_work | the amp agent should just reject any connections that don't supply a client cert that is valid against the installed CA | 07:09 |
| rm_work | (the CA which is sent over from the CW during the nova boot process) | 07:09 |
| f0o | you can just curl -vk the amp and it returns json | 07:09 |
| rm_work | uhhh | 07:09 |
| rm_work | wat | 07:09 |
| rm_work | maybe just the api info resource? | 07:09 |
| rm_work | can you get anything else? | 07:09 |
| *** ccamposr has joined #openstack-lbaas | 07:10 | |
| f0o | http://paste.openstack.org/show/dJ1MxDt9gqkuuOiJFRz9/ | 07:10 |
| f0o | what URL should I try? | 07:10 |
| rm_work | err /1.0/details | 07:10 |
| *** rpittau|afk is now known as rpittau | 07:10 | |
| rm_work | ah nm you're on 0.5 | 07:11 |
| rm_work | err | 07:11 |
| rm_work | ah no it does say 1.0 there | 07:11 |
| rm_work | so /1.0/details or /info | 07:11 |
| f0o | {"hostname":"amphora-207117bf-c718-457f-86e9-d8fe44b567fc","haproxy_version":"1.8.8-1ubuntu0.4","api_version":"1.0","networks":{},"active":true,"haproxy_count":0,"cpu":{"total":119417,"user":"1481","system":"748","soft_irq":"16"},"memory":{"total":1009056,"free":742220,"buffers":17416,"cached":104264,"swap_used":0,"shared":644,"slab":32996},"disk":{"used":914747392,"available":8615702528}, | 07:11 |
| f0o | "load":["0.00","0.00","0.00"],"topology":"SINGLE","topology_status":"OK","listeners":[],"packages":{},"udp_listener_process_count":0,"keepalived_version":"1:1.3.9-1ubuntu0.18.04.2","ipvsadm_version":"1:1.28-3ubuntu0.18.04.1"} | 07:11 |
| f0o | works fine | 07:11 |
| f0o | with 1.0/info => {"hostname":"amphora-207117bf-c718-457f-86e9-d8fe44b567fc","haproxy_version":"1.8.8-1ubuntu0.4","api_version":"1.0","keepalived_version":"1:1.3.9-1ubuntu0.18.04.2","ipvsadm_version":"1:1.28-3ubuntu0.18.04.1"} | 07:12 |
| rm_work | uhhhhhh | 07:12 |
| rm_work | hold on | 07:12 |
| f0o | curl -vk https://172.16.3.228:9443/1.0/info | 07:12 |
| *** pcaruana has joined #openstack-lbaas | 07:15 | |
| rm_work | that should NOT work | 07:16 |
| f0o | works just fine :'D | 07:16 |
| f0o | anything else that shouldn't work? | 07:16 |
| f0o | I'm going to deploy ssh-keys to actually log into one and see what's up | 07:17 |
| rm_work | the machine you're doing this from have new enough openssl to do TLSv1.2? :D I assume | 07:19 |
| f0o | ubuntu LTS | 07:19 |
| f0o | so I guess so | 07:19 |
| f0o | it pulled octavia from the stein repo for ubuntu | 07:19 |
| f0o | OpenSSL 1.1.1 11 Sep 2018 | 07:20 |
| *** luksky has joined #openstack-lbaas | 07:21 | |
| rm_work | yeah so, gunicorn should be rejecting you unless you pass a valid client cert | 07:22 |
| rm_work | if it's not, something is F@$&ed | 07:22 |
| rm_work | and I can't even comprehend how it could be broken | 07:22 |
| f0o | where would the cert be on the amphora? | 07:22 |
| rm_work | look in /etc/octavia/amphora_agent.conf | 07:24 |
| rm_work | the path will be listed in [amphora_agent]/agent_server_ca | 07:24 |
| rm_work | maybe the file is ... blank? would that make it allow no-cert? | 07:25 |
| rm_work | I actually don't know but seems like a bug to me | 07:25 |
| rm_work | (if so) | 07:25 |
| f0o | got /etc/octavia/certs/server.pem and /etc/octavia/certs/client_ca.pem | 07:26 |
| f0o | both looks fine and validate against their ca's | 07:26 |
| rm_work | it should be literally impossible to communicate with the amphora agent without sending it a client cert | 07:27 |
| f0o | xD | 07:27 |
| f0o | it works tho | 07:27 |
| f0o | does it work for your setup? | 07:27 |
| rm_work | no | 07:27 |
| f0o | is there a repo where you supply tested and validated amphorae images? | 07:28 |
| f0o | because at this rate I guess the DIB just failed | 07:28 |
| rm_work | hmmmmmmmmmmmmmmmm i take that back, it is working in my cloud | 07:31 |
| rm_work | wtf | 07:31 |
| rm_work | uhhh | 07:31 |
| rm_work | i'm gonna ... be back... in a bit | 07:31 |
| f0o | :D | 07:32 |
| rm_work | this is really really bad | 07:39 |
| rm_work | cgoncalves: you around? | 07:39 |
| *** ivve has joined #openstack-lbaas | 07:41 | |
| f0o | I wrote a bugreport https://storyboard.openstack.org/#!/story/2006660 - but I need to add team-members, who's the best to add? | 07:42 |
| f0o | rm_work cgoncalves I added you two for now, please feel free to expand | 07:43 |
| rm_work | wish johnsom was not on vacation presently. and also that it wasn't midnight. | 07:44 |
| f0o | so what happens when you open 9443 on the tenant's side... will the tenant be able to connect to it too? | 07:46 |
| f0o | or will the 9443 only listen on the management network through netns? | 07:46 |
| rm_work | if the tenant has access to the management network, yes | 07:46 |
| f0o | ah good | 07:46 |
| f0o | so it's not totally bad then | 07:46 |
| rm_work | well | 07:46 |
| rm_work | a lot of people (including my deployment) run with management-on-vip-net | 07:46 |
| rm_work | because the cert validation should be secure | 07:47 |
| * rm_work coughs | 07:47 | |
| f0o | :D | 07:47 |
| f0o | so I guess I should keep the story in private mode then | 07:47 |
| f0o | Do you guys want a CVE assigned for this? | 07:48 |
| rm_work | maybe | 07:48 |
| rm_work | I am currently trying to figure out WTF | 07:48 |
| rm_work | I know this USED to work | 07:48 |
| rm_work | because it used to be super annoying to test amp commands | 07:48 |
| f0o | maybe because it was super annoying a patch slipped through | 07:49 |
| f0o | I'm requesting a CVE | 07:49 |
| rm_work | i'm looking at the code for it right now -- either we're passing the options badly to gunicorn, or gunicorn has a bug | 07:49 |
| rm_work | i need to test gunicorn isolated | 07:50 |
| cgoncalves | hi, morning. | 07:51 |
| cgoncalves | I can open that story. I see it is private due to being related to a security issue | 07:52 |
| cgoncalves | uh, okay... | 07:53 |
| cgoncalves | IIRC it is gunicorn who enforces 2-way authentication | 07:54 |
| f0o | morning cgoncalves | 07:54 |
| cgoncalves | another Friday funday :) | 07:54 |
| rm_work | yes | 07:58 |
| rm_work | testing gunicorn in isolation | 07:58 |
| cgoncalves | f0o, in your last curl command, when you supply the client cert, you might have to also supply the server ca (amphorae's CA) | 07:58 |
| rm_work | looks like it is just ignoring the option to require client certs O_o | 07:58 |
| rm_work | that's super bad | 07:59 |
| rm_work | i tested in my cloud -- i am able to start/stop listeners on an amp just with `curk -k` | 07:59 |
| rm_work | *curl -k | 07:59 |
| rm_work | no certs provided | 07:59 |
| cgoncalves | I'm having a somewhat similar problem on centos 8 but is quite the opposite: even supplying all the good info, the server whines about a bad certificate | 07:59 |
| f0o | cgoncalves:yeah the last one was just to show that client-cert is only enforced/validated when it's supplied | 08:00 |
| rm_work | yeah that's the issue f0o was having that led us to discover this | 08:00 |
| rm_work | if no client cert is supplied .... it just... doesn't validate | 08:00 |
| cgoncalves | toggling "cert_reqs" to False, bypassed the authentication | 08:01 |
| cgoncalves | https://github.com/openstack/octavia/blob/afa917a3dee66bc687d7a6ff8140624241ed03ed/octavia/cmd/agent.py#L77 | 08:01 |
| rm_work | UMM | 08:01 |
| cgoncalves | odd. I'll need to spawn a devstack to check | 08:02 |
| rm_work | OK yeah this is bad | 08:02 |
| rm_work | soooo apparently what that takes as an arg isn't true/false | 08:02 |
| rm_work | it's supposed to take `ssl.CERT_REQUIRED` | 08:02 |
| rm_work | the value of which is *2* | 08:02 |
| cgoncalves | say what! | 08:02 |
| rm_work | WHAT | 08:02 |
| rm_work | this is fucked | 08:02 |
| rm_work | we need to patch and backport NOW | 08:03 |
| cgoncalves | http://docs.gunicorn.org/en/stable/settings.html#cert-reqs | 08:03 |
| rm_work | yeah | 08:03 |
| cgoncalves | the possible values aren't that clear to me | 08:03 |
| rm_work | "(see stdlib ssl module’s)" | 08:03 |
| rm_work | right | 08:03 |
| cgoncalves | one would assume boolean | 08:03 |
| rm_work | YES ONE WOULD | 08:03 |
| rm_work | ONE BEING ME | 08:03 |
| rm_work | *fuck* | 08:03 |
| cgoncalves | I didn't want to point fingers but..... | 08:03 |
| cgoncalves | kidding! | 08:03 |
| cgoncalves | so, resolving this might also resolve the centos 8 problem. let's go! | 08:04 |
| f0o | so for the CVE I kinda need a version for the affected component... Do we have a commit or version when it first appeared? | 08:05 |
| rm_work | yes | 08:05 |
| rm_work | i have it, sec | 08:05 |
| *** openstackgerrit has joined #openstack-lbaas | 08:05 | |
| openstackgerrit | Adam Harwell proposed openstack/octavia master: Fix urgent amphora two-way uth security bug https://review.opendev.org/686540 | 08:05 |
| rm_work | https://github.com/openstack/octavia/commit/48a1e7cbe9a07d8f26d03270a86ed093b4e817e6 | 08:05 |
| rm_work | 3 years ago >_> | 08:05 |
| f0o | ouch | 08:06 |
| rm_work | we need to backport into infinity | 08:06 |
| f0o | damn | 08:06 |
| rm_work | this is *bad* | 08:06 |
| openstackgerrit | Adam Harwell proposed openstack/octavia master: Fix urgent amphora two-way auth security bug https://review.opendev.org/686540 | 08:06 |
| f0o | so since Pike.. | 08:06 |
| cgoncalves | https://docs.python.org/3/library/ssl.html#ssl.SSLContext.verify_mode | 08:06 |
| rm_work | *somewhat* mitigated by using a management-net that's private, but.... | 08:06 |
| rm_work | yep | 08:06 |
| rm_work | "true" == 1 == cert_optional | 08:06 |
| rm_work | is what I'm guessing | 08:07 |
| f0o | lawl | 08:07 |
| f0o | yeah | 08:07 |
| f0o | guess so | 08:07 |
| cgoncalves | Changed in version 3.6: SSLContext.verify_mode returns VerifyMode enum: | 08:07 |
| f0o | do we have a relase log that I could look for versions=>commits? | 08:07 |
| cgoncalves | so this might have got broken since we started supporting py3.6 | 08:07 |
| rm_work | hmmmmmmmmm | 08:07 |
| rm_work | that would be *slightly better* | 08:07 |
| rm_work | but i think not | 08:07 |
| rm_work | https://docs.python.org/2.7/library/ssl.html#ssl.SSLContext.verify_mode | 08:08 |
| rm_work | same | 08:08 |
| openstackgerrit | Adam Harwell proposed openstack/octavia stable/train: Fix urgent amphora two-way auth security bug https://review.opendev.org/686541 | 08:08 |
| gthiemonge | ^ I confirm that the patch fixes the issue in my env | 08:09 |
| cgoncalves | you confirmed THAT fast? wow | 08:10 |
| gthiemonge | :D | 08:10 |
| f0o | wait so this has been going on since amphora-agent version 1.0.0? | 08:11 |
| rm_work | uhhh no | 08:11 |
| rm_work | since .... | 08:11 |
| rm_work | way earlier | 08:11 |
| openstackgerrit | Adam Harwell proposed openstack/octavia stable/stein: Fix urgent amphora two-way auth security bug https://review.opendev.org/686543 | 08:11 |
| openstackgerrit | Adam Harwell proposed openstack/octavia stable/rocky: Fix urgent amphora two-way auth security bug https://review.opendev.org/686544 | 08:12 |
| openstackgerrit | Adam Harwell proposed openstack/octavia stable/queens: Fix urgent amphora two-way auth security bug https://review.opendev.org/686545 | 08:12 |
| openstackgerrit | Adam Harwell proposed openstack/octavia stable/pike: Fix urgent amphora two-way auth security bug https://review.opendev.org/686546 | 08:12 |
| cgoncalves | folks, we have to consider following the vulnerability management process -- https://security.openstack.org/vmt-process.html | 08:12 |
| openstackgerrit | Adam Harwell proposed openstack/octavia stable/ocata: Fix urgent amphora two-way auth security bug https://review.opendev.org/686547 | 08:12 |
| rm_work | yes | 08:13 |
| rm_work | it's kinda late for an embargo, IRC logs and all | 08:13 |
| cgoncalves | this discussion should have taken place in a more restricted group, I think, too | 08:13 |
| cgoncalves | right | 08:13 |
| rm_work | unfortunately didn't realize the scope of the issue until it was already way too late | 08:13 |
| rm_work | we need to draft an impact description I guess | 08:14 |
| f0o | I got this draft for the CVE: | 08:14 |
| f0o | Amphora Images in OpenStack Octavia allows anyone with access to the management network to bypass Client-Certificate authentication and recover information as well as issue configuration commands via simple HTTP requests. | 08:14 |
| rm_work | we also need to figure out a tempest test to make sure this is working in the future | 08:14 |
| rm_work | f0o: seems accurate | 08:15 |
| *** rcernin has quit IRC | 08:15 | |
| f0o | I'm adding all the review URLs as reference as well as the story | 08:15 |
| f0o | filling out the versions rn then we got the CVE request done and we should get one assigned within a few hrs | 08:15 |
| rm_work | so we need a VMT coordinator? | 08:16 |
| f0o | beats me, I'm just used to write CVEs and inform vendors lol | 08:16 |
| f0o | I'm not making many new friends I'm afraid haha | 08:16 |
| cgoncalves | if this also fixes the centos8 issue, I'm your new best friend :D | 08:17 |
| rm_work | lol | 08:17 |
| rm_work | i don't know why suddenly it would CAUSE cert issues | 08:18 |
| rm_work | so ... i *doubt* it? | 08:18 |
| f0o | so i got all versions from 1.0.0 (pike) to 5.1.0dev5 (master) in the CVE now, am I missing some? | 08:18 |
| rm_work | gthiemonge: when you say "fixes the issue", you mean makes a client-cert *actually required*? | 08:18 |
| gthiemonge | I received a "curl: (35) NSS: client certificate not found (nickname not specified)" | 08:19 |
| gthiemonge | rm_work: ^ | 08:19 |
| rm_work | right, k yeah | 08:19 |
| f0o | hrm I see ocata in the reviews, now I get to dig more :'D | 08:20 |
| rm_work | yeah looked back at ocata and even it had the gunicorn swap | 08:20 |
| f0o | 0.5.0 seems fine lol | 08:21 |
| rm_work | the commit that introduced the issue was October 2016 <_< | 08:21 |
| rm_work | 0.5.0 is *forever* | 08:21 |
| rm_work | err, wait, do you mean amp version or octavia version? | 08:22 |
| rm_work | ah yes, 0.5.0 from Sept 23, 2015? lol | 08:23 |
| cgoncalves | https://releases.openstack.org/ | 08:23 |
| cgoncalves | Ocata is technically still on extended maintenance, so anyone can propose fixes | 08:23 |
| rm_work | 0.10.0 | 08:25 |
| f0o | 0.10.0 is the first vulnerable release | 08:25 |
| f0o | : | 08:25 |
| rm_work | yep | 08:25 |
| f0o | :D | 08:25 |
| *** tkajinam has quit IRC | 08:27 | |
| rm_work | FML | 08:29 |
| f0o | CVE request submitted | 08:29 |
| cgoncalves | thank you! | 08:29 |
| f0o | np | 08:29 |
| f0o | let me know when you want the story marked as public or feel free to do it yourselves if you can (idk how your tool works with RBAC and such) | 08:30 |
| rm_work | I can do anything with storyboard tickets I believe | 08:31 |
| f0o | cool :D | 08:31 |
| rm_work | guess I do need to make it public so I can add it to the commit message for carlos, lol | 08:31 |
| rm_work | what's the storyboard ID? | 08:31 |
| f0o | 2006660 | 08:32 |
| f0o | Task 36916 | 08:32 |
| *** yamamoto has quit IRC | 08:35 | |
| *** yamamoto has joined #openstack-lbaas | 08:37 | |
| rm_work | working on addressing your comments cgoncalves | 08:43 |
| *** salmankhan has joined #openstack-lbaas | 08:48 | |
| openstackgerrit | Adam Harwell proposed openstack/octavia master: Fix urgent amphora two-way auth security bug https://review.opendev.org/686540 | 08:54 |
| rm_work | ^^ look good cgoncalves? | 08:54 |
| openstackgerrit | Adam Harwell proposed openstack/octavia stable/train: Fix urgent amphora two-way auth security bug https://review.opendev.org/686541 | 08:55 |
| cgoncalves | rm_work, LGTM | 08:57 |
| rm_work | kk moving it back | 08:57 |
| f0o | LGTM | 08:57 |
| openstackgerrit | Adam Harwell proposed openstack/octavia stable/stein: Fix urgent amphora two-way auth security bug https://review.opendev.org/686543 | 08:57 |
| openstackgerrit | Adam Harwell proposed openstack/octavia stable/rocky: Fix urgent amphora two-way auth security bug https://review.opendev.org/686544 | 08:58 |
| openstackgerrit | Adam Harwell proposed openstack/octavia stable/queens: Fix urgent amphora two-way auth security bug https://review.opendev.org/686545 | 08:59 |
| openstackgerrit | Adam Harwell proposed openstack/octavia stable/pike: Fix urgent amphora two-way auth security bug https://review.opendev.org/686546 | 09:00 |
| openstackgerrit | Adam Harwell proposed openstack/octavia stable/ocata: Fix urgent amphora two-way auth security bug https://review.opendev.org/686547 | 09:00 |
| rm_work | cgoncalves: I think we should have a test in our tempest stuff that tries to connect to an amp with requests, without a cert, and makes sure it gets a deny | 09:01 |
| f0o | I'll bbl - when I get the mitre response I'll paste you the CVE; it might be in 'reserved' state until you publish the announcement but then you at least got the CVE-ID as a reference for it | 09:01 |
| rm_work | I think technically we were maybe supposed to wait for some coordinator person from the security team to file the CVE, but I think it's probably fine, I did approve the report | 09:02 |
| cgoncalves | rm_work, +1 for tempest test | 09:03 |
| rm_work | need to think about where to put that | 09:05 |
| cgoncalves | rm_work, pep8 failed. https://storage.bhs1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_979/686540/3/check/openstack-tox-pep8/979b925/job-output.txt | 09:05 |
| rm_work | eugh of course it did | 09:05 |
| rm_work | what did i do | 09:05 |
| rm_work | wtf? | 09:06 |
| rm_work | ssl before mock? | 09:06 |
| cgoncalves | A-Z sorting | 09:06 |
| rm_work | last I checked, M is before S | 09:06 |
| cgoncalves | uh, lol | 09:07 |
| rm_work | it thinks mock is 3rd-party rofl | 09:07 |
| rm_work | ok sure whatever | 09:07 |
| rm_work | ONE MORE TIME | 09:08 |
| rm_work | cgoncalves: want to look at the tempest part? | 09:08 |
| rm_work | I woke up early today >_< | 09:08 |
| openstackgerrit | Adam Harwell proposed openstack/octavia master: Fix urgent amphora two-way auth security bug https://review.opendev.org/686540 | 09:10 |
| openstackgerrit | Adam Harwell proposed openstack/octavia stable/train: Fix urgent amphora two-way auth security bug https://review.opendev.org/686541 | 09:11 |
| openstackgerrit | Adam Harwell proposed openstack/octavia stable/stein: Fix urgent amphora two-way auth security bug https://review.opendev.org/686543 | 09:12 |
| openstackgerrit | Adam Harwell proposed openstack/octavia stable/rocky: Fix urgent amphora two-way auth security bug https://review.opendev.org/686544 | 09:12 |
| openstackgerrit | Adam Harwell proposed openstack/octavia stable/queens: Fix urgent amphora two-way auth security bug https://review.opendev.org/686545 | 09:13 |
| openstackgerrit | Adam Harwell proposed openstack/octavia stable/pike: Fix urgent amphora two-way auth security bug https://review.opendev.org/686546 | 09:13 |
| openstackgerrit | Adam Harwell proposed openstack/octavia stable/ocata: Fix urgent amphora two-way auth security bug https://review.opendev.org/686547 | 09:14 |
| rm_work | OK so, *maybe this time* | 09:16 |
| *** ramishra has joined #openstack-lbaas | 09:17 | |
| cgoncalves | rm_work, I wouldn't mind but it wouldn't fit my to-do list for the next days at least :/ | 09:21 |
| rm_work | <_< k | 09:21 |
| rm_work | i mean, none of this fit my to-do list for tonight :D | 09:21 |
| cgoncalves | having the unit test already provides some assurance, at least more than before :) | 09:22 |
| rm_work | yeah | 09:22 |
| cgoncalves | I'm considering a patch to gunicorn. I think the doc is not so explicit | 09:23 |
| rm_work | yeah seems like for something that could be a *major security concern* they could have just included some real pointers | 09:26 |
| rm_work | >_> | 09:26 |
| rm_work | but they did say "go look here", I apparently just didn't | 09:26 |
| *** yamamoto has quit IRC | 09:26 | |
| cgoncalves | http://codesearch.openstack.org/?q=cert_reqs&i=nope&files=&repos= | 09:27 |
| cgoncalves | it seems we are the only ones using an integer xD | 09:27 |
| rm_work | would actually be sweet if they could detect a boolean "True" and just... either WARN/ERROR or assume the most secure option | 09:27 |
| rm_work | yep just a painfully bad quick assumption on my part without fully reading the doc, that 0 = False, and True would mean "Yes, required" | 09:29 |
| rm_work | didn't even consider there could be an intermediate option | 09:29 |
| *** yamamoto has joined #openstack-lbaas | 09:30 | |
| rm_work | it was even in the diff (because it was used correctly before) | 09:31 |
| rm_work | https://github.com/openstack/octavia/commit/48a1e7cbe9a07d8f26d03270a86ed093b4e817e6#diff-7ca3969e86a9b21e73d24b1565b22a9cL56 | 09:32 |
| cgoncalves | doh! bad gunicorn doc. you're definitely not to be blamed | 09:33 |
| rm_work | uhh, no, definitely some blame required, lol | 09:33 |
| rm_work | could have done the actual reading | 09:34 |
| rm_work | though why the doc says "0" when it could say "ssl.CERT_NONE" is kinda lame | 09:34 |
| rm_work | tempted to +A | 09:35 |
| *** yamamoto has quit IRC | 09:35 | |
| cgoncalves | no-no. one cannot and should not go that deep especially when docs say it's 0 or 1 | 09:35 |
| rm_work | they don't technically say it's 0 or 1 | 09:35 |
| rm_work | they just give an example 0 | 09:35 |
| rm_work | lol | 09:36 |
| cgoncalves | yeah but... :P | 09:36 |
| cgoncalves | dayou, hi! are you around by chance? | 09:36 |
| dayou | cgoncalves: yep, I am around | 09:36 |
| rm_work | would be nice if you'd be willing to review and +A https://review.opendev.org/#/c/686540/ | 09:37 |
| cgoncalves | dayou, cool! could you please help review these? https://review.opendev.org/#/q/I5619f5e40d7c9a2ee7741bf4664c0d2d08963992 | 09:37 |
| rm_work | or... yeah, technically all of them | 09:37 |
| rm_work | https://review.opendev.org/#/q/topic:amp-cert-reqs-fix+(status:open+OR+status:merged) | 09:37 |
| cgoncalves | rm_work, let's do a train rc2 once this merge | 09:37 |
| rm_work | yes. | 09:37 |
| rm_work | let's release *everything* once this merges. | 09:37 |
| cgoncalves | rm_work, https://review.opendev.org/#/c/685905/ | 09:37 |
| cgoncalves | everything means addressing review comments in https://review.opendev.org/#/c/683202/ | 09:38 |
| rm_work | yeah but that's easy | 09:39 |
| rm_work | they told us what they want :D | 09:39 |
| rm_work | so when we have new hashes, we can update that and fix the version numbers to their liking | 09:39 |
| cgoncalves | we should check why we backported something that bumped requirements | 09:39 |
| rm_work | and add Ocata and Pike | 09:40 |
| rm_work | oh, i think i remember that | 09:40 |
| rm_work | i don't remember WHAT, but i remember the discussion and concluding it wasn't avoidable | 09:40 |
| *** yamamoto has joined #openstack-lbaas | 09:41 | |
| cgoncalves | I don't remember that discussion. either poor memory on my part or I was on PTO, dunno | 09:42 |
| *** salmankhan has quit IRC | 09:45 | |
| cgoncalves | rm_work, https://github.com/benoitc/gunicorn/commit/3e265d4#diff-ad19fa365dab6090496ce83af25c345fR446 | 09:45 |
| cgoncalves | latest published doc not up-to-date, though | 09:45 |
| rm_work | O_o | 09:46 |
| rm_work | ok well there we go i guess | 09:46 |
| rm_work | that was done *a year ago* and still isn't published? | 09:47 |
| cgoncalves | uh, hold on | 09:47 |
| rm_work | not that one year ago would have saved us | 09:47 |
| rm_work | ah, switch to latest and it might be | 09:48 |
| cgoncalves | https://github.com/benoitc/gunicorn/blame/e147feaf8b12267ff9bb3c06ad45a2738a4027df/gunicorn/config.py#L1930 | 09:48 |
| cgoncalves | 6 years ago! | 09:48 |
| rm_work | hmmm, nope | 09:48 |
| cgoncalves | so it is the doc rendering that might be reading the value of CERT_NONE and translating to integer 0 | 09:49 |
| rm_work | rofl | 09:50 |
| rm_work | yeah | 09:50 |
| cgoncalves | latest stable (19.9.0) released 14 months ago. why... | 09:50 |
| rm_work | seems like you're correct | 09:51 |
| rm_work | so the auto-doc thing it uses actually used to translate it to a value, but NOW it doesn't? | 09:51 |
| rm_work | it did the same thing with ssl_version lol | 09:52 |
| rm_work | I assumed that meant SSLv2 but it actually means ssl.PROTOCOL_TLSv1 | 09:52 |
| rm_work | (ssl_version: 2) | 09:52 |
| cgoncalves | ah ha! | 09:56 |
| cgoncalves | https://github.com/benoitc/gunicorn/compare/19.7.1...19.8.0#diff-ad19fa365dab6090496ce83af25c345fR1254 | 09:56 |
| cgoncalves | so, it was fixed in 19.8.0. the .rst looks good from that point on | 09:58 |
| rm_work | yes | 09:58 |
| cgoncalves | but the *published* version looks that has not been refreshed | 09:58 |
| rm_work | but the html doc renderer must parse it | 09:58 |
| rm_work | or something O_o | 09:59 |
| cgoncalves | parse the .rst to go that deep and look at the .py coe? hmmm unlikely | 09:59 |
| rm_work | weird | 09:59 |
| rm_work | dunno | 09:59 |
| rm_work | all of this research just seems like a disingenuous attempt to try to shift blame to a doc issue tho, lol | 10:00 |
| rm_work | but it would be good if that was fixed | 10:00 |
| * rm_work shrugs | 10:00 | |
| rm_work | anyway, i guess we'll get the CVE # in a few hours? | 10:02 |
| rm_work | f0o: is there some sort of link to your submission? | 10:02 |
| *** mugsie has quit IRC | 10:03 | |
| cgoncalves | reporting on #gunicorn | 10:04 |
| f0o | ive just received the mail with the request ID. once I got the actual reserved ID, I will paste it here. now its just to wait for a mitre member to process the request :) (typing on phone is terrible) | 10:05 |
| *** mugsie has joined #openstack-lbaas | 10:05 | |
| f0o | im a bit curious why OpenStack hasnt applied for a CNA to be able to issue CVEs themselves | 10:08 |
| *** yamamoto has quit IRC | 10:08 | |
| f0o | OT: is anchor still alive or did it get abandoned? | 10:11 |
| *** yamamoto has joined #openstack-lbaas | 10:12 | |
| rm_work | it died | 10:12 |
| rm_work | which is sad because I really liked it and I am not aware of a true alternative | 10:13 |
| *** yamamoto has quit IRC | 10:13 | |
| f0o | too bad | 10:13 |
| rm_work | yeah, i feel like we've not done a good job at *really* following this VMT Process, but I guess it is what it is. hard to keep something embargoed when the discovery happens on public/logged IRC chat >_> | 10:13 |
| f0o | true but then this was really just discovered by random chance | 10:16 |
| rm_work | hmm looks like a couple of the backports are going to have test failures, though it's most definitely spurious stuff. guess my day tomorrow is going to be figuring out what's wrong with various stable gates | 10:18 |
| rm_work | speaking of "day tomorrow", I'm gonna leave this to you folks in Europe/Asia | 10:20 |
| cgoncalves | sure. monitoring | 10:21 |
| rm_work | f0o: ... we didn't actually fix your issue though, did we? still invalid CA I would assume | 10:22 |
| cgoncalves | uuuh | 10:22 |
| cgoncalves | https://review.opendev.org/#/c/686541/ | 10:22 |
| cgoncalves | we need to enable -train tempest jobs | 10:22 |
| rm_work | T_T | 10:22 |
| f0o | rm_work probably not but I assume the issue is on me anyway. i'll look into it later today when im back ata computer | 10:22 |
| cgoncalves | on it | 10:23 |
| rm_work | cgoncalves: ah they ran? just grenade failed? | 10:23 |
| openstackgerrit | Carlos Goncalves proposed openstack/octavia-tempest-plugin master: Enable tempest jobs from stable/train https://review.opendev.org/686565 | 10:26 |
| cgoncalves | rm_work, ^ | 10:26 |
| rm_work | looks correct | 10:27 |
| rm_work | quick +2 and heading to bed | 10:27 |
| rm_work | catch you in .... some hours | 10:27 |
| openstackgerrit | Carlos Goncalves proposed openstack/octavia stable/train: Fix urgent amphora two-way auth security bug https://review.opendev.org/686541 | 10:28 |
| cgoncalves | ^ added depends-on | 10:28 |
| cgoncalves | oops, actually... | 10:29 |
| *** ajay33 has joined #openstack-lbaas | 10:30 | |
| rm_work | ? | 10:37 |
| cgoncalves | the depends-on didn't help | 10:38 |
| cgoncalves | https://review.opendev.org/#/c/686541/ | 10:38 |
| *** gcheresh has joined #openstack-lbaas | 10:38 | |
| cgoncalves | having a brain fart now. how do we make octavia stable/train run tempest... | 10:38 |
| *** yamamoto has joined #openstack-lbaas | 10:41 | |
| *** yamamoto has quit IRC | 10:43 | |
| *** nmagnezi has quit IRC | 10:48 | |
| openstackgerrit | Merged openstack/octavia stable/train: Fix healthmonitor message v2 for UDP listeners https://review.opendev.org/685905 | 10:51 |
| *** yamamoto has joined #openstack-lbaas | 10:54 | |
| *** yamamoto has quit IRC | 11:00 | |
| *** yamamoto has joined #openstack-lbaas | 11:03 | |
| openstackgerrit | Merged openstack/octavia stable/stein: Fix healthmonitor message v2 for UDP listeners https://review.opendev.org/685906 | 11:21 |
| *** yamamoto has quit IRC | 11:36 | |
| *** yamamoto has joined #openstack-lbaas | 11:51 | |
| *** yamamoto has quit IRC | 12:00 | |
| lxkong | guys, i am curious why the CI job for stable/stein is doing `git_clone https://opendev.org/openstack/octavia.git /opt/stack/octavia master`, does that mean the job is actually testing master octavia instead of stable/stein? | 12:15 |
| lxkong | e.g. https://openstack.fortnebula.com:13808/v1/AUTH_e8fd161dc34c421a979a9e6421f823e9/zuul_opendev_logs_d79/686543/3/check/octavia-v2-dsvm-scenario-ubuntu-xenial/d79fb9a/controller/logs/devstacklog.txt.gz | 12:16 |
| lxkong | this log is for this patch https://review.opendev.org/#/c/686543/ | 12:16 |
| lxkong | rm_work, johnsom, cgoncalves ^^ | 12:17 |
| lxkong | for other core projects, i can see `git_clone https://git.openstack.org/openstack/nova.git /opt/stack/nova stable/stein` | 12:21 |
| *** goldyfruit_ has quit IRC | 12:22 | |
| *** gcheresh has quit IRC | 12:29 | |
| *** yamamoto has joined #openstack-lbaas | 12:30 | |
| cgoncalves | lxkong, I think it is picking up stable/stein | 12:37 |
| cgoncalves | https://openstack.fortnebula.com:13808/v1/AUTH_e8fd161dc34c421a979a9e6421f823e9/zuul_opendev_logs_d79/686543/3/check/octavia-v2-dsvm-scenario-ubuntu-xenial/d79fb9a/controller/logs/screen-o-cw.txt.gz | 12:37 |
| cgoncalves | 2019-10-04 09:31:39.220 | 5f72c14 Fix urgent amphora two-way auth security bug | 12:38 |
| cgoncalves | 5f72c14 matches https://review.opendev.org/#/c/686543/ | 12:38 |
| lxkong | cgoncalves: do you know where does the magic happen during the installation? | 12:40 |
| *** maciejjozefczyk has quit IRC | 12:40 | |
| *** maciejjozefczyk has joined #openstack-lbaas | 12:41 | |
| *** ivve has quit IRC | 12:41 | |
| f0o | cgoncalves rm_work CVE-2019-17134 assigned | 12:44 |
| gthiemonge | lxkong: it seems that the repo has already been cloned with the correct ref | 12:44 |
| lxkong | gthiemonge: yeah, i wonder how | 12:45 |
| *** yamamoto has quit IRC | 12:50 | |
| cgoncalves | a magician never reveals his secrets :D | 12:51 |
| cgoncalves | lxkong, I think it is this: https://github.com/openstack/devstack/blob/master/functions-common#L545 | 12:52 |
| *** vesper11 has quit IRC | 12:53 | |
| cgoncalves | hmm no, but should be around this | 12:53 |
| *** vesper11 has joined #openstack-lbaas | 12:54 | |
| gthiemonge | cgoncalves: git_ref is master, git_dest already exists and RECLONE is false, so it does nothing | 12:57 |
| gthiemonge | except git show at the end of the function | 12:57 |
| *** ramishra has quit IRC | 12:58 | |
| cgoncalves | ah | 12:59 |
| cgoncalves | https://openstack.fortnebula.com:13808/v1/AUTH_e8fd161dc34c421a979a9e6421f823e9/zuul_opendev_logs_d79/686543/3/check/octavia-v2-dsvm-scenario-ubuntu-xenial/d79fb9a/job-output.txt | 12:59 |
| *** vesper11 has quit IRC | 12:59 | |
| cgoncalves | 2019-10-04 09:21:48.647469 | controller | HEAD is now at cb214ad Merge "Fix healthmonitor message v2 for UDP listeners" | 13:00 |
| cgoncalves | 2019-10-04 09:21:48.647717 | controller | Switched to branch 'stable/stein' | 13:00 |
| cgoncalves | 2019-10-04 09:21:48.647833 | controller | opendev.org/openstack/octavia checked out to: | 13:00 |
| cgoncalves | 2019-10-04 09:21:48.647940 | controller | 5f72c1418acda635f27e3d954666b02a85b15c65 Fix urgent amphora two-way auth security bug | 13:00 |
| cgoncalves | lxkong, ^ | 13:00 |
| cgoncalves | f0o, thanks for sharing the CVE# | 13:08 |
| f0o | you're very welcome :) | 13:09 |
| f0o | I'm at a laptop now so if you need me to do anything let me know. I've informed mitre that OpenStack Security Team might engage them regarding it | 13:10 |
| *** psachin has joined #openstack-lbaas | 13:10 | |
| f0o | I've added the CVE to the story as well. As per VMT, we're supposed to write a Security Advisory now. Who's going to do that? (curious) | 13:10 |
| f0o | we did skip a few parts and went straight to the last 4 steps it seems. Apologies for this | 13:11 |
| cgoncalves | there's nothing to apologize for! | 13:20 |
| cgoncalves | https://security.openstack.org/vmt-process.html#openstack-security-advisories-ossa | 13:21 |
| cgoncalves | f0o, could you fill in that template and send it please? | 13:21 |
| f0o | sure thing | 13:21 |
| cgoncalves | either you or rm_work (PTL and author of the patch) | 13:21 |
| cgoncalves | cool, thanks! | 13:21 |
| f0o | it sais I should supply it to a repo, but I'm uncertain which one | 13:21 |
| openstackgerrit | Carlos Goncalves proposed openstack/octavia master: Fix urgent amphora two-way auth security bug https://review.opendev.org/686540 | 13:23 |
| f0o | cgoncalves https://security.openstack.org/vmt-process.html#downstream-stakeholders-notification-email-private-issues this needs to be done I presume | 13:25 |
| f0o | I will submit the template to the ossa project as outlined in a bit | 13:25 |
| *** yamamoto has joined #openstack-lbaas | 13:26 | |
| cgoncalves | ok. sorry, I'm not familiar with this VMT process either... | 13:26 |
| cgoncalves | one thing is clear to me: the patch should only be merged last in the whole process | 13:27 |
| *** vesper11 has joined #openstack-lbaas | 13:27 | |
| f0o | exactly | 13:27 |
| f0o | I'm unsure how to push to the opendev.org/openstack/ossa project | 13:28 |
| f0o | I assume this will be done via git review as well | 13:28 |
| cgoncalves | it seems we are in the "Embargoed disclosure" phase | 13:28 |
| cgoncalves | we also opened already a bug and push the patch | 13:28 |
| cgoncalves | I think so, yes | 13:28 |
| cgoncalves | https://review.opendev.org/#/q/project:openstack/ossa | 13:28 |
| cgoncalves | I see patch reviews there so yeah | 13:29 |
| cgoncalves | https://review.opendev.org/#/c/674909/ | 13:29 |
| cgoncalves | we could take this as an example | 13:29 |
| cgoncalves | let me know if you need help with that | 13:29 |
| f0o | will do | 13:30 |
| *** vesper11 has quit IRC | 13:31 | |
| *** vesper11 has joined #openstack-lbaas | 13:32 | |
| *** yamamoto has quit IRC | 13:36 | |
| *** AustinR has quit IRC | 13:38 | |
| *** AustinR has joined #openstack-lbaas | 13:39 | |
| *** goldyfruit_ has joined #openstack-lbaas | 13:40 | |
| f0o | cgoncalves:https://review.opendev.org/686724 | 13:41 |
| f0o | cgoncalves: should we add a Depends-On: https://review.opendev.org/686724 to the patches since merge shouldnt happen before OSSA? | 13:48 |
| cgoncalves | f0o, I'd say so, yeah | 13:49 |
| f0o | I'd do it myself but I've no clue how to operate gerrit sadly | 13:50 |
| cgoncalves | no worries, I can do that | 13:51 |
| f0o | cgoncalves: happy with this: https://storage.gra1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_878/686724/1/check/openstack-tox-docs/878bd67/docs/ossa/OSSA-2019-005.html ? | 13:52 |
| cgoncalves | LGTM but I'm no VMT expert :) | 13:53 |
| f0o | yeah I'm sure they will come back with some suggestions :D | 13:53 |
| cgoncalves | I left two comments but are just nits | 13:54 |
| *** ajay33 has quit IRC | 13:56 | |
| f0o | +1 | 13:56 |
| openstackgerrit | Carlos Goncalves proposed openstack/octavia master: Fix urgent amphora two-way auth security bug https://review.opendev.org/686540 | 13:57 |
| f0o | I'm bbl again - keeping an eye on the email/s and will come online if needed (mitre reply or comments on opendev), difficult to do this while on the move ;) | 13:57 |
| *** yamamoto has joined #openstack-lbaas | 13:57 | |
| *** sapd1_x has joined #openstack-lbaas | 13:57 | |
| openstackgerrit | Carlos Goncalves proposed openstack/octavia stable/train: Fix urgent amphora two-way auth security bug https://review.opendev.org/686541 | 13:57 |
| *** gmann_afk is now known as gmann | 13:58 | |
| openstackgerrit | Carlos Goncalves proposed openstack/octavia stable/stein: Fix urgent amphora two-way auth security bug https://review.opendev.org/686543 | 13:59 |
| openstackgerrit | Carlos Goncalves proposed openstack/octavia stable/rocky: Fix urgent amphora two-way auth security bug https://review.opendev.org/686544 | 13:59 |
| openstackgerrit | Carlos Goncalves proposed openstack/octavia stable/queens: Fix urgent amphora two-way auth security bug https://review.opendev.org/686545 | 14:00 |
| openstackgerrit | Carlos Goncalves proposed openstack/octavia stable/queens: Fix urgent amphora two-way auth security bug https://review.opendev.org/686545 | 14:00 |
| openstackgerrit | Carlos Goncalves proposed openstack/octavia stable/pike: Fix urgent amphora two-way auth security bug https://review.opendev.org/686546 | 14:01 |
| openstackgerrit | Carlos Goncalves proposed openstack/octavia stable/ocata: Fix urgent amphora two-way auth security bug https://review.opendev.org/686547 | 14:01 |
| cgoncalves | done | 14:01 |
| *** goldyfruit___ has joined #openstack-lbaas | 14:16 | |
| *** goldyfruit_ has quit IRC | 14:19 | |
| openstackgerrit | Carlos Goncalves proposed openstack/octavia stable/stein: Fix urgent amphora two-way auth security bug https://review.opendev.org/686543 | 14:25 |
| openstackgerrit | Carlos Goncalves proposed openstack/octavia stable/rocky: Fix urgent amphora two-way auth security bug https://review.opendev.org/686544 | 14:25 |
| *** ricolin_ has joined #openstack-lbaas | 14:25 | |
| openstackgerrit | Carlos Goncalves proposed openstack/octavia stable/queens: Fix urgent amphora two-way auth security bug https://review.opendev.org/686545 | 14:26 |
| openstackgerrit | Carlos Goncalves proposed openstack/octavia stable/pike: Fix urgent amphora two-way auth security bug https://review.opendev.org/686546 | 14:26 |
| *** ricolin has quit IRC | 14:26 | |
| openstackgerrit | Carlos Goncalves proposed openstack/octavia stable/ocata: Fix urgent amphora two-way auth security bug https://review.opendev.org/686547 | 14:27 |
| cgoncalves | apologies, missed import ssl. fixed now, I hope | 14:28 |
| *** goldyfruit_ has joined #openstack-lbaas | 14:36 | |
| *** goldyfruit___ has quit IRC | 14:39 | |
| *** maciejjozefczyk has quit IRC | 14:39 | |
| *** ricolin_ has quit IRC | 14:49 | |
| *** Conqueror has quit IRC | 15:05 | |
| *** gcheresh has joined #openstack-lbaas | 15:31 | |
| *** shananigans has joined #openstack-lbaas | 15:36 | |
| *** gcheresh has quit IRC | 15:37 | |
| f0o | cgoncalves: Octavia doesnt seem to have a VMT member assigned. I wonder if we need to engage/poke a VMT member or if they're having the ossa monitored | 15:41 |
| colin- | thanks for helping find this and getting the process rolling f0o et al | 15:41 |
| f0o | let me know if there's anything more I can do :) | 15:43 |
| *** rpittau is now known as rpittau|afk | 15:45 | |
| colin- | somehow you knew i had finished releasing octavia just hours prior :)? very funny | 15:51 |
| colin- | s/releasing/upgrading/ | 15:51 |
| f0o | haha, I wish I knew this ahead of time :'D | 15:52 |
| f0o | I'm still stuck with my CA mismatch issue - regardless of the CVE | 15:52 |
| *** gcheresh has joined #openstack-lbaas | 16:02 | |
| *** gcheresh has quit IRC | 16:13 | |
| *** yamamoto has quit IRC | 16:15 | |
| *** sapd1_x has quit IRC | 16:21 | |
| dswebb | cgoncalves, how long is it between the commits being done in git and a new pypi package being created? | 16:29 |
| *** goldyfruit_ has quit IRC | 16:32 | |
| *** goldyfruit_ has joined #openstack-lbaas | 16:39 | |
| *** goldyfruit_ has quit IRC | 16:46 | |
| cgoncalves | f0o, added 3 ossa core reviewers to your patch | 16:51 |
| *** yamamoto has joined #openstack-lbaas | 16:51 | |
| f0o | cgoncalves: thanks :) | 16:51 |
| cgoncalves | dswebb, it varies. depends on how frequently we propose stable releases | 16:52 |
| cgoncalves | dswebb, I have this one open: https://review.opendev.org/#/c/683202 | 16:52 |
| dswebb | just wondering how fast the cve will make it to a release | 16:52 |
| cgoncalves | oh, that. as soon as it is merged, I'll propose releases | 16:53 |
| cgoncalves | s/merged/merges/ | 16:53 |
| cgoncalves | blah, never mind. bad engrish | 16:53 |
| f0o | cgoncalves: OT - I now swapped my CAs and Certs with the ones created in create_dual_intermediate_CA.sh (adjusted to aes256, 4096 bits and 64char passphrase); Now worker explodes expecting the passphrase to be a 32char urlencoded string; so I dropped the passphrase to 30chars and it stopped complaining. But, still getting that unknown_ca issue... | 16:54 |
| f0o | Further on; I see agent.py referring to agent_tls_protocol but I cant see that documented anywhere. And to top it all, gunicorn seems to use tlsv1 ciphers instead of 1.2/1.3 | 16:55 |
| cgoncalves | f0o, the 32 chars for passphrase was a requirement for fernet | 16:55 |
| cgoncalves | https://cryptography.io/en/latest/fernet/ | 16:55 |
| cgoncalves | "A URL-safe base64-encoded 32-byte key" | 16:55 |
| f0o | I get that from the error, just odd that it wasnt mentioned (or if it was, I wasnt made aware strongly enough) | 16:55 |
| f0o | I feel like I'm 5-yo trying to do basics things lol | 16:56 |
| cgoncalves | which octavia version are you installing? | 16:56 |
| f0o | stein from ubuntu | 16:57 |
| f0o | 4.0.0-0ubuntu1.1~cloud0 | 16:57 |
| cgoncalves | oh, ok. patch hasn't been included in a stein release | 16:57 |
| cgoncalves | https://review.opendev.org/#/c/683070/ | 16:57 |
| cgoncalves | apologies for that. folks will get a lot of bug fixes released in next stable dot versions once https://review.opendev.org/#/c/683202/ merges | 16:58 |
| f0o | nice, sadly unrelated it was just something that got me stuck a bit again haha | 16:58 |
| f0o | I'm going insane... I'm going to revert all changes to create_dual_intermediate_CA.sh and run with it's defaults. if it still doesnt work then it just must be my amphora image | 17:00 |
| cgoncalves | I may have missed some messages where you were having problems with the certs | 17:02 |
| cgoncalves | looking at stein code, I don't see agent_tls_protocol anywhere | 17:02 |
| *** yamamoto has quit IRC | 17:02 | |
| cgoncalves | I know that is in train on | 17:02 |
| f0o | I give up :| | 17:07 |
| f0o | I used straight 1:1 openssl.cnf and create_dual_intermediate_CA.sh and the relevant config from https://docs.openstack.org/octavia/latest/admin/guides/certificates.html - yet I get SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)') | 17:07 |
| f0o | before, with my own CA, I had SSLError(SSLError(1, '[SSL: TLSV1_ALERT_UNKNOWN_CA] tlsv1 alert unknown ca (_ssl.c:852)') | 17:08 |
| cgoncalves | I wonder if https://review.opendev.org/#/c/678923/ has introduced a regression of some sort | 17:10 |
| cgoncalves | because last and early this week I was playing with centos 8 and I was having issues very similar to yours | 17:11 |
| cgoncalves | ubuntu bionic and centos 7 amphora images worked just fine, but hmmm | 17:12 |
| f0o | let's build a bionic image then.. | 17:12 |
| cgoncalves | what do you have now? | 17:13 |
| f0o | whatever the default is | 17:13 |
| cgoncalves | bionic lol | 17:13 |
| f0o | I just ran ./diskimage-create.sh in the venv | 17:13 |
| f0o | the help-page showed xenial as default tho | 17:13 |
| *** mjblack has left #openstack-lbaas | 17:13 | |
| f0o | now I'm doing diskimage-create.sh -d bionic | 17:14 |
| cgoncalves | ah, never mind. default is xenial in stein, yes | 17:14 |
| f0o | regarding your change link however; I also tried with a single CA where clients and servers are all signed by the same CA and no passwords - that ended up in same issues | 17:14 |
| f0o | cgoncalves: why does diskimage-create.sh, regardless of what branch/tag you're on, always fetch the latest and greatest octavia release in the image? Shouldn't they be scoped to the branch you're executing it from? | 17:23 |
| f0o | I ran it now from stable/stein and can see that it installed octavia-5.0.0dev5 | 17:23 |
| f0o | sorry 5.1.0dev5 | 17:24 |
| cgoncalves | rm_work, ^ :D | 17:25 |
| cgoncalves | f0o, see https://github.com/openstack/octavia/tree/master/diskimage-create#environment-variables | 17:25 |
| cgoncalves | set DIB_REPOREF_amphora_agent accordingly | 17:25 |
| f0o | not that in the end all my issues are because the image is running 5.1.0dev5 and my env is on 4.0.0 lol | 17:26 |
| cgoncalves | this is a recurring problem for many people, even core reviewers as we even forget about this caveat | 17:26 |
| cgoncalves | rm_work proposed https://review.opendev.org/#/c/686227/ that will hopefully help | 17:27 |
| f0o | +1 | 17:28 |
| f0o | let's see if this image with bionic and stable/stein will work magically | 17:28 |
| cgoncalves | it might fail due to still unknown reasons. we received reports of users failing to build one | 17:30 |
| f0o | official images could help :P | 17:30 |
| cgoncalves | we haven't understood yet how come it builds fine in CI | 17:30 |
| *** gcheresh has joined #openstack-lbaas | 17:32 | |
| *** psachin has quit IRC | 17:36 | |
| f0o | nope still CERTIFICATE_VERIFY_FAILED | 17:49 |
| colin- | :( | 17:49 |
| f0o | On a different note, I can see in the logs that it want to call 172.16.1.85:9443/0.5/info | 17:50 |
| f0o | but the amphora 404's on that, however 1.0/info works | 17:50 |
| cgoncalves | that is expected. for backward compat | 17:50 |
| f0o | again unrelated to the actual issue of controller unable to communicate with amphora | 17:50 |
| f0o | alright | 17:50 |
| *** gcheresh has quit IRC | 17:51 | |
| cgoncalves | rm_work, been checking gunicorn source code. I think the option agent_tls_protocol isn't used at all by gunicorn... | 18:04 |
| cgoncalves | I mean, it is but limited to a specific type of worker (a non-default one). we use the default | 18:04 |
| cgoncalves | gaiohttp uses respects desired SSL version, while sync (default worker) doesn't seem to | 18:05 |
| f0o | cgoncalves: I got my TLS issues resolved now. However as 0.5/info returns a 404, the worker will stackdump me with the exception of `octavia.amphorae.drivers.haproxy.exceptions.NotFound: Not Found` and throws the amphora into Failure/Error state | 18:05 |
| f0o | One step forward, two steps back :'D | 18:05 |
| f0o | I verified the amphora is now on stable/stein, logs say 4.2.0. The worker is 4.0.0 sharp (ubuntu upstream) | 18:06 |
| f0o | is there a change in the minor version that deprecated 0.5? | 18:06 |
| cgoncalves | phewww, I'm really sorry for all the trouble. I might know why | 18:08 |
| f0o | :D | 18:08 |
| f0o | no worries, this is an adventure | 18:09 |
| cgoncalves | so, you have an amphora image using stein code newer than your controllers | 18:09 |
| cgoncalves | and your controllers do not include https://review.opendev.org/#/c/673518/ | 18:09 |
| f0o | ok so i set DIB_REPOREF_amphora_agent to 4.0.0 and it should "fix" it? | 18:10 |
| cgoncalves | I'd say so | 18:10 |
| f0o | Do we have anyone from canonical on the team that could perhaps bump their packages? ;) | 18:11 |
| cgoncalves | that patch I liked, although merged in stable/stein, has not been released as part of a stein dot release. again, we are pending on the releases team to approve my patch | 18:12 |
| f0o | :D | 18:13 |
| f0o | https://wiki.openstack.org/wiki/Security-SIG#Security_Advisories_-_OSSA | 18:13 |
| f0o | wrong copy buffer | 18:13 |
| f0o | sorry | 18:13 |
| f0o | sometimes my shell desyncs from my klipper | 18:14 |
| f0o | or viceversa | 18:14 |
| rm_work | Ugh ok, awake | 18:18 |
| cgoncalves | eww! the more I look at gunicorn the more I dislike it | 18:19 |
| cgoncalves | I think we need this: https://github.com/benoitc/gunicorn/pull/2012 | 18:20 |
| cgoncalves | https://github.com/benoitc/gunicorn/issues/1140 | 18:20 |
| cgoncalves | alternatively, we could switch default worker to gtornado or gaiohttp and try | 18:20 |
| cgoncalves | https://docs.python.org/3/library/ssl.html#ssl.wrap_socket vs https://docs.python.org/3/library/ssl.html#ssl.SSLContext.wrap_socket | 18:21 |
| cgoncalves | "If we use gunicorn instead, we should have many less problems!" | 18:24 |
| cgoncalves | rm_work, ^ what do you have to say in your defense? :P | 18:24 |
| rm_work | Ugh | 18:24 |
| cgoncalves | https://pypi.org/project/Werkzeug/#history | 18:24 |
| cgoncalves | way more active than gunicorn | 18:24 |
| rm_work | Nothing, I resign myself to failure | 18:24 |
| rm_work | Nice that they expose options that just do nothing | 18:25 |
| rm_work | Though the CW side should successfully force the right TLS anyway so no controller will use the wrong one | 18:25 |
| cgoncalves | https://pypi.org/project/Werkzeug/#history | 18:27 |
| cgoncalves | way more active and supports python 3.3 whereas gunicorn doesn't | 18:28 |
| rm_work | Great so https://github.com/benoitc/gunicorn/issues/1934 is probably your issue? | 18:29 |
| johnsom | Werkzeug had huge problems for us, fyi. Super unstable | 18:31 |
| rm_work | Yes | 18:31 |
| rm_work | Umm.... uwsgi? T_T | 18:31 |
| rm_work | FML | 18:32 |
| f0o | stupid question, how do I update a submitted change? | 18:34 |
| f0o | because it just created a new change now :| | 18:38 |
| rm_work | Where? | 18:38 |
| colin- | i think i used this last time i was unsure f0o: https://docs.openstack.org/contributors/code-and-documentation/using-gerrit.html | 18:38 |
| rm_work | So as long as the change-id line is present in the commit message, it will update the existing change | 18:39 |
| rm_work | Don't delete that line O_o | 18:40 |
| f0o | well my initial commit didnt had any of that loine | 18:40 |
| colin- | wow sorry i totally misunderstood what you were asking. disregard | 18:40 |
| f0o | so I just added another commit and woop it created a new change haha | 18:40 |
| f0o | I'll just use the web-editor like in the link from colin- | 18:40 |
| rm_work | Ah, always use --amemd | 18:42 |
| rm_work | *--amend | 18:42 |
| rm_work | The change-id line comes from a postcommit hook | 18:43 |
| rm_work | So it was there before you typed git-review | 18:43 |
| rm_work | In Gerrit you only operate on a single commit, like if you were force-pushing to a repo | 18:44 |
| f0o | good to know | 18:45 |
| f0o | I'm so used to GitHub and GitLab with their pull requests where you just stack commits on top and have them squashed on merge | 18:46 |
| cgoncalves | FWIW, octavia is the only openstack/* project requiring gunicorn | 18:48 |
| rm_work | ugh | 18:49 |
| *** pcaruana has quit IRC | 18:51 | |
| f0o | | 8b357738-2892-4b16-90ed-7be6b3d2d5b1 | test-lb | c3caf1b55bb84b78a795fd81838e5160 | 192.168.123.167 | ACTIVE | amphora | | 18:53 |
| f0o | :O | 18:53 |
| rm_work | cgoncalves: so https://review.opendev.org/#/c/686565/1 ? | 18:54 |
| cgoncalves | must be a bug... :D | 18:54 |
| rm_work | were you fixing that? | 18:54 |
| f0o | it only took... 3 days, 1 CVE, countless of nerves from cgoncalves and rm_work - but it's finally running | 18:54 |
| colin- | easy right? | 18:54 |
| f0o | totally | 18:54 |
| colin- | :) | 18:54 |
| rm_work | f0o: what did you have to do | 18:54 |
| colin- | he was on new amp old controllers | 18:54 |
| colin- | switched that and he's good | 18:54 |
| rm_work | aahhhh yeah | 18:54 |
| cgoncalves | rm_work, see my last comment in https://review.opendev.org/#/c/686541/ | 18:54 |
| f0o | rm_work: sell my soul to the devil, then issue `DIB_REPOREF_amphora_agent=feb640d99d392167ca37e1c7c02d895a03f32172 ./diskimage-create.sh` on master to get bionic amphora and 4.0.0 agent | 18:55 |
| rm_work | ahh kk | 18:55 |
| f0o | I had to use the full commit because the tag 4.0.0 wouldnt work | 18:55 |
| rm_work | weird | 18:55 |
| rm_work | well anyway with my patch it'd just use current | 18:55 |
| rm_work | I use this: | 18:55 |
| rm_work | export DIB_REPOREF_amphora_agent=$(git --git-dir="$OCTAVIA_DIR/.git" log -1 --pretty="format:%H") | 18:56 |
| f0o | current would've killed it too, I tried stable/stein but ubuntu isn't at stable/stein, it's at 2 minors below | 18:56 |
| f0o | so i actually need to match 4.0.0 | 18:56 |
| f0o | because of /0.5/ to /1.0/ change | 18:56 |
| f0o | this was a very long adventure | 18:56 |
| f0o | and I'm running the stock aes128, 1024bit dual intermediate setup which I need to change tomorrow to something more solid | 18:57 |
| f0o | but at least I got a working concept | 18:57 |
| rm_work | cgoncalves: so we aren't actually VMT managed, and this issue isn't embargoed, i THINK we can just merge the fix whenever | 19:01 |
| rm_work | no need to wait | 19:01 |
| *** maciejjozefczyk has joined #openstack-lbaas | 19:11 | |
| cgoncalves | rm_work, agreed. someone in a warm and sunny island brought it to my attention | 19:32 |
| cgoncalves | rm_work, f0o: we can abandon f0o's ossa patch, drop the depends-on on your patch and merge | 19:33 |
| rm_work | eh dont have to abandon it i think | 19:33 |
| rm_work | but yeah don't need depends-on | 19:34 |
| *** maciejjozefczyk has quit IRC | 19:40 | |
| *** shananigans has quit IRC | 20:01 | |
| *** gcheresh has joined #openstack-lbaas | 20:06 | |
| openstackgerrit | Adam Harwell proposed openstack/octavia master: Fix urgent amphora two-way auth security bug https://review.opendev.org/686540 | 20:15 |
| openstackgerrit | Adam Harwell proposed openstack/octavia stable/train: Fix urgent amphora two-way auth security bug https://review.opendev.org/686541 | 20:15 |
| openstackgerrit | Adam Harwell proposed openstack/octavia stable/stein: Fix urgent amphora two-way auth security bug https://review.opendev.org/686543 | 20:16 |
| openstackgerrit | Adam Harwell proposed openstack/octavia stable/rocky: Fix urgent amphora two-way auth security bug https://review.opendev.org/686544 | 20:16 |
| openstackgerrit | Adam Harwell proposed openstack/octavia stable/queens: Fix urgent amphora two-way auth security bug https://review.opendev.org/686545 | 20:17 |
| openstackgerrit | Adam Harwell proposed openstack/octavia stable/pike: Fix urgent amphora two-way auth security bug https://review.opendev.org/686546 | 20:18 |
| openstackgerrit | Adam Harwell proposed openstack/octavia stable/ocata: Fix urgent amphora two-way auth security bug https://review.opendev.org/686547 | 20:18 |
| *** gcheresh has quit IRC | 20:30 | |
| *** dswebb has quit IRC | 20:45 | |
| *** dswebb has joined #openstack-lbaas | 20:45 | |
| *** goldyfruit_ has joined #openstack-lbaas | 21:38 | |
| *** ccamposr has quit IRC | 21:45 | |
| *** ccamposr has joined #openstack-lbaas | 21:45 | |
| *** goldyfruit_ has quit IRC | 22:04 | |
| *** goldyfruit_ has joined #openstack-lbaas | 22:07 | |
| *** gthiemonge has quit IRC | 22:45 | |
| *** gthiemonge has joined #openstack-lbaas | 22:46 | |
| *** luksky has quit IRC | 23:00 | |
| *** yamamoto has joined #openstack-lbaas | 23:00 | |
| *** yamamoto has quit IRC | 23:05 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!