*** vishalmanchanda has joined #openstack-lbaas | 00:06 | |
*** ianychoi_ has joined #openstack-lbaas | 00:09 | |
*** ianychoi has quit IRC | 00:11 | |
*** tkajinam has quit IRC | 00:56 | |
*** tkajinam has joined #openstack-lbaas | 00:59 | |
*** nicolasbock has quit IRC | 01:18 | |
rm_work | FYI I'm quiet this week because I'm heads down on some internal stuff for a little bit, but please ping me if there's reviews you need done, I will make time for that | 01:28 |
---|---|---|
*** tkajinam has quit IRC | 02:36 | |
*** tkajinam has joined #openstack-lbaas | 03:02 | |
johnsom | Ok, thanks! | 03:33 |
*** psachin has joined #openstack-lbaas | 03:36 | |
*** hongbin has joined #openstack-lbaas | 03:50 | |
*** TrevorV has joined #openstack-lbaas | 03:53 | |
*** TrevorV has quit IRC | 03:54 | |
*** psachin has quit IRC | 04:09 | |
*** sapd1_x has joined #openstack-lbaas | 04:25 | |
openstackgerrit | Vishal Manchanda proposed openstack/octavia-dashboard master: Remove six usage https://review.opendev.org/702336 | 04:31 |
openstackgerrit | Merged openstack/octavia-dashboard master: Drop Django 1.11 support https://review.opendev.org/700932 | 04:35 |
*** hongbin has quit IRC | 05:05 | |
*** AlexStaf has joined #openstack-lbaas | 05:05 | |
*** AlexStaf has quit IRC | 05:10 | |
*** hongbin has joined #openstack-lbaas | 05:14 | |
*** hongbin has quit IRC | 05:19 | |
*** sapd1_x has quit IRC | 05:53 | |
*** gcheresh has joined #openstack-lbaas | 05:53 | |
*** gcheresh has quit IRC | 06:05 | |
*** pcaruana has quit IRC | 06:17 | |
*** gcheresh has joined #openstack-lbaas | 06:32 | |
openstackgerrit | Vishal Manchanda proposed openstack/octavia-dashboard master: Remove six usage https://review.opendev.org/702336 | 06:40 |
openstackgerrit | Merged openstack/octavia stable/rocky: Cap hacking version to <2 https://review.opendev.org/702040 | 07:07 |
openstackgerrit | Merged openstack/octavia stable/stein: Cap hacking version to <2 https://review.opendev.org/702039 | 07:10 |
*** tesseract has joined #openstack-lbaas | 07:41 | |
*** rcernin has quit IRC | 07:52 | |
*** maciejjozefczyk has joined #openstack-lbaas | 08:06 | |
openstackgerrit | Merged openstack/octavia-lib master: Fix flake8 tox.ini directive https://review.opendev.org/693298 | 08:09 |
*** gcheresh has quit IRC | 08:15 | |
*** gcheresh has joined #openstack-lbaas | 08:20 | |
*** tkajinam has quit IRC | 08:24 | |
*** dmellado has quit IRC | 08:55 | |
*** dmellado has joined #openstack-lbaas | 08:56 | |
*** pcaruana has joined #openstack-lbaas | 08:56 | |
*** rpittau|afk is now known as rpittau | 09:16 | |
*** ramishra has quit IRC | 09:31 | |
*** gcheresh has quit IRC | 09:32 | |
*** ramishra has joined #openstack-lbaas | 09:33 | |
*** gcheresh has joined #openstack-lbaas | 09:37 | |
*** ramishra has quit IRC | 10:08 | |
*** ramishra has joined #openstack-lbaas | 10:16 | |
*** gcheresh_ has joined #openstack-lbaas | 10:17 | |
*** gcheresh has quit IRC | 10:17 | |
*** pcaruana has quit IRC | 10:23 | |
*** ajay33_ has joined #openstack-lbaas | 10:35 | |
*** ccamposr has quit IRC | 10:56 | |
*** ramishra has quit IRC | 10:58 | |
*** ccamposr has joined #openstack-lbaas | 10:58 | |
*** pcaruana has joined #openstack-lbaas | 11:02 | |
*** ramishra has joined #openstack-lbaas | 11:05 | |
*** ramishra has quit IRC | 11:10 | |
*** rpittau is now known as rpittau|bbl | 11:11 | |
*** ramishra has joined #openstack-lbaas | 11:11 | |
*** ramishra has quit IRC | 11:16 | |
openstackgerrit | Ann Taraday proposed openstack/octavia master: Add logging filter for AmpConnectionRetry exception https://review.opendev.org/700553 | 11:17 |
*** ramishra has joined #openstack-lbaas | 11:22 | |
*** ramishra has quit IRC | 11:28 | |
*** gcheresh_ has quit IRC | 11:49 | |
*** gcheresh_ has joined #openstack-lbaas | 11:54 | |
*** ramishra has joined #openstack-lbaas | 12:22 | |
*** ramishra has quit IRC | 12:58 | |
*** gcheresh_ has quit IRC | 13:03 | |
*** gcheresh_ has joined #openstack-lbaas | 13:07 | |
*** rpittau|bbl is now known as rpittau | 13:26 | |
*** tkajinam has joined #openstack-lbaas | 13:56 | |
*** AlexStaf has joined #openstack-lbaas | 14:19 | |
*** TrevorV has joined #openstack-lbaas | 14:34 | |
*** gcheresh_ has quit IRC | 14:44 | |
*** gcheresh has joined #openstack-lbaas | 14:45 | |
*** vesper has quit IRC | 14:53 | |
*** vesper11 has joined #openstack-lbaas | 14:53 | |
*** ramishra has joined #openstack-lbaas | 15:02 | |
*** tkajinam has quit IRC | 15:10 | |
*** maciejjozefczyk has quit IRC | 15:12 | |
*** fyx has quit IRC | 15:13 | |
*** dougwig has quit IRC | 15:13 | |
*** fyx has joined #openstack-lbaas | 15:17 | |
*** dougwig has joined #openstack-lbaas | 15:17 | |
*** maciejjozefczyk has joined #openstack-lbaas | 15:19 | |
*** gcheresh has quit IRC | 15:33 | |
*** gcheresh has joined #openstack-lbaas | 15:40 | |
*** fyx has quit IRC | 15:48 | |
*** dougwig has quit IRC | 15:48 | |
*** fyx has joined #openstack-lbaas | 15:51 | |
*** dougwig has joined #openstack-lbaas | 15:52 | |
*** maciejjozefczyk has quit IRC | 16:02 | |
openstackgerrit | Merged openstack/octavia stable/queens: Cap hacking version to <2 https://review.opendev.org/702041 | 16:06 |
*** ramishra_ has joined #openstack-lbaas | 16:07 | |
*** ramishra has quit IRC | 16:09 | |
*** gcheresh has quit IRC | 16:13 | |
*** dosaboy has quit IRC | 16:28 | |
*** AlexStaf has quit IRC | 16:44 | |
*** maciejjozefczyk has joined #openstack-lbaas | 16:55 | |
*** rpittau is now known as rpittau|afk | 17:01 | |
*** ccamposr__ has joined #openstack-lbaas | 17:14 | |
*** ccamposr has quit IRC | 17:14 | |
*** ajay33_ has quit IRC | 17:24 | |
*** dosaboy has joined #openstack-lbaas | 17:59 | |
*** maciejjozefczyk has quit IRC | 18:02 | |
*** maciejjozefczyk has joined #openstack-lbaas | 18:17 | |
*** maciejjozefczyk has quit IRC | 18:56 | |
*** maciejjozefczyk has joined #openstack-lbaas | 18:59 | |
*** pcaruana has quit IRC | 19:16 | |
*** maciejjozefczyk has quit IRC | 19:26 | |
*** armax has joined #openstack-lbaas | 19:30 | |
*** gmann is now known as gmann_afk | 19:36 | |
openstackgerrit | Brian Haley proposed openstack/octavia master: Refactor 'check_quota_met' and 'decrement_quota' https://review.opendev.org/596665 | 19:48 |
*** maciejjozefczyk has joined #openstack-lbaas | 20:13 | |
*** born2bake has joined #openstack-lbaas | 20:18 | |
born2bake | Hi guys | 20:18 |
born2bake | so installed octavia; good thing that at least now my amphora instances are not in spawning state and are active; however, still lb is in pending create state. octavia-worker.log - https://pastebin.com/ieq2YHju | 20:18 |
born2bake | 20.0.0.202 - lb ; 20.0.0.214 and 20.0.0.190 amphora instances | 20:18 |
born2bake | sec groups are added | 20:19 |
johnsom | born2bake How are you running your nova? Is it inside virtualbox? | 20:19 |
johnsom | The controller will keep retrying to connect (thus the pending_create state) until nova finishes booting the instance or it hits the timeout in your configuration file. At which time it will move to "ERROR" state. | 20:21 |
*** AlexStaf has joined #openstack-lbaas | 20:23 | |
*** maciejjozefczyk has quit IRC | 20:25 | |
*** AlexStaf has quit IRC | 20:27 | |
*** maciejjozefczyk has joined #openstack-lbaas | 20:29 | |
*** AlexStaf has joined #openstack-lbaas | 20:29 | |
*** ccamposr__ has quit IRC | 20:40 | |
born2bake | johnsom no, I have cluster of 4 machines. nova is backed by ceph | 20:41 |
johnsom | born2bake Hmm, ok, so it's not likely that nova is taking too long to boot the instance (this is very common). So, let's look at the controller port on the lb-mgmt-net | 20:42 |
johnsom | Where you are running the octavia worker process, it needs a route to or a port on the lb-mgmt-net that is used to communicate with the amphora. How do you have that port/route setup? | 20:43 |
born2bake | I am using kolla-ansible, everything is deployed in kolla containers. I was following this guide: https://ssup2.github.io/record/OpenStack_Stein_%EC%84%A4%EC%B9%98_Kolla-Ansible_Ubuntu_18.04_ODROID-H2_Cluster/ ; However, I am using Train version of openstack | 20:44 |
born2bake | Before that, I had issues that it was taking too long to boot the instance but now its working fine. instances are active | 20:45 |
born2bake | can you specify how do I find a port? Network setup is: I have provider network 10.0.0.0/16 and I can my instances with floating ip. I have octavia-network 20.0.0.0/24. octavia vxlan is attached to the router(with provider network). | 20:46 |
johnsom | Ah, yeah, ok. We here of a bunch of people using Kolla Ansible having problems because it doesn't setup the lb-mgmt-net for you. Most (maybe all) of the other deployment tools do. | 20:48 |
johnsom | Let me look at the kolla docs to see if I can find the section on the lb-mgmt-net | 20:49 |
*** tesseract has quit IRC | 20:50 | |
*** maciejjozefczyk has quit IRC | 20:50 | |
born2bake | with kolla, you deploy octavia containers, then you add flavor,network,sec group, you add them in conf file and redeploy octavia | 20:50 |
johnsom | Yeah, let's see, in that document they are calling the lb-mgmt-net "octavia-net" | 20:51 |
johnsom | Yeah, the issue with kolla ansible is it doesn't setup the lb-mgmt-net for you, the docs say you have to do it by hand. The other deployment tools do that automatically when they setup the containers | 20:51 |
born2bake | which other deployment tools? and what I can do if I am using kolla-ansible :) Also, thanks a lot for help cause I could not get any advice for long time :) | 20:52 |
johnsom | Ansible, tripleo, charms, puppet, .... pretty much all of them. I don't know kolla much, so I have to dig. There are so many deployment tools now the octavia team can't keep up with them really. | 20:55 |
johnsom | Give me a minute, we can work through getting your deployment going. | 20:55 |
born2bake | if you need, I can send my octavia.conf files | 20:56 |
johnsom | No, not at the moment. This is configuration outside of the octavia.conf | 20:57 |
johnsom | Ok, so this document creates the "octavia-net" as a vxlan network in neutron. Then it creates a route from the cloud's external network to this "octavia-net". Do we agree on that part? | 20:59 |
born2bake | route is added just for debug purposes as I understood. so you can ssh into amphora instances and check logs from your local network. In kolla config you need to specify 2 interfaces: | 21:00 |
born2bake | network_interface: "enp2s0f0" - internal, api, etc | 21:01 |
born2bake | neutron_external_interface: "enp2s0f1" - ip interface without ip address, which will be used by provider network so you can assign floating ip's to your instances and access them. | 21:01 |
born2bake | neutron_tenant_network_types: "vxlan,vlan,flat" | 21:01 |
born2bake | so for my external interface, I am using physnet1 for 10.0.0.0/16..in that doc they use 192.168.0. etc | 21:01 |
*** TrevorV has quit IRC | 21:01 | |
johnsom | Hmm, ok. Well, the controller container(s) that run the (worker, health manager, and housekeeping) processes must have a way to access the "octavia-net"\ | 21:02 |
johnsom | The document you linked implies they are creating a route in the container(s) to allow that access | 21:02 |
born2bake | ahh so this process shall be done inside the containers? | 21:03 |
born2bake | octavia containers* | 21:03 |
johnsom | I think that is how the document is having you do it. | 21:03 |
born2bake | route command within container is not allowed | 21:05 |
johnsom | Either way, those processes need to have a way to reach IPs on the "octavia-net". The IPs on the "octavia-net" must also be able to send a packet back to the controller IPs configured in https://docs.openstack.org/octavia/latest/configuration/configref.html#health_manager.controller_ip_port_list | 21:05 |
johnsom | What are the interfaces and IPs inside the worker container? | 21:06 |
johnsom | Did they plug them directly? | 21:06 |
openstackgerrit | Adam Harwell proposed openstack/octavia master: Conf option to use VIP ip as source ip for backend https://review.opendev.org/702535 | 21:06 |
born2bake | interfaces and ip's are the same what I have on my openstack controller | 21:08 |
born2bake | just for reference I was also following this guide: https://shreddedbacon.com/post/openstack-kolla/ | 21:11 |
born2bake | similar | 21:11 |
johnsom | Is there a port there with an IP on the 20.0.0.x subnet? | 21:11 |
born2bake | nope | 21:11 |
born2bake | http://prntscr.com/qnqjvj | 21:11 |
jrosser | imho the first example is made more complicated by using a vxlan network for octavia | 21:15 |
jrosser | becasue that implies the use of a router to de-encapsulate on the controller | 21:15 |
johnsom | Why is that? | 21:15 |
born2bake | second one is used with vlans; i dont have vlans | 21:15 |
jrosser | the vlan example doesnt need that, and is therefore simpler and the octaiva mgmt network remains un-routed | 21:16 |
jrosser | which is a ++ from my pov | 21:16 |
johnsom | Yeah, it just depends on how you get access to that neutron network. If you pop a port off via openvswitch it doesn't matter | 21:16 |
johnsom | Typically the routed designs are easier for folks to setup, but really there are thousands of ways to do this. | 21:17 |
johnsom | ok, so that is a lot of interfaces and networks if it's inside the container. | 21:18 |
jrosser | don't you need a vlan anyway between the controllers because only one of them will be the active router | 21:18 |
johnsom | Also, inside your container, it may require you use the new "ip route" syntax instead of using the older "route" command | 21:18 |
born2bake | my setup is simple...2 network interfaces; 1 is used for openstack api, services etc. 2 - up without ip address. In openstack I created external with 10.0.0.0/16 which is my LAN, internal - 192.168.x which I assign to instances. | 21:18 |
*** AlexStaf has quit IRC | 21:19 | |
born2bake | ip route add -net 20.0.0.0/24 gw 10.0.53.195 | 21:19 |
born2bake | Error: any valid prefix is expected rather than "-net". | 21:19 |
*** ccamposr has joined #openstack-lbaas | 21:20 | |
johnsom | ip route add 20.0.0.0/24 via 10.0.53.195 | 21:21 |
born2bake | ip route add 20.0.0.0/24 via 10.0.53.195 is not permitted and I cannot use sudo inside container | 21:21 |
johnsom | Ha, well, that might make things a bit hard.... | 21:21 |
*** ccamposr has quit IRC | 21:23 | |
johnsom | If you can't execute root level commands inside the container I'm not sure how you can fix/configure this | 21:23 |
johnsom | Sorry I don't have experience with kolla and how those are setup | 21:24 |
openstackgerrit | Adam Harwell proposed openstack/octavia master: Conf option to use VIP ip as source ip for backend https://review.opendev.org/702535 | 21:24 |
johnsom | Maybe asking for help in the kolla channel would get a response. Basically you need your Octavia controller containers to have a way to access the "octavia-net". | 21:24 |
jrosser | in the vxlan example the route is added on (Controller) which i take to mean the host, not the container | 21:25 |
johnsom | Would the container have access to that? | 21:26 |
jrosser | which for that style of containerisation presume the host is doing the NAT/port forwarding/dockerism necessary there | 21:26 |
johnsom | If I setup those containers I wouldn't expose all of the host networks into the container | 21:26 |
*** ccamposr has joined #openstack-lbaas | 21:26 | |
johnsom | Hmm, yeah, could be. | 21:27 |
jrosser | but i !kolla so the advice to go to their channel and ask is good | 21:27 |
johnsom | Yeah, same, never used it | 21:27 |
jrosser | my osa deployment looks very very different to that | 21:28 |
born2bake | they are doing the same :) | 21:28 |
born2bake | ask octavia lol | 21:28 |
johnsom | Sigh | 21:28 |
johnsom | Yeah, osa, tripleo, devstack, all of them are different than this. | 21:29 |
jrosser | born2bake: this all depends what you care about. this ocatvia mgmt network is one of the very rare times in openstack that the control plane and the VM's get "a bit tangled up" | 21:29 |
jrosser | and so your approach might be quite different in a trusted environment vs a public cloud, for example | 21:30 |
jrosser | it's a shame you're getting batted back here as it's really the job of deployment tooling projects to provide you with some pointers/templates of best practice | 21:31 |
*** gcheresh has joined #openstack-lbaas | 21:32 | |
johnsom | Yeah, there are so many deployment tools now the octavia team doesn't always have visibility to how they are setup. Heck, I have even lost track of how OSA is setup. | 21:34 |
jrosser | it makes a vlan provider network | 21:34 |
jrosser | and then a pile of iptables on the controllers on top of that | 21:35 |
johnsom | lol. Yeah, I think last I saw it had a bunch of odd bridges going on, but I think someone was simplifying that | 21:36 |
jrosser | but it's up to the deployer to make sure the vlan id and address range is suitable, and to do "whatever is needed" to make br-lbaas be in all the right places | 21:36 |
johnsom | Right, to us it's just a neutron network... | 21:37 |
born2bake | before you joined a channel they mentioned: | 21:38 |
born2bake | [21:23:37] <noxoid> born2bake, kolla uses net:host for docker containers. it uses the routes/ips of the parent node | 21:38 |
born2bake | [21:24:13] <noxoid> i have octavia configured as a vlan on the host and tell neutron about it. the amphora are attached to that vlan neutron network | 21:38 |
born2bake | [21:24:35] <noxoid> provider:network_type = vlan, provider:physical_network = physnet1, provider:segmentation_id = 1002 | 21:38 |
born2bake | [21:24:54] <osmanlicilegi> born2bake: it's not a kolla issue. I would recommend you to setup an all-in-one environment to learn the basics of octavia. trying to understand it on production may confuse you. | 21:38 |
johnsom | lol, well, it *is* a kolla issue..... | 21:39 |
johnsom | Ok, so that answers the question about how the container connects to the networks. | 21:39 |
johnsom | It's cloning in the host networks. | 21:39 |
jrosser | born2bake: what they kind of miss out is that the controllers will need actual IP on that vlan | 21:40 |
johnsom | So, on the "parent node" or host, whatever it is called. Do you have the route to 20.0.0.x? "ip route" should list the routes | 21:40 |
johnsom | Yeah, I didn't see that in the blog post | 21:41 |
johnsom | Or routes, you can route in/out of the lb-mgmt-net just fine | 21:41 |
born2bake | hm, not on a controller node. actually let me try to add that route to the controller | 21:41 |
jrosser | unless the addition of the static route via the public ip of a neutron router means you don't need that lbaas mgmt net ip on the host itself..... | 21:42 |
*** gcheresh has quit IRC | 21:45 | |
born2bake | okay its different now i guess | 21:46 |
born2bake | https://pastebin.com/eHzLt30K | 21:46 |
born2bake | i will try to restart octavia containers | 21:47 |
johnsom | Ok, so that log implied it could now connect, but the certificate configuration in the octavia.conf is wrong | 21:48 |
born2bake | certs in kolla shall be generated using -b 4.0.1 octavia version | 21:49 |
born2bake | http://prntscr.com/qnqzun | 21:50 |
born2bake | it doesnt support up-to-date version as I understood | 21:50 |
johnsom | Well, that screen shot is bad advice. Plus, this certificate configuration has been required since the first release of Octavia. | 21:51 |
johnsom | Here is a guide to set it up correctly: https://docs.openstack.org/octavia/latest/admin/guides/certificates.html | 21:51 |
born2bake | https://pastebin.com/EPwdidVT | 21:52 |
johnsom | Their instructions might work, but it is not optimal. | 21:52 |
johnsom | First, I would try booting another load balancer and confirming that the connectivity is working. If we get an SSL error, we know it's simply this certificate configuration that is wrong | 21:53 |
born2bake | yeah, cert conf is wrong | 21:54 |
johnsom | Ok. That guide document should help you sort it out. I went into detail in it. Granted this security is a bit complicated, but the guide is step by step | 21:55 |
born2bake | just to confirm...kolla still requires: ca_01.pem cakey.pem client.pem ; if those will not be inside config folder, deployment wont be possible. If I follow up-to-date guide generating certificates, can I just rename them? and if yes, what would be the right way to rename ? | 21:55 |
johnsom | I can't speak to the kolla side and if it needs those or not. | 21:56 |
johnsom | This section of the guide https://docs.openstack.org/octavia/latest/admin/guides/certificates.html#configuring-octavia might help you map their filenames to how the guide creates the certificates | 21:56 |
born2bake | I was confused just by the names and they dont have documentation for octavia therefore, I am not sure what ca_01.pem cakey.pem client.pem are - comparing to server_ca.cert.pem server_ca.key.pem client_ca.cert.pem client.cert-and-key.pem and etc | 21:58 |
johnsom | Yeah, the guide is for a "dual CA" deployment, which is best for production security. | 21:58 |
johnsom | It looks like kolla was trying a single CA config | 21:59 |
born2bake | okay, i will try. Thank you very much for you help | 22:04 |
johnsom | Sure, good luck! Maybe you can open some bugs for kolla to make this easier | 22:04 |
*** rcernin has joined #openstack-lbaas | 22:08 | |
*** ccamposr has quit IRC | 22:10 | |
born2bake | http://prntscr.com/qnr9ja <3 | 22:15 |
johnsom | Nice! | 22:15 |
born2bake | However, I didnt use dual CA. Needs to raise that to kolla | 22:16 |
johnsom | Now, the one thing to look at, is do the members become "operating_status" "ONLINE". If not, it may be that there is a route from the amphora to the controllers missing | 22:16 |
born2bake | how do I check that? | 22:17 |
born2bake | http://prntscr.com/qnraq7 ? | 22:18 |
johnsom | When you create the load balancer, and add a pool and members, with a health monitor, then in the UI it will have that status column | 22:18 |
johnsom | That is listener, click the pool tab | 22:18 |
born2bake | I have not added any instances to the pool | 22:19 |
born2bake | let me try | 22:19 |
born2bake | if I add instances to the pool, amphora images are spawning for ages | 22:27 |
johnsom | spawning? What do you mean by that? We don't boot anything when adding a member to a pool | 22:27 |
born2bake | when I create load balancer, it automatically creates 2 amphora instances which are spawning. | 22:28 |
johnsom | Yes, in active/standby topology configuration it creates two. It should take less than a minute from the create command for the LB to be ACTIVE | 22:29 |
johnsom | Typically about 30 seconds | 22:29 |
born2bake | http://prntscr.com/qnrg41 I guess its ok | 22:32 |
johnsom | That looks ok, if you click into the pool (which has no name), you should see the members and that they are only | 22:33 |
johnsom | online | 22:33 |
born2bake | yup, they are online. its awesome! | 22:34 |
born2bake | thanks a lot johnsom | 22:34 |
johnsom | +1 ok. Good stuff. Enjoy! | 22:34 |
born2bake | However, I noticed if I assosiate floating ip to lb, I cant ping it. | 22:46 |
born2bake | ahh got it...sorry. i can curl it :) | 22:47 |
*** gmann_afk is now known as gmann | 22:53 | |
*** tkajinam has joined #openstack-lbaas | 22:55 | |
*** armax has quit IRC | 23:30 | |
*** born2bake has quit IRC | 23:55 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!