| *** cshen has joined #openstack-ansible | 01:26 | |
| *** cshen has quit IRC | 01:31 | |
| *** ianychoi_ has quit IRC | 01:48 | |
| *** ianychoi_ has joined #openstack-ansible | 01:50 | |
| *** d34dh0r53 has quit IRC | 02:32 | |
| *** d34dh0r53 has joined #openstack-ansible | 02:35 | |
| *** spatel has joined #openstack-ansible | 02:49 | |
| *** mmethot has quit IRC | 03:13 | |
| *** spatel has quit IRC | 03:39 | |
| *** fridtjof[m] has quit IRC | 04:08 | |
| *** nicolasbock has quit IRC | 04:08 | |
| *** sep has quit IRC | 04:08 | |
| *** sshnaidm|afk has quit IRC | 04:08 | |
| *** KeithMnemonic has quit IRC | 04:08 | |
| *** noonedeadpunk has quit IRC | 04:08 | |
| *** zigo has quit IRC | 04:08 | |
| *** yolanda has quit IRC | 04:08 | |
| *** gouthamr_ has quit IRC | 04:08 | |
| *** persia has quit IRC | 04:08 | |
| *** krah has quit IRC | 04:08 | |
| *** skelly has quit IRC | 04:08 | |
| *** brad[] has quit IRC | 04:08 | |
| *** nwonknu has quit IRC | 04:08 | |
| *** tacco has quit IRC | 04:08 | |
| *** tbarron has quit IRC | 04:08 | |
| *** ioni has quit IRC | 04:08 | |
| *** bjoernt has quit IRC | 04:08 | |
| *** ChanServ has quit IRC | 04:08 | |
| *** alvinstarr has quit IRC | 04:08 | |
| *** schwicht has quit IRC | 04:08 | |
| *** maharg101 has quit IRC | 04:08 | |
| *** admin0 has quit IRC | 04:08 | |
| *** dasp_ has quit IRC | 04:08 | |
| *** pcaruana has quit IRC | 04:08 | |
| *** evrardjp has quit IRC | 04:08 | |
| *** gixx has quit IRC | 04:08 | |
| *** mnaser has quit IRC | 04:08 | |
| *** cyberpear has quit IRC | 04:08 | |
| *** antonym has quit IRC | 04:08 | |
| *** jhesketh has quit IRC | 04:08 | |
| *** janno has quit IRC | 04:08 | |
| *** Adri2000 has quit IRC | 04:08 | |
| *** jroll has quit IRC | 04:08 | |
| *** ChosSimbaOne has quit IRC | 04:08 | |
| *** d34dh0r53 has quit IRC | 04:08 | |
| *** ianychoi_ has quit IRC | 04:08 | |
| *** wpp has quit IRC | 04:08 | |
| *** jmccrory has quit IRC | 04:08 | |
| *** ebbex has quit IRC | 04:08 | |
| *** Brace has quit IRC | 04:08 | |
| *** mcarden has quit IRC | 04:08 | |
| *** poopcat has quit IRC | 04:08 | |
| *** akahat has quit IRC | 04:08 | |
| *** melwitt has quit IRC | 04:08 | |
| *** Jeffrey4l has quit IRC | 04:08 | |
| *** rpittau has quit IRC | 04:08 | |
| *** nurdie_ has quit IRC | 04:08 | |
| *** cp- has quit IRC | 04:08 | |
| *** nsmeds has quit IRC | 04:08 | |
| *** arxcruz has quit IRC | 04:08 | |
| *** gokhani has quit IRC | 04:08 | |
| *** brtknr has quit IRC | 04:08 | |
| *** mrda has quit IRC | 04:08 | |
| *** redrobot has quit IRC | 04:08 | |
| *** mloza has quit IRC | 04:08 | |
| *** trident has quit IRC | 04:08 | |
| *** crazzy has quit IRC | 04:08 | |
| *** Miouge has quit IRC | 04:08 | |
| *** bverschueren has quit IRC | 04:08 | |
| *** bradm has quit IRC | 04:08 | |
| *** tobberydberg_ has quit IRC | 04:08 | |
| *** openstackgerrit has quit IRC | 04:08 | |
| *** Nick_A has quit IRC | 04:08 | |
| *** vesper11 has quit IRC | 04:08 | |
| *** NewJorg has quit IRC | 04:08 | |
| *** mgagne has quit IRC | 04:08 | |
| *** fyx has quit IRC | 04:08 | |
| *** samueldmq has quit IRC | 04:08 | |
| *** fresta has quit IRC | 04:08 | |
| *** dave-mccowan has quit IRC | 04:08 | |
| *** sri_ has quit IRC | 04:08 | |
| *** bl0m1 has quit IRC | 04:08 | |
| *** NobodyCam has quit IRC | 04:08 | |
| *** Open10K8S has quit IRC | 04:08 | |
| *** jrosser has quit IRC | 04:08 | |
| *** guilhermesp has quit IRC | 04:08 | |
| *** mugsie has quit IRC | 04:08 | |
| *** redkrieg has quit IRC | 04:08 | |
| *** stingrayza has quit IRC | 04:08 | |
| *** dmsimard has quit IRC | 04:08 | |
| *** prometheanfire has quit IRC | 04:08 | |
| *** johanssone has quit IRC | 04:08 | |
| *** djhankb has quit IRC | 04:08 | |
| *** chandankumar has quit IRC | 04:08 | |
| *** mgariepy has quit IRC | 04:08 | |
| *** timburke has quit IRC | 04:08 | |
| *** spotz has quit IRC | 04:08 | |
| *** partlycloudy has quit IRC | 04:08 | |
| *** mubix has quit IRC | 04:08 | |
| *** mwhahaha has quit IRC | 04:08 | |
| *** alanmeadows has quit IRC | 04:08 | |
| *** irclogbot_0 has quit IRC | 04:08 | |
| *** tinwood has quit IRC | 04:08 | |
| *** logan- has quit IRC | 04:08 | |
| *** gary_perkins has quit IRC | 04:08 | |
| *** masterpe has quit IRC | 04:08 | |
| *** CeeMac has quit IRC | 04:08 | |
| *** mmercer has quit IRC | 04:08 | |
| *** johnsom has quit IRC | 04:08 | |
| *** gouthamr has quit IRC | 04:08 | |
| *** gundalow has quit IRC | 04:08 | |
| *** donnyd has quit IRC | 04:08 | |
| *** idlemind has quit IRC | 04:08 | |
| *** waxfire has quit IRC | 04:08 | |
| *** soren has quit IRC | 04:08 | |
| *** sum12 has quit IRC | 04:08 | |
| *** admin0 has joined #openstack-ansible | 04:14 | |
| *** maharg101 has joined #openstack-ansible | 04:14 | |
| *** alvinstarr has joined #openstack-ansible | 04:14 | |
| *** ChosSimbaOne has joined #openstack-ansible | 04:14 | |
| *** jroll has joined #openstack-ansible | 04:14 | |
| *** Adri2000 has joined #openstack-ansible | 04:14 | |
| *** janno has joined #openstack-ansible | 04:14 | |
| *** jhesketh has joined #openstack-ansible | 04:14 | |
| *** antonym has joined #openstack-ansible | 04:14 | |
| *** cyberpear has joined #openstack-ansible | 04:14 | |
| *** mnaser has joined #openstack-ansible | 04:14 | |
| *** gixx has joined #openstack-ansible | 04:14 | |
| *** evrardjp has joined #openstack-ansible | 04:14 | |
| *** pcaruana has joined #openstack-ansible | 04:14 | |
| *** dasp_ has joined #openstack-ansible | 04:14 | |
| *** tbarron has joined #openstack-ansible | 04:14 | |
| *** nwonknu has joined #openstack-ansible | 04:14 | |
| *** brad[] has joined #openstack-ansible | 04:14 | |
| *** krah has joined #openstack-ansible | 04:14 | |
| *** skelly has joined #openstack-ansible | 04:14 | |
| *** tacco has joined #openstack-ansible | 04:14 | |
| *** persia has joined #openstack-ansible | 04:14 | |
| *** gouthamr_ has joined #openstack-ansible | 04:14 | |
| *** yolanda has joined #openstack-ansible | 04:14 | |
| *** sep has joined #openstack-ansible | 04:14 | |
| *** nicolasbock has joined #openstack-ansible | 04:14 | |
| *** fridtjof[m] has joined #openstack-ansible | 04:14 | |
| *** Nick_A has joined #openstack-ansible | 04:14 | |
| *** openstackgerrit has joined #openstack-ansible | 04:14 | |
| *** tobberydberg_ has joined #openstack-ansible | 04:14 | |
| *** bradm has joined #openstack-ansible | 04:14 | |
| *** bverschueren has joined #openstack-ansible | 04:14 | |
| *** Miouge has joined #openstack-ansible | 04:14 | |
| *** crazzy has joined #openstack-ansible | 04:14 | |
| *** trident has joined #openstack-ansible | 04:14 | |
| *** redkrieg has joined #openstack-ansible | 04:14 | |
| *** mugsie has joined #openstack-ansible | 04:14 | |
| *** guilhermesp has joined #openstack-ansible | 04:14 | |
| *** jrosser has joined #openstack-ansible | 04:14 | |
| *** Open10K8S has joined #openstack-ansible | 04:14 | |
| *** NobodyCam has joined #openstack-ansible | 04:14 | |
| *** sri_ has joined #openstack-ansible | 04:14 | |
| *** dave-mccowan has joined #openstack-ansible | 04:14 | |
| *** mgagne has joined #openstack-ansible | 04:14 | |
| *** NewJorg has joined #openstack-ansible | 04:14 | |
| *** vesper11 has joined #openstack-ansible | 04:14 | |
| *** poopcat has joined #openstack-ansible | 04:14 | |
| *** ebbex has joined #openstack-ansible | 04:14 | |
| *** mcarden has joined #openstack-ansible | 04:14 | |
| *** Brace has joined #openstack-ansible | 04:14 | |
| *** jmccrory has joined #openstack-ansible | 04:14 | |
| *** wpp has joined #openstack-ansible | 04:14 | |
| *** ianychoi_ has joined #openstack-ansible | 04:14 | |
| *** d34dh0r53 has joined #openstack-ansible | 04:14 | |
| *** Jeffrey4l has joined #openstack-ansible | 04:14 | |
| *** melwitt has joined #openstack-ansible | 04:14 | |
| *** akahat has joined #openstack-ansible | 04:14 | |
| *** arxcruz has joined #openstack-ansible | 04:14 | |
| *** cp- has joined #openstack-ansible | 04:14 | |
| *** nurdie_ has joined #openstack-ansible | 04:14 | |
| *** nsmeds has joined #openstack-ansible | 04:14 | |
| *** rpittau has joined #openstack-ansible | 04:14 | |
| *** djhankb has joined #openstack-ansible | 04:14 | |
| *** johanssone has joined #openstack-ansible | 04:14 | |
| *** prometheanfire has joined #openstack-ansible | 04:14 | |
| *** dmsimard has joined #openstack-ansible | 04:14 | |
| *** stingrayza has joined #openstack-ansible | 04:14 | |
| *** ioni has joined #openstack-ansible | 04:14 | |
| *** mubix has joined #openstack-ansible | 04:14 | |
| *** mwhahaha has joined #openstack-ansible | 04:14 | |
| *** chandankumar has joined #openstack-ansible | 04:14 | |
| *** fyx has joined #openstack-ansible | 04:14 | |
| *** CeeMac has joined #openstack-ansible | 04:14 | |
| *** mmercer has joined #openstack-ansible | 04:14 | |
| *** johnsom has joined #openstack-ansible | 04:14 | |
| *** gouthamr has joined #openstack-ansible | 04:14 | |
| *** alanmeadows has joined #openstack-ansible | 04:14 | |
| *** samueldmq has joined #openstack-ansible | 04:14 | |
| *** gundalow has joined #openstack-ansible | 04:14 | |
| *** soren has joined #openstack-ansible | 04:14 | |
| *** donnyd has joined #openstack-ansible | 04:14 | |
| *** mgariepy has joined #openstack-ansible | 04:14 | |
| *** sum12 has joined #openstack-ansible | 04:14 | |
| *** idlemind has joined #openstack-ansible | 04:14 | |
| *** bjoernt has joined #openstack-ansible | 04:14 | |
| *** ChanServ has joined #openstack-ansible | 04:14 | |
| *** tepper.freenode.net sets mode: +o ChanServ | 04:14 | |
| *** fresta has joined #openstack-ansible | 04:14 | |
| *** partlycloudy has joined #openstack-ansible | 04:14 | |
| *** spotz has joined #openstack-ansible | 04:14 | |
| *** timburke has joined #openstack-ansible | 04:14 | |
| *** irclogbot_0 has joined #openstack-ansible | 04:14 | |
| *** tinwood has joined #openstack-ansible | 04:14 | |
| *** waxfire has joined #openstack-ansible | 04:14 | |
| *** logan- has joined #openstack-ansible | 04:14 | |
| *** gary_perkins has joined #openstack-ansible | 04:14 | |
| *** mloza has joined #openstack-ansible | 04:15 | |
| *** gokhani has joined #openstack-ansible | 04:15 | |
| *** brtknr has joined #openstack-ansible | 04:15 | |
| *** redrobot has joined #openstack-ansible | 04:15 | |
| *** mrda has joined #openstack-ansible | 04:15 | |
| *** sshnaidm|afk has joined #openstack-ansible | 04:15 | |
| *** KeithMnemonic has joined #openstack-ansible | 04:15 | |
| *** noonedeadpunk has joined #openstack-ansible | 04:15 | |
| *** zigo has joined #openstack-ansible | 04:15 | |
| *** fridtjof[m] has quit IRC | 04:15 | |
| *** markvoelker has joined #openstack-ansible | 04:15 | |
| *** dave-mccowan has quit IRC | 04:15 | |
| *** nicolasbock has quit IRC | 04:16 | |
| *** schwicht has joined #openstack-ansible | 04:16 | |
| *** guilhermesp has quit IRC | 04:16 | |
| *** mnaser has quit IRC | 04:16 | |
| *** fyx has quit IRC | 04:16 | |
| *** ioni has quit IRC | 04:17 | |
| *** cyberpear has quit IRC | 04:17 | |
| *** guilhermesp has joined #openstack-ansible | 04:18 | |
| *** fyx has joined #openstack-ansible | 04:18 | |
| *** markvoelker has quit IRC | 04:20 | |
| *** bl0m1 has joined #openstack-ansible | 04:20 | |
| *** nicolasbock has joined #openstack-ansible | 04:21 | |
| *** masterpe has joined #openstack-ansible | 04:23 | |
| *** cyberpear has joined #openstack-ansible | 04:24 | |
| *** evrardjp has quit IRC | 04:33 | |
| *** evrardjp has joined #openstack-ansible | 04:33 | |
| *** markvoelker has joined #openstack-ansible | 04:50 | |
| *** fridtjof[m] has joined #openstack-ansible | 04:51 | |
| *** ioni has joined #openstack-ansible | 04:51 | |
| *** markvoelker has quit IRC | 04:55 | |
| *** udesale has joined #openstack-ansible | 05:09 | |
| *** markvoelker has joined #openstack-ansible | 05:10 | |
| *** markvoelker has quit IRC | 05:14 | |
| *** miloa has joined #openstack-ansible | 05:20 | |
| *** miloa has quit IRC | 05:21 | |
| *** nurdie_ has quit IRC | 05:21 | |
| *** udesale_ has joined #openstack-ansible | 05:36 | |
| *** nurdie has joined #openstack-ansible | 05:37 | |
| *** mindthecap has joined #openstack-ansible | 05:38 | |
| *** udesale has quit IRC | 05:38 | |
| *** nurdie has quit IRC | 05:42 | |
| *** udesale_ has quit IRC | 05:44 | |
| *** udesale has joined #openstack-ansible | 05:55 | |
| CeeMac | morning | 06:13 |
|---|---|---|
| *** ianychoi_ has quit IRC | 06:22 | |
| *** ianychoi_ has joined #openstack-ansible | 06:23 | |
| *** this10nly has joined #openstack-ansible | 06:27 | |
| *** also_stingrayza has joined #openstack-ansible | 06:37 | |
| *** stingrayza has quit IRC | 06:39 | |
| *** also_stingrayza is now known as stingrayza | 06:47 | |
| *** cshen has joined #openstack-ansible | 07:19 | |
| *** arkan has joined #openstack-ansible | 07:21 | |
| *** tosky has joined #openstack-ansible | 07:30 | |
| *** jbadiapa has joined #openstack-ansible | 08:34 | |
| admin0 | morning .. is there a way to change a router status to master | 08:50 |
| admin0 | i have all 3 routers set to standby | 08:50 |
| openstackgerrit | Merged openstack/openstack-ansible-openstack_hosts master: Replace yum config manager with dnf alternative https://review.opendev.org/739554 | 09:05 |
| *** markvoelker has joined #openstack-ansible | 09:12 | |
| *** markvoelker has quit IRC | 09:17 | |
| openstackgerrit | Merged openstack/openstack-ansible master: Fix KeyError raised when max hostname length exceeded https://review.opendev.org/740343 | 09:58 |
| *** spatel has joined #openstack-ansible | 10:41 | |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible stable/ussuri: Fix KeyError raised when max hostname length exceeded https://review.opendev.org/740441 | 10:45 |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible stable/train: Fix KeyError raised when max hostname length exceeded https://review.opendev.org/740442 | 10:45 |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible stable/stein: Fix KeyError raised when max hostname length exceeded https://review.opendev.org/740443 | 10:45 |
| *** spatel has quit IRC | 10:46 | |
| *** markvoelker has joined #openstack-ansible | 11:13 | |
| *** markvoelker has quit IRC | 11:23 | |
| *** cshen has quit IRC | 11:23 | |
| *** alvinstarr has quit IRC | 11:23 | |
| *** maharg101 has quit IRC | 11:23 | |
| *** admin0 has quit IRC | 11:23 | |
| *** cshen has joined #openstack-ansible | 11:27 | |
| *** alvinstarr has joined #openstack-ansible | 11:27 | |
| *** maharg101 has joined #openstack-ansible | 11:27 | |
| *** admin0 has joined #openstack-ansible | 11:27 | |
| *** namrata has joined #openstack-ansible | 11:55 | |
| admin0 | https://review.opendev.org/#/c/740078/ -- does this mean its merged and can use it ? | 12:01 |
| *** markvoelker has joined #openstack-ansible | 12:06 | |
| *** markvoelker has quit IRC | 12:10 | |
| *** rh-jelabarre has joined #openstack-ansible | 12:12 | |
| namrata | admin0 Yes its ready for use | 12:13 |
| *** rh-jelabarre has quit IRC | 12:19 | |
| *** rh-jelabarre has joined #openstack-ansible | 12:19 | |
| admin0 | \o/ .. and hopefully it will allow me to upgrade further in future | 12:21 |
| *** markvoelker has joined #openstack-ansible | 12:22 | |
| mgariepy | admin0, for the routers status, you might need to check keepalived in the netns of the router. | 12:31 |
| mgariepy | out of curiosity how many routers do you have ? | 12:31 |
| mgariepy | i've seen some issue in the past (back in kilo or liberty) with the states of HA router not updating fast enough. | 12:32 |
| admin0 | i fixed it in the morning by removing the ha status, and then removing the router from all l3-agents and adding it back again | 12:34 |
| admin0 | this one router has around 70+ floating ips | 12:34 |
| mgariepy | how many router for the install ? | 12:40 |
| mgariepy | i'm not using ha router since it was causing more issue for me than no-ha router.. | 12:41 |
| *** spatel has joined #openstack-ansible | 12:42 | |
| *** spatel has quit IRC | 12:47 | |
| *** mindthecap has quit IRC | 12:48 | |
| arkan | hi guys, sorry to bother you, please can someone give me a working config for Octavia, I will pay $50 for it. | 12:50 |
| arkan | I lost 1 week for this thing | 12:50 |
| arkan | I can not play around the configs, and I'm under time pressure | 12:51 |
| arkan | I know that you have job also, but just a slice of your time, consider it as an external service | 12:51 |
| *** nurdie has joined #openstack-ansible | 12:51 | |
| arkan | everything is working except LB | 12:52 |
| *** arkan has quit IRC | 12:53 | |
| *** arkan has joined #openstack-ansible | 12:54 | |
| namrata | noonedeadpunk if you are around, i want you to thank you for the other day with dnsmasq issue, it was my heat template which was setting the environment for 5 VMs which was setting the log-facility. thanks again for your help. | 12:58 |
| *** mmethot has joined #openstack-ansible | 13:04 | |
| *** mnaser has joined #openstack-ansible | 13:08 | |
| *** jamesdenton has joined #openstack-ansible | 13:11 | |
| *** dave-mccowan has joined #openstack-ansible | 13:14 | |
| noonedeadpunk | namrata: sure, no problem at all:) btw I never thought that heat template will adjust bare metal nodes config... | 13:21 |
| noonedeadpunk | oh, or you was creating testing env with heat?:) | 13:21 |
| namrata | I was seting /etc/openstack_deploy/user_variables.yml with my heat template only | 13:21 |
| *** alanmeadows_ has joined #openstack-ansible | 13:24 | |
| *** gary_perkins_ has joined #openstack-ansible | 13:26 | |
| *** alanmeadows has quit IRC | 13:27 | |
| *** irclogbot_0 has quit IRC | 13:27 | |
| *** alanmeadows_ is now known as alanmeadows | 13:27 | |
| *** gary_perkins has quit IRC | 13:27 | |
| *** tinwood has quit IRC | 13:28 | |
| *** irclogbot_2 has joined #openstack-ansible | 13:28 | |
| *** mwhahaha has quit IRC | 13:30 | |
| *** logan- has quit IRC | 13:30 | |
| *** tinwood has joined #openstack-ansible | 13:30 | |
| *** mwhahaha has joined #openstack-ansible | 13:32 | |
| *** logan- has joined #openstack-ansible | 13:32 | |
| *** udesale_ has joined #openstack-ansible | 13:36 | |
| *** udesale has quit IRC | 13:39 | |
| arkan | guys I will triple $1000 for a working Octavia config. Kidding :)) | 13:43 |
| arkan | no really, I will play $100 for a working octavia config, but not more | 13:43 |
| arkan | this is the maximum amount that I can pay | 13:44 |
| arkan | just make this work | 13:44 |
| arkan | and make me live in peace | 13:44 |
| CeeMac | arkan: it might be worth taking a step back, taking a fresh breath and looking at the problem with a fresh pair of eyes as I think you have yourself tangled in a knot | 13:45 |
| CeeMac | I don't use octavia myself, so can't give you a working config | 13:45 |
| *** gokhani has quit IRC | 13:46 | |
| CeeMac | but I can maby talk through the issue with you | 13:46 |
| arkan | I lost a lot of time one week on Octavia | 13:46 |
| arkan | it drove me crazy | 13:46 |
| CeeMac | yes, ita can happen | 13:46 |
| arkan | from the morning until the evening | 13:46 |
| CeeMac | its easy to get bogged down in the problem so you can't see your way out | 13:46 |
| CeeMac | from what I see, and the information and links jrosser provided you're most of the way there. | 13:46 |
| arkan | pleace save my time | 13:46 |
| arkan | yes, but I could not reach the final | 13:47 |
| CeeMac | I think where you're struggling is with separating neutron networking from host and container networking | 13:47 |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/ansible-role-uwsgi master: Use X-Forwarded-For header in logs https://review.opendev.org/740469 | 13:47 |
| arkan | yes | 13:47 |
| arkan | CeeMac let's go private in order to not clutter the chat | 13:47 |
| *** cshen has quit IRC | 13:48 | |
| CeeMac | no need to worry about clutter, if it is a valid process then it could help other people too :) | 13:48 |
| CeeMac | let me dig out the link to that diagram again | 13:49 |
| CeeMac | so i get my head straight | 13:49 |
| arkan | cool | 13:49 |
| arkan | just let me know what is the right config that should I use | 13:50 |
| arkan | and I will paste it into my config files | 13:50 |
| CeeMac | i think, from what i saw of your chat, you have your OSA config pretty much sorted | 13:50 |
| CeeMac | the problem you had was getting the networking into the container correctly | 13:51 |
| arkan | everything is working, except octavia | 13:51 |
| arkan | and today I tried to another config | 13:51 |
| CeeMac | so, the way I see it | 13:51 |
| arkan | one moment | 13:51 |
| CeeMac | on the RHS you have a neutron 'provider' network, tagged with a vlan ID | 13:52 |
| arkan | this is my new configs - current one | 13:53 |
| arkan | http://paste.openstack.org/show/795755/ --> user_variables.yml | 13:53 |
| *** arkan has quit IRC | 13:54 | |
| *** arkan has joined #openstack-ansible | 13:54 | |
| CeeMac | you were using vlan 510 rather than 111 no? | 13:54 |
| arkan | openstack_user_config.yml ---> http://paste.openstack.org/show/795756/ | 13:54 |
| arkan | yes, today I tried to do what is in the diagram | 13:55 |
| CeeMac | ok | 13:55 |
| arkan | controller network config ----> http://paste.openstack.org/show/795757/ | 13:56 |
| CeeMac | ok | 13:56 |
| arkan | compute network config ---> http://paste.openstack.org/show/795758/ | 13:56 |
| arkan | brctl show (compute) ---> http://paste.openstack.org/show/795759/ | 13:57 |
| arkan | brctl show (controller) ----> http://paste.openstack.org/show/795760/ | 13:58 |
| arkan | and | 13:59 |
| CeeMac | you have br-vlan as flat? | 13:59 |
| arkan | root@compute1:~# tcpdump -i brqbc8fc1f4-e1 | 13:59 |
| arkan | tcpdump: verbose output suppressed, use -v or -vv for full protocol decode | 13:59 |
| arkan | listening on brqbc8fc1f4-e1, link-type EN10MB (Ethernet), capture size 262144 bytes | 13:59 |
| arkan | 13:58:58.010010 ARP, Request who-has 172.29.235.220 tell 172.29.232.145, length 28 | 13:59 |
| arkan | from the container there (on controller) is reaching br-vlan on compute | 14:00 |
| arkan | but it does not know how to return | 14:00 |
| arkan | container ----> br-lbaas ----> br-vlan | 14:00 |
| CeeMac | your vlan ids dont match up currently | 14:01 |
| arkan | humm | 14:01 |
| CeeMac | eno1.510 on the controller | 14:01 |
| CeeMac | br-vlan.111 on the compute | 14:01 |
| CeeMac | but I think you need to have your br-vlan as 'vlan' type in openstack_user_config as you have flat | 14:01 |
| arkan | a question, is there a way to not use br-vlan at all ? | 14:01 |
| arkan | or this is the only solution ? | 14:02 |
| CeeMac | let me check something real quick | 14:03 |
| arkan | great | 14:03 |
| CeeMac | you're using vxlan for tenant networks right? | 14:03 |
| CeeMac | br-vlan you've put in just for octavia traffic? Or other provider networks too? | 14:03 |
| arkan | yes | 14:03 |
| arkan | I didn't want to use br-vlan, but as the diagram shows, I was trying in order to make octavia work | 14:04 |
| arkan | if there is a solution without the need for br-vlan, it would be great | 14:04 |
| arkan | my provider is using net_name: "external" | 14:05 |
| arkan | which is of type "vlan", I used it for floating ips, and it works | 14:05 |
| CeeMac | sorry, brb just need to take a call | 14:06 |
| arkan | sure | 14:06 |
| janno | Does anyone use designate deployed by openstack-ansible? how does your network look? how do you connect your designate containers to the outer world? | 14:06 |
| jamesdenton | arkan you can avoid the use of br-vlan as long as a) your neutron agents are not in a container on the controller and b) you use host_bind_override | 14:07 |
| arkan | aha | 14:08 |
| arkan | jamesdenton: nice hint | 14:09 |
| jamesdenton | if your br-vlan would contain em2, then you could set host_bind_override: em2 and destroy br-vlan | 14:09 |
| openstackgerrit | Merged openstack/openstack-ansible-lxc_hosts stable/ussuri: Add centos-8 support https://review.opendev.org/740230 | 14:10 |
| arkan | what is em2 ? do you mean eno1 or eno2 in my case ? | 14:11 |
| jamesdenton | yeah, sorry, eno2 | 14:11 |
| jamesdenton | from looking at your output | 14:11 |
| arkan | but I have neutron is a container, it means that I need br-vlan as you mention in a), right ? | 14:12 |
| arkan | root@controller1:~# lxc-ls | grep neu | 14:13 |
| arkan | controller1_memcached_container-bc6c7729 controller1_neutron_server_container-05f59c60 controller1_neutron_server_container-8c8adbed | 14:13 |
| jamesdenton | neutron-server != neutron-agent | 14:13 |
| arkan | aha | 14:13 |
| jamesdenton | the agents are likely configured on the controller itself: systemctl status neutron-linuxbridge-aget | 14:13 |
| CeeMac | ah, cavalry to the rescue! hi jamesdenton | 14:13 |
| jamesdenton | systemctl status neutron-linuxbridge-agent | 14:13 |
| arkan | root@controller1:~# systemctl list-unit-files | grep neut | 14:13 |
| arkan | neutron-dhcp-agent.service enabled | 14:13 |
| arkan | neutron-l3-agent.service enabled | 14:13 |
| arkan | neutron-linuxbridge-agent.service enabled | 14:13 |
| jamesdenton | naw i'm gonna sink back into the shadows | 14:13 |
| arkan | neutron-metadata-agent.service enabled | 14:13 |
| arkan | neutron-metering-agent.service enabled | 14:13 |
| arkan | hahaha | 14:14 |
| jamesdenton | yep, cool. so you should be good to eliminate the br-vlan bridge itself and use host_bind_override for controllers and computes | 14:14 |
| arkan | all the warriors are here | 14:14 |
| arkan | ok CeeMac: what was your idea about getting rid of br-vlan | 14:15 |
| arkan | I have only 2 nodes, controller and compute nodes | 14:15 |
| arkan | eno1 (controller) ------> router | 14:15 |
| arkan | eno1 (compute) -----> router | 14:15 |
| arkan | eno2 (controller) <-----> eno2 (compute) | 14:16 |
| *** pcaruana has quit IRC | 14:17 | |
| arkan | I also tried to combine under one network_provider: to use vlan for both octavia and for providing floating ips | 14:17 |
| arkan | by using rang: "510:520" for vlans | 14:18 |
| arkan | then creating network for external networks using vlan 511 | 14:18 |
| arkan | and the 510 to be used by octavia | 14:18 |
| arkan | but I got the same problem, floating ips worked | 14:19 |
| arkan | but octavia not | 14:19 |
| arkan | arping is reaching the bridge but can not return | 14:19 |
| jamesdenton | unless i'm mistaken, the br-lbaas bridge is there for two reasons: 1) to provide access to the lbaas mgmt network from the actual octavia service container and 2) give neutron a place to connect amphora to said lbaas mgmt network. An amphora will be multi-homes, and connect to both br-lbaas and other neutron provider networks off eno2 (br-vlan) to reach the actual pool members. but i need to re-educate myself on this | 14:21 |
| jamesdenton | *multi-homed | 14:21 |
| arkan | I can see that amphora is running | 14:22 |
| arkan | ok, but how to solve this dilemma? is there a working config for my case ? | 14:23 |
| arkan | what should I change into my config to work ? | 14:24 |
| jamesdenton | your amphora, what interfaces does it have? | 14:24 |
| arkan | one moment | 14:24 |
| arkan | lbaas-mgmt | 14:26 |
| arkan | 172.29.232.104 | 14:26 |
| arkan | I can see from horizon | 14:26 |
| arkan | VNIC Type | 14:26 |
| arkan | Normal | 14:26 |
| arkan | Host | 14:26 |
| arkan | compute1 | 14:26 |
| arkan | Profile | 14:26 |
| arkan | None | 14:26 |
| arkan | VIF Type | 14:26 |
| arkan | bridge | 14:27 |
| arkan | VIF Details | 14:27 |
| jamesdenton | kk | 14:27 |
| arkan | connectivity l2 | 14:27 |
| arkan | port_filter True | 14:27 |
| jamesdenton | can you ping the 172.29.232.104 IP from the dhcp namespace? | 14:27 |
| arkan | let me see | 14:27 |
| arkan | I have only one netns | 14:29 |
| arkan | root@controller1:~# ip netns | 14:29 |
| arkan | qdhcp-bc8fc1f4-e1cf-4c40-a831-73871561fd40 (id: 38) | 14:29 |
| arkan | root@controller1:~# ip netns exec qdhcp-bc8fc1f4-e1cf-4c40-a831-73871561fd40 ping 172.29.232.104 | 14:29 |
| arkan | PING 172.29.232.104 (172.29.232.104) 56(84) bytes of data. | 14:29 |
| jamesdenton | also need to see your openstack network list and openstack subnet list output, please | 14:29 |
| arkan | k | 14:30 |
| johnsom | Amphora don't allow ping by default. | 14:30 |
| arkan | aha yes | 14:30 |
| *** pcaruana has joined #openstack-ansible | 14:30 | |
| arkan | I can add icmp | 14:30 |
| arkan | to the instance | 14:30 |
| arkan | and retry it | 14:30 |
| jamesdenton | i'm not sure thats the right namespace yet, but good to know, johnsom. i guess arp could be validated either way | 14:30 |
| *** spatel has joined #openstack-ansible | 14:33 | |
| johnsom | I am late to the party here, but if the load balancer became ACTIVE, your lb-mgmt-net is working. | 14:33 |
| arkan | it worked | 14:33 |
| arkan | root@controller1:~# ip netns exec qdhcp-bc8fc1f4-e1cf-4c40-a831-73871561fd40 ping 172.29.232.104 | 14:33 |
| arkan | PING 172.29.232.104 (172.29.232.104) 56(84) bytes of data. | 14:33 |
| arkan | 64 bytes from 172.29.232.104: icmp_seq=1 ttl=64 time=0.705 ms | 14:33 |
| arkan | I added security group icmp | 14:33 |
| arkan | and it pinged it | 14:33 |
| arkan | from the netns | 14:33 |
| jamesdenton | cool. and you've gone thru the process of adding pool members, listeners, etc? | 14:34 |
| CeeMac | jamesdenton trumps anything I would say :D | 14:34 |
| jamesdenton | naw | 14:34 |
| arkan | I did not add anything right now | 14:34 |
| arkan | no LB is added | 14:34 |
| arkan | but I can see some errors in journalctl -xf in octavia container | 14:35 |
| jamesdenton | ok. so i guess my question is, what led you to believe it wasn't working? Just curious, so we can update docs/faqs/etc | 14:35 |
| *** d34dh0r53 has quit IRC | 14:35 | |
| arkan | one moment | 14:35 |
| arkan | http://paste.openstack.org/show/795763/ | 14:36 |
| arkan | because every time I can see these errors | 14:36 |
| jamesdenton | yep, ok | 14:36 |
| arkan | also | 14:36 |
| arkan | I can create LB but it remain not functional "offline" | 14:37 |
| jamesdenton | so, this leads me back to ceemac's question: in br-lbaas on the controller we see vlan 510, but on the compute you're using vlan 111? the actual lbaas mgmt network would need to exist as a (flat) neutron provider network connected to br-lbaas | 14:37 |
| jamesdenton | you have that subnet list/network list? | 14:38 |
| *** d34dh0r53 has joined #openstack-ansible | 14:38 | |
| arkan | I can create it from cli + horizon (thanks for jrosser for helping me to use new sha256 for inistalling horizon) | 14:38 |
| CeeMac | arkan: I was going to say I'm not using br-vlan in my environment but use neutron_provider_networks in user_variables to configure my network mappings to physical interfaces | 14:38 |
| admin0 | how do I run setup-hosts, but skip ansible-hardening on re-runs ? | 14:39 |
| arkan | humm | 14:39 |
| CeeMac | admin0: apply_security_hardening: false in user_variables | 14:39 |
| jamesdenton | admin0: -e apply_security_hardening=false? | 14:39 |
| CeeMac | or that :D | 14:39 |
| arkan | CeeMac: I did not use this, and I don't know about it | 14:39 |
| admin0 | got it :) thanks | 14:40 |
| arkan | I want to get rid of br-vlan | 14:40 |
| arkan | if it's possible | 14:40 |
| *** pcaruana has quit IRC | 14:40 | |
| arkan | how to do it? show me the code :))) | 14:40 |
| jamesdenton | arkan i would forget that for now. i think you're close with the existing config, you can get it working then go from there | 14:40 |
| jamesdenton | change too many variables and you'll never get it going | 14:40 |
| arkan | We use this in our community "show me the code" :)) | 14:40 |
| CeeMac | is eno2 dedicated just for octavia traffic? | 14:41 |
| jamesdenton | "teach a man to fish" | 14:41 |
| jamesdenton | :D | 14:41 |
| arkan | CeeMac, maybe, I was not sure | 14:41 |
| CeeMac | i mean in you setup | 14:41 |
| arkan | No not sure of it | 14:41 |
| CeeMac | is your intenation that this would be a dedicated interface | 14:42 |
| CeeMac | oh | 14:42 |
| arkan | yes | 14:42 |
| arkan | eno2 | 14:42 |
| CeeMac | and that will only have the lbaas-mgmt traffic? | 14:42 |
| arkan | yes, I was thinking about it | 14:42 |
| arkan | but I was not sure if it will work | 14:42 |
| arkan | so I provided a direct cable from eno2 (controller node) to eno2 (compute node) | 14:43 |
| arkan | but I was not sure how the things will settle | 14:43 |
| arkan | if we can use eno2 for lbaas managment, it's ok, but how to do it? | 14:45 |
| CeeMac | right now, i think we need to correct the br-vlan.111 or the eno1.510 | 14:45 |
| jamesdenton | arkan: two problems in your user_variables -- octavia_provider_network_name: vlan needs to be octavia_provider_network_name: lbaas, which is the provider definition you made in openstack_user_config for lbaas mgmt. And second, octavia_provider_network_type: vlan should be changed to octavia_provider_network_type: flat if you're going to use a br-lbaas w/ eno1.510. You also have octavia_management_net_subnet_cidr | 14:45 |
| jamesdenton | defined twice, with the second one effective (172.29.232.0/22). octavia_provider_segmentation_id can be commented out. | 14:45 |
| arkan | if I will try "flat" I will receive an error | 14:46 |
| jamesdenton | I think it should also be noted that whatever changes you make to the configuration to get it going may not translate well to a larger environment due to the cross-connections you mentioned. | 14:47 |
| arkan | let me check | 14:47 |
| arkan | jamesdenton: I tried one time running using "flat" and I received this error while it was installing | 14:48 |
| arkan | "physical_network 'lbaas' unknown for flat provider network." | 14:48 |
| arkan | I used it in user_variables: | 14:49 |
| arkan | octavia_provider_network_type: flat | 14:49 |
| jamesdenton | yeah, the provider definition is raw and not flat, so it's not defined in neutron agent confs | 14:49 |
| jamesdenton | i need to look at something brb | 14:49 |
| arkan | sure | 14:49 |
| arkan | CeeMac: we can correct these, but what are the needed changes that need to be done | 14:50 |
| arkan | ? | 14:50 |
| arkan | so I can retry the installation and see | 14:50 |
| CeeMac | i think you need to switch your provider_networks config from raw to flat also | 14:51 |
| CeeMac | for br-lbaas | 14:51 |
| spatel | noonedeadpunk: i am seeing my patch still failing on some environment https://review.opendev.org/#/c/739658/ | 14:51 |
| jamesdenton | I'm not sure yet if that would break connectivity from octavia container to the bridge, though | 14:51 |
| spatel | noonedeadpunk: is there anyway i can get into box and look for issue | 14:52 |
| arkan | CeeMac: br-lbaas has vlan tag in the physical network | 14:52 |
| arkan | and if I use a config in the provider_network: to by of type "flat" | 14:52 |
| arkan | then I will receive an error, says: | 14:53 |
| jamesdenton | which is exactly why it would need to be 'flat' from a neutron perspective. That way, neutron won't tag on top of the existing tag | 14:53 |
| arkan | "can not ensalve br-lbaas...etc" | 14:53 |
| jamesdenton | ahh yes. | 14:53 |
| CeeMac | on eno1.510 yes | 14:54 |
| arkan | I think it would work only if it's attached directly to eno1 as the case in br-ext | 14:54 |
| CeeMac | hmm | 14:54 |
| arkan | but I can not attach eno1 two times to different bridges | 14:54 |
| CeeMac | indeed | 14:54 |
| CeeMac | and you can't have 2 tagged subinterfaces with the same id | 14:55 |
| jamesdenton | you might try 'host_bind_override: eno1.510', then | 14:55 |
| CeeMac | for br-lbaas? | 14:55 |
| arkan | jamesdenton: but I don't have this !! | 14:55 |
| arkan | should I create a dummy interface ? | 14:56 |
| CeeMac | you don't have what arkan ? | 14:56 |
| jamesdenton | you output implies there is a eno1.510 | 14:56 |
| arkan | eno1.510 | 14:56 |
| CeeMac | its in the netplan config | 14:56 |
| arkan | humm | 14:56 |
| CeeMac | and the brctl output | 14:56 |
| arkan | yes | 14:56 |
| arkan | but that has vlan tag | 14:57 |
| arkan | can I use it in the host_bing_ovveride ? | 14:57 |
| arkan | *host_bind_override | 14:57 |
| jamesdenton | you can, as long as it's removed from the br-lbaas bridge on the compute | 14:57 |
| jamesdenton | and the network type is changed to flat | 14:58 |
| arkan | ok | 14:58 |
| arkan | let's take it step by step | 14:58 |
| jamesdenton | i'm looking for a working config, bear with me | 14:58 |
| arkan | let's begin with physical networks files | 14:58 |
| arkan | is there any changes that need to be done there ? | 14:59 |
| arkan | in the netplan? | 14:59 |
| arkan | if the current setup ok, then: | 14:59 |
| arkan | what changes should I need to make in openstack_user_config.yml ? | 15:00 |
| arkan | in br-lbaas net ----> add "host_bind_override": "eno1.510", right ? | 15:01 |
| arkan | should I leave it with type:"raw"? | 15:01 |
| jamesdenton | change to flat | 15:02 |
| noonedeadpunk | spatel: I think for gnocchi you need to rename vars file just to redhat | 15:02 |
| arkan | are you sure that this will not throw "can not enslave br-lbaas" ? | 15:03 |
| jamesdenton | as long as you set host_bind_override: eno1.510 you should be OK | 15:03 |
| spatel | noonedeadpunk: oh!! i think you are right.. let me see | 15:03 |
| arkan | great | 15:03 |
| spatel | hold on | 15:03 |
| CeeMac | and remove it from the bridge on the host? | 15:03 |
| jamesdenton | you just need to remove that from the br-lbaas bridge on the compute. neutron will plug it into a brq bridge | 15:03 |
| arkan | CeeMac: from netplan ? | 15:04 |
| CeeMac | yes | 15:04 |
| arkan | ok | 15:04 |
| arkan | I will remove it from both compute and controller | 15:05 |
| arkan | just a question, br-lbaas was attached to the interface in the vlans eno1.510 | 15:06 |
| arkan | I will delete only br-lbaas and I will leave the interface eno1.510 | 15:06 |
| arkan | on both the nodes | 15:06 |
| CeeMac | br-lbaas: interfaces: [] | 15:07 |
| arkan | aha | 15:07 |
| openstackgerrit | Satish Patel proposed openstack/openstack-ansible-os_gnocchi master: Add centos-8 support https://review.opendev.org/739658 | 15:07 |
| arkan | I thought to delete it | 15:07 |
| CeeMac | no, remove it from the bridge | 15:08 |
| CeeMac | you still need the vlan interface there for the host_bind_override | 15:08 |
| arkan | ok, I will put it again but with interfaces: [] | 15:08 |
| arkan | or even no interfaces | 15:08 |
| arkan | I mean I will not declare "interfaces" | 15:08 |
| admin0 | i am guessing after all this is working, arkan will put up a blog or gist of config for us to copy to make it work :) | 15:09 |
| CeeMac | right, if that is valid with netplan, i don't netplan if I can avoid it :) | 15:09 |
| arkan | :)) | 15:09 |
| *** nurdie_ has joined #openstack-ansible | 15:10 | |
| CeeMac | check with brctl that its definitely gone | 15:10 |
| CeeMac | iirc there are issue with modifying existing virtual interfaces with netplan/networkd | 15:11 |
| arkan | http://paste.openstack.org/show/795764/ | 15:13 |
| *** nurdie has quit IRC | 15:13 | |
| CeeMac | its gone :) | 15:13 |
| arkan | you wanted to be gone | 15:14 |
| arkan | and I make it gone | 15:14 |
| arkan | is this ok ? | 15:14 |
| CeeMac | haha | 15:14 |
| CeeMac | yes | 15:14 |
| arkan | but on controller node | 15:14 |
| CeeMac | so you can make the changes as above to configuration file | 15:14 |
| arkan | it has somthing there | 15:14 |
| arkan | interface created by neutron | 15:15 |
| arkan | I did not delete it | 15:15 |
| arkan | and also | 15:15 |
| arkan | it is not attached to eno1.510 | 15:15 |
| CeeMac | thats ol | 15:15 |
| CeeMac | *ok | 15:15 |
| CeeMac | it was just the eno1.510 that was an issue as you're moving to host-bind-override | 15:15 |
| arkan | ok | 15:16 |
| CeeMac | if you make the configuration changes like jamesdenton said above next | 15:16 |
| spatel | noonedeadpunk: any idea what could be wrong with this builds, multiple failed https://review.opendev.org/#/c/739653/ | 15:16 |
| *** cshen has joined #openstack-ansible | 15:16 | |
| arkan | is this good ? http://paste.openstack.org/show/795765/ | 15:16 |
| CeeMac | seems ok | 15:17 |
| CeeMac | one way to find out :D | 15:17 |
| arkan | now let move to user_variables.yml | 15:17 |
| openstackgerrit | Merged openstack/openstack-ansible-openstack_hosts stable/ussuri: Add advanced-virtualization CentOS 8 repo https://review.opendev.org/740228 | 15:17 |
| noonedeadpunk | spatel: I guess here the role might be broken somewhere... | 15:17 |
| arkan | what needs to be changed here http://paste.openstack.org/show/795766/ ? | 15:18 |
| *** chandankumar is now known as raukadah | 15:18 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_glance stable/ussuri: Add Centos-8 support https://review.opendev.org/740232 | 15:23 |
| arkan | jamesdenton: CeeMac: what should I change here before running the installation setup? | 15:24 |
| arkan | http://paste.openstack.org/show/795766/ | 15:25 |
| CeeMac | sorry was afk | 15:26 |
| arkan | np | 15:26 |
| CeeMac | which octavia_management_net_subnet_cidr do you want to keep | 15:27 |
| CeeMac | you have it twice with different subnets | 15:27 |
| arkan | I have in openstack_user_config these | 15:27 |
| arkan | cidr_networks: &cidr_networks | 15:27 |
| arkan | container: 172.29.236.0/22 | 15:27 |
| arkan | tunnel: 172.29.240.0/22 | 15:27 |
| arkan | storage: 172.29.244.0/22 | 15:27 |
| arkan | lbaas: 172.29.232.0/22 | 15:27 |
| CeeMac | ok so you remove / comment out the first one under # Network type using 10.0.x.x | 15:28 |
| CeeMac | *you can | 15:28 |
| CeeMac | also jamesdenton mentioned to comment out octavia_provider_segmentation_id | 15:28 |
| CeeMac | and change to octavia_provider_network_type: flat | 15:29 |
| arkan | so lines: 6 & 10 will be commented out | 15:29 |
| CeeMac | correct | 15:30 |
| arkan | and leave line 5 ? | 15:30 |
| CeeMac | line 5 changes from 'vlan' to 'lbaas' | 15:30 |
| arkan | ok | 15:30 |
| CeeMac | line 9 changes from 'vlan' to 'flat' | 15:30 |
| arkan | ok | 15:31 |
| CeeMac | all based on previous conversation above | 15:31 |
| arkan | ok | 15:32 |
| arkan | we did not touch the br-vlan stuff | 15:32 |
| CeeMac | for the moment that is probably sensible | 15:33 |
| CeeMac | lets see how this change works out | 15:33 |
| arkan | ok | 15:33 |
| arkan | I will run os-neutron-install.yml | 15:33 |
| arkan | then os-octavia-install.yml | 15:33 |
| CeeMac | ok | 15:35 |
| *** namrata has quit IRC | 15:37 | |
| arkan | os-neutron-install.yml has finished with no errors | 15:48 |
| arkan | now I will run os-octavia-install.yml | 15:48 |
| CeeMac | ok | 15:48 |
| *** gyee has joined #openstack-ansible | 15:48 | |
| *** arkan has quit IRC | 15:53 | |
| *** arkan_ has joined #openstack-ansible | 15:54 | |
| *** arkan_ is now known as arkan | 15:54 | |
| openstackgerrit | Merged openstack/openstack-ansible-ceph_client stable/ussuri: Add centos-8 support https://review.opendev.org/740231 | 15:55 |
| arkan | CeeMac: os-octavia-install.yml has finished with no errors :)) | 15:56 |
| arkan | now let's see | 15:56 |
| CeeMac | lets take a look at brctl and see what has been plumbed in then | 15:57 |
| arkan | ok | 15:57 |
| arkan | brctl show (compute) --> http://paste.openstack.org/show/795767/ | 15:59 |
| arkan | brctl show (controller) ----> http://paste.openstack.org/show/795769/ | 15:59 |
| arkan | there is an instance amphora running since 14 min. | 16:00 |
| arkan | and in octavia container journal log: http://paste.openstack.org/show/795770/ | 16:01 |
| CeeMac | is tap8f2aa93f-ee the amphora interface? | 16:01 |
| arkan | one moment | 16:02 |
| arkan | it dissappeared | 16:02 |
| arkan | I think it will create another instance | 16:02 |
| CeeMac | so the controller looks good, you have eno1.510 and container eth14 in br-lbaas | 16:02 |
| arkan | it created a new amphora instance | 16:03 |
| arkan | but from octavia container it can not reach it | 16:03 |
| arkan | ... r: HTTPSConnectionPool(host='172.29.232.186', port=9443): M ... | 16:04 |
| CeeMac | i'm still not so sure about that br-vlan.111 that is dropped in the neutron bridge on compute node | 16:04 |
| arkan | I did not touch it, and now it's not used as I think | 16:04 |
| arkan | it's from the previous installation | 16:05 |
| CeeMac | hmm, there needs to be some way to get traffic out of the neutron bridge to the 510 vlan so that it can get to the compute node and br-lbaas | 16:05 |
| CeeMac | can you check the interface id of the amphora against the tap interface in the bridge | 16:05 |
| CeeMac | brqbc8fc1f4-e1 | 16:06 |
| arkan | that is from the previous installation that was used with br-vlan | 16:06 |
| CeeMac | you left br-vlan there though | 16:06 |
| CeeMac | and I guess your neutron network is still set the same that the amphora is attaching to? | 16:07 |
| arkan | yes, we said to not touch it for now | 16:07 |
| CeeMac | yes | 16:07 |
| arkan | do you want me to destroy every br-vlan on this planet ? | 16:07 |
| CeeMac | can you get an 'openstack network show' output for the neutron network | 16:07 |
| CeeMac | lets not be hasty lol | 16:07 |
| arkan | ok | 16:08 |
| arkan | you mean openstack network list | 16:08 |
| CeeMac | no | 16:09 |
| CeeMac | that would list all of the networks | 16:09 |
| CeeMac | i just want to see the specific network information for the provide network under neutron :) | 16:09 |
| arkan | show for lbaas-mgmt | 16:09 |
| CeeMac | yes | 16:10 |
| arkan | openstack network show lbaas-mgmt | 16:10 |
| CeeMac | and for the subnet for that network also | 16:10 |
| arkan | http://paste.openstack.org/show/795771/ | 16:11 |
| arkan | humm | 16:11 |
| arkan | it kept the segmentation id | 16:11 |
| arkan | I think I needed to destroy this network before running neutron | 16:11 |
| CeeMac | yes | 16:12 |
| CeeMac | it is also still vlan type | 16:12 |
| arkan | after I destroy it | 16:12 |
| arkan | I will rerun neutron-install & octavia-install again | 16:12 |
| CeeMac | sounds good | 16:12 |
| arkan | ok | 16:12 |
| CeeMac | then lets do brctl again to see how things look | 16:13 |
| arkan | ok | 16:13 |
| arkan | ok now I destroyed it | 16:14 |
| arkan | now I will run the roles | 16:14 |
| CeeMac | ok | 16:15 |
| *** udesale_ has quit IRC | 16:19 | |
| arkan | os-neutron-install.yml has finished without errors | 16:27 |
| CeeMac | great | 16:27 |
| CeeMac | can you check did the provide network get created? | 16:27 |
| arkan | on moment | 16:27 |
| CeeMac | and can we get a quick brctl | 16:27 |
| arkan | I can see this | 16:28 |
| arkan | br-lbaas8000.a22eebb1b4cbnob12e701e_eth14 | 16:28 |
| arkan | on controller node | 16:28 |
| CeeMac | and on compute? | 16:28 |
| CeeMac | can you paste full output | 16:29 |
| arkan | this | 16:29 |
| arkan | br-lbaas8000.a22eebb1b4cbnob12e701e_eth14 | 16:29 |
| arkan | eno1.510 | 16:29 |
| arkan | ok, let me past all | 16:29 |
| CeeMac | thanks | 16:29 |
| arkan | brctl show on controller ---> http://paste.openstack.org/show/795772/ | 16:30 |
| arkan | brctl show on compute1 ---> http://paste.openstack.org/show/795773/ | 16:30 |
| CeeMac | hmm | 16:31 |
| CeeMac | no eno1.510 in br-lbaas on compute | 16:31 |
| arkan | yes | 16:31 |
| CeeMac | was that there before, can't remember now | 16:31 |
| CeeMac | anyway, run the ocatvia play see what happens | 16:31 |
| CeeMac | the neutron bridge is gone, so should get recreated | 16:32 |
| arkan | yes it was there because it was attached in br-lbaas | 16:32 |
| arkan | in the physical network | 16:32 |
| arkan | as I think | 16:32 |
| arkan | as I remember well | 16:32 |
| CeeMac | yes, then we removed it due to host-bind-override | 16:32 |
| arkan | yes | 16:32 |
| CeeMac | gah, i've closed the pastes now | 16:33 |
| CeeMac | one sec | 16:33 |
| *** cshen has quit IRC | 16:34 | |
| arkan | you can give me the green light to run os-octavia-install.yml | 16:34 |
| CeeMac | go ahead | 16:35 |
| arkan | ok | 16:35 |
| CeeMac | then get a brctl output from compute once its run | 16:35 |
| arkan | ok | 16:35 |
| arkan | ok | 16:42 |
| arkan | it has finished without errors | 16:42 |
| CeeMac | great | 16:43 |
| arkan | brctl show on compute ---> http://paste.openstack.org/show/795774/ | 16:43 |
| CeeMac | ok | 16:44 |
| CeeMac | so we have eno1.510 and the tap interface in the neutron bridge | 16:44 |
| CeeMac | so that should line up | 16:44 |
| CeeMac | how is octavia looking? | 16:44 |
| arkan | brctl show on controller ---> http://paste.openstack.org/show/795775/ | 16:44 |
| arkan | ok, amphora has 3 minutes since it was started | 16:46 |
| CeeMac | hmm | 16:46 |
| arkan | but again "No route to host" in octavia container | 16:46 |
| CeeMac | yes | 16:46 |
| CeeMac | so, on both compute and controller, eno1.510 is being bound to the neutron bridge | 16:47 |
| arkan | yes | 16:47 |
| CeeMac | and br-lbaas on the controller only has the container eth14 in it | 16:47 |
| arkan | yes | 16:48 |
| arkan | root@compute1:~# tcpdump -i tap47f8c8d9-ac | 16:49 |
| arkan | tcpdump: verbose output suppressed, use -v or -vv for full protocol decode | 16:49 |
| arkan | listening on tap47f8c8d9-ac, link-type EN10MB (Ethernet), capture size 262144 bytes | 16:49 |
| arkan | 16:49:08.074000 ARP, Request who-has 172.29.235.220 tell 172.29.232.130, length 28 | 16:49 |
| arkan | 16:49:09.075653 ARP, Request who-has 172.29.235.220 tell 172.29.232.130, length 28 | 16:49 |
| arkan | the above ip 172.29.235.220 is in the eth14 inside octavia container | 16:50 |
| CeeMac | ok | 16:50 |
| arkan | 154: eth14@if155: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 | 16:50 |
| arkan | link/ether 00:16:3e:50:f9:b5 brd ff:ff:ff:ff:ff:ff link-netnsid 0 | 16:50 |
| arkan | inet 172.29.235.220/22 brd 172.29.235.255 scope global eth14 | 16:50 |
| CeeMac | the problem is, there is nothing linking the provider network to the physical br-lbaas bridge | 16:51 |
| CeeMac | so the octavia container is not connected to the same broadcast domain | 16:51 |
| CeeMac | now, after neutron play was run and before octavia play was run, on the controller, eno1.510 was attached to br-lbaas along with the container interface b12e701e_eth14 | 16:52 |
| CeeMac | now since running the octavia play, it is not | 16:52 |
| arkan | isn't it ? br-lbaas8000.a22eebb1b4cbnob12e701e_eth14 | 16:53 |
| arkan | brq53d4b297-478000.4e4db24c32b9noeno1.510 | 16:54 |
| arkan | tape6612229-1c | 16:54 |
| arkan | eno1.510 it's attached to brq53d4b297-47 | 16:54 |
| arkan | right ? | 16:54 |
| CeeMac | yes | 16:56 |
| CeeMac | which is the neutron bridge | 16:56 |
| arkan | yes | 16:56 |
| CeeMac | what is that tape6612229-1c interface attached to on the controller? | 16:57 |
| arkan | this is that I was looking at | 16:57 |
| arkan | I was thinking why is there | 16:57 |
| arkan | I found it | 16:59 |
| arkan | it is attached to this brq53d4b297-47 | 16:59 |
| CeeMac | yes | 16:59 |
| CeeMac | at one end, what is on the other end | 16:59 |
| arkan | :)) | 16:59 |
| arkan | how can I know? | 16:59 |
| arkan | is there a way to find out ? | 17:00 |
| CeeMac | maybe show arp on the bridge its attached to to see what IP it is | 17:00 |
| CeeMac | then see if you can track that back? | 17:00 |
| arkan | ok | 17:00 |
| CeeMac | or maybe openstack port list | grep 6612229-1c | 17:00 |
| CeeMac | to get the ID then port show to see what it is attached to (instance id) | 17:01 |
| arkan | openstack port list | grep tape6612229-1c returns nothing | 17:01 |
| *** cshen has joined #openstack-ansible | 17:01 | |
| arkan | maybe it's from previous installations | 17:02 |
| arkan | i used arp -a | 17:05 |
| CeeMac | dont use tap in grep | 17:05 |
| CeeMac | e6612229-1c should be the start of a port id | 17:05 |
| arkan | aha | 17:06 |
| arkan | ok | 17:06 |
| arkan | | e6612229-1cd6-457a-aaa7-d000af57789d | | fa:16:3e:0c:ef:4c | ip_address='172.29.232.30', subnet_id='6eb6d67c-5eaa-4dfa-aacb-67a74638dd31' | ACTIVE | | 17:06 |
| *** cshen has quit IRC | 17:07 | |
| arkan | CeeMac: here in openstack_user_config.yml octavia is only installed on infra hosts | 17:13 |
| arkan | octavia-infra_hosts: *infrastructure_hosts | 17:13 |
| arkan | so the containers will be only on infra | 17:13 |
| arkan | is there a possibility to run it on compute also? maybe neutron will make something similar to what it did in the controller node | 17:14 |
| arkan | but this was just a thought | 17:14 |
| arkan | maybe that's why here https://github.com/rcbops/rpc-octavia/blob/master/INSTALLATION.md they used br-vlan | 17:16 |
| arkan | in order to reach the compute node | 17:16 |
| arkan | and only br-lbaas exists on the controller node | 17:17 |
| CeeMac | sorry, lost my internet for a while there | 17:21 |
| arkan | it happens sometimes to me | 17:21 |
| CeeMac | did you find where that neutron port was plugged? | 17:22 |
| arkan | I don't if you can see what I've wrote erlier | 17:22 |
| CeeMac | yes | 17:22 |
| arkan | I will re-paste it | 17:22 |
| arkan | | e6612229-1cd6-457a-aaa7-d000af57789d | | fa:16:3e:0c:ef:4c | ip_address='172.29.232.30', subnet_id='6eb6d67c-5eaa-4dfa-aacb-67a74638dd31' | ACTIVE | | 17:22 |
| CeeMac | can you paste the full output of openstack port show | 17:23 |
| arkan | earlier messages | 17:23 |
| arkan | http://paste.openstack.org/show/795778/ | 17:23 |
| arkan | ok, I will past it | 17:24 |
| arkan | http://paste.openstack.org/show/795779/ | 17:25 |
| arkan | it's the dhcp | 17:26 |
| CeeMac | ok | 17:28 |
| CeeMac | can't quite figure out why that would have eno1.510 bound to it | 17:28 |
| *** aedc has joined #openstack-ansible | 17:28 | |
| arkan | CeeMac: do you need me to wipe out all (nuclear effect), and redeploy from zero with the current config ? | 17:29 |
| arkan | :)) | 17:30 |
| CeeMac | seems a little drastic :) | 17:30 |
| arkan | I think one month I run the deployment more than CI/CD | 17:30 |
| CeeMac | haha | 17:31 |
| CeeMac | so, br-vlan isn't doing much now by the look of it | 17:32 |
| admin0 | i have around 30k messages in ready state in nova.versioned_notifications.info | 17:33 |
| admin0 | any ideas what that is for ? | 17:33 |
| arkan | CeeMac: no I don't think is doing something | 17:34 |
| arkan | CeeMac: for my curiosity, how many NICs are you using in your installation ? | 17:35 |
| CeeMac | 3 bonded pairs on compute and network nodes | 17:36 |
| arkan | oh | 17:36 |
| arkan | you have a lot of NICs | 17:36 |
| CeeMac | yes i wanted to ensure seperation of traffic | 17:36 |
| arkan | in this diagram https://github.com/rcbops/rpc-octavia/blob/master/INSTALLATION.md they used br-vlan | 17:38 |
| CeeMac | yes | 17:38 |
| arkan | and I think that br-lbaas is installed in the controller node | 17:38 |
| CeeMac | and have veth bond between br-vlan and br-lbaas | 17:39 |
| arkan | yes | 17:39 |
| arkan | and br-vlan is on both the controller and compute | 17:39 |
| CeeMac | yes | 17:39 |
| CeeMac | with vlan tagged interface | 17:39 |
| CeeMac | i'm trying to work backwards how we've ended up where we are | 17:39 |
| CeeMac | so you've defined the "lbaas" provider-network on "br-lbaas" with host-bind-override of eno1.510 | 17:40 |
| arkan | yes | 17:40 |
| CeeMac | then you've specificed this as the octavia_provider_network_name | 17:41 |
| arkan | yes | 17:41 |
| CeeMac | which should wire the neutron network 'lbaas-mgmt' to the br-lbaas | 17:42 |
| arkan | yes | 17:42 |
| CeeMac | i'm wondering if its because that bridge is flat its pulling the bound interace out instead and putting it in the neutron bridge | 17:42 |
| CeeMac | can i get the paste of your openstack_user_config again please? | 17:43 |
| arkan | sure | 17:43 |
| arkan | http://paste.openstack.org/show/795780/ | 17:44 |
| CeeMac | there is only 1 controller and 1 compute? | 17:46 |
| arkan | yes | 17:46 |
| arkan | it's my home lab 2 rack dell servers | 17:46 |
| CeeMac | right | 17:46 |
| CeeMac | so | 17:46 |
| CeeMac | you have br-vlan defined with range 101:200,301:400 | 17:47 |
| CeeMac | is that something you've put in yourself or taken from the example? | 17:47 |
| arkan | I've taken from the example | 17:47 |
| CeeMac | ok | 17:48 |
| arkan | in order to use vlan 111, which was in the diagram | 17:48 |
| CeeMac | and do you have neutron-linuxbridge-agent running on the controller? | 17:48 |
| arkan | yes | 17:48 |
| arkan | root@controller1:~# systemctl list-unit-files | grep neut | 17:49 |
| arkan | neutron-dhcp-agent.service enabled | 17:49 |
| arkan | neutron-l3-agent.service enabled | 17:49 |
| arkan | neutron-linuxbridge-agent.service enabled | 17:49 |
| arkan | neutron-metadata-agent.service enabled | 17:49 |
| arkan | neutron-metering-agent.service enabled | 17:49 |
| arkan | also | 17:50 |
| CeeMac | ok, which explains why its getting the neutron bridge and teh br-lbaas | 17:50 |
| arkan | root@compute1:~# systemctl list-unit-files | grep neut | 17:50 |
| arkan | neutron-linuxbridge-agent.service enabled | 17:50 |
| arkan | but br-lbaas is defined on both physical net | 17:51 |
| arkan | on compute and on controller | 17:51 |
| arkan | let me paste my physical networks | 17:51 |
| CeeMac | i got them still from before | 17:51 |
| arkan | ok | 17:52 |
| CeeMac | ok, so if i'm working this out correctly, you could get rid of the second br-vlan in lines 109 to 117 | 17:53 |
| CeeMac | change 105 and 106 to vlan | 17:53 |
| *** arkan has quit IRC | 17:53 | |
| CeeMac | and add range: "501" | 17:53 |
| CeeMac | sorry range: "510" | 17:53 |
| *** arkan has joined #openstack-ansible | 17:54 | |
| arkan | I was disconnected | 17:54 |
| arkan | CeeMac: you said to get rid of lines 109-117 | 17:55 |
| arkan | right ? | 17:55 |
| CeeMac | yes | 17:55 |
| arkan | ok | 17:55 |
| arkan | should I destroy them now ? | 17:55 |
| arkan | :)) | 17:55 |
| CeeMac | change 105 and 106 to vlan | 17:55 |
| CeeMac | and add range: "510" | 17:55 |
| CeeMac | or comment them out | 17:56 |
| CeeMac | maybe comment them out | 17:56 |
| CeeMac | or take a copy of the file first in case you want/ need to revert | 17:56 |
| arkan | range: "510" will not work | 17:56 |
| arkan | "510:510" this will work | 17:56 |
| CeeMac | yes, sorry | 17:56 |
| arkan | ok | 17:57 |
| arkan | ... processing ... destroying ... | 17:57 |
| CeeMac | in user_variables, change those 'flat' back to 'vlan', uncomment segmentation_id and set it to 510 | 17:57 |
| CeeMac | sorry un-uncomment segmentation_id | 17:58 |
| arkan | I understood | 17:58 |
| CeeMac | no, i was right the first time. Either way, get rid of the # :D | 17:58 |
| CeeMac | then you'll probably need to create the veth link between br-vlan and br-lbaas as per the document / diagram | 17:59 |
| *** cshen has joined #openstack-ansible | 17:59 | |
| arkan | we will see | 17:59 |
| CeeMac | in theory that should add eno2.510 to the neutron network | 18:00 |
| CeeMac | eno2 should be in br-vlan | 18:00 |
| arkan | yes | 18:00 |
| CeeMac | and with the link between br-vlan and br-lbaas it "should work"? | 18:00 |
| arkan | and they don't use the router | 18:00 |
| arkan | they are directly connected | 18:00 |
| arkan | direct cable from controller to compute on eno2 | 18:01 |
| CeeMac | yes, direct connection but in a prod env you could have eno2 backing off to a switch stack | 18:01 |
| arkan | yeah, this one is not for prod | 18:01 |
| CeeMac | did you test giving eno2 an IP on each host and pinging each other by the way? | 18:01 |
| CeeMac | just to validate the connection is working | 18:02 |
| arkan | yes | 18:02 |
| arkan | it worked | 18:02 |
| CeeMac | great | 18:02 |
| CeeMac | dont forget to remove your neutron network before rerunning your plays | 18:02 |
| CeeMac | i need to leave "work" (my dining room) shortly, but i'll keep an eye on the channel from my phone | 18:03 |
| arkan | after the modification | 18:03 |
| CeeMac | yes | 18:03 |
| arkan | openstack_user_config.yml ---> http://paste.openstack.org/show/795781/ | 18:03 |
| admin0 | CeeMac, what app to use irc on phone ? | 18:03 |
| *** cshen has quit IRC | 18:03 | |
| arkan | user_variables.yml ---> http://paste.openstack.org/show/795782/ | 18:03 |
| CeeMac | i use irccloud | 18:04 |
| CeeMac | but other irc clients are available :) | 18:04 |
| openstackgerrit | Satish Patel proposed openstack/openstack-ansible-os_gnocchi master: Add centos-8 support https://review.opendev.org/740513 | 18:04 |
| CeeMac | octavia_provider_network_name: vlan | 18:05 |
| arkan | ok | 18:05 |
| arkan | but I think it needs _address | 18:06 |
| CeeMac | how do you mean? | 18:06 |
| arkan | vlan_address | 18:06 |
| CeeMac | you'e mixing up container network name? | 18:07 |
| arkan | as it's here https://docs.openstack.org/openstack-ansible-os_octavia/pike/ | 18:07 |
| CeeMac | provider network name is referencing the correct provider network, in this instance 'vlan' that we just ammended | 18:07 |
| arkan | octavia_container_network_name | 18:08 |
| noonedeadpunk | admin0: btw rc2 should be available now | 18:08 |
| noonedeadpunk | https://opendev.org/openstack/openstack-ansible/src/tag/21.0.0.0rc2 | 18:08 |
| CeeMac | i'm talking about octavia_provider_network_name: though arkan | 18:08 |
| arkan | ah | 18:08 |
| CeeMac | it is currently lbaas | 18:08 |
| CeeMac | need to be vlan | 18:08 |
| arkan | ok | 18:09 |
| arkan | it's vlan | 18:09 |
| arkan | and octavia_container_network_name: what should it be ? | 18:09 |
| arkan | lbaas_address ? | 18:09 |
| admin0 | noonedeadpunk, yep .. i was refreshing that page constantly.. got started as soon as i way it was merged | 18:09 |
| admin0 | i saw* | 18:09 |
| noonedeadpunk | ok, got it:) | 18:10 |
| CeeMac | I would just leave everything else as it is | 18:11 |
| arkan | ok | 18:11 |
| admin0 | seeing the fun you are having arkan, i am going to play with it this weekend :D | 18:13 |
| arkan | :)) | 18:13 |
| arkan | now it's http://paste.openstack.org/show/795783/ | 18:14 |
| arkan | just to check before the install | 18:14 |
| *** mmethot has quit IRC | 18:16 | |
| *** mmethot has joined #openstack-ansible | 18:16 | |
| CeeMac | Ok | 18:17 |
| arkan | CeeMac: is it ok now, shall I start the installation ? | 18:17 |
| arkan | great | 18:17 |
| CeeMac | You cleared the neutron network? | 18:18 |
| arkan | yes | 18:18 |
| arkan | it's destroyed | 18:18 |
| CeeMac | Great | 18:18 |
| arkan | if there is something to do first, is destroying | 18:19 |
| arkan | because it's easy to do it | 18:19 |
| arkan | :)) | 18:19 |
| CeeMac | Yes | 18:19 |
| CeeMac | :) | 18:19 |
| CeeMac | When you can't destroy something you know you're in trouble 😁 | 18:19 |
| arkan | heheh | 18:20 |
| CeeMac | You set the veth link up? | 18:21 |
| arkan | no | 18:21 |
| arkan | I said we will see then :)) | 18:21 |
| arkan | but even I need to see in netplan to do it | 18:22 |
| CeeMac | Ah, OK. Let's see how brctl looks after install then | 18:22 |
| arkan | also for my curiosity in your setup, did you need veth to do it manually ? | 18:22 |
| arkan | with post-up ? | 18:22 |
| CeeMac | I dont use octavia yet | 18:23 |
| arkan | aha | 18:23 |
| CeeMac | And I use OVS | 18:23 |
| CeeMac | Not lxb | 18:23 |
| arkan | ah | 18:23 |
| arkan | ok | 18:23 |
| arkan | so it's different | 18:23 |
| arkan | but what do you use for LB ? | 18:23 |
| CeeMac | So I'm pretty much winging this with you :D | 18:23 |
| arkan | hahaha | 18:23 |
| arkan | great | 18:23 |
| CeeMac | Itll be fine :) | 18:24 |
| CeeMac | I started looking at lbaasv2 pre-octavia | 18:24 |
| CeeMac | But had to put that on hold, we don't offer lbaas just yet | 18:24 |
| arkan | ok | 18:25 |
| CeeMac | Or deploy virtual appliance / haproxy vm if needed | 18:25 |
| arkan | if this will work, wow I will make a party today in my dream :)) | 18:25 |
| CeeMac | Don't rush out and buy balloons just yet | 18:26 |
| CeeMac | You might jinx it | 18:26 |
| arkan | yeah | 18:27 |
| arkan | but I know then how to destroy it :)) | 18:27 |
| arkan | the post up script should be created here | 18:29 |
| arkan | /etc/networkd-dispatcher/routable.d/ | 18:29 |
| arkan | we will see first | 18:30 |
| CeeMac | Yes | 18:30 |
| arkan | ok neutron has finished | 18:30 |
| arkan | now I will run octavia role | 18:31 |
| CeeMac | Ok | 18:33 |
| CeeMac | Out of curiosity how is brctl looking just now? | 18:33 |
| arkan | one moment | 18:33 |
| admin0 | do we have a straightforward ovs support now ? or we need to do a lot of overrides still ? | 18:34 |
| arkan | http://paste.openstack.org/show/795785/ (compute) | 18:35 |
| arkan | http://paste.openstack.org/show/795786/ (controller) | 18:36 |
| CeeMac | OK thanks | 18:36 |
| CeeMac | So eno1.510 is missing from both br-lbaas | 18:37 |
| arkan | don't thank me, I thank you | 18:37 |
| arkan | yes | 18:37 |
| arkan | until os-octavia-install finish | 18:38 |
| CeeMac | Pretty sure they should be there after neutron play | 18:39 |
| arkan | ok now it has finished | 18:39 |
| CeeMac | I'm willing to be wrong though :) | 18:39 |
| arkan | now it's there | 18:39 |
| arkan | after octavia | 18:40 |
| CeeMac | Show me the brctl :) | 18:40 |
| arkan | http://paste.openstack.org/show/795787/ (compute) | 18:40 |
| arkan | http://paste.openstack.org/show/795788/ (controller) | 18:41 |
| arkan | it's there eno2.510 | 18:41 |
| CeeMac | eno2.510 is there, but eno1.510 is still missing | 18:41 |
| CeeMac | Should be bound to br-lbaas | 18:42 |
| arkan | yes | 18:42 |
| arkan | I think it remained veth | 18:42 |
| arkan | it needs veth | 18:42 |
| CeeMac | No, it shouldn't need veth | 18:43 |
| arkan | to br-lbaas ? | 18:44 |
| CeeMac | Well, not for binding eno1. 510 | 18:44 |
| arkan | it needs br-lbaas <----> br-vlan | 18:44 |
| CeeMac | Yes | 18:44 |
| arkan | in the diagram is easy to watch it there :)) | 18:44 |
| CeeMac | Maybe eno1. 510 isn't necessary | 18:45 |
| arkan | yes | 18:45 |
| arkan | is not necessary | 18:45 |
| arkan | that was made to go through the router as tagged with id 510 | 18:46 |
| CeeMac | But you still had the host-bind-override for eno1. 510 on br-lbaas ? | 18:46 |
| arkan | aha yes | 18:46 |
| CeeMac | Unless you can't have the same vlan tagged on two interfaces? | 18:46 |
| CeeMac | Anyway | 18:46 |
| CeeMac | Try building the veth and see if that fixes the packet forwarding | 18:47 |
| arkan | should I use the code in https://github.com/rcbops/rpc-octavia/blob/master/INSTALLATION.md ? | 18:48 |
| arkan | # Create the post-up script <---- | 18:48 |
| arkan | and specify VLAN_ID=510 | 18:49 |
| CeeMac | Yes using vlan-id 510 | 18:49 |
| arkan | ok | 18:49 |
| CeeMac | The ifup bit won't work though | 18:51 |
| *** markvoelker has quit IRC | 18:54 | |
| arkan | ok | 18:54 |
| arkan | brctl show (controller) ---> http://paste.openstack.org/show/795789/ | 18:55 |
| CeeMac | Looks good | 18:55 |
| arkan | wow | 18:56 |
| arkan | I think something is changed now | 18:56 |
| arkan | I can see the rainbow | 18:56 |
| arkan | aha again | 18:56 |
| admin0 | i have this issue in glance mount.nfs: requested NFS version or transport protocol is not supported .. when using glance over NFS | 18:56 |
| admin0 | do you know what protocol its requesting ? | 18:56 |
| CeeMac | What's the verdict arkan? | 18:57 |
| arkan | I got enthusiasm | 18:57 |
| arkan | No route to host from octavia container | 18:57 |
| CeeMac | Hmm | 18:57 |
| CeeMac | So, so close | 18:57 |
| arkan | http://paste.openstack.org/show/795790/ | 18:58 |
| arkan | tcpdump (compute) http://paste.openstack.org/show/795791/ | 18:59 |
| arkan | this arp is coming from octavia container | 19:00 |
| arkan | it reached eno2.510 on br-vlan on compute node | 19:00 |
| arkan | I think, let me check | 19:01 |
| CeeMac | 91 is the octavia container? | 19:01 |
| arkan | I have this ip for eth14 in octavia container 172.29.235.220 | 19:02 |
| CeeMac | So 91 is asking where is 220 | 19:02 |
| arkan | 172.29.232.91 I want to see about it | 19:03 |
| CeeMac | But is also saying where 91 is, so responding to arp request | 19:03 |
| CeeMac | Looks like arp request isn't reaching 220 | 19:03 |
| arkan | it's there | 19:04 |
| CeeMac | Whats there? | 19:04 |
| arkan | lbaas-mgmt | 19:04 |
| CeeMac | You lost me | 19:04 |
| arkan | one moment | 19:05 |
| *** markvoelker has joined #openstack-ansible | 19:06 | |
| arkan | there is a port | 19:06 |
| arkan | http://paste.openstack.org/show/795792/ | 19:06 |
| CeeMac | Can you run tcpdump on eno2 on both servers? | 19:06 |
| arkan | inside lbaas-mgmt network this port has that ip | 19:06 |
| arkan | it's running on compute:nova | 19:06 |
| arkan | ok | 19:07 |
| CeeMac | Yes, that is the neutron provider network specified | 19:07 |
| *** mmethot has quit IRC | 19:08 | |
| CeeMac | Presumably the octavia container has .220 ip? | 19:08 |
| arkan | yes | 19:08 |
| CeeMac | Ok | 19:08 |
| arkan | tcpdump (compute) ---> http://paste.openstack.org/show/795793/ | 19:09 |
| arkan | tcpdump (controller) ---> http://paste.openstack.org/show/795794/ | 19:09 |
| CeeMac | OK so on the compute node you can see arp request coming from 220 asking where 91 is, and the arp reply | 19:10 |
| *** markvoelker has quit IRC | 19:10 | |
| arkan | yes | 19:11 |
| CeeMac | You can also see 91 asking where 220 is but no arp reply | 19:11 |
| CeeMac | Same on controller | 19:11 |
| arkan | you mean this one on compute Reply | 19:12 |
| arkan | Reply 172.29.232.91 is-at fa:16:3e:e1:dd:4c (oui Unknown), length 28 | 19:12 |
| CeeMac | Can you get a tcpdump of b12e701e_eth14 on controller please? | 19:14 |
| arkan | yes | 19:14 |
| arkan | 19:14:54.294827 ARP, Request who-has 172.29.232.193 tell 172.29.235.220, length 28 | 19:15 |
| arkan | 19:14:55.322870 ARP, Request who-has 172.29.232.193 tell 172.29.235.220, length 28 | 19:15 |
| arkan | 19:14:56.342750 ARP, Request who-has 172.29.232.193 tell 172.29.235.220, length 28 | 19:15 |
| arkan | only arping | 19:15 |
| arkan | no reply | 19:15 |
| arkan | only requests | 19:15 |
| CeeMac | Can you paste me the dump please? | 19:18 |
| arkan | for that ? | 19:18 |
| *** spatel has quit IRC | 19:18 | |
| CeeMac | For that interface yes | 19:18 |
| arkan | http://paste.openstack.org/show/795795/ | 19:19 |
| admin0 | how do you remove fwaas ? | 19:20 |
| admin0 | is just # in the config good enough ? | 19:20 |
| *** spatel has joined #openstack-ansible | 19:22 | |
| CeeMac | Can you do a tcpdump on v-br-lbaas.510? | 19:22 |
| arkan | ok | 19:22 |
| *** spatel has quit IRC | 19:23 | |
| CeeMac | And paste me, it's easier to te | 19:23 |
| CeeMac | *read on the phone | 19:23 |
| *** spatel has joined #openstack-ansible | 19:23 | |
| arkan | http://paste.openstack.org/show/795796/ | 19:24 |
| CeeMac | And v-br-vlan | 19:26 |
| CeeMac | On controller | 19:26 |
| arkan | CeeMac: I have a question | 19:26 |
| *** dave-mccowan has quit IRC | 19:27 | |
| CeeMac | Shoot | 19:27 |
| arkan | in the physical network, we have br-vlan but it's attached to the eno2 | 19:27 |
| CeeMac | Yes | 19:27 |
| arkan | it is not tagged as vlan | 19:27 |
| CeeMac | Yes | 19:27 |
| CeeMac | Neutron creates tagged port on eno2 | 19:28 |
| arkan | so, only the bridge created by neutron is tagged on brq23228bbf-bf which has interface eno2.510 | 19:28 |
| arkan | and we have veth between br-lbaas and br-vlan | 19:29 |
| CeeMac | If you add I think -e to the tcpdump on eno2 you should see the vlan tag | 19:29 |
| arkan | yes | 19:30 |
| *** spatel has quit IRC | 19:30 | |
| CeeMac | But we see requests and replies on compute node | 19:30 |
| CeeMac | But only requests on controller | 19:31 |
| CeeMac | And the bridges are set up identical | 19:31 |
| CeeMac | So if it works one way it should work the other | 19:31 |
| CeeMac | Let's continue on with the tcpdump and see where the edge of working is | 19:32 |
| arkan | http://paste.openstack.org/show/795797/ | 19:32 |
| CeeMac | Can you do one on v-br-vlan please? | 19:36 |
| arkan | in our setting what is the role of br-lbaas? in user_variables we used br-vlan in octavia_provider_network_type: vlan, this will use provider_network of vlan | 19:36 |
| arkan | yes | 19:36 |
| admin0 | do you guys know how can i remove fwaas and vpnaas from a running neutron .. is just removing the entry from the config and restart of neutron server enough for it ? | 19:37 |
| CeeMac | But you can see the arp reply to 220 is at least coming back to eno2 on the controller | 19:38 |
| arkan | admin0: not yet :)) | 19:38 |
| arkan | CeeMac: http://paste.openstack.org/show/795798/ | 19:38 |
| arkan | yes | 19:38 |
| CeeMac | br-lbaas is a physical bridge to attach the octavia container to, same as for br-mgmt and br-storage etc | 19:41 |
| admin0 | so at first, neutron plugin base had firewall_v2, lbv2 and vpn .. i # those and re-run the playbooks .. but they still linger around .. so trying to figure out how to remove them completely | 19:41 |
| arkan | got it | 19:41 |
| arkan | in our case it's using eno1.510 | 19:42 |
| arkan | "host_bind_override" is used in our case | 19:42 |
| arkan | it will use it instead of br-lbaas | 19:42 |
| CeeMac | Except eno1.510 isn't bound to br-lbaas now for some reason | 19:44 |
| arkan | yes | 19:44 |
| CeeMac | So the reply is getting stuck in br-vlan on the controller | 19:46 |
| arkan | but what about if we had eno1.510 bound to br-lbaas ? | 19:46 |
| arkan | yes | 19:46 |
| arkan | it's stuck | 19:47 |
| arkan | does it has to do with arp filter ? | 19:48 |
| arkan | as I know arp filter is there for security reason | 19:48 |
| CeeMac | Do you have the exact same config and kernel modules installed on both hosts? | 19:49 |
| CeeMac | Might be related to container networking | 19:49 |
| arkan | yes they should be | 19:49 |
| arkan | they are identical OS | 19:50 |
| arkan | ubuntu 18.04 | 19:50 |
| arkan | same stuff | 19:50 |
| arkan | modules, apt update & upgrade | 19:50 |
| arkan | kernel version | 19:50 |
| CeeMac | I'm wondering if it's some container bridging issue | 19:51 |
| CeeMac | I've seen similar under different circumstances | 19:51 |
| arkan | I don't know | 19:51 |
| arkan | I read that someone solved this | 19:52 |
| arkan | by using the script that we've used | 19:52 |
| CeeMac | It's working fine on compute where there are no containers right? | 19:52 |
| arkan | i'm wondering why it worked | 19:52 |
| arkan | humm, I can not tell, because my setup is using containers | 19:52 |
| arkan | lxc | 19:53 |
| CeeMac | On the compute node? | 19:53 |
| *** arkan has quit IRC | 19:53 | |
| *** arkan has joined #openstack-ansible | 19:54 | |
| arkan | I was disconnected | 19:54 |
| arkan | my question does the team here tested octavia on lxc ? | 19:54 |
| CeeMac | Are there any containers running on the compute node? | 19:54 |
| arkan | no | 19:54 |
| CeeMac | Ok | 19:54 |
| arkan | only on controller | 19:54 |
| CeeMac | So we know the exact same bridge configuration is working on compute node | 19:55 |
| CeeMac | At least to eno2 | 19:55 |
| arkan | yes | 19:55 |
| CeeMac | I guess the compute node isn't using br-lbaas | 19:55 |
| arkan | humm | 19:55 |
| CeeMac | I need to step away for an hour or so | 19:56 |
| CeeMac | Then I'll have another think when I come back | 19:57 |
| arkan | are you available tomorrow ? | 19:57 |
| CeeMac | On the phone off and on | 19:57 |
| arkan | and also what time is now ? | 19:57 |
| arkan | here is 22:57 | 19:57 |
| CeeMac | Maybe we need to tag the veth in br-vlan the same as we did for lbaas | 19:58 |
| CeeMac | 20:58 here | 19:58 |
| arkan | I'm available tomorrow from the morning till the evening | 19:58 |
| arkan | yes | 19:59 |
| arkan | I can tag it | 19:59 |
| CeeMac | OK, well try recreate v-br-vlan as v-br-vlan.510 and swap it into br-vlan instead | 19:59 |
| *** cshen has joined #openstack-ansible | 20:00 | |
| CeeMac | If I don't catch you when I'm back later I'll try catch you tomorrow | 20:00 |
| arkan | sure, just to charge my batteries (head) :)) | 20:00 |
| arkan | I will try this now | 20:00 |
| arkan | http://paste.openstack.org/show/795799/ | 20:04 |
| *** cshen has quit IRC | 20:04 | |
| arkan | tcpdump (compute) ---> http://paste.openstack.org/show/795800/ | 20:10 |
| arkan | tcpdump (controller) ---> http://paste.openstack.org/show/795801/ | 20:11 |
| arkan | octavia container after having v-br-vlan.510 ---> http://paste.openstack.org/show/795802/ | 20:12 |
| arkan | CeeMac: see you tommorrow, and thanks a lot for this investigation | 20:13 |
| CeeMac | See you arkan | 20:36 |
| *** KeithMnemonic has quit IRC | 20:47 | |
| admin0 | my 21.0.0.0rc2 fails in TASK [python_venv_build : Install python packages into the venv] -- neutron playbook | 20:49 |
| *** dave-mccowan has joined #openstack-ansible | 21:21 | |
| *** spatel has joined #openstack-ansible | 21:33 | |
| spatel | noonedeadpunk: look like something is wrong with build servers, still throwing same error | 21:34 |
| spatel | even i rename redhat-7.yml to redhat.yml | 21:35 |
| spatel | look like build server not fetching newer code | 21:35 |
| *** this10nly has quit IRC | 21:35 | |
| *** dave-mccowan has quit IRC | 21:40 | |
| *** cshen has joined #openstack-ansible | 22:01 | |
| *** cshen has quit IRC | 22:05 | |
| *** this10nly has joined #openstack-ansible | 22:34 | |
| *** tosky has quit IRC | 23:36 | |
| *** rh-jelabarre has quit IRC | 23:38 | |
| *** spatel has quit IRC | 23:40 | |
| *** rh-jelabarre has joined #openstack-ansible | 23:41 | |
| *** rh-jelabarre has quit IRC | 23:42 | |
| *** rh-jelabarre has joined #openstack-ansible | 23:42 | |
| *** rh-jelabarre has quit IRC | 23:47 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!