Tuesday, 2020-09-08

*** spatel has joined #openstack-ansible00:24
*** spatel has quit IRC00:29
*** cshen has joined #openstack-ansible01:29
*** cshen has quit IRC01:33
*** NewJorg has quit IRC01:42
*** NewJorg has joined #openstack-ansible01:44
*** noonedeadpunk has quit IRC01:47
*** gary_perkins_ has quit IRC01:47
*** gary_perkins has joined #openstack-ansible01:48
*** ChiTo has quit IRC02:03
*** spatel has joined #openstack-ansible02:21
*** cshen has joined #openstack-ansible03:29
*** cshen has quit IRC03:34
*** evrardjp has quit IRC04:33
*** evrardjp has joined #openstack-ansible04:33
*** spatel has quit IRC04:37
*** cshen has joined #openstack-ansible05:29
*** cshen has quit IRC05:34
*** cshen has joined #openstack-ansible06:00
*** noonedeadpunk has joined #openstack-ansible06:01
*** miloa has joined #openstack-ansible06:03
*** cshen has quit IRC06:04
*** d34dh0r53 has joined #openstack-ansible06:17
*** d34dh0r53 has quit IRC06:17
*** janno has quit IRC06:51
*** janno has joined #openstack-ansible06:51
*** MickyMan77 has quit IRC07:07
*** shyamb has joined #openstack-ansible07:13
*** cshen has joined #openstack-ansible07:18
*** andrewbonney has joined #openstack-ansible07:28
*** tosky has joined #openstack-ansible07:35
*** shyam89 has joined #openstack-ansible07:41
*** jbadiapa has joined #openstack-ansible07:43
*** shyamb has quit IRC07:43
*** tosky has quit IRC08:05
*** tosky has joined #openstack-ansible08:10
*** shyamb has joined #openstack-ansible08:11
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-os_senlin master: Initial commit to os_senlin  https://review.opendev.org/74936508:12
*** cshen has quit IRC08:13
*** tosky has quit IRC08:14
*** shyam89 has quit IRC08:14
*** tosky has joined #openstack-ansible08:14
*** cshen has joined #openstack-ansible08:15
*** cshen has quit IRC08:18
*** cshen has joined #openstack-ansible08:20
*** SecOpsNinja has joined #openstack-ansible09:28
*** tosky has quit IRC09:39
*** tosky has joined #openstack-ansible09:55
*** shyamb has quit IRC10:13
*** shyamb has joined #openstack-ansible10:22
*** shyam89 has joined #openstack-ansible10:33
*** shyamb has quit IRC10:35
*** shyam89 has quit IRC10:47
*** shyamb has joined #openstack-ansible10:50
*** cshen has quit IRC10:52
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-os_nova master: Bump libvirt version to prevent compute failure  https://review.opendev.org/75032010:57
noonedeadpunkjrosser: ^ this nasty thing made nova-compute fail for centos 7....10:58
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-os_nova stable/ussuri: Bump libvirt version to prevent compute failure  https://review.opendev.org/75032110:58
*** dave-mccowan has joined #openstack-ansible10:59
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-os_swift stable/ussuri: Delegate gnocchi retrievement task to setup host  https://review.opendev.org/75016910:59
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-os_glance stable/ussuri: Fix native service path  https://review.opendev.org/75013911:00
*** cshen has joined #openstack-ansible11:03
jrossernoonedeadpunk: you missing a link - what is [1] ?11:05
noonedeadpunkI am:(11:06
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-os_nova master: Bump libvirt version to prevent compute failure  https://review.opendev.org/75032011:06
noonedeadpunkhttp://paste.openstack.org/show/797571/11:06
*** dave-mccowan has quit IRC11:12
*** mgariepy has quit IRC11:15
*** dave-mccowan has joined #openstack-ansible11:15
*** stuartgr has quit IRC11:31
*** stuartgr has joined #openstack-ansible11:32
*** shyamb has quit IRC11:43
*** shyamb has joined #openstack-ansible11:45
*** MickyMan77 has joined #openstack-ansible11:54
*** mathlin has joined #openstack-ansible11:57
*** mgariepy has joined #openstack-ansible12:05
openstackgerritJay Jahns proposed openstack/openstack-ansible-os_neutron stable/ussuri: Add Initial NSX Integration  https://review.opendev.org/75032812:08
openstackgerritJay Jahns proposed openstack/openstack-ansible-os_nova stable/ussuri: Set Bridge Information for NSX Integration  https://review.opendev.org/75033012:08
*** shyamb has quit IRC12:11
*** rh-jelabarre has joined #openstack-ansible12:12
*** mathlin has quit IRC12:12
*** redrobot has joined #openstack-ansible12:17
*** cshen has quit IRC12:18
*** cshen has joined #openstack-ansible13:03
MickyMan77 I have problem to get the HAProxy to work, I'm not able to ping the VIP address. When I check the net stat I can see the vip address on all controller nodes.13:11
MickyMan77[root@controller01 ~]# netstat -pan | grep -I 443tcp        0      0 10.26.13.254:443        0.0.0.0:*               LISTEN      143691/haproxytcp        0      0 10.26.11.254:443        0.0.0.0:*               LISTEN      143691/haproxy13:11
MickyMan77---13:11
MickyMan77[root@controller01-for ~]# netstat -pan | grep -i 44313:11
MickyMan77tcp        0      0 10.26.13.254:443        0.0.0.0:*               LISTEN      143691/haproxy13:12
MickyMan77tcp        0      0 10.26.11.254:443        0.0.0.0:*               LISTEN      143691/haproxy13:12
MickyMan77---13:13
MickyMan77arp -n | grep 25413:13
MickyMan7710.26.13.254                     (incomplete)                              br-mgmt13:13
miloa@MikyMan77 If the vip address is on all controllers nodes it means (I think) that keepalived cannot check if the other nodes are up and as they are saw as down it bring up the vip on each node. Perhaps check if each node can ping each other on its main address.13:38
MickyMan77the nodes can ping the main ip address on all nodes and also ping all container ip addresses via the br-mgmt Nic.13:47
MickyMan77but not the HAProxy vip adress.13:47
*** d34dh0r53 has joined #openstack-ansible13:56
*** cshen has quit IRC13:59
miloaDid you check your log to see if there is any messages of keepalived ?13:59
MickyMan77hmm,14:03
MickyMan77tail keepalived-notifications.log14:03
MickyMan772020-09-08 11:19:30 Trying to restart haproxy to get out of faulty state14:03
*** MickyMan77 has quit IRC14:24
*** cshen has joined #openstack-ansible14:28
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-os_keystone master: Define condition for the first play host one time  https://review.opendev.org/74964914:33
*** cshen has quit IRC14:33
*** cshen has joined #openstack-ansible14:33
*** spatel has joined #openstack-ansible14:36
*** spatel has quit IRC14:44
*** spatel has joined #openstack-ansible15:03
SecOpsNinjajrosser, spatel, today i was able to put lets encrypt cert in my openstack haproxy and now magnum stoped complaining about CA cert!! What i did was creating a br-osa-dmz in infra host with static ip that corresponded to external_lb_vip_address. Now, because i want to use that network for floating ips, i added the same network to our compute host (with no ip) and configured the flat provider_15:14
SecOpsNinjanetwork to use host_bind_override: "br-osa-dmz". This was working fine, until i added this network in openstack as a external provider and create a router that connect to it. Atm the ip that was associated to the br-osa-dmz, switched to brq14a7b7c8-6e and now i can't access the openstackl public endpoint. i supose that this isn't a problem and i did something wrong but i can't understand wha15:14
SecOpsNinjat...15:14
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-os_keystone master: Define condition for the first play host one time  https://review.opendev.org/74964915:20
*** cshen has quit IRC15:32
noonedeadpunklet's skip our meeting today if there's no objections?16:00
*** cshen has joined #openstack-ansible16:01
spatelworks for me16:02
spatelnoonedeadpunk: look like very close on this patch https://review.opendev.org/#/c/749365/16:03
spateljrosser: did your senlin stuff working after fixing our last auth_url patch?16:04
*** cshen has quit IRC16:06
*** mgariepy has quit IRC16:09
*** miloa has quit IRC16:11
noonedeadpunkspatel: I posted patch to senlin https://review.opendev.org/#/c/749874/ which should cover found issue16:25
noonedeadpunkbut yeah, it's supposed to work with self-signed ssls16:25
spatelnoonedeadpunk: that is cool!16:26
spatelself-sign SSL isn't fun :)16:27
noonedeadpunkit is for CI envs especially)16:27
spatelnoonedeadpunk: why i am not seeing any centos-7 builds here - https://review.opendev.org/#/c/749365/16:28
spatelit was there last week16:28
noonedeadpunkbecause we've dropped all centos 7 jobs for master16:28
noonedeadpunkas we've planned to drop it for V16:29
spatelV ?16:30
noonedeadpunkVictoria - next openstack release16:30
spatelah! so we are not going to continue c7 anymore right?16:31
noonedeadpunkyep, we didn't. There're so many nasty workarounds for centos7 to make it work with py3...16:31
noonedeadpunkAnd distro packages has been dropped for train, so no distro path even for ussuri16:31
noonedeadpunkand ceph is barely supported as well...16:32
spatel+116:32
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-os_manila master: Add centos-8 support  https://review.opendev.org/73964616:32
spatelso i am running pike on my production with c7 in that case i can't upgrade to latest until i upgrade c8 right?16:33
noonedeadpunkso you're supposed to update to ussuri, then resetup to centos 8 and you can move forward16:33
spatelgot it16:39
*** renich has joined #openstack-ansible17:07
*** mgariepy has joined #openstack-ansible17:10
*** renich has quit IRC17:29
*** gyee has joined #openstack-ansible17:33
*** mgariepy has quit IRC17:37
*** mgariepy has joined #openstack-ansible17:38
*** carlosmss has joined #openstack-ansible17:38
carlosmssHi guys, someone seen this error with RabbitMQ, I've searched about this issue but no success. My scenario is a cluster AMPQ deployed with Openstack-Ansible.: AMQP server on xxx.xx.xx.xx:5672 is unreachable. WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeat thread processing, retrying...: ConnectionForced: Too many heartbeats missed17:44
*** MickyMan77 has joined #openstack-ansible17:51
MickyMan77I'm not able to get the keepalived to work with network bridge, it will not add the VIP ip address. If I change the keepalived config so I use ethX interface instead of br-mgmt it works perfect.18:01
MickyMan77I'm using centos 8 and the br-mgmt is port of a bond.18:02
MickyMan77*part of a bond18:02
noonedeadpunkbridge can't be bond port actually... bond can be one of the interfaces on the bridge18:03
noonedeadpunkand keepalive adds ip address o nthe interface when it has another ip from the same network iirc (may be wrong)18:04
noonedeadpunkI guess I didn't test keepalived with centos 8 on multinode setup though....18:05
noonedeadpunkspatel: did you have any issues in your lab with several centos8 controllers and keepalived?18:05
spatelnoonedeadpunk: no18:06
spateli am running 3 node controller and haven't seen any issue so far18:06
*** MickyMan77 has quit IRC18:07
spatelThis is my keepalived and haven't seen any issue so far - http://paste.openstack.org/show/797603/18:10
spatelin my keepalived its using br-mgmt interface and no issue so far.18:11
noonedeadpunkthanks for the info:)18:12
noonedeadpunktrack_script looks weird though18:13
*** mgariepy has quit IRC18:14
spatelnoonedeadpunk: yes, not sure why and how its working then18:25
noonedeadpunkhave you tested failover?18:25
noonedeadpunkbut it's more about check scripts....18:25
spatellet me test failover and see18:26
noonedeadpunkIt looks like instead of list of scripts you have only single one...18:27
spatelhmm18:28
*** mgariepy has joined #openstack-ansible18:28
noonedeadpunkbut actually by default it has to be list, and you have content as list as well....18:29
noonedeadpunkdict_keys..... hm18:29
noonedeadpunkah damn it18:29
noonedeadpunkspatel: can you try set `keepalived_scripts.keys() | list` for track_scripts in openstack_ansible/inventory/group_vars/haproxy/keepalived.yml and re-run playbook?18:30
spatellet me check18:31
spatelThis is what i have   track_scripts: "{{ keepalived_scripts.keys() }}"18:32
noonedeadpunkyeah, just add `| list` filter18:33
spatelyou want me to put "| list" ?18:33
spatelok doing it18:33
spatelwhich playbook you want me to run?18:34
noonedeadpunkhaproxy-install18:37
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible master: Fix keys() method output  https://review.opendev.org/75047318:37
spatelrunning...18:37
spatelnoonedeadpunk: neat - http://paste.openstack.org/show/797604/18:39
noonedeadpunkyeah, nice18:41
noonedeadpunkthanks so much for testing - saving me tons of time!18:42
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-os_manila master: Add centos-8 support  https://review.opendev.org/73964618:42
noonedeadpunkhttps://review.opendev.org/#/c/750473/ is going to cover that18:42
spatel+118:43
spatelnoonedeadpunk: fyi, keepalived failover is working18:50
*** viks____ has quit IRC18:58
*** d34dh0r53 has quit IRC19:08
*** d34dh0r53 has joined #openstack-ansible19:10
*** spatel has quit IRC19:18
*** SecOpsNinja has left #openstack-ansible19:21
*** cshen has joined #openstack-ansible19:28
*** cshen has quit IRC19:32
*** MickyMan77 has joined #openstack-ansible20:01
MickyMan77@noonedeadpunk the bridge is config like this.20:05
MickyMan77bond0 have this config20:05
MickyMan77BRIDGE=br-mgmt20:05
MickyMan77the nic have this,20:06
MickyMan77SLAVE=yesMASTER=bond_mgt020:06
MickyMan77and the br-mgmt is working perfect.20:06
openstackgerritMerged openstack/openstack-ansible master: Fix neutron-server default serial  https://review.opendev.org/74654620:06
MickyMan77but I'm not able to add the VIP IP address via keepalived.20:07
openstackgerritMerged openstack/openstack-ansible-os_keystone master: Define condition for the first play host one time  https://review.opendev.org/74964920:17
jrosserMickyMan77: keepalived uses multicast between the nodes to decide which node has the VIP20:28
jrosserif you use tcpdump on the different interfaces you should be able to see that from all the nodes20:28
noonedeadpunkunless it's blocked or not allowed, yeah20:28
jrosserif that traffic does not make it then things will be broken20:29
MickyMan77the logs say this.20:31
MickyMan77VRRP_Group(haproxy): Syncing instances to FAULT state20:31
MickyMan77(external) entering FAULT state20:31
MickyMan77 (internal) entering FAULT state20:31
MickyMan77(external) removing VIPs.20:32
MickyMan77(internal) removing VIPs.20:32
jrosserok but it will be necessary to debug one level deeper20:34
jrosserto look at the traffic on each node and see if it is as we should expect20:34
jrosseralso, as this is centos i think it would be a good idea to check the bridge forwarding, becasue long long ago we had to fix this for Centos AIO builds https://github.com/openstack/ansible-role-systemd_networkd/commit/242b3c3fb4a3f8ebeba5c24cf8c7a510cba1414320:34
jrosserand also a snaity check, this is actual servers and not vmwaare or other virtualisation?20:36
MickyMan77this is ibm servers, x3650 and x385020:38
*** andrewbonney has quit IRC20:39
jrosserok, thats fine20:40
jrosserOSA does not set up the host networking, and in the past there has been issues with centos bridges not forwarding by default, so that is worth checking20:40
noonedeadpunkfwiw I have the follwoing config of interfaces for ubuntu with vip 172.20.0.9 http://paste.openstack.org/show/797606/20:41
jrosserthen tcpdump on the bridge and eth interfaces to see if you can find the vrrp traffic would be the next step20:41
jrosser"If a configured script returns a non-zero exit code f times in succession, Keepalived changes the state of the VRRP instance or group to FAULT"20:45
jrossernoonedeadpunk: is it this? https://review.opendev.org/75047320:46
noonedeadpunkum, it's just fixing check scripts for keepalived. it didn't fail for spatel though....20:48
noonedeadpunkI mean without it20:48
noonedeadpunkso not sure it it will help here20:48
jrosserthe track scripts also check for ping of an upstream thing20:49
jrosserso i guess that being non-contactable is going to cause a transition to FAULT state20:50
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-os_cinder master: Use cinder_service_setup_host for qos and types creation  https://review.opendev.org/75049120:50
noonedeadpunkbtw that is pretty nasty bug I think ^20:50
noonedeadpunkand worth pending time to implement these options in openstack collection....20:50
noonedeadpunk*spending20:50
*** cshen has joined #openstack-ansible20:56
*** spatel has joined #openstack-ansible20:56
*** cshen has quit IRC21:00
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-os_cinder master: Set correct permissions for rootwrap.d  https://review.opendev.org/75049321:05
noonedeadpunkmgariepy: jamesdenton seems ovn is failing now for some reason ;( https://review.opendev.org/#/c/73301721:08
jamesdentonhtmm21:09
noonedeadpunkon tempest networking https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_ec6/733017/23/check/openstack-ansible-deploy-aio_ovn_lxc-ubuntu-focal/ec6b241/logs/openstack/aio1_utility_container-e55222e2/utility/stestr_results.html21:09
jamesdentoncool, wanted to make sure i was on the right track there.21:10
noonedeadpunkbut I think it started failing after my rebase....21:11
noonedeadpunkso might be we merged smth breaking......21:11
noonedeadpunk(we shouldn't though)21:11
jamesdentonlemme spin one up and see21:12
MickyMan77@jrosser will this found the vrrp traffic ?,      # tcpdump -vvv -n -i any host 224.0.0.1821:18
jrosseri just tried this  tcpdump -i br-mgmt vrrp21:19
jrosserand i see a bunch of 20:54:39.142988 IP 10.11.128.21 > vrrp.mcast.net: VRRPv2, Advertisement, vrid 111, prio 100, authtype simple, intvl 1s, length 2021:21
jrosserwhich is the ip of the other haproxy (i have two in this particular case)21:21
MickyMan77completely empty, tcpdump does not show anything. (I also have restated the keepalived service)21:21
* jrosser late here, enough for today21:21
spatelMickyMan77: what is the problem with keepalived?21:29
spatelfailover issue?21:29
MickyMan77@spatel it will not add VIP address on the br-mgmt21:30
spatelin last 3 years i never troubleshoot keepalived, because it just work without any issue. i am running 4 openstack cluster and non has any issue21:30
MickyMan77what os do you run on ?21:30
spatelcentos7 and centos821:31
MickyMan77hmm, using centos821:31
MickyMan77*i'm using21:31
spatellast week i build 3 controller node lab and didn't see any issue21:31
spateli am assuming something is missing in config side.21:32
MickyMan77can you give me a copy of your, openstack_user_config.yml and user_variables.yml21:33
spatelwhy don't you post your openstack_user_config.yml and user_variable.yml file21:33
MickyMan77where can I post it ?21:34
spatelhttp://paste.openstack.org/show/797612/21:34
spatelhttp://paste.openstack.org is your friend to share snippet21:35
MickyMan77thx21:35
spatelhttp://paste.openstack.org/show/797613/21:36
spateli have posted my both file.. just verify your config snippet and you should be good.21:36
*** jbadiapa has quit IRC21:41
*** d34dh0r53 has quit IRC21:42
*** d34dh0r53 has joined #openstack-ansible21:42
MickyMan77@spatel how have you. config the bridge, br-host br-mgmt21:45
spatelIn production i have bond0 with LACP21:46
spatelcreate br-mgmt and attached with bond0.XX vlan id21:47
spatelMickyMan77: something like this - http://paste.openstack.org/show/797614/21:48
spatelif you are using systemd-networkd style network in that case configuration will be little different.21:50
MickyMan77NM_CONTROLLED=no   I use NetworkManager, is that ok ?21:50
spatelI hate NetworkManager :(21:51
spatelyou have to show your network config otherwise its hard to answer whatever question you are asking21:53
spatelsystemd-networks vs legacy way21:54
MickyMan77network settings, http://paste.openstack.org/show/797615/21:57
spatelMickyMan77: looks good22:08
spatelif you able to ping other end then you should be good22:09
spateli am not seeing any issue here22:09
spateldoes ifcfg-mgt0 and ifcfg-mgt1 coming from same switch?22:10
MickyMan77yes22:10
MickyMan77they are connected to a Cisco 650022:10
spatelok22:10
MickyMan77and I can ping all nodes and container22:10
spatelcan you ping other controller node?22:10
MickyMan77the only thing that not work is the vip adress.22:11
spatelis br-mgmt is your external endpoint?22:11
spatelin my case i have br-host for external and br-mgmt for internal vip22:11
spatelbr-mgmt isn't routable to outside world (its purely for openstack control plane)22:12
spatelnot sure how config will look like when we have only br-mgmt and not br-host22:13
MickyMan77our openstack installation is only for internal use.22:13
spatelSame here, we are running in private datacenter22:14
MickyMan77and we will only use the  br-mgmt for now..22:14
spatelmay be that is your problem22:14
spatelI use br-host and br-mgmt to all my deployment and haven't seen any issue so far..22:15
-spatel- haproxy_keepalived_external_interface: br-host22:16
-spatel- haproxy_keepalived_internal_interface: br-mgmt22:16
MickyMan77I will check that tomorrow.22:16
MickyMan77need to sleep now..22:16
spatelsee this is what i have, so in your case it will be br-mgmt on both external/internal22:16
spatelMickyMan77: have a good night :)22:16
spateli gotta go too22:16
MickyMan77thx for the help22:17
*** spatel has quit IRC22:18
*** tosky has quit IRC22:55
*** cshen has joined #openstack-ansible22:56
*** cshen has quit IRC23:01
*** carlosmss has quit IRC23:05

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!