Tuesday, 2018-11-20

openstackgerritGuilherme  Steinmuller Pimentel proposed openstack/openstack-ansible-os_placement master: [WIP] Create base files to install placement  https://review.openstack.org/61882000:01
*** tosky has quit IRC00:14
logan-cloudnull: https://bugzilla.redhat.com/show_bug.cgi?id=1650342 :/00:29
openstackbugzilla.redhat.com bug 1650342 in systemd "systemd-networkd support in RHEL 8" [Unspecified,New] - Assigned to systemd-maint00:29
*** eumel8 has quit IRC00:57
*** cshen has quit IRC01:07
openstackgerritMerged openstack/openstack-ansible stable/queens: Move ARA install to end of bootstrap  https://review.openstack.org/61778601:10
openstackgerritMerged openstack/openstack-ansible stable/pike: Move ARA install to end of bootstrap  https://review.openstack.org/61778702:03
*** cshen has joined #openstack-ansible02:03
*** cshen has quit IRC02:07
openstackgerritbrandon zhao proposed openstack/openstack-ansible-apt_package_pinning master: fix tox python3 overrides  https://review.openstack.org/61893502:39
openstackgerritbrandon zhao proposed openstack/openstack-ansible-os_tacker master: fix tox python3 overrides  https://review.openstack.org/61893702:56
*** maddtux has joined #openstack-ansible03:27
*** spatel has joined #openstack-ansible03:54
spatelcloudnull: ^^03:54
spatelare you there?03:54
spatelwhat is wrong here? http://paste.openstack.org/show/735699/03:55
spateleverything is working fine.. but just noticed that when i was poking around some stuff to find out why horizon GUI is slow03:56
*** cshen has joined #openstack-ansible04:04
*** spatel has quit IRC04:05
*** udesale has joined #openstack-ansible04:09
*** cshen has quit IRC04:09
*** ivve has joined #openstack-ansible04:41
*** chandankumar has joined #openstack-ansible04:55
*** chandankumar is now known as chkumar|ruck04:56
*** cshen has joined #openstack-ansible05:49
openstackgerritChristian Zunker proposed openstack/openstack-ansible-os_cloudkitty master: Configure cloudkitty-api to use uwsgi  https://review.openstack.org/56716805:57
*** cshen has quit IRC05:57
*** radeks_ has joined #openstack-ansible05:59
*** radeks_ has quit IRC06:00
*** radeks_ has joined #openstack-ansible06:02
*** czunker has joined #openstack-ansible06:13
*** PrinzElvis has joined #openstack-ansible06:19
*** chkumar|ruck has quit IRC06:37
*** spsurya has joined #openstack-ansible06:39
*** dkopper_ has joined #openstack-ansible06:57
*** gkadam has joined #openstack-ansible07:02
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-lxc_hosts master: Minimise distro packages installed into the lxc image  https://review.openstack.org/61307807:06
*** chandankumar has joined #openstack-ansible07:14
*** chandankumar is now known as chkumar|ruck07:14
*** pcaruana has joined #openstack-ansible07:20
*** cshen has joined #openstack-ansible07:25
*** eumel8 has joined #openstack-ansible07:32
*** pcaruana has quit IRC07:34
*** dkopper__ has joined #openstack-ansible07:34
*** dkopper__ has quit IRC07:37
*** pcaruana has joined #openstack-ansible07:40
*** dkopper_ has quit IRC07:50
*** chandankumar has joined #openstack-ansible07:52
*** shardy has joined #openstack-ansible07:55
*** chkumar|ruck has quit IRC07:56
*** chandankumar is now known as chkumar|ruck07:57
*** rpittau has joined #openstack-ansible08:04
*** dcdamien has joined #openstack-ansible08:06
openstackgerritFrank Kloeker proposed openstack/openstack-ansible-haproxy_server master: Add feature Letsencrypt SSL certification  https://review.openstack.org/58677408:06
*** cshen has quit IRC08:42
*** rodolof has joined #openstack-ansible08:44
*** cshen has joined #openstack-ansible08:45
*** ahosam has joined #openstack-ansible08:49
*** priteau has joined #openstack-ansible08:54
*** lbragstad has joined #openstack-ansible08:54
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-lxc_hosts master: Minimise distro packages installed into the lxc image  https://review.openstack.org/61307808:55
openstackgerritErik Berg proposed openstack/openstack-ansible-os_neutron master: Apply provider-configuration for 'ml2.ovs.dvr' also  https://review.openstack.org/61897209:06
*** hamzaachi has joined #openstack-ansible09:09
*** thuydang has joined #openstack-ansible09:09
*** tosky has joined #openstack-ansible09:11
*** DanyC has joined #openstack-ansible09:16
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-lxc_hosts master: Minimise distro packages installed into the lxc image  https://review.openstack.org/61307809:23
*** ahosam has quit IRC09:44
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-lxc_hosts master: Minimise distro packages installed into the lxc image  https://review.openstack.org/61307809:54
*** rodolof has quit IRC10:02
*** chkumar246 has joined #openstack-ansible10:13
*** ahosam has joined #openstack-ansible10:13
*** chkumar|ruck has quit IRC10:15
*** rodolof has joined #openstack-ansible10:16
*** chkumar246 is now known as chkumar|ruck10:18
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-lxc_hosts master: Minimise distro packages installed into the lxc image  https://review.openstack.org/61307810:21
*** thuydang has quit IRC10:23
*** hamzaachi has quit IRC10:26
openstackgerritChristian Zunker proposed openstack/openstack-ansible-os_cloudkitty master: Configure cloudkitty-api to use uwsgi  https://review.openstack.org/56716810:31
odyssey4meo/ been sick since returning from berlin... morning all10:32
openstackgerritMerged openstack/openstack-ansible-os_cinder stable/queens: Add libpython2.7 as a required package.  https://review.openstack.org/61877910:34
*** nowster has joined #openstack-ansible10:34
openstackgerritMerged openstack/openstack-ansible-nspawn_hosts stable/rocky: Add missing variable for nspawn networking  https://review.openstack.org/61615910:42
dcdamienmorning odyssey4me, I was right -> http://eavesdrop.openstack.org/irclogs/%23openstack-ansible/%23openstack-ansible.2018-11-19.log.html#t2018-11-19T14:30:4410:45
*** chkumar246 has joined #openstack-ansible10:54
*** chkumar|ruck has quit IRC10:57
*** czunker has quit IRC11:00
*** fghaas has joined #openstack-ansible11:00
fghaasodyssey4me (and cloudnull if you're awake already), you'll recall the discussion that we had in Berlin about not setting wsrep_sync_wait=1 (formerly wsrep_causal_reads=on) for Galera — see http://galeracluster.com/documentation-webpages/mysqlwsrepoptions.html#wsrep-sync-wait for details on that option.11:06
fghaasI think I just thought up a scenario where this could get really, really painful and I wonder if you agree: haproxy directs mysql clients to a single backend node, because the others are set to backup. We do a simple tcp check there. Now, suppose our connections from mysql clients hover just around max_connections, so the primary backend's mysqld flaps between accepting and refusing connections. Wouldn't that mean that we woul11:06
fghaasd be getting alternate writer nodes, causing all sorts of havoc with wsrep-sync-wait=0?11:06
evrardjpcan someone point me to the etherpad used in the summit's for "Deployment tools feedback" ? I can't find it. mnaser you were there, right?11:07
mnaseryes i was there11:07
mnaseri meant to send a summary soon11:07
mnaserevrardjp: https://etherpad.openstack.org/p/BER-deployment-tools11:08
* mnaser goes back to starting day11:08
*** ahosam has quit IRC11:08
*** cshen has quit IRC11:08
*** ahosam has joined #openstack-ansible11:08
*** admin0 has quit IRC11:09
nowsterjrosser: I now have blank Ubuntu 18.0.1 server to test your patches on.11:10
*** udesale has quit IRC11:11
dcdamienWhat's wrong with Zuul? Anybody knows?11:14
mnaserwhat issues you having dcdamien ?11:17
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-lxc_hosts master: Minimise distro packages installed into the lxc image  https://review.openstack.org/61307811:17
jrossernowster: nice - i've been prodding at this a bit more https://review.openstack.org/#/c/618305/11:18
*** cshen has joined #openstack-ansible11:20
dcdamienmnaser: timeouts, not just on my checks11:20
nowsterjrosser: ta.11:21
dcdamienmnaser:11:21
dcdamienhttps://review.openstack.org/#/c/618822/11:21
dcdamienhttps://review.openstack.org/#/c/617785/11:21
dcdamienhttps://review.openstack.org/#/c/617053/11:21
mnaserdcdamien: thats weird.11:22
mnaserdid we recently merge something that affected things?11:22
evrardjpfghaas: interesting use case, but the idea was to move out from tcp check and use real checks with a proper endpoint, but I am not fully aware of the situation you're talking about (branch and other technical details)11:23
dcdamienmaybe something with moving ARA to the end of bootstrap?11:24
evrardjpfghaas: I will read more about it. Thanks for pointing to this11:24
evrardjpdcdamien: queens had a timeout in installing galera11:25
evrardjpI'd say there are a few things that might have slowed-down and need further/deeper analysis11:26
evrardjpty mnaser for the link11:27
evrardjpbut thats the comparison11:27
evrardjpI am thinking the other session11:28
evrardjpi cant really remember it though..11:28
*** chkumar246 is now known as chkumar|ruck11:30
*** rpittau_ has joined #openstack-ansible11:31
*** rpittau has quit IRC11:34
*** rpittau_ is now known as rpittau11:34
dcdamienevrardjp: when there is `TIMED_OUT` on check that it means that whole check was aborted because it took too long?11:35
*** ahosam has quit IRC11:44
*** ahosam has joined #openstack-ansible11:44
*** ahosam has quit IRC11:46
jrossernowster: i have been looking at how we get your haproxy bug fixed upstream, would you be able to help out with a test case here? https://bugs.launchpad.net/ubuntu/+source/haproxy/+bug/180406911:47
openstackLaunchpad bug 1804069 in haproxy (Ubuntu) "haproxy fails on arm64 due to alignment error" [Undecided,Incomplete]11:47
nowsterjrosser: hmm... Let me think.11:48
jrosseri expect they want a super simple config example and a curl that shows the zero length response11:50
nowsteryep... I can do11:51
jrossercool, thanks11:51
*** chkumar246 has joined #openstack-ansible11:53
nowsterjrosser: of course it's not breaking on the tests I'm doing...11:55
*** chkumar|ruck has quit IRC11:56
*** maddtux_ has joined #openstack-ansible11:56
jrossernowster: of course :)11:56
*** gkadam has quit IRC11:56
*** maddtux has quit IRC11:56
*** pabelanger has left #openstack-ansible11:58
*** noonedeadpunk[h] is now known as noonedeadpunk12:00
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-lxc_hosts master: Minimise distro packages installed into the lxc image  https://review.openstack.org/61307812:03
odyssey4mewell, that's nice - basically every single version of OSA we've released can no longer bootstrap without adjusting a-r-r: https://github.com/willshersystems/ansible-sshd/issues/9812:04
odyssey4mewe've always used the x.y.z version tag instead of the vx.y.z tag, and they've changed to the latter12:05
*** gkadam has joined #openstack-ansible12:14
guilhermespmorning osa team :)12:14
openstackgerritgit-harry proposed openstack/openstack-ansible stable/pike: Fix ansible-sshd version  https://review.openstack.org/61899512:14
*** aiyengar_ has joined #openstack-ansible12:15
*** chkumar has joined #openstack-ansible12:15
*** maddtux_ has quit IRC12:17
*** maddtux has joined #openstack-ansible12:18
*** chkumar246 has quit IRC12:18
dcdamienodyssey4me: can you please review https://review.openstack.org/#/c/617847/ ?12:21
*** aiyengar_ has quit IRC12:21
openstackgerritgit-harry proposed openstack/openstack-ansible stable/pike: Fix ansible-sshd version  https://review.openstack.org/61899512:22
jamesdentongood morning12:22
*** asettle has joined #openstack-ansible12:25
*** priteau has quit IRC12:28
*** ahosam has joined #openstack-ansible12:35
evrardjpdcdamien: yes12:41
evrardjpwow git-harry proposed something12:41
evrardjpclickbait12:41
*** ahosam has quit IRC12:44
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-os_nova master: Fix nova-lxd tests for bionic.  https://review.openstack.org/61630212:47
*** rodolof has quit IRC12:52
*** chkumar246 has joined #openstack-ansible12:54
*** chkumar246 is now known as chkumar|ruck12:54
*** chkumar has quit IRC12:57
*** maddtux has quit IRC13:00
*** mcela1 has joined #openstack-ansible13:01
mcela1Hello, anyone has experience with setting up networks for tenants using vlans? or is there any guide specifically targeting this?13:02
openstackgerritMerged openstack/openstack-ansible-nspawn_hosts stable/rocky: Add host only network option  https://review.openstack.org/61616013:04
jrossermcela1: start here with the bit of config that talks about "eth12" https://docs.openstack.org/openstack-ansible/rocky/user/prod/example.html13:05
nowsterjrosser: I've not forgotten the haproxy stuff. I'm bootstrapping openstack-ansible to the point where it fails, then I'll turn things off selectively until I have a minimal config that still breaks.13:05
*** DRoBeR has joined #openstack-ansible13:06
jrossernowster: no worries :)13:06
*** ahosam has joined #openstack-ansible13:10
*** priteau has joined #openstack-ansible13:10
*** ahosam has quit IRC13:10
mcela1jrosser: i have tried that configuration before. I can correctly create a provider network of type "vlan" with a given segmentation ID, but when i try to ping the gateway either from an instance, or from the network namespace of the router, the ARP responses of the gateway do not make it back to the instance/router namespace.13:11
*** ahosam has joined #openstack-ansible13:11
*** strattao has joined #openstack-ansible13:12
mcela1what is even better is that if i do a tcpdump of the bonded interface, i can see ARP requests from the instance/router namespace leaving (and being correctly vlan tagged), but when they come back from the gateway, they are "dropped" and i can't find why this happens13:12
*** ahosam has quit IRC13:15
jamesdentonmcela1 can you share the output of 'brctl show' on the controller/network/compute nodes and the output of 'openstack network show' for the network involved? Let's make https://pasted.tech/pastes a thing.13:15
mcela1absolutely13:16
openstackgerritMerged openstack/openstack-ansible master: [trivial] Point to the new openstack-discuss ML  https://review.openstack.org/61875813:17
mcela1jamesdenton:  The output of brctl show is as follows: https://pasted.tech/pastes/7a6a073a08cc8e522143ca9d381eeb3b44715b8213:17
jamesdentonThanks.13:18
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-ops master: MNAIO: Add ability to deploy Ceph  https://review.openstack.org/60733813:19
jamesdentonmcela1 Do you happen to know if you have a native vlan set on the bond0 interfaces on the switch/13:22
jamesdentonand have you confirmed the interfaces are configured as a trunk and not access port13:22
openstackgerritMerged openstack/openstack-ansible-lxc_hosts master: Add ansible architecture mapping for arm64  https://review.openstack.org/61850013:22
jamesdentonand what is vlan 1324?13:23
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-ops master: MNAIO: Add ability to deploy Ceph  https://review.openstack.org/60733813:27
dcdamienAre this kind of changes candidates for backport? https://review.openstack.org/#/c/617847/13:33
mcela1jamesdenton: as far as i know the native vlan is not the 1324 (the one i am trying to use for tenants). The interfaces are correctly configured, as traffic leaves tagged and we have routed access to them for the configured vlans. We suspect a bad configuration on interfaces, but have tried several reinstalls with different configurations and can't get it to work.13:35
jamesdentonmcela1 i noticed that bond0.1324 is in br-vxlan bridge, and vlan 1324 is also used for one of the neutron networks. That won't work and could be a source of trouble if that's the vlan you're working with13:36
jamesdentonyou'll have to either change the vlan used for vxlan overlay networks, or delete and recreate the neutron network and use a different vxlan id13:37
mcela1we will try to reinstall again with that configuration and will get back with results. Thank you very much!!13:41
openstackgerritMerged openstack/openstack-ansible-os_cloudkitty master: Configure cloudkitty-api to use uwsgi  https://review.openstack.org/56716813:41
jamesdentona reinstall may be scorching the earth here, but feel free and let us know!13:41
odyssey4meevrardjp jrosser wrt to https://review.openstack.org/613078 - one could debate efficiency versus explicit dependencies... I find that the 'efficient' approach of making a package install only once results in forgetting why it's there and it never being cleaned up when there is no longer a dep... versus rather making it explicitly install where it is required (in each role which requires it) so that the role is self sufficient and it13:43
odyssey4me also gets cleaned up if it's not required13:43
odyssey4methere are some exceptions, of course - that's why we have things like https://github.com/openstack/ansible-role-python_venv_build/commit/c54aa8117e79dd9b37b61189223eb4a08ad5e2a0 and may also need something in openstack_hosts13:44
jrossermy preference would be to make the lxc_hosts package list as small as possible13:45
odyssey4mejrosser agreed13:49
*** chkumar246 has joined #openstack-ansible13:49
openstackgerritGuilherme  Steinmuller Pimentel proposed openstack/openstack-ansible-os_placement master: [WIP] Create base files to install placement  https://review.openstack.org/61882013:50
*** andymccr has joined #openstack-ansible13:51
*** chkumar|ruck has quit IRC13:52
*** ansmith has joined #openstack-ansible13:56
openstackgerritMerged openstack/openstack-ansible-lxc_container_create master: Add ansible architecture mapping for arm64  https://review.openstack.org/61852313:57
*** savvas_ has joined #openstack-ansible13:58
evrardjpodyssey4me: I am fine with cleaning it up, and if necessary move it to openstack_hosts13:59
evrardjpjrosser: ^13:59
evrardjpI'd rather be explicit13:59
evrardjpand efficiency can come from openstack_hosts13:59
*** chkumar246 has quit IRC14:01
openstackgerritGuilherme  Steinmuller Pimentel proposed openstack/openstack-ansible-os_placement master: [WIP] Create base files to install placement  https://review.openstack.org/61882014:05
openstackgerritGuilherme  Steinmuller Pimentel proposed openstack/openstack-ansible-os_placement master: [WIP] Create base files to install placement  https://review.openstack.org/61882014:08
*** eumel8 has quit IRC14:11
openstackgerritGuilherme  Steinmuller Pimentel proposed openstack/openstack-ansible-os_placement master: [WIP] Create base files to install placement  https://review.openstack.org/61882014:27
*** strattao has quit IRC14:29
openstackgerritMerged openstack/openstack-ansible-os_horizon master: Add Neutron VPNaaS Dashboard to Horizon  https://review.openstack.org/61784714:37
*** DanyC has quit IRC14:39
*** DanyC has joined #openstack-ansible14:40
*** DanyC has quit IRC14:41
*** DanyC has joined #openstack-ansible14:43
noonedeadpunkCan you please check this cherry-pick? https://review.openstack.org/#/c/616988/14:46
*** DanyC has quit IRC14:47
openstackgerritMerged openstack/openstack-ansible-os_nova stable/rocky: Add libpython2.7 as a required package  https://review.openstack.org/61878314:52
*** nowster has quit IRC14:52
*** chkumar246 has joined #openstack-ansible14:53
evrardjpI generetate the bugtraige page, could people update today meeting's agenda14:55
evrardjpwow14:55
evrardjpI generated the bugtriage etherpad, could people have a look at our wiki page for meetings, to ensure the agenda is up to date? :D <-- more english14:56
*** jcosmao has joined #openstack-ansible14:59
*** udesale has joined #openstack-ansible15:05
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-os_nova master: Fix nova-lxd tests for bionic.  https://review.openstack.org/61630215:06
*** nowster has joined #openstack-ansible15:09
jrosserevrardjp: i really struggled with what to do with those nova-lxd vars - theres two different pool concepts and at least two bugs to work around......15:11
evrardjpjrosser: I think you did good, my only point for a negative vote was the inability for me to grok the contents of the reno :)15:12
evrardjpelse we can keep it that way (the test var part I mean)15:12
evrardjpjrosser: thanks for tackling this!15:12
arxcruzevrardjp: hey, regarding https://review.openstack.org/#/c/618738 did you saw my comments?15:12
arxcruzi really don't know how to do that only on os_tempest, since all os_tempest are configured on osa-tests and on os_tempest has only the templates15:13
jrosserevrardjp: once the upstream bugs in nova-lxd are ironed out those vars should not be needed in the test any more - that will make things more obvious15:14
evrardjparxcruz: I did not yet15:14
evrardjparxcruz: could we discuss this a little later, I am in a few meetings :)15:14
arxcruzevrardjp: sure15:15
evrardjpjrosser: oh great15:15
*** mcela1 has left #openstack-ansible15:17
openstackgerritMerged openstack/openstack-ansible-nspawn_hosts stable/rocky: Modify the network to ensure their more restrictive  https://review.openstack.org/61616115:22
*** jproulx has joined #openstack-ansible15:23
spotzBack in action if needed!15:24
evrardjparxcruz: https://github.com/openstack/openstack-ansible-os_neutron/blob/master/zuul.d/jobs.yaml#L16-L20 is an example for neutron. The var you're passing is then used for this: https://github.com/openstack/openstack-ansible-os_neutron/blob/master/tox.ini#L105-L11115:25
evrardjpofc it means you need to create your own override yml file to trigger a part of the code.15:26
odyssey4mespotz here are some backports which just need verification that they match master - https://review.openstack.org/#/q/topic:nspawn-rocky-fixes+status:open15:26
arxcruzevrardjp: ok, thanks, i'll take a look at this15:26
evrardjpThe idea behind that was to make sure developers can run the same things as gates15:26
odyssey4mehopefully with all those in, nspawn will be in a working state for rocky15:26
arxcruzodyssey4me: hey, what was the name of the guy working on the zuul api ?15:26
spotzodyssey4me: Ok I'll start going through them15:27
chkumar246arxcruz, do we want to run python-tempestconf and stackviz from rpm in centos-distro job?15:27
evrardjphowever, you're giving me ideas on how to change that -- we should probably consume said variables and template them for direct consumption. Deployers could just run the same kind of things.15:27
odyssey4mearxcruz hmm, my brain's only half here (due to 'flu) - maybe evrardjp will remember - he's from aus15:27
evrardjpbut for the moment, let's just do it this way15:27
evrardjpJosh?15:27
arxcruzchkumar246: eventually15:27
evrardjpJosh Hesketh?15:27
arxcruzevrardjp: some guy was doing some work/research to have a rest api in zuul to return the playbook and all variables for reproduce15:28
evrardjphttps://review.openstack.org/#/c/607077/15:28
evrardjparxcruz: ^15:28
odyssey4meevrardjp yep, that's the guy!15:28
odyssey4mejhesketh :)15:28
spotzodyssey4me: Did you start getting sick on Wednesday?15:28
evrardjpodyssey4me: I gave the link of the WIP, even better! ;)15:29
odyssey4mespotz not sure when, but it hit me hard on friday15:29
jrosserchkumar246: we may have a bit of tempest brok here in our barbican role.... is that an easy fix somewhere? - http://logs.openstack.org/36/614336/2/check/openstack-ansible-functional-distro_install-ubuntu-bionic/386a134/job-output.txt.gz#_2018-11-19_20_59_40_83289515:29
odyssey4meoh nice - I see he's got WIP for a runner too :) https://review.openstack.org/#/c/607082/115:29
spotzodyssey4me: I started a sore throat on Wednesday, then the nose on Thursday. evrardjp and kmarc came to the rescue with meds15:30
chkumar246jrosser, checking15:30
odyssey4mespotz yep, sore throat was how it started on fri - then full blown 'flu over the weekend for me15:30
spotzOSA flu!15:30
arxcruzevrardjp: i own you more jagermaster15:31
evrardjpown or owe?15:31
nowsteryuk! I developed "post nasal drip" after a heavy cold.15:31
nowsterAnd that wasn't "Conference Flu" either.15:31
evrardjpfolks you have to be sick BEFORE the conference, so you can give it better to your friends!15:32
evrardjp:p15:32
arxcruzevrardjp: next time you be in berlim, you have one shot! :)15:32
chkumar246jrosser, on puppet-side we came to know that ubuntu does not maintain tempest so it is better to run tempest from source15:33
jrosserchkumar246: actually i linked you a nv distro install job there, but they all look unhappy15:33
chkumar246jrosser, https://review.openstack.org/61551415:33
chkumar246jrosser, I am taking a look let me see how I can fix it15:34
openstackgerritArx Cruz proposed openstack/openstack-ansible-os_tempest master: Add centos-7 job with support to python-tempestconf  https://review.openstack.org/61902115:38
cloudnullmornings all15:40
mgariepymorning cloudnull15:40
cloudnullo/15:40
evrardjparxcruz: I forgot to say that those jobs should be added in jobs.yaml, and you should add said job name into projects :)15:41
*** dave-mccowan has joined #openstack-ansible15:41
arxcruzevrardjp: yes, i just realize that15:41
arxcruzworking on the patch :)15:41
*** cshen has quit IRC15:43
*** hamzaachi has joined #openstack-ansible15:43
openstackgerritArx Cruz proposed openstack/openstack-ansible-os_tempest master: Add centos-7 job with support to python-tempestconf  https://review.openstack.org/61902115:44
evrardjparxcruz: if you put [WIP] in the title, I won't bother you :p15:45
openstackgerritChandan Kumar proposed openstack/openstack-ansible-os_tempest master: Added task to list tempest tests  https://review.openstack.org/61902415:45
openstackgerritArx Cruz proposed openstack/openstack-ansible-os_tempest master: [WIP] - Add centos-7 job with support to python-tempestconf  https://review.openstack.org/61902115:45
chkumar246jrosser, ^^ a proactive approach to find which went wrong here15:45
evrardjpchkumar246 arxcruz thanks for the work done in tempest, I haven't said it enough! :p15:46
arxcruzevrardjp: now you own me a jagermaister15:47
arxcruzwhatever it writes15:47
mnasercloudnull, DimGR, d34dh0r53, hughsaunders, b3rnard0, palendae, odyssey4me, serverascode, rromans, erikmwilson, mancdaz, _shaps_, BjoernT, claco, echiu, dstanek, jwagner, ayoung, prometheanfire, evrardjp, arbrandes, scarlisle, luckyinva, ntt, javeriak, spotz, vdo, jmccrory, alextricity25, jasondotstar, admin0, michaelgugino, ametts, bgmccollum, darrenc, JRobinson__, colinmcnamara, thorst, adreznec, eil397,15:53
mnaserqwang,nishpatwa_, cathrichardson, drifterza, hwoarang, cshen, ullbeking, mnaser, nicolasbock, jrosser, cjloader, antonym, dcdamien15:53
evrardjpjagermeister :)15:53
*** spatel has joined #openstack-ansible15:53
mnaser7 minute warning :)15:53
spatelcloudnull: morning!15:53
evrardjpmnaser: thanks!15:53
chkumar246jrosser, the issue with the above failed part is tempest failed to find that tests15:53
spatelis meeting going to start?15:53
* cloudnull goes to get coffee15:53
spatelso i don't spam you ...15:53
chkumar246jrosser, above patch will clearly say whether it is discovered or not15:54
spateli have this question if its easy to answer for you http://paste.openstack.org/show/735699/15:54
chkumar246jrosser, can we add a depends on your patch?15:54
chkumar246with my patch15:54
cloudnullspatel - try to stop/start_app else restart the rabbitmq container.15:56
spatelbut i wonder how everything is working...15:56
spateleven rabbitmq is broken15:57
jrosserchkumar246: sure15:57
cloudnullspatel its likely only broken on that node15:57
spateli am seeing same behavior on all 3 nodes15:58
spatelsame error15:58
cloudnulloh then everything is not working :)15:58
cloudnullcan you restart the rabbitmq process ?15:58
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-os_barbican master: Remove unnecessary package install duplication  https://review.openstack.org/61433615:59
spatelhmm! i think this is the behavior from beginning and i never paid attention...15:59
spateli can build machine launch etc..15:59
spatellet me restart R16:00
openstackgerritMerged openstack/openstack-ansible stable/rocky: Use loop_var name in when clause  https://review.openstack.org/61659516:00
mnaserdamn woops16:00
mnaseri always do this16:00
mnaser#startmeeting openstack_ansible_meeting16:00
openstackMeeting started Tue Nov 20 16:00:59 2018 UTC and is due to finish in 60 minutes.  The chair is mnaser. Information about MeetBot at http://wiki.debian.org/MeetBot.16:01
spateljust checked all 3 has same error16:01
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.16:01
*** openstack changes topic to " (Meeting topic: openstack_ansible_meeting)"16:01
openstackThe meeting name has been set to 'openstack_ansible_meeting'16:01
mnaser#topic roll call16:01
mnasero/16:01
*** openstack changes topic to "roll call (Meeting topic: openstack_ansible_meeting)"16:01
spatelcloudnull: i will talk to you after this meeting :)16:01
mnaserlaggy bot16:01
mnasero/16:01
cloudnullo/16:01
jrossero/16:01
mnaserquiet one16:02
evrardjpo/16:02
mnaserokay so lets get started16:02
mnaser#topic what happened since last meeting16:02
*** openstack changes topic to "what happened since last meeting (Meeting topic: openstack_ansible_meeting)"16:02
mnasernot much, i think.  we were mostly all having a great time in berlin :) i had a really good deployemnt tool feedback session16:02
mnaseri had a few action items that i carried out16:02
mnaseralso, project on-boarding and project update were amazing16:02
mnaserclearly there is an amazing amonut of interest in OSA, so let's all keep up on that awesome work :D super motivating16:03
chkumar246arxcruz, thanks for suggestion it makes sense, I will update once dependent patch finishes the output16:03
mnaserother things: we're working on os_placement and hopefully we'll land that eventually16:04
mnaserit's cool because we're kinda one of the leaders in that so it would be nice for OSA to be "FIRST!"16:04
*** savvas_ has quit IRC16:04
mnaserother than that, jrosser brought up arm64 so maybe want to expand on that16:04
cloudnull++16:04
jrosserso, out of mostly curiosity i had a go on the new arm64 ci nodes16:05
odyssey4me:)16:05
jrosserand then entirely coincidentally nowster showed up here in the channel trying to do it for real16:05
mnaserjrosser: if you need any support on arm stuff, i can get people your way :)16:05
jrosserso we've been just poking at that a bit and seeing how far it gets16:05
jrossertheres a very obvious and now fixed bug in haproxy16:06
mnaserif you are going to be doing some work on that, i'd love to make an introduction to massimo who leads a lot of the open source / openstack stuff and he is a really really useful asset in terms of maybe even getting you resources to accomlpish the things you need16:06
jrosserthat would be great16:06
mnaserwonderful, ill send that soon16:06
jrosserthe other thing is that the performance of the arm64 nodes isnt great, so we need someone to help take a look at that16:06
jrosseri suspect the storage is slow16:07
mnaserperfect, he'll actually be able to help with all that stuff16:07
evrardjpthere are other ppl you might want to reach, like linaro too, should you require some help16:07
jrosserfor anyone who is interested here is the test patch https://review.openstack.org/#/c/618305/16:08
mnaseralso, i might be able to get you some.. faster resources for local testing16:08
mnaserwill follow up more in that email16:08
jrossercool :)16:08
mnaseranything else from last week?16:09
guilhermesp late o/16:09
mnaser:)16:09
guilhermespso I think you noticed that Ive been working on os_placement16:10
guilhermesphere's the pr https://review.openstack.org/#/c/618820/16:10
guilhermespI'm still building the basics of the role, but I still need to create the tests16:10
guilhermespseems that the placement api works under apache + uswgi16:11
guilhermespmnaser: mentioned that there is also red hat packages for distro install16:11
guilhermespis that right?16:11
mnaseryes, rdo apparently publishment placement packages already16:11
*** udesale has quit IRC16:11
guilhermespat the moment I'm focusing in source install16:12
guilhermespbut good to know16:12
mnaseryep lets get that and we can improve/iterate16:12
mnaseranything else or shall we move onto bug triage?16:12
cloudnulljust a quick note about the summit -- odyssey4me and I were in fhaas training session (which was great!), if any folks in the channel here were in that session and want to follow up on some of the things we talked about in the all day session please reach out.16:12
mnasers/fhaas/fghaas/ you're welcome :p16:13
cloudnull^16:13
cloudnull:)16:13
mnaser#topic bug triage16:13
*** openstack changes topic to "bug triage (Meeting topic: openstack_ansible_meeting)"16:13
mnaser#link https://bugs.launchpad.net/openstack-ansible/+bug/180420416:13
openstackLaunchpad bug 1804204 in openstack-ansible "Quickstart: AIO in openstack-ansible ubuntu instructions" [Undecided,New]16:13
mnaseryep16:14
mnaserthats confirmed16:14
*** hamzy has quit IRC16:14
evrardjpvalid16:14
evrardjplow-hanging-fruit16:14
evrardjplow ?16:14
mnaseri'd say medium considering .. its broken16:15
cloudnullsomething like this came up in the project onboarding ++ Id say medium16:15
mnaserasked if paul can push up a patch16:15
mnaserif not we can figure things out :)16:16
mnaser#link https://bugs.launchpad.net/openstack-ansible/+bug/180344416:16
openstackLaunchpad bug 1803444 in openstack-ansible "gnocchi playbook failing on 18.0.0" [Undecided,New]16:16
*** chkumar246 has quit IRC16:16
mnaserinclude vs import_playbook black magic16:16
mnaserdoes anyone knows why the above might occur?16:16
mnaserseems related to ceph_client too16:17
evrardjpchecking16:18
mnaserwait is it because the include is include-ing it regardless?16:18
mnaserbecause it looks like that deployer is using ceph16:18
mnaserso that block should be entirely ignored16:18
evrardjpI tried to avoid that playbook for a while :p16:18
* cloudnull reading 16:18
noonedeadpunkhm, never faced with this one...16:18
nowstermnaser: Just to let you know that I'm working with "Works On ARM" too (ie. Massimo).16:19
mnasernowster: oh awesome!16:19
cloudnullfatal: [infra1_gnocchi_container-7ed445f6]: FAILED! => {"msg": "No file was found when using with_first_found. Use the 'skip: true' option to allow this task to be skipped if no files are found"}16:19
evrardjpwe should probably adapt the play to only run the appropriate role if a grouping of host was conditionally successfull16:19
mnaserthat fits perfectly then :)16:19
cloudnullis generally an issue with facts16:19
cloudnulllike the OS was not found or there were no valid facts and ansible didnt regather16:19
evrardjpcloudnull: I agree on the ceph client part16:19
mnaseryeah so i think no facts were gathered for that because no tasks actually ran on the gnocchi container16:20
mnaserat least in that specific section that it was breaking on16:20
mnaserwhich was the setup-things workaround16:20
cloudnullas for the "when" in the import playbook.16:20
cloudnullim not sure that actually works any longer.16:20
evrardjpwe should maybe ask for specific runs16:20
evrardjpcloudnull: yeah my point exactly16:21
evrardjpcloudnull: well tbh I am not sure WHEN it worked :p16:21
cloudnullIDK if import_playbook allows for vars to be passed through16:21
evrardjpdouble pun inside!16:21
mnasercause an import isn't really meant to be conditional right?16:21
evrardjpcloudnull: it was never intended for that16:21
cloudnull++16:21
evrardjpmnaser: right16:21
cloudnullit worked with include because we took advantage of the overloaded include system16:21
evrardjpyeah16:22
mnaseris there include_playbook ..?16:22
evrardjpnope16:22
evrardjponly import16:22
mnaserso what we need to do is move that logic into os-gnocchi right?16:22
evrardjpbut as I said16:22
cloudnullinclude will load whatever it finds while import is more like python.16:22
evrardjpwe can move some part into the play16:22
evrardjpand include_role16:22
cloudnull++16:22
evrardjpor the meta to end the playbook early16:23
evrardjp(but I'd rather avoid that one)16:23
evrardjpfor the sake of the bug, I'd like to see a real run of gnocchi16:23
mnaseris there a quick 2 minute explanation16:23
mnaseras to why this even happens?16:23
evrardjpto see if it happens again16:23
evrardjpI mean of the gnocchi playbook16:24
mnaseralso noonedeadpunk mentions he hasnt ran into that.. do you use ceph?16:24
mnaserdo we wanna dig into this again next week?16:25
evrardjpmnaser: that sounds reasonable16:26
noonedeadpunkmnaser: yep, I'm running ceph16:26
francoishi guys, I'm the author of this bug report and am available if you need any other info :)16:26
evrardjpnoonedeadpunk: with gnocchi?16:26
evrardjpfrancois: oh great!16:26
evrardjpperfect!16:26
openstackgerritKevin Carter (cloudnull) proposed openstack/openstack-ansible-ops master: Correct clustered gating job  https://review.openstack.org/61903316:26
evrardjpfrancois: could you run just that gnocchi play again?16:26
evrardjpopenstack-ansible os-gnocchi-install.yml16:27
evrardjpit should not fail :p16:27
francoisevrardjp: ok16:27
noonedeadpunkyep. but, I'm not running swift...16:27
evrardjpif you put the log somewhere that would be nice16:27
evrardjpnoonedeadpunk: that's the deal :)16:27
noonedeadpunkAnd now I'm wondering, how I get it installed at all...16:27
evrardjpfrancois: because if it repeats itself with that case, it's not a facts problem, but a playbook problem16:28
noonedeadpunkoh, we're including gnocchi role twice...16:29
*** aedc has quit IRC16:29
mnaserfrancois: can you please update that bug with the info16:29
mnaserand we'll cotninue16:29
mnaser#link https://bugs.launchpad.net/openstack-ansible/+bug/180338216:29
openstackLaunchpad bug 1803382 in openstack-ansible "unbound-client.yml failure depending on TERMCAP envvar content" [Undecided,New]16:29
*** aedc has joined #openstack-ansible16:29
francoismnaser: yeah, will do that16:29
mnaseri ran into that bug too16:29
mnaserit happened when i was running something in screen16:30
mnaseri had *no* idea why it was happening16:30
francoisalso mine :)16:30
mnaserbut it didn't ahppen anymore in screen16:30
francoisthis one looks pretty weird!16:30
mnaseri have no idea why it honestly happens16:30
evrardjpI was told that jrosser had the same issue16:30
evrardjpwe can probably say confirmed then16:31
francoisan easy workaround is simply deleting this variable from /etc/openstack_deploy/ansible_facts/localhost16:31
dcdamienI have the same issue too16:31
evrardjpwe should unset said thing in the connection plugin16:31
francoisbut I'm quite interested in how it could happen :)16:31
mnaseri will put this as confirmed/high16:31
mnaserrunning things in screen is kinda.. important.16:31
evrardjptmux?16:32
evrardjp:p16:32
noonedeadpunk+16:32
evrardjpnever had an issue with tmux :p16:32
evrardjphahaha16:32
evrardjpanyway16:32
guilhermespbyobu rules16:32
evrardjpyeah16:32
cloudnulltmux for life!16:32
evrardjpimportant16:32
mnaserfrancois: do you have sometime to dig a little bit into that one?16:32
guilhermespbyobu = tmux lover too16:32
evrardjpI digged through a rabbit hole long ago with a rabbitmq issue of the env var that was passed16:32
evrardjpbut we changed our connection plugin now, and we are not relying on the same behaviours anymore16:33
evrardjpso you're hitting an issue in ansible itself16:33
evrardjpprobably16:33
*** dcdamien has quit IRC16:33
evrardjpworth checking in the bugs there too16:33
mnaseri put the workaround but yeah, i have no idea, we cant solve it now, just triaging :)16:33
mnaser#link https://bugs.launchpad.net/openstack-ansible/+bug/180334716:34
openstackLaunchpad bug 1803347 in openstack-ansible "Documentation mentions requirement of bridges on compute nodes" [Undecided,New] - Assigned to Mohammed Naser (mnaser)16:34
evrardjpmnaser: yeah lgtm16:34
mnaserhah i remember this16:34
noonedeadpunkYep, my question, sorry.16:34
evrardjp:)16:34
mnaserconfirmed low16:34
mnaser#link https://bugs.launchpad.net/openstack-ansible/+bug/180301916:34
openstackLaunchpad bug 1803019 in openstack-ansible "Nova generates versioned_notifications but no one consume" [Undecided,New] - Assigned to KimMinsik (for-beatitudo)16:34
noonedeadpunkYeah, it's problem but more about Nova16:35
mnaserin progress medium16:35
noonedeadpunkI know the workaround to it16:35
noonedeadpunkah, ok16:35
*** DRoBeR has quit IRC16:36
evrardjpmaybe documenting the workaround is good?16:36
noonedeadpunkit's already a patch for it present...16:37
evrardjpyeah we can talk about it in the review if you prefer16:37
noonedeadpunknot at all. But it's exactly my way of resolving this.16:38
noonedeadpunkbut I've just overrided nova.conf with extra params16:38
mnaseryep, its nice the patch has 'native' feature so itd be good to review it16:38
mnaser#link https://bugs.launchpad.net/openstack-ansible/+bug/180274216:39
openstackLaunchpad bug 1802742 in openstack-ansible "parameterized nova placement nginx error log level" [Undecided,New]16:39
mnaserthat seems like wishlist16:39
noonedeadpunkand for ocata...16:40
mnaserit probably still applies16:40
mnaserwait it doesnt16:40
mnaserwe dropped nginx right?!16:40
noonedeadpunkI think it's just plain uwsgi instead..16:40
mnaseryeah16:40
mnaseryou're right16:41
mnasermeh confiremd/wishlist16:41
mnaser#link https://bugs.launchpad.net/openstack-ansible/+bug/180246616:41
openstackLaunchpad bug 1802466 in openstack-ansible "keystone cannot generate keystone.log file" [Undecided,New] - Assigned to KimMinsik (for-beatitudo)16:41
evrardjpnot everywhere, but for nova placement I don't know16:42
mnaserplacement is uwsgi16:42
mnaseralso thats cool, we got a fix for that too16:42
mnaser+2d it16:42
mnaser#link https://bugs.launchpad.net/openstack-ansible/+bug/180219516:43
openstackLaunchpad bug 1802195 in openstack-ansible "ceph-rgw installation/configuration lacks idempotency, configuration changes break the setup" [Undecided,New]16:43
evrardjpfor the last one, I am worrying about container non container16:43
fghaasevrardjp: how so?16:44
evrardjpfghaas: well it was for the keystone cannot generate, which is not the last one, I am a little slow on typing :p16:44
evrardjpsorry for being in 2 meetings at the same time :p16:45
fghaasevrardjp: ack, thanks for clarifying :)16:45
mnaserfghaas: is that an osa issue or ceph-ansible?16:45
noonedeadpunkbtw, about keystone - I don't have /var/log/keystone at all, as all log files in journalctl16:46
fghaasit's an osa issue in that osa pulls in a state of ceph-ansible that was never meant for production.16:46
fghaasI hope the commit messages explain that fairly well16:46
*** hamzaachi has quit IRC16:47
fghaasI think that bug is only in New because the actual thing to be fixed is in Rocky, and the fix in master is Related16:47
mnaserfghaas: so does that mean its resolved ?16:47
fghaasI certainly like to think so, and haven't heard anyone scream otherwise16:48
mnaserok cool16:49
evrardjpnoonedeadpunk: that sounds more likely this is why I am concered of said patch.16:49
mnaser#link https://bugs.launchpad.net/openstack-ansible/+bug/180213216:49
openstackLaunchpad bug 1802132 in openstack-ansible "OpenStack Ansible dashboard not able to log in with admin credentials" [Undecided,New]16:49
mnasersounds like a faield deploy16:50
mnaserasked to rerun horizon install16:50
*** gkadam has quit IRC16:50
mnaserwe'll review next week16:50
mnaser#link https://bugs.launchpad.net/openstack-ansible/+bug/180207016:50
openstackLaunchpad bug 1802070 in openstack-ansible "Add resource_filter field in cinder.conf for non-admin user can retrieve volume & snapshot volume" [Undecided,New] - Assigned to KimMinsik (for-beatitudo)16:50
mnaserhttps://review.openstack.org/#/c/616113/16:51
mnaseralready a patch16:51
mnaserlets run this by the cinder team16:51
mnaserill do that16:51
mnaserjust to kinda figure out what is going on16:51
mnaserbut it should be in progress16:51
mnaser#link https://bugs.launchpad.net/openstack-ansible/+bug/180165616:52
openstackLaunchpad bug 1801656 in openstack-ansible "fix tox python3 overrides" [Undecided,New]16:52
evrardjpthanks for dealing with it mnaser16:53
mnaseri guess we can keep that one16:53
mnaserput it as confirmed16:53
mnaserto track our py3 stuff16:53
evrardjpok16:53
evrardjpfine for me16:53
evrardjpit's almost all done anyway, isn't it?16:53
mnaserit is mostly done i think16:53
noonedeadpunkreturning back to cinder, I thought, that policies resolves mentioned problem..16:53
mnaser#topic open discussion16:54
*** openstack changes topic to "open discussion (Meeting topic: openstack_ansible_meeting)"16:54
mnaserwe can talk about the bugs outside the meeting, but it does seem like it resolves things, but best to run by cinder team16:54
evrardjpI don't have any topic for open discussion16:54
spotzjungleboyj and the rest of the team should be home by now16:54
mnaserwhile we have a few minutes, got anything on side of arxcruz re: os-tempest?16:55
evrardjpexcept I hoped you had a nice trip back from the summit16:55
evrardjpoh yeah probably worth checking that direction16:55
arxcruzmnaser: we are starting to add a job to test python-tempestconf16:55
jungleboyjYep.  I am.  :-)16:55
arxcruzand from there, we will add new functionalities on python-tempestconf, like override settings, rpm package, etc16:55
jungleboyjWhat is the problem?16:56
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-os_keystone master: Add libpython2.7 as a required package  https://review.openstack.org/61904016:56
guilhermespabout os_placement, I'd really appreciate reviews and suggestions to finish the work https://review.openstack.org/#/c/618820/ :)16:56
mnasercool, thats awesome arxcruz16:56
mnaserplease send reviews16:56
arxcruzmnaser: yup, evrardjp is already reviewing it :)16:57
mnasersweet16:57
mnaser#endmeeting16:59
*** openstack changes topic to "Launchpad: https://launchpad.net/openstack-ansible || Weekly Meetings: https://wiki.openstack.org/wiki/Meetings/openstack-ansible || Review Dashboard: http://bit.ly/2xA1eZC"16:59
openstackMeeting ended Tue Nov 20 16:59:36 2018 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)16:59
openstackMinutes:        http://eavesdrop.openstack.org/meetings/openstack_ansible_meeting/2018/openstack_ansible_meeting.2018-11-20-16.00.html16:59
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/openstack_ansible_meeting/2018/openstack_ansible_meeting.2018-11-20-16.00.txt16:59
mnaserthanks everyone16:59
openstackLog:            http://eavesdrop.openstack.org/meetings/openstack_ansible_meeting/2018/openstack_ansible_meeting.2018-11-20-16.00.log.html16:59
jungleboyjmnaser:  That bug looks like an issue in Cinder.  The policies in code should handle doing that unless the file is available.16:59
spatelcloudnull: do you have min? I am seeing very strange issue with rabbitmq  http://paste.openstack.org/show/735821/17:07
spatelif you look into this output you will see "Error: unable to connect to node 'rabbit@ostack-controller-01': nodedown"17:07
spatelits trying to connect wrong node or hostname..17:07
spatelwho set that hostname? does openstack-ansible doing it?17:08
spatelit seems bug somewhere...17:08
spatelrabbitmq-server is running... but cluster not functional17:09
*** hamzaachi has joined #openstack-ansible17:10
mnaserjungleboyj: it looks like this isnt policy in code but something about resources17:11
jungleboyjmnaser:  Ok.  I have added your bug to our meeting agenda tomorrow to see if we can get some input on our end.  It seems strange to me.17:12
mnaserjungleboyj: thanks :)17:14
jungleboyjmnaser:  No problem.17:14
*** fghaas has quit IRC17:15
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-os_keystone master: Add libpython2.7 as a required package  https://review.openstack.org/61904017:22
noonedeadpunkguys, does anyone uses tooz coordination for cinder?17:23
openstackgerritKaio Kassiano Moura Oliveira proposed openstack/openstack-ansible-os_monasca-ui master: Retire openstack-ansible-os_monasca-ui  https://review.openstack.org/61720417:23
evrardjpI am done for the day17:26
evrardjpttyl everyone!17:26
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-lxc_hosts master: Minimise distro packages installed into the lxc image  https://review.openstack.org/61307817:27
spatelFolks, any idea how to fix hostname issue in rabbitmq17:27
*** tobias-urdin is now known as tobias-urdin_afk17:27
noonedeadpunkspatel if you've changed hostname, you'll probably have to re-add host to the cluster17:28
noonedeadpunkbut I'm not a big expert to be honest17:28
spatelI didn't change anything... this is all build by OSA17:28
spatelin LAB and Production same behavior.. connecting wrong hostname..17:29
*** DanyC has joined #openstack-ansible17:29
spateli am trying forget but still talking to wrong name17:30
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-ops master: MNAIO: Add ability to deploy Ceph  https://review.openstack.org/60733817:30
spatelrabbitmqctl forget_cluster_node rabbit@ostack-controller-0117:30
noonedeadpunkOh, found your paste. It seems, that rabbit is down in your container. Are you sure, that it was installed inside container? As it's name looks like it was installed bare metal...17:31
*** DanyC_ has joined #openstack-ansible17:31
*** DanyC has quit IRC17:31
spatelI have hack /etc/hosts file and lie to it about hostname and now i am seeing this error http://paste.openstack.org/show/735824/17:32
spatelI think i should open ticket.. it seems BUG17:32
noonedeadpunkrabbitmqctl won't work if rabbitmq service is down.17:32
spatelrabbitmq-server.service is running..17:33
spatelonly cluster status throwing error.. its not letting me stop/start_app also17:33
spatelI have 200 node production and everything working.. so far.. but just today i found this issue.. its not cluster correctly...17:34
spateli wonder my cluster only talking to single node and not looking into cluster17:34
noonedeadpunkThis is how rabbitmqctl works. It asks only localhost node. If you wish to ask some another member of the cluster, rabbitmqadmin should do the trick17:35
odyssey4meTime for me to head out before I feel any worse. Night all!17:36
*** shardy has quit IRC17:36
*** cshen has joined #openstack-ansible17:37
*** lbragstad has quit IRC17:38
*** lbragstad has joined #openstack-ansible17:41
noonedeadpunkOr maybe I'm mistaken...17:43
*** spatel has quit IRC17:50
*** spatel has joined #openstack-ansible18:03
spatelnoonedeadpunk:  i didn't get it what you trying to say18:06
*** cshen has quit IRC18:07
*** cshen has joined #openstack-ansible18:08
arxcruzevrardjp: what am i doing wrong here? https://review.openstack.org/#/c/619021/3/zuul.d/jobs.yaml this vars isn't being set on the job... python-tempestconf isn't being invoqued18:15
spatelI have summit bug: https://bugs.launchpad.net/openstack-ansible/+bug/180428318:17
openstackLaunchpad bug 1804283 in openstack-ansible "Rabbitmq Error: unable to connect to node 'rabbit@ostack-controller-01': nodedown" [Undecided,New]18:17
*** cshen has quit IRC18:17
openstackgerritMerged openstack/openstack-ansible-nspawn_hosts stable/rocky: Correct host only routes  https://review.openstack.org/61616218:18
*** rpittau has quit IRC18:27
*** DanyC_ has quit IRC18:27
*** DanyC has joined #openstack-ansible18:27
*** czunker has joined #openstack-ansible18:32
spatelThis is scary!!! https://www.techsfo.com/blog/2013/06/rabbitmq-breaks-when-you-rename-hostname/18:33
spatelits saying you need to re-install rabbitmq to change hostname.. damn it18:33
openstackgerritDmitriy Rabotjagov (noonedeadpunk) proposed openstack/openstack-ansible-os_horizon master: Implements masakari-dashboard to be installed in horizon  https://review.openstack.org/61905818:37
*** tobias-urdin_afk is now known as tobias-urdin18:38
openstackgerritAndy Smith proposed openstack/openstack-ansible master: Add qdrouterd role for oslo.messaging backend deployment  https://review.openstack.org/57026118:39
*** cshen has joined #openstack-ansible18:40
*** cshen has quit IRC18:45
*** fghaas has joined #openstack-ansible18:46
*** pcaruana has quit IRC18:47
spatelOh! wait...  this is interesting http://paste.openstack.org/show/735829/18:55
spatelcloudnull: noonedeadpunk ^^18:55
spatelnoonedeadpunk: ^^18:55
spatelit works but with -n option18:55
*** cshen has joined #openstack-ansible18:56
cloudnullspatel so the hostname was changed?18:56
*** strattao has joined #openstack-ansible18:56
spatelno18:56
*** fghaas has quit IRC18:56
spateli didn't touch anything...18:57
spatelrabbitmqctl cluster_status  <-- throwing error but18:57
spatelrabbitmqctl -n rabbit@ostack-controller-01-rabbit-mq-container-1bf6ede2 cluster_status  <-- is working18:57
cloudnullso then the rabbitmq custler is running ok however you have to pass the node-name flag in order to query it ?18:57
spatelyes..18:58
spatelbut why do i need to do that?18:58
cloudnulland rabbitmq is running in a container ?18:58
spatelyes in LXC18:58
cloudnullI assume the failing node is "ostack-controller-01"18:58
noonedeadpunkStrange thing is about the hostname, which rabbitmqctl is trying to ask by default...18:58
cloudnullI assume "ostack-controller-01" is the name set in /etc/hosts ?18:59
cloudnulland /etc/hostname ?18:59
spatelostack-controller-01 <- is my host node18:59
cloudnullcan you also paste the output of `hostnamectl` ?18:59
spatelhttp://paste.openstack.org/show/735830/18:59
spatelhostnamectl ->  http://paste.openstack.org/show/735831/19:00
openstackgerritDmitriy Rabotjagov (noonedeadpunk) proposed openstack/openstack-ansible master: Added masakari-dashboard repo to openstack_services  https://review.openstack.org/61906219:01
cloudnullit looks like the rabbitmqctl command thinks its hostname is "ostack-controller-01"19:01
cloudnullwhat about the files in /etc/rabbitmq ?19:01
-spatel- [root@ostack-controller-01-rabbit-mq-container-1bf6ede2 rabbitmq]# grep ostack-controller-01 /etc/hosts19:01
-spatel- 192.168.100.31ostack-controller-0119:01
cloudnullis that hostname there?19:01
cloudnullspatel that hosts entry, is that the ip address of your container ?19:02
spatel /etc/rabbitmq/rabbitmq.config  doesn't have any hostname definition anywhere19:02
spatelcontainer ip is 192.168.100.3119:02
cloudnullwhich is set in /etc/hosts19:03
cloudnullcan you change the name in the `/etc/hosts` file to the container name ?19:03
spateldoing it..19:03
spateloh! its already there..19:05
-spatel- [root@ostack-controller-01-rabbit-mq-container-1bf6ede2 rabbitmq]# grep 192.168.100.31 /etc/hosts19:05
-spatel- #192.168.100.31ostack-controller-0119:05
-spatel- 192.168.100.31 ostack-controller-01-rabbit-mq-container-1bf6ede2.openstack.local ostack-controller-01-rabbit-mq-container-1bf6ede2 ostack-controller-01_rabbit_mq_container-1bf6ede219:05
cloudnulloh so its a duplicate entry19:05
*** cshen has quit IRC19:05
spatelsomehow rabbitmq hold that entry in DB somewhere so not leaving it..19:06
spatelcloudnull: ready for fun????19:08
-spatel- [root@ostack-controller-01-rabbit-mq-container-1bf6ede2 rabbitmq]# echo $HOSTNAME19:08
-spatel- ostack-controller-01.site5.vivox.com19:08
spatelwhat the heck is this?19:08
spatel$HOSTNAME19:08
cloudnull?19:08
spatelwhy echo $HOSTNAME giving wrong hostname on lxc container19:08
spatelit should be container hostname right?19:09
cloudnullit should .19:09
spatelwho set that up?19:09
spatellet me try and see19:09
cloudnullI have no idea19:09
spatelexport HOSTNAME=ostack-controller-01-rabbit-mq-container-1bf6ede219:10
spatelit working now :)19:10
cloudnullyay :)19:10
spatelsomething is wrong in OSA...19:10
cloudnullis "ostack-controller-01.site5.vivox.com" the FQDN on the physical host ?19:10
spateli deployed 3 cloud and all three has same issue..19:10
spatelyes ostack-controller-01 is physical host19:11
cloudnullmaybe those environment variables are being passed along into the lxc-attach command?19:11
noonedeadpunkspatel: You might have forwarded ENV from host to container while entered container19:11
cloudnullI know you can tell it to drop environment variable pass though.19:11
*** strattao has quit IRC19:12
*** fghaas has joined #openstack-ansible19:12
spatelThis is what i am doing19:12
spatellxc-attach -n ostack-controller-01_rabbit_mq_container-1bf6ede219:12
spatelwhere i should tell to not use host ENV?19:13
spatelI am using all default OSA.. i didn't customized anything so it must be hiden somewhere in OSA19:14
spatelhow about if OSA do following..19:15
spateladd NODENAME=rabbit@ostack-controller-01-rabbit-mq-container-1bf6ede2  in /etc/rabbitmq/rabbitmq-env.conf19:15
spatelthat should solve that issue no matter ENV is correct or wrong..19:16
cloudnullspatel try `lxc-attach --clear-env --name $CONTAINER`19:16
cloudnull^ we run that when running the playbooks19:17
spatelcloudnull: that dropped ENV and now i am seeing correct echo $HOSTNAME19:18
spateldo i need to add --clear-env everytime?19:19
spatelwhat is the permanent solution for that19:19
*** asettle has quit IRC19:20
*** fghaas has left #openstack-ansible19:24
cloudnullcool19:24
cloudnullIDK of a permanent way to pass that flag every time.19:25
*** hamzy has joined #openstack-ansible19:25
* cloudnull looking 19:25
openstackgerritManuel Buil proposed openstack/openstack-ansible-os_neutron master: Bug fix: Bridge br-provider missing in dvr  https://review.openstack.org/61907019:26
cloudnullspatel maybe something like https://pasted.tech/pastes/c1c75800c19d7caf6e073dbf141f9e9e88cc95e219:27
spatelthat is sweet!!!19:27
cloudnullyou'd need to write that into your bash profile19:27
cloudnullbut that would do it19:27
spatelin short default behavior is to pass ENV to container19:28
cloudnullit would seem so19:28
cloudnullive not seen that behavior on ubuntu / suse19:29
cloudnullhowever it would seem that if that env var is set ($HOSTNAME) its passed through19:29
spatelhmm! so look like one more CentOS bug19:30
cloudnullwe could add that alias to the lxc-hosts role19:32
cloudnullfor all os's which would help solve this in cent19:32
spatelit would be good so folks won't get confused and get panic like me :)  let me update BUG19:32
spatelcloudnull: here is the bug if you want to go ahead and put that fix in https://bugs.launchpad.net/openstack-ansible/+bug/180428319:34
openstackLaunchpad bug 1804283 in openstack-ansible "Rabbitmq Error: unable to connect to node 'rabbit@ostack-controller-01': nodedown" [Undecided,New]19:34
cloudnullspatel https://pasted.tech/pastes/e1b1e5e67f5e0283a6d56b04e2147c51a975409119:36
cloudnullthat was my global change19:37
spatelbeautiful!!19:37
spateli am pushing them in .bashrc19:37
*** cshen has joined #openstack-ansible19:39
openstackgerritDmitriy Rabotjagov (noonedeadpunk) proposed openstack/openstack-ansible-os_horizon master: Implements masakari-dashboard to be installed in horizon  https://review.openstack.org/61905819:44
jrossercloudnull: does that interfere with ansible environment: settings?19:47
openstackgerritGuilherme  Steinmuller Pimentel proposed openstack/openstack-ansible-os_placement master: [WIP] Create base files to install placement  https://review.openstack.org/61882019:50
cloudnulljrosser it doesnt look like it would19:55
cloudnullin the plugins repo where we calll lxc-attach we use the full path19:55
cloudnullso the alias shouldnt impact those tools19:55
cloudnullI guess it might be something that could interact with a shell or command when were not using the plugins/modules.19:56
cloudnullthough idk19:56
jrosserThinking about it those environment: settings on tasks are probably created in the ansible dropped python anyway....19:56
*** cshen has quit IRC20:14
cloudnull^ yea i think thats true20:15
*** aedc has quit IRC20:15
MiougeAfter upgrading keepalived from 1.2.19 to 1.2.24 (Ubuntu 16.04) I lost the VRRP IPs. It turns out that in v1.2.20 keepalived exits if the VRRP interface has no IP, which is the case for my br-public (see relese notes https://github.com/acassen/keepalived/blob/master/ChangeLog#L3800). Should we have an assert for that?20:15
*** cshen has joined #openstack-ansible20:15
*** dcdamien has joined #openstack-ansible20:16
*** aedc has joined #openstack-ansible20:26
*** ahosam has joined #openstack-ansible20:28
openstackgerritMaxime Guyot proposed openstack/openstack-ansible master: Add a check that keepalived interfaces have IPs  https://review.openstack.org/61909220:29
*** cshen has quit IRC20:29
*** cshen has joined #openstack-ansible20:31
*** mmercer has joined #openstack-ansible20:41
*** czunker has quit IRC20:49
*** priteau has quit IRC20:52
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-os_nova master: Fix nova-lxd tests for bionic.  https://review.openstack.org/61630220:56
*** hamzaachi has quit IRC21:21
*** ahosam has quit IRC21:23
*** ahosam has joined #openstack-ansible21:24
*** dave-mccowan has quit IRC21:28
*** ahosam has quit IRC21:29
*** hamzy has quit IRC21:42
*** DanyC has quit IRC22:06
*** dcdamien has quit IRC22:27
*** spatel has quit IRC22:30
*** dcdamien has joined #openstack-ansible22:36
*** strattao has joined #openstack-ansible22:37
*** strattao has quit IRC22:48
*** dcdamien has quit IRC22:54
*** cshen has quit IRC22:55
*** mugsie has quit IRC23:06
*** ivve has quit IRC23:08
*** aedc has quit IRC23:47
*** spatel has joined #openstack-ansible23:59

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!