Tuesday, 2019-05-21

*** joshualyle has joined #openstack-ansible00:05
joshualyleanyone have any idea why the playbook for the utility container would turn up "No file was found when using with_first_found" for "pip_install : Gather variables for each operating system" ?00:05
joshualyleit only does this on os-tempest and I don't have any tempest hosts defined00:06
joshualyleI honestly do not understand. I don't have tempest hosts defined. Why is OSA even trying to look inside of os-tempest if [u'(tempest_install | default(False)) | bool or (tempest_run | default(False)) | bool'] is false?00:07
joshualylewell running ansible all -m setup seemed to fix it? Idk what it was seeing as the OS before because it wasn't matching any of the include_vars ymls from pip_install but now it is? I have no idea what has changed00:13
openstackgerritMerged openstack/openstack-ansible-os_swift master: Updated from OpenStack Ansible Tests  https://review.opendev.org/65803900:14
*** sreejithp has joined #openstack-ansible00:18
*** sreejithp has quit IRC00:32
snadgeso i figure.. i'll try the 19 rc1 tag.. and now I get an error systemd_service create temp run dir00:34
snadge"'dict object' has no attribute 'interface'"00:34
snadgeback to trying the lastest branch master then i guess00:36
*** markvoelker has joined #openstack-ansible00:41
*** markvoelker has quit IRC00:45
snadgeit would help if i knew how to use git.. i want to track remotes/origin/stable/stein00:49
snadgeso git checkout remotes/origin/stable/stein ?00:49
snadgecan just omit the remotes/origin it seems00:53
openstackgerritLogan V proposed openstack/openstack-ansible stable/queens: Bump etcd role to v3 capable SHA  https://review.opendev.org/65986800:57
openstackgerritLogan V proposed openstack/openstack-ansible stable/queens: Zuul: Simplify the integrated test playbooks  https://review.opendev.org/66022400:57
openstackgerritLogan V proposed openstack/openstack-ansible stable/queens: Add Calico networking AIO scenario  https://review.opendev.org/65986900:57
snadgei got the same error above systemd_service create temp run dir  "'dict object' has no attribute 'interface'"01:25
snadgerunning scripts/bootstrap-aio.sh allowed it to progress past that and finish.. im not sure if running it twice is going to create other problems like duplicate configurations etc01:26
openstackgerritMerged openstack/openstack-ansible stable/rocky: Bump etcd role to v3 capable SHA  https://review.opendev.org/65971301:29
*** hwoarang has quit IRC02:04
*** hwoarang has joined #openstack-ansible02:05
*** sreejithp has joined #openstack-ansible02:13
*** sreejithp_ has joined #openstack-ansible02:15
*** nicolasbock has quit IRC02:16
*** sreejithp has quit IRC02:19
*** joshualyle has quit IRC02:38
*** joshualyle has joined #openstack-ansible02:39
*** ckonstanski has joined #openstack-ansible03:41
*** udesale has joined #openstack-ansible03:47
*** yolanda_ has quit IRC03:49
openstackgerrittonybrad proposed openstack/ansible-hardening master: Update requiment source  https://review.opendev.org/65732403:50
openstackgerrittonybrad proposed openstack/openstack-ansible-os_cloudkitty master: Change git to opendev  https://review.opendev.org/65720203:51
*** sreejithp_ has quit IRC04:03
*** zerick has quit IRC04:03
*** zerick has joined #openstack-ansible04:04
openstackgerritLogan V proposed openstack/openstack-ansible stable/queens: Add Calico networking AIO scenario  https://review.opendev.org/65986904:54
*** pcaruana has joined #openstack-ansible05:02
snadgean AIO build should start horizon right? .. not sure why i can't log in05:03
snadgethat part seems to be missing from the quickstart guide05:03
*** raukadah is now known as chandankumar05:06
*** pcaruana has quit IRC05:11
snadgeahh okay ansible has chosen the wrong interface to bind to .. which is probably configurable etc05:15
snadgeim tempted to just reorder the network interface.. ie.. host first then nat.. and presumably it just picks the first one :P05:17
mcardenAnyway, yes. A successful AIO build should let you log in to Horizon and clicky clicky on things.05:18
snadgeyeah i just need to change external_lb_vip_address in openstack_user_config.yml05:19
snadgeexcept im not sure which command to run to propagate that setting to the places it needs to go05:20
mcardenHeh. Back when I used to deploy AIOs all the time, I only ever fixed a missed config by nuking from orbit and rolling the dice again.05:22
snadgethats what im doing yeah.. i saved a snapshot immediately prior to its deployment for that reason05:25
snadgesuper handy05:25
*** ivve has quit IRC05:34
*** radeks_ has joined #openstack-ansible05:36
*** udesale has quit IRC05:44
*** udesale has joined #openstack-ansible05:44
*** miloa has joined #openstack-ansible06:02
jrossersnadge: in theory just the haproxy playbook for the external vip, but beware that all of the services put entries in the keystone catalogue for their endpoints, so unfortunately that is something that really needs to be right from the outset06:05
jrosserSo it’s either re-run haproxy + setup_openstack, or rewind to a snapshot and go again06:07
*** cshen has joined #openstack-ansible06:32
*** markvoelker has joined #openstack-ansible06:36
*** ivve has joined #openstack-ansible06:43
*** starborn has joined #openstack-ansible06:47
openstackgerritJean-Philippe Evrard proposed openstack/openstack-ansible-os_masakari master: Add integrated coverage for metal jobs  https://review.opendev.org/66026006:50
openstackgerritJean-Philippe Evrard proposed openstack/openstack-ansible-os_masakari master: Use the dynamic aio scenario  https://review.opendev.org/65844506:52
*** luksky has joined #openstack-ansible07:00
openstackgerritDirk Mueller proposed openstack/openstack-ansible-galera_server master: Update package list for openSUSE Leap 15.0  https://review.opendev.org/66020407:00
openstackgerritDirk Mueller proposed openstack/openstack-ansible-galera_server master: Switch to openSUSE Leap 15 for functional testing  https://review.opendev.org/66020807:00
*** sreejithp has joined #openstack-ansible07:03
*** trident has quit IRC07:03
*** trident has joined #openstack-ansible07:05
*** sreejithp has quit IRC07:07
*** markvoelker has quit IRC07:09
*** kopecmartin|off is now known as kopecmartin07:12
openstackgerritJean-Philippe Evrard proposed openstack/openstack-ansible-os_mistral master: Use the dynamic aio scenario  https://review.opendev.org/65844807:15
openstackgerritJean-Philippe Evrard proposed openstack/openstack-ansible-os_mistral master: Use the dynamic aio scenario  https://review.opendev.org/65844807:15
openstackgerritJean-Philippe Evrard proposed openstack/openstack-ansible-os_mistral master: Add meta to mistral  https://review.opendev.org/66026507:21
*** pcaruana has joined #openstack-ansible07:21
openstackgerritChandan Kumar (raukadah) proposed openstack/openstack-ansible-lxc_hosts master: Install python-httplib2 on lxc-hosts from epel repo  https://review.opendev.org/66026607:21
openstackgerritChandan Kumar (raukadah) proposed openstack/openstack-ansible-os_tempest master: Update openstack.org -> opendev.org  https://review.opendev.org/65494207:22
evrardjpchandankumar: where is this python-httplib2 package used?07:23
evrardjpis it only in tempest?07:23
evrardjpI haven't seen the failure07:23
chandankumarevrardjp: http://codesearch.openstack.org/?q=python-httplib2&i=nope&files=.yml&repos=07:24
chandankumarevrardjp: http://logs.openstack.org/42/654942/8/check/openstack-ansible-functional-tempestconf-centos-7/72d4677/job-output.txt.gz#_2019-05-20_11_54_42_63809707:24
chandankumarevrardjp: yesterday I was discussing with logan- about the same, from #rdo folks appears centos extra repos are getting disabled at some places07:25
chandankumarwhich I donot know where it is so, we added this package to lxc host07:25
chandankumaras it is needed for os_neutron07:25
* chandankumar is not sure it is the correct fix07:25
evrardjplet me check07:26
*** udesale has quit IRC07:35
*** udesale has joined #openstack-ansible07:36
*** pcaruana has quit IRC07:39
openstackgerritChandan Kumar (raukadah) proposed openstack/openstack-ansible-os_tempest master: Update openstack.org -> opendev.org  https://review.opendev.org/65494207:39
*** tosky has joined #openstack-ansible07:41
jrosserchandankumar: more interesting question is if python-httplib2 is actually required at all, being in the -devel package list is saying it is a requirement for building python wheels07:43
evrardjpjrosser: I suppose it tries to install it but fails to find the package07:44
evrardjpfor the neutron role, as python-httplib2 is listed there07:44
evrardjpbut I suppose you meant: "Is it really required?"07:44
chandankumarevrardjp: jrosser let me remove and see what happens07:44
evrardjpnot really sure what you meant :)07:45
jrosserwell here https://github.com/openstack/openstack-ansible-os_neutron/blob/master/vars/redhat-7.yml#L5207:45
evrardjpchandankumar: do that in a separate commit, I think https://review.opendev.org/#/c/654942/ is fine07:45
jrosserthe -devel packages things needed for the venv build, rather than runtime07:45
evrardjpjrosser: yes precisely... you mean you want to remove it from the list to see what happens?07:45
jrosseryeah, it just seems a bit of an odd requirement07:45
jrosserand if it turns out to be required, we should add a comment as to why07:46
evrardjpI thought it was because those roles sometimes need to build with external things07:46
evrardjpneutron plugins, nova plugins/consoles07:46
evrardjpwhich could require this for building said things07:46
jrossercould be07:46
evrardjpif we could get hold of odyssey4me to ask why it was there, he probably remembers the context of why this was in the list for the building07:47
evrardjpwe can then add it as comment07:47
openstackgerritJean-Philippe Evrard proposed openstack/openstack-ansible-os_neutron master: Clarify supported SUSE versions  https://review.opendev.org/66027007:51
cshenHi guys, I noticed that in openstack-ansible, there arey some *-em tags, like ocata-em, pike-em. Does that mean, ocata and pike are still supported?07:55
openstackgerritJean-Philippe Evrard proposed openstack/openstack-ansible-os_monasca-agent master: Fix monasca agent for leap 15  https://review.opendev.org/66027207:56
openstackgerritJean-Philippe Evrard proposed openstack/openstack-ansible-os_nova master: Clarify supported SUSE versions  https://review.opendev.org/66027307:58
evrardjpcshen: they are not "supported"07:58
evrardjpcshen: they are under "extended maintenance"07:58
evrardjpwhich means no guarantees, probably no gating, just community efforts07:59
cshenevrardjp: Ok, thanks. Because ocata-em got its last commit about 1 month ago. I misunderstood it's still supported. Now I got it.08:00
evrardjpcshen: https://docs.openstack.org/project-team-guide/stable-branches.html#maintenance-phases08:00
evrardjpbasically dying, and nobody cares.08:00
evrardjpif you care, then don't hesitate to submit patches :)08:00
*** hamzaachi has joined #openstack-ansible08:00
evrardjpcshen: you had somehting in mind?08:01
cshenNo, it's already solved in the *-em tag. So I have nothing to commit yet.08:01
*** markvoelker has joined #openstack-ansible08:06
*** luksky has quit IRC08:11
openstackgerritJean-Philippe Evrard proposed openstack/openstack-ansible-os_octavia master: Add integrated coverage  https://review.opendev.org/66027608:11
openstackgerritJean-Philippe Evrard proposed openstack/openstack-ansible-os_panko master: Clarify supported OS versions  https://review.opendev.org/66018108:13
evrardjpjrosser: if you could give this one a nudge that would be appreciated: https://review.opendev.org/#/c/660182/108:15
jrossercould you take a look at this to keep logans stuff moving? https://review.opendev.org/#/c/659971/08:16
openstackgerritJean-Philippe Evrard proposed openstack/openstack-ansible-os_placement master: Add metal distro job template  https://review.opendev.org/66027908:19
*** priteau has joined #openstack-ansible08:20
openstackgerritJean-Philippe Evrard proposed openstack/openstack-ansible-os_placement master: Clarify supported SUSE versions  https://review.opendev.org/66028108:20
evrardjpjrosser: sure08:20
*** priteau has quit IRC08:24
openstackgerritJean-Philippe Evrard proposed openstack/openstack-ansible-os_monasca-agent master: Fix monasca agent for leap 15  https://review.opendev.org/66027208:25
*** priteau has joined #openstack-ansible08:28
openstackgerritMerged openstack/openstack-ansible-os_masakari master: Add integrated coverage  https://review.opendev.org/66012208:29
openstackgerritJean-Philippe Evrard proposed openstack/openstack-ansible-os_rally master: Add integrated coverage  https://review.opendev.org/66028408:29
openstackgerritJean-Philippe Evrard proposed openstack/openstack-ansible-os_rally master: Clarify supported SUSE versions  https://review.opendev.org/66028608:32
openstackgerritJean-Philippe Evrard proposed openstack/openstack-ansible-os_swift master: Add integrated coverage  https://review.opendev.org/66028708:37
openstackgerritJean-Philippe Evrard proposed openstack/openstack-ansible-os_swift master: Clarify supported SUSE versions  https://review.opendev.org/66028808:38
*** markvoelker has quit IRC08:39
openstackgerritJean-Philippe Evrard proposed openstack/openstack-ansible-os_tacker master: Add integrated coverage  https://review.opendev.org/66028908:43
openstackgerritJean-Philippe Evrard proposed openstack/openstack-ansible-os_tacker master: Clarify supported SUSE versions  https://review.opendev.org/66029108:44
evrardjpCeeMac: morning08:44
*** luksky has joined #openstack-ansible08:46
*** PTO has joined #openstack-ansible08:47
openstackgerrittonybrad proposed openstack/openstack-ansible-os_sahara master: Update tests git source to opendev  https://review.opendev.org/65732208:47
openstackgerritJean-Philippe Evrard proposed openstack/openstack-ansible-os_tempest stable/stein: Clarify supported SUSE versions  https://review.opendev.org/66029408:48
openstackgerrittonybrad proposed openstack/openstack-ansible-os_monasca master: Change the tests source to opendev  https://review.opendev.org/65721208:48
openstackgerrittonybrad proposed openstack/openstack-ansible-os_monasca-agent master: Change tests source to opendev  https://review.opendev.org/65721608:49
openstackgerrittonybrad proposed openstack/openstack-ansible-os_blazar master: Update the git to opendev  https://review.opendev.org/65720908:50
openstackgerritJean-Philippe Evrard proposed openstack/openstack-ansible-os_tempest master: Clarify supported SUSE versions  https://review.opendev.org/66029608:50
openstackgerrittonybrad proposed openstack/openstack-ansible-os_manila master: Update the tests source  https://review.opendev.org/65729608:51
openstackgerrittonybrad proposed openstack/openstack-ansible-rsyslog_client master: Move to opendev  https://review.opendev.org/65729708:51
openstackgerrittonybrad proposed openstack/openstack-ansible-os_panko master: Change tests source to opendev  https://review.opendev.org/65729308:52
openstackgerrittonybrad proposed openstack/openstack-ansible-repo_server master: Change the tests source to opendev  https://review.opendev.org/65730908:53
openstackgerritJean-Philippe Evrard proposed openstack/openstack-ansible-os_trove master: Add integrated coverage  https://review.opendev.org/66029808:57
openstackgerritJean-Philippe Evrard proposed openstack/openstack-ansible-os_trove master: Clarify supported SUSE versions  https://review.opendev.org/66029908:58
openstackgerritOpenStack Proposal Bot proposed openstack/openstack-ansible master: Imported Translations from Zanata  https://review.opendev.org/65873108:58
openstackgerrittonybrad proposed openstack/openstack-ansible-pip_install master: Update to opendev  https://review.opendev.org/65731808:59
openstackgerritJean-Philippe Evrard proposed openstack/openstack-ansible-os_zun master: Add integrated coverage  https://review.opendev.org/66030109:00
openstackgerritJean-Philippe Evrard proposed openstack/openstack-ansible-os_zun master: Clarify supported SUSE versions  https://review.opendev.org/66030209:00
openstackgerritJean-Philippe Evrard proposed openstack/openstack-ansible-os_zun master: Add integrated coverage  https://review.opendev.org/66030109:01
openstackgerritJean-Philippe Evrard proposed openstack/openstack-ansible-os_zun master: Clarify supported SUSE versions  https://review.opendev.org/66030209:01
CeeMacguys, how does one tell if a particular patch features in a tagged branch?09:03
*** sreejithp has joined #openstack-ansible09:03
openstackgerritChandan Kumar (raukadah) proposed openstack/openstack-ansible-os_neutron master: [DNM] remove python-httplib2 during venv build requires  https://review.opendev.org/66030709:04
openstackgerritJean-Philippe Evrard proposed openstack/ansible-config_template master: Clarify supported SUSE versions  https://review.opendev.org/66030909:05
openstackgerritJean-Philippe Evrard proposed openstack/ansible-hardening master: Clarify supported SUSE versions  https://review.opendev.org/66031109:06
evrardjpCeeMac: could you clarify your question?09:06
evrardjp(for example with the patch in question?)09:06
evrardjpwe don't tag roles anymore09:07
CeeMacevrardjp: in general i guess would be good to know, but specifically if 18.1.5 contained https://review.opendev.org/#/c/632118/1 (I've checked the file on my dep node and it looks like the changes are there)09:07
*** sreejithp has quit IRC09:07
CeeMacfor instance, I can see in the cherry-pick from master that its in branch stable/rocky, but then how would I be able to tell if that was just in the head or in one of the tagged branches (18.1.5, 18.1.6 etc)09:09
evrardjpso if you would like to know if a patch has been backported to a certain branch you can check in "included in" for example: https://review.opendev.org/#/c/621509/09:09
evrardjpif you want to see if this has been tagged recently, you can check in ansible-role-requirements sha09:09
CeeMacoooooh...  theres a dropdown right there.09:09
evrardjpso check if it's older than a month, you should have it with your last tag09:10
CeeMacolder than a month as in based on 'updated' field?09:10
evrardjpI generally do things every two weeks, but recently with summit and stuff I might be late09:10
evrardjpcheck the merged date09:11
evrardjpif not, ask me to push a new release :p09:11
CeeMac'merged by zuul'?09:11
CeeMacok, thanks09:11
CeeMacjust trying to take opportunities to get to grips with the system while I'm working through my pre-deploy checks :)09:12
evrardjpso it means the patch has landed in the git repo, which is required for the next "bump" to catch it09:12
evrardjpwe bump every two weeks09:12
CeeMacthe bump being a tag?09:12
evrardjpnope, we use shas09:12
evrardjpsee ansible-role-requirements09:12
CeeMacah, ok09:12
*** phasespace has joined #openstack-ansible09:12
evrardjpthe bump basically updates the sha to the latest it finds in the repos09:13
CeeMacso, tags are used for other reasons based on release stability for instance?09:13
evrardjptags are used in OSA main repo to say "This has been tested and should be working fine"09:13
*** openstackstatus has quit IRC09:13
*** openstackstatus has joined #openstack-ansible09:14
*** ChanServ sets mode: +v openstackstatus09:14
evrardjpbecause we don't rely on a moving target for ansible-role-requirements (head of branch), but we rely on SHAs instead, it makes the installs very stable09:14
CeeMacpresumably, if I were checked out stable/rocky, i'd need to do a pull periodically to get the latest 'bumps' then? (purely for informative purposes, i'm sticking with 18.1.5 for now)09:14
evrardjpif you come back in a month, 18.1.5 will still have the same roles content09:15
CeeMacbut stable will have moved on?09:15
evrardjpyes, you should be doing minor updates09:15
evrardjpthere is docs for that09:15
CeeMaci've seen them09:15
evrardjpso basically checkout the latest tag, rebootstrap ansible, and redeploy09:15
CeeMacfollowed the docs to get up to 18.1.5 from where I was09:16
evrardjpsounds good09:16
CeeMachappy with that process :)09:16
evrardjpbeing up to date is good09:16
CeeMaci've been reading up on git/gerrit so I'm just trying to correlate random facts with actual use cases09:16
evrardjpthe fact we do a release every two weeks (in theory, some exceptional circumstances might happen) helps too09:17
CeeMacin between trying to prep for production roll out, and my other day job antics09:17
chandankumarhttp://logs.openstack.org/42/654942/10/check/openstack-ansible-functional-tempestconf-centos-7/4b8943c/job-output.txt.gz#_2019-05-21_08_27_10_139112 still not working for python-httlip209:17
chandankumarlet me try depends on with os_neutron patch09:17
evrardjpchandankumar: nope you have another problem to dig in09:18
evrardjpwhy isn't extras available? Why can't you install python-httplib2 ?09:19
evrardjpthat's what you should dig09:19
evrardjpcheck the repos that are enabled, etc.09:19
evrardjpI would also check why this starts to fail in lxc, while it works in distro based installs09:20
*** electrofelix has joined #openstack-ansible09:30
*** PTO has quit IRC09:30
*** markvoelker has joined #openstack-ansible09:35
*** gillesMo has quit IRC09:38
openstackgerritJean-Philippe Evrard proposed openstack/openstack-ansible-galera_client master: Add integrated coverage  https://review.opendev.org/66031909:42
openstackgerritJean-Philippe Evrard proposed openstack/openstack-ansible-galera_client master: Clarify supported SUSE versions  https://review.opendev.org/66032009:43
openstackgerritMerged openstack/openstack-ansible-os_panko master: Use the dynamic aio scenario  https://review.opendev.org/66018209:45
CeeMacjrosser: you know we talked about having the deployment node and the hosts on an out of band management network?  Presumably I'll need to use the OOB IP in the o_u_c? (because it just failed with it populated using the br-mgmt ips....)09:48
openstackgerritJean-Philippe Evrard proposed openstack/openstack-ansible-memcached_server master: Add integrated coverage  https://review.opendev.org/66032209:52
nowsterchandankumar: Is https://review.opendev.org/#/c/660141/  related?09:52
openstackgerritJean-Philippe Evrard proposed openstack/openstack-ansible-memcached_server master: Clarify supported SUSE versions  https://review.opendev.org/66032309:53
openstackgerritJean-Philippe Evrard proposed openstack/openstack-ansible-galera_client master: Add integrated coverage  https://review.opendev.org/66031909:56
openstackgerritJean-Philippe Evrard proposed openstack/openstack-ansible-openstack_openrc master: Add integrated coverage  https://review.opendev.org/66033109:58
openstackgerritJean-Philippe Evrard proposed openstack/openstack-ansible-openstack_openrc master: Clarify supported SUSE versions  https://review.opendev.org/66033209:58
openstackgerritJean-Philippe Evrard proposed openstack/openstack-ansible-os_manila master: Add integrated coverage  https://review.opendev.org/66033310:01
openstackgerritMerged openstack/openstack-ansible stable/queens: Use the correct branch name to clone tests repo  https://review.opendev.org/65997110:02
openstackgerritDirk Mueller proposed openstack/openstack-ansible-galera_server master: Update package list for openSUSE Leap 15.0  https://review.opendev.org/66020410:08
openstackgerritDirk Mueller proposed openstack/openstack-ansible-galera_server master: Switch to openSUSE Leap 15 for functional testing  https://review.opendev.org/66020810:08
evrardjpgod doing this integrated testing is SOOOOO tedious10:09
*** markvoelker has quit IRC10:09
nowsterchandankumar: the mirror at http://mirror.dfw.rax.openstack.org/centos/7/ might be corrupt, perhaps?10:09
dirkevrardjp: who would have thought splitting into half a billion individual repos would have downsides? ;-)10:10
openstackgerritJean-Philippe Evrard proposed openstack/openstack-ansible-os_sahara master: Add integrated coverage  https://review.opendev.org/66033710:11
evrardjpI regret nothing!10:11
dirkcould any of the cores please review the piles of "move to opendev" patches? e.g. https://review.opendev.org/#/c/657254/ and friends10:11
evrardjpthe problem is when we set up those tests with zuul v2... We are now in a better world10:11
dirkI believe those should be easy to land and makes the list of open reviews quite a bit more bearable10:11
nowsterI'm not seeing python-httplib2 in http://mirror.dfw.rax.openstack.org/centos/7/os/x86_64/Packages/10:11
*** aedc has joined #openstack-ansible10:11
jrosserCeeMac: yes you need to list whichever IP is used to ssh to the hosts, becasue at the end of the day it's creating an ansible inventory10:12
evrardjptbh I was lazy dirk. I was hoping there would be the ML thread about using requirements's published constraints first, and get all of those in one swoop10:12
evrardjpI was lazy, dirk*10:12
jrosserthe code will read the IP from br-mgmt when that is needed specifically10:12
nowsterchandankumar: it's in "extras": http://mirror.dfw.rax.openstack.org/centos/7/extras/x86_64/Packages/python-httplib2-0.9.2-1.el7.noarch.rpm10:12
nowsterchandankumar: might be an idea to check that "extras" is enabled10:13
jrosserif you use codesearch we are pretty much the only project needing it - i'm still unsure that it's an actual requirement10:13
CeeMacthanks jrosser it makes perfect sense in hind sight!10:13
openstackgerritJean-Philippe Evrard proposed openstack/openstack-ansible-os_sahara master: Clarify supported SUSE versions  https://review.opendev.org/66033810:14
CeeMacon a semi-related note.  Do the plays need to be run by root, or wil any user with sudo work fine?  I just realised I'd locked SSH access down to prevent the root account logging in and wondered why I couldn't ssh from the deployment node using the root account......10:14
* CeeMac facepalms10:14
chandankumarnowster: got stuck with something else on tripleo side, will look10:15
openstackgerritMerged openstack/openstack-ansible-tests master: Rename opensuse functional jobs to opensuse-15  https://review.opendev.org/66020710:17
evrardjpnoonedeadpunk: did you have a look at why https://review.opendev.org/#/c/653985/ is failing?10:18
evrardjpCeeMac: by default we use root to connect10:18
evrardjpit's probably doable with your inventory though10:19
evrardjpbut it's not really tested and will be hard with delegation and the becoming with lower privileges and stuff10:19
CeeMacevrardjp: yeah, i fast forwarded to that conclusion and have slackened the security in my ssh config.  Its limited to listening on the managment IP only, so the risk is low(er)10:27
*** L_SR has joined #openstack-ansible10:29
noonedeadpunkevrardjp: xenial for master?10:32
noonedeadpunkI think we should probably update this job somehow...10:33
evrardjpwow I haven't seen that10:34
evrardjpyes we should :p10:34
L_SRHello Guys, simply trying to demystify  the fact that my deployment node is unable to complete the setup-openstack playbook. I have a pretty standard user config and variables (according to the test example). And many iterations have pin-pointed to problem to os-keystone-install, particularly an issue with MySQL.10:34
noonedeadpunkI'll try to handle this in another patch10:34
noonedeadpunkor anyone may, if have extra time on hands ofc :p10:35
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-os_glance master: Replace git.openstack.org with opendev.org  https://review.opendev.org/65400310:36
evrardjpL_SR: still not fixed?10:36
L_SRchecking HAProxy, galera's front and back are in status OPEN and UP, respectively. Keystone service front 1 and 2 are OPEN, while keystone service back is DOWN.10:36
evrardjpL_SR: so could you remind us what you're deploying? OS/Branch/stuff ?10:37
L_SRevrardjp: still does not yield. I have changed OS nodes to Bionic. Triple checked networking and all seems to be OK10:37
evrardjpbionic, and you're running stein or rocky?10:37
L_SRI have noticed that sometimes ansible galera_container -m shell \   -a "mysql -h localhost -e 'show status like \"%wsrep_cluster_%\";'" yields a CHANGED instead of a SUCCESS. But that does not seem to alter the result of the setup-openstack playbook10:38
L_SRevrardjp: currenlty in the master branch10:38
L_SRevrardjp: I have tried Rocky and Stein, both failing at the same point in the setup-openstack playbook10:38
evrardjpso master bionic, galera seems okay10:39
evrardjpoh wait10:39
evrardjpdid you crush the DB/containers between your tests, clean your facts and rebootstrapped ansible?10:39
L_SRevrardjp: the authentication problem with mysql could be bypassed by modifying keystone_db_setup.yaml and specifying linux socket for mysql. But this feels wrong10:39
L_SRevrardjp: I have done lxc-container-destroy, rm -r /etc/ansible/*, rm -r /etc/openstack_deploy; i.e. bazooka approach10:40
evrardjpcores: for the requirements constriants url, I propose we merge everything we have right now, and we clear stuff afterwards10:40
evrardjpL_SR: that's what I would have odne10:40
evrardjpyou still need to rebootstrap ansible10:41
evrardjpto have the right roles for your integrated repo10:41
evrardjpI would also check there is no galera artifacts on said machine10:41
evrardjpjust to make sure :)10:42
L_SRsure, after the bazooking, I do /opt/openstack-ansible/scripts/bootstrap-ansible10:42
jrosserL_SR: what are you deploying onto? bare metal host / vmware / other.......?10:42
evrardjpthat's good10:42
L_SRjrosser: deploying on bare-metal, i.e. Intel NUCs10:42
jrosserL_SR: it seems you have a consistent problem regardless of the branch you test, with the keystone db setup10:44
jrosserL_SR: do you run the deployment from the target host?10:44
L_SRjust a though: I have another Openstack-Ansible deployment working OK. In that cloud the infra1 node has an LXD network range in the same range this new deployment would be ( i believe): could this be an issue? (I am pretty sure NAT would block any kind of addressing/routing confusion, but just asking)10:44
L_SRjrosser: my deployment host is an LXC container on a separate computer in the same L2 segment10:44
L_SRin the same br-mgmt L2 segment10:45
L_SRevrardjp: how can I check if there are galera artifacts bugging around in my infra host?10:45
*** udesale has quit IRC10:46
L_SRjrosser: this is my setup: https://imgur.com/huWQicd10:48
jrosserL_SR: ah yes i remember that from the other day10:48
jrosserso you are still stuck at the keystone db setup tasks?10:48
jrosserso - those tasks are delegated to the deployment host https://github.com/openstack/openstack-ansible-os_keystone/blob/master/tasks/keystone_db_setup.yml#L2410:49
L_SRwhen running setup-openstack for the first time I get a Mysql authentication error, subsequent executions of the same playbook yield the following error: https://pastebin.com/trUQdffZ10:50
L_SRjrosser: yes, you commented that on a previous occasion10:50
nowsterchandankumar: the other possibility is that the python wheel for httplib2 isn't being fetched10:50
jrosserL_SR: ah well that task is not delegated10:51
L_SRjrosser: this is the error I get when I first run setup-openstack playbook: https://pastebin.com/1jmsDH24 subsequent executions yield the error I pasted above10:51
L_SRjrosser: should it be delegated to the setup host?10:53
CeeMacL_SR: you're not having a great time with this DB error :(10:56
L_SRCeeMac: it's personal now :)10:56
L_SRCeeMac: but it is true. The thing is that I have not a clear clue of how/where to proceed.10:56
CeeMacL_SR: you mentioned you have the same CIDR in another env.  Is it worth checking the arp table in this env matches what you'd expect to see for the DB container IPs etc?10:56
CeeMacjust incase there is some bleed through?10:57
nowsterchandankumar: also there's no python-httplib2 in http://mirror.dfw.rax.openstack.org:8080/rdo/centos7/a4/b4/a4b4ae1f656d7eb86e641e04add5a9b6f3511781_85df50c7/10:57
nowster(but I don't think that one's relevant)10:57
jrosserL_SR: it's probably not worth pushing on beyond that first error, thats fairly fundamental10:59
L_SRCeeMac: just checked, and the new deployment cannot reach the lxcbr0 on the old deployment11:00
L_SRjrosser: just going to destroy and redeploy again11:00
CeeMacyou checked it couldn't reach the old deployment or you checked the arp table entries are correct?11:00
jrosserthe lxcbr0 should be local to each host11:00
jrosserthey nat out of each host to provide eth0 in each container11:01
L_SRCeeMac: that it is not reachable and does not appear in the arp table11:02
L_SRjrosser: exactly. So this IP range is only local11:02
CeeMacsorry, i misread, you said lxd network, i thought br-mgmt11:02
CeeMacmy brain is not working today11:02
L_SRCeeMac :)11:02
CeeMacL_SR: when you blow away the containers, is the data in /openstack/<galera_container> being deleted?11:05
CeeMacon infra111:05
*** markvoelker has joined #openstack-ansible11:06
*** nicolasbock has joined #openstack-ansible11:16
openstackgerritMerged openstack/openstack-ansible-os_monasca-agent master: Change tests source to opendev  https://review.opendev.org/65721611:18
openstackgerritMerged openstack/openstack-ansible-os_monasca master: Change the tests source to opendev  https://review.opendev.org/65721211:19
*** sreejithp has joined #openstack-ansible11:20
openstackgerritMerged openstack/openstack-ansible-pip_install master: Update to opendev  https://review.opendev.org/65731811:22
openstackgerritMerged openstack/openstack-ansible-rsyslog_client master: Move to opendev  https://review.opendev.org/65729711:23
openstackgerritMerged openstack/openstack-ansible-os_panko master: Clarify supported OS versions  https://review.opendev.org/66018111:23
*** dave-mccowan has joined #openstack-ansible11:28
*** kplant has joined #openstack-ansible11:29
openstackgerritJean-Philippe Evrard proposed openstack/openstack-ansible master: Fix mistral metal deploy  https://review.opendev.org/66035511:39
*** markvoelker has quit IRC11:39
evrardjplogan-: jrosser noonedeadpunk ^11:39
jrosserevrardjp: i think a bunch of these "use integrated test" changes result in the role not actually being tested, zun was an example11:39
*** joshualyle has quit IRC11:40
evrardjpthat's bad, but I think it's fair that we move to this, and fix appropriately11:40
*** sreejithp has quit IRC11:40
evrardjpwhat I just ping is the problem with mistral running with lxc11:40
*** sreejithp has joined #openstack-ansible11:40
evrardjpthere are these small gaps we will have to fix11:40
jrosseryes, there needs to be corresponding support in the integrated repo to insert the right config for each role test11:41
evrardjp(I suppose for zun we need to have the right inventory)11:41
logan-all of these are ready for review: https://review.opendev.org/#/q/status:open+owner:logan2211%2540gmail.com+(topic:calico3+OR+topic:osa/opendev+OR+topic:percona-release)+label:Verified:%252B111:41
logan-jrosser: yeah we definitely need to check the logs on the "use integrated testing" patches to fish out those issues. I figured we'd run across a few issues like that11:43
logan-I guess that will be a problem for any role we haven't integrated yet, right?11:43
evrardjpbut, if you recall what mnaser told during the PTG, he said "all the roles are integrated now"11:44
jrosseri expect we've merged a few like that11:44
*** cshen has quit IRC11:44
logan-evrardjp: gotcha11:44
evrardjpso I think there is a gap11:44
*** sreejithp has quit IRC11:44
evrardjpbut I think there are many gaps in those roles11:45
evrardjpSee here: https://docs.google.com/spreadsheets/d/1coiPHGqaIKNgCGYsNhEgzswqwp4wedm2XoBCN9WMosY/edit#gid=75207069511:45
evrardjpsome only supprt one distro, some don't support package installs11:45
logan-i think we should template that aio_metal file based on the bootstrap services list11:45
evrardjplogan-: probably11:45
evrardjpbut I went for the shortest route now11:45
evrardjpjrosser: if you could vote here: https://review.opendev.org/#/c/660355/ that would close a gap for the mistral integration11:46
jrossersure - i've been doing them as they pass11:46
evrardjpone of the things we will have to do, is update the constraints for master.11:46
evrardjpin tox.ini11:46
evrardjp(or maybe other files too)11:46
evrardjpwhen we'll be doing this, I think we should ensure the roles are running correctly in the integrated11:47
evrardjpat least the jobs will be consistent and all :)11:47
openstackgerritJean-Philippe Evrard proposed openstack/openstack-ansible master: Fix mistral metal deploy  https://review.opendev.org/66035511:48
evrardjplogan-: moved even more of the things ^11:48
logan-haha yeah i just noticed that after i pushed comment11:48
logan-but oyu left the original mistral too btw11:48
openstackgerritJean-Philippe Evrard proposed openstack/openstack-ansible master: Fix mistral metal deploy  https://review.opendev.org/66035511:48
evrardjpand made a poo poo yeah11:48
evrardjpfixed it11:49
*** cshen has joined #openstack-ansible11:50
openstackgerritMerged openstack/openstack-ansible-os_cloudkitty master: Change git to opendev  https://review.opendev.org/65720211:50
*** mkuf_ has joined #openstack-ansible11:55
*** mkuf has quit IRC11:58
*** ansmith has quit IRC12:03
*** sreejithp has joined #openstack-ansible12:06
L_SRHey guys, what could be the reason why I'm getting this status from the galera container? https://pastebin.com/vaxEDKvJ12:06
L_SRsetup-hosts and setup-infrastructure finished successfuly12:06
evrardjpare you running single node galera?12:06
evrardjpthat's weird and very risky12:06
evrardjpbut it's okay we do it in different places12:07
*** markvoelker has joined #openstack-ansible12:08
L_SRyes, I only have a single infrastructure node12:08
L_SRI have also noticed that different branches give different results for ansible galera_container -m shell \ >   -a "mysql -h localhost -e 'show status like \"%wsrep_cluster_%\";'"12:09
evrardjpdid you change your inventory for that?12:09
L_SRI destroyed everything and started over12:09
evrardjpis this a PoC or somehting?12:09
evrardjpbecause you know there is an AIO scenario that can probably do what you are looking for?12:10
L_SRit is a test environment I want to build, but is giving me headaches12:10
L_SRevrardjp, I know, but this test requires multiple compute nodes12:10
evrardjpopenstack-ansible -e galera_ignore_cluster_state=true galera-install.yml12:11
evrardjpyou can also check rebuilding an AIO12:12
*** sreejithp has quit IRC12:12
evrardjpbut basically you're saying your networking is fine, you're running master as your own AIO (not using predefined scripts), haproxy is fine, galera is fine, but keystone fails12:12
evrardjpyou're installing on bionic12:12
evrardjpdid I summarize things correctly?12:12
evrardjpand you say that by default keystone cannot connect with the credentials that are in user_secrets12:13
L_SRin the first run it complains about it, if I try running the same playbook again it fails migrating the keystone db12:14
evrardjplet's focus on that first run12:14
L_SRuser_secrets is generated according to the documentation12:14
L_SRshall I destroy and create the keystone container again to get the error?12:14
evrardjpcould you run mysql commands on that galera cluster, with the appropriate user, see how it goes?12:14
openstackgerritMerged openstack/openstack-ansible master: Fix mistral metal deploy  https://review.opendev.org/66035512:15
evrardjpyes, and I would delete once again galera :)12:15
L_SRwell, at the moment I stopped at setup-infrastructure because galera does not give the SUCESS state, but only CHANGED12:15
openstackgerritMerged openstack/openstack-ansible-tests stable/queens: Fix clone list for opendev changes  https://review.opendev.org/65993012:15
evrardjpL_SR: I would follow this: https://docs.openstack.org/openstack-ansible/latest/user/aio/quickstart.html#rebuilding-an-aio12:15
L_SRok. I assume the erasing happens on my infra node12:16
evrardjpyou only have one node you told me (except compute I suppose)12:16
*** udesale has joined #openstack-ansible12:16
L_SRone infra, one compute, one storage12:16
openstackgerritOpenStack Proposal Bot proposed openstack/ansible-hardening stable/queens: Updated from OpenStack Ansible Tests  https://review.opendev.org/66036512:22
openstackgerritJean-Philippe Evrard proposed openstack/openstack-ansible-os_octavia master: [WIP] Add SUSE support  https://review.opendev.org/66036712:29
openstackgerritMerged openstack/openstack-ansible-os_blazar master: Update the git to opendev  https://review.opendev.org/65720912:29
*** mkuf has joined #openstack-ansible12:31
*** mkuf_ has quit IRC12:35
openstackgerritJean-Philippe Evrard proposed openstack/openstack-ansible-galera_client master: Clarify supported SUSE versions  https://review.opendev.org/66032012:35
*** priteau has quit IRC12:36
*** priteau has joined #openstack-ansible12:38
*** tuxjohnson has joined #openstack-ansible12:48
*** tuxjohnson has left #openstack-ansible12:48
L_SRevrardjp: same issue with galera after setup-infrastructure12:48
*** priteau has quit IRC12:51
guilhermespmorning all!12:51
evrardjpL_SR: could you rephrase that?12:52
evrardjpYou mean you know have an issue at setup-infrastructure?12:52
evrardjpinstead of keystone?12:52
L_SRevrardjp: after running setup-infrastructure, this is the status of the galera cluster: https://pastebin.com/zTGx9mUr12:52
guilhermespbtw cores, I was discussing with mnaser yesterday about the role retirement. We just have monasca ones to review and delete https://review.opendev.org/#/q/topic:retire-osa-unused-roles+status:+open12:53
guilhermespppl have shown interest in maintain but till now  we didn't hear back from them?12:53
L_SRevrardjp: I have seen this previously, but minded not much as the playbook executed successfully12:53
guilhermespI think we should move on and if someone is interested in maintain those roles, it can revert the retirement12:53
evrardjpwait I am confused L_SR12:54
evrardjpdo you mean it ran correctly or not? (setup-infrastructure)12:54
L_SRit ran correctly12:54
evrardjpand your haproxy state says?12:54
evrardjpeverything galera is fine?12:54
evrardjpand now you have to run the setup-openstack, but everything seems alright till now12:55
L_SRevrardjp: galera front -1 open, galera back up12:55
L_SRevrardjp: running setup-openstack and got the mysql auth error again12:56
evrardjpcould you paste me that error?12:56
L_SRevrardjp: this time I forgot to allow logging, https://pastebin.com/ijjjKXZt12:56
L_SRI can destroy the keystone container and try again to see the error12:57
evrardjpnope it won't matter12:57
evrardjpcreating the db is not in keystone12:57
L_SRthat's the task12:58
L_SRjrosser was highlighting the fact that the first two tasks are delegated12:58
evrardjpbut you didn't change anything in plugins or stuff12:58
evrardjpor env var12:59
evrardjpor ansible.cfg in /etc or in home12:59
evrardjpcould you run12:59
L_SRI have destroyed and re-created the deployment hosts several times and the error is the same all the time12:59
evrardjpansible -m ping galera_all12:59
*** ansmith has joined #openstack-ansible12:59
L_SRping to galera containers success: https://pastebin.com/Q6F8sTRK13:00
evrardjpcould you extract this into its own play and run it?13:01
evrardjpoh wait13:02
evrardjpbefore that13:02
evrardjpcould you connect to that galera container, and see if the db for keystone is there?13:02
L_SRevrardjp: going back to the mysql error, this is the rest of the execution of setup-openstack after the error: https://pastebin.com/GRrFShcy13:03
evrardjpthat's terribly wrong13:03
evrardjpif you container had an issue it should not be running more tasks13:04
evrardjpoh these are handlers13:04
evrardjpok I want to test something else too13:05
L_SRevrardjp: I'm at the galera container now, inside Mysql with root user I see no Keystone DB13:05
evrardjpcould you check the galera container ip?13:05
evrardjpgo back to your ansible13:06
evrardjpansible -m shell -a "ping -c 2" infra1_keystone_container-c43ef6bb13:06
openstackgerritMerged openstack/openstack-ansible-os_horizon master: Replace git.openstack.org URLs with opendev.org URLs  https://review.opendev.org/65725413:06
L_SRI think the ping succeeded: https://pastebin.com/hwTU6qb313:07
evrardjpYes I would run a different play to just focus on the db creation13:08
L_SRok, will do that13:08
evrardjpL_SR: you can run this: https://gist.github.com/evrardjp/c9c336a3dc46ad4d17d2db44aa6cd4d213:11
evrardjpwoops sorry13:12
evrardjprefresh :)13:12
evrardjpopenstack-ansible test.yml should do (please put that playbook into the osa playbooks folder13:12
openstackgerritMerged openstack/openstack-ansible-os_panko master: Change tests source to opendev  https://review.opendev.org/65729313:13
L_SRevrardjp: auth issue https://pastebin.com/8sZsZxRG13:13
evrardjpreplace login_host's value by ?13:14
L_SRevrardjp: weird, because I can use user_secrets to get into mysql in the galera container13:14
L_SRevrardjp: changing the variable for the IP succeeded13:15
evrardjpyou have a problem with haproxy13:15
*** zhongjun2_ has quit IRC13:15
L_SRevrardjp: any hint of the reason?13:15
L_SRevrardjp the haproxy is running on my infra1 host, right?13:16
evrardjpyou assured me it was working :)13:16
evrardjpin a container most likely13:16
evrardjpcan you show me your o_u_c again?13:16
L_SRI thought so: https://imgur.com/Wzp4sLS13:17
L_SRevrardjp: https://pastebin.com/bf6tDhCb13:18
evrardjpfirst can you show me your haproxy config?13:18
openstackgerritGuilherme  Steinmuller Pimentel proposed openstack/openstack-ansible-os_glance master: db_setup: refactor database setup to a common file  https://review.opendev.org/65702913:18
evrardjpthe o_u_c looks perfectly fine13:18
L_SRI have no extra config for the haproxy13:19
evrardjpbut I might have missed something it's easy to make mistakes13:19
evrardjpcan you show me the generated haproxy config?13:19
noonedeadpunkI'm wondering, why https://review.opendev.org/#/c/659754 fails...13:19
L_SRevrardjp any hint on where is that?13:19
evrardjpin the haproxy container I would say13:19
evrardjpin /etc/haproxy13:20
evrardjpor something13:20
L_SRthere is no haproxy container!13:20
L_SRgalera maybe?13:21
openstackgerritGuilherme  Steinmuller Pimentel proposed openstack/openstack-ansible-os_nova master: db_setup: refactor database setup to a common file  https://review.opendev.org/65703813:21
evrardjpL_SR: nope that was expected. It's on your host then13:21
evrardjpinfra host13:21
evrardjpit's on bare metal13:21
evrardjpok it's further closer :)13:21
evrardjpss -lntp | grep 330613:21
evrardjpcould you do this ?13:21
*** hwoarang has quit IRC13:21
evrardjpon that infra host?13:21
L_SRevrardjp haproxy config: https://pastebin.com/xULCnnbK13:22
openstackgerritGuilherme  Steinmuller Pimentel proposed openstack/openstack-ansible-os_keystone master: db_setup: refactor database setup to a common file  https://review.opendev.org/65703613:22
*** hwoarang has joined #openstack-ansible13:23
L_SRevrardjp checking mysql port: https://pastebin.com/M4bfzv5e13:23
evrardjpis that haproxy or do you have another service on that node on 3306?13:24
L_SRI wiped this node clean this morning13:25
L_SRit has a fresh bionic installation13:25
evrardjpdoing mysql operations through the LB fails, directly works13:25
L_SRno lvm, though. That's only configured on the storage host13:25
evrardjpbackend is not faulty13:26
evrardjpand you're listening correctl13:26
*** sreejithp has joined #openstack-ansible13:26
evrardjpthat's unusual13:27
evrardjpwhat's your private ip range again?13:29
evrardjpnopenopenope :)13:30
L_SR172.10.10.0/24 is mgmt13:30
evrardjpthis one!13:30
evrardjpyou don't find something weird in there?13:30
L_SRin the range itself?13:31
evrardjp(I will explain the whole story after)13:31
L_SRdon't tell me is a multicast addressing issue13:31
evrardjpbut you're close13:31
evrardjpif I tell you RFC191813:31
L_SRoh, public address13:32
evrardjp172.16.0.0/12 is the expectations13:32
L_SRgonna change to
evrardjpbecause we want the cloud to run behind the load balancer, the rest would be private13:33
evrardjpbut let me tell you why it fails13:33
evrardjpit's in there :)13:33
evrardjparound L65.13:33
evrardjpL65-L67 to be precise13:33
L_SRacl white_list src
evrardjpyw :)13:34
L_SRThanks!!!! I will redeploy now13:34
evrardjpso it's not an OSA problem :D13:34
L_SRplease, celebrate with me if it work :)13:34
evrardjpyou need to clean everything though13:35
evrardjpyour inventory is broken to the wrong ips and stuff13:35
CeeMacevrardjp: you're like some kind of troubleshooting ninja!13:35
evrardjpjust step by step :)13:35
CeeMaci feel like a dufus, I'm a network guy and I didn't spot the non-RFC address space.13:35
* CeeMac hangs up network-guy hat13:35
evrardjp0 ressembles to 6 when you read fast13:36
CeeMaci'll take that :)13:36
L_SRCeeMac Mee too!! I should retire13:37
kplantrevoke all assignments and declare it squat-space13:37
kplantproblem solved13:37
evrardjpkplant: haha13:38
evrardjpuse ipv6 only13:38
CeeMacipv6 hurts my head13:39
CeeMaci can't cope with everything in the world getting subnet equal to the entire IPv4 space at a minimum13:39
evrardjpit's better than being allergic to the character ':'13:41
kplanti like how ipv6 was necessary due to exhaustion and yet... one of the first practices is everyone gets a /6413:41
kplantor a /48 if you ask nicely with pd hints13:42
CeeMaci also like how most of the ISPs are still pushing out home routers with public IPv4 address, so if you want to run IPv6 on the edge you need to dual-stack so anyone can get to you.....13:43
jrosserL_SR: i was away - good you found it13:43
jrosserL_SR: there is a useful debug pattern which is to telnet to the backend service container port, then try the same to the LB vip to test that connectivity is good13:44
jrossermay well have been useful here13:44
CeeMacjrosser: another good tip I completely missed13:44
openstackgerritJean-Philippe Evrard proposed openstack/openstack-ansible-rabbitmq_server master: Add integrated coverage  https://review.opendev.org/66039413:45
evrardjpjrosser: I think I wrote a play for that13:45
kplant'nc -z' is a good one for that too13:45
kplanteasy to determine success after 14 hours of debugging :)13:45
evrardjpand healthcheck-hosts13:46
evrardjpmaybe I can add a test in healthcheck-hosts13:46
CeeMachindsight is 20-20 etc13:48
logan-wonder why https://review.opendev.org/#/c/660141/ isn't in the gate13:48
kplantthis may be more of a style question than anything but, do you guys use nova volumes at all?13:49
kplantor do you leave the root disk size at zero in the flavors and just use cinder for all block storage13:49
jrosserlogan-: i was wondering that too - i wonder if the depends-on change-id isnt so smart after all?13:50
logan-jrosser: i just tried abandoning the ocata/pike versions of the depends-on to see13:50
logan-iirc other branches can hold it up13:50
logan-yep sure enough13:51
logan-ther eit goes13:51
openstackgerritJean-Philippe Evrard proposed openstack/openstack-ansible master: Add rfc1918 healthcheck  https://review.opendev.org/66039913:53
evrardjpL_SR: ^13:54
evrardjplogan-: yeah this needs to be slowly done :p don't rush backports :p13:56
evrardjplogan-: also that would be fixed if you were to use the new form of urls13:56
evrardjp(Depends-On with the change url)13:56
logan-ah right13:56
jrosserevrardjp: not sure the commit message is right there? https://review.opendev.org/66039913:57
evrardjpjrosser: I was lacking proper ideas, feel free to adapt :)13:57
jrosserwell it just seems opposite to what i think you did (!)13:57
*** ansmith_ has joined #openstack-ansible13:57
openstackgerritJean-Philippe Evrard proposed openstack/openstack-ansible master: Add rfc1918 healthcheck  https://review.opendev.org/66039913:58
evrardjpjrosser: good catch13:58
*** miouge3 has joined #openstack-ansible13:59
*** gary_perkins_ has joined #openstack-ansible14:00
*** Anticime1 has joined #openstack-ansible14:01
*** _nwonknu has joined #openstack-ansible14:05
*** phasespace has quit IRC14:05
*** tosky__ has joined #openstack-ansible14:05
*** tosky has quit IRC14:06
*** ckonstanski has quit IRC14:06
*** jamesdenton has quit IRC14:06
*** joabdearaujo has quit IRC14:06
*** nwonknu has quit IRC14:06
*** bradm has quit IRC14:06
*** ansmith has quit IRC14:06
*** udesale has quit IRC14:06
*** markvoelker has quit IRC14:06
*** electrofelix has quit IRC14:06
*** aspiers has quit IRC14:06
*** miouge has quit IRC14:06
*** gary_perkins has quit IRC14:06
*** Anticimex has quit IRC14:06
*** redkrieg has quit IRC14:06
*** masterpe has quit IRC14:06
*** nsmeds has quit IRC14:06
*** udesale has joined #openstack-ansible14:07
*** electrofelix has joined #openstack-ansible14:07
CeeMacok, i've been hit by the keystone curse14:08
CeeMaci can ping the galera container IPs from the LB node14:09
CeeMacand I can telnet on port 330614:09
CeeMacif i go into hatop and put the node into maintenance then back, it come up for a second then goes in to SOCKERR14:09
CeeMacsetup-hosts, setup-inf both ran fine14:09
CeeMacrunning the setup-os plays individually, and the  os-keystone-install.yml fails14:10
CeeMacobvs if the haproxy backend is down, thats why the play is failing14:10
CeeMacgalera cluster looks fine http://paste.openstack.org/show/751870/14:12
CeeMacextract from haproxy log http://paste.openstack.org/show/751871/14:14
*** aspiers has joined #openstack-ansible14:17
*** aludwar has joined #openstack-ansible14:19
CeeMaccant see anything specific in the galera_server_error.log14:22
*** aedc has quit IRC14:22
jrosserCeeMac: https://bugs.launchpad.net/openstack-ansible/+bug/178423014:26
openstackLaunchpad bug 1784230 in openstack-ansible "Error while running setup-openstack.yml" [Undecided,Expired]14:26
openstackgerritMerged openstack/openstack-ansible-galera_server stable/queens: Update percona-release package  https://review.opendev.org/66014114:27
openstackgerritJean-Philippe Evrard proposed openstack/openstack-ansible-repo_server master: Clarify supported SUSE versions  https://review.opendev.org/66040214:34
CeeMacjrosser: good find.14:37
CeeMacinterestingly, the only_from IPs for the LB nodes is set to use the OOB IP not the br-mgmt ip14:37
CeeMacany idea where this is defined from?>14:38
openstackgerritJean-Philippe Evrard proposed openstack/openstack-ansible-rsyslog_client master: Add integrated coverage  https://review.opendev.org/66040314:38
openstackgerritJean-Philippe Evrard proposed openstack/openstack-ansible-rsyslog_client master: Clarify supported SUSE versions  https://review.opendev.org/66040414:39
openstackgerritJean-Philippe Evrard proposed openstack/openstack-ansible-rsyslog_server master: Add integrated coverage  https://review.opendev.org/66040514:40
jrosserCeeMac: which config file is that in?14:40
openstackgerritJean-Philippe Evrard proposed openstack/openstack-ansible-rsyslog_server master: Clarify supported SUSE versions  https://review.opendev.org/66040614:41
CeeMacjrosser: how do you mean?14:41
jrosserCeeMac> interestingly, the only_from IPs for the LB nodes is set to use the OOB IP not the br-mgmt ip14:42
jrosser^ that14:42
CeeMacthe only_from?  I'm picking that up from cat /etc/xinetd.d/mysqlchk | grep only in the galera container14:42
CeeMacas per the bug you sent :)14:42
CeeMacso, what is populating the mysqlchk file, and what criteria is being used to extrapolate the ip it uses14:43
jrosserCeeMac: http://codesearch.openstack.org/?q=galera_monitoring_allowed_source&i=nope&files=&repos=14:43
openstackgerritJean-Philippe Evrard proposed openstack/ansible-role-systemd_service master: Add integrated coverage  https://review.opendev.org/66040714:43
CeeMacjrosser: and i suppose its pulling groups['haproxy'] from the ansible inventory, which is configured to use the OOB IP.14:44
jrosseryes, so i have an override for this14:44
jrossergalera_monitoring_allowed_source: ""14:44
jrosser^ adjust to suit14:44
CeeMacin u_v ?14:45
openstackgerritJean-Philippe Evrard proposed openstack/ansible-role-systemd_mount master: Add integrated coverage  https://review.opendev.org/66040814:45
CeeMacany other overrides I should be aware of while we're at it?14:45
openstackgerritJean-Philippe Evrard proposed openstack/ansible-role-systemd_networkd master: Add integrated coverage  https://review.opendev.org/66040914:46
*** chandankumar is now known as raukadah14:46
jrosseri would carry on and override as necessary, so long as you're logical about finding where things don't look right then it'll be ok14:47
openstackgerritJean-Philippe Evrard proposed openstack/ansible-role-python_venv_build master: Add integrated coverage  https://review.opendev.org/66041114:48
*** luksky has quit IRC14:49
CeeMacjrosser: presumably you've whitelisted your whole br-mgmt network for future proofing against adding more galera nodes?14:49
jrosserCeeMac: well the proper answer is that this should probably be improved https://opendev.org/openstack/openstack-ansible/src/branch/master/inventory/group_vars/galera_all.yml#L3314:54
jrosserbecasue that just takes the inventory ip rather than finding br-mgmt ip for those nodes14:55
CeeMacwell yeah, but then how many people use oob management?  Is it corner-case or mainstream? /devils-advocate14:56
jrosseri think it is normal in rackspace deployments14:56
jrosserthe AIO and examples are just simplified for mortals to be able to get started14:57
CeeMacpesky mortals >_>14:57
jrosserbut you are quite right to ask "why is this internal API network the one i use to manage my hosts?"14:58
*** tosky__ is now known as tosky14:59
CeeMacok, i'll throw in an override and run the galera-install.yml play and see how it goes14:59
openstackgerritMerged openstack/openstack-ansible-os_placement master: Add metal distro job template  https://review.opendev.org/66027915:00
jrosseryou can make that specific or try to make it pick up the br-mgmt ip automagically by refactoring the existing galera group_var a bit15:00
*** tuxjohnson has joined #openstack-ansible15:05
CeeMachmm, not sure how that would work as its the haproxy ips that need picking up correctly15:05
CeeMaci'll hardcode a couple of ranges for now, and play around in test/rc once i've got prod deployed15:06
*** aedc has joined #openstack-ansible15:06
*** hamzy has quit IRC15:07
tuxjohnsonI am using "glance_default_store: cinder" in our installation and the OSA scripts never install a glance_sudoers file in the glance container.  I looked in my deployment server and there isn't a sudoers.j2 file that I can see.  Am I missing something?15:09
CeeMacok cool the galera backend is showing up now in hatop15:11
CeeMacre-running keystone play15:11
CeeMacnew error :( http://paste.openstack.org/show/751879/15:16
CeeMacverbose error: http://paste.openstack.org/show/751881/15:20
*** starborn has quit IRC15:20
jrosserCeeMac: i would guess that if you take the exact command it ran there and try it yourself on the keystone container there might be a better error message15:22
logan-could i get a last review on this stein patch https://review.opendev.org/#/c/659711/15:22
*** priteau has joined #openstack-ansible15:23
logan-jrosser: also this one if you have a moment https://review.opendev.org/#/c/659869/ since the stein patch will not gate until it merges15:25
*** cshen has quit IRC15:27
jrosserlogan-: done, this is all very cool btw getting calico 1st class citizen here15:27
*** sreejithp_ has joined #openstack-ansible15:28
logan-thanks! I agree -- will be nice to finally have the tests for it worked out15:29
*** sreejithp__ has joined #openstack-ansible15:30
*** sreejithp has quit IRC15:31
*** sreejithp_ has quit IRC15:32
CeeMaclog entry for keystone container http://paste.openstack.org/show/751886/15:33
CeeMaclooks like auth issue15:34
CeeMaci'm getting dejavu15:34
*** hamzy has joined #openstack-ansible15:35
*** miloa has quit IRC15:40
CeeMacack, i'll comb through the role tasks in the morning see if i can work out which command it is failing on and run that manually from utility container15:43
dirkevrardjp: I missed the context, most of the patches are about removing references to git.openstack.org. yes, there is also a small part of it (the tox.ini upperconstraints) but I think thats a minor part of it15:46
evrardjpdirk: yeah. Don't worry, we'll merge stuff and verify the rest afterwards15:47
*** ivve has quit IRC15:49
mnaserreviews on https://review.opendev.org/#/c/660224/2 would help land the patch above itw15:52
mnasergerrit is so slow here15:55
logan-ah yes thanks mnaser15:55
mnaserturns out 250-300ms isn't fun15:55
*** macza has joined #openstack-ansible15:55
*** macza has quit IRC15:56
*** macza has joined #openstack-ansible15:56
* mnaser waits for wiki page to load16:00
*** hamzaachi_ has joined #openstack-ansible16:00
mnaseranytime now..16:00
evrardjpdo we have meeting?16:00
mnaseryes im waiting for the wiki page to load so I can do the start meeting16:00
* mnaser is in china so .. things are slow16:01
evrardjpI want to discuss about releases/current state of jobs/constraints url16:01
mnaser..anytime now wiki16:01
mnasercan someone please just paste the start meeting string here16:01
mnaserthe wiki is not accessible for me right now16:02
evrardjp#startmeeting openstack_ansible_meeting16:02
openstackMeeting started Tue May 21 16:02:14 2019 UTC and is due to finish in 60 minutes.  The chair is evrardjp. Information about MeetBot at http://wiki.debian.org/MeetBot.16:02
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.16:02
*** openstack changes topic to " (Meeting topic: openstack_ansible_meeting)"16:02
openstackThe meeting name has been set to 'openstack_ansible_meeting'16:02
mnaser#topic office hours16:02
mnaseroh im not cool enough to do that right now16:02
evrardjp#chair mnaser16:02
openstackCurrent chairs: evrardjp mnaser16:02
*** hamzaachi has quit IRC16:02
mnaser#topic office hours16:02
*** openstack changes topic to "office hours (Meeting topic: openstack_ansible_meeting)"16:02
mnaserok now im cool enough16:02
mnaserevrardjp: btw, awesome patches on the integrated stuff on top of odyssey4me work16:03
evrardjpthat's the map of where we are. Focusing on metal. then metal distro. then lxc.16:03
mnaserI gotta vpn for that link :)16:03
mnaserethercalc's aren't blocked :p16:03
evrardjpI guessed16:04
mnaserall good, I have it open here now16:04
evrardjpIt doesn't matter for the conversation, basically there are reviews up, and things needs improvement16:04
*** errr has left #openstack-ansible16:04
evrardjpI see a few red dots in the horizon, for example, ironic, congress, blazar16:04
evrardjpoctavia, rally, tacker, trove, zun16:04
evrardjpthose are either not ready for distro installs, or they are not ready for multi distro, or both16:05
evrardjp(there are others I didn't mention)16:05
mnaserhmm, I think its mostly just adding some vars/<os>.yaml file?16:05
*** hamzaachi_ has quit IRC16:05
mnaseror is it quite a bit more complicated than that16:05
evrardjpsome of it is just that.16:05
mnaserobviously I'm making it far more trivial but yeaeh16:06
evrardjpsome are not just that, for example the packages don't exist for that distro16:06
evrardjpor need to refactor the code to get to a point that's similar to other roles16:06
evrardjpanyway this deserves love, and help is mostly welcomed16:06
*** luksky has joined #openstack-ansible16:06
evrardjpthere is a "if you want to help" on top of this page, so it should help ppl understand what's going on16:07
mnaserill try to do my part in reviewing, and anything failing debian/centos I can try and make a patch 'underneath'16:07
guilhermespevrardjp: I will take a look carefully16:07
* raukadah didnot getting time to look on python-httplib2 failure on centos16:07
guilhermespI will also continue mnaser 's work of db refactor https://review.opendev.org/#/q/topic:osa/db-refactor16:07
evrardjpmnaser: I think the most annoying is not the patch underneath for metal support for distro x, but more getting to support distro package isntalls16:07
*** hamzaachi_ has joined #openstack-ansible16:08
mnaseryeah.  that's quite a lot more work to be done there16:08
evrardjpanyway, sometimes there are patches "underneath" those links, so don't hesitate to look at the "related" patches16:08
evrardjpthat's all I have on that topic16:09
evrardjpthe other topic I have is linked to reviews, should I just continue?16:09
logan-may as well. thanks for putting up the spreadsheet and patches for the integrated tests stuff!16:10
mnasersure. I think what you discussed in on-going anyways and doesn't have anything actionable in specific other than: pay attention to reviews, try to help land some of these changes16:10
evrardjpwe'll need a map, basically, for another thing too:16:10
evrardjpWe'll have to use the constraints url file that are published by releases. I guess we'll discuss whether to use /master or /train for master (I don't have an opinion yet). But for stable branches, we're gonna have to do it too.16:12
evrardjpThe thing is... We shouldn't use that url everywhere, because that will always represent the head of the branch. So we'll have to be careful in the reviews.16:12
mnaserdon't we already pin commit hash for constraints. right now?16:12
evrardjpso those shouldn't change.16:13
evrardjpthe ones who can change are the test ones basically16:13
evrardjp(tox, etc.)16:13
mnaserso on release, we would freeze that commit, but in CI, we point to tip of requirements branch matching the release16:14
evrardjpI just want to raise awareness we shouldn't just change everything :)16:14
evrardjpI think we can keep the things frozen for requirements/repo build and many16:14
evrardjpand bump regularily16:14
evrardjpbut the tox and all should be using the new urls.16:14
noonedeadpunko/ missed start16:14
evrardjpSo basically don't use the new urls everywhere :p16:15
mnaserah yes, I agree, that's fine16:15
mnaseruse the new urls in tox only because we want to keep our pinned requirements16:15
*** vnogin has joined #openstack-ansible16:15
mnaserok, makes sense to me!16:15
*** gyee has joined #openstack-ansible16:15
evrardjpthere might be some other places we need adapting, but we need to do it smartly basically :)16:15
evrardjpthat's all I had on this topic16:15
evrardjpthe last topic I had was releases. I am late on releasing due to summit and travels and stuff.16:16
evrardjpand broken jobs16:16
evrardjpand many others16:16
logan-one other note on the new integrated testing rework stuff, you can actually implement scenarios in the roles downstream while still inheriting the integrated repo jobs16:16
logan-any pre-run playbooks in the job will run after the AIO has already been bootstrapped16:17
evrardjpbut logan helped on fixing things in Rocky/Queens, so thanks to him! I hope we'll have new releases soon16:17
mnaserack, yes, stein patch I had to recheck now16:17
mnaserI don't think we back ported the placement integration stuff, did we?16:17
evrardjplogan-: yeah that's a cool feature16:17
logan-yes thanks for pushing the reviews through.. queens is almost unblocked as of now16:17
logan-mnaser: no it is not backported16:18
mnaseryeah, that's all in master right now..16:18
guilhermespno we didn't mnaser16:18
mnaserwhat a mess it was to land it :<16:18
guilhermespand I think we haven't worked in the upgrade stuff for placement16:18
logan-^ right16:18
mnaserI'm thinking: we land most recent bump and push rc2, backport the placement stuff (so we at least have Greenfield placement in stable/stein)16:19
guilhermespi still have a picture of odyssey4me 's pseudo code :P16:19
mnaserand then we can work on the upgrade stuff afterwards.. as we usually don't release with the expectation of functional upgrades (generally)16:19
mnaserthoughts.. complaints..? :p16:19
evrardjpwe have until june to do so16:20
guilhermespI'd be ok as first step just try to have stable/stein greenfield for placement16:20
evrardjpor july?16:20
noonedeadpunksounds reasonable16:20
evrardjpcan't remember let me check16:20
logan-i think we should focus on getting stein released and then revisit the placement backport since it is optional for stein16:20
guilhermespI can take care of this backport anyways16:20
evrardjp11 July, 201916:20
mnaserlogan-: if we release stein without placement, then we don't backport placement16:21
evrardjpI think we could, as a community, agree on backporting afterwards16:21
logan-backporting placement in its current state would probably set us back quite a bit on R->S upgrade readiness right?16:21
evrardjpas long as we are clear in the ML/renos/versions16:21
mnaserit is a very invasive upgrade step so as someone who is going to update openstack, I'd be in for a mean surprise16:21
mnaserwe do things like create new databases, shut off services and migrate data from one db to another16:22
logan-yeah I think a backport is more palatable when we have code at least proposed if not landed to enable upgrades16:22
guilhermespyes, you're right evrardjp logan- mnaser16:22
mnasermany things can go wrong™16:22
mnaserI think odyssey4me suggestion at the time was that we never release OSA with the expectation of upgrades working from day 1, so that was kind at the story there.16:22
mnaserthe placement changes are pretty major, I would either release with them or without, I would not think making this back portable is good for our users16:23
evrardjpI agree with you, I just insist on deadlines16:24
mnaserif we agree that we can release without upgrades, then I think we should be (relatively) ok16:24
mnaserif we agree that a we must release with upgrades support, we might have to drop it16:24
mnaserI know odyssey4me mentioned that he volunteered to help with that upgrade effort.. so it'd be nice to hear if he can still do it (or not)16:25
evrardjpIf we don't provide the upgrade, it will be a hard mess for many, and I am not sure who will make the effort for doing so16:25
evrardjp(or when)16:25
*** mgoddard has quit IRC16:25
jrossercan we have a "placement is present but dormant" situation then if the migration tooling appears then we are good16:26
mnasernope, because it uses an entirely different database16:26
mnaserso if it does get deployed, it'll probably want to take over the service catalog entry too16:26
mnaserrunning the playbook will effectively update the service catalog entry to point @ the new placement16:27
evrardjpit makes sense to release stein with it, right?16:27
openstackgerritMerged openstack/openstack-ansible stable/rocky: Use opendev links  https://review.opendev.org/65993316:27
evrardjpit's in that cycle that it was extracted16:27
mnaserI think so.  we would save a lot of trouble for ourselves by doing it now than later16:27
evrardjpso it's easier for us to remember that's when you'll see it appear16:27
mnaserI can't remember but Jesse had a *very* good reason for including it in stein too16:28
mnaserI really don't remember it off the top of my head now16:28
logan-it seems like we're not saving ourselves trouble, it is the same amount of trouble but we're accelerating it for <some gain> (i'm not sure what the gain is yet)16:28
*** mgoddard has joined #openstack-ansible16:28
mnaseryes, that <some gain> was something Jesse mentioned that got me thinking at the PTG, not remembering though :(16:28
mnaserhopefully when he reads buffer, he can chime in16:28
evrardjpodyssey4me: ^16:29
evrardjpI guess I will propose an rc2 soon, but wait for the final release, and we can discuss this later.16:30
evrardjpthat's all I had for today16:30
logan-thanks evrardjp16:30
mnaserevrardjp: yes, I rechecked your most recent bump16:30
mnaserit passed check but failed gate so it should be ok.16:30
logan-on a separate topic, this is not urgent at all but I would like to get some thoughts on credential scoping in roles, I pushed some POC patches for nova. if these are adopted, this concept would likely apply to a number of other roles as well.16:31
logan-#link https://review.opendev.org/#/q/topic:fix-credentials-scoping16:31
* mnaser looks16:31
mnaserlogan-: I looked over that, I was hoping if you took maybe sometime to ask the nova/neutron team about possible side effects16:32
mnaserI'm thinking places where the resource was owned by X and now it is owned by Y16:32
mnasergenerally all openstack install guides have kinda historically mentioned to use users of the other service (but I totally agree with your approach tbh)16:32
logan-yep, gotcha, I will do that16:32
mnaserbut I think that's really neat, reduce the # of vars and it just makes sense that nova uses its own user to talk to neutron, rather than the neutron user16:33
evrardjplogan-: gosh this is amazing16:33
evrardjpall those vars will go away16:34
noonedeadpunkcan't agree more, looks nice:)16:34
mnaser-54 in group_vars/all is a win16:34
evrardjpimagine if the rest would be in a simple etcd? :p16:35
evrardjpor local facts?16:35
mnaser#thanks logan- awesome efforts in cleaning up and unblocking OSA gates16:35
openstackstatusmnaser: Added your thanks to Thanks page (https://wiki.openstack.org/wiki/Thanks)16:35
mnaser#thanks evrardjp dealing with OSA's painful release process!16:35
openstackstatusmnaser: Added your thanks to Thanks page (https://wiki.openstack.org/wiki/Thanks)16:35
logan-thanks for looking at those. glad the concept makes sense, it seems like the main concerns are upgrade impact, so I'll look into that some more16:36
mnaserthis should be interesting to watch - https://review.opendev.org/#/q/topic:osa/db-refactor16:36
mnaserI don't think I have time to push more on those but the 'general' concept is there16:36
guilhermespyeah I started t look at the topic today16:37
guilhermespjust did a small fix in linters but I haven't look yet at the failures16:37
guilhermespgonna be working on this16:37
mnaserjust needs a little bit of cleanup and then needs to be done across all roles, then we can use the proposal bot to keep it in sync :)16:37
mnaserbig win16:37
mnaserit also re-orders things so the db and rabbit are ready before we deploy16:38
mnaserno more crash error looping forever while we configure stuff16:38
evrardjpmnaser: commented on https://review.opendev.org/#/c/657029/216:38
logan-yeah that will be nice16:38
mnaserI'm going to be a bit 'more' back in service when I come back home on the 29th16:39
mnaserapologies for my MIA-ness, dealing with this fairly complex deployment and being in a different timezone/space/world16:39
evrardjpI think the reason we used different names for those tasks was to easily compare those, and see what this task was doing. But with ARA we don't need it anymore16:39
guilhermespI'd rename this as glance_db_sync https://review.opendev.org/#/c/657029/2/tasks/glance_db_setup.yml16:41
*** mgoddard has quit IRC16:42
guilhermespif this is a pattern across other roles btw16:42
evrardjpguilhermesp: +1 for me16:42
*** mgoddard has joined #openstack-ansible16:43
guilhermespbut that's not the case for nova tough https://github.com/openstack/openstack-ansible-os_nova/blob/master/tasks/nova_db_setup.yml :P16:43
openstackgerritMerged openstack/openstack-ansible stable/rocky: Add Calico networking AIO scenario  https://review.opendev.org/65971216:43
openstackgerritMerged openstack/openstack-ansible stable/queens: Use opendev links  https://review.opendev.org/65993616:43
mnaseryeah id work on the more simpler ones first and then clean up those other ones16:44
evrardjpguilhermesp: but we could be consistent in the future16:44
evrardjpmore consistent*16:44
guilhermespyeah evrardjp I agree refactoring stuff that makes more sense is a great first step16:44
*** hamzaachi_ has quit IRC16:46
jrosseransible 2.8.0 for master is very close to done, it needs the ceph-ansible 4.0.0 work to merge first though16:46
evrardjpjrosser: how is that last part?16:46
evrardjpI think it would be pretty early in the cycle to move to 2.8, so that's a good move16:46
jrosserthe ceph patch is looking reasonable, except that it fails tempest on some swift stuff16:47
jrosserit wasnt a very obvious failure, i bring this up in case anyone has an idea where to look next http://logs.openstack.org/03/656503/10/check/openstack-ansible-deploy-aio_ceph-ubuntu-bionic/3bfd9d8/logs/openstack/aio1_utility_container-78e37705/utility/stestr_results.html16:48
openstackgerritMerged openstack/openstack-ansible stable/queens: Restore linters job to voting  https://review.opendev.org/66001816:50
logan-jrosser: same ceph version as we currently run in master right?16:50
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-os_keystone master: Update uw_apache to run against bionic  https://review.opendev.org/66045516:52
jrosserlogan-: that patch brings it up to nautilus16:52
jrosserwhich is guess could be optional, perhaps i should try the same patch deploying mimic to separate the problems16:53
logan-if 4.0 supports mimic maybe we should split up the ceph-ansible upgrade from the ceph ugprade16:53
*** udesale has quit IRC16:53
openstackgerritMerged openstack/openstack-ansible stable/queens: Bump etcd role to v3 capable SHA  https://review.opendev.org/65986816:53
logan-because i suspect its a m->n radosgw keystone auth issue that will need to be worked out, probabyl unrelated to ceph-ansible v416:53
jrosseryes i think thats quite possible, i'll redo the patch just to bring ceph-ansible up to 4.0.016:54
*** goldenfri has joined #openstack-ansible16:57
openstackgerritJonathan Rosser proposed openstack/openstack-ansible master: Update to ceph-ansible 4.0.0  https://review.opendev.org/65650316:58
*** ivve has joined #openstack-ansible16:59
logan-thanks jrosser, if we can get ceph-ansible 4 in that will at least unblock the ansible 2.8 stuff and then we can work out the ceph upgrade separately. ill look thru the nautilius release notes to see if I can spot the change17:01
logan-last thing i've got is https://review.opendev.org/#/q/status:open+owner:logan2211%2540gmail.com+(topic:calico3+OR+topic:osa/opendev) -- only a few more reviews needed to finish up fixing the gates + getting calico both working and tested from Q->master (previously calico was working on P and earlier but broken Q->master)17:01
logan-jrosser: i wonder if the broken pipe requests in radosgw line up with the tempest fails http://logs.openstack.org/03/656503/10/check/openstack-ansible-deploy-aio_ceph-ubuntu-bionic/3bfd9d8/logs/openstack/aio1_ceph-rgw_container-2cdc0de2/ceph/ceph-rgw-aio1-ceph-rgw-container-2cdc0de2.rgw0.log.txt.gz17:13
logan-hopefully there is a debug mode where we can get some more output there because that's useless :/17:15
jrosseryes that’s possibly it17:15
jrosserThat should be on by default in for these tests if there is one17:15
*** openstack changes topic to "Launchpad: https://launchpad.net/openstack-ansible || Weekly Meetings: https://wiki.openstack.org/wiki/Meetings/openstack-ansible || Review Dashboard: http://bit.ly/2xA1eZC"17:15
openstackMeeting ended Tue May 21 17:15:53 2019 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)17:15
openstackMinutes:        http://eavesdrop.openstack.org/meetings/openstack_ansible_meeting/2019/openstack_ansible_meeting.2019-05-21-16.02.html17:15
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/openstack_ansible_meeting/2019/openstack_ansible_meeting.2019-05-21-16.02.txt17:15
openstackLog:            http://eavesdrop.openstack.org/meetings/openstack_ansible_meeting/2019/openstack_ansible_meeting.2019-05-21-16.02.log.html17:15
openstackgerritMerged openstack/openstack-ansible-os_swift master: Add integrated coverage  https://review.opendev.org/66028717:25
openstackgerritMerged openstack/openstack-ansible-os_magnum master: Add distro install jobs  https://review.opendev.org/66008617:26
*** kopecmartin is now known as kopecmartin|off17:31
openstackgerritMerged openstack/openstack-ansible-os_swift master: Clarify supported SUSE versions  https://review.opendev.org/66028817:31
*** vnogin has quit IRC17:34
*** electrofelix has quit IRC17:36
*** tuxjohnson has left #openstack-ansible17:36
openstackgerritMerged openstack/openstack-ansible-os_tacker master: Add integrated coverage  https://review.opendev.org/66028917:37
openstackgerritGeorgina Shippey proposed openstack/openstack-ansible master: Amend repo_all-back check to verify sync  https://review.opendev.org/66047717:41
*** priteau has quit IRC17:42
goldenfriHello all, If I update the provider network config and add provider groups, what do I need to re-run to deploy that to my compute nodes? setup-openstack doesn't seem to be enough17:47
goldenfrinevermind, found a typo17:51
guilhermespcores, that's something I forgot to mention in the meeting too... we are almost done with the role retirement process https://review.opendev.org/#/q/topic:drop_private+(status:open+OR+status:merged). I've seen ppl manifesting interest in maintain searchlight and monasca roles, but I understood that the ppl were trying to find time to confirm it17:52
guilhermespthat's why I proceeded with the retirement process, we have most of the other roles deleted17:53
guilhermespnow we just need to merge the open prs17:53
guilhermespI sent an email to the thread regarding the retirement but no replies. I believe if ppl really want to maintain, they could revert some of the patches to bring desired roles back to life17:54
*** hwoarang has quit IRC18:27
*** hwoarang has joined #openstack-ansible18:28
openstackgerritMerged openstack/openstack-ansible-tests stable/queens: Restore CentOS voting job  https://review.opendev.org/66016618:29
openstackgerritMerged openstack/openstack-ansible-rabbitmq_server master: Add integrated coverage  https://review.opendev.org/66039418:55
*** ansmith has joined #openstack-ansible19:04
*** ansmith_ has quit IRC19:05
*** gyee has quit IRC19:07
*** gyee has joined #openstack-ansible19:19
*** ansmith has quit IRC19:29
*** ansmith has joined #openstack-ansible19:32
*** kplant has quit IRC19:43
*** radeks_ has quit IRC19:48
*** logan- has quit IRC19:55
*** pnull has joined #openstack-ansible20:07
openstackgerritJonathan Rosser proposed openstack/openstack-ansible master: Update ansible to 2.8.0  https://review.opendev.org/65965620:08
*** ansmith has quit IRC20:08
*** ansmith has joined #openstack-ansible20:16
*** ansmith has quit IRC20:38
openstackgerritJonathan Rosser proposed openstack/openstack-ansible master: Improve repo server haproxy healthcheck to verify repo synchronisation  https://review.opendev.org/66047720:40
*** Talion has joined #openstack-ansible20:42
*** hamzy has quit IRC20:45
*** dave-mccowan has quit IRC20:50
*** dave-mccowan has joined #openstack-ansible20:56
*** aedc has quit IRC21:00
*** djhankb has quit IRC21:02
openstackgerritMerged openstack/openstack-ansible stable/stein: Bump SHAs for stable/stein  https://review.opendev.org/65996621:03
*** ut2k3 has joined #openstack-ansible21:06
ut2k3Hi guys, how can I tell OSA to install neutron-server on the controller node only as shown here: https://docs.openstack.org/security-guide/_images/1aa-network-domains-diagram.png - I've defined my two network-nodes  as `network_hosts` ... and removed already  via `inventory-manage.py -r LXC-container-name` but when I run again the `os-neutron-install.yml` it tries to install neutron-server again on the network-nodes.21:10
*** logan- has joined #openstack-ansible21:13
cloudnullut2k3 the neutron agents and server are connected in the environment config. to separate them you can create an env.d override file in /etc/openstack_deploy/env.d/ called neutron-server.yml with the following content - https://pasted.tech/pastes/b906226d245238d6c9c83d52fd60fd84cd9ce7aa - that will put the neutron server process (contianer) on the i21:16
cloudnullnfra hosts21:16
openstackgerritMerged openstack/openstack-ansible stable/queens: Zuul: Simplify the integrated test playbooks  https://review.opendev.org/66022421:17
ut2k3Awesome will check that21:17
cloudnullyou'll need to clean up the inventory with the `inventory-manage.py` script again, that override should get you there.21:17
ut2k3cloudnull, thank you for your help.21:21
cloudnullhope it works :D21:21
ut2k3I am just running it :)21:22
* cloudnull holds the beer 21:22
openstackgerritMerged openstack/openstack-ansible stable/queens: Add Calico networking AIO scenario  https://review.opendev.org/65986921:24
*** goldenfri has quit IRC21:37
*** dave-mccowan has quit IRC21:40
ut2k3Ok, next problem after upgrading to 19.0.0 / location movement of the neutron-server is now that it cries with " No 'neutron.service_plugins' driver found, looking for 'firewall'"21:48
*** sreejithp__ has quit IRC21:59
*** sreejithp has joined #openstack-ansible21:59
openstackgerritMerged openstack/ansible-role-systemd_mount master: Add integrated coverage  https://review.opendev.org/66040822:03
*** sreejithp has quit IRC22:04
openstackgerritMerged openstack/ansible-role-python_venv_build master: Add integrated coverage  https://review.opendev.org/66041122:06
openstackgerritGuilherme  Steinmuller Pimentel proposed openstack/openstack-ansible-os_glance master: db_setup: refactor database setup to a common file  https://review.opendev.org/65702922:10
cloudnullut2k3 did it work ?22:11
ut2k3Yep it worked. Having now another problem with the neutron-server but I am quite sure thats not related.22:12
ut2k3> Error loading class by alias: NoMatches: No 'neutron.service_plugins' driver found, looking for 'firewall'22:12
cloudnulllikely the firewall driver name / path has changed22:13
cloudnullI assume you had firewall as a service running22:13
cloudnulllikely need to figure out whatever the new driver name / path is and update that in your overrides.22:14
ut2k3Yep, will Digg into it.22:14
cloudnullI generally fire-up the venv python interpreter using ipython and go poke around to figure things like that out22:15
cloudnullif its not easy to find in the upstream docs that is22:15
ut2k3Ok good idea22:16
openstackgerritMerged openstack/openstack-ansible-rsyslog_client master: Add integrated coverage  https://review.opendev.org/66040322:17
openstackgerritMerged openstack/openstack-ansible-rsyslog_server master: Add integrated coverage  https://review.opendev.org/66040522:19
*** nsmeds_ has joined #openstack-ansible22:21
openstackgerritMerged openstack/ansible-role-systemd_networkd master: Add integrated coverage  https://review.opendev.org/66040922:24
*** hamzaachi has joined #openstack-ansible22:25
*** hamzaachi has quit IRC22:26
*** hamzaachi has joined #openstack-ansible22:27
*** hamzaachi has quit IRC22:28
openstackgerritMerged openstack/ansible-role-systemd_service master: Add integrated coverage  https://review.opendev.org/66040722:28
*** hamzaachi has joined #openstack-ansible22:28
*** tosky has quit IRC22:42
*** hamzaachi has quit IRC22:44
*** hamzaachi has joined #openstack-ansible22:45
*** hamzaachi has quit IRC22:45
*** hamzaachi has joined #openstack-ansible22:46
*** Talion has quit IRC22:52
*** hamzaachi has quit IRC22:54
*** pnull has quit IRC23:03
*** pnull has joined #openstack-ansible23:05
*** sreejithp has joined #openstack-ansible23:09
*** sreejithp_ has joined #openstack-ansible23:11
*** sreejithp has quit IRC23:14
*** joshualyle has joined #openstack-ansible23:14
*** ut2k3 has quit IRC23:19
*** luksky has quit IRC23:26
*** macza has quit IRC23:39
*** macza has joined #openstack-ansible23:40
*** macza has quit IRC23:47

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!