Tuesday, 2018-08-07

*** mmalchuk has joined #openstack-ansible00:05
*** gyee has quit IRC00:06
*** mmalchuk has quit IRC00:09
*** guilhermesp has quit IRC00:25
*** guilhermesp has joined #openstack-ansible00:25
*** mmalchuk has joined #openstack-ansible00:35
*** mmalchuk has quit IRC00:39
*** mmalchuk has joined #openstack-ansible00:48
*** mmalchuk has quit IRC00:52
openstackgerritLogan V proposed openstack/openstack-ansible stable/pike: Revert "Work around nested virt issues on OVH test nodes"  https://review.openstack.org/58930900:55
openstackgerritLogan V proposed openstack/openstack-ansible stable/queens: Revert "Work around nested virt issues on OVH test nodes"  https://review.openstack.org/58931000:55
*** gregoryo has joined #openstack-ansible00:55
*** mmalchuk has joined #openstack-ansible01:01
*** mmalchuk has quit IRC01:06
openstackgerritKevin Carter (cloudnull) proposed openstack/openstack-ansible-os_neutron master: Cleanup files and templates using smart sources  https://review.openstack.org/58864701:09
*** mmalchuk has joined #openstack-ansible01:14
*** mmalchuk has quit IRC01:19
*** mmalchuk has joined #openstack-ansible01:53
*** mmalchuk has quit IRC01:57
*** yolanda has quit IRC02:03
*** mmalchuk has joined #openstack-ansible02:05
*** mmalchuk has quit IRC02:10
*** lbragstad has quit IRC02:18
*** mmalchuk has joined #openstack-ansible02:18
*** mmalchuk has quit IRC02:22
*** spatel has joined #openstack-ansible02:34
*** cCain has joined #openstack-ansible02:43
cCaino/02:44
*** coolj has joined #openstack-ansible02:56
openstackgerritMerged openstack/openstack-ansible-os_nova master: Mark bionic gate jobs as non-voting  https://review.openstack.org/58916103:08
*** mmalchuk has joined #openstack-ansible03:09
*** mmalchuk has quit IRC03:14
*** jhesketh_ is now known as jhesketh03:14
*** mmalchuk has joined #openstack-ansible03:22
*** mmalchuk has quit IRC03:27
*** dave-mccowan has quit IRC03:31
*** mmalchuk has joined #openstack-ansible03:34
*** spatel has quit IRC03:37
*** mmalchuk has quit IRC03:38
*** udesale has joined #openstack-ansible03:48
*** mmalchuk has joined #openstack-ansible03:52
*** mmalchuk has quit IRC03:56
*** mmalchuk has joined #openstack-ansible04:05
*** mmalchuk has quit IRC04:09
*** poopcat has quit IRC04:09
*** Bhujay has joined #openstack-ansible04:13
*** Bhujay has quit IRC04:30
*** pdeters has joined #openstack-ansible04:59
*** shyambiradar has joined #openstack-ansible05:12
*** guilhermesp has quit IRC05:14
*** mmalchuk has joined #openstack-ansible05:33
*** shyambiradar has quit IRC05:39
*** shyambiradar has joined #openstack-ansible05:39
*** nicolasbock has joined #openstack-ansible05:46
*** hwoarang has joined #openstack-ansible05:57
*** cCain has quit IRC05:59
*** fghaas has joined #openstack-ansible06:05
*** dxiri has quit IRC06:08
*** DanyC has joined #openstack-ansible06:13
hwoaranggood morning06:18
prometheanfiremoin06:22
prometheanfirehwoarang: you going to the ptg?06:22
hwoarangnope06:22
prometheanfireboo06:26
prometheanfiredon't want a visit from thomas?06:26
prometheanfirehttps://media0dk-a.akamaihd.net/39/57/7518bfcf2a4996b3dd5d34c20c4d1101.jpg06:27
hwoaranghaha i've heard it will not be there this year :)06:29
hwoarangevrardjp: good morning. can you help me understand your comment on https://review.openstack.org/#/c/584765/ so i can 'upgrade' your vote? :)06:30
evrardjpgood morning06:30
evrardjphwoarang: let's not lose CI time, but if you need to update this, we now define the voting: false or voting: true arguments in projects.yaml not in jobs.yaml06:31
hwoarangoh ok06:31
evrardjpproject*06:31
hwoarangok will do that from now on06:32
*** pcaruana has joined #openstack-ansible06:36
prometheanfireevrardjp: hwoarang I've been bugged about https://review.openstack.org/556586 so passing the bug on to you like any good office env06:37
jrosseri have a couple moreof these ready to go if anyone has a moment https://review.openstack.org/#/q/topic:bionic-check+(status:open)06:39
jrosserevrardjp: also interested in your comments on this https://review.openstack.org/#/c/589146/ after the discussion at the last meeting06:40
*** yolanda has joined #openstack-ansible06:40
* hwoarang grabs a coffee and will be back for reviews06:41
*** Bhujay has joined #openstack-ansible06:42
*** DanyC has quit IRC06:43
*** vnogin has quit IRC06:47
*** hamzaachi has joined #openstack-ansible06:53
*** luksky has joined #openstack-ansible06:54
*** phasespace has joined #openstack-ansible06:56
phasespaceWhen upgrading between major versions, is it possible to skip one version? I.e. is it possible to upgrade from ocata to queens directly, or should we first upgrade to pike, then queens?06:56
prometheanfirephasespace: gernerally possible to skip a single version, ymmv06:57
*** shyambiradar has quit IRC07:00
*** shyambiradar has joined #openstack-ansible07:00
phasespaceOk, thanks. Is a major upgrade usually fraught with difficulty? Are there any particular recommendations on how to proceed?07:01
phasespaceI.e. if I'm on an old version of ocata, should I do a minor upgrade before the major upgrade?07:02
prometheanfirea single upgrade is generally ok, it's best to do them one at a time07:03
prometheanfireand best to be up to date before the major07:03
prometheanfireall extra work of course07:03
phasespaceI can deal with more work if it's safer07:04
*** gkadam has joined #openstack-ansible07:04
phasespaceThank you very much, prometheanfire07:04
prometheanfirenp07:08
prometheanfirelike I said, ymmv, ianal, etc.  You need to test to know how your infra works07:09
*** shyambiradar has quit IRC07:13
evrardjpprometheanfire: I will have a look07:16
prometheanfirethanks07:16
phasespaceWhat are your experiences with major upgrades? Did it go mostly without a hitch, or did most of you experience some issues that needed to be resolved?07:18
prometheanfiregoing from N to N+1 works fine in my experience, but that was mostly simple testing07:23
*** shyambiradar has joined #openstack-ansible07:23
*** phasespace has quit IRC07:26
*** phasespace has joined #openstack-ansible07:26
jrosserevrardjp: could you explain for me a bit more when we would see an lxc2->lxc3 upgrade?07:35
*** luksky has quit IRC07:41
evrardjp[m]queens 16.04 -> rocky 16.04 -> rocky 18.0407:42
*** tosky has joined #openstack-ansible07:42
jrosserbut we can't do rocky 16.04 can we?07:44
jrosseroh sorry yes we can07:44
*** rmart04 has joined #openstack-ansible07:44
evrardjp[m]rocky 16 to 18 will be up to the deployer but we'll use a different code to generate the lxc config07:44
evrardjp[m]which will put containers down07:45
jrosserand that would involve an OS reinstallation07:45
evrardjp[m]it's acceptable imo because system upgrades anyway07:45
evrardjp[m]but the code itself was not necessary if it was everything generated with lxc2 and if platform is lxc3 do the config upgrade using the tooling.07:46
jrosserwhich would youo prefer?07:46
evrardjp[m]i trust your judgment07:47
evrardjp[m]you have taken time to get this in, scratching it would be costly.07:47
*** shyambiradar has quit IRC07:47
evrardjp[m]s/scratching/abandonning/07:48
*** pdeters has quit IRC07:55
TahvokDoes it mean after upgrading to rocky we would need to insert another controller with ubuntu 18 and then transfer one by one all other controllers to 18 as well?08:07
TahvokI mean, if you want to go the safe path and not be left with 2 controllers while doing an upgrade to the third..08:07
*** DanyC has joined #openstack-ansible08:17
*** DanyC has quit IRC08:18
prometheanfireevrardjp: we support rocky on 16.04?08:19
prometheanfirethought that migration was supposed to take place in queens (nice if so)08:19
evrardjpprometheanfire: rocky will be 16.04 for source installs only.08:20
*** DanyC has joined #openstack-ansible08:20
evrardjprocky is the jump platform,like Newton was08:20
evrardjpTahvok: in that case I'd add 3 controllers, do the 18.04 upgrade, kill the 16.04 controllers, but that depens on the users I guess.08:21
prometheanfirek08:21
evrardjpTahvok: let me rephrase08:22
evrardjpif you have 3 16.04 controllers, you can upgrade them (if you're running from source) to 16.04 rocky.08:23
evrardjpthen, kill one those, you still have a cluster working.08:23
evrardjpthen you add 3 new 18.04 controllers, join to the system.08:23
evrardjpthen kill the remaining 2 controllers.08:24
evrardjpthat's how I'd do it. But it's not straightforward as usual.08:24
evrardjpThe other option is to live upgrade.08:24
evrardjpif you live upgrade and convert the containers to run on lxc3, they would still be just fine08:24
evrardjpyou can then flush facts and re-deploy08:25
evrardjpmaybe destroy on the way to get new 18.04 containers.08:25
* prometheanfire prefers expand/contract, but that takes extra infra08:25
evrardjpyeah.08:26
evrardjpI find it cleaner too prometheanfire08:26
evrardjpit's harder though08:26
prometheanfirecan be, ya08:27
prometheanfirethough side effects of in place upgrade can hit you at odd times08:27
*** electrofelix has joined #openstack-ansible08:33
hwoarangthe lxc3 is a surprise i think08:35
*** aludwar has quit IRC08:36
hwoarangwebsite says "For production environment, try to stick to LXC 1.0.x or 2.0.x as these are the long term, stable releases which we will support until April 2019 (1.0.x) or April 2021 (2.0.x)."08:36
hwoarangthey even have ubuntu repos for these releases. would it be completely wrong to stick with 2.0.X on bionic?08:37
openstackgerritJean-Philippe Evrard proposed openstack/openstack-ansible master: Replace tempest with healthchecks  https://review.openstack.org/58754408:37
*** aludwar has joined #openstack-ansible08:37
openstackgerritMarkos Chandras (hwoarang) proposed openstack/openstack-ansible master: [docs] Add documentation for 'install_method' variable  https://review.openstack.org/58939808:37
openstackgerritMarkos Chandras (hwoarang) proposed openstack/openstack-ansible master: [docs] Add documentation for 'install_method' variable  https://review.openstack.org/58939808:39
jrosserhwoarang: it doesnt look like there is a lxc2 for bionic though? https://launchpad.net/~ubuntu-lxc/+archive/ubuntu/lxc-stable08:42
hwoaranghmm true :/08:44
hwoarangmisleading info on the website i suppose08:44
openstackgerritMerged openstack/openstack-ansible-os_designate master: Mark bionic gate jobs as non-voting  https://review.openstack.org/58916008:45
*** shyambiradar has joined #openstack-ansible08:47
TahvokWhy add 3 controllers instead of adding only one, then reinstalling the rest one by one?08:49
TahvokOnly look at infra side, you would need less machines in that case...08:50
*** luksky has joined #openstack-ansible08:51
jrosserhwoarang: could you take a look at https://review.openstack.org/#/c/587532/ and https://review.openstack.org/#/c/587531/ if you have a moment?08:52
hwoarangsure08:54
*** priteau has joined #openstack-ansible08:55
evrardjphwoarang: jrosser let me ask on charms channel08:59
PradiprwtHi Everyone, Is anyone aware of this issue >> fatal: [infra1_utility_container-550bfa83]: FAILED! => {"failed": true, "msg": "No file was found when using with_first_found. Use the 'skip: true' option to allow this task to be skipped if no files are found"}09:06
hwoarangPradiprwt: what distro? looks like you are running on a distro that is not known to OSA09:07
hwoarangPradiprwt: what playbook? we need some more info09:07
openstackgerritMerged openstack/openstack-ansible master: zuul: Add aio_lxc jobs for distribution installations  https://review.openstack.org/58476509:07
PradiprwtI am using ubuntu 16.0409:07
Pradiprwthwoarang: verbose output >> http://paste.openstack.org/show/727476/09:08
hwoarangPradiprwt: what OSA version?09:09
*** gregoryo has quit IRC09:09
Pradiprwt16.0.1609:09
Pradiprwtpike09:09
TahvokPradiprwt, hwoarang: had exactly this problem 2 weeks ago09:11
TahvokThe only think that helped was to recreate the container09:11
evrardjpPradiprwt: are your facts up to date?09:11
evrardjpPradiprwt: it looks like you didn't have the proper facts.09:12
Pradiprwtevrardjp: yes, saturday i have stated deployement09:12
evrardjpCreate a play with setup: as argument, and run it with openstack-ansible and it should gather those facts.09:12
evrardjpnot sure what you mean there09:13
jrosseroh thats not good - re-running boostrap-ansible.sh shouldnt stacktrace should it?09:15
Pradiprwtshould I re-create the container ?09:16
evrardjpjrosser: nope -- could you paste somewhere?09:18
evrardjpSetuptools?09:18
evrardjpthat looks fun!09:19
jrosserevrardjp: http://paste.openstack.org/show/727477/09:19
evrardjpjrosser: remove your pip.conf : p09:20
evrardjpbut wait09:20
evrardjpis that done with bootstrap-ansible.sh ?09:20
jrosseryes, see the first line09:21
evrardjpthat's interesting09:21
evrardjpyou have the osa venv activate?09:21
evrardjpactivated* ?09:21
jrosser?09:21
evrardjpok I think we miss an --isolated flag on L19709:22
evrardjplet me fix that quick09:22
jrosseri'm just following the aio quickstart - that says nothing about venv09:22
evrardjpoh but we have.09:23
evrardjpwhivh version is that?09:23
jrosserhttps://docs.openstack.org/openstack-ansible/latest/user/aio/quickstart.html09:24
evrardjpoh well I guess you have an old milestone?09:24
evrardjpmmm09:25
evrardjpI don't think this can happen on following the docs on a clean node09:25
evrardjpdid you follow: https://docs.openstack.org/openstack-ansible/latest/user/aio/quickstart.html#rebuilding-an-aio ?09:25
jrosserahha - that'll be it09:26
jrosserthough i thought i could pull latest roles with bootstrap-ansible.sh on its own09:26
jrosser^ which is what i wanted to do - just pull in the lastest merges to the roles09:27
evrardjpit would, but you deployed already09:28
evrardjpso it means you still have the pip.conf09:28
jrosseryeah, i see that now :)09:28
evrardjpweirdly that should get ignored because --isolated.09:29
evrardjpwhich code do you run?09:29
evrardjpprecisely09:29
jrosserthis is master09:29
evrardjpsha?09:29
jrosserbd00e7c1ab8e8e7e04d1d505deb611fac5c7fa5109:29
evrardjpthat's very weird09:30
Pradiprwtevrardjp: How to create a play with setup: as argument, to gather facts ?09:32
evrardjpjrosser: L193 is super scary: http://paste.openstack.org/show/727477/09:34
*** guilhermesp has joined #openstack-ansible09:35
Pradiprwti have deleted the container but giving same error while recreating >> http://paste.openstack.org/show/727478/09:37
evrardjpjrosser: ok09:39
evrardjpPradiprwt: please gather facts like I suggested.09:40
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-os_almanach master: Switch to using project-templates  https://review.openstack.org/58753109:40
evrardjpjrosser: ok could you do something for me on your interesting node ?09:40
jrossersure09:40
jrosseroh well i did just move the pip.conf aside and reboostrap09:41
evrardjp(assuming you still have the pip.conf that is preventing stuff)09:41
jrosseri still have the pip.conf though just renamed09:41
evrardjpcould you put it back and try this command ?09:41
evrardjpvirtualenv --python=/usr/bin/python2 --clear --never-download /opt/ansible-runtime09:41
Pradiprwtevrardjp: I don't know how to do that, can you please give me some guide or refrence09:42
evrardjpPradiprwt: I told you above.09:42
jrosserevrardjp: http://paste.openstack.org/show/727479/09:42
evrardjphahah it works09:42
evrardjpjrosser: great!09:43
evrardjp #victorydance09:43
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_octavia master: Ensure the octavia flavor is private  https://review.openstack.org/58923909:43
evrardjplet me check what this code does, and since when it's shipped :D09:43
evrardjpjrosser: could you try something else real quick too?09:44
jrossersure no problem09:45
evrardjp1) re-running this without the --never-download, see if it fails09:45
jrosserstactrace09:45
noonedeadpunkodyssey4me are you here?09:45
evrardjp2)  re-running with --no-pip --no-setuptools --no-wheel09:45
*** fghaas has quit IRC09:46
evrardjpafter step 2, you need to run09:46
jrosserhold on - does 2) have --never-download or not?09:46
evrardjp/opt/ansible-runtime/bin/pip install --isolated --upgrade pip setuptools wheel09:46
evrardjp2) does not have never download09:46
odyssey4meevrardjp jrosser I'm guessing this is a new complexity after merging the change to use packages?09:47
evrardjpodyssey4me: not really09:47
evrardjpit's an upgrade case09:47
odyssey4meah ok, well good thing we catch it now :)09:47
evrardjpvirtualenv is taking the pip.conf09:47
evrardjpin upgrades we remove the pip.conf09:47
evrardjpI think there is a way to avoid this09:47
evrardjpwith virtualenv09:47
evrardjp(during virtualenv bootstrap)09:48
odyssey4meis this for queens, or rocky?09:48
evrardjprocky09:48
jrosserevrardjp: http://paste.openstack.org/show/727481/09:48
evrardjpI guess we can consider it a regression09:48
evrardjpjrosser: as expected.09:49
evrardjpgreat.09:49
evrardjpyou've confirmed things for me, that's perfect!09:49
evrardjplet me check those options09:49
openstackgerritMarkos Chandras (hwoarang) proposed openstack/openstack-ansible master: [docs] Add documentation for 'install_method' variable  https://review.openstack.org/58939809:49
odyssey4meyeah, you can't do no-pip and expect pip to be there... and we don't want to use get-pip.py to put it there any more09:50
*** shyambiradar has quit IRC09:50
evrardjpodyssey4me: oh fyi: https://virtualenv.pypa.io/en/stable/reference/#creating-your-own-bootstrap-scripts09:50
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-os_almanach master: Switch to using project-templates  https://review.openstack.org/58753109:50
evrardjpodyssey4me: yeah so my idea was that virtualenv ships with those packages anyway09:50
evrardjppip setuptools etc.09:50
odyssey4meyes it does, and we resolved to use those in your patch09:51
evrardjpbut virtualenv itself is a command that respects pip and tries to download instead of using what's shipped inside09:51
evrardjpso --never-download uses its own code09:51
*** shyambiradar has joined #openstack-ansible09:51
evrardjpI will check which version introduced it (I think it's a behavior change based on versions of virtualenv, it was default before, and is not the default anymore)09:52
odyssey4meah yes, I see that - I think we did use that once09:52
odyssey4meI vaguely recall that perhaps one of the distributions didn't support it... I guess we'll see09:53
*** rmart04 has quit IRC09:53
odyssey4meoh no, I'm mixing it up with the '--always-copy' flag09:54
odyssey4mehttps://github.com/openstack/openstack-ansible-os_keystone/blob/master/tasks/keystone_install_source.yml#L97-L9809:55
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-os_congress master: Switch to using project-templates  https://review.openstack.org/58753209:57
evrardjpodyssey4me: so never download in the code seem to have been through a series of default configurations09:57
odyssey4meevrardjp hwoarang jrosser so, I'm a little confused about https://review.openstack.org/#/c/587532/4/zuul.d/project.yaml09:57
evrardjpin 13.1.2 (minimum version of suse), it seems never download exists but gets ignored as it will not download. However latest version of virtualenv is different, so I am checking where and why it reappeared.09:58
odyssey4meby using the template, and then overriding the config to make jobs non-voting, we're essentially making a bunch of jobs run in gate which aren't voting - which is a waste of CI resources09:58
jrosserthis is my fault, sorry09:58
odyssey4meWith these repositories, does it not make sense to rather not use the template?09:58
odyssey4mePerhaps rather just implement jobs in check that are non-voting with a note to use the template when they are?09:59
jrosserconverting to the template has bought in the bionic jobs where previously they were missing09:59
jrosserand then a bunch of collatoral because the other distros are all brok for that role09:59
odyssey4mejrosser: yeah, understood10:00
odyssey4meperhaps we should be changing the templates - something like a template per distro?10:00
odyssey4mesorry, you've been hacking away at this and my questions are coming from the side10:00
jrosserhowever - for the vast majority of them the brok is actually quite small10:00
jrossermissing vars and the like for opensuse seems quite common10:01
jrosseri really don't mind which way this goes tbh10:02
odyssey4meyep, there's never been the effort made to get those roles to support more distributions - many of them are new, and many are just fringe roles10:02
evrardjpodyssey4me: https://github.com/pypa/virtualenv/commit/f50df26c44311390351d0a76c79cfc136241ece5 and https://github.com/pypa/virtualenv/commit/3d7361ff2e31472cb69d00150fbdf5a3c9af2a0d -- suse has 13.1.2 which doesn't include this code, so no defaults were applied because old code auto never downloads.10:02
evrardjpinteresting.10:02
odyssey4mein that patch in particular, you could just be adding 'openstack-ansible-functional-ubuntu-bionic' as a job in check and gate and move on - it's simple and focused on the intention of your work which is to get bionic tested and working10:02
odyssey4methe conversion to use templates sems to me a distraction10:03
odyssey4me*seems10:03
jrosseri can do that10:03
noonedeadpunkodyssey4me: lxc_containers_create fails on this step https://github.com/openstack/openstack-ansible-lxc_container_create/blob/b2020d5c10b7def43490fc192eb70969bf80517a/tasks/lxc_container_config.yml#L71, if it's run not from root user. If I run test by root, everything is okey. And the interesting thing is, that containers created by non-root user doesn't have any ethernet interface (probably they are cleaned by "Remove legacy10:03
evrardjpodyssey4me: I think we shouldn't make it non voting10:03
noonedeadpunknetwork config" step). But why it's happening only for non-root user - is a question for me now.10:03
odyssey4mewell, I'd like thoughts on the matter from hwoarang and evrardjp10:03
evrardjpwe jsut break stuff and say "please fix it if that matters to you10:03
evrardjphahah10:03
odyssey4meevrardjp: that's the point - by forcing the template to be used we're adding a bunch of tests to gate which shouldn't be there10:04
evrardjpodyssey4me: I don't mean changing the project.10:04
odyssey4meI'm thinking it'd be better to just add the bionic job and leave converting the role to use the templates out of it10:04
evrardjpodyssey4me: I think new works should be in their own template.10:05
evrardjpso for example, bringing bionic support would be in its own template.10:05
evrardjpthe point of using a template is to make sure consistency happens (to avoid the horizon problem we had earlier)10:05
evrardjp(as a reminder: horizon was voting for bionic, but keystone was not, and a patch on keystone merged that broke bionic)10:06
*** kaiokmo has quit IRC10:06
odyssey4meok, I have no idea what you mean10:06
evrardjphaha10:07
evrardjplet me rephrase that10:07
odyssey4medoing a template for a single release seems silly to me - it's a single job10:07
evrardjpwhile I think setting up jobs in each repo for new work is fine, and was the way to go, maybe we should think differently10:07
odyssey4medoing it for a single distro seems fine to me - xenial & bionic, opensuse x and y10:07
odyssey4mecentos of course is as old as the hills and doesn't have another release10:07
odyssey4mebut meh, I don't think how we use the templates should be part of jrosser's work to get bionic done... it's a distraction10:08
jrossermy priority right now is to get bionic roles voting10:08
evrardjpagreed, if ppl respect his work.10:08
odyssey4meit can be improved or re-modeled after as a seperate body of work10:08
odyssey4meso for now, I suggest, just adding the single job to any repo that doesn't use the template10:09
evrardjpjrosser: ^10:09
jrosserok well it's mostly all done, and via the template10:09
jrosserthose which arent merged i can change10:09
odyssey4meyep, works for me10:09
evrardjpjrosser: those who fail shouldn't be done using the template10:10
odyssey4methe shortest possible path to success10:10
evrardjpyeah.10:10
jrosserspeaking of which this needs moving on https://review.openstack.org/#/c/589158/10:10
jrosserhttps://review.openstack.org/#/c/586756/10:10
jrosserlikewise unless we want to edit them10:11
jrosseri was asked to get the broken stuff non-voting *before* we switch the majority to voting so this is critical path10:11
odyssey4meto me https://review.openstack.org/#/c/586756/10/zuul.d/project.yaml makes sense if someone's going to pick up making that role with with opensuse/centos very soon10:12
odyssey4meif not, we should not be making it use the template10:12
odyssey4methat's far too many jobs in gate which aren't voting10:12
jrosseras that looks just like missing vars i agree10:12
evrardjpsteps were simple before the templates: we added non voting jobs, we made them voting.10:13
evrardjpand added them to gates.10:13
evrardjpwe should keep the same path in the future.10:13
evrardjpand when it's voting on all our "core" repos part of an aio deploy, then move this to template.10:13
evrardjpthen when IN the template, it means it's an expected thing for minimum roles in the future. That means part of the maturity checking.10:14
evrardjpit seems like a story when you do it like that.10:14
evrardjpit helps clarify what is the maturity level of a role10:14
evrardjpif the role doesn't pass that maturity level, then we downgrade it, according to existing procedures.10:15
evrardjpwe evince those roles from the integrated build, and they become "community extras"10:15
odyssey4mefair enough - enough pontificating, what's the actionable outcome for jrosser ;)10:15
evrardjpodyssey4me: we need a clear story for docs.10:16
evrardjpthat looks acceptable to me.10:16
evrardjpFor jrosser, everything that's not ready for template should not use template.10:16
evrardjplike TODO(jrosser): re-use template when x is finally passing.10:17
evrardjpbut I am not 100% aware of the current state, so I trust the current status, and we'll clarify in the long run10:17
odyssey4methe exception being if the role is only not passing bionic because that's the subject of this body of work10:17
evrardjpjrosser: do you have an update on the etherpad on what's using templates?10:17
evrardjpodyssey4me: so because we introduced (my bad) template too soon, we need to deal with this.10:18
jrosserno, but you can see here https://review.openstack.org/#/q/topic:bionic-check+(status:open+OR+status:merged)10:18
*** jamesdenton has quit IRC10:18
jrosserthats what has changed, and mostly incorrectly10:18
*** electrofelix has quit IRC10:18
odyssey4meso, for example, I think https://review.openstack.org/589158 is perfectly ok - because I know that state will not last long given that bionic is the target of jrosser's work10:19
jrossercongress is totall madness though for example10:19
odyssey4meif the move to using templates has merged already, then I guess we carry a TODO once bionic is done to go back and either make those other platforms work, or switch the job config back10:19
jrosseri had planned to pass across this again after bionic and pick up the low hanging fruit of the other distros10:20
odyssey4methanks jrosser - appreciate the work you're putting into this!10:20
evrardjpyeah that's very appreciated.10:21
jrosserno worries - feels like im missing a bunch of assumed knowledge/process though :/10:21
evrardjpinteresting, almanach is passing suse?10:21
evrardjphttps://review.openstack.org/#/c/587531/6/zuul.d/project.yaml10:22
odyssey4mejrosser: it may not be assumed, it may also be that we're constantly learning - but if there is something that you find is assumed or newly learned, we should document it10:22
odyssey4mealmanach does almost nothing, so that's not all that surprising10:22
jrosseroh something has gone all a bit wrong there?10:25
jrosserlet me fix that10:26
*** olivierbourdon38 has joined #openstack-ansible10:26
*** electrofelix has joined #openstack-ansible10:26
openstackgerritJean-Philippe Evrard proposed openstack/openstack-ansible-os_magnum master: Switch to using project-templates  https://review.openstack.org/58838010:31
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-os_almanach master: Add CI jobs for bionic  https://review.openstack.org/58753110:32
hwoarangodyssey4me: leaving for lunch now so I will check backlog a bit later if that's OK10:33
hwoarangi saw some highlights there10:34
*** shyambiradar has quit IRC10:34
openstackgerritJean-Philippe Evrard proposed openstack/openstack-ansible-os_congress master: Add CI jobs for bionic  https://review.openstack.org/58942310:36
*** jamesdenton has joined #openstack-ansible10:37
openstackgerritJean-Philippe Evrard proposed openstack/openstack-ansible-os_magnum master: Switch to using project-templates  https://review.openstack.org/58838010:38
openstackgerritMaksim Malchuk proposed openstack/openstack-ansible master: Add an ability to use custom PyPi index for deploy  https://review.openstack.org/57765210:44
*** sm806 has joined #openstack-ansible10:47
*** shyambiradar has joined #openstack-ansible10:49
*** vnogin has joined #openstack-ansible10:49
*** fghaas has joined #openstack-ansible10:54
*** shyambiradar has quit IRC10:54
*** bradm has quit IRC10:56
openstackgerritJames Denton proposed openstack/openstack-ansible-os_neutron master: Adds networking-ovn support  https://review.openstack.org/58406711:03
*** galaxyblr has joined #openstack-ansible11:04
*** ansmith has quit IRC11:06
jrosseris this more broken log collection? https://github.com/openstack/openstack-ansible/blame/master/scripts/scripts-library.sh#L20411:08
jrosserbecause this http://logs.openstack.org/07/586607/8/check/openstack-ansible-deploy-aio_lxc-ubuntu-bionic/f28ca34/job-output.txt.gz#_2018-08-07_07_15_13_79538411:09
*** cshen has joined #openstack-ansible11:14
odyssey4mejrosser: yeah, wrong function name again: https://github.com/openstack/openstack-ansible/blob/master/scripts/scripts-library.sh#L14211:14
*** udesale has quit IRC11:15
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible stable/queens: Fix gate log collection  https://review.openstack.org/58942711:17
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible stable/pike: Fix gate log collection  https://review.openstack.org/58942811:17
jrosseri'm just fixing this next one11:17
*** Bhujay has quit IRC11:18
openstackgerritJonathan Rosser proposed openstack/openstack-ansible master: Fix log collection  https://review.openstack.org/58943011:22
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_congress master: Add CI jobs for bionic  https://review.openstack.org/58942311:22
odyssey4mejrosser: quick commit subject suggestion there11:31
openstackgerritJonathan Rosser proposed openstack/openstack-ansible master: Fix log compression  https://review.openstack.org/58943011:34
*** vnogin has quit IRC11:36
*** shyambiradar has joined #openstack-ansible11:41
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_cinder master: Add qdrouterd user to mq setup  https://review.openstack.org/58923411:41
odyssey4mecould I get another set of eyes on https://review.openstack.org/589234 please as I'd like to roll that into the changes I'm making to the roles11:43
*** guilhermesp has quit IRC11:45
*** guilhermesp has joined #openstack-ansible11:46
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_aodh master: Allow tags to be used for MQ tasks  https://review.openstack.org/58943711:46
*** vnogin has joined #openstack-ansible11:46
openstackgerritMerged openstack/openstack-ansible-os_almanach master: Add CI jobs for bionic  https://review.openstack.org/58753111:47
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_barbican master: Allow tags to be used for MQ tasks  https://review.openstack.org/58943811:49
jrosserthe artefact sync at the end of a job is very slow - coming up on 10mins - is that normal?11:50
odyssey4mejrosser: for roles, I noticed that the journal extraction is taking much longer11:51
jrosserit took 5 mins to copy the ara sqlite11:51
odyssey4meever since it was added, the log collection is much, much slower11:51
odyssey4mefor the integrated build we don't have that though11:51
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_ceilometer master: Allow tags to be used for MQ tasks  https://review.openstack.org/58944011:53
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_cloudkitty master: Allow tags to be used for MQ tasks  https://review.openstack.org/58944111:55
*** vnogin has quit IRC11:56
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_congress master: Allow tags to be used for MQ tasks  https://review.openstack.org/58944211:58
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_cloudkitty master: Allow tags to be used for MQ tasks  https://review.openstack.org/58944111:58
openstackgerritMerged openstack/openstack-ansible-os_congress master: Use generic vars file for ubuntu  https://review.openstack.org/58921511:58
*** Emine has joined #openstack-ansible12:00
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_octavia master: Ensure the octavia flavor is private  https://review.openstack.org/58923912:01
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_octavia master: Ensure the octavia network setup is in the right project  https://review.openstack.org/58924012:01
jrosserodyssey4me: TASK [Run log collection script] -> 20mins on centos!12:02
openstackgerritMerged openstack/openstack-ansible-os_tempest master: Mark bionic gate jobs as non-voting  https://review.openstack.org/58915812:03
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_octavia master: Ensure the octavia network setup is in the right project  https://review.openstack.org/58924012:04
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_octavia master: Add retries to service role creation  https://review.openstack.org/58924512:04
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_octavia master: Use interface instead of endpoint_type  https://review.openstack.org/58924612:05
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_octavia master: Clean up tests a bit  https://review.openstack.org/58925912:14
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_octavia master: Add retries to all os_* module tasks  https://review.openstack.org/58944512:14
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-tests master: Change bionic to be voting  https://review.openstack.org/58747812:16
openstackgerritMarkos Chandras (hwoarang) proposed openstack/openstack-ansible master: [docs] Add documentation for 'install_method' variable  https://review.openstack.org/58939812:17
savvasGM everyone12:23
*** Bhujay has joined #openstack-ansible12:24
savvasI can't seem to get Octavia to run properly, probably missing something: http://paste.openstack.org/show/727497/12:26
jamesdentonsavvas what version are you working with?12:29
savvas17.0.7 I think12:29
savvaslatest Queens stable12:29
savvasyes 17.0.712:29
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_swift master: Revert "Unblock gate: Provide default rabbitmq password"  https://review.openstack.org/58945012:31
jamesdentonsavvas In the neutron server container... can you check the bridge mappings in ml2_conf.ini?12:35
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_designate master: Allow tags to be used for MQ tasks  https://review.openstack.org/58945212:36
savvasjamesdenton: http://paste.openstack.org/show/727501/12:37
jamesdentonthx12:38
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_glance master: Allow tags to be used for MQ tasks  https://review.openstack.org/58945412:39
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_heat master: Allow tags to be used for MQ tasks  https://review.openstack.org/58945512:42
savvasAny thoughts jamesdenton? The playbook will also fail on a syntax error if I uncomment the octavia-housekeeping under group binding12:45
*** diliprenkila has joined #openstack-ansible12:46
jamesdentonsavvas was just looking at the playbooks to see if something unexpected was being drawn in. not seeing it. we should have some octavia folks around in a little bit, if not already12:47
openstackgerritMerged openstack/openstack-ansible-os_congress master: Add CI jobs for bionic  https://review.openstack.org/58942312:47
jamesdentonbut it may be worthwhile to file a bug12:47
jamesdentonon both issues12:47
savvasright, so we are in agreement that this should not be happening and that it is not a configuration issue12:50
*** mmalchuk has quit IRC12:50
*** phasespace has quit IRC12:50
*** DimGR has quit IRC12:50
savvasI wasn't sure if I was reading the docs right, but I can't make sense of it.12:50
*** mmalchuk_ has joined #openstack-ansible12:50
*** xteknoid has quit IRC12:53
jamesdentonDo you happen to have the full log you can post? I can't really tell if the physical_network name was obscured by Ansible or if ******** was the actual name it was attempting to use12:53
*** shyambiradar has quit IRC12:54
noonedeadpunkodyssey4me: for what reason masakari_venv_download was changed here https://review.openstack.org/#/c/584629/10/defaults/main.yml ? As with this change test fails on venv download attempt12:55
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_aodh master: Allow tags to be used for MQ tasks  https://review.openstack.org/58943712:56
odyssey4menoonedeadpunk: when the role joins the integrated build, the venv must not be downloaded for the monitors12:57
odyssey4menoonedeadpunk: otherwise it will have the wrong venv12:58
*** shyambiradar has joined #openstack-ansible13:01
*** ansmith has joined #openstack-ansible13:03
savvasjamesdenton: obscured by Ansible, not sure why it does that13:08
*** jamesdenton has quit IRC13:10
*** shyambiradar has quit IRC13:10
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_ironic master: Allow tags to be used for MQ tasks  https://review.openstack.org/58946113:10
*** lbragstad has joined #openstack-ansible13:11
*** jamesdenton has joined #openstack-ansible13:11
*** ArchiFleKs has quit IRC13:16
openstackgerritJean-Philippe Evrard proposed openstack/openstack-ansible master: Reduce log level on CentOS by default  https://review.openstack.org/58946313:20
openstackgerritJean-Philippe Evrard proposed openstack/openstack-ansible-tests master: Reduce log level on CentOS by default  https://review.openstack.org/58946413:22
openstackgerritJean-Philippe Evrard proposed openstack/openstack-ansible master: Set CentOS kernel logging to quiet in gates  https://review.openstack.org/58946313:23
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_keystone master: Allow tags to be used for MQ tasks  https://review.openstack.org/58946613:24
openstackgerritJean-Philippe Evrard proposed openstack/openstack-ansible master: Set CentOS kernel logging to 'quiet' in CI  https://review.openstack.org/58946313:24
openstackgerritJean-Philippe Evrard proposed openstack/openstack-ansible master: Set CentOS kernel logging to 'quiet' in AIO  https://review.openstack.org/58946313:25
openstackgerritMaksim Malchuk proposed openstack/openstack-ansible stable/queens: Add an ability to use custom PyPi index for deploy  https://review.openstack.org/57765313:29
openstackgerritJean-Philippe Evrard proposed openstack/openstack-ansible master: Set kernel logging to 'quiet' in AIOs  https://review.openstack.org/58946313:32
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_magnum master: Allow tags to be used for MQ tasks  https://review.openstack.org/58946813:33
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_masakari master: Allow tags to be used for MQ tasks  https://review.openstack.org/58947013:36
*** ArchiFleKs has joined #openstack-ansible13:41
*** galaxyblr has quit IRC13:44
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_neutron master: Allow tags to be used for MQ tasks  https://review.openstack.org/58947313:44
errris it possible to use openstack-ansible-plugins is a project that isnt part of OSA? I was looking and dont see it in pip so I wondered if it had deps that would keep it from working if I wasnt using OSA13:44
odyssey4meerrr: it's usable outside of OSA as it has no real deps - what specifically are you keen on re-using?13:45
errrodyssey4me: well you know how I love to hate federation.. Im currently doing an rpc-r gig where I have to get okta working on it13:46
*** ArchiFleKs has quit IRC13:46
errrso I want to use the keystone module13:46
odyssey4meerrr: ah ok, specifically for the federation mapping imports - or something else?13:47
odyssey4memost of what the keystone module does is now available in ansible upstream modules13:47
odyssey4mewe're actively working on phasing that module out13:47
*** ArchiFleKs has joined #openstack-ansible13:47
errryeah, Ive already robbed from the playbooks to get the shibboleth stuff all moved over for this one off thing, but now I need to do t the keystone config with setting up mapping and all that13:48
errrwell this is literally a 1 off and I just have to get it working by thursday13:48
odyssey4meerrr: ah ok - at some point I'll move onto the federation bits and remove the use of the module, as those things were needed because the federation implementation was done prior to there being any openstack/keystone client capability to do what it does13:49
odyssey4menow there is, so I'll likely switch it up to simply write out a json file from the config, then import it using the openstack client13:49
errrhey, so okta now provides free developer accounts with saml2 support so maybe a test could be made using it so things dont break when changed?13:50
savvasjamesdenton: https://bugs.launchpad.net/openstack-ansible/+bug/178582413:50
openstackLaunchpad bug 1785824 in openstack-ansible "Octavia fails on octavia-housekeeping group binding" [Undecided,New]13:50
*** jamesdenton has quit IRC13:52
errror maybe some way to automate out some testshib tests or something..13:52
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_nova master: Allow tags to be used for MQ tasks  https://review.openstack.org/58948013:52
*** jamesdenton has joined #openstack-ansible13:53
odyssey4meerrr: yeah, would be nice - one day I'll do it, but if anyone else is keen and has the know-how we'd welcome some sort of CI test13:53
*** mmalchuk_ has quit IRC13:53
errrsadly I just get dumped on to short term contracts and have just enough time to get a bug fixed when I am working on this stuff13:54
odyssey4meI've been thinking I might just do it in external CI some day, that way it could validate against MS Federation, OKTA and others.13:54
errrwe really need to get sales to make this a priority to product so one of us will get time assigned to make this happen.13:55
*** ArchiFleKs has quit IRC13:56
*** ArchiFleKs has joined #openstack-ansible13:56
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_octavia master: Allow tags to be used for MQ tasks  https://review.openstack.org/58948513:57
cloudnullmornings13:57
errrhi cloudnull13:57
*** vnogin has joined #openstack-ansible13:57
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_sahara master: Allow tags to be used for MQ tasks  https://review.openstack.org/58948914:01
*** vnogin has quit IRC14:02
*** ArchiFleKs has quit IRC14:02
*** ArchiFleKs has joined #openstack-ansible14:03
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_tacker master: Allow tags to be used for MQ tasks  https://review.openstack.org/58949014:04
openstackgerritKevin Carter (cloudnull) proposed openstack/openstack-ansible-os_neutron master: Cleanup files and templates using smart sources  https://review.openstack.org/58864714:04
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_trove master: Allow tags to be used for MQ tasks  https://review.openstack.org/58949114:06
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_watcher master: Allow tags to be used for MQ tasks  https://review.openstack.org/58949214:08
*** eglute has joined #openstack-ansible14:10
odyssey4mecloudnull: really happy to see you pick up on that work - that's been on my TODO for ages, and just haven't managed to get there14:11
cloudnull++14:11
odyssey4meit'd be super to get that into rocky if we can14:11
cloudnulltotally agree. saves a lot of operations and work for future14:12
*** vnogin has joined #openstack-ansible14:12
odyssey4meI was also thinking that we just symlink the config folder to the venv - that automatically makes rolling back an option to develop in the future for minor upgrades.14:12
cloudnull++ that was my thoughts as well14:13
cloudnulladditionally if /openstack/venvs is a bind mount the config is totally outside the container.14:13
jrosserbionic -> voting if folks agree it is time https://review.openstack.org/#/c/587478/14:14
odyssey4mecloudnull: FYI, not sure if this will give you ideas you love or hate, but I did keystone a while back: https://github.com/openstack/openstack-ansible-os_keystone/commit/8367e442304a16687be602d4a23762d3408b901014:14
cloudnull++ I took that and what mnaser did for cinder and made a pattern that we can apply to most, if not all, roles. https://review.openstack.org/#/q/topic:smart-sources+(status:open+OR+status:merged)14:15
cloudnullI did those few just to let folks see the pattern14:15
cloudnullits a fairly easy change to apply across the board14:16
odyssey4memakes sense to me, although I'm tied up right now - so others will need to review and see if it makes sense for them14:17
*** Bhujay has quit IRC14:18
odyssey4memy only concern after skimming it is that it feels complex14:18
*** nicolasbock has quit IRC14:18
odyssey4meI know it cuts down a lot of what we carry, and potentially makes the roles more multi-series friendly, but there's a cost there in the complexity of it. Unfortunately I can't afford the time to really think through whether I have any ideas to simplify it.14:19
odyssey4meIf anything, it's better than that awful sources-branch-updater.14:19
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_zun master: Allow tags to be used for MQ tasks  https://review.openstack.org/58949414:22
errrodyssey4me: I was just looking at https://docs.ansible.com/ansible/2.3/list_of_cloud_modules.html#openstack and I think mapping and federation are missing from the modules they have14:26
*** gkadam_ has joined #openstack-ansible14:27
odyssey4meerrr: yes, they are, but it's not necessary to have them14:27
errrodyssey4me: how so? The commands in https://docs.openstack.org/keystone/latest/advanced-topics/federation/federated_identity.html#mapping-combinations have you run those commands14:28
errrdo you just fall to `command:` and use the cli client?14:29
odyssey4meerrr: yep - spit out the json from our source config, then import with the openstack cli14:29
errrah14:29
*** gkadam has quit IRC14:29
*** yolanda has quit IRC14:31
cloudnullsorry in meetings :)14:31
* cloudnull the meetings will continue until moral improves 14:32
errrno they wont. they will never end no matter what14:32
errrbetter you than me though :P14:32
*** hwoarang has quit IRC14:37
*** yolanda has joined #openstack-ansible14:37
openstackgerritAmy Marrich (spotz) proposed openstack/openstack-ansible master: [docs] Add documentation for 'install_method' variable  https://review.openstack.org/58939814:41
*** priteau has quit IRC14:50
*** FrankZhang has joined #openstack-ansible14:51
*** kaiokmo has joined #openstack-ansible14:57
*** Emine has quit IRC14:58
evrardjpfolks I won't be able to stay long during the meeting of today15:06
*** nicolasbock has joined #openstack-ansible15:07
openstackgerritDmitriy Rabotjagov (noonedeadpunk) proposed openstack/openstack-ansible-os_masakari master: Basic implementation of masakari-monitors  https://review.openstack.org/58462915:07
*** pcaruana has quit IRC15:11
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_octavia master: Clean up tests a bit  https://review.openstack.org/58925915:12
*** cshen has quit IRC15:14
noonedeadpunkodyssey4me: I don't have any idea, why tests are failing, when they are run with non-root user... Moreover, it's strange, that they don't fail for root, as containers doesn't have network interfaces at all on this step. I may suggest, that ansible reaches container from root by  nsenter/lxc-attach, but I don't fully understand, how it works...15:16
*** spatel has joined #openstack-ansible15:17
evrardjpnoonedeadpunk: we are using nsenter now for our connection plugin15:17
evrardjpodyssey4me: ping?15:18
noonedeadpunkevrardjp: and how is it launched/triggered? Could you point me to this plugin (if it's public one)?15:19
evrardjphttps://github.com/openstack/openstack-ansible-plugins/blob/master/connection/ssh.py15:19
*** KeithMnemonic has joined #openstack-ansible15:20
evrardjpit's a hijack of ssh plugin, so we don't have to explicitly say it's 'our' connection plugin15:20
odyssey4meevrardjp: in meetings right now15:20
evrardjpok15:21
evrardjpprometheanfire: are you in meetings too?15:21
evrardjpI'd need to know something about requirements15:21
prometheanfiresure15:23
*** Bhujay has joined #openstack-ansible15:23
prometheanfirein meetings, but multitasking :D15:23
*** priteau has joined #openstack-ansible15:26
*** DanyC has quit IRC15:27
*** DanyC has joined #openstack-ansible15:27
*** DanyC has quit IRC15:31
*** gyee has joined #openstack-ansible15:32
*** diliprenkila has quit IRC15:35
*** dxiri has joined #openstack-ansible15:38
*** v1k0d3n has joined #openstack-ansible15:44
openstackgerritMerged openstack/openstack-ansible master: Fix log compression  https://review.openstack.org/58943015:45
*** olivierbourdon38 has quit IRC15:46
*** mmalchuk has joined #openstack-ansible15:50
evrardjpok15:53
evrardjpwho is chairing today?15:53
openstackgerritJonathan Rosser proposed openstack/openstack-ansible master: [TEST] bionic  https://review.openstack.org/58660715:59
openstackgerritJonathan Rosser proposed openstack/openstack-ansible master: [TEST] bionic  https://review.openstack.org/58660716:00
*** Bhujay has quit IRC16:00
*** luksky has quit IRC16:02
evrardjpok I guess I am chairing for 30 minutes then16:02
evrardjpcloudnull, DimGR, andymccr, d34dh0r53, hughsaunders, b3rnard0, palendae, odyssey4me, serverascode, rromans, erikmwilson, mancdaz, _shaps_, BjoernT, claco, echiu, dstanek, jwagner, ayoung, prometheanfire, evrardjp, arbrandes, scarlisle, luckyinva, ntt, javeriak, spotz, vdo, jmccrory, alextricity25, jasondotstar, admin0, michaelgugino, ametts, bgmccollum, darrenc, JRobinson__, colinmcnamara, thorst,16:02
evrardjpadreznec, eil397, qwang,nishpatwa_, cathrichardson, drifterza, hwoarang, cshen, ullbeking, Tahvok, mnaser, nicolasbock, jrosser16:02
evrardjpMeeting time :)16:03
spotzOk you didn't get booted evrardjp.... jungleboyj swears pings get you booted16:03
prometheanfireneat16:03
evrardjp#startmeeting openstack_ansible_meeting16:04
openstackMeeting started Tue Aug  7 16:04:02 2018 UTC and is due to finish in 60 minutes.  The chair is evrardjp. Information about MeetBot at http://wiki.debian.org/MeetBot.16:04
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.16:04
openstackThe meeting name has been set to 'openstack_ansible_meeting'16:04
evrardjpspotz: seems I am still here.16:04
spotzevrardjp: yep:)16:04
evrardjp#topic rollcall16:04
evrardjpo/16:05
evrardjp#topic Last week highlights16:06
evrardjpspotz: there seem to be something weird with the bot though.16:06
spotzo/16:06
evrardjpit doesn't seem to ack my commands16:06
evrardjpweirdly16:06
evrardjplet's continue and hope things are logged16:07
spotzYeah noticed that in the woo meeting16:07
evrardjpevrardjp and odyssey4me thanked jrosser for the work done in Bionic16:07
*** spatel has quit IRC16:07
evrardjpodyssey4me: reminded that there is more work to be done16:08
evrardjphwoarang reminded that leap 15 work is blocked due to mariadb, lxc, and sudo16:08
evrardjpevrardjp reminded that RC1 is this week16:08
evrardjpand evrardjp asked to fill the etherpad https://etherpad.openstack.org/p/osa-stein-ptg16:09
bgmccollumo/16:09
evrardjpis there any other last week highlights?16:09
evrardjpif not let's go to bugs16:09
evrardjp#topic bugtriage16:09
evrardjpplease see our usual etherpad: https://etherpad.openstack.org/p/osa-bugtriage16:09
evrardjp#link https://bugs.launchpad.net/openstack-ansible/+bug/178582416:09
openstackLaunchpad bug 1785824 in openstack-ansible "Octavia fails on octavia-housekeeping group binding" [Undecided,New]16:09
evrardjpI am asking more details about the openstack_user_config, as this seems an inventory issue16:10
johnsomThis was just a mis-configured host networking I think...  Ping the user, I'm pretty sure they resolved that issue on their own16:11
evrardjpjohnsom: I wouldn't be surprised about a user configuration issue.16:12
evrardjplet's move on16:12
evrardjp#link https://bugs.launchpad.net/openstack-ansible/+bug/178565116:12
openstackLaunchpad bug 1785651 in openstack-ansible "P->Q Upgrade - Nova cert shows as down" [Undecided,New]16:12
evrardjpI guess we need some cleanup of haproxy from P to Q as a service looks removed?16:13
evrardjpI am not aware of what nova-cert does, and if it really was included in nova-api . Proves my internal knowledge of nova.16:13
evrardjp:p16:13
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible stable/queens: Fix log compression  https://review.openstack.org/58953616:14
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible stable/pike: Fix log compression  https://review.openstack.org/58953716:14
evrardjphttps://github.com/openstack/nova/commit/d6d5c6be0cc0738bb7d67ca8391e5cc4f0a419f216:14
evrardjpI'd say confirmed and medium, as we need to remove things in haproxy, which we haven't done.16:15
bgmccollumAnd the containers16:15
bgmccollumAnd the service entry in Nova?16:16
evrardjpwell the thing is... I don't see anything in haproxy configuration16:16
evrardjpso I guess it's probably only the containers and service entries16:17
bgmccollumMaybe the inventory too? I went thought this myself, and forget all the places I had to clean up...16:18
evrardjpwe can mark it as confirmed then16:18
evrardjpfirst hand experience there :p16:18
cloudnullo/16:19
evrardjpnova_cert is still in the inventory16:19
evrardjpomg16:19
evrardjpin rocky16:19
evrardjpthat's fun.16:19
evrardjpconfirmed medium?16:20
evrardjpit doesn't hurt it's just ugly so I'd say "fix when convenient" is appropriate16:21
bgmccollumits not hurting anything, its just the ocata-era nova-cert starts up, then fails because of object version mismatch or somethings...and quietly goes away16:21
evrardjpok16:21
evrardjplet's move on16:21
evrardjphttps://bugs.launchpad.net/openstack-ansible/+bug/178564616:21
openstackLaunchpad bug 1785646 in openstack-ansible "Cleanup of haproxy files could be more obvious" [Undecided,New]16:21
evrardjp#link https://bugs.launchpad.net/openstack-ansible/+bug/178564616:21
evrardjpconfirmed wishlist?16:22
evrardjpit's a bug pending to happen basically16:22
evrardjpanyone agrees?16:23
bgmccollumanother doesn't hurt?16:23
evrardjpbgmccollum: not sure what you mean16:23
bgmccollumit doesn't break anything if the service is gone from the openstack side, but the haproxy config still exists...16:24
evrardjpthat's true16:24
evrardjpso confirmed wishlist?16:24
bgmccollumsure16:24
evrardjp#link https://bugs.launchpad.net/openstack-ansible/+bug/178559216:25
openstackLaunchpad bug 1785592 in openstack-ansible "dynamic inventory doesn't handle is_metal: false->true change" [Undecided,New]16:25
evrardjp(I have to go soon though)16:25
evrardjpcloudnull: could you have a look at this ^ ?16:25
cloudnullsure16:25
cloudnullassign that one to me16:25
jrosserthat felt very very similar to one we patched previously16:26
evrardjp#link https://bugs.launchpad.net/openstack-ansible/+bug/178551716:26
openstackLaunchpad bug 1785517 in openstack-ansible "iptables checksum-fill causing kernel warning error stack" [Undecided,New]16:26
*** vnogin has quit IRC16:27
evrardjpthat sounds like invalid16:27
evrardjpcloudnull: can you confirm?16:27
bgmccollumThis is the kernel-deubg slowdown issue, right?16:27
cloudnullevrardjp that kernel one seems valid16:28
*** DanyC has joined #openstack-ansible16:28
evrardjpyes but not in that sense I'd say16:28
cloudnullcent7 seems to have debug logging enabled by default16:28
*** spatel has joined #openstack-ansible16:28
bgmccollumExacerbated by the checksumming?16:28
evrardjpcloudnull: I have submitted a patch for that16:28
cloudnullthe checksumming is something that we should probably have a look at.16:28
cloudnullwe may not even need that anymore16:28
*** electrofelix has quit IRC16:29
evrardjpthat's probably true16:29
*** devx has quit IRC16:29
evrardjpbut that can be an user configuration, and with my patch explaining the change for centos, we should be good, right?16:29
evrardjpso I should mark this as invalid?16:30
evrardjpor I keep this for the checksum investigation?16:30
* cloudnull hasn't seen your patch16:30
cloudnullhowever it sounds like it covers this16:30
cloudnullor if i have seen it I dont remember16:30
bgmccollumother distros with kernel-debug not enabled do not exhibit this problem16:31
cloudnullor maybe they are and we're just not seeing it ?16:31
evrardjpcloudnull: I think that's the case16:31
cloudnullas in we're not seeing the logs16:31
evrardjpwe are not seeing it16:31
*** devx has joined #openstack-ansible16:31
cloudnull++16:31
cloudnullI would believe that16:31
evrardjpand it doesn't hurt that much on iops16:31
bgmccollumi though the slowdown was the massive amount of logging happening16:32
*** DanyC has quit IRC16:32
evrardjpyes16:32
* bgmccollum shrugs16:32
evrardjpthat's waht I'd say16:32
cloudnullhowever I'd also believe its just a crusty old cent7 issue :)16:32
evrardjpso if no logging, better behavior16:32
evrardjphaha16:32
*** lbragstad[m] has quit IRC16:32
evrardjpcloudnull: for fun we should set it to 7 1 4 7 everywhere16:32
evrardjpin the meantime I mark this as fix released16:32
bgmccollumfor science16:32
cloudnull++16:33
evrardjpbgmccollum: for science!16:33
*** DanyC has joined #openstack-ansible16:33
evrardjpfix comitted*16:33
evrardjpI have to go16:33
evrardjpcan someone take over from here?16:33
spotzLet me find the link evrardjp16:34
evrardjpspotz: https://etherpad.openstack.org/p/osa-bugtriage16:34
evrardjpwe are there:16:34
evrardjp#link https://bugs.launchpad.net/openstack-ansible/+bug/178538616:34
openstackLaunchpad bug 1785386 in openstack-ansible "Integration of Swift and Manila with Openstack-ansible having Ceph backend?" [High,New]16:34
evrardjpI have to run16:34
evrardjpttyl everyone16:35
evrardjp#chair spotz16:35
spotzgo I got it16:35
openstackCurrent chairs: evrardjp spotz16:35
evrardjpthanks spotz :)16:35
cloudnulllater evrardjp16:35
spotzOk so looking at it what are the opinions?16:35
spotzSounds kinda wishlist to me16:36
bgmccollumNot too familiar with Ceph, but shouldn't it "just work" if the rados gateway is deployed?16:36
*** DanyC has quit IRC16:36
spotzI'm taking it as they'd like a config option?16:37
*** DanyC has joined #openstack-ansible16:37
spotzNow they shouldn't have changed it to high importance but another issue:)16:37
spotzodyssey4me cloudnull jrosser?16:37
jrosserwell quite a few of these things seem to fall into the "user story" category maybe16:38
jrosseras in its probably all possible given the right config16:38
bgmccollumis manilla even an OSA deployable project?16:38
spotzIf it can be done already it's a docs issue. But not sure16:38
*** gkadam_ has quit IRC16:38
odyssey4meit makes absolutely no sense to 'integrate swift' with ceph16:38
cloudnullI think andymccr was working on that role?16:39
odyssey4meand manila is not something we have a role for16:39
bgmccollumwell there you go...wish list?16:39
spotzOr won't fix16:39
odyssey4mewell, as the question is posed there - it can be answered16:39
odyssey4meThere is no manila role in OSA. We would welcome a contribution.16:39
spotzOk changing to wishlist, and I'll mention the no manila role. Even if it doesn't make sense could they config ceph and swift?16:40
odyssey4meAnd Swift is not something that you integrate on top of Ceph. Please look at Ceph RGW if you want a swift API with a ceph back-end.16:40
bgmccollumI think that is the spirit of what he is asking...16:41
bgmccollumBut without a Manila role, there is nothing to more to provide.16:41
jrosserbitrotted stuff here https://github.com/bkreitch/openstack-ansible-os_manila16:42
spotzreply - There is not currently a Manila role within OpenStack-Ansible. We would readily accept a contribution of this role. Swift is not something that is integrated on top of Ceph. Please look at Ceph RGW if you want a swift API with a ceph back-end.16:42
bgmccollumsounds good to me16:43
spotzjrosser: But we can't support someone else's and I don't think we want to point folks to non-supported repos?16:43
odyssey4meinteresting, that role might be nice to import and iterate16:43
cloudnullhttps://github.com/andymcc/openstack-ansible-os_manila16:43
jrosserspotz: of course, looks like someones had a bash though16:43
cloudnull oh sorry.16:43
spotzNow Andy's is another story:), but anyone know the status?16:44
odyssey4meandymccr's is clearly nothing more than scaffolding16:44
odyssey4meand andymccr is unlikely to continue work on it given his work changes16:44
spotzOk I went with wish-list and what I posted in channel let me get the next one16:44
spotz#link https://bugs.launchpad.net/openstack-ansible/+bug/178536516:45
openstackLaunchpad bug 1785365 in openstack-ansible "LXC container network slow because of kernel debug mesg" [Undecided,New]16:45
spotz15 minute warning FYI16:45
spotzcloudnull: you have any thoughts?16:46
cloudnullthat was the same issue evrardjp touched on a bit ago16:46
cloudnullmaybe duplicate16:46
spotzI'll make a note in the etherpad for JP to take a look. Let me get the next one16:47
*** spatel has quit IRC16:47
spotz#link https://bugs.launchpad.net/openstack-ansible/+bug/178506816:47
openstackLaunchpad bug 1785068 in openstack-ansible "os_glance : Ensure glance service" [Undecided,New]16:47
spotzLooks like he's included info anout his setup and config settings16:48
jrosserexternal and internal vip same -> bzzzzt16:48
jrosserthat never ends well16:49
*** DanyC has quit IRC16:49
*** DanyC_ has joined #openstack-ansible16:49
spotzok told him that marked invalid and asked him to retry16:50
spotz#link https://bugs.launchpad.net/openstack-ansible/+bug/178494916:50
openstackLaunchpad bug 1784949 in openstack-ansible "Queen issue adding compute nodes " [Undecided,New]16:50
spotzNext:)16:50
spotzHas anyone done an upgrade to queens and run into this?16:55
*** Emine has joined #openstack-ansible16:55
odyssey4menot me, but this issue is not unique to queens or an upgrade16:55
spotzTrue bug or something we should document?16:56
bgmccollumSounds like he didn't run the setup-hosts.yml playbook?16:56
bgmccollumOr not...16:56
bgmccollum:.16:56
*** DanyC_ has quit IRC16:56
odyssey4methe issue is that the tasks which do the linking seem to have not ran - there's not enough here to diagnose anything16:56
*** DanyC has joined #openstack-ansible16:56
*** cshen has joined #openstack-ansible16:56
*** tosky has quit IRC16:57
jrosserif we can point to where the venv should get pulled in and ask for confirmation thats been run, and debug output if it doesnt do anything?16:57
odyssey4mewe pushed this up recently: https://review.openstack.org/#/q/Id962efe16c425424715409f071c4a304f841600116:57
jrosser"I found i don't have /openstack/venvs/nova-17.0.8 directory" <- the whole thing is missing though?16:58
odyssey4meso all that stuff is done after the venv is downloaed and extracted16:58
spotzWell May 31st, I would think he had that patch?16:58
odyssey4mewhich means that somehow this person got to that task without the venv being there16:58
spotzOk let me ask him for his steps and maybe he missed a step?16:59
spotzThis is going to be the last one we get to today16:59
odyssey4meso basically there should be a failed task somewhere around the venv deployment, or the task was skipped for whatever reason - either way, the reporter need to provide more info16:59
spotzok asked for steps and for them to look for failed tasks. And we're out of time17:00
spotz#endmeeting17:00
odyssey4methese tasks have to run, and if they didn't then the person will need to see why: https://github.com/openstack/openstack-ansible-os_nova/blob/stable/queens/tasks/nova_install.yml#L74-L12717:01
openstackMeeting ended Tue Aug  7 17:00:59 2018 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)17:01
openstackMinutes:        http://eavesdrop.openstack.org/meetings/openstack_ansible_meeting/2018/openstack_ansible_meeting.2018-08-07-16.04.html17:01
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/openstack_ansible_meeting/2018/openstack_ansible_meeting.2018-08-07-16.04.txt17:01
openstackLog:            http://eavesdrop.openstack.org/meetings/openstack_ansible_meeting/2018/openstack_ansible_meeting.2018-08-07-16.04.log.html17:01
jrosseri've been having a look at why some of the jobs are running slow - this appears to be swapping http://logs.openstack.org/30/589430/2/check/openstack-ansible-deploy-aio_lxc-ubuntu-xenial/bea65a6/logs/openstack/instance-info/dstat.html17:01
spotzthanks all17:02
odyssey4mejrosser: yeah, swapping's to be expected given how much we're putting into such a small host17:02
jrosseri see in my aio that lots of the services have their process/threds reduced17:03
jrosserbut thats not consistent17:03
odyssey4mejrosser: ah, then we should fix that17:04
jrossertons and tons of msqld, and more nova-scheduler, glance-api, nova-compute than you might expect and zillions of pypiserver17:04
jrosserthough pypiserver is small so probly doesnt matter17:05
jrosserhtop sorted by memory usage is useful17:05
odyssey4mehmm, I don't think we have a tunable for the number of pypiserver threads17:06
odyssey4methe others there should be tunables already, we're obviously just missing applying those right17:06
jrosserpypi server is only small17:06
odyssey4meI've actually been wanting to look into using uwsgi's cheaper algo for auto-scaling rather than using a fixed set of threads.17:07
*** DanyC_ has joined #openstack-ansible17:07
jrossermysql, nova-api-os-compute, nova-compute, glance-api would be a good place to start17:07
odyssey4meI did a short test and it seemed to work, but I figured that's something we should rather implement very early in a cycle so that we get maximum test time.17:07
jrosseron other news my bionic aio is getting to tempest and failing 1 out of 114, so close17:08
* jrosser travels17:08
*** DanyC_ has quit IRC17:09
*** DanyC_ has joined #openstack-ansible17:10
*** DanyC has quit IRC17:11
odyssey4meok, finally done with meetings17:13
odyssey4mexgerman_: could you revisit your review on https://review.openstack.org/#/c/589246/ and that patch chain - I think all the bases are covered for most of the patches now17:14
*** ThomasWhite has quit IRC17:15
odyssey4meyep, the last patch's tests just passed - just waiting for the results17:15
*** fghaas has quit IRC17:16
odyssey4meevrardjp: it looks like gnocchi now has a minimum requirement for setuptools>=30.3.017:20
openstackgerritAndy Smith proposed openstack/openstack-ansible-os_cinder master: Setup oslo.messaging extra packages for optional drivers  https://review.openstack.org/58263217:21
*** DanyC has joined #openstack-ansible17:22
*** DanyC_ has quit IRC17:24
openstackgerritAndy Smith proposed openstack/openstack-ansible-os_neutron master: Setup oslo.messaging extra packages for optional drivers  https://review.openstack.org/58302117:25
openstackgerritAndy Smith proposed openstack/openstack-ansible-os_nova master: Setup oslo.messaging extra packages for optional drivers  https://review.openstack.org/58445017:26
openstackgerritAndy Smith proposed openstack/openstack-ansible-os_keystone master: Setup oslo.messaging extra packages for optional drivers  https://review.openstack.org/58447917:32
*** spatel has joined #openstack-ansible17:32
spatelI am trying to add "mon_allow_pool_delete: True" in user_variable but it seems not working17:32
spateli finish my playbook but it didn't appended that option in mon config17:33
*** fghaas has joined #openstack-ansible17:39
*** markvoelker_ has quit IRC17:45
openstackgerritAndy Smith proposed openstack/openstack-ansible-os_heat master: Setup oslo.messaging extra packages for optional drivers  https://review.openstack.org/58451317:47
openstackgerritAndy Smith proposed openstack/openstack-ansible master: [WIP] Add qdrouterd role for messaging backend deployment  https://review.openstack.org/57026117:52
*** hamzaachi has quit IRC18:00
*** Leo_m has joined #openstack-ansible18:01
Leo_mhi, I destroyed the octavia container but when I try to rebuild calling "# openstack-ansible setup-hosts.yml --limit octavia_all" I'm getting a failure fatal: [aio1_octavia_server_container-4c328689]: FAILED! => {"failed": true, "msg": "No file was found when using with_first_found. Use the 'skip: true' option to allow this task to be skipped if no files are found"} with the deleted container18:03
Leo_many ideas?18:03
prometheanfirecan someone take a look at this jinja?18:07
prometheanfireenabled_hardware_types = {{ filtered_ironic_drivers.hardware_type | join(',') }}                   is returning      enabled_hardware_types = i,p,m,i18:07
prometheanfireinstead of ipmi18:07
*** poopcat has joined #openstack-ansible18:07
prometheanfireI think perhaps I need to insert a list between filtered_ironic_drivers.hardware_type and the value in the datastucture18:08
prometheanfirehttps://github.com/openstack/openstack-ansible-os_ironic/commit/fc1e4f78dc9b5e415ad71a4664638bddfa190fd2#diff-04f1b8cc0a91d47ed7905aad52cc46deR19 seems to be causing it, perhaps the filter needs updating18:10
spatelLeo_m: try to delete aio1_octavia_server_container-4c328689 in inventory and see18:10
odyssey4meLeo_m: include lxc_hosts in that limit18:10
odyssey4meLeo_m: do not do what spatel just suggested18:10
mnaserguilhermesp: so pretty much i'mm at https://review.openstack.org/#/c/584604/18:10
guilhermesplooking18:10
prometheanfiredoesn't this create a list? https://github.com/openstack/openstack-ansible-os_ironic/commit/fc1e4f78dc9b5e415ad71a4664638bddfa190fd2#diff-6d62b73b3f46cfa11f6466d08cf45303R18618:10
*** cshen has quit IRC18:11
mnaserthat's where i'm stuck so far, if you look at the logs, you'll notice that it's failing to find cinder-rtstool, that's because cinder-rtstool is located in /openstack/venv/cinder-testing/bin/cinder-rtstool, but $PATH does not include that inside the service18:11
odyssey4meprometheanfire: ah, that's because filtered_ironic_drivers.hardware_type is a string, not a list18:11
guilhermespthat's a good start to me18:11
mnaserguilhermesp: so whats needed right now is to make sure that /openstack/venv/cinder-testing/bin is inside $PATH18:12
prometheanfireodyssey4me: ya, I get that part, but the link I just did should have it as a list from my shitty jinja reading/writing18:12
mnaserand that's where i didn't have time to follow up on :X18:12
guilhermespk mnaser I'm going to focus on it asap18:12
guilhermespdon't worry :P18:12
mnasercool, lemme know if you need a hand, i know cloudnull tried his hand at adding it to $PATH but it didnt go through18:13
*** fghaas has quit IRC18:13
guilhermespgreat, I give you guys updates when I put my hands on it18:13
*** luksky has joined #openstack-ansible18:14
odyssey4meprometheanfire: I have a suspicion - just verifying18:14
Leo_modyssey4me: openstack-ansible setup-hosts.yml --limit lxc_hosts ?18:15
Leo_mor what do you mean18:15
prometheanfireodyssey4me: thanks18:15
spatelLeo_m: i think he is saying add both in limit18:15
Leo_mhow would it be?18:16
odyssey4meprometheanfire: do you have a sample output of what filtered_ironic_drivers contains?18:16
spatelLeo_m:  --limit octavia_all  --limit lxc_hosts something like that18:16
spatelor may be  --limit octavia_all, lxc_hosts18:16
odyssey4meLeo_m: --limit octavia_all,lxc_hosts18:17
odyssey4mewhenever you limit to a container set, also always include the hosts because some tasks are delegated to the hosts18:17
Leo_m--limit octavia_all  --limit lxc_hosts is going, thx!!18:17
*** cshen has joined #openstack-ansible18:18
prometheanfireodyssey4me: no :|18:18
prometheanfireodyssey4me: would that be in a gate job?18:19
odyssey4meprometheanfire: I think there's a bug in https://github.com/openstack/openstack-ansible-os_ironic/blob/master/vars/main.yml#L182-L189 because it looks to me like it will always update the same dict, so there'll only ever be one result18:19
*** diliprenkila has joined #openstack-ansible18:19
odyssey4meprometheanfire: nope, but if you had a test system running you'd be able to get it18:19
prometheanfireodyssey4me: we do, would it be in cached facts?18:19
diliprenkilaHi all, I am hitting this error while checking namespaces on network nodes18:19
diliprenkilaroot@ctrl1:~# ip netns RTNETLINK answers: Invalid argument RTNETLINK answers: Invalid argument RTNETLINK answers: Invalid argument RTNETLINK answers: Invalid argument qdhcp-60ea65d9-bf4c-4fd1-a8a9-8cae22e28662 RTNETLINK answers: Invalid argument qrouter-3755419c-2a20-4ed0-9012-af24863f2447 RTNETLINK answers: Invalid argument snat-3755419c-2a20-4ed0-9012-af24863f244718:20
odyssey4meprometheanfire: ansible -m debug -e var=filtered_ironic_drivers ironic_all18:20
prometheanfirek18:20
prometheanfire    "msg": "Hello world!"18:21
odyssey4meprometheanfire: from within the /opt/openstack-ansible/playbooks folder, to ensure you get the inventory included18:21
prometheanfirelol18:21
prometheanfireya, did that18:21
bgmccollumisn't it -a not -e ?18:22
diliprenkilaip netns RTENETLINK invalid argument https://etherpad.openstack.org/p/Bv9e3EB4gt18:22
odyssey4meprometheanfire: yes, bgmccollum is right18:22
prometheanfire    "filtered_ironic_drivers": "VARIABLE IS NOT DEFINED!"18:22
diliprenkilaon the network nodes18:22
*** cshen has quit IRC18:23
odyssey4meprometheanfire: hmm, oh of course it's not a group var so you'd have to do it in a task in the role18:23
prometheanfireadd a debug task then?18:23
mnaserevrardjp: thanks for hosting today.18:28
mnaseri'm sorry, the past few days have been overwhelmingly busy18:28
mnaseri'm almost out of it18:28
odyssey4meprometheanfire: yep, it seems clear that the way the var is put together isn't right, so you'll have to fire up a test system and work it out18:29
prometheanfirek18:29
odyssey4meor put together a test playbook on your own machine with the same starting data and figure it out18:29
prometheanfiredid so, running the debug play now18:29
odyssey4methat'd be alot quicker I suppose18:29
guilhermespmnaser: hope to help you out in order to make thing lighter for ya :D18:29
prometheanfirehttps://gist.github.com/prometheanfire/a62b68b27341505c500ad80617060eca18:30
prometheanfireso it looks like it's not generating a list for each key18:30
prometheanfireguess I play around with that filter18:31
odyssey4meprometheanfire: yes, so this is very broken: https://github.com/openstack/openstack-ansible-os_ironic/blob/master/vars/main.yml#L182-L18918:33
bgmccollumshould "set concat_drivers = {}" be "set concat_drivers = []" if you are expecting a list?18:34
odyssey4methat is constantly updating the same dict - so the last one wins18:34
odyssey4meyes, except I don't think it's a list that's expected18:34
bgmccollumok18:34
odyssey4megiven the way it's used here: https://github.com/openstack/openstack-ansible-os_ironic/blob/fc1e4f78dc9b5e415ad71a4664638bddfa190fd2/templates/ironic.conf.j2#L19-L2418:35
odyssey4meI suspect that this would all be made far simpler by just using jmespath instead18:35
prometheanfireodyssey4me: no clue what that is18:35
odyssey4meprometheanfire: check here http://jmespath.org/ - put the data in and figure out the query to get the result, then use that with the json_query filter via ansible18:36
prometheanfirethat could be better, yes18:37
odyssey4meso if you get https://github.com/openstack/openstack-ansible-os_ironic/blob/master/vars/main.yml#L182-L189 to build your list of various settings18:38
odyssey4methen use json_query to get the values you need18:38
*** madorn has quit IRC18:41
*** madorn has joined #openstack-ansible18:43
*** diliprenkila has quit IRC18:44
odyssey4meprometheanfire: this test data works for me: https://gist.github.com/odyssey4me/f2c2065a5496b09e219d5663ab304e9818:48
odyssey4meso if you get that filtered list to actually be a list of the settings, then change the .conf entries to use json_query to grab the list of attributes, you have the right combo18:49
*** openstackgerrit has quit IRC18:49
odyssey4meof course you could probably do away with the filtered list by using jmespath from the get go - do one to filter the list, then one to get the attributes18:50
*** fghaas has joined #openstack-ansible18:56
prometheanfirethat's the plan19:00
prometheanfireright now I'm figuring out how to multiselect from ironic driver types based on the list of ironic_drivers_enabled19:00
*** fghaas has quit IRC19:04
*** cshen has joined #openstack-ansible19:08
spateljamesdenton: are you there?  I migrating haproxy to F5 and i am seeing haproxy using SSL cert so do i need to move my cert to F5 ?19:11
spatelbind 10.30.2.9:6080 ssl crt /etc/ssl/private/haproxy.pem19:11
jamesdentonspatel are you using a real cert? or unsigned private cert?19:12
spatelI have real cert yes19:12
jamesdentonyes, then you'll need to create the profile on the F5 and attach it to the virtual server19:13
spatelI have wildcard certs already installed for other services i want to use that19:13
jamesdentonthen you should be able to reuse that existing profile19:14
spatelso all i need to do attach cert to VS_SERVER19:14
jamesdentonwell, the cert/key should be defined in an SSL profile, and that SSL profile is attached to VS_SERVER19:15
spatelyup!19:16
spateljust wanted to make sure this cert is not specific to openstack which we need to bring on F519:16
spatelalso i am not using SPICE console so i shouldn't use this config in F519:17
spatelbind 10.30.2.9:6080 ssl crt /etc/ssl/private/haproxy.pem19:17
spatelignore that19:17
spatelreate ltm monitor http /OSTACK/OSTACK_MON_HTTP_NOVA_SPICE_CONSOLE { defaults-from http destination *:6082 recv "200 OK" send "HEAD /spice_auto.html HTTP/1.1\r\nHost: rpc\r\n\r\n" }19:17
spatelI am using novac19:17
spatelI need to hand edit that and change port too19:17
jamesdentonyes, you'll want to edit that19:19
Leo_mspatel: where is the container inventory? I may need to delete like you mention, my recreated octavia container is not picking up the new lbaas cidr from the updated openstack_user_config.yml19:19
kaiokmojrosser: hi, since https://review.openstack.org/#/c/589189/ is merged, can you set https://review.openstack.org/#/c/588464/ to abandoned, please?19:22
*** throwsb1 has joined #openstack-ansible19:23
*** vollman has quit IRC19:25
jrosserkaiokmo: would you prefer i did it? you should see a button on gerrit i think19:33
jrossercloudnull: this feels suspicious, restart systemd-networkd followed by unable to ssh.... what do you think? http://logs.openstack.org/07/586607/10/check/openstack-ansible-deploy-aio_lxc-ubuntu-bionic/9ddb187/job-output.txt.gz#_2018-08-07_17_27_11_98524019:35
cloudnulljrosser quite possibly19:36
cloudnullit should leave the interface alone given that its already up19:40
cloudnullbut I guess if its trying to reconfigure teh interface that could be the cause19:41
cloudnullit also could be a situation where the interface is just not up/ready19:41
kaiokmojrosser: ah, my mistake. thanks :)19:42
odyssey4mejrosser cloudnull if I could get your eyes on https://review.openstack.org/589248 It'd be grand - it's a small enough patch19:43
cloudnull++19:43
odyssey4mealso cloudnull https://review.openstack.org/58906419:43
odyssey4meand https://review.openstack.org/58850819:43
cloudnulldone.19:44
* cloudnull goes back to lurking while on conference calls 19:44
odyssey4metyvm19:44
prometheanfireodyssey4me: I'm learning to hate jmespath19:45
*** openstackgerrit has joined #openstack-ansible19:46
openstackgerritMerged openstack/openstack-ansible-os_swift master: Revert "Unblock gate: Provide default rabbitmq password"  https://review.openstack.org/58945019:46
prometheanfireit doesn't even look like you can pass two sets to it so it can do comparisons19:46
*** hamzaachi has joined #openstack-ansible19:46
prometheanfireso that logic has to remain outside of it19:46
odyssey4meprometheanfire: yeah, it works best with lists of things - the trouble you have is that it's just a dict of dicts19:47
prometheanfireya, no examples for that19:47
odyssey4meso pre-process it into a list, then use jmespath to get the things you want19:47
prometheanfirenot sure that's valid either, the datastucture as is makes sense, not sure it's able to be converted without loosing data19:48
odyssey4mewell, you already are with the filtered list - it's just no generating a list, it's overwriting itself19:49
odyssey4meso change it to actually make a list, then use jmespath to extract the attributes form the list and give you a list as a result19:49
prometheanfirewat19:49
spatelLeo_m: folks won't agreed with me but thats how do workaround19:49
spatelLeo_m: do /var/opt/openstack-ansible/scripts/inventory-manage.py -l19:50
spatel /var/opt/openstack-ansible/scripts/inventory-manage.py -r <container_name>19:50
odyssey4meprometheanfire: check the test playbook in https://gist.github.com/odyssey4me/f2c2065a5496b09e219d5663ab304e9819:54
odyssey4meprometheanfire: try it out and see the results19:55
prometheanfirehow did you generate your test data?19:55
prometheanfirethat's the part I can't do19:55
odyssey4meprometheanfire: I used ansible to spit the var out in json19:56
prometheanfirewhich var19:56
* odyssey4me is lazy19:56
odyssey4mehttps://gist.github.com/odyssey4me/f2c2065a5496b09e219d5663ab304e98#file-test-playbook-yml19:56
odyssey4methe first issue is that the filtered_ironic_drivers in the role overwrites itself19:57
prometheanfireok19:57
odyssey4meit produces a single dict, not a list of dicts19:57
prometheanfirewell, you changed concat_drivers to a list19:57
prometheanfirewhich helped19:58
odyssey4methe second is that once that's fixed (see the play), you can't extract them in the conf file as it stands now because that doesn't work with a list of dicts... so the solution is to use a json_query to get a resulting list of attributes19:58
odyssey4methen some of them are duplicated, so we filter that through unique, and do the concatenation19:58
odyssey4methere might be a smart way to get rid of filtered_ironic_drivers, but meh19:59
odyssey4meI spent around 15 mins trying with just jmespath, and concluded that it wasn't the right tool19:59
prometheanfireodyssey4me: this I can work with, the change to list is what I was missing20:00
odyssey4meyep20:00
jrosserodyssey4me: can i trade you this one? https://review.openstack.org/#/c/587478/ check carefully given its another project templates one :)20:02
odyssey4mejrosser: rock and roll20:03
*** fghaas has joined #openstack-ansible20:03
jrosser\o/ /me relieved the working stuff stands a better change of staying that way20:04
odyssey4mejrosser: yep, thanks - great work getting that done in time for the RC!20:04
jrosserideally i'd like a nv integrated build too thats not a total car crash20:05
odyssey4menow if only these centos jobs would behave20:05
*** dxiri has quit IRC20:08
jrossercloudnull: i was condsidering putting in a wait_for to check the ssh was available, but it's restarted in a handler so wasnt quite sure where to put that20:08
spateljamesdenton: what is script for look like some external monitor, do i need that? https://raw.githubusercontent.com/dpham-rs/rpc-openstack/master/scripts/f5-monitor.sh20:12
spatelit says "F5 External monitor script for Liberty deployments"20:13
jamesdentonno, you don't need it20:13
spatelcool!20:13
spatelthanks!20:13
jamesdentonWe used it to get valid tokens from keystone and then hit services w/ a token20:13
logan-nice work fixing the bionic gpg stuff jrosser20:17
*** ansmith has quit IRC20:17
*** fghaas has quit IRC20:18
odyssey4mecores, I have plenty of reviews ready to look at if you're able: https://goo.gl/k9cCfF20:19
*** mmalchuk_ has joined #openstack-ansible20:21
jrosserlogan-: thanks :) it just got so ridiculously complicated - then i stopped trying to fix it and looked at what it actually did20:22
*** mmalchuk has quit IRC20:24
openstackgerritMatthew Thode proposed openstack/openstack-ansible-os_ironic master: fix ironic driver data structure usage  https://review.openstack.org/58963020:28
prometheanfireodyssey4me: ^20:28
odyssey4meprometheanfire: looks legit, have you got a test env to try it out on?20:29
prometheanfireyep, tested already20:30
prometheanfireconf output is valid20:30
prometheanfirerelocating and lunching now20:30
odyssey4meprometheanfire: hooray!20:31
openstackgerritMerged openstack/openstack-ansible-os_octavia master: Ensure the octavia flavor is private  https://review.openstack.org/58923920:33
openstackgerritMerged openstack/openstack-ansible-os_octavia master: Ensure the octavia network setup is in the right project  https://review.openstack.org/58924020:33
odyssey4meheh, check it out - zuul can now do in-line comments if you give it the right info: https://zuul-ci.org/docs/zuul/user/jobs.html#leaving-file-comments20:33
odyssey4mean example: https://review.openstack.org/#/c/589305/1/.zuul.yaml20:34
prometheanfireneat20:39
-openstackstatus- NOTICE: Due to a bug, Zuul has been unable to report on cherry-picked changes over the last 24 hours. This has now been fixed; if you encounter a cherry-picked change missing its results (or was unable to merge), please recheck now.20:41
*** nicolasbock has quit IRC20:45
jrosserlogan-: this could use a vars file for bionic https://github.com/logan2211/ansible-etcd/tree/master/vars20:48
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-repo_build master: Use a venv to build the python wheels  https://review.openstack.org/58963320:54
*** mmalchuk has joined #openstack-ansible20:57
*** mmalchuk_ has quit IRC21:00
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-repo_build master: Use a venv to build the python wheels  https://review.openstack.org/58963321:02
*** FrankZhang has quit IRC21:05
jrosserodyssey4me: "ERROR cinder.cmd.volume [-] Configuration for cinder-volume does not specify "enabled_backends". Using DEFAULT section to configure drivers is not supported since Ocata." thats likley whats breaking bionic21:07
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-repo_build master: Simplify pip/setuptools/wheel prep for venvs  https://review.openstack.org/58964321:18
*** dxiri has joined #openstack-ansible21:23
*** Leo_m has quit IRC21:40
*** Leo_m has joined #openstack-ansible21:46
*** hamzaachi has quit IRC21:46
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_magnum master: Execute service setup against a delegated host using Ansible built-in modules  https://review.openstack.org/58229621:46
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_magnum master: Execute service setup against a delegated host using Ansible built-in modules  https://review.openstack.org/58229621:47
*** cshen has quit IRC21:50
*** Leo_m has quit IRC21:51
*** Leo_m has joined #openstack-ansible21:57
openstackgerritMerged openstack/openstack-ansible-os_octavia master: Add retries to service role creation  https://review.openstack.org/58924521:59
openstackgerritMerged openstack/openstack-ansible-os_tempest stable/pike: Add tempest test blacklist capability  https://review.openstack.org/58850822:05
*** DanyC has quit IRC22:05
spateljamesdenton: I am seeing following pool but that doesn't make any sense22:09
spatelcreate ltm virtual /OSTACK/OSTACK_VS_GALERA_READ { destination 172.28.0.9:3307 ip-protocol tcp mask 255.255.255.255 pool /OSTACK/OSTACK_POOL_GALERA_READ profiles replace-all-with { /Common/fastL4 { } }  mirror enabled source-address-translation { pool /OSTACK/OSTACK_SNATPOOL type snat } }22:09
spatelPort 3307 ???22:09
spatelI don't think we need that22:09
*** priteau has quit IRC22:09
*** cshen has joined #openstack-ansible22:10
*** spatel has quit IRC22:13
*** cshen has quit IRC22:14
*** throwsb1 has quit IRC22:24
openstackgerritMerged openstack/openstack-ansible-os_octavia master: Use interface instead of endpoint_type  https://review.openstack.org/58924622:31
openstackgerritMerged openstack/openstack-ansible-os_octavia master: Add retries to all os_* module tasks  https://review.openstack.org/58944522:31
*** luksky has quit IRC22:37
*** cCain has joined #openstack-ansible22:41
*** bradm has joined #openstack-ansible22:43
openstackgerritMerged openstack/openstack-ansible stable/pike: Do not install linux-image-extra by default  https://review.openstack.org/58906522:50
openstackgerritMerged openstack/openstack-ansible stable/queens: Do not install linux-image-extra by default  https://review.openstack.org/58906422:50
*** ansmith has joined #openstack-ansible22:55
openstackgerritMerged openstack/openstack-ansible-galera_server master: Remove deprecated python-software-properties  https://review.openstack.org/58460923:07
*** priteau has joined #openstack-ansible23:14
*** pmannidi has joined #openstack-ansible23:18
*** vnogin has joined #openstack-ansible23:28
*** vnogin has quit IRC23:33
*** Leo_m has quit IRC23:46
*** Leo_m_ has joined #openstack-ansible23:46
openstackgerritKevin Carter (cloudnull) proposed openstack/openstack-ansible-ops master: Update retention process to use rollup  https://review.openstack.org/58967423:52

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!