16:00:59 <mnaser> #startmeeting openstack_ansible_meeting
16:01:00 <openstack> Meeting started Tue Nov 20 16:00:59 2018 UTC and is due to finish in 60 minutes.  The chair is mnaser. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:01:00 <spatel> just checked all 3 has same error
16:01:01 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:01:04 <openstack> The meeting name has been set to 'openstack_ansible_meeting'
16:01:11 <mnaser> #topic roll call
16:01:11 <mnaser> o/
16:01:17 <spatel> cloudnull: i will talk to you after this meeting :)
16:01:21 <mnaser> laggy bot
16:01:23 <mnaser> o/
16:01:29 <cloudnull> o/
16:01:37 <jrosser> o/
16:02:12 <mnaser> quiet one
16:02:15 <evrardjp> o/
16:02:16 <mnaser> okay so lets get started
16:02:38 <mnaser> #topic what happened since last meeting
16:02:43 <mnaser> not much, i think.  we were mostly all having a great time in berlin :) i had a really good deployemnt tool feedback session
16:02:51 <mnaser> i had a few action items that i carried out
16:02:58 <mnaser> also, project on-boarding and project update were amazing
16:03:27 <mnaser> clearly there is an amazing amonut of interest in OSA, so let's all keep up on that awesome work :D super motivating
16:03:41 <chkumar246> arxcruz, thanks for suggestion it makes sense, I will update once dependent patch finishes the output
16:04:12 <mnaser> other things: we're working on os_placement and hopefully we'll land that eventually
16:04:13 <mnaser> it's cool because we're kinda one of the leaders in that so it would be nice for OSA to be "FIRST!"
16:04:31 <mnaser> other than that, jrosser brought up arm64 so maybe want to expand on that
16:04:49 <cloudnull> ++
16:05:01 <jrosser> so, out of mostly curiosity i had a go on the new arm64 ci nodes
16:05:11 <odyssey4me> :)
16:05:24 <jrosser> and then entirely coincidentally nowster showed up here in the channel trying to do it for real
16:05:27 <mnaser> jrosser: if you need any support on arm stuff, i can get people your way :)
16:05:52 <jrosser> so we've been just poking at that a bit and seeing how far it gets
16:06:06 <jrosser> theres a very obvious and now fixed bug in haproxy
16:06:07 <mnaser> if you are going to be doing some work on that, i'd love to make an introduction to massimo who leads a lot of the open source / openstack stuff and he is a really really useful asset in terms of maybe even getting you resources to accomlpish the things you need
16:06:30 <jrosser> that would be great
16:06:47 <mnaser> wonderful, ill send that soon
16:06:54 <jrosser> the other thing is that the performance of the arm64 nodes isnt great, so we need someone to help take a look at that
16:07:10 <jrosser> i suspect the storage is slow
16:07:32 <mnaser> perfect, he'll actually be able to help with all that stuff
16:07:48 <evrardjp> there are other ppl you might want to reach, like linaro too, should you require some help
16:08:03 <jrosser> for anyone who is interested here is the test patch https://review.openstack.org/#/c/618305/
16:08:12 <mnaser> also, i might be able to get you some.. faster resources for local testing
16:08:16 <mnaser> will follow up more in that email
16:08:27 <jrosser> cool :)
16:09:12 <mnaser> anything else from last week?
16:09:21 <guilhermesp> late o/
16:09:40 <mnaser> :)
16:10:07 <guilhermesp> so I think you noticed that Ive been working on os_placement
16:10:23 <guilhermesp> here's the pr https://review.openstack.org/#/c/618820/
16:10:48 <guilhermesp> I'm still building the basics of the role, but I still need to create the tests
16:11:00 <guilhermesp> seems that the placement api works under apache + uswgi
16:11:25 <guilhermesp> mnaser: mentioned that there is also red hat packages for distro install
16:11:28 <guilhermesp> is that right?
16:11:43 <mnaser> yes, rdo apparently publishment placement packages already
16:12:07 <guilhermesp> at the moment I'm focusing in source install
16:12:19 <guilhermesp> but good to know
16:12:28 <mnaser> yep lets get that and we can improve/iterate
16:12:37 <mnaser> anything else or shall we move onto bug triage?
16:12:57 <cloudnull> just a quick note about the summit -- odyssey4me and I were in fhaas training session (which was great!), if any folks in the channel here were in that session and want to follow up on some of the things we talked about in the all day session please reach out.
16:13:17 <mnaser> s/fhaas/fghaas/ you're welcome :p
16:13:21 <cloudnull> ^
16:13:23 <cloudnull> :)
16:13:31 <mnaser> #topic bug triage
16:13:33 <mnaser> #link https://bugs.launchpad.net/openstack-ansible/+bug/1804204
16:13:35 <openstack> Launchpad bug 1804204 in openstack-ansible "Quickstart: AIO in openstack-ansible ubuntu instructions" [Undecided,New]
16:14:05 <mnaser> yep
16:14:06 <mnaser> thats confirmed
16:14:28 <evrardjp> valid
16:14:34 <evrardjp> low-hanging-fruit
16:14:37 <evrardjp> low ?
16:15:11 <mnaser> i'd say medium considering .. its broken
16:15:29 <cloudnull> something like this came up in the project onboarding ++ Id say medium
16:15:43 <mnaser> asked if paul can push up a patch
16:16:01 <mnaser> if not we can figure things out :)
16:16:01 <mnaser> #link https://bugs.launchpad.net/openstack-ansible/+bug/1803444
16:16:02 <openstack> Launchpad bug 1803444 in openstack-ansible "gnocchi playbook failing on 18.0.0" [Undecided,New]
16:16:37 <mnaser> include vs import_playbook black magic
16:16:52 <mnaser> does anyone knows why the above might occur?
16:17:52 <mnaser> seems related to ceph_client too
16:18:09 <evrardjp> checking
16:18:11 <mnaser> wait is it because the include is include-ing it regardless?
16:18:30 <mnaser> because it looks like that deployer is using ceph
16:18:30 <mnaser> so that block should be entirely ignored
16:18:43 <evrardjp> I tried to avoid that playbook for a while :p
16:18:45 * cloudnull reading
16:18:46 <noonedeadpunk> hm, never faced with this one...
16:19:05 <nowster> mnaser: Just to let you know that I'm working with "Works On ARM" too (ie. Massimo).
16:19:22 <mnaser> nowster: oh awesome!
16:19:22 <cloudnull> fatal: [infra1_gnocchi_container-7ed445f6]: FAILED! => {"msg": "No file was found when using with_first_found. Use the 'skip: true' option to allow this task to be skipped if no files are found"}
16:19:23 <evrardjp> we should probably adapt the play to only run the appropriate role if a grouping of host was conditionally successfull
16:19:27 <mnaser> that fits perfectly then :)
16:19:29 <cloudnull> is generally an issue with facts
16:19:47 <cloudnull> like the OS was not found or there were no valid facts and ansible didnt regather
16:19:50 <evrardjp> cloudnull: I agree on the ceph client part
16:20:20 <mnaser> yeah so i think no facts were gathered for that because no tasks actually ran on the gnocchi container
16:20:21 <mnaser> at least in that specific section that it was breaking on
16:20:28 <mnaser> which was the setup-things workaround
16:20:46 <cloudnull> as for the "when" in the import playbook.
16:20:53 <cloudnull> im not sure that actually works any longer.
16:20:55 <evrardjp> we should maybe ask for specific runs
16:21:02 <evrardjp> cloudnull: yeah my point exactly
16:21:09 <evrardjp> cloudnull: well tbh I am not sure WHEN it worked :p
16:21:12 <cloudnull> IDK if import_playbook allows for vars to be passed through
16:21:14 <evrardjp> double pun inside!
16:21:22 <mnaser> cause an import isn't really meant to be conditional right?
16:21:24 <evrardjp> cloudnull: it was never intended for that
16:21:28 <cloudnull> ++
16:21:29 <evrardjp> mnaser: right
16:21:50 <cloudnull> it worked with include because we took advantage of the overloaded include system
16:22:23 <evrardjp> yeah
16:22:28 <mnaser> is there include_playbook ..?
16:22:31 <evrardjp> nope
16:22:37 <evrardjp> only import
16:22:39 <mnaser> so what we need to do is move that logic into os-gnocchi right?
16:22:40 <evrardjp> but as I said
16:22:47 <cloudnull> include will load whatever it finds while import is more like python.
16:22:48 <evrardjp> we can move some part into the play
16:22:50 <evrardjp> and include_role
16:22:55 <cloudnull> ++
16:23:07 <evrardjp> or the meta to end the playbook early
16:23:16 <evrardjp> (but I'd rather avoid that one)
16:23:47 <evrardjp> for the sake of the bug, I'd like to see a real run of gnocchi
16:23:48 <mnaser> is there a quick 2 minute explanation
16:23:56 <mnaser> as to why this even happens?
16:23:58 <evrardjp> to see if it happens again
16:24:09 <evrardjp> I mean of the gnocchi playbook
16:24:11 <mnaser> also noonedeadpunk mentions he hasnt ran into that.. do you use ceph?
16:25:11 <mnaser> do we wanna dig into this again next week?
16:26:05 <evrardjp> mnaser: that sounds reasonable
16:26:08 <noonedeadpunk> mnaser: yep, I'm running ceph
16:26:14 <francois> hi guys, I'm the author of this bug report and am available if you need any other info :)
16:26:15 <evrardjp> noonedeadpunk: with gnocchi?
16:26:20 <evrardjp> francois: oh great!
16:26:24 <evrardjp> perfect!
16:26:25 <openstackgerrit> Kevin Carter (cloudnull) proposed openstack/openstack-ansible-ops master: Correct clustered gating job  https://review.openstack.org/619033
16:26:36 <evrardjp> francois: could you run just that gnocchi play again?
16:27:10 <evrardjp> openstack-ansible os-gnocchi-install.yml
16:27:15 <evrardjp> it should not fail :p
16:27:24 <francois> evrardjp: ok
16:27:27 <noonedeadpunk> yep. but, I'm not running swift...
16:27:28 <evrardjp> if you put the log somewhere that would be nice
16:27:39 <evrardjp> noonedeadpunk: that's the deal :)
16:27:51 <noonedeadpunk> And now I'm wondering, how I get it installed at all...
16:28:08 <evrardjp> francois: because if it repeats itself with that case, it's not a facts problem, but a playbook problem
16:29:12 <noonedeadpunk> oh, we're including gnocchi role twice...
16:29:17 <mnaser> francois: can you please update that bug with the info
16:29:20 <mnaser> and we'll cotninue
16:29:29 <mnaser> #link https://bugs.launchpad.net/openstack-ansible/+bug/1803382
16:29:30 <openstack> Launchpad bug 1803382 in openstack-ansible "unbound-client.yml failure depending on TERMCAP envvar content" [Undecided,New]
16:29:48 <francois> mnaser: yeah, will do that
16:29:53 <mnaser> i ran into that bug too
16:30:10 <mnaser> it happened when i was running something in screen
16:30:10 <mnaser> i had *no* idea why it was happening
16:30:10 <francois> also mine :)
16:30:13 <mnaser> but it didn't ahppen anymore in screen
16:30:25 <francois> this one looks pretty weird!
16:30:40 <mnaser> i have no idea why it honestly happens
16:30:55 <evrardjp> I was told that jrosser had the same issue
16:31:10 <evrardjp> we can probably say confirmed then
16:31:17 <francois> an easy workaround is simply deleting this variable from /etc/openstack_deploy/ansible_facts/localhost
16:31:36 <dcdamien> I have the same issue too
16:31:37 <evrardjp> we should unset said thing in the connection plugin
16:31:42 <francois> but I'm quite interested in how it could happen :)
16:31:55 <mnaser> i will put this as confirmed/high
16:31:59 <mnaser> running things in screen is kinda.. important.
16:32:03 <evrardjp> tmux?
16:32:05 <evrardjp> :p
16:32:11 <noonedeadpunk> +
16:32:13 <evrardjp> never had an issue with tmux :p
16:32:16 <evrardjp> hahaha
16:32:18 <evrardjp> anyway
16:32:20 <guilhermesp> byobu rules
16:32:20 <evrardjp> yeah
16:32:21 <cloudnull> tmux for life!
16:32:22 <evrardjp> important
16:32:47 <mnaser> francois: do you have sometime to dig a little bit into that one?
16:32:49 <guilhermesp> byobu = tmux lover too
16:32:51 <evrardjp> I digged through a rabbit hole long ago with a rabbitmq issue of the env var that was passed
16:33:23 <evrardjp> but we changed our connection plugin now, and we are not relying on the same behaviours anymore
16:33:32 <evrardjp> so you're hitting an issue in ansible itself
16:33:38 <evrardjp> probably
16:33:52 <evrardjp> worth checking in the bugs there too
16:33:56 <mnaser> i put the workaround but yeah, i have no idea, we cant solve it now, just triaging :)
16:33:58 <mnaser> #link https://bugs.launchpad.net/openstack-ansible/+bug/1803347
16:34:00 <openstack> Launchpad bug 1803347 in openstack-ansible "Documentation mentions requirement of bridges on compute nodes" [Undecided,New] - Assigned to Mohammed Naser (mnaser)
16:34:04 <evrardjp> mnaser: yeah lgtm
16:34:07 <mnaser> hah i remember this
16:34:19 <noonedeadpunk> Yep, my question, sorry.
16:34:19 <evrardjp> :)
16:34:22 <mnaser> confirmed low
16:34:42 <mnaser> #link https://bugs.launchpad.net/openstack-ansible/+bug/1803019
16:34:43 <openstack> Launchpad bug 1803019 in openstack-ansible "Nova generates versioned_notifications but no one consume" [Undecided,New] - Assigned to KimMinsik (for-beatitudo)
16:35:26 <noonedeadpunk> Yeah, it's problem but more about Nova
16:35:32 <mnaser> in progress medium
16:35:33 <noonedeadpunk> I know the workaround to it
16:35:49 <noonedeadpunk> ah, ok
16:36:46 <evrardjp> maybe documenting the workaround is good?
16:37:01 <noonedeadpunk> it's already a patch for it present...
16:37:44 <evrardjp> yeah we can talk about it in the review if you prefer
16:38:12 <noonedeadpunk> not at all. But it's exactly my way of resolving this.
16:38:39 <noonedeadpunk> but I've just overrided nova.conf with extra params
16:38:56 <mnaser> yep, its nice the patch has 'native' feature so itd be good to review it
16:39:03 <mnaser> #link https://bugs.launchpad.net/openstack-ansible/+bug/1802742
16:39:04 <openstack> Launchpad bug 1802742 in openstack-ansible "parameterized nova placement nginx error log level" [Undecided,New]
16:39:34 <mnaser> that seems like wishlist
16:40:16 <noonedeadpunk> and for ocata...
16:40:26 <mnaser> it probably still applies
16:40:27 <mnaser> wait it doesnt
16:40:29 <mnaser> we dropped nginx right?!
16:40:57 <noonedeadpunk> I think it's just plain uwsgi instead..
16:40:59 <mnaser> yeah
16:41:02 <mnaser> you're right
16:41:50 <mnaser> meh confiremd/wishlist
16:41:56 <mnaser> #link https://bugs.launchpad.net/openstack-ansible/+bug/1802466
16:41:57 <openstack> Launchpad bug 1802466 in openstack-ansible "keystone cannot generate keystone.log file" [Undecided,New] - Assigned to KimMinsik (for-beatitudo)
16:42:16 <evrardjp> not everywhere, but for nova placement I don't know
16:42:32 <mnaser> placement is uwsgi
16:42:38 <mnaser> also thats cool, we got a fix for that too
16:42:49 <mnaser> +2d it
16:43:11 <mnaser> #link https://bugs.launchpad.net/openstack-ansible/+bug/1802195
16:43:12 <openstack> Launchpad bug 1802195 in openstack-ansible "ceph-rgw installation/configuration lacks idempotency, configuration changes break the setup" [Undecided,New]
16:43:15 <evrardjp> for the last one, I am worrying about container non container
16:44:05 <fghaas> evrardjp: how so?
16:44:56 <evrardjp> fghaas: well it was for the keystone cannot generate, which is not the last one, I am a little slow on typing :p
16:45:08 <evrardjp> sorry for being in 2 meetings at the same time :p
16:45:30 <fghaas> evrardjp: ack, thanks for clarifying :)
16:45:57 <mnaser> fghaas: is that an osa issue or ceph-ansible?
16:46:12 <noonedeadpunk> btw, about keystone - I don't have /var/log/keystone at all, as all log files in journalctl
16:46:23 <fghaas> it's an osa issue in that osa pulls in a state of ceph-ansible that was never meant for production.
16:46:49 <fghaas> I hope the commit messages explain that fairly well
16:47:29 <fghaas> I think that bug is only in New because the actual thing to be fixed is in Rocky, and the fix in master is Related
16:47:59 <mnaser> fghaas: so does that mean its resolved ?
16:48:37 <fghaas> I certainly like to think so, and haven't heard anyone scream otherwise
16:49:18 <mnaser> ok cool
16:49:21 <evrardjp> noonedeadpunk: that sounds more likely this is why I am concered of said patch.
16:49:34 <mnaser> #link https://bugs.launchpad.net/openstack-ansible/+bug/1802132
16:49:35 <openstack> Launchpad bug 1802132 in openstack-ansible "OpenStack Ansible dashboard not able to log in with admin credentials" [Undecided,New]
16:50:02 <mnaser> sounds like a faield deploy
16:50:20 <mnaser> asked to rerun horizon install
16:50:33 <mnaser> we'll review next week
16:50:34 <mnaser> #link https://bugs.launchpad.net/openstack-ansible/+bug/1802070
16:50:35 <openstack> Launchpad bug 1802070 in openstack-ansible "Add resource_filter field in cinder.conf for non-admin user can retrieve volume & snapshot volume" [Undecided,New] - Assigned to KimMinsik (for-beatitudo)
16:51:22 <mnaser> https://review.openstack.org/#/c/616113/
16:51:23 <mnaser> already a patch
16:51:41 <mnaser> lets run this by the cinder team
16:51:42 <mnaser> ill do that
16:51:46 <mnaser> just to kinda figure out what is going on
16:51:51 <mnaser> but it should be in progress
16:52:38 <mnaser> #link https://bugs.launchpad.net/openstack-ansible/+bug/1801656
16:52:39 <openstack> Launchpad bug 1801656 in openstack-ansible "fix tox python3 overrides" [Undecided,New]
16:53:15 <evrardjp> thanks for dealing with it mnaser
16:53:19 <mnaser> i guess we can keep that one
16:53:23 <mnaser> put it as confirmed
16:53:26 <mnaser> to track our py3 stuff
16:53:29 <evrardjp> ok
16:53:31 <evrardjp> fine for me
16:53:37 <evrardjp> it's almost all done anyway, isn't it?
16:53:48 <mnaser> it is mostly done i think
16:53:49 <noonedeadpunk> returning back to cinder, I thought, that policies resolves mentioned problem..
16:54:18 <mnaser> #topic open discussion
16:54:29 <mnaser> we can talk about the bugs outside the meeting, but it does seem like it resolves things, but best to run by cinder team
16:54:54 <evrardjp> I don't have any topic for open discussion
16:54:59 <spotz> jungleboyj and the rest of the team should be home by now
16:55:02 <mnaser> while we have a few minutes, got anything on side of arxcruz re: os-tempest?
16:55:08 <evrardjp> except I hoped you had a nice trip back from the summit
16:55:17 <evrardjp> oh yeah probably worth checking that direction
16:55:31 <arxcruz> mnaser: we are starting to add a job to test python-tempestconf
16:55:39 <jungleboyj> Yep.  I am.  :-)
16:55:57 <arxcruz> and from there, we will add new functionalities on python-tempestconf, like override settings, rpm package, etc
16:56:07 <jungleboyj> What is the problem?
16:56:28 <openstackgerrit> Jonathan Rosser proposed openstack/openstack-ansible-os_keystone master: Add libpython2.7 as a required package  https://review.openstack.org/619040
16:56:41 <guilhermesp> about os_placement, I'd really appreciate reviews and suggestions to finish the work https://review.openstack.org/#/c/618820/ :)
16:56:47 <mnaser> cool, thats awesome arxcruz
16:56:49 <mnaser> please send reviews
16:57:09 <arxcruz> mnaser: yup, evrardjp is already reviewing it :)
16:57:17 <mnaser> sweet
16:59:36 <mnaser> #endmeeting