16:01:25 <mnaser> #startmeeting openstack_ansible_meeting
16:01:26 <openstack> Meeting started Tue Oct 30 16:01:25 2018 UTC and is due to finish in 60 minutes.  The chair is mnaser. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:01:27 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:01:29 <openstack> The meeting name has been set to 'openstack_ansible_meeting'
16:01:35 <mnaser> #topic rollcall
16:01:36 <mnaser> o/
16:01:42 <nicolasbock> o/
16:01:48 <jrosser> o/
16:02:28 <spotz> o/
16:02:39 <mnaser> jp mentioned he's out sick this week :(
16:02:41 <admin0> \o
16:02:52 <mnaser> so everyone hopefully sending gooooooood things his way
16:03:21 <mnaser> #topic last week highlights
16:03:34 <mnaser> i dont see any new highlights added, i believe jrosser has mostly resolved these issues by dropping apt-cacher-ng ?
16:03:43 <cjloader> o/
16:03:49 <jrosser> hmm yes thats a couple of weeks oldnow
16:03:52 <mnaser> (fwiw i have done 2 rocky deployments on bionic and they're running flawlessly :))
16:04:05 <mnaser> doing a few more before the summit and it's rock-y solid
16:04:09 <mnaser> so thanks for your work jrosser
16:04:26 <jrosser> :) nice to see it all working
16:04:40 <mnaser> i dont think there's much other highlights afaik
16:04:50 <mnaser> just things humming, i've been pushing more and more fixes of things i see in prod
16:04:59 <mnaser> we'll probably have to sha bump cinder and nova soon
16:05:02 <mnaser> we caught a few critical rocky bugs
16:05:18 <mnaser> (in the projects themselves, not OSA, we're obviously writing flawless code, ha)
16:05:25 <jrosser> i started some work on massive reduction in packages installed arbitrarily everywhere
16:05:37 <mnaser> that'll be tremendous in speeding things up
16:05:52 <mnaser> we should schedule a sprint maybe together
16:05:58 <mnaser> to hack on making roles test using integrated repo
16:06:07 <odyssey4me> o/ attending two meetings at once, so only half here
16:06:12 <mnaser> odyssey4me: might be interested too :) so we can kill openstack-ansible-tests
16:06:16 <mnaser> but that's probably for later discussions
16:07:00 <mnaser> onto bugzz
16:07:03 <mnaser> #topic bug triage
16:07:08 <mnaser> #link https://bugs.launchpad.net/openstack-ansible/+bug/1800655
16:07:08 <openstack> Launchpad bug 1800655 in openstack-ansible "Utility container cannot reach repo in play: name: Get list of repo packages" [Undecided,New]
16:07:13 <odyssey4me> yeah, I think a sprint for distro installs would be good to consider too - given only opensuse is really working right now
16:08:07 <mnaser> odyssey4me: agreed.  we should plan something over the ML
16:08:15 <mnaser> also would help improve our upgrade jobs too
16:08:33 <mnaser> ok that bug it looks like the user said it was all fixed when they did iptables -F
16:08:42 <mnaser> and getting icmp "unreachable - admin prohibited, length 68"
16:08:54 <mnaser> we don't really put in any iptables rules eh?
16:09:03 <mnaser> it must have been created by the user
16:11:12 <mnaser> ..anyone
16:11:14 <mnaser> :)
16:11:32 <mnaser> i guess ill go for invalid
16:12:01 <jrosser> theres some discussion of this earlier wasnt there - firewalld related?
16:12:07 <jrosser> up there ^^^^ somewhere
16:12:22 <jrosser> buy yes it looks invalid
16:12:44 <mnaser> yep thats what i did and added a comment
16:12:50 <mnaser> #link https://bugs.launchpad.net/openstack-ansible/+bug/1800425
16:12:50 <openstack> Launchpad bug 1800425 in openstack-ansible "[os_keystone] Error when installing keystone with LDAP" [Undecided,New]
16:13:39 <odyssey4me> mnaser there was discussion earlier - it seems like there may be a package installing/depending on firewalld - so ideally we need to identify that and figure out how to surgcally fix it
16:13:49 <mnaser> i believe that is fixed by I0d7e4804bdc92ffe7a679060a686e684c01fcd1b
16:13:51 <mnaser> odyssey4me: oh gross
16:14:05 <mnaser> i've been deploying on bionic here with no problems and our gate works? so im not sure
16:15:44 <odyssey4me> yeah, this is centos specific - the firewalld thing I mean
16:16:03 <odyssey4me> the ldap thing, I think there was another follow on patch after I0d7e4804bdc92ffe7a679060a686e684c01fcd1b which was needed
16:16:03 <mnaser> firewalld comes on the host by default, i havent seen it get pulled into the containers
16:16:20 <mnaser> odyssey4me: the user commented that https://github.com/openstack/openstack-ansible/blob/stable/rocky/playbooks/os-keystone-install.yml#L117 fixed for them
16:16:23 <odyssey4me> yeah, it's not the container - it's a block on the host
16:16:31 <mnaser> which was backported on here https://github.com/openstack/openstack-ansible/commit/d350df49f30f38d180422af7b7b4559ac6b31b30
16:16:35 <mnaser> so just not relesed yet?
16:16:49 <mnaser> yeah, part of what we do when we prep centos hosts is get rid of firewalld and networkmanager
16:16:59 <mnaser> should probably document that
16:16:59 <odyssey4me> oh yes, https://review.openstack.org/#/q/I0d7e4804bdc92ffe7a679060a686e684c01fcd1b was the follow on patch :p
16:17:20 <mnaser> i set that to fix released and added the reference
16:17:21 <odyssey4me> mnaser yeah, we should either document it or automate it
16:17:27 <mnaser> errr fix committed
16:17:41 <mnaser> i think document is solution, automate feels dirty in case someone actually depends on firewalld
16:17:53 <mnaser> then surrpise your firewall rules are gone
16:18:13 <mnaser> ill update that first bug
16:18:51 <odyssey4me> Agreed.
16:19:03 <odyssey4me> And for the latest bug mentioned, 17.1.2 includes the fix - I commented in the bug.
16:19:35 <mnaser> ah cool, my bad
16:19:44 <mnaser> great, i updated the first one, tagged as low-hanging-fruit too, easy simple doc change
16:19:52 <mnaser> #link https://bugs.launchpad.net/openstack-ansible/+bug/1800185
16:19:52 <openstack> Launchpad bug 1800185 in openstack-ansible "Upgrades from <Newton carry unnecessary user-space env.d files" [Undecided,New]
16:20:18 <mnaser> hmm
16:20:20 <mnaser> this is a good one
16:20:28 <mnaser> so many envs i run into with env.d/<theentireworld>.yml
16:21:07 <mnaser> it seems like a bit of a hard problem to solve because if we just verify checksums, the yaml content might be the same
16:21:11 <odyssey4me> Yep, I don't know the best way to check for exact copies and to remove those - but hopefully someone can pick that up and figure it out.
16:21:12 <mnaser> so it might involve writing some sort of utility
16:21:39 <odyssey4me> it's plausible that rsync can do it, or perhaps diff
16:22:17 <mnaser> yeah but any rearranging that we'd do of the file would make it look different on disk
16:22:21 <mnaser> even tho it evaluates to the same thing
16:22:31 <mnaser> that's just nitpicking at the problem though, i agree it does exist
16:22:46 <mnaser> i dunno how critical we can consider it though, i'm inclined to put under wishlist..
16:23:31 <odyssey4me> yep, it's more of a wishlist thing
16:23:46 <mnaser> ok cool
16:23:51 <mnaser> #link https://bugs.launchpad.net/openstack-ansible/+bug/1800169
16:23:51 <openstack> Launchpad bug 1800169 in openstack-ansible "ansible-hardening role runs out of memory when running for ~60 nodes" [Undecided,New]
16:25:47 <mnaser> odyssey4me: i know you might be busy but i feel like you're experienced at the include/include_tasks/import_tasks thing
16:26:56 <mnaser> oo
16:26:58 <mnaser> "I realized huge memory increases when the tasks was doing include_tasks. I checked the imports with static import_tasks and the memory usage remained 1.5GB."
16:27:24 <odyssey4me> yep, interesting huh - and given rocky onwards has switched to using include_tasks I think we have a pretty big problem looming
16:27:54 <mnaser> i'm going to propose to leave it open for now
16:27:58 <odyssey4me> I'll try and find some time to work on this - this is one of those things that's going to bite us.
16:28:01 <mnaser> to discuss next week with more folks hopefully
16:28:13 <mnaser> but yeah this might be a nasty surprise
16:28:30 <mnaser> #link https://bugs.launchpad.net/openstack-ansible/+bug/1800100
16:28:30 <openstack> Launchpad bug 1800100 in openstack-ansible "Aodh has wrong admin and internal endpoint in group vars" [Undecided,New] - Assigned to KimMinsik (for-beatitudo)
16:28:45 <jrosser> i think that got fixed?
16:28:48 <odyssey4me> include_tasks also doesn't let the tags work as operators might be used to, so we really need to get the right thing done in the roles
16:28:49 <mnaser> yay merged
16:29:39 <mnaser> #link https://bugs.launchpad.net/openstack-ansible/+bug/1799910
16:29:39 <openstack> Launchpad bug 1799910 in openstack-ansible "glance_cors_allowed_origin should include protocol " [Undecided,New]
16:30:13 <mnaser> aw man
16:30:15 <mnaser> with a patch too
16:30:34 <mnaser> i'm going to ask the patch submitted if they're interested in working to push it to gerrit
16:31:17 <mnaser> done, will leave it open till next week in case
16:31:26 <mnaser> #link https://bugs.launchpad.net/openstack-ansible/+bug/1799514
16:31:26 <openstack> Launchpad bug 1799514 in openstack-ansible "Test environment example in openstack-ansible" [Undecided,New]
16:31:26 * hwoarang was on an intenal meeting. sorry for being late
16:32:27 <mnaser> yup
16:32:30 <mnaser> that is indeed a mistake
16:32:56 <mnaser> ill keep it open and ask if the bug reporter would be interested in getting some help pushing up the patch
16:33:43 <mnaser> done
16:33:49 * mnaser is trying to foster helping people push patches
16:34:03 <mnaser> #topic open discussion
16:34:20 <mnaser> well
16:34:25 <mnaser> https://etherpad.openstack.org/p/OSA-berlin-planning
16:34:28 <mnaser> that doesnt seem to load for me
16:34:32 <mnaser> okay it finally is now
16:35:29 <mnaser> some cool sessions with OSA
16:35:39 <mnaser> let's try to mobilize and split/concur :)
16:36:20 <mnaser> chandankumar, arxcruz: any progress from IBM for merging our tempest testing? :p
16:36:25 <mnaser> any reviews you want eyes on, etc?
16:36:34 <arxcruz> mnaser: LOL
16:36:40 <arxcruz> it's a work in progress
16:36:50 <arxcruz> mnaser: so, tempestconf initial patch is almost done
16:36:54 <mnaser> cool, look forward for power9 support patch too :p
16:37:02 <arxcruz> some tests already pass using tempestconf instead of tempest
16:37:11 <mnaser> w00t
16:37:13 <mnaser> that sounds awesome
16:37:17 <arxcruz> https://review.openstack.org/#/c/612968/
16:37:34 <mnaser> thanks for your work
16:37:44 <arxcruz> now i'm checking why tempest.conf isn't being collected
16:38:02 <mnaser> so you get 1-2 hour attempts
16:38:03 <mnaser> fun
16:38:11 <arxcruz> and checking with other jobs if tempest.conf created by tempestconf is the same as the old tempest.conf
16:38:12 <mnaser> might be an openstack-ansible-tests change most likely
16:38:41 <arxcruz> mnaser: after that, we will add the options to remove and add specific config parameters to tempestconf
16:38:48 <arxcruz> and of course, rpm installation
16:39:37 <mnaser> cool
16:39:50 <arxcruz> and of course, i'll be in OSA berlin meeting :)
16:40:20 <mnaser> awesome!
16:40:28 <mnaser> yes quite a few of us will be there
16:40:42 <arxcruz> well, I have to, I live here :P
16:41:19 <mnaser> ahaha
16:41:23 <mnaser> didnt know but now i do :p
16:42:01 <mnaser> cool, anything else for open discussion?!
16:42:06 <ianychoi> o/
16:42:28 <arxcruz> so, that's all I have for now, I hope to have tempestconf up and running till next meeting
16:42:28 <ianychoi> I wanna ask reviews for https://review.openstack.org/#/c/613752/ and https://review.openstack.org/#/c/604567/ for translated Ansible documentation publication!
16:43:28 <mnaser> the second patch seems to be merged
16:43:55 <ianychoi> oh sorry wrong link
16:43:56 <spotz> arxcruz: You're in charge of finding a place for us to have our outing!:)
16:43:59 <odyssey4me> oh, there is another point to discuss - placement API
16:44:01 <ianychoi> For https://review.openstack.org/#/c/604567/ , seems that eumel8 is so busy with preparing Berlin Summit.. Would some Ansible team members help make templatization for For https://review.openstack.org/#/c/604567/ , seems that eumel8 is so busy with preparing Berlin Summit.. Would some Ansible team members help make templatization for https://review.openstack.org/#/c/612510/ patch?
16:44:27 <ianychoi> sorry https://review.openstack.org/#/c/612510/ is correct :)
16:44:45 <arxcruz> spotz: ack!
16:44:47 <mnaser> ianychoi: for the second one i'd be inclined to agree with jp's comment
16:44:56 <dcdamien> o/
16:45:18 <odyssey4me> ianychoi yeah I'll revise the patch after the meeting
16:45:38 <ianychoi> odyssey4me, thanks a lot! :) I agree with mnaser :)
16:45:43 <mnaser> the first one is an "honestly, i'll +W because it looks ok but i'm not too sure"
16:45:47 <dcdamien> I just want you let you know that there is still an issue with open-iscsi:
16:45:51 <mnaser> odyssey4me: yes, i wanted to bring that up, gah good call.
16:45:53 <dcdamien> #link https://bugs.launchpad.net/openstack-ansible/+bug/1704161
16:45:54 <openstack> Launchpad bug 1704161 in openstack-ansible "open-iscsi post-installation script fails in cinder-api-container" [High,New]
16:46:06 <mnaser> dcdamien: let's get to that in a second :)
16:46:08 <odyssey4me> mnaser yeah, I chose to just trust that ianychoi knows what he wants and tested it - 'cos argh
16:46:16 <mnaser> i wanted to draft up a response to matt regarding OSA
16:46:25 <mnaser> so #1 i think we need to get a placement role written first
16:46:55 <odyssey4me> mnaser yes, definitely - if someone can pick that up I'd appreciate it... I'm short on time
16:46:56 <ianychoi> Thanks a lot for kind help on two reviews! For the first one, I tested it locally and hopefully it solves all I18n publication issues
16:46:57 <mnaser> which i believe shouldn't be all that difficult
16:47:47 <odyssey4me> we'll then need to get it used across our role tests
16:47:48 <mnaser> honestly, i'm so short on time these days too.  i could hack on it this weekend but i cant guarantee much.
16:47:56 <mnaser> and then figure out upgrades.
16:48:27 <mnaser> which quite honestly is the most complicated bit
16:48:56 <mnaser> because i dont think we even have much of functional upgrade jobs to start with :x
16:49:38 <odyssey4me> in the integrated build we have a good working upgrade test, although not enough eyes are on them and actively doing fixes for where they break.
16:49:50 <mnaser> oh but "it works"?
16:49:57 <mnaser> i thought it was pretty much broken
16:50:09 <mnaser> thats pretty good to hear
16:50:29 <odyssey4me> Given our job changes have settled down a bit, I think I should be able to rebase my patch to enable roles to use the integrated build which will make using placement easier.
16:51:09 <mnaser> oh you have a wip patch for that? i think we should prioritize that, the # of things biting us from openstack-ansible-tests is just too much
16:51:20 <odyssey4me> I do, yes.
16:51:47 <odyssey4me> https://review.openstack.org/551791
16:52:16 <odyssey4me> I'll probably simplify it a bit now that I've had time to percolate and try a few things.
16:52:27 <mnaser> okay if you can take time getting that to pass, i will try to take up sometime to build os_placement
16:52:36 <mnaser> i'll probably go through the whole
16:52:39 <mnaser> add a new project in infra thing
16:53:40 <mnaser> if anyone else has time, please feel free to pick it up
16:53:48 <mnaser> as this will be sitting on a very big list :(
16:54:34 <spotz> mnaser: I might be able to work on something just finishing a peer review at work
16:55:26 <mnaser> spotz: ping me when you're done with that and i can try to defer
16:55:45 <openstackgerrit> Merged openstack/openstack-ansible-os_cinder master: tasks: cinder_qos: Use the new OpenStack CLI client  https://review.openstack.org/561218
16:56:15 <openstackgerrit> Damian Cikowski (dcdamien) proposed openstack/openstack-ansible master: Add extra volume types to AIO  https://review.openstack.org/561266
16:58:05 <spotz> mnaser: Ok, plotting with arxcruz for outing:) https://www.hofbraeu-wirtshaus.de/en/berlin/ takes reservations:)
16:58:27 <arxcruz> reservations can be done on the website
16:58:37 <arxcruz> they have people speaking english as well
16:58:45 <mnaser> i think that wraps it up
16:58:56 <mnaser> #endmeeting