16:02:14 #startmeeting openstack_ansible_meeting 16:02:15 Meeting started Tue Jul 24 16:02:14 2018 UTC and is due to finish in 60 minutes. The chair is mnaser. Information about MeetBot at http://wiki.debian.org/MeetBot. 16:02:16 mnaser: it's on the page 16:02:17 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 16:02:19 The meeting name has been set to 'openstack_ansible_meeting' 16:02:40 #topic roll call 16:02:43 o/ ? 16:02:58 o/ 16:03:02 (ʘ‿ʘ)╯ 16:03:06 o/ 16:03:16 o/ 16:03:17 bgmccollum: perfect there is a bug from you IIRC :) 16:03:41 \o 16:03:54 evrardjp for realz? 16:04:23 okay so 16:04:33 #topic last week highlights 16:04:50 i know evrardjp mentioned that he's updated the page quickly 16:04:55 * mnaser ddos f5 16:04:55 mnaser: it's done 16:05:02 odyssey4me: done too - one more question 16:05:14 given i didn't attend last meeting.. is there anything on that list from last week? 16:05:23 or are those all fresh? 16:05:37 mnaser: I will remove the old things 16:06:12 thank you evrardjp 16:06:39 mnaser: done 16:06:56 probably be translated into proper english 16:07:00 np! 16:07:13 so i guess as y'all see-- we need to do the work on bionic 16:07:30 this is extremely important in order to be able to test rocky for distro and make sure our dependencies don't mess up 16:07:37 allowing an upgrade path for those running xenial 16:07:50 maybe odyssey4me has further comments? 16:07:50 indeedy 16:08:19 * odyssey4me wishes he had time to work on it 16:08:37 odyssey4me: you did a lot recently in the cleanup. Do not worry too much, other ppl can step in too :p 16:08:43 but yes, for the distro install work on release day to work for ubuntu we need to get bionic done 16:08:47 it's an open thing, anyone is welcome to help! 16:08:49 :) 16:08:59 i am pushing some bionic patches so please keep voting :P 16:09:15 cool, so we're at least aware we gots to get things moving with that 16:09:17 yep, reviews are just as necessary as patches 16:09:43 hwoarang's work on leap 15 seems to be blocked from all sorts of external factors, so that's still a wip 16:09:56 o/ distractedly late:) 16:09:57 unfortunately we've had a storm of patches pushed up to do trivial things which make the review queue more difficult to spot the important things, so best to ask for reviews if nothing happens for a day 16:10:11 odyssey4me: ++ i agree, sometimes asking makes a big difference 16:10:13 I agree 16:10:19 * mnaser needs to do his review duty 16:10:41 I have a few ones I will ask you all to get in :p 16:10:51 we've got milestone 3 this thursday, as usual evrardjp is taking care of bumps 16:10:57 I'm back in town so poke and I'll do reviews at night 16:11:08 so while talking reviews just make sure we track those changes 16:11:13 to be on time 16:11:24 we should probably decide what level of freeze we're aiming for and what exceptions are important 16:11:25 mnaser: yeah -- so good news, we finally managed to merge a bump recently 16:11:52 when can be earliest branch? 16:11:59 I never remember 16:12:04 is that m3 or rc1? 16:12:07 rc1 16:12:10 darn 16:12:12 that's a lot 16:12:28 it's ok, we can treat it as stable/rocky :P 16:12:34 we have to tag the first rc, then branch from the tag as far as I know 16:12:48 I'd say it's possible to work on testing -- that doesn't involve architectural changes 16:12:55 improving stability 16:12:57 cleaning things up 16:13:02 no NEW feature 16:13:12 we can finish existing work if it's not fully over 16:13:30 yep 16:13:31 better something finished than half baked into prod 16:14:19 and finally i guess elections for PTL are open, fwiw i will be running 16:14:28 wooot 16:14:34 mnaser: for president! 16:14:40 Woot! 16:14:42 looks like we may have to live with repo_build for another cycle then - but next cycle is time for it to expire 16:14:58 :) thanks mnaser - appreciate your time and commitment! 16:15:10 odyssey4me: i think we had a lot of extra stuff kinda smack us like ci issues and being forced to add bionic, etc 16:15:22 hopefully a more stable upcoming cycle to clear that out 16:15:30 yeah, I'll finish up the current clean up things, then join forces ot get bionic done. 16:15:43 anyways, that's pretty much it for highlights. anything left for highlights? 16:15:43 sounds good 16:15:59 it'd be good to get that distro work done for the other roles next cycle too 16:15:59 yeah what I wrote for elections: 16:16:11 it's been very helpful to get things cleaned up 16:16:18 I am not running in case you didn't know it. 16:16:29 Time for a change. 16:16:31 ah yeah, that, sorry i kinda assume those are just infront of us 16:16:42 odyssey4me: 100% .. i wanna get centos to pass 16:16:47 it's been almost the entire cycle :'( 16:17:05 Anyone welcome to run, and I already thank ppl stepping forward. I like this :) 16:17:13 https://review.openstack.org/#/c/584604/ 16:17:22 where i'm stuck in centos anyways 16:17:33 if someone wants to help i'd be extremely welcoming of that 16:17:43 I'll continue to be social PTL:) 16:17:58 distro passes but 16:18:02 http://logs.openstack.org/04/584604/1/check/openstack-ansible-functional-centos-7/6f4abed/logs/host/cinder/cinder-volume.log.txt.gz 16:18:10 cinder-rtstool cant be found, or something.. maybe someone knows about it 16:18:15 heh, that's pretty 16:18:18 but anyways, back to meeting, i guess time for bug triage? 16:18:29 mnaser: Have you talkted to jungleboyj or smmginnis? 16:18:47 i have not yet, i'll have to do some digging still :) 16:18:58 been tryna catch up since my 2 week trip 16:19:07 #topic bug triage 16:19:10 Ok back to bugs, poke if needed. Someone did a patch for one of the two I had:) 16:19:16 mnaser: wrong shebang? 16:19:28 evrardjp: i dont know yet, gotta dig in more 16:19:31 #link https://bugs.launchpad.net/openstack-ansible/+bug/1783237 16:19:31 Launchpad bug 1783237 in openstack-ansible "git.openstack.org is super slow, we should switch to github.com" [Undecided,New] 16:19:35 check that :D 16:19:43 thats a good thing to check 16:20:11 eh. 16:20:19 its slow, but not that slow here 16:20:32 i mean it could be a doc bug 16:20:40 If there's an issue we should let Jimmy and infra know 16:20:55 but if we don't agree that it's a doc bug 16:20:57 i would put wontfix 16:21:18 thoughts? 16:21:50 well, we could make it easier to do this - and it would be nice for offline installs 16:22:11 thing is there is so many ways the url can be formed 16:22:18 We could add an alternative to use github with a warning 16:22:18 in my comment I noted a previous attempt review, and in that review I proposed a way to do it 16:22:37 so really the only way to do this is to override nova_git_repo or whatever the var is 16:22:44 which is already possible today? 16:22:53 the alternative approach was simply to pass the URL through a filter to allow search/replace when bootstrapping the roles and the repo 16:23:10 yes, today's approach is there and works, but it's a bit painful 16:23:37 so I'd say this is wishlist 16:23:50 I might just work on something for it one day 16:23:52 I'd say invalid to be honest 16:23:56 or won't fix 16:24:04 because it's already possible to do so 16:24:07 the bug title I disagree with entirely 16:24:18 ok 16:24:21 but a feature to make it easier to switch to another mirror - happy to do it 16:24:21 for me 16:24:32 odyssey4me: well we already have things like that 16:24:37 it's just not very friendly 16:24:45 wish list doesnt eixist 16:24:48 and it will change if we change how repo is made (your bp) 16:24:52 s/easier/friendlier/ 16:25:10 how to wishlist? :> 16:25:14 mnaser: 'wish list' is under 'importance' 16:25:18 mnaser: it's in importance 16:25:23 aaah 16:25:26 :) 16:25:45 for the bug title itself I'd not mark it wishlist 16:25:53 I'd mark it as won't fix 16:25:54 do we have docs for how to use a different mirror somewhere? 16:26:07 we've got a parallel git cloner too which might be of use at some point 16:26:09 we won't fix in this project how things are mirrored outside :p 16:26:26 jrosser: that's cool I wanted to write an ansible module for that. 16:26:33 boxrick: ^^ 16:26:41 odyssey4me: we have docs on how to changes the source 16:26:45 ok 16:26:47 refresh the page 16:26:48 odyssey4me: we may not have docs on a-r-r override 16:26:50 let me check 16:26:51 and tell me what you think 16:27:19 https://docs.openstack.org/openstack-ansible/latest/reference/configuration/extending-osa.html#adding-new-or-overriding-roles-in-your-openstack-ansible-installation and 16:27:27 what do you think about the bug now 16:27:27 evrardjp: I'm not seeing anything in https://docs.openstack.org/openstack-ansible/latest/user/limited-connectivity/index.html about how to use an alternative mirror for git for anything 16:27:31 https://docs.openstack.org/openstack-ansible/latest/reference/configuration/extra-python-software.html 16:27:33 please f5-ers 16:27:40 mnaser: ok 16:27:45 There is no reason we couldn't throw it into a module, its just a separate python script at the moment. 16:27:59 boxrick: I have some :) 16:28:11 heh, title change mnaser 16:28:17 i mean 16:28:20 that's really the core issue 16:28:35 and i feel like we can get to an agreement on that? 16:28:46 yeah but the conversation has lost its meaning 16:28:52 yeah, docs is certainly the easiest way to get this done 16:28:58 it's trackable though 16:29:00 so it's fine 16:29:02 At the moment it clones about 100 roles in 20 seconds? 16:29:05 ok 16:29:14 can we move on to the next bug now? :> 16:29:19 boxrick: on your amazing connection, of course 16:29:27 well 16:29:30 we do have docs though :p 16:29:32 Well this is just on my naff laptop over wifi 16:29:36 just not linked. 16:29:44 right, we can add that as a comment 16:29:45 we can add links. 16:29:47 #link https://bugs.launchpad.net/openstack-ansible/+bug/1783145 16:29:47 Launchpad bug 1783145 in openstack-ansible "Document how to use config overrides for multivalued keys in ini files" [Undecided,New] 16:29:58 is that even possible 16:29:59 confirmed low 16:30:01 yeah 16:30:05 ok 16:30:46 #link https://bugs.launchpad.net/openstack-ansible/+bug/1783132 16:30:47 Launchpad bug 1783132 in openstack-ansible "designate.conf.j2 does not respect user configured designate database name" [Undecided,New] 16:30:59 oh its hardcoded 16:30:59 shucks 16:31:19 ooo 16:31:21 patch included too 16:31:36 confirmed medium low-hanging-fruit 16:31:45 anyone to pick that fruit? 16:31:49 anyone has 2 minutes to push up that patch? 16:31:59 new contributors? 16:32:06 pretty please please please :) 16:32:18 we know how to do it, you might want to love this 16:32:26 well can give a try :) 16:32:32 yay 16:32:38 some may monitor the low-hanging-fruit tag and submit, but may not be here 16:32:42 (mnaser odyssey4me please don't fix it -- I'd rather keep it for a new person OR for an onboarding session) 16:32:47 low hang fruits will help me to reach the highest branches of the tree 16:32:54 guilhermesp: great! 16:33:12 I started with low-hanging Barbican fruit:) 16:33:12 next? 16:33:12 guilhermesp: is your lp 'guilhermesp' too? 16:33:29 sorry, lp? 16:33:33 launchpad user 16:33:37 yes 16:33:39 it is 16:33:45 :) 16:34:10 great, i've assigned it to you, thank you and please ping any of us for help 16:34:17 #link https://bugs.launchpad.net/openstack-ansible/+bug/1782856 16:34:17 Launchpad bug 1782856 in openstack-ansible "/openstack directory getting full " [Undecided,New] 16:34:20 o/ 16:34:51 Invalid 16:34:54 for me that's normal 16:34:59 OR 16:35:10 Confirmed/wishlist to document things. 16:35:21 yeah, we should really change that pattern 16:35:31 yeah but what's done is done 16:35:35 well 16:35:37 no 16:35:39 that's not what I mean 16:35:43 I looked into setting a var for it once, and got lost in the millions of places where we don't bother respecting the var 16:36:04 I meant: the code is there, so we need to think about how to change it for the future, and we'll still have to deal with the past--- so how do we do this? 16:36:08 but anyway, a little extra docs would be nice to warn people, I really did think we had something for that 16:36:17 odyssey4me: me too 16:36:48 confirmed wishlist 16:36:51 odyssey4me: that's all we have: https://docs.openstack.org/project-deploy-guide/openstack-ansible/latest/targethosts.html#configuring-the-storage 16:36:51 added comment 16:36:52 something here: https://docs.openstack.org/project-deploy-guide/openstack-ansible/latest/targethosts.html#configuring-the-storage 16:36:57 lol 16:36:57 lol, snap 16:36:58 oh my god 16:36:58 hahah 16:37:13 added that url there 16:37:21 #link https://bugs.launchpad.net/openstack-ansible/+bug/1782798 16:37:21 Launchpad bug 1782798 in openstack-ansible "AIO Build Fails on SELinux File Context Tasks" [Undecided,New] 16:37:44 it proves our doc is easy to parse, and well structured,as we came up with the SAME link. 16:37:49 #proud 16:38:00 :) 16:38:17 I'm in favor of just turnig selinux off entirely and if it's enabled, hard stop the deployment 16:38:41 none of us have any time to work that all out and maintain it, clearly 16:38:43 honestly we deploy without selinux for osa, hoping that once distro support goes up 16:38:47 and rely on upstream 16:38:55 it's silly to have our own and upstream selinux rules 16:39:36 mnaser: the reason why we stopped using the upstream packages is because they don't apply for lxc containers - we started with them, but they just didn't do the right things 16:39:45 oh btw on previous bug to talk about a-r-r we already have some mentions in this page, and it's that page that should have links to the references pages. https://docs.openstack.org/openstack-ansible/latest/user/limited-connectivity/index.html#source-code-repositories 16:39:51 IIRC mhayden said they were all docker container focused 16:40:34 i think they'll be fine when we deploy from distro 16:40:38 because the paths are all matching 16:40:40 evrardjp: yeah, I'll take a little time to add a link and be more explicit about how to implement the override 16:40:50 the biggest issue is paths are mismatching afaik 16:41:15 "We've lost the only maintainer of SELinux inside OSA, so please either try to deploy without SELinux (setenforce 0) .. or patches welcome! :)" 16:41:16 as comment 16:41:18 confirmed medium? 16:41:26 mnaser: meh, doesn't change the bug's triage - for now we should turn it all off and if you're using a source-based build it should hard stop the deployment if it's on 16:41:40 i agree 16:41:43 for source based 16:41:44 o/ sorry super late, bouncing between meetings 16:41:57 happy with confirmed medium myself 16:42:35 np cloudnull we jsut assigned all the bugs to you 16:42:36 :> 16:42:44 #link https://bugs.launchpad.net/openstack-ansible/+bug/1782388 16:42:44 Launchpad bug 1782388 in openstack-ansible "Installing Multipath But Not Enabling In Nova Causes Volume Attachment Failures" [Undecided,New] 16:42:44 yay! 16:42:58 mnaser: D 16:43:00 This is my bug report 16:43:10 bgmccollum: told ya :) 16:43:10 If there are questions... 16:43:22 :D 16:43:34 oh man, multipath 16:43:51 There is a very good chance i have no clue what im doing... 16:43:52 enabling multipath by default doesn't sound bad 16:43:55 * mnaser remembers the time of doing infiniband multipath 16:44:01 ohoh 16:44:08 mnaser: you had nice tools 16:44:11 :) 16:44:23 and a few nightmares too I guess 16:44:39 except i was trying to get it running on solaris with writing our own openstack nova driver 16:44:43 haha 16:44:46 anyways 16:45:40 ok so you run cinder-volume and nova-compute on the same host, right? 16:45:49 Yes... 16:46:13 but multipath tools were installed long ago though? 16:46:42 well, i did a Newton -> Ocata -> Pike back-to-back upgrade...then things started going sideways 16:46:54 bgmccollum: https://github.com/openstack/openstack-ansible-os_nova/commit/e9fd86ef0072f8b42997f0ca711451321cc70123 16:46:56 I dont believe it was in Newton, or at least the version of Newton I was running... 16:47:03 it was in N 16:47:12 it was in M even 16:47:20 Hrm...to something else is at play. 16:47:26 *so 16:47:34 I'd say yes. 16:47:58 i'd suspect something related to cinder-volume and nova-compute on same machine 16:48:04 I mean it's not the upgrade that changed the presence of multipath-tools 16:48:12 i got a feel like that might be conflicting 16:48:17 mnaser Yes...starting to think that. 16:48:26 or something in nova changed to 'fix' something, which broke it 16:48:26 bgmccollum: did you upgrade to Q at the end, or pike is the latest? 16:48:41 Let me check version... 16:48:43 or cinder for that matter 16:48:56 because some containers are now on bare metal in Q iirc 16:48:58 16.0.15 16:49:02 ok 16:49:04 so P 16:49:10 yes pike 16:49:11 it's also plausible that we have conflicting settings or something between the two roles 16:49:18 odyssey4me: that's what I'd say 16:49:29 but multipath-tools isn't listed anywhere else 16:49:40 I think nova cnd cinder touch lvm management if that's used - so it could be there 16:49:52 i'm more suspecting that the services are messing around, rather than our roles 16:49:55 nope it's not in cinder role 16:50:09 ill try to stage a test where the nova host has multipath d running, but the cinder host doesn't... 16:50:10 i think what odyssey4me means is that it's not the role but the services 16:50:20 worth a good look, I'd say - but hard to replicate the results given the kit required 16:50:41 could it be something OUTSIDE osa,something pre-configured on the host, or on a side thing? 16:51:12 if someone could look at a test env that bgmccollum gets the same issue showing up in, then perhaps that'd be helpful 16:51:15 not sure...i also later noticed that volume migration is broken with multipathd running 16:51:34 so something is...odd 16:51:35 * odyssey4me is clueless with this stuff though 16:51:51 ill so some more digging, and report back. 16:52:12 so for now 16:52:16 cool thanks bgmccollum 16:52:17 thanks bgmccollum - if you could post some sort of way to replicate the test enironment that might be useful, if that's possible 16:52:19 mnaser: let it open 16:52:23 okay, we'll leave it open. 16:52:29 you can assign someone to track 16:52:32 #link https://bugs.launchpad.net/openstack-ansible/+bug/1782357 16:52:32 Launchpad bug 1782357 in openstack-ansible "Undefined ceilometer variables for additional services" [Undecided,New] 16:52:32 if you prefer 16:52:38 we're running out of time 16:52:38 :X 16:53:01 I think we got this fixed some time recently? 16:53:01 confirmed and med/high 16:53:12 in Q or below? 16:53:18 not sure 16:53:24 in Q 16:53:27 (talking to odyssey4me ) 16:53:30 Dmitriy Rabotjagov (noonedeadpunk) proposed openstack/openstack-ansible-os_masakari master: Basic implemintation of masakari-monitors https://review.openstack.org/584629 16:53:51 cant find any glance_rabbitmq_telemetry_host_group references 16:53:55 looking 16:53:55 so it could still be a thing? 16:55:02 oh, other services https://github.com/openstack/openstack-ansible/commit/2fd5a4b07b6c295e0aa788be99b43a2462c62ddf 16:55:02 i think it still is 16:55:04 :p 16:55:14 so ja, probably still a thing 16:55:19 confirmed/med 16:55:27 #link https://bugs.launchpad.net/openstack-ansible/+bug/1782255 16:55:27 Launchpad bug 1782255 in openstack-ansible "novnc console error code 404, message No such file" [Undecided,New] 16:55:41 odyssey4me: that was written in my comment though :p 16:55:43 anyway 16:55:49 gah 16:55:50 wrapped logs 16:56:00 mnaser: Invalid! 16:56:02 ahaha 16:56:04 due to logs 16:56:16 I can't see anything useful in that log 16:56:17 incomplete.. what url are you trying to access.. 16:56:25 ERROR nova.console.websocketproxy self.send_error(404, "No such file") 16:56:27 i spooted this but 16:56:48 yeah 16:56:50 same 16:56:51 oh dear, centos pike 16:57:02 we need extra user cofnig 16:57:06 to know what's going on 16:57:06 I don't think that's working so good 16:57:13 :) 16:57:19 incomplete added comment asking for more info 16:57:24 sounds good 16:57:25 thanks mnaser 16:57:34 #link https://bugs.launchpad.net/openstack-ansible/+bug/1781057 16:57:37 Launchpad bug 1781057 in openstack-ansible "Support changing management network netmask" [Undecided,New] 16:57:54 confirmed wishlist 16:57:59 confirmed and ... 16:58:02 i might be doing a renumber of an osa deployed (lol end me.) 16:58:03 I'd say it's a bug 16:58:07 I thought andymccr had implemented something along these lines? 16:58:15 odyssey4me: it's not complete 16:58:21 that's another case 16:58:25 oh ok, fair enough 16:58:31 is it a bug? do we actaully claim that changing a subnet => takes care of everything? 16:58:34 the network stays the same, but network changes. 16:58:37 cidr* 16:58:57 mnaser: well the dynamic inventory is wrong, not that it changes everything. 16:59:22 _who is dreaming of static inventories now?_ 16:59:27 ok so what do you propose then? 16:59:37 perhaps best to add a known issue release note saying 'if you're changing your network config after a deploy, you're in for a bad day - don't do it' 16:59:39 confirmed and low 16:59:56 James Denton proposed openstack/openstack-ansible-os_neutron master: Adds networking-ovn support https://review.openstack.org/584067 17:00:02 anything fundamental in the infrastructure (storage, networking) should not be changed after the deploy is done 17:00:10 confirm/low and added odyssey4me's comment 17:00:14 odyssey4me: that could be documentd ^ 17:00:18 mnaser: perfect. 17:00:27 (also, lol, i am totally going to do an OSA renumber.. just... watch.. 17:00:28 out of time - 17:00:31 next meeting chair? 17:00:47 we have 1 bug but i'm pretty sure that's wip 17:00:55 yeah 17:00:56 i'll run next one unless someone really wants to 17:00:56 :) 17:01:27 no responses 17:01:28 ETIMEOUT 17:01:30 #endmeeting