16:00:00 #startmeeting OpenStack-Ansible 16:00:01 Meeting started Thu Jul 21 16:00:00 2016 UTC and is due to finish in 60 minutes. The chair is mhayden. Information about MeetBot at http://wiki.debian.org/MeetBot. 16:00:02 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 16:00:04 The meeting name has been set to 'openstack_ansible' 16:00:12 #topic Roll call 16:00:16 o/ 16:00:35 I may need to leave early, so please excuse me when I do. 16:00:46 o/ 16:00:50 o/ 16:00:52 o/ 16:00:57 o/ 16:01:32 o/ 16:02:22 o/ 16:02:33 o/ 16:03:16 o/ 16:03:23 o/ 16:03:36 o/ 16:03:44 woot, i think we have a good group here 16:03:54 #topic Action items 16:04:12 the only action item was for odyssey4me to mail the list about the mascot, and that appears to be going well 16:04:33 since mascots are on the topics list, i guess we could take a few minutes now to talk that over 16:04:46 #topic OSA Mascot 16:05:00 i was hoping to use d34dh0r53 as the mascot, but alas, that breaks the rules ;) 16:05:08 we need to agree on the final list 16:05:10 hi all 16:05:25 am i right that "yak" in modern english related to it has some negative connatation ? 16:05:41 yeah, yak shaving is not a positive thing 16:05:44 s\related to it\related to IT\g 16:05:47 original email thread: 16:05:49 #link https://openstack.nimeyo.com/90412/openstack-dev-openstack-ansible-project-mascot?qa_q=openstack-ansible+mascot&show=90412#q90412 16:05:51 : - ) ok 16:05:51 eil397_: Can be used as a synonym for vomiting 16:06:29 im partial to the "Osa Eucharitid" :) 16:06:54 but not against the Cape Buffalo 16:07:10 Palm tree. It's the beach, you're taking it easy. OSA gives you more time to visit the beach, because you're not deploying openstack. 16:07:14 that Cape Buffalo looks good, means business 16:07:34 heh, happy with a palm tree too :) 16:08:32 o? ate 16:08:38 o/ late even:) 16:08:48 how many do we need for the list? 16:08:50 do we have an etherpad started for this? 16:09:12 andymccr 3-5 16:09:20 Osa looks cool - http://www.petsfoto.com/wp-content/uploads/2011/06/Insect-Life1.jpg 16:09:23 watcher submitted top 4, to account for duplicates and ones that might violate the rules. 16:09:27 no need for an etherpad, this meeting log is record which I'll use 16:09:35 but you can etherpad if you like 16:09:39 I gotta run - bbl 16:09:40 mascot http://www.abbeyroaddals.com/images/hippiecatch.jpg !!! 16:09:51 later odyssey4me 16:09:56 i like the osa bug 16:09:56 most projects are picking animals I think, so I'm partial to something that's not an animal. 16:10:27 jmccrory: that osa bug looks like the insect version of the cape buffalo 16:10:30 I most definitely do not have strong opinions on the matter 16:10:43 mhayden: right : - ) 16:10:46 ^ i also like that suggestion. 16:10:48 perhaps we could use a peacock, automagically? :) 16:11:05 michaelgugino: re: not doing what everyone else is 16:11:06 mhayden: Go for it, a bird, a bug, a buffalo….whatevs 16:11:49 im with automagically, no strong opinions really. 16:12:07 we can have one animal in list , one isect, one tree and... 16:12:13 can someone toss that stuff into an etherpad so we can vote and move things around? 16:12:26 s\isect\insect\g 16:13:10 all the suggestions that are being put forward seem appropriate - and we need a list of 3-5 so may as well submit them unless there are major issues with any of those. 16:14:07 odyssey4me: got what you need here? :) 16:14:29 How about a camel since TMTOWTDI is possible with OSA? 16:15:52 As a last oddball suggestion, I'll throw out the Chanterelle as a mascot 16:16:04 heeheh 16:16:18 stevelle: thats a good one 16:16:43 +1 16:16:50 https://etherpad.openstack.org/p/osa-mascot 16:17:25 mhayden: You left off the dog!:) 16:18:18 lol 16:18:22 oops 16:18:28 okay, can we move to the next topic for now? 16:18:41 yes pleasew 16:18:44 haha okay 16:18:53 #topic wsgi vs uwsgi and Apache vs nginx role consistency 16:19:03 stevelle / cloudnull / odyssey4me: take it away! 16:19:20 Sure 16:20:04 back 16:20:09 We had a bit of discussion in the irc channel this week about this blueprint https://blueprints.launchpad.net/openstack-ansible/+spec/keystone-uwsgi 16:20:59 It was inspired by a change automagically proposed to fix Aodh. Upstream changed the way the API service is started. 16:21:09 #link https://review.openstack.org/#/c/344555/ 16:21:25 #link https://review.openstack.org/#/c/343918/ 16:21:40 To fix Aodh automagically proposed to run aodh-api under mod_wsgi with Apache, which conforms to how we run other wsgi services in OpenStack 16:21:58 All of those are run that way because this used to be the recommended means of running Keystone 16:22:06 particularly important with Federation 16:22:23 The Keystone community has changed their recommendation as noted in the blurprint 16:23:05 stevelle: do we know if we lose our federation capabilities by moving away from what we curerntly have? 16:23:28 We seemed to agree in IRC at the time that we would like the WSGI apps to run similarly, so if we change Keystone away from Apache / mod_wsgi, we want to look at changing the others (Horizon, Gnocchi, Aodh in this case) 16:23:40 cloudnull: I don't know, haven't investigated that yet. 16:23:44 ok 16:24:05 Until we know if federation works, I think we should table moving all the roles to uwsgi 16:24:13 I think it's worth investigating, but I don't know if it's appropriate for this cycle. 16:24:13 +1 16:24:14 i.e. I’d rather be consistent for the time being 16:24:47 I did a proof of concept for running aodh-api under uwsgi just so we could look at it. 16:24:52 #link https://blueprints.launchpad.net/openstack-ansible/+spec/keystone-uwsgi 16:24:59 #link https://review.openstack.org/#/c/344555/ 16:26:27 The PoC failed in the integrated gate due to the general merge issues. 16:26:36 id love to see all openstack api services using wsgi/uwsgi in a uniform way and not just these few - that's not really how it is right now though. 16:27:28 +1 but for Aodh and Gnocchi particularly they won't run under Werkzeug 16:28:05 Horizon also 16:28:27 sounds good to standardize esp if keystone have changed their recommendation 16:28:57 yes, uwsgi outperforms gevent/gunicorn/whatever they're using handily. 16:29:11 so it sounds like we've agreed that standardizing on uwsgi across the board makes sense? 16:29:17 andymccr: I agree, but I think the question is how long to do we think it takes to get standardized across horizon, gnocchi, aodh, keystone 16:29:46 as a long term goal, but I don't think it's appropriate for this cycle. We're a little far in, and I think we need to give greater attention to other areas, specifically Xenial and CentOS 16:29:50 mhayden where possible, yes - but the caution is that if it's not in by N3 or we don't think we can do it by N3 then it must carry over to the next cycle 16:29:51 based on the proposed change, it doesn't seem like it will be much of a challenge to move over the four. 16:30:03 odyssey4me: sounds good 16:30:21 assuming that federation support can be maintained in keystone 16:30:31 #agreed Should begin using uwsgi wherever possible -- if not available by Newton-3, it needs to happen in O 16:30:32 Do we have any gate checks around federation that can ensure we don’t break it? 16:30:40 automagically nope 16:31:10 More short term, how do we want to unblock the aodh role gate? 16:32:03 With the uncertainty around federation, I'm inclined to say we should evaluate https://review.openstack.org/#/c/343918/ 16:32:15 And Newton-3 is 8/1 correct? 16:33:24 Sorry, its 9/1 16:33:43 stevelle: +1 16:34:08 sounds like a good idea then we can evaluate a more long term approach later. 16:34:14 the aodh gate needs to be unblocked. 16:34:20 stevelle: +1 and michaelgugino +1 on needing to focus on Xenial 16:34:39 So it sounds like we are decided 16:35:14 I'm probably not going to have time to do the development for all of the above including resolving federation by Aug 1, so seems clear to me 16:35:18 #agreed Unblock the aodh gate first, then worry about a long term approach later 16:35:28 +1 16:35:37 seems dangerous to expect it to be correct and merged by Sept 1 16:36:10 next topic? 16:36:11 So, michaelgugino brought up Xenial, I’m having a hard time seeing the big picture on what work remains there 16:36:15 stevelle: woot 16:36:24 #topic Mid-cycle planning 16:36:50 Can we #topic Xenial to-do list? 16:37:07 automagically: sure, i can do that next 16:37:11 Awesome! 16:37:17 the meeting room is still good to go for the mid-cycle 16:37:25 the agenda is in the etherpad 16:37:33 is there an event bright or anything for the midcycle? 16:37:35 #link https://etherpad.openstack.org/p/osa-midcycle-newton 16:37:56 michaelgugino: not at the moment 16:37:59 michaelgugino nope, just the etherpad 16:38:07 mhayden, Could you add the meeting room into the etherpad? 16:38:09 however, i need to make sure i have an accurate list so i can send the names down to our physical security folks 16:38:14 palendae: sure 16:38:22 from what i can tell we need to do rally, babican, and zaqar to make xenial fully supporte.d 16:38:26 michaelgugino: I'm going to make some eventbrites for dinners and put the links on the etherpad 16:38:40 i've been deploying the integrated gate w/ xenial for a while now and it seems to work well. 16:38:46 palendae: added 16:38:47 I added my name as tentative, I should know by next week if I'm going, I don't think I'll have a problem getting the travel approved, but we'll see :) 16:38:54 Speaking of our physical security: pokemon go is banned in the Castle, for those non-Rackers coming 16:39:11 omg 16:39:20 haha 16:39:23 hahaha 16:39:25 palendae: Well, than I better cancel my trip 16:39:29 Don't get kicked out for something silly :) 16:39:32 okay, well please everyone make sure your name is in there 16:39:59 #topic Xenial to-do list 16:40:05 michaelgugino: once the intgrated gate is unblocked it'd be great to get more testers on xenial things 16:40:31 yeah, I think we need more manual testing, lots of code paths not checked by default (like OVS, DVR) etc 16:40:38 cloudnull: i can try out xenial on some of my lab boxes 16:40:45 -- from what i can tell we need to do rally, babican, and zaqar to make xenial fully "supported". 16:41:10 michaelgugino OVS testing is on by default for the os_neutron role right now 16:41:14 Since those are non-integrated roles, do we let them slide if they are not done? 16:41:15 -- i've been deploying the integrated gate w/ xenial for a while now and it seems to work well. but to michaelgugino point there are some manual code paths that need to be exercised. 16:41:24 brb in a sec 16:41:29 And barbican looks to be done: https://review.openstack.org/#/c/341761/ 16:41:49 awesome. 16:42:00 that's good about OVS. But yeah, just saying generally, I'm not up to date on all the different configurations. Swift comes to mind, I know we're not checking all of the backing stores 16:42:02 * cloudnull moves card 16:42:06 yeah, primary focus is on the integrated roles - secondary focus are the role we want to integrate this cycle 16:42:51 michaelgugino the roles have the ability to do multiple scenario tests now, just like the neutron role 16:42:57 cloudkitty is another one i'd like to see if we could optinally enable, similar to ironic. 16:43:04 So should we have an action item to make sure that https://etherpad.openstack.org/p/openstack-ansible-newton-ubuntu16-04 is up to date 16:43:08 all it takes is a change to tox config, then an addition of the jobs in infra 16:43:30 so if you want to test more paths, please go ahead and add the tox config to make it happen 16:43:44 you mean add another tox environment? 16:43:50 we have experimental checks for xenial/centos in the integrated repo 16:44:16 due to the master branch brokenness I haven't manage to figure out whether it's working very well - because it works slightly differently 16:44:48 ie it uses tox 16:46:07 #action automagically update xenial etherpad status 16:46:28 #action mhayden test Xenial in his lab 16:46:33 automagically: that'd be most excellent 16:46:34 indeedy 16:46:40 the lab needs a reimage anyway 16:46:58 michaelgugino yes, either adapt the existing test to make it test more... or add a new tox env for a different test entirely 16:47:13 Sounds like we have an action around figuring out the integrated gate issues for xenial 16:47:29 michaelgugino I would suggest trying to pipeline tests as far as possible, and only using multiple tox environments where the environments are structurally different 16:48:21 odyssey4me: is there an example, you mentioned OVS in neutron? 16:48:32 is there a plan to get a OSA integrated gate running on xenial? 16:48:38 michaelgugino https://github.com/openstack/openstack-ansible-os_neutron/commit/eb0bb51fafa01926a0d13021a1f97c333bbdabd8 16:48:49 mhayden there is already an experimental check 16:48:53 (about 11 minutes left in the meeting -- just a warning) 16:49:21 I'm not convinced yet that it's working because since implementing it the integrated gate has not been passing frequently. 16:49:38 that looks a little complicated, so I'll have to take some time to look it over. 16:49:40 so I'm waiting for that storm to blow over. 16:50:27 michaelgugino: Would definitely appreciate your review of that 16:50:49 I can see some tox config optimisations which can be done now that it's in. 16:50:53 I'll drop a patch shortly. 16:51:35 ok, I've been a little busy recently, but I'm planning on getting back into the full swing here at OSA, more reviews, bugs, etc. Just been a really busy month or so. :) 16:53:39 #topic Open floor 16:53:44 anyone have anything else to discuss? 16:53:48 only a few minutes left 16:54:03 so i've noticed something with regard to gate testing like crazy. IDK if this is just me, but when I have callback plugins enabled the run is notably slower. 16:54:51 Example http://cdn.pasteraw.com/gybn3gfnnum767fvnzbed2oyax66rxs 16:54:59 thats just rerunning the keystone play on an already deployed cluster. 16:55:34 I know we "use" them in the gate, but it wouls seem disabling them would be to our benifit in terms of testing speed. 16:55:35 I promised a nova-lxd patch last week. I have it cut locally, but installing from pip doesn't work because they haven't published a newton package in pypi yet, so I need to install from source. 16:55:37 ouch - time for some profiling 16:56:14 but, it's coming, I am working on it. Next week is looking good (on travel this week) 16:56:41 cloudnull I think we should kill the profiling callback 16:56:53 it's actually annoying that it only reports the top 10 tasks 16:57:15 the human readable one as well. 16:57:17 Thanks to all of you who have been reviewing the long list of patchs for https://review.openstack.org/#/q/status:open+topic:bp/multi-rabbitmq-clusters 16:57:20 Really appreciate it 16:57:21 i mean we use it . 16:57:23 i've also seen fact_caching cause a lot of problems in local deployments, lots of random slow downs. i'm wondering if that's causing some of the randomness of simple tasks to take 60+ seconds 16:57:29 why don't we remove both callbacks and see how the gate looks after? 16:57:33 but it causes a delay in processing 16:58:10 maybe for the gate we should disable fact caching? 16:58:22 okay, let's take the caching/callback discussion to #openstack-ansible and close this thing up 16:58:22 thanks everyone! 16:58:24 jmccrory: I too have been looking into that. I think a lot of those random slow downs are due to the ansible "smart" transport 16:58:28 #endmeeting