19:02:22 <lifeless> #startmeeting tripleo
19:02:23 <openstack> Meeting started Tue May  6 19:02:22 2014 UTC and is due to finish in 60 minutes.  The chair is lifeless. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:02:25 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:02:26 <jdob> o/
19:02:28 <openstack> The meeting name has been set to 'tripleo'
19:02:48 <dshulyak_> o/
19:03:19 <derekh_> hi
19:04:11 <GheRivero> o/
19:04:16 <chuckC> hi
19:04:21 <bnemec> o/
19:04:45 <lifeless> #topic agenda
19:04:46 <lsmola> hola
19:04:49 <lifeless> bugs
19:04:49 <lifeless> reviews
19:04:49 <lifeless> Projects needing releases
19:04:49 <lifeless> CD Cloud status
19:04:49 <lifeless> CI
19:04:51 <lifeless> Insert one-off agenda items here
19:04:53 <lifeless> open discussion
19:04:55 <tchaypo> \o
19:04:56 <lifeless> #topic bugs
19:05:01 <lifeless> #link https://bugs.launchpad.net/tripleo/
19:05:01 <lifeless> #link https://bugs.launchpad.net/diskimage-builder/
19:05:01 <lifeless> #link https://bugs.launchpad.net/os-refresh-config
19:05:01 <lifeless> #link https://bugs.launchpad.net/os-apply-config
19:05:01 <lifeless> #link https://bugs.launchpad.net/os-collect-config
19:05:03 <lifeless> #link https://bugs.launchpad.net/tuskar
19:05:06 <lifeless> #link https://bugs.launchpad.net/python-tuskarclient
19:05:21 <SpamapS> o/
19:06:49 <shadower> o/
19:07:26 <SpamapS> lifeless: that new trusty race is nasty
19:07:32 <tchaypo> Progress on #1290486 - mestery was able to reproduce it in a non-tripleo context iff he used vlan flows; it sounds like it's been tracked down to a bit of code that recreates some types of flow but not others
19:07:42 <lifeless> SpamapS: indeed
19:08:15 <derekh_> Patch up for https://bugs.launchpad.net/tripleo/+bug/1272803 https://review.openstack.org/#/c/91333/
19:09:01 <SpamapS> Some users have been trying to build on Debian unstable.. and failing because RabbitMQ 3.3 rejects the 'guest' user from non local hosts.
19:09:17 <SpamapS> that is bug 1315474
19:09:26 <lifeless> patch for https://bugs.launchpad.net/diskimage-builder/+bug/1314021 failed CI, I haven't looked why yet
19:09:47 <lifeless> loverly
19:09:54 <SpamapS> anyway, I feel like the bug situation is "meh" right now.. we have some.. we're not falling too far behind.. we're keeping up with triage..
19:10:05 <SpamapS> most of our bugs are actually out in the other projects really
19:11:26 <lifeless> derekh_: tiny nit in https://review.openstack.org/#/c/91333/
19:11:48 <lifeless> slagle: any news on  https://bugs.launchpad.net/tripleo/+bug/1287453 ?
19:12:02 <derekh_> lifeless: thanks will fix
19:12:17 <lifeless> derekh_: any news on https://bugs.launchpad.net/tripleo/+bug/1308407 ?
19:13:20 <SpamapS> lifeless: oh, are we tracking bugs in os-cloud-config at tripleo or do we need to add another bug link?
19:13:30 <derekh_> lifeless: fungi took a look on thrusday and mentioned prefering a deterministic approach over the random shuffly I've proposed , I'm not sure how possible this will be but have tried it yet
19:13:51 <SpamapS> I feel like it should have its own.
19:13:53 <lifeless> SpamapS: I don't think we've made the lp projest for it
19:13:55 <lifeless> SpamapS: we should
19:14:07 <lifeless> SpamapS: volunteering?
19:14:10 <SpamapS> why not
19:15:47 <lifeless> anyone know dougal matthews IRC? https://bugs.launchpad.net/tuskar/+bug/1308172
19:16:09 <bnemec> lifeless: d0ugal I think
19:16:12 <shadower> lifeless: d0ugal
19:16:29 <lifeless> d0ugal: ^
19:16:32 <Ng> .o/
19:16:50 <lifeless> Ng: o/ - wondering if you or GheRivero found time to look at the trusty race?
19:18:28 <SpamapS> https://bugs.launchpad.net/os-cloud-config no bugs, \o/ ;)
19:18:37 <Ng> lifeless: I got completely sidetracked by calls and planning of things, sorry :(
19:19:00 <Ng> lifeless: I believe Ghe was going to look at it, but I don't know if he got to it yet
19:19:03 <lifeless> SpamapS: can you link it in the wiki pages too? meeting + tripleo overview; and I guess we need a patch for incubator to describe that we use it :)
19:19:14 <lifeless> ok, moving on with the meeting
19:19:23 <dprince> SpamapS: BTW. just found this w/ heat-api. Would like to do a revert to fix it: https://review.openstack.org/#/c/92439/
19:19:56 * dprince is apparently the only one using squid lately
19:20:32 <lifeless> dprince: oh wow, classy
19:20:55 <lifeless> dprince: yeah, we no-proxy everything in devtest because of bad ip ranges vs corp proxies
19:22:09 <lifeless> #topic reviews
19:22:51 <lifeless> reviewstats still broken for tripleo at least
19:23:23 <bnemec> lifeless: Should be fixed with https://review.openstack.org/#/c/91173/
19:23:33 <bnemec> One second, I can paste the current numbers.
19:23:47 <jdob> bnemec: is that going to take into account the specs repo too?
19:23:49 <SpamapS> dprince: instead of revert, maybe just change to X-Forwarded-Host ?
19:24:20 <bnemec> lifeless: http://paste.openstack.org/show/79231/
19:24:25 <SpamapS> lifeless: http://www.stackalytics.com/report/contribution/tripleo-group/30 still is useful.
19:24:30 <bnemec> jdob: Probably not yet.
19:24:33 <SpamapS> http://www.stackalytics.com/report/reviews/tripleo-group/open too
19:24:38 <bnemec> It needs to be added to the project list.
19:25:03 <dprince> SpamapS: But I feel like this is just wrong. The Blueprint itself is actually all wrong!
19:25:24 <lifeless> SpamapS: no, there are other header folk use
19:25:35 <dprince> SpamapS: I thought about doing that. Simple enough... but this one just seemed to be off beat all around.
19:25:51 <SpamapS> mmkay
19:26:00 <SpamapS> IIRC that patch enabled using stunnel.. but maybe not
19:26:09 <SpamapS> since it is like..dead wrong ;)
19:26:23 <lifeless> huh, no, no impact on stunnel
19:26:41 <bnemec> openreviews: http://paste.openstack.org/show/79232/
19:27:19 <SpamapS> anyway, let's just leave that for #heat
19:28:59 <lifeless> Median wait time: 5 days, 1 hours, 42 minutes
19:29:00 <lifeless> 3rd quartile wait time: 11 days, 2 hours, 26 minutes
19:29:11 <lifeless> so we're still bad but not getting dramatically worse
19:29:54 <tchaypo> I notice that a lot of the really old ones are still the "make X configurable" ones. Have we -2d all the ones that could be switched to the passthrough model?
19:29:59 <lifeless> looks like a bunch of our core reviewers have dropped down a little in volume
19:30:27 <lifeless> I thought we had
19:30:32 <tchaypo> If so, it sounds like the ones that remain deserve some attention - some are 60 days old
19:30:34 <SpamapS> yeah I've been just barely > 3 per day
19:31:43 <lifeless> tchaypo: clearly :)
19:31:59 <lifeless> tchaypo: good effective non-core reviews to get things into shape can be a great help here
19:32:24 <bnemec> Yeah, I think all those glance "Make X configurable" patches need -2's.
19:32:31 <bnemec> Was on my todo list. :-)
19:32:33 * tchaypo smells some kind of hint
19:32:35 <lifeless> ok another review topic is
19:32:37 <lifeless> specs repo
19:32:39 <SpamapS> I wonder if, as cores, we should focus on those with +1's already. I tend to alternate, picking one with no votes, and then one with +1's
19:32:41 <lifeless> I mailed the list
19:32:58 <slagle> i tend to start with oldest first
19:33:01 <lifeless> but - to reinforce // provoke thought
19:33:18 <bnemec> slagle: +1
19:33:30 <lifeless> My proposal is that -specs starts with PTL only +A
19:33:43 <Ng> +1
19:33:44 <SpamapS> I'm fine w/ that
19:33:45 <lifeless> so cores review it to +2; final ack from me.
19:33:46 <greghaynes> +1
19:33:58 <lifeless> erm 2x+2 I mean
19:34:04 <jdob> i'm fine with that too, as long as you're not worried about getting buried by it
19:34:09 <lifeless> I'm toying with consensus
19:34:15 <jistr> jdob: +1
19:34:16 <bnemec> +1.  I think that's the way blueprints work in some projects anyway (only the PTL can approve).
19:34:27 <lifeless> jdob: I'm buried by blueprints atm, this is much more structured and easy to eyeball
19:34:37 <lifeless> so I hope it will be much better
19:34:42 <jdob> excellent
19:34:55 <Ng> lifeless: I'd say let's see how it goes. If you find you're just rubber stamping what the cores have already consensed, then you can devolve the +A to core folks
19:35:02 <lifeless> right
19:35:05 <slagle> sounds good to me if only you can +A, but i'd like to see more input than just 2 other +2's
19:35:07 <derekh_> sounds good to me
19:35:21 <slagle> but i guess that's on folks to make sure they're reviewing
19:35:23 <lifeless> slagle: ideally I'd like every core to be on-board with every spec
19:35:39 <lifeless> slagle: but I suspect thats not scalable
19:35:57 <slagle> yea
19:36:04 <lifeless> slagle: I think we do need every core to  follow every specs commit
19:36:11 <lifeless> so they know whats happening in the reiview pipeline
19:36:28 <greghaynes> can that be auto'd somehow?
19:36:46 <SpamapS> I hope not
19:36:47 <lifeless> greghaynes: subscribe to the branch in gerrit ?
19:36:48 <greghaynes> hehe
19:36:59 <jdob> my hesitation against saying every core is that it puts a potential bottleneck
19:36:59 <lifeless> greghaynes: and feed it into your machine-brain interface?
19:37:02 <SpamapS> auto subscribe, sure, but don't enforce that rule ...
19:37:16 <jdob> a good majority makes sense
19:37:16 <SpamapS> one person goes on vacation and we can't approve specs.. bad state
19:37:25 <jdob> SpamapS: exactly what i was just gonna say, vacation
19:37:25 <lifeless> right
19:37:33 <lifeless> I think we'll iterate
19:37:49 <marios> lifeless: btw, neutron has a rule where if it isn't a bugfix, every review must have an associated blueprint in specs repo. are also doing this?
19:37:55 <SpamapS> so just make it a reasonably wide swath of cores and perhaps a mininum amount of time before +A to allow wide review
19:38:01 <jdob> are specs going to be a separate section in this meeting?
19:38:14 <lifeless> the key points are: a) we expect cores to help review specs. b) for now, only PTL +A specs.
19:38:27 <lifeless> jdob: perhaps?!
19:38:47 <jdob> i'd say yes, after reviews treat the specs explicitly and make sure we're on top of them
19:38:57 <lifeless> marios: we haven't so far, and I'm worried about the impact on e.g. refactoring work
19:39:29 <lifeless> marios: also there are IMO many cases where bugfixes need specs
19:39:54 <marios> lifeless: yeah, my question was more rhetorical and along the lines of 'make sure we don't just slow everything down even more with this process and any rules we come up with'... like the all cores discussed above
19:39:57 <lifeless> marios: its the nature of the change, not whether or not it's unbreaking something we thought works, that drives the need for design review
19:40:10 <lifeless> marios: +1
19:40:19 * SpamapS is being pulled AFK by meatspace ... bbiab
19:40:28 <lifeless> #action lifeless to summarise specs stuff and add section to meeting
19:40:39 <lifeless> #topic
19:40:39 <lifeless> Projects needing releases
19:40:44 <lifeless> #topic Projects needing releases
19:42:17 <lifeless> ok so no volunteers ?
19:42:27 <lifeless> [see what I did there :P]
19:43:22 <lifeless> slagle: perhaps you could do a release run
19:43:23 <lifeless> ?
19:43:29 <tchaypo> I think I saw a few weeks ago that the release process is at least fairly documented, right?
19:43:32 <lifeless> slagle: I'm expecting to be totally swamped this week
19:43:33 <slagle> perhaps i could!
19:43:42 <slagle> i can do it
19:43:48 <lifeless> tchaypo: it is indeed
19:44:01 <lifeless> slagle: thankyou, muchly appreciated
19:44:18 <lifeless> #topi
19:44:24 <lifeless> #topic CD Cloud status
19:44:30 <tchaypo> I'll see if I can find the notes and have a dummy run through them
19:45:01 <lifeless> I believe both regions are behaving themselves atm
19:45:05 <jdob> tchaypo: https://wiki.openstack.org/wiki/TripleO/ReleaseManagement
19:45:08 <lifeless> derekh_: ^ ?
19:45:38 <jdob> tchaypo: i wanted to do the same thing, just haven't had a chance to yet
19:45:38 <derekh_> lifeless: yup, no major outages at the moment
19:46:09 <tchaypo> jdob: You robbed me of the fun of digging for the URL. Thanks :)
19:46:13 <lifeless> #topic CI
19:46:19 <jdob> tchaypo: hehe
19:46:29 <derekh_> lifeless: I did try to bring up a baremetal node to test differences on R1 speeds but it failed, have to get back to it
19:46:30 <lifeless> derekh_: ^
19:46:48 <lifeless> bwah, wifi jitter here is terrible
19:46:49 <derekh_> so I'm thinking we could do 3 things to help with capacity
19:46:53 <lifeless> 2s local pings
19:47:06 <derekh_> as we get a long backlog pretty much every day
19:47:07 <tchaypo> Where is here?
19:47:11 <lifeless> tchaypo: home
19:47:21 <derekh_> 1. Get rid of the seed job, until such time as its not a subset of the other 2 jobs
19:47:33 <tchaypo> Land of the long white ping?
19:47:34 <derekh_> 2. Move the ironic-undercloud to tripleo-expermental, then when it works swap it for the iron-seed job (assuming its also a subset)
19:47:43 <derekh_> 3. get us back to 8G nodepool instances https://review.openstack.org/#/c/91545/1
19:48:52 <lifeless> derekh_: 3 looks safest to me
19:49:22 <lifeless> derekh_: we can perhaps increase the HP region size a bit now more hardware is online
19:49:40 <lifeless> derekh_: note that we need to make the testenvs substantially bigger soon for HA
19:49:51 <derekh_> lifeless: I was kind of hoping we could do all 3 :-)
19:50:28 <lifeless> derekh_: I think this needs a little list discussion for 1 and 2
19:50:32 <derekh_> lifeless: ok (on testenv increase, how much bigger ?
19:50:44 <derekh_> lifeless: ok, want me to mail list ?
19:50:57 <lifeless> derekh_: well we need 3 nodes for ctl plane, and we'll need nodes for swift and cinder too very soon
19:51:23 <lifeless> derekh_: I'm thinking we perhaps need to oversubscribe the testenv hosts a little (at least on CPU)
19:51:24 <derekh_> lifeless: while we're on the testenv size, should we bring it down to 2G nodes ? as this is what devtest uses ?
19:51:35 <lifeless> derekh_: yeah
19:51:54 <lifeless> derekh_: mailing list - yes please
19:52:33 <derekh_> lifeless: ok on oversubscribing, will at patch to downlize testenvs and will mail list on some of the jobs
19:52:33 <lifeless> I'd like us to find a way to do scale tests as a periodic thing soon
19:53:01 <lifeless> I'm talking to mgmt in HP about getting some more resources to enable that
19:53:16 <derekh_> lifeless: ok, sounds good
19:53:23 <lifeless> e.g. we should run a 200 node job nightly
19:54:05 <lifeless> I'm starting to think we need to prioritise the revamp of testenv stuff to be natively hosted on nova
19:54:18 <lifeless> to be able to more flexibly use resources
19:54:41 <lifeless> I think we need a spec for TripleO that calls out the changes needed to nova/neutron/etc, which we can then seek cross-project buyin on.
19:54:51 <tchaypo> we're getting close to time - is there anything else that anyone wants to raise before we go?
19:54:56 <lifeless> any volunteers to capture such stuff ?
19:55:28 <lifeless> tchaypo: sill in the CI section for a little more :)
19:55:40 <lifeless> *still*
19:55:42 <derekh_> lifeless: if you want to point me the right direction I could have a stab at it
19:55:45 <bnemec> lifeless: Basically the TripleO on OpenStack work?
19:56:20 * derekh_ currently doesn't know what the shortcommings are but could take a educated guees
19:56:23 <lifeless> bnemec: yeah, I am sure we will touch on it at the summit, but basically I'm thinking a spec in the specs repo that identifies all the issues that testenvs let us work around
19:56:28 <lifeless> derekh_: ^
19:56:43 <lifeless> iif someone gets a draft up we can brainstorm issues on it
19:56:46 <tchaypo> lifeless: sorry, didn't mean to hurry you; I was thinking of quick "please read and comment on X" things
19:56:54 <bnemec> lifeless: Yeah, I've been poking at that this week in prep for the summit discussion.
19:57:34 <lifeless> bnemec: cool; when you have a few minutes, capturing the tripleo-on-openstack aspects into a spec would be mega useful I think
19:57:40 <bnemec> lifeless: Hopefully I can get my environment sorted out and start figuring out what does and doesn't work right now.
19:57:47 <bnemec> lifeless: Yep, will do.
19:57:59 <lifeless> #topic open discussion
19:59:23 <lifeless> 45 seconds....
19:59:41 <jdob> have fun at summit everyone :)
19:59:46 <lifeless> wooo summit
19:59:51 <lsmola> yaaay
19:59:51 <lifeless> no meeting next week
20:00:00 <lifeless> cause its ALL MEETING ALL WEEK
20:00:16 <lsmola> in person meeting next week
20:00:16 <lifeless> #endmeeting