19:01:05 <lifeless> #startmeeting tripleo
19:01:06 <jistr> hi
19:01:06 <openstack> Meeting started Tue Oct 22 19:01:05 2013 UTC and is due to finish in 60 minutes.  The chair is lifeless. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:01:07 <viktors_> hi
19:01:07 <marios_> hi
19:01:07 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:01:08 <ifarkas> hi
19:01:10 <openstack> The meeting name has been set to 'tripleo'
19:01:11 <noslzzp> hi
19:01:13 <jcoufal> o/
19:01:23 <slagle> hi
19:01:24 <tzumainn> hiya
19:01:27 <shadower> hola
19:01:51 <derekh> o/
19:01:59 <dkehn> hi
19:02:51 <lifeless> #agenda
19:02:56 <lifeless> bugs
19:02:56 <lifeless> reviews
19:02:56 <lifeless> Projects needing releases
19:02:56 <lifeless> CD Cloud status
19:02:56 <lifeless> CI virtualized testing progress
19:02:59 <lifeless> Insert one-off agenda items here
19:03:01 <lifeless> review kanban
19:03:03 <lifeless> review the tweaked reviewer rules
19:03:06 <lifeless> open discussion
19:03:08 <lifeless> #topic bugs
19:03:11 <lifeless> #link https://bugs.launchpad.net/tripleo/
19:03:13 <lifeless> #link https://bugs.launchpad.net/diskimage-builder/
19:03:16 <lifeless> #link https://bugs.launchpad.net/os-refresh-config
19:03:18 <lifeless> #link https://bugs.launchpad.net/os-apply-config
19:03:21 <lifeless> #link https://bugs.launchpad.net/os-collect-config
19:03:23 <lifeless> #link https://bugs.launchpad.net/tuskar
19:03:26 <lifeless> #link https://bugs.launchpad.net/tuskar-ui
19:03:28 <lifeless> #link https://bugs.launchpad.net/python-tuskarclient
19:03:31 <lifeless> also good morning everyone
19:03:34 <jtomasek> hi
19:04:02 <lifeless> we have bug 1241042 which is a firedrill
19:04:09 <lifeless> devtest is broken
19:04:27 <derekh> So https://bugs.launchpad.net/tripleo/+bug/1241042  , after I reported it and tried a few things my time got sucked away on something else
19:04:35 <lifeless> and we have multiple untriaged bugs
19:04:43 <dkehn> ithout a REALLY good reason
19:05:29 <lifeless> e.g. https://bugs.launchpad.net/os-apply-config/+bug/1243263
19:05:48 * SpamapS closes one old untriaged bug
19:06:04 <rpodolyaka1> lifeless: derekh's patch is very close to fix bug 1241042, I tested it today, there is one small issue with nova config template
19:06:13 <SpamapS> Oh, I just realized I'm not subscribed to bugmail for oac
19:06:16 <lifeless> rpodolyaka1: sorry to pick on that bug
19:06:22 <lifeless> rpodolyaka1: but you filed it without triaging it
19:06:31 <marios_> lifeless: i think i hit https://bugs.launchpad.net/tripleo/+bug/1241042 be the reason heat stack create for overcloud falied (even though nova instances came up) on 2 separate boxes today
19:06:32 <lifeless> rpodolyaka1: importance 'undecided'
19:06:37 <rpodolyaka1> lifeless: for some reason being in tripleo team on launchpad I can't  triage in os-apply-config :(
19:06:49 <lifeless> rpodolyaka1: ok, lets fix that!
19:07:04 <lifeless> everyone: if you find you can't do something you should be able to do, raise it!
19:07:36 <lifeless> oac had a per-project team for no good reason, switching it to tripleo
19:07:39 <lifeless> done
19:08:04 <SpamapS> tuskar should be changed to tripleo too
19:08:05 <lifeless> https://bugs.launchpad.net/tripleo/+bug/1240753 also wasn't triaged though
19:08:08 * rpodolyaka1 triages
19:08:35 <lifeless> I would ask 'is everyone making a little time to do triage'
19:08:43 <lifeless> but since we have bugs untriaged for > a week
19:08:47 <lifeless> the answer is no :(
19:08:52 <lifeless> including me, obviuosly
19:09:14 <lifeless> My excuse this week is that we've had a super busy time @ HP with conference proposals for an internal thing
19:09:21 * lifeless is sorry
19:09:47 <lifeless> however, as a group - we need to do better.
19:09:54 <lifeless> Any ideas on how ?
19:10:27 <SpamapS> I am a big believer in a unified view that helps us drive to 0
19:10:53 <SpamapS> but, in the past that has required writing little one-off launchpad scripty things to generate a report.
19:11:55 <lifeless> ok
19:12:03 <lifeless> I agree, would be wonderful
19:12:09 <lifeless> but ponies.
19:12:16 <lifeless> Unless we want to commit to triaging all of OpenStakc
19:12:29 <lifeless> which I think would be a bit tough until everyone gets on board
19:13:10 <SpamapS> We do deploy all of OpenStack.. so there would be value in doing so.. but I'm not sure we can absorb the cost of all the irrelevant things we'd have to filter out.
19:13:14 <lifeless> however, can I get everyone to commit to - just one, one day, over the next week visiting the meetings page, ctrl-clicking on the bug section links, scrolling to the bottom and triaging all 'unknown' importance bugs ?
19:13:33 <rpodolyaka1> +1
19:13:34 <lifeless> if we make a joint commitment to do that *once* each once a week, I think we can keep on top of it very easily.
19:13:35 <shadower> ya
19:13:55 * Ng nods
19:14:00 <jcoufal> +1
19:14:01 <tzumainn> sounds reasonable
19:14:02 <lsmola> ok
19:14:04 <jtomasek> ok
19:14:06 <lifeless> #vote +1 if you will triage across all tripleo LP projects [see the meetings page] once a week
19:14:07 <jistr> ok
19:14:14 <ifarkas> +1
19:14:18 <shadower> +1
19:14:19 <tzumainn> +1
19:14:20 <lsmola> +1
19:14:21 <lifeless> erm, clearly I don't know how to drive mootbot votes.
19:14:23 <marios_> k
19:14:29 <jtomasek> +1
19:14:30 <SpamapS> +1
19:14:30 <Ng> do we want to try and somehow avoid all accidentally doing it all on the same day?
19:14:31 <viktors_> +1
19:14:45 <lifeless> Ng: birthday paradox
19:14:47 <slagle> +1
19:15:00 <lifeless> +1 from me
19:15:07 <ccrouch> Ng: thats like trying to avoid winning the lottery :-)
19:15:09 <lifeless> ok, so thats triage handled.
19:15:16 <Ng> +1
19:15:17 <lifeless> Onto the critical: this is a firedrill.
19:15:32 <lifeless> It's a bit sad tht /noone/ managed to find time to drive it forwards
19:16:04 <derekh> tried a updated patch about 2 hours ago that failed, havn't looked at why yet
19:16:07 <lifeless> This is another case of joint responsibilities. And yes, company stuff will draw us away : let me apologise again for the HP folk who had papers to write with a deadline
19:16:12 <SpamapS> I did not know it was in need of driving (or even in existence actually)
19:16:43 <shadower> lifeless: I wasn't here, would you elaborate pls?
19:16:52 <lifeless> SpamapS: ok, so last week we agreed to put firedrills: a) critical in the bug tracker, b) in the firedrill column in trello and c) in the #tripleo channel topic.
19:17:31 <SpamapS> Did not look at any of those since Friday.
19:17:38 <lifeless> shadower: the change to remove file injection on the cd-undercloud broke devtest because devtest's undercloud isn't being deployed identically to the cd-undercloud
19:18:15 <SpamapS> (so therein lies the problem.. "SpamapS is scatterbrained" bug has been open for decades)
19:18:23 <lifeless> SpamapS: today is your tuesday afternoon ?
19:18:47 <lifeless> SpamapS: anyhow it's not about you specifically
19:19:02 <rpodolyaka1> so we basically need to use neutron_dhcp_agent and force use_file_injection=False? that should be done by derekh's patch with minor tweaks to nova config template
19:19:02 <lifeless> there are 15 odd folk driving tripleo as a whole, each with specialities sure
19:19:14 <SpamapS> lifeless: aye Tue 12:19 to be exact
19:19:46 <lifeless> rpodolyaka1: right, my concern here is that noone - myself included - said 'moving that bug forward is the most important thing for the team'
19:19:47 <rpodolyaka1> I managed to get working overcloud today but haven't tried to run a user VM yet
19:19:59 <lifeless> rpodolyaka1: oh, but it sounds like you have - fantastic!
19:20:37 <lifeless> anyhow, lets not obsess
19:20:41 <rpodolyaka1> I was close to reproduce/check this yesterday but hit some strage bugs with ext4_resize_fs() :(
19:20:47 <derekh> rpodolyaka1: I managed to get use_file_injection=False by forcing it to string in heat template (by wraping in quotes)
19:21:05 <marios_> i got the overcloud vms up but they didn't get the ssh keys (so couldn't init the keystone setup etc)
19:21:16 <marios_> though that happened very late in afternoon and i left work a few hours ago
19:21:18 <derekh> rpodolyaka1: will update bug with details after this meeting
19:21:30 <lifeless> derekh: oh, is this because oac has a bug too ? That bug probably needs to be critical as well, since it's blocking another critical bug.
19:21:54 <rpodolyaka1> lifeless: no, oac bug looks similar but is irrelevant
19:22:06 <rpodolyaka1> *it
19:22:12 <lifeless> rpodolyaka1: oh, ok.
19:22:31 <lifeless> ah, I see, it's the template.
19:22:37 <lifeless> Lets get on that right after the meeting
19:22:51 <lifeless> any other bug material?
19:22:58 <Ng> 8
19:23:00 <Ng> derp
19:23:18 <lifeless> orly ? ;)
19:23:24 <lifeless> #topic reviews
19:23:25 <lifeless> http://russellbryant.net/openstack-stats/tripleo-openreviews.html
19:23:29 <lifeless> http://russellbryant.net/openstack-stats/tripleo-reviewers-30.txt
19:23:32 <lifeless> http://russellbryant.net/openstack-stats/tripleo-reviewers-90.txt
19:23:44 <lifeless> 
19:23:44 <lifeless> Stats since the last revision without -1 or -2 (ignoring jenkins):
19:23:47 <lifeless> Average wait time: 1 days, 13 hours, 49 minutes
19:23:50 <lifeless> 1rd quartile wait time: 0 days, 2 hours, 56 minutes
19:23:51 <dkehn> https://review.openstack.org/#/c/50749/
19:23:52 <lifeless> Median wait time: 0 days, 4 hours, 3 minutes
19:23:55 <lifeless> 3rd quartile wait time: 1 days, 4 hours, 29 minutes
19:23:56 <lifeless> 
19:23:56 <lifeless> Longest waiting reviews (based on oldest rev without nack, ignoring jenkins):
19:23:59 <lifeless> and
19:24:02 <lifeless> 6 days, 8 hours, 37 minutes https://review.openstack.org/50341 (Add unique constraint to ResourceClass.)
19:24:05 <lifeless> 5 days, 19 hours, 58 minutes https://review.openstack.org/52236 (add python-ironicclient to openstack-clients)
19:24:08 <lifeless> 1 days, 4 hours, 29 minutes https://review.openstack.org/49729 (Add Glance image id to `resource_classes` table)
19:24:11 <lifeless> 0 days, 4 hours, 20 minutes https://review.openstack.org/50477 (WIP : Add tempest elements)
19:24:14 <lifeless> 0 days, 4 hours, 3 minutes https://review.openstack.org/53128 (Add James Slagle to tripleo-cd-admins.)
19:24:17 <lifeless> so overall we're going ok, but there are reviews waiting nearly a week
19:24:44 <viktors_> Add unique constraint to ResourceClass - there is already two +2
19:24:59 <jistr> i'm gonna approve it
19:25:07 <viktors_> thanks
19:25:12 <jistr> we didn't approve because we had failing jenkins at the time
19:25:23 <lifeless> fair enough, do you know when jenkins got fixed ?
19:25:37 <jistr> well it's in the bug that we filled for it
19:25:42 <jistr> lemme dig it up
19:25:59 <lifeless> also you can 'recheck bug XXXX' to probe and find out if jenkins is fixed without causing gate pipeline resets.
19:26:13 <lifeless> jistr: cool thanks, I'm curious how long the review sat in inventory is all
19:26:21 <jistr> lifeless: we did a common patch for it with pblaho https://bugs.launchpad.net/tuskar/+bug/1240934
19:26:32 <lifeless> https://review.openstack.org/#/c/52236/ looks like gerrit got confused
19:26:53 <lifeless> I'll ask about that one in -infra
19:28:07 <lifeless> slagle: btw its good seeing lots of reviews from you - thank you!
19:28:26 <lifeless> most folk seem to have stepped up to the plate in fact
19:28:29 <lifeless> which is awesome
19:28:47 <slagle> lifeless: thanks. been trying to keep up :)
19:29:01 <lsmola> lifeless, ??  https://review.openstack.org/#/c/52236 , it is waiting for dependency to get in, right?
19:29:30 <rpodolyaka1> indeed
19:29:57 <lifeless> lsmola: yeah
19:30:06 <lifeless> lsmola: just had cluebat applied to me in -infra :)
19:30:25 <lsmola> lifeless, :-)
19:30:28 <lifeless> so the review stats tool could benefit by taking that into account in some fashion
19:30:45 <lsmola> lifeless, +1
19:31:05 <lifeless> overall, I'm happy with where we are at with reviews: is anyone unhappy? Are you finding reviewing hard? Are you getting reviews in a timely manner? Are they supportive? Are you getting tossed all over by contradictory reviews?
19:31:50 <SpamapS> lifeless: my only complaint is that non-tripleo programs do not review nearly as rapidly as tripleo programs. ;)
19:32:05 <lsmola> hehe
19:32:10 <SpamapS> I've gotten rather spoiled
19:32:23 <lifeless> SpamapS: from a centre of excelllence....
19:32:29 <dkehn> lifeless: 625 St. Ann Street
19:32:31 <lifeless> ok, next topic
19:32:46 <dkehn> lifeless: https://review.openstack.org/#/c/50749/
19:33:17 <lifeless> #topic projects needing releases
19:33:46 <lifeless> dkehn: interesting, did you change the commit id on my draft? I'm going to guess it was 'abandoned' and thus you couldn't push to it
19:33:58 <lifeless> Ng: how did you go getting a release of everything out ?
19:34:28 <Ng> lifeless: tripleo-heat-templates and the three tuskars are still pending, I failed to drive those through in time for today
19:34:45 <lifeless> Ng: ok, want to take the challenge up for another week ?
19:34:45 <Ng> all the other bits (incubator excepted) got releases last week
19:34:51 <Ng> lifeless: I absolutely do
19:34:52 <lifeless> Ng: since many things have had commits
19:34:57 <lifeless> Ng: we need more releases :)
19:35:05 <lifeless> Ng: cool!
19:35:07 <Ng> ok
19:35:13 <lifeless> #action ng to push the release wheelbarrow
19:35:26 <lifeless> #action Ng to push the release wheelbarrow
19:35:41 <lifeless> is it just me, or am I failing to drive mootbot?
19:35:59 <lifeless> meetbot
19:36:41 <lifeless> ok
19:36:44 <lifeless> #topic CD cloud status
19:36:47 <lifeless> I'll take this one
19:36:55 <lifeless> The CD cloud is deploying very reliably *except*
19:37:07 <lifeless> every couple of days the mellanox ethernet adapter is losing the plot
19:37:14 <lifeless> the symptoms are that it starts failing
19:37:19 <lifeless> and the logs show DNS lookup errors
19:37:40 <lifeless> doing a while true; do host cd-overcloud.tripleo.org; done loop
19:37:45 <lifeless> results in one in 20 or so failing
19:37:58 <lifeless> pings /mtr to the name servers don't show a fail
19:38:02 <lifeless> I've fixed this by
19:38:42 <lifeless> rmmod mlx4_en mlx4_core; modprobe mlx4_en; ip address del <undercloud 10.x>/26 dev eth2; ovs-vsctl del-port eth2; ovs-vsctl add-port br-ctlplane eth2
19:38:50 <lifeless> and it comes good for another couple of days
19:39:02 <lifeless> I've added a card to the 'make things better' column for someone to dig into WTF is going on.
19:39:19 <SpamapS> lifeless: saucy is out, maybe we should try with its shiny new kernel. :)
19:39:19 <lifeless> Anyone tried actual workloads on the overcloud ?
19:39:42 <lifeless> SpamapS: I think that should wait for us being able to actually redeploy the undercloud :>
19:40:28 <SpamapS> lifeless: yeah we have 3 whole months before raring is dead. :)
19:40:50 * Ng has not tried workloads on the overcloud, I don't actually have any cloudy workloads I could retool for such a thing :/
19:41:05 <lifeless> remember *everyone* can get overcloud accounts
19:41:11 <lifeless> just propose yourself to the incubator
19:41:25 <lifeless> free cloud accounts on top-grade hardware. Go for it!
19:41:43 <lifeless> next topic in 1 mon
19:41:47 <lifeless> *min*
19:41:50 <Ng> and they're only going to erase your data every few hours!
19:42:07 <marios_> lol
19:42:14 <lifeless> well, the passwords are stable
19:42:22 <lifeless> so if the deployment is automated
19:42:27 <lifeless> should be pretty straight forward
19:42:38 <lifeless> and hey - next MVP is data persistence :P
19:42:39 <SpamapS> we should try juju on it
19:42:45 <lifeless> #topic CI virtualized testing progress
19:43:02 <lifeless> pleia2: how goes?
19:43:55 <pleia2> hey, so we now have an experimental check on the tripleo-incubator project
19:44:21 <lifeless> \o/
19:44:25 <lifeless> all hail automated tests
19:44:28 <pleia2> can see it being run on this patch: https://review.openstack.org/#/c/52607/1
19:44:41 <lifeless> pleia2: whats next?
19:44:53 <pleia2> right now it's just an echo script, but we have images being successfully built and managed in the tripleo cloud from nodepool
19:45:18 <pleia2> next is Iteration 2 outlined here where we actually make it do something useful: https://etherpad.openstack.org/p/tripleo-test-cluster
19:46:18 <pleia2> no updates on progress here really, have my test nodepool up to get some of the dependencies sorted first (it's currently erroring on some basic things that I need to work out)
19:46:45 <lifeless> ok cool
19:46:56 <lifeless> pleia2: btw you shouldn't need nodepool for iteration 2 at all
19:47:10 <lifeless> pleia2: it's all now within other components
19:47:13 <pleia2> lifeless: hm, fair enough
19:47:24 <lifeless> pleia2: so if I was hacking on it, I wouldn't be worrying about nodepool for now.
19:47:30 <pleia2> ok
19:48:01 <lifeless> #topic review kanban
19:48:07 <lifeless> so I added this one-off topic
19:48:15 <lifeless> I'd like folks feedback on the use of kanban so far
19:48:18 <lifeless> whats good about it?
19:48:19 <lifeless> whats bad?
19:48:23 <lifeless> What would you like changed?
19:49:20 <Ng> I'm bad at keeping track day-to-day of where we are in it
19:49:40 <SpamapS> I have found its mere presence helps me focus on the immediate.
19:49:42 <lifeless> Ng: do you look at it day to day ?
19:49:45 * derekh same as Ng, just not checking it enough
19:49:51 <SpamapS> Just seeing trello in my open tabs reminds me "go work on MVP"
19:49:57 <rpodolyaka1> it's convenient for finding out what folks are currently working on, what current MVPs are
19:50:07 <SpamapS> and it is quite a sense of pride when I get to move a card to done :)
19:50:13 <Ng> lifeless: no :)
19:51:01 <SpamapS> I do not look at it day to day, because I am following the "only have one thing assigned to yourself".. so unless I finish one thing a day.. it doesn't get a detailed look until I finish the thing I am doing now.
19:51:06 <Ng> between bug triage and trello, I'm starting to think I need to have a stricter cadence to my days/week where I'm looking around at things more
19:51:36 <lifeless> so the tension is between flow and unblocking other people
19:51:41 <lifeless> if we just do our one thing
19:51:43 <lifeless> it's easy
19:51:47 <lifeless> but other folk can get stuck
19:52:17 <lifeless> represented by bugs [untriaged], firedrills [bug/topic/kanban], reviews[no -1/-2]
19:52:18 <SpamapS> Right, so to me it is "unblock others" followed by "do MVP work"
19:52:27 <lifeless> SpamapS: yeah
19:52:46 <SpamapS> Though I admit that unblock has been 99% reviews.
19:52:56 <SpamapS> As evidenced by the lack of triage by all of us :)
19:52:57 <lifeless> SpamapS: so it seems to me we need to poll the metadata for 'is someone out there blocked' at least once a day
19:53:10 <lifeless> as a team
19:53:15 <lifeless> but possibly as individuals too.
19:53:30 <lifeless> Any other thoughts?
19:54:25 <Ng> makes sense
19:54:26 <SpamapS> seems like that would be worth a tool to do that
19:54:57 <lifeless> SpamapS: add it to the roadmap ?
19:55:02 <Ng> my ponies and rainbows tool would be an IRC bot that notices when I come on in the morning and tells me if there's a firedrill, how many untriaged bugs there are, and suchlike
19:55:05 <SpamapS> one person could probably whip up an "obvious blockers" report just pulling all of the obvious data into one place that we all start at. Given the number of people involved, probably worth the time to do it.
19:55:06 <lifeless> SpamapS: as a card we can pickup ?
19:55:15 <SpamapS> lifeless: yes doing that
19:55:18 <lifeless> cool
19:55:23 <lifeless> ok, 2nd last topic
19:55:39 <lifeless> oh and russel just fixed http://russellbryant.net/openstack-stats/tripleo-openreviews.html for us
19:56:05 <lifeless> approved patches will no longer show as stuck, because they aren't, its' their deps that are stuck and they will be getting evolved
19:56:12 <lifeless> #topic  review the tweaked reviewer rules
19:56:33 <lifeless> So, last week we decided:
19:56:43 <lifeless> - cd reviews could use two +2's from anywhere
19:57:00 <lifeless> - multiple author reviews can use a +2 from the submitters
19:57:13 <lifeless> Feedback on those changes?
19:57:20 <lifeless> Working? Keep it? Discard it?
19:57:36 <SpamapS> keep it, it has already been used and helped keep the train moving
19:57:45 <jistr> multiple author rule worked well for us on the Jenkins critical bug
19:57:57 <Ng> lifeless: keep it
19:58:06 <rpodolyaka1> +1
19:58:14 <lifeless> #agreed keep the review tweaks we introduced last week
19:58:24 <lifeless> #topic open discussion
19:58:26 <Ng> I did something on two reviews which I think was slightly outside those rules, but seemed pretty reasonable
19:58:28 <lifeless> 2 minutes y'll
19:58:58 <Ng> there was a +2 already, but a typo in the commit message, which I fixed and carried the existing +2 forwards, added my own and Approved
19:59:50 <lifeless> Ng: thats inside the rules
20:00:06 <lifeless> Ng: you + original author - one +2. Other reviewer second +2. Done.
20:00:10 <Ng> :)
20:00:14 <lifeless> Ng: the 'approve' button is orthogonal.
20:00:55 <lifeless> ok, tiems pu
20:00:58 <lifeless> thanks everyone
20:01:01 <lifeless> times up.
20:01:03 <lifeless> #endmeeting