19:04:04 <lifeless> #startmeeting tripleo
19:04:05 <openstack> Meeting started Tue Jul  1 19:04:04 2014 UTC and is due to finish in 60 minutes.  The chair is lifeless. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:04:06 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:04:09 <openstack> The meeting name has been set to 'tripleo'
19:04:12 <derekh_> hi
19:04:51 <bnemec> o/
19:05:15 <gfidente> o/
19:05:20 <pblaho> o/
19:05:39 <greghaynes> O/
19:06:16 <tchaypo> morning
19:06:31 <lifeless> #topic agenda
19:06:39 <lifeless> bugs
19:06:40 <lifeless> reviews
19:06:40 <lifeless> Projects needing releases
19:06:40 <lifeless> CD Cloud status
19:06:40 <lifeless> CI
19:06:42 <lifeless> Tuskar
19:06:44 <lifeless> Specs
19:06:47 <lifeless> Insert one-off agenda items here
19:06:49 <lifeless> open discussion
19:06:54 <lifeless> #topic bugs
19:07:02 <lifeless> #link https://bugs.launchpad.net/tripleo/
19:07:02 <lifeless> #link https://bugs.launchpad.net/diskimage-builder/
19:07:02 <lifeless> #link https://bugs.launchpad.net/os-refresh-config
19:07:02 <lifeless> #link https://bugs.launchpad.net/os-apply-config
19:07:02 <lifeless> #link https://bugs.launchpad.net/os-collect-config
19:07:04 <lifeless> #link https://bugs.launchpad.net/os-cloud-config
19:07:07 <lifeless> #link https://bugs.launchpad.net/tuskar
19:07:09 <lifeless> #link https://bugs.launchpad.net/python-tuskarclient
19:07:16 <tchaypo> This is my weekly reminder that anyone can use the link and info commands, not just lifeless - if you have something worth noting in the meeting minutes feel freeo to tag it
19:07:33 <lifeless> tchaypo: you could put that in the stuff I copy-paste :)
19:07:57 <tchaypo> I approve of this lazinesss
19:08:23 * jdob resists temptation to make a note about our ability to make notes
19:09:13 <lifeless> so we have some untriaged I see
19:09:26 <lifeless> and still crits
19:09:41 <lifeless> is michael kerrin online for the meeting? bug 1263294
19:09:42 <uvirtbot> Launchpad bug 1263294 in tripleo "ephemeral0 of /dev/sda1 triggers 'did not find entry for sda1 in /sys/block'" [Critical,In progress] https://launchpad.net/bugs/1263294
19:11:33 <lifeless> derekh_: how is 1308407 looking ?
19:12:17 <lifeless> derekh_: I believe your patch has been superceded? Does the new review reference the bug ?
19:12:33 <lifeless> GheRivero_afk: how is bug 1316985 coming ?
19:12:34 <uvirtbot> Launchpad bug 1316985 in tripleo "set -eu may spuriously break dkms module" [Critical,In progress] https://launchpad.net/bugs/1316985
19:13:12 <lifeless> no sign of rpod online - anyone know the status of bug 1317056 ?
19:13:17 <uvirtbot> Launchpad bug 1317056 in tripleo "Guest VM FS corruption after compute host reboot" [Critical,Triaged] https://launchpad.net/bugs/1317056
19:13:24 <lifeless> and lastly bug 1319979 also GheRivero_afk
19:13:25 <uvirtbot> Launchpad bug 1319979 in tripleo "rabbitmq-server fails to restart in check-tripleo-overcloud-precise" [Critical,Triaged] https://launchpad.net/bugs/1319979
19:13:58 <lifeless> I think the last one is invalid, since that job has happily passed many times since
19:14:02 <derekh_> lifeless: I added a reference to the review a few minutes ago, going to review it after meeting
19:14:05 <lifeless> we used it to debug neutron the other day
19:14:49 <lifeless> what else
19:14:58 <lifeless> bug 1329068 - slicknick was on that
19:14:58 <uvirtbot> Launchpad bug 1329068 in diskimage-builder "The trove guest-image is unable to publish the guest-config via cloud-init" [Critical,In progress] https://launchpad.net/bugs/1329068
19:15:00 <lifeless> but its still open
19:15:34 <tchaypo> I've added Closes-Bug: #1308407 to the new patchset
19:16:13 <tchaypo> via the slick new in-browser commit message editor
19:16:34 <lifeless> in 1329068 there is a fix merged to dib
19:16:46 <lifeless> I think we can close it now? SpamapS - do you know ?
19:17:00 <SpamapS> lifeless: checking
19:17:14 <SpamapS> lifeless: yeah I thought that was closed already.
19:17:42 <clarkb> tchaypo: boo
19:17:44 <bnemec> It should have - the second patch is tagged Closes-Bug.
19:17:47 <clarkb> tchaypo: you reset all the votes
19:17:48 <SpamapS> ahh yeah it's committed
19:17:53 <SpamapS> lifeless: they used partial intead of closes
19:18:17 <bnemec> Oh, the second one is trove-integration.
19:18:20 <tchaypo> clarkb: it wasn't smart enough to see that as a minor "only commit message was changed" commit?
19:18:33 <clarkb> tchaypo: no beacuse we actually test commit messages
19:18:37 <clarkb> tchaypo: for better or worse
19:19:11 <tchaypo> clarkb: to make sure they have periods in the right places and so on?
19:19:17 <clarkb> yes
19:19:21 <lifeless> stabbity stab stab
19:19:24 <tchaypo> my apologies
19:19:59 <lifeless> stabbity at the 'or worse', not you.
19:20:11 <SpamapS> lifeless: bug 1329068 was released in 0.1.21 :)
19:20:12 <uvirtbot> Launchpad bug 1329068 in diskimage-builder "The trove guest-image is unable to publish the guest-config via cloud-init" [Critical,In progress] https://launchpad.net/bugs/1329068
19:20:15 * SpamapS marks as such
19:20:24 <lifeless> ok thats all the bugs
19:20:57 <lifeless> though looking at the untriaged reminded me to collar ironic folk about removing the flavor integration :(
19:21:26 <lifeless> #topic reviews
19:21:38 <lifeless> #info There's a new dashboard linked from https://wiki.openstack.org/wiki/TripleO#Review_team - look for "TripleO Inbox Dashboard"
19:21:40 <lifeless> #link http://russellbryant.net/openstack-stats/tripleo-openreviews.html
19:21:43 <lifeless> #link http://russellbryant.net/openstack-stats/tripleo-reviewers-30.txt
19:21:46 <lifeless> #link http://russellbryant.net/openstack-stats/tripleo-reviewers-90.txt
19:22:00 <lifeless> Stats since the last revision without -1 or -2 :
19:22:00 <lifeless> Average wait time: 9 days, 10 hours, 55 minutes
19:22:00 <lifeless> 1st quartile wait time: 1 days, 6 hours, 51 minutes
19:22:00 <lifeless> Median wait time: 5 days, 18 hours, 17 minutes
19:22:00 <lifeless> 3rd quartile wait time: 13 days, 11 hours, 45 minutes
19:22:05 <lifeless> we've crept back up again
19:22:31 <lifeless> Looking at the baseline stats
19:22:40 <lifeless> *lots* of our cores are not getting 3/day.
19:22:44 <gfidente> just in case, sources for the gerrit dashboard linked from the wiki are here: https://github.com/stackforge/gerrit-dash-creator/blob/master/dashboards/tripleo.dash
19:23:09 <lifeless> #info REMINDER: Core reviewer commitment was 3 a day, rain or shine.
19:23:18 <lifeless> #link http://russellbryant.net/openstack-stats/tripleo-reviewers-30.txt
19:24:03 <tchaypo> I was looking at the stats and I noticed we seem to have lots of reviews that have just one +1
19:24:29 <tchaypo> ego https://review.openstack.org/#/c/86316/ which is 55 days since last revision, has had no negative feedback, but needs some +2s to proceed
19:24:41 <greghaynes> tchaypo: that one was just recently restored
19:24:46 <greghaynes> seems stats dont take that into account
19:25:02 <SpamapS> lifeless: Been pulled off on Heat work and also there are vacations happening around this time of year... so that throws it off. Anyway, for me I hope to slam the queue tomorrow.
19:25:38 <tchaypo> That was the case with some others that I saw too - they'd been auto-abandoned, then restored, then #crickets
19:25:56 <lifeless> SpamapS: 64 is what you get over a 30 calendar day period with 5 day work week
19:25:59 <gfidente> the dashboard actually has a nice section titled "No Negative Feedback" which should be handy for changes needing only +2
19:26:11 * gfidente encourages using the dashboard :)
19:26:18 <lifeless> SpamapS: you're over that - so thanks for the apology, but don't stress. We have folk below that by a significant marging.
19:26:44 <lifeless> margin.
19:26:51 <tchaypo> gfidente: I reckon that's worth a #info
19:26:56 <lifeless> Does anyone have specific reviews they want to call out for help or attention on ?
19:27:09 <gfidente> tchaypo, lifeless did it already :P
19:27:33 <SpamapS> lifeless: oh I aim for 90 :) weekends? what are weekends? ;-)
19:27:47 <SpamapS> actually I have family commitments I need to attend to ATM ...
19:27:54 <SpamapS> anybody need anything from me before I disappear?
19:28:07 <greghaynes> SpamapS: input on https://review.openstack.org/#/c/98709/ at some point
19:28:08 <tchaypo> oh yeah - because I put it into the paste stuff last week.
19:28:14 <tchaypo> total win for laziness!
19:28:25 <lifeless> SpamapS: not I, though -specs review please :)
19:28:30 <pblaho> can someone point me to place where I can discover diiference  http://www.stackalytics.com/report/contribution/tripleo-group/90 vs. http://russellbryant.net/openstack-stats/tripleo-reviewers-90.txt
19:28:41 <tchaypo> lifeless: Are you planning to do a stats run to look for new cores soon?
19:29:11 <lifeless> tchaypo: every day this last week, but sob its hard. Also its not a stats run - the stats is like 30 seonds. Its a systematic metareview
19:30:03 <tchaypo> lifeless: not even that, the stats are on russellbryant.net before you even think about wanting them..
19:30:06 <lifeless> tchaypo: there's been lots of knee jerk reaction to the very idea of stats being involved, so I'm trying to avoid any suggestion that the stats dominates; they never did...
19:30:19 <tchaypo> <3
19:30:35 <lifeless> pblaho: well, other than them being different? DIfferent code bases.
19:31:04 <pblaho> lifeless: :-)
19:31:04 <tchaypo> pblaho: apparently not
19:31:22 <tchaypo> gfidente: btw, how's the plan to get a hosted .html of those links going?
19:31:32 <SpamapS> ACK, o/
19:31:40 <gfidente> gfidente, sorry but didn't look into that :(
19:31:41 <lifeless> ok, next topic time
19:31:48 <gfidente> tchaypo, ^^ .(
19:31:53 <lifeless> #projects needing releases
19:32:00 <lifeless> do we have a victim^Wvolunteer
19:32:01 <jdob> i will do it. i will bring the ring to Mordor
19:32:06 <lifeless> \o/
19:32:13 <lifeless> #action jdob to release the world
19:32:21 <jdob> i'm not gonna lie, part of me volunteering was just so I could say that
19:32:26 <bnemec> One does not simply release into Mordor...
19:32:39 <tchaypo> gfidente: ah, cool
19:32:40 <lifeless> lol, grin
19:32:44 <lifeless> thats made my day
19:32:48 <lifeless> #topic CD cloud status
19:32:57 <lifeless> RH seems up - any specifics to add dprince / derekh_ ?
19:33:28 <greghaynes> Seed memory issues
19:33:29 <derekh_> lifeless: one of the TE's was failing all overcloud jobs
19:33:33 <dprince> lifeless; we got 10 more nodes today
19:33:43 <lifeless> dprince: \o/ fantastic
19:34:03 <dprince> lifeless: had a minor issues to fix because there was an IP address collision (the new DRAC's were in our undercloud range).
19:34:10 <greghaynes> I keep seeing fails like https://launchpad.net/bugs/1336479
19:34:12 <dprince> lifeless: but we've resolved that
19:34:12 <lifeless> greghaynes: righto - we still need to rebuild them; we need to make the seed size honour the node size request - I noted my local seed vm was 2G when my nodes were 3G just before
19:34:13 <uvirtbot> Launchpad bug 1336479 in tripleo "CI build failing due to insufficient memory on seed" [High,Confirmed]
19:34:16 <derekh_> lifeless: I've killed it so should bring stats up a little https://bugs.launchpad.net/tripleo/+bug/1335926
19:34:17 <uvirtbot> Launchpad bug 1335926 in tripleo "Can't destroy libvirt domain on TE host" [High,Confirmed]
19:35:02 <derekh_> Wi'll rebuild the TE host later to get bac to full capacity
19:35:21 <lifeless> dprince: are they online now or pending deploy ?
19:35:32 <derekh_> then we gotta decide when to being the new hosts into the overcloud
19:35:47 <derekh_> lifeless: their sitting there waiting  for us to use them
19:35:58 <dprince> lifeless: ^^ yep. don't rush us. Just saying we got access to them. We still need to do some burn in on them I think
19:36:07 <lifeless> cool
19:36:29 <lifeless> we need to get stevenk's fixes to toci's environment creation in too
19:36:34 <lifeless> to unlock the better Ironic API
19:36:35 <dprince> lifeless: our last round of hardware was burned-in for us. This time they pretty much just racked them I think
19:36:51 <lifeless> and we need to fix the seed sizing thing one way or another
19:36:52 <lifeless> dprince: cool
19:37:00 <derekh_> lifeless: link to the fix ?
19:37:01 <lifeless> ok so hp1 and hp2 regions
19:37:39 <lifeless> derekh_: looks to me like he analysed it and hasn't pushed it
19:37:47 <derekh_> lifeless: ok
19:37:50 <lifeless> derekh_: basically the ssh key isn't included in the per-node metadata.
19:38:09 <lifeless> derekh_: so we can't go from per-node metadata to ironic credentials field and drop the file injection
19:38:26 <lifeless> derekh_: which would mean we have /no/ difference between bm and vm for ironic, so one less template variation.
19:38:26 <derekh_> lifeless: ok will take a look once its up
19:38:42 <lifeless> #action stevenk to push up toci fix for ssh creds in nodes list
19:39:04 <lifeless> hp1 and hp2 are still down; we have the bastion on both and a uc on hp1; Ng - did you get a uc up on hp2 ?
19:39:34 <lifeless> lifeless, tchaypo and derekh_ are working on the vlan functionality re-fit to let us bring them up. hp1's machines are all vetted for config - 46 of em
19:39:54 <lifeless> hp2 we haven't commenced burnin / validation yet.
19:39:55 <derekh_> lifeless: I push a minor fix for your ensure-bridge patches today and verified the MAC thing is working, didn't get to the vlan part but its next on the list
19:40:02 <Ng> lifeless: that's my focus for the rest of the day :)
19:40:14 <lifeless> Ng: cool
19:40:28 <lifeless> derekh_: thanks. I'll pull that down, then be on the vlan thing myself
19:40:43 <lifeless> #topic CI
19:40:49 <lifeless> derekh_: da floor is yours
19:41:31 <derekh_> dprince: mentioned a error happening on some jobs pinging the user instance , I could do with somebody looking into that
19:41:39 <derekh_> if not I'll take a look
19:42:21 <derekh_> about 5 to 10% of overcloud jobs were failing because of https://bugs.launchpad.net/tripleo/+bug/1335926 but that should be ok now
19:42:23 <uvirtbot> Launchpad bug 1335926 in tripleo "Can't destroy libvirt domain on TE host" [High,Confirmed]
19:42:31 <slagle> is there a bug or a link to a failed job?
19:42:37 <slagle> (for the user instance one)
19:42:58 <derekh_> not much on the ci spec has moved recently hopfully I'll get more time on it once we get the HP cloud up and running
19:43:14 <derekh_> slagle: I'll find one and paste it
19:44:15 <derekh_> slagle: this looks like it http://logs.openstack.org/09/98709/4/check-tripleo/check-tripleo-overcloud-f20/1d5fe2c/console.html
19:44:22 <slagle> k
19:45:06 <derekh_> although there is no garentee that not related to the patch
19:46:02 <derekh_> lifeless: thats all from me at the moment
19:46:42 <derekh_> slagle: opp, you said bug, I don't think there is one yet, will do
19:47:23 <lifeless> ok
19:47:24 <slagle> oh that's np, i just wanted to see a failed log
19:47:29 <lifeless> #topic tuskar
19:47:30 <slagle> but file away if you want :)
19:47:57 <jdob> nothing specific from me this week
19:48:06 <lifeless> #topic specs
19:48:48 <lifeless> so for HA
19:48:54 <lifeless> we need 103692 reviewed
19:49:00 <lifeless> turns out the PKI stuff is actually a little complex
19:49:24 <lifeless> there's a few with +2 and no cmoments for a bit that I'm going to +A this week
19:49:26 <lifeless> also
19:49:34 <lifeless> I'd like to call everyones attention to
19:49:42 <lifeless> #info backwards compat spec - please review
19:49:54 <tchaypo> lifeless: link?
19:49:58 <lifeless> #link https://review.openstack.org/102081
19:50:05 <lifeless> tchaypo: patience
19:50:29 <tchaypo> lifeless: well that's embarrassing
19:50:48 <lifeless> whoever logged into hp1 - use screen -xr so you don't detach everyone else :)
19:51:13 <lifeless> any other specs issues to raise?
19:52:55 <lifeless> #topic open discussion
19:53:01 <lifeless> I have a thing
19:53:05 <lifeless> the mid cycle is coming up
19:53:13 <lifeless> last cycle we did a mix of talking and doing
19:53:29 <lifeless> and one bit of feedback we received from the talkers was that having a broad agenda is useful
19:53:40 <lifeless> so if you have specific bits of face to face talking you want done,please put them in the etherpad
19:54:09 <lifeless> #info if you have agenda items for the mid-cycle please add to the agenda, a schedule for discussion-times will be forthcoming from those inputs
19:54:13 <derekh_> +1, I think we would have a lot to gain if we scheduled a few design summit like session
19:55:03 <slagle> good idea
19:56:19 <gfidente> #link https://etherpad.openstack.org/p/juno-midcycle-meetup
19:59:30 <lifeless> ok, seems like we're done here! thanks everyone.
20:00:04 <lifeless> #endmeeting