07:01:35 <tchaypo> #startmeeting tripleo
07:01:36 <openstack> Meeting started Wed Jun 11 07:01:35 2014 UTC and is due to finish in 60 minutes.  The chair is tchaypo. Information about MeetBot at http://wiki.debian.org/MeetBot.
07:01:37 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
07:01:39 <openstack> The meeting name has been set to 'tripleo'
07:02:06 <greghaynes> O/
07:02:16 <d0ugal> o/
07:02:19 <derekh_> hi
07:02:22 <tchaypo> #topic agenda
07:02:26 <tchaypo> bugs
07:02:29 <tchaypo> reviews
07:02:31 <tchaypo> Projects needing releases
07:02:33 <tchaypo> CD Cloud status
07:02:35 <tchaypo> CI
07:02:37 <tchaypo> Tuskar
07:02:39 <tchaypo> Insert one-off agenda items here
07:02:41 <tchaypo> open discussion
07:02:43 <tchaypo> #topic bugs
07:02:51 <tchaypo> #link https://bugs.launchpad.net/tripleo/
07:02:54 <tchaypo> #link https://bugs.launchpad.net/diskimage-builder/
07:02:56 <tchaypo> #link https://bugs.launchpad.net/os-refresh-config
07:02:58 <tchaypo> #link https://bugs.launchpad.net/os-apply-config
07:03:00 <tchaypo> #link https://bugs.launchpad.net/os-collect-config
07:03:02 <tchaypo> #link https://bugs.launchpad.net/os-cloud-config
07:03:04 <tchaypo> #link https://bugs.launchpad.net/tuskar
07:03:06 <tchaypo> #link https://bugs.launchpad.net/python-tuskarclient
07:03:53 <marios> hi all
07:04:29 <tchaypo> I'm seeing one incomplete undecided and one new undecided in tripleo
07:04:34 <tchaypo> much better than a fortnight ago
07:04:56 <greghaynes> and no unassigned crits
07:05:09 <tchaypo> I'm going to go ahead and assign https://bugs.launchpad.net/tripleo/+bug/1320090 to lifeless as I still can't figure out what he means by it
07:05:11 <uvirtbot> Launchpad bug 1320090 in tripleo "pxe_ephemeral_format': u'ext4 left on nodes after deploys deleted" [Undecided,Incomplete]
07:06:10 <pblaho> o/
07:06:14 <rpodolyaka> o/
07:06:18 <GheRivero> o/
07:06:23 <jistr> o/
07:06:27 <proffalken> o/
07:06:30 <tchaypo> GheRivero: I see that you've looked at https://bugs.launchpad.net/tripleo/+bug/1325762 a bit, do you have a feeling for where it's at now?
07:06:32 <uvirtbot> Launchpad bug 1325762 in tripleo "intermittent tftp issues using physical hardware" [Undecided,New]
07:07:01 <GheRivero> it was a mellanox firmware issue
07:07:23 <GheRivero> we haven't seen the same behavior after upgrading it.
07:07:32 <tchaypo> Oh, there's a conclusion? That isn't documented on the issue. Do you know who would be well-placed to comment on that and close it out?
07:07:36 <lifeless> tchaypo: so the bug is what it says
07:07:57 <GheRivero> tchaypo: I will assigned it to me to doc it
07:08:12 <tchaypo> GheRivero: thanks :)
07:08:15 <lifeless> tchaypo: its rather ironic specific; if you haven't looked at what ironic does during deploys, it wouldn't make sense
07:09:29 <tchaypo> lifeless: ah. That's probably why.. I noticed that it was ironic-specific and made a mental note to try another devstack run with USE_IRONIC=1 but haven't done that yet
07:10:04 <tchaypo> let's talk about that one after this meeting or tomorrow morning
07:10:19 <tchaypo> Has anyone got anything to say about bugs in other projects?
07:11:03 <tchaypo> #topic reviews
07:11:28 <tchaypo> because I'm lazy, here's something I prepared earlier
07:11:32 <tchaypo> #link http://www.nemebean.com/reviewstats/tripleo-open.html
07:11:34 <tchaypo> #link http://www.nemebean.com/reviewstats/tripleo-30.txt
07:11:36 <tchaypo> #link http://www.nemebean.com/reviewstats/tripleo-90.txt
07:12:08 <marios> which time do we look at again?
07:12:20 <marios> average wait since last revision?
07:12:28 <tchaypo> last revision without -1 or -2
07:12:32 <tchaypo> tats since the last revision without -1 or -2 :
07:12:35 <tchaypo> Average wait time: 10 days, 10 hours, 18 minutes
07:12:37 <tchaypo> 1rd quartile wait time: 4 days, 23 hours, 19 minutes
07:12:37 <greghaynes> 7 days :/
07:12:39 <tchaypo> Median wait time: 7 days, 22 hours, 8 minutes
07:12:41 <marios> so a bit better?
07:12:41 <tchaypo> 3rd quartile wait time: 13 days, 19 hours, 17 minutes
07:12:42 <tchaypo> last week that 3rd quartile was 12 days 7 hours
07:12:52 <marios> yeah, a bit better
07:13:09 <tchaypo> the median last week was 5 days 19 hours
07:13:17 <tchaypo> to me it seems both of those have slipped
07:13:24 <lifeless> worse yes
07:13:29 <GheRivero> it's been a rough week in the gate
07:13:33 <tchaypo> even the average, last week, was 9 days 17 hours
07:13:41 <marios> oh, i was thinking avg time w/out -1/-2 was like 11 last couple weeks?
07:14:12 <tchaypo> I think it was 11 4 weeks ago, and the 3rd quartile was a bit higher as well, but we'd been moving it in the right direction
07:14:40 <tchaypo> 19:25:30 <lifeless> 1rd quartile wait time: 1 days, 1 hours, 11 minutes
07:14:42 <tchaypo> 19:25:30 <lifeless> Median wait time: 6 days, 9 hours, 49 minutes
07:14:44 <tchaypo> 19:25:30 <lifeless> 3rd quartile wait time: 13 days, 5 hours, 46 minutes
07:14:48 <tchaypo> that's from http://eavesdrop.openstack.org/meetings/tripleo/2014/tripleo.2014-05-20-19.01.log.html
07:14:49 <marios> i've been trying to clear out the oldest reviews this week. there's a fair bit of crud in there (like really small nits/changes) that ppl have moved on from/forgotten about
07:15:12 <marios> rebases needed in a few places for example
07:15:15 <tchaypo> so our 1st quartile has blown out from 1 to 4 days..
07:15:28 <tchaypo> As I understand it, simply giving those a -1 will remove them from that particular stat
07:15:44 <marios> tchaypo: yeah been doing that where applicable
07:16:05 <tchaypo> whether or not that counts as gaming the stats is a different matter - but where they're clearly stale I think that's a good move, it helps us see what still needs attention
07:16:30 <marios> e.g. lifeless https://review.openstack.org/#/c/92509/
07:17:04 <marios> probably a million other more important things you're doing and so it gets forgotten
07:17:06 * derekh_ would love to force abandon reviews that are clearly stale and forgotton about
07:17:38 <greghaynes> marios: that one a rebase would probably fix?
07:17:54 <marios> pap8
07:17:56 <marios> pep even
07:18:12 <greghaynes> Yes, but those tabs are there off master still?
07:18:29 <tchaypo> Someone should be able to do the rebase and push the new patch and see if that fixes things
07:18:29 <greghaynes> actually, youre right :/
07:18:36 <GheRivero> we should start hijacking those reviews waiting for a rebase or small nits
07:18:45 <tchaypo> in cases where they fix is simple and obvious I'm in favour of other people helping out
07:18:49 <marios> greghaynes: yeah, was a weird one because those tabs weren't introduced by lifeless
07:19:10 <greghaynes> yes, was going to propose that - for something where its straightforward (like a rebase) IMO just do it for them so we can keep moving forward
07:19:22 <tchaypo> Do we think this is worth of an email to the team asking people to pitch in?
07:19:34 <marios> tchaypo: i think it's killing our review queue
07:19:39 <GheRivero> tchaypo: +1
07:20:00 <marios> tchaypo: i'd love more eyes because doing this stuff means you get to review less cutting edge reviews so may lose what's going on currently
07:20:13 <tchaypo> #action tchaypo to email the team highlighting the state of the review queue and asking for help rebasing/fixing up/reviewing old reviews
07:20:38 <tchaypo> Is there anything else for us to talk about here?
07:21:06 * tchaypo listens to crickets
07:21:12 <tchaypo> no literally, I have crickets outside
07:21:30 <tchaypo> #topic Projects needing releases
07:21:36 <lifeless> marios: so
07:21:47 <lifeless> marios: for that sort of thing, as a reviewer, I tend to just rebase it for them
07:21:53 <rpodolyaka> tchaypo: I'm ready to help with that :)
07:22:08 <lifeless> marios: or ping them on IRC to do something about it
07:22:11 <rpodolyaka> (releases)
07:22:16 <tchaypo> rpodolyaka: #action rpodolyaka to don the superhero suit and release all the things once more
07:22:18 <lifeless> marios: mail from gerrit is basically useless because of overload.
07:22:21 <lifeless> #oops
07:22:37 <lifeless> [sorry for my slow responses here, this is my daughters bedtime, so that is taking priority]
07:22:48 <tchaypo> urr. how to i figure out if meetbot heard that or not?
07:22:49 <marios> lifeless: right. i wasn't sure what the 'etiquette' is with rebasing others patches. was just pointing it out. now that there is common understanding that we should go ahead and clear those out, will be doing so where possible
07:23:32 * tchaypo will pitch email as a clarification that "yea verily this is acceptable" rather than chiding
07:24:52 <tchaypo> #action rpodolyaka to don the superhero suit and release all the things once more
07:24:52 <lifeless> tchaypo: I believe you can find such collaborative stuff discussed previously on the -dev list
07:25:08 <tchaypo> With apologies to rpodolyaka if that shows up in the notes twice
07:25:23 <tchaypo> lifeless: thanks, I'll take a look and see if I can point back to earlier discussion
07:25:24 <rpodolyaka> tchaypo: np :)
07:25:30 <marios> rpodolyaka: 2 releases this week!
07:25:33 <rpodolyaka> heh
07:25:37 <lifeless> rpodolyaka: btw
07:25:41 <tchaypo> #topic CD Cloud status
07:25:43 <lifeless> rpodolyaka: have you seen my pbr semver spec ?
07:25:53 <rpodolyaka> lifeless: not yet, do you have a link?
07:25:54 <lifeless> rpodolyaka: I think it will help you a lot, would love your eyeballs on it
07:26:10 <lifeless> rpodolyaka: review #96608
07:26:20 <rpodolyaka> lifeless: ok, will take a look, thanks!
07:26:26 <tchaypo> Do we have anyone who can talk about CD cloud status?
07:26:43 <lifeless> tchaypo: derekh_ can, I can
07:26:45 <tchaypo> derekh_: do I remember you talking about this two weeks ago, or is that my memory playing up?
07:26:55 <lifeless> tchaypo: generally anyone in the tripleo-cd-admins team can
07:27:03 <derekh_> We're making progress on getting R1 back up and running but not their yet
07:27:03 <derekh_> looks like we got 41 working nodes to deploy onto
07:27:03 <derekh_> for the moment still running reduced load on R2
07:27:10 <tchaypo> lifeless: you've got important other commitments right now
07:27:13 <derekh_> btw while running reduced load
07:27:19 <lifeless> shes down now so I'm here
07:27:29 <tchaypo> derekh_: this is contributing to the gate buildup mentioned in the reviews topic?
07:27:31 <derekh_> people can run the ubuntu overcloud job by doing check experimental
07:27:42 <derekh_> but don't do that for all patches please :-)
07:28:19 <lifeless> derekh_: I got the vlan stuff you started working, haven't pushed it to a patchor anything
07:28:27 <tchaypo> #info derekh_ says R1 is coming back, up to ~41 working nodes. R2 still running at reduced load. People can "check experimental" to run the ubuntu overcloud job, but please don't do it for all patches
07:28:38 <lifeless> derekh_: we need the ssl certs from ng I think - but lets take this to the <- channel
07:28:55 <lifeless> folk can help us with this by:
07:29:09 <lifeless> - productionising the patches we come up with - spreading load around
07:29:13 <derekh_> tchaypo: R2 is running a reduced load, not at reduced load (small difference)
07:29:14 <lifeless> - following along on the etherpad
07:29:30 <tchaypo> #undo
07:29:31 <lifeless> - asking derekh_ and me (24hr coverage!) what needs doing next
07:29:31 <openstack> Removing item from minutes: <ircmeeting.items.Info object at 0x2690290>
07:29:45 <derekh_> lifeless: cool, will keep on it today
07:29:45 <lifeless> - joining the tripleo-cd-admins team and mucking directly in yourselves
07:29:48 <tchaypo> #info derekh_ says R1 is coming back, up to ~41 working nodes. R2 still running a reduced load. People can "check experimental" to run the ubuntu overcloud job, but please don't do it for all patches
07:30:36 <lifeless> tchaypo: might be worth capturing the how-to-help bits? dunno
07:30:58 <tchaypo> yes, I'm trying to copy and paste and summarise
07:31:08 <lifeless> ack
07:31:10 <tchaypo> can you hash-link the etherpad?
07:31:55 <lifeless> #link https://etherpad.openstack.org/p/tripleo-ci-hp1-rebuild
07:32:20 <derekh_> lifeless: FYI, I'm on PTO next week (and the following week)
07:32:22 <tchaypo> #info lifeless says folk can help us with this by: productionising the patches we come up with, following along on the etherpad, asking derekh_ or lifeless what needs doing next, or joining the tripleo-cd-admins team and mucking in
07:32:26 <lifeless> derekh_: ack
07:32:35 <tchaypo> Please Turn Over?
07:32:41 <tchaypo> Paid Time Off?
07:32:58 <proffalken> tchaypo: the first one if he's on a sun lounger?
07:32:58 <tchaypo> #undo
07:32:58 <lifeless> tchaypo: its merican for annual leave
07:32:58 <openstack> Removing item from minutes: <ircmeeting.items.Info object at 0x23d1f90>
07:33:02 <derekh_> tchaypo: yup, leave, holidays, I just wont be here :-)
07:33:16 <tchaypo> #info lifeless says folk can help us with this by: productionising the patches we come up with, following along on the etherpad, asking lifeless what needs doing next, or joining the tripleo-cd-admins team and mucking in
07:33:53 <tchaypo> Okay - anythiing else, or are we ready to move to CI?
07:34:58 <tchaypo> #topic ci
07:34:59 <derekh_> tchaypo: pull the trigger
07:35:11 <derekh_> One breakage since the last meeting, regression in nova
07:35:11 <derekh_> ci jobs were busted for a few hours but we put in a revert in tripleo ci to get it back running
07:35:11 <derekh_> which doesn't fix devtest :-(
07:35:37 <derekh_> so I'm wondering if maybe we should do the temp revert in incubator instead of tripleo-ci ? thoughts?
07:36:06 <lifeless> derekh_: well nova did appprove the revert quickly
07:36:16 <lifeless> derekh_: it wedged because the gate chose that moment to fail spectacularly
07:36:42 <greghaynes> and there was also debate as to whether to merge the revert versus the forward-fix
07:36:48 <derekh_> lifeless: yup, it was approved probably within about 4 or 5 hours, but not merged for 4 working days
07:36:57 <lifeless> derekh_: Idunno - review 96659 might point in this direction too
07:37:13 <lifeless> derekh_: perhaps have a read through that and tie this in as a use case
07:37:16 <lifeless> ?
07:37:26 <tchaypo> If it's possible for us to work around it and preserve old behaviour by using DIB_REPO* to set particular repos at a particular state
07:37:56 <derekh_> lifeless: ok, will do
07:38:05 <tchaypo> I think I'd rather document that in channel, as long as we think it's only going to be needed for a day or so
07:39:15 <tchaypo> but 4 days is pretty ouchy
07:39:30 <derekh_> I took a brief look through it yesterday, will comment later, but ya looks like it could work
07:39:34 <tchaypo> it sounds like we're done here. Anything worth noting for the minutes?
07:39:45 <derekh_> From next week for 2 weeks, If somebody could volunteer to keep and eye on http://goodsquishy.com/downloads/tripleo-jobs.html and do something if there is a problem if would be great as I wont be here.
07:40:28 <derekh_> I've also been trying to get elastic-recheck working for us but until this https://review.openstack.org/#/c/98154/ (or an alternative) is merged it wont do anything for us
07:40:43 <tchaypo> #info One breakage since the last meeting, caused by a regression in nova; we put a revert in tripleo-ci to get it back running, but that didn't fix devtest. Fix was approvied in ~4 hours but didn't merge for 4 days due to gate issues
07:41:25 <tchaypo> #info derekh_ will be away for two weeks so won't keeping an eye on http://goodsquishy.com/downloads/tripleo-jobs.html - would be good if other people could keep an eye on it and do something if there is a problem
07:41:36 <derekh_> so we have a few intermitant errors that cause failures (still to many) , their being marked with the ci tag https://bugs.launchpad.net/tripleo/+bugs?field.tag=ci
07:41:46 <derekh_> please use these if you need to recheck
07:42:24 <tchaypo> #info derekh_ has also been trying to get elastic-recheck working for us but until this https://review.openstack.org/#/c/98154/ (or an alternative) is merged it wont do anything for us. we have a few intermitant errors that cause failures (still to many) , their being marked with the ci tag https://bugs.launchpad.net/tripleo/+bugs?field.tag=ci - please use these if you need to recheck
07:42:33 <tchaypo> Anything else?
07:42:53 <derekh_> tchaypo: thats all I got :-)
07:42:54 <tchaypo> #topic tuskar
07:43:11 <tchaypo> I can never tell if I'm rushing or dallying
07:43:20 <tchaypo> derekh_: btw, lifeless had something he wanted me to tell you
07:43:30 <jcoufal> alright, I have just one request here
07:43:46 <jcoufal> we need as much eyes on the specs review as possible: https://review.openstack.org/#/c/94720/
07:43:49 <lifeless> sdague: ^ btw - can we get some of your bw to tease this apart? I hate the feeling that we're failing to reuse stuff due to miscommunication
07:43:58 <tchaypo> derekh_: I'll ping you after meeting if he isn't around to tell you himself
07:44:02 <lifeless> sdague: (the e-r review)
07:44:11 <marios> jcoufal: also https://review.openstack.org/#/c/97553
07:44:20 <lsmola__> marios: +1
07:44:26 <marios> (they are essentially 2 parts of the tuskar revamp, api, storage)
07:44:28 <jcoufal> marios: yes and that one as well
07:44:42 <derekh_> tchaypo: ok
07:44:42 <jcoufal> would be great if we can solve it ASAP and move to the implementation
07:44:47 <tchaypo> #info jcoufal requests many eyes on https://review.openstack.org/#/c/94720/ and https://review.openstack.org/#/c/97553 - 2 specs that are part of the tuskar revamp
07:44:51 <marios> this week is juno-1
07:44:59 <marios> ideally we should have those more or less landed by end of week
07:45:02 <lifeless> icehouse 0
07:45:21 <lifeless> (sorry, badum-tish)
07:45:27 <tchaypo> #info We'd really like to get these more or less landed by the end of this week
07:45:39 <marios> i was scratching my head there wondering if it was groundhog day lifeless
07:45:50 <lifeless> just a terrible joke
07:46:03 <tchaypo> thanks jcoufal, marios - it seems like having tuskar on the agenda is helping get info out
07:46:23 <lsmola__> also it depends a lot on Heat template rewrites https://review.openstack.org/#/c/97939/
07:46:29 <d0ugal> lsmola__: +!
07:46:35 <jcoufal> tchaypo: yeah, I hope after the specs period we will have more info coming here in this section
07:46:53 <tchaypo> #info lsmola__ notes that those specs depend a lot on Heat template rewrites https://review.openstack.org/#/c/97939/
07:47:55 <tchaypo> Okay, moving on
07:48:06 <tchaypo> #topic one-off agenda items
07:48:27 <tchaypo> None were raised earlier, but I didn't actually ask
07:48:37 <tchaypo> but in any case, the next item is
07:48:40 <tchaypo> #topic open discussion
07:48:44 <jcoufal> mid-cycle meetup
07:48:45 <tchaypo> so they can get raised here anyway
07:48:50 <tchaypo> Oh yes
07:48:55 <jcoufal> so just quick summary
07:49:02 <jcoufal> I've sent an e-mail to the mailing list but
07:49:15 <jcoufal> based on the etherpad: https://etherpad.openstack.org/p/juno-midcycle-meetup
07:49:42 <jcoufal> the options 1 & 3 & 4 are out of play - low number of attendees or collision with other events
07:49:52 <jcoufal> so there is one left which is option number 2
07:49:56 <tchaypo> #link http://lists.openstack.org/pipermail/openstack-dev/2014-June/036974.html
07:49:56 <jcoufal> July 21-25
07:50:24 <jcoufal> Clint objected that he can't make it and suggested to move the meetup to the end of the feature freeze
07:50:32 <shadower> jcoufal: me too
07:50:39 <tchaypo> #info Thanks to jcoufal we now have a mid-cycle meetup and dates - July 21-25 at RedHat's office in Raleigh, North Carolina. See you there!
07:50:54 <tchaypo> jcoufal: Do people need to contact you to RSVP? Update the etherpad?
07:50:56 <jcoufal> I discussed this possibility with other folks but it doesn't look as a very good idea
07:51:01 <jcoufal> reasons are in the e-mail
07:51:24 <jcoufal> the main reason is that we won't be able to propose neither get in features which we need in time
07:51:34 <StevenK> jcoufal: One thing that would help is sending an e-mail to the attendees giving them the address of the office and near-by hotels.
07:51:42 <jcoufal> so if we need to check and plan and do something it really needs to be in the middle of the cycle
07:51:44 <derekh_> So if option 2 is the selection, how about we move the voting stuff to the bottom  of the etherpad and start option 2 again ? so we can get actull numbers attending  as they get approval ?
07:51:51 <lifeless> StevenK: uhm google is better at hotels than mere humans ;)
07:51:58 <lifeless> StevenK: address is what we ned :)
07:52:05 <lifeless> derekh_: +1
07:52:09 <jcoufal> so what I wanted to do here
07:52:14 <jcoufal> we set a date
07:52:22 <jcoufal> I think July 21-25 is the right one
07:52:38 <jcoufal> and we move forward to get the list of actual attendees, the exact place and suggested hotels in time
07:52:51 <tchaypo> Yeah, I think any date is going to have some people who can't make it; this date at least means our PTL can make it
07:52:57 <lifeless> jcoufal: I agree with you; I endorse 21-25 FWIW
07:53:09 <tchaypo> and aside from that it seems on par with the next best option
07:53:14 <lifeless> tchaypo: more than me; 14 vs 27 with the other options
07:53:22 <lifeless> tchaypo: its massive ahead of the next best option
07:53:37 <lifeless> tchaypo: since the other high attendance option is a no-go conflict w/nova
07:53:38 <tchaypo> My memory is that two options had 27, but only one of those had a PTL
07:53:46 <jcoufal> great, I take it as a word and we are arranging for July 21-25
07:53:52 <tchaypo> urgh. Yeah, that's nasty.
07:54:05 <lifeless> tchaypo: your memory is correct, but the nova conflict is worse than no PTL - PTLs are nice to have :)
07:54:19 <lifeless> jcoufal: do you need anything from me?
07:54:35 <jcoufal> #agreed mid-cycle meetup date is July 21-25
07:54:52 <jcoufal> lifeless: nope, I will send a follow up to the mailing list and start with next arrangements
07:54:56 <tchaypo> thanks jcoufal
07:55:00 <jcoufal> lifeless: will keep you up to date
07:55:11 <jcoufal> thanks everyone to participation in this
07:55:14 <tchaypo> #info Look to the mailing list for next arrangements from jcoufal
07:55:20 <jcoufal> and I am very sorry if anybody can't make it that time
07:55:29 <jcoufal> but it's hard to find a time when everybody is available :(
07:55:40 <d0ugal> jcoufal: np, I can't make it but the reasoning is sound
07:56:09 <tchaypo> We have 4 more minutes
07:56:15 <tchaypo> ish
07:56:43 <StevenK> I don't know if I can troll by just saying the command to end the meeting
07:56:57 <tchaypo> once again, stevenk slides in right at the end
07:57:06 <StevenK> I've spoken before
07:57:06 <derekh_> so anybody see the game?
07:57:10 <StevenK> Not sure if the bot will ignore me, or end the meeting :-P
07:57:20 <tchaypo> StevenK: go ahead, we seem to be done
07:57:42 <StevenK> #endmeeting
07:57:50 <StevenK> Nope, I is ignored
07:57:51 <tchaypo> #endmeeting