14:02:07 <slagle> #startmeeting tripleo
14:02:08 <openstack> Meeting started Tue Sep 22 14:02:07 2015 UTC and is due to finish in 60 minutes.  The chair is slagle. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:02:09 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
14:02:11 <openstack> The meeting name has been set to 'tripleo'
14:02:18 <derekh> yo
14:02:29 <akrivoka> hi
14:02:41 <slagle> hi everyone
14:02:48 <marios> \o
14:03:26 <dprince> hello
14:03:42 * dprince loves the new meeting time already
14:03:43 <slagle> #topic agenda
14:03:43 <slagle> * bugs
14:03:43 <slagle> * reviews
14:03:43 <slagle> * Projects needing releases
14:03:43 <slagle> * CI
14:03:46 <slagle> * Specs
14:03:48 <slagle> * one off agenda items
14:03:51 <slagle> * open discussion
14:04:10 <slagle> you will notice i added a standing item for one off agenda items before open discussion
14:04:33 <slagle> #info add one off agenda items to the wiki before each meeting if you have them
14:04:57 <slagle> #link https://wiki.openstack.org/wiki/Meetings/TripleO meeting wiki page
14:05:14 <slagle> #topic bugs
14:05:35 <slagle> #link https://bugs.launchpad.net/tripleo/
14:05:35 <slagle> #link https://bugs.launchpad.net/diskimage-builder/
14:05:35 <slagle> #link https://bugs.launchpad.net/os-refresh-config
14:05:35 <slagle> #link https://bugs.launchpad.net/os-apply-config
14:05:35 <slagle> #link https://bugs.launchpad.net/os-collect-config
14:05:37 <slagle> #link https://bugs.launchpad.net/os-cloud-config
14:05:40 <slagle> #link https://bugs.launchpad.net/os-net-config
14:05:42 <slagle> #link https://bugs.launchpad.net/tuskar
14:05:45 <slagle> #link https://bugs.launchpad.net/python-tuskarclient
14:06:06 <slagle> we had a lot of untriaged bugs last week, i believe i went through and hit them all
14:06:32 <dprince> A general question about bugs I have is if something is added to the #tripleo trello do we also want to create LP bugs?
14:06:55 <dprince> sometimes I just look at LP bugs when there is an issue
14:07:01 <slagle> probably if it's a bug, then yes
14:07:48 <slagle> if it's added to the trello board as a task that someone is doing, then i dont feel those need bugs as well
14:08:04 <slagle> does that make sense?
14:08:13 <shardy> that list of bug urls needs a refresh in the wiki ;)
14:08:24 <marios> yeah i was going to ask about removing the last two shardy
14:08:26 <dprince> sure
14:08:44 <derekh> ok so this was me, I tend usually to wait until I have enough info befor I open a bug, since I started using trello I havn't been creating a bug /me will do better
14:09:06 <slagle> shardy: is there something missing?
14:09:13 <dprince> derekh: yeah, and I haven't started using the trello yet ;)
14:09:15 <marios> dprince: i'd say, nice to have, since the trello boards may not have that great visibility outside tripleo (people would expect lp is what i'm saying i guess)
14:09:22 <dprince> derekh: so I've duplicated some work...
14:09:39 <shardy> slagle: the new repos we recently added aren't there
14:09:39 <derekh> dprince: your not the only one, maybe it was a stupid idea
14:10:10 <shardy> slagle: or are they all tracked via the tripleo LP org?
14:10:12 <dprince> derekh: not stupid, I'm willing to try it. Just not part of my daily checks with regards to project visibility yet
14:10:13 <slagle> shardy: i dont think we have launchpad projects for those, but we should decide if we want them
14:10:22 <shardy> slagle: aha, Ok, my mistake
14:10:38 <d0ugal> We should probably explicitly state that in the wiki
14:10:38 <derekh> dprince: ok
14:10:46 <d0ugal> Use triple for all projects other than ...
14:11:06 <slagle> #action d0ugal to update wiki about using the tripleo project
14:11:13 <slagle> you see how that works? :)
14:11:56 <slagle> if we want to add new projects too, we can
14:11:59 <d0ugal> slagle: damn :)
14:13:48 <slagle> afaics, all the critical bugs are assigned and being worked on
14:14:02 <slagle> is there any to draw attention to?
14:14:46 <slagle> ok, moving on
14:14:48 <slagle> #topic reviews
14:15:25 <slagle> #info There's a dashboard linked from https://wiki.openstack.org/wiki/TripleO#Review_team - look for "TripleO Inbox Dashboard"
14:15:45 <bnemec> That should include all of the new projects now.
14:16:03 <slagle> i was going to point that out :)
14:16:18 <slagle> #info bnemec updated the review dashboard for all the new projects
14:16:18 <bnemec> I have a review up to add them to the dashboard configuration too.
14:16:32 <bnemec> https://review.openstack.org/#/c/223861/
14:16:41 <bnemec> in case anyone wants to generate a custom dashboard.
14:16:53 <slagle> #info if you're using custom queries or your own searches, be sure they include all the new repos
14:17:38 <slagle> i also want to point out that we are in despearate need of people to review more, across the board
14:18:01 * dtantsur tries to review moar of tripleo stuff nowadays
14:18:08 <slagle> there are a couple of new specs proposed, the common library, stable branches, container support
14:18:23 <trown> ya lots of new code in tripleo, so lots more to review
14:18:35 <slagle> i don't realistically see us able to push forward on any of those without a serious step up in reviews
14:19:34 <slagle> i don't think a "required minimum" number of reviews per day really works or is an incentive, but we should be open to whatever ideas people have on how to increase review bandwidth
14:20:10 <shardy> slagle: I guess improving CI stability is the main one
14:20:19 <slagle> i guess i'll just say (hopefully without sounding too snarkish), if you have code you want to land, you better be reviewing other peoples code
14:20:42 <shardy> slagle: I'm sure I'm not the only one who tends to wait for passing CI before devoting review bandwidth, I know it's been a bad few weeks on that front
14:20:51 <shardy> is there anything collectively we can be doing to help with that?
14:20:53 <trown> I think we are pretty close to stable CI
14:21:07 <slagle> shardy: we shouldn't do that imo (wait for ci)
14:21:08 <dprince> trown: don't jinx us man
14:21:13 <trown> there will always be stuff like the puppet change today
14:21:15 <d0ugal> lol
14:21:18 <trown> lol
14:21:51 <slagle> shardy: people miss out on getting early feedback if they're cahnge sits unreviewed for a couple of days due to failing ci
14:21:58 <shardy> slagle: it's tough though, when there's 200 pending reviews in your inbox, you're drawn to those you can most likely land (so they're not in your inbox anymore! :)
14:22:18 <slagle> oh yea, for sure :)
14:22:26 <slagle> i try to look for stuff that is ready to land
14:22:33 <slagle> but i try to balance it out too
14:23:31 <dprince> clearing out the way old stuff would help too here
14:23:45 <derekh> +1 to clearing out old stuff (and bugs)
14:23:54 <bnemec> I actually did some of that yesterday.
14:23:55 <slagle> shardy: as far as collectively helping with CI, let's discuss about that when we get to that topic in just a bit :) want to make sure we draw attention to it
14:24:08 <shardy> slagle: sure :)
14:24:27 <shardy> Yeah, I never understood why OpenStack stopped the auto-abandon thing
14:24:34 <regebro> I'll try to do more reviews. The moving in is all done now, I should be able to get more sleep and do more reviews. ;-)
14:24:41 <bnemec> Because people's feelings got hurt.  Apparently.
14:24:48 <shardy> all projects are clogged with a bazillion ancient patches which is pretty distracting at times
14:25:02 <dtantsur> ++
14:25:13 <slagle> well, i say we abandon. we can have our own policy
14:25:16 <bnemec> We could resurrect the bot for TripleO.  I thought some other projects had done that already.
14:25:20 <marios> i think neutron has an 'autocull' feature where old .. yeah what slagle said
14:25:30 <trown> I am +1 to an auto-abandon ... it is not like it is any harder to un-abandon than it will be to rebase the ancient patch anyways
14:25:30 <dtantsur> for ironic we decided to do it manually
14:25:37 <shardy> bnemec: Yeah, I remember arguing about it on the ML, personally I didn't find clicking "restore change" occasionally all that offensive :)
14:25:49 <bnemec> shardy: Nor did I.
14:26:03 <slagle> bnemec: would you like to lead this effort? :)
14:26:13 <bnemec> And I continue to say that wasting core reviewer time abandoning old patches is stupid.
14:26:16 <bnemec> slagle: Sure
14:26:22 <regebro> I don't know what the old rules are, but auto-abandoning if the poster doesn't respond is a good idea IMO.
14:26:24 <derekh> lets bring it back, I forget how many people were against it when we last brough it up
14:26:40 <slagle> #action bnemec to look into auto-abandon policy for TripleO.
14:26:55 <derekh> Auto abandon anything that has negative feedback and hasn't been dealt with in over 2 weeks
14:27:02 <bnemec> I'm not arguing it on the ML again.
14:27:13 <slagle> yea, use the list at your discrtion :)
14:27:19 <slagle> discretion
14:27:24 <dprince> bnemec: ++
14:27:31 <bnemec> I believe it was generally agreed that projects could do it if they wanted to, but it wasn't going to be re-enabled globally.
14:27:46 <shardy> I think a project specific policy is fine, maybe communicate it via a tripleo tagged ML message
14:27:59 <bnemec> shardy: Agreed.
14:28:00 <d0ugal> +1
14:28:15 <slagle> cool, any other review business?
14:28:28 <slagle> #topic releases
14:28:42 <slagle> i did a few releases last week, but didnt get to all of them
14:28:55 <slagle> i'm happy to go through them again this week
14:29:07 <slagle> #action slagle to do releases
14:29:20 <slagle> as always, please ping folks if you need a release sooner and don't feel like waiting
14:29:33 <slagle> #topic CI
14:29:55 <derekh> We've been hurting hard since the move to instack based ci (some of it not unexpected I think its a completly new job)
14:30:10 <derekh> Over the last week, we out several outright breakages for various reasons and unreliability due to usually bad mirrors.
14:30:28 <derekh> We are now pinned to specific mirrors for F21, Centos and EPEL, in jenkins nodes, delorean containers and on undercloud
14:30:49 <derekh> This I think has helped a lot but people havn't noticed due to the breakages
14:31:21 <slagle> dprince: i'm curious about your thoughts about this latest puppet-nova breakage
14:31:29 <slagle> was that revert worthy?
14:31:32 <derekh> A fix for the most recent job breakage merged a few hours ago in t-h-t but we had to wait for a new package to hit delorean/current
14:31:40 <dprince> slagle: I think Depends-On is the best solution for these
14:31:49 <slagle> it seems changes like that in the puppet modules is disastrous for people using master
14:32:11 <dprince> slagle: we could also easily pin to an older puppet-nova in our CI while we get proper review time to think about it
14:32:33 <slagle> derekh: good update. i think the pins will help as well
14:32:50 <slagle> derekh: are there things you can think of where people can contribute to help out CI?
14:33:14 <derekh> slagle: I've been adding taks to the trello boards as they pop into my head If anybody has time to help out see here https://trello.com/b/dFPVlpFZ/tripleo-short-term-tasks
14:33:24 <derekh> *tasks
14:33:43 <slagle> #link https://trello.com/b/dFPVlpFZ/tripleo-short-term-tasks ways to help out ci
14:33:45 <derekh> Maybe I should just change that trello board to ci-tasks or something, then others can have other boards if they want
14:33:55 <derekh> I've been working on these but outages have been taking priority when they happen so its been slow progress
14:34:18 <slagle> i like the use of the [toci] tag myself
14:34:28 <derekh> slagle: ack
14:35:48 <derekh> FYI: the jobs overcloud-f21-nonha and overcloud-f21puppet-nonha are identical
14:35:51 <slagle> i also wanted to point i'm trying to come up with a script that will work for ci and local environments, https://review.openstack.org/225096
14:36:00 <slagle> so we can have some consistency there
14:36:05 <dprince> slagle: ++
14:36:17 <slagle> we also have a symlink for the ci delorean repo ow
14:36:18 <slagle> now
14:36:36 <derekh> slagle: yup, switching over to that script is a must
14:36:47 <slagle> #link http://trunk.rdoproject.org/centos7/current-tripleo/ delorean repo that toci is using
14:37:10 <slagle> #info FYI: the jobs overcloud-f21-nonha and overcloud-f21puppet-nonha are identical
14:37:15 <trown> we still need a mechanism to automate that link
14:37:26 <slagle> sounds like we got a volunteer!
14:37:38 <trown> I can take that
14:37:43 <slagle> :)
14:37:49 <derekh> trown: https://trello.com/c/RXaUOfPf/14-toci-bump-delorean-repo-url
14:37:53 <dprince> I had an idea about delorean feature, or maybe it is a new (small) project called delorean-mirror
14:37:55 <slagle> #action trown to look into automation around updating the delorean repo symlink
14:38:16 <dprince> essentially, it would mirror the latest current Delorean link either locally, or to another cloud for use by our CI.
14:38:51 <dprince> this would avoid the downtime associated w/ trunk.rdoproject.org
14:39:12 <dprince> thoughts? worth putting effort into this
14:39:37 <trown> I think we should be putting effort into not having downtime w/ trunk.rdoproject.org
14:39:38 <bnemec> +1 to mirror all the things.  That's basically infra's policy on it too.
14:39:44 <trown> but mirroring is also good
14:39:46 <dprince> we could probably script it such that if we bump our DELOREAN_REPO_URL in tripleo-ci we automitically refresh the repo
14:40:05 <slagle> i'm not 100% convinced it's needed. the last downtime was painful, but other than that, how much downtime has there been?
14:40:34 <dprince> trown: better uptime for trunk.rdoproject.org is good but the maintenance there is beyond the scope of upstream tripleo
14:40:46 <derekh> dprince: +.5, I'm not sure the recent outages are a fair representation to what we can expect
14:40:48 <slagle> i guess it depends on the effort
14:40:53 <dprince> slagle: this isn't new, mirrors are always a good idea IMO
14:41:00 <trown> dprince: ya, but upstream tripleo is relying on it, so it is a bit of a strange relationship
14:41:02 <dprince> derekh: ^^
14:41:17 <dprince> derekh: lets not focus on trunk.rdoproject.org then
14:41:24 <dprince> derekh: we just need mirrors for all the things I think
14:41:27 <slagle> dprince: ok
14:41:33 <slagle> i agree mirrors are a good idea
14:41:35 <dprince> otherwise our downtime is just going to be always bad
14:41:44 <dprince> CI downtime
14:41:50 <trown> ya I am +1 to mirrors in general
14:41:57 <slagle> just depends on the effort to setup i guess
14:42:04 <derekh> Yup, I think we could just have a "mirror" server to replace out old bandersnatch mirror
14:42:19 <dprince> the idea w/ delorean-mirror would be that it is special, and requires a bit of extra work to mirror properly (perhaps in a lightweight fashion that is also useful to local devs)
14:42:22 <derekh> then aim to get as much as possible on it, no need to single anything out
14:43:13 <slagle> #agreed mirror as much as possible
14:43:24 <derekh> delorean-mirror -> [fedora@bandersnatch ~]$ sudo wget --no-parent -r http://trunk.rdoproject.org/centos7/df/03/df0377d64e1ef0b53c4e78a8ff6a50159de5131a_733f1417/
14:43:37 <dprince> right, nothing is single out. I for example already have local mirrors for Centos and Fedora I use
14:44:18 <slagle> any other ci business?
14:44:18 * bnemec should do that
14:44:31 <dprince> There was a big discussion about not "forking" delorean... i.e. running your own locally. And that is what I do. The delorean-mirror thing was an idea that might be more paletable for some as it provides the same redundency... but is just a mirror
14:44:43 <trown> I would like to get inspection back running in CI https://review.openstack.org/#/c/210436/
14:44:50 <trown> and dependent patches
14:45:04 <trown> we removed it to get past the outage at the end of last week
14:45:25 <dprince> trown: I'd like introspection on for just a single job if possible
14:45:25 <slagle> trown: +1
14:45:59 <trown> dprince: it takes almost no time now, and I put in line to change the timeout to 10m
14:46:00 <dprince> trown: the extra CI time isn't strictly required for all jobs right? Perhaps combining it with another job to minimize the # of jobs too
14:46:14 <dprince> trown: if it takes more than 5 minutes it is worth it IMO
14:46:39 <trown> it will only take more than 5 minutes on failure
14:46:46 <slagle> dprince: are you ok with turning it back on in the job we have now?
14:46:50 <slagle> and add the other job later?
14:47:00 <dprince> slagle: yep, just thinking ahea
14:47:02 <trown> but I am ok just putting it on the nonha job as derekh suggested
14:47:05 <dprince> slagle: I don't think it should be required
14:47:20 <slagle> ok
14:47:20 <trown> in reality, even with it turned on we are not actually using it
14:47:36 <trown> as in we do not use the data it collects
14:47:40 <dtantsur> it's 2 minutes for dsvm, might be slightly more for IPA
14:47:54 <shardy> trown: the test could make some basic assertions about what's in the swift data tho?
14:48:15 <slagle> #info turn introspection back on in ci, setup a separate job in the future
14:48:19 <dprince> it is actually more than we think though because if we skip it we could skip building the extra ramdisk too right?
14:48:28 <dtantsur> nope
14:48:40 <dtantsur> dprince, we're switching to IPA, which will be one ramdisk
14:49:07 <trown> ya we could skip building it today...but it would not help once deploy also uses IPA
14:49:07 <dprince> dtantsur: ah, good point. But until then it would save a bit...
14:49:27 <slagle> #info switch to IPA will mean consolidated deploy and introspecton ramdisk
14:49:33 <dtantsur> I hope that complete switch to IPA will happen in 1-2 weeks, not more
14:49:37 <dprince> dtantsur: all the more reason to switch to IP sooner then :)
14:49:40 <d0ugal> What is IPA?
14:49:42 <dtantsur> old deploy ramdisk is deprecated
14:49:49 * d0ugal is here to ask the silly questions
14:49:51 <dtantsur> d0ugal, ironic-python-agent, our new ramdisk
14:49:56 <dtantsur> not beer unfortunately :)
14:50:01 <d0ugal> dtantsur: aha, shame.
14:50:01 <derekh> d0ugal: its a beer that does discovery
14:50:05 <dtantsur> lol
14:50:08 <bnemec> It's not bash.
14:50:12 <bnemec> That's all I care about. :-)
14:50:15 <d0ugal> lol
14:50:28 <d0ugal> A step closer to beer then
14:51:06 <trown> upstream deprecated the bash ramdisk in liberty, so that is all that really matters :)
14:51:23 <slagle> ok, are we all good to move on to specs? (9 mins left)
14:51:31 <slagle> #topic specs
14:51:48 <shardy> So I'm interested to get more feedback on the release branch spec
14:51:59 <shardy> particularly if folks think we could implement it from Liberty
14:52:08 <dtantsur> I had a question on the release branches that IIRC is not covered
14:52:13 <dtantsur> what do we do with requirements?
14:52:18 <slagle> i think we could, but we gotta get the ci setup
14:52:25 <shardy> e.g it'd be great to do an announcement around the liberty release pointing to "stable tripleo" install docs
14:52:51 <slagle> #link https://review.openstack.org/#/c/221811/ stable release branch policy
14:53:12 <slagle> shardy: i'll review your latest ps today
14:53:17 <shardy> dtantsur: the release branches have their own requirements, so I guess we'd need to get syncs from openstack/requirements stable branch
14:53:32 <dtantsur> shardy, then we'll have hard time backporting features
14:53:40 <dtantsur> (and least might have)
14:53:56 <shardy> dtantsur: That is a good point, and is probably a constraint we'll have to consider wrt features that are feasable
14:54:20 <bnemec> That's how all of the other projects are doing it now though, so we won't have any unique problems around that.
14:54:20 <shardy> dtantsur: impossible requirements is probably a valid No-Backport justification
14:54:30 <slagle> this would be good to add to the spec
14:54:37 <dtantsur> bnemec, other porjects don't backport features...
14:54:39 <shardy> slagle: yup, will do
14:55:01 <bnemec> dtantsur: There's probably a reason for that. ;-)
14:55:20 <dtantsur> I know, but we plan on backporting features, right?
14:55:26 <dtantsur> and we're the only project that will do it
14:55:37 <bnemec> Why would we backport features?
14:55:38 <slagle> #topic specs / open discussion
14:55:58 <trown> I think trying to backport features is really ambitious...probably too much so for liberty
14:56:12 <tzumainn> er, can I also ask for additional eyes on the overcloud deployment library spec?
14:56:15 <bnemec> Especially if it puts us in an impossible situation wrt requirements.
14:56:24 <dtantsur> bnemec, that's what is written on the spec :) please vote there
14:56:31 <bnemec> Ah
14:56:36 * bnemec hasn't read that spec yet :-(
14:56:52 <shardy> bnemec: the idea is that if, say, nova adds a new feature, we don't prohibit e.g backporting the t-h-t patches which add support for it
14:56:59 <slagle> yea the reasoning is laid out in the spec
14:57:04 <marios> tzumainn: link would be good thing to have https://review.openstack.org/#/c/219754
14:57:06 <shardy> obviously it has to be supported in stable/liberty puppet-tripleo tho
14:57:13 <tzumainn> whoops, thank you marios!
14:57:32 <shardy> we just don't block things which would otherwise work, just because they're adding functionality vs bugfixes
14:57:43 <shardy> the other option of course is just do a normal stable branc
14:57:46 <shardy> branch
14:57:52 <bnemec> That sounds like a gigantic headache and diverges from the rest of OpenStack's stable policy.
14:57:53 <marios> #info reviews please on Mainn's library/api spec @ https://review.openstack.org/#/c/219754
14:58:11 <shardy> which might be easier, but risks making folks wait longer when we're playing catch-up with new features added to projects
14:58:30 <trown> it also calls for much tougher reviews of feature backports, and we are not reviewing new stuff enough
14:58:52 <shardy> bnemec, trown: feedback on the spec would be great, let's discuss it there :)
14:58:52 <slagle> yea, i have that concern as well.
14:58:55 <bnemec> I'll look at the review today.  We should discuss this there anyway.
14:59:00 <bnemec> +1 :-)
14:59:04 <trown> shardy: ya I will look at the spec today
14:59:12 <shardy> I'm quite willing to just do the normal stable thing if that's the consensus
14:59:30 <shardy> but I really really want *a* way to deploy stable openstack w/upstream tripleo
14:59:42 <dtantsur> ++ for that
14:59:48 <trown> ya that seems required really
14:59:51 <marios> derekh: (waiting for appropriate moment but almost out of time...) hey how do we join that board? (or the group more generally https://trello.com/tripleo/members )
15:00:23 <slagle> thanks for the participation today folks!
15:00:27 <slagle> #endmeeting