14:02:04 <dprince> #startmeeting tripleo
14:02:04 <openstack> Meeting started Tue Jan 26 14:02:04 2016 UTC and is due to finish in 60 minutes.  The chair is dprince. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:02:05 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
14:02:07 <openstack> The meeting name has been set to 'tripleo'
14:02:26 <derekh> yp
14:02:29 <derekh> yo
14:02:30 <jdob> o/
14:02:32 <dtantsur> o/
14:02:35 <marios> \o
14:02:36 <trown> o/
14:02:41 <dprince> hi everyone
14:02:53 <shadower> hey
14:02:54 <jtomasek> o/
14:03:04 <shardy> o/
14:03:07 <marios> thanks for reminder dprince am also on a bluejeans call so forgot
14:03:21 <tzumainn> hiya
14:03:23 <dprince> marios: np
14:03:28 <akrivoka> hey
14:03:30 <akrivoka> \o
14:03:31 <dprince> #topic agenda
14:03:31 <dprince> * bugs
14:03:31 <dprince> * Projects releases or stable backports
14:03:31 <dprince> * CI
14:03:31 <dprince> * Specs
14:03:33 <dprince> * one off agenda items
14:03:36 <dprince> * open discussion
14:03:59 <dprince> There are no 'one off agenda' items this week...
14:04:12 <dprince> anything else to add to the agenda?
14:04:25 <derekh> dprince: I got 2 topics to discuss
14:04:35 <trown> I have some stuff for open discussion if there ends up being time
14:04:37 <derekh> dprince: 1. critical alerts for bugs effecting tripleo
14:04:46 <derekh> dprince: 2. using launchpad for wish list bugs
14:05:17 <dprince> derekh: ack, we'll add those in..
14:05:24 <derekh> ty
14:05:39 <dprince> trown: hopefully there is time for you topic too :)
14:05:43 <dprince> lets go then
14:05:50 <dprince> #topic bugs
14:06:09 <dprince> derekh: would you like to just discuss your bugs stuff here? or later?
14:06:14 <derekh> this seems to have started today, https://bugs.launchpad.net/tripleo/+bug/1538127
14:06:15 <openstack> Launchpad bug 1538127 in tripleo "overcloud deploys timing out" [Critical,Triaged]
14:06:28 <derekh> dprince: either is fine by me
14:07:03 <derekh> on bug #1538127 I'm waiting from some logs to find the problem, will investigate after this call
14:07:05 <trown> fwiw we do not see this issue in RDOCI, I think we really need to work on getting you guys moved to a newer delorean pin
14:07:30 <trown> so that the tests in RDO are at least similar to the tests in tripleo
14:07:50 <dprince> derekh: okay, so no data yet on the timeout. we'll have to follow up on this in #tripleo as it blocks our CI
14:08:08 <trown> right now in tripleo we are really testing the ability to deploy liberty
14:08:16 <trown> with mitaka tripleo
14:08:17 <derekh> trown: yup, thanks a lot of the problems you been sorting out, we're closer now to moveing to a new repository https://review.openstack.org/#/c/229789/
14:08:50 <dprince> derekh: nice, you got one pass there :)
14:08:55 <trown> :)
14:09:22 <derekh> dprince: the ha test only failed the ping test, if this was last week we'd have switched already ;-)
14:09:26 <trown> which is one more pass than the other jobs are getting right now
14:09:34 <derekh> trown: yup
14:10:08 <derekh> ok, may aswell talk about it here
14:10:14 <derekh> critical alerts for bugs effecting ci
14:10:16 <dprince> derekh: yep, go for it
14:10:20 <derekh> Some time ago, I created (refreshed) a tripleo trello board, https://trello.com/tripleo
14:10:25 <derekh> For a while I've been using it to track almost everything I've been doing for upstream tripleo
14:10:30 <derekh> but nobody else did much, so I think I may aswell drop it
14:10:30 <derekh> the one thing I think has been good and worked is the alerts I generate from it when ci is broken
14:10:30 <derekh> I'm thinking maybe we keep that and maybe drive it from tripleo launchpad bugs that have the ci tag and are critical
14:10:30 <derekh> what do people think?
14:11:03 <trown> +1 to the alerts
14:11:25 <shardy> derekh: +1 personally I think launchpad is sufficient, although maintaining the alerts would be good
14:12:05 <dprince> derekh: +1
14:12:16 <derekh> so instead of opening a trello card (cause that hasn't been working I think), we insteand open a critical bug, with a psecial tag (maybe blocker or ci), then drive alerts from that
14:12:34 <derekh> dprince: shardy trown ok, I'll make the switch when I get a chance over the next day or 2
14:12:56 <derekh> dprince: done with that topic, unless there are objections/ better ideas///
14:12:59 <dprince> derekh: maybe just send an email out with information on the 'tag' we want to use for these things
14:13:07 <derekh> dprince: will do
14:13:13 <dprince> derekh: or update the CI wiki page too?
14:13:29 <derekh> dprince: CI wiki page?
14:13:39 <dprince> derekh: yeah, I think there is one
14:13:40 <trown> lol I thought the same
14:13:51 <trown> I have not seen this wiki
14:14:24 <dprince> derekh: I'm not finding it. Ignore me
14:14:30 <dprince> derekh: okay, wishlists?
14:14:40 <derekh> dprince: np, will have a look for it just incase
14:14:56 <derekh> wishlists, this came out of a discussion with some downstream tripleo consumers (QE), they would like to test new features as they get added to tripleo (or as soon as possible new features are added).
14:15:04 <derekh> I propose we would use launchoad wish list bugs for new features, to allow them to be tracked
14:15:18 <derekh> as with any other bug it would get automatically closed once the last patch is merged to add the feature.
14:15:23 <derekh> thoughts?
14:15:49 <dprince> small new features can be wishlists, sure
14:16:06 <shardy> derekh: In Heat, we've adopted the "spec-lite" approach outlined by glance:
14:16:09 <shardy> http://docs.openstack.org/developer/glance/contributing/blueprints.html#glance-spec-lite
14:16:13 <dprince> but larger major features (with specs) are blueprints right
14:16:21 <derekh> specs tell us we agree that something should be done but don't give us tracking to allow somebody know something has been finished
14:16:22 <shardy> basically it is tracking small features via launchpad, with a tag of spec-lite
14:16:29 <shardy> so I'm +1 on doing the same
14:17:28 <derekh> dprince: blueprints would work too, I just said wishlist bugs as they are more lightweight I've never seen any tripleo blueprints
14:17:30 <trown> are we using blueprints at all?
14:17:32 <dprince> spec light sounds fine to me, may require some occasional grooming to make the call on spec-light vs. blueprints
14:17:45 <dprince> trown: I do sometimes, yes. But not heavily
14:18:02 * dprince did for the swift artifact URL's stuff
14:18:03 <shardy> dprince: Yeah, that's part of the triage process, you set it invalid if you feel it must have a full spec
14:18:10 <trown> so maybe the proposal should be all RFEs tracked with blueprint or spec-light
14:18:42 <trown> I think the important part is having all features tracked
14:18:46 <shardy> trown: +1 that will make it *much* easier to define what we've delivered each release
14:19:40 <shardy> we may also want to consider adopting the automated release notes tools too, e.g reno
14:19:43 <shardy> http://docs.openstack.org/developer/reno/
14:19:50 <trown> ya, Ironic has been using that
14:19:59 <dprince> derekh: okay, positive feedback on both your ideas so proceed I think
14:20:12 <derekh> Either is fine with me, just learning about the spec lite process now, looks nice and light weight, then can be a full spec if needed
14:20:16 <shardy> Now we have stable branches, we'll want to be able to anounce all the new features and have accurate release notes
14:20:20 <derekh> dprince: cool, I'll summarize both to the list
14:21:17 <dprince> shardy: cool, I haven't spent time on looking at reno. Will try to have a closer look
14:21:50 <dprince> #topic CI
14:22:04 <dprince> other than our CI being broken ATM is there anything else here?
14:23:16 <derekh> I've been trying to look into some of our intermittent errors as time allows (I hadn't done this in a while), we need to get our success rate up a bit
14:23:44 <derekh> besides the current blocker, we now have what seems to be a lot of intermittent error
14:24:21 <dprince> derekh: okay, thanks for the update
14:24:24 <shardy> dprince: one thing I'm proposing to move tripleo.sh back to tripleo-ci : https://review.openstack.org/#/c/272210/
14:24:32 <trown> for the HA job, we probably need a beefier undercloud
14:24:32 <shardy> Need to get that passing, but just so folks are aware
14:24:38 <derekh> shardy: +1
14:24:57 <shardy> dprince: reason is it handles both branches, and we're wasting effort backporting everything in tripleo-common
14:24:57 <trown> I had all kinds of intermittent failures in RDOCI on the HA job before bumping the undercloud stats
14:24:57 <dprince> shardy: cool, thanks for pointing it out
14:25:31 <trown> I think the stack for the HA deploy is pretty resource intensive to create
14:25:47 <shardy> trown: Yeah, running without swap and not much RAM is bound to cause spurious failures
14:25:53 <shardy> we do have some swap now tho
14:26:10 <shardy> would be good to see if it is being used, as obviously it'll hurt performance if it is
14:26:19 <trown> for reference I use 12G RAM 4CPU undercloud in RDO
14:26:31 <dprince> shardy: true, but at some point we'll move on. Perhaps that just means we could branch tripleo-ci for the stable jobs though
14:26:38 <shardy> trown: how do your CI runtimes compare?
14:27:00 <shardy> dprince: I'd prefer if we made tripleo.sh handle all branches
14:27:06 <trown> shardy: I am doing something a bit different so it is apples to oranges, that is one of my topics for open discussion though
14:27:14 * derekh would love to just get to OVB already, then we can try all these things at a whim
14:27:39 <bnemec> derekh: What is blocking us right now?
14:27:40 <shardy> dprince: well, it already does, I mean keep it that way
14:27:50 <shardy> but we can debat that if/when the time comes ;)
14:27:53 <shardy> debate
14:28:02 <dprince> shardy: sure, sounds like a fine plan for now
14:28:11 <trown> +1 to moving tripleo.sh to tripleo-ci repo
14:28:21 * bnemec doesn't want to de-bat.  They eat mosquitoes ;-)
14:28:24 <trown> makes it clear it is a dev CI tool
14:28:30 <slagle> trown: +1
14:28:42 <derekh> bnemec: I really want to test it on 10+ compute nodes so that 1. I can test a reproducable deployment and 2. make sure its ok when things scall up to 100+ simultanious jobs
14:28:50 <slagle> i'm already seeing things i don't like with people using it as a non-dev tool :)
14:29:16 <derekh> bnemec: I don't want to risk taking down what we have only to find out we can't get it to work
14:29:33 <bnemec> slagle: That's because it's where we put all of the hacks to work around things we aren't fixing in other projects. :-/
14:29:44 <bnemec> derekh: Understood.  Maybe we need to see if we can schedule some time in the scale lab?
14:30:04 <gfidente> bnemec, right and because of that the official docs frequently doesn't work
14:30:24 <derekh> bnemec: yup, we tied at one stage, but could only gewt it one day a week, maybe we can push a bit harder
14:30:26 <bnemec> gfidente: Yeah, I've pretty much given up on the official docs.
14:30:44 <gfidente> bnemec, indeed, but that is what people will look at
14:31:05 <trown> gfidente: bnemec, I have an idea here that is the other topic I wanted to discuss in open discussion... hold tight :)
14:31:07 <gfidente> so I am thinking we should also avoid having hacks in tripleo.sh and deploy using tripleo.sh
14:31:39 <shardy> we used to run Heat tests direct from docs using scripts in the markup
14:31:47 <shardy> it would be possible to do the same I guess
14:31:56 <bnemec> derekh: Yeah, we need to do something.  Has downstream not made any progress with it?  I know they were setting up an environment too.
14:32:28 <derekh> bnemec: last I heard they were waiting on HW to be rack, I'll see if I can get an update
14:32:49 <bnemec> Bleh, hardware.
14:34:30 <dprince> okay, we skipped a topic so lets get it now
14:34:39 <dprince> #topic Projects releases or stable backports
14:35:09 <dprince> Have the backports finally slowed?
14:35:23 <shardy> dprince: there's still a large backlog because of the CI problems
14:35:52 <dprince> shardy: okay, but if we just moved over the tripleo.sh it would eliminate that issue?
14:35:56 <slagle> dprince: the backports will be quite hot and heavy once ipv6 starts landing on master
14:35:59 <marios> dprince: yeah more or less what we needed to do the initial rebase for downstream tht, though as shardy says the 'few remaining' have grown into list because ci
14:36:23 <shardy> dprince: No, that doesn't really fix anything other than making landing fixes a bit easier/quicker
14:36:31 <slagle> since ipv6, as well as any other ssl improvements, must be backported to liberty
14:36:56 <dprince> shardy: I see, so the backports are fixes to CI
14:37:23 <shardy> https://review.openstack.org/#/c/272194/ and https://review.openstack.org/#/c/272191/ are the main ones I'm aware of
14:37:49 <shardy> dprince: Yeah, we need to fix CI then there's a ton of stuff to recheck and review
14:38:01 <dprince> slagle: okay, thanks for the update
14:38:12 <shardy> same story for master I suppose, our review/merge rate is depressing there also
14:38:56 <shardy> hopefully we can work on improving that via getting CI more stable
14:38:59 <trown> side note here for cores, can we make sure that any thing that we +W has recent as in from the last day or so CI results?
14:39:01 <dprince> Yeah, I'm gonna say TripleO is struggling at best ATM... many reviews are blocked needing manual upgrade testing
14:39:21 <dprince> and features are blocked still because of the pain of backporting to stable liberty?
14:39:38 <trown> we do not run the functional tests in the gate pipeline, so things can definitely slip through with outdated CI results
14:39:39 <dprince> we seem to have upstream development backwards...
14:39:53 <shardy> dprince: I think backporting to stable is working fine, but CI reliablity is preventing us landing things in a reasonably timely fashion
14:40:22 <dprince> shardy: the backporting mechanism is fine. The fact that we are still backporting features halfway through the release is the problem :/
14:40:50 <slagle> it does feel a little silly that we are basically backporting 90% of patches
14:40:52 <dprince> shardy: until this stops it is going to really make it difficult to get in potential new features (composable roles, split stack) I think
14:41:03 <shardy> dprince: Yeah, I'm saying if it didn't take 3 months to merge to master and 2 months to merge to stable, we'd be in better shape
14:41:08 <slagle> the reality is that 90% of tripleo devs are working on liberty support still
14:41:31 <bnemec> Or Kilo
14:41:37 <slagle> bnemec: don't go there :)
14:41:43 <bnemec> :-)
14:41:51 <dprince> slagle: I realize this, and I'm trying to be mindful not to cause conflicts. But there is a cost...
14:41:57 <trown> is there some proposal to change that? otherwise I think we probably just have to take that as a given
14:42:17 <gfidente> maybe we'll have a chance to discuss this next week
14:42:38 <shardy> slagle: I'd be interested to discuss if there's an alternative
14:42:45 <slagle> i'd bring up backwards compatibility again, but i'm afraid i'd get run out of the room :)
14:43:05 <shardy> It is tough as so many features are getting backported, but we still can't really move forward on non-backportable changes on master
14:43:11 <shardy> as dprince was referring to
14:43:13 <slagle> exactly
14:43:22 <gfidente> not to mention actual bugs
14:43:31 <shardy> "90% of tripleo devs are working on liberty support"
14:43:36 <shardy> that is the actual problem :(
14:43:43 * dprince has stopped rebasing new code until it settles down
14:43:43 <slagle> i can't deny it
14:43:49 <derekh> at least its happening upstream now
14:44:07 <dprince> derekh: good point, thanks for being positive :)
14:44:33 <dprince> okay, so we've got one agenda topic left
14:44:36 <dprince> #topic Specs
14:44:42 <dprince> anything for Specs this week
14:46:20 <dprince> The TripleO API thread continues upstream. Probably not worth bringing up here but it seems to be the most debated topic we have ATM with regards to Specs
14:46:48 <trown> that has been good reading :)
14:46:52 <shardy> It is proving pretty contentious
14:47:52 <shardy> It'd be great if we can figure out a way to step back, take away some of the emotion and solve the actual problem as a community
14:48:02 <dprince> yes, it is
14:48:16 <shardy> it seems a bit like we've veered into a flamey discussion over implementation, vs the requirement
14:48:35 <dprince> FWIW the iterative feedback I've gotten from working w/ some of the UI guys on prototypes has been actually quite positive/constructive
14:49:35 <slagle> yes, that's valid, but then i'm concerned when i see reponses that we don't expect people to use the API on its own
14:50:16 <slagle> if we're just solving for the UI, then that is not what i was expecting
14:50:27 <slagle> so maybe a level set / reset on what we're doing is important
14:50:46 <tzumainn> I kinda think seeing a spec from the Mistral point-of-view is important
14:50:47 <shardy> slagle: FWIW, most of my arguments are based in expressly *not* wanting to deliver and support a "solving for the UI" layer
14:50:49 <dprince> slagle: people can use either API on its own. When I initially tried it I used the API via mistralclient for example. both solutions have APIs and they can be used outside of our python-tripleoclient UI tooling
14:51:05 <shardy> I really want it to be something more generally useful, given the overhead of maintaining it
14:51:51 <slagle> yes, the ability to use it either way is there
14:52:53 <dprince> tzumainn: I'll post a spec as soon as I can
14:53:12 <dprince> #topic open discussion
14:53:26 <dprince> trown: you had a think you wanted to mention...?
14:53:32 <dprince> thing
14:53:36 <trown> I have a couple :)
14:54:40 <trown> the first is that there is a testday for RDO Manager this Thurs/Fri, I know folks here are busy with lots of stuff, but anyone wanting to help out would be great
14:54:46 <trown> end of that topic
14:55:19 <trown> the other thing is more interesting
14:55:39 <trown> I started playing with generating docs from CI: https://review.gerrithub.io/#/c/261218/
14:56:10 <trown> that is a POC that really just templates out one bit of RDOCI, but what I wanted to present here was more the idea
14:56:33 <trown> of being able to have the same thing that runs in CI templated out to docs as simply as running `tox -e docs`
14:57:11 <derekh> trown: would we end up with tripleo.sh in the docs ?
14:57:15 <dprince> trown: the end goal being to view docs during the code review process?
14:57:32 <trown> dprince: the end goal being no CI drift from docs
14:58:05 <dprince> trown: I see, similar to how our devtest was self documenting...
14:58:05 <trown> derekh: I think we would need to template tripleo.sh similarly to what I do for RDOCI in that review
14:58:20 <derekh> trown: ack
14:58:36 <dprince> trown: cool, I like where you are going
14:58:46 <shardy> Interesting, similar to what we did for heat, only we generated a script by processing an rst file with the docs in
14:58:47 <trown> derekh: so tripleo.sh.j2 becomes a jinja template that can generate tripleo.sh for CI, as well as docs from the same template
14:59:36 <dprince> great, almost out of time this week. Thanks everyone
14:59:46 <trown> it is a pretty immature idea at this point, I just started hacking on it late last night, but wanted to see if there was interest
15:00:18 <dprince> trown: sounds cool to me
15:00:30 <dprince> #endmeeting