14:00:04 <EmilienM> #startmeeting tripleo
14:00:05 <openstack> Meeting started Tue Dec 20 14:00:04 2016 UTC and is due to finish in 60 minutes.  The chair is EmilienM. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:00:06 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
14:00:08 <openstack> The meeting name has been set to 'tripleo'
14:00:15 <EmilienM> #topic agenda
14:00:16 <EmilienM> * one off agenda items
14:00:18 <EmilienM> * bugs
14:00:20 <EmilienM> * Projects releases or stable backports
14:00:22 <EmilienM> * CI
14:00:24 <EmilienM> * Specs
14:00:26 <EmilienM> * open discussion
14:00:28 <EmilienM> hello, who is up today?
14:00:29 <d0ugal> o/
14:00:30 <beagles> o/
14:00:32 <pradk> o/
14:00:34 <mwhahaha> o/
14:00:34 <fultonj> o/
14:00:35 <marios> o/ hello
14:00:58 <mandre> o/
14:01:01 <ccamacho> hi! o/
14:01:06 <hrybacki> o/
14:01:16 <EmilienM> #topic review actions items from last week
14:01:25 <shardy> o/
14:01:30 <matbu> o/
14:01:48 <EmilienM> EmilienM to test tripleo with centos7.3 and see how it goes: we did it last week, it seems like our CI is now passing on centos 7.3
14:01:50 <dprince> hi
14:02:10 <EmilienM> EmilienM to send update on ML about releases ocata-2 and newton: done, mail sent.
14:02:17 <EmilienM> beagles propose policy edit for periodically abandoning old reviews
14:02:21 <EmilienM> HA & puppet experts to look https://review.openstack.org/#/c/408726/
14:02:41 <sshnaidm> o/
14:02:45 <EmilienM> sounds like https://review.openstack.org/#/c/408726/ is in bad shape and nobody reviewed it yet.
14:02:49 <EmilienM> we need to postpone this one
14:02:58 <EmilienM> #action HA & puppet experts to look https://review.openstack.org/#/c/408726/
14:02:59 * beagles will do that right after mtg so it doesn't slide to bottom of list again
14:03:08 <EmilienM> beagles: thanks
14:03:16 <EmilienM> EmilienM to start postponing bugs / blueprints to ocata-3 or pike-1: done
14:03:41 <EmilienM> please let me know if I moved your blueprint to pike-1 and you're not happy. We can still discuss about it, and change.
14:03:46 <EmilienM> CI folks to review https://review.openstack.org/#/q/topic:3-node and https://review.openstack.org/#/c/410131/
14:04:02 <EmilienM> https://review.openstack.org/#/c/410131/ is merged
14:04:21 <EmilienM> https://review.openstack.org/#/q/topic:3-node+status:open still have 3 patches
14:04:38 <panda> o/
14:04:50 <EmilienM> please review the 3 patches ^ so we can move forward with 3-nodes testing.
14:05:03 <EmilienM> we're done with actions, moving to off items
14:05:13 <EmilienM> #topic one off agenda items
14:05:20 <EmilienM> #link https://etherpad.openstack.org/p/tripleo-meeting-items
14:05:31 <EmilienM> there is no item this week, can we go ahead in the regular agenda?
14:06:25 <EmilienM> #topic bugs
14:07:23 <EmilienM> #link https://launchpad.net/tripleo/+milestone/ocata-3
14:08:02 <EmilienM> anyone has some specific bug to raise?
14:08:11 <trown> o/
14:08:31 <EmilienM> 18 Confirmed, 76 Triaged, 49 In Progress targetted for ocata-3
14:08:56 <trown> (that is me arriving late not raising my hand for a specific bug)
14:09:10 <EmilienM> at some points, we might need some triage on what we need to fix in ocata-3 and what can be postponed
14:09:25 <EmilienM> trown: welcome!
14:09:53 <beagles> qquestin about https://bugs.launchpad.net/tripleo/+bug/1609688
14:09:54 <openstack> Launchpad bug 1609688 in tripleo "CI: nonha jobs fails in introspection step because of mistral error" [Critical,Confirmed]
14:09:59 <EmilienM> sshnaidm, panda: could you help in triaging bugs related to CI please? some of them are open but I don't think we still hit all of them
14:10:08 <weshay> 0/
14:10:09 <beagles> is this a pretty isolated issue
14:10:11 <EmilienM> beagles: we fixed this one last night
14:10:11 <beagles> ?
14:10:13 <sshnaidm> EmilienM, sure
14:10:14 <beagles> ah okay
14:10:25 <shardy> EmilienM: perhaps we could have a bug squash day prior to ocata-3, as it'd be good to focus on landing some of those vs postpoining a lot to the RC weeks
14:10:26 <panda> EmilienM: ok
14:10:46 <beagles> I was hitting it in my env yesterday afternoon and noticed the alert tag had been punted, but missed the fix
14:11:08 <EmilienM> shardy: what an excellent idea :)
14:11:08 <shardy> I know folks will be busy with features, but perhaps we could focus on landing bugs just for one day, particularly if there are high priority ones we already know about
14:11:27 <EmilienM> beagles: update puppet-mistral / instack-undercloud / tht and you're fine
14:11:30 <trown> beagles: fix was three repo revert of fernet token stuff in tht/instack-undercloud/puppet-tripleo
14:11:34 <shardy> last cycle some folks consumed the m3 release as a preview of newton, and it didn't work out well because we released with a lot of bugs
14:11:35 <beagles> EmilienM: cool thanks
14:11:40 <beagles> trown++
14:11:45 <beagles> +1 on shardy's proposal
14:11:46 <trown> oh... s/puppet-tripleo/puppet-mistral
14:11:55 <beagles> trown: ack
14:12:11 <beagles> thx
14:12:52 <EmilienM> shardy: i'll organize this bug squash after the break and hopefully we have a a good number of folks focusing on bugs during a full day
14:13:07 <shardy> Sounds good, thanks EmilienM
14:13:18 <EmilienM> #action EmilienM to organize TripleO Bug Squash Day, that will happen after the break
14:14:03 <EmilienM> I'm not aware of any critical bug not WIP already that we need to discuss
14:14:48 <EmilienM> let's move forward (feel free to raise it if we missed something)
14:14:56 <EmilienM> #topic Projects releases or stable backports
14:15:07 <EmilienM> so 2 things:
14:15:23 <EmilienM> we released ocata-2 last week. Congrats to the team!
14:16:15 <EmilienM> I'll repeat the message sent on the mailing-list. The focus in ocata-3 should be 1) finish a feature started in ocata-1 or ocata-2 and 2) stabilization: bug fixing, CI coverage, etc
14:16:58 <EmilienM> end of january is end of ocata-3
14:17:16 <EmilienM> we have less than 4 weeks to finish the features that we want in tripleo
14:17:29 <EmilienM> that said, the second thing is, I proposed a new release for Newton
14:17:34 <EmilienM> #link https://review.openstack.org/#/c/412643/
14:17:48 <EmilienM> (-1 from jenkins because I'm waiting for a patch in tripleo-ui to land)
14:18:08 <marios> EmilienM: makes sense.. some things like the new hiera hook didn't appear in our osp10 setup so new newton release will hopefully fix that (it will end up in new osp10 builds too)
14:18:18 <EmilienM> do we have any blocker to release Newton? do you have a patch in progress that we need to wait for?
14:18:22 <marios> EmilienM: i mean makes sense for new newton release
14:18:39 <EmilienM> marios: good feedback, thanks!
14:19:18 <EmilienM> any feedback / question about release management?
14:19:45 <EmilienM> #topic CI
14:20:02 <EmilienM> so before we talk about our current CI work, I would like to thank everyone who helped hard last week in fixing TripleO CI
14:20:08 <EmilienM> as a result, we managed to promote the repos this morning
14:20:10 <gfidente> EmilienM marios sorry for going back a little but
14:20:17 <EmilienM> so again: thank yoy :-)
14:20:33 <gfidente> are you saying that with a new newton release we'll also switch to the new hiera hook?
14:20:39 <gfidente> (the Newton release
14:20:55 <EmilienM> gfidente: we are going to cut stable/newton with a new tag
14:20:58 <shardy> gfidente: No, we must not backport anything related to the hiera hook
14:21:04 <shardy> that will be part of the upgrade to ocata
14:21:07 <EmilienM> shardy: yep, exactly
14:21:21 <gfidente> shardy ack ok I was trying to understand marios' comment about the new hook
14:21:50 <shardy> marios: Yeah perhaps you can clarify, but that hook should never be added to newton builds
14:22:07 <marios> shardy: ah OK never mind then I thought we'd put that into newton
14:22:08 <shardy> I mean, the package may be available but we shouldn't require it until the ocata upgrade
14:22:15 <shardy> Ok, cool
14:22:33 <gfidente> delicate thing to upgrade
14:22:41 <EmilienM> thanks for clarifying :)
14:22:43 <gfidente> while probably transparent to end users
14:22:55 <marios> gfidente: well its currently being handled by removing the old data and installign the new hook on the upgrade init
14:23:13 <EmilienM> sshnaidm, weshay: do you mind if you could give an update on container CI job? what patch is blocking, etc... so we can dress a list of things to review
14:23:14 <shardy> yeah we might streamline that process but atm it's a pre-upgrade script
14:23:35 <dprince> to be clear we aren't planning on backporting the new optimized hiera hook into Newton are we? (I was not planning on this...)
14:23:37 <gfidente> ok thanks for clarifying and explaining
14:23:39 <weshay> EmilienM, we are good to go, I sent an update to the list.. w/ updated instructions and how to pull in the latest required patches
14:23:44 <EmilienM> dprince: not afik
14:24:01 <marios> dprince: yeah i thought we would but the discussion/clarification from shardy and now you is that we will not be backporting the new hiera hook
14:24:02 <dprince> okay, sounds good
14:24:08 <sshnaidm> EmilienM, weshay I'll start with a job then
14:24:11 <shardy> dprince: No, we're not, there's just been some confusion :)
14:25:03 <EmilienM> sshnaidm: is there any patch under review that would block this work?
14:25:20 <sshnaidm> weshay, ^
14:25:56 <EmilienM> let me ask again: what are blocking tripleo CI to have this job running?
14:26:04 <EmilienM> tripleo-ci patch? etc
14:26:06 <sshnaidm> EmilienM, I think it's https://review.openstack.org/#/c/400986/ https://review.openstack.org/#/c/393348/
14:26:23 <EmilienM> sshnaidm: those are oooq
14:26:28 <EmilienM> sshnaidm: my question is about tripleo-ci
14:27:04 <EmilienM> shardy, matbu: what is the progress on major upgrade job? I've seen some patches have been done to make it happen, how close are we?
14:27:16 <sshnaidm> weshay, are all tht patches merged?
14:27:28 <shardy> EmilienM: It's working fine locally, but matbu has been working through some remaining issues related to the environment variables in CI
14:27:43 <weshay> sshnaidm, no.. none of them
14:27:49 <matbu> EmilienM: we are close, i just get an issue related to the ci script generaly
14:27:49 <EmilienM> shardy: I guess we're close then :)
14:28:02 <sshnaidm> EmilienM, weshay so which patches are blocking us?
14:28:03 <matbu> EmilienM: https://review.openstack.org/413106
14:28:08 <matbu> shardy: ^
14:28:12 <EmilienM> weshay: could you please dress a list (using a common gerrit topic) of patches that we need?
14:28:18 <shardy> Yeah I think we're pretty close now but it would be great to get it landed this week, and potentially moved to the check queue
14:28:27 <matbu> I tried to explain the issue in the commit message, not sure if it's clear :)
14:28:41 <EmilienM> shardy: the first time we see experimental job passing, let's move it to check queue -nv (non voting)
14:28:47 <sshnaidm> EmilienM, I'd like to do it with oooq, it's more ready afaik, we can use non-merged patches there as workaround and have it running very soon
14:29:14 <EmilienM> matbu: you might want to rebase the upgrade patch on top of https://review.openstack.org/#/c/413106/
14:29:23 <EmilienM> matbu: they conflict and they can't be tested together.
14:29:28 <matbu> EmilienM: btw i think we actually never test the uc-upgrade in ci
14:29:47 <matbu> EmilienM: yep i will do
14:29:48 <shardy> matbu: thanks - that explains why we didn't catch the nova cells v2 thing in CI
14:29:57 <EmilienM> sshnaidm: yes, please create a job with oooq and let's have it in tripleo-ci
14:30:02 <shardy> it broke undercloud upgrades, and I couldn't work out why CI didn't break
14:30:07 <weshay> EmilienM, sshnaidm https://review.openstack.org/#/q/topic:oooq-containers
14:30:27 <matbu> shardy: ha that explain, cool to catch that
14:30:43 <EmilienM> matbu: oh, I remember slagle reporting this kind of feedback a while ago, too
14:30:59 <shardy> matbu: should we actually be populating deploy.env instead of removing the source of it?
14:31:22 <EmilienM> weshay, sshnaidm: where are the patches in tripleo-ci and project-config to have the job? it's WIP ?
14:31:28 <matbu> EmilienM: shardy actually, if no CI folks have strong objection (and if CI is green on the review) it's safe enough to merged it quickly
14:31:30 <slagle> EmilienM: we have a job for that now, but it is nv
14:31:35 <sshnaidm> EmilienM, job already exists
14:31:44 <matbu> shardy: idk the use of sourcing it
14:31:55 <EmilienM> slagle: right, gate-tripleo-ci-centos-7-undercloud-upgrades-nv
14:31:55 <sshnaidm> EmilienM, I mean in project-config, it already runs
14:32:03 <EmilienM> matbu: were you saying gate-tripleo-ci-centos-7-undercloud-upgrades-nv is not doing what we want?
14:32:09 <matbu> shardy: i think if tripleo-ci expert could give their advice on it
14:32:22 <sshnaidm> EmilienM, I don't have a patch still, waiting on merged in oooq side..
14:32:25 <matbu> EmilienM: yes it's doing minor upgrade (newton to newton)
14:32:37 <EmilienM> sshnaidm: and on tripleo-ci repo side? it's wip on your side?
14:32:40 <slagle> it's not doing a minor upgrade
14:32:59 <slagle> it does a major upgrade, by first installing the previous major release
14:33:09 <sshnaidm> EmilienM, yep, in WIP
14:33:13 <matbu> slagle: i think so
14:33:14 <EmilienM> sshnaidm: ok, good to know. Please keep us updates as soon as oyu have something
14:33:26 <sshnaidm> EmilienM, sure
14:33:42 <matbu> slagle: i have put an example in the review: http://logs.openstack.org/31/404831/32/check/gate-tripleo-ci-centos-7-undercloud-upgrades-nv/9f20ac6/logs/var/log/undercloud_upgrade.txt.gz#_2016-12-15_15_47_40_000
14:34:33 <matbu> slagle: in the log, the stable release is still set to newton and it configured the newton repo
14:34:33 <slagle> matbu: since the patch is against master, it starts by installing newton
14:34:52 <slagle> right, so it tests from newton to master, which is a major upgrade
14:35:27 <matbu> slagle: yes, but in toci_* the stable release to master minus 1 == newton, then the stable_release is dump in the deploy.env as newton and is never overriden
14:36:11 <matbu> slagle: in th e log ^ the delorean.repo should be master not newton
14:36:15 <matbu> smae
14:36:26 <matbu> same behavior with major overcloud upgrade
14:36:59 <EmilienM> so we want to take an action from here and discuss this topic offline?
14:37:18 <matbu> EmilienM: yep maybe
14:37:30 <EmilienM> #action matbu, slagle to figure undercloud major upgrade job status
14:37:37 <EmilienM> I'll let you figure that thing together ;-)
14:37:48 <matbu> yep thx
14:37:51 <EmilienM> #topic Specs
14:38:00 <EmilienM> https://review.openstack.org/#/q/project:openstack/tripleo-specs+status:open
14:38:17 <EmilienM> #link https://review.openstack.org/#/q/project:openstack/tripleo-specs+status:open
14:38:51 <EmilienM> some specs were waiting for +A, I want to give a last chance for them to merge
14:39:13 <EmilienM> the ones without review or with negative review, will be postponed to Pike.
14:40:20 <EmilienM> anything to discuss about specs?
14:41:16 <EmilienM> #topic open discussion
14:41:28 <EmilienM> I would like to talk about PTG a little, I forgot to add the item in the off items
14:41:44 <EmilienM> I'm about to bootstrap the discussion to prepare TripleO PTG sessions in Atlanta
14:41:56 <EmilienM> #action EmilienM sends an email to start preparing PTG
14:42:10 <shardy> EmilienM: that would be good, I was wondering about the PTG status
14:42:13 <EmilienM> one thing important to know is that we'll have a room between Wednesday and Friday included
14:42:17 <shardy> e.g how many sessions we have etc
14:42:28 <EmilienM> we'll have 3 days together
14:42:42 <EmilienM> monday / tuesday are horizontal sessions (infra, cross project, release, security, etc)
14:43:02 <marios> EmilienM: +1 shardy matbu i think an upgrades related one... iteration for P once we've got the composable /step based  and also how the upgrades + containers would work since that is coming too
14:43:26 <EmilienM> since some people will be traveling on Friday, as usual, it would make sense to make the most important sessions on Wednesday and Thursday and maybe do hack sessions on Friday
14:43:59 <EmilienM> so I'll send an etherpad
14:44:01 <gfidente> shardy is that a good forum to bring some code for the workflowexecution resource too?
14:44:07 <EmilienM> folks, please add your topics in the etherpad as soon as you can
14:44:22 <EmilienM> gfidente: it will be our regular design sessions
14:44:31 <EmilienM> gfidente: like we previously did at Summit.
14:44:34 <shardy> gfidente: Yes that seems a good place to discuss it - we can get Heat folks involved too if needed
14:44:52 <EmilienM> if needed, we can create heat/tripleo sessions
14:44:54 * shardy wasn't sure on the final consensus re the ML thread last week
14:44:54 <gfidente> EmilienM yep but this is sort of 'horizonal'
14:45:21 <gfidente> tripleo/heat
14:45:59 <shardy> marios: agreed - I think we need proper upgrade support for the container based solution from day1, and to figure out how to migrate from non-container to container based deployments
14:46:02 <EmilienM> gfidente: exactly, we'll arange a common session
14:46:05 <marios> EmilienM: just wanted to note we had an upgrades squad syncup last week... info at http://lists.openstack.org/pipermail/openstack-dev/2016-December/109022.html - may be useful for getting involved with reviews/testing for anyone interested
14:46:25 <shardy> marios: the current upgrade architecture doesn't consider that hybrid situation at all, so we'll need to think about it some more
14:46:26 <gfidente> I'll try to get cschwede involved too so we have a use case and possible some working code
14:46:30 <EmilienM> marios: excellent, and nice summary.
14:46:53 <EmilienM> marios, matbu: what is the plan for the pacemaker ansible module? Time is running out and we need to think where we put this code.
14:46:58 <EmilienM> matbu: have you made progress on that^?
14:47:20 <marios> EmilienM: in the worst case we will carry it in the tht (it is two py files) and for current testing we are even delivering/installing during the upgrade init
14:47:22 <EmilienM> maybe we can organize some swift/tripleo heat/tripleo mistral/tripleo ironic/tripleo sessions
14:47:31 <EmilienM> if you like the idea, I can't contact the PTLs and find a common slot
14:47:37 <EmilienM> I can*
14:47:53 <matbu> marios: yep thx (you are taping too fast :))
14:48:11 <marios> matbu: i *love* my new keyboard :)
14:48:29 <matbu> hehe
14:49:10 <EmilienM> feel free to bring more topics or ask any question. Otherwise I'll close the meeting in 1 min
14:49:50 <beagles> /me is now curious about marios' keyboard
14:50:26 <EmilienM> matbu: I'll ping you after meeting to know, you haven't answered.
14:50:27 <matbu> i may have one question regarding tripleo-quickstart
14:50:49 <matbu> EmilienM: about the module ? (feel free sure)
14:51:09 <EmilienM> matbu: it would be nice to know the status of this thing, again, time is running out very fast and I still see this code in THT
14:51:32 <matbu> EmilienM: yep as marios said, the plan for now is to keep it in tht
14:51:57 <EmilienM> matbu: we can discuss it on #tripleo after meeting. What was your question?
14:52:28 <matbu> regarding the review in oooq, afaik the tripleo cores is cores also on oooq
14:53:02 <trown> yep
14:53:03 <matbu> but it looks like none of them is reviewing oooq
14:53:21 <EmilienM> none? trown does iirc
14:53:27 <trown> juan does quite a few reviews too
14:53:40 <EmilienM> I think it doesn't depend of whether or not you can +2, but also on your area of expertise.
14:53:49 <matbu> i have some reviews that is taking so long for being reviewed
14:54:12 <matbu> EmilienM: yep sure, but that's why i think there is little bottleneck here
14:54:13 <marios> i am not reviewing oooq i can promise to start
14:54:16 <trown> matbu: first I have heard about them
14:54:23 <marios> or rarely anyway
14:54:37 <slagle> i dont think that is a problem specific to oooq, but all our projects really
14:54:40 <EmilienM> matbu: when you're blocked, feel free to send an email to openstack-dev
14:55:03 <EmilienM> slagle: do you think we miss reviewers in all projects?
14:55:17 <matbu> EmilienM: ack
14:55:28 <slagle> EmilienM: i'm not sure what you're asking
14:55:32 * matbu should be much noisy :)
14:55:51 <slagle> i was saying that we have reviews that in all of our projects that take a long time to land b/c they are not reviewed
14:55:57 <trown> matbu: ya squeeky wheel gets the grease
14:56:21 <EmilienM> slagle: my question was, do we miss core reviewers in tripleo?
14:57:12 <slagle> i'm not sure. what do you think?
14:57:21 <shardy> EmilienM: yes, but we've been trying to grow the team, with some success over the last couple of cycles
14:57:29 <matbu> trown: hehe nice expression :)
14:57:41 <EmilienM> shardy: exactly
14:57:52 <EmilienM> we can't force people to review code in Gerrit
14:58:15 <EmilienM> what we can do is maybe write some documentation to welcome new reviewers
14:58:22 <d0ugal> well, we can, but the reviews get worse if they are forced :)
14:58:23 <shardy> Having work tracked and prioritized against milestones helps, as it can prevent patches from getting completely lost
14:58:27 <EmilienM> and document our best practices
14:58:35 <EmilienM> shardy: also!
14:59:02 <EmilienM> slagle, matbu: but your concern is 100% valid - we need to address it.
14:59:17 <EmilienM> we have 50s left, we might want to take it offline
14:59:54 <slagle> i miss all core and non-core reviewers!
15:00:01 <trown> hehe
15:00:07 <EmilienM> #endmeeting