19:01:35 <SpamapS> #startmeeting tripleo
19:01:36 <openstack> Meeting started Tue Mar 11 19:01:35 2014 UTC and is due to finish in 60 minutes.  The chair is SpamapS. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:01:37 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:01:39 <openstack> The meeting name has been set to 'tripleo'
19:02:05 <SpamapS> #link https://wiki.openstack.org/wiki/Meetings/TripleO
19:02:10 <lifeless> SpamapS: I am here, but thanks!
19:02:20 <SpamapS> lifeless: well you're late and quiet. ;)
19:02:27 <SpamapS> I sense a ruse.. ;)
19:02:32 <SpamapS> #topic bugs
19:02:45 <tchaypo> morning
19:02:45 <SpamapS> #link https://bugs.launchpad.net/tripleo/
19:02:45 <SpamapS> #link https://bugs.launchpad.net/diskimage-builder/
19:02:45 <SpamapS> #link https://bugs.launchpad.net/os-refresh-config
19:02:45 <SpamapS> #link https://bugs.launchpad.net/os-apply-config
19:02:45 <SpamapS> #link https://bugs.launchpad.net/os-collect-config
19:02:48 <SpamapS> #link https://bugs.launchpad.net/tuskar
19:02:50 <SpamapS> #link https://bugs.launchpad.net/python-tuskarclient
19:03:34 <SpamapS> we are _swimming_ in criticals
19:04:01 <SpamapS> https://bugs.launchpad.net/tripleo/+bug/1290969 in particular has broken our CI
19:04:20 <SpamapS> revert was submitted upstream, but we're also dealing with it by listing the stores we need
19:04:31 <SpamapS> fix is winding through CI now
19:06:33 <SpamapS> There are so many, I'm not sure which ones to call out.
19:06:42 <slagle> why not +A the tripleo fix?
19:06:46 <slagle> (for the glance one)
19:06:56 <SpamapS> slagle: Waiting for it to pass. :)
19:07:07 <slagle> looks like it hit some other issue
19:07:09 <slagle> unrelated
19:08:31 <lifeless> yay breaking changes
19:08:40 <slagle> http://logs.openstack.org/31/79631/4/check-tripleo/check-tripleo-overcloud-precise/d2ff806/console.html
19:08:47 <slagle> not sure if anyone has seen that one before ^^
19:09:04 <slagle> "Authentication required" after creating endpoints, failed the overcloud job
19:09:12 <lifeless> oh *fun*
19:09:21 <slagle> so, now that's blocking the fix for glance
19:09:56 * slagle files
19:10:00 <lifeless> pragmatism vs fear interact in my head :)
19:10:06 <SpamapS> moar criticals
19:10:52 <SpamapS> suffice to say, we are a bit under water right now with openstack bugs breaking us as much as anything else
19:11:23 <SpamapS> in theory "the process is working" .. as we're feeding fixes into OpenStack.. achieving the goal that a deployment system built on OpenStack will improve OpenStack. :-P
19:11:26 <SpamapS> but it burnssss ussssss
19:11:49 <jdob> is that common this late in a cycle or a side effect of us having our feet under us better?
19:11:51 <slagle> we just need to convince everyone else :)
19:12:08 <lifeless> jdob: feature freeze  generally breaks the world
19:12:18 <jdob> sounds familiar
19:12:21 <lifeless> jdob: I hate it with a passion, but the only folk that understand are deployers.
19:12:32 <lifeless> everyone else seetakes the view that this is a good thing
19:12:49 <lifeless> I'm now focused on getting in the gate and helping them understand in a more visceral way
19:13:28 <SpamapS> right so, bugs there are many
19:13:34 <SpamapS> Triage seems to be happening more now than it was
19:13:37 <SpamapS> moving on?
19:14:06 <lifeless> DO ALL HDo all the criticals have volunteers against them ?
19:15:11 * SpamapS wishes launchpad would tell him that without opening them all :-P
19:15:33 <lifeless> SpamapS: turn on the assignee column via the cog.
19:15:44 <SpamapS> https://bugs.launchpad.net/tripleo/+bug/1270646  -  no
19:15:52 <greghaynes> https://bugs.launchpad.net/tripleo/+bug/1285269 no
19:15:58 <SpamapS> lifeless: DOHHHHH
19:16:07 <SpamapS> ok so you can all do that
19:16:35 <SpamapS> Looks like none of the triaged/confirmed ones have assignees
19:16:47 <SpamapS> so, if you are twiddling your thumbs in #tripleo.. -> grab a critical today
19:17:23 <lifeless> slagle: that authentication required is super odd
19:17:33 <lifeless> slagle: since its in the middle of setup-endpoints
19:18:45 <SpamapS> I suspect something crashed
19:18:52 <SpamapS> ENEEDLOGS BADLY
19:18:53 <slagle> likely
19:19:41 <lifeless> SpamapS: bug 1290759
19:20:35 <SpamapS> lifeless: there are two bugs there
19:20:45 <SpamapS> lifeless: one is log levels.. the other is actually shipping the logs back to jenkins
19:20:50 <SpamapS> or logstash.. or something
19:20:58 <lifeless> SpamapS: we get the logs for seed and undercloud atm
19:21:14 <SpamapS> because AFAIK, we don't have the logs from the overcloud or undercloud when something like init-keystone fails.
19:21:22 <SpamapS> oh we have undercloud now? that I missed.
19:21:31 <lifeless> SpamapS: check toci :)
19:23:03 <SpamapS> 2014-03-11 18:52:11.226 2774 WARNING keystone.common.wsgi [-] You are not authorized to perform the requested action, identity:revocation_list.
19:23:07 <SpamapS> 2014-03-11 18:52:11.697 2774 WARNING keystone.common.wsgi [-] You are not authorized to perform the requested action, identity:revocation_list.
19:23:10 <SpamapS> I bet we need a new policy.json
19:23:41 <SpamapS> anyway, should we move on or are there other bugs people want to discuss?
19:24:08 <lifeless> onwards and upwards
19:25:50 <tchaypo> and twirling
19:27:47 <bnemec> ENOSPAMAPS
19:27:57 <pblaho> o/
19:28:07 <SpamapS> doh
19:28:14 <SpamapS> my IRC host just went down
19:28:31 <lifeless> what did it go down on?
19:29:02 * SpamapS has no snark for that
19:29:14 <SpamapS> Anyway, I may have missed the "yes lets move on"
19:30:30 <SpamapS> aand we're back
19:30:37 <SpamapS> #topic reviews
19:31:00 <SpamapS> #link http://russellbryant.net/openstack-stats/tripleo-openreviews.html
19:31:18 <SpamapS> #link http://russellbryant.net/openstack-stats/tripleo-reviewers-30.txt
19:31:19 <marios> they dont work any more
19:31:24 <SpamapS> #link http://russellbryant.net/openstack-stats/tripleo-reviewers-90.txt
19:31:26 <lifeless> still down
19:31:29 <SpamapS> oh
19:31:32 <SpamapS> lovely
19:31:33 <jistr> seems like Russel took that down
19:31:36 <slagle> russel must have about a million hits on that blog post by now
19:31:39 <SpamapS> there's a different one
19:31:44 <jistr> slagle: lol yes
19:31:46 <lifeless> it was OIOOOmin his VM
19:31:57 <lifeless> ooming
19:32:13 <russellb> yes
19:32:25 <russellb> and i've been too busy with feature freeze / icehouse to deal with it
19:32:27 <russellb> so i just pulled it
19:32:32 <russellb> and would rather bring it back up in -infra
19:32:50 <bnemec> Oh, that's probably what killed my VM the other day.
19:33:20 <ccrouch> fwiw
19:33:20 <ccrouch> http://www.stackalytics.com/report/reviews/tripleo-group/open
19:33:20 <ccrouch> http://www.stackalytics.com/report/contribution/tripleo-group/30
19:33:38 <SpamapS> mikal: didn't you have a review stat thing too?
19:34:08 <SpamapS> so, our stats are going to be weird as far as landing changes right now
19:34:13 <SpamapS> because CI is like, always broken
19:34:51 <jistr> ccrouch: neat. i know about stackalytics but i missed these charts
19:35:34 <marios> indeed, thanks charles
19:36:23 <SpamapS> so in summary..
19:36:27 <SpamapS> 1) Keep reviewing!
19:36:34 <SpamapS> 2) Don't +A without CI passing!
19:36:44 <SpamapS> 3) Keep calm and recheck no bug
19:36:57 <SpamapS> zomg. new t-shirt.
19:37:22 <lifeless> so, I think that derekh is probably right about https://review.openstack.org/#/c/79502/ being key to stability
19:37:47 <lifeless> sp	I like
19:37:56 <lifeless> bah
19:38:04 <lifeless> SpamapS: I like
19:38:25 <SpamapS> that and "wow. such recheck. much no bug. very SUCCESS."
19:39:09 <SpamapS> anyway.. on a more serious note.. sdague was just reminding me that it will take a long long time to get tripleo into the gate
19:39:15 <SpamapS> and we have some really big hurdles..
19:39:27 <SpamapS> the biggest one is probably that we really need to start using ironic.
19:39:47 <ccrouch> surely that cant be the *biggest* hurdle
19:39:55 <SpamapS> ccrouch: it is huge
19:40:03 <SpamapS> nova baremetal is effectively deprecated
19:40:14 <SpamapS> and ironic is not graduated yet
19:40:20 <SpamapS> and we're not allowed to have not-graduated openstack things in the gate
19:40:28 <ccrouch> i guess I meant "biggest" == "most difficult for us to achieve"
19:40:46 <SpamapS> there are some real challenges
19:40:51 <lifeless> nova bm isn't deprecated, thats waiting for Ironic to incubate
19:40:53 <SpamapS> to switching ot Ironic I mean.
19:41:01 <SpamapS> lifeless: s/incubate/graduate/
19:41:06 <SpamapS> oh
19:41:07 <SpamapS> same diff
19:41:08 <SpamapS> n/m
19:41:13 <devananda> afaict, it's going to be a source of pain for tripleo for all of the J cycle
19:41:16 <lifeless> SpamapS: not the same, and I was being precise :)
19:41:37 <lifeless> SpamapS: but possibly wrong :)
19:41:41 <ccrouch> can i we make being in the gate a target for J ?
19:41:46 <SpamapS> lifeless: if we don't switch to Ironic now, we can get in the gate sooner you think?
19:41:47 <ccrouch> s/i//
19:41:51 <devananda> SpamapS: lifeless is correct. russellb has expclitly _not_ frozen the baremetal driver
19:41:56 <tchaypo> lifeless: definitely being saucy
19:42:24 <lifeless> SpamapS: our being in the gate is now primarily limited by tripleo-test-cloud scale
19:42:34 <devananda> ccrouch: it can be a target for K, aiui, but not J, because ironic will not be graduated until then
19:42:37 <devananda> lifeless: ^
19:42:42 <SpamapS> lifeless: yes, but if we added ironic, we'd also be limited by Ironic's status.
19:42:59 <SpamapS> hang on let me switch topics
19:43:01 <lifeless> SpamapS: for which, when the patches to make ci-overcloud regions be fully straightforward to deploy, I think we need to go hit up the folk that offered hardware and have a serious talk
19:43:03 <SpamapS> #topic Ironic and being in the gate
19:43:19 <devananda> shall i summarize?
19:43:28 <ccrouch> devananda: please
19:43:42 <devananda> integrated projects can't gate on non-integrated projects
19:43:44 <lifeless> SpamapS: Ironic's status is chicken and egg, and if we use Ironic, and are in the gate, we deliver all the criteria AFAIK, subject to Nova merging the patch.
19:43:53 <devananda> ironic is not integrated (and probably wont be until teh start of the K cycle)
19:44:06 <devananda> thus, if tripleo depends on ironic, it can't be in teh gate until ironic can be in the gate
19:44:11 <devananda> ergo, not in the J cycle
19:44:25 <lifeless> devananda: they can once the project is in incubation, because the incubation cycle is the cycle where the symmetric gating is setup.
19:44:30 <devananda> no
19:44:32 <lifeless> no?
19:44:33 <SpamapS> so this seems like something where we need to get aligned and moving forward _now_
19:44:37 <devananda> lifeless: this is a point ive continually run into
19:44:45 <devananda> lifeless: the TC and -infra teams have made it VER Yclear to me
19:44:47 <bnemec> This seems like a serious flaw in our incubation process.
19:44:50 <lifeless> sdague: ^ help me understand please
19:44:58 <devananda> ironic needs to remain ASYMMTETRRICALLY gated until AFTER graduation
19:45:24 <devananda> which means for all of J, ironic's gate is likely to break based on changes in nova, glance, keystone, neutron, etc ....
19:45:31 <lifeless> devananda: or presumably consume releases and treat Ironic like every other third party external dependency?
19:45:48 <devananda> lifeless: that's a possibility as well
19:45:48 <lifeless> except that the link to Nova internal API is a huge issue there.
19:45:52 <devananda> right
19:46:24 <devananda> bnemec: i agree
19:47:18 <SpamapS> So I think this is a really big, important conversation that we need to have, perhaps inviting the right people so we don't miss any nuances.
19:47:23 <ccrouch> apologies for missing this: but if we can't get into the gate with ironic until K, why cant we get into the gate using nova-bm?
19:47:27 <devananda> fwiw, i raised this on the list here: http://lists.openstack.org/pipermail/openstack-dev/2014-February/028459.html
19:47:44 <lifeless> ccrouch: we can
19:47:48 <devananda> where this == asymmetric gate pain
19:48:05 <SpamapS> But IMO, this is the biggest thing standing in the way of TripleO. If we're not in the gate... we have to run our own git mirrors and gate openstack ourselves or our velocity will drop to a ridiculously low level.
19:48:08 <lifeless> ccrouch: and since we want to support H for at least a cycle, we just need to use both nova-bm and Ironic, and not gate on Ironic.
19:48:17 <SpamapS> (which it already has...)
19:48:20 <lifeless> we can trivially run non-voting Ironic jobs
19:48:44 <ccrouch> lifeless: ok, great. So is that the intermediate goal, i.e. a goal for J ? To gate with nova-bm?
19:49:04 <devananda> lifeless: your goal is gate tripleo with nova-bm for J, and with ironic for K ?
19:49:22 <devananda> that is not what I thought your plan was
19:49:23 <lifeless> devananda: my goal is to get in the gate *anyway we can*
19:49:27 <devananda> ah
19:49:28 <devananda> heh
19:49:34 <ccrouch> lifeless: +10 :-)
19:49:34 <SpamapS> what he said. ;)
19:49:35 <lifeless> devananda: honestly, the baremetal backend used in nova is not the problem
19:49:38 <lifeless> as I said above
19:49:43 <devananda> sure
19:49:45 <lifeless> our problem is test capacity
19:49:48 <SpamapS> What I'd like to get confirmation on ..
19:49:53 <lifeless> any other discussion is solvable via code
19:49:54 <devananda> it's just a feature-limitation for tripleo itself -- not a gate limitation
19:49:54 <lifeless> that one isn't
19:49:58 <SpamapS> is whether we can switch to Ironic and still remain in the gate. :-P
19:50:11 <lifeless> SpamapS: Can I reframe that?
19:50:12 <devananda> lifeless: so if you want features taht ironic has AND you want tripleo in the gate
19:50:20 <devananda> lifeless: i think there's a conflict of interest in deprecating nova-bm
19:50:34 <SpamapS> lifeless: yes, but I do suspect there is a rabbit in your hat. ;)
19:50:43 <lifeless> devananda: go on
19:50:56 <devananda> lifeless: i thought that was pretty clear
19:51:43 * lifeless raises both hands palms up with a quizzical expression on his face
19:52:04 <devananda> hehe
19:52:05 <SpamapS> ~o~
19:52:06 <devananda> ok, i'll rephrase
19:52:15 <ccrouch> devananda: is saying we cant have both: if we move to ironic, we get kicked out of the gate
19:52:19 <ccrouch> iiuc
19:52:24 <devananda> aiui, tripleo needs certain features taht nova-bm does not have
19:52:27 <SpamapS> so let's not speculate
19:52:33 <SpamapS> Can we pose this quiestion to infra?
19:52:37 <devananda> some are inherently in ironic, some need to be implemented
19:52:39 <SpamapS> question too
19:52:43 <devananda> SpamapS: ++
19:52:48 <lifeless> so
19:52:53 <lifeless> I still don't understand 'move to Ironic'
19:52:58 <lifeless> thats like saying 'move to glusterfs'
19:53:02 <devananda> erm
19:53:07 <ccrouch> sigh
19:53:12 * devananda is confused
19:53:15 <ccrouch> you know what i mean :-)
19:53:18 <lifeless> we support nova-bm today
19:53:26 <devananda> lifeless: i thought we talked about "move to ironic" last week extensively
19:53:43 <devananda> perhaps not as extensively as other things
19:53:45 <lifeless> devananda: I deeply desire the ability to say that TripleO can a) deploy using and b) deploy, Ironic.
19:54:05 <devananda> sure
19:54:08 <ccrouch> i think he's drawing the distinction about tripleo would support both nova-bm and ironic
19:54:13 <lifeless> devananda: neither a) nor b) imply that TripleO cannot c) deploy using and d) deploy, Nova-BM
19:54:20 <devananda> indeed
19:54:37 <lifeless> Symmetric gating may be limited to making statements about c) and d)
19:55:00 <lifeless> but that in no way implies that a) and b) will not work, nor that we cannot do check jobs for them.
19:55:05 <devananda> lifeless: so my question becomes, does it matter to tripleo if the features which ironic and nova-bm provide are substantially different?
19:55:28 <devananda> for a case in point, the need for ephemeral (which, yes, is now in both)
19:55:40 <devananda> and then, wil tripleo be able to support back-porting those changes to nova-bm
19:55:52 <devananda> and hwo does nova feel about that
19:55:59 <lifeless> devananda: so there are threes ways Ironic might differ; it might be better (more secure/faster/leaner/more robust)
19:56:11 <lifeless> it might have additional features (e.g. UEFI)
19:56:23 <lifeless> it might have less features (e.g. missing nova rebuild --preserve-ephemeral)
19:56:46 * devananda sighs
19:56:47 <lifeless> In my mind for TripleO we have a subset of Nova-BM features (perhaps all of them minus Tilera :))
19:57:02 <lifeless> those are what we have to have to say that a) above works
19:57:12 <lifeless> but there's nothing holding Ironic back
19:57:16 <devananda> lifeless: when tripleo needs a given feature, and that feature is not in nova-bm, please, do not add it
19:57:26 <devananda> unless you guys can also add it to ironic
19:57:35 <devananda> first
19:57:37 <devananda> :)
19:57:52 <SpamapS> right so I want to summarize and then we're out of time..
19:58:17 <devananda> the more feature-ful that nova-bm becomes, the harder the testing and upgrade path becomes, and the more i get concerned taht our community fragments the way n-net and neutron did
19:58:35 <SpamapS> - We are likely to find ourselves testing with nova-baremetal and ironic for all or at least most of the J cycle.
19:58:48 <lifeless> devananda: so lets get Ironic in the gate. I just don't see how that impacts getting TripleO in the gate - a,c are distinct from c,d.
19:58:52 <SpamapS> - We should exercise rigor and not expand nova-baremetal anymore so as to not prolong that any further.
19:59:10 <devananda> lifeless: agreed. except ironic can't get in the gate until K, for political reasons
19:59:16 <devananda> or s/politic/policy/
19:59:19 <devananda> depending on your POV
19:59:19 <SpamapS> - There is a question as to whether TripleO's gate jobs can make use of an incubated but not integrated project, like Ironic, that needs answering.
19:59:28 <lifeless> devananda: if you're incubated then nova-bm is deprecated
19:59:38 <lifeless> devananda: so J should see us right there.
19:59:41 <devananda> lifeless: no. we are incubated. nova-bm is not deprecated.
20:00:07 <lifeless> up, next meeting time
20:00:17 <SpamapS> Alright, fun times
20:00:24 <SpamapS> #endmeeting