19:00:49 <slagle> #startmeeting tripleo
19:00:49 <openstack> Meeting started Tue Sep  8 19:00:49 2015 UTC and is due to finish in 60 minutes.  The chair is slagle. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:00:50 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:00:53 <openstack> The meeting name has been set to 'tripleo'
19:00:55 <slagle> i'm sure someone will hate me for that
19:01:04 <derekh> phew I thought I was late
19:01:16 <akrivoka> hiya
19:01:17 <jdob> derekh: you were, you missed the entire first meeting
19:01:30 <slagle> yea, we're on to the 2nd meeting already
19:01:48 <slagle> #topic agenda
19:02:07 <slagle> I've added several one off items to the agenda
19:02:10 <derekh> jdob: slagle: caught up on the minutes
19:02:20 <slagle> lol
19:02:33 <jdob> which, oddly enough, describe a meeting that wasn't even a minute long
19:03:07 <slagle> let's fly through the regular agenda, then hit the one off stuff at the end, if there are no objections
19:03:22 <slagle> * bugs
19:03:22 <slagle> * reviews
19:03:22 <slagle> * Projects needing releases
19:03:22 <slagle> * CI
19:03:22 <slagle> * Specs
19:03:24 <slagle> * open discussion
19:03:32 <slagle> * one off items
19:03:53 <slagle> #topic bugs
19:04:14 <slagle> there are lots of untriaged bugs that i'll plan to go through this week
19:04:16 <dprince> slagle: we've been hitting a bug or two a day recently
19:04:40 <dprince> slagle: related to CI failures, a very bad time... feature freeze in OpenStack is
19:05:02 <slagle> dprince: which are the ones that people should be focusing on?
19:05:24 <derekh> Yup, we hit at least 6 or 7 bugs/regressions last week that have killed the ci jobs
19:05:51 <dprince> so someone needs to look at the recent Horizon issue: https://bugs.launchpad.net/tripleo/+bug/1492416
19:05:52 <openstack> Launchpad bug 1492416 in tripleo "Apache fails to start, .HorizonScssFilter: command not found" [High,Triaged]
19:06:18 <dprince> the top issue IMO is this one though https://bugs.launchpad.net/tripleo/+bug/1492028
19:06:19 <openstack> Launchpad bug 1492028 in tripleo "puppet overcloud deployment timeouts" [High,In progress] - Assigned to Dan Prince (dan-prince)
19:06:45 <dprince> We might need Steve bakers help this one
19:06:56 <derekh> The ceph jobs is unreliable and the HA jobs doesn't pass, part of me wants to just ignore those and move to the instack based jobs first
19:06:59 <dprince> lots of CI failures over the last few weeks related to this
19:07:04 <derekh> then see if anything needs to be dealt with
19:07:28 <dprince> derekh: do you have a cherry pick for this yet https://bugs.launchpad.net/tripleo/+bug/1493442
19:07:29 <openstack> Launchpad bug 1493442 in puppet-glance "puppet-glance: No such file or directory - /etc/glance/glance-api.conf" [High,In progress] - Assigned to Dan Prince (dan-prince)
19:07:32 <slagle> for https://bugs.launchpad.net/tripleo/+bug/1492028, has anyone been able to repro locally?
19:07:33 <openstack> Launchpad bug 1492028 in tripleo "puppet overcloud deployment timeouts" [High,In progress] - Assigned to Dan Prince (dan-prince)
19:07:52 <dprince> slagle: I did reproduce it once
19:08:02 <dprince> slagle: but I didn't have DEBUG enabled so it was a fail
19:08:18 <dprince> slagle: We do however have DEBUG CI logs so that should help somewhat
19:08:26 <derekh> dprince: nope, my test failed and I was going to retry but I've just noticed tripleo ci jobs aren't running for some reason,
19:08:39 <dprince> slagle: I have a feeling it is race related... perhaps a DB race condition in heat or something
19:08:56 <derekh> slagle: I've tried about 8 or 9 runs in the CI rack and havn't reproduced
19:09:07 <dprince> derekh: yeah, a Delorean failure now :)
19:09:17 <derekh> dprince: ya ?
19:09:33 <dprince> slagle: to bring this up to a higher level, we probably need more help on getting CI stable
19:09:52 <dprince> slagle: because issues are almost coming in faster than we fix them
19:10:08 <slagle> yea, agreed
19:10:41 <slagle> all i can do is ask around
19:10:44 <slagle> by ask, i mean beg
19:11:15 <dprince> derekh: yes, I saw one weird delorean failure. I'll post you a link
19:11:20 <derekh> dprince:  this ? 2015-09-08 18:16:34.347 | FATAL: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected termination of the channel
19:11:33 <slagle> derekh: is the instack job  testing with master of the puppet modules?
19:11:44 <dprince> derekh: yeah, that was it
19:11:47 <slagle> just wondering if we need the devtest puppet jobs at all anymore. dprince?
19:11:54 <derekh> dprince: that happens when jenkins looses connection to the instance
19:12:09 <derekh> slagle: yes it is
19:12:12 <slagle> that would help give folks one thing to focus on, vs. switching between the 2 to try and help
19:12:17 <derekh> slagle: the plan is to remove those jobs
19:12:31 <derekh> slagle: https://review.openstack.org/#/c/220088/
19:12:33 <derekh> dprince: ^
19:12:35 <dprince> yeah, can probably consolidate
19:12:41 <derekh> that moves the ceph job
19:13:08 <derekh> and passed earlier today, I was working on the ha job next but got hit by failures
19:13:26 <slagle> ok
19:13:55 <dprince> derekh: I'm going to push a DIB_REPOREF_puppet_glance fix for the one issue
19:13:58 <slagle> #info need more people to help out with CI failures
19:14:00 <derekh> dprince: I *think* those errors you mention with delorean are actually a problem with jenkins or our cloud
19:14:18 <slagle> #link https://bugs.launchpad.net/tripleo/+bug/1493442
19:14:19 <openstack> Launchpad bug 1493442 in puppet-glance "puppet-glance: No such file or directory - /etc/glance/glance-api.conf" [High,In progress] - Assigned to Dan Prince (dan-prince)
19:14:28 <slagle> #link https://bugs.launchpad.net/tripleo/+bug/1492416
19:14:28 <openstack> Launchpad bug 1492416 in tripleo "Apache fails to start, .HorizonScssFilter: command not found" [High,Triaged]
19:14:35 <slagle> #link https://bugs.launchpad.net/tripleo/+bug/1492028
19:14:37 <openstack> Launchpad bug 1492028 in tripleo "puppet overcloud deployment timeouts" [High,In progress] - Assigned to Dan Prince (dan-prince)
19:14:49 <dprince> one more bug I just hit
19:14:52 <derekh> dprince: ok, we need to talk to infra aswell to see why we currently only processing one job at a time
19:14:54 <dprince> #link https://bugs.launchpad.net/tripleo/+bug/1493502
19:14:55 <openstack> Launchpad bug 1493502 in tripleo "Failed to validate: resources.StorageMgmtNetwork: Value must be a string" [High,In progress] - Assigned to Dan Prince (dan-prince)
19:14:58 <derekh> dprince: look at the check-tripleo queue
19:15:16 <slagle> did the storagemgt patch merge?
19:15:28 <dprince> slagle: appoligies if you wanted a quick meeting, we've got plenty to talk about :)
19:16:05 <dprince> derekh: ack on the queue
19:16:21 <slagle> dprince: i'll look at https://bugs.launchpad.net/tripleo/+bug/1492416
19:16:22 <openstack> Launchpad bug 1492416 in tripleo "Apache fails to start, .HorizonScssFilter: command not found" [High,Triaged] - Assigned to James Slagle (james-slagle)
19:17:42 <slagle> alright, so are the 4 highest priority bugs?
19:17:47 <slagle> *those
19:18:25 <dprince> slagle: yeah, and without a Horizon fix we may well have to just disable Horizon if we need to bump the DELOREAN_URL
19:19:44 <bnemec> Looks like the Horizon backport of the fix is already proposed with 1 +2 though.
19:19:58 <bnemec> Might just be a question of poking the right person to get that merged.
19:20:57 <dprince> bnemec: nice, can you link that into the bug?
19:21:06 * dprince doesn't follow Horizon to closely
19:21:18 <bnemec> dprince: Yeah, it's in the related bz.  One second.
19:21:47 <slagle> dprince: yea if we need to disable in the interim, fine by me
19:21:54 <dprince> slagle: did we talk about releases yet?
19:22:11 <david-lyle> dprince, bnemec: I can try to track down another +2, but any stable core should be able to approve, you have the Horizon +2
19:22:28 * david-lyle just gets pinged on horizon
19:22:33 <slagle> dprince: no. it's on the agenda
19:22:37 <dprince> david-lyle: great, thanks
19:23:06 <dprince> slagle: so briefly might I request a release of os-cloud-config, and the new tripleo client?
19:23:21 <dprince> slagle: only because using those in a venv is occasionally useful
19:23:42 <derekh> dprince: do we need to go through a renaming exercies of the tripleo client first ?
19:24:04 <slagle> dprince: let's talk about that when we get to releases
19:24:22 <dprince> derekh: http://git.openstack.org/cgit/openstack/python-tripleoclient/tree/
19:24:33 <dprince> derekh: we do need to finish some work, yes
19:24:37 <dprince> slagle: sure, I'll be quiet
19:24:38 <slagle> is there any other bug business?
19:25:00 <dprince> not from me
19:25:02 <slagle> #topic reviews
19:25:20 <slagle> dprince: i'm sure you want to ask for reviews :)
19:25:48 <dprince> slagle: always :), but seriously I probably have rebasing to do first
19:26:16 <slagle> there are also going to be (i suspect) a small flood of reviews to get things fixed up in the new repos that got moved to git.openstack.org
19:26:27 <dprince> slagle: seriously there is some os-net-config stuff that could land though
19:26:39 <dprince> slagle: that would help the static ctlplane patch series along...
19:26:41 <slagle> so if folks could pay attention to that as well, it will help unblock stuff
19:26:51 <slagle> dprince: can you link them here?
19:27:04 <derekh> also  we abandon a bunch of reviews for the stuff we no longer ci test ? mainly bash element stuff but that can come once things settle down a bit
19:27:10 <dprince> #link https://review.openstack.org/#/c/213351/
19:27:12 <derekh> *we can
19:27:27 <dprince> slagle: we should try to fast track the stuff for new projects I think
19:27:48 <dprince> slagle: meaning nothing other than review them as top priority
19:28:00 <slagle> dprince: yea, that's what i was thinking, there's no CI running on them currently
19:28:22 <slagle> speaking of which, derekh, can you have a look at https://review.openstack.org/221457
19:28:37 <slagle> was trying to add the CI jobs to instack*/python-tripleoclient
19:29:01 <derekh> slagle: looking
19:29:23 <derekh> slagle: gate-tripleo-ironic-overcloud-f21-nonha is currently running on them
19:29:34 <bnemec> slagle: https://review.openstack.org/#/c/220167/
19:29:37 <derekh> slagle: but ya, we should add the others
19:29:49 <slagle> oh is it? maybe it's just b/c the queue was stuck or something
19:30:03 <bnemec> Oh, that's a different one.
19:30:42 <derekh> slagle: yup, queue is stuck, dunno why only noticed befor this meeting
19:31:19 <derekh> slagle: we should merge you partch anyways, will review this evening
19:31:30 <slagle> derekh: i think it's fine to abandon the old reviews in combination with the deprecation of the bash elements
19:31:44 <derekh> slagle: ok
19:31:53 <slagle> which we should be able to do shortly
19:32:23 <slagle> anything else on reviews?
19:32:47 <slagle> #topic releases
19:33:16 <slagle> we can release all the things as needed
19:33:30 <slagle> i'm happy to do it
19:33:31 <dprince> sure, os-cloud-config would be helpful to release
19:33:54 <dprince> the "capabilities" stuff is helpful in a venv (to me at least)
19:33:57 <slagle> #info release python-tripleoclient after merge of https://review.openstack.org/#/c/221320/
19:34:23 <slagle> #action slagle to do releases
19:34:48 <slagle> #topic CI
19:35:18 <dprince> nice work derekh on the instack jobs
19:35:20 <slagle> #info CI now running an instack based  job
19:35:30 <slagle> yes, thanks to derekh :)
19:35:46 <derekh> thanks
19:35:59 <derekh> we should stop pulling the old version of instack and friends
19:36:00 <derekh> https://review.openstack.org/#/c/221474/
19:36:11 <derekh> that pulls them from the openstack donain
19:36:33 <derekh> we can remove these altogether once this merges https://review.openstack.org/#/c/220551/
19:36:44 <dprince> derekh: when CI passes you can send it :)
19:36:52 <derekh> dprince: ack
19:37:01 <dprince> derekh: or I will if I think to check it later
19:37:14 <derekh> Adding them to the projects list means we don't have to manually do it
19:37:46 <slagle> and you posted a patch for the HA and ceph jobs as well right?
19:39:32 <dprince> slagle: here is the patch series for the HA and ceph jobs https://review.openstack.org/#/c/220088/
19:39:40 <slagle> cool
19:39:42 <derekh> slagle: yup, the ceph job passed earlier, the HA then failed on the glance regression
19:40:07 <slagle> #help CI needs more volunteers to help out
19:40:18 <derekh> slagle: those gets us off devtest altogether, we can then drop one of the nonha jobs
19:40:24 <bnemec> #link https://review.openstack.org/#/c/220088/
19:40:28 <derekh> slagle: and remove a bunch of code from toci
19:41:12 <slagle> ok
19:41:16 <slagle> any other CI business?
19:41:33 <slagle> #topic specs
19:41:35 <derekh> nothing from me
19:41:40 <akrivoka> we have a spec up for a tripleo overcloud deployment library
19:41:55 * bnemec needs to look at that
19:41:56 <akrivoka> the idea is to formalize the deployment workflow and build a library to support it
19:42:05 <akrivoka> I encourage everyone to take a look and review
19:42:10 <akrivoka> #link https://review.openstack.org/#/c/219754
19:42:13 <slagle> #link https://review.openstack.org/219754 spec for overcloud deployment library
19:42:17 <slagle> :)
19:42:19 <akrivoka> :)
19:43:06 <dprince> akrivoka: nice, is this simliar to tripleoclient... but in a different format?
19:43:30 <dprince> akrivoka: I will just read the spec and then leave comments there
19:43:46 <slagle> it's building on tripleo-common
19:43:49 <akrivoka> dprince: the idea is to pull the business logic out of the tripleo client and put it in this library so that it can be reused by different consumer (CLI, GUI)
19:43:59 <bnemec> It's all the stuff in tripleoclient that's not supposed to be in tripleoclient.
19:44:10 <dprince> akrivoka: perfect
19:44:12 <akrivoka> and more! :)
19:44:35 <slagle> akrivoka: thanks for bringing it up, i'll have another look this week
19:44:46 <akrivoka> awesome, thanks
19:44:49 <slagle> the spec is not that long, so everyone please have a look
19:45:34 <slagle> #topic open discussion / one off items
19:45:55 <slagle> #link https://etherpad.openstack.org/p/tripleo-mitaka-proposed-sessions etherpad for design summit planning
19:46:09 <slagle> please add your ideas/proposals there
19:46:46 <slagle> if we get more than proposals than sessions (that would be a good thing), we can ask folks to rank them
19:46:51 <dprince> slagle: but you've already added mine ideas
19:47:13 <slagle> dprince: i did
19:47:20 <slagle> i'm sure you have more
19:47:36 <slagle> i added some other things too :)
19:47:49 <dprince> cool, looks like a good start
19:48:07 <dprince> this docker one could be fun
19:48:27 <slagle> also note that the kolla folks have offered to spend some time with tripleo during their contributor meetup
19:48:53 <slagle> so, we should plan on that, maybe some cross pollination after lunch on friday or something
19:49:04 <slagle> i think their meetup will be the 2nd half of friday
19:49:13 <dprince> slagle: yeah, apparently their PTL is buying all the kolla devs drinks
19:49:26 <sdake_> apparently ;)
19:49:29 <derekh> dprince: ya, PTL's always do that
19:49:30 <dprince> we should do that!
19:49:30 <bnemec> *hint, hint*
19:50:04 <slagle> i'll buy any devs a drink
19:50:08 <slagle> there i said it
19:50:18 <slagle> i hope water is cheap in tokyo
19:50:30 <bnemec> lol
19:50:56 <derekh> Water (0.33 liter bottle)	113.15 ¥
19:51:21 <derekh> 113 Japanese Yen equals
19:51:21 <derekh> 0.9424 US Dollar
19:51:25 <derekh> you should be ok
19:51:32 <slagle> everything from "liter" and after is gibberish to me
19:51:58 <derekh> slagle: don't worry, give me your credit card and I'll figure out the hard stuff
19:52:27 <derekh> on another note, I'd like to start using trello again
19:52:40 <derekh> #link http://lists.openstack.org/pipermail/openstack-dev/2015-September/073861.html using trello
19:53:13 <derekh> read that ^^ I'm gonna move things around tomorrow on it and populate it and home others come join the party
19:53:38 <slagle> :), wfm
19:53:45 <slagle> i like the overall picture it gives
19:53:58 <dprince> derekh: I won't stop you
19:54:43 <dprince> derekh: that is as much encouragement as I could muster about this topic. Sorry :)
19:54:54 <derekh> I'm not saying we all need to keeps fine grained details up there but I think there is a happy medium beween that and nobody knowing what we are doing
19:55:05 <derekh> dprince: Thats for the effort
19:55:08 <derekh> *thanks
19:55:41 <slagle> yea, i think we can do better about letting folks know what we're working on
19:55:45 <slagle> well i know i can anyway
19:55:53 <slagle> even if it's just amongst oursevles
19:56:11 <slagle> but it's not a hold anyone to the fire "your work must be tracked in trello" or anything
19:56:31 <derekh> slagle: yup
19:57:17 <slagle> alright everyone, i appreciate the participation today :)
19:57:38 <dprince> rock on OoO
19:57:40 <slagle> if there's nothing else, i'll see you all in #tripleo
19:57:46 <akrivoka> \o
19:57:48 <jdob> \o
19:57:53 <derekh> o/
19:57:55 <slagle> #endmeeting