19:00:59 <lifeless> #startmeeting tripleo
19:01:00 <rpodolyaka1> o/
19:01:01 <openstack> Meeting started Tue Jan 21 19:00:59 2014 UTC and is due to finish in 60 minutes.  The chair is lifeless. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:01:02 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:01:03 <jprovazn> hi
19:01:04 <openstack> The meeting name has been set to 'tripleo'
19:01:06 <slagle> hello
19:01:10 <greghaynes> hi
19:01:11 <cody-somerville> Hiya
19:01:20 <lifeless> #topic agenda
19:01:27 <lifeless> bugs
19:01:28 <lifeless> reviews
19:01:28 <lifeless> Projects needing releases
19:01:28 <lifeless> CD Cloud status
19:01:28 <lifeless> CI virtualized testing progress
19:01:30 <lifeless> Insert one-off agenda items here
19:01:32 <lifeless> open discussion
19:01:35 <lifeless> #topic bugs
19:01:39 <lifeless> #link https://bugs.launchpad.net/tripleo/
19:01:40 <lifeless> #link https://bugs.launchpad.net/diskimage-builder/
19:01:40 <lifeless> #link https://bugs.launchpad.net/os-refresh-config
19:01:40 <lifeless> #link https://bugs.launchpad.net/os-apply-config
19:01:42 <lifeless> #link https://bugs.launchpad.net/os-collect-config
19:01:44 <lifeless> #link https://bugs.launchpad.net/tuskar
19:01:47 <lifeless> #link https://bugs.launchpad.net/tuskar-ui
19:01:49 <lifeless> #link https://bugs.launchpad.net/python-tuskarclient
19:02:02 <dkehn> hi
19:02:05 <jtomasek> hi
19:02:18 <tzumainn> hiya
19:02:25 <jcoufal-mobile> Heya! o/
19:02:36 <shadower_> hey
19:02:40 <lifeless> so bugs
19:02:48 <lifeless> we've got untriaged bugs second week in a row :(
19:02:56 <lifeless> https://bugs.launchpad.net/diskimage-builder/+bug/1268678
19:03:33 <SpamapS> we've also got a non-working devtest right now.
19:03:47 <lifeless> ruh roh
19:03:48 <SpamapS> https://bugs.launchpad.net/heat/+bug/1271190
19:03:53 <SpamapS> breaking all Heat in-instance users
19:04:00 <SpamapS> the patch to fix has been approved
19:04:38 <SpamapS> Not sure if it is worth marking as affecting TripleO since it will be resolved in Heat within a few hours.
19:05:01 <lifeless> I am not a big fan of paperwork for paperwork sake
19:05:03 <lifeless> so meh
19:05:07 <SpamapS> I suppose knowing what patch to cherry-pick while it wends its way through the gate might be helpful
19:05:09 <lifeless> we have criticals already
19:05:11 <lifeless> https://bugs.launchpad.net/tripleo/+bug/1270646
19:05:22 <lifeless> this breaks tenants
19:05:29 <jomara_> ahoy
19:05:37 <lifeless> 2kbps if you're lucky
19:06:17 <lifeless> anyone with deep network experience feel like owning that and chasing it down to ground?
19:06:38 <SpamapS> I think I am the opposite of that person. :)
19:07:54 <lsmola> hehe
19:08:03 <lifeless> ok, so no volunteers; will take it to the list
19:08:14 <lifeless> https://bugs.launchpad.net/tripleo/+bug/1266513
19:08:22 <lifeless> SpamapS: you're assigned on this, whats the status ?
19:09:02 <SpamapS> lifeless: I have been badgering the upstreams to host on pypi... netaddr responded immediately, the others did not.
19:09:31 <SpamapS> lifeless: We can of course do the exception thing for pip. I dropped it late last week hoping to give the upstreams more time.
19:09:46 <SpamapS> I think there are two.. netifaces may be abandoned.. and I forget the other one.
19:09:56 <lifeless> SpamapS: so AIUI it's critical because our builds fail right?
19:11:00 <SpamapS> lifeless: I think invididual projects all adapted .. not sure how.
19:11:12 <SpamapS> Just never revisited to lower it to High.
19:11:52 <lifeless> SpamapS: since we don't seem to have any actual symptoms at the moment, I think High is better
19:11:53 <bnemec> Didn't the other projects whitelist the externally hosted deps in tox.ini?
19:12:04 <SpamapS> lifeless: ah, we're not installing any of the ones that need netifaces
19:12:11 <lifeless> SpamapS: either way :_
19:12:18 <SpamapS> and we don't install review-day so lazr.restful is not needed
19:12:21 <lifeless> SpamapS: sounds like its either not a bug for us
19:12:26 <lifeless> SpamapS: or not hitting us
19:12:32 <SpamapS> lifeless: I think I may close it.
19:12:35 <lifeless> SpamapS: I'd F-R it for us
19:12:51 <SpamapS> i want to verify that swift is not affected somehow and we're just not testing swift
19:12:52 <lifeless> ok https://bugs.launchpad.net/tripleo/+bug/1254246 is also assigned to SpamapS
19:13:06 <rpodolyaka1> this can be probably be closed too
19:13:17 <rpodolyaka1> as I already said it's kind of fixed for us
19:13:29 <rpodolyaka1> and we aren't hitting this, I believe
19:13:44 <lifeless> rpodolyaka1: you were going to do some validation or something last week?
19:13:46 <rpodolyaka1> though, there is 'proper fix' waiting on review
19:14:02 <rpodolyaka1> lifeless: yeah, it works for me
19:14:21 <lifeless> so, f-r for tripleo?
19:14:28 <rpodolyaka1> I think, so
19:14:34 <SpamapS> Yeah agreed
19:14:41 <SpamapS> Though it may also just be masked now.
19:14:42 <rpodolyaka1> will continue to track this in neutron
19:14:55 <lifeless> ok
19:15:23 <lifeless> what about https://bugs.launchpad.net/tripleo/+bug/1254555
19:15:33 <lifeless> I'm fairly sure we still have a hack in our tree
19:17:13 <rpodolyaka1> hasn't it been removed?
19:17:34 <rpodolyaka1> I thought, it was here https://github.com/openstack/tripleo-image-elements/blob/master/elements/tripleo-cd/tripleo-cd.sh
19:17:53 <lifeless> 38ac22ce7c4ea4ee9a86e1b6f2c3e5e31c8ba652
19:17:57 <lifeless> in incubator
19:19:29 <lifeless> Date:   Thu Dec 19 10:36:48 2013 +0000
19:19:29 <lifeless> Merge "Revert "Work around neutron floatingip race condition""
19:19:32 <lifeless> ok, so its gone
19:19:39 <lifeless> we can f-r that bug too
19:20:26 <lifeless> ok so that gives us two criticals, one with a fix landing in heat, one that may go all the way to ovs
19:22:04 <lifeless> #action lifeless to seek owner for network bug on list
19:22:14 <lifeless> #topic reviews
19:22:19 <lifeless> #undo
19:22:20 <openstack> Removing item from minutes: <ircmeeting.items.Topic object at 0x3a10d10>
19:22:29 <lifeless> heh, undo fail
19:22:58 <lifeless> anyhow, I was going to say - please do do one day of triage, collectively we can make sure all the projects are triaged easily
19:23:05 <lifeless> #topic reviews
19:23:16 <lifeless> http://russellbryant.net/openstack-stats/tripleo-openreviews.html
19:23:23 <lifeless> 
19:23:23 <lifeless> Stats since the last revision without -1 or -2 :
19:23:28 <lifeless> 3rd quartile wait time: 0 days, 21 hours, 40 minutes
19:23:39 <lifeless> still in ok shape - good stuff
19:23:58 <lifeless> however
19:24:43 <lifeless> 
19:24:43 <lifeless> Longest waiting reviews (based on oldest rev without -1 or -2):
19:24:43 <lifeless> 12 days, 4 hours, 1 minutes https://review.openstack.org/65699 (WIP: Make rabbitmq-server keep state in /mnt/state)
19:24:46 <lifeless> 8 days, 2 hours, 42 minutes https://review.openstack.org/52045 (Using merge.py from tuskar to generate overcloud.yaml)
19:24:49 <lifeless> 4 days, 17 hours, 28 minutes https://review.openstack.org/67348 (Add SSL configuration metadata for overclouds.)
19:24:52 <lifeless> 4 days, 16 hours, 47 minutes https://review.openstack.org/67356 (Permit deploying an overcloud with SSL endpoints.)
19:24:55 <lifeless> 4 days, 13 hours, 26 minutes https://review.openstack.org/67358 (Use OS::Heat::UpdateWaitConditionHandle)
19:24:58 <lifeless> sorry for the size
19:25:00 <lifeless> this is a collective fail
19:25:15 <SpamapS> this means we're not starting at oldest first maybe?
19:25:23 <lifeless> I don't know
19:25:27 <lifeless> but if you look at those reviews
19:25:34 <lifeless> they haven't been -1 or -2'd
19:25:42 <lifeless> they've been sitting for days
19:25:59 <slagle> i just reviewed the ssl stuff this morning
19:26:01 <slagle> i wanted to test it
19:26:24 <lsmola> lifeless, I will review the merge.pz, though I think it is still work in progress
19:26:32 <slagle> although, somehow i missed 67348
19:26:35 <lifeless> lsmola: if its work in progress, it should be marked WIP
19:26:45 <lsmola> lifeless, I will ask marios
19:26:48 <slagle> i don't think this reviewable reviews query is very accurate. b/c that one is not in my list
19:26:51 <lifeless> This isn't an individual responsibility issue
19:27:01 <lifeless> slagle: what reviewable reviews query ?
19:27:04 <marios> lsmola: lifeless: it *is* still wip in the sense that the models/controllers are yet tbd in tuskar
19:27:17 <marios> lifeless: lsmola : but -1 with appropriate comments are very very welcome at this stage
19:27:26 <marios> lifeless: lsmola: hence i removed the wip
19:27:30 <lifeless> marios: if its not ready for review yet; please mark it WIP, otherwise folk will be trying to review it to get it landed
19:27:34 <slagle> lifeless: very last link on https://wiki.openstack.org/wiki/TripleO#Review_team
19:27:36 <lifeless> marios: ack
19:27:56 <lsmola> marios, ok
19:28:37 <slagle> i don't typically use it, unless there's lots of reviews. which there have been last few days
19:29:09 <lifeless> slagle: indeed, that review is missing!
19:29:23 <slagle> weird
19:29:27 <slagle> tripleo-heat-templates is in the query
19:29:30 <lifeless> ah
19:29:34 <lifeless> remove is:reviewable
19:29:39 <lifeless> that means 'passed jenkins'
19:29:49 <lifeless> but jenkins was on holiday
19:30:14 <lifeless> slagle: I'm inclined to suggest we remove that from the wiki page link
19:30:14 <slagle> right, but then you just get all reviews, including ones you've already reviewed
19:30:26 <slagle> whereas is:reviewable removes one you've already done
19:30:46 <slagle> and your own changes for that matter
19:30:51 <lifeless> slagle: does it put them back once someone else comments ?
19:31:07 <slagle> i don't know
19:31:16 <SpamapS> slagle: sounds like the baby went out with the bathwater there..
19:31:21 <slagle> it seems to do several things. not all of them well
19:31:30 <SpamapS> slagle: I'd rather work around ones I've done and my own than miss stuff. :-P
19:31:38 <slagle> so, yes, i'm with you. let's just remove from teh wiki
19:32:01 <SpamapS> FTR I just use gerrit and round-robin among the various tripleo projects.
19:32:18 <lifeless> ok, so this isn't about individual reviewers; we have a large team
19:32:22 <pblaho> good evening
19:32:40 <lifeless> but we've consistently missed 5 reviews for 2 whole days (ignoring the weekend)
19:32:59 <lifeless> has everyone else been using the unified link as well?
19:33:21 <rpodolyaka1> me
19:33:22 <jdob> i use the second to last one, the All one
19:33:29 <jdob> not the Reviewable one
19:33:48 <lifeless> jdob: ok so you'd see everything
19:34:05 <jdob> i do see my own and things I've commented on, so assuming it aggregates right with no special sauce, then yes
19:34:30 <lifeless> yeah
19:34:55 <lifeless> so I don't think 'work harder' will fix this, we've something systemic thats happened; assuming everyone is doing some daily review
19:35:03 <lifeless> I've been a bit slack this week - sorry
19:35:53 <lifeless> so - experiment, can folk please try to review the *oldest by submission date* first ?
19:36:08 <rpodolyaka1> makes sense
19:36:13 <slagle> ack
19:36:16 <lifeless> hmm, but I can't see how to get that from the unified review
19:36:20 <lifeless> so this might be a wishlist
19:36:25 <jdob> i normally tend to check the newest, so i'll start going in reverse
19:36:25 <lifeless> shall we move on?
19:36:54 <lifeless> well, oldest on that page will be least update - which for a change w/out updates will also be oldest submitted
19:37:04 <lifeless> so yeah, oldest is probably an ok metric
19:37:15 <lifeless> trivial rebases will inherit -1's
19:37:35 <jdob> er, the ones at the top of the list is what I meant by newest, not sure if that meant by submission or state
19:37:50 <lifeless> so it should be pretty fast to open and check there is no replies etc
19:38:01 <lifeless> jdob: thats the newest by last-modification
19:38:06 <lifeless> reviewing counts as a modification
19:38:22 <lifeless> ok, moving on
19:38:25 <lifeless> #topic  Projects needing releases
19:38:43 <lifeless> AFAIK all our projects are in reasonable shape - the issue right now is just heat, right?
19:39:01 <rpodolyaka1> seems so
19:39:14 <marios> lifeless: tuskar is obviously still in flux, working out models/controllers etc
19:39:22 <lifeless> do we have a release volunteer this week? excluding tuskar* ?
19:39:24 <marios> lifeless: but no blockers atm
19:39:34 <lifeless> marios: or do you want a release cut?
19:39:45 <marios> lifeless: i don't know what we'de release (no)
19:39:52 <marios> imo
19:39:56 <jdob> i second that no
19:40:01 <lifeless> ok, so excluding tuskar*
19:40:16 <rpodolyaka1> lifeless: I can take it for this week, but will need a rain check for the next two weeks
19:40:23 <lifeless> rpodolyaka1: ok!
19:41:04 <lifeless> #topic  CD Cloud status
19:41:15 <jdob> marios: see #tripleo
19:41:27 <lifeless> HP region - generally good shape, though we've got a number of DC tickets open at the moment
19:41:51 <lifeless> undercloud is happy; cd-overcloud is I believe paused while we put the finishing touches on stateful upgrades
19:42:08 <SpamapS> lifeless: I don't actually recall why we paused most recently
19:42:11 <lifeless> ci-overcloud is happy
19:42:21 <SpamapS> ovs maybe?
19:42:35 <lifeless> SpamapS: no, not sure
19:42:45 <lifeless> lets make sure there's no local patches and restart it
19:43:02 <lifeless> in incubator/templates/elements
19:43:13 <lifeless> dprince: hows the RH region ?
19:43:16 <SpamapS> lifeless: ok, I need to get some Heat retry-on-update patches in shape in the review queue and then I will see if we can resurrect it.
19:43:32 <lifeless> SpamapS: do we have the waitcondition fix we needed ?
19:43:53 <lifeless> SpamapS: if you say 'thats waiting for review' I will lol.
19:43:53 <SpamapS> lifeless: it is awaiting review in Heat. We cherry picked it into the undercloud's heat.
19:44:04 <lifeless> SpamapS: ok, cool
19:44:11 <SpamapS> https://review.openstack.org/#/c/63245/
19:44:14 <SpamapS> lol away :)
19:44:17 <lifeless> lol
19:44:31 <SpamapS> Heat review team seems overwhelmed with i2 surge.
19:44:36 <lifeless> we'll come back to dprince I think, time is short
19:44:54 <lifeless> #topic  CI virtualized testing progress
19:45:02 <lifeless> pleia2: you is up
19:45:48 * dprince tunes in
19:46:19 <pleia2> this morning I went through the etherpad of our most recent todo list from last week and crossed off what I know was done
19:46:25 <pleia2> #link https://etherpad.openstack.org/p/tripleo-test-cluster
19:46:57 <pleia2> we might want to add the ongoing review nodepool work to that (but it was a holiday here yesterday, so I'm still catching up)
19:47:12 <lifeless> the node pool patches we need are merged
19:47:17 <lifeless> that was just network support
19:47:27 <lifeless> the big set thats open is trying to help infra with overload
19:47:48 <pleia2> ah, gotcha
19:48:18 <lifeless> whats next for ci virt testing ?
19:49:06 <pleia2> I think derekh would have a better view of that
19:49:38 <lifeless> ENODEREK
19:49:43 <lifeless> ok
19:49:54 <lifeless> dprince: RH region of tripleo-cloud, status update?
19:50:36 <dprince> lifeless: They are disconnecting ports from our internal network to prep for making it public.
19:50:44 <lifeless> \o/
19:50:46 <pleia2> sweet
19:50:54 <dprince> lifeless: in prep for this they downed the ports on the switches
19:51:00 <dprince> So I've lots access :(
19:51:09 * dprince is sad not having access
19:51:13 <lifeless> progress is great news
19:51:24 <lifeless> ok
19:51:28 <dprince> Still waiting on the IPv4 request too though
19:51:38 <lifeless> #topic open discussion
19:51:46 <pleia2> dprince: shall we have our weekly syncup tomorrow?
19:52:07 <dprince> pleia2: yes. I think they are still productive for everyone
19:52:15 <pleia2> great
19:52:33 <slagle> i'd like to discuss the stable branches for our projects
19:52:45 <slagle> sounds like we got somewhat of a general concensus on the mailing list discussion
19:53:00 <dprince> pleia2: although I myself feel like I'm struggling to make progress sometimes :-|
19:53:04 <slagle> i don't honestly know if there is enough time to do something by I2 which is the 23rd
19:53:24 <slagle> but, i plan to start making progress towards it
19:53:26 <slagle> sound ok?
19:53:33 <lifeless> slagle: I'd like some time to talk package install stuff with you too :) maybe after my run of meetings today
19:53:46 <ekarlso> Q: is there a easy way to setup a testing env for tuskar ?
19:54:04 <lsmola> ekarlso, devtest
19:54:17 <SpamapS> Hm
19:54:23 <ekarlso> lsmola: ?
19:54:25 <SpamapS> we don't currently include horizon/tuskar in devtest do we?
19:54:26 <ekarlso> devstack ?
19:54:40 <SpamapS> ekarlso: http://docs.openstack.org/developer/tripleo-incubator/devtest.html
19:54:52 <slagle> lifeless: sure, just ping me at some point. i'm on EST time (utc - 5), but am often on in the evenings
19:55:00 <lsmola> SpamapS, though it easy to install it in undercloud
19:55:01 <lifeless> slagle: cool
19:55:03 <shadower> ekarlso: devtest is a way of setting up tripleo for development/testing
19:55:06 <bnemec> There is a horizon element for use in devtest.
19:55:20 <ekarlso> does that do tuskar ? :)
19:55:20 <lifeless> so stable branches - yeah, I think if someone's going to do it, they should do it in the tent ;)
19:55:46 <shadower> ekarlso: you'll need to add it to the undercloud image elements
19:55:53 <shadower> not sure if we have the UI elementized yet though
19:55:54 <marios> ekarlso: you'll need to setup tuskar once you've got devtest and an undercloud running
19:56:02 <slagle> lifeless: the tent?
19:56:14 <shadower> marios: iirc the tuskar element's in, the UI maybe not
19:56:21 <marios> shadower: ic
19:56:29 <shadower> marios: https://github.com/openstack/tripleo-image-elements/tree/master/elements/tuskar
19:56:31 <lifeless> slagle: oh, there's a saying do you want the camel outside the tent pissing in, or inside the tent pissing out
19:56:43 <lifeless> slagle: its rather vivid :)
19:56:53 <slagle> lol. i think i get it :)
19:56:56 <lifeless> slagle: and I'm not actually comparing you to a camel :)
19:57:16 <lifeless> plus
19:57:17 <ccrouch1> i had no idea it was LBJ: http://izquotes.com/quote/241192
19:57:19 <jcoufal-mobile> shadower: we should have a look on that
19:57:22 <lifeless> on incubator specifically
19:57:45 <lifeless> I don't want stable releases being done to be blocked on, or force premature matriculation, of incubator contents
19:57:51 <shadower> ekarlso: so before the step 1. here: http://docs.openstack.org/developer/tripleo-incubator/devtest_undercloud.html you'd do `export UNDERCLOUD_DIB_EXTRA_ARGS=tuskar`
19:58:08 <lifeless> It's not that I *want* incubator released, but if you're taking a snapshot, it is what it is
19:58:26 <shadower> ekarlso: once it's launched, you'd have to ssh in and install the tuskar UI manually (for now)
19:58:41 <slagle> right, well the first thing i was going to do was to get the docs updated to say "it is what it is" and set expectations
19:58:47 <lsmola> ekarlso, install devtest then this https://github.com/openstack/tuskar-ui/blob/master/docs/install.rst
19:58:58 <lifeless> slagle: cool
19:59:13 <lifeless> ok folks, we're out of time. -> #tripleo for more
19:59:15 <lifeless> #endmeeting