19:00:59 <lifeless> #startmeeting tripleo 19:01:00 <rpodolyaka1> o/ 19:01:01 <openstack> Meeting started Tue Jan 21 19:00:59 2014 UTC and is due to finish in 60 minutes. The chair is lifeless. Information about MeetBot at http://wiki.debian.org/MeetBot. 19:01:02 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 19:01:03 <jprovazn> hi 19:01:04 <openstack> The meeting name has been set to 'tripleo' 19:01:06 <slagle> hello 19:01:10 <greghaynes> hi 19:01:11 <cody-somerville> Hiya 19:01:20 <lifeless> #topic agenda 19:01:27 <lifeless> bugs 19:01:28 <lifeless> reviews 19:01:28 <lifeless> Projects needing releases 19:01:28 <lifeless> CD Cloud status 19:01:28 <lifeless> CI virtualized testing progress 19:01:30 <lifeless> Insert one-off agenda items here 19:01:32 <lifeless> open discussion 19:01:35 <lifeless> #topic bugs 19:01:39 <lifeless> #link https://bugs.launchpad.net/tripleo/ 19:01:40 <lifeless> #link https://bugs.launchpad.net/diskimage-builder/ 19:01:40 <lifeless> #link https://bugs.launchpad.net/os-refresh-config 19:01:40 <lifeless> #link https://bugs.launchpad.net/os-apply-config 19:01:42 <lifeless> #link https://bugs.launchpad.net/os-collect-config 19:01:44 <lifeless> #link https://bugs.launchpad.net/tuskar 19:01:47 <lifeless> #link https://bugs.launchpad.net/tuskar-ui 19:01:49 <lifeless> #link https://bugs.launchpad.net/python-tuskarclient 19:02:02 <dkehn> hi 19:02:05 <jtomasek> hi 19:02:18 <tzumainn> hiya 19:02:25 <jcoufal-mobile> Heya! o/ 19:02:36 <shadower_> hey 19:02:40 <lifeless> so bugs 19:02:48 <lifeless> we've got untriaged bugs second week in a row :( 19:02:56 <lifeless> https://bugs.launchpad.net/diskimage-builder/+bug/1268678 19:03:33 <SpamapS> we've also got a non-working devtest right now. 19:03:47 <lifeless> ruh roh 19:03:48 <SpamapS> https://bugs.launchpad.net/heat/+bug/1271190 19:03:53 <SpamapS> breaking all Heat in-instance users 19:04:00 <SpamapS> the patch to fix has been approved 19:04:38 <SpamapS> Not sure if it is worth marking as affecting TripleO since it will be resolved in Heat within a few hours. 19:05:01 <lifeless> I am not a big fan of paperwork for paperwork sake 19:05:03 <lifeless> so meh 19:05:07 <SpamapS> I suppose knowing what patch to cherry-pick while it wends its way through the gate might be helpful 19:05:09 <lifeless> we have criticals already 19:05:11 <lifeless> https://bugs.launchpad.net/tripleo/+bug/1270646 19:05:22 <lifeless> this breaks tenants 19:05:29 <jomara_> ahoy 19:05:37 <lifeless> 2kbps if you're lucky 19:06:17 <lifeless> anyone with deep network experience feel like owning that and chasing it down to ground? 19:06:38 <SpamapS> I think I am the opposite of that person. :) 19:07:54 <lsmola> hehe 19:08:03 <lifeless> ok, so no volunteers; will take it to the list 19:08:14 <lifeless> https://bugs.launchpad.net/tripleo/+bug/1266513 19:08:22 <lifeless> SpamapS: you're assigned on this, whats the status ? 19:09:02 <SpamapS> lifeless: I have been badgering the upstreams to host on pypi... netaddr responded immediately, the others did not. 19:09:31 <SpamapS> lifeless: We can of course do the exception thing for pip. I dropped it late last week hoping to give the upstreams more time. 19:09:46 <SpamapS> I think there are two.. netifaces may be abandoned.. and I forget the other one. 19:09:56 <lifeless> SpamapS: so AIUI it's critical because our builds fail right? 19:11:00 <SpamapS> lifeless: I think invididual projects all adapted .. not sure how. 19:11:12 <SpamapS> Just never revisited to lower it to High. 19:11:52 <lifeless> SpamapS: since we don't seem to have any actual symptoms at the moment, I think High is better 19:11:53 <bnemec> Didn't the other projects whitelist the externally hosted deps in tox.ini? 19:12:04 <SpamapS> lifeless: ah, we're not installing any of the ones that need netifaces 19:12:11 <lifeless> SpamapS: either way :_ 19:12:18 <SpamapS> and we don't install review-day so lazr.restful is not needed 19:12:21 <lifeless> SpamapS: sounds like its either not a bug for us 19:12:26 <lifeless> SpamapS: or not hitting us 19:12:32 <SpamapS> lifeless: I think I may close it. 19:12:35 <lifeless> SpamapS: I'd F-R it for us 19:12:51 <SpamapS> i want to verify that swift is not affected somehow and we're just not testing swift 19:12:52 <lifeless> ok https://bugs.launchpad.net/tripleo/+bug/1254246 is also assigned to SpamapS 19:13:06 <rpodolyaka1> this can be probably be closed too 19:13:17 <rpodolyaka1> as I already said it's kind of fixed for us 19:13:29 <rpodolyaka1> and we aren't hitting this, I believe 19:13:44 <lifeless> rpodolyaka1: you were going to do some validation or something last week? 19:13:46 <rpodolyaka1> though, there is 'proper fix' waiting on review 19:14:02 <rpodolyaka1> lifeless: yeah, it works for me 19:14:21 <lifeless> so, f-r for tripleo? 19:14:28 <rpodolyaka1> I think, so 19:14:34 <SpamapS> Yeah agreed 19:14:41 <SpamapS> Though it may also just be masked now. 19:14:42 <rpodolyaka1> will continue to track this in neutron 19:14:55 <lifeless> ok 19:15:23 <lifeless> what about https://bugs.launchpad.net/tripleo/+bug/1254555 19:15:33 <lifeless> I'm fairly sure we still have a hack in our tree 19:17:13 <rpodolyaka1> hasn't it been removed? 19:17:34 <rpodolyaka1> I thought, it was here https://github.com/openstack/tripleo-image-elements/blob/master/elements/tripleo-cd/tripleo-cd.sh 19:17:53 <lifeless> 38ac22ce7c4ea4ee9a86e1b6f2c3e5e31c8ba652 19:17:57 <lifeless> in incubator 19:19:29 <lifeless> Date: Thu Dec 19 10:36:48 2013 +0000 19:19:29 <lifeless> Merge "Revert "Work around neutron floatingip race condition"" 19:19:32 <lifeless> ok, so its gone 19:19:39 <lifeless> we can f-r that bug too 19:20:26 <lifeless> ok so that gives us two criticals, one with a fix landing in heat, one that may go all the way to ovs 19:22:04 <lifeless> #action lifeless to seek owner for network bug on list 19:22:14 <lifeless> #topic reviews 19:22:19 <lifeless> #undo 19:22:20 <openstack> Removing item from minutes: <ircmeeting.items.Topic object at 0x3a10d10> 19:22:29 <lifeless> heh, undo fail 19:22:58 <lifeless> anyhow, I was going to say - please do do one day of triage, collectively we can make sure all the projects are triaged easily 19:23:05 <lifeless> #topic reviews 19:23:16 <lifeless> http://russellbryant.net/openstack-stats/tripleo-openreviews.html 19:23:23 <lifeless> 19:23:23 <lifeless> Stats since the last revision without -1 or -2 : 19:23:28 <lifeless> 3rd quartile wait time: 0 days, 21 hours, 40 minutes 19:23:39 <lifeless> still in ok shape - good stuff 19:23:58 <lifeless> however 19:24:43 <lifeless> 19:24:43 <lifeless> Longest waiting reviews (based on oldest rev without -1 or -2): 19:24:43 <lifeless> 12 days, 4 hours, 1 minutes https://review.openstack.org/65699 (WIP: Make rabbitmq-server keep state in /mnt/state) 19:24:46 <lifeless> 8 days, 2 hours, 42 minutes https://review.openstack.org/52045 (Using merge.py from tuskar to generate overcloud.yaml) 19:24:49 <lifeless> 4 days, 17 hours, 28 minutes https://review.openstack.org/67348 (Add SSL configuration metadata for overclouds.) 19:24:52 <lifeless> 4 days, 16 hours, 47 minutes https://review.openstack.org/67356 (Permit deploying an overcloud with SSL endpoints.) 19:24:55 <lifeless> 4 days, 13 hours, 26 minutes https://review.openstack.org/67358 (Use OS::Heat::UpdateWaitConditionHandle) 19:24:58 <lifeless> sorry for the size 19:25:00 <lifeless> this is a collective fail 19:25:15 <SpamapS> this means we're not starting at oldest first maybe? 19:25:23 <lifeless> I don't know 19:25:27 <lifeless> but if you look at those reviews 19:25:34 <lifeless> they haven't been -1 or -2'd 19:25:42 <lifeless> they've been sitting for days 19:25:59 <slagle> i just reviewed the ssl stuff this morning 19:26:01 <slagle> i wanted to test it 19:26:24 <lsmola> lifeless, I will review the merge.pz, though I think it is still work in progress 19:26:32 <slagle> although, somehow i missed 67348 19:26:35 <lifeless> lsmola: if its work in progress, it should be marked WIP 19:26:45 <lsmola> lifeless, I will ask marios 19:26:48 <slagle> i don't think this reviewable reviews query is very accurate. b/c that one is not in my list 19:26:51 <lifeless> This isn't an individual responsibility issue 19:27:01 <lifeless> slagle: what reviewable reviews query ? 19:27:04 <marios> lsmola: lifeless: it *is* still wip in the sense that the models/controllers are yet tbd in tuskar 19:27:17 <marios> lifeless: lsmola : but -1 with appropriate comments are very very welcome at this stage 19:27:26 <marios> lifeless: lsmola: hence i removed the wip 19:27:30 <lifeless> marios: if its not ready for review yet; please mark it WIP, otherwise folk will be trying to review it to get it landed 19:27:34 <slagle> lifeless: very last link on https://wiki.openstack.org/wiki/TripleO#Review_team 19:27:36 <lifeless> marios: ack 19:27:56 <lsmola> marios, ok 19:28:37 <slagle> i don't typically use it, unless there's lots of reviews. which there have been last few days 19:29:09 <lifeless> slagle: indeed, that review is missing! 19:29:23 <slagle> weird 19:29:27 <slagle> tripleo-heat-templates is in the query 19:29:30 <lifeless> ah 19:29:34 <lifeless> remove is:reviewable 19:29:39 <lifeless> that means 'passed jenkins' 19:29:49 <lifeless> but jenkins was on holiday 19:30:14 <lifeless> slagle: I'm inclined to suggest we remove that from the wiki page link 19:30:14 <slagle> right, but then you just get all reviews, including ones you've already reviewed 19:30:26 <slagle> whereas is:reviewable removes one you've already done 19:30:46 <slagle> and your own changes for that matter 19:30:51 <lifeless> slagle: does it put them back once someone else comments ? 19:31:07 <slagle> i don't know 19:31:16 <SpamapS> slagle: sounds like the baby went out with the bathwater there.. 19:31:21 <slagle> it seems to do several things. not all of them well 19:31:30 <SpamapS> slagle: I'd rather work around ones I've done and my own than miss stuff. :-P 19:31:38 <slagle> so, yes, i'm with you. let's just remove from teh wiki 19:32:01 <SpamapS> FTR I just use gerrit and round-robin among the various tripleo projects. 19:32:18 <lifeless> ok, so this isn't about individual reviewers; we have a large team 19:32:22 <pblaho> good evening 19:32:40 <lifeless> but we've consistently missed 5 reviews for 2 whole days (ignoring the weekend) 19:32:59 <lifeless> has everyone else been using the unified link as well? 19:33:21 <rpodolyaka1> me 19:33:22 <jdob> i use the second to last one, the All one 19:33:29 <jdob> not the Reviewable one 19:33:48 <lifeless> jdob: ok so you'd see everything 19:34:05 <jdob> i do see my own and things I've commented on, so assuming it aggregates right with no special sauce, then yes 19:34:30 <lifeless> yeah 19:34:55 <lifeless> so I don't think 'work harder' will fix this, we've something systemic thats happened; assuming everyone is doing some daily review 19:35:03 <lifeless> I've been a bit slack this week - sorry 19:35:53 <lifeless> so - experiment, can folk please try to review the *oldest by submission date* first ? 19:36:08 <rpodolyaka1> makes sense 19:36:13 <slagle> ack 19:36:16 <lifeless> hmm, but I can't see how to get that from the unified review 19:36:20 <lifeless> so this might be a wishlist 19:36:25 <jdob> i normally tend to check the newest, so i'll start going in reverse 19:36:25 <lifeless> shall we move on? 19:36:54 <lifeless> well, oldest on that page will be least update - which for a change w/out updates will also be oldest submitted 19:37:04 <lifeless> so yeah, oldest is probably an ok metric 19:37:15 <lifeless> trivial rebases will inherit -1's 19:37:35 <jdob> er, the ones at the top of the list is what I meant by newest, not sure if that meant by submission or state 19:37:50 <lifeless> so it should be pretty fast to open and check there is no replies etc 19:38:01 <lifeless> jdob: thats the newest by last-modification 19:38:06 <lifeless> reviewing counts as a modification 19:38:22 <lifeless> ok, moving on 19:38:25 <lifeless> #topic Projects needing releases 19:38:43 <lifeless> AFAIK all our projects are in reasonable shape - the issue right now is just heat, right? 19:39:01 <rpodolyaka1> seems so 19:39:14 <marios> lifeless: tuskar is obviously still in flux, working out models/controllers etc 19:39:22 <lifeless> do we have a release volunteer this week? excluding tuskar* ? 19:39:24 <marios> lifeless: but no blockers atm 19:39:34 <lifeless> marios: or do you want a release cut? 19:39:45 <marios> lifeless: i don't know what we'de release (no) 19:39:52 <marios> imo 19:39:56 <jdob> i second that no 19:40:01 <lifeless> ok, so excluding tuskar* 19:40:16 <rpodolyaka1> lifeless: I can take it for this week, but will need a rain check for the next two weeks 19:40:23 <lifeless> rpodolyaka1: ok! 19:41:04 <lifeless> #topic CD Cloud status 19:41:15 <jdob> marios: see #tripleo 19:41:27 <lifeless> HP region - generally good shape, though we've got a number of DC tickets open at the moment 19:41:51 <lifeless> undercloud is happy; cd-overcloud is I believe paused while we put the finishing touches on stateful upgrades 19:42:08 <SpamapS> lifeless: I don't actually recall why we paused most recently 19:42:11 <lifeless> ci-overcloud is happy 19:42:21 <SpamapS> ovs maybe? 19:42:35 <lifeless> SpamapS: no, not sure 19:42:45 <lifeless> lets make sure there's no local patches and restart it 19:43:02 <lifeless> in incubator/templates/elements 19:43:13 <lifeless> dprince: hows the RH region ? 19:43:16 <SpamapS> lifeless: ok, I need to get some Heat retry-on-update patches in shape in the review queue and then I will see if we can resurrect it. 19:43:32 <lifeless> SpamapS: do we have the waitcondition fix we needed ? 19:43:53 <lifeless> SpamapS: if you say 'thats waiting for review' I will lol. 19:43:53 <SpamapS> lifeless: it is awaiting review in Heat. We cherry picked it into the undercloud's heat. 19:44:04 <lifeless> SpamapS: ok, cool 19:44:11 <SpamapS> https://review.openstack.org/#/c/63245/ 19:44:14 <SpamapS> lol away :) 19:44:17 <lifeless> lol 19:44:31 <SpamapS> Heat review team seems overwhelmed with i2 surge. 19:44:36 <lifeless> we'll come back to dprince I think, time is short 19:44:54 <lifeless> #topic CI virtualized testing progress 19:45:02 <lifeless> pleia2: you is up 19:45:48 * dprince tunes in 19:46:19 <pleia2> this morning I went through the etherpad of our most recent todo list from last week and crossed off what I know was done 19:46:25 <pleia2> #link https://etherpad.openstack.org/p/tripleo-test-cluster 19:46:57 <pleia2> we might want to add the ongoing review nodepool work to that (but it was a holiday here yesterday, so I'm still catching up) 19:47:12 <lifeless> the node pool patches we need are merged 19:47:17 <lifeless> that was just network support 19:47:27 <lifeless> the big set thats open is trying to help infra with overload 19:47:48 <pleia2> ah, gotcha 19:48:18 <lifeless> whats next for ci virt testing ? 19:49:06 <pleia2> I think derekh would have a better view of that 19:49:38 <lifeless> ENODEREK 19:49:43 <lifeless> ok 19:49:54 <lifeless> dprince: RH region of tripleo-cloud, status update? 19:50:36 <dprince> lifeless: They are disconnecting ports from our internal network to prep for making it public. 19:50:44 <lifeless> \o/ 19:50:46 <pleia2> sweet 19:50:54 <dprince> lifeless: in prep for this they downed the ports on the switches 19:51:00 <dprince> So I've lots access :( 19:51:09 * dprince is sad not having access 19:51:13 <lifeless> progress is great news 19:51:24 <lifeless> ok 19:51:28 <dprince> Still waiting on the IPv4 request too though 19:51:38 <lifeless> #topic open discussion 19:51:46 <pleia2> dprince: shall we have our weekly syncup tomorrow? 19:52:07 <dprince> pleia2: yes. I think they are still productive for everyone 19:52:15 <pleia2> great 19:52:33 <slagle> i'd like to discuss the stable branches for our projects 19:52:45 <slagle> sounds like we got somewhat of a general concensus on the mailing list discussion 19:53:00 <dprince> pleia2: although I myself feel like I'm struggling to make progress sometimes :-| 19:53:04 <slagle> i don't honestly know if there is enough time to do something by I2 which is the 23rd 19:53:24 <slagle> but, i plan to start making progress towards it 19:53:26 <slagle> sound ok? 19:53:33 <lifeless> slagle: I'd like some time to talk package install stuff with you too :) maybe after my run of meetings today 19:53:46 <ekarlso> Q: is there a easy way to setup a testing env for tuskar ? 19:54:04 <lsmola> ekarlso, devtest 19:54:17 <SpamapS> Hm 19:54:23 <ekarlso> lsmola: ? 19:54:25 <SpamapS> we don't currently include horizon/tuskar in devtest do we? 19:54:26 <ekarlso> devstack ? 19:54:40 <SpamapS> ekarlso: http://docs.openstack.org/developer/tripleo-incubator/devtest.html 19:54:52 <slagle> lifeless: sure, just ping me at some point. i'm on EST time (utc - 5), but am often on in the evenings 19:55:00 <lsmola> SpamapS, though it easy to install it in undercloud 19:55:01 <lifeless> slagle: cool 19:55:03 <shadower> ekarlso: devtest is a way of setting up tripleo for development/testing 19:55:06 <bnemec> There is a horizon element for use in devtest. 19:55:20 <ekarlso> does that do tuskar ? :) 19:55:20 <lifeless> so stable branches - yeah, I think if someone's going to do it, they should do it in the tent ;) 19:55:46 <shadower> ekarlso: you'll need to add it to the undercloud image elements 19:55:53 <shadower> not sure if we have the UI elementized yet though 19:55:54 <marios> ekarlso: you'll need to setup tuskar once you've got devtest and an undercloud running 19:56:02 <slagle> lifeless: the tent? 19:56:14 <shadower> marios: iirc the tuskar element's in, the UI maybe not 19:56:21 <marios> shadower: ic 19:56:29 <shadower> marios: https://github.com/openstack/tripleo-image-elements/tree/master/elements/tuskar 19:56:31 <lifeless> slagle: oh, there's a saying do you want the camel outside the tent pissing in, or inside the tent pissing out 19:56:43 <lifeless> slagle: its rather vivid :) 19:56:53 <slagle> lol. i think i get it :) 19:56:56 <lifeless> slagle: and I'm not actually comparing you to a camel :) 19:57:16 <lifeless> plus 19:57:17 <ccrouch1> i had no idea it was LBJ: http://izquotes.com/quote/241192 19:57:19 <jcoufal-mobile> shadower: we should have a look on that 19:57:22 <lifeless> on incubator specifically 19:57:45 <lifeless> I don't want stable releases being done to be blocked on, or force premature matriculation, of incubator contents 19:57:51 <shadower> ekarlso: so before the step 1. here: http://docs.openstack.org/developer/tripleo-incubator/devtest_undercloud.html you'd do `export UNDERCLOUD_DIB_EXTRA_ARGS=tuskar` 19:58:08 <lifeless> It's not that I *want* incubator released, but if you're taking a snapshot, it is what it is 19:58:26 <shadower> ekarlso: once it's launched, you'd have to ssh in and install the tuskar UI manually (for now) 19:58:41 <slagle> right, well the first thing i was going to do was to get the docs updated to say "it is what it is" and set expectations 19:58:47 <lsmola> ekarlso, install devtest then this https://github.com/openstack/tuskar-ui/blob/master/docs/install.rst 19:58:58 <lifeless> slagle: cool 19:59:13 <lifeless> ok folks, we're out of time. -> #tripleo for more 19:59:15 <lifeless> #endmeeting