19:01:37 <clarkb> #startmeeting infra
19:01:38 <openstack> Meeting started Tue Jan 29 19:01:37 2013 UTC.  The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:01:39 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:01:41 <openstack> The meeting name has been set to 'infra'
19:02:10 <fungi> olaph: sends his regards. said he has no news on his topics this week but will likely have a lot of updates next week
19:02:47 <fungi> so we probably want to just reapply his action items from the last meeting
19:02:53 <clarkb> I haven't thrown an agenda together. Did jeblair add one to the wiki? if not, things we should cover: CLA, wiki, recheck enforcement, zuul queue priority
19:03:14 <fungi> i looked at the agenda from last week and it looked relevant
19:03:39 <fungi> oh, he did
19:03:41 <fungi> Wiki: Meetings/InfraTeamMeeting (last edited 2013-01-29 05:30:39 by JamesBlair)
19:04:24 <clarkb> well lets just run through them. anyone other than olaph have updates from last weeks actions?
19:04:42 <fungi> quantal slaves
19:05:06 <clarkb> #topic quantal slaves
19:05:20 <fungi> #link https://review.openstack.org/20638
19:05:42 <fungi> once that merges, we should be set to be able to do what we expect with the static quantal slaves
19:06:21 <fungi> questions there, or do we move on to the next action item?
19:06:35 <clarkb> yeah, can you quickly check that it continues to work on oneiric too?
19:07:01 <fungi> clarkb: oh, sure. as in puppet runs clean
19:07:05 <fungi> i'll check that
19:07:11 <clarkb> and if it does, I think we should just go ahead and merge it as jeblair and mordred probably won't have time to review and I don't want that to block your progress on quantal
19:07:19 <fungi> sure
19:07:53 <fungi> i'm not quite sure where it goes to head in the direction of tempest on quantal though, since we'll need a new devstack slave pool
19:08:40 <fungi> oh, wait, that says quantum not quantal
19:08:48 <fungi> okay, so next topic, cla stuff
19:08:56 <clarkb> #topic CLA
19:09:20 <fungi> announcement sent last week, and project management wiki splatted out there
19:09:24 <fungi> #link http://wiki.openstack.org/Project_Group_Management
19:09:32 <clarkb> we still on track for the 24th?
19:09:51 <fungi> nobody has told me otherwise
19:10:12 <clarkb> cool (wasn't sure if the announcement had received push back)
19:10:29 <fungi> also toddmorey got back to me last night and we'll set up a call this afternoon to update him with the new application security key
19:10:44 <fungi> *nobody* has replied to the announcement. eerie
19:11:02 <clarkb> will there be more prep changes that need to be done ahead of time or are we set until the change day?
19:11:27 <fungi> i want to stage a couple more documentation updates while we're waiting, but they're small
19:12:01 <fungi> and also i need to review what's necessary to set all the non-system groups in gerrit to self-managed. not sure if that's doable through configuration management yet
19:12:30 <clarkb> fungi: you will need to make a change to jeepyb so that the manage-projects script defaults to self managed group creation
19:12:36 <fungi> might be a manual thing, though the ultimate goal is to be able to automate group creation as part of project bootstrapping
19:12:45 <fungi> yeah, that
19:12:55 <clarkb> I am not sure what is necessary for existing groups
19:13:42 <clarkb> anything else on the new CLA?
19:13:48 <fungi> right. i'm thinking probably a "always member" option to seed a new group with a specific account too, and ensure that account (if listed) is in the group. we can enforce group configuration at the same time
19:13:55 <fungi> nothing else on cla
19:14:13 <clarkb> I am excited for this
19:14:27 <annegentle> fungi: where did the announcement go to?
19:14:29 <fungi> i am excited to have it behind me and be able to focus more on new stuff ;)
19:14:39 <fungi> annegentle: openstack-dev@lists.openstack.org
19:14:56 <fungi> let me dig up a link
19:15:18 <annegentle> fungi: yes please I'd like to take a look
19:15:19 <clarkb> #link http://lists.openstack.org/pipermail/openstack-dev/2013-January/004951.html
19:15:33 <fungi> you beat me by mere seconds
19:15:44 <annegentle> heh I just read the Gerrit unavailability and didn't read the rest
19:15:46 * annegentle hangs head
19:16:04 <clarkb> #topic Jenkins Slave Operating Systems
19:16:25 <fungi> rhel!
19:16:29 <clarkb> other than progress on Quantal is there anything else on this topic?
19:17:03 <fungi> the progress on quantal was made fin by trying to make sure i didn't break rhel slaves without having any on hand
19:17:04 <clarkb> fungi: dprince I haven't followed the creation of rhel slaves too closely. Do you guys know what is going on there?
19:17:14 <fungi> er, made fun
19:17:48 <fungi> dprince: if you've been previously testing those, you might want to confirm 20638 is safe and doesn't mess with anything rhel
19:18:02 <fungi> i'll throw you on as an optional reviewer
19:18:57 <fungi> but i know nothing else beyond that. i think jeblair was more involved in the early phases than the rest of us
19:19:24 <clarkb> yes, I believe he has been trying to sort out that it is ok for us to spin up lots of rhel nodes for testing
19:19:26 <fungi> though i see we have a rhel test slave in rs nova
19:19:34 <clarkb> and he did the testing of dprince's changes
19:20:02 <fungi> and i thought i saw in #-dev scrollback mordred saying something about progress on the licensing situation there
19:20:17 <fungi> er, no, in #-infra scrollback
19:21:21 <clarkb> we'll need to wait for them to update us on the licensing situation I think
19:21:28 <fungi> agreed
19:21:38 <clarkb> ready for the next topic?
19:21:39 <fungi> maybe add an action item for that
19:21:59 <fungi> always fun to assign stuff to absentees
19:22:05 <clarkb> #action mordred,jeblair update CI team on RHEL licensing situation
19:22:24 <fungi> and yeah, tempest next i guess
19:22:28 <clarkb> #topic Tempest gating (and quantum)
19:22:38 <fungi> on again off again quantum gating
19:22:49 <fungi> it's on again, right?
19:22:58 <clarkb> yes quantum is on
19:23:12 <clarkb> tempest smokestack tests with quantum are enabled across the board and are voting
19:23:22 <fungi> is there anything to discuss about it? other than that it's on i mean?
19:23:25 <clarkb> tempets full with quantum is enabled on tempest changes but is only non voting
19:23:45 <fungi> ahh, okay
19:23:52 <clarkb> I think people wanted to discuss the various test optimization options available to us
19:24:10 <fungi> as far as partitioning, selective testing, et cetera?
19:24:12 <clarkb> jaypipes: davidkranz sdague Did you guys want to talk about that now or give us an update on tempest +testr etc
19:24:15 <clarkb> fungi: yes
19:24:19 <clarkb> mtreinish: ^ you too
19:25:27 <clarkb> We can get back to this if they see the highlights
19:25:36 <clarkb> #topic Wiki Migration
19:26:06 <clarkb> annegentle: fungi we had a thread bounce around that seemed to settle on february 16 or 17 for the actual migration
19:26:06 <jaypipes> clarkb: no update from me on that
19:26:20 <annegentle> clarkb: I am fine with a weekend day
19:26:42 <annegentle> clarkb: which still doesn't decide between a Sat. or Sun. :)
19:26:53 <Ryan_Lane> o/
19:26:59 <fungi> yeah, i'm good with whatever day everyone else settled on, which is why i wasn't being particularly noisy on the e-mail thread
19:27:11 <clarkb> Ryan_Lane: hi there, I think we have settled on february 16 or 17 for the migration change
19:27:18 <sdague> clarkb: sorry, just saw the ping
19:27:23 <clarkb> Ryan_Lane: did you have a preference and do those dates work for you?
19:27:31 <clarkb> sdague: thats ok, can swing back to it after the wiki stuff
19:27:35 <sdague> sure
19:27:48 <Ryan_Lane> yep. fine with me
19:28:14 <pleia2> just FYI, that's president's day weekend in the states
19:28:20 <clarkb> we should settle on a day so that annegentle can send the announcement (I think annegentle was given that task in the email thread)
19:28:24 <pleia2> (in case that makes a different to anyone :))
19:28:32 <pleia2> difference
19:29:04 <annegentle> pleia2: good point, my kids will have school off that Monday
19:29:14 <clarkb> #startvote Wiki migration day? 16, 17
19:29:15 <openstack> Begin voting on: Wiki migration day? Valid vote options are 16, 17.
19:29:16 <openstack> Vote using '#vote OPTION'. Only your last vote counts.
19:29:27 <annegentle> #vote 17
19:29:36 <fungi> #vote 17
19:29:43 <clarkb> #vote 17
19:30:20 <clarkb> anyone else?
19:30:32 <clarkb> (probably didn't need the vote function but I wanted to use it :) )
19:30:39 <fungi> it's fun
19:30:43 <annegentle> it's like putting off homework until Sunday night :)
19:30:48 <torgomatic> ]\'
19:30:48 <torgomatic> \
19:30:49 <torgomatic> \
19:30:50 <pleia2> I'll be out of town, but you probably won't need me anyway for this
19:30:51 <torgomatic> \
19:30:56 <fungi> which is why i went for active assent instead of passive assent
19:30:59 <torgomatic> sorry, toddler got my keyboard
19:31:40 <clarkb> #endvote
19:31:41 <openstack> Voted on "Wiki migration day?" Results are
19:31:42 <openstack> 17 (3): clarkb, fungi, annegentle
19:31:54 <clarkb> February 17th for the wiki migration then
19:32:16 <annegentle> #action annegentle to send note to mailing list with Feb. 17th wiki migration date
19:32:21 <clarkb> Ryan_Lane: olaph wasn't able to join us, but where are we on being ready for the 17th?
19:32:32 <clarkb> I think all the image upload stuff is working. Is the last bit the skin?
19:32:37 <Ryan_Lane> yeah. skin
19:32:44 <Ryan_Lane> it's just css and logo
19:32:56 <clarkb> #action olaph finish skinning wiki-staging.o.o in prep for wiki migration
19:33:00 <Ryan_Lane> so it can be done without modifying a skin
19:33:40 <clarkb> anything else we need to cover on the wiki migration?
19:33:43 <Ryan_Lane> I think puppet is still switching the wiki to the master branch
19:33:52 <Ryan_Lane> was my change ever merged?
19:33:56 * clarkb looks
19:34:17 <clarkb> it isn't in my review queue which means probably
19:34:20 <Ryan_Lane> ah. it was
19:34:29 <Ryan_Lane> I probably just need to switch the branch
19:34:40 <annegentle> Is there a way to get a list of expected "fixes" people should do? tables, images, columns, anything else?
19:35:17 <clarkb> I think Ryan_Lane had a list somewhere
19:35:34 <fungi> probably want to test metastuff like pages written with different preprocessors, admonishments, categories
19:35:48 <Ryan_Lane> I think we still need a youtube extension
19:36:55 <Ryan_Lane> well, it's mostly obvious what is broken when looking at pages
19:37:07 <Ryan_Lane> we can make a list, though
19:37:20 <annegentle> Let's make a list...
19:37:33 <fungi> how many pages do we have at last count? easy enough to just browse them all or will we need to divide and conquer?
19:37:35 <Ryan_Lane> last time I went through this I looked at the old wiki and the new wiki and compared
19:37:54 <Ryan_Lane> last time I checked it was easy enough to browse
19:37:56 <clarkb> so thats three things. a list of fixing, git management and puppet, and a youtube extension. Who will be working on these things?
19:37:59 <annegentle> https://etherpad.openstack.org/wikimigrationemail I'll start a list
19:38:12 <clarkb> #link https://etherpad.openstack.org/wikimigrationemail
19:38:13 <Ryan_Lane> git management I'm fixing now
19:38:23 <Ryan_Lane> I can also handle the youtube extension
19:38:39 <clarkb> #action Ryan_Lane to fix mediawiki git interaction and add youtube extension
19:39:09 <clarkb> anything else?
19:39:55 <clarkb> doesn't sound like it
19:39:56 <fungi> doesn't sound like it. i guess we should swing back to tempest stuff while we have qa peeps
19:40:04 <clarkb> #topic Tempest gating (and quantum)
19:40:08 <clarkb> sdague: ready?
19:40:12 <sdague> sure
19:40:34 <sdague> ok, so we're still probably about 2 weeks out from testr conversion
19:40:50 <sdague> the unittest -> testtools did go in yesterday
19:41:03 <sdague> so that's progressing well
19:41:13 <clarkb> woot
19:41:35 <sdague> total gate run time for tempest is around 45 - 55 minutes
19:41:53 <clarkb> sdague: that is when running the full test suite?
19:41:54 <sdague> which is only about 12 - 15 mins more than smoke
19:41:57 <sdague> yes
19:42:29 <sdague> so, honestly, it would be really nice to turn on full gate prior to testr if possible
19:42:29 <clarkb> are we still seeing much longer run times on rackspace?
19:42:44 <sdague> I don't have that timing atm, jeblair might
19:42:59 <clarkb> it is available from statsd somehow
19:43:15 <fungi> i was trying to figure out how to conjure it out of graphite.o.o last week
19:43:34 <fungi> i think i found where the majic is, but my mojo is still lacking
19:43:58 <sdague> my ideal scenario would be to get full gate turned on as soon as possible, as we continue to watch bugs creep in that we would have caught
19:44:06 * ttx yawns
19:44:07 <clarkb> sdague: and by turning on the full gate you mean run all projects against the entire tempest suite?
19:44:46 <sdague> I think nova, cinder, keystone for sure
19:45:08 <sdague> the glance tests aren't really any different from smoke to full
19:45:14 <sdague> quantum is it's own beast
19:45:31 <clarkb> and quantum appears to be failing the full suite at the moment
19:45:31 <sdague> and swift kind of is as well, they are just putting the tests in now
19:45:58 <sdague> and, obvious, tempest does nothing for horizon
19:46:35 <clarkb> sounds like we will need to do some planning so that we aren't over or under testing
19:47:05 <clarkb> sdague: do you think you could send mail to the openstack-infra maillist so that we can get this conversation going with mordred and jeblair too?
19:47:05 <sdague> yeh, I guess the real question is overall CI throughput
19:47:32 <sdague> sure, I'll sort of continue the jeblair email thread from last week
19:47:38 <ttx> sdague: throughput is greatly affected by the number of flaky tests, though
19:47:38 <sdague> get that back out onto lists
19:47:41 <mordred> clarkb: what?
19:47:48 <mordred> oh - good morning. is it ci meeting time?
19:47:53 <clarkb> mordred: it is, good morning
19:48:08 <sdague> ttx: the flaky issues seem to be nailed at this point in tempest
19:48:15 <clarkb> I am wary of making large gate decisions like that without input from jeblair
19:48:16 <sdague> they were mostly overrunning the guest resources
19:48:29 <sdague> clarkb: yep, I'm fine with that
19:48:51 <clarkb> I am all for running more of the tempest suite though :)
19:48:54 <ttx> sdague: I wouldn't say that. Been pushing security fixes that pass all tests for the last 5 hours
19:49:12 <sdague> ttx: which things are flaking?
19:49:13 <ttx> they just hit various issues, all documented in rechecks
19:49:24 <ttx> Bug 1056213
19:49:25 <uvirtbot> Launchpad bug 1056213 in openstack-ci "Flakey build failures in some volume tests" [High,Confirmed] https://launchpad.net/bugs/1056213
19:49:29 <ttx> Bug 1102892
19:49:30 <uvirtbot> Launchpad bug 1102892 in glance "Fragile Test: glance.tests.functional.test_bin_glance_control.TestGlanceControl.test_respawn" [Undecided,New] https://launchpad.net/bugs/1102892
19:49:35 <ttx> Bug 1108985
19:49:36 <uvirtbot> Launchpad bug 1108985 in glance "Spurious ECONNREFUSED errors during Glance unit tests" [Undecided,New] https://launchpad.net/bugs/1108985
19:50:17 <fungi> the rechecks page is really becoming useful
19:50:21 <fungi> #link http://status.openstack.org/rechecks/
19:50:37 <ttx> sdague: not all of those are tempest... my point is that adding more tests will only add to that list
19:50:41 <sdague> ttx: well unit tests are out of scope for this, and I agree glance is under tested
19:50:54 <clarkb> #action sdague send mail to openstack-infra list to kick off discussion on running full tempest on more projects
19:51:06 <sdague> ttx: so we've got hourlies running full tempest at this point that are consistently passing
19:51:09 <ttx> and the number of flaky tests affects throughput a lot more than single test duration
19:51:13 <sdague> and have been for over a month
19:51:58 <ttx> sdague: i'm all for more tests, just saying that we need to address falikness when it's detected
19:52:05 <ttx> flakiness*
19:52:22 <sdague> ttx: sure, were these on master or stable/
19:52:41 <ttx> some of them were on stable/*
19:52:47 <sdague> because a lot of work was done on master to get rid of the flakiness, more than would be really suitable to backport to tempest/stable
19:53:10 <ttx> anyway, don't want to disrupt the meeting with my pet peeve of today
19:53:24 <clarkb> ok, anything else we want to bring up about tempest?
19:53:26 <sdague> ttx: yes, that's fair. Definitely working to close those bugs if they exist
19:53:49 <ttx> because that flakiness is the primary factor affecting throughput
19:54:28 <clarkb> #topic open discussion
19:54:39 <clarkb> this gives us about five minutes for $OTHER things
19:54:56 <ttx> mordred: got a chance to talk to jeblair about publishing python-swiftclient 1.3.0 to PyPI ?
19:55:06 <clarkb> I will be changing the console log timestamp format on jenkins.o.o and merging my logstash puppet change later today (FYI)
19:55:09 <mordred> ttx: not since last we spoke
19:55:21 <mordred> ttx: I just woke up and havent' seen him yet
19:55:31 <fungi> clarkb: i'm in favor
19:55:38 <mordred> clarkb: ++
19:55:49 <clarkb> if I get around to it I will also be fixing puppet dashboard (upgrading to a version not vulnerable to the rails vulnerability)
19:56:17 <ttx> mordred: if a permanent fix is not acceptable, would be good to just do a one-off manual publication to fix the immediate issue
19:56:22 <fungi> we've also got a stack of jjb patches ready to be merged, and it would be nice to tag a new git-review release soonish
19:56:27 <clarkb> and depending on how much work I feel like doing today I may install my zmq plugin and restart jenkins tonight when the queues are quiet
19:57:31 <pleia2> working on my reviewday action item, but it's requiring some reviewday patches before we can deploy it in our infrastructure so that's ongoing
19:57:32 <clarkb> ttx: mordred we may need to do a manual release because the tarball job borked?
19:57:48 <mordred> clarkb: yeah
19:58:00 <mordred> well, we're going to need to do a manual _something_ regardless
19:58:09 <ttx> mordred: my problem with delaying the fix is that we won't necessarily know when the next PTL will push a tag
19:58:15 <mordred> then there is also the question of the right way to fix the bork
19:58:25 <mordred> ttx: I agree with that
19:58:28 <khaido> i'm almost ready to send a big git review for gearman-jenkins plugin.
19:58:42 <ttx> whereas we know we need to push support to all projects before g3
19:58:48 <fungi> khaido: awesome!
19:59:13 <clarkb> mordred: ttx let me know if/when decisions are made and if you need someone to push the buttons
19:59:25 <khaido> yeah.  clarkb help me rewrite history.
19:59:34 <fungi> i'm happy to do so as well, if i'm still awake
20:00:08 <clarkb> and we are out of time
20:00:11 <clarkb> #endmeeting