19:04:04 <jeblair> #startmeeting infra
19:04:05 <openstack> Meeting started Tue Mar 19 19:04:04 2013 UTC.  The chair is jeblair. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:04:06 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:04:08 <openstack> The meeting name has been set to 'infra'
19:04:26 <jeblair> #topic actions from last meeting
19:05:00 <jeblair> i think we'll give clarkb another week on the new hpcs account, what with him being on vacation and all
19:05:04 <jeblair> #action clarkb set up new hpcs account with az1-3 access
19:05:08 <fungi> i'm doubting clarkb or mordred are about
19:05:29 <fungi> quantal stuff
19:05:30 <jeblair> fungi: you migrated stackforge projects to quantal; howd that go?
19:05:38 <fungi> seems fine so far
19:06:05 <fungi> at this point everything except potentially release-impacting jobs are on quantal if they were preciously precise
19:06:13 <jeblair> good, so nothing new we should consider when moving openstack projects
19:06:28 <fungi> nothing identified at this point, no
19:06:33 <jeblair> my... prrrrrrecious...
19:06:48 <fungi> thanks, gollum
19:06:58 <fungi> rhel6
19:07:11 <fungi> review needed a rebase after the oslo.config rename
19:07:36 <fungi> it should be fine at this point, and adds a non-voting shadow for all py26 unit tests on rhel6 slaves
19:08:09 <fungi> we have 8 slaves on dprince's account, and he's given some of us his account credentials to do out-of-band things with them as needed
19:08:30 <fungi> #link https://review.openstack.org/24364
19:08:49 <jeblair> fungi: creds in password file?
19:09:06 <fungi> jeblair: no, but i'll add them this afternoon
19:09:23 <fungi> wasn't sure if that was appropriate, but i suppose it's the best option
19:09:47 <jeblair> fungi: yeah, i think so.
19:10:08 <jeblair> fungi: it's a project resource more or less like the others, for the time being at least
19:10:17 <fungi> works for me then
19:10:24 <jeblair> fungi: did you create a reviewday user?
19:10:42 <fungi> yep, all set up, stuff that needs to be is added to hiera as well
19:11:03 <fungi> i think pleia2's stuff is very close to ready for merging
19:11:08 <jeblair> fungi: cool.  maybe leave a note to that effect on https://review.openstack.org/#/c/21158/
19:11:21 <fungi> i did, but that was several patchsets ago now
19:11:28 <fungi> i'll restate it
19:11:44 <jeblair> oh, sorry, i see it now
19:11:49 <mordred> o/
19:11:56 <jeblair> i searched for user.  not account.  :)
19:11:58 <mordred> (landed)
19:12:01 <fungi> just plane mordred again!
19:12:05 <jeblair> mordred: congrats!
19:12:08 <fungi> oh, no longer plane mordred
19:12:11 <mordred> whee!
19:12:18 <jeblair> just plain mordred now
19:12:23 <mordred> :)
19:12:26 <pleia2> that's all really for reviewday (don't need to revisit later in the meeting)
19:12:42 <jeblair> and the last thing is: we renamed oslo-config
19:12:56 <jeblair> oslo.config seems to be working so far
19:12:58 <fungi> and as of yet, no new complaints
19:13:01 <jeblair> it's our first repo with a dot.
19:13:33 <jeblair> the pypi (or "cheese shop") page has openstackci as a maintainer
19:14:04 <jeblair> everyone at pycon was calling it the cheese shop because saying 'pypi' sounds like 'pypy' which is a different thing.
19:14:06 <fungi> and some very runny gouda
19:14:34 <anteaya> I thought because there was none
19:14:44 <anteaya> a cheese shop without any cheese
19:15:01 <fungi> you can't actually tell the customer you have no cheese, anteaya
19:15:11 <jeblair> #topic gerrit/lp groups
19:15:22 <anteaya> but apparently you can waste their time
19:15:30 <jeblair> erm, it looks like we were all too busy to actually do that this past week
19:15:39 <mordred> heh
19:15:45 <fungi> yeah, other than i created several new gerrit groups :/
19:15:52 <jeblair> #action someone rename -drivers in gerrit to -milestone, and create -ptl groups and give them tag access
19:16:07 <jeblair> that's what we decided to do last meeting; hopefully one of us can pick that up soon
19:16:39 <jeblair> #topic jenkins job logs
19:16:52 <jeblair> fungi: ?
19:16:53 <fungi> they are many
19:17:09 <fungi> static.o.o filled up again on sunday
19:17:25 <fungi> we've done some dances to free up a few more days of runtime on it
19:17:50 <fungi> compressing subunit tempfiles, removing pre-folsom-release-day job logs
19:18:07 <fungi> tried to resize the server last night as a short-term bandaid
19:18:25 <fungi> it's still on "step 1" right now and i can't cancel it, so presumably i should open a ticket
19:18:43 <fungi> we've got a replacement built in rackspace nova with a block volume added
19:19:07 <jeblair> so it sounds like it's time to start removing old logs; do we want a rolling 6 month window, rolling 12 month, or batch delete after release?
19:19:16 <fungi> data's rsync'd from the old server, so we're going to refresh that this evening and try to move after a little validation of the content/vhosts
19:19:20 <annegentle> fungi: just a thought, can we make the doc gate jobs more specific to only build when files for that doc are updated?
19:19:24 <jeblair> (i'm thinking rolling 6 month)
19:19:32 <annegentle> (to try to alleviate filling up static)
19:19:48 <fungi> annegentle: there is a new feature for zuul to do that i think... jeblair?
19:20:00 <jeblair> annegentle: i just added a feature to zuul that would let us do that; i worry a little bit about that being fragile...
19:20:00 <annegentle> fungi: ah ok
19:20:17 <jeblair> as in, we have to specify in the zuul config which files it should run on
19:20:35 <jeblair> so if anything changes in the repo, the zuul config would need to be updated to match
19:21:09 <jeblair> but if it stays pretty static (ie, we know we'll only care about "docs/*"), that's an option
19:21:31 <jeblair> (and would mean running fewer needless jobs which is nice for jenkins)
19:21:34 <annegentle> jeblair: I think it makes sense for docs
19:22:07 <jeblair> annegentle: ok, let's give it a try
19:22:26 <jeblair> also, to be clear, i mean we're solving the space issue the right way which is to get more space
19:22:53 <jeblair> so we don't need to scrimp too much; i think annegentle's idea is just good in principle.  :)
19:22:57 <fungi> yes, and the new server with availability of separate block storage makes this as scalable as we could possibly need for the near future
19:23:13 <annegentle> jeblair: there will be a copy of all docs to start with, I suppose, but the archiving of old doc builds on drafts is unnecessary once the patch is reviewed
19:23:37 <annegentle> jeblair: and then only building when a patch indicates the build needs refreshed makes sense too
19:23:47 <jeblair> annegentle: double +1
19:24:06 <fungi> to simplify retention there, maybe we still use a rolling window for those but at 30 days instead of 180?
19:24:29 <jeblair> well...
19:24:34 <jeblair> let's just use 180 across the board
19:24:51 <fungi> that's certainly simpler still
19:24:52 <jeblair> it's not unheard of for a patch to take 30 days to merge.  :)
19:25:16 <fungi> agreed. just trying to avoid "identify checks corresponding to merged changes and purge" logic
19:25:20 <jeblair> but let's do rememebr to expire the docs draft too.
19:26:14 <fungi> i will definitely rememebr
19:26:18 <jeblair> #topic grenade
19:26:42 <jeblair> there's a non-voting grenade job, but it's failing
19:26:46 <fungi> it's currently going "boom"
19:27:10 <jeblair> we should probably point dtroyer at it so he can take a look
19:27:13 <jeblair> #link https://jenkins.openstack.org/view/All/job/dev-gate-grenade-devstack-vm/
19:27:37 <jeblair> #topic gearman
19:27:48 <jeblair> zaro, fungi: you're testing something, yah?
19:28:07 <fungi> zaro's the one testing. i'm just mashing buttons for him
19:28:11 <zaro> kinda. testing the gearman plugin
19:28:20 <mordred> w00t
19:28:42 <zaro> i believe it's pretty much complete.  i mean all the features are there.
19:29:08 <jeblair> awesome!  so i think that means we're waiting on krow for the queue change in gearmand
19:29:14 <zaro> i think only thing that doesn't work is updating when executors change.
19:29:21 <zaro> yes. that too.
19:29:43 <jeblair> zaro: ok, that's not something we do regularly, so we don't have to wait on the executors thing getting fixed
19:29:45 <zaro> i don't think he's fixed the problem he had.. CI wasn't happy with him.
19:30:00 <jeblair> zaro: did you have a puppet patch to install gearman?
19:30:11 <jeblair> i think that would be the next thing on our side
19:30:19 <jeblair> i think we can run it on the zuul server
19:30:20 <zaro> i thought we install plugin manually?
19:30:32 <jeblair> sorry, gearmand
19:30:49 <zaro> ohh yes. there's already one on puppet forge.
19:31:07 <zaro> i'll figure out how to use it so we can install one for ourselves.
19:31:18 <fungi> so we'll just need a patch to our openstack-infra/config repo to consume and configure that
19:31:49 <mordred> do we have security thoughts for that? ssl tunnel? openvpn?
19:31:53 <zaro> i'm not sure about naming the server though.
19:31:58 <jeblair> #action zaro write puppet to run gearmand on zuul.o.o
19:32:10 <jeblair> zaro: i think we can just colocate it on the server where we run zuul
19:32:15 <jeblair> mordred: we can use iptables for now
19:32:16 <mordred> ++
19:32:24 <mordred> jeblair: *facepalm* of course.
19:32:25 <zaro> cool!
19:32:37 <jeblair> mordred: that will at least scale up to a handful of jenkins masters
19:32:40 <fungi> it's tcp so unless we're worried about someone in control of the routers between zuul and jenkins...
19:32:51 <zaro> any port you prefer?
19:32:55 <jeblair> mordred: just not the 'no jenkins slaves, only masters' topology
19:33:11 <jeblair> fungi: not particularly
19:33:19 <fungi> thought not
19:33:24 <jeblair> zaro: i think there's an iana port for gearman now
19:33:41 <zaro> jeblair: sorry, iana?
19:33:46 <jeblair> (instead of using the afs3 callback port!)
19:33:59 <fungi> internet assigned names/numbers authority
19:34:06 <jeblair> 4730
19:34:14 <jeblair> #link http://gearman.org/protocol
19:34:17 <zaro> cool!]
19:34:41 <fungi> #link http://www.iana.org/
19:34:54 <fungi> (for the curious)
19:35:04 <jeblair> #topic pypi mirror/requirements
19:35:39 <jeblair> i think that's really close
19:35:45 <fungi> i'm excited
19:35:58 <jeblair> the new mirror script is running, though outputting to a subdir of the current mirror
19:36:07 <jeblair> (so we're not using it for builds quite yet)
19:36:26 <jeblair> i've done a hacky thing by installing a python2.6-only dep into the mirror manually
19:36:42 <jeblair> but mordred and i were talking about one more revision to the design:
19:36:49 <jeblair> which is to put the mirror into swift
19:36:57 <mordred> yup. I believe it will work
19:37:04 <mordred> although obvs. needs some testing :)
19:37:15 <jeblair> that _should_ work because we can use the cdn to generate index pages for us
19:37:28 <jeblair> and we only need one cname added for the mirror
19:37:29 <mordred> yes. and we don't need the dir structure we use right now
19:37:39 <mordred> we can put all th efiles in one dir in swfit and use a single index
19:38:04 <jeblair> so then we can run the mirror build on multiple platforms and have them all upload to swift easily
19:38:15 <jeblair> and that should get us the python2.6-only packages
19:38:53 <fungi> on a related note, we probably ought to scour the slaves looking for other hung puppet processes again, based on the couple we hit in the last 24 hours missing the new mirror-using wrapper scripts
19:39:19 <jeblair> so i think the plan is: test more things to make sure the new mirror is okay, when it is, switch the builders to using it
19:39:26 <fungi> i'll run a loop over them and check here in a bit in case there are others needing restarts
19:39:30 <jeblair> then immediately start work on the swift version of the mirror before the 2.6 problem bites us
19:40:24 <jeblair> #topic baremetal testing
19:40:47 <jeblair> pleia2: are you getting your questions answered?
19:40:57 <pleia2> so echohead has been working on the devstack replacement for the baremetal testing here: https://review.openstack.org/#/c/24273/
19:41:08 <mordred> jeblair: re: above mirror stuff real quick ...
19:41:12 <pleia2> it's a lot, but I'm tagging along and working my way through devstack-gate scripts :)
19:41:37 <anteaya> pleia2: go you!
19:41:41 <mordred> I'd like to inject that after swift-mirror I intend to invesitgate our possible use of hte new wheel format for extra fast goodness
19:41:44 <pleia2> no active questions right at this moment, but I'll be checking in with devananda regularly as needed
19:42:00 <mordred> pleia2: awesome
19:42:13 <jeblair> mordred: oh, yes, and since wheel may contain arch-specific files, we should be prepared for that by having multiple mirror builders
19:42:52 <jeblair> pleia2: that is a lot of files.
19:42:55 <pleia2> oh, and speaking of d-g, I updated the docs a bit too so they're less wall-of-text: https://review.openstack.org/#/c/24727/
19:43:05 <pleia2> jeblair: heh, yeah, it's adding the new thing to their image creation script
19:43:21 <pleia2> which will have to be called by d-g instead of deploying devstack
19:43:22 <jeblair> pleia2: thanks for that!
19:44:17 <jeblair> #topic releasing git-review
19:44:26 <fungi> we should, i think
19:44:31 <mordred> yeah. I think it's about that time
19:44:51 <mordred> fungi: is there anything specific I can do to help you with the testing?
19:44:53 <fungi> there's a lot of bug fixes, error message clarifications, and some new features
19:45:09 <fungi> mordred: it's been getting tested all along at this point
19:45:15 <mordred> fungi: k. cool.
19:45:33 <fungi> though i still intend to write up a real non-network-dependent test harness for it
19:45:41 <mordred> fungi: can we perhaps land https://review.openstack.org/#/c/20452/ before we release?
19:45:45 <fungi> i have a model for how it'll work, just need to do it
19:45:47 <mordred> fungi: (I'll go fix/finish it)
19:45:55 <fungi> mordred: yes, please
19:46:19 <fungi> was wanting you to at least do whatever you wanted to do with that after saper added his input
19:47:25 <fungi> since he had more or less refactored it down into more exception handling, which clarkb had some design concerns over
19:48:27 <jeblair> #topic open discussion
19:48:29 <mordred> kk
19:48:56 <fungi> for a longer than usual agenda, we closed the list out quickly
19:49:15 <fungi> ttx: any infra stuff irking you this week?
19:49:31 <jeblair> well, 'releasing git-review' didn't turn out to be very contentious.  :)
19:49:36 <fungi> heh
19:49:53 <ttx> fungi: nope
19:50:08 <ttx> fungi: RC time is actually a bit less critical than feature freeze time
19:50:38 <jeblair> of course release-week is typically when jenkins security updates come out.
19:50:59 <zaro> anybody need my help with anything?  i'm just waiting on code reviews and i don't think the gearman install to zuul.o.o will take very long.
19:51:00 <ttx> Ah! i can't wait for that to happen.
19:51:20 * ttx will be back in 70min for the release meeting
19:51:44 <fungi> zaro: i'm doing my best to review your outstanding changes, though java is not my strong suit
19:51:57 <pleia2> now that reviewday is pretty much done (we'll see in the next hour :)) I could also use a break from learning about d-g and baremetal, so feel free to ping me as needed for tasks
19:52:08 <zaro> fungi: thanks.
19:52:52 <fungi> hopefully soon we'll get that caught up to the version you're testing on jenkins-dev
19:53:00 <zaro> fungi: if nothing else.  i'll spend some more time adding automated tests to plugin.
19:54:07 <jeblair> going once...
19:54:41 <jeblair> thanks everyone!
19:54:44 <jeblair> #endmeeting