19:01:50 <clarkb> #startmeeting infra
19:01:51 <openstack> Meeting started Tue Feb  5 19:01:50 2013 UTC.  The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:01:52 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:01:54 <openstack> The meeting name has been set to 'infra'
19:02:31 <fungi> old business?
19:02:41 <clarkb> on the agenda for today: CLA, wiki, logs, reviewday, jenkins plugins, and whatever else we want to talk about
19:02:51 <fungi> sounds goof
19:02:57 <fungi> good too
19:03:03 <clarkb> #topic CLA
19:03:18 <clarkb> fungi: how are things in redoing the CLA land?
19:03:33 <fungi> no news is good news. also known as "i've been busy with other things"
19:03:47 <fungi> we're on track
19:04:05 <fungi> toddmorey and i keep missing each other to exchange the new key, but other than that good
19:04:11 <clarkb> the dummy endpoint for CLA checking landed on review-dev correct?
19:04:20 <fungi> a few weeks back, yeah
19:04:59 <clarkb> other than exchanging a key is there any other work we should keep on our radar?
19:05:05 <fungi> annegentle posted a follow-up to the ml, but nobody seems to have piped up past that
19:05:31 <fungi> the actual cut-over will be happening on the 24th of this month, so keep that in mind i guess
19:05:50 <fungi> i'll remind everybody to get some reviews done on the pending wip patches when we get closer
19:06:07 <clarkb> ok sounds good to me
19:06:10 <fungi> they'll likely need a rebase between now and then anyway
19:06:28 <clarkb> I will do my best to write changes that conflict with yours :)
19:06:33 <fungi> anyway, unless there are questions, we can move on to the next topic
19:06:40 <clarkb> none from me.
19:06:51 <fungi> though i think we skipped the action items from last week
19:07:13 <fungi> not that it probably matters much since most of that's on the agenda anyway
19:07:22 <fungi> given that we never revised teh agenda ;)
19:07:39 <clarkb> #link http://eavesdrop.openstack.org/meetings/infra/2013/infra.2013-01-29-19.01.html <- last weeks notes
19:08:00 <clarkb> I am going to skip the wiki for now because I don't think ryan_lane is here yet
19:08:05 <fungi> k
19:08:07 <clarkb> #topic quantal slaves
19:08:36 <fungi> i've got four up and running. our automation for launching new slaves has been fixed to accommodate quantal
19:09:03 <fungi> over the weekend i tested the master branch unit tests for all core and incubated projects on quantal slaves and they're fine
19:09:08 <clarkb> I think the tripleo folks have volunteered to guinea pig for us
19:09:30 <fungi> i need to update jclouds to be capable of adding extra quantals
19:09:48 <fungi> but we can press forward before that as long as we don't switch too many projects over
19:09:54 <clarkb> fungi: did you want to go ahead and propose changes that migrate projects one by one to quantal? (or in groups smaller than doing all at once)
19:10:24 <fungi> i assume we want buy-in from the core devs on each project, right?
19:10:42 <clarkb> yeah I would have them +1 the change (or get the PTL to)
19:10:47 <fungi> so probably separate changes, that way they don't block progress if some projects are slow reviewing
19:11:07 <fungi> i can batch the server and client projects together
19:11:13 <fungi> as pairs
19:11:14 <clarkb> ++
19:11:39 <fungi> i'll add jclouds first though, so that we can be lazy about adding persistent slaves if needed
19:12:02 <clarkb> sounds good to me
19:12:39 <clarkb> any questions about quantal slaves?
19:13:08 <clarkb> #topic Jenkins Build Logs
19:13:24 <fungi> also known as how to lose a monday
19:13:29 <pleia2> heh :)
19:13:56 <clarkb> yesterday (Monday) static.o.o which hosts the logs.o.o vhost decided it would run out of disk space
19:14:27 <fungi> #link http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=309&rra_id=all
19:14:38 <clarkb> this caused the gate to effectively grind to a halt as Jenkins could not copy build logs to the log server and zuul interpreted these failures as job failures
19:15:22 <clarkb> fungi managed to run the compression job aggressively to keep things semi sane but compressing things and cleaning up old unneeded cruft wasn't helping much
19:15:30 <fungi> #link https://bugs.launchpad.net/openstack-ci/+bug/1115391
19:15:32 <uvirtbot> Launchpad bug 1115391 in openstack-ci "Root filesystem full on static.openstack.org" [High,In progress]
19:16:49 <clarkb> after looking at the problem more closely we realized that the tempest full runs were producing 42MB syslog which was being duplicated by screen logs (all uncompressed). That plus the other logs we were capturing put each devstack tempest full run at near 100MB of log data
19:17:28 <clarkb> jenkins was not compressing this data before copying it and each change runs 2 to 3 of these jobs. tl;dr it didn't take many jenkins jobs to eat up gigs of data before things got compressed
19:17:53 <fungi> and we normally only compress on static.o.o every 4 hours
19:18:10 <fungi> the monday rush overran that rapidly
19:18:47 <clarkb> the temporary solution is we are no longer copying the openstack service screen logs as syslog duplicates that data, and we are compressing the files before jenkins copies them to the log server
19:19:11 <clarkb> vishy pointed out that syslog truncates messages so I have written a change to increase the MaxMessageSize in rsyslog on our slaves
19:19:22 <clarkb> #link https://review.openstack.org/#/c/21233/
19:20:01 <clarkb> this should keep things reasonable while we sort out a long term solution to this problem
19:20:50 <clarkb> long term options include: growing the disk space on static.o.o, storing all logs in swift, rotating logs more aggressively (there are many ways of doing this), and I am sure we can come up with many more
19:21:22 <clarkb> #action clarkb start discussion on long term log archival options when jeblair gets back
19:21:41 <fungi> in the meantime we've got about 4gb free right now
19:22:01 <fungi> we may need to come up with something we can purge between now and next week, depending
19:22:27 <clarkb> and if the gate queue ever quiets down we may just go ahead and grow that disk
19:22:54 <fungi> my only concern there is that it comes with a bigger vm all around, and we can't as easily scale back
19:23:05 <fungi> but it definitely is an easy way out
19:23:23 <pleia2> well, growing it a little now could give some breathing room (certainly isn't a long term solution)
19:23:28 <pleia2> 4G doesn't seem like a lot right now
19:23:29 <clarkb> yeah, and we are trying to move more items to static.o.o so I don't think making it a bigger server is a bad idea in general
19:23:44 <fungi> i'll buy that
19:23:53 <fungi> next step up is 4g ram and 160g disk
19:24:04 <fungi> right now it's at 2 and 80
19:24:59 <clarkb> anything else on logs?
19:25:19 <clarkb> #topic reviewday
19:25:40 <pleia2> so, last week I cleaned up the code so now it passes pep8 and pyflakes tests
19:26:02 <pleia2> also made it so it can take a command line argument to specify the output directory
19:26:19 <pleia2> puppetizing is now in progress
19:26:51 <pleia2> planning on putting this on static.o.o at the moment (it's small and doesn't really grow)
19:27:24 <pleia2> that's all I've got
19:27:35 <clarkb> pleia2: are there outstanding changes that need review?
19:27:49 <clarkb> I have been particularly bad keeping up with the review queue lately (will try to fix that today)
19:28:05 <pleia2> clarkb: I checked in a change, but it's more for sharing with fungi at the moment to fill in the missing pieces (not ready for proper review)
19:28:18 <clarkb> ok
19:28:32 <fungi> though i saw pabelanger had some suggestions on it already, so you'll want to have a look at those
19:28:46 <pleia2> yeah, most of those are known, but it is helpful
19:28:49 <fungi> and i'll start going over it after the meeting, before dinnertime
19:28:56 <pleia2> much appreciated
19:28:59 <clarkb> cool
19:29:04 <clarkb> #topic Jenkins Plugins
19:29:07 <zaro> i need some of my java stuff reviewed if anyone wants to do it.
19:29:30 <clarkb> #link https://review.openstack.org/#/q/project:openstack-infra/gearman-plugin,n,z
19:29:52 <fungi> zaro is a.k.a. khaido i guess. i miss nick changes sometimes
19:29:55 <clarkb> trying to get the number of outstanding changes down to a reasonable number to simplify git management
19:29:57 <clarkb> fungi: yes
19:30:30 <zaro> sorry didn't inform about nick change. but it's true
19:30:37 <clarkb> zaro demoed gearman running jobs across multiple jenkins masters for me. it is pretty awesome
19:31:04 <fungi> nice!
19:31:15 <fungi> scalability, here we come
19:31:16 <zaro> i've been holding off further changes until reviews come back.
19:31:35 <zaro> working on documenting it and further tests.
19:31:51 <clarkb> zaro has also started a gearman puppet module. so there is an assortment of related items around the gearman plugin that are ready for review
19:32:14 <fungi> i'll try to take a closer look in the next couple of days. i've also been very slack on reviews of late
19:32:25 <clarkb> me too.
19:33:09 <clarkb> I skimmed jeblair's LCA talk and he mentioned the ZMQ plugin there. And implied it would be available through the normal openstack-infra channels. So I will try getting that into gerrit sometime in the near future too
19:33:47 <fungi> jeblair's already releasing your future work. awesome
19:33:50 <zaro> have a link to his talk?
19:34:05 <pleia2> zaro: http://mirror.linux.org.au/linux.conf.au/2013/mp4/OpenStack_Zuul.mp4
19:34:05 <clarkb> the plugin is running on jenkins and jenkins-dev and is still spewing events so I think it is working now
19:34:15 <pleia2> zaro: watched last night, it's great :)
19:34:53 <fungi> and that gets us a step closer to logstash indexing of job logs, right?
19:35:12 <clarkb> fungi: yes, it makes the notification of "hey logs are available" very simple
19:35:19 <zaro> pleia2: got a link to talk?
19:35:25 <pleia2> zaro: the mp4 link above
19:35:43 <zaro> pleia2: opps.  thanks.
19:36:04 <clarkb> fungi: jeblair was suggesting zuul may use it too
19:36:18 <fungi> oh, i can see where that would be helpful, yes
19:36:38 <clarkb> still no ryan_lane but I think we should move onto talking about the wiki move
19:36:45 * annegentle is here
19:36:51 <clarkb> #topic wiki
19:37:09 <clarkb> olaph: how are things?
19:37:15 <olaph> the underlying skin on wiki-staging.o.o is different from the one I was using, but I've ported over all the style changes required for 'v1'
19:38:26 <clarkb> https://wiki-staging.openstack.org is not returning bytes to me. is that expected?
19:38:49 <olaph> https://wiki-staging.openstack.org/wiki/Main_Page
19:38:56 <fungi> i got a cert from it, but yeah, no content
19:39:12 <clarkb> ooh shiny
19:39:28 <clarkb> we should probably make apache forward you to /wiki/Main_Page when you hit /
19:39:31 <olaph> I would assume that would resolve correctly when they go live...  :)
19:39:35 <fungi> timing out getting a response after the handshake
19:39:36 <clarkb> ah
19:39:38 <pleia2> nice
19:40:26 <clarkb> annegentle: olaph I know jeblair requested we move the migration up a day
19:40:27 <annegentle> looking good
19:40:45 <clarkb> I was hoping ryan_lane would be here to chime in on that
19:41:04 <pleia2> he works next door to where I live, shall I go knock? :)
19:41:38 <clarkb> pleia2: I won't stop you, but I don't think it is necessary :)
19:41:44 <clarkb> annegentle: how do you feel about that?
19:41:55 <annegentle> no problem from me, either day works
19:42:00 <ttx> clarkb: o/
19:42:00 <clarkb> I think both fungi and I are available on the 17th if we want to stick wit hthe original day that everyone agreed to
19:42:09 <fungi> yeah, i'm cool either way
19:42:10 <clarkb> ttx: heelo
19:42:29 <ttx> olaph: nice work on the theme
19:42:47 <clarkb> ok sounds like either day works for most people. if we can get a hold of ryan today and he is ok with the 16th then I think we should move it
19:42:50 <clarkb> ttx: ^ that work with you?
19:43:09 <ttx> I'll be in jetlag mode but yes, works for me
19:43:21 <fungi> if we do that, i guess we also need to follow up to the original announcement on the ml
19:43:31 <clarkb> fungi: yes we will need to send an update
19:43:42 <annegentle> yeah makes sense, I'll send an update as needed
19:44:14 <clarkb> is anyone aware of any additional outstanding items? I believe image upload works and the template looks good
19:45:25 <clarkb> I guess not
19:45:45 <clarkb> #action clarkb to find out if ryan_lane is ok with migrating the wiki on Feb 16 instead of Feb 17.
19:45:59 <clarkb> #topic open discussion
19:46:45 <annegentle> #action annegentle to send update to mailing list if moved to Feb 16
19:46:51 <ttx> clarkb: was wondering if we could do something to get python-swiftclient 1.3.0 to Pypi
19:47:01 <ttx> been chasing mordred about it to not avail
19:47:13 <clarkb> ttx: that was the tag that didn't push properly because the jenkins job was off?
19:47:38 <ttx> that was the tag that didn't push properly because a script was missing
19:47:54 <clarkb> yes, we can manually push that to pypi
19:48:09 <mordred> oh, sorry
19:48:16 <clarkb> mordred: did you want to do that?
19:48:24 * ttx blames mordred for not doing it earlier
19:48:40 <mordred> clarkb: I can, unless someone beats me to it
19:48:55 <mordred> clarkb: I was going to re-trigger the job in jenkins to re-test it
19:49:00 <clarkb> #action mordred to upload python-swiftclient 1.3.0
19:49:02 <mordred> rather than pushing the tag manually
19:49:06 <clarkb> ++ to retriggering jenkins job
19:49:24 <ttx> mordred: while you're in acceptance mode, remember to push new versioning code to projects asap
19:49:41 <mordred> ttx: https://review.openstack.org/#/q/status:open+branch:master+topic:oslo-version,n,z
19:49:54 <ttx> mordred: cool, thx
19:49:59 <fungi> mordred: any news on the rhel slave licensing situation?
19:50:21 <mordred> fungi: I learned new things at LCA, but need to follow up on them
19:50:29 <fungi> okay, awesome
19:52:05 <clarkb> we will probably want to follow http://people.canonical.com/~ubuntu-security/cve/2013/CVE-2013-0169.html
19:52:06 <uvirtbot> clarkb: ** RESERVED ** This candidate has been reserved by an organization or individual that will use it when announcing a new security problem.  When the candidate has been publicized, the details for this candidate will be provided. (http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2013-0169)
19:52:19 <ttx> mordred: also I'll probably start up summit.openstack.org on the same cloudserver instance as for past summits
19:52:34 <ttx> mordred: couldn't push it to infra proper and now time is running very short
19:52:40 <mordred> ttx: ok
19:52:50 <ttx> mordred: I kinda want to do it myself as an exercise
19:52:57 <ttx> so I'll do that early in H.
19:53:07 * ttx remembers saying that 6 months ago too.
19:53:22 <mordred> :)
19:53:27 <clarkb> we could trap ttx in a bar in portland and make him do it then
19:54:05 <clarkb> anything else?
19:54:30 <clarkb> #endmeeting