19:02:31 <jeblair> #startmeeting infra
19:02:32 <openstack> Meeting started Tue Dec 17 19:02:31 2013 UTC and is due to finish in 60 minutes.  The chair is jeblair. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:02:33 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:02:36 <openstack> The meeting name has been set to 'infra'
19:02:36 <jeblair> #link https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting
19:02:41 <jeblair> #link http://eavesdrop.openstack.org/meetings/infra/2013/infra.2013-12-10-19.02.html
19:03:03 <fungi> jeblair: before you go reassigning the tarballs action items to yourself, don't. almost done...
19:03:43 <jeblair> fungi: oh cool, thanks.  :)
19:03:54 <clarkb> ninja fungi
19:04:07 <jeblair> fungi: how's the quota increase request?
19:04:27 <jeblair> #topic actions from last meeting
19:04:34 <SergeyLukjanov> o/
19:04:35 <fungi> done. our openstackci account can go up to 25tb in rackspace now, and up to 100 cinder volumes
19:04:41 <clarkb> fungi: awesome
19:04:41 <jeblair> fungi: yaaay!
19:04:53 <anteaya> fungi: woohoo
19:04:54 <fungi> i've added a 200gb volume for tarballs on static.o.o
19:04:59 <fungi> rsync'd the contents in
19:05:01 <anteaya> what did you say in the request?
19:05:03 <fungi> checked out the vhost
19:05:34 <fungi> lowered ttl on the dns record to 5 minutes
19:05:34 <fungi> one minor cosmetic issue outstanding... can't get the new filesystem usage to show up in cacti
19:05:54 <fungi> restarted snmpd on static, re-ran the graph creation scripts on cacti manually, no good
19:05:59 <jeblair> fungi: cool, so i think next maybe just put jenkins.o.o in shutdown mode so it doesn't generate new tarballs, then do an rsync/dns switch
19:06:33 <fungi> agreed. catch-up rsyncs are taking on the order of 10 seconds looks like, so should go quickly
19:06:41 <clarkb> ++
19:07:07 <fungi> #action fungi move tarballs.o.o to static.o.o
19:07:40 <fungi> sligtly closer to tearing down the old wiki server
19:07:46 <jeblair> almost there!
19:07:50 <jeblair> #topic Tripleo testing (lifeless, pleia2)
19:08:12 <jeblair> pleia2: anything to coordinate on here?
19:08:31 <pleia2> I don't think so
19:08:43 <jeblair> cool
19:08:51 <pleia2> I now have derekh's setup to test, but that's more on my side than infra just yet
19:09:54 <jeblair> pleia2: do you know if anyone has volunteered/been assigned to do ipv6 nodepool/jenkins work?
19:10:12 <pleia2> jeblair: afaik, no one yet
19:10:16 <jeblair> k
19:10:22 <jeblair> #topic Savanna testing (SergeyLukjanov)
19:10:25 <SergeyLukjanov> hey
19:10:32 <jeblair> SergeyLukjanov: anything new here?
19:10:43 <SergeyLukjanov> everything is ok, waiting for review for tempest patches
19:10:50 <SergeyLukjanov> nothing new atm
19:11:09 <clarkb> SergeyLukjanov: the jobs are running though correct?
19:11:15 <clarkb> SergeyLukjanov: they just don't actually test much yet
19:11:16 <SergeyLukjanov> yup!
19:11:32 <SergeyLukjanov> only api for node group templates
19:11:38 <SergeyLukjanov> waiting for review
19:11:45 <jeblair> that's the best way to go -- things will be self-testing as they go into tempest
19:11:49 <SergeyLukjanov> and then will add test for the rest api edpoints
19:12:14 <SergeyLukjanov> hope to receive some reviews this week
19:12:23 <SergeyLukjanov> tempest guys are very busy as I see
19:12:32 <anteaya> tempest people
19:13:11 <jeblair> SergeyLukjanov: cool, thanks
19:13:11 <SergeyLukjanov> btw we're starting using zuul+nodepool to run savanna-ci and I hope that will return back with some patches to support neutron in nodepool
19:13:39 <jeblair> SergeyLukjanov: yeah, that'd be great
19:13:58 <SergeyLukjanov> changes are pretty small atm
19:14:15 <SergeyLukjanov> and I'd like to start discussion about dib jobs
19:14:27 <SergeyLukjanov> but I'm not prepared atm, so, let's do it offline
19:14:36 <SergeyLukjanov> I'll try to prepare some initial queestions
19:14:36 <jeblair> (though i hope you don't have to run savanna-ci much longer as we move things into openstack)
19:14:59 <SergeyLukjanov> jeblair, we'll need it to run slow tests
19:15:10 <SergeyLukjanov> like sequential scaling of clusters
19:15:32 <jeblair> well, as much as we can :)
19:15:41 <SergeyLukjanov> yep :)
19:15:54 <SergeyLukjanov> we'd like to have at least all tests in tempest
19:16:07 <SergeyLukjanov> and run them if needed in savanna-ci but from tempest
19:16:20 * fungi imagines a 100-node hadoop cluster being spun up for each change
19:16:35 <SergeyLukjanov> :)
19:16:37 <jeblair> fungi: we'll need you to write more nice quota requests
19:16:42 <SergeyLukjanov> we've tested 200 nodes clusters
19:16:45 <jeblair> #topic  Trove testing (mordred, hub_cap)
19:17:01 <hub_cap> heyo jeblair
19:17:02 <jeblair> hub_cap: heya!
19:17:21 <hub_cap> so SlickNik has updates (hes working on the dib elements)
19:17:25 <SlickNik> hey guys.
19:17:34 <hub_cap> #link https://blueprints.launchpad.net/trove/+spec/trove-tempest
19:17:40 <anteaya> people
19:17:54 <hub_cap> i think next hes going to work on the image caching, right SlickNik? ;)
19:18:06 <SlickNik> I didn't have much of a chance to work on this last week, but I'm going to be working on this 100% this week.
19:18:34 <SlickNik> Yup image caching and devstack-vm-gate changes to run the tests.
19:19:25 <SlickNik> A couple of other folks from the trove team signed on to get started moving trove integration tests to tempest.
19:19:34 <hub_cap> and we have some people from mirantis working on server side tests, and we have some client tests in a review (iirc) already
19:20:23 <jeblair> SlickNik, hub_cap: yes tarballs.o.o is where we will stick images we build
19:20:38 <SlickNik> flying-bond (Debashish) and dlakunchikov (Dmitri)
19:21:36 <hub_cap> horray for progress
19:22:03 <jeblair> hub_cap: sounds good; any questions or blockers atm?
19:22:22 <hub_cap> none from myself
19:22:26 <hub_cap> <3
19:22:29 <SlickNik> jeblair: none at the moment. I'll likely be bugging people for reviews this week, so stay tuned!
19:22:43 * hub_cap turns a prop radio nob
19:22:54 <jeblair> cool, looking forward to it!
19:23:04 <jeblair> #topic  Jenkins 1.540 upgrade (zaro, clarkb)
19:23:16 <jeblair> so that happened, briefly, then unhappened.
19:23:43 <anteaya> the reason for the unhappening was lost or truncated logs, was it not?
19:23:44 <zaro> so i'm trying to setup latest jenkins with scp plugin to see what happened there.
19:23:45 <clarkb> ya it was sad
19:24:00 <jeblair> zaro: cool.  clarkb and i have both worked on that plugin
19:24:10 <clarkb> anteaya: correct, new version of jenkins didn't play nice with teh scp plugin console copying
19:24:16 <anteaya> ah
19:24:18 <anteaya> :(
19:24:59 <jeblair> zaro: i think if you write a job that emits 10 or 20k lines to the console, that will probably be enough to replicate
19:25:36 <zaro> yeah i have it setup in my dev env now, but having difficulties even getting plugin to connect to a server.
19:25:54 <zaro> still working on it.
19:26:10 <jeblair> #topic Maven clouddoc plugin move (zaro, mordred)
19:26:33 <zaro> #link https://etherpad.openstack.org/p/java-release-process
19:26:50 <zaro> haven't heard from sharwell since last wedn. 12/11.
19:27:09 <zaro> should we just go ahead with this? #link https://review.openstack.org/#/c/58349/
19:27:31 <fungi> it looks like i need to go into the sonotype jira and open a case requesting a dedicated groupId, based on subsequent info from dcramer
19:28:00 <zaro> fungi: yes, that does need to happen
19:28:06 <fungi> zaro: so i think that brings us back to the etherpad i originally prepopulated with all the info they want in the jira ticket fields
19:28:23 <anteaya> so is the situation that you were coordinating with someone and now someone else is involved in the process, with no access to the prior person?
19:28:28 <fungi> need to figure out all the little details about our org.openstack.cloud.api
19:29:05 <zaro> fungi: i think you'll need to coordinate with sharwell on those fields.
19:29:34 <zaro> according to dcramer sharwell can provide access.
19:29:38 <fungi> okay, i guess they need to match what's on org.rackspace.cloud.api?
19:29:53 <fungi> er, com.rackspace
19:30:16 <zaro> ohh, wait that's right this is a new groupId.
19:30:17 <fungi> i'll find out
19:30:27 <zaro> then i think you can just make it your own.
19:30:34 <fungi> yeah, we have to ask sonotype to create it in maven central
19:30:35 <zaro> i mean create it like new.
19:31:03 <zaro> yes, you can probably create without sharwell or dcramer input then.
19:31:05 <fungi> okay. do we request org.openstack.cloud.api or just org.openstack and then get the ability to create sub-ids i wonder
19:31:17 <fungi> i'll check with them
19:31:23 <zaro> i think former.
19:32:04 <zaro> ohh definately former.  cannot create subs.
19:32:22 <jeblair> #topic  Private gerrit for security reviews (zaro, fungi)
19:32:44 <jeblair> (the zaro-fungi part of the meeting continues)
19:32:46 <zaro> just got good feedback from fungi on the change.
19:33:01 <zaro> yes, nothing new ATM, just WIP
19:33:15 <fungi> sorry it's taken me so long to find time to go over it, but i think it's close to what we need
19:33:27 <zaro> good to hear!
19:33:38 <fungi> probably worth bringing to the group is whether we want to start it on latest gerrit rather than giving ourselves yet one more gerrit to upgrade from 2.4
19:33:53 <clarkb> I would be all for starting it on new gerrit
19:33:53 <zaro> i think fungi mentioned that we should wait until 2.8 upgrade.
19:34:00 <zaro> or after 2.8 upgrade
19:34:22 <fungi> well, or just build it on 2.8 (there's not a lot special it really needs for the workflow we outlined)
19:34:23 <zaro> yeah, ++
19:34:31 <jeblair> yeah, i think for the moment we can say we'll target the rollout of security after we deploy 2.8
19:34:45 <fungi> but i'm fine with prioritizing the upgrade project, given limited resources
19:34:51 <jeblair> i don't think we should try to run it on 2.8 while we're running regular gerrit on 2.4
19:34:52 <fungi> yep, totally agree
19:35:06 <zaro> ++
19:35:36 <jeblair> though since we don't know for certain everything that will be involved in the 2.8 upgrade and timeline yet, we should feel free to revisit that...
19:36:03 <jeblair> if it looks like it'll be 3 months till we upgrade and security is ready to go, it'd probabl be better to go ahead and deploy security on 2.4 and upgrade it too.
19:36:19 <fungi> okay
19:36:37 <zaro> cool.
19:36:44 <jeblair> #topic  Upgrade gerrit (zaro)
19:36:46 <jeblair> speaking of
19:37:06 <zaro> Blueprint https://blueprints.launchpad.net/openstack-ci/+spec/gerrit-2.8-upgrade
19:37:29 <zaro> Etherpad #link https://etherpad.openstack.org/p/gerrit-2.8-upgrade
19:37:40 <zaro> jeblair: had a question in there about alternative to WIP plugin.
19:38:13 <zaro> also I’m blocked waiting for approval on #link https://review.openstack.org/#/c/61542/
19:38:27 <jeblair> _david_ wrote up some text about the upgrade, so i copied it into the etherpad
19:38:30 <jeblair> #link https://etherpad.openstack.org/p/gerrit-2.8-upgrade
19:38:33 <jeblair> #link https://blueprints.launchpad.net/openstack-ci/+spec/gerrit-2.8-upgrade
19:38:40 <jeblair> and then annotated it with some of my thoughts
19:38:56 <jeblair> fungi, clarkb: ^ that's probably worth a read over and your initial feedback too
19:39:03 <fungi> adding to my list
19:39:05 <jeblair> it has some deployment choices
19:39:11 <clarkb> jeblair: ok bookmarking
19:39:24 <fungi> zaro: on 61542 i think we were waiting for mordred to chime in, but he's been absent for a few days
19:39:41 <jeblair> yeah, if he doesn't vote this afternood, let's aprv
19:39:49 <jeblair> afternoon
19:40:14 <clarkb> wfm
19:40:20 <jeblair> i'd like to continue the tradition of unanimous approvals of ssh access if we can.  :)
19:40:51 <fungi> agreed
19:41:08 <jeblair> #topic  Zuul release (2.0?) / packaging (pabelanger)
19:41:16 <jeblair> this might be stale...
19:41:22 <jeblair> and pabelanger isn't here...
19:41:27 <jeblair> #topic  Open discussion
19:41:36 <pleia2> if I could get feedback here, that would be useful: http://lists.openstack.org/pipermail/openstack-infra/2013-December/000515.html
19:41:38 <zaro> can we circle back to clouddocs?
19:41:49 <pleia2> working through publications, but we need branch names that make sense
19:42:15 <jeblair> pleia2: eek, i missed that mail, sorry.
19:42:20 <zaro> not sure i got an answer whether we should just go ahead with  https://review.openstack.org/#/c/58349/
19:42:35 <jeblair> zaro: i think we should sit on it for now.
19:42:40 <pleia2> and I also confirmed that we have all history from https://github.com/openstack-ci/publications so it can be deleted
19:42:51 <zaro> jeblair: np
19:43:37 <fungi> pleia2: i think the concern originally expressed was that until we move those into branches in the new location (and out of old git commits in the history) they're not exposed anywhere easily consumable
19:44:01 <pleia2> fungi: fair enough, so we'll have that problem solved soon
19:44:07 <jeblair> yeah, so let's keep ci/pub around until we finish the other branches
19:44:13 <jeblair> and then delete
19:44:14 <fungi> i think it's safe to hold off deleting from github until then
19:45:16 <jeblair> so it turns out that crm114 adds enough time to log processing that the workers got backlogged
19:45:38 <jeblair> i'm working on a logstash worker puppet module refactor that will let us colocate multiple logstash workers on a single host
19:45:57 <jeblair> to better utilize cpu there -- especially once we move the workers to rax performance nodes
19:45:59 <fungi> oh, one other thing which sprang to mind for the tarballs move. the target path changes slightly on the new server, so i'll need to tweak the publisher location on jenkins.o.o for it after it quiesces
19:46:23 <jeblair> and we'll add some more nodes as well
19:46:40 <jeblair> it would be swell if we could graph the gearman queue...
19:46:54 <jeblair> clarkb: maybe we could have the log client splat that to statsd/graphite?
19:47:05 <clarkb> jeblair: right I was thinking of adding that feature to geard directly
19:47:14 <clarkb> jeblair: unless yo uthink that is better off living external
19:47:44 <jeblair> fungi: ok, is that a change to the publishers in jobs, or is it a change to the scp site in the global config?
19:47:53 <zaro> fungi: will all the jjb jobs refer to static instead of tarballs.o.o now?
19:47:55 <fungi> jeblair: the latter
19:47:57 <jeblair> clarkb: hrm; adding it to geard has a certain elegance
19:48:21 <fungi> zaro: they won't. the jobs stay the same because the publisher target stays the same
19:48:33 <clarkb> jeblair: yeah may be generally useful to other geard users
19:48:49 <fungi> zaro: jeblair: it's the "Root Repository Path" which i'll need to update
19:49:12 <jeblair> clarkb: yep.  we probably _don't_ want it for zuul though.
19:49:58 <jeblair> oh, and i've proposed two changes to zuul that should allow us to start using templates in layout.yaml will will make it much smaller
19:49:59 <jeblair> https://review.openstack.org/#/q/status:open+project:openstack-infra/zuul,n,z
19:50:27 <fungi> saw the titles, haven't had time to review yet but very excited by the promise they make
19:50:45 <zaro> ohh that would be nice!
19:51:31 <fungi> oh, and stable/havana backports of the tox.ini sync are proposed now... https://review.openstack.org/#/q/branch:stable/havana+topic:tox-sync,n,z
19:52:05 <fungi> mostly working, sdague and mtreinish helped me on missing/broken prereqs in devstack and tempest
19:52:14 <jeblair> fungi: cool
19:52:54 <jeblair> anyone have anything else?
19:52:56 <clarkb> fungi: is grizzly affected?
19:53:24 <fungi> clarkb: grizzly affected grenade upgrades to the havana patches, so there was some involvement there
19:53:39 <fungi> for tempest anyway
19:54:06 <clarkb> thank
19:54:11 <fungi> though havana and grizzly stable branches of most of the servers are back to being testable again as of this week
19:54:18 <fungi> finally
19:55:24 <clarkb> I don't have anything else
19:55:42 <zaro> yep, all done
19:55:59 <jeblair> thanks all!
19:56:01 <fungi> nothing else for me
19:56:02 <jeblair> #endmeeting