19:02:19 <jeblair> #startmeeting infra
19:02:20 <openstack> Meeting started Tue Dec  3 19:02:19 2013 UTC and is due to finish in 60 minutes.  The chair is jeblair. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:02:21 * mordred hands morganfainberg his voice
19:02:21 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:02:21 <morganfainberg> mordred is more useful i'm sure in that regard
19:02:24 <openstack> The meeting name has been set to 'infra'
19:02:33 <jeblair> #link https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting
19:02:33 <jeblair> #link http://eavesdrop.openstack.org/meetings/infra/2013/infra.2013-11-26-19.02.html
19:02:51 <tristanC> aww, I thought there was going to be an open meeting (for keystone) at the end :(
19:03:03 <jeblair> #topic agenda
19:03:05 <morganfainberg> tristanC, come chat w/ us in #openstack-dev
19:03:20 * morganfainberg quiets up so Infra can do awesome stuff.
19:03:36 <jeblair> because last week we did not get through the whole agenda
19:03:43 <jeblair> and also there was this holiday in the us...
19:03:52 * coolsvap is away: Away!!!
19:04:05 <jeblair> i think we should skip things covered last week and start with things we did not get to.
19:04:20 <jeblair> #action jeblair file bug about cleaning up gerrit-trigger-plugin
19:04:20 <jeblair> #action jeblair move tarballs.o.o and include 50gb space for heat/trove images
19:04:20 <jeblair> #action fungi grow logs and docs-draft volumes so they're 50% full
19:04:20 <jeblair> #action fungi update docs for static to recommend 1t cinder volumes
19:04:20 <jeblair> #action mordred to harrass hub_cap until he's writen the tempest patches
19:04:37 <jeblair> #topic  Maven clouddoc plugin move (zaro, mordred)
19:04:56 <jeblair> zaro, mordred: you're up first! :)
19:05:07 <mordred> the repo moved
19:05:14 <clarkb> \o/
19:05:26 <zaro> ok. patch is up for review.
19:05:50 <zaro> #link https://review.openstack.org/#/c/46099
19:06:03 <zaro> #link https://review.openstack.org/#/c/47937
19:06:31 <jeblair> i think that's a future topic
19:06:33 <mordred> I find this work exciting, btw
19:06:34 <zaro> opps previous one was wrong. it's https://review.openstack.org/#/c/58349/
19:07:10 <zaro> ohh and i think we'll need core to create maven nexus accounts for this to work.
19:07:32 <zaro> #link https://review.openstack.org/#/c/58349/
19:07:45 <jeblair> zaro: can you point us at how to create those accounts?
19:08:07 <zaro> I believe you just go onto oss.sonotype.org and create an account.
19:08:22 <clarkb> its a bit more tricky than that
19:08:25 <jeblair> zaro: is dwcramer up to speed?  he knows the repo has been moved?
19:08:30 <clarkb> as their release process requires gpg signing of things
19:08:35 <jeblair> zaro: i don't see any reviews from him...
19:08:37 <zaro> jeblair: yes, been working with him.
19:08:58 <zaro> jeblair: what review were you expecting?
19:09:03 <clarkb> but yes, you go to the sonotype site, submit a jira ticket ! and eventually you get an account. I was hoping for more solid footing on the gpg stuff before doing that though
19:09:05 <anteaya> o/
19:09:18 <jeblair> clarkb: so jenkins needs to gpg sign artifacts before uploading?
19:09:27 <clarkb> yes
19:09:38 <jeblair> ok.  i think we have a gpg key for jenkins, btw.
19:09:38 <clarkb> well something needs to sign it with a key that upstream trusts
19:10:11 <clarkb> cool I didn't know that
19:10:12 <zaro> jeblair: yes, i sign the package when i do a release.
19:10:18 <jeblair> clarkb: i believe that would be jenkins, in our system, right?
19:10:50 <jeblair> we're talking about tag based versioning, right?  where jenkins creates an artifact and uploads it based on a tag in gerrit...
19:10:53 <clarkb> jeblair: yes, unless fungi has sorted out a better way of doing it
19:11:01 <jeblair> zaro: i don't think there's an opportunity there for you to sign something.
19:11:18 <clarkb> I know fungi had ideas, but having jenkins sign is most straightforward
19:11:36 <clarkb> zaro: jeblair: right, individual signs the git tag, jenkins signs the package
19:11:41 <fungi> one proposed alternative is to add a manual step for committing a detached signature to a repo and then having automation key off that change merging to do the upload of the package and sig together
19:11:41 <zaro> jeblair: ohh.  i seem to remember having to enter my password or something upon creating a tag.
19:12:00 <jeblair> zaro: yes, the _tag_ should be signed.
19:12:03 <jeblair> by you
19:12:29 <jeblair> fungi: of what would the signature be?
19:12:36 * ttx lurks
19:12:39 <fungi> signature of the generated release artifact
19:12:47 <jeblair> fungi: roger
19:12:55 <SergeyLukjanov> mvn repo requires signed artifacts
19:13:09 <fungi> download the tarball or whatever it is, create a detached sig of it, and then upload a change containing that sig (to a separate signatures repo or branch or something)
19:13:47 <fungi> this was the suggested alternative (or addition) to having automated package signing
19:13:54 <zaro> clarkb: so what's the tricky bit again?  same process as the jenkins deployments right?
19:13:58 <jeblair> i think we should proceed with jenkins signing for the clouddocs repo, for expediency
19:14:13 <clarkb> zaro: the tricky bit is the signing, we can go the easy route or we can do fancy things that fungi suggests
19:14:26 <clarkb> but I don't want to setup accounts until we have at least a solid idea of which route we are going
19:14:31 <clarkb> jeblair: ++
19:14:33 <jeblair> fungi: because what you are discussing sounds more like something we want to do long-term for openstack releases
19:14:41 <mordred> I agree with jeblair
19:14:43 <fungi> well, i only suggest them as an alternative to people who have expressed concerns about the simpler way
19:14:58 <clarkb> right the simpler way seems like signing to have a signature
19:15:02 <clarkb> rather than signing to actually trust a thing
19:15:03 <fungi> i am cool with the automated solution
19:15:35 <jeblair> clarkb: well, it means you can trust that this thing was created by our jenkins.
19:15:58 <jeblair> clarkb: and put the appropriate level of trust in 'a thing created by our jenkins'.  which may not be high.  :)
19:16:00 <fungi> the upload slave can certainly provide some assertion. i already have a proposed mechanism in that bug, elegantly simple, to have it confirm the tag on the git repo matches the contents of the tarball it retrieves
19:16:48 <fungi> the slave can't assert that it is not compromised itself, but it can at least indicate whether the tarball and/or git repo have been tampered with
19:16:56 <mordred> I don't think the jenkins sig is perfect
19:16:57 <mordred> BUT
19:16:59 <mordred> it's better than no sig
19:17:06 <mordred> and it's sooner than a perfect human sig
19:17:13 <mordred> so we should at least do it
19:17:17 <jeblair> yup
19:17:21 <mordred> while we figure out a way to do the other thing thta doesn't kill us
19:17:21 <clarkb> ++
19:17:34 <jeblair> zaro: have you signed up a core person to get a nexus account?
19:17:36 <fungi> no question. and i think it's possible to tack on the human sig solution without significant retooling later
19:17:45 <mordred> ++
19:18:05 <zaro> jeblair: nope, looking for volunteers.
19:18:06 <clarkb> jeblair: zaro: I had looked at it at one point, and can look at it again. Unless someone more familiar with gpg wants to do it
19:18:19 <fungi> sign me up
19:18:30 <fungi> zaro: get me the specifics after the meeting
19:18:35 <zaro> fungi: you got it
19:18:41 <jeblair> #action fungi create maven nexus account
19:18:43 <zaro> fungi: cool
19:18:56 <jeblair> clarkb: fungi seemed to want it more.  :)
19:19:14 <jeblair> #topic  Private gerrit for security reviews (zaro, fungi)
19:19:32 <clarkb> :) np my brain melted a little the first time I looked at it (probably overthinking the signing problem)
19:19:38 <fungi> zaro: sorry on this one--i haven't caught up far enough in my review backlog to revisit the current patchset
19:20:17 <zaro> ohh.  i'm trying to remember todo list for this one.  but i think this one needs accounts as well.
19:20:43 <zaro> the bigger thing it needs is for us to figure out how the projects will work.
19:21:03 <fungi> you mean as far as git replication vs uploaded changes?
19:21:23 <zaro> clarkb says we should set acls in all-projects and have per-project acls
19:21:23 <fungi> or the manage-projects pieces
19:21:28 <zaro> that's a lot of work.
19:21:33 <clarkb> zaro: we do it today
19:21:52 <zaro> yea, seems like a lot of work.
19:21:54 <clarkb> its not that much work once you have it in place, the acls themselves don't change, just group membershio
19:22:25 <fungi> right, that part. so the trick was working out a mechanism to manage the all-projects acl outside of gerrit's interface, including bootstrapping it. and the current suggestion is to punt because that's how we do the prod one
19:22:29 <zaro> well.  if that's the way we do it now.  then i guess we replicate for private gerrit?
19:22:45 <fungi> yes
19:22:56 <fungi> i think so, anyway
19:23:17 <zaro> ok.  then i'll need to work with fungi to figure out all-project &  per-project acl settings.
19:23:27 <fungi> sounds good
19:23:32 <jeblair> i'm okay with punting for this
19:23:33 <zaro> then somehow get that all into config?
19:23:52 <jeblair> i'm hoping that some of the folks working on downstream "infras" will contribute all-projects bootstrapping
19:24:22 <fungi> zaro: yeah, i think it can be documented (the central bits) and then the per-project special acls can be kept in the config repo i think, applied via manage-projects like we do on prod
19:24:37 <zaro> ohh yea, also i remember looking at this and it seems like a lot of work change pp files to templates.
19:24:49 <jeblair> fungi, mordred: though after bootstrapping, can't we manage all-projects acls in the usual way?
19:25:03 <clarkb> jeblair: yes
19:25:05 <fungi> jeblair: yes, except for the part where it might lock you out
19:25:20 <jeblair> fungi: ssh?
19:25:33 <jeblair> fungi: more words:  in such a situation, we could fix it by shelling into the server, surely?
19:25:37 <fungi> since the all-projects acl also controls whether your automation account can push changes for acls
19:25:55 <jeblair> fungi: oh, and in that case we can even fix it using the webui.
19:26:02 <fungi> yes, we could ssh in and hand-edit the acl on disk and hope we get it correct (or revert it would make more sense)
19:26:42 <fungi> but correct, we could move that into config
19:26:50 <jeblair> zaro, fungi: maybe the security gerrit can guinea-pig managing all-projects in git?
19:26:58 <jeblair> zaro, fungi: and then we can extend that to regular gerrit
19:27:17 <clarkb> ++ I like that idea
19:27:29 <fungi> sure. an easy first step would be simply documenting the minimal initial acl needed to get manage-projects able to apply the full one
19:27:49 <zaro> why don't we guinea-pig in review-dev?
19:27:57 <jeblair> that would help the full bootstrapping process.
19:28:00 <jeblair> zaro: that's fine too.
19:28:20 <guitarzan> win 23
19:28:26 <zaro> i'de prefer to keep it seperate because to get all the acls right would be a large change.
19:28:49 <jeblair> zaro: what do you mean?
19:29:15 <zaro> i think the right way to do the per-project acls would be to template the current set of alc config files.
19:29:29 <fungi> though it's worth noting that there will be far fewer per-project acls for security reviewing. we can have one per program (not per project) receiving security support through the vmt, which is no more than a dozen in total
19:29:31 <zaro> i'm not even sure how to do that yet.
19:30:04 <zaro> ohh, that's a revelation.  then maybe it's not such a big change.
19:30:32 <fungi> i'd suggest a field in the jeepyb projects.yaml which indicates the name of the security acl, and then maybe a little tweak to manage-projects to determine which kind of acl to apply
19:30:50 <jeblair> fungi: sounds good.
19:30:54 <fungi> that's just off the top of my head. might be better alternatives
19:31:00 <jeblair> mordred: ^ you've been in this code lately; thoughts?
19:31:11 <clarkb> why not two different project lists?
19:31:27 <fungi> also doable, though results in more duplication
19:31:31 <clarkb> maybe that is too much trouble, but I don't see us putting stackforge projects on that server
19:32:00 <jeblair> clarkb: that's a good idea.... also, might we want to use the track-upstream feature?
19:32:05 <fungi> i think we can either ignore the extra repos, or we can simply specialize manage-projects and the one yaml file a little further
19:32:08 <jeblair> to have security-gerrit track regular gerrit?
19:32:33 <fungi> but tracking might be a good feature to leverage there instead of using one list, agreed
19:32:40 <clarkb> jeblair: I think we want to go the other way nad have regular gerrit update security gerrit
19:32:50 <clarkb> jeblair: but tracking may be more sane depending on workflow
19:32:57 <jeblair> clarkb: i think we said the same thing? :)
19:33:21 <clarkb> jeblair: replication vs tracking
19:33:22 <fungi> yeah, tracking branches on the security gerrit would pull in updates from public gerrit
19:33:30 <jeblair> clarkb: security gerrit is downstream of regular gerrit.
19:33:35 <clarkb> I prefer replication over tracking, but am not involved in the workflow
19:34:07 <jeblair> clarkb: can you elaborate on what you mean by replication vs tracking?
19:34:25 <fungi> replication also ends up sending the refs/for/stuff though, right? i mean, it can probably be ignored--i don't think it would cause collissions--but...
19:34:27 <clarkb> jeblair: I want review.o.o to replicate its repos to review-secure.o.o. So that review-secure.o.o is always up to date
19:34:44 <clarkb> jeblair: we might need to pick a crazy change number offset and that might still bite us, so not perfect
19:35:10 <jeblair> clarkb: isn't this what jeepyb's upstream-tracking is designed for?  to keep a downstream repo up-to-date with an upstream one?
19:35:15 <fungi> change numbers are only a concept inside the db (change-id headers are also unlikely to collide)
19:35:17 <clarkb> jeblair: sort of
19:35:31 <clarkb> jeblair: I am not a fan of tracking fwiw
19:35:47 <clarkb> it keeps certain branches in sync
19:36:02 <clarkb> but not the branches you want to work against in most cases, that is a manual step
19:36:10 <fungi> clarkb: which, for security support development, is probably fine
19:36:21 <clarkb> right it depends on the workflow
19:36:55 <fungi> we essentially want to track master and one or two stable branches per project we're supporting for this
19:38:06 <clarkb> fungi: so in this case all changes could be proposed against upstream/stable/foo rather than stable/foo
19:38:13 <mordred> jeblair: sorry - was afk for a sec
19:38:17 <clarkb> and since changes won't get submitted and merged it is probably fine
19:38:26 <mordred> jeblair: I agree that the work we're doing with track-upstream is EXACTLY trying to solve this issue
19:38:30 <jeblair> clarkb: ok, i could see how replicating might be preferable by being a bit more real-time (and not depending as much on manage-projects)
19:38:32 <clarkb> the problems arise when you need to take upstream/stable/foo and stable/foo and not hate each other
19:38:44 <fungi> clarkb: right. just provides a convenient place to iterate through code review, and then they get discarded, downloaded, and pushed to proper public gerrit
19:39:05 <mordred> the problem is exactly the same as a company working on a private fork of openstack that tehy're trying to track upstream on
19:39:19 <fungi> mordred: agreed
19:39:21 <clarkb> mordred: except in this case we shouldn't have the merge problem so I am ok with it
19:39:35 <mordred> :)
19:39:49 <mordred> how about we take an action to sync up on the tracking thoughts
19:39:59 <mordred> I think it's slightly verbose of a topic to cover this instant
19:40:05 <clarkb> ++
19:40:09 <zaro> ++
19:40:15 <fungi> yes, meeting topic sufficiently beaten to death for this week
19:40:27 <jeblair> i'm glad we uncovered this.  :)
19:40:30 <jeblair> #topic  Upgrade gerrit (zaro)
19:40:40 <zaro> ok. best news of the week.
19:40:45 <zaro> #link https://gerrit-review.googlesource.com/#/c/48254
19:40:48 <zaro> it's merged!
19:40:51 * fungi cheers loudly
19:40:51 <pleia2> \o/
19:40:56 <pleia2> congrats, zaro :)
19:41:07 <jeblair> yaaaay!
19:41:13 <zaro> what now?
19:41:29 <jeblair> zaro: this is post 2.8, and should be in 2.9, right?
19:41:35 <jeblair> zaro: eta on 2.9?
19:41:38 <zaro> yes.
19:41:40 <mordred> woohoo!
19:41:48 <fungi> does this achieve upstream feature parity with our current workflows?
19:42:01 <jeblair> fungi: or what we hope will be equivalent
19:42:01 <fungi> or is there more outstanding
19:42:11 <clarkb> I would be ok with running a forked 2.8 with that patch on top of it if we can iterate that quickly
19:42:15 <clarkb> then go to 2.9 when that happens
19:42:29 <jeblair> clarkb: me too, but i'm trying to figure out if that's necessary...
19:42:33 <mordred> ++
19:42:49 <clarkb> gerrit doesn't have release cycles aiui so 2.9 could happen whenever
19:42:56 <clarkb> though they seem better at doing them regularly now
19:43:04 <fungi> carrying patches we know are already merged upstream and will go away when we upgrade seems okay *if& we need to do that
19:43:20 <clarkb> _david_ might know
19:43:25 <jeblair> zaro: so, assuming 2.9 isn't expected in, say, 2 weeks....
19:43:58 <jeblair> zaro: we pull 2.8 into openstack-infra/gerrit, build it, and deploy on review-dev
19:44:15 <jeblair> figure out what needs to happen to do the upgrade (if it's not fully automatic)
19:44:18 <zaro> jeblair: sounds good to me
19:44:19 <jeblair> make config changes, etc...
19:44:37 <jeblair> there are some _serious_ problems with the gerrit ui that have developed since the last time i looked
19:44:45 <mordred> oh yeah?
19:44:48 <zaro> jeblair: do we want to do WIP vote or WIP button (plugin)
19:44:50 <zaro> ?
19:44:53 <clarkb> change screen 2 is not fungi device happy
19:44:53 <jeblair> mordred: look at the link zaro sent
19:45:05 <mordred> WOAH
19:45:28 <mordred> that's crazy looking
19:45:39 <jeblair> comments are too wide, the tabs can get even wider, meaning it won't fit even on a 1280 pixel wide screen
19:45:52 <jeblair> it uses g+ avatars (i assume/hope that can be disabled)
19:46:04 <clarkb> I like some of the changes, in particular in the diff screen
19:46:18 <mordred> that cherry-picks thing is interesting
19:46:25 <fungi> that is sorta wideish. i still have to scroll sideways when fullscreening a browser on a 1080px-wide display
19:46:56 <jeblair> so anyway, i expect we have some tuning we may need to do.
19:46:56 <fungi> i'm hoping 1440px will be wide enough
19:47:04 <mordred> I like that the reply button popup is better
19:48:02 <fungi> maybe they could make the layout configurable/skinnable
19:48:31 <jeblair> fungi: i honestly don't know what's tunable, which is why i think we should get it up on review-dev and start poking at it
19:48:36 <clarkb> ++
19:48:36 <mordred> jeblair: ++
19:48:43 <fungi> yup, absolutely
19:48:51 <jeblair> hopefully there are things we can do in config and css
19:49:05 <jeblair> but there's a possibility that some things may need upstream patches (if just to expose a parameter in config)
19:49:13 <jeblair> so we should be prepared for that.
19:49:40 <SergeyLukjanov> btw how it looks like on MBA (1440x900) - https://dl.dropboxusercontent.com/u/33687550/Screen%20Shot%202013-12-03%20at%2011.48.21%20PM.png
19:49:44 <jeblair> at any rate, as much as i'd like to roll this out as quickly as possible... we have _a lot_ of developers, so we need to get the details right.
19:50:25 <fungi> yes, i think the fewer number of times we change this up on them in production, the happier everyone will be
19:50:36 <fungi> it's going to be disruptive no matter how smoothly it goes
19:51:10 <jeblair> #topic  Proposed next bug day: Tuesday, December 17th at 1700 UTC (pleia2)
19:51:24 <clarkb> we should also maybe compile a list of the good new features so that people are aware of them eg better search
19:51:25 <pleia2> so I was looking at the calendar
19:51:38 <pleia2> our final two meetings of the month land on Christmas Eve and New Years Eve
19:51:52 <pleia2> I'm going to be doing the family thing for the holidays, so I'll be family + airplane respectively on these days
19:52:04 <pleia2> but it would be nice to have a bug day before the new year, so December 18th
19:52:07 <pleia2> err 17th
19:52:15 <fungi> i can be around for both those meetings, but suspect i might be one of the only people in here
19:52:17 <clarkb> 17th isn't good for me, but 18th should work
19:52:24 <clarkb> or next week is good
19:52:58 <zaro> i'll be around
19:53:00 <clarkb> isn't icehouse-1 this week?
19:53:06 <pleia2> yeah
19:53:12 <clarkb> we could try to stick to week after milestone which would mean bug day next week
19:53:18 <pleia2> wfm
19:53:28 <pleia2> so December 10th
19:53:30 <jeblair> i think we may cancel the christmas and new years meetings.  :)
19:53:38 <zaro> ++
19:53:41 <pleia2> jeblair: yeah, I'm thinking that too
19:53:44 <clarkb> pleia2: I like that plan
19:54:10 <fungi> i'm good with the 10th for bugs
19:54:15 <anteaya> me too
19:54:18 <pleia2> great, I'll add it to our wiki
19:54:42 <fungi> thanks pleia2!
19:55:02 <jeblair> pleia2: thanks!
19:55:05 <jeblair> #topic  Zuul release (2.0?) / packaging (pabelanger)
19:55:33 <jeblair> pabelanger: i take this to mean you think it's time.  :)
19:55:40 <jeblair> it's probably a bit past time, really
19:55:47 <pabelanger> jeblair: Ya, I think we are ready to get started on packaging
19:56:04 <pabelanger> and since it is a big change after 1.3 for the triggers, I guess we should consider 2.0?
19:56:05 <clarkb> though it might be worth waiting a bit to see if the split gate bug can be tracked down
19:56:05 <jeblair> i will try to take a look and make sure it's ready (docs caught up with features, etc)
19:56:23 <jeblair> and yes, definitely 2.0 -- it's full of breaking changes (documented in the NEWS file i believee)
19:56:37 <fungi> 2.z0mg
19:56:45 <clarkb> jeblair: ++
19:56:46 <pabelanger> Event if it is not a full release, maybe an alpha?
19:56:54 <pabelanger> Even*
19:58:00 <jeblair> pabelanger: i think we can do a full release soon
19:58:07 <pabelanger> Perfect
19:58:42 <fungi> we certainly have tested it (in production!)
19:58:54 <jeblair> indeed!
19:59:02 <jeblair> #topic Open discussion
19:59:12 <jeblair> anything else in the next minute? :)
19:59:14 <ryanpetrello> o/ not much time left
19:59:19 <ryanpetrello> thought I'd mention a pecan review I have open
19:59:19 <ryanpetrello> https://review.openstack.org/#/c/58643/1
19:59:31 <ryanpetrello> it's failing to run ceilometer tests, as it can't connect to mongodb for some reason
19:59:43 <ryanpetrello> I nagged Monty about it yesterday, but just thought I'd ask around again
19:59:44 <mordred> I have looked at the above and find it very odd
19:59:54 <clarkb> mongodb is only available on the centos slaves
19:59:55 <mordred> because the log does show mongo launching successfully
19:59:57 <ryanpetrello> I can confirm that it works just fine when run via tox on my own precise images on DH's cloud
19:59:58 <clarkb> where are those jobs running?
20:00:03 <fungi> i will be disappearing for a couple hours tomorrow afternoon
20:00:07 <fungi> local osug meetup
20:00:07 <clarkb> mongodb on precise is too old for ceilometer
20:00:16 <clarkb> oh I am disappearing tomorrow mid day due to dentist
20:00:17 <mordred> ah. that might be the problem ryanpetrello is seeing then :)
20:00:22 <ryanpetrello> yep, that might be it
20:00:56 <jeblair> thanks everyone
20:00:58 <jeblair> #endmeeting