19:01:42 <jeblair> #startmeeting infra
19:01:43 <openstack> Meeting started Tue Apr 22 19:01:42 2014 UTC and is due to finish in 60 minutes.  The chair is jeblair. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:01:44 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:01:46 <openstack> The meeting name has been set to 'infra'
19:01:57 <jeblair> agenda:
19:01:57 <jeblair> #link https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting
19:02:02 <jeblair> last meeting:
19:02:02 <jeblair> #link http://eavesdrop.openstack.org/meetings/infra/2014/infra.2014-04-15-19.01.html
19:02:07 <jeblair> #topic Actions from last meeting
19:02:11 <jeblair> mordred to test zaro's gerrit upgrade instructions (with timing info)
19:02:17 <jeblair> mordred: ?
19:03:11 <jeblair> <sigh>
19:03:23 <clarkb> zaro: do you know if he did that yet?
19:03:46 <nibalizer> im here
19:03:47 <zaro> i do not know
19:03:53 <SergeyLukjanov> o/
19:03:59 <sweston> o/
19:03:59 <anteaya> do we have a back up plan?
19:04:13 <jeblair> anteaya: we trust that zaro has tested his instructions :)
19:04:23 <anteaya> go zaro
19:04:42 <jeblair> it would have been nice to get a second set of eyes on the process, but zaro's been working on this for a while so i assume he knows what he's doing.  :)
19:05:00 <jeblair> the backup plan is that if something goes minorly wrong, we wing it.
19:05:04 * clarkb has randomly poked at it with zaro and it seems to be well handled
19:05:07 <jeblair> the backup backup plan is we roll back
19:05:13 <anteaya> good enough
19:05:15 <sweston> I can test them!
19:05:43 <jeblair> which, should be fairly easy to do since i think we will be building the replacement on another host
19:05:44 <sweston> I don't know about the timing though, someone may have to work with me on that.
19:05:55 <jeblair> #topic  Gerrit 2.8 upgrade (zaro)
19:06:01 <jeblair> we might as well move into that ^ :)
19:06:31 <anteaya> that was my understanding, this is on a new host
19:06:53 <jeblair> #link https://etherpad.openstack.org/p/remaining-gerrit-upgrade-changes
19:07:08 <jeblair> sweston: i think ^ has all the info
19:07:11 <zaro> i've verified that puppet is able to deploy review-dev and verified that review-dev is working as expected.
19:07:16 <anteaya> sweston: I would offer to help but I still can't get current gerrit up with puppet
19:07:30 <jeblair> zaro: awesome -- that's very reassuring
19:07:38 <zaro> jeblair: i've rebased my changes on top of your changes for nodepool.  need that reviewed.
19:07:55 <jeblair> zaro: which?
19:08:02 <zaro> anteaya: i wasn't able to either.  clarkb did it for me.
19:08:23 <jeblair> zaro: did mordred not update the 2.8 branch for you?
19:08:31 <zaro> jeblair: https://review.openstack.org/#/c/88026
19:08:37 <jeblair> (i'm noticing those items are still on the list and not struck out)
19:08:59 <zaro> jeblair: i don't think it's been updated. but let me check again.
19:09:21 <sweston> anteaya: I was able to do it, fortunately :-)  I am always available in IRC for questions.
19:10:07 <anteaya> sweston: awesome, hopefully you can follow through zaro's instructions then
19:11:06 <sweston> yup, I will refresh my environment and then start testing this afternoon
19:11:20 <anteaya> sweston: let me work with you on that, maybe you can teach me
19:11:45 <sweston> anteaya: that sounds lovely.  Anyone else is welcome to join, the more, the merrier!
19:11:54 <zaro> jeblair: looks like openstack stable-2.8 has been synced with gerrit upstream stable-2.8
19:12:13 <zaro> jeblair: don't know which method was used to do it though.
19:12:39 <jeblair> zaro: doesn't matter, so what next?  do we need an openstack/2.8.4 branch?
19:13:10 <sweston> anteaya: that will work well, as I have an action item to create some documentation for the upgrade process as well.
19:13:20 <anteaya> sweston: great
19:13:21 <zaro> we need to update openstack/2.8 branch with commits from stable-2.8 branch
19:13:25 <anteaya> yay docs
19:13:38 <sweston> +1 yay docs!
19:13:57 <jeblair> let's focus on the branch topic for a sec
19:14:06 <anteaya> sorry
19:14:12 <zaro> mordred didn't do that.  so anybody want to take care of that?
19:14:24 * sweston gets excited too easily
19:14:24 <clarkb> I can create the branch if you give me a sha1
19:14:39 <jeblair> i'm not sure we're all following the same process here
19:14:58 * clarkb holds off for jeblair to explain
19:15:18 <jeblair> zaro: what is the difference between the openstack/2.8 and openstack/2.8.3 branches?
19:15:19 <zaro> i'm referring to line #7 on etherpad
19:15:46 <jeblair> what is that branch for?
19:15:55 <zaro> ohh, gerrit is now at ver 2.8.4 release.  but more changes were added to the stable-2.8 branch.
19:15:59 <clarkb> jeblair: its so that we didn't have to put upstream commits atop our commits
19:16:12 <clarkb> jeblair: so that our fork sits nicely atop upstream instead of being mixed in
19:16:21 <jeblair> clarkb: okay, so there's an upstream/stable-2.8 branch
19:16:31 <jeblair> that should have whatever is, well, upstream.
19:16:59 <jeblair> usually what we do is build our version on top of releases
19:17:28 <jeblair> so we'd branch openstack/2.8.4 from the 2.8.4 tag, then propose our modifications on top of it
19:17:47 <zaro> ok, that wfm
19:18:12 <jeblair> i think zaro is asking that we somehow do a force push to the openstack/2.8 branch, which seems to already have some commits in it
19:18:46 <clarkb> I interpreted that as making the 2.8.4 branch
19:19:01 <zaro> i think in this case we should just git rid of openstack/2.8 ?
19:19:25 <zaro> make openstack/2.8.4 instead?
19:19:46 <jeblair> zaro: do we need anything in upstream/2.8 that is past 2.8.4?
19:20:24 <zaro> i assume you mean upstream stable-2.8 ?
19:20:32 <jeblair> yeah sorry
19:20:33 <jeblair> http://git.openstack.org/cgit/openstack-infra/gerrit/log/?h=upstream%2Fstable-2.8
19:21:11 <zaro> there seems to be a major fix with secondary indexing.
19:22:17 <jeblair> okay, how about we branch openstack/2.8.4 from upstream/stable-2.8 so we get all of that
19:22:17 <zaro> deadlocks in subIndex or something.
19:22:47 <jeblair> zaro, clarkb: ^ make sense?
19:22:58 <clarkb> yup
19:23:09 <zaro> wfm
19:23:15 <jeblair> zaro: then you can propose the local changes on top of that
19:23:27 <jeblair> #action clarkb make openstack/2.8.4 gerrit branch
19:23:28 <zaro> ok.
19:23:39 <jeblair> okay, let's get excited about docs now! :)
19:23:56 <jeblair> sweston: you volunteered to write up some docs for developers about the changes; any progress?
19:25:04 <sweston> jeblair: yes, but I need to dot some i's and cross some t's first.
19:25:40 <jeblair> i think what we need is something highlighting the major differences.  for example:
19:25:45 <jeblair> 'important changes' view is going away -- how to make an equivalent custom dashboard
19:25:51 <jeblair> workinprogress button is going away, how to use the workflow label instead
19:26:00 <jeblair> note that the approval label is changing to workflow (but the value +1=approved is the same, and the approval process in general is the same)
19:26:23 <jeblair> i'd like to send something out with a reminder announcement asap (ideally, today), even if it's brief
19:26:44 <jeblair> because i don't want to tell 2000 developers after the fact that, btw, we changed your workflow.
19:27:21 <zaro> there are also some changes to permission, like stream events are no longer available to all registered users.
19:27:24 <sweston> jeblair: okay, let me spend some time on it then .. I do have some other code I am trying to get merged into Neutron, so that is taking up quite a bit of my time as well.
19:27:42 <clarkb> zaro: we can preserve the old behavior though and hopeflly people won't notice the difference
19:27:44 <hashar> ( for important changes maybe consider a per user version of http://status.openstack.org/reviews/ )
19:27:47 <anteaya> sweston: do you have your notes on an etherpad?
19:28:05 <anteaya> sweston: then we can refine what you have created thus far?
19:28:19 <jeblair> clarkb: zaro: yeah, we should set permissions correctly so that does not change
19:28:27 <sweston> anteaya: no, but I can put them out there tomorrow, if that is okay for everyone?
19:28:58 <anteaya> sweston: jeblair said he would like to send something out today
19:29:08 <anteaya> can I help you get them on an etherpad today?
19:29:25 <jeblair> it's < 1 week out and we basically haven't told developers anything about this, so i'm going to write up a followup reminder announcement and let people know about the changes
19:29:31 <sweston> anteaya: yup!
19:29:38 <anteaya> sweston: great thanks
19:29:48 <jeblair> we can continue to work on getting sweston's documentation in shape and point to it when ready
19:30:01 <sweston> anteaya: absolutely
19:30:16 <sweston> jeblair: sounds good
19:30:34 <jeblair> clarkb: did you ever send out something about changing the ssh keys for heartbleed?
19:30:53 <anteaya> are we all caught up on heartbleed?
19:30:54 <clarkb> I did not...
19:31:08 <clarkb> jeblair: you had mentioned that it wasn't worth sending a note at the time
19:31:18 <clarkb> a general note at least
19:31:23 <jeblair> okay.  we will do that during the gerrit upgrade as well and we'll do it then
19:31:26 <clarkb> maybe I misinterpreted what that meant about ssh keys
19:31:31 <clarkb> ok sounds good
19:31:32 <mordred> o/
19:31:36 <anteaya> hey
19:31:37 <jeblair> clarkb: i don't think i ever would suggest that we should change the ssh keys without telling anyone.
19:31:41 <anteaya> look who it is
19:31:41 <mordred> sorry I'm late. no, I did not get to the upgrade
19:31:54 <clarkb> jeblair: you didn't. Just that I shouldn't send the note last monday
19:32:05 <clarkb> where I interpreted the note to be generic thing + keys
19:32:10 <jeblair> clarkb: i feel certain something was misinterpreted.
19:32:15 <clarkb> ya probably was
19:32:39 <jeblair> but anyway, we will change the ssh keys on monday and my reminder announcement today will include that
19:32:47 <clarkb> great thanks
19:33:40 <jeblair> clarkb, zaro: let's try to get our patches on 2.8.4++ running on review-dev today
19:33:51 <clarkb> yup going to cut branch after meeting then go grab lunch
19:33:56 <jeblair> cool
19:34:03 <jeblair> anything else on the gerrit upgrade?
19:34:48 <jeblair> #topic  manage-projects status (mordred)
19:35:01 <jeblair> mordred: what's the news?
19:35:01 <mordred> manage-projects works again
19:35:06 <mordred> and is live
19:35:15 <mordred> so landing new projects changes should work
19:35:23 <anteaya> you said you were going to switch it off to test something last night
19:35:29 <anteaya> you turned it back on then
19:35:45 <mordred> yes - I was testing a few quick and dirty ways to decrease the run-time, but they didn't help
19:35:51 <mordred> so I gave up on them
19:35:51 <anteaya> k
19:36:02 <anteaya> no data from your tests?
19:36:32 <mordred> it takes 10 minutes ish to run manage-projects as is,
19:36:46 <mordred> whcih means it can complete before the next puppet run
19:37:25 <mordred> if we remove the github chatter, it can be about twice as fast - but we'd be losing features
19:37:48 <mordred> so I think the next step is just continuing to refactor some of the different thigns (like upstream tracking) into disconnected processes
19:37:56 <mordred> but that's not urgent
19:38:15 <mordred> manage-projects is NOT running on cron
19:38:20 <mordred> only on projects.yaml change
19:38:28 <mordred> so we're only getting upstream tracking when we add projects
19:38:36 <mordred> this is the main motivation behind splitting out upstream tracking
19:39:07 * anteaya makes a note to look at the sections of code that handle upstream tracking
19:39:16 <mordred> that's all Iv'e got - questions?
19:39:26 <anteaya> thanks for fixing it
19:40:39 <clarkb> mordred: no questions
19:40:54 <jeblair> #topic Open Discussion
19:41:06 <jeblair> nibalizer: any progress on the lp->storyboard script?
19:41:20 <jeblair> i think it would be nice to move zuul bugs over
19:41:33 <clarkb> I will be AFK next week. And plan to get red hat tripleo CI cloud added to nodepool this afternoon
19:41:58 <anteaya> enjoy hawaii
19:42:09 <jeblair> anteaya: there are only 4 open jeepyb bugs
19:42:14 <jeblair> anteaya: https://bugs.launchpad.net/openstack-ci?field.searchtext=jeepyb&search=Search&field.status%3Alist=NEW&field.status%3Alist=INCOMPLETE_WITH_RESPONSE&field.status%3Alist=INCOMPLETE_WITHOUT_RESPONSE&field.status%3Alist=CONFIRMED&field.status%3Alist=TRIAGED&field.status%3Alist=INPROGRESS&field.status%3Alist=FIXCOMMITTED&field.assignee=&field.bug_reporter=&field.omit_dupes=on&field.has_patch=&field.has_no_package=
19:42:38 <jeblair> anteaya: i was thinking we should file a story on 'separate upstream tracking' for jeepyb
19:42:51 <SergeyLukjanov> clarkb, have a good vacation!
19:42:57 <jeblair> maybe we should just move those 4 bugs over
19:43:02 <_nadya_> hi folks! I'd like to ask about status of fedora20 on gating. In Ceilometer we would like to move py27 jobs from ubuntu12 to fedora20. Because fedora20 has new mongo. As far as I understand #link https://review.openstack.org/#/c/86842/ needs to be merged to create testing job. Are there any concerns about this change?
19:43:07 <jeblair> clarkb: yes, enjoy!
19:43:09 <anteaya> I have no objections
19:43:26 <anteaya> it would be good to test them on storyboard, I agree
19:43:32 <clarkb> _nadya_: yes, we cannot gate on F20 as is
19:43:35 <pleia2> oh yes, I'm leaving on Friday and have a few days off and then LOPSA-East next week, then the following Monday and Tuesday I have the Open Source Business Conference
19:43:48 <clarkb> _nadya_: you can definitely test on F20 when we get it in, but it won't replace the ubuntu tests
19:43:52 <anteaya> pleia2: safe travels to you too
19:43:56 <pleia2> anteaya: thanks :)
19:44:07 <SergeyLukjanov> oh, I have a small notice too - I'll be in CA starting from May 3 (and traveling day before)
19:44:14 <_nadya_> clarkb: where can I find the reasons :)?
19:44:17 <mordred> SergeyLukjanov: beer!
19:44:20 <anteaya> SergeyLukjanov: awesome
19:44:26 <clarkb> _nadya_: because we don't have redundant providers for F20 images
19:44:37 <anteaya> I will be at home next week, on my computer and hopefully not bailing my basement
19:44:44 <clarkb> if someone teach nodepool to use DIB we can change that
19:45:47 <_nadya_> clarkb: maybe ubuntu14 will be available soon? And it will be easier then enabling fedora20. What's the plan about it?
19:46:02 <jeblair> i don't think we can run ceilometer jobs on f20 until it's available everywhere
19:46:06 <nibalizer> jeblair: no, still unable to reproduce a running storyboard
19:46:16 <nibalizer> mordred: and i beat on it again last night and couldn't figure it out
19:46:30 <nibalizer> :( maybe someone more familiar with storyboard should take this over
19:46:34 <clarkb> _nadya_: we currently have the same issue with trusty that we have with F20
19:46:49 * mordred is going to try to connect with nibalizer later today and figure out why it's not working for him
19:46:51 <clarkb> _nadya_: not all of our cloud providers have images yet, so we need a way to provide our own images or wait for the providers
19:46:57 <anteaya> nibalizer: I have faith in you
19:47:07 <nibalizer> anteaya: woooooo
19:47:11 <anteaya> :D
19:47:20 <nibalizer> mordred: lets tag team it!
19:47:21 <SergeyLukjanov> I think that I saw f20 images in rackspace cloud
19:47:31 <jeblair> mordred: are you still working on nodepool dib?
19:47:36 <clarkb> SergeyLukjanov: yup rax has f20 and trusty
19:47:39 <pleia2> SergeyLukjanov: yeah, rackspace has them but not hp
19:48:19 <mordred> jeblair: yes. that, I believe, is my current top priority, other than helping nibalizer and testing gerrit upgrade (sorry, lost that one last week, my bad)
19:48:20 <SergeyLukjanov> oh, got it
19:48:38 <eglynn_> is it the voting status of the job that requires redundant image providers?
19:48:42 <_nadya_> clarkb:  sounds sad... and no estimates? Or somebody works on it?
19:48:45 <jeblair> mordred: sweston has volunteered to test gerrit upgrade
19:48:51 <mordred> jeblair: woot
19:48:53 <eglynn_> (re. f20 and/or trusty gating_
19:48:55 <eglynn_> )
19:48:58 <mordred> then I will double down on dib nodepool
19:49:01 <clarkb> _nadya_: no estimate for when our providers will have images up
19:49:14 <clarkb> _nadya_: mordred may have an estimate on DIB for nodepool
19:49:19 <clarkb> or anyone else can give one and do the work
19:49:32 <eglynn_> clarkb: would a non-voting job be equally objectionable?
19:49:41 <mordred> it's not TOO far off - the main parts for the first stab are mostly in place
19:49:45 <jeblair> eglynn_: we can't run jobs in either check or gate queues (regardless of voting status) if the node it runs on isn't HA
19:49:45 <eglynn_> clarkb: (if the image was only available on a single provider)
19:49:51 <mordred> I need to test and slightly rework the elements
19:50:01 <jeblair> eglynn_: because that would block changes equally if there were a problem with that provider
19:50:12 <mordred> and then hand off to jeblair for help with making sure the image upload step works right
19:50:31 <eglynn_> jeblair: a-ha, I see, because even non-voting jobs have to complete?
19:50:36 <jeblair> eglynn_: yep
19:50:44 <eglynn_> k, thanks for the clarification
19:51:19 <eglynn_> ... so would it be possible to push HP to upload said image?
19:51:19 <jeblair> np.  we definitely want these running asap
19:52:55 <pleia2> eglynn_: I'm sure they're working as quickly as possible for at least trusty, it usually just takes a few weeks
19:53:11 <jeblair> i don't know.  i'd honestly rather we put our efforts into dib for nodepool; then we can potentially use the same image across providers
19:53:17 <clarkb> ++
19:53:23 <mordred> ++
19:53:23 <pleia2> yeah, that would be great
19:53:27 <jeblair> which will mean much less work and much more consistency in tests
19:53:30 <clarkb> and avoid a lot of problems with image building that we currently have
19:53:32 <mordred> it's not far off - I think I can get it done this week
19:53:35 <eglynn_> pleia2: cool enough, it would be great to see f20 there also
19:53:41 <mordred> it just needs finishing and testing
19:53:56 <mordred> I'll try to get it asap to a point where other people can also test
19:54:06 <clarkb> also I think we have somewhat agreed that we would just drop tests run on F20 when it goes out of support
19:54:06 <jeblair> mordred: great, thanks
19:54:14 <clarkb> that isn't concrete but is the current train of thought
19:54:24 <clarkb> so you don't really want to replace tests that don't have other coverage when using F20
19:54:30 <eglynn_> we've had long-running issues with the ceilometer gate being hobbled because of the lack of a mongo package newer than the ancient 2.0.4 available on precise
19:54:37 <SergeyLukjanov> dib and nodepool, sounds awesome
19:54:45 <jeblair> clarkb: yeah; that's why everything at least needs to run on lts
19:54:49 <pleia2> clarkb: yeah
19:54:53 <sdague> has anyone considered dib as part of projects we gate on? Because there is a use case for building heat capable images as well
19:54:55 <jeblair> eglynn_: fortunately, juno can test on trusty
19:55:18 <_nadya_> eglynn_: maybe we should think about ubuntu14 mostly, not f20?
19:55:45 <pleia2> for reference, the in-progress-ish dib in nodepool patch from mordred: https://review.openstack.org/#/c/88479/
19:55:51 <eglynn_> _nadya_: if trusty nodes are available soon, then yes ... main goal is for mongo to be testable
19:55:52 <jeblair> sdague: afaik it's doable, people have mentioned doing that, no one has done it.
19:55:55 <pleia2> (I thought I remembered this from my reviews :))
19:56:33 <SergeyLukjanov> fyi climate new name (blazar) has been approved by foundation, so, folks want to rename someday too
19:56:33 <eglynn_> ... but would be great to have more OS distro choices in the gate to avoid such issues in the future
19:56:36 <sdague> I feel like there was resistance to dib getting commit gated before, if that's changed, it would make some things easier
19:56:57 <jeblair> eglynn_: so we don't decide the distro support policy for the project
19:57:10 <jeblair> eglynn_: the tc does that -- we try to support that
19:57:27 <mordred> sdague: I think we weren't gating gating it before due to consumption in the gate being primarily via pip releases
19:57:36 <eglynn_> jeblair: a-ha, I see, thanks for the clarification
19:57:53 <sdague> mordred: yeh, well pip doesn't work with it, heat's been stalled on that for 5 months
19:58:03 <mordred> however, I think we very well may want to revisit that :)
19:58:05 <mordred> sdague: ++
19:58:12 <jeblair> the current policy is to support latest ubuntu and fedora, and don't break rhel and ubuntu lts; that leaves us with a baseline of only rhel and lts being supported for the life of our releases, so the bulk of our testing is there
19:58:41 <mordred> yah. but with the door open to adding backport repos, such as the UCA repos
19:58:53 <mordred> (which is how we pull in stuff from latest ubuntu)
19:58:59 <jeblair> eglynn_: we can do additional testing on fedora and ubuntu latest, but basic functionality really ought to be done in the context of one of the long-term distros
19:59:05 <clarkb> mordred: well we don't do that
19:59:13 <clarkb> mordred: we have attempted it but it never worked
19:59:13 <mordred> clarkb: right. I said door open to
19:59:15 <jeblair> mordred: yeah, everytime we try to do that it fails
19:59:21 <mordred> from a policy perspective
19:59:31 <jeblair> mordred: indeed
20:00:19 <jeblair> oh look at the time
20:00:26 <jeblair> thanks everyone!
20:00:29 <jeblair> #endmeeting