19:01:55 <jeblair> #startmeeting infra
19:01:56 <openstack> Meeting started Tue Apr 15 19:01:55 2014 UTC and is due to finish in 60 minutes.  The chair is jeblair. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:01:57 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:01:59 <openstack> The meeting name has been set to 'infra'
19:02:00 <pleia2> o/
19:02:08 <mordred> o/
19:02:09 <jeblair> agenda:
19:02:11 <jeblair> #link https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting
19:02:17 <jeblair> last meeting:
19:02:19 <jeblair> #link http://eavesdrop.openstack.org/meetings/infra/2014/infra.2014-04-08-19.01.html
19:02:31 <jeblair> #topic  Actions from last meeting
19:03:05 <jeblair> #link http://lists.openstack.org/pipermail/openstack-tc/2014-April/000608.html
19:03:17 <jeblair> the item was: jeblair send revised repo rename list to tc
19:03:36 <jeblair> i did that, and as you can see the tc was wildly enthusiastic.
19:03:48 <jeblair> i'm going to interpret that as no one having any objections and we should move forward
19:03:53 <jeblair> so i'll put those on the rename list
19:04:04 <jeblair> mordred make an abbreviated projects.yaml with only projects using storyboard as their primary tracker
19:04:09 <mordred> done
19:04:14 <jeblair> woo!
19:04:15 <mordred> or, rather
19:04:18 <mordred> implemented differently
19:04:28 <mordred> added storyboard flag to existing projects.yaml list
19:04:40 <jeblair> the outcome is there -- it is not confusing to know where to report bugs now :)
19:04:44 <jeblair> nibalizer write lp->storyboard migration script
19:04:44 <mordred> ++
19:05:54 <clarkb> I know nibalizer has been poking at getting a storyboard running for testing
19:06:06 <clarkb> but apparently the puppet was giving him a hard time, I didn't have any inside info for him though
19:06:42 <jeblair> okay cool.  i think we'll want to use something like that for zuul
19:06:56 <krotscheck> I helped him out a little this morning, but he was on the bus and never got back to me.
19:07:32 <jeblair> we should probably file a story about that actually
19:07:46 <krotscheck> If only we had a place to file stories.
19:08:09 <jeblair> and assign it to him
19:08:15 <mordred> ++
19:08:29 <jeblair> #topic  Gerrit 2.8 upgrade
19:08:40 <jeblair> so this is coming up soon
19:08:50 <jeblair> i'd like to do a few things:
19:09:10 <jeblair> 1) review the list of outstanding changes and poke infra-core to merge whatever is safe now
19:09:24 <jeblair> 2) make any outstanding decisions that we haven't gotten around to making
19:09:36 <jeblair> 3) schedule a time
19:09:44 <jeblair> 4) start sending announcements
19:09:58 <jeblair> zaro: do you have an etherpad to help with the first 2 items?
19:09:58 <aburaschi> (hi)
19:10:05 <zaro> #link https://etherpad.openstack.org/p/remaining-gerrit-upgrade-changes
19:11:02 <jeblair> mordred: oh, the 2.8 branch isn't updating because of manage projects, right?
19:11:06 <zaro> i wasn't sure if 1st item on the list is needed.
19:11:21 <nibalizer> ohai
19:11:41 <zaro> but i think we need so we can test review-dev with jenkins.
19:11:46 <clarkb> jeblair: correct
19:11:50 <mordred> jeblair: that's right
19:12:15 <zaro> i'm working on verifying that puppet can deploy review-dev.o.o.  I don't have it quite working yet.
19:12:15 <jeblair> mordred: is there a way to run just the remote tracking bit?
19:12:49 <mordred> jeblair: it's on the todo list to fully split that out - but I could make a patch to make a flag to do that pretty easily
19:12:56 <mordred> jeblair: I can prioritize that
19:12:58 <zaro> puppet will install gerrit and it looks like it starts without problem, but i can't access the site.  I think it's the apache stuff that's not quite setup right.
19:13:32 <jeblair> zaro: i know this is required:   https://review.openstack.org/#/c/60893 (Allow encoded path separators in URLs)
19:13:49 <jeblair> zaro: it will manifest as permission denied errors if you try to look at any change
19:14:00 <jeblair> zaro: i believe that's applied manually on review-dev right now
19:14:40 <nibalizer> zaro: I'm seing review-dev as up
19:14:43 <zaro> yeah, it's on there to approve before upgrade
19:14:58 <zaro> nibalizer: i'm testing on my personal VM
19:15:06 <nibalizer> zaro: oh okay
19:15:26 <zaro> jeblair: i have that change on my test vm
19:15:49 <zaro> i can't access http://host
19:16:14 <jeblair> zaro: ok, hrm.  well, we should start prioritizing fixing this now to make sure we're ready
19:16:27 <jeblair> zaro: so feel free to bug us for help if you get stuck :)
19:16:28 <zaro> actually it says web page not available.  i need help with it.
19:17:32 <jeblair> ttx: the 'off week' is april 28 - may 2, right?
19:17:46 <ttx> yes
19:17:52 <zaro> anyways i think etherpad has all the tasks that's necessary to do the upgrade.  anyone see anything missing?
19:17:54 <pleia2> good, that's when I'm away ;)
19:18:00 <clarkb> I am away that week too :)
19:18:05 <ttx> pleia2: same here :)
19:18:22 * mordred will be here
19:18:27 <jeblair> right, but fungi, mordred, and zaro are around, right?
19:18:49 <jeblair> i mean, that's like right when fungi gets back, but still... :)
19:18:52 <mordred> yah - fungi is gone now, which I believe puts him back by that week
19:19:19 <jeblair> so should we just get started first thing monday morning?
19:19:23 <zaro> i'm around
19:19:37 <mordred> jeblair: works for me
19:19:50 <clarkb> yup fungi should be back during off week
19:20:08 <zaro> my announcement says gerrit upgrade to happen 5/1-5/7.  isn't that the off week?
19:20:44 <zaro> yup, that's what it says https://wiki.openstack.org/wiki/Icehouse_Release_Schedule
19:20:48 <jeblair> zaro: april 28 - may 2; ttx names weeks by their thursdays
19:21:03 * mordred thinks ttx is strange
19:21:07 <zaro> ok, fine by me
19:21:52 <jeblair> okay, so we'll start at, oh, 9am pdt april 28
19:21:54 <zaro> better for me anyways
19:22:16 <ttx> mordred: that's another weird thing we inherited from ubuntu
19:22:35 <ttx> (release days are on Thursdays, so weeks are name after Thursdays too)
19:22:45 <jeblair> i think the announcement should probably tell people to expect a couple hours of downtime followed by several more hours of automated systems not quite working, but they shouldn't notice because they should all be on vacation.  ;)
19:22:47 <mordred> OpenStack: the love child of Ubuntu and Drizzle ...
19:23:05 <mordred> jeblair: ++
19:23:19 <jeblair> zaro: are you interested in writing/sending the announcements?
19:23:28 <zaro> jeblair: can do.
19:23:37 <mordred> in a perfect world, do we have any idea how much downtime we actually expect?
19:24:01 <jeblair> zaro: cool; if you don't mind, write up a draft in an etherpad and we can review it before you send it
19:24:09 <mordred> I mean, annoucing a couple of hours sounds great- just curious if we have a sense of the actual or not
19:24:12 <jeblair> (these should be a bit more formal and we'll want to make sure they are correct)
19:24:38 <jeblair> zaro: do you know how long schema upgrades and secondary indexing might take?
19:25:04 <zaro> there are manual steps to upgrade the db, so i'm guessing probably 4 hrs.
19:25:25 <jeblair> zaro: did we get you a production dump to test that with?
19:26:08 <zaro> yes, fugi provided and i believe i have tested.
19:26:26 <jeblair> how long did that take to convert?
19:26:52 <jeblair> also...
19:27:03 <zaro> last i tested everything it was ver 2.8.1, now it's 2.8.4.  not sure if it's worth retesting?
19:27:15 <jeblair> i'm thinking we should implement this by spinning up a new host
19:27:16 <zaro> conversion is pretty quick.
19:27:52 <zaro> i believe it was matter of a few mins.
19:28:34 <jeblair> so hopefully actual downtime should be short, unless the initial indexing (for the secondary index) is very slow
19:29:05 <mordred> jeblair: perhaps we should test by having one of us who hasn't done any of the upgrades yet try to perform one on a new copy host based on instructions from zaro and see how it goes?
19:29:13 <jeblair> our current gerrit host is very old; we should get a new one that's a performance pvhvm node
19:29:22 <mordred> jeblair: ++ to pvhvm
19:29:23 <clarkb> ++
19:30:49 <jeblair> mordred: that would be swell; are you volunteering? :)  note that if anyone spins up a second gerrit with the production data, you should probably take care to make sure no email leaves the machine; just in case
19:31:05 <mordred> jeblair: yeah - I'll take a stab at it
19:31:12 <jeblair> cool
19:31:33 <jeblair> #action mordred to test zaro's gerrit upgrade instructions (with timing info)
19:32:21 <zaro> mordred: #link https://etherpad.openstack.org/p/gerrit_upgrade_script
19:32:32 <jeblair> anything else on the gerrit upgrade?  we'll do one more checkin next week, and then the upgrade itself is the following week
19:33:17 <zaro> jeblair: did you see the gerritlib part?
19:33:44 <jeblair> zaro: yeah, remind me this afternoon and i'll tag a release
19:34:01 <zaro> we also need an infra-core to update openstack/2.8 branch
19:34:10 <zaro> any volunteers?
19:34:13 <jeblair> zaro: mordred is going to take care of that
19:34:15 <sweston> I have been working quite a lot with the gerrit upgrade as well in my own environment, so I might be helpful as well.  I will be available in infra (most of the time) if anyone thinks of something a relative noob can do.
19:34:30 <mordred> sweston: awesome, thanks!
19:34:36 <jeblair> ++
19:34:40 <zaro> sweston: will be in touch
19:34:43 <jeblair> oh, i just thought of something else...
19:35:05 <jeblair> two things actually...
19:35:27 <jeblair> zaro: what's your thought on workflow-label vs wip plugin?
19:35:51 <sweston> zaro, mordred:  glad to help out in any way I can ;-)
19:36:11 <jeblair> zaro: i'm guessing it's "stick with workflow-label" since we're getting close and wip doesn't look like it's installed on review-dev; but wanted to double check.
19:36:16 <zaro> IMO i think label makes a lot of sense because it just works with everything else.
19:36:35 <zaro> i mean like the prolog rules
19:36:46 <zaro> and searching
19:37:12 <jeblair> the second thing is, we should have some documentation to help devs with the changes.  in particular, we need something on the workflow label, and probably something about how to make custom dashboards
19:37:13 <zaro> WIP is not available on 2.8
19:37:33 <mordred> jeblair: ++
19:37:47 <jeblair> sweston: do you think you could pitch in there?
19:37:48 <zaro> is that something that would be on openstack wiki?
19:37:58 <jeblair> zaro: yes, i think so
19:38:04 <sweston> jeblair: absolutely!!
19:38:31 <mordred> jeblair: wiki or ci.o.o ?
19:38:48 <jeblair> mordred: wiki for now i think
19:38:57 <mordred> nod
19:39:16 <jeblair> mordred: i still need to finish my "user manual" idea, when i do, it should go there, but right now i think the most appropriate place for this stuff is the wiki
19:39:22 <sweston> documentation is fairly sparse as it is, I found myself reading through a lot of java code to figure things out.  Any time that I could save for anyone else would be a bonus.
19:39:43 <jeblair> and we'll like to it in the later announcements
19:39:47 <jeblair> link to it
19:41:00 <jeblair> zaro: thanks again for working on this
19:41:17 <jeblair> #topic  Fedora gate support (ianw)
19:41:46 <jeblair> ianw: any introductory remarks before we dig into the subtopics?
19:41:52 <ianw> hi; a quick procedural thing, are we supposed to remove items from the agenda on the wiki manually?  i had thought they got reset each week
19:42:15 <ianw> really, the only thing I want to bring up this week is https://review.openstack.org/#/c/86842/
19:42:35 <jeblair> ianw: heh, i try to clean it up sometimes, but i missed last weeks meeting and didn't notice this was on the agenda. :)
19:42:50 <pleia2> I wasn't around much last week, but I'll be catching up with fedora stuff and doing reviews this week and next
19:43:15 <ianw> Derek and I both tested manual installs and it seemed ok
19:43:27 <pleia2> great
19:43:39 <jeblair> cool, well if those things are already decided...
19:43:43 <ianw> pleia2: ok, if review of that is on your todo list, all good
19:43:54 <jeblair> #topic Open discussion
19:43:56 <pleia2> ianw: it is now, thanks for mentioning it :)
19:43:59 <sdague> ianw: is there a proposed commit to put this on devstack in experimental or non voting?
19:44:17 <sdague> I'd like to start running it on devstack as soon as we can to shake out issues
19:44:35 <ianw> sdague: not yet, i just wanted to make sure someone was looking at adding f20 to nodepool, then we can do that
19:44:43 <sdague> cool, great
19:45:08 <jeblair> those infra-root folks are still changing all of our credentials for everything, so we're going to be less responsive than usual for the next bit; patience there would be appreciated
19:45:35 <jeblair> and the best way to help is to review infra/config changes so that when we get back to it, we can just approve them without thinking about them because they have so many +1s :)
19:45:44 <nibalizer> im still woring on storyboard migration, still stuck on getting a local dev up
19:46:00 <nibalizer> im picking krotscheck's brain for tips and have some things to try when I get home
19:46:13 <jeblair> nibalizer: thanks for doing that
19:47:16 <nibalizer> ya, as i said earlier this morning I'm thinking right now that there is state on storyboard.o.o thats not expressed in the puppet code, or that I'm not aware of, since I'm running basically that and its no-worky
19:48:10 <jeblair> nibalizer: :(  that's definitely not what we want.
19:48:22 <mordred> agree
19:48:33 <jeblair> nibalizer: let me know if there's anything i can check on the prod server
19:48:39 <mordred> nibalizer: one thing - I assume you've created the db?
19:48:50 <mordred> we dont' create it in the puppet because we're using cloud databases for the storyboard server
19:49:32 <nibalizer> actually we do create the db, or we do a db-migrate thing in puppet
19:49:41 <nibalizer> we do an 'update head'
19:49:46 <mordred> we do - but that doesn't create the _Schema_ - just the tables in teh schema
19:49:56 <nibalizer> hrm maybe this is the problem!
19:50:05 <nibalizer> no need to chew up more meeting time tho
19:50:13 <nibalizer> we can follow up tonight when im on my home workstation
19:50:16 <mordred> kk
19:50:24 <reterdedddd> hello guys
19:51:08 <sdague> jeblair: any thoughts about if there are additional infra sessions for design summit you want proposed?
19:51:23 <clarkb> reterdedddd: o/
19:51:42 <reterdedddd> im back on freenode
19:51:46 <reterdedddd> but where is freenode
19:52:20 <jeblair> sdague: ttx or krotscheck is going to propose a storyboard session
19:52:48 <nibalizer> are the sessions going to be livestreamed?
19:53:03 <sdague> nibalizer: not design summit
19:53:15 <nibalizer> okay thanks
19:53:34 <sdague> jeblair: do we want to talk about multinode nodepool ?
19:53:52 <sdague> now that that is close to a thing
19:54:09 <retardedd_polish> im not clone ;_;
19:54:23 <jeblair> we might want to talk about the direction of the config repo (how to make it a better upstream, etc)...
19:54:50 <mordred> ++
19:54:52 <nibalizer> i would like to see the storyboard module split out into its own module that we then bring in
19:54:58 <nibalizer> not sure if there is precedent for that
19:55:15 <jeblair> nibalizer: there absolutely is; we want that for pretty much all of our modules
19:55:16 <mordred> nibalizer: that's actually a long-standing todo list to do more of
19:55:49 <nibalizer> woot, i'll follow up with you all later on how to requistion a new gerrit repo to plop it into
19:55:53 <mordred> we stalled on it a little bit due to there not being spectacular ways to automatically upload to pupept forge
19:55:53 <jesusaurus> mordred: jeblair: whats preventing it from happening for most of the modules?
19:56:13 <jeblair> jesusaurus: people need to type things to make it happen, and also what mordred said about uploading
19:56:16 <clarkb> I think cleanup is a big part of it
19:56:22 <mordred> jesusaurus: puppetforge upload automation OR a better way to pull things in which is not puppet forge that doesn't involve crazy complex new things
19:56:25 <clarkb> we need to do the decoupling
19:56:26 <mordred> and cleanup
19:56:32 <mordred> defeinitely cleanup :)
19:56:37 <sdague> yeh, there is definitely a lot of coupling today
19:56:40 <mordred> but in general  - highly supportive of the effort
19:56:49 <retardedd_polish> which server is good?
19:56:49 <nibalizer> ya thats a big project, but a fun one I think
19:57:09 * jesusaurus puts seperating out modules at the bottom of his todo list
20:00:00 <krotscheck> nibalizer: What are you thinking, something like storyboard::api, storyboard::db, storyboard::webclient?
20:00:42 <retardedd_polish> fuck NSA
20:00:45 <jeblair> thanks all!
20:00:47 <jeblair> #endmeeting