19:01:27 <fungi> #startmeeting infra
19:01:28 <openstack> Meeting started Tue Aug 18 19:01:27 2015 UTC and is due to finish in 60 minutes.  The chair is fungi. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:01:29 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:01:31 <openstack> The meeting name has been set to 'infra'
19:01:39 <fungi> #link https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting#Agenda_for_next_meeting
19:01:46 <SpamapS> o/
19:01:48 <ociuhandu> o/
19:01:56 <fungi> #topic Announcements
19:02:00 <tchaypo> \o
19:02:01 <ianw> o/
19:02:18 <fungi> #link http://lists.openstack.org/pipermail/openstack-infra/2015-August/003069.html
19:02:44 <fungi> just a reminder that we're planning to mass rename repos from the stackforge namespace into a common openstack namespace, on an opt-in basis
19:03:10 <mmedvede> o/
19:03:16 <pleia2> o/
19:03:17 <fungi> and then mark any that haven't opted in as read-only/archived after a time
19:03:25 <zaro> o/
19:03:28 <fungi> follow up to the thread with further discussion on scheduling
19:03:37 <jhesketh> Morning
19:03:43 <fungi> #link http://lists.openstack.org/pipermail/openstack-infra/2015-August/003075.html
19:04:21 <fungi> due to recent growth of core reviewers in neutron, they've requested that we clear job changes through mestery, armax or dougwig unless it's really urgent
19:04:58 <fungi> so just bear in mind, when reviewing changes to job configuration that could potentially impact what's tested for neutron that we're looking for a +1 from at least one of them if possible
19:05:25 <fungi> #topic Actions from last meeting
19:05:38 <fungi> #link http://eavesdrop.openstack.org/meetings/infra/2015/infra.2015-08-11-19.02.html
19:05:49 <fungi> pabelanger,jasondotstar look into restore-from-backup testing
19:06:01 <fungi> i know that's ongoing, we've been discussing it in channel
19:06:11 <jasondotstar> we've captured our thoughts here: https://etherpad.openstack.org/p/infra-backup-brainstorm
19:06:21 <jasondotstar> the next step is to run through our test strategy
19:06:26 <fungi> #link https://etherpad.openstack.org/p/infra-backup-brainstorm
19:06:50 <fungi> jasondotstar: pabelanger: thanks! we can work from there i think
19:06:57 <jasondotstar> we'd like to also ask folks to think of any other backup/restore scenarios worth testing (manually)
19:07:09 <jasondotstar> put them there on the etherpad.
19:07:52 <jasondotstar> once we identify the manual steps and validate that they work
19:07:53 <fungi> sounds good
19:08:01 <SpamapS> Just chiming in.. there is one set of data in infra-cloud that is hard to reproduce, and that is the Ironic inventory. We have a plan to store most of it in git, with the secret bits in hiera.
19:08:02 <jasondotstar> we can think about automating.
19:08:26 <jasondotstar> SpamapS: ack.
19:08:29 <SpamapS> So that just means any backup systems need to make sure git and hiera are backed up.
19:08:38 <fungi> SpamapS: cool, it's definitely the hard-to-reproduce things we actually care about backing up
19:08:41 <SpamapS> Also hiera backups, I assume, need to be encrypted.
19:08:48 <fungi> yeah
19:08:58 <clarkb> which is an exercise to be completed :/
19:09:04 <jasondotstar> the hiera data is primarily on the puppetmasters...or somewhere else?
19:09:14 <clarkb> yes would be on the puppetmaster(s)
19:09:21 <fungi> jasondotstar: it's a static file on the puppetmaster
19:09:21 <jasondotstar> clarkb: ack.
19:10:02 <jasondotstar> so we should think of an encryption method for sensitive data such as this...
19:10:29 <clarkb> the tricky thing is the chicke and egg with backing up the encryption key/sharedsecret
19:10:43 * jasondotstar agrees
19:10:48 <clarkb> I think thats where I got distracted last time I looked into this
19:11:22 <jasondotstar> pabelanger and i will capture this on the etherpad and see what we can come up with...
19:11:27 <fungi> yeah, it's turtles all the way down. we were considering encrypting it to the keys of the infra root admins or with a passphrase known only to infra root and leave it at that
19:11:59 <SpamapS> IMO, encrypt the backups w/ GPG to all of infra-root's keys. Anyone in infra-root then can restore the backup on their machine.
19:12:07 <fungi> but anyway, that's veering a bit off topic for validating our current backup mechanism
19:12:14 <nibalizer> ohai
19:12:17 <SpamapS> agree, just make sure it gets backed up
19:12:21 <jasondotstar> +1
19:12:21 <fungi> yeah
19:13:00 <fungi> #topic Specs approval
19:13:08 <fungi> #link https://review.openstack.org/#/q/status:open+project:openstack-infra/infra-specs,n,z
19:13:26 <fungi> we're not approving any specs in meeting, just a reminder that there are some awaiting reviews
19:13:51 <fungi> also the new storyboard dev team has explicitly requested review/approval of some outstanding storyboard specs
19:14:29 <pleia2> I didn't add it to the agenda, but this translations site spec is ready for review: https://review.openstack.org/#/c/184559/
19:14:44 <pleia2> I've been working with the i18n folks to refine it, and I think it's in a good place now
19:15:02 <fungi> since openstack project infra don't have a huge stake in it other than avoiding breaking it until we switch to something else, would be nice for us to give the storyboard team relatively wide berth on their specs i guess
19:15:05 <SotK> The StoryBoard spec in question is https://review.openstack.org/#/c/202989/
19:15:22 <fungi> #link https://review.openstack.org/202989
19:15:25 <fungi> thanks SotK!
19:15:35 <fungi> and apologies for us being somewhat absentee landlords on that
19:15:59 <clarkb> well didn't we say see how it goes?
19:16:11 <SotK> we did
19:16:14 <fungi> yep. they seem to be doing stuff
19:16:18 <clarkb> I am guessing this is and indication that infra isn't really interested in that?
19:16:35 <fungi> potentially, yes
19:17:10 <fungi> anyway, specs. we have some. i need to review them, and would be thrilled if others do to
19:17:45 <fungi> #topic Schedule Project Renames
19:18:18 <fungi> it was suggested in the aforementioned planning thread that we should probably also do some renames in september for projects that have already requested them
19:18:52 <fungi> we don't need to decide it today, but just making sure we keep it in mind and see if we can pick a good window when september draws a little closer
19:19:52 <fungi> #topic Priority Efforts
19:19:54 <clarkb> any initial suggestions?
19:19:56 <clarkb> oh too late
19:20:04 <fungi> #undo
19:20:05 <openstack> Removing item from minutes: <ircmeeting.items.Topic object at 0xa1ee7d0>
19:20:16 <fungi> it's not too late!
19:20:31 <Clint> well played
19:20:43 <fungi> i didn't have any initial suggestions though, but my calendar is wide open for september
19:20:48 <clarkb> there is the long weekend for MURICA we should avoid but otherwise maybe avoid burning man? I think that is next to the long weekend
19:21:05 * fungi has done a great job at avoiding burning man so far
19:21:08 <pleia2> after labor day I'm around all month
19:21:39 <clarkb> thats basically the first week of september is no go
19:21:43 <clarkb> maybe do second weekend?
19:22:18 <fungi> second weekend... like friday the 11th'/
19:22:20 <fungi> ?
19:22:22 <clarkb> ya
19:22:34 <pleia2> 11th and 12th are good for me
19:22:54 <fungi> wfm. i have some guests in town the following weekend
19:23:27 <fungi> #info tentative window to clear already requested project renames on september 11 or 12
19:23:31 <zaro> are we planning or have plans to move any infra projects around?  like from sourceforge to openstack or openstack-infra?
19:23:53 <fungi> s/ource/tack/
19:24:02 <fungi> not that i'm aware, no
19:25:18 <fungi> okay, nothing else on this i guess, so moving on
19:25:24 <fungi> (for real this time!)
19:25:29 <fungi> #topic Priority Efforts
19:26:02 <fungi> if there aren't any urgent updates for any of these, i'm going to move ahead to the backlog
19:26:09 <clarkb> jhesketh has a first pass at the swift index gen stuff
19:26:15 <fungi> anyone have a priority effort that's blocked/on fire?
19:26:15 <clarkb> so swift logs continues to plod along
19:26:36 <zaro> gerrit upgrade might be blocked forever.
19:26:48 <fungi> zaro: oh? details appreciated... link?
19:26:53 <yolanda> i'd like to request more reviews for several items on downstream-puppet
19:26:57 <zaro> nobody is working to fix the jgit issue #link https://groups.google.com/forum/#!msg/repo-discuss/CYYoHfDxCfA/BxkgOOx6CgAJ
19:27:05 <fungi> zaro: ugh
19:27:10 <clarkb> I think upstream gerrit may have given up on functioning properly
19:27:25 <fungi> ETOOHARD
19:27:34 <jhesketh> Yeah swift logs are slowly going. Not as fast as I'd like sorry
19:28:22 <fungi> no worries
19:28:26 <fungi> okay, moving along to the backlog
19:28:35 <fungi> #topic Restore from backup test (jeblair)
19:28:45 <fungi> this was mostly covered in the action items
19:29:04 <fungi> we should keep working through it, but i don't know that it needs any additional discussion here for the time being
19:29:23 <fungi> #topic Fedora 22 snapshots and / or DIBs feedback (pabelanger)
19:29:32 <fungi> looks like pabelanger's not here
19:29:42 <nibalizer> I believe he can be summoned
19:29:56 <nibalizer> maybe not
19:30:03 <ianw> we should be close to having dib images going
19:30:13 <ianw> hpcloud don't have f22 images, rax does
19:30:19 <fungi> yeah, i've been semi-reviewing the proposed changes
19:30:21 <pabelanger> opps
19:30:23 <fungi> i'll link them here
19:30:25 <ianw> if someone from hpcloud wants to fix that, great
19:30:26 <pabelanger> sorry, lost track of time
19:30:32 <fungi> #link https://review.openstack.org/211703
19:30:39 <fungi> #link https://review.openstack.org/211294
19:30:45 <fungi> #link https://review.openstack.org/186619
19:31:05 <ianw> devstack is broken on f22 at the moment due to a pip issues, hope to have something on that today .au time
19:31:21 <ianw> a real fix + workaround in the mean-time probably
19:31:23 <fungi> okay, great
19:31:39 <fungi> anything else specific you needed to cover for this in the meeting?
19:31:58 <clarkb> will we drop f21?
19:32:08 <ianw> sort of more priority effort, but request for review of the yum/dnf package caching for dib ... getting links
19:32:38 <ianw> #link https://review.openstack.org/208321
19:32:42 <fungi> phrased differently, should we expect the current jobs that run on f21 to work on f22 obce we have it booting?
19:33:03 <fungi> er, once
19:33:26 <ianw> my suggestion is to unify apt/yum paths to minimise our need to pass different env variables
19:33:56 <fungi> whatcha mean "unify apt/yum paths"?
19:34:17 <ianw> sorry, i mean in that review use common environment variables to indicate "keep the package cache"
19:34:28 <fungi> oh, right-o
19:34:36 <clarkb> +1
19:34:37 <ianw> as to the jobs, yes for devstack+tempest
19:34:43 <fungi> within dib's baked-in elements you mean
19:35:03 <ianw> i *think* there might be some docker-ish type stuff i know little about.  i imagine it can transition
19:35:16 <fungi> because it definitely has some widely varied caching behaviors/implementation between the dpkg element and the yum element
19:35:36 <ianw> yeah, that's what i've tried to unify with those changes
19:36:10 <fungi> okay, cool
19:36:18 <fungi> anything else on this?
19:36:29 <ianw> anyway, i think that it's all pretty contained within the reviews, so that's it
19:36:40 <fungi> f21 deprecation?
19:36:50 <fungi> don't see that answered
19:37:09 <clarkb> ianw naswered said yes for devstack + tempet
19:37:13 <clarkb> which is most of where we f21
19:37:17 <fungi> ahh, right-o
19:37:22 <ianw> the devstack job is super-unstable.  i haven't bothered fixing it because f22 is so close
19:37:35 <ianw> so it's half-dead already :)
19:37:38 <fungi> the puppet crowd i guess are the other primary consumers of those images?
19:37:49 <clarkb> and kolla/magnum
19:37:54 <fungi> tripleo have their own images, so can transition at their own pace i suppose
19:38:12 <ianw> yep; pabelanger & me will work with people to transition
19:38:21 <fungi> alright, great
19:38:36 <ianw> if anyone ever asked me, i've always told them the cost of testing on fedora is they're going to be on the upgrade cycle
19:39:07 <fungi> yeah, that's why we've generally not had important jobs run on non-long-term releases
19:39:28 <ianw> yep, so it's fair to expect people to put some effort into moving
19:39:28 <fungi> since we can't continue to maintain them in that state for our stable release branches
19:39:55 <fungi> #topic Normally there are no infra-roots in Europe, some of us want to volunteer for it (yolanda)
19:40:13 <yolanda> hi so the topic raised that week on some gerrit outage
19:40:42 <yolanda> we know we have jhesketh on Australia, and Sergey on Europe, but reality is that sometimes is difficult to have roots at certain hours
19:40:52 <mrmartin> good idea, if you don't plan to sleep in the next 6-12 months
19:40:52 <fungi> yolanda: thanks for being interested in helping more! general workflow for this is to sync up with the ptl and let them drive the conversation, so probably best to bend jeblair's ear when he's no longer in the midst of the ops mid-cycle
19:41:05 <yolanda> AJaeger raised the problem that there were no volunteers, so from Gozer we'd like to volunteer
19:41:23 <yolanda> yes, we'll contact jeblair for it
19:41:26 <fungi> yolanda: i think that's awesome
19:41:29 <mrmartin> +1 for EU infra root next to Sergey
19:41:52 <mrmartin> we are really loosing entire work days when something fails
19:41:59 <yolanda> we are 3 members on Europe that are able to help, and have the expertise, so we'll be contacting jeblair for it
19:42:27 <fungi> yep, anyway, lets let this conversation take its course through the proper channels when jeblair can be around
19:42:43 <yolanda> sure
19:42:55 <fungi> i just wanted to go ahead and bring it forward since you had it on the agenda. don't want you to think i'm skipping you!
19:43:04 <fungi> #topic puppet-pip vs puppet-python (rcarrillocruz)
19:43:21 <fungi> this got some discussion on the mailing list, though i think it might have reached a lull
19:43:48 <fungi> #link http://lists.openstack.org/pipermail/openstack-infra/2015-August/003015.html
19:44:38 <fungi> looks like it was covered in last week's meeting, so we can just kick that thread and see if we can bring it to a resolution i guess
19:45:02 <fungi> #topic Nodepool REST API spec (rcarrillocruz)
19:45:14 <fungi> this also looks like it's left over from last week
19:45:23 <nibalizer> yep
19:45:28 <fungi> was there anything else that needed covering on this one?
19:45:28 <pabelanger> I haven't had a chance to review the spec and leave comments
19:45:42 <fungi> yep, please review open specs ;)
19:45:51 * fungi is one of the worst offenders
19:46:22 <fungi> #topic Open discussion
19:46:34 <fungi> i give you 13.5 minutes
19:46:41 <yolanda> i wanted to raise the zuul services problem, we had some discussion on the mailing list
19:46:47 <yolanda> with several opinions
19:47:10 <yolanda> i'd like that we get on some agreement
19:47:20 <fungi> which list? infra or dev?
19:47:47 <fungi> or was it before august?
19:47:58 <yolanda> infra, we had the discussion last week
19:48:00 <fungi> not immediately finding the thread to gain some context
19:48:08 <yolanda> about if puppet modules should spin up services or not
19:48:22 <fungi> oh, that
19:48:27 <fungi> #link http://lists.openstack.org/pipermail/openstack-infra/2015-August/003038.html
19:48:38 <yolanda> ah, i didn't have the link handy
19:48:50 <sdake> fungi clarkb i am good from a kolla perspective with f21 deprecation in 1-2 weeks
19:49:08 <sdake> we want f22 though :)
19:49:15 <fungi> sdake: awesome, well we need the f22 images working anyway, yeah ;)
19:49:59 <fungi> yolanda: so i think we mostly settled on it being fine to parameterize the service ensure for running/stopped/undef
19:50:03 <sdake> in 1-2 weeks I will be removing our fedora-21 gating job entirely
19:50:12 <yolanda> fungi, that works for me
19:50:25 <sdake> as soon as my gating work finishes and merges for the new kolla build tool
19:50:41 <fungi> i think someone (crinkle? nibalizer? clarkb?) said ensure=>undef will have the desired outcome for our infra
19:50:50 <sdake> fungi but for f22, that gating job is furthe rout and not  an immediate priority
19:51:12 <clarkb> ensure => undef is really annoying
19:51:12 <crinkle> yes
19:51:18 <clarkb> it behaves differently in puppet3 and puppet4
19:51:19 <nibalizer> ya my last message on that thread is what I think we should do
19:51:26 <clarkb> but if thats the hack I guess ok?
19:51:41 <crinkle> clarkb: I don't think it will behave differently?
19:51:46 <nibalizer> clarkb: it does?
19:51:53 <clarkb> crinkle: using undef as a param makes it use the dfeault
19:52:01 <clarkb> in puppet3 but in 4 its actually a nil value aiui
19:52:15 <clarkb> so if there is a default for the thing you get the default which is probably not what you want if explicitly undefing
19:52:24 <crinkle> there's no default for ensure in the service type, is why not specifying it has worked so far
19:52:57 <clarkb> oh I thought it defaulted to running
19:53:13 <crinkle> if it defaulted to running then we wouldn't be arguing about adding ensure => running
19:53:13 <clarkb> but you must be right since its just dropped
19:53:15 <clarkb> ya
19:53:18 <fungi> and yeah, i feel relatively strongly that the way we run our services in the openstack project infrastructure should be the default behavior for the modules we publish, because anything else is hypocrisy, but it's perfectly fine to make the behavior configurable so that downstream consumers can choose to override that default to whatever behavior they prefer
19:53:42 <crinkle> fungi: ++
19:54:24 <pabelanger> puppet-lint voting jobs for system-config? What was decided?
19:54:32 <pabelanger> re-enable or not?
19:54:45 <fungi> but if the people responsible for maintaining zuul.openstack.org feel that ensure-running for the zuul services is a liability, then it feels wrong for us to make that a non-default behavior for the openstackinfra/zuul module on the forge
19:55:09 <clarkb> fungi: agreed we should keep the current behavior as default for all the reaons argued for on the thread
19:55:20 <pabelanger> I am infavor of expoising something to allow people to control it, does not matter the name or defaults.
19:56:16 <yolanda> i agree with pabelanger
19:56:42 <yolanda> parameterize , be flexible and not opinionated on it, and i'm perfectly fine on having the default non running for infra
19:56:53 <fungi> pabelanger: i think we have a consensus to not -1 over style concerns, but the current core reviewers for the system-config repo didn't express an interest in having that repo linted since it's not going to be published to the forge
19:57:03 <clarkb> well if you put it that way I think we _should_ be opinionated :P
19:57:14 <clarkb> we know a lot about zuul
19:57:18 <clarkb> we know a lot about running zuul
19:57:22 <clarkb> we should share that knoweldge
19:57:40 <crinkle> we have https://review.openstack.org/#/c/175226/1 to add linting, I think we should get that rebased and merged
19:58:44 <fungi> oh, i hadn't notived that one. so we need linting to give us forewarning of things we're currently doing in system-config that might break under puppet 4?
19:58:49 <fungi> er, noticed
19:59:04 <fungi> #link https://review.openstack.org/175226
19:59:18 <fungi> anyway, we're running out of time
19:59:24 <fungi> anything else in these last few seconds?
19:59:41 <Clint> no
19:59:50 <fungi> thanks everybody!
19:59:55 <fungi> #endmeeting