19:02:38 <fungi> #startmeeting infra
19:02:39 <openstack> Meeting started Tue Jan 19 19:02:38 2016 UTC and is due to finish in 60 minutes.  The chair is fungi. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:02:40 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:02:42 <openstack> The meeting name has been set to 'infra'
19:02:57 <fungi> #link https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting#Agenda_for_next_meeting
19:03:00 <zaro> o/
19:03:01 <olaph> o/
19:03:06 <fungi> #topic Announcements
19:03:47 <fungi> #info Volunteers sought to present Upstream Development track at the Newton summit in Austin, TX
19:03:58 <fungi> #link https://www.openstack.org/summit/austin-2016/call-for-speakers/
19:04:08 <fungi> #link https://etherpad.openstack.org/p/austin-upstream-dev-track-ideas
19:04:32 <fungi> pabelanger (who isn't in here?) has already volunteered, but let me know if you want to do something for this
19:04:40 <craige> o/
19:04:51 <SergeyLukjanov> o/
19:05:17 <fungi> #info Delegate sought for Cross-Project Specs Liaison
19:05:35 <AJaeger> fungi, if there's a topic in my area, I'm happy to (co)present
19:06:01 <pabelanger> o/
19:06:06 <fungi> basically we need someone reviewing the cross-project specs regularly and spotting when there are infra-specific items in them
19:06:29 <fungi> i'm on the hook for it right now, but if anyone's interested in taking that on instead, get up with me later
19:06:51 <fungi> #link https://wiki.openstack.org/wiki/CrossProjectLiaisons#Cross-Project_Spec_Liaisons
19:07:06 <fungi> any other important announcements not covered in meeting topics on the agenda>?
19:07:21 <fungi> going once... twice...
19:07:35 <jeblair> fungi: looks like there was an ml thread about the cpl?
19:07:35 <fungi> #topic Actions from last meeting
19:07:44 <fungi> jeblair: yeah, there was
19:08:08 <fungi> there were no action items last week, good for us!
19:08:20 <fungi> #topic Specs approval
19:08:21 <mordred> \o/
19:09:01 <fungi> we have two proposed for council vote
19:09:06 <jeblair> ( http://lists.openstack.org/pipermail/openstack-dev/2016-January/084136.html for those who are curious)
19:09:20 <fungi> PROPOSED: Move docs.o.o/releases to releases.o.o
19:09:28 <fungi> #link https://review.openstack.org/266506
19:09:38 <fungi> PROPOSED: Make translation set up consistent
19:09:47 <fungi> #link https://review.openstack.org/262545
19:09:55 <dhellmann> I need a bit of guidance on the releases.o.o one, but I think anteaya has explained one of my questions. The other question is how to handle the redirects.
19:10:04 <fungi> anyone object to setting the council vote deadline on these to thursday at 19:00?
19:10:15 <AJaeger> dhellmann: I can help with redirects - those are don in openstack-manuals
19:10:21 <dhellmann> AJaeger : excellent, thanks
19:10:22 <nibalizer> o/ (sorta)
19:10:26 <anteaya> I has no objection
19:10:33 <anteaya> english hard
19:10:43 <clarkb> fwiw I have long been an unfan of the proliferation of doc sites, but I also accept that we have decided to do it for various reasons. Might be good to keep track of the number of times we go back and forth and have to reconfigure apache and dns for this stuff to quantify why it is painful
19:10:45 <fungi> dhellmann: oh, is it still being tweaked, or are you ready for it to go up for approval?
19:11:01 <dhellmann> fungi : I am ready to go, I will just need help with the implementation
19:11:04 <jeblair> fungi, AJaeger, dhellmann: no objection here -- i take it from AJaeger's participation in the conversation that this won't come as a surprise for the docs folks?
19:11:10 <dhellmann> fungi : several of the patches to implement it are already up for review
19:11:31 <AJaeger> jeblair: I'm not sure - let me tell them...
19:11:44 <dhellmann> jeblair : I didn't consult with them, but they don't have much to do with that repo so I wasn't worried about it. I'll make sure they know the new URL.
19:11:46 <AJaeger> jeblair: you mean dhellmann's change, correct?
19:11:51 <jeblair> AJaeger, fungi: maybe we should get a docs ptl +1 on that?
19:11:52 <jeblair> AJaeger: yep
19:12:02 <clarkb> for translation consistency big +1
19:12:04 <AJaeger> loquacities: ^
19:12:09 <clarkb> it is something we need to do just havne't found time for it
19:12:11 <fungi> i agree
19:12:15 <jeblair> loquacities: specifically https://review.openstack.org/266506
19:12:17 <dhellmann> jeblair :  This isn't a docs repo. It's not for them to say.
19:12:43 <AJaeger> clarkb, fungi: I hope I covered everything for translations...
19:12:43 <jeblair> dhellmann: i mostly don't want to yank content from docs.o.o without them at least knowing about it
19:12:46 <fungi> oh, i thought we were talking about the translation consistency spec
19:12:51 <annegentle> dhellmann: honestly I think it's better for SEO and for people to understand where to fix stuff, but the "not for them to say" is a bit much :)
19:13:04 <annegentle> jeblair: yes, definitely let loquacities know
19:13:05 <dhellmann> jeblair : sure. I'll make sure they're aware of the new URL
19:13:07 <AJaeger> fungi: the translation one does not handle docs at all. That's a followup ;)
19:13:17 <fungi> but yeah, i guess giving the docs team a heads up that the release details are moving to a separate site is reasonable
19:13:26 <annegentle> fungi: +1
19:13:28 <fungi> AJaeger: oh, right--thanks
19:13:32 <AJaeger> translation of docs works fine, the lots of python and django projects cause trouble
19:13:56 <jeblair> if they -1 then we can have the territorial dispute, but if they +1 we can dodge the issue.  :)
19:14:23 <fungi> okay, i'll put them up for council vote and see if we can get loquacities to weigh in on the releases.o.o spec as a courtesy before then
19:14:58 <fungi> #info Voting is open on "Move docs.o.o/releases to releases.o.o" spec until 19:00 UTC Thursday, January 21
19:15:04 <AJaeger> fungi: I'm writing an email to docs team
19:15:15 <fungi> #info Voting is open on "Make translation set up consistent" spec until 19:00 UTC Thursday, January 21
19:15:32 <annegentle> thanks AJaeger
19:15:48 <fungi> anything else we need to handle regarding specs voting? if not, we're at 44 minutes left
19:16:36 <fungi> #topic Priority Efforts: Ansible Puppet Apply
19:16:43 <mordred> we're ready
19:16:49 <fungi> note here says "Ready to go live"
19:16:49 * jeblair boggles!
19:16:58 <mordred> all of the changes that are not the actual puppet apply go live patch have landed
19:16:58 <pleia2> \o/
19:17:10 <mordred> I have verified that we're copying the hiera files and the puppet files appropriately
19:17:14 <fungi> great! so we're on track to wrap that up later this week?
19:17:20 <mordred> (and in fact ran puppet apply by hand on git01.openstack.org)
19:17:25 <jeblair> or this afternoon? :)
19:17:26 <mordred> fungi: I was thinking this afternoon actually
19:17:36 <fungi> (or before the meeting is over?)
19:17:41 <fungi> heh, cool
19:17:51 <fungi> and congrats to all involved!
19:17:56 <fungi> this was a biggie
19:18:00 <jeblair> huzzah!
19:18:02 <nibalizer> neat
19:18:15 <nibalizer> mordred: and you said puppetdb posting was functional ya?
19:18:25 <fungi> someone also needs to propose a patch to infra-specs to do the cleanup (move it to implemented, take it out of the priority efforts list and query/url)
19:18:30 <mordred> nibalizer: I will verify that before landing the change
19:18:33 <fungi> nibalizer: as of just before the meeting, yep
19:18:39 <fungi> nibalizer: oh, neverminf
19:18:40 <nibalizer> so cool
19:18:44 <mordred> nibalizer: but I have all the pieces in place to be able to verify that and be satisfied with the verification
19:18:48 <fungi> you meant still functional once we switch to applying
19:18:55 <mordred> yes
19:19:20 <fungi> and what's the change for go-live, just for the record?
19:19:36 <fungi> someone #link us a url for that
19:19:38 <mordred> landing the go live patch - it should be mostly unnoticable
19:19:54 <mordred> #link https://review.openstack.org/249492
19:19:58 <fungi> appreciated
19:20:03 <mordred> the docs on the new system are in that patch
19:20:04 <clarkb> mordred: is there a doc change for explaining how to run puppet apply locally?
19:20:07 <mordred> so folks might want to review it
19:20:09 <mordred> yes
19:20:13 <clarkb> mordred: its not complicated and I did it myself at one point but oh good
19:20:21 <jeblair> what a cool number
19:20:28 <mordred> although luckily the answer is "puppet apply /opt/system-config/production/manifests/site.pp"
19:20:39 <mordred> so it's not bad
19:20:45 <fungi> okay, anything else we need to cover before moving on to swift logs?
19:20:48 <clarkb> ya I did it at one point testing things
19:20:48 <mordred> it even works on git*.o.o where we have the project-config sha fact
19:21:04 <mordred> I also included docs on how to run puppet manually from the puppetmaster via ansible
19:21:14 <mordred> in case you want to get hiera things copied and modules updated and whatnot
19:22:11 <mordred> ansible-playbook --limit "$host;localhost" /opt/system-config/production/playbooks/remote_puppet_allyaml  ... in case anybody was curious
19:22:23 <mordred> s/allyaml/all.yaml/
19:22:24 <fungi> #link http://docs.openstack.org/infra/system-config/puppet.html
19:22:31 <fungi> is where the bulk of that documentation is winding up
19:22:40 <fungi> for the benefit of those reading the meeting log
19:23:23 <fungi> #topic Priority Efforts: Store Build Logs in Swift
19:23:30 <fungi> Reviews still needed
19:23:35 <jeblair> mordred: 249492 is a beautiful change!  mostly docs and like 3 lines of code
19:23:40 <fungi> #link https://review.openstack.org/#/q/status:open+topic:enable_swift
19:23:44 <clarkb> I reviewed the zuul changes which need to go in before the config changes can happen
19:23:57 <clarkb> so I think the zuul changes would be the proirity
19:24:06 <mordred> jeblair: statistically I've got to at least SOMETIME produce one of those, right?
19:24:14 <fungi> looks like some of those should be ready to go in
19:24:34 <fungi> unless anyone else spots issues in there
19:25:02 <fungi> otherwise looks like same situation we were in last week, so moving on
19:25:05 <jeblair> er
19:25:14 <jeblair> what's the problem that 262112 is trying to solve?
19:25:17 <fungi> or not moving on yet
19:25:39 <clarkb> jeblair: we need to write to two containers with the second having a different root
19:25:48 <clarkb> jeblair: so that we can build a file system directory
19:26:09 <jeblair> clarkb: are we not having the log server do that for us?
19:26:15 <clarkb> jeblair: no
19:26:28 <jeblair> clarkb: i've clearly missed something.  how can i catch up?
19:26:31 <clarkb> jeblair: because that requires some method to write back to the server from unprivileged slaves
19:26:34 <fungi> does the spec update clarify why we aren't putting the filesystem metadata on the same container?
19:26:37 <clarkb> jeblair: the spec with that topic should catch it up
19:26:45 <clarkb> fungi: in a different container and yes
19:27:16 <mmedvede> clarkb: I think my change would allow to do that, have different container root https://review.openstack.org/#/c/229582/
19:27:37 <jeblair> okay.  i will read the spec update.  in the mean time, i'm not sure i'm on board.
19:28:10 <fungi> ah, got it, so it's the out-of-tree requirement driving having it on a different container
19:28:38 <fungi> this does, i think, mean that any malicious change could in theory trash the filesystem metadata, right?
19:29:01 <clarkb> fungi: yes, though in theory its hashed so semi difficult to trash particular things
19:29:07 <clarkb> the spec calls that out iirc
19:29:13 <anteaya> I read that
19:29:27 <anteaya> in the spec
19:29:28 <fungi> right, now i see the security section there
19:29:36 <clarkb> honestly I am beginning to think we shouldn't use swift at all
19:29:47 <jeblair> i also feel like this is a substantial enough change/setback that we should at least reconsider the alternatives of "a) keep using the big log server" and "b) use afs"
19:30:05 <clarkb> jeblair: or C) something else
19:30:07 <clarkb> I agree
19:30:10 <jeblair> clarkb: ya
19:30:13 <fungi> so the worker won't actually have access granted to upload anything into the metadata container
19:30:15 <fungi> ?
19:30:25 <fungi> (e.g. to overwrite files in it)
19:30:50 <clarkb> fungi: it will have enough access to write the metadata object, I am not sure if the acls allow us to prevent writing to other metadata objects
19:31:02 <fungi> that's what i was concerned about
19:31:06 <jeblair> clarkb: also a+b: keep jenkins scping to the big log server but have it actually backed with afs
19:31:16 <anteaya> it seems that uploading logs to a thing that has a file structure is useful
19:31:32 <clarkb> right so the underlying issue here is we actually find the posix filesystem to be useful
19:31:43 <clarkb> swift doesn't provide that so we either have ot build it ourselves or not use swift
19:32:05 <anteaya> and the build it ourselves has a security concern
19:32:35 <fungi> and teh answer from swift developers (which is entirely reasonable from their perspective) is that if our application is dependent on posix filesystem implementation details/features then we should redesign that application to need something different
19:33:20 <fungi> though i'm not sure that's possible if you consider the use case to be presenting a filesystem-like browsing experience to our end users
19:33:40 <krotscheck> o/
19:33:43 <fungi> it basically means we should tell the consumers of our log data that their expectations are wrong
19:33:59 <anteaya> that makes people sad
19:34:05 <anteaya> both the teller and the listener
19:34:09 <jeblair> i am sad
19:34:26 <fungi> so anyway, i guess let's get this additional feedback into the proposed changes
19:34:37 <fungi> and see if there's a compromise to be had
19:35:07 <fungi> #topic Priority Efforts: maniphest migration
19:35:24 <fungi> craige wanted some input on direction, according to the agenda
19:35:49 <craige> I did.
19:35:49 <fungi> craige: any details on what those questions were?
19:35:55 <fungi> (or where?)
19:37:04 <craige> Not at present.
19:37:35 <fungi> oh
19:37:37 * anteaya finds it hard to provide input
19:37:40 <craige> I'll have to punt them to -infra when I'm more awake.
19:37:50 <craige> it does. Sorry.
19:38:13 <fungi> craige: okay, please do. it's entirely reasonable to ask questions in #openstack-infra when they occur to you rather than waiting for the weekly meeting
19:38:31 <fungi> #topic puppetlabs-apache migration (pabelanger)
19:38:41 <fungi> #link https://review.openstack.org/205596
19:38:53 <pabelanger> ohau
19:38:55 <pabelanger> hi*
19:39:03 <fungi> pabelanger: what's the status and/or blocker on this work?
19:39:11 * anteaya likes ohau as a greeting
19:39:20 * jeblair likes oahu
19:39:25 <pabelanger> so, I wanted to see if people are still interested in the puppetlabs-apache migration?  I've had nodepool up for a month or so, but have no feedback
19:39:44 <pabelanger> so, before going crazy with patches, want to see if I can have some puppet core ready for reviews
19:40:00 <pabelanger> and make sure people are happy with apache::vhost:custom patch
19:40:58 <jeblair> maybe we need to ask when nibalizer is more around?
19:41:13 <pabelanger> as a reminder, this should finally get us of our puppet-http fork
19:41:49 <clarkb> pabelanger: maybe start by getting the apache module update change in?
19:41:58 <clarkb> pabelanger: it is currently marked WIP so not likely to get much attention
19:42:06 <clarkb> oh it merged
19:42:10 <clarkb> with WIP...
19:42:18 <jeblair> ha!
19:42:21 <pabelanger> Ya, not sure why that merged honestly :)
19:42:24 <pabelanger> but, it is in
19:42:40 <clarkb> well then ya probably best to start with a service and make sure it is happy then move on
19:42:55 <fungi> okay, so apache::vhost::custom is a working thing now. and looks from the example change like we don't lose our current capabilities
19:43:18 <fungi> thanks for driving that to a successful conclusion upstream!
19:43:24 <pabelanger> Ya, I can take this offline, like I say, just wouldn't mine puppet-core to confirm they are happy and I can get started on others
19:43:50 <clarkb> I think my biggest concern at this point would be making ure the module code works across platforms as we roll out
19:43:59 <clarkb> since we tripped over the apache mod stuff before
19:44:22 <pabelanger> Ya, puppetlabs-apache has some good test coverage for other OSs.
19:44:26 <pabelanger> and beaker jobs upstream
19:45:08 <fungi> anything else on this?
19:45:19 <pabelanger> none here
19:45:21 <fungi> i apparently need to run and sign for a package at the door real fast
19:45:26 <fungi> #topic gerritlib release for jeepyb to set gerrit project descriptions (yolanda, zaro)
19:45:58 <anteaya> #link https://review.openstack.org/#/c/110009/
19:46:03 <zaro> jeepyb feature to set project desriptions needs a gerritlib release
19:46:11 <zaro> to work.
19:46:17 <zaro> can someone do that?
19:46:46 <zaro> there hasn't been a release in a while.
19:47:04 <fungi> sorry about that, back now
19:47:17 <anteaya> https://review.openstack.org/#/admin/groups/730,members
19:47:47 <jeblair> i can do that, but i haven't been watching gerritlib and have no idea what's in the new version
19:47:52 <fungi> has anyone looked at what the divergence is from the last tag (number of significant changes) yet?
19:48:00 <fungi> er, what jeblair just said
19:48:19 <jeblair> zaro: are you relatively confident it's ready for release, won't cause major problems, etc?  also, are there any backwards incompatible changes?
19:48:47 <zaro> not atm. i would have to probably take another look.
19:48:57 <Swanson> I'm currently spamming the universe.  For a CI I want to chop down the jobs section of zuul's layout.yaml file to just what I care about, right?
19:48:57 <zaro> i have been doing reviews on changes though.
19:49:26 <anteaya> Swanson: please move to -infra
19:49:51 <zaro> i can take a look and let you know on -infra.
19:49:52 <fungi> skimming, maybe half a dozen notable changes in the log since 0.4.0
19:50:14 <Swanson> anteaya,  Whups!  Clicked the wrong tab.
19:51:08 <fungi> zaro: i can go ahead and tag this afternoon. looks like it's probably a 0.5.0
19:51:36 <fungi> #action fungi release gerritlib 0.5.0
19:51:44 <zaro> fungi: cool
19:51:59 <fungi> just get up with me first on whether the master branch tip testing is successful
19:52:16 <fungi> and i'll hold off tagging until you confirm
19:52:48 <zaro> the only openstack-infra thing i know that depends on gerritlib is jeepyb, does anything else depend on it?
19:52:49 <fungi> #topic update on gerrit performance after adding memory. Anybody experiencing slowness with pushes to gerrit? (zaro)
19:53:20 <fungi> zaro: we probably need a much longer discussion on this topic than the next 6 minutes will allow
19:53:40 <AJaeger> there're lots of reports of 404s and 502s - and some of these cause slowness since git review will retry...
19:53:41 <fungi> was there anything real quick you needed to bring to the team? or should we try to dig into it on #openstack-infra later?
19:54:11 <clarkb> my theory is as memory gets constrained we can't servce requests fast enough
19:54:13 <fungi> we could probably spend half a meeting just talking about the performance issues at a high level
19:54:23 <clarkb> then apache decides the backend has gone away and 500s everyone for a minute
19:55:11 <fungi> i want to sacve the last 5 minutes for pleia2's updates/reminders on the mid-cycle too
19:55:28 <anteaya> we have also experienced 502/503 after a service restart where gc isn't running
19:55:33 <zaro> we can take this offline.  i was just looking for feedback.  put this on the schedulde before yesterday's restart. so now i know we still have big problem.
19:55:34 <anteaya> fungi: agreed
19:55:53 <fungi> yeah, the one i saw yesterday right after restart seemed to coincide with a large outbound traffic spike
19:56:05 <fungi> so maybe an inadvertent dos condition
19:56:17 <anteaya> zaro: yes, we still have performance issues
19:56:26 <fungi> #topic Infra-cloud sprint (pleia2)
19:56:34 <fungi> #link https://wiki.openstack.org/wiki/Sprints/InfraMitakaSprint
19:56:42 <pleia2> so first, reminder that we want  the final attendee count to HPE for catering planning Friday, January 29th (11 days from now)
19:56:58 <pleia2> so if you haven't signed up, please do :)
19:57:15 <fungi> #info Deadline for attendee sign-up for the mid-cycle is Friday, January 29th (11 days from now)
19:57:20 <pleia2> HPE is also sending some infra-branded swag, so that'll be fun (t-shirts and stuff)
19:57:26 <fungi> neat!
19:57:31 <fungi> (thanks hpe!)
19:57:42 <pleia2> the last thing is hotels, I haven't booked yet but I don't want to end up at the hotel where no one else is since we'll need rides/cars/shuttles to the office in the morning
19:57:45 <clarkb> assuming I end up buying this house I may not end up going... would get keys just a couple days before sprint. Will keep things updated and will remove my name from list for catering purposes if that happens
19:57:46 <nibalizer> wooo
19:58:00 <pleia2> has anyone booked? I added a hotel column to the sign up
19:58:10 <mordred> oh - I have booked
19:58:10 <fungi> pleia2: i was going to just pick the nearest hotel, but i haven't yet no
19:58:11 <Clint> i'm at the courtyard
19:58:19 <anteaya> clarkb: would love to see you but also would love you to have a good house
19:58:25 <mordred> but I'm not at any of those hotels
19:58:39 <crinkle> i booked at the hilton
19:58:41 <fungi> but yeah, happy to stay at whichever is most popular
19:58:49 <anteaya> those hotels are within a rock's toss of each other
19:58:57 <anteaya> they basically share the same parking lot
19:59:03 <jeblair> mordred: where are you staying?
19:59:10 <pleia2> ok, can folks who book add their hotels to https://wiki.openstack.org/wiki/Sprints/InfraMitakaSprint#Registration ?
19:59:16 <pleia2> will help inform the rest of us ;)
19:59:19 <mordred> jeblair: I'm staying at the Fort Collins Marriott
19:59:23 <pleia2> then we can also coordinate rides with each other
19:59:31 <mordred> also - I'll only be there the last 2 days
19:59:44 <anteaya> yeah I don't post the hotel I'm staying at
19:59:50 <anteaya> anyone who wants to know can pm me
20:00:13 <fungi> yep, i agree privacy in these matters is entirely understaneable
20:00:13 <pleia2> anteaya: that's fine
20:00:14 <mordred> (with the marriott/starwood merger, I'm falling back to marriott now when starwood is not a choice on the thinking that perhaps the merger will merge my accounts)
20:00:18 <fungi> and we're out of time
20:00:21 <pleia2> thanks fungi
20:00:33 <anteaya> thank you
20:00:44 <fungi> dimtruck: thomasem: we'll have to get you next time
20:00:54 <thomasem> /sadpanda
20:00:55 <thomasem> okey dokey
20:00:57 <dimtruck> fungi: no worries!  we'll ping on -infra
20:00:58 <fungi> SergeyLukjanov: i didn't get to bring up the renames, but maybe we can talk in #openstack-infra later
20:01:00 <thomasem> Migth just hit y'all up in infra
20:01:01 <dimtruck> thanks
20:01:01 <thomasem> channel
20:01:04 <fungi> @endmeeting
20:01:06 <fungi> endmeeting
20:01:09 <fungi> #endmeeting