19:02:42 <fungi> #startmeeting infra
19:02:43 <openstack> Meeting started Tue Sep 27 19:02:42 2016 UTC and is due to finish in 60 minutes.  The chair is fungi. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:02:44 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:02:46 <openstack> The meeting name has been set to 'infra'
19:02:50 <zaro> o/
19:02:52 <fungi> #link https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting#Agenda_for_next_meeting
19:02:57 <fungi> #topic Announcements
19:03:04 <fungi> #info This is the last week to suggest topics for the cross-project workshop slots in Barcelona
19:03:06 <fungi> #link http://lists.openstack.org/pipermail/openstack-dev/2016-September/104579.html Cross Project Summit Sessions Planning ML thread
19:03:12 <fungi> #link https://etherpad.openstack.org/p/ocata-cross-project-sessions Cross Project Session Proposals pad
19:03:25 <fungi> #info fungi will be on vacation October 8-16; pleia2 has generously agreed to chair the October 11 meeting instead
19:03:40 <fungi> #topic Actions from last meeting
19:04:22 <fungi> #link http://eavesdrop.openstack.org/meetings/infra/2016/infra.2016-09-13-19.02.html
19:04:31 <fungi> fungi Test whether gerrit flush-caches significantly reduces memory utilization in Gerrit's JVM
19:04:38 <fungi> i tried and it didn't make any noticeable difference either in the javamelody graphs or overall performance
19:04:44 <clarkb> :/
19:04:47 <fungi> (gerrit still descended into a java gc death spiral shortly thereafter)
19:04:52 <fungi> however, zaro has excellent news on this front!
19:04:53 <fungi> which we'll cover later in the meeting agenda
19:05:07 <fungi> #topic Specs approval
19:05:12 <fungi> nothing new this week
19:05:19 <fungi> though i'm not expecting much else in the way of additions to crop up between now and the summit anyway
19:05:34 <fungi> unless maybe someone does a cleanup patch to move some to implemented
19:05:39 <fungi> #topic Priority Efforts: Zuul v3 (jeblair)
19:05:43 <fungi> jeblair has an update from discussions/work in walldorf and also linked the current tasks
19:05:49 <fungi> #link https://etherpad.openstack.org/p/zuulv3 Zuul v3 tasklist
19:05:58 <jeblair> howdy!
19:06:24 <jeblair> in walldorf we talked about what some zuul jobs would actually look like as ansible playbooks
19:06:47 <fungi> more representative examples than the early mock-ups?
19:06:56 <jeblair> yeah
19:07:03 <jeblair> in particulary, we dived deeply into devstack gate
19:07:05 <jeblair> and all of its uses
19:07:27 <jeblair> both for running devstack, and other non-devstacky things
19:07:39 <jeblair> one thing we realized is that we had not planned enough around roles in zuulv3
19:07:49 <jeblair> so i think i will write a spec update to try to capture that
19:07:58 <jeblair> maybe have it ready next week
19:08:06 * mordred is also going to document the sample playbook things we whiteboarded
19:08:10 <fungi> ooh, excellent
19:08:16 <jeblair> ^^ ++
19:08:37 <jeblair> and the other thing we realized is that we could start decomposing devstack-gate into ansible roles *today*
19:08:56 <fungi> that seems like a great way to get a head start on this
19:09:00 <jeblair> so that will help us get a head start on having some really nice jobs ready for v3
19:09:02 <jeblair> yep :)
19:09:07 <jeblair> and is nicely parallelizable
19:09:08 <fungi> i mean, we already rely semi-heavily on ansible in d-g now
19:09:24 <jeblair> so i have an etherpad where i jotted down things that i think people are working on
19:09:31 <jeblair> linked earlier ^ https://etherpad.openstack.org/p/zuulv3
19:09:45 <jeblair> at the bottom of that, we itemized all the roles we can extract from devstack-gate
19:10:04 <pabelanger> plan to work on the swap role tomorrow
19:10:13 <jeblair> so if you want to work on those, feel free to put your name next to it, and just go ahead and do it in the devstack-gate repo
19:11:01 <fungi> those look like a great breakdown of the reusable/boilerplate bits that aren't necessarily devstack-gate-specific
19:11:16 <jeblair> yeah, we figured they would be useful in all kinds of jobs
19:11:23 <jeblair> and would eventually live outside of the devstack-gate repo
19:11:28 <fungi> exactly. i think this is awesome
19:11:36 <fungi> modularize and simplify
19:11:43 <jeblair> (but before zuulv3, we can evolve them inside the d-g repo)
19:11:56 <clarkb> probably the biggest piece of that was using composition of roles rather than templates to get many jobs that are similar
19:11:56 <clarkb> which means we can actually start to chop up d-g into those peices before we zuulv3
19:12:11 <jeblair> (and if you're interested in any of the other areas in that etherpad, check in with me)
19:12:41 <fungi> also i realized teh other day, d-g is still not using zuul-cloner yet except for the clone it makes of itself?
19:12:50 <clarkb> fungi: yup
19:12:55 <clarkb> also woo lag
19:13:14 <jeblair> i *think* that's about it for zv3
19:13:31 <fungi> that's a lot!
19:13:42 <fungi> glad to see so much coming out of so few days together
19:13:49 <jeblair> i'm super excited about the d-g stuff
19:13:55 <mordred> it was really good to have a day in the room with people
19:14:16 <jhesketh> +1
19:14:22 <jeblair> yeah, just the thing we needed at just the right time
19:14:23 <fungi> anybody have any lingering questions for this?
19:15:12 <fungi> #topic Priority Efforts: Docs Publishing via AFS (jeblair)
19:15:14 <fungi> jeblair apparently has an update on current status of this effort
19:15:29 <jeblair> i both have and am soliciting updates :)
19:15:35 <fungi> hah
19:15:42 <jeblair> mostly, the week in germany distracted me to the point where i need to regroup on it
19:15:49 <jeblair> we have an etherpad here:
19:15:51 <jeblair> #link https://etherpad.openstack.org/p/afs-docs
19:15:52 <fungi> i will admit, i did nothing on this one while people were sprinting
19:15:58 <jeblair> and i think it's up to date
19:16:05 <fungi> but i'm curious to hear who did and what
19:16:23 <jeblair> and from poking this morning, i think the status is that we should have the keytab files in place now, and are ready to start publishing to the afs locations
19:16:39 <jeblair> we need to update the layout job to allow 'afs' publishers
19:16:41 <fungi> oh, that's, like, the bulk of it other than the import/migration work
19:16:59 <jeblair> we need to write the apache vhost defns for files.o.o
19:17:05 <fungi> so really closing in on this quickly
19:17:11 <clarkb> do we also need to resync the current cloud sites content?
19:17:16 <clarkb> I guess we do taht when we cut over?
19:17:21 <jeblair> clarkb: yeah, but closer to the end
19:17:34 <jeblair> clarkb: and recall, the current site content is just being used as a backup
19:17:42 <jeblair> we're going to switch over to entirely newly created content
19:17:52 <clarkb> right
19:17:56 <jeblair> http://files.openstack.org/ exists now
19:17:59 <fungi> and redirect questions about where content went to the docs team ;)
19:18:04 <jeblair> that just browses our afs root
19:18:13 <jeblair> fungi: yeah, and then copy things over manually if needed i guess
19:18:32 <jeblair> i can write the jjb plugin to handle afs
19:18:52 <jeblair> anyone want to volunteer for writing the vhosts?
19:18:57 <fungi> looks like mordred and pabelanger were the other two primary volunteers on this
19:19:08 <pabelanger> I can do the vhosts
19:19:12 <mordred> yes! I volunteered and then promptly did nothing
19:19:28 <fungi> and here's your chance!
19:19:58 <jeblair> mordred: i think pabelanger and i will have some patches for you to +3 soon :)
19:20:15 <fungi> so this is great, we're further along on it than i realized
19:20:22 <fungi> i know AJaeger and the docs team will be thrilled too
19:20:30 <AJaeger> Yeah!
19:20:31 <mordred> jeblair: yay!
19:20:50 <jeblair> yeah, i think the only other thing would be to start preparing patches that add afs publishers... but i think i'd hold off on that right now and wait until we see it succed for infra jobs first
19:20:58 <jeblair> just in case we find we need to change something about them
19:21:18 <fungi> definitely agree dogfooding this ourselves like we usually do is a smart way forward
19:21:30 <jeblair> fungi: yeah, that patch is here: https://review.openstack.org/#/c/377930/
19:21:38 <jeblair> it needs the jjb plugin i mentioned in order to land
19:21:42 <fungi> #link https://review.openstack.org/377930
19:21:44 <jeblair> but i think we'll have it by eod
19:21:59 <fungi> #undo
19:22:00 <openstack> Removing item from minutes: <ircmeeting.items.Link object at 0x7f66f46cabd0>
19:22:02 <fungi> #link https://review.openstack.org/377930 Publish infra docs to AFS
19:22:29 <jeblair> assuming my characterization of where we stand is correct, that's eot from me :)
19:22:44 <fungi> thanks! anything else we need to cover on this? anybody have related questions at this point?
19:23:46 <fungi> #topic Images mirror in AFS (therve)
19:23:49 <fungi> quoting from the agenda:
19:23:52 <fungi> It'd be useful to have a local mirror of minimal images (but bigger than cirros) to be used in devstack test runs. Heat currently download a Fedora image every run (about 200M), and it proves to be unreliable especially at peak hours. It's also fairly wasteful of network resources. We're not particularly attached to Fedora, though it's relatively small for what we need.
19:24:18 <fungi> therve isn't in channel, and mentioned in the agenda he may not be around
19:24:35 <fungi> but we've already discussed some of this elsewhere
19:24:59 <pabelanger> Ya, I it has been asked a few times.
19:25:32 <jeblair> ooh it's a FAQ
19:25:43 <fungi> i don't think there's been a lot of disagreement that a mirror volume with (reasonably small) guest images needed by infrequently-to-semifrequently-used jobs makes some sense
19:25:45 <mordred> compared to the other things we mirror, I don't see it as being any _harder_
19:26:22 <pabelanger> maybe something to try after afsdocs land
19:26:26 <fungi> right. i think the missing piece is some framework where we can declare what blobs we want to download/update, and arrange them in afs in a sensible fashion
19:27:09 <fungi> i'd imagine a couple dozen line script and a yaml config would just about cover it
19:27:24 <mordred> we could also write it in C++
19:27:31 <mordred> but I agree with fungi's suggestion
19:27:41 <jeblair> mordred: or ansible?
19:27:44 * fungi thinks ada is the right language for this
19:27:46 <fungi> do we want a spec for this? or just someone to step forward with patches?
19:28:02 * fungi would also have accepted lisp
19:28:03 <jeblair> i would not mind a super-short spec
19:28:22 <jeblair> mostly just to have a place to agree on mechanism and interface
19:29:00 <fungi> #agreed Guest images in our AFS mirrors make sense, and should begin with a (brief) spec where we can refine the configuration and automation model
19:29:02 <jeblair> (but will not stand on principle if someone has patches ready)
19:29:55 <fungi> now hopefully therve checks the meeting minutes, or we can point him at them later
19:30:09 <clarkb> (I have missed the last ~7 minutes)
19:30:15 * Shrews perks up at C++ mention
19:30:15 <anteaya> clarkb: welcome back
19:30:23 <jeblair> clarkb: afs and c++ were discussed
19:30:31 <fungi> also ada and lisp
19:30:42 <anteaya> clarkb: agreement on the suggestion, spec desired but patches will win
19:30:51 <clarkb> C++ for?
19:30:56 <fungi> clarkb: a joke
19:30:58 <clarkb> ah
19:31:14 <fungi> because if mordred were serious he would have said "rust"
19:31:26 <jeblair> though... i think c++ is a joke like how we used to use afs as a joke.
19:32:31 <fungi> i don't mind writing c++ (or even just straight c99, and would be willing to write k&r c even), though i don't think this is a particularly sane use case for it
19:32:50 <fungi> anyway, i think we can move on
19:32:55 <mordred> clarkb: it likely would have been clearer if it was s/afs and c++ were discussed/afs was discussed. mordred said c++/
19:33:14 <fungi> #topic Infra support for final release tagging on 6 Oct (dhellmann)
19:33:16 <clarkb> mordred: indeed
19:33:30 <fungi> quoting from the agenda:
19:33:31 <fungi> It would be helpful to have someone on standby Thurs 6 Oct when we are tagging the final Newton release, just in case something goes wrong during the process.
19:33:32 <dhellmann> fungi : o/
19:33:38 <fungi> dhellmann: count me as around and willing to spend the day focused on this
19:33:43 <dhellmann> excellent, thanks
19:33:44 <fungi> though we may want someone from apac/emea around as well to provide support before i wake up?
19:33:51 <jeblair> that's a normal workday for me
19:34:11 <dhellmann> well, we did it early last time because I was in europe, but this time I think we'll probably go back to doing it ~14:00 UTC
19:34:16 <fungi> yeah, i only occasionally take thursdays off to go drinking
19:34:32 <pabelanger> I'm happy to lend a hand if needed
19:34:40 <mordred> I will be in Berlin - but it seems there will be other humans working and not in Berlin
19:34:45 <anteaya> that is the first day of your vacation too fungi according to your earlier announcement
19:34:45 <clarkb> I will be around
19:34:53 <fungi> dhellmann: i expect we'll have a mostly full compliment of infra-root and other team members on hand
19:34:54 <dhellmann> the process will be me submitting a patch to tag finals for all of the milestone projects, and then watching while the bots do their thing
19:35:14 <dhellmann> good, I appreciate that
19:35:31 <fungi> anteaya: nope, announcement said i wasn't leaving until the 8th, and i mostly trust it
19:35:40 <jhesketh> I'm technically on leave but happy to watch for pings/be on pager duty for apac
19:35:40 <dhellmann> I sort of expected so, but I thought it wise and polite to officially coordinate
19:36:15 <anteaya> fungi: ah so you are here on the 8th, gone the 9th?
19:36:22 <anteaya> dhellmann: thank you
19:36:26 <fungi> dhellmann: also i'll make sure we're well iced over for potentially disruptive changes late in the week
19:36:43 <dhellmann> fungi : oh, that would be good, too
19:36:43 <fungi> anteaya: i said "on vacation October 8-16"
19:37:05 <fungi> release day is october 6
19:37:25 <anteaya> ah sorry thought we were discussing the 8th, my confusion
19:37:25 <dhellmann> I want to say last cycle it took ~2 hrs, though maybe a little more.
19:38:05 <fungi> well, i plan to expect you to need the entire day, just in case
19:38:23 <dhellmann> yeah, that's a good plan
19:38:37 <dhellmann> and that's all I had, unless anyone has questions?
19:38:50 <jeblair> (and that way, if we finish early, nothing else to do but start drinking)
19:39:02 <dhellmann> jeblair : I'm supposed to wait until we're done?
19:39:04 * dhellmann makes a note
19:39:18 <fungi> no questions from me, just good luck with the release, and if it catches on fire i hope it's not because of anything _i_ built! ;)
19:39:23 <jeblair> dhellmann: and take a picture!  (see ttx precedent)
19:39:40 <fungi> indeed, table covered in diagrams and checklists and wine
19:39:44 <clarkb> fwiw e-r is happy again so we can use that to track last minute things
19:39:55 <anteaya> clarkb: yay
19:40:02 <fungi> let's try and keep on top of diagnostic tools heading into release week, yes
19:40:09 <dhellmann> jeblair : https://doughellmann.com/blog/2016/04/07/openstack-mitaka-release-complete/
19:40:15 <fungi> they're our best early warning signs of possible hiccups
19:40:33 <jeblair> dhellmann: welp, can't top that.
19:40:47 <dhellmann> I've been working all cycle on my costume...
19:41:12 <anteaya> that's a visual
19:41:21 <fungi> a dancing martini shaker?
19:41:29 <fungi> i'm not sure i can un-see this now
19:41:40 <dhellmann> heh
19:41:52 <fungi> okay, ~20 minutes left for two topics
19:41:55 <fungi> thanks dhellmann!
19:42:01 <dhellmann> thanks!
19:42:04 <fungi> #topic Potential Jgit Performance Fix (zaro)
19:42:11 <fungi> #link https://groups.google.com/d/msg/repo-discuss/oj3h3JdioGs/_pNKjlUGBAAJ our saviors
19:42:36 <zaro> so i created a custom gerrit 2.11 build with custom jgit fix
19:42:46 <zaro> i think we should give that a try.
19:42:57 <fungi> "custom jgit fix" meaning backporting new jgit to gerrit 2.11?
19:43:28 <zaro> yes, just a few changes.
19:43:31 <fungi> and you have it running on review-dev right now, correct?
19:43:38 <zaro> yes
19:43:50 <fungi> this is supposedly the version of jgit shipping in current gerrit 2.13 builds?
19:44:08 <zaro> no, not exactly.
19:44:32 <zaro> the one with 2.13 has more than the receipe on the mailing list
19:44:56 <fungi> oh, i see
19:45:23 <zaro> the custom jgit i built only contains cherry-picks specified in the mail on top of jgit 4.0.x
19:45:26 <fungi> so the recipe from the discussion is what you ported to 2.11, rather than pulling down jgit 4.4.1
19:45:35 <zaro> jgit with 2.13 is jgit ver 4.4.x
19:46:05 <fungi> is this preferable over switching to 4.4.1?
19:46:39 <zaro> yes, you are correct.
19:47:12 <zaro> yes, it was what was recommended. less risky than using jgit 4.4.x with gerrit 2.11
19:47:17 <fungi> anyway, i guess what i wanted to get a feel for is whether we should shoot for restarting gerrit with patched/updated jgit in the next few days, before we get into release week, or wait until after newton is done (and i'm on vacation)?
19:48:03 <fungi> i didn't feel comfortable trying it while mostly on my own last week and unable to get a lot of input from the team
19:48:10 <anteaya> how much time do we have after you are back from vacation before people start flying to summit?
19:48:37 <fungi> i think i'm home something like 4 days before i have to travel to barcelona
19:49:07 <anteaya> do we want to try it in that window? or too close to summit?
19:49:32 <clarkb> I would rather not do it nowish with the release happening and the other things we are juggling seems like it would be a lot to get in order
19:49:37 <fungi> my calendar says i'm home 5 days (17th through 21st)
19:49:47 <anteaya> there is also oct 7
19:49:56 <anteaya> the day after release and the day before you vacation
19:50:16 <fungi> okay, so suggestion is to keep restarting gerrit when gc load gets high, at least until after release day
19:50:35 <anteaya> +1 don't change before release
19:50:51 <clarkb> also are we using the bitmapped indexes like spearce suggests?
19:51:04 <fungi> i mean, we can also consider a gerrit upgrade to 2.13 immediately after the summit
19:51:31 <zaro> i think there's things to consider before jumping to 2.13
19:51:33 <anteaya> is 2.13 released and in production somewhere?
19:51:59 <fungi> zaro: oh, as far as caveats that might cause us to want to stick with 2.11 or maybe only upgrade to 2.12?
19:52:38 <fungi> anteaya: there's even a 2.13.1 release now
19:52:42 <anteaya> wow
19:52:47 * anteaya is behind the times
19:52:49 <zaro> initial release was not good, already had a few patched releases i think.  so we might want to wait a little
19:53:08 <fungi> well, "immediately after the summit" is still more than a month away
19:53:28 <zaro> also i think we may need to do some prep stuff.  like get a zuul-dev in place so we can test gerrit->zuul->nodepool testing
19:53:54 <fungi> anyway, i think we've at least established we'll push this jgit update test off a couple weeks or more, so good enough for today
19:54:16 <fungi> #agreed Wait until after Newton release day to try updating Jgit on review.openstack.org
19:54:28 <fungi> thanks for your work on this, zaro!
19:54:35 <anteaya> thanks zaro
19:54:53 <fungi> #topic Ocata Summit Planning (fungi)
19:55:14 <fungi> just an update, i've got our planning pad updated with the now or soon-to-be finalized schedule
19:55:29 <fungi> #link https://etherpad.openstack.org/p/infra-ocata-summit-planning Ocata Summit Planning pad
19:56:06 <fungi> we're short on time during today's meeting, but try and make sure to capture any additional session topic ideas you've had in there
19:56:24 <fungi> especially stuff that might have come up in walldorf
19:56:50 <fungi> also, note that most of our workroom slots cover friday morning, and are the same room as friday afternoon's sprint
19:57:16 <fungi> so we have the option of turning up to 4 workroom slots into additional sprint time, if we want to start the sprint earlier in the day
19:57:52 <fungi> i'll go ahead and switch to open discussion for the next couple minutes, but we can continue talking about summit sessions during that time if people want
19:57:57 <fungi> #topic Open discussion
19:58:50 <anteaya> I don't know as it is realistic to expect ptls from nova neutron glance and cinder to all be able to be in the same room at summit
19:59:03 <anteaya> it is a timing issue
19:59:05 <clarkb> anteaya: it isn't, but if you can get a sampling I think that would be good and possible
19:59:11 <pleia2> I wrote about our thing: http://princessleia.com/journal/2016/09/openstack-qainfrastructure-meetup-in-walldorf/
19:59:17 <russellb> cross project session time is good for that.
19:59:18 <anteaya> sure if you can
19:59:58 <fungi> yea, agree with russellb, cross-project sessions for that sort of thing. see announcement from earlier for proposing those
20:00:00 <fungi> and we'
20:00:00 <anteaya> russellb: would a cross project session dedicated to debugging infra cloud be accepted?
20:00:04 <fungi> we're out of time
20:00:17 <russellb> i don't know, i'm not organizing it
20:00:20 <fungi> thanks everyone!
20:00:20 <ttx> anteaya: sounds like fun, why not
20:00:28 <fungi> #endmeeting