19:02:43 <fungi> #startmeeting infra
19:02:45 <openstack> Meeting started Tue Aug  1 19:02:43 2017 UTC and is due to finish in 60 minutes.  The chair is fungi. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:02:46 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:02:48 <openstack> The meeting name has been set to 'infra'
19:02:51 <pabelanger> o/
19:02:51 <fungi> #link https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting#Agenda_for_next_meeting
19:03:26 <fungi> #topic Announcements
19:03:26 <jeblair> o/
19:03:37 <fungi> #info Don't forget to register for the PTG if you're planning to attend
19:03:41 <fungi> #link https://www.openstack.org/ptg/  PTG September 11-15 in Denver, CO, USA
19:05:15 <fungi> #info Inactive accounts for several former infra-root sysadmins are being disabled; we thank them for their awesome service to the community and welcome them back any time!
19:05:53 <clarkb> does that come with a plaque to hang on their walls?
19:05:55 <fungi> #link https://review.openstack.org/489698 Disable inactive infra-root accounts
19:06:04 <jeblair> clarkb: more like a plague
19:06:10 <fungi> if only we had funding for placards
19:06:19 <fungi> but yes, they can hang a plague on their walls i suppose
19:06:25 <jeblair> fungi: you can send them stickers
19:06:28 * mordred waves
19:06:41 <AJaeger> signed by everybody?
19:06:53 <fungi> i do indeed have stickers
19:06:58 <fungi> not a terrible idea
19:07:08 <fungi> "see, if you come back, you can get even _more_ stickers"
19:07:24 <pabelanger> ++
19:07:32 <fungi> as always, feel free to hit me up with announcements you want included in future meetings
19:07:39 <fungi> #topic Actions from last meeting
19:07:55 <fungi> #link http://eavesdrop.openstack.org/meetings/infra/2017/infra.2017-07-25-19.03.html Minutes from last meeting
19:08:08 <fungi> #action fungi get switchport counts for infra-cloud
19:08:16 <fungi> still waiting to get word back from hpe at this point
19:08:33 <fungi> that was the only one we had from last meeting, looks like
19:08:52 <fungi> #topic Specs approval
19:08:59 <fungi> we don't seem to have anything new up this week
19:09:38 <fungi> #topic Priority Efforts - Gerrit 2.13 Upgrade: Request for testing (clarkb)
19:10:03 <clarkb> the zuul testing got done and everything worked great once we got tobiash's change in place
19:10:14 <clarkb> without that we did indeed see funny behavior and lack of ability to submit changes
19:10:37 <jeblair> we still need to merge that to zuul master, right?
19:10:57 <jeblair> and we should only do that after we change the label case in our current layout.yaml
19:11:02 <clarkb> jeblair: yes. I think it is still an open question about how to do that to master since it is backward incompatible (but allows greater gerrit compatibility)
19:11:26 <fungi> oh, hrm, might need to be toggleable?
19:11:38 <jeblair> 1) send email telling folks to change case; 2) change our case; 3) wait 1 week; 4) merge change ?
19:11:57 <clarkb> thinking about schedules for upgrading zuul and gerrit I think we (infra we) may not need it on zuulv2 but we likely want to support our other zuul users on v2 with newer gerrit as well
19:12:32 <jeblair> clarkb: ah, if we're pushing gerrit upgrade to post-ptg, then maybe leave master hanging?
19:12:53 <clarkb> jeblair: ya I'm thinking we can point zuulv2 users to the patch and they can apply locally if they find they need it
19:13:01 <clarkb> that way we don't have to worry about breaking people with existing installs
19:13:27 <jeblair> wfm
19:13:48 <clarkb> the other big item is removing reliance on the gerrit contact store feature which no logner exists in gerrit 2.13
19:14:09 <fungi> we got a bit near to release frenzy to feel safe taking a long downtime and significant behavior changes to the review system until next cycle
19:14:29 <clarkb> fungi has edits to the election roll generation scripts pushed up to update election roll generation which relied on the coupling of accounts via contact store
19:14:35 <clarkb> fungi: yup
19:14:39 <fungi> at this point the only thing we needed the contact store for is that
19:14:54 <clarkb> for the contact store stuff when can we update gerrit config to disable it? we can do that before we upgrade to 2.13 right?
19:15:04 <fungi> so we can in theory disable it whenever we want, though i'd like to have ptl elections behind us using the updated script
19:15:27 <clarkb> fungi: maybe you can push a change to update the config when you are comfortable doing so?
19:15:38 <fungi> yeah, i'll write that up in the next few days
19:15:40 <clarkb> and that leads us to scheduling the actual upgrade
19:15:55 <clarkb> last week was openstack feature freeze so we are deep into release activities and infra being slushy
19:16:22 <fungi> really the risk that someone will create a new account, not join the foundaiton, create a change, get it merged in a project with more than one ptl candidate and be surprised they didn't get a ballot is likely vanishingly small at this point
19:16:50 <fungi> between now and the contribution cut-off for qualifying to vote in the upcoming elections anyway
19:16:58 <clarkb> pike releases august 31, we then have a week without release or other major activity, then ptg, then another empty week
19:17:43 <clarkb> I think our good options for upgrading gerrit are teh week before or the week after ptg. I don't know how much stuff we want before ptg with plans for zuul upgrade
19:18:03 <clarkb> whcih leaves us with doing it week after ptg whcih is why I don't think we need to worry about case sensitivty fix on zuul master
19:18:13 <fungi> well, not without any release activity... the cycle-trailing projects are scrambling during that timeframe, as is documentation
19:18:40 <pabelanger> I don't mind week after PTG, unless we need 2.13 before PTG
19:18:43 <clarkb> fungi: those happen during ptg I think
19:18:54 <AJaeger> not sure about documnetation - we should have moved all release critical stuff to the project repos...
19:18:54 <fungi> clarkb: fair point, they do indeed
19:19:21 <fungi> clarkb: though there is still run-up to that release activity in the intervening week
19:19:21 <clarkb> I spoke to the release team this morning as well and they were happy with week before or after ptg
19:19:27 <clarkb> fungi: ya
19:19:38 <fungi> cool
19:19:47 <clarkb> also we have to do a full offline reindex so expect the downtime to easily be 5 hours or so probably longer
19:19:49 <fungi> AJaeger: that helps too, thanks
19:20:20 <fungi> right, this'll be one of those where we have it down for a good chunk of a weekend or something
19:21:03 <clarkb> september 17 18 19 20 21 22 23 is week after ptg
19:21:04 <fungi> may need to make sure we start early in the day on whatever day we pick, and/or plan to have people available to take over monitoring the reindex and ready to pick up the remaining tasks after that completes
19:21:34 <clarkb> yup I think early day to start is good idea and that way we don't have to leave it offline overnight
19:22:06 <fungi> "early in the day" relative to the bulk of the people volunteering
19:22:27 <clarkb> I'm thinking doing it monday the 18th might be a good option because people will be traveling and returning to work and shouldn't have as much a reliance on gerrit (but thats an assumption based on not really any data)
19:22:39 <clarkb> I fly home saturday myself
19:22:57 <fungi> yeah, i'll be available that monday if there's consensus around it
19:23:17 <pabelanger> same, fly home on saturday
19:23:23 <pabelanger> Monday works for me
19:23:29 <fungi> i can also get some prep going before the western americas volunteers if wanted
19:23:41 <fungi> since i'll be awake a little earlier
19:23:47 <clarkb> any other input from infra-roots on doing it monday?
19:24:05 <fungi> though if we have emea folk wanting to help with some of the early tasks that's cool too
19:24:24 <jeblair> logistically it works for me; a little concerned about doing it at start of week and not friday
19:25:02 <jeblair> clarkb: but i think you may be on to something with your post-ptg travel/burnout idea
19:25:06 <fungi> i recall a not so distant upgrade where we though doing it right before the weekend would be a good idea, and then didn't spot issues with it until activity picked up on monday
19:25:08 <jeblair> we'll just have to push through our own burnout.  :)
19:25:16 <fungi> s/though/thought/
19:25:21 <clarkb> right so doing it at the end of the week is the alternative. The upside to that is it will give us more tiem to prep without worrying about travel. Downsides are less friendly to emea and potentially makes that week really short for many PTGers
19:25:22 <jeblair> fungi: see, we enjoyed the weekend!  :)
19:25:31 <fungi> this is true
19:25:40 <clarkb> the other nice thing about friday is if we have to go into the next day it is saturday and not tuesday
19:26:06 <fungi> you certainly have an interesting definition of "nice" ;)
19:26:16 <clarkb> well for our users not necessarily us :)
19:26:25 <jeblair> i think you've pushed me over the fence; slight preference for monday here
19:26:27 <clarkb> fungi: the no users on saturday problem is a good point too I hadn't really considered
19:26:33 <clarkb> fungi: I think that makes me prefer monday
19:27:42 <clarkb> should we tentatively say 1500UTC monday and europe and east coast can get things rolling (I'll likely wake up early myself)
19:27:51 <fungi> wfm
19:27:57 <clarkb> that then gives us basically all day for north americans to finish it up
19:28:32 <clarkb> ok why don't we pencil that in and I'll send email about it tomorrow if no one can think of a reason to change it by then
19:28:47 <fungi> ohh
19:28:48 <AJaeger> clarkb: and then hand over to Australia and back to Europe ;)
19:28:57 <fungi> i need to check my calendar real quick :/
19:29:45 <fungi> yup, just confirmed that date sounded familiar... i may need to get permission for monday the 18th as that's my wife's birthday
19:29:54 <clarkb> uh oh
19:30:22 <clarkb> fungi: should we leave it penciled in while you talk about it? or should we not even push it :)
19:30:26 <fungi> let's say that's still the tentative plan, best case i just need to skip out early for dinner reservations or something
19:30:35 <clarkb> ok
19:30:52 <fungi> worst case i can't help with the upgrade, but we have more people
19:31:04 <jeblair> fungi: you volunteered for the early shift anyway :)
19:31:09 <fungi> too true
19:31:09 <clarkb> I think that is about it for gerrit then. If you haven't get given 2.13 a go on review-dev  please do so and give feedback
19:31:23 <clarkb> I confirmed it fixes the verified +1 but rechecked doesn't go into gate problem
19:31:30 <clarkb> so event stream does what zuul needs there to make the right choices
19:31:54 <fungi> #agreed Tentative Gerrit 2.13 upgrade date is Monday, September 18; outage starting around 15:00 UTC
19:32:10 <fungi> that sound about like what we agreed on?
19:32:14 <clarkb> yup
19:32:21 <jeblair> ya
19:32:32 <fungi> how long before the upgrade do we need to announce it on the lists?
19:32:42 <clarkb> think about it if there is a reason not to do it then let everyone know and we can rethink. But I want to send notice to the dev list soon. Probably tomorrow evening
19:32:51 <clarkb> just to give them as much lead time as possible
19:33:22 <fungi> that's some 7 weeks from yesterday i guess?
19:33:34 <clarkb> yup
19:33:46 <AJaeger> what about asking release team t oadd to their calendar at https://releases.openstack.org/pike/schedule.html - or queens?
19:33:47 <fungi> so definitely a nice advance warning
19:34:10 <clarkb> AJaeger: we could do that, not sure what they will say about it. I'lldo that after our meeting
19:34:24 <clarkb> *ask them, but only update once ml post is sent
19:34:36 <fungi> and what's the deal with the change screen? i guess polygerrit doesn't happen until 2.14 right?
19:34:37 <AJaeger> clarkb: agreed
19:34:58 <clarkb> fungi: you can run polygerrit against 2.13 but it isn't bundled as an option like it is in 2.14
19:35:03 <clarkb> so I was mostly ignoring polygerrit for 2.13
19:35:17 <tobiash> we don't have polygerrit in our 2.13
19:35:18 <clarkb> we can always deploy polygerrit bits later if we want (requires separate daemon and all that)
19:35:23 <fungi> okay, good, so not a lot of ui adjustment for people this time anyway
19:35:26 <fungi> i'm not so certain infra maintenance stuff is a fit for the release calendar, but we'll see what they say
19:36:00 <clarkb> 2.14 is going to be a big upgrade in all the ways
19:36:22 <clarkb> notedb, java 8, polygerrit, etc. That is why I didn't try pivoting to it, 2.13 is much simpler and easier to get done
19:36:30 <fungi> yup
19:36:57 <fungi> #action clarkb send advance notice of Gerrit 2.13 upgrade to mailing lists
19:37:16 <fungi> okay, anything else you wanted to cover before we switch topics?
19:37:24 <clarkb> nope that is all I had, thanks
19:37:49 <fungi> thanks!
19:37:51 <fungi> #topic Priority Efforts - Zuul v3 (jeblair)
19:38:05 <fungi> last-minute addition at jeblair's request
19:38:15 <jeblair> hi!  i mostly wanted to point out mordred's draft email about the ptg zuulv3 cutover
19:38:42 <jeblair> #link draft zuulv3 cutover announcement https://etherpad.openstack.org/p/ImHsFudauN
19:38:45 <fungi> thanks for the reminder. i need to make sure to read that shortly after the meeting
19:39:03 <jeblair> hopefully we can send that out soon (like today, tomorrow?)
19:39:06 <mordred> ++
19:39:19 <fungi> mordred: you're planning to send that one i take it?
19:39:32 <AJaeger> would be nice to merge https://review.openstack.org/#/c/489677 - so that mordred can link to it from his email
19:39:32 <mordred> fungi: yah - as soon as folks are ok with it
19:39:37 <mordred> AJaeger: ++
19:39:58 <fungi> #action mordred send advance notice of Zuul v3 cutover to mailing lists
19:40:02 <jeblair> and i had a suggestion at the end to link to an infra-manual page where we can have a living document to help devs with the transition
19:40:10 <jeblair> which is what AJaeger just linked
19:40:26 <fungi> awesome. also on my priority review list
19:41:02 <jeblair> probably that first change is enough for now.  it has an introduction, answers some basic questions, and has a skeleton we can fill in later.
19:41:31 <jeblair> so i think once that change lands, it's enough to link to from the announcement email as long as we contextualize it as a work in progress which will be more complete as we approach cutover
19:42:19 <jeblair> it's also 1 of three sets of docs i think we need before we go live:
19:42:47 <jeblair> openstack developer transition guide, zuul documentation, and job documentation
19:43:04 <jeblair> i'm working on getting the last published and cross-linked this week
19:43:20 <jeblair> (and continuing to improve zuul docs)
19:43:29 <jeblair> that's eot from me
19:43:42 <fungi> excellent. sounds entirely tractable
19:43:49 <fungi> questions?
19:44:50 <fungi> crickets? tumbleweeds?
19:44:59 <fungi> thanks jeblair!
19:45:32 <fungi> #topic PTG planning (fungi)
19:46:05 <fungi> #link https://etherpad.openstack.org/p/infra-ptg-queens Infra planning pad for Queens PTG in Denver
19:46:39 <fungi> seems we've grown a few more ideas there, this is shaping up nicely
19:47:13 <fungi> i see a number of things that clearly fit into either the helproom timeframe on monday/tuesday or as stuff we can hack on wednesday-friday
19:48:47 <jeblair> i agree with fungi's typing
19:49:04 <fungi> i've tried to tag them just now
19:49:10 <clarkb> I see the tags
19:49:18 <jeblair> what do we need to do to schedule that zuulv3 session thingy?
19:49:21 <fungi> [reserve] is for stuff where we should reserve a slot in one of the cross-project rooms
19:49:52 <fungi> there will be an ethercalc page
19:49:58 <fungi> already is
19:50:04 <jeblair> when do we do it?
19:50:46 <fungi> #link https://ethercalc.openstack.org/Pike-PTG-Discussion-Rooms Pike PTG - Extra Discussion Rooms schedule
19:51:00 <fungi> i think just pick times and stick stuff there
19:51:06 <fungi> first come, first served
19:51:42 <fungi> 30-minute slots, and the instructions say don't do more than two consecutive
19:51:55 <fungi> so we can book hour-long double slots if we need
19:52:22 <jeblair> monday is locked
19:53:02 <fungi> i wonder if there's a reason for that
19:53:15 <fungi> diablo_rojo or ttx might know, if either are around
19:53:52 <jeblair> i assume we can unlock it, but yeah, it may be best to assume that's on purpose and ask first.  :)
19:54:11 <jeblair> assuming it's possible, something like macon, monday, 2pm?
19:54:29 <fungi> sounds reasonable
19:54:39 <fungi> that's early enough in the week but after lunch
19:54:52 <jeblair> yeah, i assume we'll be very busy before lunch.
19:55:01 <fungi> that's my expectation as well
19:55:03 <jeblair> maybe still after lunch too, but gotta do it sometime.  :)
19:55:04 <clarkb> as will everyone else
19:55:35 <fungi> we could try to do 1:30-2:30 in macon but it'll likely be a slow start with everyone wrapping up lunch
19:55:54 <jeblair> and oh yeah, should we try to keep it to 30m?
19:56:12 <fungi> alternative would be to overlap neutron/ironic or service catalog discussions
19:56:20 <jeblair> maybe increase the likelihood that people will actually show up if we keep it brief
19:56:46 <fungi> could also do 4:30-5:30 but the first half of that overlaps a watcher discussion and the second half people will likely be scouting nearby bars alerady
19:56:50 <fungi> already
19:57:18 <fungi> if you think half an hour is a good target for the session, i'm down with that plan
19:57:44 <jeblair> it will be rushed, but i think that will be good.  i can literally talk for hours about this stuff.
19:57:58 <jeblair> no one wants that
19:58:06 <fungi> and not expecting more than ~50 attendees?
19:58:37 <jeblair> would be useful to have more.  think it'll happen?
19:58:52 <fungi> also plan to expect neutron/ironic people who may be disinterested in the topic wandering into macon before 2:30
19:58:59 <clarkb> my ptg room on debugging the gate was packed the first day, almost empty the second
19:59:10 <clarkb> so I expect yuo'll get quite a few people to show up if you do it once :)
19:59:34 <fungi> okay, we're about out of time
19:59:39 <jeblair> i'll try to get something scheduled
20:00:09 <fungi> thanks jeblair! i'll see if diablo_rojo or ttx can enlighten us on any reason for the locked monday cells
20:00:22 <fungi> thanks everyone for a productive meeting!
20:00:29 <fungi> #endmeeting