19:02:43 #startmeeting infra 19:02:45 Meeting started Tue Aug 1 19:02:43 2017 UTC and is due to finish in 60 minutes. The chair is fungi. Information about MeetBot at http://wiki.debian.org/MeetBot. 19:02:46 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 19:02:48 The meeting name has been set to 'infra' 19:02:51 o/ 19:02:51 #link https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting#Agenda_for_next_meeting 19:03:26 #topic Announcements 19:03:26 o/ 19:03:37 #info Don't forget to register for the PTG if you're planning to attend 19:03:41 #link https://www.openstack.org/ptg/ PTG September 11-15 in Denver, CO, USA 19:05:15 #info Inactive accounts for several former infra-root sysadmins are being disabled; we thank them for their awesome service to the community and welcome them back any time! 19:05:53 does that come with a plaque to hang on their walls? 19:05:55 #link https://review.openstack.org/489698 Disable inactive infra-root accounts 19:06:04 clarkb: more like a plague 19:06:10 if only we had funding for placards 19:06:19 but yes, they can hang a plague on their walls i suppose 19:06:25 fungi: you can send them stickers 19:06:28 * mordred waves 19:06:41 signed by everybody? 19:06:53 i do indeed have stickers 19:06:58 not a terrible idea 19:07:08 "see, if you come back, you can get even _more_ stickers" 19:07:24 ++ 19:07:32 as always, feel free to hit me up with announcements you want included in future meetings 19:07:39 #topic Actions from last meeting 19:07:55 #link http://eavesdrop.openstack.org/meetings/infra/2017/infra.2017-07-25-19.03.html Minutes from last meeting 19:08:08 #action fungi get switchport counts for infra-cloud 19:08:16 still waiting to get word back from hpe at this point 19:08:33 that was the only one we had from last meeting, looks like 19:08:52 #topic Specs approval 19:08:59 we don't seem to have anything new up this week 19:09:38 #topic Priority Efforts - Gerrit 2.13 Upgrade: Request for testing (clarkb) 19:10:03 the zuul testing got done and everything worked great once we got tobiash's change in place 19:10:14 without that we did indeed see funny behavior and lack of ability to submit changes 19:10:37 we still need to merge that to zuul master, right? 19:10:57 and we should only do that after we change the label case in our current layout.yaml 19:11:02 jeblair: yes. I think it is still an open question about how to do that to master since it is backward incompatible (but allows greater gerrit compatibility) 19:11:26 oh, hrm, might need to be toggleable? 19:11:38 1) send email telling folks to change case; 2) change our case; 3) wait 1 week; 4) merge change ? 19:11:57 thinking about schedules for upgrading zuul and gerrit I think we (infra we) may not need it on zuulv2 but we likely want to support our other zuul users on v2 with newer gerrit as well 19:12:32 clarkb: ah, if we're pushing gerrit upgrade to post-ptg, then maybe leave master hanging? 19:12:53 jeblair: ya I'm thinking we can point zuulv2 users to the patch and they can apply locally if they find they need it 19:13:01 that way we don't have to worry about breaking people with existing installs 19:13:27 wfm 19:13:48 the other big item is removing reliance on the gerrit contact store feature which no logner exists in gerrit 2.13 19:14:09 we got a bit near to release frenzy to feel safe taking a long downtime and significant behavior changes to the review system until next cycle 19:14:29 fungi has edits to the election roll generation scripts pushed up to update election roll generation which relied on the coupling of accounts via contact store 19:14:35 fungi: yup 19:14:39 at this point the only thing we needed the contact store for is that 19:14:54 for the contact store stuff when can we update gerrit config to disable it? we can do that before we upgrade to 2.13 right? 19:15:04 so we can in theory disable it whenever we want, though i'd like to have ptl elections behind us using the updated script 19:15:27 fungi: maybe you can push a change to update the config when you are comfortable doing so? 19:15:38 yeah, i'll write that up in the next few days 19:15:40 and that leads us to scheduling the actual upgrade 19:15:55 last week was openstack feature freeze so we are deep into release activities and infra being slushy 19:16:22 really the risk that someone will create a new account, not join the foundaiton, create a change, get it merged in a project with more than one ptl candidate and be surprised they didn't get a ballot is likely vanishingly small at this point 19:16:50 between now and the contribution cut-off for qualifying to vote in the upcoming elections anyway 19:16:58 pike releases august 31, we then have a week without release or other major activity, then ptg, then another empty week 19:17:43 I think our good options for upgrading gerrit are teh week before or the week after ptg. I don't know how much stuff we want before ptg with plans for zuul upgrade 19:18:03 whcih leaves us with doing it week after ptg whcih is why I don't think we need to worry about case sensitivty fix on zuul master 19:18:13 well, not without any release activity... the cycle-trailing projects are scrambling during that timeframe, as is documentation 19:18:40 I don't mind week after PTG, unless we need 2.13 before PTG 19:18:43 fungi: those happen during ptg I think 19:18:54 not sure about documnetation - we should have moved all release critical stuff to the project repos... 19:18:54 clarkb: fair point, they do indeed 19:19:21 clarkb: though there is still run-up to that release activity in the intervening week 19:19:21 I spoke to the release team this morning as well and they were happy with week before or after ptg 19:19:27 fungi: ya 19:19:38 cool 19:19:47 also we have to do a full offline reindex so expect the downtime to easily be 5 hours or so probably longer 19:19:49 AJaeger: that helps too, thanks 19:20:20 right, this'll be one of those where we have it down for a good chunk of a weekend or something 19:21:03 september 17 18 19 20 21 22 23 is week after ptg 19:21:04 may need to make sure we start early in the day on whatever day we pick, and/or plan to have people available to take over monitoring the reindex and ready to pick up the remaining tasks after that completes 19:21:34 yup I think early day to start is good idea and that way we don't have to leave it offline overnight 19:22:06 "early in the day" relative to the bulk of the people volunteering 19:22:27 I'm thinking doing it monday the 18th might be a good option because people will be traveling and returning to work and shouldn't have as much a reliance on gerrit (but thats an assumption based on not really any data) 19:22:39 I fly home saturday myself 19:22:57 yeah, i'll be available that monday if there's consensus around it 19:23:17 same, fly home on saturday 19:23:23 Monday works for me 19:23:29 i can also get some prep going before the western americas volunteers if wanted 19:23:41 since i'll be awake a little earlier 19:23:47 any other input from infra-roots on doing it monday? 19:24:05 though if we have emea folk wanting to help with some of the early tasks that's cool too 19:24:24 logistically it works for me; a little concerned about doing it at start of week and not friday 19:25:02 clarkb: but i think you may be on to something with your post-ptg travel/burnout idea 19:25:06 i recall a not so distant upgrade where we though doing it right before the weekend would be a good idea, and then didn't spot issues with it until activity picked up on monday 19:25:08 we'll just have to push through our own burnout. :) 19:25:16 s/though/thought/ 19:25:21 right so doing it at the end of the week is the alternative. The upside to that is it will give us more tiem to prep without worrying about travel. Downsides are less friendly to emea and potentially makes that week really short for many PTGers 19:25:22 fungi: see, we enjoyed the weekend! :) 19:25:31 this is true 19:25:40 the other nice thing about friday is if we have to go into the next day it is saturday and not tuesday 19:26:06 you certainly have an interesting definition of "nice" ;) 19:26:16 well for our users not necessarily us :) 19:26:25 i think you've pushed me over the fence; slight preference for monday here 19:26:27 fungi: the no users on saturday problem is a good point too I hadn't really considered 19:26:33 fungi: I think that makes me prefer monday 19:27:42 should we tentatively say 1500UTC monday and europe and east coast can get things rolling (I'll likely wake up early myself) 19:27:51 wfm 19:27:57 that then gives us basically all day for north americans to finish it up 19:28:32 ok why don't we pencil that in and I'll send email about it tomorrow if no one can think of a reason to change it by then 19:28:47 ohh 19:28:48 clarkb: and then hand over to Australia and back to Europe ;) 19:28:57 i need to check my calendar real quick :/ 19:29:45 yup, just confirmed that date sounded familiar... i may need to get permission for monday the 18th as that's my wife's birthday 19:29:54 uh oh 19:30:22 fungi: should we leave it penciled in while you talk about it? or should we not even push it :) 19:30:26 let's say that's still the tentative plan, best case i just need to skip out early for dinner reservations or something 19:30:35 ok 19:30:52 worst case i can't help with the upgrade, but we have more people 19:31:04 fungi: you volunteered for the early shift anyway :) 19:31:09 too true 19:31:09 I think that is about it for gerrit then. If you haven't get given 2.13 a go on review-dev please do so and give feedback 19:31:23 I confirmed it fixes the verified +1 but rechecked doesn't go into gate problem 19:31:30 so event stream does what zuul needs there to make the right choices 19:31:54 #agreed Tentative Gerrit 2.13 upgrade date is Monday, September 18; outage starting around 15:00 UTC 19:32:10 that sound about like what we agreed on? 19:32:14 yup 19:32:21 ya 19:32:32 how long before the upgrade do we need to announce it on the lists? 19:32:42 think about it if there is a reason not to do it then let everyone know and we can rethink. But I want to send notice to the dev list soon. Probably tomorrow evening 19:32:51 just to give them as much lead time as possible 19:33:22 that's some 7 weeks from yesterday i guess? 19:33:34 yup 19:33:46 what about asking release team t oadd to their calendar at https://releases.openstack.org/pike/schedule.html - or queens? 19:33:47 so definitely a nice advance warning 19:34:10 AJaeger: we could do that, not sure what they will say about it. I'lldo that after our meeting 19:34:24 *ask them, but only update once ml post is sent 19:34:36 and what's the deal with the change screen? i guess polygerrit doesn't happen until 2.14 right? 19:34:37 clarkb: agreed 19:34:58 fungi: you can run polygerrit against 2.13 but it isn't bundled as an option like it is in 2.14 19:35:03 so I was mostly ignoring polygerrit for 2.13 19:35:17 we don't have polygerrit in our 2.13 19:35:18 we can always deploy polygerrit bits later if we want (requires separate daemon and all that) 19:35:23 okay, good, so not a lot of ui adjustment for people this time anyway 19:35:26 i'm not so certain infra maintenance stuff is a fit for the release calendar, but we'll see what they say 19:36:00 2.14 is going to be a big upgrade in all the ways 19:36:22 notedb, java 8, polygerrit, etc. That is why I didn't try pivoting to it, 2.13 is much simpler and easier to get done 19:36:30 yup 19:36:57 #action clarkb send advance notice of Gerrit 2.13 upgrade to mailing lists 19:37:16 okay, anything else you wanted to cover before we switch topics? 19:37:24 nope that is all I had, thanks 19:37:49 thanks! 19:37:51 #topic Priority Efforts - Zuul v3 (jeblair) 19:38:05 last-minute addition at jeblair's request 19:38:15 hi! i mostly wanted to point out mordred's draft email about the ptg zuulv3 cutover 19:38:42 #link draft zuulv3 cutover announcement https://etherpad.openstack.org/p/ImHsFudauN 19:38:45 thanks for the reminder. i need to make sure to read that shortly after the meeting 19:39:03 hopefully we can send that out soon (like today, tomorrow?) 19:39:06 ++ 19:39:19 mordred: you're planning to send that one i take it? 19:39:32 would be nice to merge https://review.openstack.org/#/c/489677 - so that mordred can link to it from his email 19:39:32 fungi: yah - as soon as folks are ok with it 19:39:37 AJaeger: ++ 19:39:58 #action mordred send advance notice of Zuul v3 cutover to mailing lists 19:40:02 and i had a suggestion at the end to link to an infra-manual page where we can have a living document to help devs with the transition 19:40:10 which is what AJaeger just linked 19:40:26 awesome. also on my priority review list 19:41:02 probably that first change is enough for now. it has an introduction, answers some basic questions, and has a skeleton we can fill in later. 19:41:31 so i think once that change lands, it's enough to link to from the announcement email as long as we contextualize it as a work in progress which will be more complete as we approach cutover 19:42:19 it's also 1 of three sets of docs i think we need before we go live: 19:42:47 openstack developer transition guide, zuul documentation, and job documentation 19:43:04 i'm working on getting the last published and cross-linked this week 19:43:20 (and continuing to improve zuul docs) 19:43:29 that's eot from me 19:43:42 excellent. sounds entirely tractable 19:43:49 questions? 19:44:50 crickets? tumbleweeds? 19:44:59 thanks jeblair! 19:45:32 #topic PTG planning (fungi) 19:46:05 #link https://etherpad.openstack.org/p/infra-ptg-queens Infra planning pad for Queens PTG in Denver 19:46:39 seems we've grown a few more ideas there, this is shaping up nicely 19:47:13 i see a number of things that clearly fit into either the helproom timeframe on monday/tuesday or as stuff we can hack on wednesday-friday 19:48:47 i agree with fungi's typing 19:49:04 i've tried to tag them just now 19:49:10 I see the tags 19:49:18 what do we need to do to schedule that zuulv3 session thingy? 19:49:21 [reserve] is for stuff where we should reserve a slot in one of the cross-project rooms 19:49:52 there will be an ethercalc page 19:49:58 already is 19:50:04 when do we do it? 19:50:46 #link https://ethercalc.openstack.org/Pike-PTG-Discussion-Rooms Pike PTG - Extra Discussion Rooms schedule 19:51:00 i think just pick times and stick stuff there 19:51:06 first come, first served 19:51:42 30-minute slots, and the instructions say don't do more than two consecutive 19:51:55 so we can book hour-long double slots if we need 19:52:22 monday is locked 19:53:02 i wonder if there's a reason for that 19:53:15 diablo_rojo or ttx might know, if either are around 19:53:52 i assume we can unlock it, but yeah, it may be best to assume that's on purpose and ask first. :) 19:54:11 assuming it's possible, something like macon, monday, 2pm? 19:54:29 sounds reasonable 19:54:39 that's early enough in the week but after lunch 19:54:52 yeah, i assume we'll be very busy before lunch. 19:55:01 that's my expectation as well 19:55:03 maybe still after lunch too, but gotta do it sometime. :) 19:55:04 as will everyone else 19:55:35 we could try to do 1:30-2:30 in macon but it'll likely be a slow start with everyone wrapping up lunch 19:55:54 and oh yeah, should we try to keep it to 30m? 19:56:12 alternative would be to overlap neutron/ironic or service catalog discussions 19:56:20 maybe increase the likelihood that people will actually show up if we keep it brief 19:56:46 could also do 4:30-5:30 but the first half of that overlaps a watcher discussion and the second half people will likely be scouting nearby bars alerady 19:56:50 already 19:57:18 if you think half an hour is a good target for the session, i'm down with that plan 19:57:44 it will be rushed, but i think that will be good. i can literally talk for hours about this stuff. 19:57:58 no one wants that 19:58:06 and not expecting more than ~50 attendees? 19:58:37 would be useful to have more. think it'll happen? 19:58:52 also plan to expect neutron/ironic people who may be disinterested in the topic wandering into macon before 2:30 19:58:59 my ptg room on debugging the gate was packed the first day, almost empty the second 19:59:10 so I expect yuo'll get quite a few people to show up if you do it once :) 19:59:34 okay, we're about out of time 19:59:39 i'll try to get something scheduled 20:00:09 thanks jeblair! i'll see if diablo_rojo or ttx can enlighten us on any reason for the locked monday cells 20:00:22 thanks everyone for a productive meeting! 20:00:29 #endmeeting