19:01:50 #startmeeting infra 19:01:51 Meeting started Tue Oct 10 19:01:50 2017 UTC and is due to finish in 60 minutes. The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot. 19:01:53 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 19:01:55 The meeting name has been set to 'infra' 19:02:06 #link https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting#Agenda_for_next_meeting 19:02:07 o/ 19:02:23 #topic Announcements 19:02:25 ooh, meetins 19:02:41 we'll talk about this more later but 19:02:44 #link http://lists.openstack.org/pipermail/openstack-dev/2017-October/123337.html Zuulv3 Back to Production 19:02:50 * mordred waves 19:03:03 Zuulv3 will be going back to managing the gate tomorrow 19:03:32 The other item I wanted ot call out is that TC nominations are happening now 19:03:50 They end at 23:45UTC today (whihc is about 5 hours and 40 minutes from now 19:04:35 if you intend on running don't forget to get nominated. Also good luck to those of you already running 19:04:54 #topic Actions from last meeting 19:05:07 #link http://eavesdrop.openstack.org/meetings/infra/2017/infra.2017-10-03-19.01.txt Minutes from last meeting 19:05:28 last meeting was quite a bit more informal. Just wanted to make sure we weren't letting other important items slip while we were preoccupied with zuul 19:05:53 #topic Specs approval 19:06:15 #link https://review.openstack.org/#/c/509257/ mark gerrit upgrade as complete 19:06:35 I plan to merge ^ later today if there aren't any objections. 19:06:57 Other than that I think we've mostly been swamped with zuul related things the last couple weeks which takes us to 19:07:03 #topic Priority Efforts 19:07:11 #topic Zuul v3 19:07:58 as mentioned earlier the plan is to put this back in production tomorrow as our gate keeper 19:08:13 i've been cleaning up: 19:08:17 #link https://etherpad.openstack.org/p/zuulv3-issues 19:08:28 so that we can continue to use it to help firefighting tomorrow 19:08:40 much cleaner! 19:08:48 it has pretty clear sections now -- triage, job issues, zuul issues 19:08:55 also fixed issues 19:09:10 our list of fixed issues from the past week is *REALLY LONG* yay! 19:09:16 so i moved it to 19:09:20 #link https://etherpad.openstack.org/p/zuulv3-fixed-issues 19:09:41 basically, i think it's good to keep fixed issues on the main etherpad for a little while, for communication purposes 19:10:12 but we didn't need all those there, so that's a place they can retire to 19:10:45 i plan to restart zuul with a bunch of fixes soon 19:10:53 i think those are the main things for folks to know? 19:10:53 \o/ 19:11:05 yah. things seem WAY better now 19:11:18 Earlier this morning there seemed to be ocnfusion around what projects should/can be doing nowish to prepare 19:11:26 i'll roll the repartition into the restart too 19:11:29 might be good to send another email nowish explaining the general procss for not infra? 19:13:16 clarkb: agree - I can take a stab at writing that if that's helpful 19:13:23 mordred: that would be great 19:14:33 jeblair: and I guess just need reviewers to keep going through that list of reviews at the top of the etherpad? 19:15:21 clarkb: yeah; looks like a lot of job changes showed up recently 19:15:29 so there's currently more job work than system work 19:16:03 i do expect we were sending some mixed messages on whether zuul -1 votes were worth looking into because of the previously semi-prevalent post_failures 19:16:41 yeah. we also plan to have some more post_failures before tomorrow as well (to help track down a bug) 19:17:04 i think the message should be something like "all zuul -1 errors should at least be triaged at this point" 19:17:50 and i'll admit the backlog for normal check in v3 right now is making it hard for people to know whether the previous failures are still relevant or just stale data for things we've already fixed, because rechecks are taking over a day to process 19:18:12 ya, unfortunately we had to pull a lcoud out of the rotation too so that likely is only going to get worse 19:18:25 it'll get "better" in maybe 1 hour when i drop the queues 19:18:31 yah 19:18:34 then, of course, start to get worse again 19:18:46 until around noon utc tomorrow 19:18:53 then hopefully better again :) 19:19:00 heh, yeah my browser couldn't load the status page yesterday 19:19:06 but at least between now and then, votes will be a little more current 19:19:19 request list has more than doubled since yesterday, so it's not getting better w/o some intervention 19:19:20 ianw: my browser couldn't load the status page days ago, so you must have one heck of a browser 19:19:28 mordred: maybe include a note about the fresh restart in the email? 19:20:19 7340 corvus 20 0 2042744 427196 64988 R 100.0 1.3 3203:13 chromium-browse 19:20:28 that's my dedicated status page browser 19:20:34 (i use ff for real work) 19:20:42 wow, nuts 19:21:00 also, is unusually warm in here 19:21:06 are you actually able to scroll the page that way? 19:21:21 not well 19:21:47 how big is the check queue now? eg is it so much larger than we expect it to be nomrally? 19:21:55 looks like ~1100 19:22:13 and generally we have ~300 changes in status page in total if we ignore merge-check (we aren't using it in v3) 19:22:16 and then there's periodic 19:22:20 plus 467 in periodic 19:22:48 probably tripleo-check too 19:23:02 but ya firefox 57 beta almost manages to keep up fwiw 19:23:14 121 in check-tripleo 19:23:49 jeblair: yah (re fresh restart) 19:23:53 oh, also if i'm interpreting the discussion on #-dev correctly it sounds like the dedicated tripleo pipelines might be able to go away in another week-ish? that will simplify the config somewhat 19:24:04 ooh. that'd be nice 19:24:04 so, we'll manage through the cutover. status page improvements to come after cutover. 19:24:20 might be worth explaining ^ in the update email too 19:24:36 basically status page is in an exceptional state now and not expected to stay that way post cut over 19:24:53 ++ 19:25:13 yeah, i have a feeling lots of people think the status page is entirely unusable given experiences with attempting to render the extreme backlog status data 19:25:31 when in actuality it'll probably be semi-usable under normal peak load 19:27:32 ok, anything else zuul related, especially cut over related? 19:27:55 nak 19:28:17 just... you know... excited! 19:28:26 #topic General Topics 19:28:32 and also going to bed early tonight 19:28:49 I had one item for this, the new contributor portal project needs to be created 19:28:58 Do we want to wait for post cutover to worry about that? 19:29:10 otherwise we'll need to think carefully about running it in both versions of zuul 19:29:19 #link https://review.openstack.org/#/c/509943/ 19:30:16 thingee: if you are around, are you able to wait a couple days while we do the next zuulv3 rollout? 19:30:31 sounded like they're sort of pushing to announce it rsn 19:33:16 Looks like AJaeger has +2'd, I'll need to follow up with thingee and see what timeline they are operating under as Ithink my preference would be to merge the change after the v3 rerollout 19:33:42 but it has configs for both versions so maybe we just merge it now as is? 19:33:52 anyone have concerns with ^ 19:34:15 i don't object 19:34:45 he's done the work to set it up for v2 and v3, so i don't think we have a good reason to hold it up 19:34:51 yeah, let's go for it 19:34:54 ok 19:35:00 I'll give it a review post meeting 19:35:15 #topic Project Renames 19:35:25 I think we are actually ready to think about doing these again :) 19:35:34 wow, it's been a while! 19:35:44 #link https://review.openstack.org/#/c/500768 19:35:50 we have at least one rename in the pipeline 19:36:35 This week is probably bad as is week before summit travel, but there is one week between, 16th-20th where we could potentially do some renames 19:37:30 do we want to provisionally plan for the 20th, then commit to it in a week at our next meeting if we've caught up with everything else? 19:37:41 i don't have any atypical obligations that week, so count me in 19:38:27 i usually try to catch the release team meeting 15:00-16:00 on fridays, but otherwise nothing specific planned 19:38:39 I've got a dentist visit and a user group talk that week on the 17th and 19th but should be able to help out on the 20th 19:39:35 ok lets aim for the 20th and actually commit to it next meeting if everything else is calmer :) 19:39:51 say at 1600UTC ish 19:39:52 wfm 19:40:33 #info plan for project renames around 1600UTC October 20 assuming we have cauhgt up on other items by then. Decide if we are ready for that in the next meeting 19:40:41 #topic General Discussion 19:40:55 #undo 19:40:56 Removing item from minutes: #topic General Discussion 19:41:04 #topic Open Discussion 19:41:08 fungi: you mentioned last time wanting a hand with writing a project update? 19:42:34 Was that the summit project update? 19:42:38 ianw: well, wanting some volunteers to help the ptl do that anyway (which i'm also willing to help with) 19:42:57 ya, I think my rough plan for that was to dig in after the zuul re roll out 19:42:58 how does this: https://etherpad.openstack.org/p/OCPH0jEDn9 look for email about what people shold do? 19:43:00 and possibly also appearing on stage with said ptl 19:43:09 as I think that will largely affect the content :) 19:43:23 if you are interested in helping with that let me know and we can start drafting it up together 19:43:38 (I think fungi and paul have already expressed interestg in helping, but hte more the merryier) 19:44:02 ok, etherpad? 19:44:16 ianw: ya I think we'll start on an etherpad 19:44:38 #link https://etherpad.openstack.org/infra-sydney-summit-project-update <- how about that one 19:45:14 mordred: lgtm, added one note under "A few notes about tomorrow" 19:45:34 clarkb: excellent, open in a tab right next to zuulv3-issues :) 19:48:00 Oh I've also signed us up for an onboarding session that we will share with tonyb in sydney 19:48:12 mordred: lgtm 19:48:25 I'll likely be working off of what fungi put together for Boston in that session. If you want to help with that too let me know 19:48:57 mordred: might be good to add something about please talk to us 19:49:16 mordred: I think the projects that we would consider to be successful in all the transition stuff have been willing to work and talk with us directly 19:49:42 clarkb: good point - adding 19:51:35 * clarkb gives it a couple more minutes for any last items but then everyone can go find $meal early :) 19:52:47 Review request: https://review.openstack.org/#/c/509670/ openstack-infra/os-loganalyze Support the log level "CRITICAL" 19:53:23 As it is now. If you select ERROR, then you will not see CRITICAL messages. 19:53:25 clarkb: how's that? 19:53:37 Or if you basically select any log level you won't see CRITICAL messages. 19:54:03 mordred: I like that, really hte important part is communicating to help make sure we don't miss important stuff 19:54:03 jlvillal: oh wow - when did CRITICAL get added? 19:54:13 mordred: it is apparnetly part of python logging 19:54:25 mordred: Uh, not sure. It is part of the python logging standard. 19:55:23 ok, thanks everyone. You canfind us in #openstack-infra or on the infra mailing list. Please do let us know if you have questions about the v3 rollout or see weird unexpected behavior 19:55:26 and with that 19:55:28 #endmeeting