19:02:16 <fungi> #startmeeting infra
19:02:17 <openstack> Meeting started Tue Aug 30 19:02:16 2016 UTC and is due to finish in 60 minutes.  The chair is fungi. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:02:18 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:02:20 <openstack> The meeting name has been set to 'infra'
19:02:27 <fungi> #link https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting#Agenda_for_next_meeting
19:02:34 <fungi> #topic Announcements
19:02:34 <bkero> o/
19:02:42 <fungi> #info Track chairs have confirmed there will be a Zuul v3 presentation at the Ocata Summit
19:02:51 <fungi> jeblair: did you have any additional details on that?
19:02:57 <Shrews> o/
19:03:02 <rcarrillocruz> o/
19:03:28 <jeblair> fungi: no, that's about it.  i mean, i may make a sign-up table for ansi artists.... ;)
19:03:46 * mordred thinks someone should storm the room dressed as the staypufft marshmellow man
19:03:48 <fungi> heh
19:04:07 <pleia2> #link https://www.openstack.org/summit/barcelona-2016/summit-schedule/events/16615/zuul-v3-openstack-and-ansible-native-cicd
19:04:18 <fungi> perfect--thanks pleia2!
19:04:23 <jeblair> mordred: do we have anyone with experience dressing up in ridiculous costumes at summits?
19:04:26 <fungi> i was just trying to hunt that down
19:04:33 <pleia2> I can make stick puppets
19:04:49 <fungi> #info Gerrit downtime on Friday 2016-09-02 at 18:00 UTC
19:04:55 <fungi> #link http://lists.openstack.org/pipermail/openstack-dev/2016-August/102281.html
19:05:16 <fungi> #info Reminder: late-cycle joint Infra/QA get together to be held September 19-21 (CW38) in at SAP offices in Walldorf, DE
19:05:20 <fungi> #link https://wiki.openstack.org/wiki/Sprints/QAInfraNewtonSprint
19:05:22 <fungi> there are still 5 slots remaining, but if you're going you should put your name in the registration table on that wiki page soon lest you end up on the standby waiting list
19:05:43 <fungi> as always, get me announcements before next week's meeting and i'll include them
19:05:53 <fungi> #topic Actions from last meeting
19:06:03 <fungi> #link http://eavesdrop.openstack.org/meetings/infra/2016/infra.2016-08-23-19.01.html
19:06:08 <fungi> AJaeger start a docs ml thread to refresh constraints around retention of old/unmaintained documents
19:06:10 <fungi> #link http://lists.openstack.org/pipermail/openstack-docs/2016-August/009003.html
19:06:13 <fungi> that happened
19:06:15 <fungi> thanks!
19:06:23 <fungi> pleia2 send maintenance announcement for upcoming gerrit project renames
19:06:30 <fungi> (already linked in the announcements earlier)
19:07:17 <fungi> i may add an ad-hoc topic later if there's time to discuss project rename logistics for friday
19:07:27 <fungi> #topic Specs approval: PROPOSED "Update doc publishing spec for AFS"
19:07:33 <fungi> #link https://review.openstack.org/276482 Update doc publishing spec for AFS
19:07:35 <ianw> o/
19:07:43 <jeblair> me puts down sandwich
19:07:48 <fungi> looks like we're set to finish voting on this one!
19:08:11 <jeblair> yeah, i think we have what we need
19:08:27 <jeblair> i updated it yesterday based on feedback from loquacities that came out of the docs ml thread
19:08:37 <fungi> good with approving it at/after 19:00 utc thursday if no objections are voiced?
19:08:46 <pabelanger> o/
19:08:55 <jeblair> the good news is that not needing to do the migration makes things simpler, and gets rid of a significant work item
19:08:58 <jeblair> fungi: yes
19:09:01 <fungi> #info Council voting will remain open on the "Update doc publishing spec for AFS" change until 19:00 UTC on Thursday, September 1
19:09:08 <jeblair> this could use some volunteers
19:09:13 <jeblair> mordred: has volunteered
19:09:21 <mordred> yah. I will help
19:09:33 <jeblair> we could probably use another infra person, and probably a docs team member as well
19:09:42 <fungi> additional volunteers for some of the tasks would be good
19:09:49 <jeblair> it will be highly parallelizable
19:10:13 <jeblair> in fact, almost all of the work items are paralellizable
19:10:16 <pabelanger> sure, I can help
19:10:23 <jeblair> so if people wanted to pile on this, we could probably get it done very quickly
19:10:30 <fungi> AJaeger was already starting to dig through the existing content and trying to mirror a copy to see what's involved. i don't know if he has any new insights yet
19:10:44 <AJaeger> fungi: not yet - still mirroring content ;(
19:10:50 <jeblair> (and there is some urgency around this since cloud sites is being spun out, and apparently we use too many inodes)
19:10:51 <fungi> oh, wow
19:11:01 <mordred> we are big inode consumers
19:11:07 <pleia2> hehe
19:11:24 <pleia2> we break a lot of clouds
19:11:35 <fungi> AJaeger: still downloading after how long? how much do you have so far?
19:11:40 <AJaeger> Fastest way to safe inodes: Don't push up sources - and don't separate trees for each tags
19:12:12 <AJaeger> fungi, still downloading after 36 hours. I had to break after 24 hours ;( and restarted. Luckly lftp can mirror, so it's not starting from zero...
19:12:30 <pleia2> oy
19:12:32 <AJaeger> at 594184 files and 9.8 GB now...
19:13:03 <jeblair> i wonder if doing that inside of rax would be any faster?
19:13:22 <fungi> yeah, thankfully with the current version of the plan just needing an "old docs site" published somewhere after we get site generation going to the new location, having that take a few days to pull is tractable
19:13:25 <AJaeger> shouldn't be much longer....
19:13:31 <jeblair> fungi: yep
19:13:55 <fungi> anything else we need to cover on that spec right now?
19:13:57 <persia> Sadly, for many-small-files, there is limited speedup for low-latency.
19:13:57 <clarkb> just 2ish more GB
19:14:15 <AJaeger> regarding the inode usage: I have indentifed some really unused content (we even have redirects away from it)
19:14:22 <AJaeger> And biggest offender is _source:
19:14:32 <AJaeger> 705 MB; 156k inodes
19:14:34 <jeblair> fungi: nope.  we have some volunteers, and i will requisition others as needed.  :)
19:14:55 <AJaeger> more in the next days...
19:15:05 <jeblair> fungi: oh, you had an idea of making that a priority effort...
19:15:24 <fungi> jeblair: ahh, yep. maybe we can propose that next week
19:15:33 <jeblair> oh, also, i'll be away next week
19:15:43 <fungi> (or the week after!)
19:16:03 <fungi> #topic Priority Efforts: Zuul v3 (jeblair)
19:16:07 <fungi> #link https://review.openstack.org/360817 Zuul v3: update branches job keyword
19:16:09 <fungi> #link https://review.openstack.org/360819 Zuulv3: drop variable interpolation and add nodesets
19:16:20 <jeblair> me puts sandwich down again
19:16:37 <jeblair> the first change is trivial
19:16:49 <jeblair> so go bikeshed that if you want.  :)
19:16:59 <jeblair> i just didn't want to be committing code that didn't match the spec
19:17:02 <jeblair> the second is interesting
19:17:08 <jeblair> it's a substantive change
19:17:25 <jeblair> oh i thought i updated the agenda
19:17:40 <jeblair> the link should be to https://review.openstack.org/361463
19:17:50 <jeblair> #link https://review.openstack.org/361463 Zuulv3: drop variable interpolation and add nodesets
19:18:06 <jeblair> i think the commit msg explains the rationale
19:18:44 <fungi> aha, yep you had 360819 linked in the agenda, and i didn't double-check it
19:18:49 <fungi> sorry about that
19:19:25 <jeblair> well, i left a forwarding address in gerrit, so if anyone looked at the agenda before now, they will have ended up in the right place anyway :)
19:19:44 <fungi> heh
19:20:17 <jeblair> so i don't know if we want to open formal voting on this, or what.
19:20:32 <fungi> maybe. good that it has a clear example
19:20:48 <fungi> though your editor seems to enjoy injecting hard tabs
19:20:48 <pabelanger> I like the change
19:21:32 <jeblair> yeah, tabs are awesome, but i can remove them in a followup if we want to formally vote on the non-tab content.
19:21:49 <clarkb> reading the commit message makes sense to me
19:22:28 <fungi> i can add 361463 to the list for formal voting. 360817 i think i'll just approve as administrivia
19:22:51 <jeblair> sounds good
19:22:54 <fungi> #info Council voting will remain open on the "Zuulv3: drop variable interpolation and add nodesets" change until 19:00 UTC on Thursday, September 1
19:23:31 * mordred steals jeblair's sandwich before he can pick it up again
19:23:41 <jeblair> too late
19:23:48 <Shrews> lgtm
19:23:53 <Shrews> the review, not the sandwich
19:23:54 <fungi> hurry and take a bite
19:23:57 <fungi> #topic Priority Efforts: Nodepool: Use Zookeeper for Workers (jeblair)
19:24:00 <fungi> #link https://review.openstack.org/360821 Update nodepool zookeeper spec with branch
19:24:09 <jeblair> me puts down sandwich for the last time
19:24:22 <jeblair> this one is probably administrivia too
19:24:36 <jeblair> i think we discussed this a week or 2 ago, and started using the branch since then
19:24:37 <fungi> yep, just looked, i'm cool with approving now
19:24:43 <jeblair> so thought we ought to update the spec for it
19:25:18 <fungi> thanks
19:25:42 <fungi> #topic Design summit session planning (fungi)
19:25:47 <fungi> #link https://etherpad.openstack.org/p/infra-ocata-summit-planning Infra Ocata summit planning pad
19:26:29 <fungi> just following up quickly... i put in a preliminary request for one fishbowl and five workrooms plus the half-day collaboration space on friday
19:26:39 <Zara> \o/
19:26:57 <fungi> if anyone disagrees, i still have until tomorrow to amend our allocation request
19:27:50 <fungi> also, are most people sticking around for friday? if so, it might make sense to request some of our workrooms be on friday morning so we can treat it like additional general collaboration time
19:28:27 <rcarrillocruz> i do plan to fly back on Saturday, so yeah
19:28:39 <bkero> me too
19:28:39 <mordred> I will be flying on saturday
19:28:42 <pleia2> same
19:28:48 <clarkb> saturday too
19:28:49 <pabelanger> I too
19:28:53 <fungi> cool, that was the hope
19:29:08 <jeblair> me too
19:29:21 <fungi> i'll let you all know what we end up with, but in the meantime keep adding session ideas to the etherpad
19:29:28 <mordred> fungi: not related to that - I will only be in germany monday and tuesday, flying to stockholm from frankfurt at noon on wednesday, in case anyone cares about such information
19:30:09 <fungi> mordred: good to know. as previously discussed, i won't be in germany but i have hopes someone will feed me highlights
19:30:24 * pleia2 takes all the notes
19:30:33 <fungi> pleia2 rocks
19:30:48 <fungi> #topic Sprint to test Gerrit online indexing (zaro)
19:31:12 <zaro> i'm looking for help to test gerrit's online reindex.
19:31:25 <zaro> using review-dev.o.o
19:31:26 <fungi> zaro: my current plan is to try it in production during our project rename maintenance in a few days, but any testing you get done before then would be awesome
19:31:53 <zaro> fungi: ok.  if you want to skip then cool with me
19:32:16 <zaro> worst case is you'll just need to do offline reindex
19:32:28 <pleia2> yeah, we scheduled a 4 hour window just in case
19:32:30 <fungi> well, if you had anything you were going to try on review-dev in the next few days, feedback from that would be awesome
19:32:49 <fungi> just so we have a better idea of what to anticipate
19:33:06 <zaro> well, i'm suggesting that i run online reindex and need volunteers to use gerrit while it's reindexing
19:33:14 <zaro> that's basically the test
19:33:35 <zaro> the reindex takes about 1.5 hrs without any activity on gerrit
19:33:52 <fungi> yep. anyone have time to poke at uploading changes to and reviewing them on review-dev while there's an online reindex of a large repo happening there?
19:34:17 <pleia2> sure
19:34:41 <fungi> zaro: when do you expect to be around to kick off the reindexing and keep an eye on it?
19:34:55 <pleia2> I'm around all week, so whenever works
19:35:05 <zaro> how about tomorrow morning?
19:35:09 <pleia2> wfm
19:35:34 <fungi> also, does gerrit show-caches or show-queues or something display the reindexing progress?
19:35:55 <fungi> curious how best to keep tabs on it during our maintenance window on friday
19:35:56 <zaro> yes
19:36:03 <fungi> okay, cool
19:36:11 <zaro> show-queues
19:36:26 <fungi> thanks!
19:36:42 <zaro> how about start at 1700 utc?
19:37:03 <pleia2> sounds good
19:37:08 <fungi> i'll get to pleia2's topic on the agenda first, and then i'll inject mine about the gerrit rename maintenance logistics and we can talk a little more about reindexing as part of that
19:37:48 <zaro> cool, was going to suggest a sprint, but probably not needed.
19:37:59 <fungi> #info Gerrit online reindex testing will be run on review-dev Wednesday, August 31 starting at 17:00 UTC
19:38:11 <pleia2> zaro: maybe we can recruit some folks as we go ;)
19:38:23 <fungi> sounds good
19:38:30 <fungi> #topic Another bug day? (pleia2)
19:38:38 <pleia2> so, the last time we did one of these was back in April
19:38:44 <fungi> i agree, seems like we're overdue
19:38:45 <pleia2> bugs are piling up again
19:38:56 <pleia2> I think it would be nice to get this out of the way before our sprint in germany
19:39:14 <pleia2> next week is bad for me because there's a Jury Summons on my desk, my schedule is unpredicable
19:39:18 <pleia2> but any time the week after
19:39:43 <clarkb> what about monday of thr week after?
19:39:51 <clarkb> next monday is a us holiday
19:39:51 <pleia2> wfm
19:40:02 <pleia2> september 12th
19:40:47 <fungi> yeah, that seems fine for me
19:41:08 <fungi> i have nothing on my calendar except a weekly vmt thing
19:41:09 <pabelanger> I think that works for me too
19:41:38 <pleia2> ok, I'll get that organized w/ etherpad and add to the sprint channel schedule
19:42:00 <fungi> #info Infra bug sprint tentatively scheduled for Monday, September 12
19:42:28 <fungi> action pleia2 set up sprint booking for infra bug day
19:42:33 <fungi> thanks!
19:42:37 <fungi> #action pleia2 set up sprint booking for infra bug day
19:42:44 <fungi> helps if i # those
19:43:15 <fungi> anything else we should cover now for bug day planning?
19:43:26 <pleia2> I don't think so
19:44:04 <fungi> #topic Gerrit rename maintenance logistics (fungi)
19:44:30 <fungi> i got to going back over the change that added teh rename playbook
19:44:44 <fungi> #link https://review.openstack.org/175147 Add an ansible playbook for project renames
19:45:06 <fungi> unfortunately i hadn't gotten around to reviewing that before it merged last year
19:45:25 <fungi> it seems we took out the steps for renaming/transferring repos in github but i don't see what in the playbook takes care of that
19:45:41 <fungi> i'm assuming i should add that back into the manual part of the process
19:46:07 <fungi> also it's doing a graceful stop of zuul via kill -URS1 which i fear will take faaaaar too long for us
19:46:20 <fungi> er, kill -USR1
19:46:41 <yuval> (yuval from smaug => karbor here, arriving late :)
19:46:50 <jeblair> yeah, that'd take 24 hours
19:47:04 <fungi> and it'll need updating to remove the offline reindexing step now that we can do online
19:47:32 <fungi> just trying to pick everyone's brain that's done prior renames to see if there's any other gotchas that jump out at you
19:49:34 <fungi> and i'm fuzzy on how any of this actually works
19:49:40 <jeblair> has it ever been used?
19:49:56 <fungi> not in our gerrit at least
19:50:20 <fungi> for example, the shell items that echo db queries through |mysql reviewdb
19:50:22 <jeblair> yolanda is not here, though she decided it was ready to be merged...
19:50:26 <pleia2> and not in prod afaik
19:50:39 <jeblair> mordred: do you have thoughts?
19:51:34 <mordred> jeblair: not useful ones
19:52:10 <fungi> it looks like `echo ... |mysql reviewdb` is relying on ~root/.my.cnf being set up correctly, so that's probably fine
19:52:27 <fungi> as i too rely on that going to our trove instance correctly
19:52:53 <fungi> i'll doublecheck that storyboard.o.o has a similarly correct ~root/.my.cnf
19:53:51 <fungi> anyway, i'm still game for giving this a shot, but would appreciate people getting up with me with any concerns over it before friday so i can revert to our old mostly-manual process if necessary
19:54:11 <jeblair> well, i certainly wouldn't stop zuul that way
19:54:39 <clarkb> its looking likr I may be sitting with at least one foundation staff member to go over using gerrit during the renames so I will defer to others
19:54:52 <fungi> right. things i already know need to be changed there: zuul stop/start/reenqueue, offline reindex. also adding teh github steps back into the manual steps
19:54:59 <clarkb> (we can use review dev while review is down)
19:55:10 <pleia2> I'll have a look through too to see if anything jumps out at me
19:55:30 <fungi> appreciated
19:56:27 <zaro> clarkb: hopefully it will only be down for a little bit
19:57:04 <fungi> #topic Open discussion
19:57:35 <yuval> hey, yuval from smaug/karbor - is anything required on our part during/after the rename?
19:58:06 <fungi> yuval: we'll generate a change you'll want to approve that updates your .gitreview file
19:58:33 <fungi> yuval: but also you may find that some of your jobs are broken by the new repo name and, if so, you'll probably want to be around to patch those as needed
19:59:20 <AJaeger> fungi, the changes look complete - but there're always surprises ;)
19:59:26 <fungi> that's one of the reasons we usually do these right before the weekend, so you have a little time to get tests updated before they'll impact a large number of new changes in review
19:59:55 <fungi> AJaeger: yeah, it's often going to be things like repo urls hard-coded in devstack plugins or something
19:59:59 <fungi> okay, we're out of time
20:00:02 <fungi> thanks everyone!
20:00:06 <fungi> #endmeeting