19:01:21 <fungi> #startmeeting infra
19:01:21 <openstack> Meeting started Tue May 17 19:01:21 2016 UTC and is due to finish in 60 minutes.  The chair is fungi. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:01:22 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:01:24 <openstack> The meeting name has been set to 'infra'
19:01:28 <fungi> #link https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting#Agenda_for_next_meeting
19:01:38 <fungi> #topic Announcements
19:01:45 <fungi> none for this week
19:01:51 <fungi> #topic Actions from last meeting
19:02:02 <jeblair> not yet, sorry
19:02:21 <fungi> #link http://eavesdrop.openstack.org/meetings/infra/2016/infra.2016-05-10-19.03.html
19:02:23 <fungi> heh
19:02:32 <fungi> #action jeblair investigate mail filtering by recipient options for infra-root@o.o inbox
19:02:40 <fungi> #topic Specs approval
19:02:50 <fungi> none new this week
19:03:01 <fungi> we still need to flesh out the wip task tracking spec
19:03:16 <fungi> i was going to pick it up, but got busy with other stuff
19:03:37 <SotK> o/
19:03:51 <fungi> i did move the translation improvements spec to implemented last week
19:04:00 <jeblair> ++
19:04:05 <pleia2> great
19:04:16 <fungi> #topic Gerrit 2.11 disk utilization and Revisit Gerrit git gc (fungi, zaro)
19:04:25 <fungi> #link http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=2839&rra_id=all
19:04:50 <fungi> a couple weeks ago i ended up growing the backing volume for the gerrit homedir on review.o.o
19:05:15 <fungi> since right around the 2.11 upgrade, disk utilization inside the git tree there has been experiencing unbounded growth
19:05:32 <fungi> and anteaya noticed that we were almost out of disk
19:05:38 <jeblair> that is... substantial.
19:05:54 <fungi> so i doubled it and we've gained some breathing room for the other half of this topic
19:05:56 <pleia2> we've also been adding about one project a day, do we know whether it's gerrit or project growth? or both?
19:06:15 <fungi> no, like the nova repo was some 10x larger than in january
19:06:23 <pleia2> wow
19:06:27 <fungi> #link https://www.mail-archive.com/openstack-infra@lists.openstack.org/msg04264.html
19:06:47 <fungi> zaro: has done some more experimentation with the gerrit background git gc task
19:07:02 <fungi> and it looks like he wanted to discuss it in the meeting
19:07:29 <Shrews> o/
19:07:36 <zaro> yup.  so i think there was some lingering concern since last testing i did
19:08:06 <zaro> i've now tested it on review-dev.o.o running gc on the nova repo and nothing bad happend to it.
19:08:28 <zaro> also put it up there in case anybody else wanted to test other stuff on it
19:09:04 <zaro> wanted to see if poeple still have further questions?
19:09:09 <jeblair> are we sure we are running the existing repack cron?
19:09:32 * fungi checks syslog on review.o.o
19:10:00 <jeblair> it is at least starting...
19:10:08 <jeblair> May 15 04:07:01 review CRON[42940]: (gerrit2) CMD (find /home/gerrit2/review_site/git/ -type d -name "*.git" -print -exec git --git-dir="{}" repack -afd \;)
19:10:36 <fungi> yeah
19:10:50 <jeblair> no idea if it's finishing
19:10:53 <fungi> i suppose i could try manually running the same as the gerrit2 user and seeing what happens
19:11:09 <fungi> any objection to me starting it now?
19:11:24 <clarkb> not here
19:11:27 <fungi> no idea how long it runs
19:11:55 <jeblair> zaro: the only thing i haven't seen is how this impacts the git servers (which i think was one of the things i mentioned in my original email?)
19:12:40 <zaro> i guess i wasn't aware of that.
19:13:06 <jeblair> zaro: http://paste.openstack.org/show/497409/
19:13:11 <jeblair> zaro: that paragraph from the email
19:14:00 <fungi> i've got a screen session as gerrit2 on review.o.o with that command copied from crontab -l running, if another infra-root needs to attach: sudo su - gerrit2; script /dev/null; screen -x
19:15:09 <zaro> jeblair: are you referring to the cgit servers?
19:15:18 <nibalizer> o/ (only sorta)
19:15:20 <zaro> git.o.o?
19:15:45 <fungi> zaro: yes, those
19:15:47 <jeblair> zaro: yes.  that is the principal thing i'm worried about.
19:15:56 <jeblair> i'm sure that gerrit gc will make gerrit faster
19:16:03 <jeblair> i'm not sure what will happen to the git servers
19:16:14 <zaro> so doesn't gerrit replicate to those?
19:16:18 <clarkb> I thought that was tested and hashar? provided independent confirmation
19:16:35 <jeblair> zaro: it does
19:16:39 <jeblair> clarkb: i was unaware of that
19:17:05 <clarkb> ya regular git used less memory for them after they started GCing iirc
19:17:30 <zaro> would it be possible to test with some other repo on review.o.o?
19:17:31 <jeblair> and what impact did it have on network traffic and the resulting size of repos that were cloned from a git server?
19:18:02 <jeblair> i think this is easy to test.  just get a repo, gc it, and serve it with cgit and the apache smart git protocol
19:18:05 <clarkb> I don't recall specifics on network traffic and size of cloned repos
19:18:16 <jeblair> this is a one way trip for us
19:18:50 <jeblair> so before we embark on it, i think it would be good to know how it's going to alter one of the more performance-critical pieces of our infrastructure
19:19:04 <jeblair> however, if other folks don't share that concern, we can yolo it.
19:19:50 <fungi> zaro: jeblair: we could test what happens with replication to the local mirror on review-dev?
19:19:51 <zaro> jeblair: are you suggesting we setup a seperate cgit server for this test?
19:19:54 <fungi> i assume we have one configured
19:20:06 <clarkb> fungi: we do
19:20:31 <jeblair> zaro: the testing can be done locally
19:20:33 <zaro> i thought review-dev was only replicating to github?
19:20:40 <clarkb> zaro: it has a local replica too
19:20:47 <clarkb> the stuff served out of /p/foo/bar
19:21:19 <jeblair> git.o.o has lots of protocols
19:21:37 <jeblair> git smart http via apache, git native, cgit
19:21:57 <jeblair> (also, even occasionally git dumb via http)
19:21:58 <clarkb> the first two are trivial to test on review-dev, the last would require a cgit install
19:22:06 <jeblair> all of them may behave differently
19:22:25 <fungi> yeah, and cgit install means probably separate centos server
19:22:47 <jeblair> i'm not trying to slow this down
19:22:49 <zaro> so nova-gc repo is already on review-dev.o.o so we can just look at the local replication?
19:23:00 <jeblair> i seriously think this does not require any special access
19:23:03 <jeblair> just do it locally
19:23:12 <zaro> not sure why we need a cgit install now.  thought you said we only need local test?
19:23:24 <clarkb> zaro: because cgit serves git repos
19:23:35 <jeblair> yes, install cgit on a local server
19:23:39 <jeblair> or a cloud server somewhere
19:23:46 <clarkb> so the test is to ensure we don't negative impact the memory/cpu/network/disk of cgit when we switch
19:23:49 <jeblair> don't worry about review-dev or setting up some kind of new public cgit server
19:25:31 <cody-somerville> \o_
19:25:59 <fungi> private gerrit vm replicating to private cgit vm, test performance of cloning and fetching from the cgit vm via http/https/git protocols before and after turning on gerrit background git gc?
19:26:19 <jeblair> fungi: sure.  actual gerrit optional.  :)
19:26:40 <jeblair> (since we could forego using gerrit gc and just use git gc to give us more control)
19:27:22 <jeblair> just, whichever of those zaro wants to actually do :)
19:27:26 <fungi> btw, faux-cron repack still going on review.o.o in my screen session
19:27:33 <fungi> hasn't bombed yet anyway
19:28:09 <zaro> i guess it depends whether there's value in having a cgit test server somewhere?
19:28:21 <zaro> and having gerrit replicate to it?
19:28:38 <zaro> or is this something we only do one time?
19:29:07 <jeblair> this is a one time thing
19:29:43 <zaro> ok. then i'll try the less work route
19:29:45 <jeblair> that's the reason i put the tarball of the git repo up -- because none of this requires access to production servers
19:30:19 <hashar> o/
19:31:48 <zaro> ok, so i guess we prefer to test cgit before doing? is that the decision?
19:32:44 <fungi> hashar: clarkb mentioned your gc'd git replica performance testing back around 19:16 utc in the meeting log. do you have a link to that analysis handy?
19:32:54 <jeblair> i don't mind either way (though if there are surprises, i won't be fixing them)
19:33:32 <hashar> clarkb: fungi: I might have wrote it in the infra mailing list directly
19:33:45 <fungi> hashar: do you recall roughly when that was?
19:33:58 <hashar> nop but can look it up
19:34:09 <fungi> if you can narrow it down to a particular month, i can dig up a ml archive url for it easily
19:34:10 <hashar> I was replying to some email from Zaro for sure
19:34:46 <zaro> hashar: sorry i don't remember :(
19:36:54 <zaro> maybe we should continue and round back?
19:37:31 <fungi> yeah, let's proceed to teh next (also gerrit-related) topic
19:37:44 <hashar> yup I will post to the mailling list if I find something
19:37:45 <fungi> it's "gerrit week" here in infra
19:37:52 <fungi> thanks hashar! thanks zaro!
19:38:04 <fungi> #topic Gerrit project renames using the Gerrit import/delete plugin (zaro)
19:38:14 <fungi> #link https://gerrit.googlesource.com/plugins/importer/+/stable-2.11/src/main/resources/Documentation/about.md
19:38:36 <zaro> so i've tested what's described there to rename projects.  it's real slick and easy
19:38:44 <fungi> zaro: so if i understand the proposal, it's to use that plugin to "import" a gerrit project to a new name and then delete the old one?
19:38:49 <zaro> and no restart or rindex required
19:39:05 <zaro> fungi: yup, that's really all there is too it.
19:39:12 <fungi> zaro: reading that, it seems to import from a second gerrit. does importing from itself also work?
19:39:15 <jeblair> neato!
19:39:20 <clarkb> https://gerrit.googlesource.com/plugins/importer/+/stable-2.11/src/main/resources/Documentation/about.md#Project-Rename is the section for renames
19:39:21 <zaro> yes.
19:39:26 <fungi> aha
19:39:34 <fungi> i skimmed poorly!
19:39:49 <zaro> you can copy, and even move it to a subproject
19:40:08 <zaro> or i mean across subprojects
19:40:09 <fungi> and it keeps open changes, subscriptions, et cetera?
19:40:16 <zaro> yes
19:40:25 <fungi> starred changes carry over too?
19:40:37 <zaro> hmm, i didn't check that.  but can later
19:40:47 <fungi> do they end up with new change urls? new change-ids?
19:41:07 <fungi> just thinking through the things we normally copy when updating the db
19:41:17 <clarkb> fungi: ya I would worry about that, not sure how you can preserve the change numbers between two servers
19:41:25 <clarkb> seems like collisions would be a big issue
19:41:45 <zaro> for the changes. no, it keeps the same url, change id and commit id
19:41:55 <fungi> i have a feeling the long change-ids remain the same at least
19:42:05 <fungi> but i'm surprised it could have the same change number
19:42:30 <zaro> sorry, actually i think change ids do change.
19:42:35 <fungi> i thought that was a unique incremented integer column in the table
19:42:37 <jeblair> especially since, at some point during the process, the project exists in two places...
19:43:04 <jeblair> so this would break all of our urls
19:43:17 <zaro> opps, i mean change number changes, but change id stays the same
19:44:08 <fungi> ti definitely sounds worthy of further investigation. i don't mind there being some caveats to a rename since that may help discourage them further, but the compromises will need to be weighed against the benefits (no downtime and no offline reindex are biggies)
19:44:16 <zaro> here's a project copy in progress: https://review-dev.openstack.org/#/x/importer/list
19:45:20 <fungi> i wonder if the plugin could be extended to have a proper "rename" mode which encapsulates the copy and delete but preserves id numbers and anything else which would otherwise be awkward for an import/copy
19:46:54 <fungi> so anyway, it's an interesting proposal for sure. there's some serious merit if it gets us free of having maintenance windows to rename projects
19:47:12 <fungi> more research needed for now though
19:47:24 <fungi> #topic Project rename scheduling (fungi)
19:47:28 <fungi> and on that note...
19:47:56 <fungi> when do we want to rename openstack/openstack-ansible-ironic to openstack/openstack-ansible-os_ironic, and possibly also openstack-infra/ansible-puppet to openstack-infra/ansible-role-puppet
19:48:16 <fungi> mordred added the second one, but hasn't written the change to actually implement it
19:48:45 <fungi> i can tell everyone is excited at the prospect of yet another gerrit maintenance window
19:49:18 <prometheanfire> woooo
19:49:45 <fungi> i guess let's not shoot for "May 23-27" since that's our server upgrades sprint week
19:50:15 <pleia2> memorial day in the states is the following week, but maybe later in that week?
19:50:45 <fungi> clarkb and i are also not around for much of the week of june 5
19:51:15 <pleia2> how about the 3rd?
19:51:34 <fungi> i could see doing it on friday the 3rd of may
19:51:41 <clarkb> I won't be around the 3rd, but you don't need me
19:51:45 <pleia2> june ;)
19:51:49 <fungi> er
19:51:51 <fungi> right
19:51:53 <fungi> months
19:51:58 <fungi> i could see doing it on friday the 3rd of JUNE
19:52:06 <pleia2> I'll be around
19:52:18 <fungi> yeah, i don't have any plans that day
19:52:35 <fungi> if there's at least two infra-root volunteers, it's probably fine?
19:52:46 <fungi> #link http://releases.openstack.org/newton/schedule.html release schedule
19:52:47 <pleia2> I'll be on east coast time, so earlier is better
19:53:04 <fungi> that's the end of n1 milestone week
19:53:11 <pleia2> ah, hm
19:53:25 <fungi> which is probably fine? earlier that week less so of course
19:53:33 <pleia2> yeah
19:54:17 <clarkb> I have family arriving the econd and traveling the 4th so the 3rd is likely family day
19:54:33 <fungi> dhellmann: any concerns with an extended gerrit maintenance window friday the 3rd of june, the last day of n1 milestone week?
19:54:52 <dhellmann> fungi : when would you want to start?
19:55:21 <dhellmann> it probably won't be an issue, as long as we get the milestone tags done before you start
19:55:38 <dhellmann> and it should be possible to do that early in the day EST
19:55:40 <fungi> pleia2 and i are volunteering to work on it, and we'll both be in edt that week, so probably not super late
19:55:54 <pleia2> 2000?
19:56:03 <fungi> 20:00 utc works fine for me
19:56:10 * dhellmann converts to local time
19:56:14 <pleia2> now :)
19:56:19 <fungi> that's 4pm augusta time
19:56:44 <dhellmann> also athens but yeah, that should work fine
19:56:58 <fungi> #agreed Gerrit rename maintenance tentatively scheduled 20:00-24:00 UTC Friday, June 3
19:57:11 <fungi> dhellmann: ahh, one of those a towns. at least i didn't say atlanta ;)
19:57:28 <jeblair> all the "A" towns in georgia are conveniently in the same timezone
19:57:37 <fungi> so desu nee
19:57:38 <prometheanfire> can I bring up something real quick? have a meeting to run to
19:57:48 <fungi> still one topic to go
19:57:51 <dhellmann> jeblair : augusta is actually spelled disgusta, but yeah ;-)
19:57:55 <fungi> #topic Privileged or sanitized log access for openstackid.org (fungi)
19:58:13 <prometheanfire> ok, guess I'll be late
19:58:26 <fungi> we temporarily granted smarcet root access to openstackid.org shortly before the summit so he could review logs
19:58:58 <fungi> just wanted to see if anyone is interested in helping me work out a solution for puppeting privileged log access or alternatively a log sanitization and publishing solution for that server
19:59:25 <fungi> come find me in #openstack-infra to discuss since it's almost tc meeting time now
19:59:33 <fungi> #topic Open discussion
19:59:38 <prometheanfire> thanks
19:59:38 <fungi> prometheanfire: what's the question?
19:59:47 <jeblair> i think priv access is better; i'm not super keen on exposing (even sanitized) authn logs
19:59:49 <prometheanfire> I've been working on getting gentoo into nodepool
20:00:11 <prometheanfire> it was brought up that maybe I should make a spec for it, do I need to?
20:00:15 <clarkb> jeblair: agreed, particularly since they carry potentially personally identifiable infos
20:00:20 <clarkb> jeblair: mapping names and users to IPs
20:00:35 <fungi> prometheanfire: i don't personally see a need to have a spec for that unless there's a lot of review confusion around it
20:00:50 <prometheanfire> I don't think there is
20:00:52 <fungi> also we're out of time. don't want to anger the lovely tc
20:00:59 <fungi> prometheanfire: then i'd just go forth and code
20:01:01 <prometheanfire> just have to touch a lot of projects
20:01:01 <dims_> haha fungi
20:01:05 <fungi> thanks everyone!
20:01:05 <prometheanfire> yep, sgtm
20:01:08 <prometheanfire> cya
20:01:09 <fungi> #endmeeting