19:02:08 <fungi> #startmeeting infra
19:02:09 <openstack> Meeting started Tue Oct 20 19:02:08 2015 UTC and is due to finish in 60 minutes.  The chair is fungi. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:02:11 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:02:13 <openstack> The meeting name has been set to 'infra'
19:02:21 <jhesketh> Morning
19:02:26 <anteaya> stevemar_: yeah, no kidding
19:02:29 <ruagair> o/
19:02:30 <fungi> #topic Announcements
19:02:47 <ianw> o/
19:03:06 <pabelanger> o/
19:03:07 <nibalizer> o/
19:03:08 <zaro> o/
19:03:13 <fungi> i think lots of people are travelling this week, so while i'm happy to hold a meeting, be aware we're short at least a few council members and there's unlikely to be much of importance decided
19:03:30 <fungi> #topic Actions from last meeting
19:03:58 <fungi> oh, our agenda! i almost forgot to link it
19:04:07 <olaph> o/
19:04:10 <fungi> #link https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting
19:04:19 <fungi> no idea where my head is today, clearly not here though
19:04:43 <fungi> and as for the last meeting's minutes...
19:04:49 <fungi> #link http://eavesdrop.openstack.org/meetings/infra/2015/infra.2015-10-13-19.01.html
19:04:59 <fungi> the only action item we had was:
19:05:04 <fungi> zaro send a discussion starter to the openstack-dev mailing list for feedback on Gerrit 2.11
19:05:12 <anteaya> that happened
19:05:13 <mordred> o/
19:05:15 <fungi> #link http://lists.openstack.org/pipermail/openstack-dev/2015-October/076962.html
19:05:23 <zaro> hey, i think we can say that everybody looks forward to 2.11
19:05:37 <fungi> it's been a lively discussion too. some mix of excitement and armchair development of gerrit 2.12 ;)
19:05:47 <anteaya> zaro: I got that feeling
19:06:12 <fungi> anyway, let's not dwell on the action item other than to thank zaro: he's up momentarily with updates on it in the priority efforts portion of the meeting anyway
19:06:19 <fungi> but thanks for starting that zaro!
19:06:25 <jasondotstar> zaro: +1
19:06:30 <fungi> #topic Specs approval
19:06:49 <fungi> #info Approved: StoryBoard Worklists and Boards spec
19:07:01 <Zara> \o/
19:07:03 <fungi> #link http://specs.openstack.org/openstack-infra/infra-specs/specs/storyboard_worklists_boards.html
19:07:28 <fungi> looking forward to talking about storyboard and its possible futures at the summit
19:07:38 <Zara> :)
19:07:56 <fungi> we seem to have no proposed specs for today's agenda, which is just as well since we probably shouldn't vote on more until we summit next week
19:08:24 <fungi> #topic Priority Efforts: Gerrit upgrade
19:08:28 <fungi> #link https://etherpad.openstack.org/p/test-gerrit-2.11
19:08:45 <fungi> zaro's been a busy bee
19:09:11 <fungi> anything we need to know about this before we have our sessions next week, zaro?
19:09:12 <zaro> ahh yes,.
19:09:29 <zaro> gerrit upgrade is a thing again. #link https://review.openstack.org/237701
19:09:51 <zaro> these were the items i had slated for testing:  #link https://etherpad.openstack.org/p/test-gerrit-2.11
19:10:00 <anteaya> I have a point here
19:10:23 <zaro> it's mostly working as expected, just need to test these following things: gerritlib, zuul pipelines, manage projects and migration.
19:10:26 <anteaya> last time we upgraded we needed to give some folks  4 weeks heads up to change ip things in firewalls
19:10:43 <anteaya> do we need that again this time, if so can we give them the ip addresses now?
19:10:52 <fungi> anteaya: that's been when replacing servers (like for an distro upgrade)
19:11:02 <anteaya> fungi: ah okay thanks
19:11:05 <fungi> we don't necessarily need to do a side-by-side upgrade for this one
19:11:09 <zaro> I'm a bit concerned with regards to fiasco last time so i'm wondering if we need to have a rollback script before trying again?
19:11:11 <anteaya> so we don't need to distro upgrade again
19:11:14 <anteaya> okay thanks
19:11:22 <fungi> though that's certainly worth discussing as we get closer to a maintenance
19:11:43 <anteaya> sure I just didn't want to agree to a time and then have to wait another few weeks
19:11:57 <zaro> I also attempted a migration of prod data yesterday.  done on my local laptop but it was way too slow.
19:12:13 <fungi> zaro: a start to a rollback script wouldn't hurt. most of it was really mordred unwinding the db migrations (including some implicit migrations)
19:12:17 <zaro> took all nigth and only got thru ~600 projects out of ~780
19:12:39 <zaro> fungi: i'll look at it but might need some help with that.
19:13:05 <zaro> now i'm planning to try the migration again with a beefy 12cpu VM.
19:13:17 <anteaya> yay, I hope you have better results
19:13:18 <zaro> 16 GB ram
19:13:25 <anteaya> zaro: let's make notes there?
19:13:35 <fungi> zaro: as for a proper migration test, it's reasonable to backup the review-dev db/repos, and then place them with the redacted prod copies and see what sort of timeframe we're looking at on a more representative system
19:13:35 <zaro> where?
19:13:48 <fungi> having a timeframe estimate would be great before jumping into a maintenance
19:13:54 <anteaya> zaro: in an etherpad about what resouces you use for a successful migration
19:14:19 <zaro> fungi: yeah, i was planning to capture time as well
19:14:37 <anteaya> zaro: so if 12cpu and 16 GB ram work
19:14:51 <zaro> i'm guessing that the VM flavor i picked is representivite of what we have for review.o.o
19:15:07 <fungi> zaro: checking now
19:15:10 <zaro> although i plan to use local mysql
19:15:39 <fungi> zaro: we're using 7 vcpus and 30gb ram
19:15:57 <fungi> er, 8 vcpus
19:15:59 <zaro> cool, close enough i guess.
19:16:12 <zaro> though 7 was kina odd
19:16:19 <fungi> it's next to the 8 ;)
19:16:31 <zaro> anyways that's all for now.
19:16:36 <fungi> okay, anything else urgent on this? we have a fairly full agenda
19:16:45 <fungi> thanks zaro!
19:16:57 <fungi> we'll get into more details in person next week too
19:17:04 <zaro> ahh yeah, priority effort topic is 'gerrit-upgrade'
19:17:27 <fungi> #topic Priority Efforts: Maniphest (craige / ruagair)
19:17:38 <fungi> #link https://review.openstack.org/235802
19:18:05 <ruagair> Not much more to add beyond my update in the agenda.
19:18:07 <fungi> what's blocking you on this ruagair? or just a quick status update?
19:18:53 <fungi> so you've got a test instance running, you've gotten the migration script working, snf you're futzing with auth now (cauth?)
19:18:54 <ruagair> Currently bnutting out a Phsbricator auto configuration issue
19:19:07 <fungi> got it
19:19:12 <ruagair> yes.
19:19:37 <ruagair> you can experience it at os01.mcwhirter.io
19:20:10 <ruagair> any eyes or thoughts welcome.
19:20:16 <fungi> #link http://os01.mcwhirter.io/
19:20:32 <fungi> okay, well we've got a task tracking fishbowl slot in tokyo next week if you're going to be there, and if not we'll get some additional details
19:20:47 <fungi> thanks for working on this!
19:20:59 <tchaypo> Was the plan to use a plugin in front of phabricator to redirect to the opened provider and get auth info from it?
19:20:59 <fungi> #topic Stackforge namespace retirement
19:21:00 <zaro> ruagair: not much to experience there
19:21:09 <ruagair> I'm there for those sessions.
19:21:24 <zaro> ruagair: it says auth failure
19:21:31 <fungi> just a reminder that we did complete the stackforge namespace retirement maintenance over the weekend
19:21:40 <fungi> thanks to everyone who was involved!
19:21:42 <ruagair> yes zero :-)
19:21:46 <jasondotstar> fungi++
19:21:58 <fungi> AJaeger has a couple cleanup patches proposed but it's basically behind us now
19:22:08 <tchaypo> Experience the failure (of phabricator, not stackforge)
19:22:14 <fungi> there was one straggler repo which will need to get moved in a future rename maintenance
19:22:38 <fungi> it's on the renames list in the agenda already but needs a project-config change proposed to do the actual renaming
19:22:52 <anteaya> kevinbenton: ^^
19:23:07 <anteaya> I'll work with kevinbenton to get it into shape
19:23:12 <anteaya> and I'm sure AJaeger will too
19:23:33 <fungi> thanks again for a smooth maintenance, everyone
19:23:47 * AJaeger doubts that thre's anything to do after anteaya helped ;)
19:23:48 <fungi> it went surprisingly quickly given the hundreds of repositories involved
19:23:51 <zaro> why the straggler?
19:23:54 <anteaya> AJaeger: ha ha ha
19:24:07 <fungi> zaro: the maintainers of the repo misread the announcements
19:24:16 <anteaya> zaro: I don't even, I"m so...
19:24:29 <anteaya> I even asked in -neutron on the wednesday prior
19:24:36 <anteaya> anyway
19:24:37 <kevinbenton> here is the project-config change https://review.openstack.org/237753
19:24:56 <anteaya> kevinbenton: thank you
19:25:01 <kevinbenton> i was copycatting so i'm not sure if it's correct
19:25:04 <fungi> anything else on the stackforge namespace retirement? anyone aware of anything that got missed or issues which have cropped up since?
19:25:21 <asselin> o/
19:25:30 <zaro> we planning to move openstack-infra projects too?
19:25:46 <anteaya> zaro: oh please no
19:25:49 <zaro> or i mean do we have plan to
19:25:52 <nibalizer> noooo
19:25:55 <anteaya> I have a hard time finding what I want now
19:25:56 <AJaeger> zaro, there's little move from openstack-infra to openstack, so no need IMHO
19:25:57 <fungi> i don't think so at this stage. it would likely be a lot more disruptive to all the other projects if we did
19:26:14 <anteaya> zaro: I have no plan to
19:26:27 <zaro> ok, just wondering
19:26:35 <anteaya> zaro: yup, fair
19:26:37 <fungi> and the stackforge namespace retirement work was driven by a specific technical committee resolution
19:26:51 <fungi> okay, moving on
19:26:58 <fungi> #topic Summit planning
19:27:16 <fungi> as i mentioned on the ml(s), the schedule is up
19:27:26 <fungi> #link http://mitakadesignsummit.sched.org/type/Infrastructure
19:27:52 <fungi> this is down to the wire, but if you happen to spot any _serious_ schedule conflicts i didn't take into account, please speak up asap
19:28:28 <fungi> i'm working on getting the etherpads for them fleshed out and will add them to the schedule and to the etherpads list on the wiki within the next day or two
19:29:12 <fungi> anyone have any burning questions or issues related to our summit planning?
19:29:30 <nibalizer> nope
19:30:01 <zaro> looks good
19:30:13 <fungi> note that the work session names are generic intentionally (for all tracks, not just infra's) to deter random walk-ins
19:30:24 <fungi> so you need to hover or click them to see which ones are which
19:30:57 <anteaya> if you can bother to click, chances are higher you will get something from the session
19:31:19 <anteaya> and have something to contribute
19:31:49 <fungi> also clarkb had some good tips about the print view working well for custom schedules
19:32:15 <anteaya> oh I missed those
19:32:20 <fungi> no idea if he's around at the moment
19:32:24 <anteaya> was it in channel?
19:32:29 <anteaya> I'll find them
19:32:31 <fungi> i don't think he mentioned them in public, no
19:32:45 <fungi> i'm just putting him on the spot if he's around, or at least crediting him
19:33:06 <nibalizer> im proxy for clarkb
19:33:14 <fungi> perfect
19:33:19 <anteaya> nibalizer: what are the tips?
19:33:25 <fungi> anyway, there's a little printer icon up in the top right there
19:33:31 <anteaya> ah
19:33:33 <nibalizer> "if you click print in sched.org, there is a persobal schedule pribtout"
19:33:54 <zaro> nice~
19:33:57 <anteaya> nibalizer: cool, thank you
19:34:01 <anteaya> nibalizer: thanks clarkb
19:34:29 <fungi> i'm going to print mine and bring it with me, because we're not guaranteed there will be paper schedules available at the venue (i know, we ask for it every year, but paper is scarce or something)
19:35:01 <fungi> (in seriousness, the organizers already to an awesome job and i can understand how paper schedules might be a little tough to squeeze in on our short scheduling timeline)
19:35:14 <anteaya> fungi: yes, I can see that
19:35:24 <anteaya> awesome work organizers
19:35:47 <fungi> so anyway, bringing a paper backup is good since we've had sched.org reachability/outage issues or even general conference network issues regularly in the past
19:35:52 * zaro loves paper
19:36:01 <anteaya> zaro: ha ha ha
19:36:15 <anteaya> or if they think we are all ddosing the site
19:36:20 <fungi> it might not be completely up to date if someone tweaks something at the last minute, so prefer the electronic version if you have access, but paper is still handy
19:36:30 <anteaya> ah memories of hong kong
19:36:42 <ruagair> no ical exports?
19:37:00 <fungi> memories of just about every venue until vancouver honestly. that was the first time our networks and schedule access have been really, really solid
19:37:20 <fungi> ruagair: i think the smart phone app might do that? not sure
19:37:28 <anteaya> fungi: yay vancouver
19:37:33 <ruagair> it did in Paris
19:37:56 <fungi> interesting. i'll ask my more foundationy cohorts
19:37:59 <mordred> hey all - back online - what did I miss?
19:38:07 <fungi> mordred: just the meeting ;)
19:38:19 <fungi> your requested topic is up next actually. i put it at the end
19:38:42 <mordred> yay!
19:38:52 <anteaya> mordred: oh and zaro something something gerrit db migration rollback plan
19:39:04 <mordred> yes. we should hav eone of those
19:39:12 <anteaya> mordred: he'd like your help
19:39:15 <mordred> because we had to come up with one suddenly last time
19:39:19 <mordred> I would like to help
19:39:22 <anteaya> yay
19:39:34 <fungi> cool. okay, i think we're as prepped on the summit scheduling topic as can be, so i'll move on to the last item on the agenda
19:39:51 <fungi> #topic Release the Hound! (mordred, taron)
19:39:59 <taron> oh hello
19:40:00 <zaro> mordred: yeah, was wondering how you cobbled that together so quickly last time.
19:40:04 <fungi> mordred: so what's all this then?
19:40:05 <anteaya> taron: hey!
19:40:09 <mordred> so I dropped a hound link into the keystone meeting as an answer to a question
19:40:16 <mordred> and then people were like "zomg what's that??"
19:40:18 <anteaya> mordred: ha ha ha
19:40:23 <mordred> which made me think "we should make it a thing"
19:40:24 <nibalizer> users!
19:40:27 <stevemar_> mordred: can you blame us?
19:40:28 <taron> indeed
19:40:34 <taron> it is quite shiny
19:40:41 <mordred> and I notiecd it still had a bunch of stackforges in it
19:40:44 * AJaeger loves hound
19:40:47 <mordred> so maybe we need to restart something post rename
19:40:55 <anteaya> mordred: yes, it doesn't self update on renames
19:40:56 <taron> I still haven't figured out why the config.json is still causing puppet to complain
19:40:58 <mordred> which I htin we agreed would be ok from a process perspective
19:41:16 <mordred> so - my questions are ... should we consider the current one "live"?
19:41:24 <anteaya> which is the current one?
19:41:25 <mordred> and shoudl we restart hound so that it picks up the renames?
19:41:32 <taron> current one is hound.o.o
19:41:43 <taron> which does not run my puppet, but that can be changed once I've sorted the config issue
19:41:44 <anteaya> if taht is waht it takes to pick up the renames, yes
19:41:45 <fungi> mordred: the current one is not puppeting according to puppetboard, so i don't consider it live
19:41:53 <mordred> ah. kk
19:41:58 <anteaya> yeah, I'm not a fan of what mordred is running
19:42:05 <anteaya> it isn't current with code in git.o.o
19:42:06 <mordred> so we're still waiting to roll out taron's puppet
19:42:12 <anteaya> mordred: no offense
19:42:15 <anteaya> I am waiting
19:42:15 <mordred> cool.
19:42:17 <mordred> no  me neither
19:42:19 <mordred> :)
19:42:25 <fungi> also, it would be nice to rebuild it from scratch at least once more to make sure the deployment config management is doing the right thing
19:42:25 <anteaya> the issue was a failing test taron was debugging
19:42:32 <taron> no idea what's going on with my puppet, so I'm going to spin up a test vm and see what happens
19:42:39 <mordred> ossum
19:42:40 <anteaya> taron: do ask for help
19:42:47 <anteaya> taron: so that you don't stay blocked
19:42:57 <anteaya> taron: I was looking for you to ask, but I never saw you in channel
19:43:02 <mordred> I nominate nibalizer to help if it goes south ... I hear he understands puppet
19:43:08 * mordred is excited
19:43:09 <nibalizer> sure
19:43:10 <taron> ha, I was just about to poke him
19:43:10 <anteaya> taron: keep coming back to channel, please
19:43:16 <taron> I never leave :P
19:43:21 <anteaya> taron: good thank you
19:43:32 <anteaya> taron: I'm looking forward to us running your code
19:43:34 * mordred hands nibalizer and taron some parts of a very nice salmon
19:44:09 <taron> nibalizer: I shall have some questions about how resources work later
19:44:23 <nibalizer> sure thing
19:44:25 <fungi> i always have questions about how resources work
19:44:25 <taron> probably write them down and then poke you on thursday
19:44:34 <anteaya> taron: etherpad?
19:44:39 <anteaya> taron: then we can all look
19:44:43 <taron> good point
19:44:50 <fungi> or just a code review even
19:45:07 <fungi> sometimes code is the best means of communicating these things
19:45:19 <fungi> whatever works for you
19:45:22 * taron nods
19:45:25 <fungi> anything else hound-related?
19:45:26 <anteaya> even psuedocode
19:45:38 <anteaya> then folks can comment in line
19:45:58 <taron> come to think of it I don't have anything to do at work at the moment, so I'll start drawing that up now
19:46:11 <anteaya> taron: okay great
19:46:17 <fungi> your job sounds like mine! ;)
19:46:17 <anteaya> taron: thank you
19:46:29 <fungi> they never give me anything to do, so i just hack on openstack instead
19:46:38 <mordred> that's all I had
19:46:44 <taron> heh
19:46:48 <fungi> #topic Open discussion
19:47:33 <fungi> heads up, i'll be absent (in body, not just mind for a change!) for nearly two weeks in the middle of next month. call it november 12-23
19:47:41 <anteaya> fungi: nice
19:47:51 <anteaya> fungi: I hope you are schduling fun things
19:48:11 <fungi> yeah, messing around in boats
19:48:17 <anteaya> nice
19:48:30 <nibalizer> fungi: sweet
19:48:32 <fungi> yucatan coast (southern mex and central-am)
19:48:35 <nibalizer> have fun
19:48:43 <anteaya> oh I like it down there
19:48:58 <anteaya> the mex part, never been to the central-am part
19:49:15 <anteaya> that does sound like fun
19:49:27 <fungi> i expect so
19:50:00 <fungi> internet access is pricey on the ship, so i expect to hack offline on some personal projects and as such won't be around
19:50:32 <anteaya> fungi: good call
19:50:59 <anteaya> fungi: are you deputizing anyone in case of needing to make a decison on anything while you are away?
19:51:09 <ianw> a request for any nodepoolers -> moving f22 nodes to fedora-minimal builds : https://review.openstack.org/237389
19:51:12 <anteaya> or shall we do our best to devine your will?
19:51:29 <anteaya> divine
19:51:33 <ianw> tihs will help me getting f22 up and f21 gone...
19:51:33 <fungi> i think my will is your will, so you can all get by without me
19:51:42 <anteaya> fungi: okey dokey
19:52:20 <zaro> would be nice to get ball rolling with Gerrit 2.11 builds, just need this patch series https://review.openstack.org/#/c/236010
19:52:41 <fungi> i doubt things are going to be crazy a couple weeks after the summit anyway, so seemed like a good opportunity for some downtime
19:53:18 <anteaya> fungi: good call
19:53:39 <anteaya> fungi: if I can get enough folks together I might do the rename in your absence
19:53:47 <anteaya> fungi: I'm guessing you would be okay with that
19:53:51 <zaro> ahh, we had some discussion of planning a Gerrit cleanup which should happen before upgrade and happen seperate from the upgrade.
19:53:54 <fungi> more than okay with it ;)
19:53:59 <anteaya> fungi: heh
19:53:59 <zaro> should we leave that for another meeting?
19:54:13 <anteaya> zaro: we have a few minutes
19:54:19 <anteaya> zaro: what gerrit cleanup?
19:54:32 <fungi> zaro: yeah, we can talk about it now and/or in our channel, and/or in tokyo
19:54:41 <zaro> cool.
19:54:47 <fungi> anteaya: some ancient cruft from older renames/deletions years back
19:54:53 <zaro> anteaya: gerrit git folder and db are out of sync
19:54:55 <anteaya> ah okay cool
19:54:58 <nibalizer> anteaya: before our project rename procedure was good we left cruft in the db
19:55:01 <fungi> before i think we had entirely nailed down what's involved
19:55:02 <anteaya> zaro: okey dokey
19:55:11 <anteaya> nibalizer: yup, getting the picture
19:55:22 <anteaya> do we need downtime for the cleanup?
19:55:29 <fungi> yeah
19:55:38 <anteaya> how about when the renames are scheduled?
19:55:40 <zaro> anteaya: yes, but minimal
19:55:42 <anteaya> there are only 3
19:56:04 <anteaya> so 3 renames and gerrit cleanup in a gerrit downtime window a few weeks after summit
19:56:05 <fungi> it'll likely be quick: 1 stop gerrit, 2. backup database, 3. run cleanup query, 4. reindex lucene, 5. start gerrit and test to make sure it's okay
19:56:07 <anteaya> yeah?
19:56:34 <zaro> fungi: ugh! always forget the reindex.
19:56:49 <zaro> that might be slowish
19:56:54 <fungi> it might make sense to not mix the cleanup with the rename maintenance just in case something goes wrong and we need to roll back
19:57:03 <zaro> ++
19:57:09 <ruagair> ++
19:57:17 <anteaya> fungi: ah okay, yeah that would make sense
19:57:58 <zaro> i'll have an good esitmate from my testing though
19:58:08 <fungi> thanks zaro!
19:59:38 <fungi> well, if nobody's got anything else, i'll go ahead and close the meeting
19:59:49 <anteaya> thank you
19:59:50 <fungi> all -> #openstack-infra
19:59:58 <zaro> yes sir
20:00:02 <fungi> thanks again everybody!
20:00:06 <fungi> #endmeeting