19:03:16 <fungi> #startmeeting infra
19:03:16 <openstack> Meeting started Tue Jun 21 19:03:16 2016 UTC and is due to finish in 60 minutes.  The chair is fungi. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:03:17 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:03:20 <openstack> The meeting name has been set to 'infra'
19:03:23 <fungi> #link https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting#Agenda_for_next_meeting
19:03:29 <fungi> #topic Announcements
19:03:36 <fungi> #info The planned CI outage Friday, June 17 was postponed due to insufficient advance notification to the community
19:03:46 <fungi> pabelanger has a topic proposed later in the meeting to reschedule
19:03:57 <fungi> #info Reminder: late-cycle joint Infra/QA get together to be held September 19-21 (CW38) in at SAP offices in Walldorf, DE
19:04:24 <fungi> #link https://wiki.openstack.org/wiki/Sprints/QAInfraNewtonSprint
19:04:32 <fungi> #info Reminder: StoryBoard Bug Squash this week! (tomorrow and thursday, 22nd and 23rd)
19:04:38 <fungi> #link http://lists.openstack.org/pipermail/openstack-infra/2016-June/004402.html
19:04:41 <anteaya> yay!
19:04:49 <fungi> #topic Actions from last meeting
19:04:58 <fungi> #link http://eavesdrop.openstack.org/meetings/infra/2016/infra.2016-06-14-19.02.html
19:05:08 <clarkb> fungi what sorts of rsvp are sap/organizers looking for on that midcycle? also double check but gerrit isbhaving a meetup there at the same time too so possibilities for collaboration there too
19:05:32 <anteaya> clarkb: have you a link to the gerrit meetup info?
19:05:33 <fungi> looks like there were no action items recorded last meeting
19:05:42 <clarkb> anteaya: I can try to find it
19:05:48 <anteaya> clarkb: thanks
19:06:05 <fungi> clarkb: probably need to check with mkoderer on rsvp details if they're not in the updated info he added to that wiki page
19:06:32 <fungi> #topic Specs approval: PROPOSED Priority specs cleanup/update for Newton cycle (fungi)
19:06:40 <fungi> #link https://review.openstack.org/331903 Priority specs cleanup/update for Newton cycle
19:06:49 <mtreinish> fungi: the only rsvp I know of is putting your name on the table at the bottom of the wiki page
19:07:03 <fungi> mtreinish: thanks
19:07:06 <clarkb> https://groups.google.com/forum/m/#!topic/repo-discuss/HDHKtGUYMoY
19:07:11 <clarkb> anteaya: ^
19:07:11 <anteaya> clarkb: thanks
19:07:38 <jhesketh> Morning
19:07:42 <fungi> so anyway, as most of you following the infra ml probably saw, i proposed a (later than i would have liked) update to our priority specs list for the remainder of the newton cycle
19:08:05 <anteaya> morning jhesketh
19:08:06 <fungi> this is mostly cleanup based on the discussion which transpired earlier in that thread
19:08:26 <fungi> we can (separately) discuss if we want to add new priorities as additional patches
19:08:37 <anteaya> fungi: I'm fine with what you proposed but did reply asking about ipsilon
19:08:39 <fungi> but wanted to get this reset out of the way first
19:08:57 <anteaya> which I guess can be a separate discussion and possiblely a new patch
19:08:59 <fungi> anteaya: awesome, thanks. i think that's possibly a prime candidate for a priority addition
19:09:05 <anteaya> thanks
19:09:17 <odyssey4me> o/
19:09:23 <fungi> but we're on hold with it pending smarcet returning from his honeymoon to discuss www.o.o feature needs
19:09:39 <fungi> #info Council voting is open on "Priority specs cleanup/update for Newton cycle" until 19:00 UTC, Thursday, June 23
19:10:00 <anteaya> fungi: very good, thanks
19:10:43 <fungi> similarly, i think jhesketh had a good suggestion that we should probably still prioritize solving some of our log publishing stability issues, but that's going to require making a choice between a couple of new options we have outlined
19:11:01 <fungi> which is why it's not (yet) reflected on the updated priorities
19:11:15 <fungi> #topic Specs approval: PROPOSED Finglonger (nibalizer)
19:11:22 <fungi> #link https://review.openstack.org/310948 Finglonger
19:11:25 <nibalizer> we talked about this at the summit
19:11:26 <fungi> this looks interesting
19:11:43 <nibalizer> there are a couple ideas on how ti implement it
19:12:03 <fungi> nibalizer: you're going to need to add it to the index.rst file
19:12:05 <nibalizer> but before i go down that rabbit hole it seems like we should get some consensus?
19:13:23 <bkero> Are we talking about possibly implementing it ourselves?
19:13:38 <nibalizer> probably
19:14:01 <fungi> nibalizer: consensus about the four described implementation options i guess?
19:14:17 <mordred> nibalizer: I'm broadly in favor of the concept - I do not have smart things to say yet about specific implementations
19:14:21 <bkero> Does anyone think that the listed 'alternatives' are a good idea?
19:14:55 <fungi> i think focusing on zuulv3 is a great idea, but i don't see how it's directly related
19:15:02 <nibalizer> fungi: i think the plan in general needs some consensus? i could also go down the path of an implementation if that is what people want
19:15:03 <ianw> "There are tasks that are not managed in puppet that we currently do not have good process around." <- like what?  is debugging/firefighting considered a task that this would be used for?
19:15:07 <fungi> (one of the listed alternatives)
19:15:18 <mordred> yah - I think focusing on zuul v3 is also going to happen
19:15:40 <mordred> ianw: "hey - can we restart gerrit it's seeming slow" is a good example
19:15:44 <clarkb> is rundeck the large java app that dossnt come with good record kwwping like jenkisn?
19:15:45 <fungi> was that one just in there as a representative of "our effort is better spent solving other problems for now"?
19:15:54 <fungi> (the zuulv3 reference i mean)
19:16:04 <anteaya> clarkb is on a phone
19:16:18 <clarkb> anteaya: yes surrounded by boxes
19:16:31 <anteaya> woooo
19:16:32 <nibalizer> clarkb: its a java app, i dont have much experience with it
19:16:48 <jeblair> fungi: with zuulv3 potentially able to log into production machines and run playbooks, it may be possible to implement similar functionality with it.
19:16:49 <nibalizer> the whole point of it is tk keep an audit log though, so i assume it does
19:17:03 <fungi> jeblair: thanks--that was the connection i was missing
19:17:11 <jhesketh> Without much knowledge on the subject; why not rundeck?
19:17:12 <jeblair> fungi: but that's far enough off, and complex enough, that i think doing something like finglonger makes sense now
19:17:15 <clarkb> nibalizer: but its all one off hand edited scripts with no history like jenkins jobs iirc
19:17:24 <clarkb> jhesketh: because ^
19:17:42 <mordred> yah
19:17:43 <clarkb> it basically reintroduces all of the jenkins issues we spent 4 years fighting until we deleted it
19:17:49 <mordred> I don't think rundeck is a fit for us
19:18:09 <nibalizer> but if you look at implementation #4 its possible to do with a cronjob and a git hook
19:18:10 <jeblair> especially since finglonger is trying to be pretty simple.  and if we *do* start doing things like that with zuul3, we will have the experience of finglonger to inform how do use it
19:18:20 <mordred> jeblair: ++
19:18:27 * jhesketh is a slow typer this morning
19:18:29 <mordred> also, finglonger is just a great name
19:18:42 <anteaya> jhesketh: well it is early for you
19:18:59 <jesusaur> has ansible made any mention of open-sourcing tower? or is that a pie-in
19:19:05 <jesusaur> -the-sky alternative?
19:19:07 <mordred> it will happen
19:19:12 <mordred> timeline is the question
19:19:14 <nibalizer> jeblair: yea, i want finglonger to be stupid sinmple
19:19:18 <clarkb> anyways I like the idea psrticularly for precanned but infrequent tasks
19:19:24 <mordred> clarkb: ++
19:19:32 <anteaya> jesusaur: I was thinking pie-in was a new buzzword
19:19:33 <nibalizer> then it shoukd inform the CD parts of zuulv3 yea
19:20:05 <jesusaur> anteaya: ha, no, just my - and enter being too close together
19:20:11 <anteaya> :)
19:20:38 <mordred> anteaya: I think pie-in should be a new buzzword
19:20:44 <anteaya> mordred: I agree
19:20:55 <Zara> buy-in as a result of convincing pie-charts?
19:21:01 <clarkb> restart gerrit, rebuild nodepool images, etc
19:21:08 <anteaya> Zara: oh, thinker you are
19:21:33 <nibalizer> make a new volume, add to afs, clean out files created by accident
19:21:35 <jhesketh> If tower isn't opened sourced soon somebody else will beat them to it. I'd prefer to use existing tooling but I realise that requires waiting around on  unknowns
19:21:38 <anteaya> save the check and gate queue
19:21:47 <jhesketh> (And we could be the ones to beat them to it)
19:21:59 <fungi> i think as long as these scripts and their triggered invocation go somewhere to be very, _very_ carefully reviewed for possible risk (not only in their direct effects but also the side effects of publishing their output), it's not particularly any more dangerous than what we already do with puppet/ansible and our current interactive work
19:22:02 <mordred> there is also the thing that we'd still have to figure out how to tell tower to do something
19:22:03 <nibalizer> jhesketh: also if you look at #4 finglonger can be made very very easily
19:22:14 * reed drools at open source tower :)
19:22:18 <nibalizer> fungi: es
19:22:19 <mordred> like, if tower was open sourced tomorrow, we'd STILL need to do work
19:22:20 <nibalizer> yes
19:22:25 <mordred> in terms of triggering, etc
19:22:37 <mordred> so I imagine that finglonger could still have a place in a post-tower-open-source world
19:22:43 <jeblair> we would need some kind of very long finger to reach up to the button in the tower
19:22:48 <mordred> jeblair: yup
19:23:02 <nibalizer> jeblair: groan
19:23:02 <fungi> nibalizer: honestly, i'm more concerned about publishing the output since that can't be reviewed directly until it's public
19:23:16 <clarkb> fungi: thats a good point
19:23:21 <mordred> fungi: speaking of that - we should really check out ara more at some point
19:23:27 <clarkb> a task could cat private key data
19:23:46 <mordred> and see if/how it can/cannot help us
19:23:48 <nibalizer> fungi: well with precanned things we can socialize a rule that the first time a job runs it doesnt publish to www
19:23:50 <fungi> clarkb: well, we'd presumably catch something overt in reviewing the proposed command
19:23:56 <nibalizer> to verify
19:23:59 <mordred> nibalizer: ++
19:24:43 <fungi> but yeah, maybe this is only for commands we've already run in production at least once and are comfortable they don't leak any sensitive data in their output (as well as comfortable they are unlikely to break something, obviously)
19:24:52 <nibalizer> sice i inagine most finglonger scripts to end in | ssh logs.openstack.org cat - > /var/www/something
19:25:29 <fungi> nibalizer: or, as mordred says, ara
19:25:53 <jesusaur> ara?
19:25:57 <nibalizer> eh maybe
19:26:11 <nibalizer> we went down that road with puppetboard, and now injust want shell output
19:26:19 <mordred> ara is orthogonal to this discussion
19:26:26 <fungi> nibalizer: yep, i can sympathize
19:26:28 <nibalizer> i also expect more finglonger scripts to be shell than ansible
19:26:35 <nibalizer> but thats a guess
19:26:38 <mordred> nibalizer: oh, I do not expect that at all
19:26:44 <fungi> oh, good point, not "shell run via ansible" but just "shell"
19:26:55 <clarkb> jesusaur: google says its a modular phone so I have no idea
19:27:05 <jesusaur> clarkb: yeah, I got the same thing
19:27:09 <nibalizer> mordred: we will see
19:27:12 <mordred> I do not like the idea of having a bunch of non-ansible shell scripts being run by another system
19:27:21 <nibalizer> its a comminity web ansible report viewer thing
19:27:22 <mordred> like, I don't like that idea very much
19:27:29 <fungi> clarkb: jesusaur: an ansible dashboardy thing dmsimard has written and proposed as project in gerrit
19:27:31 <odyssey4me> clarkb jesusaur https://github.com/dmsimard/ara
19:27:39 <clarkb> oh that thing
19:27:45 <mordred> because then we'd have THREE systems for running things
19:27:46 <clarkb> ya thats orthogonal
19:27:53 <nibalizer> oh its in our stuff now? neat
19:28:17 <fungi> dunno if that new project change request has merged yet, but he's wanting to maintain it through gerrit anyway
19:28:27 <fungi> but regardless, side topic
19:28:43 <nibalizer> is there enough in the spec to vote on it?
19:28:48 <jesusaur> mordred: so you're thinking finglonger would look like a playbook with the special timer trigger magic?
19:28:57 <jesusaur> rather than a collection of shell scripts?
19:29:08 <mordred> jesusaur: I'm thinking neither thing
19:29:15 <mordred> I'm thikning it would be something that would run ansible
19:29:23 <mordred> we already hav ea system for running ad hoc commands on hosts
19:29:23 <jesusaur> ahh
19:29:28 <mordred> we just don't hav ea system to drive it
19:29:41 <mordred> so I REALLY don't want to see a new system invented for running ad hoc commands
19:29:43 <fungi> so it sounds like there is some consensus the goal with finglonger still makes sense (as discussed in austin) but you still have four implementation options (listed at "three ideas..." yay math!) which need to be decided between i guess?
19:30:47 <nibalizer> one was a late addition
19:30:52 <fungi> heh
19:31:31 <nibalizer> so we coukd vote in the spec then the people working on it (probably me) could build what works and makes sense
19:31:31 <fungi> i posit we want to see the implementation section whittled down to one design before we put this up for council vote
19:31:53 <fungi> the design can change, but at least say what you expect to try first
19:31:58 <nibalizer> do people have strong implementation opinions?
19:32:00 <nibalizer> ok
19:32:12 <fungi> mordred seemed to have a very strong implementation opinion, for example
19:32:28 <fungi> (that opinion being, we should trigger ansible somehow)
19:32:38 <mordred> well, I have a strong opinion on one portion :)
19:32:38 <nibalizer> ok
19:32:46 <bkero> I think the general idea of code as a liability applies here
19:33:25 <fungi> but yeah, it's nibalizer's spec, i'd at least like to see his list of implementations to try turned into a "nibalizer's preferences" ordering
19:33:31 <nibalizer> ok i will flesh out only one implemenation secrion of the spec then
19:33:46 <nibalizer> or a list haha
19:33:48 <fungi> you can mention alternate implementations, just suggest one
19:33:51 <nibalizer> ok
19:33:58 <nibalizer> thanks
19:34:02 <mordred> nibalizer: can you just implement all of the options, and then we can all sit in a room and judge you?
19:34:26 <nibalizer> lol
19:34:39 <fungi> sounds like the plan for barcelona
19:34:47 <pleia2> haha
19:35:30 <fungi> nibalizer: also see my (trivial but important) -1 when you're revising
19:35:40 <fungi> anything else on this one?
19:35:41 <nibalizer> yep
19:35:43 <nibalizer> nope
19:35:57 <fungi> thanks nibalizer! this looks like it is shaping up to be awesome
19:36:09 <mordred> ++
19:36:17 <fungi> #topic Priority Efforts
19:36:27 <fungi> i'll refresh this list once we decide on my proposed update
19:36:41 <fungi> then we can have a run-through hopefully next week
19:36:57 <fungi> #topic New date for zuul.o.o / static.o.o trusty upgrades (pabelanger)
19:37:09 <fungi> pabelanger is still not in here
19:37:31 * mordred hands fungi a slightly used pabelanger he found at the mall
19:37:31 <fungi> however, he left us a lovely etherpad
19:37:43 <pleia2> mordred: he's sticky
19:37:44 <clarkb> did we get feedback that more lead time was needed? I think we gave a couple weeks?
19:37:49 <pleia2> ick, mall things
19:37:57 <mordred> pleia2: inorite?
19:38:04 <fungi> #link https://etherpad.openstack.org/p/upgrade-zuul-trusty proposed maintenance notification
19:38:08 <clarkb> not sure what constraints we are operating under here
19:38:39 <anteaya> clarkb: well we decided on a couple weeks, but the announcement didn't happen in time to give a couple weeks
19:38:52 <pleia2> ah yes, I can make sure the announcement goes out this time
19:38:56 <clarkb> ah ok
19:39:04 <anteaya> clarkb: so we knew about it but noone else did
19:39:16 <fungi> i don't recall how much advance notice we planned for, but regardless there wasn't one as of last thursday so i suggested not doing it the following day
19:39:21 <anteaya> <-- very slow library wifi, sigh
19:39:21 <jeblair> it was published in locked filing cabinet in the basement
19:39:42 <mordred> jeblair: that's a totally reasonable place to publish notice
19:39:49 <pleia2> Beware of the Leopard.
19:39:53 <fungi> however i had discarded the sign on the lavatory door saying "beware of teh leopard"
19:40:04 <anteaya> fungi: we had planned for 2 weeks
19:40:18 <anteaya> fungi: but I think one week is enough, but yeah not 24 hours
19:41:10 <fungi> anteaya: yeah, sort of two weeks. at the meeting on the 7th we said we'd do it on the 17th, so more like a week and a half (to give time to have an announcement go out at least a week ahead)
19:41:24 <anteaya> yeah
19:41:59 <anteaya> so next Friday July 1 is Canada Day so pabelanger and I will be on holiday
19:42:03 <fungi> so anyway, with that in mind, who's around on july 1st? it's the day before a popular holiday weekend in the usa so i expect there may not be a lot of infra-root admins on hand
19:42:09 <fungi> right-o
19:42:36 <anteaya> and the week following I may or may not be in China, I don't know yet
19:42:42 <mordred> fungi: I will be around
19:42:50 <anteaya> but we don't need me for this
19:42:53 <mordred> the week following I will definitely be in China
19:42:56 <fungi> the release schedule is pretty open up through mid-august
19:43:04 <pleia2> I need to leave a bit early on the 1st, so it's not a great day for me
19:43:05 <clarkb> I can likely be around on july 1
19:43:13 <pleia2> the 8th is better
19:43:17 <fungi> with the exception of july 11-15 for n2
19:43:32 <jeblair> i think i'm available on the 1st
19:43:33 * bkero will be around for moral support
19:43:34 <fungi> i can do july 1 with no problem
19:44:18 <fungi> anybody want to volunteer to update pabelanger's proposed announcement to say july 1 and then send it? that give us a solid week+ of warning
19:44:38 <pleia2> I can do that
19:44:46 <jeblair> (yes, i am available on the 1st)
19:44:48 <fungi> i'm comfortable upgrading static.o.o and with jeblair around for support i'm comfortable with the zuul.o.o upgrade as well
19:44:57 <fungi> thanks pleia2!
19:45:35 <fungi> #action pleia2 send maintenance notification for static/zuul server upgrades 20:00 utc friday july 1
19:46:14 <fungi> #info CI outage Friday, July 1 so we can upgrade the operating system for zuul.openstack.org and static.openstack.org, 20:00-22:00 UTC
19:46:45 <fungi> anything else we need to discuss about the upgrade/scheduling?
19:47:01 <anteaya> I'm good
19:47:32 <fungi> i don't see any other late entry topics on the agenda, so...
19:47:34 <fungi> #topic Open discussion
19:48:00 <fungi> did anybody see that sports match where the team in one color triumphed over the inferior specimens wearing that other color?
19:48:05 <anteaya> so tomorrow's storyboard bug sprint is sort of a new contributor onboarding thing too
19:48:16 <anteaya> tomorrow and Thursday
19:48:23 <anteaya> so pleae attend and bring a friend
19:48:40 <fungi> i'm quite excited. i'd like to spend a fair amount of time focused on storyboard bits since i haven't really up to now
19:48:41 <pleia2> thanks for the reminder!
19:48:48 <anteaya> thank you
19:48:50 <clarkb> fungi: I didnt, the game was broadcast on fs1 instead of fox
19:48:59 <anteaya> pleia2: fungi looking forward to seeing you there
19:49:05 <bkero> fungi: i am a fan of citynamesportsteam
19:49:09 <Zara> :D
19:49:12 <SotK> \o/
19:49:23 <jeblair> fungi: i did not see it, but everyone else in my town did and i heard them yelling about it and was able to follow it fairly well nonetheless.
19:49:24 <fungi> bkero: those losers? they can't compare to othercitynamesportsteam
19:49:27 <pleia2> I should have zaro's hiera changes in for the gerrit<>sb dev integration by EoD today
19:49:28 <anteaya> Manchester United!
19:49:43 <AJaeger> jeblair asked for consensus on adding back the files section on project-config, see https://review.openstack.org/#/c/330821/ - how do we want to continue there?
19:49:48 <anteaya> pleia2: yay, thank you
19:50:20 <fungi> jeblair: yeah, when i lived in an apartment in chapel hill for a number of years, i could follow the games by the thumping coming from my ceiling, the general hooting through walls, and of course the incessant honking of horns all over town afterward
19:50:49 <jeblair> i think it would be really nice if we could get more folks to weigh in one way or another on https://review.openstack.org/#/c/330821/
19:51:02 <anteaya> after the world cup in germany the germans made noise for an entire 10 minutes, then got in their cars and went home
19:51:06 <anteaya> it was a sight to see
19:51:19 <Zara> Manchester is the worst place to live if you want to avoid football fans
19:51:33 <anteaya> Zara: or going there on the train too
19:51:51 <Zara> just being anywhere within 200 miles, probably
19:51:54 <jeblair> since it reverts a change with 2 +2s, none of whom have voted on the revert
19:52:00 <bkero> anteaya: i am in germany, they are still making noise
19:52:15 <anteaya> bkero: really? how interesting
19:52:31 <bkero> yeah, its a special day too. Lots of live music all over berlin
19:52:33 <jeblair> i think reviews from clarkb, fungi, yolanda on 330821 would be especially good
19:52:43 <docaedo> speaking of football, any more thoughts about my IRC spec? https://review.openstack.org/#/c/319506/
19:52:44 <zaro> pleia2: i Will not be installing storyboard plugin with puppetry. my patches won't land in time.  youll just need to give me the token
19:52:56 <anteaya> I did find though there there was only one file clip that could be played for audio/video demonstrations
19:52:57 <fungi> clarkb: any chance you can wrangle phone access to weigh in on 330821? you were the other concerned voice earlier
19:53:10 <pleia2> zaro: ok, we'll chat after the meeting
19:53:16 <AJaeger> jeblair: clarkb just said on #openstack-infra that he's fine with reverting it, let me find his quote
19:53:32 <AJaeger> "AJaeger: I have already said my piece if others want to take the time to review and keep those up tondate thats fine"
19:53:59 <clarkb> ya I am ok with it if other people want to be maintaining rukes as files move and are added/removed
19:54:18 <fungi> my only caveat is that we need to make sure we have at least one job which runs on any change even if it's a new filename we haven't predicted
19:54:37 <fungi> there have been points in time where that was not the case, and then we couldn't merge changes
19:54:44 <AJaeger> The layout change is the one that always runs - and I'm not going to approve again a files section for that. Been there, reverted ;)
19:55:01 <jeblair> i think that was a long time ago
19:56:39 <fungi> looks like we've run out of open discussion
19:57:14 <AJaeger> ok, I'll ping yolanda for the change and if she's fine, let's merge it...
19:57:36 <docaedo> fungi: except for my IRC topic (don't forget you're the spirit guide for it ;) )
19:58:07 <fungi> docaedo: indeed--i tried out your demo server for a couple days too. seemed to reconnect and do stuff nicely!
19:58:15 <anteaya> fungi is everyone's favourite spirit guide
19:58:15 <docaedo> yay!
19:58:39 <docaedo> I think of this as a sort of eternal september except for IRC instead of usenet
19:58:51 <fungi> let's hope not?
19:58:53 <docaedo> oh wait, that might not turn out so great...
19:58:54 <docaedo> haha
19:58:59 <fungi> i keep waiting for eternal september to end
19:59:15 <mordred> fungi: we're moving mailing lists to NNTP???
19:59:28 <fungi> it actually basically has. there's almost nobody on usenet anymore. not even spammers
19:59:29 <mordred> that's what I just heard at leats
19:59:46 <anteaya> well if the spammers have left...
19:59:51 <fungi> we could claim it for our own and stake our flag there
19:59:54 <mordred> anteaya: that's what I'm saying!
19:59:59 <fungi> anyway, that's it for time this week
20:00:03 <fungi> thanks everybody!
20:00:07 <fungi> #endmeeting