19:02:51 <jeblair> #startmeeting infra
19:02:51 <openstack> Meeting started Tue May 14 19:02:51 2013 UTC.  The chair is jeblair. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:02:52 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:02:55 <openstack> The meeting name has been set to 'infra'
19:03:07 <jeblair> pabelanger, mordred: ping
19:03:44 <fungi> public service reminder: when mass-deleting servers from the cli, make quadruply sure to use the right pattern
19:03:58 <fungi> triple-checking is clearly insufficient (for me at least)
19:04:08 <jeblair> #link https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting
19:04:35 <jeblair> #action actions from last meeting
19:04:39 <jeblair> #topic actions from last meeting
19:04:42 <jeblair> even
19:05:06 <clarkb> I have put up a bug day schedule on our wiki page
19:05:16 <jeblair> #help
19:05:18 <mordred> o/
19:05:26 <jeblair> meetbot?
19:05:33 <clarkb> #link https://wiki.openstack.org/wiki/InfraTeam#Bugs
19:05:40 <fungi> metbot is in here...
19:06:09 <ttx> o/
19:06:40 <fungi> however, meetbot is missing his cloak
19:06:41 <clarkb> each bugday is the tuesday after a milestone and starts at 1600UTC so that we can go through them before this meeting
19:06:51 <pleia2> yay
19:07:31 <fungi> i think he came back in on a netsplit while services was still absent or something. want me to bounce the process?
19:07:32 <jeblair> ah, so it may just not be able to set the topic
19:07:42 <jeblair> fungi: nah, we'll just live without the topic changes for now
19:07:49 <jeblair> and assume the logging commands are working
19:07:50 <fungi> k
19:07:56 * fungi checks that
19:08:09 <pleia2> my hope is that these bug days will become less needed and shorter as we get better at using the tracking day to day
19:08:21 <clarkb> I have also got hpcloud az3 in the devstack-gate rotation using the new account. I will do az2 and az1 next over the course of today assuming things rotate out quick enough
19:08:28 <clarkb> pleia2: ++
19:08:32 <mordred> ++
19:08:38 <fungi> meetbot seems to be logging to eavesdrop okay
19:09:01 <jeblair> fungi: is our foundation meeting at 1600 utc?
19:09:26 <fungi> jeblair: possibly. looking
19:10:11 <fungi> oh, right normally yes
19:10:20 <clarkb> we can shift the time as necessary. I threw something out there so that we can have this discussion :)
19:10:26 <fungi> we meet tuesdays at 1600 utc (foundation staff)
19:10:39 <clarkb> fungi: is 1700 to early then?
19:10:43 <jeblair> maybe we could do it at 1700?  that still gives us 2 hours
19:10:58 <clarkb> that works for me. I will update the wiki
19:11:01 <fungi> clarkb: 1700 is fine by me
19:11:21 <clarkb> we can continue post this meeting if we really need to but as pleia2 says hopefully we get good at it and that isn't necessary :)
19:11:38 <jeblair> clarkb: testr?
19:12:24 <jeblair> "clarkb to ping markmc and sdague about move to testr"
19:12:37 <clarkb> I filed https://bugs.launchpad.net/python-keystoneclient/+bug/1177924 to track testr adoption across the projects that haven't switched
19:12:39 <uvirtbot`> Launchpad bug 1177924 in python-keystoneclient "Use testr instead of nose as the unittest runner." [Undecided,In progress]
19:13:16 <clarkb> had a quick chat with sdague and realized he probably isn't the best person to drive this. ttx mentioned getting it targeted to a milestone so I have been trying to ping people involved with those projects to drum up support
19:13:50 <clarkb> which has started the ball rolling
19:14:02 <jeblair> clarkb: cool; it looks like there are reviews listed too
19:14:28 <clarkb> yup. I have resurrected mordred's python-keystoneclient change and asalkeld_ is doing ceilometer
19:14:48 <clarkb> oh and sbaker is doing heatclient
19:16:03 <jeblair> clarkb: anything else re testr?
19:16:23 <clarkb> I don't think so. basically going to continue arguing for it so that we can get interest and effort
19:16:31 <jeblair> "fungi open a bug about replacing mirror26 and assign it to himself"
19:16:32 <clarkb> and support as necessary
19:16:42 <fungi> bug opened, worked and closed
19:16:58 <jeblair> woow
19:17:09 <fungi> mirror26 now runs centos and the old oneiric one is gone
19:17:14 <jeblair> i deleted lists and eavesdrop
19:17:27 <jeblair> i also moved paste and planet to new servers
19:17:40 <jeblair> mordred did inform the tc of current testing plans
19:17:42 <jeblair> i saw him do it
19:17:46 <jeblair> mordred: any yelling?
19:17:48 <fungi> righteous
19:18:06 <clarkb> I don't remember any
19:18:13 <jeblair> fungi open bug about spinning up new precise slaves, then do it
19:18:22 <jeblair> that happened.
19:18:25 <clarkb> he did it twice ;)
19:18:27 <jeblair> twice, even, for good measure
19:18:30 <fungi> yeah
19:18:37 <fungi> collateral damage aside
19:18:39 <fungi> all done
19:18:48 <fungi> and the right quantal slaves are deleted now
19:18:52 <jeblair> "clarkb to get hpcloud az3 sorted out"
19:19:05 <fungi> (not the quantal slaves which were named precise[0-9]+)
19:19:24 <clarkb> that seems to be mostly sorted. trying to pay attention to test results to make sure az3 isn't failing devstack gate tests for weird reasons
19:19:44 <jeblair> clarkb: cool.  probably good to double check the runtime too
19:19:48 <clarkb> yup
19:19:52 <jeblair> "fungi add change to disable nova py26 tests for folsom"
19:20:15 <fungi> that was done when the remaining jobs were changed over to centos6 node labels
19:20:26 <jeblair> annegentle: ping
19:20:32 <fungi> and that bug is wrapped up and closed out now as well
19:21:21 <jeblair> cool.  i think there was some overlap with agenda items and items from last meeting
19:21:31 <jeblair> so i think we're nominally ready to skip to the docs topics
19:21:45 <jeblair> though i think clarkb wanted annegentle here for those...
19:21:58 <clarkb> yeah particularly the for the one about publishing old docs
19:22:23 <clarkb> basically do we want to publish the docs based on the eol tags? or just not bother?
19:22:24 <jeblair> clarkb: do we have a quorum for the maven topic?
19:22:38 <clarkb> jeblair: I think so
19:22:54 <jeblair> clarkb: yeah, good question for annegentle.  :)
19:23:00 <jeblair> #topic clouddocs-maven-plugin internalization
19:23:08 <jeblair> #link https://bugs.launchpad.net/openstack-ci/+bug/1082812
19:23:09 <uvirtbot`> Launchpad bug 1082812 in openstack-ci "Move rackspace/clouddocs-maven-plugin to openstack-dev/clouddocs-maven-plugin" [High,Triaged]
19:24:02 <clarkb> so I am a horrible person and forget names, but do we want to make clouddocs-maven-plugin a stackforge project and have them drive most of the things or put it under the openstack-infra umbrella and make things go?
19:24:44 <jeblair> clarkb: openstack-infra i think.
19:24:53 <jeblair> clarkb: is it a problem that https://github.com/rackspace/clouddocs-maven-plugin does not exist?
19:25:13 <clarkb> jeblair: yes, I think that means it is currently a private repo
19:25:14 <mordred> jeblair: sorry - got distracted - no yelling, they were fine with it
19:25:24 <jeblair> i think it moved to https://github.com/rackerlabs/clouddocs-maven-plugin
19:25:27 <clarkb> ah
19:25:38 <jeblair> David Cramer authored a commit a day ago
19:25:46 <jeblair> clarkb: i think it was david cramer we talked to.  :)
19:26:04 <clarkb> ok, in that case I can whip up a change to slurp it in and have dwcramer give us the go ahead
19:26:20 <fungi> dcramer_ seems to be in here... the same?
19:26:21 <clarkb> the other piece to that is publishing it to maven nexus, zaro made that sound easy so I am not too worried about it
19:26:21 <jeblair> clarkb: my notes said that the release process would be:
19:26:28 <jeblair> generate two commits (remove snapshot, inc+add snapshot) to pom.xml
19:26:33 <jeblair> then tag the correct one after merging
19:26:44 <jeblair> (inc == increment version number)
19:26:53 <zaro> clarkb: theoretically
19:27:00 <clarkb> yup then the tagging will kick off a maven nexus publish build
19:27:36 <clarkb> (it is worth noting that this release dance is one of the reasons Gerrit is trying Buck)
19:27:48 <jeblair> and all of this is probably very similar to what we should need to do for the gearman plugin
19:28:45 <jeblair> clarkb: eot?
19:28:47 <mordred> yes. teaching jenkins/gerrit about mvn release process - or teaching mvn about gerrit process is the game
19:28:48 <clarkb> yup
19:29:05 <jeblair> #topic What can infra do to enable remote participation at the next Design Summit?
19:29:10 <jeblair> reed: ping
19:29:14 <reed> ping
19:29:23 <reed> I can't play that game
19:29:45 <reed> we have basically two issues for the next summit:
19:30:07 <reed> -  we have never found a good technical solution for remote participation at the design sessions
19:30:37 <reed> -  we only practiced IRC for meetings as a community
19:31:24 <reed> I think we should have a way for the community to practice how to hold discussions across different places
19:31:57 <reed> before I start the conversation online where I know people will suggest we use google hangout, i'd like to explore what the infra team can do
19:32:35 <reed> the main alternatives I see to hangout/skype are XMPP+jingle if we want video and mumble/murmur
19:32:39 <clarkb> is the problem we are trying to solve general realtime collaboration or realtime collaboration specific to the summit because I think they are different
19:32:43 <clarkb> *different problems
19:32:56 <fungi> thinking about the summit design sessions, what goes on is mostly a three-way combination of voice discussion, slideshow presentations and etherpad note taking
19:33:03 <reed> clarkb, the issue is specific to summit
19:33:15 <clarkb> in the general case you can assume everyone is using the same set of tools, but in design sessions half of your attendance will be in one place and the other half will use the tools
19:33:20 <reed> and to user group meetings, if you want
19:33:55 <fungi> etherpad is already remotely joinable, so we basically need something to bridge the gap for voice and slideshows i think
19:34:14 <jeblair> fungi: i don't think slideshows are that important.  they should be discouraged for design summit sessions anyway.
19:34:18 <clarkb> we could demand no slide decks to solve half that problem :)
19:34:21 <clarkb> jeblair: jinx
19:34:24 <mordred> jeblair: ++
19:34:26 <fungi> jeblair: granted, i agree with you on that
19:34:33 <reed> fungi,  faces would be a nice addition
19:34:33 <russellb> or post slides somewhere you can download them ...
19:34:55 <reed> slides can also be done in html :)
19:34:58 <russellb> sure.
19:35:09 <russellb> so it's the voice and video part.
19:35:16 <russellb> asterisk can do it
19:35:29 <ttx> russellb: you have a highlight on... XMPP ?
19:35:36 <reed> russellb, SIP?
19:35:50 <russellb> reed: SIP and XMPP at the same time, if you'd like
19:36:01 <clarkb> does that place unreasonable requirements on the network at the conference center? they never seem to handle things well
19:36:06 <reed> consider we'll be in Hong Kong, so networking can be challenging, too :)
19:36:10 <jeblair> how about focusing on just audio -- i'm a little concerned about bandwidth and network reliability...
19:36:19 <clarkb> jeblair: ++
19:36:25 <russellb> any voice+video (*especially* video) solution is going to be bandwidth hungry
19:36:35 <reed> clarkb, god only knows what will happen there... but we can make them pay super-serious attention to that
19:36:38 <russellb> just audio is even easier
19:36:39 <fungi> without faces, i think we still need something which will pop up a subtitle indicating who's talking
19:36:55 <ttx> one-way audio with IRC feedback is what Ubuntu used to use at UDS
19:36:59 <clarkb> I am not familiar with asterisk but mumble does rooms like irc channels
19:37:25 <ttx> with a projection of an IRC channel all the time, difficult to ignore
19:37:30 <jeblair> ttx: that seemed very reliable
19:37:48 <ttx> critical participants can be hooked in in two-way voice, if necessary
19:37:59 <fungi> having been on my share of two-way conference calls, it gets unmanageable once you have 10 or more people trying to keep from talking over top of one another, without a moderator to voice and devoice them
19:38:02 <russellb> i like that.
19:38:13 <reed> the main drawback of that is that I can't see how we can have that system tested/deployed before the Summit
19:38:20 <reed> I explain:
19:38:24 <ttx> I mean, the session moderator knows if he has a ccritical participant and can hook him in
19:38:38 <ttx> everyone else can have one-way audio + IRC
19:38:49 <reed> my idea is to ask user groups to test the remote system we put in place so we get to HK with practice and knowledge
19:38:49 <jeblair> ttx: +1
19:38:50 <russellb> just need decent audio equipment so that it's actually usable ...
19:38:51 <fungi> that sounds reasonable to me
19:39:32 <ttx> we could even abandon the idea of lurkers on one-way audio + IRC and just support critical participants in a ad-hoc manner
19:39:36 <reed> I don't see how IRC+audio streaming can be made enough for user group Atlanta to chime in the meeting in Chicago, for example
19:39:50 <fungi> russellb: we had "decent" audio equipment in san diego (not awesome, but usable). the techs who wired the rooms did a less than perfect job though
19:39:53 <jeblair> reed: i think we could still test that in the manner you are discussing
19:40:10 <ttx> frankly, you know when you've a critical person missing from the discussion.
19:40:12 <reed> jeblair, I think we need video, too
19:40:13 <russellb> fungi: was that when we had the webex things?
19:40:29 <fungi> russellb: yes, webex itself is crap though, in my opinion
19:40:44 <russellb> needing video, but being concerned about conference internet, is going to be a bit concerning
19:40:51 <jeblair> reed: the thing we can't test is how the network in hong kong is going to perform, which is why i think focusing on resilient low-bandwith protocols is our best way to prepare for that.
19:40:54 <ttx> so I think having decent phones in each room that a small set of people, invited by the PTL, can call in...
19:41:07 <ttx> is all we need from a technical perspective
19:41:22 <reed> russellb, we will take care of the infrastructure later, once we have the specs ready... it's a matter of writing down the contracts
19:41:23 <ttx> one-way audio + IRC is a plus
19:41:53 <russellb> just a phone is a bit of a pain, not everyone will be able to hear it, and the people on the phone wont' hear the whole room
19:42:02 <jeblair> reed: wasn't there already a spec for the network?  i seem to remember some bandwidth number is the proposal responses from the venue
19:42:05 <russellb> not much better than someone's laptop
19:42:06 <ttx> Personally I don't mind if we lose the US lurkers. We'll have the APAC ones in the rooms to replace them.
19:42:19 <reed> jeblair, the bandwidth was never an issue afaik
19:42:26 <russellb> problem is the lurkers complain when they can't hear something
19:42:51 <reed> AFAIK we never had issues at the summit with streaming videos
19:42:57 <ttx> russellb: that's why, personally, I would get rid of the intermediary setup
19:43:24 <ttx> russellb: have a solution for critical participants if we need them
19:43:25 <reed> the problems users saw at this summit were for few failed hardware and some closed ports
19:43:52 <ttx> russellb: but not try and fail at something that pretends to enable remote participation
19:44:41 <fungi> in san diego, i had to investigate and restart webex on average twice a day, and i was only responsible for one room
19:44:58 <reed> is there a system that we can put in place that supports chat, audio and video and, if networking fails, drop video?
19:45:16 <jeblair> ttx: so that's a big first question we need to answer: do we want to enable general streaming for everyone, or focus just on the critical participants (invite only)?
19:45:33 <clarkb> I think chat is IRC and should remain IRC as we already have a ton of tooling around it and it is familiar
19:45:36 <reed> fungi, what was the cause of that? networking or webex failures?
19:45:43 <ttx> jeblair: yes
19:45:45 <fungi> reed: networking and hardware failures
19:46:07 <fungi> reed: my room had the laptop that kept locking up too
19:46:25 <reed> so we need something that can basically run as a daemon
19:46:28 <ttx> jeblair: I think we can do a good job at handling critical participation... not convinced we can build a good story for general remote participation... and doing a bad one is worse than not doing it
19:46:31 <reed> headless preferrably
19:46:48 <jeblair> ttx: indeed, things will go wrong, so we should keep it as simple as we can
19:46:53 <clarkb> ++
19:47:10 <reed> ttx: for critical participation we're investigating the possibility to pay for their travel
19:47:12 <fungi> so the critical-participants-only solution seems to be in line with the atc-only physical access to the rooms. open remote participation is likely to draw non-participant lurkers
19:47:47 <ttx> reed: it's a bit difficult though. some people don't know they are critical until the session is up
19:47:51 <fungi> much in the same way that discouraging non-atc participants helped a bunch in portland
19:47:57 <ttx> flying them on short notice is kinda expensive
19:48:07 <reed> ttx, got it, make sense
19:48:22 <ttx> in all cases you'll have critical people that won't show up
19:48:30 <ttx> even when it's in the US we have that scenario
19:48:50 <reed> yeah
19:49:03 <reed> so what you're suggesting is to ...
19:49:27 <ttx> so having a conference phone and bridges set up in the rooms might be all we need
19:49:30 <reed> proceed like we did in Portland? nothing  globally, and ad-hoc per room?
19:49:36 <fungi> dedicated conference bridge number per room, with a conference room phone
19:49:43 <ttx> that ^
19:50:05 <clarkb> and dedicated irc channels and etherpads per session
19:50:15 <jeblair> reed: which isn't ad-hoc at all.  i definitely think we should spend some time setting this up and making sure it works.
19:50:19 <clarkb> or per room. eg somewhere people can find the info if mordred doesn't do it ahead of time
19:50:55 <reed> jeblair, not very sexy, but we can throw it out there
19:51:26 <jeblair> reed: i love things that work, regardless of how sexy they are.
19:51:31 <fungi> i'd rather have functional than sexy
19:51:36 <fungi> or what jeblair said
19:51:41 <ttx> clarkb: re: dedicated irc channels -- that's a bit useless as we ignore them
19:51:41 <reed> you guys
19:52:01 <clarkb> ttx: yeah I am just thinking that a lot of the sessions don't have a remote meet point until they start
19:52:04 <jeblair> ttx: if we projected them like UDS
19:52:16 <clarkb> ttx: so we need someway of making sure that the important participants know what things are ahead of time
19:52:20 <ttx> clarkb: it's actually harder to ignore the etherpad, and the chat there
19:52:21 <jeblair> alternatively, we could encourage the use of the etherpad chat
19:52:25 <russellb> yeah, projecting them would make it work, i think
19:52:25 <clarkb> irc or etherpad etc
19:52:33 <reed> we'll need a webpage to summarize it all
19:52:48 <ttx> sure. We can have a box to project the etherpad in each room
19:52:50 <reed> the agenda will have to contain the IRC channel and the phone number for the room
19:53:17 <ttx> reed: hmm. If too many people join the bridge it will be useless for the critical people
19:53:20 <russellb> etherpad chat is ok too though, really ... everyone is looking at etherpad already anyway, and it's archived with the notes
19:53:28 <ttx> I'd rather have the moderator invite critical people in
19:53:30 <fungi> we'd still need a little more cultural pressure to not ignore them. i saw a tendency in some design sessions to not pay attention to the etherpad chat window even though it was on the projector
19:53:35 <reed> I still think that we won't find a way to test this system with the user groups, it's not useful for that use case
19:53:35 <russellb> you can set up a bridge to be listen-only by default unless you have a code to be a talker
19:53:55 <jeblair> russellb: should we attempt to use ip telephony or pots?
19:54:05 <ttx> russellb: ah, interesting
19:54:28 <russellb> either ... ip should be fine, but depends on reliability of your network
19:54:33 <ttx> russellb: it's just that... I suspect we would set up the phone only in sessions where we know someone is missing
19:54:34 <russellb> (of course)
19:54:58 <jeblair> ttx: if we're doing this with asterisk, we might be able to scale out listening to a large number of listeners, and allow them to listen in-browser, etc.
19:55:08 <russellb> also ip would be way cheaper.
19:55:19 <jeblair> (so lurkers aren't using up telephone resources)
19:55:19 <reed> ok, sounds good
19:55:23 <ttx> jeblair: that means setting up the phone call at every session
19:55:41 <ttx> ..or you would just keep it running ?
19:56:02 <reed> how do you suggest we test this?
19:56:17 * ttx remembers reed running between rooms to check that the Webex stuff was still up
19:56:22 <clarkb> we could try running this meeting with the system one day
19:56:34 <fungi> i think it would just be a conference phone dialed into the bridge number for that room as the call moderator, with written instructions on how to redial if it's disconnected
19:56:40 <reed> not exactly the way I intended...
19:57:25 <reed> so to summarise
19:57:45 <reed> you're suggestion Asterisk server/SIP and phones in each room
19:58:00 <jeblair> russellb: would you be willing to help?
19:58:04 <russellb> yes
19:58:14 <jeblair> i intend to ask pabelanger too when he's around.  :)
19:58:22 <fungi> now that's half a dozen international calls from hong kong to wherever our bridge number is, connected 8 hours a day for 5 days, more or less. not sure what the expense winds up looking like for that
19:58:24 <reed> I'm still not convinced this is the best way to proceed
19:58:43 <russellb> fungi: via IP though, free
19:58:48 <russellb> hopefully
19:58:49 <jeblair> fungi: that does make trying to use SIP compelling.
19:58:49 <reed> but I do note the convenience of good old telephones
19:58:52 <fungi> unless we do voip over the conference network yes
19:59:17 <russellb> and ... need to check regulatory issues
19:59:20 <russellb> some countries block voip :(
19:59:25 <fungi> i note the stability of good old telephones vs voip across the world from a conference
20:00:12 <jeblair> we should probably found out how much pots calls would cost from the facility
20:00:15 <ttx> russellb: but this is Hong-Kong... oh wait.
20:00:18 <fungi> voip participants make sense, but for the conference room an actual phone line per room would likely be less finicky
20:00:31 <jeblair> okay, we're at time...
20:00:43 <reed> I think we can keep discussing this next week, I'll start a thread on the list, probably
20:00:43 <jeblair> ttx: is there a meeting in here now?
20:00:49 <ttx> jeblair: no.
20:00:52 <reed> the TC...
20:01:05 <ttx> you can overrun... a bit.
20:01:05 <russellb> canceled today
20:01:08 <ttx> No TC meeting this week
20:01:21 <fungi> want me to bounce the bot real quick between meetings? i can do it after we wrap this one up i guess
20:01:22 <jeblair> reed: yeah, let's do that; we'll resume next week
20:01:45 <reed> yeah, I think we wont' go too far anyway and we have time :)
20:01:47 <jeblair> reed: and try to get people who are willing to help here
20:01:56 <reed> yep, will do
20:02:01 <reed> thanks everybody
20:02:11 <jeblair> i'm certainly willing to arrive at the summit early and help set up
20:02:18 <fungi> as am i
20:02:44 <reed> ok, gotta go
20:02:47 <jeblair> ok, thanks
20:02:56 <jeblair> thanks everyone, see you next week
20:03:02 <jeblair> #endmeeting