19:14:20 <kgriffs> #startmeeting marconi
19:14:21 <openstack> Meeting started Thu Jun  6 19:14:20 2013 UTC.  The chair is kgriffs. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:14:22 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:14:24 <openstack> The meeting name has been set to 'marconi'
19:15:22 <kgriffs> #link http://youtu.be/sDUw1Je_JJE
19:16:00 <flaper87> I'm in but I can't hear anything
19:16:17 <kgriffs> it said you left
19:16:26 <kgriffs> https://plus.google.com/hangouts/_/43e14b608e33ae127ec6ab4852f0128ae0d8a878
19:16:30 <kgriffs> try again?
19:16:33 <ametts> Flavio Percoco Premoli joined group chat.
19:16:33 <ametts> Flavio Percoco Premoli left group chat.
19:16:33 <ametts> Flavio Percoco Premoli joined group chat.
19:16:34 <ametts> Flavio Percoco Premoli left group chat.
19:17:00 <kgriffs> you were there for a sec
19:17:02 <flaper87> it throws some error on the video call
19:17:11 <kgriffs> can you do sound only?
19:17:37 <kgriffs> ok, give up?
19:18:11 <kgriffs> #topic audio meetings
19:18:44 <kgriffs> #action flaper87 to fix his mic
19:18:47 <kgriffs> :D
19:18:47 <flaper87> sorry guys, this stupid update doesn't seem to work correctly
19:18:50 <flaper87> LOOOL
19:19:09 <kgriffs> #topic H1 milestone closure
19:19:15 <flaper87> I think I'm in
19:19:56 <flaper87> kgriffs: saw you
19:19:58 <kgriffs> ok, let's try again another time - i think it had something to do with the "on air"
19:20:15 <kgriffs> so, re H1
19:20:18 <flaper87> kk
19:20:20 <kgriffs> it is now behind us
19:20:51 <kgriffs> so, we need to update the bp's and trello cards
19:20:57 <flaper87> kk
19:21:42 <kgriffs> we made good progress on all the stuff we planned, but just didn't bring them all to closure
19:22:27 <kgriffs> so, I can just move back everything, but wanted to ask if there was anything we had scoped for H1 that turned out to be lower priority and should be moved maybe to H3 instead of H2
19:22:45 <kgriffs> https://trello.com/board/openstack-marconi/511403287d138cd6200078e0
19:23:13 * flaper87 checking
19:23:23 <kgriffs> for example, "validate input" I think we need before launch, but not immediate
19:23:30 <kgriffs> (look at the "next up")
19:23:52 <kgriffs> I'm wondering if we need that "next up" list, or just merge with the milestone lists?
19:24:03 <flaper87> kgriffs: +1 for merge
19:24:10 <kgriffs> any objections?
19:24:14 <flaper87> the more focus in 1 column we can be the better
19:24:25 <flaper87> focused*
19:25:15 <kgriffs> #agreed Merge "next up" list with milestone list
19:26:22 <kgriffs> also, I'd like to add a Havana-3.a
19:26:32 <flaper87> kgriffs: +1
19:26:49 <kgriffs> Basically, I would like something to target for "pretty much production ready"
19:27:01 <kgriffs> H3 will be for stabilization and whatever
19:27:52 <flaper87> makes sense
19:28:06 <flaper87> there's a limit for introducing new features
19:28:13 <kgriffs> OK, let me do this. I will take a first pass and moving the cards around, and you guys can complain loudly in #openstack-marconi if you don't like what you see.
19:28:15 <flaper87> IIRC it is september
19:28:24 <flaper87> ok
19:28:29 <flaper87> LIKE THIS ?
19:28:57 <kgriffs> that's about right. :D
19:29:06 <ametts> Is some of the Havana-2 stuff optional?  We really going to add WebScoket, qpid, Kombu, etc?
19:29:25 <kgriffs> #action kgriffs to clean up trello board, get things prioritized better
19:29:38 <flaper87> ametts: I read those as "Nice to Have"
19:29:41 <flaper87> TBH
19:29:53 <ametts> Me too.
19:30:15 <kgriffs> so, re incubation, can we put all the alternate drivers in September-ish timeframe?
19:30:18 <flaper87> but we should definitely target them and focus on the column as "Must Have"
19:30:38 <flaper87> kgriffs: sure
19:30:45 <kgriffs> OK
19:31:27 <kgriffs> #topic bugs
19:31:47 <kgriffs> So, just wanted to get updates on bug fixes that are in progress
19:32:26 <malini> before that, can we also define a time line for the marconi client ?
19:32:40 <flaper87> kgriffs: I was working on Autoreconnect
19:32:49 <kgriffs> malini: let's do that next topic
19:32:51 <flaper87> that discussion is still pending
19:32:53 <kgriffs> #link https://bugs.launchpad.net/marconi/+bug/1169821
19:32:56 <kgriffs> OK
19:33:13 <kgriffs> so, let's talk about that for a couple mins
19:33:17 <flaper87> so, It basically works but it could be improved before letting it land
19:33:31 <flaper87> cool
19:33:56 <kgriffs> ok, so your wrapper is only for reads, correct?
19:34:06 * kgriffs is looking
19:34:16 <flaper87> nope
19:34:25 <flaper87> it is for everything, basically
19:34:32 <flaper87> the big problem are reads, though
19:34:47 <flaper87> the reason for that is that reads happen in the Cursor
19:35:00 <kgriffs> oh, is there any danger of getting part way through a bulk insert and getting a disconnect error?
19:35:29 <kgriffs> (ConnectionFailure)
19:35:32 <flaper87> Autoreconnect should be raised before sending the bulk
19:35:40 <flaper87> so, it would just retry
19:35:40 <kgriffs> #link https://review.openstack.org/#/c/30230/2/marconi/storage/mongodb/utils.py
19:36:16 <kgriffs> if the socket connection simply get's dropped, what happens?
19:36:47 <flaper87> pymongo tries to connect without raising any error
19:36:57 <flaper87> it raises an error when that attempt of connection fails
19:37:04 <kgriffs> even halfway through a write?
19:37:08 <flaper87> ah, you mean in the middle of the operation
19:37:19 <flaper87> erm, well...
19:37:41 <flaper87> we could remove the retry stuff
19:37:59 <kgriffs> basically, I just want to be sure we aren't doing a generic retry halfway through an insert, since the caller needs to reset the generator
19:39:02 <flaper87> I'm honestly not sure whether we should be handling it at all. I mean, Marconi doesn't know what to do when AutoReconnect happens
19:39:09 <flaper87> right? It just knows there was an error
19:39:22 <flaper87> should we just return an error in the API?
19:39:34 <flaper87> that question is valid even if we handle the Autoreconnect
19:39:49 <flaper87> since mongodb might be completely innaccesible
19:39:58 <flaper87> (dunno if that word exists)
19:40:22 <flaper87> The other thing is that Cursors are lazy
19:40:34 <kgriffs> well, the trouble is, in the case of a bulk message post, there is no way to know how many messages succeeded if we drop the connection halfway through. I guess with Mongo there isn't much to do about it?
19:40:44 <flaper87> and none of the read operations will raise any error until they are truly consumed
19:40:48 <flaper87> and that happens in the transport side
19:40:55 <kgriffs> kk
19:41:21 <kgriffs> so, iterating that cursor could raise an AutoReconnect?
19:41:27 <flaper87> yup
19:41:38 <flaper87> so, crazy idea
19:41:48 <kgriffs> the plot thickens...
19:42:22 <flaper87> what if we have some kind of "abstract decorator" implemented in every storage that "knows" how to handle "storage specific errors" that might happen in the transport side?
19:42:47 <flaper87> or just possible exceptions
19:42:57 <flaper87> so that it is possible to do in the transport side something like
19:43:02 <kgriffs> ok, so the transport could ask the driver for that thing, maybe it could be a context manager?
19:43:16 <flaper87> try ... except self.storage.CatchThatStuff: ...
19:43:25 <kgriffs> yeah, ok
19:43:26 <flaper87> yeah, something like that
19:43:28 <kgriffs> something like that could work
19:43:40 <flaper87> oke-doke
19:43:41 <kgriffs> ok, so we have two things to solve here
19:43:46 <kgriffs> first, reads
19:43:49 <kgriffs> second, writes
19:43:55 <malini> maybe we are getting too focussed on a single bug ? *brings out her irritating cow bell*
19:44:09 <ametts> Rock on, malini!
19:44:10 <kgriffs> do we need to experiment to see if AutoReconnect errors are possible halfway through a bulk insert?
19:44:31 <kgriffs> heh
19:44:31 <flaper87> I think they are
19:44:44 <kgriffs> OK, let's move on, but sounds like there's some homework
19:44:45 <flaper87> at least a ConnectionFailure could be raised
19:44:57 <ametts> And hey, if everything works besides reads and writes, we're good... no?
19:44:59 <flaper87> ok, I'll re-write that patch along these lines
19:45:08 <kgriffs> OK, can u verify what happens with bulk inserts?
19:45:14 <flaper87> kgriffs: will do
19:45:18 <kgriffs> (simulate a primary failover, see what happens)
19:45:24 <kgriffs> excellent
19:45:25 <flaper87> ametts: as far as deletes work, we're fine
19:45:31 <flaper87> :P
19:45:38 <ametts> Sounds like an awesome queue. :)
19:45:52 <kgriffs> #action flaper87 to experiment with bulk insert behavior on mongo connection errors
19:45:55 * flaper87 STFU
19:46:53 <kgriffs> #action flaper87 to update AutoReconnect patch to handle read cursor errors
19:47:17 <kgriffs> ok, I know this bug is also in progress, any quick thoughts before we move off the bugs topic?
19:47:20 <kgriffs> #link https://bugs.launchpad.net/marconi/+bug/1187280
19:47:52 <flaper87> patch waiting for review
19:48:19 <kgriffs> kk, I think we can land this one pretty quickly. Looks like Jenkins is happy again
19:48:26 <kgriffs> Can everyone re-review?
19:48:29 * flaper87 hates eventlet
19:48:44 <ametts> FWIW, there's a fair amount of overlap between several of these bugs, input validation, and failing system tests.
19:49:00 <flaper87> marconi/tests/common/__init__.py
19:49:02 <flaper87> renamed from lib/marconi_paste.py
19:49:04 <flaper87> WTF?
19:49:08 <flaper87> Jenkins is crazy
19:49:08 <kgriffs> yes, the remaining bugs are mostly due to holes in the implementation re the API spec
19:49:14 <kgriffs> heh
19:49:28 <kgriffs> #topic blueprints
19:49:34 <flaper87> ametts: they exist just to make us beleive we're going faster
19:49:39 <kgriffs> QA cluster news?
19:49:40 <flaper87> we can close 2 bugs with 1 patch
19:49:45 * flaper87 STFU
19:49:51 <flaper87> I know, I've said it before
19:49:57 <ametts> flaper87: LOL
19:50:01 * kgriffs didn't hear anything
19:50:30 <malini> We havent made much progress on the QA cluster since last time
19:50:38 * kgriffs sad panda
19:50:47 <kgriffs> any blocking issues?
19:51:16 <malini> its just that oz_akan had other priorities this week
19:51:45 <kgriffs> kk, will he be back on this next week?
19:51:53 <malini> yes
19:52:07 <kgriffs> kk, keep us posted
19:52:07 <ametts> QA cluster runs system tests with every commit?
19:52:27 <malini> its not hooked into jenkins yet
19:52:33 <malini> But we'll be there soon
19:52:37 <kgriffs> +1
19:52:43 <kgriffs> how about performance tests?
19:53:00 <malini> I have the xmls (i.e defining scenarios/loads etc.) ready
19:53:16 <malini> So we are ready to run them when we have the cluster
19:53:25 <kgriffs> excellent
19:53:28 <kgriffs> nice work
19:53:34 <malini> Also, can I move those xmls to the marconi repo ?
19:53:50 <malini> under tests/performance ?
19:53:52 <kgriffs> and that will be triggered by patch submissions as well?
19:53:54 <malini> yes
19:54:07 <kgriffs> flaper87: thoughts on where those should live?
19:54:26 * ametts Wants to make sure we save time for a quick marconiclient discussion
19:54:49 <flaper87> mmhh, not right now, not sure if those should go into the tree
19:55:03 <kgriffs> ok, let's discuss in #openstack-marconi
19:55:08 <malini> ok
19:55:08 <flaper87> kk
19:55:17 <kgriffs> #topic python-marconiclient
19:55:36 <kgriffs> next steps, timing?
19:55:41 <malini> we are stalled on the client because of Alessio's patch.
19:55:55 <ametts> Just noticed that stackforge/python-marconiclient doesn't have jdprax's original stuff  in it, but it does have Alessio's patch pending.
19:56:10 <ametts> What's the plan -- we start with the patch and fold in jdprax's original stuff?
19:56:48 <flaper87> ametts: so, the plan was to give alessio's code a try, and help improving that code
19:57:11 <flaper87> The code proposed there is meant to be a shared client library that will, eventually, land in oslo
19:57:18 <flaper87> but, it needs a lot of work and cleaning
19:57:31 <kgriffs> can malini help with that work and cleaning?
19:58:01 <ametts> Ok -- so that forms the base and it gets called by other stuff for marconi-specific things?
19:58:02 <malini> I emailed Alessio & at this point he badly needs reviews to merge his stuff into oslo-incubator
19:58:36 <kgriffs> wait - I thought he was merging this into python-marconiclient first?
19:58:42 <ametts> Is there anything of value that gets reused from the jdprax stuff?
19:58:48 <flaper87> kgriffs: yup
19:58:54 <flaper87> first into marconiclient
19:59:07 <flaper87> kgriffs: malini can help for sure
19:59:16 <malini> hmm..So we dont need the oslo incubator stuff ?
19:59:17 <flaper87> we should start reviewing that patch 'til the last bit
20:00:18 <malini> So if we can get https://review.openstack.org/#/c/29255 fixed, we shud be good to go ?
20:00:32 <kgriffs> (note: we are out of time, let's wrap up)
20:00:52 <flaper87> malini: sort of, yeah
20:01:06 <flaper87> we can keep discussing this stuff right now on #os-marconi
20:01:38 <kgriffs> #action malini and flaper87 to continue api client discussion elsewhere
20:01:58 <kgriffs> two parting thoughts
20:02:14 <kgriffs> malini, can you add a few very basic security tests, maybe just fuzzing?
20:02:22 <kgriffs> one possibility: http://gauntlt.org/
20:02:46 <malini> yes, I'll start looking at what we can add
20:02:48 <kgriffs> second thought, please review this RFC on the project: https://etherpad.openstack.org/queuing-scratch
20:02:58 <kgriffs> ok, thanks folks
20:03:15 <kgriffs> we will shift some agenda items to next week.
20:03:31 <kgriffs> #endmeeting