15:00:02 <flaper87> #startmeeting Zaqar
15:00:03 <openstack> Meeting started Mon Dec  1 15:00:02 2014 UTC and is due to finish in 60 minutes.  The chair is flaper87. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:04 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:00:06 <openstack> The meeting name has been set to 'zaqar'
15:00:14 <flaper87> good morning world
15:00:19 <flaper87> #topic Roll Call
15:00:24 <flaper87> (o/
15:00:26 <kragniz> \o/
15:00:28 <flaper87> (o)
15:00:30 <flaper87> \o)
15:00:35 <flaper87> kragniz: you ruined my dance
15:00:47 <kragniz> flaper87: sorry, man
15:00:52 <kragniz> \o\
15:00:54 <kragniz> /o/
15:01:08 <flaper87> kragniz: are you skiing ?
15:01:13 <flaper87> that's not dancing
15:01:15 <flaper87> :P
15:01:20 <flaper87> vkmc: ?
15:01:24 <vkmc> o/
15:01:35 <flaper87> I guess kgriffs will join in a bit
15:01:43 <flaper87> aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaanyway
15:01:49 <flaper87> #topic Agenda
15:01:56 <flaper87> #link https://wiki.openstack.org/wiki/Meetings/Zaqar#Agenda
15:02:01 <flaper87> that's what we have for today
15:02:14 <flaper87> We still have 2 more specs to review and we're dong with that for a bit
15:02:35 <flaper87> #topic Deprecate sqlalchemy for the data plane
15:02:40 <flaper87> #link https://review.openstack.org/#/c/134248/
15:02:54 <flaper87> Any initial comments on that?
15:03:10 <flaper87> The TL;DR is that we were planning to get rid of sqlalchemy
15:03:16 <flaper87> at least on the data plane
15:03:30 <flaper87> That means you would still be able to use it to store pools, flavors etc
15:03:34 <flaper87> but not to store messages
15:03:57 <vkmc> how hard would be to update it and maintain it in the codebase?
15:04:03 <flaper87> However, after some extra consideration, we (or was it just me?) also mentioned that we should just get rid of it entirely
15:04:30 <vkmc> community has shown interest in sql backends
15:04:46 <flaper87> We could keep it outside as an external driver
15:04:55 <vkmc> that sounds good
15:05:02 <flaper87> In stackforge
15:05:07 <vkmc> we know that for Zaqar's use case sql is no good
15:05:09 <flaper87> it'd have gates and whatnot
15:05:16 <vkmc> well, maybe we have to look into that again and look for other sql alternatives
15:05:29 <vkmc> I'm speaking without certainty
15:05:39 <flaper87> vkmc: that may be an overstatement. Depending on how it's implemented and configured, rdbms could also work very well
15:06:03 <flaper87> however, as of now, the implementation is not good, not optimized and it's being quite a pain to maintain
15:06:10 <vkmc> yeah, that's why I'm saying that we might need to look into that again
15:06:17 <vkmc> all right
15:06:18 <vkmc> another concern
15:06:25 <flaper87> Shipping something that works with things like mysql but doesn't work well is not good for the project reputation
15:06:25 <vkmc> we use sqlite for testing... a lot
15:06:35 <flaper87> right
15:06:36 <vkmc> removing it would make testing heavier
15:06:41 <flaper87> right
15:07:07 <vkmc> so... we need something in between maybe
15:07:23 <flaper87> Do we?
15:07:34 <flaper87> I mean, is it really that bad to ask devs to have mongodb installed ?
15:07:41 <flaper87> or redis
15:07:50 <vkmc> I don't think it is
15:07:53 <vkmc> but
15:07:59 <flaper87> We used to have an sqlite only driver but again, we have to maintain it
15:08:18 <vkmc> I'm not an operator
15:08:27 <vkmc> we should ask this on the list and try to get feedback
15:08:41 <vkmc> if you didn't do that already :)
15:08:43 <flaper87> wait, this is devs we're talking about
15:08:59 <flaper87> I really hope there's noone deploying Zaqar + sqla
15:09:10 <vkmc> yeah me too
15:09:17 <vkmc> well... in the case of devs...
15:09:29 <flaper87> we gave it a full cycle for deprecation
15:09:35 <flaper87> now it's time to remove it
15:09:39 <vkmc> I dunno... if you are deploying a cloud I could assume you won't have trouble deploying mongo
15:10:01 <vkmc> right...?
15:10:13 <flaper87> As far as tests go, I'm just concerned about devs installing mongodb somewhere
15:10:18 <flaper87> to test this whole thing
15:11:08 <flaper87> to be fair
15:11:30 <flaper87> we could also just keep the sqlalchemy code, rename it to sqlite, use it as is and allow just "sqlite" url's
15:12:00 <flaper87> That would keep testing easy
15:12:25 <flaper87> we can try to maintain that code just for sqlite
15:12:38 <vkmc> mh..
15:12:48 <flaper87> if someone is interested in making it work well with mysql and psql then fine
15:12:53 <vkmc> how that is different from keeping sqlalchemy?
15:13:47 <flaper87> we'd just keep it as-is, rename the package to sqlite (or not), forbid url's for things that are not sqlite
15:14:40 <vkmc> I like that approach
15:14:57 <flaper87> what about the renaming?
15:15:12 <vkmc> nobody will use sqlite for production and its default in most distros
15:15:27 <vkmc> let's adjust sqlalchemy to be sqlite
15:15:28 <flaper87> renaming it would clear any doubt
15:15:31 <flaper87> ok
15:15:38 <vkmc> for tests we need sqlite
15:15:48 <flaper87> If someone wants to copy that code, make it better, etc. Then, be my guest
15:15:55 <vkmc> neat
15:16:00 <vkmc> we should submit a blueprint for that
15:16:02 <vkmc> if its not already there
15:16:10 <flaper87> There's a spec but I now have to update it
15:16:21 <flaper87> #link https://review.openstack.org/#/c/134248/
15:16:22 <vkmc> <3
15:16:36 <flaper87> ok, moving on
15:16:45 <sriram> just a thought here, have you looked at mocking frameworks?
15:16:54 <sriram> mongomock, fakeredis.
15:17:03 <flaper87> sriram: erm, nope
15:17:07 <sriram> that could remove dependency on sqlite for tests.
15:17:08 <flaper87> sriram: sounds interesting
15:17:12 <flaper87> sriram: do you have a link ?
15:17:18 <sriram> sure gimme one sec.
15:17:32 <vkmc> sriram, that sounds cool
15:17:33 <sriram> https://github.com/jamesls/fakeredis
15:17:49 <sriram> https://github.com/vmalloc/mongomock
15:17:59 <sriram> obviously there could be better versions out there.
15:18:06 <sriram> these are ones I know of.
15:18:22 <vkmc> neat!
15:18:24 <flaper87> awesome, I'll take a look and update the spec based on that
15:18:28 <flaper87> thanks, sriram
15:18:33 <vkmc> :D thanks sriram
15:18:34 <sriram> glad I can help :)
15:18:47 <flaper87> #topic Deprecate queues in favor of topics
15:18:51 <flaper87> #link https://review.openstack.org/#/c/134015/
15:19:10 <flaper87> So, no need to decide that right now, that will likely slip to k-2 anyway
15:19:25 <flaper87> but please, lets take a better look at that spec and review it thoroughly
15:19:33 <flaper87> it's quite a change and we need to be sure about it
15:20:11 <flaper87> there were interesting comments w.r.t getting messages from the topic
15:20:17 <flaper87> and how the URL should be formatted
15:21:07 <flaper87> comments?
15:21:08 <flaper87> thoughts?
15:21:11 <flaper87> worries?
15:21:15 <flaper87> gummy bears?
15:21:26 <flaper87> hugs ?
15:21:39 * flaper87 is talking to himself
15:21:45 * vkmc checks the spec
15:21:52 <vkmc> gummy bears and hugs pls
15:22:17 <flaper87> vkmc: awesome
15:22:34 <flaper87> no need to go throught right now, Lets have a more detailed discussion about it next week
15:22:40 <vkmc> so... well, I agree that we might no need to change the url
15:23:51 <flaper87> ok, moving on
15:23:52 <vkmc> as Gordon mentioned, it doesn't mean that we are dealing with containers (queues/topics) as first-class citizens
15:23:57 * flaper87 stops
15:24:09 <vkmc> and it allows us to have a more intuitive API
15:24:14 <flaper87> right, but it does have a meaning for REST Apis
15:24:20 <flaper87> it means there's a "resource"
15:24:26 <vkmc> oh... true that
15:24:31 <flaper87> the difference is that it'll be a virtual resource
15:25:21 <vkmc> does REST specify how virtual resources should be accessed?
15:25:46 <flaper87> AFAIK, it says nothing about virtual resources
15:25:48 <flaper87> I just made that up
15:25:51 <flaper87> :P
15:25:59 <vkmc> lol
15:26:06 <vkmc> you should totally propose the term
15:26:11 <vkmc> and add some specs for it
15:26:14 * flaper87 for president
15:26:19 <vkmc> haha
15:26:45 <vkmc> ok, let's discuss about this later if you please
15:26:54 <vkmc> I'm really sorry that we didn't get feedback in the mailing list
15:26:59 <flaper87> sure thing
15:27:00 <flaper87> yeah
15:27:02 <vkmc> that would make things easier
15:28:01 <flaper87> ok
15:28:03 <flaper87> moving on
15:28:15 <flaper87> #topic Do we need message_data in config files?
15:28:38 <flaper87> So, exploreshafali is working on splitting the control and the data planes
15:28:54 <flaper87> That means we'll have 2 new sections in our config files
15:28:59 <flaper87> message_plane and data_plane
15:29:11 <flaper87> Those sections will be used to configure the database for each plane
15:29:36 <flaper87> There are 2 questions:
15:29:52 <flaper87> 1) Do we want to keep supporting non-pooled Zaqar deployments ?
15:30:46 <flaper87> 2) Assuming pools are enabled, can we just forbid using Zaqar in non-pooled mode?
15:31:12 <flaper87> I mean, if someone enables pools, we currently allow them to use it with and without pools
15:31:15 <flaper87> (AFAIR :P)
15:31:29 <vkmc> yeah
15:31:30 <flaper87> Which creates the whole problem with both planes being together
15:32:03 <flaper87> What if instead of splitting them, we just forbid people to use Zaqar in a non-pooled way when pools are enabled
15:32:38 <vkmc> I think that would be a better solution
15:32:54 <flaper87> I think she's going to kill me
15:32:56 <flaper87> you tell her
15:32:58 <flaper87> :P
15:33:05 <vkmc> haha
15:33:19 <vkmc> we can get another thing for her
15:33:31 <flaper87> ok, I'll run this through Kurt as well before we make a final call
15:33:39 <flaper87> I'd also like to get her feedback on this
15:33:44 <vkmc> sounds good
15:34:03 <vkmc> IMO we shouldn't work to fix users negligence
15:34:12 <vkmc> but constrain it
15:34:36 <flaper87> yeah
15:34:38 <vkmc> splitting planes would take more work than forbbiding 0 pools in a pooled deployment
15:34:47 <vkmc> for that, you have the non-pooled deployment
15:34:48 <flaper87> also, there's the whole deprecation thing
15:34:52 <flaper87> and more sections
15:34:53 <vkmc> yeah
15:34:57 <flaper87> and config duplication
15:34:59 <flaper87> ok
15:35:05 <vkmc> more trouble
15:35:09 <vkmc> more manteinance
15:35:09 <vkmc> more code
15:35:18 * vkmc -> moar!
15:35:19 <flaper87> Yeah, we don't want more trouble... we already have you
15:35:39 <vkmc> that is my mission here in Zaqar
15:35:51 <flaper87> trouble-maker
15:36:01 <flaper87> ok, that's all I had
15:36:01 <vkmc> haha
15:36:18 <flaper87> I guess we can jump into open discussions or just close the discussion for today
15:36:19 <flaper87> :P
15:36:26 <vkmc> sure :)
15:36:30 <flaper87> it's not like we won't keep talking in #os-zaqar anyway
15:36:34 <vkmc> we also have the osprofiler spec
15:36:40 <vkmc> oh wait
15:36:42 <vkmc> wait...
15:36:45 <flaper87> vkmc: do we? I didn't see it
15:36:51 <flaper87> kgriffs: just joined
15:36:53 <vkmc> kgriffs has joined
15:36:54 <flaper87> miracles happen
15:36:54 <vkmc> hahahaha
15:37:05 <kgriffs> lol, sorry had a meeting conflict
15:37:18 <flaper87> yeah yeah, that's what everyone says
15:37:35 <vkmc> https://review.openstack.org/#/c/135612/
15:37:36 <flaper87> kgriffs: we went through some specs but we deferred the discussion 'til next week
15:37:42 <flaper87> we need to provide more reviews there
15:38:06 <flaper87> that said, we were now talking about whether we really need to split control and data plane
15:38:18 <kgriffs> oic
15:38:33 <flaper87> kgriffs: TL;DR: If pooling is enabled, can't we just forbid users to use zaqar in a non-pooled way?
15:38:48 <flaper87> ah wait
15:39:05 <flaper87> no yeah, that
15:39:07 <flaper87> :P
15:39:13 <flaper87> sorry, had one of those moments
15:39:23 <kgriffs> mmm
15:39:42 <flaper87> that way we can still keep it under the same config section, avoid duplicating options
15:39:47 <flaper87> and deprecating the ones we have
15:40:01 <flaper87> fewer docs to write, etc
15:40:12 <flaper87> The "etc" is the most important bit there
15:40:14 <kgriffs> so
15:40:14 <flaper87> :D
15:40:26 <kgriffs> is this different from having "always on pools"
15:40:35 <flaper87> That was the other question
15:40:43 <flaper87> but yes, it's different
15:40:45 <kgriffs> ok
15:40:55 <vkmc> that other question.. I would say no
15:41:01 <flaper87> It'd, however, make the transition to "always pools" easier
15:41:09 <flaper87> (if we ever decide to do so)
15:41:46 <kgriffs> let me see if I understand
15:42:06 <kgriffs> I thought that when you enable pooling, you already can't do anything "non-pooled"
15:42:12 <kgriffs> like, it is all-or-nothing
15:42:38 <vkmc> kgriffs, you can enable pooling and don't have pools
15:43:21 <kgriffs> when you enable pooling, doesn't that cause the pooled driver to get loaded in between every interaction, so you have to have pools then?
15:43:43 <kgriffs> (technically, the proxy controllers that the pool driver exposes)
15:43:49 <flaper87> kgriffs: yes, but I think you can still create a queue and post messages
15:44:01 * vkmc tries it
15:44:05 <flaper87> if the queue doesn't exist, I believe it just uses the default config
15:44:10 <flaper87> OMG, we should know this
15:44:39 <kgriffs> before flavors, it would just shard the queues across all available pools
15:45:01 <flaper87> but if there were no pools it'd just use the same configs
15:45:04 <flaper87> AFAIR
15:45:12 <kgriffs> you create a queue and it would hit the proxy queue controller. that would add the queue to the catalog so it is mapped to a pool
15:45:30 <kgriffs> flaper87: I thought it would just raise an error if there were no pools to choose from
15:45:41 <flaper87> mmh, damn, now I'm confused
15:45:43 <kgriffs> or perhaps that is what I would have expected, if it isn't the reality
15:45:50 <flaper87> kgriffs: same here
15:45:55 <kgriffs> in my mind, pools were always all-or-nothing
15:46:01 <flaper87> if that's the case, then why are we doing this in the first place?
15:46:08 <flaper87> I mean, splitting both planes
15:46:20 <flaper87> I think I just lost track of the motivation behind this work
15:46:30 <kgriffs> let me see
15:46:34 <flaper87> actually, the benefits
15:47:04 <kgriffs> I think we discussed that an operator may not want to use the same data store type for control vs. data
15:47:12 <kgriffs> or they may want to use different settings
15:47:26 <kgriffs> like have more strict/safe replication settings in the connection settings
15:47:31 <kgriffs> (for control)
15:47:43 <flaper87> right but pools are configured through the API
15:47:57 <flaper87> When you create a pool, you have to specify the connection URI
15:48:16 <kgriffs> good point
15:48:17 <flaper87> it wouldn't use the same store unless you tell it to do so
15:48:39 <kgriffs> so then you should still be able to mix-and-match
15:48:53 <flaper87> right
15:48:55 <kgriffs> but really, I recall something about the original motivation
15:49:14 <kgriffs> being more architectural - it was messy to glom the two drivers together in the same bucket
15:49:23 <flaper87> ok
15:49:32 <kgriffs> also
15:50:06 <kgriffs> it is confusing to have the same config section be used for the catalog/control when pooling is enabled, but the data store when pooling is not enabled.
15:50:12 <kgriffs> that being said
15:50:14 <vkmc> btw, it raises an error
15:50:23 <flaper87> vkmc: good
15:50:23 <kgriffs> perhaps we are taking a hatchet to this when we really need a scalpel
15:50:30 <vkmc> http://paste.openstack.org/show/142676/
15:50:45 <flaper87> kgriffs: indeed, I started thinking about that earlier today
15:51:22 <kgriffs> i think the root of the problem is the dual-mode
15:51:36 <kgriffs> if you do not use pooling, yeah then having a couple config sections is necessary
15:51:44 <kgriffs> (in order to mix-and-match)
15:52:06 <flaper87> so, having some nodes non-pooled and others pooled
15:52:25 <kgriffs> if you do have pooling, a single config section (as we have today) is groovy. We may still want to move control stuff into its own module inside each driver directory, but that is just refactoring
15:52:41 <kgriffs> flaper87: no, I mean when non-pooled
15:52:58 <kgriffs> I may want to use different driver settings for my catalog vs. my message data
15:53:12 <flaper87> kgriffs: why if you're in a non-pooled mode?
15:53:22 <flaper87> AH WAIT
15:53:25 <flaper87> I remember the real motivation
15:53:38 <kgriffs> red bull?
15:53:39 <flaper87> we wanted to move queue's out of the data plane
15:54:03 <kgriffs> oh, and put their info somewhere else?
15:54:08 <flaper87> therefore we needed a way to split both planes even in a non-pooled mode
15:54:11 <flaper87> yeah
15:54:23 <kgriffs> yeah, I you are right. that was another reason
15:54:27 <kgriffs> hmm
15:54:33 * flaper87 is right
15:54:43 * flaper87 won't give kgriffs gummy bears
15:54:46 <kgriffs> (happens sometimes on accident)
15:54:47 <kgriffs> ;)
15:54:50 <flaper87> :P
15:54:59 <flaper87> ok, we solved the mistery
15:55:02 <vkmc> ok
15:55:03 <vkmc> please
15:55:04 <vkmc> write it down
15:55:05 <kgriffs> ok, so here is the thing
15:55:06 <vkmc> somewhere
15:55:07 <flaper87> It was the majordomo
15:55:13 <flaper87> in the living room
15:55:16 <flaper87> with a knife
15:55:18 <flaper87> :P
15:55:27 <kgriffs> in an ideal world we would say pools are always on
15:55:28 <flaper87> That dude cut both planes into 2
15:55:36 <flaper87> kgriffs: I kinda agree
15:55:41 <kgriffs> then you always have that config section mean the same thing
15:55:55 <kgriffs> and it already gives you a way to use different data stores
15:56:04 <flaper87> with just 1 config section
15:56:05 <kgriffs> so if you want that, please use pools
15:56:06 <flaper87> :)
15:56:23 <flaper87> oh no wait
15:56:28 <flaper87> we would still want to have 2
15:56:31 <flaper87> probably
15:56:34 <flaper87> to have a default one
15:56:40 <flaper87> no idea
15:56:42 <flaper87> that'd be confusing
15:56:57 <vkmc> 2'
15:57:04 <kgriffs> we are almost out of time, but I don't think there is another meeting after us is there?
15:57:05 <flaper87> vkmc: SSSSSSSSSSSHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH
15:57:13 <flaper87> kgriffs: I've oslo's
15:57:22 <flaper87> we should discuss this further
15:57:26 <flaper87> in the channel, perhaps
15:57:29 <kgriffs> yes
15:57:35 <flaper87> rock on
15:57:42 <flaper87> thanks for all those thoughts
15:57:58 <krotscheck> Yep. Stoyrboard is here in 3
15:58:05 <flaper87> krotscheck: 2
15:58:07 <flaper87> :P
15:58:10 <flaper87> ok, we're out!
15:58:15 <vkmc> thanks guys o/
15:58:16 <flaper87> Thank y'all
15:58:20 <flaper87> #endmeeting