15:00:31 <kgriffs> #startmeeting marconi
15:00:32 <openstack> Meeting started Tue Mar 11 15:00:31 2014 UTC and is due to finish in 60 minutes.  The chair is kgriffs. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:33 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:00:35 <openstack> The meeting name has been set to 'marconi'
15:00:45 <flaper87> <o/ <o> \o>
15:00:48 <sriram> o/
15:00:50 <kgriffs> roll call
15:00:52 <kgriffs> o/
15:00:57 <malini> o/
15:01:04 <flaper87> flapercop is here o/
15:01:22 <flwang> o/
15:01:26 <alcabrera> o/
15:01:31 <balajiiyer> o/
15:01:59 <funzo> o/
15:02:03 <kgriffs> thanks everyone for coming!
15:02:06 <alcabrera> hurray
15:02:19 <kgriffs> hmmm... missing our interns
15:02:45 <kgriffs> well, let's get going
15:02:55 <kgriffs> #link https://wiki.openstack.org/wiki/Meetings/Marconi#Agenda
15:03:00 <kgriffs> #topic graduation prep
15:03:25 <kgriffs> so, we have docs and sqla to finalize, plus tempest
15:03:34 <malini> plus devstack issues
15:03:34 <kgriffs> anything else?
15:03:45 <flaper87> kgriffs: sqla is done
15:04:02 <flaper87> says flaper87 before finding a new bug in sqla
15:04:04 <alcabrera> lol
15:04:14 <kgriffs> flaper87: ok, I wasn't sure if you still had something cookin'
15:04:26 <flaper87> kgriffs: just testing it and fixing things as I find them
15:04:33 <flaper87> but looks like it's in good shape now
15:04:34 <kgriffs> have we tested it with pgres?
15:04:37 <flaper87> it runs on mysql, at least
15:04:46 <alcabrera> and sqlite
15:04:47 <flaper87> kgriffs: I knew you were going to ask that
15:04:50 <flaper87> :P
15:04:52 <flaper87> kgriffs: not yet
15:04:55 <flaper87> that's the next one
15:04:57 <alcabrera> cool
15:05:01 <balajiiyer> kgriffs: pecan eval is not part of graduation, but based on the discussion yday Im going to redo benchmarks. Might need a day or two to do this right.
15:05:21 <alcabrera> #note flaper87 has tested sqlalchemy on mysql and sqlite - looks good, needs postgres
15:05:26 <kgriffs> ok. well, we should do that. but graduation req. was we work on a non-AGPL db, which we do now (MySQL)
15:05:47 <kgriffs> balajiiyer: actually, it is part of graduation
15:05:54 <flaper87> pecan is part of graduation
15:05:54 <kgriffs> we agreed to do a serious Pecan eval
15:06:01 <kgriffs> sorry, I forgot to mention above
15:06:05 <flaper87> what is not part of the graduation requirement is to migrate to pecan
15:06:14 <kgriffs> right
15:06:18 <flaper87> but we have to have a good understanding of where we're standing
15:07:02 <kgriffs> flaper87: I understand that infra doesn't like to take on additional deps in the gate
15:07:13 <kgriffs> but, isn't Falcon already running in the gate with no adverse affects?
15:07:45 <flaper87> it's running in the gate
15:07:51 <flaper87> it's part of the global requirements
15:08:19 <balajiiyer> so with pecan eval, there is no patch reqd, so thats good. Im working on the benchmarks, took help from kgriffs on how to structure the email that will be sent out. Im drafting the email, and will send it out for review.
15:08:28 <kgriffs> some dependency must have caused a big problem in the past for them to be so cautious now, methinks
15:08:44 <kgriffs> #topic pecan eval
15:08:51 <flaper87> kgriffs: many dependencies, starting with sqlalchemy
15:09:16 <kgriffs> flaper87: after this mtg, can you elaborate?
15:09:27 * kgriffs is seeking first to understand
15:10:06 <kgriffs> #note balajiiyer working on benchmarks for Pecan eval, will take another day or two to finish that up and draft the report
15:10:11 <balajiiyer> alright, looks like I jumped the gun a bit
15:10:15 <flaper87> kgriffs: sure
15:10:42 <vkmc> o/
15:11:08 <alcabrera> vkmc: \o/!
15:11:19 <kgriffs> so, just so everyone knows, I just suggested a rough outline for the report, but have been trying to let balajiiyer work out the details
15:11:57 <kgriffs> balajiiyer: btw, one more criterion you might add is runtime support (Python 2.6-3.3, PyPy)
15:12:09 <kgriffs> I'm curious to know whether Pecan works on PyPy, for example
15:12:19 <balajiiyer> kgriffs: noted
15:12:22 <kgriffs> vkmc: o/
15:12:26 <kgriffs> balajiiyer: thanks
15:12:30 <alcabrera> I'd like to narrow that a bit: 2.6, 2.7, 3.3
15:12:35 <alcabrera> 3.2 is not necessary
15:12:39 <kgriffs> alcabrera: +1
15:12:41 <alcabrera> also, pypy
15:13:52 <alcabrera> so, regarding pecan eval
15:13:58 <kgriffs> so, I suggested that balajiiyer get some folks outside our team to review the draft before it goes out
15:14:00 <alcabrera> we've got some benchmarks and writing to do
15:14:36 <kgriffs> just to make sure it is as reasonable and as objective as possible
15:14:48 <alcabrera> sounds good
15:15:05 <alcabrera> I'm looking forward to the report. :)
15:15:12 <sriram> +1
15:15:32 <alcabrera> any other thoughts on pecan eval?
15:15:51 <kgriffs> balajiiyer is putting together a weighted decision matrix, and I'm helping him set up an autobench run so we can get some serious numbers
15:15:59 <flwang> maybe we can take a look at Ceilometer
15:16:01 <kgriffs> that's all I had
15:16:07 <flwang> since it's using Pecan + WSME
15:16:57 <kgriffs> it would be interesting to use WSME and redo the benchmarks, but that may have to wait for a "round #2"
15:17:02 <alcabrera> #note ceilometer uses Pecan + WSME
15:17:47 <kgriffs> #note would be nice to benchmark WSME some time
15:17:54 <kgriffs> #topic ATL summit
15:18:13 <kgriffs> just a couple reminders
15:18:28 <kgriffs> #1 design session proposals are now open - submit yours now!
15:18:38 <vkmc> I saw somewhere that Oslo working on building a common library for Pecan as well
15:18:49 <kgriffs> #2 Make your travel arrangements ASAP if you haven't already
15:19:11 <kgriffs> vkmc: at the last summit it was mentioned they wanted to put together common code into a shared lib
15:19:25 <flwang> kgriffs: what's the context for #2?
15:19:43 <kgriffs> travel to the summit
15:20:02 <kgriffs> I hope we can get as many marconi team members there as possible
15:20:03 <vkmc> kgriffs, Indeed, thanks
15:20:12 <flwang> kgriffs: seems there is a team dinner?
15:20:23 <flwang> kgriffs: got it
15:21:02 <alcabrera> flwang: there will be many team dinners, if we can help it. :D
15:21:04 <kgriffs> flwang: yes, evening of the 10th. We can also do another meetup during the week
15:21:32 <flwang> kgriffs: alcabrera: fabulous........................
15:21:43 <kgriffs> vkmc: I am a little hesitant to endorse the common Pecan lib approach - i would rather see us creating generic libs that can be dropped into any kind of hook method
15:22:02 <kgriffs> that would allow us to contribute upstream these libs to the broader Python community
15:22:16 * kgriffs doesn't like tight coupling if he can help it
15:22:29 <kgriffs> one more thing wrt the summit
15:22:33 <alcabrera> kgriffs: +1
15:22:39 <kgriffs> I was thinking of these themes for Juno:
15:22:47 <kgriffs> #1 Operational Maturity
15:22:50 <kgriffs> #2 Notifications
15:22:55 <kgriffs> #3 Flavors
15:23:19 <vkmc> kgriffs, Its reasonable, +1
15:23:23 <kgriffs> I think most of the stuff we've been talking about for Juno can fit in one of those
15:23:27 <flaper87> kgriffs: re-conceptualize marconi ? :D
15:23:30 <sriram> Sounds good.
15:23:31 <malini> Does Operational Maturity include new gatecheck jobs etc ?
15:23:31 <alcabrera> haha
15:23:34 <flaper87> s/queuing/messaging/
15:23:45 <kgriffs> malini: tests, security, logging, etc.
15:23:47 <flwang> flaper87: +1
15:23:48 <alcabrera> I'm favorable towards these themes.
15:23:58 <malini> We have a lot of infra work tht needs to be done
15:23:59 <flwang> flaper87: it's important, IMHO
15:24:04 <kgriffs> flaper87: we can rework #3 to fit in that stuff
15:24:23 <kgriffs> meaning, re-conceptualize
15:24:42 <flaper87> kgriffs: sounds good
15:24:52 <kgriffs> let me think on #3 - I think we can play around with that theme, broaden it a bit
15:24:56 <flwang> flaper87: pls involve me when you start to work on the topic/tag stuff
15:25:02 <kgriffs> #topic sqla driver GC
15:25:05 <flaper87> flwang: sure thing
15:25:24 <kgriffs> so, iirc we delete expired messages each time someone posts a new one?
15:25:37 <flaper87> kgriffs: yeah, we haven't changed that bit yet
15:25:44 <flaper87> but I think we may want to
15:25:59 <kgriffs> is there a plan to add an out-of-band GC?
15:26:18 <flaper87> kgriffs: not a concrete plan but I think we should
15:26:25 <kgriffs> flaper87: can you file a bug for that?
15:26:29 <flaper87> kgriffs: sure thing
15:26:49 <kgriffs> thanks!
15:27:11 <alcabrera> out of band GC would unify the operational semantics of sqla vs. mongodb
15:27:13 <kgriffs> #action flaper87 to add a bug for sqla GC
15:27:27 <alcabrera> so I like this, however we manage to solve it
15:27:45 <kgriffs> kewl
15:27:53 <kgriffs> #topic api v1.1 updates
15:28:08 <kgriffs> so, there are three proposals
15:28:19 <kgriffs> #1 lazy-create queues
15:28:46 <kgriffs> this is where we auto-create a queue when someone posts a message
15:29:00 <flaper87> #link https://bugs.launchpad.net/marconi/+bug/1290907
15:29:09 <flaper87> kgriffs: yeah
15:29:20 <kgriffs> I think if we do this, app developers will be less likely to think about lifecycle management for queues
15:29:23 <flaper87> so, we need to:
15:29:28 <flwang> kgriffs: will we introduce the topic/tag concept at that stage?
15:29:28 <flaper87> 1) Decide if what I proposed makes sense and vote
15:29:32 <flaper87> 2) Decide what's the migration plan towards that
15:29:35 <kgriffs> meaning, they will neglect deleting them as well
15:29:47 <kgriffs> flwang: nope, not until 2.0
15:30:12 <flaper87> FWIW, the plane we came up with yday still makes sense to me
15:30:29 <kgriffs> flaper87: ok, let's jump to your proposal and circle back
15:30:30 <flaper87> #link http://blog.flaper87.com/post/531cd585d987d24e83f082a5/
15:30:32 <alcabrera> so, it sounds like we might want to make queue auto-deletion configurable; a kind of queue TTL
15:30:33 <kgriffs> #topic queues -> topics
15:30:37 <flwang> flaper87: seems we didn't touch the migration plan a lot
15:30:45 <flaper87> flwang: we did
15:31:16 <flwang> after I dropped :)?
15:31:31 <flaper87> So, TL;DR of the proposal is that we don't need the queue to be a first-citizen resource nor it to be exposed to the client
15:31:43 <flaper87> users want to post messages and have a nice way to group them together
15:32:02 <flaper87> the concept of queue is getting old
15:32:15 <flaper87> and we could adopt topics as a way of grouping messages
15:32:33 <flaper87> the main difference between topics and queues is that queues are a resource on top of messages
15:32:42 <flaper87> whereas topics are attributes of the message
15:32:50 <alcabrera> what benefits do you see us reaping over time as a result of changing the way we address messaging, flaper87?
15:32:55 <flaper87> they don't require to be created beforehand
15:32:55 <flwang> flaper87: the 'migration' I'm saying is migrating a legacy Marconi deployment to new
15:33:01 <flwang> flaper87: are we on the same page?
15:33:15 <flaper87> flwang: hold on, we'll get there
15:33:25 <flwang> flaper87: ok
15:33:38 <flaper87> alcabrera: Usability is the main one
15:33:56 <flaper87> then we'll make creation of amqp 1.0 based drivers easier
15:34:14 <kgriffs> I would also say, it gives us more flexibility. For example, now we could have a message be associated with more than one topic
15:34:22 <flaper87> we'll reduce the space required since we don't need a separate storage for the queue
15:34:36 <flaper87> A table in sqla, a collection in mongodb
15:34:41 <alcabrera> cool
15:34:43 * alcabrera takes notes
15:34:45 <flaper87> kgriffs: right
15:34:45 <sriram> yes the space required for storage might reduce.
15:35:08 <flaper87> and it'll be easier to route messages based on topics
15:35:12 <alcabrera> #note +1 - queues as topics leads to usability benefits: easier to use marconi
15:35:18 <flaper87> ah also, we won't need to query the queues collection everytime
15:35:21 <flaper87> every time*
15:35:28 <flwang> flaper87: but we may need another table for sqla against topics/tags
15:35:37 <alcabrera> #note +1 - queues as topics leads to flexibility: message associated with multiple topics
15:35:37 <flaper87> (the query of the queue's collection depends on the driver)
15:35:46 <kgriffs> idk about the space - assuming no metadata, a normalized schema can save space. but that is assuming people always delete their queues when they are done with them. :p
15:35:53 <alcabrera> #note +1 - queues as topics leads to storage reduction: no concrete store allocated for topics
15:36:11 <kgriffs> anyway, adding a bit of text with topic name to each record is not a big deal
15:36:13 <flaper87> kgriffs: yeah but during the queue life, that space is required
15:36:28 <flwang> flaper87: right
15:36:32 <flaper87> anyway, those are the main ones
15:36:33 <sriram> Does the working of claims change in anyway due to this?
15:36:48 <flaper87> sriram: nope
15:36:55 <flaper87> sriram: claims work on messages
15:36:59 <alcabrera> I don't expect it to, since claim status is a property of a message
15:37:02 <alcabrera> what flaper87 said
15:37:05 <flaper87> sriram: so, we'll still be able to claim messages
15:37:10 <sriram> cool, was just making sure.
15:37:21 <kgriffs> alcabrera: topics are kind of like RSE channels
15:37:53 <flwang> yes, queues is like a 'container' for messages, but now we don't need it anymore
15:38:01 <kgriffs> ok, any other questions before we vote
15:38:01 <alcabrera> kgriffs: fair enough, though I admit I never studied RSE in depth. :)
15:38:07 <alcabrera> hmmm
15:38:12 <flwang> we can just give messages a topic/tag to distinguish it
15:38:33 <flwang> +1 from me
15:38:49 <sriram> yeah, it sounds good to me as well.
15:38:51 <flwang> for the lazy queue in v1.1
15:38:57 <kgriffs> #startvote Queues -> Topics in v2.0: yes, no
15:38:58 <openstack> Unable to parse vote topic and options.
15:39:02 <kgriffs> crap
15:39:05 <kgriffs> can't remember the syntax
15:39:08 <flaper87> #vote yes
15:39:35 <alcabrera> #startvote Should we vote now? Yes, No, Maybe
15:39:39 <kgriffs> #startvote Queues -> Topics in v2.0? yes, no
15:39:40 <openstack> Begin voting on: Queues -> Topics in v2.0? Valid vote options are yes, no.
15:39:41 <openstack> Vote using '#vote OPTION'. Only your last vote counts.
15:39:45 <flaper87> #vote yes
15:39:46 <alcabrera> #vote yes
15:39:48 <sriram> #vote yes
15:39:52 <kgriffs> #vote yes
15:40:03 <kgriffs> balajiiyer: ?
15:40:09 <balajiiyer> yes
15:40:11 <flaper87> malini: ?
15:40:14 <flwang> #vote yes
15:40:22 <malini> #vote abstain
15:40:23 <openstack> malini: abstain is not a valid option. Valid options are yes, no.
15:40:24 <kgriffs> balajiiyer: can you revote with the hash
15:40:33 <malini> :(
15:40:35 <kgriffs> malini: sorry, forgot to add that option.
15:40:40 <kgriffs> malini: noted your abstain
15:40:52 <balajiiyer> #vote yes
15:41:07 <vkmc> #vote yes
15:41:20 <kgriffs> closing the vote in 30 seconds
15:41:30 <cpallares> #vote yes
15:41:43 <flaper87> cpallares: you bet >.>
15:41:44 <flaper87> :P
15:41:48 <cpallares> :P
15:42:01 <kgriffs> #endvote
15:42:02 <openstack> Voted on "Queues -> Topics in v2.0?" Results are
15:42:04 <openstack> yes (8): alcabrera, balajiiyer, vkmc, kgriffs, cpallares, flwang, sriram, flaper87
15:42:09 <kgriffs> ok, thanks!
15:42:15 <flaper87> awesome
15:42:22 <kgriffs> #agreed queues -> topics in v2.0
15:42:27 <flaper87> so, migration plan
15:42:34 <kgriffs> #topic migration plan
15:42:35 <flaper87> mind if I take the floor ?
15:42:42 <kgriffs> go for it
15:42:55 <flaper87> so, we discussed it yday a bit and this is what we came up with
15:43:11 <flaper87> 1) We introduce lazy queues in v1.1 (this doesn't affect the user)
15:43:38 <flaper87> 2) We remove queue's metadata in v1.1 (this affects the user, if they cry, we can add it back in v2.0 somehow)
15:43:55 <flaper87> 3) We rename queues into topics in v2.0 and complete the work
15:44:02 <flaper87> That's roughly what we discussed
15:44:10 <flaper87> The migration will require updating the client
15:44:18 <flaper87> I added some notes on the blog post about the client
15:44:23 <kgriffs> I have some thoughts re metadata
15:44:27 <flaper87> there's a way to migrate the client without breaking users code
15:44:31 <flaper87> kgriffs: shoot
15:44:35 * flaper87 STFU
15:44:40 <kgriffs> :)
15:45:02 <kgriffs> so, I was thinking that people will probably want to access the same queue from both v1.0 and v1.1, right?
15:45:39 <kgriffs> if that is true, then once way to accomplish that is lazy metadata creation
15:46:11 <kgriffs> wait, I'm conflating this with #1
15:46:16 <flwang> kgriffs: i'm curious how to implement the lazy metadata creation for sqla?
15:46:36 <kgriffs> so, my idea was to make queue an attribute of each message
15:46:49 <kgriffs> and then simulate it being it's own resource
15:46:59 <kgriffs> so, we only create a separate resource if someone sets the metadata
15:47:10 <flaper87> kgriffs: I'd leave that to v2.0, though
15:47:18 <kgriffs> listing queues and such would have to do a grouping query
15:47:18 <flaper87> I was thinking the v1.1 work to be something like:
15:47:31 <flaper87> 1) Check if the queue exist (we already do this) and create it if it doesn't
15:48:26 <flaper87> I'd prefer making the change in v1.1 as minimum as possible
15:48:36 <flaper87> and slightly change the API towards a lazier one
15:48:52 <flaper87> then SBRANG, we swap it in v2.0 and laugh watching users crying
15:48:56 <flaper87> wait, did I say that?
15:48:56 <kgriffs> mmm, now that you mention it, I think messing with the data schema may be best left to 2.0 after all
15:49:06 <flaper87> :P
15:49:06 <kgriffs> :p
15:49:29 <alcabrera> heh
15:49:32 <flaper87> so, really small change for v1.1 (since we don't even have sqla migrations yet) and then bigger change in v2.0
15:49:42 <alcabrera> yeah, let's hold off on schema changes as much as possible until v2.0
15:49:45 <kgriffs> users may be less likely to think about deleting queues they no longer use if they are doing lazy-create
15:50:04 <kgriffs> but, I guess those records don't take much space
15:50:15 <flaper87> yeah
15:51:12 <flwang> alcabrera: yep, and as an end user, I don't want to the see the application is changing drastically
15:51:35 * flaper87 reads flwang message and laughs.... muahahaha muahahah MUAHAHAHAHAHHAHAHA
15:51:36 <flaper87> :D
15:51:49 <alcabrera> lol
15:52:04 <flwang> and the developers are laughing me :D
15:52:07 <alcabrera> flaper87 - smiter of end users
15:52:26 <cpallares> lol
15:52:39 <sriram> developer vs end user. Stay Tuned.
15:52:54 <vkmc> Maybe we could use something like a dirty-bit for lazy queues deletion... it may need checking those periodically though
15:53:05 <kgriffs> ok, so if an operator ends up with a ton of dead queues, that should be taken care of by the eventual migration to v2.0
15:53:09 <flwang> then user will beat the developer if they can meet at somewhere :)
15:53:22 <flaper87> kgriffs: plus, the operator can delete them too :)
15:53:51 <kgriffs> flaper87: yes, I suppose so
15:54:23 <kgriffs> vkmc - yeah, I think we will need to be creative when we plan the v2.0 upgrade process
15:54:45 <kgriffs> we'll have to decide whether we want v2.0 topics to be exposed as v1 queues
15:54:47 <kgriffs> and stuff
15:54:59 * flaper87 was thinking something along: "Stop marconi, delete the database; upgrade marconi; start marconi"
15:55:02 <flaper87> :D
15:55:03 <vkmc> Yeah... it would require some thinking
15:55:10 <kgriffs> ok, one last thought re metadata
15:55:31 <sriram> oh yeah. the topic-> queue exposure will be important
15:55:32 <kgriffs> I was thinking we could ask rackspace to check and see how many people are using that feature
15:56:08 <flwang> kgriffs: it would be great
15:56:14 <kgriffs> otherwise, I'm cool with lazy queues and removing metadata for queues in v1.1
15:56:23 <flaper87> awesome
15:56:24 <kgriffs> any vehement objections?
15:56:25 <flwang> kgriffs: to get some thoughts from the RAX product manager and the end uers
15:56:30 <kgriffs> flwang: +1
15:56:31 <flwang> before they beat us :D
15:56:32 <flaper87> we should actionize this things and create some bps for v1.1
15:56:34 <cpallares> kgriffs: +1
15:56:35 <flaper87> I can do that
15:56:44 <kgriffs> flaper87: bps already there, man
15:56:48 * kgriffs lives in the future
15:56:50 <flaper87> s/this/these/
15:56:55 <flaper87> kgriffs: waaaaaaaat?
15:57:10 <flaper87> holy moly
15:57:19 <kgriffs> #topic open discussion
15:57:28 <alcabrera> we made it through all the topics
15:57:31 <alcabrera> I'm pleased. :)
15:57:31 <flaper87> o/
15:57:31 <flaper87> o/
15:57:32 <flaper87> o/
15:57:37 <alcabrera> we're getting really good at this
15:57:38 <vkmc> :)
15:57:44 <sriram> nice. :)
15:57:53 * alcabrera picks flaper87
15:58:17 <flaper87> so, Yday was the last day of cpallares intership. I would like to thank her for her really AMAZING job, for being around all this time, participating in meetings and providing the help he provided
15:58:27 <flaper87> so, lets all thank her and a huge round of appluse
15:58:34 * kgriffs claps loudly
15:58:34 <cpallares> flaper87: :D
15:58:40 <kgriffs> cpallares: THANK YOU!!!
15:58:41 <alcabrera> cpallares: I'm so happy to have worked with you! Thanks fo joining in with us. :D
15:58:43 <malini> thank you cpallares!!
15:58:48 <alcabrera> *for
15:58:49 * sriram claps
15:58:49 <flwang> cpallares: thank you for your contribution :D
15:58:49 <malini> hope you come back
15:58:49 <vkmc> Yay cpallares \o/
15:59:02 <flaper87> malini: oh, she's not going anywhere
15:59:03 <sriram> Thank you cpallares! :)
15:59:05 <kgriffs> cpallares: Hopefully this was a good experience for you and you have some good takeaways.
15:59:10 <flaper87> I said the intership ended not that she's free to go
15:59:11 <flaper87> :P
15:59:15 <alcabrera> lol
15:59:17 <cpallares> haha
15:59:18 <balajiiyer> cpallares: thank you
15:59:20 <cpallares> kgriffs: I did :)
15:59:24 <malini> flaper87: that was a big LIE!
15:59:32 * flaper87 is obviously kidding
15:59:41 <malini> but anyways..cpallares happy to have you around
15:59:56 <cpallares> thanks malini :)
16:00:05 <cpallares> I will stick around thanks to flaper87
16:00:14 <kgriffs> #endmeeting