14:01:32 <isviridov> #startmeeting magnetodb
14:01:33 <openstack> Meeting started Thu Jan 22 14:01:32 2015 UTC and is due to finish in 60 minutes.  The chair is isviridov. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:01:34 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
14:01:37 <openstack> The meeting name has been set to 'magnetodb'
14:01:48 <isviridov> Today agenda https://wiki.openstack.org/wiki/MagnetoDB/WeeklyMeetingAgenda#Agenda
14:01:55 <isviridov> o/
14:02:03 <ikhudoshyn> щ/
14:02:19 <ominakov> o/
14:03:04 <isviridov> ikhudoshyn cyrillic here
14:03:15 <ikhudoshyn> oh, really ?!
14:03:30 <isviridov> It looks like Bostin is not with us today
14:03:45 <ikhudoshyn> gonna be fast?
14:03:52 <isviridov> Let us go throught action items
14:04:07 <isviridov> ajayaa start mail discussion about TTL implementation
14:04:22 <isviridov> #topic Go through action items isviridov
14:05:04 <isviridov> So, the question was rised
14:05:47 <isviridov> Everybody seen this email?
14:06:39 <isviridov> ikhudoshyn aostapenko?
14:06:55 <ikhudoshyn> hm
14:07:03 <ikhudoshyn> oh, yea
14:07:06 <ikhudoshyn> there was
14:07:17 <isviridov> #link http://osdir.com/ml/openstack-dev/2015-01/msg00944.html
14:07:36 <ikhudoshyn> isviridov: tnx
14:07:56 <ikhudoshyn> move on?
14:08:24 <isviridov> ikhudoshyn I expected to hear any oppinions
14:09:03 * isviridov I know it is hard without dukhlov
14:09:30 <ikhudoshyn> i don't like the idea of only enabling ttl on insert
14:09:37 <isviridov> The question is how to implement update item with TTL
14:09:54 <isviridov> ikhudoshyn +1
14:10:50 <ikhudoshyn> i think first should agree if Dima's native LSI are prod-ready, then switch to them
14:11:13 <ikhudoshyn> and then evaluate re-evaluate required efforts
14:11:45 <ikhudoshyn> from what i've seen our insert/update/delete logic becomes much cleaner with the latter
14:12:04 <isviridov> ikhudoshyn I think that we have adopted LSI already
14:12:26 <ikhudoshyn> we've merged them, it's not the same for me))
14:12:43 <isviridov> Here it is the same :)
14:13:00 <isviridov> Merged and swithed our configuration
14:13:07 <ikhudoshyn> ok, so we could go with the 2nd step)
14:13:26 <ikhudoshyn> are we to keep out outdated stuff forever?
14:13:44 <ikhudoshyn> (like old LSI implementation)
14:14:28 <ikhudoshyn> if we think that new LSI is good enuogh lets get rid of old
14:14:38 <isviridov> ikhudoshyn now, just we have migration we can foget about it
14:14:49 <isviridov> I see no reason to support it anymore
14:14:54 <aostapenko> I don't think we should support previous implementation
14:15:03 <ikhudoshyn> hurray
14:15:18 * ikhudoshyn loves throwing away old stuff
14:15:25 * isviridov not sure if ominakov has created BP for migration
14:15:37 * isviridov loves it as well
14:15:53 <ikhudoshyn> getting back to ttl, let's re-check estimates
14:16:17 <ominakov> isviridov, i'll do it
14:16:32 <isviridov> The key problem is: C* doesn't support TTL per row
14:16:57 <ikhudoshyn> yep, we should emulate it
14:17:47 <ikhudoshyn> we may consider having it per item but thus we'd diverge from AWS API
14:17:48 <isviridov> Means we have to update whole row
14:18:03 <ikhudoshyn> isviridov: exactly
14:18:25 * isviridov is checking TTLs support in AWS
14:18:54 <isviridov> ikhudoshyn are there any TTL in AWS?
14:19:13 <ikhudoshyn> hm, i guess it was
14:20:09 * ikhudoshyn seems to be wrong
14:20:26 <ikhudoshyn> http://www.datastax.com/dev/blog/amazon-dynamodb no ttl in dynamodb
14:20:42 <isviridov> ikhudoshyn it doesn't accroding to AWS API
14:21:29 <isviridov> Ok, let us return back to TTL per row in mdb api only :)
14:21:39 <ikhudoshyn> the only issue i could think of, is a backend that wouldn't have native ttl
14:22:02 <ikhudoshyn> i mean, we could have it per attribute, but..
14:22:11 <isviridov> ikhudoshyn it is responcibility of driver aouthor to implement it or not
14:22:39 * isviridov don't like per table TTL
14:22:54 <ikhudoshyn> if we'd want to support another backend w/o ttl, emulating ttl per item would be much more complex
14:24:02 <ikhudoshyn> so..
14:24:11 <isviridov> Do you mean that TTL feature is not needed?
14:24:15 <ikhudoshyn> are we to have per item ttl?
14:25:00 <ikhudoshyn> isviridov: no, i just mean that emulate per-row ttl is easier than per item
14:25:06 <isviridov> #link https://blueprints.launchpad.net/magnetodb/+spec/row-expiration by Symantec
14:26:19 <ikhudoshyn> ok, let's not use C* native ttl))
14:26:23 <isviridov> Item == row
14:26:48 <ikhudoshyn> *easier than per-field
14:27:35 <isviridov> charlesw is TTL per field is expected to be needed?
14:28:32 <charlesw> I'd think so
14:28:57 <charlesw> until C* comes up with a solution
14:29:02 <ikhudoshyn> we can't have both at the same time
14:29:09 <isviridov> ikhudoshyn why?
14:29:43 <charlesw> I was reading C* may expose row marker as a column, then we can set ttl on row marker
14:29:50 <ikhudoshyn> usage seems to be far too complex
14:30:38 <isviridov> charlesw really interesting. Could you point us where we can read it?
14:30:43 <charlesw> http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Setting-TTL-to-entire-row-UPDATE-vs-INSERT-td7595072.html
14:32:20 <ikhudoshyn> that's exactly what ajayya told in nis email
14:32:27 <ikhudoshyn> *his
14:33:16 <isviridov> doesn't look like row marker is available via CQL ap
14:33:26 <isviridov> *api
14:34:05 <ikhudoshyn> "Wouldn't it be simpler if Cassandra just let us change the ttl on the row marker?" --> This is internal impl details, not supposed to be exposed as public API
14:34:15 <ikhudoshyn> that's from that thread
14:35:26 <isviridov> Better to say not exposed
14:35:57 <isviridov> #idea suggest exporing row marker for C* community
14:35:58 <ikhudoshyn> we could agree 'it's not exposed right now'
14:36:25 <isviridov> #idea overwrite all colums with new TTL
14:36:40 <isviridov> Does it look correct?
14:36:51 <ikhudoshyn> #idea implement per-row ttl manually
14:37:07 <ikhudoshyn> using dedicated ttl field
14:37:09 <charlesw> is ttl allowed on primary key?
14:37:22 <ikhudoshyn> i doubt
14:37:55 <charlesw> if not, setting ttl on all columns won't work
14:38:17 <isviridov> charlesw +1
14:38:43 <isviridov> But according to you #link http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Setting-TTL-to-entire-row-UPDATE-vs-INSERT-td7595072.html it should work
14:39:37 <isviridov> ikhudoshyn do you mean manually check TTL and manually delete it?
14:39:49 <ikhudoshyn> isviridov: exactly
14:40:14 <isviridov> Sounds like async job and check on each read
14:40:29 <ikhudoshyn> isviridov: yup
14:40:46 <isviridov> ikhudoshyn will you share your view in ML?
14:40:55 <ikhudoshyn> ok will do
14:41:13 <isviridov> Great, I'll join
14:41:39 <isviridov> charlesw we would like hearing from you as well
14:41:48 <isviridov> Moving on?
14:41:54 <ikhudoshyn> +1
14:41:59 <charlesw> sure I'd love to join
14:42:35 <isviridov> Next action item
14:42:40 <isviridov> achudnovets update spec
14:43:10 <isviridov> achudnovets_ was going to drive to clinic with his son
14:43:19 <isviridov> I think we can go ahead
14:43:32 <ikhudoshyn> lets do
14:43:35 <isviridov> #topic Discuss https://blueprints.launchpad.net/magnetodb/+spec/test-concurrent-writes aostapenko
14:43:43 <isviridov> aostapenko stage it your
14:44:16 <isviridov> *yours
14:44:17 <aostapenko> Yes, I'm going to implement this scenarios with tempest
14:44:38 <isviridov> With conditional writes?
14:45:30 <aostapenko> I did not write cases yet. working on framework
14:45:50 <aostapenko> I will share list of scenarios
14:46:46 <isviridov> I believe it is the only reasonable way to update the row
14:47:16 <aostapenko> So will use it. We'll have negative cases too
14:47:52 <isviridov> charlesw any hints how aostapenko can do it?
14:48:15 * isviridov Paul would be useful here
14:49:00 <charlesw> I'll ask Paul. Andrei could you please write a doc so we are clear on our cases?
14:49:46 <isviridov> charlesw do you think bp itself is a good place for this?
14:50:04 <charlesw> yes
14:50:06 <aostapenko> charlesw: sure
14:50:15 <isviridov> #action aostapenko charlesw isviridov brainstorm the scenario
14:50:32 <isviridov> aostapenko till then not approved'
14:50:59 <isviridov> Moving on
14:51:11 <isviridov> #topic Open discussion isviridov
14:51:30 <isviridov> Anything for this topic?
14:51:35 <ikhudoshyn> guys pls review aostapenko's patch about swift and lets merge it
14:51:47 <isviridov> Link?
14:51:48 <charlesw> After discussion with Dima, I plan to refactor some of the notification code
14:52:21 <isviridov> charlesw great
14:52:27 <charlesw> moving notification from storage manager layer to API controller layer
14:52:32 <ikhudoshyn> https://review.openstack.org/#/c/146534/
14:52:34 <isviridov> Anything else critical for review?
14:52:39 <charlesw> what do you guys think
14:52:51 <ikhudoshyn> charlesw: why?
14:53:45 <charlesw> make storage manager code cleaner
14:53:51 <isviridov> #action dukhlov_ charlesw aostapenko isviridov review https://review.openstack.org/#/c/146534/
14:54:23 <isviridov> charlesw how will you measure table async tasks duration if so?
14:54:27 <ikhudoshyn> charlesw: ... and make API code messier
14:54:29 <ikhudoshyn> ?
14:55:36 <ikhudoshyn> i don't mind adding notifications to API layer
14:55:38 <charlesw> And the request metrics collection can use the notification mechanism. So we won't have two sets of notification (in API/middleware using StatsD, and other places using messaging)
14:55:45 <isviridov> ikhudoshyn the more notification the more information we have about system
14:56:25 <ikhudoshyn> isviridov: +1, i just wouldn't like to remove existing notifications from storage
14:56:44 <miqui_> ..hi...
14:56:50 <dukhlov_> ikhudoshin: now we have unstructured notifications
14:57:12 <isviridov> miqui_ hello
14:57:17 <ikhudoshyn> dukhlov_: what d'you mean 'unstructured'
14:57:21 <ikhudoshyn> miqui_: hi
14:57:28 <charlesw> hi miqui
14:57:32 <dukhlov_> we are sending notification somehere
14:57:35 <isviridov> charlesw what do you mean two sets?
14:57:59 <dukhlov_> we don't have any strategy when where and what we need to send
14:58:25 <charlesw> in middlware/API controller, we sends StatsD metrics, in storage, we use messaging
14:58:42 <dukhlov_> maybe it is because we don't have customer's real usecase for that
14:59:15 <dukhlov_> but now we have first one - integrate statsd to notification mechanism
14:59:39 <dukhlov_> for this case we need request based notification
15:00:09 <dukhlov_> like request done or request failed and took some time for this job
15:00:11 <ikhudoshyn> dukhlov_: so lets consider ADDING motifications to API
15:00:18 <isviridov> Let us return back to use case
15:00:36 <isviridov> 1. we need to send information to ceilometer
15:00:46 <charlesw> we plan to have a central event registry, it will describe each event: type, messaging or metrics event name, delivery mechanism(messaging/metrics/or both). And use one notifier to decide what to do based on event description.
15:01:06 <dukhlov_> which information exactly?
15:01:22 * isviridov will listen a bit
15:01:51 <dukhlov_> if we just add notifications it will be duplicate each other in storage and api
15:02:20 <charlesw> +1
15:02:26 <aostapenko> +1
15:02:47 <isviridov> The official meeting finished
15:02:52 <isviridov> #endmeeting