15:05:09 <kgriffs> #startmeeting marconi-notifications-brainstorm
15:05:10 <openstack> Meeting started Tue Dec  3 15:05:09 2013 UTC and is due to finish in 60 minutes.  The chair is kgriffs. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:05:11 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:05:14 <openstack> The meeting name has been set to 'marconi_notifications_brainstorm'
15:05:26 <kgriffs> #link https://etherpad.openstack.org/p/marconi-notifications-brainstorm
15:05:43 <kgriffs> #topic goals
15:05:44 <amitgandhi> #link https://github.com/amitgandhinz/cloud_notifications#api
15:05:59 <flaper87> o/
15:06:08 <kgriffs> OK, first off I wanted to get a rough consensus around what use cases we are trying to enable
15:06:26 <kgriffs> thoughts?
15:06:37 <megan_w> app development
15:06:41 <megan_w> alerting
15:07:00 <amitgandhi> reporting
15:07:15 <flaper87> and notification
15:07:16 <thomasem> definitely reporting
15:07:17 <alcabrera> notifications, in the sense of, when X happens, users [Y] get a notifications in one or more of [Z] forms.
15:07:23 * amitgandhi moves to etherpad
15:07:24 <flaper87> (events)
15:07:36 <flaper87> alcabrera: out of my head
15:07:38 <flaper87> thanks
15:07:42 <alcabrera> flaper87: :)
15:07:44 <flwang> alarm
15:08:05 <kgriffs> ok, let's get some concrete examples for those
15:08:19 <kgriffs> alerting?
15:08:39 <megan_w> 1.  send me email when i use more than 100GB of CDN usage
15:08:51 <kgriffs> megan_w: is that an alarm?
15:09:05 <megan_w> maybe i'm confusing alarm and alerting
15:09:06 <megan_w> sorry
15:09:07 <alcabrera> a disk drive is reaching 90% capacity -> ops manager gets an sms and an email
15:09:08 <thomasem> Send me a report at the end of each day detailing the operation failures
15:09:39 <thomasem> But, I guess that's more from a deployers perspective.
15:09:49 <flaper87> lets all go to the etherpad
15:09:53 <alcabrera> +1
15:10:01 * mpanetta watches
15:10:03 <flaper87> and then we can come back here
15:10:11 <thomasem> Okay, cool
15:10:23 <mpanetta> I'm lurking, I hope that is ok. :)
15:10:33 <alcabrera> mpanetta: lurk away!
15:10:35 <kgriffs> flaper87:  +1
15:10:42 <kgriffs> just go and type in examples
15:12:45 <flwang> an alarm is like the alarm using in Ceilometer
15:14:54 <gordonsim> I'm not entirely clear on what exactly notifications means here. On the one hand there is a general pub-sub type mechanism to allow subscribers to be notified when messages of interest are published. On the other is a service built on top of something like that, which is essentially a precanned listener that handles the notification of a message by generating some 'external' notification (e.g. sends an SMS/Emai
15:15:19 <gordonsim> is this more of the second?
15:15:25 <gordonsim> or is the first in scope as well?
15:16:12 <flaper87> gordonsim: it's pretty much both, TBH!
15:16:26 <flaper87> The final goal is to allow people to do the second
15:16:44 <flaper87> gordonsim: want to join? https://etherpad.openstack.org/p/marconi-notifications-brainstorm
15:17:10 <megan_w> i suppose there is something non-immediate about a report, right?  that's what makes it differnt from alerting on some of these use cases..?
15:17:20 <gordonsim> flaper87: yes, I'm there
15:17:34 <flaper87> gordonsim: ah sorry, I don't see your nick
15:17:36 <flaper87> :D
15:17:44 <alcabrera> megan_w: I'm with you on that idea. The alarm would be triggered on some condition, whereas the report probably has a periodic nature.
15:17:54 <megan_w> right
15:18:02 <alcabrera> alarm behaves very much like alert
15:18:14 <kgriffs> megan_w: noted
15:19:20 <flwang> alcabrera: yep, but I prefer to use alarm to be consistent with the term using in Ceilometer
15:20:20 <flwang> and generally, alarm will trigger a binding action
15:20:39 <alcabrera> flwang: noted - the earlier we agree on common terminology, the better off we'll be in the long run. :)
15:21:02 <flwang> alcabrera: +1
15:27:16 <kgriffs> #info Need way to avoid 2 email workers from resending the same message
15:28:23 <kgriffs> OK folks, looks like we are winding down on use cases and terminology
15:28:36 <kgriffs> let's wrap that up so we can move on to the next topic
15:28:50 <flwang> kgriffs: +1 :D
15:33:36 <mpanetta> So templates would be for presentation only?  Why not use them to define the types of notifications as well?  Or does that not make sense?
15:35:06 <alcabrera> mpanetta: hmm... I think template == output, so it could be the envelope for a given notification as presented to the target device/user. Trigger should be the input, in my mind.
15:35:33 <mpanetta> Ah ok
15:36:13 <kgriffs> mpanetta: yeah, each sink could have an associated template and trigger/pattern thingy
15:36:54 <mpanetta> So the templates would be a way to define what you are subscribing to?
15:38:15 <alcabrera> mpanetta: nah, that would be part of the pattern/trigger. The pattern/trigger might say, "when X arrives on channel [Y], notify [Z] using this particular Template"
15:38:34 <mpanetta> Ok, tht makes sense.
15:38:54 <kgriffs> #topic elephants
15:38:58 <kgriffs> (see etherpad)
15:39:46 <mpanetta> So colorful heh.  Never done this etherpad thing before.
15:40:07 <alcabrera> mpanetta: this is one of the prettiest ones I've seen. 13 users. :D
15:41:09 <kgriffs> ok, great thoughts so far!
15:41:15 <kgriffs> we have 20 minutes left
15:41:27 <kgriffs> Moving on
15:41:54 <thomasem> cool
15:42:05 <kgriffs> #topic features
15:42:09 <mpanetta> Everything is an 'event' :P
15:42:15 <kgriffs> Any other features people want to add/talk about?
15:42:54 <kgriffs> (see etherpad under "Features: Stuff Users See")
15:43:41 <megan_w> sort of
15:43:55 <megan_w> i assume we leave the actual SMS, email, etc up to someone else?
15:44:05 <kgriffs> megan_w: +1
15:44:51 <kgriffs> I think the marconi project should provide 2-3 reference sinks.
15:45:25 <kgriffs> email
15:45:28 <kgriffs> webhook
15:45:30 <kgriffs> one more?
15:45:40 <amitgandhi> sms
15:45:43 <megan_w> yeah
15:46:10 <amitgandhi> and i think we need to do APN
15:46:22 <mpanetta> APN?
15:46:28 <amitgandhi> push notifications
15:46:34 <mpanetta> AH ok
15:46:36 <amitgandhi> it gives us the mobile market
15:46:52 <amitgandhi> amazon SNS introduced it a few months back also
15:47:58 <megan_w> totally agreed.  people expect push here
15:49:30 <kgriffs> I like APN
15:49:40 <kgriffs> I want to get people thinking mobile apps
15:49:45 <kgriffs> let's do that one
15:50:03 <kgriffs> people can insert their APN creds into the reference sink to try it out
15:51:33 <alcabrera> For the record, since I wasn't aware of APN...
15:51:36 <alcabrera> #info  Apple Push Notification Service (APNS)
15:51:45 <thomasem> Thanks, was about to ask. =]
15:52:02 <kgriffs> Google has something similar for Android
15:52:10 <kgriffs> (I forget the name)
15:52:23 <kgriffs> OK, we are short on time
15:52:30 <flaper87> APN = http://en.wikipedia.org/wiki/Access_Point_Name
15:52:36 <kgriffs> One more topic I wanted to touch on, then we will wrap up this session
15:53:09 <alcabrera> #info GCM = Google Cloud Message, Android's analogue to APNS
15:53:10 <kgriffs> flaper87: The "s" in "APNs" is significant, it turns out. :p
15:53:27 <flaper87> :P
15:53:35 <mpanetta> TMTLA
15:54:51 <kgriffs> #topic how to publish messages to a sink
15:55:43 <kgriffs> OK, so thoughts on how events will get published?
15:55:47 <kgriffs> worker pool?
15:56:02 <kgriffs> pipleline stage within marconi itself?
15:56:13 <kgriffs> something else?
15:56:21 <alcabrera> I'm +1 for worker pool, so that the workers themselves can be scaled as needed
15:56:25 * flaper87 has to step out for a bit!
15:56:31 <amitgandhi> +1 for WP
15:56:38 * flaper87 has a call
15:56:39 <alcabrera> flaper87: D:
15:56:47 <flwang> I prefer pipeline
15:56:48 <alcabrera> flaper87: jk, see you soon.
15:56:49 <kgriffs> flaper87: need your vote on the above real quick
15:56:55 <flaper87> if you guys want to discuss that, go ahead! I'll catch up. I don't want to break the flow
15:56:57 <kgriffs> (or when you get back)
15:56:58 <flaper87> sure
15:57:25 <flaper87> mmh, WP, it'll allow us to run them concurrently and / or distributed
15:57:31 <flaper87> actually, concurrently :D
15:57:44 <alcabrera> I like the idea of a transformation pipeline, since it has a functional flavor to it. However, if a pipeline was the connecting mechanism, I'd want that to delegate out to the workers.
15:57:44 <flaper87> we don't want to send yet another message to send a notification
15:58:01 <alcabrera> flaper87: +1
15:58:09 <flwang> pipeline allow us to add numbers of transformer
15:58:18 <flwang> and the transformer can be customied by user
15:58:39 <kgriffs> so, with the pipeline approach we could just run lots of wsgi workers. non-blocking I/O would be crucial.
15:59:10 <kgriffs> also, as flwang pointed out, you can build up a simple workflow (think filter pipeline in photoshop)
15:59:32 <kgriffs> downside is you lose some efficiency since you can't scale queues and notifications independently
15:59:38 <kgriffs> oz_akan_: thoughts?
15:59:49 <dragondm> ^ we are working on this for Ceilometer..
16:00:03 <alcabrera> My primary concern with the pipeline approach is the matter of independent worker scaling - would we just launch as many marconi instances as needed to meet the demand for all types of sinks (sms, apns, hooks, etc.)?
16:00:19 <amitgandhi> i like that worker pools can be scaled horizontally and are more resilient to workers crashing
16:00:27 <oz_akan_> hi, I wasn't following the thread, let me have a look
16:00:41 <kgriffs> we might also consider some kind of hybrid idea
16:00:59 <kgriffs> for example, you could set up a distributed pipeline by stacking queues end-to-end
16:01:30 <mpanetta> Oh how unixy of you
16:01:32 <kgriffs> leave marconi-queues to do it's think. marconi-notifications sets up the queues, manages subscriptions and workers
16:01:42 <kgriffs> mpanetta: why thank you! I try. :D
16:02:00 <mpanetta> :)
16:02:20 <mpanetta> marconi == | :)
16:02:59 <kgriffs> app | marconi | worker | sms
16:03:21 <alcabrera> yup - functional transforms all the way through
16:03:39 <kgriffs> app | queue1 | worker_a | queue2 | worker_b | sms
16:03:44 <kgriffs> something like that.
16:03:52 <mpanetta> Sounds good to me
16:03:58 <kgriffs> I think we just invented workflow-as-a-service
16:04:05 * kgriffs winks at Mirantis
16:04:05 <alcabrera> lol
16:04:07 <mpanetta> lol
16:04:17 <alcabrera> so that's another good elephant
16:04:21 <alcabrera> actually
16:04:28 <alcabrera> yeah, what's the deal with Mirantis + workers?
16:04:31 <alcabrera> Is that something we can use?
16:04:36 <flwang> kgriffs: it's Mistral workflow-as-a-service
16:04:49 <alcabrera> flwang: thanks for the clarification!
16:04:53 <kgriffs> yep
16:05:07 <alcabrera> #link https://wiki.openstack.org/wiki/Mistral
16:05:19 <kgriffs> Is there a concrete use case for doing multiple transformations or whatever?
16:05:30 <amitgandhi> oooph cloud cron!
16:05:33 <amitgandhi> *oooh
16:05:39 <kgriffs> Just thinking we may be getting ahead of ourselves
16:05:51 <kgriffs> amitgandhi: ooh, that would be nice
16:06:04 <alcabrera> anyway
16:06:06 <amitgandhi> its a mistral use case
16:06:09 * alcabrera gets back to topic
16:06:13 <kgriffs> we could set up a "cron" job that hits a web hook for the event data, then pushes it to a topic
16:06:21 <alcabrera> "how to publish messages to a sink"
16:06:38 <kgriffs> ok folks, looks like we are out of time
16:06:51 <thomasem> hmm? kgriffs Ceilometer is already and will be doing pipeline work for its own stuff. Is this in an effort to expose a pipeline to clients?
16:06:56 <kgriffs> I really appreciate everyones thoughts. We had a very productive brainstorming session!
16:07:04 <thomasem> Indeed
16:07:12 <amitgandhi> yeh this was fun
16:07:29 <kgriffs> thomasem: I definitely don't want to reinvent workflow/distributed pipeline
16:07:30 <amitgandhi> next steps?
16:07:33 <mpanetta> Mistral looks very interesting...
16:07:42 <thomasem> Haha, yeah. That'd be a bit wild.
16:07:44 <thomasem> :P
16:07:58 <kgriffs> OK, so I will attempt to summarize the etherpad into a pretty wiki
16:08:03 <kgriffs> And link that to the bp
16:08:09 <kgriffs> #link https://blueprints.launchpad.net/marconi/+spec/notifications
16:08:14 <flwang> cool
16:08:16 <thomasem> =] Cheers everyone!
16:08:44 <alcabrera> awesome, good session. :D
16:09:11 <mpanetta> kgriffs: So, about that merge... ;)
16:09:12 <kgriffs> then maybe next monday we can discuss sub-blueprints and get volunteers to start laying down code
16:09:19 <kgriffs> #endmeeting