15:00:17 #startmeeting StoryBoard 15:00:21 Meeting started Mon May 11 15:00:17 2015 UTC and is due to finish in 60 minutes. The chair is krotscheck. Information about MeetBot at http://wiki.debian.org/MeetBot. 15:00:22 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 15:00:23 hi hi, anyone here? 15:00:26 The meeting name has been set to 'storyboard' 15:01:15 o/ 15:01:29 Woo, that's one! 15:01:35 Well, I'm 2 15:02:34 We don't reallly have quorum yet, but then our agenda isn't really very full 15:02:54 So let me just run through the updates. 15:03:10 Yeah, I did some reviews and broke it, but the revert seems to be enough to fix it 15:03:13 Agenda: 15:03:19 #Link https://wiki.openstack.org/wiki/StoryBoard#Agenda 15:03:26 hi 15:03:32 Ohai! 15:03:34 More people! 15:03:40 #topic Actions from Last Week: 15:03:41 None 15:03:45 #topic Urgent Items 15:04:00 Looks like there was an unexpected side effect of one of my patches that broke login. 15:04:14 mordred was kind enough to fast-track a revert of that. 15:04:26 krotscheck, do you know details, or it needs more investigation? 15:04:27 I didn't do it? 15:04:31 oh. yeah I did 15:04:46 yolanda: Well, after I saw what patch it was I deduced the reason. 15:05:32 I'm assuming that my "Hey redirect things to the webclient" patch on the API also caught any legitimate redirects to the auth endpoints, and thus shortcircuited the login redirects. 15:06:12 Since I still feel that's the correct way to go about building links to individual tickets, I'll add a few tests onto the original patch and resubmit it. 15:06:32 #action krotscheck Revisit link redirect patch and make it sane for login. 15:06:37 Any other urgent items? 15:06:40 ok, maybe adding some exception for the auth ones? 15:06:56 yolanda: Yep, that's the idea. 15:07:58 Alright, assuming no urgent items. 15:08:02 #topic User Feedback 15:08:04 Anyone get any? 15:08:14 nope 15:09:09 Alrightey 15:09:15 #topic In Progress (email) 15:09:18 Righto.... 15:09:28 So a ton of email patches landed this morning, which I was _not_ expecting. 15:09:39 Leading to the above mentioned login bug. 15:09:54 krotscheck: I thought those were the reviews you were looking for 15:10:02 ttx: Well, yes, they were. 15:10:10 when you called for reviews earlier 15:10:14 ttx: But they've been languishing for so long that I'd almost given up hope :) 15:10:32 (After all, we still have infra-specs reviews from january) 15:11:22 Anyway, I put in the remaining code pieces for single-event email workers, here: https://review.openstack.org/#/c/179542/ 15:13:10 I only put one of the text email templates into it, so that discussion can focus on the actual implementation of the event handling rather than the email templates themselves. 15:13:28 I will add additional templates in various other followup patches. 15:13:46 checking 15:13:48 But with that patch, the remaining piece is to make sure everything is kosher with the puppet module. 15:14:39 krotscheck, any reason to have the template name on caps? 15:14:40 whoah, landing those email patches was not cool 15:15:00 i was patiently waiting for krotscheck to have a conversation with me about them 15:15:11 jeblair: I responded to your comments.... lemme check 15:15:23 krotscheck: yeah, i wasn't around then 15:15:49 i mean, i left my comments in februrary, and at krotscheck's request, waited two months for krotscheck to be ready to engage in a dialog 15:15:53 jeblair: lemme go look at the patch chain. 15:16:16 it is not okay to just, while i am on a plane, leave some comments and then merge the series 15:16:16 jeblair: That was right around when mordred's thread on the viability of storyboard landed. 15:16:59 krotscheck: i do not think mordred's thread invalidates my concerns with those patches 15:17:05 jeblair: The comment I'm referring to was on April 28th. 15:17:19 That was 2 weeks ago. 15:17:40 For reference: https://review.openstack.org/#/c/151413/ 15:18:10 krotscheck: i see that. i was traveling then. i expect that after waiting for two months for you to engage with me in a dialog, i could be given a more than a few days to respond. 15:18:25 krotscheck: i'm not upset with you, to be clear 15:18:34 i believe that the patches should not have been merged 15:18:34 Well, I was waiting :) 15:18:41 Then ttx swoops in and +A's it 15:18:56 And the entire chain came with it. 15:19:38 But with that in mind, removing the outbox patch that you had a specific issue with is actually fairly easily done. In the actual send worker patch, you can replace all instances of with get_outbox() as outbox: with get_sender as sender: 15:19:50 generally when someone raises a substantial point on a -1 we wait until it has actually been resolved with the person who raised it 15:20:11 And, quite frankly, the more I worked on the problem the more I realized that the outbox piece was premature optimization. 15:20:12 I waited a few weeks and approved it 15:20:29 I thought the "better than nothing" argument wins 15:20:36 I'm fine with reverting though 15:20:51 ttx: yeah, it would have been better to say "hey jeblair -- are you going to address krotscheck's points or should i just merge this?" 15:20:57 communicating is always the right answer 15:21:11 right, my mistake 15:21:20 Ok, so here's a question. 15:21:46 * krotscheck gets away from the 'whodunnit' and gets back to the communicating about the code. 15:21:49 My impression was taht this code was not used yet, so there was no permanent damage 15:21:58 It comes down to whether having an intermediary storage place for emails before they are sent is a thing we need. 15:22:27 My gut says it's premature optimization, but not necessarily bad. 15:22:52 Let's say we don't know where our sender is going. It could be exim, it could be mail.google.com, it could be exchange.... 15:22:53 krotscheck: i have read your comment and i still believe you are over-engineering. everything you described is handled by an MTA. it is designed for reliable high-throughput high-concurrency activity. 15:23:35 Also I misread the timeline there, thought it was one month ago 15:23:37 So for every single event that happens in StoryBoard, which "could" be many, we have two options. 15:23:43 not 3 days ago 15:24:12 Either 1: Send the email immediately, which incurs the connection overhead to our sender for every single email. 15:24:19 (which is small) 15:24:34 (i measure it at 0.005 seconds) 15:24:40 (unless it's exchange, in which case *hand wavey*) 15:24:47 it should never be exchange 15:25:12 jeblair: Infra isn't the only customer. 15:25:16 and the reason i say that is that if you take on the responsibility of reliably talking to exchange, you are _really_ becoming an MTA 15:25:19 krotscheck: exactly 15:25:28 if you need to talk to exchange, then you need to run a local mta 15:25:56 if you want to handle that directly in storyboard then you will need to implement all kinds of retry rules and queue expirations, etc. 15:26:13 instead, just let an mta handle it. 15:26:27 jeblair: I get that. However at this time we're still discussing the need for an outbox, given a Sender Of Unknown Latency. 15:26:40 what is a sender of unknown latency? 15:27:19 It is a sender that is abstracted behind get_sender(), which has a connection latency which we, as application developers, cannot reliably predict for every single install of storyboard in the wild. 15:27:48 jeblair: Can I finish explaining the cases before you jump into this argument? I promise I'll be fast. 15:28:22 krotscheck: um. i don't know what i just did to make you ask me not to jump into an argument. 15:28:23 Case 2: Roll up all the emails into an outbox, and send them as a batch. 15:28:42 There. 15:28:55 Those are the two options under discussion right now. 15:29:35 Now, jeblair's point is that the sender, in its little black box, will likely have a ridiculously small connection overhead, and therefore batching things is unecessary. 15:29:45 krotscheck: have you reversed the meaning of "sender"? 15:29:54 The thing that sends the emails? 15:29:59 krotscheck: right, storyboard. 15:30:18 The component of storyboard that actually accepts a message and ensures its delivery. 15:30:20 Better? 15:30:31 (basically the get_sender() method. 15:30:58 The counterargument is that batching is good, and we cannot predict how long the latency of establishing a connection to send an email will be. 15:31:07 And, well, the latter argument has no data. 15:31:25 krotscheck: yes, that is an excellent argument which is why i fully agree with you and therefore storyboard should require a local MTA in all installations. 15:31:34 krotscheck: it should not, however, implement it itself. 15:31:55 krotscheck: if then, because storyboard is installed at a site that uses exchange, that local MTA is configured to forward to exchange, that's fine. 15:32:26 jeblair: Dude. I agree with you. 15:32:45 But fact is that nobody who's contributing code to storyboard has the necessary expertise to put that together. 15:33:11 So we're left with the hammer, and something that looks like a nail. 15:33:28 krotscheck: it's quite simpla and i'm happy to help with that 15:34:03 jeblair: Neat! 15:34:11 * krotscheck should have said that sentence in february 15:34:33 krotscheck: well, in februrary you said you wanted to finish this entire line of development before having a conversation about it 15:34:39 Ok, so to be clear. 15:34:41 Well, true. 15:34:42 my bad. 15:34:53 even so, we're losing a single patch out of the entire chain. 15:35:02 * krotscheck doesn't think the rest of his implementation has a problem. 15:35:16 (template rendering, gathering resources to render, etc etc) 15:35:19 ok, i didn't look much past the first patch 15:35:38 yeah, sounds like that's the case 15:35:41 jeblair: Oh! 15:36:07 Alrighty then, how about I give a quick overview of how all the things work. 15:36:39 sure 15:38:34 so shall we revert my misstep ? 15:38:48 or try to fix it and keep most of what landed in the patch chain ? 15:38:59 ttx: Yes. Since I understand the code well enough I can switch the outbox code to use get_sender instead. 15:39:08 That last patch didn't land 15:39:16 But on to my explanation.... 15:39:17 The basic logic for the non-digest worker is that it receives an event from the API (Worker plugin framework), figures out who subscribed to it (subscription api), determines whether that person wants email (user preference api), resolves the necessary resources (DB session persistence courtesy of NikitaKonovalov), and then chooses a template from a URL pattern ({resourcename} 15:39:58 The resource pattern is storyboard/email/templates/{{resourcename}}/{{HTTP_ACTION}}(_subject)?.(txt|html) 15:40:19 It requires at least a subject template and a text template, but will also look for an html template. 15:40:35 It uses jinja2 to render those templates, builds a message, and passes it to the (outbox|sender) 15:41:23 The batching happens because each event may have N subscribers, so that's in a loop. 15:41:33 Anyway, that's the gist. 15:41:47 #action krotscheck Make storyboard use outbox instead of sender. 15:41:47 okay, yeah that all makes sense; i think we can fix the plumbing around that 15:41:48 Oops 15:42:02 #action krotscheck Ignore previous action, use sender instead of outbox. 15:42:29 jeblair: So what piece will you provide? 15:43:09 krotscheck: i will not be able to do anything until after the summit. it is a very busy time for me. 15:43:18 jeblair: I get that much. 15:43:27 * krotscheck is likely swamped as well 15:43:34 jeblair: I just want to get the actual task into place. 15:44:08 I'm guessing it's mainly puppet work, plus some additions to storyboard's documentation? 15:44:56 no puppet work should be necessary; there's already an MTA in place; so yeah, probably just the outbox/sender switch you mentioned and doc work 15:45:48 I'm guessing that ptyhon's email library already knows how to talk to a local mta? 15:47:02 The puppet module is the one I'm a little worried about, because I'd like that to install a fully functional piece. 15:47:27 krotscheck: we install a working exim that delivers to the internet with our puppet 15:47:42 i'm not certain that we should consider mta configuration in scope for puppet-storyboard 15:47:48 jeblair: StoryBoard's puppet will need to know how to do that. 15:48:06 i would expect most sysadmins to want to be able to configure their mta of preference 15:48:53 the mta won't need any storyboard-specific configuration, it just needs a site-appropriate configuration 15:49:07 * krotscheck pondres this 15:49:18 Ok, let me get back to you on whether I want that in the puppet module. 15:49:20 (our "site-appropriate" config is "just deliver to the durn internet"; others might be "smarthost to our exchange server") 15:49:40 krotscheck: ok, happy to discuss further 15:49:44 I'm trying to figure out what the userbase is going to be. Whether we're only-sysadmins or whether there's a "Hey we want joe random to install this on his home box" 15:50:15 I suspect that it's going to be the latter.... until someone decides to do a cheap Storyboard-as-a-Service and then suddenly it's only sysadmins again 15:50:40 Aaaanyway 15:50:57 #action jeblair add references to exim configuration to storyboard docs. 15:51:04 (or should that be mta) 15:51:06 Anywa 15:51:25 Are there any other points on Email right now? 15:51:59 nak 15:52:19 Alrightey. 15:52:27 #topic Discussion Topics (vancouver) 15:52:34 We've got 8 minutes. 15:52:48 Also, no meeting next week! 15:52:56 #action Krotscheck Send no meeting announcement. 15:54:10 i'm finishing my sb slides now 15:54:20 i'll send to ttx and krostcheck for review 15:54:27 just short 10 minutes session 15:55:25 Yay! 15:55:35 #link http://libertydesignsummit.sched.org/event/6dac7819d75ce432ecacca66010e3280#.VVDHY-SVtpg 15:56:09 jeblair: Neat! 15:56:39 One question- in my discussions with mordred, he mentioned that some pressure was coming from working groups and the board. How do we make sure those individuals are represented? 15:56:43 yeah, i hope we'll talk about mordred's email and come out of there with a direction 15:57:16 Right now they are really just "vague THEM!" 15:57:20 krotscheck: I'll try to get the to come to the session 15:57:38 mordred: Thanks. I'll come to the board meeting sunday too, see if I can talk to them :) 15:57:52 krotscheck: and, once I've informed them that it exists, if they do not come and express themselves, they can live with our decisions, whichever they may be 15:58:06 That's what I'm hoping for :) 15:58:12 Ok, thanks everyone! 15:58:18 #endmeeting