15:01:32 #startmeeting Marconi 15:01:33 Meeting started Tue Jan 14 15:01:32 2014 UTC and is due to finish in 60 minutes. The chair is flaper87. Information about MeetBot at http://wiki.debian.org/MeetBot. 15:01:34 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 15:01:37 The meeting name has been set to 'marconi' 15:01:40 o/ 15:01:55 #link https://wiki.openstack.org/wiki/Meetings/Marconi#Agenda 15:02:06 #topic actions from last meeting 15:02:40 so, most of the action items where on kurt 15:02:55 it also seems like he was able to complete most of the actions, too. 15:03:00 there's one on me which I didn't address. I'll do it right after this meeting 15:03:11 #action flaper87 to document service catalog name - make it canonical or whatever 15:03:16 alcabrera: right 15:03:36 I saw the updates on the graduation bps 15:03:59 Did you guys reviewed 1.1 spec ? 15:04:04 Any comments? 15:04:08 #link https://blueprints.launchpad.net/marconi/+spec/graduation 15:04:20 I have not reviewed that spec - recently, anyway. 15:04:25 I need to look at v1.1 15:04:34 My last impression, of long ago, was that it was looking good. 15:04:41 #link https://wiki.openstack.org/wiki/Marconi/specs/api/v1.1#Changes_from_v1.0 15:04:57 flaper87: thanks! 15:05:04 I'm very happy about the required x-project-id 15:05:05 alcabrera: agreed, that was my impression too! We also discussed the health API a bit 15:05:21 also, that the client-id is required across the board is also cool. more consistent. 15:05:29 and we finally sorted out the message_max_size thing 15:05:53 yes! :) 15:05:57 #info the spec 1.1 is looking good! We should go through it one more time and have a final review next week 15:06:02 +1 15:06:11 re: max message size 15:06:34 it's been simplified nicely, so that we limit based on content length, rather than invesitigating the mesasge contents 15:06:49 so, lemme wrap that up in an #info 15:06:57 alcabrera: go ahead 15:06:58 thanks 15:07:17 #info new message max length approach - simply limit based on content length rather than investigating messages 15:07:22 all set 15:07:24 ok, next topic 15:07:29 #topic I-2 status 15:07:32 #link https://launchpad.net/marconi/+milestone/icehouse-2 15:07:50 we still have a bunch of open bugs 15:08:13 Oh yes. 15:08:16 #info open bugs: 3 -> High 4 -> Medium 15:08:26 we should all focus on close them all 15:08:40 There are also a bunch of blueprints under development 15:08:57 let's check the progress on those. 15:08:58 and they need to be updated 15:09:05 alcabrera: out of my head 15:09:06 thanks! 15:09:08 :D 15:09:10 :) 15:09:17 Evaluate Pecan FW ? 15:09:29 balajiiyer has been working on that. He won't be able to join us this morning. 15:09:33 I'd say slow progress atm. 15:09:51 #link https://blueprints.launchpad.net/marconi/+spec/pecan-framework 15:10:00 #info pecan-framework slow progress 15:10:03 I'll update the BP 15:10:18 #link https://blueprints.launchpad.net/marconi/+spec/tempest-integration 15:10:20 malini: ^ 15:10:21 thanks! 15:10:27 I saw some of your patches in devstack landed 15:10:29 congrats! 15:10:33 thanks for the hard work there 15:10:34 flaper87: thanks :) Reviews are the bottleneck 15:10:45 malini: yeah, :( 15:10:46 So I wud still rate is Slow Progress 15:10:50 it* 15:11:03 marconi is a low priority for the tempest team 15:11:12 #info tempest-integration slow progress 15:11:25 flaper87: Do you think I shud push to have a higher priority? 15:11:30 I've +1'd all the things on that one, malini. :) 15:11:41 it shouldn't be low priority, TBH 15:11:46 See https://blueprints.launchpad.net/tempest/+spec/add-basic-marconi-tests 15:11:48 it's just like every other project 15:12:01 I do think gate issues and blockers should have a higher priority than marconi 15:12:18 but adding marconi support to tempest should have a high priority 15:12:35 I'll try to get tht bp upgarded to high 15:12:39 I see they're targeting i-3 15:12:46 which means we should move our bp to i-3 15:13:16 cool - consistency + 1 15:13:20 done 15:13:29 cross transport 15:13:29 thanks! 15:13:32 cpallares: update ? 15:13:35 #info tempest integration targeting i-3 15:13:39 #link https://blueprints.launchpad.net/marconi/+spec/cross-transport-api-spec 15:14:01 I'm working on the generic communication between the api and the storage 15:14:05 alcabrera: thnx 15:14:19 cpallares: I moved it to i-3, it won't be ready for i-2 15:14:26 ok 15:14:32 #info moved cross=transport-api-spec to i-3 15:14:40 cpallares: anything blocking you? 15:14:53 (besides me) 15:14:53 :D 15:15:17 Nope :) 15:15:24 not even me... good! 15:15:32 haha 15:15:45 #info cross-transport-api-spec moving slow 15:15:48 #link https://blueprints.launchpad.net/marconi/+spec/load-test 15:15:52 malini: ^ ? 15:16:00 it's targeted for i-2 and good progress 15:16:07 I don't see any patch 15:16:13 I dont know why I never submitted a patch for tht :-$ 15:16:22 I have everything in the rackerlabs repo 15:16:30 I'll submit a patch today for tht 15:16:36 Is it something you can do this week and we can get approve before i-2 ? 15:16:44 I can do it this week 15:16:46 malini: is i-2 still realistic ? 15:16:57 when is i2 deaddline ? 15:17:03 Jan 23rd 15:17:13 next week 15:17:24 I can get it in this week 15:17:31 its justa bunch of xmls & readme 15:17:33 if you submit it today, we can all review it and get it merged before next week 15:17:38 sure 15:17:39 malini: WHAT? 15:17:42 you said xml? 15:17:44 just a 15:17:45 forget it 15:17:47 yes :) 15:17:51 that won't get in, EVER! 15:17:53 :D 15:17:57 * flaper87 obviously kidding 15:18:05 lol 15:18:06 You'll get a special json poptart 15:18:15 pls, add 1 line to the bp explaining this 15:18:17 :) 15:18:20 ok 15:18:34 {"to": "flaper87", "contents": ["$"]} 15:18:39 :D 15:18:53 #info load-test bp, good progress! i-2 realistic but kinda mission impossible... pan pan panpan panpan (mission impossible soundtrack) 15:19:07 :) 15:19:08 #link https://blueprints.launchpad.net/marconi/+spec/perf-testing 15:19:15 erm, Oz ? ^ 15:19:18 is that still valid? 15:19:25 does any of you know? 15:19:31 oz is away this morning 15:19:41 But I dont think its possible for i2 15:19:54 is icehouse realistic at all? 15:20:03 if no, I'll remove it from the Ith series 15:20:33 It'll be better to move it from Ith 15:20:40 malini: agreed 15:21:07 kgriffs: o/ 15:21:08 #info removed perf-testing from Ith 15:21:10 #link https://blueprints.launchpad.net/marconi/+spec/limit-number-queues 15:21:14 o/ 15:21:22 that hasn't even been approved yet 15:21:24 * kgriffs *really* hates traffic 15:21:24 kgriffs: yo yo! 15:21:27 how ya doing? 15:21:36 sorry guys, the commute was horribad today 15:21:46 I removed it from i2 15:21:46 no worries 15:21:47 kgriffs: thts what happens when you move away from ATL :| 15:21:51 lol 15:21:54 kgriffs: no worries! 15:21:54 ATL was bad too 15:22:03 * kgriffs wishes he worked from home 15:22:04 tht didnt work :( 15:22:07 #info limit-number-queues removed from i2 15:22:21 #link https://blueprints.launchpad.net/marconi/+spec/http-error-bodies 15:22:25 #chair kgriffs 15:22:25 Current chairs: flaper87 kgriffs 15:22:36 kgriffs: we're going through https://launchpad.net/marconi/+milestone/icehouse-2 15:22:39 flaper87: carry on 15:22:43 we're at http-error-bodies 15:23:08 I guess we could move it to i-3 15:23:10 flaper87: you can keep driving the meeting if you like 15:23:31 kgriffs: sure, let me finish with the bps and then you can go on if you want 15:23:31 flaper87: +1 for moving it 15:23:36 to i-3 15:23:52 so, this sort of falls under "user docs" 15:24:03 #info http-error-bodies moved to i-3 15:24:08 #link https://blueprints.launchpad.net/marconi/+spec/redis-storage-driver 15:24:09 so, I guess that means "low" but we bump it up to "high" for Juno 15:24:17 kgriffs: agreed 15:24:25 alcabrera: ? 15:24:25 +1 15:24:46 oh 15:24:49 that 15:24:51 yes 15:24:53 it's been very delayed 15:25:01 so it's in the same state as a month ago 15:25:06 alcabrera: i-3 then ? 15:25:08 boooooo 15:25:14 lol 15:25:20 j-1, realistically 15:25:21 btw, is this somthing we still want to pull into Marconi's code base ? 15:25:32 Redis you mean? 15:25:40 hmmm 15:25:43 yep 15:25:50 mmm 15:26:02 I remember we discussed something about keeping the num of sotrage/transport drivers low 15:26:05 On the one hand, having a memory-mostly driver would be a nice analogue to disk-stroage targeting drivers 15:26:16 On the other hand, it's another driver to add to core. 15:26:24 *storage 15:26:26 I'm not against having it 15:26:35 I think this is really useful when you start talking queue flavors 15:26:43 and we don't have that feature yet either 15:26:47 megan_w: thoughts? 15:26:57 what is the priority on this? 15:27:00 I think we should've 1 nosql, 1 rdbms, 1 memory and 1 amqp(or some real messaging backend) 15:27:09 kgriffs: just moved it out from ith 15:27:15 kgriffs: sounds like Jth thing 15:27:24 flaper87: that's the logic I was considering. :) 15:27:33 yeah, TBH I don't think we will have time to do the "1 memory and 1 AMQP/bridge" 15:27:36 ah, flavors you mean, erm dunnon if we have a priority for that 15:27:38 (in Icehouse) 15:27:49 flaper87: should be a bp for flavors 15:27:51 not enough time for that, agreed 15:27:58 that should be scheduled to land same time as Redis 15:28:18 fwiw, I approved the redis patch, not scheduled yet, though. 15:28:30 #link https://blueprints.launchpad.net/marconi/+spec/nose-testr 15:28:31 #link https://blueprints.launchpad.net/marconi/+spec/marconi-queue-flavors 15:28:40 (fwiw ^^^) 15:28:43 cool 15:28:50 nose-testr is getting close 15:28:54 (there is no "Juno" series yet - we can schedule those later) 15:29:03 good progress, pypy decided to block me, though! 15:29:14 hopefully, I'll sort that out this week 15:29:41 #link https://blueprints.launchpad.net/marconi/+spec/make-ttl-optional 15:29:49 kgriffs: yup, I'm just removing the series for now 15:30:08 I haven't started working on that (AFAIK) :P 15:30:20 so, I guess i-3 15:30:58 I think it makes sense for Ith, though. 15:31:04 so keeping it in the Ith series 15:31:07 +1 15:31:07 just moving it to i-3 15:31:14 yes, since it requires api version rev 15:31:20 so we want to get that in along with v1.1 15:31:24 #link https://blueprints.launchpad.net/marconi/+spec/flag-private-log-data 15:31:26 kgriffs: agreed 15:31:38 that bp looks empty :P 15:32:02 I just removed the i-2 milestone 15:32:07 hmmm 15:32:13 let's see... 15:32:24 kgriffs: mind explaining it a bit 15:32:26 ? 15:32:29 at the moment, are we logging anything we shouldn't? 15:32:42 basically, we don't want to log sensitive data by accident 15:32:43 is it related to the logging discussion we had in #solum ? 15:32:47 flaper87: yes 15:32:53 cool 15:33:15 The wiki page linked to is very informative 15:33:22 alcabrera: agreed! 15:33:25 really, oslo logging needs to support designating sensitive data 15:33:38 the bp says: Track what solum is doing. 15:33:45 Will it require doing some coding? 15:33:49 anyway, I kinda think this can wait for i3, low or even Juno 15:33:50 We can audit our log calls using `grep` and find out where we're logging sensitive details. 15:33:58 I'm +1 for j-1 15:34:02 kgriffs: it didn't have a series assigned 15:34:08 i know we aren't logging creds 15:34:12 or message bodies afaik 15:34:14 so, I removed the milestone and I'd lean towards Jth 15:34:21 kgriffs: correct 15:34:21 flaper87: make it so 15:34:27 we don't even have creds, AFAIK! 15:34:29 :P 15:34:32 I mean, keystone but... 15:34:34 tokens 15:34:42 yeah, we barely touch them 15:34:49 (which is a good thing) 15:34:52 s/touch/use/ 15:34:57 kgriffs: agreed 15:35:05 kk 15:35:17 ok, so for i-2 we now have 3 bps pending 15:35:25 pecan sounds like it will be moved to i-3 15:35:33 and a bunch of bugs that we should fix asap 15:35:45 those seem like good fit for i-2 15:36:06 flwang is not around 15:36:08 nor is ykaplan 15:36:18 so, the extra items in the meeting agenda can't be addressed 15:36:30 #link https://launchpad.net/marconi/+milestone/icehouse-2 15:36:45 kgriffs: want to talk about Push-based message consumption ? 15:37:06 sure 15:37:18 #info Today we have a mid-cycle incubation review 15:37:26 #topic Push-based message consumption 15:37:31 kgriffs: floor is yours 15:37:41 so, we sort of touched on this a couple weeks back 15:38:18 basically, we need to decide first whether we need it, and second the simplest way to do it that could possibly work 15:38:33 so, I think we need it for notifications, if nothing else 15:38:41 thoughts? 15:38:51 hmmmm 15:39:09 we do need it for notifications 15:39:13 there's no way around that 15:39:37 let me play devil's advocate 15:39:41 mmh, the way I see it is that: 1) It's transport specific, 2) Although we need it for notifications, we shouldn't limit it to notifications 15:39:48 dunno if that makes sense 15:40:31 flaper87: what you say makes sense 15:40:35 why do we need it for notifiactions? 15:40:41 * kgriffs takes off devil horns 15:40:45 heh 15:40:52 so, good question 15:41:13 If we need it for notifications, it's because we want to take the burden of polling for new notifications off the client. 15:41:28 can we quantify that burden? 15:41:57 let us assume the following: 15:42:00 I'd say it's more related to the technical requirement than the usability itself 15:42:02 yup, though it's tricky. I'm thinking of mobile, battery-powered clients on one end. 15:42:10 Marconi can serve a poll request in 20-30ms 15:42:30 for example, it's cheaper to keep the connection open than opening every time 15:42:38 Marconi can enable keep-alive for extended periods (say 2-3 minutes) 15:42:49 also, push notifications imply a nearer real-time messaging delivery 15:43:13 consider that a client gets batches of messages, not just one at a time 15:43:32 so, if I get a new big batch every 20-30 ms 15:43:40 wait, are we just talking about http or transports in general ? 15:43:45 and I am pushing to something like SMS, email, APN 15:43:59 the bottleneck is going to be the target sink, not the polling 15:44:15 flaper87: specifically, notifications with HTTP 15:44:25 this is a thought experiment to see if it is viable 15:44:33 ohhhhhh, that changes my thoughts, then! 15:44:34 (without long-polling or websocket) 15:45:05 if we enable keep-alive you're doing long-polling 15:45:11 s/you're/we're/ 15:45:25 there's also this coming in the not-too-distant future: http://http2.github.io/http2-spec/#PushResources 15:45:35 ah, yes 15:45:36 http 2.0 15:45:37 w00t 15:45:40 :) 15:45:41 so, http specific notifications 15:45:46 right 15:45:50 I agree with you 15:45:51 so, here is what I think 15:46:25 you could do something pretty decent just using polling plus keep-alive 15:46:35 the pro is this keeps the backend simple 15:46:41 the important bit here is: "We're not preventing anyone to do it, we just don't want it in the code base because polling is good enough based on these assumptions" 15:46:58 flaper87: +1 15:47:02 (i.e., keeping track of everyone listening to different channels on different boxes and notifying them when something happens is a PITA) 15:47:20 flaper87: so, right 15:47:37 kgriffs: marconiclient itself can handle this, which removes the burden from the user 15:47:42 if we do implement true push messaging 15:48:14 that driver is going to be pretty complicated unless you assume that every client that is listening on a particular queue is routed to the same physical box 15:48:34 and even then, you would probably want the same proc, but you could use IPC 15:48:48 anyway, it is doable with zmq and stuff, just not trivial 15:48:55 kgriffs: agreed 15:49:06 kgriffs: correct! 15:49:18 the overall architecture suggests that using polling is better 15:49:37 (Marconi's architecture, I mean) 15:49:43 so, to flaper87 15:49:48 i mean 15:49:52 to flaper87's point 15:50:00 we don't want to prevent someone from doing all that work 15:50:25 but I don't feel like it is high priority to do this within Marconi for a while yet, at least 15:50:35 agreed 15:50:42 I think it's still premature to think about this 15:50:54 we don't want to get ahead of ourselves with this specific feature 15:51:09 I think time and users will dictate what should happen 15:51:12 with, say, a zmq transport driver 15:51:16 I think we'll get a better idea about what we need when we start working on notifications 15:51:21 we can just have RPC style request-response 15:51:28 kgriffs: I did that in glance 15:51:30 :) 15:51:33 if we want to do AMQP then we kinda have to do push, right? 15:51:42 glance-api -> glance-registry (RPC over http) 15:51:46 anyway 15:51:53 kgriffs: yea 15:51:55 kgriffs: yeah 15:52:06 tbh, I have had more people request JMS than AMQP 15:52:13 but that may just be the circles I walk in 15:53:01 #agreed Lets not worry about push messaging support. The overall architecture suggests that polling is good enough for the current status of the project and use cases 15:53:12 let's run with that 15:53:14 cool 15:53:17 awesome 15:53:20 we will have to actually try it to be sure 15:53:23 any other thoughts on push-based messaging? 15:53:31 that's it from me 15:53:35 #topic Open Discussion 15:53:41 let me just plug this: https://wiki.openstack.org/wiki/Marconi/specs/api/v1.1 15:53:43 Everyone, remember today's TC meeting 15:53:49 megan_w: ^^^ 15:53:54 I have it on my calendar! :D 15:53:57 where's balaji? 15:54:12 he's out for the day, I think 15:54:14 oh 15:54:17 ok 15:54:21 Georgia Tech or some such. :) 15:54:41 malini: have you had a chance to review the v1.1 draft? 15:54:56 I'd like everyone to review the v1.1 spec for next time 15:54:57 I did not :( 15:55:05 I'll review the 1.1 spec 15:55:06 kk, no worries. Just curious 15:55:12 kgriffs: we kinda talked about it. The summary of our comments is: "It looks good, we need to go through it one more time" 15:55:23 but it looks good and we're all happy 15:55:27 malini: can you copy the error responses tables ? 15:55:40 gotta run 15:55:43 I would like a v1.1 version. I think status codes may change in a couple spots 15:55:47 kgriffs: could you end the meeting 15:55:51 flaper87: thanks for leading the mtg! 15:55:53 will do 15:55:59 ok folks, anything else? 15:56:01 I'll update the minutes. 15:56:03 :) 15:56:14 kgriffs: https://wiki.openstack.org/wiki/Marconi/specs/api/v1/errors 15:56:16 alcabrera: ur my hero! 15:56:19 lol 15:56:41 malini: right, let's copy that and but under /v1.1/errors 15:56:47 s/but/put 15:57:03 kgriffs: ok 15:57:13 #action malini to copy v1/errors page to v1.1/errors 15:57:16 thanks! 15:57:19 #endmeeting