19:04:57 #startmeeting marconi 19:04:58 Meeting started Thu Mar 21 19:04:57 2013 UTC. The chair is kgriffs. Information about MeetBot at http://wiki.debian.org/MeetBot. 19:04:59 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 19:05:01 The meeting name has been set to 'marconi' 19:05:10 o/ 19:05:53 ok, before we get rolling, any items folks want to add to the agenda? 19:06:21 https://wiki.openstack.org/wiki/Meetings/Marconi 19:07:39 #topic sate of the projects 19:07:52 #topic state of the project 19:08:33 so, any questions or should I just give a summary? 19:09:09 kgriffs: summary sounds good 19:09:25 some questions might come out of there 19:09:40 OK, so we have queue management mostly done 19:09:58 This week we started working on message CRUD 19:10:26 We are trying to get closure on the v1 API, but I think it's stable enough to implement on 19:10:44 Doubtless there will be tweaks in the coming months, esp. at the summit 19:11:29 We are close to getting python-marconiclient on stackforge, etc. 19:11:52 kgriffs: +2 storage drivers in good progress 19:12:05 (the foundational stuff is in a different repo) 19:12:20 yes, so we have sqlite and mongo drivers in progress 19:12:38 sqlite is pretty much for testing, demos, and stuff like that 19:12:47 kgriffs: perhaps we should send an email to openstack-dev requesting reviews for the python-marconiclient 19:13:48 flaper87: Yeah, I just got some additional feedback and a -1. https://review.openstack.org/#/c/24807/ 19:14:28 They want me to add pypi configs before merging. 19:14:33 :-/ 19:14:52 Doesn't look like a big deal. 19:15:00 jdprax: cool 19:15:37 A few more things I just wanted to mention real quick FYI. We have 3 core contributors now handling code reviews through Gerrit: Jamie (jdprax), Flavio (flaper87), and myself (Kurt). 19:16:14 We've also been busy adding blueprints, and updating the wiki. 19:16:40 Also, our code is now pep8-friendly, and we have a pretty good HACKING file (still needs a few tweaks) 19:16:56 so, there's never been a better time to become a Marconi contributor (hint, hint) 19:17:18 Our first big milestone is a demo at the Portland summit 19:17:41 (our talk was accepted) 19:17:54 w000000t 19:18:02 so, if you have any cool ideas for a demo give us a shout out in #openstack-marconi 19:18:20 OK, any questions before we move on? 19:19:23 * kgriffs listens to crickets chirping 19:19:33 #topic API - Maximum queues/page 19:20:01 OK, so as the blueprints are currently written, Marconi limits queue metadata to 4 KiB. 19:21:17 So, if we allow clients to list 100 queues at a time, that's not a ton of data - should be doable even if you buffer in memory, although we'll likely code such that everything is streamed. 19:21:46 Questions/concerns? https://wiki.openstack.org/wiki/Marconi/specs/api/v1#List_Queues 19:22:09 sounds good to me! 19:22:57 +1 19:23:01 irc://card.freenode.net:7000/#topicĀ API - Maximum queues/page 19:23:36 same deal. 4 KiB per message body, 100 messages per request returned 19:23:53 #topic API - Maximum queues/page 19:24:55 OK, so I'll take that as a "+1" 19:25:24 :D 19:25:30 :D 19:25:41 Anyone have a good ascii art cricket? 19:25:51 #topic API - Claim Messages 19:26:07 https://wiki.openstack.org/wiki/Marconi/specs/api/v1#Claim_Messages 19:26:38 so, it would be cool if everyone could read over that real quick and share your thoughts 19:29:16 any concerns there? 19:29:37 kgriffs: just re-read it and seems ok so far 19:29:50 The WARNING under Claim Messages, can you explain that a little more? 19:29:53 I'm still a bit worried about the ttl thing 19:30:03 sure 19:30:13 I don't see how you can ever know that the TTL of the claim will be less than that of the messages 19:30:43 typically your application would agree on something reasonable. 19:30:58 So producers would always set ttl of messages to X or at minimum X 19:31:29 kgriffs: so, here's an idea 19:31:40 and workers would set claims to some percentage of X, depending on average time to process a given message 19:32:05 typically you'd just set the ttl for messages to something like 5 days and the ttl for a claim to 5 minutes 19:32:31 flaper87: shoot 19:34:52 since clients don't know, as bryansd-home also pointed out, what the reasonable ttl would be and we won't be able to "make sure" that messages being claimed wont expire before the claim expires, what if don't make claimed messages expire until the claim expires as well 19:35:32 so, if the message would expire before the claim expires we would extend the message expiration / ttl 'til the claim expires 19:35:54 Sounds reasonable to me. 19:36:00 +1 19:36:06 so, adjust the claim ttl to be min(requested_ttl, max(claimed_messages_ttl)) 19:36:14 sory 19:36:16 max 19:37:22 kind of. So, if a message expires in 30s and it was claimed and the claim ttl is 60, the message's ttl would be 60s 19:37:32 we give it a 30s life coupon 19:37:35 :) 19:37:58 oic 19:37:59 sure 19:38:07 +1 19:38:17 cool, place an agreed there 19:38:19 :P 19:38:27 I like this. Then a consumer can have some confidence it'll have enough processing time to get through the message. 19:38:36 Yep. 19:38:45 bryansd-home: correct, were processing time == claim ttl 19:38:46 Whereas before it may "claim" a message with only 1s ttl... chaos 19:38:53 bryansd-home: lol 19:39:40 #agreed claim ttls should be adjusted for 30s past the largest age of the messages at claim time 19:39:49 #note use age of messages not ttl 19:40:17 no wait, that's not the idea 19:40:19 :P 19:40:24 kgriffs: I think we're talking about extending the age of messages, not claims 19:40:31 bryansd-home: correct 19:40:39 oh, oops. got it backwards 19:41:42 so, extend age of messages to 30s past expiration of the claim? 19:41:53 (if needed) 19:42:09 actually, seems like you want more fudge room than that 19:42:09 if (msg_ttl - msg_age) < claims_ttl: msg_ttl += claim_ttl - msg_ttl 19:43:02 well, consider that if workers poll once every, say, 5 minutes for work 19:43:10 but a worker dies. 19:43:24 the claim must expire and the messages still need to live long enough for another worker to pick them up 19:43:53 kgriffs: nope, because the would expire with the claim (if their life was extended) 19:44:08 but I don't want them to expire with the claim 19:44:28 kgriffs: they would have expire anyway 19:44:38 expired* 19:44:40 isn't the whole point of having a ttl on claims is so if a worker dies another can pick up the slack? 19:45:29 kgriffs: sure 19:46:03 so, the suggestion is: 19:47:14 Extend messages' expiration time to the claim's expiration time for messages that would expire before the claim does 19:47:28 flaper87: +1 19:47:30 no fudge factor? 19:47:52 so, clients would still be responsible for doing that themselves if they need it 19:48:14 kgriffs: clients would set claim's ttl 19:49:12 yes, but if a worker client dies and you want a different worker to pick up the unprocessed messages, those messages will need long enough TTLs that they don't expire before another worker picks them up and claims them, thereby extending their lifetimes again. 19:50:32 so, setting messages to expire along with a claim only solves the issue of them expiring while a claim is active, not what happens when an agent crashes. 19:51:08 kgriffs: yes but consider that if those message wouldn't have been claimed they would have expired anyway 19:51:39 I agree that it is useful to do, and avoids surprises. 19:51:58 My point is just that the latter problem needs to be addressed somehow as well. 19:52:34 basically you need some grace period 19:52:44 to ensure messages aren't dropped 19:52:57 I wonder if a claim could have a "grace" field? 19:53:18 Or should message TTLs be renewed when they get claimed? 19:53:18 the the formula would become: 19:53:32 kgriffs: what I don't understand of your concern is: Why should a claimed message have a grace period and a not claimed message shouldn't ? 19:53:47 claims_ttl: msg_ttl += claim_ttl - msg_ttl + grace 19:54:13 kgriffs: I'm ok with the "after_claim_grace" 19:54:57 kgriffs: well, you are right actually. the claimed messages deserve a grace period because the were hold from being processed by the stupid, suicide, worker 19:55:11 hah! 19:55:16 :) 19:55:32 so everyone was right! 19:55:33 so, +1 for extending and +1 for grace_post_claim 19:56:24 #agreed Extend lifetime of claimed messages to be at least as long as the lifetime of the claim itself, plus a specified grace period to deal with crashed workers 19:56:30 19:56:46 OK folks. 19:56:47 w00000t 19:56:58 we have 4 mins 19:57:03 open discussion 19:57:27 o/ 19:58:29 so, we created now a new milestone series (havana) and a new milestone (havana-1). Marconi is not part of core projects so we're not expected to follow upstream release cycles but it seems to be that it would be good to start "fitting" into those release cycles since now 19:58:47 so that we're a step ahead when the time will come for marconi to be evaluated 19:58:55 as a project, organization and blah blah blah 19:59:21 +1 19:59:32 there are some upstream milestones we're following (havana-1) and some marconi's milestones (trello) and I think they both can work together 19:59:42 actually, they would make everything better 20:00:12 so, for anyone with new ideas, proposals or whatsoever, pls, write a blueprint and lets set a milestone to that blueprint based on the workload 20:00:14 I guess we could make the summit demo "grizzly-1" 20:00:19 (which is huge, btw) 20:00:40 then everything after that would be havana milestones 20:01:19 Want to propose Marconi sessions at summit.openstack.org, but there doesn't seem to be a suitable option in the "Topics" drop-down. (https://wiki.openstack.org/wiki/Marconi) 20:01:49 Anyone know who can help us get an option added to the drop-down? Or at least a "Incubated Projects" option? 20:01:57 also, I think it'd be good to get as much feedback as possible from other people 20:02:09 ametts-atl: mmh, ttx ? 20:02:19 ametts-atl: Thierry 20:02:35 Thierry == ttx? 20:02:38 ametts-atl: yep 20:02:45 you'll find him in #openstack-dev 20:03:03 ametts-atl: at least he can point us to the right direction 20:03:18 Yeah, I just posted there. No response yet. Maybe I'll try an old fashioned email. 20:03:29 ametts-atl: sounds better 20:03:39 OK folks, anything else? 20:03:48 ametts-atl: he's based in France, IIRC, so most likely he's doing something else now 20:03:57 kgriffs: not from me! Great meeting folks 20:04:06 But you're in Italia, no?! 20:04:19 ametts-atl: yep, but I'm lifeless and workaholic :P 20:04:23 :) 20:04:45 OK, so next community meeting is 4 April 20:05:17 in the meantime, folks can catch us in #openstack-marconi 20:05:23 #endmeeting