Thursday, 2014-07-24

*** shakamunyi has joined #openstack-marconi00:43
openstackgerritA change was merged to openstack/marconi: Remove check queue exists endpoint  https://review.openstack.org/10805100:54
*** oz_akan has joined #openstack-marconi01:01
*** mwagner_lap has joined #openstack-marconi01:32
*** nosnos has joined #openstack-marconi01:43
openstackgerritA change was merged to openstack/marconi: API v1.1 - Encapsulate message post bodies in a JSON object  https://review.openstack.org/10879502:21
*** haomai___ has quit IRC02:36
peoplemergewpf: ping02:51
*** prashanthr_1 has joined #openstack-marconi03:07
prashanthr_1flaper87: Good morning :)03:09
*** nosnos has quit IRC03:15
*** oz_akan has quit IRC03:42
*** vkmc has quit IRC03:43
*** prashanthr_1 has quit IRC03:45
*** chandankumar has joined #openstack-marconi03:53
*** oz_akan has joined #openstack-marconi04:05
*** nosnos has joined #openstack-marconi04:18
*** prashanthr_ has joined #openstack-marconi04:21
*** oz_akan has quit IRC04:44
*** oz_akan has joined #openstack-marconi04:45
*** oz_akan_ has joined #openstack-marconi04:46
*** oz_akan has quit IRC04:46
*** chandankumar has quit IRC04:49
*** oz_akan_ has quit IRC04:50
*** prashanthr_1 has joined #openstack-marconi04:52
*** prashanthr_ has quit IRC04:52
*** chandankumar has joined #openstack-marconi04:59
*** haomaiwa_ has joined #openstack-marconi05:13
*** k4n0 has joined #openstack-marconi05:24
*** celttechie has joined #openstack-marconi05:28
*** celttechie is now known as celttechie_05:32
*** prashanthr_1 has quit IRC05:39
*** prashanthr_ has joined #openstack-marconi05:41
*** celttechie_ has left #openstack-marconi05:43
*** prashanthr_ has quit IRC05:44
*** prashanthr_ has joined #openstack-marconi05:46
*** celttechie has joined #openstack-marconi05:51
*** celttechie has quit IRC06:03
*** celttechie has joined #openstack-marconi06:04
wpfpeoplemerge:  pong , sorry,  this is my remote desktop , I forgot to  set it "away", and now , I am back -:006:04
openstackgerritOpenStack Proposal Bot proposed a change to openstack/marconi: Imported Translations from Transifex  https://review.openstack.org/10919306:07
*** oz_akan has joined #openstack-marconi06:16
*** celttechie has quit IRC06:18
*** oz_akan has quit IRC06:21
prashanthr_flwang: Hi Good afternoon :)06:33
*** ChanServ changes topic to "OpenStack Queuing and Notification Service || Smile :D || Meetings every Tuesday @ 15:00 UTC || Wiki: https://wiki.openstack.org/wiki/Marconi || Paste: http://paste.openstack.org/ || Send messages and make some noise :D"06:33
flwangprashanthr_: hi06:33
prashanthr_flwang : Can i know how expired messages are purged from the mongodb database ?06:33
prashanthr_is it daemon based ?06:34
*** pquerna has left #openstack-marconi06:38
wpfwith a quick look, I did not find the *purged* code (I remembered I ever saw some), are they just be filtered out?06:44
prashanthr_wpf: Ohh okay. So messages remain in the system until someone claims them06:46
wpfexpired messages can't be claimed I thought06:50
wpfhoping flwang can provide some more details about the purge  -:)06:50
wpfprashanthr_:  you did not find the code to purge them either ?06:51
prashanthr_wpf: Nope.06:53
prashanthr_I used to just filter out the expired messagesz06:53
prashanthr_when kgriffs|afk pointed it out06:54
prashanthr_and in Redis memory is very crucial06:54
prashanthr_so i think it's better to get rid of the expired messages06:54
wpfin the sqlalchemy code, even the *purged* codes were commented, but the idea is to delete those one when posting messages06:56
*** shakamunyi has quit IRC07:03
prashanthr_wpf: I did not get it sorry.07:06
wpfthe idea to purge the expired messages in sqlalchemy code is using the sql statement to delete them when posting messages07:09
wpfbut the code was commented out07:09
prashanthr_wpf: Ohh okay. I have not checked the sqlalchemy code at all.07:12
* prashanthr_ is lazy 07:12
*** oz_akan has joined #openstack-marconi07:17
*** oz_akan_ has joined #openstack-marconi07:21
*** oz_akan has quit IRC07:22
*** oz_akan_ has quit IRC07:26
*** prashanthr_ has left #openstack-marconi09:16
*** flwang_ has joined #openstack-marconi09:42
*** k4n0 has quit IRC10:07
flwang_flaper87: around?10:17
*** k4n0 has joined #openstack-marconi10:21
*** prashanthr_ has joined #openstack-marconi10:30
*** oz_akan has joined #openstack-marconi10:58
*** k4n0 has quit IRC11:01
*** oz_akan has quit IRC11:03
*** chandankumar has quit IRC11:17
*** k4n0 has joined #openstack-marconi11:19
*** chandankumar has joined #openstack-marconi11:32
*** amitgandhi has joined #openstack-marconi11:44
*** amitgandhi has joined #openstack-marconi11:44
*** flwang_ has quit IRC11:56
*** k4n0 has quit IRC11:59
*** oz_akan has joined #openstack-marconi11:59
*** oz_akan has quit IRC12:01
*** nosnos has quit IRC12:09
*** amitgandhi has quit IRC12:10
*** denis_makogon has quit IRC12:16
*** denis_makogon has joined #openstack-marconi12:18
*** tongli has joined #openstack-marconi12:34
*** abettadapur has joined #openstack-marconi12:37
*** sriram has joined #openstack-marconi12:45
*** catherin_ has joined #openstack-marconi13:06
*** chandankumar has quit IRC13:10
*** jmckind has joined #openstack-marconi13:11
*** catherin_ has left #openstack-marconi13:13
*** mwagner_lap has quit IRC13:14
*** vkmc has joined #openstack-marconi13:16
*** vkmc has quit IRC13:16
*** vkmc has joined #openstack-marconi13:16
*** amitgandhi has joined #openstack-marconi13:20
*** amitgandhi has quit IRC13:20
*** amitgandhi has joined #openstack-marconi13:21
*** celttechie has joined #openstack-marconi13:23
*** amitgand_ has joined #openstack-marconi13:25
*** amitgandhi has quit IRC13:25
vkmcmorninggg13:30
*** Obulpathi has joined #openstack-marconi13:32
prashanthr_vkmc: Good mrng :)13:33
*** oz_akan has joined #openstack-marconi13:36
vkmcprashanthr_, morning!13:37
*** whenry has joined #openstack-marconi13:47
*** mkoderer has quit IRC13:57
*** flwang_ has joined #openstack-marconi13:57
*** flwang_ has quit IRC14:01
*** whenry has quit IRC14:09
*** whenry has joined #openstack-marconi14:27
*** cpallares has joined #openstack-marconi14:28
*** rwsu has joined #openstack-marconi14:28
*** shakamunyi has joined #openstack-marconi14:31
vkmccpallares, o/14:32
cpallareshey vkmc!14:33
cpallaresvkmc: how are you?14:33
vkmccpallares, all good and you? :)14:33
*** mwagner_lap has joined #openstack-marconi14:38
cpallarescpallares: pretty good :)14:39
*** tonytan4ever has joined #openstack-marconi14:39
*** malini has joined #openstack-marconi14:42
*** kgriffs|afk is now known as kgriffs14:43
*** tonytan4ever has quit IRC14:44
vkmcI'm watching flaper87 talking about brokers https://www.youtube.com/watch?v=pYzIOrAkb-A14:45
*** amitgand_ has quit IRC14:45
vkmche rocked it14:45
*** whenry has quit IRC14:46
*** sriram has quit IRC14:51
*** shakamunyi has quit IRC14:53
kgriffsprashanthr_: as a first step, you could trigger deleting expired messages by each message post or message listing operation.14:55
kgriffshowever, that could add a lot of load to the system14:55
*** Obulpathi has quit IRC14:55
kgriffsso, the better long-term solution is to have some kind of separate GC daemon that runs on the box14:55
kgriffsand every, say, 60 seconds it deletes all expired messages.14:56
prashanthr_kgriffs: Good morning ! :)14:56
prashanthr_Actually the deletion of the messages can be scheduled using setex call on the message id14:56
*** abettadapur has quit IRC14:56
prashanthr_but i also maintain a list of all messages in a queue14:57
prashanthr_which cannot be removed using the setex call14:57
kgriffsoh, I see.14:57
* kgriffs learned something new14:57
kgriffsguess that would make sense to have setex since people often use redis in place of memcache14:57
kgriffsbut yeah, there will need to be a way to periodically clean up list of all messages in the queue14:58
*** malini has quit IRC14:58
kgriffsI suppose another way is to keep a timestamp in there of the last cleanup time14:58
kgriffsthen on each message post operation, check the timestamp and also do a cleanup if 60 seconds has elapsed14:58
kgriffsbut that would still add some overhead to the post operation14:59
kgriffsprashanthr_: good evening, btw!14:59
prashanthr_kgriffs: he he thanks :) I tried to use a technique similar to that for claims. But alcabrera|afk suggested that modules must have decoupling.14:59
prashanthr_and perform just one simple function15:00
kgriffsok15:00
kgriffswhat do you think about having a GC daemon?15:00
kgriffsor it could just be a script that we tell people to put in crontab15:01
prashanthr_kgriffs: I think the crontab idea would be great15:01
prashanthr_as it would allow people the flexibility to schedule15:01
kgriffsyep15:02
kgriffsit's pretty simple, so doesn't seem to be much value having the script do its own scheduling15:02
*** chandankumar has joined #openstack-marconi15:02
*** whenry has joined #openstack-marconi15:02
prashanthr_kgriffs: agreed.15:03
prashanthr_Also a message lives on until there is a valid claim on it15:03
prashanthr_am i right here ?15:03
*** tonytan4ever has joined #openstack-marconi15:04
kgriffsactually, no it expires according to it's TTL15:09
prashanthr_kgriffs: Sure.15:10
prashanthr_I will start working on it.15:10
kgriffsthe caveat is15:10
kgriffsthat when you claim a message, it's lifetime will be extended so it does not expire before the claim does15:10
*** whenry has quit IRC15:10
kgriffsyou can see that logic in the mongo driver15:10
kgriffs(actually, it is extended to claim expiration + grace seconds)15:11
kgriffsanyway, some people will use marconi simply as a message feed and not claim messages15:12
kgriffssort of the same way you might use http://atomhopper.org/15:12
prashanthr_kgriffs: Ahh this makes sense. I shall have a look into atom hopper15:12
prashanthr_also i had a look at trollius15:13
kgriffsso yeah, messages need to expire eventually in that use case and be deleted because you don't want the data set to grow forever.15:13
kgriffsprashanthr_: oh yeah? I just came across it recently, but it looked like a promising way to get async I/O that works across py2 and py315:13
kgriffswhat have you found out about it?15:13
prashanthr_kgriffs: Yes i saw a video by Guido Van Rossum about it15:13
prashanthr_he just explains the asyncio part which is just py3.3+ compat15:14
prashanthr_but trillio works along the same lines15:14
kgriffsright15:14
prashanthr_I think we can add a AsyncConnection class to redis-py15:14
prashanthr_which is basically an extension of the trollius.Protocol15:15
kgriffswe may also need to submit a patch to pymongo so we can have async support on py3, but that is a project for another day15:15
kgriffs(FWIW)15:15
prashanthr_which can send and receive data from the redis server15:16
kgriffsprashanthr_: the asyncio stuff in 3.4 is pretty elegant, but yeah, depends on py3 language features15:16
kgriffsprashanthr_: sounds good15:16
prashanthr_kgriffs: Do we have to work with redis-py community for this15:17
prashanthr_?15:17
prashanthr_or we just extend it with our driver ?15:17
kgriffsprashanthr_: wait, I thought there was a way to do it without patching redis-py?15:17
kgriffshttps://github.com/andymccurdy/redis-py#connections15:17
kgriffs"You can create your own Connection subclasses as well."15:18
prashanthr_true.15:20
prashanthr_kgriffs:Right.15:20
prashanthr_Currently I am just developing it with a fork of the redis-py15:22
prashanthr_https://github.com/PrashanthRaghu/redis-py15:22
prashanthr_I will write a basic connection and then test it from there15:22
prashanthr_It currently has a dummy AsyncConnection class15:22
*** wirehead_ has quit IRC15:28
*** wirehead_ has joined #openstack-marconi15:29
*** prashanthr_ has quit IRC15:30
*** oz_akan has quit IRC15:34
*** denis_makogon has quit IRC15:38
vkmckgriffs, morning!15:41
kgriffshowdy!15:41
vkmcI'm working on the client and finding some stuff that may need to be changed15:42
vkmcin the API we removed the exists method for queues, should we remove it in the client as well?15:43
kgriffswell, it still exists in the 1.0 API15:43
vkmcyeah, that was my concern15:43
* kgriffs didn't intend to pun15:43
* kgriffs but finds it funny anyway15:43
vkmclol15:44
vkmcgotcha15:44
kgriffsvkmc: so... the question is what is our versioning strategy for the client?15:44
vkmcyeah :p15:44
kgriffsideally, the client can query to server to figure out which apis are available15:44
kgriffsI think there was a bp somewhere for exposing an endpoint for that15:45
kgriffsbut sans adding an api discovery endpoint, the client could just try to get homedocs for all known versions15:45
openstackgerritA change was merged to openstack/marconi: Imported Translations from Transifex  https://review.openstack.org/10919315:46
*** cpallares has quit IRC15:48
kgriffsoh, interesting. I just looked at this15:48
kgriffshttps://github.com/openstack/python-marconiclient/blob/master/marconiclient/queues/client.py15:48
kgriffslooks like some thought has already been put into versioning15:48
vkmcit looks like that15:49
kgriffsso, by default you could return the most recent version15:49
kgriffsand if you wanted to be even smarter, the default would probe the server to find out what it supports, and by default return the latest support api15:49
vkmcI got confused with the client version and the api version15:49
kgriffslooks like it may be a 1:1 relationship?15:50
kgriffsv1 api, v1 client15:50
kgriffsv1.1 api, v1.1 client15:50
vkmcI noticed we have many features that has been modified in v1.1, so I was submitting bugs accordingly15:50
vkmcyeah15:50
vkmcbut we have v1 currently15:50
vkmcso it's ok to have the exist method and have shards instead of pools15:51
kgriffsvkmc: we may just want to create one blueprint for all the client work to implement v1.115:51
vkmckgriffs, sounds good15:51
kgriffsFWIW, I listed the client work under API v1.1 on the roadmap: https://wiki.openstack.org/wiki/Roadmap_(Marconi)#Upcoming_milestones15:51
vkmcright now there are some bps for v1.115:51
kgriffsdid you see one for the client specifically? I can't remember if one was ever created15:52
kgriffsI haven't yet gotten launchpad all cleaned up for j-315:52
vkmckgriffs, https://blueprints.launchpad.net/python-marconiclient/+spec/api-v1.115:53
kgriffsbut I did update the roadmap so we can figure out what belongs15:53
kgriffsah, cool15:53
kgriffsI will move to j-315:53
vkmc:) cool15:53
vkmcI could work on that, for j-315:53
vkmcright now I'm on the CLI but I found some dependencies15:54
kgriffsok, cool15:54
kgriffsthanks!15:54
vkmcI think that the CLI for v1 will be ready for j-3... the client for v1.1 depends on all those bps15:54
kgriffsyep, we have tons of work to do. :p15:55
vkmcwell, the shards to pool thing affects v1 too :) so the report is valid15:56
vkmchaha we will do it!15:56
vkmcbtw kgriffs, I'm a bit worried about the AMQP transport15:56
vkmcdriver15:56
vkmcI saw it in the roadmap for J-315:56
vkmcbut I'm still not sure how we are going to deal with the binary messages AMQP can handle15:57
vkmcwhat do you think would be better for Marconi in this stage?15:57
kgriffshmmm15:57
*** flwang_ has joined #openstack-marconi15:58
vkmcwe also have some limitations for the claim support15:58
vkmcso... the AMQP transport would support the prod/cons pattern only15:59
*** cpallares has joined #openstack-marconi15:59
kgriffsoic16:00
kgriffsso, a few thoghts16:00
kgriffsthoughts16:00
kgriffsit's hard to know what is going to work sometimes before trying it (science!)16:01
vkmcof course :)16:01
kgriffsso, your work with AMQP has been really helpful in advancing our thinking16:02
*** flwang_ has quit IRC16:02
vkmcwell, that's good to hear16:04
vkmcif it helped for something, then I'm happy16:04
kgriffssounds like we still have more experimenting and work to do though that won't all fit in j-316:04
kgriffsis that what you were thinking?16:04
vkmcyeah, my main concern is that this was targeted for Juno and I couldn't deliver16:05
kgriffsI really think there is an AMQP + Marconi story somewhere that could be a really useful thing for the community.16:05
vkmcso I was looking for an alternative version, perhaps more limited, that would still be useful for Marconi16:06
kgriffsI think that we will get there, but it will need some more time to sort out16:06
vkmcsure16:07
kgriffsso, let me see if I can somehow indicate that on the roadmap.16:07
vkmcok :)16:08
vkmcso... for the last part of my internship I'll focus on the client16:08
vkmcso at least I can contribute with some code, in other area16:08
vkmcand continue with the research efforts for AMQP16:09
kgriffsyeah, I think the AMQP research you've done is really valuable as I said before. But for j-3 we could definitely use help on the client.16:10
vkmcwill do that then16:12
vkmcthanks kgriffs16:12
kgriffsyw16:13
kgriffsvkmc: btw, is there a wiki page somewhere that summarizes what you have learned about AMQP?16:14
kgriffsI think that would be a great to have.16:14
kgriffs...16:18
kgriffs..16:19
kgriffsBTW everyone, j-2 is done16:19
vkmckgriffs, yes there is16:21
vkmc(sorry, got distracted by the phone)16:21
vkmcI have to update it with the latest16:22
vkmchttps://wiki.openstack.org/wiki/Marconi/specs/amqp/api/v116:22
kgriffsah, cool16:22
vkmcso... I'll try to leave a POC done16:23
vkmcthe client.py mentioned in that wiki is a simple implementation of an AMQP client I'll upload too16:23
kgriffscool, that would be hugely helpful. I just want to make sure we have everything squared away by the end of the internship16:26
vkmcthat AMQP transport implementation would only be able to receive and retrieve messages in a prod/cons fashion16:26
vkmcyup16:26
vkmcas I told flaper87 last week, I'm planning to keep contributing to Marconi after the internship so I will be around to keep with this research/implementation16:27
kgriffsthat would be awesome!16:27
vkmcI'm done with college so I now can devote more time to open source :)16:27
kgriffswe have a a few contributors who have been less active lately, so it would be great to have your continued support16:27
kgriffsvkmc: oh, congrats!16:27
vkmckgriffs, thanks!16:28
kgriffsbbl16:31
*** whenry has joined #openstack-marconi16:35
*** kgriffs is now known as kgriffs|afk16:41
*** tonytan4ever has quit IRC16:42
*** chandankumar has quit IRC16:44
*** celttechie is now known as celttechie|NotHe16:54
*** rwsu has quit IRC16:58
*** rwsu has joined #openstack-marconi16:59
*** celttechie|NotHe has quit IRC17:05
* peoplemerge is still in meetings17:06
*** tonytan4ever has joined #openstack-marconi17:12
*** tonytan4ever has quit IRC17:19
*** cpallares has quit IRC17:22
*** cpallares has joined #openstack-marconi17:23
*** celttechie has joined #openstack-marconi17:27
*** kgriffs|afk is now known as kgriffs17:31
*** whenry has quit IRC17:40
*** kgriffs is now known as kgriffs|afk17:41
*** kgriffs|afk is now known as kgriffs17:42
*** flwang_ has joined #openstack-marconi17:59
*** kgriffs is now known as kgriffs|afk18:00
*** cpallares1 has joined #openstack-marconi18:02
*** cpallares has quit IRC18:02
*** flwang_ has quit IRC18:03
*** kgriffs|afk is now known as kgriffs18:13
*** cpallares1 is now known as cpallares18:46
*** mwagner_lap has quit IRC18:53
*** vkmc has quit IRC18:54
*** celttechie is now known as celttechie|NotHe19:05
*** mkoderer_cloud has joined #openstack-marconi19:09
openstackgerritKurt Griffiths proposed a change to openstack/marconi: Deprecate partial field in v1.0, remove in v1.1  https://review.openstack.org/10939119:12
*** malini has joined #openstack-marconi19:23
*** kgriffs is now known as kgriffs|afk19:26
*** celttechie|NotHe has quit IRC19:30
*** vkmc has joined #openstack-marconi19:37
*** ametts has quit IRC19:43
*** tonytan4ever has joined #openstack-marconi19:44
*** flwang_ has joined #openstack-marconi20:00
*** oz_akan has joined #openstack-marconi20:03
*** ametts has joined #openstack-marconi20:04
*** flwang_ has quit IRC20:05
*** chandankumar has joined #openstack-marconi20:10
*** chandankumar has quit IRC20:17
*** flwang_ has joined #openstack-marconi20:26
*** kgriffs|afk is now known as kgriffs20:26
*** mwagner_lap has joined #openstack-marconi20:26
*** flwang_ has quit IRC20:28
*** cpallares has quit IRC20:29
*** kgriffs is now known as kgriffs|afk20:36
*** ametts has quit IRC20:40
*** kgriffs|afk is now known as kgriffs20:44
*** jmckind has quit IRC20:56
*** mpanetta has joined #openstack-marconi20:59
*** mpanetta has quit IRC21:01
*** mpanetta has joined #openstack-marconi21:02
flwangkgriffs: ping21:21
*** tonytan4ever has quit IRC21:22
kgriffso/21:25
kgriffshey there21:25
kgriffswhats up?21:25
*** tongli has quit IRC21:29
*** malini has quit IRC21:37
*** celttechie has joined #openstack-marconi21:39
flwangjust want to confirm how the sqlalchemy remove the expired messages21:40
flwangkgriffs: somebody asked me the question yesterday, but I just know mongo driver is depends on the TTL of mongodb, but I found the code of sqlalchemdy about this is commented, am I missing anything?21:42
*** mkoderer_cloud has quit IRC21:42
*** rwsu has quit IRC21:44
*** oz_akan has quit IRC21:48
*** vkmc has quit IRC21:48
*** haomaiwa_ has quit IRC21:55
kgriffswhen the sqlite code was ported to sqlalchemy, deleting expired messages somehow fell through the cracks. The old sqlite code was copied over and commented out as a reference, but never ported to sqla21:58
kgriffshope that makes sense...21:59
kgriffs:)21:59
flwangkgriffs: did you mean sqlalchemy can support similar TTL mechanism like mongo to remove the expired messages?22:02
kgriffsno22:02
kgriffsat least, not to my knowledge22:03
kgriffswhat I mean is that the raw SQL needs to be ported to sqla syntax22:03
kgriffsflwang: I've actually got to run now, and I will be offline until Monday...thanks for helping sort this out. You now know everything I do about it. :)22:04
kgriffshave a good weekend!22:05
flwangok, see you22:05
kgriffso/22:08
kgriffscheers22:08
*** kgriffs is now known as kgriffs|afk22:09
*** vkmc has joined #openstack-marconi22:22
*** flwang_ has joined #openstack-marconi22:29
*** flwang_ has quit IRC22:33
*** balajiiyer has joined #openstack-marconi22:56
*** oz_akan has joined #openstack-marconi23:26
*** mpanetta has quit IRC23:47
*** celttechie has quit IRC23:57
*** celttechie has joined #openstack-marconi23:58

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!