16:06:00 #startmeeting marconi 16:06:01 Meeting started Mon Sep 9 16:06:00 2013 UTC and is due to finish in 60 minutes. The chair is kgriffs. Information about MeetBot at http://wiki.debian.org/MeetBot. 16:06:02 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 16:06:02 * ykaplan is here 16:06:05 The meeting name has been set to 'marconi' 16:06:06 #link https://wiki.openstack.org/wiki/Meetings/Marconi#Agenda 16:06:21 #topic review actions from last time 16:06:32 #link http://eavesdrop.openstack.org/meetings/marconi/2013/marconi.2013-08-26-16.07.html 16:06:49 1b 16:07:02 flaper87: ^^ 16:07:06 kgriffs: yo! 16:07:29 not much happened w.r.t creating new blueprints but I did organize some of them 16:07:33 and created dependencies 16:07:48 I'm started working on the client this week 16:07:49 ok, at least we got some forward progress there 16:07:54 yeah 16:08:02 well, actually 16:08:04 are there still things on the therpad that need to be turned into specs? 16:08:13 thinking about that again, we are not ussing Alessios code anymore 16:08:19 so, I think we can remove that action 16:08:24 w00t 16:08:37 because the required blueprints for the next steps were already eregistered 16:08:43 ok. 16:08:47 gtk 16:08:57 malini: 1c 16:09:04 as for now, we're basing the client's development on what Alej wrote on the wiki page 16:09:07 plus the blueprints 16:09:08 * flaper87 STFU 16:09:20 flaper87: cool 16:09:31 great 16:09:33 1c is almost there..I am reviewing the last two of flaper87's patches 16:09:56 ok, I will take a look as well today 16:10:29 I will skip 1d since that is bundled up in the same slew of patches 16:10:39 1e 16:11:18 I went through and attempted to update some bps, but some I wanted to get feedback on later during the mtg before I do much with them 16:11:38 megan_w: 1g 16:12:13 done! 16:12:17 w00t 16:12:19 incubation applicaiton process complete 16:12:21 16:12:23 bam 16:12:27 ok. 16:12:31 nice work everybody 16:12:35 great job, guys 16:13:35 megan_w: can you help me keep an eye on the incubation milestones? We need to be sure that we are ready to rock 6 weeks before the Icehouse release if we want to apply for graduation 16:13:49 which also means we need to start tying up loose ends a few weeks before that 16:13:55 yes. i was going to propose that we talk about that 16:14:05 megan_w: go for it 16:14:16 * kgriffs rips up the floor and gives it to megan_w 16:14:20 haha, sorry 16:14:35 let's get a spot on the wiki to keep track of things the TC wanted us to address 16:15:03 and perhaps we can review the open ones each week here. that'll make sure everyone keeps their eye on the prize 16:15:15 +1 16:15:26 flaper87: did you already make a wiki page? 16:16:18 maybe we just have an incubation section. and link to the application wiki, but focus on the ongoing work 16:16:20 hmmm, we seem to have momentarily lost Mr. Percoco 16:16:33 mmh 16:16:35 back 16:16:37 :D 16:16:43 ah, there you are! 16:16:45 kgriffs: :D 16:16:51 for the incubation taskes ? 16:16:53 tasks ? 16:16:59 right 16:17:09 no, there are some tasks listed in the incubation page itself 16:17:16 plus, I creaed bluprints for them 16:17:18 #link https://wiki.openstack.org/wiki/Marconi/Incubation#Incubation_Period_Notes 16:17:21 and tagged as essential 16:17:28 meaning, those or nothing :P 16:17:38 ok. megan_w: would you like to create a tracking page that links to those bps? 16:17:44 ja 16:18:06 I am just reading thru the TC meeting logs, did they want Tempest tests before graduation ? 16:18:07 #action megan_w to create a graduation tracking page 16:19:03 malini: yeah 16:19:04 seems like they wanted all new programs to be fully integrated into the usual CI stack before graduation 16:19:30 moving on 16:19:34 1j 16:19:41 IMHO, it shouldn't be that difficult to complete those tasks 16:19:51 it won't be* 16:19:53 kk 16:19:57 * kgriffs crosses fingers 16:20:02 ;) 16:20:04 kgriffs: 1j was done 16:20:35 so, "openstack" is now eavesdropping on #openstack-marconi, nicht? 16:20:39 yup 16:20:50 w00t 16:20:52 #link http://eavesdrop.openstack.org/irclogs/ 16:21:06 :D 16:21:16 1k. 16:21:21 flaper87: 1k 16:21:50 where are we wrt the client? 16:21:54 done, I mean, I started working on the client. I rebased cppcabrera's patch, migrated client's tox file and started working on the auth support for the client 16:22:08 ok 16:22:20 Alex_Gaynor giving us any love lately? 16:22:21 I was going to start working on the API but I would like to discuss a couple of things with Alej before working on that 16:22:47 He sent a patch to pypy-test marconiclient :D 16:22:53 ah, yes 16:23:16 ok, let's see if we can't identify some more volunteers to help out 16:23:18 hopefully, I'll have more news w.r.t the client, next week 16:23:24 cool beans 16:24:02 ok 16:24:08 next... 16:24:22 4d 16:24:43 So, I submitted a patch there 16:25:17 Got one more comment from flaper87 to "deal with" 16:25:26 * kgriffs makes a quick phone call 16:25:52 I will work on that today 16:26:18 also, I am discussing a way to make marker generating faster since find_one seems to be a bottleneck as well 16:26:24 and two more perf patches: 16:26:28 (to try) 16:26:52 1. use UNX timestamps instead of datetime objects in message records 16:27:13 ?? i'm writing 1) 16:27:19 excellent 16:27:25 I was hoping to get some help. :D 16:28:09 2. use one messages collection per project 16:28:13 3. eat mor chikin 16:28:20 anyway, I digress 16:28:22 let's keep going 16:28:26 2) you sure... 16:28:29 ? 16:28:43 zyuan: not sure, needs testing 16:28:45 i think oz_akan_ 's multi db patch is a good approach 16:28:53 yes, we need that as well 16:28:54 it use 2 collections 16:29:02 two collections or two DBs? 16:29:24 two collections, iirc 16:29:34 because 1 server can only has 1 db 16:29:48 should be two DBs to reduce lock % 16:29:56 it's like "marconi-odd" and "marconi-even" 16:29:59 one server can have many DBs, for the record 16:30:10 guys :D 16:30:12 but his test shows this appraoch can improve performance 16:30:15 yeah, 16:30:18 we can break this out after 16:30:20 no lock between collections 16:30:24 * kgriffs rings bell 16:31:05 #topic • SQL Backend 16:31:08 #link https://blueprints.launchpad.net/marconi/+spec/sql-storage-driver 16:31:19 o/ 16:31:22 1 port, 1 db. but the total throughput is somewhat limited 16:31:39 sqlalchemy core +1 16:32:09 so, after some reading, the performance penalty of using sqlalchemy shouldn't be that high 16:32:15 I think a driver shall be specific to a storage backend 16:32:27 core is basically a query constructor 16:32:29 flaper87: do we have to use the ORM? 16:32:36 kgriffs: no 16:32:37 no 16:32:44 but, I also share oz_akan_ thought. 16:32:47 * kgriffs not a fan of ORMs 16:32:47 no orm, orm die. 16:33:05 flaper87: pls. elaborate 16:33:22 what I'm not yet is whether sqlalchemy.core allows you to "customize" things based on the backend it is talking to 16:33:38 flaper87: it allows you to customize enough things 16:33:44 datatype, length 16:33:55 some questions: how complicated is it going to get with sqlalchemy? Does it have restrictions? Are we going to override it in some cases to get most out of MySQL or etc? How is the performance? 16:34:11 kgriffs: what I mean is, I'd rather have 1 backend per RDBMs but, I'm seeing more benefits from using sqlalchemy w/o the orm than having 1 backend per database 16:34:12 very simple 16:34:28 then, if someone wants to write 1 backend specific for psql and do some magic in there, then be my guest 16:35:06 as for now, I think we should focus on cover RDBMs gap w/ performance penalties 16:35:15 I think sqlalchemy.core allows us to do that 16:35:17 ok 16:35:19 now, other advantages: 16:35:28 1) Migrations 16:35:39 sqlalchemy.core even allows you to pick mysql storage backends 16:35:43 like innodb 16:35:48 2) Models tests (not sure we need this but, you know) 16:35:50 zyuan: +1 16:35:51 so it's very, low level 16:35:54 don't worry 16:36:26 zyuan: I'm not that worried about .core not allowing us to do something, I'm more worried about what the implementation will look like 16:36:36 I mean, a bunch of if backend == mysql ? 16:36:56 but lets not discuss that now, I think we should focus on whether +1 sqlalchemy or not 16:37:06 flaper87: .core keeps query the same 16:37:10 would it replace the SQLite driver 16:37:11 ? 16:37:16 question: what is out motivation to support mysql ? 16:37:17 kgriffs: yup, we can drop that 16:37:24 because we use chained expression to create queries 16:37:32 kgriffs: (memory backend) 16:37:33 cusomization happens before that 16:37:48 I like having it because pip install marconi; marconi-server is really useful for devs, library implementors 16:37:57 indeed we can do that; an engine of sqlite :memory: will do the job 16:38:12 but i want to keep the existing sqlite driver for reasoning 16:38:22 oz_akan_: 1) MySQL was chosen instead of psql because it's more comon throughout OS community 16:38:28 i sometimes forgot what i did in the driver... 16:38:31 not just projects, also deployments 16:38:31 zyuan: reasoning == prototyping? 16:38:43 zyuan: tell me about it :P 16:38:51 * flaper87 doesn't remember what he had for lunch 16:38:52 kgriffs: for reference. because i know i implemented it very precicely 16:38:54 ykaplan: you around? 16:39:18 so, ykaplan raised her hand to work on the sql backend 16:39:24 flaper87 I'm "listening" 16:39:41 oz_akan_: also since some people want a non-AGPL backend 16:39:43 she'll implement it along the lines of sqlite's but changing what needs to be changed 16:39:50 and I know this because I can read minds 16:39:56 ... 16:39:56 OK ? 16:39:58 :D 16:40:04 zyuan: we can keep it around for a while until it is no longer needed for reference 16:40:21 no jokes, I do read minds 16:40:26 ok, nevermind 16:40:28 flaper87 read my mind 16:40:30 if i do, i'll implement is based on sqlite as well, but with .core 16:40:33 * kgriffs puts on his tinfoil hat 16:40:45 kgriffs: hehe 16:40:46 lol 16:40:50 shouldn't we focus on redis instead of mysql? 16:40:59 oz_akan_: cppcabrera is working on that 16:41:04 oz_akan_: we can't focus on redis 16:41:07 but IMHO, mysql's has higher priority 16:41:07 redis will come with some caveats 16:41:14 plus, ^ 16:41:23 because redis' model does not quite fit marconi 16:41:29 but we can do it anyway 16:41:33 ^ 16:41:35 :D 16:41:40 * flaper87 likes that symbol 16:41:42 Redis is best IMHO for lots of tiny messages with short TTLs 16:41:51 so mysql is just because of non-agpl license 16:41:57 oz_akan_: precisely 16:42:02 yea. while your primry goal is public service 16:42:03 imo 16:42:04 for now 16:42:08 oz_akan_: I wouldn't say it's just because of that 16:42:26 flaper87: what are other reasons? 16:42:34 i guess it's also that some people just are more comfy with MySQL 16:42:37 ? 16:42:47 like me ") 16:42:53 :) 16:43:06 there are some scenarios where mongodb doesn't fit (know-how? storical reasons? what not?) but they already have some RDBMS (psql? mysql?) 16:43:15 kgriffs: exactly 16:43:16 :D 16:43:24 makes sense 16:43:33 and there are mongodb hatters out there 16:43:42 haters* 16:43:47 anyway... 16:43:48 #startvote use SQLAlchemy for SQL storage driver? Yes, No 16:43:49 Begin voting on: use SQLAlchemy for SQL storage driver? Valid vote options are Yes, No. 16:43:50 Vote using '#vote OPTION'. Only your last vote counts. 16:43:51 lets make it official 16:44:02 #vote Yes 16:44:02 kgriffs: dude, i am the one that can read minds 16:44:06 damn it 16:44:09 hehe 16:44:10 #vote Yes 16:44:18 what is not yes and not no? 16:44:21 * kgriffs lives 5 seconds in the future 16:44:30 #vote Yes 16:44:36 oz_akan_: abstai, which kgriffs abstained to add 16:44:38 :D 16:44:38 oz_akan: abstain 16:44:43 :p 16:44:47 #vote abstain 16:44:49 oz_akan_: abstain is not a valid option. Valid options are Yes, No. 16:44:53 hehe 16:44:58 * kgriffs blushes 16:45:13 #vote no 16:45:30 oz_akan_: concern? 16:45:43 if 2 more vote no, we'll get to a dead-vote 16:46:01 I am not convinced that it is easy to implement, won't increase the number of bugs, will be easier to maintain, will be as fast... 16:46:01 does meetbot have a dice option ? 16:46:12 easy? 16:46:22 SQLAlchemy.core is very easy 16:46:26 eventually it is another layer 16:46:32 easier than sql itself 16:46:57 oz_akan_: it's not. it's just a chain of sql statements in python function calls 16:47:05 If it will never replace SQLite, then I might vote no. I don't want to maintain two different SQL drivers 16:47:11 like mognodb trick we did for claims, if there is another layer in the middle, it might not be easy to do these tricks 16:47:15 kgriffs: it can 16:47:18 kgriffs: it will replace it 16:47:27 I'd have voted no otherwise 16:47:28 ok 16:47:29 :D 16:47:43 but i'm not sure whether it will, because i can't predict the future 16:47:46 guys if you say it is as fast, and easy, I can change my vote 16:47:50 zyuan: looool 16:47:51 i like the idea of only maintaining one SQL driver 16:47:52 but for reference, it can 16:47:57 kgriffs: +1 16:48:22 IMHO, if other folks want specific drivers then they can implement it as a third-party backend 16:48:29 and it won't get into the codebase 16:48:34 closing the vote in 30 seconds 16:48:35 it can live elsewhere 16:49:43 endvote ? 16:49:46 flaper87: i agree. if we have to maintain SQLite and MySQL separately, i don't think SQLAlchemy buys us anything 16:49:52 but if we merge them, then it is helpful 16:50:02 kgriffs: yup 16:50:03 they can be merged 16:50:03 #endvote 16:50:04 Voted on "use SQLAlchemy for SQL storage driver?" Results are 16:50:05 Yes (3): zyuan, flaper87, kgriffs 16:50:06 No (1): oz_akan_ 16:50:15 ok, anything else on the topic? 16:50:19 not from me 16:50:21 SQLAlchemy can have engine specifc options 16:50:22 flaper87: who is assigned? 16:50:23 ykaplan: ? 16:50:30 ykaplan: she is! 16:50:32 :D 16:50:33 kk 16:50:57 not from me 16:50:59 #topic Commit tsung configs in Marconi's tree 16:51:06 malini: ^^^ 16:51:16 ykaplan: has a turkish last name which means tiger 16:51:19 we have the xmls in rackerlabs now 16:51:31 so, I added that to the agenda because I thought it would be nice to have them there 16:51:38 so that other folks could use them as well 16:51:47 ykaplan: ?? 16:51:47 we can have a contrib dir with those configs 16:51:55 but I don't know much about tsung 16:51:57 oz_akan_: LOL 16:52:00 so, malini it's up to you 16:52:14 I wud love to have them in marconi repo 16:52:24 who is assigned? 16:52:29 zyuan: ykaplan is 16:52:32 ok 16:52:36 flaper87: is there a bp for that? 16:52:41 (tsung) 16:52:53 The current rackerlabs repo is a bit obsolete & I need to update them to reflect the tests we are running now. 16:52:57 kgriffs: no, it just corssed my mind after Allan commented on my last blog post 16:53:02 kk 16:53:08 #link http://blog.flaper87.com/post/522b9e560f06d32542ede77f/ 16:53:14 (in case you guys want to read it) 16:53:16 :D 16:53:20 I can clean the tests, improve the readme & submit a new patch 16:53:28 #action flaper87 to register a blueprint for tsung configs 16:53:28 malini: cool 16:53:34 (and a readme) 16:53:50 malini: one thing, make sure they are not env dependant 16:53:51 flaper87: go ahead and assign whatever priority you feel is appropriate 16:53:57 and schedule for icehouse-1 I guess? 16:54:08 I mean, that the tests can be run without having to install M$ office 16:54:12 (seems like we have more than enough to do for H3) 16:54:34 kgriffs: oh yes, H-3 ends next month btw, and we're already in feature-freeze 16:54:44 flaper87: env depenedent ? As is , you can point them to any marconi server 16:55:01 flaper87: let's discuss in a breakout after this 16:55:10 (H3) 16:55:25 flaper87: loved your blog post :) 16:55:44 malini: :) 16:56:08 I'll register a bp for adding tsung xmls & assign me 16:56:31 oh, ok 16:56:48 sorry, missed the earlier action .. 16:56:49 #action malini to register tsung XML blueprint 16:57:01 I'll let you two play rock-paper-scissors 16:57:07 :D 16:57:08 malini: how dare you! 16:57:13 you just stole my action 16:57:14 almost out of time 16:57:16 :P 16:57:23 two bugs I want to bring up real quick 16:57:24 https://bugs.launchpad.net/marconi/+bug/1214973 16:57:28 #link https://bugs.launchpad.net/marconi/+bug/1214973 16:57:47 We currently propagate those out to the WSGI driver iirc 16:58:05 IIRC, the original implementation retried on failures (as per 1214973) 16:58:05 kgriffs: yes 16:58:09 Seems like we should retry in the mongo driver so the client isn't any wiser? 16:58:17 I guess we can restore that behavior just for inserts 16:58:26 i mean, the transport can retry I gues 16:58:41 kgriffs: I'd leave that to the backend 16:58:50 i guess it might be better to let storage do it 16:59:04 (stepping out) 16:59:16 kgriffs: also, there's to be a max_retries somewhere 16:59:19 if transport retry, storage's interface is too complex 16:59:38 ok, if someone needs something to do, pls. take a look 16:59:47 similar: 16:59:48 #link https://bugs.launchpad.net/marconi/+bug/1214974 17:00:07 things can get ugly with this one 17:00:26 we need to handle it as gracefully as possible, but I understand we won't totally mitigate things like this 17:00:33 (network down or something) 17:01:02 we have partial claims, right? 17:01:13 i don't think so... 17:01:24 claim is one to multiple 17:01:31 ok, nevermind, my memory sucks 17:01:43 ok, any last thoughts before we wrap this up? 17:01:58 * kgriffs always plans too much on the agenda 17:02:03 back to the topic, I agree we have to handle that the best way and at first sight, partial inserts seems reasonable 17:02:07 the client will have to retry 17:02:18 as long as we give enough info to the client to do that 17:02:22 I think it is fine 17:02:23 yep 17:02:24 ok 17:02:26 (same for delete) 17:02:32 ok, wrap wrap wrap wrap 17:02:42 #info I agree we have to handle that the best way and at first sight, partial inserts seems reasonable 17:02:56 #endmeeting