Tuesday, 2013-09-03

*** amitgandhi has joined #openstack-marconi00:05
*** amitgandhi has quit IRC00:09
*** kgriffs_afk is now known as kgriffs00:23
*** kgriffs is now known as kgriffs_afk00:33
*** flaper87 is now known as flaper87|afk00:50
*** kgriffs_afk is now known as kgriffs01:24
*** kgriffs is now known as kgriffs_afk01:33
*** kgriffs_afk is now known as kgriffs02:24
*** kgriffs is now known as kgriffs_afk02:34
*** kgriffs_afk is now known as kgriffs03:25
*** kgriffs is now known as kgriffs_afk03:34
*** kgriffs_afk is now known as kgriffs04:25
*** kgriffs is now known as kgriffs_afk04:34
*** kgriffs_afk is now known as kgriffs05:25
*** kgriffs is now known as kgriffs_afk05:35
*** kgriffs_afk is now known as kgriffs06:26
*** kgriffs is now known as kgriffs_afk06:35
*** ykaplan has joined #openstack-marconi06:42
*** flaper87|afk is now known as flaper8706:51
*** kgriffs_afk is now known as kgriffs07:26
*** kgriffs is now known as kgriffs_afk07:36
*** ykaplan has quit IRC07:39
*** ykaplan has joined #openstack-marconi07:42
openstackgerritFlavio Percoco proposed a change to stackforge/marconi: Implement embedded marconi-server execution  https://review.openstack.org/4474907:45
openstackgerritFlavio Percoco proposed a change to stackforge/marconi: Run functional tests under tox  https://review.openstack.org/4472307:45
*** kgriffs_afk is now known as kgriffs08:27
*** fifieldt has quit IRC08:35
*** kgriffs is now known as kgriffs_afk08:36
*** key4 has joined #openstack-marconi08:51
openstackgerritFlavio Percoco proposed a change to stackforge/marconi: Implement embedded marconi-server execution  https://review.openstack.org/4474908:57
*** kgriffs_afk is now known as kgriffs09:27
*** kgriffs is now known as kgriffs_afk09:37
*** al-maisan_ is now known as al-maisan09:40
*** ykaplan has quit IRC10:23
*** kgriffs_afk is now known as kgriffs10:28
*** kgriffs is now known as kgriffs_afk10:37
*** oz_akan_ has joined #openstack-marconi11:07
*** oz_akan_ has quit IRC11:11
*** flaper87 is now known as flaper87|afk11:16
*** ykaplan has joined #openstack-marconi11:24
*** flaper87|afk is now known as flaper8711:28
*** kgriffs_afk is now known as kgriffs11:28
*** tedross has joined #openstack-marconi11:37
*** kgriffs is now known as kgriffs_afk11:37
*** flaper87 is now known as flaper87|afk11:41
*** flaper87|afk is now known as flaper8711:58
*** flaper87 is now known as flaper87|afk12:10
*** oz_akan_ has joined #openstack-marconi12:12
*** _alexr_ has joined #openstack-marconi12:15
*** kgriffs_afk is now known as kgriffs12:28
*** oz_akan_ has quit IRC12:32
*** oz_akan_ has joined #openstack-marconi12:33
oz_akan_hi all12:37
*** kgriffs is now known as kgriffs_afk12:38
*** kgriffs_afk is now known as kgriffs12:59
*** ayoung has joined #openstack-marconi13:06
*** flaper87|afk is now known as flaper8713:08
flaper87oz_akan_: hey :)13:08
oz_akan_flaper87: how are you today?13:08
flaper87oz_akan_: alll good, you?13:09
oz_akan_flaper87: great thanks13:09
oz_akan_yesterday was holiday here, so double good :)13:09
kgriffsgood afternoon/morning folks13:09
flaper87oz_akan_: hehehehe13:10
flaper87kgriffs: morning :)13:10
kgriffsI put together IKEA furniture yesterday. I bet you are *all* jealous.13:10
openstackgerritFlavio Percoco proposed a change to stackforge/marconi: Implement embedded marconi-server execution  https://review.openstack.org/4474913:13
oz_akan_:)13:14
flaper87https://review.openstack.org/#/q/status:open+project:stackforge/marconi+branch:master+topic:refactor-system-tests,n,z13:14
flaper87ssshhhhhhh, don't say those things in here.13:14
flaper87my gf is trying to convince me to go to Ikea13:14
kgriffsLOL13:17
flaper87soooo, guys, we need to discuss the API thing before today's meeting13:18
openstackgerritFlavio Percoco proposed a change to stackforge/marconi: Implement embedded marconi-server execution  https://review.openstack.org/4474913:25
oz_akan_flaper87: kgriffs have you guy started to review proxy code?13:26
kgriffsoz_akan_: sorry, not yet. I am hoping to get to it later today13:27
flaper87oz_akan_: nope, will start doing that today, I've spent some time making functional tests work under tox and preparing things for today's meeting13:27
oz_akan_ok13:28
kgriffsflaper87: yes, we need to do that. Is now a good time?13:28
flaper87yup13:28
flaper87it is13:28
kgriffsok13:28
kgriffsso, my take on it is this:13:28
*** amitgandhi has joined #openstack-marconi13:28
kgriffsjudging by the survey and some comments made by TC members, it seems like it may not be in our best interest to pursue an AMQP driver. The semantics we have defined in the course of designing the API simply don't line up well with it, and I am hesitant to redesign the API to make it work at this point in the game.13:30
kgriffsAm I totally off base here?13:31
*** vkmc has joined #openstack-marconi13:33
* kgriffs watches a tumbleweed roll by13:33
flaper87you're not. I agree with you w.r.t our API not lining up fine with AMQP technologies but I don't want us to exclude the possibility to redesign some parts of our API in the next version. I'm not saying we should do that now13:33
kgriffsok, so we may be able to do something in v2.0 in order to better support AMQP backends13:34
kgriffsbut for now it isn't practical?13:34
flaper87correct, as for now, those backends can be implemented as third-party libraries w/ partial - or aweful workarounds - support of our API13:35
amitgandhii agree13:35
flaper87awful*13:35
flaper87I also think we should develop another backend that can be used in production environments13:37
kgriffsok. That will also give us time to gather feedback on integration use cases with AMQP so we have a better idea of people's real-world needs13:37
kgriffsflaper87: +113:37
kgriffs(for the other backend)13:37
flaper87kgriffs: exactly (for the gather feedback)13:38
flaper87My fear about supporting AMQP brokers is that, it aint scale!!! :D13:38
flaper87we'll still have one of the issues we want to fix with marconi13:38
flaper87we'd be just barrying it down the application stack13:38
kgriffsflaper87: what do you suggest re alternative backends to Mongo?13:38
kgriffsI'd like to do 1-2 more "official" backends during incubation, assuming we are accepted13:39
flaper87kgriffs: TBH, I wasn't thinking about any relational backend BUT, I think that will bring more users and more feedback13:39
zyuan_sqlalchemy, hyperdex13:40
flaper87I'm thinking about psql and swift13:40
flaper87or mysql (since it's loved throughout OS)13:40
zyuan_sqlalchemy can handle both...13:40
flaper87zyuan_: yeah, I know, but there's a discussion we should have about pros / cons of sqlalchemy13:41
flaper87TBH13:41
zyuan_flaper87: so what's the cons?13:41
flaper87lets say, we should have a backend for a RDBMS and something else13:42
kgriffsflaper87: sounds good13:42
flaper87zyuan_: can we discuss that later?13:42
kgriffs1x RDBMS13:42
kgriffs1x Redis or Hyperdex13:42
flaper87well, cppcabrera has done some work on the redis backend13:43
kgriffstrue13:43
zyuan_sqlite -> sql*, easy13:43
zyuan_mongo -> hyperdex, also easy13:43
kgriffshyperdex and rethinkdb seem slightly more related to mongo than does redis13:43
zyuan_actually redis might be the most complex one13:44
kgriffsso, maybe redis the way to go. not saying other backends would be cool, but I want a wide spread of "flavors" as reference drivers.13:44
flaper87kgriffs: +! for redis13:44
kgriffsI gotta run to the dentist (fun!) but will be back in ~90 minutes. Let me recap...13:45
flaper87kk13:45
flaper87glad we discussed this13:45
kgriffsFirst, we will not target AMQP for our initial release13:45
flaper87I'll keep playing with tests in the meantime13:45
kgriffssecond, we would like to keep the mongo driver and add 2 more13:45
kgriffs(from different families)13:46
kgriffsfor sure an RDBMS13:46
kgriffsand then a Redis or something else (TBD)13:46
kgriffsflaper87: one more thought13:46
kgriffs(to think about_13:46
kgriffsrather than making a compound product+name field in the messages collection, what would be the pros/cons of having one collection per product ID?13:47
kgriffsanyway, gotta run. bbl.13:47
flaper87you mean project, right?13:48
flaper87kk, ttyl13:48
flaper87mmhh13:48
* flaper87 thinking13:48
zyuan_flaper87: so... what do think about *sql vs sqlalchemy?13:49
*** ametts has joined #openstack-marconi13:50
kgriffsflaper87: sorry, yeah, meant project13:51
kgriffsttfn13:51
flaper87zyuan_: I've nothing against sqlalchemy, TBH. My only concern about it (as for marconi) is it slowing down operations due to all the things it does internally. I guess we could access the core API which would bypass some of those things but, it's something we should meassure. I guess.13:52
*** kgriffs is now known as kgriffs_afk13:52
flaper87There are many things it would help us with (models, migrations, etc)13:52
zyuan_flaper87: me too. i hope core api is not that slow13:52
zyuan_at least, from the api design, i think it can hardly be slow...13:52
flaper87yeah, I was also thinking we could have a BaseSqlController for Queues, Messages, Claims that other rel-based backends could inherit from. This BaseSqlCotnroller would define queries, schemas etc. and leave the rest to the implementation13:54
flaper87or we could go for the sqlalchemy-based one and let people aiming for more performance to implement their own w/ direct access to the database13:54
flaper87that's what I've thought so far.13:55
zyuan_hmm. i don't think we should have a base class for that13:58
zyuan_different db should have different schema13:58
flaper87zyuan_: agreed13:58
zyuan_for efficiency.  even for sqlalchemy, i still want some fine control over the datatypes13:59
zyuan_(while it's impossible for sqlite)13:59
openstackgerritFlavio Percoco proposed a change to stackforge/marconi: Implement embedded marconi-server execution  https://review.openstack.org/4474914:09
flaper87is cppcabrera out ?14:09
zyuan_jury duty14:11
zyuan_i'm out from tomorrow, goingnative 201314:12
flaper87aaaaaaaaaaahhh right, he said that14:12
flaper87zyuan_: COOOOOOOOOOOL!!!!14:14
flaper87you guys have those cool events there14:14
flaper87why is this check needed? https://github.com/stackforge/marconi/blob/master/marconi/tests/storage/base.py#L14214:16
openstackgerritFlavio Percoco proposed a change to stackforge/marconi: Implement embedded marconi-server execution  https://review.openstack.org/4474914:17
zyuan_i don't know whether it's needed or not, but it looks reasonable14:27
*** kgriffs_afk is now known as kgriffs14:29
zyuan_flaper87: self.process.join(1)14:35
zyuan_then the process quit?!14:35
flaper87zyuan_: nope, that just gives the process a second to boot, otherwise some operations may fail because of the process not being ready, I don't like that piece of code, Ill work on a better proposal in a separate patch.14:36
zyuan_flaper87: but if it takes more than 1 sec to boot it raise an timeout exception, right?14:38
*** kgriffs is now known as kgriffs_afk14:38
zyuan_... it does not...14:39
flaper87no, it shouldn't raise any exception14:40
flaper87I'll remove that join() in a separate patch. I'm thinking a better way to share resources and server instances througout our functional tests14:41
flaper87(ideas are very welcome)14:41
flaper87:D14:41
*** ykaplan has quit IRC14:52
*** briancline has joined #openstack-marconi14:59
zyuan_http://stackoverflow.com/questions/11769366/why-is-sqlalchemy-insert-with-sqlite-25-times-slower-than-using-sqlite3-directly/11769768#1176976815:01
*** kgriffs_afk is now known as kgriffs15:18
* kgriffs is back15:24
* kgriffs has a numb mouth15:24
kgriffsguys, don't forget about the TC meeting today at 20:00 UTC15:26
megan_whi all.  we should make sure we have some requirements set for a solution to be considered as a storage backend15:29
flaper87kgriffs: re https://review.openstack.org/#/c/44340/3/marconi/storage/mongodb/utils.py15:33
flaper87I'm not 100% sure, I'm still worried about the case where we get 10 messages back from the backend and they're all expired15:34
flaper87sending back an empty collection to the user doesn't feel right15:34
flaper87it doesn't feel right sending expired messages either, though15:34
flaper87:P15:34
kgriffsflaper87: TBH I am on the fence here as well.15:35
kgriffsmegan_w: so, we have functional requirements already: flaper8715:36
kgriffshttp://goo.gl/a8z7KG15:36
kgriffsI guess we need to create a wiki page that brings all the requirements together15:36
flaper87if you guys give me your gmails, I'll give you write access15:36
kgriffswe have some bp pages we can pull from15:36
kgriffsflaper87: sffirgk@gmail.com15:37
megan_wso the question to me is do we support backends that don't support the full api?15:37
flaper87kgriffs: we also answered the question wrt "Why Falcon"15:37
flaper87we should write that down in the wiki before the meeting15:37
megan_wanne mentioned that it could make documentation complex, which i agree15:37
kgriffsgood point15:37
megan_wflaper87: megan.wohlford@gmail.com15:37
kgriffsflaper87: on the incubation page?15:37
flaper87kgriffs: yeah!15:38
flaper87we can have a subsection like: "Raised questions and answers"15:38
megan_wgood idea15:38
megan_wit15:38
kgriffsflaper87: let me add that15:41
megan_wabout backends supporting all API calls...15:43
flaper87megan_w: hehehe, we keep avoiding that question15:43
flaper87LOl15:43
flaper87sorry about that15:44
flaper87:D15:44
megan_wwhat happens as Marconi wants to launch new features?15:44
flaper87sooooo15:44
megan_ware the backends required to support them in a certain period of time?15:44
flaper87here's what we decided today. We won't aim to support AMQP backends in v1 but we'll improve the API (v2?)  in orther to make those backends implementation easier15:44
megan_wor do we make the backends support certain core api calls?15:44
flaper87as for whether we will allow partial support of the API, erm, I'm not 100% sure what would be the best thing.15:45
megan_wi tend to agree15:45
megan_wkgriffs: ??15:45
flaper87I'm thinking about: If the backend wants to get into Marconi's source tree then yes, the baackend has to fully support our API15:45
kgriffsyeah, I would say so15:46
flaper87but, mmh, you know.... Diversity :D15:46
flaper87anyway, that's my thinking right now15:46
kgriffsbut we may allow partial support in the external repo15:46
flaper87kgriffs: megan_w I added you both to the doc15:47
megan_wok, now what about expanding the marconi api?  how do we enforce ongoing support for new features?15:47
megan_wor do we?15:47
flaper87feel free to add sqlserver there15:47
flaper87but hey, it's all logged15:48
flaper87:P15:48
kgriffsre AMQP, I think it would be good to also mention that it will be considered as part of the v2.0 design process, taking into account community needs and feedback.15:48
flaper87kgriffs: +!15:48
flaper87kgriffs: +115:48
flaper87megan_w: Unittests15:48
flaper87megan_w: I mean, tests!15:48
flaper87we're working hard on creating a good test structure that can be used by thrid-party libraries as well15:49
megan_wgot it.15:49
flaper87so that we can use that to certify the full support of the API15:49
megan_wso having a backend "officially supported" means we take on responsibility to make it a part of our test suite, right?15:50
amitgandhikgriffs: the other thing to consider is that if we want to support an AMQP backend, but have a wsgi transport, that it should use our consistent api (ie dont make significant changes unless it is needed for *most* backends).  If you want to support amqp, then maybe make an amqp transport layer which *could* have a different transport definition that is semantically identical to using amqp directly15:50
flaper87megan_w: implementers do that. If we implement the backend ourselves the yes, it's our responsibility15:50
flaper87otherwise it's up to the guy / company / ET working on the implementation15:51
megan_wgot it15:51
* flaper87 loves pulling ETs into technical discussions15:51
flaper87kgriffs: when you get a chance, could you take a look at the test patches? starting from here: https://review.openstack.org/#/c/44475/16:01
kgriffssure, once I get email/wiki stuff squared away I'll take a pass though the patches16:03
flaper87awesome, thanks!16:03
kgriffsso, SQLAlchemy or just MySQL directly?16:04
kgriffs(working on the incubation wiki FAQ)16:04
flaper87I guess you could write MySQL16:04
flaper87and we can decide later whether to use sqlalchemy16:05
kgriffsor pg?16:05
kgriffsok16:05
kgriffssounds good16:05
* flaper87 loves pg16:05
kgriffsseems like OS operators favor MySQL, though, nicht?16:07
kgriffsor does it matter?16:07
flaper87they do favor MySQL since that's what most projects support and it was the first db supported16:09
flaper87which means, most deployments using OS have MySQL installed16:09
flaper87(at least, it's more probable)16:09
kgriffsok16:11
kgriffswe will run with that then16:11
*** jergerber has joined #openstack-marconi16:12
*** _alexr_ has quit IRC16:24
kgriffsflaper87, megan_w: https://wiki.openstack.org/wiki/Marconi/docs/drivers16:34
flaper87kgriffs: thanks16:34
kgriffsflaper87: could you work on that page?16:34
flaper87kgriffs: sure16:34
kgriffsthanks!16:34
kgriffsI am going to mention it in the incubation q&a16:35
megan_wit'll be good to have this documented16:35
flaper87Heading to dinner, will work on it as soon as I get back.16:35
flaper87kk16:35
kgriffsno problem16:35
flaper87brb16:35
kgriffsttfn16:35
kgriffsflaper87: FYI: https://blueprints.launchpad.net/marconi/+spec/use-pecan-framework16:44
kgriffsrenamed: https://blueprints.launchpad.net/marconi/+spec/pecan-framework17:11
kgriffsfeedback welcome. I'll be out for a bit to lunch.17:13
oz_akan_kgriffs: flaper87 what is the motivation behind investigating pecan ?17:19
amitgandhioz_akan_: just as an alternative to falcon since falcon isnt used in any other OS project.  Pecan can be built as another wsgi transport option, and depending on perf compared to falcon marconi could switch, or stick with falcon if it continues to prove to be faster17:27
oz_akan_I am worried to loose the quality in what we have if we try to support so many17:29
oz_akan_do we think falcon is slow?17:30
oz_akan_amitgandhi: ^^17:31
amitgandhiwe dont think so17:32
amitgandhithe comment came from last weeks incubation meeting17:32
oz_akan_I must have missed that17:32
amitgandhioz_akan: https://wiki.openstack.org/wiki/Marconi/Incubation#Raised_Questions_.2B_Answers17:33
amitgandhilast one in the list17:34
oz_akan_got it thanks17:34
amitgandhinp17:35
oz_akan_I wonder if an API that mimics SQS would be useful. Imagine, one uses SQS and can easily try marconi17:51
oz_akan_kgriffs: flaper87 will you guys be checking claim problem?17:54
oz_akan_it might be related to too many documents in messages collection issue17:55
amitgandhioz_akan_: the argument to wrap all openstack api's to support sqs api semantics is often made.  i think the merits to do such a thing with marconi would need to be made.  i see advantages in it as long as marconi supports all sqs features, and expected results are the same.  But if behaviour differs in any way then it could be dangerous as users will have made preconceived expectations on how the api behaves17:59
oz_akan_+118:00
*** oz_akan_ has quit IRC18:01
*** oz_akan_ has joined #openstack-marconi18:02
flaper87back18:04
flaper87half back18:04
flaper87oz_akan_: we're investigating Pecan not because Falcon is slow but because it's widely used throughout OpenStack. It's just a way to have some standard in the whole project. It's not a strong requirement and it doesn't mean we'll be replacing Flacon as default fw - unless Pecan proves to be way faster than Falcon.18:06
oz_akan_flaper87: got it thanks18:06
flaper87kgriffs: around?18:18
kgriffsback19:00
kgriffs(was in a mtg)19:00
zyuan_kgriffs: is there a patch to use less timeutils?19:05
*** ayoung has quit IRC19:09
kgriffszyuan_: so, couple things on that19:09
kgriffsjust a sec (in a mtg19:10
*** ayoung has joined #openstack-marconi19:22
kgriffsback19:25
kgriffszyuan_: so, we need to update our oslo-incubator code to get this: https://review.openstack.org/#/c/44363/19:25
kgriffsthen, we can just use utcnow_ts()19:25
kgriffsthat will allow us to continue having testability (via override_time) but also it will still be almost as fast as calling time.time() directly.19:26
kgriffsmake sense?19:26
zyuan_thid has been merged19:27
zyuan_this19:27
zyuan_but we havn't merged it yet...19:30
kgriffsyes, let me update our copy of oslo real quick19:33
flaper87back19:34
openstackgerritKurt Griffiths proposed a change to stackforge/marconi: chore: Update openstack.common to get latest timeutils  https://review.openstack.org/4494419:36
kgriffszyuan_, flaper87: ^^19:36
* flaper87 clicks19:38
flaper87kgriffs: approved19:38
flaper87actually, ninja approved19:38
kgriffsthanks d00d19:38
kgriffsdon't you mean ninjapproved?19:39
kgriffs:D19:39
flaper87hahahahhahahah, damnit :P19:39
openstackgerritA change was merged to stackforge/marconi: chore: Update openstack.common to get latest timeutils  https://review.openstack.org/4494419:40
kgriffsflaper87: https://wiki.openstack.org/wiki/Marconi/Incubation#Raised_Questions_.2B_Answers19:41
kgriffsI tried to consolidate recent feedback19:42
kgriffsis it *mostly* accurate? :p19:42
flaper87kgriffs: read it, it looks good. I'm putting together some ides for the drivers page. I started adding random paragraphs and then I'll clean it a bit19:43
kgriffscool, sounds awesome19:43
flaper87ok guys, the meeting is 10mins away19:45
flaper87see you all in #openstack-meeting19:46
kgriffs+119:49
flaper87kgriffs: btw, +1 for the email to the list. I wanted to tell you that w/ my last ping19:56
kgriffsflaper87: I think our brains our quantum linked somehow19:58
* kgriffs checks for scars on the back of the neck19:58
megan_wmake sure everyone goes over to the #openstack-meeting channel20:02
flaper87ok, meeting started20:03
flaper87everyone in there20:03
zyuan_kgriffs: hehe, i was looking at the 'e' handling, then i found a "bug": mongo's "grace" logic is different from sqlite's...20:19
zyuan_err, no20:20
zyuan_...nothing obvious20:24
flaper87w0000000000000000000000000000000000000000000000000000000000t20:27
flaper8711 YESSSSSSS20:27
zyuan_!!!!!!20:27
openstackzyuan_: Error: "!!!!!" is not a valid command.20:27
zyuan_\!!!!!!20:27
zyuan_\o/20:27
amitgandhi=D20:28
amitgandhiwell done team!20:30
megan_wyay!!20:30
megan_wlet's keep that scope creep in our minds20:30
amettsI was looking for some sort of official pronouncement -- the discussion with those abstain people was worrying me.20:30
amitgandhiits ok, notications wont be "part" of marconi20:31
amitgandhiits always one use case of many for marconi20:31
kgriffsinteresting, I didn't expect them to want *fewer* drivers20:31
flaper87yeah, me neither20:31
amitgandhii like they want the focus on features and performance first20:31
flaper87aaaaaaaaaaaanyway, can we go party now ?20:31
kgriffssounds like they want us to be super focused and not worry so much about chasing higher-level use cases20:31
amitgandhiget the product working and then expand horizontally20:32
flaper87I would say we need to focus on both20:32
kgriffs(let the broader community chase them for us)20:32
amitgandhiparty in italy!20:32
megan_wflaper87: we're coming to you!!20:32
flaper87meaning, we gotta split the work very well and make sure we get all those things done20:32
amettsflaper87:  The RAX people have to get our deployment working before they can party.  :)20:32
* ametts cracks whip20:32
flaper87come on people, RT https://twitter.com/flaper87/status/37499291110174310420:32
flaper87:D20:32
flaper87ametts: BOOOOO BOOOOOOOOO!!!!!20:33
flaper87party while deploying20:33
flaper87that makes it even better20:33
kgriffsright, start with a solid foundation and don't compromise that by chasing lots of driver flavors - we can do that later and/or encourage a vibrant "3rd-party" ecosystem20:33
amitgandhithe best deployments are done with tequila20:33
megan_wso what now?  does the TC formally give us requirements for graducation?  or is that later on?20:33
flaper87https://wiki.openstack.org/wiki/Governance/Approved/Incubation20:34
megan_w(just wondering if we should be tracking our progress against some master list for graduation)20:34
megan_wgot it.  so next step is getting our policy board mentor20:35
flaper87yeah and we need to migrate from stackforge to openstack on GH (AFAIK)20:35
kgriffs"ability to install marconi in a real production manner without having to install something AGPL"20:36
kgriffsthat is one thing to track, at least. :)20:36
kgriffsalso, get into devstack and the usual CI process as soon as possible20:37
oz_akan_kgriffs: I couldn't the issues with AGPL20:37
oz_akan_I couldn't get20:37
megan_wseems to be some general interest in horizon20:37
kgriffsoz_akan_: i personally think it's a bit silly, but some people are pretty religious about licensing20:37
kgriffsmegan_w: not sure if that's a requirement for graduation?20:38
flaper87in order to integrate well w/ horizon we need a client library that can be used from there20:38
flaper87It's not a strong requirement though20:38
kgriffsmegan_w: but yeah, we should be looking into that20:38
oz_akan_AGPL is the like the one without any restriction, I am not sure why it might be a problem20:38
kgriffsflaper87: +120:38
flaper87devstack, heat are, AFAICT20:38
flaper87if you guys don't mind, I'd like to take care of the whole migration from stackforge to openstack20:39
kgriffsflaper87: that would be groovy.20:40
flaper87cool beans20:43
* flaper87 learned that from kgriffs20:43
amitgandhiwhen is graduation?20:43
flaper87amitgandhi: Jth20:43
amitgandhiso oct 2014 ish20:45
*** ayoung has quit IRC20:45
kgriffsso, we incubate during Icehouse20:46
flaper87amitgandhi: no, April 201420:46
flaper87We should apply for graduation during the next release cycle20:47
amettsBTW, we also need to capture/track the requirements for DevStack, Tempest, and the gating tests.20:47
flaper87and that should be our very first, official release20:47
flaper87ametts: I'll take care of that20:47
flaper87I'll create blueprints for them20:47
kgriffsflaper87: thanks!20:47
kgriffsI added a few notes to the bottom of the incubation wiki20:47
kgriffshttps://wiki.openstack.org/wiki/Marconi/Incubation#Raised_Questions_.2B_Answers20:47
flaper87kgriffs: awesome, thanks20:47
flaper87we need to add Heat to that list20:48
kgriffson there20:48
amitgandhineed to add devstack  tempest etc to that list20:48
kgriffswasn't sure whether it is a "nice to have" or a hard requirement20:48
flaper87Heat, devstack, tempest are all "must have"20:49
kgriffsoops - had that in my notes, somehow missed it20:49
amitgandhiit sounds like a new hard requirement (esp for day 0 incubators)20:49
kgriffsstand by20:49
kgriffsupdated that wiki20:52
flaper87kgriffs: awesome20:52
kgriffsso, re timeline, we need to graduate 6 weeks priorior to J release in order to be integrated, right?20:52
kgriffsicehouse release is in the Spring, isn't it?20:53
flaper87kgriffs: right20:53
* kgriffs brings up release page20:53
kgriffshttps://wiki.openstack.org/wiki/Releases20:53
kgriffsso… doesn't that mean the first integrated release Marconi would be in would be next fall?20:54
amitgandhithats how i read it20:55
flaper87mmh no if we graduate before J is released20:55
flaper87AFAIK20:55
kgriffsisn't the J release in the fall?20:55
kgriffs(Last week someone from the TC mentioned that we would be too late for Icehouse)20:55
flaper87ah damn, I meant Ith20:55
flaper87sorry20:55
flaper87my bad20:55
kgriffsoic20:56
amitgandhiH is released in Oct,  I is released in Apr (too late for that), so J in Oct 201420:56
kgriffswe would have to graduate pretty quickly to make it into I20:56
kgriffsnict?20:56
kgriffss/nict/nicht20:56
* ametts is so confused20:56
flaper87kgriffs: TBH, I don't think it's going to take us much time to implement all the required bits20:56
amitgandhiThe I release design summit is in Nov 201320:57
flaper87amitgandhi: yeah, I meant to say Ith release and kept writing J20:57
kgriffsamitgandhi: right, that is planning for the Ith release20:57
amitgandhiwe arent going to be ready for them to commit us to the release then20:57
flaper87amitgandhi: why not?20:57
kgriffsHavana is released Oct 17, then in November planning starts in ernest for Icehouse20:57
amitgandhiwont they require us to meet all the graduation requirements by then (2 months), and wasnt there something about having 2 releases or something along those lines20:58
kgriffsI suppose that if we can graduate 6 weeks prior to Apr 2014, we should be able to make it into the integrated release.20:58
flaper87amitgandhi: no, we need to meet all the graduation requirements 6 weeks prior to the next release20:59
kgriffshttps://wiki.openstack.org/wiki/Governance/Approved/NewProjectProcess20:59
amettsWho's on first?20:59
kgriffs"All projects must be approved for promotion to core 6 weeks before a release cycle's design summit."20:59
flaper87which means we have6 1/2 months starting now20:59
kgriffsTBH, I'm still a little fuzzy on "core" vs. "integrated release"20:59
flaper87kgriffs: same #%@$ different ways to call it21:00
flaper87:D21:00
* kgriffs thought so21:00
amitgandhioh ok so as long as we meet their requirements 6 weeks before the release date, we can be part of that release21:01
kgriffswow.21:01
kgriffsthis is really interesting considering the thinking around "core" ~1 year ago21:01
flaper87amitgandhi: yup21:01
kgriffsok folks, I think this calls for a round of Pop-Tarts™21:03
oz_akan_it seems like we have 404s not 204s for claims21:04
* kgriffs starts flinging pop-tarts all over21:04
oz_akan_I updated the bug report21:04
oz_akan_trying to understand if it happens when we have more than a number of documents in messages collection21:04
* ametts already popped a bag of Rackspace Atlanta popcorn.21:04
flaper87oz_akan_: you don't like pop-tarts, do you?21:04
* flaper87 jumps around and catches his and oz_akan_'s pop-tarts21:05
oz_akan_flaper87: I like what you like21:05
* flaper87 thinks about very disgusting things he "likes"21:07
oz_akan_hehe21:07
oz_akan_I like a claim returning 20121:07
oz_akan_or 20421:07
flaper87or 50021:07
amitgandhii thought malini said something about the 404 happening after the 204's21:08
amitgandhilike the 204 was giving a null location and hence the subsequent 40421:08
amettsamitgandhi:That's an artifact from her test rig21:08
oz_akan_ok, it is about the frequency21:08
flaper87ah, btw, before migrating marconi under openstack/ we need to get all those patches merged21:08
flaper87:)21:08
oz_akan_if I send so many requests I start to get 404s21:08
oz_akan_even when there is little number of messages21:08
flaper87or, re-submit them if needed21:08
amettsThe 404 wasn't really a bug in Marconi itself.21:08
flaper87lets not just merge patches if we're not sure about the code21:09
oz_akan_we shall return 404 only when there isn't a queue21:09
amitgandhiis this with the slaves disabled?21:10
amitgandhis/slaves/secondaries21:10
oz_akan_this when we write primary and read from secondary21:11
amitgandhiit sounds like a replication issue to me getting back 404's.21:11
oz_akan_if there was no data, because of replication lag, it would return 20421:12
oz_akan_but returns 40421:12
* amitgandhi reads above and recalls this is for claims21:13
amitgandhiand this is when attempting to POST a claim right (not getting an existing claim?)21:14
oz_akan_yes, post claim21:15
kgriffsif a queue was just barely created, but the queue collection record has not replicated, you will get a 404 when POSTing a claim21:16
kgriffsI have a patch pending that changes it to return a 204 in that case, but that is beside the issue21:17
kgriffsso, it could still be an eventual consistency thing21:18
kgriffseven if something reads messages after creating the queue and gets some back, they may be getting them off 1 secondary that is up to date, but the other is lagging, and that's the one that is hit when the POST claim comes in21:19
kgriffsanyway, only way to know for sure is to force reads from the master OR set w=3 so responses are only ack'd after replication to both secondaries.21:20
kgriffsflaper87: what do you think about a separate collection per project ID?21:20
kgriffs(switching topics)21:20
oz_akan_https://bugs.launchpad.net/marconi/+bug/121901921:21
oz_akan_i put my findings there21:21
oz_akan_I think this is something wrong with mongodb and marconi both21:22
oz_akan_i will re-install mongodb on these servers with latest version tomorrow to see if it will help21:22
oz_akan_have to leave now21:22
*** oz_akan_ has quit IRC21:23
amitgandhihttp://en.wikipedia.org/wiki/File:Piles_of_Salt_Salar_de_Uyuni_Bolivia_Luca_Galuzzi_2006_a.jpg21:27
amitgandhisalt stack?21:27
* amitgandhi oops wrong room21:27
flaper87sorry, my laptop hung up21:28
flaper87kgriffs: I thought a bit about that after you bring it up today. I think we would reach namespace limit very easily21:28
flaper87In a deployment with many tenants / project21:29
kgriffsbut in such a deployment, wouldn't marconi-proxy be used anyway and you would end up with disjoint clusters?21:29
kgriffsdo you think a single "partition" would not be able to handle enough projects?21:30
kgriffsIf my math is correct, a single Mongo server can support is 3,417,89021:32
flaper87kgriffs: mmh, this is the limit http://docs.mongodb.org/manual/reference/limits/#Number of Namespaces21:32
kgriffs2047 MB = 2146435072 bytes21:33
kgriffs2146435072 / 628 = 341789021:34
kgriffs(check me)21:34
flaper87there21:34
kgriffs16 * 2**20 / 62821:35
flaper87ops21:35
flaper87there's just 1 namespace file per database21:35
amettsflaper87, kgriffs: Either of you know when they'll open up technical session proposals for HKG?  Does our new incubation status mean we get rooms now, or are we still relegated to unconference sessions?21:35
flaper87ametts: I think we get the chance to have sessions as well21:35
amettsWe should be on the lookout for that opportunity.21:36
flaper87kgriffs: IIRC21:36
kgriffsametts: +121:36
flaper87ametts: +121:36
kgriffsflaper87: "Namespace files can be no larger than 2047 megabytes."21:36
kgriffs"The limitation on the number of namespaces is the size of the namespace file divided by 628"21:37
kgriffsit isn't clear if they mean size in bytes or MB, but it seems like bytes, since 16 * 2**20 / 628 is roughly 24,00021:37
flaper87if you 'ls' mongodb's data dir, there should be just 1 .ns per database21:38
flaper87kgriffs: that's bytes21:38
kgriffsok21:38
flaper87they always measure everything in bytes21:38
flaper87(hopefully, they didn't change the standard in that particullar case)21:38
kgriffsso, ~3 million namespaces per namespace file?21:38
kgriffs(if you set nssize to the max)21:39
flaper87anyway, I also think we should design things to work perfectly on a single marconi-server instance21:39
flaper87then they can be faster if marconi-server is scaled out21:39
kgriffssure, but I think you also have to say that if people want unlimited tenants and queues, they will need marconi-proxy21:39
kgriffsthat being said, a single partition should be no slouch21:40
flaper87would that be true for every backend?21:40
flaper87or just mongodb?21:40
kgriffswell, say there were a Redis backend21:40
kgriffsyou would be limited by RAM21:40
flaper87right, but that's a hardware limit, not software21:41
kgriffs(assuming no db-level sharding)21:41
kgriffsflaper87: sure, I'm just saying, the backend makes a difference on how far you can scale (and in which ways you scale)21:41
flaper87kgriffs: agreed21:42
flaper87kgriffs: would it be fair to have a "partitioned" config param ?21:42
flaper87that would let the backend know it's being partitioned21:42
kgriffsflaper87: re namespaces limitation, does 3 million sound right? Seems like a heck of a lot to me.21:42
kgriffsflaper87: what might the backend do with that knowledge?21:43
flaper87kgriffs: 3M sounds right, I'd like to verify the limit w/ #mongodb guys21:43
flaper87kgriffs: use different storage strategies?21:43
flaper87I'm afraid that would make migrations from 1 strategy to another very difficult21:43
kgriffsok. If that is correct, seems like it would be plenty to support projects (multiple queues will continue to share the same collection, per project)21:43
flaper87ah, also, what would the exact benefit of having 1 col per project?21:44
flaper87the lock is at DB level anyway21:44
kgriffsyes, but now you don't have to have the project in the index21:44
flaper87kgriffs: oh, right, good point21:44
kgriffsand hot queues will have their indexes in RAM, not getting swapped out due to interleaving with requests for other queues21:45
flaper87kgriffs: yeah, sounds like a good strategy to me21:45
kgriffsalso, you don't repeat the project ID a gazillion times in the document store21:45
flaper87well thought21:45
kgriffsso, I was trying to find out the cons21:45
kgriffscan you think of any others other than namespace limitations?21:45
kgriffs(not really a con, I guess, but requires the operator to set nssize before the DB is created)21:46
kgriffsI guess you can ping the #mongodb guys21:46
flaper87yeah, I will do that21:46
kgriffsok, cool. I think this would be the last major change along with changing to use unix timestamps instead of date objects21:47
*** tedross has quit IRC21:47
kgriffs(change to the schema)21:47
kgriffsoh, also we have multi-db support (Oz's patch) which he is working on - he mentioned that he will try to get that submitted later this week21:48
*** openstackgerrit has quit IRC21:48
flaper87how will that patch integrate with current code?21:48
*** openstackgerrit has joined #openstack-marconi21:48
flaper87and with this change we're planning21:48
kgriffsso, there will be a configurable number of DB's to partition the data across21:50
kgriffsif you configure 4, then you end up with marconi-1, marconi-2, marconi-3, marconi-421:50
kgriffsmurmur3(project_id + queue_name) % 421:50
kgriffssomething like that21:50
kgriffs(I haven't seen the latest code)21:50
kgriffswithin each DB we would partition the messages collection by project21:51
flaper87kk, it sounds to me that both things can be the default behavior of mongodb's driver21:51
flaper87I don't see why multi-db should be turned of / on21:51
flaper87one can simply specify to use 1 db21:52
flaper87for the whole thing21:52
kgriffsyep21:52
flaper87kk21:52
kgriffsyou may get a tiny boost from avoiding the murmurhash, but probably not noticeable21:52
kgriffsand as long as operators are advised to increase nssize, partitioning the messages collection by project ID should be OK as well21:52
kgriffsjust thought of something21:54
flaper87kgriffs: a good documentation should do that for us21:54
kgriffsold tenants may clog up the namespace, so maybe we need eventually a way to GC inactive projects21:54
flaper87kgriffs: if queue's are empty, then we can delete them21:55
flaper87I guess21:55
flaper87kgriffs: namspace creation is really cheap in mongodb21:55
kgriffssure, just somebody may someday hit 3 million limit. :p21:56
kgriffs(that would be a lot of happy customers!)21:56
* ametts will be happy to have that problem21:56
flaper87If we hit that issue, the driver should rebalance the dbs21:57
flaper87and put the project in a separate db21:57
kgriffsgood point21:57
kgriffshttps://blueprints.launchpad.net/marconi/+spec/partition-messages-collection21:58
kgriffssince this would be a breaking change, I'd like to do it sooner rather than later21:58
kgriffsbut let's wait to see what you find out from 10gen peeps21:59
flaper87kgriffs: Mongodb Inc :D22:00
flaper87the rebranded 10gen22:00
flaper87:D22:00
flaper87they*22:00
kgriffsno kidding?22:00
flaper87no kididng22:00
kgriffshttp://www.mongodb.com/22:00
kgriffsheh22:00
kgriffszyuan_: are you working on making the unix timestamp vs. date object change, or just investigating?22:01
* kgriffs just realized he didn't get to any reviews today22:01
flaper87kgriffs: tests tests tests22:01
flaper87:D22:01
kgriffsflaper87: re filtering out expired...22:02
kgriffsmegan_w: you around to discuss ^^^22:02
flaper87btw, lets hold some patches back 'til we migrate under openstack22:02
kgriffs?22:02
kgriffsflaper87: ok22:02
*** ayoung has joined #openstack-marconi22:02
kgriffsflaper87: ETA on the migration?22:02
flaper87I'd say 1 or 2 days, I'll submit a patch tomorrow morning and start pinging the hell out of -infra22:02
kgriffsheh22:03
kgriffsOK, I will refrain from approving anything for a couple days. If oz needs something sooner he can cherry-pick or whatever22:04
kgriffsflaper87: just saw the cache patch got deferred until Icehouse22:05
kgriffsbummer22:05
flaper87yeah :(22:06
*** russellb has joined #openstack-marconi22:07
flaper87Ok guys, it's happening: https://review.openstack.org/#/c/44963/22:28
flaper87kgriffs: FYI, I was told that everything will happen automagically so, no need to hold patches back22:31
kgriffssweet, thanks22:33
*** oz_akan_ has joined #openstack-marconi22:34
*** oz_akan_ has quit IRC22:38
*** jergerber has quit IRC22:48
*** amitgandhi has quit IRC22:49
*** kgriffs is now known as kgriffs_afk23:01
*** vkmc has quit IRC23:12
*** kgriffs_afk is now known as kgriffs23:30
*** kgriffs is now known as kgriffs_afk23:44
*** amitgandhi has joined #openstack-marconi23:51

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!