13:59:48 <SergeyLukjanov> #startmeeting sahara 13:59:49 <openstack> Meeting started Thu Apr 23 13:59:48 2015 UTC and is due to finish in 60 minutes. The chair is SergeyLukjanov. Information about MeetBot at http://wiki.debian.org/MeetBot. 13:59:50 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 13:59:53 <openstack> The meeting name has been set to 'sahara' 13:59:58 <SergeyLukjanov> #link https://wiki.openstack.org/wiki/Meetings/SaharaAgenda 14:00:19 <SergeyLukjanov> #topic sahara@horizon status (crobertsrh, NikitaKonovalov) 14:00:25 <SergeyLukjanov> #link https://etherpad.openstack.org/p/sahara-reviews-in-horizon 14:00:33 <SergeyLukjanov> how is it going folks? 14:00:40 <crobertsrh> Mostly the same status as last week. 14:00:56 <crobertsrh> I have one more change queued-up not on the list, but it will require a new python-saharaclient 14:01:03 <SergeyLukjanov> btw 0.9.0 client released and global requirements will be updated next week I think (after end of dep freeze for Kilo release preparations) 14:01:06 <tosky> hi 14:01:09 <crobertsrh> great 14:01:14 <elmiko> nice 14:01:33 <crobertsrh> I had a few small bug fixes get merged quickly, which was nice 14:01:51 <SergeyLukjanov> crobertsrh, cool! 14:01:59 <huichun> hi 14:02:27 <crobertsrh> If anyone wants to suggest UI enhancements, even before summit, that is great too. 14:02:39 <NikitaKonovalov> I've actually got no updates on UI 14:03:24 <SergeyLukjanov> #topic News / updates 14:03:27 <SergeyLukjanov> #link https://etherpad.openstack.org/p/sahara-liberty-proposed-sessions 14:03:41 <SergeyLukjanov> folks, please continue filling the etherpad with summit ideas 14:04:00 <Nikolay_St> working on the spec for cluster health check everybody welcome to review 14:04:11 <sreshetnyak> i'm working on scripts for building oozie and swiftfs 14:04:15 <SergeyLukjanov> #info next IRC meeting (April 30) will be mostly dedicated to design summit sessions discussion 14:04:22 <Nikolay_St> #link https://review.openstack.org/#/c/175363/ 14:04:42 <huichun_> working on the scheduled edp jobs 14:04:45 <crobertsrh> I think we need more session ideas 14:04:51 <vgridnev_> i am finished work with deprecation of direct engine and also working with autotune hadoop configs 14:04:57 <SergeyLukjanov> I'll start working on composing ideas to summit sessions early next week, so, please, dump your ideas ;) 14:05:25 <egafford> Working on unified job interface map impl again, now that we're in a good place on Kilo. 14:05:30 <sreshetnyak> also i started writing kilo release notes 14:05:36 <sreshetnyak> #link https://etherpad.openstack.org/p/sahara-kilo-release-notes 14:06:01 <elmiko> i've been working on filling out the apiv2 session pad, also the security pad, and examining some of the sahara-image-elements changes 14:06:28 <crobertsrh> I have to say that I am really liking the sahara-image-elements changes 14:06:40 <elmiko> crobertsrh: +1 14:06:44 <egafford> crobertsrh: +1 14:06:50 <elmiko> sreshetnyak, pino|work, tosky, great work! 14:07:00 * pino|work bows and thanks 14:07:12 <SergeyLukjanov> crobertsrh, +1 pino|work great woek! 14:07:13 * tosky just did some testing, nothing important 14:07:28 <SergeyLukjanov> oh, some updates on gate jobs 14:07:33 <sreshetnyak> folks, please add more information in release notes 14:07:49 <SergeyLukjanov> we now have all images building for image elements and hadoop-openstack lib and oozie building for -extra 14:08:09 <SergeyLukjanov> all of this jobs are now working ok and so I'm thinking about enabling them as voting 14:08:14 <SergeyLukjanov> what do you think folks? 14:08:22 <egafford> tosky: (Testing is kinda important...) 14:08:34 <elmiko> sounds cool 14:08:43 <egafford> SergeyLukjanov: Enabling those jobs as voting can really only help. 14:09:00 <sreshetnyak> +1 14:09:02 <elmiko> SergeyLukjanov: mattf had a question about the gate and usage of -all vs -engine/-api 14:09:16 <crobertsrh> +1 14:09:31 <pino|work> just note that, given that they download lots of stuff from internet at every generation, any connection/remote website can be blocking, even when unrelated 14:09:56 <SergeyLukjanov> pino|work, but "recheck" could be used for it 14:10:02 <elmiko> are we testing distributed mode at the gate yet? 14:10:04 <pino|work> yeah i now 14:10:07 <pino|work> *know 14:10:34 <SergeyLukjanov> elmiko, in sahara-ci all jobs except one with suffix -aio are distributed (-api + few -engine) 14:10:50 <elmiko> SergeyLukjanov: awesome, thanks! 14:11:03 <SergeyLukjanov> elmiko, in gate (jenkins) -all user, but we'd like to migrate soon to distrib and disable -all in devstack at all 14:11:25 <pino|work> SergeyLukjanov: do you want to turn image generation jobs as voting, only in master or in all the branches? 14:11:28 <elmiko> nice, sounds good 14:11:36 <SergeyLukjanov> I've shared etherpad with you guys with CI plans and I've already discussed it with OpenStack QA folks as well 14:12:04 <SergeyLukjanov> pino|work, I think starting from current master 14:12:09 <SergeyLukjanov> the same for -extra 14:12:10 <tosky> SergeyLukjanov: yep, sounds nice (the CI plan) 14:12:21 <pino|work> SergeyLukjanov: if master only, i'd say +1 14:12:43 <SergeyLukjanov> #link https://etherpad.openstack.org/p/sahara-liberty-sahara-ci 14:12:50 <elmiko> my only concern for voting on master is how frequently the centos builds break =( 14:13:03 <tosky> SergeyLukjanov: just a question, the lines "For Sahara client/client's tempest tests (in gate some day?)", is it about the Sahara client tests in the sahara repository, not the ones in python-saharaclient, right? 14:13:28 <SergeyLukjanov> sreshetnyak, should know the correct answer, but I think yes ;) 14:13:48 <SergeyLukjanov> in fact the main goal is to enable different OpenStack topologies testing based on fake plugin in gate 14:14:07 <SergeyLukjanov> and the rest (test all plugins on one topology) will be tested in sahara ci 14:14:28 <pino|work> elmiko: do they break often? 14:14:39 <elmiko> pino|work: they seem to break for me often ;) 14:14:41 <SergeyLukjanov> elmiko, we could see how often we'll face it 14:14:54 <elmiko> yea, sounds good. and i'm +1 for letting them vote anyways 14:15:41 <SergeyLukjanov> great 14:16:03 <SergeyLukjanov> we'll be able to make them non-voting anyway if they'll hurt us a lot 14:16:05 <elmiko> SergeyLukjanov: what did you think about adding bandit as a non-voting gate job at some point? 14:16:09 <elmiko> yea 14:16:29 <SergeyLukjanov> elmiko, trying to understand the conf file, it's over complicated IMO 14:16:52 <SergeyLukjanov> elmiko, if you could help with adding tox env for running it in sahara - I'll easily add the job itself 14:17:05 <elmiko> SergeyLukjanov: ack, i'll take a look =) 14:17:23 <SergeyLukjanov> elmiko, I was looking at https://github.com/openstack/keystone/blob/master/bandit.yaml and a bit surprised :) 14:18:04 <elmiko> yea... 14:18:18 <elmiko> this is something i can bring up with the ossg too. i'm sure they would love feedback on usage 14:18:27 <SergeyLukjanov> elmiko, that's cool 14:19:01 <elmiko> SergeyLukjanov: thanks for looking into it though =) 14:19:52 <elmiko> one more question about the aio testing, are we going to completely remove aio testing? 14:20:05 <SergeyLukjanov> my plan for gate improvements in order I will work on it - enable image, swiftfs and oozie building jobs as voting, add them to the gate, then enable fake plugin based scenario testing as experimental, implement all scripts for its flexible exec in gate and somewhere in parallel - bandit 14:20:27 <SergeyLukjanov> elmiko, I think about keeping one jobs based on fake plugin with aio in gate 14:20:36 <elmiko> ack 14:20:37 <SergeyLukjanov> it's in etherpad with CI plans 14:20:41 <SergeyLukjanov> I think it'll be enough 14:21:12 <SergeyLukjanov> heh, I forget to switch to open discussion 14:21:18 <elmiko> i need to take another pass at that etherpad, it looked good to me the first time through 14:21:18 <SergeyLukjanov> #topic Open discussion 14:21:23 <elmiko> hehe 14:21:44 <SergeyLukjanov> elmiko, :) 14:21:57 <Nikolay_St> ho 14:22:01 <SergeyLukjanov> elmiko, you're not so sure now, isn't it? :) 14:22:14 <pino|work> SergeyLukjanov: what do you think about my comment in https://review.openstack.org/#/c/175476/ ? 14:22:24 <Nikolay_St> I have a request for this topic :D and it's related to the review process :D 14:22:25 <elmiko> lol, no i think it still sounds good. i just wanted to make sure about aio 14:22:55 <SergeyLukjanov> pino|work, ++, could you please make a patch? 14:23:05 <pino|work> sure thing 14:23:15 <SergeyLukjanov> pino|work, great, I like you idea 14:23:24 <SergeyLukjanov> Nikolay_St, ? 14:23:31 <Nikolay_St> guys, it's not neccessary to give a guy more than one "-1" with "rebase it comment" or typo comment. 14:23:58 <elmiko> Nikolay_St: noted, and good point 14:24:17 <Nikolay_St> if you want to track this patch you can just add yourself to the reviewers using "+" in the upper right ;) 14:24:34 <Nikolay_St> elmiko: it's a common point in last 2 weeks (may be more) 14:24:52 <tmckay> hi folks. my bad, still not used to the time change. I came here at 9am but nobody was here :) 14:25:00 <SergeyLukjanov> Nikolay_St, sometimes there is an issue when you'd like to mark reviewed such patch to hide it from review list, how it could be done? :) 14:25:07 <elmiko> Nikolay_St: yea, and i've done it too. but you are right, no need to pile on more -1 14:25:13 <SergeyLukjanov> tmckay, good morning :) 14:25:24 <weiting> SergeyLukjanov, we would like to share our customer use case for Sahara in Design Summit and is it possible to add this topic in sprint session? 14:25:25 <tmckay> not too many updates for me, working on internal plan for us. First meeting in 35 minutes :) 14:25:50 <elmiko> weiting: that sounds great! 14:25:51 <SergeyLukjanov> weiting, yeah, just add a topic to etherpad https://etherpad.openstack.org/p/sahara-liberty-proposed-sessions 14:26:02 <Nikolay_St> SergeyLukjanov: this thing is not working for me, so I forgot about it :D 14:26:05 <Nikolay_St> my bad 14:26:35 <SergeyLukjanov> I think the topic "End user use cases" is very intersting and could help us a lot to concentrate on more important for users stuff 14:26:46 <tmckay> weiting, yes, customer use cases would be great! High priority, I think. 14:26:52 <tmckay> it informs everything else 14:27:02 <weiting> Another question is for Sahara EDP there are two topics in working and sprint, is it possible to combine them into one? 14:27:02 <elmiko> SergeyLukjanov: +1 14:27:30 <SergeyLukjanov> the main goal is to collect ideas 14:27:39 <SergeyLukjanov> we could move and combine them 14:27:54 <weiting> They are just evaluating sahara, but we think some feedbacks are great and we can discuss them. 14:28:11 <elmiko> definitely, we could really use more user feedback 14:28:15 <SergeyLukjanov> all design sessions on previous summits were build based on the proposed ideas and mostly always it was a mix of proposed ideas ;) 14:28:30 <SergeyLukjanov> weiting, sure, it's very important 14:28:47 <crobertsrh> I'd love to hear an actual customer use case/story 14:28:55 <elmiko> same 14:29:21 <tmckay> ++ We have our own pet issues based on experience, but we're developers. Might be completely different. 14:29:44 <tmckay> for instance, I have a suspicion that the job executions page might be better if organized by name instead of ID 14:30:03 <tmckay> When I go to look for a job for relaunch, I have to guess at which one it is 14:30:14 <tmckay> or, maybe sorted by time at least 14:30:16 <egafford> tmckay: +1; id has no semantics at all. 14:30:26 <tmckay> but, is that something customers are feeling too? 14:30:27 <crobertsrh> I don't think that's a dev-only "pet" issue :) 14:30:37 <elmiko> i thought everyone loved doing business in uuids ;) 14:30:41 <tmckay> :) maybe a bad example. But you get my point. 14:30:55 <egafford> Sometimes, even devs can tell when things aren't easy, but yes, tmckay, point well taken. 14:30:55 <tmckay> elmiko, I almost named my kids uuids 14:30:59 <elmiko> lol! 14:31:11 <SergeyLukjanov> any small imporovements ideas should be posted somewhere as well 14:31:32 <SergeyLukjanov> I mean not somewhere but it should be implemented atm or added to https://etherpad.openstack.org/p/sahara-liberty-proposed-sessions 14:31:33 <elmiko> maybe we should make a "small improvements" sprint, and put all ideas in that pad? 14:31:39 <tmckay> SergeyLukjanov, good idea. I meant to create an EDP "dump ideas" page and link to it from the summit pad, I'll do that today 14:31:51 <SergeyLukjanov> we'll have the whole day to discuss things 14:31:58 <elmiko> tmckay: +1 14:32:04 <SergeyLukjanov> tmckay, +2 14:32:06 <SergeyLukjanov> :) 14:32:15 <elmiko> hehe 14:32:35 <SergeyLukjanov> btw we'll have rc2 today 14:32:49 <tmckay> yay! 14:32:52 <elmiko> \o/ 14:32:56 <SergeyLukjanov> oops, we already have it :) 14:33:12 <Nikolay_St> SergeyLukjanov: :D 14:33:19 <SergeyLukjanov> so, 99.999% that it'll be our Kilo release 14:33:27 <elmiko> cool 14:33:39 * tmckay looks for 0.001 bug 14:33:42 <SergeyLukjanov> so, time to final check our favourite project :) 14:34:08 <tmckay> cinder? jk :) 14:34:18 <huichun_> tmckay: Trevor I am writing the spec of adding more oozie workflow into edp, so sahara can support scheduled jobs 14:34:30 <SergeyLukjanov> huichun_, yay, cool! 14:34:53 <elmiko> sounds like a nice improvement 14:35:11 <tmckay> huichun_, great. Did you see my note about that? Maybe we want to extend the EDP engine interface to support scheduling and job coordination in general, and then implement in the individual EDP engines 14:35:18 <tmckay> just an idea 14:35:37 <tmckay> trying to think how best to handle enhancements, when we want to move away from being so Oozie centric 14:35:49 <SergeyLukjanov> tmckay, sounds interesting 14:35:58 <elmiko> would be nice to create some sort of abstraction there 14:36:27 <tmckay> yes. Oozie used to be everything for use, but now it's just one of the things (the biggest, but still ...) 14:36:31 <weiting> tmckay, so you would like to create a new abstract layer to do so? 14:36:35 <tmckay> use/us 14:36:47 <SergeyLukjanov> would be nice to 606 abstraction layers with 13 services - numbers will probably help the hell working :) 14:37:09 <elmiko> lol 14:37:16 <egafford> Interface consistency and feature parity across plugins is always nice to have. SergeyLukjanov: Heh. 14:37:22 <tmckay> weiting, maybe, I haven't thought about it too much. But, since we can split engines (even split engines between plugin versions) we should take advantage of that, I think. 14:37:34 <SergeyLukjanov> egafford, ++, process names for example 14:37:58 <tmckay> I am cautious of implementing something only in the Oozie engine, for instance 14:38:18 <egafford> tmckay: Especially if it drives an interface change (which it will.) 14:38:18 <tmckay> for scheduling, it could be in the job manager, as long as it applies to all job types 14:38:40 <tmckay> If we are using Oozie features to do it, that argues for an EDP engine interface, so the backend can be something else for spark/storm 14:39:27 <SergeyLukjanov> btw, do we like to backport something else to stable/kilo of -image-elements and extra? last chance - I'm going to tag them with rc2 after the meeting 14:39:57 <elmiko> SergeyLukjanov: is the python package stuff going into that tag? 14:40:06 <egafford> tmckay: +1 to both DAG generation and scheduling interfaces being Sahara-side rather than implemented entirely within plugins. 14:40:21 <tmckay> nothing that I am aware of. tosky, pino|work, ^^ you guys test a lot :) 14:40:25 <SergeyLukjanov> elmiko, which python package? 14:40:33 * pino|work has nothing to backport so far 14:40:43 <elmiko> SergeyLukjanov: the new sahara-image-create script, with setup.py/cfg 14:40:52 * tosky neither 14:41:08 <SergeyLukjanov> elmiko, nope, it's done in master I think 14:41:18 <elmiko> SergeyLukjanov: ok, cool 14:41:29 <elmiko> (we're still finding issues with it) 14:41:36 <huichun_> tmckay: sorry, lost connection just now, i will talk with you offline in details 14:41:44 <pino|work> elmiko: (and fixing too) 14:41:47 <tmckay> huichun_, okay 14:41:55 <elmiko> pino|work: exactly ;) 14:43:16 <egafford> The new sahara-image-create script isn't a particularly critical feature, and it's easy enough to patch in to any packaged distribution if distros are feeling risky. It can wait, thinks this packager. 14:43:42 <elmiko> egafford: yea, i agree, i just wanted to make sure 14:43:47 <egafford> elmiko: Totally. 14:47:26 <SergeyLukjanov> anything else to chat about? 14:47:50 <elmiko> anyone looked at apache atlas? 14:49:34 <elmiko> (that was my only chat topic) ;) 14:49:38 <SergeyLukjanov> elmiko, hm 14:49:47 <SergeyLukjanov> elmiko, I've heard about it but don't remember 14:49:53 <elmiko> https://wiki.apache.org/incubator/AtlasProposal if anyone is curious 14:50:21 <elmiko> looks interesting and imo has some overlap with what we are doing. not sure it would be useful to us, but interesting none-the-less 14:50:38 <SergeyLukjanov> Documentation 14:50:38 <SergeyLukjanov> There is documentation in a private github repository at: https://github.com/hortonworks/metadata 14:50:38 <SergeyLukjanov> Initial Source 14:50:38 <SergeyLukjanov> The source is currently in a private github repository at: https://github.com/hortonworks/metadata 14:50:43 <SergeyLukjanov> heh, private HWX repos 14:51:09 <elmiko> i guess this was an hwx internal project that is getting migrated to the ASF 14:51:39 <elmiko> maybe something we can discuss more at summit ;) 14:52:53 <SergeyLukjanov> yup 14:54:03 <tmckay> gotta drop folks, see you in the other channel 14:55:39 <SergeyLukjanov> thx folks! 14:55:44 <SergeyLukjanov> #endmeeting