14:01:20 <SergeyLukjanov> #startmeeting sahara 14:01:21 <openstack> Meeting started Thu Jan 15 14:01:20 2015 UTC and is due to finish in 60 minutes. The chair is SergeyLukjanov. Information about MeetBot at http://wiki.debian.org/MeetBot. 14:01:22 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 14:01:23 <SergeyLukjanov> hi folks 14:01:25 <openstack> The meeting name has been set to 'sahara' 14:01:25 <crobertsrh> hello/ 14:01:36 <SergeyLukjanov> #link http://wiki.openstack.org/wiki/Meetings/SaharaAgenda 14:01:38 <elmiko> mornin 14:01:39 <SergeyLukjanov> crobertsrh, morning 14:01:44 <SergeyLukjanov> elmiko, morning 14:02:20 <SergeyLukjanov> #topic sahara@horizon status (croberts, NikitaKonovalov) 14:02:25 <SergeyLukjanov> crobertsrh, NikitaKonovalov, please 14:02:42 <crobertsrh> Usual update: Horizon reviews linger. 14:02:56 <SergeyLukjanov> heh 14:02:56 <NikitaKonovalov> So we have a bunch of changes as always waiting 14:03:07 <crobertsrh> I'm part way finished with a "cluster creation guide" mode 14:03:08 <SergeyLukjanov> crobertsrh, NikitaKonovalov, could you please make a single etherpad 14:03:19 <SergeyLukjanov> with all changes on review in horizon related sahara 14:03:32 <SergeyLukjanov> with launchpad bug id, title and state 14:03:37 <crobertsrh> That's a fine idea. 14:03:53 <NikitaKonovalov> I guess we could use a single topic in the changes, and then filter by it in gerrit 14:04:00 <aignatov> o/ 14:04:15 <sreshetnyak> hi 14:04:41 <NikitaKonovalov> but the etherpad will definitely work 14:05:22 <sreshetnyak> i'm working on migration to openjdk and new integration tests 14:06:45 <tmckay> I've just about got spark/swift support done for EDP. There is one issue, though, which is the jackson-mapper-asl jar version. Not sure what to do about that. Upgrade of CDH version might fix it. 14:07:12 <tmckay> Or, we may have to carry a version of the jar ourselves, or we might have to fix up the hadoop plugin to avoid the issue 14:07:29 <tmckay> Currently I am adding the jackson jar to the nodes manually for development 14:07:53 <tmckay> Spark wrapper is based on oikawa 's Oozie Java wrapper, thanks oikawa! 14:08:11 <tmckay> maybe he's not here 14:08:43 <kazuki_oikawa> / 14:08:55 <tmckay> oh, hi! wrong name :) 14:09:25 <tmckay> SergeyLukjanov, also, we should talk about location to store wrapper jars and sources 14:09:42 * tmckay hasn't checked comments on the review last couple day 14:10:54 <tmckay> no new comments on https://review.openstack.org/#/c/115884/ 14:10:57 <huichun> Hi sreshetnyak are you working on adding new integration tests for the others services by writing the examples ? 14:11:00 <tmckay> #link https://review.openstack.org/#/c/115884/ 14:11:49 <SergeyLukjanov> (/me forgot to switch topic) 14:12:26 <sreshetnyak> Hi huichun. I'm working on adding new integration tests for all services and all plugins 14:12:53 <SergeyLukjanov> #topic News / updates 14:12:53 <sreshetnyak> #link https://review.openstack.org/#/c/147219/ 14:13:50 <crobertsrh> I'm still working on a "guided mode" for cluster creation. It's almost done. Then I'll do something for job creation/execution. 14:14:12 <elmiko> i've been continuing work on the sahara security guide, there is a lot of ground to cover but i'm making progress. also, i was able to run bandit against the sahara codebase, it went smoothly and reported some thing that we might investigate. 14:14:16 <elmiko> #link https://gist.github.com/elmiko/94f063583912a5d8c3cb 14:14:19 <SergeyLukjanov> tmckay, the best location is tarballs.o.o I'm still working on generic solution to store things there for sahara and hope to make it soon 14:14:20 <elmiko> ^^ bandit output 14:14:32 <crobertsrh> In general, it's a single page with plenty of descriptive text for each step and links to the required forms to complete each step. Hopefully, it will help the newer users find their way. 14:14:43 <tmckay> SergeyLukjanov, okay. Maybe we can store a jackson jar there if we need to, as well 14:14:53 <SergeyLukjanov> tmckay, yup I think 14:15:04 <kchen> hi sreshetnyak. Huichun is mainly focus on CDH plugin services. 14:15:23 <kchen> I think his work is a bit different from yours. 14:16:44 <kchen> crobertsrh. what do you mean by "guided mode"? 14:17:14 <crobertsrh> It's a "wizard-like" mode to help novice users through cluster creation. 14:17:37 <crobertsrh> It currently utilizes the existing forms, but ties them together in an understandable way 14:17:57 <kchen> it sounds good to me. I am also thinking of this question. 14:18:02 <tmckay> kchen, maybe huichun can help me figure out the CDH spark assembly jackson-mapper-asl version issue 14:18:27 <crobertsrh> kchen: I hope to have a patch up sometime soon. 14:18:39 <tmckay> crobertsrh, it occurred to me yesterday that the UI might not have the support in it yet for buttons to set the data source substitution options 14:18:41 <crobertsrh> Or, I may just have screenshots for next week's meeting. 14:18:55 <kchen> crobertsrh: great. 14:18:55 <crobertsrh> tmckay: that is correct. 14:19:00 <tmckay> crobertsrh, we haven't talked about that since 2014 :) 14:19:32 <SergeyLukjanov> :) 14:19:32 <tmckay> patch should be merged soon. I thought it was already but SergeyLukjanov dinged my commit message ;-) 14:19:44 <SergeyLukjanov> oops, sorry 14:19:48 <huichun_> hi sreshetnyak sorry for lost connection, I am working on the integration tests for the new servcies too, but met some issue 14:19:53 <tmckay> haha, no problem 14:20:02 <kchen> crobertsrh, do we have a review link for that patch? 14:20:18 <crobertsrh> Not yet. 14:20:43 <crobertsrh> It's still rather ugly at the moment. I'm still playing with some ideas and trying things that may or may not work. 14:21:14 <huichun_> for example. how can we add classpath in the example jar? for example. we wrote a HBaseTest.jar, but it depends on hbase-client.jar, so how can we pass `hbase classpath` by using java action? 14:21:14 <tmckay> huichun_, do you know a lot about CDH? I might need to talk with you about the jackson issue I've run into 14:21:31 <huichun_> hi tmckay 14:21:50 <tmckay> I'm talking about the Spark image specifically 14:22:11 <weiting> tmckay, which version you used in Spark, version 4, right? 14:22:21 <tmckay> weiting, yes 14:22:30 <kchen> I am working on CDH version management now, and I expected to have a patch soon. The first patch draft maybe by early next week. 14:22:35 <tmckay> 5 might fix the issue, haven't checked 14:22:54 <huichun_> tmckay, sorry for lost connection, miss your words just now 14:24:01 <tmckay> huichun_, that's okay. I have an issue with the spark assembly jar in the Spark image for Sahara. I wondered if you had some expertise there (it is a CDH image) 14:24:46 <SergeyLukjanov> kchen, that's cool, we're really missing version management for CDH 14:25:01 <SergeyLukjanov> oh, we have an action item from the last meeting for elmiko -- ACTION: elmiko to run bandit against codebase 14:25:07 <elmiko> yea 14:25:09 <elmiko> #link https://gist.github.com/elmiko/94f063583912a5d8c3cb 14:25:12 <elmiko> that's the result 14:25:29 * SergeyLukjanov looking 14:25:29 <elmiko> it went pretty smoothly, i just followed the bandit instructions and it ran without issue 14:25:42 <huichun_> hi tmckay, we could talk it offline by email? 14:25:53 <tmckay> huichun_, certainly, thanks 14:26:17 <kchen> oh one thing. currently cdh integration edp and hive test can pass. so need we turn it back to voting? 14:26:28 <elmiko> SergeyLukjanov: it mostly doesn't like us using /tmp, but probably doesn't realize that's on the nodes. 14:26:42 <elmiko> still, might be something to investigate for improving on-node security 14:26:48 <tmckay> kchen, +1 14:27:20 <huichun_> hi, SergeyLukjanov, should we add shell action and sqoop action in edp job type? 14:27:44 <SergeyLukjanov> huichun_, good question, I think spec needed for it and could be discussed in spec review 14:28:18 <elmiko> SergeyLukjanov: i think next steps with regards to bandit will be to dig into these warnings and determine what the OSSG recommends 14:28:19 <tmckay> huichun_, also, one thing we've talked about but never had time to implement is raw Oozie workflows 14:28:24 <huichun_> ok. I have made a bp, and will update the specs 14:28:32 <tmckay> huichun_, user uploads own workflow.xml 14:29:04 <tmckay> that would allow users to use more Oozie features, like dags and so on 14:29:43 <huichun_> tmckay, you mean user can upload his own workflow.xml by horizion page? 14:29:44 <SergeyLukjanov> huichun_, thx 14:29:52 <tmckay> huichun_, yes 14:29:58 <SergeyLukjanov> elmiko, thanks for making security work 14:30:13 <elmiko> SergeyLukjanov: np, i'll keep pushing forward 14:30:57 <huichun_> i found this problem, currently we have no way to add like classpath for the Java action, right ? tcmky 14:31:04 <huichun_> tmckay 14:31:57 <tmckay> huichun_, hmmm. There is a java options config that can be set, but I'm not sure how Oozie handles that 14:32:34 <tmckay> huichun_, but we have a general mechanism of "edp.xxx" configs where we can pass anything we need to the EDP engines. So we should be able to add it if we need something else 14:33:18 <tmckay> huichun, edp.java.java_opts 14:33:52 <huichun> hi, tmckay, we have tried 14:34:09 <huichun> but it's not work, we pass `hbase classpath` in the Java_opts, but failed 14:34:34 <tmckay> huichun, I can help you investigate, I think 14:35:01 <huichun> hi, tmckay, we can talk this offline 14:35:04 <huichun> thanx tmckay 14:35:10 <tmckay> huichun, agreed 14:36:14 <SergeyLukjanov> okay, let's move on :) 14:36:25 <SergeyLukjanov> #topic Bug/doc/spec days 14:36:35 <SergeyLukjanov> big topic I'd like to chat about 14:36:52 <SergeyLukjanov> IMO we need bug triage and bug fix days 14:36:57 <elmiko> +1 14:37:01 <SergeyLukjanov> we already have many issues in lp 14:37:04 <SergeyLukjanov> too many imo 14:37:14 <crobertsrh> +1. Been a long time since last bug day 14:37:14 <SergeyLukjanov> docs are not bad 14:37:31 <elmiko> we also need more reviews on bug fixes 14:37:59 <tmckay> +1 14:38:04 <tmckay> maybe even 2 days 14:38:08 <tellesnobrega_> +1 14:38:20 <tmckay> next week? 14:38:35 <SergeyLukjanov> https://review.openstack.org/#/q/sahara-specs+AND+status:open,n,z specs are in a good shape IMO 14:39:01 <SergeyLukjanov> yeah, we need two separated days 14:39:14 <SergeyLukjanov> one to triage and plan what should be fixed, find volunteers 14:39:26 <SergeyLukjanov> and one to only fix prioritized issues 14:39:45 <elmiko> i'd be ok with a 2 day hack-a-thon ;) 14:39:54 <elmiko> er, bug-fix-a-thon 14:40:14 <SergeyLukjanov> it could be done like a virtual sprint 14:40:54 <SergeyLukjanov> it means we'll agree on some time to start on 14:41:12 <elmiko> tmckay proposed sometime next week, is that too soon? 14:41:14 <SergeyLukjanov> nope, it works bad for this task, rollback :) 14:41:36 <SergeyLukjanov> what's about next two mondays? 14:41:44 <SergeyLukjanov> jan 19 and 26 14:41:53 <tmckay> +1, sounds good to me 14:42:00 <elmiko> 19th for planning/triage, 26th for fixing? 14:42:05 <SergeyLukjanov> it makes volunteers able to fix some bugs after triaging and before the bug fix day 14:42:09 <SergeyLukjanov> elmiko, yeah 14:42:12 <elmiko> +1 14:42:13 <crobertsrh> Those should work for me. +1 14:42:40 <SergeyLukjanov> at what time could we try to start between locations? 14:42:41 <tellesnobrega_> i can help out as well 14:42:47 <sreshetnyak> +1 14:43:21 <kchen> will we hold a meeting for that or not? 14:43:30 <elmiko> SergeyLukjanov: for planning we could the same time as this meeitng 14:43:43 <tmckay> SergeyLukjanov, when the first contributors wake up on Monday til the last ones go to bed :) 14:44:00 <tmckay> So maybe it starts Sunday night in my time zone 14:44:27 <elmiko> well, if we plan for the 19th at UTC14:00 in #openstack-sahara, we could try to catch many people. (although it will be tough for west coast USA) 14:45:03 <tmckay> elmiko, there is always eavesdrop, too, if someone has to catchup 14:45:19 <SergeyLukjanov> elmiko, ++ for Jan 18 1400 UTC 14:45:21 <elmiko> kchen, weiting, kazuki_oikawa, how late/early is UTC14:00 for you guys? 14:45:31 <kchen> 10pm 14:45:48 <tmckay> SergeyLukjanov, 18th or 19th 14:45:50 <weiting> UTC+8 in our zone 14:46:06 <elmiko> k, thanks. kinda late to start triaging bugs though... 14:46:11 <kchen> the time is ok for us 14:46:42 <kazuki_oikawa> 11pm (UTC+9). but the time is ok. 14:46:50 <elmiko> ok, great then 14:46:55 <huichun> the same time as we are right now 14:47:23 <elmiko> yea, i just thought it might be late to start triaging. but if you guys are ok with it that's awesome =) 14:47:24 <SergeyLukjanov> tmckay, 19th 14:47:51 <SergeyLukjanov> yeah, that's awesome! 14:48:17 <SergeyLukjanov> #agreed bug triage day at Jan 19th starting at 1400 UTC 14:48:25 <SergeyLukjanov> what's about bug fixing day? 14:48:30 <SergeyLukjanov> the same, but Jan 26th? 14:48:38 <mattf> isn't that every day? 14:49:05 <elmiko> lol 14:49:14 <tmckay> mattf, some days are bug creation days 14:49:17 <kchen> :D 14:49:29 <mattf> tmckay, good call 14:50:02 <tmckay> SergeyLukjanov, +1 same on 26th 14:50:12 <elmiko> SergeyLukjanov: +1 14:51:14 <tosky> so is it also for specs, or are they handled in the usual way? 14:51:51 <elmiko> tosky: we are mostly up to date on specs 14:51:54 <SergeyLukjanov> specs are in a good shape 14:52:00 <elmiko> but too many bugs =( 14:52:03 <SergeyLukjanov> yeah 14:52:18 <SergeyLukjanov> #agreed bug fix day at Jan 26th starting at 1400 UTC 14:52:39 <SergeyLukjanov> 7 mins left, anything else to discuss? 14:52:46 <SergeyLukjanov> #topic Open discussions 14:53:20 <elmiko> always looking for more eyes on the ongoing security guide etherpad 14:53:25 <elmiko> #link https://etherpad.openstack.org/p/sahara-security-guide-notes 14:54:30 <tellesnobrega_> i have a question about storm edp, are we planing to have it in kilo? 14:55:52 <tosky> can we talk about the new yaml-based test scenario spec on the channel after the meeting? 14:55:59 <tosky> also: did you recently try to generate the hdp2 centos6._6_ image? 14:56:21 <tmckay> tellesnobrega_, is it done? 14:56:30 <SergeyLukjanov> tosky, sure (re new tests) 14:56:38 <tmckay> tellesnobrega_, apologies, I'm behind on reviews 14:57:02 <tmckay> tellesnobrega_, if it's finished, and we have integration tests, and it passes ... and it's in by the feature freeze 14:57:17 <tmckay> then I would say "sure!" ^^ 14:57:42 <tmckay> otherwise L 14:58:06 <tellesnobrega_> not ready, i will send the spec tomorrow or the day after 14:58:20 <tellesnobrega_> still need to approve that in order to implement it all 14:58:24 <SergeyLukjanov> #link https://wiki.openstack.org/wiki/Kilo_Release_Schedule 14:58:25 <tellesnobrega_> when is feature freeze? 14:58:32 <SergeyLukjanov> FPF is March 5 14:58:33 <tellesnobrega_> SergeyLukjanov, thanks 14:58:56 <SergeyLukjanov> so, I think you have enough time to get approves :) 14:59:09 <tmckay> tellesnobrega_, that's proposal freeze, feature freeze is March 19 14:59:16 <SergeyLukjanov> oh, folks, please, review Indirect access done by alazarev 14:59:26 <tellesnobrega_> yeah 14:59:40 <SergeyLukjanov> #endmeeting