17:59:53 <SergeyLukjanov> #startmeeting sahara
17:59:54 <tellesnobrega> o/
17:59:54 <openstack> Meeting started Thu Dec 11 17:59:53 2014 UTC and is due to finish in 60 minutes.  The chair is SergeyLukjanov. Information about MeetBot at http://wiki.debian.org/MeetBot.
17:59:55 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
17:59:58 <openstack> The meeting name has been set to 'sahara'
18:00:03 <SergeyLukjanov> #topic sahara@horizon status (croberts, NikitaKonovalov)
18:00:09 <SergeyLukjanov> any critical updates folks?
18:00:32 <aignatov> o/
18:00:35 <NikitaKonovalov> nothing critical
18:00:43 <NikitaKonovalov> but one very annoying thing
18:00:45 <NikitaKonovalov> https://review.openstack.org/#/c/125927/
18:01:07 <NikitaKonovalov> this change fixes Job Executions table
18:01:09 <SergeyLukjanov> yeah, I see
18:01:10 <crobertsrh> You haven't even hit 30 patch sets yet
18:01:20 <elmiko> lol
18:01:21 <SergeyLukjanov> folks, we all need to spam it with +1s ;)
18:01:23 <NikitaKonovalov> crobertsrh: I know
18:01:42 <SergeyLukjanov> #topic News / updates
18:01:49 <crobertsrh> The filtering change is up for horizon, been sitting a week or so now.  Need some more eyes.
18:01:54 <NikitaKonovalov> and I wish it lands a bit earlier this time
18:02:02 <SergeyLukjanov> folks, please
18:02:05 <crobertsrh> #link https://review.openstack.org/#/c/136459/
18:02:08 <alazarev> I hixed a number of horizon bugs
18:02:14 <alazarev> *fixed
18:02:20 <tmckay> crobertsrh, I'll try to take a look today, and your spec too
18:02:22 <tellesnobrega> storm is in, and i need a plan to start storm edp
18:02:26 <elmiko> i've been working on the security guide. talked with the sec. and doc. folks and we will have a chapter in the official sec guide.
18:02:38 <elmiko> #link https://etherpad.openstack.org/p/sahara-security-guide-notes
18:02:49 <elmiko> i'm going to start collecting notes here, and setting up the chapter outline
18:02:58 <alazarev> also moved security groups to a separate tab: https://review.openstack.org/#/c/140485/
18:03:01 <crobertsrh> I've been working on specs for editing node group and cluster templates as well as a spec for "default templates".
18:03:06 <crobertsrh> #link https://review.openstack.org/#/c/140448/
18:03:15 <crobertsrh> Default templates spec should be up today.
18:03:22 <SergeyLukjanov> tellesnobrega, yay!
18:03:24 <weiting> ken chen is working on to enable the rest services in cdh cluster.
18:03:28 <tmckay> I have been working substituting data_source paths for references in job_config values -- a little more complex than I thought.  But, almost done.  I need unit tests.  And, it occurred to me in the process that all the workflow creation unit tests should be using xml utils to test results, not self.assertIn
18:03:47 <aignatov> guys, I’m gonna add an example of pig job which was top todoers at summit before last
18:03:49 <tmckay> tellesnobrega, I can help you with storm EDP questions
18:04:00 <alazarev> tellesnobrega, I can help either
18:04:11 <aignatov> it’s more informative than just trim spaces I think :)
18:04:19 <tellesnobrega> alazarev, tmckay, thanks. will ping when i have some time
18:04:32 <aignatov> Also I hope we will switch our ci integration tests to it
18:04:43 <huichun> I am current focus on the integration tests on CDH plugin, to make sure the new services(HBase etc) works well in the cluster
18:04:46 <elmiko> tellesnobrega: don't know if you saw this, but someone was asking about storm in sahara. https://ask.openstack.org/en/question/55810/storm-as-openstack-data-processing/
18:05:18 <SergeyLukjanov> huichun, nice, we're still not enabled cdh voting but I hope it'll be done early next week
18:05:25 <tellesnobrega> elmiko, cool, had not seen that.
18:05:28 <elmiko> aignatov: ahh, but trim whitespace is so usefule! ;)
18:05:37 <huichun> thx SergeyLukjanov
18:05:48 <aignatov> elmiko, yep, I just gonna add one more example :)
18:05:57 <aignatov> trim space will be there
18:06:01 <SergeyLukjanov> :)
18:06:18 <SergeyLukjanov> folks, please, don't forget about k1 next week
18:06:21 <SergeyLukjanov> kilo-1 dev milestone
18:06:25 <elmiko> aignatov: +1
18:06:43 <SergeyLukjanov> aignatov, could you add it to integration tests too?
18:06:52 <elmiko> SergeyLukjanov: i don't think sec-guide will be in for k-1, but k-2 seems very likely
18:07:03 <alazarev> SergeyLukjanov, anything specific about kilo-1? do we have todo list?
18:07:08 <SergeyLukjanov> elmiko, that's awesome anyway to have it ;)
18:07:21 <SergeyLukjanov> alazarev, I think it's a launchpad page for release
18:07:27 <SergeyLukjanov> we need some cleanup for bugs
18:07:46 <aignatov> SergeyLukjanov: I’ll add just an example and with second CR I’ll add switching to the new job, would that work?
18:07:54 <SergeyLukjanov> I'm going to move unimplemented features to k2 at monday
18:08:06 <alazarev> #link https://launchpad.net/sahara/+milestone/kilo-1
18:08:18 <SergeyLukjanov> aignatov, yup, probably not switch but just add one more
18:08:47 <SergeyLukjanov> we defenetly need some reviews on https://blueprints.launchpad.net/sahara/+spec/indirect-vm-access
18:08:50 <aignatov> hmmm, you mean one more edp test in our ci testing?
18:09:07 <SergeyLukjanov> zhidong, do you have any updates about CDH license?
18:09:15 <SergeyLukjanov> oh, I think he's not available now
18:09:31 <weiting> Zhidong will not be online today.
18:09:35 <SergeyLukjanov> aignatov, yup, we're already running a few jobs, just add one more
18:09:54 <SergeyLukjanov> next week we'll be at 1400 UTC, folks
18:09:58 <SergeyLukjanov> don't forget about it
18:10:06 <SergeyLukjanov> #topic Open discussion
18:10:15 <aignatov> it could increase time of job executions
18:10:17 <weiting> For CDH License, Cloudera is still working on to check the license internally.
18:10:34 <weiting> They need some time to check it.
18:10:42 <SergeyLukjanov> re building and publishing SwiftFS - work in progress and I think it'll be in next week
18:10:42 <aignatov> in our tests
18:10:44 <alazarev> aignatov, even with small data?
18:10:53 <SergeyLukjanov> I'm now looking on building it for both Hadoop versions
18:10:57 <aignatov> alazarev: I think yes
18:11:05 <SergeyLukjanov> aignatov, we should check it
18:11:08 <aignatov> defenitely we should try
18:11:34 <SergeyLukjanov> aignatov, on the other hand we could stop testing hadoop-1 on each patch set and add more powerful jobs to the rest tests
18:11:43 <aignatov> it costs about 8 mins in hadoop 2 for 100mb of data
18:11:57 <aignatov> for hadoop 1 it takes 3-5 mins  :)
18:12:00 <SergeyLukjanov> aignatov, and probably 10 mins for 1.5 GB :)
18:12:29 <SergeyLukjanov> so, what's about to stop test hadoop 1 on each patch set?
18:12:31 <aignatov> SergeyLukjanov: not sure, non-linear dependency here :)
18:12:37 <SergeyLukjanov> aignatov, yup
18:12:40 <elmiko> do we want to try and plan a bug triage day for sometime after k-1 release but before holiday break?
18:13:26 <alazarev> are we deprecating hadoop1 in kilo?
18:13:27 <SergeyLukjanov> elmiko, probably in Jan
18:13:37 <elmiko> SergeyLukjanov: cool, sounds good
18:14:18 <SergeyLukjanov> elmiko, I'm afraid some of us will be on vacations in the last two weeks of December and first two weeks of Jan
18:14:27 <SergeyLukjanov> alazarev, I'm not sure about it, but we should think about it
18:14:28 <aignatov> alazarev: aws still runnning hadoop-1?
18:14:33 <elmiko> SergeyLukjanov: yea, i figured as much. just thought i'd bring it up
18:14:41 <SergeyLukjanov> for example, HDP 1 plugin is extremely outdated
18:15:07 <SergeyLukjanov> I'll be on vacation most probably Dec 29 - Jan 11
18:15:16 <tmckay> we could deprecate hdp, but leave vanilla
18:15:26 <SergeyLukjanov> tmckay, we can do it, yup
18:15:36 <tmckay> as long as apache is supporting vanilla1, why deprecate?  It's stable
18:15:43 <aignatov> before removing hadoop-1 we need to check carefully if all our features are supported by hadoop-2…. I mean data localoity, swift integration. performance aspects and so on :)
18:15:56 <aignatov> tmckay: +1 to as well
18:15:59 <SergeyLukjanov> aignatov, could point
18:16:02 <SergeyLukjanov> tmckay, +1
18:16:33 <aignatov> also our changes are minimal, not huge efforts there to hadoop-1
18:16:43 <tmckay> as for hdp plugin out of date ... different issue :)
18:17:01 <tmckay> we can send ping of shame to hwx folks, or draft community member to fix it
18:17:45 <SergeyLukjanov> tmckay, hwx folks promising to assign several engineers to work on their plugin
18:17:55 <tmckay> yay!
18:18:00 <SergeyLukjanov> at end of the November...
18:18:17 <tmckay> 2015 ;-)
18:18:21 <SergeyLukjanov> :)
18:18:27 <tmckay> must be, they're not here yet
18:18:31 <SergeyLukjanov> shame on me, I forget to ask about year ;(
18:18:45 <elmiko> lol
18:19:00 <alazarev> they also promised to share their sahara vision... in the beginning of Dec
18:19:01 <tellesnobrega> lol
18:19:33 <alazarev> I also didn't ask for year... stupid I am
18:20:05 <SergeyLukjanov> oh, we're now testing Mirantis OpenStack release 6.0 (Juno) and Sahara works very well (only CDH and some fixes backported from Kilo, no custom patches)
18:20:18 <elmiko> nice!
18:20:26 <tmckay> so, anybody know why we didn't write job_manager unit tests to use xml utils for test results?
18:20:39 <SergeyLukjanov> tmckay, nope
18:20:45 <tmckay> we use assertIn on strings, very fragile
18:21:06 <tmckay> I helped perpetuate this too, way back when :)
18:21:10 <tmckay> but it's no good
18:21:12 <elmiko> maybe open a bug?
18:21:19 <tmckay> I'm writing new ones, I'll experiment
18:21:29 <tmckay> if it works for the subst stuff, I'll add a bug
18:21:40 <elmiko> cool
18:22:00 <elmiko> tmckay: i agree in general, we should be using the xml libs to parse the workflow for testing
18:22:59 <elmiko> as a side note, i have generated a small script to create a basic swagger compliant json object for our api
18:23:02 <elmiko> https://github.com/elmiko/sahara-doc
18:23:16 <elmiko> i'm going to be adding more to it, and we will be talking about it at the next api working group meeting
18:23:17 <tmckay> excellent
18:23:22 <crobertsrh> Neat!
18:23:48 <elmiko> it's very proto at the moment, but it does produce the base level of swagger2.0 compliant doc
18:23:52 <SergeyLukjanov> elmiko, cool!
18:24:41 <elmiko> if the api wg decides to use swagger for the api-ref stuff, we might be able to incorporate something like this to automatically produce reference api docs for sahara
18:24:45 <tmckay> oh, SergeyLukjanov, egafford should have a spec sometime in the next few weeks about EDP args.  We talked about an idea for adding an argument mapping template to a job, so that a job can define unambiguously the type and order of arguments it takes
18:25:01 <tmckay> UI can be driven by the map, potentially
18:25:42 <tmckay> it ties in to the data_source substitution stuff I am working on, and may get rid of the need for the fixed input/output paths in hadoop jobs
18:25:48 <SergeyLukjanov> tmckay, nice
18:25:52 <tmckay> opens up possibility of N data paths
18:26:08 <crobertsrh> Speaking of UI.  I suspect that after the first of the year, I'll be working heavily on user experience things (aka, "wizard stuff").  If anyone wants to be a part of that (even if it's just ideas, not code), let me know.
18:26:12 <tmckay> maybe Kilo, maybe L, depending on complexity, and UI guys availability, etc
18:26:13 <SergeyLukjanov> tmckay, multiple times nice :)
18:27:33 <SergeyLukjanov> anything else to chat about today
18:27:35 <SergeyLukjanov> ?
18:28:59 <crobertsrh> nothing from me
18:29:20 <alazarev> nothing from me
18:29:21 <SergeyLukjanov> ok, sounds like we ready to continue working :)
18:29:22 <tmckay> Nothing from me
18:29:39 <SergeyLukjanov> thank you folks, see you next week 1400 UTC
18:29:41 <SergeyLukjanov> #endmeeting