14:01:19 <SergeyLukjanov> #startmeeting sahara
14:01:20 <openstack> Meeting started Thu Jun 18 14:01:19 2015 UTC and is due to finish in 60 minutes.  The chair is SergeyLukjanov. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:01:21 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
14:01:23 <openstack> The meeting name has been set to 'sahara'
14:01:25 <alazarev> o/
14:01:46 <SergeyLukjanov> #topic sahara@horizon status (crobertsrh, NikitaKonovalov)
14:01:53 <SergeyLukjanov> #link https://etherpad.openstack.org/p/sahara-reviews-in-horizon
14:02:15 <NikitaKonovalov> I've no updates here since I've been focused on other things
14:02:38 <NikitaKonovalov> But I guess not much has changed form the last meeting
14:02:52 <vgridnev> Event log change still on review, that is sad
14:02:57 <SergeyLukjanov> NikitaKonovalov, do you probably know anything about moving sahara dashboard to contrib dir in horizon?
14:03:07 <elmiko> vgridnev: =(
14:03:08 <crobertsrh> Template editing changes still on review == sadd
14:03:32 <crobertsrh> I haven't added any new UI patches recently, been working more on service side of things.
14:03:33 <NikitaKonovalov> SergeyLukjanov: haven't seen any movement there
14:03:56 <alazarev> NikitaKonovalov, any movements in other projects?
14:04:56 <NikitaKonovalov> looks like no
14:05:01 <SergeyLukjanov> sahara was just enabled for horizon integration tests https://review.openstack.org/#/c/192645/
14:05:49 <tosky> oh, merged, nice; I missed the commit message
14:06:02 <NikitaKonovalov> SergeyLukjanov: does that mean that we can have fake plugin tests
14:06:23 <tosky> (another review will reenable the horizon integration tests as voting)
14:06:49 <SergeyLukjanov> NikitaKonovalov, hm, I think so
14:06:54 <NikitaKonovalov> I mean make UI create a cluster with a fake plugins an check that all panels, configs, etc are in place
14:07:00 <tosky> NikitaKonovalov: I though Horizon integration tests are for full deployment
14:07:22 <SergeyLukjanov> NikitaKonovalov, tosky I think it should be possible now
14:07:52 <kchen> We have registered a bp to integrate manila and sahara in https://blueprints.launchpad.net/sahara/+spec/edp-manila-hdfs. We are working on the spec.
14:07:58 <SergeyLukjanov> https://github.com/openstack/horizon/blob/master/openstack_dashboard/test/integration_tests/tests/test_sahara_job_binaries.py
14:08:04 <kchen> Hope we can have a spec by the next week
14:08:50 <tmckay> kchen, excellent
14:08:57 <NikitaKonovalov> job_binaries are quite standard == easy to test
14:09:06 <NikitaKonovalov> clusters should be more interesting
14:09:53 <SergeyLukjanov> NikitaKonovalov, yeah, but seems like no difference if we have full devstack install
14:10:00 <SergeyLukjanov> anything else re sahara@horizon?
14:10:29 <crobertsrh> nothing I can think of
14:10:33 <NikitaKonovalov> nothing from me
14:10:34 <SergeyLukjanov> so, let's move on
14:10:38 <SergeyLukjanov> thx folks
14:10:39 <SergeyLukjanov> #topic News / updates
14:11:50 <NikitaKonovalov> I've been working on refactoring of sahara.utils module. There are too many things now some of which are required by provisioning plugins and some not
14:11:54 <elmiko> i was out part of last week and at spark summit this week. i have talked with apavlov about the keystone sessions, i think we are close to a spec for that. also i've been investigating using gabbi to fuzz test a live sahara server. also, lots of cool stuff at spark summit, i think we need to improve our spark support =)
14:12:13 <vgridnev> Finished work with recommendations provider, really need for review some my changes
14:12:15 <crobertsrh> I've been doing a little experimental work getting a Spark 1.3 cluster up and running the Zeppelin notebook stuff on top of the cluster.  Looks promising so far.
14:12:21 <vgridnev> #link https://review.openstack.org/#/q/owner:%22Vitaly+Gridnev%22+status:open,n,z
14:12:28 <sreshetnyak> I'm working on HDP 2.2 plugin
14:12:29 <NikitaKonovalov> So the idea is to move the code from utils to where it's actually needed so the sahara-plugins split will be easier
14:13:30 <SergeyLukjanov> NikitaKonovalov, at least in theory, if we'll agree to do it :)
14:13:33 <esikachev> i'm working on custom scenarios
14:13:43 <tmckay> reviewing/testing egafford's job interface mapping, I think it's almost ready for a +2 from me
14:13:46 <huichun> working on scheduler and recurrence edp job
14:13:47 <tmckay> looks good
14:13:51 <alazarev> I'm trying to test hadoop performance with disks attached directly to VM via cinder driver
14:15:52 <SergeyLukjanov> sounds like time to move on
14:16:05 <SergeyLukjanov> #topic Liberty-1
14:16:08 <tmckay> maybe for open topics, but I think someone needs to work on hdp plugin ci tests (if not already) or we should move them to non-voting for now.  They break too often, and HWX is not here to fix them :)
14:16:13 <SergeyLukjanov> so, we have a liberty-1 next week
14:16:31 <SergeyLukjanov> tmckay, esikachev working on CI failures now
14:16:39 <tmckay> yay!  thanks
14:17:20 <SergeyLukjanov> #info Liberty-1 next
14:17:25 <SergeyLukjanov> #undo
14:17:26 <openstack> Removing item from minutes: <ircmeeting.items.Info object at 0x9c87690>
14:17:39 <SergeyLukjanov> #info Liberty-1 next week (Jun 23-25)
14:17:44 <SergeyLukjanov> #link https://wiki.openstack.org/wiki/Liberty_Release_Schedule
14:18:02 <SergeyLukjanov> #info Liberty release tag for sahara will be 3.0.0
14:18:11 <elmiko> neat
14:18:12 <SergeyLukjanov> so, Liberty-1 is 3.0.0b1
14:18:46 <SergeyLukjanov> the Sahara changes was applyed already in I6a35fa0dda798fad93b804d00a46af80f08d475c
14:18:55 <SergeyLukjanov> and I'm now working on doing the same for the rest repos
14:19:30 <SergeyLukjanov> so, note that epoch should be increased :)
14:19:36 <SergeyLukjanov> any questions about it?
14:20:35 <SergeyLukjanov> #topic Next client releases
14:20:56 <SergeyLukjanov> I'd like all of you to think about all potential changes to client that should be done in Liberty
14:21:06 <SergeyLukjanov> especcially new features addition
14:21:22 <SergeyLukjanov> and I'd like to make a release schedule for the client
14:21:45 <alazarev> SergeyLukjanov, why do we need release schedule for it?
14:21:58 <alazarev> SergeyLukjanov, usually it is shipped when needed
14:22:03 <SergeyLukjanov> note: we should have final "feature" release of client by liberty-3
14:22:04 <crobertsrh> I have at least 2 client changes that will be needed, for editing data sources and job binaries.
14:22:10 <SergeyLukjanov> alazarev, it's needed for us
14:22:32 <vgridnev> at least one change with auto-configuration parameter to client
14:22:34 <tellesnobrega> i would like to add storm job submission on the client
14:22:37 <tmckay> I have one tiny one that tosky noted.  default templates should be marked somehow in the template list output
14:22:51 <SergeyLukjanov> there are two main issues I'd like to solve by listing needed client features - not miss any of them in official Liberty client and decrease latency of releasing new features
14:23:09 <tellesnobrega> and also if multiple clusters creation enters I would have to implement on the client as well (if we use a new api call)
14:23:17 <egafford> Interface field on both job and job template need to get into the client as well.
14:23:30 <elmiko> SergeyLukjanov: +1
14:23:30 <SergeyLukjanov> tellesnobrega, will it be different comparing to the other jobs?
14:23:40 <tellesnobrega> no
14:24:03 <tellesnobrega> i have to take a look to see if any changes are needed
14:24:41 <tmckay> sounds like we need an etherpad for this (did I miss one above?)
14:25:18 <alazarev> tmckay, +1
14:25:24 <SergeyLukjanov> https://etherpad.openstack.org/p/sahara-liberty-client
14:25:38 <tmckay> yay!  good work SergeyLukjanov.  Fast too :)
14:26:10 <SergeyLukjanov> Please, add links to the specs / CRs, put your name to have a contact, if you have an idea, when the patch will be ready, put the date as well.
14:26:26 <elmiko> #link https://etherpad.openstack.org/p/sahara-liberty-client
14:26:31 <elmiko> just for the logs =)
14:26:48 <SergeyLukjanov> tmckay, I've been creating it in time when you write the idea :)
14:26:52 <SergeyLukjanov> elmiko, thx
14:26:58 <tmckay> parallel, excellent
14:27:06 <SergeyLukjanov> tmckay, yeah, like a hadoop :)
14:27:20 <SergeyLukjanov> so, it'll help a lot to plan client release and merge all of the stuff in time
14:27:22 <tmckay> We Are a Cluster
14:27:32 <tellesnobrega> +1
14:27:33 <SergeyLukjanov> yay!
14:27:49 <SergeyLukjanov> I'll write a follow-up to mailing list as well
14:28:09 <tmckay> all, make sure you actually create the bug/spec/bp if you put ( words ) on the etherpad!
14:28:18 <tmckay> I'll do mine today
14:28:26 <SergeyLukjanov> random idea: do we probably want to make a saharaclient-core group with all sahara-cores plus someone else?
14:29:01 <elmiko> SergeyLukjanov: is it a different group in gerrit?
14:29:06 <tmckay> hmmm, do other projects do this?
14:29:16 <SergeyLukjanov> tmckay, swift already doing it
14:29:21 <crobertsrh> My first instinct is that It seems like it might be overkill
14:29:24 <SergeyLukjanov> tmckay, probably someone else
14:29:28 <SergeyLukjanov> crobertsrh, ++
14:29:39 <elmiko> yea, kinda agree with crobertsrh
14:29:41 <tmckay> what about combined client? will python-saharaclient go away eventually?
14:29:56 <SergeyLukjanov> but it could be helpful if will have someone very active in client
14:30:02 <SergeyLukjanov> not the case right now
14:30:04 <tosky> tmckay: the CLI client could go away, but not the library
14:30:05 <tmckay> I think we only have a few client changes per cycle, not too bad
14:30:10 <tmckay> gotcha
14:30:20 <SergeyLukjanov> tmckay, its python part still needed, it's only about combined CLI
14:30:20 <egafford> Agreed that it seems like overkill.
14:30:32 <tmckay> so I hear -1
14:30:39 <tosky> ("client" is a bit overloaded word in OpenStack, unfortunately)
14:30:47 <SergeyLukjanov> yeah
14:31:08 <SergeyLukjanov> #agreed no need for the separate saharaclient core group
14:31:16 <tmckay> crobertsrh, btw, I was going to jam through those spec approvals today
14:31:24 <tmckay> everyone has had more than enough time to comment :)
14:31:25 <crobertsrh> great
14:31:35 <tmckay> if someone has a big issue, they can re-open as a CR
14:32:06 <SergeyLukjanov> #topic Hadoop 1 drop
14:32:13 <SergeyLukjanov> I'd like to chat about it again
14:32:15 <tmckay> poor hadoop 1
14:32:40 <tosky> question: vanilla is the first, and then it will be for all other plugins, or is it up to each plugin "maintainer"?
14:33:00 <SergeyLukjanov> there were no requests for the hadoop 1 for the whole sahara life (even when it was savanna, eho, etc)
14:33:14 <SergeyLukjanov> tosky, HDP plugin is now maintained by community
14:33:20 <SergeyLukjanov> due to the lack of HWX support
14:33:21 <huichun> so we won't support hadoop1?
14:33:30 <SergeyLukjanov> huichun, do you need it? :)
14:33:48 <huichun> need not
14:34:46 <SergeyLukjanov> my reasoning for it - not to support the obsolete thing, it requires CIs, images, etc.
14:34:53 <SergeyLukjanov> but noone asking for it
14:35:06 <elmiko> +1
14:35:26 <egafford> SergeyLukjanov: HDP1 has been a repeat offender in terms of failing all the CI jobs, as well.
14:35:38 <tmckay> so do we need a deprecation cycle?
14:35:56 <tosky> SergeyLukjanov: is also Mapr 3.1.1 based on Hadoop 2?
14:35:57 <tmckay> deprecated for L, removed in "M"onster
14:36:07 <tellesnobrega> +1
14:36:10 <SergeyLukjanov> tmckay, probably not for the Hadoop 1
14:36:23 <SergeyLukjanov> not sure that we should keep them while noone using them
14:36:42 <elmiko> yea, makes no sense to keep it if there have been no requests for it.
14:36:46 <tellesnobrega> even apache has lost a bit of interest in hadoop 1
14:36:48 <tmckay> truth is, plugin SPI is relatively stable, wouldn't be hard for someone to maintain out-of-tree if they really wanted to
14:37:00 <SergeyLukjanov> tellesnobrega, correct
14:37:03 <egafford> tmckay: From a customer perspective, everyone's on 2. For enthusiasts, less sure; there could be someone out there running Hadoop 1 on Sahara in his/her basement who'd be very angry at us.
14:37:14 <SergeyLukjanov> tmckay, if we'll be going to extract plugins, it'll be additional work
14:37:20 <elmiko> egafford: lol
14:37:22 <tmckay> egafford, they always have Kiko
14:37:37 <SergeyLukjanov> plus sreshetnyak is now working on the HDP plugin rework and we'll be able to drop current HDP plugin as well
14:37:49 <tmckay> ack.  I have no problem with it
14:38:23 <SergeyLukjanov> and Liberty will be shipped to customers like early next year, so, I think it's pretty safe :)
14:38:37 <SergeyLukjanov> NikitaKonovalov, have you already created Hadoop 1 drop spec?
14:38:48 <NikitaKonovalov> yes
14:38:54 <tmckay> plus it makes the gate faster ++
14:39:04 <SergeyLukjanov> tmckay, exactly!
14:39:11 <NikitaKonovalov> #link https://review.openstack.org/#/c/192563/
14:39:14 <SergeyLukjanov> great
14:39:24 <SergeyLukjanov> so, we could discuss it offline in spec
14:39:40 <SergeyLukjanov> it sounds like we have partial agreement on dropping hadoop 1
14:40:02 <SergeyLukjanov> at least no users here ;)
14:40:16 <tmckay> we can always send an email about it to openstack-dev and see if anyone screams
14:40:28 <tmckay> Like a doctor saying "Does this hurt?"
14:40:43 <SergeyLukjanov> tmckay, yeah
14:40:45 <SergeyLukjanov> #topic Open discussion
14:40:49 <tmckay> then we can prescribe Hadoop 2 as the cure
14:41:03 <elmiko> has anyone talked with the cognitive team?
14:41:16 <elmiko> to find out what it is they are putting together?
14:41:20 <tmckay> good question.  channel is kind of quiet
14:41:27 <tmckay> (by kind of, I mean totally)
14:41:41 <tosky> oh, I have a question about unit tests
14:41:55 <tosky> quick copy&paste coming :)
14:41:57 <tosky> we have a global dependency on testtools>=0.9.6, but some unit test implicitely depends on newer versions
14:42:02 <tosky> for example, I've seen usage of assertRegex which works only with testtools>=1.2 (thanks to unittest2); and other stuff like that
14:42:26 <tosky> should we consider these usages as wrong and file bugs for them? Otherwise the global requirements have no meanings
14:42:30 <tosky> It's not a problem on the gates because they test only the last version of deps (so testtools 1.8)
14:43:06 <tellesnobrega> i have a question about the multiple clusters spec. sreshetnyak talked to me this week suggesting that i created a new api call, clusters/multiple, this way we keep compatibility and we dont to wait until API v2 to have this feature in
14:43:38 <tmckay> tosky, so shouldn't the global dep be moved past 0.9.6, if the gates use 1.8?
14:43:52 <tellesnobrega> sorry tosky, kinda broke the thought there
14:44:07 <tosky> tmckay: that's the other possibility, but I'm not sure how it works in that case
14:44:25 <tosky> tmckay: maybe there are requirements from distributions; ubuntu has 0.9.6, we ship 1.1...
14:44:32 <tosky> I have no idea
14:44:33 <tmckay> I see
14:44:51 <elmiko> tosky: maybe talk with the infra team to learn a little more about global reqs and why it's at 0.9.6?
14:45:05 <elmiko> otherwise, yea bugs would be good
14:45:10 <tosky> it would be worth to check, but maybe SergeyLukjanov knows something with his infra hat
14:45:36 <tosky> or at least, who can we talk about for this?
14:45:39 <tmckay> tosky, well, technically, it's not bounded.  It doesn't say >= 0.9.6, < XXXX
14:45:54 <tmckay> so, it's not really a bug per se.  I see your point, though
14:46:15 <tosky> tmckay: it is a bug, IMHO; it's not working with a supported version
14:46:17 <tmckay> I think we need to fix the global req.  bump it up, or bound it, or everything is okay
14:46:22 <elmiko> guess it depends how each distro satisfies those reqs
14:46:26 <tosky> so either you raise the dependency or you fix the code
14:46:32 <elmiko> tosky: +1
14:47:27 <tmckay> yeah, I think investigate moving the version in the global req, and if it can't be moved, then bound it, and find bugs
14:47:36 <SergeyLukjanov> oops, I'm return back :)
14:47:46 <pino|work> (imho the global requirements.txt that everybody must take is a sort of annoyance)
14:48:04 <tmckay> SergeyLukjanov, lots of talk about infra stuff ^^
14:48:10 <elmiko> pino|work: necessary evil i suppose ;)
14:48:11 <SergeyLukjanov> tosky, IMO it's good to bump testtools version
14:48:26 <SergeyLukjanov> tosky, there were the same input from heat team
14:48:39 <tosky> SergeyLukjanov: yep, they have a similar issues in their unit tests
14:48:39 <vgridnev> there are already one bump patch for testtools
14:48:40 <vgridnev> https://review.openstack.org/#/c/192574/
14:48:47 <tosky> that's what my grep says :)
14:49:08 <tosky> oh, from yesterday
14:49:15 <SergeyLukjanov> in fact, we should be able to work on a min version
14:49:21 <tosky> if the dependency is raised, will this be backported to kilo?
14:49:26 <tosky> because also kilo is affected
14:49:31 <SergeyLukjanov> but, the specific solution for it now is better to bump min version
14:49:46 <elmiko> if we can bump, that seems like the ideal
14:50:15 <SergeyLukjanov> let's review and +1/2 https://review.openstack.org/#/c/192574/
14:50:35 <elmiko> sounds good
14:51:11 <elmiko> tellesnobrega: i think adding new endpoints to v1.1 is fine with regards to v2. v2 will be more about improving the api and perhaps breaking some bad patterns from the past.
14:51:37 <tellesnobrega> sure
14:51:45 <tellesnobrega> anyone opposes to that?
14:51:52 <SergeyLukjanov> elmiko, ++
14:52:28 <tellesnobrega> i'm going to rewrite the spec detailing that i'm going to add a new call to the API
14:52:50 <tellesnobrega> also going to write the spec for the saharaclient
14:53:26 <elmiko> tellesnobrega: sounds good
14:53:31 <elmiko> SergeyLukjanov: have you talked with the cognitive team at all? (i'm curious what kind of overlap we might have with them)
14:53:58 <SergeyLukjanov> elmiko, (trying to remember what is cognitive)
14:54:05 <elmiko> MLaaS
14:54:32 <elmiko> i'm wondering if they are going to deploy spark or something
14:54:37 <tmckay> (mailing lists as a service)
14:54:42 <elmiko> lol
14:54:54 <tellesnobrega> tmckay, thats what i read
14:54:55 <tellesnobrega> lol
14:54:56 <elmiko> MachineLearning-aas
14:55:34 <tmckay> elmiko, we could just start posting stuff in the channel
14:56:07 <elmiko> yea, i'm just curious if anyone has talked to them yet. that will be the next step, but i have too much on my plate already ;)
14:56:16 <elmiko> SergeyLukjanov: fyi https://wiki.openstack.org/wiki/Cognitive
14:56:30 <SergeyLukjanov> oh, I remeber there were an email about it
14:56:48 <SergeyLukjanov> and I answered something like folks, please, don't duplicate :)
14:56:57 <SergeyLukjanov> it's good if they will use sahara as a base
14:57:07 <SergeyLukjanov> for example to provisioning cluster for ML
14:58:13 <elmiko> yea, that would be awesome
14:59:43 <tmckay> wow, we used the whole meeting
15:00:22 <tellesnobrega> the same with happened with the project Surge (stream processing on openstack)
15:00:35 <elmiko> interesting...
15:00:58 <carl_baldwin> Hi, L3 meeting scheduled for now.
15:01:24 <tellesnobrega> i guess our time is up
15:01:28 <tmckay> yep, we have to leave
15:01:30 <kchen> bye all
15:01:38 <elmiko> carl_baldwin: sorry
15:01:39 <huichun> bye
15:01:39 <esikachev> bye
15:01:43 <tellesnobrega> bye
15:01:47 <elmiko> just need SergeyLukjanov to #endmeeting
15:01:50 <carl_baldwin> elmiko: thanks!
15:02:00 <pc_m> hi
15:02:01 <openstack> carl_baldwin: Error: Can't start another meeting, one is in progress.  Use #endmeeting first.
15:02:16 <tmckay> #endmeeting