18:00:50 <SergeyLukjanov> #startmeeting sahara
18:00:51 <openstack> Meeting started Thu Jan 22 18:00:50 2015 UTC and is due to finish in 60 minutes.  The chair is SergeyLukjanov. Information about MeetBot at http://wiki.debian.org/MeetBot.
18:00:52 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
18:00:54 <openstack> The meeting name has been set to 'sahara'
18:00:58 <weiting> Hi
18:01:02 <elmiko> yo/
18:01:18 <alazarev> o/
18:01:21 <tmckay> hello
18:02:03 <SergeyLukjanov> okay, let's start
18:02:04 <jodah> yo
18:02:08 <crobertsrh> hello/
18:02:16 <SergeyLukjanov> #link http://wiki.openstack.org/wiki/Meetings/SaharaAgenda
18:02:30 <SergeyLukjanov> #topic sahara@horizon status (crobertssh, NikitaKonovalov)
18:02:36 <SergeyLukjanov> crobertsrh, *
18:02:44 <crobertsrh> Here's the list of open sahara-horizon reviews:  https://etherpad.openstack.org/p/sahara-reviews-in-horizon
18:02:58 <crobertsrh> I don't think I've seen much change since last week.
18:03:01 <ylobankov> hello
18:03:26 <alazarev> my horizon patches are still on review, no much progress
18:03:30 <SergeyLukjanov> crobertsrh, thanks!
18:03:39 <NikitaKonovalov> yes, looks like all the changes are where they were a week ago
18:03:48 <SergeyLukjanov> heh :(
18:03:56 <SergeyLukjanov> #topic News / updates
18:04:06 <crobertsrh> I will try to rattle the chains of the horizon people at their next meeting.
18:05:06 <tosky> can we talk about the proposed blueprint for dynamic integration tests and the two proposed reviews about the same topic?
18:05:13 <egafford> Hello
18:05:16 <elmiko> continuing work on the security chapter, i've got good engagement with the OSSG adn things are moving along. investigating possible barbican usage within sahara.  also put a bug fix and following several reviews.
18:06:05 <alazarev> I was busy with multithreading for sahara engine, don't know why, but it interfere with sqlalchemy, for now I'm ready to give up
18:08:19 <SergeyLukjanov> anything else?
18:08:35 <weiting> we are working on adding service test in cdh-plugin
18:08:39 <SergeyLukjanov> tosky, let's discuss it a bit later
18:08:50 <tosky> sure
18:08:53 <SergeyLukjanov> tosky, I mean after defined topic
18:08:58 <weiting> And ken is working working on cdh version management.
18:09:06 <SergeyLukjanov> weiting, cool
18:09:17 <SergeyLukjanov> we're waiting for the version management :)
18:09:36 <weiting> Yes, we will update it ASAP.
18:10:30 <SergeyLukjanov> #topic Kilo release schedule
18:10:52 <SergeyLukjanov> #link https://wiki.openstack.org/wiki/Kilo_Release_Schedule
18:10:55 <tmckay> oops, I missed updates :)
18:11:04 * SergeyLukjanov just wants to be sure that everyone now dates
18:11:08 <tmckay> In open discussion I'll ask for reviews :)
18:11:21 <SergeyLukjanov> any questions re schedule?
18:11:23 <SergeyLukjanov> tmckay, okay :)
18:12:25 <SergeyLukjanov> #topic Bug / doc / spec days
18:12:33 <SergeyLukjanov> IMO bug triage was awesome
18:12:40 <SergeyLukjanov> thanks folks
18:12:51 <SergeyLukjanov> and remember about bug fix day next monday
18:12:57 <elmiko> still a few undecideds out there, but we'll get em =)
18:13:18 <tmckay> some of the DIB and cluster launch bugs took a long time to check :)
18:13:25 <elmiko> yea...
18:13:25 <tmckay> looong time
18:14:00 <tmckay> SergeyLukjanov, we might want to do another set of triage/fix later in the cycle, depending on how many "news" we still have
18:14:16 <alazarev> also there is a number of HDP bugs, and no one fixes them...
18:14:24 <SergeyLukjanov> tmckay, yup, agree
18:14:39 <SergeyLukjanov> alazarev, you could volunteer to fix them
18:15:01 <SergeyLukjanov> :)
18:15:08 <SergeyLukjanov> okay, let's move on
18:15:12 <SergeyLukjanov> on tosky request
18:15:15 <SergeyLukjanov> #topic "New integration tests for sahara" https://review.openstack.org/#/c/147219/
18:15:25 <SergeyLukjanov> tosky, I presume you have questions?
18:15:34 <SergeyLukjanov> tosky, or just want to -2 it? :)
18:16:29 <SergeyLukjanov> hm
18:16:44 <elmiko> i think he had issue with how complex the code generation part was getting
18:16:45 <SergeyLukjanov> I'm adding this topic to the next meeting too, because I think it should be good discussed
18:17:01 <elmiko> it started off as a simple template, but now that template is growing
18:17:06 <tosky> well, not -2, questions before going with votes
18:17:14 <elmiko> i also have a meta issue with this process
18:17:24 <elmiko> we have way too many reviews going up before the specs are merged
18:17:24 <SergeyLukjanov> elmiko, IMO it's not an issue if we'll cover it with unit tests
18:17:34 <tosky> first question is the one from elmiko, about the growing hidden complexity of the template
18:17:35 <SergeyLukjanov> elmiko, IMO it'll be pretty static code
18:18:10 <SergeyLukjanov> elmiko, it's ok to have reviews for not approved specs
18:18:11 <tosky> do you think that the template coming from the second review is generic enough that it will be quite stable?
18:18:25 <SergeyLukjanov> elmiko, we should just be sure that we aren't merging them
18:18:40 <elmiko> SergeyLukjanov: ok, i'll hold off on -2 unless they are about to merge
18:18:58 <elmiko> SergeyLukjanov: my worry is that people are implementing things before we finish debate on the specs, sometimes...
18:18:59 <tosky> but at least please update the specs so that they match the code, which changed a bit if I see it correctly
18:19:08 <elmiko> tosky: +1
18:19:13 <SergeyLukjanov> tosky, I have some ideas on adding complex features like if you specified two images than two test cases will be created
18:19:43 <SergeyLukjanov> tosky, code could be not in sync, it's marked work in progress I think
18:19:59 <tosky> SergeyLukjanov: I fear that the spec need more investigation on the complex cases
18:20:31 <elmiko> i have the same fear as tosky ;)
18:20:36 <SergeyLukjanov> tosky, my PoV is to start with the simple case and than add some complexity if it'll be really useful
18:20:39 <tosky> ok, code not in sync, but they are connected
18:21:08 <SergeyLukjanov> I have a bit different fear - current integration tests
18:21:17 <elmiko> SergeyLukjanov: also a valid fear
18:21:20 <tosky> yes, I'm from going on and refactoring, but I always fear, in such a cases, that a some constraint in the initial phase could make difficult to change things later
18:21:22 <SergeyLukjanov> they are extremely boilerplated
18:21:54 <SergeyLukjanov> let's try to split the question
18:21:58 <tosky> I totally agree with the idea behind the change, as I wrote
18:22:34 <SergeyLukjanov> the first is "does everybody agree that we need new new testing framework?"
18:22:38 <tosky> yes
18:22:53 <SergeyLukjanov> the second is "does everyone agree that it should be flexible and very configurable?"
18:22:58 <tosky> yes again
18:23:08 <elmiko> yes and yes
18:23:24 <SergeyLukjanov> and the third is "wtf it should be?"
18:23:32 <tosky> the first two points are not in discussion here; it's about the implementation
18:23:39 <SergeyLukjanov> ack
18:23:57 <SergeyLukjanov> I just would like to be sure that you're agree with base part
18:23:58 <tosky> which should be good enough to not need a refactoring at the next cycle
18:24:07 <SergeyLukjanov> yeah
18:24:26 <SergeyLukjanov> personally I prefer to have a very complex yaml -> test case transformer
18:24:32 <SergeyLukjanov> covered by tests
18:24:41 <SergeyLukjanov> that'll support jobs and etc. definitions
18:24:52 <SergeyLukjanov> that'll support several yaml files
18:25:02 <SergeyLukjanov> that'll support auto gen for array values
18:25:08 <tosky> I personally agree too with having an engine with parses the code, the things I'm not sure are something later
18:25:31 <SergeyLukjanov> I'd like to support features like "images: cdh-fedora, cdh-ubuntu, cdh-windows"
18:25:54 <SergeyLukjanov> tosky, could you please add more details?
18:25:57 <tosky> for example: why writing down the test class with a template, instead of creating the test classes in memory with metaprogramming? Because reporting will show results always from the "runner" class?
18:27:34 <SergeyLukjanov> tosky, because we're using discover + testr
18:27:45 <SergeyLukjanov> discover searches through the python file directories
18:28:00 <SergeyLukjanov> and generates list of test that should be executed
18:28:06 <tosky> ok, limitation of the tool, ack
18:28:12 <SergeyLukjanov> tosky, yeah
18:28:15 <tosky> or design of the tool, whatever :)
18:28:21 <SergeyLukjanov> tosky, yup
18:28:38 <SergeyLukjanov> and we're really like to avoid creating new tools for running tests ;)
18:28:44 <tosky> sure, sure
18:29:46 <tosky> oki, then I have some question about the way the code is going; for example, the reviews show json instead of yaml and a draft of composed files (something I asked on the spec), but the state of both things (spec and code reviews) work in progress but not in sync confused me a bit
18:30:30 <SergeyLukjanov> tosky, sorry about this
18:30:41 <SergeyLukjanov> tosky, I'd like to use yaml parser
18:30:57 <SergeyLukjanov> and if user's prefer to use json they could
18:31:10 <SergeyLukjanov> because json is a subset of yaml
18:31:46 <SergeyLukjanov> I'll ask sreshetnyak to update spec to the actual state
18:31:54 <elmiko> cool
18:32:05 <tosky> oki, the question about the complexity: the second revision of the testcase.py.mako shows already a more general (with dynamic testcases) but some "if" (special cases)
18:32:09 <tosky> https://review.openstack.org/#/c/148935/1/sahara/tests/scenario/testcase.py.mako
18:32:16 <SergeyLukjanov> spec should be the main source of answers
18:32:24 <elmiko> SergeyLukjanov: +1
18:32:46 <tosky> so I fear again that, if this the direction, this could not be generic enough, and we would need many template
18:32:54 <tosky> which could be fine, but it's something I would like to see in the spec
18:33:14 <SergeyLukjanov> okay
18:33:16 <tosky> or, on the other side, the idea is to have a really generic template with some trick and parameters
18:33:18 <tosky> that's it :)
18:33:19 <SergeyLukjanov> sreshetnyak, ^^
18:33:31 <SergeyLukjanov> IMO template is only a view
18:33:38 <SergeyLukjanov> and ready data should be passed to it
18:33:39 <tosky> it's also to avoid more work for sreshetnyak and ylobankov , as they won't need to refactor more and more
18:33:49 <SergeyLukjanov> and no conditions on the view side
18:34:06 <elmiko> i'm +1 for templates without logic embedded
18:34:25 <tosky> yep
18:34:28 * SergeyLukjanov likes mvc, mvvm, etc. so I hate business logic in view
18:34:29 <sreshetnyak> hello
18:35:42 <SergeyLukjanov> sreshetnyak, please, read notes in meeting minutes :)
18:35:57 <sreshetnyak> okay :)
18:35:59 <SergeyLukjanov> tosky, elmiko anything else on it?
18:36:07 <elmiko> nothing from me
18:36:16 <tosky> I think it's all from my side, thanks again
18:36:29 <SergeyLukjanov> so, re this new framework
18:36:38 <SergeyLukjanov> my dream is to run sahara-ci using it
18:36:46 <SergeyLukjanov> make acceptance testing of milestones
18:37:00 <SergeyLukjanov> and deprecate current tests
18:37:09 <elmiko> nice dream =)
18:37:15 <tosky> but doable
18:37:21 <elmiko> agreed
18:37:31 <alazarev> +1
18:37:32 <SergeyLukjanov> for me there are two levels of feature parity - first one is to replace tests for sahara-ci
18:37:45 <SergeyLukjanov> the second is the full feature parity with current tests
18:39:04 <tosky> sounds like a plan
18:39:26 <elmiko> that would be really cool
18:39:38 <tmckay> +1, the current tests vary widely depending on when and who implemented them
18:40:53 <crobertsrh> +1 from me.  Sounds like a good plan.
18:41:15 <SergeyLukjanov> okay, nice to see only +1s ;)
18:42:00 <SergeyLukjanov> sreshetnyak, will update the spec tomorrow to the actual state and probably we need to land it if everyone will ok with it
18:42:13 <elmiko> ack
18:42:14 <SergeyLukjanov> anything else on it?
18:43:12 <sreshetnyak> SergeyLukjanov, yes, need to update spec
18:43:17 <SergeyLukjanov> #topic Open discussion
18:43:40 <elmiko> #link https://etherpad.openstack.org/p/sahara-security-guide-notes
18:43:45 <elmiko> still looking for more feedback
18:44:00 <elmiko> there are some questions as well about how much we should recommend
18:44:05 <elmiko> with regards to hadoop and security
18:44:17 <tmckay> #link https://review.openstack.org/#/c/146659/
18:44:17 <tmckay> #link https://review.openstack.org/#/c/147949/
18:44:26 <tmckay> #link https://review.openstack.org/#/c/147955/
18:44:40 <tmckay> swift - spark integration, almost done, please reveiw :)
18:44:55 <tmckay> I had it dependent on the Java Oozie compat stuff but I split it off
18:45:44 <tmckay> last question: what do do about jackson jar?  1) update CDH, hopefully that fixes it 2) add jackson jar to sahara images as an element (host with the hadoop-swift.jar 3) carry it as resource in Sahara and upload until we have it fixed
18:45:51 <elmiko> SergeyLukjanov: so on the 147955 review, why -2?
18:46:20 <tmckay> oh, elmiko, because I wrote the code before the spec :)
18:46:27 <tmckay> I am bad.  I forgot
18:46:27 <elmiko> ahh ok
18:46:35 * elmiko wants to learn the rules better
18:46:37 <tmckay> that's why we need the reviews
18:47:06 <elmiko> ack
18:47:12 <SergeyLukjanov> yup
18:47:20 <SergeyLukjanov> we need a bit more active reviews of specs
18:47:33 <tmckay> what do you all think about the jackson jar issue?  I don't want to modify the spark assembly by hand, so depending on the particular package, we might have to carry patch jars
18:47:40 * crobertsrh has been slacking on spec reviews
18:47:46 <tmckay> is that okay in general, do you think?
18:48:20 <tmckay> ack, I have been slow on reviews in 2015, that will change ASAP.  Today :)
18:49:36 <tmckay> so the spark assembly pulls in a version of a class that conflicts with the hadoop-swift impl.  This could  happen again.
18:50:01 <tmckay> I am fine with adding jars to the mirantis hosted stuff and adding elements as needed
18:50:02 <elmiko> ooph
18:50:42 <tmckay> for spark, we can mess with the classpath on the nodes, no problem
18:51:03 <tmckay> but I want to know if you all think I am crazy :)
18:51:35 <elmiko> it seems like patch jars might be worst-best solution :/
18:51:54 <elmiko> or would that be best-worst? lol
18:52:17 <tmckay> don't know.  Like I said, if CDH 5 fixes it, all gone.  But it might not.
18:52:34 <elmiko> well, and what about getting spark into other images?
18:52:34 <SergeyLukjanov> 8 mins left
18:52:34 <tmckay> other possibility is modify hadoop-swift and try to code around it.
18:52:56 <elmiko> that sounds tougher
18:53:05 <tmckay> elmiko, I have to look at just where that is being pulled from (the spark assembly)
18:54:18 <tmckay> since no one seems to have a strong opinion, I'll assume if the problem can't be fixed with an up-rev that we can carry an additional jar as an element
18:54:32 <elmiko> sounds like no opposition
18:54:42 <crobertsrh> No opposition here
18:54:43 * tmckay should propose other things
18:54:57 <elmiko> lol
18:55:15 <tmckay> I propose hot pretzels with mustard in the Sahara pod at Summit
18:55:23 <elmiko> +1
18:56:37 <SergeyLukjanov> :)
18:56:38 <tmckay> oh, everyone, general question
18:56:50 <crobertsrh> I propose that tmckay makes that happen
18:57:10 <tmckay> we have seen lately on F21 that launching a cluster (nova) cause the network on the host to be taken out when the bridging is done
18:57:21 <tmckay> seems to be something in a recent F21 update
18:57:28 <tmckay> F20 is fine, older F21 is fine
18:57:38 <tmckay> anybody see anything like this?
18:57:46 * tosky is still on F20
18:57:53 <crobertsrh> I see it on my machine
18:57:55 <tmckay> It has something to do with the routing table, I think, but it's not obvious to me
18:58:14 <tmckay> I am not a routing table expert
18:58:24 <tmckay> maybe the nova/neutron guys could help
18:58:29 <crobertsrh> I tried asking about it in openstack-nova, but got nothing but crickets
18:58:50 <elmiko> might try in openstack-dev
18:58:58 <elmiko> lots of devstack questions there
18:59:04 <tmckay> yeah, probably worth a shot
18:59:28 <tmckay> I am revving back to F20 this afternoon/evening
18:59:48 <elmiko> tmckay: also in fedora, devstack will attempt to remove firewalld, that usually doesn't cause issues, but maybe in f21 it does
19:00:07 <tmckay> hmm, maybe
19:00:39 <elmiko> times up!
19:01:03 <SergeyLukjanov> #endmeeting