18:02:39 <SergeyLukjanov> #startmeeting sahara
18:02:40 <NikitaKonovalov> o/
18:02:41 <openstack> Meeting started Thu Oct  9 18:02:39 2014 UTC and is due to finish in 60 minutes.  The chair is SergeyLukjanov. Information about MeetBot at http://wiki.debian.org/MeetBot.
18:02:42 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
18:02:43 <dmitryme> o/
18:02:44 <sreshetnyak> o/
18:02:45 <openstack> The meeting name has been set to 'sahara'
18:02:50 <aignatov> o/
18:02:56 <ylobankov1> hello
18:03:13 <elmiko> o/
18:03:20 <SergeyLukjanov> ping ErikB, NikitaKonovalov, RobLevas, SergeyLukjanov, aignatov, alazarev, bob_nettleton, crobertsrh, dmitryme, elmiko, jspeidel, mattf, skostiuchenko, sreshetnyak, tellesnobrega, themistymay, tmckay, tosky, ylobankov
18:03:27 <alazarev> o/
18:03:49 <SergeyLukjanov> #link https://wiki.openstack.org/wiki/Meetings/SaharaAgenda
18:04:01 <tosky> o/
18:04:07 <SergeyLukjanov> #no action items from the last meetings
18:04:14 <SergeyLukjanov> #info no action items from the last meetings
18:04:22 <SergeyLukjanov> #topic sahara@horizon status (croberts, NikitaKonovalov)
18:04:39 <SergeyLukjanov> any last moment issues w/ sahara@horizon?
18:04:43 <NikitaKonovalov> Ok, there is still one CR not merged yet
18:04:59 <NikitaKonovalov> the one that should fix job executions table
18:05:12 <NikitaKonovalov> but that's minor one I guess
18:05:25 <NikitaKonovalov> everything else looks good
18:05:26 <SergeyLukjanov> is it important for J release?
18:05:32 <alazarev> NikitaKonovalov, what's wrong with job executions table?
18:05:52 <SergeyLukjanov> but honestly it's too late to include something not super-critical to J
18:06:05 <tmckay> if you delete a cluster and leave executions that ran on it, it breaks the display
18:06:06 <NikitaKonovalov> alazarev: It fails with exception if a cluster was deleted
18:06:17 <tmckay> I found this yesterday :)
18:06:21 <alazarev> NikitaKonovalov, oh
18:06:23 <tmckay> nice fix, it works :)
18:06:40 <NikitaKonovalov> #link https://review.openstack.org/#/c/125927/
18:06:44 <NikitaKonovalov> here is the fix
18:07:10 <SergeyLukjanov> okay, it'll be the known issue I presume
18:07:27 <tmckay> you should be able to work around by using the CLI to delete old job executions
18:07:38 <tmckay> then you can use the UI for job executions again
18:07:44 <SergeyLukjanov> NikitaKonovalov, please ensure that it'll be included to stable/juno release (2014.2.1)
18:07:53 <SergeyLukjanov> tmckay, yeah
18:07:56 <tmckay> only reason to go in there is relaunch, or check on how you submitted
18:08:05 <NikitaKonovalov> SergeyLukjanov: ok, I'll ping horizon reviewers
18:08:25 <tmckay> I gave a +1 :)
18:08:28 <SergeyLukjanov> great, let's move on
18:08:44 <SergeyLukjanov> #topic News / updates
18:08:46 <SergeyLukjanov> folks, please
18:09:14 <SergeyLukjanov> #info Juno RC2 successfully released https://launchpad.net/sahara/juno/juno-rc2
18:09:33 <tmckay> I discovered an edge case yesterday, sort of.  Not sure if it's important enough to mention in the docs.
18:09:37 <SergeyLukjanov> and we've included mostly everything we've been wanted to include to Juno
18:09:41 <elmiko> i've been doing a bunch of juno packaging for RDO, and some icehouse cleanup for rhel. the namespace/rootwrap review is looking good but i could use a few more eyes on it, https://review.openstack.org/#/c/125518/
18:10:08 <SergeyLukjanov> elmiko, we'll include it into the first stable release for sure
18:10:17 <elmiko> SergeyLukjanov: cool
18:10:34 <SergeyLukjanov> tmckay, do you want to approve it? :)
18:10:34 <sreshetnyak> I rebuild all images for Sahara with bug fix for shellshock
18:10:35 <tmckay> If you are running an isolated Sahara against devstack without an endpoint, it's not enough to fix up get_sahara_url as described here http://docs.openstack.org/developer/sahara/horizon/dev.environment.guide.html
18:10:51 <tmckay> You have to remove the permissions for data_processing, too
18:11:16 <sreshetnyak> all images available on http://sahara-files.mirantis.com/
18:11:16 <SergeyLukjanov> tmckay, could you make a patch for docs please?
18:11:19 <tmckay> SergeyLukjanov, approve elmiko fix?
18:11:33 <SergeyLukjanov> tmckay, https://review.openstack.org/#/c/125518/11
18:11:36 <tmckay> SergeyLukjanov, okay
18:11:42 <tmckay> (on both :) )
18:12:26 <tmckay> approved!
18:13:01 <alazarev> I've added initial documentation to openstack manuals
18:13:12 <SergeyLukjanov> alazarev, great work
18:13:19 <tmckay> I was running Sahara on a cluster where I'm not supposed to have admin privileges to add an endpoint :)
18:14:37 <SergeyLukjanov> tmckay, elmiko, are you planning to make some screencasts or something like that?
18:14:55 <elmiko> we hadn't talked about it
18:15:00 <tmckay> SergeyLukjanov, for swift stuff?
18:15:06 <SergeyLukjanov> tmckay, elmiko it'll be really great to have some from you guys
18:15:27 <SergeyLukjanov> tmckay, elmiko for any stuff - it could be Juno overview or swift stuff
18:15:32 <elmiko> SergeyLukjanov: i've starting writing a blog post about configuring swift proxies and whatnot
18:15:36 <SergeyLukjanov> tmckay, elmiko or awesome spark jobs demo :)
18:15:41 <SergeyLukjanov> elmiko, yay!
18:15:52 <crobertsrh> +1 to some more sahara videos :)
18:15:54 <tmckay> SergeyLukjanov, okay, maybe we can come up with something
18:16:00 <elmiko> i have one up now about configuring sahara using the python client
18:16:13 <SergeyLukjanov> NikitaKonovalov is now working on "What's new in Juno" blog post with demo screencast
18:16:18 <tmckay> I know some guys who probably have awesome spark demos :-D
18:16:24 <elmiko> lol
18:16:25 <tmckay> It will have to be hdfs though
18:16:41 <tmckay> swift for spark is still, um, custom at this point
18:16:44 <SergeyLukjanov> yeah, I'd like to popularize sahara :)
18:17:00 <SergeyLukjanov> more blog posts, more videos ;)
18:17:19 <SergeyLukjanov> folks, please review https://wiki.openstack.org/wiki/Sahara/ReleaseNotes/Juno
18:17:22 <elmiko> SergeyLukjanov: i need to add it to openstack planet, but elmiko.github.io is where i have been writing stuff about sahara and openstack
18:17:42 <tmckay> hmm, tellesnobrega screencast for storm would be good too
18:17:43 <SergeyLukjanov> elmiko, great!
18:17:50 <SergeyLukjanov> tmckay, exactly
18:18:18 <SergeyLukjanov> okay
18:18:20 <SergeyLukjanov> #topic Juno status (SergeyLukjanov)
18:18:33 <SergeyLukjanov> #link https://wiki.openstack.org/wiki/Juno_Release_Schedule
18:18:36 <alazarev> what I don't like in current release notes is that "New Key Features" is small, but "Upgrade Notes" is huge
18:18:39 <SergeyLukjanov> release will be next week ;)
18:18:58 <SergeyLukjanov> as I said before rc2 happens
18:19:12 <SergeyLukjanov> does everybody remember "ship happens"?
18:19:19 <elmiko> alazarev: +1
18:19:42 <SergeyLukjanov> alazarev,  agreed, we need to work on the new features part
18:19:55 <elmiko> i've got the rc2 tarballs deployed into the RDO infrastructure, so we are in good shape there
18:20:04 <SergeyLukjanov> #info We need review for https://wiki.openstack.org/wiki/Sahara/ReleaseNotes/Juno
18:20:13 <SergeyLukjanov> elmiko, awesome
18:20:25 <SergeyLukjanov> and we'll have debian stuff done by zigo
18:20:51 <elmiko> nice
18:20:54 <aignatov> hmm, alazarev, I’d like how it looks now
18:21:12 <SergeyLukjanov> let's move on
18:21:15 <SergeyLukjanov> #topic Upcoming release schedule (SergeyLukjanov)
18:21:19 <aignatov> upgrade notes has the same size in page as new key features
18:21:25 <SergeyLukjanov> #link https://wiki.openstack.org/wiki/Kilo_Release_Schedule
18:21:33 <SergeyLukjanov> so, please, take a look on the Kilo release schedule
18:22:00 <SergeyLukjanov> and the next summit will be in Vancouver
18:22:06 <elmiko> aignatov: i think maybe it would be better to format the "new key features" similiar to "upgrade notes", basically paragraphs instead of unordered list
18:22:17 <SergeyLukjanov> elmiko, +1
18:22:35 <aignatov> ops, my html page was broken some how :) now I see what you meant alazarev
18:22:56 <SergeyLukjanov> #topic Design Summit @ Paris
18:23:06 <SergeyLukjanov> folks, we have only 3 weeks before the summit
18:23:14 <SergeyLukjanov> so, time to start working on sessions
18:23:20 <SergeyLukjanov> #link https://etherpad.openstack.org/p/kilo-sahara-summit-topics
18:23:38 <SergeyLukjanov> so, please, add all of ideas to this etherpad
18:23:48 <crobertsrh> 5 sessions + half day meetup, right?
18:23:49 <aignatov> will we vote for sessions as it happens in etherpad right now?
18:24:09 <SergeyLukjanov> and let's allocate the most part of the next meeting to define list of sessions and their contents
18:24:12 <SergeyLukjanov> crobertsrh, exactly
18:24:17 <elmiko> aignatov and i are using the same color on etherpad for increased stealth ;)
18:24:21 <SergeyLukjanov> #info 5 sessions + half day meetup
18:24:49 <crobertsrh> What do we expect the "meetup" to consist of?  Is it really just extra sessions?
18:25:04 <SergeyLukjanov> aignatov, let's discuss it next meeting - I'd like to initially discuss the most important topics
18:25:05 <aignatov> SergeyLukjanov: when will the last day for proposals?
18:25:15 <SergeyLukjanov> aignatov, November 1
18:25:20 <SergeyLukjanov> or 2nd
18:25:31 <SergeyLukjanov> crobertsrh, it's a schedule-free time
18:25:43 <SergeyLukjanov> crobertsrh, we need to have a list of topics to discuss in a free form
18:25:55 <SergeyLukjanov> crobertsrh, we could do it by groups'
18:25:56 <tmckay> or we can just write code
18:26:04 <aignatov> tmckay: lol
18:26:07 <elmiko> i nominate crobertsrh to bring the beverages for the meetup
18:26:13 <crobertsrh> Ok....maybe some of the topics that don't get their own session :)
18:26:14 <tmckay> Beverly
18:26:17 <elmiko> lol
18:26:17 <SergeyLukjanov> and it's a good time to brainstorm some important stuff
18:26:28 <crobertsrh> crobertsrh only brings Beverly and Bourbon
18:26:34 <elmiko> gaah
18:26:34 <SergeyLukjanov> heh :)
18:26:39 <crobertsrh> and I'm all out of Beverly
18:26:57 <tmckay> I'm flying through Atlanta, I'll put some in a barrel on the way
18:27:01 * SergeyLukjanov need to aggregate a list of great places in Paris
18:27:06 <elmiko> SergeyLukjanov: +1
18:27:28 <crobertsrh> Yes....There is at least one '"official" party I see on the schedule.
18:27:38 <SergeyLukjanov> we'll have aignatov, dmitryme, NikitaKonovalov and me from the team this time on summit
18:28:09 <elmiko> tmckay, crobertsrh, mattf, and me from rh
18:28:15 <SergeyLukjanov> awesome
18:28:16 <tosky> and me
18:28:18 <elmiko> ooh and tosky
18:28:26 <crobertsrh> awesome tosky!  I didn't know you were going!
18:28:31 <tmckay> and some paris rh guys I think
18:28:36 <elmiko> yea, i almost forgot the best part =)
18:29:08 <tosky> well, colleagues from eNovance will be there I guess
18:29:12 <SergeyLukjanov> heh, it'll be the easiest summit flight for me and Russian folks - just ~3 hours direct flight
18:29:24 <elmiko> nice
18:29:48 <tmckay> well, I can't complain, Atlanta was short for me
18:29:51 <SergeyLukjanov> we need to have a jetlag-fighting party the day before summit :)
18:29:56 <elmiko> lol
18:30:04 <tmckay> I should be there Sunday around 11am
18:30:12 <tmckay> just in time for lunch
18:30:40 <elmiko> i think crobertsrh and i will be there around the same time
18:30:44 <SergeyLukjanov> cool, I don't know when we'll arrive but hope not at midnight
18:30:59 <crobertsrh> Yeah, I think we get in around 8am on Sunday
18:31:39 <SergeyLukjanov> okay
18:31:45 <SergeyLukjanov> #topic Specs re-org (SergeyLukjanov)
18:31:47 <aignatov> so, there will not be exact schedule for sessions. am I right, SergeyLukjanov ? If you say that Nov 1st is the last day for proposals, so there is no time to compose final schedule to post it in offecial site?
18:32:19 <SergeyLukjanov> aignatov, there will be official schedule for 5 sessions, but meetup schedule will be maintained by me
18:32:42 <SergeyLukjanov> aignatov, more than that, I'm sure that we could push updates to sessions schedule till the start of summit
18:33:00 <SergeyLukjanov> aignatov, but it's better to avoid it to make other folks able to compose own schedule for summit
18:33:13 <SergeyLukjanov> so, talking about specs re-org
18:33:17 <aignatov> so this link https://etherpad.openstack.org/p/kilo-sahara-summit-topics contains topics for meetup?
18:33:27 <aignatov> or for 5 sessions?
18:33:32 <SergeyLukjanov> it contains topics for both sections
18:34:03 <SergeyLukjanov> specs
18:34:13 <aignatov> ok, thx, sorry for interrruption, go ahead
18:34:19 <SergeyLukjanov> so, I'd like to propose re-org
18:34:22 <SergeyLukjanov> let me share the link
18:34:38 <SergeyLukjanov> http://specs.openstack.org/openstack/nova-specs/
18:34:55 <SergeyLukjanov> I really like the idea of implemented / approved specs dirs
18:35:14 <elmiko> seems nice
18:35:23 <alazarev> +1
18:35:29 <SergeyLukjanov> it will make us able to keep track on what stuff has been implemented and what stuff is just approved
18:35:40 <tmckay> +1
18:35:46 <elmiko> SergeyLukjanov: is that page auto-generated?
18:36:01 <SergeyLukjanov> so, if there will be no objections I'll propose changes to specs repo later today or tomorrow
18:36:02 <tmckay> elmiko swagger plugin
18:36:09 <tosky> is this up-to-date or just a draft, then? http://specs.openstack.org/openstack/sahara-specs/
18:36:29 <SergeyLukjanov> elmiko, not at all, we'll need to have nested dirs for implemented / approved
18:36:34 <elmiko> tmckay: i have been doing some research on the side
18:36:37 <SergeyLukjanov> tosky, it's an up-to-date page
18:36:54 <aignatov> we always could see on implemented/approved blueprints because it’s actually has direct ampping for specs :)
18:36:58 <elmiko> SergeyLukjanov: ack, still +1 for me
18:37:06 <SergeyLukjanov> more than that you could subscribe to rss on http://specs.openstack.org/openstack/sahara-specs/rss/
18:37:07 <aignatov> but I’m +1 for this re-org :)
18:37:12 <crobertsrh> +1
18:37:21 <SergeyLukjanov> aignatov, we'll do it eventually
18:37:34 <SergeyLukjanov> cool, so, sounds like no objections right now
18:37:37 <SergeyLukjanov> thanks
18:37:58 <SergeyLukjanov> #agreed re-org to have approved / implemented separation for specs
18:38:34 <SergeyLukjanov> and I think we're fully ready to say that specs is working very good
18:38:38 <elmiko> tmckay: i think what i'm going to do is work up an example to show off and then we can debate it's value to the project
18:38:43 <SergeyLukjanov> to improve quality of our blueprints
18:38:54 <SergeyLukjanov> and IMO specs are very useful for us
18:39:20 <tmckay> elmiko, sounds good
18:39:25 <elmiko> SergeyLukjanov: agreed, i think we could be a little more rigorous about the depth of our specs.
18:39:38 <alazarev> SergeyLukjanov, agreed, specs work good
18:39:41 <elmiko> some of the specs are pretty light on details
18:39:56 <SergeyLukjanov> elmiko, yeah, I think KIlo is a time for mandatory detailed specs
18:40:04 <elmiko> SergeyLukjanov: +1
18:40:17 <SergeyLukjanov> we could discuss it on summit and have "approved-on-summit" sign for it
18:40:24 <elmiko> nice
18:40:54 <SergeyLukjanov> what's is the swagger plugin?
18:41:17 <SergeyLukjanov> oh, I find it
18:41:21 <elmiko> swagger is a language agnostic spec for describing rest apis
18:41:29 <SergeyLukjanov> so, I have a bad news for it
18:41:39 <SergeyLukjanov> as being official OpenStack project
18:41:51 <SergeyLukjanov> we need to have a WADL definition for our REST API
18:42:13 <SergeyLukjanov> and we need to do it early KIlo
18:42:22 <SergeyLukjanov> probably any volunteers for it?
18:42:48 <elmiko> i'd be down for looking at it
18:42:59 <SergeyLukjanov> elmiko, thank tou
18:43:05 <crobertsrh> It does seem to jive with your swagger
18:43:05 <tmckay> not familiar with WADL, I'll have to take a look
18:43:10 <SergeyLukjanov> it's really an important thing for us
18:43:11 <elmiko> some of swagger stuff is a super-set of WADL, so it's close
18:43:43 <SergeyLukjanov> we need to have it to include Sahara rest api to http://developer.openstack.org/api-ref.html
18:44:06 <SergeyLukjanov> that's awesome IMO to have our API exposed at this doc
18:44:06 <alazarev> can it be partly autogenerated?
18:44:19 <elmiko> alazarev: ideally it is all auto-generated
18:44:38 <alazarev> because it looks a really huge task to write it manually
18:44:49 <tmckay> ack, we need tooling for sure
18:44:55 <SergeyLukjanov> yeah
18:44:58 <tmckay> because otherwise it will never be right
18:45:05 <elmiko> we would implement some meta-coding into the rest decorators, and similar, that will allow the creation of the api spec
18:45:11 <SergeyLukjanov> I'll try to find folks how have been working on it for other projects
18:45:19 <SergeyLukjanov> elmiko, yup
18:45:43 <elmiko> ideally, a properly implemented server can look at sahara as a client to autodiscover the rest api
18:45:45 <SergeyLukjanov> so, elmiko I'm writing you as initial assignee for this stuff, thank you!
18:45:53 <elmiko> SergeyLukjanov: ack
18:46:29 <SergeyLukjanov> #topic Open discussion
18:47:01 <elmiko> tosky and i have come across some weirdness with regards to the run_tests.sh script. this pertains to ci testing as well.
18:47:11 <tosky> yes
18:47:30 <SergeyLukjanov> honestly it's deprecated and CI isn't using it at all
18:47:36 <tosky> just running ./run_tests.sh -P -V --subunit , both icehouse and master branches, leads to
18:47:40 <elmiko> ok, that answers the question
18:47:42 <tosky> oh
18:47:50 <tosky> so what do you use to run the tests? tox directly?
18:47:51 <elmiko> SergeyLukjanov: is tox preferred?
18:47:57 <tosky> or testr directly?
18:48:03 <SergeyLukjanov> tox and testr
18:48:07 <tosky> (to easily extract test results)
18:48:13 <SergeyLukjanov> I've never used run_tests.sh :)
18:48:33 <tosky> is the exact line of the runner on the sahara-ci-config repository?
18:48:41 <tmckay> tools/with_venv.sh seems to be broken too.  Path is wrong for sahara, I think
18:48:46 <tosky> moreover: can run_tests.sh be removed if it's not used/deprecated?
18:48:50 <tmckay> nobody seems to notice
18:48:55 <aignatov> tmckay, alazarev : I’ve lookd at the current approach for preparing hdfs dirs with job binaries when they stored in swift
18:49:27 <SergeyLukjanov> tosky, it was added by mattf and I don't know anyone else who have been using it
18:49:28 <aignatov> I think we can refactor it by using distcp job to move them directly form swift to hdfs instead of copying to sahara then to VMs then to hdfs
18:49:47 <aignatov> I could start work on that
18:49:55 <SergeyLukjanov> aignatov, +1
18:49:56 <tmckay> aignatov, yes, I think we talked about this a long time ago. +!
18:49:56 <alazarev> aignatov, do we need them in hdfs at all?
18:49:59 <elmiko> aignatov: should work if you have the hadoop-swiftfs stuff in there
18:50:01 <tmckay> +1 even
18:50:20 <alazarev> jobs can read from swift directly
18:50:37 <alazarev> what is the purpose of distcp?
18:50:38 <tmckay> alazarev, hmmm, I think oozie wants hdfs
18:50:47 <aignatov> alazarev: I think yes, that’s how edp works via oozie
18:50:50 <tmckay> for the launcher, etc
18:50:54 <tosky> on the other side, many of the other projects have a run_tests.sh, but they are copy and copy
18:51:24 <alazarev> tmckay, hdfs and swift are the same level schemas, don't see difference
18:51:26 <aignatov> oozie expects libs dir for workflow.xml, not sure how it will work with swift
18:52:02 <tmckay> agree with aignatov, swift paths might break oozie, no idea
18:52:07 <alazarev> aignatov, yeap, workflow.xml need to be in swift too in this case
18:52:22 <SergeyLukjanov> tosky, IMO supporting run_tests.sh is bad because it's not used on CI
18:52:39 <tosky> SergeyLukjanov: ack; so, on CI, is it only a simple call for tox?
18:52:43 <tmckay> it might work.  Easy enough to play with from oozie command line
18:52:45 <aignatov> anyway, current approach seems very rough, it’s time to change that
18:52:46 <SergeyLukjanov> tosky, yup
18:52:55 <tosky> SergeyLukjanov: I'm digging into sahara-ci-config but I'm not sure which is the right call
18:53:26 <SergeyLukjanov> tosky, it's running tox -e integration
18:53:31 <alazarev> aignatov, +1
18:53:33 <SergeyLukjanov> tosky, to run tests itself
18:53:58 <SergeyLukjanov> tosky, and in openstack ci it runs tox -e py26/py27/pep8 etc
18:54:10 <tosky> SergeyLukjanov: isn't -e integration for integration tests? I was looking for unit tests
18:54:16 <alazarev> python HDFS driver could help too, but there is no good one :(
18:54:20 <tosky> SergeyLukjanov: oh, so simple tox -e py27 I guess then
18:54:30 <SergeyLukjanov> tosky, oh, unit tests are executed by the openstack ci
18:54:33 <aignatov> also oozie expects libraries put to classpath, currently they are stored in hdfs by default
18:54:45 <SergeyLukjanov> tosky, it's config located in openstack-infra/project-config
18:54:58 <tosky> SergeyLukjanov: thanks!
18:55:27 <alazarev> aignatov, yesp, but in htfs:// form, it can be easily changed to swift://
18:55:59 <tmckay> this is the longest meeting we've had in a while :)
18:56:11 <aignatov> maybe, need to investigate possibility of storing oozie stuff in swift
18:56:28 <aignatov> I’ll start work on this
18:56:53 <tmckay> yeah, I think we need to play with it outside of sahara, just use /opt/oozie/bin/oozie directly
18:56:58 <SergeyLukjanov> time for lunch
18:57:08 <aignatov> time for sleep :)
18:57:26 <tmckay> at least 4 time zones.  actually, at least 5
18:57:46 <tmckay> crazy
18:58:00 <SergeyLukjanov> yeah :)
18:58:05 <SergeyLukjanov> that's cool
18:58:08 <alazarev> tmckay, earth is round :)
18:58:09 <aignatov> 10:58PM for me
18:58:14 <SergeyLukjanov> and I'm really glad that our community is growing well
18:58:20 <SergeyLukjanov> 11:58am :0
18:58:22 <tmckay> 2:58
18:58:24 <elmiko> yea, very cool =)
18:58:52 <SergeyLukjanov> thank you folks! see you in #openstack-sahara
18:58:56 <SergeyLukjanov> #endmeeting