14:00:03 <tellesnobrega> #startmeeting sahara
14:00:04 <openstack> Meeting started Thu Apr  6 14:00:03 2017 UTC and is due to finish in 60 minutes.  The chair is tellesnobrega. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:00:05 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
14:00:08 <openstack> The meeting name has been set to 'sahara'
14:00:14 <esikachev> hi
14:00:20 <jeremyfreudberg> o/
14:00:31 <tosky> o/
14:00:42 <elmiko> ᶘ ᵒᴥᵒᶅ
14:00:55 <tellesnobrega> elmiko, that's new
14:01:04 <elmiko> gotta keep it fresh
14:01:12 <tellesnobrega> true
14:01:18 <shuyingya> o/
14:01:54 <tellesnobrega> #topic News/Updates
14:01:58 <vgridnev> o/
14:02:35 <vgridnev> nothing new from me, worked a little bit on job updates in project config
14:02:48 <tellesnobrega> I've been working on image gen for CDH, I'm almost done with that image, testing it but it fails with Cant Format NameNode, I'm working on it, hopefully we will have something by next week
14:02:51 <esikachev> i am working on sahara ci patches
14:03:12 <tellesnobrega> other than that I updated versions of spark and storm, so if you guys have time, please review those
14:03:15 <vgridnev> #link https://review.openstack.org/#/q/status:open+project:openstack-infra/project-config+branch:master+topic:sahara
14:03:23 <vgridnev> you are welcome to review those
14:03:26 <vgridnev> ^^
14:03:32 <shuyingya> I am thinking about how to refactor CDH plugin and going to update CDH 5.10
14:03:33 <tellesnobrega> vgridnev, will do
14:04:05 <shuyingya> https://blueprints.launchpad.net/sahara/+spec/refactor-cdh-plugin
14:04:11 <tellesnobrega> shuyingya, nice
14:04:44 <mariannelm> o/
14:04:56 <vgridnev> the general idea is to replace common methods into base classes
14:05:04 <vgridnev> nothing new, I guess
14:05:23 <tellesnobrega> yes, should be "simple"
14:05:34 <tellesnobrega> not fast or easy, but simple to do it
14:06:05 <shuyingya> OK, let's keep it simple.  will post few patch latter
14:06:10 <tellesnobrega> thanks
14:06:25 <shuyingya> thanks vgridnev tellesnobrega
14:06:42 <jeremyfreudberg> shuyingya, also good to take a look how this was done for vanilla (common utils in "hadoop2" folder, only the version-specific stuff separate)
14:07:14 <tellesnobrega> about sahara CI, the problem was resources, but I'm seeing a lot of failure on the patches, is that expected and is being worked on? or the errors are "random" failures
14:07:16 <tellesnobrega> ?
14:08:07 <shuyingya> yes,  I have thought about mkdir "cdh5" like "hadoop2". but looks like unecessary
14:08:47 <tosky> I'm just reviewing stuff for now, but preparing for future changes
14:09:22 <tellesnobrega> mariannelm, what have you been working on lately?
14:10:46 <tellesnobrega> about outreachy, there were two official applications to the Sahara project, our goal is to allow import and export of templates into sahara, it is something simple, but I would love to see that in, specially for moving around clouds, that makes things easier
14:11:10 <tellesnobrega> there will be more mention of it in the future if we do get someone
14:12:09 <tellesnobrega> do we have any more news or updates?
14:13:16 <tellesnobrega> lets move on them
14:13:20 <mariannelm> I didn't have much time this week, but I'm currently working on JSON changes of APIv2, and planning to fix some bugs this week
14:13:46 <tellesnobrega> mariannelm, nice, if you have some questions about api v2, stay tunned
14:13:53 <tellesnobrega> #topic APIv2
14:14:11 <tellesnobrega> I wanted to add this topic so we can bring our expert on the matter elmiko
14:14:45 <elmiko> o(^▽^)o
14:14:51 <shuyingya> welcome back ~  elmiko
14:14:56 <shuyingya> ^^
14:14:58 <elmiko> so happy to see some movement on this!
14:15:06 <tellesnobrega> as you've probably seen, I sent the email about microversions on sahara and I was talking with michael yesteday and asked him to come over and gives a little overview on that, and why this should be done and when so we are all on the same page on it
14:15:26 <tellesnobrega> elmiko, shuyingya and mariannelm are the ones doing the hard work on api v2
14:15:32 <elmiko> sweet
14:15:48 <elmiko> tellesnobrega: i did not see the email on the list, do you have a link to the archive?
14:15:59 <tellesnobrega> I'm kinda of organizing the effort, but they are the brains and hands on it
14:16:01 <tellesnobrega> yes, just a second
14:17:08 <elmiko> shuyingya mariannelm i am usually in the sahara channel but mostly quiet, if you run into issues or have questions feel free to ping me
14:17:14 <tosky> and we have a slot reserved into the API WG meeting in less than two hours
14:17:18 <tosky> (about microversioning)
14:17:52 <shuyingya> great elmiko. thanks
14:18:00 <mariannelm> elmiko, cool, thanks!
14:18:01 <tellesnobrega> elmiko, this is the title [Sahara][APIv2] Microversions looking ofr a link
14:18:23 <elmiko> tellesnobrega: ok, i'll search again. i have it on digest now
14:18:24 <jeremyfreudberg> http://lists.openstack.org/pipermail/openstack-dev/2017-April/115022.html
14:18:40 <elmiko> jeremyfreudberg++
14:18:48 <tellesnobrega> thanks jeremyfreudberg
14:19:12 <elmiko> for future reference, if you want to attract the api-wg, put [api] in the subject
14:19:22 <tellesnobrega> elmiko, will do
14:19:28 * elmiko reads
14:20:07 <elmiko> so, in answer to the question from the email
14:20:15 <elmiko> i would say prioritize the other work before microversions
14:20:54 <elmiko> you will need to have a solid foundation to build the microversion stuff on top of (ie making sure you can support older code paths), but so long as you are aware of that i think you can wait until the new api looks stable to add them
14:21:14 <tellesnobrega> that makes total sense
14:21:14 <elmiko> in the case of sahara it probably won't be that big a jump
14:21:33 <elmiko> especially since the v1 v1.1 and v2 apis all have different top level URIs
14:22:04 <elmiko> once v2 is rolling, you will need to be aware that any changes which bump the microversion _may_ have impact on the code paths
14:22:19 <elmiko> for example
14:22:49 <elmiko> returning a resource payload on a request for a version X.Y should ensure that it doesn't include payload information from X.Y+1
14:23:07 <elmiko> but that will largely depend on how the api grows after release
14:23:55 <elmiko> and of course, try to follow https://review.openstack.org/#/c/446138/3/guidelines/microversion_specification.rst
14:24:00 <elmiko> er oops, wrong link
14:24:18 <elmiko> #link http://specs.openstack.org/openstack/api-wg/guidelines/microversion_specification.html
14:24:24 <elmiko> that's the correct one
14:24:29 <tellesnobrega> thanks elmiko
14:24:44 <tellesnobrega> does anyone have any concerns on API microversions?
14:24:56 * tosky has
14:25:14 <tellesnobrega> I know of your concern tosky :)
14:25:21 <tosky> ok, apart from me then :)
14:25:53 <tellesnobrega> I was going to ask you to bring it up, just wanted to know if anyone else had anything to discuss on the topic
14:25:54 <tosky> just to summarize: I'd like to have a "sane" use of microversioning, avoiding incompatible changes in the API methods, even if microversioning allows for it
14:25:59 <tosky> yep
14:26:00 <elmiko> imo, tosky's concern can definitely be handled appropriately by the team
14:26:17 <elmiko> even if the governance doesn't strictly support that stance
14:26:32 <elmiko> i would document your usage heavily though if you intend to go down that road
14:26:41 <tosky> that's for sure, let's see
14:27:07 <elmiko> and honestly, i'm guessing that most folks would say that if you are going to introduce backward in-compat changes, you should really consider bumping major rev.
14:27:22 <elmiko> so DONT DO THAT!
14:27:28 <elmiko> =)
14:27:43 * elmiko mostly agrees with tosky vision on this
14:27:54 <tellesnobrega> tosky, awesome, whoever will work on this, will have to write a spec on it, and we will keep an eye to have that part documented in the spec and on sahara docs
14:28:13 <tosky> it seems a bit more of work, but it reduces the test matrix later
14:28:19 <tellesnobrega> that is the general assumption, if it breaks bump major version
14:28:22 <elmiko> tosky++
14:28:28 <tellesnobrega> tosky++
14:29:28 <tellesnobrega> do we anything else on APIv2?
14:29:45 <tellesnobrega> mariannelm, shuyingya any concerns or doubts you want to clear out while elmiko is around?
14:30:18 <mariannelm> small thing about json changes, that we were talking about on sahara channel...
14:30:32 <tellesnobrega> go ahead
14:30:44 <shuyingya> hi
14:30:59 <shuyingya> elmiko  what do you think about this https://review.openstack.org/#/c/448113/
14:31:11 * elmiko looks
14:32:08 <elmiko> i think that for the internal job binaries, the big trick about using the common job endpoints will be allowing the client to upload the data
14:32:33 <elmiko> that is the only sticking point i see, but in general my preference is to coalesce all jobs to a single group of endpoints
14:32:59 <elmiko> so, even if you needed to make /jobs/internal that would be preferable to having /jobs and /jobs-internal or similar
14:33:22 <tosky> wasn't the idea to kill the internal job binaries?
14:33:28 <elmiko> that was my preference
14:33:31 <mariannelm> about renaming hadoop_version -> plugin_version, I'm creating a new schema for the APIv2 with this change, and dealing with it manually in validationcoode, does it makes sense?
14:33:37 <elmiko> i think storing th binaries in the database is non-ideal
14:33:38 <mariannelm> I'm following: http://specs.openstack.org/openstack/sahara-specs/specs/backlog/api-v2-experimental-impl.html#data-model-impact
14:33:54 <mariannelm> *validation code
14:34:05 <shuyingya> But about the /jobs/internal, should we implement all of CRUD method
14:34:19 <shuyingya> I think it is unecessary
14:34:28 <tellesnobrega> shuyingya, I don't think we need jobs/internal at all
14:34:46 <elmiko> mariannelm: yes, that makes sense. a new schema is required
14:34:58 <shuyingya> oops, misunderstanding
14:35:05 <mariannelm> tellesnobrega++
14:35:05 <elmiko> shuyingya: i agree with tellesnobrega, if the team can drop internal jobs then so much the better
14:35:20 <shuyingya> yeah, make sense.
14:35:24 <tellesnobrega> we shouldn't store binaries on our internal db, we can use swift and bring stuff from there, or manila or whatever
14:35:28 <elmiko> imo, jobs should leverage cloud storage not sahara's internal database. that is just begging for problems
14:35:36 <elmiko> tellesnobrega++
14:35:44 <mariannelm> thanks elmiko
14:35:50 <shuyingya> got it.
14:35:54 <tellesnobrega> in our db we keep the link to the binary and that is all :)
14:35:56 <shuyingya> :)
14:36:02 <elmiko> yup
14:36:12 <tellesnobrega> as elmiko said, it is trouble, and we don't want any trouble
14:36:37 <elmiko> remember, if sahara starts managing job storage, then it is now in the storage business as well. there are plenty of cloud services which provide much better storage options than sahara can create.
14:36:50 <tellesnobrega> true
14:37:13 <elmiko> it sounds to me like you all have this covered nicely
14:37:25 <elmiko> kudos to the team
14:37:53 <tellesnobrega> elmiko, as I said to you, we are planning on finishing it in Pike and hopefully have it stable to release in Queen, but lets see how that goes
14:38:13 <elmiko> that will be awesome, good luck!
14:38:18 <tellesnobrega> thanks
14:38:31 <tellesnobrega> and thanks for dropping by and helping out clarify this
14:38:38 <tellesnobrega> lets move on
14:38:43 <elmiko> man, this started like back in icehouse...
14:38:54 <elmiko> no problem, thanks for inviting me =)
14:38:55 <mariannelm> an spec will be needed to deprecate job binary internals? How this will exactly be handled?
14:39:21 <tellesnobrega> mariannelm, yes, I think that is the best way to go
14:39:36 <elmiko> +1
14:40:37 <tellesnobrega> #link https://wiki.openstack.org/wiki/Sahara/api-v2
14:40:53 <tellesnobrega> just in case someone haven't seen it, this is the outline of apiv2 work
14:41:27 <tellesnobrega> we are trying to keep it up to date, and please look at the ones that are started and take some time to review it
14:42:26 <tellesnobrega> #topic OpenDiscussion
14:42:37 * elmiko sinks back into the shadows
14:42:47 <tellesnobrega> elmiko, thanks again for being here
14:42:53 <elmiko> =)
14:43:09 <tellesnobrega> jeremyfreudberg, did you take a look into vanilla 2.8 and if you will be able to work on that?
14:43:19 <tellesnobrega> I think you said you would like to
14:43:45 <jeremyfreudberg> I want to, but haven't had time yet. I can look into it this week, but I probably can't get the work done until after the summit
14:44:25 <tosky> vgridnev: just a reminder about https://review.openstack.org/#/c/448644/ - also because I'd like to backport it to Ocata
14:44:35 <tellesnobrega> ok, I don't think that it will be an issue, just take a look and see how complicated it will be and let us know if you want to take that
14:44:49 <jeremyfreudberg> tellesnobrega, sure
14:44:50 <vgridnev> waiting for jenkins to pass
14:45:08 <tellesnobrega> jeremyfreudberg, thanks
14:45:36 <tosky> vgridnev: right; the jobs mostly passed the build phase, but there was a failure in uploading the images, if I read the logs correctly from sahara-ci
14:45:42 <tosky> vgridnev: but let's see now :)
14:46:09 <vgridnev> no, I'm saying about just jenkins, which should be able to build all images
14:46:12 <tellesnobrega> jeremyfreudberg, about the Pig job issue, I did take a quick look on it, didn't see any reason why it should be integrated with oozie
14:47:16 <tellesnobrega> my opinion is that if you want you can move it out, but it is up to you
14:47:24 <tellesnobrega> I won't have the time to do it now
14:48:05 <jeremyfreudberg> I definitely want to move it out. I will publish a spec when I figure out how to do it (I have a few questions for the oozie team first)
14:48:21 <tellesnobrega> jeremyfreudberg, great
14:49:37 <jeremyfreudberg> that's it for me, gtg
14:49:44 <tellesnobrega> thanks
14:49:58 <tellesnobrega> anyone else has anything to discuss?
14:51:09 <tellesnobrega> SotK, please keep up with the reviews, we have quite a bit of patches to do so, and lets keep up the work
14:51:13 <tellesnobrega> thanks all
14:51:35 <tellesnobrega> you all have 9 minutes back
14:51:39 <tosky> \o/
14:51:43 <tellesnobrega> #endmeeting