14:00:15 <tellesnobrega> #startmeeting sahara
14:00:16 <openstack> Meeting started Thu Jan 11 14:00:15 2018 UTC and is due to finish in 60 minutes.  The chair is tellesnobrega. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:00:17 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
14:00:19 <openstack> The meeting name has been set to 'sahara'
14:00:24 <tosky> o/
14:00:56 <jeremyfreudberg> o/
14:01:52 <shuyingy_> o/
14:01:57 <tellesnobrega> we all are here
14:01:58 <shuyingy_> Hi guys
14:02:03 <tellesnobrega> that is new :)
14:02:05 <tellesnobrega> hey all
14:02:11 <tellesnobrega> lets get started
14:02:16 <tellesnobrega> #topic News/Updates
14:03:25 <tellesnobrega> I'm doing small fixes on the images, not much left to do there. Now I'm focusing on fixing some of the bugs we talked about in Denver (file transfer is the first). Also I want to start testing S3 integration as well as APIv2
14:04:24 <tosky> still checking the images produced by sahara-image-pack; finally fixed the CLI tests (the patch for tempest just landed); starting to tackle sahara-extra and upload of oozie tarballs
14:04:26 <shuyingy_> Business travel takes me half of the week. Get back yesterday and start working on solve problem in vanilla upgrade
14:04:53 <tosky> (and nagging and complaining for mass-doc changes, but that touched sahara only partially)
14:04:53 <jeremyfreudberg> i haven't had much time to focus on upstream.... but i have (finally) fixing s3_hadoop on my mind. and also apiv2 in client
14:05:46 <jeremyfreudberg> re apiv2 in client, i think i will post a patch doing version discovery the wrong way, then have monty fix it (The patch isn't just that, it also will have all the other v1->v2 changes reflected)
14:06:12 <tellesnobrega> jeremyfreudberg, we need to start testing APIv2 asap, when are you sending the patch?
14:06:25 <tellesnobrega> a rough estimation is ok
14:07:10 <jeremyfreudberg> late tonight
14:07:13 <jeremyfreudberg> hopefully
14:07:17 <tellesnobrega> perfect
14:07:18 <tellesnobrega> thanks
14:07:29 <jeremyfreudberg> np
14:07:38 <tosky> wrong patch forcing monty to fix it, I like the strategy :)
14:08:00 <jeremyfreudberg> yes, my version of cunningham's law
14:08:09 <tellesnobrega> what could go wrong using that?
14:09:18 <tellesnobrega> more updates?
14:10:06 <jeremyfreudberg> well, about force deletre
14:10:08 <jeremyfreudberg> delete
14:10:40 <jeremyfreudberg> i have a patch that you guys can review BUT actually i am thinking about it some more and i think stack abandon might be not the best choice
14:11:04 <tellesnobrega> what is the choice you are thinking of?
14:11:38 <jeremyfreudberg> just do a stack delete, but don't wait for deletion. then, unhide the stack
14:12:10 <tosky> let's discuss the details now or let's switch topic?
14:12:24 <jeremyfreudberg> tosky, yes, probably switch topic if no more updates
14:12:29 <jeremyfreudberg> jumped the gun a bit
14:12:33 <tellesnobrega> let me switch topic so we can discuss it
14:12:44 <tellesnobrega> #topic Force Delete details
14:12:53 <tellesnobrega> you have the floor jeremyfreudberg
14:13:39 <jeremyfreudberg> current version of the patch does force delete like this: 1) put stack in deleting state 2) immediately abandon it 3) drop from db and other cleanup
14:13:44 <jeremyfreudberg> drop cluster, i mean
14:14:16 <jeremyfreudberg> my new idea is: 1) put stack in deleting state, don't wait for deletion to complete 2) unhide stack 3) drop cluster from db
14:14:54 <jeremyfreudberg> the problem with abandon was that the user has no idea what was orphaned
14:15:15 <tellesnobrega> the step that I don't understand, probably due to lack of knowledge on heat, but what does unhide stack means
14:15:28 <shuyingy_> could you explain about the step 2
14:15:38 <jeremyfreudberg> yes, unhide stack is actually a very funny thing
14:15:42 <jeremyfreudberg> i will explain
14:16:10 <jeremyfreudberg> sahara actually has a special arrangement with heat, that all stacks with a tag data-processing-cluster are "hidden"
14:16:38 <jeremyfreudberg> so you need to specifically query for hidden stacks to see them in api/cli
14:17:02 <shuyingy_> yes, heat stack-list -a
14:17:07 <jeremyfreudberg> but if i remove the tag from the stack, suddenly the user can see the stack and will know to clean stuff up
14:17:44 <tellesnobrega> by user you mean cloud operator?
14:18:04 <jeremyfreudberg> by user i mean the real user
14:18:32 <tellesnobrega> ok
14:18:39 <tellesnobrega> that makes sense to me
14:18:42 <tellesnobrega> shuyingy_, tosky ?
14:19:37 <shuyingy_> I don't understand what can we benefit if we see the stack
14:19:59 <tosky> and the "drop cluster from db" solves the issue when heat stack is clean, but the cluster is stuck in deletion state from the sahara point of view?
14:21:15 <jeremyfreudberg> tosky, my force delete solution fixes (by my estimate) 90%+ of cases where cluster is stuck deleting because heat stack won't finish deleting
14:21:41 <jeremyfreudberg> by providing alternate way of getting to the final "drop from db" step that doesn't wait for heat stack delete to finish
14:22:21 <jeremyfreudberg> your case where sahara is still in deleting but stack is already gone is not fixed by my patch, but if the stack is already gone and you issue the delete call again the delete should finish fine
14:22:41 <shuyingy_> jeremyfreudberg, I have tried to delete the stack in "heat stack-list -a", but most of time sahara can't delete because stack status is failed
14:23:13 <shuyingy_> I see
14:23:33 <tellesnobrega> after the stack is completely gone that should be ok shuyingya
14:23:48 <jeremyfreudberg> basically here's my point:
14:23:56 <shuyingya> I got it.
14:24:10 * jeremyfreudberg will summarize anyway
14:24:17 <tellesnobrega> please do jeremyfreudberg
14:25:28 <jeremyfreudberg> the only way we can have a meaningful force delete option here is if we don't await stack deletion and proceed on to drop cluster from db regardless of stack status
14:25:31 <jeremyfreudberg> the implications of that:
14:25:45 <jeremyfreudberg> - user might still want to know what resources didn't end up deleted
14:26:11 <jeremyfreudberg> - but, if the stack really was undeleteable (DELETE_FAILED) then that still doesn't mean that the user will have any easier of a time
14:26:31 <jeremyfreudberg> i mean the user's manual cleanup still won't be easy
14:27:50 <jeremyfreudberg> i think we all get the idea, i don't want to waste half the meeting on it
14:28:01 <jeremyfreudberg> please follow up on the patch itself if you have concerns
14:28:07 <shuyingya> jeremyfreudberg, If I remember correctly, heat has an option to configure what kind of stack should be hidden. how can you unhiden it?
14:28:26 <jeremyfreudberg> you can do a PATCH on stack to update tags list
14:28:33 <jeremyfreudberg> i was planning just to remove the tag
14:28:44 <jeremyfreudberg> the conf option stays the same
14:29:12 <shuyingya> thanks
14:29:20 <shuyingya> totally understand it
14:29:33 <jeremyfreudberg> np
14:30:05 <tellesnobrega> are we all settled on this topic?
14:30:20 <tosky> yep
14:30:24 <jeremyfreudberg> yep
14:30:24 <shuyingya> nope
14:30:50 <jeremyfreudberg> shuyingya, continue
14:31:04 <shuyingya> jeremyfreudberg, why stack-abandon is not a proper way?
14:33:46 <jeremyfreudberg> it would work okay, but it's unnecessary. it basically means "drop stack from heat db" but why should we be hiding stuff from user, i guess is my point
14:33:54 <shuyingya_> sorry, network broken
14:34:51 <tellesnobrega> I guess the point boils down to, does the end user need to know about heat stack status when using sahara?
14:35:40 <jeremyfreudberg> tellesnobrega, basically yes
14:37:16 <jeremyfreudberg> i will continue to think about it, but i really think we should move on so we can use meeting time more efficiently
14:37:27 <shuyingya_> I need take some time to think about it
14:37:35 <shuyingya_> let's move on
14:37:43 <tellesnobrega> we can continue on the patch
14:37:59 <tellesnobrega> #topic Dublin PTG
14:38:09 <shuyingya_> did you receive the email form me?
14:38:20 <tellesnobrega> I started an etherpad so we can start putting stuff together for the PTG
14:38:31 <jeremyfreudberg> shuyingya_, yes, saw email
14:38:39 <tellesnobrega> #link https://etherpad.openstack.org/p/sahara-rocky-ptg
14:38:57 <shuyingya_> let's continue tellesnobrega's topic
14:39:00 <shuyingya_> :)
14:39:12 <tosky> yep, we can talk about oozie afterwards, I have few questions too
14:39:12 <tellesnobrega> please take some time to write down topic ideas, so we can start getting ready for the PTG
14:39:29 <shuyingya_> will do
14:39:52 <tellesnobrega> also, I know tosky and I will be there, jeremyfreudberg almost sure, how about you shuyingya_ ?
14:39:55 <tosky> sure (I was thinking about copy-pasting the final TODO list from Queens PTG for easier comparison)
14:40:27 <tellesnobrega> tosky, we can certainly do that
14:40:28 <shuyingya_> I am not sure right now. maybe not
14:40:47 <tellesnobrega> if you can apply to TSP please do
14:41:02 <tellesnobrega> you are a core reviewer, they will probably see you as high priority
14:41:41 <tellesnobrega> I was also thinking, what do you guys think of creating a trello board so we can have a better tracking system of sahara todos, feature status and so on
14:41:42 <tellesnobrega> ?
14:41:56 <tellesnobrega> wait, before you answer that question
14:42:06 <tellesnobrega> I will close this topic and change to Open Discussion
14:42:18 <tellesnobrega> and after this we can jump into oozie and openstack-hadoop stuff
14:42:25 <tellesnobrega> #topic Open Discussion
14:42:33 <tellesnobrega> what do you guys think of creating a trello board so we can have a better tracking system of sahara todos, feature status and so on?
14:43:41 <jeremyfreudberg> trello is a decent idea (and actually i am already doing that)
14:44:05 <shuyingya_> this tool looks like cool
14:44:30 <tellesnobrega> tosky, shuyingya_, any objections? I know some people hate it
14:44:49 <shuyingya_> I am OK with it.
14:45:17 <tosky> why not trying storyboard then?
14:45:17 <jeremyfreudberg> the one objection that someone could have is why not storyboard maybe
14:45:25 * tosky <- someone
14:45:31 <jeremyfreudberg> yes, beat tosky by one second
14:45:39 <tellesnobrega> could be, I never used storyboard
14:46:10 <tellesnobrega> by trello I meant some place we can have a better tracking system than keep going back to an 3-6 months old etherpad
14:46:21 <shuyingya_> I never used storyboard and trello :(
14:46:34 <tosky> in fact "migrate to storyboard" is something I'd like to propose for Rocky - it's not going to be a community goal, but still we will use it at some point
14:47:12 <jeremyfreudberg> tosky, is it possible to do a "soft" transition, meaning keep bugs on LP for now and just to do Task tracking on Storyboard?
14:47:38 <tosky> jeremyfreudberg: I don't know, it's something to ask to... uhm... I don't know :)
14:47:51 <tellesnobrega> TC maybe?
14:47:59 <tosky> probably (or they can redirect)
14:48:09 <tellesnobrega> tosky, do you have a link to storyboard?
14:48:37 <tellesnobrega> and I will follow up with that
14:48:53 <tellesnobrega> since we mainly agree that we need something better than what we have today
14:49:10 <jeremyfreudberg> yes, as long as we have some tracking tool then i will be happy
14:49:29 <jeremyfreudberg> let's use the last ten minutes for shuyingya_ oozie stuff
14:49:37 <tellesnobrega> yeah,
14:49:40 <tosky> http://storyboard.openstack.org/
14:49:49 <tellesnobrega> thanks
14:50:53 <shuyingya_> do you have any questions about the description in the email?
14:51:07 <tosky> tellesnobrega: topic!
14:51:17 <tosky> oh, open discussion already, sorry
14:51:46 <tellesnobrega> yes
14:51:47 <jeremyfreudberg> shuyingya_, basically you are saying that running the job normally against swift is working, but when you use oozie to launch job, something is missing
14:51:55 <shuyingya_> good news is that all testcases has passed.
14:52:40 <shuyingya_> jeremyfreudberg, do you mean you want know how to start the job?
14:53:03 <shuyingya_> the command is :openstack dataprocessing job execute --cluster gogo --job-template vanilla-job-template --configs mapred.map.class:org.apache.oozie.example.SampleMapper mapred.reduce.class:org.apache.oozie.example.SampleReducer mapreduce.framework.name:yarn fs.swift.service.sahara.username:admin fs.swift.service.sahara.password:admin --input 92e6b254-66a0-4b05-a989-40993bb5d9b7 --output 0cced149-5400-4afa-9d49-32295d74065d
14:53:24 <tosky> one thing that I noticed and I don't understand is: why the generated jar is always hadoop-openstack-3.0.0-SNAPSHOT.jar , even if we specify different values of hadoop.version?
14:53:46 <jeremyfreudberg> tosky, i think that naming is given by something in pom, but it doesn't mean anything
14:53:49 <jeremyfreudberg> hardcoded somewhere
14:53:50 <tosky> what is the difference then?
14:54:23 <jeremyfreudberg> shuyingya_, what i meant was just to clarify the meaning of the email. there is the oozie way (sahara edp) and the normal way (hadoop command?)
14:54:55 <jeremyfreudberg> tosky, https://github.com/openstack/sahara-extra/blob/master/hadoop-swiftfs/pom.xml#L22
14:54:59 <shuyingya_> I run it with sahara comand
14:55:28 <tellesnobrega> jeremyfreudberg, for us, the normal way  is using oozie (sahara way)
14:55:36 <jeremyfreudberg> shuyingya_, i see. my mistake. i was imagining things
14:55:39 <shuyingya_> yes, hadoop-openstack-3.0.0-SNAPSHOT.jar doesn't mean anything
14:56:34 <jeremyfreudberg> shuyingya_, i'm wondering if you can try the job with `hadoop jar ...` maybe the oozie classpath is different than normal hadoop classpath
14:57:40 <tellesnobrega> what changed so httpclient is not present anymore? and what is the best way to put it back on?
14:57:56 <jeremyfreudberg> shuyingya_, we can also try to get this change in: https://github.com/apache/hadoop/commit/62579b69a0a294ba1ea14cf76c650b640f89f331#diff-7debb0b87e1651538db48ac873db6a9c
14:58:17 <jeremyfreudberg> apache people just stopped importing httpclient completely
14:58:26 <jeremyfreudberg> and rewrote to use a different library
14:59:15 <tellesnobrega> from our side? what need httpclient? can we change it to use httpresponse?
14:59:23 <shuyingya_> jeremyfreudberg, I know what you mean and I also thought about to apply the change in 3.0.0. have no time to test
14:59:30 <jeremyfreudberg> https://github.com/openstack/sahara/blob/b3ff28609a916362fe113b2f9fb9b48db33aac09/sahara/plugins/vanilla/hadoop2/run_scripts.py#L179
14:59:40 <shuyingya_> :) I will try it out tomorrow
15:00:05 <jeremyfreudberg> if you say uploading the jar fixes it, we can just do that too
15:00:30 <tellesnobrega> quick first fix until we come up with something better
15:00:35 <tellesnobrega> we are at the top of the hour
15:00:41 <tellesnobrega> we have to close it now
15:00:50 <shuyingya_> yes
15:00:57 <tellesnobrega> thanks guys, shuyingya_ update us on progress on this and what do you think will be the best approach for now
15:01:08 <tellesnobrega> jeremyfreudberg, if you need help with APIv2 let me know
15:01:14 <shuyingya_> If you guys OK, I will post the patch with work around method tomorrow
15:01:15 <tellesnobrega> tosky, do your thing :)
15:01:25 <tellesnobrega> sounds good shuyingya_
15:01:29 <tosky> O.o
15:01:29 <tellesnobrega> thanks all
15:01:31 <tosky> o/
15:01:40 <tellesnobrega> #endmeeting