14:00:48 <tellesnobrega> #startmeeting sahara 14:00:49 <openstack> Meeting started Thu Nov 30 14:00:48 2017 UTC and is due to finish in 60 minutes. The chair is tellesnobrega. Information about MeetBot at http://wiki.debian.org/MeetBot. 14:00:50 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 14:00:52 <openstack> The meeting name has been set to 'sahara' 14:01:19 <tellesnobrega> waiting for people to show hands 14:01:20 <shuyingya> o/ 14:01:42 <shuyingya> Wait me 3 minutes please 14:01:52 <tellesnobrega> sure 14:02:01 <jeremyfreudberg> o/ 14:04:52 <tellesnobrega> waiting until 5 after so we can start 14:05:07 <jeremyfreudberg> sure 14:05:15 <tellesnobrega> and that is time 14:05:22 <tellesnobrega> #topic News/Updates 14:05:24 <shuyingy_> Hi I am comming 14:05:39 <tosky> here, sorry 14:05:52 <jeremyfreudberg> o/ 14:06:22 <tellesnobrega> I'm currently finishing up the work on decommission of specific node. I just need to check shuyingy_'s comment on the order of deletion, and write tests 14:06:39 <tellesnobrega> other than that, I need to get back to mapr image generation 14:06:53 <tellesnobrega> and try to squeeze some time to work on plugin outside sahara code 14:07:03 <tellesnobrega> not sure it will happen, the last one, but I will try 14:07:41 <tellesnobrega> Oh, I also need to do the decommission work on the sahara client and UI 14:07:56 <tellesnobrega> UI should be a pain, but it is a necessary evil 14:09:08 <jeremyfreudberg> so, I'm working on revising s3_hadoop element, finishing up s3 job binary mostly 14:09:37 <tellesnobrega> good, do you believe we will have the feature in queens? 14:09:41 <shuyingy_> Today is the last day of business travel. this week mostly working on deploying sahara on k8s by helm. If you are interesting, please review https://review.openstack.org/#/c/477718/. and will continue to work on HBase support next week. hopefully it can be finished next week. 14:09:53 <jeremyfreudberg> tellesnobrega, s3 job binary should defniitely be done 14:10:21 <jeremyfreudberg> data source is another story, those are always a bit trickier since they have to work on the cluster itself, it's not just an abstraction 14:10:22 <tellesnobrega> that is great shuyingy_, did it all work well: 14:10:23 <tellesnobrega> ? 14:10:50 <tellesnobrega> jeremyfreudberg, I understand, we still have some time, let me know if you need help 14:10:58 <jeremyfreudberg> tellesnobrega, sure 14:11:35 <shuyingy_> I am try to review s3 patch. but doesn't find any problem because of lack of knowledge about s3 14:12:12 <jeremyfreudberg> yes, it would be good have someone who really knows S3 to say whether or not we are exposing enough boto options 14:12:20 <jeremyfreudberg> i made some nodes about s3 region and such 14:12:25 <jeremyfreudberg> there are lots of fancy kwargs that can go there 14:12:31 <jeremyfreudberg> but who knows what users actually need 14:12:36 <jeremyfreudberg> *notes 14:12:47 <tellesnobrega> that might be tricky to find 14:13:03 <tellesnobrega> s3 knowledge interested in OpenStack, even more on sahara 14:13:04 <shuyingy_> yes, it works well. I need try to find more frequently changed configurations for user to configure sahara. 14:14:14 <tellesnobrega> tosky, any news/updates? 14:14:33 <tosky> I'm swamped in internal testing, but I'm still keeping an eye on the zuul changes (and nagging zuul developers) 14:14:52 <tellesnobrega> cool 14:15:01 <tosky> so yes, we will have proper native jobs way before queens (hopefully before the end of the year) 14:15:13 <tellesnobrega> nice :) that is great to hear 14:16:07 <tosky> I think I already wrote it: I could have pushed some changes, and then fixed them; but infra is not pushing to remove the old jobs, so I'm lazily waiting to avoid some loops of write/fix 14:16:27 <tellesnobrega> makes sense 14:16:31 <jeremyfreudberg> tosky, i see 14:16:32 <shuyingy_> now is in middle of Queen, right? I am sorry that i didn't finished HBase support and keypair injecting util now 14:16:52 <jeremyfreudberg> shuyingy_, that's ok 14:16:58 <jeremyfreudberg> i have the same schedule problems 14:17:02 <tellesnobrega> shuyingy_, yes, we are approaching q-2. It is on December 7th 14:17:33 <jeremyfreudberg> feature freeze is between Jan 22 - Jan 26, so you still have time after that 14:17:39 <tosky> up to the PTL, but as long as the new features are additional and do not impact the existing code, it should not be a problem if they land a bit later 14:17:39 <tellesnobrega> yes 14:18:00 <tellesnobrega> and we can always get FFE if needed 14:18:11 * jeremyfreudberg reminds everyone he has to log off today by 14:27 UTC 14:18:18 <shuyingy_> that is great 14:18:20 <tellesnobrega> it won't be a problem I believe. But we can't wait too much 14:19:36 <tellesnobrega> lets move on, we have a important topic for today 14:19:44 <jeremyfreudberg> every topic is important 14:19:47 <jeremyfreudberg> ;) 14:19:51 <shuyingy_> :) 14:19:51 <tellesnobrega> true 14:19:54 <shuyingy_> aha 14:19:57 <tellesnobrega> critical one 14:20:07 <tellesnobrega> #topic plugins update 14:20:26 <jeremyfreudberg> indeed, this is is a critical topic 14:20:35 <tellesnobrega> At the PTG we decided that we should try to have all updates by Q2 14:20:46 <tellesnobrega> that means we have about 7 days to finish them all 14:21:02 <tosky> or defines exceptions :) 14:21:09 <tellesnobrega> this isn't a must, and if we fail, nobody will die. 14:21:35 <tellesnobrega> My plan here is for us to have a list of the plugins that needs to be updated and we can start working on it asap 14:21:42 <jeremyfreudberg> yes, nobody will die, and i put a provision here https://releases.openstack.org/queens/schedule.html#q-sahara-plugin-deadline that telles can basically suspend this rule if he wants 14:21:59 <tosky> and an estimated rough complexity, if you can 14:21:59 <jeremyfreudberg> yes, let's figure out what can get updated 14:22:04 <jeremyfreudberg> yes 14:22:08 <tellesnobrega> so we can have it finished close to our schedule 14:22:33 <jeremyfreudberg> so, there is a finally as of a few weeks ago a stable Hadoop 2.8.x release 14:22:37 <jeremyfreudberg> it ended up 2.8.2 14:22:45 <jeremyfreudberg> there is no stable 2.9 or 3.0 yet 14:22:51 <tellesnobrega> yes 14:23:42 <jeremyfreudberg> so vanilla plugin can be upgraded if we want 14:23:51 <jeremyfreudberg> along with pig and hive and oozie(?) on that image 14:24:06 <tosky> and will 2.9 be close 2.8, before the big breakage with 3.0? 14:24:16 <tellesnobrega> sounds about right 14:24:28 <tosky> I wonder why they still plan full development on the old series, but well... 14:24:41 <jeremyfreudberg> tosky, 2.9 is close to 2.8 yes 14:24:46 <jeremyfreudberg> and there will be parallel development 14:24:58 <jeremyfreudberg> not quite like Python 2 / Python 3, but something like that :( 14:25:15 <jeremyfreudberg> so, vanilla 14:25:24 <jeremyfreudberg> spark plugin can can be upgraded too 14:25:31 <shuyingy_> cdh too 14:25:42 <shuyingy_> it has cdh 5.13 right now 14:25:44 <jeremyfreudberg> cdh maybe not, i had some problem with a package no longer available 14:25:58 <jeremyfreudberg> we had to cancel our 5.12 upgrade because we didn't have time to figure it out 14:26:09 <tellesnobrega> #action update vanilla to 2.8.2 also pig, hive and oozie(?) 14:26:27 <jeremyfreudberg> for spark there is 2.1.2 and 2.2.0 so far 14:26:52 <jeremyfreudberg> the cdh thing was navigator trustee if i recall correctly 14:26:55 <tellesnobrega> shuyingy_, jeremyfreudberg is right, we have to take a deeper look into CDH, they changed some packages to enterprise only, so we may have hit a roadblock there 14:27:07 <jeremyfreudberg> btw, i have to sign off basically now 14:27:13 <jeremyfreudberg> but tellesnobrega can figure out storm 14:27:17 <tellesnobrega> thanks jeremyfreudberg 14:27:20 <tosky> bye jeremyfreudberg o/ 14:27:29 <jeremyfreudberg> mapr and hdp i'm sure have something new 14:27:30 <shuyingy_> jeremy byebye 14:27:35 <jeremyfreudberg> not sure if that's absolutely essential 14:27:37 <jeremyfreudberg> bye all! 14:27:42 <jeremyfreudberg> send me an email if you have urgent questions 14:27:44 <tellesnobrega> see ya 14:27:48 <tellesnobrega> will do 14:27:50 <jeremyfreudberg> i will read the trasncript of ths meeting asap 14:28:06 <tellesnobrega> do we want both new spark versions? 14:28:36 <shuyingy_> I don't think so. 14:28:39 <tosky> uhm, do you mean basic spark and cdh/spark? 14:28:41 * tosky confused 14:28:46 <tellesnobrega> I think spark 2.1.0 should not be deprecated yet, we can add the newest one 14:28:51 <tellesnobrega> upstream spark 14:28:55 <tosky> oh, I see 14:29:12 <tellesnobrega> and later, if needed we replace 2.1.0 with 2.1.2 14:29:24 <tosky> isn't 2.1.2 just a minor update for 2.1? I think we didn't solve the problem of minor compatibile versions 14:29:40 <shuyingy_> we don't need update minor version I think 14:29:43 <tosky> maybe we could just allow the code to define multiple tags for each plugin/version 14:29:54 <tosky> so can we register it as 2.1.0 and it should work? 14:30:05 <tellesnobrega> that would be nice 14:30:56 <tellesnobrega> to be sure, spark is going to 2.2.0 14:31:22 <tellesnobrega> and we keep 2.1.0 and see if we can work tags to have minor versions 14:31:32 <tellesnobrega> all agreed? 14:32:14 <shuyingy_> do you mean we can build an image with spark 2.1.2 and tag it as 2.1.0? 14:32:30 <tosky> I would say that it should be fine for n ow 14:32:32 <tosky> now 14:32:47 <tellesnobrega> shuyingy_, yes 14:32:51 <shuyingy_> from code side, that's is not difficult. sound nice 14:33:01 <shuyingy_> \o/ 14:33:12 <tellesnobrega> #action update spark to 2.2.0 14:33:20 <tellesnobrega> lets move on to storm 14:33:27 <tellesnobrega> storm newest version is 1.1.1 14:33:43 <tellesnobrega> we have 1.1.0, so we can take the same approach we did for spark 14:34:24 <tellesnobrega> sounds ok? 14:35:14 <shuyingy_> sure. it is really convenient to user 14:35:41 <tellesnobrega> cool 14:35:46 <tellesnobrega> MapR 14:35:57 <tellesnobrega> I check their website, seems like the newest version is 5.2 14:36:00 <tellesnobrega> which we already have 14:36:30 <tosky> it would be nice to understand why they disappeared also from sahara 14:36:37 <tosky> but no new upstream versions too, uhm 14:37:19 <tellesnobrega> wait, seems like there is a mapr 6 14:38:21 <tellesnobrega> but in tr mapr the newest version is 5.2 14:40:00 <tellesnobrega> yeah, seems like 5.2 is the last one on downloads 14:40:19 <tellesnobrega> Ambari has a new version 2.6 14:41:21 <tellesnobrega> https://cwiki.apache.org/confluence/display/AMBARI/Installation+Guide+for+Ambari+2.6.0 14:41:22 <tosky> I can't find the release date for 6, but there is the documentaiton 14:41:35 <shuyingy_> we may not have too much energy to update them all. MapR used to update by MapR themself, right? 14:41:43 <tosky> and which HDP versions can be deployed by ambari 2.6? 14:41:56 <tellesnobrega> tosky, checking that now 14:42:00 <shuyingy_> https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.0/bk_support-matrices/content/ch_matrices-ambari.html#ambari_stack 14:42:15 <tellesnobrega> 6.0.0 November 21, 2017 N/A 14:42:19 <tellesnobrega> tosky, ^ 14:42:44 <tosky> tellesnobrega: argh, one week 14:42:51 <tosky> I would say we can push it to Rocky 14:42:57 <tellesnobrega> tosky, for sure 14:43:17 <tellesnobrega> seems like hdp doesn't support ambari 2.6 yet 14:43:25 <tosky> that would be weird 14:43:46 <tosky> I remember that Ambari x.y should support at least HDP x.y 14:43:54 <tosky> but I may remember incorrectly 14:44:15 <tellesnobrega> the link is looking was wrong 14:45:29 <tellesnobrega> on HDP website there is only the matrix up to ambari 2.5 14:46:05 <tellesnobrega> https://hortonworks.com/blog/hdp-2-6-3-dataplane-service/ 14:46:14 <tellesnobrega> tosky, shuyingy_ ^ 14:46:27 <tosky> uhm uhm 14:46:47 <tellesnobrega> both ambari 2.6.0 and HDP 2.6.3 14:46:50 <tosky> at least we are not the only one with documentation that requires some updates :) 14:46:58 <tellesnobrega> lol 14:47:01 <tellesnobrega> true 14:47:13 <shuyingy_> haha 14:47:16 <shuyingy_> :) 14:47:18 <tellesnobrega> so we want to update ambari to 2.6.0 with HDP to 2.6.3 14:47:22 <tellesnobrega> all in favor? 14:48:33 <shuyingy_> up 14:48:36 <shuyingy_> agree 14:49:17 <tellesnobrega> #action update ambari to 2.6 with HDP to 2.6.3 14:49:41 <shuyingy_> Now we should decide how to divide the work 14:49:45 <tellesnobrega> I think we covered all plugins 14:49:50 <tellesnobrega> yes 14:49:52 <shuyingy_> yes 14:50:10 <tellesnobrega> I believe that spark and storm should be fairly quick work, I can take them both 14:50:33 <tosky> only HDP 2.6, or should we try to switch to ambari 2.6 to deploy also the older versions? 14:50:36 <shuyingy_> btw, I will share you the performance comparison of vanilla to aws emr next week 14:50:51 <tellesnobrega> cool 14:50:52 <tosky> it may be complicated, but it could also simplify part of the matrix 14:51:17 <tellesnobrega> tosky, not sure, ambari/hdp has always confused me 14:51:28 <shuyingy_> I will update vanilla if I can. otherwise I can invite my colleague to help us. 14:51:31 <tosky> Ambari can deploy few versions 14:51:50 <shuyingy_> ambari/hdp has always confused me too 14:52:06 <shuyingy_> the image name of ambari let me crazy 14:53:02 <tosky> yeah 14:53:29 <tellesnobrega> tosky, what do you think is the best way to go with ambari? 14:53:32 <tosky> from the previous document it seems that Ambari 2.5 was able to deploy all the old versions 14:53:44 <tosky> so it may be the same with Ambari 2.6 14:54:16 <tosky> right now, if I'm not mistaken, we deploy two slightly different versions of ambari for the two version of hdp 14:54:43 <tosky> but if ambari 2.6 can deploy 2.6, 2.4 (and 2.3, if we don't want to deprecated it), maybe it's easier to handle 14:55:03 <tosky> otherwise we would need different codepaths on our side if the various ambari versions differs 14:55:15 <tosky> but if ambari hides the differences between HDP versions, well 14:55:18 <tellesnobrega> yeah, I think we have 2.2.0.0 and 2.2.1.0 14:55:52 <tellesnobrega> our ambari version is very old at this point 14:57:35 <tellesnobrega> so we have ambari and vanilla to update 14:57:36 <tosky> I think we support ambari 2.4 too 14:57:44 <tellesnobrega> yeah 14:58:03 <tellesnobrega> not sure 14:58:04 <shuyingy_> 2.4 and 2.5 both support 14:58:27 <tellesnobrega> on image gen the versions are 2.2.0.0 and 2.2.1.0 14:58:39 <tosky> but again, if HDP 2.6 can be deployed with ambari 2.6 only (ok, also 2.5), we need to update to support that ambari 14:58:59 <tosky> if, on the other side, that ambari can deploy all hdp versions, why not kill the olders ambari? 14:59:07 <tellesnobrega> sounds good, lets take a closer look on that and get back to it 14:59:18 <tellesnobrega> that would be good too 14:59:34 <tellesnobrega> and maybe we should take a class on ambari/hdp 14:59:36 <tellesnobrega> lol 15:00:14 <tellesnobrega> shuyingy_, do you have time to update any plugin? 15:00:17 <shuyingy_> so I will update vanilla 15:00:31 <shuyingy_> you may miss my message 15:00:37 <shuyingy_> :) 15:00:42 <tellesnobrega> great 15:00:54 <tellesnobrega> I will try to check ambari and see if jeremy can work on it 15:00:58 <shuyingy_> at 22:51, I said it :) 15:01:02 <tellesnobrega> otherwise I will work on it was well 15:01:04 <tellesnobrega> yes 15:01:07 <shuyingy_> haha 15:01:08 <tellesnobrega> we have to close it 15:01:11 <tellesnobrega> thanks guys 15:01:15 <shuyingy_> bye 15:01:17 <tellesnobrega> see you next week 15:01:20 <tellesnobrega> bye 15:01:22 <shuyingy_> see you next week 15:01:28 <tosky> o/ 15:01:39 <tellesnobrega> #endmeeting