14:00:02 <tellesnobrega> #startmeeting sahara
14:00:03 <openstack> Meeting started Thu Nov 22 14:00:02 2018 UTC and is due to finish in 60 minutes.  The chair is tellesnobrega. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:00:04 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
14:00:06 <openstack> The meeting name has been set to 'sahara'
14:00:16 <tosky> o/
14:01:51 <tellesnobrega> maybe jeremy is not joining today
14:01:57 <tellesnobrega> lets give a couple more minutes
14:02:09 <tosky> Thanksgiving!
14:02:15 <fungi> if you just need _a_ jeremy i'm around ;)
14:02:38 <tellesnobrega> hi fungi
14:02:45 <tellesnobrega> true tosky, forgot that
14:02:49 <fungi> but i suspect i'm a poor stand-in ;)
14:03:22 <tellesnobrega> lets start
14:03:34 <tellesnobrega> #topic News/Updates
14:04:13 <tellesnobrega> we are on a really good state on split plugins
14:04:24 <tellesnobrega> we have now all plugins on a separate repo
14:04:45 <tosky> testing repository (for the future readers of these logs)
14:05:03 <tellesnobrega> and devstack plugin is updated to install plugins from repos
14:06:01 <tellesnobrega> right now I'm working on return payload for api v2
14:06:09 <tellesnobrega> replacing tenant_id for project_id
14:06:29 <tosky> the fixes for the devstack plugin allows anyone to easily test the plugin
14:06:44 <tosky> would you mind pasting here the address of the testing repositories?
14:07:31 <tellesnobrega> sure
14:07:51 <tellesnobrega> #link https://github.com/tellesnobrega/sahara-plugin-cdh/
14:08:02 <tellesnobrega> #link https://github.com/tellesnobrega/sahara-plugin-ambari
14:08:13 <tellesnobrega> #link https://github.com/tellesnobrega/sahara-plugin-mapr
14:08:23 <tellesnobrega> #link https://github.com/tellesnobrega/sahara-plugin-spark
14:08:31 <tellesnobrega> #link https://github.com/tellesnobrega/sahara-plugin-storm
14:08:41 <tellesnobrega> #link https://github.com/tellesnobrega/sahara-plugin-vanilla
14:08:46 <tellesnobrega> and finally sahara
14:09:22 <tellesnobrega> #link https://github.com/tellesnobrega/sahara/tree/split-plugins
14:09:32 <tellesnobrega> pay attention to the correct branch
14:09:36 <tellesnobrega> split-plugins
14:10:19 <tellesnobrega> and that is all of them
14:10:59 <tosky> that's good; let me report my status and I think we can talk about it
14:11:04 <tellesnobrega> sure
14:11:16 <tosky> I've been working on some internal testing (plus vacation \o/) but I'm mostly back on track
14:11:47 <tosky> I still need to finish the patch for sahara-tests to enable S3 testing, but it's "almost" there
14:12:46 <tosky> I sent out a testing review to see if everything works with bionic nodes instead of xenial (where sahara runs, not the hadoop images), and indeed everything seems fine
14:13:31 <tellesnobrega> that is good to hear
14:13:48 <tosky> I guess we can move on
14:14:11 <tellesnobrega> ok
14:14:20 <tellesnobrega> api v2 next?
14:14:52 <tosky> or plugin split, whatever you prefer first :)
14:14:52 <tellesnobrega> or anything you want to cover before?
14:15:09 <tellesnobrega> lets go plugin split
14:15:19 <tellesnobrega> #topic Plugin Split
14:15:57 <tellesnobrega> ok, we are on a solid code state
14:16:08 <tosky> what we need to do is: a bit more testing
14:16:13 <tellesnobrega> we need to start moving on with infra, and tests
14:16:23 <tosky> and I definitely need to discuss with the packagers on how to handle to split
14:16:49 <tellesnobrega> yes
14:16:51 <tosky> especially on the RDO side, or we would block their continuous promotion pipeline, but not only them
14:17:21 <tosky> my plan is to send out an email, but I'd like to talk with RDO packagers (which I'm technically part of) first to remove the totally stupid solutions
14:18:01 <tosky> *an email to packagers and operators describing the change
14:18:13 <tosky> so I will try to do it before next meeting
14:18:18 <tellesnobrega> thanks tosky
14:18:22 <tellesnobrega> when you say more testing, you mean what exactly? more people testing or more of me testing?
14:18:30 <tosky> more people testing :)
14:18:30 <tellesnobrega> I would like to see more people testing it
14:18:33 <tellesnobrega> cool
14:18:37 <tellesnobrega> I appreciate that
14:19:03 <tosky> I'd need to reprovision a machine but it may take a bit
14:19:15 <tellesnobrega> let me bring jeremyfreudberg here, when you read this please contact us on your availability on testing this
14:20:26 <tosky> question: we have a pending review which adds sahara-image-pack support for spark, do we want to revive it and merge it before the split, or send it again afterwards?
14:20:52 <tellesnobrega> I can send it again afterwards
14:21:24 <tosky> oki
14:21:40 <tellesnobrega> I really want to see the split going forward now no stops
14:21:41 <tellesnobrega> lol
14:21:58 <tosky> sure, no more complicated rebases for you :)
14:22:04 <tellesnobrega> yes :)
14:22:46 <tellesnobrega> anything else on plugins split you want to cover?
14:22:55 <tosky> nothing more on my side about this
14:23:37 <tellesnobrega> ok
14:23:42 <tellesnobrega> #topic APIv2
14:24:50 <tellesnobrega> ok, so M2 is january 10th
14:25:01 <tellesnobrega> we have to get apiv2 going fast for it to be done in time
14:25:38 <tosky> so the point is: what is the status?
14:25:40 <tellesnobrega> unfortunately jeremyfreudberg isn't around today, so we may not be able to talk through it properly
14:25:54 <tosky> from my point of view, we still lack the basic tempest client, so tempest tests
14:26:21 <tosky> I think that Jeremy tried to hack the scenario tests. As they use the official clients, it should be easier to make them working
14:27:05 <tosky> you mentioned at least a fix that you need to write down
14:27:24 <tosky> I think we also miss the official activation of microversioning (there is a pending patch, on hold)
14:27:46 <tellesnobrega> I'm currently working on the payload
14:27:54 <tellesnobrega> it should not be complicated
14:28:04 <tellesnobrega> I just have to make one decision
14:28:20 <tosky> maybe another look by mordred just to be sure (was it him that helped answering some questions about APIs?)
14:28:40 <tellesnobrega> elmiko is also a good source for sure
14:29:22 <tellesnobrega> about the payload change, do we want to keep models as is for now and replace tenant_id for project_id on apiv2
14:29:39 <tellesnobrega> or do we want to change the model and make the change on apiv1?
14:30:18 <tosky> uhm, maybe if we avoid changes to v1 at this point...
14:32:08 <tellesnobrega> ok, so I will do the change now on apiv2, and once we remove apiv1 we can change the models
14:32:29 <tosky> just to be sure: models == db model?
14:32:51 <tellesnobrega> yes
14:34:38 <tellesnobrega> that is where the change should be done to finally remove old apiv1 stuff, like hadoop_version and tenant_id
14:34:42 <tosky> can we change them at any point without disrupting everything? Or better: if we change them now, shouldn't APIv1 hide the change?
14:34:50 <tosky> oh
14:35:08 <tellesnobrega> we can hide the change on apiv1 for sure
14:35:08 <tosky> well, I suppose that we don't want to remove APIv1 anyway for few cycles, given the amount of time that it was around
14:35:43 <tellesnobrega> that is basically the question I asked before, change now and hide for apiv1 or simply hide for apiv2 and change it once apiv1 is gone
14:38:27 <tosky> it's not the work I'm going to do, but I'd suggest to prioritize other work (like tests)
14:39:01 <tellesnobrega> so, keep it simple now and test more
14:39:04 <tosky> as the goal is "enable APIv2 as stable before R2", if that models change takes too much and it can be done later, maybe it's better to postpone
14:39:15 <tellesnobrega> that means hide for apiv2
14:39:22 <tellesnobrega> that is what I think is best as well
14:39:28 <tellesnobrega> considering the time we have
14:41:53 <tosky> oki
14:43:01 <tellesnobrega> awesome
14:43:08 <tellesnobrega> regarding tests, what do we need?
14:43:22 <tellesnobrega> tempest? scenarios are in good shape?
14:44:20 <tosky> we don't have any APIv2 test - and I'm not even sure how to quickly enable APIv2
14:45:04 <tosky> there is a documented procedure, and I may remember it incorrectly, but it may be easier to enable
14:45:25 <tellesnobrega> I think right now all that is needed is to change the endpoint to v2
14:45:44 <tellesnobrega> do you think we have time to test it before m2?
14:45:48 <tosky> so we need to finalize the patch to deploy with unversioned endpoints
14:46:19 <tosky> about testing: if we can deploy it, and if the support in python-saharaclient is complete, it should be easy to setup scenario tests
14:46:33 <tosky> tempest API tests requires a v2 data-processing tempest client
14:47:12 <tosky> if we have at least the scenario tests in an APIv2-only environment, it would be enough before M2, but then we'd *need* the full tempest client before M3
14:47:25 <tellesnobrega> ok
14:47:54 <tellesnobrega> that sounds like a solid plan
14:48:11 <tellesnobrega> I will follow up with jeremy and see how we can organize ourselves to get that done
14:49:42 <tosky> oki
14:49:50 <tellesnobrega> hopefully by next meeting we will have a more concrete plan
14:50:52 <tellesnobrega> I guess that covers what we have for today
14:51:02 <tellesnobrega> anything else on your mind tosky?
14:51:36 <tosky> nothing else, we covered The Big Two
14:51:54 <tosky> well, there is python3 porting, but let's see after the split
14:52:10 <tellesnobrega> ok
14:52:54 <tosky> uh!
14:53:00 <tellesnobrega> thanks tosky, see you around
14:53:01 <tosky> the tooz bug with python3 seems to have been fixed
14:53:10 <tellesnobrega> that is good :)
14:53:15 <tosky> I will rebase my "full-python3" test review
14:53:21 <tellesnobrega> sounds good
14:53:25 <tellesnobrega> keep me posted on that
14:53:26 <tosky> for the record, it was:
14:53:27 <tellesnobrega> please
14:53:49 <tosky> I thought that most of the deals happened online nowadays
14:53:57 <tosky> ups, wrong paste
14:54:04 <tosky> it was: https://bugs.launchpad.net/python-tooz/+bug/1530888
14:54:05 <openstack> Launchpad bug 1530888 in tooz "Bytes versus text issues on Python 3" [Medium,Triaged]
14:54:19 <tellesnobrega> hmmm, I've seen that issue before
14:54:26 <tosky> yep
14:54:36 <tellesnobrega> with the files copy bug
14:54:41 <tosky> so, let's use the almost-empty gates today and tomorrow
14:54:53 <tellesnobrega> I had to use a similar approach
14:54:59 <tellesnobrega> sounds good
14:55:47 <tosky> nothing else on my side
14:55:55 <tellesnobrega> great
14:56:00 <tellesnobrega> thanks tosky :)
14:56:17 <tellesnobrega> see you next week
14:56:38 <tellesnobrega> #endmeeting