18:00:34 <SlickNik> #startmeeting trove
18:00:35 <openstack> Meeting started Wed Jan 14 18:00:34 2015 UTC and is due to finish in 60 minutes.  The chair is SlickNik. Information about MeetBot at http://wiki.debian.org/MeetBot.
18:00:36 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
18:00:39 <openstack> The meeting name has been set to 'trove'
18:01:25 <SlickNik> Giving folks a few minutes to trickle in
18:01:35 <SlickNik> Meeting agenda at: https://wiki.openstack.org/wiki/Meetings/TroveMeeting
18:01:38 <SlickNik> #link https://wiki.openstack.org/wiki/Meetings/TroveMeeting
18:01:46 <vkmc> o/
18:02:31 <georgelorch> o/
18:03:46 <SlickNik> Fairly light meeting agenda today — so let's get started.
18:03:54 <dougshelley66> o/
18:04:02 <bartash> o/
18:04:14 <SlickNik> #topic dib elements for DB2 Express-C
18:04:34 <johnma> Thats mine.
18:04:39 <SlickNik> johnma: go for it
18:04:49 <SlickNik> #link https://review.openstack.org/#/c/133856/
18:04:58 <peterstac> o/
18:05:33 <vgnbkr> o/
18:05:37 <johnma> To give everyone an idea of what this item is about - it is to add support for DB2 Express-C. One of the problems we identified early on with creating images for DB2 express-C was that it required user to go through a registration process even though its the free version of DB2
18:06:22 <johnma> like other databases Trove supports today, DB2 Express-C cannot be downloaded from a public repository either
18:07:20 <dougshelley66> johnma, do you mean "unlike"
18:07:25 <johnma> we have been pretty much having lots of internal discussions within IBM throughout last month and there isnt a way to bypass the registration process. Its there to prevent users from embargoed countries from downloading DB2
18:07:41 <johnma> yes unlike, mistyped
18:07:56 <johnma> thanks dougshelley66 :)
18:08:57 <johnma> so the solution we came up with was to have users go through the registration process and have them download the packages to a local repository from where trove can then access the packages.
18:09:31 <johnma> Then we can go through the usual process of installating DB2 Express-C and creating users and all that good stuff.
18:09:40 <XO19_> o/
18:10:02 <SlickNik> johnma: How would this work with a public CI system (like the one OpenStack has)?
18:10:08 <annashen> o/
18:10:59 <johnma> so what we can do is download the packages into a local directory on the local filesystem and then copy those packages to the temp filesystem that the dib has access to
18:11:00 <SlickNik> I think the primary driver for this requirement is for a CI system (OpenStack / public 3rd party) to be able to pick up the bits, and build an image that it can then use as part of CI?
18:12:10 <vkmc> right now, the image creation process is external to Trove, isn't that right?
18:12:18 <dougshelley66> SlickNik, are there plans to have the CI test more than mysql?
18:12:24 <SlickNik> johnma: I think that's okay as long as the CI system is able to download the packages — how does that work with the current registration wall?
18:12:25 <johnma> so whoever that wants to create an image for DB2 express-C would download the db2 package into a local directory and wouldnt have to worry about putting it on a local repository
18:14:05 <johnma> there is a website that the user will have to go and register. So user will have to go to the website and register and then download the packages and place it in the local directory. There isnt an automatic way to go through the registration process
18:14:24 <SlickNik> vkmc: Yes, and no. The image creation happens as a separate step, but the function tests that run as part of the CI today test both the image build, and int-tests.
18:14:27 <vkmc> dougshelley66, +1 I have the same concern
18:14:29 <vgnbkr> So every time you ran redstack kick-start it would make you register and re-download the database software?
18:14:36 <amrith> o/
18:14:55 <vkmc> SlickNik, I see
18:15:31 <SlickNik> dougshelley66: The plan is to have new datastores use a third party CI system to ensure that some sort of CI runs on that new datastore type.
18:15:52 <amrith> as I understand it, this approach will not work well for the CI system
18:16:07 <amrith> as it requires a person to download the db package and put it someplace accessible to the ci
18:16:13 <SlickNik> dougshelley66: So that we make sure that we don't regress functionality for those datastore types with changes in common code.
18:16:14 <johnma> the assumption is that once youregister and download the packages in a local directory that the dib has access to, everytime you run redstack kickstart it will automatically copy the package from your local dir to the temp FS and continue with the installation process
18:16:27 <johnma> so you wouldnt have to go through the registration process every time
18:16:32 <amrith> and that place, whatever it is, is one that would be for the most part, tantamount to redistributing the database, would it not?
18:17:16 <johnma> thats right amrith
18:17:42 <dougshelley66> johnma, so IBM lawyers are ok with people booting a pre-created image where the person booting the image didn't go thru the click wrap registration?
18:17:52 <johnma> I am assuming this could be a model that works for other databases which require licenses as well. I can speak for DB2 licensed versions and Cloudant
18:18:30 * amrith waits for answer to dougshelley66's question with bated breath
18:19:06 * vkmc as well
18:19:26 * amrith predicts that the answer will be no
18:19:29 <johnma> dougshelley66, thats right. For DB2 what matters is, the person downloading it isnt from an embargoed country.
18:19:55 <dougshelley66> johnma, what if the person booting the image is in an embargoed country?
18:19:59 <vkmc> and if the person using the pre created image with DB2 is in ...
18:20:01 <vkmc> that
18:20:03 <amrith> so the pre-created image will be created by someone who isn't in an embargoed country. What protection is there on theperson booting the miage
18:20:57 <shayneburgess> I believe the EULA you agree to when downloading these things says you can’t redist to an embargoed country. The intent is to do due diligence not full prevention
18:21:20 <amrith> the eula says you can't redistribute. <period>
18:21:33 <amrith> a guest image would violate that eula
18:21:48 <amrith> <disclaimer>I'm not a lawyer, I don't play one on IRC</disclaimer>
18:21:56 <johnma> thats possible and I did bring it up during our discussions and what mattered was the country from where the packages were downloaded. Now I can verify again but I am pretty sure they are going to be fine
18:22:11 <johnma> :) , no theses are good questions amrith
18:22:41 <SlickNik> amrith: Yes, which is probably the reason that we cannot cache the guest image and make it available for download in this case.
18:22:57 <amrith> SlickNik, yes. and for the same reason you can't build them on the fly
18:23:01 <amrith> which is the catch-22
18:23:31 <saurabhs> slicknik: is it necessary that the openstack CI system builds and tests every datastore. for non-opensource datastores/the one requiring user registration etc. should we allow similar build approach that johnma pointed out. and keep CI/CD for such datastore outside the scope of the openstack CI. let owners worry about doing CI/CD on these and in turn worry about the how they want to keep their images pacakges uptodate?
18:24:06 <vkmc> saurabhs, +1
18:24:19 <amrith> saurabhs, while your approach is attractive, how do I ensure that some change I make doesn't break some IBM code for db2?
18:24:19 <johnma> that would make my life easy
18:24:29 <SlickNik> saurabhs: We need to ensure that some sort of CI/CD Is happening on this as it's part of the trove codebase.
18:24:37 <shayneburgess> maybe the simple way to solve this is to clearly explain to the legal team how it will be usd. If they sign-off then it’s probably outside our area of concern/expertise
18:24:41 <amrith> recently I had a chkin which I though may impact RedHat so I requested vkmc and sgotliv to test
18:24:44 <amrith> but that isn't scalable.
18:24:49 <SlickNik> saurabhs: or else we're shipping code that's not tested.
18:25:07 <saurabhs> amrith: if your change breaks IBM code, their CI/CD should catch it and vote on your review of comment with approprite fix
18:25:35 <SlickNik> saurabhs: Yes, so we need to ensure that there is a 3rd party CI/CD system that is testing this in the public.
18:25:36 <amrith> except the CI/CD for IBM would then have to cherry pick each patch and test it in near real time
18:25:37 <johnma> when you say break IBM code, do you mean the guestagent code for DB2?
18:25:49 <amrith> and I wouldn't know because that CI/CD wouldn't know to notify me.
18:25:51 <amrith> I think
18:25:58 <amrith> johnma, yes
18:26:17 <dougshelley66> Just to be clear - at this point, the CI only tests mysql, right?
18:26:18 <shayneburgess> and do we want to extend the checkin gate time to whatever is the longest gate time of the third party gates
18:26:21 <saurabhs> amrith: I think we should take the approach that if you are not notified you shouldn'
18:26:22 <johnma> ok, that is a valid point Amrith
18:26:26 <saurabhs> shouldn
18:26:37 <saurabhs> shouldn't be worried about breaking it.
18:26:57 <SlickNik> dougshelley66: Yes, that is correct — currently there's only the OpenStack CI which tests only mysql.
18:27:13 <saurabhs> thirdpaty ci/cd should integrate with Openstack CI for voting.
18:27:15 <SlickNik> although there is an experimental mongo job
18:27:50 <peterstac> SlickNik: would the idea of 3rd party testing ...
18:28:02 <peterstac> ^^^ what saurabhs said ...
18:28:05 <saurabhs> I think there are lot of thirdparty drivers who have their CI/CD voting on various other projects. like neutron. there are many proprietary network drivers which vote on each neutron patch
18:28:08 <SlickNik> saurabhs: agreed. The issue here is that the 3rd party CI can't download and install the datastore in public.
18:28:53 <SlickNik> As long as that can be done, and the 3rd party CI system is set up to do it — I think we're okay.
18:28:59 <peterstac> well, in this case the 3rd party CI would be run on IBM's machines (presumably) so no issue
18:29:06 <peterstac> right?
18:29:17 <shayneburgess> also IMHO a non-voting CI will be ignored for the most part. Ownership of break in the third party gate is not clear
18:29:28 <saurabhs> Slicknik: not in public but they will be able to do it within their CI/CD system.
18:30:07 <amrith> as shayneburgess said. heck some projects ignore non-voting jobs in the openstack CI.
18:30:57 <saurabhs> looking for results from non-voting ci job can be part of the review process.
18:31:05 <SlickNik> amrith: and if the 3rd party CI job keeps failing for a long time , the code is pulled out of the project since it's deemed non-working (iirc)
18:31:32 <peterstac> saurabhs: probably won't happen, though
18:31:40 <vkmc> saurabhs, could be and should be
18:31:46 <johnma> petersac: I guess that would work
18:32:43 <SlickNik> I think the important part here is that we decide what tests are a requirement for the 3rd party CI system to run.
18:33:07 <shayneburgess> just curious. if my fix causes a break in the *example* IBM agent because of a seperate bug in the agent who owns fixing it before my checkin goes in?
18:33:16 <SlickNik> Getting the licensing right to download  the bits / create the image, and actually run the tests are up to the 3rd party CI system.
18:35:13 <SlickNik> shayneburgess: You bring up a good point — at that point presumably you'd have to work with the IBM folks to see what went wrong and one interested parties would have to propose a fix.
18:35:13 <johnma> so SlickNik, who owns the dib elements for DB2 in this case, do they become part of the trove-integration project
18:36:36 <johnma> so who is the IBM contact - right now I guess its just me. Whats the process of setting a contact from IBM or any other company for that matter - is there an existing process
18:36:52 <saurabhs> I believe dib elements should still go in trove-integration and third party CI should run everything from upstream trove-integration. so that if somebody else wants to build/test this they can (by registering and downloading the packages) etc.
18:36:56 <SlickNik> johnma: Yes, I'd assume that the dib elements would be part of the trove-integration project. I don't see a reason to keep it out.
18:36:56 <shayneburgess> SlickNick: makes sense. So third parties can add a gate and ultimate ownership of the quality of that resides with them, not Trove as a whole. It’s an indicator for the owner of that gate the quality of their datastore support
18:37:14 <amrith> SlickNik, a question for you ...
18:37:17 <amrith> related to that
18:37:28 <amrith> I just fixed https://review.openstack.org/#/c/146147/
18:37:38 <amrith> It involved making changes for redhat and ubuntu
18:37:50 <amrith> in the future, I would (in theory) have to make changes for db2 as well
18:38:00 <amrith> is it incumbent upon me as a developer to also make those changes?
18:38:28 <amrith> the issue is that unless reviewers catch the fact that the change hasn't been made for db2, the openstack CI won't break
18:38:50 <amrith> and no one will be the wiser and we'll get in the same kind of situation we're in with redhat where 'kick-start mysql' doesn't work!
18:38:52 <amrith> only worse
18:38:59 <SlickNik> amrith: You wouldn't have to make that change for DB2, those elements are shared across datastores, IIRC.
18:39:09 <SlickNik> you'd have to make that change for a new OS, perhaps
18:39:46 <amrith> SlickNik, you take my example more literally than I'd intended.
18:39:56 <amrith> I was only testing ubuntu but got help to test redhat
18:40:08 <amrith> if I make a change that potentially impacts multiple datastores, then I'd test with those
18:40:15 <amrith> like mongo, cassandra, mysql, ...
18:40:21 <amrith> in future would I also have to test db2?
18:40:21 <vkmc> its impossible to make a change and make sure it works for n datastores
18:40:44 <vkmc> more if they are not open source
18:41:03 <dougshelley66> vkmc, but if the CI tests all datastores, you would have to do that
18:41:05 <peterstac> vkmc: unless you test it against n datastores :)
18:41:09 <SlickNik> If we're designing this right, the footprint of code which is shared across datastore is not duplicated across the n-datasores.
18:41:41 <vkmc> dougshelley66, peterstac, that's my concern :)
18:41:59 <saurabhs> IMHO its impractical that a feature needs to be supported across each datastore. we should be able to add a feature to any datastore of our choice and if anybody else is interested they can get that feature ported/working for datastore of their choice
18:42:23 <saurabhs> that way when number of datastore increase in future our feature dev time doesn't increase with it
18:42:35 <edmondk> Are we proposing then every time a proprietary datastore is added we add a specific CI that runs on that companies machines and we then have to have a contact to that company if something breaks? Doesn't feel very open to me.
18:42:45 <amrith> saurabhs, I agree. the converse is my question. I don't believe that it is reasonable to say I want a fix in one datastore only and if I happen to break another datastore then the interested parties can fix that there.
18:42:49 <vkmc> edmondk, +1
18:43:36 <edmondk> What happens if we added Oracle, Microsoft SQL Server, do we need servers from Microsoft and Oracle to run a CI?
18:43:40 <saurabhs> amrith: correct we can take that approach only for new features for existing feature we should make sure that we don't break them for any datastore
18:44:14 <saurabhs> edmondk: if they don't want to put their images/packages on public repos
18:44:35 <amrith> saurabhs, the issue is how do I as a person fixing bugs ensure that I don't regress a datastore to which there's no CI as a direct part of my chkin process.
18:44:50 <shayneburgess> edmondk: +1. It’s not clear if Oracle or MS has any interest in us supporting their datastores
18:44:51 <SlickNik> edmondk: No — we're proposing a similar model to the Neutron Driver model. For any new datastore that anyone wants to add to Trove, we require a minimum set of int-tests to run on a 3rd party open public CI system that is transparent and votes on the gerrit reviews, with logs that are accessible in case of failures.
18:46:16 <SlickNik> 3rd party because the CI system and the corresponding resources are set up by the interested party in question.
18:46:17 <amrith> the issue with that proposal SlickNik is that it be an open public CI.
18:46:27 <vkmc> maybe it would be a good idea to ask Neutron folks how that work for them
18:46:50 <vkmc> are all those drivers in Neutron code base? or are them in external repos?
18:47:37 <SlickNik> vkmc: If you want your driver/datastore in an external repo — feel free and have at it. You can do whatever you like.
18:48:01 <SlickNik> I believe this is the minimum requirement for in-repo drivers (or in this case datastores)
18:48:45 <vkmc> SlickNik, so...you are saying that if the driver developer wants their driver to be in Trove codebase, then they also should provide a minimum set of int tests
18:48:51 <SlickNik> So that there is a minimum guarantee to the quality of the code that ships as part of the OpenStack project.
18:49:35 <edmondk> SlickNik: The scenario then would be IBM adds code for DB2 in upstream Trove, add int-tests, then also provide their own CI build machine that hooks into our gerrit review system to run those specific DB2 int-tests?
18:49:35 <SlickNik> vkmc: minimum set of int-tests, and a minimum infrastructure to run it and ensure that those tests and consequently the in-repo code is working.
18:50:10 <SlickNik> edmondk: Yes. that's the suggestion.
18:50:39 <amrith> SlickNik, would the jobs run on the IBM CI (in this case) be gating jobs in the openstack CI?
18:51:04 <vkmc> what happens if a change in Trove requires the 3rd party to change their code?
18:51:18 <SlickNik> amrith: Yes, if the jobs run stable — they can be promoted to voting jobs.
18:51:23 <SlickNik> amrith: http://ci.openstack.org/third_party.html
18:51:56 <peterstac> SlickNik, amrith: That's essentially what rdjenkins used to be, correct?
18:52:37 <SlickNik> vkmc: If it's a stable 3rd party job, that change in trove won't be able to land without a corresponding change in the 3rd party job.
18:52:49 <SlickNik> ^ amrith:
18:53:09 <amrith> SlickNik, I'm reading that link ...
18:53:29 <SlickNik> peterstac: Yes, more or less.
18:53:48 <iccha> hey guys not to interrupt, but please leave 5 min in the end to go over Anna's changes based on yesterday's discussion
18:54:07 <SlickNik> Yes, I think there's a lot here to cover for 3rd party stuff.
18:54:20 <SlickNik> Let's move on and we can revisit this topic in the channel.
18:54:28 <johnma> ok, sounds good
18:54:54 <SlickNik> #topic Review #131610
18:55:00 <iccha> XO19_: ^
18:55:12 <SlickNik> #topic Review 131610
18:55:24 <amrith> #link https://review.openstack.org/#/c/131610/
18:55:47 <SlickNik> I think XO19_ proposed a new patch set recently
18:55:56 <SlickNik> I haven't yet had a chance to review it.
18:56:08 <iccha> SlickNik: yes so yest in the channel amrith suggested we give the tenant to block log access to admin
18:56:17 <iccha> the changes in the review reflect that
18:57:19 <vkmc> I see the new patches proposed the addition of the admin_log_access boolean param
18:57:31 <vkmc> if that is set to true
18:57:41 <XO19_> and might want to decide if its better off implemented as an extension
18:57:57 <vkmc> does that mean that only the admin user of the project is the only one capable of accesing to the new endpoints?
18:58:13 <vkmc> or every user in the tenant can access them?
18:58:24 <iccha> vkmc: nope amrith wanted the tenant to decide whether admin can have accessin addition to the tenant
18:59:30 <SlickNik> I'm indifferent to the switch. IMO an admin can basically alter the trove code running underneath, so there are no guarantees that the switch would even have to be honored.
19:00:26 <SlickNik> IMO A malicious admin can pretty much do whatever he wants.
19:00:34 <vkmc> SlickNik, you are thinking about the cloud admin, not the project/tenant admin user
19:00:37 <vkmc> right?
19:01:12 <amrith> <time-check>
19:01:29 <SlickNik> Looks good to me either way.
19:01:50 <iccha> ok thank you, folks feel free to leave comments on review if any
19:02:00 <iccha> thanks XO19_ for working on the spec
19:02:15 <SlickNik> vkmc: yes — for a user in the same tenant they'd have access to the logs anyway.
19:02:15 <vkmc> thanks XO19_!
19:02:23 <SlickNik> #topic Open Discussion
19:02:47 <SlickNik> Anything?
19:03:06 <SlickNik> If so, let's continue the discussion in #openstack-trove.
19:03:06 <amrith> yes
19:03:14 <amrith> but acn we convene in #openstack-trove
19:03:19 <SlickNik> #endmeeting