18:00:34 #startmeeting trove 18:00:35 Meeting started Wed Jan 14 18:00:34 2015 UTC and is due to finish in 60 minutes. The chair is SlickNik. Information about MeetBot at http://wiki.debian.org/MeetBot. 18:00:36 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 18:00:39 The meeting name has been set to 'trove' 18:01:25 Giving folks a few minutes to trickle in 18:01:35 Meeting agenda at: https://wiki.openstack.org/wiki/Meetings/TroveMeeting 18:01:38 #link https://wiki.openstack.org/wiki/Meetings/TroveMeeting 18:01:46 o/ 18:02:31 o/ 18:03:46 Fairly light meeting agenda today — so let's get started. 18:03:54 o/ 18:04:02 o/ 18:04:14 #topic dib elements for DB2 Express-C 18:04:34 Thats mine. 18:04:39 johnma: go for it 18:04:49 #link https://review.openstack.org/#/c/133856/ 18:04:58 o/ 18:05:33 o/ 18:05:37 To give everyone an idea of what this item is about - it is to add support for DB2 Express-C. One of the problems we identified early on with creating images for DB2 express-C was that it required user to go through a registration process even though its the free version of DB2 18:06:22 like other databases Trove supports today, DB2 Express-C cannot be downloaded from a public repository either 18:07:20 johnma, do you mean "unlike" 18:07:25 we have been pretty much having lots of internal discussions within IBM throughout last month and there isnt a way to bypass the registration process. Its there to prevent users from embargoed countries from downloading DB2 18:07:41 yes unlike, mistyped 18:07:56 thanks dougshelley66 :) 18:08:57 so the solution we came up with was to have users go through the registration process and have them download the packages to a local repository from where trove can then access the packages. 18:09:31 Then we can go through the usual process of installating DB2 Express-C and creating users and all that good stuff. 18:09:40 o/ 18:10:02 johnma: How would this work with a public CI system (like the one OpenStack has)? 18:10:08 o/ 18:10:59 so what we can do is download the packages into a local directory on the local filesystem and then copy those packages to the temp filesystem that the dib has access to 18:11:00 I think the primary driver for this requirement is for a CI system (OpenStack / public 3rd party) to be able to pick up the bits, and build an image that it can then use as part of CI? 18:12:10 right now, the image creation process is external to Trove, isn't that right? 18:12:18 SlickNik, are there plans to have the CI test more than mysql? 18:12:24 johnma: I think that's okay as long as the CI system is able to download the packages — how does that work with the current registration wall? 18:12:25 so whoever that wants to create an image for DB2 express-C would download the db2 package into a local directory and wouldnt have to worry about putting it on a local repository 18:14:05 there is a website that the user will have to go and register. So user will have to go to the website and register and then download the packages and place it in the local directory. There isnt an automatic way to go through the registration process 18:14:24 vkmc: Yes, and no. The image creation happens as a separate step, but the function tests that run as part of the CI today test both the image build, and int-tests. 18:14:27 dougshelley66, +1 I have the same concern 18:14:29 So every time you ran redstack kick-start it would make you register and re-download the database software? 18:14:36 o/ 18:14:55 SlickNik, I see 18:15:31 dougshelley66: The plan is to have new datastores use a third party CI system to ensure that some sort of CI runs on that new datastore type. 18:15:52 as I understand it, this approach will not work well for the CI system 18:16:07 as it requires a person to download the db package and put it someplace accessible to the ci 18:16:13 dougshelley66: So that we make sure that we don't regress functionality for those datastore types with changes in common code. 18:16:14 the assumption is that once youregister and download the packages in a local directory that the dib has access to, everytime you run redstack kickstart it will automatically copy the package from your local dir to the temp FS and continue with the installation process 18:16:27 so you wouldnt have to go through the registration process every time 18:16:32 and that place, whatever it is, is one that would be for the most part, tantamount to redistributing the database, would it not? 18:17:16 thats right amrith 18:17:42 johnma, so IBM lawyers are ok with people booting a pre-created image where the person booting the image didn't go thru the click wrap registration? 18:17:52 I am assuming this could be a model that works for other databases which require licenses as well. I can speak for DB2 licensed versions and Cloudant 18:18:30 * amrith waits for answer to dougshelley66's question with bated breath 18:19:06 * vkmc as well 18:19:26 * amrith predicts that the answer will be no 18:19:29 dougshelley66, thats right. For DB2 what matters is, the person downloading it isnt from an embargoed country. 18:19:55 johnma, what if the person booting the image is in an embargoed country? 18:19:59 and if the person using the pre created image with DB2 is in ... 18:20:01 that 18:20:03 so the pre-created image will be created by someone who isn't in an embargoed country. What protection is there on theperson booting the miage 18:20:57 I believe the EULA you agree to when downloading these things says you can’t redist to an embargoed country. The intent is to do due diligence not full prevention 18:21:20 the eula says you can't redistribute. 18:21:33 a guest image would violate that eula 18:21:48 I'm not a lawyer, I don't play one on IRC 18:21:56 thats possible and I did bring it up during our discussions and what mattered was the country from where the packages were downloaded. Now I can verify again but I am pretty sure they are going to be fine 18:22:11 :) , no theses are good questions amrith 18:22:41 amrith: Yes, which is probably the reason that we cannot cache the guest image and make it available for download in this case. 18:22:57 SlickNik, yes. and for the same reason you can't build them on the fly 18:23:01 which is the catch-22 18:23:31 slicknik: is it necessary that the openstack CI system builds and tests every datastore. for non-opensource datastores/the one requiring user registration etc. should we allow similar build approach that johnma pointed out. and keep CI/CD for such datastore outside the scope of the openstack CI. let owners worry about doing CI/CD on these and in turn worry about the how they want to keep their images pacakges uptodate? 18:24:06 saurabhs, +1 18:24:19 saurabhs, while your approach is attractive, how do I ensure that some change I make doesn't break some IBM code for db2? 18:24:19 that would make my life easy 18:24:29 saurabhs: We need to ensure that some sort of CI/CD Is happening on this as it's part of the trove codebase. 18:24:37 maybe the simple way to solve this is to clearly explain to the legal team how it will be usd. If they sign-off then it’s probably outside our area of concern/expertise 18:24:41 recently I had a chkin which I though may impact RedHat so I requested vkmc and sgotliv to test 18:24:44 but that isn't scalable. 18:24:49 saurabhs: or else we're shipping code that's not tested. 18:25:07 amrith: if your change breaks IBM code, their CI/CD should catch it and vote on your review of comment with approprite fix 18:25:35 saurabhs: Yes, so we need to ensure that there is a 3rd party CI/CD system that is testing this in the public. 18:25:36 except the CI/CD for IBM would then have to cherry pick each patch and test it in near real time 18:25:37 when you say break IBM code, do you mean the guestagent code for DB2? 18:25:49 and I wouldn't know because that CI/CD wouldn't know to notify me. 18:25:51 I think 18:25:58 johnma, yes 18:26:17 Just to be clear - at this point, the CI only tests mysql, right? 18:26:18 and do we want to extend the checkin gate time to whatever is the longest gate time of the third party gates 18:26:21 amrith: I think we should take the approach that if you are not notified you shouldn' 18:26:22 ok, that is a valid point Amrith 18:26:26 shouldn 18:26:37 shouldn't be worried about breaking it. 18:26:57 dougshelley66: Yes, that is correct — currently there's only the OpenStack CI which tests only mysql. 18:27:13 thirdpaty ci/cd should integrate with Openstack CI for voting. 18:27:15 although there is an experimental mongo job 18:27:50 SlickNik: would the idea of 3rd party testing ... 18:28:02 ^^^ what saurabhs said ... 18:28:05 I think there are lot of thirdparty drivers who have their CI/CD voting on various other projects. like neutron. there are many proprietary network drivers which vote on each neutron patch 18:28:08 saurabhs: agreed. The issue here is that the 3rd party CI can't download and install the datastore in public. 18:28:53 As long as that can be done, and the 3rd party CI system is set up to do it — I think we're okay. 18:28:59 well, in this case the 3rd party CI would be run on IBM's machines (presumably) so no issue 18:29:06 right? 18:29:17 also IMHO a non-voting CI will be ignored for the most part. Ownership of break in the third party gate is not clear 18:29:28 Slicknik: not in public but they will be able to do it within their CI/CD system. 18:30:07 as shayneburgess said. heck some projects ignore non-voting jobs in the openstack CI. 18:30:57 looking for results from non-voting ci job can be part of the review process. 18:31:05 amrith: and if the 3rd party CI job keeps failing for a long time , the code is pulled out of the project since it's deemed non-working (iirc) 18:31:32 saurabhs: probably won't happen, though 18:31:40 saurabhs, could be and should be 18:31:46 petersac: I guess that would work 18:32:43 I think the important part here is that we decide what tests are a requirement for the 3rd party CI system to run. 18:33:07 just curious. if my fix causes a break in the *example* IBM agent because of a seperate bug in the agent who owns fixing it before my checkin goes in? 18:33:16 Getting the licensing right to download the bits / create the image, and actually run the tests are up to the 3rd party CI system. 18:35:13 shayneburgess: You bring up a good point — at that point presumably you'd have to work with the IBM folks to see what went wrong and one interested parties would have to propose a fix. 18:35:13 so SlickNik, who owns the dib elements for DB2 in this case, do they become part of the trove-integration project 18:36:36 so who is the IBM contact - right now I guess its just me. Whats the process of setting a contact from IBM or any other company for that matter - is there an existing process 18:36:52 I believe dib elements should still go in trove-integration and third party CI should run everything from upstream trove-integration. so that if somebody else wants to build/test this they can (by registering and downloading the packages) etc. 18:36:56 johnma: Yes, I'd assume that the dib elements would be part of the trove-integration project. I don't see a reason to keep it out. 18:36:56 SlickNick: makes sense. So third parties can add a gate and ultimate ownership of the quality of that resides with them, not Trove as a whole. It’s an indicator for the owner of that gate the quality of their datastore support 18:37:14 SlickNik, a question for you ... 18:37:17 related to that 18:37:28 I just fixed https://review.openstack.org/#/c/146147/ 18:37:38 It involved making changes for redhat and ubuntu 18:37:50 in the future, I would (in theory) have to make changes for db2 as well 18:38:00 is it incumbent upon me as a developer to also make those changes? 18:38:28 the issue is that unless reviewers catch the fact that the change hasn't been made for db2, the openstack CI won't break 18:38:50 and no one will be the wiser and we'll get in the same kind of situation we're in with redhat where 'kick-start mysql' doesn't work! 18:38:52 only worse 18:38:59 amrith: You wouldn't have to make that change for DB2, those elements are shared across datastores, IIRC. 18:39:09 you'd have to make that change for a new OS, perhaps 18:39:46 SlickNik, you take my example more literally than I'd intended. 18:39:56 I was only testing ubuntu but got help to test redhat 18:40:08 if I make a change that potentially impacts multiple datastores, then I'd test with those 18:40:15 like mongo, cassandra, mysql, ... 18:40:21 in future would I also have to test db2? 18:40:21 its impossible to make a change and make sure it works for n datastores 18:40:44 more if they are not open source 18:41:03 vkmc, but if the CI tests all datastores, you would have to do that 18:41:05 vkmc: unless you test it against n datastores :) 18:41:09 If we're designing this right, the footprint of code which is shared across datastore is not duplicated across the n-datasores. 18:41:41 dougshelley66, peterstac, that's my concern :) 18:41:59 IMHO its impractical that a feature needs to be supported across each datastore. we should be able to add a feature to any datastore of our choice and if anybody else is interested they can get that feature ported/working for datastore of their choice 18:42:23 that way when number of datastore increase in future our feature dev time doesn't increase with it 18:42:35 Are we proposing then every time a proprietary datastore is added we add a specific CI that runs on that companies machines and we then have to have a contact to that company if something breaks? Doesn't feel very open to me. 18:42:45 saurabhs, I agree. the converse is my question. I don't believe that it is reasonable to say I want a fix in one datastore only and if I happen to break another datastore then the interested parties can fix that there. 18:42:49 edmondk, +1 18:43:36 What happens if we added Oracle, Microsoft SQL Server, do we need servers from Microsoft and Oracle to run a CI? 18:43:40 amrith: correct we can take that approach only for new features for existing feature we should make sure that we don't break them for any datastore 18:44:14 edmondk: if they don't want to put their images/packages on public repos 18:44:35 saurabhs, the issue is how do I as a person fixing bugs ensure that I don't regress a datastore to which there's no CI as a direct part of my chkin process. 18:44:50 edmondk: +1. It’s not clear if Oracle or MS has any interest in us supporting their datastores 18:44:51 edmondk: No — we're proposing a similar model to the Neutron Driver model. For any new datastore that anyone wants to add to Trove, we require a minimum set of int-tests to run on a 3rd party open public CI system that is transparent and votes on the gerrit reviews, with logs that are accessible in case of failures. 18:46:16 3rd party because the CI system and the corresponding resources are set up by the interested party in question. 18:46:17 the issue with that proposal SlickNik is that it be an open public CI. 18:46:27 maybe it would be a good idea to ask Neutron folks how that work for them 18:46:50 are all those drivers in Neutron code base? or are them in external repos? 18:47:37 vkmc: If you want your driver/datastore in an external repo — feel free and have at it. You can do whatever you like. 18:48:01 I believe this is the minimum requirement for in-repo drivers (or in this case datastores) 18:48:45 SlickNik, so...you are saying that if the driver developer wants their driver to be in Trove codebase, then they also should provide a minimum set of int tests 18:48:51 So that there is a minimum guarantee to the quality of the code that ships as part of the OpenStack project. 18:49:35 SlickNik: The scenario then would be IBM adds code for DB2 in upstream Trove, add int-tests, then also provide their own CI build machine that hooks into our gerrit review system to run those specific DB2 int-tests? 18:49:35 vkmc: minimum set of int-tests, and a minimum infrastructure to run it and ensure that those tests and consequently the in-repo code is working. 18:50:10 edmondk: Yes. that's the suggestion. 18:50:39 SlickNik, would the jobs run on the IBM CI (in this case) be gating jobs in the openstack CI? 18:51:04 what happens if a change in Trove requires the 3rd party to change their code? 18:51:18 amrith: Yes, if the jobs run stable — they can be promoted to voting jobs. 18:51:23 amrith: http://ci.openstack.org/third_party.html 18:51:56 SlickNik, amrith: That's essentially what rdjenkins used to be, correct? 18:52:37 vkmc: If it's a stable 3rd party job, that change in trove won't be able to land without a corresponding change in the 3rd party job. 18:52:49 ^ amrith: 18:53:09 SlickNik, I'm reading that link ... 18:53:29 peterstac: Yes, more or less. 18:53:48 hey guys not to interrupt, but please leave 5 min in the end to go over Anna's changes based on yesterday's discussion 18:54:07 Yes, I think there's a lot here to cover for 3rd party stuff. 18:54:20 Let's move on and we can revisit this topic in the channel. 18:54:28 ok, sounds good 18:54:54 #topic Review #131610 18:55:00 XO19_: ^ 18:55:12 #topic Review 131610 18:55:24 #link https://review.openstack.org/#/c/131610/ 18:55:47 I think XO19_ proposed a new patch set recently 18:55:56 I haven't yet had a chance to review it. 18:56:08 SlickNik: yes so yest in the channel amrith suggested we give the tenant to block log access to admin 18:56:17 the changes in the review reflect that 18:57:19 I see the new patches proposed the addition of the admin_log_access boolean param 18:57:31 if that is set to true 18:57:41 and might want to decide if its better off implemented as an extension 18:57:57 does that mean that only the admin user of the project is the only one capable of accesing to the new endpoints? 18:58:13 or every user in the tenant can access them? 18:58:24 vkmc: nope amrith wanted the tenant to decide whether admin can have accessin addition to the tenant 18:59:30 I'm indifferent to the switch. IMO an admin can basically alter the trove code running underneath, so there are no guarantees that the switch would even have to be honored. 19:00:26 IMO A malicious admin can pretty much do whatever he wants. 19:00:34 SlickNik, you are thinking about the cloud admin, not the project/tenant admin user 19:00:37 right? 19:01:12 19:01:29 Looks good to me either way. 19:01:50 ok thank you, folks feel free to leave comments on review if any 19:02:00 thanks XO19_ for working on the spec 19:02:15 vkmc: yes — for a user in the same tenant they'd have access to the logs anyway. 19:02:15 thanks XO19_! 19:02:23 #topic Open Discussion 19:02:47 Anything? 19:03:06 If so, let's continue the discussion in #openstack-trove. 19:03:06 yes 19:03:14 but acn we convene in #openstack-trove 19:03:19 #endmeeting