20:14:45 <heckj> #startmeeting
20:14:46 <openstack> Meeting started Tue May  1 20:14:45 2012 UTC.  The chair is heckj. Information about MeetBot at http://wiki.debian.org/MeetBot.
20:14:47 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
20:14:48 <bcwaldon> thank you
20:14:50 <notmyname> heckj: thanks :-)
20:14:54 <vishy> thanks heckj!
20:15:02 <heckj> #topic: 3rd party API policy
20:15:04 <mtaylor> heckj: try not to break #endmeeting again
20:15:12 <heckj> mtaylor: pbbbbbbbbbbbb
20:15:21 <notmyname> so we can either have a general policy or let project decide right? and if it's a general policy, what is it?
20:15:22 <heckj> who has proposals?
20:15:36 <vishy> so i think the first question is do we have a 3rd party api policy
20:15:46 <notmyname> I don't think we do right now
20:15:55 <vishy> should we have one?
20:16:06 <notmyname> I think we should :-)
20:16:09 <vishy> i think it is important that the ppb has an opinion
20:16:09 <bcwaldon> guidelines, at least
20:16:14 <devcamcar_> what does it mean to have one?
20:16:22 <mtaylor> for the sake of the peanut gallery, what is meant by 3rd party api policy?
20:16:22 <vishy> even if we say up to the ptls
20:16:36 <vishy> mtaylor: so other apis esp. open standards
20:16:43 <vishy> do we include them in core projects
20:16:44 <mtaylor> kk
20:16:45 <cloudfly> a generic api rules of conformity would be nice.
20:16:59 <anotherjesse> devcamcar_: it means if someone whats to build another API to openstack, they would know how to do it …  (eg, if we aren't going to accept it into the core projects they would know they should build a proxy or external plugin)
20:17:25 * mtaylor now understands the conversation
20:17:38 <ttx> so the question is: is each project free to add a new external API, or does it need to go through PPB approval ?
20:17:47 <anotherjesse> we currently have AWS interop APIs in the projects.  one thought is that openstack should only expose the openstack API, but have external projects that provide translation to the openstack api
20:17:49 <ttx> or is it more complex than that ?
20:18:00 <vishy> ttx: that is one of the quesions
20:18:14 <bcwaldon> anotherjesse: not necessarily tranlation to the OpenStack API, but even able to plug into the compute API (nova/compute/api.py)
20:18:22 <vishy> we have stated in the past that openstack is about the apis, not the implementation
20:18:33 <anotherjesse> bcwaldon: no, I was saying one proposal is that we don't do that, we just have http proxies
20:18:33 <mtaylor> ttx: I think the first question is "should openstack have a policy on this at all" ... then "should it be global or per project" ... then, if global "what should the policy be"
20:18:38 <vishy> if so does that mean that core openstack should only include the openstack api
20:18:41 <anotherjesse> bcwaldon: another is that it is a 3rd party plugin
20:18:43 <bcwaldon> anotherjesse: ok, loud 'n clear
20:18:58 <anotherjesse> another is that we have a contrib folder with alternative APIs
20:19:20 <anotherjesse> ttx: I think the issue is that multiple groups care about different apis but using openstack implementation - so it would be good for them to know how we want them to interact with us
20:19:26 <devcamcar_> contrib implies python only?
20:19:29 <vishy> also, if we decide that apis are external, is there a process by which they can become part of core
20:19:38 <mtaylor> vishy: ++
20:19:52 <bcwaldon> vishy: would becoming core mean OpenStack has full control over that contract?
20:19:56 <vishy> or do we just say, not our problem, talk to the distributions
20:19:57 <bcwaldon> vishy: thinking about EC2 here...
20:20:12 <anotherjesse> mtaylor / vishy - what advantage would having a way of becoming core ?
20:20:14 <vishy> bcwaldon: no, i mean, included in the core tarballs with the release
20:20:18 <bcwaldon> ok
20:20:28 <anotherjesse> I personally lean towards making openstack have one api that others proxy against
20:20:34 <vishy> bcwaldon: no advantage for us, lots of advantage for the 3rd parties :)
20:20:43 <notmyname> devcamcar_: not really. contrib could simply be a list of compatible plugins manages separately
20:20:46 * mtaylor leans towards anotherjesse
20:20:46 <vishy> er anotherjesse: ^^
20:20:52 <anotherjesse> keeps the surface area of the project smaller from a security and complexity point of view
20:20:53 * bcwaldon leans towards mtaylor
20:21:17 <bcwaldon> the proxy approach is the easiest for us, but we'd have to bite the bullet performance-wise
20:21:22 <anotherjesse> vishy: what is the advantage?  that we are now in charge of keeping it up-to-date and working?
20:21:25 <mtaylor> also means external api proxies are themselves easier to verify in isolation
20:21:45 <anotherjesse> it also has the disadvantage of coupling release
20:21:48 <vishy> anotherjesse: the advantage for them is that they gain visibility
20:22:06 <devcamcar_> anotherjesse: did that just become a proposal to remove ec2 API?
20:22:26 <vishy> devcamcar_: unfortunately that is not so easy
20:22:33 <devcamcar_> I know
20:22:48 <vishy> devcamcar_: after rpc versioning it is possible, but we still have to figure out what to do with the mapping tables.
20:22:50 <anotherjesse> devcamcar_: long term yes - it would require a project to exist and the APIs to fully expose the required features
20:22:59 <anotherjesse> but that would be good to do ;)
20:23:03 <mtaylor> ++
20:23:05 <heckj> devcamcar_: while I'd prefer any *additional* APIs be external myself, I think asserting the removal of EC2 support would send a very poor message publicly
20:23:32 <jog0> heckj: +1
20:23:34 <mtaylor> well, if we talk about it in terms of elevating it to a core (separate) project rather than "removing it"
20:23:34 <devcamcar_> I'm not proposing it
20:23:35 <ttx> We should treat EC2 as a historical corner case
20:23:50 <mtaylor> it might could be shaped to not send the wrong message
20:23:50 <anotherjesse> heckj: there is a group already working on a http-proxy based aws implementation -- if that got traction we might we able to "remove it"
20:23:54 <notmyname> removal of AWS APIs doesn't mean that it doesn't exist. it would be moved into an external proxy/plugin right?
20:24:10 <mtaylor> yeah. kind of like we're "removing nova-volume" right now
20:24:11 <ttx> rather than having it influence the decision on how to treat additional ones
20:24:18 <cloudfly> the phasing out of major components should have a formal process
20:24:39 <anotherjesse> ttx: ++
20:25:01 <mtaylor> ttx: ++
20:25:08 <ttx> but yes, the decision we'll take will influence where EC2 might end up one day.
20:25:16 <heckj> anotherjesse: sorry, I missed your original bits - lost connectivity for a bit. Do you have a discrete proposal that's in the logs?
20:25:19 <notmyname> cloudfly: not really. the way swift has done it in the past is to provide the pieces in another location
20:25:24 <ttx> s/EC2/EC2 APi support/
20:25:41 <notmyname> s/EC2/AWS/
20:25:46 <anotherjesse> heckj: I was saying I propose 3rd party APIs be done as http proxies
20:25:52 <heckj> anotherjesse: thanks
20:26:02 <ttx> i don't have strong feelings either way. I'd lean towards considering 3rd party APIs as optional  add-ons
20:26:25 <ttx> be them magic plug-ins that plug into the internal compute API... or proxies
20:26:33 <anotherjesse> ttx: I'm not against that, it requires us to stabilize a plugin api - which we haven't done
20:26:34 <devcamcar_> anotherjesse: +1
20:26:35 <vishy> is there any way that we can provide a process for a 3rd party api to be shipped as "part" of openstack?
20:26:41 <ttx> the second option sounding a lot more doable in the short term
20:26:43 <heckj> anotherjesse: +1
20:26:46 <vishy> does it mean applying for incubation?
20:26:52 <anotherjesse> vishy: no
20:26:54 <devcamcar_> vishy: could also
20:26:57 <anotherjesse> incubation is a path to core
20:27:02 <vishy> i know
20:27:03 <devcamcar_> make it an upstream problem?
20:27:19 <vishy> so we are saying that no other api could ever be in core?
20:27:46 <heckj> vishy: I think we're asserting an API can become part of core if it applies for and gains acceptance through the incubation process.
20:27:48 <devcamcar_> if not core then upstream packaging issue, if core then it's included
20:27:51 <anotherjesse> vishy: requirements change, but under the current situation I think that would be ideal
20:27:53 <ttx> We should have official projects developed within openstack that are not part of the core product.
20:28:11 <heckj> devcamcar_: I think you mean downstream, yes? (i.e. ubuntu, fedora, etc)
20:28:26 <mtaylor> ttx: ++
20:28:38 <devcamcar_> heckj: yes
20:28:40 <mtaylor> ttx: we should propose a formal definition of that at some point
20:28:44 <ttx> so that we can extend our ecosystem of projects without diluting the values of our core product
20:28:56 <ttx> mtaylor: rather soon than late, I think
20:28:57 <vishy> ttx: what does an official project mean in that case?
20:28:57 <anotherjesse> if you belive that most systems will be cloud enabled, and we hope that openstack is a major part of that, then the surface area of a complete cloud solution is huge
20:29:03 <anotherjesse> we shouldn't try to own everything
20:29:10 <anotherjesse> we should focus on being a great "kernel" for the cloud
20:29:12 <ttx> vishy: developed withing our infrastructure ?
20:29:22 <heckj> anotherjesse: +1
20:29:24 <mtaylor> anotherjesse: ++
20:29:24 <anotherjesse> trying to own every layer and impose our processes on it will limit inovation
20:29:40 <notmyname> anotherjesse: +1
20:29:46 <mtaylor> we already use many non-us components too ... mysql, rabbit, etc
20:29:49 <anotherjesse> having resources available (like mtaylor's stackforge) that help developers / projects out is great
20:29:59 <vishy> ttx: ok I'm not sure what the benefit is though.  Why would a third party project go through the overhead of being 'official'? What do they get out of it?
20:30:21 <ttx> vishy: some amount of openstack branding ?
20:30:34 <ttx> if you have openstack core, openstack official and ecosystem...
20:30:53 <ttx> For example, as of today, openstack-common is an official project.
20:30:57 <anotherjesse> owning the ecosystem could limit the ecosystem
20:31:04 <ttx> it's not core. It's like... used to build core
20:31:10 <notmyname> anotherjesse: s/could/would/
20:31:18 <ttx> but it's still very much officially openstack
20:31:22 <devcamcar_> core^2
20:31:22 <vishy> ttx: ok, so the current proposal is third party apis are not allowed in core, they are welcome in the ecosystem, and at some point there will be a process to become official?
20:31:32 <anotherjesse> supporting external projects that live outside openstaack but work with openstack would be my proferrence.
20:31:42 <anotherjesse> that might mean test suites, api work, ...
20:31:52 <devcamcar_> vishy: we better plan how to plan for that process though
20:32:16 <devcamcar_> otherwise we are just deferring the issue
20:32:18 <anotherjesse> because in the end the people selling / distributing openstack will go through and make decsisions about what their users get (what is possible / what is default)
20:32:29 <ttx> vishy: they should be developed as a plug-in or a proxy, so that they can be either a separate plug-in, an official plug-in (tested with core), or a core plugin (shipped with core)
20:32:39 <vishy> devcamcar_: i think we should also mandate an effort by the core projects to make 3rd party apis as easy as possible.
20:32:45 <anotherjesse> vishy: again - I don't see advantages to making them core
20:33:03 <ttx> vishy: +1
20:33:07 <vishy> anotherjesse: there is no core in the above proposal
20:33:12 <notmyname> an API "becoming official" would be up to the PTLs. look at how the project is being used and make the best choice for the project (?)
20:33:20 <anotherjesse> vishy: sorry, I meant making them official
20:33:41 <vishy> anotherjesse: there is no advantage for us, but it is good for the ecosystem
20:33:45 <heckj> damn, bad network day for me
20:33:47 <ttx> anotherjesse: if they don't see an advantage, they will stay non-official
20:34:06 <ttx> For example, I wouldn't gate on something non-official
20:34:09 <anotherjesse> ttx: what advantage for openstack
20:34:15 <ttx> so Tempest will be an official project (not core=
20:34:17 <ttx> )
20:34:26 <anotherjesse> not that I don't think they should exist
20:34:33 <anotherjesse> err, not that I think they shouldn't exist
20:34:34 <ttx> anotherjesse: gives us some amount of control.
20:34:52 <anotherjesse> ttx: we don't want it.  owning our official APIs are enough work
20:35:22 <anotherjesse> and being a cloud platform - instead of a cloud product - means we need to give control to people to write stuff on top of openstack
20:35:23 <mtaylor> yeah - and other people, such as canonical, who have vested interests in other apis can own those for their purposes
20:35:26 <ttx> anotherjesse: think devstack being used as a gate mechanism. Making it official forces devstack to play by our rules... and lets us influence it so that we can keep using it as a gate
20:35:33 <vishy> anotherjesse, ttx: perhaps that is exactly the definitition of official.  We feel it is an important enough use case to do some gating on it
20:35:51 <mtaylor> vishy: ++
20:35:54 <heckj> vishy: I can stand by that definition of official
20:35:58 <anotherjesse> ttx: and if devstack guys didn't want to play by those rules DVCS has easy ways of *OPENSTACK* adding process without limiting the external project
20:36:00 <ttx> I hate gating on something we don't control.
20:36:26 <ttx> anotherjesse: sure.
20:36:39 <mtaylor> dvcs solves all the worlds problems...
20:36:59 <vishy> anotherjesse: but that is just adding a dependency
20:37:05 <ttx> Personally I see our copre infrastructure as an official project too. Though we don'ty formally gate on it... it's THE GATE.
20:37:09 <anotherjesse> vishy: exactly
20:37:11 <ttx> core*
20:37:13 <vishy> anotherjesse: which is different from actively working on the code
20:37:22 <vishy> anotherjesse: and keeping it up to date
20:37:37 <anotherjesse> openstack will want to expose a stable interface to build things upon
20:37:57 <anotherjesse> if we break upstream projects like a http proxy api - then will probably are breaking LOTS of other people building custom integrations
20:38:07 <anotherjesse> but that doesn't mean we have to mark something as official
20:38:26 <anotherjesse> I want to know if we are breaking for users or use-cases
20:38:39 <anotherjesse> but that doesn't imply that those users/use-cases code needs to be official
20:38:42 <anotherjesse> we are an API
20:38:59 <anotherjesse> we need to support the users of the API regardless of "official"-ness
20:39:24 <mtaylor> but we can support them by testing the api... we don't have to test all of the users of the api, yeah?
20:39:33 <anotherjesse> and if we start blessing the high level projects on top of openstack we will end up being even more bureaucratic
20:39:43 <vishy> anotherjesse: from a community perspective I think there is a lot of perceived value to having some kind of official openstack support.
20:39:44 <anotherjesse> mtaylor: exactly - that is what we can focus on to help these guys
20:40:17 <vishy> anotherjesse: i don't know how you can avoid it. So either we do it via technical requirements or we punt let marketing people decide what to do with it
20:40:29 <anotherjesse> vishy: I'm saying you are looking at it backwards
20:40:29 <mtaylor> memcached has a great tool called memcapable that you can run against your api thing to ensure that your non-memcached implementation works
20:40:38 <ttx> vishy: yes, maybe "official" is not the best term to describe that state.
20:40:51 <anotherjesse> we should have APIs that 3rd parties can say they work with and have tests suites that show that they have integration
20:41:07 <soren> mtaylor: A memcached that just doesn't cache anything?
20:41:09 <mtaylor> if we're testing the api with something, and we can hand that something to people consuming the api that they can use to verify their side of things, then we're taking care of those consumers, right?
20:41:12 <mtaylor> soren: yup
20:41:19 <ttx> anotherjesse: so you would prefer 3rd party APIs being completely separate ?
20:41:20 <soren> mtaylor: I'm not sure what the equivalent would be here?
20:41:26 <vishy> anotherjesse: that still makes no sense, they run the test suite themselves and tell us it works
20:41:37 <anotherjesse> vishy: yes
20:41:37 <vishy> anotherjesse: and we say, awesome, use our logo?
20:42:57 <anotherjesse> vishy: we say "here are the branding guidelines" - if you are compatable with openstack 2.0 api, then you say that
20:42:58 <vishy> anotherjesse: that still fits perfectly in what i was proposing, just that we have some way to specify officialness
20:43:10 <anotherjesse> just like software boxes show mac  / windows logos on the side of the box ...
20:43:13 <ttx> vishy: did you raise the thread on this on the ML ? I haven't seen it, but I'm a bit behind on ML processing
20:43:17 <vishy> we can say: must pass these tests
20:43:24 <vishy> ttx: no i did not
20:43:35 <anotherjesse> vishy: we can say that "this logo means you pass the tests" - but we don't run the tests
20:43:37 <anotherjesse> they do
20:43:50 <vishy> anotherjesse: that's fine
20:43:52 <anotherjesse> and if they lie then users will be pissed - at both parties probably
20:44:00 <anotherjesse> ;)
20:44:22 <vishy> anotherjesse: that is a hard sales pitch when the tests don't even exist though
20:44:29 <vishy> anotherjesse: who's going to write them?
20:44:51 <mtaylor> won't tempest wind up being an API compat test?
20:44:55 <ttx> vishy: I think some emails would go a long way to crystalize the different options before we can confront them
20:44:59 <mtaylor> if it's testing everything black box via the APIs?
20:45:21 <ttx> vishy: we seem to still struggle with the breadth of the topic :)
20:45:23 <vishy> mtaylor: that doesn't give 3rd party apis or plugins any pth forward.
20:45:33 <mtaylor> vishy: hrm. k. fair
20:45:35 <devcamcar_> mtaylor: I don't think that's a stated goal of tempest
20:45:51 <anotherjesse> vishy: if we don't write them (as in all of openstack) then how is integrating them directly into the project any better
20:45:59 <anotherjesse> we end up with a larger project with more brittle integrations
20:46:12 <vishy> anotherjesse: I don't want to integrate them into the project
20:46:29 <anotherjesse> vishy: I don't want to be the gatekeeper of the labeling of "works with openstack"
20:46:35 <vishy> anotherjesse: I want to be able to tell the authors of these that there is a specific process by which they can become "part of openstack".
20:46:53 <anotherjesse> I want people to compete for adoption rather than wanting to be blessed
20:47:00 <vishy> because otherwise there is too much perceived value in core
20:47:08 <vishy> and so they will keep pushing to be in core
20:47:11 <ttx> vishy: +1
20:47:26 <ttx> Part of the "official" concept is to distract people from bwatning "core or nothing"
20:47:34 <ttx> wanting*
20:47:47 <anotherjesse> vishy: my point is being part of openstack needs to be bigger than just being in core …  I can be part of linux by writing a software project that distros include
20:48:00 <anotherjesse> I don't have to get the kernel's blessing to write a game for linux
20:48:05 <anotherjesse> or for distros to include it
20:48:20 <vishy> anotherjesse: I understand your point, but they don't
20:48:45 <ttx> anotherjesse: in that case we are talking external APIs. Which is like proposing a new executable format for the kernel.
20:48:49 <anotherjesse> vishy: k - so if we agree that we should be going for an ecosystem that is market driven, then lets work towards enabling that
20:49:08 <ttx> anotherjesse: not like adding a new game.
20:49:33 <anotherjesse> ttx: not true, since it is a compatability layer.  for performance reasons you might want to be in the kernel but there is no reason to not implement it in userland
20:49:58 <soren> anotherjesse: It may or may not be a compatibility layer.
20:50:04 <soren> anotherjesse: That's the point, isn't it?
20:50:14 <ttx> soren: +1
20:50:18 <vishy> anotherjesse: you can do a new filesystem in userland as well
20:50:20 <soren> anotherjesse: It's not necessarily just a proxy. Just like the EC2 API implementation in Nova now.
20:50:29 <vishy> anotherjesse: but there are valid reasons for proposing it into the kernel
20:50:40 <anotherjesse> soren: we are proposing that the way it is in  nova right now is wrong
20:51:05 <anotherjesse> that you can do it that way but we aren't mature enough to do both http API and internal api with stability of interfaces
20:51:12 <devcamcar_> that ec2 API needs to be a proxy to openstack API
20:51:22 <devcamcar_> correct?
20:51:33 <vishy> devcamcar_: undetermined
20:51:50 <devcamcar_> what tests do you have to pass?
20:51:51 <vishy> devcamcar_: either a proxy or communicating through defined rpc api
20:52:14 <ttx> again, EC2 is a historical corner case, let's not turn this into a EC2 API question
20:52:35 <vishy> ttx: I don't know if we can consider it that while it still in the code base
20:52:37 <ttx> the question is: what do we do for the "other" cloud standard APIs
20:52:45 <soren> anotherjesse: I know that's what being discussed, but the point remains that it doesn't have to be the proxy model. Not eveyone thinks the proxy model is super awesome.
20:52:52 <devcamcar_> ttx: yep, not trying to, but it's best example we have atm
20:52:54 <soren> anotherjesse: Present company included.
20:53:38 <soren> ttx: Do you think we might keep the EC2 API beucase it's been grandfathered in, but have other policies for new API's that come along?
20:53:39 <ttx> soren: the alternative is a clean/versioned internal API coupled with a plug-in system, I think
20:53:40 <anotherjesse> soren: we could go with the linux codebase model where everything is in core - including 3rd party stuff
20:53:54 <devcamcar_> soren: my question is how do you measure compatibility for a non proxy api?
20:53:54 <anotherjesse> soren: but our gerrit process makes it cumbersome already
20:54:11 <soren> devcamcar_: Compatiblity with what? (and why do I care?)
20:54:23 <anotherjesse> soren: for instance in the volume code we have drivers that 3rd parties maintain - and we basically have to blind accept them as they come through
20:54:28 <ttx> soren: yes
20:54:32 <mtaylor> anotherjesse: for sake of argument, that can be fixed (and remind me to chat with you about some ideas we had)
20:54:39 <soren> ttx: Ok.
20:55:01 <ttx> soren: but the decision we take here would definitely influence where the EC2 API could go in the future.
20:55:02 <devcamcar_> soren: exactly my point. what are we even trying to measure here?
20:55:05 <soren> anotherjesse: I'm not sure where you're going?
20:55:15 <soren> devcamcar_: Are we trying to measure something?
20:55:17 <soren> :)
20:55:29 <anotherjesse> soren: if there are snippets of code which could live on their own, they should
20:55:34 <anotherjesse> since our process sucks for those
20:55:50 <soren> We could make the OpenStack API external, too.
20:56:03 <anotherjesse> soren: that was our original proposal a year ago but we lost that battle
20:56:09 <heckj_> I'm having the worst damn day with networking
20:56:09 <devcamcar_> we have 4 mins
20:56:37 <devcamcar_> anything we can vote on thats tangible? :)
20:56:39 <heckj_> [tempchair]: we have 4 minutes left. I propose that we synthesize this conversation, post on the mailing list, and discuss further there. Then reconvene next week here to formally decide
20:56:44 <anotherjesse> devcamcar_: no
20:56:48 <ttx> heckj: you should endmeeting as soon as you recover "heckj"
20:57:00 <anotherjesse> we should enable optional components to thrive with their own coding process, release process and marketing demand process …  or figure out how to fix our processes to work with those
20:57:13 <anotherjesse> right now our processes don't work well for subgroups
20:57:14 <mtaylor> anotherjesse: or both
20:57:37 <heckj_> .
20:57:56 <anotherjesse> and adding more "blessings" where people either land-rush or agreements with project seems less than ideal
20:58:05 <anotherjesse> which is what the process has been so far
20:58:23 <ttx> heckj: you should endmeeting
20:58:23 <soren> heckj_: "heckj" just left. You can /nick heckj
20:58:28 <ttx> while you can
20:58:39 <heckj> finally!
20:58:42 <soren> Oh, look! It's heckj !
20:58:45 <heckj> #endmeeting