| *** emagana has joined #openstack-tc | 01:23 | |
| *** emagana has quit IRC | 01:28 | |
| *** emagana has joined #openstack-tc | 03:08 | |
| *** emagana has quit IRC | 03:20 | |
| *** david-lyle has quit IRC | 04:38 | |
| openstackgerrit | Artur Basiak proposed openstack/governance master: Add monasca-events-api https://review.openstack.org/491401 | 08:49 |
|---|---|---|
| openstackgerrit | Duong Ha-Quang proposed openstack/governance master: Add kolla-ansible-plugins https://review.openstack.org/491413 | 09:25 |
| *** cdent has joined #openstack-tc | 09:28 | |
| *** sdague has joined #openstack-tc | 09:47 | |
| *** dtantsur|afk is now known as dtantsur | 09:48 | |
| openstackgerrit | Duong Ha-Quang proposed openstack/governance master: Add kolla-ansible-plugins https://review.openstack.org/491413 | 10:06 |
| openstackgerrit | Witold Bedyk proposed openstack/governance master: Update goals status for Monasca https://review.openstack.org/491458 | 11:41 |
| *** cdent has quit IRC | 12:02 | |
| *** cdent has joined #openstack-tc | 12:33 | |
| cdent | let the great CD debates begin! | 12:47 |
| dims | #those_who_deploy_from_trunk_please_raise_your_hands | 12:54 |
| * fungi deploys from trunk... in the gate tens of thousands of times a day? | 13:12 | |
| fungi | needless to say, the rate of success there is about what you'd expect ;) | 13:12 |
| *** gcb has joined #openstack-tc | 13:28 | |
| *** lbragstad has joined #openstack-tc | 13:36 | |
| dhellmann | we should qualify that question by asking for folks who deploy from trunk and expect API stability | 13:37 |
| dhellmann | because if we have a lot of trunk deployers who don't care about api stability, that's also an interesting answer | 13:37 |
| cdent | our lack of reliable way to ask questions is problematic | 13:39 |
| dhellmann | I understand the foundation's reluctance to have lots of separate surveys going, but we do need to work something out. | 13:42 |
| dhellmann | in this case, based on past user survey results, adding a question there is likely to produce the result that extremely few (if any) users are deploying from trunk | 13:43 |
| dhellmann | and we have no way of knowing if that information is accurate, or biased based on survey respondants | 13:43 |
| cdent | we could break some stuff and wait for people to come out of the woodwork | 13:44 |
| dhellmann | you would expect anyone deploying from trunk to be relatively well engaged in the community, though, so maybe asking on the operators list would give us useful answers? | 13:44 |
| dhellmann | cdent : we could make nova's API for creating a server return a message | 13:45 |
| dhellmann | "please contact the TC about your trunk deployment by emailing ..." | 13:45 |
| cdent | I’m no longer convinced that that expectation holds. That was the main thrust of Josh’s message: if these people exist how come we can’t see them? | 13:45 |
| cdent | I like it! | 13:46 |
| dhellmann | I think we probably have 1-2 very large older deployments still doing CD from upstream. Whether that's trunk, and how far behind, was never really clear | 13:46 |
| dhellmann | RAX is the one usually cited when this discussion has come up in the past | 13:46 |
| dhellmann | speaking of, we might find some info by reviewing mailing list archives. at least some names of companies to contact directly. | 13:47 |
| dhellmann | I guess it depends on how much time we want to invest into research. | 13:47 |
| cdent | yeah, I’m not sure. To some extent there are enough habits burned in, that changing would be very disruptive. | 13:48 |
| *** marst has joined #openstack-tc | 14:09 | |
| sdague | I think there is also a discipline in using the CD model to ensure the release to release path is smooth, because if you solve the any to any+ problem, you get the N -> N+1 problem for more or less free | 14:24 |
| sdague | which makes the release sanity checking a lot smoother | 14:24 |
| dhellmann | right, I think it makes sense for us to keep doing the upgrade testing in our CI. I'm just not sure it makes sense to spend lots of time agonizing over API stability between official releases. | 14:25 |
| sdague | yeh, I tend to think it's just discipline to think through all the API changes as if they were going to be in the release, as it means you don't have to consider it as part of the other things you have to think about at release time | 14:34 |
| cdent | yeah, for the most part, the habits are pretty useful | 14:40 |
| cdent | but I suppose there might be times when they could be usefully broken | 14:40 |
| * cdent shrugs and goes home | 14:40 | |
| sdague | yeh, everything needs an exception criteria. It's just norms actually let you think less about each decision. Always do it this way unless there are exceptional circumstances. We already put in the exception explicitly about closing security issues. | 14:41 |
| smcginnis | The discipline is good, but I still have yet to see where we've ever stated officially that we would support CD. | 14:44 |
| smcginnis | I've seen it stated plenty of times for reasons why we can't just quickly fix a mistake though. | 14:44 |
| *** cdent has quit IRC | 14:44 | |
| *** gcb_ has joined #openstack-tc | 14:45 | |
| *** gcb has quit IRC | 14:47 | |
| sdague | smcginnis: yeh, I think it is all part of the culture of the "before times" | 14:49 |
| sdague | where everything wasn't written down | 14:50 |
| smcginnis | sdague: Which is fine. (My take too btw). But if it is still something we want to adhere to, I want the outcome to be that we declare that publicly in writing. | 14:51 |
| *** gcb_ has quit IRC | 14:52 | |
| sdague | smcginnis: sure, that's fair | 14:54 |
| smcginnis | Or more accurately, I want us to declare that we don't care unless it's an official release, but either way, I want it more explicit. ;) | 14:57 |
| *** emagana has joined #openstack-tc | 15:02 | |
| *** david-lyle has joined #openstack-tc | 15:04 | |
| sdague | yeh, explicit is fine | 15:05 |
| *** cdent has joined #openstack-tc | 15:08 | |
| *** emagana has quit IRC | 16:01 | |
| *** emagana has joined #openstack-tc | 16:01 | |
| *** emagana has quit IRC | 16:05 | |
| ttx | also worth noting, we already consider that a vulnerability introduced in master but not released is not worth an OSSA | 16:07 |
| ttx | so we already have different commitments for people tracking master and people tracking releases | 16:08 |
| smcginnis | ttx: Good point. | 16:09 |
| *** Qiming has quit IRC | 16:18 | |
| *** Qiming has joined #openstack-tc | 16:20 | |
| fungi | also, i don't every remember openstack officially stating it would "support" deployments from arbitrary points in master branches, or continuous deployment. lots of _individuals_ in the project said they kept the possibility that people might do that in mind, which is cool... i'm all for enabling additional use cases where we can without overtaxing ourselves. but a declaration of support for that seems | 17:57 |
| fungi | far too strong | 17:57 |
| fungi | s/every/ever/ | 17:57 |
| fungi | there's lots of ways people can and do use openstack software, but just because there are people doing things with it doesn't necessarily mean we can officially support all those things | 17:59 |
| fungi | the recent statement on postgresql seems like a relevant example | 17:59 |
| fungi | you might be able to use openstack with postgresql, and upstream developers aren't actively trying to make that harder for you, but we also can't promise to support it | 18:00 |
| fungi | i do recall some teams in the past pushing for continuous deployment support on their own, but openstack as a whole hasn't prioritized that use case that i've ever seen | 18:01 |
| fungi | and the organizations i know of who do that or did it in the past coupled their deployment process with an entire shadow deployment and test infrastructure where they could shake out/postpone breaking changes and integrate their local forks (because they pretty much always have them) before a commit from upstream master ever trickles into production | 18:04 |
| *** emagana has joined #openstack-tc | 18:10 | |
| *** dtantsur is now known as dtantsur|afk | 18:12 | |
| sdague | yeh, well you have to keep CDing in mind when writing software so that you have continuous transitions for upgrade | 18:20 |
| smcginnis | fungi: ++ | 18:52 |
| sdague | fungi: you clearly need your own local CD testing infrastructure if you are going that way | 18:59 |
| sdague | but, if the upstream software isn't even trying to work in a CDing manner, you can't do local CD | 19:00 |
| fungi | sure, just saying i think it's so far been possible by accident/conscience, not because we declared officially to provide support for doing that | 19:01 |
| fungi | teams have perhaps been trying to avoid gratuitously breaking it | 19:02 |
| dims | fungi : so write down what we are doing now already? (tell projects to continue to honor the CD target) | 19:18 |
| cdent | dims: do all projects do that or just some? | 19:24 |
| smcginnis | Maybe we need a follows-cd tag. | 19:27 |
| dhellmann | we're talking about 2 different aspects of CD, though, right? | 19:28 |
| dhellmann | sdague correctly points out that upgrades are important | 19:28 |
| smcginnis | I think you need to keep D in mind in order to have upgrades, but the C part is another apect. | 19:28 |
| smcginnis | *aspect | 19:28 |
| dhellmann | the thing I'm more worried about is artificially locking us into supporting an API that isn't fully baked | 19:28 |
| smcginnis | dhellmann: ++ | 19:28 |
| dhellmann | or where some bug slips through, as in the keystone case | 19:28 |
| smcginnis | And DB migrations. | 19:28 |
| dhellmann | I think we can relax the rules on the latter, while continuing to support the former | 19:29 |
| dhellmann | db migrations would be part of the upgrade testing, right? | 19:29 |
| dhellmann | or are you saying it's OK to change a migration before a release? | 19:29 |
| sdague | dhellmann: I hope not | 19:29 |
| smcginnis | dhellmann: Yes, but we've had cases where we've realized something wasn't the way we want, so rather than changing the migration to make it right, we've had to then add yet another migration because "someone might have picked that up already". | 19:30 |
| smcginnis | I would like to be able to just fix the intial migration rather than keep around crud just because of a "maybe". | 19:30 |
| dhellmann | that would make our upstream upgrade testing a bit more complex | 19:30 |
| dhellmann | or would it? | 19:30 |
| smcginnis | Or have it clear that we should fully support that "maybe" and be strict about it. | 19:31 |
| dhellmann | do we deploy from the last full release, then test upgrading to master? | 19:31 |
| smcginnis | dhellmann: I don't think it would. | 19:31 |
| smcginnis | Last full release in grenade. | 19:31 |
| dhellmann | oh, ok, then it seems less risky | 19:31 |
| smcginnis | That's the boundary I would prefer to operate on. Or at least last milestone "deliverable". | 19:31 |
| sdague | right, so here is the thing, if we think we ever want CD to be a thing we do need to bake some of these ideas in that once things are released they are real | 19:32 |
| dhellmann | milestones seems reasonable, since I could see someone doing test deployments with those | 19:32 |
| sdague | I do get that's harder engineering and requires more thought | 19:32 |
| dhellmann | sdague : right, but I think we're talking different definitions of "released" (committed vs. tagged) | 19:32 |
| smcginnis | sdague: I don't think we have agreement that we all "want CD to be a thing" though. | 19:32 |
| smcginnis | Big C continuous. | 19:32 |
| dhellmann | there's also a question of how far to extend that, to just deployer facing changes or all the way to end-user facing changes | 19:33 |
| dhellmann | which is why I was pointing out the difference we might want between upgrade testing and API testing | 19:33 |
| sdague | I'm not sure why we think milestone tags are special at this point. We don't have soft freezes for those any more. | 19:35 |
| smcginnis | Which is why I would rather be between releases really. Especially since that's all we test with grenade. | 19:35 |
| smcginnis | I would much rather support Ocata->master than master~1->master | 19:36 |
| sdague | I think that if we ever want to close the gap with out of tree features and providers being 2 - 5 release behind upstream, we need to encourage the CD model | 19:36 |
| sdague | because no one is going to ocata -> master if they can't master -> pike | 19:37 |
| sdague | which means no one is going to do anything other than consume stable branches, 3 - 9 months after we publish them | 19:37 |
| dhellmann | I'm not sure how I'm seeing why suggesting that APIs only have to be stable on release boundaries breaks anyone's ability to upgrade. | 19:38 |
| sdague | which puts their deployed experience a full year out of step usually with where the source code is, which makes it quite hard to integrate operators in | 19:38 |
| sdague | because if they deploy at a random git commit, that's the interface their users start writing to | 19:38 |
| sdague | and then if they upgrade away from it, boom | 19:38 |
| dhellmann | right, well, I think it's a bad idea to encourage that | 19:38 |
| dhellmann | maybe I just think CD is a bad delivery model for a stack this complex; I don't know | 19:39 |
| dhellmann | I'm looking for ways to introduce some slack into the upstream processes | 19:39 |
| smcginnis | dhellmann: Agree 100% | 19:39 |
| cdent | maybe the stack being this complex is the real issue? ;)/2 | 19:41 |
| smcginnis | cdent: Well, there is that. :) | 19:41 |
| dhellmann | yes, that's completely fair, too | 19:41 |
| sdague | personally, part of what made me passionate about OpenStack is that it functioned in an already ready to deploy mode | 19:42 |
| sdague | if we're taking that off the board, that would make me quite sad | 19:42 |
| cdent | there’s a different between ready to deploy and continuously deployable from master, isn’t there? | 19:43 |
| sdague | s/already/always/ | 19:43 |
| sdague | not if you want to support upgrades | 19:43 |
| smcginnis | I don't agree with that statement. | 19:43 |
| sdague | if you say you can deploy from any commit, but you might be stuck, that's not really saying you can deploy from any commit and have long term working software | 19:43 |
| cdent | which “that” smcginnis ? | 19:44 |
| smcginnis | Sometimes it's easier to support upgrades by not locking yourself into a mistake. | 19:44 |
| sdague | smcginnis: easier for who? | 19:44 |
| smcginnis | So I absolutely do not see CD off of master as ar equirement for upgradability. | 19:44 |
| smcginnis | sdague: All involved. | 19:44 |
| dhellmann | well, we wouldn't be saying "you can deploy from any commit" | 19:44 |
| dhellmann | we would be saying "we work toward release boundaries, pick one" | 19:45 |
| sdague | smcginnis: I don't understand how it is easier for end users to consume if API features come and go | 19:45 |
| smcginnis | sdague: They don't come and go if they go off of release boundaries. | 19:45 |
| dhellmann | sdague : you're assuming a configuration that is something we would specifically not support | 19:45 |
| sdague | it's easier for developers of openstack, because they can fix more mistakes | 19:45 |
| smcginnis | By all means test what's on master and don't wait until release to even look at it. | 19:45 |
| sdague | dhellmann: I'm saying the grey state right now | 19:45 |
| smcginnis | But don't put that on your end users. | 19:45 |
| sdague | smcginnis: how do you test it without having consumers | 19:46 |
| dhellmann | sdague : well, I'm saying we would not encourage users to deploy from untagged commits *and then* we would change the policy to say API changes are only stable at tagged releases. | 19:46 |
| sdague | dhellmann: right, I understand what you are proposing. I think we as a community loose a lot in the process | 19:46 |
| sdague | and I'm trying to state where we currently stand | 19:46 |
| dhellmann | ok, well, it sounds like you're backing that with a justification of if they do something other than what we want to support we're somehow doing them a disservice | 19:47 |
| sdague | I'm saying that if you relax the dev teams striving for a CD able upstream tree, it's not really possible to retrofit it later | 19:48 |
| sdague | it's like retrofiting pypy support or something | 19:48 |
| dhellmann | I'm not sure if I agree or disagree, but I'm also not sure that's a goal I would have. | 19:49 |
| cdent | sdague: so part of your concern is that if we give it up we can’t get it back? | 19:50 |
| sdague | cdent: yes | 19:50 |
| cdent | a fair concern | 19:50 |
| sdague | and, that even if the number of folks doing it is small, that's incredibly valuable feedback to the dev teams, because they can expose issues before we hit stable releases | 19:51 |
| cdent | I’m struggling to develop a solid opinion | 19:51 |
| sdague | and it would be great to have more folks in that camp because it also gives us better stable releases | 19:52 |
| cdent | I wish we had a stronger more visible set of participants | 19:52 |
| sdague | cdent: I do as well | 19:52 |
| sdague | but we will never get those folks if we walk away from the goal of supporting CDing | 19:53 |
| sdague | both RAX and HP in their prime for public cloud were trunk chasing. They both stopped when they effectively decommitted from the project. | 19:54 |
| sdague | we went through a lull | 19:54 |
| sdague | now we're getting a lot more public cloud folks | 19:55 |
| sdague | and I hope / expect more of them will get on this bad wagon | 19:55 |
| sdague | it fits closer to the public cloud model where you've got skilled folks deploying via ansible with a smallish number of custom site fixes. It's not product turnkey stuff. | 19:58 |
| cdent | maybe the foundation should make a trystack-like thing which is CD’d | 19:58 |
| cdent | free for 24 hour vms, ripe for abuse! | 19:59 |
| fungi | in the latest case, i think it's more that if you as a cd'er start relying on a feature which hasn't seen a release yet, then you should be mindful the behavior of that feature may still change prior to release and prepare accordingly | 20:05 |
| cdent | fungi: the problem with that model is that a user may not know they are talking to a cloud that is cd’ing | 20:07 |
| cdent | (although I’m not sure what you mean by “the latest case”?) | 20:07 |
| fungi | can we mark new interfaces/features "experimental" somehow before they hit an official release? | 20:07 |
| fungi | also "continuously deployable from master" isn't necessarily the same thing as "continuously upgradable from any arbitrary point in master" | 20:08 |
| smcginnis | I think there was some talk of using microversion x.9999 to do that. | 20:08 |
| cdent | smcginnis: where x.9999 is > ‘latest’? | 20:08 |
| fungi | also, if we have to start supporting features and not breaking regressing them as soon as they hit master, we're going to see a proliferation of "feature branches" just so devs have an out to avoid anyone depending on something which is still in flux | 20:09 |
| smcginnis | cdent: Something like that. I just remember hearing something about it in passing. I'm not sure about practical implementation. | 20:09 |
| sdague | fungi: why? | 20:09 |
| sdague | fungi: this has been the norm state for the last 7 years | 20:09 |
| fungi | will we then declare hopping between different feature branches or upgrading back and forth between master and feature branches is supported once cd'ers realize they can't get the bleeding edge features they want from master any longer? | 20:10 |
| fungi | i wonder where that road ever ends, really | 20:10 |
| sdague | fungi: except, we're already on that road | 20:10 |
| sdague | and we've been for a long time, and we didn't have that | 20:10 |
| sdague | what we got was the invention of microversions so that you could stay on that road and still signal to your users what's going on | 20:10 |
| smcginnis | Maybe we do need to start using feature branches more if we're going to actually declare this as a principle. | 20:10 |
| cdent | don’t we have experience that says that feature branches often cause more trouble than they are worth | 20:11 |
| cdent | however, I think it is important that we don’t let the experience of the bigger more complex projects (e.g. nova) to necessary implicate other less complex projects | 20:12 |
| fungi | sdague: i don't see where making a tc resolution that to be an official openstack service you must provide support for anyone who wants to run and upgrade between changes on your master branch is "the norm for the last 7 years" | 20:13 |
| fungi | we have devs who have tried to make that feasible as much as possible without compromising their ability to advance their platforms | 20:13 |
| sdague | fungi: nope, it was just the baked in culture that I showed up | 20:14 |
| *** mriedem has joined #openstack-tc | 20:14 | |
| fungi | which is more what seems like "the norm for the last 7 years" | 20:14 |
| dhellmann | I think the reality was that some teams adopted that stance more than others | 20:14 |
| sdague | dhellmann: sure, that's fair | 20:14 |
| fungi | similarly, as soon as we land methods/functions in an unreleased branch of a library, do we need to do full version bumps if we alter their behavior before the next tag? | 20:15 |
| dhellmann | we only consume libraries at releases, so we have not enforce that | 20:15 |
| sdague | fungi: I think that the set of constraints you give yourself for it isn't all that constraining. But that's just personal opinion. It's about sane deprecation, data model migrations, and having a progressively changing API | 20:15 |
| dhellmann | *d | 20:15 |
| fungi | even if it's to fix an undesirable behavior we didn't catch in review? | 20:15 |
| sdague | dhellmann: we release libraries more than every 6 months | 20:15 |
| dhellmann | sdague : we could release services more often, too | 20:16 |
| fungi | well, "we" aren't the only ones consuming the libraries we produce either | 20:16 |
| dhellmann | and we still only have stable branches for libraries every 6 months | 20:16 |
| fungi | so if someone wants to continuously deploy any of our libraries, the same concerns come up as far as i can see | 20:16 |
| dhellmann | we haven't supported CD of the libraries since we switched oslo off of alpha releases | 20:17 |
| fungi | i'm curious how it was "supported" prior to that | 20:17 |
| sdague | fungi: one key difference is you can pip install the libraries | 20:17 |
| dhellmann | although I'm not sure that's written down | 20:17 |
| sdague | which you can't do for the projects | 20:17 |
| dhellmann | you can pip install some of them | 20:17 |
| sdague | dhellmann: to a working state? | 20:18 |
| fungi | you can pip install the services too, you just need to pip install them from a local tarball/path and do some steps afterward to get configuration and data files into the right places, right? | 20:18 |
| dhellmann | sdague : can you install an RPM and get a working nova? | 20:18 |
| sdague | fungi: ok, I guess I | 20:18 |
| sdague | dhellmann: you get a lot closer for sure | 20:19 |
| mriedem | rpm lays down your config files and creates nova user/group | 20:19 |
| fungi | you can't pip install most of the services directly off pypi, but that doesn't mean that you can't use pip to install them at all | 20:19 |
| sdague | anyway, I think we're sort of in circular camp now | 20:19 |
| mriedem | and rootwrap etc | 20:19 |
| fungi | revisiting the continuous deployment argument, it seems that maybe api behaviors are "special" in this regard because that api has a version number independent of the software release version, and (currently) no way to mark api features under development as experimental | 20:30 |
| fungi | whereas features in a library have only one version, and the version for an unreleased library is clearly an experimental version number | 20:31 |
| mriedem | there is actually a status with the versions https://developer.openstack.org/api-ref/compute/#list-all-major-versions | 20:32 |
| mriedem | nova doesn't have EXPERIMENTAL anymore but i think glance v3 had/have it | 20:32 |
| mriedem | https://developer.openstack.org/api-ref/image/versions/index.html#api-versions | 20:34 |
| mriedem | "id": "v2.6", "links": [ { "href": "http://glance.openstack.example.org/v2/", "rel": "self" } ], "status": "EXPERIMENTAL" | 20:34 |
| mriedem | i don't know how glance moves something out of experimental | 20:34 |
| sdague | I think the project history with experimental was that it turns out to not be incredibly useful in practice | 20:34 |
| mriedem | i'm not advocating it, | 20:34 |
| sdague | because by the time your consumers deploy things to experiment with them, you've already committed to it or deleted it | 20:34 |
| mriedem | just saying, there was a system before microversions | 20:35 |
| sdague | because they are releases behind you | 20:35 |
| sdague | mriedem: yeh, I wasn't saying you were, I was providing context about why it went away | 20:35 |
| mriedem | i hear that you're saying that i wasn't saying something, and am pleased. | 20:38 |
| * mriedem investment in https://www.amazon.com/Couples-Guide-Communication-John-Gottman/dp/0878221271 is already paying for itself | 20:40 | |
| cdent | mriedem++ | 20:43 |
| *** mriedem has quit IRC | 20:56 | |
| fungi | just wondering how you can have an intermediate state between versions that't clearly unsupported, and with api versioning we don't presently have a mechanism to indicate to end users that they shouldn't rely on a feature (specifically because continuous deployment may be exposing that to them when it's not yet fully baked) | 21:02 |
| fungi | an alternative, perhaps, is to guard in-progress api features behind testing-mode switches that we tell operators (via documentation) not to turn on | 21:04 |
| fungi | allowing us to run some jobs with those experimental features enabled while avoiding cd'ers from exposing them to end users before we're ready to support their behaviors | 21:04 |
| dims | fungi : ++ (k8s uses --feature-gates for example - https://kubernetes.io/docs/admin/kube-apiserver/ with clearly marked features and ALPHA/BETA designations) | 21:20 |
| *** marst has quit IRC | 22:36 | |
| *** sdague has quit IRC | 23:21 | |
| *** cdent has quit IRC | 23:22 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!