| *** kumarmn has joined #openstack-tc | 00:03 | |
| *** kumarmn has quit IRC | 00:28 | |
| *** harlowja has quit IRC | 00:45 | |
| *** kumarmn has joined #openstack-tc | 00:58 | |
| *** kumarmn has quit IRC | 01:01 | |
| *** kumarmn has joined #openstack-tc | 01:01 | |
| *** mriedem has quit IRC | 01:06 | |
| *** mriedem has joined #openstack-tc | 01:10 | |
| *** kumarmn has quit IRC | 01:20 | |
| *** kumarmn has joined #openstack-tc | 01:21 | |
| *** kumarmn has quit IRC | 01:22 | |
| *** kumarmn has joined #openstack-tc | 01:22 | |
| *** liujiong has joined #openstack-tc | 01:27 | |
| *** kumarmn has quit IRC | 01:36 | |
| *** kumarmn has joined #openstack-tc | 01:36 | |
| *** kumarmn has quit IRC | 01:36 | |
| *** kumarmn has joined #openstack-tc | 01:37 | |
| *** kumarmn has quit IRC | 02:07 | |
| *** kumarmn has joined #openstack-tc | 02:15 | |
| *** kumarmn has quit IRC | 02:21 | |
| *** kumarmn has joined #openstack-tc | 02:22 | |
| *** kumarmn has quit IRC | 02:26 | |
| *** openstackstatus has quit IRC | 02:32 | |
| *** purplerbot has quit IRC | 02:32 | |
| *** amrith has quit IRC | 02:32 | |
| *** ChanServ has quit IRC | 02:32 | |
| *** ChanServ has joined #openstack-tc | 02:34 | |
| *** barjavel.freenode.net sets mode: +o ChanServ | 02:34 | |
| *** openstackstatus has joined #openstack-tc | 02:36 | |
| *** purplerbot has joined #openstack-tc | 02:36 | |
| *** amrith has joined #openstack-tc | 02:36 | |
| *** barjavel.freenode.net sets mode: +v openstackstatus | 02:36 | |
| *** lbragstad has quit IRC | 03:44 | |
| *** kumarmn has joined #openstack-tc | 04:08 | |
| *** kumarmn has quit IRC | 04:13 | |
| *** rosmaita has quit IRC | 04:13 | |
| *** kumarmn has joined #openstack-tc | 04:30 | |
| *** mriedem has quit IRC | 04:49 | |
| *** kumarmn has quit IRC | 04:53 | |
| *** kumarmn has joined #openstack-tc | 04:53 | |
| *** kumarmn has quit IRC | 05:21 | |
| *** kumarmn has joined #openstack-tc | 05:21 | |
| *** kumarmn has quit IRC | 05:23 | |
| *** kumarmn has joined #openstack-tc | 05:24 | |
| *** kumarmn has quit IRC | 05:29 | |
| *** openstackgerrit has quit IRC | 06:47 | |
| *** liujiong has quit IRC | 08:31 | |
| *** jpich has joined #openstack-tc | 09:02 | |
| *** jpich has quit IRC | 10:22 | |
| *** kumarmn has joined #openstack-tc | 10:25 | |
| *** dtantsur|afk is now known as dtantsur | 10:25 | |
| *** kumarmn has quit IRC | 10:29 | |
| *** openstackgerrit has joined #openstack-tc | 11:17 | |
| openstackgerrit | Dai Dang Van proposed openstack/governance master: Update policy goal for watcher https://review.openstack.org/527299 | 11:17 |
|---|---|---|
| *** cdent has joined #openstack-tc | 12:00 | |
| cdent | anybody have the cliff-notes version of the "longer development cycles" thread? | 12:21 |
| *** kumarmn has joined #openstack-tc | 12:25 | |
| *** kumarmn has quit IRC | 12:30 | |
| ttx | cliff-notes version? | 12:35 |
| cdent | "cliff-notes" are a study aid in the US, usually for novels that students don't want to read | 12:38 |
| cdent | https://www.cliffsnotes.com/literature/s/the-sound-and-the-fury/book-summary | 12:39 |
| *** rosmaita has joined #openstack-tc | 12:49 | |
| ttx | ah | 13:09 |
| ttx | I can summarize | 13:09 |
| ttx | cdent: People generally agree with the premise, which is that a lot of developers, especially the part-time ones, are struggling with our rhythm of $stuff | 13:10 |
| ttx | That said, some people disagree that the proposed solution would actually help in that regard | 13:10 |
| cdent | ttx, I was mostly joking; I'm reading everything now | 13:10 |
| ttx | There are concerns around the message it sends. Also concerns around the missed-train effect that would get more significant if we make the gap between releases that people want be one year | 13:12 |
| ttx | Some concern that it would result in less facetime as people would not want to do midcycles again | 13:13 |
| ttx | (that one is kinda weird, and comes from the same people who complained that the PTG would replace the midcycles) | 13:13 |
| ttx | (I read it as: stop changing the events, as it's always a good reason employers use to reduce their travel strategy | 13:15 |
| ttx | I also sense party lines between old-timers who are not really happy with it and new-comers/smaller-players who seem generally more in favor | 13:16 |
| cdent | thus far what I've read in the thread I'm not seeing much response from people who are new-timers to old projects | 13:18 |
| ttx | Some concerns around reducing cross-project collaboration, too | 13:18 |
| ttx | cdent: Some concerns around what this would mean for stable branches / number of supported for how long | 13:20 |
| ttx | anyway, open to suggestions. This one is definitely not consensual | 13:21 |
| ttx | I just thought it better to be discussed openly rather than privately | 13:21 |
| ttx | Even if that's causing me a bit of ad-hominem on Twitter | 13:22 |
| cdent | It is good that it is happening in the open. I'm still working out my reaction in my head, but initially I'm having one of my standard responses: we should do more analysis before design. | 13:28 |
| ttx | I find the tangent on "if we had more independent components that would not be that much of a problem" | 13:30 |
| ttx | interesting | 13:30 |
| cdent | I don't think that's a tangent. That's an effort to get to the underlying causes, which is what I mean by analysis before design | 13:33 |
| cdent | There are, however, likely several underlying causes, not just tight coupling. | 13:34 |
| cdent | If we're going to consider making such a big change, we might consider ways to make the change an actual change, in the guts, not just in timing. | 13:35 |
| ttx | I consider it a tangent because I don't think we can solve it before we die from contributor attrition, if we don't find a way to make part-time contributors more comfortable | 13:36 |
| cdent | then in that case, isn't it more important to talk about "contributor attrition" than cycle timing? | 13:37 |
| ttx | well the proposal talks about it. It's about making sure we can embrace part-time contributors, because we can't rely only on full-time people | 13:38 |
| cdent | Yes, I know, but it presents itself as a solution to multiple problems, and as others have mentioned, by being so well formed it feels a bit like a fait accompli (even though it is not). | 13:38 |
| cdent | It might have been better to decompose and list each of the individual problems | 13:39 |
| cdent | I dunno, there are of course horrible thread wastage risks with that approach too | 13:39 |
| cdent | I'll keep working on my thoughts on the matter and try to come up with a coherent response. It's a bit challenging as I'm seeing all this through a nasty head cold, making it hard to think. | 13:39 |
| ttx | agreed, there was a bit of time pressure as the ideal yearly timing happens to fall soon. I wish I had that idea 6 months ago :) | 13:39 |
| ttx | although the way it looks we'll soon have 12 months to wrap our heads around it | 13:40 |
| cdent | my interpretation of the current outcome of the thread is not as clear cut as yours seems to be | 13:41 |
| cdent | biab | 13:41 |
| ttx | I mean, to be able to pass it in time for Rocky, it would have required broad consensus, and I don't think we have that... so the next possibility would be to change in 2019 | 13:43 |
| ttx | which gives us time to wrap our heads around it | 13:43 |
| *** mriedem has joined #openstack-tc | 13:51 | |
| cdent | oh, that assumes a year that coincides with the calendar year, which doesn't have to be the case | 13:51 |
| persia | I suspect that a number of consumers would prefer something that could appear to match a calendar year, for their own internal convenience of thinking. I may be mistaken. | 13:57 |
| cdent | you're probably right persia | 13:58 |
| cdent | but that leads to one of the points I think we should think about: ways of detaching the consumption cycle from the development cycle | 13:59 |
| cdent | they are already not the same thing | 13:59 |
| cdent | so why not take advantage of it | 13:59 |
| persia | Ah, yes. And to be clear, I think the consumption cycle is best to be aligned with the calendar year. I think development cycles are best aligned with events, the scheduling of which have different constraints. | 14:00 |
| ttx | ++ | 14:04 |
| *** kumarmn has joined #openstack-tc | 14:19 | |
| ttx | cdent: what was your alternate name suggestion for "strategic contributions" ? ISTR you had a good one | 14:20 |
| cdent | I've forgotten (see above about cold), but it should be in the log, I'll look. | 14:20 |
| cdent | I can't find it :( | 14:24 |
| *** kumarmn has quit IRC | 14:24 | |
| *** kumarmn has joined #openstack-tc | 14:24 | |
| *** kumarmn has quit IRC | 14:29 | |
| ttx | yeah, I couldn't either | 14:31 |
| cdent | to make up a new one I think "community oriented contributions" is at least less ambiguous, but unfortunately a bit of a mouthful | 14:35 |
| persia | That name sounds dangerous to me: tossing folk upstream without direction often leads to dissatisfaction by the sponsor. "strategic" at least suggests some value may be gained. | 14:39 |
| ttx | "generally-useful contributions" ? | 14:41 |
| ttx | "Good citizen" ? | 14:41 |
| flaper87 | jeez, the release thread exploded before I even had a chance to read the first email | 14:42 |
| ttx | heh | 14:42 |
| cdent | persia: I don't have a problem with the word strategic itself, rather that without a modifier associated with it, we don't know for whom it is strategic | 14:42 |
| ttx | project-strategic contributions ? | 14:43 |
| * flaper87 reads the backlog and searches for a summary | 14:43 | |
| persia | cdent: I consider that to be a value: carries implications of being strategic for both the project and the contributor (or contributing organisation) | 14:43 |
| ttx | flaper87: I did try a summary around 13:09 UTC today | 14:43 |
| ttx | although it's very partial, more a "learnings" than a :summary" | 14:44 |
| ttx | oh joy 6 more responses | 14:44 |
| cdent | flaper87: ttx's summary was good but I would recommend doing a reading of your own, as ttx's summary inevitably reflects his biases :) | 14:44 |
| flaper87 | ttx: cdent ok, thanks | 14:45 |
| cdent | persia: my concerns comes from my initial understanding when ttx first used the term. Out of context when I heard "strategic contribution"I thiought he meant "contributions solely for the benefit of the company" | 14:45 |
| * flaper87 will never catch up T_T | 14:45 | |
| cdent | which is the _exact_ opposite of what he meant | 14:45 |
| cdent | and then again when I was writing the recen tc report I forgot again, and almost wrote the wrong thing and had to correct myself | 14:46 |
| mugsie | is the term for that not "tactical contributions"? | 14:46 |
| persia | Ah, if the goal is to exclude that which is in the (selfish) interest of the contributor/contributing org, then yes, "Community oriented contributions" is right. I hope that isn't being promoted as a good idea. | 14:46 |
| cdent | mugsie: only because you are sitting in the position of the community | 14:47 |
| mugsie | true | 14:47 |
| *** lbragstad has joined #openstack-tc | 14:50 | |
| persia | Re-reading the TC report: maybe "long-term contributions" or "project quality contributions"? | 14:50 |
| persia | My fear is mostly that if there isn't a selfish value to doing them, they may be difficult to justify for other than emotional reasons. | 14:50 |
| cdent | I think we need to address that gap. The style of development that is OpenStack is based on a fairly emotion justification: by working together we can make something that is better. | 14:51 |
| cdent | That obliges to the participants to be something other than entirely selfish. | 14:51 |
| cdent | And also an awareness that the long term gains in doing something not immediately selfish are selfish | 14:52 |
| persia | That last point is the one I think most important. | 14:53 |
| persia | In general, when soliciting corporate sponsorship of open source activities, I spend a fair amount of time helping folk appreciate how spending time resolving technical debt, improving test frameworks, etc. are of selfish benefit to them, in excess of the investment. | 14:54 |
| * cdent nods | 14:54 | |
| cdent | I often feel we're not speaking about that out loud enough, often enough. | 14:55 |
| persia | When we create a dichotomy between "corporate interest" and "upstream interest", we end up creating a situation where it is implied that corporations are not interested in upstream goals, which I believe to be unhealthy for the long-term. | 14:55 |
| persia | Many organisations contributing to openstack do so because they have a downstream codebase (maybe a project, maybe a deployment, maybe something else). | 14:56 |
| persia | In most cases, I suspect they spend a lot of time downstream trying to catch up to upstream. Fixing this by making it easier to land stuff is in their selfish interest. Making it easier to land stuff may require spending several months refactoring code or dealing with edge cases. | 14:57 |
| cdent | I am not at all convinced that a longer cycle will help with landing code | 14:57 |
| persia | But if framed in terms of the benefit, this sort of contribution is easier to solicit. On the other hand, everyone has this message: staff instructing management that "I'm working on X, because we need that for Y, which you told me to do this quarter" is a better way to pass the message than "Org B needs to send N people "upstream" necause they have J people downstream,and need to provide support to balance the support those folk need." | 14:58 |
| persia | I thought that was the proposal: that a longer cycle would let people with less time to contribute be able to get their features ready within a cycle, rather than spending so much time rebasing. | 14:59 |
| cdent | I dunno, I sometimes feel like it ought to be as straightforward as: "you have a product based on openstack therefore you are ethically bound to provide bountiful upstream contributors for the sake of long term health" | 15:00 |
| cdent | persia: I think that's the proposal, I just don't think it will work | 15:00 |
| cdent | tc-members, it is office hour and we've been at it for hours already | 15:00 |
| smcginnis | The door is always open. ;) | 15:00 |
| fungi | oof, you're saying i get to spend office hour catching up on scrollback ;) | 15:01 |
| persia | I think that straightforward statement is hard to consume in light of fiduciary duty to stockholders by the majority of joint stock organisations. | 15:01 |
| cdent | fungi: indeed | 15:01 |
| *** marst has joined #openstack-tc | 15:01 | |
| pabelanger | o/ | 15:01 |
| persia | Contrast "If you strategically depend on this open source project for your business, you expose yourself to significant existence risk by not ensuring you have sufficient active upstream contributors to ensure the project continues to meet your future needs." | 15:02 |
| smcginnis | The other option would be to do more frequent releases and just make them less of a big deal. Kind of what Ben was getting at earlier on in the LTS discussion I think. | 15:02 |
| smcginnis | So if you miss a release, it's really not a big deal to wait a few months. | 15:02 |
| smcginnis | The hard part being distros deciding which release to pick up as their official productized version. | 15:02 |
| cdent | persia: I sometimes feel that fiduciary duties to stockholder is an imaginary shared myth and something I really wish we could stop with | 15:03 |
| cdent | smcginnis: I think I'm in the more frequent and leave it up to the distros to do what they like camp. | 15:03 |
| flaper87 | smcginnis: fwiw, I think I'd be more inclined to even shorter releases and making them less of a big deal. I've been looking at how releases in Kubernetes are done and how features are carried across multiple releases | 15:03 |
| flaper87 | cdent: yeah | 15:04 |
| persia | cdent: I'd be delighted to live under different rules. That said, I often find risk management a well-received argument. | 15:04 |
| cdent | persia: this is why I'm glad you exist | 15:04 |
| flaper87 | I haven't had time to wrap my head around all the discussion but I have been looking closer and closer to how the k8s releases are done | 15:04 |
| smcginnis | The big concern for me then is if each distro picks a different release to make their "LTS" release. | 15:05 |
| cdent | why is that "our" problem? | 15:06 |
| mugsie | I will tell you trying to productise k8s is even more painful than OpenStack right now | 15:06 |
| smcginnis | And that resulting in either conscious or unconcious focus from different folks of trying to get certain features in specific releases. | 15:06 |
| smcginnis | cdent: Just because of shenanigans like that. ^ | 15:06 |
| cdent | in fact why is anything to do with productising "our" problem? | 15:06 |
| cdent | I'm not saying it shouldn't be, but why do we persist with it? | 15:06 |
| flaper87 | mugsie: but has that been because of the release cycle? | 15:07 |
| persia | smcginnis: If different folk push different things into different releases, doesn't that automatically provide some distribution of demand to allow PTLs to only concentrate on a few things each cycle? | 15:07 |
| mugsie | cdent: because right now, with the turn around time with some vendors, a new install may be out of upstream support before it is even installed | 15:07 |
| mugsie | flaper87: yes | 15:07 |
| smcginnis | cdent: Fair point. But I think due to $CURRENCY we are kind of forced to have to consider it somewhat. | 15:07 |
| smcginnis | persia: Another fair point. | 15:07 |
| flaper87 | mugsie: mmh, mind elaborating? | 15:07 |
| cdent | mugsie: but why is _that_ a problem? Is the reason the vendors (of open source stuff) exist so they can provide support? | 15:08 |
| mugsie | you work on get $release ready, find all the new issues, fix them, adapt tooling, then release your new version, just as the next version of k8s is released. | 15:08 |
| cdent | s/Is/Isn't/ | 15:08 |
| mugsie | then you get shouted at by customers looking for $feature, but you are behind | 15:08 |
| mugsie | and the cycle starts again | 15:09 |
| flaper87 | mugsie: tbh, I don't think that's a problem the upstream community should try to solve | 15:09 |
| persia | On $CURRENCY: while it makes sense for mainline projects (e.g. OpenStack) to make is easy to productise their output, actually constructing a product reduces the opportunity for downstream differntiation and value creation: too much focus in mainline means less reason for folk to invest. | 15:09 |
| TheJulia | Spending the last little bit reading the scroll back and to persia's point of long lag times for downstream integration, every large operator I've spoken to that has had to do anything custom downstream is >= 1 year behind the current release, and simply cannot keep up with the cadence. | 15:09 |
| mugsie | flaper87: how many users of OpenStack take is directly from source? | 15:09 |
| persia | TheJulia: 1 year is a dream for some folk I speak with :) | 15:09 |
| mugsie | it hurts *our* suers | 15:10 |
| mugsie | users* | 15:10 |
| * persia regularly sees operators upgrading after 2-3 years, including rebasing custom code | 15:10 | |
| smcginnis | mugsie: I couldn't get any verified CD users when I tried a few months back. | 15:10 |
| flaper87 | mugsie: very few or none but again, I think those are problems solvable downstream too. | 15:10 |
| smcginnis | mugsie: Just a lot of "we might have users deploying from main". | 15:10 |
| mugsie | especially when they try to make fixes, and the branch is gone. | 15:10 |
| TheJulia | persia: And thinking about it, if you can't jump versions, your doing a new deployment, so your taking 3+ months for hardware, 3+ months to configure/systems validation/install, and then business acceptance could take six or more months.... | 15:11 |
| ttx | ohai | 15:11 |
| fungi | at least in the past when i've dealt with this for other projects, the customers demanding newer upstream features in your productized version don't _actually_ want to use you productized version and would ultimately be happier consuming upstream releases directly | 15:11 |
| persia | mugsie: I submit that most of the major operators consume mainline, and most of the smaller deployments consume some product. From my limited information, I believe it to be related to a balance between the pain of waiting for features/bugfixes vs. the cost of hiring staff to do things directly. | 15:11 |
| persia | TheJulia: Lots of folk redistribute hardware between deployments, but yeah, it's not pretty. | 15:12 |
| pabelanger | I always thought it would be an interesting project in openstack, to try and CD openstack on the hardware we are in infracloud for example. Having a group of devs / ops working together to try and make it happen, then use the resources in nodepool for openstack-infra. | 15:12 |
| mugsie | persia: I would suggest that quite a few major operators actually have a hybrid, some custom, and some vendored | 15:12 |
| pabelanger | then take what we learn and feedback loop into the project | 15:12 |
| fungi | or, at least, after they get to try and consume upstream directly for a while they get a better understanding of why you're productizing something which isn't moving as fast or quite as up to date | 15:12 |
| mugsie | pabelanger: that would be great - but a lot of work | 15:12 |
| persia | fungi: Very commonly, although there are still a few organisations with private clouds that have stupid policies that require them to install vendor software, rather than own it themselves. | 15:12 |
| ttx | So around "more frequent releases"... I like that, and I don't think it's necessarily the opposite to longer development cycles | 15:12 |
| persia | mugsie: Agreed. | 15:13 |
| pabelanger | mugsie: Yup, i think you'd need to work with a deployment project in openstack already to make it happen | 15:13 |
| persia | ttx: Could you expand on how you would detangle release cycles from development cycles? | 15:13 |
| ttx | They already are. | 15:13 |
| mugsie | pabelanger: when we operated Designate in a public cloud it was great, we learnt so much, and the version in production was just a week or 2 behind master | 15:14 |
| flaper87 | persia: I think they are detangled already | 15:14 |
| TheJulia | pabelanger: I seem to remember that was the original dream when the idea of infracloud was pitched in a conference room during a midcycle long ago. | 15:14 |
| ttx | a development cycle is more like an epic, starts with F2F, has goals | 15:14 |
| persia | I thought a majority of the projects in the coordinated release had freeze scheduling to align development activities with teh release cycle. If that isn't true, then it isn't important. | 15:15 |
| pabelanger | mugsie: yah, to some degree, osic-cloud1 that mrhillsman was part of did the same too. I think it helped them greatly. | 15:15 |
| persia | To my mind, the other path involves a lot more effort on feature branches, for-next style arrangements, etc. | 15:15 |
| pabelanger | TheJulia: yes! Sadly, we lost people to do the work for it. But agree it is still an important thing we could do as a project | 15:15 |
| ttx | persia: only the projects following the one-release-per-cycle model (which arguably includes the largest ones) | 15:15 |
| fungi | pabelanger: well, also, we don't have fresh hardware or anywhere offering to host it long-term either | 15:16 |
| persia | Heh, indeed. If we can ignore that small minority, then I retract my concern :) | 15:16 |
| dmsimard | persia, mugsie: where do you draw the line between "mainline" and "vendor" ? For me, installing packages from Ubuntu cloud archive or RDO isn't vendor, but rather packaged source -- if an operator deals with Red Hat, Mirantis, Canonical, etc for an actual supported product offering, that's what I'd call vendor. | 15:16 |
| ttx | see Nick Barcet email on the thread, 7 min ago | 15:16 |
| ttx | he is proposing more coordinated intermediary releases | 15:17 |
| ttx | (and longer dev cycles) | 15:17 |
| pabelanger | fungi: agree, times have changed a little since fort collins | 15:17 |
| dmsimard | Like, would infracloud be "vendor" because it uses packages from Ubuntu Cloud Archive despite providing it's own installation mechanism ? | 15:17 |
| persia | dmsimard: For me, "mainline" is the master branch on git.openstack.org, "vendor" is some source provided from somewhere else, at some delay. This includes the stable branches. | 15:17 |
| smcginnis | persia: ++ | 15:17 |
| mugsie | yeah, productised is somehting that happens after the tag and tarball are put on *.openstack.org | 15:18 |
| persia | dmsimard: Infracloud is *definitely* "vendor",. The vendor is Infra. Infra builds it based on vendors to infra, which may include some mainline. | 15:18 |
| dmsimard | persia: you really believe that most of the operators are running off of master ? I would be surprised if that's the case | 15:18 |
| pabelanger | So, one of the comments I heard in passing a bout 1 year release cycle, was by moving to it, would allow openstack to stablize more for releases. However, I admit I am not sure if that is going to be accurate, I would imagine with a longer development cycle, projects would want to get more features in, not less | 15:19 |
| TheJulia | persia: would the feature branch idea.... could it be lighter weight on testing, and the proposed merge to say master branch would be the branch with all of the cross project gating/upgrade testing/etc | 15:19 |
| dmsimard | I mean, data from the user survey puts most of the operators a few releases old | 15:19 |
| persia | dmsimard: Most of the large operators with whom I've had the opportunity to discuss internal arrangments frequently pull bits of this and that from master, usually cherry-picking things. The alternative is too long a wait to land that bit of code. | 15:19 |
| persia | The better ones then send their rebase results back to the stable teams, but folk often claim they don't have time for that. | 15:20 |
| persia | The core of what is being run is likely to be several releases old: very rarely less than a year, | 15:20 |
| dmsimard | FWIW I end up agreeing on your definition of vendor vs mainline | 15:20 |
| fungi | TheJulia: we could probably approximate that model without feature branches simply by running fewer jobs in the check pipeline than gate | 15:21 |
| dmsimard | Although I still draw a line where (vendor supported) productization comes in | 15:21 |
| persia | TheJulia: I'd probably do full testing on each branch, with an always-trusted release branch, but yes, that could be possible: it depends on test resources vs, number of things to test. | 15:21 |
| TheJulia | fungi: That could possibly free resources, keep the queue down, but projects would have to be onboard | 15:21 |
| fungi | TheJulia: or we've discussed adding an intermediary pipeline too which only runs jobs after at least one core +2 review | 15:21 |
| persia | dmsimard: What is the support line? Does it still apply for the larger customers who are able to instruct vendors to provide non-standard solutions under standard support? | 15:22 |
| TheJulia | That is not a bad idea either, the downside that I see is if something is broken elsewhere, then the fix to the fix could end up in a long nit-picking cycle | 15:22 |
| TheJulia | which becomes a cultural problem almost | 15:22 |
| fungi | TheJulia: but all of those (feature branches with fewer jobs as well) seem to mostly be about reducing ci load, at the expense of more immediate feedback to developers | 15:23 |
| TheJulia | well, feedback now is not immediate | 15:23 |
| fungi | agreed | 15:23 |
| TheJulia | For some check pipelines it is hours | 15:23 |
| * TheJulia realizes the horse is dead | 15:23 | |
| mrhillsman | pabelanger: it would be lovely to have cd and work for this was happening during osic | 15:23 |
| fungi | some of those very-long-running jobs may make more sense to punt into peeriodic as well | 15:24 |
| fungi | periodic | 15:24 |
| mrhillsman | https://github.com/osic/qe-jenkins-onmetal https://github.com/osic/qe-jenkins-baremetal and i have been still thinking about this since the dissolve of osic | 15:24 |
| mrhillsman | even yesterday after reading all the replies to the release change proposal thread | 15:26 |
| mrhillsman | as a way to address a subset of concerns | 15:26 |
| TheJulia | fungi: I think at least one grenade/upgrade job would be needed for the check gate, and those are longish to begin with, since they can catch some major issues well in advance, but everyones milage is going to vary | 15:26 |
| dmsimard | persia: I can't speak for Ubuntu Cloud Archive packages (or SUSE's) but despite an obvious personal bias, RDO takes a lot of pride in packaging upstream source as-is without custom patches -- for me that's very close to mainline. And yet, it is unsupported in the sense that you can't go and call Red Hat to get their engineers to help you with a problem through a support contract. That's what I mean by | 15:27 |
| dmsimard | vendor support. | 15:27 |
| mrhillsman | maybe something we can entertain in the openlab space | 15:27 |
| pabelanger | mrhillsman: yah, i think it would be great if we could show that you could CD openstack some how | 15:27 |
| ttx | wow that thread is getting big. those openstack-dev ML stats were a bit down for 2017, figured I should fix that | 15:27 |
| mrhillsman | we were chasing master with that effort | 15:27 |
| mrhillsman | and it as actually making some damn good progress | 15:28 |
| smcginnis | ttx: :D | 15:28 |
| mrhillsman | ttx: lol | 15:28 |
| ttx | next week I'll propose moving to GitHub | 15:28 |
| TheJulia | I've heard of several teams that have gone off doing CD deployments chasing master with decent success, fwiw | 15:28 |
| TheJulia | ttx: oh my | 15:28 |
| persia | dmsimard: Ah. I see what you mean. In my experience, this experience differs for certain customers for most of that class of vendor, but to the core, when considering vendor solutions, I think each operator takes a different set of choices about whether things like "support" are meaningful, and so picking any variable to separate them seems awkward. | 15:29 |
| * dmsimard nods | 15:29 | |
| ttx | tc-members: another question I had for you is around SIG governance. Currently since SIGs are not firmly below TC nor UC, if there is some issue in there (say the co-leads don't like each other anymore, or the participants don't like their leads) there is no place of appeals | 15:30 |
| ttx | One way to fix it is to have the TC and the UC bless the Meta SIG and ask them to police that stuff | 15:31 |
| mrhillsman | ttx: i thought we decided on the meta sig handling that for now? | 15:31 |
| mrhillsman | ah ok, needs that blessing :) | 15:31 |
| ttx | mrhillsman: we did say that it could be a solution, still need that blessing | 15:31 |
| mrhillsman | ++ | 15:31 |
| cdent | who is the meta sig at this point? | 15:31 |
| mrhillsman | lol, me and ttx | 15:31 |
| ttx | so I'm checking if it actually flies or if anyone has a better suggestion | 15:31 |
| ttx | cdent: basically a rep from each body :) | 15:32 |
| cdent | seems a reasonable course of action to me | 15:32 |
| ttx | The other solution is to call for common meetings the day it happens, but then you enter crappy vote territory | 15:32 |
| fungi | or we could just say disagreements aren't allowed ;) | 15:33 |
| smcginnis | I think by having TC and UC representation in the meta-sig, that should be a good place for any issues to escalate. | 15:33 |
| fungi | yeah, wfm | 15:33 |
| persia | Maybe a combined Meta-SIG review meeting once in a (rare) while, with all of TC and UC present? | 15:34 |
| * persia is thinking once or twice a year for that | 15:34 | |
| mrhillsman | ++ | 15:34 |
| mrhillsman | i like that idea | 15:34 |
| cdent | meta-meta-sig should have governance over meta-sig, and have pre-meetings whenever they need to meet | 15:34 |
| mrhillsman | that was so redundant | 15:34 |
| mrhillsman | hehe | 15:34 |
| ttx | OK, let's say Meta SIG should be co-lead by one TC and one UC member, and is tasked with solving issues, and worst case scenario calls for common meetings with TC/UC | 15:36 |
| ttx | like in case they need more input to make the call | 15:36 |
| smcginnis | ttx: Works for me. | 15:36 |
| mrhillsman | + | 15:36 |
| ttx | each rep reports back to their $C | 15:37 |
| cdent | yup | 15:37 |
| fungi | sounds fin | 15:37 |
| fungi | e | 15:37 |
| ttx | mrhillsman: I'll draft a resolution on our side to capture that | 15:37 |
| flaper87 | ++ | 15:37 |
| ttx | you should plan to get a similar thing approved on yours | 15:37 |
| mrhillsman | cool | 15:37 |
| mrhillsman | will do | 15:37 |
| ttx | mrhillsman: maybe check that the idea works for them too, before I get busy drafting | 15:38 |
| mrhillsman | will send email shortly | 15:38 |
| ttx | On the 1-year thing, where do y'all stand ? 1/good idea let's do it for Rocky, 2/might be a good idea but we need careful consideration and a long discussion, so not rocky, 3/worst idea ever | 15:41 |
| cdent | 4/ there are problems to be solved, this doesn't solve them | 15:42 |
| flaper87 | cdent: you literally stole my words | 15:42 |
| TheJulia | I'm a 2 and 4 | 15:42 |
| flaper87 | cdent: 4 | 15:42 |
| fungi | i'm open to the idea of a 1-year cycle, but waiting to see more feedback before i can be sure we're not missing something | 15:42 |
| flaper87 | if anything, I'd rather do more frequent releases, as we mentioned earlier in the office hour. But again, #4 | 15:42 |
| ttx | for those on 4, any alternate suggestion to fix the problems ? | 15:43 |
| cdent | ttx have you seen my message on the thread? I think there's some thinking to do around decoupling development timing from consumption timing | 15:43 |
| flaper87 | ttx: not off ot the top of my head but I'll gladly spend time trying to figure out another way to solve these problems | 15:43 |
| fungi | i wouldn't say i'm #2 necessarily, because >24 hours of discussion doesn't necessarily mean "long discussion" (even though i'll grant that thread has exploded quickly) | 15:43 |
| ttx | fungi: 1.5 :) | 15:44 |
| ttx | (I'm 1.5 too | 15:44 |
| ttx | ) | 15:44 |
| EmilienM | 4/ | 15:44 |
| cdent | I partially agree with what fungi has said: we haven't actually played this out that much. It's been in awareness for a day. Sometime might push out some brilliance in the next week. | 15:44 |
| fungi | well, i can't say whether i'm #1 until i see the discussion unfold. i could be #4 | 15:44 |
| EmilienM | I'm not sure we tackle the problem on the right side (which problem by the way, it seems folks happy about this proposal have different problems to solve) | 15:45 |
| fungi | we're getting early feedback from our highest-engagement ml participants so far | 15:45 |
| TheJulia | ttx: Cycle bound the large behavior changing features, rapid release new features during a cycle resulting in more frequent releases? I'm not sure CD really, truly solves the operator lag issue beyond confidence in end of cycle release to reduce the actual release overhead for packagers. | 15:45 |
| flaper87 | cdent: I'm holding off on posting any opinions on that thread until I've gotten enough time to process everything | 15:45 |
| ttx | ok, all that points to a timing too short to actually change anything for Rocky. is that the common view ? | 15:45 |
| ttx | (the release team kinda needs to work on the cycle now :) ) | 15:46 |
| flaper87 | Rocky is def off, in my opinion. | 15:46 |
| smcginnis | ttx: Yeah, I think we are going to have to wait a bit longer on this one. | 15:46 |
| fungi | i wouldn't rule out changing for rocky, at least not yet | 15:46 |
| ttx | TheJulia: looks a bit like what NickBarcet suggested in the thread? | 15:47 |
| smcginnis | ttx: Let's give it a week maybe and see how the mood shifts. | 15:47 |
| mrhillsman | i like the idea floated by mriedem about 6 months feature, 6 months bugs | 15:47 |
| dmsimard | I feel like despite the intent of the development cycle change is not about LTS, the notion of LTS might come into play so it might be a good idea to wait how that discussion is going to end | 15:47 |
| flaper87 | I just don't think there's any need to rush it and try to make this change in Rocky does feel like it | 15:47 |
| cdent | flaper87: so you're going to be the one who provides the brilliance in the next week? :) | 15:47 |
| fungi | but i can't form a solid opinion until more of the community has a chance to provide feedback | 15:47 |
| TheJulia | ttx: I've not dug into my email yet today :\ | 15:47 |
| flaper87 | we're talking about taking more time for releases and preparing things so, let's do the same for planning this change | 15:47 |
| flaper87 | cdent: it might all come down to a pic of me surfing | 15:47 |
| flaper87 | :D | 15:47 |
| cdent | that's it! | 15:47 |
| smcginnis | fungi: The only rush is that it would need to be now, or in a year. | 15:47 |
| ttx | flaper87: yeah, the rush was more linked to the release window opening nowish | 15:47 |
| cdent | 1 year long cycle, six months of development, six months of surfing, interspersed. | 15:48 |
| flaper87 | ttx: understood, makes sense to want to have answers now, regardless of the answer :) | 15:48 |
| fungi | smcginnis: sure, but you must at least concede that one day is not enough time to say we have any representative amount of feedback on the proposal ;) | 15:48 |
| flaper87 | cdent: see? didn't have to wait to next week | 15:48 |
| cdent | fungi++ | 15:48 |
| cdent | what we have is feedback from the early actors, which is almost exactly not the group that we're trying to solve for, yeah? | 15:49 |
| EmilienM | the main thing I've heard about why people are happy about this proposal is "we'll have more time" - but I'm not sure it's accurate, if we commit to do more things in a longer release. In fact, it highlight the problem that projects don't do a good job in planning their roadmap for cycles | 15:49 |
| flaper87 | fungi: smcginnis let's put it this way, I barely caught up with the thread 10mins ago | 15:49 |
| fungi | we have a lot of feedback, but from a very specific subset of our community who is probably not a representative sample of opinions | 15:49 |
| mrhillsman | i'd have to agree with that partially cdent | 15:49 |
| smcginnis | fungi: Absolutely! I'm just saying that was the only reason why we are even considering any kind of time pressure on this. | 15:49 |
| flaper87 | so, yeah, we gotta let it mature fore another week. I bet we go over 500 emails | 15:49 |
| mrhillsman | could definitely use more time for the late responders | 15:49 |
| cdent | EmilienM++ I think we'd just end up trying to pack even more in | 15:50 |
| flaper87 | for* | 15:50 |
| dmsimard | EmilienM: this is not so much for OpenStack developers than it is about *users* and *operators* IMO. | 15:50 |
| pabelanger | EmilienM: yes, more time for X. But I honestly don't see that, just means twice the amount of things will be added in 1 year over 6months | 15:50 |
| mriedem | dmsimard: how is this about users? | 15:50 |
| *** dansmith has joined #openstack-tc | 15:50 | |
| mriedem | stability? | 15:50 |
| mriedem | features? | 15:51 |
| mriedem | 1 year cycle doesn't magically get you stability | 15:51 |
| * flaper87 agrees with mriedem | 15:51 | |
| dmsimard | mriedem: wrong vocabulary, what I really meant was operators | 15:51 |
| mrhillsman | oh shit, you have awoken the beast dmsimard | 15:51 |
| fungi | also, i don't quite see the rush. why can't we, a few weeks into rocky, say we've reached consensus to bump out the release date on it by an extra 6 months? | 15:51 |
| EmilienM | mriedem++ | 15:51 |
| pabelanger | ttx: cdent: right now 4, problems to solve, but unsure corrently if this solves it. I'm trying to keep up with all the replies and process them. | 15:51 |
| flaper87 | fungi: good point | 15:51 |
| *** edleafe has joined #openstack-tc | 15:51 | |
| fungi | i mean, i agree waiting too far into the release cycle would be bad, but... | 15:51 |
| flaper87 | however, let's first reach consensus. | 15:52 |
| mrhillsman | jk mriedem | 15:52 |
| flaper87 | also, I think there's a planing problem, fungi | 15:52 |
| mriedem | dmsimard: this doesn't give ops LTS either | 15:52 |
| mriedem | most ops are'nt even 1 year out right? | 15:52 |
| mriedem | they are waiting until start to eol the oldest supported upstream branch | 15:52 |
| dmsimard | mriedem: I realize that it has nothing to do with LTS | 15:52 |
| TheJulia | fungi: contributing businesses might get some heartburn from the community changing after what they perceive to have been committed to. | 15:52 |
| flaper87 | the release cycle is planned before the cycle and extending it would translate to doing the planning again | 15:52 |
| mrhillsman | some folks may take that the wrong way | 15:52 |
| fungi | nothing gives ops lts other than a group of people stepping up to solve maintaining an lts | 15:52 |
| dmsimard | mriedem: but releases are bound to be supported for longer than the actual 1 year support cycle because otherwise it means as soon as we ship a new release, the previous release goes EOL | 15:52 |
| pabelanger | I sitll think release early, realse often is the way to go, but still trying to wrap my brain around how a 1 year cycle for affect that. Even if we do encourage projects to release more | 15:53 |
| mriedem | dmsimard: supported by whom? | 15:53 |
| mrhillsman | yeah, i am trying also since yesterday mriedem to figure out how it gets to the root of ops/users concerns | 15:53 |
| mriedem | upstream stable team, or vendors? | 15:53 |
| dmsimard | mriedem: by upstream | 15:53 |
| ttx | I think it's squarely for developers. If it doesn't benefit devs (especially part-time devs that we ned to attract more of) then we should not do it. | 15:53 |
| fungi | TheJulia: maybe, but this would be the first time we've changed our minds on a release date in ~7 years of the project existing so i can't imagine they'd get too bent our of shape | 15:53 |
| mrhillsman | ^ | 15:53 |
| mriedem | dmsimard: the stable team hasn't said they are going to change the stable policy of 1 year | 15:53 |
| EmilienM | Tim said it would actually affect the feedback loop from operators | 15:53 |
| ttx | there are way better answers for ops needs around releases, and that's skip-release upgrades, and LTS branches | 15:53 |
| EmilienM | we would have to wait more time to know if what we're doing work | 15:53 |
| mriedem | this in no way helps a part time dev that is pushing a non-priority feature | 15:54 |
| dmsimard | mriedem: releases.openstack.org shows each release currently going EOL a year after their release, if we move to a 1 year development cycle without touching the support cycle, it means we will eventually only have one stable release instead of two | 15:54 |
| EmilienM | like, if I implement a feature early in the cycle, I would have to wait almost if not one year because it's in production | 15:54 |
| mriedem | dmsimard: those are the breaks i guess | 15:54 |
| mriedem | dmsimard: if we extend the stable policy because of this thing, then we're getting into upstream LTS areas | 15:54 |
| mriedem | which seems underhanded to me | 15:55 |
| EmilienM | ttx: devs don't need more time, they need to learn about doing a schedule | 15:55 |
| ttx | unlikely. LTS means 3-5 years, not 2 | 15:55 |
| EmilienM | and right now, we all over-commit | 15:55 |
| EmilienM | and some of us are burnt and thing we need more time. No, we need better scheduling | 15:55 |
| mriedem | EmilienM: agree | 15:55 |
| mriedem | this is why i said in the thread, the team needs to agree to less stuff per cycle | 15:56 |
| dmsimard | mriedem: if we're inflexible on the support cycle and leave it to one year, it means we're expecting operators to upgrade in a matter of weeks after the new release comes out because their current release goes EOL almost immediately and I don't think that's realistic | 15:56 |
| mrhillsman | is that really because of what employer has you doing or things you take on yourself | 15:56 |
| mugsie | something I have seen said is that features take too long to hit large amounts of users, and the bug they find - would this cause that loop to double (at least) | 15:56 |
| EmilienM | we as a community aren't good at saying "no we can't do it" | 15:56 |
| mriedem | and be OK with telling non-priority bps that they didn't make the release | 15:56 |
| EmilienM | because we want to be nice and accept everything | 15:56 |
| EmilienM | and look now, we recognize 6 months isn't enough so we want to extend to 1 year | 15:56 |
| TheJulia | EmilienM: absolutely agree, although not just scheduling, the culture is vital to be on the same page to work together in the same direction. | 15:56 |
| cdent | mrhillsman: that's a good question and I think the answer is "it's complicated" | 15:56 |
| mrhillsman | EmilienM is that employee driven or community driven? | 15:56 |
| cdent | some people see voids and try to fill them, they are just drawn that way | 15:56 |
| mriedem | mrhillsman: for me it's both | 15:56 |
| fungi | dmsimard: the way i see it, we'd be maintaining one stable branch at a time for purposes of validating master development toward the subsequent release (to confirm we can upgrade, and that we maintain some backward compatibility), but that one-year mark could be where the lts team takes over maintenance | 15:57 |
| EmilienM | mrhillsman: yeah it's both... | 15:57 |
| mriedem | mrhillsman: i don't like feeling like an asshole by telling someone no to their unicorn feature | 15:57 |
| mrhillsman | i think from the community there is a strong desire to slow down feature addition and shore up more technical debt but it is hard from what i hear when your employer is pushing for more | 15:57 |
| EmilienM | mrhillsman: my employer will probably ask us to do twice the work for each release if we take the proposal | 15:57 |
| cdent | mrhillsman: I'm not sure I believe that. I keep hearing from NFV people "why aren't you supporting X yet?" | 15:57 |
| dmsimard | fungi: it's an important discussion to be had IMO, the development cycle and the associated support cycle go hand in hand | 15:57 |
| mriedem | EmilienM: agree - this won't create stability cycles unless each project decides to do that | 15:57 |
| EmilienM | mriedem: we'll have to at some point | 15:58 |
| cdent | EmilienM: upstream, why? | 15:58 |
| EmilienM | cdent: why what? sorry | 16:00 |
| pabelanger | wouldnt the cost of a 1 year stability be more expensive then 6 month? could projects go that long without landing new features? | 16:00 |
| cdent | EmilienM: "we'll have to [create stability cycles] at some point". To which I'm asking: Why is that something that needs to happen upstream? | 16:00 |
| cdent | Of course I'm not sure I'm understanding what "stability cycles" means. | 16:01 |
| EmilienM | cdent: because upstream is where we develop things? | 16:01 |
| dmsimard | OTOH, 1 year is a long time.. I mean, k8s was a fraction of what it is today even two years ago. I'm concerned about the landscape changing enough that the plan set at the beginning of the development cycle ends up being inaccurate over that period. | 16:01 |
| ttx | dmsimard: agreed -- if we can't find a way to release more often in a longer cycle, that just won't work | 16:02 |
| EmilienM | cdent: not stability cycles, but cycles with less features probably | 16:02 |
| fungi | well, there _was_ a time when openstack released every 3 months. we do seem to be cooling off in some areas of development, so it seems natural our need to have coordination points becomes less frequent as time marches on | 16:02 |
| cdent | EmilienM: I think was conflating "stability cycles" with some of the LTS needs, but you mean "cycles to fix stuff and not add stuff". I'm not sure why we need cycles for that, we should figure out ways to do both, better. | 16:02 |
| mrhillsman | a number i would like to see is of the 20% of contributors who do 80% of the work how many are employeed to work on openstack 80% of their day | 16:03 |
| ttx | nijaba suggested a longer cycle with more coordinated releases in it -- not sure that would relax pressure as much but that's another way to slice it | 16:03 |
| smcginnis | So if we did a one year cycle with something like quarterly releases, we could "highly recommend" that that last release in the cycle be a stability release. | 16:03 |
| EmilienM | "Release early. Release often. And listen to your customers." if we go one year, we'll listen them too late and get serious problems | 16:03 |
| ttx | EmilienM: ++ | 16:03 |
| EmilienM | cdent: no I didn't mean that, sorry if I wrote it | 16:03 |
| smcginnis | mrhillsman: That would be interesting to know. | 16:03 |
| EmilienM | cdent: I meant to say, the problem is not in the duration, but in the content | 16:03 |
| cdent | Do we think of our customers as the people who use openstack, or the people who package it, or the people who deploy it? We talk about all of them, but then when it gets into the details it usually just one for any given decisions. That's complicated. | 16:04 |
| * TheJulia wonders if openstack was to plan/execute on 3 months increments for a 1 year stable release, with exepcation that new features could be available along the way... | 16:04 | |
| fungi | i tend to think of all of those as customers | 16:04 |
| cdent | If our customers are 1 to 2 years behind, already, then they will be even further behind on a 1 year cycle. | 16:04 |
| EmilienM | in that case, "customers" means all of us, devs, ops, users | 16:04 |
| fungi | or "users" | 16:04 |
| mrhillsman | i like the model at rackspace, probably at some other companies to, of everyone is a customer | 16:04 |
| cdent | Yes, I agree everyone is. The issue isn't that. It's that we don't remember that in the details of decision making. | 16:05 |
| EmilienM | OpenStack Infra is a big customer | 16:05 |
| mrhillsman | but you will not satisfy 100% of the customers 100% of the time | 16:05 |
| mrhillsman | :( | 16:05 |
| ttx | TheJulia: that's a bit like what Nick Barcet proposes on that thread | 16:05 |
| fungi | well, i think it's a cop-out to say that everyone's a customer who should be treated with equal priority | 16:05 |
| EmilienM | imagine we wait one year to get an update on the features provided by public clouds that Infra is using | 16:05 |
| TheJulia | ttx: \o/ | 16:05 |
| fungi | but yes we still need to keep all of them in mind when making decisions | 16:05 |
| EmilienM | instead of having small updates every month | 16:06 |
| mrhillsman | fungi: agreed | 16:06 |
| ttx | TheJulia: longer cycles, more coordinated releases, but only one that gets a stable branch per cycle | 16:06 |
| ttx | basically just do the same thing we are doing, but only have stable branches / PTL / PTG / Goals once per year | 16:06 |
| * TheJulia thinks it is just time to take the car to the dealership for repair work and read the mailing list | 16:06 | |
| mriedem | ask the mechanic what he thinks | 16:07 |
| dtantsur | the question holds: who is using the intermediary releases? | 16:07 |
| mriedem | dtantsur: no one | 16:07 |
| cdent | mriedem++ | 16:07 |
| TheJulia | mriedem: sure! :) | 16:07 |
| dansmith | nor would they in a 1-year cycle, which means they're just as useless as they are now, IMHO | 16:07 |
| ttx | dtantsur: define intermediary releases | 16:07 |
| smcginnis | mrhillsman: How do you know it's a he? | 16:07 |
| smcginnis | :P | 16:07 |
| EmilienM | dtantsur: excellent question | 16:07 |
| dtantsur | ttx: in a sense of cycle-with-intermediary (do I spell it right?) | 16:07 |
| TheJulia | dtantsur: then the question is, who would use it if it was available across the board | 16:07 |
| dtantsur | like what ironic is doign nowadays | 16:07 |
| mrhillsman | smcginnis: touche | 16:08 |
| ttx | dtantsur: depends. Swift ones are used | 16:08 |
| mriedem | the *only* reason i think nova would do intermediate releases is so we can do a freeze schedule a few times within the year coordinated release | 16:08 |
| ttx | dtantsur: also peripheral projects ones are used, since they have depends only one way | 16:08 |
| dtantsur | TheJulia: well, if they get bug fix backports (at least security) and upgrade support - sure | 16:08 |
| mriedem | so we can impose deadlines | 16:08 |
| TheJulia | we know people have used intermediary ironic releases, but largely stand-alone users who are installing what is the current latest available "release" | 16:08 |
| EmilienM | let's be realistics, most of products or deployments don't use intermediate releases | 16:08 |
| ttx | dtantsur: most of the other cebntral things like Nova don't do intermediary releases anyway, so yeah they aren't used | 16:09 |
| dtantsur | how is swift supporting their intermediary releases? do they provide bug fixes? | 16:09 |
| mriedem | swift is also a standalone thing | 16:09 |
| mriedem | so intermediate releases makes more sense there | 16:09 |
| EmilienM | I invited mnaser to join here, he's building one of the biggest OpenStack clouds in Canada. Let's ask him what he uses. | 16:09 |
| *** mnaser has joined #openstack-tc | 16:09 | |
| mnaser | o/ | 16:09 |
| mriedem | as noted in the thread, you can run mixed versions of the services and it should be fine - we don't CI that way, so it's a risk, but it's just not something that distros package that way either | 16:09 |
| EmilienM | mnaser: welcome here, I have a question for you. How do you deploy OpenStack? From final releases or from intermediate releases? | 16:09 |
| ttx | mnaser welcomne | 16:09 |
| mrhillsman | yeah, mentioned vexxhost earlier | 16:09 |
| mrhillsman | wondering how he do that pike 2 weeks after release :) | 16:10 |
| ttx | mnaser: does vexxhost even use a component that releases intermediary things ? | 16:10 |
| fungi | dtantsur: my understanding, from a vmt perspective, is that the only reason swift has stable branches is for critical security fixes. and even then they strive for 100% backward compatibility so they'd prefer users just upgrade to latest | 16:11 |
| ttx | swift maybe? | 16:11 |
| mnaser | So we upgrade to major releases when they are out to avoid being in a situation where we are behind and need to do a whole bunch of upgrades. | 16:11 |
| dtantsur | fungi: that's a fair answer, but I'm not sure "upgrade to latest" is something we're ready to tell people | 16:11 |
| dansmith | mnaser is also a good example of keeping up with the releases | 16:11 |
| dansmith | a poster child, if you will | 16:12 |
| mnaser | The only reason we would upgrade to an intermediate release is if it contains a bug fix. | 16:12 |
| *** mgagne has joined #openstack-tc | 16:12 | |
| dtantsur | mnaser: major in a sense of sem-ver? or in a sense of named releases? | 16:12 |
| pabelanger | is there an easy way to see the project that are using cycle-with-intermediary tag? I'm using grep right now, but not sure if that is published to web some place | 16:12 |
| EmilienM | dansmith: with major releases, not intermediary | 16:12 |
| fungi | dtantsur: for some set of "we" anyway... i mean there are already a ton of free software projects which say exactly that. if you report a bug many will close it and ask you to try to reproduce with the latest release | 16:12 |
| EmilienM | mnaser: thanks for your feedback! | 16:12 |
| mnaser | If we run into a bug, we run stable branch until a release is cut | 16:12 |
| dtantsur | on governance.o.o you can get a list of tags, I think | 16:12 |
| dansmith | EmilienM: right, but my point is, he keeps up with the major releases and thus I expect isn't overly concerned about the yearly releases alleviating pain | 16:12 |
| dansmith | presumably maybe he likes it less because he gets features less often | 16:13 |
| mnaser | Then we go back to the release which includes the bug fix | 16:13 |
| ttx | saying nobody ever deploys intermediary releases of their keystone/nova/cinder/neutron deply sounds a bit disingenious, since those don't do intermediary releases at all :) | 16:13 |
| EmilienM | dansmith: good point | 16:13 |
| mrhillsman | pabelanger there is, i remember seeing i think john post a one-liner to email | 16:13 |
| mrhillsman | he parsed releases repo i think... | 16:13 |
| mnaser | I hope that clears it out from our side | 16:14 |
| mnaser | I’d be happy to answer any other questions | 16:14 |
| EmilienM | yes it does, thanks | 16:14 |
| mriedem | ttx: http://eavesdrop.openstack.org/irclogs/%23openstack-operators/%23openstack-operators.2017-12-13.log.html#t2017-12-13T17:44:28 | 16:14 |
| smcginnis | mnaser: Thank you, that's useful info. | 16:14 |
| mriedem | ttx: i framed the intermediate question to ops yesterday, | 16:14 |
| mnaser | But I wouldn’t like a 1 year release cycle though :( it would slow things down a lot imho | 16:14 |
| dmsimard | mnaser: you consume packages from RDO right ? | 16:14 |
| mriedem | in the form of - do you actually pick up the stable branch patch releases | 16:14 |
| mriedem | as a test | 16:14 |
| ttx | mnaser: do you use any component that actually does intermediary releases ? Swift maybe ? | 16:14 |
| fungi | ttx: i recall at one point zigo was packaging the milestone tags for debian/unstable (or maybe those were going into experimental)... not sure if any other distros did the same | 16:14 |
| mnaser | From a “can I get this into the next release” perspective .. 1 year is so long | 16:15 |
| dansmith | mnaser: two if you miss the first one | 16:15 |
| persia | fungi: I believe that was experimental | 16:15 |
| ttx | mriedem: oh, so you stable stable point releases. not intermediary releases ? | 16:15 |
| ttx | you mean* | 16:15 |
| EmilienM | how mnaser could continue to report so much bugs and good feedback if we wait one year to provide him a major release? | 16:15 |
| mnaser | And if those intermediary releases become ones with features, it makes it even harder to upgrade, which brings us back to square one | 16:15 |
| mrhillsman | 44 release-model: cycle-trailing | 16:15 |
| fungi | persia: yeah, i think you're right. so that he could get security fixes for our release versions through unstable into testing naturally rather than via tpu | 16:15 |
| tdasilva | pabelanger: just for reference: https://wiki.openstack.org/wiki/Swift/version_map | 16:15 |
| mrhillsman | 147 release-model: cycle-with-intermediary | 16:15 |
| mrhillsman | 37 release-model: cycle-with-milestones | 16:16 |
| mrhillsman | 2 release-model: untagged | 16:16 |
| dmsimard | fungi, ttx: FYI RDO packages trunk continuously -- every commit that lands in master and stable branches are immediately packaged and mirrored for consumption, regardless of tags or milestones | 16:16 |
| EmilienM | mnaser: right and we couldn't garantee to CI all cycle-with-intermediary together | 16:16 |
| mrhillsman | [openstack-dev] Upstream LTS Releases - John Dickinson on Nov 10 | 16:16 |
| mnaser | Sorry I’m on mobile I’m a bit slow on answers | 16:16 |
| fungi | dmsimard: so presumably rdo would have little use for milestone tags/intermediate releases anyway? | 16:16 |
| mnaser | But yes we use rdo stable branches and I trust running stable branches 100% because I trust the OpenStack and rdo ci | 16:17 |
| mriedem | ttx: yes the closest nova gets to "intermediate releases" is stable point releases | 16:17 |
| mriedem | so i asked if ops even pick those up | 16:17 |
| mriedem | since they should, | 16:17 |
| mnaser | The stable point releases is just to have any easy number | 16:17 |
| mriedem | because severe bug fixes | 16:17 |
| dmsimard | fungi: from our perspective, our automation already ships tagged releases to stable mirrors, whether there's more or less tags doesn't mean much for us | 16:17 |
| pabelanger | EmilienM: I would imagine switching to CD deployments for new feature branch | 16:17 |
| EmilienM | mnaser: if you have time today, you can reply to the public thread about this topic, so we get your feedback recorded on the ML as well. Thanks | 16:18 |
| EmilienM | pabelanger: easy to say... | 16:18 |
| mnaser | I will try to take sometime to do a write up | 16:18 |
| dmsimard | mnaser: thanks! | 16:18 |
| mnaser | Also honestly OpenStack can just not be deployed with CD imho | 16:18 |
| mnaser | It’s too complex. Too many moving parts. | 16:18 |
| pabelanger | EmilienM: of course, why I think it would be an important project in openstack to actually manage a cloud used by nodepool and CD openstack | 16:18 |
| mnaser | And that weird workaround you did 3 years ago will break your next upgrade in some unexpected way | 16:19 |
| pabelanger | EmilienM: to show it is possible and feedback loop | 16:19 |
| dmsimard | Is there even any operator still *really* operating a production environment on a CD basis off of master ? Last I heard (although I don't know for sure) even RAX doesn't do that anymore. | 16:19 |
| mgagne | from ops perspective, I more or less don't care about release cadence as long as I have a way to catch up/fast-forward to latest versions. Upgrading has so far taken up 1 year to complete and we skipped a version which some projects strongly not recommend. | 16:19 |
| EmilienM | mgagne: what deliverables do you deploy? Major releases? | 16:20 |
| pabelanger | dmsimard: I am not sure, osic-cloud8 is the model I always point too, mrhillsman | 16:20 |
| mgagne | and about CD, to reflect what has been said on the mailinglist already: wet dream. I think I would be the first in line to want to implement CD but I just can't. We don't have the resources. | 16:20 |
| smcginnis | I think we need to drop the idea of CD, but I've stated that before and been yelled at. | 16:20 |
| EmilienM | mgagne: good feedback. | 16:20 |
| dmsimard | pabelanger: osic-cloud as the cloud that we had for nodepool ? | 16:20 |
| EmilienM | smcginnis++ | 16:20 |
| cdent | smcginnis++ | 16:20 |
| mnaser | Given that I can imagine the amount of work involved to test FF upgrades and i don’t think anyone is putting in the resources to do it :( | 16:20 |
| smcginnis | IMO, it causes more problems than benefits. | 16:20 |
| mgagne | EmilienM: I deploy latest version of a major release at the time we decided to upgrade and then we are more or less stuck to that version until next upgrade (we cherry-pick bug fixes) | 16:20 |
| cdent | whatever happened with mordred's thread of packaging the services to pypi? | 16:21 |
| EmilienM | mgagne: thanks for the info | 16:21 |
| edleafe | smcginnis: agree, but CD is like a religion in Nova | 16:21 |
| mrhillsman | yeah, osic-cloud8 was based on our CD stuff | 16:21 |
| mriedem | dropping the idea of CD support means allowing changes knowing they are backward incompatible and broken with the assumption you will fix them before you release | 16:21 |
| mriedem | and ^ is dangerous | 16:21 |
| fungi | cdent: my take was that there was a minimal amount of feedback but no real strong arguments against it | 16:21 |
| EmilienM | so today we know that 2 (big) public clouds are 1) deployming major releases 2) aren't interested by extending a cycle to one year. | 16:21 |
| mrhillsman | honestly the CD was helping to increase feedback and work for osic folks | 16:22 |
| mnaser | We don’t ever run anything other than upstream, we usually make sure things are merged and approved to stable branches | 16:22 |
| * dansmith yells at smcginnis | 16:22 | |
| mnaser | Which is nice because it makes sure nova cores approve our bug fixes before we break things. | 16:22 |
| smcginnis | :) | 16:22 |
| ttx | EmilienM: trying to make sure I captured your concern correctly -- you're saying that 6 months is still the sweet spot between what we can deliver with proper bells and whistles (coordinated, stable branch, etc), and that people do not consume things that don't have those bells and whistles | 16:22 |
| mriedem | holding to a CD standard, regardless of whether or not people are using it, improves stability | 16:22 |
| mriedem | because you actually make people think about not breaking shit | 16:22 |
| mrhillsman | we were doing at least nightly deployments from master and working to do deployments every 6-8 hours | 16:22 |
| mgagne | mriedem: what's CD for you? deploying in a production environment? Or running tempest or whatever against each change? | 16:22 |
| fungi | cdent: and ultimately we'd like services released to pypi because it massively simplifies some sorts of testing to not treat them as special | 16:22 |
| ttx | EmilienM: and anything longer would therefore mean longer feedback cycles, which is bad | 16:22 |
| mnaser | I like 6 months because it lets us pace ourself relatively well. Upgrading every year leaves such a huge period of time for things to go wrong and transitions | 16:23 |
| EmilienM | ttx: you got it | 16:23 |
| dansmith | mriedem: ++ | 16:23 |
| EmilienM | ttx: and we should rather work together at improving our planning methods | 16:23 |
| mriedem | mgagne: pre-prod syncing changes downstream daily (or more than once per day), CI on that sync, and eventually, when you're ready, publish to prod | 16:23 |
| mnaser | Let’s say nova release wants to roll out cells v2 and wants to pace out the roll out over several releases to make it easier for consumers to upgrade | 16:23 |
| mnaser | With 1 year we would have a one big cutover where so much can go wrong | 16:23 |
| smcginnis | mriedem: Doesn't sound dangerous to me. Sounds like you can actually do things with ability to not have to make a lot of design compromises to fix things because someone somewhere might have picked up the bad code. | 16:23 |
| pabelanger | dmsimard: yes, we have 2 at one point. As I understand it osic-cloud1 was slow moving, production cloud. and osic-cloud8 was faster moving, master cloud. Find issues in osic-cloud8, then propose fixes to osic-cloud1 when upgrading | 16:23 |
| mrhillsman | we were using openstack-ansible to deploy, pulling master branches every X hours, and standing up what is traditionally deployed in production for folks starting off as rax private cloud customers | 16:23 |
| mnaser | Or things can take a really long time to happen | 16:23 |
| dansmith | mnaser: indeed, larger spans mean a much larger speed bump at each release | 16:24 |
| mrhillsman | 22 physical nodes basically, ha, etc, the reference architecture publicly available | 16:24 |
| mnaser | Example: transition to adding placement service would have either taken 2 years to do smoothly | 16:24 |
| mriedem | smcginnis: then we shouldn't do microversions either | 16:24 |
| EmilienM | I'm pretty sure y'all read it before, but I can't resist to share it again here : http://www.catb.org/~esr/writings/cathedral-bazaar/cathedral-bazaar/ar01s04.html | 16:24 |
| mnaser | Or it would have been a big 1 change that could go very wrong | 16:24 |
| mgagne | mriedem: ok, well I guess I'm not the one that will be performing CD =) | 16:24 |
| smcginnis | mriedem: Totally different. But we should just have one microversion bump per release IMO. | 16:24 |
| mnaser | I much rather more smaller steps rather than large bigger ones. | 16:24 |
| dmsimard | mriedem: I think there's a distinction between landing stuff that isn't broken (knowing you'll fix it later) and *actually* continuously updating your production environment with real customers and real SLAs. We see it first hand when RAX live migrates VMs in openstack-infra, some of the nodes become unresponsive, etc. | 16:24 |
| ttx | EmilienM: we have a lot of people skipping releases because they can't upgrade every 6 months though. Like SUSE doesn't even package more than a release per year | 16:24 |
| ttx | I think the good answer there is skip-upgrade | 16:25 |
| dmsimard | mriedem: doing CD would mean updating the production without impacting the customers and if you're able to do that, please tell me how | 16:25 |
| EmilienM | ttx: I think the good answer is FFU (Fast Forward Upgrades) | 16:25 |
| mrhillsman | and we were using rally to benchmark and test the deployment | 16:25 |
| ttx | EmilienM: right, what I meant | 16:25 |
| persia | skip-upgrade is indeed the answer to vendor distribution solutions that are less frequent than deployment cycles. | 16:25 |
| mgagne | ttx: if we can get people onboard the skip-upgrade/fast-forward ugprade, I'm all in. | 16:25 |
| EmilienM | ttx: you can't skip anything, but you can go faster | 16:25 |
| pabelanger | mriedem: ++, I don't think we can drop CD also | 16:25 |
| mrhillsman | and working closely on the effort to push skip-level upgrades which i believe is not baked into openstack-ansible | 16:26 |
| mrhillsman | s/not/now | 16:26 |
| mrhillsman | or ffe, cannot remember | 16:26 |
| mnaser | I don’t think anyone is against FFU. I think the problem is who is willing to dedicate and commit the resources to work on the CI needed to manage it. | 16:26 |
| mrhillsman | ffu | 16:26 |
| EmilienM | FFU | 16:26 |
| EmilienM | and not only OSA | 16:26 |
| EmilienM | but tripleo, kolla, etc | 16:26 |
| EmilienM | FFU is a real thing, that both devs & ops would like | 16:26 |
| mnaser | And it’s very nice but if the people who need it don’t put down the resources to do it, then I can’t say much about it :( | 16:27 |
| mgagne | s/like/need/ | 16:27 |
| ttx | EmilienM: I'd like to get to the bottom of why people would not consume releases that don't come with a stable branch though | 16:27 |
| EmilienM | ttx: because they aren't tested properly? | 16:27 |
| mgagne | I *can't* afford NOT skipping version. | 16:27 |
| ttx | EmilienM: what makes them less tested ? | 16:27 |
| dmsimard | ttx: if there are tags instead of branches, this means there no expectations of backports of any kind ? | 16:27 |
| fungi | also ffu testing upstream i think depends on us to solve some other challenges related to (or perhaps even involving) lts since it's going to be quite hard to test upgrading from a version we can no longer land fixes in | 16:28 |
| *** kumarmn has joined #openstack-tc | 16:28 | |
| ttx | dmsimard: yes | 16:28 |
| mnaser | Maybe this is my business mind talking but at the end of the day we all come from organizations with specific business requirements that we work on OpenStack to deliver | 16:28 |
| dmsimard | ttx: then I can see that as a hinderance to that model -- maybe I don't want to ugprade to a new milestone or release but want to benefit from bugfixes to improve the stability of my environment | 16:29 |
| mnaser | I don’t think anyone has ever blocked anything that people haven’t put the effort into doing | 16:29 |
| ttx | dmsimard: ok, just checking that it boils down to that | 16:29 |
| EmilienM | ttx: I guess it's more complex to understand and test properly (not only upstream but downstream as well) | 16:29 |
| dmsimard | ttx: well, that's my opinion, at least | 16:29 |
| persia | ttx: In other contexts, where a project doesn't have stable branches, I have watched folk install mirroring infrastructure and maintain private stable branches: the main argument seems to be that folk don't trust internal developers to maintain long-lived branches, which is important in terms of being able to access the specific code used in production when troubleshooting issues. | 16:30 |
| mnaser | So if X organization needs Y feature, it can work with the others to put up the resources to work together to get they feature tbh. | 16:30 |
| mgagne | ttx: most people don't consume them and it's only when a stable branch is cut that (some) people consume it and report bugs. | 16:30 |
| mnaser | persia: I think those folks should get a bit more involved and instead of duplicating effort maintaining a stable branch, they can do less work with others who can help them. | 16:31 |
| mnaser | But that’s not always easy to change. | 16:31 |
| EmilienM | ttx: one day I would like my organization to ship at every milestone - but we're not here yet. I guess what I'm saying is our current model is fine imho, except we commit too much work in the cycles. | 16:31 |
| ttx | EmilienM: ok that makes sense. I agree that if we can't somehow produce something consumable in the middle of a year-long cycle, then it's just too long for feedback loop | 16:32 |
| persia | mnaser: I share your opinion: the "other contexts" bit was intentional, and related to the fact that some mainline teams are not always downstream friendly (in one extreme case, when a downstream demoed something using code, mainline was deleted from public mirrors, depite licensing). Where cooperation can be done (like openstack), it should be done: that doesn7t mean folk may not still want to be consuming software labeled "stable". | 16:32 |
| ttx | and it sounds like "consumable" implies most of the cycle treats like stable branching | 16:32 |
| mriedem | mgagne: if most people don't consume stable branches from upstream, then why do people care that we keep the upstream stable branches around longer? | 16:33 |
| mgagne | so... to be that guy: is my employer too poor to afford OpenStack since it can't 100% pay for the human resources it needs for development, maintenance and upgrade and perform CD? =) | 16:33 |
| persia | mgagne: No: your employer is only too poor to afford openstack if it cannot pay for the human resources it needs for maintainance of it7s business processes using openstack. | 16:34 |
| cdent | mriedem: I've wondered that. | 16:34 |
| persia | mgagne: Key is to make sure that those allocating resources understand that by assigning resources to work closer to trunk, there is more opportunity for shared benefit, so less overall work may be required for any specific feature. | 16:35 |
| mnaser | I don’t think longer cycles will help those who can’t keep up with upgrades do them any more often | 16:35 |
| mgagne | persia: welcome to the enterprise world and money isn't free. Asking for more isn't working here. | 16:35 |
| dmsimard | OpenStack is free </s> | 16:35 |
| mriedem | i assume the answer is so that when you hit a problem downstream on N-2 branch, hopefully someone has already fixed it for you upstream | 16:35 |
| mriedem | mnaser: totally agree | 16:35 |
| mgagne | persia: sorry, those people don't care about your argument, I tried already. | 16:35 |
| mriedem | mnaser: it's the opposite, IMO, it supports people more that aren't upgrading as often | 16:35 |
| persia | mgagne: My sympathies. | 16:35 |
| mnaser | I think inherently if you look at how much money you’ve saved by getting OpenStack for “free”. The small resources you put into upgrading it and keep it healthy is pennies for what you’re getting. | 16:36 |
| pabelanger | mnaser: I tend to agree, I think all it does it push back the issue another 6 months. However, maybe that 6 months is what people need? | 16:36 |
| mgagne | persia: thanks. now I would like people to understand that I'm not alone in this kind of boat and that I'm doing my best to do what I can with what I'm given. | 16:36 |
| mrhillsman | ^ that statement could be made for a lot of things | 16:36 |
| mrhillsman | mnaser | 16:36 |
| mnaser | I think more time means that the OpenStack install will be even more out of date when upgrade time happens | 16:37 |
| mnaser | And then more issues will happen in the upgrade because it will be such a big upgrade | 16:37 |
| mnaser | Unless we slow down the pace of development but that’s a déterrement for OpenStack. | 16:37 |
| cdent | We've strayed a long long long way from helping part-time contributors. Any ideas on that? | 16:37 |
| mriedem | yes - unless we have self-imposed stability periods within that yearly cycle | 16:37 |
| mgagne | like I said, I don't care about the release cadence as long as I can skip/FFU because some times I will have the "budget" to upgrade. some time I won't and will need to play catch up later. | 16:38 |
| persia | mgagne: I don't understand your position. I have trouble believing it is either "someone else should do all the work and I still get the latest stuff optimised for me" or "don't bother working on openstack: we're looking for something else." | 16:38 |
| smcginnis | Maybe the more frequent release idea would make it easier for part time contributors because there would be less pressure. | 16:38 |
| mnaser | cdent: I kinda jumped into the conversation. I guess you’d like to bring the topic at hand. | 16:38 |
| mgagne | persia: I used to contribute upstream, ask EmilienM. I just don't have the budget anymore. | 16:38 |
| pabelanger | cdent: I think the more we talk about it, I agree it seems less to be about part-time contributors | 16:38 |
| cdent | pabelanger: yes, except that was the original justification, so we're on a strange path | 16:39 |
| EmilienM | persia: mgagne was one of the main contributors to Puppet OpenStack modules | 16:39 |
| mnaser | I do think part time contributors would be disappointed that if they needed something it might be a year till it’s released, which will increase the likeliness of them maintaining their own branch. | 16:39 |
| cdent | mnaser++ | 16:39 |
| persia | mgagne: Sorry if I rubbed salt in wounds. | 16:39 |
| pabelanger | cdent: yes, I agree. As I listen more to the discussions, it feels like we need to keep reminding ourself of that. not sure if that is good or bad | 16:40 |
| persia | But I still believe that organisations that put significant funding into downstream work could likely get more features faster for less investment (this means firing folk) working closer upstream. Working forther downstream keeps more folk employed, but maybe means mainline moves more slowly. | 16:40 |
| EmilienM | mnaser: good feedback | 16:41 |
| mriedem | a yearly cycle does not help a part time contributor unless *everyone* slows down development | 16:41 |
| EmilienM | mnaser: iirc, this proposal was made for part time contributors | 16:41 |
| mriedem | and a yearly cycle does not mean your low priority vendor specific complicated feature that no one wants magically gets priority now | 16:41 |
| EmilienM | mriedem: exactly. Planning. | 16:41 |
| dmsimard | persia: that's assuming there is any downstream work at all | 16:41 |
| mriedem | we just have more time to not care about it | 16:41 |
| mgagne | yea... but life isn't always giving you that opportunity or surround you with likely minded people. | 16:41 |
| EmilienM | mriedem: and we can do planning in 6 months or in 3 months even. Not sure why extending to 1 year is needed | 16:41 |
| ttx | mriedem: it helps them writing code within the timefrane of one cycle | 16:42 |
| EmilienM | ttx: then split the code in smaller pieces | 16:42 |
| mriedem | right, smaller patches, | 16:42 |
| mriedem | and, | 16:42 |
| mnaser | mgagne: im sorry that you're in that position, i understand that there's not much that you can do (and its not what you'd like to do either) | 16:42 |
| persia | dmsimard: If there is no downstream work, doesn't it cease to matter, as the organisation isn't using openstack? If nothing else, I would expect some minor integration and usage. | 16:42 |
| dtantsur | in my practice, writing code is not an issue. getting it reviewed - is. agreeing on a design - sometiems as well | 16:42 |
| EmilienM | maybe developers aren't good a designing a feature into small chunks | 16:42 |
| mriedem | ttx: it's irrelevant | 16:42 |
| mriedem | ttx: you don't get it in the 6 month cycle because no one cares, or you don't get it in the 12 month cycle because no one cares | 16:42 |
| smcginnis | dtantsur: ++ | 16:43 |
| fungi | mriedem: i think a missing part of the problem statement is that development pace _is_ slowing, and so maybe it makes sense to slow our release cadence as well | 16:43 |
| cdent | fungi: it may be slowing in some places, but it certainly doesn't feel like the pressure it lightening in nova | 16:43 |
| mriedem | if development pace is slowing, then shouldn't upgrades be easier? | 16:43 |
| mriedem | because less churn? | 16:43 |
| fungi | mriedem: maybe it doesn't seem to a lot of us who work 100% (or nearly so) upstream because there are fewer and fewer of us to keep up with the load? | 16:43 |
| * mwhahaha doesn't believe things are slowing down in all areas | 16:43 | |
| * dims paying attention | 16:43 | |
| mgagne | mnaser: thanks. FFU is the only way for me to catch up and maybe free up time so I can contribute. (I still contribute when I find bugs or need a feature) Otherwise I'm just stuck, they won't hire people. That's life. | 16:43 |
| dmsimard | persia: the fact that an operator integrates openstack with their customer portal or whatever has no value upstream | 16:44 |
| mriedem | in which specific projects does a yearly cycle help them? | 16:44 |
| mriedem | glance? | 16:44 |
| mgagne | mriedem: one of the aspect slowing down my upgrades is *major* changes in architecture like cellsv2 (because we had the *great* idea to use cellsv1) | 16:45 |
| ttx | Drop data: https://etherpad.openstack.org/p/srxw36lNbL | 16:45 |
| EmilienM | mwhahaha: right I want to know which major projects are slowing down | 16:45 |
| fungi | it does seem to me like openstack is continuing to put pressure on itself to deliver the same amounts of feature and bugfix velocity, but with fewer and fewer people to do it | 16:45 |
| ttx | mwhahaha: ^ | 16:45 |
| mriedem | mgagne: yes i understand that part of your particular pain | 16:45 |
| persia | dmsimard: No, but it's still "work", in the sense that the organisation that does that (usualy) must pay for it to be done. Where there is integration upstream (as in some of the cooperative goals the public cloud team is trying to accomplish), the amount of work per organisation is reduced. Org response might be "hire more folk to add more features faster", but is likely "let folk go: this runs smoothly". | 16:45 |
| dtantsur | ttx: I'd question this data | 16:45 |
| mgagne | it's a *serious* speed bump in the road for us | 16:45 |
| mriedem | mgagne: which we've talked about a few times :) | 16:45 |
| ttx | EmilienM: around 30% drop over past year | 16:45 |
| dtantsur | ttx: was just about to say that ironic is heating up still | 16:45 |
| mriedem | fungi: i agree with that | 16:46 |
| mriedem | however, | 16:46 |
| mgagne | mriedem: yes, I appreciate your help with it. getting there, slowly ;) | 16:46 |
| mriedem | i've been tracking nova bp output since newton, | 16:46 |
| mriedem | and it's steadily going down each release with the loss of major key contributors, as one would expet | 16:46 |
| mwhahaha | ttx: considering we had resoruces go away (companies bailing), i don't think that means there's less work | 16:46 |
| mriedem | *expect | 16:46 |
| EmilienM | ttx: the public thread is huge imho. not sure how we're going to reach consensus here | 16:46 |
| ttx | dtantsur: I can give you detailed numbers for Ironic to back that up:) | 16:46 |
| mriedem | however, we're still cranking out like 50 bps per release | 16:46 |
| dtantsur | ttx: I trust you have numbers right :) I'm not sure these numbers have practical sense | 16:47 |
| mwhahaha | i'd rather we land 3 features every 6 months and get it all tested for upgrades, updates, bugs, etc than push that out to 4 features over 12 months | 16:47 |
| TheJulia | mriedem: the mechanic did not grok the question | 16:47 |
| dtantsur | ttx: I don't see pressure reducing, even though we keep the team roughly the same size | 16:47 |
| dansmith | dtantsur: agree.. a commit is not a uniform thing | 16:47 |
| mriedem | TheJulia: damn his hairy hide! | 16:47 |
| ttx | number of contributors, patchsets proposed, commits merged | 16:47 |
| EmilienM | mwhahaha: or 6 not upgraded tested | 16:47 |
| mwhahaha | i don't see a push to improve the user experiance in the existing features so unless that would be included in this extra 6 months to polish stuff, i see it as more half backed thigns implemented | 16:47 |
| mgagne | EmilienM: hehe | 16:47 |
| fungi | TheJulia: in fairness, i don't yet entirely grok the question either ;) | 16:48 |
| mriedem | mwhahaha: agree | 16:48 |
| dansmith | nova has had such a hugely long tail of contributors that losing 40% of them doesn't seem significant to me in terms of overall project activity | 16:48 |
| TheJulia | mriedem: I'm not sure that is the look she was going for, honestly | 16:48 |
| TheJulia | fungi: good point | 16:48 |
| mriedem | dansmith: not sure i follow that statement | 16:49 |
| mriedem | i'd say losing alaski, sdague and johnthetubaguy are significant impacts | 16:49 |
| ttx | dansmith: so you'd say you haven't lost much "core" activity ? | 16:49 |
| ttx | hm what mriedem said | 16:49 |
| dansmith | ttx: is this number covering cores or number of people with a commit during a cycle? | 16:49 |
| mriedem | i think we are still putting stuff out, at the expense of fewer people loading the burden to do it | 16:49 |
| ttx | dansmith: it's covering number of patchsets proposed, but also number of commits merged | 16:50 |
| dansmith | my point being that if we have 1000 contributors and 500 (or more) of them are one-shot snowflake patches, that losing those contributors does not mean the project is slowing down or dropping in activity | 16:50 |
| dtantsur | ttx: the number of changes merged reducing may mean that pressure is increasing, not decreasing | 16:50 |
| cdent | dansmith++ | 16:50 |
| fungi | i continue to get the impression that the remaining active contributors to a lot of these projects don't perceive a drop in activity because it's been meaning consistent (or increasing) amounts of work for them as contributors continue to peel away | 16:50 |
| dansmith | ttx: right, and that seems like a nonsense metric, so I was arguing about the contributor one | 16:50 |
| mriedem | dansmith: agree with that | 16:50 |
| dtantsur | it may mean that we're not coping with the work. or it may mean that we have less work. | 16:50 |
| dansmith | ttx: our changes have been getting much more complex over time | 16:50 |
| dansmith | ttx: so commit rate dropping doesn't mean much to me | 16:51 |
| ttx | dansmith: indeed | 16:51 |
| mriedem | fungi: i fully perceive it | 16:51 |
| mriedem | a 1 year cycle, however, doesn't fix that imo | 16:51 |
| ttx | I like to think that the 30% drop is more a sign of maturity. Things above that drop are a concern though | 16:51 |
| dtantsur | I like to think that too :) | 16:52 |
| mriedem | there are always going to be contributors that are simply just more interested or engaged in a project and that fuels their desire to want to work on it | 16:52 |
| fungi | mriedem: yeah, i didn't mean to imply that changing the cycle length necessarily solves the contribution drop, more than as development velocity slows it may simply be natural to slow our coordination points as well | 16:52 |
| mriedem | cycle length doesn't change that | 16:52 |
| pabelanger | mriedem: yes, I agree with that | 16:52 |
| mriedem | fungi: ok, but dev velocity isn't slowing in nova in dramatic ways | 16:52 |
| mrhillsman | i think the discussion is cyclical re 1 yr cycle when focusing only on dev aspect | 16:52 |
| mrhillsman | aspects | 16:53 |
| dtantsur | fungi: then we should talk about some 'load' metric, like # of features per contributor | 16:53 |
| mriedem | fungi: which is why i asked which specific projects are getting killed by a 6 month cycle | 16:53 |
| mriedem | the deployment/packaging projects i can understand | 16:53 |
| dtantsur | this is something I feel is not decreasing too much | 16:53 |
| mriedem | but which actual service projects? | 16:53 |
| cdent | fungi: you used the dreaded "just" word in you email! For shame. | 16:53 |
| mrhillsman | i think where it does not change/hurt anything, those should not be basis for y/n | 16:53 |
| *** kumarmn_ has joined #openstack-tc | 16:53 | |
| mrhillsman | 0 review points :) | 16:53 |
| ttx | mriedem: how much activity is lost to boilerplate activities linked to the cycle ? Would you say that's negligible ? | 16:53 |
| mriedem | yes | 16:54 |
| mrhillsman | nothing changes, just good info, fodder even at times, but i think a pro|con , helps|hurts chart may be good to compose or ask in a survey | 16:54 |
| mriedem | i don't know who in nova is doing boilerplate besides me | 16:54 |
| mriedem | if boilerplate == admin stuff | 16:54 |
| dansmith | maybe he means things like spec/code deadlines? | 16:54 |
| mriedem | if boilerplate is coming up with talks for summits, then sure | 16:54 |
| fungi | cdent: oh, so i did! for shame :/ | 16:54 |
| ttx | mriedem: well it's also feature freeze and other deadlines happening twice a year | 16:54 |
| ttx | and PTg prep | 16:55 |
| mriedem | ttx: if anything, feature freeze increases activity | 16:55 |
| dansmith | ptg prep, lol | 16:55 |
| dansmith | mriedem: ++ | 16:55 |
| mriedem | if we don't have a freeze deadline, | 16:55 |
| mriedem | we'll all be fishing | 16:55 |
| dansmith | one FF per year would mean much less activity than two, IMHO | 16:55 |
| dtantsur | but also more pressure from people who see their features missing the deadline | 16:55 |
| dansmith | to the point that we'd _have_ to have another FF per cycle in order to stimulate thing I think | 16:55 |
| mriedem | dansmith: yes i totally think nova would have to do that | 16:56 |
| ttx | dansmith: yeah, that was a question I had actually | 16:56 |
| mriedem | i've said that a few times in the thread and in here i think | 16:56 |
| dansmith | mriedem: but it'll be harder, because we won't have an integrated deadline to hold to | 16:56 |
| dansmith | mriedem: which means we'll be pressured to slip it forever | 16:56 |
| mriedem | nova will still likely have a release at least every 6 months | 16:56 |
| *** kumarmn has quit IRC | 16:56 | |
| mriedem | even if no one consumes it | 16:56 |
| mriedem | i'm not sure how we'd CI that thing wrt upgrades though | 16:56 |
| mriedem | what our support statement would be | 16:57 |
| dansmith | but it'll be a lot of work for just that deadline, because we can't deprecate anything on that intermediate release | 16:57 |
| dansmith | so it'll be a release just to force the activity, but nobody will run it | 16:57 |
| dansmith | _that_ seems like more wasted busywork to me | 16:57 |
| dtantsur | that's what we do: intermediary releases to keep us in shape (minus upgrades ofc) | 16:57 |
| mriedem | it would also unfreeze for new specs for the 2nd half | 16:57 |
| ttx | To come back to what EmilienM said earlier about better planning -- what do you need to do that ? More time ? :) | 16:57 |
| mriedem | i don't think we necessarily need more time to plan | 16:58 |
| ttx | (my proposal implies that relaxing the pressure would lead to better org) | 16:58 |
| mriedem | we need more quality contributors to review and test code | 16:58 |
| mriedem | and operators and users to give feedback <2 years | 16:58 |
| EmilienM | we already have the PTG | 16:58 |
| EmilienM | who does (actual) planning at PTG? | 16:58 |
| ttx | EmilienM: and yet as you said we need to get better at planning | 16:59 |
| EmilienM | like, listing all features, evaluating the work, and tasking | 16:59 |
| fungi | EmilienM: infra did | 16:59 |
| pabelanger | yah, infra has some ptg planing for sure | 16:59 |
| mriedem | we did some planning in denver, | 16:59 |
| ttx | The release team does planning at PTG. But then we are small | 16:59 |
| fungi | but then again, infra doesn't produce deliverables involved in the coordinated release | 16:59 |
| mriedem | but it was mostly about priorities | 16:59 |
| EmilienM | good, so some projects do it | 16:59 |
| dtantsur | EmilienM: kind-of planning, yes | 16:59 |
| mriedem | ptg is about discussing designs and issues for nova, and coming up with what we think we can do in the release and what is our priority list | 16:59 |
| EmilienM | do you feel we plan too much? | 16:59 |
| dansmith | mriedem: I call that planning :) | 16:59 |
| pabelanger | fungi: agree, but with a few people it wasn't too painful I think | 16:59 |
| EmilienM | do we overcommit *sometimes* ? | 16:59 |
| dtantsur | EmilienM: always | 17:00 |
| EmilienM | (innocent question) :-) | 17:00 |
| EmilienM | right. So this is the problem. | 17:00 |
| EmilienM | extending the cycle to one year won't solve this one | 17:00 |
| dtantsur | we nearly do it on purpose to be able to be flexible in case something gets delayed | 17:00 |
| dtantsur | fwiw I don't see it as a problem | 17:00 |
| EmilienM | we'll commit too much for one year and then why not extending to 2 years lol | 17:00 |
| ttx | EmilienM: so we need to get better qualitatively at planning, not necessarily quantitatively ? | 17:00 |
| EmilienM | you know what, we have a ton of features, let's release in 5 years | 17:00 |
| fungi | EmilienM: i would say from the infra team perspective, release cycle planning is more about "when would be the least disruptive time to the developer community for us to make major changes to the infrastructure (lengthy outages for needed upgrades, et cetera) | 17:00 |
| EmilienM | ttx: exactly | 17:01 |
| mnaser | i'm ready for a flood fo answers but -- what's wrong with the current cycle? | 17:01 |
| EmilienM | and train developers to break down features better, so we can land pieces every cycle | 17:01 |
| EmilienM | and not postpone a whole feature to the next cycle | 17:01 |
| cdent | fungi: you're in a unique (in this context) situation where the distance and path between you and your "customer" is close and relatively clear. | 17:01 |
| cdent | the feedback loops are quite tight, that's not the case otherwise | 17:01 |
| ttx | mnaser: I had lots of reports that our rhythm was too fast for part-time devs to jump on the train | 17:02 |
| EmilienM | mnaser: imho, the only thing wrong right now, is we overcommit every cycle | 17:02 |
| pabelanger | EmilienM: so, with 6months window today, what would you think is needed not to overcommit in rocky? Have PTL impose some limit on BP? Or some sort of cutoff timeline for features to be done then drop into S release | 17:02 |
| mnaser | part time devs meaning, able to commit little # of hours OR come-and-fix-one-thing-and-go ? | 17:02 |
| ttx | mnaser: the former | 17:02 |
| mgagne | ttx: is it because they couldn't complete a feature within the 6 months window? | 17:02 |
| dansmith | ttx: I've never understood why the cycle and rhythm means anything for a 20%er | 17:02 |
| fungi | cdent: yes, and it also means that we get the luxury of being able to rely on the release schedule to mostly divine when teams will have the most time to adjust to major changes or outages | 17:02 |
| EmilienM | pabelanger: not only PTL, but the whole team buying it | 17:03 |
| ttx | dansmith: for a part-time PTL it does | 17:03 |
| mriedem | how many projects are in that bucket? | 17:03 |
| dansmith | ttx: so we're looking to change the whole thing so that we can facilitate part-time PTLs? | 17:03 |
| mriedem | glance, trove, what else? | 17:03 |
| EmilienM | mriedem said it, it's hard for a PTL to pushbash features from other contributors | 17:03 |
| ttx | mriedem: more and more | 17:03 |
| EmilienM | mriedem: if I understood correctly | 17:03 |
| fungi | ttx: i wonder to what extent part-time contributors are perceiving the release cadence to drive change velocity, and it's actually the latter which makes getting involved hard? | 17:03 |
| dansmith | because that seems like exactly what I said on the ML: aiming for part-time participation of everything, which means openstack never moves past where we are today, IMHO | 17:03 |
| mnaser | now question -- are part time contributors involved in huge feature changes that will be the sort of change that spans few months? | 17:03 |
| EmilienM | it's a culture and education thing, we need to push | 17:03 |
| ttx | fungi: yeah, maybe | 17:04 |
| mgagne | because if there is overcommitting like EmilienM said, I suspect this could also impact review time and therefore delay merge for those part-time contributors. | 17:04 |
| mnaser | i feel like big changes that take a long time to complete are more taken up by full time contributors | 17:04 |
| mriedem | mnaser: totally | 17:04 |
| mriedem | ex: placement and cellsv2 | 17:04 |
| EmilienM | sometimes I see folks frustrated because their 2000+ LOC patch doesn't land on time for a cycle | 17:04 |
| EmilienM | well, first maybe split it? | 17:04 |
| mriedem | ha yes! | 17:04 |
| mgagne | and like mnaser, need also to see what's the size of the changes contributed by those part-time contributors | 17:04 |
| dtantsur | ++++++++ for split it up | 17:04 |
| pabelanger | EmilienM: right, so would it make sense to have teams cut back X in an effort to simulate having more time to do Y, but in 6 month cycle for Rocky as an experiment | 17:04 |
| EmilienM | we're not robots | 17:04 |
| mnaser | i think its good to look into how big is the size of patches of these part time contributors | 17:05 |
| mriedem | there have to be countless summit talks on how to actually get code merged in openstack that we can link people to yes? | 17:05 |
| mnaser | if they're all few lines bug fixes, dont think the year makes a difference to them | 17:05 |
| ttx | I mean, I've been a hard defender of 6-month cycles, but I also feel like we are optimizing for a group that is disappearing, and preventing its replacement by another group, so I've been trying to put myself in their shoes and interviewing lots of them lately | 17:05 |
| mriedem | if there is a bug fix that the majority of ops need, then show up and tell us to prioritize it | 17:05 |
| mnaser | but (im maybe generalizing here), i dont think part time contributors are the ones delivering 2000 LOC changes | 17:05 |
| dtantsur | I've seen quite big things coming from part-timers in ironic, fwiw | 17:05 |
| dtantsur | including 2000 LOC, yes | 17:06 |
| dansmith | ttx: I think we're a long ways away from the majority of major projects with a part-time ptl no? | 17:06 |
| mnaser | dtantsur: thats awesome to see | 17:06 |
| dansmith | ttx: if we hammer that last nail ahead of time, we're sealing our fate it seems to me | 17:06 |
| persia | One of the organisations I work with is reluctant to assign part-timers, because engaging someone with openstack means they are individually important, so it becomes hard to rotate the assignment in a team. | 17:06 |
| ttx | dansmith: I interviewed more than just PTLs | 17:06 |
| mugsie | yeah - a lot of our large changes come from part timers - they work on a feature in islolation, and then push it up en-mass | 17:06 |
| mgagne | mnaser: hence, are projects overcommitting themselves? do they have the bandwidth to review? 1 year won't fix it imo, changes will just pile up overtime | 17:06 |
| EmilienM | and it's up to projects to say "no, your patch is too big, this isn't how we work" | 17:06 |
| dtantsur | okay, I"m going to look a bad person, but the most common cause of delays in big contributors from part-timers is the quality of submissions | 17:07 |
| ttx | dansmith: like people trying to get involved in Nova and realizing they will never have a say in that project because to be core you need to spend 120% of your time onit | 17:07 |
| EmilienM | and say "we don't want you to be frustrated so if you want the code to land, break it down so it gets merged quicker" | 17:07 |
| dansmith | ttx: okay, but 20% people, even if cores, don't lead the project right? | 17:07 |
| mnaser | mgagne: i agree, changes will pile up. but as a committer, i ask myself if i want to review the code i just submitted | 17:07 |
| mnaser | and if its not, then i split it up | 17:07 |
| mgagne | mnaser: my code is perfect, of course =) | 17:07 |
| EmilienM | dtantsur: can I be the bad person too? you probably didn't do a good job at training newcomers then | 17:07 |
| EmilienM | dtantsur: do you have coding style documented? | 17:07 |
| pabelanger | honestly, zuul makes it much easier IMO to split patches over multiple commit, but maybe new developer don't really understand the power that zuul provides or our testing? | 17:07 |
| EmilienM | dtantsur: or an onboarding guide that says what we expect from contributors? | 17:07 |
| mnaser | maybe we need to document that .. "hey, changes should be ideally X size" | 17:07 |
| dtantsur | EmilienM: it's not about code style even. it about code simply working. | 17:08 |
| ttx | dansmith: I'd like to think people could be core reviewers and part-time contributors. | 17:08 |
| EmilienM | dtantsur: i'm pretty sure if I sent a patch to Ironic right now, you'll -2 my sh**t :-) | 17:08 |
| EmilienM | dtantsur: and you would be right to do it | 17:08 |
| dtantsur | EmilienM: you'd be surprised | 17:08 |
| ttx | dansmith: I realize some projects are a long way from that | 17:08 |
| ttx | but that's another thread | 17:08 |
| dansmith | ttx: I think we have part-time contributor cores | 17:08 |
| dtantsur | EmilienM: I trust you to not send patches that are simply not working and cannot work even in theory | 17:08 |
| mgagne | EmilienM: your change will be forcefully abandoned :D /jk | 17:08 |
| persia | EmilienM: We should consider that when we write code, some is landable, even as a first intro to a project. | 17:09 |
| EmilienM | anyway, what I'm saying is we should train and help our contributors, by giving them the tool and access to knowledge to be better contributors | 17:09 |
| dtantsur | EmilienM: to be clear: this is not about nit-picks in docstrings | 17:09 |
| EmilienM | dtantsur: good, trust is critical here | 17:09 |
| mnaser | i agree with EmilienM. if the problem is 2000 LOC patches, educate people to split their changes into smaller more reasonable ones | 17:09 |
| ttx | dansmith: people who spend less than one day on Nova per week ? | 17:09 |
| cdent | ttx, dansmith : if that's another thread it is a very closely related thread | 17:09 |
| mriedem | ttx: yes | 17:09 |
| mriedem | ttx: john and sean | 17:09 |
| EmilienM | how many times I've -1 patches without even looking at what it does, just because 2000LOC? | 17:09 |
| cdent | kenichi? | 17:09 |
| ttx | oh, so emeritus core | 17:09 |
| dansmith | ttx: maybe not that part-time, but considering the size and complexity of nova, I think proportionally we're covered there | 17:09 |
| EmilienM | I usually jump on IRC and try to engage with the author | 17:09 |
| mriedem | yeah kenichi more and more | 17:10 |
| EmilienM | and be like "hey, thanks for your work, you're awesome. But the patch is too big, can we split it?" | 17:10 |
| ttx | people who used to be 100% | 17:10 |
| mriedem | to be core doesn't mean you have to be full time | 17:10 |
| EmilienM | that's all, it takes 2 min and it avoids frustration | 17:10 |
| ttx | and can continue helping | 17:10 |
| mgagne | EmilienM: although I have been away from puppet for far too long, zuul helped me catch up stuff I missed in my recent changes. so there is that =) | 17:10 |
| mriedem | it means you have to be good at REVIEWS | 17:10 |
| ttx | I'm more trying to attract new people | 17:10 |
| mriedem | here is an example, i'm goign to pick on mel | 17:10 |
| dansmith | ttx: but, if the goal here is to slow nova to the pace at which a 20%er wants, then we're really just making a choice there and not dealing with any sort of overwhelming reality | 17:10 |
| mriedem | http://stackalytics.com/report/contribution/nova/90 | 17:10 |
| dtantsur | EmilienM: except that some people thing that 5x patches will take 5x time to land.. and occasionally they are right | 17:10 |
| mriedem | 56 reviews in 90 days | 17:10 |
| mriedem | however, when mel does a review, it's effing solid | 17:11 |
| mriedem | that's why mel is core | 17:11 |
| persia | I think attracting new people. retaining old people, and encouraging cross-project contributions are the same problem: the barrier to landing a patch when one isn't current about project internal communications is very high. | 17:11 |
| mriedem | you can be +1ing changes every day of the week - that doesn't get you on the core team | 17:11 |
| persia | If we can reduce the level of engagement required to land things, that would help several sorts of part-time contributors. | 17:11 |
| mnaser | in that same subject, nitpicky reviews will kill engagement of part time developers | 17:12 |
| mnaser | a full time committer doesnt mind fixing a typo he made in the comment | 17:12 |
| mnaser | a part time dev just gets super frustrated with that | 17:12 |
| EmilienM | dtantsur: not sure about that, but I'm maybe biased by tripleo | 17:12 |
| pabelanger | mnaser: why not push up a patch to fix the nitpick? if you know a part-time dev | 17:12 |
| ttx | dansmith: no the goal here was to see if reducing the pressure a bit would lead to better results. I can see that you think it would have more drawbacks than benefits | 17:12 |
| dtantsur | EmilienM: well, at least if CI is not stable (speaking of tripleo ;), it may mean 5x rechecks | 17:13 |
| ttx | which is good input, and why I started the thread | 17:13 |
| dtantsur | mnaser: ++++ we need a culture of NOT nitpicking people to death | 17:13 |
| mnaser | pabelanger: we're dealing with people. ownership and pride goes into "hey i just merged my first change in openstack!" | 17:13 |
| mnaser | it's very normal to us. it's entirely different when others do it for the first time | 17:13 |
| persia | pabelanger: Why should you have to know the person to do that? | 17:13 |
| pabelanger | mnaser: agree, there is a line I would say | 17:13 |
| EmilienM | dtantsur: touché | 17:14 |
| ttx | dansmith: so thanks for sharing | 17:14 |
| dtantsur | we're trying to practice allowing people to follow-up with fixes for their nitpicks, if the patch is close to landing | 17:14 |
| EmilienM | dtantsur: now I go to cry. | 17:14 |
| * cdent hugs EmilienM | 17:14 | |
| dtantsur | EmilienM: don't :( | 17:14 |
| mnaser | hell, i remember my first change in nova in 2011. it was "hey, that's awesome." -- and makes you want to come and do better or soo | 17:14 |
| EmilienM | lol | 17:14 |
| *** jungleboyj has joined #openstack-tc | 17:14 | |
| ttx | mriedem: same -- thanks for caring and sharing | 17:14 |
| EmilienM | ttx: what's the next steps? do we summarize our today's discussion? the thread is already *huge* | 17:15 |
| smcginnis | mriedem is a caring and sharing kind of guy. | 17:15 |
| ttx | I have a more complete (and complex) picture of the problem now | 17:15 |
| * jungleboyj is wishing I hadn't missed this discussion. | 17:15 | |
| pabelanger | persia: yah, I can see people taking offense of updating a patchset. Thinking of my experiences in infra, we all have enough stackalytics points already so often we just opt for another person to work on a patch if needed to move it along | 17:15 |
| pabelanger | or trivial rebase | 17:16 |
| mriedem | mnaser: agree - and it's totally fine to fix a thing yourself and then +2, or do the follow up - people don't realize that because they don't want to step on toes | 17:16 |
| EmilienM | mgagne, mnaser : thanks for joining us and representing ops world today ;-) it really helped. | 17:16 |
| ttx | EmilienM: let's see if the thread calms down. The discussion here was good, avoided a lot of back-and-forth between you, me and mriedem I think | 17:16 |
| mgagne | EmilienM: =) | 17:16 |
| mnaser | aha, for the fun of it, i found the first commit i ever have in nova | 17:16 |
| mnaser | how did this ever get merged | 17:16 |
| mnaser | https://github.com/openstack/nova/commit/9e7c7706a76ad76612ba75314d436a8ba419a3eb | 17:16 |
| mnaser | :P | 17:16 |
| EmilienM | 2011! | 17:17 |
| mriedem | mnaser: becaues it was a TypeError otherwise | 17:17 |
| mriedem | *because | 17:17 |
| * EmilienM switches on something else now, thanks for the nice chat | 17:17 | |
| mriedem | how it got merged without a TEST is another question | 17:17 |
| mnaser | mriedem: https://github.com/openstack/nova/commit/eba9e4abd6271e0265899a2d260b54068d78ee51 | 17:17 |
| mnaser | :P | 17:17 |
| *** alex_xu has quit IRC | 17:17 | |
| persia | pabelanger: I think worrying about causing offense means that we end up crushing people who don7t know how we work (or have time to do it if they do). Needing to do a complex rebase because of a typo is a frustrating first experience, and the one likely to be received if we only help those we know. | 17:17 |
| EmilienM | mriedem: it wouldn't happen if at that time we had one year cycle. Ok I'm out now ;-à) | 17:18 |
| *** alex_xu has joined #openstack-tc | 17:18 | |
| mnaser | but np, thanks for the discussion. and I agree mriedem, im all for follow up fixes (funny, thats exactly how i got started) | 17:18 |
| mnaser | merged something with no tests, follow up added tests | 17:18 |
| TheJulia | dtantsur: I think part of the conundrum is different perceptions of "ready to land" :\ | 17:18 |
| mnaser | who knows if i would be here if i was told "go write some tests" :D | 17:18 |
| mriedem | persia: yup - need the core team to help tell people it's not cool to -1 for a typo in a big series or complicated change | 17:18 |
| mriedem | it's a culture thing | 17:18 |
| dtantsur | TheJulia: correct | 17:18 |
| edleafe | win 13 | 17:19 |
| edleafe | doh! | 17:19 |
| dtantsur | edleafe: no such window | 17:19 |
| TheJulia | heh | 17:19 |
| persia | mriedem: I would argue we should go further, and push an update to the change with the typo fixed, expecting the original submitter to participate in review. key is preserving the Author: header, so credits go to the right place when it finally lands. | 17:20 |
| mriedem | sure the author shouldn't change | 17:21 |
| dtroyer | ttx: I think I'm unclear on the notion of 'attracting new people" in the world of the vast majority of us working on OpenStack are corporate dev being assigned. Who is making that decision to be attracted? What I've seen from the 3 companies I've been a part of during this time is that really is not the individual contributors decision, except when they decide to move on. | 17:23 |
| ttx | dtroyer: I mean getting ops and users more directly involved upstream | 17:23 |
| dtroyer | I have little direct experience with operators here, maybe that is where this happens? | 17:24 |
| ttx | We have hundreds/thousands of users, if each would commit one part-time dev we'd be in another story | 17:24 |
| dtroyer | still though, how many are able to make that decision on their own? | 17:24 |
| ttx | and yet we don't, so I tried to follow that lead | 17:24 |
| ttx | and asked them | 17:24 |
| cdent | dtroyer: I think that's the underlying thing here. The contributor map needs a backup. | 17:24 |
| cdent | but yes, it's not really an individual decision, is it? | 17:25 |
| ttx | dtroyer: I asked them what is preventing them | 17:25 |
| ttx | a surprising part of them replied, the rhythm is too fast for us to jump on that train | 17:25 |
| dtroyer | ttx: the potential contributors or the management of their employers? | 17:25 |
| cdent | I think we way underestimate the extent to which openstack is an joint collaborative operation by enterprises | 17:25 |
| ttx | some replied: my boss won't allow it | 17:25 |
| ttx | but not that many | 17:25 |
| dtroyer | also, how much of this is really only about the Big Three (Nova, Cinder, Neutron)? | 17:26 |
| dtroyer | is the cadence too fast for someone who wants to add a process to Heat? | 17:26 |
| ttx | dtroyer: actually most of the people said that the Big Three are so much out tehre that they don't even think of jumping on that train. Most were talking about getting involved in smaller projects | 17:27 |
| TheJulia | wait, is it code, features, cadence of process, or context that moves too quickly? | 17:27 |
| ttx | a mixture. I'd say it boils down with keeping up with what's happening | 17:27 |
| ttx | which arguably won't get much easier with longer cycles | 17:28 |
| dtroyer | TheJulia: /me think back to contributing to other projects over the years, it's just keeping up with only 2-8 hours a week tp spend. None of those were my $DAY_JOB | 17:28 |
| dtroyer | I don't think OpenStack attracts many of those people at all | 17:29 |
| ttx | I also asked some of the pTLs who stepped up recently, and they mentioned taht cycles were short and they could not get anything done in one | 17:29 |
| smcginnis | ttx: Was Nick Barcet's ML response the one that you referred to a few times earlier? | 17:29 |
| ttx | it was release time already, that kind of thing | 17:29 |
| ttx | which made me think 9-month cycles might be a calmer, more appeased option | 17:29 |
| ttx | but then 9 months is tricky to organize anything around | 17:30 |
| cdent | (dtroyer good message on the thread, cuts deeply, appropriately) | 17:30 |
| fungi | a lot of it is also familiarity with contributing to large free software projects. some of them move quickly, and you need to provide as much context with your contribution as possible, attempt to at least get some minimal familiarity with the community there, and have a lot of patience. i'm reminded of patches i've gotten into gerrit, or python packaging tools, or... | 17:30 |
| ttx | smcginnis: yes | 17:31 |
| fungi | i expect a lot of new contributors who have experience contributing to free software in the past have only experienced it with small, slow-moving projects | 17:31 |
| ttx | anyway i need to go -- thanks for the discussion all, that really helps | 17:31 |
| fungi | and there's only so much we can do to provide that same atmosphere in openstack | 17:31 |
| dtroyer | fungi: we both have those experiences as individual contributors. What I hear ttx is saying is that we really need to focus on individual corporat-sponsored contributors. Less self-motivation, but still enough to be an issue | 17:31 |
| ttx | despite what some people imply I'm just looking for solutions and improvements | 17:31 |
| dtroyer | thanks for getting this rolling ttx, we needed an mega-thread before the holidays! | 17:32 |
| ttx | because saying we don't have a problem or we can't change anything is not getting us anywhere | 17:32 |
| dtantsur | ++ it is very helpful to reflect on something we've been taking as granted | 17:32 |
| fungi | it's all ultimately so we have something to read while vacationing, right?> | 17:32 |
| ttx | dtroyer: actually the -dev ML stats showed a drop in 2017, so I'm trying to fix that | 17:32 |
| fungi | heh | 17:32 |
| mriedem | fungi: when i was new to openstack, having sdague, dansmith and cyeoh was key | 17:33 |
| dtroyer | my Twitter stats show the same thing | 17:33 |
| mriedem | to tell me what not to do | 17:33 |
| * dtroyer goes to fix that too | 17:33 | |
| ttx | a couple more hundreds replies and we should be good | 17:33 |
| fungi | mriedem: yep, i expect we've also lost some of that mentorship drive in more recent years as we all get overwhelmed by the volume of activity | 17:34 |
| cdent | mriedem: if I recall correctly you were able to devote that time to being mentored because you were willing/able to use up "extra" time for it? | 17:34 |
| mriedem | cdent: opposite, | 17:34 |
| mriedem | i was working over time by choice because i loved working on it | 17:34 |
| cdent | yes, that's what I meant | 17:34 |
| mriedem | i went from packaging openstack and internal CI to doing upstream dev | 17:34 |
| mriedem | with mentoring from sean, dan and chris | 17:34 |
| mriedem | but, | 17:34 |
| cdent | you _chose_ to work _over_ time | 17:35 |
| cdent | which is great, good you're here | 17:35 |
| mriedem | part of my yearly goal from my boss was "work on nova" | 17:35 |
| cdent | but is that the sort of thing we can/want-to require? | 17:35 |
| dansmith | there was no upstream university or any of those thousand "how to contribute" resources when mriedem started, I might add | 17:35 |
| mriedem | i don't expect it's a requirement | 17:35 |
| dansmith | nor when I started | 17:35 |
| mriedem | as i said much earlier, the people that become maintainers are doing it because they actually really enjoy it | 17:36 |
| cdent | nor me, but we're all unique snowflakes who are either allowed or able to devote absurd amounts of time | 17:36 |
| mriedem | despite the pains in the ass | 17:36 |
| cdent | and I think that's fine | 17:36 |
| fungi | i'll admit to similar desires. i like free software and already did a lot of free software contribution in my personal time (not my employer's time) and saw taking a job working on openstack as an opportunity to quite my day job and do free software all the time | 17:36 |
| fungi | s/quite/quit/ | 17:36 |
| cdent | but it means that anything "we" want or need or find useful, is not going to be aligned with other folk, is it? | 17:36 |
| fungi | i think the same goes no matter who you are | 17:37 |
| mriedem | i don't understand the question | 17:37 |
| fungi | i will agree that my driving desires aren't likely aligned with those of a majority of others, but the same could likely be said of just about any single individual | 17:38 |
| mriedem | there are lots of things i review and spend my day working on which i don't need, nor does my employer care about | 17:38 |
| cdent | mriedem: the experiences of mriedem, dansmith, cdent are not good guides for helping to encouage or enable people, who through no faul of their own, have to be "part time" | 17:38 |
| mriedem | but i consider ^ my duty and responsibility for having the opportunity to be mostly full time upstream | 17:38 |
| cdent | exactly | 17:38 |
| mriedem | cdent: i'm not sure what we're trying to enable or encourage | 17:39 |
| mriedem | with part time people | 17:39 |
| mriedem | if it's "hey i have 4 hours per week to work on nova, and i want to be core, please invest your time in mentoring me" | 17:40 |
| mriedem | that's gonna be tough | 17:40 |
| cdent | I'm not entirely sure either, other than the general notion of making it more possible for more of them to contribute | 17:40 |
| fungi | we all come to this from different perspectives and backgrounds, and what will make us successful leaders is if we have an ability to understand and respect the positions others are coming from | 17:40 |
| cdent | yeah, I don't think it's that | 17:40 |
| mriedem | the person that did this bp in queens https://specs.openstack.org/openstack/nova-specs/specs/queens/approved/rebuild-keypair-reset.html | 17:40 |
| mriedem | was totally part time | 17:40 |
| mriedem | and it got done because there was shared understanding and value in seeing it happen | 17:40 |
| mriedem | the reason https://specs.openstack.org/openstack/nova-specs/specs/queens/approved/scaleio-ephemeral-storage-backend.html is going release to release, | 17:41 |
| mriedem | is because there is not shared understanding or value in seeing it happen | 17:41 |
| fungi | i know i've said it before, and i'm not the only one, but we need to do a better job of highlighting stories like those | 17:41 |
| mriedem | both are part time contributors | 17:41 |
| cdent | mriedem: much like you're not entirely sure what I'm trying to say, I'm not clear on what you're trying to say | 17:41 |
| fungi | people who want to get involved need examples of what works | 17:41 |
| mriedem | https://specs.openstack.org/openstack/nova-specs/specs/pike/implemented/nova-support-attached-volume-extend.html was mgagne who is part time dev, and i helped because i understood the shared need | 17:42 |
| mriedem | cdent: i'm saying i think it is possible for part time people to contribute | 17:42 |
| dansmith | I think what mriedem is saying is that the shared understanding, shared need, and shared feeling of importance is like 90% of what gets something landed | 17:42 |
| cdent | I don't think anyone is saying that it is not possible. | 17:42 |
| mriedem | the success of their contribution depends on what and how they are going about that contribution | 17:42 |
| mriedem | "cdent: I'm not entirely sure either, other than the general notion of making it more possible for more of them to contribute" | 17:42 |
| cdent | _more_ | 17:43 |
| mriedem | ok, i don't know how to make it more possible to propose code | 17:43 |
| mriedem | visibility of that code is another issue, | 17:43 |
| cdent | right, which is why I think it is important to stop being distracted from band-aid ideas like extending the cycle, and focus more on the core issue | 17:43 |
| mriedem | if we did slots then that might fix that issue | 17:44 |
| cdent | might do | 17:44 |
| mriedem | e.g. slots: priority A -> priority B -> vendor snowflake no one cares about -> priority C, etc etc | 17:44 |
| mriedem | kanban | 17:44 |
| mriedem | it would at least make things more fair | 17:44 |
| ttx | fungi: ++ | 17:45 |
| * cdent nods | 17:45 | |
| dtantsur | mriedem: we do something slightly reminding it with our weekly priorities | 17:45 |
| dtantsur | we have a slot for each in-tree vendor to propose their patch | 17:46 |
| fungi | dtantsur: is there some expectation that you'll actually merge the vendor-proposed feature enhancements? or is this mostly for helping prioritize bug fixes for drivers? | 17:48 |
| dtantsur | fungi: only for prioritizing | 17:48 |
| dtantsur | no promises made | 17:48 |
| fungi | seems reasonable enough | 17:49 |
| mriedem | i'll also say, i fully realize that the 'vendor need' in nova in no way compares to cinder, neutron or any other project that supports a bazillion vendor drivers in tree | 17:49 |
| mriedem | making it a huge distraction to focus on core functionality | 17:49 |
| *** dtantsur is now known as dtantsur|afk | 18:01 | |
| mnaser | why dont the vendor drivers live out of tree and focus on implementing an interface | 18:08 |
| mnaser | kinda like nova's virt drivers (easier said that done obviously) | 18:08 |
| fungi | nova's probably the most extreme example of why doing that is complicated | 18:10 |
| fungi | some other teams do already either have all drivers except a reference driver out of tree, or some mix of in-tree and out-of-tree | 18:11 |
| fungi | i have only the most shallow understanding of nova, but i get the impression that abstracting away the differences between different hypervisors is nontrivial to begin with (and where it can be, libvirt is already filling much of that gap on its own)? | 18:12 |
| fungi | when people have tried to maintain their own out-of-tree hypervisor integration for nova, it's rarely managed to keep up with changes to nova's internals | 18:15 |
| mriedem | oh i also wanted to say, i think a bigger help to part time contributors is actually documenting what's going on, be that specs, forum/ptg session recap summaries, and weekly digests of upstream activity, like in keystone and what cdent does for placement in nova | 18:17 |
| mriedem | ^ is useful for full time contributors as well | 18:18 |
| mriedem | because i forget what it was i said about something 3 months ago | 18:18 |
| cdent | mriedem: thankfully there's been a pretty strong acknowledgement of that recently. In many of the discussion of "how to deal with new and part timers", "write shit down" has been a biggie | 18:18 |
| cdent | and you're right that most of the problematic contributors are the ones that will never have read that stuff | 18:19 |
| cdent | they are _too_ part time | 18:19 |
| mriedem | also useful for apac full timers | 18:19 |
| cdent | fungi: nova virt driver interface officially declared not-stable, isn't it? | 18:20 |
| cdent | out of tree drivers undesirable | 18:20 |
| cdent | (which I think is a shame, but is the current state of affairs) | 18:20 |
| fungi | cdent: yep, that's more or less what i was trying to say in answering mnaser | 18:21 |
| dtroyer | cdent, mriedem: is the interface to nova-compute just as explicitly not-stable? The fact that there is a network boundary makes it at least more intentional | 18:23 |
| mriedem | nova-compute has a stable rpc interface | 18:23 |
| mriedem | there is no stable interface between nova ComputeManager and the virt driver | 18:24 |
| mriedem | we can add/remove/change virt driver method signatures at will | 18:24 |
| dtroyer | ok, that's what I thought. | 18:24 |
| mriedem | it doesn't happen often, but it can happen | 18:24 |
| mriedem | there are several virt drivers that run successfully out of tree with i think minimal impact | 18:24 |
| dtroyer | how crazypants would it be to build a nova-compute that only did, say, libvirt and bypass the internal interface? | 18:24 |
| mriedem | efried and jichenjc can keep me honest | 18:25 |
| mriedem | dtroyer: not a priority | 18:25 |
| dtroyer | ie, draw the line at the rpc boundary? | 18:25 |
| mriedem | spending core time on something like this is wayyyyyyyyyyy down on the priority list, IMO | 18:25 |
| * TheJulia opens up the entire thread and begins reading | 18:25 | |
| dtroyer | I'm not asking about priority, I am curios about feasability and gut reactions to the idea | 18:25 |
| mriedem | dtroyer: i'm not sure i understand the idea honestly | 18:26 |
| dtroyer | becasue a hypothetical hypervisor vendor could take on all of that work given a stable-enough interface | 18:26 |
| fungi | dtroyer: idea being to treat libvirt as the plugpoint at tell people who want to support their own hypervisors to just go talk to the libvirt community? | 18:26 |
| dtroyer | the notion of splitting off nova-compute into its own thing has been raised before | 18:26 |
| dtroyer | not libvirt, the nova RPC interface | 18:26 |
| fungi | ahh | 18:26 |
| dtroyer | I'm not suggesting that the nova team do this, I'm just trying to sort out where the natural stable interfaces exist | 18:27 |
| * cdent passes TheJulia some coffee, some reading classes, some light soothing music, a salty snack and a blanket | 18:27 | |
| TheJulia | cdent: <3 | 18:28 |
| mriedem | dtroyer: the rpc interface is to the ComputeManager, not the ComputeDriver | 18:28 |
| mriedem | the compute manager does the orchestratoin | 18:28 |
| mriedem | because "build me an instance" is a shit load of other stuff than just calling driver.spawn( | 18:29 |
| *** harlowja has joined #openstack-tc | 18:29 | |
| mriedem | excuse my salty snacklike language | 18:30 |
| dtroyer | sure. how much of that happens on the compute node? its the interface to nova-compute process that I am thinking about. I have not looked at this since before DB access was taken away from it so I'm way out of date here | 18:30 |
| mriedem | dtroyer: 95% | 18:30 |
| mriedem | if we moved the volume and network stuff to conductor | 18:31 |
| mriedem | like we've talked about for ages | 18:31 |
| mriedem | that would help | 18:31 |
| mriedem | but the contributors working on moving network thingies to conductor were osic | 18:31 |
| mriedem | RIP | 18:31 |
| * dtroyer pauses for a moment | 18:32 | |
| edleafe | cdent: "reading classes"? What are you implying about TheJulia? | 18:33 |
| TheJulia | lol, I groked it as glasses | 18:34 |
| cdent | and that's how it was meant | 18:34 |
| cdent | as we all know, I type by engrams | 18:34 |
| cdent | and they are broken | 18:35 |
| cdent | oh noes, yet another one of my typos forever engraved into the twitter memory | 18:37 |
| TheJulia | muahahahaha | 18:38 |
| fungi | mriedem: which out-of-tree hypervisor backends are currently successful? are xenserver and zvm out-of-tree? | 18:44 |
| mriedem | xen is in tere | 18:44 |
| mriedem | *tree | 18:44 |
| mriedem | powervm and zvm and lxd i'd say | 18:45 |
| mriedem | powervm and zvm are working on getting in-tree | 18:45 |
| mriedem | lxd has never broached the subject | 18:45 |
| fungi | ahh, i assumed xenserver didn't use the in-tree xen support | 18:45 |
| fungi | since it's a separate (proprietary) thing | 18:45 |
| mriedem | i don't know who actually uses the xenserver driver outside of rax | 18:45 |
| fungi | i didn't realize rax used xenserver either, thought they just used xen | 18:46 |
| mriedem | and i assume rax is still on juno | 18:46 |
| fungi | vaguely remember BobBall doing third-party ci for citrix xenserver integration | 18:47 |
| mriedem | yeah the citrix team is still maintaining it | 18:47 |
| fungi | but no clue if that's still there | 18:47 |
| fungi | ahh | 18:47 |
| mriedem | yup | 18:47 |
| mriedem | alive and kicking | 18:47 |
| mriedem | adding vgpu support in queens | 18:47 |
| fungi | but that's separate from the xen support, right? | 18:48 |
| mriedem | how they get paid idk | 18:48 |
| mriedem | libvirt+xen is different yes | 18:48 |
| fungi | k | 18:48 |
| mriedem | hell we have libvirt+lxc | 18:48 |
| mriedem | and libvirt+uml | 18:48 |
| mriedem | 0 idea if those work anymore | 18:48 |
| fungi | so people doing vanilla xen with nova are generally going through libvirt, but if they want to use xenserver instead the driver for that is in-tree in nova? | 18:48 |
| mriedem | yup | 18:48 |
| fungi | cool | 18:49 |
| * dtroyer remembers uml, retreats back under his rock | 18:51 | |
| fungi | hey, i ran some very successful production virtualization on uml | 18:52 |
| fungi | for many years | 18:52 |
| fungi | if memory serves, linode based their service on it for a long time as well | 18:53 |
| fungi | i mean, the only competition for it in linux back then was chroot ;) | 18:54 |
| fungi | but yes, i can't imagine anyone actually using it with openstack today | 18:54 |
| fungi | the only real competition for uml across the free software spectrum back then was the jails implementation on freebsd, for that matter | 18:55 |
| fungi | (which was also remarkably awesome) | 18:55 |
| fungi | i think my initial foray into uml was when i decided we needed to separate our authoritative and recursive dns resolvers, but management wouldn't give me budget for twice as many nameservers (and bind views were nearly impossible to use to accomplish that at the time), so i carved the nameservers up into uml guests for each role | 18:58 |
| fungi | back in the days when cache poisoning attacks were still a relatively new threat | 18:58 |
| *** openstack has joined #openstack-tc | 20:32 | |
| *** ChanServ sets mode: +o openstack | 20:32 | |
| *** pabelanger has quit IRC | 20:39 | |
| *** pabelanger has joined #openstack-tc | 20:39 | |
| *** SamYaple has quit IRC | 21:07 | |
| *** SamYaple has joined #openstack-tc | 21:08 | |
| *** openstack has joined #openstack-tc | 21:11 | |
| *** ChanServ sets mode: +o openstack | 21:11 | |
| *** SamYaple has quit IRC | 21:22 | |
| *** SamYaple has joined #openstack-tc | 21:23 | |
| *** diablo_rojo has quit IRC | 22:04 | |
| *** diablo_rojo has joined #openstack-tc | 22:07 | |
| * cdent waves goodnight | 22:12 | |
| *** cdent has quit IRC | 22:12 | |
| *** ChanServ has quit IRC | 22:17 | |
| *** kumarmn_ has quit IRC | 22:20 | |
| *** ChanServ has joined #openstack-tc | 22:24 | |
| *** barjavel.freenode.net sets mode: +o ChanServ | 22:24 | |
| *** ChanServ has quit IRC | 22:28 | |
| *** ChanServ has joined #openstack-tc | 22:31 | |
| *** barjavel.freenode.net sets mode: +o ChanServ | 22:31 | |
| *** kumarmn has joined #openstack-tc | 22:39 | |
| *** kumarmn has quit IRC | 22:41 | |
| *** kumarmn has joined #openstack-tc | 22:41 | |
| *** kumarmn has quit IRC | 23:24 | |
| *** kumarmn has joined #openstack-tc | 23:25 | |
| *** kumarmn has quit IRC | 23:30 | |
| *** harlowja has quit IRC | 23:32 | |
| *** harlowja has joined #openstack-tc | 23:39 | |
| *** harlowja has quit IRC | 23:40 | |
| *** harlowja has joined #openstack-tc | 23:42 | |
| harlowja | woah, to much to read, lol | 23:46 |
| *** harlowja has quit IRC | 23:50 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!