15:01:51 <fungi> #startmeeting tc
15:01:52 <openstack> Meeting started Thu Jun 14 15:01:51 2018 UTC and is due to finish in 60 minutes.  The chair is fungi. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:01:53 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:01:56 <openstack> The meeting name has been set to 'tc'
15:01:56 <cdent> tc-members office hours
15:01:57 <fungi> #topic Office Hour
15:02:11 <ttx> Been spending a few cycles this afternoon reviewing https://docs.openstack.org/project-team-guide/
15:02:13 <fungi> we've been using "tc" because that's what the index on eavesdrop links to for our office hour logs
15:02:31 <ttx> One things that it's missing is a plug for basic design tenets, like the fact that you can assume teh base services will be around
15:02:48 <ttx> Anyone interested to etherpadding that with me ?
15:03:15 <dhellmann> o/
15:03:28 <ttx> cdent/zaneb: that could address some of the things we wanted to add to technical vision too
15:03:36 <cdent> yeah, agree
15:04:07 <pabelanger> o/
15:04:11 <cdent> I can't help etherpadding today but either can help tweak, or participate tomorrow afternoon/monday
15:04:23 <fungi> sure, i've had my head in the base services space recently, happy to spitball some content for that
15:04:31 <ttx> Will be at https://etherpad.openstack.org/p/project-team-guide-basic-design-tenets-chapter
15:05:43 <zaneb> in related news, the technical vision thing is getting closer to the top of my to-do list
15:05:58 <zaneb> I know y'all will be excited to hear this update about the state of my to-do list
15:06:05 <smcginnis> :)
15:06:13 <cdent> it's the only thing I think about
15:07:31 <cdent> If we've got nothing else in particular to talk about today, I'd like to selfishly draw people's attention to jay's rant in http://lists.openstack.org/pipermail/openstack-dev/2018-June/131498.html
15:07:44 <EmilienM> zaneb: couldn't sleep, waiting for it
15:07:53 <cdent> that's a long standing nova problem that we keep talking about in a sideways fashion
15:07:56 <cdent> but it never goes away
15:07:59 <EmilienM> zaneb: you have the etherpad handy right?
15:08:12 <cdent> (not talking in this instance about the number of cores, but rather the volume of change in a complex setting)
15:08:51 <zaneb> EmilienM: always, I keep it next to my bed. https://etherpad.openstack.org/p/tech-vision-2018
15:09:41 <mnaser> cdent: that was a good read.  i'd love to see nova do something like split into smaller teams/packages for drivers for example
15:09:41 <smcginnis> cdent: I missed that one.
15:10:03 <cdent> mnaser: yeah. like jay's comments, your response is not new :)
15:10:04 <mnaser> nova-libvirt, nova-vmware, etc.  rather than having the entire tree handled by the core team for changes that may not really touch the internals
15:10:04 <dims> o/
15:10:24 <smcginnis> It really does seem like nova should be decomposed somehow. Whether that's structurally or just process/team-wise.
15:10:29 <cdent> mnaser: which I think means it is probably the right thing to do, but it never happens
15:10:55 <mnaser> can gerrit work with some sort of acl that you can only vote on files within a specific path?
15:10:58 <mnaser> (as a start)
15:11:40 <EmilienM> @all did everyone had a chance to put their names on https://wiki.openstack.org/w/index.php?title=OpenStack_health_tracker ?
15:11:52 <EmilienM> I think dhellmann wanted to do it for today
15:12:01 <dhellmann> mnaser : unfortunately, no. we have had to rely on teams to trust each other for that level of granularity
15:12:18 <dims> yep, i did EmilienM
15:12:28 <smcginnis> EmilienM: I put my name on a few but wanted to go back later to do more once others had a chance to grab some.
15:12:42 <mnaser> dhellmann: i see. it does seem a lot of work to split those things out
15:12:50 <mnaser> and affects a lot of downstream things
15:13:17 <zaneb> cdent: as an outsider to Nova, it's bewildering/frustrating that everyone inside Nova understands perfectly the problem and exactly how to fix it, and they still won't do it. for many years now.
15:13:32 <mnaser> i think it's more of a lack of resources
15:13:39 <dhellmann> the placement work is a step in that direction, isn't it, zaneb ?
15:13:52 <smcginnis> ++
15:14:05 <cdent> zaneb: I think it depends on how your define "inside", for one thing
15:14:06 <EmilienM> mnaser: in tripleo, we use "Trust" (crazy right?) to make Gerrit ACLs
15:14:07 <zaneb> dhellmann: yes, that's true
15:14:09 <smcginnis> But even with that, there seems to be some reluctance to actually split out placement.
15:14:11 <dims> "still won't do it" is pretty strong ... current experiments are the runway and the placement possibly moving out
15:14:18 <EmilienM> mnaser: we promote people "core" on certain files/topics in tripleo
15:14:24 <EmilienM> mnaser: and you know what? it works :(
15:14:28 <cdent> But also the splitting is _hard_. nova is very coupled with itself, with a wiggle over here cascading to impact way over there
15:14:54 <dhellmann> looking in from the outside it seems nothing has changed, but when I talk to the nova team I learn about things that are happening. I think it's just going more slowly than anyone likes. Which makes me wonder if we can somehow give them support to say "no" to feature requests that are not related to decomposing the project, to speed things along.
15:14:55 <zaneb> there was a previous failed attempt to move placement out too though, wasn't there?
15:14:59 <dims> smcginnis : i was in the placement break out session, there's work to be done to split it out. need hands now i think
15:15:01 <EmilienM> mnaser: (not trying to compare, but giving an example where we also wanted Gerrit ACLs on files)
15:15:06 <cdent> as far as I can tell the reason that placement extraction is happening at all is because a) i'm stubborn, b) i've been exceptionally lucky to have three employers in a row who let me do much of what I want. Other people are not so lucky
15:15:08 <zaneb> (not suggesting this one is going to fail)
15:15:23 <mnaser> EmilienM: i think that can be something that would be proposed "you are a nova-vmware-core, while you can +2 everything, you can only +2 vmware changes"
15:15:26 <fungi> cdent: i saw that discussion beginning to build, and can't say i'm surprised at the discovery that you can't force people to review what you identify as important (i think all current/former ptls probably know this already)
15:15:27 <dhellmann> zaneb : my understanding is that previous approach attempted to forklift it without a transition plan, but I may be mixing up bits of history there
15:15:28 <dims> ++ cdent
15:15:53 <cdent> there was gantt which was sort of different
15:16:02 <dims> fungi : totally (jay's email from this morning)
15:16:32 <cdent> and extracting placement won't really change anything other than bring things back to their 2016 state and these coversations go back further than that (recall dan berrange's giant thread in 2014)
15:16:46 * TheJulia feels like we're also semi-rehashing some of the discussions from yvr and wonders what the next steps would be to promote trust
15:16:59 <mnaser> TheJulia: agreed
15:17:11 <ttx> I still hope they split functionality into separate review domains before they run out of "supercores" (the limited number of Nova cores that have the full picture in their brains and can really approve changes)
15:17:21 <zaneb> cdent: yeah, that thread is what I keep thinking back to
15:17:25 <dhellmann> we have several different proposed solutions, each of them taking a different approach. Which one does the nova team support?
15:17:33 <ttx> because it's hard to raise new ones
15:17:37 <cdent> yeah, TheJulia, my intent here was not to really address the problem here, just draw people to the thread
15:18:26 <TheJulia> Yeah, I hate to say it, but the only way movement is going to be made on the various issues is to toss concepts against a wall and see what sticks (for lack fo a better term to use)
15:18:33 <cdent> ttx: yes, supercores are expiring
15:18:40 <TheJulia> some of those concepts might be code, and from there sell everyone on it
15:18:57 <smcginnis> cdent: That sounds grim. :)
15:19:09 * cdent is a grim guy
15:19:09 <ttx> cdent: well their jobs aren't getting any easier as time passes
15:19:18 <cdent> tru
15:20:08 <dhellmann> what if we applied the maintenance-mode tag to nova?
15:20:18 <mnaser> why don't we add another 'voting' category in gerrit
15:20:24 <TheJulia> cdent: no, your realistic. There is a huge difference :)
15:20:33 <mnaser> just like we have code review and rollcall-vote
15:20:36 <cdent> dhellmann: that is an interesting idea. It basically maps to what jay is saying.
15:20:40 <ttx> mnaser: I wanted to discuss next steps on the diversity thing with you, maybe off meeting though
15:20:57 <ttx> dhellmann: I don't think that would solve anything
15:21:12 <mnaser> we can have a +2 in "domain-core" and then +2 in "super-core" and only a super-core can +W
15:21:31 <mnaser> so a supercore doesnt have to do a very deep dive if he knows that it doesn't touch anything beyond that specific domain
15:21:37 <ttx> mnaser: I fear that would insitutionalize the wrong approach
15:21:47 <dims> do we expect some of the starlingx's nova folks to show up anytime soon?
15:21:56 <dims> dtroyer ^^
15:22:01 <ttx> You can't extend trust beyond a certain number anyway
15:22:11 <dhellmann> yeah, I don't want us to build more complicated voting solutions to deal with the fact that we have a highly coupled system. we should look for ways to help the team have the space to fix the code.
15:22:12 <zaneb> mnaser: technical solution to a social problem
15:22:20 <mnaser> but i don't think trust scales
15:22:26 <cdent> btw, talking about this amongst ourselves is probalby the wrong approach. we should at least invite melwitt, mriedem, dansmith, jaypipes into this instead of speculating
15:22:32 <ttx> The only solution is to split into separate trust domains
15:22:48 <ttx> i.e review domains
15:22:58 <ttx> But that involves letting go of some control
15:23:07 <ttx> (some would say some overall quality)
15:23:10 * cdent in neither core nor supercore
15:23:12 <fungi> which, as they've noted, is hard with nova because of a lot of tight coupling between various parts of the service
15:23:25 <zaneb> ttx: exactly. nothing scales, and that's why we need to avoid projects that are Too Big To Fail
15:23:46 <dhellmann> based on what jay said in that thread, and I've heard from other folks, the rate of change in the code makes it difficult for them to change processes or tools or have time to grow the team. can we address that rate of change in the code somehow?
15:23:50 <mnaser> (ttx: yes, let's talk about organizational diversity after if you have sometime)
15:24:10 <dansmith> dhellmann: saying nova is in maintenance mode isn't going to slow things down
15:24:18 <dansmith> if that's what you're suggesting
15:24:30 <ttx> yeah I think it would only have negative effects
15:24:32 <dhellmann> dansmith : yeah. I'm throwing out ideas to find ways to give you options to say "no"
15:24:43 <dansmith> dhellmann: I say no a lot :)
15:24:44 <EmilienM> mnaser: trust scales, when you setup the right culture
15:24:55 <dhellmann> dansmith : heh, ok
15:25:03 <EmilienM> mnaser: https://review.openstack.org/#/admin/groups/190,members
15:25:15 <ttx> EmilienM: it's really difficult.
15:25:21 <smcginnis> cdent: You're a super non-core. ;)
15:25:22 <dansmith> dhellmann: but I agree, I'd like more people to say no more often.. I don't think artificially labeling nova as in maintenance mode really does anything though
15:25:26 <ttx> One would argue that this group is tied by external trust
15:25:28 <EmilienM> I don't think I said it was easy :)
15:25:30 <EmilienM> I said it's possible
15:25:34 <dims> EmilienM : check number of redhat.com on that :)
15:25:34 <ttx> Like read the email domain
15:25:43 <EmilienM> it's not about redhat come on
15:25:55 <EmilienM> I don't care in Gerrit if this reviewer is redhat or not
15:25:56 <mnaser> ;p
15:26:07 <ttx> I'm just saying the trust is sustained by external constructs
15:26:07 <EmilienM> what I care is who is doing it, and the knowledge/capacity to review
15:26:29 <mnaser> ttx: while i agree with you, i think we can bring those external constructs to the open regardless of the org
15:26:34 <ttx> It's really difficult to trust someone to review the way you would have reviewed. Works if you have enough shared culture
15:26:46 <ttx> That shared culture can come from company culture too
15:26:47 <dims> EmilienM : there's extra processes / hierarchy that helps too
15:26:48 <mnaser> "if i merge things in the wrong place, it will make my peers (other nova devs/cores) unhappy and i might lose my core"
15:26:52 <dhellmann> based on the *many* conversations I've had with the nova team about this, I do not think the issue is just "trust". Nova *is* complex. Learning that takes time. Investing in reviewers to build them to core takes time.
15:27:15 <ttx> Dunbar tribe bumber
15:27:16 <mnaser> dansmith: you seem to be the only one whos mostly around.. are most changes going in specific domain?
15:27:19 <ttx> number*
15:27:20 <EmilienM> the fact tripleo is redhat only doesn't mean it's easy for us to trust each others; but it's in our (project) culture to setup trust and we move forward
15:27:28 <dansmith> mnaser: we've talked about the domain core thing so many times
15:27:34 <dhellmann> so instead of saying the nova team should already have fixed this by just trusting more people isn't really digging into the underlying problems
15:27:36 <ttx> EmilienM: and yet most other team stalls around 15 in core teams
15:27:36 <EmilienM> dims: like all projects
15:27:42 <dims> right
15:27:43 <dansmith> mnaser: and I really really think that it's not that easy
15:27:59 <mnaser> dansmith: i understand but i'm just trying to get a grasp if things mostly seem to be going in one direction
15:28:00 <EmilienM> believe me or not it's not just because it's a redhat only project.
15:28:08 <mnaser> aka, is it a lot of the libvirt stuff that moves most of the time and others are mostly idle
15:28:11 <dims> EmilienM : i trust you on that
15:28:16 <EmilienM> I was PTL for Puppet OpenStack and we promoted a bunch of non redhat core
15:28:30 <EmilienM> we were diverse during some time (well not anymore)
15:28:34 <mnaser> yeah, it involves trust. and sometimes people break things.  and you tell them, and they learn :)
15:28:52 <mnaser> and sometimes even long term cores will break things when missing something, and i think that's *ok*
15:28:53 <ttx> EmilienM: agree that there are things you can do to set up project culture that allow to scale to larger groups
15:28:57 <dims> EmilienM : question is what did we learn from there that can be applied to nova
15:29:01 <EmilienM> anyway I was not trying to give a lesson on something, just giving some examples based on my personal experience.
15:29:28 <ttx> But it still generally stalls at some point, unless sustained by some other cultural alignment
15:29:29 <dhellmann> dansmith : are there things the nova team wishes they could do (or maybe just you wish) that the TC could help you with to address the situation?
15:29:45 <mriedem> i'm around, but not engaging...
15:29:46 <dims> ++ ttx
15:30:09 <dhellmann> mriedem : I'll ask the same question of you then :-)
15:30:15 <TheJulia> Tossing in a disruptive thought into this track. People don't want to give up control in many cases because of responsibility or perception of responsibility. Fear of making a mistake, if that is reduced, would people not be able to be more trusting because they wouldn't have to have their guard up constantly?
15:30:19 <ttx> Like if you set up a "ask for forgiveness, not for permission" culture you can grow review teams from 15 to ~25
15:30:23 * dims drags mriedem to the table
15:30:54 <mnaser> ttx: maybe this links to the idea of "master should be CD-able"
15:31:01 <dhellmann> could we, maybe, stop rehashing the same arguments about this we've had for 8 years and talk to the folks directly involved about the help they need?
15:31:13 <ttx> mnaser: good point yes
15:31:17 <mnaser> ah, yes, good point dhellmann.
15:31:31 <dansmith> dhellmann: I don't really think the TC is going to fix anything like this from the outside, I guess is my summary
15:31:34 <dhellmann> since mriedem and dansmith are actually here
15:31:41 <dims> ++ dhellmann
15:31:49 <EmilienM> ttx: yes, that's the kind of culture I'm talking about
15:31:49 <ttx> dansmith: agree with you on that
15:31:51 * mriedem remembers barcelona
15:31:55 <dhellmann> dansmith : let's not assume that. If you could change something, what would that be?
15:31:57 <mriedem> and atlanta
15:32:01 <mriedem> and every retrospective
15:32:12 <dansmith> dhellmann: let's not assume what?
15:32:17 <mriedem> that the tc can't help
15:32:23 <mriedem> assuming the tc can help, what do we need?
15:32:39 <dhellmann> dansmith : that no one outside the team can help
15:32:40 <dhellmann> or the tc in particular
15:32:52 <mriedem> mirantis and osic layoffs not happening? :)
15:33:25 <mriedem> once placement splits out it should help some, but the core team to start is still likely going to be similar to what it is now
15:33:34 <mriedem> i don't know how that works when cinder split out, i wasn't around that early
15:33:44 <mriedem> *worked
15:33:53 <ttx> mriedem: should be easier to become core on placement than to become core on Nova I guess ?
15:34:08 <mriedem> yeah
15:34:10 <dansmith> cinder was an obvious piece that could be cleaved off,
15:34:10 <mriedem> it's a much smaller footprint
15:34:15 <EmilienM> is the neutron project a good example as well? afik they've split a bunch of things as well
15:34:16 <dansmith> and not  having to review tons of nova-network features reduced our workload for sure
15:34:21 <dhellmann> is there another obvious piece, after placement?
15:34:22 <ttx> so maybe the teams would start to diverge soon enough (hope)
15:34:51 <dansmith> placement hasn't yet reduced the amount of complexity or code in the rest of nova, but will eventually
15:35:10 <dims> looks like dtroyer is not around to say something about nova folks from starlingx ...
15:35:12 <dansmith> but not by as much asthe volume or network splits
15:35:51 <ttx> I still have that hope that hypervisor drivers could be split off. If teh API there could be declared stable
15:35:59 <ttx> Sounds domain-specific enough
15:36:15 * mnaser doesn't know if there is enough work on other drivers that is affecting cores that much
15:36:15 <dansmith> we certainly could do that,
15:36:23 <ttx> But I know we've been chasing that stabel API for a long time and nobody really has time to work on it
15:36:34 <dansmith> but hypervisor drivers are way more integrated and complicated than network drivers and volume drivers
15:36:49 <dansmith> ttx: we're actively changing right now.. it's always changing.. substantially
15:37:07 <dhellmann> what part of the code causes the most review burden?
15:37:14 <dansmith> it's not that nobody is pursuing a stable virt api, it's that it's something that is pretty tightly coupled and complex
15:37:24 <mriedem> runways has helped focus efforts on getting non-libvirt driver blueprint changes in
15:37:30 <mriedem> xen and powervm got several bps merged in rocky b/c of runways
15:37:38 <dansmith> yup
15:37:44 <dansmith> xen for the first time in a while
15:37:44 <ttx> cool.
15:37:45 <mriedem> so runways is a new thing
15:37:54 <mriedem> i thiink pike saw a lot of ironic driver features get in
15:38:19 <TheJulia> I think we had also been working on those for >1 year
15:38:31 <mriedem> TheJulia: also lots of those were blocked on getting the ironic stuff done first
15:38:35 * EmilienM likes the name "runway" fwiw, it reminds me "checkpoint" that we use in my internal team, which is a similar process that we're doing
15:38:39 <mriedem> so it's not like nova was just sitting there
15:38:45 <TheJulia> indeed
15:38:45 <dansmith> TheJulia: and to the point, those changes were tied to the virt driver interface, compute manager, scheduler, and placement
15:38:55 <ttx> EmilienM: of course you like the name "runway"
15:38:57 <dansmith> so you know, they were complicated to get right and land without breaking people too much
15:39:06 <dansmith> ttx: ;)
15:39:08 <EmilienM> I like runways in general, always better than a field :P
15:39:08 <TheJulia> dansmith: completely agree
15:39:20 * EmilienM puts his pilot hat
15:39:20 <dhellmann> I'm much less interested in having things go faster than I am in making it so the team doesn't burn out.
15:39:21 <mriedem> also, nova tends to ask for actual CI coverage of features when possible if they are big enough, and that delays things
15:39:26 <dims> i keep reading runway as runaway!
15:39:31 <TheJulia> dansmith: I was only providing context for the discussion in that some of these things take a long long time
15:39:38 <dansmith> dhellmann: okay I want to revise my answer
15:39:44 <EmilienM> dims: just don't runaway on a runway plz
15:40:01 <ttx> Yeah, my main concern at this point is losing the last supercores faster than we can decompose to simplify their work
15:40:07 <dhellmann> dansmith : I'm all ears
15:40:10 <mriedem> queens, as noted in our retrospective at the ptg, was much less hectic in part b/c we had fewer approved blueprints to focus on
15:40:17 <dansmith> dhellmann: one thing that really makes me feel like ragequitting is feeling like everyone is complaining all the time about us merging a ton of code, but not enough code. I would love to feel like the TC was supporting me more and trying to fix me less
15:40:27 <dansmith> dhellmann: brutal honesty there, please take it in good faith
15:40:33 <mriedem> ^ same
15:41:07 <dhellmann> dansmith : yes, I'm trying to make this conversation about helping you rather than fixing you, and I hope that's coming across.
15:41:11 <mriedem> we took on less in queens and got more done, which was good,
15:41:23 <mriedem> and b/c of that, it seems we dipped way into the debt tank in rocky and are way overcommitted agin
15:41:24 <mriedem> *again
15:41:24 <dims> dansmith : mriedem : +100 on what dhellmann said above
15:42:08 <mriedem> lots of big things take project management help too,
15:42:11 <dansmith> dhellmann: dims: well, it doesn't feel like that a lot of the time, which is just being honest, but I'm glad to hear that
15:42:13 <mriedem> sdague used to do that for things he cared about
15:42:14 <melwitt> dhellmann: thank you for making that effort, because I have to say I feel similar. we merge a _lot_ of code, yet are constantly the subject of "problems" and "not enough cores"
15:42:17 <dhellmann> dansmith : IIRC from looking at the stats, nova is doing at least an order of magnitude more review work than other teams. The rate of code churn is incredible. So I'd like to help you lower your stress levels, because you're already doing more than is reasonable.
15:42:17 <mriedem> ildikov did it for multiattach
15:42:41 <mnaser> i think we're looking to find ways to reduce the load on you because you're valuable members with huge knowledge
15:42:53 <dansmith> dhellmann: thank you for saying that
15:42:57 * dansmith prints and frames
15:42:59 <ttx> mnaser: ++
15:44:31 <dhellmann> melwitt : if I say there's a problem, I mean that the team is exhibiting signs of stress which leads to bad interactions. I think that's a problem for the long term health of the people, and the project. I want people to stay happy working on OpenStack in general and nova in particular.
15:44:34 <mnaser> i totally appreciate the work done by the nova core team, it's been invaluable as an operator :>
15:44:35 <cdent> I'd just like to make it clear that this topic came up because I drew people's attention to jay's message today about feeling overstretched and people's sympathy about that is what is driving this talk.
15:44:37 <zaneb> dansmith: as a likely culprit in that perception, I'd like to acknowledge for the record that Nova is merging an enormous amount of code and that the current cores are working *incredibly* hard. I guess we all just wish you didn't _have_ to.
15:44:58 <cdent> zaneb++
15:45:05 <ttx> Yeah at this point my concern is much more "how do we make the work of the Nova team sustainable", not "how do we make Nova approve $foo faster"
15:45:15 <dansmith> so, to be honest,
15:45:27 <dansmith> I can handle the workload to a large degree
15:45:43 <dansmith> it's the workload plus the feeling that nobody will ever be happy that really wears me out
15:45:57 <cdent> is it fair to say, dansmith, that your ability to handle the workload is fairly unique?
15:46:00 <dtroyer> IMO the feelings some have about Nova (too much, not enough, fix it, etc) have become entrenched folklore
15:46:06 <cdent> as in dan is special (in the good way)?
15:46:21 <dtroyer> and get passed on to new generations of people coming to OpenStack
15:46:22 <dansmith> cdent: I can only speak for myself, which is why I said "I"
15:46:33 <dansmith> dtroyer: very much agree
15:46:44 <dansmith> dtroyer: and that very much contributes to my cynicism about "help" being offered
15:47:06 <dhellmann> dansmith : does that mean maybe we could help with setting expectations somehow?
15:47:26 <melwitt> I feel the same, it's all the complaining and criticism from outside that gets me down. nova is a lot of work but I like working on it and I like helping operators, users, and devs
15:47:35 <dtroyer> this feels like one place we need to make a decision to intentionally change the culture
15:47:49 <mriedem> honestly i'm going to put in the same amount of time and effort regardless
15:48:30 <ttx> maybe we should categorize that criticism and address it. I hope that none of this discussion is seen as criticism
15:48:45 <dansmith> dhellmann: I don't know the answer to that question. I have a cynical feeling about it, which may not match the reality or possibility :)
15:49:01 <dtroyer> dims:  ((your nova people from starlingx question: that is unlikely to be a sudden change))
15:49:16 <dims> ack dtroyer. thanks
15:49:21 <ttx> dims: also they won't become supercores tomorrow. or cores
15:49:21 <mriedem> i would question how much review happens on the starlingx stuff either
15:49:35 <mriedem> internal forks and stuff are generally done by teams that are paid to just get stuff done, not maintain it
15:49:44 <mriedem> and therefore don't care as much about the long-term costs or quality
15:49:49 <mriedem> hence why maintaining forks is SUPER HARD
15:50:24 <ttx> dansmith: mriedem: melwitt: also sometimes Nova is taken as the example, while the criticism is actually on OpenStack in general
15:50:25 <dtroyer> mriedem: that is why this is challenging to everyone involved :)
15:50:27 <mriedem> and throwing people at this problem doesn't really help
15:50:35 <dhellmann> dansmith : ok. I'm reluctant to say that because no one has figured out how to help with that in the past we can't come up with any new ideas today.
15:50:57 <ttx> Nova is singled out very often. Like in this discussion
15:50:57 <dansmith> dhellmann: ack, which is why I hedged and exposed my cynicism :)
15:51:29 <mriedem> i need a "my sensitive ego has been bruised" alert bot
15:51:42 <dims> dansmith : mriedem : melwitt : totally love and respect your work and try to hang out with you all every chance i get. So please don't feel your work is not appreciated. let's try an emoji - 💖
15:52:18 * mriedem sees a lego brick
15:52:19 <mnaser> yeah honestly my experience with the nova team is so awesome as an operator
15:52:21 <ttx> dhellmann: I think handling Nova through the TC liaison mechanism rather than make it an all-TC exercise will help, too.
15:52:24 <EmilienM> ++ lot of respect for nova and other projects
15:52:32 <fungi> yeah, given the feedback i heard from the nova cores who skimmed some of the divergence in the starlingx nova fork, it doesn't sound like the choices they made are especially compatible with the level of review the current nova core reviewers would expect (no offense meant to the starlingx crew who were maintaining those)
15:52:45 <ildikov> mriedem: it's a heart
15:52:51 <mnaser> at the summit i was stopped twice by two different cores to ask about how impactful a change nova was doing was going to be for us.  that's awesome
15:53:02 <mnaser> they totally didn't have to do that but the did
15:53:05 <dhellmann> One of the things jay mentions in his email is an "endless stream of feature requests". Can someone give any detail about the source of those requests?
15:53:12 <dansmith> mnaser: brag all you want, everyone knows the nova team <3's mnaser :)
15:53:34 <mnaser> :)
15:53:36 <mriedem> vexxhost (mnaser) and cern get special treatment b/c of their level of openness and involvement in the project
15:53:46 <dims> dansmith : i mean ... who doesn't?
15:53:52 <dansmith> dims: indeed
15:54:03 <mnaser> i really want to do a talk on "how to work with upstream" and add more orgs to that list
15:54:05 <dansmith> mriedem: and they actually contribute positively
15:54:10 <mriedem> cburgess and med were also in that boat
15:54:17 <mriedem> RIP!
15:54:20 <mriedem> SAD!
15:54:23 <mnaser> i think if people understand the value of working with upstream, internal culture will change massively
15:55:05 <mriedem> mnaser: true story, one of our guys from china that was in vancouver for the first time commented after the nova cells v2 / cern session, 'wow you guys are very involved in helping cern and they are with making sure their needs are met'
15:55:08 <mriedem> it was like a revelation
15:55:30 <dansmith> well, they do also have a very big electron gun...
15:55:44 <mnaser> lols
15:55:53 <mriedem> i'm just personally interested in helping the people that show up
15:55:57 <fungi> yeah, don't wanna be on the wrong end of that sucker
15:56:00 <mriedem> mgagne and the godaddy folks also come to mind
15:56:03 <mriedem> jmlowe too
15:56:12 <mnaser> maybe as the tc we should also look at our projects and acknowledge the good progress they've done so far
15:56:12 <dtroyer> fungi: the choices made in starlingx were for expediency and had the advantage of a highly opinionated configuration rather than having to work with N possibilities.  overcoming that is going to be one of our biggest challenges in clearing the debt backlog
15:56:18 <mnaser> rather than always trying to say "hey so let's fix this"
15:56:31 <mnaser> it might give the messaging of "we're fixing you because you don't have your stuff together" in a way
15:56:44 <dhellmann> mriedem : helping folks who show up seems like exactly the right approach to day. That's how open source is supposed to work, imo.
15:56:50 <mriedem> mnaser: it totally does,
15:56:51 <dhellmann> ugh, *to take
15:56:55 <mriedem> at least it always has to me
15:57:12 <dansmith> the culture in openstack has changed recently,
15:57:19 <dansmith> we used to celebrate the development side
15:57:27 <mnaser> i think we had 3 nova cores pretty much mention that they felt the messaging is "we're fixing you" so maybe we should look into changing that
15:57:28 <dansmith> focus on the wins
15:57:35 <fungi> dtroyer: yeah, i get that. just don't want people getting the impression that they were doing the same sort of work the nova team needs now, and the learning curve for them is likely to be as steep as it is for most people coming to the nova team for the first time
15:57:45 <dansmith> nowadays, it feels like the developers are the ones in the back room not doing enough,
15:58:00 <dtroyer> fungi: ++   (more for the record than anything :)
15:58:10 <dhellmann> dansmith : I've noticed that a bit, too
15:58:12 <dansmith> which means we focus on the projects getting the most complaints
15:58:16 <dims> ++ fungi
15:59:12 <mnaser> this was good and i'd like to thank dansmith, melwitt and mriedem for speaking up
15:59:31 <mriedem> having said that, i was happy with the press in queens for multiattach and vgpus and the like
15:59:34 <TheJulia> dansmith: I agree that could be, but I also think that as time has evolved, we've lost valuable people, people have become more spread thin... so perceptions might not match reality in terms of their ability to contribute
15:59:34 <mriedem> that was nice to see
15:59:51 <mriedem> cern was extremely gracious in vancouver
15:59:56 <fungi> dansmith: dhellmann: in recent years i've seen lots of feedback that celebrating developer activity caused other parts of the community to feel like second-class citizens and start lobbying to stop doing things they saw as idolizing developers. i don't agree with that assessment but it seems to have pervaded a lot of the extended community
16:00:07 <melwitt> I guess I want to say quickly that I appreciate the concern from all of you. we are a team of human beings and sometimes we get stressed and exhausted and I think it's okay to be open and honest about that, IMHO. at the same time, I don't want it to cause a rush of concern and fixing if someone opens up about something. we open up to try to make things better and truly evaluate the processes we're trying out
16:00:15 <mriedem> fungi: the core party aside
16:00:34 <fungi> glad the core reviewer parties are a thing of the past, yes
16:00:44 <fungi> i expect those fueled a lot of that sentiment
16:00:49 <dhellmann> I was glad melwitt had a chance to talk about the work in nova on the keynote stage. I wish there had been more time for that sort of thing.
16:01:11 <TheJulia> That was really nice
16:01:12 <fungi> agreed
16:01:16 <fungi> it was awesome
16:01:38 <mriedem> med mentioned during the summit feedback session something like that for superusers during keynotes but the community contributor awards
16:01:46 <mriedem> that would be a solid 'hey we actually care about contributors' thing from the foundatoin
16:01:56 <dhellmann> melwitt: agreed. at the same time, we'd like to try to help before the pressure gets so bad, so maybe we should talk about this more often in smaller doses
16:02:16 <dhellmann> mriedem : yeah, that's high on my list of things
16:02:35 <fungi> i also can't help but feel like there must be something the tc could do to help have the right conversations to stem the overwhelming tide of feature requests jaypipes mentioned and i've seen others (not even just in nova) bring up from time to time
16:02:39 <dhellmann> I started a conversation with mark c. about that after dublin, but didn't follow through in time to make it happen for yvr
16:02:41 <smcginnis> Unpopular opinions, but I actually liked the core parties. Maybe I didn't get exposure to the downsides of that.
16:02:58 <dhellmann> fungi : right, that's what I was hoping we'd get to
16:03:23 <dansmith> dhellmann: I think the assumption that we need to be parented before we get into a fight is the wrong approach, FWIW
16:03:29 <zaneb> fungi: shameless plug: I think this is something that a technical vision could help with
16:03:42 <fungi> indeed!
16:03:43 <dhellmann> dansmith : oh, I communicated poorly, then, because that's not what I mean
16:04:09 <dansmith> dhellmann: well, that's what it feels like whenever someone venting turns into the TC showing up to separate the children on the playground :)
16:04:31 <dansmith> (to me)
16:04:35 <fungi> smcginnis: the opportunity for a quiet place to get away and have laid-back conversations was nice. the exclusivity if it was sort of divisive and reinforced a caste system dynamic
16:04:40 <dhellmann> dansmith : but as an example, if there's a flood of feature requests that won't stop, maybe that's something we could deal with at the TC/board level so you don't have to have the fight repeatedly with every vendor who comes along
16:04:48 <mriedem> we've done things in the past to try and stem the tide of feature requests,
16:05:00 <mriedem> but the thing is, then you get nailed for not landing enough features that people want
16:05:07 <mriedem> it's a damned if you do, damned if you don't situation
16:05:12 <TheJulia> mriedem: define nailed
16:05:27 <dhellmann> dansmith : you may be referring to one specific instance where I did feel a discussion had deteriorated to a point where a third-party could help mediate
16:05:29 <mriedem> constant critisizm?
16:05:42 <mriedem> constant "how do we fix nova?"
16:05:45 <smcginnis> fungi: I liked the quiet place to have conversations aspect of it.
16:05:46 <TheJulia> mriedem: could it just still then be a perceptions issue?
16:06:01 <ttx> FTR we like to fix Swift too.
16:06:06 <smcginnis> fungi: And felt like it was a good way to foster communication between the people in different teams that could work together to actually get things done.
16:06:08 <zaneb> so I have a theory that is probably unpopular :)
16:06:24 <dansmith> dhellmann: I dunno that the board/TC really have the ability to stem that tide I guess
16:06:38 <mriedem> TheJulia: idk, i think some people just don't care about anything but their special snowflake
16:06:38 <TheJulia> I guess maybe I genuinely think people are well intentioned, and maybe criticism comes from a lack of context
16:06:40 <dansmith> dhellmann: I'm not referring to any one incident, fwiw
16:06:54 <smcginnis> fungi: And if having a party incentivized some people to get more involved and become cores to be able to go to it, that's good in my book.
16:07:14 <dhellmann> dansmith : if we said openstack was going to focus on technical debt during 2019 and not accept new features, and you received new feature requests, you could just send people to me instead of having to talk to them yourself. Would that help?
16:07:19 <TheJulia> mriedem: true, or they don't have the capacity to care about anything outside of the overall venn diagram defining their role
16:08:02 <cdent> zaneb: ?
16:08:05 <fungi> dansmith: there is the idea that if the vendors pushing for new features are represented on the board or in the set of foundation member companies, then there's the potential for product management and executive management direction conversations to stop trying to cram new features into a project which is already under stress to keep up
16:08:11 <dansmith> dhellmann: that doesn't seem realistic to me, but if it did happen I guess it would be nice :)
16:08:14 <zaneb> the theory goes that due in part to the hype cycle and in part to the lack of a detailed technical vision, OpenStack became a thing that tried to be all things to all people. It tried to be both a cloud offering and a self-service full VMWare/oVirt replacement complete with live migration, shelve/unshelve, and generally a lot of stuff that increased the complexity of the code and led to a steady torrent of n
16:08:15 <zaneb> ew feature requests
16:09:10 <ttx> dansmith: we kinda said Ocata could be used to to a stable release... not sure that worked really though
16:09:12 <dansmith> zaneb: I agree with that theory, but what point did it support?
16:09:17 <dhellmann> dansmith : I would support that and fight for it if it would help you have time/space to do more work to split parts out of nova or take whatever other steps you need to help resolve the technical issues that make it hard for you to bring in new folks to the core team
16:09:38 <zaneb> if we could have picked one then things would have been easier to manage
16:09:57 <mriedem> zaneb: but that ship has sailed
16:10:07 <ttx> yeah we kinda have users now
16:10:10 <mriedem> and the hordes are now doing it to k8s is my understanding
16:10:12 <jroll> dhellmann: fwiw, I think that people external to the TC generally don't believe that the TC/board have influence on what contributing companies do. I follow the TC's work fairly closely and it isn't even clear to me that there is influence there
16:10:21 <jroll> dhellmann: if there is influence, it isn't very visible
16:10:25 <zaneb> mriedem: yes, in many ways it's too late to fix that now because we have existing users to support
16:10:26 <dansmith> jroll: agree
16:10:37 <TheJulia> jroll: ++
16:10:41 <dansmith> dhellmann: I wouldn't say that a year of tech debt is going to accomplish or even work towards that goal
16:10:56 <dhellmann> jroll : we have tried to be very hands-off in the past, but the bylaws of the project give us basically unlimited power if we choose to use it
16:11:02 <zaneb> but we can still do better for the future by saying clearly what we do and do not want OpenStack to be
16:11:05 <mriedem> i'm pretty sure my employer still wants/needs to see features going in every release regardless of what upstream does, so that doesn't help me really
16:11:14 <dansmith> mriedem: mine as well
16:11:35 <dansmith> dhellmann: he said influence over the companies, not the developers
16:11:54 <dansmith> dhellmann: I don't think the bylaws give you unlimited power over, say, huawei's actions
16:12:08 <dhellmann> dansmith : if we have to make it so that the TC has to approve patches, we could do that. I wouldn't want to, but we could.
16:12:22 <dhellmann> my point is that we have more power than we use
16:12:24 <dansmith> okay I guess I'm missing something
16:12:32 <mriedem> and then nova will just lobby to be a top level project in the foundatoin and outside the tc pervue
16:12:43 <mriedem> *purview?
16:12:47 <mriedem> when is swift going to do that btw?
16:13:01 <dhellmann> we say no one has any influence over these companies, but the community sets the contribution rulues
16:13:32 <jroll> dhellmann: my point being you can offer the TC to help with the negativity or turning down feature proposals, but most people won't believe that the TC can successfully help there
16:13:46 <dims> jroll : we can try ... right?
16:14:02 <jroll> sure
16:14:19 <dhellmann> jroll : walk softly and carry a big stick
16:14:32 <jroll> not saying it isn't possible
16:14:34 * mriedem looks at the clock and realizes he hasn't reviewed anything he wanted to review yet today
16:14:44 <ttx> I think we need to better define the problem (if there is a problem) before wielding a big hammer in any direction
16:14:47 <dhellmann> mriedem , dansmith , melwitt : thanks for your time today
16:14:57 <dims> +1000
16:15:07 <TheJulia> ttx: definitely
16:15:09 <fungi> jroll: in fairness, i also have limited expectations in our ability to get board members to take specific actions, or that the board members even necessarily have the ear of the right people in their own organizations to make such things happen, much less in other member companies. still, not trying guarantees it won't happenm
16:15:32 <notmyname> well this is an interesting buffer scrollback to start the day with
16:15:55 <dhellmann> I'll just point to thingee's work on forcing third-party ci through for the cinder team as an example of how, when we set the rules to support the teams doing the work, we can make big changes happen.
16:15:59 <jroll> I'm just offering a POV from outside of the TC, especially as to why people generally are meh on the TC trying to take action here
16:16:01 <ttx> notmyname: hi!
16:17:06 <ttx> fungi: should we close the hour?
16:17:13 <fungi> seems we've wound down
16:17:19 <fungi> unless notmyname has something? ;)
16:17:29 <ttx> If we continue I need another drink
16:17:31 <jroll> it's especially difficult to answer the question "what can the TC do to help here?" if one doesn't think the TC has the influence or power to do much
16:17:41 * jroll is done, I think
16:18:16 <jroll> do much with toning down the slew of patches or negativity from contributors*
16:18:24 <fungi> jroll: yeah, we've been mostly loathe to exercise any nuclear options
16:19:03 <fungi> in part, because those options put the project teams in a position of having to decide whether they want to continue to be part of openstack or go it on their own
16:19:25 <zaneb> the TC has a lot of influence, but mostly only power that is too strong to use
16:19:28 <notmyname> fungi: nah, still reading scrollback. don't wait for me for anything
16:19:33 <zaneb> kind of like the Queen
16:19:50 <dhellmann> we could, however, back up teams that want to exercise some of that power themselves
16:20:47 <fungi> though i think that comes down to those teams feeling comfortable to exercise that power, and knowing we've got their back if their decisions get escalated to us
16:21:44 <dhellmann> which is exactly why I keep asking the team what would help
16:22:15 <jroll> well, if we stick to the specific nova thing
16:22:17 <fungi> what we probably can't be much help with is taking away the stress the employers of team members may place on them
16:22:21 <jroll> we concluded the major part of the problem is that the constant negativity is what really drags down the folks working on it
16:22:29 <fungi> at least as far as said nuclear options are concerned
16:22:30 <smcginnis> fungi: Should you #endmeeting?
16:22:31 <mriedem> yeah, i was going to say,
16:22:36 <jroll> how can the TC reduce people saying "nova doesn't want my features!!"
16:22:51 <jroll> "nova sux they won't merge this completely out of scope thing"
16:22:54 <jroll> etc
16:23:13 <mriedem> stop talking about 'fixing' things, more emotional support, and understand we're trying - we realize there are issues, and we're generally our own worst critics
16:23:23 <fungi> jroll: it sounds like the bigger problem is "my employer will reassign me and i won't have time to be a nova core reviewer any longer if i can't at least also make some progress on features they've requested"
16:23:42 <TheJulia> mriedem: That should be a presentation for berlin
16:23:51 <mriedem> TheJulia: did that in boston
16:23:53 <mriedem> it was the ptl talk
16:23:59 <TheJulia> I know :)
16:24:23 <fungi> was a very excellent talk, btw
16:24:25 <TheJulia> it should be in the playlist on repeat of sorts
16:24:27 <mriedem> fungi: really depends on what the core is focusing on i think
16:24:31 <jroll> fungi: I think that is a smaller risk than the cores totally burning out
16:24:37 <mriedem> yeah
16:24:47 <mriedem> huawei is totally happy to have me working on fixing scale and resiliency issues
16:25:01 <mriedem> but we have lower priority feature needs for operators too
16:25:31 <fungi> huawei sounds like they're doing it right ;)
16:26:25 <mriedem> hey,
16:26:29 <mriedem> huawei is best way
16:26:38 * mriedem drops mic
16:26:45 <jroll> please tell me that's the official slogan
16:26:53 <fungi> hypothetically, if nova were to want to take a cycle merging no new features and just working on decomposition, tech debt, usability fixes, bugs... any feel for how many of the current nova core reviewers' management chains would reassign them to somewhere else?
16:28:03 <TheJulia> It wouldn't just be cores that we would need to worry about
16:28:05 <jroll> I've always assumed employers would continue having people do the same amount of feature work, with a note that it won't be merged until the following cycle
16:28:08 <TheJulia> but actual on the ground contributors
16:28:29 <jroll> and then double the feature requests the following cycle
16:28:39 <fungi> TheJulia: if the actual on the ground contributors who get reassigned wer ethe ones pushing the unending flood of feature review requests, then that sounds like what they'd want?
16:29:26 <jroll> fungi: except those same devs probably also contribute bug fixes, right?
16:29:34 <jroll> and suddenly you've lost some percentage of bug fixers
16:29:34 <fungi> or is the general makeup of a random (not necessarily core reviewer) contributor to nova doing a mix of implementing new features and fixing bugs?
16:29:43 <fungi> jroll: yeah, that's what i'm wondering
16:29:49 <jroll> good question
16:30:23 <fungi> tragedy of the commons scenarios tend toward a lot of people pushing agenda patches and a handful working on fixing existing issues
16:30:23 <smcginnis> dhellmann: That might be an interesting metric for your next set of reports. ^
16:30:37 <TheJulia> I think it would be like trying to cross the doldrums with a sailboat.
16:30:43 * jroll needs food, this was a good conversation. I'm bummed I came in so late
16:30:45 <smcginnis> dhellmann: How many contributors are doing feature work, how many are doing bug fixes, and what's the venn diagram look like of the two.
16:31:06 <dhellmann> yeah, it can be hard to tell the difference between a bug fix and a feature request
16:31:24 <dhellmann> I would like to figure that out, though, so if anyone has ideas
16:31:29 <smcginnis> I suppose it depends on the team's policies around using Closes-bug for actual bugs.
16:31:46 <jroll> dhellmann: closes-bug: vs implements: would get you most of the way in nova
16:31:51 <dhellmann> yeah, and teams moving to storyboard will just have "story" or "task"
16:31:58 <smcginnis> Probably not complete, but it would get part way there.
16:32:50 <fungi> it would be a very fuzzy metric, but might be striking enough to still hold some insights
16:33:19 <fungi> okay, so we've basically wrapped up discussion i suppose? i'll close this log out
16:33:22 <cdent> _many_ implements are tech debt fixes
16:33:30 <cdent> not strictly bugs
16:33:56 <dhellmann> I'm more interested in the difference between the things the team wants and the things the team has thrust upon it
16:34:13 <fungi> a lot of teams also don't have strict policies about filing bugs if there's already a fix available... so generalizing across teams could be tough
16:35:28 <fungi> for infra work, bug reports were generally placeholders for we know something is broken but don't know what the fix should be yet. if it was faster to write the patch than the bug, we didn't saddle contributors with having to do both
16:36:39 <TheJulia> Yeah, when it takes 20 minutes to file the bug, and a minute to fix it, and maybe 2-3 minutes to type a verbose commit message summarizing why... it adds up quick.
16:37:49 <TheJulia> (because the bug has to be written with the bus factor accounted for)
16:39:00 <mgagne> Speaking of negative perception... (sorry if topic is past already)
16:39:01 <zaneb> you have to balance that against how many minutes future readers spend trying to figure out why the heck that change was made
16:39:09 <mgagne> On a positive note, when I attempted to contribute to Nova in the past (like years ago), I used to perceive Nova in a very negative way.
16:39:09 <mgagne> I felt I was ignored and got no feedback whatsoever for months. The only feedback was an automated reply saying "sorry, deadline has past, please propose for next version". It didn't feel like I was interacting with humans at all. Only some kind of big administration where forms and numbers are the norm.
16:39:15 <mgagne> During that same period, I was also contributing to Cinder and had a totally different (positive) experience. So it played a lot in my negative perception of the Nova project as I was expected the same (positive) experience.
16:39:15 <mgagne> Nowadays I get a human reply and feedback. (thanks to mriedem, melwitt, dansmith and others I forgot) It drastically changed my perception of the Nova project. I agree it's not perfect but at least it feels more human. So there is that which I feel is a great improvement. ++
16:39:53 <fungi> zaneb: an excellent argument for very detailed commit messages
16:39:56 <TheJulia> zaneb: indeed, or be able to pick it up with sufficent context.
16:40:24 <fungi> i just don't see a ton of benefit, personally, to copying my detailed commit message into a bug and immediately closing it
16:40:39 <cdent> mgagne++
16:40:42 <mriedem> mgagne: that reminds me of something mrhillsman told me once, or maybe smcginnis said it from an ops meetup - that ops people feel like they shouldn't or can't ask questions or raise issues in the -nova channel
16:40:48 <mriedem> which is definitely a perception issue
16:40:48 <mgagne> cdent: you too ++
16:41:09 <TheJulia> mgagne: I was kind of thinking the same thing earlier this week that I actually went on a business trip, got back and had two nova patches merged and I was like "WOW!"
16:41:39 <mgagne> mriedem: you know what I said at an ops meetup? "you need a friend to get stuffed reviewed or merged"
16:42:09 <zaneb> fungi: benefits are that other people reporting/searching for the same bug may find it, and you can track the status in multiple branches. obviously this applies less to infra than some other things
16:42:17 <mriedem> well, honestly knowing a person that is running a certain system does help, like when i think 'caching scheduler' i know who i can talk to
16:42:20 <mgagne> TheJulia: I appreciate your feedback for the times I attempt to contribute to Ironic ++
16:42:20 <melwitt> mgagne: thank you, and we appreciate the effort you make to work with us and get things fixed in nova
16:42:32 <cdent> mgagne: yeah, I agree that there is definitely a FOAF aspect to openstack contribution that isn't always ideal, but isn't always a negative either
16:42:50 <mgagne> mriedem: so I'm the caching scheduler guy now, don't know how to feel about that one :P
16:42:59 <TheJulia> mgagne: I really do try, sadly there is no way for me to reply to every single patch :( (and I actually feel bad about that reality....)
16:43:00 <mriedem> well, it was rax
16:43:04 <mriedem> but now you're the main person
16:43:12 <cdent> mgagne: tarred!
16:43:27 <mriedem> mgagne: you can also be the "loves network_data.json guy"
16:43:31 <mriedem> if you want
16:43:35 <fungi> zaneb: it may also just be that i'm used to looking at git history to find why changes were made (at least if they're minor enough changes to not require a release note or security advisory)
16:44:32 <TheJulia> cdent: OpenStack is not alone in that.... In some other projects I've had to reach out through a friend because I couldn't illicit a response. :\
16:45:01 <cdent> TheJulia: yeah wasn't trying to say openstack was unique in that regard, just that it happens
16:45:02 <mriedem> the random dead cat questions also matter, those get ignored
16:45:07 <cdent> I suspect it part of scale
16:45:17 <mriedem> "i deployed openstack and i can't create a server, please help"
16:45:20 <TheJulia> cdent: Yeah
16:45:28 <mriedem> like, that's where we tell people to hit the forum or #openstack channel
16:45:45 <mriedem> but, an operator saying, "i'm upgrading and hitting x, i think it's due to y but need some help here"
16:45:53 <mriedem> mnaser: ^ is great about that
16:46:05 <TheJulia> mriedem: Then there is also the "Uhh, that is weird, uhhhhhh Could you give us this laundry list of things we need to help you?" and then crickets are beamed in through time using a transporter
16:46:39 <fungi> #chair TheJulia cdent zaneb dhellmann smcginnis dims ttx
16:46:40 <openstack> Current chairs: TheJulia cdent dhellmann dims fungi smcginnis ttx zaneb
16:46:56 <fungi> okay, i'm about to disappear and grab lunch/run errands. our office hour is nearly an office double-hour at this point but someone can #endmeeting when appropriate
16:47:49 <TheJulia> Do we have anything else to discuss? Things to get off our chests? great beers or coffees to share?
16:48:27 <dhellmann> fungi : nice reworking of the base services draft
16:48:46 <dhellmann> I think it's safe to close the logs at this point
16:48:50 <dhellmann> TheJulia : do you want to do the honors?
16:49:13 <TheJulia> Sure! Since it seems like the time ship relativity beamed in crickets.
16:49:34 * cdent chirps
16:49:34 <TheJulia> #endmeeting