20:01:20 <ttx> #startmeeting tc
20:01:21 <openstack> Meeting started Tue Feb  7 20:01:20 2017 UTC and is due to finish in 60 minutes.  The chair is ttx. Information about MeetBot at http://wiki.debian.org/MeetBot.
20:01:23 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
20:01:25 <openstack> The meeting name has been set to 'tc'
20:01:29 <dims> o/
20:01:42 <ttx> Hi everyone!
20:01:44 <ttx> Our agenda for today is at:
20:01:47 <ttx> #link https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee
20:01:55 <ttx> remember to use #info #idea and #link liberally to make for a more readable summary
20:02:02 <ttx> #topic Deprecate postgresql in OpenStack
20:02:06 <ttx> #link https://review.openstack.org/427880
20:02:10 <ttx> sdague: o/
20:02:21 <ttx> quick intro ?
20:02:21 <sdague> right, this came up on the mailing list again recently
20:02:47 <sdague> the basic issue is that pg is really very unevenly supported in our community
20:02:58 <sdague> we dropped the bulk of the gate testing 5 months ago
20:03:01 <fungi> and almost entirely untested
20:03:05 <fungi> right, that
20:03:14 <sdague> the number of actual users is small (per survey results)
20:03:22 <ttx> from the comments on the thread and review, it appears some people still use it though
20:03:32 <AJaeger> SUSE's OpenStack product uses postgres and tests if heavily
20:03:35 <johnthetubaguy> sdague: +1 making this explicit for users, it keeps surpising people
20:03:47 <sdague> and with only so many resources to go around, I think we should say once and for all that this isn't an upstream supported thing
20:04:13 <sdague> ttx: some people still use a number of things
20:04:25 <sdague> I'm sure more use nova-net than postgresql
20:04:31 <Rockyg> quick point of info...Huawei uses a DB based on Postgres and that mean DT, China Mobile, etc.
20:04:43 <Rockyg> So, so big users, or soon to be big.
20:04:45 <dhellmann> this seems like one of those things like stable branches and requirements, where folks want to use it but don't contribute to it
20:04:49 <flaper87> We've done this with other tools technologies in the community
20:04:50 <fungi> the trick will be coming up with a soft enough way to say we're not actively testing it and won't know what we're breaking without coming out and implying that it just won't work
20:04:52 <mordred> yah
20:04:57 <mordred> what dhellmann said
20:05:02 <dhellmann> Rockyg : what is "a database based on postgres"?
20:05:08 <mtreinish> o/
20:05:10 <mordred> the problem isn't anything specifically against postgres - it's that it breaks and nobody cares
20:05:12 <flaper87> we have removed support for qpidd, for example
20:05:20 <fungi> dhellmann: sounds like a postgres fork
20:05:20 <Rockyg> Iwait a sec...
20:05:28 <mordred> postgres is obviously a fine database platform
20:05:52 <Rockyg> G
20:05:55 <flaper87> so, nothing against postgres but unless people raise their hands and help maintaining it, I think I'm fine with removing support for it
20:05:57 <johnthetubaguy> mordred: +1 this is more our inability to support both
20:05:58 <Rockyg> Guass
20:06:00 <Rockyg> db
20:06:03 <mordred> but if people think it's important from an _upstream_ perspective, people need to pony up to support it
20:06:03 <dhellmann> I still prefer postgres over mysql because I don't get angry every time I have to figure out how to set up a user
20:06:06 <EmilienM> flaper87: +1
20:06:07 <Rockyg> GaussDB
20:06:13 <gordc> mordred: i'm not sure that's accurate. i think it just takes longer to raise since it doesn't get surfaced as early
20:06:18 <edleafe> So is there an option for one of these users to step up and offer to fully support it?
20:06:18 <jroll> to be fair, if someone wanted to come upstream and "contribute to making postgres work", it's super unclear what that person should do, other than waiting for it to break and jumping on it
20:06:20 <mordred> dhellmann: funny, that's one of my big beefs when I try to use postgres :)
20:06:26 <dhellmann> mordred : heh
20:06:31 <ttx> I wish we'd use an abstraction layer that lets us ignore it
20:06:34 <mtreinish> dhellmann: heh, I have the opposite problem. I can never remember how to add postgres users :)
20:06:43 <dhellmann> ttx: ideally that's what sqlalchemy would be
20:06:54 <mordred> ttx: well - so this is my other opinino, where I become more opinionated
20:06:56 <Rockyg> So, this just got raised with Huawei by Gordon.  I'll need to spin them up for support in community
20:06:58 <jroll> e.g. should that person go contribute patches to optimize better for postgres, if that's what is chosen?
20:07:02 <ttx> it feels like it leaks implementation details up though
20:07:04 <fungi> i think we also need to keep in mind that the term "support" in our (upstream) case just means we do or don't actively try to avoid breaking something, distinct from distributors "supporting" some combined solution they're actively testing and taking contracts from customers over
20:07:11 <flaper87> ttx: should be oslo.db and/or sqlalchemy for that matter
20:07:15 <sdague> ttx: abstraction layers are fine when you don't need optimizations or are using basic common parts
20:07:29 <mordred> large scale things on rdbms' should be able to use advanced features of their rdbms
20:07:32 <sdague> but at some point you have to decide you are either using your technology stack effectively
20:07:34 <flaper87> but apparently it's not abstract enough because there are specific optimizations
20:07:34 <ttx> sdague: the question then becomes, how much do we rely on those advanced features
20:07:41 <mordred> unfortunately, that is not the place where databases are similar enough to abstract
20:07:45 <sdague> or just abstracting for abstracting sakes
20:08:00 <dims> contribs/patches welcome, 3rd party ci welcome, no CI jobs in community?
20:08:26 <jroll> mordred: sdague: +1, I think that is the real reason to do this, not that people aren't willing to work on it
20:08:33 <dims> but does that let us optimize for mysql?
20:08:49 <ttx> I mean, currently we are (about to) saying that projects can assume an oslo.db database is present in an openstack installation
20:08:51 <fungi> given that there are distributors preferring postgresql over mysql for openstack, i wonder whether us officially stating that we don't support it will cause them to rethink that or will just resort in more downstream patching/forks so they don't have to switch to mysql
20:08:56 <mordred> dims: I would like to open the doors for our team to optimize for _a_ database
20:08:58 <ttx> Shoud we say "MySQL" instead ?
20:09:05 <sdague> mordred: ++
20:09:08 <dims> mordred : agree
20:09:19 <mordred> I believe the inertia is there for "A database" to be mysql
20:09:25 <sdague> especially because that also opens up optimizing the HA scenarios into the environment
20:09:29 <mordred> but I would be just as happy if a database was postgres
20:09:37 <mordred> sdague: exactly
20:09:43 <ttx> And if we say that we write code for MySQL only, then should ditching PG support (in projects that support it) follow deprecation policies ?
20:09:48 <edleafe> If we have to specify one, can it at least be an Open database? Like MariaDB?
20:09:58 <johnthetubaguy> so I had a question, currently its MySQL "like" as in does it include MySQL+galera, MariaDB, MariaDB+galera, etc
20:10:11 <sdague> johnthetubaguy: I think mysql-like is the right answer
20:10:14 <mordred> edleafe: the differences between maria and mysql are miniscule
20:10:27 <sdague> the optimizations across those sets would be pretty close
20:10:27 <mordred> edleafe: if we get to the point where they matter, let's deal with it then
20:10:28 <dtroyer> mordred:  how long will that be the case?
20:10:30 <ttx> sdague: doesn't it expose some implementation details that would require extra testing though ?
20:10:33 <johnthetubaguy> sdague: yeah, I wasn't sure how to expess that I guess
20:10:34 <edleafe> mordred: in production, sure. I meant the licensing
20:10:39 <fungi> over time i have a feeling mysql, mariadb and the other mysql forks will diverge to the point where we would still need to "pick one"
20:10:44 <sdague> ttx: possibly, but we have to start somewhere
20:10:46 <ttx> fungi: ++
20:10:49 <dims> so 2 more releases with no postgres in our CI, then we optimize for mysql-like databases?
20:11:01 <dtroyer> we'll have the same problem again just withing a smaller family
20:11:11 <mordred> edleafe: has mysql stopped being gpl and I didn't noticed?
20:11:15 <dtroyer> fungi: ++
20:11:18 <gordc> is this 'optimising' still all hypothetical?
20:11:30 <ttx> So we could do a two-stage thing. Say now that in the future only MySQL will be supported. Then in two releases ditch all PG support
20:11:33 <cdent> the concern with future divergence supports my assertion that the resolution should state "yes mysql" not "no postgresql"
20:11:38 <dhellmann> if we say "mysql-like" does that include the clustering thing the folks from oracle are interested in us supporting?
20:11:43 <ttx> cdent: yes
20:11:44 <mordred> cdent: ++
20:12:05 <thingee> ++
20:12:05 <dhellmann> because that seems to have some schema constraints
20:12:06 <EmilienM> cdent: yes, I thought the same thing
20:12:20 <sdague> so, I'm fine adjusting the language as long as we are getting to the same place before we all rage quit
20:12:21 <cdent> (not that I really want to vote for mysql if the reason for it is because we want "optimizations" for our little databases ;) )
20:12:24 <mordred> dhellmann: it's a great question. I would hope so - but I think we need to think about it
20:12:30 <ttx> cdent: I would likely retrofit it in the "base services" stuff, so it would be "yes MySQL"
20:12:35 <persia> gordc: I have read reports of some tools not working with pg.  Nothing very widely deployed.
20:12:39 <edleafe> mordred: IANAL, but stuff like this makes me concerned: https://www.mysql.com/about/legal/licensing/foss-exception/
20:12:47 <mordred> cdent: when I say "optimizations" I do not mean like speed optimizations
20:12:54 <mordred> edleafe: that's been there literally for forever
20:13:00 <johnthetubaguy> dhellmann: I would rather we picked a clustering one because thats more restrictive, like MariaDB + galera, but yeah, there are differences
20:13:02 <EmilienM> some operators won't want to remove pg from their cloud, they probably have customers running clouds in production, they'll have to maintain the deployments. Not sure we've thought about it until now
20:13:12 <edleafe> mordred: And I've not liked it for literally forever
20:13:12 <mordred> edleafe: it allows foss projects that link against libmysqlclient to not need to gpl their application
20:13:22 <gordc> persia: it is being used so i imagine it's just the lag between product team noticing and patching it upstream
20:13:44 <ttx> sdague: does the two-stage thing I posted above make sense ?
20:13:44 <sdague> EmilienM: but I think they did that based on an unrealistic expectation of what is happening in upstream
20:13:48 <dhellmann> johnthetubaguy : right, my point is if we're worried about managing support for 1 database in order to optimize for it, having 2 clustering solutions with different requirements seems like it's going to be a hard thing to sell
20:14:14 <dtroyer> dhellmann: that feels like the same problem in a slightly different flavor
20:14:16 <mordred> cdent: when I say optimizations- I mean what sdague mentioned - being able to bake in more assumptions about how HA and friends work
20:14:19 <dhellmann> dtroyer : yep
20:14:32 <EmilienM> AJaeger: what is your longterm roadmap at Suse, to keep pg? Have you already prepare a migration etc?
20:14:32 <ttx> sdague: i.e. make a statement now, then start actively ditching things starting in two releases
20:14:43 <mordred> because HA stories and whatnot are where the differences between things like mysql and postgres diverge MASSIVELY
20:14:45 <johnthetubaguy> dhellmann: in my head I was supporting the top 4 most deployed databases, as best we can, I know that means we will have to do things differently to whats happening today
20:14:49 <dhellmann> mordred : it's also a matter of training folks in the community to recognize the sorts of things that can/must be done for optimization
20:14:53 <sdague> ttx: I think that the dev tooling should dump things now, we stopped testing with it in Newton
20:14:55 <EmilienM> (it would be great to also hear from operators)
20:14:56 <mordred> dhellmann: yup
20:15:03 <AJaeger> EmilienM: no migration planned yet.
20:15:07 <gordc> EmilienM: we have no plans to migrate from postgres either
20:15:14 <sdague> in pike we should remove it from devstack
20:15:28 <ttx> sdague: apparently doesn't prevent people from using it. They are surely at risk, but not a reason to remove features with no warning
20:15:36 <EmilienM> what gordc and AJaeger said is a concern we can't ignore it
20:15:37 <AJaeger> EmilienM: but this discussion here started already discussion
20:15:44 <dhellmann> johnthetubaguy : I guess we need to be clear about what "support" means there, because I think 4 may be too big of a number
20:15:49 <jroll> sdague: "we stopped testing with it in Newton" <- which we is this? nova? openstack?
20:15:53 <ttx> sdague: not sure what you mean by dev tooling, might be saying the same thing I say
20:15:56 <sdague> jroll: openstack
20:16:00 <mordred> jroll: openstack
20:16:04 <mtreinish> jroll: we removed the dsvm postgres jobs
20:16:08 <johnthetubaguy> dhellmann: +1 I meant to ask that question, I am not quite sure what that means yet either
20:16:10 <sdague> the main base layer jobs were deleted
20:16:12 <jroll> all of them?
20:16:18 <jroll> huh, TIL
20:16:19 <sdague> pretty much
20:16:25 <jroll> gate-tempest-dsvm-ironic-pxe_ipmitool-postgres-ubuntu-xenial-nv
20:16:28 <jroll> ironic tests it, apparently
20:16:32 <dhellmann> except apparently for ceilometer, since that's how the issue came up
20:16:34 <gordc> jroll: we had it in ceilometer which is where we always find the postgres issues
20:16:41 <ihrachys_> neutron has a periodic job for psql
20:16:46 <AJaeger> and ceilometer has a job up to remove it as well
20:16:54 <mtreinish> jroll: we didn't delete *postgres* just the job that ran everywhere
20:16:55 <AJaeger> has a change I mean
20:16:59 <fungi> jroll: this came up because ceilometer still tries to do some tests with postgres and there were recent changes which broke it because we don't generally integration test with that any longer
20:17:02 <mtreinish> so projects that had one offs kept them
20:17:06 <jroll> mtreinish: so "we" is the integrated gate. not openstack.
20:17:08 <fungi> er, what gordc said
20:17:14 <gordc> fungi: :)
20:17:27 <jroll> fungi: yep, I've been reading the thread
20:17:35 <dhellmann> ISTM that there's a lot more going on with postgresql in the community, dev and deployer, than we originally thought
20:17:41 <thingee> reference discussion of dropping gate job http://lists.openstack.org/pipermail/openstack-dev/2016-August/thread.html#101892
20:17:47 <sdague> dhellmann: I actually don't think that's true
20:17:55 <jroll> thingee: thanks for that
20:17:56 <ttx> sdague: basically I agree that we should align documentation with reality (now). I'd like to give a transition period before we actively break anything for people for which that may actually be working fine
20:18:10 <dhellmann> sdague : I hear you saying we're not testing it, and I hear 3 projects saying they are
20:18:11 <ttx> of which there seems to be a lot
20:18:11 <sdague> ttx: ok, but define actively break
20:18:18 <fungi> ttx: we seem to be breaking things anyway
20:18:21 <dims> dhellmann : the main thing i am worrying about is moving all the existing user deployments..
20:18:25 <mordred> the lack of postgres representation in the user survey has been very consistent
20:18:26 <fungi> (because of not testing it)
20:18:27 <jroll> to be clear, I'm not advocating for or against dropping postgres here, I just want to be clear that we're talking about the integrated gate, not openstack as a whole
20:18:28 <sdague> right, we took away all that testing
20:18:30 <sdague> mordred: ++
20:18:45 <ttx> sdague: like remove code that works around pgsql glitches
20:18:48 <thingee> mordred: ++
20:18:54 <EmilienM> dims: what about the ones who don't want to leave pg?
20:18:54 <ttx> (not in tests, in service code)
20:18:56 <dhellmann> jroll : my impression is that we're talking about encouraging all projects to drop psql support
20:19:04 <dhellmann> that's what the end of the resolution says
20:19:06 <dims> EmilienM : right
20:19:14 <flaper87> I think, like with everything else, we should provide a transition plan out of psql
20:19:22 <ttx> dhellmann: I think the resolution encourages all projects to ignore it
20:19:23 <fungi> it would be interesting to get more information on wy some operators are deploying with postgresql, and even moreso why some distributions chose to standardize their openstack designs on it
20:19:30 <ttx> flaper87: my thoughts too
20:19:32 * flaper87 doesn't like to assume it'll all be good
20:19:32 <jroll> dhellmann: sorry, I mean when we talk here about how we've stopped testing it
20:19:45 <thingee> dhellmann: yeah I prefer to think the way cdent put it. yes mysql.
20:19:47 <sdague> so, I don't think there will be a transition plan
20:19:48 <dhellmann> ttx: "And that the recommendation is they remove it from their code, documentation, and testing at the earliest convenience."
20:19:50 <EmilienM> I understand the "why" of the proposal. I'm just very worried about our community who use pg, like Suse or Huawei and probably much more
20:20:08 <pabelanger> sdague: ++
20:20:09 <sdague> the fact is, it's level of community support is very low
20:20:13 <johnthetubaguy> when we say drop support, do we mean actively break it, or make it low priority and let fixes in still, or deprecate then break it?
20:20:30 <mtreinish> EmilienM: that's why we have a deprecation procedure
20:20:36 <EmilienM> flaper87: yes that but until it's planned, we shouldn't remove support or jobs like we're doing
20:20:40 <ttx> dhellmann: right, that's the part I disagree with.
20:20:41 <gordc> johnthetubaguy: i think the current state is the 2nd
20:20:42 <sdague> saying "we have to have a transition plan" means we actually need a ramp up on it, to have any idea that it works, plus transition tooling, plus... docs?
20:20:45 <flaper87> sdague: transition plan, 1 cycle notice.. tomatos, tomatos
20:20:50 <stevemar> johnthetubaguy: " make it low priority and let fixes in " seems the least harmful
20:20:53 <flaper87> EmilienM: yup
20:20:54 <mtreinish> in reality it's a small percentage of users, and we can drop support for it gracefully
20:20:56 <bswartz> manila gate tests do ensure postgres compatibility, but if the rest of the projects start allowing postgres bugs to go unfixed we'll eventually have to drop support too because we won't be able to test
20:20:58 <ttx> dhellmann: I disagree with actively breaking it without a deprecation period
20:20:59 <dims> sdague : so another way is to get folks to show up to help? (dunno how, but...)
20:21:11 <sdague> dims: but we don't want other folks to show up
20:21:27 <dhellmann> sdague : it is not clear that everyone feels that way. I'm unconvinced.
20:21:29 * jroll points out that switching database engines is WAY harder to do than a normal feature or config deprecation, which is what our deprecation process is designed for
20:21:29 <thingee> I'm also not advocating for keeping pgsql, but this decision will further prevent the user survey changing or for it to become the defacto
20:21:31 <johnthetubaguy> ttx: I think I am with you, I would also say "formal deprecation" period
20:21:33 <mordred> right. we _want_ to be able to do better things with HA and similar stories so that we can provide a better product for our operators
20:21:39 <stevemar> jroll: ++
20:21:40 <dims> jroll : true
20:21:54 <sdague> mordred: ++
20:21:55 <AJaeger> I suggest to have at least "low prio and get bugs in " for the transition period.
20:21:55 <dims> folks probably have more things built on top of openstack
20:22:04 <EmilienM> because of the fact we did support pgsql (probably without many testing), we can't just throw it away like this
20:22:04 <sdague> and better compatibility on the API
20:22:10 <johnthetubaguy> jroll: +1
20:22:13 <ttx> mordred: I agree with the end goal. I disagree with actively breaking people for which that might accidentally work
20:22:14 * AJaeger likes to be able to upstream patches
20:22:20 <mordred> sdague: yes compat on the API
20:22:21 <sdague> EmilienM: you assume that we did support pg officially
20:22:26 <ttx> at least not without a deprecation notice/period
20:22:28 <dims> AJaeger ++
20:22:35 <mordred> ttx: sure. I do not have any problem with deprecation periods
20:22:46 <sdague> That's the crux of my issue with the requirement of making this super graceful to get rid of it
20:22:52 <EmilienM> sdague: we did or we didn't. The reality is some Clouds probably in production are using it and we can't ignore it now
20:23:12 <flaper87> EmilienM: ++
20:23:13 <sdague> the reason pg was in the gate, was because I spent 2 months making that happen in 2013 over some unrelated licensing concerns
20:23:21 <johnthetubaguy> the way I see this, the deprecation period here would help concerned folks get together I find the best way forward, the really important job we need to do here is wave the big red flag that something needs doing
20:23:33 <dhellmann> johnthetubaguy : ++
20:23:34 <mordred> johnthetubaguy: ++
20:23:42 <fungi> we collectively allowed people who figured out how to get openstack running with postgres to add sections to official deployment documentation saying how to do it, for example
20:23:43 <dims> agree johnthetubaguy
20:23:50 <ttx> we could also say "unless someone shows up to actually test it, we'll remove support for it"
20:23:54 <dhellmann> fungi : ++
20:23:59 <sdague> fungi: we did, without any resolution
20:24:01 <ttx> "because in OpenStack, what's not tested is broken"
20:24:10 <gordc> ttx: what does 'test it' mean?
20:24:12 <sdague> and this is about setting direction that we want out of that game
20:24:20 <cdent> is it allowed to directly go ask people who we know are running it to pony up resource to support it? the passive option of waiting doesn't seem great
20:24:25 <ttx> gordc: maintain functioning gate ?
20:24:31 <mordred> ttx: I'd rather we didn't say that, to be quite honest, because there is still a ceiling on how good we can make openstack when we're trying to engineer around 2 different wildly-divergent database backends
20:24:33 <sdague> cdent: I honestly don't want to go down that road
20:24:37 <sdague> mordred: ++
20:24:41 <EmilienM> ttx: "test it" and also "proactively maintain it"
20:24:52 <ttx> mordred: that is a good point. Like I said, I agree with the end goal
20:24:57 <gordc> ttx: hardware resources? or development? iirc, the original reason for removing was mainly due to hardware constraints
20:24:58 <mordred> nobody in the world who is running LARGE systems does it on "either mysql or postgres, kinda whichever"
20:25:03 <dhellmann> I'd be OK with the idea of saying we don't want maintenance if we have an extended deprecation period
20:25:07 <sdague> that's the big point, by making this an abstraction layer instead of a foundation, we fundamentally limit openstack progress
20:25:08 <fungi> worse, when trying to engineer around nuances of johnthetubaguy's four different database backends ;)
20:25:12 <mordred> sdague: ++
20:25:31 <johnthetubaguy> fungi: better than five though right ;)
20:25:35 <ttx> sdague: so we should not say we remove pgsql because it's broken, but because it prevents us from relying on advanced DB features
20:25:37 <dhellmann> fungi : right, I think we're going to end up having to pick a clustering solution, too
20:25:38 <sdague> and, I hear, people get frustrated and rant about pace of openstack progress some times
20:25:44 <EmilienM> gordc: I think the biggest challenge is to maintain CI jobs running well with pg (and deal with failures when they happen, like we currently do with mysql)
20:25:56 <flaper87> so, I think there's agreement on the goal but not entirely on how we get there
20:26:05 <mordred> dhellmann: yes. being able to pick one of those would be amazing and allow us to move the needle forward
20:26:11 <flaper87> should we proceed with voting either here or on the review?
20:26:14 <ttx> sdague: are they saying we are too fast or too slow ? Because I hear both
20:26:23 <mordred> ttx: I hear both at the same time many times :)
20:26:32 <gordc> EmilienM: periodic gate like neutron has now? i know the bugs we find at product end have been pushed back into upstream
20:26:34 <dtroyer> Being more opinionated about things like this (messaging, etc too) will let us speed up somewhat
20:26:51 <gordc> but there's a noticeable lag between upstream patch to product team
20:26:51 <johnthetubaguy> sdague: +1 we will have to be more opinionated to go faster, and keep going
20:27:02 <flaper87> I feel we're starting to say the same things so I'd rather move onto the next phase of the discussion
20:27:10 <mordred> flaper87: ++
20:27:14 <ttx> flaper87: yes please
20:27:18 <johnthetubaguy> so what do we write on the big red flag?
20:27:27 <mordred> johnthetubaguy: a sonnet?
20:27:34 <ttx> Sounds like there is some consensus on the end goal, but diagreement on the way to state it and how fast to proceed
20:27:34 <flaper87> so, should we vote on whether we should have a deprecation period here or on the review ?
20:27:44 <johnthetubaguy> mordred: if that makes people read it, it could totally work :)
20:27:47 <ttx> flaper87: I propose we continue on the review
20:27:48 <jroll> I think a good first step would be make sure install guide says mysql
20:27:49 <mordred> johnthetubaguy: :)
20:27:52 <mordred> ttx: ++
20:27:52 <flaper87> ttx: I like that
20:27:53 <jroll> I don't think any of us would disagree with that
20:27:59 <ttx> it will take some time to hammer out
20:28:13 <jroll> the changing existing pg deployments is the hard part
20:28:21 <AJaeger> jroll: It does already
20:28:23 <mordred> ttx: maybe we should also tee up a session at the Forum to discuss this
20:28:30 <dhellmann> mordred : ++
20:28:32 <AJaeger> jroll: and only MySQL/MariaDB
20:28:33 <gordc> ++
20:28:33 <fungi> also as has been pointed out, this will serve as prior art for the message bus discussions still to come
20:28:34 <Rockyg> mordred, ++
20:28:35 <jroll> AJaeger: it doesn't mention postgres at all?
20:28:37 <jroll> oh nice
20:28:44 <mordred> fungi: ++
20:28:51 <Rockyg> also at ops midcycle
20:28:55 <AJaeger> jroll: Correct -t he Install Guide shows basically one path - and opinoined way to install
20:28:55 <gordc> jroll: it says postgres is also supported in install guide
20:28:56 <ttx> #action mordred to propose a pgsql discussion at PTG (and probably another one at Forum) when time comes
20:29:03 <sdague> jroll: again, what gets exposed in the community around pg is really inconsistent
20:29:06 <flaper87> mordred: thanks
20:29:06 * jroll reads conflicting information
20:29:07 <gordc> but all the steps are for mysql
20:29:07 <mordred> ttx: wow. I shoudl have seen that coming
20:29:12 <ttx> mordred: I'm fast
20:29:18 <ttx> OK, I propose we move on
20:29:21 <flaper87> +1
20:29:21 <jroll> sdague: yeah so we should clean up stuff that new deployments would use to just mysql
20:29:24 <thingee> yes
20:29:25 <johnthetubaguy> I think we need something before then, though, but we can talk on the review about that
20:29:26 <jroll> and go from there with existing stuff
20:29:28 <sdague> jroll: yep
20:29:35 <johnthetubaguy> then==forum
20:29:35 <fungi> i still mostly want to know _why_ people are chosing the less popular option here, so that we have better data on whether it's just an arbitrary choice or there are definite behavior/performance needs not being met by the default option
20:29:42 <ttx> let's iterate on review, and discuss next topic
20:29:56 <ttx> dolphm: around ?
20:29:56 <fungi> sounds good
20:30:06 <johnthetubaguy> fungi: +1 me too
20:30:08 <dolphm> ttx: o/
20:30:34 <ttx> #topic Disallow downtime of controlled resources during upgrades
20:30:37 <ttx> #link https://review.openstack.org/404361
20:30:42 <ttx> The language in the latest draft looks good to me
20:30:52 <ttx> I guess the main unknown is how exactly you would validate "accessibility of a controlled resource" throughout the upgrade process
20:31:00 <ttx> Some grenade code ?
20:31:29 <stevemar> ttx: some tests, yes, could be grenade
20:31:34 <johnthetubaguy> an ssh session being open, kept coming up, just saying that for reference really
20:31:35 <mtreinish> ttx: grenade does poll things while the services are shutdown
20:31:42 <dolphm> i'm hoping to answer that at the PTG, but yes... my thought is to extend/adapt the resources that grenade creates beforre an upgrade, to also be exercised during the upgrade
20:32:16 <dolphm> mtreinish: so, if that becomes an asynchronous an ongoing validation process, i think ti satisfies the tag for the projects it covers
20:32:34 <dhellmann> that seems like a reasonable approach, and I think we can add details to the tag with some examples later when we have some
20:32:43 <ttx> mtreinish: ok, so enough to assess "accessibility of a controlled resource"
20:32:45 <sdague> I think that's going to be the basic challenge, everything we do in testing today is sequential
20:33:25 <sdague> because it's not just the validation, it's how the actual upgrades are done as well
20:33:38 <johnthetubaguy> sdague: +1 was just tying something like that
20:34:21 <mtreinish> ttx: well it's not a continous check more after thigns are shutdown can I still ping server x
20:34:40 <ttx> mtreinish: I think that the new wording is good with that
20:35:12 <dolphm> ttx: i disagree, the keyword being "throughout" the upgrade process
20:35:22 * ttx looks up latest patchset
20:35:34 <dolphm> today, the accessibility of controlled resources is tested basically once during the upgrade process
20:35:46 <sdague> dolphm: well, 3 times
20:35:46 <dolphm> ttx: L59 https://review.openstack.org/#/c/404361/8/reference/tags/assert_supports-accessible-upgrade.rst
20:36:05 <sdague> before services drop, when they are down, after they are upgraded
20:36:11 <dolphm> sdague: once before the upgrade begins, once during the upgrade process, and once after the upgrade?
20:36:11 <ttx> Should we add some precision so that testing 3 times is enough ?
20:36:24 <ttx> I just want to remove the uncertainty there
20:36:24 <mordred> one could imagine perhaps opening an ssh connection to a vm in a thread/subprocess, then doing the upgrade, then making sure the ssh session doesn't drop
20:36:36 <fungi> on the ssh session idea... ssh can resume seamlessly after rather lengthy periods of total packet loss
20:36:39 <ttx> mordred: or that
20:36:42 <dhellmann> I don't think we need to solve the implementation ideas in this meeting, do we?
20:36:48 <mordred> nope
20:36:52 <dims> nope
20:36:53 <mordred> not evne a little bit :)
20:37:01 <sdague> well, I think we do want to have an implementation before the tag though
20:37:10 <ttx> ok, so we can judge that when the tag is proposed
20:37:18 <ttx> to a specific project
20:37:20 <dolphm> that's true
20:37:21 <mordred> I'm more musing on that by way of exploring what sorts of things this could be wanting to require
20:37:21 <sdague> because if we think it's testable, we should work that out first
20:37:26 <dhellmann> I'm not sure we do. We've clearly agreed on "throughout". Different projects may have different ways of testing that.
20:37:30 <fungi> you can ssh to an instance, suspend the instance for an hour or more (in the right network environments at least), resume the session and then go back to using that same socket without it ever disconnecting
20:37:30 <johnthetubaguy> we get to revise the tag until someone gets the tag?
20:37:45 <ttx> I suspect we'll refine the tag definition when a project actually asks for it :)
20:37:46 <flaper87> dhellmann: ++
20:37:47 <sdague> plus, we all need to realize that testing is a model
20:37:53 <johnthetubaguy> sdague: I still wonder if it has value in saying "hey we don't test this thing yet"
20:37:54 <dhellmann> johnthetubaguy : or after, if it's a backwards-compatible change
20:37:59 <sdague> the point isn't to catch everything
20:38:04 * ttx is good with that. Baby steps
20:38:09 <mordred> sdague: ++
20:38:10 <sdague> the point is to catch representative errors that people missed in code review
20:38:19 <ttx> If you feel good, pile up +1s and I'll approve now
20:38:38 <johnthetubaguy> sdague: +1
20:38:50 <fungi> right, which is why i think it's sufficient to just test sample (once, twice, whatever) during the process and assume things weren't offline in between and miraculously came back just in time for the next poll
20:39:04 <fungi> though that's also a good way to sneak nondeterminsitic failures in
20:39:08 <ttx> One more +1 to approval
20:39:14 <ttx> Any objection ?
20:39:18 <johnthetubaguy> I feel we should merge what we have, and adjust it if we need to before we approve the first project to get the tag
20:39:24 <ttx> johnthetubaguy: yes
20:39:25 <sdague> johnthetubaguy: that seems fine
20:39:30 <dhellmann> fungi : ++ to using reasoning to apply good, minimal testing practices
20:39:33 <dtroyer> +1
20:39:39 <thingee> +1
20:39:41 <EmilienM> johnthetubaguy: +1, baby steps
20:39:41 <ttx> Alright, approved
20:39:53 <ttx> Thanks dolphm !
20:39:59 <fungi> dhellmann: and keeping an eye out for races. polling-related test like that are a great surface area to grow race conditions on
20:40:00 <stevemar> ++ thanks dolphm
20:40:03 <ttx> #topic Progress on stalled reviews
20:40:07 <dhellmann> fungi : true
20:40:08 <ttx> * Status update on Golang CTI
20:40:11 <ttx> #link https://review.openstack.org/410355
20:40:14 <ttx> dtroyer: how is that going ?
20:40:21 <dolphm> cool, landing these tags has certainly driven a lot of conversations in the right direction (reasonable testing practices, as someone just mentioned)
20:40:23 <dolphm> so thank you all!
20:40:29 <Rockyg> fungi, ++
20:40:52 <dtroyer> I need to get back to it, runup to release has consumed me, but I think it's pretty close
20:40:53 <dhellmann> dolphm : thanks for working on this
20:41:17 * dims needs to review that golang_cti in depth
20:41:23 <ttx> dtroyer: ok, so WIP but you'll pick it up soon
20:41:26 <dtroyer> I kept flaper87's doc in mind and IIRc there may only be minor changes to fit that
20:41:27 <ttx> On that same topic...
20:41:30 <ttx> flaper87: do we want to grandfather in that step 1 for golang addition (the use case analysis) based on the Swift case ?
20:41:37 <ttx> Did you find any volunteer to file that ?
20:42:02 <flaper87> I sent the email out and reached out but no one signed up for it AFAICT
20:42:08 <flaper87> unless I missed something entirely obvious
20:42:21 <flaper87> I would love to do a first runtest of this process with Swift
20:42:24 <dtroyer> I've been operating under the assumption that we can refer to that as necessary
20:42:25 <ttx> ok, maybe reach out to the ones that should be interested see if they still care
20:42:42 <flaper87> ttx: erm, done that already
20:42:51 * flaper87 shrugs
20:42:56 <ttx> ok. Maybe we could tie loose ends at PTG
20:43:02 <ttx> * Status update on Driver discoverability / need for driver team recognition
20:43:10 <ttx> A couple weeks ago we tabled that discussion while waiting for (some) progress on driver discoverability
20:43:11 <flaper87> yes, I'm hoping for that and feel free to action me on it
20:43:18 <ttx> I was wondering when we would be comfortable to reconsider those various options for driver team recognition
20:43:41 <ttx> #action flaper87 to tie loose ends at PTG re: step 1 for golang adoption (use case analysis)
20:43:54 <ttx> So when is that ? post-PTG ? Later ?
20:44:14 <ttx> thingee: I suspect there won't be much visible progress on discoverability before a while ?
20:44:26 <flaper87> +!1 for post PTG to take the drivers discussion again
20:44:35 <flaper87> +1*
20:44:46 <ttx> Also a bit of PTG beers might convince the ones or the others
20:44:49 <stevemar> yeah, post-PTG
20:44:56 <stevemar> beers works too
20:45:39 <fungi> it sounded like there was a fair amount of interest from the www site admins on revamping/improving the driver marketplace and solving the need to automatically aggregate driver details from structured data under the control of individual teams
20:45:48 <thingee> ttx: not yet
20:45:54 <ttx> fungi: yes, but will likely take some time to implement
20:45:58 <fungi> agreed
20:46:16 <fungi> it seems we have a path forward on that front at least, if not a timeline (yet anyway)
20:46:21 <ttx> ok, let's revive it post-PTG and use that week to informally discuss it
20:46:27 <thingee> ttx: current noticable progress is neutron moving towards the same common format that nova started for listing drivers. I would like to start communication with other projects to follow
20:46:45 <ttx> thingee: sounds like a good step forward!
20:46:52 <thingee> cinder as I believed from various people in the team back in barcelona are also going to be moving towards this same common format
20:46:52 <ttx> * Progress on Introduce assert:supports-api-compatibility
20:46:59 <dhellmann> thingee : is neutron listing drivers that are outside of its repos?
20:47:04 <ttx> #undo
20:47:05 <openstack> Removing item from minutes: #action flaper87 to tie loose ends at PTG re: step 1 for golang adoption (use case analysis)
20:47:10 <ttx> arh
20:47:12 <thingee> once that's in place the foundation will work with the community on displaying things in the market place with the common format.
20:47:18 <ttx> #action flaper87 to tie loose ends at PTG re: step 1 for golang adoption (use case analysis)
20:47:23 <mtreinish> ttx: hah :)
20:47:26 <flaper87> ttx: stop taking work away from me
20:47:32 <flaper87> no wait, do it! take work away from me
20:47:34 <ttx> I need #subtopic
20:47:39 <thingee> dhellmann: yes
20:47:42 <mtreinish> that meetingbot always surprises
20:47:44 <dhellmann> thingee : ok, good
20:47:51 <mtreinish> everytime I undo it's never what I want either
20:47:51 <ttx> #link https://review.openstack.org/418010
20:47:59 <mordred> ttx: patches accepted :)
20:48:03 <ttx> This one was blocked while the API-WG refreshes their "API changes guidelines" that this proposal references
20:48:12 <cdent> yeah, still working on that
20:48:23 <mordred> it's a fun mailing list thread
20:48:35 <cdent> in the rush of release, has been slow generating the necessary input to get a full picture of the context
20:48:38 <ttx> mordred: already need to add Slack-like 'notify me when I'm mentioned in another channel the bot lurks in" feature
20:48:41 <thingee> dhellmann: https://review.openstack.org/#/c/318192/
20:48:44 <cdent> but it's getting there
20:48:50 <mordred> thingee: EDONOTWANT
20:48:52 <mordred> gah
20:48:57 <mordred> ttx: EDONOTWANT
20:49:19 <thingee> mordred: ?
20:49:20 <ttx> mordred: would let us get rid of common meeting channels though :)
20:49:23 <dhellmann> thingee : maybe we can move some of that css and templating into oslosphinx when we have 2 that are similar enough
20:49:50 <ttx> cdent: ok cool, the other review is stalled while we wait for that step to complete
20:50:01 <thingee> dhellmann: yes there is already duplication happening with the script that generates it.
20:50:12 <cdent> ttx: yeah, that's the somewhat regrettable but desired outcome
20:50:13 <mtreinish> ttx: to be fair a lot of projects have been living under the current guideline for a while
20:50:23 <mtreinish> it was a tc approved thing back in the day
20:50:34 <smcginnis> Cinder patch for driver info: https://review.openstack.org/#/c/371169/
20:50:38 <ttx> mtreinish: so technically we could pursue both in parallel ?
20:50:40 <thingee> smcginnis: excellent!
20:50:49 <mtreinish> err or ppb: https://wiki.openstack.org/wiki/Governance/Approved/APIStability
20:50:54 <cdent> ttx, mtreinish I don't think that a good idea
20:50:56 <dhellmann> thingee : either oslosphinx or a new project, whatever's easier
20:51:03 <cdent> the existing guidelines are flat out wrong in many ways
20:51:04 <smcginnis> I'd also like to point out we have driver information automatically pulled out of the source and published here: http://docs.openstack.org/developer/cinder/drivers.html
20:51:08 <ttx> mtreinish: archeowiking is not that great
20:51:41 <cdent> and i think the modern reality of the big tent and the concept of tags changes the concepts of stability quite a bit
20:51:44 * ttx keeps on adding ETOOOLD notices to pages that some people exhume
20:51:54 <mtreinish> ttx: sure, I'm just saying this tag was more my attempt to modernize the rules projects have been using
20:51:55 <cdent> at least the discussion thus far certainly suggests that to be the case
20:51:56 <johnthetubaguy> smcginnis: very nice
20:52:12 <ttx> ok, so let's see where cdent's modernization effort takes us first
20:52:19 <cdent> shouldn't be too much longer
20:52:20 <ttx> #topic Open discussion
20:52:21 <dims> go cdent!
20:52:26 <ttx> * Board+TC workshop on "OpenStack Futures"
20:52:32 <ttx> So finally it looks like this will happen March 8-9 in Boston
20:52:33 <thingee> smcginnis: although hmm, I was kind of hoping we'd have one way to generate these. This seems different from neutron did with the .ini file
20:52:40 <sdague> cdent: to be fair, I think that 90% of the guideline is pretty much going to be the same
20:52:41 <ttx> a.k.a. "week of March 6 in Boston" in the poll
20:52:44 <dims> nice thanks ttx
20:52:49 <flaper87> mmh, does dates are horrible for me :(
20:52:50 <sdague> cdent: only 10% really gets tweaked
20:52:54 <ttx> There would be a 1-day workshop with the board on March 8, followed by some common evening activity (which should include food)
20:52:54 <flaper87> I won't be able to attend
20:52:56 <thingee> smcginnis: how is nova generating theirs?
20:53:01 <cdent> sdague: that's not the impression I've had from discussion, actually
20:53:02 <dhellmann> #info all milestone-based projects have tagged their RC1 and branched stable/ocata
20:53:07 * flaper87 thought we were voting for March 6th
20:53:14 <cdent> which is why I've been stalling following through on rubber stamping it
20:53:15 <ttx> On March 9 the Board will hold a traditional board meeting, while interested TC members would work on the "TC vision", maybe with a facilitator from ZingTrain if gothicmindfood can arrange that
20:53:19 <smcginnis> thingee: I'll have to see what they did. I'd rather not maintain an ini file, but if that makes everyone consistent, then all for it.
20:53:22 <cdent> because there's lots of input that contradicts the cw
20:53:28 <ttx> flaper87: frankly, no date was perfect
20:53:29 <EmilienM> flaper87: :(
20:53:38 <ttx> but yes, we assumed it was ok with you based on that vote
20:53:44 <ttx> Those TC members not interested in working on the TC vision nor in the board meeting could go home early
20:54:03 <ttx> I'm confirming all the details and will send something out
20:54:08 <ttx> as we need to RSVP
20:54:08 <thingee> smcginnis: I'll take a look at nova and bring it up in the thread.
20:54:15 <smcginnis> thingee: Cool!
20:54:25 <flaper87> bah :(
20:54:26 <ttx> Will very likely happen in DLA Piper offices, which means we have to RSVP
20:54:27 <sdague> thingee: nova's is generated with a custom sphinx extension, johnthetubaguy wrote it
20:54:40 <sdague> oh, sorry, danpb originally wrote it
20:54:41 <ttx> The address is 33 Arch Street
20:54:45 <thingee> sdague: https://review.openstack.org/#/c/371169/14/doc/ext/driver_support_matrix.py ?
20:54:53 <johnthetubaguy> yeah, its evolved from dan's
20:54:54 <sdague> thingee: yeh
20:55:06 <mtreinish> ttx: ok cool
20:55:10 <sdague> ini file was used because parser was already in python
20:55:25 <sdague> and, the alternatives were actually pretty much just a gross
20:55:39 <johnthetubaguy> thingee: the plan was to move that into some oslo lib, not quite done that bit yet
20:55:54 <ttx> let me know if you have any questions. We coupled both things to make the travel more worthwhile and avoid yet another trip for TC visioning
20:56:10 <dhellmann> thingee: I haven't looked at the rendering code for the driver stuff, but it sounds like the sort of thing sphinxcontrib-datatemplates would be good for
20:56:29 <dhellmann> ttx: how firm are the dates?
20:56:46 <stevemar> ttx: will a formal invite (or something else) be sent out ?
20:56:48 <fungi> yeah, it's right by boston common. at least there are lots of places to eat ;)
20:57:01 <ttx> dhellmann: my understanding is that they are final. I'd like to doublecheck that though, since it's VERY fresh
20:57:06 <ttx> like 5min ago
20:57:09 <dhellmann> ttx; ok
20:57:12 <EmilienM> ttx: can you send details asap so we can book flights, etc?
20:57:15 <dhellmann> I'll wait to book a flight :-)
20:57:26 <ttx> That is my plan. After all I'm the one coming from the furthest away
20:57:29 <johnthetubaguy> dhellmann: thingee: there are some nova OSIC folks looking at tidying ours up again next cycle, let me know if there is a better direction to consider
20:57:30 * stevemar still needs to be book his ptg flight
20:57:39 * flaper87 is literally going on PTO on the 8th, something that doesn't happen often
20:57:46 <thingee> smcginnis: http://git.openstack.org/cgit/openstack/nova/tree/doc/source/support-matrix.ini
20:57:54 <fungi> ttx: well, some of the board members are probably attending from asia ;)
20:58:08 <dhellmann> flaper87 : enjoy the time off!
20:58:11 <ttx> fungi: I won't send them any confirmation though
20:58:13 <EmilienM> flaper87: well deserved
20:58:18 <fungi> ttx: fair enough
20:58:32 <smcginnis> thingee: Looks like I could probably adapt our automated way to generate most of that ini.
20:58:43 <ttx> flaper87: I don't expect anything final to come out of those two days. Will be missing you though
20:58:55 <ttx> Anything else, anyone ?
20:59:41 <ttx> Oh, re: PTG we have an ethercalc instance now, so I'll be setting up booking forms for the discussion room and other shared resources
20:59:49 <ttx> not forms, more like tables
21:00:08 <ttx> so we'll be able to start booking extra stuff
21:00:14 <mordred> https://ethercalc.openstack.org/ <-- yay
21:00:18 <stevemar> ttx: the info will all be here right? https://wiki.openstack.org/wiki/Governance/Foundation/8Mar2017BoardMeeting
21:00:19 <ttx> and we are out of time
21:00:34 <ttx> stevemar: mayyyybe ? I'll send something to the TC list
21:00:41 <stevemar> ttx: thanks
21:00:52 <ttx> mordred: yes, please test and crash it so that we know if we can rely on it or not :)
21:01:02 <ttx> and we are out of time
21:01:06 <ttx> Thanks everyone
21:01:08 <ttx> #endmeeting