20:00:09 <redrobot> #startmeeting barbican
20:00:09 <openstack> Meeting started Mon Sep 28 20:00:09 2015 UTC and is due to finish in 60 minutes.  The chair is redrobot. Information about MeetBot at http://wiki.debian.org/MeetBot.
20:00:10 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
20:00:12 <openstack> The meeting name has been set to 'barbican'
20:00:15 <kfarr> o/
20:00:24 <redrobot> #topic Roll Call
20:00:32 <silos> /o/
20:00:36 <rellerreller> o/
20:00:37 <dave-mccowan> \o/
20:00:53 <hockeynut> (looking for chellygel-approved ascii sequence)
20:00:57 <hockeynut> (not finding one)
20:00:59 <hockeynut> o/
20:01:09 <redrobot> hockeynut lol
20:01:18 <jkf> o/
20:01:26 <spotz> o/
20:01:27 <redrobot> as usual the meeting agenda can be found here:
20:01:32 <redrobot> #link https://wiki.openstack.org/wiki/Meetings/Barbican
20:01:45 <dave-mccowan> ¯\_(ツ)_/¯
20:02:03 <hockeynut> oh dave-mccowan thats way too cool
20:02:10 <arunkant> 0/
20:02:25 <woodster_> o/
20:03:05 <elmiko> o/ (sorta)
20:04:58 <rm_work> o/
20:05:53 <jhfeng> o/
20:06:15 <redrobot> lots of barbicaneers here
20:06:19 <redrobot> let
20:06:23 <redrobot> 's get started
20:06:33 <redrobot> #topic Liberty RC1
20:06:45 <redrobot> in case you missed it, RC1 was released over the weekend
20:07:05 <redrobot> #link https://launchpad.net/barbican/liberty/liberty-rc1
20:07:38 <redrobot> Looks like we'll be needing an RC2 to address this bug
20:07:42 <redrobot> #link https://bugs.launchpad.net/barbican/+bug/1499876
20:07:42 <openstack> Launchpad bug 1499876 in Barbican "Attempt to delete project preferred CA should return 409" [Critical,Triaged]
20:07:44 <woodster_> Already planning rc2, or will that do it? :)
20:07:58 <redrobot> woodster_ read your mind ;0
20:08:19 <dave-mccowan> woodster_ is your db-manage patch a RC2 candidate?
20:08:43 <woodster_> dave-mccowan: do you mean that latest one?
20:08:51 <dave-mccowan> y
20:09:25 <woodster_> probably not worth it by itself...it affects CI/CD processes that utilize it though
20:09:39 <alee> o/
20:12:08 <redrobot> woodster_ what's the bug #?
20:12:25 <woodster_> #link https://bugs.launchpad.net/barbican/+bug/1500448
20:12:25 <openstack> Launchpad bug 1500448 in Barbican "db_manage.py script doesn't return error code" [High,New] - Assigned to John Wood (john-wood-w)
20:13:42 <redrobot> woodster_ thanks
20:13:54 <redrobot> I can add that to RC2 since we'll be having to spin one anyway
20:14:04 <redrobot> any other questions/comments about RC1 or RC2 ?
20:14:36 <woodster_> redrobot: what else is going into rc2?
20:14:52 <redrobot> woodster_ so far only the two bugs we just talked about
20:15:07 <redrobot> woodster_ and possibly alee's support for sub-cas in DogTag
20:15:08 <woodster_> redrobot: oh got it, sorry
20:15:13 <alee> redrobot, and dogtag plugin suport
20:15:15 <alee> :)
20:15:44 <alee> redrobot, yes -- almost done with that. just getting the functional tests running
20:16:12 <redrobot> ok, moving on
20:16:56 <redrobot> #topic Testing Alembic migrations at the gate
20:17:02 <redrobot> woodster_ this is your topic
20:17:24 <woodster_> redrobot: thanks. This is regarding the use of the alembic migration script to populate teh database from scratch
20:17:46 <woodster_> ...vs just using SQLAlchemy to populate the database as is done now in the gate job
20:17:53 <silos> woodster_.: so would this use the db_manage script then?
20:18:00 <woodster_> silos: that's correct
20:18:06 <redrobot> silos so the issue is that alembic never gets tested at the gate
20:18:22 <redrobot> and this is the second cycle where we scramble during RC week to get migrations working
20:18:39 <redrobot> I think that adding a Grenade job would help this to some extent
20:19:09 <rellerreller> redrobot what else would be needed?
20:19:33 <redrobot> rellerreller I'm not 100% on Grenade jobs, but I think they only test N-1-> HEAD schema
20:19:49 <redrobot> rellerreller so they wouldn't cover the brand-new-db -> HEAD migration
20:20:09 <rellerreller> redrobot would SQLAlchemy cover the latter use case?
20:20:28 <redrobot> rellerreller no... the problem is that SQLAlchemy bypasses Alembic at the gate right now
20:21:01 <redrobot> it is possible now, to add alembic migrations that don't match up to the actual models in sqlalchemy
20:21:21 <rellerreller> so if I use SQLAlchemy to populate the db and Alembic, what is the delta for new-db use case?
20:21:27 <woodster_> For most ci/cd processes, the db-manage script would be used to keep an existing database up to date. It would be better for this to be run using a privileged user as well, separate from the user the barbican api uses. It would just make sense to use the db-manage script for the time-zero database case as well.
20:22:07 <redrobot> rellerreller alembic was flat out failing, but woodster_ submitted a few patches to get it up to date
20:22:12 <woodster_> rellerreller:  the new db case starts with the juno script that populates the basic tables
20:23:28 <woodster_> the gate would at least test the viability of the scripts for MySQL. Eventually a gate for postgreSQL would be good to have.
20:23:45 <rellerreller> woodster_ why does the new db case start with the juno script?
20:24:01 <redrobot> rellerreller because juno is still supported by our team
20:24:01 <woodster_> rellerreller: that is the first script to run in the sequence
20:24:11 <rellerreller> I'm not opposed to any of this. I'm asking for clarification. I'm not certain of how all of this works, just vaguely
20:24:30 <rellerreller> I only know Hibernate :(
20:24:33 <redrobot> rellerreller once Liberty final is released, we will probably want to squash the migration such that Kilo is the new base
20:24:51 <hockeynut> redrobot could also look into javelin - its for general resource validation after install/upgrade, but not sure if it handles DB or not
20:25:05 <rellerreller> redrobot so the install for a fresh, new machine is run the kilo script to setup db and then migrate to liberty?
20:25:07 <dave-mccowan> it's now possible to make changes to the models without matching alembic migrations.  is that a "feature" to speed development or is that a problem to be fixed?
20:25:54 <woodster_> as mentioned, an issue that could creep in is that the latest SQLAlchemy models and the migration scripts diverge. My thinking is that running the full suite of functional tests against the final database is a good check on that though. We might also add explicit schema comparison to the gate as well eventually.
20:26:02 <redrobot> dave-mccowan reviewers are expected to check the validity of alembic migrations when a model change is introduced.
20:26:24 <woodster_> yeah, unfortunately folks can hand code the alembic scripts
20:26:48 <woodster_> and ugly things like the dreaded 'op.execute()' call get pulled in
20:26:49 <rellerreller> woodster_ :( I doubt that is happening
20:27:07 <redrobot> woodster_ I think that devs are expected to edit the alembic files.
20:27:17 <woodster_> rellerreller: take a look at the CR I put in recently :)
20:27:35 <rellerreller> I am in favor of a migration gate check. I think any change that changes a model should fail unless model and alembic scripts are good.
20:27:43 <woodster_> redrobot: The auto generation process isn't perfect, that's true
20:27:56 <rellerreller> I do not like it when patches are submitted that change model and do not have migration.
20:28:19 <redrobot> So, the proposed change is to: 1. Stop generating the current schema by looking at the models. 2. Use alembic to generate the schema and then run the functional tests.
20:28:40 <woodster_> well, my basic proposal is to not auto popualte the db from barbican as done now in the gate, but to rather disable that and add a db-migration step
20:28:51 <woodster_> redrobot: ha, yep!
20:29:13 <redrobot> anyone opposed to that gate change?
20:29:28 <woodster_> eventually we'll add a grenade job as well
20:30:03 <redrobot> my only concern is that we need to be able to do the alembic migration in other gate jobs that are enabling Barbican in their gate
20:30:17 <rellerreller> Is there a way that Alembic could work but creating from SQLAlchemy could fail?
20:30:37 <rellerreller> My gut says no, but I wanted to float the question.
20:30:47 <woodster_> rellerreller: well, the app uses SQLAlchemy directly, so I wouldn't expect that is possible
20:30:53 <rellerreller> I figure the functional tests would catch any differences.
20:31:01 <redrobot> rellerreller possibly, but that should be caught by a failing functional test
20:31:06 <rellerreller> woodster_ we are on the same page :)
20:31:12 <woodster_> rellerreller: that is my thinking as well...ha, yes!
20:31:20 <rellerreller> haha
20:31:30 <dave-mccowan> do we want to add a liberty_release.py script to RC2?
20:31:35 <woodster_> redrobot: what other gate jobs? Such as the dogtag one?
20:31:35 <hockeynut> if there's a scenario that needs a test then by all means lets get them in
20:31:46 <redrobot> #agreed we're all in violent agreement. :)
20:32:03 <redrobot> dave-mccowan I don't think so
20:33:02 <woodster_> redrobot: we could just start with the basic devstack gate
20:33:04 <dave-mccowan> there's a kilo_release.py now.  when would we add liberty_release.py?
20:33:46 * rellerreller packing up for the day and getting ready to leave
20:34:30 <woodster_> dave-mccowan: is this the all-tables-for-liberty module? If so then I think you start two releases back, so N would have the liberty one?
20:36:59 <dave-mccowan> juno_initial.py, kilo_release.py, <20 files for liberty>.py   it would seem nice to me if we squashed the 20 files for liberty into one before the release.
20:38:02 <redrobot> I wonder if there is some sort of guideline for this?  surely we're not the only ones using alembic
20:38:04 <dave-mccowan> (or maybe misunderstanding why we have *_release.py files)
20:38:20 <redrobot> dave-mccowan my impression is that xxxx_release.py is only created when that release becomes the new baseline
20:38:34 <redrobot> and liberty won't be a baseline until the N release
20:38:46 <woodster_> redrobot: +1, that's my impression as well
20:39:12 <dave-mccowan> why do we have kilo_release.py?
20:39:28 <redrobot> dave-mccowan not entirely sure... I think it was pushed last week
20:39:36 <woodster_> we can make a liberty_release.py after the final liberty release
20:40:11 <woodster_> I think each release can get its own version tagging module
20:40:13 <redrobot> woodster_ +1 that was my thinking also
20:40:30 * rellerreller walks out the door to exit the meeting
20:40:37 * redrobot waves at rellerreller
20:40:42 <woodster_> rellerreller: take care!
20:41:43 <woodster_> I think the difference is the very first release is the full db load...the other release pys are just tagging ones
20:43:10 <redrobot> woodster_ yeah... a no-op migration...
20:44:16 <woodster_> yep, so we'd need to convert the kilo no op to a real time-zero db load during the M release cycle, and drop the juno release
20:45:17 <dave-mccowan> there must be something that we should have done for liberty during the liberty cycle.
20:45:50 <redrobot> dave-mccowan kilo is actually in the middle of all the xxxxx.py migrations.  the tree does not go from juno->kilo->xxxx, but rather juno->xxxxx->kilo->xxxxx
20:46:09 <redrobot> dave-mccowan I don't believe we need to push that before
20:46:26 <redrobot> dave-mccowan it's really just a placeholder, so that you know where kilo ended
20:46:42 <woodster_> yeah, I think by the time you release liberty, you should have a time zero load two releases before (so Juno). Seems we are on track with that
20:46:47 <redrobot> dave-mccowan or in liberty case, once liberty final is out, it will be a placeholder to let us know where liberty ended.
20:47:08 <redrobot> dave-mccowan it's only used after n+2 releases when we need to squash all the migrations
20:48:07 <woodster_> yep, so for the next release cycle, that kilo_release.py will be come the initial version file, and so be converted from a no-op to a time zero db load.
20:48:32 <redrobot> woodster_ +1 ... and all the migrations between juno and kilo get squashed into that new kilo_release.py
20:49:06 <redrobot> dave-mccowan we definitely need a liberty_release.py, but it does not actually need to be part of liberty
20:49:20 <redrobot> dave-mccowan just needs to be there before we push the first Mitaka migration
20:49:34 <woodster_> I believe we could have time zero loads for each release...Alembic supports multiple branches. Not sure how helpful that is though
20:49:38 <woodster_> redrobot: +1
20:52:53 <redrobot> only a few more minutes left
20:53:25 <jaosorior> Any workflows for this? https://review.openstack.org/#/c/198732/ would be nice to finally get it merged :D
20:53:51 <woodster_> ...and arunkant 's ACL CRs too
20:54:09 <hockeynut> I think we only require a single +2 now, so a second +2 should be workflow?
20:54:20 <arunkant> thanks _woodster, yes it has been pending for a while
20:54:35 <redrobot> hockeynut 2nd reviewer should give both a +2 and a +Workflow
20:54:40 <hockeynut> arunkant see my comments - some easy ones.
20:54:45 <hockeynut> thx redrobot - gotcha
20:54:52 <jaosorior> arunkant: I've +2ed your first two patches but I haven't gotten around reviewing the functional tests
20:54:52 <redrobot> hockeynut unless the 2nd reviewer doesn't want to workflow just yet, so they could give a +2 only.
20:55:00 <woodster_> jaosorior: I'll take a look...I had +2ed in the past
20:55:06 <hockeynut> makes sense redrobot
20:56:03 <arunkant> hockeynut : will look into your comments
20:56:12 <hockeynut> thx!
20:56:25 <jaosorior> woodster_: thanks
20:56:38 <redrobot> dave-mccowan any last questions re: migrations?
20:57:15 <dave-mccowan> redrobot no, i'm good.
20:57:22 <redrobot> awesome
20:57:27 <redrobot> I think that's all we have time for today
20:57:31 <redrobot> thanks everyone for coming!
20:57:34 <redrobot> #endmeeting