20:00:09 <redrobot> #startmeeting barbican 20:00:09 <openstack> Meeting started Mon Sep 28 20:00:09 2015 UTC and is due to finish in 60 minutes. The chair is redrobot. Information about MeetBot at http://wiki.debian.org/MeetBot. 20:00:10 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 20:00:12 <openstack> The meeting name has been set to 'barbican' 20:00:15 <kfarr> o/ 20:00:24 <redrobot> #topic Roll Call 20:00:32 <silos> /o/ 20:00:36 <rellerreller> o/ 20:00:37 <dave-mccowan> \o/ 20:00:53 <hockeynut> (looking for chellygel-approved ascii sequence) 20:00:57 <hockeynut> (not finding one) 20:00:59 <hockeynut> o/ 20:01:09 <redrobot> hockeynut lol 20:01:18 <jkf> o/ 20:01:26 <spotz> o/ 20:01:27 <redrobot> as usual the meeting agenda can be found here: 20:01:32 <redrobot> #link https://wiki.openstack.org/wiki/Meetings/Barbican 20:01:45 <dave-mccowan> ¯\_(ツ)_/¯ 20:02:03 <hockeynut> oh dave-mccowan thats way too cool 20:02:10 <arunkant> 0/ 20:02:25 <woodster_> o/ 20:03:05 <elmiko> o/ (sorta) 20:04:58 <rm_work> o/ 20:05:53 <jhfeng> o/ 20:06:15 <redrobot> lots of barbicaneers here 20:06:19 <redrobot> let 20:06:23 <redrobot> 's get started 20:06:33 <redrobot> #topic Liberty RC1 20:06:45 <redrobot> in case you missed it, RC1 was released over the weekend 20:07:05 <redrobot> #link https://launchpad.net/barbican/liberty/liberty-rc1 20:07:38 <redrobot> Looks like we'll be needing an RC2 to address this bug 20:07:42 <redrobot> #link https://bugs.launchpad.net/barbican/+bug/1499876 20:07:42 <openstack> Launchpad bug 1499876 in Barbican "Attempt to delete project preferred CA should return 409" [Critical,Triaged] 20:07:44 <woodster_> Already planning rc2, or will that do it? :) 20:07:58 <redrobot> woodster_ read your mind ;0 20:08:19 <dave-mccowan> woodster_ is your db-manage patch a RC2 candidate? 20:08:43 <woodster_> dave-mccowan: do you mean that latest one? 20:08:51 <dave-mccowan> y 20:09:25 <woodster_> probably not worth it by itself...it affects CI/CD processes that utilize it though 20:09:39 <alee> o/ 20:12:08 <redrobot> woodster_ what's the bug #? 20:12:25 <woodster_> #link https://bugs.launchpad.net/barbican/+bug/1500448 20:12:25 <openstack> Launchpad bug 1500448 in Barbican "db_manage.py script doesn't return error code" [High,New] - Assigned to John Wood (john-wood-w) 20:13:42 <redrobot> woodster_ thanks 20:13:54 <redrobot> I can add that to RC2 since we'll be having to spin one anyway 20:14:04 <redrobot> any other questions/comments about RC1 or RC2 ? 20:14:36 <woodster_> redrobot: what else is going into rc2? 20:14:52 <redrobot> woodster_ so far only the two bugs we just talked about 20:15:07 <redrobot> woodster_ and possibly alee's support for sub-cas in DogTag 20:15:08 <woodster_> redrobot: oh got it, sorry 20:15:13 <alee> redrobot, and dogtag plugin suport 20:15:15 <alee> :) 20:15:44 <alee> redrobot, yes -- almost done with that. just getting the functional tests running 20:16:12 <redrobot> ok, moving on 20:16:56 <redrobot> #topic Testing Alembic migrations at the gate 20:17:02 <redrobot> woodster_ this is your topic 20:17:24 <woodster_> redrobot: thanks. This is regarding the use of the alembic migration script to populate teh database from scratch 20:17:46 <woodster_> ...vs just using SQLAlchemy to populate the database as is done now in the gate job 20:17:53 <silos> woodster_.: so would this use the db_manage script then? 20:18:00 <woodster_> silos: that's correct 20:18:06 <redrobot> silos so the issue is that alembic never gets tested at the gate 20:18:22 <redrobot> and this is the second cycle where we scramble during RC week to get migrations working 20:18:39 <redrobot> I think that adding a Grenade job would help this to some extent 20:19:09 <rellerreller> redrobot what else would be needed? 20:19:33 <redrobot> rellerreller I'm not 100% on Grenade jobs, but I think they only test N-1-> HEAD schema 20:19:49 <redrobot> rellerreller so they wouldn't cover the brand-new-db -> HEAD migration 20:20:09 <rellerreller> redrobot would SQLAlchemy cover the latter use case? 20:20:28 <redrobot> rellerreller no... the problem is that SQLAlchemy bypasses Alembic at the gate right now 20:21:01 <redrobot> it is possible now, to add alembic migrations that don't match up to the actual models in sqlalchemy 20:21:21 <rellerreller> so if I use SQLAlchemy to populate the db and Alembic, what is the delta for new-db use case? 20:21:27 <woodster_> For most ci/cd processes, the db-manage script would be used to keep an existing database up to date. It would be better for this to be run using a privileged user as well, separate from the user the barbican api uses. It would just make sense to use the db-manage script for the time-zero database case as well. 20:22:07 <redrobot> rellerreller alembic was flat out failing, but woodster_ submitted a few patches to get it up to date 20:22:12 <woodster_> rellerreller: the new db case starts with the juno script that populates the basic tables 20:23:28 <woodster_> the gate would at least test the viability of the scripts for MySQL. Eventually a gate for postgreSQL would be good to have. 20:23:45 <rellerreller> woodster_ why does the new db case start with the juno script? 20:24:01 <redrobot> rellerreller because juno is still supported by our team 20:24:01 <woodster_> rellerreller: that is the first script to run in the sequence 20:24:11 <rellerreller> I'm not opposed to any of this. I'm asking for clarification. I'm not certain of how all of this works, just vaguely 20:24:30 <rellerreller> I only know Hibernate :( 20:24:33 <redrobot> rellerreller once Liberty final is released, we will probably want to squash the migration such that Kilo is the new base 20:24:51 <hockeynut> redrobot could also look into javelin - its for general resource validation after install/upgrade, but not sure if it handles DB or not 20:25:05 <rellerreller> redrobot so the install for a fresh, new machine is run the kilo script to setup db and then migrate to liberty? 20:25:07 <dave-mccowan> it's now possible to make changes to the models without matching alembic migrations. is that a "feature" to speed development or is that a problem to be fixed? 20:25:54 <woodster_> as mentioned, an issue that could creep in is that the latest SQLAlchemy models and the migration scripts diverge. My thinking is that running the full suite of functional tests against the final database is a good check on that though. We might also add explicit schema comparison to the gate as well eventually. 20:26:02 <redrobot> dave-mccowan reviewers are expected to check the validity of alembic migrations when a model change is introduced. 20:26:24 <woodster_> yeah, unfortunately folks can hand code the alembic scripts 20:26:48 <woodster_> and ugly things like the dreaded 'op.execute()' call get pulled in 20:26:49 <rellerreller> woodster_ :( I doubt that is happening 20:27:07 <redrobot> woodster_ I think that devs are expected to edit the alembic files. 20:27:17 <woodster_> rellerreller: take a look at the CR I put in recently :) 20:27:35 <rellerreller> I am in favor of a migration gate check. I think any change that changes a model should fail unless model and alembic scripts are good. 20:27:43 <woodster_> redrobot: The auto generation process isn't perfect, that's true 20:27:56 <rellerreller> I do not like it when patches are submitted that change model and do not have migration. 20:28:19 <redrobot> So, the proposed change is to: 1. Stop generating the current schema by looking at the models. 2. Use alembic to generate the schema and then run the functional tests. 20:28:40 <woodster_> well, my basic proposal is to not auto popualte the db from barbican as done now in the gate, but to rather disable that and add a db-migration step 20:28:51 <woodster_> redrobot: ha, yep! 20:29:13 <redrobot> anyone opposed to that gate change? 20:29:28 <woodster_> eventually we'll add a grenade job as well 20:30:03 <redrobot> my only concern is that we need to be able to do the alembic migration in other gate jobs that are enabling Barbican in their gate 20:30:17 <rellerreller> Is there a way that Alembic could work but creating from SQLAlchemy could fail? 20:30:37 <rellerreller> My gut says no, but I wanted to float the question. 20:30:47 <woodster_> rellerreller: well, the app uses SQLAlchemy directly, so I wouldn't expect that is possible 20:30:53 <rellerreller> I figure the functional tests would catch any differences. 20:31:01 <redrobot> rellerreller possibly, but that should be caught by a failing functional test 20:31:06 <rellerreller> woodster_ we are on the same page :) 20:31:12 <woodster_> rellerreller: that is my thinking as well...ha, yes! 20:31:20 <rellerreller> haha 20:31:30 <dave-mccowan> do we want to add a liberty_release.py script to RC2? 20:31:35 <woodster_> redrobot: what other gate jobs? Such as the dogtag one? 20:31:35 <hockeynut> if there's a scenario that needs a test then by all means lets get them in 20:31:46 <redrobot> #agreed we're all in violent agreement. :) 20:32:03 <redrobot> dave-mccowan I don't think so 20:33:02 <woodster_> redrobot: we could just start with the basic devstack gate 20:33:04 <dave-mccowan> there's a kilo_release.py now. when would we add liberty_release.py? 20:33:46 * rellerreller packing up for the day and getting ready to leave 20:34:30 <woodster_> dave-mccowan: is this the all-tables-for-liberty module? If so then I think you start two releases back, so N would have the liberty one? 20:36:59 <dave-mccowan> juno_initial.py, kilo_release.py, <20 files for liberty>.py it would seem nice to me if we squashed the 20 files for liberty into one before the release. 20:38:02 <redrobot> I wonder if there is some sort of guideline for this? surely we're not the only ones using alembic 20:38:04 <dave-mccowan> (or maybe misunderstanding why we have *_release.py files) 20:38:20 <redrobot> dave-mccowan my impression is that xxxx_release.py is only created when that release becomes the new baseline 20:38:34 <redrobot> and liberty won't be a baseline until the N release 20:38:46 <woodster_> redrobot: +1, that's my impression as well 20:39:12 <dave-mccowan> why do we have kilo_release.py? 20:39:28 <redrobot> dave-mccowan not entirely sure... I think it was pushed last week 20:39:36 <woodster_> we can make a liberty_release.py after the final liberty release 20:40:11 <woodster_> I think each release can get its own version tagging module 20:40:13 <redrobot> woodster_ +1 that was my thinking also 20:40:30 * rellerreller walks out the door to exit the meeting 20:40:37 * redrobot waves at rellerreller 20:40:42 <woodster_> rellerreller: take care! 20:41:43 <woodster_> I think the difference is the very first release is the full db load...the other release pys are just tagging ones 20:43:10 <redrobot> woodster_ yeah... a no-op migration... 20:44:16 <woodster_> yep, so we'd need to convert the kilo no op to a real time-zero db load during the M release cycle, and drop the juno release 20:45:17 <dave-mccowan> there must be something that we should have done for liberty during the liberty cycle. 20:45:50 <redrobot> dave-mccowan kilo is actually in the middle of all the xxxxx.py migrations. the tree does not go from juno->kilo->xxxx, but rather juno->xxxxx->kilo->xxxxx 20:46:09 <redrobot> dave-mccowan I don't believe we need to push that before 20:46:26 <redrobot> dave-mccowan it's really just a placeholder, so that you know where kilo ended 20:46:42 <woodster_> yeah, I think by the time you release liberty, you should have a time zero load two releases before (so Juno). Seems we are on track with that 20:46:47 <redrobot> dave-mccowan or in liberty case, once liberty final is out, it will be a placeholder to let us know where liberty ended. 20:47:08 <redrobot> dave-mccowan it's only used after n+2 releases when we need to squash all the migrations 20:48:07 <woodster_> yep, so for the next release cycle, that kilo_release.py will be come the initial version file, and so be converted from a no-op to a time zero db load. 20:48:32 <redrobot> woodster_ +1 ... and all the migrations between juno and kilo get squashed into that new kilo_release.py 20:49:06 <redrobot> dave-mccowan we definitely need a liberty_release.py, but it does not actually need to be part of liberty 20:49:20 <redrobot> dave-mccowan just needs to be there before we push the first Mitaka migration 20:49:34 <woodster_> I believe we could have time zero loads for each release...Alembic supports multiple branches. Not sure how helpful that is though 20:49:38 <woodster_> redrobot: +1 20:52:53 <redrobot> only a few more minutes left 20:53:25 <jaosorior> Any workflows for this? https://review.openstack.org/#/c/198732/ would be nice to finally get it merged :D 20:53:51 <woodster_> ...and arunkant 's ACL CRs too 20:54:09 <hockeynut> I think we only require a single +2 now, so a second +2 should be workflow? 20:54:20 <arunkant> thanks _woodster, yes it has been pending for a while 20:54:35 <redrobot> hockeynut 2nd reviewer should give both a +2 and a +Workflow 20:54:40 <hockeynut> arunkant see my comments - some easy ones. 20:54:45 <hockeynut> thx redrobot - gotcha 20:54:52 <jaosorior> arunkant: I've +2ed your first two patches but I haven't gotten around reviewing the functional tests 20:54:52 <redrobot> hockeynut unless the 2nd reviewer doesn't want to workflow just yet, so they could give a +2 only. 20:55:00 <woodster_> jaosorior: I'll take a look...I had +2ed in the past 20:55:06 <hockeynut> makes sense redrobot 20:56:03 <arunkant> hockeynut : will look into your comments 20:56:12 <hockeynut> thx! 20:56:25 <jaosorior> woodster_: thanks 20:56:38 <redrobot> dave-mccowan any last questions re: migrations? 20:57:15 <dave-mccowan> redrobot no, i'm good. 20:57:22 <redrobot> awesome 20:57:27 <redrobot> I think that's all we have time for today 20:57:31 <redrobot> thanks everyone for coming! 20:57:34 <redrobot> #endmeeting