16:04:47 <krotscheck> #startmeeting storyboard
16:04:48 <openstack> Meeting started Thu Mar 20 16:04:47 2014 UTC and is due to finish in 60 minutes.  The chair is krotscheck. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:04:49 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:04:52 <krotscheck> Roll call, who's here?
16:04:52 <openstack> The meeting name has been set to 'storyboard'
16:05:10 <NikitaKonovalov> I'm here
16:05:15 <ttx> o/
16:05:23 <ttx> (for the first 15min or so)
16:05:35 <krotscheck> Ok, let's get over the big stuff quick
16:05:43 <krotscheck> #topic MVP
16:06:03 <krotscheck> Current status is "everything's working except for comments"
16:06:25 <krotscheck> jeblair wants comments before infra copts it. We've been dogfooding it ourselves to some extent.
16:06:27 <NikitaKonovalov> The controller is no review
16:06:38 <krotscheck> NikitaKonovalov: And it looks fantastci
16:06:44 <NikitaKonovalov> and thaks to ruhe, it's tested
16:06:49 <krotscheck> The UI is causing me a bit more issues, because I'm still trying to get my paging patch working
16:07:15 <krotscheck> So I haven't gotten to work on that, current ETA this afternoon.
16:07:16 <ttx> krotscheck: I think the MVP needs task state change as well ?
16:07:40 <krotscheck> ttx: That doesn't seem to be explicitly listed: https://etherpad.openstack.org/p/StoryboardMeetup
16:08:09 <ttx> krotscheck: probably not
16:08:20 <krotscheck> ttx: I stand corrected, there's a mention of setting a task as "invalid"
16:08:21 <ttx> krotscheck: I just don't see how comments can be more useful than task changes
16:08:23 <krotscheck> It's implicit
16:08:38 <ttx> since the idea is to track completion of tasks overall :)
16:08:40 <krotscheck> Status changes aren't that hard, thankfully
16:08:46 <ttx> yeah, i figured
16:08:53 <krotscheck> Ok, I'll add that to the agenda
16:08:57 <jeblair> o/
16:09:05 <ttx> krotscheck: could be considered MVP+1
16:09:43 <krotscheck> Ok, so outstanding for jeblair is comments, outstanding for ttx are task statuses.
16:10:09 <ttx> krotscheck: if infra thinks they can use it without task changes, i'm fine with leaving them out :)
16:10:15 <krotscheck> jeblair: ?
16:10:22 <ttx> in all cases it's probably the next step
16:10:29 <jeblair> oh i think they're pretty important :)
16:10:35 <ttx> krotscheck: about comments... the POC tracked the history of changes as comments. Do you think we should too ?
16:11:02 <ttx> i.e. it would interleave "system remarks" with "human comments"
16:11:14 <ttx> the same way Lp does
16:11:30 <krotscheck> ttx: Let's postpone that discussion until we get to comments.
16:11:32 <jeblair> ttx: that's a good idea, but something i was thinking about recently is that we might want to record the history of editing the story description...
16:11:35 <krotscheck> (That's on the agenda)
16:11:36 <ttx> I think it makes it easier to follow the flow
16:11:45 * jeblair waits for appropriate time in agenda
16:11:52 * ttx shuts up
16:11:58 <krotscheck> #topic Auth Research
16:12:02 <jeblair> (where's the agenda?)
16:12:13 <krotscheck> #link https://etherpad.openstack.org/p/StoryboardPerms
16:12:20 <krotscheck> jeblair: #link https://wiki.openstack.org/wiki/StoryBoard#Meeting
16:12:47 <krotscheck> NikitaKonovalov: were you able to do some research into ACL systems that might help us moving forward?
16:13:47 <NikitaKonovalov> well first we still need to move tokens to database
16:14:02 <krotscheck> (For those of you new to this discussion, I threw some ideas into an ether pad. They are very draft-ish)
16:14:03 <NikitaKonovalov> or we will be not able to scale API controllers
16:14:23 <NikitaKonovalov> But to make things work faster we may use memcached
16:14:51 <NikitaKonovalov> keystone has this functionality implemented
16:15:25 <krotscheck> NikitaKonovalov: I don't think memcached is an option - one of our requirements is that API keys can be generated and persisted for Infra daemons. If we keep those tokens in memcache we risk losing them.
16:16:08 <krotscheck> (unless there's a memcache persistence layer that I'm not familiar with)
16:16:15 <NikitaKonovalov> I mean, keep them in database an save to memcache for performance
16:16:54 <NikitaKonovalov> and expiration policy for memcache, so it does not get full of usless tokens
16:17:38 <krotscheck> I'm not certain memcache is the right answer here.
16:17:53 <krotscheck> Don't get me wrong - having the performance would be fantastic.
16:18:09 <krotscheck> And using the expiration policy as our  token lifetime would be pretty simple.
16:18:24 <NikitaKonovalov> I was mostly looking in keystone, so that's the way they do it
16:18:30 <krotscheck> Hrm
16:18:43 <NikitaKonovalov> and actually they support some other key-value storages
16:18:47 <krotscheck> Ok, so in-use tokens would get cached into memcache.
16:18:54 <NikitaKonovalov> right
16:19:10 <krotscheck> I like that better.
16:19:32 <krotscheck> How easy would it be to adapt keystone's code and make it more pecanish?
16:20:23 <krotscheck> That's probably a broader question.
16:20:32 <ttx> krotscheck: do we need to optimize token storage right now ?
16:20:38 <krotscheck> ttx: I was about to ask that
16:20:53 <krotscheck> Since we need to put it in the database anyway, let's start there.
16:20:58 <ttx> we expect like 10 users max at this point :)
16:21:08 <NikitaKonovalov> there is an WSGI middleware in keystone that does everything realted to tokens, but I'm not sure we can take a part from it
16:21:10 <ttx> sounds like something that's easy to abstract after the fact
16:21:19 <krotscheck> We should be able to layer on memcache later.
16:21:20 <NikitaKonovalov> ttx: agree
16:21:33 <krotscheck> krotscheck: And, to be honest, we might want to use memcache not only for our tokens.
16:21:38 <NikitaKonovalov> let's start with databsae only
16:21:47 <krotscheck> Any disagreements?
16:21:54 <krotscheck> (do we need a vote?_
16:22:10 <ttx> nope go  for it
16:22:26 <jeblair> ++later
16:22:34 <jeblair> (but also ++memcache)
16:22:40 <krotscheck> #agreed First step on Storyboard ACL's is to get the tokens into the db, further design discussion postponed until then.
16:22:57 <krotscheck> #idea Layer on memcache to optimize token queries.
16:23:14 <krotscheck> #topic Database migration pain
16:23:31 <krotscheck> ruhe did a great job removing the sqlite pain over the weekend.
16:23:38 <krotscheck> ttx: You mentioned there was another potential issue?
16:23:56 <ttx> krotscheck: I worked around it. Foreign keys in the initial DB definition make it hard to remove them
16:24:03 <krotscheck> ttx: Got it
16:24:16 <ttx> but looks like I hacked around it: https://review.openstack.org/#/c/81562/
16:24:35 <ttx> basically we didn'(t name the constraints, and you need to have their name to remove them afterwards
16:25:02 <krotscheck> FYI for everyone: Unit tests are now run off of sqllite using the sqla manifest generated from our models. Our migration tests then run a table comparison to see whether the DB generated by that manifest matches the DB generated by our migrations.
16:25:02 <ttx> the hack is to retroactively name them after what they end up being named in mySQL DB
16:25:26 <NikitaKonovalov> ttx: will the migrations work after you have changed the initial one?
16:25:31 <ttx> the leftover quetsion is... should we just name them all NOW (or remove them all NOW) to avoid future pain
16:25:41 <ttx> NikitaKonovalov: sure
16:25:50 <jeblair> i'm in favor of removing them all now...
16:25:51 <NikitaKonovalov> that's great
16:25:55 <ttx> NikitaKonovalov: it will only break people that have postgresql deployed and do CD
16:26:07 <jeblair> krotscheck: oh, mordred and i had a convo in #storybeard earlier and i don't think you were online
16:26:09 <jeblair> let me paste it
16:26:14 <ttx> because our overwrite of the initial migration doesn't correspond to their current state
16:26:18 <krotscheck> Heh. Storybeard.
16:26:26 <ttx> Like it
16:26:37 <ttx> I'm also in favor of removing them all now
16:26:46 <jeblair> #link http://paste.openstack.org/show/73920/
16:26:55 <ttx> I know cody-somerville expressed the other preference
16:27:07 <krotscheck> jeblair: Yeah, my znc bouncer is fubar right now.
16:27:08 <jeblair> krotscheck: storybeard is extra hipster
16:27:15 <krotscheck> .....
16:27:17 <NikitaKonovalov> lgtm, because thous are not the migrations actually, they are kind of "ajusting the model"
16:27:22 <krotscheck> I think we have an april fool's joke.
16:27:29 <jeblair> ++
16:27:52 <ttx> That said, if we remove them now it will look a bit funni in the migrations
16:27:57 <ttx> or funny
16:28:08 <jeblair> so the tldr is that since alembic creates the schema, we can simply omit fk definitions from alembic, but still have the sqla orm understand the relationships
16:28:10 <ttx> nothing a migration consolidation can't fix
16:28:29 <ttx> jeblair: yes
16:28:30 <cody-somerville> please don't get rid of fk constraints.
16:28:32 <jeblair> (as a practice for eliminating fks in the db going forward)
16:28:46 <krotscheck> cody-somerville has the floor!
16:29:26 * ttx is happy to defers to the MySQL overlords
16:30:35 <ttx> if mordred says they are useless and potentially a barrier to scale, he knows a lot more tahn I do on that toic
16:30:40 <ttx> +p
16:31:04 <krotscheck> I'm in the same camp. I've relied on FK's a lot in the past, but mordred's argument makes sense.
16:31:13 <cody-somerville> we will never be at that scale.
16:31:26 <cody-somerville> plus, bugs!
16:31:33 <cody-somerville> too high of a risk that we mess up and corrupt our data
16:31:41 <cody-somerville> our data deserves the extra protection fk provides
16:31:57 <jeblair> i don't think it provides any extra protection
16:31:57 <cody-somerville> it will not affect our ability to scale
16:32:05 <jeblair> the application needs to understand and enforce these constraints anyway, so it's wasteful to have the database do it
16:32:12 <cody-somerville> until have like terabytes of data
16:32:16 <ttx> cody-somerville: i'm fine either way. Just want to pick a way and stick to it, rather than do it 10 migrations down the road
16:32:23 <jeblair> and they cause all sorts of problems with schema changes (including but not limited to what we've just seen)
16:32:34 <cody-somerville> jeblair: data lives a long time, code not so much.
16:33:24 <ttx> I think we can have that discussion off-meeting
16:33:32 <cody-somerville> and the issue we're seeing is due to alembic being a young project. It's just a missing feature or it's doing something wrong.
16:33:43 <krotscheck> Seems like a longer argument. How about we take this to openstack-dev?
16:33:44 <ttx> just need to make sure cody and monty are in the room at the same time rather than talk past each other
16:33:56 <jeblair> ttx: ++
16:34:13 <ttx> otherwise i just agree with the last one who spoke
16:34:17 <krotscheck> Seems like a big enough 'best practice' argument that the rest of the community would care.
16:34:45 <ttx> krotscheck: my personal feeling would be: FKs just created pain for me, let's get rid of them
16:35:58 * jeblair agrees with the last thing ttx said
16:36:06 <krotscheck> ttx: Well, that's SQLA's abstraction that's causing the pain. If you were working natively that wouldn't be that much of a problem, but I see your point.
16:36:18 <krotscheck> natively on one database that is.
16:36:22 <cody-somerville> This is sort of a similar argument to type systems.
16:36:30 <jeblair> they will bite us again on some migration that's impossible to do because of a fk constraint
16:36:35 <cody-somerville> For what it's worth, the south migration engine handles this just fine.
16:37:09 <krotscheck> Alright, let's take this offline. Who wants to start the dev-list thread?
16:37:09 <jeblair> or even some code error that the fk system _didn't_ catch but is impossible to clean up due to it.  etc.
16:37:15 <cody-somerville> but yes, fk constraints beyond that can cause issues and it's usually because we want it to. (oh hey, you're doing something that fundamental breaks your data model integrity).
16:37:20 <jeblair> krotscheck: my recommendation would not be to look to openstack-dev for a decision.  you will get opinions, but we already have those.  :)
16:37:24 <ttx> cody-somerville: I think jeblair's argument is that migartion engines work better with simple DB definitions, and ORMs preserve integrity from the app perspective
16:37:40 <jeblair> ttx: wow why can't i summarize my thinking like that.
16:37:42 <cody-somerville> What happens when someone decided to write some raw SWL?
16:37:51 <cody-somerville> *decides
16:37:56 <krotscheck> jeblair: It's a distraction tactic. I actually think the current status is painful but understood, and therefore manageable, therefore we can focus on other features :)
16:38:09 <cody-somerville> What happens when we do major refactoring? and maybe we mess up.
16:38:17 <ttx> jeblair: we form a synergetic double-human
16:38:41 <krotscheck> cody-somerville: We write more tests :)
16:38:45 <cody-somerville> the data is just too valuable to risk
16:38:46 <jeblair> cody-somerville: so we need good real tests of schemas and migrations (on mysql and not sqlite)
16:38:47 <jeblair> cody-somerville: ++
16:39:02 <jeblair> cody-somerville: and if that fails and we hose our production instance, we restore from backup.  :)
16:39:25 <krotscheck> #topic Comments
16:39:25 <cody-somerville> people suck at writing tests. writing good tests is hard.
16:39:31 <cody-somerville> This is like an argument on type safety
16:39:35 * krotscheck is forcing this conversation offline
16:39:36 <jeblair> cody-somerville: (which is the same thing we would do with any kind of delete lots of data error, which fks not only don't protect against but also can facilitate)
16:39:53 <ttx> we shouldn't spend the whole meeting on that. Better to expose each arguments in a well-thought ML post and see if we can reach consensus
16:40:05 <ttx> krotscheck: +1
16:40:14 <ttx> I'd go post on -infra
16:40:49 <NikitaKonovalov> agree, let's discuss that in mailing list
16:40:52 <krotscheck> Ok: jeblair wants comments! We all want comments! We have simple non-threaded comments on a patch right now, but I haven't been able to look at/polish the UI yet.
16:41:01 <jeblair> yay!
16:41:25 <krotscheck> Current start ETA is this afternoon, assuming I don't get into another argument with jeblair on paging :)
16:41:48 <krotscheck> jeblair and tax mentioned the idea that each task should have a log of changes.
16:41:53 <krotscheck> *ttx
16:42:13 <jeblair> right, ttx was suggesting incorporating that as comments
16:42:26 <krotscheck> My perspective is that a history log is valuable, however it's a slightly different problem than comments, and should be treated differently.
16:42:38 <NikitaKonovalov> we can handle that with comments, by just setting a type
16:42:40 <jeblair> i was throwing out the idea that things like 'editing the description' may want to be logged too, but might be too verbose for comments, so we might want a side-channel log of things like that
16:43:04 <jeblair> i don't know if that means that we should put all actions in the side channel log, or some, or what...
16:43:09 <ttx> krotscheck: so there are two types of "history commenst"
16:43:11 <NikitaKonovalov> what about a checkbox for show/hide service commnets
16:43:40 <jeblair> ttx: but i do agree with the idea that peoples actions on stories should be visible, and makes sense to put them in the chronology of comments
16:43:54 <NikitaKonovalov> the UI can filter them before rendering
16:43:58 <ttx> krotscheck: the first type is meaningful for the discussion. For example, an autocomment saying you added a task affecting Nova spares you from having to post a "Oh BTW this also affects Nova apparently" comment
16:44:00 <krotscheck> ttx: Well, there's an event log for what has happened to the story, and there's a comment sections. The event log would include something like "so and so left a comment", and the UI could resolve that comment, but it would also include state changes, etc.
16:44:14 <ttx> the second type is more an activity log
16:44:57 <ttx> krotscheck: if you look at LP, whenever you change status you can easily add a comment about it to justify iy
16:44:59 <ttx> it*
16:45:27 <ttx> so for example, if you change task status to "merged" you can add a link to the review that was merged
16:45:48 <ttx> I hate it when people change something and don't explain why
16:45:50 <jeblair> (mildly distracting but i want to mention it anyway: launchpad's rolling up of several actions into one is really nice)
16:45:58 <ttx> So I don't want us to discourage that
16:46:08 <ttx> (jeblair: yes)
16:46:11 <krotscheck> ttx: I don't want to overload comments with additional functionality that isn't really part of the design discussion.
16:46:20 <krotscheck> s/design/story/
16:46:36 <ttx> krotscheck: well, the story discussion is all about steps to resolve bugs
16:47:03 <ttx> so things like "this also affects nova" are pretty critical in the discussion
16:47:04 <krotscheck> Ok, so we're conflating "features" and "design" here.
16:47:22 <krotscheck> What jeblair is asking for is the ability to collect several actions into one, which I think is good.
16:47:38 <krotscheck> What ttx is asking for is the ability to see a full log of a story/tasks history
16:47:56 <ttx> krotscheck: no
16:48:01 <krotscheck> I'm sure we have a bunch of other ways that people want to see the lifecycle of a task.
16:48:12 <ttx> krotscheck: i'm saying that key status changes are part of the discussion
16:48:14 <jeblair> let's not focus on the thing i mentioned; it's distracting and can be addressed later
16:48:28 <jeblair> ttx: ++
16:48:42 <ttx> krotscheck: and that's based on my experience with openstack and LP
16:48:50 <ttx> not just somethign I deeply believe in
16:49:33 <krotscheck> ttx: What I'm saying is that what you're asking for is a feature, and can easily be represented in the UI that way, however how that actually ends up being implemented technically is irrellevant.
16:49:47 <jeblair> here's a nice bug: https://bugs.launchpad.net/openstack-ci/+bug/1266513
16:49:49 <uvirtbot> Launchpad bug 1266513 in openstack-ci "Some Python requirements are not hosted on PyPI" [Critical,In progress]
16:49:51 <jeblair> with lots of status changes
16:50:01 <jeblair> along with associated comments
16:50:03 <krotscheck> ttx: So when we talk about "comments", to me that says : The area in the UI that you're interacting with that contains comments"
16:50:19 <ttx> krotscheck: I see your point
16:50:21 <krotscheck> ttx: When I say comments, I mean the comments API that handles someone actually talking.
16:50:54 <ttx> krotscheck: I'm just saying whatever we end up doing should support displaying more than just "comments" in the "timeline"
16:51:19 <jeblair> i think there may also be the implication that the "update task state" api may need a comment parameter as well
16:51:25 <krotscheck> ttx: So my suggestion is that when someone leaves a comment, or changes a status, they actually create an event associated with that story, which includes some reference to the comment and/or status change.
16:51:31 <ttx> the simplest way to do that is to have a single table with comment types, like in the POC
16:51:48 <ttx> and have status chnages be fed into that table
16:51:52 <ttx> but that's implementation.
16:51:55 <krotscheck> And then the UI actually renders a filterable list of events, which themselves resolve the related resources for display
16:52:04 <krotscheck> Right.
16:52:20 <krotscheck> So right now, is anything other than a single-threaded comment list asked for?
16:52:27 <ttx> yes, I think using "timeline" (and "events") clarifies what we mean
16:52:50 <ttx> i was just not using "comments" for the thing you meant
16:52:54 <krotscheck> ttx: Using the timeline metaphor can easily be extended to projects, features, stories, releases....
16:52:57 <jeblair> krotscheck: that sounds reasonable (especially the 1:many event:action-or-comment aspect)
16:53:12 <ttx> if comments are just a part of a timeline that will contain other types of important events, ++
16:53:33 <krotscheck> kk
16:53:51 <krotscheck> Further design discussion pending, I think what NikitaKonovalov gave us is sufficient for MVP, but it's good to know this is on the horizon
16:53:57 <ttx> 7min left
16:53:59 <krotscheck> #topic Paging
16:54:32 <krotscheck> I'm altering my paging patch to use oslo's marker/limit rather than offset/limit, and to make the max/default page sizes configurable via storyboard.conf.
16:54:48 <krotscheck> Hopefully the patch will be updated today.
16:54:58 <krotscheck> #topic Task status
16:55:28 <krotscheck> ttx: Any ask for anything more than a status FK-ish link to a table of valid task statuses?
16:55:44 <ttx> krotscheck: nope
16:55:52 <krotscheck> NikitaKonovalov: Think you can add that?
16:56:16 <ttx> my point was just: until we add that, we can list work todo but can't track completion of anything
16:56:30 <ttx> so it's like the next thing after MVP imho
16:56:55 <krotscheck> ok.
16:56:56 <NikitaKonovalov> we can have more complicated queryis to fetch that
16:56:59 <ttx> (personally I find it more critical than comments, but that's the result-oriented me)
16:57:00 <jeblair> ttx: we can't mark the mvp as complete until we do, so it's self-necessitating! :)
16:57:39 <ttx> jeblair: nice
16:57:40 <krotscheck> We're running out of time, and I don't want to start another design discussion.
16:58:07 <jeblair> i think we all agree they are important
16:58:09 <krotscheck> Let's move this to offline, NikitaKonovalov, I'm just going to assume you make your magic happen and the feature will appear.
16:58:10 <ttx> krotscheck: should be easy since at this point we have only 3 states and don't plan to get smarter than that
16:58:35 <ttx> FWIW the end goal is to have that set automatically on merge for git-backed projects
16:58:40 <krotscheck> ttx: Yeah, a simple implementation could just be an enum.
16:58:48 <ttx> but in all cases we need to be able to manually set it too
16:58:52 <NikitaKonovalov> story/<id>/stats may render everything about how many thing are in which state
16:59:15 <krotscheck> NikitaKonovalov: That feels more like a /story?state=foo to me.
16:59:21 <krotscheck> But we can argue about that.
16:59:23 <krotscheck> later
16:59:25 <krotscheck> #topic Requested New Feature
16:59:30 <ttx> in other news, I plan to spend more time on Storyboard in the coming weeks. It's a nice distraction from boring release matters
16:59:43 <krotscheck> Anything burning a hole in someone's brain that we need to put ahead of comments and task status?
16:59:45 <ttx> and I want to stay up to snuff
17:00:05 <ttx> krotscheck: nope
17:00:09 <krotscheck> Awesome.
17:00:12 <krotscheck> #endmeeting storyboard