16:00:25 <krotscheck> #startmeeting StoryBoard
16:00:26 <openstack> Meeting started Mon Dec  1 16:00:25 2014 UTC and is due to finish in 60 minutes.  The chair is krotscheck. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:00:27 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:00:30 <openstack> The meeting name has been set to 'storyboard'
16:00:52 <krotscheck> Agenda! https://wiki.openstack.org/wiki/StoryBoard#Agenda
16:01:05 <krotscheck> #topic Actions from last week.
16:01:25 * krotscheck reorganized things. Roadmap and meeting agendas are updated.
16:02:08 <krotscheck> I also pinged persia and rainya about their own pieces. persia stated that he’d finally start having time again this week, rainya is in singapore on a school trip and will likely be able to come back on board after.
16:02:24 <fungi> awesome
16:02:25 <krotscheck> fungi: I totally forgot to ping you about storyboard-dev.openstack.org last week.
16:02:48 <fungi> krotscheck: 's okay. i totally didn't have time. maybe later this week
16:02:48 * krotscheck may have been suffering a turkey coma.
16:02:57 <krotscheck> Alright!
16:03:11 <krotscheck> #action krotscheck Pester fungi about storyboard-dev.openstack.org
16:03:18 <fungi> sounds great
16:03:23 <krotscheck> #topic User Feedback
16:03:28 <ttx> o/
16:03:29 <krotscheck> I’ve captured a few issues raised in channel.
16:03:54 <krotscheck> The first is related to mbitard’s inability to log in. At all.
16:04:26 <krotscheck> I’ve been poking at it, but haven’t been able to figure out exactly why he hasn’t been able to do so. Apparently he’s encountering a reproducible 500 error.
16:04:30 <fungi> did you get any clearer details on that? or retries?
16:04:37 <krotscheck> No retries, unfortunately
16:05:02 <krotscheck> Does anyone want to take this on and see if we can at least coordinate logs + retry?
16:05:08 <fungi> if it happens at a known time i can try to pare down the logs well enough to spot it, but whatever the issue it didn't mention that username
16:05:11 <krotscheck> And capture a good output?
16:05:14 <fungi> knowing the source ip address would also be great
16:05:42 * SergeyLukjanov lurking
16:05:52 <krotscheck> No volunteers? Hoookay.
16:05:55 <yolanda> i can do it
16:06:02 <krotscheck> yolanda: Thanks!
16:06:14 <yolanda> krotscheck, is there any story filed?
16:06:26 <krotscheck> yolanda: Not yet, since he couldn’t log in to file a story.
16:06:33 <yolanda> of course :)
16:06:40 <jeblair> krotscheck: was he in irc?
16:06:45 <krotscheck> #action krotscheck File a story for mbitard’s login issue.
16:06:49 <krotscheck> jeblair: He was. This was 2 weeks ago.
16:07:09 <krotscheck> jeblair: We tried to go off of his comment timestamp, that didn’t work :/
16:07:16 <yolanda> krotscheck, will be useful to have logs about it, do we have that available?
16:07:44 * SergeyLukjanov looking on server logs
16:08:25 <krotscheck> yolanda: We already had fungi go look back when it happened. He wasn’t able to whittle things down, so we’re hoping to capture mbitard and an infra-core online at the same time so we can capture an actual event.
16:08:36 <krotscheck> yolanda: That’d be the first step.
16:08:40 <fungi> yolanda: SergeyLukjanov: i already went through the logs. the trick is that we're logging all manner of errors but none which mention that account name and we have no feedback from the user as to when they got the error
16:09:20 <SergeyLukjanov> okay
16:09:21 <krotscheck> #action krotscheck File a story for storyboard spamming log errors.
16:09:22 <fungi> so step 1 is to find this user and get them to try again under more controlled circumstances (get approximate timestamps, find out the source ip address, et cetera)
16:09:40 <fungi> otherwise i don't think it will be able to move forward
16:10:27 <yolanda> do we have contact details for that user?
16:10:29 <krotscheck> Next issue: reed reported that the project group summary page does not actually filter stories by project group.
16:10:32 <SergeyLukjanov> fungi, where the log files located? I see the /var/log/storyboard empty
16:10:39 <fungi> SergeyLukjanov: apache logs
16:10:40 <krotscheck> yolanda: he’s mbitard on irc.
16:10:48 <SergeyLukjanov> fungi, thx
16:10:56 <fungi> SergeyLukjanov: they're the access and error logs for the storyboard vhost
16:11:15 <fungi> SergeyLukjanov: since it runs as a subprocess of the apache daemon
16:12:11 <krotscheck> Case and point: https://storyboard.openstack.org/api/v1/stories?project_group_id=57
16:12:19 <krotscheck> (57 is storyboard)
16:12:43 <krotscheck> So that’s something that should be fairly straightforward to build some test cases and fix.
16:13:07 <krotscheck> Anyone want to pick into the API and figure this out? Else I’ll do it.
16:13:42 <krotscheck> #action krotscheck File a story for project group ID not working on stories API, fix.
16:14:06 <krotscheck> Third issue is how Storyboard tends to log you out in a nonrecoverable manner. jedimike is on that.
16:14:24 <krotscheck> #action krotscheck make sure that there’s a story for the login issue and assign it to jedimike.
16:14:33 <yolanda> krotscheck, i created a story for it
16:14:39 <krotscheck> yolanda: Awesome.
16:14:47 <krotscheck> yolanda++
16:14:54 <krotscheck> #undo :)
16:14:55 <openstack> Removing item from minutes: <ircmeeting.items.Action object at 0x3dd1f10>
16:15:03 <krotscheck> Whoa, that works?
16:15:04 <krotscheck> Neat!
16:15:18 <krotscheck> #topic Urgent Items
16:15:26 <krotscheck> None on the agenda, does anyone have one to raise?
16:15:34 <rcarrillocruz> i have something, but not urgent
16:15:37 <rcarrillocruz> not sure if applicable now
16:15:39 <rcarrillocruz> https://storyboard.openstack.org/#!/story/153
16:15:54 <rcarrillocruz> i've looked a bit about it this weekend
16:16:03 <rcarrillocruz> how about a websocket streaming api
16:16:44 <rcarrillocruz> i think we could do this API around gevent-socketio, so consumers can subscribe to streams of events based on filters
16:16:45 <krotscheck> For reference: https://review.openstack.org/#/c/105252/
16:16:45 <NikitaKonovalov> rcarrillocruz: I thought krotscheck had a spec for that
16:16:45 <rcarrillocruz> etc
16:17:04 <rcarrillocruz> k thx
16:17:14 <krotscheck> And yes, it says WebSocket.
16:17:17 <krotscheck> :D
16:17:23 <rcarrillocruz> heh
16:17:26 <krotscheck> (great minds, think alike, rcarrillocruz )
16:17:31 <rcarrillocruz> i'm glad it wasn't a totally mad idea
16:17:49 <rcarrillocruz> krotscheck: have you started with it yet?
16:17:52 <krotscheck> I know that there are other opinions about using the jabber protocol, however I’ve only received those in person.
16:17:59 <krotscheck> rcarrillocruz: I have not. Go nuts!
16:18:05 <rcarrillocruz> i haven't fiddled with storyboard yet, but i'm looking to start with something
16:18:08 <krotscheck> i.e. go for it.
16:18:12 <rcarrillocruz> awesome
16:18:16 <krotscheck> rcarrillocruz: Fair warning, taht’s a large piece of code.
16:18:17 <rcarrillocruz> you can put me as an action
16:18:20 <jedimike> o/ sorry i'm late
16:18:25 <rcarrillocruz> yeah, i'll take an exploratory approach
16:18:25 <jeblair> krotscheck: should i add that spec to the priority spec list?
16:18:32 <NikitaKonovalov> As for an implementation of that I've not seen WebSocket frameworks used in OpenStack
16:18:45 <krotscheck> jeblair: I don’t think so, it’s not on our critical path.
16:19:05 <krotscheck> #action rcarrillocruz Start working on the storyboard streaming API.
16:19:11 <NikitaKonovalov> so we need to find a framework for that
16:19:26 <krotscheck> NikitaKonovalov: Yeah, i don’t think openstack does a lot of web things :D
16:19:34 <krotscheck> Any more urgent items?
16:19:37 <jeblair> krotscheck: okay, i want infra-core to review that one before we get too far
16:19:48 <jeblair> krotscheck: if someone is ready to start on impl, then i think we should
16:19:48 <krotscheck> jeblair: You got it.
16:20:17 <krotscheck> #topic Discussion
16:20:43 <krotscheck> #topic Discussion: Story Types
16:20:47 <krotscheck> ttx: Floor is yours.
16:20:58 <krotscheck> #link https://review.openstack.org/#/c/129267/
16:21:16 <ttx> So the numerous reviewers reported that the only thing that make story types tricky is trgheir interaction with branches
16:21:38 <ttx> namely, the fact that some story types would restrict the target branches for tasks
16:21:57 <ttx> that said, it's a bit difficult to discuss that until we actually have support for branches at all
16:22:18 <ttx> since it leads to red herrings like "we need to support projects that don't really have branches"
16:22:27 <ttx> which is a bit oethogonal to that discussion
16:22:32 <ttx> +r
16:22:43 <ttx> So... I'll draft the spec on branch support first :)
16:22:57 <ttx> and revisit story types once that is done
16:23:35 <krotscheck> Alrightey, shall I replace the discussion topic with branch support, or are you not yet ready to talk about that?
16:23:36 <NikitaKonovalov> ttx: do you want the branches to look like they do in LP now?
16:23:50 <ttx> NikitaKonovalov: well, minus the weird part
16:24:27 <ttx> (the weird part being... they have magic branch that is both current series and master
16:24:38 <ttx> which is VERY confusing
16:24:46 <ttx> )
16:24:50 <krotscheck> #topic Discussion: Branch Support
16:25:13 <ttx> so yeah, overall the idea is to allow to have tasks that are targeted toward the development branch (master)
16:25:22 <ttx> and tasks that are about backporting to older branches
16:25:47 <krotscheck> Ok, so where is a branch associated to a repository? Do projects need repo support?
16:26:03 <ttx> the notion of development branch / master could degrade gracefully to mean "the current thing"
16:26:18 <NikitaKonovalov> afaik, LP allows projects define their own branches and series, should we do the same?
16:26:27 <ttx> krotscheck: oh, you mean, where would SB learn about the branches there are ?
16:26:45 <krotscheck> ttx: Yep
16:26:48 <ttx> krotscheck: importing branches from git would be neat
16:26:56 <ttx> but LP forces you to define them
16:27:09 <ttx> so both approaches are possible I guess
16:27:21 <ttx> I'll make sure to cover that in spec
16:27:26 <krotscheck> ttx: Got it,
16:27:35 <jeblair> i think since we're expeciting storyboard's git support to be external...
16:28:01 <jeblair> ..sort of like how we work with lp now -- we have scripts that interact with lp based on gerrit actions..
16:28:07 <ttx> krotscheck: fwiw I still think having a project name and a git URL in a separtate field would make sense
16:28:26 <jeblair> ..we could have branches in storyboard automatically be created by those external tools
16:28:31 <ttx> so that we stop wasting space listing "openstack/" or "openstack-infra/" on every task
16:28:57 <jeblair> which still allows for the 'create arbitrary branch' use-case which may be useful for non-git projects
16:29:11 <ttx> jeblair: yes
16:29:18 <krotscheck> Yeah, this one’s going to get tricky :)
16:29:28 <ttx> we just need to make sure projects without branches don't look too funny
16:29:40 <ttx> I guess if there is a single branch, we could omit talking about "master"
16:29:54 <ttx> that would look just like it looks today
16:30:02 <krotscheck> So, first things first: A git repository can be associated with a project, yes?
16:30:23 <ttx> (just no branch column in case the project only has one branch (or has no git repo)
16:30:32 <jeblair> i think some non-git projects would actually want to use the term 'master' (after all -- many of them are still in service of things that have git branches)
16:31:05 <ttx> krotscheck: I'd say yes
16:31:14 <krotscheck> Can a project have more than one git repository assocaited with it?
16:31:20 <ttx> krotscheck: I'd say no
16:31:22 <NikitaKonovalov> krotscheck: I'd say we need a git project url field for a project first, and then think of a flow how how to load branches if there is one
16:31:35 <krotscheck> NikitaKonovalov: I agree.
16:31:41 <NikitaKonovalov> so probably listening on gerrit events
16:31:50 <krotscheck> NikitaKonovalov: We can either gerrit-event that or cron that.
16:31:56 <ttx> NikitaKonovalov: could be cron
16:32:09 * krotscheck is working on a new iteration of the cron plugin.
16:32:15 <NikitaKonovalov> that depend on how often do the branches appea
16:32:16 <NikitaKonovalov> r
16:32:23 <jeblair> just to be clear -- i do not think that listing to gerrit events and acting on them should not be a core part of storyboard
16:32:37 <krotscheck> jeblair: That’s a double negative ;)
16:32:38 <jeblair> storyboard should be useful without gerrit, or with other source control systems
16:32:41 <krotscheck> jeblair: I agree.
16:32:49 <NikitaKonovalov> It think that mostly happens when OS projects hit release time
16:32:50 <jeblair> sorry, heh
16:32:57 <jeblair> just to be clear -- i do not think that listing to gerrit events and acting on them should be a core part of storyboard
16:33:11 <krotscheck> jeblair: The question becomes whether it’ll be a storyboard plugin in a separate repo, or a gerrit thing that talks to storyboard via the SDK
16:33:14 <NikitaKonovalov> jeblair: it could be a plugin
16:33:24 <jeblair> so yeah, we should develop a full suite of tools to do this, but not depend on them architecturally
16:33:29 <jeblair> sounds like we're on the same page :)
16:33:35 <krotscheck> Yep.
16:33:38 <krotscheck> Approach may vary.
16:33:43 <jeblair> i just really didn't want to get that one wrong :)
16:35:04 <ttx> OK, should draft that branch thing this week
16:35:10 <krotscheck> Alright, so it sounds like the first step is to get some way of adding a git repo field to a project, and let ttx write a spec from there.
16:35:18 <krotscheck> That seems simple enough.
16:35:24 <krotscheck> Anything else before we move on?
16:35:50 <ttx> krotscheck: would that allow to make projects names be smaller
16:36:04 <krotscheck> ttx: ....peeerhaps?
16:36:19 <krotscheck> ttx: Those project names come from projects.config.yaml though
16:36:22 <ttx> since we don't allow duplicates across orgs, "openstack-infra/storyboard" could be "storyboard"
16:36:26 <krotscheck> So you’ll have to argue with jim about that :)
16:37:01 <ttx> I hate losing horizontal space. Pet peeve of mine
16:37:02 <krotscheck> From a technical perspective there’s nothing preventing the project names from being shorter.
16:37:14 <krotscheck> Ok, next topic
16:37:19 <ttx> and we'll need more horizontal space with branches :)
16:37:20 <krotscheck> #topic Discussion: API Paging
16:37:27 <krotscheck> jedimike: THis one’s yours!
16:37:34 * jedimike cowers
16:37:36 <jedimike> :)
16:38:14 <jedimike> ok, I guess we need to decide if we're going to go with some more expensive marker + results snapshot system, or just simple offset/limit?
16:38:50 <krotscheck> What’s your recommendation?
16:39:26 <jedimike> if it's essential that our users don't miss or duplicate any data when paging, the marker+snapshot
16:39:33 <jedimike> if it's not really an issue, offset+limit
16:40:19 <jedimike> I would say that, probably, storyboard isnt going to lose people money or bring systems down if they miss out on a few records as they page
16:40:50 <krotscheck> I agree - in most cases the things that will make people lsoe money should be at the top of the sort list anyway.
16:41:16 <jedimike> yeah. If this were Horizon, I'd be firmly saying we need the more expensive, absolutely reliable option. But I don't think we do.
16:42:11 <krotscheck> I’m somewhat curious about the performance implications on someone using an API client to go through All The Things (TM), but I feel that that particular use case can be fixed by removing the max results limit on the API.
16:42:25 <krotscheck> Assuming that the query prep on deep pages becomes an issue.
16:42:39 <jedimike> depends on the number of results we'd typically deal with
16:43:08 <krotscheck> jedimike: I’d say, conservatively, 100K or so?
16:43:09 <NikitaKonovalov> jedimike: search by one tag may return a really large set
16:43:22 <jeblair> i think the api case is key -- if you can't reliably interface with the system using the api (and "dropping results" is not reliably interfacing) then it's going to be of much lower value than we want
16:43:37 <jedimike> and how often would someone page to the thousandth page if we had a result set of 100k?
16:43:56 <krotscheck> jedimike: Well, we haz reports :)
16:44:01 <jedimike> hmmm ok...
16:44:25 <krotscheck> I get what you’re saying though.
16:44:59 <krotscheck> Personally, I can see cases in which a marker-based page will also result in skipped records.
16:45:05 <jedimike> the thing is, a non-snapshotted marker based paging system fails all of its goals when you can order by arbritary columns
16:45:05 <krotscheck> Especially in sorted situations.
16:45:36 <jedimike> if you snapshot, you can be golden, but it's much more expensive to do
16:45:46 <krotscheck> Yeah, that requires all kinds of caching.
16:46:05 <jedimike> yeah. can shard nicely though ;)
16:46:12 <krotscheck> Though the request becomes more of a POST -> Create this result set for me, and then a 302 redirect to where that result set lives.
16:46:28 <jedimike> yes, basically it does
16:46:31 <krotscheck> And the POST can include a “Keep it around for X days"
16:47:08 <krotscheck> Again, though, marker based paging gives us no idea of where within the result set a user is.
16:47:19 <jedimike> and the position can change at any time, with no notification
16:47:32 <krotscheck> So I kindof feel that the need to snapshot and the method by which we traverse a result set are two different discussions.
16:47:35 <jedimike> you could find yourself back on the first page after paging through 100 pages, with no idea it's happened
16:48:18 <jedimike> if we snapshot, we can use markers to navigate and I'd recommend we do because of the nice use of indexes it brings
16:48:33 <jedimike> and because we're snapshotting, underlying data changes won't affect where we are
16:48:58 <krotscheck> jedimike: How about the usability bit of “You’re on record X of YYYY”?
16:49:02 <jedimike> but snapshotting 100k results for a search, or every time a search *changes* is not in any sense cheap
16:49:25 <jedimike> krotscheck, that's easily done by caching counts, seeing how many records are after the marker and how many ther are in total
16:49:44 <krotscheck> jedimike: That requires though that you start at the beginning in the client.
16:50:04 <krotscheck> jedimike: If you don’t, you’ve got no idea.
16:50:06 <jedimike> krotscheck, not if you calculate the page markers and cache them
16:50:15 <jedimike> then you can jump to pages
16:50:23 <krotscheck> jedimike: Right, or if the cached record set gives you an order index as well.
16:50:36 <krotscheck> (Because users can be annoying and change their page sizes.
16:50:49 <jedimike> and if you know the marker you can get back where marker > than this.marker to see how many follow, and you know the total
16:50:59 <jedimike> yes
16:51:11 <krotscheck> Yeah, sounds like there’s a solution there.
16:51:35 <jedimike> in the case of an ordered result set, the marker becomes a combination of pk and the value in the ordered column
16:51:44 <krotscheck> That would also greatly improve API performance, because we can include expiration headers on search results and use the browser cache to reduce API calls.
16:52:25 <jedimike> it just means a lot more inserts/deletes, and we need to figure out, based on our  (predicted) usage patterns, which is going to serve us better
16:52:44 <krotscheck> jedimike: You mean prewarmed result caches?
16:53:21 <jedimike> that's interesting, but cache invalidation becomes a problem that needs solving there
16:53:58 <jedimike> i mean, which would be lesser - the overhead of snapshotting, or the overhead of using offsets, with the usage pattern we think we'll see when this gets big?
16:54:01 <krotscheck> jedimike: Maybe. Just take a snapshot of the prewarmed results at that moment?
16:54:36 <krotscheck> jedimike: Don’t forget the overhead of breaking things downstream if we change our paging parameters.
16:54:39 <krotscheck> Twice.
16:55:09 <jedimike> krotscheck, in the past, i've only snapshotted the pk and the value ordered by so that i can join on the main data table and highlight any changes to the ordered value in the ui without having the change affect the order
16:55:39 <krotscheck> jedimike: That’s sane.
16:56:06 <krotscheck> It feels like we’re coming to a consensus on using marker paging with snapshots.
16:56:19 <jedimike> yeah, so your table is narrow and if you left join, you even cope with deletes quite easily, without affecting any navigation
16:56:24 <krotscheck> And that things like what-is-shapshotted and so forth should just be detailed out in a spec.
16:56:28 <krotscheck> (Also we’re running out of time)
16:56:58 <jedimike> yeah, and how the snapshots are sharded, the housekeeping around them, etc.
16:57:12 <krotscheck> jedimike: Yeah, that’s implementation details :)
16:57:16 <jedimike> indeed :)
16:57:22 <krotscheck> jedimike: For the sake of sanity, can you put that into a spec for us?
16:57:47 <jedimike> yes, I can write up the approach i outlined on the mailing list into a generic paging spec
16:58:01 <krotscheck> jedimike: Works for me.
16:58:05 <krotscheck> #action jedimike Write paging spec.
16:58:08 <jedimike> cool :)
16:58:12 <NikitaKonovalov> jedimike: works for me as well
16:58:36 <krotscheck> Since we only have a little time yet, I’m going to bounce out directly to open discussion and we’ll visit the other discussion topics next week or in channel.
16:58:43 <krotscheck> #topic Open Discussion
16:59:05 <krotscheck> Anything?
16:59:08 * krotscheck has nothing.
16:59:09 <NikitaKonovalov> I've finally got a client created
16:59:17 <NikitaKonovalov> So there are a few changes up
16:59:27 <krotscheck> WOO
16:59:29 <rcarrillocruz> NikitaKonovalov: nice
16:59:37 <rcarrillocruz> where ? :-)
16:59:38 <CTtpollard> cool NikitaKonovalov
16:59:57 <NikitaKonovalov> if the general approach is OK, I'll start covering it with tests docs and enpoint support
17:00:12 <NikitaKonovalov> the repo is openstack-infra/python-storyboardclient
17:00:28 <krotscheck> Cool
17:00:43 <rcarrillocruz> awesome
17:00:49 <CTtpollard> :)
17:01:04 <krotscheck> Alright, thanks everyone! We’re out of time.
17:01:07 <krotscheck> #endmeeting