15:09:39 <jeblair> #startmeeting storyboard 15:09:40 <openstack> Meeting started Mon Mar 31 15:09:39 2014 UTC and is due to finish in 60 minutes. The chair is jeblair. Information about MeetBot at http://wiki.debian.org/MeetBot. 15:09:41 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 15:09:43 <openstack> The meeting name has been set to 'storyboard' 15:10:06 <jeblair> it looks like we have enough folks to have a meeting 15:10:26 <ttx> yep 15:10:47 <jeblair> though we do not have the person needed for agenda item 1 15:10:51 <jeblair> #topic agenda 15:10:58 <jeblair> #link https://wiki.openstack.org/wiki/StoryBoard 15:11:22 <jeblair> the wiki page with the agenda has just been updated; any other last minute items to add? 15:12:00 <NikitaKonovalov> nothing from me 15:12:46 <jeblair> then i'll reorder the topics in case krotscheck shows up later 15:12:52 <jeblair> #topic Easier access to logs (cody-somerville) 15:12:58 <jeblair> cody-somerville: what's up? 15:13:34 <cody-somerville> Hi. I was wondering if it would be possible to somehow get easier access to logs for storyboard.o.o? I understand it currently requires a member of openstack-infra core with ssh access to retrieve. 15:14:57 <NikitaKonovalov> those logs wold help to solve a 'random 500' issue 15:15:30 <NikitaKonovalov> this one https://storyboard.openstack.org/#!/story/40 15:15:44 <jeblair> cody-somerville: in principle we try to make as much information about infra systems operation as possible public 15:16:29 <jeblair> cody-somerville: to that end, all our monitoring is unrestricted, puppet logs are public, and in some cases (nodepool image builds) entire logs are public 15:17:14 <jeblair> cody-somerville: i think there would be two things to consider in making storyboard logs public -- a) the security of account credentials, and b) user privacy 15:18:04 <jeblair> a) i'm not sure if we can guarantee than storyboard is never going to dump the password into the logs (courtesy of, say sqlalchemy?) 15:18:18 <NikitaKonovalov> jeblair: at this point we do not store any private information in our database 15:18:34 <NikitaKonovalov> the password goes to Launchpad only 15:19:02 <jeblair> and b) some people might consider their ip address private information and want to avoid being associated with it publicly 15:19:35 <jeblair> i don't know if that's a widely held belief in this community, but i do know some communities/organizations feel that way 15:20:34 <jeblair> oh, tokens show up in the logs 15:21:38 <NikitaKonovalov> jeblair: agree, tokens should stay private 15:21:46 <jeblair> so storyboard auth tokens themselves might be accessible in the logs... it's possibly that it's only intermediary tokens; i'm not certain 15:22:56 <jeblair> so on the other side of this, we have the thought that it should be easy for devs to run a test instance, and so most problems should be able to be debugged locally 15:23:11 <NikitaKonovalov> authorization codes are single-use, so nothing bad will happen if those leak 15:23:27 <cody-somerville> I would suggest that maybe we can more lenient with log access until MVP 15:23:27 <jeblair> that leaves only bugs that are unique to our production instance as the ones needing support from logs 15:24:21 <cody-somerville> but would be happy with restricted mechanism (maybe shared username and password coupled with scouts honour). 15:25:07 <jeblair> one class of those bugs are the ones where we screw up the database because we're not actually testing it. i firmly believe we need to fix that with testing. 15:25:42 <jeblair> that leaves things like this random 500 error, which, one assumes is not a pervasive db problem (we don't know yet because no one has looked) 15:26:38 <jeblair> there are four people who can check the production server logs, and are happy to do it in these cases 15:26:57 <persia> Note that the recommended dev setup may be fairly different than the current production setup, so there may be a set of associated issues (e.g. mod_wsgi vs. other means of running it) 15:29:08 <jeblair> persia: good point; i think they should be as close as possible. mod_wsgi may be a big thing to ask of a developer. i don't think we should require it, but i do think it's reasonable for a serious developer to use it on occasion (i've certainly done that for similar projects in the past). 15:29:10 <cody-somerville> and then after MVP, we can go back to requiring member of infra core with ssh access 15:29:59 <persia> jeblair: Just updating the storyboard docs to explain *how* would be a big step. The current prod setup says "pip install storyboard ... mod_wsgi is recommended". 15:30:26 <jeblair> cody-somerville: so far, i've only been asked to look at logs once, and i was happy to do so and provided the information within about a minute; to my knowledge, no one has asked for us to look at the most recent 500 errors 15:31:07 <jeblair> cody-somerville: so for the moment, i'd like to stick with "if you need a log entry, ask in #openstack-infra" 15:31:17 <ttx> and let's see if that doesn't scale 15:31:37 <jeblair> ttx: right 15:31:50 <ruhe> persia: i could take AI to update docs about prod deployment 15:31:55 <persia> Are there timezone concerns here? Is there good infra coverage for folk in +2 or +3? 15:31:58 <SergeyLukjanov> jeblair, ++ 15:32:23 <persia> ruhe: If you did, I'd be very pleased to do the review, including validating the docs by performing such a setup and running the test suite against it. 15:32:23 <ttx> persia: Sergey is at... +5? 15:32:58 <jeblair> though SergeyLukjanov is infra-core, not infra-root. we still need to expand infra-root coverage 15:32:59 <SergeyLukjanov> it was re -- [19:31:07] <jeblair> cody-somerville: so for the moment, i'd like to stick with "if you need a log entry, ask in #openstack-infra" 15:33:07 <persia> ttx: Then even moreso. From a +9 perspective, infra coverage is good because of overlap with -7, but I doubt that to be the case for +5. 15:33:49 <SergeyLukjanov> my tz is +4, so, probably we could expand access for me in this case 15:33:55 * persia suddenly understands and is shamed 15:34:13 <persia> SergeyLukjanov: Apologies: I just don't see you chattering a lot on IRC in the "quiet time" :) 15:34:55 <SergeyLukjanov> persia, my bad, I'm mostly accessible on-highlight 15:34:57 <jeblair> anyway, if this doesn't scale, we can look into something like password restricted logs, or a shared production-test environment 15:35:44 <SergeyLukjanov> jeblair, openstack-dev with some fake auth could be an option 15:35:51 <SergeyLukjanov> storyboard-dev* 15:36:08 <jeblair> but production is always going to be tightly restricted, so as we learn how to develop and run a cd system like this, i think it's good to know what the parameters are and to start designing around them early 15:36:09 <persia> Even if the logs contain authentication tokens? 15:37:11 <persia> A shared production test environment sounds the most flexible: it also allows deployment of stuff for verification that the infra changes won't break the commits. 15:37:25 <persia> How much effort is it to set up such an environment? 15:37:33 <jeblair> persia: yes. the main reason i'd be resistent to that is because that's the job of the test system. 15:37:48 <jeblair> persia: and production is going to be continuously deployed anyway 15:38:00 <jeblair> persia: so you'd just end up with two broken systems. 15:38:05 <persia> Ah, so the tests should include testing mod_wsgi, etc.? 15:39:03 <jeblair> persia: if mod_wsgi is critical to the operation of it, yes. however, i don't think it is, so i don't think its necessary. 15:39:32 <jeblair> i think we're getting a bit off topic; this is starting to feel like open discussion... 15:39:46 <persia> I'm just using that as an example of where the recommended config differs, with the idea that any point of variance can be a source of unexpected issues. 15:39:55 <jeblair> so before we go there, maybe we can spend a few minutes on the other item 15:40:01 <jeblair> #topic MVP status (krotscheck) 15:40:19 <jeblair> krotscheck isn't here, but i bet we can figure out what's going on 15:41:01 <NikitaKonovalov> the last thing missing was comments and pagination, I guess 15:41:11 <NikitaKonovalov> so both of them are merged 15:42:01 <jeblair> wasn't there a problem with pagination? i think it was disabled because of that? 15:42:15 <persia> https://review.openstack.org/#/c/83847/ is reputedly important for firefox users. 15:42:54 <jeblair> it looks like we can update status as well 15:43:00 * ttx looks 15:43:37 <jeblair> https://storyboard.openstack.org/#!/story/11 says it has 'landed' 15:44:35 <jeblair> i'm pretty sure we said that should be "change merged" rather than landed... 15:44:45 <jeblair> but hey, it's there :) 15:44:55 <ttx> I think it may be useful now 15:45:19 <jeblair> ttx: i think i would agree (well, for me, once that ff patch lands) 15:46:11 <ttx> the thing behaves more than weirdly when you click on description but then not on save 15:46:18 <persia> It won't scale much yet: no searching, no tagging. But that won't matter until there are hundreds of stories. 15:46:19 <ttx> places "undefined" stuff everywhere 15:46:32 <jeblair> ttx: yeah, that's the ff patch 15:46:32 <persia> ttx: Thats the FF patch. 15:46:36 <ttx> oh. 15:46:36 <persia> Someone using FF needs to review it. 15:46:45 <ttx> that would be me 15:46:58 <ttx> I'll review it 15:47:20 <jeblair> speaking of which, i thought we had docs-draft copies of the webclient configured to use the production api server, but it doesn't seem to be working for me 15:47:30 <jeblair> if we did, that would make reviewing those a little easier 15:47:41 <ttx> as soon as I find 2 min 15:47:48 <persia> jeblair: `tox -e grunt server:prod` doesn't work? 15:48:02 <jeblair> persia: that's beside the point 15:48:54 <jeblair> anyway, back to topic -- i think we should continue to poke at it for a bit, but i think we're getting close to a point where we should discuss how to use it for more infra-related tasks and really start dogfooding it. 15:49:14 <persia> I thought that was the webclient configured to use the production server. krotscheck told me that there is no current support for running the client and API server on different hosts. 15:51:03 <jeblair> persia: hrm, that seems unecessary; i think it's important that the client be able to be run anywhere, including docs-draft and developer workstations (a dev should be able to test out changes against the production server) 15:51:21 <persia> Yes, but there needs to be a local proxy on the dev workstation 15:51:39 <persia> krotscheck says we need CORS support to do it the other way. 15:51:46 <jeblair> persia: ah that makes sense 15:52:23 <persia> tox -e grunt server:prod sets up the local proxy to point at production API server, so very good for quick testing of UI stuff if you don't have your own API server environment running somewhere. 15:52:39 <jeblair> #topic open discussion 15:53:42 <jeblair> i have not had time to make headway on the 'test on mysql' change since last week, but my plan is to continue work in that direction 15:54:54 <NikitaKonovalov> jeblair: I've been running tests on both mysql and postgres installed locally 15:55:06 <NikitaKonovalov> and they are running fine so far 15:55:21 <jeblair> NikitaKonovalov: i mean _all_ the tests; they definitely don't all pass right now 15:55:56 <ruhe> jeblair: this one https://review.openstack.org/#/c/82872/ ? 15:56:04 <jeblair> ruhe: yes, that 15:56:12 <jeblair> NikitaKonovalov: you can see a list of issues in the comments on that change 15:56:57 <jeblair> the next step for that is to either fix the sqlalchemy model to specify the engine, or switch to alembic for creating the model for the tests; i'm going to explore using alembic 15:57:15 <jeblair> it might end up being crazy and slow, but i'd like to at least see what that looks like 15:57:56 <jeblair> anyway, once the tables are being created correctly for the unit tests on jenkins, the other issues will be visible 15:58:11 <ruhe> jeblair: with the current number number of tests running alembic on each test might be fine; but i'd expect a lot more tests in near future 15:58:37 <ruhe> but we always can re-use old code which runs migrations only once and captures DB DDL and applies it in each tests 15:58:53 <jeblair> ruhe: yes, it might be a problem then. openstack compresses upgrades periodically though, we could adopt something similar... 15:59:01 <jeblair> ruhe: or that ^ could work too 15:59:20 <jeblair> anyway, i think time is up 15:59:32 <jeblair> thanks everyone! hopefully we'll have a few more people next week. :) 15:59:47 <persia> It helps to have had a meeting to get used to the new day/time :) 15:59:48 <jeblair> #endmeeting