18:02:24 <SotK> #startmeeting storyboard
18:02:25 <openstack> Meeting started Thu Oct 17 18:02:24 2019 UTC and is due to finish in 60 minutes.  The chair is SotK. Information about MeetBot at http://wiki.debian.org/MeetBot.
18:02:26 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
18:02:28 <openstack> The meeting name has been set to 'storyboard'
18:02:35 <diablo_rojo> o/
18:02:52 <SotK> #topic https://wiki.openstack.org/wiki/Meetings/StoryBoard Agenda
18:03:29 <diablo_rojo> Heh, you mean link? ;)
18:03:37 <SotK> #undo
18:03:38 <openstack> Removing item from minutes: #topic https://wiki.openstack.org/wiki/Meetings/StoryBoard Agenda
18:03:44 <SotK> its that kind of day
18:03:48 <SotK> #link https://wiki.openstack.org/wiki/Meetings/StoryBoard Agenda
18:04:21 <diablo_rojo> Nailed it :)
18:04:40 <SotK> no announcements this week, and I assume the same for migration updates, since attachments aren't done yet
18:05:08 <diablo_rojo> Correct
18:06:01 <SotK> #topic PTG + Forum
18:06:15 <SotK> #link https://www.openstack.org/ptg/#tab_schedule Schedule
18:06:19 <SotK> we are on the schedule!
18:06:29 <SotK> 10:30am on Wednesday
18:06:35 <SotK> thanks for organising that diablo_rojo
18:06:40 <diablo_rojo> No problem
18:06:45 <diablo_rojo> We've got two hours
18:06:51 <diablo_rojo> I figured that should be alright.
18:06:53 <fungi> now if i just manage to get a visa so diablo_rojo isn't left answering questions alone
18:07:07 <diablo_rojo> if we need more time we can always get more time friday
18:07:20 <diablo_rojo> fungi, you assume they will actually honor my 10 year visa ;)
18:07:37 <fungi> fair
18:08:22 <SotK> I will be asleep I'm afraid
18:08:44 <fungi> i'll be struggling to stay awake
18:08:47 <diablo_rojo> SotK, no worries. I will give you the lowdown at the next meeting or right after
18:08:51 <diablo_rojo> fungi, you and me both
18:08:56 <fungi> no coffee allowed in the ptg rooms! :/
18:09:14 <diablo_rojo> fungi, yeah. Not sure how this venue gets away with that.
18:09:27 <SotK> diablo_rojo: thanks
18:09:53 <diablo_rojo> SotK, no problem
18:10:51 <SotK> anything else PTGish to discuss?
18:11:03 <diablo_rojo> Ehhh I dont think so.
18:11:12 <diablo_rojo> Its going to be a wait and see sort of thing
18:11:18 <diablo_rojo> See who shows up that is
18:12:04 <SotK> makes sense
18:12:14 <SotK> hopefully it goes well :)
18:12:20 <SotK> #topic In Progress Work
18:12:38 <SotK> to answer the question on the agenda, attachments are indeed still waiting for me
18:12:39 <diablo_rojo> I guess we could send an email out to the discuss list annoucing we are doing onboarding there.
18:12:49 <SotK> that's a good idea
18:13:10 <diablo_rojo> I can do that in like a week or so.
18:13:18 <diablo_rojo> SotK, is there anything we can do to help?
18:15:39 <SotK> the main thing that needs doing is replacing swiftclient with openstack-sdk, but when I last tried openstack-sdk didn't work with the legacy swift auth used by the swift docker container I was using for testing
18:16:04 <fungi> ahh, yeah, it probably wants keystone?
18:16:30 <SotK> yeah I expect so
18:17:44 <SotK> so its a matter of me hunting down some alternatives that allow me to set up keystone and swift for testing easily and reliably, ideally in docker with the rest of the things
18:19:01 <diablo_rojo> Okay so kinda sounds like a one person job then?
18:19:52 <SotK> indeed
18:19:54 <fungi> there's likely already something like that available which zuul is using to test their swift integration
18:20:19 <fungi> so may not require much hunting
18:20:31 <SotK> sounds probable, I'll try to find time to look/ask about what they're doing soon and get things moving again
18:20:51 <fungi> we can inquire in #zuul and probably get an answer, or i can go diving in the job configs
18:21:59 <SotK> asking sounds easier, lets do it after this meeting I guess
18:22:27 <fungi> wfm
18:23:04 <SotK> we also still have slow queries
18:23:22 <SotK> iirc the worst offender was the general story browsing query, which makes sense
18:23:35 <SotK> I've not done any drilling down into the details of why though
18:24:24 <diablo_rojo> If you want to try to point me somewhere to investigate I can try to info gather for you.
18:26:59 <fungi> if someone can give me the syntax for triggering that via the api, it'll probably speed up my ability to test the cache memory pressure theories too
18:28:40 <SotK> I *think* the massive query was just the standard one generated by doing a GET on /api/v1/stories?limit=100 (with whatever extra filtering you fancy)
18:29:56 <SotK> my first port of call was going to be seeding my dev instance with enough data to replicate the slowness we see in production, and then experiment with changing https://opendev.org/opendev/storyboard/src/branch/master/storyboard/db/models.py#L450-L475
18:30:36 <SotK> and/or adding some indexes to the relevant tables
18:30:53 <SotK> (on task status for example)
18:30:53 <fungi> cool, that helps
18:30:57 <diablo_rojo> Noted.
18:31:24 <fungi> running `time wget -qO/dev/null 'https://storyboard.openstack.org/api/v1/stories?limit=100'` gets me a baseline of "real 0m6.058s" at the moment
18:32:00 <fungi> after the meeting i'll run a battery of those and take averages before and after restarts of apache, mysql and rabbit
18:32:15 <diablo_rojo> Sounds like a good plan to me.
18:32:17 <SotK> nice, thanks
18:32:23 <fungi> as well as record the timestamps so we can correlate to cacti graphs of memory utilization
18:32:49 <fungi> (and i'll get some direct memory utilization numbers from the server before and after each as well)
18:33:25 <diablo_rojo> Do we want to throw all this research in a single etherpad?
18:34:02 <diablo_rojo> I feel like it would be good to have it all + links to other slowquery logs in a single place.
18:34:16 <fungi> #link https://etherpad.openstack.org/storyboard-query-performance-analysis
18:34:23 <fungi> i'll start doing it there
18:34:28 <diablo_rojo> Beautiful.
18:34:39 <SotK> excellent
18:35:03 <diablo_rojo> Thanks fungi!
18:36:08 <SotK> the agenda also mentions the multiple task race condition issue
18:36:17 <SotK> iirc we fixed that but caused a new bug with the fix
18:36:22 <SotK> new/similar
18:36:47 <SotK> I think we'd come up with a solution but will need to dig in IRC logs to remember properly I expect
18:37:03 <diablo_rojo> Oh okay I forgot the solution.
18:37:13 <diablo_rojo> If we have one I can try to implement it.
18:38:04 <fungi> yeah, i think the beer got to those memory cells for me
18:38:27 <diablo_rojo> I would blame it on the 'too many things at once' state I live in
18:38:32 <fungi> would similarly need to go hunting for discussion or at least the commit we thought was fixing it
18:38:53 <fungi> was the fix one which came from an intern or prospective intern? i want to say it was
18:39:43 <diablo_rojo> Oh maybe
18:40:03 <diablo_rojo> I feel like I should do a pass over all the open patches from outreachy folks
18:40:09 <diablo_rojo> reviews + updates
18:40:21 <SotK> yeah, we just added an oslo_db decorator that handles retrying on deadlock
18:40:32 <diablo_rojo> Oh thats easy.
18:40:39 <SotK> but it moved the issue to somewhere else
18:40:45 <diablo_rojo> Oh
18:41:16 <fungi> i think it had to do with it all happening within a transaction?
18:41:33 <SotK> yeah, something along those lines
18:41:52 <fungi> so retrying didn't help because it was all in the context of the same transaction
18:42:13 <diablo_rojo> Ah
18:42:17 <diablo_rojo> That makes sense
18:43:06 <fungi> but also, retrying in the face of contention is a hack
18:43:29 <diablo_rojo> Yeah would be better to know what exactly is causing it.
18:43:52 <fungi> and i believe if that *had* actually "worked" it would have just shifted the limit on number of simultaneous tasks you can create to the number of retries you perform
18:44:41 <SotK> indeed
18:45:03 <fungi> so if you try 3 times then you'll break on simultaneous creation of 4 tasks
18:45:15 <fungi> not really a solution
18:46:19 <timburke> SotK, just catching up, but fwiw, swiftclient offers a v1password plugin for keystoneauth; i know i've had some success using it with openstackclient... might need https://review.opendev.org/#/c/687774/ though. but if the goal is to get rid of swiftclient *entirely*... i might not be in a position to help much ;-)
18:46:58 <fungi> i think the goal was to simplify testing/local dev implementation
18:47:39 <fungi> but the result is going to need to work with clouds which provide keystone authentication, so we're probably better off doing a keystone+swift for that testing anyway
18:47:54 <timburke> 👍
18:51:31 <SotK> indeed, keystone + swift makes sense really
18:51:48 <SotK> we also wanted to discuss the js draft build issue
18:52:01 <diablo_rojo> (8 min left)
18:52:01 <fungi> yeah, i can cover that real quick
18:52:19 <diablo_rojo> Yes please
18:52:45 <fungi> since opendev moved from hosting build artifacts/logs on a static logs.openstack.org server to publishing them to (swift) object storage, we can no longer easily predict what hostnames those are going to be served from
18:53:41 <fungi> the storyboard api is currently designed to require you to whitelist the domain names of such deployments both for cors middleware and for openid auth
18:54:30 <fungi> because those domains are unpredictable, and because the storyboard-dev deployment they point to is not supposed to be used for sensitive information, i would like to basically turn off the cors and openid whitelisting for it
18:54:58 <fungi> the easiest practical means of doing that is to allow the config fields for those to accept a wildcard indicator
18:55:26 <fungi> as a naive first implementation i'd like to just make them consider '*' to mean any hostname should be considered valid
18:55:52 <SotK> that seems like a good enough solution to me
18:56:12 <fungi> it seems there was no disagreement in #storyboard when i suggested that, i just need to write the implementation. i've found what i believe are the affected codepaths
18:56:33 <fungi> so it shouldn't take long
18:56:45 <fungi> but would be nice to prioritize reviews of that once i get it proposed
18:57:01 <fungi> as i gather it's blocking us from having confidence in our reviews of webclient changes
18:57:19 <fungi> (also i think the change will be pretty simple)
18:57:38 <fungi> anyway, that's all i've got on that topic
18:57:43 <SotK> thanks fungi
18:57:58 <SotK> I'll prioritize reviewing them when they're ready
18:58:15 <fungi> appreciated. i'll try to get to that tomorrow
18:58:19 <diablo_rojo> I am happy to review once I see the patches go up
18:58:48 <diablo_rojo> SotK, after the meeting (while pinging the Zuul channel) would you mind reviewing that last show email patch I have out so that we can merge it?
18:58:56 <diablo_rojo> (assuming its okay)
18:59:07 <SotK> diablo_rojo: yeah sure
18:59:51 <SotK> and we're pretty much out of time, thanks for coming folks
18:59:54 <diablo_rojo> https://review.opendev.org/#/c/589713/
18:59:57 <fungi> yeah, attempting to review that is what reminded me of the draft breakage
18:59:59 <diablo_rojo> SotK, ^^
19:00:12 * SotK opens
19:00:16 <SotK> #endmeeting