19:01:27 <clarkb> #startmeeting infra
19:01:28 <openstack> Meeting started Tue Oct 30 19:01:27 2018 UTC and is due to finish in 60 minutes.  The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:01:29 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:01:32 <openstack> The meeting name has been set to 'infra'
19:01:40 <clarkb> #topic Announcements
19:01:55 <clarkb> Hopefully this doesn't surprise anyone but the Berlin summit is less than two weeks away
19:02:22 <clarkb> This is your warning to do any necesary prep work :) I need to put together a project update and onboarding session
19:02:41 <clarkb> #link https://civs.cs.cornell.edu/cgi-bin/vote.pl?id=E_aac97f1cbb6c61df&akey=b9e448b340787f0e Vote for OpenStack T release name
19:02:54 <clarkb> tonyb put together a poll for us to pikc a name. You should really vote for train
19:03:06 <fungi> note that if you share an ip address you'll need to find an alternative from which to place your vote
19:03:28 <fungi> say, an unpopular starbucks
19:03:39 <clarkb> or via your phone?
19:03:39 <fungi> or maybe the public library (they still have those, right?)
19:03:44 <ianw> o/
19:04:12 <clarkb> Anything else worth announcing?
19:04:38 <fungi> not to my knowledge
19:04:44 <clarkb> #topic Actions from last meeting
19:04:51 <clarkb> #link http://eavesdrop.openstack.org/meetings/infra/2018/infra.2018-10-23-19.01.txt minutes from last meeting
19:05:15 <clarkb> Not in that ^ meeting but in the opendev messaging meeting I took an action to start drafting an email to talk about opendev more broadly
19:05:37 <clarkb> I've got time set aside for that on this meeting agenda but if you want a sneak preview https://etherpad.openstack.org/p/WUNOTv8MuP
19:05:56 <clarkb> #topic Priority Efforts
19:05:56 * fungi sneaks a peek
19:06:07 <clarkb> #topic Update Config Management
19:06:29 <clarkb> May as well jump right in. I don't think much happened on this last week as we were distracted by zookeeper, openstacksdk, and nodepool related things (at least I was)
19:07:04 <clarkb> gerrit query for reviews: topic:puppet-4 and topic:update-cfg-mgmt
19:07:10 * mordred aplogizes for his hand in the openstacksdk things
19:08:13 <fungi> i, for one, am glad you choose to do openstacksdk things
19:08:14 <fungi> don't apologize
19:08:16 <clarkb> have I missed related work on ^ worth talking about? I guess we may still need a decision on where to store the ara database? I think the conversation we had a week ago mostly ended up with maybe we should rebuild bridge server to be appropriately sized for ansible and mysql
19:08:34 <clarkb> then we reduce data leakage concerns and have server that performs as expected across the board
19:09:15 <fungi> well, there was already a trove instance created, so i thought we said in the interim we can use that in parallel with any work to replace the bridge server
19:09:24 <clarkb> ah
19:09:39 <mordred> I support all of the choices
19:09:47 <fungi> but that in sizing the replacement bridge.o.o we should take running a mysql db for ara into account
19:09:53 <clarkb> I was distracted by zookeeper move at the time so fungi's memory likely better than mine
19:10:08 <clarkb> ++
19:10:39 <fungi> yeah, the information leak is theoretical, and so short-term concerns while we're getting this up and running aren't as significant as the longer-term risks of continuing to use a separate db system
19:10:54 <clarkb> sounds good
19:11:04 <clarkb> mordred: hae a storyboard containerization update?
19:11:27 <mordred> oh my
19:11:30 * fungi hopes it's not an old margarine tub
19:11:56 <mordred> so - I have decided to step back from that just a little bit- a bit more to wrap my head around
19:12:11 <mordred> and gonna just do a normal update - so I can properly think about what the containerized version looks like
19:13:09 <mordred> I'll hopefully get some patches up for review later today or early tomorrow
19:13:11 <fungi> now thinking that "margarine-tub" would have been a great alternative name for pbrx
19:13:18 <mordred> fungi: still might be
19:13:31 <clarkb> seems reasonable. I do think we shouldn't be afraid of doing the quicker get off trusty moves without blocking too hard on containers if it isn't straightforward upfront
19:13:37 <mordred> yup
19:13:43 <mordred> that's kinda where I ended up
19:13:53 <mordred> also - to be fair - I'm *also* overthinking it
19:14:21 <clarkb> anything else related to updating config management before we go to storyboard?
19:14:25 <clarkb> (this is the segway)
19:14:29 <fungi> so basically upgrading storyboard-dev.o.o and then storyboard.o.o from trusty to xenial for now?
19:14:34 <mordred> fungi: yah
19:14:47 <clarkb> er segue? english weird
19:14:48 <fungi> wfm
19:14:59 <fungi> sanguine
19:15:01 <mordred> I'd like to eventually end up with storyboard running in python3 in containers on bionic
19:15:07 <clarkb> mordred: shiny
19:15:10 <mordred> but let's just make progress
19:15:29 <clarkb> #topic Storyboard
19:15:38 <clarkb> I did review the storyboard spec
19:15:45 <fungi> me too! (finally)
19:16:03 <fungi> i also need to apologize to diablo_rojo for the pile of comments i left on it
19:16:11 <clarkb> and frickler too. Thank you all for doing that
19:16:18 <clarkb> Looks like we are waiting on an update on that now?
19:16:45 <fungi> if any of my comments are unclear or need added detail, please let me know and i'm happy to expand on them
19:17:33 <clarkb> are tehre other storyboard updates? I had to miss the meeting last week due to dentist visit
19:18:46 <fungi> some talk of forum planning
19:19:17 <fungi> there's a remaining blockers to migration session on the forum schedule
19:19:21 <clarkb> fungi: project moves have slowed down?
19:19:22 <fungi> #link https://www.openstack.org/summit/berlin-2018/summit-schedule/events/22839/storyboard-migration-the-remaining-blockers Summit Session
19:19:27 <clarkb> I'm guessing that is due to the remaining blockers?
19:19:30 <fungi> for the moment, yes
19:19:37 <fungi> well, and also some roadblocks in testing them
19:20:02 <fungi> diablo_rojo has discovered that test imports of neutron abruptly stop partway into what we would expect to have loaded
19:20:27 <fungi> suspecting at this point that we're missing support for lp api result pagination or something along those lines
19:20:34 <clarkb> I guess it is good that we test them then :)
19:20:55 <fungi> but need to add some more debugging output to figure it out
19:21:09 <fungi> indeed!
19:21:13 <mordred> oh joy
19:21:41 <fungi> neutron is by far the largest single lp project we've attempted to import, so no surprises it's where we're finding new problems
19:22:09 <clarkb> mordred: ^ probably want to coordinate the redeployment of -dev with neutron debugging
19:22:26 <clarkb> (just to avoid deleting old server in the middle of a test migration)
19:22:52 <mordred> clarkb: yes - I agree.
19:22:53 <clarkb> #topic General Topics
19:23:10 <clarkb> Opendev. As mentioned earlier I've started drafting an email to communicate the things
19:23:32 <clarkb> It depends a bit on the work lsell will be doing foundation side, but I'm trying to set a tone of we aren't abandoning you and this is something people have wanted for a while
19:23:48 <clarkb> I think it is also helpful that we now have data saying 98.5% of all the test resources we use are openstack related
19:23:57 <fungi> point of order. i move that any specs related to the opendev transition be deemed priority efforts
19:23:58 <mordred> ++
19:24:15 * mordred seconds fungi's point of order
19:24:17 <clarkb> That said I am not sure I'm entirely happy with that draft so please give me feedback or suggest wys to make it better
19:24:21 <clarkb> mordred: fungi I'll third that
19:24:22 * mordred also knows that's not how robert's rules work
19:24:38 <fungi> yep
19:24:48 <fungi> i was abusing terms for fun
19:24:53 <mordred> me too
19:24:59 <clarkb> #link https://etherpad.openstack.org/p/WUNOTv8MuP draft email announcement
19:25:08 <mordred> just wanted to be clear in case any robert's rules superfans were lurking
19:25:08 <clarkb> now the link shows up in the correct spot in the meeting logs
19:25:41 <clarkb> I'll try to sync up with lsell real soon now to see how the other bits are going
19:26:37 <clarkb> Doesn't look like the dns configurations are fully in place for opendev.org yet, but progress was amde on that lst week too
19:26:48 <clarkb> I expect corvus will be motivated to finish that when back at work
19:27:40 <clarkb> Trusty server upgrades. As hinted to earlier with storyboard this continue to happen. The zookeeper cluster work last week was basically a trusty to xenial upgrade for our zookeeper server. The trusty zookeeper is running on nodepool.openstack.org. It is on my list of todos today to delete that server
19:28:04 <clarkb> this server has been around for a long time and was a fairly important piece of the puzzle for us so if you know of anything that needs saving or cleanup before I delete it please let me know
19:28:18 <clarkb> (possbily useful openstack api related scripts)
19:28:41 <fungi> also, no obvious problems after switching from single-node zk to a cluster
19:28:47 <fungi> so that was nice
19:29:06 <clarkb> correct. I chceked the zk stat output on all three servers for days after and it has continued to look happy
19:29:10 <ianw> ++, now we just need to get it ssl :)
19:29:11 <clarkb> cacti also makes it looks happy
19:29:39 <clarkb> Once nodepool.o.o is gone I'm going to grab another server off the list and go look at what that server needs. Maybe pbx?
19:30:01 <clarkb> I feel like I've got some momentum with this and am hoping to keep doing a server ish a week if I can
19:30:08 <fungi> i guess that did eventually get rebuilt on ubuntu, huh?
19:30:12 <clarkb> fungi: yup
19:30:14 <ianw> i'm hoping to get to some upgrades too, if nothing else pops up
19:30:49 <fungi> are the git servers and load balancer our only centos servers now?
19:30:58 <clarkb> ianw: great. Please do let me know if I can help (that goes for anyone doing this work I am highly motivated to get this type of thing done and happy to help where I can)
19:31:01 <clarkb> fungi: yes
19:31:35 <clarkb> #link https://etherpad.openstack.org/p/201808-infra-server-upgrades-and-cleanup list of servers that need upgrading
19:32:42 <clarkb> Zuul queue backlogs. Since the PTG we've been experiencing long zuul queue backlogs. There have been a varity or reasons for this over the last month and a half many of which we've fixed. But the general situation is that we are still suffering from this
19:32:58 <clarkb> #link http://lists.openstack.org/pipermail/openstack-dev/2018-October/136141.html Most recent update from clarkb on the Zuul backlog situation
19:33:14 <clarkb> The good news is we are getting better data from zuul around where the cost is
19:33:25 <clarkb> Using that we can measure if our changes improve things
19:33:32 <clarkb> and also prioritize which work would be most effective
19:33:38 <mordred> ++ this is great
19:34:50 <clarkb> Another item worth pointing out is that I think we are in part running into lack of logging in jobs that help us understand where the time spend is within a job. I've been asking various projects to improve the logging so that the statement isn't just "infra is slow or unreliable" and instead is "on infra $task takes this long and is $resource bound"
19:35:19 <clarkb> For example tripleo points to docker image download costs. But as far as we know we don't have a measurement of that so step 0 is lets measure that then we'll know if changes improve it
19:35:24 <clarkb> (or if it is even a major cost at all)
19:35:42 <clarkb> OSH has performance issues but we don't knwo if they are related to cpu/disk/network/etc so to improve that we need that information
19:35:56 <mordred> yup. logging here is also helpful, because while we do have docker mirrors, it may not always be clear if we're actually using them or not
19:36:01 <clarkb> There is zuul work to build in some performance data into base jobs too which should help
19:36:17 <clarkb> I think we've taken for granted that devstack does all of this for you
19:36:21 <mordred> yup
19:36:29 <clarkb> and now that we are doing more and more non devstack testing the lack of that data is apparent
19:37:06 <clarkb> If random projects drop by asking for help implementing that stuff any help you can give will likely go a long way
19:38:21 <clarkb> And that was it for the agenda I had written down
19:38:30 <clarkb> #topic Open Discussion
19:38:40 <clarkb> Anything else? we've got time so feel free
19:39:14 <fungi> i've got nothin'
19:40:25 <clarkb> my kids have a halloween parade thing tomorrow at school. I am not yet sure if I am expected to go view that
19:40:31 <clarkb> but I may be afk for an hour or two tomorrow for that
19:40:51 <clarkb> fungi: ^ did the storm take precedence over halloween planning?
19:41:50 <clarkb> I'll give this until 19:45 UTC and if we don't have anything to talk about by then I'll end the meeting
19:42:16 <fungi> yeah, we sort of didn't decorate much
19:42:36 <fungi> gonna chill with some friends and frighten/rot the teeth of children in their neighborhood this year
19:44:57 <clarkb> oh! we get off DST this weekend in this country at least
19:45:14 <mordred> oh - is that this weekend?
19:45:18 <clarkb> friendly reminder that meeting times are UTC so if your local government observes DST you may have the meeting shift by an hour
19:45:26 <fungi> yeah, i need to skim my recurring schedule for any meetings which shift relative to dst
19:45:37 <fungi> or rather which follow dst and so shift in utc
19:46:02 <clarkb> the meeting will be an hour earlier in my local timezone next week
19:46:24 <clarkb> and with that I think I'll call the meeting. Thank you everyone
19:46:28 <clarkb> #endmeeting