19:01:10 <clarkb> #startmeeting infra
19:01:11 <openstack> Meeting started Tue Dec 17 19:01:10 2019 UTC and is due to finish in 60 minutes.  The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:01:12 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:01:12 <ianw> o/
19:01:14 <openstack> The meeting name has been set to 'infra'
19:01:34 <diablo_rojo> o/
19:01:45 <clarkb> #link http://lists.openstack.org/pipermail/openstack-infra/2019-December/006560.html Our Agenda
19:02:02 <clarkb> This agenda hasn't changed much from last time. I expect we'll get through this quickly
19:02:08 <clarkb> #topic Announcements
19:02:39 <clarkb> It is that time of year where many of us work less and do things with family and friends. As such we'll not have a meeting the next two weeks. See you here on January 7th for the next meeting.
19:03:25 <clarkb> I've got family in town starting thursday so my hours will get weird after that. I expect I'll be around for most of this week then less around next week
19:03:36 <clarkb> #topic Actions from last meeting
19:03:41 <clarkb> #link http://eavesdrop.openstack.org/meetings/infra/2019/infra.2019-12-10-19.01.txt minutes from last meeting
19:03:55 <clarkb> No actions from last meeting. Lets dive right into the other topics then
19:04:01 <clarkb> #topic Priority Efforts
19:04:07 <clarkb> #topic Update Config Management
19:04:25 <clarkb> mordred: I believe I saw a change merge to manage review-dev with all of our automation stuff and dockerized gerrit
19:05:04 <clarkb> maybe it was an update to that change and I read it as merged
19:05:26 <Shrews> i thought i saw something similar merge
19:05:48 <clarkb> I'm having a hard time finding it now but I really thought I saw that go by in IRC
19:06:02 <corvus> https://review.opendev.org/630406
19:06:05 <clarkb> thanks
19:06:17 <corvus> that merged on sunday
19:06:37 <clarkb> review-dev is still running a gerrit that I can reach over https so I assume that went well?
19:07:08 <corvus> bf0870ac373c  docker.io/opendevorg/gerrit:2.13  /bin/sh -c /usr/l...  7 weeks ago  Up 7 weeks ago         gerrit-podman_gerrit_1
19:07:33 <corvus> review-dev is in emergency file
19:07:48 <clarkb> ah ok
19:08:15 <corvus> i believe the status is that mordred has manually run that on review-dev, it worked, the change to automate it has merged, and next step is to remove r-d from emergency so it actually runs
19:08:23 <clarkb> sounds good
19:08:34 <fungi> out of curiosity, is the content of /root/.gerrit_db.cnf on review.o.o outdated? i can't seem to authenticate to mysql with it currently
19:08:36 <corvus> i reckon any of us could do that at any time
19:09:26 <fungi> though that file was last touched 19 months ago, so doubtful anything recent has altered it while working out the review-dev deployment
19:09:30 <clarkb> fungi: podman says there is no mysql running as a container
19:09:45 <fungi> well, it's still pointed at trove
19:09:58 <fungi> (this is review.o.o, production, not review-dev)
19:10:35 <corvus> fungi: /home/gerrit2/review_site/etc/gerrit.config says it's configured to use a cloud db
19:10:40 <corvus> oh
19:10:42 <fungi> right
19:10:46 <corvus> i was looking at r-d
19:11:05 <fungi> well, we can move on, i don't think it's related to the container stuff
19:11:18 <fungi> just thought i'd raise it as we were discussing gerrit
19:11:25 <corvus> i can say that the container config does not mount that file into the container
19:11:40 <corvus> so we have 2 pieces of evidence it seems to be a forgotten file
19:12:48 <clarkb> its used for db backups
19:13:05 <corvus> well that's no good
19:13:52 <clarkb> so ya if the password or host changed then we'll need to update puppet/hiera/anibslehostvars to apply to that file
19:13:55 <fungi> strangely, the credentials in ~gerrit2/review_site/etc/secure.config also aren't working with the mysqlclient there
19:14:04 <fungi> at least for me
19:14:20 <fungi> ohhh
19:14:25 <fungi> it's trying to use ssl, i think
19:14:32 <fungi> i wonder if a default changed
19:16:57 <fungi> so anyway, it seems likely that our db backups are now failing
19:17:08 <fungi> empty files getting written into /home/gerrit2/mysql_backups/
19:17:30 <clarkb> yes mysql --skip-ssl seems to work
19:17:56 <fungi> we'll likely need to update our mysqldump invocation
19:18:00 <clarkb> ++
19:18:26 <clarkb> Any other config management updates to consider?
19:18:27 <corvus> fungi: thanks for catching that
19:18:39 <fungi> it was sheer coincidence
19:18:45 <fungi> but my pleasure
19:19:25 <clarkb> seems like that was working in april ish
19:19:33 <clarkb> because we did db backups and such pre opendev migration
19:20:06 <clarkb> Alright lets move on
19:20:09 <clarkb> #topic OpenDev
19:20:21 <clarkb> There is a gitea 1.10.1 now. Reading its changelog I don't expect it to fix our git issues
19:20:35 <clarkb> But it does have a number of bug fixes which we should consider upgrading for.
19:21:02 <clarkb> ianw: mordred: that process is basically bump the tag in our dockerfile and diff our template files against upstream's templates to make sure we don't need edits in those?
19:21:18 <clarkb> I can give that a go tomorrow likely
19:21:31 <ianw> i believe so, i think with a . release should be minimal
19:21:41 <clarkb> ya it should be but I'll still double check
19:22:30 <clarkb> I haven't heard any new feedback on the opendev governance proposal thread. I take it there aren't any major objectiosn to the latest revision in that thread?
19:22:44 <clarkb> Should we start planning to take that up with the openstack TC?
19:24:00 <corvus> i'm pleased by the direction it's going
19:25:06 <clarkb> k. I expect the TC will also slow down activity over the next while but this could be a good early year activity for us.
19:25:50 <clarkb> #topic General Topics
19:25:58 <clarkb> fungi: I assume that there are no wiki updates?
19:26:42 <fungi> negatory
19:26:53 <clarkb> Next up is static.o.o replacement
19:27:03 <clarkb> ianw it sounds like a bit of progress has been made?
19:27:55 <ianw> https://review.opendev.org/#/q/status:open+topic:static.opendev.org
19:28:26 <clarkb> ianw: are you happy with the state of testing in those changes now?
19:28:34 <ianw> that stack is ready for review, which is the config for a basic server to provide security.openstack.org and governance.openstack.org
19:28:56 <ianw> there's a couple of other docs sites to publish in similar ways
19:29:30 <clarkb> exciting. I've probably forgotten and this was in the spec. But why aren't we using the existing filesXY server(s)?
19:29:41 <clarkb> is this beacuse the files servers are still "openstack" and we want to avoid that?
19:29:43 <ianw> then we have to think about tarballs/releases ... but it's a start :)
19:29:54 <ianw> yeah, to move to 100% ansible environment
19:30:00 <clarkb> gotcha
19:31:14 <clarkb> alright that has been added to my review list after the review grenade zuulv3 jobs changes and the fix for mysql skip ssl once we've got one of those
19:31:47 <clarkb> And I think that takes us to the end of our regularly scheduled agenda
19:31:52 <clarkb> #topic Open Discussion
19:32:01 <clarkb> There were a couple other things I should mention
19:32:15 <clarkb> We have a project rename request. I don't really have interest in doing that during the holidays
19:32:28 <clarkb> but probably sometime in january we'll do a project rename?
19:33:04 <clarkb> The other was I wanted to point out I've been doing some investigating of zuul job retries. These happen because pre-run tasks fail or ansible reports a networking error.
19:33:26 <clarkb> What I found is that we actually do have a fair number of retries especially when a cloud mirror has a sad. The feature is very helpful to us as a result
19:33:45 <clarkb> But we also have jobs that end up retrying over and over consistently which I've tried to identify and report to the responsible parties
19:34:09 <clarkb> As zuul operators I think this is something we'll want to keep taking the tempurature of to make sure we aren't masking larger problems
19:34:59 <clarkb> Also I decided to give geoguessr a go. https://geoguessr.com/challenge/9TSJaHwD74CdiSNp if anyone wants to do something fun now that the meeting is winding down :)
19:35:07 <clarkb> I have no idea how well this will work
19:35:11 <ianw> i did have on the nodepool container work i thought, but it's ok, most of it has review, except for
19:35:15 <ianw> #link https://review.opendev.org/#/c/693306/7
19:35:51 <clarkb> oh did that get added? I thought there weren't any edits. My bad if so
19:36:00 <ianw> it will be the new year now, but then i'd like to bring up a builder with nodepool containers
19:36:36 * AJaeger pushed some changes to remove trusty jobs - now the major remaining user of trusty nodes is infra
19:36:58 <ianw> AJaeger: ++ thanks for those; will be great to stop building that
19:37:40 <ianw> (as background, we found that you couldn't build trusty on bionic hosts, because bionic's mkfs.ext4 makes a filesystem that trusty-era kernels don't like with metadata_csum flags)
19:37:44 <clarkb> oh yes I should review those
19:38:04 <ianw> ... dib has a very hacky workaround but it would be ideal if we just didn't care
19:38:36 <clarkb> ++
19:39:01 <AJaeger> once those are in, let's review which jobs we still need to run ourselves - If anybody tells me, I can cleanup...
19:39:23 <clarkb> AJaeger: sounds like a plan
19:48:43 <clarkb> And sounds like that may be it for the meeting. Thanks everyone!
19:48:45 <clarkb> #endmeeting