19:01:10 #startmeeting infra 19:01:11 Meeting started Tue Dec 17 19:01:10 2019 UTC and is due to finish in 60 minutes. The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot. 19:01:12 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 19:01:12 o/ 19:01:14 The meeting name has been set to 'infra' 19:01:34 o/ 19:01:45 #link http://lists.openstack.org/pipermail/openstack-infra/2019-December/006560.html Our Agenda 19:02:02 This agenda hasn't changed much from last time. I expect we'll get through this quickly 19:02:08 #topic Announcements 19:02:39 It is that time of year where many of us work less and do things with family and friends. As such we'll not have a meeting the next two weeks. See you here on January 7th for the next meeting. 19:03:25 I've got family in town starting thursday so my hours will get weird after that. I expect I'll be around for most of this week then less around next week 19:03:36 #topic Actions from last meeting 19:03:41 #link http://eavesdrop.openstack.org/meetings/infra/2019/infra.2019-12-10-19.01.txt minutes from last meeting 19:03:55 No actions from last meeting. Lets dive right into the other topics then 19:04:01 #topic Priority Efforts 19:04:07 #topic Update Config Management 19:04:25 mordred: I believe I saw a change merge to manage review-dev with all of our automation stuff and dockerized gerrit 19:05:04 maybe it was an update to that change and I read it as merged 19:05:26 i thought i saw something similar merge 19:05:48 I'm having a hard time finding it now but I really thought I saw that go by in IRC 19:06:02 https://review.opendev.org/630406 19:06:05 thanks 19:06:17 that merged on sunday 19:06:37 review-dev is still running a gerrit that I can reach over https so I assume that went well? 19:07:08 bf0870ac373c docker.io/opendevorg/gerrit:2.13 /bin/sh -c /usr/l... 7 weeks ago Up 7 weeks ago gerrit-podman_gerrit_1 19:07:33 review-dev is in emergency file 19:07:48 ah ok 19:08:15 i believe the status is that mordred has manually run that on review-dev, it worked, the change to automate it has merged, and next step is to remove r-d from emergency so it actually runs 19:08:23 sounds good 19:08:34 out of curiosity, is the content of /root/.gerrit_db.cnf on review.o.o outdated? i can't seem to authenticate to mysql with it currently 19:08:36 i reckon any of us could do that at any time 19:09:26 though that file was last touched 19 months ago, so doubtful anything recent has altered it while working out the review-dev deployment 19:09:30 fungi: podman says there is no mysql running as a container 19:09:45 well, it's still pointed at trove 19:09:58 (this is review.o.o, production, not review-dev) 19:10:35 fungi: /home/gerrit2/review_site/etc/gerrit.config says it's configured to use a cloud db 19:10:40 oh 19:10:42 right 19:10:46 i was looking at r-d 19:11:05 well, we can move on, i don't think it's related to the container stuff 19:11:18 just thought i'd raise it as we were discussing gerrit 19:11:25 i can say that the container config does not mount that file into the container 19:11:40 so we have 2 pieces of evidence it seems to be a forgotten file 19:12:48 its used for db backups 19:13:05 well that's no good 19:13:52 so ya if the password or host changed then we'll need to update puppet/hiera/anibslehostvars to apply to that file 19:13:55 strangely, the credentials in ~gerrit2/review_site/etc/secure.config also aren't working with the mysqlclient there 19:14:04 at least for me 19:14:20 ohhh 19:14:25 it's trying to use ssl, i think 19:14:32 i wonder if a default changed 19:16:57 so anyway, it seems likely that our db backups are now failing 19:17:08 empty files getting written into /home/gerrit2/mysql_backups/ 19:17:30 yes mysql --skip-ssl seems to work 19:17:56 we'll likely need to update our mysqldump invocation 19:18:00 ++ 19:18:26 Any other config management updates to consider? 19:18:27 fungi: thanks for catching that 19:18:39 it was sheer coincidence 19:18:45 but my pleasure 19:19:25 seems like that was working in april ish 19:19:33 because we did db backups and such pre opendev migration 19:20:06 Alright lets move on 19:20:09 #topic OpenDev 19:20:21 There is a gitea 1.10.1 now. Reading its changelog I don't expect it to fix our git issues 19:20:35 But it does have a number of bug fixes which we should consider upgrading for. 19:21:02 ianw: mordred: that process is basically bump the tag in our dockerfile and diff our template files against upstream's templates to make sure we don't need edits in those? 19:21:18 I can give that a go tomorrow likely 19:21:31 i believe so, i think with a . release should be minimal 19:21:41 ya it should be but I'll still double check 19:22:30 I haven't heard any new feedback on the opendev governance proposal thread. I take it there aren't any major objectiosn to the latest revision in that thread? 19:22:44 Should we start planning to take that up with the openstack TC? 19:24:00 i'm pleased by the direction it's going 19:25:06 k. I expect the TC will also slow down activity over the next while but this could be a good early year activity for us. 19:25:50 #topic General Topics 19:25:58 fungi: I assume that there are no wiki updates? 19:26:42 negatory 19:26:53 Next up is static.o.o replacement 19:27:03 ianw it sounds like a bit of progress has been made? 19:27:55 https://review.opendev.org/#/q/status:open+topic:static.opendev.org 19:28:26 ianw: are you happy with the state of testing in those changes now? 19:28:34 that stack is ready for review, which is the config for a basic server to provide security.openstack.org and governance.openstack.org 19:28:56 there's a couple of other docs sites to publish in similar ways 19:29:30 exciting. I've probably forgotten and this was in the spec. But why aren't we using the existing filesXY server(s)? 19:29:41 is this beacuse the files servers are still "openstack" and we want to avoid that? 19:29:43 then we have to think about tarballs/releases ... but it's a start :) 19:29:54 yeah, to move to 100% ansible environment 19:30:00 gotcha 19:31:14 alright that has been added to my review list after the review grenade zuulv3 jobs changes and the fix for mysql skip ssl once we've got one of those 19:31:47 And I think that takes us to the end of our regularly scheduled agenda 19:31:52 #topic Open Discussion 19:32:01 There were a couple other things I should mention 19:32:15 We have a project rename request. I don't really have interest in doing that during the holidays 19:32:28 but probably sometime in january we'll do a project rename? 19:33:04 The other was I wanted to point out I've been doing some investigating of zuul job retries. These happen because pre-run tasks fail or ansible reports a networking error. 19:33:26 What I found is that we actually do have a fair number of retries especially when a cloud mirror has a sad. The feature is very helpful to us as a result 19:33:45 But we also have jobs that end up retrying over and over consistently which I've tried to identify and report to the responsible parties 19:34:09 As zuul operators I think this is something we'll want to keep taking the tempurature of to make sure we aren't masking larger problems 19:34:59 Also I decided to give geoguessr a go. https://geoguessr.com/challenge/9TSJaHwD74CdiSNp if anyone wants to do something fun now that the meeting is winding down :) 19:35:07 I have no idea how well this will work 19:35:11 i did have on the nodepool container work i thought, but it's ok, most of it has review, except for 19:35:15 #link https://review.opendev.org/#/c/693306/7 19:35:51 oh did that get added? I thought there weren't any edits. My bad if so 19:36:00 it will be the new year now, but then i'd like to bring up a builder with nodepool containers 19:36:36 * AJaeger pushed some changes to remove trusty jobs - now the major remaining user of trusty nodes is infra 19:36:58 AJaeger: ++ thanks for those; will be great to stop building that 19:37:40 (as background, we found that you couldn't build trusty on bionic hosts, because bionic's mkfs.ext4 makes a filesystem that trusty-era kernels don't like with metadata_csum flags) 19:37:44 oh yes I should review those 19:38:04 ... dib has a very hacky workaround but it would be ideal if we just didn't care 19:38:36 ++ 19:39:01 once those are in, let's review which jobs we still need to run ourselves - If anybody tells me, I can cleanup... 19:39:23 AJaeger: sounds like a plan 19:48:43 And sounds like that may be it for the meeting. Thanks everyone! 19:48:45 #endmeeting