19:01:29 <ianw> #startmeeting infra 19:01:30 <openstack> Meeting started Tue Aug 13 19:01:29 2019 UTC and is due to finish in 60 minutes. The chair is ianw. Information about MeetBot at http://wiki.debian.org/MeetBot. 19:01:31 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 19:01:33 <openstack> The meeting name has been set to 'infra' 19:01:52 <ianw> #link https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting#Agenda_for_next_meeting 19:02:09 <ianw> #topic Announcements 19:02:29 * mordred waves at ianw 19:02:48 <ianw> no clarkb, no fungi today, at least 19:02:58 <ianw> #topic Actions from last meeting 19:03:11 <ianw> #link http://eavesdrop.openstack.org/meetings/infra/2019/infra.2019-08-06-19.01.html Minutes from last meeting 19:03:35 <ianw> only one action item for clarkb to collect ssh logs from gitea containers 19:03:49 <ianw> #link https://review.opendev.org/#/c/674930/ 19:03:52 <ianw> i believe that is done 19:04:20 <ianw> #topic Priority Efforts 19:04:33 <ianw> #topic Storyboard 19:05:12 <ianw> i think it is probable that anyone likely to give useful updates here is otherwise engaged 19:05:24 <corvus> i'll buy that 19:06:20 <ianw> #topic Update config management 19:06:49 <ianw> not sure there's anything particular here; we have some general topics with some cross over 19:07:47 <ianw> #topic Zuul as CD engine 19:08:54 <ianw> again, if anyone has something the floor is open, i don't think think there's been particular focus on this, this week 19:09:16 <corvus> i'm unaware of any current work 19:09:29 <ianw> #topic OpenDev 19:10:46 <ianw> slightly related; the testinfra maintainer (which we use for a lot of our verification of role actions during CI jobs) has expressed interest in getting more people onboard 19:11:15 <ianw> he is looking at moving under various ansible-ish or python-qa organisations on github 19:11:52 <ianw> i've tried to do a bit of a hard sell for opendev as a home, we'll see ... anyway he has added a few more maintainers to the existing project too 19:12:13 <ianw> we'll see 19:12:29 <corvus> ianw: thanks, that'd be cool. and even if he doesn't bite, it'd be nice to know why not 19:12:56 <ianw> #topic General topics 19:13:14 <ianw> #topic Trusty ugprade process 19:13:24 <ianw> i think the most exciting thing that falls under this is the logs work 19:13:33 <ianw> #link http://lists.zuul-ci.org/pipermail/zuul-announce/2019-July/000046.html deprecation notice 19:13:39 <corvus> oh *that's* why it's under this topic, i was wondering :) 19:13:52 <corvus> i get it now. it's "retire the trusty logs server" :) 19:14:06 <ianw> most would have noticed the new links being provided in results to zuul overview pages 19:14:51 <corvus> i haven't heard of any issues with that. it's been 24h, so i suspect we won't roll back. which i think means we can make the switch to swift itself any time. 19:15:07 <corvus> we can do that today if we want. 19:15:16 <ianw> i think it's looking great, thanks corvus 19:15:21 <corvus> (or first thing tomorrow) 19:15:58 <ianw> yeah, probably best to wait until a few more eyes than just me are on it :) 19:16:49 <corvus> how about i propose the change, and we'll see what the +2s look like :) 19:16:52 <mordred> corvus: yeah - I think making the swift switch seems safe 19:17:20 <mordred> corvus: I'm still having the chrome rendering issue, which is going to anger me enough one of these days to figure out what's happening 19:17:39 <mordred> but I don't think it's actionable enough to hold up anything - and I can always switch to log tabs if needed 19:18:04 * mordred looks forward to an insane amount of console.log statements 19:18:06 <ianw> mordred: what's the short version of the rendering issue? 19:18:15 <corvus> mordred: is it possible to run other browsers on your platform? 19:18:38 <mordred> corvus: I do not know the answer to taht question - haven't tried - it's worth a shot 19:18:39 <ianw> (just so if anyone reports it i can say "we know" :) 19:18:52 <corvus> (it would be interesting to see if chromium or even ff has the same issue) 19:19:14 <mordred> ianw: for tasks with LONG output, chrome draws the box the right size for the content length, but stops drawing the text part way through 19:19:24 <mordred> no error messages in console log 19:19:41 <mordred> further tasks exist properly 19:19:44 <corvus> and i am unable to reproduce that with chromium on bionic 19:19:50 <mordred> I expect it's going to be super weird 19:19:51 <mordred> yeah 19:20:17 <mordred> since I imagine the number of openstack devs running on chromebooks as their primary platform is probably pretty low ... I think it's safe to consider me an outlier for the moment 19:20:42 <corvus> mordred: yes, that is the reason i consider you an outlier :) 19:20:47 <ianw> ok; yeah i run pretty recent firefox from fedora 30 and haven't noticed anything, but will keep an eye 19:20:53 <mordred> corvus: I'm sure it's not the only reason 19:21:19 <corvus> it's today's reason 19:21:43 <mordred> corvus: I'm sure it's not today's only reason :) 19:22:03 <ianw> speaking of logs 19:22:18 <ianw> #link https://review.opendev.org/#/c/675226/ reduce oscwrap output in devstack 19:22:40 <ianw> i bet there's more that could be done, that cuts a thousand or so lines 19:23:01 <ianw> anyway, with logs more discoverable maybe people will work to make them more readable too 19:23:26 <corvus> ++ 19:24:30 <ianw> so we will watch for swift logs; then we can think more about the logs.o.o host i guess. i'll be happy to help there, but after a week or two of the new situation bedding in 19:24:58 <ianw> #topic AFS mirroring status 19:25:16 <ianw> i just wanted to call out the buster mirror issues 19:25:30 <ianw> fungi is working on this per 19:25:33 <ianw> #link http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2019-08-12.log.html#t2019-08-12T11:00:57 19:26:16 <corvus> we're hoping to improve the mirror roles based on that 19:26:29 <corvus> #link https://review.opendev.org/675703 mirror role 19:26:43 <corvus> but that's going to be the basis for the new replacement role 19:27:10 <corvus> it's not clear to me whether to solve the immediate issue we should roll forward with that, or we need to also do some kind of hack for the current role 19:27:55 <corvus> i think as long as we don't need the new feature yesterday, we can roll forward. writing the replacement won't be that hard. i just want to get more feedback on it, to increase the chances we've covered all the cases we're likely to hit 19:28:02 <corvus> (so we don't have to do a third version) 19:29:01 <ianw> yeah, i seemed that making a blank buster-updates repo wasn't going to work iirc 19:29:14 <ianw> which leaves "client side", as it not having it there in the first place 19:29:46 <corvus> also, we should probably take the changes fungi is proposing to that role and think about whether we need any similar changes for the rpm distros 19:30:14 <corvus> it's much more thought out than my initial "copy what we already have" attempt 19:32:14 <ianw> yeah, i can't remember anything analogous we've worked around for rpm 19:33:04 <ianw> anyway, i think this has a path forward, if anyone was wondering what's going on 19:33:19 <ianw> #topic new backup server 19:33:47 <ianw> i just wanted to call out that i've worked out the kinks with the vexxhost backup server and it all seems to be working 19:34:13 <ianw> so the last step is to turn it all on with 19:34:14 <ianw> #link https://review.opendev.org/#/c/675537 19:34:59 <ianw> i'd like some feedback there, because we discussed that we don't want the server in the regular ansible runs, and there are few ways we could do that 19:35:44 <ianw> i feel like if it's in emergency it's way to easy to forget 19:36:13 <ianw> the only host using it right now is review-dev, but i will migrate other hosts 19:37:10 <ianw> then i think i'll start a new backup02 in rax.ord and have that as a secondary host. once that's working, we can retire the original backup host (but keep it's volumes attached to the new server) 19:38:23 <ianw> #topic PTG planning 19:38:36 <ianw> #link https://etherpad.openstack.org/p/OpenDev-Shanghai-PTG-2019 19:39:11 * fungi pops onto internet to wave at friendly infra people during lunch 19:39:26 <ianw> i'm sure clarkb will have more on this next time, but for now, i'm sure if anyone wants to dump something there it will be addressed 19:40:07 <ianw> #topic Open Discussion 19:41:31 <ianw> i've been trying to get a bit more dib functional testing on arm64 working, to make sure that's staying up to date 19:41:50 <ianw> seen a couple of node allocation failures which might be a slow cloud or maybe something else, will look into a little today 19:42:40 <ianw> #link https://review.opendev.org/#/c/676120/ add ubuntu-ports to mirror script 19:42:41 <ianw> would help 19:44:41 <ianw> ok, i think that's about it then 19:45:30 <ianw> #endmeeting