| clarkb | Anyone else here for the meeting? | 19:00 |
|---|---|---|
| clarkb | I think we'll be short fungi. It will likely be a quick one | 19:00 |
| frickler | o/ | 19:00 |
| frickler | did I miss some note about fungi? | 19:00 |
| ianw | o/ | 19:01 |
| clarkb | he is attempting to have some vacation time | 19:01 |
| clarkb | #startmeeting infra | 19:01 |
| opendevmeet | Meeting started Tue Aug 16 19:01:14 2022 UTC and is due to finish in 60 minutes. The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot. | 19:01 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 19:01 |
| opendevmeet | The meeting name has been set to 'infra' | 19:01 |
| clarkb | But if you watch #opendev you might not be able to tell the difference | 19:01 |
| clarkb | #link https://lists.opendev.org/pipermail/service-discuss/2022-August/000352.html Our Agenda | 19:01 |
| clarkb | #topic Announcements | 19:01 |
| clarkb | OpenDev service coordinator nominations end today. | 19:01 |
| clarkb | I haven't seen any nominations yet. I'm taking that as a sign that everyone thinks i should do it again | 19:02 |
| clarkb | I guess if no one else says they are interested I'll go ahead and make my own nomination official later this afternnon | 19:03 |
| clarkb | #topic Topics | 19:04 |
| clarkb | #topic Improving Grafana Management Tooling | 19:05 |
| clarkb | I was going to call out https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/851955 as the last remaining change to close this out, but looks like it merged sometime between when I sent the agenda and our meeting starting | 19:05 |
| clarkb | ianw: is there anything else to call out on this subject or can we consider this compelted? | 19:05 |
| ianw | I think we can call it done, thanks | 19:06 |
| clarkb | great, thank you for putting that together | 19:07 |
| clarkb | #topic Bastion Host Updates | 19:07 |
| clarkb | This got side tracked by venv installation management iirc. Any other updates on this one? | 19:08 |
| ianw | no, sorry, i've been sidetracked away from this, but it's 2nd on my todo list | 19:09 |
| clarkb | No problem. I think those venv updates will be generally useful (as already indicated by the borg work) | 19:09 |
| clarkb | #topic Upgrading Bionic servers to Focal/Jammy | 19:09 |
| ianw | but yeah, i want to get things more isolated before we tackle the upgrade | 19:09 |
| clarkb | thats a good lead into this subject. Things all end up related to each other. | 19:09 |
| clarkb | #link https://etherpad.opendev.org/p/opendev-bionic-server-upgrades Notes on the work that needs to be done. | 19:09 |
| clarkb | In general though we're laying groundwork to make it possible to update to Jammy as well as make certain servers like bridge cleaner to update | 19:10 |
| clarkb | Did anyone else have questions/concerns/plans for upgrades that wanted to discuss? | 19:12 |
| ianw | not really for me, i guess bridge is the first one I'd like to get into | 19:12 |
| clarkb | onward then | 19:13 |
| clarkb | #topic Mailman 3 | 19:13 |
| clarkb | #link https://review.opendev.org/c/opendev/system-config/+/851248 WIP change to deploy a mailman 3 instance | 19:13 |
| clarkb | Much progress has been made with mailman 3 since our last meeting | 19:14 |
| clarkb | In particular I believe that exim is properly configured to forward mail to mailman and mailman and exim know how to send mail outbound | 19:14 |
| clarkb | A permissions issue with xapian write locations was addressed allowing hyperkitty to spin up successfully and list (empty) archives | 19:15 |
| clarkb | I've also updated the testing to force mailman's hourly cron jobs to run during CI which makes things like hyperkitty bootstrap properly | 19:15 |
| clarkb | There are now two major things to work on. The first is testing the migrations of our existing lists from mailman 2 to mailman 3. My plan is to work with fungi on that using a held CI node | 19:16 |
| clarkb | We want to make sure that both the data (subscribers and archives) migrate cleanly as well as the configuration (handling dmarc, publicly accessibly vs not, and so on) | 19:16 |
| clarkb | The other big item is dealing with uid:gid mappings between the host and the containers. Currently I'm punting on this because there doesn't seem to be a good answer for this. But the tl;dr is that mailman is uid 100 in btoh mailman-web and mailman-core and gid 100 in one of those and 65535 in the other | 19:17 |
| clarkb | that makes mapping it onto users on the host a bit painful, particularly since they both differ with gids. | 19:18 |
| clarkb | My current thinking on that is we can take the upstream images, inherit from them and change the uid:gid pair on both images to be consistent and also map them to something like 10500:10500 well out of the way of anything else on the system | 19:18 |
| clarkb | That does mean we'll be building our own images though which adds some overhead, but nothing we haven't done for other pieces of software. | 19:19 |
| clarkb | If others have better ideas or suggestions I am all ears :) | 19:19 |
| clarkb | Oh and finally there is a currently held node that you should feel free to poke at. | 19:19 |
| ianw | hrm, is the uid/gid thing a bug? | 19:20 |
| clarkb | it might be. There are actually a couple of other things I've modified locally that are probably worth filing against upstream. Worst case they tell us that they won't fix it | 19:21 |
| clarkb | Why don't I work on filing those issues today before I take any drastic steps like building our own images based on theirs | 19:21 |
| ianw | sounds sane, ... maybe just nobody has tried running them for real on the same host or something? | 19:22 |
| frickler | at least finding out if there is some hidden reasoning behind those choices would also have been my idea | 19:22 |
| clarkb | ya or they use docker volumes exclusively and don't think about the mapping | 19:22 |
| clarkb | I'll work on filing those today. The other issues are ALLOWED_HOSTS is basically useless as it doesn't get interpolated into the django config properly and the django config hardcodes assumed docker hostnames | 19:23 |
| ianw | wouldn't you still have the same issue if the volume was shared? | 19:23 |
| clarkb | ianw: I think the uid overlap means it isn't an issue | 19:24 |
| clarkb | definitely not very clean and still not good to have uid 100 in a volume mapping to _apt on the host | 19:24 |
| clarkb | one issue is if they change things on their side they may break anyone using the images :/ | 19:24 |
| clarkb | not likely to be easy to fix, but making people aware of it and double checking we aren't missing something makes it worthwhile | 19:24 |
| clarkb | Anything else related to mailman 3? I'm hapyp to answer questions if people have them | 19:26 |
| ianw | not for me, but thanks for working on it! | 19:27 |
| clarkb | #topic Gitea 1.17 | 19:27 |
| clarkb | #link https://review.opendev.org/c/opendev/system-config/+/847204 | 19:28 |
| clarkb | Reviews on that change would be appreciated, but I do think waiting for 1.17.1 to happen before we upgrade is worthwhile | 19:28 |
| clarkb | 1.17.1's milestone has quite a few bugs that seem like good things to have fixed before we upgrade | 19:28 |
| clarkb | #link https://github.com/go-gitea/gitea/milestone/122 | 19:28 |
| clarkb | I do think a lot of them don't affect us because we are disabling the new package repo system they added | 19:29 |
| clarkb | But we don't have an immediate need to update and this release seems like it could use some improving. I'm happy to wait :) | 19:30 |
| ianw | ++ | 19:30 |
| clarkb | There are a lot of breaking chagnes in the release notes that I've noted in the commit message and tried to explain how they affect us if at all | 19:31 |
| clarkb | The aerly review is worthwhile simply to get through all of that | 19:31 |
| clarkb | #topic Open Discussion | 19:33 |
| clarkb | That was everything on the agenda | 19:33 |
| clarkb | Anything else | 19:33 |
| ianw | not for me, i've been a bit side-tracked into some zuul regression testing for the console streaming bits after i broke that with the zuul restart on the weekened | 19:35 |
| clarkb | Tripleo has been having ansible ssh failures in their jobs. From what I can tell the ssh connections are from 127.0.0.1 to 127.0.0.2 as the zuul user. Journald seems to record the ssh connection succeeds but then claims the remote closes the connection and ansible says ""Failed to connect to the host via ssh: kex_exchange_identification: Connection closed by remote | 19:35 |
| clarkb | host\r\nConnection closed by 127.0.0.2 port 22", "unreachable": true" | 19:35 |
| clarkb | I know that ansible will report ssh errors when it cannot do its remote node bootstrapping due to full disks and similar problems | 19:36 |
| clarkb | I suspect this is some sort of ansible specific problem and that ssh itself is functioning, but that is still mostly a hunch at this point | 19:37 |
| ianw | well i wasted about ... too many hours ... with a similar unreachable host | 19:38 |
| clarkb | was that the fstrings thing? | 19:39 |
| ianw | "ssh-keyscan localhost -p 2022" does *not* scan the ssh running on port 2022. "ssh-keyscan -p 2022 localhost" does. if you have a ssh on port 22, the first will give you the keys for that | 19:39 |
| clarkb | oh fun | 19:39 |
| ianw | and ansible will give you very little clue about host key mismatches | 19:40 |
| clarkb | right ansible's error reporting here is I think what is hampering further debugging. I've suggested they maybe increase verbosity so help sort it out | 19:40 |
| ianw | yep then you can see the ssh calls | 19:40 |
| clarkb | Sounds like that may be everything? | 19:43 |
| ianw | nothing more from me | 19:44 |
| clarkb | Thank you everyone! We'll be back here next week at the same time and location | 19:45 |
| clarkb | #endmeeting | 19:45 |
| opendevmeet | Meeting ended Tue Aug 16 19:45:04 2022 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 19:45 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/infra/2022/infra.2022-08-16-19.01.html | 19:45 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/infra/2022/infra.2022-08-16-19.01.txt | 19:45 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/infra/2022/infra.2022-08-16-19.01.log.html | 19:45 |
| frickler | thx clarkb | 19:45 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!