| opendevreview | Merged opendev/system-config master: Use prod_bastion group in gate bootstrap https://review.opendev.org/c/opendev/system-config/+/864991 | 00:12 |
|---|---|---|
| *** rlandy is now known as rlandy|out | 00:25 | |
| opendevreview | Ian Wienand proposed opendev/system-config master: rax-dns-backup: fix parsing https://review.opendev.org/c/opendev/system-config/+/865083 | 00:50 |
| ianw | ^ we should probably do another cleanup purge through openstack.org and others to get rid of old stuff too | 00:53 |
| opendevreview | Ian Wienand proposed opendev/system-config master: bridge: Disable writing known_hosts files https://review.opendev.org/c/opendev/system-config/+/865092 | 04:04 |
| *** yadnesh|away is now known as yadnesh | 04:10 | |
| opendevreview | Ian Wienand proposed opendev/system-config master: bridge: Disable writing known_hosts files https://review.opendev.org/c/opendev/system-config/+/865092 | 04:30 |
| *** ysandeep|out is now known as ysandeep | 04:59 | |
| *** ysandeep is now known as ysandeep|ruck | 04:59 | |
| opendevreview | Ian Wienand proposed opendev/system-config master: launch-node : make into a small package https://review.opendev.org/c/opendev/system-config/+/861284 | 05:30 |
| StutiArya[m] | Hi, I am trying to set-up a new stack with the below commands in python 3.9.15 and Ubuntu 20.04.4 LTS. I created a stack user and cloned devstack using 'git clone https://opendev.org/openstack/devstack' and checkout yoga and granted all permissions to user with help of chmod 755. Creating the local.conf file using below configuration.... (full message at <https://matrix.org/_matrix/media/r0/download/matrix.org/xucoaZnhyzuOlxTsWPaMIfEG>) | 06:11 |
| *** yadnesh is now known as yadnesh|afk | 07:43 | |
| *** yadnesh|afk is now known as yadnesh | 08:34 | |
| *** jpena|off is now known as jpena | 08:36 | |
| *** ysandeep|ruck is now known as ysandeep|ruck|lunch | 09:46 | |
| *** pojadhav is now known as pojadhav|afk | 10:22 | |
| *** ysandeep|ruck|lunch is now known as ysandeep|ruck | 10:26 | |
| *** gthiemon1e is now known as gthiemonge | 11:00 | |
| *** rlandy|out is now known as rlandy|rover | 11:06 | |
| *** dviroel|out is now known as dviroel | 11:23 | |
| *** pojadhav- is now known as pojadhav | 11:44 | |
| *** ysandeep|ruck is now known as ysandeep|ruck|brb | 12:00 | |
| *** dviroel_ is now known as dviroel | 12:12 | |
| *** ysandeep|ruck|brb is now known as ysandeep|ruck | 12:12 | |
| fungi | StutiArya[m]: this is the channel for discussing collaboration services run as part of the opendev collaboratory, you're probably looking for the #openstack-qa channel (primary channel for the team which maintains devstack) | 12:34 |
| ysandeep|ruck | folks o/ We build tripleo containers in a content-provider job, this job is hitting transient issue during containers build- unable to resolve the mirror. Appreciate any pointers. | 13:04 |
| ysandeep|ruck | https://c6e907431413b276a63b-23a7172f5dc1e58445390ec61883d218.ssl.cf2.rackcdn.com/864994/8/gate/tripleo-ci-centos-9-content-provider/9d99392/logs/container-builds/522fec05-8e3f-4cc7-8901-45f0ca049eed/base/os/horizon/horizon-build.log | 13:05 |
| ysandeep|ruck | ~~~ | 13:05 |
| ysandeep|ruck | https://c6e907431413b276a63b-23a7172f5dc1e58445390ec61883d218.ssl.cf2.rackcdn.com/864994/8/gate/tripleo-ci-centos-9-content-provider/9d99392/logs/container-builds/522fec05-8e3f-4cc7-8901-45f0ca049eed/base/os/horizon/horizon-build.log | 13:05 |
| ysandeep|ruck | [MIRROR] python3-PyMySQL-0.10.1-6.el9.noarch.rpm: Curl error (6): Couldn't resolve host name for http://mirror.bhs1.ovh.opendev.org/centos-stream/9-stream/AppStream/x86_64/os/Packages/python3-PyMySQL-0.10.1-6.el9.noarch.rpm [Could not resolve host: mirror.bhs1.ovh.opendev.org] | 13:05 |
| ysandeep|ruck | ~~~ | 13:05 |
| ysandeep|ruck | Different mirror/cloud provider: https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_2ea/864469/1/gate/tripleo-ci-centos-9-content-provider/2ea04fe/logs/container-builds/3b86b709-7cd1-4929-bbc8-7ce5c35137a4/base/os/glance-api/glance-api-build.log | 13:05 |
| fungi | sounds like dns resolution issues. are you using the local caching resolver which runs on the loopback on our test nodes, or directly querying some remote resolver? | 13:06 |
| ysandeep|ruck | fungi, https://b61c6185702ea5ab9478-5863b3c6c7dfee81e4a8aea3295e1b03.ssl.cf2.rackcdn.com/864814/2/check/tripleo-ci-centos-9-content-provider-zed/7653d78/logs/undercloud/etc/resolv.conf .. looks like remote server | 13:07 |
| fungi | i wonder if that could be made to use the local resolver instead. we intentionally cache dns responses on the test node in order to make dns resolution a little less likely to break at random | 13:08 |
| ysandeep|ruck | Even though ubound service running, I don't see 127.0.0.1 entry in resolv.conf | 13:08 |
| ysandeep|ruck | fungi: Should we use unbound? | 13:10 |
| * ysandeep|ruck need to cross-check if the config here can works directly https://b61c6185702ea5ab9478-5863b3c6c7dfee81e4a8aea3295e1b03.ssl.cf2.rackcdn.com/864814/2/check/tripleo-ci-centos-9-content-provider-zed/7653d78/logs/undercloud/etc/unbound/index.html | 13:10 | |
| fungi | just double-checking a devstack job for comparison, and it does: https://zuul.opendev.org/t/openstack/build/f4ed66a491334120a6e3a95c3dcb983b/log/controller/logs/resolv_conf.txt | 13:12 |
| fungi | i think we normally set it up in our base job so should be inherited by all jobs unless they undo/overwrite it somehow | 13:13 |
| fungi | yeah, it's set up in the base pre playbook here: https://opendev.org/opendev/base-jobs/src/branch/master/playbooks/base/pre.yaml#L48-L55 | 13:16 |
| ysandeep|ruck | okay looks like we are running that pre in our job: https://zuul.openstack.org/build/14fb711246c542a6842edf4e04db5b03/log/job-output.txt#147-182 , maybe overriding the resolv.conf entry somewhere in our job. | 13:20 |
| *** mrunge_ is now known as mrunge | 13:24 | |
| ysandeep|ruck | fungi: thanks for the pointer, We will try local resolver if issue continues. | 13:25 |
| fungi | ysandeep|ruck: for the record, it appears we bake that resolv.conf into our diskimages here: https://opendev.org/openstack/project-config/src/branch/master/nodepool/elements/nodepool-base/finalise.d/89-boot-settings#L159 | 13:26 |
| fungi | (sorry for the delay, took a bit of digging to refresh my memory of where that happens) | 13:27 |
| fungi | rather, we bake in an rclocal script which should overwrite it at boot | 13:28 |
| fungi | ysandeep|ruck: are the issues you're seeing primarily (or only) on centos-9-stream nodes? | 13:29 |
| ysandeep|ruck | fungi: yes we use centos-9-stream nodes for tripleo test and that's where we are seeing these errors. | 13:30 |
| fungi | checking https://nb01.opendev.org/centos-9-stream-0000007862.log for the image build, it seems there should be an /etc/rc.local on those images which overwrites /etc/resolv.conf with an entry for nameserver 127.0.0. | 13:31 |
| fungi | 1 | 13:31 |
| fungi | so it might be that /etc/rc.local no longer gets executed at boot on stream 9 | 13:32 |
| fungi | the nameserver in the example you pasted is one which would have been set at boot by the cloud provider instead, making me suspicious | 13:33 |
| fungi | ysandeep|ruck: do you happen to know if stream 9 executes /etc/rc.local at all? | 13:33 |
| fungi | looks like maybe it relies on having the rc-local service enabled in systemd | 13:35 |
| ysandeep|ruck | fungi, I am not sure about that | 13:35 |
| fungi | i don't see anywhere we explicitly enable it, but ianw did write a brief essay in code comment form about it: https://opendev.org/openstack/project-config/src/branch/master/nodepool/elements/nodepool-base/finalise.d/89-boot-settings#L117-L132 | 13:37 |
| ysandeep|ruck | from the service logs, rc-local service is enabled | 13:40 |
| ysandeep|ruck | â— rc-local.service - /etc/rc.d/rc.local Compatibility | 13:40 |
| ysandeep|ruck | Loaded: loaded (/usr/lib/systemd/system/rc-local.service; enabled-runtime; vendor preset: disabled) | 13:40 |
| ysandeep|ruck | https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_14f/852485/5/check/tripleo-ci-centos-9-content-provider/14fb711/logs/undercloud/var/log/extra/services.txt | 13:40 |
| fungi | i checked a centos-9-stream node which is currently running a job for a ceilometer change and it has an /etc/resolv.conf with nameserver 127.0.0.1 | 13:46 |
| ysandeep|ruck | fungi: thanks for digging in, I will check further incase we are modifying the resolv.conf somewhere in our steps. | 13:47 |
| fungi | i checked another one which is running a job for tripleo-heat-templates and it has entries for both 127.0.0.1 and 1.1.1.1 with a comment line at the top that says "Generated by NetworkManager" | 13:47 |
| fungi | (the example from ceilometer did not have that comment nor the extra entry for 1.1.1.1) | 13:48 |
| fungi | i checked a node running a tripleo-ansible job and it just has "nameserver 127.0.0.1" at the moment | 13:49 |
| fungi | so i'm reasonably sure the nodes are starting out configured to query the local unbound cache and then something is overwriting them (either directly in your code or a service you're enabling/restarting overwrites it as a side effect?) | 13:50 |
| ysandeep|ruck | fungi: agreed, I will let you know once I found what's overwriting it. thanks again! | 13:51 |
| fungi | my pleasure | 13:52 |
| fungi | my hamfisted approach at debugging it would probably start by catting resolv.conf into logger at various points in the code and then searching through the collected syslog at the end of the build to see when it changes, which would probably help narrow it down | 13:54 |
| fungi | another possibility is that this is varying by provider, and we see different behavior in providers which rely on dhcp for network configuration vs those which rely on configdrive | 13:56 |
| fungi | but we'll need more data points to rule that out | 13:56 |
| *** dasm|off is now known as dasm | 14:48 | |
| *** ysandeep|ruck is now known as ysandeep|ruck|dinner | 15:11 | |
| Clark[m] | Note that the containers themselves (in which container images are built) may also have a different network config. This is because 127.0.0.1 in a container network namespace is different than the host and no unbound is listening there | 15:13 |
| Clark[m] | In this case because the problem is in a container (build) the host networking may be completely ignored | 15:13 |
| *** dviroel is now known as dviroel|lunch | 15:14 | |
| *** yadnesh is now known as yadnesh|away | 15:18 | |
| fungi | well, in this case it also sounds like something has reverted the node's resolv.conf contents, or prevented rc-conf from running the script to overwrite them | 15:24 |
| *** jpodivin_ is now known as jpodivin | 15:34 | |
| fungi | Clark[m]: a couple of new django warnings on the latest mailman migration test run: | 16:02 |
| fungi | ?: (2_0.W001) Your URL pattern '^$' has a route that contains '(?P<', begins with a '^', or ends with a '$'. This was likely an oversight when migrating to django.urls.path(). | 16:02 |
| fungi | ?: (2_0.W001) Your URL pattern '^admin/' has a route that contains '(?P<', begins with a '^', or ends with a '$'. This was likely an oversight when migrating to django.urls.path(). | 16:02 |
| fungi | not sure what to make of that, but maybe latest django needs a full match already? | 16:03 |
| fungi | also these warnings which i hadn't seen before either but are likely benign: | 16:05 |
| fungi | Unable to convert mailing list attribute: msg_footer with value "...some data..." | 16:05 |
| fungi | Unable to convert mailing list attribute: digest_footer with value "...some data..." | 16:06 |
| fungi | i logged the full output so will scour for more occurrences since it doesn't say why it can't import them. not sure yet if it's for every list or just a few | 16:06 |
| fungi | for the django.urls.path() warnings, looks looks like we're setting those in docker/mailman/web/mailman-web/urls.py so maybe i need to double-check that matches latest | 16:08 |
| fungi | mmm, nope, identical | 16:10 |
| fungi | matches the equivalent file in the tip of the main branch | 16:10 |
| clarkb | fungi: I'm guessing ?P< is the control code for match entire string? | 16:12 |
| fungi | no idea, our routes don't have that but they do have starting with a ^ or ending with a $ | 16:12 |
| fungi | it's warning that the entry has at least one of those three characteristics, but not what to do about it | 16:13 |
| fungi | this may be related to why the upstream docker config is still pinning to older django and mailman | 16:13 |
| *** ysandeep|ruck|dinner is now known as ysandeep|ruck | 16:13 | |
| clarkb | we're pinning the same way though? | 16:14 |
| clarkb | our dockerfile should be an exact replica | 16:14 |
| clarkb | looks like django 2.0's path method dropped regex support | 16:16 |
| clarkb | but we're on django 4.something and were on 3.something previously iirc | 16:16 |
| clarkb | you need to use re_path() for regex routes | 16:16 |
| clarkb | seems like this may be an actual upstream bug? But I would've expected it to be present regardless of the current or previous django versions | 16:17 |
| *** dviroel|lunch is now known as dviroel | 16:18 | |
| clarkb | fungi: looking at both of those routes in that file I suspect that we can replace the first one with empty string and the second with admin/. Or replace path() with re_path() | 16:21 |
| *** ysandeep|ruck is now known as ysandeep|out | 16:23 | |
| clarkb | yes I've confirmed that django is pinned <4.1. I'm not seeing where hyperkitty is explicitly installed | 16:23 |
| clarkb | from the requirements.txt and that lists pinned versions of the mailman side as well | 16:24 |
| clarkb | so I think we're pinning everything? | 16:24 |
| fungi | ahh, okay right it wasn't django i unpinned for mailman 3.3.7 it was sqlalchemy | 16:31 |
| clarkb | ya I'm like 99% certain this is an upstream bug | 16:31 |
| fungi | the update to urls.py in the latest patchset replaced a bunch of url() entries with path() but didn't alter the parameters, so maybe they should have? | 16:32 |
| fungi | if you load change 860157 and compare patchsets 14 and 15 on the urls.py file, that shows it pretty clearly | 16:33 |
| clarkb | yup, I think that is it. For ^admin/ I think we can simply drop the ^ and that matches accounts/ on the line above? Except maybe it depends on what is in admin.site.urls and whether or not it expects a regex? | 16:33 |
| fungi | oh, in fact they did change some of the entries just not all of them | 16:33 |
| fungi | url(r'^postorius/', became path(r'postorius/', for example | 16:33 |
| fungi | okay, i can push up that edit and hold another node, just want to see if there's anything else i spot in the logs which also needs fixing up while at it | 16:34 |
| clarkb | upstream mailman may have a urls file we can look at too | 16:34 |
| clarkb | let me see if I can find that really quickly before we push an update | 16:34 |
| fungi | sure, much obliged | 16:35 |
| fungi | oh, one thing worth noting... this would result in two entries for path(r'', ...) | 16:36 |
| fungi | one does a RedirectView.as_view() and the other does an include() | 16:37 |
| clarkb | fungi: https://gitlab.com/mailman/mailman-web/-/blob/master/mailman_web/urls.py | 16:37 |
| fungi | nice, thanks! | 16:37 |
| clarkb | and ya that source file has two '' entries | 16:37 |
| fungi | perfect | 16:38 |
| clarkb | fungi: assuming that makes things happy we can do a PR to sync that up in maxking's repo too | 16:38 |
| fungi | interestingly, the upstream one in mailman-web uses "mailman3" instead of "postorius" and "archives" in stead of "hyperkitty" for the paths | 16:39 |
| fungi | not sure why they're different in the docker versions | 16:39 |
| clarkb | I wonder if that is done to allow them to change the components out in the future | 16:40 |
| clarkb | might be worth raising with maxking? | 16:40 |
| fungi | oh, could be that's in order to support side-by-side installs of 2.x and 3.x | 16:40 |
| fungi | well, no because almost all 2.x deployments put "pipermail" in the archive path | 16:41 |
| fungi | i'm stumped | 16:41 |
| fungi | looks like it's been that way since the file was introduced in 2017 (commit c110bb1) | 16:43 |
| fungi | maybe upstream changed their naming convention for those in the past 5 years | 16:44 |
| clarkb | I suspect this is something that is semi configurable and you can set to your liking | 16:44 |
| fungi | indeed, 6d7de87 changed it in 2020 for mailman-web | 16:45 |
| clarkb | ianw: I +2'd https://review.opendev.org/c/opendev/system-config/+/861284 but didn't approve it due to the minor thing called out inline. I mostly wanted to make sure that was intentional before we land the change | 16:45 |
| clarkb | fungi: I suspect maxking cna't change the docker images without breaking all of the existing docker image users | 16:45 |
| clarkb | ianw: but feel free to approve or fixup or do a followup change I'm happy however that happens | 16:46 |
| fungi | clarkb: right, that's what i was thinking too. so leaves us with the dilemma: do we want to be more like the docker version's config or more like upstream mailman's? | 16:46 |
| fungi | i'm leaning toward changing it to be like upstream, since it looks like tech debt/baggage for existing docker users | 16:47 |
| fungi | i'll probably have to adjust our redirects and tests to match though | 16:47 |
| clarkb | fungi: ya I think worst case for us down the road we have apache do some redirecting to match both sets | 16:47 |
| *** marios is now known as marios|out | 16:57 | |
| clarkb | ysandeep|out: fungi: one other thing I notice after pulling up the launchpad bug is that multiple domains are failing to resolve which means it is unlikely to be a dns server issue for that domain | 17:11 |
| clarkb | (basically points to the local systems reinforcing suspcion there) | 17:11 |
| clarkb | also it looks like people are confusing where we store logs with where the jobs run https://bugs.launchpad.net/tripleo/+bug/1997202/comments/3 | 17:12 |
| fungi | yeah, i have a feeling the resolvers maintained by one or more of our donor cloud providers are overloaded/broken, but in theory more local caching on the test nodes should mostly shield us from that | 17:12 |
| clarkb | ++ | 17:13 |
| fungi | combined with the fact that we tell unbound to forward to the larger bulk resolver services, completely bypassing the resolvers run by our cloud donors | 17:14 |
| fungi | in particular, in the past we saw rackspace running an attack mitigation solution which would block abusers' ip addresses from querying the resolvers they were providing, but the blocking didn't react to those ip addresses getting recycled to other customers, so we found that some test nodes would end up being completely unable to do dns resolution through them | 17:16 |
| clarkb | I left acomment to clarify that log location isn't the same as buidl loction | 17:19 |
| fungi | thanks | 17:19 |
| clarkb | another interesting tidbit is they are hitting this in vexxhost but I don't think vexxhost is currently running any normally sized jobs in opendev | 17:22 |
| fungi | what node label? | 17:22 |
| clarkb | that makes me think the issue is less provider specific and more in their builds themslves | 17:22 |
| clarkb | fungi: oh I mean rdoproject's CI system hits this in vexxhost. But opendev shouldn't run any tripleo jobs in vexxhost currently | 17:22 |
| fungi | oh, there's something i didn't check. so the example resolv.conf file which was collected contained a rackspace resolver address, but i simply assumed that meant the job ran in rackspace... what if it was an embedded resolv.conf in the image which was lingering from the image getting built in rackspace, and something kept the rc.local script from running to replace it? | 17:24 |
| fungi | digging back to see if i can find it again | 17:24 |
| fungi | whoa. get this... | 17:26 |
| fungi | job ran in ovh-gra1 but has "nameserver 213.186.33.99" in the resolv.conf | 17:26 |
| fungi | no, nevermind, that's an ovh resolver | 17:27 |
| clarkb | that is an ovh ya | 17:27 |
| clarkb | I think we still expect that to all get replaced though | 17:27 |
| clarkb | so could be that the underlying issue is not using the local caching resolver and that happens across the providers and they are all just flaky enough to not resolve reliably | 17:27 |
| fungi | the resolv.conf also contains this comment "Generated by NetworkManager" | 17:27 |
| fungi | maybe something is restarting nm and that causes it to overwrite resolv.conf? | 17:27 |
| clarkb | given that comment that seems very possible | 17:28 |
| fungi | or maybe nm races with rc-local on centos for some reason and it's a dice roll as to which wins | 17:28 |
| clarkb | we could update our base job to dump /etc/resolv.conf contents early in jobs | 17:32 |
| clarkb | I think we should wait on tripleo's debugging before we do that though as that will affect all jobs | 17:33 |
| clarkb | (slow them down by a few seconds) | 17:33 |
| fungi | yeah, i thought we splatted it with the rest of the host info role | 17:34 |
| clarkb | currently tripleo jobs have resolv.conf recorded but only for the end of the job after we've failed | 17:34 |
| fungi | but we don't seem to | 17:34 |
| fungi | clarkb: oh! we do have it in the ansible host info record though | 17:35 |
| clarkb | oh neat /me looks | 17:35 |
| fungi | ansible_dns: nameservers: - 127.0.0.1 | 17:36 |
| *** jpena is now known as jpena|off | 17:36 | |
| clarkb | https://zuul.opendev.org/t/openstack/build/22500697c5244d31b0687057040cf1af/log/zuul-info/host-info.primary.yaml#187-189 then later we record resolv.conf without a resolver: https://zuul.opendev.org/t/openstack/build/22500697c5244d31b0687057040cf1af/log/logs/undercloud/etc/resolv.conf | 17:36 |
| fungi | so yes, at the start of the job ansible thought the dns resolver list was just 127.0.0.1 | 17:36 |
| clarkb | so ya almost certainly this is a bug in their jobs not our images | 17:36 |
| fungi | right, seems it had to have changed at some point during job runtime | 17:37 |
| clarkb | if I had to guess why that job had no resolver in it is because it is in rax where we don't do dhcp and NM is doing dhcp and failing | 17:37 |
| fungi | it was correct when the job started | 17:37 |
| clarkb | ysandeep|out: rlandy|rover ^ fyi | 17:37 |
| * rlandy|rover reads back | 17:38 | |
| clarkb | rlandy|rover: the tldr is that in a failing job we see 127.0.0.1 as a resolver in the host info file recorded at the start of the job. THen after things fail your job records resolv.conf as having no resolvers. THis indicates that it is almost certianly an issue in the build itself updating the config | 17:39 |
| rlandy|rover | clarkb: ok - we'll look into this | 17:40 |
| clarkb | this would also better explain why it happens across providers and CI systems since it isn't related to the provider or image but instead the build workload (but still no hard confirmation of this just hints towards it) | 17:40 |
| rlandy|rover | there are a few places we set this | 17:40 |
| rlandy|rover | thanks for looking into it | 17:40 |
| clarkb | rlandy|rover: ok, you might consider not setting it at all. Our CI system attempts to set it correctly for you | 17:40 |
| rlandy|rover | clarkb: yeah - the playbooks are used across various providers and zuul systems | 17:41 |
| rlandy|rover | so it could be we needed to set the value for other system | 17:41 |
| rlandy|rover | in opendev, we should skip that | 17:41 |
| clarkb | even then it is probably better to update those systems to boto with correct settings (what opendev does) and not manage it in job payload | 17:41 |
| clarkb | unless your jobs are specifically testing DNS having the system set a system appropriate setting is probably the best option | 17:41 |
| rlandy|rover | only the ipa should do that | 17:42 |
| clarkb | infra-root https://review.opendev.org/c/opendev/system-config/+/862152 has been on my backlog too long. I'd like to go ahead and approve it now as I should be able to monitor image builds using our python images this week (until the holiday anyway). Any objection to that ? | 17:58 |
| clarkb | corvus: ^ fyi this may also impact zuul and nodepool though I did my testingwith nodepool in particular and it seemed fine | 17:59 |
| corvus | clarkb: ack, thanks | 18:00 |
| fungi | clarkb: yep, i agree that would be good to get merged | 18:03 |
| clarkb | ok approved. Please call out any problems if you see them. I have no issues reverting either if that is easiest | 18:04 |
| clarkb | ianw: I'm digging through my backlog of changes and https://review.opendev.org/c/opendev/system-config/+/857239 seems relevant to the bridge work once we start upgrading ansible on bridge. Either we can abandon it by going straight to ansible 6 or we keep it because we use ansible 5. | 18:11 |
| clarkb | infra-root and for openafs https://review.opendev.org/c/opendev/system-config/+/857520 is something I wrote a while back in response to some logging we had on mirror nodes | 18:11 |
| opendevreview | Merged opendev/system-config master: Switch python-builder/python-base to pip wheel https://review.opendev.org/c/opendev/system-config/+/862152 | 18:35 |
| clarkb | fungi: were you going to push an update to the mm3 stack to update the urls file or were you just going to modify that locally on the test node you already have held? | 20:04 |
| fungi | yes, i'm just going over the import logs looking for anything else that might need fixing first | 20:05 |
| fungi | and then i'll push it up and do another hold and import test | 20:05 |
| clarkb | infra-root I've updated the meeting agenda. Please add content or remove it as appropriate. | 20:08 |
| ianw | clarkb: i was actually thinking yesterday we go straight to 6. we're running the ansible devel job which is working, so there seems to be no reason to get as close to that as possible | 20:23 |
| clarkb | ok, the main thing would be double checking our use of shebang lines in modules, but otherwise I agree shouldn't be an issue | 20:24 |
| ianw | clarkb: dropped a comment on the openafs client flags; maybe a few tests would help | 20:24 |
| clarkb | ianw: ya looks like the gitea-git-repos module at least is broken with the shebang line | 20:25 |
| clarkb | hrm ya that may only work for the initial install as written and need a restart for existing nodes | 20:26 |
| opendevreview | Ian Wienand proposed opendev/system-config master: [wip] Bump bridge ansible to 6.6.0 https://review.opendev.org/c/opendev/system-config/+/865195 | 20:27 |
| ianw | ^ i think that should trigger everything | 20:27 |
| ianw | clarkb: i think even on initial install the client will be started before it writes that out | 20:28 |
| clarkb | ianw: but we don't install the package until after that file is written? | 20:29 |
| ianw | oh sorry yes i agree. sorry i had it in my head this was in a mirror role | 20:31 |
| ianw | probably still worth a quick test infra to make sure it applies | 20:31 |
| ianw | speaking of openafs; https://review.opendev.org/c/opendev/system-config/+/864148 is a quick one to just grab the make logs from dkms in our test builds | 20:32 |
| ianw | makes it quite a bit easier to debug if the build stops working | 20:32 |
| clarkb | ianw: re ansible 6 note some modules will run with the usr/bin/env python lines because they don't depend on any deps outside of stdlib but it is still wrong. I think we should clean those up either wya | 20:32 |
| ianw | another quick one is https://review.opendev.org/c/opendev/system-config/+/864600 which adds some links to the statusbot on the homepage. that will give us a green tick on the mastodon account | 20:33 |
| ianw | clarkb: ++ i'll go through and add it | 20:33 |
| ianw | clarkb: re 861284 ... i think it's two lines between classes -- most of the other files don't have the extra space? just looked weird as i added stuff to the file | 20:35 |
| ianw | it looks like bridge01 /root/.ssh/known_hosts hasn't come back. i was kind of wondering if any part of what we do might result in it being written out, but i guess everything it needs is in the /etc/ssh/known_hosts now, which is good | 20:37 |
| ianw | i still think https://review.opendev.org/c/opendev/system-config/+/865092 to disable it is good belt-and-suspenders, but i'm not so worried we're doing anything weird now | 20:38 |
| clarkb | ianw: I thought it was two lines between any top level file entries. But I agree other files don't do that | 20:39 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Up openafs client -stat value https://review.opendev.org/c/opendev/system-config/+/857520 | 20:40 |
| ianw | ^^ not sure what thoughts are on doing that globally. for example we could add the backup known host keys to /etc to (instead of root user), and pretty much disable it everywhere. i'd have to think about review/gerrit | 20:40 |
| clarkb | something like ^ for the afs test | 20:40 |
| opendevreview | Merged opendev/system-config master: rax-dns-backup: fix parsing https://review.opendev.org/c/opendev/system-config/+/865083 | 20:46 |
| clarkb | ianw: I went ahead and approved the afs dkms logs chagne since it is straightforward. I did leave a note about being careful about deep log dirs though | 20:47 |
| fungi | out of the full import of all lists from 7 sites, only 5 mailing lists generated complaints about "Unable to convert mailing list attribute: (digest_footer|msg_footer)" | 20:48 |
| clarkb | re 865092 I'm trying to check the assertion it should never ssh to anything else | 20:50 |
| fungi | er, i take that back. lots of lists on lists.openinfra.dev did, but not any on lists.starlingx.io for example | 20:50 |
| clarkb | I guess the possibility of that would require we do things outside of the ansible inventory. SSH'ing by hand to random nodes maybe? | 20:50 |
| clarkb | fungi: maybe there is no digest_foot/msg_footer equivalent in mm3 and the old import process just didn't awrn us? | 20:51 |
| fungi | that was my first thought, but all the old lists seem to have that set so that can't be why it only complains about ome | 20:52 |
| fungi | about some | 20:52 |
| fungi | my next guess is that some of these have outdated replacement macros | 20:52 |
| fungi | so i'm comparing them | 20:53 |
| fungi | unfortunately, comparing one it complained about and one it didn't side-by-side, i don't see any difference in the field content | 20:56 |
| ianw | clarkb: yeah, that's the type of thing we *shouldn't* be keeping a cache of imo (sshing to random nodes via bridge). it just means if we write anything to make it a less-random node, we have a window to forget we added it | 21:01 |
| clarkb | fungi: does the error say "with bad replacements" or "with value" ? | 21:02 |
| clarkb | fungi: looking at the source code it seems those are the only cases and that hsould hopefully give a better indication of what is failing | 21:02 |
| clarkb | ianw: but it will also force us to accept host keys every time which might be less secure? I guess the fact that it is unlikely to happen means we should not worry about it and deal with it if it happens | 21:03 |
| ianw | true i guess, but that OTOH can also be a good indication that "this is not a host under management, so think about that" :) tomato tomato :) | 21:07 |
| fungi | clarkb: they're all "with value" | 21:10 |
| fungi | no occurrences of "bad replacements" | 21:10 |
| ianw | looks like a couple of hard failures with the ansible 6 upgrade in the gate -- which is good :) i'll debug them this morning | 21:11 |
| clarkb | if expanded_text and '%' in expanded_text <- do you possibly have % in the footer? | 21:11 |
| clarkb | It seems to use that as a flag for unreplaced values but maybe if there are actual % signs it would trip the message | 21:11 |
| fungi | yes, the expansion macros are all things like %(real_name)s and %(cgiext)s | 21:11 |
| clarkb | note in that situation it doesn't seem to continue so would proceed with writing out the file | 21:11 |
| clarkb | the other case is when decorate_template raises a KeyError which does cause it to continue and not write the file | 21:12 |
| opendevreview | Merged opendev/system-config master: openafs: copy dkms log directory https://review.opendev.org/c/opendev/system-config/+/864148 | 21:12 |
| clarkb | I think we can narrow this down by checking for output files post migration and checking for actual % in the input | 21:12 |
| fungi | the error messages, unhelpfully, seem to use the post-conversion replacement macro syntax like ${display_name} and ${listname} | 21:13 |
| clarkb | fungi: yes it is outputting the result of the conversion | 21:13 |
| clarkb | er wait not its a half conversion | 21:14 |
| clarkb | text = text.replace(oldph, newph) <- it does that before emitting it to you | 21:14 |
| fungi | and just being sure, this is when running the import21 utility that the errors seem to be emitted | 21:14 |
| clarkb | I suspect oldph and newph are things like % and $ | 21:14 |
| clarkb | ya those strings only show up in the one file in mailman-core | 21:14 |
| clarkb | but ya I would check to see if it ended up writing a file anyway (in which case % was in the output and it may be a non issue) and if not then there is a key error | 21:15 |
| clarkb | mailman/src/mailman/utilities/importer.py if cloned from https://gitlab.com/mailman/mailman | 21:15 |
| fungi | o | 21:18 |
| fungi | i'm not sure where to find the output file | 21:18 |
| fungi | also, i don't recall seeing these errors when importing with mm 3.3.6 | 21:19 |
| clarkb | dfa72b74beb2e65649b551acd0603227de41acdd this commit from april added the if % in text check | 21:20 |
| clarkb | that was not in 3.3.5 but was in 3.3.6. However 3.3.6 is from the end of october and I don't recall if we tested on it or not | 21:20 |
| clarkb | I strongly suspect that we've got extra %s in the input text and we're tripping this check and it might be fine because we write the files anyway | 21:21 |
| fungi | https://gitlab.com/mailman/mailman/-/commit/dfa72b7 | 21:22 |
| fungi | yeah, just found it myself | 21:22 |
| clarkb | fungi: var/templates/lists/<LIST_ID>/list:member:regular:footer.txt and var/templates/lists/<LIST_ID>/list:member:digest:footer.txt | 21:22 |
| fungi | so yes, i think we went from 3.3.5 to 3.3.7 | 21:22 |
| clarkb | those are the types of output dirs | 21:22 |
| fungi | /var/lib/mailman/core/var/templates/lists/ does seem to have a small subset of our total set of lists, but i see some in there which complained and some which did not | 21:24 |
| clarkb | fungi: its possible that some tripped the keyerror and some tripped this issue. If you open those files that do exist it should hopefully be clearer why those failed to convert | 21:25 |
| fungi | indeed, /var/lib/mailman/core/var/templates/lists/airship-announce.lists.airshipit.org/en/list:member:regular:footer.txt has some old % macros still unconverted in it | 21:25 |
| fungi | /var/lib/mailman/core/var/templates/lists/zuul-announce.lists.zuul-ci.org/en/list\:member\:regular\:footer.txt exists but is entirely empty | 21:26 |
| fungi | so i'll try to dissect an example | 21:27 |
| clarkb | hrm I didn't expect any completely empty file ssince it seems to only open and write the file if it doesn't continue in the loop | 21:27 |
| clarkb | and the only cas ewhere it doesn't do that with that error is when % is in the text and that would result in non empty output | 21:28 |
| fungi | so this is the mm2 version of the field for the airship-announce ml: https://paste.opendev.org/show/bS0VoH2UztFt2p3EI9S3/ | 21:29 |
| fungi | and this is what ended up in the template file on the mm3 host: https://paste.opendev.org/show/bprYIACe00uV8Ueqhv4j/ | 21:30 |
| fungi | this is what the error logged during conversion looked like: https://paste.opendev.org/show/bq3SsusO00dvqyKsA6JN/ | 21:31 |
| fungi | https://gitlab.com/mailman/mailman/-/blob/master/src/mailman/utilities/importer.py#L495 | 21:32 |
| fungi | that should be replacing the problem line with an empty string | 21:32 |
| fungi | i wonder if some of these are lacking a trailing \n | 21:33 |
| clarkb | fungi: your example input didn't have the newline at the end. I bet that is it | 21:35 |
| clarkb | And previously it didn't complain but now does since 3.3.6 | 21:35 |
| fungi | though that doesn't appear to be it, because it didn't error about openstack-announce and its msg_footer is identical | 21:38 |
| fungi | oddly, there is no /var/lib/mailman/core/var/templates/lists/openstack-announce at all though | 21:38 |
| fungi | maybe it only writes templates there under certain conditions? | 21:39 |
| fungi | and for whatever reason it didn't try to parse the msg_footer for openstack-announce? | 21:39 |
| fungi | wow, you can't search gitlab.com issues and merge requests without having an account | 21:43 |
| Clark[m] | They could be separate issues. No newline in this case. Something else for the others | 21:45 |
| fungi | well, my point is that no errors were emitted when importing the openstack-announce ml, and its msg_footer field is identical to that of airship-discuss which generated these errors | 21:47 |
| fungi | but the more i look into this, the more i think we should just blow away the offending fields. if we had done the import with 3.3.5 they wouldn't have been copied into postorius at all | 21:48 |
| fungi | 3.3.6 and later try to import them, but then break when they can't | 21:48 |
| fungi | fiddling with these configs in bulk is a pain because they're stored on disk as python pickle files | 21:53 |
| fungi | i may be better off just updating them all with the webui to remove that url line | 21:53 |
| fungi | in the meantime, i'll get the other fix pushed up and an autohold set so we can have a new one ready for another test import | 21:53 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: Fork the maxking/docker-mailman images https://review.opendev.org/c/opendev/system-config/+/860157 | 21:55 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: DNM force mm3 failure to hold the node https://review.opendev.org/c/opendev/system-config/+/855292 | 21:55 |
| *** dasm is now known as dasm|off | 22:01 | |
| clarkb | fungi: ya maybe we should reset all footers to the mm3 default | 22:03 |
| *** rlandy|rover is now known as rlandy|rover|biab | 22:07 | |
| fungi | well, some of our lists intentionally blank out the footers, so i don't want to interfere with that choice | 22:09 |
| clarkb | fungi: fwiw I think that is why openstack-announce didn't get a file. It produced a default footer which caused the importer to skip it | 22:21 |
| clarkb | which would make me think tha maybe the two inputs are different in some subtle way | 22:21 |
| fungi | well, openstack-announce had that line in its msg_footer but didn't generate an error (or even a template) | 22:30 |
| fungi | openstack-discuss has an empty string for its msg_footer, which generated an empty template file | 22:31 |
| fungi | anyway, in order not to burn too much more time on it, i'm just removing that url line from any list footers which have it, since that's what the importer wants to do anyway | 22:32 |
| clarkb | fungi: right if the resulting output of the conversion is the same as the mm3 default it doesn't write a template at all | 22:33 |
| clarkb | I'm suggesting that it seems likely that is what happened to openstack-announce and explains the lack of an mm3 file | 22:34 |
| clarkb | it also suggests there is some difference between that list and the one that generated the error | 22:34 |
| clarkb | the empty to empty conversion also makes sense since the default is non empty I guess | 22:34 |
| fungi | right, maybe it sets something else which would have caused it to need a template | 22:34 |
| fungi | problem is, how does it determine that the resulting output would be the same as the default if it can't parse the input? | 22:35 |
| clarkb | since the replacement does seem to explicitly match on a newline after that line though I strongly suspect that is at least part of the problem for the one iwth the error | 22:35 |
| clarkb | fungi: it doesn't in that case. | 22:35 |
| fungi | getUtility(ITemplateLoader).get(newvar, mlist) | 22:36 |
| fungi | i guess that's what causes it to be skipped | 22:36 |
| clarkb | https://gitlab.com/mailman/mailman/-/blob/master/src/mailman/utilities/importer.py#L551-554 that checks if the result is the same as the default and it continues otherwise | 22:37 |
| clarkb | er continues if it is the same which doesn't allow the file to be written further down in that function | 22:37 |
| fungi | er, nevermind, i was looking at the wrong loop | 22:37 |
| opendevreview | Martin Kopec proposed opendev/irc-meetings master: Update Interop meeting details https://review.opendev.org/c/opendev/irc-meetings/+/865201 | 22:37 |
| clarkb | but that strongly implies to me that there is a difference between the two list footers. Or maybe withe the available text replacements | 22:37 |
| fungi | yeah, the odd thing is that the mm2 tool for parsing and displaying the pickle contents doesn't show a trailing newline on any of them. but when i pull some of them up in the webui they have a trailing newline, while others don't | 22:39 |
| fungi | so maybe it's the config_list util in mm2 not being entirely truthful and stripping strings before outputting them | 22:40 |
| fungi | one tool thinking newlines are irrelevant and not showing them to you, another tool insisting on their presence | 22:40 |
| fungi | hooray for consistency | 22:41 |
| clarkb | it might be a valid update to the importer to check for newline or EOF | 22:41 |
| clarkb | I wonder why only that replacement gets the newline suffix | 22:41 |
| clarkb | might just be a bug | 22:42 |
| fungi | yes, i expect it is, but i wouldn't want to hold up our migration efforts waiting for that to merge, nor necessarily dirty-patch the import21 utility in our deployments in the meantime | 22:42 |
| clarkb | ya I agree, its minor enough especially if we can just add a newline to the source data and have ti be happy | 22:44 |
| *** dviroel is now known as dviroel|afk | 22:45 | |
| opendevreview | Ian Wienand proposed opendev/system-config master: [wip] Bump bridge ansible to 6.6.0 https://review.opendev.org/c/opendev/system-config/+/865195 | 22:45 |
| opendevreview | Ian Wienand proposed opendev/system-config master: borg-backup-server: build borg users betterer https://review.opendev.org/c/opendev/system-config/+/865202 | 22:45 |
| opendevreview | Ian Wienand proposed opendev/system-config master: letsencrypt-install-txt-record: build txt record list betterer https://review.opendev.org/c/opendev/system-config/+/865203 | 22:45 |
| fungi | well, i'm just removing the "%(web_page_url)slistinfo%(cgiext)s/%(_internal_name)s" line from all the msg_footer and digest_footer fields | 22:47 |
| fungi | same end result | 22:47 |
| clarkb | is it? | 22:48 |
| clarkb | oh! because the line gets replaced with empty string | 22:48 |
| clarkb | ok ya | 22:48 |
| fungi | yep | 22:49 |
| fungi | the only reason the importer tries to match that line at all is so it can remove it | 22:49 |
| fungi | so i'm just saving it the trouble | 22:50 |
| opendevreview | Ian Wienand proposed opendev/system-config master: borg-backup-server: build borg users betterer https://review.opendev.org/c/opendev/system-config/+/865202 | 23:11 |
| opendevreview | Ian Wienand proposed opendev/system-config master: letsencrypt-install-txt-record: build txt record list betterer https://review.opendev.org/c/opendev/system-config/+/865203 | 23:11 |
| opendevreview | Ian Wienand proposed opendev/system-config master: [wip] Bump bridge ansible to 6.6.0 https://review.opendev.org/c/opendev/system-config/+/865195 | 23:11 |
| opendevreview | Ian Wienand proposed opendev/system-config master: system-config-run-gitea: use standard bridge host https://review.opendev.org/c/opendev/system-config/+/865204 | 23:11 |
| opendevreview | Merged opendev/system-config master: launch-node : make into a small package https://review.opendev.org/c/opendev/system-config/+/861284 | 23:19 |
| fungi | okay, i have them all cleaned up | 23:27 |
| fungi | and 104.130.140.226 is the latest held node | 23:31 |
| fungi | it's in rax though, so i'll need to move /var/lib/mailman to the ephemeral disk | 23:32 |
| clarkb | fungi: you cleaned them up in prod? or in your test copy? | 23:34 |
| fungi | in prod, so we won't have this issue on future actual imports | 23:35 |
| fungi | similar to some of the other fields we ran into which were too large to import into the database | 23:35 |
| fungi | i'm shifting gears to the rsync from prod to the test node now | 23:36 |
| fungi | just need to make sure there's enough space for all the production data | 23:36 |
| fungi | okay, /var/lib/mailman is now the 80gb ephemeral disk | 23:39 |
| fungi | on the held node | 23:39 |
| fungi | and rsync from prod servers to the held node is now in progress | 23:43 |
| clarkb | fungi: assuming that the urls.py update makes things happier did you want to do a PR for maxking or should I work on that once confirmed? | 23:53 |
| fungi | i can probably find time for it, and we should also likely do a mr on gitlab for the importer | 23:56 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!