| ianw | "No details about project 'ironic-discoverd' available at https://pypi.org/simple/ironic-discoverd/" | 00:53 |
|---|---|---|
| ianw | somehow i don't think we need that wheel | 00:53 |
| *** Trevor is now known as Guest9718 | 02:00 | |
| ianw | ok of the 7587 projects we build wheels for, 3906 of them have all our on-disk wheels upstream | 02:34 |
| fungi | i suspected it would be >50%, yeah | 02:37 |
| ianw | it's still adding up how much disk that is | 02:41 |
| ianw | i'll clean up this script and post it for auditing, but i'm fairly sure it's right | 02:42 |
| ianw | 24511115915 bytes to be exact, ~22gb | 02:47 |
| opendevreview | Ian Wienand proposed opendev/system-config master: tools/upstream-wheel-audit.py https://review.opendev.org/c/opendev/system-config/+/879239 | 04:18 |
| opendevreview | Ian Wienand proposed opendev/system-config master: tools/upstream-wheel-audit.py https://review.opendev.org/c/opendev/system-config/+/879239 | 04:25 |
| *** amoralej|off is now known as amoralej | 06:32 | |
| *** dasm is now known as Guest9758 | 06:52 | |
| opendevreview | Thierry Carrez proposed openstack/project-config master: Fix PTL-Approved column in RelMgt dashboard https://review.opendev.org/c/openstack/project-config/+/879319 | 09:48 |
| *** amoralej is now known as amoralej|lunch | 12:09 | |
| fungi | 22gb is not trivial | 12:43 |
| opendevreview | Thierry Carrez proposed openstack/project-config master: Fix PTL-Approved column in RelMgt dashboard https://review.opendev.org/c/openstack/project-config/+/879319 | 12:55 |
| *** amoralej|lunch is now known as amoralej | 13:07 | |
| *** Guest9758 is now known as dasm | 13:32 | |
| dpawlik | fungi, Clark[m]: Hey, just FYI: Opensearch update to 1.3 is still ongoing. When it finish I will trigger update to 2.4 or 2.5 | 14:36 |
| dpawlik | but it will take a time (probably something like 4 hours more) | 14:36 |
| dpawlik | just FYI | 14:36 |
| fungi | thanks for the heads up | 14:38 |
| dpawlik | Update to 1.3 done. Triggered upgrade to 2.5. When I come back will start logscraper and logs should be available soon :) | 15:22 |
| * dpawlik afk for awhile | 15:22 | |
| slittle1_ | hmmm, looks like github changed there ssh key. known hosts checks are failing in our zuul jobs | 15:31 |
| clarkb | slittle1_: yes they announced it publicly. Apparently they accidentally exposed the private key material which necessitated the change | 15:31 |
| slittle1_ | I think wee need a 'ssh-keygen -f "/home/zuul/.ssh/known_hosts" -R "github.com"' somewhere... just not sure if that's an admin action, or something I need to embed in all our .zuul.yaml | 15:34 |
| slittle1_ | thoughts? | 15:39 |
| clarkb | slittle1_: we don't do anything official with github so unlikely to require admin action. But you need to be more specific about what exactly is failing (have build log links?) for me to say more | 15:41 |
| slittle1_ | I'm not sure how Zuul works under the covers. I just know that we have zuul configured to push to a mirror on github on dozens of our git repos | 15:41 |
| slittle1_ | All thus push requests are now failing on the ssh key change | 15:41 |
| clarkb | ah yes openstack ran into that too. You need to update the public key material in the secret for those jobs | 15:42 |
| clarkb | let me see if I can find the openstack change | 15:42 |
| clarkb | slittle1_: https://review.opendev.org/c/openstack/project-config/+/878616/1/zuul.d/secrets.yaml you will need to do similar for your jobs | 15:42 |
| *** amoralej is now known as amoralej|off | 16:03 | |
| opendevreview | Clark Boylan proposed openstack/project-config master: Require function = NoBlock to be set on Gerrit labels https://review.opendev.org/c/openstack/project-config/+/879364 | 16:25 |
| opendevreview | Clark Boylan proposed openstack/project-config master: Fix PTL-Approved column in RelMgt dashboard https://review.opendev.org/c/openstack/project-config/+/879319 | 16:47 |
| opendevreview | Thierry Carrez proposed openstack/project-config master: Fix PTL-Approved column in RelMgt dashboard https://review.opendev.org/c/openstack/project-config/+/879319 | 16:48 |
| opendevreview | Clark Boylan proposed openstack/project-config master: Require function = NoBlock to be set on Gerrit labels https://review.opendev.org/c/openstack/project-config/+/879364 | 16:53 |
| opendevreview | Clark Boylan proposed openstack/project-config master: Require function = NoBlock to be set on Gerrit labels https://review.opendev.org/c/openstack/project-config/+/879364 | 17:07 |
| clarkb | infra-root ^ ps2 on that change removed function = NoBlock from infra-specs rollcall votes to show that it is working. I think that https://zuul.opendev.org/t/openstack/build/56fb606e751e46e89d2ef4e42bf1d1c6/log/job-output.txt#759-770 shows this working as intedned | 17:08 |
| fungi | yep, looks like it worked. thanks! | 17:12 |
| opendevreview | Merged openstack/project-config master: Fix PTL-Approved column in RelMgt dashboard https://review.opendev.org/c/openstack/project-config/+/879319 | 18:20 |
| dpawlik | fungi: yup. Logs are pushed to the Opensearch. Tomorrow will send an email | 20:14 |
| dpawlik | enjoy your day | 20:14 |
| clarkb | I've just edited the meeting agenda for tomorrow. Please take a look and add content or edit as necessary | 20:28 |
| clarkb | fungi: ianw: I'm struggling to talk to rax cinder api again :/ using all the different openstack installs I can see on bridge | 20:42 |
| clarkb | do we recall what the last state of that is/was? I'm getting 'public endpoint for volumev2 service in DFW region not found' | 20:43 |
| clarkb | this is for working on an etherpad replacement. I'm shifting to static instead since it doesn't need a volume | 20:44 |
| clarkb | and listing endpoints results in unrecognized schema errors | 20:49 |
| fungi | i think the openstackclient on the default path has too-old something or other | 20:51 |
| fungi | i can volume list with ~/fungi/foo/bin/openstack | 20:51 |
| clarkb | ya I had it working with my oldenv at one point but maybe I broke it. I'll use your env thanks! | 20:52 |
| fungi | i'm testing a reinstall in another throwaway venv now to see what's changed | 20:53 |
| fungi | you can keep using the one in foo, though that was the one i made to test installation of the launch package in system-config | 20:53 |
| fungi | okay, installed ~fungi/system-config/launch in a new venv at ~/fungi/xyzzy and it also works | 20:54 |
| clarkb | ya I thought this was sorted out so that the launch-env venv would work too. But it doesn't seem to work | 20:54 |
| fungi | time to see if the current system-config/launch is different | 20:54 |
| clarkb | using your venv I've created a new volume for etherpad02 | 20:56 |
| fungi | i created one in ~fungi/plugh installing current state of system-config/launch and it works too | 20:57 |
| fungi | so i think /usr/launcher-venv is just old and never got fixed after we got the package info in system-config sorted out? | 20:58 |
| clarkb | oh that could be | 20:58 |
| clarkb | basically we fixed it but only in CI :) | 20:58 |
| fungi | would be good to confer with ianw on it when he's around | 20:58 |
| clarkb | ++ I'm going to try to boot the etherpad02 server now setting the volume and mount paths so we'll find out if that breaks | 20:59 |
| clarkb | it may be that the subset action of attaching a volume does work | 20:59 |
| fungi | in theory we should be able to move /usr/launcher-venv aside and let the next deploy job recreate it, but i don't want to break anything if there's a god reason we haven't done that | 21:00 |
| fungi | s/god/good/ too | 21:00 |
| clarkb | and what is the story with reverse dns in rax? I know I reviewed changes around this that use some new tool to simplify things iirc | 21:08 |
| clarkb | but it isn'y clear to me if that runs automatically or not | 21:08 |
| clarkb | it looks like the launch env was able to attach the volume I created with fungi's env | 21:10 |
| ianw | umm i thought that we went for a separate openstackclient venv ... | 21:13 |
| clarkb | ianw: yes its a separate install. I'm saying that launchenv doesn't seem to work either | 21:14 |
| clarkb | (I think it may even be a docker image for `openstack` | 21:14 |
| ianw | sorry, definite oversight in the linter to not enforce function being there | 21:14 |
| opendevreview | Clark Boylan proposed opendev/zone-opendev.org master: Add static02 https://review.opendev.org/c/opendev/zone-opendev.org/+/879382 | 21:15 |
| ianw | clarkb: did you use /usr/launcher-venv/bin/launch-node ? | 21:16 |
| ianw | oh sorry, reading backwards | 21:16 |
| clarkb | ianw: yes I tried the regular install as well as that env. I think it may be specific to volume list as volume attach did work with the launcher env | 21:17 |
| ianw | right, /usr/local/bin/openstack is a link to that /usr/local/bin/openstack -> /usr/launcher-venv/bin/openstack | 21:19 |
| ianw | on the rax rdns; it should update. there's a tool in launch node rax-rdns which should hit the api and update it when on rax | 21:21 |
| clarkb | hrm dig -x isn't ersolving it yet | 21:22 |
| clarkb | maybe it didn't run for some reason? | 21:22 |
| ianw | it may also have *not* worked ... that's a possibility | 21:22 |
| ianw | i think it's, to be polite, "lightly tested" | 21:23 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Add static02 to inventory https://review.opendev.org/c/opendev/system-config/+/879383 | 21:24 |
| clarkb | ianw: I guess I should try running it manually? | 21:29 |
| opendevreview | Merged openstack/project-config master: Require function = NoBlock to be set on Gerrit labels https://review.opendev.org/c/openstack/project-config/+/879364 | 21:29 |
| clarkb | it doesn't seem to have done reverse dns for etherpad02.opendev.org either | 21:30 |
| ianw | clarkb: that would be good -- but i can take a look in a bit too if it's not going as expected. i was under the assumption it was working till just a little ago :) | 21:31 |
| fungi | to clarify, `git clone https://opendev.org/opendev/system-config;python3 -m venv plugh;plugh/bin/pip install system-config/launch` in my homedir on bridge gets me an openstackclient which can volume list in rax just fine | 21:31 |
| clarkb | ianw: ok let me see where I end up once I get sosreport installs fixed on the servers and changes up for forward dns and system-config updates | 21:32 |
| clarkb | I do have to fix sosreport on the rax jammy image too fwiw. I'm just doing a dist-upgrade to install the newer package then unattended upgrades will be clean | 21:32 |
| ianw | fungi: interesting, does that differ in pip list to /usr/launch-env in any way? | 21:34 |
| fungi | significantly | 21:34 |
| fungi | /usr/launcher-venv has comparatively somewhat old openstacksdk and python-openstackclient versions | 21:36 |
| ianw | huh, i thought we were updating daily with https://review.opendev.org/c/opendev/system-config/+/866644 | 21:38 |
| ianw | we do use that https://opendev.org/opendev/system-config/src/branch/master/playbooks/roles/install-launch-node/tasks/main.yaml#L5 | 21:39 |
| ianw | is it possible i'm wrong about --upgrade v latest: here? https://review.opendev.org/c/opendev/system-config/+/866645/6/playbooks/roles/create-venv/tasks/main.yaml | 21:40 |
| opendevreview | Clark Boylan proposed opendev/zone-opendev.org master: Add etherpad02 to DNS https://review.opendev.org/c/opendev/zone-opendev.org/+/879384 | 21:41 |
| ianw | here's what it does -> https://paste.opendev.org/show/bHLLHnCBk7adKyYeThkc/ | 21:44 |
| ianw | it upgrades pip+setuptools. not the environment. i'm now not sure what i intended | 21:45 |
| fungi | huh, yeah /usr/launcher-venv/requirements-venv.txt does not seem to reflect the contents of system-config/launch/pyproject.toml's project.dependencies list | 21:46 |
| ianw | no, that's a stamp file for the venv itself by create-venv | 21:47 |
| ianw | that updates once a day (or, if we change version pins) and makes pip/setuptools update itself | 21:47 |
| fungi | btu we don't reinstall/upgrade system-config/launch in it | 21:48 |
| fungi | s/btu/but/ | 21:48 |
| ianw | nope | 21:48 |
| ianw | i'm not sure if i intended to only upgrade pip/setuptools | 21:48 |
| ianw | we could just run "pip install --upgrade" there to freshen everything in the venv once a day? | 21:49 |
| fungi | in theory, yes | 21:49 |
| ianw | this was inspired by https://review.opendev.org/c/opendev/system-config/+/866633 | 21:49 |
| ianw | where we wanted to pull in ansible point releases, but also not pull every single bridge bootstrap run | 21:49 |
| fungi | "ERROR: You must give at least one requirement to install (see "pip help install")" | 21:50 |
| fungi | probably want something like `pip install --upgrade /home/zuul/src/opendev.org/opendev/system-config/launch` | 21:50 |
| ianw | yeah | 21:51 |
| ianw | that seems a bit hard to do generically from create-venv | 21:51 |
| ianw | i guess we want to do a similar stamp file in the launch installer, and run pip upgrade once a day | 21:51 |
| fungi | ooh, also not what i pasted, pip tries to create a temporary src/opendev_launch.egg-info | 21:51 |
| ianw | hrm it's more difficult with the launch installer, because the requirements are in pyproject.toml. so how do we trigger it to upgrade if the requirements change | 21:53 |
| fungi | and i can't easily figure out how to do it via git+https://... because the package is in a subdirectory | 21:54 |
| fungi | i suppose we could just blow it away and recreate? | 21:54 |
| ianw | https://opendev.org/opendev/system-config/src/branch/master/playbooks/roles/install-launch-node/tasks/main.yaml#L15 | 21:55 |
| fungi | pip caches everything anyway | 21:55 |
| ianw | you don't think we can put --upgrade on that? | 21:55 |
| ianw | the annoying thing is it upgrades it *every* time | 21:56 |
| ianw | unless we do something like save the hash of the current pyproject.toml, and compare it to the new one | 21:56 |
| ianw | write a stamp file with the date, and the shasum of the pyproject.toml. if either changes, run update/reinstall | 21:57 |
| fungi | "error: could not create 'src/opendev_launch.egg-info': Permission denied" | 21:57 |
| fungi | not sure where it's trying to create that | 21:57 |
| fungi | i think it may be trying to create it under /home/zuul/src/opendev.org/opendev/system-config/launch | 21:58 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Make etherpad configuration more generic for multiple hosts https://review.opendev.org/c/opendev/system-config/+/879385 | 21:59 |
| fungi | so if we run it as the zuul user it won't be able to write under /usr/launcher-venv/ but if we run it as root it will create non-zuul-owned files in /home/zuul/src/opendev.org/opendev/system-config/launch/ | 21:59 |
| ianw | fungi: or more it *has*, right? | 22:00 |
| fungi | i don't see evidence it has, maybe owing to different pip versions? | 22:01 |
| fungi | also trying with --no-build-isolation breaks on me in a different place | 22:01 |
| ianw | i guess we run it as root, from /usr/laucnher-venv/bin/pip | 22:01 |
| fungi | just testing as a non-root user (fungi), `/home/fungi/plugh/bin/pip install --upgrade file:///home/fungi/system-config/launch` works fine, but `/home/fungi/plugh/bin/pip install --upgrade file:///home/zuul/src/opendev.org/opendev/system-config/launch` fails with that write error | 22:06 |
| fungi | so it must be trying to write to zuul's homedir when i do that... because pip? | 22:07 |
| clarkb | I've updated host vars with db passwords for etherpad02 they are distinct to etherpad01 and in a separate host file for this reason. In doing this I've noticed that the etherpad.yaml group var file appears to have old puppet content in it (ssl certs etc). I'm going to clean that up then I'm also going to prune things like the ethercalc vars and so on | 22:09 |
| clarkb | infra-root ^ any concerns with me doing that? | 22:09 |
| fungi | none from me | 22:09 |
| fungi | ianw: i expect the "clean" way to do it is to have zuul run the build tool to create a wheel and then install that as root | 22:10 |
| fungi | seems like on-the-fly wheel building by one user from source owned by a different user is not something current pip/setuptools is designed for | 22:11 |
| ianw | cleanup sounds good | 22:21 |
| clarkb | ok I have removed group and host vars for ethercalc, etherpad-dev, elasticsearch, logstash-workers, subunit-workers, and pbx/asterisk | 22:22 |
| clarkb | Looking at the lists of things there is definitely more we can cleanup but I'm confident in that set | 22:22 |
| clarkb | oh also firehose | 22:22 |
| ianw | ++ | 22:23 |
| clarkb | I'll get the change up to add etherpad02 once I see my refactor of the ansible group stuff for etehrpad is passing (just to avoid thrashing things with lots of extra patchsets) | 22:27 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Add etherpad02 to inventory https://review.opendev.org/c/opendev/system-config/+/879386 | 22:33 |
| clarkb | and there it is | 22:33 |
| opendevreview | Ian Wienand proposed opendev/system-config master: [wip] upgrade launch env https://review.opendev.org/c/opendev/system-config/+/879387 | 22:33 |
| clarkb | ianw: I went ahead and added the launch env and reverse dns things to the meeting agenda under server replacements just so that we take those to some conclusion | 22:34 |
| clarkb | ianw: looking at manually running rax_rdns I'm not sure how to manually get the server href? openstack server show doens't seem to show it | 22:39 |
| * clarkb takes a break but will look closer in a bit | 22:42 | |
| clarkb | I think I see why it didn't fire. I'll work on a patch. Still not sure what the best way to fix the existing hosts is though | 22:56 |
| ianw | probably just log in and set it | 23:06 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Fix rax reverse DNS setup in launch https://review.opendev.org/c/opendev/system-config/+/879388 | 23:09 |
| clarkb | ianw: ^ I think that is the "fix" we already auto call the print dns method so it feels weird to need to rerun it by hand | 23:10 |
| clarkb | but also rerunning by hand should fail since ip4 and ip6 are undefined I think | 23:10 |
| clarkb | I fixed that too | 23:10 |
| clarkb | I think maybe we want to review and/or land that then we can run the dns script against the two servers I created to test it works? | 23:11 |
| clarkb | one thing that isn't clear to me is how dns.py connects to a cloud. It must be env vars? | 23:11 |
| clarkb | last call on meeting agenda updates. I need to get that out soon | 23:36 |
| opendevreview | Ian Wienand proposed opendev/system-config master: install-launch-node: upgrade launch env periodically https://review.opendev.org/c/opendev/system-config/+/879387 | 23:36 |
| ianw | clarkb: dns.py only prints out the dns bits for us to put into opendev zone files | 23:40 |
| clarkb | ianw: it tries to do the rax bits too https://opendev.org/opendev/system-config/src/branch/master/launch/src/opendev_launch/dns.py#L93-L110 which is currently broken due to the undefined vars (my change should fix that bit) | 23:45 |
| clarkb | ohI see your comment now | 23:46 |
| clarkb | ianw: should I move set_reverse_rax_dns() as well? | 23:46 |
| ianw | clarkb: perhaps we should, now we're thinking about it? it does feel like it's probably the wrong place in there | 23:47 |
| ianw | but i could go either way | 23:47 |
| clarkb | ok let me send out the agenda and update that change | 23:48 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: promote-container-image: add promote_container_image_method https://review.opendev.org/c/zuul/zuul-jobs/+/879009 | 23:56 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: remove-registry-tag: role to delete tags from registry https://review.opendev.org/c/zuul/zuul-jobs/+/878614 | 23:56 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: promote-container-image: use generic tag removal role https://review.opendev.org/c/zuul/zuul-jobs/+/878740 | 23:56 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: remove-registry-tag: update docker age match https://review.opendev.org/c/zuul/zuul-jobs/+/878810 | 23:56 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!