Monday, 2023-04-03

ianw"No details about project 'ironic-discoverd' available at"00:53
ianwsomehow i don't think we need that wheel00:53
*** Trevor is now known as Guest971802:00
ianwok of the 7587 projects we build wheels for, 3906 of them have all our on-disk wheels upstream02:34
fungii suspected it would be >50%, yeah02:37
ianwit's still adding up how much disk that is02:41
ianwi'll clean up this script and post it for auditing, but i'm fairly sure it's right02:42
ianw24511115915 bytes to be exact, ~22gb02:47
opendevreviewIan Wienand proposed opendev/system-config master: tools/
opendevreviewIan Wienand proposed opendev/system-config master: tools/
*** amoralej|off is now known as amoralej06:32
*** dasm is now known as Guest975806:52
opendevreviewThierry Carrez proposed openstack/project-config master: Fix PTL-Approved column in RelMgt dashboard
*** amoralej is now known as amoralej|lunch12:09
fungi22gb is not trivial12:43
opendevreviewThierry Carrez proposed openstack/project-config master: Fix PTL-Approved column in RelMgt dashboard
*** amoralej|lunch is now known as amoralej13:07
*** Guest9758 is now known as dasm13:32
dpawlikfungi, Clark[m]: Hey, just FYI: Opensearch update to 1.3 is still ongoing. When it finish I will trigger update to 2.4 or 2.514:36
dpawlikbut it will take a time (probably something like 4 hours more)14:36
dpawlikjust FYI14:36
fungithanks for the heads up14:38
dpawlikUpdate to 1.3 done. Triggered upgrade to 2.5. When I come back will start logscraper and logs should be available soon :)15:22
* dpawlik afk for awhile15:22
slittle1_hmmm, looks like github changed there ssh key.  known hosts checks are failing in our zuul jobs15:31
clarkbslittle1_: yes they announced it publicly. Apparently they accidentally exposed the private key material which necessitated the change15:31
slittle1_I think wee need a 'ssh-keygen -f "/home/zuul/.ssh/known_hosts" -R ""'  somewhere... just not sure if that's an admin action, or something I need to embed in all our .zuul.yaml15:34
clarkbslittle1_: we don't do anything official with github so unlikely to require admin action. But you need to be more specific about what exactly is failing (have build log links?) for me to say more15:41
slittle1_I'm not sure how Zuul works under the covers.  I just know that we have zuul configured to push to a mirror on github on dozens of our git repos15:41
slittle1_All thus push requests are now failing on the ssh key change15:41
clarkbah yes openstack ran into that too. You need to update the public key material in the secret for those jobs15:42
clarkblet me see if I can find the openstack change15:42
clarkbslittle1_: you will need to do similar for your jobs15:42
*** amoralej is now known as amoralej|off16:03
opendevreviewClark Boylan proposed openstack/project-config master: Require function = NoBlock to be set on Gerrit labels
opendevreviewClark Boylan proposed openstack/project-config master: Fix PTL-Approved column in RelMgt dashboard
opendevreviewThierry Carrez proposed openstack/project-config master: Fix PTL-Approved column in RelMgt dashboard
opendevreviewClark Boylan proposed openstack/project-config master: Require function = NoBlock to be set on Gerrit labels
opendevreviewClark Boylan proposed openstack/project-config master: Require function = NoBlock to be set on Gerrit labels
clarkbinfra-root ^ ps2 on that change removed function = NoBlock from infra-specs rollcall votes to show that it is working. I think that shows this working as intedned17:08
fungiyep, looks like it worked. thanks!17:12
opendevreviewMerged openstack/project-config master: Fix PTL-Approved column in RelMgt dashboard
dpawlikfungi: yup. Logs are pushed to the Opensearch. Tomorrow will send an email20:14
dpawlikenjoy your day20:14
clarkbI've just edited the meeting agenda for tomorrow. Please take a look and add content or edit as necessary20:28
clarkbfungi: ianw: I'm struggling to talk to rax cinder api again :/ using all the different openstack installs I can see on bridge20:42
clarkbdo we recall what the last state of that is/was? I'm getting 'public endpoint for volumev2 service in DFW region not found'20:43
clarkbthis is for working on an etherpad replacement. I'm shifting to static instead since it doesn't need a volume20:44
clarkband listing endpoints results in unrecognized schema errors20:49
fungii think the openstackclient on the default path has too-old something or other20:51
fungii can volume list with ~/fungi/foo/bin/openstack20:51
clarkbya I had it working with my oldenv at one point but maybe I broke it. I'll use your env thanks!20:52
fungii'm testing a reinstall in another throwaway venv now to see what's changed20:53
fungiyou can keep using the one in foo, though that was the one i made to test installation of the launch package in system-config20:53
fungiokay, installed ~fungi/system-config/launch in a new venv at ~/fungi/xyzzy and it also works20:54
clarkbya I thought this was sorted out so that the launch-env venv would work too. But it doesn't seem to work20:54
fungitime to see if the current system-config/launch is different20:54
clarkbusing your venv I've created a new volume for etherpad0220:56
fungii created one in ~fungi/plugh installing current state of system-config/launch and it works too20:57
fungiso i think /usr/launcher-venv is just old and never got fixed after we got the package info in system-config sorted out?20:58
clarkboh that could be20:58
clarkbbasically we fixed it but only in CI :)20:58
fungiwould be good to confer with ianw on it when he's around20:58
clarkb++ I'm going to try to boot the etherpad02 server now setting the volume and mount paths so we'll find out if that breaks20:59
clarkbit may be that the subset action of attaching a volume does work20:59
fungiin theory we should be able to move /usr/launcher-venv aside and let the next deploy job recreate it, but i don't want to break anything if there's a god reason we haven't done that21:00
fungis/god/good/ too21:00
clarkband what is the story with reverse dns in rax? I know I reviewed changes around this that use some new tool to simplify things iirc21:08
clarkbbut it isn'y clear to me if that runs automatically or not21:08
clarkbit looks like the launch env was able to attach the volume I created with fungi's env21:10
ianwumm i thought that we went for a separate openstackclient venv ...21:13
clarkbianw: yes its a separate install. I'm saying that launchenv doesn't seem to work either21:14
clarkb(I think it may even be a docker image for `openstack`21:14
ianwsorry, definite oversight in the linter to not enforce function being there21:14
opendevreviewClark Boylan proposed opendev/ master: Add static02
ianwclarkb: did you use /usr/launcher-venv/bin/launch-node ?21:16
ianwoh sorry, reading backwards21:16
clarkbianw: yes I tried the regular install as well as that env. I think it may be specific to volume list as volume attach did work with the launcher env21:17
ianwright, /usr/local/bin/openstack is a link to that /usr/local/bin/openstack -> /usr/launcher-venv/bin/openstack21:19
ianwon the rax rdns; it should update.  there's a tool in launch node rax-rdns which should hit the api and update it when on rax21:21
clarkbhrm dig -x isn't ersolving it yet21:22
clarkbmaybe it didn't run for some reason?21:22
ianwit may also have *not* worked ... that's a possibility21:22
ianwi think it's, to be polite, "lightly tested"21:23
opendevreviewClark Boylan proposed opendev/system-config master: Add static02 to inventory
clarkbianw: I guess I should try running it manually?21:29
opendevreviewMerged openstack/project-config master: Require function = NoBlock to be set on Gerrit labels
clarkbit doesn't seem to have done reverse dns for either21:30
ianwclarkb: that would be good -- but i can take a look in a bit too if it's not going as expected.  i was under the assumption it was working till just a little ago :)21:31
fungito clarify, `git clone;python3 -m venv plugh;plugh/bin/pip install system-config/launch` in my homedir on bridge gets me an openstackclient which can volume list in rax just fine21:31
clarkbianw: ok let me see where I end up once I get sosreport installs fixed on the servers and changes up for forward dns and system-config updates21:32
clarkbI do have to fix sosreport on the rax jammy image too fwiw. I'm just doing a dist-upgrade to install the newer package then unattended upgrades will be clean21:32
ianwfungi: interesting, does that differ in pip list to /usr/launch-env in any way?21:34
fungi/usr/launcher-venv has comparatively somewhat old openstacksdk and python-openstackclient versions21:36
ianwhuh, i thought we were updating daily with
ianwwe do use that
ianwis it possible i'm wrong about --upgrade v latest: here?
opendevreviewClark Boylan proposed opendev/ master: Add etherpad02 to DNS
ianwhere's what it does ->
ianwit upgrades pip+setuptools.  not the environment.  i'm now not sure what i intended21:45
fungihuh, yeah /usr/launcher-venv/requirements-venv.txt does not seem to reflect the contents of system-config/launch/pyproject.toml's project.dependencies list21:46
ianwno, that's a stamp file for the venv itself by create-venv21:47
ianwthat updates once a day (or, if we change version pins) and makes pip/setuptools update itself21:47
fungibtu we don't reinstall/upgrade system-config/launch in it21:48
ianwi'm not sure if i intended to only upgrade pip/setuptools21:48
ianwwe could just run "pip install --upgrade" there to freshen everything in the venv once a day?21:49
fungiin theory, yes21:49
ianwthis was inspired by
ianwwhere we wanted to pull in ansible point releases, but also not pull every single bridge bootstrap run21:49
fungi"ERROR: You must give at least one requirement to install (see "pip help install")"21:50
fungiprobably want something like `pip install --upgrade /home/zuul/src/`21:50
ianwthat seems a bit hard to do generically from create-venv21:51
ianwi guess we want to do a similar stamp file in the launch installer, and run pip upgrade once a day21:51
fungiooh, also not what i pasted, pip tries to create a temporary src/opendev_launch.egg-info21:51
ianwhrm it's more difficult with the launch installer, because the requirements are in pyproject.toml.  so how do we trigger it to upgrade if the requirements change21:53
fungiand i can't easily figure out how to do it via git+https://... because the package is in a subdirectory21:54
fungii suppose we could just blow it away and recreate?21:54
fungipip caches everything anyway21:55
ianwyou don't think we can put --upgrade on that?21:55
ianwthe annoying thing is it upgrades it *every* time21:56
ianwunless we do something like save the hash of the current pyproject.toml, and compare it to the new one21:56
ianwwrite a stamp file with the date, and the shasum of the pyproject.toml.  if either changes, run update/reinstall21:57
fungi"error: could not create 'src/opendev_launch.egg-info': Permission denied"21:57
funginot sure where it's trying to create that21:57
fungii think it may be trying to create it under /home/zuul/src/
opendevreviewClark Boylan proposed opendev/system-config master: Make etherpad configuration more generic for multiple hosts
fungiso if we run it as the zuul user it won't be able to write under /usr/launcher-venv/ but if we run it as root it will create non-zuul-owned files in /home/zuul/src/
ianwfungi: or more it *has*, right?22:00
fungii don't see evidence it has, maybe owing to different pip versions?22:01
fungialso trying with --no-build-isolation breaks on me in a different place22:01
ianwi guess we run it as root, from /usr/laucnher-venv/bin/pip22:01
fungijust testing as a non-root user (fungi), `/home/fungi/plugh/bin/pip install --upgrade file:///home/fungi/system-config/launch` works fine, but `/home/fungi/plugh/bin/pip install --upgrade file:///home/zuul/src/` fails with that write error22:06
fungiso it must be trying to write to zuul's homedir when i do that... because pip?22:07
clarkbI've updated host vars with db passwords for etherpad02 they are distinct to etherpad01 and in a separate host file for this reason. In doing this I've noticed that the etherpad.yaml group var file appears to have old puppet content in it (ssl certs etc). I'm going to clean that up then I'm also going to prune things like the ethercalc vars and so on22:09
clarkbinfra-root ^ any concerns with me doing that?22:09
funginone from me22:09
fungiianw: i expect the "clean" way to do it is to have zuul run the build tool to create a wheel and then install that as root22:10
fungiseems like on-the-fly wheel building by one user from source owned by a different user is not something current pip/setuptools is designed for22:11
ianwcleanup sounds good22:21
clarkbok I have removed group and host vars for ethercalc, etherpad-dev, elasticsearch, logstash-workers, subunit-workers, and pbx/asterisk22:22
clarkbLooking at the lists of things there is definitely more we can cleanup but I'm confident in that set22:22
clarkboh also firehose22:22
clarkbI'll get the change up to add etherpad02 once I see my refactor of the ansible group stuff for etehrpad is passing (just to avoid thrashing things with lots of extra patchsets)22:27
opendevreviewClark Boylan proposed opendev/system-config master: Add etherpad02 to inventory
clarkband there it is22:33
opendevreviewIan Wienand proposed opendev/system-config master: [wip] upgrade launch env
clarkbianw: I went ahead and added the launch env and reverse dns things to the meeting agenda under server replacements just so that we take those to some conclusion22:34
clarkbianw: looking at manually running rax_rdns I'm not sure how to manually get the server href? openstack server show doens't seem to show it22:39
* clarkb takes a break but will look closer in a bit22:42
clarkbI think I see why it didn't fire. I'll work on a patch. Still not sure what the best way to fix the existing hosts is though22:56
ianwprobably just log in and set it23:06
opendevreviewClark Boylan proposed opendev/system-config master: Fix rax reverse DNS setup in launch
clarkbianw: ^ I think that is the "fix" we already auto call the print dns method so it feels weird to need to rerun it by hand23:10
clarkbbut also rerunning by hand should fail since ip4 and ip6 are undefined I think23:10
clarkbI fixed that too23:10
clarkbI think maybe we want to review and/or land that then we can run the dns script against the two servers I created to test it works?23:11
clarkbone thing that isn't clear to me is how connects to a cloud. It must be env vars?23:11
clarkblast call on meeting agenda updates. I need to get that out soon23:36
opendevreviewIan Wienand proposed opendev/system-config master: install-launch-node: upgrade launch env periodically
ianwclarkb: only prints out the dns bits for us to put into opendev zone files23:40
clarkbianw: it tries to do the rax bits too which is currently broken due to the undefined vars (my change should fix that bit)23:45
clarkbohI see your comment now23:46
clarkbianw: should I move set_reverse_rax_dns() as well?23:46
ianwclarkb: perhaps we should, now we're thinking about it?  it does feel like it's probably the wrong place in there23:47
ianwbut i could go either way23:47
clarkbok let me send out the agenda and update that change23:48
opendevreviewIan Wienand proposed zuul/zuul-jobs master: promote-container-image: add promote_container_image_method
opendevreviewIan Wienand proposed zuul/zuul-jobs master: remove-registry-tag: role to delete tags from registry
opendevreviewIan Wienand proposed zuul/zuul-jobs master: promote-container-image: use generic tag removal role
opendevreviewIan Wienand proposed zuul/zuul-jobs master: remove-registry-tag: update docker age match

Generated by 2.17.3 by Marius Gedminas - find it at!