| noonedeadpunk | does anybody recall how we fixed `ModuleNotFoundError: No module named 'packaging'` for ceph-ansible? | 07:47 |
|---|---|---|
| noonedeadpunk | as I bet I saw that on master and we did patch that... but can't find it | 07:48 |
| noonedeadpunk | https://opendev.org/openstack/openstack-ansible-openstack_hosts/commit/434602a59e8d503bf4c0cbf47a358b1d822777aa | 07:50 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-openstack_hosts stable/2023.2: Ensure python3-packaging is installed for distros https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/928119 | 07:51 |
| jrosser | o/ morning | 08:27 |
| noonedeadpunk | o/ | 08:45 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-rabbitmq_server master: Include feature flags enablement only during upgrades https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/928124 | 09:13 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-rabbitmq_server master: Include feature flags enablement only during upgrades https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/928124 | 09:18 |
| jrosser | do we have actual breakage on the horizon role `Failed to set module mpm_event to enabled` | 09:38 |
| jrosser | i see that on two jobs here https://review.opendev.org/c/openstack/openstack-ansible-os_horizon/+/927953 | 09:38 |
| jrosser | which suggests that we're not running horizon on the metal jobs elsewhere, which might not be ideal | 09:38 |
| grauzikas | Hello, im testing OSA for some time and noticed issue when im reinstalling everything im using this steps: 1: remove instances, 2: remove all ceph pools, osd, 3: remove all lxc containers, 4: remove from all nodes all by me found packages like haproxy, keepalived, ovs, ovn, remove openstack directories and so on and on, later on compute nodes remove libvirtd and other stuff, 5: wipe drives on ceph osd nodes and so on, | 09:47 |
| grauzikas | remove /etc/openstack_deploy dir in deploy node. When reruning playbooks all the time im getting issue that OSA playbook cant start libvirtd socket service and then i need manually connect to compute nodes and restart these services and redun playbook again: https://paste.openstack.org/show/b8JnnDlhujF7JevNnJVg/ then everything works. May be in playbook is missing something? | 09:47 |
| jrosser | they playbook is not tested against anything except freshly installed systems so it is quite possible that something is not accounted for with a manually cleaned up node | 09:49 |
| jrosser | grauzikas: `"Unable to start service libvirtd.socket: Job failed. See \"journalctl -xe\" for details.\n"` <- did you do this? | 09:50 |
| jrosser | if you have some good PXEboot setup it is simpler to just make fresh nodes each time | 09:54 |
| grauzikas | https://paste.openstack.org/show/b7PIh00yuFDOQ6L7fGBO/ | 09:57 |
| grauzikas | no i dont have pxe so simply trying to remove everything what osa applyed | 10:00 |
| grauzikas | so now waiting till it reaches nova, then crashes, then restarting these services and running again :) | 10:02 |
| grauzikas | or may be for next time i will append to playbook reboot of services | 10:03 |
| *** rambo is now known as rambo2412 | 10:18 | |
| rambo2412 | Hi All This is regarding Ussuri to Victoria upgrade which I am discussing for last couple of days. I am checking each of the playbook to forecast the expected impact. now I am checking rabbitmq_install.yml playbook. | 10:20 |
| rambo2412 | I can see there are main 2 tasks (we have 3 hosts of rabbitmq) what I could interpret , first host2, host3 rabbitmq service is stopped and next upgrade is done for all three hosts sequentially starting host1. so I need to confirm if there will be any time all three rabbitmq services will be down during this time? | 10:22 |
| noonedeadpunk | yes, there will be | 10:29 |
| rambo2412 | okay thanks for confirmation. | 10:34 |
| noonedeadpunk | but it's usdually not causing too much troubles as oslo.messaging ensures reconnects to rabbitmq once it\s up | 10:45 |
| vicent | Hi! I am trying OSA 29.0.2 with the services on metal and I seem to have a problem with nova-compute not finding nova-rootwrap. https://paste.openstack.org/show/bSEP8iyuNc7PNSOADyx7/ Any idea of what could be wrong? | 11:28 |
| noonedeadpunk | hey | 11:55 |
| noonedeadpunk | darn good question | 11:56 |
| noonedeadpunk | ugh, I don't have any sandbox handy with caracal to check on that | 12:07 |
| noonedeadpunk | vicent: don't you accidentally have distro install path? | 12:07 |
| vicent | noonedeadpunk: I did install_method: source, I didn't install the distro packages. On the service unit, the path points to the venv. | 12:11 |
| vicent | # grep -i execstart /etc/systemd/system/nova-compute.service | 12:11 |
| vicent | ExecStart = /openstack/venvs/nova-29.0.2/bin/nova-compute | 12:11 |
| noonedeadpunk | ah, yes true-true | 12:11 |
| noonedeadpunk | I kind of really wonder if the issue is in not finding nova-rootwrap or not | 12:28 |
| noonedeadpunk | ah | 12:30 |
| noonedeadpunk | vicent: what `exec_dirs` you have for /etc/nova/rootwrap.conf ? | 12:30 |
| vicent | I am now reinstalling the OS, but IIRC, the nova venv was included there | 12:38 |
| noonedeadpunk | as eventually the folder is a symlink, so in case you'd manually upgrade package inside venv - it could bring in unpatched version of rootwrap.conf | 12:44 |
| noonedeadpunk | there are also other cases where such thing can happen iirc | 12:45 |
| noonedeadpunk | ie - issues in venv build | 12:45 |
| vicent | I get the result after a clean install following the deploy guide https://docs.openstack.org/project-deploy-guide/openstack-ansible/2024.1/overview.html | 13:15 |
| vicent | no upgrade at all | 13:15 |
| noonedeadpunk | huh, and no failures during nova deployment? | 13:18 |
| noonedeadpunk | also - you get that error while creating a VM, right? | 13:19 |
| vicent | noonedeadpunk: no errors on deployment. And yes, while creating a VM. Or if I add lvm on images_type, I get that on the nova logs after deployment. Similar error. | 13:20 |
| noonedeadpunk | as you're resintalling OS I assume that right now there's no possibility to check anything on the host? if so - once you'll complete redeployment, can you share if issue did re-occur? | 13:20 |
| vicent | I got the OS reinstalled. Now I am installing OSA, so I can check stuff on the host. | 13:21 |
| noonedeadpunk | ++ | 13:21 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-rabbitmq_server master: Add retries for feature flags check https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/928133 | 13:22 |
| vicent | noonedeadpunk: Same error. And the venv is on the exec_dirs | 13:41 |
| vicent | $ sudo grep exec_dirs /etc/nova/rootwrap.conf | 13:42 |
| vicent | exec_dirs = /openstack/venvs/nova-29.0.2/bin,/sbin,/usr/sbin,/bin,/usr/bin,/usr/local/bin,/usr/local/sbin | 13:42 |
| vicent | It is like the nova-compute service doesn't have the venv activated | 13:46 |
| noonedeadpunk | just in case - `sudo` is present on host? | 13:51 |
| noonedeadpunk | can you share please your OS so I could try to reproduce? | 13:51 |
| vicent | Ubuntu 22.04.4, and sudo is present | 13:52 |
| noonedeadpunk | huh | 13:52 |
| noonedeadpunk | so it's just metal installation? anything specific to keep in mind when spawning sanddbox? Like some torage driver or smth like that? | 13:53 |
| noonedeadpunk | *storage | 13:53 |
| noonedeadpunk | As we've just this week upgraded one of our regions on 22.04 to 29.0.2 | 13:54 |
| noonedeadpunk | and haven't seen anything weird so far ,except indeed things being slower then on 2023.1 | 13:55 |
| vicent | It is an ml2/ovn + sriov Openstack. I have some pci passthrough configuration. But I don't think that's relevant. I reduced the instance to not use those devices. | 13:56 |
| noonedeadpunk | yeah, as it;s failing somewhere on storage allocation | 13:56 |
| noonedeadpunk | and you're trying to use LVM, right? | 13:56 |
| noonedeadpunk | or? | 13:56 |
| noonedeadpunk | in terms of `images_type`? | 13:57 |
| vicent | LVM was also failing, so I just increased the root fs size. Single node. https://github.com/openstack/openstack-ansible/blob/master/etc/openstack_deploy/env.d/aio_metal.yml.example | 13:57 |
| vicent | I have this | 13:58 |
| noonedeadpunk | oh, so that is aio as well? o_O | 13:58 |
| vicent | and no_containers: true on the host | 13:58 |
| vicent | No, I didn't follow the AIO, just that conf to set the services to metal | 13:59 |
| noonedeadpunk | ++ | 13:59 |
| vicent | Should I try 29.0.1? | 14:03 |
| noonedeadpunk | that should not actually matter much (I guess) | 14:04 |
| noonedeadpunk | I think 29.0.2 includes a CVE fix though | 14:05 |
| noonedeadpunk | https://security.openstack.org/ossa/OSSA-2024-002.html | 14:05 |
| * noonedeadpunk spawning a sandbox | 14:08 | |
| noonedeadpunk | vicent: hm, do you have failure of VM creation when you catch the error? | 14:10 |
| noonedeadpunk | As according to comment in nova code - it might be "expected" failure for $reasons: https://opendev.org/openstack/nova/src/commit/cd4e58173a1533878eccc6efabbda0560dfde613/nova/virt/libvirt/imagebackend.py#L57-L80 | 14:11 |
| noonedeadpunk | though it's not an OSError.... | 14:12 |
| noonedeadpunk | but still `FailedToDropPrivileges` | 14:12 |
| noonedeadpunk | vicent: just in case to verify - /etc/sudoers.d/nova_sudoers do have correct path to nova as well? | 14:19 |
| vicent | Yes, the nova_sudoers have the correct file | 14:21 |
| vicent | It seems like the nova service is not running on the virtualenv. | 14:21 |
| noonedeadpunk | well according to paste it does | 14:22 |
| noonedeadpunk | as stack trace is totally from inside of venv | 14:22 |
| noonedeadpunk | oh | 14:22 |
| vicent | If I run the command that fails manually on the virtualenv, it works fine | 14:22 |
| noonedeadpunk | I think there should be smth more for service to look for binaries in expected folder | 14:22 |
| noonedeadpunk | not for me actually | 14:23 |
| noonedeadpunk | actually I have weird output | 14:24 |
| noonedeadpunk | https://paste.openstack.org/show/bXZTKy3TOgXuXk8UiJ2V/ | 14:24 |
| noonedeadpunk | ah, I should change user... | 14:26 |
| vicent | $ sudo -u nova bash -c '. /openstack/venvs/nova-29.0.2/bin/activate ; echo $PATH' | 14:26 |
| vicent | /openstack/venvs/nova-29.0.2/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games | 14:26 |
| noonedeadpunk | yeah | 14:26 |
| vicent | Where do you see on the paste that the service is running on the virtualenv? | 14:27 |
| noonedeadpunk | so one possibility is that this command is not run as `nova` user | 14:27 |
| noonedeadpunk | in stack trace you have like `/openstack/venvs/nova-29.0.2/lib/python3.10/site-packages/nova/compute/manager.py` | 14:28 |
| vicent | Works fine: | 14:28 |
| vicent | $ sudo bash -c '. /openstack/venvs/nova-29.0.2/bin/activate ; echo $PATH' | 14:28 |
| vicent | /openstack/venvs/nova-29.0.2/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games | 14:28 |
| noonedeadpunk | vicent: it did not work for me, because the venv path was not in secure_path for sudo | 14:29 |
| noonedeadpunk | so if I run as root I get `command not found` | 14:29 |
| noonedeadpunk | but as nova user it works | 14:29 |
| vicent | $ grep -i execstart /etc/systemd/system/nova-compute.service | 14:29 |
| vicent | ExecStart = /openstack/venvs/nova-29.0.2/bin/nova-compute | 14:29 |
| vicent | This doesn't mean that the venv is activated AFAIK | 14:30 |
| vicent | right? | 14:30 |
| noonedeadpunk | it should not be activated I assume | 14:30 |
| noonedeadpunk | as you have `Defaults:nova secure_path="/openstack/venvs/nova-28.1.0.dev87/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"` | 14:30 |
| noonedeadpunk | which should adjust PATH for the nova user if sudo is involve | 14:31 |
| noonedeadpunk | (in /etc/sudoers.d/nova_sudoers) | 14:31 |
| vicent | but that is not enought to modify PATH env: | 14:31 |
| vicent | $ sudo -u nova bash -c 'echo $PATH' | 14:31 |
| vicent | /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games | 14:31 |
| noonedeadpunk | but can you try setting that secure_path globally in sudoers? I really wonder if it's that nova does not run as "nova" somehow.... | 14:31 |
| noonedeadpunk | vicent: this command is very different | 14:32 |
| vicent | The service unit explicitly runs as nova | 14:32 |
| vicent | I can see it on the process list | 14:32 |
| noonedeadpunk | secure_path - Path used for every command run from sudo. If you don't trust the people running sudo to have a sane PATH environment variable you may want to use this. Another use is if you want to have the “root path” be separate from the “user path”. | 14:33 |
| noonedeadpunk | so if secure_path is set - once you try to use `sudo` to run command - sudo will ignore PATH and look inside secure_path | 14:33 |
| noonedeadpunk | ideally | 14:33 |
| vicent | aha, I was missing that knowledge | 14:34 |
| noonedeadpunk | jsut try to `su nova; sudo nova-rootwrap` | 14:34 |
| noonedeadpunk | and it should find the binary | 14:35 |
| vicent | yeah, failing: "sudo: nova-rootwrap: command not found" | 14:35 |
| noonedeadpunk | oh | 14:35 |
| noonedeadpunk | it works for me | 14:35 |
| noonedeadpunk | even without activating venv | 14:35 |
| noonedeadpunk | ok, that's interesting | 14:35 |
| noonedeadpunk | are you sure you don't have any issues in sudoers files ?:) | 14:36 |
| noonedeadpunk | visudo -c ? | 14:36 |
| vicent | Looks fine to me: https://paste.openstack.org/show/bnQwhBfuyMaNtPHXKYBL/ | 14:36 |
| noonedeadpunk | I wonder if you for some reason don't have include if /etc/sudoers.d/ | 14:37 |
| noonedeadpunk | ie I have that https://paste.openstack.org/show/b4at0HPTbJ0kzAsaauR7/ | 14:37 |
| noonedeadpunk | and `@includedir /etc/sudoers.d` is last line in /etc/sudoers | 14:38 |
| vicent | Yeah! That's it! I don't have that include. Probably my organization removed that :S | 14:39 |
| noonedeadpunk | we had same issue when just `template` /etc/sudoers so random crap... | 14:39 |
| noonedeadpunk | ok, cool, revived nmy rusty memories of how rootwrap works :D | 14:40 |
| noonedeadpunk | though not it being present after os re-setup is weird | 14:40 |
| vicent | I think there is some pxe magic and customizations in my lab | 14:41 |
| noonedeadpunk | ah | 14:41 |
| noonedeadpunk | might be that we should ensure presence of includes somewhere in openstack_hosts... | 14:43 |
| vicent | nova-rootwrap manually works now | 14:43 |
| vicent | And the vm gets created! | 14:44 |
| vicent | Many thanks noonedeadpunk! | 14:44 |
| vicent | I think devstack makes sure the include is there. I have another maching with devstack and I could see it. | 14:55 |
| fungi | warehouse (pypi) is removing some expensive xmlrpc api methods: https://mail.python.org/archives/list/pypi-announce@python.org/message/5VOX33ARFQUYKIMKM5NS7PM7Z6ZNCSJY/ | 15:18 |
| fungi | the only match i found in codesearch was this: https://opendev.org/openstack/openstack-ansible/src/branch/master/scripts/get-pypi-pkg-version.py#L35 | 15:19 |
| fungi | someone might want to rethink that routine if it's of critical importance | 15:19 |
| *** rambo is now known as Guest2636 | 20:10 | |
| *** jonher_ is now known as jonher | 22:34 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!