| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-rabbitmq_server master: Fail in when installing RabbitMQ/Erlang versions not possible https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/931798 | 08:01 |
|---|---|---|
| noonedeadpunk | I think we should look into adding infra caching mirrors for rabbitmq | 08:07 |
| noonedeadpunk | we're getting too much timeouts lately | 08:07 |
| jrosser | well that would not be the first time i had asked if that could happen | 08:07 |
| jrosser | but anyway :/ | 08:07 |
| noonedeadpunk | I can't recall what were reasons not to do that? | 08:07 |
| noonedeadpunk | except diskspace? | 08:07 |
| jrosser | required space iirc | 08:07 |
| jrosser | and being told just to use the distro package | 08:08 |
| noonedeadpunk | pffffff | 08:08 |
| jrosser | we just try again :) | 08:08 |
| noonedeadpunk | yeah | 08:09 |
| kleini | Still on A and trying to get hosts from Ubuntu 20.04 to 22.04: How would you migrate VMs from a compute host away? I am looking for as much automation as possible. | 09:15 |
| noonedeadpunk | nova host-evacuate-live? | 09:21 |
| noonedeadpunk | But I think you'll need to disable compute for scheduling first | 09:21 |
| noonedeadpunk | https://docs.openstack.org/nova/latest/admin/evacuate.html | 09:22 |
| noonedeadpunk | oops. it's just for offline evacuate | 09:22 |
| kleini | I found that documentation, too. But there is no documentation about evacuating a compute host, that is still alive with alive VMs. | 09:24 |
| noonedeadpunk | Well, command is still there https://docs.openstack.org/python-novaclient/latest/cli/nova.html#nova-usage | 09:25 |
| noonedeadpunk | `host-evacuate-live` | 09:25 |
| noonedeadpunk | ah, that's the link to the section: https://docs.openstack.org/python-novaclient/latest/cli/nova.html#nova-host-evacuate-live | 09:26 |
| noonedeadpunk | IIRC - that part never made it to openstack client or sdk | 09:26 |
| noonedeadpunk | I had some WIP patch for that, but never finished it | 09:26 |
| kleini | okay, then I will stick to my while read ID; do openstack server migrate --live-migration --wait $ID; done < <(openstack server list --all-projects --host computeSomething) | 09:28 |
| kleini | interesting: nova host-evacuate-live seems to utilize some faster block migration than openstack server migration --live-migration --block-migration according to network throughput in monitoring | 09:45 |
| noonedeadpunk | it should be exactly the same actually | 09:51 |
| noonedeadpunk | as after all it does almost same as you've pasted, just if refactor it to use xargs instead :D | 09:52 |
| kleini | nova host-evacuate-live works pretty well. 1 live migration running, all others queued, no more interaction except monitoring needed | 09:54 |
| opendevreview | Merged openstack/openstack-ansible-rabbitmq_server master: Update rabbitmq and erlang repo locations to the current recommended locations. https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/931753 | 10:20 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-rabbitmq_server master: Use json formatter for feature_flags fetcher https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/931903 | 11:40 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-rabbitmq_server master: Register _cluster_state as a separate variable https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/931905 | 11:45 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-rabbitmq_server master: Register _cluster_state as a separate variable https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/931905 | 11:46 |
| noonedeadpunk | I'm thinking and can't decide if https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/931801/1/playbooks/rabbitmq_server.yml should be part of the playbook or role | 11:48 |
| noonedeadpunk | as it feels it might make sense to add it to the role directly | 11:48 |
| noonedeadpunk | but then we need way bigger refactoring | 11:48 |
| noonedeadpunk | (which I guess I'm about to od now | 11:48 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-plugins master: Remove feature flags pre-upgrade step https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/931801 | 11:57 |
| andrewbonney | noonedeadpunk: that step is intentionally before 'stop rabbitmq nodes which are not the upgrader' as it doesn't work when they're down. It did always seem odd that the stop step happened in the playbook though, so if that moved to the role it would be no bad thing | 11:59 |
| noonedeadpunk | yeah. I\m jsut looking at completely refactoring how upgrades are done | 12:03 |
| noonedeadpunk | in order to make them rolling rather then blue/green or whatever we have | 12:03 |
| opendevreview | Merged openstack/openstack-ansible-os_tacker master: Ensure that first/last host detection is deterministic https://review.opendev.org/c/openstack/openstack-ansible-os_tacker/+/924640 | 12:11 |
| luvn | hi everyone! yesterday i installed openstack-ansible @ master in an ubuntu vm and everything was working fine | 12:17 |
| luvn | however, today i noticed that all openstack-related commands are returning gateway timeout (http 504) | 12:17 |
| luvn | and it looks like my / directory is pretty much full | 12:18 |
| noonedeadpunk | o/ | 12:18 |
| luvn | has anyone encountered this problem before? | 12:18 |
| noonedeadpunk | I assume that was AIO or smth? | 12:18 |
| luvn | yes, aio! | 12:18 |
| noonedeadpunk | As there're quite extensive logging | 12:18 |
| noonedeadpunk | So potentially you need to clean-up journald | 12:19 |
| noonedeadpunk | Also, we're using sparse files, which advertise more diskspace then there's actually available on system | 12:19 |
| noonedeadpunk | so if you go with real data usage on volumes or vms inside - that can go out of control | 12:19 |
| jrosser | i think that the AIO (as it's primarily a test fixture) enables debug logging on pretty mich everything | 12:20 |
| jrosser | so that will go pretty nuts with the log volumes | 12:20 |
| luvn | i was basically just testing some volumes functionalities with a cirros instance | 12:20 |
| luvn | and that's right, pretty much all containers run on debug mode | 12:20 |
| luvn | (or services) | 12:20 |
| jrosser | /var/lib/nova can get large as well | 12:20 |
| luvn | to clean up journald is it as simple as `journalctl --vacuum-size=<size>`, something like that? | 12:21 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-rabbitmq_server master: Use json formatter for feature_flags fetcher https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/931903 | 12:26 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-rabbitmq_server master: Register _cluster_state as a separate variable https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/931905 | 12:26 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-rabbitmq_server master: Register _cluster_state as a separate variable https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/931905 | 12:27 |
| noonedeadpunk | yup | 12:28 |
| noonedeadpunk | I thin kwe do have some defaults set for this though... | 12:28 |
| noonedeadpunk | https://opendev.org/openstack/openstack-ansible/src/branch/master/tests/roles/bootstrap-host/templates/user_variables.aio.yml.j2#L44-L50 | 12:29 |
| luvn | that's interesting | 12:30 |
| luvn | i freed up some space but i'm not sure it helped | 12:32 |
| luvn | is it a problem if i reboot my host with aio deployed? | 12:32 |
| noonedeadpunk | so, we had history of not recovering after reboots.... but I _think_ it's better now | 12:32 |
| mgariepy | maybe the losetup stuff won't come back | 12:33 |
| noonedeadpunk | yeah | 12:33 |
| noonedeadpunk | there were issues with ordering of losetup and lvm | 12:33 |
| noonedeadpunk | even if losetup is back, it might not be loaded as lvm devices | 12:33 |
| luvn | better not to do that then haha | 12:36 |
| mgariepy | what is consuming the disk space ? | 12:36 |
| luvn | the top consuming directories in my vm are: /openstack (64gb) and /var (23gb) | 12:38 |
| luvn | for instance | 12:38 |
| luvn | /dev/vda1 97G 95G 2.8G 98% / | 12:39 |
| mgariepy | maybe check in the /openstack dir to see which one is consuming space or if it's spread accross all dir ? | 12:42 |
| luvn | inside the /openstack the heaviest files are the *.img ones, swift{1,2,3}.img, nova.img, manila.img, cinder.img and ceph{1,2,3}.img | 12:44 |
| luvn | they are all listed as 1.0T files, but i guess it's not their real size, though? | 12:44 |
| noonedeadpunk | nah, these are sparse files | 12:44 |
| noonedeadpunk | but it's where your images and volumes will end up in | 12:45 |
| luvn | after all these *.img files the heaviest one is a logfile from the galera container (1.0G approx.) | 12:46 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-rabbitmq_server master: Move verification of cluster_state to a separate file https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/931905 | 13:08 |
| opendevreview | Merged openstack/openstack-ansible-os_octavia master: Move rpc_conn_pool_size to oslo_messaging_rabbit https://review.opendev.org/c/openstack/openstack-ansible-os_octavia/+/930469 | 13:24 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-rabbitmq_server master: Refactor rabbitmq_upgrade behavior https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/931962 | 14:55 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-plugins master: Move flags check and service stop into role https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/931801 | 14:58 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-rabbitmq_server master: Refactor rabbitmq_upgrade behavior https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/931962 | 14:59 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-rabbitmq_server master: Use json formatter for feature_flags fetcher https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/931903 | 15:04 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-rabbitmq_server master: Move verification of cluster_state to a separate file https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/931905 | 15:04 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-rabbitmq_server master: Move verification of cluster_state to a separate file https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/931905 | 15:05 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-rabbitmq_server master: Refactor rabbitmq_upgrade behavior https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/931962 | 15:05 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-rabbitmq_server master: Move verification of cluster_state to a separate file https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/931905 | 15:29 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-rabbitmq_server master: Refactor rabbitmq_upgrade behavior https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/931962 | 15:29 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-rabbitmq_server master: Refactor rabbitmq_upgrade behavior https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/931962 | 15:32 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-rabbitmq_server master: Cleanup unneeded upgrade tasks https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/931973 | 16:04 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-rabbitmq_server master: Move RabbitMQ restart to handlers https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/931974 | 16:28 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-rabbitmq_server master: Use rabbitmq_cluster_state task instead of command https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/931975 | 16:35 |
| noonedeadpunk | /o\ | 16:35 |
| noonedeadpunk | I wonder how many things I've broke... | 16:36 |
| noonedeadpunk | *broken | 16:36 |
| noonedeadpunk | And still a patch to allow force re-bootstrapping is needed | 16:36 |
| noonedeadpunk | and then potentially - 4.0 upgrade | 16:38 |
| noonedeadpunk | I tried to split that all into chunks as much as I could | 16:40 |
| jrosser | 8-O blimey | 16:50 |
| jrosser | what could possibly go wrong :) | 16:50 |
| noonedeadpunk | yeah, lol | 16:55 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!