| *** Guest0 is now known as prometheanfire | 00:29 | |
| *** dviroel|afk is now known as dviroel|out | 01:26 | |
| *** ysandeep|rover|out is now known as ysandeep|rover | 04:50 | |
| *** ysandeep|rover is now known as ysandeep|rover|lunch | 07:40 | |
| jrosser | i made good progress on centos-9 but remember it is completely dependany on https://review.opendev.org/q/topic:osa-gluster | 08:03 |
|---|---|---|
| jrosser | and try-outs of that would be great | 08:04 |
| jrosser | the patches failing in that topic are doing so becasue we do not yet merge this https://review.opendev.org/c/openstack/openstack-ansible/+/837589 | 08:05 |
| jrosser | and i think it is circular to add another depends-on | 08:05 |
| noonedeadpunk | I will try to spawn up a sandbox during the day if everything remain calm :) | 08:15 |
| noonedeadpunk | I actually want to test out like focal/yammy/centos stack of gluster | 08:16 |
| * noonedeadpunk wonders what can possibly go wrong there | 08:16 | |
| noonedeadpunk | but first want to look through bugs... | 08:20 |
| kleini | I am testing currently a minor upgrade in staging from 23.2.0 to 23.2.2. the upgrade guide tells to use "-e rabbitmq_upgrade=true". this seems to cause to shutdown all RabbitMQ instances, which is kind of bad. OSA console log shows to stop [0] first and then [1:]. then second task does not seem to be executed, as OSA console log shows no green or yellow line under that. but all RabbitMQs are then down. | 08:22 |
| kleini | Am I doing something wrong or is there maybe some problem in the minor upgrade guide or the rabbitmq-server role? | 08:23 |
| kleini | When looking at the tasks/rabbitmq_stopped.yml, there must then be a problem with the rabbitmq_primary_cluster_node variable. Will try to check that. | 08:26 |
| admin1 | morning . | 08:33 |
| noonedeadpunk | kleini: -e rabbitmq_upgrade=true works nicely for me as used it no later then yestarday to have that said. But it could take time to shutdown fully rabit | 08:38 |
| noonedeadpunk | Though you likely don't need to have that flag for minor upgrade (at least for this upgrade) | 08:39 |
| noonedeadpunk | as we haven't changed rabbit version between these releases | 08:39 |
| jrosser | I was going to say rabbit upgrade is not normal for minor releases? | 08:39 |
| jrosser | kleini: are you looking at the major or minor upgrade docs? | 08:40 |
| noonedeadpunk | It depends I guess? I can recall when we had to bump rabbit version on stable for some OS upgrade purposes? like bionic->focal | 08:40 |
| noonedeadpunk | so minor version upgrade along with rabbit upgrade was required for that | 08:40 |
| noonedeadpunk | So I'd say it's fine to be in docs in general. And that does't hurt that much, as downtime of rabbit is usually about 30 sec for me | 08:41 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/ansible-role-systemd_networkd master: Update documentation on role requirements https://review.opendev.org/c/openstack/ansible-role-systemd_networkd/+/841691 | 08:42 |
| jrosser | noonedeadpunk: I think there is a good chance that a test of focal/jammy/centos repo servers might actually do multi-os properly with all the gluster patches applied | 08:42 |
| jrosser | maybe some small bugs but I feel like we should be in pretty good shape for mixed OS / architectures | 08:43 |
| noonedeadpunk | I'm a bit scared to mix gluster versions... | 08:43 |
| jrosser | well there is that….. | 08:43 |
| noonedeadpunk | but in theory it should work... | 08:44 |
| jrosser | kind of cool benefit from removing lsyncd really | 08:44 |
| noonedeadpunk | I think we should do milestone release once merge that | 08:45 |
| jrosser | having a small ARM repo node just for wheel builds would be great | 08:45 |
| noonedeadpunk | um, and can you build wheels on arm for x86? | 08:45 |
| jrosser | oh well just in addition to existing x86 | 08:46 |
| noonedeadpunk | yeah, as our logic won't even pick arm as destination | 08:46 |
| noonedeadpunk | (for x86) | 08:46 |
| *** ysandeep|rover|lunch is now known as ysandeep|rover | 08:49 | |
| opendevreview | Dmitriy Rabotyagov proposed openstack/ansible-role-systemd_service master: Update documentation on role requirements https://review.opendev.org/c/openstack/ansible-role-systemd_service/+/841692 | 08:50 |
| kleini | okay, will then go without that rabbitmq_upgrade then. thanks. | 08:53 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/ansible-role-systemd_mount master: Update documentation on role requirements https://review.opendev.org/c/openstack/ansible-role-systemd_mount/+/841694 | 08:54 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_gnocchi master: Add variables to manage redis url https://review.opendev.org/c/openstack/openstack-ansible-os_gnocchi/+/841699 | 09:20 |
| admin1 | help .. i can't seem to figure out this galera error during setup-infrastructure ... i have deleted and created the containers and retried the steps three times already .. https://gist.githubusercontent.com/a1git/18dd5ca261db0b860cd7ac2e87c0faaa/raw/fd2a248adf0e3c6185bc33e76b9b741b14fbf325/gistfile1.txt | 10:08 |
| admin1 | mysql is running in the 1st container | 10:10 |
| kleini | that is some haproxy python script failing with "connection refused". Can you connect from that galera container to the haproxy server? | 10:11 |
| admin1 | thanks for the hint .. | 10:13 |
| admin1 | indeed c3( the active haproxy) cannot ping the galera container on c1 | 10:13 |
| admin1 | some(dept) played with the vlans :D | 10:13 |
| admin1 | doh! .. i can ping the ips on br-mgmt from server to server, but not the lxc ips on the same bridge .. | 10:16 |
| admin1 | fixed on a reboot .. could not make sense .. no firewall, ip_forward was set to 1 .. tcpdump showed the ips coming in the interface .. but the container not responding . | 10:44 |
| admin1 | i am unblocked :) | 10:44 |
| admin1 | me and spatel will be attending the openstack summit in berlin .. anyone else from who might be coming ? | 10:45 |
| admin1 | we can have a coffee /wine time :) | 10:46 |
| *** dviroel|out is now known as dviroel | 11:00 | |
| kleini | oh, contributor tickets are cheap until Tuesday. have to buy a ticket. | 11:02 |
| admin1 | you will be there as well | 11:03 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/ansible-hardening master: Clean out SSH options we managing https://review.opendev.org/c/openstack/ansible-hardening/+/841716 | 11:04 |
| noonedeadpunk | o/ coming | 11:04 |
| admin1 | nice .. | 11:09 |
| noonedeadpunk | has anybody seen issues with DB performance after upgrade to Xena? | 12:13 |
| mgariepy | haven't upgraded to X yet. | 12:34 |
| kleini | sorry, still on W | 12:46 |
| kleini | hmm, X is the last one supporting bionic. so either X first and then focal or the other way round... | 12:48 |
| kleini | jrosser, noonedeadpunk: I tested again in staging with rabbitmq_upgrade=true. First all not upgrader RabbitMQ instances are stopped. The the upgrader instance is stopped and upgraded. After upgrade, it does not start any more. In the logs that instance is waiting for Mnesia tables. | 13:13 |
| spatel | kleini did you try to wipe Mnesia? | 13:15 |
| kleini | no, I did not. it helped to run rabbitmqctl force_boot and then another systemd restart | 13:16 |
| kleini | how do I wipe Mnesia tables? | 13:16 |
| spatel | kleini i did this mostly - rm -rf /var/lib/rabbitmq/mnesia/* | 14:15 |
| spatel | after that when you re-build rabbitMQ it will automatically create fresh database. | 14:16 |
| *** dviroel is now known as dviroel|lunch | 15:12 | |
| admin1 | if i get rabbit issues, i normally lxc-containers-destroy , create and then re-run the setup-openstack playbooks | 15:55 |
| admin1 | rabbit issues during planned upgrade i.e | 15:55 |
| *** ysandeep|rover is now known as ysandeep|out | 15:55 | |
| *** dviroel|lunch is now known as dviroel | 16:00 | |
| noonedeadpunk | NeilHanlon: do you know if it's intended not to have /etc/pam.d/passwd on Rocky? | 18:13 |
| noonedeadpunk | or smth non-default provides that? | 18:14 |
| noonedeadpunk | As Rocky the only thing that failed https://zuul.opendev.org/t/openstack/build/dfa94b7f08314901a2d0ce46bbcc09a0/log/job-output.txt#5343 | 18:14 |
| jrosser | noonedeadpunk: i think this is to fix that https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/840353 | 18:18 |
| jrosser | if only there wasnt some other brokenness there too :/ | 18:18 |
| noonedeadpunk | hm... would passwd provide pam.d plugin? interesting... | 18:18 |
| jrosser | i think mgariepy did some investigation here | 18:19 |
| jrosser | im wondering if it's an actul bug with the nfv repo, or we are just very very unlucky on that job | 18:20 |
| spatel | jrosser finally i have created playbook to create glusterfs on infra node for glance storage - https://paste.opendev.org/show/bTBvHD0zlTFfuh2GT6pG/ | 18:26 |
| jrosser | errrrm :) | 18:27 |
| jrosser | https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/837582 | 18:28 |
| spatel | :) | 18:28 |
| spatel | hmm | 18:28 |
| jrosser | we also added glusterfs support to the systemd_mount role https://review.opendev.org/c/openstack/ansible-role-systemd_mount/+/837750 | 18:30 |
| spatel | my playbook is for very specific task and i don't think anyone else would go on that route :) | 18:30 |
| damiandabrowski[m] | jrosser: I remember about Your gluster patches but didn't have enough time for this yet :/ Today I've started preparing dev environment so hopefully I'll be able to focus on this on Monday | 18:30 |
| NeilHanlon | noonedeadpunk, jrosser: yeah, passwd in EL is all backed into PAM/authselect | 18:30 |
| jrosser | damiandabrowski[m]: no problem, all good for next week | 18:31 |
| spatel | jrosser i like systemd_mount, let me take that code and apply on my play | 18:31 |
| NeilHanlon | (It's also linked with glibc 🙃) | 18:31 |
| jrosser | spatel: something like this https://review.opendev.org/c/openstack/openstack-ansible-repo_server/+/837587/8/tasks/repo_pre_install.yml | 18:32 |
| jrosser | with data a bit like this https://review.opendev.org/c/openstack/openstack-ansible-repo_server/+/837587/8/defaults/main.yml | 18:32 |
| jrosser | but you need master branch of systemd_mount role for this | 18:32 |
| jrosser | spatel: the neat thing with doing it with systemd_mount is that if somewhere else you need like NFS or something, you just change the data | 18:33 |
| jrosser | rather than having to mess with the playbook | 18:33 |
| spatel | totally! let me see how i can get it work with my playbook | 18:34 |
| spatel | currently i am using fstab but i don't like that | 18:34 |
| jrosser | oh i also made it work in containers if you ever need that, theres trouble with fuse otherwise | 18:37 |
| spatel | wish we can expand to glance also in non-ceph scenario | 18:39 |
| spatel | I am doing on host machine because its easy | 18:40 |
| jrosser | you are kind of in luck for that | 18:40 |
| jrosser | https://review.opendev.org/c/openstack/openstack-ansible-os_glance/+/837550 | 18:40 |
| jrosser | ^ this should let you use whatever systemd_mount can do with glance | 18:40 |
| jrosser | refactor to make it not specific to NFS | 18:41 |
| jrosser | so i mean that you should easily be able to make glance be a glusterfs client in the next release | 18:41 |
| jrosser | if you put gluster on the host machine that should work out fine with the right data in `glance_remote_client` | 18:42 |
| spatel | nicee! | 18:44 |
| spatel | that is neat | 18:44 |
| jrosser | btw if you look at what i did in the plugins gluster role, it also handles recovering from a completely destroyed brick | 18:45 |
| jrosser | took a wile to figure that out as theres a bunch of contradictory info out there | 18:45 |
| spatel | what do you mean destroyed brick? | 18:46 |
| jrosser | like if you re-install one of your gluster servers, how to rebuild it an get it back in the cluster | 18:46 |
| spatel | like replacing failed node ? | 18:46 |
| jrosser | yes | 18:46 |
| spatel | oh okkk | 18:46 |
| jrosser | make sure you know about how to find the UUID for each node too :) | 18:46 |
| spatel | UUID for what, sorry i didn't work on gluster in that deep :) | 18:48 |
| spatel | did you set auth etc for gluster volume? | 18:50 |
| spatel | just incase so someone else can't mount it on LAN | 18:50 |
| jrosser | no, i didnt do anything with auth, but it is on my list of things to look at | 18:52 |
| noonedeadpunk | Today I even failed to go through all recent bugs :facepalm: | 18:52 |
| noonedeadpunk | due to this weird neutron slowness.... | 18:52 |
| noonedeadpunk | I'm really wondering if it only us who face that... | 18:53 |
| spatel | is it related to CI ? | 18:53 |
| jrosser | noonedeadpunk: did you ever manage to use config_template for ini format, just with no [sections] in the file? | 19:02 |
| jrosser | flat key:value structure | 19:03 |
| noonedeadpunk | I can recall talking about it | 19:03 |
| noonedeadpunk | But seing that for rabbitmq I haven't used it, sounds like no | 19:04 |
| noonedeadpunk | but not sure if I put any time on that tbh | 19:05 |
| noonedeadpunk | from what I see it looks like we'd need to have another config format or some exceptions in place.... | 19:09 |
| noonedeadpunk | The main problem is that to parse config we're parenting ConfigParser | 19:11 |
| noonedeadpunk | Which can not parse config without sections iirc | 19:12 |
| noonedeadpunk | And google offers solutions like that https://stackoverflow.com/questions/2819696/parsing-properties-file-in-python/2819788#2819788 but I'd say we should jsut have another config type.... | 19:14 |
| noonedeadpunk | Easy workaround though - to define such config as a dict somewhere in variables, and then you can just combine overrides with default one. and in tempalte only have for loop and {{ key }} = {{ value }}.... | 19:15 |
| jrosser | right - seems like an extra type of 'properties' would be the proper name for this | 19:30 |
| *** dviroel is now known as dviroel|out | 21:57 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!