| noonedeadpunk | If it will happen - I will be there as well (at least I aim to be there :)) | 07:47 |
|---|---|---|
| noonedeadpunk | well we probably can ensure ca-certs are latest with openstack_hosts role and install them separately from other packages. | 07:49 |
| jrosser | lxc_hosts role already deals with ca-certificates for containers | 10:56 |
| jrosser | theres also going to be present vs. latest on the apt module to consider | 10:56 |
| noonedeadpunk | but in case of lxc_hosts I think it can only build new container image? Or manage only hosts? | 10:57 |
| damiandabrowski[m] | ca-certificates are listed in `lxc_cache_distro_packages` which is used in `lxc_hosts/templates/prep-scripts/ubuntu_20_prep.sh.j2` | 11:00 |
| damiandabrowski[m] | so IMO it won't upgrade this package during openstack upgrade | 11:00 |
| noonedeadpunk | yeah so I guess you need to re-create containers to get it updated. | 11:07 |
| noonedeadpunk | eventually re-creating containers is possible but probably overkill. However, ad-hoc might be really best thing to do here, dunno if we should do anything | 11:08 |
| noonedeadpunk | but if look wider, on all packages update during openstack upgrade, we might need to set some variable explicitly while running upgrade to adjust package state | 11:09 |
| noonedeadpunk | and we already have package_state for that | 11:09 |
| noonedeadpunk | so it's more how we overwrite it only during upgrade... | 11:11 |
| noonedeadpunk | eventually documenting that is the way as well. Ie run - setup-openstack -e package_state=latest | 11:11 |
| noonedeadpunk | or setup-hosts -e package_state=latest | 11:11 |
| jrosser | i think that ad-hoc command is a good approach | 11:56 |
| jrosser | state=latest can have some unexpected side effects, like new point releases of pretty much everything might restart services you don't expect | 11:57 |
| noonedeadpunk | yeah, but I meant that during openstack upgrade it's good to include that command? | 12:10 |
| noonedeadpunk | which we don't do right now | 12:11 |
| noonedeadpunk | might be including minor upgrades as well? | 12:11 |
| noonedeadpunk | because it's not obvious that it won't be done. But maybe just documenting this would be enough | 12:11 |
| mgariepy | noonedeadpunk, https://paste.opendev.org/show/811164/ | 13:38 |
| mgariepy | jrosser, ^^ | 13:38 |
| mgariepy | i think we do try to run mysql_upgrade for no really good reason .. :/ | 13:39 |
| jrosser | do you need to attach gdb and extract the stack trace so see where it is stuck? | 13:42 |
| mgariepy | since it's already running from the | 13:42 |
| jrosser | sounds like the core file on it's own is not enough, from the jira ticket | 13:42 |
| mgariepy | yep that sure. | 13:43 |
| mgariepy | i want to debug a bit more. | 13:43 |
| mgariepy | we do run the mariadb-upgrade in a task but this is already done by systemd. | 13:43 |
| admin1 | osa+octavia .. 23.1.2 branch ... octaiva_key . .. where is the private_key stored ? | 13:47 |
| admin1 | or are we supposed to delete this key, and then recreate one with the same name and save the private key | 13:47 |
| noonedeadpunk | mgariepy: oh hm | 13:49 |
| noonedeadpunk | actually I wonder why debian-start runs mysql_upgrade while it should be a symlink to mariadb-upgrde from 10.6 iirc | 13:49 |
| mgariepy | if I stop mariadb, and start it it does comes back | 13:49 |
| mgariepy | it's symlink | 13:49 |
| noonedeadpunk | so debian-start just not got updated for the release? | 13:50 |
| noonedeadpunk | well, it'a packaging issue though, non-critical one btw | 13:50 |
| mgariepy | it's all symlink in fact. /etc/default/mysql -> /etc/default/mariadb | 13:50 |
| jrosser | admin1: i think the key is put on your utility host | 13:50 |
| mgariepy | and /usr/bin/mysql_upgrade -> mariadb-upgrade | 13:51 |
| noonedeadpunk | mgariepy: have you checked what spawns debian-start actually? | 13:51 |
| noonedeadpunk | hm https://opendev.org/openstack/openstack-ansible-galera_server/src/branch/master/templates/debian.cnf.j2#L14 | 13:52 |
| mgariepy | yep, it does run the upgrade with "/usr/bin/mysql_upgrade --defaults-extra-file=/etc/mysql/debian.cnf --version-check"" | 13:52 |
| mgariepy | https://paste.openstack.org/show/811165/ | 13:52 |
| mgariepy | https://paste.opendev.org/show/811164/ err | 13:53 |
| mgariepy | arf. | 13:53 |
| mgariepy | i wish i could delete stuff lol | 13:53 |
| mgariepy | 65 is the debian-start. | 13:54 |
| noonedeadpunk | aha ExecStartPost=/etc/mysql/debian-start | 13:54 |
| noonedeadpunk | so yes, we hit race condition | 13:55 |
| noonedeadpunk | and yes, we don't need to run that all times | 13:55 |
| mgariepy | let me try something.. | 13:55 |
| noonedeadpunk | I wonder if centos has smth common | 13:55 |
| mgariepy | with --version-check is doesn't fully run | 13:55 |
| mgariepy | and it seems to do the correct thing | 13:55 |
| mgariepy | it's race condition .. | 13:58 |
| noonedeadpunk | well, we for sure should not always run mysql_upgrade | 13:59 |
| noonedeadpunk | and I'm not sure we should run it at all... | 13:59 |
| mgariepy | and we shoud wait for mysql_upgrade to finish before adding users.. | 13:59 |
| noonedeadpunk | oh, well, yes... | 13:59 |
| noonedeadpunk | I wonder if we can somehow track systemd unit status... | 14:00 |
| mgariepy | rerunning another time to see if it's the issue.. | 14:00 |
| noonedeadpunk | let me also spawn centos8 to see how things are done there... | 14:00 |
| mgariepy | https://paste.opendev.org/show/811166/ | 14:01 |
| mgariepy | racydebug lol | 14:01 |
| mgariepy | journactl is awesome | 14:03 |
| mgariepy | you can follow the unit before it's created :D | 14:04 |
| noonedeadpunk | oh, yes, that's actually very nice thing! | 14:04 |
| mgariepy | with my super patch with sleep .. it works all the time. before that it was failing. | 14:05 |
| mgariepy | all the time :D | 14:05 |
| mgariepy | well .. mysql shouldn't let us add users if it lock the DB. | 14:08 |
| mgariepy | after.. | 14:08 |
| admin1 | jrosser, thanks .. got it | 14:09 |
| jrosser | admin1: this actually may be a bug that they end up there, really they should be placed on the deployment host | 14:09 |
| noonedeadpunk | I can recall fixing this tbh... | 14:10 |
| noonedeadpunk | ah, no :( | 14:11 |
| noonedeadpunk | https://opendev.org/openstack/openstack-ansible-os_octavia/src/branch/master/defaults/main.yml#L258 | 14:11 |
| noonedeadpunk | so I fixed it only locally I believe.... | 14:11 |
| mgariepy | if i remove the mariadb-upgrade it does fix the issue.. | 14:17 |
| mgariepy | the user get created.. and the mariadb-upgrade finish at some point. | 14:21 |
| mgariepy | i'll spawn a centos8-stream to see what it does | 14:24 |
| noonedeadpunk | ah, I laready did spawned | 14:40 |
| mgariepy | lol | 14:40 |
| mgariepy | so what does it do ? | 14:40 |
| noonedeadpunk | trying to check out atm ) | 14:42 |
| mgariepy | centos is so much fun... not having the usr/local/bin in PATH.. | 14:43 |
| noonedeadpunk | well I have other thing I stuck with... | 14:43 |
| noonedeadpunk | https://paste.opendev.org/show/811170/ | 14:44 |
| mgariepy | systemctl status mariadb? | 14:45 |
| noonedeadpunk | ah, I recalled now | 14:45 |
| noonedeadpunk | module_hotfixes=1 was missing | 14:46 |
| noonedeadpunk | (for dnf repo) | 14:46 |
| noonedeadpunk | centos is really so much fun... | 14:46 |
| mgariepy | wow. | 14:47 |
| spatel | did we think about Rocky Linux ? | 14:47 |
| noonedeadpunk | yep and decided that we won't explicitly say about support, since we can't test it and lack interested parties | 14:47 |
| noonedeadpunk | but at the same time it should jsut work as long as os_faminly is redhat | 14:48 |
| spatel | yep | 14:48 |
| noonedeadpunk | mgariepy: it just missing upgrade part from systmd unit | 14:50 |
| noonedeadpunk | (I meant debian-start) | 14:50 |
| mgariepy | if you run the upgrade is it done magically before ? | 14:50 |
| mgariepy | does it returns : This installation of MariaDB is already upgraded to 10.6.5-MariaDB, use --force if you still need to run mysql_upgrade | 14:51 |
| mgariepy | ? | 14:51 |
| noonedeadpunk | should I with ` --version-check`? | 14:51 |
| mgariepy | it don't think it matter much. | 14:52 |
| noonedeadpunk | it just runs | 14:52 |
| mgariepy | i just run without args and its spits that. | 14:52 |
| noonedeadpunk | so no, it does not run magically | 14:52 |
| noonedeadpunk | https://paste.opendev.org/show/811171/ | 14:52 |
| mgariepy | wow.. | 14:53 |
| noonedeadpunk | so, suggestion | 14:53 |
| noonedeadpunk | let's add condition here then https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/817384/2/tasks/galera_server_setup.yml to run only against redhat and only when galera_upgrade? | 14:54 |
| noonedeadpunk | (or galera_force_bootstrap or galera_ignore_cluster_state) | 14:54 |
| noonedeadpunk | or we need to add some hook to wait for post to finish | 14:55 |
| noonedeadpunk | mgariepy: have you posted systemctl status mariadb somewhere? | 14:56 |
| noonedeadpunk | I'm wondering if unit is `activating` until it finish mariadb_upgrade | 14:57 |
| mgariepy | https://paste.opendev.org/show/811164/ | 14:57 |
| noonedeadpunk | `active (running)` damn | 14:58 |
| mgariepy | nop it's not. | 14:58 |
| mgariepy | from what monty__ says on #maria, it needs to be run and if not complete some query or privileges may not work. | 15:01 |
| mgariepy | <monty__> the purpose of mysql_upgrade is to update privilege tables (and view) definitions if needed. It also fixes indexes that has a change in character sort collections | 15:01 |
| mgariepy | <monty__> This means in practice that before mysql_upgrade has been run, some queries and/or privileges may not work | 15:01 |
| noonedeadpunk | oh I missed all fun | 15:02 |
| noonedeadpunk | reads back | 15:02 |
| mgariepy | it it's not done yet i guess if query that we call fails the retry will kick in and pass at some point. | 15:04 |
| mgariepy | on focal with lxc the msyql_upgrade does take like 20 sec.. | 15:04 |
| mgariepy | i wonder how long it takes on a brownfield with a bigger DB. | 15:08 |
| noonedeadpunk | well looking on debian-start I'm pretty sure we don't ever need to run mysql_upgrade there manually | 15:12 |
| mgariepy | yep | 15:12 |
| mgariepy | so only on rhel. | 15:14 |
| mgariepy | and i guess it can be run all the time we run the playbook. not only on upgrades. | 15:14 |
| mgariepy | we could also add args to it but i'm not sure it does differ a lot from the default. | 15:14 |
| mgariepy | yep: version-check TRUE | 15:15 |
| *** akahat|rover is now known as akahat|dinner | 15:19 | |
| jrosser | interesting, OSA X release patches just got created | 15:19 |
| noonedeadpunk | nah, it's automated thing | 15:21 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-galera_server master: Update mariadb to 10.6.5 https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/817384 | 15:34 |
| noonedeadpunk | mgariepy: damn, but we also need to wait before creating users as well? | 15:35 |
| mgariepy | not really | 15:37 |
| noonedeadpunk | oh, ok | 15:52 |
| mgariepy | some stuff may not work but i guess we can continue like that and fix stuff if we cross that bridge sometimes. | 15:53 |
| mgariepy | not seeing how bad of an issue this can cause. | 15:53 |
| *** akahat|dinner is now known as akahat|rover | 16:58 | |
| mgariepy | https://zuul.opendev.org/t/openstack/stream/a9901352be5c4b30ba68ec0c7050eebe?logfile=console.log | 18:23 |
| mgariepy | arf. | 18:23 |
| mgariepy | let's try to reproduce.. | 18:30 |
| mgariepy | noonedeadpunk, jrosser can we have access to that node ? | 18:36 |
| mgariepy | before it gets deleted ? | 18:36 |
| jrosser | if it’s not yet failed the job the infra people can set up a hold on the node | 18:37 |
| mgariepy | what channel ? | 18:37 |
| jrosser | #opendev I think | 18:37 |
| mgariepy | great i'm in. | 18:51 |
| mgariepy | hmm. not sure where to start.. | 19:01 |
| jrosser | the paths to the repos should be in the log I think | 19:04 |
| mgariepy | yeah that part i know :D | 19:05 |
| mgariepy | haha | 19:05 |
| mgariepy | not sure where to start for the galera issue. | 19:05 |
| mgariepy | it's hung. at the same place as the other issue. | 19:10 |
| mgariepy | jrosser, to dump a full trace can i install the debug after the fail ? then generate it with gdb and the core dump ? | 19:37 |
| jrosser | i think you need ther debug symbols, which may be their own package | 19:37 |
| jrosser | and gdb can attach to the running process | 19:37 |
| jrosser | and from memory its 'bt' to dump the backtrace | 19:38 |
| jrosser | that will give you the same as the core dump + debug symbols | 19:38 |
| jrosser | or core dump + binary(with symbols) | 19:38 |
| mgariepy | hmm | 19:39 |
| jrosser | it's been a while though | 19:39 |
| mgariepy | ok let's try that. | 19:39 |
| mgariepy | lol first time for me :D | 19:39 |
| jrosser | first you can attach gdb and do bt | 19:39 |
| jrosser | i think when you attach the debugger it gives some info about if it finds what it wants | 19:39 |
| mgariepy | ho. seems to dump stuff. | 19:43 |
| mgariepy | 160kb of stuff | 19:43 |
| jrosser | it'll dump some stuff per thread, and if there are lots of threads there will be lots of stuff | 19:45 |
| mgariepy | so should i copy the mysql db as well ? | 19:47 |
| jrosser | I don’t know really, you can keep the geld node for a while and see if the mariadb people want more stuff | 20:06 |
| mgariepy | i attached the data to the ticket. | 20:07 |
| mgariepy | it was only 1.5M compressed.. | 20:07 |
| mgariepy | i'm at 21 run without any hangs on my setup. | 20:09 |
| mgariepy | it is really puzzeling. | 20:09 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!