| *** dviroel|afk is now known as dviroel | 00:07 | |
| *** ysandeep|out is now known as ysandeep | 02:09 | |
| *** carloss_ is now known as carloss | 06:12 | |
| *** ysandeep is now known as ysandeep|afk | 07:02 | |
| fanfi_ | hi, is there sb who can help me with bootstrap script ? https://paste.opendev.org/show/bgD9fuewymHEY2Ip1kYL/ | 07:57 |
|---|---|---|
| jrosser_ | fanfi_: which branch is that? master? | 08:12 |
| fanfi_ | * master | 08:12 |
| fanfi_ | remotes/origin/HEAD -> origin/master | 08:12 |
| jrosser_ | did you start from a clean host there? | 08:13 |
| fanfi_ | yes | 08:13 |
| fanfi_ | ubuntu 20 | 08:16 |
| noonedeadpunk | that sounds like installation of collections fails | 08:16 |
| noonedeadpunk | ah | 08:16 |
| jrosser_ | i think we need to address this https://opendev.org/openstack/openstack-ansible/src/branch/master/scripts/openstack-ansible.rc#L56 | 08:17 |
| jrosser_ | here https://github.com/openstack/openstack-ansible/blob/master/scripts/bootstrap-ansible.sh#L194-L204 | 08:18 |
| noonedeadpunk | hm. how we haven't catched that? | 08:19 |
| noonedeadpunk | I was running AIO multiple times... | 08:19 |
| noonedeadpunk | will you patch that? | 08:19 |
| jrosser_ | yes let me take a look | 08:28 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Fix tasks name for collections bootstrap https://review.opendev.org/c/openstack/openstack-ansible/+/847134 | 08:29 |
| Adri2000 | hello jrosser_, I've just replied to https://review.opendev.org/c/openstack/openstack-ansible/+/846787 - but I understand my patch may be wrong, given that the CI fails anyway. happy to discuss further here to try to understand what would be a proper fix | 08:34 |
| jrosser_ | noonedeadpunk: ^ regarding this - we have a bunch of code with conditionals on `(galera_root_user != 'root')` and i cant' remember if that stuff is all transitional | 08:35 |
| jrosser_ | and it's really confusing to read actually | 08:36 |
| Adri2000 | my understanding is that this patch https://opendev.org/openstack/openstack-ansible-galera_server/commit/931f3c74a78774f319acfb6867ff9742b9b46e3d has the consequence that /root/.my.cnf is no longer dropped in the galera container. and that has to be done by the client part of the galera role, later in the process. | 08:37 |
| * noonedeadpunk in a meeting | 08:43 | |
| anskiy | Not sure if this relates, but I've been doing this: https://paste.opendev.org/show/bA0puUg2Sv9ueZtgaR5j/, prior to running any openstack-ansible things. | 08:44 |
| jrosser_ | fanfi_: i am not managing to reproduce your error :( | 08:56 |
| jrosser_ | i made a fresh 20.04 VM, apt update / dist-upgrade, git clone https://opendev.org/openstack/openstack-ansible.git, cd openstack-ansible, ./scripts/bootstrap-ansible.sh | 08:58 |
| fanfi_ | yes, i did same on fresh ubuntu 20.04 VM https://docs.openstack.org/project-deploy-guide/openstack-ansible/latest/deploymenthost.html | 08:59 |
| noonedeadpunk | So. | 08:59 |
| noonedeadpunk | When user is root, we need my.cnf to be installed since role will set password on the user and auth with socket won't be available | 09:00 |
| fanfi_ | i am using MAAS so it easy to make redeploy the VM, let me try it again | 09:01 |
| jrosser_ | hold on | 09:01 |
| noonedeadpunk | When user is not root, we don't need that, as socket path is default. We basically don't even need my.cnf inside galera containers, as socket auth will be default way as well | 09:01 |
| fanfi_ | oka | 09:01 |
| jrosser_ | fanfi_: can you try this? https://paste.opendev.org/show/b19m4Pl8NhG9LA0bmsx8/ | 09:01 |
| damiandabrowski[m] | wasn't it fixed by https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/829260 ? | 09:01 |
| noonedeadpunk | Adri2000: ^ | 09:02 |
| noonedeadpunk | so basically question - why you need my.cnf inside galera containers? | 09:03 |
| anskiy | damiandabrowski[m]: I'll give it a try, thank you! | 09:03 |
| Adri2000 | noonedeadpunk: hmm ok. in my case, running `mysql` in the galera container was failing, it'd not connect to the socket. I'm currently setting up an AIO master to check whether I can reproduce the problem or if at least I can understand what was wrong in my deployment. | 09:03 |
| Adri2000 | noonedeadpunk: indeed I don't need my.cnf explicitely, I just need to be able to login to mysql | 09:03 |
| noonedeadpunk | Well, that could be the case in your deployment, if you upgraded OS as example? | 09:04 |
| jrosser_ | i think this is my confusion about why we have that conditional code still | 09:04 |
| jrosser_ | if the client is never needed in the galera containers in newer deploys..... | 09:04 |
| noonedeadpunk | So root user is still set for password auth (ie it was older deployment when we used root user), but when you re-deployed contianer, my.cnf jsut vanished | 09:04 |
| noonedeadpunk | jrosser_: it's needed if user is root | 09:05 |
| noonedeadpunk | if it's been explicitly overriden as an example | 09:05 |
| noonedeadpunk | Adri2000: so basically, if you indeed re-deployed containers and can't connect, then likely you need either to set `galera_root_user: root` or reset root user auth to socket instead of password | 09:06 |
| noonedeadpunk | As we indeed didn't cover upgrade path in terms of containers re-deployment or OS upgrade. | 09:07 |
| noonedeadpunk | or well. you can always revert to using root user | 09:07 |
| Adri2000 | it's a deployment where we recently upgraded from ubuntu 18.04 to 20.04, and then from victoria to wallaby... from what I see the transition from "root" to "admin" was done correctly | 09:07 |
| Adri2000 | let me do some more testing and I'll come back later today with some explanation, hopefully :) | 09:08 |
| *** ysandeep|afk is now known as ysandeep | 09:09 | |
| noonedeadpunk | Maybe you need to jsut drop 'root'@'%' ? | 09:09 |
| noonedeadpunk | Ie https://review.opendev.org/c/openstack/openstack-ansible/+/775684/8/releasenotes/notes/galera_root_user-43c292688ddc4f1d.yaml | 09:09 |
| noonedeadpunk | Sure! | 09:09 |
| admin1 | https://docs.openstack.org/openstack-ansible-os_neutron/latest/configure-network-services.html -- i am guessing firewall_v2 and vpnaas is not ON by default ? | 09:43 |
| noonedeadpunk | We should drop fwaas mentioning from docs I believe... | 09:51 |
| noonedeadpunk | but no, they're not | 09:51 |
| jrosser_ | fwaas is coming back some time isnt it | 09:53 |
| jrosser_ | but no idea if that is only OVN or smth | 09:54 |
| noonedeadpunk | huh | 09:55 |
| noonedeadpunk | it's not ovn | 09:55 |
| noonedeadpunk | https://docs.openstack.org/neutron/latest/admin/fwaas-v2-scenario.html | 09:55 |
| noonedeadpunk | but how it should be isntalled... | 09:55 |
| noonedeadpunk | I also not understand how it;s different from port security now... | 09:56 |
| jrosser_ | its at the router isnt it? | 09:58 |
| noonedeadpunk | v1 yes on the router | 09:59 |
| noonedeadpunk | v2 on ports? | 09:59 |
| noonedeadpunk | https://docs.openstack.org/neutron/latest/admin/fwaas.html | 09:59 |
| noonedeadpunk | but indeed I see that https://opendev.org/openstack/neutron-fwaas revived recently | 09:59 |
| noonedeadpunk | ugh | 09:59 |
| noonedeadpunk | need to catch up with that | 10:00 |
| anskiy | noonedeadpunk: fwaas works only with OVS | 10:18 |
| noonedeadpunk | yeah... that's true. | 10:19 |
| anskiy | it doesn't break if you're using OVN tho, just silently allows all traffic :) | 10:20 |
| admin1 | so fwaaas is out . is vpnaas future proof ? | 10:29 |
| admin1 | ovs -> ovn proof | 10:29 |
| *** dviroel__ is now known as dviroel | 11:28 | |
| *** ysandeep is now known as ysandeep|afk | 12:23 | |
| *** ysandeep|afk is now known as ysandeep | 13:22 | |
| Adri2000 | noonedeadpunk, jrosser_: tested on a brand new OSA AIO master: indeed `mysql` will allow to login inside the galera container. however it seems that works as it authenticates with the root user via unix socket. my understanding of what you said, is that it should work with the admin user via unix socket. `show grants for admin@localhost` will show that the admin user is | 14:41 |
| Adri2000 | actually not authorized to login via unix_socket. in other words if you drop the root user (as recommended in the release note), then `mysql` in the container will no longer log you in. | 14:41 |
| Adri2000 | also, is it at all expected that the user is present on a brand new deployment? if we recommend deleting it on upgrades, it should not be there on a fresh install? | 14:42 |
| Adri2000 | finally, I noticed /etc/mysql/debian.cnf mentions the root user twice - shouldn't it be replaced with admin? | 14:43 |
| Adri2000 | that's OSA master on Ubuntu 20.04, fwiw | 14:43 |
| Adri2000 | *** I meant: is it expected that the *root* user is present on a brand new deployment | 14:44 |
| jrosser_ | i think that the intention is that for everything beyond boostrapping the admin user is done fron the utility container | 14:45 |
| noonedeadpunk | Adri2000: but in release notes recommended to drop root@%. There's still root@localhost that should be able to login to the socket? | 14:58 |
| noonedeadpunk | and root@% is not present on brand new deployments I believe | 14:58 |
| *** dviroel is now known as dviroel|lunch | 14:59 | |
| *** ysandeep is now known as ysandeep|out | 15:00 | |
| Adri2000 | right only root@localhost is present. so it's expected that login from the galera container happens via root/unix_socket rather than admin/unix_socket? | 15:11 |
| noonedeadpunk | yup | 15:12 |
| Adri2000 | ok. what about the root+password credentials in /etc/mysql/debian.cnf? I'm not sure it causes actual problems, but it should be something else right? either admin+password, or just root/unix_socket | 15:12 |
| lowercas_ | Has anyone ran into an issue where openstack nova is complaining about cell v2 when upgrading from wallaby to xena where it appears the api is calling an older api version which isn't compatible with the new xena version? | 15:12 |
| noonedeadpunk | Basically this comes from recommendation never to mess up with root user on mariadb | 15:13 |
| Adri2000 | anyway, thanks noonedeadpunk and jrosser_ for your help. I'll rethink about all of this tomorrow as I had enough mysql for today =) and will certainly drop my patch | 15:13 |
| noonedeadpunk | I can't recall, but that could be opened as separate packaging bug or smth | 15:13 |
| Adri2000 | yeah, I can do a bug report for that part | 15:14 |
| jrosser_ | Adri2000: if you think we could document better, it would be cool to know where / what | 15:14 |
| noonedeadpunk | as what mariadb folks told us is that modifying root user in anyway is smth that should be avoided and not good practise | 15:14 |
| Adri2000 | understood | 15:15 |
| noonedeadpunk | btw isn't in debian.cnf some other user used by default then root? | 15:16 |
| noonedeadpunk | like system-maint or smth like that? | 15:16 |
| Adri2000 | maybe by default, but I believe OSA will rewrite it: https://opendev.org/openstack/openstack-ansible-galera_server/src/branch/master/tasks/galera_server_post_install.yml#L151-L155 and root is hardcoded in the template: https://opendev.org/openstack/openstack-ansible-galera_server/src/branch/master/templates/debian.cnf.j2 | 15:18 |
| noonedeadpunk | ok, that is weird/wrong.... | 15:19 |
| lowercas_ | found it: https://bugs.launchpad.net/openstack-ansible/+bug/1736731 | 15:21 |
| lowercas_ | ^ for my bug | 15:21 |
| noonedeadpunk | lowercas_: don't we cover that with https://opendev.org/openstack/openstack-ansible-os_nova/src/branch/master/tasks/nova_db_setup.yml#L60-L87 ? | 15:23 |
| noonedeadpunk | looking at patch, it should be already present for Wallaby | 15:24 |
| lowercas_ | noonedeadpunk: https://paste.centos.org/view/ee5f5b81 | 15:30 |
| lowercas_ | you're query doesn't appear to be grabbing the uuid, its grabbing the whole line? | 15:30 |
| noonedeadpunk | I think we rely more on the return code then on the content | 15:33 |
| noonedeadpunk | btw | 15:33 |
| lowercas_ | hmm, yea i see that. | 15:34 |
| noonedeadpunk | So bsically, when there's already cell1 present `Create the cell1 mapping entry in the nova API DB` jsut should not run | 15:35 |
| lowercas_ | root@infra1-nova-api-container-f0847039:/openstack/venvs/nova-24.1.0# bin/nova-manage cell_v2 verify_instance --uuid 70bd4168-3fc1-4b6e-b2bb-6b0faf314702 | 15:36 |
| lowercas_ | Instance 70bd4168-3fc1-4b6e-b2bb-6b0faf314702 is not mapped to a cell (upgrade is incomplete) or instance does not exist | 15:36 |
| lowercas_ | that's essentially where i am stuck | 15:36 |
| *** lowercas_ is now known as lowercase | 15:37 | |
| opendevreview | Merged openstack/openstack-ansible master: Fix tasks name for collections bootstrap https://review.opendev.org/c/openstack/openstack-ansible/+/847134 | 16:12 |
| *** dviroel|lunch is now known as dviroel | 16:31 | |
| spatel | Any outstanding issue to upgrade from Wallaby->Xena ? | 16:56 |
| fanfi_ | hi, guys did you get following error during bootstrap ? https://paste.opendev.org/show/b7ndfzMTPjsKujtj6ANU/ | 19:07 |
| noonedeadpunk | fanfi_: galaxy.ansible.com can have plenty of different issues :( | 19:10 |
| noonedeadpunk | most of they are intermittent though and should restore on themselves | 19:11 |
| fanfi_ | :( sweet | 19:11 |
| noonedeadpunk | we have only 2 collections being installed from galaxy though | 19:14 |
| noonedeadpunk | (maybe 3) | 19:14 |
| noonedeadpunk | you can replace them with installation from github but there're drawbacks | 19:15 |
| noonedeadpunk | as they won't have version when installed | 19:15 |
| noonedeadpunk | fanfi_: but to have that said it's all really about galaxy.ansible.com which we don't control | 19:19 |
| jrosser_ | fanfi_: i could not reproduce either your error with the linear strategy nor the SSL error which you showed me earlier | 19:21 |
| fanfi_ | sure, i will need to check the collection what must by installed | 19:21 |
| jrosser_ | are you behind some strange corporate SSL man-in-the-middle firewall? | 19:21 |
| noonedeadpunk | you can also create user-collection-requirements.yml with content https://paste.openstack.org/show/bVtF9LZofRiWIB8ShzqD/ | 19:22 |
| noonedeadpunk | It could be also some CDN thing... | 19:23 |
| fanfi_ | not sure, probably not. it is lab env. but in corporate building | 19:23 |
| noonedeadpunk | once I wrote it, I'm not sure this would replace galaxy source.. .Or well, I'm pretty sure it's not... | 19:24 |
| noonedeadpunk | but you can just in-place manually overwrite them in /opt/opensatck-ansible/ansible-collections-requirements.yml | 19:25 |
| fanfi_ | okay ...looks like our corporate issue with ssl :( ..so i need to install the collections by different way | 19:33 |
| spatel | Any idea about this error ? - https://paste.opendev.org/show/bIur5LQFlctDbuCnAfn4/ | 19:33 |
| spatel | deploying 23.3.0 wallaby on new cloud | 19:33 |
| jrosser_ | what version of the community.general collection do you have | 19:38 |
| jrosser_ | it should be 2.1.1 https://github.com/openstack/openstack-ansible/blob/stable/wallaby/ansible-collection-requirements.yml#L6 | 19:39 |
| spatel | checking.. | 19:41 |
| admin1 | spatel, i have faced that error somewhere .. but can't remember when | 19:42 |
| spatel | its version: 2.1.1 | 19:42 |
| admin1 | i upgraded a few to 23.3.0 without issues | 19:42 |
| spatel | let me re-run playbook | 19:42 |
| admin1 | check haprpxy stats .. does it show all OK ? | 19:43 |
| spatel | its showing ok | 19:43 |
| admin1 | galera ,. one should be green and 2 other in standby type ( and not error ) | 19:44 |
| spatel | i found some DNS related issue which i just fixed so lets see | 19:44 |
| admin1 | in my case, my galera was not in sync/broken | 19:44 |
| *** dviroel is now known as dviroel|afk | 19:48 | |
| spatel | something is really messed up.. i am destroying container and re-building it | 19:58 |
| spatel | galera is not happy somehow | 19:58 |
| spatel | jrosser_ admin1 no error after destroy container and re-deploy galera | 20:04 |
| admin1 | yours is a new cloud, so nice | 20:05 |
| spatel | yes | 20:05 |
| spatel | building 78 nodes cloud in singapore region | 20:06 |
| admin1 | mine was an existing one where upgrade ( somehow ) wiped my old galera and brought up a new database .. ( it said cluster name mismatch ) .. but i had backups just 5 mins before, so had to manually bring up galera and re-run | 20:06 |
| spatel | once it done, my goal is to go back to OVN and test LXD to OVN migration stuff | 20:06 |
| jrosser_ | spatel: if you read the error message it is a task delegated from the galera container to the host, managing the haproxy settings | 20:10 |
| spatel | what does that means? | 20:14 |
| spatel | This is what i think, i had DNS problem on haproxy public vip and that may created issue. | 20:15 |
| spatel | it could be just me and my error | 20:16 |
| spatel | I have to leave now but i will see if i hit any other error :) | 20:18 |
| spatel | soon i have plan to upgrade all production clouds from Wallaby->Xena so hope all go well | 20:18 |
| spatel | Good night! see you tomorrow | 20:19 |
| *** dviroel|afk is now known as dviroel | 20:43 | |
| *** dviroel is now known as dviroel|afk | 21:22 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!