| opendevreview | Jimmy McCrory proposed openstack/openstack-ansible-galera_server master: Include CA cert in client my.cnf https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/900266 | 06:21 |
|---|---|---|
| jrosser | nixbuilder: is that only with firefox? other browsers are OK? | 08:45 |
| opendevreview | Niklas Schwarz proposed openstack/openstack-ansible-rabbitmq_server master: Add ability to add custom configuration for RabbitMQ https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/900023 | 08:50 |
| damiandabrowski | is there something wrong with CI? I spotted a lot of POST_FAILURES yesterday | 09:23 |
| damiandabrowski | https://zuul.opendev.org/t/openstack/builds?project=%09openstack%2Fopenstack-ansible-os_nova&result=POST_FAILURE&skip=0 | 09:23 |
| jrosser | i think there might be failure in rackspace object storage for log uploads | 09:38 |
| damiandabrowski | ahh thanks | 09:42 |
| nixbuilder | jrosser: No... other browsers fail as well. But I think I found a workaround in just turning off SSL on horizon. | 10:21 |
| jrosser | nixbuilder: what kind of build is that? can you reproduce it with an all-in-one? | 10:27 |
| nixbuilder | This is our production build... AIO works fine. In our production environment I have three infra, five compute and two haproxy servers. After install I will ultimately have ~15 or so compute servers. | 10:32 |
| jrosser | if there is a difference between production and AIO then that could point to a configuration error | 10:44 |
| jrosser | i am guessing that this is an internal system rather than internet facing if you're trying to connect to 10.255.60.29 | 10:44 |
| jrosser | for my internet facing endpoints i use some of the online SSL checkers to validate the https setup | 10:45 |
| anskiy | nixbuilder: you can try issuing curl/openssl s_client on it: could get less vague error message | 10:46 |
| nixbuilder | jrosser: This is our second 'new' cloud. But yes, both clouds are all internal and have no access to the outside world as they sit behind the corporate firewalls. | 10:46 |
| jrosser | where did the SSL certificate come from? | 10:47 |
| jrosser | like anskiy says you can do some quite useful debugging with CLI tools against the horizon service | 10:47 |
| nixbuilder | jrosser: I assume the SSL certificate were the self-signed certificates generated by the OSA scripts. | 10:48 |
| nixbuilder | anskiy: I did do some wget/curl debugging as well as openssl. But then decided to eliminate https just like our old system. | 10:49 |
| jrosser | you could try this https://testssl.sh/ | 10:50 |
| noonedeadpunk | I kinda wonder if that's also related to some proxy settings potentially? | 10:51 |
| noonedeadpunk | tjhat would explain why CLI works, as it doesn't go through proxy | 10:51 |
| jrosser | that depends on environment vars | 10:51 |
| noonedeadpunk | well, true | 10:52 |
| noonedeadpunk | but usually we suggest adding internal network to no_proxy in docs iirc | 10:52 |
| noonedeadpunk | and then there's a question - does CLI from your local machine also works, or you was reffering only to the utility container cli? | 10:53 |
| anskiy | nixbuilder: you've disabled https for horizon right now, if I understood you correctly? | 10:55 |
| nixbuilder | I did do a packet capture on the infra node and basically it said I was getting a 400 Bad Request error... wireshark also noted that it saw unencrypted HTTP traffic over HTTPS. That is when I decided to turn off SSL on install and re-install. | 10:57 |
| nixbuilder | anskiy: Yes, I have disabled HTTPS on horizon and am in the process of re-installing from scratch. | 10:57 |
| nixbuilder | noonedeadpunk: The cli was on the infra node. | 10:58 |
| nixbuilder | No containers... all bare metal | 10:58 |
| anskiy | well, seeing non-https traffic in what's supposed to be https connection explains the error, so it's either, like noonedeadpunk suggested: some problem with proxy on the machine where you're running firefox, or something wrong with haproxy's config | 10:59 |
| opendevreview | Merged openstack/openstack-ansible-os_ironic stable/2023.1: Use common value for inspector callback URL https://review.opendev.org/c/openstack/openstack-ansible-os_ironic/+/900080 | 10:59 |
| nixbuilder | anskiy: I suspected haproxy misconfiguration but since I don't have any experience with haproxy as our existing production system didn't use it, I decided to turn off SSL. | 11:01 |
| nixbuilder | anskiy: Our existing production system does not not use HTTPS on horizon. | 11:02 |
| noonedeadpunk | I also wonder if that could be some kind of HSTS issue as well.... | 11:17 |
| jrosser | even so - a full metal deploy is an interesting thing as not many people do that | 11:21 |
| * jrosser can't remember if we actually enable horizon in CI for that | 11:22 | |
| noonedeadpunk | we don't generally | 11:23 |
| jrosser | nixbuilder: if you run into problems like these please do ask, if there is a deployment bug we should try to fix it | 11:23 |
| anskiy | some time ago I was trying to disable horizon on is_metal, and it didn't work :) | 11:23 |
| jrosser | otherwise we end up with a kind of FUD situation with "metal deploys don't work for https" | 11:23 |
| anskiy | it was still deployed as part of the shared-infra, IIRC | 11:24 |
| noonedeadpunk | yeah, in case it's in shared-infra - it will get deployed. But in CI we don't use shared-infra defenition | 11:37 |
| nixbuilder | ALL: Thanks for the help. If I run into problems I will send up smoke signals :-) | 11:49 |
| opendevreview | Merged openstack/openstack-ansible-os_ceilometer master: Enable Ceilometer resource cache https://review.opendev.org/c/openstack/openstack-ansible-os_ceilometer/+/888032 | 12:11 |
| NeilHanlon | o/ mornin' folks | 13:11 |
| noonedeadpunk | \o/ | 13:33 |
| NeilHanlon | i'm glad i looked at my calendar.. forgot about daylight savings.. | 14:46 |
| NeilHanlon | see y'all in 15 ;) | 14:46 |
| * noonedeadpunk hates daylight savings | 14:49 | |
| mgariepy | lol. | 14:54 |
| mgariepy | we should all be in UTC for everything. | 14:54 |
| noonedeadpunk | #startmeeting openstack_ansible_meeting | 15:01 |
| opendevmeet | Meeting started Tue Nov 7 15:01:00 2023 UTC and is due to finish in 60 minutes. The chair is noonedeadpunk. Information about MeetBot at http://wiki.debian.org/MeetBot. | 15:01 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 15:01 |
| opendevmeet | The meeting name has been set to 'openstack_ansible_meeting' | 15:01 |
| noonedeadpunk | #topic roll call | 15:01 |
| noonedeadpunk | o/ | 15:01 |
| NeilHanlon | o/ heya | 15:01 |
| mgariepy | hey | 15:01 |
| * NeilHanlon resists urge to use other meetbot's commands | 15:01 | |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-plugins master: Set the default domain for the role_assignment https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/900329 | 15:04 |
| noonedeadpunk | #topic office hours | 15:04 |
| noonedeadpunk | so. I have barely catched up with what happened last wekk | 15:04 |
| jrosser | o/ | 15:07 |
| NeilHanlon | heya jrosser | 15:07 |
| noonedeadpunk | From what I see we've almost landed quorum_queues, with very valid comment on the ceilometer role | 15:08 |
| NeilHanlon | there was also a question last week about quorum queues for Keystone and Swift | 15:08 |
| noonedeadpunk | We also do have zun, magnum and manila roles broken | 15:08 |
| noonedeadpunk | yeah, I guess that's related to the comment | 15:09 |
| noonedeadpunk | I think I've skipped these 2 because neither keystone nor swift do not use them for RPC | 15:09 |
| noonedeadpunk | but I think they indeed should be covered | 15:09 |
| noonedeadpunk | #action noonedeadpunk to propose quorum_queues patches for keystone and swift | 15:10 |
| NeilHanlon | depending on the work required, I think it'd be ok to postpone to after release, since as you said they are not used for RPC | 15:10 |
| noonedeadpunk | For broken roles I had a look only at manila, and it's broken due to ceph-ansible being incompatible with ansible-core 2.15 | 15:10 |
| NeilHanlon | also FYI RHEL 9.3 released today. So.. Rocky 9.3 will come at some point. I'll give advanced notice | 15:11 |
| anskiy | NeilHanlon: yeah, instead of remembering that thing, I've manaaged to put valid comment :) | 15:11 |
| noonedeadpunk | And I even proposed PR: https://github.com/ceph/ceph-ansible/pull/7466 | 15:11 |
| NeilHanlon | anskiy: thank you :D I did also put it on our last PTG etherpad so I would remember (somehow) | 15:12 |
| noonedeadpunk | But I kinda concerned if it will merge and if it will - I highly doubt it would be backported by release due time... | 15:12 |
| noonedeadpunk | regarding magnum and zun - they seem to both fail with DB upgrade. I haven't looked there yet. And that has also happened after updating SHAs... | 15:13 |
| noonedeadpunk | Would need to have a closer look what's wrong there. | 15:14 |
| NeilHanlon | hm | 15:14 |
| noonedeadpunk | from good news - distro path seems to "just work" after switching to Bobcat :) | 15:16 |
| NeilHanlon | 🥳 | 15:16 |
| noonedeadpunk | oh, well, magnum is slightly different. but not less confusing | 15:17 |
| jrosser | something odd is happening there | 15:17 |
| jrosser | `The container-infrastructure-management service for default:RegionOne exists but does not have any supported versions.` | 15:18 |
| noonedeadpunk | yeah | 15:19 |
| noonedeadpunk | but sometimes it pass... | 15:19 |
| noonedeadpunk | so would need to reproduce this one | 15:19 |
| jrosser | that error actually is from the SDK, so it could be trouble anywhere from SDK onward | 15:20 |
| noonedeadpunk | For openstack-resources topic: trove seems good except upgrade jobs. Apparently, role trying to change the network that's in use and fails. Likely I've missed something, as that works nicely for octavia which is pretty much simmilar: https://review.opendev.org/c/openstack/openstack-ansible-os_trove/+/899284 | 15:21 |
| noonedeadpunk | so magnum and standalone playook left to reach the minimum par for the role | 15:21 |
| noonedeadpunk | no progress regarding skyline so far | 15:32 |
| anskiy | for some reason I've been poking at the Vagrantfile inside openstack-ansible: I've added some cache (which breaks a bit at simultaneous package installation in containers) and roles/collections passthrough: https://github.com/dbalagansky/openstack-ansible/commit/b9168a5d022a5321c7bca4b7f5bbb470eccbaa7c | 15:33 |
| noonedeadpunk | anskiy: to be frank - we didn't maintained that for quite a while now | 15:38 |
| noonedeadpunk | so I can hardly comment on that and I was kinda thinking to clean that up with functional tests repo and functional jobs from tox | 15:39 |
| jrosser | i think there was even the start of an effort to remove all of that | 15:40 |
| noonedeadpunk | and now with vagrant being BSL-licensed... | 15:40 |
| NeilHanlon | yeahhhh | 15:41 |
| nixbuilder | Anyone have any ideas where I can start looking to fix this error: "infra01 neutron-rpc-server[8037]: SQL connection failed. 10 attempts left." Keeps repeating over and over. Plus connecting to port 9696 yet that port is open on the server. | 15:42 |
| noonedeadpunk | I think it 's a good idea to have some easy way to spawn aio is some kind of VM on the localhost without much fuss. But I kinda failed to find a good option except virt-manager.... | 15:46 |
| anskiy | noonedeadpunk: well, I was hoping to fix two thing: eliminate running with patches between the host and VM, and ability to easily instantiate MNAIO | 15:51 |
| anskiy | and the last part is what I've had previously, but with some additional roles in the middle (which rendered openstack_user_config, for example) | 15:52 |
| anskiy | I can probably try rewriting this thing on plain ansible with `community.libvirt` collection, if this is really something that's welcome :) | 15:53 |
| noonedeadpunk | I actually was thinking about re-writing MNAIO for quite a while but never had a chance | 15:55 |
| noonedeadpunk | And my idea was also to have some test_openstack_user_config that would be parsed by dynamic_inventory and provided as input to some role that will spawn resources | 15:56 |
| jrosser | i think jamesdenton has looked also at the MNAIO | 15:56 |
| noonedeadpunk | yeah | 15:56 |
| NeilHanlon | crap forgot to bring up my thing but it's not very long or important.. I made some progress on Incus for Fedora and Enterprise Linux, and made a connection with someone who did a much better job than I at packaging it, too :) so -- will be working on that in my spare time to get it ready for Rocky 9 and friends | 15:57 |
| noonedeadpunk | and yeah, community.libvirt was on the radar as one "driver", while another "driver" was to use already existing openstack project (and probably leverage openstack_resources role for that purpose) | 15:58 |
| noonedeadpunk | NeilHanlon: oh, these are great news! | 15:58 |
| NeilHanlon | there is apparently some *drama* happening with some of the dependencies of LXD, so will be interesting to see how it plays out | 15:58 |
| NeilHanlon | `cowsql` and `raft`, specifically | 15:59 |
| noonedeadpunk | I assume that incus folks should be also quite interested to provide some help with that effort | 15:59 |
| NeilHanlon | I assume so as well | 15:59 |
| noonedeadpunk | yeah, I think I've read smth about cowsql at least in one of their blogposts | 16:00 |
| NeilHanlon | raft apparently has had two ABI changes this year without a soname bump, which is.. ugh | 16:00 |
| NeilHanlon | #link https://github.com/ganto/copr-lxc4 | 16:00 |
| NeilHanlon | mostly that's for me, but, yeah. that's where I'll be looking to build some crap on EPEL with :) | 16:01 |
| NeilHanlon | notably, it also should mean we can eventually remove the dependency on my personal copr for lxc4 | 16:01 |
| NeilHanlon | (https://opendev.org/openstack/openstack-ansible-lxc_hosts/src/branch/master/vars/redhat-host.yml#L19) | 16:02 |
| noonedeadpunk | at some point I think I've proposed patch for that.... | 16:02 |
| noonedeadpunk | I _think_ it was build for epel even? | 16:02 |
| noonedeadpunk | https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2022-968b01292a | 16:03 |
| noonedeadpunk | #endmeeting | 16:03 |
| opendevmeet | Meeting ended Tue Nov 7 16:03:41 2023 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 16:03 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/openstack_ansible_meeting/2023/openstack_ansible_meeting.2023-11-07-15.01.html | 16:03 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/openstack_ansible_meeting/2023/openstack_ansible_meeting.2023-11-07-15.01.txt | 16:03 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/openstack_ansible_meeting/2023/openstack_ansible_meeting.2023-11-07-15.01.log.html | 16:03 |
| NeilHanlon | hmm. maybe we are using that one for the `templates-extra` thing, then | 16:03 |
| NeilHanlon | thanks for running the meeting btw noonedeadpunk | 16:03 |
| noonedeadpunk | yeah, could be for the templates.... | 16:04 |
| noonedeadpunk | worth revising that actually | 16:04 |
| jrosser | nixbuilder: neutron rpc server is trying to connect to the database | 16:05 |
| NeilHanlon | I think the consensus when I spoke with `thm` about it was that we shouldn't build two things in the same spec file, so I think I can (probably) just introduce a distinct `lxc-templates-extra-legacy` package which provides the legacy templates. or.. something like that anyways | 16:05 |
| jrosser | so you need to check some things, that the database is OK, that haproxy *thinks* that the database is OK, and the database port on the internal VIP is reachable from wherever neutron-rpc-server is running | 16:05 |
| NeilHanlon | or maybe we just move away from the legacy templates? i don't remember what's involved with that | 16:06 |
| nixbuilder | jrosser: Yeah... I think the manilla install may have broke mysql | 16:06 |
| jrosser | well, that would be surprising :) | 16:07 |
| noonedeadpunk | NeilHanlon: to be frank - me neither... | 16:08 |
| jrosser | haproxy status is a good place to start, using `hatop` | 16:08 |
| noonedeadpunk | potentially we indeed don't need them. But iirc on ubuntu they were providing profiles for apparmour or smth like that... | 16:09 |
| NeilHanlon | psh | 16:11 |
| NeilHanlon | security | 16:11 |
| NeilHanlon | who needs it | 16:12 |
| NeilHanlon | although, i did recently find a guide for openstack bobcat on centos stream 9 that had selinux profiles... | 16:12 |
| noonedeadpunk | well, I'm pretty sure we're running with apparmour profiles being active | 16:13 |
| noonedeadpunk | never managed to deal nicely with selinux though.... | 16:13 |
| anskiy | they don't actually have much in common :) | 16:13 |
| opendevreview | Merged openstack/openstack-ansible-os_nova stable/2023.1: Always disable libvirt default network https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/900190 | 16:15 |
| noonedeadpunk | https://paste.openstack.org/show/bhf0TBfoChPcdv1eaFBw/ | 16:15 |
| noonedeadpunk | well... they both try to restrict apps from not doing weird things? :p | 16:15 |
| NeilHanlon | ostensibly | 16:18 |
| anskiy | noonedeadpunk: if you put it that way, yes, but for selinux it's just a part of it: because it can operate on users/network primitives/inter-process comms/filesystem objects | 16:18 |
| noonedeadpunk | but I close to never run openstakc with EL in any sort of production (except migrating 1 deployment from centos 7 to debian 10 or smth) | 16:18 |
| NeilHanlon | they're both security theater :) | 16:18 |
| noonedeadpunk | anskiy: yeah, but satisfying it is really... time consuming, I would say? | 16:19 |
| anskiy | noonedeadpunk: sure :) I was just trying to say that selinux is a bit bigger in what it could do, in comparison to apparmor | 16:20 |
| anskiy | and you probably wouldn't even need it for the case, when it's not a multiuser system | 16:21 |
| nixbuilder | jrosser: Well something surely broke mysql... it was working up until the manilla install (https://paste.opendev.org/show/bY3q5tmyC2itBA4jFGI1/) | 16:21 |
| jrosser | nixbuilder: is there anything else using the IP/ports that you'd expect mariadb to be using there (maybe manila related?) | 16:23 |
| jrosser | also something has caused mariadb to restart? | 16:25 |
| jrosser | thats worth understanding | 16:25 |
| nixbuilder | jrosser: To be honest, I didn't realize that manilla would install... my bad. I took the openstack_user_config.yml from my previous AIO, edited it, and guess I missed the manilla portion and that's why the script started to install manilla. | 16:25 |
| jrosser | remember that we run metal AIO for manila, and this works | 16:26 |
| jrosser | +/- ceph-ansible trouble of course | 16:26 |
| jrosser | but regardless, the database should not be restarted by installing a new service | 16:26 |
| nixbuilder | jrosser: I don't understand what this error means... "Failed to open backend connection: -110 (Connection timed out)"... this is coming from the mariadb. | 16:28 |
| jrosser | there is a database cluster, the different mariadb instances need to talk to each other | 16:29 |
| damiandabrowski | sorry, i couldn't attend to the meeting | 16:29 |
| damiandabrowski | jrosser: regarding magnum and `The container-infrastructure-management service for default:RegionOne exists but does not have any supported versions.` | 16:29 |
| damiandabrowski | I believe this patch will fix it: https://review.opendev.org/c/openstack/openstack-ansible-os_magnum/+/897526 | 16:29 |
| jrosser | damiandabrowski: ahh cool - maybe we can get within the timeout now the jobs are adjusted a bit | 16:30 |
| damiandabrowski | fingers crossed | 16:30 |
| jrosser | nixbuilder: this https://mariadb.com/kb/en/galera-cluster-address/ | 16:31 |
| jrosser | nixbuilder: there is also some guide for database maintainance https://docs.openstack.org/openstack-ansible/latest/admin/maintenance-tasks.html | 16:35 |
| noonedeadpunk | damiandabrowski: ah, right, thanks fro reminding about it | 16:35 |
| NeilHanlon | btw if anyone needs reviews, ping me.. i'm distracted like usual but I am happy to help out | 16:39 |
| nixbuilder | jrosser: Well I got mysql running on one node by turning off the cluster in /etc/my.cnf.d/cluster.cnf | 16:39 |
| nixbuilder | jrosser: just need to try and figure out how to fix the cluster. | 16:39 |
| jrosser | this suggests that there is something broken, perhaps with your networking | 16:40 |
| jrosser | the playbooks should bring up the database cluster and it should then work on it's own | 16:40 |
| nixbuilder | jrosser: Reading through the link you sent me. | 16:41 |
| jrosser | it's important to get this all to the position where you can run/r-run the playbooks without incident as thats the basis of an upgrade | 16:42 |
| jrosser | *re-run | 16:42 |
| opendevreview | Merged openstack/openstack-ansible-os_nova stable/xena: Always disable libvirt default network https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/900193 | 16:53 |
| nixbuilder | jrosser: Yeah... when I tried to recover the database I kept getting these... manilla did something to my database: https://paste.opendev.org/show/bfbKAaXWRJtf2cuopFcF/ | 17:08 |
| noonedeadpunk | nixbuilder: I don't think it's really smth wrong with the dvb | 17:15 |
| noonedeadpunk | *db | 17:15 |
| noonedeadpunk | I think I have quite some of such messages in logs | 17:15 |
| noonedeadpunk | it might be that wait_timout is unaligned... | 17:16 |
| noonedeadpunk | (though we've attempted to sync it lately) | 17:16 |
| jrosser | nixbuilder: you should try to get the db cluster working correctly according to the maintainance tasks document | 17:17 |
| jrosser | if you think that you have manila services deployed that 1) you don't want 2) you believe are somehow interfereing with the db, just stop those services | 17:18 |
| nixbuilder | jrosser: Thanks for the help! | 17:19 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_cinder master: Restart cinder-purge-deleted service only on abnormal exit https://review.opendev.org/c/openstack/openstack-ansible-os_cinder/+/900347 | 18:04 |
| opendevreview | Merged openstack/openstack-ansible-os_nova stable/yoga: Always disable libvirt default network https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/900191 | 18:35 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_octavia master: Adjust condition for availability_zone definition https://review.opendev.org/c/openstack/openstack-ansible-os_octavia/+/899127 | 19:11 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-repo_server master: Fix example playbook linters https://review.opendev.org/c/openstack/openstack-ansible-repo_server/+/900359 | 19:13 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-repo_server master: Ensure mounts are present only when they are expected to exist https://review.opendev.org/c/openstack/openstack-ansible-repo_server/+/899063 | 19:13 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-repo_server master: Cleanup upgrade tasks https://review.opendev.org/c/openstack/openstack-ansible-repo_server/+/899064 | 19:13 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_octavia master: Remove obsoleted provider drivers https://review.opendev.org/c/openstack/openstack-ansible-os_octavia/+/885519 | 19:15 |
| opendevreview | Merged openstack/ansible-role-zookeeper master: Add upgrade jobs for zookeeper https://review.opendev.org/c/openstack/ansible-role-zookeeper/+/897754 | 21:33 |
| opendevreview | Merged openstack/openstack-ansible master: Disable wheels build for metal AIO deployments https://review.opendev.org/c/openstack/openstack-ansible/+/899319 | 21:50 |
| jamesdenton | been a while, but here's what i was doing with MNAIO: https://github.com/busterswt/MNAIOv2 | 22:59 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!