| farbod | I am using stable/2023.2 branch | 06:50 |
|---|---|---|
| farbod | jrosser: How can i know why i am getting this? | 06:51 |
| farbod | Any idea? | 06:53 |
| farbod | Could be a problem with HAProxy? | 07:19 |
| farbod | here is my user configuration: https://paste.opendev.org/show/bo1jwGVb4xG302VaEZpK/ | 07:21 |
| farbod | and this is the user variables: https://paste.opendev.org/show/bXQ28f9DUOa77gMDRe9q/ | 07:21 |
| farbod | Also I am deploying on debian 11 | 07:31 |
| noonedeadpunk | farbod: so, what's the issue you have?:) | 08:14 |
| farbod | https://paste.opendev.org/show/bhMxyJTavEuxvV4sqZsT/ | 08:15 |
| noonedeadpunk | ok, I see. What's the content of /etc/apt/sources.list.d/MariaDB.list ? | 08:16 |
| noonedeadpunk | inside the galera container? | 08:16 |
| noonedeadpunk | As it seems that you don't have access to mariadb repositories that are used by default | 08:21 |
| noonedeadpunk | Could be some regional restrictions, for instance | 08:21 |
| hamidlotfi_ | noonedeadpunk: Maybe he used from VPN server and Mariadb servers blocked his connection. | 08:22 |
| noonedeadpunk | farbod: can you curl https://downloads.mariadb.com/MariaDB/mariadb-10.11.2/repo/debian/dists/buster/Release for isntance? As that's the repo that's configured by default | 08:23 |
| farbod | Let me check | 08:24 |
| noonedeadpunk | you can override the repository with variable galera_repo_url - like try to use some local mirror or smth... | 08:24 |
| noonedeadpunk | hamidlotfi_: oh, ok, didn't know that tbh | 08:25 |
| noonedeadpunk | though never tried that | 08:25 |
| hamidlotfi_ | noonedeadpunk: I hope you never need to use it. 😉 | 08:26 |
| noonedeadpunk | yeah, true | 08:28 |
| gokhan | good morning noonedeadpunk , I am upgrading yoga to antelope, but I am getting warning like [WARNING]: Failed to reset connection:Control socket connect(/root/.ansible/cp/bfe4c07115): No such file or directory | 08:36 |
| gokhan | when upgrading keystone it throws error when creating database play. İt can not ssh to infra node. ı am suspicious about upper warning | 08:37 |
| noonedeadpunk | gokhan: well, usually this issue is intermittent and IIRC was related to SSH connection persistance settings | 08:40 |
| noonedeadpunk | So I'd suggest try to re-run this individual playbook and see if issue is still there | 08:41 |
| noonedeadpunk | I think we landed https://opendev.org/openstack/openstack-ansible/commit/cbdba67ad0b5a3e29db390c8e6b66721719184c0 back in the days to address that at least partially | 08:42 |
| gokhan | I will try again | 08:44 |
| gokhan | noonedeadpunk, https://paste.openstack.org/show/beYK0iMCdu56PSvG1LDW/ again same eror | 08:47 |
| noonedeadpunk | gokhan: it kinda feels that issue is different here | 08:50 |
| noonedeadpunk | and does `mysql` works/connects from the utility container? | 08:52 |
| gokhan | it is not working ERROR 2026 (HY000): TLS/SSL error: Broken pipe (32) | 08:55 |
| noonedeadpunk | ok, so that's likely the reason | 08:57 |
| gokhan | there need to be a variable which disables ssl I think | 08:57 |
| gokhan | for galera | 08:58 |
| noonedeadpunk | is running `update-ca-certificates` fixes the issue? | 08:58 |
| noonedeadpunk | or well. It kinda depends on the outcome you want - to have mysql connection encrypted or not | 08:58 |
| gokhan | how can I update ca-certificates | 09:02 |
| noonedeadpunk | um. did it work or not?:) | 09:03 |
| noonedeadpunk | So it actually depends if root should be distributed or it's already there | 09:03 |
| gokhan | it didn't work | 09:04 |
| gokhan | ERROR 2026 (HY000): TLS/SSL error: wrong version number | 09:06 |
| noonedeadpunk | Hm, ok, I mistreated error I guess... | 09:10 |
| noonedeadpunk | Do you have SSL settings in `/etc/mysql/mariadb.cnf` ? | 09:10 |
| noonedeadpunk | So I'm actually trying to understand if SSL is configured for server at all and be used | 09:11 |
| noonedeadpunk | you should be able to set `galera_use_ssl: False` explicitly to disable SSL | 09:12 |
| gokhan | there is no ssl seettings in both server and client side | 09:14 |
| gokhan | default value is false https://github.com/openstack/openstack-ansible-galera_server/blob/stable/2023.1/defaults/main.yml#L239C1-L240C1 | 09:17 |
| gokhan | but ssl verify is true | 09:17 |
| noonedeadpunk | gokhan: well, there's an override in openstack_ansible/inventory/group_vars | 09:21 |
| noonedeadpunk | https://opendev.org/openstack/openstack-ansible/src/branch/master/inventory/group_vars/all/infra.yml#L52 | 09:22 |
| gokhan | I see, I need rerun haproxy, galera and utility | 09:25 |
| gokhan | now the error is ERROR 1130 (HY000): Proxy header is not accepted from 10.13.201.21 | 09:28 |
| *** tosky_ is now known as tosky | 10:19 | |
| noonedeadpunk | gokhan: sorry had a meeting | 10:42 |
| noonedeadpunk | 10.13.201.21 is not a VIP, is it? | 10:42 |
| gokhan | noonedeadpunk, no worries :) no it is not wip, it is infra1 ip | 10:43 |
| gokhan | it is haproxy issue, when I changed ip to galera, it work | 10:44 |
| noonedeadpunk | so... is it included in `proxy-protocol-networks` in /etc/mysql/mariadb.cnf? | 10:44 |
| gokhan | yes it is not inclued, ips in there are wrong, it needs to be br-mgmt ips, but ips in protocol network ips are public endpoints | 10:49 |
| noonedeadpunk | um, yes, they should be br-mgmt ips | 10:50 |
| noonedeadpunk | and it should take mgmt ip: https://opendev.org/openstack/openstack-ansible/src/branch/master/inventory/group_vars/galera_all.yml#L43-L49 | 10:50 |
| noonedeadpunk | or well..... | 10:50 |
| noonedeadpunk | not on yoga, but on antelope | 10:51 |
| noonedeadpunk | and if you're having management_address set to smth different then br-mgmt - that could be the reason.... | 10:52 |
| noonedeadpunk | But it comes from the inventory iirc | 10:52 |
| gokhan | yes, I have overrided it in user variables for before versions :( thanks noonedeadpunk :) | 10:56 |
| jrosser | https://github.com/ceph/ceph-ansible/commit/a9d1ec844d24fcc3ddea7c030eff4cd6c414d23d | 11:12 |
| gokhan | noonedeadpunk, where is management_address is assigned ? it is wrong in my deployment | 11:13 |
| noonedeadpunk | it's coming from openstack_inventory. I think that inventory taking from `is_management_address: true` in provider_networks from openstack_user_config | 11:15 |
| noonedeadpunk | Again? | 11:16 |
| noonedeadpunk | gokhan: or well, is_container_address: true should also work as fallback | 11:19 |
| gokhan | noonedeadpunk, https://paste.openstack.org/show/bvdVttJZxqsglndTfNCu/ it is in there but it didn't work | 11:26 |
| gokhan | management_adress is different in our inventory | 11:33 |
| noonedeadpunk | that's weird | 11:35 |
| noonedeadpunk | gokhan: but, does host defenitions in openstack_user_config contain IP from the same subnet, or you're using a different one for SSH? | 11:38 |
| noonedeadpunk | as, for instance, we also doing smth like this here: https://opendev.org/openstack/openstack-ansible/src/branch/master/doc/source/reference/inventory/configure-inventory.rst#having-ssh-network-different-from-openstack-management-network | 11:40 |
| noonedeadpunk | when we want ansible to access hosts through different network rather then via management one | 11:41 |
| noonedeadpunk | so ansible_host != management_address | 11:41 |
| gokhan | noonedeadpunk, yes you are right, host definitions in openstack user config contains ip from different subnet, we are using a vm for deployment host. it seems we need to override it from user_variables | 12:01 |
| jrosser | it doesnt really matter about using a vm for a deployment host | 12:10 |
| jrosser | you need connectivity / routing / bastion of some kind between the deployment host and the host definitions in openstack_user_config | 12:12 |
| spatel | Folks, looking for help to understand Ceph rgw eraser coding.. and bucket placement. I have 2 node with some HDD and I want EC pool to use HDD nodes. I did this - https://paste.opendev.org/show/bbk0QGA0zwpgTbsQLDFd/ | 14:16 |
| spatel | As per my understanding ahm.rgw.buckets.data should live on HDD right? | 14:17 |
| jrosser | spatel: it depends what performance you need where you put the pools | 14:57 |
| spatel | I want to put pool on HDD nodes | 14:57 |
| jrosser | if you only have two hosts for an hdd ec pool thats really not enough? not even enough for replica-3 | 14:58 |
| spatel | Tomorrow adding one more node | 15:00 |
| spatel | so I will total have 3 nodes | 15:00 |
| jrosser | so you want ahm.rgw.buckets.data on the EC pool and leave everything else replicated i think | 15:04 |
| spatel | I want ahm.rgw.buckets.data on EC pool using HDD | 15:04 |
| jrosser | and the index wants to be on the fastest thing you have | 15:04 |
| spatel | You are correct index on SSD which is by default SSD | 15:05 |
| spatel | How do I tell ceph to use .data pool to HDD | 15:11 |
| mgariepy | it's defined by the crush rule for the pool. | 15:38 |
| spatel | My my "ceph orch ps" command hanging after reboot mgr daemon :( | 15:40 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/ansible-hardening master: Use replace module instead of lineinfile for disabling dynamic motd https://review.opendev.org/c/openstack/ansible-hardening/+/907095 | 16:24 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/ansible-hardening master: Use replace module instead of lineinfile for disabling dynamic motd https://review.opendev.org/c/openstack/ansible-hardening/+/907095 | 16:26 |
| jrosser | noonedeadpunk: i looked at how the mariadb repo setup script works, and it points to slightly different urls, like https://paste.opendev.org/show/b3t86KRzC1mT7aCqgTOI/ | 16:52 |
| jrosser | did you see that before | 16:53 |
| jrosser | and browsable version like https://dlm.mariadb.com/browse/mariadb_server/228/1915/ | 16:53 |
| noonedeadpunk | So, I think it leads to some kind of mirrors by default | 16:54 |
| noonedeadpunk | I had some talk with mariadb folks veeery long ago and they said kinda - whatever. But not 100% sure | 16:56 |
| jrosser | yeah it is very confusing about what the "right" repo url to use | 16:58 |
| jrosser | because also they give https://mirrors.xtom.nl/mariadb/repo/10.11/ubuntu/ as a mirror location | 16:58 |
| noonedeadpunk | Well. Kinda as long as we don't have any stability issues with original one, potentially there's no reason to switch... | 17:04 |
| noonedeadpunk | The only very concerning thing is MariaDB was mariadb in rest places, which makes switching between mirrors very inconvenient | 17:05 |
| noonedeadpunk | as you need to have full url then | 17:05 |
| jrosser | indeed - i double checked due to someone earlier having repo trouble | 17:05 |
| jrosser | and interesting to see how many different places this can come from :/ | 17:05 |
| TheCompWiz | I'm stuck. Neutron doesn't seem to want to start on my compute node with the error "ERROR neutron.plugins.ml2.managers [-] No type driver for tenant network_type: vxlan. Service terminated!" ... and if I take a look at "ml2_conf.ini" I can see that the vxlan is missing from "type_drivers". This is all ansible deployed, but I cann't figure out why it would be missing the vxlan. (it has geneve,vlan,flat) | 20:44 |
| admin1 | TheCompWiz, you can pass it via neutron_ml2_drivers_type variable | 21:01 |
| admin1 | also on the user_config, you have type with vxlan and the range for the vxlan ids to be used | 21:02 |
| jrosser | what does it say in ml2_conf.ini for tenant_network_types? | 21:08 |
| TheCompWiz | admin1: Shouldn't the openstack-ansible be smart enough to add the vxlan to the drivers config? | 21:25 |
| TheCompWiz | jrosser: tenant_network_types does include the vxlan | 21:25 |
| TheCompWiz | tenant_network_types = vxlan,flat | 21:26 |
| jrosser | this is all determined by what you put in openstack_user_config | 21:28 |
| jrosser | with "modern" openstack the network uses ovn as the ml2 driver | 21:28 |
| TheCompWiz | jrosser: what/where should the vxlan be placed? the provider_networks for vxlan already has the type set to "vxlan" | 21:29 |
| jrosser | and in ovn the default for tunneled tenant networks is geneve rather than vxlan | 21:29 |
| jrosser | so the question really is if you are wanting vxlan at all | 21:29 |
| TheCompWiz | jrosser: sadly, I don't know the difference in capabilities between vxlan and ovn | 21:30 |
| jrosser | they are not comparable things | 21:30 |
| jrosser | vxlan is an network overlay technique, similar but different to geneve | 21:30 |
| jrosser | ovn is a framework for software defined networks | 21:31 |
| jrosser | https://www.ovn.org/en/ | 21:31 |
| TheCompWiz | ok... so are you suggesting I should reconfigure to use geneve? | 21:32 |
| TheCompWiz | and just abandon vxlan? | 21:32 |
| jrosser | it depends | 21:32 |
| jrosser | but lets say, if you were to build an openstack-ansible all-in-one today from the latest release, the default would be geneve | 21:33 |
| jrosser | but you are free to use whatever you need | 21:33 |
| TheCompWiz | ok... so do you know of a reason why the vxlan wouldn't be included in the ml2 drivers? | 21:34 |
| jrosser | vxlan might interoperate with some physical switches in certain use cases, but those are niche | 21:34 |
| jrosser | what ends up in the ml2 config file is determined by what you put in openstack_user_config | 21:34 |
| jrosser | and any variables you have overidden for neutron | 21:35 |
| TheCompWiz | I have nothing overriden in neutron... that's what bugs me. | 21:35 |
| jrosser | then can you share your openstack_user_config in a paste somewhere? | 21:36 |
| TheCompWiz | https://paste.openstack.org/show/bIp5WearMO1rAXHF54qz/ | 21:37 |
| jrosser | well, this looks like a setup for ovs rather than ovn | 21:40 |
| jrosser | due to specifying neutron_openvswitch_agent in places there | 21:41 |
| jrosser | which network driver would you like to use? | 21:41 |
| TheCompWiz | Honestly, I don't know what I don't know. I'm not even sure how to answer that. | 21:42 |
| admin1 | if you use pvn, it defaults to geneve .. and its good and its what the industry is moving towards | 21:42 |
| admin1 | ovn* | 21:42 |
| jrosser | TheCompWiz: did you build an all-in-one? | 21:43 |
| jrosser | ^ i think we may have discussed this before | 21:43 |
| TheCompWiz | jrosser: I haven't. | 21:43 |
| TheCompWiz | I am hesitant to use the all-in-one... because I wouldn't learn much. | 21:44 |
| TheCompWiz | more than likely, I'll want OVN ultimately. | 21:45 |
| jrosser | if you can make a ~8 core 8G VM anywhere, you can build an all-in-one | 21:45 |
| jrosser | the trouble with openstack is that it is a collection of software | 21:46 |
| jrosser | each part has <N> options for which driver or storage vendor or whatever | 21:46 |
| jrosser | so the matrix of all possibilties is gigantic, and really for practical purposes there are a set of choices that make the most sense | 21:46 |
| TheCompWiz | jrosser: I understand that. but using a pre-built config to do an all-in-one skips all the "learning" part of "learning" | 21:47 |
| TheCompWiz | and I'm still in the learning phase of the config. | 21:47 |
| jrosser | one of the things that the all-in-one does is make a bunch of those "sensible defaults" for you, and you get to a reference point in openstack-ansible terms very quickly | 21:47 |
| jrosser | the config is unfortunately pretty infinite | 21:48 |
| jrosser | you can put any key/value you like in any config file on any of your hosts | 21:48 |
| TheCompWiz | jrosser: True. Just out of curiosity... how difficult is it to take an all-in-one setup... and then later break it down into separate machines? | 21:48 |
| jrosser | it's possible | 21:48 |
| jrosser | one of the things that the all-in-one does is hide everything behind one interface and one ip | 21:49 |
| jrosser | primarily to make it useful for CI and as a developer environment, rather than for actual deployments | 21:49 |
| jrosser | it is the very same thing that has to pass tests when we merge code | 21:49 |
| TheCompWiz | "possible" sounds more like... "yeah... you could, but you'd be shooting yourself in the face by the end of it." | 21:50 |
| jrosser | so there is some networking wierdness there with some NAT and iptables | 21:50 |
| jrosser | imho you should treat the AIO as a disposable reference environment | 21:50 |
| TheCompWiz | jrosser: I can understand that. That's what I assumed it was. | 21:50 |
| TheCompWiz | something along the lines of "hey... see... it works.... now delete it and try it the better way" | 21:51 |
| jrosser | but for example, you would get a known good openstack_user_config and see neutron working | 21:51 |
| jrosser | if you have access to a virtualisation environment of any kind then you can use that | 21:51 |
| jrosser | but anyway - your openstack_user_config is currently setup for OVS | 21:52 |
| jrosser | and it defines vxlan type network | 21:52 |
| jrosser | but the default in recent openstack-ansible is for OVN and geneve | 21:52 |
| TheCompWiz | ok. | 21:52 |
| jrosser | so i think thats probably the basis of the trouble here, that the default setup of the ansible roles does not match up with what openstack_user_config says | 21:53 |
| TheCompWiz | jrosser: ok. Time to nuke/pave once more. ... this time with more gusto :D | 21:53 |
| jrosser | if you want to look at some config files, all the CI results are publically accessible | 21:54 |
| TheCompWiz | thanks for your consult jrosser . | 21:54 |
| jrosser | you would start here https://review.opendev.org/q/project:openstack/openstack-ansible+status:open | 21:54 |
| jrosser | then we can pick the first thing in the list https://review.opendev.org/c/openstack/openstack-ansible/+/906750 | 21:54 |
| jrosser | oh, bad example, thats just documentation update | 21:55 |
| jrosser | third one https://review.opendev.org/c/openstack/openstack-ansible/+/906148 | 21:55 |
| jrosser | press "zuul summary" https://review.opendev.org/c/openstack/openstack-ansible/+/906148?tab=change-view-tab-header-zuul-results-summary | 21:55 |
| jrosser | pick the environment you are interested in, perhaps ubuntu jammy with LXC containers https://zuul.opendev.org/t/openstack/build/215d36c446644d489f8f91a00ffca1a0 | 21:56 |
| jrosser | choose logs https://zuul.opendev.org/t/openstack/build/215d36c446644d489f8f91a00ffca1a0/logs | 21:56 |
| jrosser | then we can navigate to the ml2_conf.ini https://zuul.opendev.org/t/openstack/build/215d36c446644d489f8f91a00ffca1a0/log/logs/etc/host/neutron/plugins/ml2/ml2_conf.ini.txt | 21:57 |
| jrosser | or see the openstack_user_config https://zuul.opendev.org/t/openstack/build/215d36c446644d489f8f91a00ffca1a0/log/logs/etc/host/openstack_deploy/openstack_user_config.yml.txt | 21:57 |
| TheCompWiz | wouldn't the ml2_conf.ini be auto-generated by ansible? | 21:57 |
| jrosser | yes all these files are grabbed off the test node once the CI job finishes | 21:57 |
| jrosser | otherwise there is no way to debug :) | 21:58 |
| jrosser | but these jobs all run the AIO | 21:58 |
| jrosser | so if you want to poke around the config / resulting files without running it yourself, it's all there to see | 21:58 |
| TheCompWiz | much appreciated. | 22:03 |
| TheCompWiz | on a side note... is there a bootcamp you would recommend or online tutorial for someone who has just started getting his feet wet with openstack? | 22:04 |
| noonedeadpunk | there're quite some ops meetups and/or openinfra days during May | 22:24 |
| noonedeadpunk | TheCompWiz: I can also check on Cleura Academy - though not super cheap: https://shop.cleura.com/ | 22:26 |
| noonedeadpunk | hm, it looks smth is off with tempest excludelist patterns.... | 22:49 |
| noonedeadpunk | https://zuul.opendev.org/t/openstack/build/21874c048592450fb6582f8f363193fd/log/logs/etc/host/openstack_deploy/user_variables_ironic.yml.txt#9 | 22:49 |
| noonedeadpunk | But then we apparently failing on supposedly disabled test: https://8a5247649d3115bcd2e6-5f55b49caeaffd6ebf7a83fc93de4834.ssl.cf1.rackcdn.com/906353/2/check/openstack-ansible-deploy-aio_metal-ubuntu-jammy/21874c0/logs/openstack/aio1-utility/stestr_results.html | 22:50 |
| noonedeadpunk | or smth off with tempest itself - I in fact can't find the include/exclude lists generated in logs :( | 22:51 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Restore Ironic test_chassis_node_list test https://review.opendev.org/c/openstack/openstack-ansible/+/907154 | 22:54 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_ironic master: Allow to extend default ironic_driver_types https://review.opendev.org/c/openstack/openstack-ansible-os_ironic/+/907155 | 22:54 |
| noonedeadpunk | I'm kinda sure that magnum fails due to the same reason eventually | 22:56 |
| noonedeadpunk | Tests that are failing here: https://0f52ec42c8d2def0cbb2-9a428750761834a84d7f79789f3b4734.ssl.cf2.rackcdn.com/901185/7/check/openstack-ansible-deploy-aio_metal-ubuntu-jammy/25f4c0a/logs/openstack/aio1-utility/stestr_results.html | 22:57 |
| noonedeadpunk | are excluded https://opendev.org/openstack/openstack-ansible/src/branch/master/tests/roles/bootstrap-host/templates/user_variables_magnum.yml.j2#L56-L59 | 22:57 |
| noonedeadpunk | feels like we landed something.... but not sure... | 22:58 |
| * noonedeadpunk spawning aio | 23:03 | |
| * noonedeadpunk and goes to bed :p | 23:03 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!