moha7 | Hi there; I did a new deploy. But still have problem with up-ing the port 9696! Galera is active and running but with this error in the status output: [ERROR] Invalid (old?) table or database name '#tmp', --> http://ix.io/4jFA | 08:22 |
---|---|---|
moha7 | neutron.slice had this error: ValueError: :6642: bad peer name format, but after restarting this service, it's now active and running | 08:24 |
moha7 | port 9696 is not up on ct1-neutron-server-container-8d8a1c64 | 08:24 |
moha7 | I can not login to Horizon dashboard because this error: https://ibb.co/pv52pCT | 08:25 |
kleini | I delete those tmp files in /var/lib/mysql manually before starting MariaDB. | 09:05 |
noonedeadpunk | moha7: what are values of tmpdir and ignore_db_dirs you have in /etc/mysql/my.cnf ? | 10:13 |
moha7 | tmpdir = /var/lib/mysql/#tmp | 10:26 |
moha7 | ignore_db_dirs = '#tmp',lost+found | 10:27 |
moha7 | noonedeadpunk: ^ | 10:27 |
moha7 | openstack_user_config.yml: http://ix.io/4jB4 | 10:28 |
moha7 | user_variables.yml: http://ix.io/4jGa | 10:28 |
noonedeadpunk | that looks quite legit | 10:28 |
noonedeadpunk | give me couple of mins to setup multinode with galera to check that | 10:29 |
noonedeadpunk | As I'm quite sure that these #tmp shouldn't be an issue as they're ignored | 10:29 |
moha7 | how you do that in mins! It takes exactly 5 hours each time I go for fresh deploy | 10:30 |
kleini | I had issues with those tmp folders, too. Don't actually remember, if they are named "#tmp" or just "tmp.something". | 10:30 |
noonedeadpunk | kleini: well, we've defined couple of vars by default to avoid such issues | 10:31 |
noonedeadpunk | moha7: `./scripts/gate-check-commit.sh aio_infra_lxc deploy source` :D | 10:32 |
noonedeadpunk | but it's all in one at the end | 10:32 |
moha7 | ct1-galera-container --> systemctl status mariadb --> http://ix.io/4jGb | 10:33 |
moha7 | ct3-galera-container --> systemctl status mariadb --> http://ix.io/4jGc | 10:33 |
noonedeadpunk | well, I would totally suggest cleaning up /var/lib/mysql from these `tmp.tyvhAZxFbU` or `#mysql50#tmp.tyvhAZxFbU` - the only "valid" thing there from temp stuff should be `#tmp` folder | 10:37 |
noonedeadpunk | I don't think it's critical issue though that prevents you from having healthy cluster. | 10:37 |
noonedeadpunk | So. what does mysql -e "SHOW GLOBAL STATUS LIKE 'wsrep_%';" says? | 10:38 |
moha7 | >> http://ix.io/4jGe | 10:43 |
moha7 | after removing tmp.*, noe there's only this one in the status output: `[ERROR] Invalid (old?) table or database name '#tmp'` | 10:46 |
noonedeadpunk | moha7: based on that galera is healthy | 10:46 |
moha7 | now* | 10:46 |
noonedeadpunk | I will check this #tmp thingy in a while... That sounds like some bug, but not sure when ignore_db_dirs stopped working.... | 10:47 |
noonedeadpunk | so back to the neutron thing? | 10:48 |
noonedeadpunk | You said it's active and running now, but it's not having port 9696? | 10:49 |
noonedeadpunk | *not listens | 10:49 |
moha7 | yes | 11:00 |
moha7 | Oh, no! | 11:01 |
moha7 | w8 | 11:01 |
moha7 | neutron.slice is active, but not running with same error like yesterday: | 11:02 |
moha7 | still having this error: ValueError: :6642: bad peer name format | 11:04 |
moha7 | ==> http://ix.io/4jGm | 11:05 |
noonedeadpunk | well, ovn is not my strength (yet). I'd summon jamesdenton for that. But he said it's related to northd | 11:07 |
moha7 | seems something else other than network-northd_hosts and network-gateway_hosts is missing in `user_variables.yml` and `openstack_user_config.yml` | 11:07 |
moha7 | noonedeadpunk: Thanks for your help | 11:07 |
moha7 | jamesdenton: Should be there an neutron-ovn-northd container in `lxc-ls` output? | 11:41 |
noonedeadpunk | yes, there should | 11:54 |
noonedeadpunk | don't you have that? | 11:54 |
opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Remove ANSIBLE_ACTION_PLUGINS override https://review.opendev.org/c/openstack/openstack-ansible/+/824595 | 12:51 |
opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Add tempest repos to required-repos https://review.opendev.org/c/openstack/openstack-ansible/+/834289 | 12:52 |
opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Respect haproxy_bind_internal_lb_vip_* variables https://review.opendev.org/c/openstack/openstack-ansible/+/852243 | 12:52 |
jamesdenton | good morning | 13:28 |
jamesdenton | moha7 i suspect that at a minimum, you're missing the "network-northd_hosts" group from openstack_user_config.yml. And if you want gateway node functionality, you'll want the "network-gateway_hosts" group, too. I'll work on fixing the northd group piece, as ideally it wouldn't be necessary IMO | 13:30 |
cloudnull | Merry Christmas / happy holidays all - I hope you’re all doing wonderful. | 14:17 |
jamesdenton | cloudnull! same to you | 14:33 |
noonedeadpunk | thanks cloudnull, merry christmas and happy new year for you too! | 14:39 |
noonedeadpunk | huh, I somehow generated desired inventory using env.vars during summer and now I can't reproduce.... | 14:40 |
moha7 | noonedeadpunk: no, there's not ant neutron-ovn-northd! --> lxc-ls: http://ix.io/4jdC | 14:48 |
moha7 | penstack_user_config.yml: http://ix.io/4jB4 | 14:49 |
moha7 | user_variables.yml: http://ix.io/4jGa | 14:49 |
moha7 | jamesdenton: ^ | 14:49 |
noonedeadpunk | well, network-northd_hosts is defined there | 14:50 |
noonedeadpunk | isn't northd in `./scripts/inventory-manage.py -g` ? | 14:53 |
jamesdenton | hmm | 14:58 |
jamesdenton | moha7 this is zed/master, correct? | 14:59 |
moha7 | git branch --> * master | 15:00 |
moha7 | I got it by: | 15:01 |
moha7 | `git clone -b master https://opendev.org/openstack/openstack-ansible /opt/openstack-ansible && cd $_ && scripts/bootstrap-ansible.sh` | 15:01 |
jamesdenton | ok, thanks | 15:01 |
jamesdenton | and this was a fresh deployment? | 15:02 |
jamesdenton | Is there anything in these groups? neutron_ovn_northd or neutron_ovn_northd_container | 15:03 |
jamesdenton | you can prob use "./scripts/inventory-manage.py -g" like noonedeadpunk pointed out, or jq: cat /etc/openstack_deploy/openstack_inventory.json | jq .neutron_ovn_northd_container; cat /etc/openstack_deploy/openstack_inventory.json | jq .neutron_ovn_northd | 15:04 |
moha7 | Yup, a fresh deplyment. `inventory-manage.py -g`: http://ix.io/4jHx | 15:11 |
moha7 | "neutron_ovn_northd_container": { "hosts": [] } | 15:14 |
moha7 | jamesdenton: those groups are empty! | 15:14 |
jamesdenton | yes, very odd | 15:14 |
jamesdenton | one sec | 15:14 |
noonedeadpunk | oh.... My issue wiht inventory caused by this https://opendev.org/openstack/openstack-ansible/commit/c76a4d5dfe6c22b17064a11d7c60aa5b629ab439 | 15:19 |
noonedeadpunk | (I guess) | 15:19 |
noonedeadpunk | damn... | 15:19 |
jamesdenton | :| | 15:21 |
noonedeadpunk | jrosser: maybe you do recall this... I do how we discussed that back then... So basically my problem is related to AZ implementation | 15:22 |
jamesdenton | moha7 i am attempting to recreate your issue, takes a little bit of time to stand it up | 15:22 |
moha7 | Thanks | 15:23 |
noonedeadpunk | So I had config like that https://paste.openstack.org/show/b6R6axfFPZMsrGgU7jo4/ | 15:24 |
noonedeadpunk | and in inventory had smth like https://paste.openstack.org/show/bl11c9SwQgDS2TNBp01F/ | 15:26 |
noonedeadpunk | But now I'm not understanding how I can properly add all host containers | 15:27 |
noonedeadpunk | hm... | 15:31 |
noonedeadpunk | jsut reverting code doesn't help... | 15:52 |
noonedeadpunk | damn | 15:52 |
noonedeadpunk | I'm soooo tired of figuring out the way... Probably I should have focused on implementing a key or smth that would allow do what I want | 15:52 |
opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Update tox.ini https://review.opendev.org/c/openstack/openstack-ansible/+/868680 | 15:56 |
opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Update tox.ini https://review.opendev.org/c/openstack/openstack-ansible/+/868680 | 15:57 |
jamesdenton | moha7 i'm not able to recreate your issue so far. Can you confirm the contents of /etc/openstack_deploy/env.d and conf.d? | 16:04 |
opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Update tox.ini https://review.opendev.org/c/openstack/openstack-ansible/+/868680 | 16:18 |
jamesdenton | noonedeadpunk is centos still borked? | 16:18 |
noonedeadpunk | just fixed | 16:18 |
noonedeadpunk | like hour ago or smth | 16:19 |
jamesdenton | oh nice | 16:19 |
noonedeadpunk | But now tox 4 is borked :D | 16:19 |
jamesdenton | oh no | 16:19 |
noonedeadpunk | (like docs, renos, functional leftovers) | 16:19 |
noonedeadpunk | I'm going to check through and propose patches | 16:19 |
jrosser | i am pretty sure that the changes i did to the inventory code were right | 16:20 |
noonedeadpunk | I'm not saying they're not... Just trying to realize what I did during summer that I cna't reproduce anymore | 16:24 |
noonedeadpunk | Basically what I want - is when host added to another group defined in env.d - it's containers should appear there as well | 16:25 |
noonedeadpunk | So given I define az1_all / az1_hosts / az1_containers and define like comput01, control01, net01 to az1_hosts, I want all containers to appear in az1_containers this way or another | 16:26 |
noonedeadpunk | And I somehow managed that, but control01_host-containers were children of az1_containers that I assume made it work... | 16:26 |
noonedeadpunk | (though reverting code didn't help to be fair) | 16:27 |
noonedeadpunk | But I wonder if it's simply not possible with code we have today or I just forgot how to do that, despite have old inventory file along with env.d..... | 16:28 |
noonedeadpunk | Anyway | 16:28 |
jrosser | well, something i think i have seen is that the output of inventory-manage might not be as complete as what ansible actually sees | 16:28 |
jrosser | but thats possibly something quite different again | 16:28 |
noonedeadpunk | Yeah, but I'm checking json directly | 16:28 |
noonedeadpunk | (as I'm aware about inventory-manage already) | 16:29 |
noonedeadpunk | So I'm either brainfarting or it's simply not possible in an easy wya | 16:30 |
noonedeadpunk | But I can't get on with that as I see old generated inventory that had this working in the sandbox... | 16:31 |
jrosser | well - i have done this sort of thing only with compute nodes to make new groups | 16:34 |
jrosser | like "with GPU" or whatever | 16:34 |
jrosser | so never with anything on the control plane | 16:35 |
*** dviroel_ is now known as dviroel | 16:38 | |
opendevreview | Dmitriy Rabotyagov proposed openstack/ansible-role-uwsgi master: Update tox.ini to work with 4.0 https://review.opendev.org/c/openstack/ansible-role-uwsgi/+/868683 | 16:53 |
opendevreview | Dmitriy Rabotyagov proposed openstack/ansible-role-pki master: Update tox.ini to work with 4.0 https://review.opendev.org/c/openstack/ansible-role-pki/+/868684 | 16:53 |
opendevreview | Dmitriy Rabotyagov proposed openstack/ansible-role-systemd_networkd master: Update tox.ini to work with 4.0 https://review.opendev.org/c/openstack/ansible-role-systemd_networkd/+/868685 | 16:53 |
opendevreview | Dmitriy Rabotyagov proposed openstack/ansible-role-qdrouterd master: Update tox.ini to work with 4.0 https://review.opendev.org/c/openstack/ansible-role-qdrouterd/+/868686 | 16:53 |
opendevreview | Dmitriy Rabotyagov proposed openstack/ansible-role-systemd_mount master: Update tox.ini to work with 4.0 https://review.opendev.org/c/openstack/ansible-role-systemd_mount/+/868687 | 16:53 |
opendevreview | Dmitriy Rabotyagov proposed openstack/ansible-role-python_venv_build master: Update tox.ini to work with 4.0 https://review.opendev.org/c/openstack/ansible-role-python_venv_build/+/868688 | 16:53 |
opendevreview | Dmitriy Rabotyagov proposed openstack/ansible-role-systemd_service master: Update tox.ini to work with 4.0 https://review.opendev.org/c/openstack/ansible-role-systemd_service/+/868689 | 16:53 |
opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-apt_package_pinning master: Update tox.ini to work with 4.0 https://review.opendev.org/c/openstack/openstack-ansible-apt_package_pinning/+/868690 | 16:53 |
opendevreview | Dmitriy Rabotyagov proposed openstack/ansible-hardening master: Update tox.ini to work with 4.0 https://review.opendev.org/c/openstack/ansible-hardening/+/868691 | 16:53 |
opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-ceph_client master: Update tox.ini to work with 4.0 https://review.opendev.org/c/openstack/openstack-ansible-ceph_client/+/868692 | 16:53 |
opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-haproxy_server master: Update tox.ini to work with 4.0 https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/868693 | 16:53 |
opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-lxc_container_create master: Update tox.ini to work with 4.0 https://review.opendev.org/c/openstack/openstack-ansible-lxc_container_create/+/868694 | 16:53 |
opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-memcached_server master: Update tox.ini to work with 4.0 https://review.opendev.org/c/openstack/openstack-ansible-memcached_server/+/868695 | 16:53 |
opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-lxc_hosts master: Update tox.ini to work with 4.0 https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/868696 | 16:53 |
opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-galera_server master: Update tox.ini to work with 4.0 https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/868697 | 16:53 |
opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-openstack_hosts master: Update tox.ini to work with 4.0 https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/868698 | 16:53 |
opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-openstack_openrc master: Update tox.ini to work with 4.0 https://review.opendev.org/c/openstack/openstack-ansible-openstack_openrc/+/868699 | 16:53 |
opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-ops master: Update tox.ini to work with 4.0 https://review.opendev.org/c/openstack/openstack-ansible-ops/+/868700 | 16:53 |
opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_manila master: Update tox.ini to work with 4.0 https://review.opendev.org/c/openstack/openstack-ansible-os_manila/+/868702 | 16:53 |
opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_magnum master: Update tox.ini to work with 4.0 https://review.opendev.org/c/openstack/openstack-ansible-os_magnum/+/868701 | 16:53 |
opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_murano master: Update tox.ini to work with 4.0 https://review.opendev.org/c/openstack/openstack-ansible-os_murano/+/868703 | 16:53 |
opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_senlin master: Update tox.ini to work with 4.0 https://review.opendev.org/c/openstack/openstack-ansible-os_senlin/+/868704 | 16:53 |
opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_sahara master: Update tox.ini to work with 4.0 https://review.opendev.org/c/openstack/openstack-ansible-os_sahara/+/868705 | 16:53 |
opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_swift master: Update tox.ini to work with 4.0 https://review.opendev.org/c/openstack/openstack-ansible-os_swift/+/868706 | 16:53 |
opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-plugins master: Update tox.ini to work with 4.0 https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/868707 | 16:53 |
opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-rabbitmq_server master: Update tox.ini to work with 4.0 https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/868708 | 16:53 |
opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-tests master: Update tox.ini to work with 4.0 https://review.opendev.org/c/openstack/openstack-ansible-tests/+/868709 | 16:53 |
opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-repo_server master: Update tox.ini to work with 4.0 https://review.opendev.org/c/openstack/openstack-ansible-repo_server/+/868710 | 16:53 |
BobZAnnapolis | folks, running some RefStack tests - getting this error msg for a handful of tests "Missing Identity Admin API credentials in configuration" - nothing in the RefStack documentation (that is obvious to this newbie to RefStack) that says where/how to set these up, tia | 16:56 |
jrosser | BobZAnnapolis: i would start here https://codesearch.opendev.org/?q=Missing%20Identity%20Admin%20API%20credentials%20in%20configuration | 16:59 |
jrosser | which takes you here https://opendev.org/openstack/tempest/src/branch/master/tempest/test.py#L282-L288 | 17:00 |
noonedeadpunk | yeah, GPU is easy | 17:01 |
noonedeadpunk | Eventually, Damian has found that for GPU and stuff you even don't need to define custom env.d | 17:01 |
noonedeadpunk | as adding anyting to openstack_user_config that ends with `-hosts` will result in creating such group in inventory | 17:02 |
noonedeadpunk | It won't create containers part - but for computes that's not needed | 17:02 |
jrosser | yes we do that a lot with -hosts things | 17:02 |
jrosser | it is not obvious | 17:02 |
noonedeadpunk | I learned abut that literally last week | 17:03 |
noonedeadpunk | As was creating env.d for everything | 17:03 |
noonedeadpunk | I didn't dare to try tbh | 17:04 |
BobZAnnapolis | jrosser: thumbs up, thanks - you beat me to it :-) i was headed into the code as well, there are 'admin' parameters in the etc/accounts.yaml file but looks like not enough - thank you | 17:07 |
moha7 | `ls /etc/openstack_deploy/env.d` --> aio_metal.yml.example calico.yml.example cinder-volume.yml.container.example extra_container.yml.example neutron.yml nova.yml | 17:29 |
moha7 | conf.d/ --> http://ix.io/4jI8 --> all files have .aio or .example --> then there's not any actove config file in ../conf.d/. | 17:32 |
moha7 | jamesdenton: ^ | 17:32 |
jamesdenton | can you show be env.d/neutron.yml? | 17:33 |
jamesdenton | *me | 17:33 |
jamesdenton | and nova | 17:33 |
moha7 | I copied contents for them from this page: https://satishdotpatel.github.io/openstack-ansible-multinode-ovn/ | 17:34 |
jamesdenton | ahh ok, you won't need them anymore | 17:34 |
jamesdenton | since OVN is default for OSA now, those env.d overrides are not necessary and prob not compatible | 17:35 |
moha7 | Ah, noonedeadpunk mentioned yesterday, but I think he means that they are ignored. My bad | 17:35 |
jamesdenton | so if you rename neutron.yml and nova.yml or move them, try regenerating the inventory with /opt/openstack-ansible/inventory/dynamic_inventory.py | 17:35 |
moha7 | jamesdenton: Now the group `neutron_ovn_northd_container` has values in that *json* file. Going to deploy from scratch: `/opt/openstack-ansible/playbooks/setup-everything.yml` after destroying current LXCes. | 18:01 |
jamesdenton | no need to destory current containers if thats the only change | 18:01 |
jamesdenton | as long as you didn't make changes to the inventory (other than addition of northd) you should be OK | 18:02 |
jamesdenton | you would want to rerun setup-hosts.yml and you can use --limit <northd container>, to get it setup, and then rerun neutron plays. Or, setup-everything is fine ,too | 18:02 |
moha7 | Thanks for this time-saving tip | 18:04 |
jamesdenton | sure, just double check after running setup hosts that the container shows up. it should | 18:06 |
moha7 | Am I using the command in wrong way: `openstack-ansible /opt/openstack-ansible/playbooks/setup-hosts.yml --limit neutron_ovn_northd_container` because it's failed with: http://ix.io/4jIm --> 'dict object' has no attribute 'distribution'. The error appears to be in '/etc/ansible/roles/lxc_container_create/tasks/lxc_container_create_dir.yml' | 18:17 |
jrosser | moha7: that looks like you are missing facts gathering for the physical hosts | 18:28 |
jrosser | they are only valid in the cache for (I think) 24 hours | 18:29 |
moha7 | One or two hours have passed since 24 hours. Should I install from the beginning, or is there a way to generate the facts twice? | 18:34 |
jrosser | none of this ever really involves starting again from the beginning, only for massive mess | 18:35 |
jrosser | perhaps expand the limit to include infra_all or some other group including the physical hosts | 18:37 |
moha7 | Well, I run it with `--limit network-infra_all` | 18:44 |
jamesdenton | moha7 is it running now? sorry, stepped away | 19:21 |
moha7 | No, by network-infra_all passed that previous error, but failed with another one: fatal: [ct1_neutron_server_container-56963a6e -> {{ _oslodb_setup_host }}]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} | 19:23 |
jamesdenton | ok, just try without the limit then | 19:23 |
jamesdenton | openstack-ansible setup-hosts.yml | 19:23 |
moha7 | +1 | 19:23 |
jamesdenton | sorry | 19:24 |
jamesdenton | some of these need --limit localhost,<group or host>. who knows | 19:24 |
ElDuderino | anyone know of a good practical tuning guide for rabbitmq as it relates to openstack-ansible based HA installs? O | 19:55 |
ElDuderino | I'm watching https://www.openstack.org/videos/summits/austin-2016/tuning-rabbitmq-at-large-scale-cloud and am trying to determine how relevant it still is (2016 video) although we're still on Rocky so some of it might be relevant. | 19:56 |
*** dviroel is now known as dviroel|out | 20:17 | |
ElDuderino | We have a 3 node rabbit cluster and run it on the controllers in an lxc container. I'm trying to figure out if the OSA community has a 'canonical' way for any of us admins/installers that use OSA at scale to reference | 20:20 |
ElDuderino | It appears that Mirantis has this approach, perhaps we do also? https://docs.mirantis.com/mcp/q4-18/mcp-deployment-guide/advanced-config/tune-rabbitmq-perf.html | 20:22 |
opendevreview | Merged openstack/openstack-ansible master: Update tox.ini https://review.opendev.org/c/openstack/openstack-ansible/+/868680 | 23:38 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!