| noonedeadpunk | mornings | 07:22 |
|---|---|---|
| noonedeadpunk | sry, was away these couple of days | 07:22 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible master: Update haproxy healthcheck options https://review.opendev.org/c/openstack/openstack-ansible/+/887285 | 09:10 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-rabbitmq_server master: Adjust wildcard definition https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/887779 | 09:21 |
| opendevreview | Damian Dąbrowski proposed openstack/openstack-ansible-os_rally stable/2023.1: Include proper commit in rally_upper_constraints_url https://review.opendev.org/c/openstack/openstack-ansible-os_rally/+/887679 | 09:35 |
| opendevreview | Damian Dąbrowski proposed openstack/openstack-ansible-os_rally stable/zed: Include proper commit in rally_upper_constraints_url https://review.opendev.org/c/openstack/openstack-ansible-os_rally/+/887680 | 09:35 |
| opendevreview | Damian Dąbrowski proposed openstack/openstack-ansible-os_rally stable/yoga: Include proper commit in rally_upper_constraints_url https://review.opendev.org/c/openstack/openstack-ansible-os_rally/+/887681 | 09:35 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-rabbitmq_server master: Adjust wildcard definition https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/887779 | 09:39 |
| opendevreview | Damian Dąbrowski proposed openstack/openstack-ansible master: Adjust default value for *_backend_ssl https://review.opendev.org/c/openstack/openstack-ansible/+/887785 | 09:55 |
| opendevreview | Damian Dąbrowski proposed openstack/openstack-ansible stable/2023.1: Adjust default value for *_backend_ssl https://review.opendev.org/c/openstack/openstack-ansible/+/887683 | 09:58 |
| damiandabrowski | hamburgler: ^ | 09:58 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Remove Ubuntu 20.04 support https://review.opendev.org/c/openstack/openstack-ansible/+/886517 | 10:36 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Bump ansible-core to 2.15.1 and collections https://review.opendev.org/c/openstack/openstack-ansible/+/886527 | 10:36 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_cinder master: Add quorum support for service https://review.opendev.org/c/openstack/openstack-ansible-os_cinder/+/875408 | 11:59 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_glance master: Add quorum support for glance https://review.opendev.org/c/openstack/openstack-ansible-os_glance/+/873632 | 12:05 |
| mgariepy | https://bugs.launchpad.net/ubuntu/+source/openvswitch/+bug/1979070 hmm :/ | 12:10 |
| noonedeadpunk | yup, we have that as well | 12:29 |
| mgariepy | is it neutron that is not closing the ssl stuff correctly ? | 12:30 |
| mgariepy | or it's something else ? | 12:30 |
| noonedeadpunk | Not sure to be frank | 12:30 |
| noonedeadpunk | saw that on northd and southsd | 12:30 |
| noonedeadpunk | under `we` I meant CI in OSA - internally I don't run OVN - still on OVS | 12:33 |
| mgariepy | hmm ok | 12:36 |
| noonedeadpunk | (but I bet it would be same) | 12:48 |
| mgariepy | i thuink this part must be missing in neutron or something like that. https://github.com/ovn-org/ovn/commit/5fe179987d14ff38cce345dbbe57ef1ffe7853cc | 12:54 |
| anskiy | mgariepy: I don't think neutron ever should close the connection | 12:54 |
| mgariepy | anyway it's probably not my issue :D | 12:54 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_neutron master: Add quorum support for the service https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/887803 | 12:55 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_cinder master: Add quorum queues support for service https://review.opendev.org/c/openstack/openstack-ansible-os_cinder/+/875408 | 12:56 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_neutron master: Add quorum queues support for the service https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/887803 | 12:56 |
| anskiy | is there any reason, why this thing: https://docs.openstack.org/openstack-ansible-memcached_server/latest/configure-ha.html (Memcached HAProxy frontend/backend) is not the default? | 12:56 |
| noonedeadpunk | we had discussion about that and there were concerns of not making it a default, as like oslo_cache.memcahced_pool should handle things nicely (it's not in fact). | 12:59 |
| noonedeadpunk | and concerns that it's just l3 balancing, so haproxy is not aware of cache state and in case of failure or recovery will get cache invalidated | 13:00 |
| noonedeadpunk | but it worked nicely for us at least. | 13:00 |
| noonedeadpunk | And also, proper solution would be to have smth like mcrouter https://github.com/facebook/mcrouter | 13:01 |
| anskiy | this is a massive boost for gnocchi for me (for which [cache] is not configured at all as defaults) | 13:01 |
| noonedeadpunk | But it's a facebook thingy, so it's basically - use on your own risk | 13:01 |
| noonedeadpunk | oh, well, likely we should indeed add cache to [keystone_authtoken] section... I would say it's worth doing it for each service actually, as I believe we miss that in quite some places | 13:03 |
| anskiy | oh, sorry, not gnocchi -- ceilometer (for ceilomter-agent-notification) -- it's caching metrics so they don't destroy gnocchi | 13:04 |
| noonedeadpunk | oh ,wait, we have that there :) | 13:04 |
| noonedeadpunk | ah, yes | 13:04 |
| noonedeadpunk | there's no reason why it's not there | 13:05 |
| anskiy | "pr's are welcome", got it :) | 13:05 |
| noonedeadpunk | yup :) | 13:06 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_neutron master: Add quorum queues support for the service https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/887803 | 13:43 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_nova master: Add quorum queues support for the service https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/887849 | 13:43 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Replace HA policies for RabbitMQ with quorum https://review.opendev.org/c/openstack/openstack-ansible/+/873618 | 13:48 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: [DNM] Integrational test of quorum queues https://review.opendev.org/c/openstack/openstack-ansible/+/887850 | 13:48 |
| hamburgler | damiandabrowski: ty :) re: https://review.opendev.org/c/openstack/openstack-ansible/+/887683 | 14:27 |
| noonedeadpunk | damiandabrowski: I've reviewed tls-backend patches and have some comments there | 14:51 |
| damiandabrowski | ack, will respond during the evening | 14:53 |
| mgariepy | lol.. neutron_plugin_base router != ovn-router.. | 14:58 |
| mgariepy | >< | 14:58 |
| ptoal | Looking for a pointer to where to look... After running openstack-ansible, with all success, trying to log in to the portal, I get: "Something went wrong!" "An unexpected error has occurred..." | 15:57 |
| ptoal | Where would I look for help troubleshooting this? | 15:57 |
| noonedeadpunk | ptoal: under portal you mean horizon, I assume? | 16:01 |
| noonedeadpunk | I think you need to change logs inside the horizon container then | 16:01 |
| noonedeadpunk | s/change/check/ | 16:01 |
| noonedeadpunk | as this message jsut basically says that 500 error has happened | 16:01 |
| ptoal | Maybe? I'm new to OpenStack. Decided to get familiar with it in expert-mode, for some reason. | 16:01 |
| noonedeadpunk | all details are in logs. I assume, that might be related to some plugin that is broken. | 16:02 |
| ptoal | GET /project/ HTTP/1.1" 500 1867 "https://osp.example.com/auth/login/?next=/" | 16:02 |
| noonedeadpunk | what projects have you deployed, except placement, keystone, nova, cinder, glance and neutron? | 16:02 |
| ptoal | 'Recoverable error: Policy doesn't allow os_compute_api:os-simple-tenant-usage:show to be performed. (HTTP 403)' | 16:03 |
| noonedeadpunk | nah, you should look for smth with 500 and big stack trace | 16:04 |
| ptoal | https://gist.github.com/ptoal/b263ce4ef62dc7fc9d2a7a50b50c9a7b <-- openstack_user_config.yml | 16:08 |
| noonedeadpunk | huh, so you have just basics plus heat | 16:11 |
| ptoal | Hmm.. I do have an error that includes: neutronclient.common.exceptions.ServiceUnavailable: <html><body><h1>503 Service Unavailable</h1> | 16:13 |
| noonedeadpunk | meh... It should be all in journalctl btw | 16:14 |
| noonedeadpunk | ie, `journalctl -u apache2` | 16:14 |
| ptoal | https://gist.github.com/ptoal/0e0ecb9e1de57c9a3db46eb966384d00 | 16:17 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-galera_server master: Remove warn argument for command/shell https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/887861 | 16:18 |
| jrosser | ptoal: i think your neutron service is not heathy | 16:20 |
| jrosser | check the status of all the services on haproxy (you can use hatop to do this) | 16:20 |
| noonedeadpunk | it's weird that horizon raises 500 though... | 16:22 |
| noonedeadpunk | As I've just ran horizon having only keystone and glance and it was fine/happy about that | 16:22 |
| noonedeadpunk | I guess when it detects neutron in catalog it becomes worse, but it's kinda... weird | 16:23 |
| noonedeadpunk | anyway... | 16:23 |
| ptoal | Yep... neutron-server-back is DOWN | 16:23 |
| noonedeadpunk | yeah, check neutron-server state :) | 16:23 |
| ptoal | Well, I must have messed something up with my ovn setup... | 16:27 |
| ptoal | `ValueError: :6642: bad peer name format` | 16:27 |
| ptoal | in my ml2_conf.ini, it has: ovn_sb_connection = ssl::6642 | 16:28 |
| opendevreview | Damian Dąbrowski proposed openstack/openstack-ansible master: Remove haproxy_accept_both_protocols from repo_all https://review.opendev.org/c/openstack/openstack-ansible/+/886586 | 16:28 |
| ptoal | It probably needs a hostname/ip? | 16:28 |
| jrosser | that would come from here https://opendev.org/openstack/openstack-ansible-os_neutron/src/branch/master/defaults/main.yml#L482 | 16:30 |
| ptoal | Yeah... Maybe I missed a host declaration in my config. | 16:30 |
| ptoal | Hm... used _ where I should have used - ... | 16:32 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Return PIP_OPTS for load_nodepool_pip_opts https://review.opendev.org/c/openstack/openstack-ansible/+/887862 | 16:36 |
| ptoal | So, if I fix my openstack_user_config.yml to add the correct neutron host config, do I have to go back to the beginning of the process? Or can I just run the setup-infrastructure / setup-openstack playbooks? | 16:37 |
| jrosser | "it depends" | 16:38 |
| jrosser | for some things you can get away with just running the service specific playbook on it's own | 16:38 |
| jrosser | note that setup-openstack actually just includes a whole bunch of other playbooks | 16:39 |
| ptoal | So I'll just give os-neutron-install a try? | 16:39 |
| jrosser | so if the ml2_conf.ini needs updating it's quite possible that just the neutron role would do | 16:40 |
| jrosser | it's also possible that `--tags neutron-config` would speed that up | 16:42 |
| jrosser | but this is an area where quite often we have bugs | 16:42 |
| opendevreview | Damian Dąbrowski proposed openstack/openstack-ansible master: Allow to update AIO config prior to an upgrade https://review.opendev.org/c/openstack/openstack-ansible/+/885190 | 16:44 |
| noonedeadpunk | Uhg, gather subset is marked for removal for ansible-core=2.18: https://docs.ansible.com/ansible/latest/reference_appendices/config.html#default-gather-subset | 16:54 |
| noonedeadpunk | So we need to somehow replace that with module_defaults in all playbooks... | 16:54 |
| noonedeadpunk | If you anyone was interested what will happen if update ansible-lint to latest version, here's the answer: Failed: 497 failure(s), 131 warning(s) on 92 files. | 16:59 |
| jrosser | is this something to do with role vars require prefixes? | 17:01 |
| noonedeadpunk | 200+ is about not using fqcn for modules | 17:06 |
| noonedeadpunk | let me try to fetch results to some paste... | 17:08 |
| noonedeadpunk | first 800 lines https://paste.openstack.org/show/bZcrKvZBXw5Xi1UUhnzm/ | 17:11 |
| noonedeadpunk | I think `yaml[truthy]` is weird check.... | 17:11 |
| noonedeadpunk | same goes for `key-order[task]` | 17:11 |
| noonedeadpunk | actually, here's summary https://paste.openstack.org/show/bSjyFdkm9JSqwwm8Yu54/ | 17:13 |
| noonedeadpunk | (and these are only for playbooks, not including roles) | 17:15 |
| noonedeadpunk | huge problem is that for ansible-core 2.15 we need to bump ansible-lint to smth pretty recent | 17:22 |
| noonedeadpunk | not sure about 2.14... | 17:23 |
| noonedeadpunk | wtf is that? `schema[playbook]: '{{ osa_gather_facts | default(True) }}' is not of type 'boolean'` | 17:30 |
| jrosser | does it want | bool ? | 17:31 |
| opendevreview | Damian Dąbrowski proposed openstack/openstack-ansible master: Add tls upgrade jobs https://review.opendev.org/c/openstack/openstack-ansible/+/885194 | 17:33 |
| noonedeadpunk | `'{{ osa_gather_facts | default(True) | bool }}' is not of type 'boolean'` - no? | 17:34 |
| noonedeadpunk | maybe that variable should be defined? | 17:34 |
| opendevreview | Damian Dąbrowski proposed openstack/openstack-ansible stable/2023.1: Remove support for 'tls-transition' scenario https://review.opendev.org/c/openstack/openstack-ansible/+/887866 | 17:42 |
| noonedeadpunk | jrosser: it doesn't like expression there | 17:42 |
| opendevreview | Merged openstack/openstack-ansible-os_nova master: Apply always tag to nova_virt_detect.yml https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/885337 | 17:45 |
| jrosser | well unless thats not allowed that would be a linter bug | 17:45 |
| noonedeadpunk | yup, likely it is | 18:01 |
| noonedeadpunk | it just involves npm | 18:06 |
| noonedeadpunk | I've pushed this to cover the issue: https://github.com/ansible/ansible-lint/pull/3606 | 20:03 |
| noonedeadpunk | will see if that will be merged... | 20:03 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_nova master: Add quorum queues support for the service https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/887849 | 20:19 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/2023.1: Bump SHAs for stable/2023.1 https://review.opendev.org/c/openstack/openstack-ansible/+/887513 | 20:22 |
| ptoal | Thanks jrosser and noonedeadpunk. I got my stack running. | 21:24 |
| opendevreview | Merged openstack/ansible-hardening master: Remove warn argument for command/shell https://review.opendev.org/c/openstack/ansible-hardening/+/887376 | 22:33 |
| opendevreview | Merged openstack/ansible-role-pki master: Convert loop labels to strings https://review.opendev.org/c/openstack/ansible-role-pki/+/887374 | 23:37 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!