| opendevreview | Andrew Bonney proposed openstack/openstack-ansible-haproxy_server stable/wallaby: Fix PKI regen behaviour https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/813568 | 07:25 |
|---|---|---|
| opendevreview | Andrew Bonney proposed openstack/openstack-ansible-haproxy_server stable/wallaby: Fix typo for user supplied certificate variable https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/813569 | 07:25 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-haproxy_server stable/wallaby: Fix PKI regen behaviour https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/813568 | 09:51 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-haproxy_server stable/wallaby: Fix typo for user supplied certificate variable https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/813569 | 09:51 |
| opendevreview | Andrew Bonney proposed openstack/openstack-ansible-haproxy_server stable/wallaby: Fix typo for user supplied certificate variable https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/813569 | 10:00 |
| opendevreview | Merged openstack/openstack-ansible-os_tempest stable/wallaby: Pin neutron-tempest-plugin to v1.6.0 https://review.opendev.org/c/openstack/openstack-ansible-os_tempest/+/813198 | 12:17 |
| noonedeadpunk | andrewbonney: should we also merge and backport https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/811985 ? | 12:53 |
| andrewbonney | Yes it's ready to go as far as I'm concerned. I've done two production deploys with the patch in place | 12:53 |
| noonedeadpunk | nice | 12:54 |
| andrewbonney | Same for this backport https://review.opendev.org/c/openstack/openstack-ansible/+/813099 | 12:55 |
| noonedeadpunk | oh, yes, I missed it( | 12:58 |
| fridtjof[m] | hm, I'm encountering issues on a v -> w upgrade :( | 14:11 |
| fridtjof[m] | os-keystone-install is failing with this: https://paste.opendev.org/show/809924/ | 14:11 |
| fridtjof[m] | what's confusing me is that keystone shouldn't have a repo directory, right? that should only be in a repo container | 14:11 |
| opendevreview | Merged openstack/openstack-ansible-haproxy_server stable/wallaby: Fix PKI regen behaviour https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/813568 | 14:25 |
| opendevreview | Merged openstack/openstack-ansible-haproxy_server stable/wallaby: Fix typo for user supplied certificate variable https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/813569 | 14:28 |
| jrosser | fridtjof[m]: in the tasks before the one pasted you should see where that file has been written to | 14:32 |
| jrosser | and you can see that the task was delegated to the repo container "[infra1_keystone_container-42e7d94a -> 10.1.60.153]" | 14:33 |
| fridtjof[m] | ah, i didn't realize that. | 14:35 |
| fridtjof[m] | Looking at the tasks before that, it reads like it might've skipped more than it should? | 14:36 |
| fridtjof[m] | I don't have logs for it unfortunately (too short terminal buffer :( ), but an earlier attempt failed somewhere else related to the repository (i think?) | 14:36 |
| fridtjof[m] | so that might actually be a knock-on effect | 14:36 |
| fridtjof[m] | i could try re-running whatever sets up the repo containers... | 14:37 |
| fridtjof[m] | (full log of this run: https://paste.debian.net/1215151/) | 14:39 |
| fridtjof[m] | nope, same results after re-running repo-install | 14:48 |
| mgariepy | are you sure all the osa git repo are up-to-date ? | 14:50 |
| mgariepy | i had issue recently for some reason that repo-build wasn't updated and it caused something similar to that. | 14:50 |
| mgariepy | fridtjof[m], ^^ | 14:57 |
| fridtjof[m] | I just went for the latest 23.* tag, checked out today | 14:58 |
| fridtjof[m] | I'll check again later | 14:59 |
| noonedeadpunk | #startmeeting openstack_ansible_meeting | 15:00 |
| opendevmeet | Meeting started Tue Oct 12 15:00:23 2021 UTC and is due to finish in 60 minutes. The chair is noonedeadpunk. Information about MeetBot at http://wiki.debian.org/MeetBot. | 15:00 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 15:00 |
| opendevmeet | The meeting name has been set to 'openstack_ansible_meeting' | 15:00 |
| noonedeadpunk | #topic office hours | 15:00 |
| noonedeadpunk | so, next week is week of PTG | 15:01 |
| noonedeadpunk | and https://etherpad.opendev.org/p/osa-yoga-ptg is pretty empty :( | 15:01 |
| noonedeadpunk | also - I guess it's we time switched master to stable/xena not to follow master and not to face bugs? | 15:02 |
| noonedeadpunk | *it's time we switched | 15:03 |
| spatel | noonedeadpunk i added one line :) | 15:15 |
| spatel | jamesdenton i have blog out for VPN with keepalived for high availability https://satishdotpatel.github.io/ha-strongswan-ipsec-vpn/ | 15:28 |
| noonedeadpunk | #endmeeting | 15:36 |
| opendevmeet | Meeting ended Tue Oct 12 15:36:18 2021 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 15:36 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/openstack_ansible_meeting/2021/openstack_ansible_meeting.2021-10-12-15.00.html | 15:36 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/openstack_ansible_meeting/2021/openstack_ansible_meeting.2021-10-12-15.00.txt | 15:36 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/openstack_ansible_meeting/2021/openstack_ansible_meeting.2021-10-12-15.00.log.html | 15:36 |
| jamesdenton | thanks, spatel | 15:36 |
| noonedeadpunk | well, vpnaas is not the case for OVN :( | 15:36 |
| noonedeadpunk | I wish it was though | 15:37 |
| spatel | why not ? | 15:37 |
| spatel | because of no namespace? | 15:37 |
| noonedeadpunk | yeah | 15:37 |
| noonedeadpunk | and no interest/maintainers even for ovs/lxb implementation ( | 15:38 |
| spatel | there must be other workaround | 15:38 |
| noonedeadpunk | create several VM's with keepalived on them or following your blogpost? | 15:38 |
| jamesdenton | probably just easier to build some heat template or something for VPN concentrator | 15:39 |
| noonedeadpunk | yeah, or that actually | 15:39 |
| noonedeadpunk | or leverage magnum ? I guess there should be smth deployable inside k8s? | 15:39 |
| spatel | my blog is for my new datacenter VPN access to have redundancy in keepalived but sure look like we have to deploy VPN outside openstack | 15:39 |
| noonedeadpunk | well, I guess it can work inside openstack as well if needed... | 15:40 |
| spatel | https://specs.openstack.org/openstack/neutron-specs/specs/xena/vpnaas-ovn.html | 15:40 |
| noonedeadpunk | oh, wow | 15:44 |
| noonedeadpunk | there's even a patch https://review.opendev.org/c/openstack/neutron-vpnaas/+/765353 | 15:45 |
| noonedeadpunk | wow | 15:45 |
| spatel | yeah!! | 15:46 |
| spatel | i should test that :) | 15:47 |
| spatel | noonedeadpunk currently we don't have good way to recover rabbitMQ in OSA :( | 15:47 |
| noonedeadpunk | well, during last PTG iirc neutron folks was like talking that they're not going even to spend time on this atm | 15:47 |
| noonedeadpunk | recover in terms of..? | 15:48 |
| noonedeadpunk | if it's fully dead and re-setuped? | 15:48 |
| spatel | last week two time i nuke rabbitMQ cluster and rebuild it | 15:48 |
| noonedeadpunk | or to complete maintenances/upgrades beter? | 15:48 |
| spatel | its keep dying, i think i need to upgrade more memory | 15:48 |
| mgariepy | spatel. stop all and start back | 15:48 |
| spatel | nothing work :( | 15:48 |
| noonedeadpunk | there's a tag `common-mq` so you can run setup-openstack.yml with it and it's creating all vhosts/users for all services relatively fast I'd say | 15:49 |
| spatel | i did stop but that stop went for 1 hour and then finally i have to kill it and whole cluster was in non-recoverable mode | 15:49 |
| spatel | wow! where is that common-mq ? | 15:49 |
| spatel | i wrote my own script to create vhost/user/password/ etc.. and i hate that.. | 15:50 |
| noonedeadpunk | in each role where mq_setup is included, ie https://opendev.org/openstack/openstack-ansible-os_glance/src/branch/master/tasks/main.yml#L75 | 15:50 |
| noonedeadpunk | well, it's not instant, but acceptable imo | 15:50 |
| noonedeadpunk | and worked for me lately | 15:51 |
| spatel | so if i run openstack-ansible rabbitmq-server.yml --tags common-mq will create all account? | 15:51 |
| noonedeadpunk | `setup-openstack.yml` | 15:51 |
| spatel | oh! | 15:51 |
| mgariepy | is that documented ? :/ | 15:52 |
| noonedeadpunk | but it will run only tasks regarding rabbit setup | 15:52 |
| noonedeadpunk | ofc not | 15:52 |
| spatel | why don't we have something more quicker way, just read endpoint and create those account quickly | 15:52 |
| noonedeadpunk | because we kind of don't know credentials? | 15:53 |
| noonedeadpunk | since they are defined for each role independently | 15:53 |
| spatel | user_secret.yml file | 15:53 |
| noonedeadpunk | and vhost? username? | 15:53 |
| spatel | can't we read-password from that place? | 15:53 |
| noonedeadpunk | if we should use ssl? And what rabbit server to talk to? | 15:54 |
| noonedeadpunk | because back to the usecase - we had a rabbit cluster per service | 15:54 |
| spatel | rabbitmqctl add_user nova c737e038d2a49696411240e12c362d4f431bbb67a405a0bcb8e4 | 15:54 |
| spatel | rabbitmqctl add_user glance c42769fddfe6799a6efd8e3330ee5f29e8c99387dfa8c8 | 15:54 |
| noonedeadpunk | how do you know username is nova? | 15:54 |
| spatel | and craft this command to run | 15:54 |
| spatel | [root@ostack-osa-2 ~]# cat /etc/openstack_deploy/user_secrets.yml | grep nova_o | 15:55 |
| spatel | nova_oslomsg_rpc_password: c737e038d2a49696411240e12c362d4f431bbb67a405a0bcb8e4 | 15:55 |
| noonedeadpunk | it doesn't mean username is nova | 15:55 |
| noonedeadpunk | because that is defined with nova_oslomsg_rpc_userid | 15:56 |
| spatel | why someone will change userid? | 15:56 |
| spatel | i am seeing all service has same userid what ever service name | 15:57 |
| spatel | glance / nova/ neutron etc... | 15:57 |
| noonedeadpunk | well, returning back to usecase I was having - separate rabbit clusters for each service | 15:58 |
| spatel | whatever script i created it read this user_secret.yml file and craft all rabbitMQ command to add username/password and run them on rabbit container, its working fine so far.. | 15:58 |
| spatel | noonedeadpunk no kidding :O | 15:58 |
| noonedeadpunk | so you actually override glance_oslomsg_rpc_host_group | 15:58 |
| spatel | why are you doing that way? | 15:58 |
| spatel | each service has own cluster? | 15:59 |
| noonedeadpunk | yes, this can be done, but it would be pretty opionated | 15:59 |
| noonedeadpunk | in the way of not taking into account plenty of other usecases | 15:59 |
| spatel | i know what you saying.. | 15:59 |
| noonedeadpunk | yep, each service had own independant rabbitmq clusters | 15:59 |
| spatel | that would be overkill right? | 15:59 |
| noonedeadpunk | while running playbook would still do exactly the same thing without need of extra maintenance, when new role is being added | 16:00 |
| spatel | only neutron and nova is bigger consumer of rabbit (forget about gnocchi) | 16:00 |
| noonedeadpunk | depends on the region size... Because eventually the only way to scale up rabbit is to make more clusters or disable ha_queues | 16:00 |
| noonedeadpunk | well, yes, I agree that you don't need that for _each_ service | 16:01 |
| spatel | i heard disable ha_queues cause issue | 16:01 |
| spatel | is that true o? | 16:01 |
| noonedeadpunk | but if you automate that process - it's just easier to align all of them kind of | 16:01 |
| noonedeadpunk | it causes issues when one of rabbit nodes outages.... | 16:01 |
| noonedeadpunk | well, it used to cause issues | 16:01 |
| noonedeadpunk | but, having ha_queues is a performance penalty... | 16:02 |
| spatel | i am thinking to remove ha_queue (i don't need HA because this is my private cloud) | 16:02 |
| spatel | do you know good way to disable HA in OSA ? | 16:02 |
| spatel | any variable etc.. | 16:02 |
| noonedeadpunk | sure | 16:03 |
| noonedeadpunk | just define `rabbitmq_policies: []` in overrides | 16:04 |
| noonedeadpunk | to override that https://opendev.org/openstack/openstack-ansible/src/branch/master/inventory/group_vars/all/infra.yml#L27-L31 | 16:04 |
| spatel | oh! | 16:04 |
| SiavashSardari | @noonedeadpunk what do you mean by not using ha queues used to cause issue? | 16:05 |
| spatel | i heard somewhere, i don't have personal experience because i am running in ha | 16:06 |
| spatel | if anyone running in non-HA please tell us how do you feel , i would love to know | 16:06 |
| noonedeadpunk | SiavashSardari: so, when one of rabbitmq cluster members was going down, and returning back to life - it didn't have queues, but services were supposing it has them, so while cluster was healthy, and services were able to communicate with all members, there were some interminnent erorrs while doing requests to apis | 16:08 |
| noonedeadpunk | But I was seeing that in stein last time. After that I switched to HA queues and forgot about the issues | 16:08 |
| SiavashSardari | couple of years ago I used zeromq due to higher performance and if I recall correctly we didn't have any problem but the fear of failure led us to rabbit with ha queues | 16:08 |
| noonedeadpunk | so, the problem was at time, when "faulty" node was re-joining cluster | 16:09 |
| noonedeadpunk | but it might be somehow fixed in oslo or actually with rabbit version bumps | 16:09 |
| noonedeadpunk | I haven't really tested anywhere since then | 16:10 |
| spatel | problem is in lab you won't see this kind of issue until you deploy in production with real-workload | 16:10 |
| SiavashSardari | interesting... | 16:10 |
| noonedeadpunk | also solution to that (except ha queues) was jsut dropping all queues in cluster, so that services jsut re-created them from scratch | 16:10 |
| noonedeadpunk | (which is actually what rabbitmq-install.yml -e rabbitmq_upgrade` also does...) | 16:11 |
| jamesdenton | QQ re: octavia and trove roles. Are we hardset on the use of a second bridge and related wiring to provide access from container->amphora and db VMs, or can we phase that out in favor of L3 connectivity? Or at least maybe provide a way to choose? | 16:11 |
| noonedeadpunk | I'd vote on simplification of networking about octavia/trove, but I'm not sure I understand the way how l3 networking could be done there in other way? | 16:14 |
| noonedeadpunk | like just routing? | 16:14 |
| jamesdenton | yeah, just routing | 16:15 |
| jamesdenton | right now, the lxc container is wired up to second bridge and some funky stuff is used to connect | 16:15 |
| jamesdenton | i need to spend a little time on it | 16:16 |
| johnsom | Yeah, I never understood why so many bridges were needed for the Octavia lb-mgmt-net in OSA. | 16:16 |
| noonedeadpunk | yeah. but I guess you would need to have vlan network to route that? | 16:16 |
| jamesdenton | IIRC it was meant to be 'temporary' :D | 16:17 |
| noonedeadpunk | and vlan should be managed not by l3 router but by upstream router? | 16:17 |
| jamesdenton | noonedeadpunk yes, it would need to be a vlan. lbaas-mgmt would need to be a provider vlan, essentially | 16:17 |
| SiavashSardari | while we're on the networking part of octavia and trove. I had an issue regarding the connectivity of containers to amphora. the root cause was the use of http proxy on containers, I override 'global_environment_variables' in group vars and my problem was kinda solved. but I think there may be a better solution here | 16:17 |
| noonedeadpunk | I mean - there should be a point which would have access to both - some container net and octavia net | 16:18 |
| noonedeadpunk | and I guess just l3 connectivity is just matter of documentation? | 16:19 |
| jamesdenton | docs + diagram | 16:19 |
| noonedeadpunk | well, yes | 16:19 |
| jamesdenton | i'm doing it now w/ octavia, but need to look at the overrides i have in place to make that go. Trove is less forgiving. | 16:19 |
| johnsom | Octavia fully supports a routed solution. There is nothing special about that network, it's just a neutron network the amps sit on for command/control. | 16:19 |
| jamesdenton | exactly | 16:20 |
| noonedeadpunk | with trove it's a bit harder, because dbaas net should be not on trove_api but on rabbitmq container | 16:20 |
| jamesdenton | rabbit? really? | 16:20 |
| noonedeadpunk | yeah | 16:20 |
| noonedeadpunk | trove communicates with agents through rabbit | 16:20 |
| noonedeadpunk | so like rabbit exposed to VMs | 16:21 |
| jamesdenton | well, that's no different that lxb or ovs agent, right? | 16:21 |
| noonedeadpunk | (and that is why it's kind of better to have independent rabbit cluster for it as well) | 16:21 |
| noonedeadpunk | yeah, I guess so) | 16:21 |
| noonedeadpunk | but to sum up - I'm all for to document the way for just l3 connectivity | 16:25 |
| jamesdenton | cool | 16:25 |
| noonedeadpunk | if somebody have interest or time for that:) | 16:25 |
| jamesdenton | i will work on that | 16:25 |
| noonedeadpunk | awesome! | 16:25 |
| johnsom | jamesdenton Add me as a reviewer and I will also give it a look over | 16:25 |
| noonedeadpunk | because it will really simplify lives for many | 16:25 |
| jamesdenton | thanks johnsom | 16:26 |
| *** wt is now known as _wt_ | 23:25 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!