Monday, 2022-03-14

*** aussielunix_ is now known as aussielunix00:56
*** aussielunix is now known as aussielunix_04:51
*** aussielunix_ is now known as aussielunix04:51
*** frenzyfriday|PTO is now known as frenzyfriday05:20
noonedeadpunkOutBackDingo_: sounds like you don't have lsync running properly or indeed you just missed repo_setup.yml08:06
*** odyssey4me|away is now known as odyssey4me09:00
*** odyssey4me is now known as odyssey4me|away09:00
*** odyssey4me|away is now known as odyssey4me09:02
*** odyssey4me is now known as odyssey4me|away09:25
*** odyssey4me|away is now known as odyssey4me09:25
jrossermorning10:00
noonedeadpunko/10:01
opendevreviewJonathan Rosser proposed openstack/openstack-ansible-tests master: Add ansible.utils collection requirement  https://review.opendev.org/c/openstack/openstack-ansible-tests/+/83359610:09
opendevreviewJonathan Rosser proposed openstack/openstack-ansible-plugins master: Change location of ipaddr filter  https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/83153010:11
noonedeadpunkjrosser: I believe we need netcommon there as for utils there's no redirect iirc so ipaddr without full path was failing as well 10:11
jrosseroh so i need netcommon in the same patch?10:12
noonedeadpunkyeah, I guess so. Not 100% sure, but ~90%10:12
opendevreviewJonathan Rosser proposed openstack/openstack-ansible-tests master: Add ansible.utils collection requirement  https://review.opendev.org/c/openstack/openstack-ansible-tests/+/83359610:13
jrossertheres a couple things needing collections installing with openstack-ansible-tests which is failing10:14
opendevreviewJonathan Rosser proposed openstack/openstack-ansible-os_keystone stable/xena: add oauth support  https://review.opendev.org/c/openstack/openstack-ansible-os_keystone/+/83355210:23
*** arxcruz is now known as arxcruz|lunch11:04
jrosserso this fails on not having the uwsgi role https://review.opendev.org/c/openstack/openstack-ansible-tests/+/83359611:06
jrosserwhat did we break there11:06
*** dviroel is now known as dviroel|ruck11:14
noonedeadpunkansible.netcommon should be 2.5.1 I guess, but not sure it was reason11:15
noonedeadpunkoh, well, uwsgi not here https://opendev.org/openstack/openstack-ansible-tests/src/branch/master/test-ansible-role-requirements.yml11:16
noonedeadpunkAnd I guess it's result of switching keystone to uwsgi11:17
noonedeadpunkhttps://opendev.org/openstack/openstack-ansible-os_keystone/commit/cb7eaa7ce305328a8677bcbe6a34f7577b77b04111:17
noonedeadpunkjrosser:  ^11:17
noonedeadpunkI think we need to patch everything in one commit anyway...11:18
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible-tests master: Add ansible.utils collection requirement  https://review.opendev.org/c/openstack/openstack-ansible-tests/+/83359611:33
*** arxcruz|lunch is now known as arxcruz12:04
*** odyssey4me is now known as odyssey4me|away12:18
*** odyssey4me|away is now known as odyssey4me12:18
*** odyssey4me is now known as odyssey4me|away12:33
jrosseri wonder if they change how the ipaddr filter works12:43
jrossersystemd-networkd.service /etc/systemd/network/8-osa_testing-br-lbaas.network:5: Invalid address '['172.29.232.100/22']', ignoring assignment: Invalid argument12:43
jrosserfrom here https://zuul.opendev.org/t/openstack/build/501de017a698406c8c952edc93b9add4/log/logs/host/systemd-networkd.service.journal-17-40-50.log.txt12:43
*** odyssey4me|away is now known as odyssey4me12:45
jrosserhmm https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_501/831603/1/check/openstack-ansible-deploy-aio_metal-ubuntu-focal/501de01/logs/etc/host/systemd/network/8-osa_testing-br-lbaas.network.txt12:45
*** odyssey4me is now known as odyssey4me|away12:46
*** odyssey4me|away is now known as odyssey4me12:50
*** odyssey4me is now known as odyssey4me|away12:50
mgariepyarf it returns a list of ips instead of an ip..12:50
jrosserindeed, but kind of not previously12:54
jrosserwhich is wierd12:55
mgariepyhttps://github.com/ansible-collections/ansible.utils/blob/main/plugins/filter/ipaddr.py12:58
mgariepyhttps://github.com/ansible-collections/ansible.netcommon/commit/db4920ebf6bae6476ff8829e2cf475f19f83a990#diff-0121234d8f291cc643e532655eb9bf3d94e0377b723e5e867177baca8311a752L148-L15313:00
*** odyssey4me|away is now known as odyssey4me13:00
mgariepyfrom here : https://github.com/ansible-collections/ansible.netcommon/commit/db4920ebf6bae6476ff8829e2cf475f19f83a990#diff-0121234d8f291cc643e532655eb9bf3d94e0377b723e5e867177baca8311a752L50913:00
jrosserwell i guess i more mean that the implementation in netcommon didnt used to return a list?13:21
mgariepyyep. it was returning a string13:26
mgariepynetcomm was returning a string and utils do return lists.13:27
jrosserthats unfortunate13:33
jrosserprobably means we have some circular dependancies then13:33
jrosseras [0] on a string is not the same as [0] on a list13:33
mgariepythere are some QA missing on ansible .. 13:35
noonedeadpunkjrosser: there was a bug for utility claiming on same issue13:37
jrosseri was not spotting anything here https://github.com/ansible-collections/ansible.utils/issues13:37
noonedeadpunkhttps://github.com/ansible-collections/ansible.utils/issues/15113:37
jrosserahha yes just found that13:38
noonedeadpunkthey have 2.5.2 now13:38
jrosserhttps://github.com/ansible-collections/ansible.utils/issues/14813:38
opendevreviewJonathan Rosser proposed openstack/openstack-ansible-tests master: Add ansible.utils collection requirement  https://review.opendev.org/c/openstack/openstack-ansible-tests/+/83359613:39
opendevreviewJonathan Rosser proposed openstack/openstack-ansible master: Add ansible.utils collectoin requirement  https://review.opendev.org/c/openstack/openstack-ansible/+/83152513:48
opendevreviewJonathan Rosser proposed openstack/openstack-ansible master: Update netcommon collection  https://review.opendev.org/c/openstack/openstack-ansible/+/83311713:49
NeilHanlonnoonedeadpunk: would some love for the openstack-ansible-ops graylog role be welcome? Seems it's in a bit of a bad state right now w.r.t. archived upstream dependencies and old versions13:58
noonedeadpunkNeilHanlon: they will for sure, yes13:59
NeilHanlon👍14:00
noonedeadpunklove for everything is always welcome :p14:00
NeilHanlon:D 14:00
NeilHanloni have a terrible habit of finding more work for me to do hehe14:00
spatelgood habit though NeilHanlon 14:00
NeilHanlonvery true, I'm never not busy lol14:01
spatelnoonedeadpunk is xena 24.0.1 is ready to rollout in prod, hope no lingering issue :) 14:02
noonedeadpunkmmmm, I guess we could have...14:03
spatelthen i would like to hold on 14:03
noonedeadpunkhm, and why 24.1.0 not released yet14:03
spatelhmm14:04
noonedeadpunkI'm also now seeing issues regarding decreased db pooling I believe in our biggest region14:05
noonedeadpunkand https://review.opendev.org/q/topic:bug%252F1961603 is also related14:05
spatelwhat do you mean decreased db pool?14:05
noonedeadpunkspatel: mean this https://review.opendev.org/q/topic:db-pooling14:32
spateloh! i thought this patch is part of 24.0.0 14:34
spatelyou are saying in order to upgrade xena we have to increate db pool size ?14:35
opendevreviewMarc Gariépy proposed openstack/openstack-ansible-lxc_container_create master: Change location of ipaddr filter  https://review.opendev.org/c/openstack/openstack-ansible-lxc_container_create/+/83311814:36
noonedeadpunkspatel: yep, that is part of 24.0.0 but seems it's causing us issues14:39
opendevreviewMarc Gariépy proposed openstack/openstack-ansible-lxc_hosts master: Change location of ipaddr filter  https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/83311914:39
noonedeadpunkwell, I'm not 100% sure it's it, but the only thing to suspect atm and testing actively14:40
noonedeadpunkthe problem is that upon connection_recycle_time even non-idle connections will be recycled, and with conjuction decreasing of pool of connections we hit interminent failures for some mysql requests14:41
spatelNothing wrong to increate pooling limit correct? 14:42
jrosserandrewbonney: ^ you seen this?14:42
andrewbonneyI don't believe so14:43
jrosserthis somehow feels like a bug if active connections are recycled14:43
andrewbonneyDoes it prompt any tracebacks or similar? I could have a search if so14:44
noonedeadpunkI also pushed https://review.opendev.org/c/openstack/oslo.db/+/833623 as this could be likely other way around14:45
noonedeadpunkRelated doc: https://docs.sqlalchemy.org/en/14/core/pooling.html#disconnect-handling-pessimistic14:45
noonedeadpunkandrewbonney: I have sample traceback14:46
noonedeadpunkhttps://paste.citynetwork.se/ozipediyegirimo.rb14:46
noonedeadpunkdamn :)14:46
noonedeadpunkhttps://paste.openstack.org/show/bcyfeUiNi3HTIxHKrZos/14:47
andrewbonneyOk, I can't see any sign of that here14:50
MrClayPoleWhile testing the an OSA upgrade from train (20.2.6) to victoria (20.4.1 ) following the guide https://docs.openstack.org/openstack-ansible/victoria/admin/upgrades/major-upgrades.html, it fails with following error on all containers during the "openstack-ansible setup-hosts.yml --limit '!galera_all:!rabbitmq_all'" with the error "(item={'key': 'lxcbr0_address', 'value': {'bridge': 'lxcbr0', 'interface': 'eth0', 'type':15:00
MrClayPole 'veth'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lxcbr0_address", "value": {"bridge": "lxcbr0", "interface": "eth0", "type": "veth"}}, "msg": "AnsibleError: template error while templating string: no filter named 'ipaddr'." I've tried a manual "pip install ipaddr" on one of the containers but this doesn't resolve the issue.15:00
MrClayPoleThis is during this part of the playbook https://opendev.org/openstack/openstack-ansible-lxc_container_create/src/branch/stable/victoria/tasks/lxc_container_network_new.yml#L2115:00
jrosserMrClayPole: probably this https://review.opendev.org/c/openstack/openstack-ansible/+/83185315:03
jrosseransible have made breaking changes unfortunately15:03
MrClayPoleThanks for the pointer. I'll get installed and see if that helps.15:04
opendevreviewMarc Gariépy proposed openstack/openstack-ansible-lxc_hosts master: Change location of ipaddr filter  https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/83311915:10
*** dviroel|ruck is now known as dviroel|ruck|lunch15:36
noonedeadpunkwe actually see that only in one region out of 3 upgraded15:44
noonedeadpunkso can be unrelated but I can hardly imagine any other reason tbh15:45
*** melwitt_ is now known as melwitt16:14
admin1is there a way to override polling.yaml of ceilometer .. or add more more stuff  via osa ? 16:18
admin1add more items in the meters:  in polling.yaml 16:18
noonedeadpunkI bet there was16:19
noonedeadpunkiirc you could even provide your own one16:19
noonedeadpunk`ceilometer_polling_default_file_path` - for your own file or ceilometer_polling_yaml_overrides for overrides16:20
admin1thanks ..16:21
admin1got it16:21
*** dviroel|ruck|lunch is now known as dviroel|ruck16:22
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible-tests master: Add ansible.utils collection requirement  https://review.opendev.org/c/openstack/openstack-ansible-tests/+/83359616:28
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible-plugins master: Change location of ipaddr filter  https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/83153016:28
noonedeadpunkspatel: 24.1.0 released jsut in case16:29
spatel+1 thanks i will push out to my dev openstack first and validate 16:30
MrClayPoleThanks jrosser, that patch fixed my issue.16:46
*** arxcruz is now known as arxcruz|off17:54
spatelAny idea what is wrong here? I deployed k8s cluster and when trying to obtain creds/config getting this error - https://paste.opendev.org/show/bhkwVmwwerPHDM46Ymgv/18:43
spatelNevermind upgrade of magnumclient resolved issue18:58
opendevreviewMerged openstack/openstack-ansible master: Add ansible.utils collectoin requirement  https://review.opendev.org/c/openstack/openstack-ansible/+/83152519:01
*** dviroel|ruck is now known as dviroel|ruck|afjk20:50
*** dviroel|ruck|afjk is now known as dviroel|ruck|afk20:50
opendevreviewMerged openstack/ansible-role-systemd_networkd master: Change location of ipaddr filter  https://review.opendev.org/c/openstack/ansible-role-systemd_networkd/+/83160322:46
*** dviroel|ruck|afk is now known as dviroel|ruck23:49

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!