| opendevreview | Verification of a change to openstack/kolla-ansible master failed: Fix Bash variable expansion issues in openrc file https://review.opendev.org/c/openstack/kolla-ansible/+/881469 | 00:16 |
|---|---|---|
| opendevreview | Merged openstack/kolla-ansible master: Fix Bash variable expansion issues in openrc file https://review.opendev.org/c/openstack/kolla-ansible/+/881469 | 01:58 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: always add service_user section to nova.conf https://review.opendev.org/c/openstack/kolla-ansible/+/882893 | 04:20 |
| mnasiadka | morning | 06:09 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: ansible: bump min to 2.13 and max to 2.14 https://review.opendev.org/c/openstack/kolla-ansible/+/881018 | 06:10 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: ansible: bump min to 2.13 and max to 2.14 https://review.opendev.org/c/openstack/kolla-ansible/+/881018 | 06:10 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: ansible: bump min to 2.13 and max to 2.14 https://review.opendev.org/c/openstack/kolla-ansible/+/881018 | 06:12 |
| opendevreview | Maksim Malchuk proposed openstack/kolla-ansible stable/zed: Fix Bash variable expansion issues in openrc file https://review.opendev.org/c/openstack/kolla-ansible/+/883228 | 06:57 |
| opendevreview | Maksim Malchuk proposed openstack/kolla-ansible stable/yoga: Fix Bash variable expansion issues in openrc file https://review.opendev.org/c/openstack/kolla-ansible/+/883229 | 07:04 |
| opendevreview | Maksim Malchuk proposed openstack/kolla-ansible stable/xena: Fix Bash variable expansion issues in openrc file https://review.opendev.org/c/openstack/kolla-ansible/+/883230 | 07:05 |
| opendevreview | Maksim Malchuk proposed openstack/kolla-ansible stable/wallaby: Fix Bash variable expansion issues in openrc file https://review.opendev.org/c/openstack/kolla-ansible/+/883231 | 07:07 |
| mmalchuk | CI is broken | 07:46 |
| mmalchuk | kolla-ansible | 07:46 |
| mmalchuk | all stable jobs failed on kolla-toolbox ansible run | 07:47 |
| mmalchuk | X,Y,Z atleast | 07:48 |
| mmalchuk | https://zuul.opendev.org/t/openstack/build/2440e1a7460f485ba6512fb0731116c3/log/primary/logs/ansible/deploy | 07:48 |
| mmalchuk | an example of the failure | 07:49 |
| mmalchuk | looks like new json library issue | 07:50 |
| mnasiadka | breakage after breakage, why not | 07:51 |
| SvenKieske | mhm, where got that introduced? | 07:51 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: ansible: bump min to 2.13 and max to 2.14 https://review.opendev.org/c/openstack/kolla-ansible/+/881018 | 07:53 |
| mmalchuk | master and W not affected | 07:54 |
| SvenKieske | I got a question, are the graphs for k-a supposed to work here? https://grafana.opendev.org/d/c0d59dad13/kolla-failure-rate?orgId=1&from=now-30d&to=now the kolla graphs seem to be okay, but k-a mostly shows "no data" | 07:56 |
| SvenKieske | or are there really no failures in the last 30 days? | 07:57 |
| mnasiadka | I know that frickler was looking into this, maybe the jobs names changed? | 07:57 |
| SvenKieske | I somehow doubt that :D | 07:57 |
| mnasiadka | looking into this in the past, so probably it worked | 07:57 |
| mnasiadka | and kolla build failures include only debian and rocky | 07:58 |
| mnasiadka | so probably job names :D | 07:58 |
| SvenKieske | ok | 07:58 |
| mnasiadka | mmalchuk: which json library? | 07:58 |
| SvenKieske | so where do you look daily to detect job failures? just directly in zuul? | 07:58 |
| mmalchuk | mnasiadka json in callback, don't know yet, trying to investigate... seems very strange... help needed | 07:59 |
| mnasiadka | SvenKieske: yes, zuul.openstack.org -> Builds ;-) | 08:00 |
| SvenKieske | ah, it's the same as zuul.opendev.org :D, okay, now that one already | 08:01 |
| SvenKieske | *kno | 08:01 |
| * SvenKieske gives up correcting his writing in the morning. | 08:02 | |
| mnasiadka | SvenKieske: https://review.opendev.org/c/openstack/project-config/+/883234 - that should fix grafana | 08:05 |
| mnasiadka | but would be nice to get some dashboards for stable branches as well | 08:05 |
| SvenKieske | I'm a fan of generating grafana boards/yaml via jsonnet: https://monitoring.mixins.dev/ would make this yaml maybe easier to review. the downside is that jsonnet is poorly documented. | 08:11 |
| mnasiadka | SvenKieske: if you want to convince opendev to change something, this is not the correct channel :D | 08:14 |
| SvenKieske | okay, which one would that be? | 08:16 |
| mnasiadka | #opendev ;-) | 08:17 |
| opendevreview | Verification of a change to openstack/kolla-ansible stable/zed failed: hacluster: Use nodename to align with nova service names https://review.opendev.org/c/openstack/kolla-ansible/+/875609 | 08:25 |
| opendevreview | Verification of a change to openstack/kolla-ansible stable/zed failed: cloudkitty: set cafile for fetcher_keystone https://review.opendev.org/c/openstack/kolla-ansible/+/877764 | 08:38 |
| opendevreview | Merged openstack/kolla-ansible stable/wallaby: hacluster: Use nodename to align with nova service names https://review.opendev.org/c/openstack/kolla-ansible/+/875672 | 08:43 |
| opendevreview | Verification of a change to openstack/kolla-ansible stable/yoga failed: hacluster: Use nodename to align with nova service names https://review.opendev.org/c/openstack/kolla-ansible/+/875670 | 08:43 |
| opendevreview | Verification of a change to openstack/kolla-ansible stable/yoga failed: Fix deploy/genconfig in check mode https://review.opendev.org/c/openstack/kolla-ansible/+/880311 | 08:50 |
| opendevreview | Verification of a change to openstack/kolla-ansible stable/xena failed: Fix deploy/genconfig in check mode https://review.opendev.org/c/openstack/kolla-ansible/+/880312 | 08:56 |
| opendevreview | Verification of a change to openstack/kolla-ansible stable/yoga failed: RabbitMQ: Support setting ha-promote-on-shutdown https://review.opendev.org/c/openstack/kolla-ansible/+/878338 | 09:27 |
| opendevreview | Verification of a change to openstack/kolla-ansible stable/yoga failed: Add flag to change RabbitMQ ha-mode definition https://review.opendev.org/c/openstack/kolla-ansible/+/878339 | 09:27 |
| opendevreview | Verification of a change to openstack/kolla-ansible stable/yoga failed: Add flags for RabbitMQ message TTL & queue expiry https://review.opendev.org/c/openstack/kolla-ansible/+/880314 | 09:27 |
| opendevreview | Verification of a change to openstack/kolla-ansible stable/zed failed: Fix deploy/genconfig in check mode https://review.opendev.org/c/openstack/kolla-ansible/+/880310 | 09:29 |
| SvenKieske | is that all the json breakage? these changes seemed to be mostly fine before? | 09:56 |
| SvenKieske | mmalchuk: do you have any clues yet, regarding the breakage? if I can assist somehow in finding a root cause ping me | 09:59 |
| mmalchuk | SvenKieske no, cant' find. If you have a time please do | 10:08 |
| mnasiadka | ok, found the perpetrator | 10:19 |
| mnasiadka | ansible.posix.json callback plugin 1.5.4 is broken | 10:19 |
| mnasiadka | I thought pinning to <2 is a good idea because it wouldn't break us :D | 10:19 |
| mmalchuk | cool! thanks | 10:20 |
| opendevreview | Verification of a change to openstack/kolla-ansible stable/zed failed: RabbitMQ: Support setting ha-promote-on-shutdown https://review.opendev.org/c/openstack/kolla-ansible/+/878336 | 10:21 |
| opendevreview | Verification of a change to openstack/kolla-ansible stable/zed failed: Add flag to change RabbitMQ ha-mode definition https://review.opendev.org/c/openstack/kolla-ansible/+/878337 | 10:21 |
| opendevreview | Verification of a change to openstack/kolla-ansible stable/zed failed: Add flags for RabbitMQ message TTL & queue expiry https://review.opendev.org/c/openstack/kolla-ansible/+/880313 | 10:21 |
| opendevreview | Michal Nasiadka proposed openstack/kolla master: toolbox: pin ansible.posix to <1.5.4 https://review.opendev.org/c/openstack/kolla/+/883245 | 10:35 |
| opendevreview | Verification of a change to openstack/kolla-ansible stable/xena failed: Add flags for RabbitMQ message TTL & queue expiry https://review.opendev.org/c/openstack/kolla-ansible/+/880315 | 10:39 |
| kevko | Hi, folks have someone issues with neutron and db deadlocks on yoga ? | 10:55 |
| kevko | Neutron ml2 ovn | 10:55 |
| opendevreview | Dawud proposed openstack/kolla-ansible master: Use friendly prometheus instance labels https://review.opendev.org/c/openstack/kolla-ansible/+/876357 | 11:04 |
| opendevreview | Dawud proposed openstack/kolla-ansible master: Use friendly prometheus instance labels https://review.opendev.org/c/openstack/kolla-ansible/+/876357 | 11:05 |
| mnasiadka | kevko: db deadlocks? We've had those before we started using mariadb-clustercheck, but I don't think I've seen any for a year or more ;-) | 11:06 |
| kevko | mnasiadka: when i run 250 instances at once neutron is broken | 11:19 |
| mnasiadka | like create 250 instances in one second? | 11:19 |
| kevko | Some instances just jump into error | 11:19 |
| kevko | Horizon -> count 250 :) | 11:20 |
| kevko | I've already tuned some configs and now working ...but it still looks like some type of bug in neutron | 11:20 |
| mmalchuk | mnasiadka maybe there need RP+ for https://review.opendev.org/c/openstack/kolla/+/883245 ? CI is broken | 11:52 |
| mmalchuk | folks please review ^^^ | 11:52 |
| SvenKieske | I'm on it! | 12:50 |
| SvenKieske | mhm, should the kolla-toolbox container download from quay.io actually work? I seem to remember someone complaining about this some time ago? | 13:21 |
| SvenKieske | because it doesn't work currently | 13:21 |
| SvenKieske | nvm, guess I had the wrong tag | 13:22 |
| opendevreview | Merged openstack/kolla master: toolbox: pin ansible.posix to <1.5.4 https://review.opendev.org/c/openstack/kolla/+/883245 | 13:32 |
| opendevreview | Michal Nasiadka proposed openstack/kolla stable/zed: toolbox: pin ansible.posix to <1.5.4 https://review.opendev.org/c/openstack/kolla/+/883121 | 13:33 |
| opendevreview | Michal Nasiadka proposed openstack/kolla stable/yoga: toolbox: pin ansible.posix to <1.5.4 https://review.opendev.org/c/openstack/kolla/+/883122 | 13:33 |
| opendevreview | Michal Nasiadka proposed openstack/kolla stable/xena: toolbox: pin ansible.posix to <1.5.4 https://review.opendev.org/c/openstack/kolla/+/883123 | 13:33 |
| opendevreview | Dawud proposed openstack/kolla-ansible master: Use friendly prometheus instance labels https://review.opendev.org/c/openstack/kolla-ansible/+/876357 | 13:45 |
| opendevreview | Mark Goddard proposed openstack/kayobe master: Fix firewalld configuration for monitoring hosts https://review.opendev.org/c/openstack/kayobe/+/883263 | 13:58 |
| spatel | SvenKieske after yoga they have changed tag name | 13:58 |
| spatel | I had similar issue. | 13:59 |
| SvenKieske | spatel: it's okay, just wanted to check something, thanks for the reminder | 14:05 |
| spatel | SvenKieske are you using public images? or building own? | 14:17 |
| SvenKieske | depends, for short lived testing like this I use public stuff due to laziness | 14:30 |
| SvenKieske | building from source yourself is recommended, I'd say :) | 14:30 |
| mmalchuk | https://review.opendev.org/q/I2f879e59a64b9039a8a27c40bd462fde4df73f43 | 14:57 |
| mmalchuk | failed upgrade jobs | 14:57 |
| mmalchuk | so we need merge in backward starting from xena | 14:58 |
| mmalchuk | then recheck each review | 14:58 |
| mnasiadka | yup | 15:06 |
| mmalchuk | I see you merge Xena. thanks | 15:07 |
| mnasiadka | let's hope no more breakages ;) | 15:08 |
| mmalchuk | bless us | 15:08 |
| opendevreview | Merged openstack/kolla-ansible master: always add service_user section to nova.conf https://review.opendev.org/c/openstack/kolla-ansible/+/882893 | 15:17 |
| opendevreview | Merged openstack/kolla stable/xena: toolbox: pin ansible.posix to <1.5.4 https://review.opendev.org/c/openstack/kolla/+/883123 | 16:16 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!