| opendevreview | wu.chunyang proposed openstack/kayobe master: Add python 3.10 to setup.cfg metadata https://review.opendev.org/c/openstack/kayobe/+/886546 | 01:40 |
|---|---|---|
| opendevreview | wu.chunyang proposed openstack/kolla master: Add python 3.10 to setup.cfg metadata https://review.opendev.org/c/openstack/kolla/+/886547 | 01:40 |
| opendevreview | wu.chunyang proposed openstack/kolla-ansible master: Add python 3.10 to setup.cfg metadata https://review.opendev.org/c/openstack/kolla-ansible/+/886548 | 01:40 |
| opendevreview | Merged openstack/kolla stable/2023.1: opensearch-dashboards: Fix permissions https://review.opendev.org/c/openstack/kolla/+/886478 | 01:43 |
| opendevreview | wu.chunyang proposed openstack/kolla-ansible master: Add python 3.10 to setup.cfg metadata https://review.opendev.org/c/openstack/kolla-ansible/+/886548 | 01:47 |
| opendevreview | Maksim Malchuk proposed openstack/kolla stable/yoga: Fix cinder-volume: add missing nvme-cli rpm package https://review.opendev.org/c/openstack/kolla/+/886513 | 07:51 |
| opendevreview | Michal Nasiadka proposed openstack/kolla master: rabbitmq: use erlang-25 ppa on Debian/Ubuntu https://review.opendev.org/c/openstack/kolla/+/886574 | 07:53 |
| opendevreview | Maksim Malchuk proposed openstack/kolla stable/xena: Fix cinder-volume: add missing nvme-cli rpm package https://review.opendev.org/c/openstack/kolla/+/886514 | 07:53 |
| opendevreview | Maksim Malchuk proposed openstack/kolla stable/wallaby: Fix cinder-volume: add missing nvme-cli rpm package https://review.opendev.org/c/openstack/kolla/+/886515 | 07:54 |
| opendevreview | Michal Nasiadka proposed openstack/kolla master: opensearch: drop unnecessary workarounds https://review.opendev.org/c/openstack/kolla/+/886467 | 08:04 |
| opendevreview | Maksim Malchuk proposed openstack/kolla stable/yoga: Fix cinder-volume: add missing nvme-cli rpm package https://review.opendev.org/c/openstack/kolla/+/886513 | 08:38 |
| opendevreview | Pierre Riteau proposed openstack/kayobe master: [DNM] Revert "CI: Enable bare metal testing for Ubuntu upgrades" https://review.opendev.org/c/openstack/kayobe/+/884525 | 08:44 |
| opendevreview | Verification of a change to openstack/kolla stable/zed failed: Disable build of Gnocchi containers https://review.opendev.org/c/openstack/kolla/+/886503 | 09:10 |
| opendevreview | Maksim Malchuk proposed openstack/kolla stable/xena: Fix cinder-volume: add missing nvme-cli rpm package https://review.opendev.org/c/openstack/kolla/+/886514 | 09:13 |
| opendevreview | Adam Stackhouse proposed openstack/kolla-ansible master: Adding mariadb_port to wsrep sync status so alterative ports can be used https://review.opendev.org/c/openstack/kolla-ansible/+/886581 | 09:15 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: Fix issue with octavia security group rules creation https://review.opendev.org/c/openstack/kolla-ansible/+/885831 | 09:19 |
| opendevreview | Adam Stackhouse proposed openstack/kolla-ansible master: Updating proxysql.yaml.j2 to use mariadb_port for backends https://review.opendev.org/c/openstack/kolla-ansible/+/886583 | 09:27 |
| kevko | ^^ i like that some users use my proxysql pathc <3 | 09:29 |
| kevko | :) | 09:29 |
| opendevreview | Verification of a change to openstack/kolla stable/zed failed: Disable build of Gnocchi containers https://review.opendev.org/c/openstack/kolla/+/886503 | 09:30 |
| mnasiadka | There was a topic on the PTG to use it by default instead of the bash script to check galera health - maybe time to propose it? | 09:30 |
| kevko | well, i don't know who (maybe frickler) ..but someone had some reservatons against proxysql ...security or what ...i didn't get it ... we are using for all customers | 09:32 |
| kevko | or fl1ng | 09:34 |
| kevko | fl1nt | 09:34 |
| SvenKieske | never used proxysql myself, but I find the general tech very interesting to solve some mysql scaling problems | 09:34 |
| mnasiadka | kevko: it was fl1nt, but he never came back with details - so let's just discuss that on the meeting today? | 09:34 |
| SvenKieske | I'd be interested to hear why it should not be used | 09:34 |
| kevko | i am open to discuss ... but proxysql is widely used and developer is very nice person | 09:35 |
| mmalchuk | +1 for pexysql | 09:36 |
| mmalchuk | proxysql | 09:36 |
| mnasiadka | kevko: one developer, seems fantastic ;) | 09:36 |
| kevko | mnasiadka: 2012 - 2023 | 09:37 |
| kevko | it's company , but i always seen rene cannano | 09:37 |
| mnasiadka | just laughing | 09:38 |
| mnasiadka | Kolla is like what, less than 10 regular contributors | 09:38 |
| SvenKieske | I remember his presentations at fosdem introducing proxysql, it's at least a company now which is hiring, so I guess they make some money from it | 09:38 |
| kevko | 99 contributors | 09:39 |
| kevko | well, when i was implementing some time ago , the first proposal was maxscale ...but then i realized they are using some business licence which is no free .. | 09:40 |
| kevko | mnasiadka: agree | 09:40 |
| mmalchuk | lets open the 'Kolla Ltd.' company) | 09:46 |
| kevko | i would like to be a director and have my own secretary :) | 09:49 |
| kevko | :D :D | 09:49 |
| kevko | mnasiadka : https://review.opendev.org/c/openstack/kolla/+/886093 << can u add +2 and +w as i already added cells job and working ? | 09:50 |
| kevko | thank you very much | 09:51 |
| opendevreview | Michal Nasiadka proposed openstack/kolla master: rabbitmq: use erlang-25 ppa on Debian/Ubuntu https://review.opendev.org/c/openstack/kolla/+/886574 | 09:53 |
| opendevreview | Michal Nasiadka proposed openstack/kolla master: rabbitmq: use erlang-25 ppa on Debian/Ubuntu https://review.opendev.org/c/openstack/kolla/+/886574 | 09:57 |
| opendevreview | Adam Stackhouse proposed openstack/kolla-ansible master: Closes-Bug: #2024554 Adding mariadb_port to wsrep sync status so alterative ports can be used https://review.opendev.org/c/openstack/kolla-ansible/+/886581 | 10:00 |
| opendevreview | Adam Stackhouse proposed openstack/kolla-ansible master: Closes-Bug: #2024554 * Adding mariadb_port to wsrep sync status so alterative ports can be used https://review.opendev.org/c/openstack/kolla-ansible/+/886581 | 10:02 |
| opendevreview | Adam Stackhouse proposed openstack/kolla-ansible master: Adding mariadb_port to wsrep sync status so alterative ports can be used https://review.opendev.org/c/openstack/kolla-ansible/+/886581 | 10:03 |
| opendevreview | Adam Stackhouse proposed openstack/kolla-ansible master: Adding mariadb_port to wsrep sync status so alternative ports can be used https://review.opendev.org/c/openstack/kolla-ansible/+/886581 | 10:04 |
| opendevreview | Michal Nasiadka proposed openstack/kolla master: rabbitmq: use erlang-25 ppa on Debian/Ubuntu https://review.opendev.org/c/openstack/kolla/+/886574 | 10:48 |
| opendevreview | Merged openstack/kolla stable/yoga: Disable build of Gnocchi containers https://review.opendev.org/c/openstack/kolla/+/886504 | 11:02 |
| opendevreview | Verification of a change to openstack/kolla master failed: opensearch: drop unnecessary workarounds https://review.opendev.org/c/openstack/kolla/+/886467 | 11:32 |
| opendevreview | Verification of a change to openstack/kolla master failed: opensearch: drop unnecessary workarounds https://review.opendev.org/c/openstack/kolla/+/886467 | 12:14 |
| frickler | really need to get a grip on those mysql failures. how certain are we that https://review.opendev.org/c/openstack/kolla-ansible/+/884182 would fix those? | 12:22 |
| *** mgoddard- is now known as mgoddard | 12:28 | |
| mnasiadka | frickler: you mean the ones in stable branches? | 12:32 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible stable/2023.1: Refactor MariaDB and RabbitMQ restart procedure https://review.opendev.org/c/openstack/kolla-ansible/+/886485 | 12:32 |
| mnasiadka | first of all we need this in antelope | 12:32 |
| SvenKieske | strictly speaking, I didn't see that it was mentioned this would fix any mysql failures? should it? | 12:34 |
| opendevreview | Merged openstack/kolla master: Bump proxysql version https://review.opendev.org/c/openstack/kolla/+/886093 | 12:36 |
| mnasiadka | we've been seeing some mariadb deployment failures mainly on debian/ubuntu in stable branches from time to time, I think frickler is speculating a bit it would fix those - I've been thinking the same | 12:36 |
| opendevreview | Michal Nasiadka proposed openstack/kolla master: Revert "Pin setuptools=67.2.*" https://review.opendev.org/c/openstack/kolla/+/876460 | 12:36 |
| mnasiadka | Don't we need to backport 886389: Fix rabbitmq image build | https://review.opendev.org/c/openstack/kolla/+/886389 to yoga and earlier? | 12:40 |
| mmalchuk | not really, images built because of folsom | 12:43 |
| mmalchuk | focal | 12:43 |
| mmalchuk | focal not affected | 12:43 |
| opendevreview | Merged openstack/kolla stable/xena: Disable build of Gnocchi containers https://review.opendev.org/c/openstack/kolla/+/886506 | 12:43 |
| kevko | it's needed only if we want to have the same code in a branches ...but as mmalchuk said .. focal is not affected | 12:44 |
| mnasiadka | ah ok, so let's leave as is | 12:44 |
| mmalchuk | but whats about aarch64? maybe backport to have new changes cherrypicked without conflicts? | 12:48 |
| mmalchuk | not affected but I'm +1 to backport | 12:49 |
| mmalchuk | to have same code only | 12:50 |
| opendevreview | Merged openstack/kolla-ansible master: Add python 3.10 to setup.cfg metadata https://review.opendev.org/c/openstack/kolla-ansible/+/886548 | 12:50 |
| opendevreview | Jakub Darmach proposed openstack/kolla master: Designate infoblox port 53 permission denied https://review.opendev.org/c/openstack/kolla/+/886636 | 12:52 |
| mnasiadka | mgoddard mnasiadka hrw bbezak frickler kevko SvenKieske mmalchuk - meeting in 5 | 12:55 |
| frickler | I was thinking about the failures in master actually, like https://review.opendev.org/c/openstack/kolla/+/886467 | 12:57 |
| mnasiadka | in the ubuntu-upgrade job? | 12:59 |
| opendevreview | Merged openstack/kolla stable/zed: Disable build of Gnocchi containers https://review.opendev.org/c/openstack/kolla/+/886503 | 13:00 |
| mnasiadka | #startmeeting kolla | 13:00 |
| opendevmeet | Meeting started Wed Jun 21 13:00:52 2023 UTC and is due to finish in 60 minutes. The chair is mnasiadka. Information about MeetBot at http://wiki.debian.org/MeetBot. | 13:00 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 13:00 |
| opendevmeet | The meeting name has been set to 'kolla' | 13:00 |
| mnasiadka | #topic rollcall | 13:00 |
| mnasiadka | o/ | 13:00 |
| ramona-beermann[m] | o/ | 13:01 |
| frickler | \o | 13:01 |
| kevko | \o/ | 13:01 |
| bbezak | o/ | 13:01 |
| SvenKieske | o/ | 13:01 |
| ihalomi | \o | 13:01 |
| mmalchuk | \o | 13:03 |
| mnasiadka | #topic agenda | 13:03 |
| mnasiadka | * CI status | 13:03 |
| mnasiadka | * Release tasks | 13:03 |
| mnasiadka | * Regular stable releases (first meeting in a month) | 13:03 |
| mnasiadka | * Current cycle planning | 13:03 |
| mnasiadka | * Additional agenda (from whiteboard) | 13:03 |
| mnasiadka | * Open discussion | 13:03 |
| mnasiadka | #topic CI status | 13:03 |
| mnasiadka | It was red, but we're getting to green with RMQ and gnocchi unbuildable fixes | 13:03 |
| opendevreview | Verification of a change to openstack/kayobe master failed: Revert "Use OpenStack 2023.1 release" https://review.opendev.org/c/openstack/kayobe/+/886313 | 13:03 |
| opendevreview | Verification of a change to openstack/kayobe master failed: set previous_release to 2023.1 https://review.opendev.org/c/openstack/kayobe/+/886396 | 13:03 |
| kevko | btw, sorry for stupid question ..but where i can check ci status page for kolla, kolla-ansible ? | 13:04 |
| frickler | well yellow at best I'd say | 13:04 |
| frickler | lots of failures like https://review.opendev.org/c/openstack/kolla/+/886467/comments/ed6de50a_0c6c9a3b needing recheck | 13:04 |
| mnasiadka | yeah, looking now it's yellow ;) | 13:04 |
| mnasiadka | kevko: we basically update the whiteboard with status, of what we see every day | 13:04 |
| SvenKieske | kevko: some of it here: https://grafana.opendev.org/d/c0d59dad13/kolla-failure-rate?orgId=1 | 13:04 |
| mnasiadka | there is a dashboard on grafana.opendev - but it needs some love | 13:04 |
| spatel | \o/ | 13:05 |
| SvenKieske | it got some love from you, but it could certainly need more :) | 13:05 |
| kevko | thanks, bookmarked | 13:05 |
| SvenKieske | sources are here: https://opendev.org/openstack/project-config/src/branch/master/grafana/kolla.yaml | 13:06 |
| mnasiadka | #topic Release tasks | 13:08 |
| mnasiadka | So, we're on the path to use master again on master | 13:08 |
| mnasiadka | so that's covered | 13:08 |
| mnasiadka | #topic Current cycle planning | 13:08 |
| mnasiadka | ihalomi: how's podman? | 13:08 |
| ihalomi | not good, im stuck on problem with rabbitmq | 13:08 |
| ihalomi | solution about mounting subdirectories of run and using tmpfiles.d config seems working but i cant make deployment finish successfully | 13:09 |
| kevko | some log ? | 13:10 |
| ihalomi | it fails on podman exec rabbitmq rabbitmqctl wait /var/lib/rabbitmq/mnesia/rabbitmq.pid | 13:10 |
| ihalomi | where it timeouts with Waiting for pid file '/var/lib/rabbitmq/mnesia/rabbitmq.pid' to appear but pid file is there and when I check that process is running | 13:11 |
| SvenKieske | that seems like not a correct directory to write a pid file? sure it is there? oO | 13:11 |
| SvenKieske | ah ok | 13:11 |
| SvenKieske | did you check permissions? can rabbitmq user access the pid file?` have seen such errors many times | 13:12 |
| ihalomi | i didnt change anything in there so it looks like some hidden podman problem | 13:12 |
| SvenKieske | if you could share all this stuff on some etherpad or somewhere maybe ppl could help :) | 13:12 |
| ihalomi | i can write some summary and post the newest patchset so you can look | 13:13 |
| SvenKieske | that would be nice | 13:13 |
| mnasiadka | good progress anyway ;) | 13:14 |
| mnasiadka | #topic Additional agenda (from whiteboard) | 13:14 |
| mnasiadka | nothing on the whiteboard | 13:15 |
| mnasiadka | but I have one | 13:15 |
| mnasiadka | Wallaby and Xena EOL | 13:15 |
| ihalomi | is this correct permissions for that pid? -rw-r--r--. 1 rabbitmq rabbitmq 2 Jun 21 08:56 /var/lib/rabbitmq/mnesia/rabbitmq.pid | 13:15 |
| mnasiadka | Anybody wants to maintain those branches or are we marking them as EOL? | 13:15 |
| mmalchuk | lets fixes on review landed and ok for EOL | 13:16 |
| mnasiadka | ihalomi: correct | 13:16 |
| mnasiadka | ok, so we should have a look on all reviews in open state for those branches and post EOL patch | 13:16 |
| frickler | as already mentioned +1 for EOL from me | 13:16 |
| mmalchuk | mnasiadka I can only Xena | 13:16 |
| SvenKieske | ihalomi: looks like it; I hope nobody needs execute rights on this stuff | 13:17 |
| ihalomi | also when i was rebasing on the newest patch there was few newly added docker commands so could you try keep in mind that now kolla_container_engine is used? fix is here https://review.opendev.org/c/openstack/kolla-ansible/+/886391 | 13:17 |
| mnasiadka | ihalomi: we'll try our best ;) | 13:18 |
| frickler | ihalomi: oh, that's going to be tough I fear. can we lint that somehow? | 13:18 |
| mnasiadka | we could probably define a custom check to ansible lint? | 13:19 |
| frickler | we could make a CI job where we rename the command maybe. mv docker foobar | 13:19 |
| frickler | or reviewers could just try to keep that in mind, yes | 13:20 |
| mnasiadka | custom lint rule would probably be better, but can take some more time ;) | 13:21 |
| mnasiadka | ok, so I think we have an agreement on EOL | 13:22 |
| mnasiadka | #agreed to mark Wallaby and Xena EOL | 13:22 |
| mnasiadka | #topic Open discussion | 13:22 |
| mnasiadka | anyone anything? | 13:22 |
| mmalchuk | https://review.opendev.org/q/topic:fix-app-cred-access-rules | 13:22 |
| mmalchuk | discussed before | 13:22 |
| mmalchuk | not breaking changes | 13:23 |
| mmalchuk | lets merge | 13:23 |
| mmalchuk | 2. lack of reviewers in Kayobe | 13:23 |
| mmalchuk | https://review.opendev.org/c/openstack/kayobe/+/879554 | 13:24 |
| mmalchuk | https://review.opendev.org/c/openstack/kayobe/+/861397 | 13:24 |
| mmalchuk | ask from time to time and nothing changes( | 13:24 |
| mnasiadka | and vacation season is approaching, so it's not going to get better | 13:24 |
| * mmalchuk sad | 13:25 | |
| mmalchuk | +RP ? | 13:25 |
| mnasiadka | on the Kolla User Forum in Vancouver - there was a proposal on supporting multiple RabbitMQ clusters (or at least one additional for notifications) | 13:25 |
| mnasiadka | anybody here has similar thoughts? | 13:25 |
| mmalchuk | +RP ? | 13:26 |
| kevko | mnasiadka this would be helfull for multiple cells | 13:27 |
| mnasiadka | yeah, that's another thing - multiple cells | 13:27 |
| SvenKieske | this is also a recommendation from the large scale sig afaik | 13:27 |
| SvenKieske | so it's a good feature to have, I'd say | 13:28 |
| mnasiadka | If somebody is giving - sure :) | 13:28 |
| kevko | mnasiadka multiple rabbitmq feature can be copied from my patch for mariadb | 13:28 |
| mnasiadka | labedz: if you guys are willing to work on this (as spoken on the Summit) and plan to submit a patch this cycle - I can add that to the cycle priorities | 13:29 |
| frickler | and then we need a 6-node CI job to validate it? | 13:29 |
| mnasiadka | we had 6 node CI job for Monasca | 13:29 |
| kevko | well, you can spin 2/3 one node rabbitmq cluster i think | 13:29 |
| kevko | two | 13:30 |
| mnasiadka | I don't know how big the cells job is currently | 13:30 |
| kevko | one node and two nodes if i remember correctly | 13:30 |
| kevko | with two mariadb proxied through proxysql ..but one rabbit | 13:30 |
| mnasiadka | so it should be easy to improve that, once somebody wants to work on it :) | 13:31 |
| mnasiadka | ok, let's try to stabilise things before next week | 13:31 |
| mnasiadka | btw - I'm going on vacation again for 3 weeks of July | 13:31 |
| mnasiadka | Is anybody going to be around to run the meetings - or should we cancel them? | 13:31 |
| mnasiadka | (that's mainly vacation season everywhere) | 13:32 |
| mmalchuk | me too on vacation in July) | 13:32 |
| frickler | fine to cancel I'd say, we can discuss things ad hoc as needed | 13:33 |
| mnasiadka | ok then, I'll send a mail next week to the ML | 13:33 |
| mnasiadka | That's all for me for today - thanks for coming :) | 13:33 |
| mnasiadka | #endmeeting | 13:33 |
| opendevmeet | Meeting ended Wed Jun 21 13:33:36 2023 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 13:33 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/kolla/2023/kolla.2023-06-21-13.00.html | 13:33 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/kolla/2023/kolla.2023-06-21-13.00.txt | 13:33 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/kolla/2023/kolla.2023-06-21-13.00.log.html | 13:33 |
| frickler | moving a bit to offtopic, is there a difference between vacation and holiday? | 13:33 |
| mmalchuk | thanks mnasiadka | 13:33 |
| mmalchuk | big difference | 13:33 |
| mnasiadka | frickler: I don't know, maybe it's a US vs Europe thing, there's also PTO ;-) | 13:34 |
| mmalchuk | holiday at home, vacation on the seashore) | 13:34 |
| mmalchuk | please +w https://review.opendev.org/c/openstack/kolla/+/886515 | 13:35 |
| SvenKieske | vacation is mostly an americanism of holiday, according to https://grammar.collinsdictionary.com/de/verwendung-von-englisch/what-is-the-difference-between-holiday-and-vacation | 13:35 |
| kevko | can I ask ? https://zuul.opendev.org/t/openstack/build/5ae19a520e68495f83c9c0250a253976 << why rabbitmq, kolla-toolbox and gnocchi failing ... | 13:35 |
| kevko | i think we already merged no ? | 13:35 |
| kevko | i mean fixes for them | 13:35 |
| mnasiadka | yes, but it looks like mirror issue | 13:36 |
| mnasiadka | Failed to fetch https://ppa1.novemberain.com/rabbitmq/rabbitmq-erlang/deb/ubuntu/dists/focal/main/binary-amd64/Packages.gz File has unexpected size (102916 != 95299). Mirror sync in progress? [IP: 2606:4700:3035::ac43:8c3f 443] | 13:36 |
| frickler | gnocchi fix wasn't working, marked as unbuildable | 13:36 |
| kevko | Failed to fetch https://ppa1.novemberain.com/rabbitmq/rabbitmq-erlang/deb/ubuntu/dists/focal/main/binary-amd64/Packages.gz File has unexpected size (102916 != 95299). Mirror sync in progress? [IP: 2606:4700:3035::ac43:8c3f 443] | 13:36 |
| mmalchuk | kevko gnocchi? rebase on top | 13:36 |
| kevko | this is weird | 13:36 |
| mnasiadka | gnocchi is not failing in this change | 13:36 |
| frickler | typical mirror updates being done in an unsafe way | 13:36 |
| mmalchuk | kevko said) | 13:37 |
| kevko | yeah, sorry .. i was too quick | 13:37 |
| kevko | so, recheck will help ? | 13:37 |
| mnasiadka | kevko: ppa1.novemberain.com is some personal mirror of a RMQ/VMWare employee, maybe it's not that reliable after all ;) | 13:37 |
| frickler | mmalchuk: please don't rebase. gerrit and zuul rebase on their own as needed. recheck is enough | 13:37 |
| kevko | or i should wait for ppa.launchpad.net revert ? | 13:37 |
| mnasiadka | revert but not revert ;) | 13:37 |
| mmalchuk | frickler I know) | 13:37 |
| kevko | you know what i meant :D | 13:38 |
| mmalchuk | frickler there is no issue already | 13:38 |
| mnasiadka | https://review.opendev.org/c/openstack/kolla/+/886574 | 13:38 |
| mnasiadka | this one | 13:38 |
| kevko | so let's merge | 13:38 |
| kevko | https://review.opendev.org/c/openstack/kolla/+/886574 | 13:38 |
| kevko | frickler ^ ? | 13:38 |
| mnasiadka | let's wait for zuul and merge | 13:39 |
| frickler | I was waiting on the discussion to settle down on the patch, approved now | 13:39 |
| mnasiadka | frickler: the mariadb problems in master that I've seen now - they happen in reconfigure step - because we restart all three nodes at the same time | 13:39 |
| mnasiadka | and we do that because of a bug in our systemd code ;-) | 13:40 |
| kevko | what bug ? | 13:40 |
| mnasiadka | we use docker_restart_policy: no in CI | 13:40 |
| mnasiadka | and we always compare the systemd unit, but don't write it out when docker_restart_policy==no | 13:40 |
| mnasiadka | ;-) | 13:41 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: Introduce oneshot docker_restart_policy https://review.opendev.org/c/openstack/kolla-ansible/+/881707 | 13:41 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: Introduce oneshot docker_restart_policy https://review.opendev.org/c/openstack/kolla-ansible/+/881707 | 13:41 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: Introduce oneshot docker_restart_policy https://review.opendev.org/c/openstack/kolla-ansible/+/881707 | 13:41 |
| mnasiadka | that one might help, but needs probably some more love - let's see | 13:42 |
| mmalchuk | mnasiadka after oneshot restart should we recover mariadb? | 13:49 |
| mmalchuk | really don't understand how this would help | 13:50 |
| mnasiadka | mmalchuk: as I said, when docker_restart_policy == no we are not creating the systemd unit, but we are always comparing the systemd unit, and since we'd like in CI to have the systemd units, but with Restart=No - then introducing a new policy that sets Restart=No helps | 14:00 |
| mnasiadka | mmalchuk: and we run bootstrap containers for a lot of services that should not get a systemd unit, that's why "oneshot" | 14:00 |
| mnasiadka | ;-) | 14:00 |
| mmalchuk | for all services as I see | 14:01 |
| mmalchuk | okay | 14:02 |
| mmalchuk | let's test this widely | 14:02 |
| mmalchuk | thanks for clarification | 14:02 |
| SvenKieske | mmalchuk: I left some comments on "mdns" aka mini dns vs "mdns" aka "multicast dns"; I cursed a lot at the designate devs already in the past regarding that naming clash.. | 14:05 |
| mmalchuk | i see, thanks | 14:06 |
| mmalchuk | mdns lol | 14:06 |
| * mmalchuk didn't use infoblox | 14:06 | |
| SvenKieske | it's beyond me how you can not check for basic naming clashes in the domain your working in and then go on ignoring them.. | 14:06 |
| mmalchuk | there is no 'mini dns' in the domain) this is copyright of the infoblox only | 14:07 |
| SvenKieske | ah, I don't know anything about infoblox, but why did designate devs incorporated a company term in their openstack architecture design as a service name? | 14:12 |
| SvenKieske | seems even more insane, imvho | 14:12 |
| opendevreview | Adam Stackhouse proposed openstack/kolla-ansible master: Adding mariadb_port to wsrep sync status so alterative ports can be used https://review.opendev.org/c/openstack/kolla-ansible/+/886581 | 14:13 |
| opendevreview | Adam Stackhouse proposed openstack/kolla-ansible master: Adding mariadb_port to wsrep sync status so alterative ports can be used https://review.opendev.org/c/openstack/kolla-ansible/+/886581 | 14:16 |
| mmalchuk | SvenKieske agree. anyway thanks) | 14:17 |
| opendevreview | Adam Stackhouse proposed openstack/kolla-ansible master: Adding mariadb_port to wsrep sync status so alterative ports can be used https://review.opendev.org/c/openstack/kolla-ansible/+/886581 | 14:17 |
| opendevreview | Verification of a change to openstack/kolla stable/zed failed: Fix cinder-volume: add missing nvme-cli rpm package https://review.opendev.org/c/openstack/kolla/+/886512 | 14:17 |
| opendevreview | Ivan Halomi proposed openstack/kolla-ansible master: Add support of podman deployment https://review.opendev.org/c/openstack/kolla-ansible/+/799229 | 14:20 |
| ihalomi | here is etherpad with rabbitmq problem https://etherpad.opendev.org/p/rabbitmq_pid_issue | 14:23 |
| spatel | mmalchuk I am stuck at nova token issue.. How do it fix. I have replied you in mailing list | 14:24 |
| opendevreview | Verification of a change to openstack/kolla stable/2023.1 failed: Fix cinder-volume: add missing nvme-cli rpm package https://review.opendev.org/c/openstack/kolla/+/886511 | 14:25 |
| mmalchuk | spatel I see, don't know how to help right now. may be you use an old containers? | 14:27 |
| opendevreview | Verification of a change to openstack/kolla-ansible master failed: Fix issue with octavia security group rules creation https://review.opendev.org/c/openstack/kolla-ansible/+/885831 | 14:27 |
| spatel | mmalchuk I believe my issue is that nova.conf didn't get updated by kolla-ansible because its stuck in upgrade check error | 14:28 |
| spatel | This is look like chicken-egg issue. | 14:28 |
| spatel | kolla-ansible can't update nova.conf file for service_user snippet until upgrade TASK move forward.. | 14:30 |
| spatel | Let me blowup container and try | 14:30 |
| mmalchuk | you can run ansible with -vvvv and check the results | 14:31 |
| SvenKieske | spatel: the timestamps you mailed indicate that you didn't deploy this deployment after this was patched, so you most likely still have the old versions | 14:32 |
| spatel | SvenKieske I think you might be correct. | 14:33 |
| SvenKieske | so verify that you have the appropriate commits mmalchuk mailed you. in the worst case you could edit the conf files manually I guess | 14:33 |
| spatel | I was running yoga and upgraded to Zed and then antelop | 14:33 |
| opendevreview | Adam Stackhouse proposed openstack/kolla-ansible master: Updating proxysql.yaml.j2 to use mariadb_port for backends https://review.opendev.org/c/openstack/kolla-ansible/+/886583 | 14:34 |
| spatel | Just trying to understand what went wrong with me that it didn't push that patch | 14:34 |
| spatel | my /etc/kolla/nova-api/nova.conf doesn't have service_user that means I need to add there | 14:35 |
| opendevreview | Adam Stackhouse proposed openstack/kolla-ansible master: Updating proxysql.yaml.j2 to use mariadb_port for backends https://review.opendev.org/c/openstack/kolla-ansible/+/886583 | 14:38 |
| spatel | mmalchuk it works now after adding service_user manually.. | 14:39 |
| mmalchuk | lol. manual upgrade?) | 14:39 |
| spatel | Yes I have added https://paste.opendev.org/show/bo7J3hQn0ouHN8fXX3vq/ in /etc/kolla/nova-api/nova.conf and run kolla-ansible -i multinode upgrade -t nova | 14:40 |
| spatel | it works! without errro | 14:40 |
| spatel | that piece was missing | 14:41 |
| spatel | I don't know how that was missing.. must be some race condition | 14:41 |
| opendevreview | Adam Stackhouse proposed openstack/kolla-ansible master: Adding mariadb_port to wsrep sync status so alterative ports can be used https://review.opendev.org/c/openstack/kolla-ansible/+/886581 | 14:41 |
| mmalchuk | spatel ineresting. the upgrade jobs passed on CI without any problems | 14:42 |
| spatel | I will re-build my lab and give it a try again.. | 14:42 |
| spatel | I don't like surprises | 14:43 |
| mmalchuk | create the bugreport if you find something | 14:43 |
| spatel | +1 | 14:43 |
| spatel | mmalchuk I have question I can see service_user snippet in /opt/venv-kolla/share/kolla-ansible/ansible/roles/nova/templates/nova.conf.j2 then why ansible didn't push it to /etc/kolla/nova-api/nova.conf ? | 14:50 |
| spatel | at what stage that template initialized ? | 14:51 |
| opendevreview | Verification of a change to openstack/kolla stable/zed failed: Fix cinder-volume: add missing nvme-cli rpm package https://review.opendev.org/c/openstack/kolla/+/886512 | 14:52 |
| opendevreview | Verification of a change to openstack/kolla stable/2023.1 failed: Fix cinder-volume: add missing nvme-cli rpm package https://review.opendev.org/c/openstack/kolla/+/886511 | 14:58 |
| mmalchuk | spatel config or config-bootstrap | 14:59 |
| mmalchuk | tasks ^^^ | 14:59 |
| mmalchuk | so deploy or rolling_upgrade stage | 15:00 |
| spatel | Hmm! This is what is did to upgrade zed to antalop tell me if i miss something | 15:01 |
| spatel | https://paste.opendev.org/show/bmw4IuHC1ndHnD9z8Mkr/ | 15:01 |
| mnasiadka | using pypi releases is not the best idea, to have the latest patches (and latest bugs) install directly from git ;-) | 15:02 |
| spatel | Does pip3 install --upgrade kolla-ansible==16.0.0 will upgrade all configuration in side /opt/venv directory? | 15:02 |
| opendevreview | Merged openstack/kolla stable/yoga: Fix cinder-volume: add missing nvme-cli rpm package https://review.opendev.org/c/openstack/kolla/+/886513 | 15:03 |
| opendevreview | Merged openstack/kolla stable/xena: Fix cinder-volume: add missing nvme-cli rpm package https://review.opendev.org/c/openstack/kolla/+/886514 | 15:03 |
| spatel | mnasiadka that could be my issue :( | 15:03 |
| mmalchuk | spatel you can see upgraded or not in the output of the pip install --upgrade command | 15:04 |
| mmalchuk | and youre brave one to run upgrade without prechecks | 15:04 |
| mnasiadka | spatel: and you could have used code before we merged the OSSA service_token patches for Nova and Cinder, so upgrade failed on upgrade checkers | 15:04 |
| spatel | I don't have that output in hand but look like it doesn't upgrade my /opt/venv... :( | 15:04 |
| mnasiadka | sometimes it's good to use refreshed code and run deploy before you upgrade | 15:05 |
| spatel | mnasiadka totally.. | 15:05 |
| spatel | what method you prefer for upgrade? | 15:05 |
| spatel | pip3 install --upgrade git+https://opendev.org/openstack/kolla-ansible@zed ? | 15:06 |
| mnasiadka | stable/zed, but yes ;) | 15:06 |
| spatel | Ok.. I will try that out | 15:06 |
| mmalchuk | mnasiadka should we update docs about the upgrade procedure? | 15:08 |
| spatel | It would be great!! | 15:08 |
| mnasiadka | mmalchuk: feel free ;-) | 15:08 |
| mnasiadka | or even spatel could contribute if he's interested | 15:09 |
| spatel | current doc is little confusion one.. | 15:09 |
| mnasiadka | when this rmq drama will end... | 15:09 |
| spatel | it would be good to put some TIPs to validate if codes got updated or not :) I am very interested but I am new also so not sure my input worth expert input :( | 15:09 |
| mmalchuk | mnasiadka good idea. spatel you can refer the code from tests/run.yml and tests/upgrade.sh to change the documentation | 15:11 |
| spatel | Give me sometime.. let me try both method pip3 install --upgrade and pip3 install --upgrade git+https://opendev.org/openstack/kolla-ansible@zed and see what is the difference between them | 15:11 |
| spatel | I can come up with some doc which you guys can review and put your input | 15:12 |
| mnasiadka | spatel: that's how it usually works | 15:12 |
| spatel | Just fyi, I can see my virtual environment files got updated with pip3 install --upgrade kolla-ansible==16.0.0 | 15:16 |
| spatel | that shows it has updated all ansible roles. Now question is why ansible didn't push them to /etc/kolla/nova-api/nova.conf ? | 15:16 |
| spatel | If i have template in virtual environment then ansible should update them on each run | 15:17 |
| mmalchuk | spatel btw, you can use: pip3 install --upgrade 'kolla-ansible==16.*.*' | 15:18 |
| mmalchuk | this will install the latest 16th version exist in pypi | 15:19 |
| mmalchuk | don't forgot quotes) | 15:19 |
| spatel | good to know! in past i did many upgrade using that method and i believe it work for me but this time i don't know what was wrong | 15:19 |
| mnasiadka | scottsol: are you using Infoblox Designate backend by any chance? | 15:19 |
| mnasiadka | frickler: https://review.opendev.org/c/openstack/kolla-ansible/+/886485 should we merge that? | 15:28 |
| SvenKieske | I agree on updating the docs, last week there was already some discussion why some bugfix was not released as a pip package, we should probably warn with big red letters to use the stable git branches. don't want to know how many people are using outdated releases and think they have the new stuff | 15:29 |
| mnasiadka | we should have a note that while we do release every month (or strive to), we recommend using stable branches from git | 15:31 |
| mnasiadka | SvenKieske: basically we could try to redo the docs, we've had multiple approaches to it, and they usually failed - but I would like to hear from a new user like spatel - what really needs rework and how can we make the entry point easier from a user perspective :) | 15:38 |
| mnasiadka | spatel: would you be able to provide such feedback? etherpad with some summary would be nice | 15:39 |
| spatel | mnasiadka I will | 15:39 |
| spatel | Let me produce rough doc and challenges and put it on etherpad to review/edit | 15:40 |
| SvenKieske | spatel: very cool! ty! | 15:43 |
| SvenKieske | mnasiadka: agreed | 15:43 |
| frickler | mnasiadka: maybe wait until next week to give it some more testing on master? | 15:53 |
| mmalchuk | mnasiadka please +W the last backport of nvme-cli https://review.opendev.org/c/openstack/kolla/+/886515 | 15:55 |
| mmalchuk | thanks | 15:55 |
| opendevreview | Maksim Malchuk proposed openstack/kolla stable/2023.1: Fix cinder-volume: add missing nvme-cli rpm package https://review.opendev.org/c/openstack/kolla/+/886511 | 15:58 |
| mmalchuk | mnasiadka and this one, sorry: https://review.opendev.org/c/openstack/kolla/+/886511 | 15:59 |
| kevko | haha, i am not using kolla version for years :D | 16:00 |
| kevko | i have my own pretty nice docker image :P | 16:00 |
| mmalchuk | there issues with rabbitmq mirror sync... looks like it synced during working hours( | 16:01 |
| mnasiadka | it gets fixed with 886574 - since we are going back to ppa's on launchpad, not somebody's home computer | 16:07 |
| mnasiadka | frickler: sure, but that also stops pinning ansible-core version on antelope, which might be a bit confusing for users | 16:08 |
| mmalchuk | fixed, but not merged yet | 16:16 |
| opendevreview | Merged openstack/kolla master: rabbitmq: use erlang-25 ppa on Debian/Ubuntu https://review.opendev.org/c/openstack/kolla/+/886574 | 16:19 |
| mmalchuk | now merged))) | 16:19 |
| frickler | mnasiadka: ah, I keep forgetting about that link. then tomorrow I'd say | 16:35 |
| opendevreview | Michal Nasiadka proposed openstack/kolla stable/2023.1: rabbitmq: use erlang-25 ppa on Debian/Ubuntu https://review.opendev.org/c/openstack/kolla/+/886672 | 17:17 |
| opendevreview | Michal Nasiadka proposed openstack/kolla stable/zed: rabbitmq: use erlang-25 ppa on Debian/Ubuntu https://review.opendev.org/c/openstack/kolla/+/886673 | 17:17 |
| opendevreview | Michal Nasiadka proposed openstack/kolla stable/yoga: rabbitmq: use erlang-25 ppa on Debian/Ubuntu https://review.opendev.org/c/openstack/kolla/+/886674 | 17:17 |
| mnasiadka | argh, abandoned yoga, not needed | 17:19 |
| opendevreview | Michal Nasiadka proposed openstack/kolla master: Revert "rabbitmq: pin to 3.11.16" https://review.opendev.org/c/openstack/kolla/+/886675 | 17:20 |
| mnasiadka | that can be reverted now, and tomorrow I'll try to fix centos/rocky aarch64 and we're done with RMQ for this week... | 17:20 |
| opendevreview | Merged openstack/kolla stable/zed: Fix cinder-volume: add missing nvme-cli rpm package https://review.opendev.org/c/openstack/kolla/+/886512 | 17:23 |
| opendevreview | Maksim Malchuk proposed openstack/kolla stable/2023.1: Fix cinder-volume: add missing nvme-cli rpm package https://review.opendev.org/c/openstack/kolla/+/886511 | 18:14 |
| opendevreview | Verification of a change to openstack/kolla-ansible master failed: Fix issue with octavia security group rules creation https://review.opendev.org/c/openstack/kolla-ansible/+/885831 | 18:18 |
| opendevreview | Verification of a change to openstack/kolla-ansible master failed: Fix issue with octavia security group rules creation https://review.opendev.org/c/openstack/kolla-ansible/+/885831 | 20:18 |
| BobZAnnapolis_ | apologise if this is the wrong channel; kayobe aio install is failing at the 'control host bootstrap' step w/the following msg : "[WARNING]: - mrlesmithjr.manage-lvm was NOT installed successfully: - sorry, mrlesmithjr.manage-lvm was not found on https://galaxy.ansible.com/api/." ideas ? tia | 20:53 |
| mmalchuk | BobZAnnapolis_ you have an old code. this issue already fixed in all versions. | 21:08 |
| mmalchuk | which version you use? | 21:09 |
| opendevreview | Merged openstack/kolla stable/2023.1: Fix cinder-volume: add missing nvme-cli rpm package https://review.opendev.org/c/openstack/kolla/+/886511 | 22:05 |
| opendevreview | Verification of a change to openstack/kolla stable/2023.1 failed: rabbitmq: use erlang-25 ppa on Debian/Ubuntu https://review.opendev.org/c/openstack/kolla/+/886672 | 23:20 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!