| opendevreview | Dawud proposed openstack/kolla master: Change logic for plugins in grafana image build https://review.opendev.org/c/openstack/kolla/+/913184 | 00:19 |
|---|---|---|
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible stable/2023.2: Fix kolla-ansible systemd restart behaviour https://review.opendev.org/c/openstack/kolla-ansible/+/923020 | 05:49 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible stable/2023.1: Fix kolla-ansible systemd restart behaviour https://review.opendev.org/c/openstack/kolla-ansible/+/923021 | 05:49 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible stable/2023.1: Fix kolla-ansible systemd restart behaviour https://review.opendev.org/c/openstack/kolla-ansible/+/923021 | 05:50 |
| opendevreview | Roman Krček proposed openstack/kolla-ansible master: Performance: use filters for service dicts https://review.opendev.org/c/openstack/kolla-ansible/+/914997 | 07:15 |
| SvenKieske | morning | 07:44 |
| SvenKieske | \o/ | 07:44 |
| mnasiadka | SvenKieske: so tell me - what's the chance that we break people that don't use docker live-restore with that patch? ;-) | 08:02 |
| SvenKieske | it's alright, I shouldn't comment that late apparently :D | 08:02 |
| SvenKieske | I already had the start of a headache, I should've left it at that point, I suppose. | 08:03 |
| mnasiadka | haha :) | 08:04 |
| SvenKieske | I added 4 small patches to the whiteboard, I'll not be present next week, except monday (vacation, yay!) but these are only asking for review/RP+1, so I guess no input needed from my side. | 08:04 |
| mnasiadka | I will be off on Monday, but will have a look next week ;) | 08:04 |
| SvenKieske | any stuff you guys need reviews on? I guess I'll look over the stuff from other people on the whiteboard | 08:08 |
| opendevreview | Matúš Jenča proposed openstack/kolla-ansible master: Patch service-cert-copy role to be used w/o HAProxy https://review.opendev.org/c/openstack/kolla-ansible/+/915901 | 08:22 |
| kevko | morning \o/ | 08:49 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: Fix prechecks for interfaces with dashes https://review.opendev.org/c/openstack/kolla-ansible/+/922478 | 08:49 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: CI: test interface names with dashes https://review.opendev.org/c/openstack/kolla-ansible/+/794545 | 08:49 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: CI: test interface names with dashes https://review.opendev.org/c/openstack/kolla-ansible/+/794545 | 08:50 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: CI: test interface names with dashes https://review.opendev.org/c/openstack/kolla-ansible/+/794545 | 08:50 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: [DNM] Debug podman failed if healthcheck is disabled https://review.opendev.org/c/openstack/kolla-ansible/+/923032 | 08:57 |
| kevko | guys, i found a problem in our kolla-ansible code and i really don't know how to deal with it | 09:00 |
| kevko | it's problem with ansible_python_interpreter and venv ... and i think there is no fix for it :/ | 09:02 |
| kevko | i have some ideas how to fix ..but it's little bit exotic fixes :D | 09:02 |
| kevko | and we probably need to resolve somehow with debian bookworm and ubuntu mantic coming :/ | 09:04 |
| jovial | What is the problem? | 09:05 |
| kevko | well, for now i am implementing some stuff for testing ... tempest/rally ...but that's not interisting for now ...main problem is that in some places in a code i am using openstack.cloud module which has dependency for openstacksdk ... as you know ..the workflow for openstack.cloud is -> use the module -> delegate_to localhost ...etc ... but i | 09:07 |
| kevko | have openstacksdk installed in python environment ..so ansible_python_interpreter doesn't fit the location | 09:07 |
| kevko | moreover in CI we are using global override in globals to ansible_python_interpreter: /usr/bin/python3 | 09:07 |
| kevko | but this is not issue only for my patch currently writing ...it's problem globally with ansible_python_interpreter and if user installing in virtualenv or not ... | 09:08 |
| kevko | on newest distros as you know .. install pip globally it's prohibited ...lastly we fxed this .... i've built debian package and we installed globally | 09:09 |
| kevko | in my case ...openstack.cloud modules are just fauling ..because kolla-ansible with openstacksdk are installed in venv ...but ansible by default is trying to use global system interpreter | 09:10 |
| kevko | so simply said ..we need to somehow resolve this question globally ..if using venv or not .. | 09:10 |
| kevko | i don't know if i described well ..ask please ... | 09:10 |
| jovial | Was a pretty clear explanation - thanks. To me it seems like we should recommend to always use a virtualenv and do that in the CI too. | 09:12 |
| jovial | If we weren't recommending setting it at the extra vars level, we could set ansible_python_interpreter at the task level and that could use a venv with openstacksdk installed | 09:19 |
| kevko | just want to point that this is not only for openstacksdk ...but also for docker/podman currenctly using and it's crucial dependency | 09:22 |
| kevko | jovial: yeah, as you said ...we should use everywhere same location of venv and install stuff there ...then we can use our globally /var/lib/kolla/venv/python3 | 09:26 |
| kevko | jovial: another question how to deal with kolla-ansible is running on some host where we are installing packages in venv ... so docker/podman and potentionally others packages are installed in /var/lib/kolla/venv ...but in CI for example kolla-ansible code and openstacksdk etc needs to be installed also in this venv ..so interpreter can be set | 09:37 |
| kevko | globally ..and this can be broken .. | 09:37 |
| SvenKieske | I would just install everything into a venv if possible at the current state. | 09:39 |
| SvenKieske | but yes, might be problematic with libraries which are directly interfacing with host packages like docker/podman | 09:40 |
| SvenKieske | it also would have some benefits though. | 09:40 |
| SvenKieske | we might need to be able to select e.g. docker-py version in venv in order to match package docker outside venv, as it could be that different library versions support different distro package version ranges | 09:41 |
| SvenKieske | maybe not a good example in the docker case, afaik we use upstream docker repo, no? | 09:41 |
| kevko | SvenKieske: no :D | 09:42 |
| kevko | SvenKieske: it's packages from me I've uploaded to debian directly (yes) ...but debian hias 2 years cycle ...so it's hosted on zigo's server I asked him for host | 09:43 |
| SvenKieske | ok, i always have to look this stuff up, I don't like memorizing stuff that's in git :D | 09:43 |
| SvenKieske | kevko: I know :) | 09:43 |
| kevko | SvenKieske: yeah, another option is to open new terminal window and prepare openstacksdk apt package and again ask zigo to host for me :D | 09:44 |
| SvenKieske | ah wait, ok, was not aware we are using docker from zigos server? nice | 09:44 |
| kevko | SvenKieske: i don't know if docker ...but 100 % sure that podman yes | 09:44 |
| SvenKieske | I have also some prior experience building packages etc. from my time with gentoo, debian and ubuntu. but I would say we rather should not build more debs I guess? :D | 09:44 |
| SvenKieske | I mean that's why we stopped providing binary images, wasn't it? (I was not really around at that time, only as a user) | 09:45 |
| kevko | SvenKieske: https://github.com/openstack/ansible-collection-kolla/blob/cf7b405c4aa788bb307020b739c93094ae0050a9/roles/podman_sdk/defaults/main.yml#L37 | 09:46 |
| SvenKieske | but only for non venv deployments it seems :) | 09:46 |
| kevko | SvenKieske: yeah ! :D that's the point ...new debian and new ubuntu forbid to install without env :D | 09:47 |
| SvenKieske | I would just like to have everything be the same, where possible. greatly reduces variability, maintenance burden and test matrix | 09:47 |
| kevko | SvenKieske: okay, you can say, fine ..we will finally use virtualenv everywhere | 09:47 |
| SvenKieske | at least I would like to. I'm pretty sure there are also nice problems with venv everywhere :D | 09:48 |
| SvenKieske | but maybe we can solve them, don't know just yet :D | 09:48 |
| kevko | SvenKieske: but then you have to know that if you install minimal requirements for kolla run on hosts /var/lib/kolla/venv ...you also need to install dependencies for kolla-ansible itself (and openstacksdk, and clients for CI ..etc) .... then you have exactly one venv ...that venv you previously installed ...why ? because you have to need | 09:49 |
| kevko | ansible_python_interpreter to be the same | 09:49 |
| kevko | got the point ? | 09:49 |
| kevko | and there can be potentional conflicts ...and also ...normally you don't need to mix those rquirements for kolla-ansible itself and remote hosts reuqirements | 09:50 |
| kevko | another solution is finally have kolla-ansible container and use same path for virtualenv ... | 09:50 |
| SvenKieske | yes, I would suggest we have/create a playbook, or rather bootstrap script for that, don't you think? so users don't get insane from dozens of installation instructions, just good old "curl $foo | sudo " :D | 09:50 |
| kevko | that's not about playbook | 09:51 |
| SvenKieske | actually I like the container idea more :) | 09:51 |
| kevko | you just need to have everything in one venv | 09:51 |
| kevko | for example ...i don't have a problem ..because i am using container | 09:51 |
| SvenKieske | just do "docker pull kolla-ansible-bootstrap" or something and you are good to go | 09:51 |
| SvenKieske | but do we use a venv inside the container, or not? ;) | 09:51 |
| kevko | so my python interpreter is set to /usr/bin/python3 | 09:51 |
| kevko | (because i have kolla-ansible installed globally in container ..with dependencies also ) | 09:52 |
| kevko | and also i have gloablly installed everything on host | 09:52 |
| kevko | so on every place i have /usr/bin/python3 | 09:52 |
| kevko | i found this problem just on CI :D | 09:52 |
| SvenKieske | kevko: we use https://github.com/osism/container-image-kolla-ansible | 09:53 |
| kevko | SvenKieske: i have my own | 09:53 |
| kevko | hmmmm, probably i can install openstacksdk in kolla-toolbox as it's isolated and call there | 09:54 |
| kevko | let me try | 09:54 |
| kevko | thanks god it's there :D | 09:55 |
| SvenKieske | I fuzzily remember the sdk already being installed there? But maybe I mix up the container with one of our downstreams | 09:56 |
| kevko | i hope you are right | 09:57 |
| kevko | hate if i need to resolve bunch of problems because something simple :D | 09:57 |
| SvenKieske | https://github.com/openstack/kolla/blob/master/docker/kolla-toolbox/Dockerfile.j2#L73 | 09:57 |
| SvenKieske | it's there, from pip | 09:57 |
| kevko | cool | 09:58 |
| kevko | (kolla-toolbox)[root@controller0 /]# pip3 freeze | grep -i openstacksdk | 09:58 |
| kevko | openstacksdk==3.1.0 | 09:58 |
| kevko | but then i need to also install openstack.cloud cloud collections :( | 09:58 |
| kevko | trap next to a trap | 09:59 |
| SvenKieske | check requirements.yaml it's also there | 10:00 |
| SvenKieske | https://github.com/openstack/kolla/blob/master/docker/kolla-toolbox/requirements.yml#L11 | 10:00 |
| kevko | another trap :D | 10:05 |
| kevko | <3 | 10:05 |
| kevko | :D | 10:05 |
| kevko | SvenKieske: but coooool, working | 10:22 |
| SvenKieske | so, we shove everything into toolbox now? :D | 10:24 |
| kevko | well, kolla-toolbox was designed for this :D | 10:35 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: Add rally role https://review.opendev.org/c/openstack/kolla-ansible/+/922900 | 10:42 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: Try to test via rally https://review.opendev.org/c/openstack/kolla-ansible/+/922901 | 10:42 |
| kevko | podman broken if user override healthcheck_enabled: "no" | 10:43 |
| kevko | https://review.opendev.org/c/openstack/kolla-ansible/+/923032 | 10:43 |
| opendevreview | Matúš Jenča proposed openstack/kolla-ansible master: Add support for RabbitMQ internode tls https://review.opendev.org/c/openstack/kolla-ansible/+/921381 | 10:51 |
| jovial | :q | 11:46 |
| opendevreview | Michal Arbet proposed openstack/kolla master: Trivial fix letsencrypt base image https://review.opendev.org/c/openstack/kolla/+/923045 | 12:19 |
| opendevreview | Merged openstack/kolla-ansible master: Patch service-cert-copy role to be used w/o HAProxy https://review.opendev.org/c/openstack/kolla-ansible/+/915901 | 12:26 |
| bbezak | kevko, frickler: got a sec to look into https://review.opendev.org/c/openstack/kolla/+/913184 ? | 12:44 |
| dougszu | A small one that's been blocked on minor changes for a while: https://review.opendev.org/c/openstack/kolla/+/913184 | 12:44 |
| kevko | bbezak: yep | 12:46 |
| bbezak | thx kevko | 12:47 |
| kevko | bbezak: commented | 12:48 |
| kevko | bbezak: btw, >> can u ? >> https://review.opendev.org/c/openstack/kolla/+/923045 << typo only | 12:49 |
| bbezak | syre | 12:49 |
| bbezak | sure :) | 12:49 |
| bbezak | another typo | 12:49 |
| bbezak | and then cherry pick to 2023.2. probably worthy a launchpad bug kevko if I'd be super picky :) | 12:53 |
| kevko | bbezak: well, it's bug ..yeah ..but not affecting users ... | 12:54 |
| kevko | bbezak: so i would prefer to not create launchpad | 12:54 |
| kevko | i just think it can be quickly cherry-picked to stable ... | 12:54 |
| bbezak | ack | 12:54 |
| kevko | bbezak: check the cooment about those plugins install ..i think reasonable | 12:55 |
| bbezak | k | 12:56 |
| SvenKieske | that's a weird "typo" almost all characters are different, how do you type "cinder" instead of "letsencrypt"? ;) but just another case in point that nobody will spot all errors in +600lines long changesets | 13:28 |
| SvenKieske | so we really should do smaller changes where possible, split stuff up. :) | 13:29 |
| SvenKieske | mhm, should we backport https://review.opendev.org/c/openstack/kolla-ansible/+/915901 ? I did vote +1 on backport and nobody objected, but the change actually doesn't have any associated bug. | 13:30 |
| SvenKieske | if we agree on backporting I can create a bugreport for that | 13:30 |
| kevko | SvenKieske: I replied to your comment ... how you can split the patch which is depends-on in kolla-ansible ? | 13:31 |
| SvenKieske | kevko: for example the addition of rsync package can be split out | 13:32 |
| SvenKieske | I guess the complete ssh stuff could be split as well | 13:33 |
| kevko | SvenKieske: what is this ? i think it's bugfix ... as it is in master ..and also in 2024.1 and 2023.2 https://review.opendev.org/c/openstack/kolla/+/920279 <<< | 13:33 |
| kevko | please explain to me | 13:33 |
| kevko | SvenKieske: no it can't ... how ? | 13:33 |
| kevko | SvenKieske: if you remove haproxy-ssh ..job for letsencrypt will fail ... | 13:34 |
| kevko | SvenKieske: i really didn't get it ...another example ? | 13:34 |
| SvenKieske | it's not about removing but adding. if $a implies $b that doesn't necessarily mean $b implies $a. so if you say "I can't remove foo, else bar will break" that doesn't mean you can't also "introduce foo without introducing bar" | 13:35 |
| SvenKieske | https://review.opendev.org/c/openstack/kolla/+/920279 was a bit special, because gating and publish jobs where broken | 13:36 |
| SvenKieske | and ftr I strongly dislike having "special" patches which have different rules. I wouldn't have them if I could rule everything. But I can look at it from other people's point of view and understand that unbrekaing gate has higher prio for them then following some rule. | 13:37 |
| kevko | SvenKieske: so you are saying that this was special because it didn't affect users ... | 13:37 |
| kevko | SvenKieske: this is same story ... | 13:37 |
| SvenKieske | kevko: I'm unsure about the impact on users wrt your LE patch, that's why I didn't -1 it :) | 13:38 |
| SvenKieske | so I tend to believe you that there is no impact, thus no -1. I still think a reno is good everytime. | 13:38 |
| SvenKieske | and I also understand being frustrated when rules are not applied to every patch always the same, no matter the author or content. | 13:39 |
| SvenKieske | but I think I wouldn't attribute to malice which can be equally explained by us all just being poor imperfect humans who make mistakes all the time with applying rules. that's why we invented computers I think ;9 | 13:40 |
| kevko | SvenKieske: A little lightheartedness - do you know what rules are for? They are there to be broken :) | 13:41 |
| SvenKieske | that's why I personally would just have a bot scan commit messages and if it says "fixes" and there is no reno -> no merge possible W-1. so it's enforced everytime and nobody can complain, well reality would be everyone would complain ;) | 13:41 |
| SvenKieske | yes, we always need escape hatches, because rules are, well for general cases, there will always be the super special case where some rule doesn't apply. "checks and balances" like the us americans like to say. | 13:43 |
| kevko | SvenKieske: I just want to say that you need a certain amount of sensitivity when evaluating; release notes are primarily for users. That means, from my perspective, they shouldn't be filled with nonsense, but with real bug fixes. | 13:43 |
| SvenKieske | still I would like to see most rules be enforced by CI bots, humans are very poor at it, and it's a lot of work that can be automated. you can always have a core reviewer overrule a bot in special cases imho. | 13:43 |
| SvenKieske | kevko: absolutely agree! as I said: I'm really unsure about the impact of that LE change. why didn't anybody notice? what is it good for if it didn't break anybody? :D | 13:44 |
| SvenKieske | I find the more interesting part how we introduce this bug, by these huge patches, where nobody reads every line with 100% scrutiny and bugs slip through. I'm more interested in preventing bugs in the first place. | 13:45 |
| kevko | SvenKieske: firstly, it's base image ..so if anyone found that mistake ..he just didn't used base_image variable for additional install ..but used another two variables (in case he wanted to install something additional into images ) | 13:46 |
| SvenKieske | kevko: any opinion on backporting the service-cert-copy stuff? https://review.opendev.org/c/openstack/kolla-ansible/+/915901 I can handle bug creation and backports. just don't want to do it, if a backport is deemed to high risk | 13:47 |
| kevko | SvenKieske: and I am not sure ... but i think this is only about naming convention ...if you will add 'foo' as name ..it will be working in same way | 13:47 |
| kevko | SvenKieske: because there is | customizable(packages) | 13:47 |
| SvenKieske | yeah, I guess I need to have a look myself, my question is, what is in "letsencrypt_base_packages" that is so useless that nobody noticed that it wasn't installed? | 13:50 |
| kevko | As you know, I'm lenient about this :) ^^ | 13:50 |
| SvenKieske | mhm, weird, that seems to be completely unused?Is my tree out of date? :D | 13:51 |
| kevko | SvenKieske: yeah :D | 13:52 |
| opendevreview | Michal Arbet proposed openstack/kolla master: Trivial fix letsencrypt base image https://review.opendev.org/c/openstack/kolla/+/923045 | 13:52 |
| kevko | SvenKieske: we were blind again :D | 13:52 |
| kevko | SvenKieske: That means even extremely small patches don't help :D :D :D | 13:53 |
| SvenKieske | well, we did spot it in this one liner, because that made me actually grep for both variables in the code base. you don't do that with every variable in 500 line changes, do you? | 13:54 |
| SvenKieske | so I would say it really helped :) | 13:55 |
| SvenKieske | take my +1 | 13:55 |
| kevko | SvenKieske: i am checkouting the change and yeah ..i am doing it , grepping for unused vars in kolla-ansible ..and some other stuff | 13:56 |
| kevko | and 600 lines it's not too much | 13:56 |
| SvenKieske | very honorable, I try to do that as well, but it seems 4-6 people missed that, and that's even easy to spot, right? I mean why would "cinder" crop up in a LE only dockerfile? | 13:57 |
| SvenKieske | that could've already been spotted even without any grepping the codebase. just by looking at it. I'm just mad at myself I didn't see it :D | 13:57 |
| kevko | because i written that patch ...and i am just lazy to write them all ..so i am doing cp xyz/ new-image/ | 13:58 |
| kevko | and then i am rewriting | 13:58 |
| kevko | something as reno new templates/bugfix ...or how it is that command :D | 13:58 |
| SvenKieske | that reinforces something I learned some years ago: there are _no_ trivial changes, like at all. That day I learned that by reviewing a one line change from the main author of a codebase with over 20 years of experience. he had 3 bugs on this one line. | 13:59 |
| SvenKieske | mankind is not built for computers, we are too error prone :) | 13:59 |
| kevko | SvenKieske: i am open to git commit --amend and git review every time even if need to fix some comment ...but that means for example loose of one +2 ..which will result in month delay actually (sometimes ) | 14:00 |
| SvenKieske | kevko: sure :D I know that pain | 14:09 |
| SvenKieske | mhm I can't check if the base package var for cinder is empty when only building le base container, need to repair my kolla venv first..I guess I changed something in the requirements file.. | 14:20 |
| kevko | i have more important work :D | 14:26 |
| opendevreview | Matúš Jenča proposed openstack/kolla-ansible master: Add documentation for caching. https://review.opendev.org/c/openstack/kolla-ansible/+/918285 | 15:08 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: Fix prechecks for interfaces with dashes https://review.opendev.org/c/openstack/kolla-ansible/+/922478 | 16:45 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: CI: test interface names with dashes https://review.opendev.org/c/openstack/kolla-ansible/+/794545 | 16:46 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: CI: test interface names with dashes https://review.opendev.org/c/openstack/kolla-ansible/+/794545 | 16:46 |
| opendevreview | Merged openstack/kolla master: Trivial fix letsencrypt base image https://review.opendev.org/c/openstack/kolla/+/923045 | 17:24 |
| opendevreview | Merged openstack/kolla master: Change logic for plugins in grafana image build https://review.opendev.org/c/openstack/kolla/+/913184 | 18:23 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: Fix prechecks for interfaces with dashes https://review.opendev.org/c/openstack/kolla-ansible/+/922478 | 18:38 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: CI: test interface names with dashes https://review.opendev.org/c/openstack/kolla-ansible/+/794545 | 18:38 |
| Balsa | Running Kolla 18 (2024.1) I want to use ACME/LetsEncrypt certs for SSL, is there a guide floating around to do this | 22:01 |
| Balsa | I looks like its built in, but I'm getting deployment errors about invalidated certs when haproxy comes up | 22:02 |
| Balsa | the "Generating TLS certificates with Let’s Encrypt" part of the docs doesn't seem complete, its missing what dependencies in the configs it needs it seems | 22:12 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!