| opendevreview | Merged openstack/python-octaviaclient master: Update master for stable/2025.1 https://review.opendev.org/c/openstack/python-octaviaclient/+/943737 | 09:08 |
|---|---|---|
| opendevreview | Merged openstack/python-octaviaclient stable/2025.1: Update .gitreview for stable/2025.1 https://review.opendev.org/c/openstack/python-octaviaclient/+/943735 | 09:08 |
| opendevreview | Merged openstack/python-octaviaclient stable/2025.1: Update TOX_CONSTRAINTS_FILE for stable/2025.1 https://review.opendev.org/c/openstack/python-octaviaclient/+/943736 | 09:08 |
| opendevreview | Merged openstack/octavia-lib master: Update master for stable/2025.1 https://review.opendev.org/c/openstack/octavia-lib/+/943732 | 09:32 |
| opendevreview | Merged openstack/octavia-lib stable/2025.1: Update .gitreview for stable/2025.1 https://review.opendev.org/c/openstack/octavia-lib/+/943729 | 09:32 |
| opendevreview | Merged openstack/octavia-lib stable/2025.1: Update TOX_CONSTRAINTS_FILE for stable/2025.1 https://review.opendev.org/c/openstack/octavia-lib/+/943730 | 09:32 |
| opendevreview | Merged openstack/octavia master: Drop VIRTUAL_ENV https://review.opendev.org/c/openstack/octavia/+/941188 | 12:58 |
| kevko | Hi folks, i want to just ask .. we found a bug in kolla config or better said combination of config and octavia redis sentinel implementation - Version of octavia - 2023.1 ... so I thought it will be enough to backport https://review.opendev.org/c/openstack/octavia/+/907669 and https://review.opendev.org/c/openstack/octavia/+/907676 and | 13:20 |
| kevko | https://review.opendev.org/c/openstack/octavia/+/912355 to octavia ... i've also installed taskflow taskflow==5.9.1 and redis==5.2.1 (as i have this dependencies on another stack where it's working - 2024.2 ) ...but still nothing :( ...what i am missing ? please can u help me ? here LOG and config | 13:20 |
| kevko | https://paste.openstack.org/show/bCcp7cDuspEmEIy5m69k/ I am asking also because i remember I was discussing this with tkajinam in a past with connection to the kolla - thank you very much | 13:20 |
| kevko | + patch for kolla-ansible working in master - https://review.opendev.org/c/openstack/kolla-ansible/+/942799 << Original bug i see in 2023.1 with original config is that if master dissapear.. jobboard is not working :/ | 13:21 |
| kevko | do i need to backport something else ? i mean some patch ? | 13:21 |
| tkajinam | kevko, "AUTH <password> called without any password configured for the default user." | 13:50 |
| tkajinam | this is likely the cause and you may have to check any difference in redis version or its configuration | 13:51 |
| tkajinam | if you do not enable redis acl then you may not need the username parameter set, iirc | 13:51 |
| tkajinam | I don't see any options which may be relevant to ACL https://dd43e0f9601da5e2e650-51b18fcc89837fbadd0245724df9c686.ssl.cf1.rackcdn.com/942799/5/check/kolla-ansible-ubuntu-octavia/5415d9d/primary/logs/kolla_configs/redis-sentinel/redis.conf | 13:54 |
| tkajinam | and I guess you are using the legacy auth mechanism which does not accept user-based auth | 13:54 |
| tkajinam | anyway the error does not really look relevant to octavia or taskflow iiuc | 13:54 |
| kevko | tkajinam: check the kolla-ansible patch ... it is working in upstream | 14:05 |
| kevko | tkajinam: and upstream didn't change the redis images ..or configuration ... | 14:05 |
| kevko | tkajinam: so it needs to be something in octavia or taskflow | 14:06 |
| kevko | tkajinam: that's the reason why i am asking ..what i am missing | 14:07 |
| kevko | tkajinam: what about sentinel auth-pass kolla prJdpxQpU4uYvY3h3kIWLt9u7fOSlANY74v49yUM | 14:08 |
| kevko | (as there is no user ..it should be default ) | 14:08 |
| tkajinam | is that exactly same with the working configuration in master ? | 14:08 |
| kevko | yeah .. | 14:09 |
| tkajinam | hm | 14:14 |
| kevko | tkajinam: we are using 2023.1 right ? and we found that if master is down ..jobboard is not working ...so what i've done ? spin the master stack and said to myself ..okay ..i know where is the problem and fix the kolla-ansible config for octavia.conf jobboard section and said heureka ..it's working | 14:15 |
| kevko | tkajinam: ^^ (master patch for octavia.conf in kolla-ansible project ) | 14:16 |
| kevko | tkajinam: then I backported to kolla-ansible 2023.1 and expected that it will work ..but it did't | 14:16 |
| kevko | tkajinam: so i found that there are changes for user auth and ssl and sentinel_fallbacks ... | 14:16 |
| kevko | tkajinam: so i've taken that patches and patched 2023.1 octavia branch and build kolla-images + of course the same redis-py and taskflow pip packages ... and still dont work ..but i expected it will | 14:17 |
| kevko | tkajinam: so - that's the reason why i am asking ..and what i am missing ... | 14:17 |
| kevko | tkajinam: redis.conf or redis-sentinel.conf was not changed during several version in kolla-ansible orchestration if i am correct | 14:18 |
| tkajinam | hmm wait do you enable auth in sentinel in master, right ? | 14:22 |
| tkajinam | https://review.opendev.org/c/openstack/taskflow/+/912346 | 14:23 |
| tkajinam | I think we have no mechanism to enable auth for sentinel in taskflow now | 14:23 |
| tkajinam | I don't remember details but I was asked to not require auth for sentinel by someone from your group | 14:24 |
| tkajinam | and had to revert the change, IIRC | 14:24 |
| tkajinam | https://bugs.launchpad.net/python-tooz/+bug/2056656 | 14:26 |
| kevko | tkajinam: well, as i said ..check redis-sentinel config in master kolla job .. | 14:28 |
| kevko | tkajinam: it's there and working ..how it is possible ..no errors in octavia-worker | 14:28 |
| kevko | tkajinam: it was me .. but it was something sligtly different maybe ? | 14:28 |
| tkajinam | Some deployment tools such as kolla already rely on the previous | 14:29 |
| tkajinam | behavior which requires authentication for only redis. | 14:29 |
| tkajinam | I remember I checked actual config generated in kolla at that time which had auth enabled for redis but not for redis-sentinel | 14:29 |
| tkajinam | backport indicates it was 2024.1 release | 14:29 |
| tkajinam | so I expect the same for 2023.1 unless you "fix" it in older branches later | 14:30 |
| kevko | tkajinam: this is master -> https://dd43e0f9601da5e2e650-51b18fcc89837fbadd0245724df9c686.ssl.cf1.rackcdn.com/942799/5/check/kolla-ansible-ubuntu-octavia/5415d9d/primary/logs/kolla_configs/index.html | 14:32 |
| kevko | tkajinam: redis-sentinel https://dd43e0f9601da5e2e650-51b18fcc89837fbadd0245724df9c686.ssl.cf1.rackcdn.com/942799/5/check/kolla-ansible-ubuntu-octavia/5415d9d/primary/logs/kolla_configs/redis-sentinel/redis.conf | 14:32 |
| kevko | tkajinam: redis https://dd43e0f9601da5e2e650-51b18fcc89837fbadd0245724df9c686.ssl.cf1.rackcdn.com/942799/5/check/kolla-ansible-ubuntu-octavia/5415d9d/primary/logs/kolla_configs/redis/redis.conf | 14:32 |
| kevko | tkajinam: config octavia-worker https://dd43e0f9601da5e2e650-51b18fcc89837fbadd0245724df9c686.ssl.cf1.rackcdn.com/942799/5/check/kolla-ansible-ubuntu-octavia/5415d9d/primary/logs/kolla_configs/octavia-worker/octavia.conf | 14:34 |
| tkajinam | hmm so redis.conf for sentinel has no requirepass | 14:36 |
| tkajinam | which indicates no auth for sentinel | 14:36 |
| kevko | tkajinam: so what is this ? entinel auth-pass kolla prJdpxQpU4uYvY3h3kIWLt9u7fOSlANY74v49yUM | 14:41 |
| kevko | sentinel auth-pass kolla prJdpxQpU4uYvY3h3kIWLt9u7fOSlANY74v49yUM | 14:41 |
| kevko | i think this is auth to the sentinel no ? | 14:41 |
| tkajinam | that the password used in communication between sentinel and redis | 14:41 |
| tkajinam | which is different from requireauth defining the password in redis sentinel itself | 14:41 |
| kevko | tkajinam: hmm | 14:42 |
| kevko | tkajinam: it's same | 14:42 |
| kevko | tkajinam: i have a little gulash in it :( ... and still don't understand why it is working in master and not in my stack | 14:43 |
| kevko | + https://github.com/openstack/taskflow/commit/3fbd05078f84fc5b8190201fc6eeb7d005bf4988 | 14:45 |
| tkajinam | kevko, are you using the same dogpile.cache ? | 14:45 |
| tkajinam | redis_sentinel driver in oslo.cache uses consistent auth for redis and sentinel (because that's a new feature at that moment we didn't care incompatibility with kolla at that time) | 14:46 |
| tkajinam | and I see you use redis_sentinel cache driver in master now for example in nova.conf, so it indicates you fixed disabled auth in sentinel somehow | 14:47 |
| tkajinam | I could not really find a clear clue about it from config but I think you first dig into the history around introduction of redis_sentinel oslo.cache backend in kolla | 14:47 |
| kevko | tkajinam: this shouldn't be anything about oslo.cache no ? | 14:48 |
| tkajinam | The problem is not relevant to oslo.cache, but the current setting of oslo.cache I see in master is that you expect auth for sentinel | 14:49 |
| tkajinam | which it didn't in 2024.1 | 14:49 |
| tkajinam | which *you* didn't, I mean | 14:49 |
| kevko | tkajinam: lemme check | 14:49 |
| kevko | tkajinam: are you talking about some redis_sentinel driver oslo.cache not merged gerrit review ? | 14:51 |
| kevko | because in nova merged kolla-ansible templates we still using meemcached | 14:51 |
| tkajinam | hmm wait I might be looking at a wrong file from too many tabs | 14:52 |
| kevko | :D | 14:52 |
| tkajinam | hmm seems I do | 14:53 |
| kevko | tkajinam: so i really don't know what's wrong there :/ | 15:06 |
| kevko | tkajinam: i was wondering that i will just upgrade octavia to newer images ..but on the other side..i want to fix it in currect version of octavia | 15:06 |
| tkajinam | I may check how https://review.opendev.org/c/openstack/kolla-ansible/+/943950/ goes into CI to see if auth is actually enabled in sentinel or it's ignored by octavia | 15:07 |
| kevko | tkajinam: I'll spare you the wait :) I think you will get this | 15:11 |
| kevko | tkajinam: https://paste.openstack.org/show/bQ3ika8Rf14rUl7iZLKx/ | 15:11 |
| kevko | tkajinam: this is my local cluster deployed | 15:11 |
| kevko | ^^ | 15:11 |
| tkajinam | kevko, so you see now sentinel has its auth enabled, right ? | 15:12 |
| tkajinam | and can you check the same in that existing 2023.1 cluster ? | 15:12 |
| tkajinam | redis-cli -h <host> -p 26379 may be the thing you can try/check | 15:17 |
| kevko | tkajinam: so change the config as you proposed in DNM patch ? in my testing 2023.1 with octavia patched and taskflow redis ? | 15:17 |
| tkajinam | kevko, I'm saying if you can try connecting to sentinel by a tool (not by octavia or actual service code) to see if auth is enabled | 15:18 |
| tkajinam | that's why I'm suggesting redis-cli | 15:18 |
| tkajinam | to make sure redis-sentinel behaves in the same way | 15:19 |
| kevko | tkajinam: ah ok | 15:19 |
| kevko | tkajinam: lemme check | 15:19 |
| kevko | tkajinam: (octavia-worker)[root@controller0 /]# redis-cli -h 192.168.205.11 -p 26379 SENTINEL get-master-addr-by-name kolla | 15:22 |
| kevko | 1) "192.168.205.10" | 15:22 |
| kevko | 2) "6379" | 15:22 |
| kevko | ^^ master | 15:22 |
| kevko | (redis-sentinel)[root@controller0 /]# redis-cli -h 192.168.205.11 -p 26379 SENTINEL get-master-addr-by-name kolla | 15:23 |
| kevko | 1) "192.168.205.10" | 15:23 |
| kevko | 2) "6379" | 15:23 |
| kevko | 2023.1 | 15:23 |
| tkajinam | so... no auth | 15:24 |
| kevko | hmm | 15:24 |
| kevko | okay, so why it is working with the config in master ? | 15:26 |
| kevko | and the 2nd ...so you are saying ..you don't know why it's working in master ..but in my 2023.1 it should be enough to remove sentinel auth ? | 15:31 |
| tkajinam | I have no idea about #1 but I suspect so for #2 | 15:33 |
| kevko | tkajinam: 2025-03-10 15:50:48.219 20 ERROR taskflow.jobs.backends.impl_redis [-] <redis.client.Redis(<redis.sentinel.SentinelConnectionPool(service=kolla(master))>)> | 15:51 |
| kevko | tkajinam: hmm, w8 a minute | 15:59 |
| tkajinam | kevko, so octavia jobs are passing without sentinel username/password options https://zuul.opendev.org/t/openstack/build/09105b4e69094938b4870beb71d9eecf | 16:18 |
| kevko | tkajinam: hmm | 16:31 |
| kevko | tkajinam: i will check every result and try locally and will let you know ... thank you very much for the help | 16:32 |
| opendevreview | Sergey Kraynev proposed openstack/octavia master: Ignore load_balancer graph in task results to avoid break write data in jobboard DB https://review.opendev.org/c/openstack/octavia/+/943985 | 18:38 |
| kevko | tkajinam: okay, thanks Takashi, working on both stacks, removed sentinel auth from the conf | 20:26 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!