*** zzzeek has quit IRC | 00:15 | |
*** zzzeek has joined #openstack-kolla | 00:15 | |
*** maharg101 has joined #openstack-kolla | 00:16 | |
*** maharg101 has quit IRC | 00:20 | |
*** zzzeek has quit IRC | 00:23 | |
*** zzzeek has joined #openstack-kolla | 00:26 | |
*** LinPeiWen has joined #openstack-kolla | 00:42 | |
*** brinzhang has joined #openstack-kolla | 00:43 | |
*** jamesbenson has joined #openstack-kolla | 00:52 | |
*** jamesbenson has quit IRC | 00:57 | |
*** rohit02 has joined #openstack-kolla | 02:27 | |
*** ricolin has joined #openstack-kolla | 02:52 | |
*** ricolin has quit IRC | 03:38 | |
*** zzzeek has quit IRC | 03:47 | |
*** zzzeek has joined #openstack-kolla | 03:49 | |
*** zzzeek has quit IRC | 03:53 | |
*** zzzeek has joined #openstack-kolla | 03:55 | |
*** zzzeek has quit IRC | 03:59 | |
*** zzzeek has joined #openstack-kolla | 04:01 | |
*** ricolin has joined #openstack-kolla | 04:02 | |
*** zzzeek has quit IRC | 04:14 | |
*** maharg101 has joined #openstack-kolla | 04:17 | |
*** zzzeek has joined #openstack-kolla | 04:18 | |
*** maharg101 has quit IRC | 04:22 | |
*** vishalmanchanda has joined #openstack-kolla | 04:24 | |
*** zzzeek has quit IRC | 04:57 | |
*** zzzeek has joined #openstack-kolla | 04:58 | |
*** zzzeek has quit IRC | 05:03 | |
*** zzzeek has joined #openstack-kolla | 05:05 | |
*** cah_link has joined #openstack-kolla | 05:11 | |
*** cah_link has quit IRC | 05:20 | |
*** maharg101 has joined #openstack-kolla | 06:18 | |
*** maharg101 has quit IRC | 06:23 | |
*** cah_link has joined #openstack-kolla | 06:41 | |
*** benj_ has quit IRC | 06:48 | |
*** benj_ has joined #openstack-kolla | 06:52 | |
osmanlicilegi | morning | 06:53 |
---|---|---|
*** zijlboot has joined #openstack-kolla | 07:30 | |
*** shyamb has joined #openstack-kolla | 07:33 | |
*** shyamb has quit IRC | 07:35 | |
*** e0ne has joined #openstack-kolla | 07:38 | |
*** e0ne has quit IRC | 07:39 | |
*** LinPeiWen has quit IRC | 07:50 | |
*** bengates has joined #openstack-kolla | 07:57 | |
*** bengates has quit IRC | 07:57 | |
*** amoralej|off is now known as amoralej | 08:06 | |
*** gfidente has joined #openstack-kolla | 08:06 | |
*** bengates has joined #openstack-kolla | 08:09 | |
*** andrewbonney has joined #openstack-kolla | 08:10 | |
*** bengates has quit IRC | 08:11 | |
*** bengates has joined #openstack-kolla | 08:11 | |
*** bengates has quit IRC | 08:11 | |
*** bengates has joined #openstack-kolla | 08:24 | |
*** LinPeiWen has joined #openstack-kolla | 08:24 | |
*** maharg101 has joined #openstack-kolla | 08:25 | |
*** bengates has quit IRC | 08:25 | |
*** bengates has joined #openstack-kolla | 08:26 | |
*** rpittau|afk is now known as rpittau | 08:34 | |
hrw | morning | 08:41 |
*** k_mouza has joined #openstack-kolla | 08:58 | |
*** rohit02 has quit IRC | 08:59 | |
*** rohit02 has joined #openstack-kolla | 09:04 | |
*** e0ne has joined #openstack-kolla | 09:05 | |
*** k_mouza has quit IRC | 09:05 | |
*** k_mouza has joined #openstack-kolla | 09:08 | |
*** e0ne has quit IRC | 09:10 | |
*** jbadiapa has joined #openstack-kolla | 09:36 | |
openstackgerrit | Mark Goddard proposed openstack/kolla-ansible stable/victoria: Replace db-sock with db-nb-sock and db-sb-sock https://review.opendev.org/c/openstack/kolla-ansible/+/776882 | 09:44 |
openstackgerrit | Mark Goddard proposed openstack/kolla-ansible stable/ussuri: Replace db-sock with db-nb-sock and db-sb-sock https://review.opendev.org/c/openstack/kolla-ansible/+/776883 | 09:44 |
openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: CI: Move from ceph-ansible to cephadm https://review.opendev.org/c/openstack/kolla-ansible/+/771418 | 09:54 |
mgoddard | morning | 09:57 |
*** zzzeek has quit IRC | 09:59 | |
*** zzzeek has joined #openstack-kolla | 10:01 | |
mnasiadka | morning | 10:06 |
openstackgerrit | Mark Goddard proposed openstack/kolla-ansible master: Add support for Ceph RadosGW integration https://review.opendev.org/c/openstack/kolla-ansible/+/776662 | 10:07 |
*** e0ne has joined #openstack-kolla | 10:11 | |
*** bengates_ has joined #openstack-kolla | 10:11 | |
*** bengates has quit IRC | 10:13 | |
openstackgerrit | Mark Goddard proposed openstack/kolla-ansible master: WIP: CI: monasca job follow up https://review.opendev.org/c/openstack/kolla-ansible/+/773881 | 10:14 |
*** skramaja has joined #openstack-kolla | 10:18 | |
openstackgerrit | Mark Goddard proposed openstack/kolla-ansible master: DNM: Test chrony apparmor fix https://review.opendev.org/c/openstack/kolla-ansible/+/776411 | 10:21 |
*** k_mouza has quit IRC | 10:21 | |
*** rohit02 has quit IRC | 10:30 | |
*** kevko has joined #openstack-kolla | 10:35 | |
*** kevko_ has joined #openstack-kolla | 10:37 | |
*** k_mouza has joined #openstack-kolla | 10:38 | |
*** kevko has quit IRC | 10:40 | |
*** rohit02 has joined #openstack-kolla | 10:42 | |
*** kevko_ has quit IRC | 10:53 | |
*** kevko has joined #openstack-kolla | 10:54 | |
openstackgerrit | Merged openstack/kolla-ansible master: CI: Initial jinja2 templates syntax checks https://review.opendev.org/c/openstack/kolla-ansible/+/728894 | 10:57 |
*** k_mouza has quit IRC | 11:01 | |
*** k_mouza has joined #openstack-kolla | 11:06 | |
*** zijlboot has quit IRC | 11:11 | |
*** zijlboot has joined #openstack-kolla | 11:12 | |
openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: CI: Move from ceph-ansible to cephadm https://review.opendev.org/c/openstack/kolla-ansible/+/771418 | 11:13 |
*** jan00 has joined #openstack-kolla | 11:16 | |
*** k_mouza has quit IRC | 11:17 | |
*** jan00 has quit IRC | 11:18 | |
openstackgerrit | Merged openstack/kolla-ansible stable/victoria: ovn: Fix disabling of gateway chassis https://review.opendev.org/c/openstack/kolla-ansible/+/776436 | 11:29 |
openstackgerrit | Merged openstack/kolla-ansible stable/ussuri: ovn: Fix disabling of gateway chassis https://review.opendev.org/c/openstack/kolla-ansible/+/776437 | 11:29 |
*** k_mouza has joined #openstack-kolla | 11:32 | |
yoctozepto | mgoddard: https://review.opendev.org/c/openstack/kolla-ansible/+/773246 - quick review perhaps? | 12:00 |
openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: CI: Move from ceph-ansible to cephadm https://review.opendev.org/c/openstack/kolla-ansible/+/771418 | 12:04 |
openstackgerrit | Merged openstack/kayobe stable/victoria: Test building seed deployment images in the seed job https://review.opendev.org/c/openstack/kayobe/+/775327 | 12:06 |
*** jamesbenson has joined #openstack-kolla | 12:07 | |
kevko | hi, guys .. on ussuri I am facing some issue (don't know if it is OSstack or my env or what ..) when I restart keepalived and IP is switched to secondary keepalived host ...keystone is unresponsive for 1 minute .. did someone see this ? | 12:10 |
openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: CI: Move from ceph-ansible to cephadm https://review.opendev.org/c/openstack/kolla-ansible/+/771418 | 12:11 |
kevko | mnasiadka: just curious why are u replacing ceph-ansible with cephadm ? | 12:15 |
kevko | mnasiadka: asking because I'm also using ceph-ansible in my in-house pipelines and I am OK with it .. | 12:16 |
*** Luzi has joined #openstack-kolla | 12:29 | |
mnasiadka | kevko: because it's dead | 12:31 |
mnasiadka | or soon to be dead | 12:31 |
mnasiadka | and it's so slooow :) | 12:33 |
openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: CI: Move from ceph-ansible to cephadm https://review.opendev.org/c/openstack/kolla-ansible/+/771418 | 12:36 |
mnasiadka | kevko: even tripleo stopped using ceph-ansible, and red hat ceph storage 5 will use cephadm | 12:37 |
mgoddard | mnasiadka: who will write openstackadm? | 12:46 |
kevko | hmm, hope that kolla-ansible will not stop ..because it looks like everything what is ansible is considered as slow :D | 12:46 |
kevko | :P | 12:46 |
kevko | i am using ceph-ansible for 3 small nodes of ceph in test env ..and it is running about 20 minutes .. kolla-ansible is running hours :D | 12:47 |
mnasiadka | mgoddard: I thought you're already writing it? :) | 12:48 |
kevko | openstackadm ? :D | 12:48 |
kevko | mgoddard, mnasiadka: did you ever see issue what I have in my test-env ? (few messages above from me ...) could you advice ? | 12:50 |
mgoddard | kevko: yeah, I think it's due to DB pools | 12:50 |
yoctozepto | mnasiadka, mgoddard: kollaadm | 12:50 |
mgoddard | you can reduce the pool size to 1 | 12:50 |
kevko | hmm, really ? let me try it | 12:51 |
yoctozepto | kevko: yeah, I confirm what mgoddard says | 12:51 |
kevko | hmm, i already have one pool :D | 12:52 |
mgoddard | connection timeouts? | 12:55 |
kevko | to db ? | 12:56 |
mgoddard | haproxy -> db | 12:56 |
kevko | well, i have everything kolla-default | 12:56 |
openstackgerrit | Merged openstack/kolla-ansible master: Use trove.conf for all trove services https://review.opendev.org/c/openstack/kolla-ansible/+/763191 | 12:57 |
*** amoralej is now known as amoralej|lunch | 13:00 | |
openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: CI: Move from ceph-ansible to cephadm https://review.opendev.org/c/openstack/kolla-ansible/+/771418 | 13:02 |
*** zzzeek has quit IRC | 13:05 | |
*** zzzeek has joined #openstack-kolla | 13:07 | |
hrw | mgoddard: we have 'openstack_tag' and 'docker_namespace' which look like same thing... | 13:16 |
hrw | # Docker image tag used by default. | 13:17 |
hrw | openstack_tag: "master-0221-1" | 13:17 |
hrw | mgoddard: we have 'openstack_tag' and 'docker_namespace' which look like same thing...# Namespace of images: | 13:17 |
hrw | ops | 13:17 |
hrw | # Namespace of images: | 13:17 |
hrw | docker_namespace: "master-0221-1" | 13:17 |
hrw | fells weird | 13:17 |
openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: CI: Move from ceph-ansible to cephadm https://review.opendev.org/c/openstack/kolla-ansible/+/771418 | 13:19 |
hrw | hm. will be fun as host os is fedora | 13:20 |
*** LinPeiWen has quit IRC | 13:25 | |
openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: CI: Move from ceph-ansible to cephadm https://review.opendev.org/c/openstack/kolla-ansible/+/771418 | 13:30 |
*** jamesbenson has quit IRC | 13:30 | |
openstackgerrit | Merged openstack/kolla-ansible stable/ussuri: Replace db-sock with db-nb-sock and db-sb-sock https://review.opendev.org/c/openstack/kolla-ansible/+/776883 | 13:32 |
openstackgerrit | Merged openstack/kolla-ansible stable/victoria: Replace db-sock with db-nb-sock and db-sb-sock https://review.opendev.org/c/openstack/kolla-ansible/+/776882 | 13:32 |
hrw | ah, sorted out | 13:33 |
hrw | ^[[A[hrw@jagular ~]$ docker run -it -u root puchatek.lan:5000/master-0222-1/debian-source-kolla-toolbox:11.1.0 bash | 13:34 |
hrw | ()[root@c20676b2293b /]# exit | 13:34 |
hrw | works | 13:34 |
hrw | raise cls(e, response=response, explanation=explanation)\\ndocker.errors.NotFound: 404 Client Error for http+docker://localhost/v1.41/images/create?tag=11.1.0&fromImage=puchatek.lan%3A5000%2Fmaster-0221-1%2Fdebian-source-kolla-toolbox: Not Found (\"manifest for puchatek.lan:5000/master-0221-1/debian-source-kolla-toolbox:11.1.0 not found: manifest unknown: manifest unknown\")\\n'"} | 13:34 |
hrw | kolla-ansible deploy does not work | 13:34 |
hrw | wth. | 13:38 |
hrw | does it require going back to using raw IP addresses again? | 13:39 |
*** dougsz has joined #openstack-kolla | 13:41 | |
dougsz | `. /etc/kolla/admin-openrc.sh 2/home/zuul/.ansible/tmp/ansible-tmp-1613996189.8381832-271-273167133369856/test-monasca.sh: line 59: /etc/kolla/admin-openrc.sh: Too many open files 3 ` | 13:41 |
dougsz | Too many files open? Has anyone hit that in CI? Seems reproducible. | 13:42 |
mnasiadka | I had that once with Octavia, but never happened again ;) | 13:42 |
mnasiadka | hrw: master-0222-1 vs master-0221-1 ? | 13:43 |
hrw | mnasiadka: 3> | 13:44 |
dougsz | hmm, thanks mnasiadka, I will retry again to get another sample | 13:44 |
dougsz | I guess it could be related to the HVs the job is landing on | 13:45 |
openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: ovn: Move openvswitch related tasks to openvswitch role https://review.opendev.org/c/openstack/kolla-ansible/+/776421 | 13:46 |
openstackgerrit | Mark Goddard proposed openstack/kolla-ansible master: Add support for Ceph RadosGW integration https://review.opendev.org/c/openstack/kolla-ansible/+/776662 | 13:48 |
*** jamesbenson has joined #openstack-kolla | 13:48 | |
*** rafaelweingartne has joined #openstack-kolla | 13:49 | |
openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: Add missing octavia-driver-agent https://review.opendev.org/c/openstack/kolla-ansible/+/761872 | 13:51 |
openstackgerrit | Mark Goddard proposed openstack/kayobe master: CI: Add Ubuntu seed job https://review.opendev.org/c/openstack/kayobe/+/775267 | 13:55 |
openstackgerrit | Mark Goddard proposed openstack/kayobe master: CI: Add Ubuntu seed job https://review.opendev.org/c/openstack/kayobe/+/775267 | 13:55 |
hrw | CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES | 13:57 |
hrw | 30d6fa75154a puchatek.lan:5000/master-0222-1/debian-source-placement-api:11.1.0 "dumb-init --single-…" 3 minutes ago Up 3 minutes (healthy) placement_api | 13:57 |
hrw | mnasiadka: that health checks look nice | 13:57 |
kevko | where is it checking ? | 13:58 |
*** amoralej|lunch is now known as amoralej | 14:00 | |
*** rafaelweingartne has left #openstack-kolla | 14:00 | |
yoctozepto | kevko: it checks whether the container's http service returns 2xx on / | 14:01 |
yoctozepto | or 3xx that leads to 2xx | 14:01 |
yoctozepto | well, you get the idea | 14:01 |
kevko | yeah | 14:01 |
kevko | some time ago I've seen that zigo was adding some healh-checks to openstack projects throuhg review | 14:02 |
kevko | some built-in checks which are not in paste-ini | 14:02 |
*** rohit02 has quit IRC | 14:04 | |
*** rohit02 has joined #openstack-kolla | 14:05 | |
hrw | kevko: https://paste.centos.org/view/98f2e166 is how looks here | 14:06 |
openstackgerrit | Mark Goddard proposed openstack/kolla-ansible master: Performance: Split out prometheus node exporter and cadvisor https://review.opendev.org/c/openstack/kolla-ansible/+/740082 | 14:07 |
*** jamesbenson has quit IRC | 14:09 | |
*** rohit02 has quit IRC | 14:11 | |
mnasiadka | yoctozepto: do you remember what happened with supporting /healthcheck backend by all API services? We could move to using that in healthchecks, if that thing finally works. | 14:22 |
hrw | 2021-02-22 15:21:59.399 7 ERROR nova.compute.manager [instance: eff222d7-b006-4a5d-8c66-f099fb804086] libvirt.libvirtError: Unable to read from '/sys/fs/cgroup/../../machine.slice/machine-qemu\x2d1\x2dinstance\x2d00000001.scope/cgroup.controllers': No such file or directory | 14:23 |
hrw | that's something new? | 14:23 |
*** zzzeek has quit IRC | 14:28 | |
yoctozepto | mnasiadka: not remember | 14:29 |
*** zzzeek has joined #openstack-kolla | 14:29 | |
openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: CI: Move from ceph-ansible to cephadm https://review.opendev.org/c/openstack/kolla-ansible/+/771418 | 14:33 |
kevko | hrw: do you have review for that feature ? i would like to port it locally to ussuri :P | 14:34 |
*** sean-k-mooney has joined #openstack-kolla | 14:36 | |
kevko | hrw: probbably this, isn't it ? https://review.opendev.org/c/openstack/kolla/+/676128 | 14:36 |
mnasiadka | kevko: that's the Kolla part, you also need the Kolla-Ansible part | 14:37 |
mnasiadka | kevko: https://review.opendev.org/q/topic:%22bp%252Fcontainer-health-check%22+(status:open%20OR%20status:merged) | 14:37 |
kevko | mnasiadka: thanks | 14:42 |
*** jamesbenson has joined #openstack-kolla | 14:45 | |
*** brinzhang_ has joined #openstack-kolla | 14:52 | |
*** brinzhang_ has quit IRC | 14:53 | |
*** brinzhang_ has joined #openstack-kolla | 14:53 | |
*** brinzhang has quit IRC | 14:54 | |
*** skramaja has quit IRC | 15:03 | |
yoctozepto | mnasiadka: could you look at https://review.opendev.org/c/openstack/kolla-ansible/+/773246 and let me know your thoughts? I'd rather we merged this sooner than later because there is a stack of optimisations/fixes that depends on it | 15:05 |
yoctozepto | this change is small | 15:05 |
mnasiadka | yoctozepto: as long as the message on 137 is self explanatory, could be a nice thing :) | 15:08 |
yoctozepto | mnasiadka: it's as explanatory as it can honestly be :-) | 15:12 |
mnasiadka | yoctozepto: Well, there's still some time in the cycle to battle test it and see if it breaks anything :) | 15:13 |
yoctozepto | mnasiadka: that's my thought | 15:14 |
openstackgerrit | Mark Goddard proposed openstack/kolla-ansible master: Add support for Ceph RadosGW integration https://review.opendev.org/c/openstack/kolla-ansible/+/776662 | 15:14 |
openstackgerrit | Merged openstack/kayobe master: Update IPA docs and test build with extra-hardware https://review.opendev.org/c/openstack/kayobe/+/775093 | 15:21 |
*** brinzhang_ has quit IRC | 15:32 | |
*** brinzhang_ has joined #openstack-kolla | 15:33 | |
openstackgerrit | Pierre Riteau proposed openstack/kayobe stable/victoria: Update IPA docs and test build with extra-hardware https://review.opendev.org/c/openstack/kayobe/+/776947 | 15:38 |
*** rohit02 has joined #openstack-kolla | 15:44 | |
atmark | hello, is there a way I can override in the globals.yml to have custom fqdn for horizon and keystone? for horizon and keystone, I would like the fqdn to be http://openstack.dash that rest of services openstack.internal | 15:45 |
atmark | I have custom certs for keystone and horizon which I can override through haproxy config | 15:48 |
*** rohit02 has quit IRC | 15:50 | |
*** ysirndjuro has joined #openstack-kolla | 16:13 | |
kevko | guys, can u check this ? https://download.kevko.ultimum.cloud/video_debug.mp4 it is weird .. some advices ? :P | 16:13 |
kevko | switching VIP between two hosts with docker restart keepalived ..and keystone is weird .. | 16:14 |
mgoddard | atmark: horizon_public_endpoint and keystone_public_endpoint? | 16:22 |
kevko | (sending token issue with curl in 10 parallel bashesh) | 16:26 |
*** rohit02 has joined #openstack-kolla | 16:29 | |
*** jamesbenson has quit IRC | 16:32 | |
*** vishalmanchanda has quit IRC | 16:34 | |
priteau | kevko: I've seen that restarting keepalived isn't a great way to move the VIP as open database connections keep talking with the old VIP node | 16:35 |
priteau | Restarting haproxy too will help as it closes the connections | 16:35 |
priteau | That's also why we've reduce the use of connection pooling | 16:35 |
kevko | priteau: well, I know :D | 16:35 |
kevko | priteau: but, from my perspective of view it is not HA ... if you will loose connection to server ..you are in same situation | 16:37 |
kevko | our client want to fix from us .. | 16:37 |
priteau | Which release is this? | 16:38 |
mgoddard | kevko: why are you restarting keepalived? | 16:39 |
kevko | mgoddard: simulating lost connection | 16:40 |
kevko | mgoddard: or VIP switched ..whatever reason it is | 16:41 |
mgoddard | kevko: as pierre mentioned, it fares better when haproxy goes down too | 16:41 |
mgoddard | which is more likely to happen in reality | 16:41 |
atmark | mgoddard: I don't see keystone_public_endpoint and horizon_public_endpoint variable in the playbooks. Is the variable named something else? | 16:41 |
mgoddard | try killing the node | 16:41 |
kevko | mgoddard: yes, sure, i know | 16:41 |
mgoddard | kevko: if you know so much, why all the questions? | 16:42 |
mgoddard | :p | 16:42 |
kevko | mgoddard: so you think the situation when machine just lost network connection -> and keepalived will switch IP to another host because of heartbeat can't happen ? :D | 16:43 |
mgoddard | kevko: of course it could. Is that what you are testing by restarting keepalived though? | 16:44 |
kevko | mgoddard: yes, it is quite similar | 16:44 |
mgoddard | deactivating the NIC might be closer | 16:45 |
priteau | kevko: which release are you running? | 16:45 |
kevko | currently ussuri | 16:45 |
mgoddard | atmark: keystone_public_url | 16:46 |
*** Luzi has quit IRC | 16:46 | |
mgoddard | atmark: horizon is not in the keystone endpoint list, so maybe not required? | 16:46 |
atmark | Thanks | 16:48 |
atmark | I'll take a look for horizo | 16:48 |
atmark | horizon* | 16:48 |
priteau | kevko: I was using train when testing something similar. I don't remember seeing the wsgi logs you highlight | 16:49 |
atmark | Do I need to override internal and admin for keystone as well or just the public? | 16:50 |
*** jamesbenson has joined #openstack-kolla | 16:51 | |
kevko | priteau: well, it is only about fact that when i switch IP one backend is just silent ..and that's weird | 16:51 |
kevko | ctrl0, ctrl1 working both (two different IPs) , traffic -> VIP --> ctrl0 or ctrl1 round-robin ====>>>> Switch VIP traffic--> ctrl0 working , ctrl1 no logs, just silent :D | 16:52 |
yoctozepto | mgoddard: thanks for testing the kolla docker config check | 16:54 |
mgoddard | yoctozepto: np. Would be good to move forward with it | 16:54 |
yoctozepto | mgoddard: could you reply back once more? | 16:54 |
yoctozepto | mgoddard: I agree | 16:54 |
*** jamesbenson has quit IRC | 16:55 | |
kevko | priteau: I'm just wondering if keepalived is checking haproxy is alive ..and if not is going to switch VIP .. haproxy can do something similar, can't ? Just check if VIP is configured and if no ..kill a haproxy process and autorestart will spawn container again ...hmm ? | 16:56 |
mnasiadka | yoctozepto, mgoddard: https://review.opendev.org/c/openstack/kolla-ansible/+/771418 - seems to work, reviews please | 16:57 |
mnasiadka | once we agree it looks ok, I'll add the usual CI jobs | 16:58 |
*** jamesbenson has joined #openstack-kolla | 16:58 | |
priteau | kevko: Sorry, I am struggling to follow | 16:58 |
kevko | priteau: I'm just saying that if there is a problem with *IP migrated BUT Haproxy Running* | 17:00 |
mnasiadka | or maybe I'll add them now :D | 17:00 |
kevko | priteau: solution could be to check status of assigned VIP to host, and if not restart itself | 17:01 |
openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: CI: Move from ceph-ansible to cephadm https://review.opendev.org/c/openstack/kolla-ansible/+/771418 | 17:01 |
mgoddard | yoctozepto: done | 17:01 |
mgoddard | kevko: I might be wrong, but I think haproxy needs to bounce before the VIP is lost to make it graceful | 17:02 |
mgoddard | kevko: too late once the VIP is gone | 17:02 |
mgoddard | the connections are already broken | 17:02 |
mgoddard | but you are welcome to try it | 17:03 |
openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: CI: Move from ceph-ansible to cephadm https://review.opendev.org/c/openstack/kolla-ansible/+/771418 | 17:05 |
yoctozepto | mgoddard: thanks, that's what I needed | 17:11 |
yoctozepto | tbh, I tested in a similar way | 17:11 |
yoctozepto | but I guess I waited too long | 17:11 |
yoctozepto | and the backoff was high enough that I did not encounter these issues | 17:11 |
*** cah_link has quit IRC | 17:14 | |
kevko | mgoddard: yes, it's too late ..but better to restart haproxy in very short period of time after VIP is gone than not at all | 17:16 |
*** bengates_ has quit IRC | 17:19 | |
mgoddard | kevko: well, does it help? | 17:23 |
openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: ovn: Move openvswitch related tasks to openvswitch role https://review.opendev.org/c/openstack/kolla-ansible/+/776421 | 17:23 |
kevko | mgoddard: let me try :) | 17:23 |
openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: Add missing octavia-driver-agent https://review.opendev.org/c/openstack/kolla-ansible/+/761872 | 17:26 |
mnasiadka | yoctozepto, mgoddard: when I'm looking at fluentd CI change to fail if fluentd errors... I see there's a lot of work to make it clean -> https://99f0252829f9ba0090bd-b0aa4b46a373faa89ed1ae99b23368b2.ssl.cf5.rackcdn.com/742399/11/check/kolla-ansible-centos8-source/efc87d7/primary/logs/kolla/fluentd-errors.txt | 17:31 |
*** amoralej is now known as amoralej|off | 17:40 | |
*** dougsz has quit IRC | 17:40 | |
*** rohit02 has quit IRC | 17:40 | |
*** rohit02 has joined #openstack-kolla | 17:40 | |
*** jamesbenson has quit IRC | 17:47 | |
yoctozepto | mnasiadka: but it might actually be just a few extra rules | 17:47 |
yoctozepto | hard to tell really | 17:47 |
mnasiadka | probably it is, but we need to fix this before we get way more problems... | 17:47 |
*** rpittau is now known as rpittau|afk | 17:49 | |
*** k_mouza has quit IRC | 17:52 | |
*** k_mouza has joined #openstack-kolla | 17:53 | |
openstackgerrit | Michal Nasiadka proposed openstack/kayobe master: Change docker_registry network_mode to host https://review.opendev.org/c/openstack/kayobe/+/760371 | 17:54 |
mnasiadka | ok, enough for today | 17:56 |
mnasiadka | will look into fluent tomorrow | 17:56 |
*** bengates has joined #openstack-kolla | 17:58 | |
*** jamesbenson has joined #openstack-kolla | 18:02 | |
*** bengates has quit IRC | 18:03 | |
openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: CI: Move from ceph-ansible to cephadm https://review.opendev.org/c/openstack/kolla-ansible/+/771418 | 18:11 |
*** jamesbenson has quit IRC | 18:12 | |
openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: CI: Ansible-lint - update Exxx to names https://review.opendev.org/c/openstack/kolla-ansible/+/776977 | 18:14 |
*** jonaspaulo has joined #openstack-kolla | 18:18 | |
*** amoralej|off is now known as amoralej | 18:19 | |
*** zul has joined #openstack-kolla | 18:29 | |
*** jamesbenson has joined #openstack-kolla | 18:33 | |
*** rohit02 has quit IRC | 18:35 | |
*** rohit02 has joined #openstack-kolla | 18:35 | |
*** jamesbenson has quit IRC | 18:59 | |
*** jamesbenson has joined #openstack-kolla | 18:59 | |
*** rohit02 has quit IRC | 19:02 | |
*** cah_link has joined #openstack-kolla | 19:12 | |
*** jamesbenson has quit IRC | 19:24 | |
*** k_mouza has quit IRC | 19:27 | |
*** jamesbenson has joined #openstack-kolla | 19:28 | |
*** cah_link has quit IRC | 19:56 | |
*** jonaspaulo has quit IRC | 19:59 | |
*** jamesbenson1 has joined #openstack-kolla | 19:59 | |
*** jamesbenson has quit IRC | 20:02 | |
*** jamesbenson1 has quit IRC | 20:03 | |
*** andrewbonney has quit IRC | 20:39 | |
kevko | ok, found weird thing in keystone ... | 20:58 |
kevko | in kolla env | 20:59 |
kevko | when I am not using apache and using uwsgi instead ..everything is OK | 20:59 |
*** kevko has quit IRC | 21:08 | |
*** ysirndjuro has left #openstack-kolla | 21:15 | |
*** zzzeek has quit IRC | 21:26 | |
*** zzzeek has joined #openstack-kolla | 21:28 | |
*** gfidente is now known as gfidente|afk | 21:46 | |
*** maharg101 has quit IRC | 22:08 | |
*** jamesbenson has joined #openstack-kolla | 22:36 | |
*** jamesbenson has quit IRC | 22:53 | |
*** jamesbenson has joined #openstack-kolla | 22:55 | |
*** jamesbenson has quit IRC | 23:13 | |
*** e0ne has quit IRC | 23:20 | |
*** e0ne has joined #openstack-kolla | 23:24 | |
*** e0ne has quit IRC | 23:24 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!