Thursday, 2019-10-24

« Wednesday, 2019-10-23 Index Friday, 2019-10-25 »
*** schwicht has joined #openstack-kolla00:01
*** dciabrin_ has quit IRC00:06
*** dciabrin has joined #openstack-kolla00:16
*** schwicht has quit IRC00:56
*** spsurya has joined #openstack-kolla01:59
openstackgerritgugug proposed openstack/kolla-ansible master: Bump Sphinx lower constraints to 1.8.0  https://review.opendev.org/69078202:24
openstackgerritMerged openstack/kolla-ansible master: Remove delegate_to from service-ks-register tasks  https://review.opendev.org/68500803:00
*** dannins has joined #openstack-kolla03:16
*** shyamb has joined #openstack-kolla03:59
*** shyamb has quit IRC04:19
*** radeks has joined #openstack-kolla04:21
*** shyamb has joined #openstack-kolla04:42
*** shyamb has quit IRC05:39
*** shyamb has joined #openstack-kolla05:46
*** cah_link has joined #openstack-kolla05:51
*** dpawlik has joined #openstack-kolla05:59
*** dpawlik has quit IRC06:03
*** radeks has quit IRC06:07
*** dr_feelgood has joined #openstack-kolla06:16
*** dpawlik has joined #openstack-kolla06:23
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: WIP: Ceph-Ansible CI  https://review.opendev.org/67637606:32
*** pcaruana has joined #openstack-kolla06:41
yoctozeptomorning06:43
mnasiadkamorning06:43
hrwkawusia++06:57
mnasiadkatripleo rc packages still didn't make to mirror.centos.org... grr07:01
*** jaicaa has quit IRC07:02
*** jaicaa has joined #openstack-kolla07:05
yoctozeptomnasiadka: are they going to be mirrored?07:09
yoctozeptomaybe you are waiting for something that is not going to happen07:09
mnasiadkayoctozepto: RDO guys promised they would be there yesterday in the evening, let's give them some more time.07:09
*** jbadiapa has joined #openstack-kolla07:13
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible master: Default to disabling haproxy for memcached  https://review.opendev.org/69086907:13
yoctozeptomnasiadka: what mirror are we talking about? RDO's or infra's?07:13
mnasiadkayoctozepto: first it needs to be on mirror.centos.org, to be on infra's :D07:14
yoctozeptomnasiadka: all right, I thought you were about the infra07:14
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible master: Bump Sphinx lower constraints to 1.8.0  https://review.opendev.org/69078207:16
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: Add proper wsgi loglevel when openstack_logging_debug  https://review.opendev.org/69087107:30
*** gfidente has joined #openstack-kolla07:34
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: WIP: Ceph-Ansible CI  https://review.opendev.org/67637607:42
*** tonythomas has joined #openstack-kolla07:47
*** rpittau|afk is now known as rpittau07:52
*** rgogunskiy has joined #openstack-kolla07:54
*** shyamb has quit IRC08:02
*** amoralej|off is now known as amoralej08:04
*** k_mouza has joined #openstack-kolla08:13
mgoddardmorning all08:18
mgoddardbtw yoctozepto, mnasiadka, hrw: I will be out next Monday - Wednesday (inclusive)08:18
hrwok08:18
hrwmnasiadka: asked boss?08:18
mnasiadkahrw: in the process, awaiting answer ;)08:19
mnasiadkamgoddard: have fun08:19
hrwto be honest I did not asked my wife yet08:19
*** skramaja has joined #openstack-kolla08:19
hrwbut it was never an issue08:19
hrwbb in ~30 minutes08:20
mnasiadkamy wife only complains behind back and reminds about it for the next 30 years ;)08:22
yoctozeptoso hrw needs 30 minutes to ask wifey08:35
*** k_mouza_ has joined #openstack-kolla08:42
openstackgerritMerged openstack/kolla-ansible master: Zun: fix Cinder (volume) iSCSI support  https://review.opendev.org/69061408:42
*** k_mouza has quit IRC08:43
openstackgerritMark Goddard proposed openstack/kolla-ansible master: Fix swift.conf custom configuration path collision  https://review.opendev.org/69088108:44
*** k_mouza_ has quit IRC08:48
hrwyoctozepto: no, it was other reason to be out08:48
yoctozeptohrw: just kidding :-)08:49
*** k_mouza has joined #openstack-kolla08:49
hrwmanylinux2014 is coming. pypi stuff08:51
yoctozeptohrw: I guess will support aarch6408:53
*** chrizl has joined #openstack-kolla08:53
hrwaarch64, ppc64le, armv7l etc08:53
hrws390x08:53
hrwhttps://github.com/pypa/manylinux/pull/34808:54
yoctozeptohrw: glad you have that under control09:01
openstackgerritMark Goddard proposed openstack/kolla-ansible master: CI: Add mariadb test  https://review.opendev.org/65566309:10
yoctozeptoERROR:kolla.common.utils.barbican-base:error creating overlay mount to /var/lib/docker/overlay2/2bb23bb31aff64351da2932a683c0b72529f6e72c31492e2298f8f14b5837fe4-init/merged: device or resource busy09:26
yoctozepto'tis lame09:26
openstackgerritNuman Siddique proposed openstack/kolla master: ovn: Create the ovn dbs only if the schema file exists  https://review.opendev.org/69006509:28
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: Add proper wsgi loglevel when openstack_logging_debug  https://review.opendev.org/69087109:28
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: Add proper wsgi loglevel when openstack_logging_debug  https://review.opendev.org/69087109:33
openstackgerritMerged openstack/kolla-ansible master: Bump Sphinx lower constraints to 1.8.0  https://review.opendev.org/69078209:33
*** shyamb has joined #openstack-kolla09:35
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: WIP: Ceph-Ansible CI  https://review.opendev.org/67637609:47
*** skramaja has quit IRC09:50
openstackgerritMark Goddard proposed openstack/kolla-ansible master: CI: Add mariadb test  https://review.opendev.org/65566309:55
* hrw out10:09
yoctozeptowe have three outputs from timeouts10:15
yoctozepto2 result in 50410:15
yoctozepto1 results in 50010:15
yoctozeptoif service knows it timeouts (due to rabbitmq/mysql/other service) - it shouts 50010:15
yoctozeptoif wsgi timeouts we get 504 from apache (though the timeout seems to be 5 minutes, still longer than haproxy's)10:16
yoctozeptoif apache timeouts (which timeouts due to wsgi above), we get 504 from haproxy10:16
*** dpawlik has quit IRC10:19
mgoddardyoctozepto: is 500 actually a timeout from haproxy? or just from the backend10:20
yoctozepto500 only if service is quick to discover timeout from its dependency10:21
mgoddardok10:21
yoctozeptothen it goes back to the user10:21
yoctozeptoall other cases 50410:21
yoctozepto(which are actually 2)10:21
mgoddarddoes wsgi not detect that the client (haproxy) has disconnected?10:21
yoctozeptokeystone-wsgi is a real hanger10:21
mgoddardpossibly not10:21
mgoddardyeah, I've seen keystone-wsgi hang10:21
yoctozeptonope, it continues10:21
mgoddardguess that makes sense10:22
yoctozeptowhich increase the future failover rate10:22
mgoddardI saw a case where one of the keystone backends would always hang10:22
yoctozeptoI wonder if I fix the keepalived stability10:22
mgoddardand because every operation requires 3 calls to keystone, it always failed10:22
yoctozeptoto let VIP addr stay on one node10:22
yoctozeptoyeah, this surely makes it hard10:22
mgoddardcan we get more info from keystone on what it's doing?10:23
yoctozeptowe already have it in debug10:23
yoctozeptomaybe mnasiadka's change give more from wsgi10:23
mgoddardstrace?10:23
mgoddardwill be a big log :)10:23
yoctozeptoyeah xD10:23
*** rgogunskiy has quit IRC10:23
yoctozeptofor the next run I take mnasiadka's extra logging + keepalived fix if I make it happen10:26
yoctozeptothen either it fixes the issue or we get some more info about it10:26
*** shyamb has quit IRC10:27
yoctozepto(or nothing helps and we are bound to strace)10:27
mgoddardwhat do you think is the problem with keepalived?10:29
*** rgogunskiy has joined #openstack-kolla10:31
*** shyamb has joined #openstack-kolla10:37
*** shyamb has quit IRC10:43
*** shyamb has joined #openstack-kolla10:43
*** dpawlik has joined #openstack-kolla10:44
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible master: DNM: increase keepalived timeout  https://review.opendev.org/69092010:45
yoctozeptomgoddard: keepalived queries haproxy every 2 seconds10:45
yoctozeptowaiting 1 second for reply on the socket10:45
yoctozeptonormally haproxy responds in 1/10th of a sec10:46
yoctozeptoor less10:46
yoctozeptobut here it seems not to10:46
yoctozeptoI increased the timeout to 3 seconds10:46
yoctozeptoand had to increase interval to 3 seconds as well but I think it's ok for production either, it only extends discovery of failed haproxy by 2 seconds10:47
yoctozeptodue to 2 tries10:47
yoctozeptobefore announcing the failure10:47
yoctozeptohaproxy socket down would result in 1, not timeout kill10:48
yoctozeptolooks like haproxy is busy10:48
yoctozeptoI seriously wonder why keystone discovered it has b0rken mysql connection 20 minutes after vip change10:50
yoctozeptoand whether that caused its problems10:51
yoctozeptoit was the first request after vip change after all10:52
*** jbadiapa has quit IRC10:55
*** shyam89 has joined #openstack-kolla10:58
*** shyamb has quit IRC10:59
*** amoralej is now known as amoralej|lunch11:00
yoctozeptothe other two keystone instances discovered the issue pretty immediately and reconnected11:04
yoctozeptowell, they just sent immediate 500 and then were working11:05
yoctozeptoalso, wsrep issues pop out after vip addr change, wonder what that means11:07
*** radeks has joined #openstack-kolla11:07
*** shyamb has joined #openstack-kolla11:08
openstackgerritMark Goddard proposed openstack/kolla-ansible master: Tidy up release notes for Train  https://review.opendev.org/68897311:08
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: WIP: Ceph-Ansible CI  https://review.opendev.org/67637611:10
*** shyam89 has quit IRC11:11
openstackgerritMerged openstack/kolla-ansible master: Limit open file descriptors for Neutron agent containers  https://review.opendev.org/68954811:16
*** shyam89 has joined #openstack-kolla11:17
mnasiadkayoctozepto: maybe it's a docker mtu issue?11:17
mnasiadkaI know we use --network=host, just speculating...11:18
*** X-dark has left #openstack-kolla11:19
*** shyamb has quit IRC11:19
*** k_mouza_ has joined #openstack-kolla11:23
*** kplant has joined #openstack-kolla11:23
*** shyam89 has quit IRC11:24
*** radeks_ has joined #openstack-kolla11:24
*** k_mouza has quit IRC11:25
*** radeks has quit IRC11:26
*** skramaja has joined #openstack-kolla11:27
*** jbadiapa has joined #openstack-kolla11:29
*** shyamb has joined #openstack-kolla11:30
*** jbadiapa has quit IRC11:32
yoctozeptomnasiadka: unlikely, remember we already had 504 before, only adding keepalived+haproxy resulted in more of them11:34
mnasiadkabefore vxlan?11:34
yoctozeptoas we have it in all jobs now11:34
yoctozeptoyeah11:34
*** k_mouza_ has quit IRC11:34
mnasiadkamaybe I was blind for them :)11:34
yoctozeptomaybe, but they were rarer11:35
*** k_mouza has joined #openstack-kolla11:35
yoctozeptosinglenode cannot have a failover in keepalived11:35
yoctozeptoso this failure vector was out of scope11:35
yoctozeptomgoddard: how strong your RP-1 is for https://review.opendev.org/690869 - I strongly advise we get rid of this quirk in Train already ;p11:37
*** jbadiapa has joined #openstack-kolla11:41
mnasiadkayoctozepto: well, unless it breaks something and there's a bug filed...11:43
*** janki has joined #openstack-kolla11:44
yoctozeptomnasiadka: you are not helping :-)11:44
mnasiadkayoctozepto: policy is policy11:44
mnasiadkawe only do bugs now11:44
mnasiadka(and RDO)11:44
mnasiadka:)11:44
yoctozeptoeh, whatever ;p11:45
yoctozeptoThu Oct 24 10:25:54 2019: /check_alive.sh exited due to signal 1511:45
yoctozeptoThu Oct 24 10:25:54 2019: VRRP_Instance(kolla_internal_vip_98) Transition to MASTER STATE11:45
yoctozeptoThu Oct 24 10:25:54 2019: VRRP_Instance(kolla_internal_vip_98) Entering MASTER STATE11:45
yoctozeptoThu Oct 24 10:25:54 2019: VRRP_Instance(kolla_internal_vip_98) using locally configured advertisement interval (1000 milli-sec)11:45
yoctozeptoThu Oct 24 10:26:03 2019: VRRP_Instance(kolla_internal_vip_98) Received advert with lower priority 2, ours 3, forcing new election11:45
yoctozeptoThu Oct 24 10:26:03 2019: VRRP_Instance(kolla_internal_vip_98) Received advert with lower priority 2, ours 3, forcing new election11:46
yoctozeptoThu Oct 24 10:26:03 2019: VRRP_Instance(kolla_internal_vip_98) Received advert with lower priority 2, ours 3, forcing new election11:46
yoctozeptothis bug does not help either (just like mnasiadka)11:46
hrwI have patch waiting for centos8 support replacing centos711:46
yoctozeptocentos8? :O11:46
mnasiadkahrw: there is no RDO for CentOS 811:46
mnasiadkaand there won't be for the next 3-4 weeks11:46
yoctozeptodon't tell me we are going to hold on 3 months11:46
hrwthat's why I did not even sent it for review ;D11:46
hrwI have a copy of kolla which probably do not even work with py211:47
yoctozeptogood, py2 is to be dropped in ussuri11:47
yoctozeptodrop drop drop11:47
mnasiadkayoctozepto: hrw wanted to tell you to wait until U to merge this change ;)11:47
hrwImportError: No module named queue11:51
hrwyes!11:51
hrwmission accomplished11:51
yoctozeptoxD11:54
yoctozeptomnasiadka: ok, ok, I'm giving up11:55
*** dave-mccowan has joined #openstack-kolla11:55
*** shyamb has quit IRC11:59
*** dave-mccowan has quit IRC12:00
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible master: DNM: haproxy timeout - raise value for server  https://review.opendev.org/69070212:01
*** amoralej|lunch is now known as amoralej12:01
* hrw -> meeting12:01
yoctozeptowith wifey12:01
*** radeks_ has quit IRC12:13
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: WIP: Ceph-Ansible CI  https://review.opendev.org/67637612:14
hrwyoctozepto: company one this time12:17
hrwyoctozepto: but if you need psychotherapy then I can arrange you a meeting with my wife12:17
hrw;D12:19
yoctozeptoyou mean legit psychotherapy?12:19
mnasiadkahrw: who doesn't need it :D12:19
yoctozeptoor psychoshock12:19
yoctozepto:D12:19
hrwyoctozepto: legit. certified etc.12:20
mgoddardhrw, yoctozepto: centos8 & source testing can begin without RDO12:22
mgoddardactually that was mnasiadka12:22
mnasiadkamgoddard: well, sure - but we'll soon get to a point we need some packages from RDO12:23
mgoddardmaybs12:23
yoctozeptohrw: wow, nice! :D12:23
mnasiadkaso in T we will have both centos7 and centos8 images? :)12:23
mgoddardin T centos7 only12:24
mnasiadkaok, so we go for c8 in U12:24
mgoddardyeah12:24
mnasiadkamakes sense12:24
yoctozeptore centos8 - I already wanted centos8 for source but we want consistency too much :-)12:24
yoctozeptoit needs work for sure12:24
yoctozeptobut hrw can publish that patch12:24
mnasiadkayoctozepto: even in source we rely on RDO packages, e.g. openvswitch12:24
mgoddardwe discussed backporting fixes to T after RDO release, if they are limited12:25
yoctozeptomnasiadka: ah, yeah, that is a core component, argh12:25
hrwc8 for U+. if someone will want to backport then feel free, will not review12:25
mnasiadkaand the centos8 build system is still not available for SIGs (like Cloud SIG) ;)12:25
yoctozeptostart with linuxbridge12:25
yoctozeptoyeah, U+ definitely12:26
hrw12:01 (0s) hrw@j13-qrep-04:~$ docker run -it centos:812:26
hrwUnable to find image 'centos:8' locally12:26
hrw8: Pulling from library/centos12:26
hrwdocker: no matching manifest for linux/arm64/v8 in the manifest list entries.12:26
hrwso c8 work for me then yet ;D12:27
mnasiadkauh, arm64 will come... in future :D12:28
*** schwicht has joined #openstack-kolla12:30
hrwmnasiadka: my x86-64 boxes are 4-8 cores with 20-32GB ram. arm64 one has 46 cores and 96GB ram so I do quick builds in tmpfs12:30
mnasiadkawell, that's understandable :)12:31
mnasiadkago fix centos on arm64 then ;)12:31
hrwhad to give away fast machines...12:31
hrws/away/back12:31
yoctozeptois arm64 supported upstream? in rhel8 I mean12:33
hrwyoctozepto: yes12:34
hrwyoctozepto: rhel8 has four equal architectures: aarch64, ppc64le, s390x, x86-6412:34
hrwpackage has to build on each12:34
hrwunless does not make sense like dmidecode which is aarch64 and x86-64 only etc12:35
openstackgerritBharat Kunwar proposed openstack/kayobe master: Example should use hyphen instead of underscore  https://review.opendev.org/69094912:35
openstackgerritBharat Kunwar proposed openstack/kayobe master: Example should use hyphen instead of underscore  https://review.opendev.org/69094912:36
yoctozeptohrw: aye, thx12:36
openstackgerritBharat Kunwar proposed openstack/kayobe master: Example should use hyphen instead of underscore  https://review.opendev.org/69094912:37
hrwyoctozepto: centos will add armv7l and i686 probably as altarch but both can be ignored12:37
hrwi686 means old junk people are too sentimental to throw away, armv7l means toys-like hardware (same sentimental stuff)12:38
mnasiadkahrw: I think there are some problems with kolla builds on Debian...12:41
mnasiadkahttps://zuul.opendev.org/t/openstack/build/0101b0e167a74fbb8146ae6beda27987/log/primary/logs/build/000_FAILED_placement-base.log#612:41
hrwfucking docker...12:41
hrwit is problem with docker on anything with fresh enough kernel12:42
mnasiadkaadd support for podman and buildah :)12:43
hrwmnasiadka: one condition: you do podman in k-a and changes in kolla to get k-a working12:44
hrwmnasiadka: as podman runs on user account so GID/UID stuff can be complicated12:44
hrwfrom what I heard12:44
yoctozeptohrw: we will run it as root then ;p12:44
mnasiadkaand we would need systemd units for each container...12:44
yoctozeptoyeah, for persistency12:45
openstackgerritMerged openstack/kolla master: Fix mariabackup arguments  https://review.opendev.org/69058912:45
openstackgerritMerged openstack/kolla master: ovn: Create the ovn dbs only if the schema file exists  https://review.opendev.org/69006512:45
mnasiadkahrw: well, we could start at least building images using buildah12:45
openstackgerritMerged openstack/kolla-ansible master: Tidy up release notes for Train  https://review.opendev.org/68897312:45
hrwyoctozepto: ;D12:45
yoctozeptowe surely need to start toying with that12:45
yoctozeptoas it looks like the way forward12:45
hrwworks for tripleo guys with our templates12:45
mnasiadkaeverything looks like a way forward until you use it extensively12:46
yoctozeptofair point12:46
mnasiadkahrw: yeah well, we would need to rework image build python code, to support both docker and podman ;)12:46
* hrw -> food12:47
*** schwicht has quit IRC12:50
mnasiadkahrw: well, they thought they fixed it - but they didn't: https://github.com/docker/for-linux/issues/71112:52
*** born2bake has joined #openstack-kolla12:56
born2bakeIs anyone had the same issue - https://bugs.launchpad.net/kolla-ansible/+bug/1847905 ?12:56
openstackLaunchpad bug 1847905 in kolla-ansible "Octavia loadbalancer pending create state" [Undecided,New]12:56
*** goldyfruit has joined #openstack-kolla13:00
yoctozeptomnasiadka: commented there13:05
openstackgerritMark Goddard proposed openstack/kayobe master: Use docker_custom_config variable  https://review.opendev.org/69096513:07
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: WIP: Ceph-Ansible CI  https://review.opendev.org/67637613:13
*** janki has quit IRC13:13
mnasiadkaborn2bake: it seems like an Octavia code issue - have you raised a bug in Octavia on Storyboard?13:16
openstackgerritMerged openstack/kayobe master: Add nova-super-conductor to inventory  https://review.opendev.org/68872513:23
yoctozeptono keepalived instability, no other issues at hand, still hang of keystone admin13:27
yoctozeptohttps://a30336fa6a8fca5c6dba-fe779e5654b21fdff79727b204dfb7d6.ssl.cf1.rackcdn.com/690920/1/check/kolla-ansible-centos-source-ceph/8ef81a0/13:27
yoctozeptomariadb connections were broken here though as well13:28
yoctozeptocould be the results of rebootstraps happening in reconfigure13:28
*** spsurya has quit IRC13:29
born2bakemnasiadka no, is it octavia issue? i thought its related to kolla13:44
mnasiadkaborn2bake: it is, read Dincer’s comment13:44
mnasiadkaWhat he is doing is a workaround13:45
born2bakeI am not confident enough to customize kolla images lol13:46
born2bakealso, I do have similar problem with magnum...cluster is just in pending create state... https://bugs.launchpad.net/kolla-ansible/+bug/184647613:46
openstackLaunchpad bug 1846476 in kolla-ansible "K8s cluster deployment is stuck at kube_cluster_deploy OS::Heat::SoftwareDeployment" [Undecided,New]13:46
born2bake@mar13:47
born2bakemarkmcclain in case they fix the isses with octavia code, how do I update my images so the bug will be fixed?13:47
*** born2bake has quit IRC13:49
*** amoralej is now known as amoralej|mtg14:00
openstackgerritMark Goddard proposed openstack/kolla-ansible master: Restart nova scheduler to refresh cell cache  https://review.opendev.org/69098214:03
*** learnstack has joined #openstack-kolla14:05
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible master: Allow passing arguments to RabbitMQ server  https://review.opendev.org/68636914:06
*** cah_link has quit IRC14:07
*** dpawlik has quit IRC14:11
*** pcaruana has quit IRC14:19
mnasiadkayoctozepto: so now what, we need to triple check mariadb state?14:39
*** learnstack has quit IRC14:48
*** evrardjp_ is now known as evrardjp14:48
*** amoralej|mtg is now known as amoralej14:49
openstackgerritMark Goddard proposed openstack/kayobe master: Use docker_custom_config variable  https://review.opendev.org/69096514:51
*** learnstack has joined #openstack-kolla14:59
*** gouthamr_ is now known as gouthamr15:01
*** learnstack has quit IRC15:06
*** born2bake has joined #openstack-kolla15:09
*** priteau has joined #openstack-kolla15:09
openstackgerritWill Szumski proposed openstack/kolla-ansible stable/stein: Apply Resource Constraints to Services.  https://review.opendev.org/69100015:10
*** rgogunskiy has quit IRC15:10
*** gedcib has joined #openstack-kolla15:16
openstackgerritMark Goddard proposed openstack/kolla-ansible master: Fix kolla-bootstrap-servers with docker_storage_driver set  https://review.opendev.org/69100115:17
openstackgerritMark Goddard proposed openstack/kayobe master: Use docker_custom_config variable  https://review.opendev.org/69096515:18
openstackgerritMark Goddard proposed openstack/kolla-ansible master: Use mariabackup for database backups  https://review.opendev.org/69059815:20
openstackgerritMark Goddard proposed openstack/kolla-ansible master: CI: Add mariadb test  https://review.opendev.org/65566315:21
gedcibhi guys, when i add custom keystone.conf for one of the domains and run kolla-reconfigure it fails with this error:15:23
gedcibTASK [keystone : Creating admin project, user, role, service, and endpoint] *******************************************************************************************************************15:23
gedcibfatal: [nested]: FAILED! => {"msg": "The conditional check '(keystone_bootstrap.stdout | from_json).changed' failed. The error was: Invalid control character at: line 1 column 134 (char 133)"}15:23
openstackgerritMark Goddard proposed openstack/kolla-ansible master: Use mariabackup for database backups  https://review.opendev.org/69059815:28
openstackgerritMark Goddard proposed openstack/kolla-ansible master: CI: Add mariadb test  https://review.opendev.org/65566315:28
openstackgerritWill Szumski proposed openstack/kolla-ansible stable/stein: Apply Resource Constraints to Services.  https://review.opendev.org/69100015:29
openstackgerritMark Goddard proposed openstack/kolla-ansible master: CI: Add mariadb test  https://review.opendev.org/65566315:31
openstackgerritMerged openstack/kolla-ansible master: Add proper wsgi loglevel when openstack_logging_debug  https://review.opendev.org/69087115:34
openstackgerritWill Szumski proposed openstack/kolla-ansible stable/stein: Limit open file descriptors for Neutron agent containers  https://review.opendev.org/69100715:39
*** gedcib has quit IRC15:40
*** goldyfruit has quit IRC15:54
*** goldyfruit has joined #openstack-kolla15:56
*** k_mouza_ has joined #openstack-kolla16:02
*** rpittau is now known as rpittau|afk16:05
*** k_mouza has quit IRC16:06
openstackgerritAdeberg proposed openstack/kolla-ansible master: Update swift templates to support swift_extra_ring_files  https://review.opendev.org/69101416:32
*** igordc has joined #openstack-kolla16:39
*** k_mouza_ has quit IRC16:45
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: WIP: Ceph-Ansible CI  https://review.opendev.org/67637616:47
openstackgerritMark Goddard proposed openstack/kolla-ansible master: Use mariabackup for database backups  https://review.opendev.org/69059816:49
openstackgerritMark Goddard proposed openstack/kolla-ansible master: CI: Add mariadb test  https://review.opendev.org/65566316:49
*** spiette has quit IRC16:52
*** spiette has joined #openstack-kolla16:55
*** priteau has quit IRC16:59
*** igordc has quit IRC17:01
*** igordc has joined #openstack-kolla17:02
mnasiadkahrw: https://review.opendev.org/#/c/689121/ - are we merging it?17:04
hrwmnasiadka: I would not write it otherwise17:04
mnasiadkahrw: rpm_setup_config has the same issue, or does it work? :)17:05
hrwmnasiadka: should just work as it is handled other way17:05
mnasiadkaso +2+w and we're off17:05
hrwthanks17:05
*** k_mouza has joined #openstack-kolla17:06
* hrw off17:06
mnasiadkathis was the last bug in kolla attached to Train milestone17:06
mnasiadka(last not fixed)17:06
hrwmnasiadka: rpm_setup_config is handled in docker/base/Dockerfile.j2 L141-14317:08
hrwas kolla/image/build.py builds command which is then run17:08
*** k_mouza has quit IRC17:10
kplantis there a recommended way to safely reboot control nodes?17:13
kplanti need to restart the docker daemons to update trusted CAs :-|17:14
*** stingrayza has quit IRC17:15
noxoididk if its recommended but i just restart the docker daemons one at a time to let any quorum services (ceph mon/mariadb/rabbitmq) catch up17:18
noxoidassuming docker doesnt lose track of the runc processes and orphans them17:19
kplanti've done that before and things kind of fell down17:19
kplantrabbitmq shit itself17:19
kplantthankfully i don't have ceph on the os controllers17:20
noxoidare you using autoheal or pause_minority?17:20
kplantwhichever the default kolla-ansible is configuring for rmq17:21
noxoidwell it changed recently :P17:22
kplantoh17:23
kplantlet me check with cluster_status17:23
noxoidhttps://github.com/openstack/kolla-ansible/commit/cdfc1c23442f43b6d86010cbd5147802c521493417:23
noxoidin stein its still autoheal17:23
kplantyeah - it's autoheal17:24
noxoidyou may have better luck by switching it to pause_minority17:24
kplantappreciate it17:24
noxoidbut even when i was on autoheal it could still handle rolling restarts17:24
noxoidor at least i didnt notice any problems17:24
kplantmaybe i didn't wait long enough between booting nodes?17:24
noxoidits possible17:24
noxoidi usually wait 3-4 minutes between each17:24
kplanti couldn't get rabbit to recover until i booted the entire physical server17:24
*** henriqueof has joined #openstack-kolla17:25
noxoidid run cluster_status as well as node_health_check before moving on to the next node17:25
noxoidive seen cases where node_health_check will report issues but cluster_status would not17:25
kplantappreciate the insight17:25
noxoidno problem. rabbitmq is the only component of openstack that im still having unknown issues with17:25
*** stingrayza has joined #openstack-kolla17:25
kplanti'm going to start with the machine that doesn't have the vip to be safe17:26
kplantif it recovers and there are no partitions i'll keep rolling17:27
noxoidsounds good17:27
kplantit's immediately healthy - that's skeptical17:32
kplantnothing works that well17:32
*** stingrayza has quit IRC17:48
*** stingrayza has joined #openstack-kolla17:49
*** amoralej is now known as amoralej|off17:52
kplantwell all three nodes are healthy17:54
*** stingrayza has quit IRC17:54
kplantwe'll see if anything blows up :]17:54
*** kplant has quit IRC17:55
*** stingrayza has joined #openstack-kolla18:02
*** schwicht has joined #openstack-kolla18:13
openstackgerritMerged openstack/kolla master: get 'apt_*' options of kolla-build.conf working again  https://review.opendev.org/68912118:58
*** born2bake has quit IRC19:34
*** goldyfruit_ has joined #openstack-kolla19:57
*** goldyfruit has quit IRC19:59
*** KeithMnemonic has quit IRC20:41
henriqueofInstalled a let's encrypt using kolla-ansible, the certificate expired just now, how can I change it?20:41
*** KeithMnemonic has joined #openstack-kolla20:52
*** goldyfruit_ has quit IRC21:38
*** schwicht has quit IRC21:43
*** goldyfruit has joined #openstack-kolla22:03
*** gfidente has quit IRC22:06
*** igordc has quit IRC22:08
*** schwicht has joined #openstack-kolla22:29
*** schwicht has quit IRC22:36
*** schwicht has joined #openstack-kolla22:43
*** k_mouza has joined #openstack-kolla22:46
*** igordc has joined #openstack-kolla22:47
*** schwicht has quit IRC22:51
*** k_mouza has quit IRC22:51
*** zbr has quit IRC23:13
*** henriqueof has quit IRC23:33
*** schwicht has joined #openstack-kolla23:34
« Wednesday, 2019-10-23 Index Friday, 2019-10-25 »

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!