| *** Limech_ has quit IRC | 00:38 | |
| *** hongbin has quit IRC | 00:47 | |
| *** hongbin has joined #openstack-kolla | 00:50 | |
| *** e0ne has joined #openstack-kolla | 00:55 | |
| *** Torel has quit IRC | 00:55 | |
| *** Torel has joined #openstack-kolla | 00:57 | |
| *** e0ne has quit IRC | 00:59 | |
| openstackgerrit | Hongbin Lu proposed openstack/kolla-ansible stable/stein: [DNM] Test Zun job in stable/stein https://review.opendev.org/733999 | 01:07 |
|---|---|---|
| *** noxoid has quit IRC | 01:53 | |
| *** noxoid has joined #openstack-kolla | 02:00 | |
| *** noxoid8 has joined #openstack-kolla | 02:02 | |
| *** noxoid8 has quit IRC | 02:04 | |
| *** noxoid has quit IRC | 02:05 | |
| *** bsanjeewa_ has joined #openstack-kolla | 02:19 | |
| *** bsanjeewa has quit IRC | 02:23 | |
| openstackgerrit | Hongbin Lu proposed openstack/kolla-ansible stable/rocky: CI: Fix Ironic and Zun scenarios testing https://review.opendev.org/734001 | 02:27 |
| openstackgerrit | Hongbin Lu proposed openstack/kolla-ansible stable/rocky: [DNM] Test Zun job https://review.opendev.org/733997 | 02:28 |
| *** diurnalist has joined #openstack-kolla | 02:28 | |
| *** Torel has quit IRC | 02:39 | |
| *** bsanjeewa has joined #openstack-kolla | 03:12 | |
| *** bsanjeewa_ has quit IRC | 03:15 | |
| *** bsanjeewa has quit IRC | 03:38 | |
| *** diurnalist has quit IRC | 03:43 | |
| *** bsanjeewa has joined #openstack-kolla | 04:05 | |
| *** ricolin has joined #openstack-kolla | 04:14 | |
| *** ykarel|away is now known as ykarel | 04:23 | |
| *** vishalmanchanda has joined #openstack-kolla | 04:33 | |
| *** abdysn has joined #openstack-kolla | 04:57 | |
| oklhost | Good morning. | 05:00 |
| oklhost | Something broken in all-in-one deploy? http://paste.openstack.org/show/794445/ | 05:01 |
| *** k_mouza has joined #openstack-kolla | 05:07 | |
| *** k_mouza has quit IRC | 05:11 | |
| *** hongbin has quit IRC | 05:12 | |
| *** sorin-mihai has quit IRC | 05:32 | |
| *** factor has quit IRC | 06:04 | |
| *** seco has joined #openstack-kolla | 06:12 | |
| *** nikparasyr has joined #openstack-kolla | 06:42 | |
| *** lvdombrkr has joined #openstack-kolla | 06:46 | |
| openstackgerrit | Pierre Riteau proposed openstack/kayobe master: Fix documented name of time configuration file https://review.opendev.org/734020 | 06:53 |
| *** zijlboot has quit IRC | 06:58 | |
| *** zijlboot has joined #openstack-kolla | 06:59 | |
| *** stingrayza has joined #openstack-kolla | 07:03 | |
| *** skramaja has joined #openstack-kolla | 07:04 | |
| *** also_stingrayza has quit IRC | 07:05 | |
| yoctozepto | oklhost: nope, you simply don't have kolla-ansible properly installed | 07:06 |
| yoctozepto | morning channel | 07:06 |
| yoctozepto | oklhost: ansible & kolla-ansible must live in the same python environment | 07:06 |
| yoctozepto | that does not seem to be the case in the attached logs | 07:07 |
| oklhost | yoctozepto: +1, thanks ;-) | 07:09 |
| *** rpittau|afk is now known as rpittau | 07:14 | |
| *** bsanjeewa_ has joined #openstack-kolla | 07:16 | |
| *** bsanjeewa has quit IRC | 07:19 | |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible stable/rocky: CI: Fix Ironic and Zun scenarios testing https://review.opendev.org/734001 | 07:24 |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible stable/queens: CI: Fix Ironic and Zun scenarios testing https://review.opendev.org/734024 | 07:24 |
| *** bengates has joined #openstack-kolla | 07:26 | |
| mgoddard | morning | 07:27 |
| *** bengates has quit IRC | 07:27 | |
| *** bengates has joined #openstack-kolla | 07:28 | |
| mgoddard | lvdombrkr: password injection issue looks like a nova python3 bug | 07:28 |
| mgoddard | try raising in #openstack-nova | 07:28 |
| lvdombrkr | mgoddard: morning. yes its bug, opesntack-nova already confirmed | 07:29 |
| lvdombrkr | is there any other ways to inject password geenrated by openstac? | 07:30 |
| *** nathharp_ has quit IRC | 07:31 | |
| openstackgerrit | Mark Goddard proposed openstack/kayobe stable/train: Don't disable SELinux if it is not installed https://review.opendev.org/731593 | 07:32 |
| *** nathharp has joined #openstack-kolla | 07:32 | |
| openstackgerrit | Mark Goddard proposed openstack/kayobe stable/stein: Don't disable SELinux if it is not installed https://review.opendev.org/734025 | 07:32 |
| openstackgerrit | Mark Goddard proposed openstack/kayobe stable/rocky: Don't disable SELinux if it is not installed https://review.opendev.org/734026 | 07:32 |
| *** dougsz has joined #openstack-kolla | 07:33 | |
| *** sorin-mihai has joined #openstack-kolla | 07:47 | |
| *** amoralej|off is now known as amoralej | 07:52 | |
| *** bsanjeewa has joined #openstack-kolla | 07:54 | |
| *** e0ne has joined #openstack-kolla | 07:55 | |
| *** bsanjeewa_ has quit IRC | 07:57 | |
| sorin-mihai | good morning. repeating myself saying the same thing i said on Sunday. a few days ago my all-in-one node decided to ignore /etc/hosts even though before it worked. node fqdn, internal fqdn and external fqdn where pointing to private IPs. as a result almost everything stopped to work. i had to create records in the public DNS pointing to the private IPs to make some things work again. i'm not sure if this is the reason why, but | 07:59 |
| sorin-mihai | mariadb stopped being reachable over the internal fqdn via haproxy. if i keep database_address set to kolla_internal_fqdn mariadb is not reachable, so i tried to change database_address from kolla_internal_fqdn to api_interface_address, but that didn't solve anything. with that change i can deploy/reconfigure everything except nova, which still doesn't manage to connect to database. if i don't make the change i can't even login | 07:59 |
| sorin-mihai | to horizon it will always, if i do make the change, i can login in horizon but almost nothing works in horizon. this is the worst situation i've had so far with this all-in-one, i don't know what to look for, please give me some ideas | 07:59 |
| *** e0ne has quit IRC | 08:00 | |
| openstackgerrit | Merged openstack/kolla-ansible stable/queens: CI: Fix Ironic and Zun scenarios testing https://review.opendev.org/734024 | 08:00 |
| *** numans has joined #openstack-kolla | 08:00 | |
| *** nathharp has quit IRC | 08:01 | |
| mgoddard | sorin-mihai: are you running haproxy? | 08:03 |
| sorin-mihai | yes | 08:06 |
| *** wuchunyang has joined #openstack-kolla | 08:08 | |
| sorin-mihai | mgoddard, here is my /etc/kolla/haproxy/services.d/mariadb.cfg http://paste.openstack.org/raw/L4VtM5gpDnKplZ5lG2eq/ | 08:10 |
| mgoddard | sorin-mihai: is the host listening on the VIP? | 08:11 |
| sorin-mihai | mgoddard, i believe it is. i can telnet on 80 and 443 on both VIP and FQDN, but on on 3306. in the browser i can see the login page, but when i try to login it times out | 08:15 |
| mgoddard | sorin-mihai: you can check by running 'ip address | grep $vip' | 08:16 |
| mgoddard | it should be a secondary IP | 08:16 |
| mgoddard | have you checked the mariadb logs? | 08:16 |
| sorin-mihai | yeah, the VIP is there, haproxy listens on it. not sure what to look for in the mariadb logs. i can docker exec -it mariadb bash and i can login just fine, also can telnet 3306 from other hosts in the same subnet but only the node's IP not the VIP | 08:19 |
| mgoddard | sorin-mihai: firewall? | 08:20 |
| mgoddard | MAC/IP filtering? | 08:21 |
| sorin-mihai | it's a centos node, firewalld is stopped | 08:21 |
| yoctozepto | selinux? :D | 08:22 |
| sorin-mihai | permisive, i never touched it. i don't see anything that should block the traffic from haproxy to mariadb inside the same node | 08:23 |
| sorin-mihai | from any other host in the subnet, 'telnet 10.254.255.253 3306' doesn't work (haproxy internal VIP), 'telnet 10.254.0.2 3306' works (node's IP). same thing from inside the node itself, so i guess it's some haproxy issue | 08:26 |
| hrw | ing | 08:28 |
| wuchunyang | you can exec "netstat -atnp | grep 3306" then, you will know whether haproxy listen on this ip or not. | 08:28 |
| sorin-mihai | tcp 0 0 10.254.0.2:3306 0.0.0.0:* LISTEN - | 08:29 |
| sorin-mihai | tcp 0 0 10.254.255.253:3306 0.0.0.0:* LISTEN - | 08:29 |
| yoctozepto | sorin-mihai: can you ping the vip address at all? | 08:30 |
| wuchunyang | can you telnet 10.254.255.253 localhost ? | 08:31 |
| wuchunyang | or is this openstack vm ? | 08:31 |
| *** bsanjeewa_ has joined #openstack-kolla | 08:36 | |
| *** Fl1nt has joined #openstack-kolla | 08:37 | |
| Fl1nt | Good morning everyone! | 08:37 |
| *** born2bake has joined #openstack-kolla | 08:38 | |
| *** bsanjeewa has quit IRC | 08:40 | |
| sorin-mihai | yoctozepto, wuchunyang, i can ping, i can telnet, from both inside and outside the node. the only thing that doesn't work is telnet 10.254.255.253 3306, which makes me think that for some reason haproxy is not doing the thing properly | 08:40 |
| yoctozepto | sorin-mihai: anything in haproxy logs? | 08:40 |
| openstackgerrit | Merged openstack/kayobe master: Fix documented name of time configuration file https://review.opendev.org/734020 | 08:41 |
| yoctozepto | is xinetd listening on 4569 ? | 08:41 |
| yoctozepto | since you are using the httpchk | 08:41 |
| sorin-mihai | {"Payload":"Jun 4 09:38:00 haproxy[64]: 10.254.255.253:47406 [04/Jun/2020:09:38:00.082] mariadb mariadb/<NOSRV> -1/-1/0 0 SC 1/1/0/0/0 0/0","log_level":"info","Hostname":"stk1-blr1.blockgemini.net","programname":"haproxy"} | 08:42 |
| sorin-mihai | i assume xinet is not running, telnet to 4569 says connection refused | 08:43 |
| *** e0ne has joined #openstack-kolla | 08:43 | |
| sorin-mihai | xinetd i mean | 08:44 |
| *** e0ne has quit IRC | 08:48 | |
| wuchunyang | try "netstat -atnp | grep 4569" ? | 08:48 |
| yoctozepto | sorin-mihai: so haproxy marked the only server down because of lack of xinetd mysql check | 08:48 |
| yoctozepto | sorin-mihai: I wonder did you disable it? | 08:49 |
| yoctozepto | this is new one in ussuri | 08:49 |
| yoctozepto | maybe it failed to deploy for some reason? | 08:49 |
| wuchunyang | yoctozepto it this health check ? | 08:50 |
| yoctozepto | it is much more reliable than the previous mysql check from haproxy | 08:50 |
| yoctozepto | wuchunyang: yeah, it is | 08:50 |
| yoctozepto | checks whether wsrep is fine | 08:50 |
| wuchunyang | it is listened by xinetd ? | 08:51 |
| wuchunyang | is it listened by xinetd ? | 08:52 |
| *** cah_link has joined #openstack-kolla | 08:53 | |
| *** nathharp has joined #openstack-kolla | 08:53 | |
| *** nathharp has quit IRC | 08:54 | |
| *** nathharp has joined #openstack-kolla | 08:55 | |
| sorin-mihai | yoctozepto, nothing in netstat | 08:56 |
| *** gfidente has joined #openstack-kolla | 08:59 | |
| sorin-mihai | yoctozepto, is there some way to just start xinetd manually and try to redeploy? | 08:59 |
| *** k_mouza has joined #openstack-kolla | 09:03 | |
| openstackgerrit | Mark Goddard proposed openstack/kayobe master: Fix seed VM interface ordering https://review.opendev.org/734040 | 09:06 |
| *** eliaswimmer has joined #openstack-kolla | 09:08 | |
| *** wuchunyang has quit IRC | 09:08 | |
| *** jschlueter has joined #openstack-kolla | 09:13 | |
| openstackgerrit | Michal Arbet proposed openstack/kolla-ansible master: Fix chrony permissions inside docker container https://review.opendev.org/734042 | 09:14 |
| openstackgerrit | Mark Goddard proposed openstack/kayobe master: WIP: Add seed VM provisioning CI job https://review.opendev.org/559706 | 09:15 |
| *** kevko has joined #openstack-kolla | 09:15 | |
| *** ricolin has quit IRC | 09:18 | |
| openstackgerrit | Mark Goddard proposed openstack/kayobe master: DNM: use cirros for seed VM https://review.opendev.org/734050 | 09:28 |
| openstackgerrit | Merged openstack/kolla-ansible stable/rocky: CI: Fix Ironic and Zun scenarios testing https://review.opendev.org/734001 | 09:29 |
| *** bsanjeewa has joined #openstack-kolla | 09:31 | |
| *** bsanjeewa_ has quit IRC | 09:33 | |
| yoctozepto | sorin-mihai: mariadb role should start it up | 09:38 |
| yoctozepto | sorin-mihai: it's another container :D | 09:38 |
| sorin-mihai | what's the name of the container? i've run multiple deploy/reconfigure -t mariadb and that didn't fix anything, so maybe something is broken with the role on my side | 09:40 |
| openstackgerrit | Mark Goddard proposed openstack/kayobe master: Fix seed VM interface ordering https://review.opendev.org/734040 | 09:42 |
| *** wuchunyang has joined #openstack-kolla | 09:44 | |
| openstackgerrit | Mark Goddard proposed openstack/kayobe master: Fix seed VM interface ordering https://review.opendev.org/734040 | 09:46 |
| openstackgerrit | Mark Goddard proposed openstack/kayobe master: Fix seed VM interface ordering https://review.opendev.org/734040 | 09:48 |
| openstackgerrit | Mark Goddard proposed openstack/kayobe master: DNM: use cirros for seed VM https://review.opendev.org/734050 | 09:49 |
| openstackgerrit | Mark Goddard proposed openstack/kolla stable/train: Fix multiple issues https://review.opendev.org/733557 | 09:53 |
| *** seco has quit IRC | 09:57 | |
| *** bsanjeewa_ has joined #openstack-kolla | 09:59 | |
| *** jbadiapa has joined #openstack-kolla | 10:02 | |
| openstackgerrit | Mark Goddard proposed openstack/kayobe stable/stein: Install libselinux-python on Ansible control host https://review.opendev.org/734054 | 10:02 |
| *** bsanjeewa has quit IRC | 10:03 | |
| openstackgerrit | Mark Goddard proposed openstack/kayobe stable/train: CentOS 8: separate kolla build tag from deploy tag https://review.opendev.org/734055 | 10:03 |
| *** seco has joined #openstack-kolla | 10:06 | |
| openstackgerrit | Mark Goddard proposed openstack/kayobe master: WIP: Add seed VM provisioning CI job https://review.opendev.org/559706 | 10:08 |
| *** Tony31 has joined #openstack-kolla | 10:08 | |
| *** seco has quit IRC | 10:11 | |
| Tony31 | Hi @mgoddard hope you had a good weekend. Do you have a minute? Could I ask for help on this? I am a bit stuck. When I run "deploy" (first time deployment) I get error "TASK [ironic : Ensure dnsmasq_dhcp_range configured for Ironic Dnsmasq] ********************************************************************************** | 10:12 |
| Tony31 | fatal: [juc-kocon1-prd]: FAILED! => {"changed": false, "msg": "ironic_dnsmasq_dhcp_range must be set when enable ironic\n"}" | 10:12 |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla-ansible master: CI: move playbook files to tests/playbooks/ dir https://review.opendev.org/715872 | 10:13 |
| hrw | let's check | 10:13 |
| Tony31 | This is one of the controller nodes. It's being added to the ironic group in the hosts. I didnt change any config when I copied over the etc/kayobe so I'm confused on this and am struggling to figure out where this is coming from | 10:13 |
| *** seco has joined #openstack-kolla | 10:14 | |
| *** rpittau is now known as rpittau|bbl | 10:15 | |
| *** wuchunyang has quit IRC | 10:16 | |
| *** k_mouza has quit IRC | 10:17 | |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla-ansible master: CI: move playbook files to tests/playbooks/ dir https://review.opendev.org/715872 | 10:19 |
| *** k_mouza has joined #openstack-kolla | 10:22 | |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla-ansible master: CI: move playbook files to tests/playbooks/ dir https://review.opendev.org/715872 | 10:24 |
| hrw | "msg": "Could not find or access '/home/zuul/src/opendev.org/openstack/kolla-ansible/tests/setup_gate.sh' on the Ansible Controller.\nIf you are using a module and expect the file to exist on the remote, see the remote_src option" | 10:32 |
| hrw | and file exists... | 10:33 |
| Tony31 | possibly this file has changed since I last done a fresh deployment back in jan/feb: venvs/kolla-ansible/share/kolla-ansible/ansible/site.yml. I was able to comment out ironic (and cinder-backup) and the deployment is progressing, now | 10:35 |
| Tony31 | # - enable_ironic_{{ enable_ironic | bool }} | 10:35 |
| openstackgerrit | Mark Goddard proposed openstack/kayobe master: WIP: Add seed VM provisioning CI job https://review.opendev.org/559706 | 10:37 |
| *** gfidente has quit IRC | 10:37 | |
| hrw | can someone look at https://review.opendev.org/715872 failure? | 10:39 |
| patchbot | patch 715872 - kolla-ansible - CI: move playbook files to tests/playbooks/ dir - 7 patch sets | 10:39 |
| mgoddard | Tony31: set kolla_enable_ironic to false | 10:46 |
| Tony31 | Hi @mgoddard - so do I have to enable the variable then set it as false? | 10:50 |
| mgoddard | Tony31: right - uncomment, then set it to false | 10:51 |
| Tony31 | thank you - that must be the problem | 10:51 |
| yoctozepto | sorin-mihai: mariadb-clustercheck is the name, xinetd runs in it | 10:54 |
| yoctozepto | sorin-mihai: sorry for being unclear | 10:54 |
| *** k_mouza has quit IRC | 10:59 | |
| *** lvdombrkr has quit IRC | 11:00 | |
| *** wuchunyang has joined #openstack-kolla | 11:05 | |
| openstackgerrit | Michal Arbet proposed openstack/kolla-ansible master: Fix chrony permissions inside docker container https://review.opendev.org/734042 | 11:12 |
| kevko | mgoddard: hi, Mark, could you check https://review.opendev.org/#/c/734042 again, as I am not sure if release note is OK, i'm not writing it every day :/ | 11:13 |
| patchbot | patch 734042 - kolla-ansible - Fix chrony permissions inside docker container - 2 patch sets | 11:13 |
| *** lvdombrkr has joined #openstack-kolla | 11:15 | |
| *** k_mouza has joined #openstack-kolla | 11:17 | |
| mgoddard | kevko: looks fine. You didn't respond to my comment though. Why is it not running as the chrony user on debian? This works on other distros | 11:18 |
| *** gfidente has joined #openstack-kolla | 11:21 | |
| kevko | mgoddard: debian package use different user -> _chrony:x:111:115:Chrony daemon,,,:/var/lib/chrony:/usr/sbin/nologin | 11:22 |
| mgoddard | <facepalm> | 11:22 |
| kevko | mgoddard: ubuntu of course too :) | 11:22 |
| mgoddard | _chrony? | 11:22 |
| kevko | mgoddard: here you can check postinst of chrony package -> https://salsa.debian.org/debian/chrony/-/blob/master/debian/postinst | 11:23 |
| mgoddard | kevko: would it be better to change the owner to _chrony then? | 11:24 |
| mgoddard | this does sidestep our UID configuration for the chrony user | 11:24 |
| mgoddard | hopefully not too much of a problem in this case | 11:25 |
| *** wuchunyang has quit IRC | 11:26 | |
| kevko | mgoddard: i'm going to check if helps | 11:29 |
| kevko | mgoddard: simply changed owner to _chrony , but same issue | 11:32 |
| *** Torel has joined #openstack-kolla | 11:32 | |
| kevko | mgoddard: i think 0644 is not too open and working ..so , why not set to 644 ? | 11:33 |
| hrw | to make sure that suid/sgid are cleared? | 11:34 |
| openstackgerrit | Mark Goddard proposed openstack/kayobe master: Remove DIB kernel cleanup workaround https://review.opendev.org/734063 | 11:35 |
| *** Gtv has joined #openstack-kolla | 11:36 | |
| mgoddard | kevko: another issue is log files. We set the /var/log/kolla/ owner to chrony | 11:39 |
| mgoddard | in the same JSON file you changed | 11:39 |
| openstackgerrit | Will Szumski proposed openstack/kayobe master: Add kolla_enable_zookeeper feature flag https://review.opendev.org/734064 | 11:41 |
| openstackgerrit | Merged openstack/kolla-ansible stable/ussuri: barbican: Use python3 plugin in uwsgi config https://review.opendev.org/733804 | 11:42 |
| openstackgerrit | Merged openstack/kolla-ansible master: Fix bug in deploying monasca_agent_forwarder https://review.opendev.org/733871 | 11:42 |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible stable/ussuri: Fix bug in deploying monasca_agent_forwarder https://review.opendev.org/734066 | 11:47 |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible stable/train: Fix bug in deploying monasca_agent_forwarder https://review.opendev.org/734067 | 11:48 |
| sorin-mihai | mgoddard, yoctozepto, much obliged for the help! once 'deploy' reached the point of checking if mariadb is available via haproxy, i removed 'option httpchk' from /etc/kolla/haproxy/services.d/mariadb.cfg, restarted haproxy and now the deployment seems to be going on fine with the other tasks, though i still don't see any mariadb-clustercheck container, no idea why | 11:51 |
| *** Luzi has joined #openstack-kolla | 11:53 | |
| kevko | mgoddard: well, i don't think we are understanding each other, i just said that default user/group in ubuntu/debian packages is _chrony:_chrony ... but kolla chrony container executing directly, check launchpad bug log ->> echo 'Running command: '\''/usr/sbin/chronyd -d -f /etc/chrony/chrony.conf'\''' ... so ..i think everything should be ok, but it is not | 12:01 |
| *** skramaja has quit IRC | 12:10 | |
| *** skramaja has joined #openstack-kolla | 12:10 | |
| Fl1nt | @yoctozepto, I've found a minor bug/hiccups with the way you introduced LOCALES on Train release for the base dockerfile. | 12:11 |
| *** wuchunyang has joined #openstack-kolla | 12:12 | |
| Fl1nt | additionally, @mgoddard where are you finding sqlalchemy-collectd package within: https://opendev.org/openstack/kolla/src/tag/9.1.0/docker/openstack-base/Dockerfile.j2#L88 <-- can't find it on any CentOS/RDO or EPEL repository. | 12:13 |
| openstackgerrit | Merged openstack/kayobe stable/stein: Remove newline from default public_net_name https://review.opendev.org/733833 | 12:13 |
| Fl1nt | which trigger a build image fail | 12:13 |
| *** rpittau|bbl is now known as rpittau | 12:13 | |
| openstackgerrit | Merged openstack/kayobe stable/train: Make the root disk image build command more visible https://review.opendev.org/733839 | 12:13 |
| Fl1nt | when using CentOS 8 as base image for Kolla build. | 12:13 |
| Fl1nt | with the 9.1.0 train tag | 12:13 |
| Fl1nt | @yoctozepto, what could fix our issue with the lang/locales issue with the base dockerfile would be to put it after the REPO files copy as you can't install the required packages without having copied those repo list when dealing with local internal PYPI mirrors. | 12:15 |
| hrw | Fl1nt: can you share logs? | 12:15 |
| Fl1nt | @hrw, for SQLAlchemy? | 12:15 |
| Fl1nt | or for the LANG things? | 12:16 |
| hrw | Fl1nt: when you have a failure using kolla it is good to share log showing it | 12:16 |
| Fl1nt | I know, sure I'll do my best, but the logs are pretty explicit indeed about that SQLAlchemy thing and for the LANG it's something that I actually solved with template-overrides.j2 but I would rather like to push this change as a patch. | 12:18 |
| Fl1nt | This platform being kind of "secured" I'll need few minutes to extract/redact the interesting part of logs | 12:19 |
| *** amoralej is now known as amoralej|lunch | 12:22 | |
| *** wuchunyang has quit IRC | 12:22 | |
| hrw | ok | 12:23 |
| Fl1nt | @hrw, ok, here are the logs and information that I can extract: http://paste.openstack.org/show/c8ezVYbcX5MByXds51wr/ | 12:25 |
| Fl1nt | so the point is | 12:25 |
| hrw | I am building master now to solve some issue with my patch | 12:26 |
| hrw | will look later at train | 12:26 |
| Fl1nt | none of the listed repos actually knowns about this SQLalchemy-collectd package, however cbs.centos.org is actually able to find a RPM package. | 12:26 |
| hrw | Fl1nt: please add enabling of opstools? | 12:26 |
| hrw | http://mirror.centos.org/centos/8/opstools/x86_64/collectd-5/Packages/p/python3-sqlalchemy-collectd-0.0.6-1.el8.noarch.rpm | 12:27 |
| Fl1nt | aaaaah here we are ^^ | 12:27 |
| Fl1nt | hence my first question to mark ^^ which repo is providing this package :D | 12:28 |
| Fl1nt | thanks a lot | 12:28 |
| Fl1nt | so | 12:28 |
| Fl1nt | this raise another question | 12:28 |
| Fl1nt | How do you know which repo is actually providing a package when you don't get this repo listed on your default CentOS machine? like, I mean, obviously a dnf info <package> won't work without the repodata being available ^^ | 12:29 |
| hrw | Fl1nt: I live by logs. so seeing logs often just switch some brain parts | 12:29 |
| hrw | Fl1nt: cbs, koji.mbox, going through mirror, asking on #centos-devel? | 12:29 |
| Fl1nt | unfortunately those logs doesn't complain about opstools missing metadata which is usually how I track them down ^^ | 12:30 |
| Fl1nt | so, anyway, I'll finish with my train tag build, and I'm really glad that it doesn't need a patch as it would spare me another template override ^^ | 12:31 |
| Fl1nt | Regarding the SYNC and LANG problematic block within the BASE image: https://opendev.org/openstack/kolla/src/tag/9.1.0/docker/base/Dockerfile.j2#L65 | 12:32 |
| Fl1nt | would you be OK to get a patch where we just move that block AFTER this repo file COPY For Loop ? https://opendev.org/openstack/kolla/src/tag/9.1.0/docker/base/Dockerfile.j2#L158 | 12:33 |
| hrw | Fl1nt: you can sneak your repos in dnf.conf file few lines above | 12:34 |
| Fl1nt | the problem being, when you're on an internal only environment, the Centos8 base image come with appropriate repo file but pointing to the internet and not internally which lead to the build to fail by not being able to retrieve the packages. | 12:34 |
| Fl1nt | what do you mean by sneak your repos? | 12:35 |
| Fl1nt | using dnf.conf ? | 12:35 |
| hrw | yes | 12:35 |
| Fl1nt | It's cleaner to not touch dnf.conf and just override the base_yum_repo_files variable. | 12:35 |
| hrw | dnf does not care much where repo info is written | 12:35 |
| hrw | so you can have own ones in dnf.conf | 12:36 |
| Fl1nt | When I say cleaner, I mean for Ops related stuff, it's quicker to know which repo is override when using the template-overrides.j2 file. | 12:37 |
| Fl1nt | so for instance, for now, I've made my repo list to copy our internal ones and then emptied the sync/lang variable and move the whole block within the post copy block. https://opendev.org/openstack/kolla/src/tag/9.1.0/docker/base/Dockerfile.j2#L172 | 12:39 |
| Fl1nt | this doesn't conflict with anything from a standard way of things as it's still the first packages that you install but you let ops and admins to overrides initial repo files without cluttering the dnf conf file. | 12:40 |
| Fl1nt | it's help a lot when debugging things :D | 12:40 |
| hrw | ok, so again - what is the problem with locales? | 12:40 |
| sorin-mihai | mgoddard, yoctozepto, deployment finished, no erros, but the mariadb-clustercheck container was not deployed. master git branch with locally built images from master also | 12:44 |
| *** seco has quit IRC | 12:45 | |
| *** born2bake has quit IRC | 12:54 | |
| Fl1nt | @hrw, sorry, I'm switching between that desk and the laptop. So, the issue is | 12:54 |
| Fl1nt | when you're building a platform, that have to rely on an internal mirror for everything | 12:55 |
| Fl1nt | you'll be able to do the FROM Centos-8 part of the build as it will pull your official CentOS image replica from your internal docker registry | 12:56 |
| Fl1nt | BUT | 12:56 |
| *** bengates has quit IRC | 12:56 | |
| Fl1nt | that official image isn't modified and come with mirrorlist and/or baseurl pointing to the internet endpoint. | 12:56 |
| *** bengates has joined #openstack-kolla | 12:56 | |
| Fl1nt | you then need to override the default CentOS-<RepoName>.repo file using the templates-override.j2 | 12:57 |
| Fl1nt | in order to make this image point to the correct internal mirrors. | 12:57 |
| Fl1nt | the thing is | 12:57 |
| hrw | Fl1nt: override base_dnf_conf block with adding own repos/ | 12:57 |
| hrw | ? | 12:57 |
| hrw | Fl1nt: this way you can configure own repos before any package attempt is done | 12:58 |
| hrw | Fl1nt: sure, later you have to do it again but that should solve your problem | 12:58 |
| openstackgerrit | Pierre Riteau proposed openstack/kolla-ansible stable/queens: CI: Fix installation of legacy Docker packages on CentOS https://review.opendev.org/733882 | 12:58 |
| Fl1nt | you don't get this block within 9.1.0 tags or train branch | 12:58 |
| hrw | Fl1nt: or make centos-modified:8 image and use it as --base-image | 12:59 |
| *** born2bake has joined #openstack-kolla | 12:59 | |
| hrw | Fl1nt: then base_yum_conf which is same | 12:59 |
| Fl1nt | nope, you don't do that within proper ops, it will mess your infrastructure as you're diverging from vanilla and people will be inclined to add more and more stuff because why not. | 12:59 |
| *** bsanjeewa has joined #openstack-kolla | 13:00 | |
| hrw | sure | 13:00 |
| hrw | Fl1nt: so base_yum_conf override | 13:00 |
| Fl1nt | yeah, but then you're doing it twice, once at base_yum_conf level, which is not really clean as it's name conf, not repo and so you're mixing things, and additionally there is another block dedicated to that purpose already. | 13:01 |
| *** bsanjeewa_ has quit IRC | 13:01 | |
| Fl1nt | moving the lang block under that block is far cleaner and without any consequances as it still install the lang package as the first packages. | 13:01 |
| Fl1nt | moving one block only is way more easier/cleaner than having to put your custom repos at two different levels. | 13:02 |
| hrw | Fl1nt: so please send us patch for master to move that block. we discuss, merge and then we may consider it for backports | 13:02 |
| Fl1nt | perfect! | 13:03 |
| Fl1nt | should I start to fill a bug report on launchpad or directly make the patch and send a pull request? | 13:03 |
| hrw | Fl1nt: note that you can do your repos once even now | 13:03 |
| hrw | bug + patch would be best | 13:04 |
| Fl1nt | ok, perfect | 13:04 |
| Fl1nt | what do you mean? | 13:04 |
| hrw | no PR - we use 'git review' | 13:04 |
| Fl1nt | yeah, I was stuck at that level the other day. | 13:04 |
| hrw | Fl1nt: base_yum_conf adds your repos as 'flint_*.repo' files and then other override removes all kolla provided ones | 13:05 |
| hrw | Fl1nt: you can even put your repo files in /etc/dnf/ and then just remove all in /etc/yum.repos.d/ | 13:05 |
| Fl1nt | still two step instead of one and still a split responsability issue tho. | 13:05 |
| hrw | sure, but until your patch gets into train some time will pass | 13:06 |
| Fl1nt | for now on, I don't really get an issue, as I've already done it using template-overrides.j2, that's an acceptable solution btw, just that I think we can definitely improve something for people not being able to reach internet while not broking anything else for anyone else ^^ | 13:07 |
| hrw | sure | 13:08 |
| Fl1nt | I've got a more problematic additional patch to submit once this one finished as CentOS decided few times ago that it would be a good idea to embed GPG Keys within packages rather than having them within the repository itself. | 13:08 |
| hrw | for centos-release-* packages it is quite popular way | 13:10 |
| hrw | we import those keys | 13:10 |
| Fl1nt | yeah I know, I'm really like that when I'm on my personal lab that get access to internet (or at least the proxy), but once again, when doing a internet-less installation, that suxx as it override your repo files once again and create an issue. | 13:11 |
| *** ykarel is now known as ykarel|afk | 13:11 | |
| Fl1nt | I'll scratch my head to find a solution to that one but honnestly I think that corporations that just sync the mirror without introspecting it and modifying the repo files at fly will be screwed and required to either disable gpg check | 13:13 |
| Fl1nt | or get a two phase installation | 13:13 |
| mgoddard | sorin-mihai: maybe you could provide logs of the mariadb role deploy? | 13:13 |
| Fl1nt | 1°/- Install the package, validate/import the GPG Key from etc/pki/ | 13:14 |
| *** amoralej|lunch is now known as amoralej | 13:14 | |
| Fl1nt | 2°/ - Override the installed CentOS-*.repo files | 13:14 |
| Fl1nt | @amoralej, hey! didn't forget you, I'm just in a rush corridor, I'll update the RDO request as soon as possible to let it go until Barbican can actually use the release-cycle validation. | 13:15 |
| amoralej | Fl1nt, ok, no problem | 13:16 |
| sorin-mihai | mgoddard, you mean output with increased verbosity, or is there some log file for 'deploy'? | 13:16 |
| mgoddard | no verbosity is fine. Just want to see if the task to deploy clustercheck ran | 13:17 |
| hrw | Fl1nt: you sync whole mirror. and you install packages from it. so where is problem with keys? | 13:17 |
| hrw | Fl1nt: note that you can even sed out mirrorlist= lines and then use sed to enable baseurl=http://your.own.mirror.of.centos.internal/ like we do on CI | 13:18 |
| *** Torel has quit IRC | 13:18 | |
| hrw | Fl1nt: look in tests/templates/template_overrides.j2 base_centos_repo_overrides_post_yum block | 13:20 |
| openstackgerrit | Mark Goddard proposed openstack/kayobe master: DNM: use cirros for seed VM https://review.opendev.org/734050 | 13:23 |
| *** Torel has joined #openstack-kolla | 13:25 | |
| Fl1nt | @hrw, this part isn't the issue ^^ the problem is our mirror. | 13:30 |
| sorin-mihai | mgoddard, http://paste.openstack.org/raw/6sFn8wqketjkz0s4ZFYT/ | 13:31 |
| Fl1nt | base_centos_repo_overrides_post_yum <-- You'll have the chicken and egg issue with this one unfortunately ^^ | 13:31 |
| *** Torel has quit IRC | 13:32 | |
| hrw | Fl1nt: so fix the mirror? :D | 13:32 |
| yoctozepto | mgoddard, sorin-mihai: odd, it looks like it templated out haproxy for this but does not have the necessary tasks, confusing | 13:32 |
| *** Torel has joined #openstack-kolla | 13:33 | |
| Fl1nt | Unfortunately I can't and none of us can, this is a security rules/policy issue, it's easier to create a patch for kolla to be able to working within both online and offline installation ^^ | 13:33 |
| hrw | D: | 13:34 |
| hrw | ok, cores - anyone for my simple patches? https://review.opendev.org/731487 https://review.opendev.org/730331 | 13:34 |
| patchbot | patch 731487 - kolla-ansible - CI: drop pre-ussuri upgrade support - 3 patch sets | 13:34 |
| patchbot | patch 730331 - kolla - CI: use wheels from infra mirror - 6 patch sets | 13:34 |
| Fl1nt | Oh, and there is a third patch that i can contribute too regarding the way we install/upgrade pip on RPM related stuff. | 13:36 |
| sorin-mihai | mgoddard, i don't think any of the changes i've done in my local git branch should influence this. currently none of those changes should affect anything, those components are not enabled in globals.yml http://paste.openstack.org/raw/j0exxHdRi10vSePreL3h/ | 13:36 |
| openstackgerrit | Mark Goddard proposed openstack/kayobe master: WIP: Add seed VM provisioning CI job https://review.opendev.org/559706 | 13:36 |
| hrw | Fl1nt: good! | 13:36 |
| openstackgerrit | Mark Goddard proposed openstack/kayobe master: DNM: use cirros for seed VM https://review.opendev.org/734050 | 13:36 |
| hrw | Fl1nt: this is how all those quirks and hacks gets sorted out. you make them, look at them, clean them and send upstream as you find them interesting. | 13:37 |
| sorin-mihai | mgoddard, and this would by the diff between my current kolla-build.conf and the original one generated in master branch http://paste.openstack.org/raw/g1RBtttIV2gR5kMu892q/ | 13:40 |
| mgoddard | sorin-mihai: I see one problem which might affect you. I'll propose a patch | 13:40 |
| Fl1nt | yep, I always try to contribute back when I can, unfortunately companies doesn't want it all the time, I'm fortunately enough to currently work with a company that is actually sponsoring that and is looking to better contribute to the community. | 13:41 |
| sorin-mihai | mgoddard, i hope it's just me. lol | 13:41 |
| *** Luzi has quit IRC | 13:41 | |
| mgoddard | Fl1nt: we basically make it part of our way of working - upstream first, downstream only in exceptions. Customers are generally on board with it. | 13:42 |
| *** zijlboot_ has joined #openstack-kolla | 13:43 | |
| *** TrevorV has joined #openstack-kolla | 13:43 | |
| Fl1nt | You're glad to get it that way, I'm sometimes dealing with people that can't really explain why they doesn't want it. so usually I or one of my team mate contribute back as an individual. | 13:43 |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible master: Fix handler notification for mariadb-clustercheck https://review.opendev.org/734090 | 13:43 |
| mgoddard | Fl1nt: easier if it's not a question. This is how we do things | 13:44 |
| hrw | Fl1nt: I remember cooperation with one such company. I had one line patch to some project and knew that developer next to me had to patch that project similar way. I made a patch, shown it to him with "have you had to do such one?" question. Answer was 'kind of yeah'. | 13:44 |
| mgoddard | sorin-mihai: https://review.opendev.org/734090 | 13:45 |
| patchbot | patch 734090 - kolla-ansible - Fix handler notification for mariadb-clustercheck - 1 patch set | 13:45 |
| sorin-mihai | mgoddard, yeah, already applied and testing | 13:45 |
| Fl1nt | As I now get my own company is really more easier to contribute back either with code or sponsoring, it was sooo difficult to get Big N companies to play it fair... It just exhausted me ^^ | 13:45 |
| mgoddard | sorin-mihai: cool. I'm not sure why other tasks don't trigger it though. Are you running kolla-ansible genconfig? | 13:45 |
| Fl1nt | @hrw, yeah, that kind of non sense reactions ^^ | 13:46 |
| mgoddard | Or maybe it failed once then you tried again | 13:46 |
| sorin-mihai | mgoddard, no, actually i don't think i've ever run that previously | 13:46 |
| *** zijlboot has quit IRC | 13:47 | |
| Fl1nt | Fixed the OpsTools repo issue, currently building, cross-finger fellows ^^ | 13:47 |
| hrw | Fl1nt: can you check is that opstools issue is related to your config or to train? | 13:52 |
| hrw | mgoddard: tripleo should now be in sync with kolla on ussuri - my tripleo patch got +W | 13:53 |
| mgoddard | hrw: nice job | 13:55 |
| *** bsanjeewa_ has joined #openstack-kolla | 13:55 | |
| mgoddard | ok, pip question | 13:55 |
| mgoddard | pip2 install -c https://opendev.org/openstack/requirements/raw/branch/stable/train/upper-constraints.txt 'diskimage-builder<2.30' | 13:55 |
| mgoddard | this gives me 2.30, which is obviously outside the version constraint | 13:55 |
| mgoddard | why? | 13:56 |
| sorin-mihai | mgoddard, http://paste.openstack.org/raw/JVrTnm5lgWimcGYmGqMV/ no error, mariadb_clustercheck container is present in docker ps | 13:56 |
| Fl1nt | @hrw, For now the build passed the image that missed the package so it was on my side of things. | 13:56 |
| hrw | mgoddard: u-c says ===2.30? | 13:57 |
| mgoddard | hrw: yes | 13:57 |
| hrw | mgoddard: probably has higher priority | 13:57 |
| Fl1nt | but still I'll fill a bug for that LANG thing and few patches for other things. | 13:57 |
| mgoddard | hrw: it's a constraint, not a req | 13:57 |
| hrw | Fl1nt: this is the way | 13:58 |
| hrw | mgoddard: u-c is also a way to say 'whatever, I want you to use THOSE versions' | 13:58 |
| *** bsanjeewa has quit IRC | 13:59 | |
| hrw | at least this is how I understand u-c | 13:59 |
| *** zijlboot_ has quit IRC | 14:02 | |
| *** zijlboot has joined #openstack-kolla | 14:03 | |
| *** priteau has joined #openstack-kolla | 14:05 | |
| *** ykarel|afk is now known as ykarel | 14:09 | |
| mgoddard | this one needs some love to unblock stable CI: https://review.opendev.org/733557 | 14:10 |
| patchbot | patch 733557 - kolla (stable/train) - Fix multiple issues - 2 patch sets | 14:10 |
| mgoddard | hrw, yoctozepto : ^ | 14:11 |
| *** diurnalist has joined #openstack-kolla | 14:11 | |
| hrw | mgoddard: please mark 'train only' | 14:12 |
| mgoddard | hrw: then stein will stay broken :) | 14:12 |
| hrw | mgoddard: in gneeral: add note which branches | 14:12 |
| hrw | mgoddard: stein will move to bifrost 7.2.1 too? | 14:14 |
| hrw | +1 | 14:15 |
| mgoddard | no, but it has a release | 14:17 |
| hrw | so it will be separate but similar patch. send it for review then | 14:19 |
| yoctozepto | mgoddard, sorin-mihai: I bet it is the 'failed before handler triggered' case | 14:21 |
| yoctozepto | + we missed the other trigger | 14:21 |
| yoctozepto | which is an obvious bug ;-) | 14:21 |
| Fl1nt | By the way @hrw you're pretty ARM oriented aren't you? | 14:21 |
| yoctozepto | mgoddard: are we giving up on fix renos? | 14:21 |
| yoctozepto | mgoddard: more trouble than they are worth? | 14:21 |
| hrw | Fl1nt: working on ARM since 2004, AArch64 since 2012 | 14:23 |
| hrw | Fl1nt: why ask? | 14:23 |
| Fl1nt | Cool, I've a small side project involving Raspberry PI and Openstack :p | 14:23 |
| hrw | Fl1nt: gt.o with it :D | 14:24 |
| Fl1nt | gt.o ?? | 14:24 |
| hrw | Fl1nt: r/pi is piece of shit | 14:25 |
| hrw | and I am as politically correct as I can | 14:25 |
| Fl1nt | I'm building a micro-server (1U form factor) that will have few RPi used to host a full control plan ^^ | 14:25 |
| Fl1nt | Already got 5 RPI 4B 8gb it's pretty cool ^^ | 14:25 |
| hrw | Fl1nt: sorry, I do not support rpi | 14:25 |
| Fl1nt | just received them two days ago :p | 14:26 |
| hrw | Fl1nt: sorry, I do not support rpi | 14:26 |
| Fl1nt | Yeah don't worry, let's say I'm working on AArch64 platform :p | 14:26 |
| Fl1nt | So the question is rather, did you had any success running a vanilla cluster? | 14:27 |
| Fl1nt | at least the control plan part | 14:27 |
| Fl1nt | on ARM AARCH64? | 14:27 |
| hrw | Fl1nt: openstack infra has aarch64 nodes running on kolla deployed cloud | 14:28 |
| hrw | Fl1nt: on normal aarch64 servers | 14:28 |
| Fl1nt | cool, so it means at least it works :D | 14:28 |
| hrw | at least? | 14:28 |
| hrw | at least? | 14:28 |
| hrw | trying to offend you are | 14:29 |
| Fl1nt | joke aside, what don't you like with rpi ? the GPU limbo? | 14:29 |
| hrw | D: | 14:29 |
| Fl1nt | I'm feeling offended by you being offended on such a remark good sir :p | 14:29 |
| openstackgerrit | Mark Goddard proposed openstack/kayobe master: WIP: Add seed VM provisioning CI job https://review.opendev.org/559706 | 14:29 |
| openstackgerrit | Mark Goddard proposed openstack/kayobe master: DNM: use cirros for seed VM https://review.opendev.org/734050 | 14:29 |
| hrw | Fl1nt: one day raspberrypi foundation will learn how to release and support their hw. in 2060 or later | 14:29 |
| hrw | Fl1nt: until then I just prefer to not talk about it on public channels. nor on privs | 14:30 |
| mgoddard | yoctozepto: I don't add renos if the feature hasn't been released | 14:30 |
| Fl1nt | yeah, well, as long as I've got a working DTS/DTB and a blob to support the GPU part, it's fine for me :D | 14:30 |
| yoctozepto | mgoddard: I see a more general trend | 14:30 |
| yoctozepto | mgoddard: but that's fine by me, we can stick to hot ones, or huge ones ;D | 14:31 |
| mgoddard | yoctozepto: oh. I'm probably inconsistent about it | 14:31 |
| mgoddard | I don't generally push it too hard for new contributors with small fixes | 14:31 |
| mgoddard | yoctozepto: FYI mnasiadka is away this week | 14:33 |
| hrw | Fl1nt: I do not like wasting time on hardware which is not supported by mainline kernel | 14:33 |
| yoctozepto | mgoddard: ah, right, I forgot | 14:33 |
| Fl1nt | so just to share with you @hrw I'm working to get that 1U micro-server things, as it's a hobby/side project, I'm building everything from scratch, HW/SW, that will be a good way to gain skills on many things that I'm interested by, especially the ARM part, any suggestion about the HW, like, any good company that I could request for building a custom board? | 14:33 |
| hrw | Fl1nt: but this is not a channel for this discussion | 14:33 |
| yoctozepto | mgoddard: sure, I agree regarding renos | 14:33 |
| Fl1nt | @hrw, hum... BCM2711 is actually supported on mainline. | 14:34 |
| yoctozepto | mgoddard: it felt like too much trouble to get otherwise fine fixes in | 14:34 |
| mgoddard | yoctozepto: I do like having fixes for renos, but the bar for contribution is already quite high | 14:34 |
| yoctozepto | mgoddard: or renos for fixes | 14:34 |
| mgoddard | well both really | 14:34 |
| *** TrevorV has quit IRC | 14:36 | |
| *** TrevorV has joined #openstack-kolla | 14:37 | |
| *** wuchunyang has joined #openstack-kolla | 14:39 | |
| *** abdysn has quit IRC | 14:43 | |
| *** k_mouza has quit IRC | 14:44 | |
| openstackgerrit | Mark Goddard proposed openstack/kayobe master: DNM: use cirros for seed VM https://review.opendev.org/734050 | 14:49 |
| *** Torel has quit IRC | 14:49 | |
| openstackgerrit | Mark Goddard proposed openstack/kolla stable/train: Fix multiple issues https://review.opendev.org/733557 | 14:52 |
| *** Torel has joined #openstack-kolla | 14:52 | |
| hrw | +2 | 14:54 |
| openstackgerrit | Mark Goddard proposed openstack/kayobe master: WIP: Add seed VM provisioning CI job https://review.opendev.org/559706 | 14:55 |
| openstackgerrit | Mark Goddard proposed openstack/kayobe master: DNM: use cirros for seed VM https://review.opendev.org/734050 | 14:55 |
| *** k_mouza has joined #openstack-kolla | 14:56 | |
| *** nikparasyr has quit IRC | 15:01 | |
| kevko | mgoddard: here? | 15:04 |
| mgoddard | kevko: sort of | 15:05 |
| *** skramaja has quit IRC | 15:06 | |
| kevko | mgoddard: still debugging chrony debian image ... it's weird , look -> http://paste.openstack.org/show/794478/ | 15:07 |
| mgoddard | kevko: oh dear, strace comes out :) | 15:08 |
| kevko | mgoddard: from strace it's look like root is going to open /etc/chrony/chrony.conf ..but get permission denied ... if i change /etc/chrony/chrony.conf to root:root , it is working | 15:08 |
| mgoddard | kevko: could be apparmor? the chrony container uses the root user | 15:09 |
| kevko | mgoddard: somehow it is running as root ..but don't have permissions ... | 15:11 |
| kevko | mgoddard: http://paste.openstack.org/show/794480/ | 15:11 |
| kevko | mgoddard: if i change root:root /etc/chrony/chrony.conf ... chrony has another issue with /var/log/kolla/chrony/tracking.log of course | 15:12 |
| *** Torel has quit IRC | 15:12 | |
| mgoddard | kevko: is apparmor or selinux enabled? | 15:12 |
| *** bsanjeewa has joined #openstack-kolla | 15:12 | |
| *** bsanjeewa_ has quit IRC | 15:14 | |
| *** Torel has joined #openstack-kolla | 15:14 | |
| kevko | mgoddard: nope | 15:14 |
| mgoddard | kevko: that's the usual reason why root would not be able to do something | 15:15 |
| kevko | mgoddard: no apparmor installed | 15:15 |
| mgoddard | kevko: what user ID does the chrony process have? maybe it drops privileges and switches user? | 15:15 |
| mgoddard | might be so it can listen on a low port number as root, then switch to chrony/_chrony user | 15:16 |
| kevko | mgoddard: if i change config.json from 0600 to 0644 i can see that process is runnning under chrony as it should ... | 15:16 |
| kevko | mgoddard: if i leave 0600 in config ..i can't exec to container because still restarting :/ | 15:16 |
| mgoddard | kevko: does it work if you replace all instances of chrony user with _chrony? | 15:19 |
| kevko | mgoddard: nope | 15:20 |
| mgoddard | weird | 15:21 |
| *** Gtv has quit IRC | 15:21 | |
| kevko | yes :D | 15:21 |
| *** Torel has quit IRC | 15:22 | |
| kevko | mgoddard: work if i replace chrony user in config.json for /etc/chrony/chrony.conf by root user and leave 0600 as is | 15:22 |
| *** priteau has quit IRC | 15:22 | |
| *** Torel has joined #openstack-kolla | 15:24 | |
| *** bsanjeewa has quit IRC | 15:26 | |
| *** k_mouza has quit IRC | 15:28 | |
| *** k_mouza has joined #openstack-kolla | 15:28 | |
| *** k_mouza has quit IRC | 15:29 | |
| *** k_mouza has joined #openstack-kolla | 15:29 | |
| *** noxoid has joined #openstack-kolla | 15:30 | |
| *** bengates_ has joined #openstack-kolla | 15:34 | |
| *** bengates has quit IRC | 15:34 | |
| openstackgerrit | Mark Goddard proposed openstack/kayobe master: DNM: use cirros for seed VM https://review.opendev.org/734050 | 15:39 |
| openstackgerrit | Mark Goddard proposed openstack/kolla stable/stein: Fix multiple issues https://review.opendev.org/734131 | 15:42 |
| hrw | +2 | 15:43 |
| openstackgerrit | Merged openstack/kolla-ansible stable/ussuri: Fix bug in deploying monasca_agent_forwarder https://review.opendev.org/734066 | 15:50 |
| *** ykarel is now known as ykarel|away | 15:51 | |
| Fl1nt | @mgoddard, I may have spotted an issue with the tempest role on k-a too ^^ | 15:52 |
| Fl1nt | so | 15:52 |
| Fl1nt | if we want to be on par with what we do on pretty much all services | 15:53 |
| Fl1nt | we should fix that task: https://opendev.org/openstack/kolla-ansible/src/tag/9.1.0/ansible/roles/tempest/tasks/config.yml#L28https://opendev.org/openstack/kolla-ansible/src/tag/9.1.0/ansible/roles/tempest/tasks/config.yml#L28 | 15:53 |
| Fl1nt | with something similar to: https://opendev.org/openstack/kolla-ansible/src/tag/9.1.0/ansible/roles/nova/tasks/config.yml#L45 | 15:54 |
| Fl1nt | just stumble upon the issue | 15:54 |
| *** Torel has quit IRC | 15:54 | |
| Fl1nt | where each services of my node_custom_config get a proper folder | 15:54 |
| Fl1nt | with its service.conf in it | 15:54 |
| Fl1nt | but that didn't worked for tempest ^^ | 15:54 |
| Fl1nt | I know why now ^^ | 15:55 |
| Fl1nt | so, if we want things to behave a little more consistently, I would go for a small patch on that too ^^ | 15:55 |
| *** Torel has joined #openstack-kolla | 15:55 | |
| *** k_mouza has quit IRC | 16:02 | |
| *** k_mouza has joined #openstack-kolla | 16:03 | |
| Fl1nt | We're OK that Train is still open for backport patches? | 16:05 |
| Fl1nt | like, RabbitMQ Server is now available directly from CentOS storage using 3.8 series, and it work, just tested it on my installation, tempest/rally tested all PASSED. | 16:06 |
| *** rpittau is now known as rpittau|afk | 16:08 | |
| *** wuchunyang has quit IRC | 16:09 | |
| *** jonaspaulo has joined #openstack-kolla | 16:10 | |
| *** k_mouza has quit IRC | 16:14 | |
| openstackgerrit | Christian Berendt proposed openstack/kolla-ansible master: zun: create /opt/cni/bin directory https://review.opendev.org/734135 | 16:14 |
| *** k_mouza has joined #openstack-kolla | 16:14 | |
| *** jonaspaulo has quit IRC | 16:25 | |
| *** k_mouza has quit IRC | 16:37 | |
| lvdombrkr | hello all. i have changed my compute node hostname but on docker containers hostname is still old one. any clues how to change hostname on containers ? | 16:39 |
| *** k_mouza has joined #openstack-kolla | 16:41 | |
| lvdombrkr | i mean to change permanently | 16:42 |
| openstackgerrit | Merged openstack/kolla-ansible stable/train: Fix bug in deploying monasca_agent_forwarder https://review.opendev.org/734067 | 16:47 |
| openstackgerrit | Merged openstack/kolla master: Switch to newer openstackdocstheme and reno versions https://review.opendev.org/729644 | 16:47 |
| *** cah_link has quit IRC | 16:48 | |
| *** k_mouza has quit IRC | 16:52 | |
| *** amoralej is now known as amoralej|lunch | 16:52 | |
| *** amoralej|lunch is now known as amoralej|off | 16:52 | |
| *** k_mouza has joined #openstack-kolla | 16:56 | |
| *** dking has joined #openstack-kolla | 16:56 | |
| Fl1nt | @mgoddard, when Train release was created, it was in between CentOS7 and CentOS8 release isn't ? | 17:03 |
| mgoddard | Fl1nt: Train was initially released for C7. C8 support has since been added | 17:04 |
| Fl1nt | ok, that explain things ^^ thanks | 17:04 |
| dking | From what I understand, Train is the transition period between 7 and 8, because both are supposed to be supported, but 7 is no longer supported after it. | 17:05 |
| dking | I've come across some issues trying to use CentOS 8 on Train so, I'm wondering if all the issues have been ironed out yet. | 17:06 |
| Fl1nt | so, I'll request my customer for dedicated days for contributing back as there are a lot of fix to be made on train to improve it especially around package topic, I'll check if some can already be just backported from master. | 17:06 |
| openstackgerrit | Mark Goddard proposed openstack/kayobe master: WIP: Add seed VM provisioning CI job https://review.opendev.org/559706 | 17:08 |
| openstackgerrit | Mark Goddard proposed openstack/kayobe master: DNM: use cirros for seed VM https://review.opendev.org/734050 | 17:08 |
| mgoddard | Fl1nt: we don't generally upgrade things like rabbitmq after release | 17:09 |
| mgoddard | at least not minor upgrades | 17:09 |
| Fl1nt | Even if they pass the CICD gates etc? | 17:09 |
| mgoddard | dking: no doubt there will be issues, it was a big transition. Bug reports & fixes appreciated | 17:10 |
| mgoddard | Fl1nt: well that is a requirement for any patch :) | 17:10 |
| mgoddard | but generally stable means stable | 17:10 |
| mgoddard | (in terms of branch names) | 17:10 |
| mgoddard | anyway, off now o/ | 17:11 |
| Fl1nt | ok, so I should rather work on master | 17:11 |
| Fl1nt | see ya \o | 17:11 |
| dking | Thank you. That last one really helped. However, immediately following it, I had an issue building images. I wasn't able to finish tracking down the issue, though, before it seemed to just resolve itself. | 17:15 |
| *** dougsz has quit IRC | 17:15 | |
| *** vishalmanchanda has quit IRC | 17:20 | |
| *** bsanjeewa has joined #openstack-kolla | 17:24 | |
| *** k_mouza has quit IRC | 17:28 | |
| *** bengates has joined #openstack-kolla | 17:28 | |
| Fl1nt | ok, time to go too, see ya all! | 17:28 |
| *** Fl1nt has quit IRC | 17:29 | |
| *** bengates_ has quit IRC | 17:32 | |
| *** headphoneJames has joined #openstack-kolla | 17:32 | |
| *** headphoneJames has left #openstack-kolla | 17:59 | |
| *** headphoneJames has joined #openstack-kolla | 18:00 | |
| *** headphoneJames is now known as generalfuzz | 18:14 | |
| *** bengates has quit IRC | 18:18 | |
| *** generalfuzz is now known as headphoneJames | 18:18 | |
| *** headphoneJames has left #openstack-kolla | 18:20 | |
| *** headphoneJames has joined #openstack-kolla | 18:20 | |
| *** headphoneJames has left #openstack-kolla | 18:20 | |
| *** generalfuzz has joined #openstack-kolla | 18:21 | |
| *** k_mouza has joined #openstack-kolla | 18:29 | |
| *** gfidente is now known as gfidente|afk | 18:32 | |
| *** k_mouza has quit IRC | 18:33 | |
| openstackgerrit | Merged openstack/kolla stable/train: Fix multiple issues https://review.opendev.org/733557 | 18:38 |
| *** k_mouza has joined #openstack-kolla | 18:38 | |
| *** generalfuzz is now known as headphoneJames | 18:49 | |
| *** muhaha has joined #openstack-kolla | 18:51 | |
| *** k_mouza has quit IRC | 18:51 | |
| stackedsax | headphoneJames: just making sure you got back in here after the net split | 18:52 |
| lvdombrkr | guys, any clues how to change running container hostname? i changed hostname on my compute node, but on contaiers still old one | 18:52 |
| bsanjeewa | Hello, could anyone please tell me what is the alternative for ntp_config_server: in ntp.yml when using time.yml. Would it be time_config_server: ? | 18:52 |
| *** nathharp_ has joined #openstack-kolla | 18:55 | |
| *** nathharp has quit IRC | 18:58 | |
| openstackgerrit | Merged openstack/kolla stable/ussuri: [docs] Ussuri - Ubuntu has Ceph Octopus https://review.opendev.org/733830 | 19:04 |
| *** dougsz has joined #openstack-kolla | 19:08 | |
| openstackgerrit | Christian Berendt proposed openstack/kolla-ansible master: Add 'ovn' to list of valid values of neutron_plugin_agent https://review.opendev.org/734174 | 19:12 |
| *** bsanjeewa has quit IRC | 19:22 | |
| *** bornie2bake has joined #openstack-kolla | 19:24 | |
| *** Torel has quit IRC | 19:28 | |
| *** Torel has joined #openstack-kolla | 19:28 | |
| openstackgerrit | Christian Berendt proposed openstack/kolla-ansible master: Remove XenAPI integration https://review.opendev.org/734180 | 19:31 |
| *** Torel has quit IRC | 19:49 | |
| *** Torel has joined #openstack-kolla | 19:50 | |
| openstackgerrit | Merged openstack/kolla-ansible master: Fix handler notification for mariadb-clustercheck https://review.opendev.org/734090 | 20:24 |
| openstackgerrit | Christian Berendt proposed openstack/kolla-ansible master: Fix wrong value for neutron_ovn_metadata_agent_image https://review.opendev.org/734224 | 20:24 |
| *** bornie2bake has quit IRC | 20:25 | |
| *** jbadiapa has quit IRC | 20:32 | |
| *** muhaha has quit IRC | 20:40 | |
| kplant | what is the correct procedure for adding compute nodes to an existing cluster? | 20:41 |
| kplant | i assumed it was edit inventory, ka bootstrap-servers --limit <compute_hosts>, ka deploy --limit <compute_hosts> | 20:41 |
| kplant | i've done that but now have 10 compute nodes that aren't showing in nova | 20:42 |
| kplant | a did a full reconfigure with no '--limit' | 20:47 |
| kplant | hmm, i think it's an /etc/hosts problem | 20:47 |
| *** k_mouza has joined #openstack-kolla | 20:52 | |
| *** k_mouza has quit IRC | 20:56 | |
| kplant | kolla-ansible reconfigure//deploy don't update the /etc/hosts file | 21:08 |
| kplant | do i have to do this manually? | 21:08 |
| *** dougsz has quit IRC | 21:09 | |
| *** born2bake has quit IRC | 21:10 | |
| *** Torel has quit IRC | 21:13 | |
| *** Torel has joined #openstack-kolla | 21:14 | |
| *** lvdombrkr has quit IRC | 21:17 | |
| *** rlljorge has joined #openstack-kolla | 21:43 | |
| *** rlljorge has left #openstack-kolla | 21:43 | |
| *** rlljorge has joined #openstack-kolla | 21:44 | |
| *** rlljorge has quit IRC | 21:45 | |
| *** Torel has quit IRC | 21:54 | |
| *** Torel has joined #openstack-kolla | 21:57 | |
| *** Torel has quit IRC | 22:04 | |
| *** Torel has joined #openstack-kolla | 22:05 | |
| *** threestrands has joined #openstack-kolla | 22:15 | |
| *** threestrands has quit IRC | 22:16 | |
| *** threestrands has joined #openstack-kolla | 22:17 | |
| *** threestrands has quit IRC | 22:18 | |
| *** threestrands has joined #openstack-kolla | 22:18 | |
| *** threestrands has quit IRC | 22:19 | |
| *** threestrands has joined #openstack-kolla | 22:20 | |
| *** threestrands has quit IRC | 22:21 | |
| *** threestrands has joined #openstack-kolla | 22:21 | |
| *** threestrands has quit IRC | 22:22 | |
| *** threestrands has joined #openstack-kolla | 22:23 | |
| *** threestrands has quit IRC | 22:24 | |
| *** threestrands has joined #openstack-kolla | 22:24 | |
| *** threestrands has quit IRC | 22:25 | |
| *** threestrands has joined #openstack-kolla | 22:26 | |
| *** threestrands has quit IRC | 22:27 | |
| *** threestrands has joined #openstack-kolla | 22:27 | |
| *** threestrands has quit IRC | 22:28 | |
| *** threestrands has joined #openstack-kolla | 22:29 | |
| *** threestrands has quit IRC | 22:30 | |
| *** threestrands has joined #openstack-kolla | 22:30 | |
| *** Torel has quit IRC | 22:48 | |
| *** Torel has joined #openstack-kolla | 22:50 | |
| *** nathharp has joined #openstack-kolla | 23:06 | |
| *** nathharp_ has quit IRC | 23:08 | |
| *** Torel has quit IRC | 23:08 | |
| *** Torel has joined #openstack-kolla | 23:09 | |
| *** dmellado_ has joined #openstack-kolla | 23:10 | |
| *** dmellado has quit IRC | 23:11 | |
| *** dmellado_ is now known as dmellado | 23:11 | |
| *** TrevorV has quit IRC | 23:19 | |
| *** Torel has quit IRC | 23:29 | |
| *** Torel has joined #openstack-kolla | 23:33 | |
| *** dciabrin has quit IRC | 23:49 | |
| *** dciabrin has joined #openstack-kolla | 23:50 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!