opendevreview | James Kirsch proposed openstack/kolla master: Add LetsEncrypt images for cert request/renewal https://review.opendev.org/c/openstack/kolla/+/741339 | 00:17 |
---|---|---|
opendevreview | James Kirsch proposed openstack/kolla master: Add LetsEncrypt images for cert request/renewal https://review.opendev.org/c/openstack/kolla/+/741339 | 00:21 |
opendevreview | Verification of a change to openstack/kayobe stable/wallaby failed: ntp: Fix service mask when service doesn't exist https://review.opendev.org/c/openstack/kayobe/+/827403 | 01:25 |
opendevreview | James Kirsch proposed openstack/kolla master: Add LetsEncrypt images for cert request/renewal https://review.opendev.org/c/openstack/kolla/+/741339 | 03:51 |
shyamb | Hi | 05:42 |
shyamb | As we are stopping centos8 builds, are we going to build centos stream -8 images for Train, Victoria and Ussuri? | 05:42 |
jingvar | Hi! | 07:45 |
jingvar | How does it work? | 07:45 |
jingvar | etc/kayobe/kolla/globals.yml:kolla_external_fqdn: "public.dev.cloud" | 07:45 |
jingvar | etc/kolla/globals.yml:kolla_external_fqdn: "10.228.4.100" | 07:45 |
jingvar | etc/kolla/globals.yml:kolla_external_fqdn: public.dev.cloud | 07:45 |
jingvar | is it ok, to place 2 same variables ? | 07:46 |
frickler | jingvar: the second entry should overwrite the first one | 08:04 |
jingvar | I now but it looks not pretty | 08:24 |
jingvar | warnings generator? | 08:25 |
mnasiadka | shyamb: no | 08:58 |
mnasiadka | shyamb: actually there are c8s images for Victoria | 08:59 |
mnasiadka | but no Train/Ussuri | 08:59 |
shyamb | mnasiadka: Okay. So there is no plan to create images for Train and Ussuri? Or it's still in discussion phase? | 09:00 |
mnasiadka | shyamb: we can't, there are no Train/Ussuri RDO packages for CentOS 8 Stream | 09:16 |
shyamb | mnasiadka: Got it. Thank you. | 09:17 |
shyamb | mnasiadka: victoria-centos8s is this the correct tag ? | 09:18 |
mnasiadka | shyamb: yes | 09:18 |
shyamb | I saw this tag in dockerhub | 09:18 |
opendevreview | Michal Nasiadka proposed openstack/kolla stable/victoria: Drop CentOS 8 Linux image builds https://review.opendev.org/c/openstack/kolla/+/827608 | 09:20 |
shyamb | mnasiadka: Thank you for the information. | 09:23 |
opendevreview | Pierre Riteau proposed openstack/kayobe stable/wallaby: ntp: Fix service mask when service doesn't exist https://review.opendev.org/c/openstack/kayobe/+/827403 | 09:32 |
opendevreview | Michal Nasiadka proposed openstack/kolla stable/victoria: Drop CentOS 8 Linux image builds https://review.opendev.org/c/openstack/kolla/+/827608 | 09:48 |
dardelean | hello, I have a question about the current kolla-ansible stable | 10:47 |
dardelean | here https://launchpad.net/kolla-ansible/+series is says xena series Pre-release Freeze | 10:47 |
dardelean | and wallaby series Current Stable Release | 10:47 |
dardelean | so is kolla xena officially released by kolla? | 10:48 |
dardelean | as stable I mean | 10:48 |
mnasiadka | damn, Ceph Nautilus repos in CentOS are gone, we need to use Pacific in CentOS 8 Stream Victoria | 10:48 |
yoctozepto | mnasiadka: just drop centos and close this chapter of indecency | 10:49 |
mnasiadka | yoctozepto: CentOS Stream builds in Victoria use CentOS Ceph repos (non-stream) :D | 10:49 |
mnasiadka | https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_bad/827608/2/check/kolla-build-centos8s-source/bada98d/kolla/build/000_FAILED_zun-compute.log | 10:50 |
yoctozepto | mnasiadka: ugh, so we made more mess than we thought yesterday, oh well! | 10:53 |
mnasiadka | Well | 11:01 |
mnasiadka | https://review.rdoproject.org/r/c/rdo-infra/rdo-release/+/38913 | 11:01 |
mnasiadka | This got merged yesterday | 11:01 |
yoctozepto | mnasiadka: "skip_if_unavailable=True" | 11:02 |
yoctozepto | khekhe | 11:02 |
mnasiadka | it's a bloody mess | 11:03 |
mnasiadka | asked the RDO guys... | 11:03 |
shyamb | Have we updated install document for victoria centos stream 8 containers? | 11:07 |
shyamb | How one can upgrade from centos8 victoria to centos stream8 victoria | 11:07 |
opendevreview | Merged openstack/kolla-ansible master: Add OIDCDiscoverURL mod_oidc option https://review.opendev.org/c/openstack/kolla-ansible/+/793622 | 11:19 |
opendevreview | Piotr Parczewski proposed openstack/kolla-ansible stable/xena: Add OIDCDiscoverURL mod_oidc option https://review.opendev.org/c/openstack/kolla-ansible/+/827637 | 11:25 |
opendevreview | Piotr Parczewski proposed openstack/kolla-ansible stable/wallaby: Add OIDCDiscoverURL mod_oidc option https://review.opendev.org/c/openstack/kolla-ansible/+/827638 | 11:26 |
hrw | morning | 11:31 |
opendevreview | Pierre Riteau proposed openstack/kayobe stable/wallaby: ntp: Fix service mask when service doesn't exist https://review.opendev.org/c/openstack/kayobe/+/827403 | 11:38 |
parallax | afternoon hrw :) | 12:17 |
yoctozepto | parallax: bold to provoke hrw again to paste the "universal greeting time" info | 12:18 |
opendevreview | Mark Goddard proposed openstack/kolla-ansible master: certificates: generate libvirt TLS certificates https://review.opendev.org/c/openstack/kolla-ansible/+/826723 | 12:18 |
opendevreview | Mark Goddard proposed openstack/kolla-ansible master: CI: enable libvirt TLS in TLS jobs https://review.opendev.org/c/openstack/kolla-ansible/+/825845 | 12:18 |
hrw | yoctozepto: ;DD | 12:27 |
hrw | UGT ftw! | 12:27 |
opendevreview | Maksim Malchuk proposed openstack/kayobe master: Multiple DHCP Subnets for Ironic Inspector https://review.opendev.org/c/openstack/kayobe/+/827653 | 12:31 |
opendevreview | Michal Nasiadka proposed openstack/kolla stable/victoria: Drop CentOS 8 Linux image builds https://review.opendev.org/c/openstack/kolla/+/827608 | 12:39 |
dardelean | hello, I have a question about the current kolla-ansible stable | 12:41 |
dardelean | here https://launchpad.net/kolla-ansible/+series is says xena series Pre-release Freeze | 12:42 |
dardelean | and wallaby series Current Stable Release | 12:42 |
dardelean | so is kolla xena officially released by kolla? as stable I mean | 12:42 |
mnasiadka | dardelean: refresh | 12:43 |
mnasiadka | (as in refresh the page in browser) | 12:44 |
dardelean | mnasiadka yes thank you :0 | 12:44 |
mnasiadka | hrw: regarding centos aarch64 erlang version - is it still max 22? | 12:51 |
opendevreview | Michal Nasiadka proposed openstack/kolla master: rabbitmq: update to 3.9 on x86 https://review.opendev.org/c/openstack/kolla/+/827656 | 12:56 |
opendevreview | Michal Nasiadka proposed openstack/kolla stable/victoria: Drop CentOS 8 Linux image builds https://review.opendev.org/c/openstack/kolla/+/827608 | 12:57 |
hrw | mnasiadka: let me check how we are on it | 12:58 |
mnasiadka | hrw: google found also this repo: http://packages.erlang-solutions.com/erlang/rpm/centos/8/aarch64/ (if it helps) | 12:58 |
opendevreview | Michal Arbet proposed openstack/kolla master: Add mariadb-arbitrator image https://review.opendev.org/c/openstack/kolla/+/780967 | 13:07 |
hrw | - nothing provides libwx_baseu-3.0.so.0()(64bit) needed by erlang-wx-24.0.4-1.el8.aarch64 | 13:08 |
hrw | epel time | 13:09 |
hrw | mnasiadka: thanks | 13:10 |
opendevreview | Michal Nasiadka proposed openstack/kolla stable/victoria: Drop CentOS Linux 8 image builds https://review.opendev.org/c/openstack/kolla/+/827608 | 13:12 |
opendevreview | Marcin Juszkiewicz proposed openstack/kolla master: erlang: use packages from Erlang Solutions on AArch64 https://review.opendev.org/c/openstack/kolla/+/827663 | 13:16 |
hrw | mnasiadka: on top we can bump to erlang 24 and rmq 3.9 | 13:16 |
mnasiadka | hrw: nice | 13:17 |
opendevreview | Marcin Juszkiewicz proposed openstack/kolla master: erlang: use packages from Erlang Solutions on AArch64 https://review.opendev.org/c/openstack/kolla/+/827663 | 13:18 |
hrw | x86-64 built | 13:23 |
hrw | aarch64 failed | 13:24 |
opendevreview | Michal Nasiadka proposed openstack/kolla stable/victoria: Drop CentOS Linux 8 image builds https://review.opendev.org/c/openstack/kolla/+/827608 | 13:27 |
hrw | mnasiadka: do we plan to move to ansible 2.12+? | 13:33 |
mnasiadka | https://review.opendev.org/c/openstack/kolla-ansible/+/819135 - merged | 13:33 |
hrw | (just noticed 'will require python 3.8' message) | 13:33 |
admiyo | What does it mean if a container (in this case Neutron Bootstrap) failed with No such file or directory: '/var/lib/kolla/config_files/config.json' | 13:33 |
mnasiadka | but we use it only on Ubuntu in CI | 13:33 |
hrw | admiyo: that it was not deployed | 13:34 |
hrw | INFO:kolla.common.utils.rabbitmq:Successfully tagged rmq3/centos-source-rabbitmq:13.1.0 | 13:34 |
hrw | aarch64 built | 13:34 |
opendevreview | Marcin Juszkiewicz proposed openstack/kolla master: erlang: use packages from Erlang Solutions on AArch64 https://review.opendev.org/c/openstack/kolla/+/827663 | 13:35 |
admiyo | what expectation was not met that it failed that way...the funny thing is that I was looking on line for explanations and found a blog post *I* had writted a couple years back with the same message | 13:35 |
hrw | admiyo: our images require deployment done by k-a. then they get that config.json file etc | 13:35 |
hrw | if you built and just tried 'docker run image-name' then it fails the way you shown | 13:36 |
admiyo | and I was in the middle of a long k-a run that used to work. I changed from 12.0.1 to 12.1.1 wasthe primary difference | 13:36 |
admiyo | but no change to config files | 13:36 |
admiyo | the ansible call is a call to kolla_docker | 13:36 |
admiyo | so I assume that there should be something that maps in the config from there | 13:37 |
admiyo | KOLLA_CONFIG_STRATEGY: "{{ config_strategy }}" | 13:37 |
admiyo | which expands to always_copy or similar | 13:37 |
admiyo | # grep -rn config_strategy /etc/kolla/ | 13:39 |
admiyo | /etc/kolla/globals.yml:6:config_strategy: "COPY_ALWAYS" | 13:39 |
admiyo | "+ sudo -E kolla_set_configs", so not on the kolla_docker call directly I guess | 13:41 |
admiyo | But its not like there is a running container image I can look at to see if the file is there. I'll start wiuth the assumption that it is not | 13:43 |
opendevreview | Michal Nasiadka proposed openstack/kolla stable/victoria: Drop CentOS Linux 8 image builds https://review.opendev.org/c/openstack/kolla/+/827608 | 13:44 |
admiyo | hrw what that says to me is that the k-a run is not dropping the config file in the container. That would not be a container build issue, would it? I did rebuild the containers, but It looks like this file is added during the actual docker call | 13:50 |
hrw | admiyo: you use same branch of kolla images and kolla-ansible, right? then you deployed using k-a, right? | 13:51 |
hrw | admiyo: and then 'docker logs CONTAINERNAME' tells you 'no such file' error? | 13:51 |
admiyo | I *think* so. I've inherited some automation here... | 13:51 |
admiyo | the error was from the ansible run | 13:52 |
admiyo | but yes, I deployed using K-a Um...but the wallby branch....maybe I need a specific tag.... | 13:52 |
admiyo | * (HEAD detached at 12.3.0) | 13:53 |
admiyo | hrm...or not | 13:53 |
admiyo | that was k-a repo... | 13:54 |
admiyo | cd ../kolla | 13:54 |
admiyo | root@openstack01sys-r116:/usr/local/src/kolla# git branch | 13:54 |
admiyo | * (HEAD detached from 12.1.0) | 13:54 |
admiyo | ok, so one inconsistency | 13:54 |
hrw | just use git head of stable/wallaby in both cases? | 13:56 |
admiyo | Now you are talking crazy talk | 13:56 |
opendevreview | Mark Goddard proposed openstack/kayobe master: DNM: enable iPXE for UEFI https://review.opendev.org/c/openstack/kayobe/+/827667 | 13:58 |
hrw | admiyo: no. it is the way | 13:58 |
hrw | admiyo: we do releases of stable/* from time to time and recommend to just use git HEAD of stable/* to not miss any fixes | 13:59 |
opendevreview | Pierre Riteau proposed openstack/kayobe stable/wallaby: ntp: Fix service mask when service doesn't exist https://review.opendev.org/c/openstack/kayobe/+/827403 | 14:00 |
opendevreview | Mark Goddard proposed openstack/kayobe master: DNM: Enable PXE UEFI booting https://review.opendev.org/c/openstack/kayobe/+/827669 | 14:01 |
jingvar | can I use kayobe/stable/xena for wallby? there is the option to set openstack release | 14:02 |
bsanjeewa | any core in the mood for WF+ ? https://review.opendev.org/c/openstack/kolla-ansible/+/824722 | 14:03 |
priteau | jingvar: It's not recommended to mix releases like this | 14:05 |
opendevreview | Michal Nasiadka proposed openstack/kolla stable/victoria: Drop CentOS Linux 8 image builds https://review.opendev.org/c/openstack/kolla/+/827608 | 14:11 |
admiyo | Yeah...I was kidding. I know that, and I concur. I thought that is what the automation I am using was doing...but someone thought to get things by tags. | 14:13 |
jingvar | priteau: understand, but the option confused me | 14:23 |
opendevreview | Mark Goddard proposed openstack/kolla-ansible master: certificates: generate libvirt TLS certificates https://review.opendev.org/c/openstack/kolla-ansible/+/826723 | 14:32 |
mnasiadka | yoctozepto: what do you mean that ceph ci fixed itself? have you seen it not failing anywhere? :) | 14:32 |
opendevreview | Mark Goddard proposed openstack/kolla-ansible master: CI: enable libvirt TLS in TLS jobs https://review.opendev.org/c/openstack/kolla-ansible/+/825845 | 14:32 |
mnasiadka | Hmm, Error: Unable to find a match: python3-sqlalchemy-collectd | 14:38 |
mnasiadka | hrw: was it one of the missing packages in opstools? | 14:38 |
hrw | yes | 14:38 |
hrw | 16:26 < hrw> Error: Unable to find a match: collectd-dpdk_telemetry collectd-libpod-stats collectd-sensubility python3-sqlalchemy-collectd | 14:38 |
mnasiadka | well, openstack-base is failing on that - are we waiting for it to magically appear? | 14:45 |
mnasiadka | mrunge: any eta? :) | 14:47 |
mrunge | dpdk_telemetry is dead and gone | 14:48 |
mrunge | I didn't had the chance to look at the other packages, and tbh. they are not my top priority | 14:49 |
mnasiadka | so basically we can mark collectd as unbuildable for now and skip installing python3-sqlalchemy-collectd | 14:49 |
mrunge | collectd is buildable | 14:49 |
mrunge | the other packages are packages having collectd in their name | 14:50 |
mrunge | but they are different source packages | 14:50 |
opendevreview | Michal Nasiadka proposed openstack/kolla master: Remove missing collectd packages https://review.opendev.org/c/openstack/kolla/+/827679 | 14:53 |
frickler | mnasiadka: both ubuntu and c8s are beyond the failure point now, so I assume yoctozepto was right. seems they had broken docker images that got fixed now? | 15:01 |
jrwr | with openstack kolla, how would I configure neurton to use an existing network bridge I have on all the compute nodes (and controller) this bridge is attached to a vlan tagged interface that is also used for most openstack traffic | 15:04 |
mnasiadka | frickler: might be | 15:09 |
mnasiadka | frickler: as always I've chosen the worst time to look into it :) | 15:09 |
opendevreview | Michal Nasiadka proposed openstack/kolla master: Remove missing collectd packages https://review.opendev.org/c/openstack/kolla/+/827679 | 15:14 |
opendevreview | Maksim Malchuk proposed openstack/kayobe master: Multiple DHCP Subnets for Ironic Inspector https://review.opendev.org/c/openstack/kayobe/+/827653 | 15:15 |
hrw | if someone wants to check videos of FOSDEM talks for 2015-2022 then I made a page listing them all (with search): https://marcin.juszkiewicz.com.pl/download/tables/fosdem/videoyears.html | 15:18 |
frickler | hrw: that's pretty cool, may I share this outside this community, too? | 15:20 |
frickler | mnasiadka: well, if that's what it takes to make the issue disappear, I'm fine with it ;) | 15:21 |
hrw | frickler: free to share wherever you want. It is done for public use. | 15:21 |
opendevreview | Merged openstack/kolla-ansible master: Deploy Zun with Cinder Ceph support https://review.opendev.org/c/openstack/kolla-ansible/+/824722 | 15:49 |
yoctozepto | hrw: oh my, I died at the sheer size | 15:54 |
spatel | yoctozepto how do i clean up my controller node and remove all kolla related stuff? | 15:56 |
opendevreview | Buddhika Sanjeewa proposed openstack/kolla-ansible stable/xena: Deploy Zun with Cinder Ceph support https://review.opendev.org/c/openstack/kolla-ansible/+/827649 | 16:31 |
mnasiadka | yoctozepto, hrw, mgoddard: this fixes the CI on master... need to backport it all the way to victoria, and then we can merge the drop centos8 patch :) | 17:51 |
mnasiadka | https://review.opendev.org/c/openstack/kolla/+/827679 | 17:51 |
yoctozepto | mnasiadka: you mean SQUASH and merge :D | 18:01 |
mnasiadka | yoctozepto: yeah, right ;) | 18:03 |
mnasiadka | but anyway it's nice to have at least master CI working | 18:03 |
yoctozepto | indeed | 18:06 |
yoctozepto | frickler: you around? to help with https://review.opendev.org/c/openstack/kolla-ansible/+/827512 | 18:10 |
yoctozepto | because it needs an infra folk to get debugged | 18:11 |
frickler | yoctozepto: I can take a look | 18:12 |
frickler | yoctozepto: POST-RUN END RESULT_TIMED_OUT: [untrusted : opendev.org/openstack/kolla-ansible/tests/post.yml@master] | 18:15 |
frickler | seems "TASK [Run diagnostics script]" timed out on secondary1 | 18:16 |
frickler | if you don't know why that could happen, I can hold another set of nodes | 18:17 |
yoctozepto | frickler: yeah, that would be helpful | 18:35 |
frickler | already on it | 18:35 |
yoctozepto | btw, where does it say this "POST-RUN END RESULT_TIMED_OUT" ? | 18:40 |
yoctozepto | ok | 18:40 |
yoctozepto | I know | 18:40 |
yoctozepto | I thought all logs were gone | 18:40 |
frickler | mnasiadka: do you want to do some rechecks on the cephadm test or do we assume all is good and I cancel the hold? | 18:44 |
mnasiadka | frickler: cancel the hold for now, it seems it feels better now, but surely needs some love :) | 18:45 |
frickler | yoctozepto: no, the usual job log is in place, it also shows that on the two other nodes the scripts finished in a matter of seconds, the timeout hits after 30 minutes then | 18:45 |
frickler | mnasiadka: ok | 18:45 |
yoctozepto | frickler: ack, weird it's 100% on ubuntu/debian and 0% on centos | 18:46 |
yoctozepto | I wonder which step is hangs on | 18:47 |
yoctozepto | it | 18:47 |
mnasiadka | hmm, would we want the single node jobs to also run cephadm? would it bring us any good? | 18:51 |
opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: CI: cephadm - copy config/keys to all nodes https://review.opendev.org/c/openstack/kolla-ansible/+/827298 | 18:59 |
yoctozepto | mnasiadka: what for? | 19:05 |
mnasiadka | yoctozepto: just asking, cinder? seems cephadm has an out of the box single node install option that sets proper policies | 19:05 |
opendevreview | Merged openstack/kolla-ansible master: certificates: generate libvirt TLS certificates https://review.opendev.org/c/openstack/kolla-ansible/+/826723 | 19:11 |
opendevreview | Merged openstack/kolla-ansible master: CI: enable libvirt TLS in TLS jobs https://review.opendev.org/c/openstack/kolla-ansible/+/825845 | 19:11 |
opendevreview | Radosław Piliszek proposed openstack/kolla-ansible master: [CI] Test Glance interaction with Cinder https://review.opendev.org/c/openstack/kolla-ansible/+/827264 | 19:12 |
yoctozepto | mnasiadka: hmm, I imagine we could make the ceph jobs singlenode and have another multinode job so that we can test our multinode parts instead of crying that ceph broke again | 19:27 |
yoctozepto | wdyt? | 19:27 |
yoctozepto | it looks really promising to me | 19:28 |
mnasiadka | well, question if we need shared storage for cinder in those multinode jobs :D | 19:28 |
yoctozepto | mnasiadka: well, we don't test HA so multiple separate would do | 19:29 |
yoctozepto | and also | 19:29 |
yoctozepto | you don't have to test cinder there | 19:29 |
yoctozepto | separation of concerns :-) | 19:29 |
yoctozepto | then we will see how bad our multinode is because of us (and our deps, including openstack, in general) and not because of ceph | 19:30 |
yoctozepto | we could gate on one multinode job then | 19:31 |
yoctozepto | the ipv6 job is multinode iirc | 19:31 |
opendevreview | Merged openstack/kayobe stable/wallaby: ntp: Fix service mask when service doesn't exist https://review.opendev.org/c/openstack/kayobe/+/827403 | 19:33 |
yoctozepto | added to the whiteboard for the next meeting's agenda | 19:42 |
* yoctozepto off | 19:42 | |
frickler | yoctozepto: wow, this I wouldn't have expected. it's "parted -l" that hangs, trying manually gives https://paste.opendev.org/show/812511/ | 19:46 |
frickler | I killed the hanging process, so we should get the remainder of logs collected | 19:47 |
opendevreview | Marcin Juszkiewicz proposed openstack/kolla master: Switch to use Yoga binary packages https://review.opendev.org/c/openstack/kolla/+/826488 | 19:48 |
admiyo | Well, hrw successfully escaped me....but switching to wallaby stable did not fix my problem. I still get the error launching the neutron bootstrap container from k-a: FileNotFoundError: [Errno 2] No such file or directory: '/var/lib/kolla/config_files/config.json' | 20:12 |
admiyo | what am I doing wrong | 20:12 |
opendevreview | Merged openstack/kolla master: Remove missing collectd packages https://review.opendev.org/c/openstack/kolla/+/827679 | 21:04 |
opendevreview | Verification of a change to openstack/kolla-ansible stable/wallaby failed: Add OIDCDiscoverURL mod_oidc option https://review.opendev.org/c/openstack/kolla-ansible/+/827638 | 21:04 |
yoctozepto | frickler: nice, I did not read irc so issued a recheck | 21:11 |
yoctozepto | perhaps some iscsi volume is hanging | 21:12 |
yoctozepto | will think tomorrow | 21:12 |
opendevreview | Merged openstack/kolla-ansible stable/xena: Add OIDCDiscoverURL mod_oidc option https://review.opendev.org/c/openstack/kolla-ansible/+/827637 | 21:39 |
opendevreview | Verification of a change to openstack/kayobe stable/xena failed: ntp: Fix service mask when service doesn't exist https://review.opendev.org/c/openstack/kayobe/+/827404 | 22:13 |
opendevreview | Verification of a change to openstack/kayobe master failed: ntp: Fix service mask when service doesn't exist https://review.opendev.org/c/openstack/kayobe/+/827486 | 22:13 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!