Thursday, 2022-02-03

opendevreviewJames Kirsch proposed openstack/kolla master: Add LetsEncrypt images for cert request/renewal  https://review.opendev.org/c/openstack/kolla/+/74133900:17
opendevreviewJames Kirsch proposed openstack/kolla master: Add LetsEncrypt images for cert request/renewal  https://review.opendev.org/c/openstack/kolla/+/74133900:21
opendevreviewVerification of a change to openstack/kayobe stable/wallaby failed: ntp: Fix service mask when service doesn't exist  https://review.opendev.org/c/openstack/kayobe/+/82740301:25
opendevreviewJames Kirsch proposed openstack/kolla master: Add LetsEncrypt images for cert request/renewal  https://review.opendev.org/c/openstack/kolla/+/74133903:51
shyambHi05:42
shyambAs we are stopping centos8 builds, are we going to build centos stream -8 images for Train, Victoria and Ussuri?05:42
jingvarHi!07:45
jingvarHow does it work?07:45
jingvaretc/kayobe/kolla/globals.yml:kolla_external_fqdn: "public.dev.cloud"07:45
jingvaretc/kolla/globals.yml:kolla_external_fqdn: "10.228.4.100"07:45
jingvaretc/kolla/globals.yml:kolla_external_fqdn: public.dev.cloud07:45
jingvaris it ok, to place 2 same variables ? 07:46
fricklerjingvar: the second entry should overwrite the first one08:04
jingvarI now but it looks  not pretty08:24
jingvarwarnings generator?08:25
mnasiadkashyamb: no08:58
mnasiadkashyamb: actually there are c8s images for Victoria08:59
mnasiadkabut no Train/Ussuri08:59
shyambmnasiadka: Okay. So there is no plan to create images for Train and Ussuri? Or it's still in discussion phase?09:00
mnasiadkashyamb: we can't, there are no Train/Ussuri RDO packages for CentOS 8 Stream09:16
shyambmnasiadka: Got it. Thank you.09:17
shyambmnasiadka: victoria-centos8s  is this the correct tag ?09:18
mnasiadkashyamb: yes09:18
shyambI saw this tag in dockerhub09:18
opendevreviewMichal Nasiadka proposed openstack/kolla stable/victoria: Drop CentOS 8 Linux image builds  https://review.opendev.org/c/openstack/kolla/+/82760809:20
shyambmnasiadka: Thank you for the information.09:23
opendevreviewPierre Riteau proposed openstack/kayobe stable/wallaby: ntp: Fix service mask when service doesn't exist  https://review.opendev.org/c/openstack/kayobe/+/82740309:32
opendevreviewMichal Nasiadka proposed openstack/kolla stable/victoria: Drop CentOS 8 Linux image builds  https://review.opendev.org/c/openstack/kolla/+/82760809:48
dardeleanhello, I have a question about the current kolla-ansible stable10:47
dardeleanhere https://launchpad.net/kolla-ansible/+series is says xena series Pre-release Freeze10:47
dardeleanand wallaby series Current Stable Release10:47
dardeleanso is kolla xena officially released by kolla?10:48
dardeleanas stable I mean10:48
mnasiadkadamn, Ceph Nautilus repos in CentOS are gone, we need to use Pacific in CentOS 8 Stream Victoria10:48
yoctozeptomnasiadka: just drop centos and close this chapter of indecency10:49
mnasiadkayoctozepto: CentOS Stream builds in Victoria use CentOS Ceph repos (non-stream) :D10:49
mnasiadkahttps://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_bad/827608/2/check/kolla-build-centos8s-source/bada98d/kolla/build/000_FAILED_zun-compute.log10:50
yoctozeptomnasiadka: ugh, so we made more mess than we thought yesterday, oh well!10:53
mnasiadkaWell11:01
mnasiadkahttps://review.rdoproject.org/r/c/rdo-infra/rdo-release/+/3891311:01
mnasiadkaThis got merged yesterday11:01
yoctozeptomnasiadka: "skip_if_unavailable=True"11:02
yoctozeptokhekhe11:02
mnasiadkait's a bloody mess11:03
mnasiadkaasked the RDO guys...11:03
shyambHave we updated install document for victoria centos stream 8 containers?11:07
shyambHow one can upgrade from centos8 victoria to centos stream8 victoria11:07
opendevreviewMerged openstack/kolla-ansible master: Add OIDCDiscoverURL mod_oidc option  https://review.opendev.org/c/openstack/kolla-ansible/+/79362211:19
opendevreviewPiotr Parczewski proposed openstack/kolla-ansible stable/xena: Add OIDCDiscoverURL mod_oidc option  https://review.opendev.org/c/openstack/kolla-ansible/+/82763711:25
opendevreviewPiotr Parczewski proposed openstack/kolla-ansible stable/wallaby: Add OIDCDiscoverURL mod_oidc option  https://review.opendev.org/c/openstack/kolla-ansible/+/82763811:26
hrwmorning11:31
opendevreviewPierre Riteau proposed openstack/kayobe stable/wallaby: ntp: Fix service mask when service doesn't exist  https://review.opendev.org/c/openstack/kayobe/+/82740311:38
parallaxafternoon hrw :)12:17
yoctozeptoparallax: bold to provoke hrw again to paste the "universal greeting time" info12:18
opendevreviewMark Goddard proposed openstack/kolla-ansible master: certificates: generate libvirt TLS certificates  https://review.opendev.org/c/openstack/kolla-ansible/+/82672312:18
opendevreviewMark Goddard proposed openstack/kolla-ansible master: CI: enable libvirt TLS in TLS jobs  https://review.opendev.org/c/openstack/kolla-ansible/+/82584512:18
hrwyoctozepto: ;DD12:27
hrwUGT ftw!12:27
opendevreviewMaksim Malchuk proposed openstack/kayobe master: Multiple DHCP Subnets for Ironic Inspector  https://review.opendev.org/c/openstack/kayobe/+/82765312:31
opendevreviewMichal Nasiadka proposed openstack/kolla stable/victoria: Drop CentOS 8 Linux image builds  https://review.opendev.org/c/openstack/kolla/+/82760812:39
dardeleanhello, I have a question about the current kolla-ansible stable12:41
dardeleanhere https://launchpad.net/kolla-ansible/+series is says xena series Pre-release Freeze12:42
dardeleanand wallaby series Current Stable Release12:42
dardeleanso is kolla xena officially released by kolla? as stable I mean12:42
mnasiadkadardelean: refresh12:43
mnasiadka(as in refresh the page in browser)12:44
dardeleanmnasiadka yes thank you :012:44
mnasiadkahrw: regarding centos aarch64 erlang version - is it still max 22?12:51
opendevreviewMichal Nasiadka proposed openstack/kolla master: rabbitmq: update to 3.9 on x86  https://review.opendev.org/c/openstack/kolla/+/82765612:56
opendevreviewMichal Nasiadka proposed openstack/kolla stable/victoria: Drop CentOS 8 Linux image builds  https://review.opendev.org/c/openstack/kolla/+/82760812:57
hrwmnasiadka: let me check how we are on it12:58
mnasiadkahrw: google found also this repo: http://packages.erlang-solutions.com/erlang/rpm/centos/8/aarch64/ (if it helps)12:58
opendevreviewMichal Arbet proposed openstack/kolla master: Add mariadb-arbitrator image  https://review.opendev.org/c/openstack/kolla/+/78096713:07
hrw  - nothing provides libwx_baseu-3.0.so.0()(64bit) needed by erlang-wx-24.0.4-1.el8.aarch6413:08
hrwepel time13:09
hrwmnasiadka: thanks13:10
opendevreviewMichal Nasiadka proposed openstack/kolla stable/victoria: Drop CentOS Linux 8 image builds  https://review.opendev.org/c/openstack/kolla/+/82760813:12
opendevreviewMarcin Juszkiewicz proposed openstack/kolla master: erlang: use packages from Erlang Solutions on AArch64  https://review.opendev.org/c/openstack/kolla/+/82766313:16
hrwmnasiadka: on top we can bump to erlang 24 and rmq 3.913:16
mnasiadkahrw: nice13:17
opendevreviewMarcin Juszkiewicz proposed openstack/kolla master: erlang: use packages from Erlang Solutions on AArch64  https://review.opendev.org/c/openstack/kolla/+/82766313:18
hrwx86-64 built13:23
hrwaarch64 failed13:24
opendevreviewMichal Nasiadka proposed openstack/kolla stable/victoria: Drop CentOS Linux 8 image builds  https://review.opendev.org/c/openstack/kolla/+/82760813:27
hrwmnasiadka: do we plan to move to ansible 2.12+?13:33
mnasiadkahttps://review.opendev.org/c/openstack/kolla-ansible/+/819135 - merged13:33
hrw(just noticed 'will require python 3.8' message)13:33
admiyoWhat does it mean if a container (in this case Neutron Bootstrap) failed with No such file or directory: '/var/lib/kolla/config_files/config.json'13:33
mnasiadkabut we use it only on Ubuntu in CI13:33
hrwadmiyo: that it was not deployed13:34
hrwINFO:kolla.common.utils.rabbitmq:Successfully tagged rmq3/centos-source-rabbitmq:13.1.013:34
hrwaarch64 built13:34
opendevreviewMarcin Juszkiewicz proposed openstack/kolla master: erlang: use packages from Erlang Solutions on AArch64  https://review.opendev.org/c/openstack/kolla/+/82766313:35
admiyowhat expectation was not met that it failed that way...the funny thing is that I was looking on line for explanations and found a blog post *I* had writted a couple years back with the same message13:35
hrwadmiyo: our images require deployment done by k-a. then they get that config.json file etc13:35
hrwif you built and just tried 'docker run image-name' then it fails the way you shown13:36
admiyoand I was in the middle of a long k-a run that used to work.  I changed from 12.0.1 to 12.1.1 wasthe primary difference13:36
admiyobut no change to config files13:36
admiyothe ansible call is a call to kolla_docker13:36
admiyoso I assume that there should be something that maps in the config from there13:37
admiyo KOLLA_CONFIG_STRATEGY: "{{ config_strategy }}"13:37
admiyowhich expands to always_copy or similar13:37
admiyo# grep -rn  config_strategy /etc/kolla/13:39
admiyo   /etc/kolla/globals.yml:6:config_strategy: "COPY_ALWAYS"13:39
admiyo"+ sudo -E kolla_set_configs",  so not on the kolla_docker call directly I guess13:41
admiyoBut its not like there is a running container image I can look at to see if the file is there.  I'll start wiuth the assumption that it is not13:43
opendevreviewMichal Nasiadka proposed openstack/kolla stable/victoria: Drop CentOS Linux 8 image builds  https://review.opendev.org/c/openstack/kolla/+/82760813:44
admiyohrw what that says to me is that the k-a run is not dropping the config file in the container.  That would not be a container build issue, would it? I did rebuild the containers, but It looks like this file is added during the actual docker call13:50
hrwadmiyo: you use same branch of kolla images and kolla-ansible, right? then you deployed using k-a, right?13:51
hrwadmiyo: and then 'docker logs CONTAINERNAME' tells you 'no such file' error?13:51
admiyoI *think* so.  I've inherited some automation here...13:51
admiyothe error was from the ansible run13:52
admiyobut yes, I deployed using K-a  Um...but the wallby branch....maybe I need a specific tag....13:52
admiyo* (HEAD detached at 12.3.0)13:53
admiyohrm...or not13:53
admiyothat was k-a repo...13:54
admiyo cd ../kolla13:54
admiyoroot@openstack01sys-r116:/usr/local/src/kolla# git branch 13:54
admiyo* (HEAD detached from 12.1.0)13:54
admiyook, so one inconsistency13:54
hrwjust use git head of stable/wallaby in both cases?13:56
admiyoNow you are talking crazy talk13:56
opendevreviewMark Goddard proposed openstack/kayobe master: DNM: enable iPXE for UEFI  https://review.opendev.org/c/openstack/kayobe/+/82766713:58
hrwadmiyo: no. it is the way13:58
hrwadmiyo: we do releases of stable/* from time to time and recommend to just use git HEAD of stable/* to not miss any fixes13:59
opendevreviewPierre Riteau proposed openstack/kayobe stable/wallaby: ntp: Fix service mask when service doesn't exist  https://review.opendev.org/c/openstack/kayobe/+/82740314:00
opendevreviewMark Goddard proposed openstack/kayobe master: DNM: Enable PXE UEFI booting  https://review.opendev.org/c/openstack/kayobe/+/82766914:01
jingvarcan I use kayobe/stable/xena for wallby? there is the option to set openstack release14:02
bsanjeewaany core in the mood for WF+ ? https://review.opendev.org/c/openstack/kolla-ansible/+/82472214:03
priteaujingvar: It's not recommended to mix releases like this14:05
opendevreviewMichal Nasiadka proposed openstack/kolla stable/victoria: Drop CentOS Linux 8 image builds  https://review.opendev.org/c/openstack/kolla/+/82760814:11
admiyoYeah...I was kidding.  I know that, and I concur.  I thought that is what the automation I am using was doing...but someone thought to get things by tags.14:13
jingvarpriteau: understand, but the option confused me14:23
opendevreviewMark Goddard proposed openstack/kolla-ansible master: certificates: generate libvirt TLS certificates  https://review.opendev.org/c/openstack/kolla-ansible/+/82672314:32
mnasiadkayoctozepto: what do you mean that ceph ci fixed itself? have you seen it not failing anywhere? :)14:32
opendevreviewMark Goddard proposed openstack/kolla-ansible master: CI: enable libvirt TLS in TLS jobs  https://review.opendev.org/c/openstack/kolla-ansible/+/82584514:32
mnasiadkaHmm, Error: Unable to find a match: python3-sqlalchemy-collectd14:38
mnasiadkahrw: was it one of the missing packages in opstools?14:38
hrwyes14:38
hrw16:26 < hrw> Error: Unable to find a match: collectd-dpdk_telemetry collectd-libpod-stats collectd-sensubility python3-sqlalchemy-collectd14:38
mnasiadkawell, openstack-base is failing on that - are we waiting for it to magically appear?14:45
mnasiadkamrunge: any eta? :)14:47
mrungedpdk_telemetry is dead and gone14:48
mrungeI didn't had the chance to look at the other packages, and tbh. they are not my top priority14:49
mnasiadkaso basically we can mark collectd as unbuildable for now and skip installing python3-sqlalchemy-collectd14:49
mrungecollectd is buildable14:49
mrungethe other packages are packages having collectd in their name14:50
mrungebut they are different source packages14:50
opendevreviewMichal Nasiadka proposed openstack/kolla master: Remove missing collectd packages  https://review.opendev.org/c/openstack/kolla/+/82767914:53
fricklermnasiadka: both ubuntu and c8s are beyond the failure point now, so I assume yoctozepto was right. seems they had broken docker images that got fixed now?15:01
jrwrwith openstack kolla, how would I configure neurton to use an existing network bridge I have on all the compute nodes (and controller) this bridge is attached to a vlan tagged interface that is also used for most openstack traffic15:04
mnasiadkafrickler: might be15:09
mnasiadkafrickler: as always I've chosen the worst time to look into it :)15:09
opendevreviewMichal Nasiadka proposed openstack/kolla master: Remove missing collectd packages  https://review.opendev.org/c/openstack/kolla/+/82767915:14
opendevreviewMaksim Malchuk proposed openstack/kayobe master: Multiple DHCP Subnets for Ironic Inspector  https://review.opendev.org/c/openstack/kayobe/+/82765315:15
hrwif someone wants to check videos of FOSDEM talks for 2015-2022 then I made a page listing them all (with search): https://marcin.juszkiewicz.com.pl/download/tables/fosdem/videoyears.html15:18
fricklerhrw: that's pretty cool, may I share this outside this community, too?15:20
fricklermnasiadka: well, if that's what it takes to make the issue disappear, I'm fine with it ;)15:21
hrwfrickler: free to share wherever you want. It is done for public use.15:21
opendevreviewMerged openstack/kolla-ansible master: Deploy Zun with Cinder Ceph support  https://review.opendev.org/c/openstack/kolla-ansible/+/82472215:49
yoctozeptohrw: oh my, I died at the sheer size15:54
spatelyoctozepto how do i clean up my controller node and remove all kolla related stuff? 15:56
opendevreviewBuddhika Sanjeewa proposed openstack/kolla-ansible stable/xena: Deploy Zun with Cinder Ceph support  https://review.opendev.org/c/openstack/kolla-ansible/+/82764916:31
mnasiadkayoctozepto, hrw, mgoddard: this fixes the CI on master... need to backport it all the way to victoria, and then we can merge the drop centos8 patch :)17:51
mnasiadkahttps://review.opendev.org/c/openstack/kolla/+/82767917:51
yoctozeptomnasiadka: you mean SQUASH and merge :D18:01
mnasiadkayoctozepto: yeah, right ;)18:03
mnasiadkabut anyway it's nice to have at least master CI working18:03
yoctozeptoindeed18:06
yoctozeptofrickler: you around? to help with https://review.opendev.org/c/openstack/kolla-ansible/+/82751218:10
yoctozeptobecause it needs an infra folk to get debugged18:11
frickleryoctozepto: I can take a look18:12
frickleryoctozepto: POST-RUN END RESULT_TIMED_OUT: [untrusted : opendev.org/openstack/kolla-ansible/tests/post.yml@master]18:15
fricklerseems "TASK [Run diagnostics script]" timed out on secondary118:16
fricklerif you don't know why that could happen, I can hold another set of nodes18:17
yoctozeptofrickler: yeah, that would be helpful18:35
frickleralready on it18:35
yoctozeptobtw, where does it say this "POST-RUN END RESULT_TIMED_OUT" ?18:40
yoctozeptook18:40
yoctozeptoI know18:40
yoctozeptoI thought all logs were gone18:40
fricklermnasiadka: do you want to do some rechecks on the cephadm test or do we assume all is good and I cancel the hold?18:44
mnasiadkafrickler: cancel the hold for now, it seems it feels better now, but surely needs some love :)18:45
frickleryoctozepto: no, the usual job log is in place, it also shows that on the two other nodes the scripts finished in a matter of seconds, the timeout hits after 30 minutes then18:45
fricklermnasiadka: ok18:45
yoctozeptofrickler: ack, weird it's 100% on ubuntu/debian and 0% on centos18:46
yoctozeptoI wonder which step is hangs on18:47
yoctozeptoit18:47
mnasiadkahmm, would we want the single node jobs to also run cephadm? would it bring us any good?18:51
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: CI: cephadm - copy config/keys to all nodes  https://review.opendev.org/c/openstack/kolla-ansible/+/82729818:59
yoctozeptomnasiadka: what for?19:05
mnasiadkayoctozepto: just asking, cinder? seems cephadm has an out of the box single node install option that sets proper policies19:05
opendevreviewMerged openstack/kolla-ansible master: certificates: generate libvirt TLS certificates  https://review.opendev.org/c/openstack/kolla-ansible/+/82672319:11
opendevreviewMerged openstack/kolla-ansible master: CI: enable libvirt TLS in TLS jobs  https://review.opendev.org/c/openstack/kolla-ansible/+/82584519:11
opendevreviewRadosÅ‚aw Piliszek proposed openstack/kolla-ansible master: [CI] Test Glance interaction with Cinder  https://review.opendev.org/c/openstack/kolla-ansible/+/82726419:12
yoctozeptomnasiadka: hmm, I imagine we could make the ceph jobs singlenode and have another multinode job so that we can test our multinode parts instead of crying that ceph broke again19:27
yoctozeptowdyt?19:27
yoctozeptoit looks really promising to me19:28
mnasiadkawell, question if we need shared storage for cinder in those multinode jobs :D19:28
yoctozeptomnasiadka: well, we don't test HA so multiple separate would do19:29
yoctozeptoand also19:29
yoctozeptoyou don't have to test cinder there19:29
yoctozeptoseparation of concerns :-) 19:29
yoctozeptothen we will see how bad our multinode is because of us (and our deps, including openstack, in general) and not because of ceph19:30
yoctozeptowe could gate on one multinode job then19:31
yoctozeptothe ipv6 job is multinode iirc19:31
opendevreviewMerged openstack/kayobe stable/wallaby: ntp: Fix service mask when service doesn't exist  https://review.opendev.org/c/openstack/kayobe/+/82740319:33
yoctozeptoadded to the whiteboard for the next meeting's agenda19:42
* yoctozepto off19:42
frickleryoctozepto: wow, this I wouldn't have expected. it's "parted -l" that hangs, trying manually gives https://paste.opendev.org/show/812511/19:46
fricklerI killed the hanging process, so we should get the remainder of logs collected19:47
opendevreviewMarcin Juszkiewicz proposed openstack/kolla master: Switch to use Yoga binary packages  https://review.opendev.org/c/openstack/kolla/+/82648819:48
admiyoWell, hrw successfully escaped me....but switching to wallaby stable did not fix my problem.  I still get the error launching the neutron bootstrap container from k-a:  FileNotFoundError: [Errno 2] No such file or directory: '/var/lib/kolla/config_files/config.json'20:12
admiyowhat am I doing wrong20:12
opendevreviewMerged openstack/kolla master: Remove missing collectd packages  https://review.opendev.org/c/openstack/kolla/+/82767921:04
opendevreviewVerification of a change to openstack/kolla-ansible stable/wallaby failed: Add OIDCDiscoverURL mod_oidc option  https://review.opendev.org/c/openstack/kolla-ansible/+/82763821:04
yoctozeptofrickler: nice, I did not read irc so issued a recheck21:11
yoctozeptoperhaps some iscsi volume is hanging21:12
yoctozeptowill think tomorrow21:12
opendevreviewMerged openstack/kolla-ansible stable/xena: Add OIDCDiscoverURL mod_oidc option  https://review.opendev.org/c/openstack/kolla-ansible/+/82763721:39
opendevreviewVerification of a change to openstack/kayobe stable/xena failed: ntp: Fix service mask when service doesn't exist  https://review.opendev.org/c/openstack/kayobe/+/82740422:13
opendevreviewVerification of a change to openstack/kayobe master failed: ntp: Fix service mask when service doesn't exist  https://review.opendev.org/c/openstack/kayobe/+/82748622:13

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!