Wednesday, 2022-04-27

opendevreviewJames Kirsch proposed openstack/kolla-ansible master: Add support for LetsEncrypt-managed certs  https://review.opendev.org/c/openstack/kolla-ansible/+/74134000:15
dardeleanHi, does kolla-ansible upgrade also perform the bootstrap phase?06:45
dardeleanthis would be problematic for the existing ceph containers, they may be wiped by bootstrap06:45
watzdardelean: no, it doesn't07:02
dardeleanthanks07:03
watzi just updated from pike to xena (going in steps of 2)07:03
watzwe also upgraded the base os (ubuntu) and installed the newest docker version from the docker repo07:04
watzboostrap would try to install old docker-engine oder docker-ce packges from old, non-exising repos07:04
watzwe skipped that and made sure that basic docker functionality was available07:05
watzso we totaly could skip the bootstrap process07:05
vbelgood morning/evening, trying to build base (ubuntu, centos): docker run -ti quay.io/centos/centos:stream8 sh07:28
vbelsh-4.4# rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch07:28
vbelcurl: (22) The requested URL returned error: 403 07:28
vbelerror: https://artifacts.elastic.co/GPG-KEY-elasticsearch: import read failed(2).07:28
vbelis the site over?07:29
mnasiadkaworks on my end07:31
vbelmnasiadka, thanks, looks my updated docker problem07:36
vbelhttps://artifacts.elastic.co/GPG-KEY-elasticsearch    - this site is forbidden for me :(07:42
vbeli cannot download elastic search on any device07:46
vbelwill test in air :)07:46
dardeleanon  the topic of upgrade. about globals.yaml, the docos say this "or globals.yml relevant changes should be merged into a copy of the new template, and then replace the file in /etc/kolla with the updated version. For passwords.yml, see the kolla-mergepwd instructions in Tips and Tricks."07:46
dardeleanthis is not dome automatically, right? I have to manually config the new globals.yaml, right?07:46
mnasiadkaif you need any changes - yes07:48
opendevreviewPierre Riteau proposed openstack/kayobe-config master: [release] Synchronise with latest kayobe changes for Yoga  https://review.opendev.org/c/openstack/kayobe-config/+/83939707:53
opendevreviewVladislav Belogrudov proposed openstack/kolla master: Add multipath to cinder-volume  https://review.opendev.org/c/openstack/kolla/+/83948107:57
opendevreviewPierre Riteau proposed openstack/kayobe master: Use rsync to update kayobe-config  https://review.opendev.org/c/openstack/kayobe/+/83948207:59
wuchunyangHi,  How often do we push images to quay.io? 08:00
opendevreviewMerged openstack/kayobe-config master: [release] Synchronise with latest kayobe changes for Yoga  https://review.opendev.org/c/openstack/kayobe-config/+/83939708:03
hrwjamesbenson: each revision of change needs same change-id08:05
opendevreviewPierre Riteau proposed openstack/kayobe-config stable/yoga: [release] Synchronise with latest kayobe changes for Yoga  https://review.opendev.org/c/openstack/kayobe-config/+/83934508:06
hrwjamesbenson: "git review -d URL" will fetch change from gerrit and make a branch for it. edit, 'git commit --amend' changes and then send back with 'git review'08:06
mnasiadkahrw, yoctozepto, mgoddard: anybody wants to run todays meeting? I'm still a bit sick and would prefer to rest at this time ;)08:10
watzdardelean: yes, that's true .. we merged globals.yaml with the example one on each upgrade step08:19
hrwmnasiadka: do not know yet will I be able to attend.08:24
opendevreviewlixuehai proposed openstack/kolla-ansible master: add path_grouping_policy, failback, no_path_retry default value  https://review.opendev.org/c/openstack/kolla-ansible/+/83949308:27
opendevreviewlixuehai proposed openstack/kolla-ansible master: add path_grouping_policy, failback, no_path_retry default value  https://review.opendev.org/c/openstack/kolla-ansible/+/83949308:31
dardelean@watz thanks08:37
opendevreviewPierre Riteau proposed openstack/kolla master: Use rsync to update kayobe-config  https://review.opendev.org/c/openstack/kolla/+/83949609:05
dardeleanis there a way for me to pin a specific docker image versiion? For example right now I am testing an upgrade to Xena, with images that were built 6 hours ago. Let's say that next week I'll do the upgrade in prod, is there a way to pin the same images? 09:08
watzin debian/ubuntu via apt pinning, yes09:09
watzah .. docker image .. not docker package .. sorry .. my fault09:10
dardeleanyes, I want to make sure I put in prod the same OS/docker version/images I tested in test env09:11
fricklerdardelean: don't use upstream images in production, run your own registry, build your own images, tag them as you need them09:11
watzin that case, i would use a docker registry and pull from there09:11
dardeleanI see09:11
fricklermnasiadka: if noone else comes along, I can give it a try. get some rest and get well soon09:14
mnasiadkafrickler: thanks09:22
hrwdardelean: like frickler said - keep your own copy of images. This way when you want to add more machines you have exact same images available.09:26
dardeleanhrw thanks for the info09:26
dardeleanmakes sense09:27
hrwand saves lot of problems09:27
hrwyou can use own namespace, own tags09:27
hrwproduction-xena/debian-source-nova-compute:20220427 shows exactly branch and build date09:28
hrwor even production-xena-g2493257ae/debian-source-nova-compute:20220427 to keep git HEAD of used kolla code09:29
hrw(id random)09:29
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: ovn: Change NB/SB connection setup to allow usage of inactivity probe  https://review.opendev.org/c/openstack/kolla-ansible/+/83950109:35
mnasiadkacongnt: ^^09:35
opendevreviewMerged openstack/kayobe stable/yoga: Fix no_proxy configuration  https://review.opendev.org/c/openstack/kayobe/+/83934409:52
opendevreviewMerged openstack/kayobe-config stable/yoga: [release] Synchronise with latest kayobe changes for Yoga  https://review.opendev.org/c/openstack/kayobe-config/+/83934509:52
opendevreviewRafael Weingartner proposed openstack/kolla-ansible master: Use 'cloudkitty_influxdb_use_ssl' when creatign InfluxDB database  https://review.opendev.org/c/openstack/kolla-ansible/+/83939310:01
dardeleanAlso, for kolla-ansible upgrade, it's totally fine to do this process node by node, right? First update controller1, then controller2, etc10:22
dardeleanor does it have logic that required all nodes10:23
dardeleanI'm doing it with --limit controller1, etc10:23
opendevreviewPierre Riteau proposed openstack/kayobe-config-dev master: [release] Synchronise with latest kayobe changes for Yoga  https://review.opendev.org/c/openstack/kayobe-config-dev/+/83951010:42
opendevreviewPierre Riteau proposed openstack/kayobe-config-dev stable/yoga: [release] Synchronise with latest kayobe changes for Yoga  https://review.opendev.org/c/openstack/kayobe-config-dev/+/83951110:42
dardeleanseems like the controllers need to upgraded at the same time, not one by one10:58
mnasiadkacorrect, basically due to databases10:59
mnasiadkaand if you're running OVN, the situation is a bit more complex.10:59
dardeleanalso a nova check failed11:04
dardeleanbut I guess that for the computes it can be done one by one11:04
jingvarI've placed ovndb behind VIP11:27
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: ovn: Change NB/SB connection setup to allow usage of inactivity probe  https://review.opendev.org/c/openstack/kolla-ansible/+/83950111:29
opendevreviewRafael Weingartner proposed openstack/kolla-ansible master: Use 'cloudkitty_influxdb_use_ssl' when creatign InfluxDB database  https://review.opendev.org/c/openstack/kolla-ansible/+/83939311:51
*** amoralej is now known as amoralej|lunch12:30
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: ovn: Change NB/SB connection setup to allow usage of inactivity probe  https://review.opendev.org/c/openstack/kolla-ansible/+/83950112:40
jamesbensonhrw, should they all have the same change-id from the first commit or the last or doesn't matter?12:51
hrwjamesbenson: same12:55
hrwjamesbenson: you do a change. it gets change-id after you commit. then each new version is just revision of change. so keeping same changeid is needed12:56
mnasiadkabut it needs to be one commit, multiple commits - multiple changes :D12:56
hrwah, right. if you have series of changes12:57
opendevreviewJames proposed openstack/kolla-ansible master: Add radosgw doc to enable ceph as swift backend  https://review.opendev.org/c/openstack/kolla-ansible/+/83927212:59
jamesbensonhrw, do you mind if I pm you?13:00
jamesbensonI'm not sure I'm doing this right.  This is the final commit that I'm interested in. https://review.opendev.org/c/openstack/kolla-ansible/+/83941813:04
hrwjamesbenson: 'git review -d https://review.opendev.org/c/openstack/kolla-ansible/+/839418', do changes, 'git commit --amend', 'git review'13:05
hrwjamesbenson: in other words: fetch, amend, resend13:05
jamesbensonokay, so fetch the very first commit, update it to the latest version that I have and then git review it, right?13:07
hrwit fetches latest revision of change13:07
jamesbensonbut only if it has the same change ID, in my case it doesn't. because I didn't use those commands.13:08
hrwyou can always just update change-id by hand. 13:09
hrwjamesbenson: fetching with git-review makes sure that you fetch latest revision of change - someone could update it in meantime13:09
hrwjamesbenson: gerrit operates on changes. 839418 is a change. each revision of it is separate commit and all of those commits has the same change-id line in message.13:11
*** amoralej|lunch is now known as amoralej13:12
opendevreviewJames proposed openstack/kolla-ansible master: update external-ceph document with additional settings  https://review.opendev.org/c/openstack/kolla-ansible/+/83926913:15
hrwjamesbenson: you can use 'git review -x ID' to fetch other patch on top of current branch. This way you can queue them as one set.13:18
dardeleanI've upgraded the controller nodes and lost connectivity to the VMs :(13:20
jamesbensonhrw: I'm sorry, I just don't use gerrit/git enough to fully understand what is needed here.  What, if anything, else do I need to do?13:35
hrwjamesbenson: you are new to it. ignore my last line13:35
jamesbensonhrw: sorry, my use of git is primarily me and my own projects, so this is beyond my typical use.  And it's been several years since I've contributed to kolla, so I've forgotten the process a bit.13:36
hrwjamesbenson: no problem. get familiar with tool by sending single patches13:37
jamesbensonhrw: I know it's just a doc change, but it is it good order now to be reviewed properly?13:37
hrware users expected to set so many ceph options?13:45
hrwcan't it be done by kolla-ansible role/task?13:45
mnasiadkawe're not configuring Ceph anymore13:47
mnasiadkawe are just using it in Kolla-Ansible13:47
dardelean@watz did you loose connectivity to the VMs during the upgrade, when controllers were on a different version than the computes?13:48
dardeleanafter I upgraded the computes to xena, I got connectivity once more13:49
dardeleanbut when controllers were on xena, computes on wallaby, VMs were not accessible13:49
fricklermeeting in 10 mgoddard mnasiadka hrw egonzalez yoctozepto rafaelweingartne cosmicsound osmanlicilegi bbezak parallax Fl1nt adrian-a13:50
frickler#startmeeting kolla14:00
opendevmeetMeeting started Wed Apr 27 14:00:21 2022 UTC and is due to finish in 60 minutes.  The chair is frickler. Information about MeetBot at http://wiki.debian.org/MeetBot.14:00
opendevmeetUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.14:00
opendevmeetThe meeting name has been set to 'kolla'14:00
frickler#topic rollcall14:00
frickleranyone up for a meeting? /me is doing this for the first time, so please bear with me14:01
hrwo/14:01
mgoddard\o14:02
frickler#topic agenda14:04
frickler* Roll-call14:04
frickler* Agenda14:04
frickler* Announcements14:04
frickler* Review action items from the last meeting14:04
frickler* CI status14:04
frickler* Release tasks14:04
frickler* Current cycle planning14:04
frickler* Additional agenda (from whiteboard)14:04
frickler* Open discussion14:04
frickler#topic Announcements14:04
fricklernothing that I know about, anyone anything to mention?14:04
hrwUbuntu 22.04 released14:05
hrwinfra mirror exists, we lack external repos so k-a CI will not pass14:05
frickleryes, image is available in infra (no arm yet)14:05
hrwno fluentd, influxdb, mariadb14:06
fricklerI've started testing devstack, that also has some issues in neutron it seems14:06
fricklerhrw: can we use focal repos for those?14:07
hrwno idea14:07
hrwI left ubuntu update for someone else this time14:07
hrwwe have similar issues with cs9 update14:07
fricklero.k., maybe just wait and see, then. I can look at it once I get devstack working14:08
hrwnext topic?14:08
frickler#topic Review action items from the last meeting14:08
fricklerI didn't see any14:08
frickler#topic CI status14:08
fricklerthe grafana dashboard need updating14:09
frickler#link https://grafana.opendev.org/d/c0d59dad13/kolla-failure-rate?orgId=114:09
fricklerlots of binary job references14:09
hrwI wonder is anyone using it14:09
fricklerit would be a good view on CI status if it were up to date, I'll take an action to try and update it14:10
frickler#action frickler update grafana dashboard14:10
frickleranything else on CI things?14:11
hrwfrickler: we have new names in Zed so it would be nice to have14:11
hrwhttps://review.opendev.org/c/openstack/kolla-ansible/+/839047 needs second check - adds binary->source upgrade job14:11
hrwnot remember other issue14:12
fricklero.k., I'll look at that later14:12
frickler#topic Release tasks14:12
fricklerI must admit I don't know where to look for these14:12
hrwhttps://docs.openstack.org/kolla/latest/contributor/release-management.html iirc14:13
hrwhttps://review.opendev.org/q/project:openstack%252Fkolla-ansible+branch:stable%252Fyoga lists 3 patches not merged yet14:14
fricklerseems we are at R-23 now https://docs.openstack.org/kolla/latest/contributor/release-management.html#r-23-development-begins14:14
hrwkolla and kayobe are fine in stable/yoga14:14
frickler[all] Communicate end of feature freeze via IRC meeting14:15
hrwfrickler: R-23 for Zed. But did we released Yoga?14:15
fricklerhrw: hmm, good point14:15
hrwwe need to review and merge/reject 3 k-a patches and do RC or release14:16
fricklermgoddard: do you know how to proceed with that? otherwise I guess we'll need to wait for mnasiadka being well again14:16
fricklerI'll look at the open patches14:17
frickleranyway, I guess let's move on, then14:18
mgoddardI don't think any of the patches are critical - just bugfix backports14:18
mgoddardso in theory we could release14:18
mgoddardfeels a bit too soon though. Where are the usual release blockers?14:19
mmalchukmgoddard 817076 is not bugfix14:19
mmalchukhttps://review.opendev.org/c/openstack/kolla-ansible/+/81707614:19
hrwmmalchuk: that's master. not stable/yoga14:20
mmalchukoops14:20
mmalchuksorry14:21
hrwnp14:21
frickler#topic Current cycle planning14:22
hrwnow we talk Zed14:23
hrwwe need a task in k-a to cleanup old containers after upgrade14:23
fricklerhrw: containers or just images?14:23
hrwhttps://review.opendev.org/c/openstack/kolla-ansible/+/839047 job shows old ubuntu-binary-* images on hosts14:24
hrwfrickler: images, right14:24
fricklerdo you want to do that or do we need to find someone?14:25
hrwI did not checked, maybe k-a has something for it already14:26
mgoddardhrw: this is how it always is14:26
hrwlong time since my last deployment14:26
hrwmgoddard: ok14:26
mgoddardhrw: after wallaby -> xena upgrade you still have wallaby images14:26
mgoddardthey're just not running14:26
hrwok, so something for me in spare time ;D14:27
frickleranything else or should we go to hrw's topic?14:28
hrwfrickler: the ones in whiteboard are for last week14:29
frickleroh, so we can drop them?14:29
hrwdropped all as we discussed them week ago14:30
fricklerthen we'd just have ...14:30
frickler#topic Open discussion14:30
hrwRDO team just announced Yoga release14:30
hrwfor CS8 and CS914:30
jamesbensonQ: Do we want to offer any support to ceph in ka or just purely the docs?14:32
jamesbensonI know it used to be supported many moons ago, just a thought.14:32
fricklerjamesbenson: iiuc it was removed and assumed to be handled externally14:33
hrwjamesbenson: ceph is only external. we dropped installing it and no way back14:33
mmalchukagree, there are more tools to deploy ceph, eg. cephadm, ceph-ansible, ceph-deploy... users should use any14:34
jamesbensonokay, so no bringing it back then. thanks14:34
frickleranother note: nova dropped support for py36, so once we switch to zed/master, forced end of CS8 support I guess unless14:36
fricklerunless someone wants to bring in py38 there?14:36
hrwwe agreed on move to CS914:37
hrwexternal repos are problem14:37
hrwwe have same issue with CS9 and ubuntu 22.04 - no external repos yet14:38
hrwfor fluentd image we need td-agent build with openssl 3 which also means ruby 3 so upstream has some stuff to handle14:39
hrwhttps://github.com/fluent/fluentd/discussions/3704 for more14:39
frickleryeah, seems we only can wait for that and maybe start testing initially with a reduced set of external stuff14:41
hrwfrickler: https://review.opendev.org/c/openstack/kolla/+/836664 moves to cs914:43
fricklerjammy comes with mariadb 10.6 at least natively14:43
hrwadded you14:43
fricklerhrw: do you mean to W-1 instead of review -1?14:44
fricklerbut I'll have a look at that anyway14:44
hrweffect is the same - ignore zuul, not ready for merge, ready for review14:44
hrww-1 is too much "work in progress, can ignore"14:45
hrwr-1 is more 'I do not like it, review to check do you agree'14:45
fricklerhrw: review -1 isn't technically blocking, though, so if you want to avoid it getting merged, I'd prefer W-114:45
fricklero.k., asking for more negative feedback is a valid use14:45
hrwmoved to w-114:46
hrwwill restore k-a job to make sure it does not pass zuul ;D14:46
fricklerafter looking at the patch, I hope noone would approve something that removes so much testing14:47
fricklerbut better be safe than sorry, yes14:48
frickleranything else? otherwise we could end a bit early14:48
hrwnothing more from my side14:49
fricklero.k., thx everyone14:49
frickler#endmeeting14:50
opendevmeetMeeting ended Wed Apr 27 14:50:03 2022 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)14:50
opendevmeetMinutes:        https://meetings.opendev.org/meetings/kolla/2022/kolla.2022-04-27-14.00.html14:50
opendevmeetMinutes (text): https://meetings.opendev.org/meetings/kolla/2022/kolla.2022-04-27-14.00.txt14:50
opendevmeetLog:            https://meetings.opendev.org/meetings/kolla/2022/kolla.2022-04-27-14.00.log.html14:50
mmalchukmgoddard could you please take a look at https://review.opendev.org/c/openstack/kolla-ansible/+/81707614:51
opendevreviewMarcin Juszkiewicz proposed openstack/kolla master: [WIP] Move to CentOS Stream 9  https://review.opendev.org/c/openstack/kolla/+/83666414:52
hrwnow zuul should properly fail14:52
*** mat_fechner is now known as matfechner14:58
mnasiadkahrw: mariadb.com packages for jammy should show up in next iteration of versions, which means in around 2 days looking at their JIRA dashboard...15:02
mnasiadkaI started building that locally, will submit some very WIP patch tomorrow15:03
mnasiadkabut fluentd has the same problem15:03
jamesbensonSvenKieske: I saw your comment about the 'rgw_verify_ssl' want me to just change that to true?  Also, do you want me to add any notes about self-signed certs? 15:03
SvenKieskejamesbenson: yeah would set it to true, add a comment or if you have the time: the proper steps to add a correctly signed cert (lets encrypt?) and maybe a note about disabling this with self signed certs15:36
opendevreviewMichal Nasiadka proposed openstack/kolla master: veryWIP: Move to Ubuntu Jammy (22.04)  https://review.opendev.org/c/openstack/kolla/+/83958515:48
opendevreviewMichal Nasiadka proposed openstack/kolla master: veryWIP: Move to Ubuntu Jammy (22.04)  https://review.opendev.org/c/openstack/kolla/+/83958515:50
jamesbensonSvenKieske: I can't test with correctly signed certs unfortunately, only self-signed.  I'm doing some testing now though with enabling this for self signed and necessary steps. 15:51
*** amoralej is now known as amoralej|off15:57
opendevreviewMichal Nasiadka proposed openstack/kolla master: veryWIP: Move to Ubuntu Jammy (22.04)  https://review.opendev.org/c/openstack/kolla/+/83958515:58
opendevreviewMerged openstack/kayobe master: Bump stackhpc.drac role  https://review.opendev.org/c/openstack/kayobe/+/83905617:03
jamesbensonSvenKieske: It looks like `rgw_verify_ssl true` works with self signed hosts as long as they are trusted by the hosts.  I'll change to true and add a comment about trusting certs17:03
opendevreviewMerged openstack/kayobe-config-dev master: [release] Synchronise with latest kayobe changes for Yoga  https://review.opendev.org/c/openstack/kayobe-config-dev/+/83951017:31
opendevreviewMerged openstack/kayobe-config-dev stable/yoga: [release] Synchronise with latest kayobe changes for Yoga  https://review.opendev.org/c/openstack/kayobe-config-dev/+/83951117:41
opendevreviewMichal Nasiadka proposed openstack/kolla master: veryWIP: Move to Ubuntu Jammy (22.04)  https://review.opendev.org/c/openstack/kolla/+/83958517:42
opendevreviewJames proposed openstack/kolla-ansible master: Enable SSL settings and add description.  https://review.opendev.org/c/openstack/kolla-ansible/+/83926919:41
opendevreviewJames proposed openstack/kolla-ansible master: enable SSL settings and add description  https://review.opendev.org/c/openstack/kolla-ansible/+/83963319:41
opendevreviewJames proposed openstack/kolla-ansible master: add radosgw doc to enable ceph as swift backend using secure settings.  https://review.opendev.org/c/openstack/kolla-ansible/+/83963319:42
opendevreviewRadosław Piliszek proposed openstack/kolla-ansible stable/yoga: [CI] Always use quay.io via infra's mirror  https://review.opendev.org/c/openstack/kolla-ansible/+/83954820:31
opendevreviewRadosław Piliszek proposed openstack/kolla-ansible stable/xena: [CI] Always use quay.io via infra's mirror  https://review.opendev.org/c/openstack/kolla-ansible/+/83954920:31
opendevreviewRadosław Piliszek proposed openstack/kolla-ansible stable/wallaby: [CI] Always use quay.io via infra's mirror  https://review.opendev.org/c/openstack/kolla-ansible/+/83965020:31
opendevreviewPierre Riteau proposed openstack/kayobe stable/yoga: Bump stackhpc.drac role  https://review.opendev.org/c/openstack/kayobe/+/83965121:44
opendevreviewPierre Riteau proposed openstack/kayobe stable/xena: Bump stackhpc.drac role  https://review.opendev.org/c/openstack/kayobe/+/83965221:44
opendevreviewJames Kirsch proposed openstack/kolla-ansible master: Add support for LetsEncrypt-managed certs  https://review.opendev.org/c/openstack/kolla-ansible/+/74134022:41

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!