Monday, 2021-08-23

*** rpittau|afk is now known as rpittau07:51
opendevreviewMark Goddard proposed openstack/kolla-ansible master: Support Ansible max_fail_percentage  https://review.opendev.org/c/openstack/kolla-ansible/+/80559808:23
opendevreviewMark Goddard proposed openstack/kolla-ansible master: nova: don't fail all hosts on service registration failures  https://review.opendev.org/c/openstack/kolla-ansible/+/80559908:23
kevkomorning :)08:24
opendevreviewRadosław Piliszek proposed openstack/kolla-ansible stable/ussuri: Fix regression in image pull action v2  https://review.opendev.org/c/openstack/kolla-ansible/+/80560208:29
opendevreviewMark Goddard proposed openstack/kayobe-config-dev master: Use kolla_internal_vip_address instead of aio_vip_address  https://review.opendev.org/c/openstack/kayobe-config-dev/+/80524008:30
opendevreviewMark Goddard proposed openstack/kolla-ansible stable/wallaby: Fix haproxy precheck when kolla_externally_managed_cert is used  https://review.opendev.org/c/openstack/kolla-ansible/+/80556008:32
opendevreviewMark Goddard proposed openstack/kolla-ansible master: Remove chrony role from kolla  https://review.opendev.org/c/openstack/kolla-ansible/+/79174308:56
opendevreviewMichal Nasiadka proposed openstack/kolla master: haproxy: Use haproxy 2.2.x  https://review.opendev.org/c/openstack/kolla/+/80451309:33
mnasiadkamorning09:35
jingvarmorning09:39
kevko\o/09:42
opendevreviewVerification of a change to openstack/kolla stable/wallaby failed: Add some system users to fix user uid inconsistencies  https://review.opendev.org/c/openstack/kolla/+/80513409:43
opendevreviewRadosław Piliszek proposed openstack/kolla-ansible stable/ussuri: Fix regression in image pull action v2  https://review.opendev.org/c/openstack/kolla-ansible/+/80560210:24
opendevreviewMerged openstack/kayobe master: Add twentyFiveGigE to port descriptions in inspector rules  https://review.opendev.org/c/openstack/kayobe/+/80270910:41
opendevreviewwu.chunyang proposed openstack/kolla master: Remove chrony image from kolla  https://review.opendev.org/c/openstack/kolla/+/79175310:51
opendevreviewMichal Arbet proposed openstack/kolla master: Move clustercheck script to mariadb-base  https://review.opendev.org/c/openstack/kolla/+/80561311:12
Fl1ntGoooood whatever chan!!!!11:27
Fl1ntRadek, are you there?11:29
Fl1ntQuestion about the Erlang Bump from 22 to 23, what's the reasoning behind that?11:29
Fl1nton RabbitMQ11:29
Fl1nt:D11:29
Fl1ntcause for those using the CentOS Messaging repo, it's not yet available :-(11:30
opendevreviewMichal Arbet proposed openstack/kolla-ansible master: Use Docker healthchecks for mariadb-server service  https://review.opendev.org/c/openstack/kolla-ansible/+/80561611:33
kevkoanyone have some time for some reviews :) ? 11:39
opendevreviewMark Goddard proposed openstack/kolla-ansible stable/victoria: Add Neutron DHCP agent to OVN networking setup  https://review.opendev.org/c/openstack/kolla-ansible/+/80556111:39
kevkoFl1nt: don't know ..but maybe have erlang same in all images ? (bullseye has 1:23.2.6+dfsg-1 in repo https://packages.debian.org/bullseye/erlang-base)11:45
opendevreviewMerged openstack/kolla-ansible stable/victoria: Use more RMQ flags for less busy wait  https://review.opendev.org/c/openstack/kolla-ansible/+/80527211:57
opendevreviewMerged openstack/kolla-ansible stable/ussuri: Use more RMQ flags for less busy wait  https://review.opendev.org/c/openstack/kolla-ansible/+/80527312:00
opendevreviewMerged openstack/kolla-ansible stable/wallaby: Use more RMQ flags for less busy wait  https://review.opendev.org/c/openstack/kolla-ansible/+/80527112:02
opendevreviewMichal Nasiadka proposed openstack/kolla master: haproxy: Use haproxy 2.2.x  https://review.opendev.org/c/openstack/kolla/+/80451312:11
Fl1ntkevko, yeah probably but still aarch64 is still on 22 as it doesn't yet get any 23 release, so would have been more generic or standard to use 22 everywhere isn't?12:27
Fl1ntAnyway, I'll override on our template-override :D12:27
Fl1ntoh, sweet clever RMQ Patch :D12:29
Fl1ntok understand the 23 release now12:29
opendevreviewMerged openstack/kolla stable/wallaby: Add some system users to fix user uid inconsistencies  https://review.opendev.org/c/openstack/kolla/+/80513412:33
kevkois this working ? https://etherpad.opendev.org/p/kolla-mentorship-signup :) ? 12:39
mnasiadkajudging by the list of names, this is really old...12:43
kevkoyeah, i thought :D 12:44
yoctozeptothe history shows it's from 201712:44
kevkobut, is there something similar ? or kolla is closed ? 12:44
mnasiadkaif there's a need for something similar, we could revive it - please add a topic to the next meeting and we can discuss it12:45
yoctozeptohow is kolla closed?12:45
yoctozeptowe are mentoring you all the time, kevko12:45
kevko:D 12:45
kevkowell, i was thinking about to join core team... so I was trying to find some document, procedure whatever - what is needed to be done .. but what I found is only that one document 12:47
mgoddardkevko: I think I have said this before, but... how to join core team? Demonstrate understanding of the project and trustworthiness through doing code reviews.12:51
mnasiadkakevko: it's not something you can sign up to, like a certification :)12:51
mgoddardkevko: from what I have seen you would be a suitable candidate, but we would need more examples of code reviewing12:53
mgoddardkevko: we have these slides, which show various levels of involvement: https://docs.google.com/presentation/d/1YyRbp-UW9EXTtG9_sTaRov13X6V8FjCztj9c0EkT754/edit12:54
mgoddard"Demonstrate responsibility, understanding & care"12:54
yoctozeptoI am with mgoddard on this one; you certainly are a good candidate but we need more reviews from you; cores focus on reviews and project maintenance, not necessarily feature implementation12:55
yoctozeptomgoddard: perhaps we should link this up in a few places around?12:55
mgoddardkevko: it would be good to see you in the top 10 reviewer list: https://www.stackalytics.io/?module=kolla-group12:55
yoctozeptoI like our creation12:55
mgoddardkevko: although that is not the whole story: quality >> quantity12:56
yoctozeptomgoddard, kevko: ++, and not with ratio too close to 100% because those are often too artificial :P12:56
yoctozeptoyeah, that's about that quality part there12:56
mgoddardtbh, a -1 is more useful than a +112:57
Fl1nthum... I think this stackalytics thing may well loose few stats :D12:57
Fl1ntI'm not on the whole classement ^^12:57
yoctozeptomgoddard: my thinking indeed12:57
mgoddard-1 when warranted of course :)12:58
Fl1ntthe 37th one is having 1 review and is on the list :p12:58
yoctozeptoFl1nt: you are missing entirely?12:58
mgoddardFl1nt: you are on https://www.stackalytics.io/report/contribution?module=kolla-group&project_type=openstack&days=3012:58
mgoddard#3912:58
Fl1ntMy colleague Martin is on the list and I'm not he's having 1 review and I'm having way more ^^12:58
yoctozeptoI can see your 7 in total Fl1nt12:59
yoctozeptobut none on Xena12:59
yoctozeptosomehow they were not counted against the current release13:00
Fl1ntyeah, exactly, just saw you were using Xena filter ^^ I'm only contributing on master and then backport :D13:00
yoctozeptoremember these are reviews13:00
yoctozeptoXena is master ;-)13:00
yoctozeptothese are reviews13:00
yoctozeptonot change proposals13:00
Fl1ntooooh ok :D13:00
yoctozeptohttps://www.stackalytics.io/?module=opendev.org/openstack/kolla-ansible&release=xena&metric=patches13:01
Fl1ntYeah, I'm bad at reviewing stuff, I'm usually having hard time finding something to say ^^13:01
yoctozeptohere you are in xena13:01
yoctozeptoFl1nt: ack, no problem13:02
mgoddardreviewing takes practice13:02
mgoddardbut it gets easier13:02
Fl1ntDoing my best when I'm highlighted tbh :D13:02
mgoddardyour input is appreciated Fl1nt 13:02
yoctozeptoI'm dropping kolla-cli13:02
yoctozeptomgoddard, Fl1nt ++13:02
yoctozeptomgoddard: I meant kolla-cli from the slides13:03
mgoddardyoctozepto: fire++13:03
mgoddardoh13:03
Fl1ntlike with the SAML2/OIDC patch for instance, but sometimes I'm having a hard time writing in a way it doesn't sounds pedentic to the contributor :(13:03
mgoddardyoctozepto: I think we still need to drop the project13:03
Fl1nt++ yoctozepto 13:03
yoctozeptomgoddard; yeah, it's been only deprecated I think13:03
yoctozeptofrom Victoria precisely13:03
kevkosorry, was smoking ..going to read :) 13:03
yoctozeptoso it should be gone entirely with ussuri going away13:03
yoctozeptosomething for the new ptl to consider ensuring happen13:04
mgoddardkevko: oh, sorry - no smokers! :p13:04
yoctozeptomnasiadka: ^13:04
mnasiadkadrop kolla-cli... huh, well, whatever - I'll add it to my list.13:05
mgoddardmnasiadka: PTL pro-tip: delegate_to13:05
mnasiadkamgoddard: I didn't say I'm going to do it, just add to list and find a volunteer ;-)13:06
mgoddardthat's the spirit13:06
yoctozeptodelegate but ensure it's done ;-)13:08
kevkohmm, so i will do reviews, but I'm glad I broke the silence all he day with my question :D 13:14
yoctozepto:D13:14
yoctozeptoif you have any comments regarding the slides, they are welcome too13:14
yoctozeptoI like what we have done there13:15
yoctozeptoI was delivering that presentation and its followup13:15
yoctozeptothose were the times...13:15
opendevreviewMerged openstack/kolla-ansible master: Add ability to retry image pulling  https://review.opendev.org/c/openstack/kolla-ansible/+/80382813:22
Fl1ntTalking about reviews :D13:22
Fl1ntCould someone had a look at: https://review.opendev.org/c/openstack/kolla-ansible/+/80008013:22
opendevreviewRadosław Piliszek proposed openstack/kolla-ansible stable/wallaby: Add ability to retry image pulling  https://review.opendev.org/c/openstack/kolla-ansible/+/80556513:27
opendevreviewRadosław Piliszek proposed openstack/kolla-ansible stable/victoria: Add ability to retry image pulling  https://review.opendev.org/c/openstack/kolla-ansible/+/80556613:27
opendevreviewRadosław Piliszek proposed openstack/kolla-ansible stable/ussuri: Add ability to retry image pulling  https://review.opendev.org/c/openstack/kolla-ansible/+/80556713:27
opendevreviewRadosław Piliszek proposed openstack/kolla-ansible stable/ussuri: Add ability to retry image pulling  https://review.opendev.org/c/openstack/kolla-ansible/+/80556713:27
opendevreviewMaksim Malchuk proposed openstack/kolla master: Unify curl use (customisation options)  https://review.opendev.org/c/openstack/kolla/+/80564013:28
opendevreviewRadosław Piliszek proposed openstack/kolla-ansible stable/ussuri: Fix regression in image pull action v2  https://review.opendev.org/c/openstack/kolla-ansible/+/80560213:37
opendevreviewRadosław Piliszek proposed openstack/kolla-ansible stable/ussuri: Add ability to retry image pulling  https://review.opendev.org/c/openstack/kolla-ansible/+/80556713:37
yoctozeptomgoddard: ^ ansible quirks episode 7861113:38
yoctozeptohad to fix ansible linter13:39
yoctozeptochecked it's all fine now13:39
yoctozeptowell, more like work around the linter13:39
yoctozeptobut you get the point13:39
opendevreviewMaksim Malchuk proposed openstack/kolla stable/victoria: Unify curl options  https://review.opendev.org/c/openstack/kolla/+/80564313:44
mnasiadkaSo, we're using the linter just to work it around? :)13:45
opendevreviewMaksim Malchuk proposed openstack/kolla stable/ussuri: Unify curl options  https://review.opendev.org/c/openstack/kolla/+/80564413:46
yoctozeptomnasiadka: if it has bugs...13:46
yoctozeptoI wonder how it worked outside of the block13:47
yoctozeptoas it should be hitting the very same code13:47
yoctozeptoMEH13:47
yoctozepto:D13:47
yoctozeptokill it with fire before it lays exceptions13:47
yoctozepto^ an old adage13:47
opendevreviewGaël THEROND proposed openstack/kolla-ansible master: Add missing cloudkitty documentation.  https://review.opendev.org/c/openstack/kolla-ansible/+/80565014:07
opendevreviewMerged openstack/kolla master: ironic-neutron-agent: enable for Debuntu binary  https://review.opendev.org/c/openstack/kolla/+/79961114:22
opendevreviewMerged openstack/kolla master: Allow build debian/ubuntu-binary cloudkitty images  https://review.opendev.org/c/openstack/kolla/+/79743314:22
opendevreviewMerged openstack/kolla master: Config: Set default home to /var/lib/{user}  https://review.opendev.org/c/openstack/kolla/+/80268514:25
opendevreviewGaël THEROND proposed openstack/kolla-ansible master: Add missing cloudkitty documentation.  https://review.opendev.org/c/openstack/kolla-ansible/+/80565014:51
opendevreviewMerged openstack/kolla-ansible stable/victoria: Add Neutron DHCP agent to OVN networking setup  https://review.opendev.org/c/openstack/kolla-ansible/+/80556114:55
opendevreviewGaël THEROND proposed openstack/kolla-ansible master: Add missing cloudkitty documentation.  https://review.opendev.org/c/openstack/kolla-ansible/+/80565015:11
opendevreviewwu.chunyang proposed openstack/kolla-ansible master: Modernize the swift role  https://review.opendev.org/c/openstack/kolla-ansible/+/79749815:19
ignaziocassanoHello All, I am new in kolla deplpyment. I read every week new sources images are available. I have just pulled and pushed new images (date is 31 hours ago) on my local registry then I run kolla-deply but on computes and controllers the images are stil 8 days ago old 15:26
ignaziocassanoMust I run before kolla pull for computes and controllers ?15:27
ignaziocassanoPlease, any help ?15:28
mnasiadkaignaziocassano: If you're using the same tag  for new images - you need to run `pull`15:31
ignaziocassanomnasuadka: an then I must renu deploy ?15:32
ignaziocassanomnasuadka: an then I must run deploy ?15:32
ignaziocassanoI tired it but redeploying with updated images, stop all my instances15:34
ignaziocassanoThis is my procedure: pull  and push new images on local registry. Pull new images from local registry to compute and controllers nodes. Deploy15:36
ignaziocassanoOn an installation with the same openstack version (wallaby) it stops all runiing virtual machines) :-(15:37
mnasiadkacheck libvirt log, that shouldn't be the case...15:39
kevkoyoctozepto: btw, have you some time to review proxysql thing ? :) 15:57
kevkoor, do someone have a time to review ? 15:57
*** rpittau is now known as rpittau|afk15:58
ignaziocassanomnasiadka: please, I would like to ask if my procedure is correct:   1 pulled new images on my local registry 2 pushed new images on my local registry 3 pushed new images from my local registry to computes and controllers nodes (kolla-ansuble pull --limit control,compute. 4 kolla-ansible deploy16:01
mnasiadkaignaziocassano: that's correct, if your vms are getting stopped - there is some problem somewhere, you should check most probably in libvirt/nova-libvirt log16:04
ignaziocassanoI will check it. Thanks16:04
ignaziocassanomnasiadka: seems some errors when it redeploy libvirt container 2021-08-23 15:32:38.221+0000: 2288093: error : virNetSocketReadWire:1833 : End of file while reading data: Input/output error 16:09
opendevreviewGaël THEROND proposed openstack/kolla-ansible master: Add missing cloudkitty documentation.  https://review.opendev.org/c/openstack/kolla-ansible/+/80565016:11
ignaziocassanomnasiadka: may be masakary ? During the update the following appears in masakari log:Exception caught: crmadmin command output unexpected host status.: Exception: crmadmin command output unexpected host status.  16:20
ignaziocassanoMust I disable masakari before updating ?16:20
mnasiadkaignaziocassano: might be, yoctozepto ^^ something for you16:20
kevkoignaziocassano: haha, that's the bug which i fixed already 16:21
kevkoignaziocassano: what images do you use ? debian ? 16:21
ignaziocassanokevko: ubuntu source16:21
kevkoignaziocassano: yeah, it's the same ..16:21
kevkoi mean ..same issue 16:22
ignaziocassanokevko: what must I do ?16:22
kevkoignaziocassano: well, you posted here bug which is already fixed (but probably image is not available yet)16:22
ignaziocassanokevko: sorry as I wrote before I am new in kolla. How can I check when the image withi your fix is available ?16:23
kevkoignaziocassano: but it's only masakari fix .. i don't know if your problem is related .. (because from time to time i had some turned off instances during the upgrade - but not all)16:24
kevkobut probably it will be related 16:25
kevkoyou can switch images in configuration ..or build new images 16:25
ignaziocassanokevko: what exactly I must do for testing it ?16:26
ignaziocassanokevko: Or can I try a workaround.....Can I disable masakary before updating ?16:27
kevkoignaziocassano: it will be best if you build images locally and upload to your docker repository 16:27
kevkoignaziocassano: stable/wallaby kolla repo should have that patches included 16:27
kevkoignaziocassano: btw, you can check this bug report https://bugs.launchpad.net/kolla/+bug/1934753 16:28
kevkoignaziocassano: you can also stop masakari,hacluster containers ..and upgrade with --tag or use --skip-tags16:29
kevkoyou can try and verify if it is masakari problem 16:29
ignaziocassanokevko: I will read the bug. Meanwhile next week I will try a workaround. I must disable the masakari segment befor updading o stopping the masakari containers ?16:30
ignaziocassanokevko: many thanks. You wrote the answer while I was writing :-)16:30
kevkowell, as I said ..i don't know if it is masakari issue ..but yes I can imagine that masakari is reacting to lost libvirt process via processmonitor ..but it's buggy .. so it will die somewhere in the middle 16:32
kevkoso, if you think it should be your issue .. 1. stopping masakari,hacluster containers on your nodes 2. Be sure that you are upgrading without hacluster,masakari tags/roles16:33
kevkoyou can also build your images from latest stable/wallaby branch 16:34
ignaziocassanokevko: thanks. I will try it, but when I try to upgrate only masakari it should works, in a second step16:41
kevkoyeah, but if you will not replace images ..you will have still buggy/not-functional masakari 16:43
yoctozeptokevko, ignaziocassano: that issue was not affecting ubuntu, only debian, and only binary for some reason...16:45
yoctozeptoduring update it's normal that services report other services going down16:46
kevkoyoctozepto: nope, ubuntu also 16:46
kevkoyoctozepto: i think 16:46
yoctozeptokevko: nope, it worked16:46
yoctozeptoignaziocassano: have you checked the logs of masakari-engine? has it actually done any action?16:46
kevkoyoctozepto: ah, yes, i was little bit blind ..output is different 16:47
ignaziocassanoException caught: crmadmin command output unexpected host status.: Exception: crmadmin command output unexpected host status. 2021-08-23 17:35:46.582 7 WARNING masakarimonitors.hostmonitor.host_handler.handle_host [-] 'tst2-osctrl01' is unstable state on cluster.: Exception: crmadmin command output unexpected host status.16:47
ignaziocassanothis is during update16:47
yoctozeptoyeah, that's fine16:47
ignaziocassanoyoctozepto: that's fine ? Do you mean I  met the bug ?16:48
yoctozeptoignaziocassano: no, I mean the problem must be somewhere else16:49
kevkobtw, yoctozepto what is your upgrade procedure, are u upgrading in one batch or live-migrating instances somewhere else ..upgrade compute and live-migrate to upgraded node ...upgrade another ..etc16:50
kevkobecause I also saw turned-off instances, but not all at once (during upgrade)16:50
ignaziocassanoNo, I am updating without moving instances 16:51
ignaziocassanoone step only16:51
ignaziocassanoI ha only two instances on the same node.16:52
ignaziocassanoBut the issue happened also two months ago16:52
ignaziocassanoAnd if you remember another user wrote about this issue 2 months ago, if I remember well16:53
yoctozeptokevko: if I don't need to update the kernel (so I don't need to reboot), then I simply roll out the containers updates starting with one node to check if it does not break the world16:53
yoctozeptoper flavour and then just roll out all the others16:53
yoctozeptoignaziocassano: I've already asked you to check masakari-engine logs as that is the service that is taking actual actions16:54
yoctozeptoif it acted upon something, then we need to figure why16:54
yoctozeptoif it didn't, then we need to find another cause16:55
kevkoyoctozepto: so you never saw stopped instance after upgrade ? 17:00
kevkoyoctozepto: btw, will you have time for proxysql ? so I can fix things at evening :D 17:01
yoctozeptokevko: no, never17:02
yoctozeptothis should not be happening17:02
yoctozeptokevko: no, I am as sorry as you are even if not more :-(17:02
yoctozeptoI like the change but too many higher priorities now17:03
yoctozeptoI need my own team ;D17:03
opendevreviewPiotr Parczewski proposed openstack/kayobe master: Add OpenStack release name suffix to base_path  https://review.opendev.org/c/openstack/kayobe/+/80569619:57
opendevreviewPiotr Parczewski proposed openstack/kayobe master: Add OpenStack release name suffix to base_path  https://review.opendev.org/c/openstack/kayobe/+/80569620:07
-opendevstatus- NOTICE: The Gerrit service on review.opendev.org has been restarted for a patch version upgrade, resulting in a brief outage21:42
*** gilou_ is now known as Gilou21:43
*** yoctozepto4 is now known as yoctozepto23:37

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!