Tuesday, 2024-02-13

jrossergood morning08:34
noonedeadpunko/08:40
noonedeadpunkoctavia looks waaaay better now btw: https://review.opendev.org/c/openstack/openstack-ansible-os_octavia/+/86846208:41
noonedeadpunkbut taking a while....08:42
jrosseryeah, i did wonder if we needed to backport the tempest concurrency patch for the upgrade jobs08:43
jrosserbut somehow we don't collect /etc for upgrade jobs so it's not possible to see how it ends up in the config file08:43
noonedeadpunk /root/workplace/etc you meant I guess...08:49
noonedeadpunkbut yes08:49
noonedeadpunkmight be08:49
jrosserwell i was looking for the user_variables and things in here https://zuul.opendev.org/t/openstack/build/efa2b66d6d0f453e928c2b1d6fe11d88/logs08:54
jrosseri'm sure we looked at this before tbh08:54
noonedeadpunkah!08:56
noonedeadpunkok, yes08:56
noonedeadpunktrue08:56
noonedeadpunkbtw. folks discovered today, that after upgrade to 22.04 haproxy does not log to journalctl anymore08:56
noonedeadpunkwith `[ALERT]    (2432049) : sendmsg()/writev() failed in logger #1: Permission denied (errno=13)`08:59
andrewbonneyWhere are you getting that message? We seem to have journal logging working fine09:00
jrosseryeah i just looked at ours and it's right there in the journal ok09:00
noonedeadpunkchown haproxy:haproxy /var/lib/haproxy/dev/log fixes it09:03
noonedeadpunksomehow, it was root:root for me....09:03
jrosserours is `srw-rw-rw- 1 root root 0 Jan 31 13:24 /var/lib/haproxy/dev/log`09:03
noonedeadpunkmine srwxr-xr-x 1 root root 0 Feb  8 15:59 /var/lib/haproxy/dev/log09:04
noonedeadpunkhuh09:04
jrosserunrelatedly rsyslog journal is just tons of error messages on this haproxy box09:07
jrosser` action 'action-3-builtin:omfile' suspended (module 'builtin:omfile'), retry 0. There should be messages before this one giving the reason for suspension. [v8.2112.0 try https://www.rsyslog.com/e/2007 ]`09:08
jrosser`action 'action-3-builtin:omfile' resumed (module 'builtin:omfile') [v8.2112.0 try https://www.rsyslog.com/e/2359 ]`09:08
noonedeadpunkhuh, and chmod +w doesn't work...09:10
jrosserthats a chroot isnt it09:17
jrosseri wonder what creates that file, and if this is all perhaps umask related or something09:17
jrosserbecause there is likley some stuff done by haproxy when it starts as root (like enter the chroot and setup $stuff) then it drops priviledges and continues as haproxy user09:18
noonedeadpunksystemctl stop systemd-journald-dev-log.socket; rm /run/systemd/journal/dev-log; systemctl start systemctl stop systemd-journald-dev-log.socket does the trick and socket created with proper permissions09:18
noonedeadpunkSo it's smth that messes up in our system with perms....09:18
*** tosky_ is now known as tosky09:48
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible-os_neutron master: Replace voting jobs with Rocky Linux  https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/90889210:54
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible-os_neutron master: Add VPNaaS OVN support  https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/90834110:54
opendevreviewMerged openstack/openstack-ansible-plugins master: Add role for provisioning default variables for install methods  https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/90817911:29
opendevreviewOpenStack Proposal Bot proposed openstack/openstack-ansible-os_masakari stable/2023.2: Updated from OpenStack Ansible Tests  https://review.opendev.org/c/openstack/openstack-ansible-os_masakari/+/90307311:57
opendevreviewOpenStack Proposal Bot proposed openstack/openstack-ansible-os_manila stable/2023.2: Updated from OpenStack Ansible Tests  https://review.opendev.org/c/openstack/openstack-ansible-os_manila/+/90307211:57
jrossernoonedeadpunk: there are now two cirros versions we setup after this patch https://github.com/openstack/openstack-ansible-os_tempest/commit/f25f81ebd1fd43e5c797bcda6845026bf209c34612:05
jrosserhow does tempest choose which to use?12:05
noonedeadpunkah......12:07
noonedeadpunkso, one should be set for alternate image 12:08
noonedeadpunkhttps://opendev.org/openstack/openstack-ansible-os_tempest/src/branch/master/templates/tempest.conf.j2#L75-L7612:08
noonedeadpunkand we set it like this: https://opendev.org/openstack/openstack-ansible-os_tempest/src/branch/master/tasks/tempest_resources.yml#L247-L24912:09
jrosseryeah i just saw12:09
noonedeadpunkI guess we used same image to save some time....12:09
jrosserhaving to rely on the ordering in the response there12:09
noonedeadpunkyeah, I don't like that much, but it's used to be like that for a while now...12:10
noonedeadpunkoh12:10
noonedeadpunkI probably missed that order can be different from time to time....12:10
noonedeadpunkDoesn't make difference for CI, but otherwise...12:11
jrosserit was in my mind that perhaps some test failures are due to using one or the other image12:11
noonedeadpunkit actually can be....12:11
noonedeadpunkas I do recall some issues related to cirros 0.612:12
jrosseri updated the capi patch etherpad a bit https://etherpad.opendev.org/p/osa-capi12:23
jrosserthere are some small things that are easy to review12:24
jrosserfinal outstanding thing is to not run the tempest role and create the public network somewhere else12:24
opendevreviewMerged openstack/openstack-ansible-os_nova stable/2023.2: Always distribute qemu config file  https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/90826513:32
*** Adri2000_ is now known as Adri200013:42
opendevreviewMerged openstack/openstack-ansible master: Add Octavia OVN provider to required repos  https://review.opendev.org/c/openstack/openstack-ansible/+/90829214:04
opendevreviewMerged openstack/openstack-ansible master: Remove galera_client from required projects  https://review.opendev.org/c/openstack/openstack-ansible/+/90832414:04
noonedeadpunkOk14:07
noonedeadpunkI guess it's time to fight some bugs?:)14:07
mgariepysure but your doodle said in an hour :P14:12
noonedeadpunkoh, lol, ok14:15
mgariepyhaha :)14:16
noonedeadpunkyes, right, in an hour14:16
noonedeadpunkThen will go and grab some lunch :D14:16
opendevreviewMerged openstack/openstack-ansible-ceph_client master: Don't load systemd parent service for object cache  https://review.opendev.org/c/openstack/openstack-ansible-ceph_client/+/90878214:28
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible-ceph_client stable/2023.2: Don't load systemd parent service for object cache  https://review.opendev.org/c/openstack/openstack-ansible-ceph_client/+/90880914:29
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible-ceph_client stable/2023.1: Don't load systemd parent service for object cache  https://review.opendev.org/c/openstack/openstack-ansible-ceph_client/+/90881014:29
opendevreviewMerged openstack/openstack-ansible-os_nova master: Fix nova device_spec to support multiple values  https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/90796914:37
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible-os_nova stable/2023.2: Fix nova device_spec to support multiple values  https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/90881114:47
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible-os_nova stable/2023.1: Fix nova device_spec to support multiple values  https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/90881214:47
opendevreviewMerged openstack/ansible-role-systemd_networkd master: Use OriginalName instead of Name in systemd.link  https://review.opendev.org/c/openstack/ansible-role-systemd_networkd/+/90752714:54
opendevreviewDmitriy Rabotyagov proposed openstack/ansible-role-systemd_networkd stable/2023.2: Use OriginalName instead of Name in systemd.link  https://review.opendev.org/c/openstack/ansible-role-systemd_networkd/+/90881314:57
opendevreviewDmitriy Rabotyagov proposed openstack/ansible-role-systemd_networkd stable/2023.1: Use OriginalName instead of Name in systemd.link  https://review.opendev.org/c/openstack/ansible-role-systemd_networkd/+/90881414:57
opendevreviewDmitriy Rabotyagov proposed openstack/ansible-role-systemd_networkd stable/zed: Use OriginalName instead of Name in systemd.link  https://review.opendev.org/c/openstack/ansible-role-systemd_networkd/+/90881514:58
noonedeadpunk#startmeeting openstack_ansible_meeting15:01
opendevmeetMeeting started Tue Feb 13 15:01:31 2024 UTC and is due to finish in 60 minutes.  The chair is noonedeadpunk. Information about MeetBot at http://wiki.debian.org/MeetBot.15:01
opendevmeetUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.15:01
opendevmeetThe meeting name has been set to 'openstack_ansible_meeting'15:01
mgariepyhey !15:01
jrossero/ hello15:01
damiandabrowskihi!15:01
noonedeadpunkWell, since our bug fighting day co-located with our regular meeting - let's combine these 2 things I guess?15:01
noonedeadpunkBut before that we have 1 thing to discuss I guess15:02
noonedeadpunkAs you might seen, unmaintained/yoga branch was created as part of the new process by TC15:02
NeilHanlono/ 15:03
noonedeadpunkright now kinda nobody have permissions there15:03
noonedeadpunkSo idea by TC was that unmaintained branches are "maintained" by separate group of individuals if they want to15:03
noonedeadpunkBut probably it makes sense to add just osa-core group to osa-unmaintained-cores?15:04
NeilHanlonthat makes sense to me15:05
NeilHanlonif the time comes that we have people really wanting to take care of those and we (current cores) don't want to, we can reevaluate ;) 15:05
mgariepyto me as well.15:05
noonedeadpunkWhy it was not done by default, is because some teams feel too accountable and resopnsible when they're assigned somewhere as cores15:05
noonedeadpunkso if we don't have cultural issues for being "responsible" for unmaintained/* I will propose to combine these 2 for now15:06
noonedeadpunkok15:06
jrossersounds ok15:06
damiandabrowski+115:06
noonedeadpunk#agreed to assign current cores to unmaintained cores15:07
noonedeadpunkok, great15:07
noonedeadpunkdoes anybody want to raise smth before bug triage starts?15:07
noonedeadpunk(except that it's time to tag 28.1.0)15:08
opendevreviewMerged openstack/openstack-ansible-plugins master: Add role to do common setup tasks for lxc containers  https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/90052915:08
noonedeadpunk#topic bug triage15:09
noonedeadpunkSo, I guess we might want to go through bugs based on their last update date15:09
noonedeadpunk#link https://bugs.launchpad.net/openstack-ansible/+bugs?orderby=-date_last_updated&start=015:09
noonedeadpunkAnd probably we can split between pages and then beginning/end of the page?15:10
NeilHanlonsounds good15:10
mgariepythere are a lot of them :) haha15:11
noonedeadpunkYeah, but we potentially can start from older ones, saying pre-202015:11
noonedeadpunkor starting 202015:12
noonedeadpunkThere's extreme chance to just close them with Fix released15:12
noonedeadpunkor Won't fix because it's about smth we've re-worked havily lately15:12
noonedeadpunkIf not sure about anything - just throw link here and let's have short discussion15:13
noonedeadpunkor.. we can have an etherpad15:13
noonedeadpunk#link https://etherpad.opendev.org/p/osa-feb-2024-bug-triage15:13
noonedeadpunkso we can organize/record better15:14
noonedeadpunkthe ones we're not sure about15:15
jrosserso things that are obviously out of date we can just deal with without discussing? or shall we record everything there?15:15
noonedeadpunkno-no, jsut record what we are unsure about15:16
noonedeadpunkas old =! invalid in some cases15:16
noonedeadpunkLike https://review.opendev.org/c/openstack/ansible-role-systemd_networkd/+/908815 was good example of 5yo buld...15:16
noonedeadpunk*bug15:16
noonedeadpunkno need to record everything15:17
mgariepysome have patches but lp wasn't updated to fix released. 15:17
noonedeadpunkyup15:17
noonedeadpunkplenty of Rocky issues actually15:21
noonedeadpunkjust for the record - we've started with 265 bugs15:22
damiandabrowskihow can we avoid verifying same bug report multiple times?15:33
damiandabrowskifor ex. person A looks at a bug report and decides that there's no update required.15:33
noonedeadpunkLet's split efforts by pages?15:34
damiandabrowskihow to avoid person B spending time on this bug report?15:34
* noonedeadpunk trying to find away to configure bugs per page15:34
noonedeadpunkbut there're 4 pages...15:35
noonedeadpunkSo you can say - working on page # from beggining/end15:35
damiandabrowskibut pages are dynamic :D if you mark few bugs with "fix released" and refresh page, you'll get different output15:36
noonedeadpunkyeah....15:37
noonedeadpunkI know.15:37
noonedeadpunkI don't have good answer unfortunatelly. 15:40
noonedeadpunkSo if you have an idea it's always welcome15:41
damiandabrowskino, i also don't see any good solution :/15:44
damiandabrowskii'll start from the top of this page: https://bugs.launchpad.net/openstack-ansible/+bugs?orderby=-importance&memo=75&start=7515:44
opendevreviewMerged openstack/openstack-ansible-os_ironic master: Fix a typo in pxe_redfish definition  https://review.opendev.org/c/openstack/openstack-ansible-os_ironic/+/90635315:50
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible-openstack_hosts master: Place sysctl options to it's own file  https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/90891215:56
NeilHanlonbit distracted fixing some networking things here, but should be able to help out more in a few minutes15:58
opendevreviewMerged openstack/openstack-ansible master: Remove tempest concurrency exception for Octavia AIO  https://review.opendev.org/c/openstack/openstack-ansible/+/90883116:03
opendevreviewMerged openstack/openstack-ansible master: Fix formatting issue for SPICE HAProxy httpcheck  https://review.opendev.org/c/openstack/openstack-ansible/+/90874616:28
opendevreviewMerged openstack/openstack-ansible-os_nova stable/2023.1: Always distribute qemu config file  https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/90826616:29
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible stable/2023.2: Fix formatting issue for SPICE HAProxy httpcheck  https://review.opendev.org/c/openstack/openstack-ansible/+/90892216:30
mgariepyback to deprecated.. https://github.com/ceph/ceph-ansible/commit/a9d1ec844d24fcc3ddea7c030eff4cd6c414d23d17:19
jrosserdo we need to end the meeting?17:40
noonedeadpunkoh, yes17:42
noonedeadpunk#endmeeting17:42
opendevmeetMeeting ended Tue Feb 13 17:42:52 2024 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)17:42
opendevmeetMinutes:        https://meetings.opendev.org/meetings/openstack_ansible_meeting/2024/openstack_ansible_meeting.2024-02-13-15.01.html17:42
opendevmeetMinutes (text): https://meetings.opendev.org/meetings/openstack_ansible_meeting/2024/openstack_ansible_meeting.2024-02-13-15.01.txt17:42
opendevmeetLog:            https://meetings.opendev.org/meetings/openstack_ansible_meeting/2024/openstack_ansible_meeting.2024-02-13-15.01.log.html17:42
jrosserwell i mainly did weeding out invalid / ancient / already fixed bugs17:44
jrosserand we are down to 174 now17:45
mgariepythat's a good progress i think.17:45
noonedeadpunkWe closed almost 100 of that17:45
noonedeadpunk*them17:45
noonedeadpunkwhich is really great17:45
noonedeadpunkbut quite some actions points in etherpad17:46
noonedeadpunkOne of most tought questions is what to do with ceph-ansible though17:46
noonedeadpunkLike we've talked about forking/simplifying it one day...17:46
noonedeadpunkBut no idea if there're still audience for that17:46
mgariepyi think ppl will notice when they will check for an upgrade.17:48
mgariepyi didn't saw it before today .. 17:48
mgariepylunch time for me17:48
noonedeadpunkyeah, we raised it couple of times....17:48
noonedeadpunkI think Jonathan spotted first17:49
jrosserwe already had a fork anyway as getting things merged upstream was pretty tough https://github.com/bbc/ceph-ansible/commits/main/17:50
noonedeadpunkare you planning to accept PRs if anything?:)17:51
jrosserwhy not :)17:51
noonedeadpunkdamiandabrowski: your thoughts as active ceph-ansible user?17:52
jrosserbut we really dont look at the testing or anything, thats another matter17:52
jrosserfork != maintainer17:53
noonedeadpunkyeah17:53
noonedeadpunkas CI is likely quite heavy17:54
damiandabrowskii'm not sure what to think about it yet :/ i need more time17:54
spatelI am having strange issue, after reboot nova-* service nova-novncproxy stopped working.. :(18:02
spatelconsole not showing anything.. 18:02
spatelnothing changed so far18:02
noonedeadpunkcould be that certificate has changed if novnc is using tls?18:02
spatelI am not running anything in TLS 18:03
spatelall simple http 18:03
spatellogs also not telling anything 18:03
spatelwhat ports nova-novnc use ?18:04
spatelwhat ports nova-novnc use on compute nodes so I can run tcpdump 18:04
noonedeadpunkum, so on computes it's a range of ports, you can get vnc port/address from virsh dumpxml for instance18:05
spatelINFO nova.console.websocketproxy [None req-b4721407-b6e4-4072-adb6-01a7c8f2becb - - - - - -] handler exception: Origin header protocol does not match this host.18:05
spatel<graphics type='vnc' port='5920' autoport='yes' listen='192.168.8.18'>18:07
noonedeadpunkbasically you should be able to connect with any VNC client given you have network access :)18:08
spatelI need token or auth right?18:08
opendevreviewMerged openstack/openstack-ansible master: Remove unused variable  https://review.opendev.org/c/openstack/openstack-ansible/+/90877818:09
spatelI did virsh console <instance> and it showing console but my vnc crap not working 18:10
spatelall I am getting this error - https://paste.opendev.org/show/bckMBQazDPx9wq0h7J2p/18:11
spatelNot sure its error or not :)18:11
noonedeadpunkand the compute is up in compute service list?18:13
noonedeadpunkand you're running Antelope?18:13
spatelYes everything up and running I am on 2023.1 release18:14
spatelI can able to create new vms and everything looks green 18:14
noonedeadpunk`Origin header protocol does not match this host.` is kinda weird...18:14
noonedeadpunkspatel: question - have you tried to live migrate VM?18:14
spatelYes I did 18:15
spatelBut only 2 vms not all 18:15
noonedeadpunkyeah, just 1 would be enough to confirm 18:15
spatelI am having issue with all 100 vms 18:15
noonedeadpunkas in case of change of auth scheme instance migration might be needed to recover console18:15
spatelwhat is the relation with live-migrate and vnc ?18:16
noonedeadpunkand did console worked after migration?18:16
noonedeadpunkhttps://opendev.org/openstack/openstack-ansible-os_nova/src/branch/master/defaults/main.yml#L334-L33918:16
spateleverything was working 2 hour ago.. 18:16
spatelI made cpu_allocation_ration and restart all compute service and boom console stopped working :(18:16
noonedeadpunkso you've explicitly disabled `nova_qemu_vnc_tls`?18:17
spatelI am running kolla not osa :) 18:18
spatelhttps://paste.opendev.org/show/bTiIvDwQ60KpwhWZ5vlR/18:18
noonedeadpunkah, well18:18
spatelI hate TLS so always keep disable 18:19
noonedeadpunkI guess it's wrong channel then lol18:19
spatellol.. but this is not deployment issue.. thought if you guys know about any bug or something else18:20
noonedeadpunkwe had that behaviour only once - when were enabling TLS for VNC18:20
noonedeadpunknever happened otherwise18:20
spatelwhat is the way to debug this crap :( 18:21
noonedeadpunkand then yes -restarting nova-compute would break instance console until you migrate it out18:21
noonedeadpunkand after live migration console was revived18:21
spatelBut I just live-migrate single vm not 100s :)18:23
spateljust single vm can create entire cloud VNC? 18:23
noonedeadpunkand console worked after migration for it?18:23
spatelbefore live migrate everything work working..( Its hard to know this happened because of live migrate or I made change in nova for cpu overcommit and restart services) 18:24
spatelI don't know what event cause this issue.. 18:24
spatelI have restarted all the service again but same issue 18:24
spatellet me google handler exception: Origin header protocol does not match this host and understand error 18:25
mgariepyhmm wtf ?https://docs.openstack.org/project-deploy-guide/openstack-ansible/draft/19:14
NeilHanlonlol wat19:15
mgariepyreally not sure where that page points to haha19:18
noonedeadpunkwhaaaaat19:55
noonedeadpunkyeah, that's really very weird.19:56
noonedeadpunkI assume, this can be actually somewhere on openstack-manuals19:57
noonedeadpunkie there's some redirect for ironic: https://opendev.org/openstack/openstack-manuals/src/branch/master/www/.htaccess#L27519:59
noonedeadpunkand... it;'s totally in a sitemap20:00
noonedeadpunkhttps://opendev.org/openstack/openstack-manuals/src/branch/master/www/static/sitemap.xml#L26630720:00
mgariepydigging old bugs makes us discover stuff.20:04
noonedeadpunkyup20:04
mgariepydocumentation is always out of sync.20:04
noonedeadpunksometimes I'd rather not know about it... But it's great eventually20:04
noonedeadpunkYeah, but docs is not everything that has been discovered I guess20:05
noonedeadpunkbut yeah, huge chunk20:05
mgariepyyep20:05
mgariepyhttps://bugs.launchpad.net/openstack-ansible/+bug/180334720:08
mgariepylol20:08
noonedeadpunklol20:09
noonedeadpunkI do recall how it was created20:09
noonedeadpunkIIRC it was made during PTG20:09
noonedeadpunkLikely - Berlin 201820:09
mgariepyi'll close it. 20:09
mgariepyunless you tell me not to close it haha20:10
noonedeadpunkI think we kinda fixed it partially20:10
noonedeadpunkIdea there was about br-vxlan and br-storage20:10
noonedeadpunkwhich are really not needed on computes20:10
mgariepyno ?20:11
noonedeadpunkinterfaces are, but not bridges20:11
noonedeadpunklike no sense to have br-vxlan which will be added to ovs bridge?20:11
noonedeadpunkbut I think we've already mentioned in couple of places that these can be interfaces, not really bridges...20:13
noonedeadpunkso you probably should feel free to close indeed20:13
spatelnoonedeadpunk here is the error and not sure what is wrong :( - https://paste.opendev.org/show/baZjIpTcibpksgajegav/20:15
spatelI can directly connect VNC server with vnc client from desktop20:16
noonedeadpunkno idea, rly20:21
noonedeadpunksounds like hostname issue or smth like that20:21
jrosseror the scheme is wrong http vs https compared to what the server things it is doingg20:27
jrosseror even vnc:// vs http(s)://20:28
spatelhmm but everything was working before. This is not something just deployed 20:48
spatelI am totally clueless here :(20:49
spatelDamn! problem solved 21:09
spatelnovncproxy_base_url =  21:09
spatelfeels like we did change this in past but didn't cook into ansible override 21:09
spatelre-run playbook knock them out 21:09
spatelafter 8 month what a mess 21:09

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!