Tuesday, 2024-01-16

tonybclarkb: Yup I'm still interested.  Today is supposed to be a holiday for you.  Would you prefer to do it tomorrow?00:06
Clark[m]Sure we can do tomorrow 00:22
opendevreviewMerged openstack/project-config master: Add netdata onfiguration repo to NebulOuS  https://review.opendev.org/c/openstack/project-config/+/90529305:44
*** benj_9 is now known as benj_07:41
opendevreviewJan Marchel proposed openstack/project-config master: Add new projects to NebulOuS: ontology-server, brokerage-quality-assurance-server, service-level-agreement-generator  https://review.opendev.org/c/openstack/project-config/+/90565709:04
opendevreviewJan Marchel proposed openstack/project-config master: Add new projects to NebulOuS: ontology-server, brokerage-quality-assurance-server, service-level-agreement-generator  https://review.opendev.org/c/openstack/project-config/+/90565710:10
opendevreviewJan Marchel proposed openstack/project-config master: d new projects to NebulOuS: ontology-server, brokerage-quality-assurance-server, service-level-agreement-generator  https://review.opendev.org/c/openstack/project-config/+/90565710:16
rtomczykWe have some issues with neutron-vpnaas libreswan driver. What is the best way to get some community support on it ?10:35
opendevreviewJan Marchel proposed openstack/project-config master: Add new projects to NebulOuS: ontology-server, brokerage-quality-assurance-server, service-level-agreement-generator  https://review.opendev.org/c/openstack/project-config/+/90565710:36
fungiyoctozepto: i assume you were okay with 905293 merging? i should have left a note on it to hold approvals until you had time to confirm14:18
funginot sure if rtomczyk was expecting an answer out of band, since they seem to have left the channel one minute after asking their question14:19
fungiif they're reading this log, https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack.org/ is usually where people ask questions about neutron-vpnaas or other openstack components14:20
fungior in the #openstack-neutron irc channel here on oftc14:22
opendevreviewMerged openstack/project-config master: Add new projects to NebulOuS: ontology-server, brokerage-quality-assurance-server, service-level-agreement-generator  https://review.opendev.org/c/openstack/project-config/+/90565714:54
opendevreviewSeongsoo Cho proposed openstack/project-config master: Add weblate api key in zuul secret  https://review.opendev.org/c/openstack/project-config/+/90570116:13
clarkbI plan to mention this in our meeting today during open discussion, but I've been thinking about trying OpenDev ptg time again. Less office hours for other projects to engage with us (thoguh we can probably do some of that) and more time to focus on some of the tech debt we've been digging up (afs mirroring and nodepool test node images, zuul sql db hosting, etc)17:06
clarkbMentioning it now so that people have a bit of time to formulate opinions on that before the meeting17:06
opendevreviewMerged opendev/system-config master: Remove bullseye python3.11 image builds  https://review.opendev.org/c/opendev/system-config/+/90501819:51
clarkbI'm going to figure out lunch now then go for one last walk in the snow before it turns to ice. tonyb does sometime after 2300 UTC work for the linaro cloud cert stuff?19:59
tonybclarkb: perfect.20:00
tonybWith reference to: https://review.opendev.org/c/openstack/project-config/+/90570120:05
tonybIs there a way infra-root can verify that secret has been encrypted correctly/will decrypt and be useful in jobs?20:07
fungitonyb: https://docs.opendev.org/opendev/system-config/latest/zuul.html#secrets but we generally don't bother20:09
tonybfungi: Thanks.  I feel obliged in this case as it's the first time I've helped create said secret, so I'm being extra careful/supportive20:15
fungitonyb: as for mutt access to the shared inbox, here's a redacted subset of my config with the relevant options. fair warning, i use mutt's folder-hook option to switch out multiple muttrc files depending on which mailbox i switch to, so this isn't a complete muttrc: https://paste.opendev.org/show/bBdZP6e9WlefjKjUmwo8/20:16
tonybfungi: Thanks.20:16
tonybfungi: when I used mutt I did the same thing with folder-hooks20:16
fungiyeah, i have mine hooked to 3 different accounts at separate providers currently20:17
fungimuch more convenient than running separate mutt processes for every account20:18
tonybIt's super handy.20:21
fungiwhich, to be honest, is what i did before i learned about folder-hook20:24
tonybIt's the easy way to get started.  I used to have separate configs/processes when I had two accounts.  When I added a 3rd account I looked for a better way .... enter folder-hook20:26
fungialso saved a fair amount of resident memory on the 1gb vm where i run my mua, irc client and calender/reminder software20:27
tonybI noticed that jq isn't installed on several of the production servers.21:12
tonybWhat's the best way to get that added on "all the servers" .... I'm assuming there is an existing task to do that $somewhere21:13
tonybin the meantime any objection to me installing jq on zuul02 ?21:14
tonybLike ... adding jq to: opendev/system-config/playbooks/roles/base/server/vars/Debian.yaml21:17
SvenKieskegood evening from europe21:53
SvenKieskeanybody has any idea why https://pypi.org does stop responding _during_ the TLS handshake when called via curl? this is on a single machine of a customer21:53
SvenKieskeother curl TLS requests to other domains work from the very same machine21:54
SvenKieskeit stops at: TLSv1.3 (OUT), TLS handshake, Client hello (1):21:54
SvenKieskeso right at the start21:54
SvenKieskeand sorry for the slightly off topic question, but I figured we have some experienced people here.21:55
SvenKieskeI still think this is something local21:55
SvenKieskeother server has no problem :/21:55
SvenKieskethere's a pfsense firewall involved..mhm let's ask google21:58
clarkbSvenKieske: is this system too old to do SNI?21:58
clarkbSvenKieske: pypi disabled non SNI stuff a while back21:59
clarkbbut I would do `openssl s_client -connect pypi.org:443` and see what that says21:59
SvenKieskeit's ubuntu 20.04 based22:00
tonybSvenKieske: do the working and not working systems share the same networking config?  Any tunnels or VPNs involved?22:00
clarkb20.04 should be new enough to SNI22:00
clarkbIts xenial era that was a problem I think22:00
SvenKieskewe currently suspect some MTU misconfiguration with the pfsense22:01
tonybSvenKieske: that's kinda why I asked about the networking config22:02
SvenKieskeyeah, thanks so far, currently working our way through pinging with no fragment header set :D22:09
SvenKieskeoriginally we wanted to upgrade an openstack installation..oh well..22:09
tonybSvenKieske: you can't just set the MTU to something "low" on the problem machine to verify?22:10
tonybSvenKieske: or enable PMTU discovery?22:10
clarkbI don't recall needing to do anything special for MTUs on my pfsense router22:12
clarkbtonyb: I forwarded you the email with background on linaro cloud cert renewal steps. I'm happy to simply answer questions via irc as you do that or hop on a call and do it more paired programming style22:18
SvenKieskeI guess I could, it's a customers machine and I'm merely consulting/helping them22:18
SvenKiesketonyb: in my experience PMTU discovery rarely works, do you had success with it in the past? I must admit I didn't really try it that often after being discouraged by stuff I read about it and advice from older networking gurus :D22:19
clarkbthe problem with PMTU that you often hit around openstack is that only L3 "devices" can participate22:20
clarkbOpenStack environments often have many many interfaces and "devices" that are either virtual l1 or l2 equivalents and they all have MTUs that can't be adjusted via PTMU22:21
SvenKieskeanother problem I have found multiple times is, that you need ICMP for automatic mtu discovery and network engineers seem to really like blocking ICMP everywhere.22:24
SvenKieskebut whatever, I hope my problem is solved by this, thanks all for your help :)22:25
fungiyes, blocking icmp error responses will break pmtud and create mtu black holes. never block icmp22:30
fungithe rest of tcp/ip doesn't work without icmp. if you really don't want machines participating on the internet, unplug them22:31
fungiat least that's what i always told my customers as the "in-house security expert" when i worked for a hosting company22:32
clarkbfollowing up on the stream multiple packages using too much disk thing we currently have 5 thunderbird packages mirrored22:37
clarkbeahc one is just over 100MB22:37
clarkbsimilar story with firefox22:38
clarkbthis is for centos 9 stream22:38
clarkbcurrently waiting for the sort by size update request to load22:39
clarkbI can see having two copies a current and a pervious but it isn't clear to me why you would need 522:39
clarkbtonyb: also left some short notes on the wiki etherpad section22:45
fungias did i22:59
ianwhrm, https://review.opendev.org/c/opendev/system-config/+/885557?tab=change-view-tab-header-zuul-results-summary doesn't look as i'd hope for the openafs installation23:13
ianwthe 9-stream log shows that the kernel module build worked -> https://224b312f67b64e43bc51-0845cf503ed6ea2ba7b6515ac4a85fb6.ssl.cf1.rackcdn.com/885557/1/check/system-config-zuul-role-integration-centos-9-stream/6f209f3/dkms-make-logs/var/lib/dkms/openafs/1.8.10-1.el9/5.14.0-407.el9.x86_64/x86_64/log/make.log23:14
ianwbut then in the messages https://224b312f67b64e43bc51-0845cf503ed6ea2ba7b6515ac4a85fb6.ssl.cf1.rackcdn.com/885557/1/check/system-config-zuul-role-integration-centos-9-stream/6f209f3/messages.txt23:14
ianwJan 16 21:29:51 np0036427074 systemd[1]: Starting OpenAFS Client Service...23:14
ianwJan 16 21:29:51 np0036427074 modprobe[82159]: modprobe: FATAL: Module openafs not found in directory /lib/modules/5.14.0-404.el9.x86_6423:14
ianwCC [M]  /var/lib/dkms/openafs/1.8.10-1.el9/build/src/libafs/MODLOAD-5.14.0-407.el9.x86_64-SP/afspag.mod.o23:16
ianwit build the module for kernel 5.14.0-407 ... but it can't find a module for 5.14.0-404 ... which suggests to me that this is runing 5.14.0-404 kernel but has headers for 507 ...23:16
clarkbianw: possibly because we did a system update prior to building but without a reboot?23:17
clarkbtonyb: let me know what you think re cert renewal process. Happy to jump on a call but in about an hour and a half I'll have to step away for a bit for dinner23:24
clarkbbut also happy to just have you work through it and use irc if you prefer.23:25
ianwwe might be in a period where a new kernel has released but we haven't updated and rebooted the hosts.  or, the image building is broken and the hosts aren't being updated because the new image isn't created :/23:28
ianwthe arm64 versions worked, so that's good at least23:29
tonybclarkb: I'm available now(ish) to look at the cert updates.  I don't have a preference for call vs IRC23:30
tonybIRC makes it easier to multitask so lets go with that23:31
clarkbsounds good. Hopefully what I sent makes sense23:32
clarkbthe school just sent an email but it is a newsletter not another day of cancellations23:32
clarkbthe worst offender in centos stream 9 multiple packages for the same major release of software appaers to be openjdk 11. There are 10 packages ranging from 246M to 319M depending on the specific minor version23:38
clarkbdotnet sdks, firefox, thunderbird, libreoffice, grafana, texlive, gcc and on and on. The tail is actually quite long here23:39
clarkbI have a very strong suspicion this is why we're growing rapidly in those mirrosr though23:39
clarkbthey seem to append to the distro and not clean up the older stuff23:40
clarkbI wonder if pruning that is feedback they would be receptive to.23:40
clarkbEven if they keep older packages around it seems like it would be reasonable to not do so super long term?23:41
clarkbtrying to look in other locations (that was the AppStream location)23:42
clarkbunder BaseOS the big offender is linux kernel related stuff. Firmware and debug modules and so on23:43
clarkbNeilHanlon: ^ you might know why that is done?23:44
ianware we sure we're not running with rsync flags that don't prune?23:49
clarkbwe pass --delete23:53
clarkbreading the rsync manpage if you try and sync foo/* then --delete doesn't work but we don't do that23:54
clarkb"Prior to rsync 2.6.7, this option would have no effect unless --recursive was enabled.  Beginning with 2.6.7, deletions will also occur when --dirs (-d) is enabled, but only for directories whose contents are being copied." Maybe that?23:56
clarkbwe do pass -r nevermind23:57

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!