Tuesday, 2021-10-26

fungihopefully the docs volume will be done replicating in the next few minutes02:06
fungiit just finished. running again to make sure it's a ~noop02:30
fungiit's taking longer on the rerun than i would have expected02:32
fungiokay, finished much faster that time02:55
fungithe flock is now cancelled02:57
fungiand i'm watching logs for the next cron pass02:58
fungithe tarballs volume is taking a while to catch up03:22
opendevreviewIan Wienand proposed openstack/diskimage-builder master: [wip] regenerate initramfs with FEDORA-2021-e4843341ca  https://review.opendev.org/c/openstack/diskimage-builder/+/81538503:32
*** ykarel|away is now known as ykarel04:44
opendevreviewIan Wienand proposed openstack/diskimage-builder master: fedora-container: regenerate initramfs for F34  https://review.opendev.org/c/openstack/diskimage-builder/+/81538505:09
opendevreviewIan Wienand proposed openstack/diskimage-builder master: dracut-regenerate: drop Python 2 packages  https://review.opendev.org/c/openstack/diskimage-builder/+/81540905:09
ianwclarkb/fungi: ^ that's about my best guess for how to make a bootable f34 image05:09
*** pojadhav is now known as pojadhav|ruck05:34
*** ysandeep|out is now known as ysandeep06:28
opendevreviewAnanya proposed opendev/elastic-recheck rdo: DNM: WIP: ER bot with opensearch for upstream  https://review.opendev.org/c/opendev/elastic-recheck/+/81325006:54
fricklerianw: this seems to be hard failing the arm64-bionic jobs. maybe add a "mkdir -p"? https://review.opendev.org/c/openstack/diskimage-builder/+/814081/9..13/roles/dib-functests/tasks/main.yaml06:54
opendevreviewdaniel.pawlik proposed openstack/project-config master: Add project openstack/ci-log-processing  https://review.opendev.org/c/openstack/project-config/+/81526007:01
opendevreviewAnanya proposed opendev/elastic-recheck rdo: Fix ER bot to report back to gerrit with bug/error report  https://review.opendev.org/c/opendev/elastic-recheck/+/80563807:08
opendevreviewAnanya proposed opendev/elastic-recheck rdo: Fix ER bot to report back to gerrit with bug/error report  https://review.opendev.org/c/opendev/elastic-recheck/+/80563807:24
*** jpena|off is now known as jpena07:31
ianwfrickler: hrm, https://review.opendev.org/c/openstack/diskimage-builder/+/814846 switches things to bullseye07:37
ianwi'm just going to approve that through, as the other func tests and the nodepool image are all now bullseye07:39
opendevreviewMerged openstack/diskimage-builder master: Switch ARM64 testing to bullseye  https://review.opendev.org/c/openstack/diskimage-builder/+/81484608:54
*** ykarel is now known as ykarel|lunch08:55
*** ysandeep is now known as ysandeep|lunch09:44
*** ysandeep|lunch is now known as ysandeep10:33
*** ykarel|lunch is now known as ykarel10:56
*** dviroel|rover|out is now known as dviroel|rover11:11
*** jpena is now known as jpena|lunch11:24
*** tosky_ is now known as tosky11:25
damiandabrowski[m] (review02.opendev.org) is listed b.barracudacentral.org DNSBL, so some of the mail servers using this blacklist(like mine) may not be able to receive emails from review.opendev.org.11:28
damiandabrowski[m]Can I ask someone authorized to look at this and send removal request? https://barracudacentral.org/rbl/removal-request11:28
fungi#status log Requested removal of review.opendev.org's IPv4 address from the barracudacentral.org RBL11:43
opendevstatusfungi: finished logging11:43
fungidamiandabrowski[m]: thanks for the heads up, hopefully it's just a one-time thing and the same user doesn't re-report gerrit notifications as spam11:44
*** pojadhav|ruck is now known as pojadhav|brb11:45
fungiBBR21635248847-93103-2709 is our confirmation number for that removal request11:45
damiandabrowski[m]thank You!11:46
fungia couple of amusing things i noticed... their rbl search doesn't support ipv6 addresses, and also after submitting the removal request they recommend listing e-mail emitting domains with https://www.emailreg.org which is entirely unresponsive. makes me suspect the barracuda spam filtering "service" may be in a semi-abandoned state, or at least derelict11:48
*** ysandeep is now known as ysandeep|brb12:03
*** pojadhav|brb is now known as pojadhav|ruck12:04
*** jpena|lunch is now known as jpena12:25
*** ysandeep|brb is now known as ysandeep12:25
fricklerfungi: related question: do we check for bounces from review.o.o? I guess mail-admins might consider continually sending mails to adresses that no longer exist also to be spam12:41
fungifrickler: i don't think gerrit has a feature for that, but we could probably set up something once we have the account collisions fixed12:49
fricklerfungi: grep 550 /var/log/exim4/mainlog|cut -f5 -d\ |sort|uniq -c|wc => 130, with 1400 msgs in total. the question would be what action to take, disable the affected accounts completely might be a bit harsh?12:59
fungiyes, we could probably batch something to remove the e-mail addresses, from their accounts, but i don't know what gerrit will do in such cases13:01
fungialso we wouldn't want to take action on a single bounce, we'd need some sort of tracking to look for multiple bounces and no successful deliveries in recent history13:02
*** Guest3656 is now known as redrobot13:03
opendevreviewHervĂ© Beraud proposed openstack/project-config master: Allow to check release approval for DPL teams  https://review.opendev.org/c/openstack/project-config/+/81549713:08
opendevreviewHervĂ© Beraud proposed openstack/project-config master: Allow to check release approval for DPL teams  https://review.opendev.org/c/openstack/project-config/+/81549713:09
opendevreviewMichal Nasiadka proposed openstack/diskimage-builder master: Add dnf versionlock support  https://review.opendev.org/c/openstack/diskimage-builder/+/81194513:21
*** lbragstad6 is now known as lbragstad13:28
opendevreviewMerged opendev/elastic-recheck rdo: Changing no of days for query from 14 to 7  https://review.opendev.org/c/opendev/elastic-recheck/+/81379513:53
*** sshnaidm_ is now known as sshnaidm14:03
clarkbfungi: frickler: removing emails from accounts is dangerous due to gerrit's consistency checking. That could cause us to create new accounts for people when they next log in. Instead I suspect the two options we can look at are disabling accounts (and then work with people who discover their accounts are disabled to fix them) or removing the preferred email address from the account14:35
clarkbsettings (this would need testing but I think gerrit may not send email if you don't have a preferred email set and the preferred email setting isn't as highly scrutinized by gerrit for account creation on login)14:35
clarkbrelated the user that has a problem with an email conflict did get back to me and confirmed they want us to clean up the new email from the old account allowing gerrit to make a new account for them. I'll look into that after some morning meetings. If someone else wants to give that a go I'm willing to help too14:36
fungiyeah, whatever avenue we take will need testing14:36
clarkbI've approved the ci-log-processing repo creation change14:42
opendevreviewMerged openstack/project-config master: Add project openstack/ci-log-processing  https://review.opendev.org/c/openstack/project-config/+/81526014:55
venkatakrishnatHi, can someone please suggest me the correct channel to discuss about Devstack stack related issues15:03
fungivenkatakrishnat: #openstack-qa is where the devstack maintainers hang out15:03
venkatakrishnat@fungi, thank you15:04
*** ysandeep is now known as ysandeep|out15:08
*** pojadhav|ruck is now known as pojadhav|out15:15
*** ykarel is now known as ykarel|away15:16
*** marios is now known as marios|out15:54
*** jpena is now known as jpena|off16:33
opendevreviewYuriy Shyyan proposed openstack/project-config master: Reinstating InMotion cloud  https://review.opendev.org/c/openstack/project-config/+/81553217:56
opendevreviewMartin Kopec proposed opendev/system-config master: Adjust RefStack build for osf->openinfra rename  https://review.opendev.org/c/opendev/system-config/+/80848018:00
opendevreviewMerged openstack/project-config master: Reinstating InMotion cloud  https://review.opendev.org/c/openstack/project-config/+/81553218:18
clarkbfungi: https://keys.openpgp.org/about also seems to be a thing19:13
clarkboh wait that is what emailed us so I think you just typo'd in the other channel19:14
fungier, yep i mistyped in the meeting19:15
opendevreviewJeremy Stanley proposed opendev/system-config master: Update artifact signing key management process  https://review.opendev.org/c/opendev/system-config/+/81554719:42
opendevreviewJeremy Stanley proposed openstack/project-config master: Replace old Xena cycle signing key with Yoga  https://review.opendev.org/c/openstack/project-config/+/81554819:53
ianwfungi: "something like a caff-style signature approval mechanism" .. oh wow, nostalgia ... it's probably been 15 years since I last attended a mass key-signing and apt-get installed signing-party20:00
ianwit was so long ago that we all put up our license/passports on a big projector in the room and nobody even thought about "someone could take a high quality photo of this"20:01
fungibut the point is that caff basically gives the keyholder control over choosing which third-party signatures get published20:03
fungiwhich could alleviate the previous pollution issues with unsolicited keysig pollution20:04
fungier, previous security issues20:04
fungithe keys.openpgp.org maintainers have indicated they'd be willing to support something like that, but somebody needs to do the work to implement it20:05
fungiso it's not as if publication/distribution of third-party keysigs will never be a thing again, there's just no good option available for it right now20:06
opendevreviewYuriy Shyyan proposed openstack/project-config master: Increasing max servers  https://review.opendev.org/c/openstack/project-config/+/81555220:18
yuriyslooking good today20:18
fricklerfungi: well we could publish a txt export of our collective signatures somewhere locally. while not quite as convenient for users, it may be comforting for some, like distro managers maybe?20:20
fungifrickler: yeah, i think we can iterate on it, i just wanted to get the process to a usable state again so we could rotate keys as they're overdue20:22
fungiwe'd definitely need a bit more orchestration around it if we want to continue publishing our individual keysigs somewhere20:22
*** dviroel|rover is now known as dviroel|rover|afk20:42
fungilooks like 815552 would be good to put some more stress on the inmotion-iad3 environment21:35
clarkbfungi: yuriys done21:36
fungihttps://grafana.opendev.org/d/4sdNjeXGk/nodepool-inmotion is showing error-clean so far21:37
yuriysyep, and ive confirmed the nova-scheduler/placement rando properties are helping quite a bit21:37
yuriysbig thanks to fungi and melwitt for the research material on optimizing that21:38
corvusclarkb: i'm seeing some gerrit http timeouts in the zuul scheduler log21:40
corvusit just got a lot better21:40
corvuszuul may have been exhibiting sytmptoms of either a slow gerrit or bad zuul<->gerrit network21:41
fungii see periodic gerrit http(s) timeouts from gertty too21:41
corvusthen probably the former21:41
clarkbcorvus: ok, so hopefully a transient issue and it will catch backup on the backlog of events now21:41
fungii've been suspecting issues reaching it, possibly limited to ipv621:41
clarkbload average isn't bad on review02 right now21:41
clarkbwhich is the symptom we've seen in the past for slow gerrit (huge load average spikes)21:42
clarkbI guess we continue to monitor then but nothing worth panicing over yet :)21:42
fungii also get semi-frequent errors running git-review, where it acts like it times out trying to establish an ssh connection (again over ipv6)21:42
fungibut yeah, i assumed it was a network issue in my house21:42
fungior with my isp's ipv6 routes21:43
johnsomI am also noticing some slowness today. Also on IPv6. No errors, just slow to load. I am doing reviews, I will leave the dev console open and see if I get any hints.21:47
corvusfungi: plot twist:  gerrit was in the house the whole time!21:48
opendevreviewMerged openstack/project-config master: Increasing max servers  https://review.opendev.org/c/openstack/project-config/+/81555221:55
fungiyuriys: ^ it has deployed in the last few minutes22:39
yuriysYep! watching all the things :)22:43

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!