Tuesday, 2021-07-20

*** cloudnull9 is now known as cloudnull00:44
opendevreviewIan Wienand proposed opendev/system-config master: gerrit: fix Launchpad credentials write  https://review.opendev.org/c/opendev/system-config/+/80122700:54
opendevreviewIan Wienand proposed opendev/system-config master: [dmn] testing mariadb connector  https://review.opendev.org/c/opendev/system-config/+/80141001:43
opendevreviewIan Wienand proposed opendev/system-config master: Remove review01 references  https://review.opendev.org/c/opendev/system-config/+/80141201:58
ianwclarkb: https://gerrit-review.googlesource.com/c/gerrit/+/312602 fixes the mariadb connector issue. is up with the fix, but it doesn't do anything exciting; just doesn't give you 500 errors when looking at files :)06:23
*** amoralej|off is now known as amoralej06:56
*** rpittau|afk is now known as rpittau07:06
*** odyssey4me is now known as Guest148808:33
*** ksambor is now known as ksambor|lunch10:01
*** ksambor|lunch is now known as ksambor10:53
*** odyssey4me is now known as Guest149811:38
*** amoralej is now known as amoralej|lunch13:10
*** amoralej|lunch is now known as amoralej14:05
opendevreviewDanni Shi proposed openstack/diskimage-builder master: Add a keylime-agent element and a tpm-emulator element  https://review.opendev.org/c/openstack/diskimage-builder/+/78960114:06
clarkbianw: upstream gerrit just merged your mariadb connector fix \o/ It isn't clear to me if we need to push that patch to stable-3.3, stable-3.4, and master now though14:54
clarkbmordred: corvus: ^ do you grok their forward/backward porting system and do you know if they'll do that for you or not?14:54
*** dviroel is now known as dviroel|lunch14:54
corvusclarkb: my understanding is someone will do it if you don't; but if you do it's appreciated and may happen faster.  basically just 'git merge' to each branch in series14:57
corvusso master -> stable$latest -> $stable$latest-1 etc14:57
clarkbcorvus: oh ok so you push merge commits and not cherrypicks? I'll look into that after some breakfast since in theory I'll need to do it for my change too if/when it lands14:58
corvuscorrect, merges14:58
clarkbcorvus: ya I think that must be correct and that is why their git history is always interesting to look at14:58
corvusclarkb: looks like they include 'git log --oneline' in the merge commit msgs too.  look for the string "Merge branch" in git history for an example14:59
corvusi don't know if there's a tool to do that automatically, or they just copy/paste14:59
clarkbinteresting. I wonder if they have a developr doc on this15:00
clarkbI'll dig around, this helps ensure I'm looking for the right stuff though15:00
mordredcorvus: it's master -> stable$latest and not stable$oldest -> stable$nextoldest -> ... master?15:02
clarkbmordred: I think the arrows are git dag direction which implies $stable$latest-1 is oldest in that series15:03
mordredI'm sure you're right - just trying to remember last time I did this15:03
mordredah. I was thinking of it in terms of order of operations for dev15:03
mordredI think they are "land on oldest applicable branch then merge forward" as opposed to openstack which is "land on master then cherry-pick backwards"15:04
clarkbya and in theory that sounds great as it ensures you aren't forgetting to fix bugs, but it makes for really odd histories. I always find it difficult to determine if a particular patch is included in any release without checking it out locally (the gitiles type tools all seem to fail me)15:05
clarkbnot the end of the world to fetch it locally though15:06
corvusmordred, er yeah, oldest to newest, sorry15:11
fungiseems like it merged initially to stable-3.215:16
fungiso i guess that's the oldest relevant branch15:16
clarkbfungi: well I think that is what ianw pushed to15:17
clarkbcertainly old enough for us :)15:17
clarkb"Fixes that are known to be needed for a particular release should be pushed for review on that release's stable branch. They will then be included into the master branch when the stable branch is merged back.15:20
clarkbreading elsewhere they say "start on the newest `stable-*` branch that you can run" as a general rule of thumb for improvements and bug fixes15:23
clarkbI've found at least one example in the git log where they made a change to master and then cherry-picked it backward in a way we are more familiar with15:25
clarkb(good to know that is an option)15:25
clarkbcorvus: I'm noticing that their merge rollups don't seem to go through gerrit?15:28
clarkbAlso looks like they may try to bundle up more than one change when merging forward. I guess that means we can wait for my change to land before worrying about doing ianws15:28
*** dviroel|lunch is now known as dviroel16:00
*** marios is now known as marios|out16:22
*** rpittau is now known as rpittau|afk16:45
*** amoralej is now known as amoralej|off16:50
clarkbinfra-root I updated the maintenance.html on review01 to help make things more clearly for people who may have DNS hardcoded to the old server18:01
fungioh, great idea18:03
clarkbthe gerrit user cleanup notes I've been taking have been moved to the new server as well. I'll need to do a check over the other stuff there to see if I want to copy anything else (I knew I wanted those files)18:17
*** cloudnull9 is now known as cloudnull18:19
fungifor the gerrit upgrade, did we blow away the accountpatchreviewdb content?19:48
clarkbfungi: yes19:48
fungior whatever it's CALLED19:48
fungigah, capslock19:48
fungihas anyone even noticed? ;)19:49
clarkbnot that I've heard19:49
clarkbI did explicitly test it afterwards just to be sure no new weird behavior popped up and that is how I found the issue with the mariadb connector19:49
clarkbdansmith has reported the occasional slow page load (I'm guessing that is related to caches which is why I mentioend we can tune those more now) but also says submitting changes seems to be snappier19:51
clarkboverall I think the server is running in a much happier way19:51
ianwi actually now think this issue with the connector ignoring dups is related to the failed import19:52
clarkboh! ya it could be19:52
clarkbsince you were using the gerrit built in migration tool ya?19:52
clarkbat this point I think its fine. The value is minimal and very ephemeral19:53
clarkbemail sent about the yum-puppetlabs mirror19:55
ianwi think the main benefit is that we are masters of our own database and can easily point ansible where we need it to go19:55
fungioh, here's one significant hurdle with dropping stretch... https://opendev.org/opendev/base-jobs/src/branch/master/zuul.d/nodesets.yaml#L5119:58
fungithough not a lot of references to debian-stable in job configs on master branches either, thankfully20:00
clarkbfungi: I guess step 0 is bumping that to buster with a note bullseye will follow shortly?20:02
fungior getting rid of that alias20:03
clarkbor that20:04
clarkbI think yum-puppetlabs hit its quota limit a couple of months ago and no one noticed. That is why it hasn'ed updated since sometime in may20:06
ianwheh, yeah, those lines converge20:24
fungiokay, i've sent the dropping stretch proposal to openstack-discuss20:24
opendevreviewClark Boylan proposed opendev/system-config master: Trim yum-puppetlabs content  https://review.opendev.org/c/opendev/system-config/+/80152920:34
clarkbthat is based on feedback from mwahahah on my email. I'll ask them to review it quickly too before we land anything20:35
ianwoh the other thing i noticed was https://bugs.launchpad.net/ubuntu/+source/glibc/+bug/1926918 had broken package installs on review0221:38
ianwi'll poke around on our focal systems to see if they have a similar issue21:39
clarkbianw: I started looking at that since you had it on the etherpad. Did it get fixed on review02 already? and now we have to address the others ptentially? what was the fix downgrading glibc?21:40
ianwthe fix was a manual downgrade, which then allowed an upgrade past the broken libc package21:41
ianwi've checked a few focal systems and they don't seem affected21:42
ianwok i think i understand the problem better now.  2.31-0ubuntu9.3 was released and withdrawn, so focal-updates only has 2.31-0ubuntu9.222:02
ianwthe problem is if you're on a system that has 9.3 libc installed, then you try to install libc-dev (such as via build-essential, which we tried to do when we pulled in borg, because we need to build fuse from source)22:03
ianwmany of our systems have 9.3 installed via automated updates, but most have pulled along libc-dev with them (e.g. executors that require that for building dkms, etc)22:04
ianwso if i can come up with something that finds hosts with 2.31-0ubuntu9.3 libc installed, but !libc-dev, they are in the potentially broken state22:06
clarkbianw: should we remove 9.3 everywhere if it was withdrawn?22:07
clarkb(I'm not sure how painful that will be and maybe we juts wait for 9.4?)22:07
ianwyeah, i guess that when they put something back in focal-updates, they will make sure it has a higher version22:07
ianwok, here is the list of servers that are potentially broken if they (or something they depend on) try to install libc6-dev currently -> https://paste.opendev.org/show/807610/22:32
ianwthe two options are a) downgrade them to 9.2, the latest release in -updates or b) ignore this, assuming they won't need that package until ubuntu upstream release a new libc >9.322:33
clarkbianw: we should probably consider why 9.3 was pulled when evaluating ?22:34
clarkbif it is a major issue then probably worth downgrading22:34
ianwit think it was pulled by unattended-upgrades22:39
clarkbianw: oh I meant why was it pulled from the repository22:39
clarkbpresumably there was something wrong with it22:39
clarkbthough I guess if it was really bad they would have released a 9.4 to replace it with the old code?22:44
ianw"caused regressions" from the bug ... trying to see if i can find exactly what it was22:45
ianwi think it might actually have to do with WSL (!)22:45
ianwThe fixed version has been moved back to focal-proposed due to regressions caused by LP: #1914044.22:51
ianw[SRU] gstreamer fails with "cannot allocate memory in static TLS block" error on aarch64 Edit22:52
ianwThis fix proved to be too intrusive to be released via focal-updates. It requires an immediate restart of services and a reboot 22:53
ianw... I'm preparing 2.31-0ubuntu9.4 _without_ this fix but including other fixes which are safe to release to Focal.22:54
ianwi think we have a good handle on what's actually wrong now, and i've condensed my research into https://bugs.launchpad.net/ubuntu/+source/glibc/+bug/1926918/comments/11 because this problem is all over and not explained terribly well23:08
ianwi think we just leave things alone, wait for 9.4 packages, and in the very unlikely case we need to pull in something that depends on libc6-dev be aware of the broken servers and our options23:09
fungithat sounds fine to me based on what you've quoted so far23:11
fungiany idea how long that's been in limbo waiting for the updated package?23:12
ianwthe "i'm preparing" message was  2021-05-2023:18
clarkbmaybe we can drop a comment there and ask if this is still the plan?23:19
ianwi can't see any other alternatives.  i'm very surprised deleting a package from a stable release was an option23:24

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!