Wednesday, 2021-06-23

ianwmy preference might be to do that, to avoid ongoing confusion00:00
fungiyeah, they're all in afs, i suppose i could just grep -r to build up a file list and then sed them or something00:00
ianwi don't mind doing that seeing as i created the problem00:00
fungiwell, it's also not as urgent as correcting the config. i don't mind working on it myself, but would likely save it for tomorrow00:01
ianwi can do it, i think it makes the most sense 00:03
ianwmy thinking was that although the logs are stored on, we should just hide that behind (hence the proxy there)00:03
ianwand if the long-term future of eavesdrop-ing is not eavesdrop01 (matrix bot, etc) then we just create a static copy of the logs for archival on afs 00:04
fungiyep, i understand and agree with the design00:04
ianwi'll see that change through and modify the incorrect meeting logs, thanks for finding it!00:05
fungianyway, if you want to work on fixing up the meeting logs and/or channel logs feel free, but don't feel solely responsible. i'm sure if i looked back at the review which updated that file i likely reviewed if not approved it00:05
fungii'm happy to help more with fixing it up too if you've got other things going on00:06
ianwno worries00:09
ianwunless mordred or someone can tell me how to make the query above not overwhelm the trove server, i don't even know how i'd clean up the existing account_patch_reviews db00:10
ianwergo, at this point, unless someone has other ideas i think it will drop transitioning it from the review update checklist procedure00:11
fungiyeah, i think if it gets reset to an empty table when we move the service that's entirely fine00:12
fungiwe should probably make sure to include that impact in communications, but i have a feeling it won't really affect our users significantly00:13
fungimost of them likely never even noticed it tracked that, and of the ones who did and relied on it, they probably only care about the current reviews they're looking at, so it'll catch back up enough for their needs in hours if not days00:14
ianwthat reminds me, isn't in hte backup list.  i think we should add that00:16
ianwis the incorrect logs00:22
ianw is the fixup plan00:26
fungiianw: also i saw that gitea01 backups have been failing. not urgent since we haven't added new repo redirects lately, but i haven't had time to look into it yet00:29
ianwmysqldump: Error 2013: Lost connection to MySQL server during query when dumping table `action` at row: 696800:32
ianwi think we have seen this before a bit00:32
opendevreviewIan Wienand proposed opendev/system-config master: Add to backup group
fungiianw: looking at your grep output, i guess there are no absolute urls embedded in the html files? also that's just the meeting logs right? should we do channel logs too?00:51
ianwyeah, i think the text just points to the html00:55
ianwbut the html doesn't have any links00:55
ianwi'll grep the channel logs now to see if there's anything other than "this address isn't working" or this discussion :)00:56
ianwohh, the comments in the end of the meeting, i see what you mean00:59
opendevreviewMerged opendev/system-config master: Correct the meeting base URL for our meetbot
opendevreviewMerged opendev/system-config master: centos-mirror: add dry run mode
opendevreviewMerged opendev/system-config master: centos-mirror: exclude ppc64le
fungiianw: right, what the bot echoes to the channel. that's how it originally came up, someone tried to follow the link to the minutes that the bot reported in-channel02:53
ianwfungi: ok, all the files where opendevmeet has incorrectly stamped things are @
ianwi think what i'll do is merge and apply that backup change, and run backups.  then i can do the sed replacements but feel good we have redundant copies03:16
opendevreviewMerged opendev/system-config master: Add to backup group
*** diablo_rojo is now known as Guest26704:00
ianwletsencrypt failed because of nb0104:01
*** ykarel|away is now known as ykarel04:37
opendevreviewMerged openstack/project-config master: Add publish-to-pypi job template for tripleo-operator-ansible
*** marios is now known as marios|ruck05:23
opendevreviewIan Wienand proposed opendev/system-config master: borg-backup: exclude /var/lib/snapd
ianw#status log cleaned up and rebooted nb01/nb0205:40
opendevstatusianw: finished logging05:40
ianwi am running a manual run of backs on eavesdrop05:41
opendevreviewIan Wienand proposed opendev/system-config master: review02: enable replication
opendevreviewIan Wienand proposed opendev/system-config master: backups: add
*** ysandeep|out is now known as ysandeep05:52
*** odyssey4me is now known as Guest27305:55
ianwWed Jun 23 06:02:08 UTC 2021 Backup finished successfully06:26
ianwok, i will do the sed replacements now06:26
ianw#status log manually corrected links in recent meetings/logs on to; see
opendevstatusianw: finished logging06:44
ianwjust to be clear on the files i updated; &
*** ChanServ changes topic to "OpenDev is a space for collaborative Open Source software development | | channel logs"06:48
*** jpena|off is now known as jpena06:56
*** rpittau|afk is now known as rpittau07:07
zbrwhat is the best irc channel to discuss pbr issues?07:23
*** bhagyashris_ is now known as bhagyashris08:27
*** ykarel is now known as ykarel|lunch08:52
*** ykarel|lunch is now known as ykarel09:56
fricklerinfra-root: the bot expanding bug urls seems down/gone, not sure which one that was. I'm in meetings today so cannot check myself11:28
*** jpena is now known as jpena|lunch11:30
*** dviroel|out is now known as dviroel11:37
*** marios|ruck is now known as marios|ruck|call12:00
*** ysandeep is now known as ysandeep|afk12:26
*** jpena|lunch is now known as jpena12:33
*** marios|ruck|call is now known as marios|ruck12:47
fungifrickler: it was the meetbot, so now opendevmeet. it was switched from the abandoned supybot framework to limnoria a couple weeks ago, so maybe we're missing the bug expansion plugin? when did you notice it last working?12:50
fungizbr: officially, #openstack-oslo since pbr is an oslo deliverable12:50
zbrfungi: thanks. so far I identified 3 places for a fix: cpython, setuptools and the newer distutils, a joy.12:54
fungii have a feeling there will be little interest in fixing distutils, since it's fully deprecated now12:55
zbrkindof, being taken out. but what was not clear to me was which copy is used by pbr itself, as it would much easier to vender our fixed copy in pbr12:56
fungiprobably pbr should have a plan for replacing its use of distutils. i count 6 modules in it (aside from tests) which import distutils or something from it12:57
* zbr is looking for a link...12:58
mordredyeah - we probably need to make a pbr modernification plan12:58
fungi"In Python 3.12, distutils will no longer be installed by make install or any of the first-party distribution. Third-party redistributors should no longer include distutils in their bundles or repositories."12:59
fungihowever it goes on to say that setuptools may continue to provide a copy itself12:59
zbrusually i would drop use of pbr, but i still need it for packages that install stuff in no-kosher places, like installing ansible roles (which is not-supported by the modern packaging)12:59
mordredI would not drop use of pbr :)13:00
zbrin fact setuptools has two copies of the broken code inside (for some reason), but i have a patch to fix it, after the reference distutils project is patched.13:00
mordredbut I do agree - it's going to need to learn how to co-exist with the new python packaging ecosystem sooner or later13:00
mordredI haven't checked recently - but my understanding of where things are going is that we should be able to reach a point where we can have a pbr that doesn't use setuptools under the covers in any way13:01
mordredwhich would likely make several of the things pbr wants to do much easier to accomplish (rather than needing to monkeypatch things that dont' necessarily want to be extended :) )13:02
fungilike acting as a pep-517 build backend?13:02
zbr is the line that ends-up calling the buggy bits13:02
mordredfungi: yeah13:03
mordredfungi: my brain kindof has had a pause in the back waiting for that to be ready13:03
*** ysandeep|afk is now known as ysandeep13:04
zbri do not know pbr internals, but based on debuggin include_pattern seems to point to `distutils.filelist.include_pattern`13:06
mordredalso - I haven't dug in far enough to see whether we can get away with not having requirements or version listed in pyproject.toml yet13:06
mordredzbr: interesting that the recursive symlink issue only expresses on osx and not on linux13:07
zbrin fact it does impact both13:07
zbrbut the experience may be different from person to person, it is not even infinite because it stops when it reaches maxpath length.13:08
zbrbut the feeling is that is infinite, with 100% CPU usage too.13:08
mordredisn't the prefix=ei_cmd.egg_info part of that call supposed to be telling it to find all of the files in the egg_info dir?13:09
mordredthat dir is usually quite small - so I wonder if someting in that invocation is not doing that any longer13:10
zbrthink about editable installs, that method ends up looking are git root repo, at least that is what I seen13:12
fricklerfungi: that's a good question, I was pretty sure it had been working not so long ago, but checking my logs, the latest working response I could find was on the 10th
opendevreviewFlorian Haas proposed opendev/git-review master: Doc updates for core.hooksPath support
mordredzbr: ^^ that's where ei_cmd.egg_info *should* be being set - I wonder if in your editable context it's getting set to something else13:15
mordredzbr: do you have a setup where it would be possible to print out what ei_cmd.egg_info is?13:16
mordred(on the line right before the include_pattern call)13:16
fungifrickler: that sounds like it might coincide with the switch to limnoria in that case: "2021-06-11 06:23:10 UTC meetbot/logging now running from limnoria on"
mordredzbr: also - the class we're overriding there has changed upstream13:18
mordredzbr: maybe also try replacing the include_pattern call with:13:18
mordredand see if that solves your issue? (mostly - you have a reproducible env set up - so if updating the call to the new upstream call fixes it - that'll be good!)13:19
mordred FTR13:19
zbrapparently it worked, but i also had to install wheel package (probably unrelated)13:29
zbrtbh, i did print the ei_cmd.egg_info and it pointed to a temp directory (already cleaned by the time command finished)13:30
mordredbut the graft worked?13:32
zbrwell, i did not test if the install was really correct, but at least pip reported success13:33
mordredcool. it should be - that method is an override of the one from setuptools and the content is copied in (there isn't actually a proper hook point so we have to copy-and-extend) - updating it to match what the setuptools method is doing is almost certainly the correct thing to do13:34
zbrmordred: now that I look at implementation of graft, i have reason to say that is also prone to the same bug:
zbrin the end all of them endup calling -- which we know chokes with symlinks-loops13:38
mordredyah - but maybe for now it's not a thing we need to solve because it fixed it for you and it's what setuptools itself is doing14:03
*** ysandeep is now known as ysandeep|out14:14
*** marios|ruck is now known as marios|ruck|call14:31
*** sboyron is now known as Guest32214:44
*** marios|ruck|call is now known as marios|ruck14:51
*** ykarel is now known as ykarel|away15:20
*** gthiemon1e is now known as gthiemonge15:49
*** marios|ruck is now known as marios|out15:55
*** jpena is now known as jpena|off16:00
*** rpittau is now known as rpittau|afk16:08
*** frenzy_friday is now known as anbanerj|rover16:19
*** sshnaidm is now known as sshnaidm|afk16:35
opendevreviewDmitriy Rabotyagov proposed openstack/project-config master: Deprecate OpenStack-Ansible nspawn repositories
opendevreviewDmitriy Rabotyagov proposed openstack/project-config master: Deprecate OpenStack-Ansible nspawn repositories
opendevreviewDmitriy Rabotyagov proposed openstack/project-config master: Remove noop jobs for OpenStack-Ansible nspawn projects
opendevreviewShnaidman Sagi (Sergey) proposed openstack/project-config master: Allow RDO CI on ansible-collections-openstack
sshnaidm|afktristanC, mnaser ^^ please take a look18:51
opendevreviewShnaidman Sagi (Sergey) proposed openstack/project-config master: Allow RDO CI on ansible-collections-openstack
opendevreviewGhanshyam proposed openstack/project-config master: End project gating for retiring scientific-wg repo
opendevreviewGhanshyam proposed openstack/project-config master: Remove scientific-wg from infra
opendevreviewGhanshyam proposed openstack/project-config master: Remove arch-wg from infra
opendevreviewGhanshyam proposed openstack/project-config master: Remove openstack-specs from infra
opendevreviewGhanshyam proposed openstack/project-config master: Remove project-navigator-data from infra
opendevreviewGhanshyam proposed openstack/project-config master: Remove enterprise-wg from infra
opendevreviewGhanshyam proposed openstack/project-config master: Remove workload-ref-archs from infra
opendevreviewGhanshyam proposed openstack/project-config master: Remove ops-tags-team from infra
opendevreviewGhanshyam proposed openstack/project-config master: Remove governance-uc from infra
opendevreviewGhanshyam proposed openstack/project-config master: Remove uc-recognition from infra
*** dviroel is now known as dviroel|out21:13
opendevreviewGhanshyam proposed openstack/project-config master: End proejct gating for puppet-openstack-specs
opendevreviewGhanshyam proposed openstack/project-config master: Remove puppet-openstack-specs from infra
ianwfungi / frickler: yeah, there's some NOTE type things in the Dockerfile about putting that plugin back in22:23
JayFwhoops, apologies22:29
corvusfungi: did we lose the zuul secret decrypting script on the server move?22:30
fungiJayF: that's one very long password you're going to need to reset, or a very dexterous cat22:30
JayFyubikey that has a super extra sensitive button22:31
corvusmy guess is OG yubikey?22:31
fungicorvus: hmm... maybe, though the server may still exist as well. if you need the pypi creds they're also in our usual list22:31
JayFit's a USB-C yubikey, which has the capacitive button on both side, so if you have it in your laptop and shift just right, it activates :|22:31
corvus(there's a limited character set for keyboard compat)22:31
fungii'll check if the old server can be revived22:31
corvusfungi: don't worry, i think i have a local copy :)22:32
fungicorvus: okay good, because the only zuul* server in rax-dfw now is zuul02.opendev.org22:33
corvusfungi: copied back to zuul02:~root22:33
corvuswe could probably check that into zuul's tree.22:33
fungiyeah, not a bad idea, though it might also be simplified by exposing zuul's decrypt function as an entrypoint22:34
fungibut we have a number of random utility scripts in tools/ if memory serves, so would fit fine in there22:35
corvusconsidering it requires a zuul install, the entrypoint option may be best actually22:36
fungialternative is to do it with openssl, but that's more fiddly22:40
fungii suppose if we add a zuul-decrypt entrypoint, it would make sense to have it be able to fetch keys from zk as well in preparation for when we no longer maintain on-disk copies22:47
opendevreviewIan Wienand proposed openstack/project-config master: nodepool: pause gentoo and tumbleweed builds
ianwprometheanfire / frickler : ^ i've put some pastes of failures in the above change, full logs are @
ianwunfortunately, i haven't seen the dib gate pass in a few days.  i'll investigate now but i don't think we have a quick path to deploying a change right now23:30
ianwit looks like on centos8-stream "2021-06-21 10:54:17.806 | >   Running scriptlet: grub2-tools-1:2.02-99.el8.x86_64                    73/121" just hangs until the job times out23:35
opendevreviewMerged openstack/project-config master: nodepool: pause gentoo and tumbleweed builds
fungiianw: hanging during boot device detection?23:57
ianwfungi: maybe ... trying to put a node on hold now.  i'm not sure if it's doing it on the builders23:58

Generated by 2.17.2 by Marius Gedminas - find it at!