Tuesday, 2022-11-15

opendevreviewOpenStack Proposal Bot proposed openstack/project-config master: Normalize projects.yaml  https://review.opendev.org/c/openstack/project-config/+/86445602:18
opendevreviewMerged openstack/project-config master: Normalize projects.yaml  https://review.opendev.org/c/openstack/project-config/+/86445603:07
*** ysandeep|out is now known as ysandeep05:57
*** soniya29|rover is now known as soniya29|rover|afk05:59
fricklerssbarnea: stephenfin: you may want to have a look at https://review.opendev.org/c/opendev/git-review/+/864098 and my CI fix below it07:27
*** soniya29|rover|afk is now known as soniya29|rover07:46
*** yadnesh|away is now known as yadnesh07:48
*** slaweq_ is now known as slaweq07:57
*** jpena|off is now known as jpena08:21
*** soniya29|rover is now known as soniya29|rover|lunch08:25
*** ysandeep is now known as ysandeep|lunch08:45
*** soniya29|rover|lunch is now known as soniya29|rover09:22
*** akahat|ruck is now known as akahat|ruck|afk09:53
*** ysandeep|lunch is now known as ysandeep10:20
*** pojadhav is now known as pojadhav|dr_appt11:09
*** akahat|ruck|afk is now known as akahat|ruck11:32
noonedeadpunkhey there! Any idea where this redirect may come from? https://paste.openstack.org/show/bCqeAofiSsTUOtrN5l4r/11:48
noonedeadpunkLike in gerrit these are 2 different projects: https://review.opendev.org/admin/repos/q/filter:zookeeper11:49
noonedeadpunkBut gitea does redirect for $reason....11:50
noonedeadpunkah, here we go.... https://opendev.org/opendev/project-config/src/branch/master/renames/20190419.yaml#L131712:09
fungiyes, the old project name is now in use by a new project. we likely have to unwind that somehow12:28
fungiin this particular case, the original project wasn't used much so it's probably fine to do so. more generally, i wonder how and whether we need to catch that sooner12:30
fungii think gitea stores its redirects in database records, so cleaning it up across the entire farm of gitea servers is probably going to be a bit of work12:31
fungisince they don't share a database12:31
noonedeadpunkwell, to be fair I asked if I should rename repo somehow :)12:58
noonedeadpunkfungi: I wonder if I should push patch to drop that rename? I guess not?12:59
funginoonedeadpunk: we'll need to talk through the options when there are more folks around (maybe add it to the meeting agenda for today if we can't work out a good solution sooner)13:01
noonedeadpunkI'm not sure if dropping/renaming repo will be much easier?13:04
fungino, i expect cleaning up the old redirects will be the simplest path forward, but want to get some consensus on that from other sysadmins and to have a better understanding of the steps involved in it13:21
*** pojadhav|dr_appt is now known as pojadhav13:25
*** dasm|off is now known as dasm13:58
opendevreviewMerged opendev/git-review master: Fix nodesets for tox jobs  https://review.opendev.org/c/opendev/git-review/+/86410614:02
noonedeadpunkyeah, that's totally fair but smth I can't actually help with. But I'm really open for solutions if there's an easy way14:06
*** ysandeep is now known as ysandeep|dinner14:41
*** pojadhav is now known as pojadhav|dinner14:42
*** soniya29|rover is now known as soniya29|rover|out14:47
*** yadnesh is now known as yadnesh|away14:58
*** ysandeep|dinner is now known as ysandeep15:00
*** pojadhav|dinner is now known as pojadhav15:31
*** akahat|ruck is now known as akahat|ruck|dinner15:31
opendevreviewLajos Katona proposed openstack/project-config master: Add toggleWipState to networking projects  https://review.opendev.org/c/openstack/project-config/+/86456315:33
clarkbfungi: noonedeadpunk: the tldr is that openstack/ansible-role-zookeeper redirects to windmill/ansible-role-zookeeper in gitea due to a previous rename of openstack/ansible-role-windmill to windmill/ansible-role-zookeeper? I'm good with undoing the redirect since windmill is largely unmaintained at this point and windmill users can directly point to the main name if necessary15:45
fungiclarkb: precisely. is that going to involve cleaning up our rename metadata in opendev/project-config and then deleting redirect records from the mariadb on each gitea server?15:46
fungior is there more to it?15:46
fungior is there an api to use for manipulating those redirects instead of poking directly in the db?15:46
clarkbthose redirects are inserted into the gitea redirect database as a side effect of renaming the projects in gitea. I am unaware of any direct management for that so database updates are likely necessary15:47
clarkbI don't think we need to clean up are rename metadaa that is just record keeping. We might want to add a note to it to prevent readding that redirect in hte future though15:47
clarkbbasically keep the existing metadata that says we did a rename but amend it to indicate url redirects are undesireable (with a yaml comment probabl)15:47
fungimainly concerned if we use it in the future to rebuild the redirects15:47
fungiyeah, that should work15:48
fungifor the db, at least it's easy to test on one server, then put back if it causes a problem15:48
fungicould even temporarily remove it from the haproxy pools if we're really worried15:48
clarkbif we didn't want to edit the db directly we might eb able to rename windmill/ansible-role-zookeeper back to openstack/ansible-role-zookeeper then recreate windmill/ansible-role-zookeeper and tell gerrit to replicate both projects to resync the git content15:48
fungithough that's likely overkill15:48
clarkbfungi: my suggestion would be to hold a gitea system-config-run test node and test the redirect removal there15:49
clarkbfungi: we do project renames in those jobs so everything but the redirect removal will be done for you15:49
fungioh, yep that's even better15:50
clarkbcan even test the rename back to original name to see if it collapses the db records (I suspect not, I suspect it just adds another redirect entry to the db then the software collapses things)15:51
clarkbensuring we don't break any schema expectations with unenforced foreign keys etc (seems like that was popular for performance reasons once upon a time,  ut I have no idea if that is an issue here) is probably the main thing if editing the db directly15:51
fungilonger term, we may want to check the renames data when adding new repos... that could probably be done pretty easily with a zuul job15:52
clarkbfungi: also, I think logging in as the admin user and double checking there isn't a ui toggle for this would be good ( a lot of things in the ui don't have api mechanisms so lack of api doesn't mean no button)15:52
clarkbfungi: ++++15:52
clarkbI'm double checking the api swagger doc now15:53
fungiclarkb: in unrelated news, upstream mailman3 containers got updated for the 0.4 tag we were tracking, which broke the setup automation. fixing that then broke the forked dockerfile builds. i've updated those as well, but seems like there may be a behavior change breaking our tests now, which i still need to look into15:55
clarkbinteresting. They had indicated you should be able to upgrade between 0.4 rleases. Might be a good thing we hit this before deployment so we can learn about their update processes :)15:56
fungihaving been through the exercise, i guess my main concern with the fork is all the == pinning for pypi reqs, which leads to dependency hell any time we want to update the images to new mailman releases, and also means we won't be automatically updating mailman when there's a security issue15:56
fungiyeah, they decided that 0.4.4 should switch from python 3.8 to 3.915:57
fungiprobably driven by needing to update dependencies or new python features used in newer mailman15:57
fungianyway, ironing the rest of that out is my main priority today if i get user requests under control15:59
clarkbre the == dependencies those come frmo upstream because aiui they only work with very specific deps16:00
fungijust noting that we have to either copy theirs exactly or figure it out ourselves if there's an urgent mm3 update16:01
fungifor example, there's a newer mailman-core available on pypi than what's in the containers, but upgrading that needs a newer django too16:02
*** ysandeep is now known as ysandeep|out16:02
clarkbright. I linked to a mm3 mailing list thread a little while back that basically said this is a big issue for them and they need to do better16:03
fungioh, right, i remember that now16:04
clarkbit definitel doesn't help with confidence levels, but I'm not sure there is much we can do as end users unless we volunteer to fix that for them. Maybe hey need a lockfile/constraints16:05
fungisure, it's the process of generating that list that's challenging of course16:06
*** marios is now known as marios|out16:28
clarkbfungi: https://review.opendev.org/c/opendev/system-config/+/864060 and https://review.opendev.org/c/opendev/system-config/+/864234 would be good to land. First fixes ehterpad logging rotation (or lack of it) and second updates our gerrit image building so that it works again16:30
clarkbthere is a followup to the second changethat will update our gerrit imgaes to the latest versions from last week too16:31
clarkbnothign super urgent in those updaets but it does improve performance for the copy approvals step we need to take before upgrading to 3.6 which is great16:32
fungiawesome, thanks. will take a look16:32
fungiokay, so the system-config-run-lists3 failure with our updated mm3 container seems to be related to database configuration/population, but i'm not finding a lot to go on in the collected logs: https://zuul.opendev.org/t/openstack/build/08603d23d29a4c5f90d9c0c995e9ec5516:47
clarkbfungi: but it does work with their images?16:49
clarkbmight need to exec into the two images and compare pip freeze output or do something similar to that16:50
fungialso possible that was a random failure, i'm checking the child change to see if it had the same issue16:50
clarkboh wait16:50
clarkbthats one of my tasks to ensure we don't operate on the deployment before the databse has content we expect. Is it possible the content has changed?16:50
fungientirely possible16:51
clarkbowuld be odd for the parent to work in that case, but I think first step is to do what that task does by hand on a held node and then take it from there16:51
fungii think i have a held node from the dnm change, and yeah same error on it too16:51
clarkbmaybe django changed the table name or something16:52
fungithough doesn't explain why the test worked with the upstream image16:52
clarkbhttps://zuul.opendev.org/t/openstack/build/08603d23d29a4c5f90d9c0c995e9ec55/log/lists99.opendev.org/docker/mailman-compose_database_1.txt indicates it may be an auth issue?16:52
clarkbbut the db itself is up16:53
clarkbya I think once we understand the failure then we can dig into the difference between the images16:53
fungioh, that's where the db log is. i even looked in that subtree but my eyes just went right past it16:53
clarkbI've got to find breakfast and prep for the infra meeting and so on, but let me know if I can help more with that. Definitely start by manually trying to run that task to see why it is failing (auth failure?) and once we know how to fix it we can compare images with that knowledge16:54
fungiAccess denied for user 'root'@'' (using password: NO)16:54
fungiyeah, maybe we're missing an update for that16:55
clarkbI know getting the quoting right for those tasks was a chore16:55
clarkbmaybe something changed to break the quoting?16:55
fungii'm going to sxs compare the db log from the working build16:56
fungibut also is the held node, i can test things on there16:57
fungion the working build, th elog looks very similar except after a couple of access denied entries for 'root'@'' it appears to switch to using 'mailman'@''17:02
clarkbthe command does use -u mailman17:02
fungioh, actually the failing log has indications of those connections too17:03
fungiso this may be a red herring17:03
clarkbso maybe the root access denieds are a red herring and instead "Aborted connection 629 to db: 'mailmandb' user: 'mailman' host: '' (Got an error reading communication packets)" is what we should look at17:03
fungiwe have those in the successful one too though17:03
clarkbya so it could just be a problem with parsing the output17:05
clarkbntil: django_db_exists.stdout_lines | length > 1 and django_db_exists.stdout_lines[1] == "auth_user"17:06
fungithere is no auth_user table on the held node17:09
clarkbya so django must've changed. I don't understand why that would work on the parent change though17:11
clarkbthe idea behind all of that is there isn't a good way to check if mailman3 is up and running. So we do a number of checks for things that get us there. A db check, a rest api check, and I think another one17:12
fungiunless me copying from master of the dockerfiles isn't reflecting the 0.4.4 tag17:12
fungii'll double-check the diff17:12
clarkbalso maybe django isn't starting properly17:12
clarkbwe may need extraconfig17:12
clarkbfungi: the failing job is on python3.10 not python3.917:14
fungioh, huh17:15
clarkboh but they both are17:15
fungithe only commit after 0.4.4 was "Add g++ (gcc-cpp) as build dependency" which only updated core/Dockerfile.dev abd some postgresql configuration17:16
clarkbfungi: https://zuul.opendev.org/t/openstack/build/08603d23d29a4c5f90d9c0c995e9ec55/log/lists99.opendev.org/docker/mailman-web.txt#1624-1668 that doesn't show up in the successful job17:16
fungibut maybe i should more thoroughly diff the upstream dockerfile against ours17:16
clarkbso ya I'm backto thinking something about the django install has broken the db17:17
clarkband that would be in the image17:17
fungioh, and yeah the mailman-web container did get updated to python 3.10, mailman-core only to 3.917:18
clarkbthats weird that they would split them up like that17:18
fungiapparently they like to use different python versions in different containers, because that's a sane thing to do17:18
fungiyeah, mailman-core uses alpine:3.15 while mailman-web has alpine:3.16.217:19
fungiand mailman-web did get a django cap increase from <3.2 to <4.117:27
fungialong with updating django-mailman3 from 1.3.7 to 1.3.817:27
fungii'll look through changelogs in a bit, but need to take a moment to go pick up some lunch17:28
clarkbit might be better to diff the image content directly17:29
clarkbmaybe something without a pin got updated17:29
*** jpena is now known as jpena|off17:42
clarkbfungi: https://github.com/maxking/docker-mailman/commit/208017f9a00fdaf1b2d0671e97fb879e804fd911 I think you need to incorporate these updates18:15
clarkbhere https://review.opendev.org/c/opendev/system-config/+/860157/14/docker/mailman/web/mailman-web/urls.py but should cross check all of those files18:16
fungioh, thanks!18:16
fungibut yeah, back now and lunch has been consumed, so i'll start plugging at it18:17
clarkbdoes it make sense to also convert to using a requirements file like upstrem just to keep the delta down? Our install should be equivalent though18:17
clarkbfungi: https://review.opendev.org/c/opendev/system-config/+/860157/14/docker/mailman/web/mailman-web/uwsgi.ini also needs updating18:22
fungioh fun18:26
fungiso path changes to web/mailman-web/urls.py and refresh the uwsgi.ini file we're using18:29
clarkbya  Ithink so18:29
clarkbseparately I was unable to exec into our forked image using --entrypoint bash but was able to do so with the 0.4 upstream image tag18:30
clarkbwe should test that after we fix those issues18:30
fungiis our docker/mailman/web/mailman-web/urls.py just a direct copy from their repo, or does it have local edits?18:30
clarkboh wait that might be my own derp18:31
clarkbfungi: everything in the image fork I pushed up was a direct copy except for the addition of lynx to the package install list18:31
*** akahat|ruck|dinner is now known as akahat|ruck18:31
clarkbfungi: all of the overrides we are doing are happening via ansible config management18:31
fungiwell, and our dockerfiles don't use requirements.txt files, the packages are merged instead18:32
clarkbfungi: that is because the split to requirements.txt is newer than my fork18:32
fungior did that change recently upstream>18:32
fungiaha, so maybe i should do that too18:32
clarkbyes the exec thing was my own derp. DOn't try to provide a command as well as an entrypoint18:34
clarkbfungi: also there is an even newer mailman release18:36
clarkbhttps://docs.mailman3.org/projects/mailman/en/latest/src/mailman/docs/NEWS.html#news-3-3-7 not sure if we want ot try and update to those or just stick with being a very lightweight fork18:36
clarkblooks like it should be pretty non eventful for us and adds a bugfix though18:36
fungilooks like the only uwsgi.ini edit is adding buffer-size = 819018:37
fungiclarkb: yeah, i tried it, but 3.3.7 needs newer django18:37
fungicould try uncapping django but then not sure what else that'll shake loose18:38
clarkbinteresting that the notes don't say anyhting about that :/18:39
fungiyeah, the dockerfile limits Django<4.1 but mailman 3.3.7 wants Django>=4.1 (or something like that)18:40
fungiso presumably lockstep django updating18:41
clarkbfungi: I think this is separate18:42
clarkbit is mailman-core that got updated not mailman-web18:42
clarkbhttps://review.opendev.org/c/opendev/system-config/+/860157/14/docker/mailman/core/Dockerfile#20 is the line to update18:43
clarkb(entirely possible there are other deps issues though, its just that django shouldn't be one of them)18:43
fungiwell, now it's in the requirements.txt with the update i'm about to push18:44
clarkbright but its part of the non django portion of mm318:44
fungianyway, we can try updating that as a different revision if we want18:44
fungioh! wasn't django18:45
fungithe dockerfile for mailman-core has sqlalchemy<1.4.0 and mailman 3.3.7 needs sqlalchemy>=1.4.018:45
fungithat's what it was18:45
fungibumping sqla might not be too onerous18:45
fungior it might need newer alpine18:45
clarkbya that should be safe to change based on my read of the changelog18:45
clarkbthey had that pin in place because of python3.6 support aiui18:46
fungioh, well i'll give it a shot here in that case and we can see what happens18:46
opendevreviewJeremy Stanley proposed opendev/system-config master: Fork the maxking/docker-mailman images  https://review.opendev.org/c/opendev/system-config/+/86015718:49
opendevreviewJeremy Stanley proposed opendev/system-config master: DNM force mm3 failure to hold the node  https://review.opendev.org/c/opendev/system-config/+/85529218:49
fungii've blown away my last held node and set a new autohold18:49
funginow to see about holding a gitea test node for fiddling with redirect removal18:51
clarkbunrelated: our jitsi meet install should upgrade overnight tonight due to a new release and our periodic jobs18:52
opendevreviewJeremy Stanley proposed opendev/system-config master: DNM: Fail gitea testing to hold a node  https://review.opendev.org/c/opendev/system-config/+/86458118:53
clarkbfungi: re ^ before you do the db mangling I would definitely log in as the admin use rand see if this is toggleable via the web ui18:54
fungido we have global admin set up in prod?18:54
opendevreviewVishal Manchanda proposed zuul/zuul-jobs master: [DNM] Enable and started dbus.service  https://review.opendev.org/c/zuul/zuul-jobs/+/86458218:54
clarkbfungi: we have an admin user yup. But you have to modify each node separately18:55
fungithat much i figured. anyway, i won't have time to look at it until after the meeting18:56
clarkbyou can also login as admin on the test node. Everything should align behaviorwise since we deploy a specific version18:56
fungiyeah, that's what i was going to look at first18:57
fungii guess we have a test credential for admin in the job setup18:57
opendevreviewMerged opendev/system-config master: etherpad: redirect container logs  https://review.opendev.org/c/opendev/system-config/+/86406019:35
fungithe latest mm3 container fork change seems to test clean19:37
fungiso i should be able to proceed with migration testing on it this evening19:37
fungifor the fips token situation, i can understand the concerns on both sides, and am willing to accept that encoding the ua token in a zuul secret really amounts to obfuscation which may not be a great choice for transparency purposes20:04
fricklermaybe it would also help to see how the token would be used. is there some documentation that you could link to? or show how a job using it would look like?20:08
clarkbI expect the jobs will use the `ua` command to enroll the node and enable the features / do package updates then reboot into the updated kernel settings20:08
fricklerso that's essentially the same we did with the old xenial nodes? (or was it even older ones?) I was assuming fips to be a different feature20:10
opendevreviewMerged opendev/system-config master: gerrit-build: jammy updates: update to nodejs 18.x, allow submodule clones  https://review.opendev.org/c/opendev/system-config/+/86423420:13
clarkbfrickler: yes, the ua tool is used to manage all of those separate things. THey are distinct items, just sharing the same management tool20:14
fungifrickler: the extent of the documentation i got for it is included in the commit message20:17
fungipro attach {{ token }}20:17
fungibut my understanding is that it will then allow adding sources.list entries for the fips enablement package repository so that packages can be installed from there20:18
fungii'm mainly hoping the people with a vested interest in doing fips testing will be the ones to work out the details20:19
fricklerfor me the main question still is where we as opendev team (and this is where IMO this leaves the pure openstack realm) draw the line on running non-free software on test nodes20:25
frickler we have this, we might be asked to run RHEL or possibly even windows instances by some project. and this is for me a fundamentally different thing from accessing external APIs only that may require a secret to access them20:26
fungiit's not entirely clear that the software in question is closed source/proprietary, but also there's nothing preventing a user from installing software in a test from somewhere on the internet regardless of what license it's distributed under20:27
fricklerwell the download of the fips certified software obviously isn't free. otherwise we could run these jobs without needing that token20:28
fungiwe've been asked to run rhel in the past. we didn't say no because it requires licenses, red hat said no because they didn't feel like working out the legal agreements it would entail20:28
fungi"free" (libre) software has nothing to do with whether the distributor charges money for it20:28
fungiif we're not allowed to redistribute the software we've received then that's another matter of course20:29
fricklerthere different flavors of free of course20:30
fungiof course20:30
fricklerwell we are not allowed to redistribute the token we received, or are we?20:30
fungieven the gpl doesn't require someone to distribute software for no charge, only that they also supply source code to anyone to whom they distribute binary builds20:30
fungiis a token "software"?20:30
fungithis is where a lot of these existential arguments about software freedom start to break down20:31
fungiis a software license also software? many open source licenses disallow alteration of the license files. can you say it's acceptable to waive software freedoms for license texts, but expect credentials to be open source?20:32
fungiis it unacceptable to use a private credential in order to obtain privileged access to someone's build of an open source project, but okay to use a private credential to authenticate a test to someone's proprietary service on the internet?20:34
fungiis the twitter login our statusbot uses free software?20:34
fricklerlots of good questions indeed. I'll sleep about it. fwiw feedback I saw from the TC so far was similarly mixed, so not helpful in making up my mind, either20:35
JayFfungi: I'd frame the question this way: Can we redistribute OS images that contain Ubuntu-FIPS code?20:35
fungiJayF: we weren't planning to do so, but it's a valid question to ask canonical20:35
JayFlike, is the software itself under some kind of license agreement or will opendev/infra be free to distribute it as long as the key is not disclosed20:35
JayFI feel like if you get a key that gives you access to a repo full of FOSS20:35
JayFthere's no way they give you the key without you agreeing to some sort of restriction, right?20:36
fungiwe weren't planning to distribute the software anyway, but i do agree it's a good data point20:36
JayFIt's just helpful to have a concrete perspective to look from when talking about "software freedom"20:36
JayFand I think about it in context of this: there's a failing FIPs run in Ironic. I ask infra to hold a failing node.20:37
JayFDo I have to worry about copying secrets off of that public CI node I've been given access to?20:37
JayFDo I agree transitively to not disclose or distribute information about the FIPs configuration that I now have access to?20:37
JayFlicense key itself is an authentication credential; I get that part being hidden; but if the work product is licensed as such that we can't redistribute it, then you're going to handcuff any developer who has to go deep troubleshooting on this 20:38
fungicanonical representatives said not to worry about that, because the main value in that token and those builds of packages is whether a user can engage them for support. if someone wants free access to fips-enabled builds of software they can build the same themselves from source or switch to a different distribution which provides them already20:38
JayFIs that going to be in writing in some public place?20:39
fungiit's summarized in a code comment20:39
JayFpart of being open is those promises being made in the open so that everyone can rely on it; not just the folks having these conversations (or the second order conversations like this one)20:39
JayFThis all sounds fine then? 20:39
fungithe code comment could certainly be expanded on20:40
Clark[m]Fwiw openstack already uses secrets to access proprietary software from our jobs (GitHub replication)20:40
JayFAs we were talking about in #openstack-tc yesterday; it's not any different than having features that can't be tested without explicit hardware20:40
Clark[m]I don't think that aspect is a concern unless we want to revert a lot of precedent 20:40
JayFand sometimes even hardware that needs a software-unlock-key-license (e.g. some iLo features that Ironic supports)20:40
JayFI've been trying to drive a wedge somewhere to see if that ^ case is different than the fips case; and it just isn't.20:40
fungitinwood may also be able to clarify some of the technical points about whether there's any non-osi-approved licenses involved in the packages distributed for ubuntu fips enablement (he's the one who arranged the token for us)20:43
fungimy assumption is that it's just specifically configured builds of the usual open source projects which have been subjected to additional testing on the distro side, and are normally being distributed for a cost in order to be able to fund that work, but i could be mistaken20:46
JayFyeah tbh given what I know about it now; it makes me wish that they just gated the actual-support and not the code20:54
opendevreviewIan Wienand proposed opendev/statusbot master: Add mastodon support  https://review.opendev.org/c/opendev/statusbot/+/86458621:10
ianw^ basically you just send a bit of json with a bearer token21:11
fungioh nice21:15
opendevreviewIan Wienand proposed opendev/statusbot master: Add mastodon support  https://review.opendev.org/c/opendev/statusbot/+/86458621:16
clarkbianw: where did we end up with paste and werkzeug. Do we need a system-config-run job?21:17
clarkboh we have that job I just need a depends on to the lodgeit side I guess /me does this21:17
ianwyeah that should do a complete bringup and screenshot21:19
opendevreviewClark Boylan proposed opendev/system-config master: DNM testing lodgeit image updates  https://review.opendev.org/c/opendev/system-config/+/86458721:19
clarkbre etherpad it looks like we are getting the log files now. However, it seems like we might still be logging to the docker log which is what was filling up?21:26
clarkbso this might be half of the fix we need. The other half would be to somehow reduce retention of the docker log? I don't know the answer yet just calling out what I see21:26
fungior configure etherpad/nodejs to not also log to stdout?21:27
opendevreviewIan Wienand proposed opendev/statusbot master: Add mastodon support  https://review.opendev.org/c/opendev/statusbot/+/86458621:28
opendevreviewIan Wienand proposed opendev/statusbot master: Update ancient hacking version  https://review.opendev.org/c/opendev/statusbot/+/86458821:28
clarkbwell I think that is how the docker -> syslogging works21:28
clarkbit takes what docker sees, tags it with a syslog tag, then syslog puts it into files21:28
ianwsorry i have to afk for a bit, i can look when i get back in ~1 hour21:29
clarkboh unless `docker log` knows to look in syslog for that info?21:29
clarkbya that may be it21:30
fungier, right i reviewed that no more than an hour ago and already forgot the config was telling docker to log to syslog, not telling nodejs to do that21:30
clarkbcontainer-cached.log is the file we end up with in the container dir21:34
clarkbmaybe that won't grow forever and instead acts like a cache for the info going to syslog?21:34
clarkbI think we can monitor and observe this.21:34
clarkbhttps://docs.docker.com/config/containers/logging/dual-logging/ I suspect that it may be doing something like this?21:40
clarkbthere are also docker daemon settings that will rotate logs. It might not be a bad idea for us to set those too?21:41
clarkbianw: https://1b684d4e3822b9d14821-fd1d23de4951783ce4905cf96230b70e.ssl.cf5.rackcdn.com/864587/1/check/system-config-run-paste/e56772e/bridge99.opendev.org/screenshots/paste-main-page.png21:54
*** rcastillo|rover is now known as rcastillo21:55
clarkbhttps://zuul.opendev.org/t/openstack/build/e56772e28f75450c97e5ac18073f28cc/log/paste99.opendev.org/docker/lodgeit-compose_lodgeit_1.txt shows it using python3.10 as expected21:55
*** dasm is now known as dasm|off23:02
ianwclarkb: excellent, thanks!  LGTM23:03
clarkbfungi: I've poked around on your held gitea node to try and find a way to edit redirects via the ui. I've also looked at the API swagger doc with no luck. I suspect that the db is our only choice23:08
clarkbLooking at the db the repo_redirect table appears to have the bits we need. That table has a redirect id, then owner_id + repository pair that indicate the source of a redirect and redirect_repo_id that points to the target23:09
clarkbon our test node we have openstack/diskimage-builder and opendev/diskimage-builder redirecting to opendev/disk-image-builder23:09
clarkbI think we get two redirects because we changed both the name and org in this case23:10
clarkbthe owner_id maps onto user.id and redirect_repo_id onto repository.id23:11
clarkbyou should double check all of that but hopefully this helps with your investigating23:11
ianwclarkb: i agree journalctl -f is still showing things, i had expected that would be out of the loop too23:19
fungithanks for the extra detail clarkb!23:26
fungii got sidetracked with other stuff and hadn't a chance to dig into it yet, but at this rate probably will save it for tomorrow morning (barring any new emergencies overnight)23:27
clarkbianw: ya I think we can probably monitor it and see how it does23:34
clarkbianw: if that cache file doesn't grow indefinitely then its probably fine23:34
ianwclarkb: i think this is related, and i have deja-vu about this -> https://github.com/docker/for-linux/issues/1151#issuecomment-73435186623:34
ianwhttps://review.opendev.org/c/opendev/system-config/+/845316 is why i have deja-vu i think.  same but different ...23:36
clarkbdoes that imply we need to down then up it?23:36
ianwi think it implies that we should have a separate rsyslogd socket that docker logs to directly, bypassing /dev/log23:37
ianwif we don't want the logs in journald too23:37
clarkboh, I'm ok with them being in journald23:38
clarkbits the docker direct collection that grows indefinitely and created problems (I think journald has some sort of pruning?)23:38
clarkbfungi: are you interested in the stack at https://review.opendev.org/c/opendev/lodgeit/+/864222 which updates lodgeit testing and python versions?23:40
fungilooks straightforward to me, and still seems to run the same tests23:51
clarkbthe child changes swap the docker image over to python3.10 and remove 3.6 testing23:51
clarkband are tested by https://review.opendev.org/c/opendev/system-config/+/864587 as well23:52

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!