Friday, 2023-03-24

corvusmain things: 1) different org names possible; 2) copy all architectures; 3) copy all tags00:00
opendevreviewIan Wienand proposed zuul/zuul-jobs master: build-container-image: directly push with buildx
opendevreviewIan Wienand proposed zuul/zuul-jobs master: build-container-image: enhance buildx documentation
opendevreviewIan Wienand proposed zuul/zuul-jobs master: test-registry: split docker and container paths
opendevreviewMerged zuul/zuul-jobs master: push-to-intermediate-registry: look for container_images variable
opendevreviewIan Wienand proposed zuul/zuul-jobs master: test-registry: split docker and container paths
opendevreviewJames E. Blair proposed zuul/zuul-jobs master: Move container-image-promote login block
opendevreviewMerged zuul/zuul-jobs master: Move container-image-promote login block
opendevreviewJames E. Blair proposed zuul/zuul-jobs master: Add --insecure-policy to skopeo promote command
opendevreviewMerged zuul/zuul-jobs master: Add --insecure-policy to skopeo promote command
hitesh1409_Hi Everyone, 06:04
hitesh1409_I've made changes in the python-jenkins library. Please review it and let me know if more changes are needed:
fricklergithub changed their hostkey, not sure if we have that cached somewhere. for sure it will affect anyone doing git ops via ssh-rsa with them
*** jpena|off is now known as jpena08:41
hasharhi for jjb/python-jenkins I am in the `core` group and I am wondering how I can get myself added to the `release` group,members13:24
hasharshould I reach out to people there so they can add me via the Web UI or are there some YAML files to define group membership13:24
fricklerhashar: the groups are self-managed by the members, if you cannot contact anyone there, feel free to ping infra-root for help13:28
hasharahh that explains it thank you frickler !13:28
fungijust to follow up here as well, but as mentioned in #openstack-jjb i added hashar's gerrit account to the python-jenkins-release group since i am familiar with the history for that project and the current situation13:59
hasharthank you fungi. And for record tracking:  I have send an email to the jjb list to have jjb/python-jenkins to be maintained by the same set of people that maintains jjb/jenkins-job-builder
fungicool, happy to help14:47
clarkbfungi: if you have a moment this morning can you double check the backups moved from gitea01 to gitea09 in a manner that looks good to you then we can probably approve Once that is approved I'll shutdown services on those hosts and if that doesn't turn up any problems after a bit we can delete the servers14:56
clarkbpretty sure you reviewed that before the backups got moved. I did check the backups moved myself already but am hoping for a second set of eyeballs on that. We primarily care about the database iirc14:56
opendevreviewJames E. Blair proposed zuul/zuul-jobs master: WIP: Update promote-container-image to copy from intermediate registry
fungisure, taking a peek15:13
fungiclarkb: when i try to borg list it tells me "Warning: The repository at location /opt/backups-202010/borg-gitea09/backup was previously located at /opt/backups/borg-gitea09/backup"15:38
fungiis that normal?15:38
fungii get the same warning on both backup servers15:38
clarkbI think that has to do with how things were mounted?15:39
clarkbon the backup server15:39
fungiright, following the initial part of just to check the listings15:40
clarkbfungi: thats on the client side though15:40
clarkbI think he warning is entirely server side saying these backups were at location X but are now at location Y (I don't know yet if this is a problem)15:41
fungiit says "log into the backup server, sudo su - to switch to the backup user for the host to be restored, run /opt/borg/bin/borg list ./backup"15:41
clarkboh so it does15:41
clarkbI think that is why you are getting the warning its mounting the backups at location /opt/backups/borg-gitea09/backup under /opt/backups-202010/borg-gitea09/backup or vice versa15:42
clarkbI've only ever done the mounting from the client nodes myself15:42
clarkbanyway borg is sensitive about the location of its repos15:43
clarkbif things have moved it may complain. This happens to me when dynamic dhcp IPs change with my offsite backups location it warns about the ip change15:43
fungiwell, if i proceed, it does give me a listing that includes filesystem and mysql backups from 2023-03-24 on the vexxhost backup server and 2023-03-23 on the rackspace server, so lgtm15:44
clarkbok, as far as that specific warnings goes I think we need to determine if we've moved things inappropriately15:44
clarkblooking at fstab /opt/backups-202010 is where we host the device15:46
fungilrwxrwxrwx 1 root root 20 Oct  8  2020 /opt/backups -> /opt/backups-202010/15:46
fungii think it's that we're accessing it via symlink15:46
fungiso we say "use /opt/backups" and then borg dereferences the symlink and says oh but this is /opt/backups-20201015:46
clarkbya the homedir is listed as /opt/backups15:46
clarkbya that makes sense. I don't think the warning is an issue then15:47
clarkbwe might make note of it in the docs15:47
fungiso probably not moved incorrectly or anything, just borg being paranoid about symlinks15:47
clarkbits information then we have to decide if it is a problem or not. Another good example is that my recovery plan for personal backups is to drive over to my brothers house and retrieve the device and bring it home and plug it in and mount it locally. Borg would warn me then too15:48
clarkbbut that is alsook because the change is due to me mounting the device under a new name locally to speed up recovery (cut out the network)15:48
clarkbin this case I think we are using the symlinks so that we can move old devices to the side and mount new devices without udpating all the configs15:49
clarkball that to say I agree it is a non issue here, and annoying because unlike an intentional move or a known problem for dynamic IPs its a bit more subtle and unexpected15:51
clarkbfungi: should we go ahead and approve the change then? I'm not sure anyone else is likely to review it this morning16:00
clarkbthats going to take a bit due to its modification of the inventory16:02
*** jpena is now known as jpena|off17:22
opendevreviewMerged opendev/system-config master: Remove gitea01-04 from configuration management
clarkbthats going to spend some time running through all the jobs due to the inventory update, but I think I'm good to go ahead and shutdown the docker services on those 4 hosts in the meantime17:39
clarkbthats done. On gitea01 I restarted the mariadb container to prevent email from failed db backup cron jobs from spamming us17:42
clarkbplease keep an eye out for any unexpected behavior but these should have all been removed from the load balancer and gerrit replication so nothing should care about this change.17:43
*** Guest8553 is now known as atmark17:48
*** atmark is now known as Guest875517:49
*** Guest8755 is now known as atmark17:58
clarkbI found good reason to start my python3.11 migration locally. The new tumbleweed x86-64-v3 support finally works on my hardware (still need to track down what the issue was) and python3.11 on tumbleweed includes x86-64-v3 packages. In theory this makes the python 3.11 install even faster than normal21:03
clarkbI now have a new python3.11 venv with fancy newer cpu support and git-review tox nox reno installed into it21:09
clarkbnot that I need those tools to be particularly fast, but maybe when I run things with tox/nox I'll see adifference21:09
fungii've just been compiling my own 3.11 in mo homedir (and 3.12 alphas)21:19
fungipython 3.11 for workgroups21:20
clarkband I guess turning on flags to take advantage of your cpu features21:20
clarkblooking at my installed packages a number of boost libs, all the compression libs, cryptography libs (like openssl) and some image (jpeg not vm variety) stuff seem to have the add on packages to overlay the cpu extra vroom stuff21:23
Hitesh1409Can someone tell me the channels for jjb/python-Jenkins repository ? 21:24
fungiHitesh1409: #openstack-jjb21:24
clarkbassuming they havne't found a new home since that was used?21:25
Hitesh1409I’ve pushed changes since last two days, but I’m unable to contact people for reviewing it. 21:25
fungithey were using that channel this morning when i talked to hashar and added him to the python-jenkins-release group21:25
fungithey also have a mailing list21:25
clarkbfungi: oh cool21:26
fungitoday's discussion log from there:
Hitesh1409Hey fungi, thanks for helping out. Can you please ask them to review my changes:
fungiHitesh1409: i was just helping them with some access changes to the code review system. if you /join #openstack-jjb you can ask them, or it looks like their mailing list is
hasharHitesh1409: I will eventually, over the week-end21:28
clarkbalso it looks like the change was actively being reviewed and the reviewer asked for some time21:28
hasharunless I end up too busy with family duties, but I haven't played with Python in a while so reviewing a few changes would probably be a great nap exercise21:29
hasharat least i got access to approving change and even cutting new release (python-jenkins 1.8.0 I have cut earlier today)21:29
Hitesh1409Your time will be highly appreciated for reviewing it. Thanks in advance. 21:30
hashar also that library is not necessarily actively developed nor is there that many people still using jjb, we might be just a few dozen21:30
clarkbhashar: I've definitely run into people with jenkins installs still using jjb21:31
hasharI do! :]21:31
clarkbI think kata is using it for example. But I'm not sure how many of them feel a need to develop it as well21:31
hasharI guess Jenkins REST API has barely changed over the last decade or so and the python lib is thus rather low maintenance21:32
hasharon another subject, I have noticed Zuul has different tenants ( )    and jjb/python-jenkins happens to be in the "openstack" tenant21:34
fungii would have assumed jenkinsx changed everything21:34
hitesh1409__Yes, I'm aware that the repo is not that active.21:34
hasharshould it be moved to "opendev" as a non openstack project?21:34
clarkb"As of March 24, 2023, Docker has reversed its decision to sunset the “Docker Free Team” plan."21:34
fungihashar: it would probably be trivial to move to a different tenant, yes, depending on whether it reuses any openstack job definitions (i doubt it does)21:34
fungiclarkb: hah, i guess we can take our time on the move then21:35
hitesh1409__How can we contribute to the jenkins rest api, any ideas?21:35
clarkbit definitely makes it a lower priority, but at this point I think we're really close to working things out so probably need to keep the momentum going21:35
clarkbI guess I can prioritize more server replacmeents instead? I'll see what corvus thinks as losing momentum is probably not a good thing21:36
hasharfungi: jjb/jenkins-job-builder has some `openstack-*` jobs but I think I got a change to remove those.  Then I don't know what are the criteria for being in opendev, I am guessing it is simply the default for anything not tied to OpenStack21:36
clarkbhashar: we host open source projects. We're happy to have you as long as the license remains FOSS21:37
hasharhitesh1409__: through the Jenkins project itself I guess? Looks like is their starting page.  Their repo is on  and they have a few mailing lists one for developers at
hasharclarkb: can I bring the thousands of MediaWiki related repos? :-]21:39
hitesh1409__hashar: thanks for the info, I'll check.21:39
fungihashar: it might need some discussion around logistics, but if you're serious we could probably work it out21:39
clarkbhashar: its theoretically possible :) we might need to do some sizing and planning etc21:40
clarkbya what fungi said21:40
fungiwe already have thousands of repos, what's a few thousand more? ;)21:40
hasharI just need Zuul v2.521:41
* hashar ducks21:41
hasharI am kidding, we do self host on principle and for historical reason. Albeit our stack is miles behind the OpenDev one21:41
hasharbut it is something I am sometime contemplating on my own.  At least if I have an idea for a new project outside of Wikimedia I will consider OpenDev for sure21:42
fungiif there's ever interest in openinfra foundation/wikimedia foundation collaboration on infrastructure though, i can help get the right people in touch21:43
fungii know we all have great respect for what wmf and its communities have accomplished21:44
hasharfungi: thanks I will keep that in mind :]21:51
hasharand the respect goes the other way. Our OpenStack cluster powers a lot of the tool developed by our volunters and is a key in feeding content to the wiki. I also got so much help setting up the CI stack eleven years ago, which in all honesty was merely just me copy pasting from all the work openstack-infra has done21:53
clarkbgithub had a number of expired and mis served certs today22:07
clarkbin addition to the rsa key rotation22:07
fungihashar: the openinfra foundation did somewhat recently add a new (non-paying) associate member organization class for nonprofits and educational/research institutions we partner and collaborate with, if that might be something wmf is interested in:
ianwi'd agree that the work we're doing to abstract from dockerhub is generically useful, sunsetting expiry of free hosting or not22:20
ianwwe've found corners in image creation, tagging, deleting that it's nice to work out 22:21
hasharfungi: thanks! :)22:23
fungiianw: maybe now we have time to develop and popularize a dockerhub read-only api replacement which is redirectable ;)22:24
fungiwith no root level namespace so dockerhub is no longer an implicit root for clients implementing the new image fetching protocol22:26
fungiand also fix the proxying/caching problems with the existing protocol while we're at it22:27

Generated by 2.17.3 by Marius Gedminas - find it at!