| opendevreview | Merged opendev/system-config master: Add linaro mirror node https://review.opendev.org/c/opendev/system-config/+/868370 | 00:33 |
|---|---|---|
| ianw | hrm, the base job timed out, but it looks like it's due to a hung connection trying to talk to the gitea lb | 02:55 |
| ianw | 76 root 20 0 0 0 0 R 92.8 0.0 1523:21 kswapd0 | 02:57 |
| ianw | this is not a happy host | 02:58 |
| ianw | $ ps -aef | grep git | wc -l | 02:58 |
| ianw | 3913 | 02:58 |
| ianw | somethign has gone bananas | 02:58 |
| ianw | gitea08 | 02:58 |
| ianw | load average: 86.02, 94.86, 106.24 | 02:59 |
| ianw | i'm just going to completely reboot the host | 02:59 |
| Clark[m] | ianw: taking it out of the lb is probably smart as we may need to force a resync from Gerrit to it | 03:01 |
| ianw | ok, it is back | 03:02 |
| ianw | i've forced a replication to gitea08 | 03:07 |
| ianw | gerrit show-queue --by-queue" | grep 'push ssh' | wc -l | 03:10 |
| ianw | 2275 | 03:10 |
| ianw | for my own laziness i'll just watch that instead of dropping it from the cluster. if there's issues will drop it and report back | 03:10 |
| ianw | i'll do a manual base and mirror run, to make sure this new mirror is deployed and not going to break things too | 03:11 |
| ianw | ahh, it doesn't start | 03:40 |
| ianw | Cache size (50000000) must be less than 95% of partition size (which is 48722080). Lower cache size | 03:40 |
| opendevreview | Ian Wienand proposed opendev/system-config master: Linaro mirror: shrink volume, add letsencrypt cert handlers https://review.opendev.org/c/opendev/system-config/+/868375 | 03:49 |
| *** yadnesh|away is now known as yadnesh | 03:50 | |
| ianw | #status log gitea08 restarted and resynced to gerrit | 04:14 |
| ianw | #status log gitea08 restarted and resynced to gerrit | 04:20 |
| opendevstatus | ianw: finished logging | 04:21 |
| opendevreview | Steve Baker proposed openstack/diskimage-builder master: Check and grow the GPT structure to the end of the volume https://review.opendev.org/c/openstack/diskimage-builder/+/827558 | 04:21 |
| opendevreview | Steve Baker proposed openstack/diskimage-builder master: Reduce thin pool by one more extent https://review.opendev.org/c/openstack/diskimage-builder/+/868376 | 04:21 |
| opendevreview | Steve Baker proposed openstack/diskimage-builder master: Reduce thin pool by one more extent https://review.opendev.org/c/openstack/diskimage-builder/+/868376 | 04:30 |
| opendevreview | Steve Baker proposed openstack/diskimage-builder master: Check and grow the GPT structure to the end of the volume https://review.opendev.org/c/openstack/diskimage-builder/+/827558 | 04:30 |
| opendevreview | Ian Wienand proposed opendev/system-config master: Linaro mirror: shrink volume, add letsencrypt cert handlers https://review.opendev.org/c/opendev/system-config/+/868375 | 04:38 |
| *** chandankumar is now known as chkumar|rover | 05:27 | |
| opendevreview | Merged opendev/system-config master: Linaro mirror: shrink volume, add letsencrypt cert handlers https://review.opendev.org/c/opendev/system-config/+/868375 | 06:47 |
| *** jpena|off is now known as jpena | 08:24 | |
| noonedeadpunk | So, we again have issues with CentOS mirrors :( I will try to check out with mirror maintainer (and find that info again) | 10:18 |
| noonedeadpunk | aha, it was Epel mirror last time. Now I think this is about Base ones | 10:19 |
| noonedeadpunk | and we're using facebook mirror.... | 10:21 |
| noonedeadpunk | https://opendev.org/opendev/system-config/src/branch/master/playbooks/roles/mirror-update/files/centos-stream-mirror-update#L42 | 10:21 |
| noonedeadpunk | I kind of wonder why don't we use rackspace mirror? Do we have point of contact of mirror maintainers in facebook? | 10:43 |
| noonedeadpunk | As there's both rsync://mirror.rackspace.com/centos-stream/9-stream/ and rsync://mirror.rackspace.com/centos-stream/SIGs/9-stream/ | 10:45 |
| *** dviroel|out is now known as dviroel | 10:48 | |
| noonedeadpunk | aha https://opendev.org/opendev/system-config/commit/e76e0089d1c4a7fe781027feb7977ae874899443 | 10:51 |
| noonedeadpunk | lol https://opendev.org/opendev/system-config/commit/cc2dd16d3a7194a4185ad6e1da854cb4fde01b1c | 10:52 |
| noonedeadpunk | There's also ord.mirror.rackspace.com :D | 10:52 |
| noonedeadpunk | damn | 10:55 |
| *** dasm|off is now known as dasm | 14:10 | |
| opendevreview | Dmitriy Rabotyagov proposed opendev/system-config master: Revert "Revert "Use rackspace mirror to sync centos stream repos"" https://review.opendev.org/c/opendev/system-config/+/868392 | 14:16 |
| opendevreview | Dmitriy Rabotyagov proposed opendev/system-config master: Revert "Revert "Use rackspace mirror to sync centos stream repos"" https://review.opendev.org/c/opendev/system-config/+/868392 | 14:17 |
| *** yadnesh is now known as yadnesh|away | 15:08 | |
| *** dviroel is now known as dviroel|afk | 15:46 | |
| clarkb | noonedeadpunk: did you check if the rackspace mirror is up to date? | 16:03 |
| noonedeadpunk | clarkb: at least they don't seem to contain python3-3.9.16-1 | 16:39 |
| noonedeadpunk | latest is python3-3.9.14-1 in there from what I see | 16:40 |
| noonedeadpunk | well | 16:40 |
| noonedeadpunk | facebook mirror also doesn't contain it.... | 16:41 |
| noonedeadpunk | Could it be that we're missing delete flag or smth? | 16:41 |
| noonedeadpunk | I see it's there. but well... | 16:41 |
| clarkb | https://mirror.bhs1.ovh.opendev.org/centos-stream/timestamp.txt is when we last updated. I think it kicks off the sync process every four hours so should get another sync shortly | 16:43 |
| clarkb | you can check the log here https://mirror.bhs1.ovh.opendev.org/logs/rsync-mirrors/ once it publishes | 16:44 |
| clarkb | maybe wait for the next sync to see if it the upstream synced in the interim | 16:44 |
| clarkb | but if both upstreams have the same file then I don't think swapping will help | 16:44 |
| noonedeadpunk | both doesn't have it now, but yeah, same applies | 16:55 |
| *** dviroel|afk is now known as dviroel | 17:09 | |
| clarkb | the cron jobs are set to fire every 6 hours not every 4 | 17:14 |
| *** jpena is now known as jpena|off | 17:35 | |
| noonedeadpunk | ok. worth waiting then... and crossing fingers | 17:37 |
| *** dviroel is now known as dviroel|out | 20:15 | |
| opendevreview | Ian Wienand proposed opendev/system-config master: Remove old nodepool.yaml testing variables https://review.opendev.org/c/opendev/system-config/+/868432 | 20:16 |
| opendevreview | Ian Wienand proposed opendev/system-config master: linaro: add nodepool cloud configuration https://review.opendev.org/c/opendev/system-config/+/868433 | 20:16 |
| ianw | nodepool.yaml is like a little time capsule. we've gone from linaro -> linaro_london -> linaro_us -> linaro completing the circle | 20:41 |
| fungi | revert to the future! | 20:47 |
| opendevreview | Ian Wienand proposed opendev/system-config master: Remove old nodepool.yaml testing variables https://review.opendev.org/c/opendev/system-config/+/868432 | 20:53 |
| opendevreview | Ian Wienand proposed opendev/system-config master: linaro: add nodepool cloud configuration https://review.opendev.org/c/opendev/system-config/+/868433 | 20:53 |
| opendevreview | Ian Wienand proposed opendev/system-config master: Run nodepool on testing credentials change https://review.opendev.org/c/opendev/system-config/+/868436 | 20:53 |
| clarkb | the tox unpin is in the gate now. We'll probably need https://review.opendev.org/c/opendev/system-config/+/867269 | 20:53 |
| ianw | clarkb: lgtm although the change says its overrides which it doesn't any more? | 20:54 |
| clarkb | oh ya I removed it but didn't update the commit message whoops | 20:54 |
| clarkb | I think we should just send it in unless we feel strongly enough to update the commit message and go throughcheck again | 20:55 |
| ianw | i'm fine with that | 20:55 |
| clarkb | I approved it | 20:55 |
| ianw | ++ | 20:56 |
| clarkb | noonedeadpunk: according to the timestamp in the mirror volume it did update as scheduled. I don't know what I'm looking for in the meirror to knowif it updated to include the content you expect but may be worth checking | 21:01 |
| noonedeadpunk | python 3.9.16 that should not exist | 21:02 |
| noonedeadpunk | it's coming from BaseOS | 21:03 |
| noonedeadpunk | I can trigger re-check ofc... | 21:03 |
| clarkb | you can also just look at the mirror content | 21:04 |
| clarkb | looking at https://mirror.bhs1.ovh.opendev.org/logs/rsync-mirrors/centos-stream.log I don't see it listing python in its incremental file list though | 21:05 |
| noonedeadpunk | well. it should be deleted from our mirror | 21:06 |
| clarkb | our python3 pacakge list seems to be synced with http://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/ | 21:06 |
| clarkb | comparing upstream to https://mirror.bhs1.ovh.opendev.org/centos-stream/9-stream/BaseOS/x86_64/os/Packages/ there are a number of python3 packages ut they seem in agreement between the two | 21:07 |
| noonedeadpunk | So, that's actually the issue we faced https://zuul.opendev.org/t/openstack/build/c4c3101b31c9402287cda4f6cee95166/log/job-output.txt#4887 | 21:07 |
| noonedeadpunk | eventually I'm not 100% sure where this python comes from... | 21:07 |
| clarkb | ok I think I've just established that our mirror is in sync with the facebook mirror. And earlier you asserted the facebook mirror was in sync with the rackspace mirror. | 21:08 |
| noonedeadpunk | but https://9de4e16122dd30a896f8-2244a524143d7353df0994b89e76e3ec.ssl.cf1.rackcdn.com/867954/3/check/openstack-ansible-deploy-aio_lxc-centos-9-stream/c4c3101/logs/etc/host/yum.repos.d/ doesn't give much options | 21:08 |
| clarkb | Do we know if they are in sync with upstream? | 21:08 |
| noonedeadpunk | Oh..... | 21:09 |
| noonedeadpunk | `Package python3-3.9.16-1.el9.x86_64 is already installed.` | 21:09 |
| opendevreview | Merged opendev/system-config master: Update tox.ini for tox v4 https://review.opendev.org/c/opendev/system-config/+/867269 | 21:09 |
| noonedeadpunk | Eventually, that's not mirror... It's nodepool image I guess that al;ready contains python that is not in the mirrors yet | 21:10 |
| noonedeadpunk | How that could happen? | 21:10 |
| clarkb | noonedeadpunk: the nodepool images are likely not built with the mirrors. I know we configure the mirrors for deb platforms. Not sure why we wouldn't on rpm platforms | 21:10 |
| clarkb | ok rackspace is NOT in sync with facebook | 21:11 |
| noonedeadpunk | but still, if everything in sync | 21:11 |
| clarkb | rackspace has the newer python so switching to rackspace would fix it. Facebook is behind | 21:11 |
| clarkb | I've +2'd your change as a result | 21:11 |
| noonedeadpunk | huh, indeed. | 21:12 |
| noonedeadpunk | I was sure it wasn't there... | 21:12 |
| noonedeadpunk | But maybe overlooked... | 21:12 |
| noonedeadpunk | But you're right clarkb - using mirrors for dib would reduce amount of hassle in case of mirrors being de-synced | 21:13 |
| clarkb | I know we do it for this reason on the deb platforms. I can only assume there is some reason we don't do it for the rpm platforms but I don't know what it is | 21:13 |
| noonedeadpunk | from one side we simply won't notice that. but it's a good thing? | 21:13 |
| noonedeadpunk | Maybe nobody cared as ubuntu is main platform? | 21:14 |
| noonedeadpunk | Like the same reason why there're no rocky mirrors are present at all | 21:14 |
| noonedeadpunk | usage vs profit vs effort | 21:15 |
| clarkb | looks like openeuler and opensuse images did it too. SO ya maybe it just needs to be added? | 21:15 |
| clarkb | well we have the mirrors for centos stream | 21:15 |
| clarkb | configuring dib to use them is just one line of config per image | 21:15 |
| clarkb | rocky doesn't have mirrors because bootstrapping mirrors from nothing is a lot of work (and disk space) | 21:15 |
| noonedeadpunk | and devstack wasn't even failing with this de-sync, as for devstack it simply ignored dnf install error | 21:17 |
| clarkb | iirc dnf and yum will do that if you install a list of things and at least one of them succeeds it treats the whole lot as a success | 21:18 |
| ianw | iirc devstack should be looking at the output to detect that. but it might have changed/broken | 21:19 |
| noonedeadpunk | ah, yes, it's one of 2 packges that failed | 21:20 |
| ianw | i don't know how these mirrors keep getting out of sync. i wish we had someone on the centos side to talk to about it, because i wonder if it's actually the remote mirrors, or something from the central infrastructure | 21:20 |
| ianw | it seems like it's a black box, i've tried sending messages to one of the mirror admin lists before and got nothing | 21:20 |
| noonedeadpunk | at least epel mirror maintainer said it happens during new "release" as too much packages are being added/change for mirrors to couple with them. This also involved Fedora releases | 21:21 |
| clarkb | noonedeadpunk: in theory they can be syncd such taht the new packages aren't visible until all packages are synce though | 21:21 |
| noonedeadpunk | So they had like mirror with super minimal set of distros supported to eliminate effect of releases | 21:21 |
| clarkb | via repodata.xml or something | 21:22 |
| noonedeadpunk | for devstack it was https://zuul.opendev.org/t/openstack/build/1b37b0e31dd14aa398a71377ee27eb4b/log/job-output.txt#2264 fwiw | 21:22 |
| ianw | if only there was some system where you could publish a globally visible volume and release a known-good version atomically ... | 21:23 |
| noonedeadpunk | heh, yeah | 21:23 |
| noonedeadpunk | so back to dib... I'm not sure where it's defined :( | 21:24 |
| * noonedeadpunk quite ashamed | 21:24 | |
| opendevreview | Clark Boylan proposed zuul/zuul-jobs master: Avoid tox-bindep in test-tox job https://review.opendev.org/c/zuul/zuul-jobs/+/866945 | 21:24 |
| opendevreview | Clark Boylan proposed zuul/zuul-jobs master: Flip ensure-tox to tox v4 by default https://review.opendev.org/c/zuul/zuul-jobs/+/866943 | 21:24 |
| ianw | https://opendev.org/openstack/diskimage-builder/src/branch/master/diskimage_builder/elements/openstack-ci-mirrors/environment.d/11-dib-distribution-mirror.bash is something like it | 21:28 |
| ianw | i'd have to context switch back in all the variables in play, IIRC it's not quite as simple as debootstrap because of the way we make the initial chroot | 21:29 |
| clarkb | openeuler does it but maybe it has a different generation process | 21:30 |
| clarkb | or maybe it is set but not taking effect | 21:30 |
| opendevreview | Ian Wienand proposed openstack/project-config master: ARM64 nodes : rename larger types to be more descriptive https://review.opendev.org/c/openstack/project-config/+/868439 | 21:32 |
| noonedeadpunk | huh, I see NODEPOOL_UBUNTU_MIRROR being exported only here for example https://opendev.org/opendev/base-jobs/src/branch/master/roles/mirror-info/templates/mirror_info.sh.j2#L59 But I'm not sure that mirror-info anyhow affect nodepool-builder? | 21:44 |
| noonedeadpunk | interesting is that it's setting NODEPOOL_CENTOS_MIRROR but in dib DIB_OS_CI_YUM_REPOS is used. | 21:45 |
| noonedeadpunk | So if mirror-info is correct place so set that up, then it's likely it | 21:46 |
| clarkb | noonedeadpunk: no that does not affect the image builds. | 21:46 |
| clarkb | The clue is that nodepool builds the images not zuul (and that is zuul job configuration). There is a plan to change this in the future though | 21:46 |
| clarkb | noonedeadpunk: openstack/project-config/nodepool/nodepool.yaml is the file to look in | 21:46 |
| ianw | yeah, there is a special element that sets up mirrors during CI | 21:57 |
| opendevreview | Ian Wienand proposed openstack/project-config master: ARM64 nodes : rename larger types to be more descriptive https://review.opendev.org/c/openstack/project-config/+/868439 | 22:12 |
| opendevreview | Ian Wienand proposed openstack/project-config master: nodepool: remove ubuntu-bionic-arm64-large label https://review.opendev.org/c/openstack/project-config/+/868441 | 22:12 |
| opendevreview | Ian Wienand proposed openstack/project-config master: Add new linaro cloud https://review.opendev.org/c/openstack/project-config/+/868442 | 22:12 |
| opendevreview | Ian Wienand proposed openstack/project-config master: nl03 : reorganise labels https://review.opendev.org/c/openstack/project-config/+/868443 | 22:12 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: use-buildset-registry: support microk8s https://review.opendev.org/c/zuul/zuul-jobs/+/867063 | 22:41 |
| opendevreview | Ian Wienand proposed openstack/project-config master: nl03 : reorganise labels https://review.opendev.org/c/openstack/project-config/+/868443 | 22:46 |
| opendevreview | Clark Boylan proposed zuul/zuul-jobs master: Avoid tox-bindep in test-tox job https://review.opendev.org/c/zuul/zuul-jobs/+/866945 | 22:49 |
| opendevreview | Clark Boylan proposed zuul/zuul-jobs master: Flip ensure-tox to tox v4 by default https://review.opendev.org/c/zuul/zuul-jobs/+/866943 | 22:49 |
| ianw | ok, i think everything for the linaro cloud is in https://review.opendev.org/q/topic:linaro-2022 | 23:00 |
| ianw | it will want monitoring though, as i'm not 100% how we'll go booting the nodepool images in the new cloud | 23:01 |
| ianw | moving the builder is still a todo item | 23:02 |
| ianw | we're going to have to manage our space more carefully as i think disk will be our limit on this cloud | 23:02 |
| clarkb | ianw: for the builder did one get deployed there too? Or is that still todo | 23:02 |
| clarkb | and re disk concerns and the builder should we see if we can move the builder into osuosl? | 23:02 |
| ianw | clarkb: still 100% todo, i haven't deployed the node at all. the mirror is up and should be fine | 23:02 |
| clarkb | got it | 23:03 |
| clarkb | maybe the thing to do since we are concerned about disk space is see if Lance has any concerns with deploying that to osuosl instead | 23:03 |
| ianw | yeah, it's probably worth thinking about | 23:04 |
| ianw | i think we have about 1.2tb | 23:04 |
| opendevreview | Clark Boylan proposed zuul/zuul-jobs master: Avoid tox-bindep in test-tox job https://review.opendev.org/c/zuul/zuul-jobs/+/866945 | 23:40 |
| opendevreview | Clark Boylan proposed zuul/zuul-jobs master: Flip ensure-tox to tox v4 by default https://review.opendev.org/c/zuul/zuul-jobs/+/866943 | 23:40 |
| opendevreview | Clark Boylan proposed zuul/zuul-jobs master: Stop trying to use a %s tox target https://review.opendev.org/c/zuul/zuul-jobs/+/868451 | 23:40 |
| *** ianw is now known as ianw_pto | 23:43 | |
| *** dasm is now known as dasm|off | 23:57 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!