| ianw | re 263490 -- 34094 * 5 / 60 /60 == 47.3 hours by my calculation. that's essentially two days worth of cpu time we give to failing diff query from stackalytics a month | 00:01 |
|---|---|---|
| ianw | at least if we mined bitcoin with that we might have something to show for it :) | 00:02 |
| clarkb | ooh that is interesting analysis | 00:05 |
| *** rlandy_ is now known as rlandy|out | 00:05 | |
| clarkb | I wonder why it cares about diffs anyway | 00:08 |
| clarkb | maybe line count info. | 00:08 |
| clarkb | but also why does it need to requery it if it is failing already | 00:08 |
| clarkb | ianw: re https://review.opendev.org/c/opendev/system-config/+/445616/ the old system set it to 4096 and then multiplied that by two for ulimits. Then we ported that to the docker system with the chagne you linked I think. So this cahnge proposed way back when doubling it to 8192 * 2 and I updated it to do that under docker | 00:27 |
| clarkb | basically I think we maintained status quo with the chagne you pushed. We didn't actually double as proposed | 00:27 |
| ianw | oh, hrm, ok very possible i misread the history :) | 00:28 |
| ianw | i don't see any of those errors in the logs | 00:29 |
| clarkb | now whether or not we really need that I dunno its been years at the lower limit. But bumping it seems safe and we thought it might be a good idea at one time | 00:29 |
| clarkb | ya I think we've been ok with the current limits. Possibly the thing that changed is we repack more often now | 00:29 |
| clarkb | which means far fewer object files for gerrit to open | 00:29 |
| clarkb | anyway if we want to say this isn't needed anymore we should abandon the change | 00:30 |
| clarkb | and that is a perfectly valid answer | 00:30 |
| ianw | # ls -l /proc/1877/fd | wc -l | 00:31 |
| ianw | 480 | 00:31 |
| ianw | that's how many files it current has open | 00:32 |
| fungi | so about an order of magnitude headroom | 00:32 |
| ianw | we'd probably need to track it to be sure, especially around say periodic job kick-off, etc. | 00:34 |
| clarkb | ya though an order of magnitude difference is a good indication we're good as is | 00:35 |
| clarkb | I've got to help with dinner now but if you want to -1 with "open file count is <500" that is good for me | 00:36 |
| ianw | yeah updated my comment | 00:36 |
| opendevreview | James E. Blair proposed zuul/zuul-jobs master: Add per-build WinRM cert generation https://review.opendev.org/c/zuul/zuul-jobs/+/837416 | 00:43 |
| opendevreview | Merged opendev/git-review master: Drop support for Python 3.5 https://review.opendev.org/c/opendev/git-review/+/837222 | 03:47 |
| opendevreview | Ian Wienand proposed opendev/git-review master: Enforce minimum git version https://review.opendev.org/c/opendev/git-review/+/837451 | 04:26 |
| chkumar|ruck | ianw: hello | 05:14 |
| chkumar|ruck | ianw: https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_103/836104/21/gate/tripleo-ci-centos-9-containers-multinode-wallaby/103c74c/job-output.txt is failing with following error | 05:14 |
| chkumar|ruck | mirror-workspace-git-repos : Synchronize src repos to workspace directory task | 05:15 |
| chkumar|ruck | ECDSA host key for 198.72.124.40 has changed and you have requested strict checking. | 05:15 |
| chkumar|ruck | Host key verification failed. | 05:15 |
| chkumar|ruck | and another example is this https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_971/836104/21/gate/tripleo-ci-centos-9-containers-multinode-wallaby/97147e1/job-output.txt | 05:16 |
| chkumar|ruck | Can you check on infra side? when free, thanks! | 05:16 |
| ianw | hrm, has this happened multiple times? | 05:16 |
| chkumar|ruck | https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-9-containers-multinode-wallaby&project=openstack/tripleo-quickstart | 05:17 |
| chkumar|ruck | one with retry and one with retry_limit | 05:17 |
| chkumar|ruck | another retry does not have logs | 05:17 |
| chkumar|ruck | ianw: just 2 logs | 05:18 |
| ianw | https://zuul.opendev.org/t/openstack/build/97147e1f93714596a3db512f7512f35b/logs - iweb-mtl | 05:20 |
| ianw | https://zuul.opendev.org/t/openstack/build/103c74c36e3443be9a5ae0ae9d53f4d4 - iweb-mtl | 05:20 |
| chkumar|ruck | so it is linked with the provider iweb-mtl01. | 05:21 |
| chkumar|ruck | https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-9-containers-multinode-wallaby jobs running on ovh-bhs1 is working fine, can we avoid this provider iweb-mtl01 ? | 05:25 |
| ianw | maybe; seems like recycled nodes possibly ... | 05:28 |
| ianw | 53f4d4 was 198.72.124.40 | 05:28 |
| ianw | 512f35b was 198.72.124.92 | 05:29 |
| chkumar|ruck | ianw: ok, thank you for looking into that. | 05:36 |
| chkumar|ruck | ianw: is there anything we can do to avoid that? or can we remove recycled nodes? | 05:36 |
| ianw | it's clearly not supposed to do that :) i'm just trying to see if there's any obvious errors | 05:36 |
| chkumar|ruck | ok | 05:36 |
| ianw | 198.72.124.40 has been up for 1:17 | 05:52 |
| ianw | that would be since ... just after 04:36 | 05:53 |
| ianw | 53f4d4 failed at 2022-04-12 03:27:05 | 05:56 |
| *** pojadhav- is now known as pojadhav | 05:57 | |
| ianw | i dunno, i can't see clearly what caused this failure | 06:03 |
| chkumar|ruck | ianw: I will keep an eye on this job, if more comes will let you know with a bug | 06:20 |
| chkumar|ruck | ianw: ++ thank you for checking on it :-) | 06:20 |
| ianw | well, an unsatisfying conclusion so far. it could be something on the cloud side, with the floating-ip not updating, or being reused | 06:21 |
| ianw | I think at this point, yeah, we should monitor for more cases | 06:21 |
| opendevreview | Ian Wienand proposed opendev/git-review master: Enforce minimum git version https://review.opendev.org/c/opendev/git-review/+/837451 | 06:30 |
| frickler | ianw: those two IPs seem to be long running instances again. wasn't iweb supposed to get turned down anyway or am I mixing this up? | 06:37 |
| frickler | also iirc mgagne wanted to look into cleaning things up some time ago but I don't remember seeing a response | 06:38 |
| opendevreview | Dr. Jens Harbott proposed openstack/diskimage-builder master: Update gentoo python version to 3.9 https://review.opendev.org/c/openstack/diskimage-builder/+/836237 | 07:27 |
| *** Tengu_ is now known as Tengu | 08:26 | |
| hrw | can zuul directory view can get sortable headers? | 09:07 |
| hrw | https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_0e6/837293/4/check/kolla-ansible-ubuntu-source-masakari/0e653ec/primary/logs/ansible/ begs for 'sort by date' being available | 09:07 |
| *** rlandy|out is now known as rlandy | 10:31 | |
| fungi | frickler: inap has ended their openstack public cloud, but the iweb staff (different company) are in control of the servers there are trying to keep things running for us a while longer | 11:30 |
| fungi | hrw: sortability might be hard since that index you linked is a static html file served from a swift container. if you can come up with a way to serve sortable indexes from the arbitrary swift services in public cloud providers, then perhaps | 11:32 |
| fungi | ianw: `ssh root@198.72.124.40 uptime` says "up 2 days" | 11:34 |
| fungi | the problem is that there's more than one virtual machine with that ip address, so just because the one you end up at when you check has a short uptime, that doesn't mean it's the problem vm | 11:34 |
| fungi | anyway, i'm going to tell it to poweroff | 11:34 |
| hrw | fungi: ok. | 11:35 |
| hrw | or will look for some userscript to handle that | 11:36 |
| fungi | hrw: it's far more likely the logs view of the build result page could grow some smarter indexes, since it's already a javascript app. i almost never use the raw indexes in swift anyway | 11:37 |
| hrw | times when I understood javascript passed long time ago | 11:40 |
| hrw | ;( | 11:41 |
| fungi | i still don't understand javascript, if that helps | 11:41 |
| hrw | https://gist.github.com/ameboide/368797ecaa18bb9ac11f866e12156c95 adds some simple sorting. good enough | 11:43 |
| hrw | fungi: ;) | 11:46 |
| frickler | repo mirror issues seem to continue, now something fedora in case somebody wants to dig deeper https://78fd1c0a167dfb96b269-6e86dc8a2337ee139316a184fc899cae.ssl.cf5.rackcdn.com/836237/2/check/dib-functests/ae0b17d/logs/fedora_build-succeeds.FAIL.log https://zuul.opendev.org/t/openstack/build/ae0b17d50bcb488b8b5206fac036aa66 | 11:52 |
| frickler | and I also never use the build result page for logs because it tends to clog up my firefox | 11:53 |
| hrw | not only yours | 11:54 |
| fungi | Status code: 404 for http://mirror.iad3.inmotion.opendev.org/fedora/updates/35/Everything/x86_64/repodata/026490e6ea124e04df9d50fd577848426ac58a09b7401a8dca4a7cc43bde172d-filelists.xml.zck | 11:55 |
| hrw | repodata out of sync | 11:55 |
| fungi | frickler: that looks like what we saw last week when there was some significant churn for f35 package updates and the second-tier mirror we were syncing from was apparently mid-update | 11:55 |
| fungi | it cleared on its own eventually when they finished updating their mirror | 11:56 |
| fungi | https://static.opendev.org/mirror/logs/rsync-mirrors/fedora.log | 11:58 |
| frickler | would be nice if we had a way to keep snapshots of working states until that happens | 11:58 |
| fungi | frickler: the trick would be to identify "working states" | 11:59 |
| fungi | if we could do that automatically, we could just have the rsync script check mirror consistency and exit nonzero for problems, at which point we'd avoid calling vos release and the read-only replicas would remain unchanged until the version in the read-write volume achieved a consistent state | 12:00 |
| frickler | fungi: just select some set of test jobs that you can point at a proposed update | 12:00 |
| fungi | our mirror updates aren't triggered by zuul though | 12:00 |
| fungi | "some random jobs pass" is a rather poor (and likely inefficient) means of checking that the mirror content is internally consistent | 12:01 |
| fungi | maybe the fedora maintainers have a tool which can be used to check mirror consistency quickly? | 12:02 |
| frickler | maybe inefficient but essentially the kind of check we care about in the end | 12:02 |
| fungi | no, not really. "all jobs are fine with the mirror content" is what we care about | 12:02 |
| fungi | "random jobs are fine with the mirror content" isn't the same | 12:02 |
| fungi | if there's a corrupt package then you won't find out in your canary if it doesn't install that package | 12:03 |
| frickler | I didn't suggest random, selecting a representative subset of all jobs might be possible | 12:03 |
| frickler | also it doesn't need to have 100% coverage, the usual repodata issues should get caught by almost any job I guess | 12:04 |
| frickler | these also don't seem to be specific to fedora, more a general issue for rpm-based distros | 12:06 |
| fungi | a tool we can run from the rsync script would be an efficient way to block releasing an inconsistent/incomplete content update. finding ways to have jobs run against candidate states of the mirror and then somehow provide a signal which blocks calling vos release until it's working seems very fragile, as well as unlikely to provide complete coverage | 12:08 |
| fungi | if we redid our mirror update mechanism to be entirely run from zuul, that might be slightly easier to engineer, but still at a minimum implies shadow paths or vhosts for the mirror sites so that nodes could be switched to fetch the candidate content | 12:10 |
| fungi | since we'd also need to serve the read-write volumes alongside the read-only replicas | 12:10 |
| fungi | and would still be an indirect/partial test of mirror consistency | 12:12 |
| hrw | https://fedoraproject.org/wiki/Infrastructure/Mirroring can be handy | 12:12 |
| fungi | yep, quite familiar with it, but we don't qualify to use that and sync from their first-tier mirrors based on prior discussions | 12:13 |
| hrw | imho CI should count as private mirror | 12:14 |
| fungi | https://fedoraproject.org/wiki/Infrastructure/Mirroring#How_can_someone_make_a_private_mirror.3F "Private mirrors cannot pull from the master Fedora download servers. They must pull from another listed public mirror." | 12:16 |
| hrw | so Fedora master -> 3rd party mirror -> infra mirror, right? | 12:18 |
| fungi | yes, and the problem arises when "3rd party mirror" is in an inconsistent state, we sync from it, and then our mirrors reflect the same state | 12:20 |
| fungi | i've been trying to figure out if https://pagure.io/quick-fedora-mirror/blob/master/f/quick-fedora-mirror has any consistency checking features, i know we've discussed using it in the past | 12:21 |
| hrw | ok | 12:22 |
| fungi | looks like we've been talking about quick-fedora-mirror for 5 years now: https://meetings.opendev.org/irclogs/%23openstack-infra/%23openstack-infra.2017-04-04.log.html#t2017-04-04T17:40:22 | 12:26 |
| fungi | unfortunately i'm still not seeing anything in quick-fedora-mirror which would notice when the mirror you're syncing from is in an inconsistent state | 12:29 |
| fungi | not even a simple index reference checker | 12:29 |
| fungi | we could probably write one, but i get the impression rpm/yum/dnf/whatever change their indexing models over time so it may need to do different things for centos 7 mirror indices than for fedora 35 | 12:30 |
| fungi | and if it's not something written/used by the distros themselves, then it will probably bitrot and need regular maintenance | 12:31 |
| hrw | this is where reprepro from Debian does the job better iirc as it fetches packages and rebuilds indexes locally | 12:33 |
| hrw | iirc because I use it for own repos only nowadays | 12:33 |
| fungi | with the downside being that the package manager can no longer rely on the distro's index signatures | 12:35 |
| fungi | (we could sign them ourselves and bake our signing key into our images, but there's not much point) | 12:35 |
| fungi | and yeah, i'm familiar enough with apt repository indices that i could whip up a consistency checker fairly easily, but i know very little about what dnf expects | 12:36 |
| hrw | no idea too | 12:43 |
| Clark[m] | Not really here yet today, but I feel like official public mirrors are broken and we reflect that state isn't a bug on our end. If it is something we can address via verification (as with reprepro) then great let's do it. But it is also a very low priority thing for me and ideally upstream would fix their update order | 12:54 |
| Clark[m] | It's similar to CentOS breaking ping. We could fix that but it would be counterproductive to as it would require everyone to magically know for a month while it is broken how we fixed it. Instead pressure should be pushed upstream to correct those problems. We correctly caught and reported them and that is a success of the ci system | 12:56 |
| *** rlandy is now known as rlandy|mtg | 13:03 | |
| *** pojadhav is now known as pojadhav|afk | 14:02 | |
| *** dviroel is now known as dviroel|mtg | 14:15 | |
| opendevreview | Tristan Cacqueray proposed opendev/system-config master: Update gerritbot-matrix version to include wipness https://review.opendev.org/c/opendev/system-config/+/837580 | 14:42 |
| * frickler likes "wipness" :) | 14:56 | |
| clarkb | fungi: https://review.opendev.org/c/opendev/bindep/+/837394 and its child may be of interest to you | 15:49 |
| clarkb | bindep updates for dependency change and alma linux support | 15:49 |
| fungi | thanks, and yeah i've been meaning to look at those | 15:50 |
| *** dviroel|mtg is now known as dviroel|lunch | 16:02 | |
| *** rlandy|mtg is now known as rlandy | 16:03 | |
| clarkb | the gerrit mailing list has an interesting discussion on rehoming a project from one gerrit server to another | 16:08 |
| clarkb | the tldr is that you cannot do this easily because you need to update the installation uuid in the repo and all of the change numbers in a repo need to be update so that they do not collide with existing changes in the existing server | 16:09 |
| clarkb | calling this out beacuse this is theoretically something we need to be on the lookout for when importing projects into our gerrit. However, I think the way jeepyb does the imports it will fail to push any existing changes into our gerrit thus bypassing the issue | 16:09 |
| *** pojadhav|afk is now known as pojadhav | 16:14 | |
| fungi | i thought the ideas with the new project-specific change urls was that gerrit would eventually be able to move away from a global index number | 16:24 |
| fungi | but i guess that will take a while and break lots of other assumptions | 16:24 |
| clarkb | ya so far they haven't done that. YOu can still go to https://review.opendev.org/$number and it works | 16:26 |
| opendevreview | sean mooney proposed openstack/project-config master: update Review-Priority lable for nova related projects https://review.opendev.org/c/openstack/project-config/+/837595 | 16:32 |
| *** marios is now known as marios|out | 16:40 | |
| *** dviroel|lunch is now known as dviroel | 16:58 | |
| clarkb | fungi: re iweb it seems they haev gone into a "all nodes are stuck in deleting" state | 17:00 |
| *** artom_ is now known as artom | 17:01 | |
| clarkb | on the nodepool side that seems to manifest as a timeout waiting for node cleanup. I suspect that means the apis are still up otherwise we'd get 500 errors or network connectivity problems | 17:01 |
| opendevreview | Jeremy Stanley proposed opendev/git-review master: Revert "Drop support for Python 3.5" https://review.opendev.org/c/opendev/git-review/+/837602 | 17:21 |
| clarkb | I guess due to centos 7 we'd need to detect git version anyway which means that old python can live another day too | 17:23 |
| fungi | yeah, i meant for those two changes to be approved together, they were just separated for ease of evaluation | 17:25 |
| clarkb | fwiw I think I'm ok with telling centos 7 users to install old git-review | 17:25 |
| clarkb | which is what I also suggested that potential old python3 users could do | 17:25 |
| clarkb | they would do it for different reasons but that end result would be the same. | 17:26 |
| fungi | yep, i'm okay with it either way, we just technically don't need to drop py35 testing unless there's a specific change which causes us to be unable to continue | 17:26 |
| clarkb | right | 17:26 |
| clarkb | I have removed my WIP from https://review.opendev.org/c/opendev/system-config/+/831441 with an indication of why I think it is safe to land that now. SHould be an easy review if you ahve a moment | 17:36 |
| clarkb | Not sure how much longer we want to keep the grafana graphs for that cloud: https://review.opendev.org/c/openstack/project-config/+/831442 but that chagne is still WIP'd | 17:37 |
| mgagne | clarkb fungi : I'm working on it right now. I managed to find and delete a bunch of rogue instances. There was also an incident that messed up our cells v1 databases. | 17:59 |
| fungi | oh thanks mgagne! | 18:01 |
| mgagne | I'll take a look at the ones stuck in deleting. | 18:01 |
| clarkb | mgagne: not urgent I just wanted to make note of that as I took a look to ensure it wasnt' something on our end | 18:01 |
| clarkb | and thank you! | 18:02 |
| fungi | mgagne: also i ssh'd in and directly powered off a couple of rogue virtual machines we noticed at 198.72.124.40 and 198.72.124.92 which had been up for a day or two and were conflicting with new nova instances which got those addresses assigned | 18:04 |
| mgagne | yes, I found some rogue instances that were marked as deleted in the api cell database but not compute cell database. we are still running cells v1 because why not. | 18:05 |
| fungi | heh | 18:06 |
| fungi | clarkb: this is an interesting behavior... look at the three most recent zuul comments on https://review.opendev.org/837118 | 18:08 |
| fungi | side effect of the depends-on failure reporting when used for another change in the same project? | 18:08 |
| mgagne | instances stuck in deleting are now deleted. I see a whole bunch of instances in spawning, I suppose a new focal image was uploaded recently? | 18:14 |
| clarkb | sorry was grabbing early lunch due to the noon infra meeting | 18:33 |
| clarkb | fungi: looks like the parent did fail twice? | 18:35 |
| clarkb | so I think those messages are correct | 18:36 |
| clarkb | and I would expect them to be posted even without the depends on | 18:36 |
| fungi | mgagne: looks like the most recent focal image uploaded to iweb was 1.5 hours ago | 18:36 |
| clarkb | mgagne: yes an hour an a half ago according to an image listing | 18:36 |
| fungi | clarkb: oh! it looks like maybe my comment on 837394 at 17:59 triggered a reentry into the gate pipeline | 18:39 |
| fungi | i didn't start it with the word "recheck" so wasn't expecting that to happen | 18:39 |
| clarkb | ah | 18:39 |
| fungi | i wonder why that triggered it | 18:39 |
| clarkb | the failures are related to the fedora mirroring torubles that were discussed earlier today | 18:40 |
| fungi | right, hence my comment | 18:41 |
| fungi | though looking into the mirror-update logs, the last time rsync saw file changes was around 13 hours ago, so the mirror we're pulling from must be stuck in a bad state | 18:42 |
| ianw | thanks for looking into those nodes. it seems obvious now that the 198.72.124.40 I ended up at might not have been the same 198.72.124.40 seen from another location. i wonder if there's some sort of LB effect going on where the source location matters | 19:19 |
| fungi | it's almost certainly just your basic arp overwrites at work | 19:21 |
| fungi | so which one you end up connecting to depends on which mac the gateway has seen use that address and recorded to its arp table at the time | 19:21 |
| ianw | I wonder why https://mirror.aarnet.edu.au/pub/fedora/linux/updates/35/Everything/x86_64/repodata/ has more files than http://mirror.iad.rax.opendev.org/fedora/updates/35/Everything/x86_64/repodata/ | 19:45 |
| opendevreview | Merged opendev/system-config master: Remove byobu profile configuration https://review.opendev.org/c/opendev/system-config/+/529426 | 19:46 |
| clarkb | ianw: we filter out things like source rpms and large image files | 19:47 |
| ianw | this is the repodata though, this other mirror has different -primary.xml files etc. | 19:48 |
| clarkb | ah | 19:48 |
| ianw | i'm wondering if maybe we should be doing additive updates or something, then pruning old files after a given time (maybe, clutching at straws) | 19:49 |
| clarkb | ianw: have you checked our upstrema though? | 19:49 |
| clarkb | I htink fungi was indicating it too lacks that content | 19:49 |
| fungi | i don't think delayed deletes help in this case, without knowing that we need to delay until the parent mirror finishes updating and reaches coherency | 19:50 |
| ianw | yes, we look the same as upstream (http://pubmirror2.math.uh.edu/fedora-buffet/fedora/linux/updates/35/Everything/x86_64/repodata/) | 19:51 |
| ianw | different to https://lon.mirror.rackspace.com/fedora/updates/35/Everything/x86_64/repodata/ though | 19:52 |
| ianw | so http://mirror.iad.rax.opendev.org/fedora/updates/35/Everything/x86_64/repodata/repomd.xml == https://lon.mirror.rackspace.com/fedora/updates/35/Everything/x86_64/repodata/repomd.xml | 19:55 |
| ianw | but 8da0646b3c0dea2866247260d4b2d05073d92a7addf1c9612b09b2c67b1dadc2-primary.xml.gz just doesn't exist on the mirror | 19:57 |
| fungi | yeah, that's about where i got to with it | 20:01 |
| ianw | i feel like if i go back through history i'm going to see lon.mirror.rackspace.com being taken out for some reason, but it does seem like a good option... | 20:02 |
| ianw | although does that "lon." stand for "london"? | 20:02 |
| ianw | Sunbury-on-Thames, | 20:03 |
| ianw | Surrey, | 20:03 |
| ianw | England, | 20:03 |
| ianw | United Kingdom, | 20:03 |
| ianw | Europe | 20:03 |
| opendevreview | Clark Boylan proposed opendev/base-jobs master: Stop submitting logstash/subunit gearman jobs https://review.opendev.org/c/opendev/base-jobs/+/837619 | 20:03 |
| clarkb | fungi: ^ I went ahead and decied to remove the roles too. But let me know if you think we should do it more carefully only removing the use of the roles first. I figure a revert is mostly the same either way so why not rip it all out | 20:04 |
| clarkb | ianw: we have asked a couple of times for thoes involved in fedora to try and help us find a better mirror and people tend to point at the most top level mirror which we cannot pull from as we are not publicly listed | 20:05 |
| ianw | weird, so that's the one listed in the fedora mirror page. but mirror.rackspace.com responds and is us local | 20:05 |
| fungi | i didn't check the git history for that line, but wouldn't be surprised to find it there | 20:06 |
| ianw | i'll try a manual pull from mirror.rackspace.com on mirror-update for f35 and see if it works ... | 20:06 |
| ianw | it looks in sync | 20:06 |
| fungi | awesome | 20:07 |
| opendevreview | Merged opendev/statusbot master: twitter: send log level messages https://review.opendev.org/c/opendev/statusbot/+/837082 | 20:07 |
| clarkb | fungi: I've +A'd https://review.opendev.org/c/opendev/git-review/+/837451/ I think to remove frickler's wip vote on the parent change we need to escalte privs? Is that osmething you want to do? | 20:07 |
| corvus | ianw: clarkb fungi : ^ i left a comment on the twitter change | 20:08 |
| clarkb | hrm I'm not familiar enough with twitter to know if there are expectations one way or the other about verbosity | 20:09 |
| clarkb | to be fair we tend to not #status log super often either and that info is still probably within realms of reason? | 20:10 |
| corvus | i think it's less a platform expectation, and what is it we were actually trying to offer to folks | 20:10 |
| corvus | i log boring stuff with commit shas all the time | 20:10 |
| corvus | but if "log of zuul commit shas deployed" is what our users want, who am i to argue :) | 20:11 |
| fungi | i'm okay with reverting it too, i similarly don't use twitter so i'm not sure what the balance should be between "look opendev is transparent and you can follow what's going on in excruciating detail" vs "please not another log entry about package upgrades" | 20:11 |
| clarkb | I wonder if we hashtagged them with say #notice or #log if users could filter them more readibly | 20:12 |
| clarkb | *readily | 20:12 |
| clarkb | that might be a good compromise? | 20:12 |
| corvus | yeah. it's a tradeoff and i don't know what balance we're aiming for. mostly i wanted to highlight that if the goal is "make sure people know about service interruptions and changes" this may be changing the balance in the other direction. | 20:13 |
| corvus | not a twitter expert, but i think the options are pretty coarse. another option may be to use multiple accounts (one for high-value/low-volume, another for firehose). | 20:14 |
| corvus | pretty sure twitter themselves have tons of official accounts :) | 20:14 |
| fungi | that's certainly an idea | 20:14 |
| corvus | anyway, that's totally a +0 comment from me. i am not in the constituency :) | 20:15 |
| ianw | well, it hasn't worked for, a long time -- since twitter dropped part of their API -- so wasn't achieving anything. maybe many years ago, when we had 2-to-3 times as many people actively involved in infra things, it would be "too much". clearly, since i proposed it, i think we're better to try and drive some engagement with the day-to-day things at this point though | 20:16 |
| clarkb | ya I think I'm willing to try something different and if people complain we can revert or split to two accounts etc | 20:17 |
| corvus | okay, i just didn't see any discussion of the flip side. should i continue to log "boring" things, or only "interesting" things? is there any change we should make to the structure of log entries now that they have an audience? | 20:17 |
| corvus | i certainly have phrased notice and alerts different from log because of the differences in publishing targets | 20:18 |
| corvus | like, i use capital letters and complete sentences | 20:18 |
| clarkb | I don't think we need to change how we use status log. Its always professional and has useful info (if potentially boring) | 20:19 |
| opendevreview | Merged opendev/system-config master: Use deb-docker mirror for upload image too https://review.opendev.org/c/opendev/system-config/+/689858 | 20:20 |
| corvus | wfm | 20:22 |
| ianw | i guess these rackspace mirrors don't actually do rsync, despite what the mirror page says :/ | 20:23 |
| opendevreview | Clark Boylan proposed openstack/project-config master: Remove geard graphing from zuul-status dashboard https://review.opendev.org/c/openstack/project-config/+/837621 | 20:26 |
| clarkb | corvus: ^ thtas the graph cleanup which includes clenaup of an old zuul geard graph too | 20:26 |
| corvus | hah, that really messes up the layout doesn't it? we need to add 2 graphs or drop 1 :) or maybe change that to a 2x2? | 20:28 |
| corvus | i like the 2x2 idea | 20:29 |
| corvus | row1: zuul jobs/hour | node requests | 20:29 |
| corvus | row2: gerrit events/hour | test nodes | 20:30 |
| clarkb | let me see how to do that | 20:30 |
| corvus | clarkb: i think change it to `span: 6` for the remaining 4 graphs | 20:30 |
| corvus | they're already in the right order so that should be it | 20:30 |
| corvus | (a row is 12 units total) | 20:31 |
| opendevreview | Clark Boylan proposed openstack/project-config master: Remove geard graphing from zuul-status dashboard https://review.opendev.org/c/openstack/project-config/+/837621 | 20:31 |
| clarkb | got it | 20:31 |
| ianw | well, trying a dry-run with mirrors.us.kernel.org brings in : Everything/x86_64/repodata/8da0646b3c0dea2866247260d4b2d05073d92a7addf1c9612b09b2c67b1dadc2-primary.xml.gz | 20:38 |
| ianw | i.e. should be in sync | 20:38 |
| ianw | again, i feel like we've switched away from this mirror before | 20:38 |
| clarkb | yes pretty sure we've used that one before | 20:39 |
| ianw | https://review.opendev.org/c/opendev/system-config/+/553052 | 20:40 |
| ianw | though that was "mirrors.kernel.org", rather than "mirrors.us.kernel.org" ... that may make a difference | 20:40 |
| clarkb | they resolve to the same location for me | 20:41 |
| clarkb | but that may not be true from our mirror updater | 20:41 |
| ianw | yeah it does seem to change | 20:41 |
| ianw | i got a sfo once, then mirror-update just got pdx | 20:41 |
| ianw | i think that it's also been probably years since we tried it, so it's not like a constant flip-flop | 20:42 |
| ianw | tibbs is in #fedora-admin as is noted as the contact, i'll try reaching out quickly | 20:46 |
| opendevreview | Ian Wienand proposed opendev/system-config master: mirror-update: switch fedora to kernel.org mirror https://review.opendev.org/c/opendev/system-config/+/837625 | 20:50 |
| *** dviroel is now known as dviroel|out | 20:55 | |
| opendevreview | Merged opendev/system-config master: mirror-update: switch fedora to kernel.org mirror https://review.opendev.org/c/opendev/system-config/+/837625 | 22:17 |
| *** rlandy is now known as rlandy|bbl | 22:35 | |
| ianw | i'm running a fedora sync with the lock | 22:37 |
| fungi | thanks! | 22:42 |
| clarkb | we're looking at a thing in #openstack-qa that seems to be causing all devstack jobs on ubuntu focal to fail due to pip/pbr/setuptools not being able to install keystone | 22:44 |
| clarkb | I'm working on holding a node to look at and there is some suspicion that a git package update from today could be to blame | 22:44 |
| clarkb | just fyi in case we start to see problems like that elsewhere | 22:44 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!