| -@gerrit:opendev.org- Monty Taylor https://matrix.to/#/@mordred:inaugust.com proposed: [opendev/system-config] 1007077: Mirror pigsty/silo https://review.opendev.org/c/opendev/system-config/+/1007077 | 00:16 | |
| @mordred:waterwanders.com | Clark: ^^ there's the followup to your zuul-jobs patch :) corvus | 00:17 |
|---|---|---|
| -@gerrit:opendev.org- Monty Taylor https://matrix.to/#/@mordred:inaugust.com proposed: [opendev/system-config] 1007077: Mirror pigsty/silo https://review.opendev.org/c/opendev/system-config/+/1007077 | 00:23 | |
| @mordred:waterwanders.com | this time spelled correctly | 00:23 |
| -@gerrit:opendev.org- Zuul merged on behalf of Clark Boylan: [zuul/zuul-jobs] 1005764: Convert mc to mcli in s3 tests https://review.opendev.org/c/zuul/zuul-jobs/+/1005764 | 00:32 | |
| -@gerrit:opendev.org- Zuul merged on behalf of Clark Boylan: [zuul/zuul-jobs] 1005755: Switch minio/minio to pigsty/silo https://review.opendev.org/c/zuul/zuul-jobs/+/1005755 | 00:32 | |
| -@gerrit:opendev.org- Zuul merged on behalf of James E. Blair https://matrix.to/#/@jim:acmegating.com: [opendev/system-config] 1006632: Reset podman data on zuul components https://review.opendev.org/c/opendev/system-config/+/1006632 | 14:37 | |
| -@gerrit:opendev.org- Zuul merged on behalf of Monty Taylor https://matrix.to/#/@mordred:inaugust.com: [opendev/system-config] 1007077: Mirror pigsty/silo https://review.opendev.org/c/opendev/system-config/+/1007077 | 14:46 | |
| -@gerrit:opendev.org- Clark Boylan proposed: [openstack/project-config] 1007269: Add prometheus as a grafana datasource https://review.opendev.org/c/openstack/project-config/+/1007269 | 16:07 | |
| @clarkb:matrix.org | I have no idea if ^ is correct, but I think if it passes testing then getting that in will make it easier to test in the running instance or with held test nodes? | 16:07 |
| @clarkb:matrix.org | I'm also going to approve https://review.opendev.org/c/opendev/system-config/+/1000873 to have prometheus collect greptimedb metrics | 16:08 |
| @clarkb:matrix.org | actually before I do that I'm going to review the metrics reported at that endpoint | 16:09 |
| @clarkb:matrix.org | ya I don't see anything concerning in there. It does list some paths that greptimedb uses but I think those are all in public config anyway | 16:15 |
| @clarkb:matrix.org | Jens Harbott: did you have a chance to look at https://review.opendev.org/c/opendev/system-config/+/1006976 fungi pointed out that one of the two routers does not appear to currently be valid. I'm not sure if we can determien whether it should be without asking vexxhost though? | 16:20 |
| @clarkb:matrix.org | in the meantime do we want to proceed with config that has a single router in it? I can't imaging that would be worse than the current situation? | 16:28 |
| @fungicide:matrix.org | i don't think it will be different than the current situation | 16:28 |
| @clarkb:matrix.org | Wouldn't it be better as long as the single router doesn't go down? | 16:29 |
| @fungicide:matrix.org | i haven't seen any cases of review.o.o having incorrect entries in its routing table | 16:29 |
| @clarkb:matrix.org | The issue aiui is that our config is being unconfigured by stray RAs. Switching to a static config that ignores those RAs should be more reliable? | 16:29 |
| @fungicide:matrix.org | oh maybe, but i haven't observed that | 16:29 |
| @fungicide:matrix.org | in the cases i personally saw, one of the two core routers was getting incoming packets and didn't know where the review server was | 16:30 |
| @clarkb:matrix.org | oh so this is different to the behavior we saw on review02 then? In that case the static config may not help? | 16:30 |
| @clarkb:matrix.org | I was under the impression that we thought it was the same thing with RAs coming from elsewhere confusing things | 16:31 |
| @fungicide:matrix.org | it seems different to me, but it's been getting debugged independently by multiple people who may have been coming to different conclusions | 16:31 |
| @clarkb:matrix.org | ya double checking right now what I see is ping6 google.com does not succeed. But ip addr reports the expected ipv6 global address and ip -6 route shows the route to the one of two routers that we expect to work right now. | 16:32 |
| @clarkb:matrix.org | Which I think backs up what you are saying and statically configuring things may not actually help here (since the host isn't confuised about who it is or who it should talk to) | 16:33 |
| @clarkb:matrix.org | oh the pings do eventually go through they just take a while to get going | 16:33 |
| @clarkb:matrix.org | I'll defer to others who have done much more debugging on whether or not we want to try the static config afterall. I'm happy to edit it to drop the second route if so (as that one appears to be invalid at the moment) | 16:35 |
| @fungicide:matrix.org | also it sounded like a lot of the troubleshooting had been focused on the mirror server in that region, and i don't know that we can assume whatever's happening on it is the same problem review is having | 16:36 |
| @mnasiadka:matrix.org | These are the same problems, any routing outside of that network there is problematic (in any direction) | 16:37 |
| @fungicide:matrix.org | yeah, in which case static or dynamic route configuration on the servers won't do anything to fix inbound routing | 16:38 |
| @mnasiadka:matrix.org | Kolla-Ansible had to disable mirror usage in that region and set to prefer ipv4 in gai.conf | 16:38 |
| @mnasiadka:matrix.org | And to make things more stable - before VexxHost fixes the problem - probably removing AAAA entry for review and switching gai.conf to prefer ipv4 on the mirror would be the things that would help regain stability | 16:39 |
| @mnasiadka:matrix.org | This is the merged K-A patch for reference: https://review.opendev.org/c/openstack/kolla-ansible/+/1007102 | 16:40 |
| @fungicide:matrix.org | we could drop the aaaa record for the mirror server there as well | 16:40 |
| @clarkb:matrix.org | with the mirror it proxies to pypi and elsewhere which is why the ipv4 preference would help I think | 16:41 |
| @clarkb:matrix.org | but ya dropping AAAA records may be necessary if we think this is all upstream of us | 16:41 |
| @mnasiadka:matrix.org | Ipv6 works flawlessly inside the network, any connection to/from outside is flawed (I tested both mirror and review from another US ipv6 capable cloud and my home broadband in Europe) | 16:42 |
| @mnasiadka:matrix.org | The worst problem is it’s intermittent :) | 16:43 |
| @mnaser:matrix.org | Sorry we're working on this but it's been hard identifying the root exact cause.. but we have a lead | 16:43 |
| @fungicide:matrix.org | sounds promising, thanks mnaser! | 16:43 |
| @mnasiadka:matrix.org | mnaser: no need to be sorry, networking is hard :) maybe that’s some OVN bug? | 16:43 |
| @mnaser:matrix.org | No it's actually a physical router thing it seems | 16:44 |
| @clarkb:matrix.org | how exciting | 16:45 |
| @fungicide:matrix.org | i recommend percussive maintenance, the bigger the hammer the better | 16:45 |
| @clarkb:matrix.org | I'll go ahead and WIP the static config change as thsi seems to confirm it is unlikely to help | 16:46 |
| -@gerrit:opendev.org- Zuul merged on behalf of Michal Nasiadka: [opendev/system-config] 1000873: prometheus: Add scraping of greptimedb metrics https://review.opendev.org/c/opendev/system-config/+/1000873 | 16:48 | |
| @mnasiadka:matrix.org | mnaser: Arista is playing tricks? Interesting | 16:56 |
| -@gerrit:opendev.org- Clark Boylan proposed: [openstack/project-config] 1007269: Add prometheus as a grafana datasource https://review.opendev.org/c/openstack/project-config/+/1007269 | 18:29 | |
| @clarkb:matrix.org | Looks like there may be name normalization and if you're not using a normalized name then uid stuff gets confused and we try to create a datasource that already exists? In any case I'm hoping that simply using a normalized name will make that happier | 18:30 |
| @clarkb:matrix.org | ok 1007269 passes now. In theory if we get that landed we can start constructing new dashbaord via grafana's web ui that we then port into grafyaml config for everyone to see | 19:44 |
| @clarkb:matrix.org | The pre ptg is next week. Organizational notes and other details are going here: https://etherpad.opendev.org/p/opendev-preptg-202609 I've been trying to add a few more ideas on topics to cover as they occur to me. Feel free to add your own or add more information/questions/etc to the ones already on there | 20:58 |
| @fungicide:matrix.org | looks like anubis 1.28 is going to add webassembly challenges for browsers that support it (which will likely improve the human experience since it can take advantage of hardware accelerated computation too) | 21:40 |
| @fungicide:matrix.org | v1.28.0-pre2 was tagged a week ago, so the full release is probably coming in the next few weeks | 21:41 |
| @fungicide:matrix.org | i've upgraded one of my personal servers to it, just to try it out early | 21:45 |
| @clarkb:matrix.org | it will but it won't be used by default. Its also a bit unclear to me if it will actually be better at keeping the bots at bay (I think the theory is that memory is more expensive than compute now so memory heavy challenges are better_ | 21:48 |
| @fungicide:matrix.org | yeah i don't think it's aimed as much at making things harder for bots than the prior version, just making it less annoying for !bots | 21:49 |
| @clarkb:matrix.org | right but we rely on it slowing down the bost currently :) | 21:50 |
| @fungicide:matrix.org | doesn't look like it will make it any easier for the bots though | 21:51 |
| @clarkb:matrix.org | I think it is expected to be much quicker | 21:51 |
| @clarkb:matrix.org | which is the part I'm not sure I understand properly | 21:51 |
| @clarkb:matrix.org | since the time required is what effectively acts as the metering function | 21:51 |
| @clarkb:matrix.org | (the bots are doing the challenge they are not failing it for the most part these days but anubis is still helpful and keeping the thundering herd at bay due to the time required) | 21:52 |
| @fungicide:matrix.org | bots able to run webassembly challenges and offload them to hardware acceleration may be able to punch through easier, yes | 21:52 |
| @fungicide:matrix.org | it also might make it less annoying for normal users even if we wanted to shorten the cookie validity lifetime | 21:54 |
| @fungicide:matrix.org | so basically we could make things harder/slower for bulk access without as much impact to normal browser users | 21:55 |
| @fungicide:matrix.org | anecdotal evidence, upgrading from 1.27.0 to 1.28.0-pre2 with no changes to configuration, trying it out with some of my smaller netbooks that often struggle to solve anubis challenges quickly are now getting through to sites almost instantaneously | 21:58 |
| @clarkb:matrix.org | fungi: I thought the wasm was opt in so required config updates | 22:05 |
| @clarkb:matrix.org | you have to set the challenge method or something | 22:06 |
| @clarkb:matrix.org | ya I think you haev to set the challenge algorithm to argon2id | 22:06 |
| @fungicide:matrix.org | interesting that i'm seeing better performance automatically | 22:08 |
| @fungicide:matrix.org | but this is also admittedly not scientific benchmarking | 22:08 |
| @clarkb:matrix.org | https://anubis.techaro.lol/blog/2026/anubis-wasm/ the first config block shows an example. its also possible they shipped it by default and the blog post is wrong or out of date | 22:09 |
Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!