| -@gerrit:opendev.org- Goutham Pacha Ravi proposed: [openstack/project-config] 987313: Add devstack-plugin-lustre project https://review.opendev.org/c/openstack/project-config/+/987313 | 03:25 | |
| -@gerrit:opendev.org- Goutham Pacha Ravi proposed: [openstack/project-config] 987313: Add devstack-plugin-lustre project https://review.opendev.org/c/openstack/project-config/+/987313 | 05:14 | |
| @tkajinam:matrix.org | I'm afraid review.opendev.org is down (or at least unreachable for a few people) | 06:08 |
|---|---|---|
| @mnasiadka:matrix.org | tkajinam: I started it back, most probably another hypervisor OOM kill | 06:34 |
| @mnasiadka:matrix.org | #status Started review03 after it was found in a SHUTOFF state. | 06:35 |
| @status:opendev.org | @mnasiadka:matrix.org: unknown command | 06:35 |
| @mnasiadka:matrix.org | * #status log Started review03 after it was found in a SHUTOFF state. | 06:36 |
| @mnasiadka:matrix.org | #status log Started review03 after it was found in a SHUTOFF state. | 06:36 |
| @status:opendev.org | @mnasiadka:matrix.org: finished logging | 06:36 |
| -@gerrit:opendev.org- Zuul merged on behalf of Ivan Anfimov: [openstack/project-config] 960321: Add translation-jobs to mistral-dashboard https://review.opendev.org/c/openstack/project-config/+/960321 | 10:54 | |
| @harbott.osism.tech:regio.chat | I'm still seeing the "no results" issue on fresh changes, e.g. https://zuul.opendev.org/t/openstack/build/b33300c2ccbe4459948d33bdba45168f, interestingly only the "Task Summary", "Logs" and "Console" tabs are affected, the Artifacts show up fine | 12:49 |
| @harbott.osism.tech:regio.chat | mnaser: did you have a chance yet to look into the repeated issues with review03? | 12:51 |
| @harbott.osism.tech:regio.chat | Doug Goldstein: do you happen to have any insight on how IPv6 support for raxflex is progressing? if that were solved, we could consider moving gerrit and other services over there | 12:52 |
| @fungicide:matrix.org | Jens Harbott: for your "no results" build example there, i'm getting normal content. that would imply the zuul webui in your browser is having trouble fetching https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_b33/openstack/b33300c2ccbe4459948d33bdba45168f/job-output.json in order to populate the content | 12:53 |
| @mnasiadka:matrix.org | > <@harbott.osism.tech:regio.chat> I'm still seeing the "no results" issue on fresh changes, e.g. https://zuul.opendev.org/t/openstack/build/b33300c2ccbe4459948d33bdba45168f, interestingly only the "Task Summary", "Logs" and "Console" tabs are affected, the Artifacts show up fine | 12:53 |
| It looks fine for me | ||
| @harbott.osism.tech:regio.chat | well without fetching the job-output.json, how could it display the artifacts tab? note that I also restarted my server (and thus firefox), too, but still getting this | 12:59 |
| @jim:acmegating.com | the artifacts list comes from the zuul api | 13:37 |
| @jim:acmegating.com | Jens Harbott: you might open up your browser network inspection panel before loading up zuul results to see if network connection errors show up there | 13:37 |
| -@gerrit:opendev.org- Clark Boylan proposed: [opendev/system-config] 848181: DNM intentional Gitea failure to hold a node https://review.opendev.org/c/opendev/system-config/+/848181 | 15:08 | |
| -@gerrit:opendev.org- Clark Boylan proposed: [opendev/system-config] 840972: DNM force etherpad failure to hold node https://review.opendev.org/c/opendev/system-config/+/840972 | 15:10 | |
| -@gerrit:opendev.org- Monty Taylor https://matrix.to/#/@mordred:inaugust.com proposed: [openstack/project-config] 987373: Add gerrit and zuul plugin repos for hermes-agent https://review.opendev.org/c/openstack/project-config/+/987373 | 15:11 | |
| -@gerrit:opendev.org- Clark Boylan proposed: [opendev/system-config] 987375: Update Gerrit images to 3.12.7 and 3.13.6 https://review.opendev.org/c/opendev/system-config/+/987375 | 15:25 | |
| @clarkb:matrix.org | infra-root one thing I remember doing in the past for test nodes was setting a kernel parameter to limit the total amount of memory that is usable by the operating system. We could do something like that with Gerrit if we think it will help avoid OOMkiller problems in the future. The two issues I see with this are that Gerrit likes memory so limiting it may create other problems and we don't know how much free space we have the hypervisor so we may still end up getting killed. I think it would be `mem=96G` or similar to limit to 96GB from 128GB | 15:28 |
| @clarkb:matrix.org | Another option would be to restart the Gerrit JVM and possibly the Gerrit VM itself every few days to reset memory consumption | 15:28 |
| @clarkb:matrix.org | mnaser: ^ before we do anything like that are you able to weigh in on whether or not we can address this on the hypervisor side of things? | 15:29 |
| @mordred:waterwanders.com | Clark: wow, we can't set memory limits with jvm parameters? | 15:37 |
| @clarkb:matrix.org | mordred: we can and we already do. But linux will helpfully cache lots of files in memory too | 15:39 |
| @clarkb:matrix.org | mordred: the problem is that the hypervisor is OOMkilling the VM so the total memory consumption by the VM is the issue | 15:40 |
| @mordred:waterwanders.com | wat | 15:40 |
| @clarkb:matrix.org | the actual linux VM not the jvm | 15:40 |
| @clarkb:matrix.org | If we do reduce the amount of linux memory then we should reduce teh amount we let the jvm use as I Think the jvm is currently set to allow up to 96GB | 16:05 |
| -@gerrit:opendev.org- Monty Taylor https://matrix.to/#/@mordred:inaugust.com proposed: [openstack/project-config] 987373: Add gerrit and zuul plugin repos for hermes-agent https://review.opendev.org/c/openstack/project-config/+/987373 | 16:07 | |
| -@gerrit:opendev.org- Michal Nasiadka proposed wip on behalf of Christian Berendt: [openstack/project-config] 962116: Remove refstack projects https://review.opendev.org/c/openstack/project-config/+/962116 | 16:11 | |
| -@gerrit:opendev.org- Michal Nasiadka proposed wip on behalf of Christian Berendt: [openstack/project-config] 962116: Remove refstack projects https://review.opendev.org/c/openstack/project-config/+/962116 | 16:12 | |
| -@gerrit:opendev.org- Michal Nasiadka proposed wip on behalf of Christian Berendt: [openstack/project-config] 962116: Remove refstack projects https://review.opendev.org/c/openstack/project-config/+/962116 | 16:12 | |
| @mnasiadka:matrix.org | Thats funny now - I was able to rebase ^^ using Gerrit UI, but I can't move it out of WIP :) | 16:16 |
| -@gerrit:opendev.org- Michal Nasiadka proposed wip on behalf of Christian Berendt: [openstack/project-config] 962116: Remove refstack projects https://review.opendev.org/c/openstack/project-config/+/962116 | 16:30 | |
| -@gerrit:opendev.org- Monty Taylor https://matrix.to/#/@mordred:inaugust.com proposed: [openstack/project-config] 987373: Add gerrit and zuul plugin repos for hermes-agent https://review.opendev.org/c/openstack/project-config/+/987373 | 16:47 | |
| @harbott.osism.tech:regio.chat | `Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_b33/openstack/b33300c2ccbe4459948d33bdba45168f/job-output.json. (Reason: CORS request did not succeed). Status code: (null).` seems this is blocked by noscript. if I allow `ovh.net` in noscript, the page works normally. but that's not something I want to do permanently. also I've been using noscript for years, the issue only appears since sunday, did something change in the way the job json is being handled/loaded? | 16:53 |
| @jim:acmegating.com | Jens Harbott: i am not aware of any changes in zuul or the log upload roles that would affect that. further, the CORS headers are correct for those files, so it's strange that you get that error. when i use noscript with firefox, i get a more clear "Blocked by NoScript" error in the network panel. I am able to configure noscript to allow the fetch for opendev.org with this configuration: | 17:03 |
| @jim:acmegating.com | perhaps something changed with noscript. latest release for firefox was april 23. | 17:04 |
| @clarkb:matrix.org | maybe it is starting to treat json as content that needs extra scrutiny? I think it may have already done that for js? Thats weird though as json should only be data | 17:05 |
| @jim:acmegating.com | 17:05 | |
| like, maybe: `x Remove "fetch" capability from DEFAULT preset unless | ||
| "script" is there too (thanks Security Research Labs for | ||
| suggestion)` | ||
| @jim:acmegating.com | from https://addons.mozilla.org/en-US/firefox/addon/noscript/versions/ | 17:06 |
| @mordred:waterwanders.com | weird | 17:11 |
| @harbott.osism.tech:regio.chat | oh, wow, I never used this custom settings tab before, but enabling exactly that `fetch` capability solved the issue, thanks for the pointer | 17:15 |
| @jim:acmegating.com | you're welcome. and seems like allowing it only for ...opendev.org for the "top page" is reasonable (would be nice to enable it only for zuul.opendev.org, but i didn't see an option for that) | 17:19 |
| @clarkb:matrix.org | 200.225.47.56 is a held gitea-lb node for testing gitea 1.26.1. I've done an `ssh -D 1080 200.225.47.56` socks proxy then configured firefox to speak to port 1080 for socks. Then I can browse https://gitea99.opendev.org:3081/opendev/system-config | 17:20 |
| @clarkb:matrix.org | All of this is necessary now due to blocking direct access to the backends via firewall rules and anubis filtering allowed domains. Another approach would be to edit /etc/hosts to point at that IP address for opendev.org and i Think that would also work | 17:20 |
| @clarkb:matrix.org | anyway I don't see anything obviously wrong from this testing. But others should feel free to test too if they are interested | 17:21 |
| @clarkb:matrix.org | I also have an etherpad 2.7.2 held node I need to look at at some point | 17:21 |
| @clarkb:matrix.org | also looking at screenshots for Gerrit 3.13 on the change to update to 3.13.6 the config to disable the AI prompt button appears to be working | 17:23 |
| @clarkb:matrix.org | (Nothing against that idea in theory, but as implemented in 3.13 there is no configuration other than disable/enable and that was only added in the release today. Which means the prompt content etc is completely hardcoded and unlikely to be useful for us) | 17:24 |
| @clarkb:matrix.org | Supposedly 3.14 makes this much more configurable so we'll have to look at it more closely when we upgrade to that version | 17:24 |
| @mordred:waterwanders.com | Clark: dare I ask what the gerrit ui AI prompt button is supposed to do? | 18:13 |
| @clarkb:matrix.org | mordred: in 3.13 it emits a hardcoded prompt to copy and paste into your favorite chatbot to review the change | 18:18 |
| @clarkb:matrix.org | mordred: in 3.14 it has a similar purpose I think but is far more configurable. I know one of sean-k-mooney's concerns with the hardcoded version is that it only looks at the change diff whcih isn't enough context | 18:19 |
| @clarkb:matrix.org | mordred: you can see the upstream verison at https://gerrit-review.googlesource.com/c/gerrit/+/579641 its the Create Ai Review prompt button | 18:21 |
| @mordred:waterwanders.com | how fascinating | 18:26 |
| @mordred:waterwanders.com | Clark: I think I see how that's intended to be helpful, but it is clear to me that different people are doing things very differently in this space :) | 18:28 |
| @jim:acmegating.com | mordred: https://youtu.be/sZ9SmMjnH6k?si=iceT240-ultDJQ61&t=1012 and https://gerrit.googlesource.com/plugins/ai-code-review/ | 18:35 |
| but also, yes, all of those are a different approach compared to agents | ||
| @jim:acmegating.com | * mordred: https://www.youtube.com/watch?v=sZ9SmMjnH6k&t=1012s and https://gerrit.googlesource.com/plugins/ai-code-review/ | 18:36 |
| but also, yes, all of those are a different approach compared to agents | ||
| -@gerrit:opendev.org- Goutham Pacha Ravi proposed: [opendev/irc-meetings] 987415: Add schedule for the TC weekly meeting (2026.2) https://review.opendev.org/c/opendev/irc-meetings/+/987415 | 19:01 | |
| @clarkb:matrix.org | Gerrit 3.14 is expected next week | 20:49 |
| -@gerrit:opendev.org- Clark Boylan proposed: [opendev/zuul-providers] 987430: Collect disk usage info for dib data dirs https://review.opendev.org/c/opendev/zuul-providers/+/987430 | 20:55 | |
| @clarkb:matrix.org | corvus: ^ I pushed that to get a better sense of how much disk we can save with your idea of managing the format conversions outside of dib | 20:55 |
| @clarkb:matrix.org | and maybe we'll learn something useful that can be utilized via another approach? | 20:56 |
| @clarkb:matrix.org | 158.69.67.192 is the held etherpad 2.7.2 test instance. I've created a pad called clarkb-test on it and interacted with it from firefox and chrome. It seems to be working as expected. I did notice that when you create a new pad the initial text is now using the color of the pad creator (in the past it was rendered without attribution/color) | 21:29 |
| @clarkb:matrix.org | corvus: ok we're already clearing out the dib_cache that was one of the earlier chagnes you made. So its largely just the dib_tmp/dib-images content which as you mention is the raw + qcow2 + both vhd versions | 21:31 |
| @clarkb:matrix.org | corvus: looking at https://opendev.org/openstack/diskimage-builder/src/branch/master/diskimage_builder/lib/img-functions#L149-L151 we seem to copy the raw file to the intermediate path then we do the intermediate conversion in place? I wonder if we drop that cp and have the conversion from source to intermediate take the original as the source and output the new intermediate content if we might be more disk efficient that way? | 21:36 |
| @clarkb:matrix.org | I'm trying to reason about how ext4 might handle that. I think it is possible if we have two different file descriptors for the two files with the same path that the kernel/filesystem may be keeping logical copies while the process runs? So we'd have raw source copy, copy of raw source copy, output intermediate version. Whereas if we drop the copy we could possible only have 2? | 21:38 |
| -@gerrit:opendev.org- Clark Boylan proposed: [openstack/diskimage-builder] 987436: Avoid extra copies during VHD image conversion https://review.opendev.org/c/openstack/diskimage-builder/+/987436 | 21:44 | |
| -@gerrit:opendev.org- Clark Boylan proposed: [opendev/zuul-providers] 987430: Collect disk usage info for dib data dirs https://review.opendev.org/c/opendev/zuul-providers/+/987430 | 21:45 | |
| @clarkb:matrix.org | those two updates explore this idea | 21:46 |
| @clarkb:matrix.org | Looks like the vhd-util conversion may consume the source that is why it creates a .bak file? | 22:19 |
| @clarkb:matrix.org | What a weird way to do it | 22:19 |
| @clarkb:matrix.org | (So my change isn't working) | 22:19 |
| -@gerrit:opendev.org- Clark Boylan proposed: [opendev/zuul-providers] 987430: Collect disk usage info for dib data dirs https://review.opendev.org/c/opendev/zuul-providers/+/987430 | 22:28 | |
| @mordred:waterwanders.com | Clark: vhd-util is the gift that keeps on giving | 22:34 |
| @clarkb:matrix.org | https://zuul.opendev.org/t/opendev/build/5550e94221e543d9aa2f7f7dd33bfeca/log/job-output.txt#10180-10182 is interesting. I didn't expect that the raw and qcow2 would haev the same size. but I'm betting that is due to empty blocks not actually consuming disk space for those two but vhd does ? | 22:58 |
| @clarkb:matrix.org | doing some napkin math: we have a ~15GB raw source image + all the build artifacts and caches when we start the conversion process for vhd. So thats 15GB source, 15 GB source copy, and 24GB intermediate image for a total of ~54GB of image. Then after the intermediate image is produced we had 15 GB + 24GB for ~40GB total. Then we produce the final image which would be 15GB + 24GB + 24GB (as we end up with the .bak copy that we delete immediately afterwards) for ~63GB of disk | 23:03 |
| @clarkb:matrix.org | which is getting close to the limits of the rackspace ephemeral disk size | 23:04 |
| @clarkb:matrix.org | corvus: One thing I see is that we appear to only put /opt/dib_tmp/ on the ephemeral drive so all the caches and everything else are on / and aren't competing for space | 23:05 |
| @clarkb:matrix.org | though I think we're typically running out of disk during the next step after conversion where we zstd compress the images? | 23:07 |
| @clarkb:matrix.org | so maybe dib build raw image. Convert to vhd, compress vhd, upload vhd, delete vhd, convert to qcow2, compress qcow2 (maybe its already compressed), upload qcow2, delete qcow2, compress raw, upload raw really is a good approach here | 23:09 |
| @clarkb:matrix.org | essentially use the minimal amount of disk space for each format of the image and do them sequentially so that we're never doubling up | 23:09 |
| @clarkb:matrix.org | (to be clear this was corvus' idea from our meeting today, I'm just gathering data and trying to size things out at each step to see if that would help and I think it would) | 23:09 |
| @mordred:waterwanders.com | ++ that sounds like a good approach. only real downside is that you'd need to lift the conversions out of dib, which honestly isn't *that* terrible | 23:19 |
| @clarkb:matrix.org | yup exactly | 23:28 |
| -@gerrit:opendev.org- Zuul merged on behalf of James E. Blair https://matrix.to/#/@jim:acmegating.com: | 23:56 | |
| - [opendev/zuul-providers] 984866: Run vhd builds first and delete cache https://review.opendev.org/c/opendev/zuul-providers/+/984866 | ||
| - [opendev/zuul-providers] 982182: Add Ubuntu resolute image build job https://review.opendev.org/c/opendev/zuul-providers/+/982182 | ||
| @clarkb:matrix.org | woah they merged! | 23:56 |
| @clarkb:matrix.org | (we should still try to make things better overall but that is great news) | 23:56 |
Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!