| mnasiadka | fungi: accurate - both Kolla-Ansible and OpenStack-Ansible, it’s a lot of small html files in Kolla it’s around 150-200 megabytes of HTML... | 06:01 |
|---|---|---|
| jrosser | mnasiadka: https://github.com/jrosser/ara/commit/f9af69eaef4ea1228f4fc641e36b9d8df5adbaaa | 08:01 |
| jrosser | from ages ago this was | 08:01 |
| mnasiadka | jrosser: yeah, but the concern here is ARA generates HTML files and does not delete them automatically - so the storage bloat is growing very fast | 08:09 |
| jrosser | right - but that patch to ara i made deals with the ara db and renders the html | 08:58 |
| jrosser | so there was no need to keep the html files in the CI job, you could retrieve it as required | 08:58 |
| jrosser | mnasiadka: ^ | 08:58 |
| mnasiadka | Well, you can keep sqlite in the job, that’s my plan - just upload sqlite to job logs and add a readme for users how to use it (run ara-server locally using that sqlite file) | 09:00 |
| jrosser | yes thats exactly what it did | 09:00 |
| jrosser | it was a seperate service, that given the build id, could retrieve the db from the job logs then render the html on demand | 09:00 |
| corvus | deleting the files shouldn't be a concern: all of the build logs opendev uploads are deleted automatically after 30 days. | 13:37 |
| fungi | by setting object expirations when uploading to swift | 13:40 |
| clarkb | corvus: the zuul launcher fix landed yesterday. Should I plan to restart the launchers to pick that up or do you want ot do it? | 13:56 |
| clarkb | I've got a PTG session to join in a few minutes but can do it after | 13:56 |
| corvus | i'll do it a little later this morning | 13:57 |
| mnasiadka | That would be great, I got a bit bored by doing rechecks for node failures ;-) | 13:59 |
| cardoe | random question about meetpad... are we not able to replace the default etherpad for the meeting with another etherpad? Like if we had one before the meetpad was creaetd | 14:02 |
| fungi | cardoe: yes, you set the meetpad room name to be the same as the pad name | 14:03 |
| fungi | #track url https://meetpad.opendev.org/whatever_your_pad_name_is | 14:04 |
| fungi | the meetpad rooms are plumbed directly to identically-named pads in etherpad | 14:04 |
| cardoe | ah okay so in the future I'll do that for Ironic. | 14:04 |
| cardoe | Cause we basically have a URL to another etherpad which is our PTG notes. | 14:05 |
| fungi | just make sure to use all lower-case for the pad name, because jitsi-meet room names are case-insensitive but etherpad has case-sensitive pad names configured (for historical reasons) | 14:05 |
| fungi | cardoe: right, if you change the pad name in ptgbot then always make sure to change the room name to match | 14:05 |
| clarkb | as an alternative you can just use the auto populated etherpad when initially set up (But I know some projects start creating notes before that is ready | 14:06 |
| cardoe | yeah that's what happened with Ironic | 14:24 |
| frickler | if only there was a documented scheme that would allow to predict these names ... not a new issue really | 14:45 |
| clarkb | I think there is | 14:46 |
| clarkb | I think the main issue is the team names "changing" | 14:47 |
| clarkb | whcih for a team like ironic isn't a problem | 14:47 |
| clarkb | $month$year-ptg-$team. All lower case | 14:48 |
| clarkb | also you can just copy paste teh data between pads when the actual pad is known. You'll lose authorship but typically before the event that is less of an issue (because topics are already annotated with authors) | 14:49 |
| clarkb | infra-root I expect to be around all day today if we want to upgrade gitea https://review.opendev.org/c/opendev/system-config/+/964899 | 14:51 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Upgrade etherpad to 2.5.1 https://review.opendev.org/c/opendev/system-config/+/956593 | 15:19 |
| opendevreview | Clark Boylan proposed opendev/system-config master: DNM force etherpad failure to hold node https://review.opendev.org/c/opendev/system-config/+/840972 | 15:19 |
| clarkb | the 2.5.0 to 2.5.1 update looks straightforward from our side as Dockerfile and settings files don't change between those versions so I went ahead and updated the proposed change and put a hold in place. I've also requested holds for the gerrit jobs and rechecked the gerrit testing change | 15:20 |
| clarkb | I'm going to step out and do morning things now that I'm acught up. infra-root if you have time to check on https://review.opendev.org/c/opendev/system-config/+/964899 I'm happy to babysit that today | 15:41 |
| corvus | #status log restarted zuul launchers to pick up node reassignment fix | 16:05 |
| opendevstatus | corvus: finished logging | 16:05 |
| corvus | infra-root: i'm seeing a few errors like this in the launcher logs: https://paste.opendev.org/show/bDmip1DwP60fgOZ8pqQf/ | 16:18 |
| corvus | i think we have 2 or 3 instances that we can't delete because of that | 16:18 |
| corvus | i'm manually stepping through the api calls, and this is the list of fips we get back for that server: | 16:18 |
| corvus | [{'version': 4, 'addr': '174.143.59.44', 'OS-EXT-IPS:type': 'floating', 'OS-EXT-IPS-MAC:mac_addr': 'fa:16:3e:b4:5c:6f'}] | 16:18 |
| corvus | i think the problem is that the same fip is assigned to two servers: | 16:19 |
| corvus | | b0b14af7-e256-4f70-829f-2abc13f364d2 | np8b28eed0e1384 | ACTIVE | opendevzuul-network1=10.0.19.4, 174.143.59.44 | centos-10-stream-6b04a7a00e0641a98362200fc1644b24 | gp.5.4.8 | | 16:19 |
| corvus | | 620042e4-4a3e-4db4-8d26-a20fefcbfcb5 | np3eca225498334 | ACTIVE | opendevzuul-network1=10.0.17.200, 174.143.59.44 | debian-bookworm-bdfd61a77dd447c59be890d1244ed9e6 | gp.5.4.8 | | 16:19 |
| corvus | that seems like a potential openstack problem? | 16:20 |
| fungi | this is in rackspace flex i guess, based on the flavor name? | 16:26 |
| corvus | yes dfw3 | 16:26 |
| Clark[m] | You may be able to manually remove the fip from the unused server as a workaround, but yes Openstack shouldn't allow that to happen imo since ipv4 doesn't work that way | 16:27 |
| corvus | is this something that should be escalated to the flex folks? | 16:28 |
| Clark[m] | ++ at least to make sure they are aware | 16:28 |
| corvus | any chance you or fungi want to do that? i'd like to focus on why the launcher thinks there are artifacts missing | 16:29 |
| clarkb | sure the pasted lines above are cloud uuid, hostname, status, network, image name, flavor? | 16:31 |
| clarkb | just want to make sure the data above is interpreted correctly when I share it | 16:31 |
| corvus | yeah that's openstack server list | 16:31 |
| clarkb | perfect thanks | 16:32 |
| fungi | i'm in another ptg session right now, but can put it on my list for after lunch this afternoon if clarkb doesn't get to it | 16:32 |
| corvus | | ID | Name | Status | Networks | Image | Flavor | | 16:32 |
| clarkb | dan_with: dan_with_ ^ is that something I can report to you here or would it be better to email cloudnull and james denton? | 16:32 |
| corvus | convenient paste: https://paste.opendev.org/show/bTgyg6Y7xXQTxWe0UVpN/ | 16:33 |
| corvus | i think there's at least one other, lmk if i should run that down | 16:33 |
| clarkb | I suspect that having a single example is probably sufficient for them to run it down | 16:34 |
| clarkb | I'll work on an email now and not send it if dan_with is able to discuss here otherwise I'll get it sent soon | 16:34 |
| clarkb | email sent | 16:46 |
| fungi | thanks! | 16:54 |
| tonyb | cardoe: If you're creatign the pad ahead of time the "default" name follows a std. format you could do likewise. but also you can update the url as pointed out | 16:57 |
| tonyb | cardoe: You can even do so now to impact sessions for ironic for the rest of the ptg if that isn't too confusing | 16:58 |
| fungi | i'm on my way out to lunch, back soon | 16:58 |
| cardoe | Yeah I'll do that for tomorrow. | 17:00 |
| tonyb | clarkb: for the record I'm happy for you to do gitea whenever it works for you. | 17:02 |
| clarkb | tonyb: thanks, you're the only reviewer so far and was hoping soneone else would sanity check since its gitea, but if not I can probably proceed in a bit | 17:09 |
| tonyb | clarkb: that's fair. | 17:21 |
| clarkb | gitea 1.25.0 just released less than an hour ago | 17:41 |
| clarkb | usually we upgrade from the last point release of the prior release to the latest point release of the next "major" release nyway so that doesn't change any plans for us | 17:42 |
| tonyb | okay | 17:43 |
| clarkb | https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_189/openstack/18971ee2a1364d39b1c0f9963d840141/bridge99.opendev.org/screenshots/etherpad-main.png etherpad 2.5.1 css still looks meh on the main page | 17:45 |
| tonyb | I'm planning to look at etherpad today | 17:46 |
| tonyb | that doesn't look so different from the existing install | 17:47 |
| clarkb | tonyb: ya its minor around the open pad by name button | 17:48 |
| clarkb | I'm filing a followup issue now | 17:48 |
| tonyb | ahh I see | 17:51 |
| clarkb | interestingly the problem doesn't seem to exist in chrom* | 17:54 |
| tonyb | hmmmm I'm on my phone so I can't check the held node | 17:56 |
| tonyb | when I'm at my desk I will | 17:56 |
| clarkb | https://github.com/ether/etherpad-lite/issues/7202 is the new issue | 17:58 |
| clarkb | considering it seems firefox specific I'm prbably a bit more willing to live with it | 18:01 |
| clarkb | note I haven't tested the other functionality under 2.5.1 yet | 18:01 |
| clarkb | (but it passed our CI tests which is a good sign) | 18:01 |
| clarkb | fungi: any interest in checking the gitea upgrade change or do you think I should just go for it? | 18:06 |
| fungi | i'm back, can take a look at the gitea upgrade in a minute | 18:10 |
| stephenfin | clarkb: fungi: If any of you have time in the next few days, I'd appreciate thoughts on https://review.opendev.org/c/openstack/pbr/+/965315/ | 18:29 |
| clarkb | stephenfin: the commit message says enable build isolation but the commit seems to disable it? I guess I'm lacking context for what the goal is there | 18:30 |
| clarkb | maybe its the parent which rewords tests that I should be looking at first | 18:31 |
| stephenfin | Sorry, it should read "disable". And yes, it only makes sense with the parent commit | 18:31 |
| clarkb | ack | 18:31 |
| stephenfin | For some reason, disabling isolation breaks packages building because pbr isn't built correctly (I see versions of 0.0.0 for everything). But it should work, because I'm building the sdists, dumping them to a temporary directory (dist_dir), and telling pip where to find them (and to skip pypi) with the '-d <dir> --no-index' options | 18:31 |
| stephenfin | and pbr itself doesn't get installed in the environment, despite being a runtime requirements of pkgReq package, which suggests much of the pbr machinery is broken. It's all very confusing | 18:32 |
| clarkb | the fix I pushed after the pip update was to not use isolation though so it does work in some contexts I guess? | 18:39 |
| clarkb | as for verison 0.0.0 I think thati sb ecause the packages we're building are not in git contexts | 18:39 |
| clarkb | the version info comes from git but we aren't faking that out in the test fixtures so it must default to that or maybe we're using the env var override | 18:39 |
| fungi | stephenfin: my first guess is thet build isolation isn't copying the whole git repository, so pbr can't figure out its version. this has happened in the past | 18:44 |
| corvus | not super urgent, but fyi, i noticed that the launchers have some "upload" records in ZK with no associated "image build artifacts". these are accumulating, and there's some associated logspam. there is no cleanup path for this since it's not supposed to happen. i think i identified the bug and fixed it in https://review.opendev.org/965318 | 18:44 |
| corvus | if that gets merged before the weekend, then i can manually clean up the znodes next week. | 18:45 |
| stephenfin | clarkb: hmm, the env var point is a good one | 18:45 |
| * stephenfin looks | 18:45 | |
| stephenfin | fungi: Sorry, with build isolation is works just fine. It's when I disable it that things break | 18:46 |
| fungi | oh, odd... | 18:46 |
| stephenfin | so at risk of repeating myself, how I *think* that test is working is that we build a venv with minimal dependencies (pip, wheel, build), ... | 18:47 |
| stephenfin | then we build sdists for pbr and our _dependency_ package (which depends on pip) and dump them to 'dist' | 18:48 |
| stephenfin | then we install our _test_ package (which depends on the dependency package) using the sdists we just built | 18:49 |
| fungi | clarkb: +2 on the gitea 1.24.7 upgrade change, but didn't approve in case you want to more tightly control the deployment window | 18:49 |
| stephenfin | and I'd expect pip to install both pbr (to satisfy the dependency package) and the dependency package (to satisfy the test package) from those sdists | 18:50 |
| stephenfin | but obviously not. A problem to pick up again tomorrow I think | 18:50 |
| clarkb | fungi: nope I think now is good. I'l lapprove it | 18:53 |
| clarkb | I should be able to eat lunch while that runs through the gate | 19:02 |
| clarkb | stephenfin: thinking out loud here: if it is helpful we can hold pbr test nodes too so that it is easier to inspect the test cases. I think these specific test cases are the integration ones which are a bit harder to run locally | 19:02 |
| opendevreview | Nicolas Hicher proposed zuul/zuul-jobs master: Refactor: multi-node-bridge to use linux bridge https://review.opendev.org/c/zuul/zuul-jobs/+/959393 | 19:41 |
| opendevreview | Merged opendev/system-config master: Upgrade gitea to 1.24.7 https://review.opendev.org/c/opendev/system-config/+/964899 | 19:49 |
| clarkb | that was quick. I'll keep an eye on it | 19:49 |
| clarkb | https://gitea09.opendev.org:3081/opendev/system-config/ is updated. Testing cloen now | 19:54 |
| clarkb | clone of system-config seems to work | 19:55 |
| clarkb | all six backends are upgraded | 20:08 |
| clarkb | https://zuul.opendev.org/t/openstack/buildset/924471729ffe4d75ab86ba50add83aa7 and deployment buildset is a success | 20:09 |
| clarkb | now I just need someone to push code to double check replication | 20:09 |
| corvus | i just pushed https://review.opendev.org/965237 ps2 | 20:15 |
| clarkb | thanks https://opendev.org/zuul/zuul/commit/fca71a069aba5bf309689d58d91ddd8b65d73fa5 is there | 20:15 |
| clarkb | I think the upgrade lgtm. | 20:16 |
| clarkb | I won't look at 1.25.0 just yet. Happy for someone else to do so if interested otherwise I can probably start on that tomorrow | 20:16 |
| clarkb | but now I want to review that mediawiki change | 20:16 |
| clarkb | tonyb: https://review.opendev.org/c/opendev/system-config/+/921321 lgtm for the most part but there is one little image publication accounting thing that needs to be added I think. Then it also needs a rebase due to a merge conflict so two birds one stone there hopefully | 20:27 |
| clarkb | but with those addressed I agree we shoud be able to start publishing the image | 20:27 |
| clarkb | tonyb: oh I guess the other thing to consider is do we want to publish to quay.io which is slightly different job setup | 20:27 |
| clarkb | if we are going to deploy the new server on noble we should but not if jammy | 20:28 |
| clarkb | I'll leave another comment | 20:28 |
| fungi | seems like deploying on jammy is going to be more work in the long term | 20:30 |
| clarkb | ya thats what I'm thinking, if we skip these extra steps and go directly to noble that is probably best | 20:31 |
| fungi | unless we can't run the container for the old mw on noble | 20:31 |
| clarkb | I doubt that is an issue (cases where that happens are usually the other way around: your kernel is too old to support the software in the container iirc) | 20:31 |
| fungi | right, seems unlikely | 20:32 |
| clarkb | Going to do a school run, back in a bit | 20:57 |
| clarkb | tonyb: I think Ive' gotten through the ansible ara testing stack now as well. let me know if there are qusetions or concerns | 21:50 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Add trixie mirror config to reprepro https://review.opendev.org/c/opendev/system-config/+/965334 | 21:57 |
| clarkb | I decided to go ahead and push that up ^. Looking in grafana we should have both plenty of space in afs as well as the debian volume quota | 21:57 |
| clarkb | that said I don't have an exact size figure for trixie. I just figure it should be about half of the total consumption for bookworm and bullseye | 21:58 |
| clarkb | which is about 260gb | 21:58 |
| fungi | noonedeadpunk: ^ heads up in case you'd also started working on that | 22:02 |
| *** dhill is now known as Guest30146 | 22:08 | |
Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!
