| *** mrunge_ is now known as mrunge | 08:09 | |
| opendevreview | Benjamin Schanzel proposed zuul/zuul-jobs master: Add ensure-dib role https://review.opendev.org/c/zuul/zuul-jobs/+/922910 | 09:06 |
|---|---|---|
| opendevreview | Benjamin Schanzel proposed zuul/zuul-jobs master: Add build-diskimage role https://review.opendev.org/c/zuul/zuul-jobs/+/922911 | 09:06 |
| opendevreview | Benjamin Schanzel proposed zuul/zuul-jobs master: Add a role to convert diskimages between formats https://review.opendev.org/c/zuul/zuul-jobs/+/922912 | 09:06 |
| opendevreview | Benjamin Schanzel proposed zuul/zuul-jobs master: Add ensure-dib role https://review.opendev.org/c/zuul/zuul-jobs/+/922910 | 09:12 |
| opendevreview | Benjamin Schanzel proposed zuul/zuul-jobs master: Add build-diskimage role https://review.opendev.org/c/zuul/zuul-jobs/+/922911 | 09:12 |
| opendevreview | Benjamin Schanzel proposed zuul/zuul-jobs master: Add a role to convert diskimages between formats https://review.opendev.org/c/zuul/zuul-jobs/+/922912 | 09:12 |
| *** oschwart_ is now known as oschwart | 10:07 | |
| opendevreview | Benjamin Schanzel proposed zuul/zuul-jobs master: Add ensure-dib role https://review.opendev.org/c/zuul/zuul-jobs/+/922910 | 10:18 |
| opendevreview | Benjamin Schanzel proposed zuul/zuul-jobs master: Add build-diskimage role https://review.opendev.org/c/zuul/zuul-jobs/+/922911 | 10:18 |
| opendevreview | Benjamin Schanzel proposed zuul/zuul-jobs master: Add a role to convert diskimages between formats https://review.opendev.org/c/zuul/zuul-jobs/+/922912 | 10:18 |
| opendevreview | Benjamin Schanzel proposed zuul/zuul-jobs master: Add ensure-dib role https://review.opendev.org/c/zuul/zuul-jobs/+/922910 | 11:00 |
| opendevreview | Benjamin Schanzel proposed zuul/zuul-jobs master: Add build-diskimage role https://review.opendev.org/c/zuul/zuul-jobs/+/922911 | 11:00 |
| opendevreview | Benjamin Schanzel proposed zuul/zuul-jobs master: Add a role to convert diskimages between formats https://review.opendev.org/c/zuul/zuul-jobs/+/922912 | 11:00 |
| opendevreview | Benjamin Schanzel proposed zuul/zuul-jobs master: Add build-diskimage role https://review.opendev.org/c/zuul/zuul-jobs/+/922911 | 11:08 |
| opendevreview | Benjamin Schanzel proposed zuul/zuul-jobs master: Add a role to convert diskimages between formats https://review.opendev.org/c/zuul/zuul-jobs/+/922912 | 11:08 |
| opendevreview | Jan Gutter proposed zuul/zuul-jobs master: [wip] Fix k8s-crio buildset registry test https://review.opendev.org/c/zuul/zuul-jobs/+/926013 | 11:11 |
| opendevreview | Jan Gutter proposed zuul/zuul-jobs master: [wip] Fix k8s-crio buildset registry test https://review.opendev.org/c/zuul/zuul-jobs/+/926013 | 12:09 |
| opendevreview | James Page proposed openstack/project-config master: Prep sunbeam single charm repos for retirement https://review.opendev.org/c/openstack/project-config/+/917827 | 12:22 |
| opendevreview | Jan Gutter proposed zuul/zuul-jobs master: [wip] Fix k8s-crio buildset registry test https://review.opendev.org/c/zuul/zuul-jobs/+/926013 | 12:31 |
| opendevreview | Jan Gutter proposed zuul/zuul-jobs master: [wip] Fix k8s-crio buildset registry test https://review.opendev.org/c/zuul/zuul-jobs/+/926013 | 12:46 |
| rpittau | hi all! o/ out of curiosity how often the opendev repositories are mirrored to github? | 13:17 |
| fungi | rpittau: hard to say since different projects have their own mirroring jobs, we don't control that | 13:17 |
| opendevreview | Jan Gutter proposed zuul/zuul-jobs master: [wip] Fix k8s-crio buildset registry test https://review.opendev.org/c/zuul/zuul-jobs/+/926013 | 13:18 |
| rpittau | fungi: thanks, I was wondering as I removed an eoled branch from ironic today and it still appears in github | 13:18 |
| fungi | rpittau: branch removals may not trigger openstack's github mirroring job, have any changes merged to ironic since then? | 13:19 |
| rpittau | fungi: oh I see! no, I don't think so | 13:19 |
| rpittau | thanks, that makes sense actually | 13:19 |
| fungi | i think it's only set up to run on ref updates for branch tips and tag additions, just a sec and i'll check | 13:20 |
| rpittau | fungi: thank you :) | 13:20 |
| fungi | rpittau: but also, here's the build history for the openstack-upload-github-mirror job running on openstack/ironic: https://zuul.opendev.org/t/openstack/builds?job_name=openstack-upload-github-mirror&project=openstack/ironic | 13:21 |
| fungi | last run was 2024-08-08 17:16:52 | 13:21 |
| fungi | (utc) | 13:21 |
| rpittau | so yesterday | 13:21 |
| rpittau | well that explains it :) | 13:22 |
| rpittau | thanks a bunch fungi :) | 13:22 |
| fungi | and you can see it's triggered in the post and release pipelines, which matches my recollection | 13:22 |
| fungi | so the next time a change merges or a tag is created for any branch in ironic that should get refreshed | 13:23 |
| rpittau | yeah, I'll check again once we merge something | 13:23 |
| rpittau | mmm even a tag, ok, interesting, cause I created one | 13:24 |
| rpittau | but maybe it's only from the release pipeline | 13:24 |
| fungi | rpittau: yeah, sorry, versioned tags i mean. here's where it gets pipeline assignments: https://opendev.org/openstack/openstack-zuul-jobs/src/branch/master/zuul.d/project-templates.yaml#L3520-L3534 | 13:25 |
| rpittau | no worries, that makes sense | 13:26 |
| fungi | we could replace pre-release and release there with tag and then it would run on any tag regardless of whether it has a version pattern | 13:26 |
| fungi | also could try to figure out what kind of events gerrit emits on branch deletion and see if the job could be adapted to run for those as well, but they're relatively rare enough that openstack hasn't seen it as a priority | 13:27 |
| rpittau | that's ok, I was just puzzled as the branch was still there, but now I get it | 13:28 |
| rpittau | it's really not that big of a deal | 13:28 |
| fungi | but the job itself is essentially idempotent and relies on no real context other than the triggering project's git repository, so any event that causes the job to run performs a full git mirror push | 13:28 |
| fungi | we could even stick a periodic pipeline entry in that template just as a fallback in case projects have git state changes that didn't trigger replication for a day or more | 13:30 |
| rpittau | mmm that could be helpful I guess | 13:30 |
| fungi | since the job doesn't use any nodes at all (only runs on the zuul executors) and it's very quick (~30sec), running it more often probably isn't a huge deal | 13:31 |
| rpittau | yep execution looks pretty quick, maybe set it to run only if it didn't run in the last 24 hours, if possible | 13:33 |
| rpittau | or just run, if the repos are in sync it won't do anything anyway | 13:34 |
| fungi | from zuul's perspective, figuring out whether it ran is about the same amount of work as running it again. i suppose one concern might be additional impact on github rate limits/account quotas, not sure if pushes count toward those or not | 13:40 |
| clarkb | I think git pushes are completely separte from api rate limits | 15:36 |
| fungi | cool, so probably no impact from that | 15:40 |
| fungi | the only other down-side i can think of is that it will create additional periodic buildsets for any projects which don't currently run periodic jobs, so could be extra load on the zuul mergers when the timer trigger fires | 15:41 |
| fungi | oh, and for periodic we'd only want it to run once for a project, not once for each branch of the project, so might need an explicit branch matcher with that addition | 15:42 |
| clarkb | both gerrit and etherpad have new bugfix release in the last ~24 hours. Once I've caught up a bit I plan to push changes to update both of those versions in our systems | 15:43 |
| fungi | thanks! | 15:46 |
| opendevreview | James Page proposed openstack/project-config master: Prep sunbeam single charm repos for retirement https://review.opendev.org/c/openstack/project-config/+/917827 | 15:51 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Update Gerrit image to 3.9.6 https://review.opendev.org/c/opendev/system-config/+/926075 | 16:02 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Update etherpad to 2.2.2 https://review.opendev.org/c/opendev/system-config/+/926078 | 16:20 |
| clarkb | that ^ ended up being a bit more invovled than expected. They changed some of the build stuff so had to sync that into our dockerfile | 16:20 |
| opendevreview | James Page proposed openstack/project-config master: Prep sunbeam single charm repos for retirement https://review.opendev.org/c/openstack/project-config/+/917827 | 16:28 |
| opendevreview | Merged openstack/project-config master: Prep sunbeam single charm repos for retirement https://review.opendev.org/c/openstack/project-config/+/917827 | 16:56 |
| opendevreview | Clark Boylan proposed opendev/system-config master: DNM force etherpad failure to hold node https://review.opendev.org/c/opendev/system-config/+/840972 | 16:58 |
| clarkb | I've put a hold in place for ^ so we can double check the plugins work as called out in the commit message for the parent change | 16:59 |
| clarkb | other infra-root and opendev users may want to followup on https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/thread/CWVR5HRTHGWZTKQEFCOOP77GFFVUXBA4/ to see if there is consensus one way or another | 17:38 |
| clarkb | The gerrit update change passed testing. Remember that when we land that we will need to manually update the production deployment (by pulling the new image and then manually restarting things). | 17:40 |
| clarkb | I don't think we need to hold a node for that one if the screenshots and tests look good (tests did pass but screenshots are manual check) | 17:41 |
| clarkb | held etherpad 2.2.2 is at 104.130.140.173 and I'm jumping in to the clarkb-test pad to check on it if anyone else wants to look | 17:44 |
| clarkb | hrm I'm actually getting a perpetual loading splash screen | 17:45 |
| clarkb | so this may not be functional | 17:45 |
| clarkb | just in case the issue is due to cached js/css I've moved to incognito tabs but get the same result there. The server logs don't immediately show any errors either. (it does appear to minify things on demand the first time requiests are made but I don't think that would take more than 5 minutes. I will try refreshing in a bit to see if that assumption is wrong) | 17:47 |
| fungi | for the base url where you can create new pads, the button text is missing for me | 17:47 |
| fungi | and the buttons on that page are also nonfunctional | 17:48 |
| fungi | and yes, going to a pad url i just get perpetual "loading" too | 17:48 |
| fungi | in a browser that has never connected to the production site and keeps no cache | 17:49 |
| clarkb | looking at browser debug tools I think /padbootstrap-$hash.min.js is redirecting to /p/padbootstrap-$hash.min.js and that appears to be getting back html not js (maybe due to our proxy mod rewrite rules) | 17:51 |
| clarkb | I suspect that this may be contributing to the issue since the expected js is coming back as not js so it can't run the things | 17:51 |
| clarkb | oh yup the console log says there is a syntax error in that file so this is almost certainly the issue | 17:52 |
| clarkb | will need to figure out if this change in content serving location is expected and if so probably modify our apache mod rewrite rules | 17:52 |
| clarkb | I'm not sure I have time to dig into that right this moment (I have paperwork type things to get done this week and its Friday) | 17:53 |
| clarkb | I'll mark the change WIP though | 17:53 |
| clarkb | actually I think I see the issue maybe? I'll push an update and make a new hold | 17:57 |
| clarkb | we expected the js content to be at /static/ or similar and don't proxy that to /p/ but now the path is rooted at the base of the url so we redirect it to /p/ | 17:58 |
| clarkb | ok that got things into a slightly better state but now it is erroring in the js rather than blowing up because it got html intead of js. I'll still push this up as it illustrates the issue | 18:07 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Update etherpad to 2.2.2 https://review.opendev.org/c/opendev/system-config/+/926078 | 18:10 |
| opendevreview | Clark Boylan proposed opendev/system-config master: DNM force etherpad failure to hold node https://review.opendev.org/c/opendev/system-config/+/840972 | 18:10 |
| clarkb | ok hold is rotated so now you should get tcp connection errors | 18:12 |
| fungi | looks like 104.130.172.149 is the new hold | 20:14 |
| fungi | TypeError: Co is not a function | 20:15 |
| clarkb | ya thats what I got on the previous hold when testing the updates. Not sure if that is a proper bug in etherpad or another installation specific thing | 20:16 |
| clarkb | I think it is possible that some js isn;t loading beacuse of our installation, but it could just as easily be a proper bug | 20:16 |
| fungi | the apache error log has a couple of entries for my client, not sure whether they're benign... | 20:22 |
| fungi | [Fri Aug 09 20:16:31.650412 2024] [proxy:error] [pid 17009:tid 140141150594624] (70014)End of file found: [client 47.133.137.230:60809] AH03307: ap_proxy_transfer_between_connections: error on origin - ap_pass_brigade | 20:22 |
| fungi | [Fri Aug 09 20:16:31.650476 2024] [proxy:error] [pid 17009:tid 140141150594624] [client 47.133.137.230:60809] AH10221: proxy: websocket: client flushing failed (-102) | 20:22 |
| fungi | could also be secondary errors from my session never fully establishing | 20:23 |
| clarkb | ya | 20:23 |
| fungi | doesn't look like my browser requested any js files which were refused or nonexistent | 20:25 |
| clarkb | maybe we should check the etherpad issue tracker to see if anyone else has hit this error | 20:27 |
| clarkb | I'm not finding any evidence for similar issues already reported | 21:27 |
| clarkb | unfortunately I suspect that Co is a minified variable name | 21:28 |
| clarkb | fungi: looks like the landing page loads now at least | 21:29 |
| clarkb | fungi: var Co, Nc, Nc = Fo().$, Co = Ts(), P4 = ["ep_headings/static/css/editor.css"], js = ["h1", "h2", "h3", "h4", "h5", "h6", "code"], M4 = function() { is where Co gets defined. Note the ep_headings css stuff I think maybe this is exploding in the plugin ep_headings that we install | 21:35 |
| fungi | yeah, the base page and pad creation form seemed to work | 21:36 |
| clarkb | and then that code is being appended to the single min.js file which makes it extra confusing to trace back again | 21:36 |
| clarkb | we could confirm this by pushing a change up to disable/remove that plugin for now | 21:36 |
| fungi | or maybe just manually disable it on the held node? | 21:37 |
| clarkb | I think they may get built into the image, but maybe that is doable | 21:37 |
| fungi | oh | 21:37 |
| clarkb | ya we run a pnpm command to install the plugins list | 21:38 |
| clarkb | I'm not sure we can easily untangle that on the running image. It is probably doable but figuring out the method for how is the trick | 21:38 |
| clarkb | https://github.com/fourplusone/etherpad-plugins/blob/master/ep_headings/static/js/index.js#L85-L89 I believe this is the code that is failing | 21:39 |
| fungi | makes sense that it could be a plugin problem if those plugins don't get much testing with new releases of the base service | 21:40 |
| clarkb | and Co is _ there. Ironic that minification made that bigger | 21:40 |
| clarkb | var _ = require('ep_etherpad-lite/static/js/underscore'); | 21:41 |
| clarkb | I suspect that dropping require() in the main etherpad codebase means that all plugins need to drop it too | 21:41 |
| clarkb | and require() there is assigning something useless/null to _ and we get this error | 21:41 |
| clarkb | I'll file a bug with that information and see if upstream can give us anymore hints | 21:42 |
| clarkb | hrm that plugin's repo is archived too... I haev a feeling this will end up in a state of "sorry no longer supported use a different plugin if something exists" | 21:44 |
| clarkb | https://github.com/ether/ep_headings2 does exist. No idea if it is compatible with the other plugin. I'm going to push a new pair of patchsets that use that plugin to see if it works. Better to file a bug against it if it doesn't | 21:46 |
| clarkb | and then if it does work we can figure out if the old markup is compatibile with this plugin too | 21:46 |
| clarkb | https://github.com/ether/ep_headings2/issues/4 I am not hopeful :/ | 21:52 |
| fungi | eep | 21:53 |
| clarkb | but one step at a time. Lets see if this is even workable with the new plugin | 21:53 |
| clarkb | if we simply lose the old rendering but keep the markup that is probably workable but that issue implies there are errors instead | 21:55 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Update etherpad to 2.2.2 https://review.opendev.org/c/opendev/system-config/+/926078 | 21:55 |
| opendevreview | Clark Boylan proposed opendev/system-config master: DNM force etherpad failure to hold node https://review.opendev.org/c/opendev/system-config/+/840972 | 21:55 |
| clarkb | I'm leaving the old autohold in place for now as it should haev different errors which may be useful depending on how further debugging goes | 21:56 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!