| cardoe | Sorry. Kid afterschool stuff pulled me away. | 01:20 |
|---|---|---|
| fungi | don't apologize, we operate very asynchronously | 02:00 |
| fungi | i'm going ahead and approving 926078 so that it will hopefully be deploying early enough in the day that we can get confirmation of production functionality out of the way | 13:32 |
| fungi | this is taking longer than anticipated... it's been an hour and system-config-run-etherpad is only just now getting a node request | 14:28 |
| fungi | it has finally started running | 14:34 |
| fungi | estimated 15 minutes to merge so deploy should hopefully beat the hourly jobs | 14:35 |
| opendevreview | Merged opendev/system-config master: Update etherpad to 2.2.4 https://review.opendev.org/c/opendev/system-config/+/926078 | 14:52 |
| clarkb | I have arrived just in time | 14:53 |
| fungi | infra-prod-service-etherpad is running already | 14:55 |
| clarkb | ya the container hs restarted too | 14:56 |
| clarkb | it says it is healthy | 14:57 |
| fungi | loading existing pads seems to be fine, with headings too | 14:57 |
| clarkb | same for me | 14:57 |
| clarkb | I'm going to try loading them in an incognito tab just to rule out caching making things work | 14:57 |
| clarkb | that still seems happy so this is looking good | 14:58 |
| fungi | and on that note, i need to disappear for some quick lunch/errands, but can help test meetpad when i get back in an hour or so | 14:58 |
| clarkb | thanks | 15:00 |
| clarkb | fwiw etherpad introduced a new local db type in addition to its "dirtydb" local type in 2.2.3. But we override that explicitly in settings to use mysql/mariadb so should be unaffected | 15:00 |
| clarkb | the fact that we can still load old dbs implies we haven't gotten the db settings wrong | 15:00 |
| clarkb | *still load old etherpads implies we are using the old db | 15:01 |
| corvus | clarkb: have a test pad? | 15:02 |
| clarkb | corvus: https://etherpad.opendev.org/p/gerrit-upgrade-3.7 is the one I was looking at since it has a bunch of headers in it but maybe best to not edit that one | 15:03 |
| clarkb | corvus: https://etherpad.opendev.org/p/isitbroken is one that we can type in | 15:05 |
| clarkb | that seems to be working for me between firefox and chrome | 15:08 |
| corvus | lgtm too | 15:08 |
| clarkb | we did break meetpad fwiw. And it is almost certainly related to the rewrite rule adjustments | 15:54 |
| clarkb | I'm guessing we need to add similar pass throughs for these things in the meetpad nginx config | 15:54 |
| clarkb | hrm though I see a successful response for padboostrap-*.min.js in my browser debug tooling so maybe that isn't the issue | 15:56 |
| clarkb | would be annoying if the 2.2.2 updates for code loading are just incompatible currently | 15:56 |
| clarkb | the console log shows the error occurs in padBoostrap trying to read an undefined property of skinName | 15:57 |
| clarkb | firefox has a slightly different error and says parent.parent.clientVars is undefined | 16:06 |
| corvus | clarkb: fungi i feel like maybe we didn't sufficiently discuss "system-config" as a potential home for the image build jobs... pros: it's an untrusted repo that deals with how opendev runs services; cons: it's a big repo that's *mostly* focused on ansible-based system operations. | 16:07 |
| corvus | but i made my original test change in that repo; so that's a rough sketch of what it could look like: https://review.opendev.org/848792 | 16:10 |
| corvus | would probably call that top level directory something like "zuul-images" instead of "nodepool" | 16:10 |
| clarkb | ya I guess taht would work. Might be a little awkward due to being in the openstack tenant and having other responsibilities if we want to load the configs into other tenants later (so they get access to the images?) | 16:10 |
| corvus | oh ha i forgot about that | 16:11 |
| corvus | i have a strong preference for doing this in the opendev tenant, so i'll stick with the new-repo plan :) | 16:12 |
| fungi | okay, back | 16:13 |
| opendevreview | James E. Blair proposed openstack/project-config master: Add opendev/zuul-jobs repo https://review.opendev.org/c/openstack/project-config/+/928945 | 16:14 |
| opendevreview | James E. Blair proposed openstack/project-config master: Add opendev/zuul-jobs to Zuul https://review.opendev.org/c/openstack/project-config/+/928946 | 16:14 |
| fungi | and yeah, i would have made the same point about opendev/system-config still being in the openstack tenant | 16:14 |
| corvus | (my strong preference comes from the new pipelines we're going to be experimenting with; we may end up adding them to openstack eventually, but that's not certain now, so i'd like them to be in opendev) | 16:15 |
| corvus | we can always move em later | 16:15 |
| clarkb | https://opendev.org/opendev/system-config/src/branch/master/playbooks/roles/jitsi-meet/files/meet.conf#L104-L117 this is the code block for proxying /etherpad/ to https://etherpad.opendev.org in meetpad's nginx conf | 16:16 |
| clarkb | and looking in developer tooling there is no obvious failure to load files | 16:16 |
| clarkb | but there does seem to be missing content (particularly settings I guess?) in the running js | 16:17 |
| clarkb | but things like locales.json are being loaded just fine | 16:17 |
| clarkb | its almost like it thinks it has loaded everything then it starts to run the js and explodes. But all of the previous requests 200 and appear to return the rough type of data that I woudl expect. I guess I can compare this set of requests to what a normal etherpad with jitsi meet looks like | 16:21 |
| fungi | thanks, i was about to ask if anyone was already working on the path changes for meetpad | 16:23 |
| clarkb | fungi: well I'm not sure yet that the path changes are needed since we basically send everything under /etherpad/ to the etherpad server | 16:24 |
| clarkb | comparing network traces between etherpad alone and meetpad I see etherpad alone requests and receives a manifest.json but I don't see that in meetpad at all | 16:24 |
| fungi | possible something in etherpad itself has regressed to using a full/explicit path instead of relative | 16:25 |
| clarkb | looks like meetpad does serve a manifest.json for jitsimeet. etherpad's manifest.json is actually html.... so maybe those two things are getting confused? | 16:26 |
| fungi | oh, one shadowing the other? | 16:27 |
| clarkb | however I don't see a request to /etherpad/p/manifest.json at all under meetpad but etherpad alone does request /p/manifest.json | 16:27 |
| clarkb | ok here's somethign that may actually be useful. If I open the wss messages pane for our websocket comms under etherpad alone i see messages for client vars and skinname | 16:28 |
| clarkb | so this is probably a websocket issue if we're not getting that data back under meetpad? | 16:28 |
| clarkb | hrm meetpad shows those messages too though | 16:29 |
| clarkb | according to the console in chrome the initial failure occurs during init, but then there are subsequent errors of the same "TypeError: Cannot read properties of undefined (reading 'skinName')" in handle message from server | 16:35 |
| clarkb | so ya I think this must be some sort of websocket read failure | 16:35 |
| clarkb | maybe not of the socket itself but at least of the data coming out of the socket | 16:36 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Don't redirect etherpad manifest.json https://review.opendev.org/c/opendev/system-config/+/928951 | 16:41 |
| clarkb | at the very least I think ^ should be fixed | 16:41 |
| clarkb | I don't know if that will fix this issue though | 16:41 |
| clarkb | but I'm fairly stumped otherwise without a better udnerstanding of etherpad js and how everythign is tied together with the data coming over the websocket | 16:42 |
| fungi | yeah, let's see what happens after that deploys | 16:43 |
| fungi | hashar: i'm curious, any idea if the machine learning work mentioned in https://x.com/chrisalbon/status/1833873528757903434 is running on openstack? | 16:43 |
| hashar | oh | 16:44 |
| fungi | if so, i know people who would love to help you promote that | 16:44 |
| hashar | well that is how I learn our ML team is doubling size :] | 16:44 |
| fungi | hah! okay | 16:45 |
| hashar | I think they run on a dedicated Kubernetes cluster built on top of ML dedicated baremetal | 16:45 |
| hashar | I imagine cause they need some GPUs | 16:45 |
| fungi | fair enough! | 16:45 |
| fungi | openstack does have gpu management capabilities, but i can understand just wanting something dedicated | 16:45 |
| hashar | they are on Libera.chat in #wikimedia-ml if you want to ask. Chris Albon is the team director | 16:46 |
| fungi | thanks! | 16:46 |
| hashar | we don't use OpenStack in production. It was baremetal and Ganeti for VMs | 16:47 |
| hashar | we now have some kubernetes cluster which afaik is built on top of baremetal | 16:47 |
| hashar | but we do have a cloud offering | 16:47 |
| fungi | neat, thanks for the details | 16:47 |
| fungi | as one of the directors for spi, i similarly like to see when spi associated projects are relied on in production ;) | 16:50 |
| fungi | it's not often i come across anyone mentioning ganeti these days | 16:51 |
| hashar | I think the idea was that for our production use case, OpenStack was largely overkill | 16:52 |
| fungi | particularly if you didn't need multi-tenancy, yes | 16:52 |
| hashar | but the cloud we offer to our volunteers is powered by openstack with some kubernetes built on top of it | 16:53 |
| fungi | awesome! | 16:53 |
| hashar | https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS | 16:53 |
| hashar | so yeah there is a whole ecosystem built on top of OpenStack :] | 16:54 |
| fungi | i'll pass that along. i saw someone asking about the wmf ai/ml activity mostly because there's been a marketing push lately about the various organizations who are doing that sort of stuff on openstack, but i'm always happy to know about any uses of it really | 16:56 |
| hashar | well they can reach out to Chris Albon directly, he will know for sure | 16:56 |
| hashar | better reach out to the canonical source than relaying on my hearsays | 16:57 |
| hashar | afaik the very first thing he started when joining was to create a Kubernetes cluster. | 16:57 |
| fungi | yeah, i mostly just abuse your presence here because i like to have opportunities to talk to you ;) | 16:58 |
| * hashar blushes | 16:59 | |
| hashar | else for our cloud offer the team has communication channels at https://wikitech.wikimedia.org/wiki/Help:Cloud_Services_introduction#Communication_and_support | 16:59 |
| hashar | I am not sure who would be the best person to talk to | 17:00 |
| hashar | but for sure bd808 (here in this channel) knows all about it since AFAIK he did a lot of product management / definition | 17:00 |
| fungi | thanks! | 17:01 |
| hashar | and please ping me anytime, I am happy to chat :-] | 17:01 |
| fungi | same, of course ;) | 17:02 |
| JayF | hashar: if there's ever anything we can do to help you all get those BM pieces running on Ironic (should you want to move your BM into OpenStack, too), feel free to reach out. (OpenStack+)Ironic with k8s on top is an extremely common setup. | 17:06 |
| hashar | JayF: I don't know much about how SRE manages our machines, but we have a couple decades of experience doing that and I imagine they would be reluctant to add an abstraction layer | 17:09 |
| hashar | they picked up Ganeti cause it was good enough for our use case | 17:10 |
| JayF | Ironic was mostly written by SRE-types for SRE-types fwiw; not trying to sell you a thing just telling you we're here and actively willing to help if you ever wanna :) | 17:10 |
| hashar | sure thing :] | 17:11 |
| JayF | we always say you can come chat with us just if you wanna complain about horrible hardware with us :D | 17:11 |
| hashar | then I am not involved in the SRE team or the baremetal layers, so I can't tell much about it | 17:11 |
| opendevreview | Merged opendev/system-config master: Don't redirect etherpad manifest.json https://review.opendev.org/c/opendev/system-config/+/928951 | 17:31 |
| clarkb | as expected ^ didn't fix things but did fix serving of that file | 17:43 |
| clarkb | this is fun the chat messaging functionality works but not the pad content | 17:45 |
| clarkb | chat messaging uses websocket traffic too so the websocket is working at least minimally (which seemed previously confirmed by the fact that the messages are in the developer tool log) | 17:45 |
| opendevreview | James E. Blair proposed openstack/project-config master: Add opendev/zuul-jobs repo https://review.opendev.org/c/openstack/project-config/+/928945 | 17:48 |
| opendevreview | James E. Blair proposed openstack/project-config master: Add opendev/zuul-jobs to Zuul https://review.opendev.org/c/openstack/project-config/+/928946 | 17:49 |
| clarkb | looks like initializing the editorcontainer is where things break so what may explain why chat works but not the editable pad area | 18:00 |
| clarkb | and then subsequent message handling fails because the editorcontainer isn't initialized? | 18:00 |
| clarkb | https://github.com/ether/etherpad-lite/blob/v2.2.4/src/static/js/ace2_inner.ts#L242-L243 I suspect this is the code that is exploding though am not 100% positive of that (I'm inferring it based on the two different errors from ff and chrome involving parent.parent.clientVars and skinName) | 18:06 |
| clarkb | that code is nwe from august 17 | 18:06 |
| clarkb | oh yup I think I figured out how to confirm that and yes that appears to tbe the line | 18:08 |
| clarkb | the system knows what the skin name is because it is requesting the no-skin/pad.css file | 18:17 |
| clarkb | https://github.com/ether/etherpad-lite/issues/6618 this is it | 18:31 |
| clarkb | unfortunately instead of linking to the fix on the develop branch they've just suggested we deploy the develop branch.... | 18:31 |
| clarkb | https://github.com/ether/etherpad-lite/commit/a61f634586017dcadffd859820b66cd5916cef3a | 18:32 |
| clarkb | the issue is that `parent` is the meetpad window I think and so they move to `window` to select the inner iframe? | 18:32 |
| clarkb | do we want to cherry pick that fix on top of v2.2.4 and see if it works, or deploy develop, or just wait for the next release? | 18:33 |
| clarkb | I'll stew on that for a bit while I do other things | 18:34 |
| fungi | i guess without knowing when the next release will be, that might be "roll back to 2.2.1 and restore the database backup... then wait"? | 18:49 |
| clarkb | fungi: I suspect we may not need a db backup rollback because I don't think they've made changes to the db format. But also I think everything mostly works except for the embedded etherpad in meetpad (meetpad itself should still work for comms) so maybe we just roll forward? | 18:50 |
| clarkb | unrelated to everything else https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack.org/thread/H6NWHXBZI4FKTLY6VY3GVNYNWN7XSTFH/ is an interseting email it doesn't have content in the archive I think baceuse they didn't send a plain text version too? | 18:50 |
| clarkb | fungi: maybe we reach out on the issues and ask if we cherry pick that one commit if that is a workable compromise or if there aer other commits we need too? | 18:51 |
| clarkb | or ask if a 2.2.5 will be happening soon | 18:52 |
| clarkb | the commit log since 2.2.4 seems liek a small number of bugfixes and may make sense to do another quick bugfix release if we wask them | 18:52 |
| fungi | i don't think it's just the lack of plaintext part in that message that results in the weird archive result, if memory serves there are some message types defaulted in clients more popular with southeast asian script users that end up with a mime part that is an encoded message object consisting of additional mime parts, and hyperkitty may have trouble decoding that correctly to pick which part | 18:56 |
| fungi | is the actual message text | 18:56 |
| clarkb | ah. There are other odd things too liek I think maybe I was bcc'd but also the headers are there for the mailing list so my mua knows how to reply to list but if I hit reply to all it only replys to the sender not the list and sender like it does normally | 18:57 |
| clarkb | definitely doing weird things with their client | 18:57 |
| fungi | it seems these anomalies originate from people who don't typically use e-mail for communication | 18:58 |
| clarkb | fungi: ya the more I think about this the more it makes sense to me that etehrpad should make a bugfix release soon given the bugfixes on the develop branch. I think we should just respond on the issue and ask if they are open to doing that | 18:58 |
| clarkb | ya also using screenshots to capture text logs is painful to me | 18:59 |
| fungi | or may only use it for inter-office communication on some bespoke corporate group communications platform where compatibility with external mail systems is a half-implemented afterthought | 18:59 |
| clarkb | fungi: I posted https://github.com/ether/etherpad-lite/issues/6587#issuecomment-2344526946 and now time for lunch | 19:21 |
| fungi | yeah, fingers crossed. but at least we can run a snapshot in the interim if we want | 20:44 |
| fungi | headed out to grab dinner, but will check back in later | 20:44 |
| clarkb | should I prep a build of current develop or a cherry pick of that specific change? | 20:45 |
| clarkb | considering the blast radius of this is fairly small I think I'm willing to wait a bit to see if etherpad has a better answer for us before blazing ahead in any particular direction | 20:50 |
| clarkb | but I'm happy to be more proactive if others disagree | 20:50 |
| opendevreview | James E. Blair proposed openstack/project-config master: Add opendev/zuul-jobs repo https://review.opendev.org/c/openstack/project-config/+/928945 | 21:25 |
| opendevreview | James E. Blair proposed openstack/project-config master: Add opendev/zuul-jobs to Zuul https://review.opendev.org/c/openstack/project-config/+/928946 | 21:25 |
| cardoe | So might be the wrong place to ask but since OpenInfra is OIDC. Do you guys use that for Keystone and OpenStack services at all? | 21:26 |
| clarkb | we don't run any of our own keystones and I'm not aware of any that integrate with the openinfra oidc server so no on that front. There was some initial movement to moev login for thinsg to that but that stalled out, then we came up with the even better idea to have keycloak act as an identity proxy so that you can login with whatever you prefer including openinfra oidc but that is | 21:29 |
| clarkb | currently stalled out on getting a proxy layer in place for openid to ubuntu one (since that is what the bulk of current users are using having that will ease the transition). tonyb is working on getting that going currnetly though | 21:29 |
| clarkb | for example I think zanata (and maybe weblate) translation stuff authenticated with openinfra's identity server | 21:29 |
| jrosser | i integrate keystone with my employers OIDC (including keycloak in the middle as an identity broker) | 21:31 |
| jrosser | the deployment projects have examples of this | 21:31 |
| opendevreview | Merged openstack/project-config master: Add opendev/zuul-jobs repo https://review.opendev.org/c/openstack/project-config/+/928945 | 22:51 |
| cardoe | Guess I should tinker with keycloak then cause I’ve been messing with dex for the proxy piece. | 23:11 |
| cardoe | The docs around keystoneauth (client wise) are sparse for OIDC. The examples I’ve seen you have to have a lot of variables / data exported. I’ve got something working with vexxhost’s websso but that’s a third party plugin. | 23:12 |
| opendevreview | Merged openstack/project-config master: Add opendev/zuul-jobs to Zuul https://review.opendev.org/c/openstack/project-config/+/928946 | 23:17 |
| cardoe | It’s really the client side authentication piece that seemed awkward to me. | 23:20 |
| fungi | cardoe: you might ask for tips in #openstack-keystone too. i recall knikolla mentioning doing something similar for mass open cloud's federation, he's not been around much lately but other folks in there may have also done similar things | 23:54 |
| cardoe | Good point. | 23:58 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!