| fungi | yep, looks like your suggestion is compatible with ttx's poc, we can just reparent it | 00:00 |
|---|---|---|
| corvus | there's a poc? | 00:00 |
| fungi | https://review.opendev.org/718479 | 00:00 |
| fungi | only applied to openstack/release-test repo for now | 00:00 |
| fungi | not sure if it's even been triggered yet | 00:01 |
| corvus | neat, that's slightly unsafe due to the caveats i mentioned in my message | 00:01 |
| corvus | a hypothetical "x/nova" project could use that to overwrite the nova repo | 00:02 |
| fungi | yep, agreed | 00:02 |
| fungi | well, at least fight with openstack/nova over it anyway | 00:02 |
| *** mlavalle has quit IRC | 00:02 | |
| fungi | corvus: clarkb had suggested that current zuul limitations on running jobs with secrets would require adding them to project-pipeline within a trusted config repo, is that not the case? | 00:04 |
| clarkb | in order to "share" the secret I thought that was required, Though in this case maybe you cna make a variat that changes the var? | 00:05 |
| corvus | fungi: i don't think so? i think this is just like the doc promote jobs... | 00:05 |
| fungi | or is it just the case that we'd need to do it that way to account for triggering on tag events as well (because we want to replicate those too when they're pushed, but tag events don't convey a branch and so the job won't get run if added to a branched project due to implicit branch matcher behavior) | 00:06 |
| corvus | fungi: that is the case | 00:06 |
| fungi | corvus: got it, so yes we definitely need your regex idea, but also we'll still need to do the pipeline additions in the config repo | 00:07 |
| fungi | replying on-list | 00:07 |
| clarkb | fwiw I also suggested the job should be protected | 00:07 |
| clarkb | but that didn't seem to end up in the latest patchset, I should've left comments in gerrit not irc | 00:07 |
| corvus | fungi, ttx, mnaser: feel free to ping me on stuff like this. i wasn't intentionally ignoring that, i just had no idea any of this was happening. been a bit busy lately. | 00:09 |
| corvus | i would recommend reverting 479 (jobs like that scare me), then redoing it as suggested on the ml | 00:10 |
| fungi | sure, thanks! revert on the way | 00:11 |
| openstackgerrit | Jeremy Stanley proposed openstack/project-config master: Revert "Introduce job for granular GitHub mirroring" https://review.opendev.org/718839 | 00:14 |
| fungi | corvus: clarkb: mnaser: ttx: ^ | 00:14 |
| corvus | you might just be able to do "git_mirror_repository: {{ zuul.project.name }}" and that would be enough. that will certainly work for openstack/*. and would even work for foo/* if the project name were the same on github and that org also added the same user. and if that weren't the case, it would fail harmlessly. but if we want to avoid attempting to push to github in that case, then we'd need the | 00:15 |
| corvus | regex check. | 00:15 |
| fungi | yep, also your idea is more flexible anyway, since it would allow us to make versions for mirroring openstack/ namespace repos to a more diverse set of gh orgs if desired | 00:17 |
| fungi | like putting services under openstack-api and shared libraries under openstack-lib and so on | 00:17 |
| fungi | which was one of the ideas the folks who care about organizing things on github were todding around anyway | 00:17 |
| fungi | er, tossing around | 00:18 |
| mnaser | corvus: I’m happy to help somewhat in this effort as it’s something that’s helpful for our opendev tenant | 00:18 |
| fungi | i don't think todd was involved at all ;_ | 00:18 |
| openstackgerrit | Merged openstack/project-config master: Revert "Introduce job for granular GitHub mirroring" https://review.opendev.org/718839 | 00:32 |
| *** ysandeep|out is now known as ysandeep | 00:34 | |
| mnaser | post jobs for manage-projects seem to have failed | 03:22 |
| mnaser | Unable to freeze job graph: Job infra-prod-remote-puppet-else depends on infra-prod-update-system-config which was not run. | 03:22 |
| mnaser | http://zuul.opendev.org/t/openstack/job/infra-prod-remote-puppet-else | 03:24 |
| mnaser | that job's parent is infra-prod-service-base which depends on infra-prod-update-system-config running | 03:25 |
| mnaser | does this mean we need to include infra-prod-update-system-config in post for openstack/project-config? | 03:25 |
| *** ysandeep is now known as ysandeep|off | 03:28 | |
| fungi | hrm, that's a good question... i don't *think* any of the deployment actions we trigger off project-config ought to implicitly depend on having latest system-config, but i could be wrong | 03:31 |
| fungi | mordred? | 03:31 |
| openstackgerrit | Mohammed Naser proposed openstack/project-config master: Add infra-prod-update-system-config to deploy https://review.opendev.org/718877 | 03:31 |
| mnaser | fungi: ^ i pushed that up if that makes sense, it might be entirely wrong, but its something to start from | 03:31 |
| fungi | i mean, yes the way the jobs are defined right now that see,s to be the case | 03:31 |
| fungi | but it may mean we want to change up how that one is parented | 03:32 |
| mnaser | yep | 03:32 |
| mnaser | its a soft dependency when running in other projects but a hard one in openstack/system-config | 03:32 |
| mnaser | so maybe it might just be another variant with no dependencies | 03:33 |
| *** sgw has quit IRC | 03:38 | |
| *** sgw has joined #opendev | 04:06 | |
| ianw | clarkb: my preference for restoring suse would be to merge https://review.opendev.org/718299 and get building working without pip-and-virtualenv, and then the stack like https://review.opendev.org/#/q/status:open+project:zuul/zuul-jobs+branch:master+topic:ensure-pip | 04:18 |
| ianw | clarkb: i think that the primary motivation for suse is also devstack, which should work without pip-and-virtualenv; if it doesn't have a a couple of patches up to devstack that would make it work, i haven't checked on their merge status | 04:19 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: ensure-pip: export ensure_pip_virtualenv_command https://review.opendev.org/718224 | 04:53 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: fetch-zuul-cloner: use ensure-pip https://review.opendev.org/717882 | 04:53 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: fetch-subunit-output test: use ensure-pip https://review.opendev.org/718225 | 04:53 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: ensure-tox: use ensure-pip role https://review.opendev.org/717663 | 04:53 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: Update Fedora to 31 https://review.opendev.org/717657 | 04:53 |
| ianw | clarkb: ^ thanks for comments. responded inline, but on the order of installation -- i think we have to let the distribution own "pip" in the packaged case; in that case it doesn't matter if you install python3 or 2 first as only one package owns that generic namespace | 04:55 |
| ianw | it's when you're installing outside the system packages with git-pip.py that you end up fighting over /usr/local/bin/pip | 04:56 |
| *** DSpider has joined #opendev | 05:48 | |
| *** dpawlik has joined #opendev | 06:20 | |
| AJaeger | ianw: did you see corvus' comment on https://review.opendev.org/718284 ? | 07:16 |
| AJaeger | corvus: in your email you say that the documentation promote job has something similar - could you point me to that one, please? I'm not seeing that directly | 07:31 |
| *** tosky has joined #opendev | 07:32 | |
| AJaeger | mnaser, noonedeadpunk, after the removal of pip_install and repo_build, we have now failures in openstack tenant, please fix: http://zuul.opendev.org/t/openstack/config-errors | 07:35 |
| *** rpittau|afk is now known as rpittau | 07:36 | |
| *** hashar has joined #opendev | 07:52 | |
| *** roman_g has quit IRC | 08:18 | |
| ttx | corvus: your proposal looks indeed a lot safer. I was operating under the assumption that the job would only be available from openstack/project-config's projects.yaml, and therefore openstack-infra reviewers could make sure it's not abused by a x/nova thing. Was that a bad assumption? | 08:28 |
| AJaeger | ttx, for that you would need to mark it as such - for example using "protected" | 08:35 |
| ttx | Ah ok, someone told me that the secret would not be available to children jobs | 08:37 |
| ttx | since those would be defined in a different repo | 08:37 |
| AJaeger | not children - but you could still add it as is AFAIK | 08:38 |
| openstackgerrit | Bernard Cafarelli proposed openstack/project-config master: Update Grafana dashboards for stable Neutron releases https://review.opendev.org/718676 | 08:51 |
| noonedeadpunk | AJaeger: yeah, having a look | 08:57 |
| noonedeadpunk | I also noticed missing jobs today | 08:59 |
| AJaeger | noonedeadpunk: yeah, jobs cannot run with such an error ;( | 09:04 |
| ttx | re: meetpad, there are a number of Jitsi limitations that might block us for using it for big events | 09:09 |
| ttx | Hard limit at 75 users in a room, with performance strongly degrading starting at 35: https://community.jitsi.org/t/maximum-number-of-participants-on-a-meeting-on-meet-jit-si-server/22273/16 | 09:09 |
| ttx | We need some mechanism to make sure only trusted people join the room, and can be kicked out. By default anyone can create a room in Jitsi, and then the first people in it is moderator and can set (and reset) a password. So we might need some way to communicate out-of-band password change, or some chanserv-like bot to keep the room | 09:12 |
| ttx | Finally there is GDPR compliance, which is not there, but they are apparently working on it: https://community.jitsi.org/t/privacy-gdpr/26388/5 | 09:14 |
| ttx | corvus: ^ | 09:14 |
| *** persia_ is now known as persia | 10:07 | |
| *** rpittau is now known as rpittau|bbl | 10:19 | |
| *** rpittau|bbl is now known as rpittau | 11:58 | |
| ttx | corvus, fungi: re: that github-mirroring, see http://lists.openstack.org/pipermail/openstack-discuss/2020-April/014046.html | 12:03 |
| fungi | ttx: that jitsi issue seems to be about the insufficiency of gdpr compliance statements on the instance hosted at meet.jit.si | 12:31 |
| fungi | though i'll grant that if opendev is going to start putting gdpr statements on all of its services it would need to do so for meetpad as well | 12:32 |
| fungi | i guess that issue is further asking for a gdpr statement template so that people who are comfortable publishing generic legal statements without getting a lawyer involved to write them can do so | 12:34 |
| fungi | though i have a feeling most organizations who feel like they need to publish a gdpr statement will also feel compelled to engage legal counsel to write it for them rather than trust the authors of the application | 12:35 |
| openstackgerrit | Thierry Carrez proposed openstack/project-config master: [check-approval] Use committer instead of owner https://review.opendev.org/718994 | 12:36 |
| * fungi notes our gerrit instance also has "personal information" and doesn't say anywhere how it's used | 12:36 | |
| fungi | same probably goes for anything interactive we host which people can log into or otherwise input data in (mailman, etherpad, ethercalc, mediawiki, zanata, limesurvey, ...) | 12:38 |
| fungi | (storyboard, lodgeit, askbot, asterisk, maybe our irc bots...) | 12:40 |
| ttx | indeed | 12:40 |
| fungi | for that matter, basically every service we run at least records the ip addresses which connect to it, and that alone is considered personal data which we'd probably at least be expected to explain we only look at for troubleshooting and diagnostic purposes | 12:42 |
| fungi | so i wouldn't consider lack of gdpr integration in jitsi a blocker for using it in opendev any more than lack of gdpr integration is a blocker for any of the other services we run | 12:43 |
| mnaser | fwiw i think the only issue is that it may mean a different story about using jitsi for something like an "openinfra event" | 12:45 |
| mnaser | because then that might introduce some sort of liability wrt to the OSF | 12:46 |
| fungi | if it's something the osf is saying is an official part of the event, i suppose... though we use ptgbot and etherpad and gerrit and other things in nearly as officially linked a capacity for past ptgs | 12:49 |
| fungi | mnaser: were you saying you were interested in taking this task on? http://lists.openstack.org/pipermail/openstack-discuss/2020-April/014046.html | 13:13 |
| *** prometheanfire has quit IRC | 13:16 | |
| fungi | ttx: and yeah, we were wrong about the replication job being restricted to adding in individual repos, though clarkb's suggestion in irc to mark it as a protected job would have mitigated that as well (i forgot he'd mentioned that when i was reviewing the change) | 13:25 |
| fungi | i need to disappear to pick up our weekly grocery order, but should return in time to help prep for the maintenances | 13:46 |
| fungi | bbiab | 13:46 |
| mnaser | fungi, ttx: yes, i can help with that, but i think i need a little bit of guidance/discussion with corvus around the implications of all of it because it's not fully clear to me yet | 14:00 |
| corvus | o/ | 14:02 |
| corvus | mnaser, ttx: i think start with a job like ttx wrote, then use an empty nodeset, then do something like https://opendev.org/opendev/base-jobs/src/branch/master/zuul.d/jobs.yaml#L252 | 14:04 |
| corvus | note how docs_branch_path is tied to the secret; it's not just a job variable, which means it can't be overridden. the person with the secret decides what the secret can be used for. | 14:05 |
| corvus | so do that to pass the git_mirror_repository variable to the role | 14:06 |
| corvus | then, i think there are 2 options: 1) in openstack, set that variable to "{{ zuul.project.name }}", so that x/nova won't fight with openstack/nova. that could mean that if someone doesn't have it set up correctly, it will attempt to push and fail. that's probably fine. | 14:07 |
| corvus | but if we don't want that to happen (ie, we're concerned about x/foo setting up replication with openstack's key and github seeing a bunch of failed pushes from that account and blocking it), then 2) add an extra task to the playbook for the job which does some kind of validation. | 14:08 |
| corvus | honestly, i think that's pretty low risk. i would probably just do #1 and stop there. | 14:08 |
| corvus | AJaeger: ^ | 14:08 |
| mnaser | yes i agree that 1 is enough | 14:09 |
| corvus | ttx: re meetpad -- frickler pointed us to a system for horizontally scaling jitsi bridges for more capacity; i only briefly reviewed it, but i suspect we could do that if called upon | 14:10 |
| corvus | ttx: i certainly agree that we would need extra support for managing rooms for larger events; though i wonder if for design-session style events if "trust everyone" might work? but then again, zoom-bombing has been invented, so maybe not. maybe the first-person-is-the-moderator would be enough for design sessions, and we ask leaders to join the room a bit early? | 14:13 |
| corvus | ttx: if we wanted to have bots hold the room, i suspect that would be possible -- apparently this whole thing works based on xmpp, so maybe we could have ptgbot join the xmpp room for scheduled sessions, have privileges, and annoint moderators... | 14:14 |
| corvus | (right now, the xmpp server is hidden, but it can be exposed) | 14:14 |
| corvus | fungi: ^ | 14:15 |
| corvus | ttx, fungi: though also, i think we have a little more evaluation to do once we're done with the etherpad transition to see if we think it's feasible. the last time we tried, it was "mostly working". | 14:16 |
| AJaeger | corvus: thanks | 14:31 |
| openstackgerrit | Mohammed Naser proposed opendev/base-jobs master: opendev-upload-git-mirror: add job https://review.opendev.org/719032 | 14:41 |
| mnaser | corvus, ttx, AJaeger, fungi ^ i think that is step 1, i'll try to work on pt 2 (aka in openstack/project-config) but landing that would be nice so the job becomes available | 14:42 |
| mnaser | hmmm | 14:43 |
| mnaser | i wonder if i should make git_mirror_repository variable passd through the secret | 14:43 |
| corvus | mnaser: yep i'm leaving a comment to that effect for clarity | 14:43 |
| mnaser | because most likely we won't be able to pass a 'root' value in a secret, afaik that's not possible | 14:43 |
| mnaser | ok cool | 14:44 |
| AJaeger | corvus: I left a question: How can we enforce that its run in trusted repo only? | 14:44 |
| AJaeger | should the job be abstract? | 14:45 |
| mordred | AJaeger: I think the idea is that we don't worry about it (see corvus comment in scrollback starting with "then, I think there are 2 options" | 14:48 |
| corvus | AJaeger, mnaser: responded to both | 14:50 |
| AJaeger | thx | 14:51 |
| *** rpittau is now known as rpittau|afk | 14:57 | |
| mordred | infra-root: I also put etherpad01.openstack.org into emergency - probably don't want puppet restarting it while we have it turned off | 15:29 |
| clarkb | mordred: we are doing review first right? Also it occurred to me we may need to add bup to our ansible for etherpad, review etc? | 15:31 |
| mordred | well - we already have bup ansible | 15:37 |
| mordred | HOWEVER - the way we have it set up is a little strangeish and I do think we need to do it for new etherpad | 15:37 |
| mordred | basically - we need to add the host to the backup group, run a pulse, then re-remove it | 15:38 |
| mordred | (at least that's what the comments in inventory/groups.yaml say) | 15:38 |
| corvus | that is weird; why? | 15:38 |
| mordred | I don't know | 15:38 |
| mordred | I've been meaning to bring this up with folks to wrap our heads around it | 15:39 |
| mordred | corvus: oh - sorry - process is slightly different ... | 15:39 |
| fungi | yeah, i noticed the bup run for the new etherpad server sent some weird cronspam | 15:39 |
| mordred | we need to add the server to the backup group in general | 15:39 |
| mordred | but we need to do one pulse with the backup-server uncommented | 15:39 |
| mordred | because we don't usually run ansible against the backup server normally | 15:40 |
| clarkb | ya that makes sense | 15:40 |
| corvus | got it | 15:40 |
| mordred | also - this makes me think - should we be putting the db dumps from etherpad somewhere so that bup is backing them up? | 15:40 |
| mordred | or will they be fine? | 15:41 |
| ttx | corvus: re: load, scaling bridges for more capacity IMHO only helps with the number of meetings, not capacity in a single meeting. We'll see in testing | 15:41 |
| ttx | corvus: re: securing access, I think ideally we would support some dynamic creation of room names and post them on the ptgbot somehow | 15:41 |
| mordred | we only have review-dev in backup right now | 15:41 |
| ttx | That way the meeting lead would jump to a random room, then add the link to the PTGbot for others to join | 15:41 |
| ttx | Makes the work of Zoombombers a lot more difficult | 15:42 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Add review and etherpad to backup group https://review.opendev.org/719036 | 15:42 |
| clarkb | mordred: that is why we backup to the local fs, bup will find them there unless the path is in bups exclude list | 15:43 |
| mordred | cool | 15:43 |
| mnaser | btw, new project addition is broken right now -- i pushed up https://review.opendev.org/#/c/718877/ but its probably not the best cleanest aproach | 15:43 |
| mordred | we should probably consider if there are additional hosts we should be backing up - because it looks like we only have review-dev in the backup list right now - at least til that patch ^^ | 15:43 |
| mnaser | zuul error message: "Unable to freeze job graph: Job infra-prod-remote-puppet-else depends on infra-prod-update-system-config which was not run." | 15:43 |
| clarkb | mordred: well puppet has all the puppet things backing up. And because we didn't replace review.o.o its still backing up. However in general we should have ansible manage that | 15:44 |
| clarkb | mordred: we should backup a single gitea too probably | 15:44 |
| corvus | ttx: yeah, that link to the jitsi forum had some interesting info on scaling; it all seems unclear. (i wonder if we muted video for most participants if it would scale more.) | 15:44 |
| fungi | so... here's one possibility. etherpad has "read-only" view urls which are an unguessable hash, i wonder if we could tie meetpad to those | 15:44 |
| mordred | mnaser: poo. actually - remove update-system-config and add a dependencies: [] to manage-projects | 15:44 |
| mnaser | mordred: oh gotcha | 15:45 |
| mordred | mnaser: (see service-nodepool) | 15:45 |
| clarkb | mordred: but yes we should ensure that new and old hosts don't lose their bup backups | 15:45 |
| fungi | ahh, though i gues sthe idea is that the etherpad view in jitsi-meet is writeable by each client? | 15:45 |
| fungi | i keep forgetting it's not just a view-only video stream there | 15:46 |
| clarkb | fungi: yes | 15:46 |
| fungi | so yeah, ignore that suggestion ;) | 15:46 |
| openstackgerrit | Mohammed Naser proposed openstack/project-config master: Drop dependencies from manage-projects https://review.opendev.org/718877 | 15:46 |
| mnaser | mordred: updated, and then once that lands it'll be nice if someone can kick that off manually too | 15:47 |
| corvus | mnaser: if you remove the file matcher then add it back in a second change it'll run :) | 15:48 |
| openstackgerrit | Mohammed Naser proposed opendev/base-jobs master: opendev-upload-git-mirror: add job https://review.opendev.org/719032 | 15:50 |
| mordred | corvus: speaking of - https://review.opendev.org/#/c/718827/ | 15:52 |
| corvus | status notice review.opendev.org is being restarted for scheduled maintenance; see http://lists.opendev.org/pipermail/service-announce/2020-April/000003.html | 15:55 |
| corvus | how's that? | 15:55 |
| clarkb | lgtm | 15:56 |
| corvus | mordred: are you doing the typing? you want to do a screen or anything, or just let us know everything worked? :) | 15:56 |
| fungi | that notice wfm | 16:00 |
| corvus | i'll wait until we have a mordred to send it | 16:01 |
| AJaeger | mnaser, corvus , shouldn't opendev-upload-git-mirror show up in the docs? | 16:02 |
| corvus | do we know if everything on the server is ready to go? are all the docker changes in place, or do we need to do something to make that happen? | 16:02 |
| clarkb | corvus: aiui we just have to stop and disable the systemd service then start the docker compose thing | 16:03 |
| mnaser | AJaeger: i wonder why they're not showing up.. | 16:03 |
| fungi | i assumed it was all in place because the changes i'm aware of since last attempt have merged and the server isn't in the emergency list | 16:03 |
| clarkb | but we should haev mordred confirm | 16:03 |
| corvus | AJaeger: yeah, need to add it to doc/source/jobs.rst | 16:03 |
| AJaeger | mnaser: you need to add the job manually. Also, lets make the job abstract, shouldn't | 16:03 |
| corvus | AJaeger: why abstract? | 16:03 |
| mordred | sorry - was reading a document - ready to go now | 16:04 |
| corvus | (it's runnable as-is) | 16:04 |
| mnaser | i'll make a follow up that adds all the jobs to base-jobs | 16:04 |
| corvus | #status notice review.opendev.org is being restarted for scheduled maintenance; see http://lists.opendev.org/pipermail/service-announce/2020-April/000003.html | 16:04 |
| openstackstatus | corvus: sending notice | 16:04 |
| mordred | yeah - I'll do the typing - let me set up a screen | 16:04 |
| AJaeger | corvus: really runnable as-is? | 16:04 |
| -openstackstatus- NOTICE: review.opendev.org is being restarted for scheduled maintenance; see http://lists.opendev.org/pipermail/service-announce/2020-April/000003.html | 16:04 | |
| corvus | AJaeger: sure, just add secret :) | 16:04 |
| AJaeger | corvus: so, just passing in a secret but that could be done in project-stanza? | 16:04 |
| mordred | I'm in a root screen session on review | 16:04 |
| corvus | AJaeger: yep | 16:04 |
| AJaeger | mnaser: so, ignore the request for abstract ;) | 16:05 |
| fungi | attached | 16:05 |
| clarkb | I'm attached as well | 16:05 |
| corvus | me too | 16:05 |
| mnaser | AJaeger: if you wanna +w, i'll push up a follow-up adding all docs for all jobs | 16:05 |
| mordred | for stopping - is it systemctl stop gerrit? | 16:05 |
| AJaeger | mnaser: ok | 16:05 |
| fungi | mordred: yep, or service gerrit stop | 16:05 |
| mordred | (I gotta say this might be one of my favorite parts about the dockerification) | 16:05 |
| AJaeger | mnaser: done | 16:05 |
| fungi | btw i love that systemd decided reversing the parameter order was no big deal | 16:06 |
| * mordred will wait until we get the "done sending notice" | 16:06 | |
| * fungi forgot to add his <sarcasm> tag | 16:06 | |
| clarkb | mordred: you'll want to disable the unit too `systemctl disable gerrit` | 16:06 |
| openstackgerrit | Merged openstack/project-config master: Drop dependencies from manage-projects https://review.opendev.org/718877 | 16:06 |
| clarkb | (to prevent systemd from trying to manage it going forward) | 16:06 |
| mnaser | AJaeger: im going to see how we can make opendev/base-jobs linters get mad if you dont have a documented job | 16:06 |
| mnaser | or figure out how we do it in zuul/zuul-jobs | 16:06 |
| mordred | clarkb: yeah - I've got a todo list item to go remove the old units and stuff - but I'll do that right now | 16:06 |
| openstackstatus | corvus: finished sending notice | 16:07 |
| mordred | ok. here we go - everybody ready? | 16:07 |
| corvus | ++ | 16:08 |
| * fungi braces for impact | 16:08 | |
| mordred | ok. gerrit seems to be down | 16:09 |
| *** mlavalle has joined #opendev | 16:09 | |
| mordred | starting | 16:09 |
| fungi | yep, and disabled effectively | 16:09 |
| corvus | what's gerritcompose_shell? | 16:10 |
| mnaser | AJaeger: hmm i think that might involve writing a job/role to do the autodoc stuff as i might have to steal it from zuul/zuul-jobs, me has ENOTIME for that, so ill just manually add them | 16:10 |
| mordred | it's an extra container that does nothing for using for random exec tasks | 16:10 |
| corvus | gertty is happy, web site seems up | 16:11 |
| clarkb | so are we happy with that? the error log is clean | 16:11 |
| mordred | there was a reason why just using the gerrit one wasn't always a good idea | 16:11 |
| openstackgerrit | Merged opendev/system-config master: Collect logs from manage-projects runs https://review.opendev.org/718827 | 16:11 |
| corvus | merges are apparently possible :) | 16:12 |
| mordred | yeah | 16:12 |
| mordred | that was awesome :) | 16:12 |
| clarkb | [2020-04-10 16:11:54,207] [HookQueue-1] INFO com.googlesource.gerrit.plugins.hooks.HookTask : hook[change-merged] output: FileNotFoundError: [Errno 2] No such file or directory: '/home/gerrit2/projects.yaml' | 16:12 |
| clarkb | the hooks are broken | 16:12 |
| mordred | poo. so we're missing a bind-mount | 16:12 |
| clarkb | but general operation seems to be fine | 16:12 |
| corvus | also /home/gerrit2/review_site/etc/gerrit.config | 16:12 |
| openstackgerrit | Merged opendev/base-jobs master: opendev-upload-git-mirror: add job https://review.opendev.org/719032 | 16:13 |
| corvus | which is used by the welcome-message hook | 16:13 |
| clarkb | [2020-04-10 16:12:17,250] [HookQueue-1] INFO com.googlesource.gerrit.plugins.hooks.HookTask : hook[patchset-created] output: FileNotFoundError: [Errno 2] No such file or directory: '/home/gerrit2/review_site/etc/gerrit.config' | 16:13 |
| mordred | shall we just manual that in to the compose file and do another restart? | 16:13 |
| mordred | then followup with a change? | 16:13 |
| clarkb | mordred: how is it that the gerrit config isn't mounted? | 16:13 |
| fungi | i'm fine with that | 16:13 |
| openstackgerrit | Mohammed Naser proposed opendev/base-jobs master: docs: add docs for all jobs https://review.opendev.org/719042 | 16:13 |
| clarkb | gerrit needs that to start so I think there may be more to this? | 16:14 |
| fungi | clarkb: i think it's just not at that exact path? | 16:14 |
| mordred | it's in a different path in the container | 16:14 |
| clarkb | fungi: ya so adding bind mounts won't fix that | 16:14 |
| mordred | it will | 16:14 |
| clarkb | unless we bind mount to the other path too? | 16:14 |
| mnaser | AJaeger: ^ fyi re adding docs | 16:14 |
| fungi | right, we can bindmount it to additional paths for now, and we can also fix the hook to use the same one the gerrit process does | 16:14 |
| clarkb | got it | 16:15 |
| corvus | where's the path coming from? can we do something other than double-bind-mount it? | 16:15 |
| * fungi checks | 16:15 | |
| clarkb | corvus: I'm guessing jeepyb scripts hard code those paths | 16:15 |
| mordred | we could do that | 16:15 |
| clarkb | but maybe its configurable | 16:15 |
| corvus | like can we update the scriptos to use /var/gerrit/... | 16:15 |
| mordred | corvus: yeah - I think we should udpate the scripts - that'll just take a review cycle | 16:15 |
| fungi | yeah, there's a ton of /home/gerrit2/... defaults in jeepyb | 16:15 |
| fungi | most seem to be overridable | 16:15 |
| corvus | i'm okay with the double-bind-mount just to see if there are any more problems | 16:16 |
| mordred | or maybe it doesn't matter since it'll be a restart to roll out the fixed version anyway? | 16:16 |
| mordred | let's do that for now | 16:16 |
| mordred | does what I've got htere look ok? | 16:16 |
| corvus | but yeah, we're looking at more restarts no matter what | 16:16 |
| mordred | yeah | 16:16 |
| mordred | luckily they are _quick_ restarts | 16:16 |
| fungi | jeepyb/cmd/welcome_message.py will need editing though, yes, it hard-codes BASE_DIR = '/home/gerrit2/review_site' | 16:16 |
| clarkb | if /home/gerrit2 doesn't exist in the container will the bind mounting process sort that out for us? | 16:16 |
| mordred | yeah | 16:16 |
| clarkb | also I think it may be trying to get db connection info which is in the other secure.conf now? | 16:17 |
| mordred | clarkb: liek that? | 16:17 |
| clarkb | fungi: ^ can you check really quickly to see what it is trying to get from gerrit.config and if it looks in secure.config (or whatever it is called) too? | 16:17 |
| clarkb | mordred: ya I expect its looking in both for the db connection info | 16:18 |
| fungi | yeah, seems jeepyb/gerritdb.py will obey envvars for GERRIT_CONFIG and GERRIT_SECURE_CONFIG but defaults those to /home/gerrit2/review_site/etc/gerrit.config and /home/gerrit2/review_site/etc/secure.config respectively | 16:18 |
| corvus | mordred: lgtm | 16:18 |
| clarkb | so having both is probably a good thing (otherwise we'll get failure on the other file when we restart) | 16:18 |
| mordred | ok. cool | 16:18 |
| mordred | oh - we can set env vars? | 16:18 |
| fungi | mordred: for at least some of this, yes | 16:18 |
| clarkb | mordred: fungi depends on whether or not gerrit will set those when calling hooks | 16:18 |
| mordred | why don't we do that for those real quick - can we do that for projects.yaml? | 16:18 |
| mordred | oh - good point | 16:18 |
| fungi | anything which uses the jeepyb/gerritdb.py module and is breaking there at least | 16:19 |
| mordred | let's do this - then investigate more | 16:19 |
| clarkb | mordred: wfm | 16:19 |
| fungi | right, we may need to export those in the hook script wrappers | 16:19 |
| mordred | I'm going to save that, do a compose down and up | 16:19 |
| mordred | fungi: that would work | 16:19 |
| mordred | k. seems down. starting | 16:20 |
| fungi | basically gerrit is configured to run some shell scripts and pass command-line parameters to them, which then in turn call jeepyb entrypoint wrappers with more specific command-line options set | 16:20 |
| clarkb | did you ps to check if it was down? | 16:20 |
| clarkb | error log never showed it being down | 16:20 |
| mordred | I didn't - but the container was gone, so I don't think it could still be running right? | 16:21 |
| clarkb | there is only one gerrit ps | 16:21 |
| clarkb | mordred: I think it could be if it got reparented all the way back up to systemd | 16:21 |
| AJaeger | mnaser: I'll look into the linter for jobs in opendev/base-jobs | 16:21 |
| fungi | yeah, the only java processes i see are for the container | 16:21 |
| clarkb | but in this case seems fine it just didn't log it like I expected it to | 16:21 |
| mordred | yeah- I wonder if we need to configure compose to send a graceful shutdown signal first | 16:21 |
| corvus | seems to be up again | 16:22 |
| clarkb | mordred: oh ya that may be it | 16:22 |
| fungi | also a spontaneous thought, don't need to discuss it in the middle of this, but #opendev-meeting might be good for maintenance windows too | 16:22 |
| corvus | fungi: wfm. maybe we should do the 1700 maint there | 16:23 |
| fungi | i'm up for that | 16:23 |
| mordred | ++ | 16:23 |
| mordred | we could even startmeeting them so we can log todo list items | 16:23 |
| openstackgerrit | Mohammed Naser proposed openstack/project-config master: Revert "Revert "Introduce job for granular GitHub mirroring"" https://review.opendev.org/719047 | 16:24 |
| mordred | projects.yaml and projects.ini seem to be settable via env vars too | 16:24 |
| mordred | so I think we can not do the double-mount and instead just set those vars in teh scripts | 16:24 |
| mnaser | corvus (when you're not in the middle of doing gerrit things), ttx, AJaeger, fungi ^ i believe that this is a good point for github mirroring job | 16:24 |
| corvus | jeepyb seems really unhappy? | 16:24 |
| clarkb | [2020-04-10 16:24:14,247] [HookQueue-1] INFO com.googlesource.gerrit.plugins.hooks.HookTask : hook[patchset-created] output: fatal: not a git repository: '/home/gerrit2/review_site/git/openstack/project-config.git' | 16:24 |
| fungi | hrm, yeah, looks like it's expecting the bare repos at the old path? | 16:25 |
| openstackgerrit | Andreas Jaeger proposed opendev/base-jobs master: Check that all jobs are documented https://review.opendev.org/719048 | 16:25 |
| AJaeger | mnaser: ^ | 16:25 |
| fungi | longer term, so much of these gerrit hooks could be come zuul jobs | 16:26 |
| fungi | er, become | 16:26 |
| mordred | sigh | 16:26 |
| corvus | it also seems to have problems with the gerrit config file? | 16:26 |
| corvus | [2020-04-10 16:25:51,996] [HookQueue-1] INFO com.googlesource.gerrit.plugins.hooks.HookTask : hook[patchset-created] output: configparser.DuplicateOptionError: While reading from '<???>' [line 115]: option 'footer' in section 'trackingid "launchpad-bug"' already exists | 16:26 |
| mordred | yeah - I think we can do one more double-bind-mount for the git thing - but what's the issue with the gerrit config file | 16:26 |
| mordred | yeah- I agree - there are three footer options in section trackingid "launchpad-bug" | 16:27 |
| mordred | is it possible python3 configparser is just stricter? | 16:28 |
| fungi | oh, yep i bet it doesn't like duplicate options | 16:29 |
| mordred | we could maybe rework the gerrit.config to list those as three different trackingid sections rather than one with 3 footers | 16:29 |
| mordred | the download section is totally going to break it though | 16:30 |
| fungi | that would probably work since they're not tied to an "its" plugin | 16:30 |
| fungi | there's magic name glue between tracking ids and its plugins | 16:30 |
| mordred | maybe we just need to rework that script to have its own config file that we write out | 16:30 |
| mordred | so that it doesn't have to read this one | 16:30 |
| mordred | in any case - that's not going to be a quick fix | 16:30 |
| fungi | i'd rather rewrite it as a zuul job if we're doing that | 16:30 |
| mordred | yeah | 16:30 |
| mordred | can we though? doesn't it need acess to the gerrit db? | 16:31 |
| fungi | oh, yeah for working out assignees | 16:31 |
| corvus | that's not going to work long run anyway | 16:31 |
| fungi | right, it maps openids to lp usernames via gerrti db query and lp api call | 16:31 |
| corvus | we should go ahead and add "not access gerrit db" to our requirements if we want to upgrade to nodetdb | 16:32 |
| corvus | notedb | 16:32 |
| mordred | good point | 16:32 |
| fungi | i'm for hacking around it in the config for now if we can while we design something more effective long-term | 16:32 |
| mordred | in either case, I think we're going ot need to once more shut down and start the old way :( | 16:33 |
| fungi | we also never got task assignment implemented in the storyboard-its plugin, for similar reasons | 16:33 |
| mordred | because I don't know how to hack anything in place temporarily | 16:33 |
| mordred | maybe we give up on task assignment | 16:33 |
| fungi | yeah, i'm willing to entertain that | 16:34 |
| corvus | i agree that we've arrived at restart with old method | 16:34 |
| mordred | but - for now - let's shut back down, start the old way, and make a list of things we need to fix yes? | 16:34 |
| clarkb | mordred: yes, do you want to test graceful shutdown? | 16:34 |
| fungi | at least it'll be shorter than the last list | 16:34 |
| clarkb | I've got a change just about ready to push up based on reading the init script | 16:34 |
| clarkb | oh its already stopped | 16:34 |
| mordred | clarkb: oh - sorry | 16:35 |
| clarkb | its ok will push it up for review when its up again :) | 16:35 |
| mordred | ++ | 16:35 |
| openstackgerrit | Andreas Jaeger proposed opendev/base-jobs master: Check that all jobs are documented https://review.opendev.org/719048 | 16:35 |
| mordred | I added four bullet points to the bottom of https://etherpad.openstack.org/p/gerrit-2020-03-20 | 16:36 |
| AJaeger | mnaser: ok for me to update https://review.opendev.org/#/c/719042/ ? | 16:36 |
| mordred | do they look good to people? | 16:37 |
| corvus | mordred: yeah, updated with slight mod | 16:38 |
| clarkb | ya that looks good | 16:38 |
| corvus | i tested out the configparser issue and put in a solution | 16:39 |
| corvus | (strict=False) | 16:39 |
| fungi | ooh, good find on strict=false royal purple! | 16:39 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Use HUP to stop gerrit in docker-compose https://review.opendev.org/719051 | 16:40 |
| mordred | awesome. strict=false seems like a great short-term solution there | 16:40 |
| clarkb | others should compare ^ to the init script. The timeout is actualyl shorter now than I remembered | 16:40 |
| fungi | olive drab: i added something about the bare git repos since we saw jeepyb complain about that too | 16:40 |
| clarkb | but I still went with 5 minutes beacuse why not. | 16:40 |
| mordred | ++ | 16:40 |
| clarkb | also it really looks like sighup is used to stop gerrit gracefulyl which is weird | 16:40 |
| corvus | that is weird | 16:40 |
| fungi | only weird if you don't consider that the original signal was a call to "hangup" the line | 16:41 |
| clarkb | but I also discovered it uses sigterm if not using start-stop-daemon | 16:41 |
| clarkb | our server has start stop daemon isntaelled so pretty sure it uses the sighup path | 16:41 |
| corvus | fungi: i do remember that, but still think it's weird :) | 16:41 |
| fungi | (sighup will terminate shells) | 16:41 |
| corvus | i almost never want to stop gerrit when i hang up. | 16:41 |
| fungi | but yeah, i agree it's not a traditional signal i would expect for a graceful shutdown of a headless daemon | 16:42 |
| clarkb | yall shoudl read the init script and check that I interpreted that correctly :) | 16:42 |
| fungi | granted, those usually end up being some sigusrN | 16:42 |
| fungi | or just sigterm | 16:42 |
| mordred | so I think step one is just a jeepyb patch that updates the hardcoded filepaths - and maybe a patch to system-config to add envvars to set the paths? | 16:43 |
| mordred | and also strict=False | 16:43 |
| openstackgerrit | Andreas Jaeger proposed opendev/base-jobs master: docs: add docs for all jobs https://review.opendev.org/719042 | 16:43 |
| AJaeger | mnaser: ^ | 16:43 |
| fungi | mordred: i think so, yes. the hook scripts are in system-config | 16:43 |
| fungi | so we can export our preferred paths from there | 16:43 |
| mordred | ok. I'll work on those patches | 16:43 |
| mordred | I think maybe adding an envvar override to the scripts where it's just hardcoded | 16:44 |
| mordred | to match the other ones | 16:44 |
| mordred | is the smallest jeepyb change | 16:44 |
| fungi | that may be all we need actually. i don't see where BASE_DIR in welcome_message.py ever even gets used | 16:48 |
| *** prometheanfire has joined #opendev | 16:48 | |
| fungi | it's used in jeepyb/cmd/notify_impact.py and jeepyb/cmd/update_blueprint.py and jeepyb/cmd/update_bug.py hard-coded (checking those out now) | 16:49 |
| fungi | but seems to be just cargo-culted cruft in jeepyb/cmd/welcome_message.py | 16:49 |
| fungi | yeah, i think those are going to need to grow some envvar lookups | 16:50 |
| openstackgerrit | Monty Taylor proposed opendev/jeepyb master: Fix issues from rolling out containers https://review.opendev.org/719052 | 16:50 |
| fungi | they do git commands with --git-dir=BASE_DIR/git/... | 16:50 |
| mordred | fungi, clarkb, corvus : ^^ | 16:50 |
| mordred | I think that should fix the git dir and strict parsing issues | 16:50 |
| fungi | heh, exactly what i was thinking! | 16:51 |
| mordred | oh - poo - update-bug | 16:51 |
| openstackgerrit | Andreas Jaeger proposed opendev/base-jobs master: docs: add docs for all jobs https://review.opendev.org/719042 | 16:51 |
| fungi | right, that one too | 16:51 |
| AJaeger | sorry, corvus for hurting your eyes! | 16:52 |
| openstackgerrit | Monty Taylor proposed opendev/jeepyb master: Fix issues from rolling out containers https://review.opendev.org/719052 | 16:52 |
| corvus | AJaeger: i left a +2 :) | 16:52 |
| AJaeger | corvus: updated already | 16:53 |
| *** dpawlik has quit IRC | 16:53 | |
| fungi | -> #opendev-meeting for etherpad maintenance (6 minute warning!) | 16:54 |
| openstackgerrit | Mohammed Naser proposed openstack/project-config master: Revert "Revert "Introduce job for granular GitHub mirroring"" https://review.opendev.org/719047 | 16:57 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Set env vars pointing to correct file locations https://review.opendev.org/719053 | 16:58 |
| mordred | infra-root: ^^ there's the other half | 16:59 |
| clarkb | mordred: and we don't need similar on the manage proejct side because it is all config file driven already right? | 16:59 |
| mordred | yeah - it already works and does its own bind-mounts | 17:00 |
| -openstackstatus- NOTICE: etherpad.openstack.org will be offline for about 30 minutes while it is migrated to a new server with a new hostname; see http://lists.opendev.org/pipermail/service-announce/2020-April/000003.html | 17:01 | |
| AJaeger | mnaser: want to +2 719048 now that it passes - and +2 719042 | 17:03 |
| mnaser | AJaeger: +2 +w | 17:05 |
| AJaeger | thx | 17:06 |
| openstackgerrit | Andreas Jaeger proposed zuul/zuul-jobs master: Fix check_jobs_documented linter https://review.opendev.org/719054 | 17:09 |
| openstackgerrit | Merged opendev/base-jobs master: docs: add docs for all jobs https://review.opendev.org/719042 | 17:10 |
| *** hashar has quit IRC | 17:20 | |
| openstackgerrit | matthew wagoner proposed opendev/system-config master: Fix typo on website https://review.opendev.org/719057 | 17:34 |
| fungi | !!! ^ | 17:37 |
| openstack | fungi: Error: "!!" is not a valid command. | 17:37 |
| fungi | hush, openstack, i wasn't talking to you | 17:37 |
| mnaser | so would it be okay to requeue manage-projects job into post or should i make a noop commit? | 17:44 |
| mnaser | (there was an issue with post jobs not working for manage project that didnt create repos) | 17:44 |
| mnaser | example of repo that failed in post: https://review.opendev.org/#/c/718824/ | 17:44 |
| mnaser | we merged the fix but yeah, now something needs to kick it off | 17:45 |
| *** diablo_rojo has joined #opendev | 17:45 | |
| -openstackstatus- NOTICE: The etherpad migration is still in progress; revised estimated time of completion 18:30 UTC | 17:51 | |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Set env vars pointing to correct file locations https://review.opendev.org/719053 | 17:57 |
| *** melwitt has quit IRC | 17:59 | |
| *** mugsie has quit IRC | 17:59 | |
| *** cmurphy has quit IRC | 17:59 | |
| *** Open10K8S has quit IRC | 17:59 | |
| openstackgerrit | Monty Taylor proposed opendev/jeepyb master: Fix issues from rolling out containers https://review.opendev.org/719052 | 18:01 |
| *** melwitt has joined #opendev | 18:03 | |
| *** cmurphy has joined #opendev | 18:03 | |
| *** mugsie has joined #opendev | 18:03 | |
| *** Open10K8S has joined #opendev | 18:03 | |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Run ansible on the backup server https://review.opendev.org/719076 | 18:21 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Turn backup server back off https://review.opendev.org/719077 | 18:21 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Add review and etherpad to backup group https://review.opendev.org/719036 | 18:22 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Run ansible on the backup server https://review.opendev.org/719076 | 18:22 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Turn backup server back off https://review.opendev.org/719077 | 18:22 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Set env vars pointing to correct file locations https://review.opendev.org/719053 | 18:25 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Set env vars pointing to correct file locations https://review.opendev.org/719053 | 18:29 |
| openstackgerrit | Merged opendev/system-config master: Change Etherpad default intro text https://review.opendev.org/718763 | 18:37 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Add root cron jobs to gerrit https://review.opendev.org/719088 | 18:40 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Add review and etherpad to backup group https://review.opendev.org/719036 | 18:46 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Run ansible on the backup server https://review.opendev.org/719076 | 18:46 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Turn backup server back off https://review.opendev.org/719077 | 18:46 |
| mordred | clarkb, corvus : ^^ had to fix a linter failure - we were checking group membership and I didn't udpate the fixture | 18:46 |
| mordred | now that we're 100% zuul-driven - we _could_ remove the update-system-config playbook and push the zuul content to the zuulcd user's home dir in our prod playbook, making zuul driven runs run with the context zuul prepares, rather than just the tip | 18:49 |
| corvus | i like that | 18:49 |
| mordred | I mention this because thinking about that backup stack - if we were running from zuul state, we could ladn all three patches of that stack and be sure that we would have run once with each state | 18:49 |
| corvus | ++ | 18:49 |
| corvus | that will also rely on the mutex of one right now, but we don't anticipate changing that any time soon | 18:50 |
| mordred | yeah | 18:50 |
| mordred | we'd need to update the ansible config to point to /home/zuulcd/src/opendev.org/opendev/system-config for inventory and roles instead of /opt/system-config ... or we could make a symlink | 18:51 |
| clarkb | thinking out loud here, how does that impact our ability to run things outside of zuul (I think this may be something to think about either way actually) | 18:51 |
| clarkb | with the existing setup if we put bridge in emergency it won't update things | 18:51 |
| clarkb | with the proposed setup we'd probably want to have a separate git repo for this? | 18:51 |
| mordred | for what? | 18:51 |
| mnaser | if anyone has some spare cycles, i'd appreciate: sudo zuul enqueue-ref --tenant openstack --trigger gerrit --pipeline post --project openstack/project-config --ref refs/heads/master --oldrev abc9da185343c691fa66585ee108586eff84a9f9 --newrev 395dfed08ca3c009d0040cc7c7a326ebd3dd0bf5 --- this should rerun the manage-projects and create the projects that were missed in the zuul misconfig | 18:52 |
| mordred | this is onlky talking about the state of the system-config repo - it _is_ the separate repo :) | 18:52 |
| clarkb | mordred: running kick.sh for whatever reason | 18:52 |
| mordred | clarkb: yeah - I don't think that would be impacted | 18:52 |
| clarkb | mordred: right but it has group vars and playbooks and stuff | 18:52 |
| mordred | still not impacted | 18:52 |
| mordred | the only thing that would be impacted is if we had specific inventory state | 18:52 |
| clarkb | mordred: isn't it? if you are expecing one state when running kick.sh but zuul updates that state | 18:52 |
| mordred | clarkb: oh - yeah - sorry - I didn't say a followup | 18:53 |
| clarkb | today the way we address that is bridge in emergency aiui | 18:53 |
| mordred | I think we want to keep /opt/system-config as a location from which we can run adhoc things | 18:53 |
| clarkb | and that will fix the git repo until removed | 18:53 |
| clarkb | mordred: got it | 18:53 |
| clarkb | and zuul would have its own thing | 18:53 |
| mordred | and - we probably want to go ahead and install the inventory files in install-ansible | 18:53 |
| mordred | rather than pointing directly at the files in system-config | 18:53 |
| mordred | that way it won't be weird | 18:53 |
| mordred | yup | 18:54 |
| * mordred will work up a patch | 18:54 | |
| clarkb | ok I need to pop out for lunch things now | 18:54 |
| clarkb | back in a bit (but then probably out again to bike) | 18:54 |
| *** moppy has joined #opendev | 18:54 | |
| * mordred is going to run manage-projects for mnaser | 18:54 | |
| mnaser | mordred: thank you :> | 18:55 |
| mordred | mnaser: k. check your projects - they ok? | 18:56 |
| mnaser | i see https://opendev.org/vexxhost/rbac-helm and https://opendev.org/vexxhost/documentation so yay | 18:56 |
| mordred | \o/ | 18:57 |
| mnaser | thanks mordred - ill watch for new project creations if the fix did the right thing in teh future | 18:57 |
| AJaeger | anybody willing to review to move the documentation linter from zuul-jobs to opendev/base-jobs, please? https://review.opendev.org/#/c/719048/ | 18:57 |
| openstackgerrit | Merged opendev/system-config master: Fix typo on website https://review.opendev.org/719057 | 19:22 |
| AJaeger | and a sync back for the job linter for review, please - https://review.opendev.org/719054 | 19:22 |
| mordred | clarkb: https://etherpad.opendev.org/p/clarkb-test is not loading for me - is it loading for you? | 19:28 |
| openstackgerrit | Merged opendev/base-jobs master: Check that all jobs are documented https://review.opendev.org/719048 | 19:30 |
| AJaeger | have a great easter weekend everybody! | 19:31 |
| mordred | AJaeger: you too! | 19:31 |
| * AJaeger waves good night | 19:31 | |
| *** sgw has quit IRC | 19:36 | |
| clarkb | mordred: hrm not loading for me either | 19:41 |
| corvus | i'll see if i can help it load by also trying | 19:42 |
| corvus | the gerrit maint etherpad from earlier works: https://etherpad.opendev.org/p/gerrit-2020-03-20 | 19:43 |
| clarkb | I wonder if the utf8 stuff is still weird there | 19:43 |
| clarkb | like maybe the db dump didnt dump that successfully? | 19:43 |
| mordred | yeah | 19:47 |
| mordred | that dump would have had the mb4 settings active, where the previous one didn't | 19:47 |
| *** moppy has quit IRC | 19:47 | |
| corvus | we probably should have dived the plan :) | 19:48 |
| mordred | maybe we should try removing the most recent edit from that pad? | 19:48 |
| *** moppy has joined #opendev | 19:48 | |
| clarkb | ya could roll back versions and see if it starts workinh | 19:48 |
| corvus | i'm worried about other pads though | 19:48 |
| corvus | basically, is every pad with a pile-of-poo dead? | 19:48 |
| clarkb | there is something poetic about that. I think rolling back versions of the test pad will help us confirm this | 19:50 |
| corvus | perhaps we will lose less data if we shut down the new server, and re-export the data | 19:50 |
| corvus | we'll lose changes from this morning, but maybe people are not doing much today? | 19:50 |
| clarkb | how would we export differently? that isnt clear to me | 19:50 |
| mordred | the way I did it before | 19:50 |
| mordred | run the dump on old etherpad server, copy the dump file over, then run it in on new server | 19:51 |
| corvus | my understanding is that the export process we did in production is not what we tested, due to mysql settings being different. also, perhaps running on a different host? (this may be the reason it took longer?) | 19:51 |
| clarkb | I see | 19:51 |
| mordred | we did todays' process yesterday to find the time length ... but we didn't chek for data integrity afterwords | 19:52 |
| corvus | oh, so that's not the reason it took longer | 19:52 |
| mordred | old etherpad is still shut down - how about I go ahead and start a dump, since it won't change regardless | 19:52 |
| mordred | while we investigate this | 19:52 |
| corvus | i'm concerned that if we do anything other than re-export, we will have a long tail of reports like "this pad from 2 summits ago where we had a 4-byte char doesn't work" | 19:53 |
| fungi | seems a reasonable next step, though we may also ought to temporarily shut production back down? | 19:53 |
| fungi | just so nobody makes changes and then gets surprised when they disappear | 19:53 |
| corvus | clarkb: if we did roll back recent edits of the test pad, what would that confirm? | 19:53 |
| corvus | fungi: i think so | 19:53 |
| corvus | fungi: we might need an announcement | 19:54 |
| clarkb | corvus: that the recent utf8mb4 edits arelikely to blame | 19:54 |
| clarkb | corvus: if rolling back doesnt fix then probably something else is involved | 19:54 |
| corvus | clarkb: ack. wfm | 19:54 |
| fungi | i can draft an announcement (not in an etherpad), just a moment | 19:54 |
| mordred | yeah | 19:54 |
| *** sgw has joined #opendev | 19:54 | |
| * mordred is running the dump from etherpad.openstack | 19:55 | |
| corvus | fungi, clarkb: let's do the rollback test before shutting it down | 19:55 |
| mordred | ++ | 19:55 |
| fungi | k | 19:55 |
| corvus | clarkb: you know how to do that? are you in a good place to do it? | 19:55 |
| clarkb | I'm not in a good spot. Currently finishing lunch | 19:55 |
| corvus | mordred, fungi: ^ either of you? | 19:55 |
| fungi | i can give it a try if someone tells me the pad name and desired revision | 19:55 |
| clarkb | in scrollback fungi had an api command for etherpad its that with a different url | 19:55 |
| corvus | https://etherpad.opendev.org/p/clarkb-test | 19:56 |
| corvus | no idea the revision | 19:56 |
| clarkb | fungi: pad name is clarkb-test and revisions will be current -1 and work backward from there | 19:56 |
| corvus | is there a wy to find the current rev? | 19:56 |
| fungi | yeah, i've done rollbacks recently, i can also probe it with the get text call to find the right rev | 19:56 |
| corvus | (via api or db query?) | 19:56 |
| corvus | cool | 19:56 |
| fungi | yeah, i can get current rev number too | 19:56 |
| fungi | etherpad api docs are pretty thorough | 19:56 |
| corvus | mordred: ^ you want to pick up the announcement work? | 19:56 |
| corvus | mordred: or, if you are otherwise occupied, i can | 19:57 |
| mordred | corvus: could you - I'm poking to make sure I'm dumping like I did before | 19:58 |
| corvus | mordred: on it | 19:58 |
| corvus | corvus is writing announcement; fungi is rolling back test pad; mordred is exporting; clarkb is lunching | 19:59 |
| fungi | oh yay, someone has defaced their wiki? https://github.com/ether/etherpad-lite/wiki/HTTP-API | 20:01 |
| fungi | anyway i have some examples in my shell history on the old server to draw on | 20:02 |
| mordred | ok - as an update- I am now dumping on etherpad01.opendev.org - which is what I did before. but I'm dumping on the _host_ not in the container | 20:03 |
| fungi | getRevisionsCount says 615 | 20:03 |
| mordred | the host does in fact have different my.cnf settings than the container | 20:03 |
| corvus | how does this look? http://paste.openstack.org/show/791941/ | 20:05 |
| mordred | also - fwiw, last time we ran it in we had utf8 and utf8_bin set in the my.cnf for the container - then we loaded the dump, then we changed the settings to allow creating the mb4 chars | 20:06 |
| mordred | corvus: yes, that looks good | 20:06 |
| fungi | corvus: lgtm | 20:06 |
| corvus | status notice Due to a database migration error, etherpad.opendev.org is offline until further notice. | 20:06 |
| mordred | yeah | 20:06 |
| fungi | clarkb: what revision did you want this rolled back to? or how can i identify it from the pad content? | 20:07 |
| corvus | fungi: keep doing N-1 | 20:07 |
| fungi | okay, can do, just a moment | 20:07 |
| corvus | #status notice Due to a database migration error, etherpad.opendev.org is offline until further notice. | 20:07 |
| openstackstatus | corvus: sending notice | 20:07 |
| -openstackstatus- NOTICE: Due to a database migration error, etherpad.opendev.org is offline until further notice. | 20:07 | |
| fungi | restored it to revision 614 now | 20:08 |
| corvus | fungi: no joy for me | 20:08 |
| mordred | nope | 20:08 |
| clarkb | fungi: pad content wise morder added some chinese characters that require 4 bytes | 20:08 |
| clarkb | its possible we have to go back more than one rev | 20:08 |
| fungi | now at 613 | 20:08 |
| mordred | nope | 20:08 |
| corvus | fungi: it may be a lot of revs, probably best for you to revert and test yourself :) | 20:08 |
| fungi | doing | 20:09 |
| corvus | (ftr i'm holding the email message until we actually confirm the problem and establish the end time) | 20:09 |
| mordred | ++ | 20:09 |
| mordred | I've put etherpad.opendev.org into the emergency file just to be safe | 20:10 |
| openstackstatus | corvus: finished sending notice | 20:11 |
| fungi | i've rewound 10 revs so far (down to 605) but will keep at it | 20:11 |
| mordred | ok. dump is done, perparing to be able to restore | 20:12 |
| mordred | if we move forward, what I'd like to do is change the mysql settings for the container back to utf8 and ut8_bin | 20:13 |
| clarkb | fungi: fwiw I'm not sure what forces a rev to happen but if each character is a rev we might have to go back ~30? | 20:13 |
| mordred | then do the restore, then change the settings back to utf8mb4 | 20:13 |
| mordred | since that is the sequence we used last time and it worked | 20:14 |
| fungi | clarkb: i'm back 20 revs to 595 now | 20:14 |
| corvus | mordred: we may also want to get a dump of the new db after we shut it down | 20:14 |
| mordred | corvus: ok | 20:14 |
| corvus | just in case someone comes to us with "i lost 40 hours of work due to the rollback" and we want to try to help | 20:14 |
| corvus | i think it's unlikely, but it seems like a precaution we could take | 20:15 |
| mordred | I will dump that one with the current db settings | 20:16 |
| corvus | sounds good. but wait until we shut it down after fungi's experiment | 20:16 |
| mordred | yes | 20:16 |
| fungi | i'm back 30 revs to 585 now | 20:17 |
| fungi | still no luck but can keep going | 20:17 |
| mordred | I thnik it;s entirely likely it's _not_ mb4 related and that it might be any multi-byte char related - and may have been an inappropriate cross-encoding | 20:18 |
| fungi | and yeah, a rev can be as little as a single character added or removed | 20:18 |
| corvus | mordred: there are old multibyte chars in that pad, so that could make the experiment inconclusive | 20:19 |
| mordred | yeah | 20:19 |
| fungi | i can try jumping back substantially further in that pad if we want | 20:19 |
| mordred | fungi: might as well | 20:19 |
| clarkb | fungi: sure its a test pad so this is what it is there for | 20:19 |
| corvus | maybe we should just shut it down, save the new db (which would then be insurance against our multibyte hunch being wrong) and try the re-import? | 20:19 |
| corvus | ^ (feel free to continue rollback experiment while considering this) | 20:20 |
| mordred | at this point probably a good idea | 20:20 |
| mordred | if the re-import doesn't work - we can always just turn old etherpad back on, apologize, and regroup for next week | 20:20 |
| fungi | i'm starting to think the problem is not recent edits in that pad | 20:21 |
| mordred | yeah | 20:21 |
| clarkb | fwiw clarkb-test on etherpad-dev hasn't worked in years | 20:21 |
| clarkb | its not entirely abnormal for etherpad to decide its unhappy :/ | 20:21 |
| corvus | clarkb: well, it worked on prod yesterday, right? | 20:21 |
| mordred | yeah | 20:21 |
| clarkb | corvus: yes this particular one was fine yesterday | 20:22 |
| corvus | does anyone have any other pads with multibyte test chars? | 20:22 |
| clarkb | I'm just pointing out that we've had some sort of issue with pads breaking and never been able to anrrow it down to a cause | 20:22 |
| clarkb | corvus: I am not aware of any at the moment | 20:22 |
| mordred | I'm driving in root screen and ready to shut down - pending folks being ready for it | 20:22 |
| corvus | clarkb: ack; but without any other tests to triangulate, i think we have to assume this could affect any multibyte pad | 20:22 |
| corvus | i'm ready | 20:22 |
| fungi | yeah, i'm now getting an "internal error" from getText() calls to the api for that pad | 20:22 |
| corvus | fungi: ? | 20:22 |
| fungi | i do not have another pad i can think of in that situation, no | 20:23 |
| corvus | clarkb, fungi: ready to shut down etherpad.opendev? | 20:23 |
| fungi | i say let's proceed with the restore | 20:23 |
| fungi | yes | 20:23 |
| clarkb | ya | 20:23 |
| mordred | ok. I'm dumping the current state of the new db into /var/lib/mysql/etherpad-new.sql | 20:25 |
| mordred | the new dump from old etherpad is etherpad.sql | 20:26 |
| mordred | once this is done, I will restart mariadb with the utf8 settings back to the other settings, restore the dump, then we can restart one more time with the settings restored | 20:26 |
| clarkb | k | 20:27 |
| clarkb | I'm to keyboard now fwiw so let me know if I can help with anything at this point | 20:31 |
| fungi | and my food is here so i may want to step away and scarf it down | 20:35 |
| rm_work | smcginnis: so on EM/EOL | 20:36 |
| rm_work | smcginnis: the ocata branch of octavia was marked EM one year ago -- https://opendev.org/openstack/releases/commit/d57f810ad6555528a4af039b693453b551da4cf2 | 20:36 |
| rm_work | same with pike: https://opendev.org/openstack/releases/commit/375a23874431bec0981e5545933e977d8f398000 | 20:36 |
| clarkb | rm_work: that might be more appropriate for #openstack-release? (we are in the middle of some operational fun so keeping on topic right now is useful) | 20:36 |
| rm_work | ah ok sorry :) will move | 20:37 |
| fungi | this situation also fits the incident response purpose for #opendev-meeting if we want to head back in there | 20:41 |
| corvus | looks like the dump of the new db is done | 20:42 |
| mordred | no - that was me accidentally pasting text into the buffer | 20:42 |
| corvus | indeed | 20:43 |
| corvus | mysqldump proc is still running | 20:43 |
| mordred | I just pasted a particularly confusing chunk of text | 20:43 |
| corvus | at least it wasn't a chapter from your tell-all book | 20:44 |
| fungi | that could describe my typical friday afternoon | 20:44 |
| mordred | corvus: I think it was | 20:44 |
| *** sshnaidm|off has quit IRC | 20:46 | |
| mordred | now it's done | 20:47 |
| corvus | looks like the dump of the new db is done | 20:47 |
| mordred | I'm going to swap the db settings restart mariadb then restore | 20:47 |
| mordred | well - luckily that output is all just people with root already :) | 20:49 |
| mordred | so - the problem, I think - is that the old db was utf8mb4 at the table level, but we did the original dump with a connection setting of utf8 - that tells the interaction "I am speaking utf8, you are utf8mb4, please transcode for me" | 20:51 |
| fungi | aha | 20:52 |
| mordred | hrm. no - my explanation doesn't hold up | 20:52 |
| mordred | because what we did today should have worked the same - just eliding 2 trancodings | 20:52 |
| mordred | shrug. I'm confused | 20:53 |
| corvus | i guess we'll found out in a few minutes | 20:53 |
| mordred | there's basically always three charsets - what the table is storing, what the server is speaking and what the client is speaking | 20:53 |
| fungi | right, without utf8mb4 set, people would have at most been able to enter 3-byte utf8 characters, which are a proper subset of 4-byte utf8 anyway | 20:53 |
| mordred | whenver one doesn't match, mysql will translate from one to the other for you | 20:54 |
| mordred | the combo that worked before was "table in utf8mb4, server in utf8, client in utf8" -> "client in utf8, server in utf8, table in utf8mb4" | 20:55 |
| mordred | what we tried today that did not work was "table in utf8mb5, server in utf8, client in utf8mb4" -> "client in utf8mb4, server in utf8mb4, table in utf8mb4" | 20:55 |
| mordred | if that makes sense | 20:56 |
| mordred | so - I think the issue is that the old server was configured to speak utf8 - but we were talking to it in utf8mb4 - so it did a step of transcoding that somehow became unhappy | 20:56 |
| fungi | there's a utf8mb5? | 21:00 |
| fungi | or was that a typo above? | 21:00 |
| mordred | that was a typo | 21:00 |
| fungi | okay, makes sense in that case | 21:01 |
| mordred | the thing is while it _should_ all work, if etherpad had been writing things without the correct settings, it could have been writing bogus-ish data that would still get re-translated back out right-ish- and if we apply the same set of tranformations and untransormations, the data stays the same | 21:02 |
| mordred | but with an unbalanced set of transformations, assertions we're making about what the data actually is start to matter | 21:03 |
| corvus | mordred: so it's the middle (server) later that you think did the deed, rather than the client difference | 21:04 |
| corvus | s/later/layer/ | 21:04 |
| mordred | corvus: yeah. *I think* | 21:04 |
| mordred | I *know* how I dumped it last time | 21:05 |
| mordred | I'm 95% positive I stored it before we swapped to mb4 - because I was actually worried about this | 21:05 |
| mordred | why that worry didn't translate to today I'm not sure | 21:05 |
| corvus | worry overload? | 21:06 |
| mordred | yeah | 21:06 |
| mordred | prolly | 21:06 |
| clarkb | I think its done | 21:18 |
| clarkb | orwait maybe the window changed? | 21:19 |
| clarkb | no I thik its done | 21:19 |
| corvus | hrm, screen is not responsive for me, but i agree there is no mysqldump process running | 21:19 |
| mordred | yes - done | 21:19 |
| mordred | I will now restart mariadb with the right settings | 21:19 |
| corvus | screen woke up | 21:19 |
| mordred | and etherpad | 21:19 |
| clarkb | should we try clarkb-test now? | 21:20 |
| mordred | wow. still broken :( | 21:20 |
| fungi | https://etherpad.opendev.org/p/clarkb-test is still hanging for me, yeah | 21:21 |
| corvus | ditto | 21:21 |
| fungi | we're sure this exact state of the pad worked on the old server? | 21:21 |
| fungi | maybe we can compare the fields in the two databases? | 21:21 |
| * fungi says with flippant wavy-hands typical of a friday evening | 21:22 | |
| clarkb | fungi: unless someone broke it overnight | 21:22 |
| corvus | we could probably restart the old server and directly access it? | 21:22 |
| corvus | i'll prepare to do that | 21:22 |
| mordred | kk | 21:22 |
| clarkb | corvus: ya that would be one way to test it | 21:22 |
| corvus | done; rejiggering my hosts file to test | 21:23 |
| fungi | if that works, then i give it 50/50 odds either the data is different in the imported db *or* new etherpad version isn't handling it like the old version did | 21:24 |
| corvus | it does not work -- chat loads but the main pad still hangs | 21:25 |
| corvus | let me do a browser restart just to clear ambiguity | 21:25 |
| mordred | ok. so it;s entirely possible that old clarkb-test between the times of imports broke | 21:25 |
| mordred | and we're seeing that reflected here | 21:25 |
| clarkb | yes, the window is small but not from a computing perspective | 21:25 |
| mordred | we were attempting to test writing mb4 chars to it | 21:26 |
| mordred | we could have caused garbage to get written | 21:26 |
| corvus | confrimed in both firefox and chromium | 21:26 |
| corvus | mordred: can you stop new server? | 21:26 |
| mordred | done | 21:26 |
| corvus | (let's avoid more deltas in case we decide to roll *forward*) | 21:26 |
| mordred | I'm now unsure of how to validate success | 21:27 |
| corvus | i think at this point we could really use a pad with multibyte chars | 21:28 |
| corvus | maybe we should go look at a pad index from the latest ptg | 21:28 |
| mordred | yeah. a pre-existing pad that peopel would expect to work | 21:28 |
| mordred | yeah | 21:28 |
| corvus | https://wiki.openstack.org/wiki/PTG/Ussuri/Etherpads ? | 21:28 |
| mordred | https://wiki.openstack.org/wiki/PTG/Ussuri/Etherpads | 21:28 |
| mordred | yeah | 21:28 |
| mordred | corvus: what's the ip for old etherpad? | 21:28 |
| corvus | 23.253.238.66 etherpad.openstack.org | 21:29 |
| fungi | etherpad01.openstack.org | 21:29 |
| corvus | is my hosts entry | 21:29 |
| fungi | will also get you the old addresses | 21:29 |
| fungi | as it's not been deleted from dns | 21:29 |
| fungi | but yeah, don't try to go to the hostname in a browser or you'll end up redirected obviously | 21:29 |
| clarkb | the i18n pad seems like it may be what we want since different langauges use different chars more often | 21:29 |
| corvus | https://etherpad.openstack.org/p/manila-shanghai-ptg-planning has some chinese chars | 21:30 |
| fungi | they may be 3-byte, but it's work checking at least | 21:31 |
| mordred | want me to start new so we can check that one? | 21:31 |
| corvus | yeah, our best bet for a 4 byte would be a pile of poop | 21:31 |
| fungi | alternative would be to figure out how it gets encoded in a mysqldump and then search the export for an example of poop | 21:31 |
| mordred | fwiw - I cannot diff the two dumps - diff runs out of memory | 21:32 |
| fungi | eww | 21:33 |
| fungi | i meant more like just diff the part of the dump for that pad, but yeah maybe that's not easy to isolate given how revisions are stored | 21:33 |
| mordred | yeah - I mean, the fact that the db format is ... INSANE ... doens't help matters | 21:34 |
| corvus | mordred says poop a lot | 21:35 |
| fungi | also i've confirmed that those three han characters in the manila pad are 3-byte | 21:35 |
| fungi | \xe6\xb5\xa6, \xe6\xb1\x9f and \xe5\xae\xb4 respectively | 21:36 |
| corvus | i don't think my "google for etherpad poop" idea is going to work | 21:36 |
| mordred | well - also worth confirming that they're good - so that's still a good pad to check | 21:36 |
| mordred | corvus: can you good for etherpad (insert poop emoji) | 21:36 |
| mordred | googling for: "💩" site:etherpad.openstack.org - did not work | 21:37 |
| fungi | pretty sure google doesn't index our etherpad (probably can't) | 21:38 |
| corvus | welp, i think the only idea i have now is to roll forward again and go test a bunch of etherpads | 21:38 |
| fungi | i mean, they have a fake browser javascript renderer engine they use for indexing js-only site content, but i have doubts it would handle something like etherpad | 21:38 |
| fungi | alternatively, create a new pad on the old server with poo in it, do a new db dump, import that... | 21:39 |
| fungi | but that's giong to eat up a ton of time | 21:39 |
| mordred | I think I'm with corvus here - I can't think of anything more specific to test | 21:40 |
| mordred | maybe before we do real quick - let's enable new etherpad and check that 3 byte page above | 21:40 |
| mordred | that way if we roll forward and it doesn't work we'll now if rolling bafck will improve things | 21:40 |
| clarkb | that sounds reasonable | 21:40 |
| corvus | heh, i found a convo from 2015, but it's about the clarkb-test etherpad | 21:40 |
| mordred | hahaha | 21:41 |
| fungi | i'm in favor of just cranking it back up, and then maybe data mining the mysqldump we imported to try and identify pads with 4-byte characters in them | 21:41 |
| * mordred starting real quick | 21:41 | |
| fungi | at this point we don't have any evidence that it's actually broken | 21:41 |
| mordred | https://etherpad.opendev.org/p/manila-shanghai-ptg-planning works | 21:41 |
| clarkb | agreed | 21:41 |
| clarkb | *agreed, that etherpad works | 21:41 |
| mordred | so at least we;re ok with the most commonest cases | 21:41 |
| mordred | I think at this point shut back down, re-roll forward - double check that one as a smoke test | 21:42 |
| corvus | ++ | 21:42 |
| mordred | and in the meantime we can try mining for more data | 21:42 |
| mordred | k. I'm about to rollforward to this afternoon's new dump - everyone ok with that? | 21:43 |
| corvus | yep | 21:43 |
| corvus | mordred: oh | 21:43 |
| corvus | mordred: what settings will you use to restore? | 21:43 |
| mordred | the current ones - they're the ones we used to dump | 21:43 |
| corvus | sounds good | 21:44 |
| fungi | go for it | 21:45 |
| clarkb | I got no better ideas | 21:47 |
| corvus | i'll stop the old etherpad server now | 21:48 |
| *** diablo_rojo has quit IRC | 21:54 | |
| clarkb | I need to get a bike ride in if I'm gonna get one in today. Back in a bit | 22:01 |
| mordred | have fun! | 22:03 |
| corvus | done | 22:09 |
| mordred | I agree | 22:11 |
| mordred | shall we start er up? | 22:11 |
| corvus | yep | 22:11 |
| mordred | ok. the shanghai pad still works | 22:11 |
| corvus | agreed | 22:12 |
| mordred | I'm going to move both of the db dumps to /opt/db wich is on the ephemeral - so that we don't keep low disk there | 22:13 |
| corvus | kk | 22:13 |
| corvus | i'll go through a bunch of the U ptg etherpad lists | 22:13 |
| mordred | I also just did a quick double-check and I'm comfortable that we have enough space on / for the db backups cron | 22:14 |
| corvus | everything A-M looks good | 22:16 |
| corvus | continuning | 22:16 |
| corvus | and cinder had a lot of non-roman chars | 22:16 |
| mordred | good for cinder! | 22:17 |
| corvus | ok, i looked at every pad on https://wiki.openstack.org/wiki/PTG/Ussuri/Etherpads and they all load ok | 22:17 |
| corvus | i think we can just skip sending the email and send a status notice saying "all good" | 22:18 |
| corvus | fungi: are you around? | 22:18 |
| fungi | yep | 22:20 |
| fungi | sounds good to me | 22:20 |
| fungi | i'm working out how to maybe identify '💩' (b'\xf0\x9f\x92\xa9') in a mysqldump | 22:21 |
| corvus | status notice Maintenance on etherpad.opendev.org is complete and the service is available again | 22:23 |
| corvus | mordred: ^ should we go ahead and send that? | 22:23 |
| fungi | that looks good to me | 22:23 |
| mordred | ++ | 22:23 |
| corvus | #status notice Maintenance on etherpad.opendev.org is complete and the service is available again | 22:23 |
| openstackstatus | corvus: sending notice | 22:23 |
| -openstackstatus- NOTICE: Maintenance on etherpad.opendev.org is complete and the service is available again | 22:23 | |
| mordred | fungi: I'm attempting to grep for 💩 | 22:24 |
| mordred | no clue if it'l work that way | 22:24 |
| mordred | it did not find any :) | 22:25 |
| openstackstatus | corvus: finished sending notice | 22:27 |
| fungi | out of curiosity, does anybody see how to set content styles in the etherpad ui now? | 22:28 |
| fungi | used to have a drop-down for heading levels along with a "code" style | 22:28 |
| fungi | i wonder if that got replaced by functionality in the new skins | 22:28 |
| mordred | nope - I do not see that | 22:29 |
| fungi | any any rate, i've created https://etherpad.opendev.org/p/tANNPB4DN0J936odiiBj with 3 and 4 byte characters | 22:32 |
| corvus | looks right to me | 22:33 |
| corvus | i think the loss of style dropdown is disappointing; i used that a lot | 22:34 |
| mordred | yeah - I think it's worth investigating to see if we can add it back somehow | 22:34 |
| fungi | as did i. i suspect they decided it was incompatible with line numbering in the default theme | 22:34 |
| fungi | we may just need a theme which is not as overblown as the colibris one they standardized on as their recommended default | 22:35 |
| fungi | mordred: i don't think "rechecj" is going to do anything ;) | 22:37 |
| mordred | it should know what I mean | 22:37 |
| mordred | fungi: I was trying to find a plugin for the heading styles ... but wow the plugins page is like o_O | 22:43 |
| mordred | fungi: maybe https://www.npmjs.com/package/ep_headings_css ? | 22:43 |
| fungi | now may also be a bad time to settle on plugins, since 1.8.1 (or .3?) is expected to break a bunch of them | 22:43 |
| mordred | neat! | 22:44 |
| mordred | fungi: we could restart with skinName: noSkin | 22:49 |
| mordred | fungi: we could restart with skinName: no-skin | 22:49 |
| mordred | we're not configuring a skin right now | 22:49 |
| mordred | oh - actually colibris isn't supposed to be default until 2.0 | 22:50 |
| mordred | so I think we're still using no-skin | 22:50 |
| fungi | we're on no-skin now, yes | 22:51 |
| mordred | fungi: colibris seems to have heading selector: https://pad.colibris-outilslibres.org/p/YAPAkpKBfh | 22:51 |
| fungi | colibris is what you get if you use their recommended configuration but we overwrite it with a config which doesn't specify a skin | 22:52 |
| fungi | so the in-code fallback is to no-skin | 22:52 |
| fungi | and yeah, i demoed the clibris skin on etherpad-dev briefly but everyone (me included) was, like, "eww" | 22:53 |
| *** hashar has joined #opendev | 22:58 | |
| mordred | nod. I think I missed that. I think it looks nice - but I'm likely weird | 23:01 |
| mordred | I can't find anything in the current source tree that would implement the style dropdown | 23:01 |
| fungi | yeah, the colibris theme seems to me like it's designed to satisfy people who want etherpad to be gdocs | 23:02 |
| fungi | so it has page margins and edges and all that | 23:02 |
| mordred | although I can't find anything in 1.7.0 that does either | 23:02 |
| fungi | and wastes a lot of space in the browser as a result | 23:02 |
| mordred | I do not know why old etherpad had that menu | 23:04 |
| fungi | could we have been using a plugin and forgotten?!? | 23:04 |
| mordred | aha! | 23:05 |
| mordred | ep_headings | 23:05 |
| mordred | is in node_modules | 23:05 |
| mordred | etherpad_lite::plugin { 'ep_headings': | 23:06 |
| mordred | k. now let me learn how to enable that | 23:06 |
| mordred | wow. it's just an npm install | 23:07 |
| mordred | well - we can have the headings dropdown back - but we will need to build out own image | 23:07 |
| mordred | so let's maybe wait for 1.8.3 and see what breaks before we dothat? | 23:07 |
| fungi | yeah | 23:07 |
| fungi | i wonder how existing pads which had headings set in them are rendering now | 23:08 |
| mordred | fungi: soooooooo | 23:11 |
| mordred | fungi: game to try something completely silly? | 23:11 |
| fungi | sure | 23:11 |
| mordred | we can bind-mount the plugin in | 23:11 |
| mordred | and in theory it should "just work" | 23:11 |
| fungi | that sounds way preferable to building new images ourselves | 23:12 |
| *** hashar has quit IRC | 23:13 | |
| mordred | fungi: I have put a copy of the ep_headings module into /etc/etherpad/node_modules | 23:14 |
| mordred | fungi: and in the root screen you can see the line I added to docker-compose.yaml | 23:14 |
| mordred | we can restart etherpad real quick to see if that works - and if it does I'll work up a change to do it properly | 23:15 |
| fungi | jumping back into that root screen now | 23:16 |
| fungi | ahh, yep | 23:17 |
| mordred | (bottom line) | 23:17 |
| fungi | up for it | 23:17 |
| mordred | k. giving it a go | 23:17 |
| fungi | service unabailable | 23:17 |
| fungi | unavailable | 23:17 |
| fungi | so maybe not | 23:18 |
| mordred | yeah. it's a permissions thing ... | 23:19 |
| mordred | there it is | 23:20 |
| mordred | k. we've learned how to fake this :) | 23:20 |
| *** tosky has quit IRC | 23:20 | |
| * mordred is going to leave etherpad.opendev.org in the emergency list until that's done in ansible | 23:21 | |
| fungi | it works! | 23:21 |
| mordred | so - basic things I did - on my laptop in a random directory I ran "npm install ep_headings" - I tarred up the node_modules dir that produced, copied it over, untarred it in /etc/etherpad - and then touched /etc/etherpad/node_modules/ep_headings/.ep_initialized and chmodded is for write | 23:22 |
| fungi | neat | 23:23 |
| fungi | so if we decide to add other plugins, that's basically how we'd go about it? | 23:23 |
| mordred | yeah | 23:23 |
| fungi | here's hoping they don't break that one in the next release, at least | 23:24 |
| mordred | yeah | 23:24 |
| *** DSpider has quit IRC | 23:26 | |
| clarkb | mordred: fungi are we giong to ansiblify that? | 23:43 |
| mordred | yup | 23:45 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Install ep_headings module https://review.opendev.org/719123 | 23:51 |
| mordred | clarkb, fungi :^^ | 23:51 |
| clarkb | +2 | 23:52 |
| openstackgerrit | Merged opendev/system-config master: Set env vars pointing to correct file locations https://review.opendev.org/719053 | 23:57 |
| clarkb | oh it just occured to me is ^ safe with the current running gerrit? | 23:58 |
| clarkb | mordred: ^ | 23:58 |
| clarkb | review is not in the emergency file so that may break us? | 23:59 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!