| Unit193 | Has anyone noticed that when the pastebin was moved, https://paste.opendev.org/json/?method=pastes.newPaste started becoming an error? | 01:12 |
|---|---|---|
| fungi | Unit193: i hadn't noticed, but what did that method do previously? | 01:15 |
| Unit193 | fungi: That's the API to post a paste, specifically how pastebinit pastes stuff. | 01:16 |
| fungi | huh, i thought i'd used it since then... checking | 01:16 |
| fungi | indeed, pastebinit is giving me "Failed to contact the server: HTTP Error 502: Proxy Error" | 01:17 |
| fungi | that's no good... i wonder if something changed in our proxying config | 01:17 |
| Unit193 | Yep, openstack has been my default for years in pastebinit. :3 Also, if you wonder why `pastebinit file.ext` doesn't work, it's literally a one line fix. At about line 353 change 'with open(filename, "rb")' to 'with open(filename, "r")' :P | 01:20 |
| fungi | yeah, i've similarly debugged and locally patched my copy of pastebinit long ago when switching to python3 | 01:21 |
| fungi | ianw added an explicit filter for the getRecent method last week in https://review.opendev.org/803418 but we're generally proxying everything else | 01:21 |
| ianw | it is likely something to do with lodgeit on the backend | 01:22 |
| fungi | maybe | 01:23 |
| fungi | we're proxying basically everything via https://opendev.org/opendev/system-config/src/branch/master/playbooks/roles/lodgeit/templates/paste.vhost.j2#L37-L46 | 01:23 |
| ianw | yeah, the proxy error would be when lodgeit returns an exception i bet | 01:23 |
| Unit193 | BTW, thank you very much for hosting a great lodgeit instance. | 01:23 |
| fungi | yep, i'm digging in logs now | 01:23 |
| fungi | AH00898: Error reading from remote server returned by /json/ | 01:25 |
| fungi | so as ianw guessed, apache is getting a bad response from the backend | 01:25 |
| ianw | yeah, check in /var/log/containers/ for the lodgeit container logs | 01:25 |
| * ianw lays a bet on unicode strings being involved | 01:26 | |
| fungi | docker-compose says it should log to syslog, but it doesn't seem to that i can tell (or it's not logging the api error) | 01:28 |
| fungi | ahh, its streaming to stdout not syslog | 01:28 |
| fungi | i should have told it to only show me recent log entries :/ | 01:28 |
| ianw | /var/log/containers/docker-lodgeit.log should be capturing it ... | 01:29 |
| ianw | Aug 12 01:27:41 paste01 docker-lodgeit[647]: File "/usr/local/lib/python3.7/site-packages/lodgeit/lib/json.py", line 45, in handle_request | 01:29 |
| ianw | Aug 12 01:27:41 paste01 docker-lodgeit[647]: response = {'data': None, 'error': str(e).decode('utf-8')} | 01:29 |
| ianw | Aug 12 01:27:41 paste01 docker-lodgeit[647]: AttributeError: 'str' object has no attribute 'decode' | 01:29 |
| fungi | File "/usr/local/lib/python3.7/site-packages/lodgeit/lib/json.py", line 45, in handle_request | 01:30 |
| fungi | yeah, you beat me to it | 01:30 |
| fungi | you were close!~ | 01:30 |
| fungi | string handling/decoding definitely involved anyway | 01:30 |
| fungi | i agree that's entirely wrong. should we just drop the .decode() or drop the str() recast? | 01:35 |
| fungi | except Exception as e should already have e as a string | 01:36 |
| fungi | i think we drop both the str() and the .decode() | 01:36 |
| fungi | oh! | 01:37 |
| fungi | it was supposed to be .encode() | 01:37 |
| fungi | this is getting stuffed back into the json response | 01:37 |
| fungi | ianw: oh, i should have read back farther in the log. that was a secondary exception. the real issue is "TypeError: pastes_new_paste() missing 2 required positional argumen | 01:44 |
| fungi | ts: 'language' and 'code'" | 01:44 |
| fungi | so we have two bugs. one to do with how *args is being passed (i think?) and one with how exceptions are being serialized into the response | 01:46 |
| fungi | the method handler is a bit spaghetti-like, i don't think i'm going to manage to debug it this late at night | 02:10 |
| fungi | going to need to dump local.request.data to examine i think | 02:11 |
| *** ykarel|away is now known as ykarel | 04:57 | |
| *** rpittau|afk is now known as rpittau | 06:22 | |
| *** dpawlik6 is now known as dpawlik | 07:07 | |
| *** jpena|off is now known as jpena | 07:33 | |
| *** elodilles_pto is now known as elodilles | 08:12 | |
| *** ykarel is now known as ykarel|lunch | 08:27 | |
| opendevreview | Ananya proposed opendev/elastic-recheck rdo: Make elastic recheck compatible with rdo elasticsearch https://review.opendev.org/c/opendev/elastic-recheck/+/803897 | 08:50 |
| *** sshnaidm|afk is now known as sshnaidm | 08:55 | |
| *** ykarel|lunch is now known as ykarel | 09:46 | |
| opendevreview | Ananya proposed opendev/elastic-recheck rdo: Make elastic recheck compatible with rdo elasticsearch https://review.opendev.org/c/opendev/elastic-recheck/+/803897 | 09:49 |
| *** dviroel|ruck|out is now known as dviroel|ruck | 11:24 | |
| *** jpena is now known as jpena|lunch | 11:33 | |
| fungi | returning to the lodgeit situation, so far i've been able to determine that when sending via pastebinit, local.request.data is coming through as an empty string | 12:12 |
| fungi | which is basically an alias for werkzeug.LocalProxy(ctx, 'request').data | 12:14 |
| fungi | unfortunately, my familiarity with werkzeug is effectively nonexistent | 12:15 |
| fungi | noonedeadpunk: any chance you've run into this and/or figured out the cause? | 12:15 |
| *** jpena|lunch is now known as jpena | 12:24 | |
| *** jpena is now known as jpena|off | 12:33 | |
| *** ysandeep|PTO is now known as ysandeep | 13:26 | |
| opendevreview | Ananya proposed opendev/elastic-recheck rdo: Make elastic recheck compatible with rdo elasticsearch https://review.opendev.org/c/opendev/elastic-recheck/+/803897 | 13:27 |
| opendevreview | Ananya proposed opendev/elastic-recheck rdo: Make elastic recheck compatible with rdo elasticsearch https://review.opendev.org/c/opendev/elastic-recheck/+/803897 | 13:37 |
| opendevreview | Ananya proposed opendev/elastic-recheck rdo: Make elastic recheck compatible with rdo elasticsearch https://review.opendev.org/c/opendev/elastic-recheck/+/803897 | 13:43 |
| fungi | starting to wonder if werkzeug doesn't werk with python 3.7 | 14:17 |
| fungi | rather, the ancient werkzeug 0.15 we pin to | 14:17 |
| fungi | 0.15.6 (2019-09-04) does claim it supports 3.7: https://pypi.org/project/Werkzeug/0.15.6/ | 14:20 |
| *** dmellado_ is now known as dmellado | 14:24 | |
| fungi | looks like the old paste server got deleted, so i'm not sure if i can figure out what versions of deps were installed on it | 14:33 |
| *** jpena|off is now known as jpena | 14:43 | |
| *** diablo_rojo__ is now known as diablo_rojo | 14:54 | |
| clarkb | fungi: did we convert to python3 on the old server? I thought we had but maybe that never happened | 15:14 |
| clarkb | also re etherpad, sleeping on it my brain woke up telling me to check the settings template and there are a few settings I want to check on the test instance. I'll restart an update there, then we can make the updates if necessary to the change and hold a new test instance to double check it | 15:15 |
| fungi | i don't think we did, but if we did that'll be reflected in puppet-lodgeit i guess | 15:15 |
| *** rpittau is now known as rpittau|afk | 15:31 | |
| opendevreview | Clark Boylan proposed opendev/system-config master: Upgrade etherpad to 1.8.14 https://review.opendev.org/c/opendev/system-config/+/804136 | 15:32 |
| opendevreview | Clark Boylan proposed opendev/system-config master: DNM force etherpad failure to hold the test nodes https://review.opendev.org/c/opendev/system-config/+/804138 | 15:32 |
| clarkb | I'll delete the current hold and set up a hold for ^ then if that still looks good I think we can consider landing this change | 15:32 |
| clarkb | mnaser: do you still need your loci keystone autohold? Doesn't look like it ever caught a failure | 15:33 |
| *** jpena is now known as jpena|off | 15:34 | |
| clarkb | while waiting for those jobs to run I'm going to find breakfast and water the garden. Back in a bit | 15:34 |
| clarkb | fungi: if you review that update ^ you may want to read through the upstream example settings.json to make sure we aren't missing anything you think is important | 15:35 |
| fungi | sure, and good call, this bit us in the past around the default "theme" change which completely redid how the interface works | 15:37 |
| *** marios is now known as marios|out | 16:08 | |
| clarkb | fungi: 23.253.160.131 etherpad.opendev.org and https://etherpad.opendev.org/p/clarkb-test if you want to test | 16:26 |
| clarkb | still looks good to me | 16:26 |
| *** ykarel is now known as ykarel|away | 16:31 | |
| roman_g | Good morning, clarkb fungi. City Network has resolved Glance issues on KNA1. Please, revert https://review.opendev.org/c/openstack/project-config/+/804281 and proceed with image uploads and what else is needed. | 16:37 |
| clarkb | roman_g: reverting that change should be all we need to do to start using it again. I'll push that up momentarily | 16:38 |
| roman_g | Great. Thank you. | 16:38 |
| fungi | roman_g: so they concluded that their problem was not us after all? ;) | 16:38 |
| roman_g | fungi of course. | 16:40 |
| opendevreview | Clark Boylan proposed openstack/project-config master: Revert "Disable airship citylcoud nodepool provider" https://review.opendev.org/c/openstack/project-config/+/804417 | 16:40 |
| roman_g | Your Zuul is one of the best canary services public cloud provider could use for free. | 16:41 |
| fungi | to be fair, it's very resource-intensive for a canary ;) | 16:41 |
| roman_g | It's more useful than fully artificial tempest workloads, for example. Doing good for community and testing infra reliability at the same time - couldn't imagine better options. | 16:44 |
| roman_g | Participating in SETI@Home or something similar does not produce this type of workloads. | 16:45 |
| roman_g | All right, I'm going home. Have a good day/evening, see you later. Big warm Thank you, fungi & clarkb. | 16:47 |
| clarkb | yup, it is one of the things we try to explain as a benefit for contributing resources. We're a flexible user that can shake out problems and potentially help fix them if we need to tie back into the upstream problems | 16:47 |
| opendevreview | Merged openstack/project-config master: Revert "Disable airship citylcoud nodepool provider" https://review.opendev.org/c/openstack/project-config/+/804417 | 17:01 |
| clarkb | fungi: let me know how if you find anything with the etherpad test server. But if not I think we should probably try and land that chagne today and upgrade? | 17:03 |
| clarkb | infra-root ^ if you think that is a bad idea for some reason let us know. I think the risk here is largely that we might break the jitsi meet integration which is harder to test pre merge | 17:04 |
| fungi | yeah, sorry, getting distracted by other discussions but testing now | 17:04 |
| clarkb | we have until october for the ptg which will hopefully be enough time to correct any of those issues | 17:04 |
| fungi | clarkb: seems to be working well for me | 17:07 |
| fungi | the other thing we'll want to test is meetpad, but i expect we need to upgrade production etherpad to really do that properly | 17:07 |
| clarkb | yes unfortauntely due to how the proxying there works | 17:07 |
| clarkb | I don't think we can hijack it the way we can with a local etc/hosts update | 17:08 |
| clarkb | since we run a proxy to etherpad on the meetpad host that jitsi meet talks to | 17:08 |
| clarkb | if we talked directly to etherpad then the local /etc/hosts override would work | 17:08 |
| clarkb | if anyone has good ideas for testing that prior to landing the change I'm open to ideas :) | 17:10 |
| fungi | clarkb: those patchfiles are reapplied to the corresponding upstream files, i take it? hard to know what i'm reviewing in those diffs | 17:10 |
| clarkb | I guess we could run a test jitsi meet and then hack it up? | 17:10 |
| clarkb | fungi: the patchfiles are all deleted because newer etherpad fixed the problem they addressed | 17:11 |
| clarkb | fungi: previously that was basically us backporting a fix because upstream hadn't done a release yet | 17:11 |
| clarkb | they came from upstream but we didn't have a release | 17:11 |
| clarkb | specifically it fixed the overlapping of colors with text iirc | 17:12 |
| clarkb | yup git log seems to confirm | 17:13 |
| clarkb | "We also make a small change to the css in the editor so that each line has its text fully rendered. See https://github.com/ether/etherpad-lite/issues/4106 for more details." | 17:13 |
| clarkb | oh and another fix was for "Patch etherpad console logging to fix cross origin error" | 17:14 |
| fungi | oh, indeed, i missed that the patchfiles were all deletions | 17:14 |
| corvus | clarkb, fungi: can you allow through https://review.opendev.org/804255 (and https://review.opendev.org/804256 ) so that, aiui, tristanC can confirm that the matrix gerritbot disconnection problem is resolved? | 17:14 |
| fungi | because i was looking at diffs, in a diff view, and my brain read the diffs which were being removed as the actual git diffs | 17:14 |
| clarkb | corvus: why not remove the line in https://review.opendev.org/c/opendev/system-config/+/804255/2/inventory/service/groups.yaml ? | 17:16 |
| clarkb | I'll apprive it as is with a note to consider deleting that line in a followup if there wasn't an explicit reason for that | 17:16 |
| fungi | clarkb: it did remove the line, and replaced it with a comment explaining why we don't need to put eavesdrop in that group | 17:18 |
| clarkb | for https://review.opendev.org/c/opendev/system-config/+/804256 the iptables role has an explicit entry in rules.v4.j2 and rules.v6.j2 for port 22. That is what we mean by hard coded ya? And as such thee is no concern with approving that change? | 17:18 |
| fungi | i saw that as a reminder to our future selves that we shouldn't think it's missing unintentionally and re-add it | 17:18 |
| clarkb | fungi: gotcha | 17:18 |
| corvus | clarkb: i left that as a comment so that people ... yes what fungi said | 17:18 |
| corvus | i could see us thinking "but eavesdrop is a webserver!" | 17:19 |
| clarkb | I want ot make sure we're all in agreement on the port 22 thing because I approve it and potentially remove ssh access | 17:20 |
| clarkb | s/because/before/ | 17:20 |
| corvus | clarkb: that is my understanding and reasoning for the change and i agree extra caution is warranted. | 17:20 |
| corvus | (this is exactly why i wish it hadn't been added in the first place and think we should nip it in the bud) | 17:21 |
| clarkb | ++ just want to double check first | 17:21 |
| clarkb | fungi: ^ is that your understanding? and if so should I approve it now? | 17:21 |
| fungi | yes, that is what i understood when i reviewed it | 17:21 |
| corvus | i also believe that testinfa checks this | 17:22 |
| fungi | we always add ssh to all servers, so removing it from the webservers group should have no adverse impact | 17:22 |
| fungi | we have other server groups which do not include ssh in them, and we still have ssh access to those servers | 17:22 |
| clarkb | alright approving now | 17:23 |
| clarkb | and ya those extra ports get iterated over a few lines below the hard coded port 22 entry | 17:24 |
| clarkb | so should be fully redundant | 17:24 |
| clarkb | and doing an iptables -L -n on eavesdrop01 shows two entries for port 22 | 17:25 |
| corvus | clarkb: maybe an extra safety would be to do an ad-hoc iptables -L -n on the webservers group | 17:27 |
| clarkb | corvus: ya that is what I did roughly on one member of that group and it does show two entries | 17:28 |
| clarkb | removing one should still leave us with the other | 17:28 |
| corvus | clarkb: i just did it on the whole group, and they always appear in pairs | 17:29 |
| corvus | eavesdrop01.openstack.org and wiki-dev01.openstack.org are unreachable | 17:29 |
| clarkb | I think that is expected for eavesdrop01.openstack.org, not sure about wiki-dev01 | 17:29 |
| clarkb | fungi: ^ | 17:29 |
| fungi | eavesdrop01.opendev.org is the server in the inventory | 17:30 |
| opendevreview | James E. Blair proposed opendev/system-config master: Remove 2 servers that don't exist https://review.opendev.org/c/opendev/system-config/+/804420 | 17:30 |
| fungi | the current wiki-dev is wiki-dev03.openstack.org | 17:30 |
| corvus | clarkb, fungi: ^ then it sounds like that change is desirable | 17:30 |
| fungi | thanks | 17:30 |
| clarkb | I think eavesdrop01.openstack.org may still exist in a shutdown state as part of queued up cleanup? ianw would know what the state is on those | 17:31 |
| fungi | but yeah, removal from the inventory is fine, it may be that there's a change out there to do that for eavesdrop already, but i clearly forgot to do one for wiki-dev01 | 17:31 |
| opendevreview | Merged opendev/system-config master: Test port 9001 on eavesdrop https://review.opendev.org/c/opendev/system-config/+/804255 | 17:52 |
| opendevreview | Merged opendev/system-config master: Remove port 22 from webservers extra ports https://review.opendev.org/c/opendev/system-config/+/804256 | 17:52 |
| clarkb | corvus: ^ | 17:53 |
| *** sshnaidm is now known as sshnaidm|afk | 17:55 | |
| clarkb | infra-root do we want to proceed with https://review.opendev.org/c/opendev/system-config/+/804136 knowing that meetpad testing is a gap we have? or try to figure out testing for meetpad first? | 18:04 |
| fungi | i'm good with going ahead, but hoped we could get at least one more core reviewer to weigh in | 18:04 |
| clarkb | wfm | 18:05 |
| fungi | clarkb: maybe unless ianw thinks he has time to review the etherpad upgrade (804136) during his morning, we can just go ahead and approve/deploy it and then test meetpad before it gets too late in your day | 19:54 |
| clarkb | fungi: wfm | 19:54 |
| clarkb | I think it may still be quite early in ianw's part of the world so may have to wait another coupel of hours before we hear back. I guess worst case we can land the change tomorrow? | 20:00 |
| fungi | i'm also around for that | 20:00 |
| clarkb | fungi: I'm trying to think of other tools that might be hit by sqlalchemy 2.0 in opendev land | 20:41 |
| clarkb | jeepyb dropped its sql stuff | 20:41 |
| clarkb | storyboard? | 20:41 |
| clarkb | I suppose for tools like that pinning isn't the end of the world either | 20:42 |
| clarkb | keeping up to date is likely far more important for zuul | 20:43 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Update our project rename docs https://review.opendev.org/c/opendev/system-config/+/804460 | 20:53 |
| clarkb | That is a first pass at updating the rename docs. I think we can land that pretty safely. It doesn't include steps about splitting up the changes because that is more complicated and can happen in a followup | 20:54 |
| clarkb | I think we'ev leaked ansible processes on bridge again | 20:57 |
| clarkb | I noticed because https://review.opendev.org/c/opendev/system-config/+/804255 timed out a couple of jobs. I don't think those timeouts will impact what it is trying to accmplish though | 20:57 |
| clarkb | Once bridge is more idle we should be able to safely clean up the processes there | 20:58 |
| fungi | yeah, i expect to just pin sb and/or jeepyb and/or lodgeit if sqlav2 becomes a problem | 21:48 |
| clarkb | corvus: the infra-prod-service-eavesdrop job is running now | 21:48 |
| clarkb | for the first change | 21:49 |
| corvus | wow that seems like yesterday | 21:50 |
| *** dviroel|ruck is now known as dviroel|ruck|out | 21:51 | |
| fungi | wait, that wasn't yesterday? | 21:54 |
| * fungi has lost his perception of time | 21:54 | |
| ianw | sorry, hi | 22:05 |
| fungi | sorry for what? also, hiii! | 22:06 |
| ianw | i fiddled with the etherpad, just reading the changes now | 22:07 |
| clarkb | I should apologize for thinking of new things to check on etherpad but only after having slept on it :) | 22:07 |
| clarkb | ianw: in addition to the new settings changes the other thing to think about is if we are worried enough about meetpad to not just land that change go for it | 22:08 |
| ianw | only things of interest seem to be a warning | 22:17 |
| ianw | [2021-08-12 16:08:59.867] [WARN] console - Declaring the sessionKey in the settings.json is deprecated. | 22:17 |
| fungi | mmm | 22:18 |
| ianw | not sure if that's important. and it doesn't seem to wait for mariadb to start; we could use the wait-for-it script if we cared | 22:18 |
| ianw | seems like it doesn't have issues retrying | 22:18 |
| ianw | (unlike lodgeit ...) | 22:19 |
| ianw | speaking of ... python3 was new to the container, it was definitely python2 on the old host | 22:19 |
| ianw | i did try updating werkzeug but it's a dependency loop | 22:20 |
| ianw | they kicked some session thing out into a new library; that library seems abanonded and so now doesn't work with the latest werkzeug | 22:20 |
| ianw | that was where i left it. | 22:21 |
| clarkb | https://github.com/ether/etherpad-lite/tree/1.8.14#tweak-the-settings looks like it wants a SESSIONKEY.txt? | 22:25 |
| opendevreview | Ian Wienand proposed opendev/system-config master: etherpad: remove session key https://review.opendev.org/c/opendev/system-config/+/804466 | 22:25 |
| clarkb | also I feel like I look this up every time I interact with it but APIKEY is generated on startup for us and random? | 22:26 |
| fungi | ianw: do you have any gut feeling for why the request data obtained from werkzeug would be an empty bytestring in those api calls? | 22:28 |
| clarkb | ya that seems to be the case for apikey | 22:28 |
| clarkb | ianw: your change lgtm and I think it is safe for old or new etherpad | 22:29 |
| clarkb | ha nice the hourly deploy jobs have the semaphore so the extra port 22 cleanup is another hour away from running :/ | 22:30 |
| fungi | the deploy for 804255 timed out on manage-projects and puppet-else builds | 22:30 |
| clarkb | ianw: fungi: I guess we'll land the etherpad upgrade change first thing tomorrow so that it applies early enough in the day for testing | 22:30 |
| clarkb | fungi: yes I made note of that above we seem to have leaked ansible processes on bridge which I think is making it slowish | 22:30 |
| clarkb | but I dno't want to clean those up until bridge is idle | 22:31 |
| fungi | ahh, thanks | 22:31 |
| fungi | and yeah, tomorrow for etherpad wfm | 22:31 |
| ianw | https://github.com/ether/etherpad-lite/commit/a3ed82ad0622aaed0777e63280a2487322c49bad is the original change | 22:31 |
| clarkb | out of fear for stopping actual ansible processes that are doing work | 22:31 |
| clarkb | ianw: oh actually one issues with your change is that if we down then up the containers we'll get a new session key value | 22:31 |
| * ianw puts this in my list of "terrible commit messages that tell you the obvious thing but are useless on why they are making the change" | 22:31 | |
| clarkb | and will that cause everyone to need to reconnect? | 22:31 |
| clarkb | maybe we are ok with that. Since it would only be when we restart the service. If the client side disconnects the random value is fine | 22:32 |
| fungi | speaking of reconnecting, i noticed the error messages improved when you open the same pad in a second browser window/tab | 22:33 |
| fungi | it now actually pops up an error in the first one saying "you've opened a new copy" or something to that effect | 22:33 |
| clarkb | oh nice | 22:33 |
| clarkb | fungi: re the upgrade I'll try to get an early start so that I can approve the change then find breakfast while zuul chews on it | 22:33 |
| fungi | hopefully fewer confused reports from users now | 22:33 |
| fungi | i have no other scheduled obligations tomorrow so will be on hand as soon as you're ready | 22:34 |
| ianw | looking at that code, it seems unlikely the override would ever go away; only if they removed that if (!exports.sessionKey) | 22:34 |
| fungi | i can even approve it earlier if you like so there's a head start on gating/deploy jobs | 22:34 |
| clarkb | fungi: up to you on that. I've pretty regularly been around by 1500 UTC but if I'm late you may have to deal with things a bit until I show up. Not that I have any plans tonight that would impact my ability to show up on time :) | 22:36 |
| clarkb | ianw: looking at that try except I think that may make bind mounting the file in tricky. We'd have to supply the secret in the file directly | 22:37 |
| clarkb | because giving it an empty file will not cause an exception I don't think | 22:37 |
| clarkb | and we can't bind mount the entire dir because that dir contains the installation :/ | 22:37 |
| ianw | hrm, docker-compose could write out a static value? | 22:40 |
| clarkb | ianw: ya we could have ansible write it out first and then bind mount that. Not too different than what we do today just in another file | 22:41 |
| clarkb | its just frustating because they seem to have done this for simplicity but then made it not any simply | 22:41 |
| clarkb | s/simply/simpler/ | 22:41 |
| ianw | i guess from their POV it's complicated to get the right length of key in the config file, and you have to explain and handle people putting in the wrong thing | 22:42 |
| clarkb | hrm ya that could be. And it could also be that 99% of the time we're fine with the generated value taht chagnes each restart | 22:42 |
| clarkb | since clients tend to have to reconnect in those cases anyway I think | 22:42 |
| ianw | it's maybe even good to require that, to avoid any weirdness | 22:43 |
| ianw | fungi: to your prior question; i didn't have any insights to the argument passing on lodgeit. i couldn't figure out a simple debug strategy; with limnoria i could jump int he container and edit the python files, and send a sighup and it would reload | 22:45 |
| ianw | but with this one it seemed to kill the container. so it was difficult to just edit in a quick debug statement to help trace things | 22:45 |
| clarkb | ianw: as a general heads up https://review.opendev.org/804256 was merged and it hasn't run its deploy jobs yet. fungi and corvus and I all tried to double check it and we're confident that won't accidentally break ssh. But I may be eating dinner when it finally starts to apply on systems | 22:45 |
| clarkb | fungi: https://review.opendev.org/c/opendev/system-config/+/804460 is the project renaming doc update | 22:47 |
| ianw | heh, ok :) not sure what the rollback strategy will be, other than to scream loudly :) | 22:47 |
| clarkb | fungi: it tries to handle everything but the split change for testing situation whcih I figure we can land in a followup if we manage to come up with a better process for that | 22:48 |
| clarkb | ianw: I think it is do recovery mounts and edit those files and reboot | 22:48 |
| clarkb | ianw: but we confirmed there were double port 22 rules on those webserver hosts and the rules templates hardcode in a port 22 rule so we don't rely on the lists of ports for ssh | 22:48 |
| mordred | ianw: I find screaming loudly to be the best strategy for most situations | 23:04 |
| fungi | ianw: oh, i have no problem editing the python scripts in the lodgeit container and stopping/starting the container | 23:15 |
| fungi | i'm just not sure what else to debug | 23:15 |
| fungi | (but that's how i confirmed the args list is an empty bytestring) | 23:16 |
| ianw | fungi: ahh, ok, was this local or a held node? | 23:16 |
| fungi | ianw: neither, i poked at the production server since i was initially trying to figure out if there was something strange about how it was running | 23:34 |
| ianw | ok, if i get a sec this afternoon i might try and put a test server on hold where i think we can fiddle as we want | 23:34 |
| fungi | yeah, at this point i'm fairly certain i can do exactly the same on a held node | 23:37 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!