| clarkb | I was trying to kill my ssh conncetion and usually when that happens I get at least one blank but in this case I don't think it even did that. Weird | 00:03 |
|---|---|---|
| opendevreview | Ian Wienand proposed opendev/system-config master: [wip] redirect pastebinit https://review.opendev.org/c/opendev/system-config/+/804918 | 00:09 |
| clarkb | oh I guess there is one space there | 00:10 |
| ianw | oh doh, using requests it's running on the bridge, no the remote host | 00:38 |
| opendevreview | Ian Wienand proposed opendev/system-config master: [wip] redirect pastebinit https://review.opendev.org/c/opendev/system-config/+/804918 | 00:44 |
| opendevreview | Gage Hugo proposed opendev/irc-meetings master: Update irc channel for security sig meetings https://review.opendev.org/c/opendev/irc-meetings/+/805093 | 01:21 |
| opendevreview | Gage Hugo proposed opendev/irc-meetings master: Update openstack-helm irc meeting channel https://review.opendev.org/c/opendev/irc-meetings/+/805094 | 01:23 |
| opendevreview | Takashi Kajinami proposed openstack/project-config master: Retire puppet-monasca - Step 1: End project Gating https://review.opendev.org/c/openstack/project-config/+/805098 | 01:59 |
| opendevreview | chzhang8 proposed openstack/project-config master: bring tricircle under x namespaces https://review.opendev.org/c/openstack/project-config/+/805102 | 02:25 |
| opendevreview | Takashi Kajinami proposed openstack/project-config master: Retire puppet-monasca - Step 3: Remove Project https://review.opendev.org/c/openstack/project-config/+/805103 | 02:32 |
| opendevreview | Ian Wienand proposed opendev/system-config master: [wip] redirect pastebinit https://review.opendev.org/c/opendev/system-config/+/804918 | 02:50 |
| opendevreview | Ian Wienand proposed opendev/system-config master: [wip] redirect pastebinit https://review.opendev.org/c/opendev/system-config/+/804918 | 03:35 |
| *** ysandeep|away is now known as ysandeep | 05:06 | |
| opendevreview | Ian Wienand proposed opendev/system-config master: Stop redirecting for the paste site https://review.opendev.org/c/opendev/system-config/+/804539 | 05:22 |
| *** rpittau|afk is now known as rpittau | 07:08 | |
| *** jpena|off is now known as jpena | 07:35 | |
| *** redrobot1 is now known as redrobot | 07:59 | |
| *** arxcruz is now known as arxcruz|off | 09:05 | |
| opendevreview | Sorin Sbârnea proposed zuul/zuul-jobs master: Include podman installation with molecule https://review.opendev.org/c/zuul/zuul-jobs/+/803471 | 10:39 |
| *** dviroel|ruck|out is now known as dviroel|ruck | 11:08 | |
| *** jpena is now known as jpena|lunch | 11:31 | |
| *** jpena|lunch is now known as jpena | 12:31 | |
| *** frenzy_friday is now known as erbot | 13:13 | |
| *** erbot is now known as frenzy_friday | 13:15 | |
| opendevreview | Clark Boylan proposed opendev/system-config master: DNM Test a gerrit 3.2 -> 3.3 upgrade https://review.opendev.org/c/opendev/system-config/+/805243 | 15:23 |
| *** jpena is now known as jpena|off | 15:41 | |
| *** ysandeep is now known as ysandeep|away | 15:42 | |
| *** rpittau is now known as rpittau|afk | 15:51 | |
| opendevreview | Merged openstack/project-config master: Retire puppet-monasca - Step 1: End project Gating https://review.opendev.org/c/openstack/project-config/+/805098 | 16:06 |
| *** sshnaidm is now known as sshnaidm|afk | 16:09 | |
| clarkb | wow I think that upgrade test worked on the first pass. I really didn't expect that so I didn't have a hold set for my assert False | 16:30 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Test a gerrit 3.2 -> 3.3 upgrade https://review.opendev.org/c/opendev/system-config/+/805243 | 16:47 |
| clarkb | if that works I think that would be mergable (though I've still got my assert False because I want ot hold a node) | 16:48 |
| clarkb | fungi: have you had a chance to test newlist on the test lists.kc.io yet? | 17:18 |
| fungi | oh, nope, i'll do that real quick | 17:19 |
| *** mtreinish_ is now known as mtreinish | 17:22 | |
| fungi | after enabling exim4 and mailman services, running newlist did send me a list admin notification e-mail which i received | 17:25 |
| clarkb | excellent | 17:25 |
| fungi | i'll add myself as a subscriber and then make sure manually injecting a message to the test list distributes it to me | 17:26 |
| clarkb | ++ that sounds great | 17:26 |
| clarkb | then I guess we confirm next week after people get back from vacation and can schedule an upgrade | 17:26 |
| fungi | yep, i was able to fake mua inject a message to testlist by telnetting to the smtp socket from my workstation, and received my post as a subscriber | 17:31 |
| fungi | you should also see it in the testlist archive now | 17:31 |
| clarkb | let me update /etc/hosts and take a look | 17:32 |
| clarkb | yup I see it and navigating to it via the web ui to the archive to the month to the thread all works too | 17:33 |
| clarkb | this looks great, thank you for doing that | 17:33 |
| fungi | np, it only took a couple minutes | 17:33 |
| fungi | glad to know my experience prank spoofing e-mails by telnet in high school can still be put to use decades later | 17:34 |
| clarkb | you even put a signature in the test message :) | 17:34 |
| fungi | helps with getting past ye olde spam filters | 17:34 |
| clarkb | fungi: I guess we shutdown exim and mailman to avoid an unexpected emails from them and can work on scheduling the upgrade now? | 17:34 |
| fungi | yep, on it | 17:35 |
| fungi | they're shut back down | 17:35 |
| clarkb | thanks again. I'm glad this seems to be working without too much intervention. | 17:36 |
| fungi | yeah, straight through to focal. very nice considering it's 4 years of package updates in one shot | 17:37 |
| fungi | infra-root: rackspace has opened a trouble ticket to let us know afs02.dfw.openstack.org needs a host migration performed by or on tomorrow | 17:42 |
| fungi | i can start working to manually initiate a migration now in order to avoid unexpected downtime for that server | 17:42 |
| clarkb | fungi: thanks. afs01.dfw runs the vos releases and afs02 shouldn't be primary for any volumes which means I think you can do that fairly safely without impact? | 17:44 |
| fungi | yeah, i'll follow https://docs.opendev.org/opendev/system-config/latest/afs.html#afs0x-openstack-org to be sure | 17:44 |
| fungi | confirmed via vos listvldb that afs02.dfw does not hold any rw volumes right now | 17:45 |
| fungi | going to `bos shutdown afs02.dfw.openstack.org` | 17:46 |
| fungi | and done. i'll poweroff the server for the duration of the host migration to increase our chances it goes quickly and doesn't hang/error on us | 17:48 |
| clarkb | ++ | 17:48 |
| fungi | and host migration is now in progress | 17:53 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Test a gerrit 3.2 -> 3.3 upgrade https://review.opendev.org/c/opendev/system-config/+/805243 | 17:56 |
| clarkb | that change should be mergable now. I've got my held node already | 17:56 |
| clarkb | two things I notice on the held node: we have sql connectivity errors, not sure if that is just a startup race or not yet. And we configure replication to the giteas on that node so we get a lot of replication errors | 17:57 |
| fungi | i've seen sql connectivity errors at docker-compose up in production as well | 17:58 |
| fungi | i think gerrit tries to connect to mariadb before the db is fully running | 17:59 |
| fungi | also see this in the git-review functional testsuite | 17:59 |
| fungi | seems gerrit is smart enough to retry until the db accepts the connection | 17:59 |
| clarkb | ok I can probably ignore that then | 18:00 |
| clarkb | I've made a copyof the 3.3 configs so that I can diff them against 3.2 configs to ensure there isn't anything new we need to write into those files | 18:01 |
| clarkb | but I'm abount to sort out the downgrade process | 18:01 |
| clarkb | I think I have successfully downgraded the test gerrit. The process is written down at https://etherpad.opendev.org/p/gerrit-3.3-upgrade-prep | 18:28 |
| fungi | neat, and good to test | 18:28 |
| clarkb | the refs/meta/version update is real git fu. I feel like I leveled up | 18:29 |
| fungi | heh | 18:30 |
| fungi | did you use git to do it, or just vi? | 18:30 |
| clarkb | you have to use git beacuse its a direct blob update and not a check out able thing | 18:31 |
| clarkb | I suspopse you could use vi if you figured out how to cache and write that objcet by hand but that would be rough | 18:31 |
| clarkb | The whole time I was paranoid that I might be affecting the prod server because we write out the hostname and everything as if it were prod | 18:32 |
| fungi | maybe we should do something in our jobs to mangle the shell prompt on test nodes | 18:33 |
| clarkb | Any ideas on how we might better indicate to users that they are on a test node? PS1 was my first thought but that gets overridden and set all over | 18:33 |
| clarkb | What I ended up doing was checking my IP address and memory count as those differ greatly to prod | 18:33 |
| fungi | #status log Completed manually-initiated host migration for afs02.dfw.openstack.org in preparation for upcoming provider maintenance | 18:34 |
| opendevstatus | fungi: finished logging | 18:34 |
| fungi | server seems to be back up and working normally again | 18:34 |
| fungi | i'll close out the ticket | 18:34 |
| clarkb | fungi: trying to get /etc/profile or /etc/bash.bashrc to update the PS1 it seems like the way debian does it is gives every users a .bashrc which makes that difficult? | 18:42 |
| clarkb | well difficult because the per user bashrc does set a PS1 | 18:42 |
| clarkb | if it wasn't setting a PS1 we could set it globally and then it wouldn't be overwritten | 18:43 |
| fungi | we could alter /etc/skel/.bashrc in our debuntu images, i guess | 18:49 |
| fungi | or early before user creation | 18:50 |
| clarkb | oh is that the template? | 18:50 |
| fungi | yes, /etc/skel gets copied as the initial homedir for each newly created user | 18:50 |
| fungi | that's typical of *bsd systems too i think, so maybe rh based would work that way as well | 18:51 |
| corvus | apparently npmjs.org is having a sad; it's taking out a lot of zuul gate jobs | 18:51 |
| fungi | though for our infra jobs, whatever works on ubuntu should be sufficient i suppose | 18:51 |
| corvus | (fyi) | 18:51 |
| fungi | corvus: ouch, thanks for the heads up | 18:51 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Test a gerrit 3.2 -> 3.3 upgrade https://review.opendev.org/c/opendev/system-config/+/805243 | 18:56 |
| clarkb | correction ^ that should be mergeable now | 18:56 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Test a gerrit 3.2 -> 3.3 upgrade https://review.opendev.org/c/opendev/system-config/+/805243 | 20:19 |
| clarkb | fungi: ^ updated to address your comment | 20:19 |
| *** dviroel|ruck is now known as dviroel|ruck|out | 21:08 | |
| clarkb | corvus: I'm going to pop out for a bit (bike ride) but happy to help with any zuul restarts we might want to do to pick up those fixes | 21:16 |
| clarkb | I assume the .1 release might happen either tomorrow or early next week? | 21:16 |
| corvus | clarkb: yeah, i'd like to restart this afternoon and release tomorrow | 21:17 |
| clarkb | don't feel like you have to wait for me either. But I'm happy to help when back | 21:17 |
| corvus | i'm digging into some thing i observed while debugging; i don't think they're critical, but i want to get to a good stapping place before restarting | 21:17 |
| corvus | so maybe after you get back sounds like good timing :) | 21:17 |
| fungi | i too am back from foodmaking and foodeating, so happy to help with any restart tasks/fallout | 21:52 |
| corvus | i'm done looking into the stuff i noticed this morning; i pushed up 2 more changes (see #zuul). i'm waffling on whether we should try to get those in too, or draw the line where we are now | 22:25 |
| corvus | i don't think they're urgent, but if we don't include them in this round, we should probably merge them immediately after and make another point release. i think i've just about talked myself into saying we should get them in now. :) | 22:26 |
| fungi | i've already +2'd one, i'm re-reading the other now because it was involved enough i wasn't sure i'd given it a thorough look the first time | 22:27 |
| fungi | but i'm also fine not including them | 22:27 |
| ianw | fungi: did you want to merge https://review.opendev.org/c/opendev/system-config/+/804539 to finalise the paste thing if you're happy | 22:32 |
| fungi | ianw: oh, yes thanks, i've been far too scattered this week, sorry | 22:33 |
| fungi | ianw: on a related note, not sure if you saw but https://review.opendev.org/804540 corrects a bug i found while trying to debug the pastebinit situation | 22:36 |
| ianw | fungi: so the response can't be a unicode string? | 22:38 |
| fungi | ianw: the response is unlikely to be unicode, but that's not the issue. the exception data object isn't utf-8 encoded so attempting to decode it raises an exception | 22:40 |
| fungi | however that codepath is only hit when returning an error, so the observable difference is that now it will return detailed error responses when intended, rather than a generic proxyerror | 22:41 |
| ianw | oh right, sorry i read the diff backwards! | 22:41 |
| ianw | i'm pretty sure you can also remove this line now you've fixed it -> https://opendev.org/opendev/system-config/src/branch/master/playbooks/roles/lodgeit/templates/paste.vhost.j2#L41 | 22:42 |
| fungi | oh was that it? still, i think blocking getrecent is probably a good hygiene measure | 22:46 |
| fungi | part of what got me trying to fix up testing for lodgeit is i want to invert the private paste toggle and make all pastes use unguessable urls | 22:46 |
| fungi | given that was essentially the source of the biggest security breach we've ever had to recover from | 22:47 |
| fungi | done properly, i'd just remove the getrecent method from the api, or make it configurable (though that sounds like more work) | 22:48 |
| opendevreview | Merged opendev/lodgeit master: Properly handle paste exceptions https://review.opendev.org/c/opendev/lodgeit/+/804540 | 22:51 |
| opendevreview | Merged opendev/system-config master: Stop redirecting for the paste site https://review.opendev.org/c/opendev/system-config/+/804539 | 23:05 |
| clarkb | Look at openstack's release schedule we are entering feature freeze and then RC periods over the next few weeks | 23:49 |
| clarkb | *looking | 23:49 |
| clarkb | Based on that I think it might be best to do big upgrades like the gerrit upgrade after Xena is out the door. Aim for the week of October 11-15 to do the gerrit 3.3 upgrade and possibly the lists.o.o uprade? | 23:50 |
| clarkb | I'm fairly confident in the gerrit 3.3 release at this point, but what I'm not so sure of is taht the release tooling won't have a sad against newer gerrit for unexpected reasons | 23:51 |
| clarkb | the gitea 1.15.0 seems to be held up by a couple of issues. Not sure when that will be available but hopefully soon. Happy to hold off on that update too unless something forces us to upgrade. We should be ready either way | 23:54 |
| fungi | makes sense | 23:58 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!