Thursday, 2021-08-19

clarkbI was trying to kill my ssh conncetion and usually when that happens I get at least one blank but in this case I don't think it even did that. Weird00:03
opendevreviewIan Wienand proposed opendev/system-config master: [wip] redirect pastebinit  https://review.opendev.org/c/opendev/system-config/+/80491800:09
clarkboh I guess there is one space there00:10
ianwoh doh, using requests it's running on the bridge, no the remote host00:38
opendevreviewIan Wienand proposed opendev/system-config master: [wip] redirect pastebinit  https://review.opendev.org/c/opendev/system-config/+/80491800:44
opendevreviewGage Hugo proposed opendev/irc-meetings master: Update irc channel for security sig meetings  https://review.opendev.org/c/opendev/irc-meetings/+/80509301:21
opendevreviewGage Hugo proposed opendev/irc-meetings master: Update openstack-helm irc meeting channel  https://review.opendev.org/c/opendev/irc-meetings/+/80509401:23
opendevreviewTakashi Kajinami proposed openstack/project-config master: Retire puppet-monasca - Step 1: End project Gating  https://review.opendev.org/c/openstack/project-config/+/80509801:59
opendevreviewchzhang8 proposed openstack/project-config master: bring tricircle under x namespaces  https://review.opendev.org/c/openstack/project-config/+/80510202:25
opendevreviewTakashi Kajinami proposed openstack/project-config master: Retire puppet-monasca - Step 3: Remove Project  https://review.opendev.org/c/openstack/project-config/+/80510302:32
opendevreviewIan Wienand proposed opendev/system-config master: [wip] redirect pastebinit  https://review.opendev.org/c/opendev/system-config/+/80491802:50
opendevreviewIan Wienand proposed opendev/system-config master: [wip] redirect pastebinit  https://review.opendev.org/c/opendev/system-config/+/80491803:35
*** ysandeep|away is now known as ysandeep05:06
opendevreviewIan Wienand proposed opendev/system-config master: Stop redirecting for the paste site  https://review.opendev.org/c/opendev/system-config/+/80453905:22
*** rpittau|afk is now known as rpittau07:08
*** jpena|off is now known as jpena07:35
*** redrobot1 is now known as redrobot07:59
*** arxcruz is now known as arxcruz|off09:05
opendevreviewSorin Sb├órnea proposed zuul/zuul-jobs master: Include podman installation with molecule  https://review.opendev.org/c/zuul/zuul-jobs/+/80347110:39
*** dviroel|ruck|out is now known as dviroel|ruck11:08
*** jpena is now known as jpena|lunch11:31
*** jpena|lunch is now known as jpena12:31
*** frenzy_friday is now known as erbot13:13
*** erbot is now known as frenzy_friday13:15
opendevreviewClark Boylan proposed opendev/system-config master: DNM Test a gerrit 3.2 -> 3.3 upgrade  https://review.opendev.org/c/opendev/system-config/+/80524315:23
*** jpena is now known as jpena|off15:41
*** ysandeep is now known as ysandeep|away15:42
*** rpittau is now known as rpittau|afk15:51
opendevreviewMerged openstack/project-config master: Retire puppet-monasca - Step 1: End project Gating  https://review.opendev.org/c/openstack/project-config/+/80509816:06
*** sshnaidm is now known as sshnaidm|afk16:09
clarkbwow I think that upgrade test worked on the first pass. I really didn't expect that so I didn't have a hold set for my assert False16:30
opendevreviewClark Boylan proposed opendev/system-config master: Test a gerrit 3.2 -> 3.3 upgrade  https://review.opendev.org/c/opendev/system-config/+/80524316:47
clarkbif that works I think that would be mergable (though I've still got my assert False because I want ot hold a node)16:48
clarkbfungi: have you had a chance to test newlist on the test lists.kc.io yet?17:18
fungioh, nope, i'll do that real quick17:19
*** mtreinish_ is now known as mtreinish17:22
fungiafter enabling exim4 and mailman services, running newlist did send me a list admin notification e-mail which i received17:25
clarkbexcellent17:25
fungii'll add myself as a subscriber and then make sure manually injecting a message to the test list distributes it to me17:26
clarkb++ that sounds great17:26
clarkbthen I guess we confirm next week after people get back from vacation and can schedule an upgrade17:26
fungiyep, i was able to fake mua inject a message to testlist by telnetting to the smtp socket from my workstation, and received my post as a subscriber17:31
fungiyou should also see it in the testlist archive now17:31
clarkblet me update /etc/hosts and take a look17:32
clarkbyup I see it and navigating to it via the web ui to the archive to the month to the thread all works too17:33
clarkbthis looks great, thank you for doing that17:33
funginp, it only took a couple minutes17:33
fungiglad to know my experience prank spoofing e-mails by telnet in high school can still be put to use decades later17:34
clarkbyou even put a signature in the test message :)17:34
fungihelps with getting past ye olde spam filters17:34
clarkbfungi: I guess we shutdown exim and mailman to avoid an unexpected emails from them and can work on scheduling the upgrade now?17:34
fungiyep, on it17:35
fungithey're shut back down17:35
clarkbthanks again. I'm glad this seems to be working without too much intervention.17:36
fungiyeah, straight through to focal. very nice considering it's 4 years of package updates in one shot17:37
fungiinfra-root: rackspace has opened a trouble ticket to let us know afs02.dfw.openstack.org needs a host migration performed by or on tomorrow17:42
fungii can start working to manually initiate a migration now in order to avoid unexpected downtime for that server17:42
clarkbfungi: thanks. afs01.dfw runs the vos releases and afs02 shouldn't be primary for any volumes which means I think you can do that fairly safely without impact?17:44
fungiyeah, i'll follow https://docs.opendev.org/opendev/system-config/latest/afs.html#afs0x-openstack-org to be sure17:44
fungiconfirmed via vos listvldb that afs02.dfw does not hold any rw volumes right now17:45
fungigoing to `bos shutdown afs02.dfw.openstack.org`17:46
fungiand done. i'll poweroff the server for the duration of the host migration to increase our chances it goes quickly and doesn't hang/error on us17:48
clarkb++17:48
fungiand host migration is now in progress17:53
opendevreviewClark Boylan proposed opendev/system-config master: Test a gerrit 3.2 -> 3.3 upgrade  https://review.opendev.org/c/opendev/system-config/+/80524317:56
clarkbthat change should be mergable now. I've got my held node already17:56
clarkbtwo things I notice on the held node: we have sql connectivity errors, not sure if that is just a startup race or not yet. And we configure replication to the giteas on that node so we get a lot of replication errors17:57
fungii've seen sql connectivity errors at docker-compose up in production as well17:58
fungii think gerrit tries to connect to mariadb before the db is fully running17:59
fungialso see this in the git-review functional testsuite17:59
fungiseems gerrit is smart enough to retry until the db accepts the connection17:59
clarkbok I can probably ignore that then18:00
clarkbI've made a copyof the 3.3 configs so that I can diff them against 3.2 configs to ensure there isn't anything new we need to write into those files18:01
clarkbbut I'm abount to sort out the downgrade process18:01
clarkbI think I have successfully downgraded the test gerrit. The process is written down at https://etherpad.opendev.org/p/gerrit-3.3-upgrade-prep18:28
fungineat, and good to test18:28
clarkbthe refs/meta/version update is real git fu. I feel like I leveled up18:29
fungiheh18:30
fungidid you use git to do it, or just vi?18:30
clarkbyou have to use git beacuse its a direct blob update and not a check out able thing18:31
clarkbI suspopse you could use vi if you figured out how to cache and write that objcet by hand but that would be rough18:31
clarkbThe whole time I was paranoid that I might be affecting the prod server because we write out the hostname and everything as if it were prod18:32
fungimaybe we should do something in our jobs to mangle the shell prompt on test nodes18:33
clarkbAny ideas on how we might better indicate to users that they are on a test node? PS1 was my first thought but that gets overridden and set all over18:33
clarkbWhat I ended up doing was checking my IP address and memory count as those differ greatly to prod18:33
fungi#status log Completed manually-initiated host migration for afs02.dfw.openstack.org in preparation for upcoming provider maintenance18:34
opendevstatusfungi: finished logging18:34
fungiserver seems to be back up and working normally again18:34
fungii'll close out the ticket18:34
clarkbfungi: trying to get /etc/profile or /etc/bash.bashrc to update the PS1 it seems like the way debian does it is gives every users a .bashrc which makes that difficult?18:42
clarkbwell difficult because the per user bashrc does set a PS118:42
clarkbif it wasn't setting a PS1 we could set it globally and then it wouldn't be overwritten18:43
fungiwe could alter /etc/skel/.bashrc in our debuntu images, i guess18:49
fungior early before user creation18:50
clarkboh is that the template?18:50
fungiyes, /etc/skel gets copied as the initial homedir for each newly created user18:50
fungithat's typical of *bsd systems too i think, so maybe rh based would work that way as well18:51
corvusapparently npmjs.org is having a sad; it's taking out a lot of zuul gate jobs18:51
fungithough for our infra jobs, whatever works on ubuntu should be sufficient i suppose18:51
corvus(fyi)18:51
fungicorvus: ouch, thanks for the heads up18:51
opendevreviewClark Boylan proposed opendev/system-config master: Test a gerrit 3.2 -> 3.3 upgrade  https://review.opendev.org/c/opendev/system-config/+/80524318:56
clarkbcorrection ^ that should be mergeable now18:56
opendevreviewClark Boylan proposed opendev/system-config master: Test a gerrit 3.2 -> 3.3 upgrade  https://review.opendev.org/c/opendev/system-config/+/80524320:19
clarkbfungi: ^ updated to address your comment20:19
*** dviroel|ruck is now known as dviroel|ruck|out21:08
clarkbcorvus: I'm going to pop out for a bit (bike ride) but happy to help with any zuul restarts we might want to do to pick up those fixes21:16
clarkbI assume the .1 release might happen either tomorrow or early next week?21:16
corvusclarkb: yeah, i'd like to restart this afternoon and release tomorrow21:17
clarkbdon't feel like you have to wait for me either. But I'm happy to help when back21:17
corvusi'm digging into some thing i observed while debugging; i don't think they're critical, but i want to get to a good stapping place before restarting21:17
corvusso maybe after you get back sounds like good timing :)21:17
fungii too am back from foodmaking and foodeating, so happy to help with any restart tasks/fallout21:52
corvusi'm done looking into the stuff i noticed this morning; i pushed up 2 more changes (see #zuul).  i'm waffling on whether we should try to get those in too, or draw the line where we are now22:25
corvusi don't think they're urgent, but if we don't include them in this round, we should probably merge them immediately after and make another point release.  i think i've just about talked myself into saying we should get them in now.  :)22:26
fungii've already +2'd one, i'm re-reading the other now because it was involved enough i wasn't sure i'd given it a thorough look the first time22:27
fungibut i'm also fine not including them22:27
ianwfungi: did you want to merge https://review.opendev.org/c/opendev/system-config/+/804539 to finalise the paste thing if you're happy22:32
fungiianw: oh, yes thanks, i've been far too scattered this week, sorry22:33
fungiianw: on a related note, not sure if you saw but https://review.opendev.org/804540 corrects a bug i found while trying to debug the pastebinit situation22:36
ianwfungi: so the response can't be a unicode string?22:38
fungiianw: the response is unlikely to be unicode, but that's not the issue. the exception data object isn't utf-8 encoded so attempting to decode it raises an exception22:40
fungihowever that codepath is only hit when returning an error, so the observable difference is that now it will return detailed error responses when intended, rather than a generic proxyerror22:41
ianwoh right, sorry i read the diff backwards!22:41
ianwi'm pretty sure you can also remove this line now you've fixed it -> https://opendev.org/opendev/system-config/src/branch/master/playbooks/roles/lodgeit/templates/paste.vhost.j2#L4122:42
fungioh was that it? still, i think blocking getrecent is probably a good hygiene measure22:46
fungipart of what got me trying to fix up testing for lodgeit is i want to invert the private paste toggle and make all pastes use unguessable urls22:46
fungigiven that was essentially the source of the biggest security breach we've ever had to recover from22:47
fungidone properly, i'd just remove the getrecent method from the api, or make it configurable (though that sounds like more work)22:48
opendevreviewMerged opendev/lodgeit master: Properly handle paste exceptions  https://review.opendev.org/c/opendev/lodgeit/+/80454022:51
opendevreviewMerged opendev/system-config master: Stop redirecting for the paste site  https://review.opendev.org/c/opendev/system-config/+/80453923:05
clarkbLook at openstack's release schedule we are entering feature freeze and then RC periods over the next few weeks23:49
clarkb*looking23:49
clarkbBased on that I think it might be best to do big upgrades like the gerrit upgrade after Xena is out the door. Aim for the week of October 11-15 to do the gerrit 3.3 upgrade and possibly the lists.o.o uprade?23:50
clarkbI'm fairly confident in the gerrit 3.3 release at this point, but what I'm not so sure of is taht the release tooling won't have a sad against newer gerrit for unexpected reasons23:51
clarkbthe gitea 1.15.0 seems to be held up by a couple of issues. Not sure when that will be available but hopefully soon. Happy to hold off on that update too unless something forces us to upgrade. We should be ready either way23:54
fungimakes sense23:58

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!