Friday, 2021-08-06

opendevreviewMerged opendev/system-config master: Serve gating.dev static website  https://review.opendev.org/c/opendev/system-config/+/80190800:20
Clark[m]The gitea upgrades should be running now00:26
Clark[m]All 8 are done. They look happy from here00:44
ianwi don't know why infra-prod-base timed out01:58
ianwthe log file seems to just stop during fact collection, so i'm not sure what host it was trying to talk to01:58
Clark[m]Could there be a server that isn't ssh'ing? ps often shows it on bridge it so02:20
opendevreviewMerged opendev/system-config master: system-config-roles: test centos-8-stream  https://review.opendev.org/c/opendev/system-config/+/80298002:24
ianwyeah there's a bunch of stuck stuff, i clean up02:25
ianwi will clean up i mean02:25
ianwthe two unresponsive hosts appear to be logstash 05 & 0802:35
Clark[m]You can reboot those safely02:49
ianwlook like the gating.dev certificate didn't work, and static is now unhappy03:09
ianwgating.dev:Verify error:DNS problem: NXDOMAIN looking up TXT for _acme-challenge.gating.dev - check that a DNS record exists for this domain might be it03:10
xinliangHii guys, ianw: I propose a patch to fix diskimage-builder ARM64 pipeline, please help to review https://review.opendev.org/c/zuul/zuul-jobs/+/80341303:36
xinliangHi guys, also please help to review openEuler diskimage-builder patch https://review.opendev.org/c/openstack/diskimage-builder/+/784363,  thanks03:43
ianwxinliang: hey, i will put them in the queue to look at 04:13
ianwthe /var/lib/bind/zones/gating.dev/zone.db didn't get the _acme-challenge record update04:13
xinliangianw: thanks a lot04:14
ianwoh, hang on, service-nameserver has to run for that04:16
*** ykarel|away is now known as ykarel05:17
*** marios is now known as marios|ruck05:20
opendevreviewMerged zuul/zuul-jobs master: Fix install podman error on Ubuntu aarch64 Bionic  https://review.opendev.org/c/zuul/zuul-jobs/+/80341305:31
*** rpittau|afk is now known as rpittau07:23
*** jpena|off is now known as jpena07:36
opendevreviewAnanya proposed opendev/elastic-recheck master: Run elastic-recheck in container  https://review.opendev.org/c/opendev/elastic-recheck/+/80286608:37
*** ykarel is now known as ykarel|lunch09:06
*** diablo_rojo is now known as Guest344709:16
opendevreviewMerged openstack/project-config master: Allow elastic-recheck cores to create branches  https://review.opendev.org/c/openstack/project-config/+/80347309:22
*** ykarel|lunch is now known as ykarel10:23
*** jpena is now known as jpena|lunch11:26
*** ykarel_ is now known as ykarel11:33
opendevreviewSorin Sbârnea proposed zuul/zuul-jobs master: Include podman installation with molecule  https://review.opendev.org/c/zuul/zuul-jobs/+/80347111:39
tristanCwin 412:01
tristanCsorry, keyboard failed :)12:14
opendevreviewSorin Sbârnea proposed zuul/zuul-jobs master: Include podman installation with molecule  https://review.opendev.org/c/zuul/zuul-jobs/+/80347112:17
opendevreviewSorin Sbârnea proposed zuul/zuul-jobs master: Include podman installation with molecule  https://review.opendev.org/c/zuul/zuul-jobs/+/80347112:24
*** jpena|lunch is now known as jpena12:31
opendevreviewAnanya proposed opendev/elastic-recheck master: Run elastic-recheck in container  https://review.opendev.org/c/opendev/elastic-recheck/+/80286612:46
opendevreviewAnanya proposed opendev/elastic-recheck master: Run elastic-recheck in container  https://review.opendev.org/c/opendev/elastic-recheck/+/80286613:05
opendevreviewAnanya proposed opendev/elastic-recheck master: Run elastic-recheck in container  https://review.opendev.org/c/opendev/elastic-recheck/+/80286613:37
opendevreviewAnanya proposed opendev/elastic-recheck master: Run elastic-recheck in container  https://review.opendev.org/c/opendev/elastic-recheck/+/80286613:42
opendevreviewAnanya proposed opendev/elastic-recheck master: Run elastic-recheck in container  https://review.opendev.org/c/opendev/elastic-recheck/+/80286613:54
*** rpittau is now known as rpittau|afk14:43
clarkbfungi: do you want to review https://review.opendev.org/c/opendev/system-config/+/802922 before I update private host vars and and approve that?15:26
clarkbit is the gerrit ssh known hosts fixup15:26
clarkbnot in a rush as I'm having a slow start this morning and need breakfast etc, but that is high on my list for when I'm caught up15:26
fungii can take a look in a sec, sure15:29
clarkbfungi: it has two +2s but you were involved in the manual fixing of that so thought you may want to take a look15:29
clarkbin exciting news I think upgrading to gerrit 3.3 might be the next big gerrit todo :)15:31
clarkbwell once known hosts are sorted out15:31
*** jpena is now known as jpena|off15:34
clarkbI'm editing host vars for review02 now to add in the ssh known_hosts info15:56
*** marios|ruck is now known as marios|out15:59
clarkbthats done. https://review.opendev.org/c/opendev/system-config/+/802922 should be good to approve16:00
fungiyeah, i'm looking at it now, just got done rebooting my crashy workstation16:01
fungiokay, approved it16:02
clarkbthanks!16:03
clarkbI made copies of the existing known_hosts files for root and gerrit2 so that we can compare or easily revert if necessary too16:03
clarkbthats interesting we build a new image for the known hosts update. I didn't expect that, but also don't expect it to be a problem16:14
*** gibi is now known as gibi_pto16:38
thomasb06hello. is this the channel of openstack?16:51
fungithomasb06: "OpenDev is a space for collaborative Open Source software development"16:52
clarkbwe help provide development resources to openstack16:52
fungithis is where we run the collaboration tools used to develop software like openstack16:52
fungithomasb06: there is a #openstack channel16:53
thomasb06ah, this morning i've been asking at libera but nobody answers16:53
thomasb06let me try #openstack then, thank you for your help16:53
fungitjall of our communities moved from freenode to oftc, as you've discovered16:53
fungithomasb06: ^16:53
thomasb06fungi: well, some move to libera. even until 2 min ago, i though all of them...16:54
fungiwe maintained backup channels on oftc for the past 6-7 years in expectation that freenode may eventually implode, so when it did we just enacted our long-standing evacuation plan16:57
thomasb06well done, to say the least16:58
fungialways have a backup ;)16:58
fungimost documentation has been updated to point users to oftc for our projects now, but if you've found some which still mentions freenode then help fixing those is appreciated16:59
thomasb06the page explicitly mentions oftc: https://docs.opendev.org/opendev/system-config/latest/17:00
fungiyep, it does look like that's been updated17:01
* thomasb06 reads to fast...17:01
fungianyway, the opendev collaboratory is a suite of software development collaboration services which are utilized by communities who build a variety of projects, one of which is openstack, but the people in this channel for the most part are not developing openstack (though we are users of a bunch of donated public openstack cloud accounts to power our services)17:04
thomasb06regarding me, i'm searching for an open source project to get involved in. apparently openstack could be a good place to start17:06
fungithomasb06: with another hat on, i completely agree. openstack has a vibrant and supportive/welcoming community, though it tends to be stretched a bit thin and in need of new people interested in helping out17:09
* fungi basically runs vulnerability management for the openstack project17:09
opendevreviewMerged opendev/system-config master: Improve gerrit known_hosts management  https://review.opendev.org/c/opendev/system-config/+/80292217:10
thomasb06fungi: cool, as i'm not as cs student i feared to be a bit light for giving a hand17:10
thomasb06if you have something for me, i take17:11
fungithomasb06: https://www.openstack.org/community/ is a good place to begin, it will likely land you at the contributor guide for openstack17:11
thomasb06great, thank you for the link17:12
fungias far as what specifically to work on, there's far more to choose from than i could ever hope to list17:12
fungithomasb06: the openstack technical committee does list some particularly desired help here: https://governance.openstack.org/tc/reference/upstream-investment-opportunities/17:13
fungithey also have a #openstack-tc channel here on oftc where i'm sure they'd be happy to answer questions or point you toward additional resources17:13
thomasb06on the link i pasted, i read that openstack is retributing companies sharing their idle computer ressources, did i misunderstood?17:16
fungithomasb06: i'm struggling to parse your question, but i think you're asking about the donated resources opendev uses to provide collaboration and testing services to openstack and other similar open source projects. we do utilize resources generously donated by public openstack clouds to host the systems which provide those services17:19
thomasb06then, i misunderstood: i though the donations could go to company sharing their idle servers...17:21
fungicurrently we have resources donated by rackspace, vexxhost, ovh, inap, citycloud, linaro, osuosl, inmotion, limestone networks, maybe others i'm forgetting at the moment17:21
thomasb06it's the oposite then17:22
fungiwe use those to run thnigs like our code review, continuous integration/testing, video conferencing, document editing, mailing lists, irc/matrix bots, and so on17:22
thomasb06understood17:23
clarkbheh because I left a comment in the inventory path on the known hosts change all the jobs are queued to run17:30
fungid'oh!17:35
corvuswe had the orange sunlight this morning due to upper atmosphere smoke in the east.18:23
clarkbcorvus: we had the bad air along with it for a few days (but nowhere near as bad as last year) then it moved up high into the air and we had the weird sunlight with very good air quality18:25
fungiweather here's nice. no swampfires yet this summer, for a change (watch i've just jinxed us now)18:45
*** odyssey4me is now known as Guest348919:03
clarkbfungi: did you still have time to test the openid email deletion thing with your secondary account?19:15
clarkbI just remembered we had mentioend we could try that yesterday19:15
fungioh, yeah just a moment19:20
clarkbfungi: I'm happy to do a screenshare or just describe the process here. Basically you need to login, then in your email settings add a second different email address and set that one to preferred. Then you can attempt to delete the email address associated with your openid and it should error19:23
clarkbthen refresh the settings page and ensure both emails are still present19:23
fungiseems straightforward, setting up an account container in mybrowser now19:25
fungiinteresting, i can't actually sign into gerrit with that account19:26
fungiredirects me to https://review.opendev.org/SignInFailure,SIGN_IN,Contact+site+administrator when i do19:27
clarkbhuh19:27
clarkbdoes it have a conflicting email already?19:27
fungipossibly?19:27
fungiif it did, that would be in your audit19:27
fungiand i'm not seeing it in there19:28
clarkbit could eb that the account settings don't have it conflicting but when you try to login the data provided by ubuntu one is conflicting19:28
clarkbif that makes sense19:28
clarkbfungi: Authentication error, account inactive19:29
fungiahh, what's the id number?19:29
fungii'll set it active19:29
clarkbnot sure it doesn't seem to tell me in the error log19:30
fungii'll check the external-ids repo19:30
clarkbya if you grep for the email address there then look in the resulting files accountID is one of the fields19:31
fungifound it by grepping for the username, but yeah, fixed and now i'm logged in19:35
clarkbfungi: if you want to try and delete the email now it should fail because it will be preferred (assuming you have only the one email associated)19:35
clarkbthey check for that on the old gerrit too, but now if you add a second and flip it to preferred we should prevent deletion of the openid associated email too19:36
fungithe ui doesn't allow you to delete a preferred address, the delete button is greyed out19:36
fungiadding a secondary address now19:37
clarkbonce you've confirmed you cannot delete the openid assoicated address deleting the new secondary email that you added is probably a good sanity check that deletions work in general19:38
clarkbI tested all that on the test node so it should all work as expected :)19:39
fungii'll have to switch the new address to be my preferred address first, doing that now19:39
fungiit allowed me to delete my original address after setting the new address as preferred19:40
fungii take it that's bad news19:40
clarkbyou didn't get an error pop up?19:41
funginope19:41
clarkbarg19:41
fungichecking to see what's in the external-ids ref now19:41
clarkband ya the next time you login you should get a new account if the external id was actually removed19:42
fungiopenid externalid entries still have the old address in them19:42
clarkbfungi: the externalid entry still exists for the openid and accountId?19:42
fungiyep19:42
fungiwith the original address19:43
clarkbok can you refresh the settings page and see if it shows you btoh addresses again?19:43
clarkbthat implies it didn't actually delete19:43
clarkbbut I'm surprised you didn't get the warning pop up thing saying error blah blah19:43
fungiyeah, the original address is back and marked as preferred again19:43
clarkboh hold on19:44
clarkbsetting a new preferred email and deleting the email address is a multi step process where you have to save iirc19:44
fungii did not hit save changes, i'll see if that's why nothing happened19:44
clarkbif that didn't happen then I think you simply queued the changes19:44
fungithat's it19:44
clarkbfungi: instead you should change the preferred then save and refresh19:44
clarkbthen try to delete the original email that will give you the error19:44
clarkbthen refresh and both should show up with new as preferred. Then you can swap back again and delete the new email19:45
clarkbya the UI here is really clunky :/19:45
fungi[An error occurred] Error 409 (Conflict): Cannot remove e-mail 'fungi-three@yuggoth.org' which is directly associated with OPENID_SSO authentication (Endpoint: /accounts/self/email/*)19:45
clarkbperfect!19:45
clarkbnow if you refresh the settings page it should still show you both email addresses19:45
fungiso if you delete the address in the ui it disappears, even if you don't click save19:45
clarkbyes19:45
fungiif you delete it and click save, you get that error19:45
thomasb06fungi: (was asking for help in #openstack-tc at libera back again...)19:45
clarkbthats a UI thing indepdnent of my update19:45
fungithomasb06: oftc not libera19:46
thomasb06maybe now I'll remember19:46
fungiall of openstack moved from freenode to oftc19:46
funginot to libera19:46
clarkbfungi: now we should confirm that if you swap the preferred email address back that you can delete the new email address (email deletion generaly works confirmation)19:46
thomasb06need to change my config once for all19:46
fungiclarkb: yeah, testing that next19:47
fungiclarkb: it allowed me to do that, as expected19:47
fungiso i think this all checks out19:47
clarkbgreat I think that means the bugfix is working as expected. Thank you for double checking19:48
clarkbI won't spam all the mailing lists about that again, but if peopel ask us questions we can be fairly confident that gerrit won't let them do the wrong thing anymore19:48
fungirefreshing the external-ids ref, the new address is gone again too19:48
fungii'm putting that account back to inactive again. i remember now i did that so that it wouldn't show up in autocompletion for people trying to find me19:49
fungisince it confused some folks19:49
clarkbfungi: sounds good, thanks again19:50
clarkband now i need to go review the merge into master change upstream. The stable-3.4 change landed19:59
clarkbok ssh known_hosts have updated on review for gerrit2 and root20:05
clarkbrunning gerrit ls-projects continues to work so we didn't break that20:06
clarkbI think we're good20:06
fungiyep, lgtm20:07
clarkbmy first run was against localhost the second against review.opendev.org both worked but when I did review.opendev.org it also added the key for the ip address20:07
clarkbI Guess that is expected sicne we don't put the ip address in the listing managed by ansible and i think that is ok20:08
clarkbya lgtm20:10
corvusinfra-root: due to an amazing review push from clarkb and tobiash on changes from swest and felixedel, all of the changes in zuul to implement moving the merge requests from gearman to zk, and also using cached config on restarts are approved.  once they land, i think it's worth restarting zuul to start using those.  and also, that should be the last restart that takes 20-30m because of cat jobs!20:17
corvusso really, once those land, i'd like to restart zuul twice :)20:17
tobiash[m]looking forward to see the improvement :)20:17
clarkbcorvus: we need to restart twice to exercise the faster startup?20:19
clarkbjust making sure I understand the process. I don't think thati s ap roblem20:19
corvusclarkb: i believe so, yes.  i don't think quite all the info will be in zk for the first startup to be fast20:20
clarkbgot it20:21
fungicorvus: i'm very much in favor. can't wait to see it in production20:22
*** dviroel is now known as dviroel|out20:23
corvusincidentally, these same changes are also the first changes where we added a second scheduler to a unit test.  we can't do more than load the config with 2 schedulers yet, but we can get that far.  it's a pretty cool milestone.  :)20:26
fungiawesome!20:29
opendevreviewMerged zuul/zuul-jobs master: Remove success-url  https://review.opendev.org/c/zuul/zuul-jobs/+/80297420:53
clarkbfungi: any chance you want to review this small puppet cleanup? https://review.opendev.org/c/opendev/system-config/+/80351821:42
opendevreviewMerged opendev/system-config master: Cleanup unused puppet modules from modules.env  https://review.opendev.org/c/opendev/system-config/+/80351823:31

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!