Saturday, 2021-08-21

mordredcorvus: more fail:
mordredcorvus: looks like baseurl and m.server swapped in the test00:48
fungiassert '"base_url": ""' in '{\n    "m.server": ""\n}\n'01:28
fungiassert '"m.server": ""' in '{\n    "m.homeserver": {\n        "base_url": ""\n    },\n    "m.identity_server": {\n        "base_url": ""\n    }\n}\n'01:28
fungiyeah, i'll see if maybe the file contents are swapped01:29
fungiyep, either the file contents are backwards or the tests are backwards...01:30
fungii think it's the tests which are backwards, the file seem to match what we're currently putting in the gitea containers01:32
fungii'll try to fix it up01:32
fungii think it's the urls which are switched? we want to see base_url in client with no cors header, and m.server in server with a cors header, yeah?01:36
fungiahh, no, we're doing the cors header on client not server01:37
fungiso it's just the string matches which are swapped in the tests01:37
opendevreviewJeremy Stanley proposed opendev/system-config master: Serve matrix well-known files from apache
fungithat ^ should hopefully solve it01:39
opendevreviewJeremy Stanley proposed opendev/system-config master: Assume gitea reverse proxy
opendevreviewJeremy Stanley proposed opendev/system-config master: Remove matrix well-known files from gitea image
fungirebase those ^ on the revised one01:42
fungier, rebased01:42
fungicloser at least...02:56
fungiAssertionError: assert 'Access-Control-Allow-Origin' in '* Added to DNS cache\n* Hostname was found in DNS cache\n*   T...3    0     0   9562      0 --:--:-- --:--:-- --:--:--  9562\n* Connection #0 to host left intact\n'02:56
fungii think the header output may not be where the test expects?02:56
fungimmm, no, using a similar curl command, i get "< Access-Control-Allow-Origin: *" from somewhere with it set, so maybe apache isn't correctly serving it?03:06
fungimod_headers seems to be enabled, at leas on our production servers, so it's probably not that03:15
fungishould have a held node here shortly04:30
corvuswow that ended up being complex :/04:45
corvusfungi: indeed, i don't see a cors header when i curl on that server04:48
corvusthat's the job doing?  it looks like it's just been sitting there for 16 minutes04:49
corvuswell, gitea process seems to be busy, maybe this is the part of the job where it's creating repos04:50
corvusi think it's proxying to gitea04:51
corvusthe ilikegitea cookie is being sent04:51
corvusoh i see the problem -- the test isn't hitting apache, it's hitting gitea directly05:01
corvusapache listens on 308105:01
opendevreviewJames E. Blair proposed opendev/system-config master: Serve matrix well-known files from apache
opendevreviewJames E. Blair proposed opendev/system-config master: Switch robots.txt test on gitea to proxy port
corvusfungi: ^ hopefully that should get it05:05
corvusthanks for pushing that along :)05:05
opendevreviewJames E. Blair proposed opendev/system-config master: Assume gitea reverse proxy
opendevreviewJames E. Blair proposed opendev/system-config master: Switch robots.txt test on gitea to proxy port
opendevreviewJames E. Blair proposed opendev/system-config master: Remove matrix well-known files from gitea image
fungiVirtualHost *:308105:07
fungiyep :/05:07
fungijust got on the held node to check05:08
fungii should have spotted that sooner05:08
fungiwell, i guess we at least proved the context check against the files gitea's serving05:08
fungiso if they also work against apache then we can be more confident they're correct05:09
fungii'll clear the autohold and knock off for the night05:10
corvusfungi: thanks!  goodnight!05:11
fungiyou too. i'll check back in after breakfast and see where we are05:11
opendevreviewMerged opendev/system-config master: Serve matrix well-known files from apache
yoctozeptoinfra-root: ethercalc is down07:57
yoctozeptoerror 50307:57
yoctozepto(the usual)07:57
*** redrobot4 is now known as redrobot08:04
yoctozeptoup again09:46
mordredfungi, corvus: the patch is landed and the well-known file looks good, but I still can't browse channels12:27
mordredI can join them12:40
fungii can confirm has a "Access-Control-Allow-Origin: *" header served with it12:41
mordredone of the gnome guys suggested that there is a homeserver config setting ^^&12:41
mordredwe might need to engage with our element friends to get them to set that?12:41
opendevreviewMerged opendev/system-config master: Assume gitea reverse proxy
yoctozeptoso the gnomes are real? ;o13:32
fungiyoctozepto: yes, every morning when i wake up, the software gnomes have added new bugs for me to troubleshoot13:33
yoctozeptofungi: :O no kidding, must be the reason there are so many bugs13:34
fungiinfra-root: our shared inbox contains a couple of "please confirm your address" messages from element/matrix which arrived around 21:00 utc yesterday. just a heads up in case someone hasn't followed up on those. if they're no longer needed i'll move them to our element service archive mailbox13:37
corvusfungi: i probably triggered those; i don't know why it asked for them... :/13:38
* corvus uploaded an image: (40KiB) < >13:38
corvusmordred: fungi do you think it could be the "Guest users" setting? 13:39
fungiwe already talked yesterday about turning that on, right?13:40
fungiseemed like there were no objections13:40
corvusfungi: i'm not 100% sure what that option entails, though i think generally it sounds like something we'd want?13:40
fungithe way it was described yesterday sounded like a net positive13:41
corvusbased on chat history in #ems -- the 'guest users' feature would probably be needed in order to allow anonymous read access to rooms, so yeah, i think it's the next step in that and seems likely to be related to the lack of ability to browse lists13:45
fungicorvus: skimming back in discussion history, it looks like the confirmation e-mails may coincide with when you tried using an admin account to delete messages from the test channel13:45
corvusfungi: no it was when i logged into the ems dashboard and it kept asking me for my name13:46
fungiso apparently that generates address confirmation e-mails13:48
corvusyeah; maybe our account got into a weird state or something.  i haven't confirmed the email (again?), but i probably should13:49
corvusi asked in #ems if there's a setting that corresponds to the link from mordred13:50
corvusi'm assuming if we turn on the 'guest users' switch this will be enabled:
corvusbut maybe it will also enable the other option?13:57
corvuswhy don't i try enabling guest access and let's see what happens?14:04
fungiworth a try14:07
fungiskimming the config, room_list_publication_rules is likely applicable14:07
corvusi enabled it, and playing around with our (unpublicized) element instance, i am able to browse the list of rooms, and see a preview of #zuul without logging in.  i get prompts to create an account if i want to join; of course that doesn't work since we don't allow registration.14:10
fungiahh, yeah, i didn't consider that the prompt to register would point to registering with our homeserver14:11
fungii wonder if there's a way to have it send people to's registration page14:12
corvusokay, if i add '' as a server, i can browse the directory now, so it looks like this did enable the feature we want14:16
corvusit's a little unfortunate that typing in "" doesn't automatically do that, so you don't get feedback that it's correct.  but this looks like the best that's possible right now14:16
corvusi think this is the real benefit of anonymous access -- visit this in a private tab:
corvusthat's the link we're going to have on the web site and docs14:18
corvusthat lets you preview the room, and, happily, it's going through, so the signup links are for the homeserver14:19
corvusall in all, i think this setting is good for us14:19
fungithe privacy policy is interesting14:19
fungi"You must be at least 16 years old to use this Service ("14:19
fungii guess this is a cookie-cutter policy adapted from element14:20
fungibut yeah, it's only a few clicks from that url to get to a view of the channel through the in-browser element client14:21
fungiactually the "sign in" and "sign up" links there default to as the location for the account when i click them14:22
corvusit's only if you do the same thing through our hidden element client that they default to opendev as the homeserver14:23
fungiahh, so it's by virtue of using the url that we get as a default account provider14:23
fungiand also i get to imagine i'm visiting a site in tonga14:25
fungivirtual vacation in the pacific islands14:26
corvus#status log enabled "Guest access" on matrix synapse server EMS control panel to allow anonymous read access to rooms which allow that, and for federated homeservers to be able to browse the published rooms list14:30
opendevstatuscorvus: finished logging14:30
opendevreviewMerged openstack/project-config master: Remove gerritbot from #zuul
opendevreviewMerged opendev/system-config master: Remove matrix well-known files from gitea image
mordredcorvus: \o/14:53
mordredcorvus: I confirm that in my element client I can now add to the server list and browse the channels and it shows me the zuul channel14:53
opendevreviewMerged opendev/system-config master: Move #zuul from OFTC to Matrix
mordredcorvus: it's happening15:09
tristanCit seems like the gerritbot didn't pick the zuul room config, is /var/lib/matrix-gerritbot/config/gerritbot.dhall updated? if it is, the autoreload seems broken and the compose should be restarted. if it isn't, there may be something wrong with the system-config ansible tasks15:40
fungii'll look15:52
fungi/var/lib/matrix-gerritbot/config/gerritbot.dhall does not have the zuul room in it yet, just the test room still15:58
fungidid the deploy job complete?15:58
mordredfungi: it has not yet started16:00
mordredfungi: there are two ahead of it - and one of them is running quite long16:00
fungithat'd 'splain it16:00
corvusthe eavesdrop job just finished16:03
corvusthere's another project-config eavesdrop job to run too16:03
corvusoh sorry, that was for an older change... that's what you meant16:04
corvusso yeah, the job for the queue item for 805464 probably won't run for another 2 hours :(16:04
corvusgood thing we didn't specify a cutover time :)16:04
mordredcorvus: maybe the following system-config changes won't deploy *everything* ?16:06
mordredcorvus, fungi : base failed on ns2 with a dpkg lock issue trying to run apt-get autoremove16:08
mordredthere seems to be a dpkg process from aug 6 that's trying to remove old kernels16:09
fungioh fun, i'll check it out16:09
mordredI'm thinking we shoudl kill that, clean the lock and re-run autoremove to get ns2 into a non-broken state16:10
fungiyeah, go for it16:10
corvusmordred: ++16:10
mordredroot     20417 20405  0 Aug06 pts/1    00:00:03 /usr/bin/dpkg --status-fd 22 --no-triggers --force-depends --remove linux-headers-4.15.0-147-generic:amd64 linux-headers-4.15.0-147:all linux-headers-4.15.0-45-generic:amd64 linux-headers-4.15.0-45:all linux-image-4.15.0-147-generic:amd64 linux-image-4.15.0-45-generic:amd64 linux-modules-4.15.0-147-generic:amd64 linux-modules-4.15.0-45-generic:amd6416:10
mordredthat's the hung process, fwiw. I'm betting there's a prompt :)16:10
mordred"A new version of /boot/grub/menu.lst is available"16:11
mordredI'm leaning towards "install the package maintainer's version" - we don't actualy manage that file, yeah?16:12
fungiwe don't16:12
fungiyou can diff it if you're concerned16:12
mordredthe new version removes references to initrd16:13
mordredso - seems like we should follow pacakge maint lead :)16:13
mordredk. ns2 should work next time16:14
fungiawesome, thanks!16:14
mordredfwiw - we've made it to remote-puppet-else16:15
mordredooh! that went quick16:16
mordredwe're on the second change now - and it's only doing gitea16:16
fungiclarkb put in some effort to speed these up16:18
fungithough there's still more we can do16:18
mordredyah. it's one of those forever tasks16:19
mordredcorvus: your change is now queued for deployment. it is going to run ALL the jobs17:02
mordredwhich I'm guessing is largely due to inventory file matchers17:03
fungiyes, it's that it touches the inventory17:09
opendevreviewMonty Taylor proposed opendev/system-config master: Restrict generic inventory matchers to inventory/base
corvusmakes sense to me17:14
opendevreviewMerged opendev/system-config master: Run service-eavesdrop after promoting the matrix eavesdrop bot
tristanCthank you for looking into that, it seems like the new matrix room is now configured for the gerritbot18:10
corvuseavesdrop is now running with the new config; i can see it on the text file; the html file should update the next time the cronjob runs18:18
corvusi think that means we have switched :)18:18
mordredcorvus: how do we (gracefully) eject people from the old room? do we just ask nicely and then all just leave?18:20
fungiwe could make the room invite only and then kick everyone18:21
fungior we could just let everyone know and then add a welcome message and topic which both let folks know we're using via matrix instead18:22
opendevreviewTristan Cacqueray proposed opendev/system-config master: Update gerritbot-matrix version to fix a message typo
corvusfungi, mordred: i'm about to send out an email -- then i think we should do fungi's suggestion, maybe for the next week or so?  then maybe see about further restriction (muting or invite only or something)18:23
corvusit looks like limnoria may still be logging, so we're sort of interleaving both rooms in the log right now18:35
corvusi'll double check its config and see if it needs to be restarted18:35
corvuserm, does anyone know where /var/lib/limnoria/opendev/conf/channels.conf comes from?18:38
corvusi think maybe that's internal state from limnoria?18:41
fungii think it must be generated, i can't find the string "lobotomized" in system-config nor project-config18:41
corvusi think our ansible creates limnoria.config, and that does not have #zuul in it18:41
corvusso i'm thinking we should restart limnoria and see if it changes18:42
fungiprobably expects to restart the container, yes18:42
fungii agree18:42
funginow seems like a great time to do that18:42
fungiyou want to or shall i?18:42
corvusi will18:42
corvushas it rejoined?  i'm not sure i see all the events18:45
corvusoh yep, i see it in element18:46
corvusah, and now i see log entries18:46
corvuswonder what the delay is there18:46
corvuslooks like it's done joining, and it has not rejoined #zuul, so i think it's all good now18:48
corvusmordred, fungi: maybe a quick +3 on ?18:51
mordredcorvus: done18:57
mordredtristanC: docker: Error response from daemon: manifest for not found: manifest unknown: manifest unknown.19:10
mordredtristanC: verified locally19:11
mordredtristanC: my local docker says:19:13   latest      75c78329e615   5 weeks ago     362MB19:13
mordred5 weeks ago seems a little old - maybe something weird happened?19:13
mordredoh - I see what you're doing19:14
mordredis 0af1f8f the more right sha then?19:14
opendevreviewTakashi Kajinami proposed openstack/project-config master: Telemetry: Switch back to launchpad from storyboard
mordredOH ...19:16
mordredtristanC: actually - 0af1f8f is the older one - I see that now. in that's the most recent one I see - I think afdc8c may have had a publication sad19:17
tristanCmordred: yeah, publication failed, the tag is now available19:24
tristanCmordred: it's a nix build i need to move in a zuul jobs. you might be interested to inspect the image, you'll see one layer per package19:31
opendevreviewMerged opendev/system-config master: Update gerritbot-matrix version to fix a message typo

Generated by 2.17.2 by Marius Gedminas - find it at!