Friday, 2021-07-09

corvusrm_you: i se an "rm_work|" (with a pipe) as a puppet user00:00
rm_work|will spin this down soon00:01
clarkbok time for dinner00:03
clarkbI've learned more things about matrix :)00:03
rm_workme too :)00:07
fungicatching up, as for the tangent about edits, the weeslack slack plugin for weechat totally uses s/this/that/ or 3s/this/that/ to edit the third most recent message you sent00:11
rm_workyeah neat, would be cool for element to do that as well :P00:12
opendevreviewGhanshyam proposed openstack/project-config master: Properly retire neutron-lbaas
opendevreviewGhanshyam proposed openstack/project-config master: Properly retire neutron-lbaas
opendevreviewTakashi Kajinami proposed openstack/project-config master: Retire tripleo-common-tempest-plugin - Step 1: End project Gating
*** ysandeep|away is now known as ysandeep04:54
*** ykarel|away is now known as ykarel05:04
opendevreviewTakashi Kajinami proposed openstack/project-config master: Retire tripleo-common-tempest-plugin - Step 3: Remove Project
*** bhagyashris|ruck is now known as bhagyashris|out05:53
*** ykarel_ is now known as ykarel06:17
*** jpena|off is now known as jpena07:11
*** ysandeep is now known as ysandeep|lunch07:29
*** amoralej|off is now known as amoralej08:05
*** ykarel is now known as ykarel|lunch08:30
opendevreviewDmitriy Rabotyagov proposed openstack/project-config master: Create repo for Hashicorp Vault deployment
opendevreviewDmitriy Rabotyagov proposed openstack/project-config master: Add Vault role to Zuul jobs
*** mgoddard- is now known as mgoddard08:40
*** ysandeep|lunch is now known as ysandeep08:42
*** mrunge_ is now known as mrunge08:59
*** ykarel|lunch is now known as ykarel10:05
opendevreviewchzhang8 proposed openstack/project-config master: register andd return back tricircle under x namespaces
*** jpena is now known as jpena|lunch11:20
*** jpena|lunch is now known as jpena12:25
*** amoralej is now known as amoralej|lunch13:07
*** ysandeep is now known as ysandeep|away13:21
opendevreviewMatthias Runge proposed openstack/project-config master: Enable deleting left-overs of panko deprecation
*** amoralej|lunch is now known as amoralej14:00
clarkbfungi: I think its not so much weeslack as it is slack14:34
clarkbthey removed that feature and people complained and then added it back again iirc (to slack proper)14:35
fungithe s/foo/bar/ editing?14:35
clarkbLooking through my todo list I think the big one for ensuring ianw's monday goes smoothly is ensuring that zuul restarts cleanly14:44
clarkbcorvus mentioend being able to do that today and I offered to help. corvus are there other changes we want to land really quickly before doing that?14:44
fungii'm happy to switch focus at some point to help with that too if needed14:45
corvusclarkb: nothing urgent, but maybe we can do a quick pass over hashtag:sos and see if there's anything else ready to land?14:50
clarkbcorvus: sure15:04
clarkbcorvus: is the one I see15:05
clarkbI can review that shortly15:05
corvusclarkb: yep, i just -1d the other 2, so that's the only candidate.15:07
opendevreviewClark Boylan proposed opendev/system-config master: Update gitea to 1.14.4
clarkbI'll mark ^ wip as the set of changes is not small. But reviews would still probably be helpful particularly in double checking the template updates.15:32
clarkbnow to review that zuul change15:32
*** jpena is now known as jpena|off16:11
clarkbcorvus: mordred: in the original well known patchset you used I'm looking at setting up the EMS instance now and it seems that they are all suffixed with I guess domains have simply changed since you set things up?16:11
corvusclarkb: i'm unaware of that... but honestly, that seems surprising... i wouldn't think they would want to use element.io16:16
*** amoralej is now known as amoralej|off16:18
clarkb"First you need to choose a hostname. This will be where you will log in to your Element chat webapp." Then the field says your_subdomain    .element.io16:23
clarkbI'm going with opendev16:23
clarkbfor custom homeserver domain I want and for custom client domain we want ?16:27
clarkbmordred: ^ I think this is the bit you were talking about16:28
clarkboh actuall I think we don't need custome client domain this is what we talked about yseterday with admins being able to talk directly to it16:28
clarkbwhen I punch in the results match what mordred pushed in ps1 of the well known files change16:32
clarkbI think they use different domains for that stuff then (it complains that the files 404 but says I can deal with that later so I will proceed)16:32
clarkbI'm disabling guests and public registration. I believe we have already decided we never want public registration but maybe we want guests later16:34
clarkbI'm going to take the secrets lock16:38
clarkbOk I've put the credentails in the usual location and logged back in again using them. If someone else can give that a go that would be great16:50
clarkbI'm going to restore mordred's original ps now16:51
opendevreviewClark Boylan proposed opendev/system-config master: Add matrix well-known files for opendev
clarkbThey do note there is some cors requirements for one of those files which may force us to host it with apache, but I think we can do that if this doesn't work16:52
clarkbcorvus: ^ do you want to send that one in if it looks good to you?16:53
mordredyah. there's no rush - that wizard will happily sit open for ages16:53
clarkbmordred: ya there is a button to tell it to recheck16:53
clarkbwe are on the trial level nicket setup with 5 users.16:54
mordredI'm betting that'll be more than enough16:55
corvusi don't see any reason for us to even use the ems-hosted element service16:57
corvusi don't think we should do that, even as admins16:57
corvusso if it's broken by cors: great :)16:57
clarkbI'm not entirely sure what the next steps are other than to land the well known file. I suppose to create an admin account in the server.16:58
clarkbAnd then a gerritbot account and then run a gerritbot?16:58
clarkboh and I guess some channels16:59
clarkbI'll probably defer to order of operations on that to others but I'm happy to try and help do the work16:59
corvusclarkb: that sounds good16:59
clarkbI've been reminded that I need to get on the bike soon before the outside becomes extra warm. I'll start working my way out the door now then when I get back can help with zuul restart stuff.17:00
clarkbinfra-root but definitely see if you can get to the EMS dashboard and maybe glance over the options there17:01
corvusi may take the same break actually :)17:01
*** melwitt is now known as Guest32217:31
mordredclarkb: I have logged in to the dashboard and things look good.17:49
fungii logged into it enough to confirm the recorded credentials work, but that's all17:52
*** melwitt_ is now known as melwitt17:57
*** melwitt is now known as jgwentworth17:58
*** TheJulia is now known as needssleep18:04
opendevreviewGoutham Pacha Ravi proposed openstack/project-config master: Add channel ops to #openstack-manila
opendevreviewMerged openstack/project-config master: Add channel ops to #openstack-manila
clarkbI'm back and can help with restarts whenever we are ready.19:24
clarkblooks like the nodepool launcers are on a 2 day old image whihc i think incluides the change we need on them19:24
fungii'm not sure if we were waiting on images from 800282 for restarting the executors, though i suppose it's immaterial since that didn't switch the default behavior anyway19:25
fungiand yes, i restarted the launchers on the new version already19:26
fungiit just happened to be convenient to do that since i already needed to restart one to free up those hanging arm node requests19:26
corvusclarkb: the promote job for the latest patch was successful, so i think we can pull & restart now19:26
corvusfungi: i think we should restart with images from 800282 -- that's the promote job i meant19:27
corvussince it secceeded, we only need to do a pull and we'll be sure we have it19:27
fungiright, that's what i assumed19:27
clarkbwe do ignore errors now though right?19:27
corvusclarkb: i think just on tag deletes?19:28
fungierrors for what?19:28
fungioh, on promote19:28
fungiyeah, that situation19:28
fungiit was just ignoring failure to delete tags on dockerhub19:28
clarkbcorvus: oh right just the cleanups19:28
clarkband ya checking docker hub images appear up to date19:28
corvusi think/hope/intended it to still fail if it can't create the desired tag19:28
clarkbcorvus: yes, I blieve that is the case19:29
corvus(is there an english word for think/hope/intend ?)19:29
corvus("would be suprised if not" is about all i can come up with right now)19:30
corvuspulling now19:30
fungi"wager" ;)19:30
corvusfungi: ideally a word without unintended financial side-effects :)19:31
JayFcorvus: expect?19:32
opendevreviewMerged opendev/system-config master: Add matrix well-known files for opendev
clarkboh hrm ^ restarting while that promotes and runs might get weird19:33
clarkbI guess worst case I can manually do the service pulls and updates for that19:33
corvusJayF: yeah, that's probably the closest :)  that's like 98% what i wanted to say, then throw in 2% of "fingers crossed" :)19:34
JayFI always joke about the world "should" for stuff like that19:34
corvusclarkb: it's waiting on the infra-prod playbook semaphore, which i guess is held by the hourly jobs19:34
JayFbecause should is the weasel word that means "this is what I expect but ANYTHING IS POSSIBLE"19:34
JayFe.g. The patch should fail in CI if it creates a bug.19:35
clarkbcorvus: yup it should get it once the infra-prod-service-nodepool job finishes. I guess if we stop now and reeqneue we'll probably be ok19:35
corvusJayF: if we capitalize it like an RFC it takes on that specific meaning, right? :)19:35
clarkbcorvus: or we wait for the nodepool job adn the gitea jobs to finish as another graceful option19:35
JayFcorvus: I've worked with enough protocol implementations to know that even in an RFC, should means 'this is expected but ANYTHING IS POSSIBLE" lol19:36
corvusclarkb: maybe let's see what happens in the next 2 minutes?19:36
clarkbcorvus: wfm19:37
corvusit's running a "cleanup old logs" task, but with 10237 logs in that directory, i'm not sure it's suceeding19:38
corvusanyway, on to gitea now, which takes ~16 minutes19:39
corvusthat's kind of interesting that the semaphore shifted pipelines like that, but i guess it makes sense due to priority19:40
clarkbyup I think that is normal19:41
clarkbwhere normal == what it has always done19:41
clarkbI see the well known files on gitea01 now19:44
clarkbit has finished through 06 I think19:57
clarkbnow 7 is done19:59
clarkb8 is being restarted right now20:00
corvusplaybook is done; just waiting for the job to tidy up20:00
corvusclarkb: look good to restart now?20:05
clarkbcorvus: I think so20:05
clarkbthe zuul playbook is running and i think it will keep running20:05
clarkbbut it doesn't start anything iirc20:05
clarkbso we should be fine20:06
corvus#status log restarted all of zuul on commit 657d8c6fb284261f1213b9eaf1cf5c51f47c383b20:06
opendevstatuscorvus: finished logging20:06
johnsomDo we need to restart our jobs or will they eventually requeue?20:14
corvusjohnsom: no action necessary20:14
fungiyeah, i see them starting to show up in the status interface now20:14
clarkband the jobs that are running seem to be doing work20:18
clarkbwhcih is good because there were changes to node requests so if we're doing real work then we're probably in a happy spot?20:18
fungii see some have already succeeded20:19
fungiso, yes20:19
fungii need to go run a quick errand since this looks probably sane, but should be back in 30-45 minutes tops20:19
clarkbno worries20:20
corvusre-enqueue done20:22
clarkbI'll be around to keep an eye on it during that time. When fungi gets back I may do some local network updates though20:22
corvusclarkb: what's the matrix status?20:22
clarkbalso apparnetly the 5.13 kernel is available now I should update to that20:23
clarkbcorvus: I need to log back in and hit the recheck button the well known file stuff20:23
clarkbI'll do that now20:23
clarkbcorvus: the server well known doc is happy now and has a green check mark. The client homeserver well known doc is not due to cors as we suspected. You indicated yuo thought that was fine so we'll just leave it as is for now?20:25
corvusclarkb: i think so20:27
clarkbLooking at the user management menu I can add a user, give it a and check a box if I want it to be admin or not. Do we want and admin account called ?20:28
corvusclarkb: sounds good20:29
clarkbok I'll add that info to the usual location20:29
clarkbso now I think if I click on our element url I'll get a new webbrowser element client whcih I can use to login as admin and make a channel?20:34
corvusclarkb: sounds right; modulo maybe that has issues due to cors?20:34
clarkbya I think the cors thing is if we want element to live under opendev.org20:35
clarkbthe url they give me is an url so I'm hoping it will work20:35
corvusyou should be able to use any element webapp to talk to any homeserver (ie, i could use my own)20:36
corvusbasically, you're just trusting that site to serve you an application which then runs entirely in-browser20:36
corvusso nobody else should trust me, but i do :)20:36
corvusclarkb: if you're going to make a room, maybe just a 'test' room for now?20:38
clarkbcorvus: sure. The first thing it does is invite me to server alerts and then when I join that channel the server tells me I must agree to a tos to use the service (wasn't really expecting that)20:39
clarkbI'm going to go ahead and accept those for our admin user :)20:39
corvusclarkb: neat, that's all EMS stuff :)20:39
clarkbcorvus: yup. Also it auto created a #general for me. Should we just use that instaed of #test?20:40
corvuswe can tombstone it later if we want20:40
clarkbI'll try and invite those of us I know that are already on matrix20:41
clarkbthen we can change the channel settings to be more public as we get more comfotable with this20:41
clarkbmordred: corvus: you have been invited. I couldn't find fungi's matrix name20:45
fungiclarkb: it's "fungicide" on element because "fungi" was taken by someone else who obviously shares my impeccable tastes and unrivaled sense of style21:02
clarkbfungi: cool on sec21:02
fungii'll get around to setting up a homeserver at some point to rectify that21:02
clarkbfungi: kinrui is the display name?21:03
fungiyeah, that was a test display name (fungi in romanji) so i wouldn't have to ghost my irc connection21:03
corvusyour irc nick can differ from your matrix display name21:04
fungii thnik i just set them to the same at the time21:04
fungisince i was fiddling with clearing out the [m]21:05
clarkbfungi: you should have invites to the two channels we're using now. Though really we're only going to use the one #test21:05
fungithanks, i haven't set up a persistent client yet anyway, will try to knock that out over the weekend21:06
clarkbOne thing to be careful of in element is adding aliases21:07
clarkbThere are two areas to do that. One allows you to create the alias on any server you have permissions to do that on I guess and the other will add it specifically to your homeserver21:08
clarkbanyway I set the alias when I was admin on the opendev homeserver so that all worked fine21:08
clarkbbut when I looked at the settings as myself with admin perms noticed the subtle shift in behavior there21:08
corvusclarkb: yep, i think we don't generally want people to add room aliases to official opendev rooms21:10
corvusbut that is how you can achieve some fault tolerance, and it's how you move rooms to other servers21:11
clarkbit might also be good to have someone else login as admin via element and make sure that is working for them and my browser isn't now forever stuck being the browser to do that. The element url is in the EMS dashboard under server info iirc21:12
clarkbthe credentials to log into the ems dashboard and then to the admin account are in the typical location21:12
clarkb(can you tell I don't want to end up as the spof here? :) )21:13
clarkbok I'm going to do those network upgrades now. I'll be back soon21:13
corvusi'm going to make an account for an eavesdrop bot21:32
corvus(which should exercise the second-user testing clarkb asked for too :)21:33
clarkbnetwork and local kernel updates complete. I'm still here so they went ok I guess :)21:34
corvusclarkb, fungi: i notice our hostname is no longer 'eavesdrop.*' is there a rebranding effort i should be aware of?21:35
corvusi'm inclined to make a user but don't want to undermine that.21:36
corvusnaming the user doesn't seem quite right though; it's not a meeting bot21:36
corvuscould go with 'logs' i guess?21:37
clarkbcorvus: one sec for me to double check something21:37 is still a thing where we run the bots (its a new server replacing the old one though not sure that is cimplete)21:38
corvusbut the cname is 'meetings.' now, yeah?21:38
clarkbas part of that replacement the hosting was centralized under the meetings and logs website at
clarkbya the idea behind that was humans interacting with the system think meetings ratehr than eavesdrop to find meeting logs21:39
corvusas a project that uses logs w/o meetings, that doesn't help, but i guess it's too late to provide feedback on that change :)21:39
clarkbya ianw did that as part of the cleanup from the oftc move21:39
clarkbI don't think the eavesdrop name will go away (though it is a redirect)21:40
corvuswell, there never was an afaik21:40
clarkboh hrm ya there may not have been21:40
clarkbThere isn't now21:40
corvusanyway, i don't expect this to get any less weird for the zuul project, like i said, that ship seems to have sailed21:41
corvusjust wondering what the name of the bot should be :)21:41
clarkbeavesdrop is sufficient generic and also descriptive that I still like that. THough I wonder if anyone had concerns with that name being creepy or something21:42
clarkbI'm happy with 'eavesdrop' or 'logs' and iirc we easily disable accounts in EMS if we want to21:42
clarkbthat means if we go with one and don't like it switching to another should be easy21:42
clarkb(without consuming user slots)21:42
corvusokay, i'll just go with logs21:43
mordredcorvus: are we thinking we'll have a log bot and a meet bot as separate bots?21:54
corvusmordred: well, i'm writing a log bot as we speak, but i don't plan on writing a meetbot since it's not required for zuul21:55
corvusif someone wants to extend what i'm writing, i have no objections21:55
fungiyeah, i didn't question the name of the new service, though it was hurried in so we could stop running a fork of supybot, which meant limnoria not deployed via puppet, and solving the irc-meetings publication which went to static.o.o so got a new site name21:57
corvusi have it on good authority naming is hard21:57
fungii expect we could bring back the eavesdrop name for the location of channel logs, and keep meeting logs on the meetings site, would just need to figure out the details21:57
mordredit's log. it's log. it's big, it's heavy, it's wood.21:58
fungibasically we had static content (irc-meetings publication which we'd prefer to serve out of afs as a normal publication job) and dynamic content written by the bot to the server21:58
fungiit's better than bad, it's good21:58
fungisince the meeting logs and channel logs are in separate file trees, it should be fairly easy to have those be different sites21:59
fungihaving the eavesdrop site served from static.o.o while the bots ran on an eavesdrop server which did not host that site would have been kinda weird to manage, so i understand ianw's need to come up with a separate name for the former22:01
fungibut to the original question, if there was any conscious rebranding i think it was merely to have something on the domain since we were rebuilding it anyway, and make the names legacy cnames/redirects22:02
corvusoy, the bots need to agree to the T&C22:03
corvusbut neat: the error for that includes the url to agree, so we can do so without having to log in (i have just done so for logs)22:04
fungianybody happen to know what might have modified /etc/accessbot/channels.yaml on eavesdrop01 around 21:43z? the last change to that file was from 800296 which merged and deployed hours prior22:33
corvusi have not logged into that server recently22:34
fungii'm trying to figure out why the accessbot run log acts like that change hadn't been included22:34
clarkbwasn't me. Are you saying the file was modified at that time but did not include the modifications you expected?22:34
fungiwondering if we have some sort of order of operations problem, though /var/log/ansible/run-accessbot.yaml.log indicates it copied the file into place before it ran accessbot22:34
fungithe file appears to have been modified a couple hours after the deploy job completed:
fungijust judging from the last modified timestamp on it22:35
fungiso i can't tell if it was updated more recently than that build to include the change which triggered that build and the deploy used a stale copy of the file22:36
fungibasically i don't know what the file on disk there actually looked like when infra-prod-run-accessbot ran, because it looks like something has touched that file since then22:37
fungii'm half expecting that if i just manually run accessbot now, the change from 800296 will get applied (it seems to be in the config currently present on disk)22:38
clarkbthe file comes from project config right? I susppose it is possible there is a race somewhere around updating project-config in the deploy job and running the deploy job22:39
fungibut without trying that, i can't say whether there's a bug in accessbot which caused it to ignore what was added with 800296 or a bug in our deployment logic that we're not deploying the version of that file which is triggering the deployment22:39
fungisimilar changes have taken effect before now, so yes i've got a strong suspicion we're "eventually consistent" by applying changes on the run after the run triggered by their merging22:40
fungiand related to that, i'm wondering what else touched the config file since it seems to have been something other than the infra-prod-run-accessbot job22:41
fungimaybe we have multiple deploy jobs writing that file and they interfere with one another?22:41
fungii only see playbooks/roles/accessbot/tasks/main.yaml in system-config writing it though22:43
fungiso maybe we have more than one job running that role22:44
clarkbI think we have an hourly job or maybe daily?22:49
fungiyep, it's infra-prod-service-eavesdrop doing it22:49
fungijust found it in the log22:49
fungithough do those jobs share a semaphore?22:50
fungii guess not, because they're triggered from different projects in different pipelines22:50
fungiso i suppose it's possible for them to race one another22:50
fungiboth infra-prod-run-accessbot and infra-prod-service-eavesdrop install /etc/accessbot/channels.yaml on the server, but only infra-prod-run-accessbot also runs accessbot22:52
clarkbfungi: we can probably make them share a semaphore?22:53
fungiso if infra-prod-service-eavesdrop started shortly before the change merged but wrote what had been the master branch copy of the config between when infra-prod-run-accessbot wrote it and when it actually ran eavesdrop (there's an image pull and other stuff which happens in between those steps so could be a lengthy window) we'd end up applying a stale config22:54
fungiall of this also happened right around the time of the zuul restart and reenqueue22:56
fungiwell, maybe not quite. the task to write that file in the run job logged at 19:26:10 and the zuul restart was 40 minutes later22:57
fungithere was an hourly run of infra-prod-service-eavesdrop which logs writing the file at 18:45:36 and again at 21:45:07 but we don't have any logs for two more runs which would have happened between those23:00
fungiso those were probably delayed by the zuul restart23:00
clarkbthe 20:00ish run would've been caught by the restart23:00
fungianyway, i don't have a smoking gun showing they were both writing the file around the same time23:01
fungitrying a manual run of accessbot now to see if it applies the current configuration23:04
fungiyep, it's applying it correctly23:07
fungithe #opendev-manila access list just jumped from 11 to 18 entries23:08
fungithe log from the previous run showed it only checking the entries from our global admins/ops for that channel, so i'm fairly certain it ran with an old copy of the channels.yaml23:09
fungimaybe this isn't a race...23:10
fungione thing i don't see the infra-prod-run-accessbot job doing is pushing an updated project-config onto the eavesdrop server23:10
fungiwhich i think would explain this behavior?23:11
fungiyeah, there's a sync-project-config task which happens in infra-prod-service-eavesdrop but not in infra-prod-run-accessbot23:13
clarkbya that could do it too23:14
opendevreviewJeremy Stanley proposed opendev/system-config master: Sync project-config before deploying accessbot
opendevreviewClark Boylan proposed opendev/system-config master: Preserve zuul executor SIGTERM behavior
fungithat would have been much easier to spot if infra-prod-service-eavesdrop hadn't run, since i would have seen the stale config and stale copy of project-config on the server23:23
clarkbNow we won't forget to do that23:23
fungithat should be fine to merge sooner, since the executors currently ignore that envvar anyway23:25
clarkbthats true. I put the depeonds on in there anyway though23:27
fungithat at least ensures we don't wind up needing to amend it23:30
fungiso fine by me23:31
corvusmordred, fungi: i'm i'm working on a container image for this eavesdrop bot; i need libolm3 but i only see libolm2 in our python-builder (debian buster i think?)23:55
corvusshould i add some backport thing, or clone+install from upstream repo?23:56
corvuslooks like it may be in buster-backports, so maybe i just add that to sources.list?23:58
fungicorvus: yep, i just confirmed buster-backports has it23:59

Generated by 2.17.2 by Marius Gedminas - find it at!