Tuesday, 2024-02-13

corvustonyb: which ze did that job run on?01:50
tonybI'm not sure.  I for the logs by looking at them all `for i in 01 ..... 12 ; do ...`01:52
tonybHow'd you find that so quickly? ... just experience?01:54
tonyb*just* .... like its a small thing01:54
corvusoh nope not that one01:54
corvusyou can do an ad-hoc ansible command to grep01:55
corvusit's ze0801:56
corvustonyb: the exceptions don't end up with build tags, so if there is one, you have to read the log and search for it01:57
corvus2024-02-13 00:29:53,859 ERROR zuul.AnsibleJob:   Exception: Variable names may only contain letters, numbers, and underscores01:57
tonybAhh okay.  That makes sense01:59
tonybI think maybe I found it.02:01
*** tosky_ is now known as tosky09:48
*** Adri2000_ is now known as Adri200013:42
opendevreviewRodolfo Alonso proposed openstack/project-config master: Implement "neutron-unmaintained-core" group  https://review.opendev.org/c/openstack/project-config/+/90891115:32
opendevreviewMerged zuul/zuul-jobs master: Introduce LogJuicer roles  https://review.opendev.org/c/zuul/zuul-jobs/+/89921215:41
TheJuliao/ folks, looks like https://zuul.opendev.org/t/openstack/status has a hung job on queued docs..... any insight or is there any way to clear it out?15:57
clarkbTheJulia: generally those get stuck due to being unable to provision the node type for the job. I've got a meeting now but can look at logs afterwards to see exactly why it is stuck16:01
opendevreviewMerged zuul/zuul-jobs master: Add zuul-tenant-conf-check role/job  https://review.opendev.org/c/zuul/zuul-jobs/+/90736316:09
clarkbthe status page does say it is waiting on a node request. If I had to guess this is fallout from the restarts of nodepool related to some recent bugfixes to nodepool. cc corvus I think we may have seen this before where a node request gets lost but it happened over the holidays and so we didn't have logs for it.16:31
clarkbcorvus: I haven't dug up logs yet though, but my hunch is this is that same issue. This time we should hopefully have logs since it happened recently16:36
clarkbThe rax-dfw provider is trying to process the node request and has been since 2024-02-12 20:09:30,572 DEBUG nodepool.PoolWorker.rax-dfw-main: [e: 4369e4e61c504735bc2f32c589152357] [node_request: 300-0023568458] Locking request16:58
clarkbIt has been failing in a loop on not enough quota remaining so it pauses and retries16:58
clarkbrax-dfw is not launching any nodes16:59
clarkbdoing a server list against that provider is taking a very long time...17:02
clarkbtrying to double check the values grafana and nodepool are reporting against what the cloud reports17:02
clarkbI suspect we may need to set max-servers in this region to 0 though while we figure this out17:02
clarkbok ya there are many servers in server list that are not known to nodepool. I'll try manually deleting them first and if that doesn't work we can set max-servers to 0 and file a ticket17:04
fungichances are they're undeletable and we'll have to get rax support to clean them up. not the first time this has happened in the past few months17:05
clarkbthere are a few listed as active and I'm startign with them rather than the ones already in error or deleting17:06
clarkbnodepool reports all the servers it knows about in that region should be deleting already so should be safe to delete anything off the list I generated. Now that I'm trying to delete stuff new servers may be booted though17:06
funginote that you should put in a wait between asking nodepool for a list of instances it knows about and then asking the cloud for a list17:10
funginodepool may be waiting for instances to come active, and won't know the uuids for those until they boot, so you could race it17:11
clarkbya in this case it has been steady state for a while (almost 10 minutes)17:12
clarkbalso I don't think any of the 30 something deletes I've done have made any changes17:13
clarkboh maybe I'm wrong17:14
clarkbI see the building graph jumping up17:14
clarkbI guess I'll continue with the deletions17:14
corvusclarkb: do they look like leaked nodepool nodes?  do they have any metadata?17:26
clarkbcorvus: yes they are all named np000XYZ...17:26
clarkbI'll check the metadata on a server after the current mass delete ends (enough are failing I'll still have examples(17:26
corvusclarkb: then my next question (assuming that there is no metadata) is whether these nodes were ever used. as in: if the cloud is messing up the metadata association, is it on creation (so we never got those nodes to be used) or on deletion (we used them and told the cloud to delete, and it deleted the metadata but not the node)17:28
clarkbTheJulia: the cleanups I'm doing above allowed your node to be scheduled and you should have a zuul report on the change now17:28
corvuscan probably answer by grepping for one of those node ids17:28
clarkbcorvus: looks like some are deletes failing because multiple nodes have the same name17:28
TheJuliaclarkb: thanks!17:28
clarkbwhich is a surprising state considering that nodepool largely runs single threaded per provider?17:29
clarkbcorvus: I'm not sure I'll have time to do that debugging for a while. I have two more meetings this morning. One of which I'm running17:29
TheJuliafrom a 30k ft guess, I bet something told it was deleting but it didn't actually get deleted17:29
corvusclarkb: we'll reuse the name if we retry launching a node.  but we perform delete operations based on nova id so it shouldn't care17:29
clarkbya also server list doesn't show me duplicates17:30
clarkbso wherever that duplicate is isn't exposed to us17:30
corvusTheJulia: that sounds likely, but we expect lies like that from the cloud so we also use metadata to detect leaks.  the worst thing is if the cloud deletes the metadata but not the instance (or conversely, creates an instance and doesn't attach metadata).  the only resolution to that is human intervention.17:31
clarkbcorvus: 8624c16c-97a0-435b-93e3-dfcf048f4e6a is np0036475154 and openstack says there is a duplicate. There is no metadata on this server17:32
clarkb10f0b06b-7019-4792-a3db-b76ab2ab2f3f was np0036473596 and had metadata. I manually deleted it though17:33
clarkbMakes me wonder if nodepool isn't able to delete things that have duplicates for some reason17:33
clarkbsince it should be a deletable node with the metadata17:33
corvusclarkb: if it had metadata and wasn't detected as a leak, that sounds like a nodepool bug worth exploring.  might want to keep it around if you find another with metadata.17:33
clarkbcorvus: ack17:34
corvusclarkb: i don't see any info about the lifecycle of np0036475154 before we started trying to delete it; i expect those logs are gone17:36
corvusgrep 0036475154 /var/log/nodepool/*|grep -v Exception|grep -v cache|grep -v Delete|grep -v delete is what i used17:36
clarkbcorvus: ya the newest one I've got is from january 2117:36
clarkbI think we may have rotated logs on most/all of these17:36
clarkb67ad5b7b-1667-4923-8470-1b274460fba0 is the newest one np003647805917:37
clarkbit was created at 2024-01-21T11:24:29Z, had a fault at 2024-01-21T12:13:39Z, then was last updated at 2024-01-21T12:15:14Z17:37
corvusmy question about the lifecycle is mostly curiosity / to help characterize what i expect is a cloud failure and not that important.  but any ongoing failure to delete leaked nodes with metadata is potentially actionable.17:37
TheJuliacorvus: still seems like that could be reconciled out of potentially :( Anyway, Thanks!17:37
corvusTheJulia: with heuristics we could automatically delete leaked nodes without metadata, but considering how quickly this system can do great damage, we avoid doing stuff like that with heuristics.17:38
fungiTheJulia: a common problem we've run into is that something times out between bits of the cloud during server creation, and the metadata nodepool relies on never gets added to the instances17:39
corvusa mistake there could delete the entire dev infrastructure in a blink.  :)17:39
TheJuliacorvus: different tenants maybe :)17:39
TheJuliafungi: ugh17:39
corvusTheJulia: opendev uses different tenants, generally, for that purpose.  but not everyone is as lucky as opendev in that regard, and even opendev in some cases has had trouble getting multiple tenants from clouds.  we avoid using those clouds for infrastructure.17:40
fungiactually, thinking back to when i looked into it last time, openstacksdk adds the metadata as a separate api call? so if it gives up waiting for the server instance then the metadata is never added17:40
fungithough i may also be confusing this with similar leaks we see with image uploads17:41
TheJuliacorvus: true17:41
clarkbcorvus: np0036478700 is also from january 21, but it has metadata17:41
clarkbcorvus: though looking at it it may be a stuck delete /me looks at nodeppol17:41
corvusclarkb: yeah it'r trying to delete that one still17:42
clarkbya it is17:42
clarkbI'm going to try manually deleting it since we know nodepool is trying now17:42
clarkbI don't expect it to actually go away but worth a shot in case the openstack client gives me any useful info17:42
corvus++ nodepool is only getting a timeout17:43
corvusclarkb: any nova error state set?17:43
clarkbcorvus: {'message': 'MessagingTimeout', 'code': 500, 'created': '2024-01-21T16:08:45Z'}17:44
clarkbits possible that we need an admin to reset the task staet so that a delete will actually be reattempted17:44
clarkbcorvus: I found one e3f0d035-71f9-4bdb-8a5d-e576de2ead87 is np0036476592 (duplicate reported) and has metadata but is not being deleted by nodepool17:45
clarkbcorvus: possibly because the state for this node is DELETED. Maybe we need to retry deletes until servers stop being listed?17:45
clarkbcorvus: so I suspect that the vast majority of these are a cloud side issue but maybe this subset is possible to have nodepool cleanup?17:45
corvusclarkb: that should already be the case17:46
corvuslemme look up the zk record for that one17:46
clarkbok I need to context switch now. There are quite a few nodes that are still leaked. I'll make a record of them in my homedir on bridge to distinguish them from the running nodes17:47
corvusclarkb: oh i see, the cloud state is deleted; yeah i think our understanding of that is that it doesn't affect quota and we can ignore it, so we don't try deleting nova nodes that are deleted17:48
corvusif we think this is a problem, we could probably do as you suggest and keep deleting the deleted nodes17:48
clarkbcorvus: I guess it isn't clear to me if that node is counting against our quota. But it continues to show up in listings several weeks later. I think it might be a good idea to keep trying simply to clear out the noise as much as possible making it easier to see the actual problems (assuming they don't count against quota)17:49
clarkbcorvus: nodes in a DELETED state are the only ones that have leaked that I see with metadata so far fwiw17:49
clarkbok file with list of leaked nodes is in my homedir if anyone else wants to look at it and avoid touching nodes that may be in use17:50
corvusclarkb: i don't disagree, but one thing to consider is it's trading one kind of noise for another (noisy nodepool logs)17:52
fungiokay, so there are two categories of leaks present: those stuck in deleting/error and those missing metadata?17:52
fungithat's fun17:52
clarkbfungi: three I've seen so far. deleted with metadata now ignored by nodepool intentionally, error with metadata not ignored by nodepool but cloud fails to delete them, and error/active/build with no metadata that nodepool ignores as a result. The first and third categores have been manually deletable if you use uuids to get around duplicate node complaints17:53
corvusif our previous assessment that nova-deleted nodes don't count against quota still holds, then category #1 is only an admin annoyance, but not an operational impediment.  if that doesn't hold true anymore then i definitely think we should continuously delete them17:56
clarkbwell #3 is a problem too because it uses up our quota then jobs get stuck. But I'm not sure nodepool can do much to solve them (so admin problem not a nodepool problem)17:58
corvusyes, i was trying to clarify that those 3 classes of "leaked nodes" may not all be causing operational issues.  #3 clearly is and i think #2 probably is (but less sure), and i think #1 is not (unless something changed since the last time we evaluated that behavior)18:00
fungiunrelated to anything else, https://zuul.opendev.org/t/openstack/status shows a change which is about to clear the gate, and has a null queue name for it18:05
fungiwondering what could lead to that18:06
clarkbcorvus: gotcha18:06
corvusfungi: not assigned to a named queue, so it's only in its automatically created per-project queue18:09
fungicorvus: interesting, other changes show the project name as the queue name in that situation rather than just a blank18:09
corvusmight be due to the backwards compat handling?18:10
fungiin this case it was for an openstack/openstack-ansible change (now it's merged so no longer showing there), but i couldn't find any queue name set in the project's config18:10
opendevreviewLajos Katona proposed openstack/project-config master: Implement "neutron-unmaintained-core" group  https://review.opendev.org/c/openstack/project-config/+/90891118:36
fungii just noticed https://github.com/lxc/lxc-ci because someone announced plans to package it in debian... looks very similar to dib, but focused on just creating lxc images instead of virtual machine images18:47
clarkbinteresting that lxc wouldn't do dockerfile like builds (d0on't have to use docker)18:50
fungithey also don't seem to have as much of a mix-in model like how dib elements work, and more repetition between related distros as a result18:53
opendevreviewMerged opendev/system-config master: Check launched server for x86-64-v2/sse4_2 support  https://review.opendev.org/c/opendev/system-config/+/90851219:17
opendevreviewMerged opendev/zone-opendev.org master: Switch the keycloak CNAME to the new server  https://review.opendev.org/c/opendev/zone-opendev.org/+/90835719:24
opendevreviewJames E. Blair proposed zuul/zuul-jobs master: Remove command.warn usage  https://review.opendev.org/c/zuul/zuul-jobs/+/90867119:27
opendevreviewSteve Baker proposed openstack/diskimage-builder master: Add setuptools for python3.12 support in venvs  https://review.opendev.org/c/openstack/diskimage-builder/+/90249719:30
fungihaving not used the zuul web admin access before now, is the "sign in" button in the top-right corner supposed to do anything? i tried it a few times a while back and it doesn't seem to ever do anything when i click that20:06
fungitried it from multiple browsers too, i don't think there's a popup blocker breaking it20:07
fungii would have expected it to send me to the configured openid provider20:08
Clark[m]Did we update the URL to drop /auth/ yet?20:09
Clark[m]It may be doing a request in the background that fails. Browser debugger may help20:09
tonyb404 openid-configutation.js20:10
opendevreviewMerged opendev/system-config master: Update Zuul auth config for new Keycloak images  https://review.opendev.org/c/opendev/system-config/+/90835320:11
tonybsorry the 404 is on openid-configuration XHR from oidc-client.min.js:120:13
tonyb```status#system-config:1 Access to XMLHttpRequest at 'https://keycloak.opendev.org/auth/realms/zuul/.well-known/openid-configuration' from origin 'https://zuul.opendev.org' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource.```20:15
Clark[m]The /auth/ prefix is at least part of the problem. Maybe it sets cors headers when hitting the valid path?20:16
tonybYeah I get data without the /auth/ iun the url.  I admit I have no idea how to test/debug CORS stuff20:17
fungiokay, so in theory it will work once 908353 is in use20:17
fungiwe probably need to restart zuul-web for that?20:17
fungisince the change is to zuul.conf, which i don't think gets read live20:18
fungihuh, when did zuul start leaving comments like "1 is the latest approved patch-set. No files were changed between the latest approved patch-set and the submitted20:20
funginote the required votes were all on patch set #220:20
fungiwas that part of the circular deps refactor?20:21
Clark[m]That's from Gerrit I think 20:22
Clark[m]All zuul is doing is clicking the submit button and then Gerrit records a comment on your honor20:22
fungioh, right that's the gerrit comment on zuul's behalf20:22
fungiso anyway, 908353 hasn't deployed yet. once it does i should restart each zuul-web container in turn?20:24
Clark[m]Yes I think that would be the next steps20:24
tonybfungi: Yup sounds good to me20:24
fungiit deployed seconds after i said that20:29
fungiissuer_id looks correct in the configs of both servers so downing/upping their containers one at a time now20:30
fungimmm, i probably should have waited longer between those20:32
fungisorry about that20:32
tonybI get redirected to keycloak now when clicking the 'sign in' button20:37
fungiyep, the webui seems to be back up and working again20:38
fungiand yes, the sign in button is also working for me20:38
fungiunfortunately, when signing in with my account credentials, i get "login in progress, you will be redirected shortly..." for what seems like is probably forever20:41
opendevreviewJeremy Stanley proposed opendev/system-config master: Document adding Zuul WebUI admins  https://review.opendev.org/c/opendev/system-config/+/90894920:44
fungieverything's working for me up to the very last step, signing into zuul20:44
fungii probably need to check the zuul-web logs for errors20:45
fungifor me it's spinning forever on the https://zuul.opendev.org/auth_callback that keycloak redirects to20:49
tonybeeek gotta do the school run20:50
fungiokay, javascript console in my browser says this:20:54
fungiCross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at https://keycloak.opendev.org/realms/zuul/protocol/openid-connect/userinfo. (Reason: CORS header ‘Access-Control-Allow-Origin’ missing). Status code: 200.20:54
fungiis that going to be on the zuul side or the keycloak side?20:56
clarkbwhich server respodned with that error?20:57
clarkbyou should see it in the network debugger with clearer network paths20:57
fungiaha, thanks, i'm entirely unfamiliar with debugging browser-based stuff20:58
fungiaside from basic html/xhtml/sgml20:58
clarkbbut I think that may be zuul saying it blocked the content from keycloak based on the same origin policy20:58
clarkbI'm not sure how to address that. I would be something in react maybe?20:58
fungilooking at the network trace, the last response is from zuul yes20:59
fungii wonder why it wasn't a problem with the old keycloak server20:59
fungii see we explicitly set Access-Control-Allow-Origin in our vhost configs for gitea, graphite and jitsi-meet21:00
clarkbbasically I think the webbrowser is saying zuul.o.o can't trust keycloak.o.o because of the policy. Which si slightly different than the CORS policy which is the server saying "you acn use this elsewhere"21:00
clarkboh except maybe keycloak needs to respond with access-control-all0w-origin ?21:00
fungihttps://zuul-ci.org/docs/zuul/latest/howtos/openid-with-keycloak.html#create-a-client covers setting "web origins" in keycloak, which should cover that21:01
clarkbyou should see that in your network trace21:03
clarkbfor the request responses from keycloak21:03
fungiweb origins for the client i created in the zuul realm is set to https://zuul.opendev.org/21:04
fungidescription for that field in the form is "Allowed CORS origins. To permit all origins of Valid Redirect URIs, add '+'. This does not include the '*' wildcard though. To permit all origins, explicitly add '*'."21:04
clarkbI woudl double check you see that in your web browser dev tools network trace21:04
fungiit'll be in a response header, right? still trying to figure out how/where it exposes those21:06
clarkbif you click on a line entry in the network trace it should open a more details very for that request and response21:06
clarkbthat will include header info21:07
clarkb*more details view21:07
fungiaha, thanks21:07
fungireferrer policy is strict-origin-when-cross-origin21:07
fungiis *21:08
clarkbwhich isn't https://zuul.opendev.org/21:09
clarkbbut also should be sufficient to allow things to happen21:09
fungithat's coming from zuul.o.o though21:09
clarkboh you need to find the requests to and from keycloak21:09
fungiaha, yeah i had to redo the login rather than just refreshing21:12
fungiit reports the origin as https://zuul.opendev.org but no Access-Control-Allow-Origin header21:13
clarkbok that is likely the problem. We need the access control headers for the browser to do the right thing21:13
clarkbmaybe there is a different setting filed in keycloak now that we have to set?21:14
fungihowever, an earlier request to keycloak.opendev.org is returning "Access-Control-Allow-Origin: https://zuul.opendev.org"21:14
fungijust not the request that it's getting stuck on21:14
fungihttps://keycloak.opendev.org/realms/zuul/protocol/openid-connect/certs has the expected header, https://keycloak.opendev.org/realms/zuul/protocol/openid-connect/userinfo does not21:16
clarkbhttps://keycloak.discourse.group/t/access-control-allow-origin-header-missing/328/28 says they had to remove trailing slashes from the origin21:17
clarkbthat seems unlikely to be the problem, but maybe that is it?21:17
fungithat was entirely it. thanks21:19
fungii'll adjust my docs update for zuul21:19
clarkbthat seems like a bug in input validation for keycloak if they don't want to accept a trailing /21:19
clarkbI've just sent the ansible 6 removal announcement email21:19
fungiinfra-root: new keycloak server is now in production, instructions as to how to add your user are provided by https://review.opendev.org/908949 (and eventually by our system-config docs once that merges)21:25
clarkbthanks I'll review that change and the zuul docs update now21:26
clarkbfungi: mhu has some comments on the zuul change21:27
fungihttps://github.com/keycloak/keycloak/issues/25522 seems to be the corresponding bug report for that21:27
fungiclarkb: thanks, forgot to address those when updating21:30
fungishould be covered now21:30
clarkbI'm going to pop out soon for a bike ride. We're going to get hit by another atmospheric river/pineapple express tomorrow and I can see blue skies right now21:30
clarkbjust a heads up I'll be afk for a bit this afternoon to take advantage of the weather21:31
fungiwe very narrowly escaped flooding today from the wind storm that just plowed through21:31
clarkbis that the same storm as the noreaster hitting the areas north of you?21:31
fungiprobably. a few inches deeper at high tide and we'd have had some interior cleanup to deal with21:31
clarkbwhen I get back I'll look into the rest of that rax dfw cleanup that we can do without admin perms21:32
fungithankfully it only just reached the edge of our patio on the waterfront side21:32
clarkbI finally made it up into the hills over the weekend and the destruction from our ice storm last month is still very evident in areas that were hit hard. One very large multistory home had an entire corner just sliced/ripped off21:33
clarkbthe massive rootball of a tree near the house gave me an idea of how big the tree must've been to do that21:33
clarkband there is still a fairly large tree hanging off utility lines next to the commuter trail21:34
fungioh yes, trees are extremely strong, but add an inch-thick coating of ice to their branches and they topple like (insanely heavy) toothpicks, taking out everything in their path21:34
fungiespecially if the soil is also wet and compromised by the same storm21:35
clarkbyup pretty classic scenario for trees coming down. There is a big push in our local media to try and rescue the reputation of trees21:36
clarkbthere is concern that everyone will chop down all the trees now. But they provide a lot of benefits like defending against urban heat islands and so on21:36
fungialso preventing or at least limiting erosion21:37
fungichop down all the trees, and you guarantee mudslides. saw it all over growing up, on mountainsides that had been clearcut21:37
clarkbya and trees work together to block wind. If you thin them out too much those that are left may be at higher risk21:38
fungionce the trees are gone, there's nothing to hold all the soil and bounders onto the substrate. so it plows downhill at insane speeds and turns whole subdivisions into parking lots21:38
clarkbits all about balance21:38
clarkbalright working on popping out now. Back later21:42
funginext (non-urgent) question for those who have used the admin functions of zuul's webui: the docs suggest that after logging in i should see the option to create an autohold, but where? it shows me logged in (displays my username in the top-right corner), yet neither the tenants list nor the autoholds page for a tenant seems to give any such option. what am i missing?21:44
fungithe docs suggest it should appear on the tenant-specific autoholds page21:45
fungimakes me wonder if i missed adding some sort of authorization, and by default my account is unprivileged21:48
corvusfungi: yeah, in opendev we set up an expectation that there would be groups with the tenant name and you'd be in that group22:23
corvusfungi: to have admin perms, you either need to be in a group with the exact name of the tenant, or in a group named `infra-root` you probably want the latter22:25
corvusi'm going to create the infra-root group now22:31
corvusand we need a mapping too; i'll set that up22:38
fungicorvus: oh, thanks! that would explain exactly what i saw. where is that expectation reflected in the zuul configuration? somewhere i'm overlooking in zuul.conf?22:56
corvusfungi: in main.yaml -- the tenant config22:57
fungiaha. i was looking in entirely the wrong place22:58
corvusfungi: okay i did 2 things: first i added a mapper for groups like we just discussed: https://keycloak.opendev.org/admin/master/console/#/zuul/clients/b4ed13af-2692-4821-a06e-03a2f356b7f3/clientScopes/dedicated22:58
fungii should probably mention this in https://review.opendev.org/c/opendev/system-config/+/908949 too22:58
corvusthat's specific to how we want to set up authz for opendev, so yeah, that probably belongs in that doc but not the zuul tutorial22:59
corvusthere was one other thing missing that probably belongs in the tutorial; after creating a client scope for the audience, you need to add it to the zuul client config22:59
corvusfungi: that's here: https://keycloak.opendev.org/admin/master/console/#/zuul/clients/b4ed13af-2692-4821-a06e-03a2f356b7f3/clientScopes23:00
fungiawesome. i can cover the the custom mapper/group creation in keycloak.rst and adding users to the infra-root group in sysadmin.rst, i guess23:01
fungii'll do that tomorrow between pre-ptg sessions23:01
corvusfungi: basically on client details page, go to "client scopes" tab, click "add client scope button" "check zuul_aud" click "add" "default"23:01
fungiperfect, i'll update the zuul docs change with that too. thanks!23:02
corvusfungi: alternatively, i think you could do what i did with the groups, which is rather than defining a standalone client scope and then adding it to the client; i think you could add a dedicated client scope.23:02
corvusfungi: since i hadn't fully explored this, we now have a standalone scope for zuul_aud and a dedicated scope for groups; if that bothers anyone else's ocd, we might want to pick one of those two styles and stick with it23:03
corvusi'm fine with either23:03
corvusfungi: since i'm here, i added you to the infra-root group23:03
corvusi'm done with the keycloak admin ui now23:04
fungithanks corvus!23:04
fungisigning out of and back into zuul, i see a link to create an autohold now23:05
fungii also have dequeue and promote options in the status view23:06
fungii'll get the various docs changes fixed up for this tomorrow23:06
corvusthere's also a little wizard hat if you open up the user info dialog23:06
corvusshould say "Logged in as: fungi <wizard hat>"23:07
fungii see the hat! i have a hat!23:07
fungivery cute23:07
fungicorvus: the doc in zuul already does cover adding zuul_aud to default. did it still show up as "none" for the assigned type?23:10
fungithe ui around that part seems to have changed somewhat between the version the original steps were written for and now, so i had to adapt it a little, but maybe i misunderstood what i was looking at and adapted incorrectly23:11
fungior maybe i just missed clicking a "save" button somewhere there23:12
corvusi didn't see zuul_aud added as a client scope to the zuul client; the zuul client had no client scopes; yeah maybe a missed save23:12
corvusfungi: hrm i see the text about adding it as a scope, but i think it's missing an explicit "Add" step23:13
corvusnot quite sure how to reconcile that; might need a clean run through?23:13
fungioh, got it. yeah this talks about setting the assigned type for the zuul_aud client scope to default23:13
fungiin the client scopes list23:14
fungii do still have a held node i can override my name resolution for and go through the same setup steps on23:14
corvusyeah; when you add it, you also select whether it's default or not (i did select default); it almost seems like that's an edit step that's missing the add23:14
fungibut can't easily test external interactions from zuul with it of course23:15
corvusor maybe the docs are correct for adding a dedicated scope, but you did a standalone scope instead in prod?23:15
corvusfungi: you can decode the jwt and check the "aud" field manually23:15
corvusit should have "zuul" in it; without the mapper it just says "account"23:16
fungioh, looking at the diff in 908855 i think i did indeed misinterpret the instructions there when adapting it to the current version. i clicked "client scopes" in the left sidebar and changed the assigned type for zuul_aud there, rather than picking "clients" in the left sidebar and going into the zuul client to change zuul_aud there23:18
*** dmitriis is now known as Guest268323:44
clarkbI've kicked off a deletion process which should delete the reamining nodes that we can delete from the rax dfw nodepool provider23:51
clarkbthere are ~25 that have stuck around23:56
clarkba big improvement but still enough that we'll probably want to file a ticket ot see if rax can clear out the remainder23:56

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!