Friday, 2023-03-17

ianw++ yeah i can see all that in the logs too00:00
clarkbianw: maybe read through the quay log jus to make sure I didn't do anything obviously dumb :)00:01
clarkbbut ya I think this should work00:03
clarkboh also to be clear the token I created was just for creating repos perms00:03
clarkbI didn't end up needing any additional permissions in that token to get this working00:04
fungiso you can push images as a robot, but precreating the repo requires a token?00:04
clarkbfungi: correct. YOu can also `docker image push` as the robot to create the repo but when you do this its default visibility is private not public00:05
clarkbfungi: this means an alternative approach would be to use the robot to create a private image then use the token to update the visibility to public00:05
fungiinteresting that a robot account is able to autocreate a repository but robot credentials can't precreate one00:05
clarkbtheir api is definitely a little odd. I assume the reason for this is the api has a completely different authentication implemtnation to the docker api part of it?00:06
fungiso we'll need two secrets per org i guess00:06
clarkbwell no00:06
funginot including our own accounts00:06
clarkba single secret with two encrypted attributes00:06
fungioh, sure that would work00:06
clarkbthe API token and then the regular docker robot auth00:07
clarkbalso I can look at it with fresh eyes tomorrow and see if I can find a way to do everything with the robot as that is definitely the ideal00:07
clarkbhowever since repo creation only happens once its not the end ot the owrld to do it the way I describe above00:07
clarkband now I need to figure out dinner00:09
ianwsince we have no private repos on the plan, i wonder how the push works if it doens't make it public?00:11
ianwseems like something i could test :)00:11
clarkbianw: ya I didn't get that far and just figured out how to make things public. Note you can also toggle them from private to public via the web ui00:12
clarkbbut once you do that you cannot toggle it back to private since we don't have perms. I guess the testing would reveal if the image is accessible anyway?00:12
ianwi can see  Repository Visibility , which says "you can't make this private"00:13
ianwbut not sure i can see one for the image00:13
clarkbianw: the repository is basically the image I think they are the same concept with the terms we've been using00:13
clarkbthey distinguish because a tag in a repo is an image and that may be different than another tag against the same repo name00:14
ianwoh right, yeah it says "repository tags"00:14
clarkbat least that is why I think they do that00:14
ianwso yeah at least i can't see that the repo can be toggled to private, or at least back to private00:14
clarkbianw: if you want it to start out as private you have to `docker login #give it your creds when prompted this is a generated password in your user account settings then docker image tag somesourceimage:sometag then docker image push
clarkbif you create via the web ui it asks if you want private or public but it won't let you do private without upgrading your account (it chagnes the submit button to upgrade now)00:16
ianwthat does say "Identify repositories and images that the robot can push images to or pull images from"00:22
ianwbut not "create repositories"00:22
ianwwhich is i think what you were saying :)00:22
Clark[m]Well I think it has create perms too? But I'm on the phone now and can't check. We/I can test creating with docker image push as the robot tomorrow 00:25
ianwwell it certainly is true that the repo starts private, you switch it to public and then can't switch it back again00:46
ianwit is intentional ->
opendevreviewJames E. Blair proposed opendev/ master: Add
opendevreviewJames E. Blair proposed opendev/system-config master: Add SSL cert for
opendevreviewJames E. Blair proposed opendev/system-config master: Add static site for
corvusinfra-root: 2 years ago, the last time we talked about moving to quay, i did some experiments that suggested that a 301 redirect for container images might be possible.  we actually discussed doing as a redirector to dockerhub/quay.  given the uncertainty around the namespace, and the possibility that we might have to move again in the future, i would like to proceed with that experiment now.01:43
corvusi'd like to create in dns, and point it to a static site with a .htaccess file and go ahead and test it redirecting to dockerhub now.  if it works, then i'll propose we actually make that the canonical location for zuul images.  we could consider doing the same for opendev as well. if it doesn't work, i'll clean it up.01:44
corvusinfra-root: i'd like to proceed quickly with that, so if you can go ahead and review those changes, that'd be great.  i can +w them and supervise.01:45
ianwcorvus: it does feel like we could put the redirect directly in the config file and gate test with a pull ... i presume you're thinking the .htaccess so you can quickly fiddle it?01:48
ianwok, so i've at least caught up to where everyone else is on the quay situation i think01:54
ianwthe robot account can push new repositories/images via docker.  confirmed that if it pushes a new repository, then it's private by default01:55
ianwhowever, the robot account can not access the quay API01:55
fungii went ahead and approved the dns addition and the ssl cert creation01:57
ianwto access the quay API you need an oauth application.  users can then create an oauth token in that application; and then API operations with that token are done by the application "on behalf" of the user that created the token01:58
Clark[m]Yup I linked a thing about that I think and that is why I named my application with my name01:59
ianwso instead of pre-creating the repo, we could also use
ianwto flip it.  either way, if it's done by pre-creating a public repository, or flipping the bit on one already pushed, you need the oauth token01:59
corvusianw: exactly -- and i agree, if it's simple, let's move it into config.  if we end up needing a redirect for each repo, probably best to put it in an afs volume for easy project-level maintenance.02:00
ianwok, i think i'd prefer to see it checked in either way, and ideally a pull or at least a listing or something via testinfra to confirm changes, but for POC sounds good02:03
opendevreviewMerged opendev/ master: Add
corvusianw: absolutely.  basically my test 2 years ago was "this looks like it might work based on local testing, but needs validation with a site with a real ssl cert".  i promise if it works, the actual redirects will end up in a config managed repo, one way or another.  :)02:05
fungilgtm then. approve at your convenience02:07
corvusjust in case you were wondering... chatgpt doesn't know if this will work.  but it's pretty sure that we can just proxy image pulls pretty easily!  ;)02:15
ianwcoruvs: just be sure you blog clearly and explicitly about the results so that it can be fed back into the machine02:21
* NeilHanlon has an idea for a (satirical?) blog post about writing for ChatGPT02:34
fungitry not to taunt it about bingbot's lobotomy03:22
fungiprobably a sensitive subject03:22
opendevreviewJames E. Blair proposed opendev/system-config master: Add static site for
corvusmixed up my - and .03:30
opendevreviewMerged opendev/system-config master: Add SSL cert for
opendevreviewMerged opendev/system-config master: Add static site for
opendevreviewIan Wienand proposed opendev/system-config master: base: split ssh setup
opendevreviewIan Wienand proposed opendev/system-config master: base: split ssh setup
*** jpena|off is now known as jpena08:24
dtantsursomeone asked me back at the time for an example of a patch where the gertty crash is reproduced. from today:
dtantsursame on, I wonder what is common between them, is it because they're from one chain?10:58
fungidtantsur: i just loaded 877387 in gertty (tip of master) and it didn't crash on me. scrolled all the way to the end of the comments and the end of the diff. what actions trigger it for you?12:18
fungisame for 87738412:19
dtantsurscroll down key12:19
fungidown arrow or pgdn? if it's the latter, i think i narrowed that down to some sort of regression in a newer release of urwid12:20
dtantsurpagedown, yeah12:20
fungilots of changes crash gertty if i use the pgdn key, something about an element not accepting keypresses12:20
fungithough i'm not getting it to trigger on those changes at the moment12:22
fungiit may be a combination of particular changes at specific terminal geometries. i use gertty in an 80x24 terminal (80x25 minus the tmux status bar)12:23
fungidtantsur: what gertty version are you using? a release or from git?12:26
fungii'm also using urwid 2.1.2 with it, under python 3.11.2 currently12:28
fungitrying to see if the behavior has recently gone away for me and maybe i didn't notice12:28
fungiyeah, at the moment no luck, software never crashes when i want it to :/12:31
dtantsurgertty==1.6.1.dev50, urwid==2.1.212:38
dtantsuralso 3.11.212:39
opendevreviewMerged zuul/zuul-jobs master: Remove ignored success-url job attribute
Stephen_OSAgood morning15:21
Stephen_OSAI have a question about heat container agent, where might the best place to ask that question be?15:21
clarkbStephen_OSA: this channel is mostly dedicated to the developer tools used to build software like heat. We don't actually know a ton about heat itself as  result. Your best bet is probably in #openstack ?15:24
clarkbor in #heat I guess15:24
clarkb(I was expecting it to be #openstack-heat and couldn't find it for a minute)15:24
dpawlikclarkb, fungi: o/ I would like to upgrade the Opensearch cluster in 2 weeks ( 03.04.2023 at 12:00 PM UTC). Are you ok with it? Soon I will send an email15:31
dpawliknormally I see in the AWS console, that the "upgrade" option is available, but I don't know if after clicking it would be working15:32
dpawlikso just small question if you are available at this time, if I don't have permission, you will just click :)15:32
clarkbdpawlik: The openstack release is next week and then the ptg the week after that. I would expect that doing it after those events is a good time. But don't really have a personal opinion beyond that15:32
clarkbdpawlik: to be super clear we were pretty explicit about not being involved in operations like this15:33
clarkbI don't even know how to login15:33
dpawlikI understand15:33
dpawlikoh xD15:33
dpawlikI was thinking that you were logged in into the AWS console15:34
dpawlikallright. So in that case, I will send an email about upgrade15:34
clarkbI think maybe fungi got an account setup? BUt I'm pretty sure I don't have one.15:34
dpawlikif I will not have permission to trigger that, I will just postpone it to later15:34
clarkbdpawlik: is that not something you can check right now?15:35
clarkbItseems weird to me that this is unknowable15:35
dpawliknot really... So I don't use the AWS API to call the upgrade, I'm doing it via web site and it seems that I got that option "update now" where it shows me when it should be done15:36
dpawlikbut when I was starting maintaining the Opensearch, the console look ok, but later I saw that I don't have permissions to check this, this this 15:37
dpawlikmoreover, there is an info "A new service software release R20221114-P2 is available. The release will be automatically deployed after April 04, 2023, 07:04 (UTC+02:00). Learn more"15:37
clarkbdpawlik: oh you are worried that it will fail after the initiate the process?15:38
dpawlikI never have done it before in AWS15:39
clarkbinfra-root: This is fun if you request with a brswer you get a 404. If you do it with curl you get a 40315:39
dpawliklike...99% of people on that channel :)15:39
dpawlikbut I don't expect that it will break anything15:40
clarkbinfra-root the next thing I need to test is whether or not calling the repo create on an existing repo is an error. If so I need to do something like ^ to query if the repo exists before creating it15:40
clarkboh I'm logged in in the browser so it shows me the 404 but I guess if you aren't logged in with perms to kow if the repo exists or not its an unauthorized error?15:40
dpawlikset the repository as "public" not private15:43
clarkbdpawlik: it is15:46
clarkbwell I should say the repo we have created it15:46
clarkbbut we need to check for uncreated repos which have no visibility status15:46
clarkbbasically it seems quay won't let you know if a repo exists or not unless authenticated with sufficient permissions to list all possible repos? Otherwise its an authentication error15:47
clarkbwe can probably assume for our purposes that a non 200 return means that the image does not exist, but that is because we will make all of our images public15:48
clarkblooks like attemtping to create a repo that already exists is a 400 errorwith a message indicating the repo already exists. I think we can just try to create things and continue if they are already there. THat seems more reliable than assuming a 403 is a 404 by proxy15:49
dpawlikI have done a tool for maintaining our quay service - . That functionality that you are looking for, probably is missing (I have done it some time ago)15:50
clarkbdpawlik: what I'm doing will end up as a small zuul-jobs ansible role using the uri module. Basically an add on when talking to quay for docker management jobs to precreate the repository (so that we can mark it public)15:51
clarkbafter this testing I think it will end up being a single uri task with a couple of followup tasks to error check15:51
clarkbessentially mimicing what you get via docker image push to docker hub15:52
dpawlikah. Now I got more context15:53
NeilHanlonfwiw clarkb, fungi, others: Rocky was accepted into DSOS this morning15:53
dpawlikthe Quay API is pretty cool. Also it works well with swagger 15:54
* dpawlik need to go in few min15:54
clarkbdpawlik: I don't like the authentication system. In particular its annoying you need to delete an application to invalidate tokens. At least via the web UI? I also don't like that I can't make a token act on behalf of a robot account15:55
NeilHanlondpawlik: w.r.t. opensearch upgrades: they have been stable in my experience15:55
fungiclarkb: dpawlik: yeah, i think i logged into the aws console once or twice when things were being set up, but i didn't have administrative control over the environment i don't think. at least not when i was trying to work out the ssl cert renewal situation anyway15:56
clarkbdpawlik: as far as the goal here, I'm mostly just trying to update the existing zuul-jobs support for image creation and pushes for quay and an expectation that repos should be public by default. I wasn't sure what the best way to check if a repo should be created is since we want to precreate via the API before doing the image push (because image push defaults to private)15:58
clarkbalso looking at token scopes I think create repo is more limted than the scope necessary to modify repos so a slighty preference for precreating than updating after the fact15:59
clarkbmodify perms would allo modification of all repo attributes looks like. But create only allows you to make a new thing once16:00
dpawlikNeilHanlon, fungi: thanks for info16:01
dpawlikclarkb: let me know when you have a PS, I'm interested to check it16:02
fungiit's possible i have opensearch admin perms but not ssl cert admin perms, i suppose16:02
dpawlikfungi: I was reading some time ago about the permissions and policy in AWS that can be applied for user... It's insane how many options they have :D16:04
* dpawlik need to go. Enjoy your weekend!16:05
clarkbyou too16:05
opendevreviewMichael Johnson proposed opendev/meetbot master: Fix white space blocking #startmeeting
johnsomA small fix so we don't bug you thinking meetbot is hung. lol16:18
fungijohnsom: oh, right! i remember talking about that16:26
johnsomYeah, a silly cut/paste error that tricked Greg and I16:26
fungitricksy, tricksy botses16:27
johnsomAh, Greg pointed out a bug in my patch, fixing16:29
opendevreviewMichael Johnson proposed opendev/meetbot master: Fix white space blocking #startmeeting
NeilHanlonrandom question: any tips/tricks for running your own meetbot? thinking of setting one up for Rocky's various projects16:34
clarkbNeilHanlon: not really. I think we set that up forever ago (I remember addinga feature to it and it existed efore I started owrking on things) and the only thing we've really done to it since is switch to limnoria from supybot16:34
NeilHanloni see -- any idea why the switch from limnoria?16:35
NeilHanlonin general, besides them being tricksey as fungi mentions... bots are fairly straightforward 16:35
johnsomsupybot is not maintained anymore16:36
clarkbright supybot was unmaintained and forked off to limnoria and we switched in order to python3 and modernize the deployment16:36
NeilHanlonoh. i misread16:41
NeilHanlonFROM supybot16:41
NeilHanlonyes that makes sense lol16:42
clarkbNeilHanlon: this is where we build our docker image for limnoria. There is some zuul magic there using "sibling" repositories to include meetbot in the install. The local build script hints at what zuul si doing and allows you to replicate it locally17:06
clarkbNeilHanlon: then is ansible that deploys the container 17:07
fungiNeilHanlon: probably the best advice i have is make sure you give your users an easy-to-find "cheat sheet" with the commands/syntax supported by your bot. especially handy if meetings are an infrequent thing, and people only occasionally need a particular feature of the bot (e.g. to hold a vote or flag specific agreement or action items in the minutes). having to stop and hunt for documentation17:10
fungican slow down meetings substantially, wasting the time of possibly many attendees17:10
fungibut meetbot is quite useful. i'm on the board of directors for spi and we hold our monthly board meetings in irc using a meetbot (not the same fork as opendev's but shares a common ancestry)17:11
*** jpena is now known as jpena|off17:24
NeilHanlonthank you, appreciate all that. we have other hurdles with people prefering video to chat in some cases.. but, slowly I'll convince them otherwise17:25
NeilHanloni'll chat with people all day on IRC... i only have so much capacity for voice/video calls17:25
fungisame here17:40
fungirax-ord utilization doesn't look worse over the past 24 hours, after going back to full capacity. it's not great but seems much better than before the various tuning efforts. still seems like the building line spikes up a lot higher than what i would expect with the the current max-concurrency=5 setting (unless we've got lots of requests for huge nodesets all hitting at once)18:48
clarkbfungi: I wonder if we can query for builds with large nodesets somehow18:49
clarkbmaybe through the historical nodepool request logs?18:49
clarkbI think knowing that a bunch of builds are running with 10 nodes is generally useful info too18:50
fungiat the 02:08z sample (just after periodics kicked off), it clocked 178 building nodes18:50
fungithat would imply we averaged 36 nodes per request at that point in time18:50
fungiwhich i'm pretty sure we have no jobs actually doing18:50
fungiso odds are there's more to that graph than just nodes that are building for a node request that's still in process18:51
fungibut the error node launch attempts graph looks a lot better than a week ago18:53
opendevreviewClark Boylan proposed zuul/zuul-jobs master: Add ensure-quay-repo role
clarkbinfra-root dpawlik ^ thats where I've ended up with the make quay create repos via api using ansible. I've run that locally using include_tasks and a lot of -e on the command line and I think it is working19:35
fungineeds moar -e19:36
clarkbI'm a bit on the fence for whther or not we should default to public there but I figure since making a private repo is trivial you'd really only use this role if making a public repo so it makes a good default19:37
fungimakes sense to me19:39
fungidid get decommissioned? i can't recall. also did the contents of the root homedir get copied from it to the new bridge? that's where we kept the keychain with the private artifact signing master keys... i'm not finding any of that on the new bridge19:54
fungispecifically the /root/signing.gnupg directory documented in
Clark[m]It was shutdown but unsure of deletion19:55
fungiyeah, i'm checking that now19:55
fungilooks like it exists, it's just powered off19:56
fungiinfra-root: any objection to me booting the old up briefly so i can copy files we seem to have overlooked from the root homedir on it?19:56
fungii'm running behind generating the next signing key, and will definitely need to regain access to the previous one so i can sign the new one with it19:57
fungiif booting that server is dangerous, i can work on restoring files from its backup19:58
Clark[m]I don't have an objection. Maybe just be on the lookout for any cronjobs it may run (though I disabled the zuul upgrade one and it won't run for another 4 hours anyway) not sure of others that may trigger if enabled19:58
Clark[m]I think 90% of the danger is just in potentially allowing cron jobs to run since zuul shouldn't talk to it anymore. I guess you can double check zuul has forgotten about the old host too l19:58
fungiokay, i'll have it up just long enough for the new bridge to slurp over a copy of the old ~root, but i'll check crontab and running processes first thing20:05
fungirax-dns-backup was still configured in the root crontab so i commented it out20:07
fungiokay, i've shut it back down again20:20
fungimanaged to use sudo with pipes at both ends of the transfer so that the tarfile i created was never written to disk by a non-root user, even though i authenticated the transfer with my non-root account ( connecting to to retrieve it)20:22
fungibasically ssh remote sudo cat file | sudo tee file20:23
fungigotta love how ssh client avoids mangling binary data streams20:24
fungianyway, ~root/signing.gnupg is on the new bridge now20:25
fungiand there's a ~root/old-bridge-root-homedir.tar on it if anyone needs anything else out of there in the near term20:25
opendevreviewClark Boylan proposed zuul/zuul-jobs master: Add ensure-quay-repo role
fungiwell, it's become ~root/old-bridge-root-homedir.tar.xz now, to save a bit of space (shrank from 7gib to 2gib)21:45
ianwoh lucky that's still there21:59
ianwi feel like i copied that sorry, but glad we found it21:59
ianwit is a good case for us having a distributed backup, however :)22:02
fungimy fault for not thinking to check that we had copied it over to begin with, really22:09
clarkbianw: re flipping the bit after you push I thought about that too but eventually decided doing a create first is better becuase the token scope for that is much smaller22:15
clarkbbut agreed both would work22:15

Generated by 2.17.3 by Marius Gedminas - find it at!