Thursday, 2023-03-16

ianwclarkb: i tested the !groupa:!groupb and I think shows that it indeed means "don't run if this host is in groupa or in groupb"00:08
Clark[m]Cool I did end up leaving a +1 after finding the extra docs. Off to dinner now but a lot more confident that does the correct thing now00:14
opendevreviewIan Wienand proposed openstack/project-config master: gerrit/acl : check for function/s-r in normalize
opendevreviewIan Wienand proposed openstack/project-config master: gerrit/acls : fix some missed NoOp functions
opendevreviewIan Wienand proposed openstack/project-config master: gerrit/acl : check for capital booleans in normalize
opendevreviewIan Wienand proposed openstack/project-config master: gerrit/acl : fix some missed NoOp functions
opendevreviewIan Wienand proposed openstack/project-config master: gerrit/acl : check for function/s-r in normalize
opendevreviewIan Wienand proposed openstack/project-config master: gerrit/acl : check for capital booleans in normalize
opendevreviewIan Wienand proposed opendev/system-config master: system-config-run-review : add review priority and backport labels
ianwok that is everything i had on my todo list for copyConditions/submit-requirements cleared out03:00
ianw(modulo +1 verified :)03:01
ianwclarkb: i think ^^ is kind of good to add the trigger votes section, seems to work ->
*** gibi_pto is now known as gibi08:04
*** jpena|off is now known as jpena08:21
*** jpena is now known as jpena|off08:43
*** jpena|off is now known as jpena08:52
mnasiadkaclarkb: I did sign up without paying for anything - but it was last year, and our plan allows 0 private repositories for openstack.kolla organization - should be for free09:18
*** elodilles_pto is now known as elodilles09:26
fricklerclarkb: I'll be off early today and away until tuesday, so you'll have to discuss docker without me. I would certainly prefer not to pay them anything in reward for their behaviour09:54
*** dhill is now known as Guest793912:29
opendevreviewMerged opendev/system-config master: dns variables : move to canonical locations
opendevreviewMerged opendev/system-config master: bind9 : drop obsolete option for later versions
opendevreviewMerged opendev/system-config master: system-config-run-dns : update nodes to jammy
dtantsur also has a free account15:26
clarkbya I'm about to try signing up this morning. I think the thing that trips us up is that all of the docs say you need to set up at least a developer level account whihc is not free but then also says you can use for free. The messaging is confusing and now I just need to give it a go15:28
clarkb"user account for this email already exists" thats news to me /me recovers an account insteadof creating a new one15:33
clarkbinteresting when I did that it redirected me to a page where i have to add all the extra info corvus  was talking about so couldn't sneak around that with an old account15:37
clarkbwhich then results in "We're sorry an internal server error occurred"15:37
clarkband if I try to login to quay I get redirected to the page to add extra details again and it fails again15:38
clarkbso ya thats fun15:40
fungimaybe they're buried under dockerhub evacuees and the system has fallen over15:51
clarkbor I've got a really old account they don't know how to make modern15:51
hasharhi, is there any guide as to which python jobs I could add to a repo? I am trying to polish up the old `jjb/python-jenkins` which uses `openstack-python35-jobs` and I guess I can replace it with `tox-py35`?15:53
hasharI guess I can try :]15:53
clarkbhashar: zuul lists them all for you at
clarkbhashar: tox-py36 37 38 39 310 311 should all work but you may need to select a test nodeset that supports that version of python15:55
clarkbI've tried both chrome and firefox to make sure this wasn't a browser difference breaking red hat. Same behavior with both15:55
clarkbIf I try to login via another location it asks me to choose if my account is corporate or developer. I'll set this personal email account to a regular developer account I guess15:56
clarkband see if that fixes things15:56
hasharclarkb: I will try thanks15:56
clarkbone problem with figuring out how to login to red hat is that red hat sso is both a product they sell and a tool they run15:57
clarkbI'm getting turned around in circles here :/15:57
fungicould it be that all the people saying "just upload to quay, it's free" got developer accounts automatically by being employed at red hat?16:00
clarkbfungi: well I'm not even trying to do anything quay related yet16:01
clarkbI'm just trying to login at sso.redhat.com16:01
clarkbI filled out the form it wanted there and it redirected me back to the same empty form after hitting submit16:01
clarkbno explicit error this time but I'm not sure it worked iether16:01
fungiright, you're trying to sign up for a developer account. i'm just wondering if anyone outside of rh is actually able to sign up for those successfully16:01
fungiand whether rh employes get signed up for it automatically so don't realize the sign-up is busted16:02
clarkb*I'm trying to update my existing account to a modern rh developer account16:02
clarkbI suspect the issue is in upgrading the account. I can try creating a new account with a different email addr16:02
fungioh, that's probably an even less well-tested path ;)16:02
clarkbI started this by trying to sign up for a new account and being told an account already existed with that email address16:03
clarkbso then I reset the password successfully but am now stuck in update your details limbo16:03
JayFWe have enough RH-adjacent people in the community that surely one of them could at least connect you to someone who could fix it?16:03
clarkbmaybe. But games of telephone are never fun.16:04
JayFOh, I dislike that they hide so much behind login as well. Not really open in the 4-opens sense... but it's probably better to send an email than bang your head against it 16:04
JayFmake them feel some of the pain from those choices too :/ 16:05
clarkbI also wonder if that account got associated with hp or itnel or some corporate account way back when and now its basically dead. But the easiest way to deal with this is a new account I think16:06
clarkbyup that worked16:09
clarkbinfra-root: ok I had to provide an email address, phone number, and physical address for this personal account. I did not have to provide any billing info. When I login and view the billing info area of the settings page it says 0 private repos is the maximum allowed by your plan and doesn't seem to force me to sign up with billing info anywhere16:11
clarkbI think this should work if you can amange to create an account :)16:11
clarkblooks like you can also set the account type to be an organization? I'm not sure if it is better to create a new organization via a normal user account or create a new normal user account for our purposes then convert it to an aorganization?16:13
clarkbalso it looks like you set a quay docker protocol password independently of your red hat sso which is nice16:14
fungiand that's what would be put in the zuul secret for the publishing job?16:14
clarkbI need to grab breakfast but I'll push a copy of our base python images to my personal account there just to make sure all the plumbing works16:15
clarkbWhen you click create organiztion you set an organization name and email (this email addr must differ from your account) and then you choose from a set of prices. One of them is "Open Source" which is 0 private repos and $016:17
fungimakes sense16:17
clarkbI think that means we have the option of having an org+regular user for "infra-root" but with different email addrs. Or just make an infra-root regular user for opendevorg/ or whatever and share that account16:18
clarkbI suspect the "correct" thing to do is create an org and a push user16:18
clarkbrather than just a user for everything16:18
clarkbthe whole phone number and unique emails requirement is a bit annoying but I thinkwe can make that work assuming infra-root+quayopendevorg and infra-root+quayopendevuser work16:19
clarkbok breakfast now. Then I'll try pushing an image and see if others can fetch it to ensure the actual workflows work16:21
fungido they test that the phone number is actually in service?16:22
clarkbfungi: I have not recieved any phone contact from them since creating the account. Admittedly not that long ago16:24
clarkbno sms or phone calls etc16:24
clarkbit is probably useful in a recovery situation and may make sense to set to a value that is likely to function.16:25
clarkbOh! also when you sign up to quay they ask for slightly less info than signing up via I'd suggest we sign up via quay for this reason16:25
clarkb(it drops the whole corporate vs personal question)16:25
fungioh, nice16:25
opendevreviewAntoine Musso proposed zuul/zuul-jobs master: Remove ignored success-url job attribute
clarkb I think that worked. If someone else wants to pull you can confirm the public access16:59
corvusclarkb: wfm.  pulled manifest and i had all the layers locally :)17:03
clarkbexcellent then despite some account management clunkyness and messaging confusion I think this should work17:03
corvuspulled on another machine without layers cached and that worked too17:03
corvusand obvs neither of these had quay logins, so was anonymous17:04
clarkbI find the UI a bit clunky too but I think that isn't a problem 99% of the time17:06
clarkblol the security checker for quay complains about debian libc due to a cve that debian has marked as not an issue17:07
clarkbI can see how notices like that will cause people to very quickly ignore the security checkers17:08
clarkbour debian libc package is up to date too17:08
clarkbI'm going to guess that centos and fedora images do not trip these messages17:09
clarkbtheir checker also shows fixes for some things in package versions that don't show up in the debian package search17:11
clarkbIf I have any real criticism of quay so far it is their security scanner produces a bunch of noise which will lead people to ignore it17:11
clarkbthe last thing I need to test is that the update to the quay docker cli password settings really did leave my sso credentials alone17:15
clarkbheh if you sign out all sessions then immediate click sign in it does so autoamtically...17:16
* clarkb opens a new browser serssion17:16
clarkbyup the passwords are distinct which is what we want17:18
*** jpena is now known as jpena|off17:23
clarkbI think what we do is have admin user accounts tied to us as individuals. We create organizations for opendevorg/ zuul/ etc and under the umbrella of an organization we can create robot accounts. These robot accounts are the ones that should go into our zuul stuff17:24
fungihopefully this means our discussion in ~1.5 hours will be a quick one17:26
clarkbit isn't clear to me if a robot account can create new rpos17:26
clarkb"For an organization-owned robot account, a robot account can be granted permission to create repositories if placed under a team with the creator permission. Otherwise, a robot account must be granted individual permissions."17:28
clarkbI think this should all work17:28
clarkbquay docs also say you can just push to a name to create a repository.17:42
clarkblet me test this17:43
clarkbok direct push works but it seems to mark the image private by default I think this is what mnasiadka was referring to17:47
clarkblooks like there are apis to set these values too but the api docs don't document valid parameter values just that the parameters exist :/17:51
clarkb is something I don't get the full text for17:52
clarkbthis seems solveable if we end up manually toggling that value in the short term17:53
clarkbof course the time we'll want it most is when we create a bunch of new images for the things we use today17:54
clarkbI wonder if you create an open source org if it solves that for you actually. Since the implication there is very much that repos will all be public. I don't want to create orgs for oepndev and/or zuul until we have a bit more consensus on this though so I'll avoid testing that for now17:55
NeilHanlonclarkb: meat of that article is18:04
NeilHanlonAdd and set following parameter CREATE_PRIVATE_REPO_ON_PUSH: false in quay config.yaml file. This helps create a public repository when first pushing the image to the quay registry.18:04
NeilHanlonnotably: there is also a CREATE_NAMESPACE_ON_PUSH parameter18:05
clarkbNeilHanlon: oh so that is for people running their own local quay deployments. That won't apply here unfrotuntaely18:06
NeilHanlonTrying to figure that, but my suspicion is the same.. that it's a server-side config18:07
NeilHanlonyeah, looks to be server side18:08
fungiso no mention of any similar parameter that can be passed in the api call18:09
NeilHanlonnot that I can tell18:09
NeilHanlonthere is some default permission settings in an organization.. but I cannot see if there's a way to grant read to 'all'18:11
NeilHanlon 18:11
clarkbya  Idon't think it is the end ofthe world considering there appears to be an api endpoint to manage it. Our flow might just be create repo with api endpoint then push18:15
NeilHanlonyeah that's probably smart18:16
NeilHanlonit also might just be me, but I've experienced trouble uploading to quay... interested to hear if you all have similar issues18:17
NeilHanlonsometimes their proxy just hangs up, it seems18:17
fungiso we should implement retries, sounds like18:17
NeilHanlonI would recommend it, yeah18:18
fungifwiw, we've seen random failures with trying to publish on dockerhub too18:18
fungithough our usual workflow is to push images built from approved changes, and then retag them in the repository with a known label after they merge18:18
clarkbI'm told emilienm may have a blog post about this18:19
clarkbI'm looking now18:19
clarkbya we allow three attempts to docker hub iirc18:19
fungier, retag them in the registry i mean18:19
NeilHanlonclarkb ?18:19
fungii think it's mainly been tag deletes where we end up with failures?18:19
clarkbNeilHanlon wins18:20
clarkbso ya this should work for us. We'll just need zuul jobs that create the public repo before we push to it if it doesn't exist already18:20
corvussounds like we could make a role to do that and include it first thing in the promote jobs18:50
clarkbyup I was looking at what it takes to get a bearer token and I think we can make an application for an organization (instead of a robot) and use the resulting token for the application to talk to the api and do the image push commands18:57
clarkbthe distinction between an application and a robot is a bit fuzy to me but it see that only an org can have an application but orgs and users can have robots?18:57
clarkbinfra-root it is 19:00 which is when we agreed to spend some time making decisions about the shutdown of free docker orgs19:00
clarkb is the etherpad where I've collected ideas/thoughts/concerns so far to try and ensure we keep the info as centralized as possible. Though I haven't put what I have learned about quay there yet19:00
clarkbLong story short on April 14 we'll lose access to manage our docker hub orgs for opendevorg/ and zuul/ images. I believe the public images hosted there will be accessible for 30 days after april 1419:01
corvuswe meeting here or -meeting?19:01
clarkbcorvus: would you like us to keep notes and use -meeting? we can do that19:01
* fungi is ambivalent about meeting minutes, good with either channel choice19:02
corvusoh no opinion, just want to make sure i'm in the right place :)19:03
clarkbafter you said it I realized its nice to have the logs separated for ease of discovery later19:03
clarkbthings can get lost in the daily scrollback in here19:03
corvusto the meeting room then!19:04
artom_So out of curiosity, if we have a Zuul tox job that fails consistently 100% of the time, but we can't reproduce locally, is it conceivable to hold a VM so that we can poke around after the job?19:32
clarkbartom_: yes, if you note the job name and change you'll trigger it with we can set a hold. We'll also need a copy of your public ssh key19:33
fungiartom_: it is, but have a link to the build result page? we mightbe able to spot it19:33
clarkbalso that19:33
fungisome of us have become rather attuned to the assumptions users make when running tests locally19:33
opendevreviewJeremy Stanley proposed openstack/project-config master: Restore rax-ord quota but lower max-concurrency
fungiclarkb: corvus: ^ after reviewing the graphs for rax-ord. i think the max-concurrency has helped but may still be too high, so want to chop it in half while we redouble the max-servers back to the original capacity20:03
fungiseems like it does a good job of using nodes there as long as it manages to boot them20:04
clarkbfungi: wfm20:04
corvusfungi: should we think about rate too?  that might help stretch out delete serialization if that's a problem (since i don't think max-concurrency does anything for deletes)20:05
corvus(but also approved)20:05
fungiwe can, though i already attenuated the rate by an order of magnitude earlier20:06
fungibut 100/sec may still be too high, yeah20:06
fungithere's not really a substantial gap between used/deleting in the graphs, that i can see20:06
* clarkb finds lunch20:07
fungicorvus: and it really seems like the underlying problems there aren't the number of api calls, rather their nova scheduler probably takes way too long to pick a host and process the create20:09
fungiand the time that takes seems to get significantly worse the more of those we ask for at once20:09
fungibut the graph is looking much closer to how their other two regions behave now, at least20:11
corvusfungi: yep; someone mentioned the delete instance thundering herd problem to me the other day and that's fresh in my mind20:13
corvusbut i don't have any evidence it applies here20:13
corvusjust something to keep in mind20:13
fungiwill do, thanks!20:13
corvuserm, stupid question: are we still infra-root at ?20:16
fungicorvus: yes, that's still the address20:16
corvuskk.  i will register with "infra-root+quay-zuul-org" at ...20:16
opendevreviewMerged openstack/project-config master: Restore rax-ord quota but lower max-concurrency
corvusbad news!  zuul already exists.  i wonder if it's owned by or ...20:18
fungizuulci, everyone's favorite italian dessert20:20
clarkbI'm doing opendevorg/ to keep the url portion the same with what we had and infra-root+quay-opendev-org at ...20:23
fungithough it's an opportunity to have opendev there if it's not already squatted20:23
clarkbopendev/ is greeen if we want it20:24
corvusi didn't find out it was unavailable until i hit submit20:24
fungiwe ended up using opendevorg on dh only because opendev was taken20:24
clarkbcorvus: ah20:24
clarkbdo we prefer opendev/ then? I can try that20:24
fungishorter is probably better, but i'm not all that concerned either way20:25
corvusi have slight preference for opendev/20:25
clarkbcool trying that then20:25
clarkbit already exists. Now trying opendevorg/20:25
corvusi wonder if the same helpful person squatted both...20:26
clarkbcorvus: do you think it worthwhile to wait on that or just move ahead?20:26
fungior if the recent dockerhub announcement has resulted in everything getting squatted20:26
corvusclarkb: for opendev i don't think i'd wait, i'd say keep going with opendevorg; since we already use it20:27
clarkb done20:28
corvusfor zuul, i sent an email to....  (since apparently that's what the faq says?) to ask20:28
corvusi also asked in #zuul in case someone helpfully squatted it and we forgot20:28
clarkbI'm going to pause here. Both coruvs and I are owners of the opendevorg org and will happily add other infra-root once you let us know what your usernames are20:34
clarkbIts bright and sunny and almost warm outside today so I want to get out on my bike. When I get back I'll look at setting up an "Application" in the org which can be used to talk to the api and push images20:34
fungilooks like i previously had a red hat account because of working ansiblefest, but now it's busted20:37
clarkbthat may explain what I ran into20:37
clarkbfwiw I don't see any emails after creating the org20:38
clarkbbut maybe when I create an application we'll get email20:38
corvusme too, so i have 2 potentially busted redhat accounts20:39
fungitried the password reset, but i receive no reset email20:40
clarkbianw: fwiw I reviewed your new acl stack and they lgtm but I dind't approve since I wasnt' sure I could pay attention to them earlier20:40
clarkbianw: but I think you can self approve if you like20:40
clarkbfungi: I did get a reset email. It was what ahpepend after setting my new password with trying to set personal info that failed20:41
fungiokay, i am now
fungiapparently your quay username isn't tied to your red hat username20:48
corvusfungi: i invited you; you should have a notification in the bell at the top right20:53
corvus(it showed up for me after i set my cli password; i don't know if that's because i needed to do that first, or it just took a few minutes)20:53
fungiaccepted. thanks!20:55
fungiand yeah, i had already seen and acted on the password setting notification20:56
ianwclarkb: thanks, will do today to clear that, i started updating the checklist page but will finish that off21:02
ianwclarkb: are you ok with me trying out the linaro thing too?  i imagine that will take some fiddling to get right, so i'll watch that closely21:05
clarkbI am but that one is probably good to get a second reviewr on?21:07
ianwcorvus: i am now
ianwfungi: maybe if you have a sec the change in question is, which adds linaro as an "unamanged" node so we can deploy users and stuff, but not take over iptables, etc. with a full base install21:10
fungisure, taking a look21:11
ianwit currently won't do anything, i'd just like to validate the path that it can actually run something.  also kevinz has agreed to us using it like that21:11
artom_clarkb, fungi, belated thanks! Was in a call, so have to drop now, but will follow up in the next few days21:13
fungiyeah, lgtm, and i saw the response from kevinz in favor as well21:13
artom_jparker, ^^ (srcoll to 15:33:02)21:13
artom_fungi, so is the build page, in case you're still around21:15
*** artom_ is now known as artom21:15
artomThe error is very specific to the project iself21:15
artomAnd does not happen then I just run `tox -re py310` locally on Jammy21:16
opendevreviewMerged openstack/diskimage-builder master: Fix double-keyed json
opendevreviewMerged openstack/diskimage-builder master: Repeat to umount filesystem when exception occurs
fungiartom: got it, looks like oslo_config isn't finding expected configuration options, and this started between 2022-12-13 and 2023-01-23 (last recorded success and first in the current string of failures). seems like py36 jobs are working but py38, py39 and py310 fail?21:57
jparkerfungi: yes its just py38,py39, and py31021:58
fungiartom: fwiw, i get the exact same error running tox -e py310 on my workstation21:58
fungi(debian sid, not ubuntu jammy, but similar)21:59
artomfungi, oh that's some Twilight Zone stuff22:12
fungiif it's working on py36 it probably makes sense to compare what versions of packages are ending up installed, because a lot of libs have dropped py36 support and it's probably a newer version of one of them, or maybe newer tox22:15
fungiactually, that's right around when tox 4.0.0 happened, yeah?22:15
fungiare you testing with tox 3.x locally?22:15
fungiartom: remove the skipsdist from tox.ini and *bam* it works22:16
fungithere's your problem22:17
fungithe timing was a dead giveaway22:17
fungiif you upgrade your local tox you'll likely see the same errors otherwise22:17
fungijparker: ^22:19
jparkerfungi: thank you! Let me try that now22:20
fungibasically the reason for the confusing error is that the tempest-whitebox-plugin isn't getting installed into the venv22:21
fungilots of our projects used to rely on usedevelop making the project always get installed, even if they set skipsdist. tox maintainers decided that was a nonsensical misbehavior, and made it so that skipsdist means never install the project even if usedevelop is specified22:22
jparkerfungi: thanks debugging and explaining the issue that's extremely helpful, artom I've got something testing now
fungimy pleasure. it was faster than setting up an autohold and giving someone access to the node22:27
artomfungi, the annoying thing is, we went through this with Nova, I just wasn't paying attention enough22:28
artomfungi, much appreciated, thank you!22:28
fungiany time22:28
opendevreviewIan Wienand proposed openstack/project-config master: openstack/release : return to non-blocking submit rule
clarkbianw: you should see an invite in quay I just got back from a bike ride and added you23:03
ianwgreat, looks like I'm in23:04
clarkbI'm looking at the application stuff and its weird that it says it will act on behalf of me I think because I was the user creating it23:07
clarkbok looks like there may be a way to do this with a robot instead so I'll explore that avenue23:07
clarkbfrom I think that there may not be a way to perform application actions as a robot. In that case I think we probably want two secrets. One is a bearer token for an admin that is able to create new repos. This will create the initial repo. Then another which is the robot which can do all of he pushing23:19
clarkbwhat I've done is create a robot and put it in a team with "creator" perms23:19
clarkband now I'll create an application that will act as me apparently to create a repo and see if I can then push as the robot23:20
opendevreviewMerged openstack/project-config master: gerrit/acl : fix some missed NoOp functions
clarkbok I am/was a derp and triedto create a repo under opendev instead of opendevorg and that is why I was not authorized. So I got a new token with more perms and tried some more before I realized. There doesn't seem to be a good way to remove tokens that I see so I guess I remove the entire application? I'll test that in a bit23:42
clarkbconfirmed that deleting the application properly clears out the tokens.23:49
opendevreviewMerged opendev/system-config master: infra-prod: run job against linaro
clarkbok here is a rundown of what I did in quay that seems to work for creating and pushing a public image/repo: 1) create robot account 2) create automationtools team 3) set default permissions to give org owners admin rights when anyone creates a new image 4) set default permissions to give write rights to the automationtools team when anyone creates a new image 5) create a new23:54
clarkbapplication to generate a bearer token that will act as me (I can't find a way around this yet) 6) use emilienm's documented curl stuff to create a public repo 7) use robot account to push image content23:54
clarkbif we need to remove bearer tokens the only option seems to be to delete the application and create a new one but this does seem to work23:55
clarkbthis is how opendevorg/clarkbtest:base-3.11-bullseye was created and pushed23:56
clarkbbecause the application is apparently acting as me I dleted the old one and made a new one that very specifically identifies this as a token related to me23:56
clarkbmake management of it easier23:56
clarkbAnother option would be to docker image push to have the robot account create the image. Then use the bearer token to modify its visibility. Not sure that is any better and I like the flow of creating it public upfront23:57
clarkbok good the log shows that the application is acting on behalf of a person rather than just that person doing things23:59
clarkband the log captures everything I just wrote down. THats a nice feature23:59
clarkbI've run out of time, but I'll pick up the ansiblification of this tomorrow23:59

Generated by 2.17.3 by Marius Gedminas - find it at!