Monday, 2020-03-23

openstackgerritMerged opendev/system-config master: Docs: Update main page for OpenDev
clarkbthat change should exercise the new restart logic03:04
clarkbmordred: ^ I wonder if there is a good way to verify that gerrit responded as expected. maybe via replication logs03:05
openstackgerritKevin Zhao proposed openstack/project-config master: Linaro US: Change bionic job to 16G RAM
openstackgerritKevin Zhao proposed openstack/project-config master: Linaro US: Change bionic job to 16G RAM
openstackgerritsebastian marcet proposed opendev/puppet-openstackid master: Fixing results from ZAP Scanning Report
openstackgerritKevin Zhao proposed openstack/project-config master: Linaro US: Change bionic job to 16G RAM
openstackgerritKevin Zhao proposed openstack/project-config master: Linaro US:  Add a 16GB RAM label for bionic
AJaegerconfig-core, please review
*** DSpider has joined #opendev07:44
*** dpawlik has joined #opendev07:44
*** ralonsoh has joined #opendev07:59
*** tosky has joined #opendev08:32
*** rpittau|afk is now known as rpittau08:52
*** ralonsoh has quit IRC09:44
*** ralonsoh has joined #opendev09:44
*** bcafarel has left #opendev09:55
*** dpawlik has quit IRC10:11
*** dpawlik has joined #opendev10:13
*** roman_g has joined #opendev10:33
*** ouroboros8 has joined #opendev11:21
*** rpittau is now known as rpittau|bbl11:47
*** roman_g has quit IRC12:46
mordredclarkb: good question13:03
*** rpittau|bbl is now known as rpittau13:08
*** roman_g has joined #opendev13:35
AJaegerinfra-root, two small fixes for review, please:  (pbrx), (netlify-sandbox),13:45
mordredfungi: could I interrupt your morning with ?13:48
mordredfungi: and from thence afterwords unto, which verily doth depend on it?13:49
fungisure, just a sec13:50
AJaegerthanks, mordred13:52
AJaegerwhy is netlify-sandbox in opendev? None of us is core there13:53
corvusAJaeger: we can add ourselves13:55
corvusit's a repo to test out the netlify/gerrit integration that we hope to (someday) use for web site cms13:56
AJaegermmh, readme says " StarlingX website at"?13:56
corvusyeah, that's the test data13:56
corvusi think so far we haven't had to approve much (it's a sandbox)13:57
AJaegermy change was just updating .gitreview - wanted to cleanup my queue13:57
mordredyeah. we've got 4 or 5 steps we need to work on13:57
corvusi'm sure we can bootstrap ourselves to land a gitreview change13:57
AJaegerthanks for background13:57
corvussay i've got a repo with a debian dir like this:
corvuswhat's the best way to build the packages?13:59
corvuscd into that and run 'dpkg-buildpackage' ?13:59
corvusrun debian/rules?14:00
mordredI normally use debbuild instead of dpkg-buildpackage14:01
mordredyou might want to run it with -uc -us - which will tell it not to try to sign the changes or source14:01
mordred(if you're not the author of the most recent change in the changelog)14:02
corvusyes i want that, thanks!14:02
mordredthere's also pdebuild which will use pbuilder to make a a chroot and then build it in there14:02
corvusi'll probably just throw all this into a container14:02
mordredbut that's like 10 year old info - one would imagine there woudl be something similar to pdebuild which would use docker nowadays14:02
corvussince that's what i'm trying to build anyway14:03
corvusthe jitsi-meet docker images are really just installs of the project's debian packages14:03
corvus(which seems weird to me, sort of a "worst of both worlds" situation; but i don't want to reinvent their system)14:03
mordredcorvus: might be a small thiung to look at that you might be able to steal commands from14:04
corvusmordred: thanks!14:06
fungii second the preference for debbuild over dpkg-buildpackage14:08
fungiand yeah, pdebuild (using pbuilder) or cdebuild (using cowbuilder) is useful if you want packages consistently built in a minimal chroot14:09
fungipbocker does look like an interesting modernization. pbuilder traditionally bootstrapped and the tarred up the chroot for future reuse14:10
fungithings folks use docker for these days14:11
fungimordred: 714307 depends on an outdated patchset of its parent change14:14
corvusfungi, mordred: does mean anything to you?14:24
mordredcorvus: yes - debian expects the directory to be named with the version14:24
mordredor, rather, for the package name14:25
mordred(version is optional)14:25
corvusoh, it seems to behave differently if i run it in the top level dir14:25
mordredso try putting it into a dir called "jitsi-meet-web" instead of "jitsi-meet"14:25
corvusinstead of debian14:25
mordredyes - sorry14:25
mordredyou should run in the top level, not in debian/14:25
corvusmordred: well, i mean, i'm just trying to do what upstream does14:25
corvusok i'll do that14:25
fungidebuild generally expects the "debian" directory to be a subdir of your pwd, creates temporary build results in ./debian/build/ and writes package files in ../14:27
corvusa build-depends would have been nice :/14:27
mordredcorvus: they don't have build-depends in the debian/control file?14:28
fungiyeah, if they do you ought to be able to tell apt-source to parse that for you14:28
openstackgerritMerged opendev/system-config master: Split gitea and gerrit services from manage-projects
clarkbfungi: AJaeger looking at those review requests the first one in the list is for #openvswitch. They use meetbot, but don't have accessbot set up and now don't want accessbot with default registration requirement?14:28
fungier, i mean apt-get source14:29
mordredfungi: one of my long-standing gripes is that there isn't a "please install the build depends for this debian/control file" command - at least not in debian. there is something in ubuntu-dev-scripts I think14:29
corvusmordred: only
corvusnone of the 'node' 'jdk' etc stuff14:29
clarkbis that captured by their docker file?14:30
fungimordred: oh, actually i was thinking of `sudo mk-build-deps -i14:31
fungi(from devscripts)14:31
corvusclarkb: no, they don't build debs with docker (i don't know how they build them)14:31
mordredfungi: ah - does that exist now?14:31
mordredprobably build them on someone's laptop14:31
fungi"Given a package name and/or control file, mk-build-deps will use equivs to generate a binary package which may be installed to satisfy all the build dependencies of the given package."14:32
clarkbcorvus: another option may just be to grab their pacakges?14:32
clarkbthough it doesn't look like that is a proper repo :/14:32
corvusclarkb: that is what their docker file does; my goal is to build the system with a (2 line) patch.14:33
openstackgerritMonty Taylor proposed opendev/system-config master: Remove registry mirror settings
corvusi want to replicate the existing docker images as much as possible, so i figure that means i should start by building debian packages, then install those into my new docker images14:34
mordredfungi: thanks ^^ it was also actually semantically wrong too14:34
mordredcorvus: you are very close to nerdsniping me in to wanting to build the underlying zuul support for speculative deb repos to do the thing you're talking about systemically14:34
openstackgerritMerged opendev/system-config master: Use dev subdir on review-dev for project-config things
openstackgerritMerged opendev/system-config master: Install utility scripts for running jeepyb commands
corvusmordred: don't let me stop you -- but i don't think ill have any image composition here14:35
corvusi'm curious what build platform they use that has node 10 and opnjdk 814:41
openstackgerritMerged openstack/project-config master: check-release-approval: handle no-review case
fungicorvus: i have node 10 and openjsk 8 installed on my sid/unstable workstation14:44
mordredI think debian buster has those14:45
corvusoh, i thought it didn't have jdk 814:46
mordredoh - no - buster has openjdk 1114:46
mordredyeah - maybe they're using the upstream nodejs package repos for their node packages14:47
fungiyeah, openjdk8 seems to be in unstable but held out of testing from buster onwards14:47
clarkbbionic has openjdk8 and nodejs8 iirc14:47
mordredso bionic + node10 upstream would give those 214:48
fungi"Migration status for openjdk-8 (- to 8u242-b08-1): BLOCKED: Rejected/violates migration policy/introduces a regression"14:48
corvustheir runtime docker images are stretch, so maybe they use upstream node?14:48
fungifor the record as to why it's not in buster or testing14:48
mordredcorvus: ++14:48
mordredcorvus: yeah - I think use upstream node14:48
mordredcorvus: that's what we're doing in our gerrit images14:48
corvusand that won't show up as a runtime dependency anyway, so shouldn't be a big deal14:49
fungiand yeah, latest nodejs in stretch is 814:49
funginot 1014:49
*** ouroboros8 has quit IRC14:49
fungiso they must do something14:49
fungiand while both of those are technically available in debian/sid(unstable), i doubt that's what they're using14:50
corvusyeah, seems unlikely if they're producing docker images running on stretch14:51
AJaegerclarkb, yes, that's my understanding as well - my ping was before that - fungi, this is again about - what shall we do?14:53
AJaegerclarkb, fungi: we had in the past accessbot as pre-requisite for meetbot - is that a hard requirement?14:58
clarkbAJaeger: yes, but I think we may be able to update the configs for certain channels to drop the registration requirement?14:59
clarkbwe may also want to consider dropping it globally?14:59
clarkbwe get some spam in the unregistered channel but not much14:59
clarkbof course if we drop the requirement that may make us a more attractive location14:59
AJaegerclarkb: yes, we could fine tune here. If we drop it globally, we might have to revert and then it's back again15:09
clarkbya, I think that is my biggest concern. If we drop the requirement we'll become interesting again15:10
mnaserhi infra, can we get a hold on for 'openstack-operator:functional'15:18
mnaserit seems to be working in local testing.. so it'd be nice to have an environemnt once it breaks15:19
openstackgerritMerged opendev/system-config master: Set zuul_work_dir for system-config-*-image
openstackgerritMerged opendev/system-config master: Put gerrit image jobs into a project-template
clarkbmnaser: can we fix logging first?15:27
clarkb(the job indicates there are pods which have not started, but none of the pods have logs)15:27
mnaserclarkb: the logging actually works fine in other jobs which have not failed, we had pod logs15:27
mnaseri am just unsure at what i'm missing from logs to be able to figure out why those pods dont have logs15:28
clarkbmnaser: right probably because the pods have started :)15:28
clarkbI think there is a gap between logging "pods have not started" and "which pods have not started and why"15:28
clarkbI would start with "which pods did not start"15:28
mnaserthat is a much easier exercise in figuring out what to capture logs if i have an actual environment with logs that arent running (seeing what state they are, introspecting the commands to output)15:29
mnaserand probably a lot less useful of ~20m of CI time for every iteration till this fails15:29
clarkbI guess I'm saying there are no logs15:29
openstackgerritRadosław Piliszek proposed openstack/project-config master: Cache CirrOS 0.5.1
clarkbstarting with something is a good idea?15:29
clarkb(I don't want holding the job to result in not being able to debug this in the future)15:30
mnaseri'm not sure what do you mean by no logs15:30
mnaserkubectl logs?  they won't have anything if the pod isn't up15:30
clarkbmnaser: the job says "pods failed to start" then if you go to the pod logs there is no info15:30
mnaseri'm trying to get an autohold so i _never_ have to autohold again15:30
clarkbyou are missing output from k8s saying "pods xyz failed for reasons abc15:30
mnaserok so i should just iterate and spend 20 minutes of ci time on every attempt till i get it right15:31
clarkbI beliee that info is typically available as airship had the same issue a while back15:31
mnaserit'd be nice if airship published their logging stuff to zuul-jobs then :)15:31
clarkbyou'd have to take that up with them15:32
clarkband yes, for many moons the expectation is that jobs add appropriate logging when failures are obtuse15:33
clarkbthen if that doesn't clarify things we hold nodes15:33
clarkbthis has been true since the early devstack days15:33
clarkb(because it helps us build jobs that properly log around failure conditions)15:33
mnaserthe only thing is that will hurt by spending 20 minutes of CI time on every iteration15:33
mnaserso i can borrow a node for a 15 minutes to introspect and generate a kubectl command that will log things out15:34
clarkbif we are going to be sure to add additional logging to the jobs after I'll go ahead and hold the node15:37
clarkbthats really the bit I don't want to miss15:37
mnaserclarkb: i absolutely have no interest in ever wanting to hold instances tbh15:38
mnaserits easier and faster for me to _not_ have to ask for holds15:38
mnaserand the way to avoid it is by adding the missing gaps for logs when introspecting failures15:38
mnaser(and also trying to avoid blindly just logging $world too)15:38
clarkbmnaser: ok its set15:39
mnaserclarkb: thank you for being understanding15:39
clarkbfwiw this seems to be a common issue with k8s15:40
clarkbyour idea of adding that bootstrapping to zuul-jobs is a good one "appropriate logging for when k8s things fail)15:40
mnaserclarkb: i absolutely will end up with a depends-on for a zuul/zuul-jobs change from this :)15:40
mnasermaybe like "gather all info" or something15:41
clarkbseems like beacuse k8s retries things until they work (at least that seemed to be the airship problem) if they fail intermittently locally but consistently in the gate just miss the data15:41
mnasercollect-kubernetes-logs .. i dunno, ill find the right place15:41
openstackgerritArtom Lifshitz proposed openstack/project-config master: Add whitebox-tempest-plugin under QA
clarkbya being able to skim k8s for error logs in particular would be useful I think15:41
openstackgerritRadosław Piliszek proposed openstack/project-config master: Cache CirrOS 0.5.1 for AArch64 too
clarkbconfig-core ^ that is probably a good one to land15:51
AJaegerclarkb: so, we bake both AArch64 and x86-64 images into all our media?15:58
clarkbAJaeger: ya, until we do the improvements that ianw would like to do15:59
clarkbI'm not quite sure what that looks like yet, but I'm pretty confident we can sort it out15:59
clarkbAJaeger: these images are like 12MB so it should be fine15:59
openstackgerritArtom Lifshitz proposed openstack/project-config master: Add whitebox-tempest-plugin under QA
AJaegerclarkb: that small? Ok...16:01
clarkbAJaeger: ya its basically busybox with scripts taht pretent to be cloud-init and dropbear16:01
clarkbthey've done a good job minimzing the images16:01
AJaegerclarkb: do you want to +2A - that's the x86-64 part. I'll approve the AArch64 now ;)16:02
AJaegerI knew they were good - just not that good ;)16:02
clarkbdone thanks16:03
AJaegerwe just had a rename slot last week - and now 714478 comes along ;(16:06
clarkbAJaeger: well with releases schedule things for openstack and having just done it that one may just have to wait :)16:06
mordredAJaeger: perhaps we should suggest that they simply move the repo to the openstack tenant for now so that it can sit alongside tempest ... but yeah, we'll need to wait for release schedule to actually do the rename16:08
AJaegerit's in the openstack tenant already, just not in the namespace16:09
AJaegerso, nothing is blocking them to work on it ;)16:10
mordredclarkb: got a sec for ?16:12
openstackgerritMerged openstack/project-config master: Cache CirrOS 0.5.1
openstackgerritMerged openstack/project-config master: Cache CirrOS 0.5.1 for AArch64 too
AJaegerclarkb: could you review an old change of yoursr that I updated, please?
clarkbmordred: AJaeger yup I'll get to those as soon as my meeting is over16:14
AJaegersure, thanks16:14
mnaser714340 which has a hold on it has failed, can i get my keys please?
mordredmnaser: on it16:24
mordredmnaser: done16:25
mnaserthanks mordred16:26
*** hashar has joined #opendev16:33
mnasermordred, clarkb: thanks, we caught the failure, so we're looking at adding kubelet logs + a kubectl describe of all resources inside the cluster (similar to collect-container-logs)16:34
mnaserthis should probably give us all the info that we need, Openk10s still using that vm to test16:34
mnaserand we'll probably have a patch up to zuul/zuul-jobs for both hopefully16:34
mordredmnaser: what's an Openk10s? I feel like I'm not cool enough to know the latest trendy things :)16:36
mnasermordred: someone's IRC nickname which is a play on words based on their name :P16:36
mordredI was thinking it was some new weird openstack k8s hybrid craziness :)16:37
mnaseralso what they're working on :p16:37
mordredclarkb: actually, first (it needs to go in first cause depends-on)16:56
mordredcorvus: ^^ if you have a sec to look at that one16:56
openstackgerritJames E. Blair proposed opendev/system-config master: Build jitsi-meet images
corvusokay, that worked locally (at least before i squashed all the RUN lines)17:00
mordredcorvus: what are apt-dpkg-wrap and apt-cleanup - are those things I should know about?17:04
corvusmordred: no, they're local to that image17:04
corvusthey get copied in from one of the 'rootfs' dirs17:04
corvusthey are simple wrapper scripts that do what you expect17:05
mordredah! gotcha17:05
corvus(that would be cool if they were some kind of universally accessible helper tool; that's what i thought/hoped at first too)17:05
openstackgerritArtom Lifshitz proposed openstack/project-config master: Add whitebox-tempest-plugin under QA
*** rpittau is now known as rpittau|afk17:08
openstackgerritArtom Lifshitz proposed openstack/project-config master: Add whitebox-tempest-plugin under QA
openstackgerritMerged openstack/project-config master: Clean up infra gerritbot irc channel configs
openstackgerritMerged opendev/system-config master: Remove registry mirror settings
openstackgerritSaul Wold proposed openstack/project-config master: Add Rook to StarlingX
*** hashar has quit IRC17:30
mordredJust going to leave this here for everyone:
*** dpawlik has quit IRC17:37
*** lpetrut has joined #opendev17:40
clarkbmordred: corvus looks like the upstream jitsi images are not like the ones we already build? They appear to produce somewhat all in one containers17:43
clarkbI don't think that is a major issue, but something to be aware of because we'll end up with a web setup running nginx, and the app, and certbot, etc17:43
clarkbalso I expect we need to update ?17:45
mordredcorvus: welp - I have run the failing task from the openstackclient promote job (the one that fails getting a JWT token) locally and it succeeded17:45
mordredbut that was with the password set directly. I guess the next step is to decrypt the zuul secret myself and see what's different?17:46
mordredclarkb: do we have a doc that points out of how to do that decrypt by hand? I have not yet had to do this debugging step17:47
clarkbmordred: ya fungi wrote a thing up17:47
clarkbthough not sure if it merged /me looks17:47
clarkbmordred: it did merge17:48
mordredclarkb: cool thanks!17:48
corvusmordred: it probably has a trailing newline17:50
mordredcorvus: yeah - it does look that way. does echo secret | zuul_encrypt  not do it right I guess?17:53
fungimordred: echo -n17:53
fungiif you don't want a trailing newline17:53
mordredcorvus: so I guess it worked for the intial push because that's using creds written to a config file for docker - but not for the retag since that's passing the creds to the uri module17:57
fungithat sounds likely17:58
mordredyeah - upload uses docker login on the command line - so it doesn't look like that would allow setting a password that validly had a trailing newline in its contents17:59
openstackgerritJames E. Blair proposed opendev/system-config master: Build jitsi-meet images
openstackgerritJames E. Blair proposed opendev/system-config master: jitsi-meet: open etherpad on join
mordredwould it be reasonable to add a strip to the uri password parameter?17:59
corvusmordred: yes, or we could change the default for encrypt-secret to strip18:00
mordredcorvus: I don't suppose leading or trailing whitespace are usually valid tokens in these contexts18:00
mordredcorvus: maybe both?18:00
corvusmordred: wfm18:01
fungiwhich makes it harder if folks want to start or end a secret with whitespace, but i suppose if it's clearly documented and there's a --no-strip-whitespace or something then it's probably less confusing than the current behavior18:01
corvusfungi: yeah, there's a --strip option; i think we should just invert the option/default18:01
fungi(because you almost never want to include whitespace at the start or end of a secret)18:01
corvusor at least default strip for stdin18:01
corvusand default to nostrip for file18:01
corvus(but that could be confusing)18:01
*** dpawlik has joined #opendev18:02
fungithat seems reasonable18:02
fungibut yeah, harder to explain18:02
fungiand harder to explain means more likely for folks to trip over18:02
* mordred will make some patches18:03
openstackgerritMonty Taylor proposed zuul/zuul-jobs master: Trim whitespace from uri password for docker promote
fungimordred: also corvus's may be worth looking at18:04
mordredfungi: cool18:08
mordredfungi: btw - in your docs it says "no need to dedent" - I did not find that to be true on zuul.o.o18:08
fungimordred: did you remove the leading - from the list items?18:09
fungithat's necessary, but removing the whitespace shouldn't be18:09
mordredI did18:09
fungibase64 -d is supposed to eat whitespace18:10
mordredif you do: "cat testsecret | base64 -d | sudo openssl rsautl -decrypt -oaep -inkey /var/lib/zuul/keys/secrets/project/gerrit/openstack/python-openstackclient/0.pem"18:10
mordredin /root on zuul.o.o you can verify18:10
mordredfungi: I get 140095120840344:error:04065072:rsa routines:RSA_EAY_PRIVATE_DECRYPT:padding check failed:rsa_eay.c:602:18:11
mordredalso - those docs aren't in system-config any more ... did they move?18:12
fungiindeed, `cat testsecret | base64 -d >/dev/null` reports "base64: invalid input" but `sed 's/^ *//' testsecret | base64 -d >/dev/null` works18:12
mordredI'd fix the docs - but they don't seem to exist any more :)18:13
clarkbthey may have got eaten by the removal of the v3 migration docs18:14
mordredah - good point18:14
openstackgerritMonty Taylor proposed opendev/system-config master: Re-add secret decrypting docs
mordredclarkb, fungi: ^^ there - and updated to include the sed command18:17
fungiactually it got eaten by
fungiwhich ripped out a bunch of the stuff i'd already updated our zuul docs with18:17
mordredwould it be better to add that seciton to the zuul docs instead?18:17
fungilooks like that change assumed the zuul.rst hadn't already been updated for v318:18
fungiand then replaced it with the zuulv3 migration doc18:18
openstackgerritJames E. Blair proposed opendev/system-config master: Add meetpad server
openstackgerritJames E. Blair proposed opendev/system-config master: Build jitsi-meet images
openstackgerritJames E. Blair proposed opendev/system-config master: jitsi-meet: open etherpad on join
openstackgerritJames E. Blair proposed opendev/system-config master: Use our jitsi-meet image for meetpad
fungiwhich we then ripped out more recently18:18
corvusokay, i think that stack of 4 changes should completely implement the meetpad spec18:18
mordredcorvus: can you explain making opendev-buildset-registry have requires: jitsi-meet-container-image? is that on purpose?18:20
corvusmordred: yes, and we should probably do it for everything; that's where pull-from-intermediate-registry is run, so if we build an image in a previous change, we need that job to see the artifact and pull it so that it's in the intermediate registry18:21
corvusmordred: i expect that stack would fail without it, because #3 builds an image without running the 'run-meetpad' job, and #4 runs 'run-meetpad' without building an image18:21
mordredcorvus: yeah - we should totally update our other jobs to do that18:22
corvusmordred: so without that, i would expect #3 to pass, and #4 to fail because there is no image on dockerhub yet18:22
corvusmordred: in the other cases, we're silently falling back to dockerhub18:22
corvusoh, that reminds me, i need to qualify image names in dockerfile18:22
corvusand docker-compose18:22
corvusupstream didn't do that, but i think that's a change we should carry18:22
mordredcorvus: requires is already soft right? so it's safe to require and not have something in the stack providing it18:23
corvusmordred: correct18:23
corvusso i think the final state should be to just have a really long list on that job18:23
mordredI think we're fine on the gerrit changes _for_now_ because they're all triggering build at the same time18:23
corvusyeah.  i think we should do that once all the gerrit changes have landed18:24
corvusbecause right now, it's going to be a big merge conflict18:24
mordredso big18:24
corvusmordred: FROM as build18:26
corvuslike that?18:26
mordredcorvus: yup18:27
mordredcorvus: "Unable to freeze job graph: Project opendev/jeepyb is not allowed to run job system-config-upload-image-gerrit-base18:28
mordredI don't have an allowed-projects in system-config ...18:28
corvusmordred: it's probably implied because of the secret18:28
mordredoh. hrm.18:29
corvusmordred: if the 2 projects trust each other with the secret, you can attach it to the pipeline in a config project18:29
mordredok. I think that's what I have to do in this case18:30
openstackgerritJames E. Blair proposed opendev/system-config master: Add meetpad server
openstackgerritJames E. Blair proposed opendev/system-config master: Build jitsi-meet images
openstackgerritJames E. Blair proposed opendev/system-config master: jitsi-meet: open etherpad on join
openstackgerritJames E. Blair proposed opendev/system-config master: Use our jitsi-meet image for meetpad
openstackgerritMonty Taylor proposed openstack/project-config master: Run gerrit image jobs on jeepyb patches
openstackgerritMonty Taylor proposed opendev/jeepyb master: Run pep8 job in-repo
openstackgerritMonty Taylor proposed opendev/jeepyb master: Clean up some more python3 things
mordredcorvus, clarkb: should be the right way to run gerrit image jobs on jeepyb18:35
openstackgerritMonty Taylor proposed opendev/jeepyb master: Run pep8 job in-repo
openstackgerritMonty Taylor proposed opendev/jeepyb master: Clean up some more python3 things
openstackgerritMonty Taylor proposed opendev/jeepyb master: Run pep8 job in-repo
openstackgerritMonty Taylor proposed opendev/jeepyb master: Clean up some more python3 things
corvusmordred: just thinking out loud: you might be able to add opendev-buildset-registry to the project-template; it should be harmless to have it in there twice (at least as long as we're not setting any attributes like 'dependencies')18:37
corvusi'm not sure i'm ready to advocate that18:37
corvusmordred: +2 as written18:37
mordredcorvus: yeah - I had a similar thought and also wasn't 100% ready for that :)18:37
* clarkb is catching up on new set of rules for living18:41
mordredcorvus: you got new rules?18:41
clarkbya bunch of businesses have to close now, but also all sports courts, pools, playgrounds, and skate parks18:42
clarkbwhich honeslty is a good thing based on my non scientific observations of neighborhoods while biking18:43
clarkbplaygrounds and sports courts in particular have been packed18:43
mordredI was sad about the tennis court closure here - because tennis is played at a distance ... but there's still a human who has to collect the money from people, so it makes sense18:44
clarkbmordred: I've seen a lot of basketball and volleyball, neither of which are very distant18:46
clarkbalso lots of kids on playgrounds18:47
mordredclarkb: well, when you're back, would be helpful18:49
clarkbya  Ithink I've gotten through it now. No real change for us since we've been social distant already and I get to work from home everyday18:49
clarkbmordred: is pep8 defined in repo? or are not worrying about pep8 anymore?18:52
mordred(which could also do with a +A :) )18:53
mordred is the ultimate prize here fwiw18:53
mordredclarkb: woot. thanks!18:55
mordredclarkb: I think once that lands we'll have an image we can do a restart on once we're ready to try that18:55
openstackgerritMerged opendev/system-config master: Re-add secret decrypting docs
fungii guess gerritbot hasn't caught up on the infra-manual rename yet18:57
fungi Restore Zuul v3 Migration Guide bits on secrets18:58
fungioh right, because review.o.o is still in the emergency disable file18:59
fungiso that part of the rename change never got applied to ir19:00
fungier, to it19:00
openstackgerritMerged openstack/project-config master: Run gerrit image jobs on jeepyb patches
mordredfungi: hopefully soon we'll undisable that ...19:05
fungiyeah, not an inconvenience, was just confused for a moment and thought gerritbot might be broken until i realized what was going on19:12
*** lpetrut has quit IRC19:12
*** diablo_rojo__ has joined #opendev19:13
*** ralonsoh has quit IRC19:16
openstackgerritMerged opendev/jeepyb master: Run pep8 job in-repo
openstackgerritMonty Taylor proposed opendev/system-config master: Add system-config to required-projects for gerrit images
mordredclarkb, fungi, corvus: ^^ sigh/whoops19:22
openstackgerritOleksandr Kozachenko proposed zuul/zuul-jobs master: add role for collecting the kubernetes pod&kubelet logs
*** hashar has joined #opendev19:37
*** diablo_rojo__ has quit IRC19:49
*** diablo_rojo__ has joined #opendev19:51
openstackgerritOleksandr Kozachenko proposed zuul/zuul-jobs master: add role for collecting the kubernetes pod&kubelet logs
openstackgerritOleksandr Kozachenko proposed zuul/zuul-jobs master: add role for collecting the kubernetes pod&kubelet logs
corvusmordred: oy, i was wrong; we only run the pull role on the docker image build job which is unfortunate for our system-run jobs20:17
corvusi think we can solve this either by running pull in the base job, or making a new almost-base job which adds that onto base (then parent the system-run jobs to that, then we also don't need to whole big list of images on the buildset-registry job)20:19
corvusi'll work on the second thing since it's more self-contained20:20
corvusoh actually, opendev-build-docker-image-base fits that description but just has a weird name20:21
mordredcorvus: I agree - the second thing sounds more correcter20:21
fungi#status log manually deleted old /afs/ which was previously copied and switched to .../osf/openstackid/20:23
openstackstatusfungi: finished logging20:23
openstackgerritJames E. Blair proposed opendev/system-config master: Use our jitsi-meet image for meetpad
corvusso we may only need that for it to work20:24
corvusmordred: then with that we can start adding requires: entries to "systtem-config-run-foo" jobs, which i think will make a lot of sense20:25
corvusmordred: in fact, if that change works, i think i might pull that part out and just do a standalone change to reparent that and start doing that.  i think we can land that without throwing a wrench into ether gerrit or meetpad.20:26
mordredcorvus: yeah - the requires entries on the run-foo jobs make more logical sense20:26
mordredcorvus: is there any reason to not just put the pull into the base job? like - could we protect it with a check to see if a docker exists?20:27
openstackgerritOleksandr Kozachenko proposed zuul/zuul-jobs master: add role for collecting the kubernetes pod&kubelet logs
corvusmordred: i think we can do that, it will just take a few changes to do (at least 2 changes to zuul-jobs, and then a nice base/base-test series in base-jobs)20:29
mordredcorvus: cool. I think that's ultimately worthwhile20:29
ianwclarkb: apropos arm64 did you see to add a new larger memory label?20:42
openstackgerritMerged opendev/system-config master: Add system-config to required-projects for gerrit images
ianwi feel like we might have done this for hrw at some point?  i have dejavu we discussed it before20:43
clarkbianw: I hadnt, but ya I think we knew going in with arm64 we'd have to be flexible with sizing since its a different system20:43
*** dpawlik has quit IRC20:43
ianwyeah, i have not i admit looked at the details of why it needs more memory.  now i think of it, we should probably add it as a task to see if we can reduce it if/when things are working devstack wise20:45
ianwi'll have to find the story ... first some breakfast :)20:45
corvusa change of mine just hit a bunch of retries on several jobs; i investigated one of them, and it appears to have been a host unreachable due to ssh host key changing error20:46
corvusi don't see anything systemically wrong on the graphs right now20:47
ianwfungi: that sounds good ... i guess if you're in the pip game, you already expect some level of release-to-release ... instability; if you don't, you're in for a surprise20:47
corvusmaybe call this a data point and see if we get another one20:47
mordredcorvus: you're a data point20:48
clarkbcorvus: we see that periodically in rax20:48
fungiianw: we've been lamenting the lack of a resolver in pip since... well since for ever basically20:48
clarkbcorvus: from an outside perspective it seems like it will happen for a short ish period of time then they clean up the bad ips20:49
clarkband it goes away20:49
corvusclarkb: just happened on ovh-bhs120:49
fungisome release jobs hit the same thing late last week20:49
clarkboh thats new then20:49
fungibut i forget which provider20:49
clarkbbeing a different cloud I mean20:49
corvusi'm also seeing weird errors here: 20:43 < clarkb> ianw: I hadnt, but ya I think we knew going in with arm64 we'd have to be flexible with sizing since its a different system20:51
corvus20:45 < ianw> yeah, i have not i admit looked at the details of why it needs more memory.  now i think of it, we should probably add it as a task to see if we can reduce it if/when things are working devstack20:51
corvus              wise20:51
corvus20:45 < fungi>
corvusi can apparently not copy/paste from the zuul console stream?20:52
ianwlast host key weirdness from my notes i saw was @ in rax20:52
ianwso there's another data point :)20:52
corvusyou'll just have to take my word that there's a permission denied error20:53
corvusthat seems unrelated to the later unreachable error20:53
mordredfungi: maybe we shoudl start running pip check in devstack after we're installed everything20:53
mordredfungi: just to be sure20:53
*** dpawlik has joined #opendev20:53
corvusit seems very strange to have 2 errors like that20:53
corvusif you look at  you can see the "Permission denied error"20:54
corvusthat later also encountered the unreachable error, but that seems to be after the upload?20:55
corvuson "get df disk usage" ?20:55
clarkbcorvus: there are quite a few layers to this based on previous investigation20:55
fungiah, the recent release job failure i'm finding from thursday hit in inap:
clarkbcorvus: things that use the control peristent connection tend to be most reliable20:55
corvusclarkb: i think i'm going to assume this is something different since it's in ovh20:55
clarkbbut then you'll get similar errors if ansibel can't exec things on the remote side potentialyl due to running out of disk20:55
corvusand there are 2 errors involved20:56
clarkbwhich is where the df check came in20:56
clarkbcorvus: I just mean in general20:56
clarkbansible errors aren't always what they seem at first glance20:56
corvusclarkb: if we see the "host key" error from ssh, is that reliable?20:56
clarkbcorvus: yes I believe the host key mismatch errors are reliable20:57
clarkbthe permission denied can be more generic iirc20:57
openstackgerritMerged openstack/project-config master: Linaro US:  Add a 16GB RAM label for bionic
mnaserclarkb: as promised - :)20:58
corvusclarkb: how about this theory: the permission denied is a red herring because these are system-config tests which rewrite our keys.  they run after everything else in the cleanup playbook and so errors normally go unnoticed.  they are still harmless.20:58
clarkbcorvus: ah ya that could be20:59
corvusso maybe what we should do there is add some failed_when lines so that the next time there's an error, we don't spin wheels :)20:59
corvusso the *actual* problem is the permission denied for the docker logs20:59
mnaserinfra-root: hold on 0015407770 can be removed, thanks21:00
openstackgerritMonty Taylor proposed opendev/system-config master: Use openstackclient from container
mordredmnaser: cool21:01
corvusokay, i think i see the problem; i think i'll fix it by doing that whole pull in the base job thing21:02
mordredcorvus: ++21:03
mordredcorvus, clarkb, mnaser: we currently have opendevorg/python-base pinned to python:3.7 - the latest python image is on 3.8.2 ... how should we approach updating that?21:09
clarkbI want to say we tried updating but something broke under it?21:09
mordredyeah - although I think that might have been stretch->buster related?21:10
clarkbpersonally I have no issue with sticking to older python21:10
mordredmaybe we should make an opendevorg/python-base:3.7 and an opendevorg/python-base:3.8?21:10
clarkbthat would probably help us transition21:10
clarkbsince we don't have to do it all at once21:10
mordredclarkb: well - I don't either - but at _some_ point updating is probably not a bad idea21:10
mordredkk. I'll make a patch to make a 3.7 and a 3.821:11
mordredwe can tag :latest still at 3.7 for now since that's all we've had - but maybe go around and either make sure we're pinning consumption-side or that we're ok bumping with a latest bump21:12
mordredcorvus: can provides: be a list?21:17
corvusmordred: yes21:19
*** dpawlik has quit IRC21:20
openstackgerritMonty Taylor proposed opendev/system-config master: Disable recommends in python-base and python-builder
openstackgerritMonty Taylor proposed opendev/system-config master: Start making 3.8 python images
mordredclarkb, corvus: I think that should do the right thing with getting us new versions but without breaking anything existing21:31
clarkbmordred: zuul says that has a syntax error21:34
mordredclarkb: zuul is almost certainly right21:34
mordredmore importantly ...21:34
mordredputting the system config jobs on jeepyb in project-config did not work:
mordredstill getting Unable to freeze job graph: Project opendev/jeepyb is not allowed to run job system-config-upload-image-gerrit-2.1321:35
clarkbis it a final job?21:35
clarkbor protected?21:35
openstackgerritMerged zuul/zuul-jobs master: add role for collecting the kubernetes pod&kubelet logs
*** hashar has quit IRC21:37
corvusmordred: maybe that needs an explicit allowed-projects?21:38
mordredcorvus: should I put it on each leaf job I suppose?21:39
corvusmordred: hang tight a minute, i'm reading code21:39
corvusmordred: there is a distinct possibility that the bug here is the job is in a project template21:40
mordredoh! well that's certainly fun21:41
corvusi'm still reading, but that's my current theory21:41
corvusno that's a bad theory21:43
corvushow about this one: the project-template is not in a config-repo21:45
fungidid we recently manage to remove puppet-beaker-rspec-puppet-4-infra in the opendev tenant without removing references to it?
corvusmordred: yep, i think that's the bug.  the thing that attaches internally to say "it's okay to run this cross-repo" is attached to the project template itself, not the use of the project template in the config project.  i think that's a bug, because we may not necessarily want to have a project-template like that defined in a config project.  we want the config project to be the one granting permission,21:48
corvusnot necessarily defining the template itself.21:48
clarkbfungi: more likely something moved into that tenant that tries to run it?21:49
clarkbfungi: opendev/sytem-config is primarily served by the openstack tenant at this point21:49
corvusmordred: immediate fixes: don't use a project-template or move the project-template to project-config.21:49
mordredcorvus: ah - yes, this makes sense21:49
mordredcorvus: yeah. I think I'll just skip using the project template for now21:49
mordredfungi: and what clarkb said - the opendev tenant has an incomplete view of system-config and is frequently unhappy21:50
mordredthose legacy jobs are one of teh reasons we can't fully move it21:50
mordredsince we dont' have openstack-zuul-jobs legacy-base in the opendev tenant (nor should we)21:50
clarkbwe're slowly getting there21:50
clarkbwe deleted a bunch of puppet jobs for centos and trusty that we didn't need anymore21:50
mordredI've actually got it on my personal cleanup list to go unlegacy those (I'd like to just delete them - but unlegacy-ing them shouldn't be hard)21:50
mordredthen we shoudl be able to move most of opendev repos into opendev tenant :)21:51
fungiclarkb: ahh, okay, yeah makes sense21:52
openstackgerritMonty Taylor proposed openstack/project-config master: Apply gerrit jobs directly to jeepyb
mordredclarkb, corvus, fungi: ^^ that shoudl work around the zuul issue corvus identitied21:53
clarkbmordred: needs yamling better21:54
clarkbmordred: the trailing : needs removing21:54
openstackgerritJames E. Blair proposed zuul/zuul-jobs master: Allow pull-from-registry to be used in base jobs
clarkbor maybe add in dependencies like the others?21:55
openstackgerritMonty Taylor proposed openstack/project-config master: Apply gerrit jobs directly to jeepyb
mordredclarkb: or - I need to properly cut-and-paste22:00
openstackgerritJames E. Blair proposed opendev/base-jobs master: Add opendev-buildset-registry-consumer job
corvusmordred: I started the base-jobs approach in 714541, but i'm kind of leaning toward the explicit parent approach in 714542, mostly just to keep the base job simple.22:06
corvus(there are some trade-offs either way)22:06
corvusoh no i have trailing whitespace22:06
ianwfungi / clarkb : (as the original reviewers on :) do you think we still need NODEPOOL_TARBALLS_PROXY ?  i'm not seeing it used anywhere from codesearch22:07
openstackgerritJames E. Blair proposed opendev/base-jobs master: Add opendev-buildset-registry-consumer job
clarkbianw: we push a lot of bw through tarballs.o.o based on the goaccess reports22:07
clarkbianw: that said I'm not sure of what jobs need to use that caching proxy so not sure where to start in making that better22:08
ianwyeah, well if anything is using, it's not, it would seem, being used via the NODEPOOL_TARBALLS_PROXY env we setup22:08
fungisome projects are/were using tarballs.o.o to cache artifacts used by subsequent jobs22:09
fungialso a number of ci jobs consume things like service vm images from tarballs.o.o22:09
ianw... but do they do that via the reverse proxy on a mirror node?22:10
clarkbya I think te service vm images may be the biggest usage of it now22:10
fungii don't know that any of them switched to using it22:10
clarkbianw: probably not if that didn't end up getting fully realized22:10
fungiand now that it's on afs anyway, we could do it more like we do distro package caches22:10
corvusclarkb: can you go ahead and look at 714542?  (it's in base-jobs, so i can't depends-on it)22:11
clarkbcorvus: thats a simplification for downstream jobs?22:12
corvusclarkb: it's for downstream jobs, but not a simplification -- it has to run in base-jobs because it uses secrets22:12
clarkboh right22:12
corvuswould we be okay making all of the system-config-run jobs depend on the buildset registry job?22:16
corvusonly about half of them use images right now, so half actually don't have to wait for it and can run quicker22:16
*** DSpider has quit IRC22:16
clarkbsome things like bridge and the nameservers may never get that relationship22:16
clarkbmaybe we want to have a second parent job to capture it for those that do?22:16
mordredit doesn't bother me to have them all wait - but I'm fine with whatever other people want22:17
corvusyeah... or... maybe we should land both changes, and use a job variable to allow pull to no-op for the jobs that don't22:18
corvus is the other22:18
openstackgerritMerged opendev/base-jobs master: Add opendev-buildset-registry-consumer job
openstackgerritIan Wienand proposed opendev/base-jobs master: Remove env var for tarballs mirror
openstackgerritJames E. Blair proposed opendev/system-config master: Use our jitsi-meet image for meetpad
openstackgerritIan Wienand proposed opendev/system-config master: Remove /tarballs proxy from mirrors
clarkbcorvus: mordred I feel like I'm missing some fundamental background thing for these changes22:21
clarkbhaven't we been running these jobs with a buildset registry (which forwards to the intermedaite registry) already?22:22
corvusmordred, clarkb: ^ okay, 714510 is what it would look like if we did that.  take a look at the run-meetpad job -- we'd just set that variable on the jobs which use it, and do nothing on the jobs that don't.22:22
corvusclarkb: kindasorta22:22
mordredclarkb: we have - but in the case of the system-config-run-* jobs ...22:22
mordredsince they don't build the images themselves, there has been an incomplete config22:22
clarkbwhere were they getting the images before?22:23
corvus(they have been silently falling back to using images from dockerhub, rather than images from previous changes)22:23
clarkbthey were still pulling them from the buildset registry right?22:23
mordredif the change in question built an image - yes - if a change above it in the stack built the image - no22:23
clarkbI see so it is the intermediate registry integration specifically that was missing22:24
mordredclarkb: so - we were pulling from the buildset registry, but missing things only from previous changes from the intermediate registry22:24
clarkbeven if we were running ab uildset registry (I thought those roles always populated the buildset registry from the intermediate registry but I guess not)22:24
corvusthe case where we miss a change is pretty rare, we *usually* build and use images together and call it done.  but i just made a stack that built an image without using it, then a second change that used it without building it.22:25
clarkbcorvus: rather than what does should we just require the buildset registry always then have two sets of parent jobs for system-config-run those with and without docker images?22:26
corvusclarkb: they populate the buildset registry from artifacts zuul tells it about, so the buildset registry job itself, since it's standalone, doesn't get any artifact data, so it doesn't pull anything.  it's only the build jobs (or, with these changes, now also the run jobs) which get the artifact data and pull them.22:26
corvusclarkb: then we would have 2 different system-config-run jobs22:27
corvus(and system-config-run isn't trivial, so i don't really want to duplicate it)22:28
corvus(we could use a mix-in parent)22:28
clarkbya thats what I'm thinking 3 base jobs essentially (or maybe just 2)22:28
corvus(but that's more typing than this)22:28
clarkbbasemost does system-config run things without docker then another that does with docker things22:28
corvusclarkb: without using multiple inheritance, we'd have to duplicate this:
corvuswe could do that without duplication if we did use multiple inheritance though; let me mock that up and see if we like it22:30
mordredI think I prefer just running the pull in more places even if we don't need it everywhere - once we start hitting multi-inheritance it adds a comprehension cost22:31
openstackgerritMonty Taylor proposed opendev/system-config master: Start making 3.8 python images
openstackgerritJames E. Blair proposed opendev/system-config master: WIP: multi inheritance system-config-run-containers
corvusclarkb, mordred: that's better than i thought it would be22:33
corvusif i'm right about this, that change should actually pass tests, even though its parent will fail (because the zuul-jobs change would need to land first since it's used in a config-project)22:34
corvusclarkb: is that sort of what you were expecting, or have i misunderstood?22:36
clarkbcorvus: the shape of it wasn't quite what I expected (because I hadn't fully appreciated that we need to parent the base-jobs job) but the functionality is what I expected22:38
clarkbthen ya jobs that use containers can parent to the containers job and those that don't don't22:38
mordredcorvus: I agree - it's not as bad as I thought22:39
openstackgerritMonty Taylor proposed opendev/system-config master: Start making 3.8 python images
corvusclarkb: yeah, *something* has to be parented to *something* in opendev/base-jobs because that's where the pull role has to run22:40
mordredcorvus: mind +Aing this: real quick?22:40
corvusmordred: +3; hopefully that will work, confirm the bug, then we can fix zuul22:41
mordredcorvus: thx22:41
mordredcorvus: and yah22:41
mordredalso - hopefully it will result in a working gerrit image22:42
clarkbcorvus: I think I like 714547 because it makes the behavior explicit22:43
clarkbcorvus: its a bit more mind warpy in the config itself, but we don't have to rely on special behavior from the roles22:43
corvuswe can probably address the mind-warp with comments22:44
mordredyeah - I think I like 714547 too22:44
mordredalthough I wasn't expecting to22:44
corvusi'll wait for some green on 547, and if it's heading the right direction, i'll squash it into 510 (which, as predicted, has started failing)22:45
clarkband ya I agree comments should help with the config understanding22:45
corvusi'll put 541 back in WIP (we could still land it anyway of course)22:46
*** persia has quit IRC22:46
openstackgerritMonty Taylor proposed opendev/system-config master: Start making 3.8 python images
*** persia_ has joined #opendev22:47
clarkbI don't know if this has been said yet but I always read meetpad as meatpad. It doesn't help that my local butcher is "The Meating Place"22:49
mordredclarkb: I tnik that's one of the things I like about it22:50
ianwroot@bridge:~# /home/ianw/env/bin/openstack --os-cloud=openstackci-rax --os-region=DFW volume list22:53
ianwInvalid volume client version '1'. must be one of: 2, 322:53
ianwhave i upgraded into a version of openstack client that doesn't support rax or something?22:53
clarkbianw: sort of22:54
clarkboh wait22:55
clarkbso we pinned the version to version 1 in our clouds.yaml22:55
clarkbbecause thats the one that worked and now its an invalid version?22:55
clarkbmordred: ^ we probably don't watn to do that if we intend on these things to keep working with clouds in the wild22:55
mordredmaybe we should unpin that?22:55
clarkbmordred: we can't22:55
mordredone sec22:55
clarkbrax only does volume api v122:55
fungiianw: the openstackclient at ~fungi/launch-env/bin/openstack is still working if you need a point of comparison22:55
clarkbthere was like a whole afternoon of figuring this out between keystone api versions and cinder22:56
fungiwith OS_CLIENT_CONFIG_FILE=/etc/openstack/all-clouds.yaml22:56
clarkbbecause they both have to be pinned back22:56
ianwfungi: ahh, 3.19.0 v 4.0.022:56
mordredclarkb: it might take me a while to fix osc in that respect - we haven't changed osc to use sdk for cinder yet22:56
openstackgerritMerged openstack/project-config master: Apply gerrit jobs directly to jeepyb
mordredbut it's obviously on the list :)22:56
ianw    # Try a small import to check if cinderclient v1 is supported22:58
ianwi am guessing this fails22:58
mordredclarkb: rax supports v222:59
clarkbmordred: it didn't work22:59
mordred>>> c.block_storage.get('').json()22:59
mordred{'versions': [{'status': 'SUPPORTED', 'updated': '2014-06-28T12:20:21Z', 'id': 'v1.0', 'links': [{'href': '', 'rel': 'self'}]}, {'status': 'CURRENT', 'updated': '2012-11-21T11:33:21Z', 'id': 'v2.0', 'links': [{'href': '', 'rel': 'self'}]}]}22:59
ianwI get "public endpoint for volumev2 service in DFW region not found" from cmd line22:59
clarkbianw: ya that22:59
mordredignore the cmd line for a sec .22:59
mordredlet's figure out what actually works and then what we need to do to configure things to use22:59
clarkbmordred: my understanding of the issue was their catalog does not list the v2 api23:00
clarkband that is how tools like osc decide whattehy can talk to23:00
clarkbif the service itself does offer it then the catalog hasn't been updated23:00
clarkb(and that was sorted out after pinning the keystone api back first)23:01
clarkb(because only keysteon v2 works)23:01
mordredyes - that's right - there is a missing catalog entry23:01
mordredhowever - we should just put in an endpoint override23:01
openstackgerritJames E. Blair proposed opendev/system-config master: Use our jitsi-meet image for meetpad
mordredsince block-storage v2 is there23:01
mordredand that's much more supportable23:02
corvusmordred, clarkb: it looked like it was shaping up as expected, so i updated 510 to use multiple-inheritance.23:02
mordred(I agree, we shouldnt' have osc not supporting v1 - and we'll fix that)23:02
clarkbmordred: can we do that separately for each region?23:02
mordredclarkb: yes23:02
mordredclarkb: we can also use a substitution variable on region name23:02
mordredbut I think doing it per-region is likely more readble23:02
mordredclarkb, ianw: I need to afk for a minute - but I really want to get this sorted properly so that we don't have to remember to use an old install of osc that's sitting /root when we talk to rax23:03
mordredso - I'll get a patch up when I get back if y'all haven't gotten to is already23:04
clarkbmordred: I'm still not sure how to express this the way you are suggesting fwiw23:04
clarkbso if you are able to push that might be a good thing23:04
ianwok ... we are sayign we need something in our clouds config to override this?23:04
clarkbianw: ya per region cinder endpoint overrides to ignore what the catalog is telling us23:05
clarkbianw: I don't know how to do that but apparently it is possible :)23:05
ianwalright, i'll do some reading too :)23:05
openstackgerritMonty Taylor proposed opendev/system-config master: WIP override rax block storage endpoint
mordredclarkb, ianw: something like that23:09
mordredof course -we should probably just put that into the vendor settings for rax too23:09
clarkbwe'll want to https those and then add it to all the clouds.yaml files with rax regions23:10
mordredthat's just the real quick "this is what it should look like"23:10
mordredbut there may be more rabbit hole for me to go down :)23:10
clarkbyup that helps a lot specifically the key for the override23:10
clarkb(that doesn't seem to be documented in any of the docs I can find)23:10
mordredI just linked to the docs for it23:11
mordredsee right before the patch23:11
mordredoh - the endpoint_override?23:11
mordredthat's a standard keystoneauth setting - we should get that document in a better place - great point23:11
mordredclarkb: fwiw - this: is also possible23:12
mordredwhen we find something that's working for us, we should update too23:12
mordredok - I'm gonna afk - will follow up on this23:12
openstackgerritClark Boylan proposed opendev/system-config master: WIP override rax block storage endpoint
clarkbthat ps touches all the files it needs to I think23:14
clarkbI haven't tested this yet23:14
clarkbianw: ^ is that something you might be able to do since you've got the newer non working install?23:15
ianwclarkb:hrm, it does no, a priori, appear to work.  debugging a bit now23:19
ianwbeing careful not to paste passwords! :)23:20
*** diablo_rojo__ is now known as diablo_rojo23:21
ianw      - name: DFW23:22
ianw        values:23:22
ianw          block_storage_endpoint_override:
ianwi think this is typo free23:22
clarkb and seems to match this doc23:23
clarkbperhaps block_storage_endpoint_override isn't quite what we need?23:23
ianwInstantiating volume client: <class 'cinderclient.v2.client.Client'>23:27
ianwMaking authentication request to
ianw "POST /v2.0/tokens HTTP/1.1" 200 179023:27
ianwpublic endpoint for volumev2 service in DFW region not found23:27
ianwit doesn't seem like we're at the point of even talking to the blockstorage api?23:28
clarkbya that looks like it is still interrogating the catalog from keystone and deciding it can't do the thing23:28
clarkbpossible this is a bug in the tooling where it still checks even with an override23:29
ianwit's not clear to me how that setting translates into the config dict @
clarkbianw: there maybe?23:46
ianwthat's looking for a config section though?  more like endpoint_override: block_storage: https://... ?23:47
clarkbianw: no its doing weird concatenation of terms23:48
ianwclarkb: instrumenting it, only gets called with a key of "interface"23:53
clarkbianw: so it never calls the block-storage version of that, interesting23:53

Generated by 2.15.3 by Marius Gedminas - find it at!