| tkajinam | o/ I wonder if I can get https://review.opendev.org/c/openstack/project-config/+/910452 merged to move the retirement process forward | 05:47 |
|---|---|---|
| opendevreview | Merged openstack/project-config master: Retire puppet-sahara: End Project Gating https://review.opendev.org/c/openstack/project-config/+/910452 | 06:16 |
| tkajinam | thanks ! | 06:17 |
| frickler | tkajinam: welcome, sorry that this got missed so long | 06:19 |
| opendevreview | Birger J. Nordølum proposed openstack/diskimage-builder master: feat: add almalinux-container element https://review.opendev.org/c/openstack/diskimage-builder/+/883855 | 07:31 |
| amoralej | may i get reviews in https://review.opendev.org/c/openstack/puppet-openstack-integration/+/910576 from unmaintained-core members? | 08:35 |
| frickler | amoralej: this likely isn't the best channel for this question, but I'm also not sure which one would be. there is #openstack-unmaintained but it is not official yet and pretty quiet | 09:13 |
| amoralej | i didn't know where to ask, tbh | 09:13 |
| amoralej | i'll try there, thanks | 09:14 |
| jrosser | similarly on unmaintained patches - what needs to happen next with this? https://review.opendev.org/c/openstack/project-config/+/911576 | 09:47 |
| jrosser | i have unmaintained/yoga patches that i want to merge for OSA and forsee a bunch more needed once the next branch renaming happens | 09:48 |
| fungi | jrosser: i went ahead and approved it, seems like the openstack-unmaintained-core folks are probably still just ramping up | 13:16 |
| opendevreview | Merged openstack/project-config master: Implement openstack-ansible-unmaintained-core group https://review.opendev.org/c/openstack/project-config/+/911576 | 13:22 |
| fungi | jrosser: i'll add you as the initial member once that ^ deploys | 13:22 |
| fungi | jrosser: i've added you now | 13:29 |
| *** gthiemon1e is now known as gthiemonge | 13:36 | |
| fungi | i've gone ahead and approved https://review.opendev.org/911381 to try out the api key for rackspace dfw on nl01 | 14:01 |
| fungi | once it's deployed, i'll put nl01 into emergency disable and pull docker://insecure-ci-registry.opendev.org:5000/quay.io/zuul-ci/nodepool-launcher:e02cb1c8119b4a00bb6c4a02e13585a4_latest restarting onto that | 14:07 |
| fungi | in fact, i've pre-pulled it there just to save time later | 14:10 |
| fungi | i also need to disappear shortly before 16:00 utc for a tax prep appointment and will be grabbing lunch out after, so probably will be gone for approximately two hours (could be less) | 14:12 |
| opendevreview | Merged openstack/project-config master: Try switching Rackspace DFW to an API key https://review.opendev.org/c/openstack/project-config/+/911381 | 14:26 |
| fungi | it's deployed, nl01 is in emergency disable now and restarted onto the test container image | 14:45 |
| Clark[m] | Now we watch the grafana graphs. Thank you for getting that going. I've still got a school run to do so won't really be around for a bit yet | 14:54 |
| fungi | no worries | 14:56 |
| fungi | keystoneauth1.exceptions.auth_plugins.NoMatchingPlugin: The plugin rackspace_apikey could not be found | 14:56 |
| fungi | mmm, think we missed something | 14:57 |
| fungi | ah, no that was before the container restart | 14:57 |
| jrosser | fungi: thanks for setting up the unmaintained group, it's working for us | 14:58 |
| fungi | jrosser: my pleasure | 14:58 |
| Clark[m] | fungi: that is a good sign it is using the new clouds.yaml credentials at least. As long as it doesn't complain post restart | 15:00 |
| fungi | right | 15:05 |
| tristanC[m] | Hello folks, it seems like the AFS cache are out of sync or something, in http://mirror.ord.rax.opendev.org/centos/8-stream/AppStream/x86_64/os/repodata/ we see missing primary files listed in the repomd.xml. Is there something we can do about it? | 15:24 |
| frickler | tristanC[m]: check the mirror update log? | 15:25 |
| Clark[m] | tristanC you need to check the upstream to determine if we are out of sync or if upstream is then work from there | 15:25 |
| tristanC[m] | frickler: ok, would you know where I can check the update log? | 15:27 |
| Clark[m] | We last updated a few hours ago according to the timestamp in that mirror so either the issue was resolved very recently or upstream is out of sync or rsync broke somehow | 15:27 |
| tristanC[m] | Clark: thanks, the upstream mirror seems consistent according to http://mirror.centos.org/centos/8-stream/AppStream/x86_64/os/repodata/ | 15:29 |
| frickler | confusingly it is logged in here https://mirror.iad3.inmotion.opendev.org/logs/rsync-mirrors/centos.log and not in centos-stream.log | 15:29 |
| fungi | has to do with where in the upstream file tree they put stream 8 vs stream 9 files | 15:30 |
| fungi | i agree it's confusing, but it's confusion chosen by centos | 15:30 |
| Clark[m] | tristanC is that where we pull from? | 15:31 |
| fungi | Clark[m]: we mirror from mirror.centos.org yes | 15:31 |
| fungi | though it seems to round-robin between multiple servers, so no guarantee they all share a common file backend | 15:31 |
| opendevreview | Tristan Cacqueray proposed opendev/system-config master: Add AFS mirror update logs location https://review.opendev.org/c/opendev/system-config/+/911869 | 15:32 |
| Clark[m] | And possible they dont do updates that are rsync safe | 15:32 |
| fungi | in unrelated news, the final unmerged devstack-single-node-centos-7 nodeset removal backport shows another project using it, as predicted: https://review.opendev.org/c/openstack/devstack/+/910986 | 15:33 |
| frickler | in master, yay | 15:34 |
| fungi | frickler: well, *at least* in master, but probably in other branches too | 15:34 |
| frickler | so likely in umpteen branches, too, yes | 15:34 |
| frickler | and zuul only shows the first project it finds | 15:35 |
| frickler | or first error really | 15:35 |
| fungi | right, there are probably more in other projects as well | 15:35 |
| frickler | codesearch has only three more, not as bad as I feared https://codesearch.openstack.org/?q=devstack-single-node-centos-7&i=nope&literal=nope&files=&excludeFiles=&repos= | 15:36 |
| fungi | but again, that's in master... there are potentially more in other branches of projects that more recently removed it from their master branches | 15:36 |
| frickler | one spontaneous idea: iirc deploying codesearch is simple. can we deploy multiple instances, at least for the current maintained branches? assuming the branch that is getting indexed is configurable | 15:37 |
| amoralej | would it be possible to rerun the centos-stream-8 mirror script manually? | 15:49 |
| fungi | amoralej: we do that sometimes if there's reason to think it's warranted | 15:49 |
| fungi | i'm gone for the next two hours, but can look at it when i return if it's still a problem at that time | 15:50 |
| clarkb | tristanC[m]: amoralej can you be more specific about which files are missing so that we can check things before we do manual intervention? linking to repodata dirs isn't very specific. Maybe you can link to failed job logs? | 16:07 |
| clarkb | I do note that the upstream mirrors repodata/ contents were all updated today | 16:07 |
| clarkb | my hunch is that they do not update those directories in an atomic manner | 16:08 |
| clarkb | and if we're going to use them as an upstream we either accept that or change to an upstream that does update more atomically | 16:08 |
| amoralej | maybe, I'm asking in centos channels | 16:08 |
| amoralej | error is in https://logserver.rdoproject.org/08/08046c811a4b3d0137652eaad951cecf31b18b2d/openstack-post/rdo-send-stable-release/18def42/job-output.txt i.e. | 16:08 |
| clarkb | because we can manually intervene everytime centos does large sets of updates | 16:08 |
| clarkb | *becase we cannot manually inervene | 16:08 |
| amoralej | it's missing metadata files | 16:08 |
| clarkb | amoralej: ack then I think that supports my hunch | 16:09 |
| clarkb | amoralej: they removed old indexes before writing new indexes and now we're broken when we rsync the intermediate state | 16:09 |
| clarkb | I don't think we should manually intervene for that. Either upstream fixes their order of operations to be more atomic or we use a different upstream | 16:09 |
| amoralej | that'd be an explanation, but given that that's the repo used for global mirroring, i'd expect them to do it in the reverse order, first add, then remove, but i can't be sure | 16:10 |
| clarkb | yes I agree, but the evidence here is that we synced about 3.5 hours ago and don't have those files | 16:11 |
| clarkb | which should be fine if we continue to sync the old stuff before the new stuff is in place as long as nothing refers to the new stuff yet | 16:11 |
| clarkb | as a side note we've always been explicit that these are not public mirrors | 16:12 |
| clarkb | we can and do delete/remove content from them (we are removing centos 7 for example) | 16:12 |
| amoralej | yes, no problem with that | 16:13 |
| clarkb | the next automatic sync should begin at 18:43 UTC based on the cronjob table | 16:14 |
| amoralej | ack, we will find out | 16:14 |
| amoralej | thanks for checking | 16:14 |
| clarkb | amoralej: tristanC[m] reading the log at http://mirror.ord.rax.opendev.org/logs/rsync-mirrors/centos.log I think this supports my hunch. You'll notice that the appstream and baseos dirs delete a bunch of .xml files but do not add any .xml files | 16:17 |
| clarkb | amoralej: tristanC[m] whereas extras/ adds xml files | 16:17 |
| clarkb | so I think upstream wrote out their filse in an incorrect order. It should be add any new packge files, add any new metadata files, update indexes to point at new files, remove old files | 16:18 |
| clarkb | then anyone performing an rsync should still have a valid mirror | 16:18 |
| amoralej | yep, that's a likely explanation | 16:21 |
| amoralej | actually, some rpms were updated in the repos, just metadata was probably not updated yet | 16:22 |
| clarkb | amoralej: `grep 'AppStream/.*/.*.xml' centos.log` shows only deleting lines | 16:23 |
| amoralej | yep, i've seen it | 16:23 |
| clarkb | compared to `grep 'extras/.*/.*.xml' centos.log` which is what it should look like | 16:23 |
| amoralej | i see public mirrors have right content and pointing to updated packages so i guess we will pull it in next run | 16:24 |
| clarkb | as long as they are not consistent and fully updated then ya our next sync should catch us up | 16:24 |
| clarkb | s/they are now/ | 16:24 |
| clarkb | I cannot type today | 16:24 |
| clarkb | infra-root the switch of the rax-dfw clouds.yaml creds over the to "rackspace" cloud has jobs failing to resolve mirror.dfw.rackspace.opendev.org instead of mirror.dfw.rax.opendev.org and are failing | 16:40 |
| clarkb | this was an unexpected side effect of our rax MFA testing. I think the testing we've done so far has been fine though | 16:40 |
| clarkb | so I'm going to manually revert the change to help make jobs happy again | 16:40 |
| clarkb | I will only touch the config on nl01 (the builders and other launchers shouldn't be affected) | 16:41 |
| corvus | clarkb: i was just checking up on that change, and i also think we should not change the metrics from rax to rackspace | 16:42 |
| corvus | clarkb: i agree the POC looks good and we should probably merge the nodepool change, then update our configs to use the api key (without any other changes like cloud or metric names) | 16:42 |
| clarkb | corvus: ++ | 16:43 |
| clarkb | the manual revert is in | 16:43 |
| corvus | i'll make a system-config change to the config files | 16:44 |
| clarkb | corvus: project-config too | 16:44 |
| corvus | ++ | 16:44 |
| clarkb | this was unexpected fallout | 16:44 |
| clarkb | and makes me wonder if we should try and decouple the clouds.yaml name from logical names | 16:45 |
| clarkb | but easy enough to undo and move forward | 16:45 |
| opendevreview | James E. Blair proposed openstack/project-config master: Revert "Try switching Rackspace DFW to an API key" https://review.opendev.org/c/openstack/project-config/+/911944 | 16:46 |
| corvus | that is safe to merge now ^ | 16:46 |
| clarkb | corvus: do you know if I have to manually restart the nl01 launcher service or should we pick up the new ocnfig? | 16:47 |
| clarkb | errors that fungi saw earlier imply that we don't need to restart | 16:47 |
| corvus | clarkb: no restart necessary | 16:47 |
| clarkb | I've approved 911944 and once that merges I'll remove nl01 from the emergency file on bridge | 16:47 |
| clarkb | something like #status notice Jobs that fail due to being unable to resolve mirror.dfw.rackspace.opendev.org can be rechecked. This error was an unexpected side effect of some nodepool configuration changes which have been reverted. | 16:51 |
| clarkb | does that look good? | 16:51 |
| corvus | okay working on this system-config change, i'm unclear why all the variable names changed from openstackci_rax_x to opendevci_rax_x -- we typically named those variables based on the account name/creds, but i didn't think the account name was changing? | 16:52 |
| corvus | clarkb: status lgtm | 16:52 |
| clarkb | corvus: I think it was simply to have a second set of names that could be used without impacting existing stuff while we tested | 16:52 |
| opendevreview | Matt Peters proposed openstack/project-config master: Add Distributed Cloud App to StarlingX https://review.opendev.org/c/openstack/project-config/+/911946 | 16:52 |
| clarkb | corvus: now that testing is done I think you can remove the new names and just put the new config under the old names | 16:52 |
| corvus | i don't think that was necessary for testing | 16:53 |
| clarkb | #status notice Jobs that fail due to being unable to resolve mirror.dfw.rackspace.opendev.org can be rechecked. This error was an unexpected side effect of some nodepool configuration changes which have been reverted. | 16:53 |
| opendevstatus | clarkb: sending notice | 16:53 |
| corvus | it seems like something else was trying to be accomplished, like a migration to a different set of names | 16:53 |
| -opendevstatus- NOTICE: Jobs that fail due to being unable to resolve mirror.dfw.rackspace.opendev.org can be rechecked. This error was an unexpected side effect of some nodepool configuration changes which have been reverted. | 16:53 | |
| corvus | (sure, a second profile -- but we didn't need to duplicate all the variables) | 16:54 |
| clarkb | corvus: it is possible fungi intended that, I am not sure. But it did allow for testing of launch node for example without impacting other uses (the cloud settings cron and prod launch node for example) | 16:54 |
| corvus | to be clear, i'm talking about the secrets and variable names | 16:54 |
| clarkb | oh I see | 16:54 |
| clarkb | I am not aware of this changing account names or projects | 16:55 |
| corvus | i have to pick one, so i'm going to stick with the old names since they match the usernames | 16:55 |
| clarkb | ++ | 16:55 |
| opendevreview | Tristan Cacqueray proposed opendev/system-config master: Add AFS mirror update logs location https://review.opendev.org/c/opendev/system-config/+/911869 | 16:55 |
| clarkb | we can have fungi double check before we approve the system-config update | 16:55 |
| opendevstatus | clarkb: finished sending notice | 16:56 |
| opendevreview | Merged openstack/project-config master: Revert "Try switching Rackspace DFW to an API key" https://review.opendev.org/c/openstack/project-config/+/911944 | 17:04 |
| opendevreview | James E. Blair proposed opendev/system-config master: Switch rackspace clouds to api key auth https://review.opendev.org/c/opendev/system-config/+/911948 | 17:06 |
| corvus | okay i think that should do it. i have made the corresponding secret hostvars additions; the removals should wait until that merges | 17:06 |
| mannie | Hi sorry to disturb but am I in the right channel for outreachy contributors | 17:09 |
| clarkb | mannie: I think it depends on what you are trying to accomplish. We help run the developer tooling for projects participating in outreachy. This means we can help with gerrit account setup and understanding how to push to gerrit etc. But if you are looking for project specific info you'll need one of the irc channels dedicated to the project | 17:10 |
| clarkb | I would say ask your question and we can redirect if necessary :) | 17:10 |
| gouthamr | mannie clarkb: this channel: #openstack-outreachy is a good starting place for new contributors applying to outreachy | 17:11 |
| clarkb | infra-root I have removed nl01 from the emergency file. Once the hourly deploy jobs finish the deploy job for 911944 should noop config update on nl01 (since I already manually made that change) but may restart nl01's nodepool container to switch us off the speculative image build | 17:15 |
| clarkb | then when the nodepool image is updated we'll switch to the final version of that conatiner and finally 911948 will move all of rax over to the new config credentials | 17:15 |
| clarkb | I have also pointed this out to the openstack release team to have them double check some recent release jobs results. I know some failed and retried and were eventually successful. There is the small chance that some retried and ultimately failed and would need to rerun though | 17:16 |
| clarkb | Other than that I think we're largely in the clear again and its just a matter of rolling the rax credential updates out gracefully again | 17:17 |
| mannie | First of thank you so much I wasn't sure how quickly my quickly my first message would be responded to I really really appreciate this for the bottom of my heart. Okay so here goes how do I go about contributing to this project, yes I am also asking relating to the "gerrit" thing, I am really confused. can I just clone for the project with the regular git clone form the github repo or there's a different way of going about this? | 17:18 |
| clarkb | mannie: note gouthamr's message as there is apparently a dedicated channel for this stuff. But yes, you clone the project (either from https://opendev.org or https://github.com) then use a tool called `git-review` to push proposed changes to gerrit where they can be reviewed and once merged will end up in the repos on opendev.org and github.com. | 17:19 |
| clarkb | mannie: git is a distributed version control system which allows us to host the git repos outside of the code review system whcih helps keep load and demand off the code review system whcih makes it responive for everyone doing code reviews | 17:20 |
| mannie | So sorry for doing this on the wrong channel, I really am but I tried joining through the link on the outreachy channel via this link https://kiwiirc.com/nextclient/irc.oftc.net:6697 but i keep getting this "Closing Link: d.clients.kiwiirc.com (Banned)" after inputting a username instead of redirecting me to the channel | 17:23 |
| clarkb | mannie: how did you connect to this channel? Are you using kiwiirc here too? | 17:24 |
| mannie | I did some digging around after trying to understand why I couldn't get in with kiwiirc then I went to this url: https://meetings.opendev.org/ and some some channels i could get help from. At the moment I am using https://webchat.oftc.net/?channels=opendev# | 17:27 |
| clarkb | mannie: got it. I think kiwiirc may have been banned from oftc (not sure why) so using that tool won't work. You should be able to change the url you just pasted above to change the channel name. Currently we are in #opendev so your url would be https://webchat.oftc.net/?channels=openstack-outreachy# to get that channel | 17:29 |
| mannie | Thanks so much clarkb | 17:30 |
| clarkb | let me know if that doesn't work and we can continue to debug | 17:31 |
| mannie | Okay so the new link you provided works, but when ever I need help can I always reach you here? | 17:33 |
| clarkb | well I sleep sometimes and may be busy with other things. But yes I am happy to help with general openstack (and opendev) developer tooling questions and problems | 17:34 |
| mannie | Haha alright, thanks again. Also I noticed I am the only person on the channel i.e for outreachy contributors. I don't wanna disturb y'all except when it is absolutely necessary but is there someone I can talk to on the outreachy chennel when ever i am confused about something. | 17:40 |
| clarkb | mannie: hrm I just joined and gouthamr is there too. Let me double check if you are there | 17:41 |
| clarkb | hrm maybe that didn't end up joining the correct channel | 17:42 |
| mannie | I just sent a message to the channel | 17:42 |
| clarkb | mannie: oh the url should be https://webchat.oftc.net/?channels=openstack-outreachy with no # suffix | 17:42 |
| clarkb | I think you ended up in the #openstack-outreachy# channel (which would've been auto created for you) rather than #openstack-outreachy | 17:43 |
| gouthamr | IRC teething troubles :) | 17:53 |
| aquin | diablo_rojo: Hi I was able to push test code for review in the sandbox project. I cc'd you as a reviewer. | 18:02 |
| opendevreview | Goutham Pacha Ravi proposed openstack/project-config master: Add op to #openstack-outreachy https://review.opendev.org/c/openstack/project-config/+/911954 | 18:02 |
| fungi | okay, back and catching up. took a little longer than expected | 18:18 |
| fungi | oof, cloud name changes alter the mirror name settings. i briefly thought about that and then forgot to ask. sorry! | 18:19 |
| fungi | this is also part of why i was originally leaning toward changing the auth for the existing providers rather than making new providers in parallel | 18:21 |
| fungi | for the record, it wasn't "for testing" but "for transition" since i got requests to have two clouds defined in parallel that we could switch back and forth between easily | 18:22 |
| fungi | i was trying to make sure that we didn't miss values for the new provider/cloud or accidentally use the old password when we meant to use the new api key | 18:23 |
| fungi | i'm happy to roll all of the recent changes back and redo them by just changing the auth settings for the existing clouds/providers, since that was my original plan | 18:23 |
| fungi | infra-root: ^ what would you prefer? | 18:24 |
| clarkb | fungi: I think corvus' change above already does that | 18:28 |
| clarkb | note we didn't create new providers | 18:28 |
| fungi | cool, i'll take a look at those | 18:28 |
| clarkb | we chagned the cloud name in existing providers and taht was sufficient to make mirror stuff unhappy | 18:28 |
| fungi | fwiw, i think it's cleaner to update the authentication for our existing provider definitions | 18:28 |
| fungi | rather than add new ones | 18:29 |
| clarkb | right thats what we've done the whole time | 18:29 |
| clarkb | oh maybe you mean cloud.yaml entries | 18:29 |
| fungi | most of the complexity in the recent changes was to satisfy the request to have new providers in parallel with the original ones | 18:29 |
| clarkb | ya I wanted to avoid taht for testing because we can't limit updated cloud entries for a single region | 18:29 |
| clarkb | but now that we've tested and it generally works I think this is ok (and I +2'd corvus' change) | 18:29 |
| clarkb | fungi: just for clarity we never wanted new providers | 18:30 |
| clarkb | fungi: an initial patchset did add a new provider but that was changed because we didn't want that | 18:30 |
| fungi | oh, i clearly misunderstood | 18:30 |
| clarkb | because adding a new nodepool provider would've orphaned the existing stuff in the old provider | 18:30 |
| fungi | i was personally satisfied when manual tests with the auth settings change worked, but was willing to add new clouds instead since others wanted that | 18:31 |
| fungi | yeah, new clouds, new providers, it wasn't clear to me that those were distinct choices | 18:31 |
| clarkb | ya I was hoping to test existing nodepool providers with new credentials and doing so required new cloud entries unless you wanted to update all of nodepool for rax at once | 18:31 |
| clarkb | and since rax is like 50% of our capacity I was hoping to avoid that for testing | 18:32 |
| clarkb | turns out it would've been better to just send it due to this unexpected behavior | 18:32 |
| fungi | i assumed we needed separate providers to use different cloud configurations in parallel | 18:33 |
| clarkb | in any case adding new providers would've broken in the same way (beacuse there is no mirror with that name). But we didn't do that | 18:33 |
| fungi | got it | 18:34 |
| clarkb | the next hourly run of the nodepool job should update all of our images to the new image with rackspaceauth installed then we can land the system-config update which swaps rax over to using rackspaceauth keys | 18:36 |
| fungi | i'll un-wip it. thanks! | 18:36 |
| clarkb | then we should be done for both control plane and nodepool and can opt into MFA early if we like | 18:36 |
| clarkb | fungi: https://review.opendev.org/c/opendev/system-config/+/911948 is the change in system-config I'm referring to | 18:37 |
| fungi | oh, though i change the cloud names and stuff | 18:37 |
| fungi | it'll need a reword | 18:37 |
| fungi | rework | 18:37 |
| clarkb | it isn't wip should be good to review (we just need to land it after the nodepool update occurs) | 18:37 |
| fungi | ah, okay, i was talking about 911229 | 18:37 |
| fungi | which, yes, was complicated by the requirement to add new clouds in parallel with the existing ones | 18:37 |
| clarkb | I think the confusion was that I only ever intended for new clouds.yaml entries to be temporary to test things | 18:38 |
| fungi | i can abandon 911229 if we don't need it now | 18:38 |
| clarkb | ya I don't think we need 911229 if we land 911948 | 18:38 |
| fungi | yeah, i assumed it was a desire to replace the old definitions with new cloud entries and then later delete the old ones once everything was moved over | 18:38 |
| clarkb | and the reason we needed new clouds.yaml entries for testing is that each cloud entry applies to all regions all at once | 18:38 |
| clarkb | but once testing was done we'd remove the new stuff and apply the config update to what prod was using normally | 18:39 |
| clarkb | which is what we've ended up with here | 18:39 |
| fungi | got it. i didn't realize it was a test-and-roll-back suggestion, though it was add-parallel-replacements-and-clean-up-originals | 18:39 |
| clarkb | if we had cloud entries for each region already then none of this would've been necesasry. But we don't (and I think that is correct for how we're handling mirror stuff in nodepool and zuul) | 18:40 |
| clarkb | otherwise we'd have mirrors like mirror.dfw.rax-dfw.opendev.org without changing stuff up there | 18:40 |
| fungi | right, i had a fleeting thought that this was going to require new dns entries, and should have remembered to follow up on that | 18:41 |
| clarkb | I completely spaced on that | 18:41 |
| clarkb | might be worth a note in our nodepool configs? | 18:41 |
| clarkb | similarly new credentials allowed for testing launchnode without impacting say tonyb trying to launch new meetpad server stuff | 18:43 |
| clarkb | anyway testing looks good other than the mirror mismatch and we can roll forward now pretty comfortably in our existing configs | 18:43 |
| clarkb | one "good" side effect of this is jobs failed quickly so we booted and deleted nodes quickly in dfw :) pretty good indication that this works well | 18:44 |
| fungi | yep, i think we're in a good place with it | 18:48 |
| fungi | so are there outstanding changes i should reference when abandoning 911229, something that isn't using topic:rackspace-mfa maybe? | 18:54 |
| clarkb | fungi: just the one from corvus | 19:04 |
| clarkb | https://review.opendev.org/c/opendev/system-config/+/911948 | 19:04 |
| clarkb | the hourly nodepool job is running now so we should get new container images deployed then can land 911948 after | 19:05 |
| clarkb | nl01's container just restarted | 19:07 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: Clean up unused Rackspace password test values https://review.opendev.org/c/opendev/system-config/+/911958 | 19:12 |
| clarkb | the nodepool job is done now and was successful. All nodepool containers should be running images capaable of using the new auth type | 19:13 |
| clarkb | we should be good to land 911948 if we are happy with that change and the private var updates | 19:14 |
| clarkb | fungi: corvus: I looked at the private vars and they look good. I think my only other fear is that if we somehow mixed up api keys and had test nodes booting in the control plane account that nodepool might try to delete those nodes. But it shouldn't because we don't let nodepool delete things without nodepool metadata | 19:19 |
| clarkb | I would expect auth to just not work beacuse usernames shouldn't match up in that case too | 19:19 |
| clarkb | all that to say I think we're good to approve 911948 now? | 19:19 |
| fungi | yep, agreed. just approved it now | 19:21 |
| clarkb | I guess after that lands we should do some openstack server lists and make sure everything looks aligned properly | 19:22 |
| opendevreview | Clark Boylan proposed openstack/project-config master: Add warning to nodepool configs about changing cloud name https://review.opendev.org/c/openstack/project-config/+/911959 | 19:29 |
| clarkb | tristanC[m]: the 18:43 sync appears to have caught things up for centos 8 | 19:47 |
| clarkb | the system-config change to update the clouds.yaml should be merging shortly. Not sure which side of the hourly deploy jobs it will end up on though | 19:50 |
| tristanC[m] | clarkb: that's great, thank you very much for the follow-up! | 19:52 |
| opendevreview | Merged opendev/system-config master: Switch rackspace clouds to api key auth https://review.opendev.org/c/opendev/system-config/+/911948 | 19:59 |
| clarkb | deploy for ^ ended up being hourly jobs but I think it may have merged soon enough that hourly jobs will use it anyway? | 20:05 |
| clarkb | yes bridge just updated | 20:05 |
| clarkb | the nodepool job is starting so nodepool should update soon too | 20:07 |
| clarkb | server listing against ord is extremely slow | 20:07 |
| clarkb | IAD isn't what I'd call quick but it is faster. And listings for both iad accounts work and I see appropriate values in both (though nodepool hasn't updated its clouds.yaml yet) | 20:10 |
| clarkb | this looks good to me. I'm not sure if we have to restart nodepool to pick up the change since only clouds.yaml updated and not the provider config in nodepool.yaml. If so then the change to add more build and upload stats to nodepool should cause a container restart across the board | 20:12 |
| clarkb | fungi: corvus: please do checking from your end if you feel it is worthwhile. I can also restart containers if we want that to happen more quickly. But in the meantime I need ot eat lunch | 20:13 |
| clarkb | and now we can start thinking about the swift credentials. Then figuring out opting into MFA early | 20:13 |
| fungi | yeah, i think we can wait for another change to restart nodepool onto the new image and read in the clouds.yaml when that happens | 20:23 |
| clarkb | nl01 just restarted due to that nodepool change merging a little while ago | 21:07 |
| clarkb | so ya no reason to do that manually and we just need to watch grafan for rax oddities | 21:08 |
| clarkb | and start figuring out the swift credentials | 21:08 |
| clarkb | we encrypt the username and project id info as well as the password | 21:09 |
| clarkb | and grabbing the secret keys out of zk is difficult now because they are encrypted? THis could be a fun puzzle to sort through | 21:10 |
| clarkb | though maybe https://review.opendev.org/c/zuul/zuul/+/908507/6/tools/decrypt_secret.py addresses those problems for us | 21:31 |
| clarkb | corvus: ^ I reviewed that chaneg it seems to be fine more the most part but would be interested to see if you are willing to review it before we run it against our secrets | 22:53 |
| corvus | clarkb: reviewed | 23:07 |
| clarkb | thanks looks like no major concerns. maybe we see if we get a new iteration overnight (the timestamps imply maybe an EU submitter) then give ti a go to see what we've got in place for these swift creds | 23:09 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!