Tuesday, 2022-05-10

opendevreviewIan Wienand proposed opendev/infra-vhd-util-deb focal: Initial import of debian build files  https://review.opendev.org/c/opendev/infra-vhd-util-deb/+/84117500:02
opendevreviewIan Wienand proposed opendev/infra-vhd-util-deb focal: [wip] test vhd-util build  https://review.opendev.org/c/opendev/infra-vhd-util-deb/+/84117900:02
ianwclarkb: https://review.opendev.org/c/opendev/infra-vhd-util-deb/+/841174 change should make it clearer00:03
ianwchange description i mean00:04
opendevreviewIan Wienand proposed opendev/infra-vhd-util-deb focal: [wip] test vhd-util build  https://review.opendev.org/c/opendev/infra-vhd-util-deb/+/84117900:05
*** rlandy|bbl is now known as rlandy|out01:22
opendevreviewIan Wienand proposed opendev/infra-openafs-deb jammy: Test generic job path  https://review.opendev.org/c/opendev/infra-openafs-deb/+/84118901:53
opendevreviewIan Wienand proposed openstack/project-config master: infra-vhd-util-deb : add jobs  https://review.opendev.org/c/openstack/project-config/+/84119002:01
opendevreviewIan Wienand proposed opendev/infra-openafs-deb jammy: Test generic job path  https://review.opendev.org/c/opendev/infra-openafs-deb/+/84118902:06
opendevreviewMerged opendev/infra-openafs-deb jammy: Test generic job path  https://review.opendev.org/c/opendev/infra-openafs-deb/+/84118903:15
*** soniya29 is now known as soniya29|ruck04:38
*** ysandeep|out is now known as ysandeep|rover04:42
opendevreviewIan Wienand proposed openstack/project-config master: infra-vhd-util-deb : add jobs  https://review.opendev.org/c/openstack/project-config/+/84119004:42
opendevreviewMerged openstack/project-config master: infra-vhd-util-deb : add jobs  https://review.opendev.org/c/openstack/project-config/+/84119004:59
opendevreviewIan Wienand proposed opendev/infra-vhd-util-deb focal: Initial import  https://review.opendev.org/c/opendev/infra-vhd-util-deb/+/84117405:04
opendevreviewMerged opendev/infra-vhd-util-deb focal: Initial import  https://review.opendev.org/c/opendev/infra-vhd-util-deb/+/84117405:16
ianwthat ~ working -> https://launchpad.net/~openstack-ci-core/+archive/ubuntu/vhd-util-deb-test05:21
*** pojadhav|afk is now known as pojadhav05:23
opendevreviewMerged opendev/system-config master: Use add instead of wget to download things for gerrit images  https://review.opendev.org/c/opendev/system-config/+/71508706:14
opendevreviewIan Wienand proposed opendev/infra-vhd-util-deb focal: Apply Citrix patch for conversion in vhd-util  https://review.opendev.org/c/opendev/infra-vhd-util-deb/+/84120906:46
opendevreviewIan Wienand proposed opendev/infra-vhd-util-deb focal: Import changes to build on Ubuntu Bionic  https://review.opendev.org/c/opendev/infra-vhd-util-deb/+/84121006:53
opendevreviewIan Wienand proposed opendev/infra-vhd-util-deb focal: Apply changes to build on Ubuntu Focal  https://review.opendev.org/c/opendev/infra-vhd-util-deb/+/84121106:53
opendevreviewIan Wienand proposed opendev/infra-vhd-util-deb focal: Import changes to build on Ubuntu Focal  https://review.opendev.org/c/opendev/infra-vhd-util-deb/+/84121106:54
opendevreviewIan Wienand proposed opendev/infra-vhd-util-deb focal: Update Debian version number  https://review.opendev.org/c/opendev/infra-vhd-util-deb/+/84121207:00
opendevreviewMerged opendev/infra-vhd-util-deb focal: Apply Citrix patch for conversion in vhd-util  https://review.opendev.org/c/opendev/infra-vhd-util-deb/+/84120907:06
opendevreviewMerged opendev/infra-vhd-util-deb focal: Import changes to build on Ubuntu Bionic  https://review.opendev.org/c/opendev/infra-vhd-util-deb/+/84121007:07
opendevreviewMerged opendev/infra-vhd-util-deb focal: Import changes to build on Ubuntu Focal  https://review.opendev.org/c/opendev/infra-vhd-util-deb/+/84121107:08
opendevreviewMerged opendev/infra-vhd-util-deb focal: Update Debian version number  https://review.opendev.org/c/opendev/infra-vhd-util-deb/+/84121207:14
*** jpena|off is now known as jpena07:30
*** soniya29|ruck is now known as soniya29|ruck|lunch08:09
*** ykarel is now known as ykarel|away08:25
*** ysandeep|rover is now known as ysandeep|rover|lunch08:49
*** soniya29|ruck|lunch is now known as soniya29|ruck09:21
*** bhagyashris is now known as bhagyashris|out09:33
*** ysandeep|rover|lunch is now known as ysandeep|rover09:47
opendevreviewA R proposed openstack/diskimage-builder master: Preserve local mirrors when using Ubuntu element  https://review.opendev.org/c/openstack/diskimage-builder/+/84124710:09
*** rlandy|out is now known as rlandy10:33
*** dviroel|afk is now known as dviroel11:28
ecsantos[m]Hello folks11:44
ecsantos[m]My team (NetApp) maintaints a third-party CI for the Cinder and Manila projects11:44
ecsantos[m]The Gerrit account we use to report job results was registered a long time ago (back in 2014 I reckon), and we can no longer access it11:45
ecsantos[m]When trying to sign in, we get a "Not Found" page with the https://review.opendev.org/SignInFailure,SIGN_IN,Contact+site+administrator URL11:45
ecsantos[m]Where can we get some help on this?11:45
fungiecsantos[m]: what's the username?11:48
ecsantos[m]fungi: I only have the display name, NetApp CI11:49
fungithe ci system has to specify its username when it authenticates to gerrit though11:49
fungido you not have access to the ci system itself?11:50
*** soniya29|ruck is now known as soniya29|ruck|break11:58
fricklerthere is an account with that name in the third-party CI group, but better double-check with how they try to log in, I agree12:11
rosmaitaaccording to this: https://wiki.openstack.org/wiki/ThirdPartySystems/NetApp_CI , the gerrit username is netapp-ci12:17
fungiecsantos[m]: if the account is "very old" it would have been manually created by our gerrit sysadmins with no ability to log into the webui. the recommendation for updating those accounts is to create a new account and ask to have the old one marked inactive12:22
*** soniya29|ruck|break is now known as soniya29|ruck12:27
ecsantos[m]fungi: Unfortunately we only have that e-mail, I can even access Launchpad with it. Would we really need a new e-mail, or would just marking the account as inactive and trying to sign in again work?12:31
fungiecsantos[m]: i think if i take that e-mail address off the old account, then you'll be able to create a new gerrit account with the launchpad login you already have12:32
ecsantos[m]fungi: That'd be okay with us12:33
fungilooks like username:netapp-ci has accountId = 1062112:35
fungiecsantos[m]: the xdl-openstack-jenkins@ address is the one you're using with it currently?12:38
fungijust want to make sure i'm looking at the right account12:38
ecsantos[m]fungi: It's ng-openstack-ci@netapp.com12:38
ecsantos[m]At least that's the e-mail it shows when we click on the Gerrit account12:39
fungiaha, that's username:NetApp-ci with accountId = 2524312:40
fungiunfortunately, gerrit usernames in our deployment are historically case-sensitive, so username:netapp-ci and username:NetApp-ci are separate accounts12:40
fungianyway, that account has an ubuntu login associated with it12:41
fungiso it's not an older account created by one of our admins12:41
fungiecsantos[m]: and you say trying to authenticate from login.ubuntu.com is failing to sign in? i'll check the logs to see if there are any associated errors12:42
fungiecsantos[m]: oh, fun, it's apparently created a new ubuntu login and that's trying to create a new account, but gerrit is refusing to let account creation complete because the same e-mail address is in use by existing account 2524312:44
fungidid you change anything you can think of in launchpad/ubuntuone which would result in a different openid?12:44
ecsantos[m]yeah I just tried loggin in12:44
ecsantos[m]actually no one in our team have tried to log in into that account in recent memory12:45
ecsantos[m]so we don't know :/12:45
fungiokay, something has definitely resulted in the openid for it changing vs the one gerrit knew. i'll see if i can add the new openid to the existing account12:46
fungioh, though i may not be able to push this through gerrit. before i go screwing something up, i want to double-check with clarkb and he's probably not awake yet12:47
ecsantos[m]last login to review.openstack.org on our side was on 2015/06/1512:48
ecsantos[m]fungi: no problem, it's not that urgent12:48
fungiecsantos[m]: yeah, we should be able to update this so that the lp account you're using logs into the gerrit account you have, it'll just take a little surgery on the account records and i want to be sure i do it right in notedb (it used to be a lot more straightforward when all this was an sql db)12:51
ecsantos[m]fungi: Thanks for the help! If you need anything please let me know12:56
fungiwill do, and i'll let you know once we're ready for you to test logging in again13:01
opendevreviewA R proposed openstack/diskimage-builder master: Preserve local mirrors when using Ubuntu element  https://review.opendev.org/c/openstack/diskimage-builder/+/84124713:07
Clark[m]fungi: correct you cannot do that push until all the existing errors are corrected in the DB unless Gerrit 3.4 changed that behavior. I sort of fizzled out on fixing the last 35 or so errors as it is quite time consuming and there is always something else to do. You can try it though and Gerrit should reject it.13:12
Clark[m]What we can do is retire the old account and remove the openids and email addresses from it then let the new openid create a new account using those attributes13:14
fungiClark[m]: so use the rest api to remove the e-mail address and username from the old account and set it inactive?13:20
Clark[m]I typically set it inactive and remove the preferred email address using a git push to the accounts specific ref. Then when that is done use the rest API to remove the external ids. The scripts I pushed into system-config for DB cleanups can do that. One does the "retirement" which sets inactive and removes the preferred email setting. The other deletes external ids13:24
Clark[m]Before that happens may be good to double check with ecsantos that they don't use the account with ssh anywhere13:25
Clark[m]I suspect they are actively reporting via ssh. And this surgery will temporarily break that until they update ssh keys in Gerrit and ssh usernames in their CI system13:26
fungihttps://review.opendev.org/Documentation/rest-api-accounts.html#delete-account-external-ids seems like it can delete arbitrary external-ids, i guess that's a workaround for not having a specific method to delete usernames?13:27
Clark[m]Ya usernames are just external ids.13:28
ecsantos[m]Clark: That's correct, it'll temporarily break job reporting for us but that's okay, SSH is used by our old Zuul v2 CI, we're migrating to v3 and as soon as we can create the new account I'll properly set up SSH keys for it13:29
Clark[m]ecsantos you'll need to configure a new username on your clients too. At least historically we've not allowed those to migrate between accounts (there is a post somewhere by Gerrit upstream indicating why doing so is a bad idea iirc)13:30
Clark[m]The other alternative is to shutdown Gerrit. Push the openid update to the existing account behind gerrit's back then offline reindex and start Gerrit again13:31
ecsantos[m]Clark: Whatever is easier for you guys works for us13:43
fungiClark[m]: so when using the retire-user.sh script, is the assumption that it will push through the gerrit ssh remote or locally to the bare repo on the backing filesystem?13:43
fungimy admin account seems to not have push rights13:49
fungito the user ref13:49
fungiwere you temporarily adding your account to a different group? project bootstrappers maybe?13:50
fungiyeah that worked13:51
fungiso refs/users/43/25243 is now retired and i should be able to clean up the external-ids13:52
Clark[m]fungi: ya it pushes to Gerrit13:52
Clark[m]The idea with those scripts is everything is done through Gerrit to avoid races and to avoid needing reindexing13:53
Clark[m]And ya to do that you need to be a bootstrapper13:53
Clark[m]You only need to be an admin to run the rest API commands13:54
Clark[m]Updating the user ref isn't a big race concern if the user can't login anyway. But all the external ids are under a shared ref and updates to those can race. This is why we have to stop Gerrit if changing them behind gerrit's back14:00
fungiyep, so the external-ids referencing that e-mail address have been removed now. on a whim i tried deleting the external-id for the username and gerrit actually refuses14:09
fungireturns a 409 error14:10
fungiecsantos[m]: okay, i think you're all set to try logging in again when you have a moment. this will create a new account with your new openid, but should be able to reuse the old e-mail address. as Clark[m] mentioned, you'll need to set a new username in gerrit (can't reuse the old username), as well as reuploading ssh keys14:11
ecsantos[m]fungi: Okay, will try logging in14:12
Clark[m]Good to know Gerrit doesn't allow the username removal. I know it causes problems if you do it and the service is protecting itself. That is good14:13
ecsantos[m]fungi Clark: It worked! Big thanks for the help folks14:15
fungihooray! thanks for confirming14:15
ecsantos[m]Just trying to think of a new username now :p14:15
fungialso good luck on your zuul v3 (hopefully really v6) deployment!14:15
ecsantos[m]Thanks! We're on v4.9 hoping to go v5 soon :) (looking forward to use community.general on v6)14:17
fungiyeah, the jump from 5 to 6 isn't a significant amount of upgrade effort, 6.0.0 mainly just signals the removal of the old ansible prohibitions so it will be able to use newer ansible versions easily14:18
fungithe upgrade past 4.9 does require some care in handling the migtation of project keys into zookeeper14:19
fungier, migration14:19
fungibut beyond that it's pretty straightforward (release notes have upgrade guidance)14:20
ecsantos[m]We're using Software Factory actually, so hopefully their upgrade playbooks will take care of that for us :)14:22
rpittauhi all! A little doubt, is it possible to override a branch in a zuul template? or do we need to create a new job based on that and override the branch there?14:25
clarkbfungi: it looks like ssl certs are still not being refreshed14:28
clarkbI need to find breakfast, but I'll probably start my day by looking at that if it isn't already solved14:28
clarkbrpittau: project-templates don't have branch settings. That is a job thing. Generally you should define the job you want to run on a branch in that branch. Then the implicit lookups will find it properly. If your job is defined in a config project or branchless project like tempest you can make a variant of that job that applies to specific branches. But do not use the branches:14:29
clarkbkeyword in jobs defined on projects with branches it will just lead to problems later when you branch that project again14:29
rpittauclarkb: the problem is that we have the template in a bugfix branch and it takes upper-constraints from master :/14:30
clarkbrpittau: I think that is usually appropriate since bug fixes tend to target master then get backported?14:31
rpittauI guess the implicit lookup takes into consideration only stable branches14:31
clarkbthat is generally how things are configured the expectation being if there aren't explicit rules for something then treat it like master for this reason14:31
rpittauclarkb: not in our case, we have bugfix branches based on stable branches14:31
clarkbno the implicit lookup applies to all branches, but if it doesn't find your branch it falls back to master14:32
rpittauok, so I guess we need to define a job and override the branch14:32
clarkbyou can add a pragma that defines mappings. Devstack does this iirc14:32
clarkbrpittau: https://opendev.org/openstack/devstack/src/branch/master/.zuul.yaml#L1-L714:32
clarkbthat says feature/r1 is equivalent to master I think14:33
rpittauclarkb: mmmm ok, interesting14:33
rpittauclarkb: thanks!14:33
clarkbrpittau: I'm curious why you would need bug fix branches targetting stable branches if the idea is to always backport fixes? I guess you are fixing bugs in stable branches but not master?14:34
clarkbso the backport begins behind master14:34
rpittauclarkb: because we have intermediate releases14:34
rpittauthat are based on those bugfix branches14:34
rpittaubecause we like complicate things :D14:35
clarkbI'm not sure I would be doing releases off of bugfix branches14:35
clarkbseems like a really good way to miss fixes14:35
clarkbfor newer releases I mean14:35
clarkbif you do a bugfix branch, merge to master, branch master off for stable intermediate release, then release you're ensuring that future releases don't miss your fixes14:36
rpittauthe bugfix branches have usually a shorter life than stable branches14:36
clarkbthat sort of process would be more in line with the general expectation of how many of openstack's jobs are designed14:36
rpittaubut in that case we also get master features14:36
clarkbright but master doesn't necessarily get your bugfix14:37
clarkb(which is the concern I'm trying to highlight)14:37
rpittaummmm no, we fix on master, then we backport to all supported stable and bugfix branches14:38
rpittaubut we cherry-pick only the fix14:38
clarkbok I may not be understanding what you mean by bugfix branch. But if you backport to it and make releases from it I'm not sure I would characterize that as a bugfix branch14:38
clarkbin any case your have bugfix/foo and if you need to map that to stable/bar instead of master I think you can use the pragmas like devstack.14:40
rpittauI probably didn't explain myself properly, we do only one host release from bugfix branches, once it's done, it's done14:41
rpittauall the subsequent fixes backport to the bugfix branch are not released, but they're kept in that branch14:41
rpittauone shot*14:41
rpittauclarkb: I'll give the pragma thing a try, thanks!14:41
fungiyeah, it's not that implicit lookups only take stable branches into consideration, it's that zuul assumes you'll have the same branch name in every project involved in a job, and if that branch is missing from some project it falls back to using the default branch for that project (typically master)14:45
clarkbright. Implicit lookups look for consistent naming across the system. The way openstack is set up is to have master and a series of stable branches. The stable branches will find each other and use consistent config. Master finds master and is consistent. If you use a bugfix/foo or feature/foo the assumption by default is that those fixes or those features apply to master and you'll14:47
clarkbget master configs. Then when they land you can backport to stable branches as necessary14:47
fungithere are ways (for example that pragma) to map branches with different names to each other between projects, but it gets complicated and not really how project integration was designed14:47
clarkbI think if this were me I'd branch stable/foo early to do the intermediate release. It will use master things until the rest of openstack makes stable/foo at which time it will become equivalent to stable/foo14:50
clarkbthen you do your stable/foo release off of that branch later with the newer commits14:50
clarkbrpittau: ^ fyi that would be the way to do it while integrating with openstack I think14:51
clarkbwhether or not that would break other things I do not know.14:51
*** dviroel is now known as dviroel|lunch|afk14:53
rpittauclarkb, fungi: thanks again, "our" bugfix branches usually don't last very long, definitely less than a stable branch, I think the pragma option could work fine for our needs14:54
clarkbright its less about timeperiods and more about representing the intent with the existing machinery14:55
clarkbpragma does that by being explicit about weird rules. The suggestion above tries to do it by operating with the existing rules14:55
clarkbbtu I'm sure it would need a bit more thought to determine if it would create other problems first14:56
*** soniya29|ruck is now known as soniya29|out15:20
clarkbacme.sh is failing because it sees and existing domain key and for some reason thinks it isn't renewing but issuring a new cert. This fails beacuse it wants you to pass --force in this instance.15:49
clarkbThing is we are renewing existing certs so I dunno why it is broken, but looking at recent changes to acme.sh they broke a bunch of renewal stuff and are fixing it so possibly related to that? However I have no clue if those fixes are sufficient to start over or if we need to more forcefully do that15:50
clarkboh we actually do the two pass thing of issuing then renewing15:52
clarkbI think it is the issue command that is bailing out15:52
fungiokay, so your first impression is some recent change to acme.sh has broken us?15:54
fungilike in the past week?15:54
clarkbor month but ya15:54
fungiwell, we didn't start getting warnings until the weekend, and that seemed to be in part becasue we stopped running the base deploy on friday15:54
fungiso i have a feeling it was in the past week, maybe even just in the past day or two15:55
clarkbthe commit that is suspicious is "fix renew server" of course there isn't more info than that15:56
fungiof course! what's the point of commit messages when you have the diff? ;)15:59
fricklerpossibly we didn't try to renew for some time before that if all certs were >30d valid?16:01
clarkbwe do an `acme.sh --server letsencrypt --cert-home /etc/letsencrypt-certs --no-color --yes-I-know-dns-manual-mode-enough-go-ahead-please --issue --dns --challenge-alias acme.opendev.org` this logs "Creating domain key" "Domain key exists, do you want to overwrite the key?" "Add '--force', and try again." "Create domain key error."16:01
clarkblooking at the source for acme.sh there is a $_ACME_IS_RENEW flag that is meant to avoid that error when renewing16:02
clarkbwhat --force will do is create a new domain key (which is probably ok? I'm not sure yet) which would be a change of behavior for us16:03
fungiwe renew a large enough number of certs that i expect the distribution of renewal dates is sufficiently distributed over time for us to notice problems like this within a few days of arising, usually16:06
clarkbthe call for createDomainKey is guarded with if [ ! -f "$CERT_KEY_PATH" ] || [ "$_key_length" != "$_key" ] || [ "$Le_ForceNewDomainKey" = "1" ]; then so maybe one of those other flags has changed16:07
fungiclarkb: are you saying we were previously skipping the domain key replacement?16:07
clarkbfungi: pretty sure we were16:07
clarkbsince the code that generates the domain key hasn't changed in a long time16:08
clarkbI'm suspecting that the changes are to when/how it is called rather than the actual routine16:08
fungii see, so the "already exists" condition was just causing it to move on to the next step up to now16:08
clarkb7f9074adbf2f2aeba61db36a3233730c4768c033 is also suspicious as it deals with key lengths which are checked in that test16:10
clarkbfungi: that is my hunch16:10
clarkbfungi: eitehr we never called createDomainKey because the guard above skipped it or we called it and then it saw it was renewing and ignored it16:10
clarkbya ok I think it is the keylength16:12
fungioho, so they switched to a longer key?16:13
clarkbthe zuul.opendev.org.conf file lists Le_Keylength=''16:13
clarkband that value gets stuffed in $_key in that condition above16:13
clarkbthen $_key_length appears to be 204816:13
clarkbI think it was always 2048 they just changed how they check it and broke us?16:13
clarkbthe commit sha above 7f907 is what I suspect broke things16:13
fungioh, interesting, so we're trying to tell it to use its default length there, and it sees that as a mismatch now16:14
clarkboh sorry its that commit and 64847afc3ff8cfe214aca7db7f793d96bee95e5e together16:14
clarkbfungi: ya I think so16:14
clarkbya I think I see the fix16:15
clarkbin review() which we call after issue() it checks if Le_Keylength (which gets stuffed in $_key in issue()) is empty and is so sets it to 204816:15
clarkbbut issue() doesn't seem to do that check and it fails16:15
clarkbanother option would be to manually edit the files and set the value to 204816:16
clarkband then I think it will just proceed from there properly16:16
*** marios is now known as marios|out16:17
opendevreviewdasm proposed opendev/elastic-recheck rdo: Fix failing tests  https://review.opendev.org/c/opendev/elastic-recheck/+/84130616:18
clarkbLet me try to collect this together more concretely so that others can weigh in16:18
opendevreviewdasm proposed opendev/elastic-recheck rdo: Enable podman-compose  https://review.opendev.org/c/opendev/elastic-recheck/+/84117216:19
opendevreviewdasm proposed opendev/elastic-recheck rdo: Updating gitignore to ignore local config files  https://review.opendev.org/c/opendev/elastic-recheck/+/82428316:19
clarkbhttps://github.com/acmesh-official/acme.sh/blob/dev/acme.sh#L1551-L1553 is where we are breaking after calling issue() per the logs on zuul01.16:23
clarkbhttps://github.com/acmesh-official/acme.sh/blob/dev/acme.sh#L4411-L4416 is where issue() calls createDomainKey. It only does this if the key doesn't already exist or you change the key length or if you pass --force16:23
*** ysandeep|rover is now known as ysandeep|out16:23
clarkbhttps://github.com/acmesh-official/acme.sh/commit/64847afc3ff8cfe214aca7db7f793d96bee95e5e updated the way key lengths are read in and handled and I'm 99% sure that the check is now comparing 2048 against "" and decided they are different which causes the createDomainKey function to be called16:24
clarkbWe can fix that by checking if $_key is "" and setting it to 2048 first. 64847 does this in the renew() path already but not the issue path. Or we can modify our key conf files to set the keylength explicitly to 2048 out of band and let it continue16:25
fungiwe could do the latter and propose the former upstream too16:25
fungiand then revert the explicit 2048 once upstream is fixed16:26
clarkbwe wouldn't want to rever that as it will write 2048 back automatically for us16:26
clarkbso after the first successful pass this should work going forward. For that reason I actually think it may be useful to just fix this upstream?16:27
clarkbalso we can probably wait for ianw to weigh in as ianw has worked upstream with them in the past and may have input on the best way to approach this16:27
fungii'm probably misunderstanding. the current problem is that we set the keylength to an empty string in our config, and that confuses acme.sh into forcing replacement even though the existing key is the default length already. if we set the config explicitly to 2048 instead of an empty string, then that would work around it temporarily. if we fixed it upstream to know not to force replacement of16:33
fungithe key, then we'd be able to safely revert back to setting the empty string in our config, right?16:33
clarkbif we set it to 2048 that would fix it permanently unless they change the code again. It would be a permanent fix because acme.sh will read 2048 and compare it against our requested size of 2048 and be happy. That will skip the first pass of createDomainKey(). After that it will write 2048 back to the conf file for us16:35
clarkbif we revert back to the empty string it will fail on the next pass16:35
clarkbbecause it expects 2048 there (this is why it writes it back again)16:35
fungier... acme.sh writes to its own configuration file? we don't write out the config for it?16:36
fungimaybe that's where i'm getting confused. i expected the configuration to be immutable from the application's perspective16:36
clarkbcorrect this file is tied to the key itself and acme.sh writes it out. We don't touch it16:37
fungibut sounds like you're saying the "configuration" is more of a state cache16:37
clarkbya in a way I guess it is16:37
clarkbacme.sh generates the key and writes a config for that key alongside it16:37
clarkbso that when it runs the next time it knows some info about the key without executing openssl against it I guess16:38
fungiokay, so i completely misunderstood the suggestion, sorry. so acme.sh was writing out an empty string for the key length field in its state metadata, essentially, but them misinterpreting it as something other than an indication it had previously used its default value16:38
funginot a problem with any configuration we were supplying16:39
clarkbcorrect and 64847af is the commit that changed their internal assumptions recently16:39
clarkbanother option is to force it to generate new keys once by setting --force16:40
clarkbI think I'm coming around to fixing this upstream though since it will adjust the config appropriately for us and there are probably other users out there with this problem16:40
fungiso to reset... your suggestion was that we could manually fix up the state information previously recorded to have an explicit key length, in order to get it back on track16:40
clarkbthough maybe we are the only ones doing the dns issuance which causes both issue and renew to be called16:40
clarkbfungi: correct. And we'd do that manually since that isn't tracked in our config management and probably shouldn't be16:41
fungigot it. i was confused by the use of the term "configuration" for this. it just happens to (ab)use openssl's config format for recording additional data about the key it generates16:42
fungibut it's not configuration from our perspective, it's just data the application is maintaining16:43
clarkbya its configuration for the key managed by acme.sh16:43
clarkbit keeps track of when things need to be renewed in there and so on16:44
clarkbI need to take a break, but will file an issue upstream and send a PR when I get back. Then our weekly meeting16:45
fungiand yeah, it's not urgent, we have weeks to figure out a plan16:46
*** jpena is now known as jpena|off17:17
*** artom_ is now known as artom17:24
fungiwhen reviewing a pr in github, going to the files changed tab there's a review changes drop-down which gives several options. if as a casual observer i want to indicate i'm in favor of the patch, is that what the "approve" option is for, or will people think it's weird for a non-maintainer of the project to "approve" a pr?17:25
fungithe available options seem to be comment, approve, or request changes... i'm essentially wanting to leave an equivalent of our code-review +117:27
clarkbI think you can leave a thumbs up emoji or similar on the PR message17:27
clarkbthumbs up, smiley, heart etc. But it probably varies by repo/project17:28
fungithanks, i'll do that instead (and yeah it's a fairly off-topic question here, except insofar as i'm reviewing a pr for a project we use heavily)17:28
fungii thumbs-upped the pr description17:29
clarkbhttps://github.com/acmesh-official/acme.sh/issues/4077 is the acme.sh issue. Now to do a pull request17:39
* fungi readies his thumbs-up emoji17:39
*** tweining is now known as tweining|off18:00
* frickler would have real issues working on code with french pronouns all day long18:47
fricklerbtw., fedora and opensuse mirroring is broken, in case someone wants to have a look. connection refused for fedora and something 15.2 missing for suse18:47
clarkbfrickler: huh I thought it may have been Le == LetsEncrypt but now I wonder if it is french18:48
fricklerclarkb: I'm convinced that they intend to mean LetsEncrypt, but my reading circuit is wired differently, in particular with that capitalization18:50
clarkbhttp://mirror.clarkson.edu/opensuse/opensuse/distribution/leap/15.2/ seems to still be thre re opensuse. I wonder if it is the OBS mirroring that is broken18:51
clarkbya I think the issue is obs https://provo-mirror.opensuse.org/repositories/Cloud%3A/OpenStack%3A/18:52
clarkbmaybe we need tojust drop OBS?18:52
clarkbI need to prep for the meeting though so can't dig into that much further right now18:53
*** dviroel|lunch|afk is now known as dviroel\18:53
*** dviroel\ is now known as dviroel18:53
ianwrsync: failed to connect to mirrors.us.kernel.org (2604:1380:4040:4f00::1): Connection timed out (110)20:06
clarkbI'm going to grab lunch now. For acme.sh I think I'm going to just wait and see what upstream does for the next few days before worrying about manual intervention. But if others want to get ahead of it thats fine with me20:06
ianwi feel like being able to force a refresh is actually a decent idea.  it would be nice for us to be able to just automatically roll all certs with the flip of a flag if we watned20:07
clarkbya that seems like a good feature20:08
ianwi've had on my todo list to "use pebble for CI" for so long that pebble is probably no longer the thing cool people use to test acme workflows20:08
ianwi'm not sure even in this case it would help, though20:08
ianwi'm guessing that maybe mirrors.us.kernel.org got pointed at something new, that now doesn't expose rsync (either through design or overlooked)20:10
ianwi can't find anything discussing this, although may be looking in the wrong places.  i've sent mail to the address listed on https://www.kernel.org/category/contact-us.html20:26
ianwi'll give a little time to see if anyone reads that, or find another mirror20:32
opendevreviewMerged opendev/elastic-recheck rdo: Fix failing tests  https://review.opendev.org/c/opendev/elastic-recheck/+/84130620:36
opendevreviewMerged opendev/elastic-recheck rdo: Updating gitignore to ignore local config files  https://review.opendev.org/c/opendev/elastic-recheck/+/82428320:39
opendevreviewClark Boylan proposed openstack/project-config master: Stop e-r gerrit alerts in #opendev  https://review.opendev.org/c/openstack/project-config/+/84133920:50
opendevreviewClark Boylan proposed opendev/system-config master: Fix opensuse upstream mirroring path  https://review.opendev.org/c/opendev/system-config/+/84134021:02
clarkbthat might be the fix for opensuse? its hard to tell without running an rsync21:02
clarkband seems like it can't be any worse than the current situation :021:03
clarkber :)21:03
ianwhttps://mirror.clarkson.edu/opensuse/opensuse/ ... that feels a lot like a trailing "/" rsync issue upstream 21:07
clarkboh ha good point21:09
ianwbut yeah, not sure what we can do about that21:09
clarkbianw: looking at acme.sh ya I think we move aside the domain.tld.conf file for the key and that causes acme.sh to not find a next renew time so it cannot skip based on it being too soon and it should reissue21:17
clarkbHowever, I'm not convinced it won't hit this same bug21:17
opendevreviewMerged openstack/project-config master: Stop e-r gerrit alerts in #opendev  https://review.opendev.org/c/openstack/project-config/+/84133921:18
clarkbbecause it will get a default value of "" or whatever for the key size which is different than 2048 and then fail because it cannot create a new key21:18
clarkbI think if we want ot get around this until upstream fixes that bug we need to use --force21:18
clarkbthere is an option to always force generating a new key rather than using a new cert each time for existing key21:19
*** dviroel is now known as dviroel|out21:22
ianwis there a reason to not use a different key every time?21:24
clarkbfungi: ^?21:25
funginot that i can think of other than efficiency21:26
clarkb--always-force-new-domain-key is the flag21:26
clarkbthere may be races with tools trying to read the key and cert where the key has been updated but not the cert?21:27
clarkbapache won't be affected by that as it waits for us to reload it before reading both files, but potentailly other tools could do that?21:27
ianwyeah, it maybe feels less risky to just use your patch 21:30
ianwperhaps we should consider adding the flags for ec-256 21:32
clarkbthatwould also require we rekey21:33
clarkbare there client and server compatibility concerns with that too?21:34
ianwlooks like it's still not standard21:37
clarkbI'm going to take a break mid afternoon to get a bike ride in while it is sunny and not rainy21:39
*** rlandy is now known as rlandy|bbl22:17
opendevreviewMerged opendev/system-config master: Fix opensuse upstream mirroring path  https://review.opendev.org/c/opendev/system-config/+/84134022:36
*** mazzy5098812929580859497 is now known as mazzy50988129295808594923:29
ianwwe probably do run our fedora syncs a little too often23:35
ianwmirror.cogentco.com says *** Please limit your rsyncs to a max of 4x per day *** and that's probably about right23:35
clarkbno objections from me to reduce our resync attempts23:39
fungiyeah, switch from every 2 hours to every 6?23:40
fungior even every 823:40
ianwnot everyone does rawhide which limits our options a bit23:51
ianwif you change the frequency in the cron ansible, i assume it overwrites the old version?23:56
clarkbianw: yes I think so23:57
clarkbit uses the task name to track it or something23:58
clarkbif you change the task name or whatever identifier it uses it won't be able to update23:58
opendevreviewIan Wienand proposed opendev/system-config master: mirror-update: switch Fedora mirror  https://review.opendev.org/c/opendev/system-config/+/84134923:59

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!