| opendevreview | Ian Wienand proposed opendev/infra-vhd-util-deb focal: Initial import of debian build files https://review.opendev.org/c/opendev/infra-vhd-util-deb/+/841175 | 00:02 |
|---|---|---|
| opendevreview | Ian Wienand proposed opendev/infra-vhd-util-deb focal: [wip] test vhd-util build https://review.opendev.org/c/opendev/infra-vhd-util-deb/+/841179 | 00:02 |
| ianw | clarkb: https://review.opendev.org/c/opendev/infra-vhd-util-deb/+/841174 change should make it clearer | 00:03 |
| ianw | change description i mean | 00:04 |
| opendevreview | Ian Wienand proposed opendev/infra-vhd-util-deb focal: [wip] test vhd-util build https://review.opendev.org/c/opendev/infra-vhd-util-deb/+/841179 | 00:05 |
| *** rlandy|bbl is now known as rlandy|out | 01:22 | |
| opendevreview | Ian Wienand proposed opendev/infra-openafs-deb jammy: Test generic job path https://review.opendev.org/c/opendev/infra-openafs-deb/+/841189 | 01:53 |
| opendevreview | Ian Wienand proposed openstack/project-config master: infra-vhd-util-deb : add jobs https://review.opendev.org/c/openstack/project-config/+/841190 | 02:01 |
| opendevreview | Ian Wienand proposed opendev/infra-openafs-deb jammy: Test generic job path https://review.opendev.org/c/opendev/infra-openafs-deb/+/841189 | 02:06 |
| opendevreview | Merged opendev/infra-openafs-deb jammy: Test generic job path https://review.opendev.org/c/opendev/infra-openafs-deb/+/841189 | 03:15 |
| *** soniya29 is now known as soniya29|ruck | 04:38 | |
| *** ysandeep|out is now known as ysandeep|rover | 04:42 | |
| opendevreview | Ian Wienand proposed openstack/project-config master: infra-vhd-util-deb : add jobs https://review.opendev.org/c/openstack/project-config/+/841190 | 04:42 |
| opendevreview | Merged openstack/project-config master: infra-vhd-util-deb : add jobs https://review.opendev.org/c/openstack/project-config/+/841190 | 04:59 |
| opendevreview | Ian Wienand proposed opendev/infra-vhd-util-deb focal: Initial import https://review.opendev.org/c/opendev/infra-vhd-util-deb/+/841174 | 05:04 |
| opendevreview | Merged opendev/infra-vhd-util-deb focal: Initial import https://review.opendev.org/c/opendev/infra-vhd-util-deb/+/841174 | 05:16 |
| ianw | that ~ working -> https://launchpad.net/~openstack-ci-core/+archive/ubuntu/vhd-util-deb-test | 05:21 |
| *** pojadhav|afk is now known as pojadhav | 05:23 | |
| opendevreview | Merged opendev/system-config master: Use add instead of wget to download things for gerrit images https://review.opendev.org/c/opendev/system-config/+/715087 | 06:14 |
| opendevreview | Ian Wienand proposed opendev/infra-vhd-util-deb focal: Apply Citrix patch for conversion in vhd-util https://review.opendev.org/c/opendev/infra-vhd-util-deb/+/841209 | 06:46 |
| opendevreview | Ian Wienand proposed opendev/infra-vhd-util-deb focal: Import changes to build on Ubuntu Bionic https://review.opendev.org/c/opendev/infra-vhd-util-deb/+/841210 | 06:53 |
| opendevreview | Ian Wienand proposed opendev/infra-vhd-util-deb focal: Apply changes to build on Ubuntu Focal https://review.opendev.org/c/opendev/infra-vhd-util-deb/+/841211 | 06:53 |
| opendevreview | Ian Wienand proposed opendev/infra-vhd-util-deb focal: Import changes to build on Ubuntu Focal https://review.opendev.org/c/opendev/infra-vhd-util-deb/+/841211 | 06:54 |
| opendevreview | Ian Wienand proposed opendev/infra-vhd-util-deb focal: Update Debian version number https://review.opendev.org/c/opendev/infra-vhd-util-deb/+/841212 | 07:00 |
| opendevreview | Merged opendev/infra-vhd-util-deb focal: Apply Citrix patch for conversion in vhd-util https://review.opendev.org/c/opendev/infra-vhd-util-deb/+/841209 | 07:06 |
| opendevreview | Merged opendev/infra-vhd-util-deb focal: Import changes to build on Ubuntu Bionic https://review.opendev.org/c/opendev/infra-vhd-util-deb/+/841210 | 07:07 |
| opendevreview | Merged opendev/infra-vhd-util-deb focal: Import changes to build on Ubuntu Focal https://review.opendev.org/c/opendev/infra-vhd-util-deb/+/841211 | 07:08 |
| opendevreview | Merged opendev/infra-vhd-util-deb focal: Update Debian version number https://review.opendev.org/c/opendev/infra-vhd-util-deb/+/841212 | 07:14 |
| *** jpena|off is now known as jpena | 07:30 | |
| *** soniya29|ruck is now known as soniya29|ruck|lunch | 08:09 | |
| *** ykarel is now known as ykarel|away | 08:25 | |
| *** ysandeep|rover is now known as ysandeep|rover|lunch | 08:49 | |
| *** soniya29|ruck|lunch is now known as soniya29|ruck | 09:21 | |
| *** bhagyashris is now known as bhagyashris|out | 09:33 | |
| *** ysandeep|rover|lunch is now known as ysandeep|rover | 09:47 | |
| opendevreview | A R proposed openstack/diskimage-builder master: Preserve local mirrors when using Ubuntu element https://review.opendev.org/c/openstack/diskimage-builder/+/841247 | 10:09 |
| *** rlandy|out is now known as rlandy | 10:33 | |
| *** dviroel|afk is now known as dviroel | 11:28 | |
| ecsantos[m] | Hello folks | 11:44 |
| ecsantos[m] | My team (NetApp) maintaints a third-party CI for the Cinder and Manila projects | 11:44 |
| ecsantos[m] | The Gerrit account we use to report job results was registered a long time ago (back in 2014 I reckon), and we can no longer access it | 11:45 |
| ecsantos[m] | When trying to sign in, we get a "Not Found" page with the https://review.opendev.org/SignInFailure,SIGN_IN,Contact+site+administrator URL | 11:45 |
| ecsantos[m] | Where can we get some help on this? | 11:45 |
| fungi | ecsantos[m]: what's the username? | 11:48 |
| ecsantos[m] | fungi: I only have the display name, NetApp CI | 11:49 |
| fungi | the ci system has to specify its username when it authenticates to gerrit though | 11:49 |
| fungi | do you not have access to the ci system itself? | 11:50 |
| *** soniya29|ruck is now known as soniya29|ruck|break | 11:58 | |
| frickler | there is an account with that name in the third-party CI group, but better double-check with how they try to log in, I agree | 12:11 |
| rosmaita | according to this: https://wiki.openstack.org/wiki/ThirdPartySystems/NetApp_CI , the gerrit username is netapp-ci | 12:17 |
| fungi | ecsantos[m]: if the account is "very old" it would have been manually created by our gerrit sysadmins with no ability to log into the webui. the recommendation for updating those accounts is to create a new account and ask to have the old one marked inactive | 12:22 |
| *** soniya29|ruck|break is now known as soniya29|ruck | 12:27 | |
| ecsantos[m] | fungi: Unfortunately we only have that e-mail, I can even access Launchpad with it. Would we really need a new e-mail, or would just marking the account as inactive and trying to sign in again work? | 12:31 |
| fungi | ecsantos[m]: i think if i take that e-mail address off the old account, then you'll be able to create a new gerrit account with the launchpad login you already have | 12:32 |
| ecsantos[m] | fungi: That'd be okay with us | 12:33 |
| fungi | looks like username:netapp-ci has accountId = 10621 | 12:35 |
| fungi | ecsantos[m]: the xdl-openstack-jenkins@ address is the one you're using with it currently? | 12:38 |
| fungi | just want to make sure i'm looking at the right account | 12:38 |
| ecsantos[m] | fungi: It's ng-openstack-ci@netapp.com | 12:38 |
| ecsantos[m] | At least that's the e-mail it shows when we click on the Gerrit account | 12:39 |
| fungi | aha, that's username:NetApp-ci with accountId = 25243 | 12:40 |
| fungi | unfortunately, gerrit usernames in our deployment are historically case-sensitive, so username:netapp-ci and username:NetApp-ci are separate accounts | 12:40 |
| fungi | anyway, that account has an ubuntu login associated with it | 12:41 |
| fungi | so it's not an older account created by one of our admins | 12:41 |
| fungi | ecsantos[m]: and you say trying to authenticate from login.ubuntu.com is failing to sign in? i'll check the logs to see if there are any associated errors | 12:42 |
| fungi | ecsantos[m]: oh, fun, it's apparently created a new ubuntu login and that's trying to create a new account, but gerrit is refusing to let account creation complete because the same e-mail address is in use by existing account 25243 | 12:44 |
| fungi | did you change anything you can think of in launchpad/ubuntuone which would result in a different openid? | 12:44 |
| ecsantos[m] | yeah I just tried loggin in | 12:44 |
| ecsantos[m] | actually no one in our team have tried to log in into that account in recent memory | 12:45 |
| ecsantos[m] | so we don't know :/ | 12:45 |
| fungi | okay, something has definitely resulted in the openid for it changing vs the one gerrit knew. i'll see if i can add the new openid to the existing account | 12:46 |
| fungi | oh, though i may not be able to push this through gerrit. before i go screwing something up, i want to double-check with clarkb and he's probably not awake yet | 12:47 |
| ecsantos[m] | last login to review.openstack.org on our side was on 2015/06/15 | 12:48 |
| ecsantos[m] | fungi: no problem, it's not that urgent | 12:48 |
| fungi | ecsantos[m]: yeah, we should be able to update this so that the lp account you're using logs into the gerrit account you have, it'll just take a little surgery on the account records and i want to be sure i do it right in notedb (it used to be a lot more straightforward when all this was an sql db) | 12:51 |
| ecsantos[m] | fungi: Thanks for the help! If you need anything please let me know | 12:56 |
| fungi | will do, and i'll let you know once we're ready for you to test logging in again | 13:01 |
| opendevreview | A R proposed openstack/diskimage-builder master: Preserve local mirrors when using Ubuntu element https://review.opendev.org/c/openstack/diskimage-builder/+/841247 | 13:07 |
| Clark[m] | fungi: correct you cannot do that push until all the existing errors are corrected in the DB unless Gerrit 3.4 changed that behavior. I sort of fizzled out on fixing the last 35 or so errors as it is quite time consuming and there is always something else to do. You can try it though and Gerrit should reject it. | 13:12 |
| Clark[m] | What we can do is retire the old account and remove the openids and email addresses from it then let the new openid create a new account using those attributes | 13:14 |
| fungi | Clark[m]: so use the rest api to remove the e-mail address and username from the old account and set it inactive? | 13:20 |
| Clark[m] | I typically set it inactive and remove the preferred email address using a git push to the accounts specific ref. Then when that is done use the rest API to remove the external ids. The scripts I pushed into system-config for DB cleanups can do that. One does the "retirement" which sets inactive and removes the preferred email setting. The other deletes external ids | 13:24 |
| Clark[m] | Before that happens may be good to double check with ecsantos that they don't use the account with ssh anywhere | 13:25 |
| Clark[m] | I suspect they are actively reporting via ssh. And this surgery will temporarily break that until they update ssh keys in Gerrit and ssh usernames in their CI system | 13:26 |
| fungi | https://review.opendev.org/Documentation/rest-api-accounts.html#delete-account-external-ids seems like it can delete arbitrary external-ids, i guess that's a workaround for not having a specific method to delete usernames? | 13:27 |
| Clark[m] | Ya usernames are just external ids. | 13:28 |
| ecsantos[m] | Clark: That's correct, it'll temporarily break job reporting for us but that's okay, SSH is used by our old Zuul v2 CI, we're migrating to v3 and as soon as we can create the new account I'll properly set up SSH keys for it | 13:29 |
| Clark[m] | ecsantos you'll need to configure a new username on your clients too. At least historically we've not allowed those to migrate between accounts (there is a post somewhere by Gerrit upstream indicating why doing so is a bad idea iirc) | 13:30 |
| Clark[m] | The other alternative is to shutdown Gerrit. Push the openid update to the existing account behind gerrit's back then offline reindex and start Gerrit again | 13:31 |
| ecsantos[m] | Clark: Whatever is easier for you guys works for us | 13:43 |
| fungi | Clark[m]: so when using the retire-user.sh script, is the assumption that it will push through the gerrit ssh remote or locally to the bare repo on the backing filesystem? | 13:43 |
| fungi | my admin account seems to not have push rights | 13:49 |
| fungi | to the user ref | 13:49 |
| fungi | were you temporarily adding your account to a different group? project bootstrappers maybe? | 13:50 |
| fungi | yeah that worked | 13:51 |
| fungi | so refs/users/43/25243 is now retired and i should be able to clean up the external-ids | 13:52 |
| Clark[m] | fungi: ya it pushes to Gerrit | 13:52 |
| Clark[m] | The idea with those scripts is everything is done through Gerrit to avoid races and to avoid needing reindexing | 13:53 |
| Clark[m] | And ya to do that you need to be a bootstrapper | 13:53 |
| Clark[m] | You only need to be an admin to run the rest API commands | 13:54 |
| Clark[m] | Updating the user ref isn't a big race concern if the user can't login anyway. But all the external ids are under a shared ref and updates to those can race. This is why we have to stop Gerrit if changing them behind gerrit's back | 14:00 |
| fungi | yep, so the external-ids referencing that e-mail address have been removed now. on a whim i tried deleting the external-id for the username and gerrit actually refuses | 14:09 |
| fungi | returns a 409 error | 14:10 |
| fungi | ecsantos[m]: okay, i think you're all set to try logging in again when you have a moment. this will create a new account with your new openid, but should be able to reuse the old e-mail address. as Clark[m] mentioned, you'll need to set a new username in gerrit (can't reuse the old username), as well as reuploading ssh keys | 14:11 |
| ecsantos[m] | fungi: Okay, will try logging in | 14:12 |
| Clark[m] | Good to know Gerrit doesn't allow the username removal. I know it causes problems if you do it and the service is protecting itself. That is good | 14:13 |
| ecsantos[m] | fungi Clark: It worked! Big thanks for the help folks | 14:15 |
| fungi | hooray! thanks for confirming | 14:15 |
| ecsantos[m] | Just trying to think of a new username now :p | 14:15 |
| fungi | also good luck on your zuul v3 (hopefully really v6) deployment! | 14:15 |
| ecsantos[m] | Thanks! We're on v4.9 hoping to go v5 soon :) (looking forward to use community.general on v6) | 14:17 |
| fungi | yeah, the jump from 5 to 6 isn't a significant amount of upgrade effort, 6.0.0 mainly just signals the removal of the old ansible prohibitions so it will be able to use newer ansible versions easily | 14:18 |
| fungi | the upgrade past 4.9 does require some care in handling the migtation of project keys into zookeeper | 14:19 |
| fungi | er, migration | 14:19 |
| fungi | but beyond that it's pretty straightforward (release notes have upgrade guidance) | 14:20 |
| ecsantos[m] | We're using Software Factory actually, so hopefully their upgrade playbooks will take care of that for us :) | 14:22 |
| rpittau | hi all! A little doubt, is it possible to override a branch in a zuul template? or do we need to create a new job based on that and override the branch there? | 14:25 |
| clarkb | fungi: it looks like ssl certs are still not being refreshed | 14:28 |
| clarkb | I need to find breakfast, but I'll probably start my day by looking at that if it isn't already solved | 14:28 |
| clarkb | rpittau: project-templates don't have branch settings. That is a job thing. Generally you should define the job you want to run on a branch in that branch. Then the implicit lookups will find it properly. If your job is defined in a config project or branchless project like tempest you can make a variant of that job that applies to specific branches. But do not use the branches: | 14:29 |
| clarkb | keyword in jobs defined on projects with branches it will just lead to problems later when you branch that project again | 14:29 |
| rpittau | clarkb: the problem is that we have the template in a bugfix branch and it takes upper-constraints from master :/ | 14:30 |
| clarkb | rpittau: I think that is usually appropriate since bug fixes tend to target master then get backported? | 14:31 |
| rpittau | I guess the implicit lookup takes into consideration only stable branches | 14:31 |
| clarkb | that is generally how things are configured the expectation being if there aren't explicit rules for something then treat it like master for this reason | 14:31 |
| rpittau | clarkb: not in our case, we have bugfix branches based on stable branches | 14:31 |
| clarkb | no the implicit lookup applies to all branches, but if it doesn't find your branch it falls back to master | 14:32 |
| rpittau | ok, so I guess we need to define a job and override the branch | 14:32 |
| clarkb | you can add a pragma that defines mappings. Devstack does this iirc | 14:32 |
| clarkb | rpittau: https://opendev.org/openstack/devstack/src/branch/master/.zuul.yaml#L1-L7 | 14:32 |
| clarkb | that says feature/r1 is equivalent to master I think | 14:33 |
| rpittau | clarkb: mmmm ok, interesting | 14:33 |
| rpittau | clarkb: thanks! | 14:33 |
| clarkb | rpittau: I'm curious why you would need bug fix branches targetting stable branches if the idea is to always backport fixes? I guess you are fixing bugs in stable branches but not master? | 14:34 |
| clarkb | so the backport begins behind master | 14:34 |
| rpittau | clarkb: because we have intermediate releases | 14:34 |
| rpittau | that are based on those bugfix branches | 14:34 |
| rpittau | because we like complicate things :D | 14:35 |
| clarkb | I'm not sure I would be doing releases off of bugfix branches | 14:35 |
| clarkb | seems like a really good way to miss fixes | 14:35 |
| clarkb | for newer releases I mean | 14:35 |
| clarkb | if you do a bugfix branch, merge to master, branch master off for stable intermediate release, then release you're ensuring that future releases don't miss your fixes | 14:36 |
| rpittau | the bugfix branches have usually a shorter life than stable branches | 14:36 |
| clarkb | that sort of process would be more in line with the general expectation of how many of openstack's jobs are designed | 14:36 |
| rpittau | but in that case we also get master features | 14:36 |
| clarkb | right but master doesn't necessarily get your bugfix | 14:37 |
| clarkb | (which is the concern I'm trying to highlight) | 14:37 |
| rpittau | mmmm no, we fix on master, then we backport to all supported stable and bugfix branches | 14:38 |
| rpittau | but we cherry-pick only the fix | 14:38 |
| clarkb | ok I may not be understanding what you mean by bugfix branch. But if you backport to it and make releases from it I'm not sure I would characterize that as a bugfix branch | 14:38 |
| clarkb | in any case your have bugfix/foo and if you need to map that to stable/bar instead of master I think you can use the pragmas like devstack. | 14:40 |
| rpittau | I probably didn't explain myself properly, we do only one host release from bugfix branches, once it's done, it's done | 14:41 |
| rpittau | all the subsequent fixes backport to the bugfix branch are not released, but they're kept in that branch | 14:41 |
| rpittau | one shot* | 14:41 |
| rpittau | clarkb: I'll give the pragma thing a try, thanks! | 14:41 |
| fungi | yeah, it's not that implicit lookups only take stable branches into consideration, it's that zuul assumes you'll have the same branch name in every project involved in a job, and if that branch is missing from some project it falls back to using the default branch for that project (typically master) | 14:45 |
| clarkb | right. Implicit lookups look for consistent naming across the system. The way openstack is set up is to have master and a series of stable branches. The stable branches will find each other and use consistent config. Master finds master and is consistent. If you use a bugfix/foo or feature/foo the assumption by default is that those fixes or those features apply to master and you'll | 14:47 |
| clarkb | get master configs. Then when they land you can backport to stable branches as necessary | 14:47 |
| fungi | there are ways (for example that pragma) to map branches with different names to each other between projects, but it gets complicated and not really how project integration was designed | 14:47 |
| clarkb | I think if this were me I'd branch stable/foo early to do the intermediate release. It will use master things until the rest of openstack makes stable/foo at which time it will become equivalent to stable/foo | 14:50 |
| clarkb | then you do your stable/foo release off of that branch later with the newer commits | 14:50 |
| clarkb | rpittau: ^ fyi that would be the way to do it while integrating with openstack I think | 14:51 |
| clarkb | whether or not that would break other things I do not know. | 14:51 |
| *** dviroel is now known as dviroel|lunch|afk | 14:53 | |
| rpittau | clarkb, fungi: thanks again, "our" bugfix branches usually don't last very long, definitely less than a stable branch, I think the pragma option could work fine for our needs | 14:54 |
| clarkb | right its less about timeperiods and more about representing the intent with the existing machinery | 14:55 |
| clarkb | pragma does that by being explicit about weird rules. The suggestion above tries to do it by operating with the existing rules | 14:55 |
| clarkb | btu I'm sure it would need a bit more thought to determine if it would create other problems first | 14:56 |
| *** soniya29|ruck is now known as soniya29|out | 15:20 | |
| clarkb | acme.sh is failing because it sees and existing domain key and for some reason thinks it isn't renewing but issuring a new cert. This fails beacuse it wants you to pass --force in this instance. | 15:49 |
| clarkb | Thing is we are renewing existing certs so I dunno why it is broken, but looking at recent changes to acme.sh they broke a bunch of renewal stuff and are fixing it so possibly related to that? However I have no clue if those fixes are sufficient to start over or if we need to more forcefully do that | 15:50 |
| clarkb | oh we actually do the two pass thing of issuing then renewing | 15:52 |
| clarkb | I think it is the issue command that is bailing out | 15:52 |
| fungi | okay, so your first impression is some recent change to acme.sh has broken us? | 15:54 |
| fungi | like in the past week? | 15:54 |
| clarkb | or month but ya | 15:54 |
| fungi | well, we didn't start getting warnings until the weekend, and that seemed to be in part becasue we stopped running the base deploy on friday | 15:54 |
| fungi | so i have a feeling it was in the past week, maybe even just in the past day or two | 15:55 |
| clarkb | the commit that is suspicious is "fix renew server" of course there isn't more info than that | 15:56 |
| fungi | of course! what's the point of commit messages when you have the diff? ;) | 15:59 |
| frickler | possibly we didn't try to renew for some time before that if all certs were >30d valid? | 16:01 |
| clarkb | we do an `acme.sh --server letsencrypt --cert-home /etc/letsencrypt-certs --no-color --yes-I-know-dns-manual-mode-enough-go-ahead-please --issue --dns --challenge-alias acme.opendev.org` this logs "Creating domain key" "Domain key exists, do you want to overwrite the key?" "Add '--force', and try again." "Create domain key error." | 16:01 |
| clarkb | looking at the source for acme.sh there is a $_ACME_IS_RENEW flag that is meant to avoid that error when renewing | 16:02 |
| clarkb | what --force will do is create a new domain key (which is probably ok? I'm not sure yet) which would be a change of behavior for us | 16:03 |
| fungi | we renew a large enough number of certs that i expect the distribution of renewal dates is sufficiently distributed over time for us to notice problems like this within a few days of arising, usually | 16:06 |
| clarkb | the call for createDomainKey is guarded with if [ ! -f "$CERT_KEY_PATH" ] || [ "$_key_length" != "$_key" ] || [ "$Le_ForceNewDomainKey" = "1" ]; then so maybe one of those other flags has changed | 16:07 |
| fungi | clarkb: are you saying we were previously skipping the domain key replacement? | 16:07 |
| clarkb | fungi: pretty sure we were | 16:07 |
| clarkb | since the code that generates the domain key hasn't changed in a long time | 16:08 |
| clarkb | I'm suspecting that the changes are to when/how it is called rather than the actual routine | 16:08 |
| fungi | i see, so the "already exists" condition was just causing it to move on to the next step up to now | 16:08 |
| clarkb | 7f9074adbf2f2aeba61db36a3233730c4768c033 is also suspicious as it deals with key lengths which are checked in that test | 16:10 |
| clarkb | fungi: that is my hunch | 16:10 |
| clarkb | fungi: eitehr we never called createDomainKey because the guard above skipped it or we called it and then it saw it was renewing and ignored it | 16:10 |
| clarkb | ya ok I think it is the keylength | 16:12 |
| fungi | oho, so they switched to a longer key? | 16:13 |
| clarkb | the zuul.opendev.org.conf file lists Le_Keylength='' | 16:13 |
| clarkb | and that value gets stuffed in $_key in that condition above | 16:13 |
| clarkb | then $_key_length appears to be 2048 | 16:13 |
| clarkb | I think it was always 2048 they just changed how they check it and broke us? | 16:13 |
| clarkb | the commit sha above 7f907 is what I suspect broke things | 16:13 |
| fungi | oh, interesting, so we're trying to tell it to use its default length there, and it sees that as a mismatch now | 16:14 |
| clarkb | oh sorry its that commit and 64847afc3ff8cfe214aca7db7f793d96bee95e5e together | 16:14 |
| clarkb | fungi: ya I think so | 16:14 |
| clarkb | ya I think I see the fix | 16:15 |
| clarkb | in review() which we call after issue() it checks if Le_Keylength (which gets stuffed in $_key in issue()) is empty and is so sets it to 2048 | 16:15 |
| clarkb | but issue() doesn't seem to do that check and it fails | 16:15 |
| clarkb | another option would be to manually edit the files and set the value to 2048 | 16:16 |
| clarkb | and then I think it will just proceed from there properly | 16:16 |
| *** marios is now known as marios|out | 16:17 | |
| opendevreview | dasm proposed opendev/elastic-recheck rdo: Fix failing tests https://review.opendev.org/c/opendev/elastic-recheck/+/841306 | 16:18 |
| clarkb | Let me try to collect this together more concretely so that others can weigh in | 16:18 |
| opendevreview | dasm proposed opendev/elastic-recheck rdo: Enable podman-compose https://review.opendev.org/c/opendev/elastic-recheck/+/841172 | 16:19 |
| opendevreview | dasm proposed opendev/elastic-recheck rdo: Updating gitignore to ignore local config files https://review.opendev.org/c/opendev/elastic-recheck/+/824283 | 16:19 |
| clarkb | https://github.com/acmesh-official/acme.sh/blob/dev/acme.sh#L1551-L1553 is where we are breaking after calling issue() per the logs on zuul01. | 16:23 |
| clarkb | https://github.com/acmesh-official/acme.sh/blob/dev/acme.sh#L4411-L4416 is where issue() calls createDomainKey. It only does this if the key doesn't already exist or you change the key length or if you pass --force | 16:23 |
| *** ysandeep|rover is now known as ysandeep|out | 16:23 | |
| clarkb | https://github.com/acmesh-official/acme.sh/commit/64847afc3ff8cfe214aca7db7f793d96bee95e5e updated the way key lengths are read in and handled and I'm 99% sure that the check is now comparing 2048 against "" and decided they are different which causes the createDomainKey function to be called | 16:24 |
| clarkb | We can fix that by checking if $_key is "" and setting it to 2048 first. 64847 does this in the renew() path already but not the issue path. Or we can modify our key conf files to set the keylength explicitly to 2048 out of band and let it continue | 16:25 |
| fungi | we could do the latter and propose the former upstream too | 16:25 |
| clarkb | ya | 16:25 |
| fungi | and then revert the explicit 2048 once upstream is fixed | 16:26 |
| clarkb | we wouldn't want to rever that as it will write 2048 back automatically for us | 16:26 |
| clarkb | so after the first successful pass this should work going forward. For that reason I actually think it may be useful to just fix this upstream? | 16:27 |
| clarkb | also we can probably wait for ianw to weigh in as ianw has worked upstream with them in the past and may have input on the best way to approach this | 16:27 |
| fungi | i'm probably misunderstanding. the current problem is that we set the keylength to an empty string in our config, and that confuses acme.sh into forcing replacement even though the existing key is the default length already. if we set the config explicitly to 2048 instead of an empty string, then that would work around it temporarily. if we fixed it upstream to know not to force replacement of | 16:33 |
| fungi | the key, then we'd be able to safely revert back to setting the empty string in our config, right? | 16:33 |
| clarkb | if we set it to 2048 that would fix it permanently unless they change the code again. It would be a permanent fix because acme.sh will read 2048 and compare it against our requested size of 2048 and be happy. That will skip the first pass of createDomainKey(). After that it will write 2048 back to the conf file for us | 16:35 |
| clarkb | if we revert back to the empty string it will fail on the next pass | 16:35 |
| clarkb | because it expects 2048 there (this is why it writes it back again) | 16:35 |
| fungi | er... acme.sh writes to its own configuration file? we don't write out the config for it? | 16:36 |
| fungi | maybe that's where i'm getting confused. i expected the configuration to be immutable from the application's perspective | 16:36 |
| clarkb | correct this file is tied to the key itself and acme.sh writes it out. We don't touch it | 16:37 |
| fungi | but sounds like you're saying the "configuration" is more of a state cache | 16:37 |
| clarkb | ya in a way I guess it is | 16:37 |
| clarkb | acme.sh generates the key and writes a config for that key alongside it | 16:37 |
| clarkb | so that when it runs the next time it knows some info about the key without executing openssl against it I guess | 16:38 |
| fungi | okay, so i completely misunderstood the suggestion, sorry. so acme.sh was writing out an empty string for the key length field in its state metadata, essentially, but them misinterpreting it as something other than an indication it had previously used its default value | 16:38 |
| fungi | not a problem with any configuration we were supplying | 16:39 |
| clarkb | correct and 64847af is the commit that changed their internal assumptions recently | 16:39 |
| clarkb | another option is to force it to generate new keys once by setting --force | 16:40 |
| clarkb | I think I'm coming around to fixing this upstream though since it will adjust the config appropriately for us and there are probably other users out there with this problem | 16:40 |
| fungi | so to reset... your suggestion was that we could manually fix up the state information previously recorded to have an explicit key length, in order to get it back on track | 16:40 |
| clarkb | though maybe we are the only ones doing the dns issuance which causes both issue and renew to be called | 16:40 |
| clarkb | fungi: correct. And we'd do that manually since that isn't tracked in our config management and probably shouldn't be | 16:41 |
| fungi | got it. i was confused by the use of the term "configuration" for this. it just happens to (ab)use openssl's config format for recording additional data about the key it generates | 16:42 |
| fungi | but it's not configuration from our perspective, it's just data the application is maintaining | 16:43 |
| clarkb | ya its configuration for the key managed by acme.sh | 16:43 |
| clarkb | it keeps track of when things need to be renewed in there and so on | 16:44 |
| clarkb | I need to take a break, but will file an issue upstream and send a PR when I get back. Then our weekly meeting | 16:45 |
| fungi | thanks! | 16:45 |
| fungi | and yeah, it's not urgent, we have weeks to figure out a plan | 16:46 |
| *** jpena is now known as jpena|off | 17:17 | |
| *** artom_ is now known as artom | 17:24 | |
| fungi | when reviewing a pr in github, going to the files changed tab there's a review changes drop-down which gives several options. if as a casual observer i want to indicate i'm in favor of the patch, is that what the "approve" option is for, or will people think it's weird for a non-maintainer of the project to "approve" a pr? | 17:25 |
| fungi | the available options seem to be comment, approve, or request changes... i'm essentially wanting to leave an equivalent of our code-review +1 | 17:27 |
| clarkb | I think you can leave a thumbs up emoji or similar on the PR message | 17:27 |
| clarkb | thumbs up, smiley, heart etc. But it probably varies by repo/project | 17:28 |
| fungi | thanks, i'll do that instead (and yeah it's a fairly off-topic question here, except insofar as i'm reviewing a pr for a project we use heavily) | 17:28 |
| fungi | i thumbs-upped the pr description | 17:29 |
| clarkb | https://github.com/acmesh-official/acme.sh/issues/4077 is the acme.sh issue. Now to do a pull request | 17:39 |
| * fungi readies his thumbs-up emoji | 17:39 | |
| clarkb | https://github.com/acmesh-official/acme.sh/pull/4078 | 17:45 |
| *** tweining is now known as tweining|off | 18:00 | |
| * frickler would have real issues working on code with french pronouns all day long | 18:47 | |
| frickler | btw., fedora and opensuse mirroring is broken, in case someone wants to have a look. connection refused for fedora and something 15.2 missing for suse | 18:47 |
| clarkb | frickler: huh I thought it may have been Le == LetsEncrypt but now I wonder if it is french | 18:48 |
| frickler | clarkb: I'm convinced that they intend to mean LetsEncrypt, but my reading circuit is wired differently, in particular with that capitalization | 18:50 |
| clarkb | http://mirror.clarkson.edu/opensuse/opensuse/distribution/leap/15.2/ seems to still be thre re opensuse. I wonder if it is the OBS mirroring that is broken | 18:51 |
| clarkb | ya I think the issue is obs https://provo-mirror.opensuse.org/repositories/Cloud%3A/OpenStack%3A/ | 18:52 |
| clarkb | maybe we need tojust drop OBS? | 18:52 |
| clarkb | I need to prep for the meeting though so can't dig into that much further right now | 18:53 |
| *** dviroel|lunch|afk is now known as dviroel\ | 18:53 | |
| *** dviroel\ is now known as dviroel | 18:53 | |
| ianw | rsync: failed to connect to mirrors.us.kernel.org (2604:1380:4040:4f00::1): Connection timed out (110) | 20:06 |
| clarkb | I'm going to grab lunch now. For acme.sh I think I'm going to just wait and see what upstream does for the next few days before worrying about manual intervention. But if others want to get ahead of it thats fine with me | 20:06 |
| ianw | i feel like being able to force a refresh is actually a decent idea. it would be nice for us to be able to just automatically roll all certs with the flip of a flag if we watned | 20:07 |
| clarkb | ya that seems like a good feature | 20:08 |
| ianw | i've had on my todo list to "use pebble for CI" for so long that pebble is probably no longer the thing cool people use to test acme workflows | 20:08 |
| ianw | i'm not sure even in this case it would help, though | 20:08 |
| ianw | i'm guessing that maybe mirrors.us.kernel.org got pointed at something new, that now doesn't expose rsync (either through design or overlooked) | 20:10 |
| ianw | i can't find anything discussing this, although may be looking in the wrong places. i've sent mail to the address listed on https://www.kernel.org/category/contact-us.html | 20:26 |
| ianw | i'll give a little time to see if anyone reads that, or find another mirror | 20:32 |
| opendevreview | Merged opendev/elastic-recheck rdo: Fix failing tests https://review.opendev.org/c/opendev/elastic-recheck/+/841306 | 20:36 |
| opendevreview | Merged opendev/elastic-recheck rdo: Updating gitignore to ignore local config files https://review.opendev.org/c/opendev/elastic-recheck/+/824283 | 20:39 |
| opendevreview | Clark Boylan proposed openstack/project-config master: Stop e-r gerrit alerts in #opendev https://review.opendev.org/c/openstack/project-config/+/841339 | 20:50 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Fix opensuse upstream mirroring path https://review.opendev.org/c/opendev/system-config/+/841340 | 21:02 |
| clarkb | that might be the fix for opensuse? its hard to tell without running an rsync | 21:02 |
| clarkb | and seems like it can't be any worse than the current situation :0 | 21:03 |
| clarkb | er :) | 21:03 |
| ianw | https://mirror.clarkson.edu/opensuse/opensuse/ ... that feels a lot like a trailing "/" rsync issue upstream | 21:07 |
| clarkb | oh ha good point | 21:09 |
| ianw | but yeah, not sure what we can do about that | 21:09 |
| clarkb | ianw: looking at acme.sh ya I think we move aside the domain.tld.conf file for the key and that causes acme.sh to not find a next renew time so it cannot skip based on it being too soon and it should reissue | 21:17 |
| clarkb | However, I'm not convinced it won't hit this same bug | 21:17 |
| opendevreview | Merged openstack/project-config master: Stop e-r gerrit alerts in #opendev https://review.opendev.org/c/openstack/project-config/+/841339 | 21:18 |
| clarkb | because it will get a default value of "" or whatever for the key size which is different than 2048 and then fail because it cannot create a new key | 21:18 |
| clarkb | I think if we want ot get around this until upstream fixes that bug we need to use --force | 21:18 |
| clarkb | there is an option to always force generating a new key rather than using a new cert each time for existing key | 21:19 |
| *** dviroel is now known as dviroel|out | 21:22 | |
| ianw | is there a reason to not use a different key every time? | 21:24 |
| clarkb | fungi: ^? | 21:25 |
| fungi | not that i can think of other than efficiency | 21:26 |
| clarkb | --always-force-new-domain-key is the flag | 21:26 |
| clarkb | there may be races with tools trying to read the key and cert where the key has been updated but not the cert? | 21:27 |
| clarkb | apache won't be affected by that as it waits for us to reload it before reading both files, but potentailly other tools could do that? | 21:27 |
| ianw | yeah, it maybe feels less risky to just use your patch | 21:30 |
| ianw | perhaps we should consider adding the flags for ec-256 | 21:32 |
| clarkb | thatwould also require we rekey | 21:33 |
| clarkb | are there client and server compatibility concerns with that too? | 21:34 |
| ianw | https://community.letsencrypt.org/t/ecdsa-availability-in-production-environment/150679 | 21:36 |
| ianw | looks like it's still not standard | 21:37 |
| clarkb | I'm going to take a break mid afternoon to get a bike ride in while it is sunny and not rainy | 21:39 |
| *** rlandy is now known as rlandy|bbl | 22:17 | |
| opendevreview | Merged opendev/system-config master: Fix opensuse upstream mirroring path https://review.opendev.org/c/opendev/system-config/+/841340 | 22:36 |
| *** mazzy5098812929580859497 is now known as mazzy509881292958085949 | 23:29 | |
| ianw | we probably do run our fedora syncs a little too often | 23:35 |
| ianw | mirror.cogentco.com says *** Please limit your rsyncs to a max of 4x per day *** and that's probably about right | 23:35 |
| clarkb | no objections from me to reduce our resync attempts | 23:39 |
| fungi | yeah, switch from every 2 hours to every 6? | 23:40 |
| fungi | or even every 8 | 23:40 |
| ianw | not everyone does rawhide which limits our options a bit | 23:51 |
| ianw | if you change the frequency in the cron ansible, i assume it overwrites the old version? | 23:56 |
| clarkb | ianw: yes I think so | 23:57 |
| clarkb | it uses the task name to track it or something | 23:58 |
| clarkb | if you change the task name or whatever identifier it uses it won't be able to update | 23:58 |
| opendevreview | Ian Wienand proposed opendev/system-config master: mirror-update: switch Fedora mirror https://review.opendev.org/c/opendev/system-config/+/841349 | 23:59 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!