| frickler | cloudflare does seem to have configured the openstack.org zone on their nameservers, but the switch of the delegation from .org is still pending? | 08:53 |
|---|---|---|
| frickler | I also notice that the TTL for e.g. the CNAME record for docs.openstack.org has been reduced from 3600 to 300? I'm hoping that that would only the for the transition? | 08:54 |
| frickler | also "curl https://openstack.org --resolve openstack.org:443:172.66.43.113" (one of the IPs in cloudflare DNS) results in "cert expired" :-/ | 09:00 |
| frickler | checking the openmetal LE issue, it seems they changed the email to be used from "infra-root@openstack.org" to something @openmetal.io, which explains why the former is getting a warning now | 10:49 |
| frickler | sadly the new acc isn't working either, the logs repeats "Account xxx@openmetal.io is not registered. Use 'run' to register a new account." | 10:50 |
| frickler | but looks to me like something the openmetal team should look at, not sure whether just sending a mail or opening a ticket via their portal would be better | 10:51 |
| frickler | .oO(if only every cloud would be doing support via IRC *sigh*) | 10:52 |
| opendevreview | Benjamin Schanzel proposed zuul/zuul-jobs master: Add build-diskimage role https://review.opendev.org/c/zuul/zuul-jobs/+/922911 | 11:09 |
| opendevreview | Benjamin Schanzel proposed zuul/zuul-jobs master: Add build_diskimage_environment role variable https://review.opendev.org/c/zuul/zuul-jobs/+/926224 | 11:09 |
| opendevreview | Benjamin Schanzel proposed zuul/zuul-jobs master: Add a diskimage-builder job https://review.opendev.org/c/zuul/zuul-jobs/+/926225 | 11:09 |
| fungi | frickler: apparently the domain registrar used by the openinfra foundation doesn't have any automated way to change ns delegation, the process is that one of the executive staff e-mails the support address of the registrar with the requested change and then waits for it to be processe | 12:28 |
| fungi | d | 12:28 |
| fungi | cscglobal.com parties like it's 1999 | 12:29 |
| frickler | welcome to the real world ;) | 12:31 |
| frickler | it still might be good to check the cert issue before the ns change becomes active | 12:32 |
| fungi | but yeah, their whois still lists rackspace's nameservers at the moment, and afilias (the org tld registrar) still returns rackspace's nameservers | 12:32 |
| fungi | frickler: interestingly, if i set openstack.org to 172.66.43.113 in /etc/hosts i don't have any problem with the cert | 12:34 |
| fungi | or rather my browser doesn't have any problem | 12:34 |
| fungi | also your curl command doesn't give me "cert expired" but rather "server certificate verification failed." | 12:36 |
| frickler | hmm, still repeats for me. likely they're anycasting things depending on the source (region) of the request. though the final result doesn't seem better in your case | 12:40 |
| fungi | and as for the ttl on records there, they all start out at "auto" which is i guess 5 minutes, but i can individually set them to fairly arbitrary lengths from a drop-down (1m,2m,5m,10m,15m,30m,1h,2h,5h,12h,1d) | 12:46 |
| fungi | looks like if it was an enterprise account there would also be a 30s ttl option | 12:47 |
| frickler | hmm, I really don't want to make you click through all our 100+ records to do that, is there any automation they offer? or a zone default that is used for "auto" which could be bumped to 1h? | 12:56 |
| fungi | there may be a bulk operation option in this interface, i just need to look around a bit | 12:59 |
| fungi | well, at the very least there's https://developers.cloudflare.com/api/operations/dns-records-for-a-zone-update-dns-record | 13:00 |
| fungi | and https://developers.cloudflare.com/api/operations/dns-records-for-a-zone-patch-dns-record | 13:01 |
| fungi | looks like a patch operation can set the ttl of a record | 13:01 |
| clarkb | frickler: when you say they changed email addresses for the LE thing in the openmetal cloud do you mean that happend without our intervention or was that a side effect of running some kolla commands? | 14:52 |
| clarkb | frickler: I think we can start with email and ask them if they want us to file a ticket instead. In particular I think we are a bit of a demo cloud so its good to talk directly to people who can figure things out rather than going through their normal ticket system I suspect | 14:53 |
| clarkb | frickler: not sure if you want to send that email as you probably have a better grasp on the situation or draft something and I can send it or just help me draft something | 14:54 |
| clarkb | re low TTLs I think people are under the assumption that is always better because it is more nimble, but it also means more opportunity for failed lookups | 14:54 |
| fungi | and also more load on resolvers | 14:59 |
| frickler | clarkb: the change of email addresses is what I can see in the log, I do see the new one in the current kolla config and I don't think that we touched that file | 15:09 |
| frickler | I can try to draft a mail later or maybe tomorrow, need to chair the TC session today so will be a bit busy with preparing | 15:10 |
| clarkb | frickler: got it so likely something they changed. I guess the email should be along the lines of "The letsencrypt provisioned cert from the initial deployment is going to expire soon (I can check the actual timeframe before sending). We noticed some recent updates to the kolla config around this, but it seems to still not be getting reissued. Is there anything we should be doing to | 15:10 |
| clarkb | help with this process?" | 15:10 |
| clarkb | frickler: thanks! | 15:10 |
| frickler | clarkb: maybe also mention the error I pasted, but then that already sounds good to go I'd say | 15:11 |
| clarkb | frickler: the error about registering the account? sounds good I can send that out after my morning meeting | 15:11 |
| frickler | yes | 15:12 |
| clarkb | email sent | 15:47 |
| fungi | the foundation just got confirmation back from the registrar that the openstack.org dns change has been put in | 16:20 |
| fungi | whois has updated now | 16:20 |
| fungi | afilias still has rackspace's nameservers, but with a 1-hour ttl so hopefully won't take too long when it updates | 16:23 |
| fungi | also ramon from openmetal already replied | 16:27 |
| clarkb | oh nice sounds like they think things should provision todayish | 16:29 |
| fungi | i just got a pair of cloudflare ns records back from b2.org.afilias-nst.org | 16:35 |
| fungi | and my local resolver is returning them now as well | 16:36 |
| clarkb | now resolve soemthign you don't already haev cached | 16:37 |
| clarkb | I too am getting back cloudflare NS records and swift.openstack.org resolves for me | 16:38 |
| clarkb | I guess I don't know what my dns server has cached though | 16:38 |
| clarkb | (for the ns records) | 16:38 |
| frickler | and I get an error from chromium for https://openstack.org/ :( | 16:47 |
| fungi | frickler: yes, we confirmed that with the folks managing it and they're putting a workaround in place | 16:47 |
| fungi | should go back to normal momentarily | 16:47 |
| fungi | thankfully it only serves a permanent redirect under normal circumstances, so hopefully most people have that cached in their browsers and won't hit it in the near term | 16:47 |
| frickler | redirect to what? www.o.o isn't working either. "too many redirects" | 16:48 |
| fungi | to that, and yeah it was working moments ago but maybe they switched it backwards | 16:49 |
| fungi | yeah, seems the attempted fix is what has now created the redirect loop there | 16:50 |
| fungi | frickler: looks like they got it worked out, try again? | 16:54 |
| frickler | yep, looking better now | 16:56 |
| fungi | thanks for confirming! | 16:56 |
| SvenKieske | don't forget to check different dns caches, e.g. via https://dnschecker.org/#A/openstack.org dns changes need most of the time more time than pure TTL to propagate | 16:56 |
| fungi | yeah, we're/they're avoiding making unrelated dns changes for a while until the ns delegation settles out | 16:57 |
| SvenKieske | good luck with the work going forward :) | 16:58 |
| fungi | thanks! luckily from an opendev perspective the only important bits for us are mostly just cnames into domains we operate the nameservers for | 16:59 |
| clarkb | there are a couple exceptions to that but ya impact to us should be limited | 16:59 |
| fungi | except for afs/kerberos which is still in openstack.org because moving domains is challenging | 16:59 |
| fungi | and with the funky (from a typical dns perspective) records involved in those protocols we should definitely keep an eye out for oddities in case something in the import wasn't quite right | 17:00 |
| opendevreview | Merged opendev/zone-opendev.org master: Add DNS for new Vexxhost mirrors https://review.opendev.org/c/opendev/zone-opendev.org/+/925437 | 19:07 |
| opendevreview | Merged opendev/system-config master: Add Noble nodes to system-config-run testing https://review.opendev.org/c/opendev/system-config/+/925447 | 19:24 |
| opendevreview | Merged opendev/system-config master: Track our OpenMetal environment HTTPS cert expiry https://review.opendev.org/c/opendev/system-config/+/926488 | 20:05 |
| clarkb | tonyb: I made one minor update to the wiki announcement etherpad (changed a '.' to a ' ' on line 22) otherwise this looks good to me | 20:07 |
| clarkb | we probably don't need to get into that level of detail for the upgrade process but I don't think it hurts either | 20:07 |
| tonyb | thanks. | 20:09 |
| tonyb | I'll refresh the reviews, with some questions on the reviews. from there we can think about timing, and send the announcement. | 20:10 |
| clarkb | sounds good. I'm going to pop out on a bike ride now. Back in a bit | 20:13 |
| tonyb | enjoy | 20:14 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!