Wednesday, 2024-01-17

NeilHanlonhm. that's a tough one if using rsync. `dnf reposync` has a '--newest-only' feature which only keeps the most recent NEVRA in a repo when syncing it, but, that has some downfalls too01:10
tonybclarkb: I think the update went okay:01:16
tonyb 0 s:CN = openinfraci.linaro.cloud01:16
tonyb   i:C = US, O = Let's Encrypt, CN = R301:16
tonyb   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA25601:16
tonyb   v:NotBefore: Jan 16 22:54:35 2024 GMT; NotAfter: Apr 15 22:54:34 2024 GMT01:16
Clark[m]tonyb: great I will check it shortly. Did the how to on the server make sense? I'm glad I wrote that last time I did this01:22
tonybYeah the doc was good.01:23
clarkb`openssl s_client -connect` lgtm as well01:24
NeilHanlonalso looks good for me fwiw01:28
NeilHanlonclarkb: might be of interest, to prune the repo. this is what I was trying to remember. 01:29
NeilHanlonessentially, run this against each repo in some manner, prune the old packages, and then run 'createrepo' on them.01:30
NeilHanlonthough, this means those paths would also have to be added to an excludelist when pruned, so they are not re-synced01:30
Clark[m]NeilHanlon: thanks that gives me something to look at02:13
opendevreviewOpenStack Proposal Bot proposed openstack/project-config master: Normalize projects.yaml
opendevreviewMerged openstack/project-config master: Add weblate api key in zuul secret
opendevreviewMerged openstack/project-config master: Normalize projects.yaml
opendevreviewTakashi Kajinami proposed openstack/project-config master: Retire heat-cfnclient: End Project Gating
opendevreviewTakashi Kajinami proposed openstack/project-config master: Retire heat-cfnclient: Remove Project from Infrastructure System
opendevreviewTakashi Kajinami proposed openstack/project-config master: Retire heat-cfnclient: Remove Project from Infrastructure System
*** benj_2 is now known as benj_08:41
*** dhill is now known as Guest1443513:36
seongsoochoHi team. I'm working on a migrate i18n translation tools from zanata to weblate. Currently, only openinfra accounts can be registered as users in weblate.  I am curious if the email that zanata previously used to call the Zuul API is also an openinfra account. 14:09
fungiseongsoocho: i'll check, hold on14:12
fungiseongsoocho: yes, it is:,all_affiliations&relations=affiliations,groups&filter[]=email==zanata@openstack.org14:13
seongsoochookay. Then, can I receive the password for that account by email? 14:14
fungiseongsoocho: i'll have to figure out where messages get routed for that address but there's a good chance i have access to it and can help there14:15
seongsoochothanks. It's not urgent. 14:16
fungiseongsoocho: it's an alias for our infra-root inbox, so yes i have access to any messages sent to that address14:17
fungiseongsoocho: also, it looks like i have a record of what the password for that account was at one time, it may still be the same14:21
fungitesting it out now14:21
Clark[m]Is that the account we use in zuul to push to zanata? If so changing that password may break those jobs14:22
fungiseongsoocho: the password we have on file for that login still works at, i just tested it14:23
seongsoochoThat's a good news14:23
fungiare you trying to create a zuul secret containing it? for what repo?14:23
seongsoochofor i18n 's new translation platform "weblate"14:24
fungiseongsoocho: i get that, but what are you trying to do with the password? create a zuul secret for use in jobs (in which case you might be able to reuse the existing one unless you're putting it in a different repo), or something else?14:25
seongsoochoOr you can just give me an api key of zanata user from weblate .
ianychoiI believe seongsoocho's intention is to exactly match to an account's secret which has openstackid for weblate login. Another way would be to create another alias like with openstackid, create an user on, and update the secret14:26
fungiand can't be authenticated via
seongsoochoYou can auth via and I want to make an api key for account.14:28's only authentication method is now via seongsoocho needs CI user's credential which push/pulls from Weblate to openstack repos14:29
ianychoi(Thanks to Brian's development!!)14:29
fungioh, you want to use the account to log into interactively, and create a weblate api key using that account, then create a zuul secret that contains that api key14:30
ianychoiExactly (and from my side I am not insisting to stay to use It can be another alias like infra@ or weblate@ ...)14:31
fungiClark[m]: ^ do you think we should reuse the existing openinfraid account for zanata, or create a new one for weblate? i can probably add the new alias with my admin access to the foundation mail hosting for the domain14:32
Clark[m]Reusing may be confusing long term. Like when we had zuul commenting as Jenkins in Gerrit. I think if we can avoid that it would be best14:35
fungiokay, i'll add a weblate@ alias and create an account in openinfraid for that14:36
fungiseongsoocho: ianychoi: infra-root: i've created a dedicated account for (password added to our list) and an e-mail alias for that forwarding to our infra-root inbox; i've also created an account on with that id and generated a rest api key there which i've added to our list. working on the zuul secret for it now14:48
Clark[m]Thank you!14:51
*** blarnath is now known as d34dh0r5314:56
opendevreviewJeremy Stanley proposed openstack/project-config master: Update secret for Weblate REST API key
fungiseongsoocho: ianychoi: ^ is that what you wanted?14:58
ianychoiYep that's whey I understand from today I18n meeting thank you fungi! I think seongsoocho is now sleeping :p15:00
fungii understand. i would be too!15:00
fungianyway, if either of you need anything else for this, just let us know15:00
ianychoiSure really appreciate it!15:01
seongsoochothanks fungi !!!!!15:01
fungiany time15:01
ianychoi(Oh he is not sleeping LoL)15:01
fungiand sorry it took me a few minutes to wrap my head around what you were asking for ;)15:01
seongsoochoit's okay.  I should have gone into more detail. Thank you for doing the favor so quickly. 15:06
fungiit was my pleasure, as always15:07
ianychoiNo worries I personally appreciate seongsoocho's help a lot, while I am still lack of lots of knowledge on the infra and opendev. I am seeing the secret part on the repo for the first time :p15:07
opendevreviewElod Illes proposed openstack/project-config master: Add puppet-ceph-release right for special stable branch handling
*** gthiemon1e is now known as gthiemonge16:41
opendevreviewJeremy Stanley proposed opendev/system-config master: Switch from legacy to new style keycloak container
opendevreviewJeremy Stanley proposed opendev/system-config master: Switch from legacy to new style keycloak container
clarkbautomated cert refreshing seems consistently broken so we'll need to dig into that. and the linaro cloud are both happy now19:15
clarkbjudging by log fiel timestamps on mirror02.ord the last time ran there was Jan 1319:41
clarkbwhich was right before it needed to renew19:42
clarkbthe timing is kind of amazingly good there19:42
clarkband that was the last time the job ran19:42
clarkbinfra-prod-base is failing19:44
clarkbwhich causes the other jobs run daily to be skipped19:44
clarkbcodesearch ran out of disk and that breaks ansible's ability to run the base playbooks on codesearch19:46
clarkbI'm digging into why codesearch is sad now19:46
clarkb/var/lib/hound/data is 34GB and / is 40GB large19:49
clarkbis an appropriate action here to stop the service (this might be difficult because docker can't run without disk space. Maybe we can compact journals) delete the contents of that dir then start the service again after a reboot and let it rebuild that data?19:49
clarkbfungi: ^ do you know?19:59
fungii wouldn't consider the timing amazing. when that job starts failing, the servers we'll get notified about soon thereafter are the ones that were just on the cusp of needing to get their certs refreshed20:02
fungiclarkb: not really sure what the ideal remediation for hound is, but i'll see if there's anything in my shell history. we should be able to add it to the emergency disable list to get deploy jobs running again though20:04
clarkbgood idea. Though we only need to do that if this persists through this evening since the periodic jobs queue at 02:00 UTC20:05
fungiclarkb: it looks like in the past i used `journalctl --vacuum-size=500M` to free up some space...20:05
clarkbya that part should be fine. Its more of a what do we do about the 34GB of codesearch data20:06
fungiand then downed and upped the container20:06
fungioh, i blew away /var/lib/hound/data/20:06
clarkbaha ok so the process I describe above is the one to use20:06
fungiwell, `sudo rm -rf /var/lib/hound/data/*` anyway20:06
fungibefore starting the container20:06
clarkbI'll start on that process. Thank you for checking20:07
fungibut also reboot20:07
clarkb++ to reboot after clearing things up and before starting services20:07
fungisince the rootfs filled up, can't really trust anything that was running after that occurred20:07
clarkbI'm going to delete the idx- and vcs- dirs separately so I can get a sense for the data distribution here20:08
clarkbindexes are about 1.9GB20:09
clarkband vcs-* was about 32GB20:10
clarkbrebooting momentarily20:10
clarkbback and the service is starting up. I believe this will take about 10 minutes?20:12
fungihopefully. i don't recall, honestly20:12
clarkbits filling data/ back up again at least. I'm watching it log the repos it is processing in the container log file20:15
clarkbit took closer to 19-20 minutes. The service is back up again20:30
clarkbthere is 28GB of disk available. I guess those vcs dirs must be leaky20:30
clarkbthe indexes could be leaky too but they were too small to account for that size delta20:31
fungilooks like our cacti tracking for that filesystem only extends back to october...
fungidefinitely looks like it could warrant a separate cinder volume for the data though20:34
fungior use the ephemeral disk20:34
clarkbya then we won't have to claer it out as often20:35
clarkbfrickler: ^ fyi I think this should enable the LE cert refresh job to run now. And if all goes well the cert alerts should go away afterwards20:36
* clarkb finds lunch20:36
frickleroh, so a totally not LE related failure source, nice21:00
fungiyeah, something failing a task on the base job has been the reason for certs not updating at least 95% of the time21:08
tonybwhen I build a new hound server I can add an additional disk for that data.22:38
fungithat's an option, but if we build it in rackspace we can just put it on the ephemeral disk (and the same can be done with the existing server for that matter)22:43
tonybfungi: Oh, I didn't realise we always got an ephemeral disk in rax.23:08
fungiyeah, it comes by default, 80gb i think?23:11
fungiwe wouldn't normally use it for important data, but since this is just all cache really it's not important23:11

Generated by 2.17.3 by Marius Gedminas - find it at!