Wednesday, 2022-06-08

puckIdeally I'd just change the email address in to match my current work email address.00:03
ianwwe probably need to add the openid as an externalaccount00:09
ianwi'm just trying to see if we have any prior history of this.  we might have to loop in clarkb, but he is currently on ~ UTC time 00:18
*** rlandy|bbl is now known as rlandy00:19
ianwahh, ok, i can see a new account created actually now00:20
ianwpuck: it's created a login for you now with an email @catalystcloud.nz00:29
puckYeah, I created that account earlier today before I remembered I already had an identity.00:29
ianwah, and you can log in with that ok?00:30
ianwwhat we want to do then is make 9950 inactive, then you can add the account to this new account00:31
puckOkay, that sounds ideal00:31
ianwpuck: ok, that is done, try adding the email and it should allow you00:36
puckError 422 (Unprocessable Entity): Identity '' in use by another account00:37
puckCan you change the other one to be something like ?00:38
*** rlandy is now known as rlandy|out00:41
ianwpuck: try again now00:44
puckWoohoo, done! Thank you00:48
opendevreviewIan Wienand proposed opendev/system-config master: gerrit docs: cleanup and use shell-session
ianwclarkb: ^ i'll sync with you at another point00:56
ianwbut the gist is that there was an external-id for 9950 "identity":"" which I removed via the api call00:58
ianwit doesn't seem like the duplicate instructions, where we disable the original account, take into account having to remove this external-id?  seems like we'd always have to do that to allow the email to be re-used?00:58
Clark[m]ianw: if you look into system-config/tools/gerrit-something are three scripts. One to collect info on those collisions, another to retire an account (disable it and remove the preferred email) and finally one to delete external ids from accounts. Typically you do need to retire and account then delete external ids in that order to fix these issues03:10
Clark[m]The reason for that is often the preferred email is also the conflicting email address and if you don't remove the preferred email first you create a new db error state for that account where the preferred email has no matching external id03:11
ianwthanks; we might just want to take a look at the duplicates instructions in the admin doc page, i do have a memory of all this now you mention those scripts with the retirement work you've done03:47
opendevreviewIan Wienand proposed openstack/diskimage-builder master: [wip] test f36 builds
opendevreviewIan Wienand proposed openstack/diskimage-builder master: [wip] test f36 builds
*** Tengu_ is now known as Tengu05:56
*** ysandeep|out is now known as ysandeep06:15
fricklerianw: clarkb: fungi: could you please put onto your review list? yet another attempt at making the requirements proposal bot work again06:32
*** ykarel__ is now known as ykarel06:48
opendevreviewMerged openstack/project-config master: propose-updates: Run ensure-python for python 3.9
fricklerianw: now the centos8 wheel builds started failing too. not sure if there actually might be a relation to your dib issue? new kernel or such?
*** jpena|off is now known as jpena07:30
ianwthat looks like openafs isn't installed, that generally means that the kernel the images are running is out of date with the upstream headers07:31
ianwwhich means we have old images07:31
ianwwhich, yes might all tie into centos issues :/07:31
frickleroh, yes,
frickler /opt is full on nb01+02. maybe more symptom than root cause07:35
fricklersadly due to the way nodepool recycels logs it will likely not be possible to look at the original failures though07:36
fricklerwow, seems all our images are two weeks old? guess I should move "make a job to check image freshness" up on my todo so at least we can notice07:39
Clark[m]With that timing jt may have been an openstacksdk 0.99.0 problem? That has been capped now so fixing thr disk issue may be sufficient07:41
fricklermaybe, I'll give it a go now07:48
ianw is a initial bug for the selinux issue07:50
ianwit will need more investigation than i have time for right now 07:51
opendevreviewDr. Jens Harbott proposed openstack/diskimage-builder master: Drop tumbleweed job
*** ysandeep is now known as ysandeep|lunch07:58
fricklerwow, that cleanup sure does take its time. anything else to rm other than /opt/dib_tmp/dib_* ?07:59
Clark[m]That's the main thing. You can also look for leaked intermediate vhd files in /opt/nodepool_dib iirc. Those files are only used in converting from raw to vhd and once that is done they are unneeded. They have a .intermediate file suffix iirc08:03
opendevreviewDr. Jens Harbott proposed opendev/glean master: Drop opensuse jobs
*** anbanerj is now known as frenzy_friday08:30
*** ysandeep|lunch is now known as ysandeep08:44
fricklero.k., c8s built successfully, now waiting to see if the uploads are working too10:01
frickler#status log cleanup up /opt on nb01+2 and rebooted them10:31
opendevstatusfrickler: finished logging10:31
fricklernb01 might need some deeper cleanup at some time
*** rlandy|out is now known as rlandy10:33
frickleriiuc nodepool also keeps around the two most recent copies of each image on each builder? which might be twice as much as we actually need10:33
fricklermaybe have a fixed allocation of images to builders instead? possibly failover if one is down? though that very likely would be a new feature for nodepool?10:34
Clark[m]frickler: two total may be two on one or one and one10:39
fricklerClark[m]: but if nb01 has built two images, then nb02 builds the next, should nb01 actually notice and delete its older copy?10:44
fricklerbecause from my casual look that doesn't seem to happen10:44
Clark[m]Yes it should. There may be some issue preventing that or maybe they leaked?11:05
fricklerthey didn't look too old at least. I'll try to trace some in more detail. maybe best once we're back to two days of stable image builds11:17
*** dviroel|afk is now known as dviroel11:19
frickler170M of setfiles warnings. I think we should disable c9 image builds until ianw can make progress. and possibly also wheel builds for it12:10
frickler#status log paused centos-9-stream image builds due to
opendevstatusfrickler: finished logging12:19
Clark[m]++ to pausing. Thank you for handling that12:22
*** ysandeep is now known as ysandeep|out14:31
*** dviroel is now known as dviroel|lunch14:47
frickleris there a way to actually see which images are paused?15:36
*** marios is now known as marios|out15:47
Clark[m]Maybe with dib-image-list?15:52
Clark[m]I'm not sure15:52
*** dviroel|lunch is now known as dviroel16:05
fbo[m]Hi, is there an issue with the opendev AFS mirror ? The centos-stream 9 mirror seems out of sync.16:37
fbo[m]I don't see libselinux-3.4-2 on but is available since yesterday on
fricklerfbo[m]: seems our upstream mirror is lagging, then:
fricklerevery call of the nodepool cli now seems to log this
rlandyfbo[m]: thanks - following here17:38
*** jpena is now known as jpena|off17:49
rlandyfrickler: is it possible to force update the mirror? 17:59
Clark[m]We sync from Facebook iirc. You should check if our upstream is updated. If not a force won't help anything18:08
dasm|ruckClark[m]: it's there:;O=A  libselinux-utils-3.4-2.el9.x86_64.rpm                              2022-06-07 12:12  183K  18:27
dasm|ruckbut it's missing everywhere else, starting with dfw.rax, then vexx18:27
dasm|ruckClark[m]: fb mirror is lagging:
dasm|ruckeven tho it shows this too:
dasm|ruckrcastillo is the one who pointed that out18:37
dasm|ruckit seems to be fetching from here;O=D18:41
opendevreviewGleb Aronsky proposed openstack/project-config master: Add KubeVirt app to StarlingX
dasm|ruckdoes anyone know how te reach out to fb to update their mirrors?19:00
rlandylooks like the current update is in the morning - we may get an update tomorrow?19:04
dasm|rucklet's have a hope for that19:05
*** dviroel is now known as dviroel|afk20:27
*** dasm|ruck is now known as dasm|ruck|afk21:05
ianwfrickler: yeah, thanks for pausing that, i realised it was probably in a death spiral doing that late last night22:02
ianwthankfully, though, i've woken up to reports that this is fixed by an upstream commit already in
ianwi was getting a little bit worried it might have been something where generically it wouldn't run on !9-stream kernels or something else particularly difficult to work-around22:04
ianwturns out it is a generic problem of running setfiles in chroots without /proc22:05
ianwahh, i see we're interested in the same 3.4-2 libselinux package anyway discussed above22:15
ianwi just tried from the regular mirrors and didn't get that new package.  i'm not really sure but i don't think it is rolled out yet22:15
ianw has "candidate" and "pending" flags22:20
rlandyianw: hi - we see libselinux-3.4-2.el9.x86_64.rpm if that helps22:30
*** rlandy is now known as rlandy|bbl22:42
ianwyeah, i'm not sure what the roll-out process for a package is23:05
ianwodcs seems to be some sort of on-demand-compose-service.  i'm not sure what that is23:16

Generated by 2.17.3 by Marius Gedminas - find it at!