Tuesday, 2022-05-24

opendevreviewJames E. Blair proposed zuul/zuul-jobs master: buildset registry: run socat in new session  https://review.opendev.org/c/zuul/zuul-jobs/+/84303800:29
opendevreviewJames E. Blair proposed zuul/zuul-jobs master: DNM: welfare check: ensure-podman  https://review.opendev.org/c/zuul/zuul-jobs/+/84304802:00
opendevreviewJames E. Blair proposed zuul/zuul-jobs master: ensure-podman: Remove kubic from Ubuntu 18.04  https://review.opendev.org/c/zuul/zuul-jobs/+/84304902:00
opendevreviewJames E. Blair proposed zuul/zuul-jobs master: ensure-podman: Use PPA for Ubuntu 20.04  https://review.opendev.org/c/zuul/zuul-jobs/+/84309302:14
*** ysandeep|out is now known as ysandeep|rover03:24
*** ysandeep|rover is now known as ysandeep|afk04:24
*** ysandeep|afk is now known as ysandeep|rover04:45
opendevreviewyatin proposed openstack/diskimage-builder master: [DNM] Test oooq build image without CentOS preshipped repos  https://review.opendev.org/c/openstack/diskimage-builder/+/79439507:22
*** ysandeep|rover is now known as ysandeep|rover|lunch08:02
*** ysandeep|rover|lunch is now known as ysandeep|rover08:38
opendevreviewyatin proposed openstack/diskimage-builder master: [DNM] Test oooq build image without CentOS preshipped repos  https://review.opendev.org/c/openstack/diskimage-builder/+/79439510:10
opendevreviewPavlo Shchelokovskyy proposed openstack/diskimage-builder master: Fix openssl example command in dynamic-login  https://review.opendev.org/c/openstack/diskimage-builder/+/84312811:13
*** dviroel|out is now known as dviroel11:21
dtantsurI think I'm seeing something weird in zuul. The 'command' ansible task ends up with stderr merged into stdout. Is it something zuul does?11:58
dtantsurI *seem* to recall that, but want to double-check11:58
fungi"merged" as in when you pull of the task detail you see things which should have been part of the sterr stream showing up the stdout box instead?12:10
fungier, pull up the task detail12:10
fungian example would help me undertand what you're seeing12:11
opendevreviewDr. Jens Harbott proposed openstack/project-config master: Add missing volumes to Grafana AFS page  https://review.opendev.org/c/openstack/project-config/+/84313312:16
chandankumarHello #opendev12:30
chandankumarwhat is the sync frequency of syncing stream contents from facebook to opendev mirrors?12:31
fungichandankumar: centos stream you mean? or a stream of something else?12:32
chandankumarfungi: centos stream 912:32
chandankumar*facebook mirror12:32
chandankumarwhat is the sync frequency of centos stream 9 contents from facebook mirror to opendev mirrors?12:33
fungichandankumar: roughly every 4 hours: https://opendev.org/opendev/system-config/src/branch/master/playbooks/roles/mirror-update/tasks/rsync.yaml#L47-L5412:34
fungier, every 6 hours12:34
chandankumarfungi: thanks :-)12:35
chandankumarI was checking this https://grafana.opendev.org/d/9871b26303/afs?orgId=112:35
chandankumarMirror release times shows 6 hours.12:35
chandankumarysandeep|rover: ^^12:35
ysandeep|roverfungi, chandankumar thanks!12:36
fricklermirror.epel shows 3 days, /me goes looking at logs12:36
frickler/afs/.openstack.org/mirror/epel/7/x86_64/repodata/repomd.xml: FAILED12:37
fricklersha1sum: WARNING: 1 computed checksum did NOT match12:37
fricklersource is "rsync://pubmirror1.math.uh.edu/fedora-buffet/epel/7/" , maybe someone wants to propose an alternative12:38
fungithat's an interesting sanity check12:39
frickleralso a good cue to alert config-core to my patch adding some missing mirror volumes to that page https://review.opendev.org/c/openstack/project-config/+/84313312:39
fungiso we're comparing the copy of repomd.xml we pulled via rsync to what's being served from the official fedora mirror i guess?12:39
fricklerwe compare to https://dl.fedoraproject.org/pub/epel/7/x86_64/repodata/repomd.xml , which looks official, yes12:40
chandankumarfrickler: Do we want to switch this https://opendev.org/opendev/system-config/src/branch/master/playbooks/roles/mirror-update/files/epel-mirror-update#L31 to facebook also?12:48
fricklerchandankumar: whatever works, feel free to propose a patch13:03
fungithough whatever you propose, you might want to try to rsync a copy of epel/7/x86_64/repodata/repomd.xml and compare it to what's being served from https://dl.fedoraproject.org/13:05
fungijust to make sure there's not some broader problem13:05
dtantsurfungi: dunno how to link to a task, check the one before the failed task: https://zuul.opendev.org/t/openstack/build/bb018fbbeda14baab8061ac96cc9b48f/console13:33
dtantsurthe first line in stdout comes from stderr13:33
fricklerthere's a permalink button at the top of the task popup https://zuul.opendev.org/t/openstack/build/bb018fbbeda14baab8061ac96cc9b48f/console#3/1/25/controller13:43
fricklerdtantsur: did you try adding "2>/dev/null" to your command?13:44
fungidtantsur: yeah, i agree that's probably an effect of https://opendev.org/zuul/zuul/src/branch/master/zuul/ansible/base/library/command.py#L460-L46713:47
fungior maybe i'm misreading that13:49
fungithe exit json is passing a stderr keyarg still13:50
fungiyeah, i'm i'm reading correctly, stdout and stderr from the command are both interleaved into _log_lines so that it forms the live console log stream, and then the aggregate of _log_lines is passed into the exit json as stdout13:55
fungier, if i'm reading correctly13:55
fungiso this is essentially a side effect of zuul forking the command module in order to hook the stdout and stderr from commands into the live console stream (something vanilla ansible doesn't support)13:56
dtantsurokay, thanks for explanation14:24
fungidtantsur: i'm not immediately seeing why we couple sdterr and stdout rather than interleaving them from separate descriptors so they could still be separately preserved in the json. the follow() function takes care of writing the stream to a separate file anyway14:30
fungimaybe someone else remembers why that was necessary14:31
fungii'm guessing it's something to do with needing to make sure they get ordered and timestamped correctly when you're getting buffered output14:32
fricklermaybe zuul could offer an unstreamed_command module for such usecases14:37
opendevreviewMerged openstack/project-config master: Add missing volumes to Grafana AFS page  https://review.opendev.org/c/openstack/project-config/+/84313314:41
clarkbfungi: you acn't separate them without losing information. specifically interleaving/timing info14:41
clarkbI think the idea was you can separate them yourself if necessary14:41
fricklerclarkb: how would you do that in the general case? although in the current scenario maybe using instance_info.stdout_lines[-1] instead of .stdout might work14:45
clarkbfrickler: command 2> record_stderr14:46
clarkbor invert it depending on which aspect you are interested in14:46
clarkbessentialy zuul has made a compromise to collect the least lossy information by default. but that does mean you need to take action if you want it to be lossy14:47
fricklerclarkb: ah, sure, I though you were saying that one could separate somehow from what zuul collects14:47
fungiyeah, i think zuul is doing the rough equivalent of 2>&1 so that the system will perform the interleaving more accurately than zuul itself would be able to after the fact when trying to mash two separate streams together14:50
fricklerfungi: clarkb: my propose-updates fix would still like to get reviewed if you have time https://review.opendev.org/c/openstack/project-config/+/84285314:51
opendevreviewMerged openstack/project-config master: Add neutron-vpnaas-stable-maint group  https://review.opendev.org/c/openstack/project-config/+/84269315:11
*** ysandeep|rover is now known as ysandeep|out15:17
opendevreviewMerged openstack/diskimage-builder master: Fix openssl example command in dynamic-login  https://review.opendev.org/c/openstack/diskimage-builder/+/84312815:46
*** dviroel is now known as dviroel|lunch15:58
*** marios is now known as marios|out15:58
clarkbfungi: you mentioned being willing to install the wiki cert when it is available. It is now available and in the typical location. Let me know if you can't find it16:18
fungithanks! i can take care of it after i get off this meeting and grab a bite to eat16:19
mnasero/ how can one request the deletion of a stale feature branch (feature/lbaasv2 on neutron-vpnaas), last change was in.. 201416:21
clarkbmnaser: the openstack release team has permissions to manage branches in poenstack projects now16:22
clarkb(new gerrit allowed us to delegate that which is awesome)16:22
mnaserclarkb: oh cool, so i'll ask there then16:22
clarkbwell to be clear they could always add branches but now they can delete them too16:22
*** jpena is now known as jpena|off16:51
*** dviroel|lunch is now known as dviroel16:58
opendevreviewJames E. Blair proposed zuul/zuul-jobs master: ensure-podman: Remove kubic from Ubuntu 18.04 and drop 20.04  https://review.opendev.org/c/zuul/zuul-jobs/+/84309317:16
opendevreviewJames E. Blair proposed zuul/zuul-jobs master: buildset registry: run socat in new session  https://review.opendev.org/c/zuul/zuul-jobs/+/84303817:16
opendevreviewJames E. Blair proposed zuul/zuul-jobs master: ensure-podman: Remove kubic from Ubuntu 18.04 and drop 20.04  https://review.opendev.org/c/zuul/zuul-jobs/+/84309317:46
opendevreviewJames E. Blair proposed zuul/zuul-jobs master: buildset registry: run socat in new session  https://review.opendev.org/c/zuul/zuul-jobs/+/84303817:46
fungithe last entry in statusbot's debug log is 2022-05-17 16:23:4117:48
fungithe daemon is still running though17:49
fungi2022-05-17 16:29:46     <--     opendevstatus (~opendevst@eavesdrop01.opendev.org) has quit (synthon.oftc.net reflection.oftc.net)17:50
fungips says it's been running since april 20 though17:50
funginothing useful in the docker logs either17:52
fungi#status log Restarted statusbot container on eavesdrop01 since it appeared to disconnect from the IRC server around 2022-05-17 16:30 UTC and did not reconnect on its own (but also logged nothing to explain why)17:54
opendevstatusfungi: finished logging17:55
clarkbcorvus: re the nodepool changes remember launchers don't auto update (builders should)17:59
opendevreviewMerged openstack/project-config master: Fix python list for requirements propose-updates job  https://review.opendev.org/c/openstack/project-config/+/84285318:18
jrosserif i have a job that passes on on 2nd or 3rd attempts, would those attempts be (potentially) in a different provider from the ones that failed18:26
jrosseri'm wondering if i have something broken with setting up the extra volume on some providers (rax?)18:28
fricklerjrosser: yes, that's possible. you should be able to see the retried jobs in the build results to verify18:29
jrosseroh, it keeps the failed ones?18:29
fricklerjrosser: do you have a link to a failure?18:29
jrosserheres one that passed on the second attempt https://zuul.opendev.org/t/openstack/build/5d6068be71af4ae1808d5d3e25aee6c418:30
jrosserbut i'm kind of wondering why the first failed18:30
jrosserah is that what "view build history" is showing18:31
fricklerjrosser: if you look at the "build history" link, you can see the "retry"18:31
jrosserexcellent, just what i need18:31
fricklerlooks like rax vs. inmotion indeed18:33
jrosseri think i've a bug which is only triggered when setting up the extra disk18:33
fricklerclarkb: it seems you missed my earlier comment about the project.opendev volume. I don't think it really needs 1TB of quota? https://grafana.opendev.org/d/9871b26303/afs?orgId=1&from=now-7d&to=now&viewPanel=6818:48
clarkbfrickler: what path is that mounted at? I wonder if it is hosting tarballs.opendev.org ?18:53
clarkbIn which case it might need some quota, but I agree 1TB seems excessive18:53
clarkbmaybe we reset it to 10GB18:54
clarkbI can look into that later today18:54
fungithe project/opendev.org mount is indeed separate from project/tarballs.opendev.org18:58
fungithough there is a tarballs.old in it18:58
clarkbin that case 1GB is probably plenty18:59
clarkbthat is about 10x the current usage19:00
clarkbI went ahead and set it to 1GB19:39
fricklerianw: I think I did set up the jammy wheel volumes now correctly. if you can double check and then go on that would be great19:58
ianwfrickler: will do20:00
ianwthanks, acl's look right, so all lgtm20:04
opendevreviewClark Boylan proposed openstack/project-config master: Install setfacl on test nodes  https://review.opendev.org/c/openstack/project-config/+/84321020:05
clarkbfrickler: infra-root ^ turns out this is one of those packages where almost everyone agrees on the package name which is nice20:05
ianwmaybe it should be a dib thing too?  i've managed to be using linux for about 30 years and never once to my knowledge manually called setfacl afaik20:09
clarkbianw: maybe? I feel like the ansible situation is a bit of a corner case but one likely in a CI system20:10
fungiit's taken a while to catch on20:10
clarkbor at least CI system that uses ansible20:10
opendevreviewMerged openstack/project-config master: Build and publish wheel mirror for Ubuntu Jammy  https://review.opendev.org/c/openstack/project-config/+/84284120:13
ianw"The acl package is a dependency of systemd, it should already be installed. "20:13
ianwi wonder if it's a reccommendation that we're not pulling in because of ... something (hand wavy something about debootstrap, etc)20:13
ianwseems to require libacl1, but maybe not "acl"20:15
ianwthe fault in my thinking is that you'd want to do anything on your system outside of having systemd do it for you.  clearly crazy :)20:17
fungisystemd must have its own facl entrypoint20:17
ianwstill, given that it falls under "makes system work with ansible" adding it to even minimal dib images i think would be fine20:19
ianwhttps://bugs.launchpad.net/ubuntu/+source/acl/+bug/1913666 similar sort of bug about it not being in base images20:21
opendevreviewJames E. Blair proposed zuul/zuul-jobs master: DNM: test buildset registry jobs for bitrot  https://review.opendev.org/c/zuul/zuul-jobs/+/84321220:31
opendevreviewMerged openstack/project-config master: Install setfacl on test nodes  https://review.opendev.org/c/openstack/project-config/+/84321020:34
*** dviroel is now known as dviroel|out21:11

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!