Thursday, 2022-05-12

fungigood luck on your talk!00:00
clarkboh yes good luck. I thought about trying to participate there, but the timing just wasn't good. The inperson but not inperson wouldn't work for me :)00:00
*** dviroel|afk is now known as dviroel00:00
ianwhrm, i'm guessing the ppa didn't like me adding ~ppa0 as it probably doesn't sort later than the version already there00:03
ianwfor vhd-util00:04
ianwthis is my fault for not uploading it with this tag in the first place00:04
johnsomOk, so I know the issue with the unbound logs being empty on centos/rhel. Maybe you folks might have an idea of how to solve it.00:13
johnsomSo, this
johnsomPuts the unbound.log in two different places depending on the OS.00:13
johnsomThe filename is the same however. So, when the devstack job comes around, I can't just add the other path to zuul_copy_output because the empty one will copy over the top of the correct one:
johnsomIt doesn't look like zuul_copy_output has any merge or rename options.00:16
johnsomMaybe this "null" option could help? I just don't fully understand how it works:
opendevreviewIan Wienand proposed opendev/infra-vhd-util-deb focal: Retrigger upload
opendevreviewMerged zuul/zuul-jobs master: add-gpgkey: trust incoming key
*** dviroel is now known as dviroel|out00:24
opendevreviewMerged opendev/infra-vhd-util-deb focal: Retrigger upload
johnsomMaybe the best answer is to stop that nodepool element from creating the /var/log/unbound.log empty file.00:29
johnsomOr just set them both to store it under /var/lib/unbound/unbound.log00:37
*** dviroel|out is now known as dviroel00:59
ianwjohnsom: yeah, i think that is a limitation of zuul_copy_output :/01:05
ianwdid you say it was something about selinux and centos not storing it under /var/log/unbound.log?  that also might be the sort of thing we merged in centos6 era and might be different now01:06
johnsomianw Yeah, the patch that added the path change mentions selinux:
johnsomIt is old....01:08
*** rlandy|bbl is now known as rlandy|out01:09
ianwyeah, that feels like the type of thing that might have changed.  not 100% sure how to gate test that though01:09
johnsomYeah, same here. No idea how to get all the parts together when it's a nodepool element.01:09
ianwyeah it's a bit of a manual process because once we merge it goes into the next build01:10
ianwwe could probably just put a node on hold and fiddle manually to see what works01:10
johnsomI need to go make dinner now, so limited availability. I will check the channel log tomorrow though if folks have thoughts on the best approach.01:11
johnsomYeah, holding a node to try moving the log path would probably be the best approach. If it can open the file in /var/log/unbound.log we know whatever issue existed doesn't anymore.01:12
ianwok, i need to eat lunch :)  i'll take a look01:12
johnsomIf we don't force a path it goes in journald, not sure if there is good tooling to pull those logs out that we should just switch to using journald01:12
johnsomianw Thank you! Drop me a note if there is something I can pick up and continue to work on tomorrow.01:13
ianwyeah it might be better to just dump a "journalctl -u unbound" type thing01:13
opendevreviewMerged opendev/infra-openafs-deb jammy: Add build stamp to push to production PPA
opendevreviewMerged opendev/infra-openafs-deb bionic: Add build stamp to push to production PPA
opendevreviewMerged opendev/infra-openafs-deb focal: Add build stamp to push to production PPA
*** ysandeep|rover|out is now known as ysandeep|rover01:19
opendevreviewMerged opendev/infra-openafs-deb xenial: Add build stamp to push to production PPA
*** dviroel is now known as dviroel|out01:24
*** ysandeep|rover is now known as ysandeep|afk02:26
*** diablo_rojo_phone is now known as Guest45802:44
opendevreviewIan Wienand proposed opendev/system-config master: Add testing for jammy openafs
opendevreviewIan Wienand proposed opendev/system-config master: [dnm] holding some centos nodes
*** ysandeep|afk is now known as ysandeep|rover04:44
*** soniya is now known as soniya|ruck05:04
opendevreviewIan Wienand proposed openstack/project-config master: Set context for unbound.log on selinux systems
*** ysandeep|rover is now known as ysandeep|rover|brb05:57
ianwjohnsom: is a 8-stream, is a 9-stream with ^ manually setup.  i think it's easier to keep it in the same location and should fix the collection issue?06:12
*** ysandeep|rover|brb is now known as ysandeep|rover06:15
*** ysandeep|rover is now known as ysandeep|rover|brb07:50
*** ysandeep|rover|brb is now known as ysandeep|rover08:03
*** Guest458 is now known as diablo_rojo_phone08:03
*** tweining|off is now known as tweining08:12
opendevreviewDmitry Tantsur proposed openstack/diskimage-builder master: Switch to the CentOS 9 IPA job
*** jpena|off is now known as jpena09:44
*** ysandeep|rover is now known as ysandeep|rover|lunch10:03
*** rlandy|out is now known as rlandy10:20
opendevreviewDmitriy Rabotyagov proposed openstack/diskimage-builder master: Adopted dkms element to work on Ubuntu Jammy and nvidia drivers
*** ysandeep|rover|lunch is now known as ysandeep|rover10:34
*** soniya29 is now known as soniya29|ruck11:08
*** soniya29|ruck is now known as soniya29|ruck|brb11:10
*** dviroel_ is now known as dviroel11:33
*** ysandeep|rover is now known as ysandeep|rover|brb12:01
*** soniya is now known as soniya|ruck12:21
*** ysandeep|rover|brb is now known as ysandeep|rover12:28
opendevreviewMerged openstack/diskimage-builder master: Switch to the CentOS 9 IPA job
*** marios_ is now known as marios13:17
slittleCan we get eyes on
slittleah... perhaps that should be opendev/project-config13:32
fungislittle: no, it's correct13:33
fungiwe're still in the (lengthy) process of moving all that stuff to the newer opendev project namespace13:33
fungibut untangling it from other things we need to wind down in the openstack namespace is taking a while13:34
fungiin part because it's also a move to a separate zuul tenant, so some things can't go until related repositories switch to the other tenant13:35
slittleList of cores for starlingx-app-sriov-fec-operator-core: Teresa Ho, Greg Waines, Steve Webster, Cole Walker13:39
opendevreviewMerged openstack/project-config master: Add SRIOV FEC Operator app to StarlingX
*** ysandeep|rover is now known as ysandeep|rover|mtg14:00
fungislittle: i've found and added all four of them. in the future, it's easier for us to just add one person and then let them add the others (since these groups are self-managed by default)14:31
clarkbinfra-root I'm lurking the TC meeting then will reboot for some updates and then my plan is to restart zuul-web instances15:41
clarkbthis should hopefully fix the scroll to log line issue with current zuul web15:41
fungii'm around to help15:41
clarkbits a relatively minor thing but something that I hit constantly as I look at job logs so I'm selfishly looking forward to having my browser scroll for me :)15:42
fungiyeah, it comes up for me a lot too. i end up scrolling thousands of lines of log to spot the one that's a slightly different color15:43
clarkbalso I'll be listening in on the gerrit hackathon which is running over PDT more than I expected it to15:44
opendevreviewMerged openstack/project-config master: Set context for unbound.log on selinux systems
clarkbjohnsom: ianw ^ that will need iamge rebuilds though15:52
*** ysandeep|rover|mtg is now known as ysandeep|rover15:56
johnsomclarkb Right, thanks! Fingers crossed for tomorrow16:00
clarkbjohnsom: is thee a particular centos flavor that would be more helpful to get signal on quicker? I can manually trigger a build for that one now16:01
clarkb*is there16:01
johnsomclarkb C9s, but I am fine waiting until tomorrow too. I have C9s test jobs I was using yesterday16:01
clarkbjohnsom: I'm impatient and want signal more quickly :)16:02
johnsomFair enough, ping me and I can run the job for confirmation16:02
*** marios is now known as marios|out16:05
clarkbits queued behind a couple of other builds now but should happen soon16:05
*** soniya|ruck is now known as soniya|out16:11
*** ysandeep|rover is now known as ysandeep|rover|out16:28
clarkbI'm going to restart zuul-web on zuul01 now17:11
clarkbthen when it is up again do zuul-web on zuul0217:12
opendevreviewMerged zuul/zuul-jobs master: Switch enable_src_repos to False in configure-mirrors
fungigotta say, from an operations standpoint i *love* the page17:17
clarkbfungi: we should be able to land whenever we are ready to do the cleanups in the repo by hand17:17
fungiclarkb: did you restart the fingergw on zuul01 as well? it's reporting the new version17:18
clarkbfungi: yes they share a docker-compose file so both got down and up'd17:18
clarkbI didn't realize ti when I did 01 but I'll be sure to treat 02 the same way17:19
fungiaha, perfect. just making sure the component api wires weren't crossed17:19
clarkbya I don't think it is a problem but it surprised me too. Not a big deal17:19
clarkbzuul-web debug log doesn't give much indication of its progress in updating system config, but I'm guessing that is normal? cc corvus 17:20
*** jpena is now known as jpena|off17:21
clarkbzuul-web does seem to be using a number of cpu cycles so I should be pateitn I guess17:26
clarkbit is logging about all the config files it is loading now17:33
clarkbI expect it will be done very soon17:33
clarkbjohnsom: ianw: dib doens't like the new selinux context setting17:34
clarkbfungi: zuul01 is up now17:34
clarkbI'm talking to it. Going to test the log viewing17:34
clarkbit works! I'm so happy I managed to track that down17:35
clarkbI'll give it a couple minutes for any problems to show up then restart zuul02's web and fingergew17:35
johnsomclarkb ack, interesting... "chcon: can't apply partial context to unlabeled file '/var/log/unbound.log'" I will pull down the patch and see if I can figure out what chcon wants.17:36
clarkbjohnsom: thanks17:36
johnsomThat file should probably be labeled17:36
clarkbas far as I can tell zuul-web is happy. Proceeding with 02 now17:40
fungiyep, seems to be working for me17:49
clarkb#status log Updated zuul-web and zuul-fingergw to 6.0.1.dev14 60e59ba67. This fixes scrolling to specific line numbers on log files.18:05
opendevstatusclarkb: finished logging18:05
fungithanks! so much more convenient that it will scroll to the line number anchors now18:06
clarkbI tested it on and it handled that relatively large log and the deeper link than some of the more trivial stuff that it was testing on pre merge18:07
clarkbbut ya seems to work and I'm happy18:07
clarkbfungi: I rechecked so that it has a +1 when we are ready to prune the mirrors18:19
clarkbit is not longer approved so it won't land but I think we can approve that whenever someone is ready to do the reprepro cleanups18:19
clarkbspecifically someone has grabbed and held the lock when we approve that I mean18:20
funginot that it would hurt if we grabbed the lock afterward18:20
clarkbya I guess reprepro will just fail and we won't publish anything as a result18:21
fungiwill it fail? we're just removing entries right? i need to re-review the change obviously to double-check18:21
clarkbiirc the first repo we tested with failed18:22
clarkbbut maybe not18:22
clarkbthere were a lot of cleanups and it could've been another thing that caused it to fail18:22
opendevreviewMichael Johnson proposed openstack/project-config master: Fix selinux context for unbound.log
johnsomclarkb ^^^ I think that will fix the error. Though I couldn't reproduce the "partial context" issue locally for some reason.19:25
opendevreviewMerged openstack/project-config master: Retire openstack-helm-docs repo, step 3.3
corvusclarkb: yeah re zuul-web.  i generally check the components page every 10m or so20:39
corvusclarkb: it'll switch from 'initializing' to 'running'20:39
clarkbright, I was mostly just surprised at how quiet it is since the schedulers are pretty verbose. But it all worked out20:39
rosmaitahello ... i'm suddenly seeing a bunch of post failures, can't find pbr 5.9.0 ... are you already aware?
clarkbnews to me20:45
rosmaitasorry to be the one to report it20:46
clarkbpbr probably should be in constraints but that is a separate problem20:46
rosmaitadoes it look like an outdated mirror situation?20:46
clarkbsort of. Pypi is merely proxy cached these days. It asks pypi for that data if the cache ttl is expired or it doesn't haev it already. If the pypi CDN has an error talking to the pypi backend it will talk to a backup backend which will serve often stale data. I looks like that situation20:47
clarkbyou've asked for pbr 5.9.0 which is the lastest version from may 5th. But the index only provided up to version 5.8.020:47
clarkbimplying you got a stale index which was likely caused by a pypi failover to their backup backend20:48
rosmaitagotcha, thanks for explaining that20:48
clarkblooks like 5.8.1 was also missing from the stale index20:48
clarkbwhich means what we got was older than february 6, 2022 :/20:48
rosmaitaso i guess the thing to do is wait a bit and then recheck?20:49
clarkbyes, you can also check as large ongoing issues with pypi tend to make it there (it appears clean right now though)20:49
rosmaitaok, i'll make a note of that so i can troubleshoot better next time20:49
clarkbseparately it is worth noting that openstack is particularly susceptible to this due to the use of constraints and keeping them up to date with more recent releases20:50
clarkbother pip installs will often just install an older (potentially vulnerable version of the package) and continue on20:50
clarkbI wish that pypi would just return an error rather than failvoer and potential give people insecure packages...20:50
fungialso it can be interesting to note which regions the failures happened in. the usual pattern is that it impacts some specific part of the globe because of the cdn pypi relies on, so you get endpoints in, say, the montreal area returning old data while the rest of the world is unaffected20:51
clarkbIt might be worth trying ot have a discussion with pypi about this behavior again. I think for the vast majority of their users and error would be much safer20:53
clarkbMore annoying but safer20:53
fungithey're currently not indicating they're aware anything is out of sorts:
clarkbya the status page doesn't always trip when this trips. But if the status page does trip we tend to see this20:57
*** dviroel is now known as dviroel|afk21:09
ianwjohnsom/clarkb: huh, it must be something to do with creating it in a non-selinux context i guess, then maybe we haven't relabeled it since?  i did "test" this on a live machine21:31
ianwi wonder about the restorecon; *maybe* that restores context from the "global" db and overwrites what we just did?  21:32
johnsomianw Yeah, I couldn't reproduce it locally either. I went ahead and proposed doing the persistent update to the local selinux via semanage which should resolve that error since it relabels the file.21:32
ianwselinux is hard enough to use when you've got it already working :/21:33
ianwdo you think the restorecon might undo that though?21:33
clarkbya I'm mostly wondering if it is easier to use the normal path and have devsatck copy it from the various locations or if it is to manage selinux directly21:33
johnsomrestorecon gives this: "Relabeld /var/log/unbound.log from unconfined_u:object_r:var_log_t:s0 to unconfined_u:object_r:named_log_t:80" as the output of the -v21:34
johnsomHad to transcribe that as it was a console session on that c9s instance21:35
ianwyeah that named_log_t was the key21:35
johnsomWithout semanage stashing it in the local file, chcon would have been undone if something ran a relabel of /var/log21:36
johnsomman semanage-fcontexts for the details of why I went this path21:37
ianwright so it updates the db, which seems right21:38
johnsomclarkb I went down the copy them both path, but the order the zuul copies happened seemed not deterministic and since the nodepool element created a empty file, it ended up getting copied over the top of the one with data. (I was using the zuul config to collect them)21:40
clarkbjohnsom: right yuo have to have devstack copy the file to a singular location and then zuul copies that21:40
clarkbthere are a number of logs dont tht way iirc21:40
johnsomHmm, I didn't see that outside of devstack-gate (which I think we don't use). Just the zuul_copy_output settings that collect them (the problem).21:42
ianwwe didn't revert anything, so 841629 is required to get builds happy again right?21:43
clarkbianw: correct something is required at least21:43
clarkbI'm happy to give 841629 a go21:43
ianwyeah i can watch it21:44
opendevreviewMerged openstack/project-config master: Fix selinux context for unbound.log
*** prometheanfire is now known as Guest022:26
*** rlandy is now known as rlandy|bbl23:02
johnsomianw Let me know if you want me to launch a test job for the log file capture.23:07
ianwlooks like nb02 built a centos-7 with it23:13
ianw2022-05-12 22:55:54.713 | + [[ -e /usr/sbin/semanage ]]23:13
ianw2022-05-12 22:55:54.713 | + semanage fcontext -a -t named_log_t /var/log/unbound.log23:13
ianw2022-05-12 22:56:06.993 | + restorecon -v /var/log/unbound.log23:13
ianwso we just have to  wait for the other builds + uploads23:13
johnsomAh, cool. Yeah, my test case is C9s23:14

Generated by 2.17.3 by Marius Gedminas - find it at!