Tuesday, 2022-02-22

opendevreviewIan Wienand proposed openstack/diskimage-builder master: bootloader: clean up EFI checking  https://review.opendev.org/c/openstack/diskimage-builder/+/83029200:08
clarkbianw: for ^ that was just a linter fixup?00:09
clarkbmeeting agenda is out00:10
ianwyep, for dib-lint's inability to deal with \ continuation indentation :/00:10
jentoioclarkb: tomorrow is fine. thanks00:35
*** rlandy|ruck|bbl is now known as rlandy|ruck00:35
opendevreviewMerged opendev/system-config master: run-production-playbook: fix path typo  https://review.opendev.org/c/opendev/system-config/+/83029400:42
*** rlandy|ruck is now known as rlandy|out00:44
opendevreviewIan Wienand proposed opendev/system-config master: run-production-playbook : fix synchronize src/dest arguments  https://review.opendev.org/c/opendev/system-config/+/83029700:54
opendevreviewIan Wienand proposed opendev/system-config master: run-production-playbook : fix synchronize src/dest arguments  https://review.opendev.org/c/opendev/system-config/+/83029700:54
opendevreviewSteve Baker proposed openstack/diskimage-builder master: rhel: work around RHEL-9 BLS issues  https://review.opendev.org/c/openstack/diskimage-builder/+/82962001:09
opendevreviewSteve Baker proposed openstack/diskimage-builder master: Detect boot and EFI partitions in extract-image  https://review.opendev.org/c/openstack/diskimage-builder/+/82861701:09
opendevreviewMerged opendev/system-config master: run-production-playbook : fix synchronize src/dest arguments  https://review.opendev.org/c/opendev/system-config/+/83029702:04
*** pojadhav- is now known as pojadhav|ruck02:53
opendevreviewMerged openstack/diskimage-builder master: bootloader: fix arm64 install path  https://review.opendev.org/c/openstack/diskimage-builder/+/83011103:17
opendevreviewIan Wienand proposed opendev/system-config master: run-production-playbook: fix stray }}  https://review.opendev.org/c/opendev/system-config/+/83030403:19
Clark[m]I wonder why the linter didn't catch those04:06
opendevreviewMerged opendev/system-config master: run-production-playbook: fix stray }}  https://review.opendev.org/c/opendev/system-config/+/83030404:25
*** ysandeep|out is now known as ysandeep04:47
opendevreviewIan Wienand proposed opendev/system-config master: run-production-playbook: fix another stray }}  https://review.opendev.org/c/opendev/system-config/+/83031105:58
opendevreviewIan Wienand proposed opendev/system-config master: encrypt-logs: fix a couple of download script issues  https://review.opendev.org/c/opendev/system-config/+/83031206:11
opendevreviewIan Wienand proposed opendev/system-config master: encrypt-logs: fix a couple of download script issues  https://review.opendev.org/c/opendev/system-config/+/83031206:11
opendevreviewMerged opendev/system-config master: run-production-playbook: fix another stray }}  https://review.opendev.org/c/opendev/system-config/+/83031106:52
opendevreviewMerged opendev/system-config master: encrypt-logs: fix a couple of download script issues  https://review.opendev.org/c/opendev/system-config/+/83031207:08
*** amoralej|off is now known as amoralej07:34
opendevreviewMerged openstack/diskimage-builder master: Add rocky support to the epel element  https://review.opendev.org/c/openstack/diskimage-builder/+/83027807:40
dpawlikianw: hey, our monitoring tool said that the logscraper01.openstack.org is down. Could you get last console log and reboot that host please?07:51
dpawlikit is on vexxhost ca-ymq-107:51
*** ysandeep is now known as ysandeep|lunch07:58
*** jpena|off is now known as jpena08:33
*** ysandeep|lunch is now known as ysandeep08:41
ianwdpawlik: it's got a bunch of oops backtraces, but it's OOM09:07
ianw67769.004533] Out of memory: Killed process 13236 (logscraper) total-vm:2587404kB, anon-rss:35764kB, file-rss:0kB, shmem-rss:4kB, UID:10210 pgtables:1072kB oom_score_adj:009:07
ianwi've just issued a hard reboot on it09:07
dpawlikthank you09:07
dpawlikianw: probably the logstash service is still down and nothing was consuming the gearman queue.09:08
opendevreviewMerged openstack/diskimage-builder master: bootloader: clean up EFI checking  https://review.opendev.org/c/openstack/diskimage-builder/+/83029209:33
*** ysandeep is now known as ysandeep|dr_appt10:59
*** rlandy|out is now known as rlandy|ruck11:15
*** dviroel|out is now known as dviroel11:21
*** ysandeep|dr_appt is now known as ysandeep13:02
*** rlandy|ruck is now known as rlandy|ruck|mtg13:04
*** amoralej is now known as amoralej|lunch13:08
*** amoralej|lunch is now known as amoralej14:13
*** rlandy|ruck|mtg is now known as rlandy|ruck14:23
fungidtantsur: the gitweb links you mentioned last week are now fixed as of the weekend, we finally have links to gitea there instead14:33
fungiyou bet, it's something i've wanted for a long time (ever since we switched from cgit to gitea in fact)14:34
*** ysandeep is now known as ysandeep|dinner14:39
*** pojadhav|ruck is now known as pojadhav|dinner14:58
*** ysandeep|dinner is now known as ysandeep15:10
*** dviroel is now known as dviroel|lunch15:11
opendevreviewyatin proposed openstack/project-config master: Update Neutron's Dashboard as per recent changes  https://review.opendev.org/c/openstack/project-config/+/83044015:15
NeilHanlonianw: with that nodepool dib version bump, should it be OK to see if the rocky image will build on the nodepool hosts? (I think clarkb has it paused, maybe)15:33
*** pojadhav|dinner is now known as pojadhav|ruck15:33
clarkbNeilHanlon: yup I'll unpause it once our deployment updates (if it hasn't already)15:41
clarkbfrickler: I didn't have time to test the config update on the held gerrit yesterday, but did ask the gerrit slack about it and their response is that it should just start applying that information to changes that are updated after the config is set.15:42
clarkbI've approved the nodepool dib version update which should get us new images once that lands15:44
NeilHanlonnice, thanks clarkb!15:45
tristanCclarkb: thank you for gitweb link, would you know the change number that did the trick?16:06
clarkbtristanC: the gitea stuff in gerrit? https://gerrit-review.googlesource.com/c/gerrit/+/329279 and then https://gerrit-review.googlesource.com/c/gerrit/+/33059916:08
tristanCi see, thanks!16:09
opendevreviewMerged openstack/project-config master: Move missed repos under openstack-ansible-roles ACL  https://review.opendev.org/c/openstack/project-config/+/82927816:09
*** dviroel|lunch is now known as dviroel16:15
*** marios is now known as marios|out17:03
*** pojadhav|ruck is now known as pojadhav|out17:05
clarkbI think the nodepool iamges should be updated nowish. I'll hop on the servers shortly to check them an unpause rocky if they did update17:13
*** ysandeep is now known as ysandeep|out17:16
NeilHanlonsweet, thank you clarkb. 🤞17:17
clarkbNeilHanlon: nodepool has updated and I've unpaused the image. But looks like we're building other images right now so may be a bit before the rocky build is started17:23
*** Guest7 is now known as diablo_rojo_phone17:24
*** diablo_rojo_phone is now known as Guest22917:24
*** jpena is now known as jpena|off17:42
*** amoralej is now known as amoralej|off17:45
mgagne_fungi: You mentioned an email your received from INAP regarding a migration. Would you mind sharing its content? It can be sent to my email found in Gerrit.18:25
fungimgagne_: gladly, there were a few messages back and forth, i'll bounce them all to you when i get out of my next meeting18:38
mgagne_fungi: tyvm18:39
opendevreviewMerged openstack/project-config master: Update Neutron's Dashboard as per recent changes  https://review.opendev.org/c/openstack/project-config/+/83044018:51
fungimgagne_: i've bounced them all to you, should be 6 messages in total so far, i think19:12
NeilHanlonthanks clarkb, i'll keep an  eye on it19:14
fungimgagne_: if they don't come through for some reason (dmarc/spf or whatever breaking resends) i can redo it with forwards via new messages instead19:16
mgagne_fungi: I got them. tyvm!19:16
NeilHanlongood ol' INAP. i think i have my badge still somewhere..19:16
fungii have not replied to that last message from jennifer yet19:16
*** pi1 is now known as tbarron19:40
frickleron I only see spinning circles instead of the images and when I click on "View File", it shows a 404. the corresponding things work fine on opendev.org19:43
clarkbfrickler: looks like the svg's render but not the pngs19:44
clarkbthats a good catch. I suspect that is a regression19:44
clarkbfrickler: if you can leave notes about that on my gitea change I can work to file an issue upstream unless you'd like to19:45
fricklerI'll comment on the change19:46
fricklerthough for me also the svg's don't show up19:47
clarkbhuh the svg's do load for me in firefox19:47
fricklerah, for the "view file" they work, in the commit view, they only spin19:50
clarkboh ya sorry they don't 404 on view file19:51
clarkbbut they do spin19:51
fricklerthere's some tracebacks in the docker log for those is seems, starting with19:58
frickler2022/02/22 19:49:36 Completed GET /opendev/system-config/raw/commit/8f8100ed28d15bdad935b82dbfd6bb2d35203614/docker/gitea/custom/public/img/logo-192.png 404 Not Found in 13.431598ms19:59
frickler2022/02/22 19:49:36 ...s/context/context.go:290:PlainTextBytes() [E] PlainTextBytes: Not found.19:59
clarkbfrickler: was there something in the changelog that prompted you to check that? or just knowing that rendering of images is likely different enough from source code that it catches edge cases?20:01
clarkbAlright I need to take a break for lunch and then problbay a quick indoor bike ride. Then I'll try to sync up on the latest with rocky and find time with jentoio to look at conatiner stuff20:04
fricklerclarkb: no, I just clicked around a bit and then noticed it, no planned action20:04
clarkbfrickler: the pngs were deleted20:38
clarkbI think there are two issues here. The spinning in general. But then also not clearly listing files that are deleted as files that are deleted20:38
clarkbin prod we see only before data for the deleted files. I think this gives us what I need to file an issue with them. Will work on that in a bit20:39
clarkbhttps://github.com/go-gitea/gitea/issues/18811 is a related bug21:01
*** dviroel is now known as dviroel|brb21:23
clarkbI have filed https://github.com/go-gitea/gitea/issues/1886121:27
clarkbI was able to find what I think is a reproduction case on their demo site and I believe this is specifically related to renamed or deleted files because it is trying to load the deleted file and getting a 404 and instead of only showing the old side it spins. In the case of the renamed files it is trying to load the old name on the current commit which 404s as well since it has a new21:30
*** dviroel|brb is now known as dviroel21:40
clarkbjentoio: I'm going to get some exercise now, but if you around in ~an hour I should be back and we can dive into stuff if you like21:47
clarkbNeilHanlon: I think nb01 may be about to successfully complete a rocky build21:49
clarkbits copy the chroot contents into the image file now if I read teh log correctly (you should see this via the web server too if you pull it up)21:49
clarkbok back in a bit21:50
NeilHanlonsweet! i've been refreshing it every so often22:21
clarkbNeilHanlon: 2022-02-22 22:22:00.274 | Build completed successfully22:30
clarkbthe next step is the image then gets uploaded to the various clouds22:30
clarkbthen in theory you can tell zuul to run a job on it22:30
clarkblooks like all of the uploads are still in progress22:30
NeilHanlonawesome! this is exciting 22:43
NeilHanlonI think this is needed, too, and then i can have zuul try and re-check my change for openstack-ansible https://review.opendev.org/c/opendev/base-jobs/+/82843622:44
clarkbThe nodeset isn't scrictly necessary. You can replace the nodeset definition in your job with the contents of the nodeset in that change.22:46
clarkbjentoio: hey anytime for the next couple of hours is good for me if that works for you. Just let me know and we can sync up22:47
NeilHanlonclarkb: like so? https://paste.opendev.org/show/bInWKVOg7LMSRUZVaj98/22:59
clarkbNeilHanlon: I think you may still need to give it a name? but ya23:00
clarkbNeilHanlon: but I think network configuration is sad :( [[0;1;31mFAILED[0m] Failed to start Glean for interface ens3.23:01
clarkbI was worried about this, but thought the nodepool functional test may have had that covered. I guess not?23:01
NeilHanlon:( 23:01
NeilHanloni was going off of https://zuul-ci.org/docs/zuul/latest/config/job.html which seems to say when doing an in-line nodeset def. the name attribute should be skipped23:01
clarkbNeilHanlon: ah ok then no name necessary :)23:02
clarkbI'm going to try booting this image in the second cloud that it uploaded to to see if this is a consistent issue23:02
clarkbsometimes problems with networking are cloud specific23:02
NeilHanlongotcha, i do have a half patch for glean around here somewhere23:03
opendevreviewNeil Hanlon proposed opendev/glean master: add in rockylinux support for glean  https://review.opendev.org/c/opendev/glean/+/83053223:03
clarkbah ya that may be the next step23:04
NeilHanloni think where i left off with that was if I needed to generate the test data and commit it, or if it's produced by CI23:05
clarkbI guess the dib nodepool test images must use dhcp all interfaces or cloud init23:05
clarkbNeilHanlon: I think you can mostly make copies and just swap out the names for rocky23:08
clarkboh except that isn't necessary23:08
clarkbsince the distro detection is done via the os-release stuff and the vendored library not via the test fixtures. The test fixtures just have the network input data and I don't think you need to udpate that23:09
NeilHanloni'm gonna take your word for it lol :D23:09
clarkbok the ovh rocky image booted with networking but not the iweb23:10
clarkbis iweb statically configured? that may explain it if so23:10
clarkbwe likely still need your glean patch to make this work everywhere, but this may limp along until this lands23:11
clarkbI'm trying to make sense of how the network ended up being configured in ovh as that may give us clues23:13
clarkbsystemd[1]: glean@ens3.service: Main process exited, code=exited, status=1/FAILURE23:13
clarkbya so I think it is working despite the issue and likely via dhcp which iweb must not use23:13
clarkbheh and manually running glean appears to have been a success23:15
NeilHanlonah, interesting23:15
clarkbI think it may be nooping because it doesn't recognize the distro23:18
clarkbhwoever that doesn't explain why I get a 0 rc and systemd gets non zero23:18
clarkbI just cannot reproduce the failure when manually invoking it23:26
clarkbOne thing I notice is that we don't seem to have the --use-nm flag set which I think we did set for centos-823:26
clarkbianw: not urgent, but I'm noticing that we haven't released glean since your refactors for python stuff?23:29
clarkbianw: the glean@.service we have installed on rocky seems to match what we have in 1.19.0 not HEAD23:30
jentoioclarkb: hey, got distracted23:30
clarkbjentoio: no worries. I found something else to look at. But now is still good if you want to loko at things23:30
clarkbjentoio: https://meetpad.opendev.org/isitbroken this is my default jitsi meet room for random stuff if that works for you23:33
ianwthis is interesting, i intuitively thought that glean would not work, but it was working in the gate jobs so ... i just assumed it was working23:57
ianwbut perhaps the gate is not testing glean as much as we'd like?23:57
*** rlandy|ruck is now known as rlandy|out23:57

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!