| opendevreview | Ian Wienand proposed openstack/diskimage-builder master: bootloader: clean up EFI checking https://review.opendev.org/c/openstack/diskimage-builder/+/830292 | 00:08 |
|---|---|---|
| clarkb | ianw: for ^ that was just a linter fixup? | 00:09 |
| clarkb | meeting agenda is out | 00:10 |
| ianw | yep, for dib-lint's inability to deal with \ continuation indentation :/ | 00:10 |
| jentoio | clarkb: tomorrow is fine. thanks | 00:35 |
| *** rlandy|ruck|bbl is now known as rlandy|ruck | 00:35 | |
| opendevreview | Merged opendev/system-config master: run-production-playbook: fix path typo https://review.opendev.org/c/opendev/system-config/+/830294 | 00:42 |
| *** rlandy|ruck is now known as rlandy|out | 00:44 | |
| opendevreview | Ian Wienand proposed opendev/system-config master: run-production-playbook : fix synchronize src/dest arguments https://review.opendev.org/c/opendev/system-config/+/830297 | 00:54 |
| opendevreview | Ian Wienand proposed opendev/system-config master: run-production-playbook : fix synchronize src/dest arguments https://review.opendev.org/c/opendev/system-config/+/830297 | 00:54 |
| opendevreview | Steve Baker proposed openstack/diskimage-builder master: rhel: work around RHEL-9 BLS issues https://review.opendev.org/c/openstack/diskimage-builder/+/829620 | 01:09 |
| opendevreview | Steve Baker proposed openstack/diskimage-builder master: Detect boot and EFI partitions in extract-image https://review.opendev.org/c/openstack/diskimage-builder/+/828617 | 01:09 |
| opendevreview | Merged opendev/system-config master: run-production-playbook : fix synchronize src/dest arguments https://review.opendev.org/c/opendev/system-config/+/830297 | 02:04 |
| *** pojadhav- is now known as pojadhav|ruck | 02:53 | |
| opendevreview | Merged openstack/diskimage-builder master: bootloader: fix arm64 install path https://review.opendev.org/c/openstack/diskimage-builder/+/830111 | 03:17 |
| opendevreview | Ian Wienand proposed opendev/system-config master: run-production-playbook: fix stray }} https://review.opendev.org/c/opendev/system-config/+/830304 | 03:19 |
| Clark[m] | I wonder why the linter didn't catch those | 04:06 |
| opendevreview | Merged opendev/system-config master: run-production-playbook: fix stray }} https://review.opendev.org/c/opendev/system-config/+/830304 | 04:25 |
| *** ysandeep|out is now known as ysandeep | 04:47 | |
| opendevreview | Ian Wienand proposed opendev/system-config master: run-production-playbook: fix another stray }} https://review.opendev.org/c/opendev/system-config/+/830311 | 05:58 |
| opendevreview | Ian Wienand proposed opendev/system-config master: encrypt-logs: fix a couple of download script issues https://review.opendev.org/c/opendev/system-config/+/830312 | 06:11 |
| opendevreview | Ian Wienand proposed opendev/system-config master: encrypt-logs: fix a couple of download script issues https://review.opendev.org/c/opendev/system-config/+/830312 | 06:11 |
| opendevreview | Merged opendev/system-config master: run-production-playbook: fix another stray }} https://review.opendev.org/c/opendev/system-config/+/830311 | 06:52 |
| opendevreview | Merged opendev/system-config master: encrypt-logs: fix a couple of download script issues https://review.opendev.org/c/opendev/system-config/+/830312 | 07:08 |
| *** amoralej|off is now known as amoralej | 07:34 | |
| opendevreview | Merged openstack/diskimage-builder master: Add rocky support to the epel element https://review.opendev.org/c/openstack/diskimage-builder/+/830278 | 07:40 |
| dpawlik | ianw: hey, our monitoring tool said that the logscraper01.openstack.org is down. Could you get last console log and reboot that host please? | 07:51 |
| dpawlik | it is on vexxhost ca-ymq-1 | 07:51 |
| *** ysandeep is now known as ysandeep|lunch | 07:58 | |
| *** jpena|off is now known as jpena | 08:33 | |
| *** ysandeep|lunch is now known as ysandeep | 08:41 | |
| ianw | dpawlik: it's got a bunch of oops backtraces, but it's OOM | 09:07 |
| ianw | 67769.004533] Out of memory: Killed process 13236 (logscraper) total-vm:2587404kB, anon-rss:35764kB, file-rss:0kB, shmem-rss:4kB, UID:10210 pgtables:1072kB oom_score_adj:0 | 09:07 |
| dpawlik | ehh | 09:07 |
| ianw | i've just issued a hard reboot on it | 09:07 |
| dpawlik | thank you | 09:07 |
| dpawlik | ianw: probably the logstash service is still down and nothing was consuming the gearman queue. | 09:08 |
| opendevreview | Merged openstack/diskimage-builder master: bootloader: clean up EFI checking https://review.opendev.org/c/openstack/diskimage-builder/+/830292 | 09:33 |
| *** ysandeep is now known as ysandeep|dr_appt | 10:59 | |
| *** rlandy|out is now known as rlandy|ruck | 11:15 | |
| *** dviroel|out is now known as dviroel | 11:21 | |
| *** ysandeep|dr_appt is now known as ysandeep | 13:02 | |
| *** rlandy|ruck is now known as rlandy|ruck|mtg | 13:04 | |
| *** amoralej is now known as amoralej|lunch | 13:08 | |
| *** amoralej|lunch is now known as amoralej | 14:13 | |
| *** rlandy|ruck|mtg is now known as rlandy|ruck | 14:23 | |
| fungi | dtantsur: the gitweb links you mentioned last week are now fixed as of the weekend, we finally have links to gitea there instead | 14:33 |
| dtantsur | thanks! | 14:33 |
| fungi | you bet, it's something i've wanted for a long time (ever since we switched from cgit to gitea in fact) | 14:34 |
| *** ysandeep is now known as ysandeep|dinner | 14:39 | |
| *** pojadhav|ruck is now known as pojadhav|dinner | 14:58 | |
| *** ysandeep|dinner is now known as ysandeep | 15:10 | |
| *** dviroel is now known as dviroel|lunch | 15:11 | |
| opendevreview | yatin proposed openstack/project-config master: Update Neutron's Dashboard as per recent changes https://review.opendev.org/c/openstack/project-config/+/830440 | 15:15 |
| NeilHanlon | ianw: with that nodepool dib version bump, should it be OK to see if the rocky image will build on the nodepool hosts? (I think clarkb has it paused, maybe) | 15:33 |
| *** pojadhav|dinner is now known as pojadhav|ruck | 15:33 | |
| clarkb | NeilHanlon: yup I'll unpause it once our deployment updates (if it hasn't already) | 15:41 |
| clarkb | frickler: I didn't have time to test the config update on the held gerrit yesterday, but did ask the gerrit slack about it and their response is that it should just start applying that information to changes that are updated after the config is set. | 15:42 |
| clarkb | I've approved the nodepool dib version update which should get us new images once that lands | 15:44 |
| NeilHanlon | nice, thanks clarkb! | 15:45 |
| tristanC | clarkb: thank you for gitweb link, would you know the change number that did the trick? | 16:06 |
| clarkb | tristanC: the gitea stuff in gerrit? https://gerrit-review.googlesource.com/c/gerrit/+/329279 and then https://gerrit-review.googlesource.com/c/gerrit/+/330599 | 16:08 |
| tristanC | i see, thanks! | 16:09 |
| opendevreview | Merged openstack/project-config master: Move missed repos under openstack-ansible-roles ACL https://review.opendev.org/c/openstack/project-config/+/829278 | 16:09 |
| *** dviroel|lunch is now known as dviroel | 16:15 | |
| *** marios is now known as marios|out | 17:03 | |
| *** pojadhav|ruck is now known as pojadhav|out | 17:05 | |
| clarkb | I think the nodepool iamges should be updated nowish. I'll hop on the servers shortly to check them an unpause rocky if they did update | 17:13 |
| *** ysandeep is now known as ysandeep|out | 17:16 | |
| NeilHanlon | sweet, thank you clarkb. 🤞 | 17:17 |
| clarkb | NeilHanlon: nodepool has updated and I've unpaused the image. But looks like we're building other images right now so may be a bit before the rocky build is started | 17:23 |
| *** Guest7 is now known as diablo_rojo_phone | 17:24 | |
| *** diablo_rojo_phone is now known as Guest229 | 17:24 | |
| *** jpena is now known as jpena|off | 17:42 | |
| *** amoralej is now known as amoralej|off | 17:45 | |
| mgagne_ | fungi: You mentioned an email your received from INAP regarding a migration. Would you mind sharing its content? It can be sent to my email found in Gerrit. | 18:25 |
| fungi | mgagne_: gladly, there were a few messages back and forth, i'll bounce them all to you when i get out of my next meeting | 18:38 |
| mgagne_ | fungi: tyvm | 18:39 |
| opendevreview | Merged openstack/project-config master: Update Neutron's Dashboard as per recent changes https://review.opendev.org/c/openstack/project-config/+/830440 | 18:51 |
| fungi | mgagne_: i've bounced them all to you, should be 6 messages in total so far, i think | 19:12 |
| NeilHanlon | thanks clarkb, i'll keep an eye on it | 19:14 |
| fungi | mgagne_: if they don't come through for some reason (dmarc/spf or whatever breaking resends) i can redo it with forwards via new messages instead | 19:16 |
| mgagne_ | fungi: I got them. tyvm! | 19:16 |
| fungi | yw! | 19:16 |
| NeilHanlon | good ol' INAP. i think i have my badge still somewhere.. | 19:16 |
| fungi | i have not replied to that last message from jennifer yet | 19:16 |
| *** pi1 is now known as tbarron | 19:40 | |
| frickler | on https://104.130.74.7:3081/opendev/system-config/commit/8f8100ed28d15bdad935b82dbfd6bb2d35203614 I only see spinning circles instead of the images and when I click on "View File", it shows a 404. the corresponding things work fine on opendev.org | 19:43 |
| clarkb | frickler: looks like the svg's render but not the pngs | 19:44 |
| clarkb | thats a good catch. I suspect that is a regression | 19:44 |
| clarkb | frickler: if you can leave notes about that on my gitea change I can work to file an issue upstream unless you'd like to | 19:45 |
| frickler | I'll comment on the change | 19:46 |
| frickler | though for me also the svg's don't show up | 19:47 |
| clarkb | huh the svg's do load for me in firefox | 19:47 |
| frickler | ah, for the "view file" they work, in the commit view, they only spin | 19:50 |
| clarkb | oh ya sorry they don't 404 on view file | 19:51 |
| clarkb | but they do spin | 19:51 |
| frickler | there's some tracebacks in the docker log for those is seems, starting with | 19:58 |
| frickler | 2022/02/22 19:49:36 Completed GET /opendev/system-config/raw/commit/8f8100ed28d15bdad935b82dbfd6bb2d35203614/docker/gitea/custom/public/img/logo-192.png 404 Not Found in 13.431598ms | 19:59 |
| frickler | 2022/02/22 19:49:36 ...s/context/context.go:290:PlainTextBytes() [E] PlainTextBytes: Not found. | 19:59 |
| clarkb | frickler: was there something in the changelog that prompted you to check that? or just knowing that rendering of images is likely different enough from source code that it catches edge cases? | 20:01 |
| clarkb | Alright I need to take a break for lunch and then problbay a quick indoor bike ride. Then I'll try to sync up on the latest with rocky and find time with jentoio to look at conatiner stuff | 20:04 |
| frickler | clarkb: no, I just clicked around a bit and then noticed it, no planned action | 20:04 |
| clarkb | frickler: the pngs were deleted | 20:38 |
| clarkb | I think there are two issues here. The spinning in general. But then also not clearly listing files that are deleted as files that are deleted | 20:38 |
| clarkb | in prod we see only before data for the deleted files. I think this gives us what I need to file an issue with them. Will work on that in a bit | 20:39 |
| clarkb | https://github.com/go-gitea/gitea/issues/18811 is a related bug | 21:01 |
| *** dviroel is now known as dviroel|brb | 21:23 | |
| clarkb | I have filed https://github.com/go-gitea/gitea/issues/18861 | 21:27 |
| clarkb | I was able to find what I think is a reproduction case on their demo site and I believe this is specifically related to renamed or deleted files because it is trying to load the deleted file and getting a 404 and instead of only showing the old side it spins. In the case of the renamed files it is trying to load the old name on the current commit which 404s as well since it has a new | 21:30 |
| clarkb | name | 21:30 |
| *** dviroel|brb is now known as dviroel | 21:40 | |
| clarkb | jentoio: I'm going to get some exercise now, but if you around in ~an hour I should be back and we can dive into stuff if you like | 21:47 |
| clarkb | NeilHanlon: I think nb01 may be about to successfully complete a rocky build | 21:49 |
| clarkb | its copy the chroot contents into the image file now if I read teh log correctly (you should see this via the web server too if you pull it up) | 21:49 |
| clarkb | ok back in a bit | 21:50 |
| NeilHanlon | sweet! i've been refreshing it every so often | 22:21 |
| clarkb | NeilHanlon: 2022-02-22 22:22:00.274 | Build completed successfully | 22:30 |
| clarkb | the next step is the image then gets uploaded to the various clouds | 22:30 |
| clarkb | then in theory you can tell zuul to run a job on it | 22:30 |
| clarkb | looks like all of the uploads are still in progress | 22:30 |
| NeilHanlon | awesome! this is exciting | 22:43 |
| NeilHanlon | I think this is needed, too, and then i can have zuul try and re-check my change for openstack-ansible https://review.opendev.org/c/opendev/base-jobs/+/828436 | 22:44 |
| clarkb | The nodeset isn't scrictly necessary. You can replace the nodeset definition in your job with the contents of the nodeset in that change. | 22:46 |
| clarkb | jentoio: hey anytime for the next couple of hours is good for me if that works for you. Just let me know and we can sync up | 22:47 |
| NeilHanlon | clarkb: like so? https://paste.opendev.org/show/bInWKVOg7LMSRUZVaj98/ | 22:59 |
| clarkb | NeilHanlon: I think you may still need to give it a name? but ya | 23:00 |
| clarkb | NeilHanlon: but I think network configuration is sad :( [[0;1;31mFAILED[0m] Failed to start Glean for interface ens3. | 23:01 |
| clarkb | I was worried about this, but thought the nodepool functional test may have had that covered. I guess not? | 23:01 |
| NeilHanlon | :( | 23:01 |
| NeilHanlon | i was going off of https://zuul-ci.org/docs/zuul/latest/config/job.html which seems to say when doing an in-line nodeset def. the name attribute should be skipped | 23:01 |
| clarkb | NeilHanlon: ah ok then no name necessary :) | 23:02 |
| clarkb | I'm going to try booting this image in the second cloud that it uploaded to to see if this is a consistent issue | 23:02 |
| clarkb | sometimes problems with networking are cloud specific | 23:02 |
| NeilHanlon | gotcha, i do have a half patch for glean around here somewhere | 23:03 |
| opendevreview | Neil Hanlon proposed opendev/glean master: add in rockylinux support for glean https://review.opendev.org/c/opendev/glean/+/830532 | 23:03 |
| clarkb | ah ya that may be the next step | 23:04 |
| NeilHanlon | i think where i left off with that was if I needed to generate the test data and commit it, or if it's produced by CI | 23:05 |
| clarkb | I guess the dib nodepool test images must use dhcp all interfaces or cloud init | 23:05 |
| clarkb | NeilHanlon: I think you can mostly make copies and just swap out the names for rocky | 23:08 |
| clarkb | oh except that isn't necessary | 23:08 |
| clarkb | since the distro detection is done via the os-release stuff and the vendored library not via the test fixtures. The test fixtures just have the network input data and I don't think you need to udpate that | 23:09 |
| NeilHanlon | i'm gonna take your word for it lol :D | 23:09 |
| clarkb | ok the ovh rocky image booted with networking but not the iweb | 23:10 |
| clarkb | is iweb statically configured? that may explain it if so | 23:10 |
| clarkb | we likely still need your glean patch to make this work everywhere, but this may limp along until this lands | 23:11 |
| clarkb | I'm trying to make sense of how the network ended up being configured in ovh as that may give us clues | 23:13 |
| clarkb | systemd[1]: glean@ens3.service: Main process exited, code=exited, status=1/FAILURE | 23:13 |
| clarkb | ya so I think it is working despite the issue and likely via dhcp which iweb must not use | 23:13 |
| clarkb | heh and manually running glean appears to have been a success | 23:15 |
| NeilHanlon | ah, interesting | 23:15 |
| NeilHanlon | hah | 23:15 |
| clarkb | I think it may be nooping because it doesn't recognize the distro | 23:18 |
| clarkb | hwoever that doesn't explain why I get a 0 rc and systemd gets non zero | 23:18 |
| clarkb | I just cannot reproduce the failure when manually invoking it | 23:26 |
| clarkb | One thing I notice is that we don't seem to have the --use-nm flag set which I think we did set for centos-8 | 23:26 |
| clarkb | ianw: not urgent, but I'm noticing that we haven't released glean since your refactors for python stuff? | 23:29 |
| clarkb | ianw: the glean@.service we have installed on rocky seems to match what we have in 1.19.0 not HEAD | 23:30 |
| jentoio | clarkb: hey, got distracted | 23:30 |
| clarkb | jentoio: no worries. I found something else to look at. But now is still good if you want to loko at things | 23:30 |
| jentoio | sure | 23:31 |
| clarkb | jentoio: https://meetpad.opendev.org/isitbroken this is my default jitsi meet room for random stuff if that works for you | 23:33 |
| ianw | this is interesting, i intuitively thought that glean would not work, but it was working in the gate jobs so ... i just assumed it was working | 23:57 |
| ianw | but perhaps the gate is not testing glean as much as we'd like? | 23:57 |
| *** rlandy|ruck is now known as rlandy|out | 23:57 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!