Tuesday, 2021-11-16

opendevreviewMerged opendev/system-config master: mirror: Add centos-stream link  https://review.opendev.org/c/opendev/system-config/+/81802600:27
opendevreviewIan Wienand proposed openstack/project-config master: grafana: add centos-stream mirror volume  https://review.opendev.org/c/openstack/project-config/+/81803100:42
opendevreviewIan Wienand proposed openstack/project-config master: grafana: update nodepool/dib images  https://review.opendev.org/c/openstack/project-config/+/81803501:48
opendevreviewMerged openstack/project-config master: grafana: add centos-stream mirror volume  https://review.opendev.org/c/openstack/project-config/+/81803101:52
ianwianw: test06:18
ianwtestianw: testing06:20
ianwtestianw: testing again06:21
ianwtestianw: testing again again06:23
ianwtestianw: testing again06:26
ianwtestianw: a message06:28
ianwtestianw: foo06:28
*** ysandeep|out is now known as ysandeep06:30
ianwwhat a yak shaving adventure.  i finally tracked down my f35 crashes to the x11 qxl driver in my vm.  so i decided to switch to virtio.  i don't know why but latency was terrible with xfce.  so now i've switched back to gnome06:33
ianw3d/opengl passthrough seems to actually work now06:34
*** gibi_ is now known as gibi08:03
opendevreviewIan Wienand proposed openstack/diskimage-builder master: source-repositories: don't use --git-dir  https://review.opendev.org/c/openstack/diskimage-builder/+/81805308:25
ianwfungi/clarkb: ^ after instrumenting git to narrow this down, ultimately I think it's our error08:27
*** ysandeep is now known as ysandeep|lunch08:29
*** ykarel is now known as ykarel|lunch09:10
*** ysandeep|lunch is now known as ysandeep09:44
*** ykarel|lunch is now known as ykarel10:26
opendevreviewJakub Darmach proposed openstack/diskimage-builder master: Added missing grubby arg DIB_BOOTLOADER_DEFAULT_CMDLINE  https://review.opendev.org/c/openstack/diskimage-builder/+/81806310:43
*** ysandeep is now known as ysandeep|brb11:07
*** ysandeep|brb is now known as ysandeep11:13
opendevreviewAndreas Jaeger proposed openstack/project-config master: Iotronic-pythonclient and Iotronic-UI update  https://review.opendev.org/c/openstack/project-config/+/81771911:26
opendevreviewMerged opendev/base-jobs master: Remove growroot logs dumping in base-test  https://review.opendev.org/c/opendev/base-jobs/+/81728911:31
opendevreviewMerged openstack/project-config master: Create repo for ProxySQL Ansible role  https://review.opendev.org/c/openstack/project-config/+/81727111:38
opendevreviewMerged openstack/project-config master: Iotronic-pythonclient and Iotronic-UI update  https://review.opendev.org/c/openstack/project-config/+/81771913:41
zigoAny idea why heat's doc job fails, when it fully builds the docs? https://9348cb89109060032864-cd07d3426af01024dbeaac2f8639d2e0.ssl.cf1.rackcdn.com/817890/2/check/openstack-tox-docs/cbe0bf9/job-output.txt13:56
*** pojadhav is now known as pojadhav|brb13:59
*** pojadhav|brb is now known as pojadhav14:08
*** ysandeep is now known as ysandeep|out14:34
fungibuild finished with problems, 4 warnings.14:39
fungiERROR: InvocationError for command /home/zuul/src/opendev.org/openstack/heat/.tox/docs/bin/sphinx-build -W --keep-going -b html doc/source doc/build/html (exited with code 1)14:39
fungizigo: i believe -W tells sphinx-build t treat warnings as errors, so exits nonzero14:40
zigoOh, ok, thanks.14:40
fungimost openstack projects strive for warning-free doc builds, i think14:40
zigoWell, in this case, it looks like it broke the CI ! :)14:41
zigoYou are right with the -W ...14:43
fungizigo: looks like it's the warnigns like "docstring of heat.db.sqlalchemy.types.Json.process_bind_param:19: WARNING: undefined label: types_typedecorator" it's concerned about14:50
zigoYeah, saw it.14:50
zigoI'm looking for how to fix ...14:50
zigoI'm not sure what this is, but looks like comming from sqlalchemy.14:51
fungiheat/db/sqlalchemy/types.py i think14:51
zigoThough types.TypeDecorator isn't on line 19, but 23, so it's kind of werid.14:51
fungiit might be on line 23 after merging to the current state of the master branch14:52
fungihttps://review.opendev.org/c/openstack/heat/+/816592 merged a few days ago and added some blank lines early in that file14:53
zigoRight ! :)14:53
fungiso that would explain the line numbers being skewed from the ones in the change, zuul tests the proposed commit as it would be if it merged to the current state of the repository14:53
funginot as-written14:54
zigoMaybe this is due to now using a higher SQLAlchemy version...14:56
fungiand yeah, i have a feeling something used to define the types_typedecorator label14:57
fungiuse of types.TypeDecorator was added to that script in 2015, so it's not like it's a recent addition14:58
fungisqlalchemy 1.4.27 released last week, and was updated in upper-constraints.txt yesterday when https://review.opendev.org/c/openstack/requirements/+/817667 merged15:00
fungiso if the problem started showing up shortly after 08:00 utc yesterday, that might be related15:00
funginothing in https://docs.sqlalchemy.org/en/14/changelog/changelog_14.html#change-1.4.27 jumps out at me, but since it's documentation-related it could have been from something below the threshold for a changelog entry15:03
zigoLet's ask zeeek? :)15:09
fungizzzeek, but yeah i'm digging through the commit history first15:10
fungihttps://github.com/sqlalchemy/sqlalchemy/commit/605d0d9 might be related15:12
fungiit added :ref:`types_typedecorator` to some docstrings, new in the rel_1_4_27 tag15:22
fungiand that seems to be getting inherited via subclassing in heat, which i think causes sphinx to want to include it, but the types_typedecorator label is defined in the doc/build/core/custom_types.rst file in the sqlalchemy repo15:23
fungiit might be possible for heat to add a types_typedecorator label in its own docs15:24
zigoWhat exactly is a label ?15:24
zigoIs this a sphinx thing?15:25
fungiits a restructuredtext thing15:25
zigoOh, it's a .. thingylikethis: right?15:25
zigoWith underscore in front, event.15:26
zigoLet's try, I'd say? :)15:26
fungiyou can see the _types_typedecorator label for sqlalchemy's docs in https://raw.githubusercontent.com/sqlalchemy/sqlalchemy/main/doc/build/core/custom_types.rst15:28
zigoMaybe something naive like this will repair? https://review.opendev.org/c/openstack/heat/+/81810415:30
fungiit's worth testing, as long as that document is included in their docs build. a slightly less hacky solution might be to find a relevant place in the heat docs to add a short section which links to https://docs.sqlalchemy.org/en/14/core/custom_types.html#augmenting-existing-types15:33
fungisince that's what the docstring is trying to reference anyway15:33
*** artom__ is now known as artom15:34
*** dviroel is now known as dviroel_15:37
opendevreviewArtem Goncharov proposed zuul/zuul-jobs master: Add no_log to the log uploading module parameters  https://review.opendev.org/c/zuul/zuul-jobs/+/81810915:39
clarkbI'm testing the growroot log removal in base-test with https://review.opendev.org/c/zuul/zuul-jobs/+/680178 I'll approve the base update if that looks happy15:46
clarkbfungi: did you want to review the dib git repo caching fix before I approve it?15:48
fungioh, right thanks, i had pulled that up before i got distracted with a meeting15:50
clarkbOnce I've caught up on a few things like ^ I'm going to work on an email to openstack-discuss asking openstack to address their zuul config errors15:50
clarkbfungi: the change seems straightforward I'll let you +A if you are happy with it15:50
fungiyeah, for some reason i never noticed git's -C option and always ended up verbosely overriding both --git-dir and --work-tree15:53
fungii agree the manpage indicates this is far simpler too15:53
clarkbI suspect that it is newer than the ~ubuntu precice centos 5/6 time period this stuff was written15:54
opendevreviewArtem Goncharov proposed zuul/zuul-jobs master: Fix zuul-jobs-test-upload-logs-s3 job  https://review.opendev.org/c/zuul/zuul-jobs/+/80713216:20
*** ykarel is now known as ykarel|away16:27
opendevreviewArtem Goncharov proposed zuul/zuul-jobs master: Add no_log to the log uploading module parameters  https://review.opendev.org/c/zuul/zuul-jobs/+/81810916:29
opendevreviewMerged zuul/zuul-jobs master: Fix zuul-jobs-test-upload-logs-s3 job  https://review.opendev.org/c/zuul/zuul-jobs/+/80713216:34
opendevreviewMerged zuul/zuul-jobs master: Add no_log to the log uploading module parameters  https://review.opendev.org/c/zuul/zuul-jobs/+/81810916:45
clarkbI've approved https://review.opendev.org/c/opendev/base-jobs/+/817290 which updates the base job to remove growroot log checking17:04
clarkbbase-test testing seemed happy17:04
opendevreviewMerged opendev/base-jobs master: Remove growroot log dumping from the base job  https://review.opendev.org/c/opendev/base-jobs/+/81729017:11
opendevreviewMerged openstack/diskimage-builder master: source-repositories: don't use --git-dir  https://review.opendev.org/c/openstack/diskimage-builder/+/81805317:28
*** marios is now known as marios|out17:29
*** sshnaidm is now known as sshnaidm|afk17:33
clarkbfungi: when is a good time to talk lists.o.o kernel stuff?17:50
clarkbI need ot eat something and prep for the infra meeting but can probably sneak ^ in now if that works17:50
*** tosky is now known as Guest605418:05
*** tosky_ is now known as tosky18:05
fungiahh, sorry, stepped away for a snack, but maybe after the meeting?18:20
clarkbfungi: now is still good if it works, but ya can do after the meeting too (though going to try a bike ride today if I can since the rain stopped)18:20
fungioh, sure, i can do it now18:21
fungii've started a root screen session on lists.o.o18:22
clarkbok I've joined (had to load my keys)18:24
clarkbfungi: in my homedir there is a dir called kernel-stuff or similar18:24
clarkbamybe we should create one in roots homedir and work from there goign forward18:24
fungii'll copy it over18:28
fungiand done18:29
clarkbfungi: extract-vmlinux is the tool from the kernel git repo18:29
clarkbyou basically give it a vmlinuz file and it writes to stdout the extracted file. So what I did previously was copied the compressed file to this kernel-stuff dir, extract it into a new file in that dir than copied the extracted file over the original in /boot18:29
clarkbthe menu.lst stuff was from figuring out the chainload situation18:30
fungiokay so there's a vmlinuz-5.4.0-90-generic.extracted in ~root/kernel-stuff now18:33
fungias well as a backup of the compressed vmlinuz-5.4.0-90-generic from /boot18:33
fungii ran it like `bash extract-vmlinux vmlinuz-5.4.0-90-generic > vmlinuz-5.4.0-90-generic.extracted`18:33
clarkbya I think that is correct. The size looks about right too18:33
clarkbI don't think we were able to come up with any better verification other than size is bigger18:34
clarkbbut nwo you can copy that file back over the /boot path that the original was at18:34
fungifile says it's an ELF 64-bit LSB executable rather than a Linux kernel x86 boot executable bzImage18:34
clarkbfungi: amybe double check permissions for the result too18:35
clarkbafter it is copied I mean18:35
clarkbthat looks good18:35
fungipermissions and ownership remained consistent with the others there, yeah18:35
fungiand the size is in the same range as the earlier extracted one18:36
fungishould we reboot now or test that at a different time?18:36
clarkbup to you I guess. I'd like to get out on a bike ride in an hour and a half or so and that might impact that :) but if you want to debug potentail failed boots I'm fine with that18:37
fungii'm happy to reboot now18:37
clarkbI won't stop you :)18:37
fungigoing ahead then18:38
clarkbit pings but is network unreachable currently. I don't recall if that was a symptom of the failed boot or not18:39
*** sshnaidm|afk is now known as sshnaidm18:39
clarkboh and now it doesn't pign so maybe it was on its way down and now we wait for it to come back18:40
fungiit was refusing port 22 connections up to a moment ago, yeah18:40
fungii think so18:40
fungipinging again now18:41
fungiLinux lists 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux18:41
fungi#status log Rebooted main lists.o.o server onto new extracted vmlinuz-5.4.0-90-generic kernel18:41
clarkbits up and I confirm uname reports the new kernel18:41
opendevstatusfungi: finished logging18:41
fungithanks! that was fairly straightforward18:41
clarkbya once we sorted out how to make it work it isn't terrible (other than it isn't currently automated)18:42
clarkbit was the figuring it out step that was painful18:42
fungialso 45 processes owned by the list user, according to ps, so everything's running that should be18:42
clarkbI can get list archives too (I needed that for writing down my meeting notes)18:43
*** tosky_ is now known as tosky18:48
clarkbis https://zuul.opendev.org/t/openstack/build/bc0075f04a8941cf82dc536f76d6ecab having a sad for anyone else too?18:50
clarkbthat caused the openstack gate to reset and I was going to take a quick look but I don't seem to get logs18:50
clarkbthat has me concerned that some of the job updtes that landed recently could affect that but spot checking other failures I get logs for them18:50
fricklerclarkb: "malformed URL"18:51
fricklerhttps://storage.bhs.cloud.********.net does it really mean it or try to hide something?18:52
clarkbya it seems there are **** in the url?18:52
clarkbhttps://zuul.opendev.org/t/openstack/build/f9f275a756534946850b2ec51541fe2a ok found another one18:53
clarkbcorvus: ^ could this be related to the recent zuul-jobs update? seems unlikely given that rax uploads seem fine?18:53
fungireplacing the ******** with ovh fixes it18:53
fricklerthat looks like it could try to obscure the result18:54
clarkbfrickler: ya that is the change, but it isn't doing it for rax. I'm really confused as to how that would happen here18:54
clarkbthe first build I linked to ran on ze0218:57
clarkbReturn log URL to Zuul is the task that would set the value, maybe it is affected18:58
clarkblog_url: "{{ upload_results.url }}/" is what that task does18:58
fungistrange that only the "ovh" in the url was replaced18:59
fungii wonder if it mistakes that for a sensitive pattern18:59
clarkbok rax has a different code path due to the cdn in the library file19:00
clarkbI need to context switch to the meeting, but we should keep looking at this. it is possible it is just coincidence and something change elsewhere but also possibly related19:00
clarkbhttps://opendev.org/zuul/zuul-jobs/src/branch/master/roles/upload-logs-base/library/zuul_swift_upload.py#L92-L101 is the special casing19:01
fungithat change was the first one to touch zuul_swift_upload.py since february19:03
opendevreviewArtem Goncharov proposed zuul/zuul-jobs master: Revert "Add no_log to the log uploading module parameters"  https://review.opendev.org/c/zuul/zuul-jobs/+/81798819:06
opendevreviewMerged zuul/zuul-jobs master: Revert "Add no_log to the log uploading module parameters"  https://review.opendev.org/c/zuul/zuul-jobs/+/81798819:18
clarkbto followup on the no_log thing ansible has a "feature' where any values in a no_log dict are removed from logs and return values20:17
clarkbin this case ovh is the name of the cloud in the cloud dict so it gets removed from the url that is returned20:18
clarkbrax doesn't suffer this as it uses different names for the cdn20:18
ianwhttps://landscape.cncf.io/?selected=open-stack is the thing scraping the repo i guess20:18
ianwi don't know how i feel about ranking projects by "latest tweet"20:19
clarkbianw: its the future!20:19
ianwalso seems zuul could have an icon in just about every box of https://landscape.cd.foundation/20:20
fungialso "github stars" are a good indication of project health, in case you didn't knopw20:21
clarkbbut no github repo so they don't let zuul in20:22
fungiwhich is amusing, because they let proprietary projects into the landscape with no public source control reference at all20:22
fungibut if it's open source, it has to be in github (isn't that the definition of "open source" now?)20:23
opendevreviewArtem Goncharov proposed zuul/zuul-jobs master: Apply no_log on the upload-logs-swift role  https://review.opendev.org/c/zuul/zuul-jobs/+/81813720:24
clarkbthings seem happy enough now. Going on that bike ride20:31
clarkbback in a bit20:31
fungihave fun!20:40
opendevreviewClark Boylan proposed openstack/project-config master: Set noop jobs on openstackid repos  https://review.opendev.org/c/openstack/project-config/+/81817022:29
opendevreviewClark Boylan proposed opendev/puppet-openstackid master: Retire this project  https://review.opendev.org/c/opendev/puppet-openstackid/+/81817222:41
*** tosky is now known as Guest607022:42
*** tosky_ is now known as tosky22:42
opendevreviewClark Boylan proposed openstack/project-config master: Complete openstackid repo retirements  https://review.opendev.org/c/openstack/project-config/+/81817622:52
clarkbopenstackid retirement shoudl clean up the errors with its zuul config ^ :)22:53
*** tosky is now known as Guest607323:07
*** tosky_ is now known as tosky23:07
ianwsmall patch for the caching issue @ https://public-inbox.org/git/YZQ5Zk0ItWvfr8sF@fedora19.localdomain/T/#u23:08
ianwof course already seen a spelling error.  i hate emailing patches23:09
clarkbhere here23:09
clarkbor is it hear hear?23:09
clarkbianw: I guess is_empty() does more than just check if it is a directory and is empty? similar with is_directory?23:12
clarkbbecause --git-dir should be a directory path right?23:12
clarkboh wait is it that empty_submodule_path.buf is set to some weird value? I bet that is it and it isn't actually a directory and probably doesn't exist at all23:13
clarkbianw: the typo appears to be in the comment? the C seems valid at least. I wonder if they will nitpick that23:15
clarkbianw: re the debugging were you recompiling git to add extra debugging or was gdb sufficient? (I'm mostly curious as its been a long time since I had to do debugging like that myself)23:16
ianwclarkb: i recompiled git; i found you can mostly fprintf(stderr, "...") at random points safely23:28
ianw--git-dir should be relative to the top of the repo.  i can't image why it's really even an option, but ... git23:28
ianwin this case, the repo was in /home/iwienand/tmp/os/openstack/ ; we work in /home/iwienand/tmp/os/ and then set --gitdir openstack/.git 23:30
ianwthe error would look like23:30
ianwFrom https://opendev.org/openstack/openstack23:30
ianw + 7af3bee3a1...c557eac228 master             -> master  (forced update)23:30
ianw * [new branch]            milestone-proposed -> milestone-proposed23:30
ianwSubmodule directory '/home/iwienand/tmp/os/manila/' not found23:30
ianwSubmodule directory '/home/iwienand/tmp/os/openstack-helm-images/' not found23:30
ianwSubmodule directory '/home/iwienand/tmp/os/tripleo-ansible/' not found23:30
ianwwith the patch applied23:30
ianw(note the missing "/openstack/")23:31
clarkbso ya the dir didn't exist at all not that it wasn't empty23:31
clarkbthat makes sense given your patch23:31
ianwyeah, that is_empty_dir function ends up conflating "is empty" and "missing"23:35

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!