Tuesday, 2022-09-27

*** dviroel|dr_appt is now known as dviroel01:16
*** dviroel is now known as dviroel|out01:31
*** ysandeep|out is now known as ysandeep02:39
*** ysandeep is now known as ysandeep|afk04:52
*** ysandeep|afk is now known as ysandeep05:35
dpawlikhello folks. Small question: do you know what from time to time generates testrepository.subunit.gz that got broken content inside?06:57
dpawlikfor example: https://42ef709f4885026f2696-9795c6ee70a51795a4d498c996d13136.ssl.cf2.rackcdn.com/periodic/opendev.org/openstack/oslo.db/master/propose-translation-update/e665a4f/testrepository.subunit.gz 06:57
dpawlikor https://42ef709f4885026f2696-9795c6ee70a51795a4d498c996d13136.ssl.cf2.rackcdn.com/periodic/opendev.org/openstack/oslo.db/master/propose-translation-update/e665a4f/testrepository.subunit.gz  or06:57
opendevreviewMichal Nasiadka proposed openstack/project-config master: Add rockylinux-9-arm64  https://review.opendev.org/c/openstack/project-config/+/85855407:24
mnasiadkaclarkb: ack, updated07:24
*** jpena|off is now known as jpena07:37
opendevreviewDmitriy Rabotyagov proposed opendev/system-config master: Add Ceph Quincy mirror  https://review.opendev.org/c/opendev/system-config/+/85932708:23
noonedeadpunkfungi: would be great if ceph quincy mirror won't be forgotten as pacific was (https://review.opendev.org/c/opendev/system-config/+/823504)08:25
noonedeadpunkbut I will try to take care of that by pinging others in some time 08:27
opendevreviewRafal Lewandowski proposed openstack/diskimage-builder master: Added cloud-init growpart element  https://review.opendev.org/c/openstack/diskimage-builder/+/85585609:00
*** soniya29 is now known as soniya29|afk11:09
*** dviroel|out is now known as dviroel11:19
fungidpawlik: is it only translation jobs?11:31
funginoonedeadpunk: yep, it's on my list for this morning once i get caught up on e-mail et cetera11:32
fungithanks for the reminder!11:32
fungii'm also hoping to get to expanding the storage on x86 nodepool builders today11:35
dpawlikfungi: hey, hard to say. I see that after adding subunit file, it download such artifacts11:44
*** ysandeep is now known as ysandeep|afk11:59
*** ysandeep|afk is now known as ysandeep|mtg12:33
*** soniya29|afk is now known as soniya2913:18
fungidpawlik: it looks like each of the subunit files you linked is basically empty (just some headings). i have a feeling the translation jobs just shouldn't be collecting those subunit files13:22
fungimaybe once clarkb's around he might know more about what's going on there13:23
dpawlikfungi: thanks13:23
dpawlikI temporary skip the file for processing in logsender13:24
dpawlikit would be helpful to remove the workaround in the future in logsender :)13:24
*** ysandeep|mtg is now known as ysandeep13:24
fungiout of curiosity, what's doing the subunit parsing?13:25
fungiwe used to do it with subunit2sql for the openstack-health backend13:25
fungii don't remember the old logstash infrastructure doing any subunit parsing, so it's pretty cool if you've managed to add some in the new importer13:26
*** dasm|off is now known as dasm13:46
*** ysandeep is now known as ysandeep|out14:01
clarkbdpawlik: fungi: some jobs had stub subunit files to help record them in the old health server. I'm not sure if that is what is happening here but could explain it possibly.14:44
clarkbI'd have to look at the job and see what is generating the file (testr/stestr is the typical case but for the stub files I think they may have written them more directly with basic info)14:44
dpawlikfungi: Sorry for delay. TBH I don't know how the whole process for subunit works. Maybe lpiwowar knows better how it works or arxcruz14:50
dpawlikclarkb: ack. 14:51
arxcruzfungi dpawlik the idea was to collect tempest info, tests running, what was failing, skipped, etc 14:52
arxcruzi think we can check in the log parsing if the subunit is from tempest or not, and only parse tempest 14:53
arxcruzor if it's empty 14:53
fungiarxcruz: cool, so basically what we used to do with subunit2sql and the openstack-health dashboard before we stopped running it14:53
clarkbdpawlik: https://opendev.org/openstack/project-config/src/branch/master/roles/fetch-translations-subunit-output/README.rst theres a hint14:54
dpawlikclarkb: I see in codesearch that there is a dedicated --include parameter that will get the file14:56
clarkbdpawlik: https://opendev.org/openstack/openstack-zuul-jobs/src/branch/master/roles/prepare-zanata-client/files/common_translation_update.sh#L79-L88 as suspected it is directly generating a stub subunit file14:56
clarkbI think the file should be valid.14:56
dpawlikthat can be it14:57
dpawlikgood point clarkb++ and fungi++14:57
clarkbit is creatinga single entry subunit with a timestamp, test/job name, and status. It uses generate-subunit so this should be completely valid14:57
dpawlikalso thanks arxcruz!14:57
clarkbany problems you've got are likely on the deserialization side given that14:57
* clarkb fetches one of the files to see14:57
clarkbyes the file I fetched appears to be perfectly valid.14:59
fungicool, so just very small14:59
clarkbsubunit-2to1 parses it properly and emits a more human readable version that indicates the info that that script is writing14:59
clarkbarxcruz: ^ I think the file is valid fwiw15:01
clarkbI didn't end up restarting apache yesterday afternoon. I'll try to remember better today15:40
fungiit may eventually clear up on its own too15:41
*** dviroel is now known as dviroel|lunch16:01
fungiclarkb: i've managed to use cinderclient to create new volumes for the builders in dfw, but can't get either osc nor novaclient to attach them16:14
fungihave you done that lately?16:14
fungimy shell history on bridge indicates i did it most recently with novaclient, but currently i'm getting this...16:16
fungiDEBUG (v2:61) Making authentication request to https://identity.api.rackspacecloud.com/v2.0/tokens16:16
fungiDEBUG (connectionpool:396) https://identity.api.rackspacecloud.com:443 "POST /v2.0/tokens HTTP/1.1" 400 7616:17
fungiDEBUG (session:975) Request returned failure status: 40016:17
fungiDEBUG (shell:821) Unrecognized schema in response body. (HTTP 400)16:17
fungilike their keystone isn't compatible with the ksa 4.2.1 installed in ~fungi/launch-env16:18
fungilooks like that's 2020/victoria era ksa16:19
fungiso hopefully before v2 api support was dropped16:19
clarkbI have not recently does attaches. I think when we cleaned up the Elasticseraches may have been the last time I did similar but that aws all removal/deletion?16:23
fungiin theory, detaching would have been similar16:24
funginova volume-detach16:24
clarkbin this case I think deleting the server implied that then I just had to delete the volume?16:24
fungioh, perhaps16:24
clarkbfungi: maybe see if you have any better luck with the venv in my homeidr?16:24
fungisame deal, "Unrecognized schema in response body."16:26
fungimakes me wonder if rackspace changed something in their keystone responses16:26
clarkbif you run with --debug or whatever the flag is do you get the response back to be able to see what it is doing?16:26
clarkbcareful pasting that output too16:27
clarkbit might include tokens/auth details16:27
fungino, --debug doesn't give me the response body (but it does show the json fro the earlier responses)16:28
clarkbmight need to hack the code to print the json response or run things under the debugger?16:29
fungilooks like it fails to parse the response and then raises keystoneauth1.exceptions.http.BadRequest16:29
clarkbhttp 400 is a failure response from rax though16:29
clarkbso is rax indicating it doesn't like a body content and responding with 400 or is "invalid schema" on the client side sort of hiding whatever caused the 40016:29
fungisame authentication parameters i was using with the cinderclient commands which worked16:30
clarkbfungi: looking at my command history I did just do a volume list then volume show then volume delete. No detaches16:31
fungireally strange since i'd expect cinderclient and novaclient to auth with ksa the same way16:31
clarkbfungi: I am able to do an openstack server show against rax. Any reason to not use osc?16:32
clarkbfungi: `openstack server add volume $server $volume`16:33
fungiyeah, because the cinder api is so old in rackspace that osc doesn't support it16:33
fungiand openstack server add volume claims the volume doesn't exist regardless of whether i specify it by name or uuid16:34
clarkb--os-volume-api 1 does adding that flag help?16:34
clarkbI see that in some of my volume commands to rax16:34
fungiInvalid volume client version '1'. must be one of: 2, 316:35
clarkbya so your osc is too new :) but really :(16:35
fungioh, i should use the osc in my venv16:35
fungiNo volume with a name or ID of '0ba0c1b7-f2df-42dd-a3a3-ac42585c5cb8' exists.16:36
clarkbya so osc is getting around the keystone issue but then having problems listing volumes16:37
fungicinder list in the same region and account shows:16:37
fungi| 0ba0c1b7-f2df-42dd-a3a3-ac42585c5cb8 | available | nb01.opendev.org/main01                              | 1024 | SATA        | false    |16:37
clarkbmaybe try by name?16:37
fungisame error message16:37
fungiNo volume with a name or ID of 'nb01.opendev.org/main01' exists.16:38
fungiopenstack volume list says:16:38
fungiNot Found (HTTP 404)16:38
fungiwhich i expect is part of the problem16:39
clarkbvolume list works for me iwth --os-volume-api 116:39
fungioh, indeed yours is old enough to return content16:40
fungithe ~fungi/launch-venv is not16:40
fungiokay, yeah your venv seems to have the magic16:41
fungisudo ~clarkb/venv/bin/openstack --os-cloud openstackci-rax --os-region-name DFW --os-volume-api 1 server add volume "nb01.opendev.org" "nb01.opendev.org/main01"16:41
fungithat worked16:41
clarkbI don't know why unfortunately16:44
clarkbpossibly just a sufficiently old client16:44
fungiyeah, i'll make a "super old osc" env in my homedir, thanks16:45
*** jpena is now known as jpena|off16:46
funginb01: /dev/mapper/main-main  2.0T  884G  1.2T  44% /opt16:50
funginb02: /dev/mapper/main-main  2.0T  686G  1.3T  35% /opt16:50
fungi#status log Added a 1TB volume to each of nb01 and nb02 in order to accommodate the recent increase in built nodepool images16:51
opendevstatusfungi: finished logging16:51
funginow i need to reboot my workstation and hope my upgraded openafs lkm is working so i can add that ceph mirror volume16:51
fungidkms still isn't building the openafs lkm :(18:02
fungiit logs compiler errors which return no hits from a web search18:03
fungii may need to find somewhere other than my computer to create the mirror volume from18:03
*** dviroel|lunch is now known as dviroel18:21
fungi/var/lib/dkms/openafs/ error: conflicting types for ‘grab_cache_page_write_begin’; have ‘struct page *(struct address_space *, long unsigned int,  unsigned int)’18:23
fungii wonder if it's this still: https://bugs.debian.org/94550618:23
fungiconfigure seems to have decided on using gcc-1118:24
fungii can't tell whether that's "the kernel compiler used for the headers" but i'm running a newer version of dkms than is mentioned there18:30
fungii have gcc-12 installed as well, so it clearly didn't just pick the latest one available on the system18:32
fungiand my /usr/bin/gcc is a symlink to gcc-1218:33
fungiso i guess it's not that bug18:33
clarkbfungi: I odn't have any good ideas18:57
fungii'm almost done adding it from mirror-update.o.o18:57
fungi#status log Added mirror volume in AFS for Ceph Quincy Debian/Ubuntu packages19:48
opendevstatusfungi: finished logging19:48
funginoonedeadpunk: ^19:48
fungiwe should be all clear for approving https://review.opendev.org/859327 if another infra-root has time to review it19:48
Clark[m]I'll look after lunch19:56
noonedeadpunkfungi: ice, thanks!19:58
clarkbfungi: noonedeadpunk  one thing needs to be cleaned up on that change20:32
clarkbdetails in gerrit20:32
opendevreviewDmitriy Rabotyagov proposed opendev/system-config master: Add Ceph Quincy mirror  https://review.opendev.org/c/opendev/system-config/+/85932720:33
clarkbhttps://github.com/mypyc/mypyc is a really interesting idea and maybe one of the more compelling reasons for type annotation sthat I've seen20:41
clarkbits frustratin gto use type annotations given all their limitations and lack of runtime checking. But if you can compile with them to something faster...20:41
clarkblooks like mypy continues to publish both sdists and python native wheels too so you keep slower portability but then platforms you've compiled for can take advantage of the quicker compiled executable20:43
*** dviroel is now known as dviroel|afk20:43
clarkbthe old apache process persists on review. I'll restart apache after this school run. That should be late neough in the day that fewre people notice20:47
*** dasm is now known as dasm|off21:31
fungisounds good21:56
clarkbok I'm going to restart the apache2 on review02 now 22:19
clarkbthats done. I can reach the esrvice and cert still looks good22:20
clarkbcorvus: 859162 in the openstack tenant says it is "waiting on repo state" for almost 2 hours22:20
clarkbI'm about to look but thought I would call that out22:21
clarkband it just started jobs22:23
clarkb526db5ba5dca4724ae20c3d839d5fd4e is the event I think22:26
clarkbit seems like zuul reports builds started well before the web ui showed that. Either I'm reading this wrong or not understanding it or maybe I have the wrong event22:37
clarkbThat said this is for openstack ansible and as we discovered they setup a ton of repos so maybe we were just waiting for a ton of merge events to finish22:37
clarkbok the "Executing jobs for change" log line does seem to align with what I saw in the UI and according to the code that seems to happen once everything else is ready22:47
opendevreviewMerged opendev/system-config master: Add Ceph Quincy mirror  https://review.opendev.org/c/opendev/system-config/+/85932722:49
clarkboh! I think I see what must've happened there was a change ahead in the queue that failed and we had a reset22:52
clarkbI just happened to look and missed the reset and only saw the aftermath22:52
clarkbya a job for 85916422:53
clarkbfungi: it looks like the maintainer of the mailman3 docker images is the mailman3 lead dev?23:03
fungioh, perhaps so23:03
clarkbthey've been active on gitlab for mailman more recently than those PRs and issues I filed https://gitlab.com/maxking23:08
clarkbI'm wondering if maybe we should reach out via some other method23:08
clarkbsomething like https://mail.python.org/mailman3/lists/mailman-developers.python.org/ or their user list?23:09
clarkbI don't want to pester, but the lynx install should be pretty straightforward23:10
clarkbfungi: you don't happen to already be on one of their lists yet do you?23:11
fungii'm not, no23:15
clarkbhttps://lists.mailman3.org/archives/list/mailman-users@mailman3.org/ thats the users list23:18
clarkbhttps://lists.mailman3.org/archives/list/mailman-users@mailman3.org/message/FK7VEHG5WBPEQCMOKYB4XRDO44UXDD6O/ and docker image maintainer does seem to respond to questions there23:19
clarkbI guess I can work on a draft of an email to that lis and see where it takes us23:20

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!