| *** dviroel|dr_appt is now known as dviroel | 01:16 | |
| *** dviroel is now known as dviroel|out | 01:31 | |
| *** ysandeep|out is now known as ysandeep | 02:39 | |
| *** ysandeep is now known as ysandeep|afk | 04:52 | |
| *** ysandeep|afk is now known as ysandeep | 05:35 | |
| dpawlik | hello folks. Small question: do you know what from time to time generates testrepository.subunit.gz that got broken content inside? | 06:57 |
|---|---|---|
| dpawlik | for example: https://42ef709f4885026f2696-9795c6ee70a51795a4d498c996d13136.ssl.cf2.rackcdn.com/periodic/opendev.org/openstack/oslo.db/master/propose-translation-update/e665a4f/testrepository.subunit.gz | 06:57 |
| dpawlik | or https://42ef709f4885026f2696-9795c6ee70a51795a4d498c996d13136.ssl.cf2.rackcdn.com/periodic/opendev.org/openstack/oslo.db/master/propose-translation-update/e665a4f/testrepository.subunit.gz or | 06:57 |
| dpawlik | https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_edc/02b003332cfd4988639efbd82a1935c24c4f5d0c/post/upstream-translation-update/edc2e39/testrepository.subunit.gz | 06:57 |
| opendevreview | Michal Nasiadka proposed openstack/project-config master: Add rockylinux-9-arm64 https://review.opendev.org/c/openstack/project-config/+/858554 | 07:24 |
| mnasiadka | clarkb: ack, updated | 07:24 |
| *** jpena|off is now known as jpena | 07:37 | |
| opendevreview | Dmitriy Rabotyagov proposed opendev/system-config master: Add Ceph Quincy mirror https://review.opendev.org/c/opendev/system-config/+/859327 | 08:23 |
| noonedeadpunk | fungi: would be great if ceph quincy mirror won't be forgotten as pacific was (https://review.opendev.org/c/opendev/system-config/+/823504) | 08:25 |
| noonedeadpunk | but I will try to take care of that by pinging others in some time | 08:27 |
| opendevreview | Rafal Lewandowski proposed openstack/diskimage-builder master: Added cloud-init growpart element https://review.opendev.org/c/openstack/diskimage-builder/+/855856 | 09:00 |
| *** soniya29 is now known as soniya29|afk | 11:09 | |
| *** dviroel|out is now known as dviroel | 11:19 | |
| fungi | dpawlik: is it only translation jobs? | 11:31 |
| fungi | noonedeadpunk: yep, it's on my list for this morning once i get caught up on e-mail et cetera | 11:32 |
| fungi | thanks for the reminder! | 11:32 |
| fungi | i'm also hoping to get to expanding the storage on x86 nodepool builders today | 11:35 |
| dpawlik | fungi: hey, hard to say. I see that after adding subunit file, it download such artifacts | 11:44 |
| *** ysandeep is now known as ysandeep|afk | 11:59 | |
| *** ysandeep|afk is now known as ysandeep|mtg | 12:33 | |
| *** soniya29|afk is now known as soniya29 | 13:18 | |
| fungi | dpawlik: it looks like each of the subunit files you linked is basically empty (just some headings). i have a feeling the translation jobs just shouldn't be collecting those subunit files | 13:22 |
| fungi | maybe once clarkb's around he might know more about what's going on there | 13:23 |
| dpawlik | fungi: thanks | 13:23 |
| dpawlik | I temporary skip the file for processing in logsender | 13:24 |
| dpawlik | it would be helpful to remove the workaround in the future in logsender :) | 13:24 |
| *** ysandeep|mtg is now known as ysandeep | 13:24 | |
| fungi | out of curiosity, what's doing the subunit parsing? | 13:25 |
| fungi | we used to do it with subunit2sql for the openstack-health backend | 13:25 |
| fungi | i don't remember the old logstash infrastructure doing any subunit parsing, so it's pretty cool if you've managed to add some in the new importer | 13:26 |
| *** dasm|off is now known as dasm | 13:46 | |
| *** ysandeep is now known as ysandeep|out | 14:01 | |
| clarkb | dpawlik: fungi: some jobs had stub subunit files to help record them in the old health server. I'm not sure if that is what is happening here but could explain it possibly. | 14:44 |
| clarkb | I'd have to look at the job and see what is generating the file (testr/stestr is the typical case but for the stub files I think they may have written them more directly with basic info) | 14:44 |
| dpawlik | fungi: Sorry for delay. TBH I don't know how the whole process for subunit works. Maybe lpiwowar knows better how it works or arxcruz | 14:50 |
| dpawlik | clarkb: ack. | 14:51 |
| arxcruz | fungi dpawlik the idea was to collect tempest info, tests running, what was failing, skipped, etc | 14:52 |
| arxcruz | i think we can check in the log parsing if the subunit is from tempest or not, and only parse tempest | 14:53 |
| arxcruz | or if it's empty | 14:53 |
| fungi | arxcruz: cool, so basically what we used to do with subunit2sql and the openstack-health dashboard before we stopped running it | 14:53 |
| clarkb | dpawlik: https://opendev.org/openstack/project-config/src/branch/master/roles/fetch-translations-subunit-output/README.rst theres a hint | 14:54 |
| dpawlik | clarkb: I see in codesearch that there is a dedicated --include parameter that will get the file | 14:56 |
| clarkb | dpawlik: https://opendev.org/openstack/openstack-zuul-jobs/src/branch/master/roles/prepare-zanata-client/files/common_translation_update.sh#L79-L88 as suspected it is directly generating a stub subunit file | 14:56 |
| clarkb | I think the file should be valid. | 14:56 |
| dpawlik | ah | 14:56 |
| dpawlik | that can be it | 14:57 |
| dpawlik | good point clarkb++ and fungi++ | 14:57 |
| clarkb | it is creatinga single entry subunit with a timestamp, test/job name, and status. It uses generate-subunit so this should be completely valid | 14:57 |
| dpawlik | also thanks arxcruz! | 14:57 |
| clarkb | any problems you've got are likely on the deserialization side given that | 14:57 |
| * clarkb fetches one of the files to see | 14:57 | |
| clarkb | yes the file I fetched appears to be perfectly valid. | 14:59 |
| fungi | cool, so just very small | 14:59 |
| clarkb | subunit-2to1 parses it properly and emits a more human readable version that indicates the info that that script is writing | 14:59 |
| clarkb | arxcruz: ^ I think the file is valid fwiw | 15:01 |
| clarkb | I didn't end up restarting apache yesterday afternoon. I'll try to remember better today | 15:40 |
| fungi | it may eventually clear up on its own too | 15:41 |
| *** dviroel is now known as dviroel|lunch | 16:01 | |
| fungi | clarkb: i've managed to use cinderclient to create new volumes for the builders in dfw, but can't get either osc nor novaclient to attach them | 16:14 |
| fungi | have you done that lately? | 16:14 |
| fungi | my shell history on bridge indicates i did it most recently with novaclient, but currently i'm getting this... | 16:16 |
| fungi | DEBUG (v2:61) Making authentication request to https://identity.api.rackspacecloud.com/v2.0/tokens | 16:16 |
| fungi | DEBUG (connectionpool:396) https://identity.api.rackspacecloud.com:443 "POST /v2.0/tokens HTTP/1.1" 400 76 | 16:17 |
| fungi | DEBUG (session:975) Request returned failure status: 400 | 16:17 |
| fungi | DEBUG (shell:821) Unrecognized schema in response body. (HTTP 400) | 16:17 |
| fungi | like their keystone isn't compatible with the ksa 4.2.1 installed in ~fungi/launch-env | 16:18 |
| fungi | looks like that's 2020/victoria era ksa | 16:19 |
| fungi | so hopefully before v2 api support was dropped | 16:19 |
| clarkb | I have not recently does attaches. I think when we cleaned up the Elasticseraches may have been the last time I did similar but that aws all removal/deletion? | 16:23 |
| fungi | in theory, detaching would have been similar | 16:24 |
| fungi | nova volume-detach | 16:24 |
| clarkb | in this case I think deleting the server implied that then I just had to delete the volume? | 16:24 |
| fungi | oh, perhaps | 16:24 |
| clarkb | fungi: maybe see if you have any better luck with the venv in my homeidr? | 16:24 |
| fungi | same deal, "Unrecognized schema in response body." | 16:26 |
| fungi | makes me wonder if rackspace changed something in their keystone responses | 16:26 |
| clarkb | if you run with --debug or whatever the flag is do you get the response back to be able to see what it is doing? | 16:26 |
| clarkb | careful pasting that output too | 16:27 |
| clarkb | it might include tokens/auth details | 16:27 |
| fungi | no, --debug doesn't give me the response body (but it does show the json fro the earlier responses) | 16:28 |
| clarkb | might need to hack the code to print the json response or run things under the debugger? | 16:29 |
| fungi | looks like it fails to parse the response and then raises keystoneauth1.exceptions.http.BadRequest | 16:29 |
| clarkb | http 400 is a failure response from rax though | 16:29 |
| fungi | right | 16:29 |
| clarkb | so is rax indicating it doesn't like a body content and responding with 400 or is "invalid schema" on the client side sort of hiding whatever caused the 400 | 16:29 |
| fungi | same authentication parameters i was using with the cinderclient commands which worked | 16:30 |
| clarkb | fungi: looking at my command history I did just do a volume list then volume show then volume delete. No detaches | 16:31 |
| fungi | really strange since i'd expect cinderclient and novaclient to auth with ksa the same way | 16:31 |
| clarkb | fungi: I am able to do an openstack server show against rax. Any reason to not use osc? | 16:32 |
| clarkb | fungi: `openstack server add volume $server $volume` | 16:33 |
| fungi | yeah, because the cinder api is so old in rackspace that osc doesn't support it | 16:33 |
| fungi | and openstack server add volume claims the volume doesn't exist regardless of whether i specify it by name or uuid | 16:34 |
| clarkb | --os-volume-api 1 does adding that flag help? | 16:34 |
| clarkb | I see that in some of my volume commands to rax | 16:34 |
| fungi | Invalid volume client version '1'. must be one of: 2, 3 | 16:35 |
| clarkb | ya so your osc is too new :) but really :( | 16:35 |
| fungi | oh, i should use the osc in my venv | 16:35 |
| fungi | No volume with a name or ID of '0ba0c1b7-f2df-42dd-a3a3-ac42585c5cb8' exists. | 16:36 |
| clarkb | ya so osc is getting around the keystone issue but then having problems listing volumes | 16:37 |
| fungi | cinder list in the same region and account shows: | 16:37 |
| fungi | | 0ba0c1b7-f2df-42dd-a3a3-ac42585c5cb8 | available | nb01.opendev.org/main01 | 1024 | SATA | false | | 16:37 |
| clarkb | maybe try by name? | 16:37 |
| fungi | same error message | 16:37 |
| fungi | No volume with a name or ID of 'nb01.opendev.org/main01' exists. | 16:38 |
| fungi | openstack volume list says: | 16:38 |
| fungi | Not Found (HTTP 404) | 16:38 |
| fungi | which i expect is part of the problem | 16:39 |
| clarkb | volume list works for me iwth --os-volume-api 1 | 16:39 |
| fungi | oh, indeed yours is old enough to return content | 16:40 |
| fungi | the ~fungi/launch-venv is not | 16:40 |
| fungi | okay, yeah your venv seems to have the magic | 16:41 |
| fungi | sudo ~clarkb/venv/bin/openstack --os-cloud openstackci-rax --os-region-name DFW --os-volume-api 1 server add volume "nb01.opendev.org" "nb01.opendev.org/main01" | 16:41 |
| fungi | that worked | 16:41 |
| fungi | thanks! | 16:41 |
| clarkb | I don't know why unfortunately | 16:44 |
| clarkb | possibly just a sufficiently old client | 16:44 |
| fungi | yeah, i'll make a "super old osc" env in my homedir, thanks | 16:45 |
| *** jpena is now known as jpena|off | 16:46 | |
| fungi | nb01: /dev/mapper/main-main 2.0T 884G 1.2T 44% /opt | 16:50 |
| fungi | nb02: /dev/mapper/main-main 2.0T 686G 1.3T 35% /opt | 16:50 |
| clarkb | thanks! | 16:51 |
| fungi | #status log Added a 1TB volume to each of nb01 and nb02 in order to accommodate the recent increase in built nodepool images | 16:51 |
| opendevstatus | fungi: finished logging | 16:51 |
| fungi | now i need to reboot my workstation and hope my upgraded openafs lkm is working so i can add that ceph mirror volume | 16:51 |
| fungi | brb | 16:52 |
| fungi | dkms still isn't building the openafs lkm :( | 18:02 |
| fungi | it logs compiler errors which return no hits from a web search | 18:03 |
| fungi | i may need to find somewhere other than my computer to create the mirror volume from | 18:03 |
| *** dviroel|lunch is now known as dviroel | 18:21 | |
| fungi | /var/lib/dkms/openafs/1.8.8.1/build/src/afs/LINUX/osi_compat.h:143:1: error: conflicting types for ‘grab_cache_page_write_begin’; have ‘struct page *(struct address_space *, long unsigned int, unsigned int)’ | 18:23 |
| fungi | i wonder if it's this still: https://bugs.debian.org/945506 | 18:23 |
| fungi | configure seems to have decided on using gcc-11 | 18:24 |
| fungi | i can't tell whether that's "the kernel compiler used for the headers" but i'm running a newer version of dkms than is mentioned there | 18:30 |
| fungi | i have gcc-12 installed as well, so it clearly didn't just pick the latest one available on the system | 18:32 |
| fungi | and my /usr/bin/gcc is a symlink to gcc-12 | 18:33 |
| fungi | so i guess it's not that bug | 18:33 |
| clarkb | fungi: I odn't have any good ideas | 18:57 |
| fungi | i'm almost done adding it from mirror-update.o.o | 18:57 |
| fungi | #status log Added mirror volume in AFS for Ceph Quincy Debian/Ubuntu packages | 19:48 |
| opendevstatus | fungi: finished logging | 19:48 |
| fungi | noonedeadpunk: ^ | 19:48 |
| fungi | we should be all clear for approving https://review.opendev.org/859327 if another infra-root has time to review it | 19:48 |
| Clark[m] | I'll look after lunch | 19:56 |
| noonedeadpunk | fungi: ice, thanks! | 19:58 |
| noonedeadpunk | *nice | 19:58 |
| clarkb | fungi: noonedeadpunk one thing needs to be cleaned up on that change | 20:32 |
| clarkb | details in gerrit | 20:32 |
| opendevreview | Dmitriy Rabotyagov proposed opendev/system-config master: Add Ceph Quincy mirror https://review.opendev.org/c/opendev/system-config/+/859327 | 20:33 |
| noonedeadpunk | done | 20:33 |
| clarkb | https://github.com/mypyc/mypyc is a really interesting idea and maybe one of the more compelling reasons for type annotation sthat I've seen | 20:41 |
| clarkb | its frustratin gto use type annotations given all their limitations and lack of runtime checking. But if you can compile with them to something faster... | 20:41 |
| clarkb | looks like mypy continues to publish both sdists and python native wheels too so you keep slower portability but then platforms you've compiled for can take advantage of the quicker compiled executable | 20:43 |
| *** dviroel is now known as dviroel|afk | 20:43 | |
| clarkb | the old apache process persists on review. I'll restart apache after this school run. That should be late neough in the day that fewre people notice | 20:47 |
| *** dasm is now known as dasm|off | 21:31 | |
| fungi | sounds good | 21:56 |
| clarkb | ok I'm going to restart the apache2 on review02 now | 22:19 |
| clarkb | thats done. I can reach the esrvice and cert still looks good | 22:20 |
| clarkb | corvus: 859162 in the openstack tenant says it is "waiting on repo state" for almost 2 hours | 22:20 |
| clarkb | I'm about to look but thought I would call that out | 22:21 |
| clarkb | and it just started jobs | 22:23 |
| clarkb | 526db5ba5dca4724ae20c3d839d5fd4e is the event I think | 22:26 |
| clarkb | it seems like zuul reports builds started well before the web ui showed that. Either I'm reading this wrong or not understanding it or maybe I have the wrong event | 22:37 |
| clarkb | That said this is for openstack ansible and as we discovered they setup a ton of repos so maybe we were just waiting for a ton of merge events to finish | 22:37 |
| clarkb | ok the "Executing jobs for change" log line does seem to align with what I saw in the UI and according to the code that seems to happen once everything else is ready | 22:47 |
| opendevreview | Merged opendev/system-config master: Add Ceph Quincy mirror https://review.opendev.org/c/opendev/system-config/+/859327 | 22:49 |
| clarkb | oh! I think I see what must've happened there was a change ahead in the queue that failed and we had a reset | 22:52 |
| clarkb | I just happened to look and missed the reset and only saw the aftermath | 22:52 |
| clarkb | ya a job for 859164 | 22:53 |
| clarkb | fungi: it looks like the maintainer of the mailman3 docker images is the mailman3 lead dev? | 23:03 |
| fungi | oh, perhaps so | 23:03 |
| clarkb | they've been active on gitlab for mailman more recently than those PRs and issues I filed https://gitlab.com/maxking | 23:08 |
| clarkb | I'm wondering if maybe we should reach out via some other method | 23:08 |
| clarkb | something like https://mail.python.org/mailman3/lists/mailman-developers.python.org/ or their user list? | 23:09 |
| clarkb | I don't want to pester, but the lynx install should be pretty straightforward | 23:10 |
| clarkb | fungi: you don't happen to already be on one of their lists yet do you? | 23:11 |
| fungi | i'm not, no | 23:15 |
| clarkb | https://lists.mailman3.org/archives/list/mailman-users@mailman3.org/ thats the users list | 23:18 |
| clarkb | https://lists.mailman3.org/archives/list/mailman-users@mailman3.org/message/FK7VEHG5WBPEQCMOKYB4XRDO44UXDD6O/ and docker image maintainer does seem to respond to questions there | 23:19 |
| clarkb | I guess I can work on a draft of an email to that lis and see where it takes us | 23:20 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!