| *** mlavalle has quit IRC | 00:05 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Install gzip instead of busybox-gzip on suse https://review.opendev.org/751919 | 00:36 |
|---|---|---|
| openstackgerrit | Merged openstack/diskimage-builder master: bootloader: remove dangling grubenv links https://review.opendev.org/750279 | 01:01 |
| *** elod has quit IRC | 02:29 | |
| *** elod has joined #opendev | 02:30 | |
| *** Topner has joined #opendev | 02:56 | |
| *** Topner has quit IRC | 03:00 | |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: ensure-kubernetes: workaround slow account creation https://review.opendev.org/752184 | 03:09 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: Retire Fedora 31 for 32 https://review.opendev.org/751975 | 03:10 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: ensure-docker: only run docker-setup.yaml when installed https://review.opendev.org/747062 | 03:10 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: update-json-file: add role to combine values into a .json https://review.opendev.org/746834 | 03:10 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: ensure-docker: Linaro MTU workaround https://review.opendev.org/747063 | 03:10 |
| *** ykarel|away has joined #opendev | 04:46 | |
| *** ykarel|away is now known as ykarel | 05:01 | |
| *** ysandeep|away is now known as ysandeep | 05:15 | |
| ianw | frickler / AJaeger: was wording the only issue with https://review.opendev.org/#/c/752184/ ? it would be good to get that stack in, but particularly the f31 drop | 05:29 |
| AJaeger | ianw: yes, I think so - gave my +2. | 05:57 |
| AJaeger | ianw: found a few nits in https://review.opendev.org/#/c/746834 | 06:02 |
| openstackgerrit | Merged openstack/project-config master: Cleanup x/devstack-plugin-glusterfs jobs https://review.opendev.org/752335 | 06:05 |
| *** qchris has quit IRC | 06:21 | |
| frickler | ianw: what about devstack? the f32 job isn't working there yet afaict, is the f31 job so broken that we should rather drop it than wait for f32? | 06:23 |
| ianw | frickler: at least for zuul-jobs, i think let's drop it rather than have people work on it | 06:24 |
| ianw | i would like to see it working with devstack, i think there has been some initial work | 06:24 |
| *** slaweq_ has joined #opendev | 06:25 | |
| frickler | ianw: oh, that only changes zuul-jobs, not retire f31 nodes completely, that's fine then | 06:30 |
| frickler | ianw: lyarwood started https://review.opendev.org/#/c/750292/ but there seems to be some issue with dhcp to instances | 06:31 |
| *** qchris has joined #opendev | 06:34 | |
| openstackgerrit | Merged zuul/zuul-jobs master: ensure-kubernetes: workaround slow account creation https://review.opendev.org/752184 | 06:39 |
| *** lpetrut has joined #opendev | 06:43 | |
| *** hashar has joined #opendev | 06:54 | |
| *** jaicaa has quit IRC | 07:01 | |
| *** jaicaa has joined #opendev | 07:04 | |
| openstackgerrit | Merged opendev/elastic-recheck master: Made parse_jenkins_failure a non static https://review.opendev.org/751999 | 07:21 |
| *** andrewbonney has joined #opendev | 07:22 | |
| *** ykarel_ has joined #opendev | 07:24 | |
| *** icey has quit IRC | 07:29 | |
| *** ykarel has quit IRC | 07:30 | |
| openstackgerrit | Pierre-Louis Bonicoli proposed zuul/zuul-jobs master: Use ansible_distribution* facts instead of ansible_lsb https://review.opendev.org/742310 | 07:31 |
| *** ykarel_ is now known as ykarel | 07:32 | |
| *** icey has joined #opendev | 07:35 | |
| *** elod has quit IRC | 07:39 | |
| *** elod has joined #opendev | 07:41 | |
| ianw | clarkb: 3.3.0 dib released, should build suse | 07:49 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: update-json-file: add role to combine values into a .json https://review.opendev.org/746834 | 07:53 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: ensure-docker: Linaro MTU workaround https://review.opendev.org/747063 | 07:53 |
| ianw | frickler/AJaeger: thanks... in the mean time haskell has updated and dropped f32 from the copr repo, and so the change is broken in the gate :/ | 07:57 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: Retire Fedora 31 for 32 https://review.opendev.org/751975 | 07:57 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: ensure-docker: only run docker-setup.yaml when installed https://review.opendev.org/747062 | 07:57 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: update-json-file: add role to combine values into a .json https://review.opendev.org/746834 | 07:57 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: ensure-docker: Linaro MTU workaround https://review.opendev.org/747063 | 07:57 |
| *** dtantsur|afk is now known as dtantsur | 07:58 | |
| *** moppy has quit IRC | 08:01 | |
| *** moppy has joined #opendev | 08:01 | |
| *** DSpider has joined #opendev | 08:07 | |
| openstackgerrit | zbr proposed zuul/zuul-jobs master: Add mode to packer https://review.opendev.org/752213 | 08:31 |
| openstackgerrit | zbr proposed zuul/zuul-jobs master: Add managed jobs to periodic-weekly https://review.opendev.org/748682 | 08:33 |
| AJaeger | ianw: oh fun ;( | 08:35 |
| AJaeger | frickler: could you review 751975, please? | 08:35 |
| *** slaweq_ is now known as slaweq | 08:48 | |
| openstackgerrit | zbr proposed opendev/bindep master: Allow bindep to mention stategy https://review.opendev.org/752417 | 09:29 |
| openstackgerrit | zbr proposed zuul/zuul-jobs master: bindep: perform serial install if needed https://review.opendev.org/693637 | 09:37 |
| *** slaweq has quit IRC | 09:38 | |
| *** slaweq has joined #opendev | 09:40 | |
| *** dtantsur has quit IRC | 09:59 | |
| *** dtantsur has joined #opendev | 10:01 | |
| openstackgerrit | zbr proposed zuul/zuul-jobs master: bindep: perform serial install if needed https://review.opendev.org/693637 | 10:06 |
| openstackgerrit | zbr proposed opendev/elastic-recheck master: Enable configuration via environment variables https://review.opendev.org/746890 | 10:07 |
| *** tosky has joined #opendev | 10:19 | |
| *** ykarel has quit IRC | 11:25 | |
| *** ykarel has joined #opendev | 11:25 | |
| *** slaweq has quit IRC | 11:38 | |
| *** slaweq has joined #opendev | 12:12 | |
| openstackgerrit | Dmitry Tantsur proposed openstack/diskimage-builder master: Rename duplicating 10-debian-minimal.bash https://review.opendev.org/752456 | 12:41 |
| *** iurygregory has quit IRC | 12:56 | |
| *** iurygregory has joined #opendev | 12:58 | |
| *** mrunge_ has joined #opendev | 13:27 | |
| *** mrunge has quit IRC | 13:28 | |
| openstackgerrit | Merged zuul/zuul-jobs master: Retire Fedora 31 for 32 https://review.opendev.org/751975 | 13:32 |
| *** hashar has quit IRC | 14:10 | |
| *** lpetrut has quit IRC | 14:14 | |
| openstackgerrit | zbr proposed opendev/elastic-recheck master: Enforce use of safe yaml loader https://review.opendev.org/752476 | 14:18 |
| *** mrunge_ is now known as mrunge | 14:22 | |
| openstackgerrit | zbr proposed opendev/elastic-recheck master: Replace testr with pytest https://review.opendev.org/752484 | 14:49 |
| *** priteau has joined #opendev | 14:59 | |
| openstackgerrit | zbr proposed zuul/zuul-jobs master: WIP: E208 work https://review.opendev.org/748606 | 15:04 |
| *** ysandeep is now known as ysandeep|away | 15:06 | |
| *** mlavalle has joined #opendev | 15:24 | |
| openstackgerrit | Merged opendev/elastic-recheck master: Enforce use of safe yaml loader https://review.opendev.org/752476 | 15:25 |
| *** ykarel is now known as ykarel|away | 15:30 | |
| *** hashar has joined #opendev | 15:44 | |
| *** dtantsur is now known as dtantsur|afk | 15:51 | |
| clarkb | ianw: thanks! looks like now we need to land a nodepool change to get a new nodepool image | 15:51 |
| clarkb | working on sorting that out next | 15:51 |
| *** ykarel|away has quit IRC | 16:05 | |
| openstackgerrit | zbr proposed zuul/zuul-jobs master: Add mode to packer https://review.opendev.org/752213 | 16:21 |
| openstackgerrit | zbr proposed zuul/zuul-jobs master: Add file mode to fetch-sphinx-tarball https://review.opendev.org/748606 | 16:29 |
| *** priteau has quit IRC | 16:50 | |
| *** hashar has quit IRC | 16:55 | |
| *** tosky has quit IRC | 17:00 | |
| *** andrewbonney has quit IRC | 17:19 | |
| fungi | our storyboard docker image promotion is apparently breaking on getting a jwt token: https://zuul.opendev.org/t/openstack/build/3fbf6bc2284d4799b8d9b76777c472b7/console#1/0/1/localhost | 17:28 |
| fungi | https://opendev.org/zuul/zuul-jobs/src/commit/c5c9f7d862c31b7b4689a11bbaab3d67dd6f9014/roles/promote-docker-image/tasks/main.yaml#L11-L22 | 17:29 |
| clarkb | fungi: we may have to create the storyboard image under our opendevorg ? | 17:29 |
| clarkb | and/or ensure we've got the correct creds in there | 17:29 |
| fungi | we were publishing images there until a few months ago | 17:30 |
| clarkb | oh I didn't realize storyboard was already publishing images | 17:30 |
| *** Topner has joined #opendev | 17:30 | |
| fungi | https://hub.docker.com/r/opendevorg/storyboard-api | 17:31 |
| clarkb | and updated within the last 3 days | 17:31 |
| fungi | the gate job is uploading them | 17:31 |
| clarkb | ya | 17:32 |
| fungi | it's the promote we're failing | 17:32 |
| fungi | if it were the creds at issue, i would expect them to be the same | 17:32 |
| clarkb | agreed. In this case we're manually manipulating the api to get a token then promote an image (rather than using the client itself) | 17:33 |
| clarkb | I guess they could've updated things breaking that | 17:33 |
| clarkb | I'm fairly sure we haven't updated things on our side but we should double check that too | 17:33 |
| clarkb | we may have to emulate what ansible is doing (or run it locally without the no_log: false stuff) and see whatthe error is | 17:34 |
| fungi | i wonder if this is impacting any other users of the job | 17:34 |
| fungi | zuul-promote-image worked today just fine | 17:36 |
| clarkb | https://zuul.opendev.org/t/zuul/build/df20f030b2254f92997d3fdc5eaee1f9 ya thats the most recent one | 17:37 |
| clarkb | was about 12 hours ago | 17:37 |
| clarkb | could be a one off | 17:38 |
| clarkb | docker servers just had a sad | 17:38 |
| clarkb | maybe we should try a reenqueue first and see if it is persistent? | 17:38 |
| fungi | i can try to reenqueue the change in promote, yeah | 17:38 |
| clarkb | and if it happens again then we try to reproduce locally so we can see what the failure is | 17:38 |
| fungi | okay, reenqueued now | 17:42 |
| fungi | running https://zuul.opendev.org/t/openstack/stream/92e8f3352fa84ba7b21167d22962970c?logfile=console.log | 17:43 |
| clarkb | https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/promote-docker-image/tasks/main.yaml#L11-L22 is the task we already retry it a bunch (so failures are maybe common?) | 17:44 |
| clarkb | that looks pretty easy to reproduce locally with ansible though whcih is good | 17:44 |
| fungi | https://zuul.opendev.org/t/openstack/build/92e8f3352fa84ba7b21167d22962970c | 17:45 |
| fungi | failed on the same task agaion | 17:45 |
| clarkb | we can watch the nodepool image promotion that should happen soon too | 17:47 |
| * clarkb looking to see what secrets we use for this job | 17:47 | |
| fungi | yeah, i wonder if we're merely referencing the wrong secret | 17:48 |
| clarkb | https://opendev.org/opendev/storyboard/src/branch/master/.zuul.yaml#L9-L23 is what we use | 17:48 |
| clarkb | https://opendev.org/opendev/system-config/src/branch/master/zuul.d/docker-images/base.yaml#L2-L16 same username there at least | 17:49 |
| fungi | yeah, just checked, storyboard-upload-opendev-image and storyboard-promote-opendev-image use the exact same vars/secret ref | 17:50 |
| openstackgerrit | Matt McEuen proposed openstack/project-config master: Switch airship's gerritbot channel https://review.opendev.org/752540 | 17:50 |
| clarkb | I think we need to take that task and run it locallywith the credentials to see what hte log details are | 17:51 |
| clarkb | maybe the return code is no longer 200 (201 would make sense possibly?) or similar change with the api | 17:52 |
| clarkb | fungi: the credentials appear to be in the normal location | 17:52 |
| clarkb | do you watn to try doing the local run or shoudl I spin up a local ansible venv for that? | 17:52 |
| clarkb | fungi: I'm reading the ansible docs and it looks like the token request should be a POST but I think we do a GET based on default uri module behavior | 17:57 |
| clarkb | https://docs.ansible.com/ansible/latest/modules/uri_module.html#parameter-method vs https://hub.docker.com/support/doc/how-do-i-authenticate-with-the-v2-api | 17:58 |
| clarkb | we should run it locally and reproduce the failure then add method: POST to the task and see if it works | 17:58 |
| *** hamalq has joined #opendev | 17:59 | |
| fungi | strange though, zuul-promote-image is parented on the same opendev-promote-docker-image that storyboard-promote-opendev-image is, and worked fine | 18:01 |
| clarkb | maybe it changed in the last 12 hours? | 18:03 |
| fungi | nah, the earlier failure was from days ago | 18:05 |
| fungi | this job has never actually run successfully: https://zuul.opendev.org/t/openstack/builds?job_name=storyboard-promote-opendev-image | 18:07 |
| fungi | dating back to december (when it was first added, i guess) | 18:07 |
| fungi | the oldest build we have ansible data from still is https://zuul.opendev.org/t/openstack/build/4702398d1aa249b1a8a1f27fabd2b3f3 which failed almost a week ago | 18:09 |
| fungi | same task | 18:09 |
| fungi | but i wouldn't be surprised if it's been failing that way ~forever | 18:10 |
| fungi | maybe the credential used in the storyboard jobs isn't authorized for whatever action it's trying to perform? | 18:10 |
| clarkb | the task that is running doesn't have any context like that yet | 18:11 |
| clarkb | its just authing in a basic fashion | 18:11 |
| clarkb | I've got a local play running now that works how I expect with bogus creds | 18:11 |
| clarkb | now to try with real creds | 18:11 |
| fungi | the upload role uses `docker login` instead, looks like | 18:13 |
| clarkb | I am not able to reproduce using real creds | 18:13 |
| clarkb | I get a token | 18:13 |
| clarkb | does the secret possibly have whitespace on either end of it? | 18:14 |
| fungi | maybe the way the credentials are encoded work in the docker login call but not with a uri task? trailing newline? | 18:14 |
| clarkb | maybe other things chomp that | 18:14 |
| fungi | hah, minds going to the same place | 18:14 |
| clarkb | fungi: maybe just reencrypt it since the utility for that chomps whitespace now I think | 18:15 |
| fungi | trying to decrypt it now | 18:15 |
| clarkb | that works too :) | 18:16 |
| fungi | mainly to work out what's there | 18:16 |
| clarkb | fwiw I think we should switch that to a POST method too but I won't go changing things until we figure this out | 18:16 |
| clarkb | corvus: ^ fyi do you knwo why we don't use method: POST to auth with docker hub? | 18:17 |
| fungi | yup! there's a trailing newline on the password string | 18:17 |
| fungi | okay, so we need to replace it with a reencrypted copy that has no newline | 18:18 |
| fungi | i already have the string here, so i can push that | 18:18 |
| clarkb | yup at least reproducing works locally when there is no trailing new line :) | 18:18 |
| fungi | i must have a change in some project which is taking gertty forever to sync things | 18:22 |
| clarkb | also interesting that that must mean docker tooling chomps | 18:24 |
| fungi | i think it's the command line parsing doing that | 18:24 |
| clarkb | ah | 18:24 |
| fungi | we inline the password in a command task string | 18:24 |
| fungi | so the shell just sees additional whitespace as a separator | 18:24 |
| fungi | on a related note, it would be nice to finally get https://review.opendev.org/714518 back | 18:25 |
| openstackgerrit | Matt McEuen proposed openstack/project-config master: Switch airship's gerritbot channel https://review.opendev.org/752540 | 18:27 |
| openstackgerrit | Jeremy Stanley proposed opendev/storyboard master: Correct docker image upload creds https://review.opendev.org/752545 | 18:31 |
| clarkb | the linaro mirror was off, I have started it. Noticed beacuse nodepool's image builds were unhappy | 18:42 |
| clarkb | we should maybe move nodepool's image builds to pull the arm wheels from an x86 mirror I'll look at that after lunch | 18:42 |
| fungi | ianw may have set up a netconsole stream for that already | 18:43 |
| clarkb | fungi: ^ fyi I think we'll need to reenqueue the nodepool change | 18:43 |
| clarkb | I smell lunch now, back in a bit to look at that more | 18:43 |
| fungi | clarkb: or approve your one-liner followup? | 18:43 |
| fungi | if my gertty ever gets done trying to consume all my swap space i'll review it | 18:44 |
| openstackgerrit | Merged opendev/infra-manual master: Restore Zuul v3 Migration Guide bits on secrets https://review.opendev.org/714518 | 18:44 |
| fungi | i should get around to switching to a vm with more than 1gb ram | 18:44 |
| openstackgerrit | Merged zuul/zuul-jobs master: Add mode to packer https://review.opendev.org/752213 | 18:52 |
| openstackgerrit | Merged openstack/project-config master: Switch airship's gerritbot channel https://review.opendev.org/752540 | 18:57 |
| fungi | discussion over in #openstack makes me realize that lower-constraints jobs don't get the benefit of our prebuilt wheelhouse in many cases because they can specify different versions from what the central requirements upper-constraints.txt does | 19:08 |
| fungi | like designate trying to install an old version of cffi with a new python interpreter | 19:09 |
| clarkb | especially if upper constraints skips versions | 19:09 |
| fungi | yeah | 19:14 |
| fungi | well, also upper-constraints is going to use them with relatively contemporary python interpreter versions | 19:14 |
| fungi | so even if it did at one time include cffi 1.11.5 it likely moved on to list a newer version before python3.8 even existed | 19:15 |
| fungi | and therefore would never have been built for cp38 | 19:15 |
| *** openstackgerrit has quit IRC | 19:21 | |
| *** openstackgerrit has joined #opendev | 19:26 | |
| openstackgerrit | Merged opendev/storyboard master: Correct docker image upload creds https://review.opendev.org/752545 | 19:26 |
| *** hashar has joined #opendev | 19:27 | |
| *** Topner has quit IRC | 19:29 | |
| clarkb | pypi's bandersnatch mirror is still not done rebuilding fwiw | 19:31 |
| fungi | no surprise there | 19:32 |
| clarkb | it will be interesting to see if it comes in under 12TB :) | 19:34 |
| clarkb | fungi: corvus I think https://review.opendev.org/#/c/750833/ is a safe followup to getting all the zuul and nodepool processes on zk tls | 19:47 |
| clarkb | https://review.opendev.org/#/c/750049/ addresses an annoying thing building new servers too | 19:48 |
| corvus | clarkb: lgtm; if we need to zk-shell we'll need to install and run it on a zkXX host after that | 20:28 |
| clarkb | yup, I've already got my venv on zk01 from previous debugging | 20:29 |
| corvus | kk, i think i used nb01 in the past; others may have as well | 20:30 |
| corvus | or nl01. or something starting with an n. | 20:30 |
| *** tosky has joined #opendev | 20:34 | |
| fungi | nl01. i'm also still able to run the nodepool cli from there for the moment | 20:45 |
| clarkb | you'll still be able to if you exec into the container | 20:46 |
| fungi | cool, i'll remember to use docker-compose exec for that in the future | 20:46 |
| clarkb | I've abandoned https://review.opendev.org/#/c/751872/ since we seem to be moving to f32 and not bother with f31 | 20:52 |
| *** hashar has quit IRC | 20:55 | |
| openstackgerrit | Merged opendev/system-config master: Block port 2181 on zookeeper hosts https://review.opendev.org/750833 | 21:07 |
| openstackgerrit | Thomas Bachman proposed openstack/project-config master: Remove openstack-python-jobs from GBP templates https://review.opendev.org/752575 | 21:41 |
| *** jhesketh has quit IRC | 21:55 | |
| *** slaweq has quit IRC | 22:07 | |
| *** tosky has quit IRC | 22:19 | |
| ianw | clarkb: did nb03 go off with the mirror? | 22:23 |
| clarkb | ianw: no | 22:23 |
| clarkb | or at least the nova api said it was up | 22:23 |
| clarkb | I didn't try to ssh in | 22:23 |
| johnsom | FYI, I just saw a docs job timeout as well. I know Julia mentioned this earlier in the week. Job setup, before the tox ran took 29.5 minutes. Much longer than I would have expected. The actual docs work only got two minutes to run before being killed. | 22:26 |
| clarkb | johnsom: can you link to a log? | 22:27 |
| clarkb | in TheJulia's case I think we confirmed that pdf generation was slow? | 22:27 |
| johnsom | https://api.us-east.open-edge.io:8080/swift/v1/AUTH_e02c11e4e2c24efc98022353c88ab506/zuul_opendev_logs_146/751918/2/check/openstack-tox-docs/1465196/job-output.txt | 22:27 |
| clarkb | johnsom: installing pdf prereqs was slow | 22:28 |
| johnsom | Yeah, that is what I see too | 22:28 |
| clarkb | I wonder if the ansible console log stuff will give us more info than what we get in the terminal console log | 22:28 |
| *** DSpider has quit IRC | 22:29 | |
| johnsom | 21 minutes in fact | 22:29 |
| clarkb | https://opendev.org/openstack/openstack-zuul-jobs/src/branch/master/roles/prepare-build-pdf-docs/tasks/main.yaml if I had to guess its the tex stuff | 22:29 |
| clarkb | since that usually expands to a billion packages, but confirming that with the ansible stuff would be good | 22:29 |
| johnsom | I really haven't seen this before, but thought I would mention it as someone else had also run into it. | 22:35 |
| *** mgagne has joined #opendev | 22:37 | |
| johnsom | https://zuul.openstack.org/builds?job_name=openstack-tox-docs&result=TIMED_OUT | 22:38 |
| johnsom | Seems to be a trend | 22:39 |
| fungi | maybe related to the recent change which added texlive-full to docs builds? | 22:40 |
| fungi | that drags in an insane number of dependencies | 22:40 |
| clarkb | fungi: ya I may have exagerated when I said a billion but its pretty close :) | 22:41 |
| clarkb | arg the console stuff doesn't work when we timeout | 22:42 |
| clarkb | I guess we can spelunk in the json | 22:42 |
| fungi | i'll see if i can find where that was | 22:42 |
| johnsom | It looks like 458 packages on one of those older runs | 22:42 |
| johnsom | clarkb I found it here on a previous failure: https://zuul.openstack.org/build/b218e3cc8dd24106b062d20a52dbba39/console | 22:42 |
| johnsom | My log the run hasn't finished yet | 22:42 |
| johnsom | Yeah, so 3GB of stuff if I read that correctly | 22:43 |
| clarkb | unfortunately no internal timing | 22:44 |
| fungi | https://review.opendev.org/738326 merged just over a week ago | 22:44 |
| clarkb | but we can probably get an idea for where the cost is if someone runs the same installation in a container or vm and pays attention | 22:44 |
| clarkb | another option would be to only install what is needed | 22:45 |
| clarkb | rather than every tex package | 22:45 |
| openstackgerrit | Mathieu Gagné proposed openstack/project-config master: Reduce available servers on inap-mtl01 https://review.opendev.org/752579 | 22:45 |
| johnsom | The title pretty much sums it up: TeX Live: metapackage pulling in all components of TeX Live | 22:46 |
| johnsom | It's a long list of dependencies. | 22:46 |
| fungi | and those dependencies also have dependencies | 22:46 |
| johnsom | So, yeah, we might want to be a bit more targeted here. | 22:46 |
| fungi | gmann: AJaeger: heads up, the texlive-full addition in the prepare-build-pdf-docs role may have started causing job timeouts | 22:47 |
| johnsom | Pulling this down for every docs job run can't be a good thing either "Need to get 3096 MB of archives." | 22:47 |
| fungi | yeah, any time i'm installing stuff on debian and it "recommends: texlive-full" i end up doing surgery because, holy heck that's a bunch of packages | 22:49 |
| johnsom | Sadly I don't have a focal instance (nor the cycles) to try to roll that back and see what is needed. | 22:49 |
| fungi | and unfortunately the failed build linked in the commit message has since expired out of swift, so not sure what the error was | 22:51 |
| johnsom | Yeah, at least that tells us it was the nova docs that triggered whatever the issue was. | 22:52 |
| fungi | i guess proposing a revert and then doing a dnm depends-on change for nova ought to recreate it | 22:53 |
| gmann | fungi: johnsom it is in many projects where doc job failed on Focal. | 22:55 |
| johnsom | gmann Yes: https://zuul.openstack.org/builds?job_name=openstack-tox-docs&result=TIMED_OUT | 22:56 |
| gmann | it is easily and 100% reproducible on Focal without that pckg | 22:56 |
| gmann | i mean without texlive-full | 22:56 |
| johnsom | Oh, I have no idea | 22:56 |
| clarkb | gmann: right but just because installing every tex package fixes it doesn't mean we need every package | 22:57 |
| gmann | we might shrink it to specific pckg which causing error but need to check that as pdf error was not clear | 22:57 |
| clarkb | there is likely a subset that would work | 22:57 |
| johnsom | I was just looking at the commit message for the patch that added texlive-full, it referenced a nova docs run. | 22:59 |
| openstackgerrit | Merged openstack/project-config master: Reduce available servers on inap-mtl01 https://review.opendev.org/752579 | 23:06 |
| clarkb | thinking about the tex stuff: I think what we should do is install the package list on bionic and xenial and diff the resulting package lists | 23:12 |
| clarkb | er bionic and focal | 23:12 |
| clarkb | I'm guessing on bionic it pulled in more for whatever reason | 23:13 |
| * clarkb spins up some containers | 23:13 | |
| clarkb | do our VMs disable recommends too? that may play into it | 23:14 |
| ianw | i think so, that has got us in trouble sometimes, openafs packages is one i can think of | 23:15 |
| johnsom | Looking at that console log it doesn't appear to be installing the suggested or recommended packages. | 23:17 |
| clarkb | ya its 182 vs 197 with bionic having more | 23:20 |
| clarkb | I'm installing them then will do a dpkg listing and diff it | 23:20 |
| *** jhesketh has joined #opendev | 23:31 | |
| clarkb | http://paste.openstack.org/show/798033/ nothing jumps out to me after I've removed overlap just with different versions | 23:36 |
| clarkb | my hunch is something moved from requires to recommends though | 23:37 |
| *** mlavalle has quit IRC | 23:44 | |
| fungi | yeah, probably need to try the pdf build steps on a focal node and then check what it actually says is missing | 23:51 |
| ianw | clarkb: mind a look at https://review.opendev.org/#/c/746272/ ; we should fix up our deb-docker usage | 23:54 |
| ianw | also https://review.opendev.org/#/c/746271/ which switches in base-jobs | 23:55 |
| ianw | then i can delete the old, non-updating bits from the mirror | 23:55 |
| ianw | (in case you forgot, we have to use deb-docker/<distro> because upstream keeps the same package names with different versions so we can't build one consolidated mirror) | 23:57 |
| clarkb | ianw: the non updating bits are what make things work right now ya? | 23:57 |
| ianw | yeah, it's just stuck in the past from the last time it happened to work | 23:59 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!