| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: [dnm] testing nested virt labels https://review.opendev.org/686510 | 00:00 |
|---|---|---|
| openstackgerrit | Paul Belanger proposed zuul/zuul master: Remove support for ansible 2.5 https://review.opendev.org/650431 | 00:01 |
| openstackgerrit | Paul Belanger proposed zuul/zuul master: Switch ansible_default to 2.8 https://review.opendev.org/676695 | 00:01 |
| openstackgerrit | Paul Belanger proposed zuul/zuul master: WIP: Support Ansible 2.9 https://review.opendev.org/674854 | 00:01 |
| *** dchen has joined #openstack-infra | 00:04 | |
| ianw | File "/usr/local/lib/python3.5/dist-packages/nodepool/driver/openstack/config.py", line 219, in load | 00:05 |
| ianw | top_label = full_config.labels[pl.name] | 00:05 |
| ianw | KeyError: 'nested-virt-ubuntu-bionic' | 00:05 |
| mnaser | someone made a nodepool oopsie? | 00:05 |
| mnaser | probably top level labels missing | 00:05 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-registry master: Add OCI containers/storage support https://review.opendev.org/686512 | 00:05 |
| mnaser | yep | 00:05 |
| mnaser | ianw: patch incoming | 00:06 |
| ianw | heh,just pushed one too | 00:07 |
| openstackgerrit | Ian Wienand proposed openstack/project-config master: Add top-level nested virt labels https://review.opendev.org/686513 | 00:07 |
| mnaser | yours was better tha mine | 00:08 |
| ianw | why didn't linting catch this though? | 00:08 |
| mnaser | how would it i dont think we lint nodepool config | 00:08 |
| ianw | we run config-validate? | 00:08 |
| mnaser | i ran into this too at some point i think | 00:09 |
| mnaser | but not with config-validate i think | 00:09 |
| ianw | i don't think we're running it | 00:10 |
| ianw | i tell a lie, it did run, and it passed | 00:20 |
| ianw | https://zuul.opendev.org/t/openstack/build/e7aeb2257151439c907cd2e53a6c8cc7/log/job-output.txt#642 | 00:21 |
| *** gyee has quit IRC | 00:21 | |
| *** yamamoto has joined #openstack-infra | 00:23 | |
| ianw | infra-root: if we can double-check https://review.opendev.org/#/c/686513/ should allow some nested-virt testing | 00:23 |
| mnaser | ianw: do you have the full stack trace | 00:24 |
| mnaser | ianw: all validate seems to do is just validate the schema from what it seems like | 00:24 |
| ianw | mnaser: http://paste.openstack.org/show/781064/ | 00:24 |
| mnaser | yeah it doesnt actually load it | 00:25 |
| mnaser | cause the failure happens in the driver level | 00:25 |
| *** yamamoto has quit IRC | 00:27 | |
| *** jtomasek has quit IRC | 00:30 | |
| *** markvoelker has joined #openstack-infra | 00:31 | |
| *** jbadiapa has joined #openstack-infra | 00:33 | |
| *** markvoelker has quit IRC | 00:35 | |
| openstackgerrit | Ian Wienand proposed zuul/nodepool master: [wip] validate diskimages have top-level labels https://review.opendev.org/686514 | 00:39 |
| ianw | mnaser: ^ something like that? can add a test case if it might work | 00:39 |
| mnaser | i was actaully working on something similar, i have a failing test case :p | 00:39 |
| mnaser | also notice for provider_dict loop exists above | 00:39 |
| mnaser | so you're looping one extra time | 00:40 |
| mnaser | and i think the normal schema validation throws an exception | 00:40 |
| *** jtomasek has joined #openstack-infra | 00:40 | |
| mnaser | ianw: at this point im actually wondering why the top level labels even exist | 00:41 |
| ianw | having it separate was sort of by design, keep each step a bit separate. but let's take yours if you've got something better | 00:41 |
| mnaser | if all we're going to do is just .. make them match | 00:42 |
| *** yamamoto has joined #openstack-infra | 00:47 | |
| *** armax has joined #openstack-infra | 00:48 | |
| openstackgerrit | Mohammed Naser proposed zuul/nodepool master: Added failing configuration check https://review.opendev.org/686515 | 01:00 |
| *** CrayZee has joined #openstack-infra | 01:10 | |
| *** panda|off has quit IRC | 01:12 | |
| *** panda has joined #openstack-infra | 01:12 | |
| *** shachar has quit IRC | 01:13 | |
| *** rfolco has quit IRC | 01:19 | |
| *** efried has quit IRC | 01:26 | |
| *** markvoelker has joined #openstack-infra | 01:26 | |
| *** efried has joined #openstack-infra | 01:26 | |
| *** signed8b_ has joined #openstack-infra | 01:27 | |
| *** dklyle has quit IRC | 01:27 | |
| *** rfolco has joined #openstack-infra | 01:27 | |
| *** dklyle has joined #openstack-infra | 01:28 | |
| *** harlowja has quit IRC | 01:30 | |
| *** signed8bit has quit IRC | 01:30 | |
| *** d34dh0r53 has quit IRC | 01:30 | |
| *** bradm has quit IRC | 01:30 | |
| *** bstinson has quit IRC | 01:30 | |
| *** markvoelker has quit IRC | 01:30 | |
| *** harlowja has joined #openstack-infra | 01:32 | |
| *** d34dh0r53 has joined #openstack-infra | 01:32 | |
| *** bstinson has joined #openstack-infra | 01:40 | |
| *** yamamoto has quit IRC | 01:42 | |
| *** markvoelker has joined #openstack-infra | 01:54 | |
| *** markvoelker has quit IRC | 01:58 | |
| *** markvoelker has joined #openstack-infra | 02:05 | |
| *** whoami-rajat has joined #openstack-infra | 02:09 | |
| *** jamesmcarthur has joined #openstack-infra | 02:13 | |
| *** yamamoto has joined #openstack-infra | 02:13 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Remove "failovermethod=priority" for Fedora (dnf) https://review.opendev.org/685876 | 02:17 |
| *** jonher has quit IRC | 02:19 | |
| *** yamamoto has quit IRC | 02:19 | |
| *** yamamoto has joined #openstack-infra | 02:20 | |
| *** ricolin has joined #openstack-infra | 02:27 | |
| *** roman_g has quit IRC | 02:34 | |
| *** markvoelker has quit IRC | 02:35 | |
| *** ykarel|away has joined #openstack-infra | 02:36 | |
| *** jonher has joined #openstack-infra | 02:41 | |
| *** exsdev has quit IRC | 02:44 | |
| *** rfolco has quit IRC | 02:45 | |
| *** exsdev has joined #openstack-infra | 02:49 | |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-registry master: Add OCI containers/storage support https://review.opendev.org/686512 | 02:52 |
| *** markvoelker has joined #openstack-infra | 03:06 | |
| openstackgerrit | Merged openstack/diskimage-builder master: yum-minimal : update mirrors for Centos 8 https://review.opendev.org/684439 | 03:22 |
| openstackgerrit | Merged openstack/diskimage-builder master: dib-python : handle centos 8 https://review.opendev.org/684441 | 03:22 |
| openstackgerrit | Ian Wienand proposed zuul/nodepool master: Validate openstack provider pool labels have top-level labels https://review.opendev.org/686514 | 03:30 |
| ianw | donnyd: ^ changes are making it through. the centos8 stack was very long, i think that with the rebasing etc and all the jobs running together probably caused undue contention | 03:33 |
| *** jamesmcarthur has quit IRC | 03:34 | |
| *** jamesmcarthur has joined #openstack-infra | 03:35 | |
| *** kjackal has joined #openstack-infra | 03:35 | |
| *** markvoelker has quit IRC | 03:36 | |
| *** markvoelker has joined #openstack-infra | 03:36 | |
| *** jamesmcarthur has quit IRC | 03:39 | |
| openstackgerrit | Ian Wienand proposed zuul/nodepool master: Validate openstack provider pool labels have top-level labels https://review.opendev.org/686514 | 03:53 |
| openstackgerrit | Ian Wienand proposed zuul/nodepool master: Validate openstack provider pool labels have top-level labels https://review.opendev.org/686514 | 04:01 |
| *** ykarel|away has quit IRC | 04:05 | |
| *** jamesmcarthur has joined #openstack-infra | 04:05 | |
| *** markvoelker has quit IRC | 04:07 | |
| *** jamesmcarthur has quit IRC | 04:12 | |
| *** gouthamr has quit IRC | 04:15 | |
| *** gouthamr has joined #openstack-infra | 04:16 | |
| openstackgerrit | Merged openstack/project-config master: Add top-level nested virt labels https://review.opendev.org/686513 | 04:24 |
| *** ykarel|away has joined #openstack-infra | 04:25 | |
| *** ykarel|away is now known as ykarel | 04:31 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Update locales for Centos 8 https://review.opendev.org/684442 | 04:34 |
| *** markvoelker has joined #openstack-infra | 04:35 | |
| *** markvoelker has quit IRC | 04:39 | |
| *** surpatil has joined #openstack-infra | 04:44 | |
| *** ykarel is now known as ykarel|afk | 04:50 | |
| *** jbadiapa has quit IRC | 04:52 | |
| *** soniya29 has joined #openstack-infra | 04:57 | |
| *** kjackal has quit IRC | 04:59 | |
| *** dchen has quit IRC | 05:00 | |
| *** dchen has joined #openstack-infra | 05:02 | |
| *** jamesmcarthur has joined #openstack-infra | 05:08 | |
| *** jamesmcarthur has quit IRC | 05:13 | |
| *** raukadah is now known as chandankumar | 05:13 | |
| *** odicha has joined #openstack-infra | 05:13 | |
| *** setuid is now known as zzZZzzetuid | 05:15 | |
| *** markvoelker has joined #openstack-infra | 05:19 | |
| *** iokiwi has quit IRC | 05:23 | |
| *** markvoelker has quit IRC | 05:24 | |
| *** iokiwi has joined #openstack-infra | 05:24 | |
| *** goldyfruit_ has joined #openstack-infra | 05:26 | |
| *** markvoelker has joined #openstack-infra | 05:28 | |
| *** markvoelker has quit IRC | 05:33 | |
| *** markvoelker has joined #openstack-infra | 05:37 | |
| *** markvoelker has quit IRC | 05:42 | |
| *** rlandy|bbl is now known as rlandy | 05:43 | |
| *** markvoelker has joined #openstack-infra | 05:47 | |
| *** ramishra has joined #openstack-infra | 05:47 | |
| *** markvoelker has quit IRC | 05:51 | |
| *** markvoelker has joined #openstack-infra | 05:56 | |
| *** ykarel|afk is now known as ykarel | 06:00 | |
| *** markvoelker has quit IRC | 06:01 | |
| *** markvoelker has joined #openstack-infra | 06:05 | |
| *** jamesmcarthur has joined #openstack-infra | 06:09 | |
| *** markvoelker has quit IRC | 06:10 | |
| openstackgerrit | Ian Wienand proposed openstack/project-config master: nodepool/elements : use abstracted commands https://review.opendev.org/686524 | 06:11 |
| *** markvoelker has joined #openstack-infra | 06:14 | |
| ianw | donnyd: so, first results on nested-vm nodes are in @ https://review.opendev.org/#/c/686510/ . it's hard to say ... not suddenly faster. however, it's a bit hard because we dont' capture the devstack/nova logs to see 100% that it chose kvm for the nested vm. feel free to update job and grab that info :) | 06:14 |
| *** jamesmcarthur has quit IRC | 06:15 | |
| *** markvoelker has quit IRC | 06:19 | |
| *** markvoelker has joined #openstack-infra | 06:23 | |
| AJaeger | ianw: https://review.opendev.org/#/c/686524/1 will break without nodepool release, correct? So, let's WIP it, please | 06:25 |
| *** markvoelker has quit IRC | 06:28 | |
| *** pgaxatte has joined #openstack-infra | 06:32 | |
| *** markvoelker has joined #openstack-infra | 06:33 | |
| *** spsurya has joined #openstack-infra | 06:33 | |
| *** dchen has quit IRC | 06:36 | |
| *** slaweq has joined #openstack-infra | 06:37 | |
| *** dchen has joined #openstack-infra | 06:38 | |
| *** markvoelker has quit IRC | 06:38 | |
| *** markvoelker has joined #openstack-infra | 06:42 | |
| *** udesale has joined #openstack-infra | 06:46 | |
| *** markvoelker has quit IRC | 06:46 | |
| *** markvoelker has joined #openstack-infra | 06:52 | |
| *** iurygregory has joined #openstack-infra | 06:54 | |
| *** markvoelker has quit IRC | 06:56 | |
| *** hwoarang has quit IRC | 06:58 | |
| *** markvoelker has joined #openstack-infra | 07:01 | |
| *** ramishra has quit IRC | 07:06 | |
| *** markvoelker has quit IRC | 07:06 | |
| *** jonher has quit IRC | 07:07 | |
| *** jonher has joined #openstack-infra | 07:07 | |
| *** rpittau|afk is now known as rpittau | 07:10 | |
| *** markvoelker has joined #openstack-infra | 07:11 | |
| *** hwoarang has joined #openstack-infra | 07:11 | |
| *** jamesmcarthur has joined #openstack-infra | 07:11 | |
| *** pcaruana has joined #openstack-infra | 07:15 | |
| *** markvoelker has quit IRC | 07:15 | |
| *** jamesmcarthur has quit IRC | 07:16 | |
| *** tosky has joined #openstack-infra | 07:18 | |
| *** markvoelker has joined #openstack-infra | 07:20 | |
| *** xek has joined #openstack-infra | 07:22 | |
| *** e0ne has joined #openstack-infra | 07:22 | |
| *** e0ne has quit IRC | 07:22 | |
| *** markvoelker has quit IRC | 07:24 | |
| *** xenos76 has joined #openstack-infra | 07:29 | |
| *** kjackal has joined #openstack-infra | 07:31 | |
| *** jpena|off is now known as jpena | 07:38 | |
| ianw | Cannot initiate the connection to mirror.ord.rax.opendev.org:80 (2001:4801:7824:103:be76:4eff:fe10:9c2). - connect (101: Network is unreachable) [IP: 23.253.216.84 80] | 07:43 |
| ianw | that's odd | 07:43 |
| *** markvoelker has joined #openstack-infra | 07:47 | |
| *** markvoelker has quit IRC | 07:52 | |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: Add OpenAPI description for enqueue, dequeue, autohold https://review.opendev.org/674257 | 07:53 |
| *** markvoelker has joined #openstack-infra | 07:57 | |
| *** pkopec has joined #openstack-infra | 08:01 | |
| *** markvoelker has quit IRC | 08:01 | |
| *** jamesmcarthur has joined #openstack-infra | 08:13 | |
| *** rcernin has quit IRC | 08:15 | |
| *** markvoelker has joined #openstack-infra | 08:15 | |
| *** gfidente has joined #openstack-infra | 08:16 | |
| *** jamesmcarthur has quit IRC | 08:18 | |
| openstackgerrit | Fabien Boucher proposed zuul/zuul master: Pagure - Support for branch creation/deletion https://review.opendev.org/685116 | 08:20 |
| *** markvoelker has quit IRC | 08:20 | |
| *** xek_ has joined #openstack-infra | 08:21 | |
| *** zbr is now known as zbr|ruck | 08:23 | |
| *** soniya29 has quit IRC | 08:24 | |
| *** xek has quit IRC | 08:24 | |
| *** lpetrut has joined #openstack-infra | 08:25 | |
| *** tkajinam has quit IRC | 08:27 | |
| *** yamamoto has quit IRC | 08:35 | |
| *** ociuhandu has joined #openstack-infra | 08:37 | |
| *** yamamoto has joined #openstack-infra | 08:37 | |
| *** derekh has joined #openstack-infra | 08:37 | |
| *** soniya29 has joined #openstack-infra | 08:41 | |
| *** markvoelker has joined #openstack-infra | 08:43 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Add security mirror override for debian-minimal https://review.opendev.org/686328 | 08:46 |
| *** markvoelker has quit IRC | 08:48 | |
| *** ociuhandu has quit IRC | 08:54 | |
| *** ociuhandu has joined #openstack-infra | 08:54 | |
| *** ociuhandu has quit IRC | 08:59 | |
| *** ociuhandu has joined #openstack-infra | 08:59 | |
| *** e0ne has joined #openstack-infra | 08:59 | |
| *** markvoelker has joined #openstack-infra | 09:02 | |
| *** markvoelker has quit IRC | 09:07 | |
| *** jamesmcarthur has joined #openstack-infra | 09:14 | |
| *** hrw has joined #openstack-infra | 09:14 | |
| hrw | morning | 09:14 |
| yoctozepto | morning infra | 09:15 |
| *** ramishra has joined #openstack-infra | 09:17 | |
| hrw | We have a problem in Kolla. Out periodic job builds and then push ~250 images in 5 jobs (centos/source, centos/binary, ubuntu/source, ubuntu/binary, debian/source). And then it timeouts in 2.5h | 09:18 |
| hrw | how much we can expand timeout? | 09:19 |
| *** ccamacho has joined #openstack-infra | 09:19 | |
| *** jamesmcarthur has quit IRC | 09:19 | |
| *** markvoelker has joined #openstack-infra | 09:21 | |
| *** kjackal has quit IRC | 09:22 | |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: Zuul Web: add /api/user/authorizations endpoint https://review.opendev.org/641099 | 09:23 |
| *** rfolco has joined #openstack-infra | 09:23 | |
| *** panda is now known as panda|bbl | 09:25 | |
| *** markvoelker has quit IRC | 09:25 | |
| *** yamamoto has quit IRC | 09:26 | |
| *** yamamoto has joined #openstack-infra | 09:30 | |
| *** yamamoto has quit IRC | 09:35 | |
| *** markvoelker has joined #openstack-infra | 09:39 | |
| *** yamamoto has joined #openstack-infra | 09:41 | |
| *** jbadiapa has joined #openstack-infra | 09:42 | |
| *** SergeyLukjanov has quit IRC | 09:44 | |
| *** markvoelker has quit IRC | 09:45 | |
| AJaeger | Why do you need to build those in a periodic job? | 09:45 |
| *** exsdev0 has joined #openstack-infra | 09:45 | |
| *** exsdev has quit IRC | 09:46 | |
| *** exsdev0 is now known as exsdev | 09:46 | |
| AJaeger | hrw: ^ | 09:47 |
| AJaeger | hrw: maximal allowed timeout is AFAIK 10800s | 09:47 |
| hrw | ok | 09:48 |
| AJaeger | and you can set that as post-timeout. | 09:48 |
| hrw | AJaeger: done in https://review.opendev.org/686561 | 09:48 |
| hrw | building is ~20 minutes. then push goes | 09:48 |
| AJaeger | then don't increase timeout, just post-timeout ;) | 09:49 |
| hrw | normal timeout was 2:50 anyway | 09:49 |
| AJaeger | you don't want it running 6hours ;) | 09:49 |
| AJaeger | mmh, so you're at the limit - I think we have some time to collect log files... | 09:49 |
| hrw | I can bear with 6h if it does the job ;( | 09:50 |
| AJaeger | hrw: so, what is the purpose of this? | 09:50 |
| hrw | AJaeger: our job is to provide images for people to deploy openstack | 09:50 |
| hrw | and with more and more images pushing time gets longer and longer | 09:51 |
| *** SergeyLukjanov has joined #openstack-infra | 09:51 | |
| AJaeger | and why -source and -binary? Isn't one enough? | 09:51 |
| mordred | wow - that's taking 2 hours just in the pushing to dockerhub | 09:52 |
| AJaeger | and you need those for all branches, correct? | 09:52 |
| hrw | AJaeger: yep. 5 branches now | 09:52 |
| AJaeger | so, 5 branches and 5 jobs pushing at the same time? | 09:52 |
| hrw | stable runs at other time iirc | 09:53 |
| AJaeger | hrw: no, all at basically same time | 09:53 |
| hrw | ok | 09:53 |
| hrw | in stable/* there are 4 jobs. we just added 5th one for master | 09:53 |
| AJaeger | hrw: stable starts at 6:01 UTC, others at 6:00 UTC | 09:53 |
| hrw | k | 09:54 |
| AJaeger | mordred: would it make sense to make them dependent on each other? So, serializing them? | 09:54 |
| AJaeger | (at least per branch) | 09:54 |
| AJaeger | Or use a semaphore so that only a few of them run at once? Or is parallel execution not a problem? | 09:54 |
| hrw | not a problem as long as dockerhub responds in time | 09:55 |
| AJaeger | hrw: So, let me ask differently (see my ramblings above): Do you think a single job run on its own would be better than 24 jobs running in parallel and uploading? | 09:55 |
| hrw | AJaeger: definitelly better | 09:56 |
| mordred | yeah - I think the issue here is simply pure volume of data uploaded in the job | 09:56 |
| *** kjackal has joined #openstack-infra | 09:56 | |
| AJaeger | hrw: consider using a semaphore for the jobs, see what we do with translations at https://opendev.org/openstack/project-config/src/branch/master/zuul.d/jobs.yaml#L1123 | 09:57 |
| hrw | AJaeger: thanks. will look | 09:57 |
| AJaeger | hrw: and how large are your images? | 09:57 |
| *** markvoelker has joined #openstack-infra | 09:58 | |
| hrw | let me check do I have just a set | 09:58 |
| hrw | will take a moment - removing extra ones | 09:59 |
| mordred | yeah - looking through the logs - it just seems like 2 hours is a really long time to just be stuck in pushing - so either the images are really big and it's just going to take a while, or dockerhub is throttling us | 09:59 |
| mordred | in which case potentially the semaphore would help, as serializing the uploads would allow more bandwidth per push | 10:00 |
| yoctozepto | AJaeger, mordred: yeah we are pushing a lot in parallel | 10:00 |
| openstackgerrit | Fabien Boucher proposed zuul/zuul master: Gitlab - Basic handling of merge_requests event https://review.opendev.org/685990 | 10:00 |
| mordred | so I think the semaphore is definitely worth a try - just to see if it helps with bandwidth between us and dockerhub | 10:00 |
| yoctozepto | best if we could semaphore post only | 10:00 |
| yoctozepto | but probably not achievable | 10:00 |
| AJaeger | yoctozepto: sure, you can add the semaphore only to the periodic jobs | 10:01 |
| AJaeger | yoctozepto: post or periodic? Is kolla pushing in post as well? | 10:01 |
| *** markvoelker has quit IRC | 10:02 | |
| *** mugsie has quit IRC | 10:03 | |
| yoctozepto | AJaeger: yeah the publishing/pushing in periodics is done inside post | 10:03 |
| yoctozepto | that's why hrw is increasing the post timeout | 10:03 |
| AJaeger | yoctozepto: Ah, post of a job run in periodic pipeline ;) | 10:04 |
| yoctozepto | indeed :-) | 10:04 |
| AJaeger | yoctozepto: I confused post-job with post pipeline ;( | 10:04 |
| yoctozepto | that ambiguity is confusing | 10:04 |
| yoctozepto | yeah, figured | 10:05 |
| * hrw rebuilds one set of images to check space need. | 10:05 | |
| yoctozepto | no problem | 10:05 |
| AJaeger | yoctozepto: so, no, you can only semaphore whole job | 10:05 |
| yoctozepto | "Semaphores, like most configuration items, are unique within a tenant" | 10:05 |
| hrw | as cleaning of not needed ones was wrong approach | 10:05 |
| yoctozepto | so we should prefix it with kolla- | 10:05 |
| AJaeger | yes, please | 10:05 |
| yoctozepto | AJaeger: yeah, read the docs | 10:05 |
| yoctozepto | thanks for the hints | 10:05 |
| yoctozepto | hrw: xD | 10:05 |
| *** mugsie has joined #openstack-infra | 10:05 | |
| yoctozepto | I will prepare the semaphore change in case we want to use it there | 10:06 |
| yoctozepto | AJaeger: any idea whether other projects are also pushing something to dockerhub? | 10:06 |
| AJaeger | so, I see images are around 350 MB - are you pushing 250 images per job - or 24 jobs, each pushes 10 images? | 10:07 |
| hrw | yoctozepto: thanks | 10:07 |
| hrw | AJaeger: each 250 | 10:07 |
| *** markvoelker has joined #openstack-infra | 10:07 | |
| hrw | ~250 | 10:07 |
| yoctozepto | if we could ask them to be nice to us | 10:07 |
| yoctozepto | AJaeger: but layers are pushed in an optimized way | 10:07 |
| AJaeger | That sounds insane - 24*250*350 MB. Guess it's more that kolla needs to be nice ;) | 10:07 |
| yoctozepto | and they share lots of layers | 10:07 |
| hrw | they can also answer "please run registry.openstack.org and point users there" | 10:08 |
| *** yamamoto has quit IRC | 10:08 | |
| yoctozepto | yeah, we would be happy to get local registry at some point | 10:08 |
| AJaeger | 24*250*350 gives me 2 TB ;( | 10:08 |
| yoctozepto | AJaeger: I already told you they share layers ;p | 10:08 |
| yoctozepto | this computation is wrong in the 250*350 part | 10:09 |
| yoctozepto | hrw is working on estimation | 10:09 |
| AJaeger | ok | 10:09 |
| hrw | give me ~30 minutes to build | 10:09 |
| yoctozepto | also, binary have less images | 10:09 |
| hrw | the good part is: we do not even try to publish arm64 images ;D | 10:10 |
| yoctozepto | AJaeger: re other projects - are you aware of any doing pushing to dockerhub? | 10:10 |
| hrw | yoctozepto: loci may. but they have small set | 10:10 |
| yoctozepto | hrw: it would need much more thought | 10:10 |
| hrw | yoctozepto: arm64 would need lot of work outside of openstack | 10:11 |
| hrw | wheel uploads etc | 10:11 |
| donnyd | ianw: so before we had 100% failure rates on FN | 10:11 |
| donnyd | Those look like passes to me | 10:11 |
| *** markvoelker has quit IRC | 10:12 | |
| *** yamamoto has joined #openstack-infra | 10:12 | |
| AJaeger | yoctozepto: you might want to check via codesearch.openstack.org | 10:12 |
| AJaeger | yoctozepto, hrw, are you building kubernetes as well as part of this? Aren't there images you can use? | 10:13 |
| *** yamamoto has quit IRC | 10:13 | |
| AJaeger | found on master "found 293 Dockerfiles" - wow | 10:13 |
| *** jamesmcarthur has joined #openstack-infra | 10:15 | |
| hrw | AJaeger: good point is that we are slowly working on deprecation or defining level of support for them. but still would build all probably | 10:16 |
| openstackgerrit | Merged openstack/diskimage-builder master: Update redhat-common pkg-map for centos 8 https://review.opendev.org/684449 | 10:16 |
| AJaeger | hrw: everything both as source and binary? | 10:17 |
| * AJaeger will be back later | 10:17 | |
| hrw | and for 3 distros as base | 10:17 |
| AJaeger | but why build those support things both as source and binary? Do you really need everything both? Anything you can optimize? | 10:18 |
| hrw | I think that this is something our project have to reconsider one day | 10:19 |
| *** jamesmcarthur has quit IRC | 10:19 | |
| donnyd | The only job that ran on FN was the centos build, but it was still failing before | 10:21 |
| *** prometheanfire has quit IRC | 10:22 | |
| *** ociuhandu has quit IRC | 10:22 | |
| hrw | fsck. my devmachine hang | 10:22 |
| *** ociuhandu has joined #openstack-infra | 10:22 | |
| *** prometheanfire has joined #openstack-infra | 10:23 | |
| *** factor has joined #openstack-infra | 10:26 | |
| * hrw out - bbl | 10:31 | |
| *** markvoelker has joined #openstack-infra | 10:35 | |
| *** markvoelker has quit IRC | 10:39 | |
| *** panda|bbl is now known as panda | 10:39 | |
| *** yamamoto has joined #openstack-infra | 10:41 | |
| *** soniya29 has quit IRC | 10:42 | |
| *** yamamoto has quit IRC | 10:43 | |
| *** kmarc has quit IRC | 10:46 | |
| *** kmarc has joined #openstack-infra | 10:46 | |
| *** jamesmcarthur has joined #openstack-infra | 10:47 | |
| *** ociuhandu has quit IRC | 10:47 | |
| *** ociuhandu has joined #openstack-infra | 10:48 | |
| *** jamesmcarthur has quit IRC | 10:51 | |
| *** ociuhandu has quit IRC | 10:52 | |
| *** dchen has quit IRC | 10:52 | |
| *** ociuhandu has joined #openstack-infra | 10:52 | |
| *** markvoelker has joined #openstack-infra | 10:53 | |
| *** yamamoto has joined #openstack-infra | 10:54 | |
| *** jaosorior has joined #openstack-infra | 10:54 | |
| hrw | set of centos/binary images (223 images + centos:7 + some dangling ones) is 16GB | 10:57 |
| *** markvoelker has quit IRC | 10:57 | |
| *** jpena is now known as jpena|lunch | 11:00 | |
| *** yamamoto has quit IRC | 11:00 | |
| *** kmarc has quit IRC | 11:01 | |
| *** markvoelker has joined #openstack-infra | 11:02 | |
| *** kmarc has joined #openstack-infra | 11:03 | |
| *** yamamoto has joined #openstack-infra | 11:03 | |
| *** markvoelker has quit IRC | 11:06 | |
| *** ociuhandu has quit IRC | 11:07 | |
| *** soniya29 has joined #openstack-infra | 11:08 | |
| *** ociuhandu has joined #openstack-infra | 11:08 | |
| *** jbadiapa has quit IRC | 11:09 | |
| yoctozepto | AJaeger: https://review.opendev.org/686569 | 11:11 |
| *** markvoelker has joined #openstack-infra | 11:11 | |
| *** ociuhandu has quit IRC | 11:12 | |
| openstackgerrit | Monty Taylor proposed zuul/zuul-registry master: WIP Consume typing from openstacksdk and keystoneauth https://review.opendev.org/686403 | 11:12 |
| *** ociuhandu has joined #openstack-infra | 11:14 | |
| *** zbr|ruck is now known as zbr|lunch | 11:15 | |
| *** markvoelker has quit IRC | 11:16 | |
| yoctozepto | AJaeger: about that semaphore - if I then want to change the number of concurrent jobs | 11:17 |
| yoctozepto | and have this semaphore in many branches | 11:17 |
| yoctozepto | how would that work | 11:18 |
| yoctozepto | would it prevent me from updating it? | 11:18 |
| *** ociuhandu has quit IRC | 11:18 | |
| donnyd | how much would it be to ask zuul if we could get the node provider where the build ran on the summary screen? Not sure if I am asking for a lot or a little | 11:19 |
| *** markvoelker has joined #openstack-infra | 11:21 | |
| *** ociuhandu has joined #openstack-infra | 11:25 | |
| *** markvoelker has quit IRC | 11:25 | |
| *** yamamoto has quit IRC | 11:36 | |
| *** markvoelker has joined #openstack-infra | 11:39 | |
| *** panda is now known as panda|eat | 11:43 | |
| *** markvoelker has quit IRC | 11:44 | |
| *** ccamacho has quit IRC | 11:45 | |
| *** jamesmcarthur has joined #openstack-infra | 11:48 | |
| *** pgaxatte has quit IRC | 11:48 | |
| *** markvoelker has joined #openstack-infra | 11:48 | |
| *** markvoelker has quit IRC | 11:49 | |
| *** markvoelker has joined #openstack-infra | 11:49 | |
| *** yamamoto has joined #openstack-infra | 11:51 | |
| *** lucasagomes has joined #openstack-infra | 11:52 | |
| *** EmilienM has quit IRC | 11:53 | |
| *** jamesmcarthur has quit IRC | 11:53 | |
| *** EmilienM has joined #openstack-infra | 11:54 | |
| *** ociuhandu has quit IRC | 11:55 | |
| AJaeger | yoctozepto: I think it's best define the semaphore in a repo that is not branched - let's see what others say. Fallback is project-config... | 11:56 |
| *** ociuhandu has joined #openstack-infra | 11:57 | |
| *** ociuhandu has quit IRC | 11:59 | |
| *** ociuhandu has joined #openstack-infra | 11:59 | |
| *** yamamoto has quit IRC | 12:00 | |
| *** jpena|lunch is now known as jpena | 12:00 | |
| *** ociuhandu has quit IRC | 12:03 | |
| *** ociuhandu has joined #openstack-infra | 12:04 | |
| yoctozepto | AJaeger: ok, replied | 12:06 |
| *** kukacz has joined #openstack-infra | 12:09 | |
| *** ociuhandu has quit IRC | 12:09 | |
| *** spsurya has quit IRC | 12:10 | |
| *** zzZZzzetuid is now known as setuid | 12:10 | |
| *** pgaxatte has joined #openstack-infra | 12:14 | |
| *** jamesmcarthur has joined #openstack-infra | 12:15 | |
| *** udesale has quit IRC | 12:16 | |
| *** udesale has joined #openstack-infra | 12:16 | |
| *** goldyfruit_ has quit IRC | 12:22 | |
| *** rh-jelabarre has joined #openstack-infra | 12:23 | |
| *** signed8b_ has quit IRC | 12:24 | |
| *** rfolco has quit IRC | 12:24 | |
| *** rfolco has joined #openstack-infra | 12:25 | |
| openstackgerrit | Monty Taylor proposed openstack/openstack-zuul-jobs master: Remove a reference to cgit from comment https://review.opendev.org/686586 | 12:26 |
| *** yamamoto has joined #openstack-infra | 12:30 | |
| yoctozepto | AJaeger: it could be nice to get a branchless repo for kolla for sure | 12:31 |
| yoctozepto | mgoddard, hrw, mnasiadka: ^ | 12:31 |
| mnasiadka | you mean a separate repo for some of the CI stuff? | 12:32 |
| AJaeger | yoctozepto: it's not worth doing it for this yet... | 12:32 |
| mgoddard | yoctozepto: a repo just for this semaphore? | 12:32 |
| mgoddard | yoctozepto: before we go down this path, are we sure it's the right solution? | 12:34 |
| mgoddard | what is the problem we are trying to solve? | 12:34 |
| yoctozepto | hmm, yeah, for now just semaphore but the same collision exists for nodesets, though we don't really need to change them hmm | 12:34 |
| *** rlandy has joined #openstack-infra | 12:34 | |
| *** ociuhandu has joined #openstack-infra | 12:34 | |
| yoctozepto | AJaeger: how can we estimate the effect of semaphore without waiting 24h for each test? | 12:35 |
| *** nhicher has quit IRC | 12:35 | |
| *** nhicher has joined #openstack-infra | 12:36 | |
| yoctozepto | mgoddard: we are trying to solve 'oh why are 2 hours not enough to push those images' | 12:37 |
| AJaeger | yoctozepto: you have to wait... | 12:37 |
| yoctozepto | so we must backport this to all the branches and then wait | 12:37 |
| mgoddard | yoctozepto: what is the difference between a normal build job vs. publish? | 12:37 |
| yoctozepto | and then get issues | 12:37 |
| yoctozepto | :D | 12:37 |
| *** panda|eat is now known as panda | 12:37 | |
| mgoddard | (in time) | 12:37 |
| yoctozepto | mgoddard: 2 hours of pushing not enough | 12:38 |
| yoctozepto | we have 2 hour limit on the push | 12:38 |
| yoctozepto | and this timeouts | 12:38 |
| mgoddard | yoctozepto: we are definitely not building during that time? | 12:38 |
| yoctozepto | hrw knows the details as well, I have to leave | 12:38 |
| yoctozepto | mgoddard: definitely | 12:38 |
| yoctozepto | this is POST | 12:38 |
| yoctozepto | till later | 12:38 |
| *** whoami-rajat has quit IRC | 12:38 | |
| hrw | mgoddard: building is 20 minutes, then pushing | 12:39 |
| *** xek_ has quit IRC | 12:39 | |
| mgoddard | hrw: it definitely takes longer than 20 minutes to build kolla images | 12:39 |
| *** xek_ has joined #openstack-infra | 12:40 | |
| *** ociuhandu has quit IRC | 12:40 | |
| *** markvoelker has quit IRC | 12:41 | |
| hrw | mgoddard: https://cb91710b8557a2f6220d-98f49207ca8d4e649788d064c2e22814.ssl.cf2.rackcdn.com/periodic/opendev.org/openstack/kolla/master/kolla-publish-debian-source/747d878/job-output.txt | 12:41 |
| mgoddard | hrw: yeah, looking at one now | 12:41 |
| *** xek_ has quit IRC | 12:42 | |
| mgoddard | https://openstack.fortnebula.com:13808/v1/AUTH_e8fd161dc34c421a979a9e6421f823e9/zuul_opendev_logs_b5e/periodic-stable/opendev.org/openstack/kolla/stable/queens/kolla-publish-centos-source/b5e20fb/job-output.txt | 12:42 |
| *** xek_ has joined #openstack-infra | 12:42 | |
| mgoddard | that one takes 1h20 to push | 12:42 |
| *** whoami-rajat has joined #openstack-infra | 12:42 | |
| hrw | mgoddard: some layers were already present at hub probably so they were reused | 12:43 |
| mgoddard | ok | 12:43 |
| *** xek_ has quit IRC | 12:43 | |
| hrw | I am swowly thinking of building those images locally, pushing to hub, waiting for periodic to push own, remove my ones | 12:44 |
| *** xek_ has joined #openstack-infra | 12:44 | |
| mgoddard | I imagine it depends a lot on which cloud provider | 12:44 |
| hrw | so it would reuse layers present on hub probably | 12:44 |
| mgoddard | hrw: I don't think you have the key for kolla docker namespace | 12:44 |
| *** kukacz has quit IRC | 12:44 | |
| fungi | yes, network bandwidth availability is one of our biggest variables between providers | 12:44 |
| hrw | mgoddard: right. no shared layers between users | 12:44 |
| *** kukacz has joined #openstack-infra | 12:45 | |
| AJaeger | fungi: and they are pushing 297 images (on master) for each job - and have 5 jobs run in parallel on master and 4 jobs for each stable branch... | 12:47 |
| AJaeger | fungi: that's why I proposed to experiment some rate limit - using a semaphore | 12:47 |
| *** jamesmcarthur has quit IRC | 12:47 | |
| *** jamesmcarthur has joined #openstack-infra | 12:47 | |
| AJaeger | mgoddard: as I said earlier: I suggest you reduce those image pushes as well, I see redis, kubernetes etc. build and pushed - are those really needed in both -source and -binary - or at all? | 12:48 |
| mgoddard | AJaeger: I was just looking at our image list | 12:48 |
| mgoddard | we have deprecated some, but maybe we should just be bold and remove | 12:49 |
| mordred | mgoddard: boldness is rewarded | 12:49 |
| * mgoddard deprecates openstack | 12:49 | |
| openstackgerrit | Merged openstack/openstack-zuul-jobs master: Remove a reference to cgit from comment https://review.opendev.org/686586 | 12:49 |
| mgoddard | too bold? | 12:50 |
| hrw | mgoddard: we should remove 'base' one - no one is using us. | 12:50 |
| *** yamamoto has quit IRC | 12:50 | |
| hrw | mgoddard: we should remove 'base' one - no one is using it | 12:50 |
| mgoddard | true | 12:50 |
| hrw | sorry, but I have to go handle some out of computer things | 12:50 |
| hrw | bbl | 12:50 |
| *** soniya29 has quit IRC | 12:51 | |
| fungi | also have you evaluated whether too much is being included in some of the images? 16gb seems a bit much for a minimal operating system, openstack python projects and their dependencies. but i've really never tried to measure it | 12:52 |
| fungi | which layers are the largest? | 12:53 |
| *** vesper11 has quit IRC | 12:53 | |
| mgoddard | we have done a few iterations of slimming down images, never with a huge amount of success | 12:53 |
| mgoddard | we could try again in Ussuri | 12:53 |
| mgoddard | I looked at multi-stage builds for source images, shaved a bit off | 12:54 |
| mnaser | is 16gb uploaded .. on every job? | 12:54 |
| mgoddard | mnaser: per-day | 12:54 |
| AJaeger | mnaser: that's my understanding. | 12:54 |
| mnaser | oh ack | 12:54 |
| mgoddard | publisher is periodic | 12:54 |
| *** vesper11 has joined #openstack-infra | 12:54 | |
| mnaser | that might be expensive over time :) | 12:54 |
| mgoddard | could be | 12:55 |
| AJaeger | So, every periodic job - and they run 5 on master, 4 on stable branches which gives 24 runs, so 384 GB per day | 12:55 |
| mgoddard | there may be shared layers | 12:55 |
| mgoddard | we've started a support matrix: https://storage.bhs1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_606/677500/18/check/openstack-tox-docs/606bd6a/docs/support_matrix.html | 12:56 |
| AJaeger | mgoddard, my understanding is that the 16 GB is without shared layers - otherwise you have much more (~300 images times 350 MB gives 1 TB) | 12:56 |
| *** udesale has quit IRC | 12:56 | |
| mgoddard | plan is to use it to help trim down our image list | 12:56 |
| mgoddard | and also categorise images | 12:56 |
| mgoddard | we could look at a less frequent publishing cycle for our non-core images | 12:57 |
| AJaeger | mgoddard: that is wrong, you build almanach (first line) in both binary and source! | 12:57 |
| mgoddard | AJaeger: I don't think so: https://opendev.org/openstack/kolla/src/branch/master/docker/almanach/almanach-base/Dockerfile.j2#L10 | 12:58 |
| *** ramishra has quit IRC | 12:58 | |
| *** markvoelker has joined #openstack-infra | 12:58 | |
| *** vesper11 has quit IRC | 12:59 | |
| AJaeger | mgoddard: http://zuul.opendev.org/t/openstack/build/78864ef8b2514e359e5e5fca858d9189/log/job-output.txt#61267 is ubuntu-binary | 13:00 |
| AJaeger | mgoddard: http://zuul.opendev.org/t/openstack/build/3a30769c75cb4c8c83845831f5125e97/log/job-output.txt#70310 is ubuntu-source | 13:00 |
| AJaeger | Aren't both building? | 13:00 |
| AJaeger | mgoddard: oh, I see the skipping now, so you start building and stop directly. confusing... | 13:01 |
| mgoddard | yes | 13:01 |
| mgoddard | fixed in master | 13:02 |
| *** rlandy is now known as rlandy|mtg | 13:02 | |
| mgoddard | we could EOL pike... | 13:03 |
| *** ociuhandu has joined #openstack-infra | 13:03 | |
| mgoddard | or at least stop publishing images | 13:03 |
| cgoncalves | hey folks. octavia tempest jobs are not being triggered in a stable/train patch: https://review.opendev.org/#/c/686541/ | 13:04 |
| cgoncalves | what am I missing? | 13:04 |
| mnasiadka | mgoddard: +1 | 13:04 |
| AJaeger | mgoddard: you have images in both binary and source like cron, is that correct? | 13:05 |
| mgoddard | AJaeger: correct | 13:05 |
| AJaeger | mgoddard: could you explain me why, please? | 13:05 |
| AJaeger | cgoncalves: the release team needs to branch devstack first to be able to do this | 13:05 |
| mgoddard | AJaeger: because it means I can just set the build type to binary/source and build all images I want | 13:05 |
| cgoncalves | AJaeger, understood. thank you | 13:06 |
| AJaeger | cgoncalves: that happens at later time, so for now we depend on good testing on master | 13:06 |
| AJaeger | mgoddard: my suggestion: Build them only once in the periodic job and not twice. | 13:06 |
| AJaeger | almanach for example is only binary as you said - why not go through with it? | 13:07 |
| mgoddard | AJaeger: sorry, I don't follow | 13:07 |
| AJaeger | looking at your documentation, it even says "The binary means that OpenStack will be installed from apt/yum. And the source means that OpenStack will be installed from source code. " | 13:07 |
| AJaeger | but for cron you have no source code, so why build such a container in the source periodic job? | 13:07 |
| mgoddard | fair point | 13:08 |
| *** roman_g has joined #openstack-infra | 13:08 | |
| mgoddard | having a cron image labelled as source makes for a better user experience - they can build all images with one type, and set one variable for the type to deploy | 13:09 |
| mgoddard | what might work is if we made those images essentially the same underneath, then tag twice in one job | 13:09 |
| *** ociuhandu has quit IRC | 13:09 | |
| mgoddard | i.e. a base layer that is not dependent on source vs binary | 13:09 |
| *** psachin has joined #openstack-infra | 13:10 | |
| AJaeger | mgoddard: but that's not what's in the docs - and not what you do, there are other examples where you only build in binary but not in source. I don't see a consistency here | 13:10 |
| fungi | would calling those undifferetiated images something like "independent" instead of either binary or source make for a reasonable user experience? | 13:10 |
| mnaser | infra-root: been doing some clean up on things in sjc1 for infra and looks like we're peaking ~50vms | 13:10 |
| mnaser | sorry took a while | 13:10 |
| fungi | thanks mnaser!!! | 13:10 |
| mgoddard | AJaeger: perhaps that's true | 13:11 |
| AJaeger | fungi, so, having three sets: binary OpenStack, source OpenStack, binary Support (your independent) | 13:11 |
| mgoddard | it's not a bad idea, and ties into our thinking around categorising images | 13:12 |
| mgoddard | I will add this as a topic of conversation for our next meeting, and we'll aim to improve the situation in Ussuri | 13:14 |
| AJaeger | So, my suggestion: Figure out how you can reduce those ~ 16 GB per job to be able to push the container images to dockerhub Bandwidth is not unlimited (neither with our providers nor with dockerhub) | 13:16 |
| AJaeger | mgoddard: thanks! | 13:16 |
| *** mriedem has joined #openstack-infra | 13:18 | |
| *** xek__ has joined #openstack-infra | 13:19 | |
| *** xek_ has quit IRC | 13:21 | |
| openstackgerrit | Merged openstack/project-config master: Override security mirror for Debian image builds https://review.opendev.org/686204 | 13:25 |
| *** dpawlik has quit IRC | 13:25 | |
| *** zbr|lunch is now known as zbr|ruck | 13:25 | |
| *** yamamoto has joined #openstack-infra | 13:26 | |
| *** vesper11 has joined #openstack-infra | 13:27 | |
| *** vesper11 has quit IRC | 13:31 | |
| *** vesper11 has joined #openstack-infra | 13:32 | |
| *** SurajPatil has joined #openstack-infra | 13:34 | |
| *** yamamoto has quit IRC | 13:36 | |
| *** surpatil has quit IRC | 13:36 | |
| *** goldyfruit_ has joined #openstack-infra | 13:40 | |
| *** yamamoto has joined #openstack-infra | 13:57 | |
| *** gmann_afk is now known as gmann | 13:58 | |
| *** ociuhandu has joined #openstack-infra | 14:02 | |
| *** woojay has joined #openstack-infra | 14:15 | |
| *** goldyfruit___ has joined #openstack-infra | 14:16 | |
| yoctozepto | AJaeger, fungi, mgoddard: yeah, we had those ideas already but they tend to come and go as their issues do | 14:17 |
| yoctozepto | we are getting more mature each release :-) | 14:18 |
| *** goldyfruit_ has quit IRC | 14:19 | |
| *** surpatil has joined #openstack-infra | 14:24 | |
| *** ricolin_ has joined #openstack-infra | 14:25 | |
| *** SurajPatil has quit IRC | 14:26 | |
| *** ricolin has quit IRC | 14:26 | |
| *** rlandy|mtg is now known as rlandy | 14:32 | |
| *** jbadiapa has joined #openstack-infra | 14:35 | |
| *** goldyfruit_ has joined #openstack-infra | 14:36 | |
| *** odicha has quit IRC | 14:38 | |
| openstackgerrit | Merged openstack/project-config master: python-dracclient zuulv3 changes https://review.opendev.org/686409 | 14:39 |
| *** goldyfruit___ has quit IRC | 14:39 | |
| openstackgerrit | Fabien Boucher proposed zuul/zuul master: Gitlab - Basic handling of merge_requests event https://review.opendev.org/685990 | 14:39 |
| openstackgerrit | Tristan Cacqueray proposed zuul/nodepool master: Switch to fedora-30 for the openshift integration job https://review.opendev.org/686737 | 14:40 |
| clarkb | when this kolla stuff has come up in the past it wasmy understanding that for the source images all of the build objects remain in the images as do the build deps | 14:40 |
| clarkb | this is one reason for their large size | 14:41 |
| clarkb | building in a separate image andcopying to the image you want to publish is how we deal with that on infra and zuul images | 14:41 |
| paladox | corvus fungi https://gerrit-review.googlesource.com/c/gerrit/+/239436 is the deadlock workaround! | 14:42 |
| *** ricolin_ has quit IRC | 14:49 | |
| *** ociuhandu has quit IRC | 14:51 | |
| *** ociuhandu has joined #openstack-infra | 14:52 | |
| *** bnemec is now known as beekneemech | 14:54 | |
| *** ociuhandu has quit IRC | 14:56 | |
| *** chandankumar is now known as raukadah | 14:57 | |
| *** xenos76 has quit IRC | 14:58 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-registry master: Fix merge error in streaming support https://review.opendev.org/686505 | 15:00 |
| *** ociuhandu has joined #openstack-infra | 15:04 | |
| *** arxcruz|ruck is now known as arxcruz|rover | 15:08 | |
| *** jamesmcarthur has quit IRC | 15:09 | |
| *** ociuhandu has quit IRC | 15:09 | |
| openstackgerrit | Merged zuul/zuul-registry master: Fix merge error in streaming support https://review.opendev.org/686505 | 15:11 |
| *** jamesmcarthur has joined #openstack-infra | 15:11 | |
| *** iurygregory has quit IRC | 15:12 | |
| corvus | infra-root: i have 2 production changes that require babysitting which i would like to do today: https://review.opendev.org/686049 https://review.opendev.org/686298 they both could use more +2s | 15:14 |
| *** ykarel is now known as ykarel|afk | 15:15 | |
| * prometheanfire is trying to find where the redirects for releases.openstack.org/constraints/upper/BRANCH_NAME are set | 15:15 | |
| * prometheanfire wants to add ussuri since reqs is branching | 15:15 | |
| clarkb | prometheanfire: I think it is an htaccess file in the releases repo | 15:16 |
| mordred | corvus: those are both awesome changes | 15:18 |
| corvus | i'm excited about them | 15:18 |
| *** donnyd is now known as donnyd_afk | 15:19 | |
| prometheanfire | that could be | 15:19 |
| *** gyee has joined #openstack-infra | 15:22 | |
| *** mattw4 has joined #openstack-infra | 15:22 | |
| *** ociuhandu has joined #openstack-infra | 15:23 | |
| fungi | prometheanfire: https://opendev.org/openstack/releases/src/branch/master/doc/source/_templates/htaccess | 15:25 |
| prometheanfire | fungi: yep | 15:26 |
| prometheanfire | controled by openstack_releases/_redirections.py | 15:26 |
| *** pkopec has quit IRC | 15:28 | |
| *** kopecmartin is now known as kopecmartin|off | 15:30 | |
| openstackgerrit | Tristan Cacqueray proposed opendev/glean master: Import ipv6 configuration done by cloud-init https://review.opendev.org/686749 | 15:38 |
| *** jaosorior has quit IRC | 15:41 | |
| fungi | mordred: https://docs.python.org/3.9/whatsnew/3.8.html#new-modules looks interesting from a pbr perspective | 15:42 |
| clarkb | fungi: I tested that with osc | 15:42 |
| fungi | oh? | 15:43 |
| clarkb | it works but is slower on python2 and only slightly quicker with python3 | 15:43 |
| fungi | ahh | 15:43 |
| clarkb | there is a pypi backport for the other python versions | 15:43 |
| clarkb | I think that means we should definitely switch to it from pkg_resources once python2 is no longer supported | 15:43 |
| clarkb | but it won't be a major improvement performance wise | 15:43 |
| fungi | i wonder if the one in stdlib performs any better or if it's just the same routines | 15:43 |
| clarkb | aiui its the same | 15:44 |
| fungi | regardless, it does reduce the required dependencies for 3.8 onward | 15:44 |
| clarkb | the major underlying problem is the disk scan and version sort | 15:44 |
| clarkb | whihc the new thing keeps from pkg_resources | 15:44 |
| clarkb | it just does it slightly more efficiently | 15:44 |
| *** rpittau is now known as rpittau|afk | 15:45 | |
| openstackgerrit | Tristan Cacqueray proposed opendev/glean master: Import ipv6 configuration done by cloud-init https://review.opendev.org/686749 | 15:46 |
| *** kjackal has quit IRC | 15:47 | |
| *** ociuhandu has quit IRC | 15:47 | |
| *** ociuhandu has joined #openstack-infra | 15:48 | |
| *** rlandy is now known as rlandy|brb | 15:49 | |
| *** piotrowskim has quit IRC | 15:55 | |
| *** ociuhandu has quit IRC | 15:56 | |
| corvus | is it known and expected that the arm64 images haven't updated in 37+ days? | 15:57 |
| clarkb | if it is the buster image yes, if it is the other arm64 images then no | 15:58 |
| corvus | it's buster, stretch, bionic and xenial | 15:58 |
| *** SurajPatil has joined #openstack-infra | 15:58 | |
| corvus | http://paste.openstack.org/show/781093/ | 15:59 |
| *** xarses has joined #openstack-infra | 15:59 | |
| *** adriant has quit IRC | 15:59 | |
| *** iokiwi has quit IRC | 15:59 | |
| *** xarses has quit IRC | 15:59 | |
| *** andreww has quit IRC | 15:59 | |
| *** xarses has joined #openstack-infra | 16:00 | |
| clarkb | looks like the builder has filled its disk | 16:00 |
| clarkb | (buster is broken because its mirror is broken iirc) | 16:00 |
| *** pgaxatte has quit IRC | 16:01 | |
| *** donnyd_afk is now known as donnyd | 16:01 | |
| *** surpatil has quit IRC | 16:01 | |
| *** cmurphy is now known as cmorpheus | 16:01 | |
| corvus | does that cause uploads to fail? | 16:01 |
| corvus | i see connection timeouts to arm64ci-nrt1 | 16:02 |
| *** iokiwi has joined #openstack-infra | 16:02 | |
| clarkb | that cloud doesn't exist anymore and I guess the cleanups for it never merged | 16:02 |
| *** xarses_ has joined #openstack-infra | 16:02 | |
| corvus | okay, do linaro-cn1 and linaro-london exist? | 16:03 |
| clarkb | london does. cn1 was in a state of flux at one point I'm not fully up to date on its status | 16:03 |
| corvus | who would know? | 16:04 |
| *** xarses has quit IRC | 16:04 | |
| clarkb | ianw is typically most up to date on that. | 16:04 |
| clarkb | hrw: may also know | 16:04 |
| *** tosky has quit IRC | 16:05 | |
| corvus | i'll clean up the disk | 16:06 |
| *** jamesmcarthur has quit IRC | 16:06 | |
| *** ykarel|afk is now known as ykarel | 16:07 | |
| corvus | rm: cannot remove 'dib_build.QolQCiqi/mnt/proc/18/net/wireless': Operation not permitted | 16:09 |
| corvus | what's that about ^? | 16:10 |
| AJaeger | corvus: /proc mounted in chroot? | 16:10 |
| clarkb | AJaeger: ya Ithink this is related to dib leaking mounts when builds fail in some cases | 16:10 |
| AJaeger | corvus: you better umount what in the chroot | 16:10 |
| openstackgerrit | Merged opendev/system-config master: Switch the insecure-ci-registry to zuul-registry https://review.opendev.org/686298 | 16:10 |
| openstackgerrit | Merged opendev/system-config master: Use the gerrit driver for the googlesource connection https://review.opendev.org/686049 | 16:10 |
| clarkb | what I've done in the past is stop and disable the builder servers, reboot, rm those files, enable the service, reboot | 16:10 |
| corvus | oh yep, i just missed it in the mount output | 16:11 |
| clarkb | (rebooting is just easy mode clear out stale mounts) | 16:11 |
| *** jamesmcarthur has joined #openstack-infra | 16:11 | |
| *** SurajPatil has quit IRC | 16:12 | |
| openstackgerrit | James E. Blair proposed openstack/project-config master: Remove arm64ci-nrt1 (1/2) https://review.opendev.org/686760 | 16:13 |
| openstackgerrit | James E. Blair proposed openstack/project-config master: Remove arm64ci-nrt1 (2/2) https://review.opendev.org/686761 | 16:13 |
| *** mriedem is now known as mriedem_lunch | 16:14 | |
| *** yamamoto has quit IRC | 16:15 | |
| openstackgerrit | James E. Blair proposed opendev/system-config master: Remove arm64ci-nrt1 (3/2) https://review.opendev.org/686763 | 16:15 |
| * corvus is bad at fractions | 16:15 | |
| clarkb | 3/2 is a valid fraction | 16:16 |
| openstackgerrit | James E. Blair proposed opendev/system-config master: Remove arm64ci-nrt1 (3/2) https://review.opendev.org/686763 | 16:16 |
| AJaeger | corvus: I think you missed one line, could you double check https://opendev.org/opendev/system-config/src/branch/master/inventory/groups.yaml#L176 , please? | 16:18 |
| corvus | AJaeger: ah yeah, i was grepping for nrt1 | 16:19 |
| AJaeger | corvus: I checked codesearch for arm64ci | 16:19 |
| openstackgerrit | James E. Blair proposed opendev/system-config master: Remove arm64ci (3/2) https://review.opendev.org/686763 | 16:20 |
| *** jbadiapa has quit IRC | 16:21 | |
| corvus | #status log nb03 dib_tmp partition was full; cleaned and restarted | 16:22 |
| openstackstatus | corvus: finished logging | 16:22 |
| corvus | AJaeger: it looks like we should wait a little longer to merge the git cache directory change since these images are out of date | 16:23 |
| *** ykarel has quit IRC | 16:24 | |
| hrw | clarkb, corvus: linaro-cn1 is no more | 16:24 |
| hrw | linaro-london is the only linaro cloud now | 16:24 |
| *** michael-beaver has joined #openstack-infra | 16:24 | |
| openstackgerrit | Tristan Cacqueray proposed opendev/glean master: Import ipv6 configuration done by cloud-init for Red Hat distro https://review.opendev.org/686749 | 16:25 |
| AJaeger | corvus: fun ;( Thanks for checking | 16:26 |
| AJaeger | corvus: I don't find linaro-cn1 in codesearch, so that one seems removed cleanly - or did you find anything? | 16:28 |
| AJaeger | We still have : https://opendev.org/opendev/system-config/src/branch/master/playbooks/templates/clouds/bridge_all_clouds.yaml.j2#L157 | 16:28 |
| AJaeger | corvus: want to put that on top of your 3/2 change? ;) | 16:29 |
| hrw | I can send patch to gerrit if you want | 16:30 |
| mordred | clarkb: 3/0 isn't | 16:30 |
| corvus | hrw: that's okay, i'll go ahead and take care of it, thanks! | 16:31 |
| hrw | ok | 16:31 |
| mordred | corvus: maybe number that patch 4/2 | 16:31 |
| *** rlandy|brb is now known as rlandy | 16:31 | |
| *** goldyfruit_ has quit IRC | 16:32 | |
| corvus | mordred: i was really tempted | 16:32 |
| openstackgerrit | James E. Blair proposed opendev/system-config master: Remove linaro-cn1 https://review.opendev.org/686770 | 16:32 |
| *** e0ne has quit IRC | 16:32 | |
| corvus | but it doesn't relate to the others so it'd be confusing | 16:33 |
| hrw | corvus: so is there any other arm64 one than linaro-london now? | 16:34 |
| corvus | hrw: no, that looks to be the last | 16:34 |
| hrw | ok | 16:35 |
| openstackgerrit | Merged openstack/project-config master: Remove arm64ci-nrt1 (1/2) https://review.opendev.org/686760 | 16:35 |
| hrw | thanks for info | 16:35 |
| hrw | I hope that situation will improve | 16:36 |
| corvus | hrw: ++ | 16:36 |
| corvus | clarkb, AJaeger, mordred: unfortunately it looks like cn1 was removed from the nodepool config without letting nodepool clean up the images first | 16:36 |
| corvus | that means there is no nodepool server that can delete them from zk | 16:37 |
| hrw | corvus: sorry for that | 16:37 |
| mordred | corvus: bother | 16:37 |
| hrw | cn1 was pain in the ass anyway | 16:37 |
| *** roman_g has quit IRC | 16:37 | |
| corvus | hrw: it's not your fault -- it's something the infra-root folks should have caught in review | 16:37 |
| mordred | yeah | 16:37 |
| mordred | that's our bad - not yours | 16:38 |
| corvus | so we actually have to add it back | 16:38 |
| hrw | ;) | 16:38 |
| clarkb | corvus: will that work for cleaning up zk even if the cloud does not exist? | 16:38 |
| corvus | clarkb: it's supposed to. let's verify that works for arm64 before we go to the trouble. | 16:38 |
| clarkb | ++ | 16:39 |
| fungi | looks like 649773 is where we retired it back in april | 16:39 |
| openstackgerrit | Tristan Cacqueray proposed opendev/glean master: Set Type=Ethernet for RedHat system https://review.opendev.org/686474 | 16:39 |
| openstackgerrit | Tristan Cacqueray proposed opendev/glean master: Import ipv6 configuration done by cloud-init for Red Hat distro https://review.opendev.org/686749 | 16:39 |
| *** goldyfruit_ has joined #openstack-infra | 16:39 | |
| *** zbr|ruck has quit IRC | 16:39 | |
| Shrews | oops. yeah, supposed to set max-servers to 0 first before removing the provider, as documented at: https://zuul-ci.org/docs/nodepool/operation.html#removing-from-the-launcher | 16:41 |
| Shrews | but easy to forget | 16:41 |
| corvus | Shrews: i think the nodes are okay, but we still have image records | 16:41 |
| *** lpetrut has quit IRC | 16:41 | |
| corvus | but same thing | 16:41 |
| openstackgerrit | Tristan Cacqueray proposed opendev/glean master: Import ipv6 configuration done by cloud-init for Red Hat distro https://review.opendev.org/686749 | 16:41 |
| corvus | set images to [] | 16:41 |
| clarkb | tristanC: re the type ethernet, that is specific to static configurations. Did you confirm that static configurations don't use the type setting? | 16:42 |
| Shrews | corvus: ah yeah. that's covered in that doc too. i wonder if we could programmatically check for those changes and prevent them? | 16:43 |
| *** dpawlik has joined #openstack-infra | 16:44 | |
| *** xenos76 has joined #openstack-infra | 16:44 | |
| openstackgerrit | Merged zuul/zuul master: web: render log manifest consistently https://review.opendev.org/686307 | 16:44 |
| *** goldyfruit_ has quit IRC | 16:46 | |
| *** gfidente has quit IRC | 16:49 | |
| fungi | Shrews: we'd need a job which analyzed the diff within the change itself i expect. i guess you could load the configuration from both HEAD and HEAD^1 and make sure that any provider missing from HEAD only had max-servers:0 and images:[] in HEAD^1 | 16:49 |
| Shrews | fungi: i was thinking more within nodepool itself | 16:50 |
| fungi | oh, got it | 16:50 |
| fungi | during configuration reload? | 16:50 |
| *** zbr has joined #openstack-infra | 16:50 | |
| Shrews | yeah, but not sure how useful that would be, tbh. unless someone is actively monitoring logs to see the warning we would issue, it might go unnoticed | 16:50 |
| fungi | so if you see a provider disappear then first treat it as if you went to max-servers:0 and images:[] on the provider and then... wait for that cleanup to complete? | 16:51 |
| Shrews | fungi: that's an option i hadn't thought of | 16:51 |
| fungi | and defer replacing the configuration until it's all empty | 16:51 |
| *** yamamoto has joined #openstack-infra | 16:51 | |
| corvus | it would be good to have a warning, but we also don't want to make it impossible | 16:51 |
| corvus | fungi: hrm... that could work, but it could also surprise folks who just want nodepool to ignore a provider for a bit | 16:52 |
| fungi | yep | 16:52 |
| corvus | (that's not how we do it, but i could see someone doing that) | 16:52 |
| Shrews | yeah | 16:52 |
| fungi | maybe a provider toggle for disable:true or something? | 16:52 |
| fungi | which might be more convenient for people who want to temporarily disable a provider than removing it from configuration | 16:53 |
| corvus | fungi: we have paused | 16:53 |
| fungi | oh, right, seems like that already would meet their use case then | 16:53 |
| *** ykarel has joined #openstack-infra | 16:53 | |
| fungi | anyway, more of a discussion for #zuul | 16:54 |
| corvus | tdasilva: fyi i'm about to switch out the opendev intermediate image registry and there could be a brief outage that could cause image build jobs to fail -- i mention it because swift is a pretty heavy user. | 16:59 |
| clarkb | corvus: re your changes you listd earlier today, one of them involves a zuul scheduler restart right? | 17:00 |
| mordred | corvus: swift is a pretty heavy user of the image registry - and the image registry is a pretty heavy user of swift | 17:00 |
| corvus | mordred: it's a virtuous cycle | 17:00 |
| clarkb | that should pick up the url linking bug fix in the dashboard | 17:00 |
| corvus | clarkb: yep | 17:01 |
| clarkb | k, let me know if I can help (I think you got the reviews you needed already) | 17:01 |
| *** yamamoto has quit IRC | 17:02 | |
| clarkb | https://review.opendev.org/#/c/686237/ is anothe rchange to zuul managementwhich we might want to get in | 17:05 |
| clarkb | one thought is that could fix issues like the include role problem in persistent-firewall if we keep ansible up to date | 17:05 |
| *** derekh has quit IRC | 17:06 | |
| corvus | we may see more job failures when point releases happen. but we can merge it and see | 17:06 |
| clarkb | due to files moving? thats a good point. Hrm | 17:06 |
| clarkb | maybe we can add manage-ansible -u to our zuul-executor restart playbook? | 17:06 |
| clarkb | we do tend to update zuul often enough and we shouldn't introduce failures (that wouldn't happen otherwise) at that point | 17:07 |
| corvus | clarkb: yeah, that's a good idea | 17:07 |
| *** jpena is now known as jpena|off | 17:08 | |
| openstackgerrit | Ghanshyam Mann proposed openstack/devstack-gate master: Update grenade settings for stable/train https://review.opendev.org/686774 | 17:08 |
| *** ociuhandu has joined #openstack-infra | 17:09 | |
| tdasilva | corvus: thanks for the heads up (cc: timburke) | 17:11 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Update zuul ansible installs when restarting zuul https://review.opendev.org/686775 | 17:13 |
| clarkb | corvus: ^ something like that | 17:13 |
| corvus | problem with the registry | 17:13 |
| corvus | /usr/local/bin/python: error while loading shared libraries: libpython3.7m.so.1.0: cannot open shared object file: No such file or directory | 17:13 |
| corvus | that's what i see in docker logs | 17:13 |
| corvus | i'm very confused and am starting to think i don't understand how containers work | 17:13 |
| corvus | it's in a restart loop because of that | 17:14 |
| corvus | clarkb, fungi, mordred: ^ any ideas? | 17:14 |
| clarkb | is /usr/local/bin/python a python2 maybe? | 17:14 |
| clarkb | and we've somehow mixed up package built for python3 (links against it) with python2 container? | 17:14 |
| mordred | shouldn't be | 17:14 |
| corvus | (this is a container image that ran a test in ci) | 17:15 |
| *** lpetrut has joined #openstack-infra | 17:15 | |
| mordred | corvus: can you boot a bash shell with the same image? | 17:15 |
| corvus | mordred: yes | 17:16 |
| openstackgerrit | Ghanshyam Mann proposed openstack/devstack-gate master: Add gate job for stable/train https://review.opendev.org/686776 | 17:16 |
| corvus | 'python' gives me a repl and is 3.7.3 | 17:16 |
| mordred | uhm | 17:16 |
| corvus | "zuul-registry --help" works | 17:16 |
| corvus | i'm going to try some voodo docker-compose down/up | 17:17 |
| mordred | kk | 17:18 |
| mordred | we're not doing anything with docker-compose that should be doing anything | 17:18 |
| corvus | no joy, same error | 17:18 |
| mordred | corvus: is the command docker-compose is running to launch the docker logged anywhere? | 17:20 |
| corvus | mordred: i don't know, but 'docker inspect registrydocker_registry_1' may answer similar questions | 17:21 |
| corvus | mordred: if we want i can down/up with --verbose | 17:21 |
| corvus | i'll do that | 17:21 |
| mordred | corvus: le's try that | 17:21 |
| clarkb | other idea: we built the package on old version of python so then the imgae we copy pcakge into updated with newer so and the file hsa moved? | 17:22 |
| corvus | mordred: /tmp/log | 17:22 |
| corvus | clarkb: why does 'zuul-registry --help' not show the problem then? | 17:22 |
| clarkb | it might not call into anything that needs the linking (though ldd should check al lthat upfront iirc) | 17:23 |
| corvus | a difference between the ci test and this is the use of swift | 17:24 |
| corvus | why don't i try setting up the actual configuration and running it from my container | 17:24 |
| mordred | libpython3.7m.so.1.0 => /usr/local/lib/libpython3.7m.so.1.0 (0x00007f7036e69000) | 17:24 |
| mordred | that's the ldd of /usr/local/bin/python in the container | 17:25 |
| clarkb | mordred: and does that file exist? | 17:25 |
| mordred | it's almost like something is blanking an env setting or something and /usr/local/lib isn't in the LD_LIBRARY_PATH | 17:25 |
| mordred | yes | 17:25 |
| corvus | wait /var/registry/etc:/etc | 17:25 |
| mordred | and I agree with corvus - when I boot a shell in the container .... | 17:25 |
| corvus | that would do it | 17:25 |
| mordred | oh | 17:25 |
| mordred | yeah | 17:26 |
| mordred | ld.so.conf would get blown away | 17:26 |
| clarkb | oh clearing out the ldd config | 17:26 |
| mordred | yeah | 17:26 |
| corvus | okay, let me manually patch docker-compose.yaml and fix | 17:26 |
| mordred | kk | 17:26 |
| mordred | PHEW | 17:26 |
| corvus | we can bind-mount in files, yeah? | 17:27 |
| mordred | yes | 17:27 |
| corvus | that's just for clouds.yaml | 17:27 |
| mordred | ah - which is probably why this didn't show in CI | 17:27 |
| clarkb | ah yup since swift isn't used in ci | 17:28 |
| corvus | okay it's up | 17:29 |
| mordred | yay! | 17:29 |
| fungi | stepped away at the wrong time, and i caught up just in time to see it solved | 17:29 |
| corvus | fungi: likely story :) | 17:29 |
| mordred | fungi: tl;dr - don't bind mount a mostly empty dir over top of /etc | 17:30 |
| fungi | i got nerdsniped trying to find a good fudge recipe in my collection of ancient cookbooks | 17:30 |
| clarkb | fungi: did you have to test a bunch to determine which are good? | 17:31 |
| fungi | not yet, that's this evening ;) | 17:31 |
| *** jtomasek has quit IRC | 17:31 | |
| corvus | i've totally done A/B testing on brownies | 17:31 |
| mordred | corvus: there are tracebacks in the log for the registry container - are they expected? | 17:31 |
| corvus | mordred: yes, i'm hitting wrong urls | 17:32 |
| fungi | really most of the time waste was spent comparing my 1938 boston cooking school cookbook to the "modern" 1984 fanny farmer edition | 17:32 |
| mordred | corvus: awesome | 17:32 |
| fungi | in the end though i think i've settled on the 1963 good housekeeping, so can get back to python | 17:33 |
| *** psachin has quit IRC | 17:36 | |
| corvus | looking for changes to recheck | 17:37 |
| corvus | apparently all the registry changes are in merge conflict now | 17:40 |
| openstackgerrit | James E. Blair proposed zuul/zuul-registry master: DNM null commit for testing https://review.opendev.org/686791 | 17:40 |
| corvus | okay zuul-registry-build-image should upload to the intermediate registry | 17:41 |
| openstackgerrit | James E. Blair proposed zuul/zuul-registry master: DNM: second null commit for more testing https://review.opendev.org/686793 | 17:42 |
| corvus | and that (because the role is naive) should download from it | 17:42 |
| *** lpetrut has quit IRC | 17:49 | |
| *** mriedem_lunch is now known as mriedem | 17:51 | |
| * paladox just ate dominos | 17:54 | |
| Shrews | corvus: could your changes be causing the POST_FAILURE on the nodepool-build-image job, by chance? | 17:55 |
| fungi | Shrews: could it have tried to upload while stuff was down? | 17:55 |
| Shrews | fungi: possible. i just returned from lunch. happened about 35min ago | 17:56 |
| fungi | ~17:13-17:29z | 17:56 |
| *** efried has quit IRC | 17:56 | |
| fungi | as best i can piece together from the discussion timeline | 17:56 |
| corvus | i may have also restarted it once after that.... | 17:56 |
| *** ykarel is now known as ykarel|away | 17:56 | |
| Shrews | seems likely then | 17:56 |
| corvus | so maybe throw a couple more minutes onto the end of that window. sorry. | 17:57 |
| fungi | mostly wanting to be sure any related failures we see aren't from after it is thought to have stabilized | 17:57 |
| Shrews | no worries. just didn't want to get bogged down in chasing yet another random fail :) | 17:57 |
| corvus | just got a post failure on the z-r job | 17:57 |
| corvus | gah | 17:58 |
| corvus | ansible overwrote my fix | 17:59 |
| corvus | refixing and updating emergency file | 17:59 |
| fungi | so let's call it ~17:13-18:00z | 18:00 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-registry master: Add type annotations https://review.opendev.org/686249 | 18:02 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-registry master: Add support for skopeo copy https://review.opendev.org/686803 | 18:02 |
| *** ociuhandu has quit IRC | 18:08 | |
| AJaeger | the bug for review.o.o links in zuul.opendev.org is not fixed - is that to be expected? I thought I read in backscroll it would be part of the deployment? | 18:08 |
| *** ociuhandu has joined #openstack-infra | 18:08 | |
| clarkb | AJaeger: it requires a zuul-scheduler restart which has not yet been done. Only the intermediate registry was restarted aiui | 18:09 |
| AJaeger | understood - thanks | 18:10 |
| *** rfolco is now known as rfolco|bbl | 18:12 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-registry master: DNM null commit for testing https://review.opendev.org/686791 | 18:12 |
| openstackgerrit | James E. Blair proposed zuul/zuul-registry master: DNM: second null commit for more testing https://review.opendev.org/686793 | 18:13 |
| corvus | INFO:registry.api:Put manifest zuul/zuul-registry aa146c4af83743eeb805df6c5892ae6a_latest digest sha256:e6c35cd710660f41ba5c14a913d4beb7ecc608a055a169b1c9fc3ab709f1b14a | 18:22 |
| corvus | that looks promising | 18:22 |
| corvus | cool, a simple image build job that pushes to the intermediate registry works | 18:23 |
| corvus | that's the typical case -- the next thing is pulling from it, which we should see in a minute. | 18:23 |
| *** jamesmcarthur has quit IRC | 18:25 | |
| corvus | 2 successful pushes | 18:27 |
| corvus | w00t http://zuul.opendev.org/t/zuul/stream/c7e9ca7514ab44dea61333831071deb6?logfile=console.log just pulled an image | 18:31 |
| corvus | i think we're all set! | 18:32 |
| AJaeger | congrats! | 18:32 |
| corvus | timburke, tdasilva: the intermediate registry should be stable now. we did have some hiccups rolling it out; image job errors from ~17:13-18:00z are likely due to that and can be rechecked. | 18:33 |
| *** jamesmcarthur has joined #openstack-infra | 18:40 | |
| openstackgerrit | James E. Blair proposed opendev/system-config master: Correct registry docker mount https://review.opendev.org/686823 | 18:44 |
| corvus | infra-root: ^ matches what's running in prod and will let us lift the emergency entry | 18:45 |
| fungi | corvus: that does look like it'll work, but i'm curious about the other files present in /var/registry/etc/ | 18:48 |
| fungi | (config.yml, hostname, hosts, resolv.conf) | 18:48 |
| fungi | were we not relying on those to end up in /etc? | 18:49 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-registry master: Add support for skopeo copy https://review.opendev.org/686803 | 18:49 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-registry master: Add type annotations https://review.opendev.org/686249 | 18:49 |
| corvus | fungi: nope, those are leftover from the previous registry | 18:49 |
| corvus | fungi: actually, the first file is leftover. i think the next 3 were written by docker when we ran in the bad config | 18:50 |
| corvus | ie, they were *exported* from the container | 18:50 |
| corvus | we can just delete those manually now | 18:50 |
| fungi | got it, that all makes perfect sense | 18:51 |
| fungi | thanks! | 18:51 |
| *** pcaruana has quit IRC | 18:51 | |
| *** ociuhandu has quit IRC | 18:51 | |
| *** ociuhandu has joined #openstack-infra | 18:52 | |
| *** petevg has joined #openstack-infra | 18:54 | |
| *** ociuhandu has quit IRC | 18:56 | |
| openstackgerrit | Tim Burke proposed zuul/zuul-registry master: Rework the stream_blob/stream_object API https://review.opendev.org/686827 | 19:00 |
| clarkb | corvus: on that registry mounting change is /var/registry/etc just a clouds.yaml file? | 19:06 |
| clarkb | ah yup that is related to what fungi was asking re hosts and resolv.conf and friends | 19:08 |
| fungi | agreed, that's what i was expecting to find, but was confused by the other (stale) files in there | 19:10 |
| *** jamesmcarthur has quit IRC | 19:14 | |
| clarkb | corvus: re restarting zuul scheduler do we want to do a full restart and if so should we get https://review.opendev.org/#/c/686775/ in first? | 19:18 |
| *** kjackal has joined #openstack-infra | 19:24 | |
| corvus | clarkb: does that mean we have the list of extra packages in 2 places? | 19:25 |
| clarkb | corvus: yes, because puppet does the normall installation. I suppose we might be able to get away with not having it in the playbook because puppet will install the packages otherwise? | 19:26 |
| corvus | looks like we didn't plumb that through to puppet-zuul | 19:27 |
| corvus | clarkb: it doesn't change that often, we can probably muddle through until we switch to images | 19:27 |
| clarkb | its in the puppet-zuul exec for zuul-manage-ansible | 19:27 |
| corvus | yeah, my plumb comment was meant to convey that we didn't make it an argument to puppet-zuul so that we could just have one list in system-config | 19:27 |
| clarkb | oh ya | 19:28 |
| corvus | clarkb: 775 lgtm; we can add that in manually if it doesn't get in place before we're ready | 19:29 |
| clarkb | corvus: sgtm, thanks | 19:29 |
| *** ykarel|away has quit IRC | 19:31 | |
| corvus | i think i'm ready to do the zuul restart now | 19:36 |
| clarkb | I've eaten lunch and can assist if necessary | 19:36 |
| *** panda is now known as panda|off | 19:38 | |
| corvus | clarkb: i've cherry-picked your change into a local repo on bridge | 19:39 |
| corvus | i think i'll wait ~5m for merges | 19:42 |
| corvus | there's a bunch of 5m changes followed by 52m; that seems like a good spot | 19:42 |
| openstackgerrit | Merged opendev/system-config master: Correct registry docker mount https://review.opendev.org/686823 | 19:46 |
| *** zbr has quit IRC | 19:46 | |
| *** e0ne has joined #openstack-infra | 19:50 | |
| *** markvoelker has quit IRC | 19:55 | |
| corvus | yay a bunch of stuff merged! | 19:57 |
| corvus | i'll get ready to restart now | 19:58 |
| corvus | restarting | 20:01 |
| corvus | 2019-10-04 20:01:18,706 INFO zuul.GerritConnection: Remote version is: 3.0.2-3196-g589fe6669e8e (parsed as (3, 0, 2)) | 20:02 |
| corvus | that's promising -- it's talking to googlesource | 20:02 |
| corvus | (we should probably add connection names into those log messages | 20:02 |
| clarkb | status dashboard is loading now | 20:08 |
| clarkb | and urls work again | 20:08 |
| corvus | reenqueing | 20:08 |
| *** e0ne has quit IRC | 20:09 | |
| *** lucasagomes has quit IRC | 20:13 | |
| corvus | that's done | 20:13 |
| clarkb | seems to be working according to the dashboard | 20:14 |
| corvus | the executors are still stopping | 20:14 |
| corvus | so no jobs are actually running yet, and we haven't seen the ansible upgrade | 20:14 |
| *** EmilienM is now known as EvilienM | 20:18 | |
| corvus | clarkb: ansible update and executor start have happened | 20:18 |
| *** EvilienM is now known as containerizes_hi | 20:19 | |
| *** containerizes_hi is now known as containerized | 20:19 | |
| clarkb | and jobs have actually started running? | 20:19 |
| *** containerized is now known as EvilienM | 20:19 | |
| corvus | clarkb: yes | 20:20 |
| clarkb | corvus: double checking ansible versions they all appear up to date on ze01 now too | 20:26 |
| corvus | \o/ | 20:26 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-registry master: Add support for skopeo copy https://review.opendev.org/686803 | 20:29 |
| *** jamesden_ has joined #openstack-infra | 20:30 | |
| *** jamesdenton has quit IRC | 20:31 | |
| *** xenos76 has quit IRC | 20:32 | |
| *** jamesmcarthur has joined #openstack-infra | 20:33 | |
| fungi | indeed, /usr/lib/zuul/ansible/2.7/bin/ansible was last modified a few minutes ago | 20:35 |
| fungi | so i guess the updates for those are working now | 20:35 |
| fungi | 2.5 is still left behind there, but i suppose that's reasonable | 20:36 |
| clarkb | fungi: left behind? | 20:36 |
| fungi | we can always manually clean up removed versions afterward if we need | 20:36 |
| clarkb | oh did we remove support for it? | 20:37 |
| fungi | -rwxr-xr-x 1 root root 5863 Mar 18 2019 /usr/lib/zuul/ansible/2.5/bin/ansible | 20:37 |
| fungi | or maybe we just didn't need to update it? | 20:37 |
| clarkb | that is the current version | 20:37 |
| fungi | i don't know if the 2.5 removal patch has merged yet | 20:37 |
| fungi | 2.6 through 2.8 were all upgraded though, so i guess that means we're finally up to date on those | 20:38 |
| corvus | https://review.opendev.org/650431 | 20:39 |
| fungi | yeah, i guess https://review.opendev.org/650431 hasn't merged yet | 20:39 |
| corvus | mordred: do you have a gerrit docker image change handy we can recheck to verify the googlesource projects work? | 20:40 |
| *** jamesden_ has quit IRC | 20:41 | |
| *** jamesdenton has joined #openstack-infra | 20:41 | |
| corvus | mordred: maybe https://review.opendev.org/682601 | 20:42 |
| corvus | which i will also review :) | 20:42 |
| *** jamesmcarthur has quit IRC | 20:44 | |
| corvus | mordred: +3 with comments | 20:45 |
| *** jamesmcarthur has joined #openstack-infra | 20:46 | |
| corvus | i'm curious about whether we want dependencies in promote... it kindof makes sense, but i also don't think it would be 100% effective in preventing errors. | 20:47 |
| clarkb | the persistent firewall issue persists after the updates | 20:53 |
| clarkb | at least we can rule that out | 20:53 |
| *** dpawlik has quit IRC | 20:54 | |
| *** kjackal has quit IRC | 20:55 | |
| *** slaweq has quit IRC | 20:59 | |
| corvus | ze03 is slowly chewing through the setup for the 2.15 job | 21:01 |
| corvus | stderr: 'ssh: connect to host gerrit-review.googlesource.com port 29418: Connection timed out | 21:05 |
| corvus | that's a problem | 21:05 |
| corvus | huh, there's no password in the zuul.conf file on the executors | 21:06 |
| clarkb | I don't recall if it was there before or not | 21:08 |
| corvus | it wouldn't have been (for the git driver) | 21:08 |
| clarkb | ah | 21:08 |
| corvus | it's there on the scheduler, so it's correctly using http, but the execs are using ssh because the password is what causes it to switch | 21:09 |
| openstackgerrit | James E. Blair proposed opendev/system-config master: Add zuul_connection_secrets to mergers and executors https://review.opendev.org/686840 | 21:12 |
| corvus | clarkb, fungi, mordred: ^ i think that should fix it? | 21:12 |
| corvus | in the mean time, jobs with googlesource repos will fail | 21:13 |
| corvus | slowly | 21:13 |
| *** xek_ has joined #openstack-infra | 21:13 | |
| fungi | ahh, yep, separate node entries in the global manifest | 21:13 |
| fungi | but same class, so all the plumbing is already there | 21:14 |
| clarkb | Looks like that key exists in hiera for the mergers and executors too (just double checked) | 21:14 |
| corvus | yeah, i added it previously assuming it would work like on the scheduler | 21:14 |
| corvus | i will dequeue the test change from gate since it'll block this | 21:15 |
| *** xek__ has quit IRC | 21:16 | |
| *** prometheanfire has quit IRC | 21:16 | |
| corvus | and maybe enqueue this one so hopefully we can fix this today | 21:16 |
| *** prometheanfire has joined #openstack-infra | 21:17 | |
| corvus | #status log removed insecure-ci-registry01 from emergency | 21:18 |
| openstackstatus | corvus: finished logging | 21:18 |
| corvus | #status log restarted all of zuul at commit e6496faf406529b4003ce7ebaa22eb1f2fa78929 | 21:18 |
| openstackstatus | corvus: finished logging | 21:18 |
| *** whoami-rajat has quit IRC | 21:18 | |
| *** derekh has joined #openstack-infra | 21:29 | |
| *** derekh has quit IRC | 21:29 | |
| *** tosky has joined #openstack-infra | 21:31 | |
| *** jamesmcarthur has quit IRC | 21:34 | |
| *** goldyfruit_ has joined #openstack-infra | 21:38 | |
| *** rlandy has quit IRC | 21:40 | |
| openstackgerrit | Merged opendev/system-config master: Add zuul_connection_secrets to mergers and executors https://review.opendev.org/686840 | 21:40 |
| *** mriedem has quit IRC | 21:47 | |
| *** jbadiapa has joined #openstack-infra | 21:50 | |
| *** EvilienM is now known as EmilienM | 21:52 | |
| *** goldyfruit_ has quit IRC | 22:04 | |
| *** goldyfruit_ has joined #openstack-infra | 22:07 | |
| corvus | fungi: do you have the fanny farmer baking book? | 22:09 |
| corvus | er fannie farmer | 22:11 |
| *** slaweq has joined #openstack-infra | 22:11 | |
| fungi | i have "the boston cooking school cook book" (fannie merritt farmer, 1938 ed.) | 22:14 |
| corvus | it's a collection by marion cunningham, who edited the revised fannie farmer cookbook; it has both old and newer recipies in it. | 22:15 |
| fungi | it's seen a lot of use, but still quite intact, you just have to look past the oil stains and whatnot | 22:15 |
| fungi | oh, *baking* book! | 22:15 |
| fungi | no, i do not | 22:15 |
| fungi | i should obtain, yes | 22:15 |
| corvus | there's some good stuff in there :) | 22:15 |
| fungi | my usual go-to is the "the king arthur flour baker's companion" but it's a modern volume | 22:16 |
| *** slaweq has quit IRC | 22:16 | |
| *** xek_ has quit IRC | 22:17 | |
| fungi | still i love that it seems like nearly a quarter of the content in it is treatises on baking science rather than just a pile of recipes | 22:17 |
| fungi | oh, and of course that they systematically include weights for all the ingredient quantities | 22:19 |
| corvus | they have a bakery and store in vermont. they're pretty good. :) | 22:20 |
| fungi | yeah, i keep meaning to plan a visit, vt is the only state in new england i've never even passed through | 22:22 |
| corvus | make a day of it and stop there on the way to ben and jerry's :) then, i guess, hike the green mountains to burn off the calories | 22:24 |
| fungi | sounds like an excellent itinerary | 22:26 |
| corvus | i'm going to run some errands while ansible does its thing and then restart the executors later | 22:32 |
| fungi | i'm around, just waiting on this fudge to reach 115c | 22:34 |
| fungi | (and then to reach 40c) | 22:35 |
| *** mattw4 has quit IRC | 22:53 | |
| *** rfolco|bbl has quit IRC | 22:54 | |
| fungi | makes me want to calculate the specific heat and thermal inertia of fudge | 22:59 |
| *** yamamoto has joined #openstack-infra | 23:00 | |
| *** yamamoto has quit IRC | 23:05 | |
| *** slaweq has joined #openstack-infra | 23:11 | |
| *** slaweq has quit IRC | 23:16 | |
| *** tosky has quit IRC | 23:39 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!