| clarkb | ianw it looks like ubuntu-ports got interrupted so it stopped running was that you? | 00:30 |
|---|---|---|
| clarkb | if not I'll start it again | 00:31 |
| ianw | i don't think so ... | 00:31 |
| clarkb | ok starting it again. | 00:31 |
| ianw | Select returned error 4: Interrupted system call ... anything in dmesg about afs? | 00:32 |
| clarkb | not recently | 00:32 |
| ianw | nope | 00:32 |
| clarkb | considering the size of the repo its probably not completely unexpected | 00:32 |
| clarkb | and we'll just keep running it until it goes happy | 00:32 |
| ianw | yeah, i didn't ctrl-c it or anything | 00:34 |
| clarkb | pretty sure I didn't either. Will just continue to watch it | 00:34 |
| clarkb | I checked quota usage and we have plenty of room there too | 00:34 |
| ianw | looks like the same just happened on the ubuntu main sync | 00:41 |
| clarkb | ianw: looks like ubuntu-ports picked up whree it left off | 00:41 |
| clarkb | is it possible the two are interacting with each other? | 00:41 |
| clarkb | they were fine for a long time :? | 00:41 |
| clarkb | ianw: maybe I should start the ubuntu main repo sync again then see if ubuntu-ports stops soon? | 00:41 |
| clarkb | oh you got it thanks | 00:42 |
| clarkb | ok I'm eating dinner now will keep checking on ti after | 00:42 |
| ianw | i just restarted the main one ... we don't have a timeout in the script do we? | 00:42 |
| clarkb | ianw: oh maybe | 00:42 |
| clarkb | we do in the cron but I thought running it manually like this might be avoiding that | 00:42 |
| ianw | yeah, i think we want NO_TIMEOUT | 00:42 |
| ianw | in fact, we do want that | 00:43 |
| clarkb | ah yup | 00:43 |
| clarkb | ok I guess next time it times out we can set that | 00:43 |
| clarkb | sorry about that I misread the script | 00:43 |
| ianw | setting it only under cron is a good idea | 00:45 |
| ianw | keep that in mind when we move it all over :) | 00:45 |
| clarkb | ++ | 00:46 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-jobs master: cabal-test: add initial haskell job https://review.opendev.org/721735 | 00:55 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: yum-minimal: strip env vars in chroot calls https://review.opendev.org/721726 | 00:57 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: [wip] switch func tests to containers https://review.opendev.org/721511 | 00:57 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-jobs master: cabal-test: add initial haskell job https://review.opendev.org/721735 | 01:10 |
| clarkb | ianw: I updated the etherpad with the timeout info | 01:27 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: [wip] Revert "opensuse: fix python 2.x install" https://review.opendev.org/721763 | 01:27 |
| clarkb | it should timeout in about 34 minutes or so | 01:27 |
| clarkb | I'll try to get them then | 01:27 |
| ianw | clarkb: i can restart them, don't worry too much if you've got better things to do :) | 01:28 |
| clarkb | ianw: ok maybe I'll take you up on that. Its add NO_TIMEOUT=1 to the beginning of the command fwiw | 01:28 |
| ianw | yep | 01:28 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: [wip] Revert "opensuse: fix python 2.x install" https://review.opendev.org/721763 | 01:30 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: [wip] Revert "opensuse: fix python 2.x install" https://review.opendev.org/721763 | 01:36 |
| openstackgerrit | Merged openstack/diskimage-builder master: Make ipa centos8 dib job voting https://review.opendev.org/717700 | 01:44 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: [wip] Revert "opensuse: fix python 2.x install" https://review.opendev.org/721763 | 01:58 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: [wip] Revert "opensuse: fix python 2.x install" https://review.opendev.org/721763 | 02:19 |
| ianw | ^ ok both now running with NO_TIMEOUT | 02:20 |
| *** DSpider has joined #opendev | 02:46 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: [wip] Revert "opensuse: fix python 2.x install" https://review.opendev.org/721763 | 02:47 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: [wip] Revert "opensuse: fix python 2.x install" https://review.opendev.org/721763 | 03:13 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Revert "opensuse: fix python 2.x install" https://review.opendev.org/721763 | 03:42 |
| ianw | dirk / mordred / frickler : ^ can we go with something like this before next dib release? | 03:45 |
| *** kevinz has quit IRC | 04:24 | |
| *** ykarel|away is now known as ykarel | 04:25 | |
| clarkb | ianw: looks like both are still going. quotas look good too | 04:51 |
| clarkb | ianw: if this ends up going past your EOD I'll just pick it up in the morning | 04:51 |
| clarkb | but now I am going to call it a night. Thanks! | 04:52 |
| *** ysandeep|away is now known as ysandeep | 05:34 | |
| dirk | ianw: can you explain why this is a revert? | 05:59 |
| dirk | It seems more like a refinement of the previous change. E.g. different way of making it work with opensuse 15 | 06:00 |
| dirk | ianw: please note that we readded python2-six etc to tumbleweed. So it is possible with tumbleweed to still create a python2 virtualenv. Parts of the change would not be needed then | 06:01 |
| openstackgerrit | Merged zuul/zuul-jobs master: helm-template: enable using values file https://review.opendev.org/721365 | 06:09 |
| AJaeger | frickler: could you review https://review.opendev.org/#/c/721719/ to retire i18n-specs, please? | 06:13 |
| *** dpawlik has joined #opendev | 06:24 | |
| ianw | dirk: for tumbleweed we don't want pip-and-virtualenv involved at all, per our removal plans in https://docs.opendev.org/opendev/infra-specs/latest/specs/cleanup-test-node-python.html | 06:33 |
| ianw | for 15; as i've mentioned several times now, that is using the python2 path to install the python3 packages (it doesn't specify _do_py3) which is wrong and makes a mess even more messy | 06:37 |
| ianw | i believe we could actually drop pip-and-virtualenv from the image right now | 06:38 |
| ianw | unfortunately last time, https://review.opendev.org/#/c/712609/ got -2'd and i didn't notice, so devstack seemed broken, and it led mordred and other on a wild goose chase trying to restore an old image | 06:39 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Revert "opensuse: fix python 2.x install" https://review.opendev.org/721763 | 06:51 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Restore SUSE tests to gate https://review.opendev.org/721779 | 06:58 |
| ianw | clarkb: yeah, not sure if i'll get back to it, it's up to l & p in universe it seems | 07:18 |
| dirk | ianw: thanks for sharing the background | 07:21 |
| dirk | Although you mentioned it several times before ;-) I don't read all of the channels all the time so I might miss comments that are not including my.nick | 07:22 |
| ianw | that's ok ... the best thing would be if we can *all* never have to think about this element again! :) | 07:23 |
| *** rpittau|afk is now known as rpittau | 07:24 | |
| *** ralonsoh has joined #opendev | 07:25 | |
| *** tosky has joined #opendev | 07:45 | |
| openstackgerrit | Sorin Sbarnea proposed openstack/project-config master: Enable promote to unarchive gz archives in addition to bz2 https://review.opendev.org/721652 | 08:08 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: tox: run default envlist if tox_envlist is empty https://review.opendev.org/721790 | 08:22 |
| *** roman_g has quit IRC | 08:23 | |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: tox: Run all environments in tox_envlist https://review.opendev.org/721790 | 08:36 |
| *** Dmitrii-Sh has joined #opendev | 08:45 | |
| *** ysandeep is now known as ysandeep|lunch | 08:46 | |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: tox: Run all environments in tox_envlist https://review.opendev.org/721790 | 09:00 |
| *** ysandeep|lunch is now known as ysandeep | 09:06 | |
| yoctozepto | infra-root: is ethercalc all right? I can't access https://ethercalc.openstack.org/kolla-infra-service-matrix | 09:12 |
| *** ykarel is now known as ykarel|lunch | 09:18 | |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: tox: update tox_envlist documentation https://review.opendev.org/721796 | 09:19 |
| frickler | yoctozepto: indeed, apache2 error log was full with "[mpm_event:error] [pid 5272:tid 140651845121920] AH00485: scoreboard is full, not at MaxRequestWorkers" starting with today's log rotation | 09:26 |
| frickler | #status log restarted apache2 on ethercalc.openstack.org which seems to have gotten stuck during today's log rotation | 09:27 |
| openstackstatus | frickler: finished logging | 09:27 |
| yoctozepto | frickler: thanks | 09:27 |
| frickler | uptime 616d, not bad for a cloud service | 09:30 |
| *** ykarel|lunch is now known as ykarel | 09:45 | |
| *** rpittau is now known as rpittau|bbl | 10:42 | |
| *** roman_g has joined #opendev | 11:15 | |
| *** ysandeep is now known as ysandeep|brb | 11:46 | |
| *** rpittau|bbl is now known as rpittau | 11:50 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Remove Babel and any signs of translations https://review.opendev.org/720673 | 11:51 |
| ttx | now running the async refs/changes cleanup script on the openstack github mirrors. Should take a few days | 11:51 |
| *** Eighth_Doctor is now known as Conan_Kudo | 12:25 | |
| *** Conan_Kudo is now known as Eighth_Doctor | 12:27 | |
| *** Eighth_Doctor is now known as Conan_Kudo | 12:28 | |
| *** Conan_Kudo is now known as Eighth_Doctor | 12:29 | |
| *** Eighth_Doctor is now known as Conan_Kudo | 12:29 | |
| *** Conan_Kudo is now known as Eighth_Doctor | 12:30 | |
| *** ysandeep|brb is now known as ysandeep | 12:41 | |
| AJaeger | config-core, could you review https://review.opendev.org/#/c/721719/ to retire i18n-specs, please? | 12:52 |
| AJaeger | ttx: thx, hope you have a stable connection ;) | 12:52 |
| *** ykarel is now known as ykarel|afk | 13:00 | |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Move in-tree hiera settings to ansible vars https://review.opendev.org/721629 | 13:08 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Add new etherpad to cacti https://review.opendev.org/721633 | 13:09 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Stop cloning a bunch of puppet modules we don't use https://review.opendev.org/720892 | 13:09 |
| ttx | AJaeger: I procured a cloud instance to do that :) Don;t want to see my IP blacklisted on GitHub | 13:12 |
| AJaeger | ;) | 13:14 |
| mordred | ttx: neat. we already landed the "don't replicate new refs/changes" config yes? | 13:23 |
| ttx | yes | 13:23 |
| ttx | https://review.opendev.org/#/c/720679/ | 13:24 |
| mordred | infra-root: holy crap! https://review.opendev.org/#/c/721098 is green! that means it fixes the speculative container support - and otherwise all works! | 13:34 |
| mordred | corvus: let me know when you've got a few minutes to chat about nodepool containers | 13:36 |
| AJaeger | mordred: time for a quick review of https://review.opendev.org/#/c/721719/ to retire i18n-specs, please? | 13:39 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Stop cloning a bunch of puppet modules we don't use https://review.opendev.org/720892 | 13:40 |
| AJaeger | thx, mordred | 13:42 |
| corvus | mordred: i'm here. also, w00t! | 13:45 |
| *** sgw has quit IRC | 13:47 | |
| corvus | mordred: left some comments on that change | 13:49 |
| *** ykarel|afk is now known as ykarel | 13:51 | |
| openstackgerrit | Merged openstack/project-config master: Retire i18n-specs repo https://review.opendev.org/721719 | 13:53 |
| openstackgerrit | Monty Taylor proposed opendev/puppet-cgit master: Retire repo https://review.opendev.org/721962 | 13:53 |
| openstackgerrit | Monty Taylor proposed opendev/puppet-accessbot master: Retire repo https://review.opendev.org/721963 | 13:55 |
| openstackgerrit | Monty Taylor proposed opendev/puppet-etherpad_lite master: Retire repo https://review.opendev.org/721964 | 13:55 |
| openstackgerrit | Monty Taylor proposed opendev/puppet-exim master: Retire repo https://review.opendev.org/721965 | 13:55 |
| openstackgerrit | Monty Taylor proposed opendev/puppet-gerrit master: Retire repo https://review.opendev.org/721966 | 13:55 |
| openstackgerrit | Monty Taylor proposed opendev/puppet-gerritbot master: Retire repo https://review.opendev.org/721967 | 13:55 |
| openstackgerrit | Monty Taylor proposed opendev/puppet-infracloud master: Retire repo https://review.opendev.org/721968 | 13:55 |
| openstackgerrit | Monty Taylor proposed opendev/puppet-ipsilon master: Retire repo https://review.opendev.org/721969 | 13:56 |
| openstackgerrit | Monty Taylor proposed opendev/puppet-iptables master: Retire repo https://review.opendev.org/721970 | 13:56 |
| openstackgerrit | Monty Taylor proposed opendev/puppet-jenkins master: Retire repo https://review.opendev.org/721971 | 13:56 |
| openstackgerrit | Monty Taylor proposed opendev/puppet-nodepool master: Retire repo https://review.opendev.org/721972 | 13:56 |
| openstackgerrit | Monty Taylor proposed opendev/puppet-openstackci master: Retire repo https://review.opendev.org/721973 | 13:56 |
| openstackgerrit | Monty Taylor proposed opendev/puppet-os_client_config master: Retire repo https://review.opendev.org/721974 | 13:56 |
| openstackgerrit | Monty Taylor proposed opendev/puppet-packagekit master: Retire repo https://review.opendev.org/721975 | 13:56 |
| openstackgerrit | Monty Taylor proposed opendev/puppet-phabricator master: Retire repo https://review.opendev.org/721976 | 13:57 |
| openstackgerrit | Monty Taylor proposed opendev/puppet-zuul master: Retire repo https://review.opendev.org/721977 | 13:57 |
| openstackgerrit | Roman Gorshunov proposed openstack/project-config master: Retire airship-in-a-bottle: end project gating https://review.opendev.org/721978 | 13:57 |
| mordred | AJaeger: ^^ retire patches! | 13:57 |
| mordred | corvus: cool - thanks | 13:57 |
| AJaeger | mordred: thanks! | 13:58 |
| mordred | corvus: so - it looks like ianw has sorted the nodepool builder container issue - which means we should either grapple with the arm question or continue with the "switch builders to use ansible" approach. did you get a chance to read the scrollback from me from the othe day about the arm stuff? | 13:58 |
| mordred | http://eavesdrop.openstack.org/irclogs/%23opendev/%23opendev.2020-04-19.log.html#t2020-04-19T15:04:32 for context | 13:59 |
| corvus | mordred: on the issue of buildset registry -- i believe it's just a request that it be in the same provider. if that provider is not capable of producing the label, then it falls back to normal behavior | 14:02 |
| corvus | mordred: so i think we can still use a single buildset registry if we want, we'll just be shipping a lot of data over the internet | 14:03 |
| mordred | corvus: ok - that's good | 14:03 |
| mordred | that part was the biggest "this might not work at all" | 14:03 |
| corvus | yeah, i think that's worth a try | 14:03 |
| corvus | mordred: what's the manifest manipulation for? i didn't understand that part | 14:04 |
| mordred | you have to create a manifest that references the other images you made - and you push that to docker hub as "the image" | 14:04 |
| corvus | ah, that makes me understand the man page now :) | 14:05 |
| mordred | so we make nodepool-builder:x86 and nodepol-builder:arm and then a manifest that is nodepool-builder:latest that refrences nodepool-builder:x86 as the image for x86 and nodepool-builder:arm as the image for arm | 14:05 |
| mordred | corvus: \o/ | 14:05 |
| *** sgw has joined #opendev | 14:06 | |
| corvus | mordred: i think the existing tooling is using docker; do we want to do that and leave buildah for later? | 14:07 |
| mordred | corvus: yeah- I think I mostly mentioned buildah in case it was easier - it seems like the manifest step is going to be a new job on a new node, so it could likely use whatever is easiest (sort of like using skopeo for the transfers) | 14:08 |
| mordred | I'm guessing a "nodepool-build-manifest" job that dependencies: ['nodepool-build-image-arm', 'nodepool-build-image-x86'] yeah? | 14:09 |
| corvus | mordred: i guess to do this incrementally, we could download the manifest, add an entry, then upload it again? | 14:09 |
| mordred | can we? | 14:09 |
| corvus | i'm wondering if we can do that without a third job; but even if we could, coordinating between those 2 jobs would be difficult. so maybe it's simplest/best to do your suggestion | 14:10 |
| mordred | corvus: we could do it incrementally perhaps if we also used a semaphore | 14:11 |
| corvus | (basically, the step of discovering whether a job was the 1st or second image would be a race) | 14:11 |
| mordred | yeah | 14:11 |
| corvus | mordred: yeah, but then we're serializing image builds which is lame | 14:11 |
| mordred | totally agree | 14:11 |
| mordred | so this will be fun - I think maybe we just start with nodepool image build jobs in repo - and then extract out a build-combined-manifest role? | 14:12 |
| mordred | or do you think we should go ahead and do a build-combined-manifest role in zuul-jobs to start with? | 14:12 |
| mordred | guess it doesn't really make it any harder | 14:12 |
| corvus | yeah, i'd do that :) | 14:13 |
| mordred | k. I'll work on this today - because I think it's more forward-looking (and honestly a cool new zuul thing) than regressing to a pip install | 14:13 |
| corvus | ok | 14:14 |
| mordred | I've got a couple of calls in the middle of the day, so I might fall into a hole for a minute, but I think it's reasonably doable | 14:14 |
| mordred | oh - I guess we should figure out what the docker architecture tag is for our arm hosts | 14:14 |
| corvus | mordred: i'm guessing the docker manifest command needs to run on a machine with dockerd, so it'll need a real node | 14:15 |
| mordred | yeah | 14:16 |
| corvus | mordred: i wonder 2 things: 1) if buildah could do it on the executor; 2) if we can just write a json file and skopeo it into to the registry, without any extra tools? | 14:16 |
| mordred | corvus: ooh - that would be neat | 14:17 |
| mordred | (the 2 one) | 14:17 |
| mordred | I mean - it IS just a json file | 14:17 |
| mordred | arm64 seems to be the architecture tag | 14:17 |
| corvus | yeah, i think that might be worth it | 14:17 |
| mordred | corvus: we could do that as a followup | 14:18 |
| corvus | it looks like we just need to know the size and sha256 of the underlying image manifests? | 14:18 |
| corvus | mordred: i think it might be easier to just start with this? | 14:18 |
| mordred | oh yeah? | 14:18 |
| corvus | mordred: we might be able to add the sha256 and size information to the zuul artifact return | 14:20 |
| corvus | mordred: which means the 'create a json' file step could just be a jinja template | 14:20 |
| corvus | we wouldn't even need to move any data around | 14:20 |
| corvus | actually, maybe we can just 'skopeo inspect' the remote image | 14:22 |
| mordred | oh yeah - almost certainly | 14:22 |
| * mordred was just docker manifest inspect ubuntu:latest - and docker manifest inspect -v ubuntu:latest - to see what it looked like | 14:23 | |
| mordred | corvus: oh - hah. I was thikning we could just start with nodepool - but really we need to start with python-base/python-builder :) | 14:25 |
| corvus | skopeo inspect docker://insecure-ci-registry.opendev.org:5000/zuul/zuul-merger:c96e6abbfd954049ab90323c268b26fc_latest | 14:26 |
| corvus | FATA[0001] Error determining repository tags: invalid status code from registry 404 (Not Found) | 14:26 |
| mordred | hrm | 14:26 |
| corvus | that's weird; it's there, i can pull it, and the registry itself only logs 200 return codes | 14:26 |
| corvus | a | 14:27 |
| corvus | 2020-04-22 14:27:07,184 INFO cherrypy.access.140475213507728: 70.36.235.80 - - [22/Apr/2020:14:27:07] "GET /v2/zuul/zuul-merger/tags/list HTTP/1.1" 404 762 "" "Go-http-client/1.1" | 14:27 |
| corvus | mordred: ^ maybe the registry is missing an api endpoint and i should flesh that out | 14:27 |
| mordred | corvus: ++ | 14:28 |
| corvus | mordred: how about we proceed under the assumption that i can get that working today and that we'll be able to use 'skopeo inspect' to get the necessary info? | 14:28 |
| AJaeger | mordred: https://docs.opendev.org/opendev/infra-manual/latest/drivers.html#step-2-remove-project-content has a slightly different README, want to update for the retirement - or stay with yours? I wouldn't -1 yours, just find the other nicer | 14:28 |
| mordred | corvus: I'll start working on the other parts of the puzzle | 14:28 |
| corvus | mordred: oh -- i don't see the size in the inspect output; only the sha | 14:29 |
| corvus | if we need size that may not be enough | 14:29 |
| mordred | corvus: I see size in inspect output from docker | 14:30 |
| mordred | oh - that's on a manifest - one sec | 14:30 |
| mordred | corvus: yeah- I see size from docker | 14:30 |
| mordred | but that's not against insecure-ci-registry - that's just against local dockerhub stuff | 14:31 |
| corvus | i pulled an image and 'docker inspect' shows the size, but 'skopeo inspect' does not; it looks like it's missing from skopeo output | 14:32 |
| mordred | corvus: "awesome" | 14:32 |
| mordred | corvus: is there a -v option or anythign to skopeo inspect? | 14:32 |
| corvus | looking | 14:33 |
| corvus | oh, i think i was getting confused with manifests and non-manifests | 14:34 |
| corvus | gimme a sec to sort this out | 14:34 |
| mordred | corvus: skopeo inspect --raw gives me what docker gives me | 14:35 |
| mordred | corvus: skopeo inspect --raw docker://docker.io/opendevorg/python-base | 14:35 |
| mordred | and skopeo inspect --raw docker://docker.io/library/ubuntu is uglier but shows the manifest version | 14:36 |
| corvus | mordred: yeah, i think that'll work -- i think the ['config']['size'] from that is what we want to put in ['manifests'][0]['size'] of the multi arch manifest | 14:37 |
| mordred | yeah | 14:37 |
| corvus | w00t, then i think we can proceed as planned :) | 14:37 |
| mordred | corvus: I think this is goign to be real nice | 14:37 |
| mordred | and as a bonus - doesn't require nested virt! (which is what the docker buildx command needs) | 14:38 |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Fix i18n-specs jobs https://review.opendev.org/721989 | 14:38 |
| corvus | mordred: it does require an arm cloud though | 14:38 |
| corvus | and we don't have a great track record with those | 14:39 |
| mordred | indeed | 14:40 |
| mordred | corvus: well - if our current arm cloud goes away, so does our need for multi-arch images | 14:40 |
| corvus | yeah; but sinc this is going to run on every nodepool change, it's going to add extra time to each. and we may need to frequently disable the job due to outages. | 14:42 |
| mordred | corvus: but if we're worried about that - we could just do parallel jobs with an explicit arm64 tag in arm specific pipelines | 14:42 |
| corvus | true | 14:42 |
| mordred | and in the deployment tell nb03 to boot from nodepool-builder:arm64 | 14:42 |
| mordred | it's less sexy - but it might be more resilient in the short term | 14:42 |
| corvus | that works for check, but it's not great for gate or promote | 14:44 |
| mordred | corvus: in fact, we could go more simple for arm and just do a post-arm64 pipeline that does the build and upload - since with parallel pipelines we aren't goign to do gate so that ^^ | 14:44 |
| mordred | count on x86 for the testing and speculative (not like we're likley to want to run full integration tests on arm anyway) | 14:45 |
| mordred | and treat the arm build as more of a packaging step | 14:45 |
| corvus | yeah, that'd work, but then the images might get out of sync for deployment | 14:45 |
| *** factor has quit IRC | 14:45 | |
| corvus | that may not be a concern if we're not going to auto restart the builders | 14:45 |
| mordred | yeah | 14:46 |
| openstackgerrit | Roman Gorshunov proposed openstack/project-config master: Retire airship-in-a-bottle: end project gating https://review.opendev.org/721978 | 14:48 |
| clarkb | infra-root ubuntu-ports is still going but ubuntu finished. Unfortunately ubuntu reports "there have been errors" so I think I'm going to rerun reprepro on ubuntu | 14:50 |
| fungi | thanks for the heads up | 14:50 |
| clarkb | in theory it should be much quicker this time because the bulk of the data has been transferred | 14:51 |
| fungi | it does still wind up scanning a bunch of what's there, but yeah, at least it doesn't need to retransmit and write | 14:51 |
| clarkb | yup | 14:51 |
| clarkb | also it appears that the ubuntu-ports reprepro might have stalled? We don't have timing info within the reprepro log output so hard to say for sure. But if it stays in its current log state for the next $breakfasttime then maybe I'll stop it start it again too | 14:53 |
| clarkb | overall though quota usage looks reasonable and about what I expected | 14:53 |
| fungi | strace the newest child process i guess? | 14:54 |
| *** mlavalle has joined #opendev | 14:55 | |
| clarkb | strace says it is doing a lot of reading. Maybe it is verifying the state of the repo? | 14:56 |
| clarkb | I'll leave it be at least until strace shows it idling for a long period of time | 14:56 |
| *** ykarel is now known as ykarel|away | 14:57 | |
| openstackgerrit | Merged openstack/project-config master: Fix i18n-specs jobs https://review.opendev.org/721989 | 15:06 |
| clarkb | looks like ubuntu main has reported "there are errors" again | 15:14 |
| clarkb | it did add and remove more pacakges though | 15:14 |
| clarkb | so maybe we just need to converge on happy state? I think that is how it works in the normal daily cron | 15:14 |
| clarkb | I'll try running it again since that ran relatively quickly and gives us more data | 15:14 |
| clarkb | actually hrm the same unhappy files show up this time around too | 15:16 |
| clarkb | lets look at those more closely | 15:16 |
| clarkb | http://paste.openstack.org/show/792549/ | 15:17 |
| clarkb | fungi: ^ I think that means the db is saying it knows about some files that aren't present on disk | 15:17 |
| clarkb | fungi: any idea on how to recover that? | 15:17 |
| openstackgerrit | Monty Taylor proposed opendev/puppet-cgit master: Retire repo https://review.opendev.org/722001 | 15:18 |
| openstackgerrit | Monty Taylor proposed opendev/puppet-etherpad_lite master: Retire repo https://review.opendev.org/722002 | 15:18 |
| openstackgerrit | Monty Taylor proposed opendev/puppet-exim master: Retire repo https://review.opendev.org/722003 | 15:18 |
| openstackgerrit | Monty Taylor proposed opendev/puppet-gerrit master: Retire repo https://review.opendev.org/722004 | 15:19 |
| openstackgerrit | Monty Taylor proposed opendev/puppet-gerritbot master: Retire repo https://review.opendev.org/722005 | 15:19 |
| openstackgerrit | Monty Taylor proposed opendev/puppet-github master: Retire repo https://review.opendev.org/722006 | 15:19 |
| openstackgerrit | Monty Taylor proposed opendev/puppet-infracloud master: Retire repo https://review.opendev.org/722007 | 15:19 |
| openstackgerrit | Monty Taylor proposed opendev/puppet-ipsilon master: Retire repo https://review.opendev.org/722008 | 15:19 |
| openstackgerrit | Monty Taylor proposed opendev/puppet-iptables master: Retire repo https://review.opendev.org/722009 | 15:19 |
| openstackgerrit | Monty Taylor proposed opendev/puppet-jenkins master: Retire repo https://review.opendev.org/722010 | 15:19 |
| openstackgerrit | Monty Taylor proposed opendev/puppet-nodepool master: Retire repo https://review.opendev.org/722011 | 15:19 |
| openstackgerrit | Monty Taylor proposed opendev/puppet-openstackci master: Retire repo https://review.opendev.org/722012 | 15:19 |
| openstackgerrit | Monty Taylor proposed opendev/puppet-os_client_config master: Retire repo https://review.opendev.org/722013 | 15:19 |
| openstackgerrit | Monty Taylor proposed opendev/puppet-packagekit master: Retire repo https://review.opendev.org/722014 | 15:19 |
| openstackgerrit | Monty Taylor proposed opendev/puppet-phabricator master: Retire repo https://review.opendev.org/722015 | 15:19 |
| openstackgerrit | Monty Taylor proposed opendev/puppet-zuul master: Retire repo https://review.opendev.org/722016 | 15:19 |
| mordred | AJaeger: ^^ fixed readme | 15:19 |
| mordred | AJaeger: SIGH | 15:20 |
| mordred | AJaeger: I did not commit ammend well | 15:20 |
| clarkb | ports just finished and didn't report errors so I am going to vos release it now | 15:21 |
| clarkb | still trying to make sense of the ubuntu main complaints | 15:21 |
| clarkb | oh I forget to remove the package indexes for trusty on ubuntu-ports. THis is what I get for doing this first thing in the morning | 15:22 |
| clarkb | that shouldn't break anything will just advertise trusty when we don't have packages for it | 15:22 |
| clarkb | I've made a note on the etherpad to do that then vos release again | 15:22 |
| fungi | clarkb: honestly, not sure, the last time i ran into inconsistency issues (for the debian/buster backports addition) i ended up removing reprepro's database and letting it recreate it | 15:23 |
| clarkb | fungi: its looks like deleteifunreferenced (the thing that fails) is a less angry version of deleteunreferenced (based on man page reading) | 15:24 |
| openstackgerrit | Monty Taylor proposed opendev/puppet-accessbot master: Retire repo https://review.opendev.org/722018 | 15:24 |
| clarkb | so I'm going to try deleteunreferenced then rerunning reprepro | 15:24 |
| fungi | worth a shot | 15:24 |
| clarkb | this was the command used at the beginning to clear out trusty anyway so should be safe | 15:24 |
| mordred | ++ | 15:25 |
| clarkb | oh you know what we might be feeding it that list of chrony things manually | 15:25 |
| clarkb | I'm checking on that and if so I might just move that file aside | 15:25 |
| clarkb | because it likely got cleared in the trusty clearing | 15:25 |
| openstackgerrit | Jan Zerebecki proposed openstack/diskimage-builder master: Retry git clone/fetch on timeout https://review.opendev.org/721581 | 15:27 |
| clarkb | ya I think that is it | 15:27 |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Finish retiring i18n-specs https://review.opendev.org/721722 | 15:28 |
| AJaeger | mordred: sorry, I shouldn't have asked ;) | 15:28 |
| clarkb | that file only had the three chrony packages in it. I've backed it up to my homedir then cleared it out | 15:29 |
| clarkb | that should get this moving I think | 15:29 |
| clarkb | assuming that reports success I'll clear out the trusty indices in ubuntu and vos release. Then do the same for ubuntu-ports once its vos release is completed | 15:30 |
| mordred | AJaeger: it's ok - I abandoned the bad ones | 15:32 |
| mordred | clarkb: \o/ | 15:32 |
| clarkb | its crazy how arcacne reprepro is | 15:33 |
| mordred | clarkb: also - fwiw - ianw got the container fixed up for nb04 - and corvus and I discussed some strategies for building arm containers | 15:33 |
| mordred | clarkb: you might enjoy reading that scrollback - there's a couple of different options we can do | 15:33 |
| clarkb | mordred: have a timestamp I should look for/ | 15:33 |
| mordred | clarkb: literally the conversation that ended right before you said the first thing today | 15:34 |
| AJaeger | mordred: https://review.opendev.org/#/c/722013/ is in merge conflict? | 15:35 |
| clarkb | mordred: corvus as an intermediate step we could possibly build nodepool-builder-arm64 and not manifest? I don't know if that makes things simpler | 15:36 |
| clarkb | the last time I looked into the docker manifest stuff it seemed to be painful because client tools weren't all up to speed but that was a while back and they may be ebtter now | 15:36 |
| corvus | clarkb: i think that's effectively mordred's latest option, except he spelled nodepool-builder-arm64 as nodepool-builder:arm64 :) | 15:37 |
| corvus | clarkb: do you recall the client problems? | 15:38 |
| clarkb | corvus: iirc they simply didn't know about the manifest thing so you'd pull an image and it wouldn't work | 15:38 |
| corvus | (it seems likely that our base debian images would be multi-arch) | 15:38 |
| clarkb | its likely that was an issue for distro shipped docker as newer docker added the support | 15:39 |
| clarkb | oh good point so ya we are probably fine there | 15:39 |
| corvus | yep, python:3.7-slim is multiarch | 15:40 |
| corvus | (and does include arm64) | 15:40 |
| clarkb | ubuntu main just exited cleanly. I'm removing the trusty indexes now and will vos release \o/ | 15:41 |
| corvus | clarkb, mordred: so i think the dedicated image post job is the way to go if we're concerned about cloud stability and build times. if/when we think our arm64 clouds are in a good enough place in terms of reliability and capacity to put them in the critical path for landing every nodepool change, then the multi-arch manifest is the way to go. | 15:42 |
| clarkb | corvus: wfm | 15:44 |
| clarkb | fungi: mordred: re focal. The next thing I notice is we don't create a backports or security index | 15:44 |
| clarkb | I believe this is a known thing with reprepro when upstream has no packages | 15:44 |
| clarkb | I forget how we dealt with that in the past but heads up that we'll likely need to handle it here too | 15:44 |
| clarkb | both ubuntu-ports and ubuntu are vos releasing now | 15:45 |
| openstackgerrit | Jan Zerebecki proposed openstack/diskimage-builder master: Retry zypper when refresh failed https://review.opendev.org/721587 | 15:45 |
| openstackgerrit | Jan Zerebecki proposed openstack/diskimage-builder master: Retry git clone/fetch on timeout https://review.opendev.org/721581 | 15:46 |
| fungi | clarkb: in the past we've avoided adding the security and backports suites to sources lists until they have packages | 15:46 |
| clarkb | fungi: doesn't that break our mirror setup on test nodes? | 15:46 |
| clarkb | maybe we addressed that and its fine | 15:46 |
| clarkb | I'm not sure anything is actually wrong here. Just calling out that the previous behavior that I Think gave us problems is present here too | 15:47 |
| fungi | yeah, i think the only thing it breaks is if systems add lines to their sources list for those suites | 15:52 |
| mordred | I believe security and backports get created upstream once final release happens | 15:52 |
| fungi | since there won't be any package indices for them until there are packages to index for them | 15:52 |
| fungi | mordred: at least for debian it's even later | 15:52 |
| mordred | yeah | 15:53 |
| fungi | usually security suite gets created after there's a security update, and backports once there's a package backported | 15:53 |
| persia | Historically, for Ubuntu, it was a week or so before release (related to the last final test milestone, as the final installer wanted to have -backports and -security available,. and that needed testing). | 15:53 |
| fungi | or, well, technically the backports and security suites for debian *do* exist earlier. the problem here is that reprepro won't create indices for them because they have no packages to mirror | 15:54 |
| persia | Yes. Reprepro is the limiting factor here. The decision to avoid adding the suites until they have packages is the correct decision for opendev. | 15:54 |
| fungi | we have to wait to rely on them until after reprepro will create indices for them, which won't be until there are some packages in them | 15:54 |
| fungi | or we need some separate process to create those empty indices outside of reprepro | 15:55 |
| clarkb | mordred: corvus re nodepool ansible/containers/etc I'd like to restart nl03 with the chagne to address quota accounting and see if that makes inap things happier | 16:17 |
| clarkb | is there any reason to not just do that now on the existing host? | 16:17 |
| mordred | clarkb: I can't think of one | 16:23 |
| clarkb | k will do that shortly | 16:26 |
| *** rpittau is now known as rpittau|afk | 16:29 | |
| clarkb | infra-root I'm stopping and starting nodepool-launcher on nl03 toget the quota accounting fix in | 16:36 |
| clarkb | inap is currently happy so the next time we have delete issues we should see if this helps with grabbing too many nodereuests | 16:37 |
| clarkb | nodepool==3.12.1.dev69 # git sha cbdf374 is what I restarted on | 16:38 |
| *** dpawlik has quit IRC | 16:40 | |
| *** dpawlik has joined #opendev | 16:40 | |
| *** dpawlik has quit IRC | 16:45 | |
| openstackgerrit | Merged opendev/base-jobs master: Enable promote to unarchive gz archives in addition to bz2 https://review.opendev.org/721706 | 16:47 |
| openstackgerrit | Merged openstack/project-config master: Enable promote to unarchive gz archives in addition to bz2 https://review.opendev.org/721652 | 16:48 |
| *** _mlavalle_1 has joined #opendev | 16:48 | |
| clarkb | I've confirmed that nodepool_pool_name='main', nodepool_provider_name='inap-mtl01' are set as properties via openstack server show on newly booted instances | 16:49 |
| *** mlavalle has quit IRC | 16:50 | |
| clarkb | mordred: next up on my list is the gerrit /opt/lib replication things. I'm about to be distracted by school for a bit though but wanted to let you know that is still on my list of help sort out | 16:57 |
| clarkb | vos releases continue | 16:57 |
| mordred | clarkb: do we still have an issue with that? | 17:06 |
| mordred | clarkb: I thought we were solid on it now | 17:07 |
| mordred | clarkb: I chowned the three repos - I think they were chown'd weird because they were created when the mounts were inconsistent | 17:07 |
| clarkb | mordred: ah ok I missed that. And yes I see now more retries in gerrit show queue | 17:07 |
| mordred | so I *think* we should be gtg there pending further failures | 17:07 |
| clarkb | mordred: do we need to trigger replication? | 17:08 |
| clarkb | or did that get done too | 17:08 |
| mordred | that said - I can't build a narrative as to exactly why they got into that state | 17:08 |
| mordred | I did that yesterday too | 17:08 |
| mordred | so I expect them to be ok now | 17:08 |
| clarkb | awesome. I completely missed that. Thank you! | 17:08 |
| mordred | sure! | 17:08 |
| mnaser | so i was confused by this for a long time | 17:40 |
| mnaser | but in our opendev images, install-from-bindep is more like install-from-bindep-and-python-wheels | 17:40 |
| clarkb | mnaser: I don't think bindep installs anything via pypi/pip | 17:41 |
| mnaser | ~maybe~ we need to rename that at some point, i dont think its hugely important, but just wanted dto throw some thought at it, cause i was cofnused for a while where the wheels were being installed | 17:41 |
| clarkb | mnaser: it should only run bindep, then run your package manager with the list bindep supplies | 17:41 |
| mnaser | clarkb: sorry i meant https://opendev.org/opendev/system-config/src/branch/master/docker/python-builder/scripts/install-from-bindep (from opendev images, not the zuul role) | 17:41 |
| clarkb | oh the docker images, ya I think thats less concerned about being "clean" | 17:42 |
| clarkb | but I agree making that clearer would be a good thing | 17:42 |
| fungi | yes, naming is confusing there, i agree | 17:42 |
| mordred | ++ | 17:42 |
| mordred | it's bad naming | 17:42 |
| mordred | never let me name things | 17:42 |
| mordred | that should be "install the bindep dev depends needed to build wheels, collect the final list of bindep and also build all the wheels and collect them" | 17:43 |
| clarkb | calling it "install-dependencies" might be more accurate | 17:43 |
| mordred | well - it's not even quite that | 17:43 |
| clarkb | oh right this is the staging area | 17:43 |
| clarkb | "build-python-dependencies" maybe | 17:43 |
| mordred | it should be "stage wheels and bindep lists" | 17:43 |
| clarkb | ++ | 17:44 |
| mordred | that it installs things in the builder image is an impl detail | 17:44 |
| mordred | it does that to have the side effect of producing wheels for the deps and their transitive deps | 17:44 |
| mordred | maybe just "stage-install-artifacts" | 17:44 |
| mordred | and then put in a big comment at the top of it that explains what it's doing and why | 17:44 |
| mnaser | mordred: this doesn't just stage it though does it? when running in base image it also installs all the wheels into the image | 17:49 |
| mnaser | unelss i'm misunderstanding things :) | 17:50 |
| mordred | mnaser: yes - it installs the wheels in the base image | 17:50 |
| mordred | mnaser: but it installs the wheels that were produced into the wheel cache by installing everything in the builder image | 17:51 |
| mordred | we do that so that we're sure we're always installing from wheels in the base image and therefore do not need any of the compile bindep depends installed | 17:51 |
| mordred | the wheel install in the base image should basically be the equiv of just untarring a bunch of pre-built python dirs | 17:51 |
| clarkb | right we use the builder to stage all that so that the result on the actual image is minimal | 17:52 |
| mordred | so the builder image installs all the wheels, which puts them and their depends into the wheel cache - then the wheels in the wheel cache are copied into the base image from the builder image | 17:52 |
| mordred | yup | 17:52 |
| clarkb | otherwise you ahve to have full compiler toolchains and end up with compile artifacts and all that | 17:52 |
| mordred | exactly | 17:52 |
| mnaser | mordred: right, for some reason, it's installing memcached in the build phase | 17:53 |
| mnaser | but its not installing it in the base image after | 17:53 |
| clarkb | mnaser: is memcached only listed as a compile dep? | 17:53 |
| mordred | yeah - that sounds like a bindep file issue | 17:54 |
| clarkb | (but also you probably don't want memcached running alongside whatever python daemon you've got) | 17:54 |
| mnaser | clarkb: i'm the king of improper info, i meant python-memcached, and its listed in extras -- exactly like https://review.opendev.org/#/c/722023/ | 17:54 |
| mnaser | i see it install in the get-extra-packages stage | 17:54 |
| mordred | oh hrm. and it's not winding up in your final image? that's super weird | 17:55 |
| mnaser | so i see: Successfully installed python-memcached-1.59 | 17:55 |
| mnaser | in my build image | 17:55 |
| mnaser | the _only_ thing that might be odd is this is based on the uwsgi-base thing | 17:56 |
| mnaser | see https://review.opendev.org/#/c/721702/5/images/horizon/Dockerfile | 17:56 |
| *** ralonsoh has quit IRC | 17:56 | |
| clarkb | possible that it isn't building a wheel for python-memcached for some reason? | 17:57 |
| clarkb | if that is the case then we won't install it after staging | 17:58 |
| clarkb | mnaser: ^ you might want to check the logs to see if it indicates how that is pull in on the build container | 17:58 |
| mnaser | let me double check, that is likely the case here | 17:58 |
| clarkb | mnaser: because its an extra it isn't in requirements | 17:59 |
| clarkb | so it would only be installed if it was in the wheel cache | 17:59 |
| mordred | we use the extras trick to install extra things in the nodepool images though - so it _should_ work | 17:59 |
| clarkb | https://pypi.org/project/python-memcached/#files publishes a wheel too so that should've been used | 18:01 |
| clarkb | and would wind up in the cache as expected | 18:01 |
| clarkb | vos releases are still running fwiw | 18:02 |
| *** sshnaidm is now known as sshnaidm|afk | 18:04 | |
| clarkb | I've signed us up for those three 2 hour blocks of PTG time I mentioned yesterday | 18:20 |
| clarkb | I've also filled out the attendance survey for opendev | 18:20 |
| clarkb | "see" you there :) | 18:20 |
| clarkb | corvus: I listed Zuul as a project to avoid conflicts with. but not sure if you planned to do zuul tiem at all | 18:21 |
| corvus | clarkb: i think our most recent plan (from the before times) was to pass this ptg by and try to focus on berlin; i don't think we ever bothered to revisit that after the apocalypse. i'm fine with that, and no one else has raised that with me, so i think we'll probably keep that as the plan | 18:23 |
| mnaser | ok checking this issue now | 18:57 |
| mnaser | clarkb, mordred: ok i can confirm that i dont have an actual python-memcache in the /output/wheels folder.. | 19:04 |
| mnaser | having said that, maybe we're doing the copy wrong because | 19:04 |
| mnaser | there is /output/wheels/wheels | 19:04 |
| mordred | mnaser: there shouldnt' be an /output/wheels/wheels | 19:04 |
| mnaser | (note this is still in the build image) | 19:04 |
| mordred | so - yeah - I'd what that sounds like a copy issue | 19:05 |
| mordred | mnaser: your copy line is fine - your'e copying /output | 19:05 |
| mordred | mnaser: that makes me wonder if we're setting teh wheel cache dir wrong | 19:05 |
| mordred | mnaser: I have a patch to clean up something - let me pull it up - maybe what we should do is iterate on it and see if we can't figure out a way to verify/test the steps | 19:06 |
| mnaser | mordred: to repro.. http://paste.openstack.org/show/792563/ | 19:06 |
| mnaser | and then docker run -it --rm on the target image locally and you can inspect and see both python-memcached missing (it isn't even in wheels folder) | 19:07 |
| mnaser | also the wheels folder seems to be "indexed" | 19:07 |
| mordred | mnaser: https://review.opendev.org/#/c/715717/ | 19:07 |
| mnaser | (like the folders are hashed) | 19:07 |
| mordred | yeah - I thnk that's expected | 19:08 |
| mordred | but I thought we were supposed to deal with it | 19:08 |
| mordred | lemme read code real quick | 19:08 |
| mordred | mnaser: and I'm going to run your repro | 19:09 |
| mnaser | pip install --force --cache-dir=/output/wheels /output/wheels/*.whl -- only installs the horizon wheel, nothing else | 19:10 |
| mnaser | ooh | 19:11 |
| mnaser | mordred: i think Open10K8S caught the issue | 19:11 |
| mordred | yeah? | 19:11 |
| mnaser | mordred: see this diff https://www.diffchecker.com/07K8988S | 19:12 |
| mnaser | any reason why we're saving the extras inside path/name/requirements.txt instead of path/requirements.txt? | 19:12 |
| mordred | yeah - originally the idea was that you could name the extra after the image name | 19:13 |
| mnaser | oh i see | 19:13 |
| mordred | so if you wanted an extra only for the zuul-executor image | 19:13 |
| mordred | BUT | 19:13 |
| mordred | I don't think that's doing a thing currently | 19:13 |
| mordred | because we tend to only run assemble once | 19:13 |
| mnaser | so the two things in play are: requirements.txt is not being modified to append those extra requirements, wheel sre installed in /output/wheels/wheels/xx/xx/foo.wheel and only horizon is inside /output/wheels/horizon.whl | 19:14 |
| mnaser | the latter might not be that much of an issue i think | 19:14 |
| mnaser | but the first one is, because nothing is making sure the memcache wheel is ending up on the final image | 19:14 |
| mordred | yeah. so - lemme look at something | 19:15 |
| mordred | mnaser: yeah - I think the extra dir isn't a ton of an issue - we install horizon and then the requirements.txt file - and in both cases we point at the cache dir - so if pip is writing the cache dir to /output/wheels/wheels - it'll read form there - so as long as it's getting the right list on the input it should do the right thing on the output | 19:18 |
| mordred | so I think Open10K8S is right - we need to update that to append the extras to requirements.txt | 19:18 |
| mordred | because I don't think we're using the "only install this extra in this image - that'sa. leftover from pbrx - we don't know that in assemble | 19:19 |
| clarkb | mordred: wouldnt it get picked up by the wheel installs though | 19:19 |
| clarkb | we install everything in the wheel cache or attempt to | 19:19 |
| mnaser | clarkb: you only install the thing we just built aka horizon, the wheel cache is a weird layout on disk like /xx/xx/foo.whl | 19:20 |
| mnaser | so we'd have to do something like installing /output/wheels/wheels/*/*/*.whl or something | 19:20 |
| clarkb | maybe just do a find | 19:20 |
| mnaser | files are like this: /output/wheels/wheels/8f/14/b5/d4c1642a6d221ac2fb3e7bbdcad180929d2f2b7c9c92cd3cdc/XStatic_term.js-0.0.7.0-py3-none-any.whl | 19:21 |
| clarkb | but I think that was the intent | 19:21 |
| mordred | no - let's fix the thing | 19:21 |
| mordred | no - it's not - the intent is that we use the wheel cache as a wheel cache | 19:21 |
| mordred | and just install the thing | 19:21 |
| clarkb | mordred I see | 19:21 |
| mordred | but installing the thing with the wheel cache pointed to the right thing should cause the right things to get installed | 19:21 |
| mnaser | Open10K8S: wanna push up a patch to fix get-extra-packages in the repo with gerrit? | 19:22 |
| mordred | clarkb: we do "/output/wheels/*whl" purely because we don't know what the wheel name is | 19:22 |
| mordred | mnaser, Open10K8S we should make sure we collect the extras and append to requirements.txt BEFORE we make the primary wheel | 19:23 |
| mordred | otherwise the metadata won't pick it up right | 19:23 |
| mnaser | ah yes | 19:23 |
| mordred | mnaser: as a followup, we should also potentially allow passing a list of extras you care about | 19:23 |
| mordred | like - keystone has several and might not want ALL of them - a set of defined extras might be a nice improvement | 19:24 |
| mnaser | ya that would be nice, allowing different build profiles | 19:24 |
| mordred | oh - actually ... | 19:24 |
| mordred | can I suggest a slightly different fix | 19:24 |
| clarkb | fwiw I've always been a fan of just installing all the things | 19:24 |
| clarkb | the python libs arerelatively small so the cost is low | 19:25 |
| clarkb | and then users dont get confused about why something doesnt work | 19:25 |
| mordred | clarkb: yah - but I thinkj some pre-existing thigns might have conflicting package sets | 19:25 |
| mnaser | in openstack context people were complaining about "oh but i dont want vmware libs when i deploy with libvirt" | 19:25 |
| mordred | so might have a setup.cfg that wouldn't work if you tried to install all of them - and they might want a get-out-of-jail | 19:25 |
| clarkb | mnaser: ya but it doesnt actually matter compared to installing python | 19:25 |
| mordred | mnaser: well - I agree with clarkb on that one - they should get over it - it's just libraries :) | 19:25 |
| clarkb | the cost ie basically nil | 19:25 |
| mnaser | please chime in then https://review.opendev.org/#/c/720107/ :) | 19:26 |
| mordred | I'm more concerned about cases where it's literally conflicting things | 19:26 |
| mordred | mnaser: instead of changing to append ... | 19:26 |
| mnaser | extra-requirements.txt? | 19:26 |
| mordred | let's change assemble to look for /output/wheels/*/extra-requirements.txt and install those too | 19:26 |
| mordred | mnaser: and then as a followup - let assemble take an option which is the name of an extra to only install in addition to the main thing | 19:27 |
| mordred | mnaser: that way we COULD have dockerfiles that run assemble with a specific extra as an argument to produce a more targetted sub-image | 19:27 |
| mordred | does that make sense? | 19:27 |
| mnaser | doesn't that mean that the wheel itself will not have that extra included as a dependency? | 19:27 |
| mnaser | so really we don't install anything in build-time and only "side-load them" in stage time | 19:28 |
| mordred | yeah - but that's ok - just do "pip install /output/wheels/*whl -f/ouput/wheels/extra1/extra-requirements.txt -f/output/wheels/extra2/extra-requrements.txt" | 19:28 |
| mordred | mnaser: oh - good point | 19:28 |
| mordred | mnaser: nevermind - let's skip what I said | 19:28 |
| mordred | we need to run the installs | 19:29 |
| mordred | let's fix it like Open10K8S found - and we can do the thign I'm thinking as a followup - since I think it'll be more involved to get right | 19:29 |
| clarkb | vos release is still going | 19:29 |
| mnaser | cool, makes me wonder how this even worked ever :) | 19:29 |
| mnaser | cause i know zuul relies on extras i think | 19:30 |
| mordred | mnaser: yeah - that's my question ... we do | 19:30 |
| mnaser | mordred: https://opendev.org/zuul/zuul/src/branch/master/Dockerfile#L46 | 19:31 |
| mnaser | aha. | 19:31 |
| mnaser | so we'd break zuul if we did this | 19:31 |
| mordred | ah - that's how | 19:31 |
| mordred | yeah | 19:31 |
| mordred | mnaser: so - maybe just doing that pattern for now? (which is sort of what I was suggesting above) | 19:32 |
| mnaser | mordred: yeah i guess that what we'll do | 19:33 |
| mordred | mnaser: and we can improve it by letting assemble take an optional list of extras to install so you don't have to do that | 19:33 |
| clarkb | how does that install extras? | 19:33 |
| clarkb | the extras arent in the requirements file right? | 19:33 |
| mordred | clarkb: that requirements file is created by the extras extraction in the builder image | 19:34 |
| clarkb | I see so it isn't part of the main repo, its generated | 19:34 |
| mordred | yeah | 19:35 |
| mordred | mnaser: clearly we need better api and better api docs here don't we? | 19:35 |
| mnaser | mordred: i think so, maybe those images could be on their own repo | 19:36 |
| mnaser | clarkb, mordred: can we land https://review.opendev.org/#/c/713953/11 ? we are using it and it works :) | 19:38 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Allow requesting a list of extras to install https://review.opendev.org/722125 | 19:39 |
| clarkb | opendev doesn't use uwsgi anywhere | 19:40 |
| clarkb | mnaser: I think we are the wrong venue for that | 19:40 |
| mordred | clarkb: I was thinking it was ok because the python-base and python-builder images are generally useful and it would be nice to be able to rebuild all three if we change any of them | 19:40 |
| mordred | although I can also see not wanting to | 19:41 |
| mordred | mnaser, clarkb: ^^ that might be some nice syntactic sugar to help with the extras install | 19:41 |
| mnaser | things like pastebin might need that image fwiw | 19:41 |
| clarkb | mnaser: why does it set PBR_VERSION=0.0 | 19:41 |
| mnaser | clarkb: /me points at mordred "he did it" | 19:42 |
| mordred | mnaser: yeah - in fact, maybe if we update pastebin to use that it would make more sense in opendev | 19:42 |
| *** diablo_rojo has joined #opendev | 19:42 | |
| clarkb | grepping for PBR_VERSION in system-configs existing images doesn't show us doing that | 19:42 |
| mordred | clarkb: there was a reason ... trying to think what it was | 19:42 |
| mordred | clarkb: it might just be leftover from previous hacking | 19:42 |
| mnaser | mordred: i think you added it when you did uwsgi install via assmeble | 19:43 |
| mnaser | bc it tries to do a pip install or something | 19:43 |
| mordred | oh | 19:43 |
| mordred | it's because we're faking a pbr install | 19:43 |
| mordred | so that python builder works | 19:43 |
| clarkb | I think thats a bad thing to do | 19:43 |
| mordred | clarkb: see the setup.cfg and friends in that repo | 19:43 |
| clarkb | all your versions will be wrong as a result | 19:43 |
| clarkb | mordred: that repo? | 19:44 |
| mordred | in that change | 19:44 |
| mordred | the other files | 19:44 |
| mordred | but I agree - let me relook at a solution for that real quick | 19:44 |
| mordred | I've got it | 19:45 |
| clarkb | out of curiousity why can't the images just use normal python-* and add uwsgi to their deps? | 19:45 |
| clarkb | this seems unnecessary if you can pip install uwsgi | 19:45 |
| mordred | because you need bindep stuff to install uwsgi - and it's not strictly a requirement for the python project itself, just for the images of that project | 19:46 |
| mordred | but - I think I have an idea of how to simplify this | 19:47 |
| clarkb | I mean if the project is expected to run with uwsgi it kind of is a dep of the python project | 19:47 |
| clarkb | I -1'd for the PBR_VERSION thing because I think that is an actual bug. That can leak around | 19:48 |
| clarkb | (its only intended to be set for the uwsgi install thing but if its in the env I think other pbr invocations will use it too) | 19:48 |
| corvus | what's the next thing i can help with re nodepool ansiblification? | 19:49 |
| clarkb | corvus: vos releases still running on ubuntu mirror updates. But I think we can move forward on ansible + containering the other x86 builders? (I don't know what that requires though) | 19:50 |
| corvus | i think, if i'm following correctly, mordred's strategy has shifted back 'get builders running in containers' since the /proc problem has been resolved | 19:51 |
| corvus | so i think that means nodepool isn't blocked on focal? | 19:51 |
| clarkb | corvus: correct | 19:51 |
| corvus | maybe the next step is the dedicated arm64 image idea | 19:51 |
| corvus | i think i need mordred to indicate whether he's started on patches for that | 19:52 |
| clarkb | that would enable a rebuild of nb03 which is probably the biggest blocker right now? since nb01/02 can be rebuilt like 04 | 19:52 |
| corvus | yeah, that's what i'm thinking | 19:52 |
| corvus | i wonder if nb04 is back in prod | 19:52 |
| corvus | i'll check on that and if not, see what's necessary while i wait for an update from mordred | 19:52 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Allow passing an arbitrary package list to assemble https://review.opendev.org/722133 | 19:54 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Add a uwsgi-base container image https://review.opendev.org/713953 | 19:55 |
| mordred | corvus: I have not - I was caught up in some calls - I'm now done and about to make the dedicated-arm image jobs | 19:55 |
| openstackgerrit | James E. Blair proposed openstack/project-config master: Revert "Move Ubuntu builds away from nb04" https://review.opendev.org/722134 | 19:56 |
| corvus | mordred, clarkb: cool; nb04 is running the latest nodepool image, so maybe we can go ahead and merge ^ now | 19:56 |
| mordred | corvus: ++ | 19:56 |
| clarkb | mordred: corvus can you point me to the place where the new image fixed the debootstrap issue? | 19:56 |
| corvus | clarkb: see commit message | 19:57 |
| mordred | clarkb: https://review.opendev.org/#/c/721394/ | 19:57 |
| clarkb | got it thanks | 19:57 |
| mordred | yeah | 19:57 |
| mordred | I would love it if we could convince debian to land those - if not, I think this is fine for now, but also think we should investigate switchign to mmdebootstrap | 19:58 |
| mordred | in dib | 19:58 |
| corvus | or using the containerfile element? | 19:58 |
| mordred | corvus: yeah - liek - I think doing this is good enough for a time - but I'd like a long-term solution to be better | 19:58 |
| corvus | i kinda we all felt that the containerfile element would be a good long term plan | 19:58 |
| mordred | yeah - although I also think that fixing dib so that you can run any of the elements in a container without breaking the container would be good from a dib pov - we can't exactly prevent nodepool users from using arbitrary elements - unless we want to update nodepool's config align more explicitlyu to the containerfile element | 20:00 |
| mordred | which is to say - yes - and I think the ppa workaround for now is fine as a workaround as we work on the long term plan | 20:00 |
| clarkb | mordred: was debian not happy with those changes? | 20:01 |
| mordred | clarkb: they aren't landed yet - I think there was review comments that wasn't happy about the need to care about being in a container and hinted at wanting things to be different | 20:02 |
| clarkb | I mean unmounting /proc seems like a bad idea in any context :) | 20:03 |
| corvus | mordred: does this error mean anything to you? https://zuul.opendev.org/t/openstack/build/3d3f5320bd9a40efa659e4aa9eebf21b/log/logs/containerfile_bionic-build-succeeds.FAIL.log#146-147 | 20:04 |
| mordred | corvus: no | 20:05 |
| corvus | k, we might need more debug logging or a node hold there | 20:05 |
| mordred | corvus, mnaser: remote: https://review.opendev.org/722135 Actually install extras from nodepool_base | 20:06 |
| mordred | is based on the things we learned from mnaser's image issue earlier | 20:07 |
| mordred | (turns out we did not succeed in adding objgraph and yappi to the nodepool image) | 20:07 |
| mnaser | mordred: your uwsgi-base attempt had a sad: [0m[91mpython: can't open file 'setup.py': [Errno 2] No such file or directory | 20:19 |
| clarkb | ubuntu-ports vos release completed | 20:19 |
| clarkb | I'm going to clean up its trusty indexes now and revos release | 20:20 |
| clarkb | oh hey maybe ther are already gone (perhaps ianw removed them?) | 20:20 |
| clarkb | oh right its because we never mirrored trusty ubuntu-ports \o/ | 20:21 |
| mnaser | mordred: it actually took the path with -z $PACKAGES in the code | 20:21 |
| clarkb | ok I'm going to remove my lock for ubuntu-ports and call that one done | 20:21 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Build arm64 versions of python-base and python-builder https://review.opendev.org/722140 | 20:23 |
| mordred | corvus: ^^ how does that look | 20:23 |
| clarkb | I'm dialing back the quota for ubuntu-ports now | 20:24 |
| mordred | mnaser: why did it do that??? | 20:25 |
| clarkb | will do that for ubuntu as well (adding that note to the etherpad) | 20:25 |
| mnaser | mordred: i have no idea bc i made a simple local assemble and the $* behaviour is right | 20:25 |
| mnaser | maybe because docker? | 20:25 |
| corvus | mordred: comment on something we should have caught earlier | 20:25 |
| mordred | corvus: oh poo. so we will need a dedicated base job | 20:28 |
| corvus | do we have reliable nested-virt yet? | 20:29 |
| corvus | (i'm just trying to weigh the reliability of the arm clouds vs nested virt) | 20:29 |
| clarkb | corvus: we ~2 clouds running nested virt evaluation flavors | 20:30 |
| openstackgerrit | Merged opendev/system-config master: Allow requesting a list of extras to install https://review.opendev.org/722125 | 20:30 |
| clarkb | I don't think its to the point of 100% relaible yet, but those clouds have indicated they are trying to make it work | 20:30 |
| clarkb | (so if it breaks they will actually try to fix it) | 20:30 |
| corvus | and one arm64 cloud | 20:31 |
| corvus | mordred: as a bonus, we need a job like that for building and publishing tags | 20:34 |
| corvus | (i mean docker images corresponding to git tags) | 20:34 |
| mordred | corvus: hrm. I'm torn - maybe the docker qemu thing is the right choice? we'd to add a flag to the build and upload jobs to say "use buildx" | 20:36 |
| corvus | mordred: yeah, i'm unsure too. buildah doesn't (yet) have support for this; but we could probably run buildah inside of qemu ourselves (oy) | 20:37 |
| corvus | (we don't use buildah, yet, just thinking ahead) | 20:37 |
| corvus | mordred: actually, are you sure we need *nested* virt? | 20:38 |
| corvus | mordred: i mean, we're talking cross-arch -- the only thing nested virt would get us is a fast build inside of qemu for the native arch | 20:38 |
| mordred | corvus: oh - yeah - maybe not | 20:38 |
| mordred | corvus: that's a very good point | 20:39 |
| corvus | mordred: but if we split it into different jobs, maybe we run the native build normally, and then a buildx qemu emulated build for the others | 20:39 |
| corvus | i wonder how long that takes; maybe i'll try that on my desktop? | 20:39 |
| mordred | corvus: docker buildx build --platform=linux/amd64,linux/arm64 . | 20:40 |
| mordred | https://github.com/docker/buildx/#with-buildx-or-docker-1903 | 20:40 |
| fungi | this problem goes on the list of things which would be so much easier if gerrit let you review tags | 20:41 |
| clarkb | oh ya nested virt may not help much good point | 20:42 |
| fungi | (the building from tags problem i mean) | 20:42 |
| corvus | mordred, fungi: well, re tags... hrm, actually... maybe we can promote them | 20:43 |
| corvus | we'd still need to map the change to the merge commit | 20:43 |
| mordred | I was thinking we were waiting on zuul v4 to implement support for promoting from the previous ... | 20:44 |
| corvus | but i think that can be done reliably with git? | 20:44 |
| fungi | oh, maybe | 20:44 |
| mordred | yeah - that - I thought last we discussed it the plan was to record something in the zuul db so that we could look up the change that built an image for a given commit so that we could promote the actual artifact | 20:44 |
| corvus | it might be possible. it's not trivial. not sure how motivated we are to solve it. | 20:45 |
| mordred | and we were waiting on v4 to implement that | 20:45 |
| corvus | mordred: that's not ringing a bell. | 20:45 |
| mordred | because ultimately when we tag a commit if we've otherwise promoted things built in gate we'd conceivably want not to rebuild the image as much as promote the previously gated image | 20:45 |
| corvus | at least, i don't see how anything planned for v4 helps | 20:45 |
| mordred | corvus: my understanding was that we'd need to record the mapping so we'd know the docker sha for a given change - but it's entirely possible I just hallucinated this | 20:46 |
| fungi | alternatively, if we built a new image from the tag, we'd want the tag to not get applied to the canonical copy of the repo unless the image is viable and can pass tests | 20:46 |
| fungi | (which is where the reviewing tags thought came into play) | 20:46 |
| mordred | yeah - that's the problem - it's non-trivial | 20:47 |
| corvus | mordred: oh, maybe the idea was to use direct-push so we know the merge commit | 20:47 |
| mordred | yeah | 20:47 |
| mordred | maybe that was it | 20:47 |
| corvus | that is one option | 20:47 |
| mordred | so that we could then reverse map the merge commit to the docker sha | 20:47 |
| corvus | i do think that digging the change out from the merge commit via git is an option | 20:47 |
| mordred | yeah | 20:48 |
| corvus | but it's decided non-trivial | 20:48 |
| clarkb | ubuntu mirror has released | 20:48 |
| mordred | corvus: incidentally - my laptop just built amd64 and arm64 in parallel of python base in short order | 20:48 |
| clarkb | I'm going to reduce the quota on that volume now then remove my locks | 20:48 |
| mordred | corvus: 84 seconds | 20:48 |
| corvus | mordred: cool, docker is giving me a really weird error | 20:49 |
| corvus | so i have no dockerx | 20:49 |
| clarkb | #status log Removed Trusty from our Ubuntu mirrors and added Focal. Updates have been vos released and should be in production. | 20:50 |
| openstackstatus | clarkb: finished logging | 20:50 |
| mordred | you have to install it as a plugin - instructions at https://github.com/docker/buildx/#with-buildx-or-docker-1903 | 20:50 |
| corvus | mordred: no i get that | 20:50 |
| corvus | mordred: i am unable to build the plugin | 20:50 |
| mordred | corvus: I'm cooking up runing instructions | 20:50 |
| mordred | corvus: oh! | 20:50 |
| mordred | corvus: maybe you need the instructions for older docker above that section? | 20:50 |
| corvus | mordred: nope | 20:50 |
| corvus | there is a definite error which i would like to share, but am not yet comfortable sharing | 20:51 |
| mordred | kk | 20:51 |
| corvus | got it. it was left over from registry testing | 20:52 |
| mordred | corvus: so - I can build the image, but I can't look at it because pushing to a registry is the only currently supported export format | 20:55 |
| mordred | I mean - I guess I could push to emonty/python-base for now | 20:55 |
| mordred | corvus: ok - I have pushed to emonty/python-base and it looks correct | 20:58 |
| clarkb | infra-root if anyone else is willing to review a gitea upgrade change from mordred, https://review.opendev.org/#/c/720202/1, I'm happy to approve that and watch it (probably tomorrow morning?) | 20:59 |
| clarkb | that was hiding on my todo list from last week | 20:59 |
| fungi | i'll take a look now | 20:59 |
| fungi | also i noticed we're getting some strange cronspam from the gitea servers, for how long i'm not sure | 20:59 |
| mordred | corvus: http://paste.openstack.org/show/792566/ | 21:00 |
| mordred | corvus: that's the sequence of commands I ran to build and push to a registry | 21:00 |
| corvus | mordred: i'm trying to get a comparisen of an arm64 build to amd64 | 21:00 |
| corvus | mordred: can you try doing those one at a time? | 21:00 |
| fungi | there's a mysqldump-in-a-container cronjob complaining "No container found for mariadb_1" | 21:00 |
| mordred | corvus: yeah - you want me to time them? | 21:00 |
| corvus | yep | 21:00 |
| corvus | (i've build an amd64 image, but i apparently need more qemu to build an arm64 image) | 21:01 |
| mordred | corvus: k. one sec | 21:01 |
| clarkb | fungi: thats our db backups I bet its fallout from the docker-compose upgrade I'll take a look | 21:01 |
| corvus | mordred: i suspect you're running qemu in your docker-provided linux vm | 21:01 |
| mordred | corvus: I'm going to skip push this time | 21:02 |
| mordred | corvus: and yes | 21:02 |
| mordred | corvus: I'm doing python-builder this time because it does more - and also so that I don't have any local cache | 21:03 |
| mordred | corvus: nicely - the buildkit output gives us times for each step too | 21:04 |
| mordred | I'll paste the whole thing | 21:04 |
| clarkb | fungi: that is really weird | 21:04 |
| clarkb | docker-compose ps shows nothing | 21:04 |
| clarkb | but docker ps does | 21:04 |
| mordred | the qemu arm is definitely slower - but not _stupid_ slower | 21:04 |
| clarkb | fungi: I've got it I think | 21:05 |
| clarkb | give me a few mintes to confirm | 21:05 |
| mordred | corvus: http://paste.openstack.org/show/792567/ | 21:07 |
| clarkb | fungi: its a path issue. I'll get a change up as soon as I can type it up | 21:07 |
| mordred | corvus: less than a second for the first two steps, 2 seconds for the third, 4 minutes total for the build - most expensive build step took 36s on x86 and 158s on arm | 21:08 |
| fungi | clarkb: cool, thanks! | 21:08 |
| corvus | mordred: so if we run them on your laptop, we can expect the builds to take ~4x | 21:09 |
| mordred | corvus: it's doing them in parallel - so we're long-poled on the arm time - x86 finished quickly | 21:09 |
| mordred | yeah | 21:09 |
| corvus | mordred: oh i thought you were doing it serially | 21:09 |
| corvus | mordred: can you repeat that and do it serially? | 21:10 |
| mordred | I did the steps serially and timed | 21:10 |
| mordred | sure | 21:10 |
| corvus | i'm confused | 21:10 |
| mordred | I am too - I'm not sure what you're wanting to see? | 21:10 |
| corvus | mordred: did the arm and amd images build in parallel or serial? | 21:10 |
| mordred | parallel | 21:10 |
| mordred | I misunderstood what things you wanted invididual timing of | 21:10 |
| corvus | mordred: okay, that's going to give us bad numbers for estimating what running multiple jobs in parallel would do | 21:10 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Fix rooted path to docker-compose https://review.opendev.org/722145 | 21:11 |
| clarkb | fungi: infra-root ^ fyi | 21:11 |
| clarkb | note docker-compose is still in /usr/bin/ on gitea0X which is why the error is odd | 21:11 |
| mordred | corvus: nod. out of curiosity - why do we think that might be a better choice? (doing the serial steps now) | 21:11 |
| corvus | mordred: so i think if you can do an amd build, then an arm build, that'll do it | 21:11 |
| clarkb | on other hosts it seems to have been removed properly | 21:11 |
| corvus | mordred: i'm not sure it will; it depends on how the build time compares to the setup time | 21:12 |
| mordred | corvus: doing that now | 21:12 |
| mordred | ah - well - the setup time was 2 seconds | 21:12 |
| *** dpanech has joined #opendev | 21:12 | |
| corvus | mordred: i mean job setup | 21:12 |
| mordred | nod | 21:12 |
| mordred | corvus: x86 took 48s | 21:14 |
| corvus | mordred: a zuul image build job takes 11 minutes to build the docker image and 42 minutes to run. to be honest, i don't know what's happening in the other 32 minutes. | 21:16 |
| fungi | clarkb: in a similar vein, the mysqldump cronjob on the etherpad server is complaining "/bin/sh: 1: /usr/bin/docker-compose: not found" | 21:17 |
| clarkb | fungi: yup its fixed in the same change | 21:17 |
| corvus | we have those nice ansible task times now, and nothing other than the docke image build steps is > 1 minute | 21:17 |
| clarkb | fungi: it was etherpad that made me realize what was going on | 21:17 |
| mordred | corvus: weird | 21:17 |
| fungi | clarkb: aha, got it | 21:17 |
| corvus | mordred: i'm looking at https://zuul.opendev.org/t/zuul/build/a8798b497e2c44baa9b514e06786e37c | 21:17 |
| mordred | corvus: fwiw: https://www.docker.com/blog/getting-started-with-docker-for-arm-on-linux/ has a couple of steps in it that I don't have to do on my laptop - notably docker run --rm --privileged docker/binfmt:820fdd95a9972a5308930a2bdfb8573dd4447ad3 | 21:17 |
| clarkb | corvus: https://zuul.opendev.org/t/zuul/build/a8798b497e2c44baa9b514e06786e37c/log/job-output.txt#11858 | 21:18 |
| clarkb | the job is pausing | 21:18 |
| corvus | mordred: yeah the binfmt situation on bionic is bad; we may want to do this on focal | 21:18 |
| corvus | clarkb: ah thanks :) | 21:18 |
| corvus | we should see if we can get that in the cosole log | 21:19 |
| corvus | okay, so the docker build is about 11 minutes out of 21 | 21:19 |
| corvus | or, really, we don't care about the time after the pause | 21:19 |
| clarkb | right thats quickstart or whatever time and isn't directly worried about in the build context aiui | 21:20 |
| mordred | corvus: yeah | 21:20 |
| mordred | corvus: 3:31 to run the arm - full log http://paste.openstack.org/show/792568/ | 21:20 |
| dpanech | Hi there, I need to push to f/centos8 branch in starlingx/kernel.git, can someone help me? I don't have the permissions. I'm splitting out the kernel from integ.git. | 21:21 |
| dpanech | sgw: please confirm it's ok | 21:21 |
| corvus | so let's say 5 minutes setup time for the job, and 11 minutes docker build time | 21:21 |
| *** DSpider has quit IRC | 21:22 | |
| sgw | Hi gang, yes dpanech needs the branch to complete the kernel split work in starlingx. I might have complicated things by creating the f/centos8 branch, so it might need to be deleted first | 21:22 |
| mordred | corvus: yeah. so we could imagine the qemu arm build would make that take 40-45 minutes - although we still might want to test it - because a ton of what's going on in that docker build is zuul-manage-ansible installing ansible a bunch - so cpu vs. disk performance is going to come in to play | 21:22 |
| mordred | or might come in to play | 21:23 |
| clarkb | sgw: dpanech give me a couple minutes | 21:23 |
| corvus | mordred: do you have an amd build yet? | 21:23 |
| mordred | corvus: yeah - it was 48s | 21:23 |
| corvus | oh sorry i missed that | 21:23 |
| sgw | clarkb: thanks, ping when your ready for us. | 21:23 |
| mordred | corvus: so still roughly 4x longer - on my laptop | 21:24 |
| fungi | sgw: dpanech: clarkb: the acl for that repo says the members of this group can create new branches in it already: https://review.opendev.org/#/admin/groups/starlingx-release | 21:24 |
| fungi | sgw: you're one of them | 21:24 |
| corvus | mordred: https://zuul.opendev.org/t/openstack/build/3199c3bc10564997a63c589b65286b68/console took 51 seconds, so i think your laptop == build nodes for this comparison | 21:24 |
| mordred | corvus: good to know :) | 21:24 |
| fungi | sgw: go to https://review.opendev.org/#/admin/projects/starlingx/kernel,branches and you should see a "create branch" button | 21:24 |
| fungi | if not, we can troubleshoot it | 21:25 |
| mordred | corvus: oh - although - it's worth noting we don't need to do this for all of our images | 21:25 |
| sgw | fungi: yea, I was able to create the branch, but what we really want is for dpanech to be abke to push the new branch. so I really need to delete the one I just created and allow dpanech to push the branch direclty | 21:25 |
| clarkb | sgw: you mean bypassing review? | 21:25 |
| fungi | sgw: oh... you have content you want to bypass code review and push directly? | 21:25 |
| mordred | corvus: really only python-base, python-builder and nodepool-builder - although it would likely be awkward to just do builder so might as well do nodepool - we might want to look at nodepool build logs instead of zuul ones | 21:25 |
| sgw | yes, in this case it's to preserve the history | 21:25 |
| mordred | since zuul's is extra long due to the manage ansible stuff | 21:25 |
| corvus | mordred: splitting into separate jobs would allow us to build an untested arm64 image while we continue jobs testing the built amd64 images | 21:26 |
| clarkb | sgw: dpanech typically that requires a gerrit admin. | 21:26 |
| fungi | sgw: it's probably safest to have one of us do it in that case, otherwise you'll need a group added to the acl which has the ability to push commits and merge requests on behalf of other users | 21:26 |
| corvus | mordred: yeah, looks like a nodepool build is going to be about 5m | 21:27 |
| corvus | mordred: (or 20m for amd64) | 21:27 |
| fungi | sgw: dpanech: is there a public copy of that branch somewhere one of us can pull from? | 21:27 |
| sgw | fungi: was just about to ask what you wanted! | 21:27 |
| mordred | so - then the question is - do we want to always do separate and always have a job to combine a manifest - or do we want both versions - do a combined build for python-base/python-builder and then do a separate build for nodepool? | 21:27 |
| sgw | dpanech: do you have a public github that you can push your work to? | 21:28 |
| dpanech | sgw: fungi: give a moment | 21:28 |
| corvus | mordred: this is one option: http://paste.openstack.org/show/792569/ | 21:33 |
| corvus | mordred: with timing http://paste.openstack.org/show/792570/ | 21:36 |
| dpanech | sgw: fungi: here you go: https://github.com/dpanech/starlingx-kernel/tree/f/centos8 | 21:36 |
| clarkb | dpanech: sgw worth noting for the future that the repo creation step will import existing branches so you can prep the import step to avoid the work at this stage | 21:37 |
| sgw | clarkb: thanks, since we have lost Dean, we lost some of that knowledge, hopefully we don't do too much spliting like this. | 21:38 |
| openstackgerrit | Mohammed Naser proposed opendev/system-config master: Allow passing an arbitrary package list to assemble https://review.opendev.org/722133 | 21:38 |
| openstackgerrit | Mohammed Naser proposed opendev/system-config master: Add a uwsgi-base container image https://review.opendev.org/713953 | 21:38 |
| mordred | corvus: yeah. and with quick-start - the total job time is likely not extended | 21:38 |
| corvus | mordred: and the other option: http://paste.openstack.org/show/792571/ | 21:39 |
| fungi | sgw: dpanech: cool, i'll pull from https://github.com/dpanech/starlingx-kernel/tree/f/centos8 and push --force to https://opendev.org/starlingx/kernel/src/branch/f/centos8 through review.opendev.org momentarily | 21:39 |
| corvus | mordred: though i may be guessing a bit on the setup times.... | 21:39 |
| clarkb | fungi: sgw dpanech fwiw it does look like its just a merge commit and the .gitreview commits? | 21:39 |
| clarkb | fungi: possible that dpanech could push that merge commit afterall? | 21:39 |
| dpanech | fungi: thanks | 21:39 |
| clarkb | (also appears to be a fastforward from the current branch head) | 21:39 |
| clarkb | but fungi pushing it should also work fine | 21:40 |
| corvus | mordred: maybe for this particular case of nodepool-builder it doesn't matter that much and we should just do it in one job? | 21:40 |
| dpanech | wait | 21:40 |
| dpanech | it's not going into master | 21:40 |
| mordred | corvus: so - the second option is quite a bit easier to implement (no need for merge images - just need to add an option to the build jobs) | 21:40 |
| dpanech | its going into f/centos8 | 21:40 |
| clarkb | dpanech: I know | 21:40 |
| dpanech | I tried pushing before and it didn't allow me | 21:40 |
| clarkb | dpanech: you have two merge commits "merge branch 'master'" | 21:40 |
| mordred | yeah - but - I think the first option has some potentially nice benefits and is also still slighly faster | 21:40 |
| fungi | dpanech: yep, sgw created the f/centos8 from master head | 21:41 |
| clarkb | dpanech: yes you need special permissions to push merge commits | 21:41 |
| sgw | dpanech: the branch needs an update .gitreveiw also | 21:41 |
| clarkb | sgw: that is tehre too | 21:41 |
| corvus | mordred: i think i'd be in favor of the second option for now and see how it goes | 21:41 |
| sgw | clarkb: ah good, something I had forgotten once! | 21:41 |
| clarkb | I was just pointing out that if you were in the correct group I think you could push those two merge commits for review | 21:41 |
| mordred | corvus: so we could followup up with option #1 as an later improvement - or if we wanted to start making arm images of zuul | 21:41 |
| clarkb | where that could fail is if the commits in the merge weren't already known to gerrit | 21:41 |
| clarkb | (and that is where an admin pushing would be required) | 21:42 |
| mnaser | mordred: i know what happened. we don't have a requires on the zuul job for uwsgi-base, so it was pulling master image | 21:42 |
| mnaser | s/master/latest/ | 21:42 |
| mordred | mnaser: hah | 21:42 |
| mordred | mnaser: that makes total sense | 21:42 |
| mordred | mnaser: and thank god :) | 21:42 |
| mnaser | yeah i was starting to lose it | 21:42 |
| mnaser | i found a little bug, we needed to quote the "$PACKAGES" in the if to avoid issues with bash complaining | 21:43 |
| mnaser | but that was it | 21:43 |
| dpanech | i'm confused, did I screw something up? | 21:43 |
| clarkb | dpanech: I don't think so | 21:43 |
| clarkb | dpanech: https://review.opendev.org/#/admin/groups/1921,members that group is allowed to push merge commits like the ones you've made into gerrit for review | 21:44 |
| sgw | I think what is going on is that dpanech could have shared this work with me and I could have pushed the merge commit directly | 21:44 |
| clarkb | dpanech: if you were a member of that group I don't think you need fungi to push for you | 21:44 |
| clarkb | sgw: yes basically | 21:44 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Allow passing an arbitrary package list to assemble https://review.opendev.org/722133 | 21:44 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Add a uwsgi-base container image https://review.opendev.org/713953 | 21:44 |
| clarkb | I'm not 100% certain of that yet because gerrit does awnt to have previously seen all commits pushed itno it via a merge commit push | 21:45 |
| mordred | mnaser: like that ^^ ? | 21:45 |
| clarkb | so depending on the contents of those merge commits this may not be the case | 21:45 |
| sgw | And looking at the group, I could have added dpanech temporarily | 21:45 |
| clarkb | and either way its fine for fungi to push them | 21:45 |
| *** tobiash has quit IRC | 21:45 | |
| mnaser | mordred: left a note :) | 21:45 |
| mordred | mnaser: so I don't need the second one you said is fine? | 21:46 |
| dpanech | also: it's not just the review commits, there's a bunch of earlier commits from integ.git's centos8 in that branch, extracted with paths re-written | 21:46 |
| mnaser | mordred: in my local testing yes only the one at the if was needed | 21:46 |
| dpanech | still confused what you guys are talking about | 21:46 |
| mnaser | mordred: or i would se things like this -- ./assemble: line 6: [: foo: binary operator expected | 21:46 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Allow passing an arbitrary package list to assemble https://review.opendev.org/722133 | 21:47 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Add a uwsgi-base container image https://review.opendev.org/713953 | 21:47 |
| clarkb | dpanech: https://github.com/dpanech/starlingx-kernel/commits/f/centos8 I'm looking at that. 4th commit down is current master's HEAD | 21:47 |
| *** tobiash has joined #opendev | 21:47 | |
| clarkb | dpanech: that means you've got a fast forward of 3 commits from master|f/centos8 to the state you want | 21:47 |
| clarkb | dpanech: two of those commits are merge commits | 21:47 |
| clarkb | you are allowed to push merge commits to gerrit of commits that are already in gerrit if you have the push merge commit permissions which "starlingx-release" does have | 21:48 |
| mnaser | mordred: yay. ok, cool. i don't think we can take advantage of those because requires/provides needs things to be in the same tenant, correct? | 21:48 |
| fungi | clarkb: looks like it's rather a lot of commits | 21:48 |
| fungi | "Your branch is ahead of 'origin/f/centos8' by 31 commits." | 21:48 |
| mnaser | aka vexxhost tenant cant just do requires | 21:48 |
| fungi | (that's after fetching https://github.com/dpanech/starlingx-kernel f/centos8 and resetting --hard to it) | 21:48 |
| clarkb | fungi: ya its probably calculating the deltas pulled in by the merge | 21:48 |
| corvus | mnaser: correct | 21:48 |
| clarkb | fungi: but I think only the merge commits matter if the other commits were already accepted by gerrit | 21:49 |
| clarkb | fungi: because those other commits are accepted and reviewed and merged. What you'd be reviewing here is the combo of the two branches together via a merge commit | 21:49 |
| clarkb | (I don't actually know if the statement above is true in this particular case) | 21:49 |
| mnaser | corvus: bummer -- ok, so we have no means of testing that till we merge uwsgi-base (or if we decide to not have that image then we'll just move it in) -- the changes mordred introduced makes it more trivial to install it in build time anyways.. | 21:49 |
| openstackgerrit | James E. Blair proposed openstack/diskimage-builder master: WIP: boot test of containerfile image https://review.opendev.org/722148 | 21:50 |
| clarkb | but generally in say the feature branch case that we support that is how it works | 21:50 |
| mordred | mnaser: how about update the logdeit image to use it? | 21:50 |
| clarkb | you don't need an admin to support feature branches, you just need to push commits that gerrit has arleady accepted | 21:50 |
| mnaser | mordred: oh i could do that | 21:50 |
| clarkb | *push merge commits for commits that gerrit has already accepted | 21:50 |
| mnaser | sounds like work but if that means we can land uwsgi-base i'll do it :p | 21:50 |
| fungi | clarkb: also some of the commits like https://github.com/dpanech/starlingx-kernel/commit/f683f85 aren't in gerrit yet | 21:51 |
| dpanech | to be clear: there are earlier commits in that branch that are not in master, not just the top few | 21:51 |
| clarkb | fungi: gotcha so it wouldn't just work in this case because it violates the commit previously existing in gerrit rule | 21:51 |
| fungi | yup | 21:51 |
| fungi | i just wanted to double-check | 21:52 |
| mordred | mnaser: I got it | 21:52 |
| mordred | mnaser: one sec | 21:52 |
| clarkb | dpanech: yup the merge commits reconcile that | 21:52 |
| mnaser | mordred: im almost done ;) | 21:52 |
| mordred | mnaser: kk | 21:52 |
| clarkb | dpanech: if we ignore your specific situation for a minute the general rule is you can push merge commits to gerrit just fine. It does require an extra bit of permissions (starglinx-release has that) and gerrit must already know about the commits (basically this means they went through reivew on some branch) | 21:55 |
| clarkb | dpanech: going forward what this means is you don't need a gerrit admin to push for you as long as you keep teh work in gerrit | 21:56 |
| openstackgerrit | Mohammed Naser proposed opendev/lodgeit master: docker: switch to using uwsgi-base https://review.opendev.org/722149 | 21:56 |
| mnaser | mordred: ^ that should work | 21:56 |
| mnaser | except we don't really have any tests for the image right now, boo :X | 21:56 |
| mnaser | ... i have a helm chart that deploys from the image .. but thats also in another tenant, lol | 21:57 |
| fungi | #status log replaced content of starlingx/kernel f/centos8 branch with f/centos8 branch from github.com/dpanech/starlingx-kernel per sgw | 21:57 |
| openstackstatus | fungi: finished logging | 21:57 |
| fungi | dpanech: sgw: http://paste.openstack.org/show/792572/ | 21:57 |
| fungi | that's the output of the push command, for the record | 21:57 |
| fungi | you should be all set | 21:57 |
| sgw | thanks guys | 21:57 |
| fungi | you can ignore the warning lines, gerrit's just nagging about well-formatted commit messages | 21:59 |
| dpanech | clarkb: sorry I have little experience with gerrit. I thought every review = one patch/commit. Not sure how I would go about importing a pile of commits from another repo via gerrit. I guess I'll read up on it later or ask someone to show me... | 21:59 |
| mnaser | mordred: oh joy of joys, opendev/lodgeit is in the opendev tenant, lol | 21:59 |
| clarkb | dpanech: yes every review is one patch/commit. Git merge commits are sort of a workaround to that where you create a single commit that has multiple parents effectively combining streams | 22:00 |
| mnaser | oh but wait, opendev/system-config is in the opendev tenant, that should be ok | 22:00 |
| clarkb | dpanech: so you can still review a merge commit in gerrit as usual. The cavest ais that each of the commits involved in those merge streams must also have been gerrit changes at one point | 22:00 |
| dpanech | clarkb: you mean it can track them by change ids across repos or something? | 22:02 |
| clarkb | dpanech: it does it by commit sha so as long as those haven't chagned I think its fine with it | 22:02 |
| fungi | the idea is that you propose changes as you go along rather than winding up with a stack outside gerrit which can't be code-reviewed | 22:03 |
| clarkb | in this case I think there was some underlying repo rearrangement and that made it an exception to the rule | 22:04 |
| clarkb | I was just trying to make sure we don't think all merge commits need to go through a gerritadmin | 22:04 |
| mnaser | im seeing a lot of retries in zuul status | 22:04 |
| dpanech | clarkb: fungi: clear as mud, but ok. Can I ask you to hold my hand next time I need this ? :) | 22:05 |
| clarkb | dpanech: sure | 22:05 |
| mnaser | im seeing a bunch of retries in zuul status, expected due to any issues with a provider? | 22:06 |
| clarkb | mnaser: no | 22:06 |
| clarkb | (the known provider issues shouldn't cause retries) | 22:06 |
| mnaser | change 713953,15 had 4 jobs that retried | 22:06 |
| corvus | mirrors or images are more likely | 22:06 |
| mnaser | 2x had 2 attempts and 2x had 3 attempts | 22:06 |
| clarkb | ya its the indexes being unexpected sizes | 22:07 |
| clarkb | https://54b78720d7619cd1224f-698cf8cdc9dda09c975180d31d914d21.ssl.cf2.rackcdn.com/717754/2/gate/neutron-grenade-multinode/8ab891d/job-output.txt | 22:07 |
| clarkb | so we updated the mirror with new indexes and now mirrors are serving stale versions? | 22:07 |
| clarkb | corvus: you ran checkvol ? check something when static was sad maybe that will fix this? | 22:07 |
| fungi | dpanech: basically it's hard to say what you could have done in this case to avoid needing a push --force to incorporate commits which were not in gerrit without knowing what the development process was which led to the creation of those commits, but the general idea is to make sure commits are pushed into gerrit as reviews normally when possible | 22:08 |
| corvus | clarkb: i'll look up our convo from last time while you log into an afs head and see if you can find what's out of date? | 22:08 |
| corvus | clarkb: (so we can confirm the problem/fix quickly) | 22:08 |
| clarkb | corvus: k | 22:08 |
| clarkb | /afs/openstack.org/mirror/ubuntu/dists/bionic-updates/main/binary-amd64/Packages.gz is 1266584 bytes according to afs | 22:10 |
| corvus | clarkb: what does mirror.iad.rax.openstack.org think it is? | 22:10 |
| clarkb | /afs/openstack.org/mirror/ubuntu/dists/bionic-updates/Release: 3ed520b3f935f958321db0b24f340d8c 1266584 main/binary-amd64/Packages.gz | 22:11 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: pip-and-virtualenv: drop f31 & tumbleweed, rework suse 15 install https://review.opendev.org/721763 | 22:11 |
| corvus | clarkb: apt expected it to be 920439 ? | 22:12 |
| clarkb | corvus: it seems to agree | 22:12 |
| clarkb | File has unexpected size (1266584 != 1266587). Mirror sync in progress? | 22:12 |
| clarkb | corvus: it expected 1266587 | 22:12 |
| clarkb | but the Release file says 1266584 | 22:12 |
| corvus | oh sorry was looking at the wrong line | 22:12 |
| ianw | clarkb: i'm guessing the release didn't go as well as hoped ... lmn if i can help, i'm around | 22:12 |
| clarkb | ianw: well I think the reprepro side went ok. This looks like fallout due to afs caches? | 22:13 |
| clarkb | corvus: so I think that particular one may be fine now | 22:13 |
| clarkb | (afs eventually started serving synced up content) | 22:13 |
| mnaser | mordred: left 2 reviews on those 2 patches which i think are the last things needed to make it work :) | 22:13 |
| corvus | clarkb: huh i don't understand is 1266587 not the correct value? | 22:13 |
| clarkb | corvus: aiui the Release file eg http://mirror.iad.rax.openstack.org/ubuntu/dists/bionic-updates/Release gives you a hash and filesize for the other files | 22:14 |
| clarkb | corvus: in this case the file above says the Packages.gz file is the correct size | 22:14 |
| clarkb | corvus: the implication is that when this failed either the Release file or the Packages.gz file were out of date | 22:14 |
| clarkb | and that caused apt to be mad | 22:14 |
| corvus | hrm | 22:15 |
| corvus | i don't understand how this happens; | 22:15 |
| corvus | it's kind of the entire value proposition of using afs for this | 22:15 |
| fungi | unless there was an interim vos release after a failed reprepro execution | 22:15 |
| corvus | i think clarkb held the lock the whole time? | 22:16 |
| clarkb | corvus: I did | 22:16 |
| clarkb | I did release it when I was done though | 22:16 |
| fungi | right, and you only called vos release at the end | 22:16 |
| clarkb | but that was after the vos release had completed | 22:16 |
| clarkb | fungi: correct | 22:16 |
| fungi | once things should have been in sync | 22:16 |
| ianw | fwiw if it was done in the background automaticaly it would show up in http://grafana.openstack.org/d/ACtl1JSmz/afs?orgId=1 | 22:16 |
| fungi | yeah, so nothing released that volume without checking the flock | 22:17 |
| clarkb | is it possible apache is caching things for us | 22:17 |
| clarkb | what I notice in my example log is that it is a multinode job | 22:17 |
| clarkb | and each node complains about different indexes | 22:17 |
| clarkb | that implies to me that apache is serving correct content to some but not others? | 22:18 |
| clarkb | corvus: ^ re afs value perhaps this is part of the problem | 22:18 |
| corvus | i didn't think we had apache caching these? | 22:18 |
| clarkb | corvus: I didn't either | 22:18 |
| clarkb | it should all be done by suffix and on that vhost should only be pypi but I'm looking now to double check | 22:19 |
| clarkb | but notice on that example job on one host it is bionic-update universe that is mad and main is fine then on the other host its bionic-updates main that is mad | 22:19 |
| corvus | "Wed Apr 22 15:44:56 2020" is the last release, so those errors didn't happen to have two requests span the release | 22:20 |
| openstackgerrit | Mohammed Naser proposed opendev/system-config master: Allow passing an arbitrary package list to assemble https://review.opendev.org/722133 | 22:20 |
| clarkb | https://opendev.org/opendev/system-config/src/branch/master/modules/openstack_project/templates/mirror.vhost.erb#L82 is how we configure caching for specific suffixes | 22:21 |
| clarkb | and ya only /pypi and /pypifiles are set on the main vhost | 22:21 |
| clarkb | so /ubuntu shouldn't be cached | 22:21 |
| clarkb | (not by apache anyway) | 22:21 |
| openstackgerrit | Mohammed Naser proposed opendev/system-config master: Add a uwsgi-base container image https://review.opendev.org/713953 | 22:21 |
| openstackgerrit | Mohammed Naser proposed opendev/system-config master: Add a uwsgi-base container image https://review.opendev.org/713953 | 22:21 |
| clarkb | fwiw it seems like jobs are settling down? eg not many have failed due to retries | 22:22 |
| clarkb | so whatever the out of sync problem was may have corrected tiself (which doesn't make it easier to debug) | 22:22 |
| clarkb | it does definitely seem like we are serving differetn content to different hosts at roughly the same time | 22:23 |
| clarkb | that implies to me that afs is probably not the problem because those fs reads should all be consistent across that time period? | 22:23 |
| corvus | clarkb: do you know how many variations we have? | 22:25 |
| mnaser | also maybe it was problematic in a specific region? | 22:25 |
| corvus | (if there are only two, then it makes some kind of old/new mixup more likely; i don't know what >3 would mean) | 22:26 |
| clarkb | corvus: so far its just those two but I'm mostly only looking at the one job so far | 22:26 |
| clarkb | let me see if logstash has more | 22:26 |
| mnaser | (my comment about specific region is because they evenutally would pass with enough retries) | 22:26 |
| clarkb | mnaser: the region with known failures reports good data to my browser | 22:27 |
| clarkb | mnaser: which is another reason I think it corrected itself | 22:27 |
| clarkb | the most recent errors of this type in logstash are from 20:36UTC ish | 22:30 |
| clarkb | (and they go back well before that) | 22:30 |
| clarkb | the failure logstash is reporting seem to largely be from rax.ord | 22:31 |
| clarkb | implying that it could be a mirror state thing of some sort | 22:32 |
| clarkb | (we don't run the bulk of our jobs in rax-ord last I checked so it shouldn't have the bulk of these failures) | 22:32 |
| clarkb | logstash shows that it is bionic-updates main and universe and bionic-security main though pretty consistently | 22:33 |
| clarkb | which is the same set we see on this job in iad | 22:33 |
| clarkb | corvus: so maybe we try to flush those 3 then check logstash in a few hours (assuming that zuul isn't complaining too much)? | 22:34 |
| * clarkb makes a list of file paths | 22:34 | |
| corvus | clarkb: ok. i'm winding down for the day, so maybe we can check in tomorrow | 22:35 |
| clarkb | corvus: if I do a flush volume is that applied globally? | 22:35 |
| clarkb | or do I have to do it on each client? | 22:35 |
| clarkb | https://etherpad.opendev.org/p/tMfB9VRcqe7NhS9a4-ZX is updated with a list of paths to flush | 22:37 |
| openstackgerrit | Merged opendev/system-config master: Fix rooted path to docker-compose https://review.opendev.org/722145 | 22:40 |
| corvus | clarkb: that's a client thing | 22:41 |
| openstackgerrit | panticz.de proposed openstack/diskimage-builder master: Install required netbase package for ubuntu-minimal element. Based on similar commit for debian-minimal Icc81635870961943707cf6b3f61a9ddbd51cb8fd https://review.opendev.org/722164 | 22:42 |
| clarkb | corvus: k I'll run checkvolume and those flushes across our mirrors | 22:43 |
| clarkb | ord and iad are done | 22:43 |
| clarkb | does anyone know where we write out the zuul deployment vars? | 22:48 |
| clarkb | I want to make sure I get the correct list of mirrors | 22:49 |
| corvus | clarkb: you mean /etc/zuul/site-variables.yaml ? | 22:52 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: pip-and-virtualenv : fix fedora 30 install https://review.opendev.org/716795 | 22:54 |
| clarkb | corvus: ya I think so | 22:55 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Bionic functional tests should be voting https://review.opendev.org/716839 | 22:56 |
| ianw | infra-root: there is a stack out @ https://review.opendev.org/#/q/topic:builder-container which is all focused on getting all our image types fully boot tested under the container builder, review appreciated | 23:04 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: yum-minimal: strip env vars in chroot calls https://review.opendev.org/721726 | 23:18 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: [wip] switch func tests to containers https://review.opendev.org/721511 | 23:18 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Restore SUSE tests to gate https://review.opendev.org/721779 | 23:18 |
| clarkb | ok I think I got all the mirrors checkvolume'd and flushvolumed | 23:23 |
| clarkb | did it wrong the first pass, needed to use -path with flushvolume and it didn't error when I didn't | 23:23 |
| clarkb | so roughly after 23:23UTC April 22 we should see things happier in logstash? | 23:23 |
| *** Dmitrii-Sh has quit IRC | 23:24 | |
| *** Dmitrii-Sh has joined #opendev | 23:32 | |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Allow passing an arbitrary package list to assemble https://review.opendev.org/722133 | 23:35 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Add a uwsgi-base container image https://review.opendev.org/713953 | 23:35 |
| tosky | uhm, if I can't clone/fetch from https://opendev.org, is it a problem of my connection? I can fetch from the gerrit remote | 23:37 |
| tosky | oh, forget that, it worked | 23:37 |
| *** _mlavalle_1 has quit IRC | 23:45 | |
| *** Dmitrii-Sh has quit IRC | 23:46 | |
| *** Dmitrii-Sh has joined #opendev | 23:47 | |
| *** tosky has quit IRC | 23:51 | |
| ianw | oh we still have trusty tests in dib ... i guess they're time has come | 23:51 |
| ianw | their time even | 23:51 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Remove Trusty testing https://review.opendev.org/722168 | 23:56 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!