Thursday, 2020-04-23

openstackgerritIan Wienand proposed openstack/diskimage-builder master: Remove Trusty testing
mnasermordred: looks like have one last thing and we're good00:01
ianwclarkb: ^^ if you wouldn't mind a quick look for some other eyes to see if you see anything else that might fall under that ... i'd like to unbreak the gate :)00:01
mordredmnaser: haha. also - why is it still not doing the right thing in the uwsgi patch? :(00:15
mnasermordred: i think it is?  the phail is happening now because we did -f00:15
mnasermordred: teh current failure is - [91mERROR: You must give at least one requirement to install (maybe you meant "pip install /output/packages.txt"?)00:16
mordredoh it is?00:16
mnaseroh wait00:16
openstackgerritMonty Taylor proposed opendev/system-config master: Allow passing an arbitrary package list to assemble
openstackgerritMonty Taylor proposed opendev/system-config master: Add a uwsgi-base container image
mnaser3.7 does the right thing00:16
mnaserbut 3.8 doesnt00:16
mnaser3.8 failed with [0m[91mpython: can't open file '': [Errno 2] No such file or directory00:16
mordredoh - interesting00:17
mordredmnaser: that's super weird - I don't see what's wrong :(00:19
mnasermordred: yeah same.. i am looking at requires and they all seem right?00:19
mordredyeah - they look right to me00:19
mnasermordred: PYTHON_VERSION=3.700:20
mnaserin the 3.8 job00:20
mnaserso its doing FROM blah:3.700:20
openstackgerritMonty Taylor proposed opendev/system-config master: Add a uwsgi-base container image
mordredthat's it00:20
mordredbut it's not requiring 3.700:20
mnaserbuild an image with uwsgi inside of it they said00:20
mordred*BRAIN MELTS*00:20
mnaserit'll be easy they said00:20
mnaserif anything, this should be further value for having an image dedicated for this :p00:21
mnasermordred: the only other problem is that i cant test it for lodgeit bc different tenants :(00:21
mnaserlodgeit was moved to opendev00:21
mordredmnaser: oh yeah00:22
mnaseri mean the change is up00:22
mnaserbut i dont think we can use/test it00:22
mordredmnaser: I'm getting us closer to being able to move all of the opendev repos into opendev00:22
mordredbut yeah - that's gonna be a few minutes00:22
mnasermordred: yeah it'd be nice to get consesus on the usage of uwsgi-base image, if not, then i can just create one locally based on the work we did00:23
mnaserthe parent change is still useful so it makes it easier to build, but yeah00:24
mnasermordred: Created wheel for uwsgi: filename=uWSGI-2.0.18-cp38-cp38-linux_x86_64.whl size=535417 sha256=4f4bfe0529382ef80439f7a6a4465a20e668385975199c2acc9761fc407232c000:42
openstackgerritIan Wienand proposed openstack/diskimage-builder master: yum-minimal: strip env vars in chroot calls
openstackgerritIan Wienand proposed openstack/diskimage-builder master: [wip] switch func tests to containers
openstackgerritIan Wienand proposed openstack/diskimage-builder master: Restore SUSE tests to gate
mnasermordred: thinking out loud why dont we just always install the contents of /output/wheels ?00:42
mnasercause the python-builder will drop in there whatever we installed anyways00:42
mnaserwait, hm no00:43
mnasermaybe we need to use 'uWSGI' explicity ?00:44
openstackgerritMohammed Naser proposed opendev/system-config master: Add a uwsgi-base container image
mnaserlets try that, maybe the pip wheel is hash based00:50
clarkbianw: +200:51
mnasermordred: well, 3.7 works and 3.8 doesn't after using 'uWSGI' explicitly.  i'm out of ideas.01:09
*** dpanech has quit IRC01:29
clarkbmnaser: fwiw Im not against a uwsgi image but it might be good to explain why it exists so that when someone wants a cherrpy or a gunicorn or whatever we know where that boundary is02:08
clarkband also to understand if python-* are deficient in some way02:08
clarkbI've only overpip installed uwsgi so to me it fits into that existing process02:09
clarkb*only ever02:09
mnaserclarkb: yeah I’m not sure how to go about the reasoning other than “it’s popular and works nicely” :p02:09
fungiinfra-root: we just got a provider notification they're having trouble with the host ze03 is on02:28
fungilooks like ze03 got ungracefully rebooted ~10 minutes ago02:29
*** mordred has quit IRC02:43
*** mordred has joined #opendev02:45
openstackgerritMerged openstack/diskimage-builder master: Remove Trusty testing
openstackgerritMatthew Thode proposed openstack/diskimage-builder master: use stage3 instead of stage4 for gentoo builds
*** ykarel|away is now known as ykarel05:02
*** roman_g has quit IRC05:30
*** DSpider has joined #opendev05:36
*** sgw has quit IRC05:55
*** dpawlik has joined #opendev06:01
*** ysandeep is now known as ysandeep|afk06:28
*** jaicaa has quit IRC06:29
*** jaicaa has joined #opendev06:32
*** diablo_rojo has quit IRC06:53
openstackgerritIan Wienand proposed openstack/diskimage-builder master: [wip] switch func tests to containers
openstackgerritIan Wienand proposed openstack/diskimage-builder master: Restore SUSE tests to gate
*** ysandeep|afk is now known as ysandeep07:10
*** tosky has joined #opendev07:24
*** rpittau|afk is now known as rpittau07:39
*** ralonsoh has joined #opendev07:44
*** ykarel is now known as ykarel|lunch08:03
*** ysandeep is now known as ysandeep|lunch08:44
*** sshnaidm|afk has quit IRC08:55
*** ykarel|lunch is now known as ykarel08:56
*** ysandeep|lunch is now known as ysandeep09:33
*** ykarel is now known as ykarel|afk09:47
*** sshnaidm has joined #opendev10:14
*** ykarel|afk is now known as ykarel10:21
*** rpittau is now known as rpittau|bbl10:21
*** roman_g has joined #opendev10:37
*** rpittau|bbl is now known as rpittau12:02
*** ykarel is now known as ykarel|afk12:22
corvusfungi, mordred: maybe we should land +12:40
openstackgerritJames E. Blair proposed openstack/diskimage-builder master: WIP: boot test of containerfile image
mordredfungi, frickler: could I get one of you to approve ? searching for the venv that has it installed on bridge is driving me batty12:42
mordredcorvus: I'm grabbing a bionic test vm so I can verify the build steps (you know, assuming things might be slightly different than on my laptop12:44
mordredcorvus: however, in good news I can report that we don't need to install buildx - the docker we install from upstream is new enough as long as we set an env flag12:44
corvusmordred: oh interesting12:44
fricklermordred: I was assuming you wanted to drop the /root/.config bindmount on 711057. if you don't, I can still approve it12:45
mordredcorvus: DOCKER_CLI_EXPERIMENTAL=enabled12:45
mordredfrickler: oh - crap, yes12:45
mordredfrickler: let me do that real quick, my bad12:45
openstackgerritMonty Taylor proposed opendev/system-config master: Use openstackclient from container
mordredcorvus: as long as docker >=19.0.3 that should work12:46
corvusmordred: bionic may have the problem that the binfmt entries don't have the 'F' flag; see
mordredblerg. I was worried about that12:48
mordredcorvus: so maybe I should go ahead and make a focal image12:48
corvusmordred: so we may need to either fix that, or use the 'docker image based installation'
corvusmordred: or a focal image12:48
mordredcorvus: oh - so - the linux instructiosn I was following already suggested using docker/binfmt12:49
mordredcorvus: let me try those on bionic and see how it goes12:50
corvusmordred: i think that would be fine for this use case (a throwaway build node)12:50
corvus(it's lame for a desktop use case)12:51
* mordred thinks a focal base would be nice anyway - but trying to reduce the task load12:51
openstackgerritPaul Albertella proposed zuul/zuul-jobs master: Add Bazel build and ensure roles
openstackgerritMerged openstack/project-config master: Revert "Move Ubuntu builds away from nb04"
openstackgerritSorin Sbarnea proposed zuul/zuul-jobs master: Add testing of fetch-sphinx-tarball role
mordredcorvus: autohold is now the only way to grab a nodepool node isn't it?12:54
corvusmordred: yep12:54
mordreddarn. I was going to cheat. :)12:54
openstackgerritSorin Sbarnea proposed zuul/zuul-jobs master: Improve linters execution
openstackgerritSorin Sbarnea proposed zuul/zuul-jobs master: Improve linters execution
mordredcorvus: fwiw - launch_node is giving us warnings about using things that will be removed in future paramiko releases13:01
mordred/usr/local/lib/python3.6/dist-packages/paramiko/ CryptographyDeprecationWarning: Support for unsafe construction of public numbers from encoded data will be removed in a future version. Please use EllipticCurvePublicKey.from_encoded_point13:01
mordredcorvus: oh - wait - that's paramiko throwing cryptography warnings13:02
openstackgerritTristan Cacqueray proposed zuul/zuul-jobs master: hlint: add haskell source code suggestions jobs
openstackgerritSorin Sbarnea proposed zuul/zuul-jobs master: Switch remaining tests to fedora-31
openstackgerritMohammed Naser proposed opendev/system-config master: Add a uwsgi-base container image
mnasermordred: ^ yay finally i think i figured it out, the reason 3.8 didnt work and 3.7 did is because we used the wrong base image later13:12
mnaserso it obivously wasnt in the cache13:12
mnaserso that might/should be the alst thing13:12
corvusmnaser, mordred: are we going to continue the pattern of adding images like that to system-config?13:12
corvusit seems like a weird place to me13:13
mnasercorvus: i think it might make sense for opendev/images or something like that at some point13:14
corvus(and yes, the existing python-base and python-builder images also seem weird to me, but they're at least within my knowlede domain to review; uwsgi is not)13:14
openstackgerritTristan Cacqueray proposed zuul/zuul-jobs master: hlint: add haskell source code suggestions job
mnasercorvus: given that we can document them too, tbh, i/we consume them (both python-builder and python-base) and that can be helpful for us to have some docs too13:14
mnaseror at least somewhere to have docs for usage eventually13:14
corvusdo we want to create a single library of generally useful images?  or do we want to enable, say, the openstack project to make uwsgi images that it uses?13:15
mnasercorvus: i guess that's up for discussion.  an example of uwsgi being useful for opendev is for lodgeit13:16
mnaserbut unfortunately they are in different tenants so is failing13:16
mnaserso i think in the case of uwsgi, it _might_ make sense for some of the python-based services that opendev runs13:19
corvusi think lodgeit may be the only one?13:19
corvusand the way we currently run it is "/usr/bin/python /srv/lodgeit/openstack/ runserver -h -p 5000"13:20
mordredcorvus: yah - but in the docker image we're installing uwsgi and running it through that13:21
corvusright, i'm just saying that's all completely new to me13:21
mordredwe're using apache mod_wsgi for storyboard - although as we shift that to containers it might make more sense to use uwsgi in the storyboard container and run apache separately with proxypass13:22
corvusi haven't grown the ability to review uwsgi changes.  and, tbh, i don't really understand why you chose to do that instead of just "CMD ["python /srv...."]"13:22
corvusmordred: maybe?  apparently apache with mod_wsgi can be super scalable....13:22
mordredcorvus: yeah - maybe we just want to make an apache image there13:23
corvusin my mind, openstack/uwsgi-image is better than opendev/images is better than opendev/system-config.  but i can see the arguments for opendev providing useful images for all of its tenants to collaborate on.  but if that's the goal, maybe we should be intentional about it13:23
mordredcorvus: yeah- that's honestly my motivation for working with mnaser on this one13:24
*** ykarel|afk is now known as ykarel13:24
mordredit's close enough to the python-base image that it feels like sharing lifecycle events would be advantageous13:24
mordredbut I can totally understand the reluctance to expand scope too much13:24
corvusi'm really reluctant to expand system-config13:24
mordredperhaps python-base, python-builder and friends should go in to a separate repo?13:24
mordredthat's just for making useful base images?13:25
corvusyeah, it sounds like you, mnaser and i may all be able to agree on that one13:25
openstackgerritTristan Cacqueray proposed zuul/zuul-jobs master: hlint: add haskell source code suggestions job
corvusmordred, mnaser: i could also see adding test jobs to that repo that exercised the images more (eg, ran a uwsgi app)13:27
mnasercorvus: i mean it "works for me locally" right now in our testing but its hard to do speculative jobs because of different tenants13:27
corvusmnaser: yep, which is why if we're going to start hosting images like that, we should probably have good tests13:28
corvusin the same tenant :)13:29
mnasercorvus: do you feel like we can at least land the parent change in the meantime that allows us to arbitrarily install things13:29
mnaserthat's tested in the follow up change13:29
mordredyeah - that way you can at least make a local uwsgi-base image easily while we figure out the larger story here13:30
mnasersigh great13:30
mnasergithub returning 500s for the uwsgi-base test13:30
corvusno objections13:30
mnaserat least the image builds now..13:31
mnaserbut the puppet tests failed because github returning 500s (unrelated to change)13:31
mordredcorvus: while booting the bionic test node, I have used nb04's nodepool container to build a focal-minimal image to upload to rax so that we can boot focal opendev servers13:32
mnaserwhich means can land13:32
corvusmordred: neat :)13:32
mordredmnaser: I'm getting closer to having all of those puppet tests not need to run on _every_ patch13:32
mnasermordred: all good :) i think there's been huge progress the past few weeks13:33
mordred++ - I'm excited about it - even though my brain is bleeding13:33
mnaserthe puppet change double failed13:33
mnaser500s talking to github13:33
mnaseroh nevermind, it failed in post because it tried to collect a directory that wasnt crated yet13:34
mnaser"Copy puppet-apply test output to log server" could probably use ignore_errors: yes -- but yea13:34
mordredcorvus: WOW - neat thing ... I'm building the nodepool image as a test with buildx13:38
*** ysandeep is now known as ysandeep|away13:38
mordredcorvus: we obviously have not pushed python-base or python-builder arm images13:38
mordredcorvus: but - since neither of them actually install binary depends - they work as overlays on the multi-arch base images13:38
mordredcorvus: so we do not actually have to make multi-arch images of python-base and python-builder13:39
corvusmordred: oh, you can just FROM them in an arm build and it works?13:39
mordredbecause I guess the reference python which is a multi-arch manfiest, so when it goes to fetch those layers, it fetches the platform specific ones13:40
corvusmordred: so if they *did* install something arch dependent, i assume the user would just find out when they ran something in the image?13:40
corvusgood ux13:40
mordredcorvus: real4m11.129s13:42
mordredcorvus: that was from a buildx build of nodepool13:42
corvusmordred: native arch? or arm?13:42
mordredcorvus: so - it's possible that my laptop actually ISN'T a good indication here13:42
corvuswow, that's way faster than expected13:42
mordredcorvus: yeah - that makes me want to think about just always building multi-arch in our jobs13:44
*** sgw has joined #opendev13:54
mordredcorvus: ok - I think I have figured out the sequence of magical incantations13:54
corvusmordred: docker docker docker?13:55
mordredcorvus: you left out a docker13:56
mordredcorvus: sigh. not quite.13:58
mordredcorvus: ok - so - the thing we can't do is export into the local docker image list14:00
*** sshnaidm has quit IRC14:01
mordredcorvus: after building, we can export to an oci tarball - or we can have the buildx command push to a registry - oh - let me try one other thing14:02
*** sshnaidm has joined #opendev14:02
corvusthat's about the most bonkers thing i've heard14:02
mordredcorvus: I mean - we can export to an oci image tarball and I'm pretty sure then use skopeo to do our push needs14:03
corvusmordred: why driver=docker-container instead of driver=docker ?14:03
mordredwell - that was what I thought I needed to do to get --load to work14:03
mordredbut it didn't work14:03
mordredso I think that's a red herring14:04
mordredit doesn't seem to make much difference14:04
corvusmordred: so maybe driver=docker (and try that with and without load?)14:05
mordredcorvus: yeah - driver=docker implies load by default14:06
fungimordred: are we ready to approve 720202 now?14:06
mordredcorvus: and it doesn't work - actually documented in the readme that it doesn't work for multi-arch images yet14:06
mordredcorvus: basically - there isn't a path, yet, to make a multi-arch image and have it show up in docker images yet14:06
corvuseverything about docker is an afterthought14:07
mordred(from buildx)14:07
mordredpushing to a registry works14:07
mordred(and then you can get it loaded in to local docker with a pull)14:07
mordredand exporting to an oci tarball works - which we can then manipulate with skopeo14:07
mordredI'm inclined to do exporting to an oci tarball14:08
mordredand then skopeo - so we've got maximum control14:08
corvusmordred: ok, well the build-docker-image pushes to the buildset registry14:08
corvusmordred: i'd do that instead of skopeo14:08
corvusmordred: currently there is no skopeo usage in build-docker-image, so you'd be adding that14:08
mordredgood point14:08
corvusmordred: instead, i'd do the push then pull approach14:08
corvussince we need to push anyway14:09
mordredwhat about the multi-tagging14:09
mordredwe currently run docker tag in a loop before the push14:09
corvusoh right14:09
corvusmordred: well, how does docker recommend you do that?14:09
mordredjeez. maybe we push, then pull, then tag in loop, then push again?14:09
mordredthey don't recommend how I do that at all :)14:09
corvusi mean, how do you push multi-arch?14:10
mnaseri think you have to generate a local manifest14:10
mnaserand do a special type of push for multiarch14:10
mordredcorvus: oh - pushing a multi-arch is just buiold --push14:10
mordredmnaser: no - we're using buildx at the moment which is handling all of that for us14:10
corvusmordred: does it take multiple tag arguments?14:11
mordredit does not14:11
corvusmordred: so maybe 'build -t {{tag}} --push' in a loop over tags?14:12
corvusthe second builds should be noop14:12
mnaserthere is docker tag14:12
mnaserdocker tag foo/bar foo/bar214:12
corvusmnaser: the image isn't in the local image list14:12
corvusmnaser: yes, that's how the existing system works :)14:12
mnaseroh i see, sorry, i'm not familiar with the whole buildx infra14:13
corvusneither am i14:13
mordredmnaser: yeah - it's a whole set of new madness14:13
mordredcorvus: that's worth a try14:13
fungi#status log restarted all mailman sites on following oom events around 12:35-12:45 utc14:13
openstackstatusfungi: finished logging14:13
mnasercorvus: would it be ok to land given at least demonstrates it works?14:13
mnasersorry to nag but its useful for us and its been hard to use that without it landing due to container speculative jobs not being possible cross-tenant14:14
corvusmnaser: wfm14:16
fungimordred: can i go ahead and approve the gitea 1.11.4/go-git v5 upgrade (720202) now, or should i wait?14:17
mordredfungi: I think it's a fine idea to approve14:17
mnasermoving forward, i'll try to find time to create opendev/images and move things there14:18
mnaserit seems like a better home (eventually)14:19
mnaserbut a lot of moving parts right now so maybe not the best time to decouple it right now14:19
mordredmnaser: I can help drive that once my head isn't in this - and yeah, I agree re timing - but also that it's potentially a good idea14:20
corvusmnaser: ++ thanks!14:21
mnaseranother reason is that it could use some more examples and basic documentation, as me and mordred found out yesterday14:22
mordredcorvus: jina question for you ...14:28
mordredif I have a list of dicts, and I want to find out if any of the dicts have a non-empty list in one of the value14:29
mordredcorvus: is there a good jinja filter way to do that?14:29
mordredbasically "has_arch=False ; for arch in arches: if arch.get('arch', []): has_arch=true"14:30
corvusmordred: i want to say we used something like that in the siblings role14:30
mordredcorvus: my thinking is to trigger the buildx logic on someone specifying that they want multi-arch in any of the images14:31
mordredcorvus: we made a python module in siblings :)14:31
corvusmordred: no i'm thinking of "{{ zuul.projects.values() | selectattr('required') | list }}"14:32
openstackgerritColleen Murphy proposed openstack/project-config master: Add core group for python-keystoneclient
corvusmordred: arches | selectattr('arch', 'iterable')14:33
corvusmordred: or maybe for this case: arches | selectattr('arch', 'defined')14:35
mordredcorvus: ah - yeah - and then arches should be empty if none of them are defined yeah?14:35
* mordred tests real quick14:35
openstackgerritMerged opendev/system-config master: Use openstackclient from container
corvusmordred: well, the result will be empty,  but yeah.14:36
mordredcorvus: woot - works. thanks!14:40
corvus\o/ np14:40
openstackgerritDmitry Tantsur proposed openstack/project-config master: Cache cirros UEC images
openstackgerritMonty Taylor proposed zuul/zuul-jobs master: WIP Support multi-arch image builds with docker buildx
openstackgerritMerged zuul/zuul-jobs master: Switch remaining tests to fedora-31
mordredcorvus: ^^ that's kind of what I'm thinking for overall shape, look ok?14:47
mordredcorvus: thing I havne't tried yet though is pushing to something that isn't dockerhub14:48
corvusmordred: generally lgtm, yeah14:49
mordredcorvus: oh -
mordredcorvus: looks like we can pass a json file and specify a list of tags14:51
corvusthat looks easy to construct14:53
openstackgerritMerged zuul/zuul-jobs master: Improve linters execution
openstackgerritMerged zuul/zuul-jobs master: Add testing of fetch-sphinx-tarball role
*** ykarel is now known as ykarel|away15:02
openstackgerritAndreas Jaeger proposed zuul/zuul-jobs master: Update ensure-javascript-packages README
*** sshnaidm has quit IRC15:22
clarkbfungi: fwiw I'm here now to help watch gitea things as well looks like it hasn't merged yet?15:29
clarkbcorvus: mordred mnaser we're removing git from assemble, but isn't that needed for pbr? We don't git clone the repo into the container we just bind mount it in iirc so we don't necessarily have git in that context15:32
fungiclarkb: yeah, it's still running the gauntlet for now15:33
clarkboh I get it now PACKAGES can't be local pacakges15:35
clarkbbecause we do isntall git if doing local packages15:35
clarkbmnaser: for the uwsgi image how do you install keystone (or whatever) into that?15:36
openstackgerritMerged opendev/system-config master: Upgrade to gitea 1.11.4
openstackgerritMerged openstack/project-config master: Cache cirros UEC images
clarkbmnaser: you use python-builder to stage keystone and its deps, then use uwsgi-base instead of python-base to produce the production image?15:38
mnaserclarkb: i have an example for you :) one sec!15:38
mnaserthis is how i did it for keystone and it worked nicely locally15:39
mnaser(it builds but what i meant by worked nicely locally as in docker run from that launched keystone)15:39
openstackgerritSorin Sbarnea proposed zuul/zuul-jobs master: Add testing of fetch-sphinx-tarball role
openstackgerritSorin Sbarnea proposed zuul/zuul-jobs master: Add testing of fetch-sphinx-tarball role
openstackgerritSorin Sbarnea proposed zuul/zuul-jobs master: tox: allow tox to be upgraded
*** mlavalle has joined #opendev15:49
*** sshnaidm has joined #opendev15:55
openstackgerritDouglas Mendizábal proposed openstack/project-config master: Add ansible role for managing Luna SA HSM
*** diablo_rojo has joined #opendev15:59
openstackgerritClark Boylan proposed opendev/system-config master: Organize zuul jobs in zuul.d/ dir
clarkbmordred: ^ I don't know if you were already starting that but I think maybe we should try and land something like that soon16:00
clarkbinfra-root ^ any feedback on how the jobs have been organized there is much appreciated. I tried to make it obvious where to look for a job if you see it failing or not running16:01
clarkbI based it on the uwsgi stack to avoid conflicting with that16:02
clarkbthat change conflicts with like a billion things fwiw16:06
clarkbso we should land it when we are all happy to rebase16:06
openstackgerritMerged opendev/system-config master: Allow passing an arbitrary package list to assemble
openstackgerritMerged opendev/system-config master: Add a uwsgi-base container image
openstackgerritSorin Sbarnea proposed zuul/zuul-jobs master: bindep: Add missing virtualenv and fixed repo install
clarkblogstash doesn't show a bunch of apt issues in the last 6 hours16:17
mordredclarkb: I hadn't started it - I was gonna wait until we'd landed the zuul and nodepool ansible changes16:17
mordredclarkb: but I agree - I think it's super important16:18
mordredclarkb: oh - one thing I was thinking was to take a page from zuul-tests.d16:21
mordredclarkb: and put the project pipeline defintions in each file too16:22
mordredclarkb: so split this: as well16:22
clarkbmordred: maybe that should be a followon? its easier to reason about this as its largely just moving content without changing it16:22
mordredjust saying - I think that should be the ultimate shape - so that it's really clear "these are the gitea docker jobs and when they run"16:23
clarkbI wonder if we want to combine docker image, system-config-run, and infra-prd for ^16:23
clarkband to them by service rather than lifecycle stage16:24
clarkbI think that could be a good improvement too16:24
openstackgerritJames E. Blair proposed openstack/diskimage-builder master: WIP: boot test of containerfile image
clarkbbut probably a followon to avoid too much shuffling in one go16:24
mordredclarkb: hrm. maybe?16:24
mordredthat's not a bad idea - although I wonder if it'll be hard to think about the dependency trees for infra-prod if we do that16:24
corvusyeah, i think it's worth looking at, but i don't think it's a slam dunk16:25
clarkbmordred: thats a good point re infra-prod16:26
clarkboh i that change going to run all the jobs too because I moved them around :)16:27
clarkbI didn't think about that before I pushed it16:27
mordredmaybe abandon it for now then - we can circle back to it once we've got fewer s-c patches in flight?16:28
clarkbmordred: well its all those s-c changes that drove me to write it :/16:28
clarkbit is getting more and more difficult to review things. But if we want to avoid rebasing the stuff in flight thats fine16:29
clarkbis there a rough point in git time where we'd be happy to start rebasing?16:29
clarkbyou mentioned zuul/nodepool. What about the puppet splitting with eavesdrop and friends?16:32
clarkblooks like its jus about done running jobs anyway so I'll WIP it instaed of abandoning16:34
mordredah - cool16:38
mordredclarkb: yeah - I think zuul and nodepool and the current eavesdrop/codesearch patches16:38
mordredthat'll get us up to date on all of the big changes - and I'll hold off on writing more puppet split changes until we've done a reorg16:38
*** njohnston has joined #opendev16:39
*** rpittau is now known as rpittau|afk16:49
AJaegermordred: what about adding new jobs already in the "correct" files when you write them? Or moving .zuul.yaml to .zuul.d/project.yaml and then start adding new files? and in the end splitting up? That would decouple your changes...16:53
AJaegerbut you can merge first what we have as well;)16:54
clarkbfwiw school time is starting in a minute but then rereviewing mordreds s-c changes is high on the list16:55
clarkbalso I think ianw had nodepool changes that will help make progress on the zuul/nodepool side16:55
clarkbcorvus: mordred ^ I haven't looked at those yet but also on my list and you may know more about them16:56
clarkblooks like gitea upgrade happened uneventfully17:04
openstackgerritClark Boylan proposed opendev/system-config master: Stop using mysqlclient ssl flag
clarkbmordred: fungi ^ noticed that looking at more cronspam17:11
clarkbI haven't tested it yet17:11
fungiyeah, so far things looks sane for gitea servers17:12
*** sshnaidm is now known as sshnaidm|afk17:50
openstackgerritSorin Sbarnea proposed openstack/project-config master: WIP: Add tripleo-ci-shared group
zbrclarkb: i need bit of assistance creating a new gerrit group18:05
zbri was not able to find any config for effectively adding the group18:06
fungizbr: groups are auto-created when referred to in an acl18:06
zbrbut who becomes the first owner?18:07
*** ralonsoh has quit IRC18:07
clarkbwe have to seed the first member with an appropriate individual18:07
fungia gerrit admin adds someone on request, usually either the ptl/infra liaison of the project, or the person who authored the change which added the acl18:08
fungizbr: basically this
zbrthanks. i am updating the change now to follow all guidelines.18:12
yoctozeptoif I wanted to check why centos-8 infra image is so bulky, where would I want to look?18:18
clarkbyoctozepto: the git repos are the bulk of it18:18
clarkbyoctozepto: but you can download the image from or and mount it locally and poke around18:19
clarkbin the past we also cached some fairly large vm images within the VM image but now we are down to cirros and etcd tarballs so that has a much smaller impact these days18:20
openstackgerritSorin Sbarnea proposed openstack/project-config master: Add tripleo-ci-shared-core group
yoctozeptoclarkb: thanks, I am worried kolla-ansible is tested against very specific environment, we see system-wide pip-installed packages there18:23
clarkbyoctozepto: yes, there is currently work in progress to stop doing that18:24
clarkbwe are currently in the build test plain images, run jobs on them and make the jobs work18:24
yoctozeptooh, cool18:24
clarkbsystem wide pip packages don't make the images large18:25
yoctozeptobut only centos 8  seems affected18:25
yoctozeptois that valid?18:25
clarkbyoctozepto: no all of them are that way except for -plain as per the spec18:25
clarkbyoctozepto: however centos doesn't use /usr/local like debian18:25
yoctozeptohmm, I guess something made centos 8 have even more pip packages18:25
clarkbso its often more noticeable18:25
yoctozeptooh, could be that!18:25
clarkbbut again this is barely a blip in the image size18:25
clarkbwe are talking a few MB vs many GB of other things18:26
openstackgerritSorin Sbarnea proposed openstack/project-config master: Add tripleo-ci-shared-core group
clarkbthe reasons to get away from that aren't image size but conflicts with how people expect to interact with the images18:27
yoctozeptoand can I already use some -plain?18:27
clarkbyoctozepto: at your own risk18:27
yoctozeptoclarkb: agreed and very much what we just hit18:27
clarkbI don't think we'll treat it as an emergency if those break right now18:27
clarkbsince we know we are in the test and figure it out stage18:27
yoctozeptoack, sensible18:27
clarkband they are likely to go away at some point too18:27
clarkbbecause we'll replace the existing image labels with them18:28
yoctozeptowill not that make hell gates open? ;p18:28
clarkbthe goal is not to18:28
clarkbwe are updating the jobs to make the old and new cases work18:28
clarkbthat is why we built -plain images18:29
yoctozeptook, I think it starts to stick with me18:29
clarkbso we can run the same jobs on both bionic and bionic-plain, validate that they both function as expected without user intervention18:29
yoctozeptothanks, clarkb :-) any eta on it?18:29
clarkbthen once we're satisfied we've done enough test coverage of that on our side we'll probably suggest others do testing if they are concerned and set a flag day to switch18:29
fungibut as clarkb points out, the majority of the image content is whatever we're caching in the images, not really the installed software18:29
yoctozeptoyeah, I think I saw plain being tested by Ian18:29
clarkbno I don't think we can put an eta on it becuse there are still a lot of unknowns about how jobs will interact18:30
yoctozeptofair point18:30
clarkbthe infra specific stuff all seems fine though. Things like glean18:30
clarkbit looks promising18:30
yoctozeptoclarkb, fungi: sorry for confusion, I meant bulkiness in terms of those extra system-wide pip packages, not raw size18:31
yoctozeptoshould have made that one sentence, oh well18:31
fungicentos-8-0000069746.qcow2 is 8.5G, ubuntu-bionic-0000104857.qcow2 is 9.2G, debian-buster-0000134995.qcow2 is 8.9G18:31
fungiso yeah, for actual images sizes, centos is smaller18:31
yoctozeptoyeah, I think clarkb correctly pointed out the /usr/local case could be the culprit that we see effects of18:31
fungiwell, there's one other problem in centos/fedora land18:32
yoctozeptobecause ubuntu seems quite pristine from our pov there18:32
fungiand that's they've written a bunch of their required system tools in python, unlike debian/ubuntu18:32
fungiso there are going to be a slew of preinstalled python libraries required by centos/fedora just because they're expected to be part of the base operating system18:33
yoctozeptosure, that's no problem, we don't rely on that fact nor depend on it18:34
yoctozeptoit was just surprising to find out pydocker available system-wide ;p18:34
fungiyeah, but it does mean that there will be a bunch of extra python libraries showing up installed18:34
clarkbyoctozepto: I'm not sure what would pull pydocker in. Maybe tox?18:34
yoctozeptotrue, but it's under control and resembles real life scenarios18:34
fungipydocker is probably part of centos18:34
clarkbwe install pip, tox, glean globally off the top of my head18:35
yoctozeptono idea, let me dig a bit more18:35
clarkbthen a smaller number of things in venvs18:35
clarkb*small number of things18:35
yoctozeptoI doubt c8 would install it considering it does not natively support docker in the first place hmm18:35
fungipodman may rely on pydocker, and centos/fedora replace docker with podman but call it docker, right?18:35
yoctozeptothat would be odd18:35
yoctozeptobut who knows18:35
yoctozeptochecking that18:36
clarkb doesn't show anything for pydocker or python-docker18:36
fungiyeah, so it must be getting installed when the job runs18:36
fungiif you've got an example build result, we can look in the logs to see when it gets installed and why18:37
yoctozeptothanks, that would be cool18:37
yoctozeptohere is the relevant change18:37
yoctozeptocentos 8 should fail as others did18:37
yoctozeptothe job is broken by design18:38
yoctozeptoI checked rpm packages but nothing relevant installed18:38
yoctozeptosimilarly jmespath should not be present18:38
clarkbyoctozepto: fungi I think its gonna be ansible installing it18:40
clarkbthat job uses a nested ansible and I bet money ansible pulls it in because modules want to talk to docker containers via python in ansible18:40
clarkbone beers worth :)18:40
yoctozeptoclarkb: it never does, we have to install it18:42
yoctozeptofor this reason ubuntu and debian fail now18:42
clarkbright wouldn't that result in it being present though?18:43
clarkbfwiw it seems that the job I'm looking at is failing because python docker is NOT installed18:43
yoctozeptoyou are probably looking at ubuntu/debian then18:44
clarkb that says centos-818:44
fungiyeah, i think what he's saying is you need to look at the logs for and see why/how pydocker is getting installed that makes that build actually work18:44
yoctozeptoah, multinode ones18:44
yoctozeptoyeah, I'm talkin singlenode18:44
clarkbyoctozepto: can you link to the exact job then?18:45
clarkbthat job succeeded18:45
yoctozeptoit did but should not18:45
yoctozeptothat's the trick18:46
clarkbok so the ubuntu behavior of Failing is what we want18:46
yoctozeptobut your observation is correct that multinode somehow do not have py docker on secondary nodes18:46
yoctozeptoI missed that18:46
clarkbyoctozepto: that implies it isn't an image issue18:47
yoctozeptoyeah, that one is not18:47
yoctozeptobut jmespath looks like it18:47
fungii'm not having much luck finding a log which records where package installation is occurring18:47
*** sgw has quit IRC18:48
yoctozeptothis is where ubu/deb fail because of lack of jmespath, yet c8 found it18:48
yoctozeptobut it looks indeed like the /usr/local thingy18:48
clarkbfungi: ya me either. There doesn't seem to be a clear linear path. The job-output log tells me fails but I can't find a log18:49
yoctozeptoso the jmespath mystery looks resolved18:49
yoctozeptoand py docker is on me to trace18:49
yoctozeptoah, I can help with that18:49
fungithe dnf and yum logs collected don't seem to provide much18:50
yoctozeptoor just browse ara
clarkbyoctozepto: except that this is likely preara problem?18:50
clarkbyoctozepto: my theory is its being pulled in by the ansible install18:50
clarkband ara can't log that18:50
yoctozeptopreara is job-output in main dir18:51
yoctozeptothen everything is ara :-)18:51
clarkbyoctozepto: but that doesn't log what is doing?18:51
yoctozeptocorrect, it is split into the log I mentioned18:51
clarkbbut ara isn't a shell script18:52
yoctozeptodo note ubu/deb fail at prechecks so deploy never even happens18:52
yoctozeptoyeah, it is not18:53
yoctozeptobut is just calling ansible18:53
clarkbyoctozepto: where does log?18:54
yoctozeptowhere deploy but in files named reconfigure*18:55
clarkbI guess thats after the expected failure point18:55
yoctozeptoit's the last step though18:55
yoctozeptothe earliest is bootstrap18:55
yoctozeptobut it treats all hosts equally so it should patch up other nodes too18:55
yoctozeptolet me verify one thing18:56
yoctozeptook, verified, it got pulled via requirements.txt18:56
yoctozeptoso docker is resolved as well18:56
openstackgerritRoman Gorshunov proposed openstack/project-config master: Retire airship-in-a-bottle
yoctozeptoand jmespath is due to not /usr/local18:56
yoctozeptoand python see all extra packages18:57
yoctozeptomakes sense18:57
yoctozeptoall my worries have gone away18:57
clarkbyoctozepto: can you point to that log since I'm curious to see where that ends up?18:57
clarkbI've basically been trying to find where ansible is installed18:57
clarkband working my way backwards from failing18:57
clarkbah there it is18:58
clarkbinstall kolla-ansible and dependencies18:58
clarkband that is where docker is pulled in18:58
yoctozeptolook for install kolla...18:59
yoctozeptoit installs the deps18:59
yoctozeptonow then why ubuntu no longer see them but centos8 still sees them18:59
yoctozeptooh my brain hurts18:59
clarkbyoctozepto: python path probably18:59
yoctozeptoyeah, that would align with jmespath19:00
openstackgerritSorin Sbarnea proposed openstack/project-config master: Add tripleo-ci-shared-core group
yoctozeptothe breaking patch just creates virtualenv with system-wide packages in19:00
yoctozeptoso based on default paths it could break it seems19:00
yoctozeptothat's a very valuable lesson19:01
yoctozeptook, I updated that change with summary comment so I have something to start off; the erratic behaviour actually reminds me of users having trouble with package resolution on ubuntu19:06
yoctozeptothanks again clarkb and fungi, best service around here19:06
clarkbmordred: I'm digging into the puppet stack (I've got more of that paged in than zuul and nodepool right now) and should be ready to go (I haven't reviewed it yet since others have)19:10
fungiyoctozepto: to be fair, it's the *only* service around here ;)19:10
clarkbmordred: has comments from corvus and I'm reviewing that one myself now19:11
openstackgerritMonty Taylor proposed zuul/zuul-jobs master: Support multi-arch image builds with docker buildx
mordredcorvus, clarkb: ^^ I think that's teh whole story with multi-arch19:20
yoctozeptofungi: shh! let me express my praise for your work and hints with no undermining!19:21
mordredclarkb: cool - let me fix those real quick19:21
fungiyoctozepto: in that case, thanks!19:22
clarkbmordred: if you give me about 5 minutes I think I'll be done reviewing it myself19:22
openstackgerritMonty Taylor proposed opendev/system-config master: Split eavesdrop into its own playbook
corvusmordred, clarkb, ianw: nb04 has built images, and i can docker exec into it, so that's looking good19:24
mordredcorvus: oh - poop. well, there it is - but I'll check your reviews as soon as you're done19:24
mordredcorvus: I also used it to build a focal image earlier19:24
corvusmordred: yep, just adding additional confirming data :)19:25
mordredcorvus: /opt/nodepool_dib/ubuntu-focal.* on nb0419:25
mordrednext step is uploading to rax-dfw19:25
mordredand then we should be able to boot production focal-based images19:26
clarkbmordred: ok comments left19:31
clarkbI think the install_only thing may be the only thing that needs changing?19:31
openstackgerritOleksandr Kozachenko proposed opendev/system-config master: Remove packages.txt after assemble
clarkbmordred: and I was hoping for a little more clarfication on the site.pp and eavesdrop.pp split19:32
clarkbI don't think its necessary but also doesn't hurt anything19:32
openstackgerritMonty Taylor proposed opendev/system-config master: Split eavesdrop into its own playbook
mordredclarkb: fixed and responded19:34
mordredclarkb: mainly making smaller .pp files so that we can do better job triggering19:35
clarkbgot it its for the zuul file matcher more than anything else19:36
mordredcause you're right - we could totally just do the playbook19:37
mordredclarkb: for that matter, we could remove the host matchers from the smaller .pp files19:37
mordredbut I figured keeping it that way would be an easier review19:38
openstackgerritMonty Taylor proposed opendev/system-config master: Stop logging puppet to syslog
openstackgerritMonty Taylor proposed opendev/system-config master: Move in-tree hiera settings to ansible vars
clarkbmordred: ya19:38
openstackgerritMonty Taylor proposed opendev/system-config master: Add new etherpad to cacti
mordredclarkb: I rebased the stack - it's mostly reviewed19:39
clarkbmordred: on the parent codesearch change you assert that openstack_project::server does nothing. It seems to configure the apt puppet module as well as install openafs client19:39
clarkbmordred: for codesearch those two things are likely not necessary but we may have to be more careful about that removal on other servers19:39
mordredclarkb: we install openafs-client with ansible now19:40
clarkbmordred: but only on ansible hosts19:40
mordredyeah - but we're doing playbooks19:40
clarkbso which is puppet still wouldn't19:40
mordredso - my thinking was to remove server and add openafs-client as needed19:40
mordredas we add the playbook19:41
clarkb(though all new mirrors are being built as mirror.* with ansible)19:41
clarkbso we could still have puppet manage old mirrors but run playbook for afs pre puppet19:41
mordredthe nice thing here is that we can actually piecewise move chunks of stuff between teh two since we have service-specific playbooks19:41
mordred(like moving accessbot to docker but leaving the others for now in eavesdrop)19:42
yoctozeptoinfra-core: one last thing before /me going to bed - could I get 2nd +2 and approval on ?19:43
clarkbmordred: minor nit on that can be a followup (or probably no one will notice if we don't fix it either)19:43
yoctozeptoclarkb: thanks :-)19:47
mordredclarkb: :)19:49
openstackgerritMonty Taylor proposed zuul/zuul-jobs master: Support multi-arch image builds with docker buildx
clarkbfungi: thinking out loud here, should we maybe start a dstat run capturing memory usage on a frequent basis then maybe rotate that weekly?20:03
clarkbfungi: hoping that eventually it will lead us to the OOM culprit on lists20:03
clarkb(sorry I meant to say run dstat on lists)20:03
clarkbmaybe to start just run it in a root shell?20:03
fungimaybe... i mean the oom dumps already list the processes consuming memory. the highest vmem consumers are all apache processes20:06
clarkbfungi: isn't it resident memory that matters more though20:07
clarkbfungi: also the other thing we can do is add more swap20:07
fungiand yeah, cacti doesn't give us much useful info, snmpd stops responding during these events20:08
clarkbdstat will give us a much better pciture of total system resource usage20:08
fungithough there's a bit of a lead-up in swap utilization hours before20:08
clarkbto try and help identify bottlenecks (we could be using more memory beacuse some activity is being slow on cpu or disk)20:09
fungiwhich is strange, fiven that most of the active memory is consumed by buffers and cache but a ton of data is getting paged out?20:09
clarkbfungi: I think linux will put caches into swap too under some circumstances20:09
clarkbbceause they can be retrived from swap more quickly than disk20:09
fungi vs
clarkb*regular fs disk20:10
clarkboh wow huh20:10
clarkbfungi: swappiness on that host is 60 which is default20:11
clarkbI know we turn it down in devstack-gate/devstack20:11
openstackgerritMerged opendev/system-config master: Set AllowEncodedSlashes NoDecode on 8080 revproxy
fungithe sustained burst of cpu utilization also sort of coincides with when the swapping began20:12
clarkbfungi: maybe we should turn down swappiness and add another 2GB of swap?20:12
clarkband then see if we can flatten that memory use out more?20:12
fungilooks like that's a nightly event20:12
clarkbfungi: bup maybe20:12
fungi(the cpu bursts i mean)20:12
fungiyeah, could be20:13
fungiit's almost all iowait20:13
clarkbfungi: what is odd is memory use went down after the host upgrade20:13
fungiand there's a ton of corresponding reads from xvda20:13
clarkbfungi: bup would certainly explain that. Other things it could be robot indexers of the web?20:14
clarkb(I don't think we should turn off indexing of the archives, but if that is a cause understanding it would be good)20:14
fungiso what i can gather from cacti is that there was a bunch of reading off xvda which gobbled up memory buffers (primarily) much of which wound up getting paged to swap20:14
fungibut yeah, maybe dstat to the rescue here, we probably want to start by identifying what's consuming so much memory earlier in that event20:16
fungiand what's also hammering the rootfs so hard20:16
clarkb fwiw on swappiness20:16
*** sgw has joined #opendev20:16
clarkbthe note about mysql might be relevant here20:16
clarkbperhaps apache and/or bup do similar20:17
clarkb++ to dstat, gather some more data then we acn decide if adjusting swappiness, swap size, or something else makes sense20:17
clarkbI've got to sort out lunch and a bike ride right now. But can help set that up in a root screen later20:18
fungilooking at the swap usage graph for longer timeframes though, today's spike may be an anomaly20:18
clarkbfungi: devstack's dstat service might contain helpful info for logging to disk20:18
fungithe cpu spikes are consistent daily, but not the spikes in memory utilization20:18
openstackgerritOleksandr Kozachenko proposed opendev/system-config master: Remove output after install from bindep
*** DSpider has quit IRC20:26
openstackgerritSean McGinnis proposed openstack/project-config master: Include octavia files when updating branch constraints
openstackgerritJames E. Blair proposed zuul/zuul-jobs master: Add tests for multiarch build
*** dpawlik has quit IRC21:09
fungi#status log running `dstat -tcmndrylpg --tcp --output dstat-csv.log` in a root screen session on lists.o.o to diagnose recurring oom issue21:10
openstackstatusfungi: finished logging21:10
openstackgerritMonty Taylor proposed zuul/zuul-jobs master: Support multi-arch image builds with docker buildx
openstackgerritOleksandr Kozachenko proposed zuul/zuul-jobs master: Fix the checking helm_values_file definition
ianwclarkb yoctozepto: centos-8 is slightly different in that it only has packaged pip installed -- we do not overwrite it with pip installed versions21:47
ianwhowever, those packages are pre-installed, as noted, and we are moving to stop doing that21:48
ianwspeaking of clarkb, maybe mordred : is the only non-trivial change required to get our suite of platforms building under our container functional tests21:49
ianwclarkb: perhaps you could look over too which mordred has already looked at21:49
ianwFunctional container tests: update to CentOS 8 ( can't work until 721726 is merged21:50
ianwwhich will fix the release test job there21:50
openstackgerritIan Wienand proposed openstack/project-config master: nb03 : update to arm64 to inheritance, drop pip-and-virtualenv
ianwclarkb / mordred: ^ as mentioned in #zuul21:56
mordredianw: cool. did you see the arm container building stuff?22:06
ianwmordred: yes sort of ... is it all in zuul-jobs or am i missing other bits to look at?22:07
mordredianw: all in zuul-jobs22:07
mordredianw: will use it - but the role is in a trusted job, so we have to land it first22:08
mordredianw: +2 on the p-c change22:11
ianwmordred: is the eventual idea for buildx to be the only path?22:13
ianwmordred: minor comment on 722339 ... generally lgtm; seems like something we should start and iterate on22:17
openstackgerritMonty Taylor proposed opendev/system-config master: Add new etherpad to cacti
mordredianw: well - don't know that we know that ... it's extra moving parts that we don't really need for non-multi-arch22:19
mordredianw: but ... maybe we'll be happy with multiarch builds enough that we do make it the only path22:19
openstackgerritMichael Johnson proposed openstack/diskimage-builder master: Stop installing python2 pip venv on focal
mordredianw: for now, definitely just keeping it to be conditional while we poke at it (it's very new and experimental from upstream docker)22:20
mordredianw: ^^ look! your favorite element!22:21
mordredianw: oh - that's a good question re: buildset_registry22:21
mordredI was mostly looking at the existing checking for if buildset_registry is defined in the role22:22
mordredbut I think that can be removed - as well as the assert22:22
ianwjohnsom / mordred: thanks, dropped a comment but i'd prefer we put it an error out for this element on focal.  it's what we're doing for f31 and tumbleweed in  we don't want to support it22:22
mordredianw: oh - wait - I know22:22
mordredianw: buildset_registry might not be defined because that lookup might fail22:23
johnsomAh, I was looking in the DIB channel and didn't see this chat.22:23
johnsomThat is also my least favorite element22:23
ianwmordred: oh, right i see now, thanks22:24
openstackgerritMonty Taylor proposed zuul/zuul-jobs master: Support multi-arch image builds with docker buildx
johnsomianw What is the alternate element? or can we just drop it and use package-installs?22:25
mordredjohnsom: oh it's so much more complicated than that :)22:25
johnsomlol, I would guess22:25
ianwyeah, for infra, but for your case, i would say yes just install python3-pip as a package22:26
mordredoh. yeah. sorry - I was thinking about the opendev images22:26
ianwunless you need something very specific22:26
* mordred forgets there are other dib users :)22:26
johnsomHappy to give the service project perspective. grin22:26
mordredianw: oh - also - I built a focal image but haven't uploaded it anywhere yet22:27
ianwand, if there's any problems with that version of pip, report them upstream :)22:27
ianwmordred: using dib?22:27
mordredianw: yah22:28
fungiianw: or to your distro probably22:28
mordredianw: /opt/nodepoolb_dib/ubuntu-focal.* on nb04 - plan is to upload to rax so we can boot control plane nodes on focal. I'll upload it tomorrow, didn't get to it today22:28
mordredianw: but it worked like a charm!22:28
* fungi notes you should generally refrain from reporting bugs in distro-packaged software to upstream software maintainers unless you are the package maintainer for that distro22:28
ianwmordred: interesting, it just worked?22:28
ianwfungi: sorry, yeah that's what i meant, the distro.  basically, anywhere but to me! :)22:29
mordredianw: I did this: docker-compose exec nodepool-builder bash /opt/nodepool_dib/make-focal.sh22:29
ianw(i jest, but that is the point that we don't have infra custom things you have to debug)22:29
johnsomianw I will give it a go with the drop the element approach and let you know how it goes.22:30
* fungi is working on opendev engagement metrics... notes that over the past 30 days 11241 gerrit changes have been touched/altered/updated/commented on in some way22:30
ianwmordred: well, that's interesting, thanks ...  i'll work on some gate jobs22:30
mordredianw: also - I think it means that this: should work22:31
mordredianw: although maybe I should add them back to nb0422:32
mordredoh - actually, I should ONLY add them to nb0422:32
mordredbecause we need newer builder to build focal22:32
ianwmordred: yeah it would be good to verify with gate tests.  i feel like the debootstrap fix should work.  but similarly, i'd like to start this element *without* pip-and-virtualenv22:33
ianwi.e. no need for -plain images; we should just start our testing/bringup on nodes without it22:34
openstackgerritMonty Taylor proposed openstack/project-config master: Start building focal images
mordredianw: kk. I'll update the config for it22:35
mordredianw: also - I agree re: gate tests22:35
openstackgerritMonty Taylor proposed openstack/project-config master: Start building focal images
mordredianw: thereyago22:37
ianwi just need to re-orient to paedology mode in my classroom ... i.e. dining room table.  back in a sec22:38
clarkbianw: catching up on reviews now and starting with the ones you linked22:46
openstackgerritIan Wienand proposed openstack/diskimage-builder master: [wip] switch func tests to containers
openstackgerritIan Wienand proposed openstack/diskimage-builder master: Restore SUSE tests to gate
ianwmordred / clarkb: ^^ the only thing with using the container tests is i'm not sure how to do an equivalent of the release test22:47
ianwthe nodepool release test installs nodepool from source, then dib etc from release, but i can't reuse that job in dib where i'd want nodepool from release and dib from source22:47
openstackgerritJames E. Blair proposed zuul/zuul-jobs master: Support multi-arch image builds with docker buildx
clarkbianw: minor but important thing on
ianwoops thanks22:50
ianwyou americans and your 'z's throw me off :)22:50
clarkbianw: we simplified english, ya'll should get on board (unfortuantely it didn't really help english still crazy)22:51
openstackgerritIan Wienand proposed openstack/project-config master: nb03 : update to arm64 to inheritance, drop pip-and-virtualenv
clarkbjohnsom: ianw I think octavia could probably use python3 venv module?22:57
openstackgerritIan Wienand proposed openstack/diskimage-builder master: [wip] add a focal test
clarkbwhich is where we are giogn to try and push things on the job nodes for bootstrappign tools too22:58
*** tosky has quit IRC22:58
ianwmordred: ^^ that will test glean22:58
johnsomclarkb Oh, yeah, I remember thinking about that. You can just do "python -m venv" right and you don't need the python3-virtualenv package installed? Is that correct.22:59
clarkbjohnsom: you need to do python3 -m venv on debuntu and you need the python3-venv package but ya22:59
clarkbthe nice thing about it is its stdlib so nothing from pip sould try and install over it22:59
clarkbwhich helps to avoid conflicts down the road23:00
johnsomianw FYI, the image built switching to just handling them in package-installs, so works-for-me. Just fired up a job to see if the image is actually functional.23:00
clarkbianw: I've reviewed the changes you linked to earlier (at the start of your day) is there a tldr for subsequent things to review?23:00
ianwclarkb: umm, so many things in progress! :)23:02
clarkbianw: I know thats why I get easily lost :)23:02
ianwclarkb: if you could also poke at ... i would like to get that into the next dib release23:02
ianwthat will, i believe, ensure opensuse builds in both -plain and !plain formats in the gate23:03
ianwand with that release of dib, i'll be pretty confident that our container builders can build all our image types23:04
ianw will confirm this too23:04
ianwafter that, we should have -plain nodes and i can get back to more focus on the pip-and-virtualenv removal23:06
clarkbianw: whats with ? I think the difference there is we don't filter for executable files. Were we writing scripts that needed to be chowned?23:08
ianwhrmmm ... that's come in via the revert ...23:09
ianw doesn't mention why it's there ...23:10
ianwit either does something, or was accidentally left in ... hard to tell :)23:11
clarkbianw: I guess we can add it back in if necessary? I think the only functional difference is filtering executables23:11
clarkband if we've got non executable files in /bin maybe we should fix that?23:11
ianwyeah, my preference would be to add it back with a changelog if there is an issue to be aware of23:12
clarkbok change is approved23:12
clarkbyou can -W if I did that too quick :)23:13
clarkbmordred: fwiw I didn't approve because it needs watching and there were a few things going on earlier today. Its late in the day now, but I can help watch that go in tomorrow morning if you'll be around23:17
openstackgerritIan Wienand proposed opendev/glean master: Update functional tests
openstackgerritIan Wienand proposed opendev/glean master: [wip] drop release test
clarkbthen once that is in I think we can land eavesdrop after assuming it gets another round of review from a second reviewer23:18
openstackgerritIan Wienand proposed openstack/diskimage-builder master: [wip] switch func tests to containers
openstackgerritIan Wienand proposed openstack/diskimage-builder master: Restore SUSE tests to gate
openstackgerritIan Wienand proposed openstack/diskimage-builder master: [wip] add a focal test
mordredclarkb: ++ I agree23:27
mordredclarkb: I think we can land a bunch of that tomorrow actually23:27
mordredclarkb: should be able to do nodepool launchers and zuul - I think they're both ready to go23:27
mordredclarkb: oh - I take that back - nodepool is - but I need to update zuul to not do compose up like we did for nodepool23:27
clarkbI'm updating my tumbleweed install right now, its removing a bunch of python2 as part of that. Unexpected removal due to this: bzr23:38
clarkbmordred: ^ I don't know if you bzr anything at all anymore but that could be important as python2 dies23:38
clarkbbasically now is a good time to bzr to git I guess23:38
ianw2020-04-23 23:28:47.039228 | primary |   "msg": "failed to create temporary content file: [Errno 104] Connection reset by peer"23:42
ianwit's unclear if that's trying to get to an infra server ...23:43
clarkbianw: it says "Download tripleo source image" I doubt we are hosting that but it could be23:43
clarkblets see if codesearch will say more23:44
clarkbianw: that is the default which isn't us23:44
clarkbbut could be set in CI to somethign else I guess23:44
ianwhrm i'm guessing that's likely it, sorry just thought it might be our git on the other end23:45
ianwsigh, and now
ianwCould not connect to (, connection timed out Could not connect to (, connection timed out23:47
clarkbthat ran in sjc1 so not an ipv6 only cloud problem23:47
clarkb*vexxhost sjc123:47
ianwyeah the other was openedge ... maybe i'm just unlucky23:48
ianw will be the one that tries to boot focal23:51
openstackgerritMerged openstack/diskimage-builder master: pip-and-virtualenv: drop f31 & tumbleweed, rework suse 15 install

Generated by 2.15.3 by Marius Gedminas - find it at!