| *** dchen has joined #openstack-infra | 00:04 | |
| *** armax has quit IRC | 00:09 | |
| openstackgerrit | Sagi Shnaidman proposed zuul/zuul-jobs master: DNM: debug vars https://review.opendev.org/702144 | 00:20 |
|---|---|---|
| *** threestrands has joined #openstack-infra | 00:38 | |
| *** zhurong has quit IRC | 00:43 | |
| *** jistr has quit IRC | 00:45 | |
| *** jistr has joined #openstack-infra | 00:48 | |
| openstackgerrit | Sagi Shnaidman proposed zuul/zuul-jobs master: Rename vars/ directory to prevent conflicts https://review.opendev.org/702144 | 00:59 |
| sshnaidm | EmilienM, will fix your job ^^ | 00:59 |
| *** threestrands has quit IRC | 01:43 | |
| *** xinranwang has joined #openstack-infra | 02:10 | |
| *** hwoarang has quit IRC | 02:11 | |
| *** hwoarang has joined #openstack-infra | 02:11 | |
| *** hwoarang has quit IRC | 03:05 | |
| *** hwoarang has joined #openstack-infra | 03:07 | |
| *** threestrands has joined #openstack-infra | 03:10 | |
| *** threestrands has quit IRC | 03:11 | |
| *** threestrands has joined #openstack-infra | 03:11 | |
| *** soniya29 has joined #openstack-infra | 03:27 | |
| *** ramishra has joined #openstack-infra | 03:41 | |
| *** ykarel has joined #openstack-infra | 04:52 | |
| *** evrardjp has quit IRC | 05:33 | |
| *** evrardjp has joined #openstack-infra | 05:34 | |
| *** threestrands has quit IRC | 05:38 | |
| *** surpatil has joined #openstack-infra | 05:41 | |
| *** udesale has joined #openstack-infra | 06:09 | |
| AJaeger | ianw: welcome back! I pushed some changes to gerrit-dash-creator starting at https://review.opendev.org/#/c/702088/1 . I suggest to merge and then tag a new version (I can tag). Could you put that on your review queue, please? | 06:18 |
| ianw | AJaeger: thanks :) will do | 06:18 |
| AJaeger | thanks | 06:22 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: Disable wheel mirror https://review.opendev.org/702166 | 06:24 |
| ianw | infra-root: ^ i'm about out of time for today, but i think we have a confluence of problems causing gate failures | 06:24 |
| ianw | http://lists.openstack.org/pipermail/openstack-discuss/2020-January/011960.html describes things; of immediate interest to this channel is the lack of version metadata on our wheels | 06:25 |
| ianw | I have filed https://storyboard.openstack.org/#!/story/2007084 for us to investigate this. basically the <a> links to wheels probably need a "data-requires-python="^gt;=3" in them, so pip doesn't try to install incompatible wheels | 06:26 |
| ianw | how, exactly, you determine this from a wheel, I do not know | 06:27 |
| AJaeger | ianw: ARgh ;( Thanks for digging into it. That's a big hammer but good bandaid | 06:28 |
| ianw | yeah, i'm not sure about it. it has exposed things we should fix | 06:28 |
| *** udesale is now known as udesale_ | 06:28 | |
| *** udesale_ is now known as udesale | 06:29 | |
| AJaeger | yes, we need to | 06:29 |
| ianw | the python2/3 thing is obviously the biggest break, but i imagine as the world moves on, this could also be relevant for python 3.X v 3.Y | 06:29 |
| AJaeger | ;( | 06:30 |
| ianw | this is the first i've heard of PEP503 ... there's always something new to learn in python packaging ... | 06:31 |
| AJaeger | ianw: I thought our wheels where able to run on both python2 and python3, is that not the case anymore? | 06:43 |
| ianw | AJaeger: the problem is that setuptools has dropped python2 support -- but python2 will still try to install our wheel because it seems it can't determine that detail | 06:44 |
| AJaeger | ah, got it | 06:48 |
| ianw | i'm asking in https://github.com/pypa/pip/issues/7586#issuecomment-573529379 if we can actually determine the value of this field from .whl files ... | 06:48 |
| ianw | if so, we could write a script to create index.html files with it | 06:48 |
| ianw | but i wonder if you can only see it from the source | 06:49 |
| *** lmiccini has joined #openstack-infra | 06:57 | |
| *** slaweq has joined #openstack-infra | 07:07 | |
| *** fdegir has quit IRC | 07:08 | |
| *** fdegir has joined #openstack-infra | 07:09 | |
| *** slaweq has quit IRC | 07:12 | |
| *** rcernin has quit IRC | 07:13 | |
| *** aarents has quit IRC | 07:16 | |
| *** slaweq has joined #openstack-infra | 07:16 | |
| *** zhurong has joined #openstack-infra | 07:17 | |
| *** pgaxatte has joined #openstack-infra | 07:18 | |
| *** dpawlik has joined #openstack-infra | 07:21 | |
| *** aarents has joined #openstack-infra | 07:28 | |
| *** ralonsoh has joined #openstack-infra | 07:29 | |
| *** aarents has quit IRC | 07:29 | |
| *** aarents has joined #openstack-infra | 07:34 | |
| *** ykarel is now known as ykarel|lunch | 07:36 | |
| *** xinranwang has quit IRC | 07:40 | |
| *** trident has joined #openstack-infra | 08:02 | |
| *** hashar has joined #openstack-infra | 08:09 | |
| *** jtomasek has joined #openstack-infra | 08:09 | |
| *** jtomasek has quit IRC | 08:12 | |
| *** jtomasek has joined #openstack-infra | 08:13 | |
| *** tesseract has joined #openstack-infra | 08:21 | |
| *** tosky has joined #openstack-infra | 08:22 | |
| yoctozepto | morning | 08:30 |
| *** rpittau|afk is now known as rpittau | 08:35 | |
| *** zbr is now known as zbr|rover | 08:37 | |
| *** gfidente has joined #openstack-infra | 08:45 | |
| *** jpena|off is now known as jpena | 08:47 | |
| *** ykarel|lunch is now known as ykarel | 08:47 | |
| *** dchen has quit IRC | 08:49 | |
| *** lucasagomes has joined #openstack-infra | 08:50 | |
| *** kjackal has joined #openstack-infra | 08:59 | |
| *** iurygregory has joined #openstack-infra | 09:02 | |
| *** ociuhandu has joined #openstack-infra | 09:07 | |
| *** ociuhandu has quit IRC | 09:12 | |
| *** xek has joined #openstack-infra | 09:16 | |
| *** apetrich has joined #openstack-infra | 09:18 | |
| *** ociuhandu has joined #openstack-infra | 09:20 | |
| *** electrofelix has joined #openstack-infra | 09:27 | |
| *** hashar has quit IRC | 09:36 | |
| *** derekh has joined #openstack-infra | 09:38 | |
| *** soniya29 has quit IRC | 09:38 | |
| *** ociuhandu has quit IRC | 09:39 | |
| *** ociuhandu has joined #openstack-infra | 09:40 | |
| *** ociuhandu has quit IRC | 09:44 | |
| openstackgerrit | Paul Albertella proposed zuul/zuul-jobs master: Add Bazel build and install roles https://review.opendev.org/693513 | 09:45 |
| *** pcaruana has joined #openstack-infra | 09:49 | |
| *** ociuhandu has joined #openstack-infra | 09:56 | |
| *** pcaruana has quit IRC | 09:56 | |
| *** ociuhandu has quit IRC | 09:58 | |
| *** ociuhandu has joined #openstack-infra | 09:59 | |
| *** ociuhandu has quit IRC | 10:03 | |
| *** ociuhandu has joined #openstack-infra | 10:05 | |
| *** ociuhandu has quit IRC | 10:07 | |
| *** iurygregory has quit IRC | 10:18 | |
| *** surpatil is now known as surpatil|mtg | 10:19 | |
| *** iurygregory has joined #openstack-infra | 10:28 | |
| *** ociuhandu has joined #openstack-infra | 10:32 | |
| *** dtantsur|afk is now known as dtantsur | 10:32 | |
| *** ociuhandu has quit IRC | 10:34 | |
| dtantsur | morning folks! maybe the setuptools situation warrants a global notice to prevent people from aggressively rechecking? | 11:01 |
| *** rpittau is now known as rpittau|bbl | 11:03 | |
| *** ociuhandu has joined #openstack-infra | 11:06 | |
| *** rcernin has joined #openstack-infra | 11:11 | |
| openstackgerrit | Paul Albertella proposed zuul/zuul-jobs master: Add Bazel build and install roles https://review.opendev.org/693513 | 11:19 |
| *** gfidente has quit IRC | 11:27 | |
| *** gfidente has joined #openstack-infra | 11:28 | |
| *** rfolco has joined #openstack-infra | 11:55 | |
| openstackgerrit | Paul Albertella proposed zuul/zuul-jobs master: Add Bazel build and install roles https://review.opendev.org/693513 | 11:55 |
| *** diablo_rojo has joined #openstack-infra | 12:09 | |
| *** diablo_rojo_phon has joined #openstack-infra | 12:14 | |
| *** zzzeek has quit IRC | 12:16 | |
| *** zzzeek has joined #openstack-infra | 12:17 | |
| yoctozepto | dtantsur: ML notification was done yesterday for the very reason :-) | 12:20 |
| yoctozepto | but yeah, maybe infra-root could send a global disaster notification on irc | 12:20 |
| *** jpena is now known as jpena|lunch | 12:23 | |
| *** ociuhandu has quit IRC | 12:23 | |
| *** udesale has quit IRC | 12:23 | |
| *** ociuhandu has joined #openstack-infra | 12:31 | |
| *** lmiccini has quit IRC | 12:32 | |
| *** michal_kowieski4 has joined #openstack-infra | 12:39 | |
| *** lmiccini has joined #openstack-infra | 12:44 | |
| sshnaidm | cores, please merge: https://review.opendev.org/#/c/702144/ | 12:45 |
| sshnaidm | cloudnull, ^^ | 12:45 |
| *** adriant has quit IRC | 12:48 | |
| *** rh-jelabarre has joined #openstack-infra | 12:48 | |
| *** adriant has joined #openstack-infra | 12:49 | |
| *** ociuhandu has quit IRC | 12:52 | |
| *** tommylikehu has joined #openstack-infra | 12:54 | |
| *** ykarel is now known as ykarel|afk | 13:00 | |
| *** rlandy has joined #openstack-infra | 13:00 | |
| tommylikehu | hey guys, does openstack has mail server except for mail list? I mean an official mail system that can be used to send/receive mail directly :) | 13:02 |
| *** Goneri has quit IRC | 13:04 | |
| *** ykarel|afk has quit IRC | 13:05 | |
| fungi | tommylikehu: to what purpose? there is a mail service hosted for openstack.org which is used for addresses of the osf staff, but the opendev/openstack infra sysadmins don't run that | 13:08 |
| fungi | it's also used for some role addresses, like for domain contact information and addresses the donor service providers associate with our keystone accounts | 13:10 |
| tommylikehu | thanks for you reply @fungi, so you are saying there are two systems one is mail list running by infra team and another mail server running by others? | 13:13 |
| fungi | yes | 13:14 |
| tommylikehu | we are wondering in which case we need a official mail server when already has deployed the mail list system. I am from openeuler community :) | 13:14 |
| smcginnis | tommylikehu: o/ | 13:15 |
| fungi | what are you needing to accomplish with e-mail? | 13:15 |
| tommylikehu | @smcginnis hey long time no see | 13:15 |
| tommylikehu | @fungi setting up a official mail server for reporing security issues | 13:16 |
| *** ociuhandu has joined #openstack-infra | 13:16 | |
| tommylikehu | i found openstack use mailist to address this | 13:16 |
| *** ociuhandu has quit IRC | 13:16 | |
| smcginnis | fungi: From what I understand, openeuler is a new community and tommylikehu is looking to replicate the infrastructure services that we have here for that community. | 13:16 |
| *** ociuhandu has joined #openstack-infra | 13:16 | |
| fungi | openstack does not use a mailing list for receiving vulnerability reports | 13:16 |
| fungi | https://security.openstack.org/#how-to-report-security-issues-to-openstack | 13:17 |
| tommylikehu | oh! your personal email address | 13:18 |
| michal_kowieski4 | Hi,Could you help me with access to Gerrit opendev repository (https://review.opendev.org/). I dont have option to clone with ssh. I have account, accepted ICLA and added ssh key. Did I forget about somethink? | 13:19 |
| fungi | tommylikehu: but if you're asking about adding mailing lists (for whatever purpose) that can be done here: https://opendev.org/opendev/system-config/src/branch/master/modules/openstack_project/manifests/lists.pp | 13:19 |
| smcginnis | michal_kowieski4: You don't clone from review.opendev.org. You should be able to drop the "review." part. | 13:20 |
| fungi | tommylikehu: each mailing list domain needs a mailman::site object defined, and then each individual mailing list needs a mailman_list | 13:20 |
| fungi | michal_kowieski4: to clarify, you clone from https://opendev.org/ and then push to gerrit (using a tool like git-review preferably) | 13:21 |
| *** jpena|lunch is now known as jpena | 13:22 | |
| fungi | michal_kowieski4: https://docs.openstack.org/infra/manual/developers.html#starting-work-on-a-new-project | 13:23 |
| tommylikehu | fungi, great thanks :) | 13:24 |
| michal_kowieski4 | My problem is that I have only anonymous http options. I want to download project via ssh. I don't see options for it. I added ssh key in settings. | 13:25 |
| fungi | michal_kowieski4: what's wrong with obtaining it like `git clone https://opendev.org/openstack/<projectname> | 13:29 |
| fungi | or git clone https://opendev.org/<namespace>/<repo>` in general | 13:29 |
| fungi | my quoting is terrible this morning, sorry | 13:29 |
| *** derekh has quit IRC | 13:30 | |
| fungi | still attempting to caffeinate my brains | 13:30 |
| *** rpittau|bbl is now known as rpittau | 13:32 | |
| fungi | we prefer the gerrit remote only be used for pushing and fetching changes, to minimize load on that server. we run an entire farm of gitea servers for cloning and pulling from | 13:32 |
| michal_kowieski4 | I plan to contribute. I am not shure how can I push without ssh. | 13:37 |
| jrosser | fetch with https from opendev.org, when you set up 'git review' it will add the ssh gerrit remote for you automatically | 13:38 |
| jrosser | work through the sandbox example and you'll see what it does | 13:39 |
| *** hashar has joined #openstack-infra | 13:39 | |
| jrosser | https://docs.openstack.org/infra/manual/sandbox.html | 13:39 |
| yoctozepto | fungi: https://review.opendev.org/702166 <- are you able to accept it? | 13:40 |
| yoctozepto | it's kind of hard to navigate CI today, everything turns red on touch | 13:41 |
| fungi | michal_kowieski4: to reinterate, you can push over ssh to review.opendev.org (please just use a tool like git-review for that) but still pull/clone over https from opendev.org | 13:48 |
| openstackgerrit | Simon Westphahl proposed zuul/zuul master: Add optional support for circular dependencies https://review.opendev.org/685354 | 13:48 |
| fungi | er, reiterate | 13:48 |
| openstackgerrit | Simon Westphahl proposed zuul/zuul master: Add optional support for circular dependencies https://review.opendev.org/685354 | 13:50 |
| michal_kowieski4 | It's work. Thanks for help! | 13:50 |
| *** michal_kowieski4 has quit IRC | 13:50 | |
| *** Goneri has joined #openstack-infra | 13:51 | |
| fungi | yoctozepto: i'm still trying to come up to speed on what changed with setuptools to cause it to stop working with the way we serve wheels. i'm hesitant to merge something which might make the situation worse without fully understanding what's happened | 13:53 |
| *** rcernin has quit IRC | 13:53 | |
| fungi | but i will make this situation a priority as soon as i get to my workstation | 13:53 |
| yoctozepto | fungi: the correct fix would be to fix the metadate in wheel index | 13:54 |
| yoctozepto | centos7 works because wheel failed to build there: http://mirror.regionone.limestone.openstack.org/wheel/centos-7-x86_64/setuptools/ | 13:54 |
| yoctozepto | on ubuntu it succeded though: http://mirror.regionone.limestone.openstack.org/wheel/ubuntu-18.04-x86_64/setuptools/ | 13:54 |
| yoctozepto | and hence it fails | 13:54 |
| yoctozepto | because metadata is missing to tell pip it's py3 only | 13:55 |
| yoctozepto | see https://github.com/pypa/pip/issues/7586 | 13:55 |
| mordred | fungi: don't you love waking up to pip breaks? | 13:55 |
| mordred | infra-root I'm mostly AFK today | 13:57 |
| yoctozepto | mordred: and all that while /me working on CI for jslib, just my luck... | 13:57 |
| fungi | yoctozepto: there is no wheel metadata at play, if i understand the problem correctly | 13:57 |
| yoctozepto | fungi: the metadata is in index | 13:58 |
| yoctozepto | the missing metadata, that is :-) | 13:58 |
| fungi | it's that we don't provide additional information via the simple api because we're just serving via apache mod_autoindex | 13:58 |
| *** KeithMnemonic has joined #openstack-infra | 13:59 | |
| fungi | so basically the python packaging ecosystem has just this weekend broken the ability to effectively serve packages without a special api | 13:59 |
| yoctozepto | fungi: yeah, that's what they did | 14:00 |
| fungi | i expect the same problem would occur with an on-disk wheelhouse too | 14:00 |
| yoctozepto | mhm | 14:00 |
| fungi | because it's just a directory full of whl files, not served by any intelligent api | 14:01 |
| yoctozepto | yes, indeed, that's what it is ;-) | 14:01 |
| yoctozepto | and it seems no ready tool to build this exists | 14:02 |
| fungi | we could stop mirroring the affected whl files to solve this, right? | 14:02 |
| fungi | like if our wheel mirror didn't include setuptools wheels | 14:02 |
| yoctozepto | yeah | 14:02 |
| *** ykarel|afk has joined #openstack-infra | 14:02 | |
| yoctozepto | though others will follow | 14:02 |
| yoctozepto | I think hammer is the way to go atm | 14:03 |
| fungi | sure, but i bet they all already publish wheels on pypi too | 14:03 |
| yoctozepto | we only mirror or build too? | 14:03 |
| fungi | our wheel mirror could be fixed to only contain platform-specific wheels we had to build because those projects can't/don't publish wheels on pypi | 14:04 |
| yoctozepto | fungi: also makes sense | 14:04 |
| *** ykarel|afk is now known as ykarel | 14:05 | |
| fungi | the "wheel mirror" we maintain exists to provide prebuilt wheels for situations where no appropriate wheels exist on pypi, so that jobs don't time out trying to build stuff like numpy from sdist | 14:05 |
| fungi | but it was apparently easier to just publish wheels for everything listed in upper-constraints.txt from open branches of openstack/requirements | 14:06 |
| *** derekh has joined #openstack-infra | 14:06 | |
| yoctozepto | heh ;D | 14:07 |
| yoctozepto | https://review.opendev.org/702220 seems helping kolla-ansible jobs | 14:10 |
| yoctozepto | so at least we know it really is that part that lacks | 14:10 |
| yoctozepto | just fix the wheels and we are golden | 14:10 |
| yoctozepto | (or rather their indices) | 14:10 |
| yoctozepto | fungi: maybe you are right the best for now would be to delete setuptools wheels completely | 14:11 |
| yoctozepto | minimal impact | 14:11 |
| yoctozepto | maximum gains :-) | 14:11 |
| *** roman_g has joined #openstack-infra | 14:22 | |
| openstackgerrit | Andreas Jaeger proposed opendev/gerrit master: Fix .gitreview https://review.opendev.org/702224 | 14:23 |
| fungi | well, we'll also need to stop the wheel building jobs so it doesn't just put that setuptools wheel back again | 14:25 |
| yoctozepto | fungi: indeed, I definitely meant a permanent solution | 14:26 |
| jrosser | fwiw in openstack-ansible we pin the versions of setuptools/pip/wheel to things we know work | 14:27 |
| jrosser | thats an alternative approach to decouple from upstream breaking your world | 14:27 |
| fungi | indeed | 14:29 |
| yoctozepto | jrosser: we use upper-constraints in kolla and don't update setuptools but try running any py2 virtualenv in zuul on ubuntu now and you are straight doomed :-) | 14:30 |
| fungi | yeah, if memory serves, virtualenv unconditionally retrieves the latest setuptools when building an environment | 14:31 |
| openstackgerrit | Simon Westphahl proposed zuul/zuul master: Ensure correct cleanup on repo update and reset https://review.opendev.org/701531 | 14:31 |
| yoctozepto | fungi: it consults pip | 14:31 |
| fungi | well, pip no longer requires setuptools since a few releases ago | 14:32 |
| fungi | or you mean it uses pip to find out what the latest setuptools version is? | 14:32 |
| fungi | if so, that matches what i recall | 14:33 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: executor: prevent restart exception on bad build log https://review.opendev.org/702228 | 14:34 |
| *** pcaruana has joined #openstack-infra | 14:34 | |
| clarkb | the difference with setuptools is it is often expected to be there already. For everything else we can useenvironment markers and largely ignore the problem | 14:34 |
| clarkb | I dont think weneed to add python version specifiers to our mirror index via html tags | 14:35 |
| yoctozepto | fungi: that virtualenv consults pip to find latest packages (pip, not setuptools) | 14:35 |
| yoctozepto | clarkb: we need to properly feed pip | 14:36 |
| clarkb | yoctozepto: yes that is ehat environment markers are for | 14:36 |
| yoctozepto | otherwise you will keep getting this issue with wheels incompatibilities between minor versions too | 14:36 |
| openstackgerrit | Simon Westphahl proposed zuul/zuul master: Ensure correct cleanup on repo update and reset https://review.opendev.org/701531 | 14:36 |
| clarkb | we already use them extensively | 14:36 |
| yoctozepto | yeah, all upper-constraints and stuff | 14:37 |
| yoctozepto | plus constraints in requirements | 14:37 |
| fungi | but i do think this is an opportunity to also make the wheel mirrors more efficient by not carrying copies of wheels which are already available on pypi | 14:37 |
| fungi | (setuptools is only one example) | 14:37 |
| yoctozepto | folks, I just want to remind that atm we have RED all around so best to do something quick atm and then think future forward :-) | 14:38 |
| jrosser | i still think this can all be pinned | 14:38 |
| jrosser | as osa stable branches have all built OK in the last 12hours | 14:38 |
| yoctozepto | jrosser: we tried with devstack already, too many places to fix, with cyclic deps | 14:38 |
| jrosser | so there is an example if anyone is interested to see how that was done | 14:38 |
| clarkb | jrosser: that is what the proposed fixes do last I chwcked (yesterday) | 14:38 |
| yoctozepto | jrosser: kolla builds fine too for that matter :-) | 14:39 |
| jrosser | ok fine:) | 14:39 |
| clarkb | the problem with the html indexhints is not all versions of pip support them and not all packages publish that info | 14:40 |
| clarkb | using env markers is farmore robust and globally applicable | 14:40 |
| clarkb | for now rempving the offending wheel seems like a good way to get unstuck | 14:40 |
| yoctozepto | clarkb: we already have them in main index | 14:40 |
| clarkb | yoctozepto: the main index is just proxied form pypi | 14:41 |
| *** ociuhandu has quit IRC | 14:41 | |
| clarkb | we dont generate that | 14:41 |
| clarkb | and not all packages have them | 14:41 |
| yoctozepto | clarkb: yeah, true | 14:42 |
| clarkb | fungi: really lxml and libvirt-python arethe only two wheels I expect we really need | 14:43 |
| clarkb | maybe some numpy wheels | 14:44 |
| fungi | well, even if we switch to a whitelist, what needs adjusting is how we copy files into the cache | 14:44 |
| fungi | because right now it's mirroring the transitive set, all wheels retrieved in the course of building that sdist | 14:45 |
| fungi | and setuptools would still show up | 14:45 |
| clarkb | ya I think first step is ro exclude setuptools | 14:45 |
| fungi | er, i mean even if we switch to a smaller whitelist (upper-constraints.txt is already a whitelist) | 14:46 |
| clarkb | which we can probably do with rsync options similar to ehat we do when mirroring fedora atomic | 14:46 |
| fungi | i wonder if there isn't a way to just exclude any wheels which were retrieved from pypi | 14:46 |
| clarkb | exclude setuptools, pip, and virtualenv | 14:46 |
| fungi | we don't need those in the wheel cache | 14:46 |
| *** jackedin has joined #openstack-infra | 14:46 | |
| fungi | because they'll just be fetched from our pypi proxy cache instead if they're not present in the wheel cache | 14:47 |
| clarkb | you might be able to parse the build output to whitelist the wheels that were buily | 14:47 |
| fungi | that's probably less fiddly than consulting pypi to check for presence of each one | 14:47 |
| mordred | clarkb: I for some reason had this thought that we had logic at some point to not put things into the wheel cache if they were downloaded as wheels | 14:49 |
| mordred | clarkb: like, I *think* we have to do extra work to get both sets? | 14:50 |
| clarkb | I thought it copied the whole wheel cache which has both | 14:50 |
| mordred | yeah - but I thought we had to do something special to get downloaded wheels put into the wheel cache | 14:51 |
| * mordred is looking | 14:51 | |
| fungi | we did something special to avoid rebuilding stuff already available on pypi | 14:53 |
| fungi | the script is openstack/project-config:roles/build-wheels/files/wheel-build.sh | 14:53 |
| fungi | in case you're hunting for it | 14:53 |
| mordred | yeah- you're right | 14:54 |
| mordred | I don't see the thing my brain had invented | 14:54 |
| fungi | $WHEELHOUSE_DIR just contains everything | 14:54 |
| *** pgaxatte has quit IRC | 14:54 | |
| fungi | and then playbooks/wheel/post-base.yaml pulls it all | 14:55 |
| clarkb | in the rsync for that job can we exclude setuptools, pip, and virtualenv for now? | 14:55 |
| clarkb | that should be an easy change to land then we remove those file? | 14:55 |
| fungi | that's what i was looking for examples to do | 14:55 |
| clarkb | ah for examples look in fedora mirror sync script | 14:56 |
| clarkb | it has a bunch of file exclusion patterns | 14:56 |
| *** jackedin has quit IRC | 14:57 | |
| *** ykarel has quit IRC | 14:57 | |
| fungi | well, the other bit is working out the file pattern | 14:57 |
| *** ykarel has joined #openstack-infra | 14:57 | |
| *** bdodd has joined #openstack-infra | 14:57 | |
| *** pgaxatte has joined #openstack-infra | 14:59 | |
| mordred | ++ - that seems like the best plan for short term | 14:59 |
| *** lmiccini has quit IRC | 14:59 | |
| clarkb | setuptools-.*\.whl | 14:59 |
| fungi | running locally, it seems to create setuptools-45.0.0-py2.py3-none-any.whl in the wheelhouse | 15:00 |
| clarkb | https://mirror.mtl01.inap.opendev.org/wheel/ubuntu-18.04-x86_64/setuptools/ | 15:00 |
| fungi | and virtualenv-16.7.9-py2.py3-none-any.whl and pip-19.3.1-py2.py3-none-any.whl | 15:00 |
| fungi | for those | 15:00 |
| mordred | fwiw - none-any might be a good thing to look for to excluse more broadly | 15:00 |
| mordred | http://mirror.dfw.rax.opendev.org/wheel/ubuntu-18.04-x86_64/lxml/ <-- if you look at that, that's wheels that are "important" that we build because of the compiling step | 15:01 |
| mordred | they've all got things like cp37 and manylinux in their wheel names - which indicate what they were compiled for | 15:01 |
| mordred | things with no compile get marked none-any | 15:01 |
| mordred | (as a next step after just excluding setuptools perhaps) | 15:02 |
| fungi | so yes something like logs/(pip|setuptools|virtualenv)-.*\.whl | 15:02 |
| *** eharney has quit IRC | 15:02 | |
| mordred | fungi: ++ | 15:02 |
| fungi | i do find it intriguing though that setuptools dropped python3 support yet still publishes a py2.py3 whl | 15:02 |
| clarkb | yesthat is a bug | 15:03 |
| *** surpatil|mtg has quit IRC | 15:04 | |
| fungi | but yeah, none-any and manylinux1 can wheels likely be excluded broadly | 15:04 |
| clarkb | manylinux is probably safer since we dont generate those | 15:05 |
| clarkb | but none any can be locally built or pypi sourced | 15:05 |
| mordred | ++ ... we should probably do that as a followup just so we can think about it - but I think clarkb's point about the win of the wheel mirror being things with compile step is a good one | 15:05 |
| fungi | even if some dependencies fail to publish a wheel and build a none-any wheel from their sdist, they probably take trivial time for jobs to build on their own | 15:05 |
| mordred | yup | 15:05 |
| clarkb | fungi: good point | 15:05 |
| fungi | it's the platform-specific wheels which are likely to have lengthy build times | 15:06 |
| mordred | yup | 15:06 |
| openstackgerrit | Paul Albertella proposed zuul/zuul-jobs master: Add Bazel build and install roles https://review.opendev.org/693513 | 15:07 |
| fungi | clarkb: when you say we have exclusion examples in the fedora mirror script, you mean for an rsync command not for an ansible synchronize task, right? | 15:07 |
| fungi | we have to stuff those into an rsync_opts parameter? | 15:08 |
| clarkb | yes and maybe | 15:08 |
| clarkb | not sureif ansible exposes that "natively" | 15:08 |
| fungi | rsync_opts: | 15:09 |
| fungi | - "--exclude=logs/(pip|setuptools|virtualenv)-.*\.whl" | 15:09 |
| fungi | something like that | 15:09 |
| fungi | as parameters to the synchronize task | 15:09 |
| fungi | we do it in roles/test-upload-logs/tasks/main.yaml | 15:09 |
| *** ociuhandu has joined #openstack-infra | 15:10 | |
| clarkb | ya docs show it isnt natively exposed | 15:10 |
| *** zxiiro has joined #openstack-infra | 15:10 | |
| fungi | oh, also rsync needs glob patterns not regular expressions | 15:11 |
| *** lpetrut has joined #openstack-infra | 15:12 | |
| *** lmiccini has joined #openstack-infra | 15:13 | |
| *** ykarel is now known as ykarel|afk | 15:15 | |
| openstackgerrit | Benjamin Schanzel proposed zuul/zuul master: Allow Passing of Jitter Values in TimerDriver https://review.opendev.org/702237 | 15:18 |
| fungi | aha, looks like we could get more fancy if we wanted to filter it in roles/copy-wheels/files/wheel-copy.sh | 15:19 |
| clarkb | yup I think that is where we need to filter? that copies directly from wheelhouse to afs | 15:23 |
| openstackgerrit | Paul Albertella proposed zuul/zuul-jobs master: Add Bazel build and install roles https://review.opendev.org/693513 | 15:23 |
| fungi | right, i'm working on a patch now to insert a find -exec rm across the source tree | 15:24 |
| mordred | ++ | 15:24 |
| fungi | so it shouldn't delete anything from the mirror, but will stop adding things which match the globs we specify | 15:24 |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Add noop-jobs for opendev/gerrit https://review.opendev.org/702242 | 15:25 |
| AJaeger | clarkb, fungi, there are no jobs defined for gerrit, is this ok? ^ | 15:25 |
| clarkb | AJaeger: I think that is because master gerrit doesn't really do anything | 15:25 |
| clarkb | AJaeger: all our gerrit dev is on the version branches and those should have jobs | 15:26 |
| *** pgaxatte has quit IRC | 15:26 | |
| *** tbachman has joined #openstack-infra | 15:26 | |
| AJaeger | clarkb: I didn't see jobs configured on those either ;( | 15:28 |
| AJaeger | clarkb: I pushed https://review.opendev.org/702224 - but no jobs run on it | 15:28 |
| AJaeger | we can also force-merge it ;) | 15:29 |
| clarkb | AJaeger: well master is never used so I don't think we need that change? | 15:29 |
| clarkb | I guess for hygiene we should land it | 15:29 |
| AJaeger | or remove .gitreview? | 15:30 |
| AJaeger | for hygine yes... | 15:30 |
| AJaeger | ok, let me abandon 702242 | 15:35 |
| openstackgerrit | Jeremy Stanley proposed openstack/project-config master: Don't mirror wheels for pip/setuptools/virtualenv https://review.opendev.org/702244 | 15:37 |
| fungi | yeah, i think removing the .gitreview file from the opendev/gerrit repo is likely better | 15:38 |
| AJaeger | fungi, clarkb, I push that as well and you can choose ;) | 15:39 |
| openstackgerrit | Andreas Jaeger proposed opendev/gerrit master: Remove .gitreview https://review.opendev.org/702245 | 15:40 |
| AJaeger | fungi, clarkb ^ | 15:40 |
| clarkb | fungi: change lgtm | 15:41 |
| fungi | mordred: https://review.opendev.org/702244 if you're still semi-around | 15:42 |
| fungi | i can remove the affected wheel from the wheel-mirror afs volume once that merges | 15:43 |
| mordred | fungi: looks great | 15:43 |
| mordred | fungi: and is both readable for this case, and probably pretty easy to change to just remove anything with none-any in the future should we choose to | 15:44 |
| clarkb | yoctozepto: jrosser: re kolla and osa jobs working you might want ot double check that you use the wheel mirror if available. That can really speed up job runtimes if building lxml and libvirt-python | 15:45 |
| clarkb | (granted don't switch until we've landed the above fixes :) ) | 15:45 |
| *** dave-mccowan has joined #openstack-infra | 15:47 | |
| *** derekh has quit IRC | 15:47 | |
| *** dave-mccowan has quit IRC | 15:52 | |
| clarkb | kevinz: hello, if you are around I had some questions about some odd image states in linaro-london | 15:53 |
| *** derekh has joined #openstack-infra | 15:54 | |
| *** lmiccini has quit IRC | 15:55 | |
| *** eharney has joined #openstack-infra | 15:56 | |
| *** rfolco is now known as rfolco|brb | 15:57 | |
| openstackgerrit | Merged openstack/project-config master: Don't mirror wheels for pip/setuptools/virtualenv https://review.opendev.org/702244 | 16:00 |
| clarkb | fungi: ^ | 16:00 |
| fungi | yup, thanks, getting to work on the deletion now | 16:01 |
| mordred | \o/ | 16:02 |
| clarkb | once deletion is done we should tryand land some of the fixes people tried to land earlier | 16:04 |
| clarkb | switching to python3 venvs seems like a good idea where possible anyway | 16:04 |
| mordred | ++ | 16:05 |
| openstackgerrit | Merged zuul/zuul-jobs master: Rename vars/ directory to prevent conflicts https://review.opendev.org/702144 | 16:06 |
| *** ociuhandu has quit IRC | 16:07 | |
| *** tbachman has quit IRC | 16:09 | |
| *** pcaruana has quit IRC | 16:12 | |
| fungi | btw, running find across /afs/.openstack.org/mirror/wheel/ is not exactly fast (want to make sure i find all copies of setuptools-45.0.0-py2.py3-none-any.whl it has) | 16:12 |
| clarkb | fungi: it should be pretty deterministic, one copy for each supported distro | 16:13 |
| clarkb | https://mirror.mtl01.inap.opendev.org/wheel/centos-7-x86_64/setuptools/setuptools-45.0.0-py2.py3-none-any.whl https://mirror.mtl01.inap.opendev.org/wheel/ubuntu-16.04-x86_64/setuptools/setuptools-45.0.0-py2.py3-none-any.whl https://mirror.mtl01.inap.opendev.org/wheel/ubuntu-18.04-x86_64/setuptools/setuptools-45.0.0-py2.py3-none-any.whl | 16:14 |
| *** gyee has joined #openstack-infra | 16:15 | |
| fungi | yep, i'm just curious why it's slow if the tree is so shallow | 16:15 |
| *** iurygregory has quit IRC | 16:17 | |
| clarkb | infra-root https://review.opendev.org/#/c/702050/ and https://review.opendev.org/702051 are changes to get insecure-ci-registry.opendev.org's cert renewed via LE | 16:17 |
| clarkb | corvus: ^ you may be interested in that as it touches zuul-registry and docker compose stuff | 16:17 |
| *** jamesmcarthur has joined #openstack-infra | 16:20 | |
| fungi | so in actuality there is no setuptools 45.0.0 wheel in our centos cache (maybe it's ceased updating in recent history?) | 16:26 |
| *** ociuhandu has joined #openstack-infra | 16:26 | |
| clarkb | fungi: it is because centos 7 is python2 | 16:27 |
| clarkb | so the wheel build will avoid pulling python3 only setuptools | 16:27 |
| fungi | oh, yep that'll be it | 16:27 |
| fungi | i have removed /afs/.openstack.org/mirror/wheel/ubuntu-16.04-x86_64/s/setuptools/setuptools-45.0.0-py2.py3-none-any.whl and /afs/.openstack.org/mirror/wheel/ubuntu-18.04-x86_64/s/setuptools/setuptools-45.0.0-py2.py3-none-any.whl and released the corresponding mirror.wheel.xenialx64 and mirror.wheel.bionicx64 volumes | 16:29 |
| fungi | #status log manually removed setuptools 45.0.0 wheels from our afs wheel cache | 16:30 |
| openstackstatus | fungi: finished logging | 16:30 |
| clarkb | I rechecked a small number of devstack and d-g changes so that we get back data on whether or not this is working | 16:32 |
| clarkb | it is possible that we have some places with old pip that don't understand the data-requires-python index fields and we'd need to fix this some other way | 16:32 |
| *** jamesmcarthur has quit IRC | 16:34 | |
| *** jamesmcarthur has joined #openstack-infra | 16:35 | |
| *** jamesmcarthur_ has joined #openstack-infra | 16:39 | |
| *** jamesmcarthur has quit IRC | 16:39 | |
| fungi | https://github.com/pypa/pip/pull/3877 merged several years ago and so pip >=9 should in theory handle this | 16:40 |
| fungi | at least https://github.com/pypa/pip/commit/b506992 says it first appeared in the 9.0.0 tag | 16:40 |
| *** rfolco|brb has quit IRC | 16:41 | |
| *** eernst has joined #openstack-infra | 16:42 | |
| *** tbachman has joined #openstack-infra | 16:42 | |
| *** avladu has joined #openstack-infra | 16:45 | |
| avladu | Hello, I have a question related zuul gates for x/cloudbase-init project | 16:46 |
| *** ykarel|afk is now known as ykarel | 16:46 | |
| avladu | I have in my zuul.yml from https://github.com/cloudbase/cloudbase-init/blob/master/.zuul.yaml, the definition of 3 gates, but a lot more get executed. Example https://review.opendev.org/#/c/701737/ . How can I disable the openstack-tox-py27 gate? | 16:48 |
| clarkb | avladu: https://opendev.org/openstack/project-config/src/branch/master/zuul.d/projects.yaml#L5021-L5033 is where the extra jobs are coming from | 16:48 |
| clarkb | you can propose and update to that file to remove jobs you don't want | 16:49 |
| *** derekh has quit IRC | 16:49 | |
| avladu | clarkb thank you very much | 16:49 |
| AJaeger | avladu: openstack-python-jobs has the tox-py27 gate | 16:49 |
| avladu | got it, will make a patch for https://opendev.org/openstack/project-config/src/branch/master/zuul.d/projects.yaml#L5021-L5033 | 16:50 |
| AJaeger | avladu: https://docs.openstack.org/infra/openstack-zuul-jobs/project-templates.html shows the definitions | 16:50 |
| *** hashar has quit IRC | 16:52 | |
| *** icarusfactor has quit IRC | 16:55 | |
| *** ykarel is now known as ykarel|away | 16:57 | |
| corvus | clarkb: both lgtm, thanks! | 17:01 |
| openstackgerrit | Adrian Vladu proposed openstack/project-config master: cloudbase-init: set zuul project to noop-jobs https://review.opendev.org/702259 | 17:02 |
| *** rpittau is now known as rpittau|afk | 17:02 | |
| mordred | clarkb: same here - although I agree with corvus' comment on the second one | 17:03 |
| *** lucasagomes has quit IRC | 17:04 | |
| openstackgerrit | Merged opendev/zone-opendev.org master: Manage insecure-ci-registry ssl with LE https://review.opendev.org/702050 | 17:04 |
| corvus | yeah, food for thought on that one | 17:04 |
| clarkb | fwiw ianw's https://review.opendev.org/#/c/702162/ is the devstack change that I think we should merge. https://review.opendev.org/#/c/702126/8 is the d-g change to land I Think | 17:04 |
| clarkb | the d-g change could use some cleanup though | 17:05 |
| *** diablo_rojo has quit IRC | 17:05 | |
| clarkb | corvus: mordred ya I'm not sure. The good news is we have independent verification that our certs are not expiring and we renew after 2 of the 3 months of cert validity | 17:05 |
| clarkb | corvus: mordred: I think the worst case there for current situation is itdoes break then we have 30 days to fix it | 17:06 |
| clarkb | with plenty of email spam to remind us | 17:06 |
| clarkb | not ideal but it shouldn't get lost in the shuffle iether | 17:06 |
| *** mattw4 has joined #openstack-infra | 17:08 | |
| AJaeger | avladu: don't use noop-jobs, you have in repo jobs. I'll comment on the review | 17:11 |
| *** pcaruana has joined #openstack-infra | 17:13 | |
| avladu | AJaeger sure, any help would be great. so what would be the best way to update the code? I would like zuul to include only the .zuul.yaml file, which I will update asap to include the formerly defined jobs in the project-config repo | 17:13 |
| AJaeger | avladu: the publish-to-pypi should stay in project-config, see the infra manual, https://docs.openstack.org/infra/manual/creators.html#central-config-exceptions - the rest can move in your repo. | 17:14 |
| AJaeger | noop-jobs is only needed if you have no jobs defined at all. | 17:15 |
| *** yolanda has quit IRC | 17:15 | |
| *** eernst has quit IRC | 17:15 | |
| *** diablo_rojo has joined #openstack-infra | 17:15 | |
| AJaeger | avladu: so, either do it like https://opendev.org/openstack/project-config/src/branch/master/zuul.d/projects.yaml#L2929 - or remove the stanza completely (if you don't publish to pypi) | 17:16 |
| *** ahosam has joined #openstack-infra | 17:19 | |
| *** eernst has joined #openstack-infra | 17:19 | |
| avladu | AJaeger I suppose I can leave the publish part there, although I need to check with which credentials that would be done, in case of a tagged release (from what I understand) | 17:19 |
| *** rfolco|brb has joined #openstack-infra | 17:20 | |
| *** nicolasbock has joined #openstack-infra | 17:21 | |
| AJaeger | avladu: pypi should be setup with our openstack user as owner, so our tools can upload without extra credentials | 17:22 |
| fungi | the "openstackci" pypi account needs at least index maintainer privileges on the project, i believe (though you could probably supply a different one using a custom zuul secret, i'm not aware of any examples where projects are doing that) | 17:24 |
| AJaeger | and the repo is setup that way: https://pypi.org/project/cloudbase-init/ | 17:25 |
| *** electrofelix has quit IRC | 17:27 | |
| avladu | got it, in that case it should already be in a working state. there wasn t a tag since 2017, and on pypi appears correct. | 17:27 |
| *** ykarel|away has quit IRC | 17:29 | |
| *** evrardjp has quit IRC | 17:33 | |
| *** eharney has quit IRC | 17:34 | |
| *** evrardjp has joined #openstack-infra | 17:34 | |
| openstackgerrit | Adrian Vladu proposed openstack/project-config master: x/cloudbase-init: remove openstack gates https://review.opendev.org/702259 | 17:35 |
| avladu | AJaeger the updated patch plus https://review.opendev.org/#/c/702270/ should do the trick. My focus was to remove the Python 2.7 gates, as there is more and more hassle to make the tests pass correctly with the openstack upper requirements. | 17:37 |
| *** jamesmcarthur_ has quit IRC | 17:38 | |
| *** eernst has quit IRC | 17:40 | |
| *** eharney has joined #openstack-infra | 17:41 | |
| *** eernst has joined #openstack-infra | 17:43 | |
| AJaeger | avladu: I suggest you update setup.cfg as well to remove mention of python 2 | 17:46 |
| AJaeger | avladu: see how the openstack repos are doing it | 17:46 |
| *** eernst has quit IRC | 17:47 | |
| avladu | AJaeger: will update the setup.cfg, thanks for the heads up. | 17:51 |
| smcginnis | avladu: It doesn't apply to an x/ repo, but you can see some of the recommended steps here - https://governance.openstack.org/tc/goals/selected/ussuri/drop-py27.html#completion-criteria | 17:51 |
| *** dtantsur is now known as dtantsur|afk | 17:53 | |
| *** lpetrut has quit IRC | 17:54 | |
| *** eernst has joined #openstack-infra | 17:54 | |
| *** ociuhandu_ has joined #openstack-infra | 17:55 | |
| *** gfidente is now known as gfidente|afk | 17:55 | |
| fungi | well, much of it does apply, it's just not mandatory | 17:56 |
| fungi | it's still a decent set of recommendations | 17:56 |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: Authorization rules: support YAML nested dictionaries https://review.opendev.org/684790 | 17:58 |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: JWT drivers: Deprecate RS256withJWKS, introduce OpenIDConnect https://review.opendev.org/701972 | 17:58 |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: OIDCAuthenticator: add capabilities, scope option https://review.opendev.org/702275 | 17:58 |
| *** derekh has joined #openstack-infra | 17:58 | |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: admin REST API: zuul-web integration https://review.opendev.org/643536 | 17:58 |
| *** ociuhandu has quit IRC | 17:58 | |
| avladu | fungi: current version of cloudbase-init will support Python 2.7, the main reason for this thread was to have control on the gets to be able to remove Py27. | 17:59 |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: admin REST API: zuul-web integration https://review.opendev.org/643536 | 17:59 |
| *** ociuhandu_ has quit IRC | 17:59 | |
| *** tbachman has quit IRC | 18:00 | |
| *** avladu has quit IRC | 18:03 | |
| *** kjackal has quit IRC | 18:04 | |
| *** kjackal_v2 has joined #openstack-infra | 18:04 | |
| *** jpena is now known as jpena|off | 18:06 | |
| *** bnemec has quit IRC | 18:14 | |
| *** diablo_rojo has quit IRC | 18:17 | |
| *** lpetrut has joined #openstack-infra | 18:17 | |
| *** bnemec has joined #openstack-infra | 18:18 | |
| openstackgerrit | Merged zuul/nodepool master: Always identify static nodes by node tuple https://review.opendev.org/701969 | 18:19 |
| *** lpetrut has quit IRC | 18:28 | |
| *** tbachman has joined #openstack-infra | 18:32 | |
| fungi | #status log used mailman's `rmlist` tool to retire the following abandoned lists.openstack.org mailing lists at their respective owners' requests (archives were preserved and not removed): elections-committee, openstack-content, staffwithtrinet, transparency, women-of-openstack | 18:33 |
| openstackstatus | fungi: finished logging | 18:33 |
| fungi | infra-root: i've confirmed none of the above lists are still present in our puppetry, so they won't be automatically recreated in error | 18:34 |
| fungi | if anybody knows of any other mailing lists which should be retired, i'm happy to look into it | 18:35 |
| *** jtomasek has quit IRC | 18:40 | |
| *** tesseract has quit IRC | 18:53 | |
| *** jamesmcarthur has joined #openstack-infra | 19:01 | |
| *** jamesmcarthur has quit IRC | 19:05 | |
| *** jamesmcarthur has joined #openstack-infra | 19:05 | |
| *** mattw4 has quit IRC | 19:18 | |
| *** mattw4 has joined #openstack-infra | 19:18 | |
| *** jamesmcarthur has quit IRC | 19:18 | |
| openstackgerrit | Clark Boylan proposed zuul/nodepool master: Delete dib images when all uploads set to deleting https://review.opendev.org/702062 | 19:30 |
| *** mattw4 has quit IRC | 19:33 | |
| *** mattw4 has joined #openstack-infra | 19:33 | |
| *** jamesmcarthur has joined #openstack-infra | 19:36 | |
| clarkb | fungi: thanks for the lceanup | 19:37 |
| clarkb | now that I've got ^ pushed up I'm going to take a break | 19:37 |
| clarkb | back in a bit | 19:37 |
| *** jamesmcarthur has quit IRC | 19:42 | |
| melwitt | clarkb: hey, your advice to propose the devstack patch to stable/train worked and I Depends-On it in nova and got a grenade failure. there are lots of errors in the mysql log but I'm not sure what to do next. I added a few comments to the patch a bit ago. wondering if something might ring a bell for you https://review.opendev.org/701478 | 19:43 |
| openstackgerrit | Clark Boylan proposed openstack/devstack-gate master: Use python3 for virtualenv if exists https://review.opendev.org/702126 | 19:44 |
| *** ociuhandu has joined #openstack-infra | 19:45 | |
| *** sshnaidm is now known as sshnaidm|bbl | 19:45 | |
| clarkb | melwitt: this job https://zuul.opendev.org/t/openstack/build/833a46b05c9641b9b22b3ee7f394e80b ? | 19:45 |
| AJaeger | amotoki, ianychoi, experts, translations jobs for horizon repos fail - see http://zuul.opendev.org/t/openstack/build/61cce1de0e9e428db25e480bac8d36e2 . Looks like pint 0.10.1 which is in upper-constraints only works with Python 3.6 - but we run on Xenial with Python 3.5 ;( | 19:45 |
| AJaeger | any ideas on how to fix that one? ^ | 19:45 |
| melwitt | clarkb: yes that's the one | 19:45 |
| melwitt | clarkb: I put the mysql error log in a paste and added it as a comment on the review (since you have to download it). and at the end of the log I noticed "[Note] InnoDB: page_cleaner: 1000ms intended loop took 6943ms. The settings might not be optimal. (flushed=1 and evicted=0, during the time.)" | 19:46 |
| melwitt | and wondered if there's anything we could or should do about our innodb settings. I found this when I googled https://dba.stackexchange.com/questions/121236/mysql-innodb-page-cleaner-settings-might-not-be-optimal | 19:47 |
| AJaeger | fungi, thanks for force-merging 702245 | 19:48 |
| yoctozepto | 16:45:31 <clarkb> re kolla and osa jobs working you might want ot double check that you use the wheel mirror if available. That can really speed up job runtimes if building lxml and libvirt-python | 19:48 |
| clarkb | melwitt: hrm that might be a better question for mordred | 19:48 |
| yoctozepto | good idea, I created https://bugs.launchpad.net/kolla/+bug/1859516 for us to remember, thanks | 19:48 |
| openstack | Launchpad bug 1859516 in kolla "use wheel mirror in CI" [Wishlist,Triaged] | 19:48 |
| clarkb | melwitt: it is possible that we should retune the db settings for current hardware | 19:49 |
| clarkb | I know some tuning work happened a long time ago but it may have grown old and need updating | 19:49 |
| clarkb | melwitt: I'll look at logs more closely after lunch | 19:50 |
| openstackgerrit | Merged opendev/gerrit master: Remove .gitreview https://review.opendev.org/702245 | 19:50 |
| fungi | AJaeger: you're welcome--thanks for looking into it! | 19:50 |
| *** ociuhandu has quit IRC | 19:50 | |
| melwitt | thanks clarkb | 19:50 |
| ianw | fungi / clarkb: thanks for looking in, so i see we've purged the problem .whls | 19:53 |
| ianw | i think we still might need the <a> tags, in general, because theoretically this could happen with python 3.x v 3.y too ... but it's not, now, an immediate concern | 19:54 |
| yoctozepto | ianw: +1 | 19:54 |
| clarkb | ianw env markers should handle that though | 19:55 |
| clarkb | and we already have tooling for tthem | 19:55 |
| clarkb | (also packages dont have to have that index data so you cant rely on it) | 19:57 |
| ianw | yeah, it is a MAY but i think bandersnatch puts it in now too | 19:58 |
| ianw | env markers i guess work for things using requirements, i'm just thinking for maximum compatibility; if somebody's job just pip install's a project we have a wheel for | 19:59 |
| fungi | i think if we make this more completely filter out all wheels which already exist on pypi the problem goes away | 20:02 |
| fungi | because we will only build and serve wheels for platforms where they can be run, with interpreters they support | 20:02 |
| fungi | also that slims down the utilization in those volumes substantially | 20:03 |
| *** eharney has quit IRC | 20:03 | |
| fungi | though that aside, i still think it's a bug that the setuptools 45.0.0 wheel's filename says py2.py3 when it only supports py3 | 20:05 |
| clarkb | ++ | 20:05 |
| *** Goneri has quit IRC | 20:05 | |
| fungi | https://github.com/pypa/setuptools/issues/1964 | 20:10 |
| ianw | i think we should skip pypi built wheels too, and we can also implement the metadata (i think it should fit into the index generation script not too bad) for maximum compatibility | 20:12 |
| ianw | that's now story 2007084 task 38083 | 20:14 |
| *** ralonsoh has quit IRC | 20:34 | |
| *** eharney has joined #openstack-infra | 20:57 | |
| clarkb | melwitt: I'm seeing no valid hosts were found errors in the n-sch and n-cond logs for that job | 21:00 |
| clarkb | that seems to line up with at least some of the tempest failures | 21:00 |
| *** eharney has quit IRC | 21:01 | |
| *** kjackal_v2 has quit IRC | 21:04 | |
| clarkb | lookingat the nova cpu logs it appears to ahve available memory, disk and cpu | 21:06 |
| clarkb | makes me think that maybe the issue is related to connectivity between the nova cpu process and conductor? | 21:06 |
| clarkb | melwitt: I'm not really sure how to check ^ something cells related maybe? | 21:08 |
| *** rfolco|brb is now known as rfolco | 21:08 | |
| clarkb | and this is a single node job so this isn't going to be related to subnode registration | 21:09 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-jobs master: fetch-tox-output: introduce zuul_use_fetch_output https://review.opendev.org/681864 | 21:14 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-jobs master: fetch-subunit-output: introduce zuul_use_fetch_output https://review.opendev.org/681882 | 21:14 |
| *** eernst has quit IRC | 21:21 | |
| openstackgerrit | Clark Boylan proposed zuul/nodepool master: Delete dib images when all uploads set to deleting https://review.opendev.org/702062 | 21:22 |
| melwitt | clarkb: right, it's getting NoValidHost as a result of a "timeout from cell database" which I'm assuming is connected to the mysql errors | 21:29 |
| *** derekh has quit IRC | 21:30 | |
| melwitt | I'm trying to figure out what is causing mysql to never respond to a read/write request | 21:30 |
| clarkb | melwitt: oh what file is the timeout in? I'm not seeing that in the conductor log | 21:30 |
| melwitt | clarkb: n-sch log, example "Timeout getting computes for cell f12ed72a-18af-44cf-b3a6-fcb45625f484" | 21:30 |
| melwitt | I'd link you but logs for this job have to be downloaded to look at | 21:31 |
| *** rfolco has quit IRC | 21:31 | |
| clarkb | ah and that is just a warning | 21:31 |
| clarkb | I found it, thanks | 21:31 |
| melwitt | ah yeah, sorry I didn't notice they are logged at warn level "Timed out waiting for response from cell f12ed72a-18af-44cf-b3a6-fcb45625f484" | 21:33 |
| melwitt | that means we waited for 60s for a result from a cell database and it didn't return anything within that time | 21:33 |
| clarkb | melwitt: looks like we don't log dstat csv files in grenade jobs | 21:35 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-jobs master: DNM: test tox-py36 on openshift node https://review.opendev.org/682049 | 21:35 |
| melwitt | this is a pretty rare event and I think it must be related to pressure on the host, there's a lot of errors in the mysql logs. but I don't know what else to do or know what's causing the database to slow down and not respond. I assumed the "[Note] InnoDB: page_cleaner: 1000ms intended loop took 6943ms. The settings might not be optimal. (flushed=1 and evicted=0, during the time.)" is related too | 21:35 |
| melwitt | clarkb: I think we do here right? https://zuul.opendev.org/t/openstack/build/833a46b05c9641b9b22b3ee7f394e80b/log/logs/screen-dstat.txt.gz | 21:35 |
| melwitt | I looked at it but admittedly I don't know if anything in there is bad or unusual | 21:36 |
| clarkb | melwitt: thats the human "readable" version which the grapher doesn't accept direclty ( you might be able to convert it though) | 21:39 |
| clarkb | melwitt: looking at that version we aren't really swapping a ton ~4MB in swap. mysqld is the biggets memory consumer | 21:39 |
| melwitt | clarkb: oh, I see, sorry | 21:40 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-jobs master: fetch-subunit-output: introduce zuul_use_fetch_output https://review.opendev.org/681882 | 21:40 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-jobs master: DNM: test tox-py36 on openshift node https://review.opendev.org/682049 | 21:40 |
| clarkb | melwitt: there is a little iowait times but its not consistently bad | 21:40 |
| melwitt | hmmm | 21:41 |
| melwitt | I wonder what is going on with the page_cleaner then, from what I googled, it means it had to clean up a lot more pages than expected | 21:43 |
| melwitt | I'll try to get mordred to comment on that part when he's around | 21:43 |
| clarkb | percona has done some work to make it more better https://www.percona.com/doc/percona-server/5.6/performance/page_cleaner_tuning.html no idea if those changes would help us though | 21:46 |
| clarkb | sounds like there is a fair bit of writing happenign to db at the time so it is taking the db longer to make those writes | 21:46 |
| clarkb | thats the sort of thing I would expect to get slower with bad iowait too | 21:46 |
| clarkb | but the io seems fine around when the first timeout happens | 21:46 |
| clarkb | it does get bad around 00:22:59 though | 21:46 |
| clarkb | (but that is about 5 minutes after the first problem) | 21:47 |
| melwitt | thanks for those insights | 21:48 |
| clarkb | melwitt: probably worthwhile to rerun the jobs a few times if necessary and collect more data too | 21:52 |
| clarkb | if this is consistent that might help us narrow it down. Or maybe it only happens on a specific provider etc | 21:52 |
| *** ociuhandu has joined #openstack-infra | 21:55 | |
| openstackgerrit | Ian Wienand proposed openstack/project-config master: Generate PEP503 index pages for wheel mirrors https://review.opendev.org/702299 | 21:58 |
| *** jamesmcarthur has joined #openstack-infra | 21:58 | |
| *** ociuhandu has quit IRC | 22:00 | |
| ianw | clarkb / fungi : ^ i think that should, in general, be helpful. note as a first step i have it writing out to a temp file, not index.html, since we don't really gate test it | 22:00 |
| *** vesper has joined #openstack-infra | 22:00 | |
| *** vesper11 has quit IRC | 22:01 | |
| *** sshnaidm|bbl is now known as sshnaidm | 22:02 | |
| *** strigazi has quit IRC | 22:12 | |
| *** slaweq has quit IRC | 22:12 | |
| *** strigazi has joined #openstack-infra | 22:13 | |
| clarkb | ianw: couple of notes inline. Also zuul is unhappy linting it | 22:22 |
| ianw | clarkb: yes, owrking on the linting, the hacking version is too old to understand f-strings | 22:23 |
| ianw | thanks :) | 22:24 |
| openstackgerrit | Clark Boylan proposed zuul/nodepool master: Use iterate_timeout in test waits https://review.opendev.org/702303 | 22:24 |
| *** xek has quit IRC | 22:24 | |
| ianw | fungi: if you have some time, the stack @ https://review.opendev.org/#/c/698128/ would be helpful. related to the new static sites | 22:24 |
| clarkb | semi related to ianw's change, pbr appears to support requires-python directives. Openstack may want to look at setting those values | 22:27 |
| clarkb | gmann: ^ this might be a good task for the python3 migration | 22:27 |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: install-docker: allow removal of conflicting packages https://review.opendev.org/702304 | 22:29 |
| *** zxiiro has quit IRC | 22:36 | |
| *** jackedin has joined #openstack-infra | 22:39 | |
| *** tkajinam has joined #openstack-infra | 22:49 | |
| *** rcernin has joined #openstack-infra | 22:57 | |
| *** jamesmcarthur has quit IRC | 23:03 | |
| *** tosky has quit IRC | 23:06 | |
| clarkb | ianw: http://paste.openstack.org/show/788263/ is a list of images that refuse to delete in linaro-london. I think you and kevinz have better overlap than I do. maybe you can try and bring that up and see if those images can be deleted cloud side? | 23:12 |
| clarkb | I've approved the insecure-ci-registry LE change | 23:13 |
| clarkb | I'll keep an eye on it | 23:13 |
| ianw | clarkb: can do | 23:15 |
| clarkb | I'ev WIP'd the insecure-ci-registry LE change beacuse I realize I forgot to put it in the LE group | 23:19 |
| clarkb | remote: https://review.opendev.org/702051 Manage insecure-ci-registry cert with LE fixes that | 23:20 |
| clarkb | not sure where gerritbot went to | 23:21 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Manage insecure-ci-registry cert with LE https://review.opendev.org/702051 | 23:21 |
| clarkb | corvus: mordred ^ that is a minor update to the change you already reviewed. Realized I needed to update the ansible group too | 23:21 |
| *** dchen has joined #openstack-infra | 23:24 | |
| *** jackedin has quit IRC | 23:27 | |
| *** ociuhandu has joined #openstack-infra | 23:30 | |
| *** ociuhandu has quit IRC | 23:35 | |
| clarkb | I'm doing much neglected software patching on my computing device. I'll be in and out for reboots for the next bit | 23:47 |
| clarkb | 1288 packages apparently :/ | 23:49 |
| clarkb | this is what i get for leaving the laptop off in a corner over the holidays | 23:49 |
| openstackgerrit | Ian Wienand proposed openstack/project-config master: Generate PEP503 index pages for wheel mirrors https://review.opendev.org/702299 | 23:49 |
| openstackgerrit | Ian Wienand proposed openstack/project-config master: Update hacking, fix errors/warnings https://review.opendev.org/702310 | 23:49 |
| ianw | clarkb: ^ thanks, notes applied. if you're happy, could apply and do a manual job trigger -- it should be a no-op as errors ignored and writing out to a temporary file | 23:51 |
| clarkb | yup rereviewing now | 23:52 |
| clarkb | both changes look good to me. If pip is looking at METADATA and not metadata.json seems like that is what we want to look at too | 23:56 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!