Friday, 2020-10-02

*** philroche has joined #openstack-infra00:05
*** armax has quit IRC00:13
*** hamalq has quit IRC00:16
openstackgerritGhanshyam Mann proposed openstack/devstack-gate master: Run bashate job on bionic nodeset  https://review.opendev.org/75562500:23
openstackgerritGhanshyam Mann proposed openstack/devstack-gate master: Set up stable/victoria  https://review.opendev.org/74961800:23
openstackgerritGhanshyam Mann proposed openstack/devstack-gate master: Update grenade setting for stable/victoria  https://review.opendev.org/75534100:23
openstackgerritGhanshyam Mann proposed openstack/devstack-gate master: Move stable branch gate job to stable/victoria  https://review.opendev.org/75541900:23
*** dchen has quit IRC00:47
*** dchen has joined #openstack-infra00:48
*** weshay|ruck has quit IRC00:54
*** weshay has joined #openstack-infra00:55
*** weshay is now known as weshay|ruck00:55
*** zzzeek has quit IRC01:17
*** zzzeek has joined #openstack-infra01:19
*** dulek has quit IRC01:20
*** dulek has joined #openstack-infra01:20
*** zxiiro has joined #openstack-infra01:33
*** Topner has joined #openstack-infra01:49
*** zzzeek has quit IRC01:51
*** zzzeek has joined #openstack-infra01:51
*** Topner has quit IRC01:54
*** dchen has quit IRC02:03
*** dchen has joined #openstack-infra02:04
*** smarcet_ has joined #openstack-infra02:12
*** smarcet has quit IRC02:13
*** smarcet_ is now known as smarcet02:13
*** zxiiro has quit IRC03:13
*** zxiiro has joined #openstack-infra03:15
*** philroche has quit IRC03:21
*** rcernin has quit IRC03:31
*** philroche has joined #openstack-infra03:49
*** Topner has joined #openstack-infra03:50
*** zzzeek has quit IRC03:52
*** zzzeek has joined #openstack-infra03:52
*** Topner has quit IRC03:54
*** rcernin has joined #openstack-infra03:59
*** rcernin has quit IRC04:13
*** rcernin has joined #openstack-infra04:30
*** evrardjp has quit IRC04:33
*** evrardjp has joined #openstack-infra04:33
*** matt_kosut has joined #openstack-infra05:09
*** bnemec has quit IRC05:23
*** gyee has quit IRC05:26
*** bnemec has joined #openstack-infra05:28
*** dchen has quit IRC05:37
*** dchen has joined #openstack-infra05:39
*** jtomasek has joined #openstack-infra05:39
*** Topner has joined #openstack-infra05:50
*** Topner has quit IRC05:55
*** zzzeek has quit IRC05:59
*** zzzeek has joined #openstack-infra06:01
*** hamalq has joined #openstack-infra06:12
*** lmiccini has joined #openstack-infra06:14
*** ralonsoh has joined #openstack-infra06:28
*** eolivare has joined #openstack-infra06:29
*** zxiiro has quit IRC06:39
*** dklyle has quit IRC06:52
*** jcapitao has joined #openstack-infra07:00
*** slaweq has joined #openstack-infra07:01
*** yolanda has quit IRC07:04
*** yolanda has joined #openstack-infra07:04
*** hamalq has quit IRC07:13
ralonsohhi folks07:17
ralonsohwe have a doubt/problem we don't know how to solve07:18
ralonsohthe same CI job is passing in mirror.mtl01.inap.opendev.org07:18
ralonsohbut not in mirror.bhs1.ovh.opendev.org07:18
ralonsohexample07:18
ralonsohpassing: https://12d9c9014a31cd38106f-d99046410a2db92aeb18e96327c94fc0.ssl.cf1.rackcdn.com/755314/1/check/neutron-tempest-plugin-scenario-openvswitch-stein/88582a4/job-output.txt07:18
ralonsohfailing: https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_ece/755314/1/check/neutron-tempest-plugin-scenario-linuxbridge-stein/eced887/job-output.txt07:18
ralonsohthe problem, so far, is the setuptools version installed07:19
ralonsohsetuptools-44.1.1 is working fine07:19
ralonsohbut not setuptools-50.3.007:19
*** andrewbonney has joined #openstack-infra07:20
*** rcernin has quit IRC07:27
slaweqfrickler: hi, can You take a look at what ralonsoh wrote ^^?07:37
slaweqI think we will need some infra experts help on that07:37
*** tosky has joined #openstack-infra07:43
*** Topner has joined #openstack-infra07:51
*** rcernin has joined #openstack-infra07:52
*** Topner has quit IRC07:56
*** rcernin has quit IRC07:56
fricklerslaweq: ralonsoh: it looks like the earlier pypi issues have reappeared, possibly in a different incarnation, see the earlier discussion in #opendev07:57
*** jpena|off is now known as jpena07:57
*** matbu has joined #openstack-infra07:58
*** hashar has joined #openstack-infra07:58
fricklersetuptools-50.3.0 is py3 only, pip2.7 shouldn't try to install it, but it still seems to try to when getting a corrupt index from pypi07:58
ralonsohfrickler, this is happening only in ovh mirror07:59
ralonsohhow could I check this specific mirror?07:59
fricklerone mitigation might be to add an explicit cap to devstack/tools/cap-pip.txt . or to conclude that py27 is really EOL by now and stop testing it07:59
ralonsohI saw this07:59
ralonsohand I was thinking about it07:59
fricklerralonsoh: well you should be able to access that mirror from anywhere07:59
ralonsohfrickler, but I don't see this setuptools version08:00
ralonsohhttps://mirror.bhs1.ovh.opendev.org/wheel/ubuntu-18.04-x86_64/setuptools/08:00
AJaegerIsn't our mirror now a proxy? So, we just download what's available from pypi...08:01
fricklerralonsoh: yeah, we don't build wheels for that, there's a second location for the pypi mirrored things08:01
frickler2020-10-02 04:30:06.626441 | controller | Looking in indexes: https://mirror.bhs1.ovh.opendev.org/pypi/simple, https://mirror.bhs1.ovh.opendev.org/wheel/ubuntu-18.04-x86_6408:04
frickler^^ the pypi/simple there08:04
ralonsohyeah, there are 20 setuptools references08:05
ralonsohI'm still trying to find the correct one08:05
*** lucasagomes has joined #openstack-infra08:07
*** slaweq has quit IRC08:11
*** slaweq has joined #openstack-infra08:16
ralonsohfrickler, sorry, but I really don't know were to look08:20
ralonsohwhere*08:20
fricklerralonsoh: I don't know the details about how pip does this either, but there's some kind of metadata hidden somewhere about which version supports which python version. likely the URL is something like https://mirror.bhs1.ovh.opendev.org/pypi/simple/setuptools/ but you also need special request params in order to see the interesting part08:24
*** derekh has joined #openstack-infra08:26
*** tkajinam has quit IRC08:29
*** tkajinam has joined #openstack-infra08:29
slaweqfrickler: but when You go to https://mirror.bhs1.ovh.opendev.org/wheel/ubuntu-18.04-x86_64/setuptools/ (from failed job) and https://mirror.iad.rax.opendev.org/wheel/ubuntu-18.04-x86_64/setuptools/ (passed one) there are the same versions of the setuptools in both08:33
slaweqand there is no even 50.0 something there :/08:33
ralonsohfungi, hi08:51
ralonsohhttps://bugs.launchpad.net/neutron/+bug/189817808:51
openstackLaunchpad bug 1898178 in neutron "Wrong setuptools version installed on stable branches gate" [Critical,Confirmed]08:51
*** lajoskatona has joined #openstack-infra08:51
ralonsohI think we have a problem with OVH mirror08:52
ralonsohwhen using py2708:52
ralonsohis installing an incorrect version of setuptools08:52
ralonsohhttps://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_ece/755314/1/check/neutron-tempest-plugin-scenario-linuxbridge-stein/eced887/job-output.txt08:52
*** yamamoto has quit IRC08:55
*** rcernin has joined #openstack-infra08:58
*** yamamoto has joined #openstack-infra09:01
*** rcernin has quit IRC09:03
*** tkajinam has quit IRC09:18
*** gfidente has joined #openstack-infra09:19
*** yamamoto has quit IRC09:37
*** ociuhandu has joined #openstack-infra09:48
*** rcernin has joined #openstack-infra09:49
*** Topner has joined #openstack-infra09:52
*** yamamoto has joined #openstack-infra09:54
*** rcernin has quit IRC09:55
*** xek has joined #openstack-infra09:55
*** ysandeep|sick is now known as ysandeep|holiday09:55
*** Topner has quit IRC09:57
*** xek has quit IRC10:10
*** xek has joined #openstack-infra10:10
*** xek has quit IRC10:17
*** ricolin_ has joined #openstack-infra10:17
*** xek has joined #openstack-infra10:19
*** xek has quit IRC10:37
*** xek has joined #openstack-infra10:37
*** jcapitao is now known as jcapitao_lunch11:09
*** yamamoto has quit IRC11:27
*** jpena is now known as jpena|lunch11:36
ralonsohfungi, hi11:43
ralonsohhttps://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_29f/755112/3/check/neutron-tempest-plugin-scenario-openvswitch-iptables_hybrid-stein/29f70a5/job-output.txt11:44
ralonsohwe have an issue with OVH mirrors, installing the incorrect version of setuptools in py2711:44
*** lpetrut has joined #openstack-infra11:44
*** dchen has quit IRC11:45
*** sshnaidm is now known as sshnaidm|afk11:46
*** yamamoto has joined #openstack-infra11:47
*** rledisez has quit IRC11:53
*** Topner has joined #openstack-infra11:53
*** iurygregory has quit IRC11:55
*** rledisez has joined #openstack-infra11:55
*** jcapitao_lunch is now known as jcapitao11:56
*** iurygregory has joined #openstack-infra11:57
*** yamamoto has quit IRC11:57
*** Topner has quit IRC11:58
*** rfolco has joined #openstack-infra11:58
*** yamamoto has joined #openstack-infra12:02
*** yamamoto has quit IRC12:02
*** rlandy has joined #openstack-infra12:07
*** rlandy is now known as rlandy|rover12:07
*** eharney has quit IRC12:08
*** slaweq has quit IRC12:09
*** slaweq has joined #openstack-infra12:10
fungiralonsoh: the "mirrors" are just proxies when it comes to installing pypi/python packages, so if they're serving different content that's usually a sign of regional inconsistencies in the cdn network for pypi itself12:26
fungithe last time something like this came up, it wound up being because the pypi admins have a "fallback" mirror that the cdn fetches from if it has trouble getting content from actual pypi, and the fallback doesn't have additional metadata12:29
fungiso if that version fallback of the index is cached in some of fastly's cdn endpoints in that part of the world, then pip may not be able to tell that it shouldn't download that version of setuptools, i suppose12:30
*** Topner has joined #openstack-infra12:30
*** jpena|lunch is now known as jpena12:30
fungiralonsoh: slaweq: also https://mirror.bhs1.ovh.opendev.org/wheel/ubuntu-18.04-x86_64/ isn't where it should be retrieving those packages from, that's a wheel cache we build to fill in the gaps when there's no wheel on pypi for a given package12:31
fungia while back we improved it to no longer copy wheels which already exist on pypi, which is why you won't find newer versions of things like setuptools there12:32
fungiwe just haven't taken the time to clean up old redundant wheels on it12:32
*** yamamoto has joined #openstack-infra12:35
fungiralonsoh: also how did you determine those ran in ovh? the first one says it ran in inap12:36
fungi(the second one you linked is ovh-bhs1 though yeah)12:37
*** xek has quit IRC12:37
*** yamamoto has quit IRC12:39
ralonsohfungi, yes, the first one passed, not the second one (in ovh)12:43
ralonsohsorry, I was having lunch12:43
fungilooking at the file list on pypi, setuptools publishes a whl and a zip for 50.3.0... pip on python2.7 isn't going to download the wheel because it's a python3-only wheel12:43
fungihow pip figures out what version it should download based on python interpreter support is the metadata served in the index file... view-source:https://mirror.bhs1.ovh.opendev.org/pypi/simple/setuptools/12:43
ralonsohI know12:44
fungidown at the end, you'll see "...data-requires-python=">=3.5">setuptools-50.3.0.zip..."12:44
ralonsohhmmm12:44
ralonsohbut we are executing that with py2712:44
fungiso pip should normally know not to retrieve setuptools-50.3.0.zip if it's running on python earlier than 3.512:44
fungi*however* the pypi admins have a fallback mirror which the fastly cdn will use if it can't reach warehouse, and that's not got the same metadata12:45
fungiaccording to earlier discussions we've had in #pypa-dev12:45
*** rcernin has joined #openstack-infra12:45
fungimy guess is that if we start dissecting the apache cache on the ovh-bhs1 mirror we're going to find some cached copies of the setuptools version index which doesn't have requires-python metadata12:46
ralonsohhow can I do this?12:46
fungiindicating fastly served it from pypi's bandersnatch mirror fallback and not warehouse12:47
ralonsohsorry, I'm a bit lost in infra...12:47
fungiyou can't, one of the opendev sysadmins will literally have to sift through compressed cache data on our proxy looking for evidence that pypi's cdn network is acting up again12:47
fungiand report it to the pypi admins12:47
ralonsohwhere can I request this?12:48
ralonsohwwe have a critical bug opened in Neutron, with this ingo12:48
ralonsohinfo*12:48
fungiyou just have, but i'm about to head out to run some errands12:48
ralonsohthanks a lot! as usual!12:48
fungiralonsoh: the short answer is it's not a bug with neutron, nor with our mirror infrastructure, it's a problem in fastly cdn network and/or the pypi admins' fallback mirror12:49
fungiso there's only so much we're going to be able to do about it besides presenting them with all the evidence we can gather and letting them decide how (and if) they want to fix their problem12:50
slaweqfungi: ralonsoh would it be maybe possible to workaround that issue in devstack, by e.g. forcing installation of setuptools < 50 in stable branches if py2 is used?12:51
ralonsohwe'll need that in devstack, in the pip installation method12:52
ralonsohadding a condition to the get-pip.sh script12:52
ralonsohI can push this patch as a workaround for now12:52
*** ricolin_ has quit IRC13:05
*** xek has joined #openstack-infra13:05
*** ricolin_ has joined #openstack-infra13:06
*** rcernin has quit IRC13:07
*** Goneri has joined #openstack-infra13:13
*** hashar has quit IRC13:25
*** dciabrin has joined #openstack-infra13:28
*** ricolin_ has quit IRC13:30
*** haleyb has quit IRC13:41
*** haleyb has joined #openstack-infra13:43
*** zxiiro has joined #openstack-infra13:51
*** ricolin_ has joined #openstack-infra13:52
*** priteau has joined #openstack-infra13:52
*** hashar has joined #openstack-infra13:55
*** eharney has joined #openstack-infra13:56
*** priteau has quit IRC13:58
fungiralonsoh: it cant' hurt. basically if pip is running on python<3.5 then we don't want to possibly request a version of setuptools which lacks support for that14:10
fungihowever setuptools probably has to be installed before pip to make that work... not entirely sure. i think we've previously capped setuptools in devstack for other reasons so there's likely already some mechanism in there for it14:11
ralonsohfungi, i've tested manually https://review.opendev.org/#/c/755783/14:12
ralonsohadding this cap in the cap file14:12
ralonsohand forces the correct version with py2714:12
*** xek has quit IRC14:14
*** xek has joined #openstack-infra14:14
fungiralonsoh: well, it's also going to naturally select the correct version most of the time... it's breaking when presented with an incomplete pypi index14:15
fungiwhich is going to be hard to reproduce locally14:15
ralonsohI know...14:16
fungii suppose you could test by dialling the setuptools version in the cap down even further than the latest version which supported python 2.7, and double-checking that it doesn't try to upgrade past that14:16
openstackgerritJavier Peña proposed openstack/project-config master: CentOS 8 Stream initial deployment  https://review.opendev.org/73479114:18
*** priteau has joined #openstack-infra14:19
*** xek has quit IRC14:20
*** xek has joined #openstack-infra14:21
fungiralonsoh: slaweq: catching up finally now that errands are done, it looks like this was already being discussed around 05:00 utc in #opendev too, impacting other providers in the same part of the world (montreal canada area)14:22
ralonsohreading now14:24
ralonsohhttp://eavesdrop.openstack.org/irclogs/%23opendev/%23opendev.2020-10-02.log.html14:24
fungivexxhost ca-ymq-1 and ovh bhs1 are both near montreal14:25
fungiso if it's a cdn issue for pypi then it makes sense they're likely landing requests on the same endpoints14:26
*** dviroel has quit IRC14:26
*** dviroel has joined #openstack-infra14:26
*** vkmc has quit IRC14:27
*** vkmc has joined #openstack-infra14:29
fungialso discussion in #pypa-dev14:35
fungihopefully once the pypi admins are online they can dig deeper14:35
*** Adri2000 has quit IRC14:36
*** yamamoto has joined #openstack-infra14:36
*** dklyle has joined #openstack-infra14:37
*** philroche_ has joined #openstack-infra14:38
*** philroche has quit IRC14:40
*** philroche_ is now known as philroche14:40
*** yamamoto has quit IRC14:40
*** armax has joined #openstack-infra14:45
*** lajoskatona has left #openstack-infra14:49
*** smarcet has quit IRC15:00
*** dklyle has quit IRC15:03
*** xek has quit IRC15:06
fungiralonsoh: slaweq: in case you're not following along in other channels, the pypi admins are discussing turning the problem site back off again, so hopefully this will resolve itself soon15:07
slaweqfungi: thx15:07
*** dklyle has joined #openstack-infra15:12
*** smarcet has joined #openstack-infra15:14
ralonsohfungi, yes, I'm following it15:17
ralonsohthanks!15:17
fungiyw15:17
*** lpetrut has quit IRC15:31
*** priteau has quit IRC15:45
*** lmiccini has quit IRC15:46
openstackgerritGhanshyam Mann proposed openstack/devstack-gate master: Update grenade setting for stable/victoria  https://review.opendev.org/75534115:47
openstackgerritGhanshyam Mann proposed openstack/devstack-gate master: Move stable branch gate job to stable/victoria  https://review.opendev.org/75541915:47
*** priteau has joined #openstack-infra15:53
*** lucasagomes has quit IRC16:02
*** eharney has quit IRC16:04
*** ricolin_ has quit IRC16:14
*** hashar has quit IRC16:28
*** slaweq has quit IRC16:32
*** yamamoto has joined #openstack-infra16:37
*** ralonsoh has quit IRC16:37
*** eharney has joined #openstack-infra16:41
*** xek has joined #openstack-infra16:41
*** xek has quit IRC16:41
*** yamamoto has quit IRC16:41
*** ociuhandu_ has joined #openstack-infra16:43
*** ociuhandu has quit IRC16:47
*** ociuhandu_ has quit IRC16:48
*** raissa has joined #openstack-infra16:50
*** hamalq has joined #openstack-infra16:58
dirkclarkb: could you please readd your +2 / W+1 to https://review.opendev.org/#/c/755550/ ?17:01
clarkblooking17:02
clarkbdone17:02
clarkbinfra-root fyi ^ bumping our default opendev.org ttl to one hour17:03
dirkclarkb: thank you17:04
clarkbcorvus: might want to consider that for zuul-ci.org too?17:05
*** andrewbonney has quit IRC17:08
*** eolivare has quit IRC17:12
*** derekh has quit IRC17:13
*** eolivare has joined #openstack-infra17:18
*** eolivare has quit IRC17:19
*** jpena is now known as jpena|off17:20
*** zxiiro has quit IRC17:25
*** bnemec is now known as beekneemech17:37
*** zxiiro has joined #openstack-infra17:46
*** gfidente has quit IRC17:46
*** Tengu has quit IRC17:48
*** jcapitao has quit IRC17:55
*** Topner has quit IRC17:59
*** raissa has quit IRC18:11
*** raissa has joined #openstack-infra18:11
*** raissa has quit IRC18:11
*** raissa has joined #openstack-infra18:12
*** raissa has quit IRC18:12
openstackgerritTristan Cacqueray proposed openstack/project-config master: infra-package-needs: add support for centos-8-stream  https://review.opendev.org/75586018:19
*** Topner has joined #openstack-infra18:23
*** Tengu has joined #openstack-infra18:25
*** Topner has quit IRC18:27
*** jamesdenton has quit IRC18:34
*** yamamoto has joined #openstack-infra18:38
*** yamamoto has quit IRC18:43
*** hashar has joined #openstack-infra18:43
*** smarcet has quit IRC18:54
openstackgerritMerged openstack/openstack-zuul-jobs master: Switch py36 jobs to centos-8  https://review.opendev.org/75111218:54
fungiwe should probably watch out for fallout from ^18:55
*** jamesdenton has joined #openstack-infra18:56
*** Topner has joined #openstack-infra18:57
openstackgerritMerged openstack/project-config master: infra-package-needs: add support for centos-8-stream  https://review.opendev.org/75586018:58
*** jtomasek has quit IRC19:02
*** Topner has quit IRC19:02
*** yolanda has quit IRC19:04
*** yolanda has joined #openstack-infra19:04
*** priteau has quit IRC19:17
openstackgerritMerged openstack/devstack-gate master: Run bashate job on bionic nodeset  https://review.opendev.org/75562519:53
*** samP has quit IRC19:56
*** samP has joined #openstack-infra19:56
*** tosky has quit IRC19:57
*** raissa has joined #openstack-infra19:57
*** tosky has joined #openstack-infra19:58
openstackgerritMerged openstack/devstack-gate master: Set up stable/victoria  https://review.opendev.org/74961820:05
*** tosky has quit IRC20:19
*** raissa has quit IRC20:27
*** cyberpear has joined #openstack-infra20:30
*** raissa has joined #openstack-infra20:32
*** zxiiro has quit IRC20:55
*** hashar has quit IRC20:56
*** rfolco has quit IRC20:58
*** Topner has joined #openstack-infra20:59
*** Topner has quit IRC21:03
*** HenryG has quit IRC21:19
*** eharney has quit IRC21:21
*** HenryG has joined #openstack-infra21:27
*** armax has quit IRC21:48
*** rlandy|rover has quit IRC21:59
*** raissa has quit IRC22:01
*** Topner has joined #openstack-infra22:02
*** Topner has quit IRC22:06
*** yamamoto has joined #openstack-infra22:08
*** yamamoto has quit IRC22:43
*** yamamoto has joined #openstack-infra23:03
*** armax has joined #openstack-infra23:05
*** yolanda has quit IRC23:23
*** yolanda has joined #openstack-infra23:24
*** ricolin_ has joined #openstack-infra23:49

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!