Monday, 2019-09-30

*** dychen has joined #openstack-infra00:02
*** yamamoto has quit IRC00:02
*** diablo_rojo has joined #openstack-infra00:09
*** yamamoto has joined #openstack-infra00:12
*** dchen has quit IRC00:13
*** dchen has joined #openstack-infra00:14
*** ricolin has joined #openstack-infra00:32
*** yamamoto has quit IRC00:36
*** jamesmcarthur has joined #openstack-infra00:41
*** jamesmcarthur has quit IRC00:48
*** jamesmcarthur has joined #openstack-infra00:49
*** diablo_rojo has quit IRC00:49
*** yamamoto has joined #openstack-infra00:50
*** diablo_rojo has joined #openstack-infra00:50
*** diablo_rojo has quit IRC00:51
*** diablo_rojo has joined #openstack-infra00:52
*** jamesmcarthur has quit IRC00:54
*** jamesmcarthur has joined #openstack-infra00:55
*** dychen has quit IRC01:01
*** jamesmcarthur has quit IRC01:10
*** jamesmcarthur has joined #openstack-infra01:11
*** jamesmcarthur has quit IRC01:16
*** jamesmcarthur has joined #openstack-infra01:21
*** yamamoto has quit IRC01:31
*** yamamoto has joined #openstack-infra01:34
*** ykarel|away has joined #openstack-infra01:55
*** diablo_rojo has quit IRC02:00
*** jamesmcarthur has quit IRC02:02
*** tkajinam_ has joined #openstack-infra02:06
*** tkajinam has quit IRC02:09
*** yamamoto has quit IRC02:24
*** yamamoto has joined #openstack-infra02:28
*** auristor has quit IRC02:32
*** yamamoto has quit IRC02:45
*** markvoelker has joined #openstack-infra02:46
*** dychen has joined #openstack-infra02:49
*** dychen has quit IRC02:50
*** auristor has joined #openstack-infra02:53
*** yamamoto has joined #openstack-infra03:38
*** bradm has quit IRC03:38
*** yamamoto has quit IRC03:42
*** yamamoto has joined #openstack-infra03:44
*** jamesmcarthur has joined #openstack-infra03:45
*** psachin has joined #openstack-infra03:55
*** dciabrin_ has quit IRC04:04
*** udesale has joined #openstack-infra04:04
*** toabctl has joined #openstack-infra04:07
*** jamesmcarthur has quit IRC04:14
*** whoami-rajat has joined #openstack-infra04:24
*** ociuhandu has joined #openstack-infra04:30
*** ociuhandu has quit IRC04:35
*** ykarel|away has quit IRC04:36
*** jtomasek has quit IRC04:36
AJaegerianw: ^04:43
AJaegerconfig-core, please review https://review.opendev.org/679806 https://review.opendev.org/684374 and https://review.opendev.org/684124 https://review.opendev.org/68127004:44
*** ykarel|away has joined #openstack-infra04:53
openstackgerritIan Wienand proposed openstack/diskimage-builder master: [wip] pip-and-virtualenv : deprecate source for CentOS 8, new variables  https://review.opendev.org/68446205:01
openstackgerritIan Wienand proposed openstack/diskimage-builder master: Fix networking for CentOS 8  https://review.opendev.org/68451305:01
openstackgerritIan Wienand proposed openstack/diskimage-builder master: CentOS 8 minimal testing and support  https://review.opendev.org/68486005:01
openstackgerritIan Wienand proposed openstack/diskimage-builder master: WIP: CentOS 8 support  https://review.opendev.org/68430805:01
openstackgerritIan Wienand proposed openstack/diskimage-builder master: [wip] simple-init: Use wrappers to call pip for glean install  https://review.opendev.org/68564305:01
*** markvoelker has quit IRC05:02
*** jaosorior has joined #openstack-infra05:03
*** pcaruana has joined #openstack-infra05:24
*** dpawlik has joined #openstack-infra05:28
*** pcaruana has quit IRC05:30
*** soniya29 has joined #openstack-infra05:31
*** raukadah is now known as chandankumar05:33
*** janki has joined #openstack-infra05:35
*** yamamoto has quit IRC05:38
*** yamamoto has joined #openstack-infra05:57
*** markvoelker has joined #openstack-infra05:58
*** dpawlik has quit IRC06:01
*** kjackal has joined #openstack-infra06:02
*** markvoelker has quit IRC06:02
*** iurygregory has quit IRC06:04
*** soniya29 has quit IRC06:04
*** soniya29 has joined #openstack-infra06:05
openstackgerritMerged openstack/project-config master: Use iptables for openSUSE  https://review.opendev.org/68323606:05
*** soniya29 has quit IRC06:12
*** soniya29 has joined #openstack-infra06:15
*** jamesmcarthur has joined #openstack-infra06:16
*** jtomasek has joined #openstack-infra06:17
*** jamesmcarthur has quit IRC06:20
openstackgerritMasayuki Igawa proposed openstack/hacking master: Add PDF documentation build  https://review.opendev.org/68565506:22
*** slaweq has joined #openstack-infra06:23
*** soniya29 has quit IRC06:28
*** dpawlik has joined #openstack-infra06:29
*** soniya29 has joined #openstack-infra06:30
*** ykarel|away is now known as ykarel06:31
*** pgaxatte has joined #openstack-infra06:38
*** whoami-rajat has quit IRC06:39
*** pcaruana has joined #openstack-infra06:41
*** bradm has joined #openstack-infra06:48
*** pgaxatte has quit IRC06:58
*** pgaxatte has joined #openstack-infra07:00
*** iurygregory has joined #openstack-infra07:01
*** whoami-rajat has joined #openstack-infra07:01
openstackgerritSimon Westphahl proposed zuul/zuul master: Add optional support for circular dependencies  https://review.opendev.org/68535407:05
*** kopecmartin has joined #openstack-infra07:05
*** jtomasek has quit IRC07:06
*** jtomasek has joined #openstack-infra07:07
*** pkopec has joined #openstack-infra07:13
*** rcernin has quit IRC07:14
*** tesseract has joined #openstack-infra07:17
*** e0ne has quit IRC07:19
openstackgerritSimon Westphahl proposed zuul/zuul master: Add optional support for circular dependencies  https://review.opendev.org/68535407:19
*** jchhatbar has joined #openstack-infra07:23
*** arxcruz|zzz is now known as arxcruz|ruck07:25
*** janki has quit IRC07:25
*** jpena|off is now known as jpena07:29
*** iurygregory has quit IRC07:34
openstackgerritMasayuki Igawa proposed openstack/hacking master: Add PDF documentation build  https://review.opendev.org/68565507:36
*** tesseract has quit IRC07:43
*** tesseract has joined #openstack-infra07:43
*** surpatil has joined #openstack-infra07:45
*** dpawlik has quit IRC07:45
*** rpittau|afk is now known as rpittau07:52
*** soniya29 has quit IRC07:53
*** ykarel is now known as ykarel|lunch07:53
*** markvoelker has joined #openstack-infra08:00
*** ralonsoh has joined #openstack-infra08:00
*** tkajinam_ has quit IRC08:00
*** dpawlik has joined #openstack-infra08:04
*** markvoelker has quit IRC08:04
*** iurygregory has joined #openstack-infra08:11
*** dchen has quit IRC08:14
*** xek has joined #openstack-infra08:23
*** dtantsur|afk is now known as dtantsur08:24
openstackgerritMerged openstack/project-config master: Update Networking OVN failure rate dashboard  https://review.opendev.org/68412408:25
*** xenos76 has joined #openstack-infra08:26
*** e0ne has joined #openstack-infra08:31
*** derekh has joined #openstack-infra08:38
*** dpawlik has quit IRC08:39
*** georgk5 has quit IRC08:41
*** georgk has joined #openstack-infra08:41
*** ykarel|lunch is now known as ykarel08:43
*** yamamoto has quit IRC08:44
openstackgerritSimon Westphahl proposed zuul/zuul master: Add optional support for circular dependencies  https://review.opendev.org/68535408:53
*** soniya29 has joined #openstack-infra08:54
*** dpawlik has joined #openstack-infra08:59
*** dpawlik has quit IRC09:04
*** tosky has joined #openstack-infra09:10
*** dpawlik has joined #openstack-infra09:13
*** ricolin has quit IRC09:13
*** smcginnis has quit IRC09:14
*** yamamoto has joined #openstack-infra09:19
openstackgerritMerged zuul/zuul-jobs master: Use iptables for openSUSE  https://review.opendev.org/68549109:21
*** zbr is now known as zbr|ruck09:24
*** yamamoto has quit IRC09:24
*** tobiash has quit IRC09:27
*** tobiash has joined #openstack-infra09:29
*** dpawlik has quit IRC09:29
*** dpawlik has joined #openstack-infra09:48
*** odicha has quit IRC09:48
*** yamamoto has joined #openstack-infra09:49
*** ociuhandu has joined #openstack-infra09:52
*** gfidente has joined #openstack-infra09:55
*** odicha has joined #openstack-infra09:56
*** aedc has quit IRC10:04
*** yamamoto has quit IRC10:13
*** gfidente has quit IRC10:16
*** tosky has quit IRC10:17
*** tosky has joined #openstack-infra10:18
*** soniya29 has quit IRC10:26
*** soniya29 has joined #openstack-infra10:26
*** dciabrin_ has joined #openstack-infra10:28
openstackgerritSimon Westphahl proposed zuul/zuul master: Add optional support for circular dependencies  https://review.opendev.org/68535410:32
*** ociuhandu has quit IRC10:40
fricklerinfra-root: exim strikes again https://exim.org/static/doc/security/CVE-2019-16928.txt10:44
*** yamamoto has joined #openstack-infra10:46
*** yamamoto has quit IRC10:52
*** ociuhandu has joined #openstack-infra10:53
*** ricolin has joined #openstack-infra10:54
frickleroh, I misread the affected versions, that's >=4.92 only, so bionic with 4.90 not affected10:56
*** dchen has joined #openstack-infra10:57
*** jchhatbar has quit IRC11:00
*** yamamoto has joined #openstack-infra11:01
*** markvoelker has joined #openstack-infra11:01
*** yamamoto_ has joined #openstack-infra11:03
*** markvoelker has quit IRC11:06
*** yamamoto has quit IRC11:07
zbr|ruckout of curiosity, can we do somethign about the log urls which became very unfriendly (long)? auth in url, non standard ports... example https://openstack.fortnebula.com:13808/v1/AUTH_e8fd161dc34c421a979a9e6421f823e9/zuul_opendev_logs_341/685466/1/gate/tripleo-ci-centos-7-containers-multinode/34175b4/job-output.txt11:09
zbr|rucksoon they will no longer fit in a single irc message11:10
openstackgerritFabien Boucher proposed zuul/zuul master: WIP - gitlab - bootstrap the driver structure + Webhook support  https://review.opendev.org/68568211:10
*** chandankumar is now known as raukadah11:11
*** gfidente has joined #openstack-infra11:12
*** udesale has quit IRC11:13
fungizbr|ruck: switch to sharing the log view from zuul's build results interface? like https://zuul.opendev.org/t/openstack/build/34175b4984fa4c70b5dc11ad9dc1b6e9/log/job-output.txt11:25
zbr|ruckfungi: the build failures are almost never on the job-output log, so you do need to browse the logs.11:26
*** odicha has quit IRC11:27
fungiwell, you linked job-output.txt in your example, but the log viewer can browse to whatever you want11:27
zbr|ruckfungi: ok, i wil be back a this, now i got pinged by ruck dufy... :(11:28
zbr|ruckduty11:28
*** rcernin has joined #openstack-infra11:28
fungiexample: https://zuul.opendev.org/t/openstack/build/34175b4984fa4c70b5dc11ad9dc1b6e9/log/logs/delorean_logs/20/e6/20e69383e10878e38a0944e2f1d0514269500e2c_dev/rpmbuild.log.txt.gz11:31
*** jpena is now known as jpena|lunch11:31
*** ociuhandu has quit IRC11:35
*** ociuhandu has joined #openstack-infra11:36
*** smarcet has joined #openstack-infra11:37
openstackgerritFeruzjon Muyassarov proposed openstack/project-config master: Create a project for Airship go-redfish client  https://review.opendev.org/68568811:37
*** ianychoi has quit IRC11:39
*** xek has quit IRC11:39
*** xek_ has joined #openstack-infra11:39
*** odicha has joined #openstack-infra11:39
*** ianychoi has joined #openstack-infra11:40
*** weshay has joined #openstack-infra11:43
openstackgerritFeruzjon Muyassarov proposed openstack/project-config master: Add airship/go-redfish project  https://review.opendev.org/68568811:55
*** yamamoto_ has quit IRC11:59
*** kjackal_v2 has joined #openstack-infra12:04
*** kjackal has quit IRC12:07
*** goldyfruit has quit IRC12:18
*** yamamoto has joined #openstack-infra12:24
*** yamamoto has quit IRC12:26
*** jamesmcarthur has joined #openstack-infra12:26
*** otherwiseguy_ is now known as otherwiseguy12:28
*** jpena|lunch is now known as jpena12:30
ykarelAre there some mirror issues in ovh?12:31
ykareli see error in our job related to it12:33
ykarelexample:- https://5e6d48fe25fecf1b2eda-cc994f454b3da94a07b55a097da7db60.ssl.cf5.rackcdn.com/685393/5/check/puppet-openstack-integration-5-scenario001-tempest-centos-7-luminous/0abbb26/job-output.txt12:33
*** rfolco has joined #openstack-infra12:33
ykarelttp://mirror.bhs1.ovh.openstack.org:8080/buildlogs.centos/centos/7/storage/x86_64/ceph-luminous/repodata/repomd.xml: [Errno 14] HTTP Error 503 - Service Unavailable12:33
ykarelalso seeing with http://mirror.gra1.ovh.openstack.org:8080/rdo, may be issues are there with more servers12:33
*** jamesmcarthur has quit IRC12:35
*** smarcet has quit IRC12:37
openstackgerritFeruzjon Muyassarov proposed openstack/project-config master: Add airship/go-redfish project - Jira Issue https://airship.atlassian.net/browse/AIR-169 - The go-redfish library is currently hosted on github.com/Nordix/go-redfish Change-Id: Ic04eb2f1f631ae84556437425e1c3ed84c8804d5  https://review.opendev.org/68568812:40
openstackgerritFeruzjon Muyassarov proposed openstack/project-config master: Add airship/go-redfish project  https://review.opendev.org/68568812:42
*** odicha has quit IRC12:44
*** Goneri has joined #openstack-infra12:44
*** odicha has joined #openstack-infra12:45
fungiykarel: those aren't really mirrors, just caching proxies. generally if they're returning a 503 it means that there was some problem reaching the real services they're proxying12:45
ykarelfungi, so from above two logs, one was returning 503, and other returning 404 Not found for some packages12:46
fungiif the official sites for buildlogs.centos and rdo are served from the same network, then maybe it's not reachable from there12:46
fungiahh, 404 would generally indicate that it did reach the backend and the backend said that file wasn't available/existent12:47
fungicurrently http://mirror.bhs1.ovh.openstack.org:8080/buildlogs.centos/centos/7/storage/x86_64/ceph-luminous/repodata/repomd.xml is returning content, so seems it's able to fetch it now12:48
fungihttp://mirror.gra1.ovh.openstack.org:8080/rdo/ is also returning content, not sure what subpath was giving you a 404 though12:49
*** rcernin has quit IRC12:50
* ykarel checks12:50
*** aaronsheffield has joined #openstack-infra12:52
ykarelfungi, 404 was with packages in same repo:- http://mirror.gra1.ovh.openstack.org:8080/buildlogs.centos/centos/7/storage/x86_64/ceph-luminous/12:57
ykarelhttps://openstack.fortnebula.com:13808/v1/AUTH_e8fd161dc34c421a979a9e6421f823e9/zuul_opendev_logs_702/685393/5/check/puppet-openstack-integration-5-scenario003-tempest-centos-7/7028141/logs/puppet.txt.gz12:58
*** dtantsur is now known as dtantsur|afk12:58
*** dklyle has quit IRC13:00
*** markvoelker has joined #openstack-infra13:02
fungiahh, okay, so buildlogs.centos again, but from an entirely different provider that time13:03
fungii don't see a 404 in that puppet.txt.gz log13:04
*** markvoelker has quit IRC13:06
*** smarcet has joined #openstack-infra13:10
*** dave-mccowan has joined #openstack-infra13:16
*** goldyfruit has joined #openstack-infra13:18
*** yamamoto has joined #openstack-infra13:22
*** mriedem has joined #openstack-infra13:25
*** smarcet has quit IRC13:27
*** whoami-rajat has quit IRC13:29
*** smarcet has joined #openstack-infra13:30
*** sthussey has joined #openstack-infra13:30
*** soniya29 has quit IRC13:31
*** yamamoto has quit IRC13:33
*** dpawlik has quit IRC13:36
*** roman_g has joined #openstack-infra13:41
*** bdodd has joined #openstack-infra13:43
*** dklyle has joined #openstack-infra13:43
*** dklyle has quit IRC13:48
*** david-lyle has joined #openstack-infra13:48
*** munimeha1 has joined #openstack-infra13:49
*** rosmaita has joined #openstack-infra13:50
*** odicha has quit IRC13:52
AJaegerpabelanger: please review https://review.opendev.org/685615 and dependencies13:54
*** jamesdenton has quit IRC13:54
clarkbfungi: I agree the error in that last log paste appears to be related to sql connection details missing13:58
*** ociuhandu has quit IRC13:58
clarkbas for debugging proxy connectivity my process tends to be: 1) fetch url on mirror and confirm it is not working 2) construct url for backend direct access and check if that also fails. 3a) if 2) fails then problem is with backend, report it upstream as able 3b) if backend works check network connectivity between mirror and backend13:59
*** redrobot has joined #openstack-infra13:59
clarkbas fungi mentions that 3a) case is likely to produce 404s the whole way through whereas 3b will be 500s of some sort14:00
openstackgerritAndreas Jaeger proposed openstack/openstack-zuul-jobs master: Testing templates in templates...  https://review.opendev.org/68571614:03
AJaegerfrickler: I doubt this works as you suggested, but let's give it a try ^14:03
AJaegerclarkb: good morning, ianw flagged https://review.opendev.org/683431 for PTL review if you want to tackle it at some point. it adds a special nested label for VMs.14:04
pabelangerAJaeger: what is the opendev docs process? Given that windmill isn't openstack, I'm unsure if we want to depend on that job?14:04
pabelangeralso, maybe it should be moved into opendev tenant14:04
openstackgerritDavid Shrewsbury proposed zuul/zuul-website master: Revert "CSS fix for ul/li in FAQ"  https://review.opendev.org/68571714:04
*** jamesdenton has joined #openstack-infra14:06
AJaegerpabelanger: there's opendev-tox-docs - but no template.14:06
pabelangerokay, let me poke into that and see14:06
pabelangerI can do that this week14:06
AJaegerthanks, pabelanger14:07
*** smcginni1 has joined #openstack-infra14:08
*** adriant has quit IRC14:09
*** iokiwi has quit IRC14:09
*** dchen has quit IRC14:09
*** smcginni1 is now known as smcginnis_14:11
*** iokiwi has joined #openstack-infra14:12
fricklerAJaeger: o.k., thanks for testing that14:14
openstackgerritMerged zuul/zuul-website master: Revert "CSS fix for ul/li in FAQ"  https://review.opendev.org/68571714:17
*** pcaruana has quit IRC14:18
AJaegerconfig-core, please review https://review.opendev.org/679806 https://review.opendev.org/684374 and https://review.opendev.org/68127014:21
openstackgerritMerged openstack/openstack-zuul-jobs master: Add Python3 project templates for Ussuri release  https://review.opendev.org/68545314:21
*** pgaxatte has quit IRC14:32
*** Lucas_Gray has joined #openstack-infra14:32
*** ociuhandu has joined #openstack-infra14:33
openstackgerritMatt Riedemann proposed opendev/elastic-recheck master: Generalize query for bug 1808010  https://review.opendev.org/68572514:35
mriedemclarkb: ^14:35
openstackbug 1808010 in OpenStack Compute (nova) "Tempest cirros ssh setup fails due to lack of disk space causing config-drive setup to fail forcing fallback to metadata server which fails due to hitting 10 second timeout." [Medium,Confirmed] https://launchpad.net/bugs/180801014:35
*** diablo_rojo has joined #openstack-infra14:46
*** smcginnis_ is now known as smcginnis14:46
*** surpatil has quit IRC14:48
*** brtknr has joined #openstack-infra14:48
*** slaweq has quit IRC14:48
brtknrhey all14:48
brtknrjust wondering if anyone know about why these jobs are failing: https://review.opendev.org/#/c/685620/14:50
*** slaweq has joined #openstack-infra14:50
brtknrthe changes are yaml file changes... so effectively a noop14:50
brtknrthe same jobs are passing locally14:50
*** goldyfruit_ has joined #openstack-infra14:51
clarkbmy suggestion here will be the same as in #openstack-qa. There isn't enough information logged by the individual test cases to understand why they are failing. You will likely need to add that logging so that we can determine the case of failure then go from there14:51
clarkbit appears that http requests are failing because the webserver isn't running14:52
clarkbif we can get logs from the webserver that will likely help a lot14:52
brtknrclarkb: if you look at the py36 results, the error is "BlockingIOError: [Errno 11] write could not complete without blocking"14:52
*** goldyfruit has quit IRC14:53
brtknronly in py27, it is "Resource temporarily unavailable"...14:53
clarkbok so your file is opened for asyncio and raises and exception if it would block? if that is the acse then you'll need to handle blocking there14:54
openstackgerritMerged opendev/elastic-recheck master: Generalize query for bug 1808010  https://review.opendev.org/68572514:55
openstackbug 1808010 in OpenStack Compute (nova) "Tempest cirros ssh setup fails due to lack of disk space causing config-drive setup to fail forcing fallback to metadata server which fails due to hitting 10 second timeout." [Medium,Confirmed] https://launchpad.net/bugs/180801014:55
clarkbalso is the unittest job running linter checks?14:57
clarkb(might be a good idea to move those into the linter job?)14:57
clarkbbrtknr: looks like this is happening with print() function calls so stdout would block14:59
clarkbI don't see magnum replacing stdout with a iostring fixture like many of the other unittests do. Its possible that the actual stdout buffer is filling as a result15:00
brtknrclarkb: could you show me an example?15:01
brtknrthe same tests were working on 26/09... they've been broken since 27/09... not a lot has changed on our end15:02
*** markvoelker has joined #openstack-infra15:02
brtknrclarkb: here's an example of a PS which passed on 26/09 and fails on 30/09: https://review.opendev.org/#/c/675511/15:03
openstackgerritTristan Cacqueray proposed zuul/zuul master: Replace tox-py36 by tox-py37  https://review.opendev.org/68215815:03
clarkbbrtknr: note if it is due to the system stdout buffer filling then it could just be a timing/performance issue15:04
brtknrclarkb: could you translate that for me please?15:04
*** iurygregory has quit IRC15:05
clarkbstdout is provided to a process by the operating system. Depending on where the operating system is running you will get different performance and timing of the job and stdout handling. If the timing lines up on certain test instances such that the buffer is full and would require blocking you may see that only on some test runs15:06
*** roman_g has quit IRC15:07
clarkbbrtknr: https://opendev.org/openstack/nova/src/branch/master/nova/tests/fixtures.py#L206-L231 is the fixture definition and https://opendev.org/openstack/nova/src/branch/master/nova/test.py#L222-L223 is where it gets used15:09
clarkbbrtknr: what nova is doing there is replacing stdout provided by the operating system with one that is process internal so that the data can be captured without interleaving streams when stestr runs the tests in parallel15:10
openstackgerritMerged zuul/zuul master: Support nodes setting 'auto' python-path  https://review.opendev.org/68227515:18
*** Jeffrey4l has quit IRC15:19
*** ykarel is now known as ykarel|afk15:20
*** Jeffrey4l has joined #openstack-infra15:20
openstackgerritMerged zuul/zuul master: Annotate cancelJobInQueue with build id  https://review.opendev.org/68437315:25
brtknrclarkb: ive added an output stream fixture: https://review.opendev.org/68573515:26
*** gyee has joined #openstack-infra15:30
*** mattw4 has joined #openstack-infra15:33
*** eernst has joined #openstack-infra15:33
*** jcoufal has joined #openstack-infra15:34
*** eernst has quit IRC15:38
*** jamesmcarthur has joined #openstack-infra15:41
*** ykarel|afk is now known as ykarel|away15:44
*** pcaruana has joined #openstack-infra15:46
*** rpittau is now known as rpittau|afk15:48
*** tesseract has quit IRC15:55
*** xek_ is now known as xek15:59
*** roman_g has joined #openstack-infra15:59
clarkbbrtknr: see comments on the change16:05
*** ociuhandu has quit IRC16:13
*** xenos76 has quit IRC16:16
*** e0ne has quit IRC16:27
mordredinfra-root: I have arrived back in Stockholm and am now enjoying an evening of jetlag. (yay!) I expect to be back online properly working tomorrow. OR - in a few hours when I wake up senselessly at 1am or something16:32
openstackgerritFabien Boucher proposed zuul/zuul master: Gitlab - bootstrap the driver structure + Webhook support  https://review.opendev.org/68568216:35
*** whoami-rajat has joined #openstack-infra16:36
openstackgerritTristan Cacqueray proposed zuul/zuul master: Replace tox-py36 by tox-py37  https://review.opendev.org/68215816:41
*** eernst has joined #openstack-infra16:41
*** cgoncalves has quit IRC16:42
*** cgoncalves has joined #openstack-infra16:42
*** eernst has quit IRC16:43
*** cgoncalves has quit IRC16:43
*** jcoufal_ has joined #openstack-infra16:44
*** Lucas_Gray has quit IRC16:45
*** cgoncalves has joined #openstack-infra16:45
*** jpena is now known as jpena|off16:45
*** Lucas_Gray has joined #openstack-infra16:45
*** bdodd has quit IRC16:46
*** jaosorior has quit IRC16:46
*** jcoufal has quit IRC16:47
*** zbr|ruck is now known as zbr16:52
*** slaweq has quit IRC16:59
*** cgoncalves has quit IRC16:59
*** derekh has quit IRC17:00
*** cgoncalves has joined #openstack-infra17:00
*** slaweq has joined #openstack-infra17:01
*** ykarel|away has quit IRC17:06
*** igordc has joined #openstack-infra17:08
*** Lucas_Gray has quit IRC17:09
*** owalsh has quit IRC17:09
*** owalsh has joined #openstack-infra17:10
*** markvoelker has quit IRC17:17
*** ociuhandu has joined #openstack-infra17:19
*** markvoelker has joined #openstack-infra17:20
*** smarcet has quit IRC17:24
*** ykarel|away has joined #openstack-infra17:24
*** ociuhandu has quit IRC17:27
*** e0ne has joined #openstack-infra17:28
*** Lucas_Gray has joined #openstack-infra17:30
*** smarcet has joined #openstack-infra17:30
*** yamamoto has joined #openstack-infra17:31
*** yamamoto has quit IRC17:35
*** e0ne has quit IRC17:36
*** anteaya has joined #openstack-infra17:37
openstackgerritClark Boylan proposed zuul/zuul-jobs master: Use import_role on persistent-firewall  https://review.opendev.org/68576517:40
*** Lucas_Gray has quit IRC17:41
*** imacdonn has joined #openstack-infra17:50
Shrewsclarkb: i'm seeing a lot of connection errors to all providers in the nb01 logs. ever seen this before?17:55
Shrewsand zk session losses, too17:55
Shrewsmaybe a reboot is in order?17:55
clarkbShrews: no usually it is provider specific due to an outage or whatever. Not recalling anything affecting all providers17:56
Shrewsseems the networking stack is bonkers17:56
*** psachin has quit IRC17:56
clarkbdoesn't look like nodepool is running at all there17:56
Shrewsoh!17:56
Shrewsdied a couple of hours ago17:57
clarkbit OOM'd and ya nodepool-builder was selected to be culled17:57
ShrewsSep 30 14:08:35 nb01 kernel: [1714559.275405] Out of memory: Kill process 1676 (nodepool-builde) score 116 or sacrifice child17:58
Shrewsfun17:58
Shrews#status log Restarted nodepool-builder on nb01 that OOM'd18:00
openstackstatusShrews: finished logging18:00
Shrewswow. there are dib processes running on nb02 that have been running for several days18:02
Shrewsi don't know what that means, off hand18:03
Shrewslots of select() related errors in the logs18:05
Shrewshttps://www.irccloud.com/pastebin/ZjsLrHZ4/18:05
Shrewsthat's polling the dib process18:06
Shrewsnodepool has 761 subprocesses. yikes18:07
Shrewsnever seen this before18:08
Shrewskilling nodepool-builder to see how many die off18:11
*** diablo_rojo has quit IRC18:15
*** smarcet has quit IRC18:15
ShrewsStill a few hundred left. I'm going to reboot the host18:16
*** e0ne has joined #openstack-infra18:19
openstackgerritTristan Cacqueray proposed zuul/zuul master: WIP: Replace tox-py36 by tox-py37  https://review.opendev.org/68215818:19
*** gmann is now known as gmann_afk18:19
*** prometheanfire has quit IRC18:22
*** prometheanfire has joined #openstack-infra18:22
Shrewsaaaaand... it does not appear reachable   *sigh*18:23
AJaegerconfig-core, please review https://review.opendev.org/679806 https://review.opendev.org/684374 and https://review.opendev.org/68127018:23
clarkbShrews: it may be fscking18:24
Shrewstrue. 'server show' says it is running18:24
*** ricolin has quit IRC18:27
*** elod has quit IRC18:28
*** xenos76 has joined #openstack-infra18:33
*** smarcet has joined #openstack-infra18:34
*** pkopec has quit IRC18:43
*** jtomasek has quit IRC18:43
*** ociuhandu has joined #openstack-infra18:45
*** whoami-rajat has quit IRC18:46
*** jtomasek has joined #openstack-infra18:48
Shrewsclarkb: starting to think it's not coming back18:49
*** jtomasek has quit IRC18:50
*** jtomasek has joined #openstack-infra18:52
*** jtomasek has quit IRC18:52
clarkbthis is 02 or 01? you can check the console via the rax web ui and or hard reboot it via the api18:54
Shrews0218:54
*** ykarel|away has quit IRC18:54
openstackgerritJeremy Stanley proposed openstack/project-config master: Record vulnerability management teams used in SB  https://review.opendev.org/68577818:54
*** elod has joined #openstack-infra18:55
fungii can take a look in the console if you want18:56
Shrewsjust at it18:57
Shrewsand as soon as i open it, ssh now works18:57
Shrewsgo figure18:57
Shrewsmaybe it needed a proper poke18:58
fungiit was probably a long fsck18:59
fungithose instances have some largeish attached filesystems18:59
*** weshay has quit IRC18:59
fungioccasional boot delays on reboot would not shock me18:59
Shrews#status log Rebooted nb02 to deal with 100's of stray dib processes18:59
openstackstatusShrews: finished logging18:59
Shrewsfungi: i'll have to remember that and be more patient19:00
fungii think you can `sudo touch /fastboot` to skip autochecking filesystems which have been to long since their last fsck19:00
fungier, too long19:00
*** jamesmcarthur has quit IRC19:01
*** ociuhandu has quit IRC19:04
*** jamesmcarthur has joined #openstack-infra19:06
*** jbadiapa has joined #openstack-infra19:17
openstackgerritTristan Cacqueray proposed zuul/zuul master: WIP: Replace tox-py36 by tox-py37  https://review.opendev.org/68215819:21
*** ccamacho has joined #openstack-infra19:22
*** jamesmcarthur has quit IRC19:34
*** jamesmcarthur has joined #openstack-infra19:41
*** e0ne has quit IRC19:41
*** kopecmartin is now known as kopecmartin|off19:47
yoctozeptointeresting listing behavior: https://storage.gra1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_d59/681573/28/check/kolla-ansible-ubuntu-source-ceph-ipv6/d59c360/primary/logs/kolla/ceph/ - the 3rd file is a trap19:47
openstackgerritTristan Cacqueray proposed zuul/zuul master: WIP: Replace tox-py36 by tox-py37  https://review.opendev.org/68215819:51
fungiyoctozepto: oh, neat, i wonder what's caused that to be indexed as a relative filename?19:57
fungipresumably a bug in our index generator19:57
*** kjackal_v2 has quit IRC19:59
clarkbfungi: if you look at the page source it isn't any different than the other files. I think it has to do with the ::s20:02
clarkbya https://storage.gra1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_d59/681573/28/check/kolla-ansible-ubuntu-source-ceph-ipv6/d59c360/primary/logs/kolla/ceph/ceph-mon.fd%3A%3A3.txt.gz works fine20:03
openstackgerritTristan Cacqueray proposed zuul/zuul master: WIP: Replace tox-py36 by tox-py37  https://review.opendev.org/68215820:04
*** markvoelker has quit IRC20:04
fungiaha, so in our index we need to escape : in filenames i guess20:08
clarkbya I've got a quick patch I'm working on20:09
fungiawesome, thanks!20:09
timburkenow i want to see what swift's staticweb middleware does with a listing like that...20:11
*** smarcet has quit IRC20:13
*** e0ne has joined #openstack-infra20:14
clarkbits going to take way more time writing a test than it is to write the actual fix :)20:14
*** smarcet has joined #openstack-infra20:19
*** e0ne has quit IRC20:19
fungitimburke: probably "the right thing"[tm]20:20
fungi(i.e., just run file urls through something like urllib3's bog standard url escaper)20:20
clarkbfungi: timburke the issue is in the webbrowser20:26
clarkbso unless swift is rewriting object payloads I don't think swift can fix this20:26
openstackgerritClark Boylan proposed zuul/zuul-jobs master: Escape special filename characters in urls  https://review.opendev.org/68578520:27
clarkbfungi: yoctozepto ^ fyi20:27
*** jbadiapa has quit IRC20:27
openstackgerritTristan Cacqueray proposed zuul/zuul master: WIP: Replace tox-py36 by tox-py37  https://review.opendev.org/68215820:27
fungidepends on what you consider the problem to be as to whether that's in the browser or in the content ;)20:29
clarkbfungi: well the browser is follwoing what it has parsed to be a url then deciding the url is invalid as it doesn't understand the scheme20:30
fungi"browser can't handle unescaped :: in url paths" vs "index generator should be passing urls through a safe escaper"20:30
clarkbwell we definitely need to escape it20:30
clarkbI just mean that a server side middleware can't handle this20:31
*** Lucas_Gray has joined #openstack-infra20:31
clarkbunless it is scanning payload data and rewriting it to do the escaping20:31
fungiahh, i wasn't sure what swift's staticweb middleware was, thought maybe it had something akin to apache mod_autoindex20:31
timburkeit's similar, and yeah, we quote() it: https://github.com/openstack/swift/blob/2.22.0/swift/common/middleware/staticweb.py#L35120:32
fungiat least that's how i interpreted timburke's comment20:32
clarkboh I see. In this case we provide our own index so that should override20:32
timburkeyup20:32
timburkei just wanted to *make sure* swift quotes when it does generate pages ;-)20:33
fungii suspect the reason we're generating our own is that some of the object stores we upload to don't have that middleware as an option (due to not actually being swift)20:33
timburkei think there was also a desire to have accurate Last Modified columns for directory-like prefixes, which Swift can't do20:34
clarkbyup20:35
clarkb(you can't currently sort on that column though, but at least the data is acurate)20:35
openstackgerritJames E. Blair proposed zuul/zuul-registry master: Initial implementation  https://review.opendev.org/68395820:36
*** yamamoto has joined #openstack-infra20:38
*** jbadiapa has joined #openstack-infra20:46
*** ralonsoh has quit IRC20:49
*** ociuhandu has joined #openstack-infra20:49
*** pcaruana has quit IRC20:50
rm_workhey, with the SDK when I do: `user_cloud = admin_cloud.connect_as_project(user_project)`, and try to access `user_cloud.identity`, it fails because it can20:51
rm_work*can't determine the identity api version20:51
rm_workbut `admin_cloud.identity` works fine20:51
rm_worksomething isn't getting copied over?20:52
*** markvoelker has joined #openstack-infra20:52
*** xeno_os76_xyz has joined #openstack-infra20:52
*** signed8bit has joined #openstack-infra20:53
rm_workI'm not sure how it's determining the version in the original but not the new one20:54
*** irclogbot_3 has quit IRC20:55
*** xenos76 has quit IRC20:55
*** irclogbot_0 has joined #openstack-infra20:55
*** ccamacho has quit IRC20:57
*** irclogbot_0 has quit IRC21:01
*** Goneri has quit IRC21:01
*** irclogbot_1 has joined #openstack-infra21:01
openstackgerritPaul Belanger proposed openstack/project-config master: Add ansible-role-zuul-registry role for windmill  https://review.opendev.org/68579121:03
openstackgerritClark Boylan proposed opendev/elastic-recheck master: Add query for 1846093  https://review.opendev.org/68579221:05
clarkbmriedem: ^ any chance I can get a review on that one? trying to track down this issue21:05
clarkbrm_work: you might have better luck in the sdk channel21:06
*** irclogbot_1 has quit IRC21:07
rm_workclarkb: on one hand, yes, it's probably the correct channel. on the other hand, no one actually responds there usually and you folks actually use shade/sdk more than probably anyone else lol21:07
*** irclogbot_0 has joined #openstack-infra21:07
rm_workso yeah, if it's annoying to see that chatter here, I can try there instead21:08
mriedemrm_work: efried is over there21:08
clarkbwell we very rarely (at this point never?) interact with the clouds as admin users21:08
mriedemmordred is traveling otherwise he'd probably help21:08
rm_workyeah i'll just summarize my question there and see, but urgency is gone since i'm working around it21:08
rm_workfigured out how to just avoid using the identity client21:09
efriedrm_work: I'm listening (here or in -sdks)21:09
clarkbrm_work: in general I believe that you are supposed to hit the root of the keystone service which gives you back a json document with versions in it. Then sdk choses one based on whether or not you've supplied v3 auth details (domains essentially)21:09
rm_workhmm k so maybe the domain stuff isn't getting set properly for autodetection21:10
rm_workanywho, you're correct about the channel so i'll move over there21:10
mriedemclarkb: comment in our e-r patch21:10
mriedem*your21:10
*** jamesmcarthur has quit IRC21:11
openstackgerritClark Boylan proposed opendev/elastic-recheck master: Add query for 1846093  https://review.opendev.org/68579221:11
clarkbmriedem: done ^21:11
openstackgerritClark Boylan proposed zuul/zuul-jobs master: Escape special filename characters in urls  https://review.opendev.org/68578521:12
*** ociuhandu has quit IRC21:12
clarkbfungi: ^ fyi that needed a linting fix. Apparently vim overindents by default21:12
*** irclogbot_0 has quit IRC21:13
*** irclogbot_1 has joined #openstack-infra21:14
*** xek has quit IRC21:17
*** jamesmcarthur has joined #openstack-infra21:17
*** e0ne has joined #openstack-infra21:18
*** irclogbot_1 has quit IRC21:19
*** Lucas_Gray has quit IRC21:19
*** irclogbot_0 has joined #openstack-infra21:20
openstackgerritTristan Cacqueray proposed zuul/zuul master: Replace tox-py36 by tox-py37  https://review.opendev.org/68215821:22
*** e0ne has quit IRC21:24
*** irclogbot_0 has quit IRC21:25
openstackgerritJames E. Blair proposed zuul/zuul-registry master: Initial implementation  https://review.opendev.org/68395821:25
*** irclogbot_1 has joined #openstack-infra21:25
*** serim has joined #openstack-infra21:30
*** irclogbot_1 has quit IRC21:33
*** gfidente has quit IRC21:35
openstackgerritJeremy Stanley proposed zuul/zuul-website master: Link to instructions on vulnerability reporting  https://review.opendev.org/68579921:35
*** irclogbot_1 has joined #openstack-infra21:35
*** eernst has joined #openstack-infra21:37
*** irclogbot_1 has quit IRC21:41
*** irclogbot_1 has joined #openstack-infra21:42
*** eernst has quit IRC21:42
openstackgerritMerged opendev/elastic-recheck master: Add query for 1846093  https://review.opendev.org/68579221:43
*** irclogbot_1 has quit IRC21:47
*** irclogbot_3 has joined #openstack-infra21:48
*** yamamoto has quit IRC21:48
*** yamamoto has joined #openstack-infra21:48
*** smarcet has quit IRC21:50
openstackgerritMerged zuul/zuul-jobs master: Escape special filename characters in urls  https://review.opendev.org/68578521:52
corvusclarkb, fungi, mordred: i wonder if we should add libc/gcc stuff to the python-build image to save time in subsequent container image builds?21:52
*** irclogbot_3 has quit IRC21:53
*** irclogbot_1 has joined #openstack-infra21:54
*** goldyfruit_ has quit IRC21:56
*** irclogbot_1 has quit IRC21:59
openstackgerritJames E. Blair proposed zuul/zuul-registry master: Fix container image build  https://review.opendev.org/68580821:59
*** irclogbot_0 has joined #openstack-infra22:00
*** irclogbot_0 has quit IRC22:05
*** dave-mccowan has quit IRC22:05
*** irclogbot_0 has joined #openstack-infra22:06
*** adriant has joined #openstack-infra22:14
*** slaweq has quit IRC22:14
*** irclogbot_0 has quit IRC22:15
*** irclogbot_2 has joined #openstack-infra22:16
fungidoes it spend a significant amount of time (re)installing those?22:19
*** goldyfruit_ has joined #openstack-infra22:23
*** tosky has quit IRC22:24
*** rcernin has joined #openstack-infra22:24
*** munimeha1 has quit IRC22:28
*** yamamoto has quit IRC22:33
*** yamamoto has joined #openstack-infra22:34
*** xeno_os76_xyz has quit IRC22:36
*** harlowja has quit IRC22:36
*** mriedem has quit IRC22:44
*** mriedem has joined #openstack-infra22:45
*** jbadiapa has quit IRC22:53
ianwinfra-root: could i get some eyes on a rather mechanical change to up the log rotation times on mirror-update; the current roll-over makes it harder than it needs to be to correlate if we're having problems : https://review.opendev.org/68278622:56
clarkbianw: frickler had a couple comments in there you may want to address?22:58
clarkbI'm ok with it as is as both items could be addressed in a follow up22:58
ianwahh ok, sorry missed that, will do now22:58
ianwthere is something wrong with the dib gate, unsure yet if it's related to our mirrors ... that's why i'm poking :)22:59
openstackgerritIan Wienand proposed opendev/system-config master: logrotate: better rotation options  https://review.opendev.org/68278623:01
*** tkajinam has joined #openstack-infra23:02
*** smarcet has joined #openstack-infra23:06
*** mriedem has quit IRC23:08
*** aaronsheffield has quit IRC23:12
openstackgerritMerged zuul/zuul-jobs master: Use import_role on persistent-firewall  https://review.opendev.org/68576523:17
corvusfungi: yeah, for anything where we might need to build binary libs, etc, as part of installing deps.  which i think most/all of the zuul components do.  basically, it seems like a nice batteries-included thing for a 'python-builder' image to provide.23:21
*** goldyfruit_ has quit IRC23:21
fungiagreed, particularly if there are no wheels fitting the platform on which those images are based23:22
*** dchen has joined #openstack-infra23:22
fungithere's a way to add those as a "build" layer which can then be excluded from the requirements for the top layer, right? or did i imagine that?23:23
corvusfungi: yes -- python-builder is a published image but it's only purpose is to serve as a throwaway build layer for other "real" images.  so anything we install there does not end up in, say "zuul/zuul-scheduler"23:24
*** mattw4 has quit IRC23:26
corvuswhat i'm seing right now, building the zuul-registry image over and over, is that, while i have 'python-builder' cached locally, we start with that, and then, since the step where we install gcc, etc, is part of that throwaway build layer, that has to happen each time.23:26
corvusbut if it were already present in python-builder, we would save that time (at the cost of the python-builder image being larger)23:27
corvusbut since it's likely to be needed by a lot of python builds, i think we'd win playing those odds.23:27
corvusand since i seem to have an error on every single character of my dockerfile, i have a lot of time to talk about this23:28
corvus1 upgraded, 32 newly installed, 0 to remove and 11 not upgraded. Need to get 42.6 MB of archives. After this operation, 172 MB of additional disk space will be used.23:29
*** yamamoto has quit IRC23:30
*** yamamoto has joined #openstack-infra23:31
fungiyeah, sounds like a clear win then23:35
openstackgerritJames E. Blair proposed zuul/zuul-registry master: Fix container image build  https://review.opendev.org/68580823:40
*** exsdev0 has joined #openstack-infra23:47
*** exsdev has quit IRC23:47
*** exsdev0 is now known as exsdev23:47
*** sthussey has quit IRC23:50
*** jamesmcarthur has quit IRC23:57
*** jamesmcarthur has joined #openstack-infra23:58

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!