Thursday, 2019-08-08

*** kjackal has quit IRC00:00
*** tdasilva has quit IRC00:03
*** tdasilva has joined #openstack-infra00:03
*** yamamoto has quit IRC00:04
*** markvoelker has joined #openstack-infra00:09
*** slaweq_ has joined #openstack-infra00:11
*** slaweq_ has quit IRC00:15
openstackgerritJames E. Blair proposed zuul/zuul master: Refactor build page tabs  https://review.opendev.org/67523500:28
openstackgerritJames E. Blair proposed zuul/zuul master: Add permalinks to task detail popup  https://review.opendev.org/67523600:28
openstackgerritJames E. Blair proposed zuul/zuul master: Scroll log line anchor into view  https://review.opendev.org/67522000:29
*** ociuhandu has joined #openstack-infra00:30
*** ociuhandu has quit IRC00:35
*** betherly has joined #openstack-infra00:35
*** betherly has quit IRC00:39
*** markvoelker has quit IRC00:42
*** markvoelker has joined #openstack-infra00:43
*** gregoryo has joined #openstack-infra00:49
*** gyee has quit IRC00:49
*** ricolin has joined #openstack-infra01:03
*** tdasilva has quit IRC01:04
*** jamesmcarthur has joined #openstack-infra01:05
*** tdasilva has joined #openstack-infra01:05
*** jamesmcarthur has quit IRC01:11
*** smarcet has joined #openstack-infra01:14
*** jamesmcarthur has joined #openstack-infra01:14
*** jamesmcarthur has quit IRC01:23
*** jamesmcarthur has joined #openstack-infra01:26
*** diablo_rojo has joined #openstack-infra01:30
*** smarcet has quit IRC01:30
*** betherly has joined #openstack-infra01:30
*** betherly has quit IRC01:35
*** smarcet has joined #openstack-infra01:36
*** markvoelker has quit IRC01:37
openstackgerritJames E. Blair proposed zuul/zuul master: Refactor build page tabs  https://review.opendev.org/67523501:37
openstackgerritJames E. Blair proposed zuul/zuul master: Add permalinks to task detail popup  https://review.opendev.org/67523601:37
*** jamesmcarthur has quit IRC01:40
*** markvoelker has joined #openstack-infra01:44
*** rlandy|rover|bbl is now known as rlandy|rover01:45
*** yamamoto has joined #openstack-infra01:45
*** armax has quit IRC01:47
*** diablo_rojo has quit IRC01:48
*** rlandy|rover has quit IRC01:50
*** jhesketh has quit IRC01:53
*** bhavikdbavishi has joined #openstack-infra02:00
*** rh-jelabarre has quit IRC02:00
*** betherly has joined #openstack-infra02:01
*** bhavikdbavishi1 has joined #openstack-infra02:03
*** bhavikdbavishi has quit IRC02:04
*** bhavikdbavishi1 is now known as bhavikdbavishi02:04
*** jamesmcarthur has joined #openstack-infra02:04
*** betherly has quit IRC02:05
*** tdasilva has quit IRC02:06
*** tdasilva has joined #openstack-infra02:06
*** apetrich has quit IRC02:08
*** slaweq_ has joined #openstack-infra02:11
*** slaweq_ has quit IRC02:15
*** jamesmcarthur has quit IRC02:17
*** smarcet has left #openstack-infra02:22
*** jhesketh has joined #openstack-infra02:22
*** jamesmcarthur has joined #openstack-infra02:28
*** bhavikdbavishi has quit IRC02:32
*** betherly has joined #openstack-infra02:40
*** ramishra has joined #openstack-infra02:42
*** yamamoto has quit IRC02:49
*** yamamoto has joined #openstack-infra02:51
*** betherly has quit IRC02:53
*** adriant has quit IRC02:54
*** iokiwi has quit IRC02:54
*** tdasilva has quit IRC03:07
*** tdasilva has joined #openstack-infra03:07
*** betherly has joined #openstack-infra03:08
*** dchen has quit IRC03:08
*** dchen has joined #openstack-infra03:09
*** whoami-rajat has joined #openstack-infra03:12
*** tdasilva has quit IRC03:15
*** igordc has joined #openstack-infra03:16
*** tdasilva has joined #openstack-infra03:16
*** tdasilva has quit IRC03:19
*** tdasilva has joined #openstack-infra03:20
*** betherly has quit IRC03:21
*** iurygregory has quit IRC03:22
*** jhesketh has quit IRC03:29
*** yamamoto has quit IRC03:30
*** psachin has joined #openstack-infra03:32
*** bhavikdbavishi has joined #openstack-infra03:33
*** jhesketh has joined #openstack-infra03:36
*** yamamoto has joined #openstack-infra03:37
*** betherly has joined #openstack-infra03:39
*** betherly has quit IRC03:44
*** raukadah is now known as chkumar|ruck03:49
openstackgerritIan Wienand proposed opendev/system-config master: Add review-dev as a new backup client  https://review.opendev.org/67524303:56
*** jamesmcarthur has quit IRC04:01
*** jamesmcarthur has joined #openstack-infra04:05
*** tdasilva has quit IRC04:08
*** eharney has quit IRC04:08
*** tdasilva has joined #openstack-infra04:09
*** udesale has joined #openstack-infra04:09
*** jamesmcarthur has quit IRC04:09
*** jamesmcarthur has joined #openstack-infra04:10
*** slaweq_ has joined #openstack-infra04:11
*** slaweq_ has quit IRC04:15
*** jamesmcarthur has quit IRC04:19
*** eharney has joined #openstack-infra04:21
*** slittle1 has quit IRC04:22
*** jamesmcarthur has joined #openstack-infra04:23
*** slittle1 has joined #openstack-infra04:30
*** jamesmcarthur has quit IRC04:33
*** jamesmcarthur has joined #openstack-infra04:34
*** betherly has joined #openstack-infra04:38
*** jamesmcarthur has quit IRC04:42
*** jamesmcarthur has joined #openstack-infra04:42
*** betherly has quit IRC04:42
*** adriant has joined #openstack-infra04:45
*** ykarel|away has joined #openstack-infra04:46
*** jamesmcarthur has quit IRC04:48
*** ykarel|away is now known as ykarel04:51
*** yamamoto has quit IRC04:59
*** spsurya has joined #openstack-infra05:02
*** dave-mccowan has quit IRC05:03
*** ykarel has quit IRC05:19
*** ykarel has joined #openstack-infra05:20
*** fnordahl has quit IRC05:23
*** lennyb has quit IRC05:26
*** betherly has joined #openstack-infra05:29
*** yamamoto has joined #openstack-infra05:29
*** yamamoto has quit IRC05:30
*** Vadmacs has joined #openstack-infra05:34
*** betherly has quit IRC05:37
*** fnordahl has joined #openstack-infra05:38
*** n-saito has joined #openstack-infra05:38
*** yamamoto has joined #openstack-infra05:41
*** exsdev has joined #openstack-infra05:50
*** jaosorior has quit IRC05:56
*** igordc has quit IRC05:59
*** betherly has joined #openstack-infra06:01
*** ccamacho has quit IRC06:03
*** ykarel is now known as ykarel|afk06:04
*** betherly has quit IRC06:06
*** jaosorior has joined #openstack-infra06:09
*** tdasilva has quit IRC06:11
*** slaweq_ has joined #openstack-infra06:11
*** slaweq_ has quit IRC06:16
*** rcernin has quit IRC06:18
*** dpawlik has joined #openstack-infra06:22
*** ykarel|afk is now known as ykarel06:22
*** ralonsoh has joined #openstack-infra06:25
*** lennyb has joined #openstack-infra06:26
*** jaosorior has quit IRC06:29
*** tdasilva has joined #openstack-infra06:31
*** jbadiapa has joined #openstack-infra06:33
*** pgaxatte has joined #openstack-infra06:40
*** udesale has quit IRC06:41
*** n-saito has quit IRC06:42
*** udesale has joined #openstack-infra06:42
*** odicha has joined #openstack-infra06:53
AJaegerpabelanger: the non-voting jobs for python-tripleoclient and barbican are not on master branch - master looks fine for both repos06:57
*** pkopec has joined #openstack-infra06:58
*** slaweq_ has joined #openstack-infra06:59
*** betherly has joined #openstack-infra07:00
*** ykarel is now known as ykarel|pto07:00
*** ianychoi has quit IRC07:01
*** ianychoi has joined #openstack-infra07:01
*** jtomasek has joined #openstack-infra07:01
openstackgerritAndreas Jaeger proposed openstack/openstack-zuul-jobs master: Update ansible-lint to version 4  https://review.opendev.org/67525407:03
*** udesale has quit IRC07:03
*** ccamacho has joined #openstack-infra07:03
*** udesale has joined #openstack-infra07:03
openstackgerritAndreas Jaeger proposed openstack/project-config master: Bump ansible-lint to version 4  https://review.opendev.org/67525507:04
*** betherly has quit IRC07:05
*** ykarel|pto has quit IRC07:07
openstackgerritAndreas Jaeger proposed openstack/project-config master: Bump ansible-lint to version 4  https://review.opendev.org/67525507:10
openstackgerritAndreas Jaeger proposed openstack/openstack-zuul-jobs master: Update ansible-lint to version 4  https://review.opendev.org/67525407:11
*** ginopc has joined #openstack-infra07:12
*** jaosorior has joined #openstack-infra07:16
*** tosky has joined #openstack-infra07:17
*** betherly has joined #openstack-infra07:20
*** iurygregory has joined #openstack-infra07:22
*** ianychoi has quit IRC07:24
*** betherly has quit IRC07:25
*** ianychoi has joined #openstack-infra07:25
openstackgerritAndreas Jaeger proposed openstack/openstack-zuul-jobs master: Update ansible-lint to version 4  https://review.opendev.org/67525407:30
*** tesseract has joined #openstack-infra07:31
*** tdasilva has quit IRC07:32
*** tdasilva has joined #openstack-infra07:32
*** panda has quit IRC07:35
*** panda has joined #openstack-infra07:38
openstackgerritAndreas Jaeger proposed opendev/system-config master: Fix some ansible linting  https://review.opendev.org/67526007:43
*** dtantsur|afk is now known as dtantsur07:43
openstackgerritAndreas Jaeger proposed openstack/project-config master: Bump ansible-lint to version 4  https://review.opendev.org/67525507:43
*** apetrich has joined #openstack-infra07:44
openstackgerritAndreas Jaeger proposed openstack/openstack-zuul-jobs master: Update ansible-lint to version 4  https://review.opendev.org/67525407:44
*** kjackal has joined #openstack-infra07:50
*** lucasagomes has joined #openstack-infra07:56
AJaegerianw, dirk, any idea why https://review.opendev.org/#/c/667698/ fails now on openSUSE 15.1 and tumbleweed? https://logs.opendev.org/98/667698/9/gate/zuul-jobs-test-multinode-roles-opensuse-15/b9a73d4/ara-report/result/9eeead3e-2d1d-4913-8a60-bcd9fabc5521/ confuses me ;(07:56
*** ociuhandu has joined #openstack-infra07:56
AJaegerThis worked last night when zbr updated the change but fails now in gate and check ;(07:57
*** udesale has quit IRC07:57
*** udesale has joined #openstack-infra07:58
zbrAJaeger: morning! I will have a look on it in two minutes, i was writing a bug report.07:58
AJaegerzbr: good morning to you as well - thanks for checking07:59
*** kopecmartin|off is now known as kopecmartin07:59
*** tdasilva has quit IRC08:00
*** tdasilva has joined #openstack-infra08:01
*** rpittau|afk is now known as rpittau08:04
openstackgerritAndreas Jaeger proposed openstack/project-config master: Bump ansible-lint to version 4  https://review.opendev.org/67525508:05
openstackgerritAndreas Jaeger proposed openstack/project-config master: Bump ansible-lint to version 4  https://review.opendev.org/67525508:06
*** takamatsu has quit IRC08:06
openstackgerritAndreas Jaeger proposed openstack/openstack-zuul-jobs master: Update ansible-lint to version 4  https://review.opendev.org/67525408:07
AJaegerianw, could you check https://review.opendev.org/675260 , please?08:07
*** betherly has joined #openstack-infra08:22
*** dchen has quit IRC08:22
*** hrw has joined #openstack-infra08:24
hrwmorning08:24
zbrAJaeger: very weird outcome on suse, iptables_rules variable is defined but lacks stdout. i wonder when this can happen with ansible. i will add a debug before the task.08:24
zbri really doubt it has anything to do with our patch08:25
openstackgerritMark Meyer proposed zuul/zuul master: Rework a cache invalidation issue  https://review.opendev.org/67442508:26
hrwAJaeger, frickler: can you look again at https://review.opendev.org/#/c/671445/ patch? new flavour for linaro-london should now be properly configured.08:26
*** natalytvinova has joined #openstack-infra08:28
openstackgerritSorin Sbarnea proposed zuul/zuul-jobs master: DNM: Test iptables_rules on Suse  https://review.opendev.org/67527008:28
*** derekh has joined #openstack-infra08:30
*** tdasilva has quit IRC08:33
*** tdasilva has joined #openstack-infra08:33
*** Lucas_Gray has joined #openstack-infra08:34
AJaegerzbr: thanks - and I wonder why it worked yesterday08:35
AJaegerhrw: I'll wait for an admin to confirm the setup... thanks for update08:35
hrwAJaeger: thx08:36
*** jbadiapa has quit IRC08:36
AJaegerzbr: https://review.opendev.org/#/c/667698 is now succeeding and has pasted the SUSE tests...08:37
AJaegernot sure what kind of random failure that was ;/08:37
zbrAJaeger: i don't know... I never used suse ;)08:41
AJaeger:)08:42
zbrbut looking at the code, I would learn something new: register on a shell command could return success and not have a stdout key in the dict.08:42
zbrthere was an issue there, use of "include" instead of "include_task", but not a reason by itself for this issue.08:43
AJaegerinteresting08:44
dtantsurhi folks, is there a way to "recheck" a post job?08:50
dtantsurother than landing an empty patch08:50
*** Adri2000 has quit IRC08:53
AJaegerdtantsur: an admin can reenqueue it - best wait until the US wakes up...08:54
AJaegerBut yes, next patch ot the repo should do the same action - might be faster08:54
*** iurygregory has quit IRC08:58
*** roman_g has quit IRC08:59
*** Lucas_Gray has quit IRC09:00
openstackgerritMerged zuul/zuul-jobs master: Be consistent about spaces before and after vars  https://review.opendev.org/66769809:02
*** gregoryo has quit IRC09:02
*** iurygregory has joined #openstack-infra09:03
*** e0ne has joined #openstack-infra09:04
*** hrw has left #openstack-infra09:04
openstackgerritCarlos Goncalves proposed openstack/diskimage-builder master: Reduce yum-minimal based OS install size footprint  https://review.opendev.org/67232909:11
*** electrofelix has joined #openstack-infra09:12
*** roman_g has joined #openstack-infra09:21
*** Lucas_Gray has joined #openstack-infra09:28
*** tdasilva has quit IRC09:29
*** Lucas_Gray has quit IRC09:35
*** Lucas_Gray has joined #openstack-infra09:39
*** diga has joined #openstack-infra09:41
openstackgerritAndreas Jaeger proposed opendev/system-config master: Fix some ansible linting  https://review.opendev.org/67526009:51
AJaegerfrickler: updated with your suggestion ^09:51
*** dmellado has quit IRC09:53
*** dmellado has joined #openstack-infra09:55
*** Lucas_Gray has quit IRC10:01
*** udesale has quit IRC10:01
*** udesale has joined #openstack-infra10:01
*** udesale has quit IRC10:02
*** udesale has joined #openstack-infra10:03
*** Adri2000 has joined #openstack-infra10:08
*** jaosorior has quit IRC10:26
*** ociuhandu has quit IRC10:26
*** Lucas_Gray has joined #openstack-infra10:30
*** pgaxatte has quit IRC10:32
*** yamamoto has quit IRC10:33
chkumar|ruckHello #Infra10:33
*** kjackal has quit IRC10:33
chkumar|ruckwe are seeing Failed to retrieve repo file from https://trunk.rdoproject.org/centos7-master/current/delorean.repo after 10 retries on three jobs from yesterday and all these jobs are running on      centos-7-limestone-regionone-000985101810:34
chkumar|ruckhttps://logs.opendev.org/85/673985/1/gate/tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates/64088df/job-output.txt.gz#_2019-08-07_15_06_10_69045610:35
chkumar|ruckhttps://logs.opendev.org/65/669165/3/gate/tripleo-ci-centos-7-standalone/19bca80/job-output.txt.gz#_2019-08-08_09_16_07_47128710:35
chkumar|ruckhttps://logs.opendev.org/65/669165/3/gate/tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates/b20b7c7/job-output.txt.gz#_2019-08-08_09_13_29_75524610:35
chkumar|ruckbut requests on delorean.repo is working fine10:35
chkumar|ruckis there some issue going with limestone ?10:36
*** udesale has quit IRC10:38
*** iurygregory has quit IRC10:43
*** ociuhandu has joined #openstack-infra10:47
*** yamamoto has joined #openstack-infra10:51
*** tdasilva has joined #openstack-infra10:52
*** jaosorior has joined #openstack-infra11:10
openstackgerritManuel Torrinha proposed openstack/diskimage-builder master: Fixes Launchpad issue #1808359  https://review.opendev.org/67529411:12
*** tobiash has quit IRC11:18
*** tobiash has joined #openstack-infra11:20
*** tobiash has quit IRC11:24
*** tobiash has joined #openstack-infra11:26
*** ginopc has quit IRC11:29
*** lpetrut has joined #openstack-infra11:34
*** lpetrut has quit IRC11:49
*** jamesmcarthur has joined #openstack-infra11:51
*** jamesmcarthur has quit IRC11:51
*** jamesmcarthur has joined #openstack-infra11:51
*** kjackal has joined #openstack-infra11:53
*** rh-jelabarre has joined #openstack-infra11:53
*** smarcet has joined #openstack-infra11:56
*** jamesmcarthur has quit IRC12:02
*** jamesmcarthur has joined #openstack-infra12:04
*** smarcet has quit IRC12:05
*** ricolin_ has joined #openstack-infra12:05
*** ricolin has quit IRC12:07
*** rfolco has joined #openstack-infra12:12
*** jamesmcarthur has quit IRC12:21
*** jaosorior has quit IRC12:22
*** ccamacho has quit IRC12:24
*** rlandy has joined #openstack-infra12:28
*** udesale has joined #openstack-infra12:28
*** rlandy is now known as rlandy|rover12:28
*** iurygregory has joined #openstack-infra12:36
*** pgaxatte has joined #openstack-infra12:49
*** eharney has quit IRC12:51
*** jcoufal has joined #openstack-infra12:54
*** jamesmcarthur has joined #openstack-infra12:54
*** smarcet has joined #openstack-infra12:55
*** aaronsheffield has joined #openstack-infra12:56
*** Lucas_Gray has quit IRC12:57
*** Vadmacs has quit IRC12:57
*** Lucas_Gray has joined #openstack-infra12:58
*** jcoufal has quit IRC13:02
*** Lucas_Gray has quit IRC13:03
*** jcoufal has joined #openstack-infra13:06
*** natalytvinova has quit IRC13:07
*** natalytvinova has joined #openstack-infra13:10
*** ekultails has joined #openstack-infra13:14
*** mriedem has joined #openstack-infra13:16
*** Lucas_Gray has joined #openstack-infra13:17
*** ccamacho has joined #openstack-infra13:23
*** Vadmacs has joined #openstack-infra13:24
*** pkopec has quit IRC13:26
*** guoqiao has quit IRC13:27
*** slaweq_ is now known as slaweq13:29
*** eharney has joined #openstack-infra13:32
openstackgerritAndreas Jaeger proposed openstack/diskimage-builder master: Stop regex warning  https://review.opendev.org/67533713:33
*** stephenfin has quit IRC13:35
*** snierodz has quit IRC13:35
*** stephenfin has joined #openstack-infra13:36
*** ginopc has joined #openstack-infra13:37
*** snierodz has joined #openstack-infra13:38
*** yamamoto has quit IRC13:38
*** Diabelko has quit IRC13:39
openstackgerritJeff Liu proposed zuul/zuul-operator master: WIP: Add zuul-operator-functional-openshift job  https://review.opendev.org/67435513:44
clarkbchkumar|ruck: limestone is an ipv6 cloud and trunk.rdoproject has no AAAA record. This means all traffic to it must go through shared NAT. Possible that is overloaded by other requests to ipv4 only locations like dockerhub13:51
clarkblogan-: might have time to quickly checl that NAT is working propely13:51
*** ricolin_ is now known as ricolin13:55
donnydMaybe the additional stress of having two ipv6 clouds has something to do with it13:55
*** jcoufal has quit IRC13:55
*** smarcet has quit IRC13:55
clarkbEach ipv6 cloud is running separate NAT gateways though13:56
*** electrofelix has quit IRC13:57
donnydOh you mean NAT on the cloud side. I'm not sure what limestone is running, but I can show what it looks like from my end when the next job rolls around13:58
*** yamamoto has joined #openstack-infra13:59
*** yamamoto has quit IRC13:59
*** yamamoto has joined #openstack-infra14:00
clarkbyes NAT on the cloud side to translate from private cloud ipv4 addr to shared public ipv4 addr to talk to trunk.rdoproject.org via ipv414:00
*** dave-mccowan has joined #openstack-infra14:00
*** smarcet has joined #openstack-infra14:01
donnydYea that makes some sense, but honestly NAT doesn't really put a load on at least my edge FW14:01
clarkbThe problem iirc isnt due to performance load but due ti limited numbers of ports for connections on a single IP?14:03
clarkbI want to say udp is more problematic though as it isnt properly stateful and the kernel has to approximate14:03
*** yamamoto has quit IRC14:04
donnydmy state table shows around 4K in connections and I am currently doing 70 nodes.14:05
*** yamamoto has joined #openstack-infra14:05
donnydBut its possible14:05
openstackgerritJeff Liu proposed zuul/zuul-operator master: WIP: Add zuul-operator-functional-openshift job  https://review.opendev.org/67435514:05
*** liuyulong has quit IRC14:05
donnydI have been watching edge traffic and I am not sure this makes any difference at all, but the majority of outbound ipv4 connections are for dns and ntp14:08
*** jbadiapa has joined #openstack-infra14:11
clarkbchkumar|ruck: donnyd the other way that can break is if upstream has source based connection limits14:11
donnydtrue story14:13
donnydhttps://logs.opendev.org/05/675305/1/check/openstack-tox-py37/bdcc5a1/job-output.txt#_2019-08-08_13_54_29_38839414:14
donnydhttps://logs.opendev.org/24/675124/3/check/tripleo-ci-centos-7-undercloud-containers/6d8f838/job-output.txt#_2019-08-08_12_12_14_27716414:14
donnydfungi: it looks to me like moving back to ipv4 for the mirror has improved performance. Before these #'s were being measured in K14:15
donnydbut it still hasn't helped the jobs that timeout14:15
donnydBut i don't think that is isolated to the v6 clouds, seems to be all of them14:16
fungidonnyd: that's definitely worth digging into then!14:19
* fungi is stuck in meetings most of his morning, just fyi14:19
fungiclarkb: chkumar|ruck: donnyd: nat port ranges are often somewhat smallish but usually also configurable. and depending on implementation the nat may also hold out previously used ports for a little longer to account for late resets and fin/acks14:23
fungiso you usually need to find where the stats on the overload nat/pat pool are recorded14:24
*** pkopec has joined #openstack-infra14:24
AJaegerfungi, clarkb, I updated ansible-lint on some other projects (with similar blacklist) and add a few spaces to make ansible future-proof. Could you review https://review.opendev.org/675260 https://review.opendev.org/675254 and https://review.opendev.org/675255 , please?14:26
openstackgerritTristan Cacqueray proposed zuul/zuul master: web: refactor the errorsIds into the build action  https://review.opendev.org/67535014:28
*** pgaxatte has quit IRC14:29
*** dave-mccowan has quit IRC14:30
*** ociuhandu has quit IRC14:30
*** ociuhandu has joined #openstack-infra14:31
*** rlandy|rover is now known as rlandy|rover|mtg14:31
*** jcoufal has joined #openstack-infra14:35
*** ociuhandu has quit IRC14:35
openstackgerritJeff Liu proposed zuul/zuul-operator master: WIP: Add zuul-operator-functional-openshift job  https://review.opendev.org/67435514:41
*** rascasoft has quit IRC14:49
*** rascasoft has joined #openstack-infra14:52
chkumar|ruckclarkb: donnyd thanks, I will take a look, thanks!14:56
*** diablo_rojo has joined #openstack-infra14:58
corvusinfra-root: i think zuul-preview can be used as an open proxy, so i've shut the service down until we can track that down15:02
fungithanks for the heads up, corvus15:02
fungii can sort of half-imagine how that might work15:03
corvusyeah, i don't think it's a fundamental design flaw, i think we probably just missed a setting15:03
*** ociuhandu has joined #openstack-infra15:04
corvus#status log shut down zuul-preview and added zp01 to emergency disable due to a suspected misconfiguration allowing it to act as an open proxy15:06
openstackstatuscorvus: finished logging15:06
mordredcorvus: oh, that's not joyous15:08
*** psachin has quit IRC15:15
*** jbadiapa has quit IRC15:18
*** rlandy|rover|mtg is now known as rlandy|rover15:20
openstackgerritMerged zuul/zuul master: Scroll log line anchor into view  https://review.opendev.org/67522015:20
*** Lucas_Gray has quit IRC15:27
*** odicha has quit IRC15:29
*** jbadiapa has joined #openstack-infra15:30
*** spsurya has quit IRC15:34
*** jbadiapa has quit IRC15:35
*** eernst has joined #openstack-infra15:40
fungicorvus: did you see evidence of it being exploited to that end? or just happened to stumble across a possible hole there?15:40
corvusfungi: the log looked like it was being used15:42
fungithanks, figured that might be it15:42
*** gyee has joined #openstack-infra15:43
*** ricolin_ has joined #openstack-infra15:43
mordredcorvus: I'm curious to see what we missed15:43
corvusmordred: i'm not working on that right now, so if you want to dig into it, be my guest15:44
corvus(i'm heads down in prepping for the log change)15:44
*** eernst has quit IRC15:44
*** armax has joined #openstack-infra15:44
mordredcorvus: totes. I'll put it on my list15:45
mordredalthough probably also won't work on it today15:45
*** ricolin_ has quit IRC15:45
*** ricolin has quit IRC15:46
*** ricolin_ has joined #openstack-infra15:46
*** ricolin_ is now known as ricolin15:47
*** sthussey has joined #openstack-infra15:47
*** natalytvinova has quit IRC15:50
*** diga has quit IRC15:57
*** lucasagomes has quit IRC16:03
*** Lucas_Gray has joined #openstack-infra16:06
*** iurygregory has quit IRC16:08
*** smarcet has quit IRC16:15
*** beekneemech has joined #openstack-infra16:16
*** e0ne has quit IRC16:16
*** beekneemech has quit IRC16:16
*** yamamoto has quit IRC16:18
openstackgerritClark Boylan proposed zuul/zuul master: Consistent handling of ansible venv install dirs  https://review.opendev.org/67540316:26
openstackgerritMerged zuul/zuul master: Render console in js  https://review.opendev.org/67436816:28
*** mattw4 has joined #openstack-infra16:29
*** smarcet has joined #openstack-infra16:30
openstackgerritMerged zuul/zuul master: Usability tweaks for the build page console  https://review.opendev.org/67514716:34
*** rpittau is now known as rpittau|afk16:36
*** diablo_rojo has quit IRC16:37
*** mattw4 has quit IRC16:39
*** mattw4 has joined #openstack-infra16:39
clarkbAJaeger: hrw's https://review.opendev.org/#/c/671445/2 lgtm if you want to be second reviewer now (catching up on scrollback now)16:40
clarkbjohnsom: we do our best to mirror or proxy cache major sources of upstream data. All of the distros we build images for have mirrors except for gentoo, we proxy cache pypi, dockerhub, rubygems, npm, and more16:41
*** chkumar|ruck is now known as raukadah16:42
openstackgerritJeff Liu proposed zuul/zuul-operator master: WIP: Add zuul-operator-functional-openshift job  https://review.opendev.org/67435516:42
johnsomclarkb Yeah, we have used the distro/package mirrors for a while. It's the DIB pip install that appears to have not been using the mirrors. I'm giving it a go today.16:43
fungiwhen analyzing traffic for fortnebula which wasn't going through the mirror server, the hit spots seemed to be dockerhub, pypi and rdo16:44
fungier, hot spots16:44
fungiwe don't currently mirror or cache rdo so that might be something to look at i guess16:45
logan-o/ .. i'll check for nat contention, but i'm a little skeptical since it seems like only trunk.rdoproject.org is coming up unreachable. i looked thru a batch of recent failures and didn't come up with other jobs failing due to network16:45
clarkbfungi: we do cache all of those in the mirror node so maybe we need to advertise that more16:45
clarkbfungi: we do actually proxy cache rdo16:45
fungioh! right on16:45
fungiso it all boils down to jobs not using the mirror host16:46
clarkblogan-: thanks, and that is a good point. If that is behind CDN could be that the node servicing limestone was down explaining why it hit limestone16:46
logan-going to do some more digging, im thinking of starting by setting up some long running tcpdumps at the network nodes to watch trunk.rdoproject.org traffic and see what we can correlate with job failures there16:47
*** tosky has quit IRC16:49
*** Lucas_Gray has quit IRC16:51
*** yamamoto has joined #openstack-infra16:51
fungithe other thing we can do is enumerate some instance names or ip addresses which reached out to rdo (or pypi or dockerhub) and use zuul logs to track those back to specific jobs16:51
*** Lucas_Gray has joined #openstack-infra16:51
fungiobviously from the failures we know at least some of the jobs in question16:52
clarkbcorvus: the open proxy thing being used is probably good evidence we should avoid open proxies for job data caching16:52
fungiclarkb: i guess you mean it validates our decision not to do generic open proxies on the mirror hosts16:53
clarkbfungi: ya16:53
mordredyeah - I think we're all in agreement about not using open proxies :)16:53
fungii concur16:53
funginot that i thought we really needed any fresh evidence that people scour the internet in search of open proxies to exploit16:54
openstackgerritMatt Riedemann proposed opendev/elastic-recheck master: Add query for nova functional test race bug 1839515  https://review.opendev.org/67540816:54
openstackbug 1839515 in OpenStack Compute (nova) "Weird functional test failures hitting neutron API in unrelated resize flows since 8/5" [Undecided,New] https://launchpad.net/bugs/183951516:54
*** armax has quit IRC16:56
*** dtantsur is now known as dtantsur|afk16:57
openstackgerritJames E. Blair proposed zuul/zuul master: Add option to report build page  https://review.opendev.org/67540916:57
openstackgerritJames E. Blair proposed zuul/zuul master: Add release note for Pagure driver  https://review.opendev.org/67541016:57
openstackgerritJames E. Blair proposed zuul/zuul master: Move admin-rules setting in tenants doc  https://review.opendev.org/67541116:57
openstackgerritJames E. Blair proposed zuul/zuul master: Make auth docs more boring  https://review.opendev.org/67541216:57
openstackgerritJames E. Blair proposed zuul/zuul master: Don't capitalize Token in docs  https://review.opendev.org/67541316:57
*** tdasilva has quit IRC16:59
openstackgerritMatt Riedemann proposed opendev/elastic-recheck master: Add query for nova functional test race bug 1839515  https://review.opendev.org/67540817:00
openstackbug 1839515 in OpenStack Compute (nova) "Weird functional test failures hitting neutron API in unrelated resize flows since 8/5" [High,Confirmed] https://launchpad.net/bugs/183951517:00
*** derekh has quit IRC17:00
*** roman_g has quit IRC17:04
*** yamamoto has quit IRC17:04
*** Lucas_Gray has quit IRC17:05
*** e0ne has joined #openstack-infra17:05
openstackgerritBen Nemec proposed openstack/devstack-gate master: WIP: Use OSCaaS to speed up devstack runs  https://review.opendev.org/67541417:06
*** roman_g has joined #openstack-infra17:06
openstackgerritMerged openstack/project-config master: Linaro London: use new bigger flavour  https://review.opendev.org/67144517:06
*** udesale has quit IRC17:08
*** sshnaidm is now known as sshnaidm|off17:10
*** markvoelker has quit IRC17:12
*** igordc has joined #openstack-infra17:13
*** ociuhandu has quit IRC17:14
*** ricolin_ has joined #openstack-infra17:16
clarkbinfra-root I'd like to approve https://review.opendev.org/#/c/674930/ now to improve gitea sshd logging. Any reason for me to not do that now?17:19
*** ricolin has quit IRC17:19
fungii've approved it17:20
clarkbperfect17:20
fungii guess we should just make sure sshd keeps running17:20
fungiotherwise replication will cease happening17:20
clarkbdocker-compose will restart it, but ya we should check it is running after the restart and then check docker logs works for it and that replication is working17:20
fungiand it might not be outwardly apparent for some time17:20
*** jamesmcarthur has quit IRC17:22
openstackgerritClark Boylan proposed zuul/zuul master: Consistent handling of ansible venv install dirs  https://review.opendev.org/67540317:25
*** markvoelker has joined #openstack-infra17:26
*** tesseract has quit IRC17:27
openstackgerritMerged opendev/elastic-recheck master: Add query for nova functional test race bug 1839515  https://review.opendev.org/67540817:28
openstackbug 1839515 in OpenStack Compute (nova) "Weird functional test failures hitting neutron API in unrelated resize flows since 8/5" [High,Confirmed] https://launchpad.net/bugs/183951517:28
*** kopecmartin is now known as kopecmartin|off17:31
*** ricolin_ is now known as ricolin17:32
*** jamesmcarthur has joined #openstack-infra17:33
*** e0ne has quit IRC17:35
*** ociuhandu has joined #openstack-infra17:36
openstackgerritJeremy Stanley proposed opendev/system-config master: Replace wiki-dev02 with wiki-dev03  https://review.opendev.org/67542517:37
*** ginopc has quit IRC17:38
clarkbfungi: did that board transparency mailing list get disabled after it gummed up the server?17:38
*** armax has joined #openstack-infra17:39
*** ociuhandu has quit IRC17:40
fungiclarkb: not yet, i have it on my to do list to ask some folks who i need to get an okay from17:41
fungier, that is, to ask some osf staff whether we need the board to say it's okay to retire/archive it17:42
fungisince it was for an osf board of directors working group17:42
*** ianychoi has quit IRC17:43
clarkbconsidering it was the transparency ml seems extra prudent to do that17:44
fungiyes17:45
fungiwell, i mean, technically it still *is* the transparency ml, it just hasn't had a post in >4 years now17:46
johnsomclarkb I have another one of those DNS failures: https://logs.opendev.org/81/584681/26/check/octavia-v2-act-stdby-dsvm-scenario-two-node/ed101de/job-output.txt.gz#_2019-08-08_16_52_24_31977817:46
clarkbjohnsom: I got as far as "it largely a centos 7 problem but happens on all the cloud regions" the other day looking at that17:46
clarkbthat job was xenial though so falls outside of that17:47
*** ianychoi has joined #openstack-infra17:47
*** panda has quit IRC17:53
*** ralonsoh has quit IRC17:54
*** ianychoi_ has joined #openstack-infra17:54
*** panda has joined #openstack-infra17:54
*** ianychoi_ has quit IRC17:55
*** ianychoi has quit IRC17:58
*** smarcet has quit IRC18:06
*** psachin has joined #openstack-infra18:07
*** psachin has quit IRC18:11
openstackgerritMerged opendev/system-config master: Collect gitea sshd logs  https://review.opendev.org/67493018:12
mriedemclarkb: are we still running devstack nodes in vms with 8vcpu and 8gb ram?18:16
clarkbmriedem: yes18:16
mriedemi haven't been able to have a stable ubuntu 18.04 devstack running on master for at least a month, tried both py27 and py36, running with 1 API_WORKER per service, and still have random crashes of nova/neutron/cinder services18:17
clarkbmriedem: do your intsances have swap? we create swap on our test VMs and devstack produces a result that needs swap with 8GB of memory now :(18:18
mriedemhmm18:20
*** smarcet has joined #openstack-infra18:20
mriedemprobably not18:21
mriedemyeah the flavor i'm using for the vm (v1-standard-8 from vexxhost) doesn't have swap18:21
clarkbI brought it up on hte ml but other than you pointing out the change to stop running cinder backup I haven't seen anything more18:21
clarkb(I think this is actually a fairly major problem with openstack right now, it affects our ability to test the software and ofr people to use it locally easily)18:22
mriedemoh an in this case, i disable basically all of cinder, etcd, tempest and horizon18:22
mriedemyeah something in the last 1-2 months and i'm unable to use devstack18:22
mriedemthis is my fake driver local conf i'm using today (except NUMBER_FAKE_NOVA_COMPUTE=2) http://paste.openstack.org/show/755667/18:23
mordredclarkb, mriedem: also edge use cases probably like it when our footprint isn't super high18:23
clarkbmriedem: to confirm this is the problem I would check dmesg for OOMKiller output18:23
clarkbmriedem: you can also check top then hit > to sort by memory percentage instead of cpu18:24
clarkbthat might quickly identify problems if memory is the problem18:24
clarkbdisabling tempets and horizon is probably not much of a win there since tempest doesn't run by default (it is an install) and horizon runs out of apache which is running anyway18:25
mriedemno OOMKiller in dmesg18:25
clarkbcinder should have an impact though18:25
clarkbmriedem: also free -m to see memory usage at a very high level18:25
mriedemi do have a crash file from apport but never tried to parse one of those things18:25
mriedem$ free -mh18:26
mriedem              total        used        free      shared  buff/cache   available18:26
mriedemMem:           7.8G        2.9G        1.7G        2.8M        3.1G        4.6G18:26
mriedemSwap:            0B          0B          0B18:26
mriedemof course i need to restart some of these dead services, doing that18:26
*** igordc has quit IRC18:28
mriedemheh, "openstack volume service list" > public endpoint for volumev3 service in RegionOne region not found18:29
mriedemthat's not good18:29
smcginnismriedem: Didn't you say you disabled cinder?18:30
mriedemoh right :)18:30
*** jamesmcarthur has quit IRC18:30
mriedemusing the fake driver so i disabled cinder, but the other day had a 'normal' devstack with libvirt18:30
*** kjackal has quit IRC18:31
mriedemwell restarting nova and neutron then18:31
mriedemok those are back up18:32
mriedem$ free -mh18:32
mriedem              total        used        free      shared  buff/cache   available18:32
mriedemMem:           7.8G        3.2G        1.5G        2.8M        3.1G        4.3G18:32
mriedemno big change there18:32
clarkbyou may need to use the services so that they grow in footprint18:33
mriedemnothing seems to be chewing up memory, mysqld is pretty stable at the top but just 5%18:34
*** bobh has joined #openstack-infra18:35
clarkbfwiw I was deinitely able to induce crashy behavior via running tempest against a default devstack cloud on an 8GB memory host with no swap18:35
clarkbthat washow I discovered swap was required when spinning up the fn cloud18:35
mriedemok i can try creating a test guest, that's when things crashed earlier18:36
* mriedem is also on a call about standing up a private dev/test cloud18:36
fungialso your instance flavor doesn't need a swap partition, you can just create a swapfile in the rootfs instead but i strongly recommend preallocating all the blocks and not just making it a sparse file18:39
fungiotherwise you get different random crashes when you run out of space on the filesystem and then try to swap18:39
fungibut yeah, given c-vol was the biggest offender for memory utilization in devstack jobs recently, if you're not running that you likely don't need any swap18:40
*** jamesmcarthur has joined #openstack-infra18:42
clarkbsomeone wasasking about similar stuffin #opemstack-qa yesterday and I think tosky was helping them18:44
clarkbI was busy enjoying aday at the coast though18:44
fungiwell-deserved18:46
*** rfolco has quit IRC18:51
*** pkopec has quit IRC18:52
mriedemof course now i can't crash the damn thing18:55
*** armstrong has joined #openstack-infra18:56
openstackgerritSaul Wold proposed openstack/project-config master: starlingx: add zuul-jobs repo  https://review.opendev.org/67544518:58
*** rfolco has joined #openstack-infra19:00
*** armax has quit IRC19:01
*** armax has joined #openstack-infra19:02
*** ramishra has quit IRC19:02
fungimriedem: if only our gate jobs were so lucky?19:06
*** bobh has quit IRC19:06
mriedemfungi: do we get frequent crashes in the gate?19:06
mriedemif so, are we detecting and auto-retrying the job or something?19:07
*** smarcet has quit IRC19:08
openstackgerritMerged zuul/zuul master: Don't always show expansion option on build console  https://review.opendev.org/67516319:08
openstackgerritMerged zuul/zuul master: Adjust results headings in build console page  https://review.opendev.org/67520319:09
fungidepends on what you mean by crashes, but devstack-managed services not starting or dying? seems like that happens fairly often due to bugs and races in openstack19:09
clarkbmriedem: fungi the major behavior we seem to notice is job slowness and timeouts19:10
clarkbif you then track that back to dstat or stackviz you see that swap is being hit hard and that causes things to break19:11
clarkbI would not be surprised if many test failures themselves were a result of that too19:11
fungirecall rather a lot of nondeterministic devstack job failures due to some service not starting19:11
fungior possibly stopping abruptly19:11
mriedemfungi: yeah i meant the services crashes, but agree with clarkb19:12
mriedemif services were crashing *during* tests that would be pretty obvious in job logs19:12
mriedemlike back when we had OOMKillers19:12
mriedemand the slowness and such related to my patch to disable c-bak19:12
clarkbif we removed swap we would see ^19:12
fungiyeah19:12
mriedemnow that you mention it, the other day when i had a normal aio devstack with libvirt, watching top and such cinder-volume was very heay19:15
mriedem*heavy19:15
mriedemseemed like a periodic was running continually or something19:16
*** e0ne has joined #openstack-infra19:16
mriedemsmcginnis: any recent changes to cinder-volume that would be running periodics heavy.19:16
mriedem?19:16
smcginnismriedem: Nothing that I'm aware of, but I haven't been reviewing the new changes as much as I was.19:18
mriedemheh you're still top dog though https://www.stackalytics.com/report/contribution/cinder/6019:18
*** bobh has joined #openstack-infra19:19
mriedemhmm https://github.com/openstack/cinder/commit/b0279f2080ea54a123a9249c7a1a8fb027909ce2#diff-724c53b8cd580ae533b4fb301df7c3e919:20
mriedemseems ok...defaults are 60 which matches the default in oslo.service19:22
fungimaybe tasks are taking more than 60 seconds to complete and so they're piling up over time?19:26
mriedemwe've seen c-vol get locked up on lvchange for more than 60 seconds19:26
mriedemleading to starving other requests to lvchange resulting in MessagingTimeouts from c-vol to c-api19:27
mriedemand 500s out of the cinder API19:27
mriedemlet's see what happens if i try to create 100 servers19:28
openstackgerritRonelle Landy proposed opendev/elastic-recheck master: Add query for UNAUTHORIZED error when pulling containers  https://review.opendev.org/67545319:29
clarkbcloudnull: ^ did adding the reauths help that at all?19:31
donnydmriedem: This is why i just had to set max_concurrent_builds=2 on FN19:31
mriedemdonnyd: in this devstack i'm using the fake nova virt driver19:31
mriedemand just 2 fake computes19:31
mriedemi mean, they are real nova-compute services in systemd, but running a fake driver19:31
donnydis c-vol using lvm?19:32
mriedemi've disabled cinder in this vm19:32
mriedemin the gate yes19:32
cloudnullclarkb I think it did, but we're adding more logs to the process to try and track things down .19:32
mriedemusing the fake driver is useful for testing things like evacuate on a single node devstack19:32
mriedemcreating 100 fake servers was ok, nova-conductor hammers CPU but that's not news19:35
*** Romik has joined #openstack-infra19:36
*** jcoufal has quit IRC19:40
*** auristor has quit IRC19:42
mriedemclarkb: looking at https://logs.opendev.org/17/675117/1/check/tempest-full-py3/6cee1e5/controller/logs/stackviz/#/stdin/timeline?test=tempest.scenario.test_security_groups_basic_ops.TestSecurityGroupsBasicOps.test_cross_tenant_traffic reminded me of something,19:42
*** noonedeadpunk has quit IRC19:42
*** noonedeadpunk has joined #openstack-infra19:43
mriedemtempest-full jobs run api tests concurrently (nproc/2) but scenario tests are run in serial,19:43
mriedemwhich is why the scenario tests are all stacked up on worker 019:43
mriedemi had a patch to try and run scenario tests with nproc/4 but it got snagged up19:43
mriedemhttps://review.opendev.org/#/c/650300/19:44
funginow that the release meeting is over i'm going to go find very late lunch, but will return in a while. if anyone wants to approve 675425 i can continue testing that when i get back19:45
*** auristor has joined #openstack-infra19:46
*** dciabrin_ is now known as dciabrin19:53
clarkbwe have sshd logs on gitea01 now19:53
clarkbthe containers were restarted as expected on all gitea hosts19:55
clarkbthere are no queued replication events on review0119:55
clarkbI'm going to trigger replication of system-config and see that it doesn't fail19:55
*** bobh has quit IRC19:56
*** bobh has joined #openstack-infra19:56
clarkband that seems to have worked19:56
*** panda has quit IRC19:57
*** eharney has quit IRC19:58
*** panda has joined #openstack-infra19:59
*** jbadiapa has joined #openstack-infra20:00
*** bobh has quit IRC20:01
openstackgerritJames E. Blair proposed zuul/zuul master: Add permalinks to task detail popup  https://review.opendev.org/67523620:01
*** noonedeadpunk has quit IRC20:03
openstackgerritTristan Cacqueray proposed zuul/zuul master: web: extract pure functions from the TaskOutput component  https://review.opendev.org/67546020:08
weshayhopefully elastic-recheck has the resources for https://review.opendev.org/#/c/675453/20:09
weshayplease review if you have a momemt20:09
openstackgerritJames E. Blair proposed zuul/zuul master: Hide "root" variable in job web page  https://review.opendev.org/67546120:11
clarkbweshay: is that query specific enough? will the name of that container only show up on failures for the unauthorized docker hub requests?20:12
clarkbmaybe we can check for 401 messages instead or similar?20:12
weshayclarkb it's just the base name of all the containers20:12
*** jbadiapa has quit IRC20:12
weshaybut if you think we should change it.. that's fine too20:12
clarkbright so wouldn't it show up in normal jobs because it is used everywhere?20:12
*** Vadmacs has quit IRC20:13
weshaydon't understand that last comment20:13
openstackgerritMerged zuul/zuul master: Add option to report build page  https://review.opendev.org/67540920:13
clarkbweshay: ideally e-r queries will only match on jobs that have failed due to the bug tied to the query. If we are searching for the name of an image that is used in every job are we going to match every job that runs regardless of whether or not it failed?20:14
weshayhttp://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Name%5C%5C%5C%22%3A%5C%5C%5C%22tripleomaster%2Fcentos-binary%5C%22%20AND%20tags%3A%5C%22console%5C%2220:15
weshaywe can refine it a bit more..20:15
weshayI get your point now20:15
*** smarcet has joined #openstack-infra20:16
*** allanb has joined #openstack-infra20:16
clarkb'"code":"UNAUTHORIZED","message":"authentication required"' looks like the string we may want to match on20:17
openstackgerritwes hayutin proposed opendev/elastic-recheck master: Add query for UNAUTHORIZED error when pulling containers  https://review.opendev.org/67545320:23
*** diablo_rojo has joined #openstack-infra20:24
weshayclarkb reworking it now.. thanks for the quick ping back..20:26
*** betherly has quit IRC20:27
*** diablo_rojo__ has joined #openstack-infra20:30
weshayrlandy|rover we're good w/ the latest?20:31
rlandy|roverweshay: ack - thanks for updating that20:31
*** jtomasek has quit IRC20:31
weshayclarkb ok.. it should hit only errors while pulling now20:31
openstackgerritMerged zuul/zuul master: Use wait for empty update queue before accepting merges  https://review.opendev.org/67503920:33
*** diablo_rojo has quit IRC20:33
clarkbmriedem: did you want to review that one too?20:35
*** e0ne has quit IRC20:37
mriedemyeah i'll take a look20:46
*** rascasoft has quit IRC20:47
*** bhavikdbavishi has quit IRC20:47
*** betherly has joined #openstack-infra20:48
*** rascasoft has joined #openstack-infra20:49
mriedemdone20:51
*** diablo_rojo__ is now known as diablo_rojo20:52
*** betherly has quit IRC20:52
*** jcoufal has joined #openstack-infra20:53
openstackgerritMerged zuul/zuul master: Refactor build page tabs  https://review.opendev.org/67523520:54
*** guoqiao has joined #openstack-infra20:57
*** kjackal has joined #openstack-infra20:57
*** tjgresha has joined #openstack-infra20:58
*** noonedeadpunk has joined #openstack-infra20:59
*** whoami-rajat has quit IRC21:01
openstackgerritMerged opendev/elastic-recheck master: Add query for UNAUTHORIZED error when pulling containers  https://review.opendev.org/67545321:04
*** rm_work has quit IRC21:05
*** rm_work has joined #openstack-infra21:05
*** diablo_rojo has quit IRC21:07
weshaythanks clarkb!21:07
openstackgerritJeff Liu proposed zuul/zuul-operator master: WIP: Add zuul-operator-functional-openshift job  https://review.opendev.org/67435521:07
*** diablo_rojo has joined #openstack-infra21:08
*** betherly has joined #openstack-infra21:08
*** betherly has quit IRC21:12
*** igordc has joined #openstack-infra21:13
fungiokay, back and pitching in^W^Wtrying not to break stuff21:15
fungiclarkb: did you happen to notice if the staggering/serialization of gitea restarts happened as designed?21:15
*** diablo_rojo__ has joined #openstack-infra21:16
clarkbfungi: I did not. The default resolution for docker ps -a's output resulted in "about an hour ago"21:16
openstackgerritMichael Johnson proposed openstack/diskimage-builder master: Fix the pypi element for multiple mirror URLs  https://review.opendev.org/67546821:16
fungigotta love the dedication to precision in the docker community21:17
fungihold my beer, i'll debug this21:17
clarkbI expect the ansible logs would probably show if they serialized properly21:17
*** rfolco has quit IRC21:18
*** diablo_rojo has quit IRC21:19
*** jamesmcarthur has quit IRC21:19
openstackgerritMerged zuul/zuul master: Add permalinks to task detail popup  https://review.opendev.org/67523621:23
*** diablo_rojo__ is now known as diablo_rojo21:25
*** betherly has joined #openstack-infra21:28
*** smarcet has quit IRC21:31
*** betherly has quit IRC21:33
*** smarcet has joined #openstack-infra21:35
*** markvoelker has quit IRC21:36
*** betherly has joined #openstack-infra21:36
*** jtomasek has joined #openstack-infra21:36
openstackgerritJames E. Blair proposed zuul/zuul master: web: refactor the errorsIds into the build action  https://review.opendev.org/67535021:40
*** betherly has quit IRC21:40
*** smarcet has quit IRC21:43
openstackgerritClark Boylan proposed zuul/zuul master: Improve functionality and docs around ansible installation  https://review.opendev.org/67540321:49
*** markvoelker has joined #openstack-infra21:53
*** armstrong has quit IRC21:55
*** betherly has joined #openstack-infra21:57
*** markvoelker has quit IRC21:57
*** betherly has quit IRC22:01
*** jcoufal has quit IRC22:02
*** panda has quit IRC22:03
*** kjackal has quit IRC22:03
*** kei-ichi has quit IRC22:03
*** panda has joined #openstack-infra22:03
*** mattw4 has quit IRC22:04
*** mattw4 has joined #openstack-infra22:04
openstackgerritJames E. Blair proposed zuul/zuul master: web: refactor the errorsIds into the build action  https://review.opendev.org/67535022:05
*** mattw4 has quit IRC22:14
*** mattw4 has joined #openstack-infra22:14
*** slaweq has quit IRC22:15
openstackgerritJames E. Blair proposed zuul/zuul master: Correctly identify failed tasks  https://review.opendev.org/67548822:16
openstackgerritJames E. Blair proposed zuul/zuul master: Refactor task result detection  https://review.opendev.org/67548922:16
*** panda has quit IRC22:16
corvusclarkb, fungi: where are we on the ara report error?22:18
corvusi can switch to working on that now, but i lost track of what was going on with ansible venvs etc22:18
*** betherly has joined #openstack-infra22:18
fungii'm so distracted/disconnected i didn't know there was an ara report error22:19
* fungi apologizes more for his cluelessness22:19
corvusin the swift logs job, the static ara report generation is failing22:19
fungiahh, i didn't know but am just about freed up and can take a look at errors22:20
corvuswe have no debugging info other than a cryptic message that says it failed22:20
*** panda has joined #openstack-infra22:20
corvusclarkb was looking into whether we maybe just needed to upgrade ara?22:20
openstackgerritTristan Cacqueray proposed zuul/zuul master: web: add buildset page  https://review.opendev.org/63007922:20
fungioh, right, i thought there were some patches about upgrading ara in our ansible envs on the executors22:21
fungiwhich led to the discussion in #zuul about preinstalled vs zuul-managed ansible envs i guess22:21
corvusyep, but that should be independent22:21
*** betherly has quit IRC22:23
*** whoami-rajat has joined #openstack-infra22:24
corvusit looks like 0.16.5 is installed on ze01 now22:27
corvusi need to find when that happened to see if we have a run of ara afterwords22:28
corvusthe timestamp on the executable file is aug 7 08:3022:28
corvusmy last recheck of swift jobs was before that.  so the next step is to recheck 674359 and see if it's fixed.22:29
fungimy guess is yes22:29
fungiclarkb has likely fixed it at this point22:29
corvusi am not going to bet against clarkb having fixed something :)22:30
fungii'm not going to risk angering the clarkb as he fixes more things than i break, and that's saying something22:30
*** ociuhandu has joined #openstack-infra22:30
openstackgerritMerged zuul/zuul master: Provide buildset.uuid in /builds API result  https://review.opendev.org/67475922:30
clarkbwell I'm hoping that the 0.16.5 installs are the fix22:31
*** ociuhandu has quit IRC22:35
openstackgerritTristan Cacqueray proposed zuul/zuul master: web: link the buildset page from the build  https://review.opendev.org/67549322:38
*** mattw4 has quit IRC22:44
openstackgerritMichael Johnson proposed openstack/diskimage-builder master: Fix the pypi element for multiple mirror URLs  https://review.opendev.org/67546822:53
*** rcernin has joined #openstack-infra22:53
*** mattw4 has joined #openstack-infra22:54
corvusclarkb, fungi: still seeing "non-zero return code"22:54
corvushttp://paste.openstack.org/show/755671/22:55
corvusthat's all we get22:55
openstackgerritTristan Cacqueray proposed zuul/zuul master: web: extract pure functions from the TaskOutput component  https://review.opendev.org/67546022:55
corvusi'll see if i can repro manually22:58
*** dchen has joined #openstack-infra23:00
*** betherly has joined #openstack-infra23:01
*** dchen has quit IRC23:01
*** dchen has joined #openstack-infra23:02
*** ricolin_ has joined #openstack-infra23:02
*** mattw4 has quit IRC23:03
corvushrm.  a simple repro case seems to work fine23:04
*** mattw4 has joined #openstack-infra23:04
*** ricolin has quit IRC23:05
*** diablo_rojo has quit IRC23:05
*** diablo_rojo has joined #openstack-infra23:06
*** betherly has quit IRC23:06
corvusi rsynced a build directory, activated the 2.8 venv, ran ara generate html, and it worked23:06
*** aaronsheffield has quit IRC23:06
dmsimardhi o/ where can I see that failure happening ?23:08
corvusdmsimard: that paste http://paste.openstack.org/show/755671/ is all we have right now23:08
dmsimardcorvus: what job is that from ?23:08
corvusi can't actually link to the logs yet because i won't know the url until the rest of the jobs have finished23:09
corvusbut it probably wouldn't be useful anyway23:09
dmsimardcurious to see it run on the executor23:10
corvusdmsimard: but it should be pretty much the same as the last buildset of https://review.opendev.org/67435923:10
dmsimardcorvus: looking23:10
corvusdmsimard: what are you looking for, maybe i can help23:10
corvusi mean, i have a streaming log window open, so i can copy/paste more of that if you want23:11
*** slaweq has joined #openstack-infra23:11
corvushere's the whole post-run playbook: http://paste.openstack.org/show/755672/23:12
dmsimardcorvus: I'm looking to see if there is a more verbose version of that error hidden somewhere :D23:13
corvusunfortunately, it's not in the job-output.json since that misses the last play -- however, i may have it in the copy of the build directory i just saved, let me see23:14
*** slaweq has quit IRC23:15
*** ekultails has quit IRC23:16
corvusthat's disappointing -- it doesn't have any more helpful info: http://paste.openstack.org/show/755673/23:16
dmsimardI found a random occurrence on ze01 and there is suspisciously really no more output http://paste.openstack.org/show/755674/23:20
corvusyeah, i'm wondering if we're going to need to set keep and verbose on the executors to continue debugging23:21
dmsimardhttp://paste.openstack.org/show/755675/ is that occurrence with more context23:22
*** armax has quit IRC23:22
dmsimardI wonder if command munges output more than shell would23:23
*** eernst has joined #openstack-infra23:23
corvusdmsimard: do you want me to get started on keep+verbose?23:23
corvusdmsimard: could this just be a command not found error?23:26
corvushrm... we check that the executable is found by bash... but then we use command and not bash to run it.23:27
*** eernst has quit IRC23:27
corvusso i would think that if it's not in the path, we would get the "not installed" error, but still there's a subtle difference there23:28
dmsimardthe role does some assertions before getting to that step but we should improve them to be more verbose23:28
dmsimardit provides "/usr/local/bin/ara"23:28
corvusclarkb, dmsimard: amusingly, that is installed *outside* the ansible venvs, but it's still ara 0.16.523:32
dmsimardthere is some amount of static files copied over so it kind of half ran https://storage.bhs1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/logs_59/674359/1/check/nodepool-zuul-functional/8d3bac7/ara-report/static/css/23:32
corvusso there are two ara installations in play -- the one in the zuul-ansible venv which is where the callback is run from, and the one that's used to generate the html, which is system-wide on the executor.23:33
corvus(which, tbh, is actually pretty typical of an ara installation)23:33
corvus(so i don't think that's bad, just something to be aware of.  anyway, they are both 0.16.5)23:34
corvusdmsimard: the incompatability with ansible that was fixed in the 0.16.4 -> 0.16.5 upgrade -- does that affect the callbacks in ansible 2.8?23:34
corvusi'm starting to wonder if we're still using the old versions of the callbacks because we haven't restarted the executor since the upgrade (which would happen if the ara callbacks are part of what the executor copies in place on startup)23:37
corvusif that's the case, then i think the next step would be to restart the executors and recheck23:37
dmsimardpretty positive a new ansible-playbook invocation would run off of an updated package23:38
corvusdmsimard: not if the package wasn't updated because it's part of what the executor copies in place23:39
dmsimard0.16.5 was mostly an administrative release23:39
corvushrm.  someone said something about fixing a 2.8 compatability issue23:39
dmsimardindeed, in 0.16.423:39
corvusoh, that may be it then, we may have been running <16.4 even23:40
corvuslemme see if i can confirm this real quick23:40
dmsimardthe regression was https://github.com/ansible-community/ara/issues/4623:40
dmsimardbut that was devel back then23:41
dmsimardtrying to see if I can reproduce the html generation issue -- the sqlite database from the executor for a build would be helpful23:42
corvusdmsimard: root@ze09:/root/corvus-test has an entire build dir right before it was deleted23:45
dmsimardsweet23:45
corvusit looks like my theory about the ara callback being old isn't holding up -- that job configured the callbacks as /var/lib/zuul/ansible/2.8/zuul/ansible/callback:/usr/lib/zuul/ansible/2.8/lib/python3.5/site-packages/ara/plugins  and that has the timestamp of the upgrade to 0.16.523:46
corvus(the first directory is the one that gets copied, the second is the venv which got updated, that has the timestamp of the 0.16.5 upgrade)23:47
corvusoh hey, i got it to reproduce23:48
corvusearlier i was running ara from the venv, not the system install23:49
corvushttp://paste.openstack.org/show/755678/23:49
dmsimardinteresting!23:49
*** mattw4 has quit IRC23:52
dmsimardthere be dragons in that general vicinity, I much prefer your implementation of the file tree view :)23:53
*** betherly has joined #openstack-infra23:54
corvusthis is curious though -- if there's no "create_file" method, how does this work at all -- doesn't the ansible zuul use static ara generation?23:57
*** sthussey has quit IRC23:57
dmsimardthat method is coming from pyfakefs, looks like we have version pyfakefs==3.5.823:57
*** diablo_rojo has quit IRC23:57
corvusoh gotcha23:57
corvus3.5.8 is in the venv, but 3.3 is on the system23:58
corvusso presumably we need to upgrade the system version23:58
dmsimardwhy suddenly though ?23:58
dmsimard3.5.8 is apparently not even the latest version, there was a 3.6.0 back in june23:58
corvusthis is the first time (in about 8 months) that we've exercised the static generation in opendev23:59
*** betherly has quit IRC23:59
corvusso the last time we ran it, everything was probably old enough that it worked23:59
dmsimardI can try with an older version perhaps, hang on23:59
corvussince then, we've upgraded ara, but not pyfakefs.  everybody else using ara static generation probably has upgraded both packages23:59

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!