| *** kjackal has quit IRC | 00:00 | |
| *** tdasilva has quit IRC | 00:03 | |
| *** tdasilva has joined #openstack-infra | 00:03 | |
| *** yamamoto has quit IRC | 00:04 | |
| *** markvoelker has joined #openstack-infra | 00:09 | |
| *** slaweq_ has joined #openstack-infra | 00:11 | |
| *** slaweq_ has quit IRC | 00:15 | |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Refactor build page tabs https://review.opendev.org/675235 | 00:28 |
|---|---|---|
| openstackgerrit | James E. Blair proposed zuul/zuul master: Add permalinks to task detail popup https://review.opendev.org/675236 | 00:28 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Scroll log line anchor into view https://review.opendev.org/675220 | 00:29 |
| *** ociuhandu has joined #openstack-infra | 00:30 | |
| *** ociuhandu has quit IRC | 00:35 | |
| *** betherly has joined #openstack-infra | 00:35 | |
| *** betherly has quit IRC | 00:39 | |
| *** markvoelker has quit IRC | 00:42 | |
| *** markvoelker has joined #openstack-infra | 00:43 | |
| *** gregoryo has joined #openstack-infra | 00:49 | |
| *** gyee has quit IRC | 00:49 | |
| *** ricolin has joined #openstack-infra | 01:03 | |
| *** tdasilva has quit IRC | 01:04 | |
| *** jamesmcarthur has joined #openstack-infra | 01:05 | |
| *** tdasilva has joined #openstack-infra | 01:05 | |
| *** jamesmcarthur has quit IRC | 01:11 | |
| *** smarcet has joined #openstack-infra | 01:14 | |
| *** jamesmcarthur has joined #openstack-infra | 01:14 | |
| *** jamesmcarthur has quit IRC | 01:23 | |
| *** jamesmcarthur has joined #openstack-infra | 01:26 | |
| *** diablo_rojo has joined #openstack-infra | 01:30 | |
| *** smarcet has quit IRC | 01:30 | |
| *** betherly has joined #openstack-infra | 01:30 | |
| *** betherly has quit IRC | 01:35 | |
| *** smarcet has joined #openstack-infra | 01:36 | |
| *** markvoelker has quit IRC | 01:37 | |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Refactor build page tabs https://review.opendev.org/675235 | 01:37 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Add permalinks to task detail popup https://review.opendev.org/675236 | 01:37 |
| *** jamesmcarthur has quit IRC | 01:40 | |
| *** markvoelker has joined #openstack-infra | 01:44 | |
| *** rlandy|rover|bbl is now known as rlandy|rover | 01:45 | |
| *** yamamoto has joined #openstack-infra | 01:45 | |
| *** armax has quit IRC | 01:47 | |
| *** diablo_rojo has quit IRC | 01:48 | |
| *** rlandy|rover has quit IRC | 01:50 | |
| *** jhesketh has quit IRC | 01:53 | |
| *** bhavikdbavishi has joined #openstack-infra | 02:00 | |
| *** rh-jelabarre has quit IRC | 02:00 | |
| *** betherly has joined #openstack-infra | 02:01 | |
| *** bhavikdbavishi1 has joined #openstack-infra | 02:03 | |
| *** bhavikdbavishi has quit IRC | 02:04 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 02:04 | |
| *** jamesmcarthur has joined #openstack-infra | 02:04 | |
| *** betherly has quit IRC | 02:05 | |
| *** tdasilva has quit IRC | 02:06 | |
| *** tdasilva has joined #openstack-infra | 02:06 | |
| *** apetrich has quit IRC | 02:08 | |
| *** slaweq_ has joined #openstack-infra | 02:11 | |
| *** slaweq_ has quit IRC | 02:15 | |
| *** jamesmcarthur has quit IRC | 02:17 | |
| *** smarcet has left #openstack-infra | 02:22 | |
| *** jhesketh has joined #openstack-infra | 02:22 | |
| *** jamesmcarthur has joined #openstack-infra | 02:28 | |
| *** bhavikdbavishi has quit IRC | 02:32 | |
| *** betherly has joined #openstack-infra | 02:40 | |
| *** ramishra has joined #openstack-infra | 02:42 | |
| *** yamamoto has quit IRC | 02:49 | |
| *** yamamoto has joined #openstack-infra | 02:51 | |
| *** betherly has quit IRC | 02:53 | |
| *** adriant has quit IRC | 02:54 | |
| *** iokiwi has quit IRC | 02:54 | |
| *** tdasilva has quit IRC | 03:07 | |
| *** tdasilva has joined #openstack-infra | 03:07 | |
| *** betherly has joined #openstack-infra | 03:08 | |
| *** dchen has quit IRC | 03:08 | |
| *** dchen has joined #openstack-infra | 03:09 | |
| *** whoami-rajat has joined #openstack-infra | 03:12 | |
| *** tdasilva has quit IRC | 03:15 | |
| *** igordc has joined #openstack-infra | 03:16 | |
| *** tdasilva has joined #openstack-infra | 03:16 | |
| *** tdasilva has quit IRC | 03:19 | |
| *** tdasilva has joined #openstack-infra | 03:20 | |
| *** betherly has quit IRC | 03:21 | |
| *** iurygregory has quit IRC | 03:22 | |
| *** jhesketh has quit IRC | 03:29 | |
| *** yamamoto has quit IRC | 03:30 | |
| *** psachin has joined #openstack-infra | 03:32 | |
| *** bhavikdbavishi has joined #openstack-infra | 03:33 | |
| *** jhesketh has joined #openstack-infra | 03:36 | |
| *** yamamoto has joined #openstack-infra | 03:37 | |
| *** betherly has joined #openstack-infra | 03:39 | |
| *** betherly has quit IRC | 03:44 | |
| *** raukadah is now known as chkumar|ruck | 03:49 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Add review-dev as a new backup client https://review.opendev.org/675243 | 03:56 |
| *** jamesmcarthur has quit IRC | 04:01 | |
| *** jamesmcarthur has joined #openstack-infra | 04:05 | |
| *** tdasilva has quit IRC | 04:08 | |
| *** eharney has quit IRC | 04:08 | |
| *** tdasilva has joined #openstack-infra | 04:09 | |
| *** udesale has joined #openstack-infra | 04:09 | |
| *** jamesmcarthur has quit IRC | 04:09 | |
| *** jamesmcarthur has joined #openstack-infra | 04:10 | |
| *** slaweq_ has joined #openstack-infra | 04:11 | |
| *** slaweq_ has quit IRC | 04:15 | |
| *** jamesmcarthur has quit IRC | 04:19 | |
| *** eharney has joined #openstack-infra | 04:21 | |
| *** slittle1 has quit IRC | 04:22 | |
| *** jamesmcarthur has joined #openstack-infra | 04:23 | |
| *** slittle1 has joined #openstack-infra | 04:30 | |
| *** jamesmcarthur has quit IRC | 04:33 | |
| *** jamesmcarthur has joined #openstack-infra | 04:34 | |
| *** betherly has joined #openstack-infra | 04:38 | |
| *** jamesmcarthur has quit IRC | 04:42 | |
| *** jamesmcarthur has joined #openstack-infra | 04:42 | |
| *** betherly has quit IRC | 04:42 | |
| *** adriant has joined #openstack-infra | 04:45 | |
| *** ykarel|away has joined #openstack-infra | 04:46 | |
| *** jamesmcarthur has quit IRC | 04:48 | |
| *** ykarel|away is now known as ykarel | 04:51 | |
| *** yamamoto has quit IRC | 04:59 | |
| *** spsurya has joined #openstack-infra | 05:02 | |
| *** dave-mccowan has quit IRC | 05:03 | |
| *** ykarel has quit IRC | 05:19 | |
| *** ykarel has joined #openstack-infra | 05:20 | |
| *** fnordahl has quit IRC | 05:23 | |
| *** lennyb has quit IRC | 05:26 | |
| *** betherly has joined #openstack-infra | 05:29 | |
| *** yamamoto has joined #openstack-infra | 05:29 | |
| *** yamamoto has quit IRC | 05:30 | |
| *** Vadmacs has joined #openstack-infra | 05:34 | |
| *** betherly has quit IRC | 05:37 | |
| *** fnordahl has joined #openstack-infra | 05:38 | |
| *** n-saito has joined #openstack-infra | 05:38 | |
| *** yamamoto has joined #openstack-infra | 05:41 | |
| *** exsdev has joined #openstack-infra | 05:50 | |
| *** jaosorior has quit IRC | 05:56 | |
| *** igordc has quit IRC | 05:59 | |
| *** betherly has joined #openstack-infra | 06:01 | |
| *** ccamacho has quit IRC | 06:03 | |
| *** ykarel is now known as ykarel|afk | 06:04 | |
| *** betherly has quit IRC | 06:06 | |
| *** jaosorior has joined #openstack-infra | 06:09 | |
| *** tdasilva has quit IRC | 06:11 | |
| *** slaweq_ has joined #openstack-infra | 06:11 | |
| *** slaweq_ has quit IRC | 06:16 | |
| *** rcernin has quit IRC | 06:18 | |
| *** dpawlik has joined #openstack-infra | 06:22 | |
| *** ykarel|afk is now known as ykarel | 06:22 | |
| *** ralonsoh has joined #openstack-infra | 06:25 | |
| *** lennyb has joined #openstack-infra | 06:26 | |
| *** jaosorior has quit IRC | 06:29 | |
| *** tdasilva has joined #openstack-infra | 06:31 | |
| *** jbadiapa has joined #openstack-infra | 06:33 | |
| *** pgaxatte has joined #openstack-infra | 06:40 | |
| *** udesale has quit IRC | 06:41 | |
| *** n-saito has quit IRC | 06:42 | |
| *** udesale has joined #openstack-infra | 06:42 | |
| *** odicha has joined #openstack-infra | 06:53 | |
| AJaeger | pabelanger: the non-voting jobs for python-tripleoclient and barbican are not on master branch - master looks fine for both repos | 06:57 |
| *** pkopec has joined #openstack-infra | 06:58 | |
| *** slaweq_ has joined #openstack-infra | 06:59 | |
| *** betherly has joined #openstack-infra | 07:00 | |
| *** ykarel is now known as ykarel|pto | 07:00 | |
| *** ianychoi has quit IRC | 07:01 | |
| *** ianychoi has joined #openstack-infra | 07:01 | |
| *** jtomasek has joined #openstack-infra | 07:01 | |
| openstackgerrit | Andreas Jaeger proposed openstack/openstack-zuul-jobs master: Update ansible-lint to version 4 https://review.opendev.org/675254 | 07:03 |
| *** udesale has quit IRC | 07:03 | |
| *** ccamacho has joined #openstack-infra | 07:03 | |
| *** udesale has joined #openstack-infra | 07:03 | |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Bump ansible-lint to version 4 https://review.opendev.org/675255 | 07:04 |
| *** betherly has quit IRC | 07:05 | |
| *** ykarel|pto has quit IRC | 07:07 | |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Bump ansible-lint to version 4 https://review.opendev.org/675255 | 07:10 |
| openstackgerrit | Andreas Jaeger proposed openstack/openstack-zuul-jobs master: Update ansible-lint to version 4 https://review.opendev.org/675254 | 07:11 |
| *** ginopc has joined #openstack-infra | 07:12 | |
| *** jaosorior has joined #openstack-infra | 07:16 | |
| *** tosky has joined #openstack-infra | 07:17 | |
| *** betherly has joined #openstack-infra | 07:20 | |
| *** iurygregory has joined #openstack-infra | 07:22 | |
| *** ianychoi has quit IRC | 07:24 | |
| *** betherly has quit IRC | 07:25 | |
| *** ianychoi has joined #openstack-infra | 07:25 | |
| openstackgerrit | Andreas Jaeger proposed openstack/openstack-zuul-jobs master: Update ansible-lint to version 4 https://review.opendev.org/675254 | 07:30 |
| *** tesseract has joined #openstack-infra | 07:31 | |
| *** tdasilva has quit IRC | 07:32 | |
| *** tdasilva has joined #openstack-infra | 07:32 | |
| *** panda has quit IRC | 07:35 | |
| *** panda has joined #openstack-infra | 07:38 | |
| openstackgerrit | Andreas Jaeger proposed opendev/system-config master: Fix some ansible linting https://review.opendev.org/675260 | 07:43 |
| *** dtantsur|afk is now known as dtantsur | 07:43 | |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Bump ansible-lint to version 4 https://review.opendev.org/675255 | 07:43 |
| *** apetrich has joined #openstack-infra | 07:44 | |
| openstackgerrit | Andreas Jaeger proposed openstack/openstack-zuul-jobs master: Update ansible-lint to version 4 https://review.opendev.org/675254 | 07:44 |
| *** kjackal has joined #openstack-infra | 07:50 | |
| *** lucasagomes has joined #openstack-infra | 07:56 | |
| AJaeger | ianw, dirk, any idea why https://review.opendev.org/#/c/667698/ fails now on openSUSE 15.1 and tumbleweed? https://logs.opendev.org/98/667698/9/gate/zuul-jobs-test-multinode-roles-opensuse-15/b9a73d4/ara-report/result/9eeead3e-2d1d-4913-8a60-bcd9fabc5521/ confuses me ;( | 07:56 |
| *** ociuhandu has joined #openstack-infra | 07:56 | |
| AJaeger | This worked last night when zbr updated the change but fails now in gate and check ;( | 07:57 |
| *** udesale has quit IRC | 07:57 | |
| *** udesale has joined #openstack-infra | 07:58 | |
| zbr | AJaeger: morning! I will have a look on it in two minutes, i was writing a bug report. | 07:58 |
| AJaeger | zbr: good morning to you as well - thanks for checking | 07:59 |
| *** kopecmartin|off is now known as kopecmartin | 07:59 | |
| *** tdasilva has quit IRC | 08:00 | |
| *** tdasilva has joined #openstack-infra | 08:01 | |
| *** rpittau|afk is now known as rpittau | 08:04 | |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Bump ansible-lint to version 4 https://review.opendev.org/675255 | 08:05 |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Bump ansible-lint to version 4 https://review.opendev.org/675255 | 08:06 |
| *** takamatsu has quit IRC | 08:06 | |
| openstackgerrit | Andreas Jaeger proposed openstack/openstack-zuul-jobs master: Update ansible-lint to version 4 https://review.opendev.org/675254 | 08:07 |
| AJaeger | ianw, could you check https://review.opendev.org/675260 , please? | 08:07 |
| *** betherly has joined #openstack-infra | 08:22 | |
| *** dchen has quit IRC | 08:22 | |
| *** hrw has joined #openstack-infra | 08:24 | |
| hrw | morning | 08:24 |
| zbr | AJaeger: very weird outcome on suse, iptables_rules variable is defined but lacks stdout. i wonder when this can happen with ansible. i will add a debug before the task. | 08:24 |
| zbr | i really doubt it has anything to do with our patch | 08:25 |
| openstackgerrit | Mark Meyer proposed zuul/zuul master: Rework a cache invalidation issue https://review.opendev.org/674425 | 08:26 |
| hrw | AJaeger, frickler: can you look again at https://review.opendev.org/#/c/671445/ patch? new flavour for linaro-london should now be properly configured. | 08:26 |
| *** natalytvinova has joined #openstack-infra | 08:28 | |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: DNM: Test iptables_rules on Suse https://review.opendev.org/675270 | 08:28 |
| *** derekh has joined #openstack-infra | 08:30 | |
| *** tdasilva has quit IRC | 08:33 | |
| *** tdasilva has joined #openstack-infra | 08:33 | |
| *** Lucas_Gray has joined #openstack-infra | 08:34 | |
| AJaeger | zbr: thanks - and I wonder why it worked yesterday | 08:35 |
| AJaeger | hrw: I'll wait for an admin to confirm the setup... thanks for update | 08:35 |
| hrw | AJaeger: thx | 08:36 |
| *** jbadiapa has quit IRC | 08:36 | |
| AJaeger | zbr: https://review.opendev.org/#/c/667698 is now succeeding and has pasted the SUSE tests... | 08:37 |
| AJaeger | not sure what kind of random failure that was ;/ | 08:37 |
| zbr | AJaeger: i don't know... I never used suse ;) | 08:41 |
| AJaeger | :) | 08:42 |
| zbr | but looking at the code, I would learn something new: register on a shell command could return success and not have a stdout key in the dict. | 08:42 |
| zbr | there was an issue there, use of "include" instead of "include_task", but not a reason by itself for this issue. | 08:43 |
| AJaeger | interesting | 08:44 |
| dtantsur | hi folks, is there a way to "recheck" a post job? | 08:50 |
| dtantsur | other than landing an empty patch | 08:50 |
| *** Adri2000 has quit IRC | 08:53 | |
| AJaeger | dtantsur: an admin can reenqueue it - best wait until the US wakes up... | 08:54 |
| AJaeger | But yes, next patch ot the repo should do the same action - might be faster | 08:54 |
| *** iurygregory has quit IRC | 08:58 | |
| *** roman_g has quit IRC | 08:59 | |
| *** Lucas_Gray has quit IRC | 09:00 | |
| openstackgerrit | Merged zuul/zuul-jobs master: Be consistent about spaces before and after vars https://review.opendev.org/667698 | 09:02 |
| *** gregoryo has quit IRC | 09:02 | |
| *** iurygregory has joined #openstack-infra | 09:03 | |
| *** e0ne has joined #openstack-infra | 09:04 | |
| *** hrw has left #openstack-infra | 09:04 | |
| openstackgerrit | Carlos Goncalves proposed openstack/diskimage-builder master: Reduce yum-minimal based OS install size footprint https://review.opendev.org/672329 | 09:11 |
| *** electrofelix has joined #openstack-infra | 09:12 | |
| *** roman_g has joined #openstack-infra | 09:21 | |
| *** Lucas_Gray has joined #openstack-infra | 09:28 | |
| *** tdasilva has quit IRC | 09:29 | |
| *** Lucas_Gray has quit IRC | 09:35 | |
| *** Lucas_Gray has joined #openstack-infra | 09:39 | |
| *** diga has joined #openstack-infra | 09:41 | |
| openstackgerrit | Andreas Jaeger proposed opendev/system-config master: Fix some ansible linting https://review.opendev.org/675260 | 09:51 |
| AJaeger | frickler: updated with your suggestion ^ | 09:51 |
| *** dmellado has quit IRC | 09:53 | |
| *** dmellado has joined #openstack-infra | 09:55 | |
| *** Lucas_Gray has quit IRC | 10:01 | |
| *** udesale has quit IRC | 10:01 | |
| *** udesale has joined #openstack-infra | 10:01 | |
| *** udesale has quit IRC | 10:02 | |
| *** udesale has joined #openstack-infra | 10:03 | |
| *** Adri2000 has joined #openstack-infra | 10:08 | |
| *** jaosorior has quit IRC | 10:26 | |
| *** ociuhandu has quit IRC | 10:26 | |
| *** Lucas_Gray has joined #openstack-infra | 10:30 | |
| *** pgaxatte has quit IRC | 10:32 | |
| *** yamamoto has quit IRC | 10:33 | |
| chkumar|ruck | Hello #Infra | 10:33 |
| *** kjackal has quit IRC | 10:33 | |
| chkumar|ruck | we are seeing Failed to retrieve repo file from https://trunk.rdoproject.org/centos7-master/current/delorean.repo after 10 retries on three jobs from yesterday and all these jobs are running on centos-7-limestone-regionone-0009851018 | 10:34 |
| chkumar|ruck | https://logs.opendev.org/85/673985/1/gate/tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates/64088df/job-output.txt.gz#_2019-08-07_15_06_10_690456 | 10:35 |
| chkumar|ruck | https://logs.opendev.org/65/669165/3/gate/tripleo-ci-centos-7-standalone/19bca80/job-output.txt.gz#_2019-08-08_09_16_07_471287 | 10:35 |
| chkumar|ruck | https://logs.opendev.org/65/669165/3/gate/tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates/b20b7c7/job-output.txt.gz#_2019-08-08_09_13_29_755246 | 10:35 |
| chkumar|ruck | but requests on delorean.repo is working fine | 10:35 |
| chkumar|ruck | is there some issue going with limestone ? | 10:36 |
| *** udesale has quit IRC | 10:38 | |
| *** iurygregory has quit IRC | 10:43 | |
| *** ociuhandu has joined #openstack-infra | 10:47 | |
| *** yamamoto has joined #openstack-infra | 10:51 | |
| *** tdasilva has joined #openstack-infra | 10:52 | |
| *** jaosorior has joined #openstack-infra | 11:10 | |
| openstackgerrit | Manuel Torrinha proposed openstack/diskimage-builder master: Fixes Launchpad issue #1808359 https://review.opendev.org/675294 | 11:12 |
| *** tobiash has quit IRC | 11:18 | |
| *** tobiash has joined #openstack-infra | 11:20 | |
| *** tobiash has quit IRC | 11:24 | |
| *** tobiash has joined #openstack-infra | 11:26 | |
| *** ginopc has quit IRC | 11:29 | |
| *** lpetrut has joined #openstack-infra | 11:34 | |
| *** lpetrut has quit IRC | 11:49 | |
| *** jamesmcarthur has joined #openstack-infra | 11:51 | |
| *** jamesmcarthur has quit IRC | 11:51 | |
| *** jamesmcarthur has joined #openstack-infra | 11:51 | |
| *** kjackal has joined #openstack-infra | 11:53 | |
| *** rh-jelabarre has joined #openstack-infra | 11:53 | |
| *** smarcet has joined #openstack-infra | 11:56 | |
| *** jamesmcarthur has quit IRC | 12:02 | |
| *** jamesmcarthur has joined #openstack-infra | 12:04 | |
| *** smarcet has quit IRC | 12:05 | |
| *** ricolin_ has joined #openstack-infra | 12:05 | |
| *** ricolin has quit IRC | 12:07 | |
| *** rfolco has joined #openstack-infra | 12:12 | |
| *** jamesmcarthur has quit IRC | 12:21 | |
| *** jaosorior has quit IRC | 12:22 | |
| *** ccamacho has quit IRC | 12:24 | |
| *** rlandy has joined #openstack-infra | 12:28 | |
| *** udesale has joined #openstack-infra | 12:28 | |
| *** rlandy is now known as rlandy|rover | 12:28 | |
| *** iurygregory has joined #openstack-infra | 12:36 | |
| *** pgaxatte has joined #openstack-infra | 12:49 | |
| *** eharney has quit IRC | 12:51 | |
| *** jcoufal has joined #openstack-infra | 12:54 | |
| *** jamesmcarthur has joined #openstack-infra | 12:54 | |
| *** smarcet has joined #openstack-infra | 12:55 | |
| *** aaronsheffield has joined #openstack-infra | 12:56 | |
| *** Lucas_Gray has quit IRC | 12:57 | |
| *** Vadmacs has quit IRC | 12:57 | |
| *** Lucas_Gray has joined #openstack-infra | 12:58 | |
| *** jcoufal has quit IRC | 13:02 | |
| *** Lucas_Gray has quit IRC | 13:03 | |
| *** jcoufal has joined #openstack-infra | 13:06 | |
| *** natalytvinova has quit IRC | 13:07 | |
| *** natalytvinova has joined #openstack-infra | 13:10 | |
| *** ekultails has joined #openstack-infra | 13:14 | |
| *** mriedem has joined #openstack-infra | 13:16 | |
| *** Lucas_Gray has joined #openstack-infra | 13:17 | |
| *** ccamacho has joined #openstack-infra | 13:23 | |
| *** Vadmacs has joined #openstack-infra | 13:24 | |
| *** pkopec has quit IRC | 13:26 | |
| *** guoqiao has quit IRC | 13:27 | |
| *** slaweq_ is now known as slaweq | 13:29 | |
| *** eharney has joined #openstack-infra | 13:32 | |
| openstackgerrit | Andreas Jaeger proposed openstack/diskimage-builder master: Stop regex warning https://review.opendev.org/675337 | 13:33 |
| *** stephenfin has quit IRC | 13:35 | |
| *** snierodz has quit IRC | 13:35 | |
| *** stephenfin has joined #openstack-infra | 13:36 | |
| *** ginopc has joined #openstack-infra | 13:37 | |
| *** snierodz has joined #openstack-infra | 13:38 | |
| *** yamamoto has quit IRC | 13:38 | |
| *** Diabelko has quit IRC | 13:39 | |
| openstackgerrit | Jeff Liu proposed zuul/zuul-operator master: WIP: Add zuul-operator-functional-openshift job https://review.opendev.org/674355 | 13:44 |
| clarkb | chkumar|ruck: limestone is an ipv6 cloud and trunk.rdoproject has no AAAA record. This means all traffic to it must go through shared NAT. Possible that is overloaded by other requests to ipv4 only locations like dockerhub | 13:51 |
| clarkb | logan-: might have time to quickly checl that NAT is working propely | 13:51 |
| *** ricolin_ is now known as ricolin | 13:55 | |
| donnyd | Maybe the additional stress of having two ipv6 clouds has something to do with it | 13:55 |
| *** jcoufal has quit IRC | 13:55 | |
| *** smarcet has quit IRC | 13:55 | |
| clarkb | Each ipv6 cloud is running separate NAT gateways though | 13:56 |
| *** electrofelix has quit IRC | 13:57 | |
| donnyd | Oh you mean NAT on the cloud side. I'm not sure what limestone is running, but I can show what it looks like from my end when the next job rolls around | 13:58 |
| *** yamamoto has joined #openstack-infra | 13:59 | |
| *** yamamoto has quit IRC | 13:59 | |
| *** yamamoto has joined #openstack-infra | 14:00 | |
| clarkb | yes NAT on the cloud side to translate from private cloud ipv4 addr to shared public ipv4 addr to talk to trunk.rdoproject.org via ipv4 | 14:00 |
| *** dave-mccowan has joined #openstack-infra | 14:00 | |
| *** smarcet has joined #openstack-infra | 14:01 | |
| donnyd | Yea that makes some sense, but honestly NAT doesn't really put a load on at least my edge FW | 14:01 |
| clarkb | The problem iirc isnt due to performance load but due ti limited numbers of ports for connections on a single IP? | 14:03 |
| clarkb | I want to say udp is more problematic though as it isnt properly stateful and the kernel has to approximate | 14:03 |
| *** yamamoto has quit IRC | 14:04 | |
| donnyd | my state table shows around 4K in connections and I am currently doing 70 nodes. | 14:05 |
| *** yamamoto has joined #openstack-infra | 14:05 | |
| donnyd | But its possible | 14:05 |
| openstackgerrit | Jeff Liu proposed zuul/zuul-operator master: WIP: Add zuul-operator-functional-openshift job https://review.opendev.org/674355 | 14:05 |
| *** liuyulong has quit IRC | 14:05 | |
| donnyd | I have been watching edge traffic and I am not sure this makes any difference at all, but the majority of outbound ipv4 connections are for dns and ntp | 14:08 |
| *** jbadiapa has joined #openstack-infra | 14:11 | |
| clarkb | chkumar|ruck: donnyd the other way that can break is if upstream has source based connection limits | 14:11 |
| donnyd | true story | 14:13 |
| donnyd | https://logs.opendev.org/05/675305/1/check/openstack-tox-py37/bdcc5a1/job-output.txt#_2019-08-08_13_54_29_388394 | 14:14 |
| donnyd | https://logs.opendev.org/24/675124/3/check/tripleo-ci-centos-7-undercloud-containers/6d8f838/job-output.txt#_2019-08-08_12_12_14_277164 | 14:14 |
| donnyd | fungi: it looks to me like moving back to ipv4 for the mirror has improved performance. Before these #'s were being measured in K | 14:15 |
| donnyd | but it still hasn't helped the jobs that timeout | 14:15 |
| donnyd | But i don't think that is isolated to the v6 clouds, seems to be all of them | 14:16 |
| fungi | donnyd: that's definitely worth digging into then! | 14:19 |
| * fungi is stuck in meetings most of his morning, just fyi | 14:19 | |
| fungi | clarkb: chkumar|ruck: donnyd: nat port ranges are often somewhat smallish but usually also configurable. and depending on implementation the nat may also hold out previously used ports for a little longer to account for late resets and fin/acks | 14:23 |
| fungi | so you usually need to find where the stats on the overload nat/pat pool are recorded | 14:24 |
| *** pkopec has joined #openstack-infra | 14:24 | |
| AJaeger | fungi, clarkb, I updated ansible-lint on some other projects (with similar blacklist) and add a few spaces to make ansible future-proof. Could you review https://review.opendev.org/675260 https://review.opendev.org/675254 and https://review.opendev.org/675255 , please? | 14:26 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: web: refactor the errorsIds into the build action https://review.opendev.org/675350 | 14:28 |
| *** pgaxatte has quit IRC | 14:29 | |
| *** dave-mccowan has quit IRC | 14:30 | |
| *** ociuhandu has quit IRC | 14:30 | |
| *** ociuhandu has joined #openstack-infra | 14:31 | |
| *** rlandy|rover is now known as rlandy|rover|mtg | 14:31 | |
| *** jcoufal has joined #openstack-infra | 14:35 | |
| *** ociuhandu has quit IRC | 14:35 | |
| openstackgerrit | Jeff Liu proposed zuul/zuul-operator master: WIP: Add zuul-operator-functional-openshift job https://review.opendev.org/674355 | 14:41 |
| *** rascasoft has quit IRC | 14:49 | |
| *** rascasoft has joined #openstack-infra | 14:52 | |
| chkumar|ruck | clarkb: donnyd thanks, I will take a look, thanks! | 14:56 |
| *** diablo_rojo has joined #openstack-infra | 14:58 | |
| corvus | infra-root: i think zuul-preview can be used as an open proxy, so i've shut the service down until we can track that down | 15:02 |
| fungi | thanks for the heads up, corvus | 15:02 |
| fungi | i can sort of half-imagine how that might work | 15:03 |
| corvus | yeah, i don't think it's a fundamental design flaw, i think we probably just missed a setting | 15:03 |
| *** ociuhandu has joined #openstack-infra | 15:04 | |
| corvus | #status log shut down zuul-preview and added zp01 to emergency disable due to a suspected misconfiguration allowing it to act as an open proxy | 15:06 |
| openstackstatus | corvus: finished logging | 15:06 |
| mordred | corvus: oh, that's not joyous | 15:08 |
| *** psachin has quit IRC | 15:15 | |
| *** jbadiapa has quit IRC | 15:18 | |
| *** rlandy|rover|mtg is now known as rlandy|rover | 15:20 | |
| openstackgerrit | Merged zuul/zuul master: Scroll log line anchor into view https://review.opendev.org/675220 | 15:20 |
| *** Lucas_Gray has quit IRC | 15:27 | |
| *** odicha has quit IRC | 15:29 | |
| *** jbadiapa has joined #openstack-infra | 15:30 | |
| *** spsurya has quit IRC | 15:34 | |
| *** jbadiapa has quit IRC | 15:35 | |
| *** eernst has joined #openstack-infra | 15:40 | |
| fungi | corvus: did you see evidence of it being exploited to that end? or just happened to stumble across a possible hole there? | 15:40 |
| corvus | fungi: the log looked like it was being used | 15:42 |
| fungi | thanks, figured that might be it | 15:42 |
| *** gyee has joined #openstack-infra | 15:43 | |
| *** ricolin_ has joined #openstack-infra | 15:43 | |
| mordred | corvus: I'm curious to see what we missed | 15:43 |
| corvus | mordred: i'm not working on that right now, so if you want to dig into it, be my guest | 15:44 |
| corvus | (i'm heads down in prepping for the log change) | 15:44 |
| *** eernst has quit IRC | 15:44 | |
| *** armax has joined #openstack-infra | 15:44 | |
| mordred | corvus: totes. I'll put it on my list | 15:45 |
| mordred | although probably also won't work on it today | 15:45 |
| *** ricolin_ has quit IRC | 15:45 | |
| *** ricolin has quit IRC | 15:46 | |
| *** ricolin_ has joined #openstack-infra | 15:46 | |
| *** ricolin_ is now known as ricolin | 15:47 | |
| *** sthussey has joined #openstack-infra | 15:47 | |
| *** natalytvinova has quit IRC | 15:50 | |
| *** diga has quit IRC | 15:57 | |
| *** lucasagomes has quit IRC | 16:03 | |
| *** Lucas_Gray has joined #openstack-infra | 16:06 | |
| *** iurygregory has quit IRC | 16:08 | |
| *** smarcet has quit IRC | 16:15 | |
| *** beekneemech has joined #openstack-infra | 16:16 | |
| *** e0ne has quit IRC | 16:16 | |
| *** beekneemech has quit IRC | 16:16 | |
| *** yamamoto has quit IRC | 16:18 | |
| openstackgerrit | Clark Boylan proposed zuul/zuul master: Consistent handling of ansible venv install dirs https://review.opendev.org/675403 | 16:26 |
| openstackgerrit | Merged zuul/zuul master: Render console in js https://review.opendev.org/674368 | 16:28 |
| *** mattw4 has joined #openstack-infra | 16:29 | |
| *** smarcet has joined #openstack-infra | 16:30 | |
| openstackgerrit | Merged zuul/zuul master: Usability tweaks for the build page console https://review.opendev.org/675147 | 16:34 |
| *** rpittau is now known as rpittau|afk | 16:36 | |
| *** diablo_rojo has quit IRC | 16:37 | |
| *** mattw4 has quit IRC | 16:39 | |
| *** mattw4 has joined #openstack-infra | 16:39 | |
| clarkb | AJaeger: hrw's https://review.opendev.org/#/c/671445/2 lgtm if you want to be second reviewer now (catching up on scrollback now) | 16:40 |
| clarkb | johnsom: we do our best to mirror or proxy cache major sources of upstream data. All of the distros we build images for have mirrors except for gentoo, we proxy cache pypi, dockerhub, rubygems, npm, and more | 16:41 |
| *** chkumar|ruck is now known as raukadah | 16:42 | |
| openstackgerrit | Jeff Liu proposed zuul/zuul-operator master: WIP: Add zuul-operator-functional-openshift job https://review.opendev.org/674355 | 16:42 |
| johnsom | clarkb Yeah, we have used the distro/package mirrors for a while. It's the DIB pip install that appears to have not been using the mirrors. I'm giving it a go today. | 16:43 |
| fungi | when analyzing traffic for fortnebula which wasn't going through the mirror server, the hit spots seemed to be dockerhub, pypi and rdo | 16:44 |
| fungi | er, hot spots | 16:44 |
| fungi | we don't currently mirror or cache rdo so that might be something to look at i guess | 16:45 |
| logan- | o/ .. i'll check for nat contention, but i'm a little skeptical since it seems like only trunk.rdoproject.org is coming up unreachable. i looked thru a batch of recent failures and didn't come up with other jobs failing due to network | 16:45 |
| clarkb | fungi: we do cache all of those in the mirror node so maybe we need to advertise that more | 16:45 |
| clarkb | fungi: we do actually proxy cache rdo | 16:45 |
| fungi | oh! right on | 16:45 |
| fungi | so it all boils down to jobs not using the mirror host | 16:46 |
| clarkb | logan-: thanks, and that is a good point. If that is behind CDN could be that the node servicing limestone was down explaining why it hit limestone | 16:46 |
| logan- | going to do some more digging, im thinking of starting by setting up some long running tcpdumps at the network nodes to watch trunk.rdoproject.org traffic and see what we can correlate with job failures there | 16:47 |
| *** tosky has quit IRC | 16:49 | |
| *** Lucas_Gray has quit IRC | 16:51 | |
| *** yamamoto has joined #openstack-infra | 16:51 | |
| fungi | the other thing we can do is enumerate some instance names or ip addresses which reached out to rdo (or pypi or dockerhub) and use zuul logs to track those back to specific jobs | 16:51 |
| *** Lucas_Gray has joined #openstack-infra | 16:51 | |
| fungi | obviously from the failures we know at least some of the jobs in question | 16:52 |
| clarkb | corvus: the open proxy thing being used is probably good evidence we should avoid open proxies for job data caching | 16:52 |
| fungi | clarkb: i guess you mean it validates our decision not to do generic open proxies on the mirror hosts | 16:53 |
| clarkb | fungi: ya | 16:53 |
| mordred | yeah - I think we're all in agreement about not using open proxies :) | 16:53 |
| fungi | i concur | 16:53 |
| fungi | not that i thought we really needed any fresh evidence that people scour the internet in search of open proxies to exploit | 16:54 |
| openstackgerrit | Matt Riedemann proposed opendev/elastic-recheck master: Add query for nova functional test race bug 1839515 https://review.opendev.org/675408 | 16:54 |
| openstack | bug 1839515 in OpenStack Compute (nova) "Weird functional test failures hitting neutron API in unrelated resize flows since 8/5" [Undecided,New] https://launchpad.net/bugs/1839515 | 16:54 |
| *** armax has quit IRC | 16:56 | |
| *** dtantsur is now known as dtantsur|afk | 16:57 | |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Add option to report build page https://review.opendev.org/675409 | 16:57 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Add release note for Pagure driver https://review.opendev.org/675410 | 16:57 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Move admin-rules setting in tenants doc https://review.opendev.org/675411 | 16:57 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Make auth docs more boring https://review.opendev.org/675412 | 16:57 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Don't capitalize Token in docs https://review.opendev.org/675413 | 16:57 |
| *** tdasilva has quit IRC | 16:59 | |
| openstackgerrit | Matt Riedemann proposed opendev/elastic-recheck master: Add query for nova functional test race bug 1839515 https://review.opendev.org/675408 | 17:00 |
| openstack | bug 1839515 in OpenStack Compute (nova) "Weird functional test failures hitting neutron API in unrelated resize flows since 8/5" [High,Confirmed] https://launchpad.net/bugs/1839515 | 17:00 |
| *** derekh has quit IRC | 17:00 | |
| *** roman_g has quit IRC | 17:04 | |
| *** yamamoto has quit IRC | 17:04 | |
| *** Lucas_Gray has quit IRC | 17:05 | |
| *** e0ne has joined #openstack-infra | 17:05 | |
| openstackgerrit | Ben Nemec proposed openstack/devstack-gate master: WIP: Use OSCaaS to speed up devstack runs https://review.opendev.org/675414 | 17:06 |
| *** roman_g has joined #openstack-infra | 17:06 | |
| openstackgerrit | Merged openstack/project-config master: Linaro London: use new bigger flavour https://review.opendev.org/671445 | 17:06 |
| *** udesale has quit IRC | 17:08 | |
| *** sshnaidm is now known as sshnaidm|off | 17:10 | |
| *** markvoelker has quit IRC | 17:12 | |
| *** igordc has joined #openstack-infra | 17:13 | |
| *** ociuhandu has quit IRC | 17:14 | |
| *** ricolin_ has joined #openstack-infra | 17:16 | |
| clarkb | infra-root I'd like to approve https://review.opendev.org/#/c/674930/ now to improve gitea sshd logging. Any reason for me to not do that now? | 17:19 |
| *** ricolin has quit IRC | 17:19 | |
| fungi | i've approved it | 17:20 |
| clarkb | perfect | 17:20 |
| fungi | i guess we should just make sure sshd keeps running | 17:20 |
| fungi | otherwise replication will cease happening | 17:20 |
| clarkb | docker-compose will restart it, but ya we should check it is running after the restart and then check docker logs works for it and that replication is working | 17:20 |
| fungi | and it might not be outwardly apparent for some time | 17:20 |
| *** jamesmcarthur has quit IRC | 17:22 | |
| openstackgerrit | Clark Boylan proposed zuul/zuul master: Consistent handling of ansible venv install dirs https://review.opendev.org/675403 | 17:25 |
| *** markvoelker has joined #openstack-infra | 17:26 | |
| *** tesseract has quit IRC | 17:27 | |
| openstackgerrit | Merged opendev/elastic-recheck master: Add query for nova functional test race bug 1839515 https://review.opendev.org/675408 | 17:28 |
| openstack | bug 1839515 in OpenStack Compute (nova) "Weird functional test failures hitting neutron API in unrelated resize flows since 8/5" [High,Confirmed] https://launchpad.net/bugs/1839515 | 17:28 |
| *** kopecmartin is now known as kopecmartin|off | 17:31 | |
| *** ricolin_ is now known as ricolin | 17:32 | |
| *** jamesmcarthur has joined #openstack-infra | 17:33 | |
| *** e0ne has quit IRC | 17:35 | |
| *** ociuhandu has joined #openstack-infra | 17:36 | |
| openstackgerrit | Jeremy Stanley proposed opendev/system-config master: Replace wiki-dev02 with wiki-dev03 https://review.opendev.org/675425 | 17:37 |
| *** ginopc has quit IRC | 17:38 | |
| clarkb | fungi: did that board transparency mailing list get disabled after it gummed up the server? | 17:38 |
| *** armax has joined #openstack-infra | 17:39 | |
| *** ociuhandu has quit IRC | 17:40 | |
| fungi | clarkb: not yet, i have it on my to do list to ask some folks who i need to get an okay from | 17:41 |
| fungi | er, that is, to ask some osf staff whether we need the board to say it's okay to retire/archive it | 17:42 |
| fungi | since it was for an osf board of directors working group | 17:42 |
| *** ianychoi has quit IRC | 17:43 | |
| clarkb | considering it was the transparency ml seems extra prudent to do that | 17:44 |
| fungi | yes | 17:45 |
| fungi | well, i mean, technically it still *is* the transparency ml, it just hasn't had a post in >4 years now | 17:46 |
| johnsom | clarkb I have another one of those DNS failures: https://logs.opendev.org/81/584681/26/check/octavia-v2-act-stdby-dsvm-scenario-two-node/ed101de/job-output.txt.gz#_2019-08-08_16_52_24_319778 | 17:46 |
| clarkb | johnsom: I got as far as "it largely a centos 7 problem but happens on all the cloud regions" the other day looking at that | 17:46 |
| clarkb | that job was xenial though so falls outside of that | 17:47 |
| *** ianychoi has joined #openstack-infra | 17:47 | |
| *** panda has quit IRC | 17:53 | |
| *** ralonsoh has quit IRC | 17:54 | |
| *** ianychoi_ has joined #openstack-infra | 17:54 | |
| *** panda has joined #openstack-infra | 17:54 | |
| *** ianychoi_ has quit IRC | 17:55 | |
| *** ianychoi has quit IRC | 17:58 | |
| *** smarcet has quit IRC | 18:06 | |
| *** psachin has joined #openstack-infra | 18:07 | |
| *** psachin has quit IRC | 18:11 | |
| openstackgerrit | Merged opendev/system-config master: Collect gitea sshd logs https://review.opendev.org/674930 | 18:12 |
| mriedem | clarkb: are we still running devstack nodes in vms with 8vcpu and 8gb ram? | 18:16 |
| clarkb | mriedem: yes | 18:16 |
| mriedem | i haven't been able to have a stable ubuntu 18.04 devstack running on master for at least a month, tried both py27 and py36, running with 1 API_WORKER per service, and still have random crashes of nova/neutron/cinder services | 18:17 |
| clarkb | mriedem: do your intsances have swap? we create swap on our test VMs and devstack produces a result that needs swap with 8GB of memory now :( | 18:18 |
| mriedem | hmm | 18:20 |
| *** smarcet has joined #openstack-infra | 18:20 | |
| mriedem | probably not | 18:21 |
| mriedem | yeah the flavor i'm using for the vm (v1-standard-8 from vexxhost) doesn't have swap | 18:21 |
| clarkb | I brought it up on hte ml but other than you pointing out the change to stop running cinder backup I haven't seen anything more | 18:21 |
| clarkb | (I think this is actually a fairly major problem with openstack right now, it affects our ability to test the software and ofr people to use it locally easily) | 18:22 |
| mriedem | oh an in this case, i disable basically all of cinder, etcd, tempest and horizon | 18:22 |
| mriedem | yeah something in the last 1-2 months and i'm unable to use devstack | 18:22 |
| mriedem | this is my fake driver local conf i'm using today (except NUMBER_FAKE_NOVA_COMPUTE=2) http://paste.openstack.org/show/755667/ | 18:23 |
| mordred | clarkb, mriedem: also edge use cases probably like it when our footprint isn't super high | 18:23 |
| clarkb | mriedem: to confirm this is the problem I would check dmesg for OOMKiller output | 18:23 |
| clarkb | mriedem: you can also check top then hit > to sort by memory percentage instead of cpu | 18:24 |
| clarkb | that might quickly identify problems if memory is the problem | 18:24 |
| clarkb | disabling tempets and horizon is probably not much of a win there since tempest doesn't run by default (it is an install) and horizon runs out of apache which is running anyway | 18:25 |
| mriedem | no OOMKiller in dmesg | 18:25 |
| clarkb | cinder should have an impact though | 18:25 |
| clarkb | mriedem: also free -m to see memory usage at a very high level | 18:25 |
| mriedem | i do have a crash file from apport but never tried to parse one of those things | 18:25 |
| mriedem | $ free -mh | 18:26 |
| mriedem | total used free shared buff/cache available | 18:26 |
| mriedem | Mem: 7.8G 2.9G 1.7G 2.8M 3.1G 4.6G | 18:26 |
| mriedem | Swap: 0B 0B 0B | 18:26 |
| mriedem | of course i need to restart some of these dead services, doing that | 18:26 |
| *** igordc has quit IRC | 18:28 | |
| mriedem | heh, "openstack volume service list" > public endpoint for volumev3 service in RegionOne region not found | 18:29 |
| mriedem | that's not good | 18:29 |
| smcginnis | mriedem: Didn't you say you disabled cinder? | 18:30 |
| mriedem | oh right :) | 18:30 |
| *** jamesmcarthur has quit IRC | 18:30 | |
| mriedem | using the fake driver so i disabled cinder, but the other day had a 'normal' devstack with libvirt | 18:30 |
| *** kjackal has quit IRC | 18:31 | |
| mriedem | well restarting nova and neutron then | 18:31 |
| mriedem | ok those are back up | 18:32 |
| mriedem | $ free -mh | 18:32 |
| mriedem | total used free shared buff/cache available | 18:32 |
| mriedem | Mem: 7.8G 3.2G 1.5G 2.8M 3.1G 4.3G | 18:32 |
| mriedem | no big change there | 18:32 |
| clarkb | you may need to use the services so that they grow in footprint | 18:33 |
| mriedem | nothing seems to be chewing up memory, mysqld is pretty stable at the top but just 5% | 18:34 |
| *** bobh has joined #openstack-infra | 18:35 | |
| clarkb | fwiw I was deinitely able to induce crashy behavior via running tempest against a default devstack cloud on an 8GB memory host with no swap | 18:35 |
| clarkb | that washow I discovered swap was required when spinning up the fn cloud | 18:35 |
| mriedem | ok i can try creating a test guest, that's when things crashed earlier | 18:36 |
| * mriedem is also on a call about standing up a private dev/test cloud | 18:36 | |
| fungi | also your instance flavor doesn't need a swap partition, you can just create a swapfile in the rootfs instead but i strongly recommend preallocating all the blocks and not just making it a sparse file | 18:39 |
| fungi | otherwise you get different random crashes when you run out of space on the filesystem and then try to swap | 18:39 |
| fungi | but yeah, given c-vol was the biggest offender for memory utilization in devstack jobs recently, if you're not running that you likely don't need any swap | 18:40 |
| *** jamesmcarthur has joined #openstack-infra | 18:42 | |
| clarkb | someone wasasking about similar stuffin #opemstack-qa yesterday and I think tosky was helping them | 18:44 |
| clarkb | I was busy enjoying aday at the coast though | 18:44 |
| fungi | well-deserved | 18:46 |
| *** rfolco has quit IRC | 18:51 | |
| *** pkopec has quit IRC | 18:52 | |
| mriedem | of course now i can't crash the damn thing | 18:55 |
| *** armstrong has joined #openstack-infra | 18:56 | |
| openstackgerrit | Saul Wold proposed openstack/project-config master: starlingx: add zuul-jobs repo https://review.opendev.org/675445 | 18:58 |
| *** rfolco has joined #openstack-infra | 19:00 | |
| *** armax has quit IRC | 19:01 | |
| *** armax has joined #openstack-infra | 19:02 | |
| *** ramishra has quit IRC | 19:02 | |
| fungi | mriedem: if only our gate jobs were so lucky? | 19:06 |
| *** bobh has quit IRC | 19:06 | |
| mriedem | fungi: do we get frequent crashes in the gate? | 19:06 |
| mriedem | if so, are we detecting and auto-retrying the job or something? | 19:07 |
| *** smarcet has quit IRC | 19:08 | |
| openstackgerrit | Merged zuul/zuul master: Don't always show expansion option on build console https://review.opendev.org/675163 | 19:08 |
| openstackgerrit | Merged zuul/zuul master: Adjust results headings in build console page https://review.opendev.org/675203 | 19:09 |
| fungi | depends on what you mean by crashes, but devstack-managed services not starting or dying? seems like that happens fairly often due to bugs and races in openstack | 19:09 |
| clarkb | mriedem: fungi the major behavior we seem to notice is job slowness and timeouts | 19:10 |
| clarkb | if you then track that back to dstat or stackviz you see that swap is being hit hard and that causes things to break | 19:11 |
| clarkb | I would not be surprised if many test failures themselves were a result of that too | 19:11 |
| fungi | recall rather a lot of nondeterministic devstack job failures due to some service not starting | 19:11 |
| fungi | or possibly stopping abruptly | 19:11 |
| mriedem | fungi: yeah i meant the services crashes, but agree with clarkb | 19:12 |
| mriedem | if services were crashing *during* tests that would be pretty obvious in job logs | 19:12 |
| mriedem | like back when we had OOMKillers | 19:12 |
| mriedem | and the slowness and such related to my patch to disable c-bak | 19:12 |
| clarkb | if we removed swap we would see ^ | 19:12 |
| fungi | yeah | 19:12 |
| mriedem | now that you mention it, the other day when i had a normal aio devstack with libvirt, watching top and such cinder-volume was very heay | 19:15 |
| mriedem | *heavy | 19:15 |
| mriedem | seemed like a periodic was running continually or something | 19:16 |
| *** e0ne has joined #openstack-infra | 19:16 | |
| mriedem | smcginnis: any recent changes to cinder-volume that would be running periodics heavy. | 19:16 |
| mriedem | ? | 19:16 |
| smcginnis | mriedem: Nothing that I'm aware of, but I haven't been reviewing the new changes as much as I was. | 19:18 |
| mriedem | heh you're still top dog though https://www.stackalytics.com/report/contribution/cinder/60 | 19:18 |
| *** bobh has joined #openstack-infra | 19:19 | |
| mriedem | hmm https://github.com/openstack/cinder/commit/b0279f2080ea54a123a9249c7a1a8fb027909ce2#diff-724c53b8cd580ae533b4fb301df7c3e9 | 19:20 |
| mriedem | seems ok...defaults are 60 which matches the default in oslo.service | 19:22 |
| fungi | maybe tasks are taking more than 60 seconds to complete and so they're piling up over time? | 19:26 |
| mriedem | we've seen c-vol get locked up on lvchange for more than 60 seconds | 19:26 |
| mriedem | leading to starving other requests to lvchange resulting in MessagingTimeouts from c-vol to c-api | 19:27 |
| mriedem | and 500s out of the cinder API | 19:27 |
| mriedem | let's see what happens if i try to create 100 servers | 19:28 |
| openstackgerrit | Ronelle Landy proposed opendev/elastic-recheck master: Add query for UNAUTHORIZED error when pulling containers https://review.opendev.org/675453 | 19:29 |
| clarkb | cloudnull: ^ did adding the reauths help that at all? | 19:31 |
| donnyd | mriedem: This is why i just had to set max_concurrent_builds=2 on FN | 19:31 |
| mriedem | donnyd: in this devstack i'm using the fake nova virt driver | 19:31 |
| mriedem | and just 2 fake computes | 19:31 |
| mriedem | i mean, they are real nova-compute services in systemd, but running a fake driver | 19:31 |
| donnyd | is c-vol using lvm? | 19:32 |
| mriedem | i've disabled cinder in this vm | 19:32 |
| mriedem | in the gate yes | 19:32 |
| cloudnull | clarkb I think it did, but we're adding more logs to the process to try and track things down . | 19:32 |
| mriedem | using the fake driver is useful for testing things like evacuate on a single node devstack | 19:32 |
| mriedem | creating 100 fake servers was ok, nova-conductor hammers CPU but that's not news | 19:35 |
| *** Romik has joined #openstack-infra | 19:36 | |
| *** jcoufal has quit IRC | 19:40 | |
| *** auristor has quit IRC | 19:42 | |
| mriedem | clarkb: looking at https://logs.opendev.org/17/675117/1/check/tempest-full-py3/6cee1e5/controller/logs/stackviz/#/stdin/timeline?test=tempest.scenario.test_security_groups_basic_ops.TestSecurityGroupsBasicOps.test_cross_tenant_traffic reminded me of something, | 19:42 |
| *** noonedeadpunk has quit IRC | 19:42 | |
| *** noonedeadpunk has joined #openstack-infra | 19:43 | |
| mriedem | tempest-full jobs run api tests concurrently (nproc/2) but scenario tests are run in serial, | 19:43 |
| mriedem | which is why the scenario tests are all stacked up on worker 0 | 19:43 |
| mriedem | i had a patch to try and run scenario tests with nproc/4 but it got snagged up | 19:43 |
| mriedem | https://review.opendev.org/#/c/650300/ | 19:44 |
| fungi | now that the release meeting is over i'm going to go find very late lunch, but will return in a while. if anyone wants to approve 675425 i can continue testing that when i get back | 19:45 |
| *** auristor has joined #openstack-infra | 19:46 | |
| *** dciabrin_ is now known as dciabrin | 19:53 | |
| clarkb | we have sshd logs on gitea01 now | 19:53 |
| clarkb | the containers were restarted as expected on all gitea hosts | 19:55 |
| clarkb | there are no queued replication events on review01 | 19:55 |
| clarkb | I'm going to trigger replication of system-config and see that it doesn't fail | 19:55 |
| *** bobh has quit IRC | 19:56 | |
| *** bobh has joined #openstack-infra | 19:56 | |
| clarkb | and that seems to have worked | 19:56 |
| *** panda has quit IRC | 19:57 | |
| *** eharney has quit IRC | 19:58 | |
| *** panda has joined #openstack-infra | 19:59 | |
| *** jbadiapa has joined #openstack-infra | 20:00 | |
| *** bobh has quit IRC | 20:01 | |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Add permalinks to task detail popup https://review.opendev.org/675236 | 20:01 |
| *** noonedeadpunk has quit IRC | 20:03 | |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: web: extract pure functions from the TaskOutput component https://review.opendev.org/675460 | 20:08 |
| weshay | hopefully elastic-recheck has the resources for https://review.opendev.org/#/c/675453/ | 20:09 |
| weshay | please review if you have a momemt | 20:09 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Hide "root" variable in job web page https://review.opendev.org/675461 | 20:11 |
| clarkb | weshay: is that query specific enough? will the name of that container only show up on failures for the unauthorized docker hub requests? | 20:12 |
| clarkb | maybe we can check for 401 messages instead or similar? | 20:12 |
| weshay | clarkb it's just the base name of all the containers | 20:12 |
| *** jbadiapa has quit IRC | 20:12 | |
| weshay | but if you think we should change it.. that's fine too | 20:12 |
| clarkb | right so wouldn't it show up in normal jobs because it is used everywhere? | 20:12 |
| *** Vadmacs has quit IRC | 20:13 | |
| weshay | don't understand that last comment | 20:13 |
| openstackgerrit | Merged zuul/zuul master: Add option to report build page https://review.opendev.org/675409 | 20:13 |
| clarkb | weshay: ideally e-r queries will only match on jobs that have failed due to the bug tied to the query. If we are searching for the name of an image that is used in every job are we going to match every job that runs regardless of whether or not it failed? | 20:14 |
| weshay | http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Name%5C%5C%5C%22%3A%5C%5C%5C%22tripleomaster%2Fcentos-binary%5C%22%20AND%20tags%3A%5C%22console%5C%22 | 20:15 |
| weshay | we can refine it a bit more.. | 20:15 |
| weshay | I get your point now | 20:15 |
| *** smarcet has joined #openstack-infra | 20:16 | |
| *** allanb has joined #openstack-infra | 20:16 | |
| clarkb | '"code":"UNAUTHORIZED","message":"authentication required"' looks like the string we may want to match on | 20:17 |
| openstackgerrit | wes hayutin proposed opendev/elastic-recheck master: Add query for UNAUTHORIZED error when pulling containers https://review.opendev.org/675453 | 20:23 |
| *** diablo_rojo has joined #openstack-infra | 20:24 | |
| weshay | clarkb reworking it now.. thanks for the quick ping back.. | 20:26 |
| *** betherly has quit IRC | 20:27 | |
| *** diablo_rojo__ has joined #openstack-infra | 20:30 | |
| weshay | rlandy|rover we're good w/ the latest? | 20:31 |
| rlandy|rover | weshay: ack - thanks for updating that | 20:31 |
| *** jtomasek has quit IRC | 20:31 | |
| weshay | clarkb ok.. it should hit only errors while pulling now | 20:31 |
| openstackgerrit | Merged zuul/zuul master: Use wait for empty update queue before accepting merges https://review.opendev.org/675039 | 20:33 |
| *** diablo_rojo has quit IRC | 20:33 | |
| clarkb | mriedem: did you want to review that one too? | 20:35 |
| *** e0ne has quit IRC | 20:37 | |
| mriedem | yeah i'll take a look | 20:46 |
| *** rascasoft has quit IRC | 20:47 | |
| *** bhavikdbavishi has quit IRC | 20:47 | |
| *** betherly has joined #openstack-infra | 20:48 | |
| *** rascasoft has joined #openstack-infra | 20:49 | |
| mriedem | done | 20:51 |
| *** diablo_rojo__ is now known as diablo_rojo | 20:52 | |
| *** betherly has quit IRC | 20:52 | |
| *** jcoufal has joined #openstack-infra | 20:53 | |
| openstackgerrit | Merged zuul/zuul master: Refactor build page tabs https://review.opendev.org/675235 | 20:54 |
| *** guoqiao has joined #openstack-infra | 20:57 | |
| *** kjackal has joined #openstack-infra | 20:57 | |
| *** tjgresha has joined #openstack-infra | 20:58 | |
| *** noonedeadpunk has joined #openstack-infra | 20:59 | |
| *** whoami-rajat has quit IRC | 21:01 | |
| openstackgerrit | Merged opendev/elastic-recheck master: Add query for UNAUTHORIZED error when pulling containers https://review.opendev.org/675453 | 21:04 |
| *** rm_work has quit IRC | 21:05 | |
| *** rm_work has joined #openstack-infra | 21:05 | |
| *** diablo_rojo has quit IRC | 21:07 | |
| weshay | thanks clarkb! | 21:07 |
| openstackgerrit | Jeff Liu proposed zuul/zuul-operator master: WIP: Add zuul-operator-functional-openshift job https://review.opendev.org/674355 | 21:07 |
| *** diablo_rojo has joined #openstack-infra | 21:08 | |
| *** betherly has joined #openstack-infra | 21:08 | |
| *** betherly has quit IRC | 21:12 | |
| *** igordc has joined #openstack-infra | 21:13 | |
| fungi | okay, back and pitching in^W^Wtrying not to break stuff | 21:15 |
| fungi | clarkb: did you happen to notice if the staggering/serialization of gitea restarts happened as designed? | 21:15 |
| *** diablo_rojo__ has joined #openstack-infra | 21:16 | |
| clarkb | fungi: I did not. The default resolution for docker ps -a's output resulted in "about an hour ago" | 21:16 |
| openstackgerrit | Michael Johnson proposed openstack/diskimage-builder master: Fix the pypi element for multiple mirror URLs https://review.opendev.org/675468 | 21:16 |
| fungi | gotta love the dedication to precision in the docker community | 21:17 |
| fungi | hold my beer, i'll debug this | 21:17 |
| clarkb | I expect the ansible logs would probably show if they serialized properly | 21:17 |
| *** rfolco has quit IRC | 21:18 | |
| *** diablo_rojo has quit IRC | 21:19 | |
| *** jamesmcarthur has quit IRC | 21:19 | |
| openstackgerrit | Merged zuul/zuul master: Add permalinks to task detail popup https://review.opendev.org/675236 | 21:23 |
| *** diablo_rojo__ is now known as diablo_rojo | 21:25 | |
| *** betherly has joined #openstack-infra | 21:28 | |
| *** smarcet has quit IRC | 21:31 | |
| *** betherly has quit IRC | 21:33 | |
| *** smarcet has joined #openstack-infra | 21:35 | |
| *** markvoelker has quit IRC | 21:36 | |
| *** betherly has joined #openstack-infra | 21:36 | |
| *** jtomasek has joined #openstack-infra | 21:36 | |
| openstackgerrit | James E. Blair proposed zuul/zuul master: web: refactor the errorsIds into the build action https://review.opendev.org/675350 | 21:40 |
| *** betherly has quit IRC | 21:40 | |
| *** smarcet has quit IRC | 21:43 | |
| openstackgerrit | Clark Boylan proposed zuul/zuul master: Improve functionality and docs around ansible installation https://review.opendev.org/675403 | 21:49 |
| *** markvoelker has joined #openstack-infra | 21:53 | |
| *** armstrong has quit IRC | 21:55 | |
| *** betherly has joined #openstack-infra | 21:57 | |
| *** markvoelker has quit IRC | 21:57 | |
| *** betherly has quit IRC | 22:01 | |
| *** jcoufal has quit IRC | 22:02 | |
| *** panda has quit IRC | 22:03 | |
| *** kjackal has quit IRC | 22:03 | |
| *** kei-ichi has quit IRC | 22:03 | |
| *** panda has joined #openstack-infra | 22:03 | |
| *** mattw4 has quit IRC | 22:04 | |
| *** mattw4 has joined #openstack-infra | 22:04 | |
| openstackgerrit | James E. Blair proposed zuul/zuul master: web: refactor the errorsIds into the build action https://review.opendev.org/675350 | 22:05 |
| *** mattw4 has quit IRC | 22:14 | |
| *** mattw4 has joined #openstack-infra | 22:14 | |
| *** slaweq has quit IRC | 22:15 | |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Correctly identify failed tasks https://review.opendev.org/675488 | 22:16 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Refactor task result detection https://review.opendev.org/675489 | 22:16 |
| *** panda has quit IRC | 22:16 | |
| corvus | clarkb, fungi: where are we on the ara report error? | 22:18 |
| corvus | i can switch to working on that now, but i lost track of what was going on with ansible venvs etc | 22:18 |
| *** betherly has joined #openstack-infra | 22:18 | |
| fungi | i'm so distracted/disconnected i didn't know there was an ara report error | 22:19 |
| * fungi apologizes more for his cluelessness | 22:19 | |
| corvus | in the swift logs job, the static ara report generation is failing | 22:19 |
| fungi | ahh, i didn't know but am just about freed up and can take a look at errors | 22:20 |
| corvus | we have no debugging info other than a cryptic message that says it failed | 22:20 |
| *** panda has joined #openstack-infra | 22:20 | |
| corvus | clarkb was looking into whether we maybe just needed to upgrade ara? | 22:20 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: web: add buildset page https://review.opendev.org/630079 | 22:20 |
| fungi | oh, right, i thought there were some patches about upgrading ara in our ansible envs on the executors | 22:21 |
| fungi | which led to the discussion in #zuul about preinstalled vs zuul-managed ansible envs i guess | 22:21 |
| corvus | yep, but that should be independent | 22:21 |
| *** betherly has quit IRC | 22:23 | |
| *** whoami-rajat has joined #openstack-infra | 22:24 | |
| corvus | it looks like 0.16.5 is installed on ze01 now | 22:27 |
| corvus | i need to find when that happened to see if we have a run of ara afterwords | 22:28 |
| corvus | the timestamp on the executable file is aug 7 08:30 | 22:28 |
| corvus | my last recheck of swift jobs was before that. so the next step is to recheck 674359 and see if it's fixed. | 22:29 |
| fungi | my guess is yes | 22:29 |
| fungi | clarkb has likely fixed it at this point | 22:29 |
| corvus | i am not going to bet against clarkb having fixed something :) | 22:30 |
| fungi | i'm not going to risk angering the clarkb as he fixes more things than i break, and that's saying something | 22:30 |
| *** ociuhandu has joined #openstack-infra | 22:30 | |
| openstackgerrit | Merged zuul/zuul master: Provide buildset.uuid in /builds API result https://review.opendev.org/674759 | 22:30 |
| clarkb | well I'm hoping that the 0.16.5 installs are the fix | 22:31 |
| *** ociuhandu has quit IRC | 22:35 | |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: web: link the buildset page from the build https://review.opendev.org/675493 | 22:38 |
| *** mattw4 has quit IRC | 22:44 | |
| openstackgerrit | Michael Johnson proposed openstack/diskimage-builder master: Fix the pypi element for multiple mirror URLs https://review.opendev.org/675468 | 22:53 |
| *** rcernin has joined #openstack-infra | 22:53 | |
| *** mattw4 has joined #openstack-infra | 22:54 | |
| corvus | clarkb, fungi: still seeing "non-zero return code" | 22:54 |
| corvus | http://paste.openstack.org/show/755671/ | 22:55 |
| corvus | that's all we get | 22:55 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: web: extract pure functions from the TaskOutput component https://review.opendev.org/675460 | 22:55 |
| corvus | i'll see if i can repro manually | 22:58 |
| *** dchen has joined #openstack-infra | 23:00 | |
| *** betherly has joined #openstack-infra | 23:01 | |
| *** dchen has quit IRC | 23:01 | |
| *** dchen has joined #openstack-infra | 23:02 | |
| *** ricolin_ has joined #openstack-infra | 23:02 | |
| *** mattw4 has quit IRC | 23:03 | |
| corvus | hrm. a simple repro case seems to work fine | 23:04 |
| *** mattw4 has joined #openstack-infra | 23:04 | |
| *** ricolin has quit IRC | 23:05 | |
| *** diablo_rojo has quit IRC | 23:05 | |
| *** diablo_rojo has joined #openstack-infra | 23:06 | |
| *** betherly has quit IRC | 23:06 | |
| corvus | i rsynced a build directory, activated the 2.8 venv, ran ara generate html, and it worked | 23:06 |
| *** aaronsheffield has quit IRC | 23:06 | |
| dmsimard | hi o/ where can I see that failure happening ? | 23:08 |
| corvus | dmsimard: that paste http://paste.openstack.org/show/755671/ is all we have right now | 23:08 |
| dmsimard | corvus: what job is that from ? | 23:08 |
| corvus | i can't actually link to the logs yet because i won't know the url until the rest of the jobs have finished | 23:09 |
| corvus | but it probably wouldn't be useful anyway | 23:09 |
| dmsimard | curious to see it run on the executor | 23:10 |
| corvus | dmsimard: but it should be pretty much the same as the last buildset of https://review.opendev.org/674359 | 23:10 |
| dmsimard | corvus: looking | 23:10 |
| corvus | dmsimard: what are you looking for, maybe i can help | 23:10 |
| corvus | i mean, i have a streaming log window open, so i can copy/paste more of that if you want | 23:11 |
| *** slaweq has joined #openstack-infra | 23:11 | |
| corvus | here's the whole post-run playbook: http://paste.openstack.org/show/755672/ | 23:12 |
| dmsimard | corvus: I'm looking to see if there is a more verbose version of that error hidden somewhere :D | 23:13 |
| corvus | unfortunately, it's not in the job-output.json since that misses the last play -- however, i may have it in the copy of the build directory i just saved, let me see | 23:14 |
| *** slaweq has quit IRC | 23:15 | |
| *** ekultails has quit IRC | 23:16 | |
| corvus | that's disappointing -- it doesn't have any more helpful info: http://paste.openstack.org/show/755673/ | 23:16 |
| dmsimard | I found a random occurrence on ze01 and there is suspisciously really no more output http://paste.openstack.org/show/755674/ | 23:20 |
| corvus | yeah, i'm wondering if we're going to need to set keep and verbose on the executors to continue debugging | 23:21 |
| dmsimard | http://paste.openstack.org/show/755675/ is that occurrence with more context | 23:22 |
| *** armax has quit IRC | 23:22 | |
| dmsimard | I wonder if command munges output more than shell would | 23:23 |
| *** eernst has joined #openstack-infra | 23:23 | |
| corvus | dmsimard: do you want me to get started on keep+verbose? | 23:23 |
| corvus | dmsimard: could this just be a command not found error? | 23:26 |
| corvus | hrm... we check that the executable is found by bash... but then we use command and not bash to run it. | 23:27 |
| *** eernst has quit IRC | 23:27 | |
| corvus | so i would think that if it's not in the path, we would get the "not installed" error, but still there's a subtle difference there | 23:28 |
| dmsimard | the role does some assertions before getting to that step but we should improve them to be more verbose | 23:28 |
| dmsimard | it provides "/usr/local/bin/ara" | 23:28 |
| corvus | clarkb, dmsimard: amusingly, that is installed *outside* the ansible venvs, but it's still ara 0.16.5 | 23:32 |
| dmsimard | there is some amount of static files copied over so it kind of half ran https://storage.bhs1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/logs_59/674359/1/check/nodepool-zuul-functional/8d3bac7/ara-report/static/css/ | 23:32 |
| corvus | so there are two ara installations in play -- the one in the zuul-ansible venv which is where the callback is run from, and the one that's used to generate the html, which is system-wide on the executor. | 23:33 |
| corvus | (which, tbh, is actually pretty typical of an ara installation) | 23:33 |
| corvus | (so i don't think that's bad, just something to be aware of. anyway, they are both 0.16.5) | 23:34 |
| corvus | dmsimard: the incompatability with ansible that was fixed in the 0.16.4 -> 0.16.5 upgrade -- does that affect the callbacks in ansible 2.8? | 23:34 |
| corvus | i'm starting to wonder if we're still using the old versions of the callbacks because we haven't restarted the executor since the upgrade (which would happen if the ara callbacks are part of what the executor copies in place on startup) | 23:37 |
| corvus | if that's the case, then i think the next step would be to restart the executors and recheck | 23:37 |
| dmsimard | pretty positive a new ansible-playbook invocation would run off of an updated package | 23:38 |
| corvus | dmsimard: not if the package wasn't updated because it's part of what the executor copies in place | 23:39 |
| dmsimard | 0.16.5 was mostly an administrative release | 23:39 |
| corvus | hrm. someone said something about fixing a 2.8 compatability issue | 23:39 |
| dmsimard | indeed, in 0.16.4 | 23:39 |
| corvus | oh, that may be it then, we may have been running <16.4 even | 23:40 |
| corvus | lemme see if i can confirm this real quick | 23:40 |
| dmsimard | the regression was https://github.com/ansible-community/ara/issues/46 | 23:40 |
| dmsimard | but that was devel back then | 23:41 |
| dmsimard | trying to see if I can reproduce the html generation issue -- the sqlite database from the executor for a build would be helpful | 23:42 |
| corvus | dmsimard: root@ze09:/root/corvus-test has an entire build dir right before it was deleted | 23:45 |
| dmsimard | sweet | 23:45 |
| corvus | it looks like my theory about the ara callback being old isn't holding up -- that job configured the callbacks as /var/lib/zuul/ansible/2.8/zuul/ansible/callback:/usr/lib/zuul/ansible/2.8/lib/python3.5/site-packages/ara/plugins and that has the timestamp of the upgrade to 0.16.5 | 23:46 |
| corvus | (the first directory is the one that gets copied, the second is the venv which got updated, that has the timestamp of the 0.16.5 upgrade) | 23:47 |
| corvus | oh hey, i got it to reproduce | 23:48 |
| corvus | earlier i was running ara from the venv, not the system install | 23:49 |
| corvus | http://paste.openstack.org/show/755678/ | 23:49 |
| dmsimard | interesting! | 23:49 |
| *** mattw4 has quit IRC | 23:52 | |
| dmsimard | there be dragons in that general vicinity, I much prefer your implementation of the file tree view :) | 23:53 |
| *** betherly has joined #openstack-infra | 23:54 | |
| corvus | this is curious though -- if there's no "create_file" method, how does this work at all -- doesn't the ansible zuul use static ara generation? | 23:57 |
| *** sthussey has quit IRC | 23:57 | |
| dmsimard | that method is coming from pyfakefs, looks like we have version pyfakefs==3.5.8 | 23:57 |
| *** diablo_rojo has quit IRC | 23:57 | |
| corvus | oh gotcha | 23:57 |
| corvus | 3.5.8 is in the venv, but 3.3 is on the system | 23:58 |
| corvus | so presumably we need to upgrade the system version | 23:58 |
| dmsimard | why suddenly though ? | 23:58 |
| dmsimard | 3.5.8 is apparently not even the latest version, there was a 3.6.0 back in june | 23:58 |
| corvus | this is the first time (in about 8 months) that we've exercised the static generation in opendev | 23:59 |
| *** betherly has quit IRC | 23:59 | |
| corvus | so the last time we ran it, everything was probably old enough that it worked | 23:59 |
| dmsimard | I can try with an older version perhaps, hang on | 23:59 |
| corvus | since then, we've upgraded ara, but not pyfakefs. everybody else using ara static generation probably has upgraded both packages | 23:59 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!