| *** rlandy|ruck|bbl is now known as rlandy|ruck | 00:07 | |
| openstackgerrit | Mohammed Naser proposed openstack/project-config master: gerrit: change retired.config acls https://review.opendev.org/737649 | 00:09 |
|---|---|---|
| *** armax has joined #openstack-infra | 00:29 | |
| *** jamesmcarthur has joined #openstack-infra | 00:30 | |
| *** ryohayakawa has joined #openstack-infra | 00:35 | |
| *** hamalq_ has quit IRC | 00:44 | |
| *** jamesmcarthur has quit IRC | 00:50 | |
| *** jamesmcarthur has joined #openstack-infra | 00:51 | |
| openstackgerrit | Ian Wienand proposed openstack/project-config master: grafana: don't use bool for refresh https://review.opendev.org/737662 | 00:55 |
| *** jamesmcarthur has quit IRC | 00:58 | |
| *** lbragstad has joined #openstack-infra | 01:08 | |
| *** rcernin has quit IRC | 01:12 | |
| *** jamesmcarthur has joined #openstack-infra | 01:14 | |
| *** rcernin has joined #openstack-infra | 01:17 | |
| *** jamesmcarthur has quit IRC | 01:23 | |
| openstackgerrit | Merged openstack/project-config master: grafana: don't use bool for refresh https://review.opendev.org/737662 | 01:23 |
| *** markvoelker has joined #openstack-infra | 01:30 | |
| *** xarses_ has quit IRC | 01:32 | |
| *** xarses_ has joined #openstack-infra | 01:32 | |
| *** markvoelker has quit IRC | 01:35 | |
| *** yamamoto has joined #openstack-infra | 01:40 | |
| *** jamesmcarthur has joined #openstack-infra | 01:42 | |
| *** jamesmcarthur has quit IRC | 01:50 | |
| openstackgerrit | Ian Wienand proposed openstack/project-config master: grafyaml: drop python2 jobs https://review.opendev.org/737666 | 01:51 |
| *** tetsuro has joined #openstack-infra | 01:52 | |
| *** yamamoto has quit IRC | 02:04 | |
| *** jamesmcarthur has joined #openstack-infra | 02:11 | |
| *** jamesmcarthur has quit IRC | 02:11 | |
| *** jamesmcarthur has joined #openstack-infra | 02:11 | |
| *** rlandy|ruck has quit IRC | 02:18 | |
| *** rcernin has quit IRC | 02:34 | |
| *** rcernin has joined #openstack-infra | 02:36 | |
| *** yamamoto has joined #openstack-infra | 02:37 | |
| *** lbragstad has quit IRC | 02:47 | |
| *** Goneri has joined #openstack-infra | 02:53 | |
| *** jamesmcarthur has quit IRC | 02:57 | |
| *** jamesmcarthur has joined #openstack-infra | 03:02 | |
| *** artom has joined #openstack-infra | 03:07 | |
| *** rfolco has quit IRC | 03:09 | |
| *** matt_kosut has joined #openstack-infra | 03:30 | |
| *** markvoelker has joined #openstack-infra | 03:31 | |
| *** Goneri has quit IRC | 03:31 | |
| *** markvoelker has quit IRC | 03:36 | |
| *** matt_kosut has quit IRC | 03:36 | |
| *** psachin has joined #openstack-infra | 03:42 | |
| *** rcernin has quit IRC | 03:46 | |
| *** rcernin has joined #openstack-infra | 03:55 | |
| *** rcernin has quit IRC | 04:04 | |
| *** rcernin has joined #openstack-infra | 04:05 | |
| *** grantza has joined #openstack-infra | 04:26 | |
| *** ysandeep|away is now known as ysandeep | 04:28 | |
| *** jamesmcarthur has quit IRC | 04:31 | |
| *** evrardjp has quit IRC | 04:33 | |
| *** evrardjp has joined #openstack-infra | 04:33 | |
| *** markvoelker has joined #openstack-infra | 04:34 | |
| *** udesale has joined #openstack-infra | 04:35 | |
| *** jamesmcarthur has joined #openstack-infra | 04:37 | |
| *** Lucas_Gray has quit IRC | 04:38 | |
| *** markvoelker has quit IRC | 04:39 | |
| *** apetrich has quit IRC | 04:43 | |
| *** jtomasek has joined #openstack-infra | 04:53 | |
| *** jamesmcarthur has quit IRC | 04:58 | |
| *** jamesmcarthur has joined #openstack-infra | 04:58 | |
| *** dklyle has quit IRC | 04:58 | |
| openstackgerrit | Ian Wienand proposed openstack/project-config master: wheel-cache: convert release to a loop https://review.opendev.org/737678 | 04:59 |
| *** matt_kosut has joined #openstack-infra | 04:59 | |
| *** jamesmcarthur has quit IRC | 05:04 | |
| *** lmiccini has joined #openstack-infra | 05:11 | |
| *** d34dh0r53 has joined #openstack-infra | 05:18 | |
| *** jamesmcarthur has joined #openstack-infra | 05:33 | |
| *** sshnaidm|afk is now known as sshnaidm|off | 05:34 | |
| *** udesale has quit IRC | 05:40 | |
| *** jamesmcarthur has quit IRC | 05:42 | |
| *** d34dh0r53 has quit IRC | 05:51 | |
| *** eolivare has joined #openstack-infra | 05:52 | |
| *** jamesdenton has quit IRC | 05:52 | |
| *** jamesdenton has joined #openstack-infra | 06:00 | |
| *** slaweq_ has joined #openstack-infra | 06:12 | |
| *** jamesmcarthur has joined #openstack-infra | 06:13 | |
| *** slaweq has quit IRC | 06:13 | |
| *** markvoelker has joined #openstack-infra | 06:14 | |
| *** flepied has joined #openstack-infra | 06:17 | |
| *** flepied has quit IRC | 06:17 | |
| *** markvoelker has quit IRC | 06:20 | |
| *** jamesmcarthur has quit IRC | 06:22 | |
| *** udesale has joined #openstack-infra | 06:25 | |
| *** udesale_ has joined #openstack-infra | 06:29 | |
| *** ysandeep is now known as ysandeep|afk | 06:30 | |
| *** udesale has quit IRC | 06:30 | |
| *** amoralej|off is now known as amoralej | 06:31 | |
| *** dtantsur|afk is now known as dtantsur | 06:50 | |
| *** gyee has quit IRC | 06:51 | |
| *** ralonsoh has joined #openstack-infra | 06:55 | |
| *** rpittau|afk is now known as rpittau | 06:57 | |
| *** jcapitao has joined #openstack-infra | 07:03 | |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Test multiarch release builds https://review.opendev.org/737315 | 07:17 |
| *** hashar has joined #openstack-infra | 07:30 | |
| *** vishalmanchanda has joined #openstack-infra | 07:33 | |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Test multiarch release builds https://review.opendev.org/737315 | 07:34 |
| *** rcernin has quit IRC | 07:39 | |
| *** ysandeep|afk is now known as ysandeep | 07:42 | |
| *** dmellado has joined #openstack-infra | 07:43 | |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Test multiarch release builds https://review.opendev.org/737315 | 07:47 |
| *** priteau has joined #openstack-infra | 07:47 | |
| *** tosky has joined #openstack-infra | 07:49 | |
| *** slaweq_ is now known as slaweq | 07:49 | |
| *** Lucas_Gray has joined #openstack-infra | 07:52 | |
| openstackgerrit | Andreas Jaeger proposed openstack/openstack-zuul-jobs master: Add py38 job templates and make py3 templates consistent https://review.opendev.org/737701 | 07:56 |
| *** jpena|off is now known as jpena | 07:56 | |
| AJaeger | ianw, fungi, clarkb, based on ianw's feedback, here's my proposal on the py38 job templates ^. Please review. Happy to rename some templates as well. | 07:57 |
| *** bhagyashris is now known as bhagyashris|lunc | 07:57 | |
| openstackgerrit | Vishal Manchanda proposed openstack/project-config master: Upadting horizon nodejs job name https://review.opendev.org/737457 | 08:02 |
| *** udesale_ has quit IRC | 08:11 | |
| openstackgerrit | Vishal Manchanda proposed openstack/project-config master: Upadting horizon nodejs job name https://review.opendev.org/737457 | 08:12 |
| *** markvoelker has joined #openstack-infra | 08:16 | |
| *** jamesmcarthur has joined #openstack-infra | 08:18 | |
| *** markvoelker has quit IRC | 08:21 | |
| *** jamesmcarthur has quit IRC | 08:23 | |
| *** apetrich has joined #openstack-infra | 08:43 | |
| *** xek_ has joined #openstack-infra | 08:46 | |
| *** pkopec has joined #openstack-infra | 08:48 | |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Test multiarch release builds https://review.opendev.org/737315 | 08:50 |
| *** gfidente has joined #openstack-infra | 08:52 | |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Test multiarch release builds https://review.opendev.org/737315 | 09:04 |
| *** bhagyashris|lunc is now known as bhagyashris | 09:08 | |
| *** derekh has joined #openstack-infra | 09:08 | |
| *** rcernin has joined #openstack-infra | 09:12 | |
| *** rcernin has quit IRC | 09:17 | |
| *** matt_kosut has quit IRC | 09:22 | |
| *** hashar has quit IRC | 09:48 | |
| *** tkajinam has quit IRC | 10:03 | |
| *** markvoelker has joined #openstack-infra | 10:05 | |
| *** markvoelker has quit IRC | 10:09 | |
| *** rcernin has joined #openstack-infra | 10:10 | |
| *** rpittau is now known as rpittau|bbl | 10:11 | |
| *** ociuhandu has quit IRC | 10:14 | |
| *** rcernin has quit IRC | 10:15 | |
| *** jamesmcarthur has joined #openstack-infra | 10:20 | |
| *** nightmare_unreal has joined #openstack-infra | 10:24 | |
| *** jamesmcarthur has quit IRC | 10:32 | |
| openstackgerrit | Jonathan Rosser proposed openstack/project-config master: Refresh openstack-ansible grafana dashboards https://review.opendev.org/737742 | 10:39 |
| *** hashar has joined #openstack-infra | 10:41 | |
| *** grantza has left #openstack-infra | 10:54 | |
| *** psachin has quit IRC | 10:57 | |
| *** jcapitao has quit IRC | 11:27 | |
| *** rcernin has joined #openstack-infra | 11:27 | |
| *** xek_ has quit IRC | 11:27 | |
| *** jcapitao has joined #openstack-infra | 11:28 | |
| *** jcapitao is now known as jcapitao_lunch | 11:28 | |
| *** jcapitao_lunch is now known as jcapitao | 11:28 | |
| *** jcapitao is now known as jcapitao_lunch | 11:28 | |
| *** rcernin has quit IRC | 11:34 | |
| *** artom has quit IRC | 11:35 | |
| *** rcernin has joined #openstack-infra | 11:39 | |
| *** jpena is now known as jpena|lunch | 11:43 | |
| openstackgerrit | Vishal Manchanda proposed openstack/project-config master: Upadting horizon nodejs job name https://review.opendev.org/737457 | 11:43 |
| openstackgerrit | Vishal Manchanda proposed openstack/project-config master: Upadting horizon nodejs job name https://review.opendev.org/737457 | 11:44 |
| *** rlandy has joined #openstack-infra | 11:45 | |
| *** rlandy is now known as rlandy|ruck | 11:46 | |
| *** ociuhandu has joined #openstack-infra | 11:52 | |
| *** markvoelker has joined #openstack-infra | 11:55 | |
| *** ysandeep is now known as ysandeep|brb | 11:58 | |
| *** markvoelker has quit IRC | 11:59 | |
| *** amoralej is now known as amoralej|lunch | 12:06 | |
| *** jcapitao_lunch is now known as jcapitao | 12:07 | |
| *** rcernin has quit IRC | 12:08 | |
| *** rfolco has joined #openstack-infra | 12:11 | |
| *** rpittau|bbl is now known as rpittau | 12:13 | |
| *** ysandeep|brb is now known as ysandeep | 12:14 | |
| *** rosmaita has joined #openstack-infra | 12:17 | |
| openstackgerrit | Jonathan Rosser proposed openstack/project-config master: Refresh openstack-ansible grafana dashboards https://review.opendev.org/737742 | 12:29 |
| *** ryohayakawa has quit IRC | 12:29 | |
| *** jamesmcarthur has joined #openstack-infra | 12:29 | |
| *** rosmaita has left #openstack-infra | 12:29 | |
| *** jamesmcarthur has quit IRC | 12:34 | |
| *** rcernin has joined #openstack-infra | 12:35 | |
| rlandy|ruck | hello - we're seeing an increase in retry_limits. Is there anything we are tracking in this regard? | 12:36 |
| *** soniya29|rover has joined #openstack-infra | 12:37 | |
| *** xek_ has joined #openstack-infra | 12:38 | |
| AJaeger | rlandy|ruck: can you point us to such an issue, please? | 12:40 |
| AJaeger | rlandy|ruck: your commit is too generic to answer anything | 12:41 |
| AJaeger | s/commit/comment/ | 12:41 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Test multiarch release builds https://review.opendev.org/737315 | 12:43 |
| rlandy|ruck | AJaeger: hi - for example ... the gate for tripleo ... the current job at the head of the gate 735815 | 12:44 |
| rlandy|ruck | we also notice this in the following gate jobs: | 12:44 |
| rlandy|ruck | https://review.opendev.org/#/c/734668/ (gate result tripleo-ci-centos-8-scenario004-standaloneRETRY_LIMIT in 33m 46s) | 12:45 |
| rlandy|ruck | https://review.opendev.org/#/c/736215/ tripleo-ci-centos-8-standaloneRETRY_LIMIT in 40m 46s | 12:45 |
| *** jcapitao has quit IRC | 12:45 | |
| rlandy|ruck | and about 6 of those yesterday | 12:45 |
| *** jcapitao has joined #openstack-infra | 12:46 | |
| AJaeger | rlandy|ruck: RETRY_LIMIT normally means that one of the pre runbooks failed, did you check already why? | 12:47 |
| *** lbragstad has joined #openstack-infra | 12:47 | |
| * AJaeger sees no log files for them ;( | 12:47 | |
| rlandy|ruck | AJaeger: correct ... I tried | 12:48 |
| rlandy|ruck | you just get sent back to the cloud page | 12:48 |
| AJaeger | infra-root, is ze10 having problems? | 12:48 |
| AJaeger | Looking at https://zuul.opendev.org/t/openstack/status/change/735815,1 - the RETRY_LIMIT points to ze10 | 12:49 |
| openstackgerrit | Thierry Carrez proposed zuul/zuul-jobs master: upload-git-mirror: check after mirror operation https://review.opendev.org/737533 | 12:49 |
| AJaeger | infra-root, and the other jobs rlandy|ruck mentioned do not have logs at all | 12:49 |
| AJaeger | rlandy|ruck: sorry, can't help further - hope somebody else can help ^ | 12:49 |
| openstackgerrit | Thierry Carrez proposed zuul/zuul-jobs master: upload-git-mirror: check after mirror operation https://review.opendev.org/737533 | 12:50 |
| fungi | i'll take a closer look at ze10 | 12:50 |
| rlandy|ruck | AJaeger: k - we're watching this issue | 12:50 |
| openstackgerrit | Thierry Carrez proposed zuul/zuul-jobs master: upload-git-mirror: check after mirror operation https://review.opendev.org/737533 | 12:51 |
| *** jpena|lunch is now known as jpena | 12:51 | |
| AJaeger | thanks, fungi | 12:53 |
| fungi | system telemetry seems normal, executor logs are showing some tracebacks for ansible 2.9 calling zuul_stream.py (for the console streaming) | 12:56 |
| mwhahaha | anyone aware why we'd be getting "ERROR: InvocationError for command could not find executable tox" for openstack-tox-pep8 on stable/train? | 12:56 |
| mwhahaha | example https://review.opendev.org/#/c/737507/ | 12:56 |
| *** jcapitao has quit IRC | 12:57 | |
| *** jcapitao has joined #openstack-infra | 12:59 | |
| fungi | AJaeger: rlandy|ruck: i'm not sure it's ze10. the first one you mentioned is https://zuul.opendev.org/t/openstack/build/746066b4c7654e01adc5a9aa532a0da3 which ran from ze03 | 13:04 |
| noonedeadpunk | fungi: hi! I guess there should be hold for https://review.opendev.org/#/c/689629/ | 13:05 |
| noonedeadpunk | can you put my keys on it? https://launchpad.net/~noonedeadpunk/+sshkeys | 13:05 |
| rlandy|ruck | fungi: unfortunately since not all retry_limit failures leave logs, it's hard to track down | 13:05 |
| fungi | AJaeger: rlandy|ruck: this one looks like "Ansible complete, result RESULT_UNREACHABLE code None" | 13:05 |
| rlandy|ruck | somehow we saw quite a few of these over the last 24 hours | 13:05 |
| fungi | so maybe we've got nodes disappearing out from under us in some provider, or our executors are having network connectivity issues | 13:06 |
| fungi | i'll check the other one you linked and see if it's the same | 13:06 |
| *** amoralej|lunch is now known as amoralej | 13:08 | |
| rlandy|ruck | fungi: thanks | 13:12 |
| fungi | rlandy|ruck: https://zuul.opendev.org/t/openstack/build/6fd612696eab452d990a8b7ea3765655 was failing the same way, node went unreachable while running tripleo-ci/toci_gate_test.sh | 13:17 |
| fungi | do these jobs really run that in the pre phase? | 13:18 |
| fungi | anyway, that's why you're not getting any logs, they were on the node and the executor stopped being able to communicate with it while the gate test script was running | 13:18 |
| rlandy|ruck | we do run a fair amount in pre | 13:19 |
| rlandy|ruck | we just couldn't nail the increase in retry_limit failures to any particular job so I wanted to check on this channel to see if this was a known problem | 13:20 |
| fungi | rlandy|ruck: did it happen to all be in jobs which run tripleo-ci/toci_gate_test.sh? | 13:22 |
| fungi | or were you hitting it for other jobs too? | 13:22 |
| rlandy|ruck | most of our jobs run that - so yes | 13:22 |
| rlandy|ruck | fungi: I'll watch the numbers today and see if they still increase | 13:23 |
| rlandy|ruck | so far we had two occurrences today | 13:23 |
| rlandy|ruck | which is probably not so out of line | 13:24 |
| corvus | rlandy|ruck, fungi: zuul retries the job if it hits unreachable in any phase | 13:25 |
| fungi | oh, actually this may be getting retried because of the unreachable result from ansible, not due to being in pre phase | 13:25 |
| fungi | corvus: yep, just dawned on me | 13:25 |
| AJaeger | mwhahaha: there's a message by gmann on "GAte Status" in openstack-discuss, please read it | 13:26 |
| mwhahaha | AJaeger: k thnx | 13:26 |
| corvus | rlandy|ruck: keep in mind that the job itself can make the node unreachable (for example, by breaking the network config for the node) | 13:26 |
| fungi | rlandy|ruck: anyway, i wouldn't rule out the possibility that there's something happening as a result of the tripleo-ci/toci_gate_test.sh script which occasionally crashes a node or takes down its network connectivity | 13:27 |
| *** stevebaker has quit IRC | 13:27 | |
| AJaeger | rlandy|ruck: you can also follow the jobs and see logs in real-life to check - best if something is reproducible | 13:27 |
| fungi | yeah, in this case watching the log stream likely doesn't help because the rate of failure is very low | 13:27 |
| *** jcapitao has quit IRC | 13:28 | |
| *** rcernin has quit IRC | 13:29 | |
| rlandy|ruck | corvus: fungi: AJaeger: thanks ... will look through your suggestions. and yes the failure rate is low and sporadic - so it's one of those hard to track down issues | 13:29 |
| *** jcapitao has joined #openstack-infra | 13:31 | |
| mwhahaha | AJaeger: so i read it, but the issue being the openstack-tox-pep8 job since we're not customizing the job, where can we inject ensure-tox | 13:31 |
| *** rlandy|ruck is now known as rlandy|ruck|mtg | 13:31 | |
| *** ociuhandu_ has joined #openstack-infra | 13:31 | |
| fungi | rlandy|ruck|mtg: also i'm seeing those failures from different executors connecting to nodes in different providers, meaning it's not executor-specific or provider-specific | 13:32 |
| fungi | making it increasingly likely it's something the job itself is causing | 13:32 |
| *** ociuhandu has quit IRC | 13:35 | |
| AJaeger | mwhahaha: interesting, pep8 should work - have a link? | 13:36 |
| mwhahaha | AJaeger: https://review.opendev.org/#/c/737507/ | 13:36 |
| * mwhahaha double checks job def | 13:37 | |
| AJaeger | mwhahaha: that is pep8 - and it runs " tox -e linters -- flake8" | 13:37 |
| AJaeger | So, you invoke tox inside tox ;( | 13:37 |
| AJaeger | That's the problem | 13:37 |
| Tengu | inceptox ? | 13:37 |
| AJaeger | Sorry, in a call - hope others can help you how to fix that | 13:37 |
| mwhahaha | fun | 13:37 |
| mwhahaha | yea we can get to it later | 13:38 |
| AJaeger | mwhahaha: others can help as well | 13:39 |
| mwhahaha | i'll also look into it, thanks for the pointer | 13:39 |
| *** jcapitao has quit IRC | 13:43 | |
| fungi | and our ensure-tox role doesn't put a tox executable in the default search path by default, you're expected to template in an ansible variable which has the path to tox (it's in a venv) | 13:45 |
| fungi | there is a parameter we can pass ensure-tox to add a symlink in /usr/local/bin but it might be better to figure out why tox is calling tox | 13:45 |
| *** jcapitao has joined #openstack-infra | 13:49 | |
| *** dklyle has joined #openstack-infra | 13:49 | |
| *** matt_kosut has joined #openstack-infra | 13:51 | |
| openstackgerrit | Aurelien Lourot proposed openstack/project-config master: Add Neutron Arista plugin charm to OpenStack charms https://review.opendev.org/737791 | 13:54 |
| *** matt_kosut has quit IRC | 13:56 | |
| openstackgerrit | Mohammed Naser proposed openstack/project-config master: gerrit: change retired.config acls https://review.opendev.org/737649 | 13:58 |
| mwhahaha | fungi: it works in ussuri+ still which is the weird part. We're calling tox because it's invoking something else via tox as part of the linters. it's likely we can rework it but this seems to be a difference in train vs ussuri+ | 14:05 |
| mwhahaha | if the tox path got set to an env var, we could add that in there but the lack of tox on the default path seems to be an issue | 14:06 |
| mordred | fungi: I kinda think we should maybe set that variable more globally for opendev | 14:11 |
| mordred | I know there's a bunch of inputs to this system - but ensure-tox not resulting in "tox" working seems to be a consistent source of confusion and one that I'm hard-pressed to explain why | 14:12 |
| mwhahaha | the ansible var works, if you're writing an ansible playbook but it doesn't translate well to other things :/ | 14:12 |
| *** jamesmcarthur has joined #openstack-infra | 14:15 | |
| AJaeger | config-core, dragonflow retirement step 1 is ready, please review https://review.opendev.org/#/c/737566/ | 14:20 |
| *** smarcet has joined #openstack-infra | 14:26 | |
| *** ysandeep is now known as ysandeep|afk | 14:29 | |
| AJaeger | mnaser: do you want to abandon https://review.opendev.org/737636? That looks not needed anymore... | 14:30 |
| *** ociuhandu_ has quit IRC | 14:32 | |
| *** ociuhandu has joined #openstack-infra | 14:33 | |
| mordred | mwhahaha: yah | 14:43 |
| *** hashar has quit IRC | 14:52 | |
| *** jamesmcarthur has quit IRC | 14:54 | |
| *** jamesmcarthur has joined #openstack-infra | 14:54 | |
| *** ysandeep|afk is now known as ysandeep | 14:56 | |
| clarkb | fungi: corvus: jobs will retry if connectivity fails during any stage but also will retry if pre fails. Are we thinking pre isn't simply failing because post never runs and collects logs? | 14:58 |
| clarkb | running a full tripleo quickstart in pre seems likely to retry due to failure | 14:59 |
| openstackgerrit | Ghanshyam Mann proposed openstack/openstack-zuul-jobs master: Remove greande jobs for EM and oldest stable https://review.opendev.org/737826 | 15:01 |
| corvus | clarkb: fungi said that it failed running toci_gate_test.sh. that script is run in the run phase, not pre. | 15:01 |
| clarkb | oh I interpreted what fungi said about pre to mean that was running in pre. I now see that was a proper question and we're failing in run due to the network connectivity trouble. Got it | 15:02 |
| corvus | clarkb: example (successful) build log: https://zuul.opendev.org/t/openstack/build/31c56b7babe24e3180c2775c216af566/console | 15:02 |
| clarkb | as a side note, nested virt from tripleo jobs has caused similar failures in the past | 15:03 |
| clarkb | I have no idea if nested virt is in use again, but we definitely had crashing test nodes in these jobs once upon a time when it was used | 15:03 |
| openstackgerrit | Ghanshyam Mann proposed openstack/openstack-zuul-jobs master: Remove greande jobs for EM and oldest stable https://review.opendev.org/737826 | 15:03 |
| openstackgerrit | Ghanshyam Mann proposed openstack/openstack-zuul-jobs master: Remove greande jobs for EM and oldest stable https://review.opendev.org/737826 | 15:06 |
| *** ysandeep is now known as ysandeep|away | 15:09 | |
| *** jamesmcarthur has quit IRC | 15:12 | |
| *** jamesmcarthur has joined #openstack-infra | 15:13 | |
| *** Goneri has joined #openstack-infra | 15:19 | |
| *** rlandy|ruck|mtg is now known as rlandy|ruck | 15:22 | |
| *** jamesmcarthur has quit IRC | 15:23 | |
| *** lmiccini has quit IRC | 15:24 | |
| *** priteau has quit IRC | 15:27 | |
| mwhahaha | we stopped doing nested virt (or should have) | 15:30 |
| mwhahaha | specifically to avoid that | 15:31 |
| mwhahaha | https://928100af618c7dde602b-73eca520d44e7bdb73532e4ac34bedaa.ssl.cf2.rackcdn.com/736215/5/check/tripleo-ci-centos-8-standalone/31c56b7/logs/undercloud/home/zuul/standalone_parameters.yaml | 15:31 |
| mwhahaha | NovaComputeLIbvirtType: qemu should prevent nested virt | 15:32 |
| clarkb | k, was just calling that out as it caused similar symptoms in the past | 15:32 |
| mwhahaha | sure, just checking we didn't accidently switch back | 15:32 |
| clarkb | is it possible to make quickstart logging stream to the console (if it isn't already)? Then you could open the console logs of running jobs and see if you can catch one failing | 15:33 |
| mwhahaha | it does stream to console | 15:34 |
| clarkb | that may then offer a clue to where it is breaking and we can work from there to figure out how best to debug further | 15:34 |
| *** yamamoto has quit IRC | 15:34 | |
| clarkb | we can also probably pull up the ansible logs on the executor | 15:34 |
| clarkb | though it may not say much more than "ran this playbook and I lost connectivity" | 15:34 |
| clarkb | oh ya I think fungi may have already done that | 15:35 |
| openstackgerrit | Sean McGinnis proposed openstack/project-config master: Make tox global for update proposal jobs https://review.opendev.org/737836 | 15:36 |
| clarkb | and ya checking the one on ze03 that fungi found its basically that it runs bash -xe $TRIPLEO_ROOT/tripleo-ci/toci_gate_test.sh then the node becomes unreachable | 15:36 |
| mwhahaha | fun with crashing :/ | 15:36 |
| fungi | the other example was identical, except different executor, different job, different change, and ran on a node ni a different provider | 15:41 |
| fungi | but still became unreachable during the toci_gate_test.sh task | 15:41 |
| openstackgerrit | Ghanshyam Mann proposed openstack/openstack-zuul-jobs master: Remove greande jobs for EM and oldest stable https://review.opendev.org/737826 | 15:41 |
| prometheanfire | is https://review.opendev.org/736194 the reason why stable branches are failing? | 15:42 |
| *** matt_kosut has joined #openstack-infra | 15:43 | |
| clarkb | prometheanfire: which stable branch jobs are failing? that change should only affect centos devstack jobs | 15:45 |
| clarkb | (of which there are not many aiui) | 15:46 |
| prometheanfire | clarkb: I think train an earlier | 15:53 |
| prometheanfire | though the last rechecks makes me think something was fixed | 15:53 |
| prometheanfire | I guess I'll give it a day | 15:53 |
| clarkb | prometheanfire: well there were all sorts of devstack and grenade fires related to uwsgi and pip and virtualenv stuff last week | 15:53 |
| *** amoralej is now known as amoralej|off | 15:53 | |
| clarkb | the bulk of them for non EM branches should be fixed | 15:53 |
| prometheanfire | cool | 15:54 |
| clarkb | gmann is probably the best resource for keeping on top of what is left (I think its largely just EM branches) | 15:54 |
| prometheanfire | ya, EM stuff is being disabled | 15:54 |
| gmann | prometheanfire: clarkb EM, gate s still blocked and need this which again need more work and debug - https://review.opendev.org/#/c/735615/ | 15:55 |
| *** ricolin has joined #openstack-infra | 15:55 | |
| gmann | non-EM will be up with these start merging - https://review.opendev.org/#/q/topic:grenade-em-nv+(status:open+OR+status:merged) | 15:55 |
| AJaeger | clarkb: 737826 would help gmann as well - please review | 15:56 |
| gmann | yeah, about to write that, thanks AJaeger | 15:56 |
| AJaeger | ;) | 15:56 |
| *** gyee has joined #openstack-infra | 15:57 | |
| *** ociuhandu has quit IRC | 15:59 | |
| openstackgerrit | Thierry Carrez proposed zuul/zuul-jobs master: upload-git-mirror: check after mirror operation https://review.opendev.org/737533 | 16:02 |
| clarkb | gmann: AJaeger the regex there has a bug. Trying to write it in an understandable way now (regexes are so much fun) | 16:03 |
| AJaeger | clarkb: thanks for catching that | 16:03 |
| AJaeger | clarkb: I'm seeing some extra () but they don't harm - curious to see the real problem... I removed my +2 for now | 16:04 |
| *** yamamoto has joined #openstack-infra | 16:04 | |
| clarkb | oh wait heh ya I think its more harmless than I thought at first glance | 16:05 |
| clarkb | I was thinking we were matching queens, rocky, stein without stable/ prefix | 16:06 |
| *** mihalis68_ has quit IRC | 16:06 | |
| clarkb | but we are. I thinkw e should fix it now if we can as it will make it easier to understand later | 16:06 |
| AJaeger | yes, let's fix it. | 16:06 |
| openstackgerrit | Ghanshyam Mann proposed openstack/openstack-zuul-jobs master: Remove greande jobs for EM and oldest stable https://review.opendev.org/737826 | 16:07 |
| gmann | clarkb: AJaeger updated ^^ | 16:08 |
| clarkb | +2'd | 16:09 |
| *** yamamoto has quit IRC | 16:10 | |
| zbr | how to i figure out where does a job comes from? (template), still trying to figureout how to remove opestack-tox-pep8 from tripleo-common, as we already have linters one. | 16:13 |
| *** ociuhandu has joined #openstack-infra | 16:13 | |
| clarkb | zbr: if you pull up the job logs for a pep8 job that ran then look at its zuul inventory there should be a configuration "history" there | 16:14 |
| clarkb | and that should show you where it originates from (including the branch infO) | 16:14 |
| *** ysandeep|away is now known as ysandeep | 16:16 | |
| fungi | zbr: an example: https://zuul.opendev.org/t/openstack/build/b3f1832a72a4422c8c12f06ef2c4fe5d/log/zuul-info/inventory.yaml#33-40 | 16:18 |
| fungi | <Job openstack-tox branches: {MatchAny:{BranchMatcher:^(?!stable/(ocata|pike|queens|rocky)).*$}} source: openstack/openstack-zuul-jobs/zuul.d/jobs.yaml@master#47> | 16:19 |
| zbr | clarkb: thanks. i think i got an idea. | 16:20 |
| zbr | apparently culprits like openstack-python3-ussuri-jobs or openstack-python3-train-jobs enforce us the pep8 job | 16:22 |
| zbr | that is quite problematic as it prevents us from removing the job | 16:23 |
| zbr | probably it would be easier to just replace the job command with a "true" | 16:23 |
| *** markvoelker has joined #openstack-infra | 16:24 | |
| *** jcapitao has quit IRC | 16:24 | |
| zbr | any other ideas? | 16:24 |
| clarkb | you can remove those templates from your project | 16:25 |
| clarkb | possibly update the template to apply those jobs only to ussuri and train then let it die off as the branches age | 16:26 |
| *** ysandeep is now known as ysandeep|away | 16:26 | |
| zbr | well, is very easy to replace pep8 job with linters, and force any consumer to update their tox file. not sure how many are there | 16:26 |
| *** markvoelker has quit IRC | 16:28 | |
| zbr | codesearch reports 163 files, so not a chance. | 16:28 |
| zbr | his setup seems to be be bit rotten as it prevents any project from evolving because there are 160 others that need to do it at the same time. | 16:29 |
| *** derekh has quit IRC | 16:30 | |
| AJaeger | clarkb: could you put https://review.opendev.org/#/c/737566/1 for dragonflow retirement on your review queue, please? | 16:35 |
| prometheanfire | can people take a quick look at this glean change (simple and tested) https://review.opendev.org/737325 | 16:37 |
| AJaeger | ianw, fungi, clarkb, based on ianw's feedback, here's my proposal on the py38 job templates: https://review.opendev.org/737701. Please review. Happy to rename some templates as well or use another option... | 16:37 |
| fungi | zbr: the likely way to solve it would be to introduce an alternate template which switches out that job, get tc approval for the transition, and then projects can switch from the old template to the new one as they get time | 16:38 |
| fungi | and the tc can assist with getting bulk-submitted changes approved if bypassing semi-inactive teams is necessary | 16:39 |
| fungi | zbr: what's preventing projects from "evolving" there is tc policy | 16:39 |
| fungi | regarding required project-templates for the openstack project testing interface | 16:40 |
| *** rpittau is now known as rpittau|afk | 16:41 | |
| openstackgerrit | Merged openstack/openstack-zuul-jobs master: Remove greande jobs for EM and oldest stable https://review.opendev.org/737826 | 16:42 |
| clarkb | AJaeger: +W on dragonflow. Looking at the template change now | 16:45 |
| clarkb | AJaeger: ya I think that template chnage works | 16:46 |
| clarkb | basically be ocnsistent across python verisons and we're good | 16:46 |
| *** Lucas_Gray has quit IRC | 16:49 | |
| openstackgerrit | Merged openstack/project-config master: Make tox global for update proposal jobs https://review.opendev.org/737836 | 16:49 |
| *** eolivare has quit IRC | 17:01 | |
| openstackgerrit | Merged openstack/project-config master: Retire dragonflow project https://review.opendev.org/737566 | 17:01 |
| *** smarcet has quit IRC | 17:03 | |
| *** jpena is now known as jpena|off | 17:03 | |
| *** lastmikoi has quit IRC | 17:06 | |
| *** lastmikoi has joined #openstack-infra | 17:06 | |
| AJaeger | clarkb: exactly | 17:07 |
| *** dtantsur is now known as dtantsur|afk | 17:08 | |
| *** markvoelker has joined #openstack-infra | 17:09 | |
| *** markvoelker has quit IRC | 17:14 | |
| *** xek__ has joined #openstack-infra | 17:16 | |
| *** nightmare_unreal has quit IRC | 17:18 | |
| *** xek_ has quit IRC | 17:19 | |
| *** ricolin has quit IRC | 17:20 | |
| *** apetrich has quit IRC | 17:20 | |
| *** apetrich has joined #openstack-infra | 17:24 | |
| *** d34dh0r53 has joined #openstack-infra | 17:25 | |
| openstackgerrit | Merged openstack/project-config master: gerrit: change retired.config acls https://review.opendev.org/737649 | 17:25 |
| *** gfidente is now known as gfidente|afk | 17:26 | |
| fungi | mnaser: now that's merged ^ do you have a list of the repos i should switch active through the gerrit api? | 17:29 |
| mnaser | fungi: i can generate something right now, one second! | 17:33 |
| fungi | there's no rush, but i'm happy to process it when you have it | 17:33 |
| mnaser | fungi: http://paste.openstack.org/show/795172/ | 17:38 |
| mnaser | feel free to test one at a time or so :) -- and maybe after that i guess we have to run manage-projects manually | 17:38 |
| mnaser | wait, oops, hang on | 17:38 |
| mnaser | fungi: http://paste.openstack.org/show/795173/ had the wrong hostname :) | 17:39 |
| *** d34dh0r53 has quit IRC | 17:43 | |
| *** priteau has joined #openstack-infra | 17:48 | |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Test multiarch release builds https://review.opendev.org/737315 | 17:57 |
| fungi | mnaser: done, all 487 have been switched active now. had to sed -i s/proejct/project/ but otherwise good | 17:58 |
| mnaser | fungi: oh woops, but yay, progress. should we run manage-projects now and make sure the acls apply? | 17:59 |
| fungi | i believe the next m-p run will do that for us, but if it doesn't i can run it manually | 17:59 |
| clarkb | note that may take some time | 18:02 |
| clarkb | since 487 is not a small number | 18:02 |
| fungi | just the acl calls took a while to complete | 18:03 |
| mnaser | oh i thought manage-projects runs on a cronjob | 18:03 |
| mnaser | s/runs on a cronjob/runs in a deploy pipeline/ | 18:03 |
| fungi | it does, so should get triggered by 737649 merging | 18:03 |
| fungi | er, i meant just the api calls took a while to complete, that was probably a confusing typo | 18:04 |
| *** apetrich has quit IRC | 18:15 | |
| *** apetrich has joined #openstack-infra | 18:16 | |
| *** priteau has quit IRC | 18:16 | |
| *** vishalmanchanda has quit IRC | 18:17 | |
| *** xek__ has quit IRC | 18:24 | |
| *** irclogbot_3 has quit IRC | 18:34 | |
| *** irclogbot_3 has joined #openstack-infra | 18:38 | |
| *** smarcet has joined #openstack-infra | 18:43 | |
| *** yamamoto has joined #openstack-infra | 18:53 | |
| *** yamamoto has quit IRC | 18:58 | |
| *** maysams has joined #openstack-infra | 19:07 | |
| *** markvoelker has joined #openstack-infra | 19:10 | |
| *** markvoelker has quit IRC | 19:15 | |
| maysams | Hello Folks, does anyone knows if the server that provides internal DNS addresses to resolve domains for the VMs running on the CI is also IPv6? | 19:21 |
| *** ralonsoh has quit IRC | 19:26 | |
| *** Goneri has quit IRC | 19:30 | |
| fungi | maysams: can you clarify? as in does it listen on the ipv6 loopback address (::1)? | 19:31 |
| fungi | or are you asking whether it returns aaaa (ipv6 address) records for requests where those are available? | 19:31 |
| clarkb | also we dont run internal dns | 19:33 |
| clarkb | its just forwards and caches public dns | 19:33 |
| fungi | by default the virtual machnies are configured in /etc/resolv.conf to use an unbound service on the ipv4 loopback address for recursion, and it forwards (with local caching) to opendns and google resolvers, either over ipv6 (if v6 global egress is available ni that provider) or v4 | 19:33 |
| *** markvoelker has joined #openstack-infra | 19:33 | |
| *** markvoelker has quit IRC | 19:38 | |
| maysams | fungi: Yes, if it listen on the ipv6 address. I'm questioning this cause there is a need to add internal coredns service in our CI and we don't want to run it directly on the host to not have a DNS resolver to the internet | 19:39 |
| maysams | as that coredns svc would first attempt to lookup it's own records and then ask the server that provides internal dns addresses, we would need to check if that server would also require ipv6 depending on the cloud | 19:42 |
| *** hamalq has joined #openstack-infra | 19:43 | |
| maysams | thanks for the info, fungi and clarkb | 19:44 |
| clarkb | I believe unbound listens on 127.0.0.1 and ::1 | 19:44 |
| clarkb | whether or not your coredns needs to listen on ipv6 depends on whetheror not you have ipv6 dns clients | 19:45 |
| maysams | okay, thanks | 19:49 |
| maysams | cc dulek | 19:49 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Test multiarch release builds https://review.opendev.org/737315 | 19:54 |
| fungi | however i don't think we put ::1 into /etc/resolv.conf (i could be wrong) | 19:58 |
| fungi | granted, from the perspective of the primary network namespace they're all just addresses bound to lo0 | 19:59 |
| *** markvoelker has joined #openstack-infra | 19:59 | |
| fungi | er, lo not lo0 (i spend too much time fiddling with *bsd it seems) | 19:59 |
| *** markvoelker has quit IRC | 20:04 | |
| clarkb | fungi: yup I think resolv.conf is 127.0.0.1 only | 20:12 |
| clarkb | but it listens on both | 20:12 |
| *** stevebaker has joined #openstack-infra | 20:14 | |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Test multiarch release builds https://review.opendev.org/737315 | 20:19 |
| clarkb | fungi: a change currently in deploy has a failing manage-projects job | 20:23 |
| clarkb | fungi: you may want to check that given the acl updates? | 20:23 |
| *** matt_kosut has quit IRC | 20:25 | |
| *** matt_kosut has joined #openstack-infra | 20:26 | |
| *** priteau has joined #openstack-infra | 20:28 | |
| *** matt_kosut has quit IRC | 20:30 | |
| *** smarcet has quit IRC | 20:31 | |
| openstackgerrit | Merged openstack/project-config master: grafyaml: drop python2 jobs https://review.opendev.org/737666 | 20:34 |
| *** smarcet has joined #openstack-infra | 20:37 | |
| *** priteau has quit IRC | 20:39 | |
| noonedeadpunk | fungi: sorry for bothering you twice, but is hold for https://review.opendev.org/#/c/689629/ ? | 20:40 |
| noonedeadpunk | can you kindly put my key on it? https://launchpad.net/~noonedeadpunk/+sshkeys | 20:40 |
| clarkb | noonedeadpunk: I can get it | 20:41 |
| noonedeadpunk | clarkb: it should be for `openstack-ansible-deploy-aio_distro_metal-centos-8` job | 20:42 |
| clarkb | noonedeadpunk: root@2607:ff68:100:54:f816:3eff:fea8:e6d2 | 20:43 |
| noonedeadpunk | oh.... | 20:43 |
| noonedeadpunk | is there ipv4 on it? | 20:43 |
| clarkb | only NAT'd its not publicly routable | 20:43 |
| * noonedeadpunk has no ipv6 :( | 20:44 | |
| clarkb | noonedeadpunk: vexxhost instances have ipv6 | 20:44 |
| clarkb | you could bounce through one | 20:44 |
| clarkb | (I do similar since my home isp has no ipv6) | 20:44 |
| noonedeadpunk | yeah, nice idea | 20:46 |
| *** rfolco has quit IRC | 20:47 | |
| fungi | clarkb: tthanks for the heads up, will take a look as soon as i finish cooking dinner | 20:48 |
| *** smarcet has quit IRC | 21:02 | |
| *** smarcet has joined #openstack-infra | 21:12 | |
| *** dchen has quit IRC | 21:25 | |
| fungi | okay, kitchen duties are out of the way... looking into manage-projects errors now | 21:29 |
| fungi | looks like /var/log/manage_projects.log was last updated in april, so we're not redirecting output there any longer | 21:30 |
| fungi | i guess the infra-prod-manage-projects zuul job is now where it's at | 21:32 |
| fungi | for reference, it's been either failing or timing out since 2020-06-19 03:17:11 | 21:33 |
| fungi | so long before the recent un-retirement of openstack repos | 21:33 |
| fungi | the earliest failure is https://zuul.opendev.org/t/openstack/build/ef79b73c7ab143a5bf365b98afc72183 | 21:35 |
| fungi | looks like we redirect stdout to /var/log/ansible/manage-projects.yaml.log on bridge.o.o now | 21:35 |
| *** slaweq has quit IRC | 21:36 | |
| clarkb | fungi: yes all of those jobs should go in a /var/log/ansible/$file based on the job name | 21:36 |
| fungi | 409 Client Error: Conflict for url: https://localhost:3000/api/v1/org/x/repos | 21:37 |
| fungi | fatal: [gitea08.opendev.org]: FAILED! | 21:37 |
| fungi | i wonder if 2020-06-19 roughly coincides with the gitea upgrade | 21:37 |
| fungi | looks like it's org creation failing against gitea servers | 21:38 |
| fungi | perhaps something in the latest version make org creation no longer idempotent | 21:38 |
| fungi | it's the "gitea-git-repos : Create Gitea Repos and Org" task which is failing against all 8 gitea servers | 21:39 |
| fungi | doesn't seem to be the gitea upgrade, we didn't merge that until 2020-06-22 | 21:41 |
| fungi | so this was already failing several days earlier | 21:42 |
| fungi | nothing suspect merged in system-config around that timeframe | 21:43 |
| fungi | moving to #opendev | 21:43 |
| clarkb | gitea01 was upgraded earlier but thats gitea08 | 21:45 |
| clarkb | we do test org and project creation but not that recreation noops in the gate | 21:45 |
| clarkb | fungi: do we get a python traceback from that? | 21:46 |
| clarkb | its running a python module | 21:47 |
| *** slaweq has joined #openstack-infra | 21:47 | |
| fungi | none in the ansible stdout there anyway | 21:49 |
| fungi | but this doesn't seem to be openstack-only so better to continue in #opendev | 21:50 |
| *** slaweq has quit IRC | 21:52 | |
| *** markvoelker has joined #openstack-infra | 22:00 | |
| *** rfolco has joined #openstack-infra | 22:04 | |
| *** markvoelker has quit IRC | 22:05 | |
| *** Tengu has quit IRC | 22:05 | |
| *** Tengu has joined #openstack-infra | 22:07 | |
| *** rlandy|ruck is now known as rlandy|ruck|bbl | 22:22 | |
| *** rfolco has quit IRC | 22:27 | |
| *** pkopec has quit IRC | 22:33 | |
| *** rcernin has joined #openstack-infra | 22:42 | |
| *** markvoelker has joined #openstack-infra | 22:49 | |
| *** tkajinam has joined #openstack-infra | 22:51 | |
| *** markvoelker has quit IRC | 22:54 | |
| *** tosky has quit IRC | 23:03 | |
| *** smarcet has quit IRC | 23:11 | |
| *** rfolco has joined #openstack-infra | 23:23 | |
| *** markvoelker has joined #openstack-infra | 23:37 | |
| *** markvoelker has quit IRC | 23:47 | |
| *** lbragstad has quit IRC | 23:48 | |
| *** dchen has joined #openstack-infra | 23:52 | |
| *** ryohayakawa has joined #openstack-infra | 23:58 | |
| *** ryohayakawa has quit IRC | 23:58 | |
| *** ryohayakawa has joined #openstack-infra | 23:58 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!