| *** apetrich has quit IRC | 00:00 | |
| mwhahaha | it's unlikely that the run dlrn was the cause but if you want an example let me go fetch that | 00:00 |
|---|---|---|
| *** apetrich has joined #openstack-infra | 00:01 | |
| clarkb | ya it may not be the cause but maybe it complains about networking or slow disk io (or io errors) or something | 00:01 |
| mwhahaha | http://logs.openstack.org/43/502943/25/check/tripleo-ci-centos-7-undercloud-oooq/e425916/job-output.txt.gz#_2018-01-09_13_13_09_761783 | 00:01 |
| *** pahuang has joined #openstack-infra | 00:01 | |
| mwhahaha | same change, just the check version | 00:01 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Add alembic.ini https://review.openstack.org/532260 | 00:02 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Fix error handling for pidfile https://review.openstack.org/532206 | 00:02 |
| clarkb | mwhahaha: so there won't be any logs in the one that timed out there? | 00:02 |
| mwhahaha | no | 00:02 |
| mwhahaha | cause they couldn't be fetched | 00:02 |
| mwhahaha | http://logs.openstack.org/43/502943/25/gate/tripleo-ci-centos-7-undercloud-oooq/26ab603/job-output.txt.gz#_2018-01-09_22_36_32_085205 | 00:02 |
| *** aeng has quit IRC | 00:03 | |
| clarkb | mwhahaha: what is interesting about that is it does show that toci seems to have timed itself out | 00:03 |
| clarkb | so I don't think zuul is at fault here | 00:03 |
| mwhahaha | no i don't think it's zuul, seems cloud related | 00:03 |
| clarkb | which does point the finger at something happening on the test node | 00:03 |
| clarkb | ya | 00:03 |
| mwhahaha | is a rainy day in the cloud :D | 00:03 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Move CRD tests to test_gerrit_legacy_crd https://review.openstack.org/531886 | 00:04 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Add skipped CRD tests https://review.openstack.org/531887 | 00:04 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: WIP: Support cross-source dependencies https://review.openstack.org/530806 | 00:04 |
| mwhahaha | here's another one from a TIMED_OUT, the node just went poof it looks like | 00:05 |
| mwhahaha | http://logs.openstack.org/09/532009/2/check/tripleo-ci-centos-7-scenario003-multinode-oooq-container/6ba31e7/job-output.txt.gz#_2018-01-09_05_47_34_604283 | 00:05 |
| mwhahaha | that's from this morning | 00:06 |
| clarkb | mwhahaha: both of those are in inap | 00:06 |
| mwhahaha | http://logs.openstack.org/09/532009/2/check/tripleo-ci-centos-7-scenario002-multinode-oooq-container/7b75b42/job-output.txt.gz also timed_out | 00:06 |
| clarkb | mgagne: ^ any known problems there? | 00:06 |
| mwhahaha | was a failure but the node went away | 00:06 |
| mwhahaha | some TIMED_OUT in puppet as well today, http://logs.openstack.org/09/527509/8/check/puppet-openstack-integration-4-scenario004-tempest-centos-7/27ed44e/ | 00:07 |
| *** xarses_ has quit IRC | 00:08 | |
| mwhahaha | also had some POST_FAILURES too, http://logs.openstack.org/50/487450/77/check/tripleo-ci-centos-7-undercloud-oooq/1f7b49a/job-output.txt.gz | 00:08 |
| mwhahaha | http://logs.openstack.org/09/527509/8/check/puppet-openstack-integration-4-scenario003-tempest-centos-7/4ea0fec/ | 00:08 |
| clarkb | http://logs.openstack.org/09/527509/8/check/puppet-openstack-integration-4-scenario004-tempest-centos-7/27ed44e/job-output.txt.gz#_2018-01-09_16_13_11_139588 hard to know if that timed out or if it was just slow | 00:08 |
| mwhahaha | that one was slow | 00:09 |
| mwhahaha | like seriously slow | 00:09 |
| clarkb | ya | 00:09 |
| mwhahaha | our timeouts are 1.5 hours and it's a 40-50min job | 00:09 |
| clarkb | just pointing out it may be a different issue there | 00:09 |
| clarkb | inap appears to be node goes away | 00:09 |
| mwhahaha | yea, resulted in a TIMED_OUT though | 00:10 |
| clarkb | I also have no idea how much clouds patching is going to affect us | 00:10 |
| mwhahaha | oh i guess that's the difference since that's zuulv3 | 00:10 |
| clarkb | but in theory that could be a fairly major impact | 00:10 |
| mwhahaha | true | 00:10 |
| *** claudiub has quit IRC | 00:10 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/git-review master: Add bindep.txt https://review.openstack.org/532364 | 00:11 |
| ianw | clarkb: ^ lack of java i'd say ... might need a few rounds but that's the gist | 00:11 |
| dmsimard | clarkb: re: syntax errors, without looking I'd say that Ansible syntax check happens prior to starting the playbook so Ansible doesn't send any callback hooks to ARA about it. It'd perhaps be an improvement opportunity for upstream Ansible to have a hook on syntax error. | 00:13 |
| dmsimard | I'll try to look later. | 00:13 |
| mwhahaha | here's a POST_FAILURE that just came through, job was ok but trying to grab logs failed http://logs.openstack.org/61/531261/12/check/tripleo-ci-centos-7-containers-multinode/569efba/job-output.txt.gz#_2018-01-09_23_32_17_379770 | 00:15 |
| *** caphrim007_ has joined #openstack-infra | 00:18 | |
| *** e0ne has quit IRC | 00:18 | |
| *** caphrim00_ has joined #openstack-infra | 00:20 | |
| *** caphrim007 has quit IRC | 00:21 | |
| *** caphrim007_ has quit IRC | 00:23 | |
| *** rlandy has quit IRC | 00:24 | |
| *** caphrim00_ has quit IRC | 00:25 | |
| openstackgerrit | Zara proposed openstack-infra/storyboard master: Remove vagrant setup https://review.openstack.org/532367 | 00:33 |
| *** GregHous- has quit IRC | 00:34 | |
| *** jamesmcarthur has joined #openstack-infra | 00:35 | |
| *** rosmaita_ has quit IRC | 00:36 | |
| *** andymccr has quit IRC | 00:36 | |
| *** GregHouse has joined #openstack-infra | 00:36 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool feature/zuulv3: builder: do not cleanup image for driver not managing image https://review.openstack.org/516920 | 00:38 |
| openstackgerrit | Clark Boylan proposed openstack-infra/git-review master: Fix compare_review's use of fetch_review https://review.openstack.org/532359 | 00:38 |
| ianw | there seem to be a lot of jobs seemingly sitting in zuul assigned but no progress bar ... | 00:38 |
| pabelanger | zuul merging things? | 00:39 |
| *** jamesmcarthur has quit IRC | 00:39 | |
| *** felipemonteiro has quit IRC | 00:40 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool feature/zuulv3: Implement a static driver for Nodepool https://review.openstack.org/468624 | 00:40 |
| *** andymccr has joined #openstack-infra | 00:40 | |
| *** markvoelker has joined #openstack-infra | 00:43 | |
| *** corvus has quit IRC | 00:44 | |
| *** markvoelker has quit IRC | 00:45 | |
| ianw | pabelanger: not sure, is node requests in http://grafana01.openstack.org/dashboard/db/zuul-status at 0 a good thing or a bad thing? | 00:46 |
| *** wolverineav has quit IRC | 00:49 | |
| *** markvoelker has joined #openstack-infra | 00:49 | |
| *** markvoelker has quit IRC | 00:50 | |
| *** andymccr has quit IRC | 00:51 | |
| *** xarses has joined #openstack-infra | 00:55 | |
| *** andymccr has joined #openstack-infra | 00:55 | |
| *** david-lyle has joined #openstack-infra | 00:58 | |
| *** hichihara has joined #openstack-infra | 01:02 | |
| ianw | ok, scrolling down, zuul apparently has 0 executors attached | 01:02 |
| ianw | http://grafana01.openstack.org/dashboard/db/zuul-status?panelId=21&fullscreen | 01:03 |
| ianw | which would probably explain why nothing is happening | 01:03 |
| *** cuongnv has joined #openstack-infra | 01:04 | |
| clarkb | ianw: did they all get nuked by the memory protection fault again? | 01:04 |
| clarkb | I'm currently distracted sorting out meltdown patching | 01:04 |
| clarkb | join us in #openstack-infra-incident if interested (though fixing zuul also a good thing) | 01:05 |
| *** hjensas has joined #openstack-infra | 01:05 | |
| ianw | [Wed Jan 10 00:13:19 2018] zuul-executor[32228]: segfault at a9 ip 0000000000513ef4 sp 00007f50e4ff68b8 error 4 in python3.5[400000+3a9000] | 01:06 |
| ianw | on ze01 | 01:06 |
| ianw | that's not so good | 01:06 |
| *** aviau has quit IRC | 01:07 | |
| *** aviau has joined #openstack-infra | 01:07 | |
| *** dave-mccowan has joined #openstack-infra | 01:08 | |
| *** markvoelker has joined #openstack-infra | 01:08 | |
| *** markvoelker has quit IRC | 01:09 | |
| *** rmcall has joined #openstack-infra | 01:10 | |
| ianw | i've restarted ze01 | 01:15 |
| ianw | same segfault ze02 | 01:16 |
| *** numans has quit IRC | 01:16 | |
| *** kiennt26 has quit IRC | 01:16 | |
| ianw | what could cause an instantaneous executor wide segfault :/ | 01:16 |
| *** kiennt26 has joined #openstack-infra | 01:17 | |
| clarkb | ianw: loading the same yaml update? | 01:17 |
| *** numans has joined #openstack-infra | 01:17 | |
| clarkb | my guess is something in yaml tripped it all up | 01:17 |
| clarkb | hrm corvus is still afk should ping corvus when he rejoins | 01:17 |
| *** daidv has quit IRC | 01:18 | |
| *** daidv has joined #openstack-infra | 01:19 | |
| ianw | alright, i'm just going to go through and restart all 9 | 01:21 |
| clarkb | 10* | 01:21 |
| *** xarses has quit IRC | 01:21 | |
| clarkb | ianw: hey actually | 01:23 |
| *** hongbin has joined #openstack-infra | 01:24 | |
| clarkb | if you haven't started the daemons yet maybe we want to patch their kernels? | 01:24 |
| *** bobh has joined #openstack-infra | 01:24 | |
| *** zhurong has joined #openstack-infra | 01:24 | |
| ianw | :/ about done | 01:24 |
| *** felipemonteiro has joined #openstack-infra | 01:25 | |
| clarkb | oh well | 01:26 |
| *** kraig has joined #openstack-infra | 01:27 | |
| ianw | ok, executors executing again | 01:29 |
| *** rosmaita has joined #openstack-infra | 01:29 | |
| ianw | jeblair: ^ simultaneous segfault if you'd like to dig into it | 01:29 |
| *** sbezverk has left #openstack-infra | 01:31 | |
| *** liujiong has joined #openstack-infra | 01:36 | |
| *** pahuang has quit IRC | 01:36 | |
| *** xarses has joined #openstack-infra | 01:38 | |
| *** dave-mccowan has quit IRC | 01:42 | |
| fungi | ianw: did it perchance correlate to a pip upgrade of any zuul deps according to puppet entries in syslog? | 01:42 |
| *** Apoorva_ has joined #openstack-infra | 01:43 | |
| *** d0ugal has quit IRC | 01:43 | |
| *** erlon has quit IRC | 01:44 | |
| *** caphrim007 has joined #openstack-infra | 01:44 | |
| *** Apoorva has quit IRC | 01:46 | |
| *** pahuang has joined #openstack-infra | 01:47 | |
| *** felipemonteiro_ has joined #openstack-infra | 01:47 | |
| *** Apoorva_ has quit IRC | 01:47 | |
| *** pahuang has quit IRC | 01:48 | |
| *** caphrim007 has quit IRC | 01:48 | |
| *** felipemonteiro__ has joined #openstack-infra | 01:51 | |
| *** felipemonteiro_ has quit IRC | 01:51 | |
| *** felipemonteiro has quit IRC | 01:51 | |
| *** pahuang has joined #openstack-infra | 01:53 | |
| *** felipemonteiro__ has quit IRC | 01:55 | |
| ianw | fungi: good point ... prior run was just a few minutes before http://paste.openstack.org/show/641768/ (segfault @00:13:19) | 01:57 |
| *** iyamahat has joined #openstack-infra | 01:57 | |
| ianw | ok, all nodepool images rebuilt ... it's been a embarrassingly long time :/ | 01:58 |
| *** rfolco has joined #openstack-infra | 01:58 | |
| *** iyamahat has quit IRC | 02:00 | |
| fungi | mildly worried based on the last two incidents that zuul daemons are now crashing at upgrade | 02:01 |
| *** d0ugal has joined #openstack-infra | 02:01 | |
| *** sdague has quit IRC | 02:01 | |
| ianw | probably the idea that we just git pull the code and hope for the best is a bit optimistic | 02:03 |
| *** felipemonteiro__ has joined #openstack-infra | 02:07 | |
| *** david-lyle has quit IRC | 02:07 | |
| *** iyamahat has joined #openstack-infra | 02:07 | |
| *** slaweq has joined #openstack-infra | 02:09 | |
| *** iyamahat has quit IRC | 02:09 | |
| *** dangers_away has joined #openstack-infra | 02:12 | |
| *** slaweq has quit IRC | 02:13 | |
| *** dave-mccowan has joined #openstack-infra | 02:16 | |
| *** mriedem has quit IRC | 02:16 | |
| *** felipemonteiro__ has quit IRC | 02:18 | |
| *** rmcall has quit IRC | 02:21 | |
| *** harlowja has quit IRC | 02:24 | |
| *** jeblair has joined #openstack-infra | 02:24 | |
| *** jeblair has quit IRC | 02:24 | |
| *** Apoorva has joined #openstack-infra | 02:29 | |
| *** iyamahat has joined #openstack-infra | 02:34 | |
| gmann | clarkb: fungi: ianw: can you check these. it will minimize the gate time for reno patches - https://review.openstack.org/#/c/531194/ https://review.openstack.org/#/c/531195/ | 02:37 |
| *** HenryG has quit IRC | 02:42 | |
| *** mordred has quit IRC | 02:42 | |
| *** HenryG has joined #openstack-infra | 02:42 | |
| *** mordred has joined #openstack-infra | 02:43 | |
| *** dougwig has quit IRC | 02:44 | |
| *** dave-mccowan has quit IRC | 02:47 | |
| *** lathiat has quit IRC | 02:51 | |
| *** lathiat has joined #openstack-infra | 02:52 | |
| *** Apoorva has quit IRC | 02:52 | |
| *** bobh_ has joined #openstack-infra | 02:53 | |
| *** lbragstad has quit IRC | 02:54 | |
| *** bobh has quit IRC | 02:56 | |
| *** rosmaita has quit IRC | 02:58 | |
| *** ijw has quit IRC | 03:00 | |
| *** ramishra has joined #openstack-infra | 03:10 | |
| *** jamesmcarthur has joined #openstack-infra | 03:19 | |
| *** annp has joined #openstack-infra | 03:20 | |
| *** zhurong has quit IRC | 03:23 | |
| *** jamesmcarthur has quit IRC | 03:24 | |
| *** jascott1 has quit IRC | 03:24 | |
| *** Apoorva has joined #openstack-infra | 03:30 | |
| *** jistr has quit IRC | 03:31 | |
| *** zhurong has joined #openstack-infra | 03:32 | |
| *** ramishra has quit IRC | 03:33 | |
| *** ramishra has joined #openstack-infra | 03:35 | |
| *** puiterwijk has quit IRC | 03:40 | |
| *** jeblair has joined #openstack-infra | 03:40 | |
| *** jeblair has quit IRC | 03:41 | |
| *** rosmaita has joined #openstack-infra | 03:41 | |
| *** DinaBelova has quit IRC | 03:44 | |
| *** andreykurilin has quit IRC | 03:44 | |
| *** andreykurilin has joined #openstack-infra | 03:44 | |
| *** DinaBelova has joined #openstack-infra | 03:46 | |
| clarkb | ianw: hey http://logs.openstack.org/62/531162/5/gate/openstack-tox-pep8/b9ad0ba/ara/result/f121b9fb-2375-4c37-8800-c7edb8cbebbc/ seems to be breaking quite a bit of stuff | 03:49 |
| *** lyarwood has quit IRC | 03:49 | |
| clarkb | ianw: the check in there merged about am onth ago https://review.openstack.org/#/c/525507/ so I'm guessing thsi is a new side effect of image builds. Maybe new tox doesn't have pip installed? | 03:49 |
| clarkb | or no this is global pip I think | 03:49 |
| clarkb | ianw: any chance you are able to debug that? | 03:50 |
| *** lyarwood has joined #openstack-infra | 03:50 | |
| clarkb | I'm running out of steam and need to finish up this infracloud stuff | 03:50 |
| ianw | ok, let me see | 03:52 |
| *** ijw has joined #openstack-infra | 03:52 | |
| *** bobh_ has quit IRC | 03:52 | |
| clarkb | ianw: just look at the failed jobs in the gate pep8 etc | 03:53 |
| *** udesale has joined #openstack-infra | 03:59 | |
| *** adriant has joined #openstack-infra | 04:01 | |
| adriant | is zuul unhappy right now? | 04:02 |
| adriant | I've got some weird build failures and one that appears to be stuck :/ | 04:02 |
| clarkb | adriant: yes, its failing ona check in one of the base tox job features | 04:03 |
| clarkb | I think ianw is looking into it | 04:03 |
| clarkb | adriant: http://logs.openstack.org/62/531162/5/gate/openstack-tox-pep8/b9ad0ba/ara/result/f121b9fb-2375-4c37-8800-c7edb8cbebbc/ | 04:03 |
| ianw | "import pip" doesn't work | 04:04 |
| adriant | yep, that's the error I'm seeing in a few places, and one job appears to be stuck in a pep8 check for hours :P | 04:04 |
| ianw | "from pip import main" | 04:04 |
| ianw | does | 04:04 |
| clarkb | ianw: huh | 04:04 |
| clarkb | oh you know what | 04:05 |
| clarkb | I think pip was changing some of their api stuff. maybe its new release of pip? | 04:05 |
| ianw | nothing's come out since 2016 | 04:06 |
| clarkb | huh | 04:07 |
| *** sree has joined #openstack-infra | 04:07 | |
| *** puiterwijk has joined #openstack-infra | 04:08 | |
| ianw | pip 9.0.1 from /usr/local/lib/python3.5/dist-packages | 04:10 |
| *** sree has quit IRC | 04:11 | |
| clarkb | ianw: is there a traceback or anything when you do import pip? | 04:12 |
| *** jeblair has joined #openstack-infra | 04:12 | |
| ianw | we haven't installed pip v2 in pip-and-virtualenv | 04:13 |
| ianw | :/ | 04:13 |
| clarkb | we only installed pip v3? | 04:13 |
| clarkb | in that case I think we pause image builds (might have to force merge that?) then delete the new images that nodepool made today? | 04:14 |
| ianw | i think it's https://review.openstack.org/#/c/519502/ | 04:14 |
| ianw | yes, i'm running out of time to fix it | 04:14 |
| ianw | greghaynes: ^ i think we *only* install pip3 now, wihch isn't what we want | 04:15 |
| clarkb | ianw: we can pretty quickly revert as long as the old imges are still around. Just need to pause image building first | 04:17 |
| clarkb | ianw: do you want to write that change or should I/ | 04:17 |
| ianw | i can | 04:18 |
| clarkb | I think we'll force merge that then also just quickly hand apply it to the builders then delete the new images | 04:19 |
| *** rosmaita has quit IRC | 04:21 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config master: Pause builds for dib 2.10 release https://review.openstack.org/532392 | 04:22 |
| ianw | clarkb: ^ pretty sure if that just merges before next build time, we'll be ok | 04:22 |
| ianw | i'll delete todays builds | 04:22 |
| clarkb | ianw: I think delting todays builds will cause new builds to start immediately | 04:23 |
| clarkb | since the old ones are more than 24 hours old | 04:23 |
| clarkb | ianw: so I think we want to get ^ in first | 04:23 |
| *** liujiong has quit IRC | 04:24 | |
| *** jeblair has quit IRC | 04:24 | |
| openstackgerrit | Merged openstack-infra/project-config master: Pause builds for dib 2.10 release https://review.openstack.org/532392 | 04:25 |
| clarkb | oh wait thats on nl01 and nl02 not the builders? | 04:26 |
| *** sree has joined #openstack-infra | 04:26 | |
| ianw | it's the same config | 04:26 |
| clarkb | isn't it using nodepool.yaml on the builders? | 04:26 |
| ianw | i think it's getting split up soon | 04:26 |
| clarkb | I think this is an artifact of our v3 switch that isn't entirely complete yet | 04:27 |
| clarkb | the builders still use nodepool.yaml | 04:27 |
| ianw | :/ | 04:27 |
| clarkb | I'll force merge a followup change to do it to the other file | 04:28 |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config master: Pause builds for dib 2.10 https://review.openstack.org/532393 | 04:29 |
| ianw | coming now | 04:29 |
| ianw | clarkb: ^ | 04:29 |
| *** jascott1 has joined #openstack-infra | 04:29 | |
| openstackgerrit | Merged openstack-infra/project-config master: Pause builds for dib 2.10 https://review.openstack.org/532393 | 04:30 |
| ianw | greghaynes: i think the solution here is to make it clear that pip-and-virtualenv sets up a python2 & a python3 environment with the latest | 04:32 |
| ianw | if you only want python3, then it's going to have to be a new element i think, to keep things backwards compatible | 04:33 |
| *** links has joined #openstack-infra | 04:37 | |
| *** hongbin has quit IRC | 04:38 | |
| *** liujiong has joined #openstack-infra | 04:39 | |
| *** zhurong has quit IRC | 04:40 | |
| *** links has quit IRC | 04:40 | |
| *** gouthamr has quit IRC | 04:41 | |
| ianw | clarkb: ok, i gave puppet a kick on there and new config should be deploeyd | 04:41 |
| *** janki has joined #openstack-infra | 04:41 | |
| *** ijw has quit IRC | 04:42 | |
| *** ijw has joined #openstack-infra | 04:42 | |
| clarkb | ianw: and now we delete the images? | 04:43 |
| *** namnh has joined #openstack-infra | 04:44 | |
| *** ijw has quit IRC | 04:45 | |
| *** david-lyle has joined #openstack-infra | 04:46 | |
| ianw | clarkb: yep, done that, they're deleting | 04:46 |
| *** links has joined #openstack-infra | 04:46 | |
| clarkb | ianw: | centos-7-0000008401 | centos-7 | nb04 | qcow2,vhd,raw | building | 00:00:23:06 | <- is one that I think raced | 04:47 |
| clarkb | but otherwise ya looks good | 04:47 |
| ianw | i was a bit fast with that one | 04:47 |
| ianw | yeah, i don't think centos is actually affected by this | 04:47 |
| clarkb | ya depending on what jobs we run there it may not be | 04:47 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Revert "Dont install python-pip for py3k" https://review.openstack.org/532395 | 04:50 |
| ianw | oh i mean the install is only broken on debuntu | 04:50 |
| clarkb | oh gotcha | 04:50 |
| ianw | clarkb / greghaynes: ^^ i think that is the fix, i will pick it back up tomorrow my time | 04:51 |
| clarkb | I too need to start stepping away from the computer | 04:51 |
| clarkb | I probably won't be up very early in the morning | 04:52 |
| ianw | thanks for shepherding the kernel stuff | 04:53 |
| *** david-lyle has quit IRC | 04:53 | |
| clarkb | and thank you for jumping on the zuul issues | 04:54 |
| ianw | there's plenty of fires to go around :) | 04:54 |
| *** Apoorva has quit IRC | 04:58 | |
| *** yamamoto has joined #openstack-infra | 04:58 | |
| *** coolsvap has joined #openstack-infra | 04:59 | |
| clarkb | #status log image builds are paused and we have reverted images to old ones after dib release produced images withou pip for python2. This lack of pip for python2 broke tox siblings in many tox jobs | 04:59 |
| openstackstatus | clarkb: finished logging | 04:59 |
| adriant | clarkb: just curious before you vanish as well, should the jobs mostly clear themselves up or is there a way to give them a kick? | 05:01 |
| adriant | I've got a merge/gate job that appears stuck and I know if theres a way for me to restart it | 05:02 |
| clarkb | adriant: all the jobs should have timeouts | 05:02 |
| clarkb | adriant: once they report back you can recheck | 05:02 |
| adriant | kk | 05:02 |
| adriant | will just head home and recheck it tomorrow | 05:02 |
| *** bhavik1 has joined #openstack-infra | 05:06 | |
| *** bhavik1 has quit IRC | 05:06 | |
| *** bhavik1 has joined #openstack-infra | 05:06 | |
| *** sree has quit IRC | 05:10 | |
| *** sree has joined #openstack-infra | 05:10 | |
| *** sree has quit IRC | 05:12 | |
| *** sree has joined #openstack-infra | 05:12 | |
| *** jeblair has joined #openstack-infra | 05:14 | |
| *** sree has quit IRC | 05:15 | |
| *** sree has joined #openstack-infra | 05:17 | |
| *** caphrim007 has joined #openstack-infra | 05:18 | |
| *** jeblair has quit IRC | 05:20 | |
| *** jeblair has joined #openstack-infra | 05:21 | |
| *** jeblair is now known as corvus | 05:21 | |
| *** sree has quit IRC | 05:23 | |
| *** dhajare has joined #openstack-infra | 05:24 | |
| *** harlowja has joined #openstack-infra | 05:24 | |
| *** bhavik1 has quit IRC | 05:24 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Install ovsdbapp master in neutron + ovsdbapp periodic job https://review.openstack.org/532350 | 05:29 |
| *** yamamoto_ has joined #openstack-infra | 05:29 | |
| *** gouthamr has joined #openstack-infra | 05:32 | |
| *** yamamoto has quit IRC | 05:33 | |
| *** bhavik1 has joined #openstack-infra | 05:35 | |
| *** gyee has quit IRC | 05:36 | |
| *** psachin has joined #openstack-infra | 05:39 | |
| *** markvoelker has joined #openstack-infra | 05:39 | |
| *** shu-mutou-AWAY is now known as shu-mutou | 05:39 | |
| *** sree has joined #openstack-infra | 05:42 | |
| *** bhavik1 has quit IRC | 05:42 | |
| *** dhajare has quit IRC | 05:44 | |
| *** yamamoto has joined #openstack-infra | 05:46 | |
| *** yamamoto_ has quit IRC | 05:49 | |
| *** sree_ has joined #openstack-infra | 05:50 | |
| *** sree_ is now known as Guest40634 | 05:50 | |
| *** sree has quit IRC | 05:50 | |
| *** sandanar has joined #openstack-infra | 05:51 | |
| *** dsariel has joined #openstack-infra | 05:58 | |
| *** edmondsw has joined #openstack-infra | 06:00 | |
| *** jbadiapa has quit IRC | 06:02 | |
| *** edmondsw has quit IRC | 06:04 | |
| *** claudiub has joined #openstack-infra | 06:05 | |
| *** oanson has joined #openstack-infra | 06:06 | |
| *** dingyichen has joined #openstack-infra | 06:08 | |
| *** lathiat has quit IRC | 06:09 | |
| *** dhajare has joined #openstack-infra | 06:09 | |
| *** snapiri has joined #openstack-infra | 06:12 | |
| *** lathiat has joined #openstack-infra | 06:12 | |
| *** zhurong has joined #openstack-infra | 06:16 | |
| *** oanson has quit IRC | 06:18 | |
| *** oanson has joined #openstack-infra | 06:22 | |
| *** gouthamr has quit IRC | 06:23 | |
| *** jbadiapa has joined #openstack-infra | 06:26 | |
| *** pcaruana has joined #openstack-infra | 06:27 | |
| *** dbecker_ has quit IRC | 06:30 | |
| *** jbadiapa has quit IRC | 06:30 | |
| *** pcaruana has quit IRC | 06:33 | |
| *** pcaruana has joined #openstack-infra | 06:34 | |
| *** sree has joined #openstack-infra | 06:40 | |
| *** Guest40634 has quit IRC | 06:41 | |
| *** dbecker_ has joined #openstack-infra | 06:43 | |
| *** sree has quit IRC | 06:44 | |
| *** sree has joined #openstack-infra | 06:44 | |
| *** sree has quit IRC | 06:50 | |
| *** hjensas has quit IRC | 06:54 | |
| *** gcb has quit IRC | 06:55 | |
| *** udesale__ has joined #openstack-infra | 06:57 | |
| *** ramishra has quit IRC | 06:58 | |
| *** udesale has quit IRC | 06:59 | |
| *** dimak has joined #openstack-infra | 07:00 | |
| *** Guest76 is now known as jlk | 07:02 | |
| *** sar has joined #openstack-infra | 07:06 | |
| *** ramishra has joined #openstack-infra | 07:06 | |
| *** sar is now known as sbr | 07:06 | |
| *** sbr is now known as sbra | 07:06 | |
| AJaeger | ianw, frickler, could you review my Zuul v3 native job for bindep, please? https://review.openstack.org/531470 | 07:06 |
| *** masber has quit IRC | 07:08 | |
| *** hichihara has quit IRC | 07:08 | |
| *** slaweq has joined #openstack-infra | 07:09 | |
| *** pcichy has joined #openstack-infra | 07:10 | |
| *** stakeda has joined #openstack-infra | 07:12 | |
| *** slaweq has quit IRC | 07:14 | |
| *** rcernin has quit IRC | 07:15 | |
| *** jbadiapa has joined #openstack-infra | 07:18 | |
| *** sree has joined #openstack-infra | 07:19 | |
| *** udesale has joined #openstack-infra | 07:19 | |
| *** jamesmcarthur has joined #openstack-infra | 07:20 | |
| *** udesale has quit IRC | 07:20 | |
| *** jascott1 has quit IRC | 07:20 | |
| *** jascott1 has joined #openstack-infra | 07:20 | |
| *** udesale__ has quit IRC | 07:21 | |
| *** sree has quit IRC | 07:23 | |
| *** jamesmcarthur has quit IRC | 07:24 | |
| *** makowals has joined #openstack-infra | 07:25 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/jeepyb master: Add checking for PyPI project ownership https://review.openstack.org/531898 | 07:28 |
| *** sree has joined #openstack-infra | 07:29 | |
| *** armaan has joined #openstack-infra | 07:30 | |
| *** sileht has quit IRC | 07:31 | |
| *** sbra has quit IRC | 07:34 | |
| *** sree has quit IRC | 07:34 | |
| *** armax has quit IRC | 07:35 | |
| *** threestrands has quit IRC | 07:36 | |
| *** sree has joined #openstack-infra | 07:38 | |
| *** dingyichen has quit IRC | 07:40 | |
| *** dingyichen has joined #openstack-infra | 07:40 | |
| *** sree has quit IRC | 07:43 | |
| *** slaweq has joined #openstack-infra | 07:43 | |
| *** sree has joined #openstack-infra | 07:44 | |
| *** sbra has joined #openstack-infra | 07:45 | |
| *** armax has joined #openstack-infra | 07:47 | |
| *** harlowja has quit IRC | 07:49 | |
| *** sree has quit IRC | 07:49 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool feature/zuulv3: Do pep8 housekeeping according to zuul rules https://review.openstack.org/522945 | 07:52 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool feature/zuulv3: Use same flake8 config as in zuul https://review.openstack.org/509715 | 07:52 |
| *** thorre_se has joined #openstack-infra | 07:56 | |
| *** thorre has quit IRC | 07:59 | |
| *** thorre_se is now known as thorre | 07:59 | |
| *** armax has quit IRC | 08:01 | |
| *** lathiat has quit IRC | 08:02 | |
| *** thorre_se has joined #openstack-infra | 08:04 | |
| *** lathiat has joined #openstack-infra | 08:04 | |
| *** dsariel has quit IRC | 08:05 | |
| *** thorre has quit IRC | 08:07 | |
| *** thorre_se is now known as thorre | 08:07 | |
| *** armax has joined #openstack-infra | 08:08 | |
| *** shardy has joined #openstack-infra | 08:09 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool feature/zuulv3: handler: fix support for handler without launch_manager https://review.openstack.org/524773 | 08:10 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool feature/zuulv3: Add a plugin interface for drivers https://review.openstack.org/524620 | 08:10 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool feature/zuulv3: builder: do not cleanup image for driver not managing image https://review.openstack.org/516920 | 08:10 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool feature/zuulv3: Implement a static driver for Nodepool https://review.openstack.org/468624 | 08:10 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool feature/zuulv3: Refactor run_handler to be generic https://review.openstack.org/526325 | 08:10 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool feature/zuulv3: Implement an OpenContainer driver https://review.openstack.org/468753 | 08:10 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool feature/zuulv3: Implement a Kubernetes driver https://review.openstack.org/521356 | 08:10 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool feature/zuulv3: Refactor NodeLauncher to be generic https://review.openstack.org/532450 | 08:10 |
| *** pgadiya has joined #openstack-infra | 08:11 | |
| *** jtomasek has joined #openstack-infra | 08:11 | |
| *** tesseract has joined #openstack-infra | 08:14 | |
| *** sree has joined #openstack-infra | 08:16 | |
| *** ralonsoh has joined #openstack-infra | 08:16 | |
| *** florianf has joined #openstack-infra | 08:18 | |
| *** hashar has joined #openstack-infra | 08:19 | |
| *** aviau has quit IRC | 08:19 | |
| *** aviau has joined #openstack-infra | 08:19 | |
| *** sree has quit IRC | 08:20 | |
| *** sree has joined #openstack-infra | 08:21 | |
| *** alexchadin has joined #openstack-infra | 08:24 | |
| *** xarses has quit IRC | 08:25 | |
| *** hjensas has joined #openstack-infra | 08:26 | |
| *** jascott1 has quit IRC | 08:27 | |
| *** alex_xu has quit IRC | 08:28 | |
| *** alex_xu has joined #openstack-infra | 08:29 | |
| *** cshastri has joined #openstack-infra | 08:34 | |
| *** ianychoi has quit IRC | 08:35 | |
| *** pcichy has quit IRC | 08:35 | |
| *** jascott1 has joined #openstack-infra | 08:38 | |
| *** sree has quit IRC | 08:39 | |
| *** sree has joined #openstack-infra | 08:41 | |
| *** iyamahat has quit IRC | 08:44 | |
| *** jpena|off is now known as jpena | 08:45 | |
| *** sree has quit IRC | 08:45 | |
| *** jaosorior has joined #openstack-infra | 08:54 | |
| *** jistr has joined #openstack-infra | 08:57 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: Really change buildset to string https://review.openstack.org/532459 | 08:58 |
| tobiash | corvus, SpamapS: ^^ | 08:58 |
| *** dingyichen has quit IRC | 08:58 | |
| tobiash | the change yesterday had a small bug and missed the according change in the driver metadata and tests | 08:59 |
| *** sree has joined #openstack-infra | 08:59 | |
| *** stakeda has quit IRC | 08:59 | |
| tobiash | I overlooked that the table schema is duplicated in the driver itself and not inferred from the real db | 08:59 |
| tobiash | ups, wrong channel | 09:00 |
| *** ijw has joined #openstack-infra | 09:01 | |
| *** jpich has joined #openstack-infra | 09:02 | |
| *** sree has quit IRC | 09:04 | |
| *** eumel8 has joined #openstack-infra | 09:06 | |
| *** tommylikehu has quit IRC | 09:07 | |
| *** tommylikehu has joined #openstack-infra | 09:08 | |
| *** markvoelker has quit IRC | 09:08 | |
| *** owalsh has quit IRC | 09:09 | |
| *** owalsh has joined #openstack-infra | 09:10 | |
| *** slaweq_ has joined #openstack-infra | 09:11 | |
| *** shu-mutou is now known as shu-mutou-AWAY | 09:12 | |
| *** jpich has quit IRC | 09:14 | |
| *** slaweq_ has quit IRC | 09:15 | |
| *** jpich has joined #openstack-infra | 09:17 | |
| tobiash | infra-root: looks like the images are still/again broken as all zuul jobs are broken currently | 09:18 |
| tobiash | http://logs.openstack.org/59/532459/1/check/tox-pep8/ebd56b0/job-output.txt.gz#_2018-01-10_09_02_06_471237 | 09:18 |
| *** sileht has joined #openstack-infra | 09:20 | |
| *** dsariel has joined #openstack-infra | 09:21 | |
| *** dizquierdo has joined #openstack-infra | 09:23 | |
| *** andreas_s has joined #openstack-infra | 09:30 | |
| *** sree has joined #openstack-infra | 09:30 | |
| *** jpich has quit IRC | 09:31 | |
| *** lucas-afk is now known as lucasagomes | 09:31 | |
| *** e0ne has joined #openstack-infra | 09:34 | |
| *** sree has quit IRC | 09:35 | |
| *** dsariel has quit IRC | 09:35 | |
| *** liujiong has quit IRC | 09:40 | |
| *** dizquierdo has quit IRC | 09:42 | |
| *** sandanar has quit IRC | 09:42 | |
| *** derekh has joined #openstack-infra | 09:42 | |
| *** sandanar has joined #openstack-infra | 09:43 | |
| kmARC | Hey guys, I want to bring the `Upstream Institute`s Virtual Environment generation under the ci system. I already started talking to fungi about these efforts last year. | 09:43 |
| kmARC | Now factored out all the vagrant-based things and I only use the diskimage-builder to create the image. The review is here, if anyone is interested (https://review.openstack.org/#/c/531525/) | 09:43 |
| kmARC | What we want to have is an automated build that publishes an `ova` archive on tarballs.o.o - who can help me with that? :-) | 09:43 |
| kmARC | fungi: ^^ | 09:43 |
| *** jpich has joined #openstack-infra | 09:44 | |
| *** gema has joined #openstack-infra | 09:46 | |
| *** dsariel has joined #openstack-infra | 09:48 | |
| *** kjackal has joined #openstack-infra | 09:51 | |
| gema | fungi: moved the conversation to the ML | 09:52 |
| gema | fungi: will give it a day or so before starting | 09:52 |
| gema | xinliang: ping | 09:52 |
| xinliang | pong ,gema | 09:53 |
| gema | xinliang: you tried nodepool on ARM64 in the past | 09:53 |
| xinliang | yes | 09:53 |
| gema | xinliang: when we left it, were you happy with being able to deploy it and use it? | 09:53 |
| xinliang | gema: a little complicate to configure and use it | 09:54 |
| gema | xinliang: but everything that needs to be built builds on our arch? | 09:54 |
| xinliang | gema: you mean deb packages? | 09:56 |
| gema | xinliang: or pip deps, yes | 09:56 |
| gema | xinliang: I mean were you lacking anything to make it work? | 09:56 |
| gema | xinliang: else we need to enlist hrw | 09:57 |
| xinliang | gema: most of them installed by pip | 09:57 |
| *** hrw has joined #openstack-infra | 09:57 | |
| hrw | morning | 09:57 |
| gema | xinliang: ack | 09:57 |
| xinliang | hrw: morning | 09:57 |
| gema | hrw: trying to figure out if we need any extra bits before we can run infra on our cloud | 09:58 |
| gema | niedbalski: could we apply this patch to the current version we have deployed: https://review.openstack.org/#/c/489951/ | 09:58 |
| hrw | gema: the question is "do we need to run VM instances". if yes then VM in VM == no hw accelearation | 09:58 |
| gema | hrw: we know that | 09:59 |
| gema | hrw: we are doing a trial | 09:59 |
| *** jpich has quit IRC | 09:59 | |
| hrw | gema: other than that I do not see arch related issues | 09:59 |
| gema | hrw: goodio, thanks | 09:59 |
| xinliang | nodepool depend on diskimage-builder. I did some hack on it to make it nodepool work | 09:59 |
| gema | xinliang: do you have notes on that? | 10:00 |
| gema | we'll need to fix it proper | 10:00 |
| xinliang | yes, i note that down | 10:00 |
| gema | xinliang: good, dig it out, please :) | 10:01 |
| gema | (no rush, just getting items ready) | 10:01 |
| *** kjackal has quit IRC | 10:03 | |
| *** jascott1 has quit IRC | 10:04 | |
| *** kopecmartin has joined #openstack-infra | 10:04 | |
| *** kjackal has joined #openstack-infra | 10:04 | |
| xinliang | gema: here are the notes: https://docs.google.com/document/d/1lOYjw6RIbBGXVrAPJBd3MK0dALoZApaL4Tzrlna3pmQ/edit# | 10:04 |
| *** stevebaker has quit IRC | 10:06 | |
| *** dmellado has quit IRC | 10:06 | |
| *** dmellado has joined #openstack-infra | 10:07 | |
| *** yamamoto has quit IRC | 10:11 | |
| *** sambetts|afk is now known as sambetts | 10:13 | |
| *** tosky has joined #openstack-infra | 10:14 | |
| gema | xinliang: thanks, made it public for good measure | 10:14 |
| *** jpich has joined #openstack-infra | 10:18 | |
| *** cuongnv has quit IRC | 10:18 | |
| *** kraig has quit IRC | 10:20 | |
| *** jamesmcarthur has joined #openstack-infra | 10:20 | |
| *** threestrands has joined #openstack-infra | 10:21 | |
| eumel8 | hi, is anyone aware about https://openstackid-resources.openstack.org/ ? | 10:23 |
| *** dtantsur|afk is now known as dtantsur | 10:24 | |
| *** jamesmcarthur has quit IRC | 10:25 | |
| *** ldnunes has joined #openstack-infra | 10:28 | |
| *** coolsvap has quit IRC | 10:29 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid master: Updated support email https://review.openstack.org/532483 | 10:30 |
| *** pbourke_ has joined #openstack-infra | 10:30 | |
| *** rochaporto has joined #openstack-infra | 10:32 | |
| *** alexchadin has quit IRC | 10:32 | |
| *** jpich has quit IRC | 10:32 | |
| *** alexchadin has joined #openstack-infra | 10:33 | |
| *** annp has quit IRC | 10:34 | |
| *** stevebaker has joined #openstack-infra | 10:35 | |
| frickler | tobiash: I can confirm that, but I'm failing to find out how to debug further, will need to wait for other infra-root to help | 10:37 |
| *** dizquierdo has joined #openstack-infra | 10:42 | |
| *** kraig has joined #openstack-infra | 10:44 | |
| *** stevebaker has quit IRC | 10:48 | |
| *** dmellado has quit IRC | 10:49 | |
| frickler | infra-root: indeed it seems new images have been built about 5h ago. I have deleted these now on nl02 | 10:50 |
| *** hashar is now known as hasharAway | 10:51 | |
| *** jpich has joined #openstack-infra | 10:51 | |
| *** namnh has quit IRC | 10:51 | |
| *** rpittau has joined #openstack-infra | 10:54 | |
| *** yamamoto has joined #openstack-infra | 10:55 | |
| *** zhurong has quit IRC | 10:56 | |
| ianw | frickler: thanks for that, we might have raced and built images before the pause got in | 10:58 |
| ianw | i have to go now, but will check in first thing | 10:58 |
| *** kiennt26 has quit IRC | 10:58 | |
| *** eyalb has joined #openstack-infra | 10:59 | |
| eyalb | Hi all my tox jobs have MODULE FAILURE when installing sibling python packages | 11:00 |
| eyalb | http://logs.openstack.org/87/530587/8/check/openstack-tox-pep8/73503fc/job-output.txt.gz#_2018-01-10_10_03_41_077428 | 11:00 |
| tosky | I've seen that too | 11:01 |
| eyalb | is anyone working on this ? | 11:01 |
| * tosky would like to know as well | 11:02 | |
| *** kraig has quit IRC | 11:03 | |
| eyalb | this is the error from ansible "RuntimeError: pip python module is required" | 11:03 |
| strigazi | I have the same here http://logs.openstack.org/98/532098/1/gate/openstack-tox-pep8/049c1cf/ara/ | 11:03 |
| frickler | ianw: I'm wondering whether the "pause: true" will not be stopping rebuilds if the image already exists | 11:04 |
| *** stevebaker has joined #openstack-infra | 11:04 | |
| openstackgerrit | Merged openstack-infra/openstackid master: Updated support email https://review.openstack.org/532483 | 11:04 |
| strigazi | eyalb: the same as you in tox_install_sibling_packages | 11:04 |
| *** jascott1 has joined #openstack-infra | 11:04 | |
| frickler | eyalb: tosky: I've cleaned up broken images, running a test job now, please wait a bit before rechecking | 11:04 |
| eyalb | ok thanks | 11:06 |
| *** alexchadin has quit IRC | 11:06 | |
| *** sbra has quit IRC | 11:06 | |
| *** alexchadin has joined #openstack-infra | 11:06 | |
| tosky | frickler: oh, thanks; what was the source of the issue? | 11:07 |
| *** andreas_s has quit IRC | 11:08 | |
| frickler | tosky: IIUC new images are missing pip, so until the build process is repaired, we reverted to previous, working images | 11:08 |
| *** markvoelker has joined #openstack-infra | 11:08 | |
| *** alexchadin has quit IRC | 11:09 | |
| *** andreas_s has joined #openstack-infra | 11:09 | |
| frickler | tosky: I still haven't read all of the backlog from last night yet | 11:09 |
| *** ldnunes has quit IRC | 11:09 | |
| tosky | ah | 11:10 |
| *** ldnunes has joined #openstack-infra | 11:10 | |
| tosky | interesting that only those jobs were affected | 11:10 |
| eyalb | frickler: fixed thanks | 11:18 |
| *** eyalb has left #openstack-infra | 11:19 | |
| *** ykarel has joined #openstack-infra | 11:20 | |
| *** Mr_Broken has joined #openstack-infra | 11:20 | |
| Mr_Broken | Hey Guys | 11:21 |
| *** mnasiadka has joined #openstack-infra | 11:21 | |
| *** b_bezak has joined #openstack-infra | 11:21 | |
| *** yamahata has quit IRC | 11:22 | |
| Mr_Broken | Any chance that shade will be supporting SubnetPool functionality ? ( create, delete , manage ? ) | 11:22 |
| *** andreas_s has quit IRC | 11:23 | |
| *** dmellado has joined #openstack-infra | 11:23 | |
| *** andreas_s has joined #openstack-infra | 11:23 | |
| *** sree has joined #openstack-infra | 11:25 | |
| ykarel | is there some issue going up with: tox_install_sibling_packages | 11:26 |
| tosky | ykarel: yep, but it should be fixed (for what I know: images were generated without pip, -infra people rolled back on old images for now) | 11:29 |
| ykarel | frickler, is this fixed or is in progress ^^ | 11:29 |
| ykarel | tosky, just checked the logs | 11:29 |
| *** sree has quit IRC | 11:30 | |
| openstackgerrit | Jens Harbott (frickler) proposed openstack-infra/system-config master: Fix nodepool pause option https://review.openstack.org/532498 | 11:30 |
| frickler | ykarel: should be fixed for now, but I think we still have to fix dib to get new working images | 11:31 |
| *** dmellado has quit IRC | 11:32 | |
| ykarel | frickler, ack | 11:32 |
| ykarel | Thanks | 11:32 |
| *** armaan has quit IRC | 11:33 | |
| *** armaan has joined #openstack-infra | 11:33 | |
| *** armaan has quit IRC | 11:34 | |
| *** armaan has joined #openstack-infra | 11:34 | |
| *** dmellado has joined #openstack-infra | 11:34 | |
| *** andreas_s has quit IRC | 11:37 | |
| *** andreas_s has joined #openstack-infra | 11:38 | |
| *** d0ugal has quit IRC | 11:39 | |
| *** alexchadin has joined #openstack-infra | 11:39 | |
| AJaeger | Mr_Broken: best ask on #openstack-sdks any shade related questions | 11:41 |
| *** alexchadin has quit IRC | 11:41 | |
| *** markvoelker has quit IRC | 11:42 | |
| openstackgerrit | Merged openstack-infra/puppet-snmpd master: Enable snmpd so it works on systemd systems https://review.openstack.org/531520 | 11:44 |
| *** jkilpatr has quit IRC | 11:44 | |
| *** andreas_s has quit IRC | 11:48 | |
| *** andreas_s has joined #openstack-infra | 11:49 | |
| AJaeger | infra-root, according to zuulv3.openstack.org, we have quite a few changes hanging - top of integrated gate is there since 4+ hours, hope it finally times out ;( And there are others. Might be worthwhile to investigate and restart zuul or take changes out of queue if it does not self-heal when you read this... | 11:50 |
| AJaeger | we currently have 91 changes in gate... | 11:51 |
| frickler | AJaeger: almost all gate builds have been failing until recently due to broken images. I'm hoping that this should resolve by itself now, but it may take some time | 11:54 |
| *** yamamoto has quit IRC | 11:55 | |
| AJaeger | frickler: that's a different problem. have a look at 531779,1 | 11:57 |
| *** d0ugal has joined #openstack-infra | 11:57 | |
| AJaeger | the docs were build at "d: 2018-01-09 23:11 | 11:57 |
| *** nicolasbock has joined #openstack-infra | 11:57 | |
| AJaeger | but there's a legacy-tempest job that is "not found" if you click on it. Somehow that job got disconnected... | 11:57 |
| *** sdague has joined #openstack-infra | 11:58 | |
| AJaeger | frickler: The trove one has the same problem | 11:58 |
| AJaeger | 531779 is the real problem since it's top of integrated gate and has all those jobs behind it. Not sure when it times out | 11:59 |
| AJaeger | frickler: the red ones below are indeed the broken images. | 12:00 |
| *** alexchadin has joined #openstack-infra | 12:00 | |
| *** yamamoto has joined #openstack-infra | 12:00 | |
| frickler | AJaeger: o.k., I'll try to look that one up in the zuul logs | 12:00 |
| evrardjp | hello, any changes in tox jobs? | 12:00 |
| evrardjp | TASK [tox : Install any sibling python packages] -- ubuntu-xenial | MODULE FAILURE | 12:01 |
| AJaeger | evrardjp: sorry, we had broken images, just fixed. Please recheck | 12:01 |
| Mr_Broken | @Ajaeger: thanks, will do | 12:01 |
| evrardjp | thanks! | 12:02 |
| claudiub | hello. I have a question. How come #openstack-hyper-v's IRC logs are not collected? | 12:02 |
| claudiub | i don't see them on http://eavesdrop.openstack.org/irclogs/ | 12:02 |
| Mr_Broken | Hey guys | 12:02 |
| Mr_Broken | Any chance that shade will be supporting SubnetPool functionality ? ( create, delete , manage ? ) | 12:02 |
| Mr_Broken | lol | 12:02 |
| Mr_Broken | wrong window, disregard :D | 12:02 |
| *** shardy is now known as shardy_lunch | 12:03 | |
| *** dizquierdo has quit IRC | 12:05 | |
| *** mnasiadka has quit IRC | 12:05 | |
| AJaeger | Mr_Broken: this is irc, no need to use @ with names at all | 12:09 |
| AJaeger | claudiub: did you configure this? See http://docs.openstack.org/infra/system-config/irc.html on how to do it | 12:10 |
| frickler | 2018-01-10 10:54:54,782 INFO zuul.ExecutorClient: Build <gear.Job 0x7fe887f260b8 handle: b'H:127.0.0.1:167463' name: executor:execute unique: 2bb1990e998c4bf4acd59f30152dffee> complete, result None | 12:10 |
| frickler | AJaeger: I'm assuming zuul should retry that. I haven't found a way to remove the review from the queue other than restarting zuul-scheduler completely | 12:11 |
| *** yamamoto has quit IRC | 12:11 | |
| claudiub | AJaeger: don't remember, tbh. i'll try to do that. | 12:11 |
| AJaeger | I'm aware of these two Abandon/restore a change; promote another change on front of it (for gate queue). | 12:11 |
| AJaeger | claudiub: you only set up statusbot but not logging - you need meetbot | 12:12 |
| AJaeger | frickler: not sure what's best - or whether to wait for timeout... | 12:13 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/jeepyb master: Add checking for PyPI project ownership https://review.openstack.org/531898 | 12:16 |
| smcginnis | I'm getting proxy timeouts and general sluggishness with gerrit. Are there known issues right now? | 12:16 |
| frickler | AJaeger: maybe uploading a new PS for those stuck reviews might help | 12:16 |
| AJaeger | frickler: or abandon/restore | 12:17 |
| *** jkilpatr has joined #openstack-infra | 12:17 | |
| frickler | smcginnis: I some some of that, too, but not enough yet to restart | 12:17 |
| AJaeger | frickler: let me rebase the one in the queue... | 12:17 |
| smcginnis | frickler: OK, I'll watch it and let folks know if it looks like it's getting unusable. | 12:17 |
| *** dizquierdo has joined #openstack-infra | 12:20 | |
| *** smatzek has joined #openstack-infra | 12:20 | |
| *** pgadiya has quit IRC | 12:20 | |
| Mr_Broken | Ajaeger: yeah right, its a habbit of mine :P | 12:23 |
| *** rhallisey has joined #openstack-infra | 12:24 | |
| *** yamamoto has joined #openstack-infra | 12:26 | |
| *** b_bezak has quit IRC | 12:26 | |
| *** lucasagomes is now known as lucas-hungry | 12:27 | |
| *** janki has quit IRC | 12:27 | |
| *** sdague has quit IRC | 12:31 | |
| *** yamamoto has quit IRC | 12:32 | |
| *** markvoelker has joined #openstack-infra | 12:39 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: Added new Endpoint getspeaker https://review.openstack.org/532514 | 12:40 |
| *** wolverineav has joined #openstack-infra | 12:42 | |
| *** gcb has joined #openstack-infra | 12:43 | |
| *** dtruong has quit IRC | 12:44 | |
| *** dtruong has joined #openstack-infra | 12:44 | |
| openstackgerrit | Merged openstack-infra/openstackid-resources master: Added new Endpoint getspeaker https://review.openstack.org/532514 | 12:45 |
| *** shardy_lunch is now known as shardy | 12:47 | |
| *** jpena is now known as jpena|lunch | 12:48 | |
| openstackgerrit | Alex Krzos proposed openstack-infra/grafyaml master: Add Grafana URL and API Key to GrafYaml CLI https://review.openstack.org/530949 | 12:48 |
| *** vivsoni has quit IRC | 12:54 | |
| *** vivsoni has joined #openstack-infra | 12:54 | |
| *** makowals has quit IRC | 12:57 | |
| *** links has quit IRC | 12:58 | |
| *** makowals has joined #openstack-infra | 12:59 | |
| andreykurilin | hi folks! Can anyone help me with strange openstack-tox error? see http://logs.openstack.org/11/532411/1/check/openstack-tox-py35/fd18492/job-output.txt.gz#_2018-01-10_10_31_30_253954 | 13:00 |
| *** dizquierdo has quit IRC | 13:00 | |
| frickler | andreykurilin: we had some broken image without pip earlier, this should be fixed now, please recheck your patch | 13:01 |
| andreykurilin | frickler: thanks, will check | 13:01 |
| *** edmondsw has joined #openstack-infra | 13:01 | |
| *** Mr_Broken has quit IRC | 13:02 | |
| openstackgerrit | Yaguo Zhou proposed openstack-infra/subunit2sql master: Replace DbMigrationError with DBMigrationError https://review.openstack.org/500858 | 13:02 |
| *** pbourke_ has quit IRC | 13:03 | |
| *** efried has joined #openstack-infra | 13:03 | |
| *** rosmaita has joined #openstack-infra | 13:04 | |
| openstackgerrit | Sean McGinnis proposed openstack-infra/project-config master: Switch to noop jobs for evoque projects https://review.openstack.org/532519 | 13:05 |
| dirk | is zuulv3.openstack.org (the webpage) unavailable for anyone else? | 13:06 |
| dirk | I just get connection timeout | 13:06 |
| smcginnis | dirk: It's been slowly dying for me. | 13:06 |
| dirk | smcginnis: whoops, then I probably just threw it over the fence | 13:07 |
| smcginnis | dirk: I wouldn't be surprised if a restart is needed soon, but it's still somewhat usable. ;) | 13:07 |
| frickler | dirk: smcginnis: ya, zuul started swapping again. /me goes to save the queues just in case | 13:09 |
| *** smatzek has quit IRC | 13:09 | |
| *** markvoelker has quit IRC | 13:12 | |
| *** dizquierdo has joined #openstack-infra | 13:13 | |
| frickler | infra-root: zuul seems to be in swapping mood again, no queue save possible currently. will wait for half an hour otherwise restart | 13:13 |
| *** smatzek has joined #openstack-infra | 13:14 | |
| *** mnasiadka has joined #openstack-infra | 13:14 | |
| *** janki has joined #openstack-infra | 13:15 | |
| *** dprince has joined #openstack-infra | 13:16 | |
| efried | When something like that happens, does a guy need to recheck patches? I've got one that's been sitting in the gate for >17h | 13:16 |
| frickler | efried: I did succeed in saving the queues now, so should be able to restore them after a restart | 13:20 |
| *** b_bezak has joined #openstack-infra | 13:20 | |
| *** yamamoto has joined #openstack-infra | 13:21 | |
| *** ttx has quit IRC | 13:21 | |
| *** lucas-hungry is now known as lucasagomes | 13:22 | |
| *** ttx has joined #openstack-infra | 13:23 | |
| *** ricolin has joined #openstack-infra | 13:23 | |
| efried | frickler Thanks. I've had a handful of patches +W for a couple days. The bottom ones have either been failing spuriously after many hours or seemingly just ignored by zuul. I've seen other stuff merge, so I was gonna ask if it was personal :) | 13:24 |
| pabelanger | frickler: can confirm swapping, if queues are saved, agree about restart | 13:25 |
| *** andreas_s has quit IRC | 13:27 | |
| *** shardy is now known as shardy_mtg | 13:28 | |
| frickler | pabelanger: yes, still swapping heavily, will restart now | 13:28 |
| *** sree has joined #openstack-infra | 13:30 | |
| frickler | efried: yes, we have had issues for almost 24h now which has lead to a bit of backlog, so please bear with us. nothing personal involved I hope ;) | 13:31 |
| efried | Thanks frickler. As always, I appreciate the work y'all do. | 13:31 |
| *** andreas_s has joined #openstack-infra | 13:32 | |
| *** daidv has quit IRC | 13:32 | |
| *** wolverineav has quit IRC | 13:33 | |
| *** rlandy has joined #openstack-infra | 13:34 | |
| *** sbra has joined #openstack-infra | 13:35 | |
| frickler | zuul restart successful, restoring queues now | 13:36 |
| *** andreas_s has quit IRC | 13:37 | |
| *** alexchadin has quit IRC | 13:38 | |
| *** andreas_s has joined #openstack-infra | 13:38 | |
| efried | whee, my two showed up in the gate, thanks frickler | 13:39 |
| frickler | pabelanger: how is your nodepool work going? still waiting for gentoo images | 13:39 |
| frickler | efried: yeah, let's hope that these run smoother now | 13:39 |
| *** makowals has quit IRC | 13:39 | |
| *** andreas_s has quit IRC | 13:40 | |
| *** andreas_s has joined #openstack-infra | 13:40 | |
| pabelanger | frickler: yah, once we merge feature/zuulv3 back to master, we can work on bringing those new images online. We are waiting for some changes to puppet-openstackci to land first | 13:41 |
| *** wolverineav has joined #openstack-infra | 13:41 | |
| fungi | what was the cause of missing pip on newer images? anyone know? | 13:41 |
| *** andreas__ has joined #openstack-infra | 13:41 | |
| frickler | fungi: I haven't been able to find that out from backlog yet | 13:41 |
| *** ykarel is now known as ykarel|away | 13:43 | |
| *** sree has quit IRC | 13:43 | |
| stephenfin | smcginnis: Do reqs in 'doc/requirements.txt' have to be tracked in upper constraints? | 13:43 |
| *** makowals has joined #openstack-infra | 13:43 | |
| pabelanger | fungi: is there an example failure? | 13:44 |
| pabelanger | I think we just started building images again after a few days of being offline | 13:45 |
| *** andreas_s has quit IRC | 13:45 | |
| frickler | fungi: pabelanger: ianw mentioned https://review.openstack.org/#/c/519502/, if that has only been recently released, that looks plausible | 13:45 |
| jokke_ | looks like rechecks are needed | 13:45 |
| *** trown|outtypewww is now known as trown | 13:45 | |
| pabelanger | frickler: fungi: we also have a new DIB release (2.10.0 yesterday) | 13:46 |
| frickler | dib-2.10.0 released yesterday, yes | 13:46 |
| frickler | so maybe we want to pin 2.9.0 until we have a better solution | 13:47 |
| pabelanger | well, what is the issue currently? I haven't seen a failure myself | 13:47 |
| frickler | jokke_: queues are still being restored, please be patient | 13:47 |
| frickler | pabelanger: all these errors about tox_siblings in the backlog | 13:48 |
| pabelanger | we can also roll back images in nodepool if needed | 13:48 |
| frickler | pabelanger: already did that | 13:48 |
| *** sree has joined #openstack-infra | 13:48 | |
| frickler | pabelanger: had to repeat because another set of broken images got published in the morning | 13:48 |
| frickler | pabelanger: sample log http://logs.openstack.org/11/532411/1/check/openstack-tox-py35/fd18492/job-output.txt.gz#_2018-01-10_10_31_30_253954 | 13:49 |
| fungi | yeah, not currently using broken images from what i gather, we're back on old images until it can be resolved | 13:49 |
| *** yamamoto has quit IRC | 13:49 | |
| *** ykarel|away has quit IRC | 13:49 | |
| pabelanger | frickler: did you set pause in nodepool.yaml? Otherwise, nodepool-builder will just build images again (since over 24hrs) and upload the brokeness | 13:49 |
| frickler | pabelanger: yes, there were two patches for that | 13:50 |
| pabelanger | great | 13:50 |
| fungi | following scrollback, the broken images were deleted and pause patches merged, but new images started building in between so had to also be deleted once they came online and were noticed | 13:51 |
| *** jpena|lunch is now known as jpena | 13:51 | |
| frickler | fungi: yes. probably because we only use one builder task, the new builds were already queued | 13:51 |
| pabelanger | yah, rebuilds are pretty aggressive, I've had to delete images twice before | 13:52 |
| *** alexchadin has joined #openstack-infra | 13:52 | |
| *** sree has quit IRC | 13:53 | |
| frickler | fungi: pabelanger: would this be o.k. or would you rather do a notice? /me thinks there has been enough spamming lately. #status log zuul-scheduler has been restarted due to heavy swapping, queues have been restored. | 13:54 |
| pabelanger | frickler: nope, feel free. Good to update people | 13:55 |
| fungi | frickler: up to you. we at least need to keep track of the restart but whether that's via log or notice doesn't matter so much for me on this one | 13:56 |
| frickler | #status log zuul-scheduler has been restarted due to heavy swapping, queues have been restored. | 13:56 |
| openstackstatus | frickler: finished logging | 13:56 |
| *** sree has joined #openstack-infra | 13:56 | |
| frickler | #status log another set of broken images has been in use from about 06:00-11:00 UTC, reverted once more to the previous ones | 13:58 |
| openstackstatus | frickler: finished logging | 13:58 |
| pabelanger | yah, it looks like we no longer install python2 pip | 13:59 |
| *** shardy_mtg is now known as shardy | 13:59 | |
| *** sdague has joined #openstack-infra | 14:00 | |
| *** armaan has quit IRC | 14:00 | |
| *** armaan has joined #openstack-infra | 14:01 | |
| openstackgerrit | Vasyl Saienko proposed openstack-infra/project-config master: Add networking-generic-switch-tempest-plugin repo https://review.openstack.org/532535 | 14:02 |
| pabelanger | seems only python2 virtualenv or python3 virtualenv. No longer support both | 14:02 |
| *** sree has quit IRC | 14:04 | |
| *** sree has joined #openstack-infra | 14:05 | |
| *** kiennt26 has joined #openstack-infra | 14:06 | |
| pabelanger | one option would be to have the job setup pip2, which would be something non DIB users would have to do | 14:07 |
| frickler | pabelanger: ianw already proposed a revert https://review.openstack.org/532395 , but maybe setting it up in the job would be cleaner. not sure about performance/stability impact though | 14:08 |
| *** markvoelker has joined #openstack-infra | 14:10 | |
| *** sree has quit IRC | 14:10 | |
| pabelanger | frickler: yah, revert is good I think, for breaking backwards, +2. | 14:10 |
| *** jamesmcarthur has joined #openstack-infra | 14:11 | |
| *** kgiusti has joined #openstack-infra | 14:11 | |
| frickler | pabelanger: fungi: what do you think about restarting gerrit, too, while we are at it? seeing sporadic 502s and memory usage close to maximum for a while now http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=27&rra_id=all | 14:13 |
| pabelanger | https://review.openstack.org/monitoring?part=graph&graph=gc&period=jour | 14:15 |
| pabelanger | looks like spike in GC | 14:15 |
| *** gouthamr has joined #openstack-infra | 14:17 | |
| *** Goneri has joined #openstack-infra | 14:19 | |
| *** smatzek has quit IRC | 14:22 | |
| *** smatzek has joined #openstack-infra | 14:22 | |
| fungi | pabelanger: we should take the opportunity to update the kernel and reboot too | 14:24 |
| *** smatzek has quit IRC | 14:24 | |
| *** dhill_ has quit IRC | 14:27 | |
| *** dhill_ has joined #openstack-infra | 14:27 | |
| *** trown is now known as trown|brb | 14:32 | |
| *** hongbin has joined #openstack-infra | 14:34 | |
| frickler | infra-root: do we have a broken mirror or some other issue here? no more time to look into this myself right now http://logs.openstack.org/78/531778/1/gate/cross-cinder-py27/15971e5/job-output.txt.gz#_2018-01-10_14_08_06_711604 | 14:34 |
| openstackgerrit | Alex Krzos proposed openstack-infra/grafyaml master: Add Grafana URL and API Key to GrafYaml CLI https://review.openstack.org/530949 | 14:36 |
| tosky | do we have any example of native zuul v3 jobs publishing something on tarballs.openstack.org, apart from the release tarballs? | 14:36 |
| openstackgerrit | Vasyl Saienko proposed openstack-infra/project-config master: Add networking-generic-switch-tempest-plugin repo https://review.openstack.org/532535 | 14:37 |
| AJaeger | tosky: https://review.openstack.org/#/c/530124 | 14:37 |
| *** mriedem has joined #openstack-infra | 14:38 | |
| *** lbragstad has joined #openstack-infra | 14:38 | |
| tosky | AJaeger: thanks! | 14:39 |
| *** mnasiadka has quit IRC | 14:39 | |
| *** coolsvap has joined #openstack-infra | 14:40 | |
| AJaeger | tosky: there might be others, hope that helps you to run some queries on | 14:40 |
| *** armax has quit IRC | 14:41 | |
| openstackgerrit | Vasyl Saienko proposed openstack-infra/project-config master: Add networking-generic-switch-tempest-plugin repo https://review.openstack.org/532542 | 14:41 |
| openstackgerrit | Vasyl Saienko proposed openstack-infra/project-config master: Add jobs for n-g-s tempest-plugin https://review.openstack.org/532543 | 14:41 |
| hongbin | hi, our pep8 and py** jobs started to fail in about 6 hours before, anyone notices a similar issue? | 14:42 |
| *** markvoelker has quit IRC | 14:43 | |
| *** alexchadin has quit IRC | 14:43 | |
| dmsimard | frickler: just wanted to high-five, you've been a great help recently keeping the ship afloat recently | 14:43 |
| * dmsimard high-fives | 14:43 | |
| pabelanger | fungi: good idea | 14:44 |
| *** iyamahat has joined #openstack-infra | 14:46 | |
| fungi | frickler: looks like that job ran in infracloud-chocolate... cacti graphs for it died at 04:00z | 14:46 |
| *** afred312 has quit IRC | 14:46 | |
| *** afred312 has joined #openstack-infra | 14:46 | |
| fungi | looks like that's around the time clarkb was updating/rebooting stuff in there for the meltdown patches | 14:46 |
| hongbin | Here is the failure on our gate: | 14:47 |
| hongbin | 2018-01-10 07:51:50.547936 | TASK [tox : Install any sibling python packages] | 14:47 |
| hongbin | 2018-01-10 07:51:50.873038 | ubuntu-xenial | MODULE FAILURE | 14:47 |
| hongbin | http://logs.openstack.org/46/531646/3/check/openstack-tox-pep8/cc9720d/job-output.txt.gz#_2018-01-10_07_51_50_547936 | 14:47 |
| *** eharney has joined #openstack-infra | 14:48 | |
| fungi | frickler: and looks like my attempt to ssh into it is going to time out, so i'll see if i can manage to reboot it but we may need to take chocolate out of nodepool for the near future | 14:48 |
| *** trown|brb is now known as trown | 14:48 | |
| pabelanger | fungi: since review.o.o is trusty, which kernel should we have installed? | 14:48 |
| pabelanger | https://wiki.ubuntu.com/SecurityTeam/KnowledgeBase/SpectreAndMeltdown seems to show a 4.x for trusy | 14:48 |
| pabelanger | trusty* | 14:48 |
| pabelanger | but currently on 3.x stream | 14:49 |
| fungi | pabelanger: i'm checking their cve tracker now... my browser's gotten really slow so just a sec | 14:49 |
| AJaeger | hongbin: known issues, we had broken images. Sorry. Please recheck your change | 14:49 |
| hongbin | AJaeger: ack | 14:49 |
| *** yamamoto has joined #openstack-infra | 14:50 | |
| fungi | pabelanger: 3.13.0-139.188 according to https://people.canonical.com/~ubuntu-security/cve/2017/CVE-2017-5754.html | 14:50 |
| *** markvoelker has joined #openstack-infra | 14:51 | |
| pabelanger | okay, looking | 14:51 |
| fungi | frickler: confirmed, server status for mirror01.regionone.infracloud-chocolate.openstack.org shows SHUTOFF | 14:52 |
| fungi | trying to reboot it through the api now | 14:52 |
| fungi | shows active | 14:52 |
| pabelanger | okay, linux-image-3.13.0-139-generic:amd64 looks to be installed, just need to reboot review.o.o | 14:52 |
| fungi | pabelanger: once it boots up, you can check dmesg output for "Kernel/User page tables isolation: enabled" | 14:53 |
| pabelanger | okay | 14:54 |
| fungi | probably want to #status notice something about the gerrit reboot | 14:54 |
| pabelanger | agree | 14:54 |
| fungi | since people will notice it's down | 14:54 |
| fungi | frickler: thanks for spotting the broken mirror! it's back online now and seems to be working since i booted it up | 14:55 |
| fungi | #status log manually started mirror.regionone.infracloud-chocolate which had been in shutoff state following application of meltdown patches to infracloud hosts | 14:55 |
| openstackstatus | fungi: finished logging | 14:55 |
| pabelanger | #status notice Gerrit is being restarted due to slowness and to apply kernel patches | 14:56 |
| fungi | (which may add new slowness!) | 14:56 |
| *** iyamahat has quit IRC | 14:56 | |
| *** pabelanger has quit IRC | 14:56 | |
| *** pabelanger has joined #openstack-infra | 14:56 | |
| *** pcichy has joined #openstack-infra | 14:56 | |
| *** iyamahat has joined #openstack-infra | 14:56 | |
| mriedem | i'm assuming this has been fixed? | 14:56 |
| mriedem | "Error when trying to get requirement for VCS system Command "git config --get-regexp remote\..*\.url" failed with error code 1 in /home/zuul/src/git.openstack.org/openstack/nova, falling back to uneditable format,Could not determine repository location of /home/zuul/src/git.openstack.org/openstack" | 14:56 |
| pabelanger | #status notice Gerrit is being restarted due to slowness and to apply kernel patches | 14:56 |
| openstackstatus | pabelanger: sending notice | 14:56 |
| tosky | AJaeger: did you see https://review.openstack.org/#/c/532522/ and the comments? | 14:57 |
| fungi | mriedem: that's usually a benign message pip emits when installing from a local repo with no remote named "origin" | 14:57 |
| *** jamesmcarthur has quit IRC | 14:57 | |
| fungi | mriedem: odds are if that was seen in a failed job, the failure message is elsewhere | 14:57 |
| mriedem | fungi: i've seen it over several nova patches from last night | 14:57 |
| *** felipemonteiro__ has joined #openstack-infra | 14:57 | |
| mriedem | oh | 14:57 |
| *** yamamoto has quit IRC | 14:58 | |
| -openstackstatus- NOTICE: Gerrit is being restarted due to slowness and to apply kernel patches | 14:58 | |
| *** dhajare has quit IRC | 14:58 | |
| AJaeger | tosky: will try to look at it later... | 14:58 |
| *** pcichy has quit IRC | 14:58 | |
| fungi | mriedem: we've had a confluence of other problems in the past ~12 hours or so... images missing pip (those have been rolled back while we investigate) and the mirror in infracloud-chocolate being offline (i just booted it, but it was missing in action after we patched that cloud for meltdown) | 14:59 |
| *** felipemonteiro_ has joined #openstack-infra | 14:59 | |
| tosky | AJaeger: sure, no rush | 14:59 |
| openstackstatus | pabelanger: finished sending notice | 14:59 |
| mriedem | fungi: ack. just trying to find a fingerprint for e-r really. | 15:00 |
| pabelanger | okay, stopping gerrit | 15:00 |
| mriedem | https://goo.gl/xJdvRt | 15:00 |
| pabelanger | now rebooting review.o.o | 15:01 |
| *** openstackgerrit has quit IRC | 15:01 | |
| *** jamesmcarthur has joined #openstack-infra | 15:02 | |
| *** felipemonteiro__ has quit IRC | 15:03 | |
| *** markvoelker has quit IRC | 15:03 | |
| *** iyamahat has quit IRC | 15:03 | |
| pabelanger | [ 0.000000] Kernel/User page tables isolation: enabled | 15:04 |
| pabelanger | starting gerrit now | 15:04 |
| *** smatzek has joined #openstack-infra | 15:05 | |
| pabelanger | actually, gerrit starts on reboot | 15:05 |
| *** smatzek_ has joined #openstack-infra | 15:06 | |
| *** jamesmcarthur has quit IRC | 15:06 | |
| *** smatzek has quit IRC | 15:08 | |
| pabelanger | okay, gerrit should be online now | 15:08 |
| *** iyamahat has joined #openstack-infra | 15:09 | |
| *** dave-mccowan has joined #openstack-infra | 15:09 | |
| fungi | mriedem: yeah, the logstash graph for that message is basically a graph of all python jobs we run. if you toggle the build_status column you'll see it's in successes as well as failures | 15:12 |
| fungi | it's noise from pip, and clarkb has a pr up for pip to "fix" that | 15:12 |
| *** dave-mcc_ has joined #openstack-infra | 15:12 | |
| *** erlon has joined #openstack-infra | 15:13 | |
| *** dougwig has joined #openstack-infra | 15:13 | |
| *** ianychoi has joined #openstack-infra | 15:14 | |
| *** armax has joined #openstack-infra | 15:14 | |
| mriedem | fungi: ah ok, cool | 15:14 |
| *** dave-mccowan has quit IRC | 15:14 | |
| *** bobh has joined #openstack-infra | 15:16 | |
| fungi | #status log manually started mirror.regionone.infracloud-vanilla which had been in shutoff state following application of meltdown patches to infracloud hosts | 15:16 |
| openstackstatus | fungi: finished logging | 15:16 |
| *** hrw has quit IRC | 15:17 | |
| tosky | AJaeger: about that publishing job for training-labs: the publish job is defined in project-config instead of in-tree for security reasons? | 15:22 |
| *** hjensas has quit IRC | 15:22 | |
| *** hrw has joined #openstack-infra | 15:22 | |
| *** psachin has quit IRC | 15:22 | |
| *** iyamahat has quit IRC | 15:24 | |
| *** jaypipes has joined #openstack-infra | 15:25 | |
| *** numans has quit IRC | 15:26 | |
| *** iyamahat has joined #openstack-infra | 15:26 | |
| *** numans has joined #openstack-infra | 15:28 | |
| *** caphrim007 has quit IRC | 15:30 | |
| *** caphrim007 has joined #openstack-infra | 15:31 | |
| *** iyamahat has quit IRC | 15:33 | |
| *** felipemonteiro_ has quit IRC | 15:33 | |
| *** felipemonteiro_ has joined #openstack-infra | 15:34 | |
| *** caphrim007 has quit IRC | 15:35 | |
| *** jamesmcarthur has joined #openstack-infra | 15:35 | |
| frickler | dmsimard: thanks, that's about the position I assumed I would end up in, filling the time-gap from australia to the states ;) | 15:36 |
| dmsimard | frickler: what timezone are you in ? | 15:36 |
| *** Apoorva has joined #openstack-infra | 15:37 | |
| *** markvoelker has joined #openstack-infra | 15:37 | |
| frickler | dmsimard: GMT+1. to be more precise location of the next european summit, as I managed to learn only today :D | 15:38 |
| *** katkapilatova1 has joined #openstack-infra | 15:38 | |
| dmsimard | Berlin ? Neat. | 15:38 |
| AJaeger | tosky: yes. | 15:38 |
| *** slaweq has quit IRC | 15:38 | |
| AJaeger | tosky: for tarballs.o.o, we might not need it, need to double checks. I was conservative. | 15:39 |
| *** slaweq has joined #openstack-infra | 15:39 | |
| *** jkilpatr has quit IRC | 15:39 | |
| *** slaweq has quit IRC | 15:43 | |
| *** david-lyle has joined #openstack-infra | 15:43 | |
| *** janki has quit IRC | 15:45 | |
| *** sandanar has quit IRC | 15:45 | |
| *** kiennt26 has quit IRC | 15:46 | |
| *** yamahata has joined #openstack-infra | 15:47 | |
| fungi | frickler: so you're saying you'll serve as our guide to find the perfect bierstube? or i guess a biergarten would be a better choice that time of year | 15:49 |
| *** mnasiadka has joined #openstack-infra | 15:52 | |
| frickler | fungi: actually I'm not that much into beer, but I'm sure we can manage something. though november might not be too cosy outside unless we are very lucky | 15:53 |
| *** vsaienk0 has joined #openstack-infra | 15:53 | |
| vsaienk0 | dtantsur: AJaeger please review https://review.openstack.org/#/c/532542/ add n-g-s-tempest-plugin repo | 15:53 |
| vsaienk0 | thanks in advance! | 15:53 |
| *** jkilpatr has joined #openstack-infra | 15:55 | |
| fungi | frickler: fair enough, you probably still have better suggestions than whatever the rest of us would come up with ;) | 15:57 |
| *** felipemonteiro__ has joined #openstack-infra | 15:57 | |
| fungi | finding good local cuisine is at least as important | 15:58 |
| cmurphy | would not recommend biergarten in berlin in november | 15:59 |
| fungi | good to know! | 16:00 |
| *** felipemonteiro_ has quit IRC | 16:01 | |
| *** tpsilva_ has joined #openstack-infra | 16:02 | |
| *** dklyle has joined #openstack-infra | 16:04 | |
| *** david-lyle has quit IRC | 16:04 | |
| *** sbra has quit IRC | 16:04 | |
| corvus | i'm sure it's still better than our "beer gardens" any time of the year | 16:06 |
| corvus | even if it no one's there... and it's closed... and locked. | 16:06 |
| mriedem | will we need to recheck patches after the gerrit restart? | 16:09 |
| frickler | mriedem: no, those are handled by zuul | 16:11 |
| fungi | mriedem: only if zuul tried to report results for them in the few minutes gerrit was offline | 16:11 |
| *** jkilpatr has quit IRC | 16:16 | |
| *** mnasiadka has quit IRC | 16:16 | |
| *** fultonj has quit IRC | 16:16 | |
| *** dizquierdo has quit IRC | 16:18 | |
| *** mpeterson has quit IRC | 16:19 | |
| *** markvoelker has quit IRC | 16:19 | |
| efried | Hey guys, sorry to bother you again, but 518633 & 521686 disappeared from the gate queue without posting any results. (They *did* appear in the gate queue after frickler's earlier restart/restore.) I don't want to keep clogging with rechecks if there's something else wrong. Please advise. | 16:21 |
| *** mpeterson has joined #openstack-infra | 16:21 | |
| mriedem | i think efried is asking about the same thing i just did | 16:22 |
| *** markvoelker has joined #openstack-infra | 16:22 | |
| *** kopecmartin has quit IRC | 16:22 | |
| efried | mriedem After the restart/restore, my patches were restored to the gate queue, and earlier frickler said we shouldn't have to recheck. But then they vanished. | 16:23 |
| *** b_bezak has quit IRC | 16:23 | |
| mriedem | that's what i saw too | 16:24 |
| mriedem | i just rechecked and now it's back in check | 16:25 |
| *** vsaienk0 has quit IRC | 16:25 | |
| efried | oh, I see 521686 in the check queue - seven times (wtf?). But 518633 is nowhere to be found. | 16:25 |
| *** vsaienk0 has joined #openstack-infra | 16:27 | |
| *** markvoelker has quit IRC | 16:27 | |
| *** armaan has quit IRC | 16:27 | |
| *** armaan has joined #openstack-infra | 16:28 | |
| *** armaan has quit IRC | 16:28 | |
| *** armaan has joined #openstack-infra | 16:28 | |
| *** jkilpatr has joined #openstack-infra | 16:29 | |
| *** andreas__ has quit IRC | 16:30 | |
| *** andreas_s has joined #openstack-infra | 16:32 | |
| *** ricolin has quit IRC | 16:33 | |
| *** andreas__ has joined #openstack-infra | 16:33 | |
| *** slaweq has joined #openstack-infra | 16:35 | |
| *** jamesmcarthur has quit IRC | 16:36 | |
| *** andreas_s has quit IRC | 16:36 | |
| *** jamesmcarthur has joined #openstack-infra | 16:38 | |
| *** caphrim007 has joined #openstack-infra | 16:38 | |
| *** fultonj has joined #openstack-infra | 16:39 | |
| *** rhallisey has quit IRC | 16:39 | |
| *** rhallisey has joined #openstack-infra | 16:39 | |
| *** slaweq has quit IRC | 16:40 | |
| *** andreas__ has quit IRC | 16:41 | |
| clarkb | efried: its showing you that 521686 is in the dependency chain of six changes in check | 16:41 |
| *** caphrim007 has quit IRC | 16:42 | |
| fungi | efried: mriedem: if those changes happened to finish their jobs in the couple of minutes that gerrit was offline for rebooting, that would explain it | 16:43 |
| *** caphrim007 has joined #openstack-infra | 16:45 | |
| *** d0ugal has quit IRC | 16:46 | |
| *** caphrim007 has quit IRC | 16:46 | |
| *** jamesmcarthur has quit IRC | 16:46 | |
| *** jaosorior has quit IRC | 16:47 | |
| *** caphrim007 has joined #openstack-infra | 16:47 | |
| *** felipemonteiro__ has quit IRC | 16:48 | |
| *** felipemonteiro__ has joined #openstack-infra | 16:48 | |
| *** jamesmcarthur has joined #openstack-infra | 16:48 | |
| *** openstackgerrit has joined #openstack-infra | 16:51 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul feature/zuulv3: Remove need to start executor as root https://review.openstack.org/532575 | 16:51 |
| *** clarkb has quit IRC | 16:53 | |
| *** gyee has joined #openstack-infra | 16:54 | |
| *** clarkb1 has joined #openstack-infra | 16:54 | |
| *** dhajare has joined #openstack-infra | 16:55 | |
| *** rosmaita has quit IRC | 16:58 | |
| *** clarkb1 is now known as clarkb | 16:58 | |
| *** sbra has joined #openstack-infra | 17:01 | |
| *** Apoorva has quit IRC | 17:01 | |
| *** dklyle has quit IRC | 17:03 | |
| *** yamahata has quit IRC | 17:05 | |
| *** e0ne has quit IRC | 17:05 | |
| *** dhajare has quit IRC | 17:06 | |
| *** vsaienk0 has quit IRC | 17:07 | |
| *** pblaho has quit IRC | 17:09 | |
| *** vsaienk0 has joined #openstack-infra | 17:09 | |
| *** ramishra has quit IRC | 17:10 | |
| *** dave-mcc_ has quit IRC | 17:11 | |
| *** pcaruana has quit IRC | 17:11 | |
| *** dizquierdo has joined #openstack-infra | 17:13 | |
| *** numans has quit IRC | 17:16 | |
| *** harlowja has joined #openstack-infra | 17:16 | |
| *** david-lyle has joined #openstack-infra | 17:16 | |
| *** dave-mccowan has joined #openstack-infra | 17:19 | |
| *** lucasagomes is now known as lucas-afk | 17:19 | |
| *** numans has joined #openstack-infra | 17:19 | |
| *** hjensas has joined #openstack-infra | 17:20 | |
| *** hjensas has quit IRC | 17:20 | |
| *** hjensas has joined #openstack-infra | 17:20 | |
| *** pcichy has joined #openstack-infra | 17:22 | |
| *** vsaienk0 has quit IRC | 17:23 | |
| *** pfallenop has quit IRC | 17:23 | |
| *** sree has joined #openstack-infra | 17:23 | |
| *** pfallenop has joined #openstack-infra | 17:24 | |
| *** rhallisey has quit IRC | 17:29 | |
| *** jascott1 has quit IRC | 17:30 | |
| *** rhallisey has joined #openstack-infra | 17:30 | |
| *** pblaho has joined #openstack-infra | 17:33 | |
| *** vsaienk0 has joined #openstack-infra | 17:34 | |
| *** jaypipes has quit IRC | 17:35 | |
| *** jpich has quit IRC | 17:36 | |
| *** florianf has quit IRC | 17:37 | |
| *** zerick has joined #openstack-infra | 17:39 | |
| *** sree_ has joined #openstack-infra | 17:39 | |
| *** sree_ is now known as Guest48418 | 17:40 | |
| *** iyamahat has joined #openstack-infra | 17:40 | |
| *** iyamahat_ has joined #openstack-infra | 17:40 | |
| *** iyamahat has quit IRC | 17:40 | |
| *** zerick has quit IRC | 17:41 | |
| *** rosmaita has joined #openstack-infra | 17:41 | |
| pabelanger | infra-root: Could I get a review on https://review.openstack.org/528807/ remove kdc02 from our manifests | 17:41 |
| *** zerick has joined #openstack-infra | 17:42 | |
| *** Guest48418 has quit IRC | 17:42 | |
| *** sree has quit IRC | 17:42 | |
| *** felipemonteiro_ has joined #openstack-infra | 17:43 | |
| *** zerick has quit IRC | 17:45 | |
| *** zerick has joined #openstack-infra | 17:46 | |
| *** felipemonteiro__ has quit IRC | 17:46 | |
| *** armaan has quit IRC | 17:47 | |
| *** armaan has joined #openstack-infra | 17:48 | |
| *** cshastri has quit IRC | 17:51 | |
| *** sree has joined #openstack-infra | 17:53 | |
| corvus | pabelanger: +3 | 17:55 |
| pabelanger | thank you | 17:56 |
| *** sree has quit IRC | 17:58 | |
| *** iyamahat_ has quit IRC | 17:58 | |
| *** derekh has quit IRC | 18:00 | |
| *** iyamahat has joined #openstack-infra | 18:00 | |
| *** Apoorva has joined #openstack-infra | 18:00 | |
| *** Apoorva has quit IRC | 18:00 | |
| *** Apoorva has joined #openstack-infra | 18:01 | |
| *** pcichy has quit IRC | 18:02 | |
| *** jamesmcarthur has quit IRC | 18:02 | |
| *** david-lyle has quit IRC | 18:07 | |
| *** gouthamr has quit IRC | 18:09 | |
| *** jpena is now known as jpena|off | 18:14 | |
| *** trown is now known as trown|lunch | 18:17 | |
| *** coolsvap has quit IRC | 18:18 | |
| *** numans has quit IRC | 18:19 | |
| *** felipemonteiro_ has quit IRC | 18:19 | |
| *** jkilpatr has quit IRC | 18:19 | |
| *** felipemonteiro_ has joined #openstack-infra | 18:19 | |
| *** jkilpatr has joined #openstack-infra | 18:20 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Add specific setup inventory https://review.openstack.org/530265 | 18:20 |
| *** numans has joined #openstack-infra | 18:22 | |
| *** hemna_ has joined #openstack-infra | 18:25 | |
| *** jascott1 has joined #openstack-infra | 18:25 | |
| *** openstack has joined #openstack-infra | 18:30 | |
| *** ChanServ sets mode: +o openstack | 18:30 | |
| *** madhuvishy is now known as madhuvishy|afk | 18:32 | |
| *** jamesmcarthur has quit IRC | 18:33 | |
| *** jamesmcarthur has joined #openstack-infra | 18:33 | |
| *** dhajare has joined #openstack-infra | 18:33 | |
| openstackgerrit | Merged openstack-infra/system-config master: Remove kdc02.o.o (replaced by kdc04.o.o) https://review.openstack.org/528807 | 18:34 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/puppet-zuul master: Start executor as 'zuul' user https://review.openstack.org/532594 | 18:34 |
| *** thorre_se has joined #openstack-infra | 18:35 | |
| Shrews | pabelanger: corvus: I *think* that ^^^ is all that's needed, yeah? | 18:35 |
| *** vsaienk0 has joined #openstack-infra | 18:36 | |
| pabelanger | Shrews: yah, believe so | 18:36 |
| Shrews | oh, interesting that zuul already -1'd that | 18:36 |
| *** tesseract has quit IRC | 18:36 | |
| *** harlowja has quit IRC | 18:37 | |
| *** thorre_ has joined #openstack-infra | 18:37 | |
| *** thorre has quit IRC | 18:38 | |
| *** thorre_ is now known as thorre | 18:38 | |
| *** ijw has quit IRC | 18:39 | |
| *** thorre_se has quit IRC | 18:40 | |
| *** jkilpatr has joined #openstack-infra | 18:41 | |
| *** vsaienk0 has quit IRC | 18:46 | |
| *** ijw has joined #openstack-infra | 18:46 | |
| *** gouthamr has joined #openstack-infra | 18:47 | |
| *** e0ne has joined #openstack-infra | 18:48 | |
| *** sshnaidm is now known as sshnaidm|afk | 18:48 | |
| *** xinliang has quit IRC | 18:53 | |
| *** gouthamr has quit IRC | 18:53 | |
| *** xinliang has joined #openstack-infra | 18:53 | |
| *** xinliang has quit IRC | 18:53 | |
| *** xinliang has joined #openstack-infra | 18:54 | |
| *** b_bezak has joined #openstack-infra | 18:55 | |
| *** vsaienk0 has joined #openstack-infra | 18:56 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul feature/zuulv3: Remove need to start executor as root https://review.openstack.org/532575 | 18:57 |
| *** b_bezak has quit IRC | 18:57 | |
| *** jamesmcarthur has quit IRC | 18:59 | |
| *** david-lyle has joined #openstack-infra | 18:59 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/system-config master: Fix zuulv3 links https://review.openstack.org/532605 | 19:02 |
| *** felipemonteiro__ has joined #openstack-infra | 19:03 | |
| thingee | a few small reviews in this topic for statusbot, just need final +2's. https://review.openstack.org/#/q/status:open+topic:statusbot-pageurl | 19:03 |
| *** felipemonteiro_ has quit IRC | 19:06 | |
| *** shardy has quit IRC | 19:08 | |
| *** sambetts is now known as sambetts|afk | 19:10 | |
| *** jamesmcarthur has joined #openstack-infra | 19:10 | |
| *** fultonj has quit IRC | 19:11 | |
| *** harlowja has joined #openstack-infra | 19:13 | |
| *** ijw has quit IRC | 19:15 | |
| *** dizquierdo has quit IRC | 19:15 | |
| *** hemna_ has quit IRC | 19:16 | |
| *** harlowja_ has joined #openstack-infra | 19:16 | |
| openstackgerrit | Matt Riedemann proposed openstack-infra/project-config master: Fix ironic jobs run in nova experimental https://review.openstack.org/532608 | 19:19 |
| *** harlowja has quit IRC | 19:19 | |
| openstackgerrit | Matt Riedemann proposed openstack-infra/project-config master: Fix ironic jobs run in nova experimental https://review.openstack.org/532608 | 19:20 |
| *** thingee has quit IRC | 19:23 | |
| *** fultonj has joined #openstack-infra | 19:24 | |
| *** vsaienk0 has quit IRC | 19:24 | |
| *** vsaienk0 has joined #openstack-infra | 19:27 | |
| *** thingee has joined #openstack-infra | 19:28 | |
| *** rmcall has joined #openstack-infra | 19:30 | |
| *** Apoorva_ has joined #openstack-infra | 19:32 | |
| *** Apoorva has quit IRC | 19:34 | |
| *** trown|lunch is now known as trown | 19:35 | |
| *** dsariel has quit IRC | 19:36 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Remove zuulv3-dev.o.o https://review.openstack.org/511986 | 19:38 |
| *** dbecker_ has quit IRC | 19:38 | |
| *** iyamahat has quit IRC | 19:38 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Remove zuul-dev.o.o https://review.openstack.org/532615 | 19:39 |
| *** rmcall has quit IRC | 19:43 | |
| *** rlandy has quit IRC | 19:43 | |
| *** hemna_ has joined #openstack-infra | 19:45 | |
| *** bmjen has quit IRC | 19:46 | |
| *** bmjen has joined #openstack-infra | 19:46 | |
| *** rmcall has joined #openstack-infra | 19:46 | |
| *** iyamahat has joined #openstack-infra | 19:48 | |
| *** rlandy has joined #openstack-infra | 19:48 | |
| *** jtomasek has quit IRC | 19:49 | |
| mtreinish | We just merged a patch to stestr switching it from argparse to cliff. If you get a sec can you give the master branch of stestr a try and see if it doesn't work for anything | 19:50 |
| *** zerick has quit IRC | 19:51 | |
| *** zerick_ has joined #openstack-infra | 19:51 | |
| *** ameliac has joined #openstack-infra | 19:51 | |
| mtreinish | I'm also seeing some signficant slowdowns (like 70%) on smaller test suites with cliff, but masayukig_ can't reproduce it on his end. So if sees anything like that it'd be useful info | 19:51 |
| *** dtantsur is now known as dtantsur|afk | 19:53 | |
| *** vsaienk0 has quit IRC | 19:54 | |
| fungi | i wonder if cliff is triggering entrypoint discovery under some circumstances? | 19:54 |
| *** vsaienk0 has joined #openstack-infra | 19:54 | |
| clarkb | ya I think it does | 19:55 |
| mtreinish | fungi: I was thinking the stevedore lookup is likely what's going on, but in my testing I wasn't seeing that across all commands | 19:55 |
| clarkb | and then your performance is based on number of packages installed and speed of your disk io | 19:55 |
| mtreinish | I figured it would be a static hit on command start | 19:55 |
| clarkb | so masayukig[m] may not have a bunch of stuff installed or has really fast disks or both | 19:55 |
| mtreinish | there's a bunch of data in: https://github.com/mtreinish/stestr/pull/100 | 19:56 |
| *** nicolasbock has quit IRC | 19:56 | |
| *** bobh has quit IRC | 19:57 | |
| *** bobh has joined #openstack-infra | 19:57 | |
| *** iyamahat has quit IRC | 19:57 | |
| mtreinish | smcginnis: ^^^ you might want to test the master branch out too since you've been playing with stestr things lately | 20:01 |
| *** jamesmcarthur has quit IRC | 20:02 | |
| *** iyamahat has joined #openstack-infra | 20:05 | |
| *** dhajare has quit IRC | 20:05 | |
| *** jamesmcarthur has joined #openstack-infra | 20:07 | |
| *** slaweq has joined #openstack-infra | 20:09 | |
| *** ralonsoh has quit IRC | 20:09 | |
| openstackgerrit | Merged openstack-infra/storyboard master: Cleanup Migration Docs https://review.openstack.org/529177 | 20:10 |
| smcginnis | mtreinish: Cool, I'll try to check it out. | 20:15 |
| *** iyamahat has quit IRC | 20:17 | |
| *** ldnunes has quit IRC | 20:17 | |
| *** jamesmcarthur has quit IRC | 20:18 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: Really change patchset column to string https://review.openstack.org/532459 | 20:18 |
| *** jamesmcarthur has joined #openstack-infra | 20:20 | |
| slaweq | hello | 20:24 |
| *** rmcall has quit IRC | 20:25 | |
| slaweq | do You maybe know about issue like http://logs.openstack.org/84/531684/2/check/neutron-functional/0efb05b/job-output.txt.gz#_2018-01-10_18_36_49_996359 ? | 20:25 |
| slaweq | I have similar issues with many test jobs | 20:25 |
| dhellmann | mtreinish , fungi , clarkb : cliff does use pkg_resources to look for entrypoints | 20:26 |
| AJaeger | mordred: I fixed your jeepyb change a bit, now the new job fails, see http://logs.openstack.org/98/531898/7/check/jeepyb-check-pypi/4be4da0/job-output.txt.gz#_2018-01-10_12_19_32_520283 - no idea why.. | 20:26 |
| AJaeger | slaweq: see http://logs.openstack.org/84/531684/2/check/neutron-functional/0efb05b/job-output.txt.gz#_2018-01-10_17_12_11_165342 - the time jump. | 20:27 |
| mtreinish | dhellmann: right, and I would assume that lookup cost would more or less be a static run time cost for all commands | 20:27 |
| *** dave-mccowan has quit IRC | 20:27 | |
| dhellmann | mtreinish : it depends on how many packages you have installed, but yes, it would be the same for all commands | 20:27 |
| fungi | each invocation presumably | 20:28 |
| mtreinish | but that doesn't match the performance numbers I was seeing in my manual testing. So I think there is something else going on too | 20:28 |
| dhellmann | ok | 20:28 |
| * dhellmann vaguely remembers having a conversation about whether cliff would be a good fit for stestr | 20:28 | |
| AJaeger | slaweq: we had some problems with jobs, I hope this is fixed now. | 20:28 |
| mtreinish | dhellmann: yeah a month or 2 ago. You reviewed the patch :) | 20:28 |
| slaweq | AJaeger: ok, so I will try to recheck again :) | 20:28 |
| slaweq | AJaeger: thx for looking on it | 20:29 |
| ianw | pabelanger: (re previous discussion) yeah, having the job install pip packages might be an idempotent solution too. as that element shows though, you can get quite tangled up if you mix pip/package installs which is what we try to smooth over | 20:29 |
| dhellmann | mtreinish : yeah, I remember that we discussed it, but not the details :-) | 20:29 |
| *** mingqian has joined #openstack-infra | 20:30 | |
| openstackgerrit | Merged openstack-infra/storyboard master: Remove Update tox Step https://review.openstack.org/530427 | 20:30 |
| gema | fungi: shall I start the changes to nodepool and zuul, or do you prefer to add the cloud first? | 20:30 |
| fungi | gema: we'd probably want to test out and add credentials to our clouds.yaml configuration first since we wouldn't be able to land the other changes without that | 20:31 |
| gema | fungi: ack, will start there | 20:31 |
| *** Odd_Bloke has left #openstack-infra | 20:31 | |
| fungi | and then we need to manually launch the nodepool builder and mirror instances | 20:31 |
| gema | fungi: what OS do they need to be? we tend to work with debian | 20:32 |
| gema | fungi: due to being at top of the tree of most packages to keep up with code churn | 20:32 |
| gema | (we have repos for that) | 20:33 |
| gema | but I am happy to try ubuntu / centos | 20:33 |
| fungi | gema: we can upload images, would likely be ubuntu xenial | 20:33 |
| clarkb | gema ubuntu is what we currently use for the dib builder and mirror node | 20:33 |
| gema | ok | 20:33 |
| gema | will start there then, thanks fungi and clarkb | 20:34 |
| fungi | i personally prefer debian (and run it on a variety of arm-based systems for that matter) but for better or worse we've standardized primarily on ubuntu since the beginning | 20:34 |
| fungi | and consistency trumps personal preference of course ;) | 20:34 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Move CRD tests to test_gerrit_legacy_crd https://review.openstack.org/531886 | 20:34 |
| gema | yep, ok | 20:35 |
| gema | fungi: for us it is not personal preference, it is how we do early previews of our patches before they trickle to the distros | 20:36 |
| gema | but we can try first with standard ubuntu and see | 20:36 |
| gema | happy to start there | 20:36 |
| openstackgerrit | Mikhail S Medvedev proposed openstack-infra/puppet-openstackci master: Add zookeeper to allinone manifest https://review.openstack.org/526758 | 20:37 |
| gema | it may not be a problem since it will all be virtualised anyway | 20:37 |
| fungi | gema: yeah, and at least i understand ubuntu has been relatively interested in keeping their arm support in good shape | 20:38 |
| fungi | granted it's one of the only architectures they support besides x86 | 20:39 |
| fungi | (unlike debian) | 20:39 |
| *** dave-mccowan has joined #openstack-infra | 20:39 | |
| *** jtomasek has joined #openstack-infra | 20:43 | |
| openstackgerrit | Ihar Hrachyshka proposed openstack-infra/project-config master: Switch grafana neutron board to non-legacy jobs https://review.openstack.org/532632 | 20:45 |
| *** mingqian_ has joined #openstack-infra | 20:47 | |
| gema | fungi: yep, there's a difference between stable and daring though, and xenial is stable :) | 20:48 |
| gema | fungi: we'll make it work, no worries | 20:48 |
| clarkb | does afs work on arm? | 20:50 |
| clarkb | that could be a gotcha with the current mirror setup | 20:50 |
| *** mingqian_ has quit IRC | 20:50 | |
| *** Jeffrey4l has quit IRC | 20:52 | |
| gema | afs? | 20:53 |
| clarkb | gema: andrew filesystem, it is a distributed fs we use to host our mirror data | 20:54 |
| clarkb | but its not in tree so unsure of other arch support | 20:55 |
| *** dprince has quit IRC | 20:55 | |
| gema | clarkb: will check | 20:55 |
| gema | added to todo | 20:55 |
| *** iyamahat has joined #openstack-infra | 20:56 | |
| gema | EOD here, talk to you guys tomorrow | 20:56 |
| mordred | clarkb: oh wow - what a fun new question | 20:56 |
| *** iyamahat_ has joined #openstack-infra | 20:56 | |
| mtreinish | clarkb: that's a good question | 20:57 |
| * mtreinish digs up the openafs source tree | 20:57 | |
| *** jamesmcarthur has quit IRC | 20:59 | |
| *** iyamahat has quit IRC | 21:00 | |
| fungi | indeed, i didn't consider whether the openafs lkm compiles on arm | 21:00 |
| mtreinish | fungi: shouldn't you know, you use a lot of arm devices. Ever mount the afs mirrors on one? | 21:00 |
| *** jamesmcarthur has joined #openstack-infra | 21:00 | |
| fungi | mtreinish: unfortunately i'm not lucky enough to possess any arm64 hardware (yet), only 32-bit armel and armhf so far | 21:01 |
| *** iyamahat has joined #openstack-infra | 21:01 | |
| *** iyamahat_ has quit IRC | 21:01 | |
| *** Jeffrey4l has joined #openstack-infra | 21:03 | |
| *** eharney has quit IRC | 21:03 | |
| *** felipemonteiro__ has quit IRC | 21:03 | |
| *** felipemonteiro__ has joined #openstack-infra | 21:04 | |
| ianw | i have a pine64 ... not sure it would run gerrit though | 21:08 |
| clarkb | we can also not have a mirror there to start if its a problem | 21:09 |
| clarkb | but would be nice ti have one | 21:09 |
| fungi | ianw: wouldn't need gerrit, just building teh openafs lkm | 21:11 |
| fungi | and yeah, i've been looking at getting one of those pine64 boards... how do you like it? | 21:11 |
| *** yamahata has joined #openstack-infra | 21:13 | |
| fungi | biggest challenge with building openafs is that we'd need viable kernel source and headers for whatever kernel we boot, and in armland that's not always easy to come by | 21:13 |
| SamYaple | that screams license violation | 21:14 |
| SamYaple | i thought you had to make that stuff available? | 21:14 |
| *** e0ne has quit IRC | 21:14 | |
| persia | SamYaple: Life is complicated, and publish-to-the-world-without-NDA is rarely a clause in open source licenses. | 21:14 |
| persia | But there are public sources of kernel sources known to work for most public ARM environments, and presumably any third-party configurator would be able to get access to kernel source for their hardware. | 21:15 |
| *** Apoorva_ has quit IRC | 21:15 | |
| SamYaple | persia: you cant distribute a kernel without distributing the source though.... thats gplv2 | 21:15 |
| *** Apoorva has joined #openstack-infra | 21:16 | |
| mtreinish | SamYaple: heh, have you looked at the arm chips used in most android phones... | 21:16 |
| mtreinish | a lot of them don't bother | 21:16 |
| persia | SamYaple: Right, so you put the kernel on hardware, and include a written offer to provide source on request. You then set up a customer support site that only allows registered customers to log in and download updates, for which source will be provided by postal mail on request. | 21:16 |
| SamYaple | oh i see your point. sure its a license violation but what can yo udo? | 21:17 |
| SamYaple | fair enough | 21:17 |
| persia | SamYaple: In practice, everything eventually goes mainline for most chips, but sometimes it takes longer than other times, and most of the distros are willing to exchange funds to produce private kernels (with non-public sources) for specific customers, where the provider-customer relationship is preserved, but the patches may not exist on the open internet. | 21:18 |
| persia | My point is that it is *not* a license violation. A copyright holder who happens to also be a recipient and is unable to get source on request is likely to have action (and should consult counsel). The rest of us have no standing. | 21:18 |
| *** smatzek_ has quit IRC | 21:19 | |
| *** smatzek has joined #openstack-infra | 21:20 | |
| SamYaple | its defientely a license violation for gplv2. i dont follow why you say its not | 21:20 |
| *** felipemonteiro_ has joined #openstack-infra | 21:20 | |
| persia | because gplv2 only requires that one make an offer to provide source to those to whom one has distributed the binaries. If you privately give a gplv2 binary to your friend, I have no reason to ask for the source. | 21:20 |
| hasharAway | what persia said | 21:21 |
| fungi | though also a number of the arb-based soc manufacturers _are_ clearly violating the gplv2 and get away with it. some eventually get pressed to cough up source | 21:21 |
| fungi | er, arm-based | 21:22 |
| *** inc0 has quit IRC | 21:22 | |
| *** smatzek_ has joined #openstack-infra | 21:22 | |
| hasharAway | as long as you do not distribute binary, you dont need to disclose the sources | 21:22 |
| *** rhallisey has quit IRC | 21:22 | |
| SamYaple | yea fungi thats what im refering too | 21:22 |
| persia | fungi: Fair enough, but one has to buy the hardware to complain. | 21:22 |
| fungi | and distributing a device with the binary preinstalled _is_ distributing the binary | 21:22 |
| *** inc0 has joined #openstack-infra | 21:22 | |
| fungi | but yes, in many cases it's done out of laziness/ignorance | 21:23 |
| persia | My personal experience with purchasing ARM servers is that some of them can only be purchased with the sort of relationship that includes an NDA, which makes it hard to have public sources. | 21:23 |
| persia | And, yes, ignorance (or lack of custodial awareness of the source) is the most common reason for violation. | 21:23 |
| fungi | and takes people who have purchased the devices complaining to the manufacturer through lawyers to get source which in cases i've seen has often been on the order of years later | 21:23 |
| *** felipemonteiro__ has quit IRC | 21:23 | |
| persia | Sometimes. Sometimes it just takes an email to the support folk at the hardware vendor, and one gets source within the hour. | 21:24 |
| fungi | yep | 21:24 |
| openstackgerrit | Monty Taylor proposed openstack-infra/jeepyb master: Add checking for PyPI project ownership https://review.openstack.org/531898 | 21:24 |
| *** smatzek has quit IRC | 21:24 | |
| mordred | AJaeger: ^^ thank you for your fixes - that should fix the job | 21:24 |
| *** smatzek_ has quit IRC | 21:26 | |
| *** sbra has quit IRC | 21:27 | |
| *** dave-mccowan has quit IRC | 21:29 | |
| openstackgerrit | Ihar Hrachyshka proposed openstack-infra/project-config master: Switch grafana neutron board to non-legacy jobs https://review.openstack.org/532632 | 21:31 |
| *** gouthamr has joined #openstack-infra | 21:32 | |
| *** felipemonteiro_ has quit IRC | 21:32 | |
| *** felipemonteiro_ has joined #openstack-infra | 21:33 | |
| *** vsaienk0 has quit IRC | 21:34 | |
| *** gouthamr_ has joined #openstack-infra | 21:35 | |
| openstackgerrit | Mikhail S Medvedev proposed openstack-infra/puppet-openstackci master: Add partial support for zuulv3 scheduler https://review.openstack.org/526759 | 21:35 |
| openstackgerrit | Mikhail S Medvedev proposed openstack-infra/system-config master: Move zuulv3.o.o definitions into ::openstackci https://review.openstack.org/526760 | 21:35 |
| *** vsaienk0 has joined #openstack-infra | 21:36 | |
| *** madhuvishy|afk is now known as madhuvishy | 21:36 | |
| *** gouthamr has quit IRC | 21:37 | |
| *** hasharAway has quit IRC | 21:38 | |
| *** trown is now known as trown|outtypewww | 21:38 | |
| *** pcaruana has joined #openstack-infra | 21:42 | |
| *** pcaruana has quit IRC | 21:43 | |
| *** kjackal has quit IRC | 21:45 | |
| *** kjackal has joined #openstack-infra | 21:46 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Remove need to start executor as root https://review.openstack.org/532575 | 21:48 |
| *** armaan has quit IRC | 21:48 | |
| openstackgerrit | Sean McGinnis proposed openstack-infra/project-config master: Retire openstack/anvil https://review.openstack.org/532643 | 21:49 |
| *** jtomasek has quit IRC | 21:51 | |
| *** threestrands_ has joined #openstack-infra | 21:52 | |
| *** threestrands_ has quit IRC | 21:53 | |
| *** threestrands_ has joined #openstack-infra | 21:53 | |
| *** threestrands has quit IRC | 21:54 | |
| mordred | infra-root: could I get some reviews on https://review.openstack.org/#/c/532216/ and https://review.openstack.org/#/c/531936/ - they are needed for us to be able to move forward with enabling the ansible/ansible tests | 21:58 |
| *** armaan has joined #openstack-infra | 22:00 | |
| *** smatzek has joined #openstack-infra | 22:00 | |
| *** tpsilva_ has quit IRC | 22:01 | |
| *** iyamahat has quit IRC | 22:01 | |
| *** iyamahat has joined #openstack-infra | 22:02 | |
| *** jascott1 has quit IRC | 22:02 | |
| *** masber has joined #openstack-infra | 22:02 | |
| *** yamahata has quit IRC | 22:03 | |
| *** jascott1 has joined #openstack-infra | 22:03 | |
| *** jascott1 has quit IRC | 22:03 | |
| *** jascott1 has joined #openstack-infra | 22:04 | |
| *** yamahata has joined #openstack-infra | 22:04 | |
| *** smatzek has quit IRC | 22:05 | |
| *** jascott1 has quit IRC | 22:05 | |
| *** gouthamr_ has quit IRC | 22:06 | |
| *** ijw has joined #openstack-infra | 22:06 | |
| *** jamesmcarthur has quit IRC | 22:07 | |
| *** jascott1 has joined #openstack-infra | 22:08 | |
| *** vsaienk0 has quit IRC | 22:08 | |
| mwhahaha | so POST_FAILURES, is that being tracked somewhere? i'm noticing a whole bunch | 22:08 |
| clarkb | mwhahaha: there may be an elastic recheck entry for it | 22:09 |
| clarkb | (sort of distracted by patching right now though) | 22:09 |
| mwhahaha | fatal: [primary]: UNREACHABLE! => {"changed": false, "msg": "SSH Error: data could not be sent to remote host \"198.72.124.206\". Make sure this host can be reached over ssh", "unreachable": true} | 22:09 |
| *** yamahata has quit IRC | 22:09 | |
| fungi | mwhahaha: a whole bunch which just happened, or a whole bunch which happened earlier? | 22:11 |
| mwhahaha | catching up, on my email so recently i think | 22:11 |
| * mwhahaha busts out his utc to normal time | 22:11 | |
| *** rcernin has joined #openstack-infra | 22:11 | |
| mwhahaha | oh maybe about 3 hours ago | 22:12 |
| pabelanger | I'm unsure if we have an elastic-recheck query for that, but sound like maybe we should | 22:12 |
| pabelanger | to help see if there is a pattern | 22:12 |
| *** aeng has joined #openstack-infra | 22:12 | |
| mwhahaha | http://logs.openstack.org/57/532557/1/check/tripleo-ci-centos-7-undercloud-upgrades/2e7e83b/job-output.txt.gz#_2018-01-10_19_12_04_794478 | 22:12 |
| *** ijw has quit IRC | 22:12 | |
| mwhahaha | maybe someone broke soemthing | 22:12 |
| * mwhahaha goes digging | 22:13 | |
| pabelanger | issue with playbook? | 22:13 |
| pabelanger | nope, just SSH issue | 22:13 |
| fungi | we are vigorously rebooting things for meltdown patches, but nothing which should have caused that i don't think | 22:13 |
| pabelanger | it is inap | 22:14 |
| mwhahaha | might have just been a batch when you were rebooting the world | 22:14 |
| pabelanger | we've seen that a few times this week | 22:14 |
| mwhahaha | or that | 22:14 |
| mgagne | pabelanger: no reboot atm related to meltdown | 22:15 |
| *** mingqian has quit IRC | 22:15 | |
| pabelanger | ansible has the concept of SSH retries: https://review.openstack.org/512130/ not that it fixes underlaying SSH issues, but an option to try more then 1 time before failing | 22:15 |
| mwhahaha | log stash indicates it was a problem a few hours ago, guess it's resolved now | 22:18 |
| mwhahaha | but it wasn't specific to a cloud | 22:19 |
| mwhahaha | http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22SSH%20Error%3A%20data%20could%20not%20be%20sent%20to%20remote%20host%5C%22 | 22:19 |
| pabelanger | mwhahaha: specific to zuul-executor? we've also see SSH issue when an executor has a high load | 22:19 |
| mwhahaha | doesn't look specifically | 22:20 |
| *** corvus is now known as jeblair | 22:21 | |
| *** jeblair is now known as corvus | 22:22 | |
| *** kjackal has quit IRC | 22:22 | |
| -openstackstatus- NOTICE: The zuul system is being restarted to apply security updates and will be offline for several minutes. It will be restarted and changes re-equeued; changes approved during the downtime will need to be rechecked or re-approved. | 22:23 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Rename _useBuilder method to useBuilder https://review.openstack.org/531883 | 22:24 |
| *** jamesmcarthur has joined #openstack-infra | 22:26 | |
| *** afred312 has quit IRC | 22:27 | |
| *** felipemonteiro__ has joined #openstack-infra | 22:29 | |
| *** jamesmcarthur has quit IRC | 22:30 | |
| *** jkilpatr has quit IRC | 22:32 | |
| *** felipemonteiro_ has quit IRC | 22:33 | |
| *** Goneri has quit IRC | 22:34 | |
| *** vsaienk0 has joined #openstack-infra | 22:36 | |
| *** kgiusti has left #openstack-infra | 22:37 | |
| *** gouthamr has joined #openstack-infra | 22:37 | |
| *** threestrands_ has quit IRC | 22:41 | |
| *** gouthamr has quit IRC | 22:42 | |
| *** jamesmcarthur has joined #openstack-infra | 22:43 | |
| *** vsaienk0 has quit IRC | 22:46 | |
| *** slaweq has quit IRC | 22:47 | |
| *** threestrands has joined #openstack-infra | 22:55 | |
| *** afred312 has joined #openstack-infra | 22:58 | |
| *** iyamahat has quit IRC | 22:59 | |
| *** edmondsw has quit IRC | 22:59 | |
| fungi | infra-root: heads up, puppetmaster will be rebooting momentarily (see #openstack-infra incident) | 23:00 |
| *** afred312 has quit IRC | 23:03 | |
| *** bobh has quit IRC | 23:03 | |
| fungi | er, openstack-infra-incident | 23:03 |
| *** ijw has joined #openstack-infra | 23:06 | |
| *** afred312 has joined #openstack-infra | 23:07 | |
| openstackgerrit | Matt Riedemann proposed openstack-infra/infra-manual master: Clarify the point of a repo in a zuulv3 job name https://review.openstack.org/532686 | 23:07 |
| *** rmcall has joined #openstack-infra | 23:09 | |
| *** felipemonteiro__ has quit IRC | 23:10 | |
| *** threestrands has quit IRC | 23:11 | |
| *** jamesmcarthur has quit IRC | 23:11 | |
| *** ijw has quit IRC | 23:12 | |
| *** tosky has quit IRC | 23:13 | |
| *** bmjen has quit IRC | 23:19 | |
| mriedem | would i need "zuul_legacy_vars" in "environment" if i'm creating a new zuulv3-based job in the nova repo? | 23:19 |
| mriedem | i'm assuming no? | 23:20 |
| mriedem | in the shell task | 23:20 |
| *** bmjen has joined #openstack-infra | 23:20 | |
| *** threestrands has joined #openstack-infra | 23:20 | |
| *** ijw has joined #openstack-infra | 23:20 | |
| mriedem | or maybe yes if it's using devstack-gate? | 23:20 |
| openstackgerrit | Merged openstack-infra/system-config master: Fix nodepool pause option https://review.openstack.org/532498 | 23:20 |
| *** numans has quit IRC | 23:21 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Adjust check for .stestr directory https://review.openstack.org/532688 | 23:22 |
| mordred | clarkb: ^^ that should fix the POST failure you pointed at | 23:23 |
| mordred | mriedem: you do not | 23:23 |
| mordred | mriedem: well, anything using devstack-gate is a legacy job by definition | 23:23 |
| *** hongbin has quit IRC | 23:23 | |
| mordred | mriedem: so, if you are creating a new devstack-gate job, then yes, you need zuul_legacy_vars | 23:23 |
| mordred | mriedem: I would recommend taking a stab at a nice new shiny job using the devstack and/or tempest base job | 23:24 |
| *** numans has joined #openstack-infra | 23:24 | |
| mordred | mriedem: well, I say that - the devstack base job is definitely ready for prime time and is being used in a bunch of places - I'd check with andreaf about the tempest base job | 23:25 |
| clarkb | mordred: is that patch correct? if the failing file exists you don't want to process subunit? | 23:25 |
| fungi | mriedem: it's strongly encouraged to omit zuul_legacy_vars from non-legacy jobs (and technically using devstack-gate is a legacy design) | 23:25 |
| *** ijw has quit IRC | 23:25 | |
| clarkb | mordred: I think you need two stats. .stestr exists and .stestr/failing does not exist | 23:25 |
| fungi | mriedem: it's a compatibility shim to provide the old $ZUUL_* envvars | 23:26 |
| fungi | mriedem: while native v3 jobs should rely on values in ansible variables (and set their own envvars if they need to pass those values along into script thunks) | 23:26 |
| mriedem | this is my thing https://review.openstack.org/532689 | 23:27 |
| mriedem | ok i'm not sure if this actually needs anything from devstack-gate | 23:28 |
| mriedem | but it is setting devstack-gate vars, because i copied it from a legacy job definition | 23:28 |
| mriedem | this scares me https://github.com/openstack-dev/devstack/blob/master/playbooks/post.yaml | 23:28 |
| fungi | can't disagree with you there... that's mildly frightening | 23:29 |
| mriedem | plus, there is some stuff that's not quite working properly yet wrt logs and such with the new jobs | 23:30 |
| mriedem | anyway, if i can get nova-multiattach working by FF i'll be happy | 23:30 |
| fungi | i'm assuming you mean the 30-odd patterns of artifacts to grab from the job run | 23:30 |
| fungi | being scary | 23:30 |
| mriedem | that and the missing tempest/test artifacts and the devstack log dumped into the console log | 23:30 |
| mriedem | plus, | 23:31 |
| mordred | clarkb: no - you only want to process subunit if the failing file exists | 23:31 |
| mriedem | i just know enough about how project-config and devstack-gate job definitions have worked that i need to ease into this tub | 23:31 |
| clarkb | mordred: but you said that fialing means there are no test results | 23:31 |
| fungi | mriedem: yeah, the result is clearly still being hammered out | 23:31 |
| fungi | for log collection with the new devstack jobs | 23:31 |
| mordred | clarkb: sorry - I may have spoked bad - if test discovery fails due to import/syntax errors, then .stestr will still be created but .stestr/failing will not be created | 23:32 |
| mriedem | https://www.nbc.com/saturday-night-live/video/james-brown-celebrity-hot-tub-party/n9133 | 23:32 |
| mordred | clarkb: so the existence of .stestr/failing is a thing we can check for to see if stestr actually ran some tests (the file being called failing isn't relevant) | 23:33 |
| mriedem | in this metaphor, i'm eddie murphy as james brown and zuulv3 is the hot tub | 23:33 |
| mordred | mriedem: that's one heck of a metaphor | 23:33 |
| clarkb | mordred: gotcha ya I think the commit message says somethingdifferent. Or I suck at reading. I have been rebooting stuff for about 30 hours now so my eyeballs likely don't work | 23:33 |
| mordred | clarkb: fair - it's also an inherent double-negative, so it's worth a triple-check | 23:34 |
| *** armax has quit IRC | 23:36 | |
| ianw | infra-root: per my latest comment in https://review.openstack.org/#/c/532395 i think that revert restores us to status quo pip setup on images. i will merge that and do a point release of dib, then we can rebuild images. i think it would be good to get pti kernels into testing, not so much for security concerns but just testing sanity | 23:36 |
| *** rlandy is now known as rlandy|bbl | 23:37 | |
| clarkb | ianw: ++ | 23:37 |
| clarkb | ianw: also thinking about that we could probably make both pythons the default behavior then let you select one or the other to meet greghaynes' use case | 23:37 |
| *** wolverineav has quit IRC | 23:37 | |
| fungi | ianw: thanks, i concur | 23:38 |
| ianw | clarkb: there's a bit of chatter about that in #openstack-dib, i think this element might be a bit of a special case | 23:39 |
| *** efried has quit IRC | 23:44 | |
| *** dingyichen has joined #openstack-infra | 23:44 | |
| *** sdague has quit IRC | 23:47 | |
| *** iyamahat has joined #openstack-infra | 23:48 | |
| *** iyamahat_ has joined #openstack-infra | 23:50 | |
| *** iyamahat has quit IRC | 23:50 | |
| *** hjensas has quit IRC | 23:52 | |
| *** freerunner has quit IRC | 23:52 | |
| *** andreykurilin has quit IRC | 23:52 | |
| *** hjensas has joined #openstack-infra | 23:52 | |
| *** SergeyLukjanov has quit IRC | 23:53 | |
| *** andreykurilin has joined #openstack-infra | 23:53 | |
| *** freerunner has joined #openstack-infra | 23:53 | |
| *** jascott1 has quit IRC | 23:54 | |
| *** SergeyLukjanov has joined #openstack-infra | 23:54 | |
| *** jascott1 has joined #openstack-infra | 23:54 | |
| *** edmondsw has joined #openstack-infra | 23:55 | |
| *** jascott1 has quit IRC | 23:57 | |
| *** yamamoto has joined #openstack-infra | 23:58 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!