| ianw | "Remove port 0089da2a-891f-45da-961b-0f56f29c02a7 [1/390]" -- ovh/gra1 ... i'm not sure how in 20 minutes since the last run of the script, we managed to get 390 ports leaked | 00:18 |
|---|---|---|
| *** eernst has joined #openstack-infra | 00:22 | |
| *** felipemonteiro has joined #openstack-infra | 00:28 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Minor documentation updates https://review.openstack.org/609636 | 00:37 |
| *** longkb has joined #openstack-infra | 00:41 | |
| *** jamesmcarthur has quit IRC | 00:49 | |
| *** jamesmcarthur has joined #openstack-infra | 00:52 | |
| *** dave-mccowan has joined #openstack-infra | 01:04 | |
| ianw | Remove port 5a5f9ed7-3061-415f-8431-c2345f49891e [189/496] ... 496 ports now? hrm | 01:09 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Remove build-openstack-sphinx-docs-python3 https://review.openstack.org/609793 | 01:21 |
| *** jamesmcarthur has quit IRC | 01:24 | |
| *** mrsoul has quit IRC | 01:30 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Add a pre-finalise.d phase https://review.openstack.org/609863 | 01:45 |
| *** jamesmcarthur has joined #openstack-infra | 01:45 | |
| *** dave-mccowan has quit IRC | 01:53 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Set EPEL mirror during openstack-ci-mirrors https://review.openstack.org/609169 | 01:54 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Provide for an EPEL mirror during build https://review.openstack.org/608834 | 01:55 |
| *** jamesmcarthur has quit IRC | 02:07 | |
| *** bhavikdbavishi has joined #openstack-infra | 02:14 | |
| *** d0ugal has quit IRC | 02:16 | |
| *** armax has quit IRC | 02:17 | |
| *** d0ugal has joined #openstack-infra | 02:20 | |
| *** jamesmcarthur has joined #openstack-infra | 02:20 | |
| *** hongbin has joined #openstack-infra | 02:24 | |
| *** jamesmcarthur has quit IRC | 02:25 | |
| *** jamesmcarthur has joined #openstack-infra | 02:26 | |
| *** jamesmcarthur has quit IRC | 02:31 | |
| *** jamesmcarthur has joined #openstack-infra | 02:46 | |
| *** eernst has quit IRC | 02:55 | |
| *** psachin has joined #openstack-infra | 02:56 | |
| *** bhavikdbavishi has quit IRC | 02:59 | |
| *** jamesmcarthur has quit IRC | 03:06 | |
| *** jamesmcarthur_ has joined #openstack-infra | 03:06 | |
| *** psachin has quit IRC | 03:11 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Set EPEL mirror during openstack-ci-mirrors https://review.openstack.org/609169 | 03:18 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Provide for an EPEL mirror during build https://review.openstack.org/608834 | 03:18 |
| *** ramishra has joined #openstack-infra | 03:18 | |
| *** jesusaur has quit IRC | 03:21 | |
| *** jesusaur has joined #openstack-infra | 03:26 | |
| *** jamesmcarthur_ has quit IRC | 03:26 | |
| *** gagehugo has quit IRC | 03:34 | |
| *** jamesmcarthur has joined #openstack-infra | 03:34 | |
| *** gagehugo has joined #openstack-infra | 03:36 | |
| *** felipemonteiro has quit IRC | 03:37 | |
| *** bhavikdbavishi has joined #openstack-infra | 03:41 | |
| *** bhavikdbavishi1 has joined #openstack-infra | 03:44 | |
| *** benj_- has joined #openstack-infra | 03:45 | |
| *** bhavikdbavishi has quit IRC | 03:46 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 03:46 | |
| *** benj_- has quit IRC | 03:46 | |
| *** benj_- has joined #openstack-infra | 03:48 | |
| *** jamesmcarthur has quit IRC | 03:52 | |
| *** benj_- has quit IRC | 03:52 | |
| *** udesale has joined #openstack-infra | 03:57 | |
| *** jamesmcarthur has joined #openstack-infra | 04:00 | |
| *** benj_- has joined #openstack-infra | 04:01 | |
| *** benj_- has quit IRC | 04:02 | |
| *** benj_- has joined #openstack-infra | 04:02 | |
| *** jamesmcarthur has quit IRC | 04:04 | |
| *** udesale has quit IRC | 04:07 | |
| *** jamesmcarthur has joined #openstack-infra | 04:15 | |
| openstackgerrit | YAMAMOTO Takashi proposed openstack-infra/project-config master: networking-midonet: Update the dashboard for recent changes https://review.openstack.org/610428 | 04:17 |
| AJaeger | config-core, could you review https://review.openstack.org/610347 , please? | 04:18 |
| *** jamesmcarthur has quit IRC | 04:20 | |
| *** udesale has joined #openstack-infra | 04:20 | |
| *** jamesmcarthur has joined #openstack-infra | 04:22 | |
| *** hongbin has quit IRC | 04:28 | |
| *** psachin has joined #openstack-infra | 04:29 | |
| *** annp has joined #openstack-infra | 04:35 | |
| *** hemna has quit IRC | 04:49 | |
| *** jamesmcarthur has quit IRC | 04:51 | |
| *** hemna has joined #openstack-infra | 04:52 | |
| *** jamesmcarthur has joined #openstack-infra | 04:58 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: [wip] ubuntu element - ignore missing openstack repos https://review.openstack.org/610430 | 05:02 |
| ianw | clarkb / slaweq: we're removing hundreds (like 600) of ports on ovh/gra1 every 20 minutes or so, using our strategy of seeing what is in DOWN state constantly for 3 minutes. I have no idea where these are all coming from; they seem to disappear and re-appear? i would say something is quite wrong | 05:08 |
| ianw | the script is running in a root session on bridge.o.o | 05:09 |
| ianw | screen 0 is gra1, screen 1 in bhs1 | 05:09 |
| chandankumar | AJaeger: ianw hello | 05:10 |
| chandankumar | AJaeger: ianw https://review.openstack.org/#/c/609938/4/.zuul.yaml please have a look at this | 05:10 |
| chandankumar | I am not sure what I am doing wrong here | 05:10 |
| *** jamesmcarthur has quit IRC | 05:11 | |
| *** jamesmcarthur has joined #openstack-infra | 05:14 | |
| promehteanfire | commented, just a guess though | 05:15 |
| dirk | mnaser: hwoarang is out, how can I see the 'brokenness'? | 05:25 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: [wip] ubuntu element - ignore missing openstack repos https://review.openstack.org/610430 | 05:26 |
| *** jamesmcarthur has quit IRC | 05:33 | |
| *** jamesmcarthur has joined #openstack-infra | 05:35 | |
| dirk | ianw: there isn't any check job anymore in dib testing the openSUSE elements, is there? | 05:36 |
| *** ccamacho has joined #openstack-infra | 05:36 | |
| *** quiquell|off is now known as quiquell | 05:37 | |
| ianw | dirk: umm... opensuse-minimal/opensuse423-build-succeeds in the extras-job | 05:38 |
| ianw | e.g. http://logs.openstack.org/63/609863/2/check/dib-dsvm-functests-python3-extras/627312b/job-output.txt.gz#_2018-10-15_04_56_31_801870 | 05:38 |
| ianw | chandankumar: you can't set variables there? | 05:39 |
| dirk | ianw: does that one boot it as well? | 05:39 |
| dirk | I guess I need to set this to test 150 instead | 05:40 |
| *** jtomasek has quit IRC | 05:45 | |
| chandankumar | ianw: I am following this one http://git.openstack.org/cgit/openstack-infra/tripleo-ci/tree/zuul.d/README.rst#n21 | 05:52 |
| *** jamesmcarthur has quit IRC | 05:52 | |
| chandankumar | ianw: Do I need to define a new job template and add the vars there? | 05:52 |
| *** kjackal has joined #openstack-infra | 05:58 | |
| *** e0ne has joined #openstack-infra | 06:00 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Add a pre-finalise.d phase https://review.openstack.org/609863 | 06:01 |
| *** jamesmcarthur has joined #openstack-infra | 06:04 | |
| AJaeger | chandankumar: promehteanfire is right, you cannot use variables for templates | 06:08 |
| AJaeger | chandankumar: you can overwrite the job | 06:08 |
| AJaeger | chandankumar: but with the changes you do, it'S better to define a new job that inherits | 06:09 |
| *** kopecmartin|off is now known as kopecmartin|ruck | 06:09 | |
| *** jamesmcarthur has quit IRC | 06:09 | |
| AJaeger | chandankumar: you're not following the link you gave - that is for a *job* but you change the *template* | 06:09 |
| *** apetrich has quit IRC | 06:09 | |
| chandankumar | AJaeger: doing that | 06:10 |
| *** jamesmcarthur has joined #openstack-infra | 06:13 | |
| *** jamesmcarthur has quit IRC | 06:17 | |
| openstackgerrit | Dirk Mueller proposed openstack/diskimage-builder master: Add test coverage for openSUSE 15.0 building https://review.openstack.org/610436 | 06:19 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: [wip] ubuntu element - ignore missing openstack repos https://review.openstack.org/610430 | 06:19 |
| *** dpawlik has joined #openstack-infra | 06:20 | |
| *** apetrich has joined #openstack-infra | 06:22 | |
| openstackgerrit | Matthew Thode proposed openstack/diskimage-builder master: enable caching for gentoo builds https://review.openstack.org/604268 | 06:26 |
| *** e0ne has quit IRC | 06:28 | |
| *** kjackal has quit IRC | 06:30 | |
| *** kjackal has joined #openstack-infra | 06:33 | |
| *** pcaruana has quit IRC | 06:37 | |
| *** jbadiapa has joined #openstack-infra | 06:39 | |
| *** jamesmcarthur has joined #openstack-infra | 06:40 | |
| openstackgerrit | Dirk Mueller proposed openstack/diskimage-builder master: Add test coverage for openSUSE 15.0 building https://review.openstack.org/610436 | 06:41 |
| openstackgerrit | Dirk Mueller proposed openstack/diskimage-builder master: WIP: Add test coverage for openSUSE 15.0 building https://review.openstack.org/610436 | 06:41 |
| *** slaweq has joined #openstack-infra | 06:43 | |
| *** jamesmcarthur has quit IRC | 06:45 | |
| *** apetrich has quit IRC | 06:45 | |
| *** Bhujay has joined #openstack-infra | 06:47 | |
| *** witek_ is now known as witek | 06:47 | |
| *** Bhujay has quit IRC | 06:49 | |
| *** bhavikdbavishi1 has joined #openstack-infra | 06:49 | |
| AJaeger | frickler, if you review project-config today, please include https://review.openstack.org/610347. Thanks! | 06:50 |
| *** bhavikdbavishi has quit IRC | 06:51 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Remove legacy-periodic-tempest-dsvm-oslo-latest-full-master https://review.openstack.org/610386 | 06:53 |
| *** bhavikdbavishi1 has quit IRC | 06:54 | |
| *** annp has quit IRC | 06:55 | |
| *** kjackal has quit IRC | 06:55 | |
| *** shardy has joined #openstack-infra | 06:55 | |
| *** mtreinish has quit IRC | 06:55 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Remove periodic-tempest-dsvm-oslo-latest-full-master job https://review.openstack.org/610385 | 06:56 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Remove legacy-periodic-tempest-dsvm-oslo-latest-full-master https://review.openstack.org/610386 | 06:56 |
| *** jtomasek has joined #openstack-infra | 06:57 | |
| *** apetrich has joined #openstack-infra | 06:57 | |
| *** mtreinish has joined #openstack-infra | 06:58 | |
| *** quiquell is now known as quiquell|brb | 06:59 | |
| openstackgerrit | YAMAMOTO Takashi proposed openstack-infra/project-config master: networking-midonet: Update the dashboard for recent changes https://review.openstack.org/610428 | 07:00 |
| *** elod is now known as elod_off | 07:01 | |
| *** pcaruana has joined #openstack-infra | 07:02 | |
| *** ginopc has joined #openstack-infra | 07:03 | |
| *** rcernin has quit IRC | 07:04 | |
| *** rcernin has joined #openstack-infra | 07:07 | |
| *** jtomasek has quit IRC | 07:11 | |
| *** jamesmcarthur has joined #openstack-infra | 07:12 | |
| *** jtomasek has joined #openstack-infra | 07:12 | |
| *** jamesmcarthur has quit IRC | 07:16 | |
| promehteanfire | ianw: if you are around I think I used the wrong var for my pre-finalize.d (and cleanup.d) stuff, I don't think it's TMP_MOUNT_PATH, you remember what it is? | 07:19 |
| *** rcernin has quit IRC | 07:21 | |
| *** evrardjp_ is now known as evrardjp | 07:21 | |
| *** kjackal has joined #openstack-infra | 07:25 | |
| *** jtomasek has quit IRC | 07:28 | |
| *** rpittau has joined #openstack-infra | 07:29 | |
| *** adriancz has joined #openstack-infra | 07:29 | |
| *** jtomasek has joined #openstack-infra | 07:30 | |
| promehteanfire | ianw: nvm, going to move more stuff to cleanup | 07:32 |
| *** annp has joined #openstack-infra | 07:35 | |
| *** jamesmcarthur has joined #openstack-infra | 07:36 | |
| openstackgerrit | Matthew Thode proposed openstack/diskimage-builder master: enable caching for gentoo builds https://review.openstack.org/604268 | 07:40 |
| *** jamesmcarthur has quit IRC | 07:41 | |
| *** quiquell|brb is now known as quiquell | 07:42 | |
| *** ssbarnea has joined #openstack-infra | 07:43 | |
| *** tosky has joined #openstack-infra | 07:45 | |
| *** AJaeger has quit IRC | 07:45 | |
| *** hashar has joined #openstack-infra | 07:48 | |
| *** AJaeger has joined #openstack-infra | 07:48 | |
| *** jtomasek has quit IRC | 07:53 | |
| *** jamesmcarthur has joined #openstack-infra | 07:54 | |
| *** roman_g has joined #openstack-infra | 07:54 | |
| *** kjackal has quit IRC | 07:55 | |
| *** bhavikdbavishi has joined #openstack-infra | 07:55 | |
| *** bhavikdbavishi has quit IRC | 07:55 | |
| *** bhavikdbavishi has joined #openstack-infra | 07:55 | |
| *** kjackal has joined #openstack-infra | 07:56 | |
| *** jamesmcarthur has quit IRC | 07:58 | |
| openstackgerrit | Merged openstack-infra/project-config master: Readd openstack-tox-py35-with-neutron-lib-master https://review.openstack.org/610347 | 07:58 |
| *** jamesmcarthur has joined #openstack-infra | 08:04 | |
| *** rascasoft has joined #openstack-infra | 08:06 | |
| *** jpich has joined #openstack-infra | 08:06 | |
| *** kaiokmo has quit IRC | 08:07 | |
| *** kaiokmo has joined #openstack-infra | 08:07 | |
| *** jamesmcarthur has quit IRC | 08:08 | |
| *** jtomasek has joined #openstack-infra | 08:10 | |
| *** jtomasek has quit IRC | 08:14 | |
| *** hashar is now known as hasharAway | 08:15 | |
| *** ChosSimba1 has joined #openstack-infra | 08:21 | |
| *** jamesmcarthur has joined #openstack-infra | 08:27 | |
| *** jamesmcarthur has quit IRC | 08:32 | |
| *** lucasagomes has joined #openstack-infra | 08:34 | |
| *** Bhujay has joined #openstack-infra | 08:36 | |
| *** Bhujay has quit IRC | 08:37 | |
| *** jamesmcarthur has joined #openstack-infra | 08:37 | |
| *** Bhujay has joined #openstack-infra | 08:37 | |
| openstackgerrit | Matthew Thode proposed openstack/diskimage-builder master: enable caching for gentoo builds https://review.openstack.org/604268 | 08:40 |
| *** derekh has joined #openstack-infra | 08:41 | |
| *** jamesmcarthur has quit IRC | 08:41 | |
| *** carl_cai has joined #openstack-infra | 08:43 | |
| *** jtomasek has joined #openstack-infra | 08:44 | |
| *** olivierb has joined #openstack-infra | 08:51 | |
| *** jamesmcarthur has joined #openstack-infra | 09:14 | |
| openstackgerrit | Dirk Mueller proposed openstack/diskimage-builder master: WIP: Add test coverage for openSUSE 15.0 building https://review.openstack.org/610436 | 09:17 |
| openstackgerrit | Dirk Mueller proposed openstack/diskimage-builder master: Update test coverage for openSUSE-minial to 15.0 https://review.openstack.org/610436 | 09:24 |
| *** lpetrut has joined #openstack-infra | 09:35 | |
| *** Bhujay has quit IRC | 09:47 | |
| *** imacdonn has quit IRC | 09:54 | |
| *** Bhujay has joined #openstack-infra | 09:54 | |
| *** imacdonn has joined #openstack-infra | 09:54 | |
| *** ramishra has quit IRC | 09:55 | |
| *** Bhujay has quit IRC | 09:55 | |
| *** Bhujay has joined #openstack-infra | 09:56 | |
| *** charz has quit IRC | 10:15 | |
| *** kjackal has quit IRC | 10:22 | |
| *** kjackal_v2 has joined #openstack-infra | 10:22 | |
| *** ramishra has joined #openstack-infra | 10:23 | |
| *** electrofelix has joined #openstack-infra | 10:25 | |
| *** psachin has quit IRC | 10:32 | |
| *** ginux has joined #openstack-infra | 10:33 | |
| *** ginux is now known as Guest94741 | 10:33 | |
| *** ginopc has quit IRC | 10:35 | |
| *** Guest94741 has quit IRC | 10:37 | |
| *** adriancz has quit IRC | 10:39 | |
| *** yamamoto has quit IRC | 10:47 | |
| *** yamamoto has joined #openstack-infra | 10:48 | |
| *** yamamoto has quit IRC | 10:52 | |
| *** ginopc has joined #openstack-infra | 10:54 | |
| *** psachin has joined #openstack-infra | 10:56 | |
| *** dave-mccowan has joined #openstack-infra | 11:01 | |
| *** lifeless has quit IRC | 11:04 | |
| *** hasharAway is now known as hashar | 11:08 | |
| *** kjackal_v2 has quit IRC | 11:10 | |
| *** psachin has quit IRC | 11:10 | |
| *** annp has quit IRC | 11:11 | |
| *** longkb has quit IRC | 11:11 | |
| openstackgerrit | Dirk Mueller proposed openstack-infra/project-config master: Reenable nodepool testing for opensuse images https://review.openstack.org/610477 | 11:16 |
| *** jamesdenton has joined #openstack-infra | 11:24 | |
| *** udesale has quit IRC | 11:24 | |
| *** yamamoto has joined #openstack-infra | 11:30 | |
| *** dtantsur is now known as dtantsur|brb | 11:30 | |
| *** jesusaur has quit IRC | 11:31 | |
| *** ansmith has quit IRC | 11:36 | |
| *** jesusaur has joined #openstack-infra | 11:37 | |
| *** kjackal has joined #openstack-infra | 11:38 | |
| *** gfidente has joined #openstack-infra | 11:42 | |
| *** yamamoto has quit IRC | 11:43 | |
| *** yamamoto has joined #openstack-infra | 11:43 | |
| quiquell | AJaeger: Do you know if it's possible to rewrite ansible_python_interpreter at zuul's job "vars:" ? | 11:47 |
| *** kjackal has quit IRC | 11:47 | |
| *** ldnunes has joined #openstack-infra | 11:47 | |
| *** fnordahl has quit IRC | 11:49 | |
| *** fnordahl has joined #openstack-infra | 11:49 | |
| *** kjackal has joined #openstack-infra | 11:51 | |
| *** kaiokmo has quit IRC | 11:54 | |
| *** kaiokmo has joined #openstack-infra | 11:54 | |
| *** quiquell is now known as quiuell|lunch | 12:00 | |
| *** dpawlik has quit IRC | 12:07 | |
| *** bhavikdbavishi1 has joined #openstack-infra | 12:07 | |
| *** bhavikdbavishi has quit IRC | 12:09 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 12:09 | |
| *** jamesmcarthur has quit IRC | 12:11 | |
| *** jamesmcarthur has joined #openstack-infra | 12:11 | |
| *** trown|outtypewww is now known as trown | 12:12 | |
| *** dpawlik has joined #openstack-infra | 12:14 | |
| *** boden has joined #openstack-infra | 12:14 | |
| *** yamamoto has quit IRC | 12:14 | |
| *** yamamoto has joined #openstack-infra | 12:15 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool master: Use quota handling code for min-ready declines https://review.openstack.org/610495 | 12:17 |
| *** kjackal has quit IRC | 12:20 | |
| *** kjackal has joined #openstack-infra | 12:20 | |
| *** yamamoto has quit IRC | 12:20 | |
| *** zul has joined #openstack-infra | 12:20 | |
| *** adriancz has joined #openstack-infra | 12:22 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool master: Remove unneeded todo comment https://review.openstack.org/610496 | 12:22 |
| *** bhavikdbavishi1 has joined #openstack-infra | 12:28 | |
| *** weshay has joined #openstack-infra | 12:29 | |
| *** bhavikdbavishi has quit IRC | 12:29 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 12:29 | |
| weshay | morning, hoping to get https://review.openstack.org/#/c/610472/ pushed to the top of the tripleo queue to resolve gate failures | 12:29 |
| *** jamesmcarthur has quit IRC | 12:30 | |
| AJaeger | weshay: that fails all RDO checks - why do you think it's working? | 12:32 |
| weshay | AJaeger, rdo-cloud is down | 12:32 |
| AJaeger | weshay: then shutdown the reporting, please | 12:32 |
| weshay | AJaeger, will try | 12:33 |
| AJaeger | weshay: also that change has not finished running checks at all... Why will this change fix the failure? And why do you now have a failure you didn't had before? Best to explain that better in your commit message... | 12:34 |
| *** rlandy has joined #openstack-infra | 12:34 | |
| AJaeger | weshay: I can't promote, just pointing out that the info you give in that change does not help anybody to understand why this is needed and the proper fix. | 12:35 |
| *** jroll has quit IRC | 12:38 | |
| AJaeger | mnaser: dirk looked into the openSUSE images, could you look at https://review.openstack.org/610477 , please? | 12:38 |
| *** jroll has joined #openstack-infra | 12:39 | |
| *** kgiusti has joined #openstack-infra | 12:40 | |
| *** agopi is now known as agopi|brb | 12:40 | |
| *** quiuell|lunch is now known as quiquell | 12:41 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool master: Ensure that completed handlers are removed frequently https://review.openstack.org/610029 | 12:41 |
| *** agopi|brb has quit IRC | 12:45 | |
| *** derekh has quit IRC | 12:46 | |
| *** yamamoto has joined #openstack-infra | 12:46 | |
| *** EmilienM has quit IRC | 12:46 | |
| *** EmilienM has joined #openstack-infra | 12:47 | |
| *** nhicher has joined #openstack-infra | 12:54 | |
| *** jamesmcarthur has joined #openstack-infra | 12:54 | |
| *** haleyb has joined #openstack-infra | 12:55 | |
| *** dtantsur|brb is now known as dtantsur | 12:59 | |
| *** fried_rice is now known as efried | 13:02 | |
| *** felipemonteiro has joined #openstack-infra | 13:03 | |
| *** agopi|brb has joined #openstack-infra | 13:03 | |
| *** bhavikdbavishi has quit IRC | 13:04 | |
| *** ansmith has joined #openstack-infra | 13:04 | |
| *** agopi|brb is now known as agopi | 13:06 | |
| *** bobh has joined #openstack-infra | 13:07 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool master: Use quota handling code for min-ready declines https://review.openstack.org/610495 | 13:07 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool master: Remove unneeded todo comment https://review.openstack.org/610496 | 13:07 |
| *** smarcet has joined #openstack-infra | 13:18 | |
| *** yamamoto has quit IRC | 13:18 | |
| *** yamamoto has joined #openstack-infra | 13:19 | |
| *** mriedem has joined #openstack-infra | 13:19 | |
| *** lbragstad has joined #openstack-infra | 13:20 | |
| *** zxiiro-pto is now known as zxiiro | 13:23 | |
| *** yamamoto has quit IRC | 13:23 | |
| *** sthussey has joined #openstack-infra | 13:23 | |
| *** SteelyDan is now known as dansmith | 13:28 | |
| *** yamamoto has joined #openstack-infra | 13:34 | |
| *** jrist has quit IRC | 13:34 | |
| *** jrist has joined #openstack-infra | 13:38 | |
| *** rh-jelabarre has joined #openstack-infra | 13:40 | |
| frickler | infra-root: I made an autohold request earlier with "--count 1", but i zuul actually did hold two nodes for two failures of that job 2.5h apart. is that a known issue? | 13:40 |
| *** dave-mccowan has quit IRC | 13:42 | |
| *** carl_cai has quit IRC | 13:42 | |
| *** roman_g has quit IRC | 13:45 | |
| *** rpioso|afk is now known as rpioso | 13:46 | |
| *** eharney has joined #openstack-infra | 13:47 | |
| *** hwoarang_ has joined #openstack-infra | 13:49 | |
| AJaeger | frickler, clarkb , https://review.openstack.org/610270 switches our lib-forward-testing template, could either of you review it, please? | 13:49 |
| dirk | frickler: which change? The opensuse 150 image boot? | 13:50 |
| *** hwoarang has quit IRC | 13:53 | |
| *** sshnaidm_ has joined #openstack-infra | 13:53 | |
| *** sshnaidm has quit IRC | 13:54 | |
| smcginnis | Not sure if anyone would know and I'm guessing it's too early for Clark, but last Friday I pointed out gerritbot isn't commenting in the #openstack-cinder channel. | 13:54 |
| *** gfidente has quit IRC | 13:55 | |
| smcginnis | That still appears to be the case. Haven't seen anything since late last week, and I submitted a patch this morning that did not show up there. | 13:55 |
| mordred | frickler: not that I know of, no | 13:56 |
| stephenfin | Is it just me or are the git-review tests broken? I'm seeing the following error http://paste.openstack.org/show/732155/ | 13:57 |
| stephenfin | Looks to be because 'ssh-keygen' is generating keyfiles in non-PEM format which BouncyCastle doesn't like, but there's no switch available on 'ssh-keygen' to fix this and even the latest version of Gerrit from tarballs.o.o doesn't fix things :/ | 13:58 |
| *** anteaya has joined #openstack-infra | 13:58 | |
| openstackgerrit | Matthew Thode proposed openstack/diskimage-builder master: enable caching for gentoo builds https://review.openstack.org/604268 | 13:59 |
| mordred | stephenfin: well that's certainly exciting | 14:00 |
| *** jesusaur has quit IRC | 14:00 | |
| openstackgerrit | Stephen Finucane proposed openstack-infra/git-review master: Remove auto-branch name https://review.openstack.org/610573 | 14:00 |
| * stephenfin pushes reviews and let's the CI take the hit | 14:00 | |
| Shrews | frickler: did you delete one of the held nodes? I see only one held by you | 14:01 |
| openstackgerrit | Stephen Finucane proposed openstack-infra/git-review master: docs: Misc updates https://review.openstack.org/610574 | 14:01 |
| mordred | stephenfin: we recently made a new functional test setup for zuul which, amongst other things, spins up a gerrit in a docker container ... once we've got that solid, maybe we should have git-review use the same construct | 14:01 |
| openstackgerrit | Stephen Finucane proposed openstack-infra/git-review master: tox: Remove dead settings/targets https://review.openstack.org/610576 | 14:02 |
| openstackgerrit | Stephen Finucane proposed openstack-infra/git-review master: Allow '@' in username https://review.openstack.org/610577 | 14:02 |
| stephenfin | mordred: Not a bad idea. Would certainly help reproducibility | 14:02 |
| * stephenfin had an idea to have tox use "backends", with virtualenv being the default but docker being another. Just need to find time to do | 14:03 | |
| mordred | stephenfin: http://git.openstack.org/cgit/openstack/dox <-- I've had similar thoughts before myself :) | 14:04 |
| stephenfin | Ooh, nifty | 14:05 |
| mordred | stephenfin: http://git.openstack.org/cgit/openstack-infra/zuul/tree/.zuul.yaml#n61 is the job we've got in zuul I was talking about | 14:05 |
| *** Bhujay has quit IRC | 14:06 | |
| *** jesusaur has joined #openstack-infra | 14:06 | |
| mordred | stephenfin: http://git.openstack.org/cgit/openstack-infra/zuul/tree/playbooks/quick-start/run.yaml#n18 <-- it already installs git-review as part of what it does | 14:06 |
| *** eernst has joined #openstack-infra | 14:09 | |
| *** munimeha1 has joined #openstack-infra | 14:12 | |
| *** hwoarang_ is now known as hwoarang | 14:19 | |
| stephenfin | mordred: Yup, that looks like a good model to work with. Not sure it would help with local dev though, unless there was a Dockerfile in tree? (I haven't checked) | 14:21 |
| *** radez has joined #openstack-infra | 14:22 | |
| openstackgerrit | Stephen Finucane proposed openstack-infra/git-review master: Remove auto-branch name https://review.openstack.org/610573 | 14:23 |
| *** promehteanfire has quit IRC | 14:24 | |
| *** anteaya has quit IRC | 14:24 | |
| *** sshnaidm_ has quit IRC | 14:25 | |
| *** quiquell is now known as quiquell|off | 14:26 | |
| *** jamielennox has quit IRC | 14:27 | |
| *** anteaya has joined #openstack-infra | 14:29 | |
| *** jamielennox has joined #openstack-infra | 14:29 | |
| openstackgerrit | Stephen Finucane proposed openstack-infra/git-review master: Remove auto-branch name https://review.openstack.org/610573 | 14:31 |
| mordred | stephenfin: there's a whole docker-compose stack actually | 14:35 |
| mordred | stephenfin: but zuul and friends are a little special WRT containers - we're using pbrx to build our containers | 14:35 |
| frickler | Shrews: yes, I deleted the first held node, checked the nodepool list to confirm deletion, and then stumbled about the second held node. I verified that they belong to different runs 2.5h apart. patch in question is https://review.openstack.org/610427 | 14:38 |
| frickler | dirk: no, different change ^^ | 14:39 |
| *** armax has joined #openstack-infra | 14:45 | |
| *** sshnaidm_ has joined #openstack-infra | 14:45 | |
| *** eernst has quit IRC | 14:51 | |
| *** eernst has joined #openstack-infra | 14:52 | |
| mordred | infra-root: I started poking at gerrit 2.15 and building/upgrading over the weekend | 14:53 |
| openstackgerrit | Merged openstack-infra/nodepool master: Update pypi metadata https://review.openstack.org/557738 | 14:53 |
| dmsimard | mordred: is someone taking bets on complaints about the new UI on openstack-dev ? | 14:53 |
| *** dpawlik has quit IRC | 14:54 | |
| mordred | dmsimard: heh :) | 14:54 |
| mordred | dmsimard: I, for one, welcome our new polygerrit overlords | 14:54 |
| dmsimard | ++ | 14:55 |
| dmsimard | I have just bad memories of how it went down the last time the UI changed ever so slightly | 14:55 |
| mordred | dmsimard: I'm guessing single-page-diff will win over most naysayers | 14:55 |
| openstackgerrit | Stephen Finucane proposed openstack-infra/git-review master: docs: Call out use of an agent to store SSH passwords https://review.openstack.org/610616 | 15:00 |
| *** beekneemech is now known as bnemec | 15:01 | |
| *** Bhujay has joined #openstack-infra | 15:02 | |
| openstackgerrit | Merged openstack-infra/zuul master: Include enqueue and report timestamps in MQTT msg https://review.openstack.org/609355 | 15:03 |
| *** promehteanfire has joined #openstack-infra | 15:03 | |
| *** cdent has joined #openstack-infra | 15:04 | |
| *** efried has quit IRC | 15:05 | |
| cdent | I'm trying to figure out if a tempest plugin can define a zuul job which other projects can use. Since tempest plugins get added dynamically I'm not clear on how/if that would work. This is related to: https://review.openstack.org/#/c/607507/ | 15:11 |
| cdent | and https://anticdent.org/gabbi-in-the-gate.html | 15:11 |
| *** efried has joined #openstack-infra | 15:13 | |
| *** roman_g has joined #openstack-infra | 15:13 | |
| *** Bhujay has quit IRC | 15:14 | |
| openstackgerrit | Merged openstack-infra/zuul master: Retry jobs failed with MERGER_FAILURE https://review.openstack.org/609945 | 15:15 |
| openstackgerrit | Merged openstack-infra/zuul master: docs: add Project Testing Interface guide https://review.openstack.org/571420 | 15:15 |
| corvus | cdent: yes, any project can use a job defined by any other project in the system (for example, from zuul's pov, there's nothing special about the devstack and tempest jobs which are defined in those respective projects) | 15:16 |
| frickler | #status log force-merged https://review.openstack.org/610484 in order to work around gate issue for OpenStack Chef cookbook CI | 15:16 |
| openstackstatus | frickler: finished logging | 15:16 |
| *** ssbarnea has quit IRC | 15:16 | |
| cdent | corvus: yeah, I got that, but I wasn't clear if a tempest plugin (loaded later in the process) would be visible as an untrusted project? | 15:16 |
| cdent | so if gabbi-tempest plugin had a zuul config file in its own repo, could openstack/placement be aware of it by default, or need to do some extra work? | 15:17 |
| AJaeger | cdent: it would just work by magic ;) | 15:17 |
| cdent | where can I look at this magic, so I can understand more better? | 15:18 |
| cdent | an important note at this point: gabbi-tempest is not in git.openstack.org _yet_ but presumably will be eventually | 15:18 |
| corvus | cdent: ah -- zuul has a global config -- it reads all of the config files in every repo and combines them -- so there's no temporality to the configuration. so openstack-placement can add "gabbi-tempest-job" to its check/gate pipelines and it will run | 15:18 |
| corvus | cdent: oh that's an important point. it does need to be in gerrit. | 15:19 |
| cdent | so basically when zuul spins up it reads openstack/*/<various zuul files> to make its jobs, yes? | 15:19 |
| corvus | cdent: yes. i realize it sounds ridiculous, but it does read all 6000 project/branch combinations. it takes about 80 seconds. | 15:20 |
| cdent | :) | 15:20 |
| corvus | cdent: (then once it's running, it just watches for changes landing and patches in the deltas) | 15:20 |
| cdent | can I force a job lookup from elsewhere, while expermenting? | 15:20 |
| cdent | or should I bite the bullet and move gabbi-tempest into openstack git? | 15:21 |
| *** gfidente has joined #openstack-infra | 15:22 | |
| *** dpawlik has joined #openstack-infra | 15:23 | |
| corvus | cdent: no, adding projects to the system is admin configuration, and we have a policy to only load job definitions from our gerrit (we can test projects in github, but we don't load zuul config from them due to the possibility of introducing errors into the global config). so best bets are to go ahead and import, or for experimenting, you should be able to do WIP changes like you've started (in | 15:23 |
| corvus | tempest, or even openstack/sandbox). | 15:23 |
| cdent | great, thanks for the info | 15:24 |
| cdent | that gives me some direction | 15:24 |
| corvus | cdent: no prob; let me know if you have more questions. the general idea you have for the job structure looks solid. | 15:25 |
| cdent | great, thanks | 15:26 |
| cmurphy | if any infra-root is bored i have a few non-dangerous puppet changes sitting in my queue that would be nice to clear through https://review.openstack.org/#/q/is:open+topic:puppet-4+NOT+label:workflow%253D-1+NOT+message:%22future+parser%22 | 15:26 |
| *** dpawlik has quit IRC | 15:27 | |
| clarkb | stephenfin: mordred I thought gerrit generated those files on startup? in any case is that saying your local version of ssh keygen produces different output than the CI system's? | 15:28 |
| *** mlavalle has joined #openstack-infra | 15:29 | |
| clarkb | mordred: as for Gerrit upgrade I learned a new thing last week and that is on 2.14/2.15 the notedb transition can apparently be done online? I think that means our upgrade path is 2.13 -> 2.14 with online migration to notedeb, then 2.15 either after that migration or once whatever changes from 2.14 are in place which may be a quick switch or long one (I know you have to do 2.13 -> 2.14 -> 2.15 and | 15:30 |
| clarkb | can't skip 2.14 from last I checked) | 15:30 |
| mlavalle | Hi, I am trying to see if gerritbot is running in the neutron channel and, if not, restart it. I tried these commands in the channegl, but don't seem to do anything: https://docs.openstack.org/infra/system-config/irc.html#troubleshooting | 15:30 |
| stephenfin | clarkb: We seem to be doing it for...some reason https://github.com/openstack-infra/git-review/blob/master/git_review/tests/__init__.py#L156-L175 | 15:30 |
| mlavalle | what am I missing? | 15:30 |
| *** ramishra has quit IRC | 15:31 | |
| clarkb | mlavalle: I think we've seen gerritbots channel joining mechanism that handles more than 120 channels (the connection limit for freenode) get broken by the freendoe global +r behavior. I think we thought we had fixed this with SASL auth but it doesn't seem to have helped | 15:31 |
| *** Bhujay has joined #openstack-infra | 15:32 | |
| *** roman_g has quit IRC | 15:32 | |
| mlavalle | clarkb: just for my education, the commands in that document are just supposed to be typed in the channel like that? | 15:32 |
| clarkb | mlavalle: the commands in that document are for infra root to debug the bot service on our servers | 15:33 |
| mlavalle | clarkb: ah ok | 15:33 |
| clarkb | mlavalle: they are terminal commands for the server running the instance of the bot | 15:33 |
| mordred | clarkb: yah - so - one of the things I've learned over the weekend is that 2.14 is pretty unbuildable at the moment | 15:33 |
| *** munimeha1 has quit IRC | 15:33 | |
| mlavalle | clarkb: Thanks for the education :-) | 15:34 |
| *** chandankumar is now known as chkumar|off | 15:35 | |
| clarkb | stephenfin: ah it is because we do a single install then copy it for the different running threads so pregenerate the host key so it works everywhere | 15:35 |
| mordred | clarkb: so my initial thought (subject to testing of course) is to just use the plain upstream 2.14 war they've published to run an online migration, then do an immediate re-upgrade to 2.15 using one we've built | 15:35 |
| clarkb | stephenfin: there is talk of using a single instance of gerrit but I'm not sure if ssbarnea has started that work | 15:35 |
| mordred | clarkb: rather than spending too much time trying to figure out how to get a build of 2.14 through zuul | 15:36 |
| *** smarcet has quit IRC | 15:36 | |
| clarkb | 2.14 is still supported upstream and they just published new wars for it like last week. I wonder what they are doing to build it | 15:36 |
| mordred | clarkb: 2.15 builds like a charm - and actually the way things work now allow us to just build a war with the plugins we care about in it in the first place, so the combo winds up being nice | 15:36 |
| stephenfin | clarkb: Ah, test parallelization. I'd completely forgotten about that for a moment | 15:36 |
| stephenfin | Makes sense though | 15:36 |
| mordred | clarkb: I get java compile errors :) | 15:36 |
| *** shardy has quit IRC | 15:37 | |
| mordred | clarkb: using java 8 | 15:37 |
| mordred | clarkb: but - since we *really* want to be on 2.15 anyway and since we don't have any substantive patches in 2.13 we need to have exist in 2.14 - I think we can just ignore the issue and use their build for a sec | 15:37 |
| clarkb | mordred: ya chances are good we can get away with that, helps that 2.14 has been aroudn a while too | 15:38 |
| mordred | ++ | 15:38 |
| mordred | clarkb: https://review.openstack.org/#/c/610395/ is a patch adding zuul jobs and container builds to an openstack/2.15 branch | 15:38 |
| clarkb | cmurphy: that reminds me that I held off on puppet 4ing static.o.o because of the TC electio nresults being hosted there. I can approve that change nowish | 15:43 |
| *** hwoarang has quit IRC | 15:44 | |
| *** sdoran has joined #openstack-infra | 15:44 | |
| *** melwitt has joined #openstack-infra | 15:45 | |
| clarkb | infra-root: once I'm caught up on email and scrollback (and things like approving futureparser changes) I'm going to dive into server upgrades and try to push on that this week in particular. I put together https://etherpad.openstack.org/p/201808-infra-server-upgrades-and-cleanup a little while back with compiled info on that | 15:46 |
| clarkb | I'm going to start with etherpad-dev. I think the needed prereq change for that has merged | 15:46 |
| clarkb | if you have time to work on that it is much appreciated and let me know if any related changes need reviews | 15:46 |
| clarkb | I'm going to restart gerritbot now. Maybe the sasl changes didn't actually take | 15:46 |
| *** smarcet has joined #openstack-infra | 15:47 | |
| *** openstackgerrit has quit IRC | 15:47 | |
| clarkb | mlavalle: ^ I've restarted the bot to be sure that our bug fix is applied and running. Can you let us know if you start seeing event messages from it (or if you don't) | 15:47 |
| *** openstackgerrit has joined #openstack-infra | 15:48 | |
| *** fuentess has joined #openstack-infra | 15:48 | |
| mlavalle | clarkb: yes, I'll keep you posted | 15:49 |
| mlavalle | clarkb: thanks! | 15:49 |
| clarkb | AJaeger: for https://review.openstack.org/#/c/610270/3 I think we want to be able to test both things. the library installed from pypi doesn't break changes to the library consumers and when the library updates we don't break the library consumers when the next release happens | 15:49 |
| clarkb | AJaeger: since that is the existing behavior of the new job I don't think we need to prvent your change from merging but we might want to double check with the qa team that we are testing what they expect to be testing | 15:50 |
| *** hwoarang has joined #openstack-infra | 15:51 | |
| AJaeger | gmann: ^ | 15:51 |
| *** roman_g has joined #openstack-infra | 15:53 | |
| *** munimeha1 has joined #openstack-infra | 15:54 | |
| openstackgerrit | Chris Dent proposed openstack-infra/project-config master: Add gabbi-tempest unofficial project https://review.openstack.org/610628 | 15:55 |
| clarkb | ianw: fyi I abandoned https://review.openstack.org/#/c/528959/ as other work seems to have superceded it (re etherpad server upgrades) | 15:57 |
| openstackgerrit | Filippo Inzaghi proposed openstack-infra/project-config master: remove job settings for OpenStackSDK repositories https://review.openstack.org/610631 | 15:57 |
| *** trown is now known as trown|lunch | 15:59 | |
| dmsimard | infra-root: I found an apparent issue with gerritbot that is still ongoing and I'm not sure what's going on. I got these logs yesterday which makes it feel like everything worked: http://paste.openstack.org/show/732129/ but gerritbot isn't actually in the channel and there doesn't seem to be any errors in the logs that I've seen. | 15:59 |
| dmsimard | This is just for #ara but I suspect it could be occurring for other channels as well | 16:00 |
| clarkb | dmsimard: ya see mlavalle's comments above and I debugged ti a bit last friday for smcginnis too | 16:01 |
| dmsimard | oh, let me read, thanks | 16:01 |
| clarkb | dmsimard: freenode +r'd the entire service and I expect this si fallout from that. fungi wrote changes to use sasl to auth gerritbot to work around that which should've fixed it. I just restarted gerritbot to be double sure the sasl auth fixes are in place | 16:01 |
| clarkb | if we see the problem persist we may need to update gerritbot to do more sophistacted channel membership tracking by having it check with freenode to see if it is joined to a channel before talking to the channel | 16:02 |
| dmsimard | I'm surprised that there doesn't seem to be any exceptions or errors raised by the unsuccessful attempt at sending a message | 16:03 |
| dmsimard | but I haven't gone as far as checking where this actually happens in the code | 16:03 |
| smcginnis | I kind of expected to see a bunch of messages in openstack-unregistered, but they aren't going there either. | 16:03 |
| *** e0ne has joined #openstack-infra | 16:04 | |
| dmsimard | oh, +r is the thing that prevents unregistered users from joining channels ? | 16:04 |
| dmsimard | they set that globally ? | 16:04 |
| clarkb | dmsimard: yes, freenode set that globally | 16:04 |
| dmsimard | yikes | 16:04 |
| clarkb | as for error messages I think from gerritbot's perspective there is no error | 16:04 |
| dmsimard | spammers have won :( | 16:04 |
| clarkb | freenode is likely silently dropping messages on the floor to make it appear to the spammers that it is going through | 16:05 |
| dmsimard | ah, perhaps | 16:05 |
| clarkb | the other possibility here is freenode has flagged gerritbot as spam | 16:05 |
| clarkb | since it writes repeated templated messages to channels and parts and joins | 16:05 |
| clarkb | I wanted to be double sure the sasl auth stuff was working before bugging freenode about that possibility though | 16:05 |
| dmsimard | That's happened to the RDO bot before, but it was a brutal k-line so that should be obvious if it's that | 16:06 |
| clarkb | ya I'm wondering if part of the ramping up efforts is to shadow ban people | 16:07 |
| clarkb | its a common technique for combating spammers on other platforms | 16:07 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Update lib-forward-testing https://review.openstack.org/610270 | 16:09 |
| *** gyee has joined #openstack-infra | 16:09 | |
| *** kjackal has quit IRC | 16:10 | |
| *** kjackal has joined #openstack-infra | 16:13 | |
| *** ginopc has quit IRC | 16:14 | |
| clarkb | dmsimard: ^ it is at least working some of the time | 16:17 |
| clarkb | ok quick breakfast before diving into config mgmt changes | 16:20 |
| *** e0ne has quit IRC | 16:21 | |
| *** sshnaidm_ is now known as sshnaidm|afk | 16:24 | |
| *** e0ne has joined #openstack-infra | 16:27 | |
| *** e0ne has quit IRC | 16:28 | |
| *** yamamoto has quit IRC | 16:30 | |
| *** cdent has left #openstack-infra | 16:32 | |
| *** yamamoto has joined #openstack-infra | 16:34 | |
| mlavalle | clarkb: the gerritbot still not working in the Neutron channel | 16:34 |
| clarkb | mlavalle: have changes merged since the restart? | 16:34 |
| clarkb | or I guess changes pushed | 16:34 |
| fungi | clarkb: dmsimard: i'm not really around, but i think we just need to double-check that gerritbot/statusbot are logging server messages. it may be that the first few /joins are failing and the bot isn't logging the error (and also just assumes the /join succeeds) | 16:34 |
| mlavalle | clarkb: no, but I pushed a couple of patches a couple of minutes ago | 16:35 |
| *** smarcet has quit IRC | 16:35 | |
| clarkb | fungi: the trouble is the irc lib doesn't log much when I looked | 16:35 |
| *** lpetrut has quit IRC | 16:37 | |
| *** ssbarnea has joined #openstack-infra | 16:38 | |
| *** anteaya has quit IRC | 16:38 | |
| *** jamesmcarthur has quit IRC | 16:39 | |
| openstackgerrit | Corey Bryant proposed openstack-dev/pbr master: Change python3.5 job to python3.7 job on Stein+ https://review.openstack.org/610659 | 16:39 |
| *** yamamoto has quit IRC | 16:40 | |
| clarkb | coreycb: not sure if you saw but there are apparently problems with paste and python3.7. There is a mailing list thread on -dev about paste support and fixes that cdent started you may want to check on | 16:40 |
| coreycb | clarkb: ok will take a look, thanks! | 16:41 |
| fungi | clarkb: dmsimard: i wonder if the bot could double-check that the /join worked (i.e. that it sees itself in the channel) and error/retry otherwise | 16:41 |
| clarkb | fungi: ya there is also an _on_join() method that we could use to add channels to the list rather than assuming it worked | 16:42 |
| * fungi returns to his frenzied cleaning | 16:43 | |
| *** anteaya has joined #openstack-infra | 16:43 | |
| *** jpich has quit IRC | 16:44 | |
| *** hashar is now known as hasharAway | 16:46 | |
| *** Bhujay has quit IRC | 16:46 | |
| mordred | fungi: I saw pictures in which there was too much water in inappropriate places in your house :( | 16:47 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Add status page to the quick-start https://review.openstack.org/610660 | 16:48 |
| *** dpawlik has joined #openstack-infra | 16:48 | |
| clarkb | cmurphy: on https://review.openstack.org/#/c/576262/2/manifests/lb.pp do you want to update the regex match as suggested by fungi? that will make it more reliable I think | 16:49 |
| *** dpawlik has quit IRC | 16:50 | |
| *** dpawlik has joined #openstack-infra | 16:50 | |
| *** ssbarnea has quit IRC | 16:51 | |
| clarkb | I'm setting up ansible vars/hieradata for etherpad-dev group now on bridge.o.o. Then will boot a new instance and stop the old one when new checks out | 16:58 |
| clarkb | looks like hieradata was already set up | 16:59 |
| openstackgerrit | Merged openstack-infra/system-config master: Turn on the future parser for static.o.o https://review.openstack.org/601840 | 17:00 |
| clarkb | I'll also watch ^ and friends | 17:00 |
| openstackgerrit | Merged openstack-infra/system-config master: Turn on the future parser for status.o.o https://review.openstack.org/601843 | 17:02 |
| *** lpetrut has joined #openstack-infra | 17:04 | |
| clarkb | I've just realized that a flaw in the plan to boot this as etherpad-dev01.opendev.org and let dns for etherpad-dev.openstack.org CNAME to that is that our manifests/site.pp says etherpad-dev\d*.openstack.org | 17:05 |
| clarkb | infra-root ^ I think I'm inclined to not worry about opendev.org until we can start to refactor site.pp but before I laucnh the new instance thoughtI'd open that to comment | 17:05 |
| *** ssbarnea has joined #openstack-infra | 17:06 | |
| openstackgerrit | Merged openstack-infra/system-config master: Turn on the future parser for survey.o.o https://review.openstack.org/601848 | 17:07 |
| clarkb | I figure there is enough value in getting off of trusty and finding any issues for production server that making forward progress at this point with the existing name is the way to go | 17:07 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Add Zuul to zuul-quick-start https://review.openstack.org/610664 | 17:08 |
| openstackgerrit | Merged openstack-infra/puppet-gerritbot master: Add beaker tests https://review.openstack.org/581691 | 17:09 |
| openstackgerrit | James E. Blair proposed openstack-infra/nodepool master: Run zuul-quick-start job https://review.openstack.org/610159 | 17:09 |
| *** dpawlik has quit IRC | 17:18 | |
| *** dpawlik_ has joined #openstack-infra | 17:18 | |
| *** Emine has joined #openstack-infra | 17:19 | |
| *** ssbarnea_ has joined #openstack-infra | 17:19 | |
| *** yamamoto has joined #openstack-infra | 17:20 | |
| clarkb | mordred: openstack inventory generation is failing with 'openstack plugin: public endpoint for compute service in Lon1 region not found' | 17:21 |
| *** studarus has joined #openstack-infra | 17:22 | |
| clarkb | ansible + puppet are currently not running. I take it that the ansible openstack modules need public endpoints because non public endpoints shouldn't be talked to? | 17:22 |
| * clarkb checks what keystone in that cloud says | 17:22 | |
| *** ssbarnea has quit IRC | 17:23 | |
| *** gfidente is now known as gfidente|afk | 17:25 | |
| clarkb | ok there is no Lon1 in citycloud anymore | 17:25 |
| clarkb | at least according to catalog list | 17:25 |
| *** anteaya has quit IRC | 17:27 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config master: Cleanup Citycloud Lon1 https://review.openstack.org/610668 | 17:28 |
| clarkb | infra-root I think ^ is necessary to get ansible and puppet running again. I may also need to manually edit the clouds.yaml on bridge.o.o to get ansible to run far enough to update the rest of the items | 17:29 |
| clarkb | I'll do that once ^ is merged | 17:29 |
| corvus | clarkb: thx | 17:30 |
| clarkb | I'll watch that the future parser changes go in quietly at that point then pick up the etherpad-dev xenial server update again | 17:31 |
| clarkb | fwiw this seemed to start around 1000UTC today | 17:31 |
| clarkb | based on syslog on static.o.o | 17:31 |
| *** ssbarnea_ has quit IRC | 17:31 | |
| *** kjackal has quit IRC | 17:33 | |
| *** kjackal has joined #openstack-infra | 17:34 | |
| *** rkukura has joined #openstack-infra | 17:35 | |
| *** panda is now known as panda|off | 17:37 | |
| clarkb | if I can get another review on ^ I may direct enqueue to the gate too | 17:37 |
| *** dtantsur is now known as dtantsur|afk | 17:38 | |
| boden | hi, I'm trying to run another projects UTs in a neutron job for cross project testing (https://review.openstack.org/#/c/605861/8/.zuul.yaml) using a similar approach as to what the requirements project does. however I'm getting a ton of UT errors that seem odd. I can't seem to put my finger on the issue. Are there any gotchas I maybe missing? | 17:40 |
| *** trown|lunch is now known as trown | 17:41 | |
| smcginnis | clarkb: I've noticed at least one merge that didn't show up in the cinder channel, so looks like the restart of GerritBot didn't take care of the issue. | 17:42 |
| clarkb | smcginnis: ya mlavalle reported similar | 17:42 |
| clarkb | boden: can you link to example weird behavior? | 17:45 |
| clarkb | smcginnis: fixing our puppetry is top of my list as is making progress on some server upgrades (simply beacuse they have been sitting idle and ignored otherwise), but I think the next step for gerritbot is to add more debugging of the actual irc comms if possible | 17:46 |
| boden | http://logs.openstack.org/61/605861/8/check/neutron-cross-vmware-nsx-py35/6ed9003/job-output.txt.gz#_2018-10-12_19_34_21_374190 I just can't explain these failures ATM | 17:46 |
| boden | clarkb ^ | 17:46 |
| *** jamesmcarthur has joined #openstack-infra | 17:48 | |
| *** roman_g has quit IRC | 17:50 | |
| *** felipemonteiro has quit IRC | 17:51 | |
| studarus | fyi: PacketHost limit on Grafana has gone to 9. Build and running pools are zero. Nodepool.yaml still has it at max-servers 100 though. | 17:52 |
| openstackgerrit | James E. Blair proposed openstack-infra/puppet-gerritbot master: Switch logging to debug https://review.openstack.org/610675 | 17:53 |
| corvus | clarkb: ^ | 17:53 |
| openstackgerrit | Chris Dent proposed openstack-infra/project-config master: Add gabbi-tempest unofficial project https://review.openstack.org/610628 | 17:53 |
| *** jamesmcarthur has quit IRC | 17:53 | |
| pabelanger | studarus: I think clarkb disabled it over the weekend, because mirror keeps dying | 17:53 |
| clarkb | boden: I think you've inverted the test setup? you want to run neutron tox with vmware nsx installed? not vmware nsx installing neutron and running neutron tests through neutron vmwarensx? In any case it is really hard to understand whati s going on there beacuse the log file is almost 100MB large | 17:53 |
| clarkb | pabelanger: studarus I pushed teh change not sure if it merged, but ya the mirror host keeps getting shutdown which causes jobs to fail | 17:54 |
| pabelanger | studarus: https://review.openstack.org/610187/ | 17:54 |
| *** diablo_rojo has joined #openstack-infra | 17:54 | |
| clarkb | boden: my hunch is that the neutron test suite isn't build to do test discovery nested in that manner | 17:55 |
| boden | clarkb: I know the log is huge.. I want to run vmware-nsx tox (py35) in the neutron gate using the neutron patch the job is running for... the idea is to detect if the neutron change will break vmware-nsx | 17:55 |
| studarus | k - I'll track down the compute host | 17:55 |
| clarkb | boden: because things are going to get mixed together with the nsx unittests that may cause weird errors? | 17:55 |
| clarkb | boden: then why is the neutron base test part of the traceback? File "/home/zuul/src/git.openstack.org/openstack/vmware-nsx/.tox/py35/lib/python3.5/site-packages/neutron/tests/base.py", line 140, in func' | 17:56 |
| clarkb | boden: if you just want the nsx unittests to run shouldn't that be independent of neutron? | 17:56 |
| *** apetrich has quit IRC | 17:56 | |
| clarkb | er of neutrons unittests | 17:56 |
| *** jamesmcarthur has joined #openstack-infra | 17:57 | |
| boden | clarkb base.py is probably there b/c a vmware-nsx test class extends base.py.... vmware-nsx UTs do run in the vmware-nsx gate; this is trying to cross test from neutron to ensure nothing in the neutron patch breaks vmware-nsx UTs | 17:57 |
| boden | clarkb: its a similar approach as in https://github.com/openstack/requirements/blob/master/.zuul.d/cross-jobs.yaml | 17:58 |
| clarkb | boden: ok, you probably want to capture the alembic logs for that test to see why the migration is out of sync with the schema | 17:58 |
| *** ssbarnea has joined #openstack-infra | 17:59 | |
| clarkb | I'm going to guess it is some unexpected interaction between the two test suite setup routines? | 17:59 |
| clarkb | pabelanger: mordred care to review https://review.openstack.org/#/c/610668/ to get ansible and puppet running again? | 18:00 |
| boden | clarkb: I'm not sure at this point... Based on the logs (I refd a min ago) it seems like neutron is getting installed into the vmware-nsx tox env.. I see 2018-10-12 19:41:07.367295 | ubuntu-xenial | b' File "/home/zuul/src/git.openstack.org/openstack/vmware-nsx/.tox/py35/lib/python3.5/site-packages/neutron/tests/unit/extensions/test_l3.py", line 3808, in setUp' and notice the neutron path is nested in the t | 18:01 |
| boden | ox env for vmware-nsx | 18:01 |
| clarkb | yup | 18:01 |
| clarkb | I wonder if the parenting of the base class for testing might be confusing the sql setup for the unittests? | 18:01 |
| boden | there's more than that SQL failure... there's also some globals not being initialized | 18:02 |
| clarkb | boden: do you get similar behavior locally? if you install tip of neutron master into the tox env for vmware nsx and run tox -epy35? | 18:03 |
| boden | clarkb: nope | 18:03 |
| boden | all green | 18:03 |
| clarkb | for the db stuff in particular those don't run unless you have a local database configured to test that | 18:03 |
| clarkb | do you have that openstack_citest db or whatever it is called setup ? | 18:03 |
| clarkb | if not you probably aren't running those particular tests | 18:04 |
| openstackgerrit | Merged openstack-dev/pbr master: tox: Suppress output https://review.openstack.org/607895 | 18:04 |
| openstackgerrit | Merged openstack-dev/pbr master: Skip test for testr hook being installed when testr is not available https://review.openstack.org/602761 | 18:04 |
| *** dmellado has quit IRC | 18:05 | |
| *** dmellado has joined #openstack-infra | 18:05 | |
| *** dpawlik_ has quit IRC | 18:06 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Don't run legacy-tempest-dsvm-neutron-full on pike https://review.openstack.org/610688 | 18:08 |
| *** anteaya has joined #openstack-infra | 18:11 | |
| openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-cgit master: Filter out link-local ipv6 address in loadbalancer https://review.openstack.org/576262 | 18:11 |
| cmurphy | clarkb: updated but commented ^ i won't be around much this week | 18:11 |
| *** electrofelix has quit IRC | 18:13 | |
| *** ssbarnea|bkp2 has joined #openstack-infra | 18:13 | |
| *** rkukura has quit IRC | 18:14 | |
| *** apetrich has joined #openstack-infra | 18:14 | |
| kmalloc | FYI, gerrit is spinning hard in "working" when loading patchsets atm | 18:17 |
| *** ansmith has quit IRC | 18:18 | |
| mordred | kmalloc: that's becauseit's ahard worker | 18:19 |
| kmalloc | mordred: hahahahahahahahha | 18:19 |
| kmalloc | mordred: ^_^ | 18:19 |
| *** dpawlik has joined #openstack-infra | 18:21 | |
| openstackgerrit | Dan Prince proposed openstack-infra/project-config master: Import puppet-crane for TripleO https://review.openstack.org/609531 | 18:22 |
| *** dpawlik has quit IRC | 18:26 | |
| *** bobh has quit IRC | 18:27 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: SQL: only create tables in scheduler https://review.openstack.org/610696 | 18:30 |
| clarkb | I have direct enqueued https://review.openstack.org/#/c/610668/ to the gate | 18:31 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: WIP: add mysql to quick-start https://review.openstack.org/610697 | 18:31 |
| *** stevebaker has quit IRC | 18:32 | |
| *** stevebaker has joined #openstack-infra | 18:41 | |
| *** lpetrut has quit IRC | 18:43 | |
| *** ssbarnea has quit IRC | 18:50 | |
| clarkb | mordred: maybe you can review https://review.openstack.org/#/c/610675/1 too and I can enqueue that/approve it once puppet/ansible are running again | 18:53 |
| mordred | clarkb: ++ | 18:54 |
| clarkb | kmalloc: when opening diffs? | 18:58 |
| clarkb | kmalloc: if this is for the large keystone rewrite the cost of those diffs is proportional to their size when it comes to rendering | 18:58 |
| openstackgerrit | Merged openstack-infra/system-config master: Cleanup Citycloud Lon1 https://review.openstack.org/610668 | 19:00 |
| *** mdrabe has joined #openstack-infra | 19:05 | |
| *** bobh has joined #openstack-infra | 19:05 | |
| *** bobh has quit IRC | 19:05 | |
| clarkb | I've manually edit /etc/openstack/clouds.yaml to remove Lon1 the 19:15 UTC run should work then put the rest of the updates from ^ in place | 19:06 |
| *** sshnaidm|afk has quit IRC | 19:06 | |
| *** bobh has joined #openstack-infra | 19:06 | |
| *** bobh has quit IRC | 19:11 | |
| openstackgerrit | Matthew Thode proposed openstack/diskimage-builder master: enable caching for gentoo builds https://review.openstack.org/604268 | 19:13 |
| clarkb | ansible is running. I'm going to make sure that etherpad-dev01 gets configured properly and that the futureparser chagnes look sane on static.o.o | 19:19 |
| *** hasharAway has quit IRC | 19:31 | |
| *** mriedem has quit IRC | 19:32 | |
| *** kjackal has quit IRC | 19:34 | |
| *** kjackal has joined #openstack-infra | 19:34 | |
| *** mriedem has joined #openstack-infra | 19:35 | |
| clarkb | puppet is running on static.o.o now | 19:47 |
| clarkb | run completed and appears to be a noop \o/ | 19:47 |
| clarkb | puppet is also configuring etherpad-dev01 now so Ih ave stopped apache and etherpad services on old etherpad.o.o | 19:48 |
| clarkb | oh and there is a bug | 19:49 |
| openstackgerrit | Clark Boylan proposed openstack-infra/puppet-etherpad_lite master: Fix env for plugin installs https://review.openstack.org/610717 | 20:01 |
| clarkb | ok ^ should fix things | 20:01 |
| openstackgerrit | Merged openstack-infra/zuul master: Add status page to the quick-start https://review.openstack.org/610660 | 20:07 |
| *** bobh has joined #openstack-infra | 20:08 | |
| ianw | clarkb: I see the last port cleanup run in gra1 removed 600(!) ports. looking at scrollback but did you get any further details on this? | 20:11 |
| clarkb | ianw: I haven't sorry | 20:12 |
| clarkb | slaweq basically made it sound like it was a known issue on the ovh side and that they were working it | 20:12 |
| clarkb | so I didn't want to push too hard and be annoying | 20:12 |
| clarkb | ianw: we could set the max-servers value to something small like 5 to mimic what we did in bhs1 to prove it works at all under minimal load | 20:13 |
| *** jamesmcarthur has quit IRC | 20:13 | |
| ianw | ok cool -- yeah the failure is seemingly different in bhs1 v gra1 ... in bhs1 we clean up a reasonable number of ports that seems to correlate roughly to a slow leak. i don't know where these huge numbers come from in gra1 | 20:14 |
| clarkb | ianw: when bhs1 was unhappy I put the nl0x for ovh into the emergency file then edited max-servers by hand to try and find the happy number | 20:15 |
| clarkb | I can do that again for gra1 if you don't want to | 20:15 |
| clarkb | ianw: https://review.openstack.org/#/c/610717/ fixes some etherpad puppetry that I Just ran into trying to ugprade etherpad-dev | 20:16 |
| ianw | clarkb: yeah, but http://grafana.openstack.org/d/BhcSH5Iiz/nodepool-ovh?orgId=1 doesn't really suggest there's crazy amounts of errors launching, or that it's near capacity? that's why it's so weird there's all the these ports | 20:17 |
| ianw | but i agree, if people on the other side (with access to stuff like logs :) are working on it, we can just monitor for now | 20:17 |
| clarkb | ianw: I think its saying the vast majority of the boots are failing | 20:17 |
| clarkb | and those failures likely lead to the leaks | 20:17 |
| clarkb | the available/in use numbers are basically all 0 :( | 20:18 |
| ianw | hrm, "error node launch attempts" maybe doesn't capture it? | 20:19 |
| clarkb | there are a lot of errors in that graph for me, but it doesn't distinguish between region | 20:19 |
| clarkb | ianw: not sure if you saw but I did followup on the linaro + rax mirror host thread | 20:20 |
| ianw | clarkb: yep, thanks, i think we need to debug the specific job a bit further on that. i think, from memory, it's looking like you get a tcp connection, but then things get reset | 20:25 |
| ianw | again iirc when looking at mirror logs maybe even saw apache recognise the connection, but then it disappeared | 20:25 |
| clarkb | that seems similar to what we saw with the http server on the nodepool builder too | 20:26 |
| ianw | hrm, you're right. I've hard-refreshed that grafana page and it now looks very different to what i just had up | 20:26 |
| *** jmart916 has joined #openstack-infra | 20:27 | |
| *** kgiusti has quit IRC | 20:28 | |
| clarkb | ianw: I think what we should do is set max servers to 0, see if port backlog goes away | 20:32 |
| clarkb | if it ends up at 0 ports then increase max servers to 5-10 and see if the cloud can handle that | 20:32 |
| ianw | clarkb: agree, just need to do some running around but can fiddle with this in about an hour today | 20:33 |
| clarkb | ianw: cool. I'm going to work on paging in the gerritbot stuff as the logging verbosity change should go into the gate shortly | 20:33 |
| openstackgerrit | Merged openstack-infra/zuul master: Add Zuul to zuul-quick-start https://review.openstack.org/610664 | 20:35 |
| *** smarcet has joined #openstack-infra | 20:36 | |
| kmalloc | clarkb: it was just clicking on the patchset | 20:42 |
| kmalloc | not a diff | 20:42 |
| clarkb | kmalloc: changing the patchset selector in the top right? | 20:43 |
| *** bobh has quit IRC | 20:43 | |
| kmalloc | clarkb: just selecting a review from the chain on the top right. | 20:44 |
| kmalloc | e.g. going from https://review.openstack.org/#/c/609500/4 to https://review.openstack.org/#/c/609535/3 | 20:44 |
| kmalloc | the related / submitted together patches | 20:44 |
| kmalloc | it seems to have resolved itself. | 20:45 |
| kmalloc | but it was 1-2 minutes to load those at one point | 20:45 |
| openstackgerrit | Clark Boylan proposed openstack-infra/project-config master: Reduce gate window floor to 10 https://review.openstack.org/610724 | 20:45 |
| clarkb | pabelanger: ^ I think I've given up on people fixing their tests in the near future. I know you suggested that change in the past. Curious what you think | 20:46 |
| *** eharney has quit IRC | 20:48 | |
| pabelanger | clarkb: +2 left comment. Agree not long term fix, but should help reduce some of the trashing | 20:54 |
| clarkb | mordred: pabelanger would you be willing to review this fix for etherpad puppetry? https://review.openstack.org/#/c/610717/1 I can approve it if the tests end up going green later | 21:00 |
| clarkb | I'm going to dig into gerritbot as soon as the debug change merges | 21:00 |
| *** fuentess has quit IRC | 21:01 | |
| pabelanger | +3 | 21:02 |
| *** trown is now known as trown|outtypewww | 21:05 | |
| *** jamesmcarthur has joined #openstack-infra | 21:16 | |
| *** ldnunes has quit IRC | 21:17 | |
| *** rh-jelabarre has quit IRC | 21:20 | |
| *** jmart916 has left #openstack-infra | 21:20 | |
| openstackgerrit | Merged openstack-infra/puppet-gerritbot master: Switch logging to debug https://review.openstack.org/610675 | 21:25 |
| clarkb | re the polygerrit attention that seems to be happening https://bugs.chromium.org/p/gerrit/issues/detail?id=5012#c2 got a bunch of recent attention. I wonder if people are looking at it now | 21:26 |
| *** ansmith has joined #openstack-infra | 21:37 | |
| *** jamesmcarthur has quit IRC | 21:42 | |
| smcginnis | Is the Tryjobs section how our CI results would be displayed? https://chromium-review.googlesource.com/c/chromium/src/+/636113 | 21:42 |
| *** Emine has quit IRC | 21:43 | |
| clarkb | smcginnis: probably not on day one, but we'd switch to using the bot reporting interface eventually | 21:44 |
| clarkb | we have to update zuul to use the new reporting mechanism which shouldn't be hard btu I don't think we want to couple too many things to the upgrade then it become even more difficult to do | 21:45 |
| smcginnis | We might want a custom CSS to display though. At least on Cinder. All our third party CI's would be painful. | 21:45 |
| smcginnis | I do like it's below the files. | 21:46 |
| smcginnis | Bah, but still no 'f' key shortcut. Sadness. | 21:46 |
| clarkb | smcginnis: shift + i is the new thing | 21:51 |
| clarkb | then jk through the list | 21:51 |
| clarkb | not quite the same but fills a similar need | 21:51 |
| smcginnis | Looks like they don't have that yet. But glad to have the functionality coming back. I still miss it from 2-3 upgrades ago. | 21:52 |
| clarkb | ok going to fiddle with gerritbot now | 21:55 |
| *** openstackgerrit has quit IRC | 21:56 | |
| *** openstackgerrit has joined #openstack-infra | 21:57 | |
| *** openstackgerrit has quit IRC | 21:58 | |
| *** mlavalle has left #openstack-infra | 21:59 | |
| *** openstackgerrit has joined #openstack-infra | 21:59 | |
| *** jtomasek has quit IRC | 22:00 | |
| clarkb | I had to tweak the hanlders list for the root DEBUG too | 22:02 |
| clarkb | but doesn't seem like irc lib is logging much on its own. Did get more paramiko logging though | 22:03 |
| *** agopi has quit IRC | 22:04 | |
| *** slaweq has quit IRC | 22:05 | |
| pabelanger | https://www.openstack.org/summit/berlin-2018/summit-schedule/events/22794/leveraging-airship-as-an-integration-and-qa-enabler is an interesting forum topic, from reading the description I am unsure if airship want to run CI/CD deployments in the gate, or have openstack projects test atop of airship | 22:10 |
| pabelanger | I am assuming it is the first | 22:10 |
| *** eernst has quit IRC | 22:10 | |
| *** slaweq has joined #openstack-infra | 22:11 | |
| *** boden has quit IRC | 22:12 | |
| *** boden has joined #openstack-infra | 22:12 | |
| *** boden has quit IRC | 22:12 | |
| *** boden has joined #openstack-infra | 22:12 | |
| *** openstackgerrit has quit IRC | 22:12 | |
| *** boden has quit IRC | 22:13 | |
| *** boden has joined #openstack-infra | 22:13 | |
| *** openstackgerrit has joined #openstack-infra | 22:13 | |
| clarkb | hrm I had to explicitly add an irc logger to get irc logging messages | 22:13 |
| clarkb | anyways have a lot more logs to look at now and hopefully that gives us a hint | 22:13 |
| *** boden has quit IRC | 22:14 | |
| *** harlowja has quit IRC | 22:14 | |
| *** boden has joined #openstack-infra | 22:14 | |
| *** boden has quit IRC | 22:14 | |
| *** boden has joined #openstack-infra | 22:15 | |
| *** boden has quit IRC | 22:15 | |
| clarkb | success: 2018-10-15 22:15:23,571 DEBUG irc.client: _dispatcher: toomanychannels | 22:15 |
| *** slaweq has quit IRC | 22:15 | |
| *** boden has joined #openstack-infra | 22:15 | |
| clarkb | the success is from me. Did freenode change the channel limit? | 22:15 |
| *** boden has quit IRC | 22:16 | |
| *** rpioso is now known as rpioso|afk | 22:16 | |
| *** boden has joined #openstack-infra | 22:16 | |
| mordred | clarkb: good question | 22:16 |
| *** boden has quit IRC | 22:17 | |
| *** boden has joined #openstack-infra | 22:17 | |
| clarkb | my personal chanlimit is 120 according to /quote version | 22:17 |
| *** gfidente|afk has quit IRC | 22:17 | |
| *** boden has quit IRC | 22:17 | |
| *** boden has joined #openstack-infra | 22:18 | |
| *** boden has quit IRC | 22:18 | |
| *** harlowja has joined #openstack-infra | 22:18 | |
| clarkb | there are 144 attempted channel joins | 22:18 |
| *** boden has joined #openstack-infra | 22:19 | |
| clarkb | now to sort thorugh how many worked before it failed | 22:19 |
| *** boden has quit IRC | 22:19 | |
| *** boden has joined #openstack-infra | 22:19 | |
| *** boden has quit IRC | 22:20 | |
| *** boden has joined #openstack-infra | 22:20 | |
| *** boden has joined #openstack-infra | 22:21 | |
| *** boden has quit IRC | 22:21 | |
| *** boden has joined #openstack-infra | 22:22 | |
| clarkb | ok grepping the logs we don't seem to part any channels | 22:22 |
| *** boden has quit IRC | 22:22 | |
| *** boden has joined #openstack-infra | 22:22 | |
| *** boden has quit IRC | 22:23 | |
| *** boden has joined #openstack-infra | 22:23 | |
| *** boden has quit IRC | 22:24 | |
| *** boden has joined #openstack-infra | 22:24 | |
| *** boden has quit IRC | 22:24 | |
| *** boden has joined #openstack-infra | 22:25 | |
| *** boden has quit IRC | 22:25 | |
| *** boden has joined #openstack-infra | 22:26 | |
| *** boden has quit IRC | 22:26 | |
| *** boden has joined #openstack-infra | 22:26 | |
| fungi | could a recent gerritbot change have regressed that? | 22:27 |
| *** boden has quit IRC | 22:27 | |
| *** boden has joined #openstack-infra | 22:27 | |
| *** smarcet has quit IRC | 22:27 | |
| *** boden has quit IRC | 22:28 | |
| clarkb | possibly. It appears that we are trying to join all the channels even if there aren't events for them | 22:29 |
| clarkb | so we try to join all 140something channels and the channels at the end of the list lose out | 22:29 |
| *** agopi has joined #openstack-infra | 22:29 | |
| clarkb | reading the logs there isn't a sending event tied to channel join for all the channels | 22:29 |
| fungi | ohhh | 22:30 |
| fungi | i bet the sasl change did that | 22:30 |
| fungi | sorry, we can revert it, but will be in a similar boat since it still races nickserv identify and doesn't realize it fails to join protected channels while waiting for nickserv to confirm | 22:31 |
| clarkb | ya there is a definite #ara join with no gerritbot log for sendnig a message to ara | 22:32 |
| clarkb | fungi: ya we probably need to actually fix it since it won't work on a revert either | 22:32 |
| fungi | the fix to make it not autojoin all the channels is likely trivial though. basically pass an empty value for channels to the irc constructor? | 22:32 |
| fungi | and then let the other bot logic decide when to join | 22:33 |
| clarkb | ya do you know where that logic is? I've still not found it | 22:33 |
| dmsimard | Ohai my ara senses are tingling | 22:33 |
| clarkb | is it the channels=channels list? | 22:33 |
| dmsimard | Yeah, the issue has been continuing after clarkb's restart | 22:33 |
| clarkb | ya I think that may be the fix. Let me push up the change and if it looks good I can ninja that in and restart and see | 22:34 |
| dmsimard | Can we open a shell ? So we can see from it's perspective what channels it has joined ? | 22:34 |
| *** pbourke has quit IRC | 22:34 | |
| openstackgerrit | Matthew Thode proposed openstack/diskimage-builder master: enable caching for gentoo builds https://review.openstack.org/604268 | 22:34 |
| *** munimeha1 has quit IRC | 22:34 | |
| dmsimard | Or even a pdb breakpoint | 22:35 |
| *** pbourke has joined #openstack-infra | 22:35 | |
| clarkb | dmsimard: we can, but I don't think we need to. WIth logging we have the info we need | 22:35 |
| dmsimard | I'm just curious on how many channels it has really joined | 22:35 |
| fungi | clarkb: https://git.openstack.org/cgit/openstack-infra/gerritbot/tree/gerritbot/bot.py#n107 i think? | 22:35 |
| dmsimard | I thought that the way it worked is that gerribot would only join channels on a need basis | 22:36 |
| dmsimard | Do we really have >100 channels with active development ? | 22:36 |
| clarkb | dmsimard: thats the intent but its joining all the channels in its list and failing | 22:36 |
| clarkb | its buggy | 22:36 |
| dmsimard | Ah :( | 22:36 |
| fungi | dmsimard: i regressed it when patching it to switch to sasl auth | 22:36 |
| clarkb | fungi: ya but reading https://github.com/jaraco/irc/blob/master/irc/bot.py#L116 channels isn't an argument | 22:36 |
| clarkb | but maybe the sasl/ssl stuff is being "helpful" | 22:36 |
| dmsimard | I can spend some time on this tonight, the notifications are very useful to me :) | 22:37 |
| clarkb | dmsimard: well I think I'm about to fix it | 22:37 |
| dmsimard | Oh, neat.. I'm not at a real keyboard yet | 22:37 |
| fungi | i'm happy to review. done with cleaning for the night i think | 22:37 |
| openstackgerrit | Clark Boylan proposed openstack-infra/gerritbot master: Don't autojoin channels https://review.openstack.org/610745 | 22:39 |
| clarkb | fungi: ^ I'm thinking that may be the fix. if that looks about right I can ninja edit the installation locally on review.o.o | 22:39 |
| clarkb | we definitely weren't passing channels through before | 22:40 |
| fungi | yeah, i would try that on review.o.o but looks like what i was thinking | 22:40 |
| clarkb | ok here goes | 22:41 |
| *** rcernin has joined #openstack-infra | 22:41 | |
| fungi | when i said "likely trivial" i wasn't kidding ;) | 22:41 |
| dmsimard | We can just submit a new patchset to test it I guess | 22:41 |
| *** kjackal has quit IRC | 22:43 | |
| *** efried has quit IRC | 22:43 | |
| *** openstackgerrit has quit IRC | 22:43 | |
| *** kjackal_v2 has joined #openstack-infra | 22:43 | |
| clarkb | I've restarted, you can push changes to test. I've asked neutron specifically to test since they found that their channel didn't work | 22:45 |
| *** jesusaur has quit IRC | 22:45 | |
| clarkb | ara would be a good one too | 22:45 |
| clarkb | ok that fixed #openstack-neutron | 22:45 |
| clarkb | the next puppet run should undo what I've done for the gerritbot install | 22:45 |
| clarkb | please review :) | 22:45 |
| clarkb | infra-root https://review.openstack.org/610745 is specifically the change to review to fix gerritbot | 22:48 |
| *** kjackal_v2 has quit IRC | 22:50 | |
| clarkb | I'll get a revert for the not quite working logging change uptoo | 22:50 |
| *** openstackgerrit has joined #openstack-infra | 22:52 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/puppet-gerritbot master: Revert "Switch logging to debug" https://review.openstack.org/610750 | 22:52 |
| clarkb | there | 22:52 |
| *** jesusaur has joined #openstack-infra | 22:52 | |
| *** smarcet has joined #openstack-infra | 22:56 | |
| openstackgerrit | Merged openstack-infra/puppet-etherpad_lite master: Fix env for plugin installs https://review.openstack.org/610717 | 23:00 |
| *** mriedem has quit IRC | 23:00 | |
| clarkb | if ^ fixes etherpad-dev I'll update dns after that | 23:01 |
| *** eernst has joined #openstack-infra | 23:01 | |
| clarkb | then I can do production server tomorrow hopefully and two more servers come off the needs upgrade list | 23:02 |
| *** dklyle has quit IRC | 23:04 | |
| clarkb | infra-root thoughts on https://review.openstack.org/#/c/610724/ which reduces the gate window floor size much appreciated | 23:07 |
| mordred | clarkb: oh - I +A'd before I saw you say thoughts ... +A removed but the +2 remains - land at will | 23:21 |
| clarkb | mordred: ya I feel like that change might need broader input? I dunno its not like things are merging quickly 10 at a time anyway :P | 23:22 |
| openstackgerrit | Clark Boylan proposed openstack-infra/zuul master: Provide some accounting of node usage in logs https://review.openstack.org/605856 | 23:24 |
| *** tosky has quit IRC | 23:25 | |
| *** dklyle has joined #openstack-infra | 23:30 | |
| *** jamesmcarthur has joined #openstack-infra | 23:31 | |
| clarkb | hrm the last run of puppet didn't update the puppet module. I guess its got to merge before the run starts? | 23:33 |
| clarkb | I'm going to try kicking the node | 23:34 |
| mordred | clarkb: yeah - there's a weird double-cycle that needs to happen sometimes that I think we should design better | 23:35 |
| mordred | clarkb: I've been meaning to come back to it | 23:35 |
| *** jamesmcarthur has quit IRC | 23:35 | |
| clarkb | mordred: I'm not sure if kick.sh is going to update the modules either but I figure trying and understanding how this works is better than nothing | 23:37 |
| clarkb | ya that doesn't seem to update the modules | 23:38 |
| clarkb | I think the puppet modules will update at 23:45 and then I can rerun kick.sh to avoid waiting half an hour | 23:39 |
| mordred | clarkb: update-system-config.yaml is the one that updates the modules | 23:45 |
| clarkb | ya the first one that runs | 23:45 |
| clarkb | I see the task running now in the log | 23:45 |
| clarkb | I'll just kick.sh again when that is done | 23:45 |
| *** eernst has quit IRC | 23:50 | |
| *** agopi has quit IRC | 23:51 | |
| *** agopi has joined #openstack-infra | 23:51 | |
| clarkb | ok that almost completely gets etherpad light working. The current issue is we are trying to set custom pad.js file content in the git tree which doesn't exist as a destination so puppet files to write the file | 23:55 |
| *** gyee has quit IRC | 23:55 | |
| clarkb | this implies to me that this is no longer a supported method for etherpad to customize things | 23:55 |
| clarkb | except it seems to work with etherpad.o.o, its the code that pins the chat window open on start | 23:56 |
| *** eernst has joined #openstack-infra | 23:56 | |
| *** carl_cai has joined #openstack-infra | 23:57 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!