| *** slaweq has quit IRC | 00:00 | |
| fungi | mnaser: yup, thanks | 00:00 |
|---|---|---|
| *** Apoorva_ has joined #openstack-infra | 00:00 | |
| *** hongbin has quit IRC | 00:01 | |
| *** jamesmcarthur has joined #openstack-infra | 00:02 | |
| *** Apoorva has quit IRC | 00:03 | |
| *** Apoorva_ has quit IRC | 00:04 | |
| *** slaweq has joined #openstack-infra | 00:05 | |
| *** xarses has joined #openstack-infra | 00:07 | |
| *** jamesmcarthur has quit IRC | 00:07 | |
| clarkb | bah getting the python version right with pip and doing pyyaml installs is more painful across all the distros than I was hoping | 00:12 |
| fungi | down to 27 executor processes on ze01 | 00:12 |
| clarkb | fungi: make sure they aren't all just ssh agents | 00:12 |
| clarkb | you can kill thsose | 00:12 |
| clarkb | (once zuul isn't running anymore) | 00:13 |
| fungi | oh, cool checking. thanks | 00:13 |
| pabelanger | fungi: I wonder if they are no longer aborting, and just finishing / timing out per usually | 00:13 |
| *** salv-orlando has joined #openstack-infra | 00:13 | |
| *** bobh has joined #openstack-infra | 00:13 | |
| fungi | but yeah, the main zuul-executor process (and a fork i think) are still running | 00:13 |
| fungi | i'll kill the ssh sessions and see if everything wraps up | 00:14 |
| clarkb | ya there are still bwrap'd jobs going | 00:14 |
| fungi | yeah, 20 still with ssh in the process string | 00:14 |
| fungi | it's safe to just kill those? or not until the parent terminates? | 00:15 |
| clarkb | I've waited on those | 00:15 |
| fungi | k | 00:15 |
| clarkb | but if executor sticks around after everything else but ssh-agent is gone I've killed the executors | 00:15 |
| fungi | noted | 00:16 |
| *** salv-orlando has quit IRC | 00:18 | |
| clarkb | bah its like whackamole | 00:18 |
| clarkb | fix one thing with pip 10 and another thing falls out | 00:18 |
| clarkb | I'm calling it, I'll be around to answer ze questions but closing laptop | 00:18 |
| fungi | thanks! i've probably got it from this point anywa | 00:19 |
| fungi | y | 00:19 |
| *** xarses has quit IRC | 00:20 | |
| *** mriedem1 has joined #openstack-infra | 00:21 | |
| *** mriedem has quit IRC | 00:22 | |
| *** gyee has quit IRC | 00:36 | |
| *** slaweq has quit IRC | 00:38 | |
| *** markvoelker_ has quit IRC | 00:38 | |
| *** bobh has quit IRC | 00:38 | |
| *** mriedem1 is now known as mriedem | 00:40 | |
| *** andreas_s has joined #openstack-infra | 00:43 | |
| jeblair | clarkb, fungi: template test failure is a missing git add, sorry | 00:45 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Ignore duplicate project-template definitions https://review.openstack.org/513853 | 00:45 |
| *** andreas_s has quit IRC | 00:47 | |
| *** slaweq has joined #openstack-infra | 00:47 | |
| openstackgerrit | Mohammed Naser proposed openstack-infra/zuul feature/zuulv3: Refactor Zuul job to use lightweight Vue components https://review.openstack.org/513885 | 00:48 |
| openstackgerrit | Mohammed Naser proposed openstack-infra/zuul feature/zuulv3: Switch change list to use Vue.js https://review.openstack.org/513886 | 00:48 |
| fungi | still 4 bwrap processes on ze01 | 00:49 |
| fungi | guess i'll give them a little more time | 00:49 |
| *** thorst has joined #openstack-infra | 00:49 | |
| *** thorst has quit IRC | 00:51 | |
| *** gmann_afk is now known as gmann | 00:54 | |
| *** ijw has quit IRC | 01:01 | |
| *** dhajare has joined #openstack-infra | 01:05 | |
| *** armax has quit IRC | 01:11 | |
| *** edmondsw has joined #openstack-infra | 01:12 | |
| *** salv-orlando has joined #openstack-infra | 01:14 | |
| *** owalsh_ has joined #openstack-infra | 01:15 | |
| *** edmondsw has quit IRC | 01:16 | |
| *** salv-orlando has quit IRC | 01:19 | |
| *** owalsh has quit IRC | 01:19 | |
| *** slaweq has quit IRC | 01:19 | |
| fungi | they finally wrapped up | 01:22 |
| fungi | i had to kill the executor process and ssh processes once there were no bwrap processes left | 01:22 |
| *** markvoelker has joined #openstack-infra | 01:23 | |
| fungi | looks like it's starting to pick up work, so i'll move on to the rest | 01:23 |
| *** markvoelker has quit IRC | 01:27 | |
| *** bobh has joined #openstack-infra | 01:28 | |
| *** armax has joined #openstack-infra | 01:29 | |
| *** slaweq has joined #openstack-infra | 01:30 | |
| *** markvoelker has joined #openstack-infra | 01:32 | |
| *** bobh has quit IRC | 01:32 | |
| *** dhajare has quit IRC | 01:35 | |
| *** markvoelker has quit IRC | 01:36 | |
| *** ihrachys has quit IRC | 01:37 | |
| *** fanzhang_ has joined #openstack-infra | 01:40 | |
| *** markvoelker has joined #openstack-infra | 01:41 | |
| *** slaweq has quit IRC | 01:59 | |
| *** rosmaita has quit IRC | 02:08 | |
| *** slaweq has joined #openstack-infra | 02:08 | |
| *** sshnaidm|off has quit IRC | 02:10 | |
| *** dhinesh has quit IRC | 02:10 | |
| *** xarses has joined #openstack-infra | 02:11 | |
| *** markvoelker has quit IRC | 02:14 | |
| *** salv-orlando has joined #openstack-infra | 02:15 | |
| *** salv-orlando has quit IRC | 02:19 | |
| pabelanger | infra-root: we may also want to consider stopping nodepool processes on nodepool.o.o, we are still running since last sundays zuulv3 rollout. | 02:20 |
| pabelanger | mostly for log file clean up | 02:20 |
| *** xinliang has joined #openstack-infra | 02:24 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: DEBUG: downgrade iptables https://review.openstack.org/513873 | 02:31 |
| fungi | #status log all zuul executors restarted to pick up the /usr/share/ca-certificates addition | 02:33 |
| openstackstatus | fungi: finished logging | 02:33 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: DEBUG: downgrade iptables https://review.openstack.org/513873 | 02:33 |
| *** bstinson has quit IRC | 02:34 | |
| *** Kevin_Zheng has joined #openstack-infra | 02:35 | |
| *** slaweq has quit IRC | 02:38 | |
| *** bstinson has joined #openstack-infra | 02:40 | |
| *** slaweq has joined #openstack-infra | 02:44 | |
| *** thorst has joined #openstack-infra | 02:52 | |
| *** dave-mccowan has quit IRC | 02:54 | |
| *** thorst has quit IRC | 02:57 | |
| kazsh | Hi Infra team, could you kindly add myself `kaz_shinohara` to heat-dashboard-core team ? Need it to move forward our journey, https://review.openstack.org/#/c/504166/ | 02:58 |
| EmilienM | clarkb: weshay|ruck and I are still debugging the iptables thing | 02:59 |
| EmilienM | clarkb: we tried to reproduce it in our env and failed, we think it has something to do with the rules, maybe not saved | 03:00 |
| EmilienM | are nodes restarted before running jobs? | 03:00 |
| jeblair | EmilienM: no, it's the first boot | 03:00 |
| EmilienM | ok | 03:00 |
| EmilienM | thx | 03:00 |
| *** markvoelker has joined #openstack-infra | 03:05 | |
| openstackgerrit | wes hayutin proposed openstack-infra/tripleo-ci master: DEBUG: downgrade iptables https://review.openstack.org/513891 | 03:09 |
| *** markvoelker has quit IRC | 03:10 | |
| clarkb | EmilienM: my patch to zuul will help us in debugging it | 03:11 |
| clarkb | I should make sure tests passed | 03:11 |
| EmilienM | clarkb: which patch please? | 03:11 |
| EmilienM | weshay|ruck: I was wondering, could we reproduce the same problem in RDO CI, if we used the same nodepool image? | 03:12 |
| EmilienM | so we can see if it's something with zuul v3 | 03:12 |
| EmilienM | clarkb: I remember you told me that you don't publish the images anymore | 03:12 |
| weshay|ruck | EmilienM, ya.. let me see how those jobs are running | 03:12 |
| EmilienM | and that I have to use DIB myself | 03:12 |
| clarkb | we've never published the images | 03:13 |
| EmilienM | weshay|ruck: maybe we can just run a multinode job with latest centos7 (and even run yum -y update before anything) in RDO CI | 03:13 |
| EmilienM | clarkb: on nodepool.o.o ? | 03:13 |
| weshay|ruck | ya | 03:13 |
| clarkb | EmilienM: ehat you can do is run the overlay on the nodes I held for you | 03:13 |
| EmilienM | clarkb: ok | 03:13 |
| EmilienM | clarkb: but did zuul v3 prepared the nodes before? | 03:14 |
| *** markvoelker has joined #openstack-infra | 03:14 | |
| clarkb | EmilienM: I'm not sure how far it got before the job was aborted | 03:15 |
| weshay|ruck | EmilienM, rdo jobs are hitting the same issue | 03:15 |
| EmilienM | weshay|ruck: awesome, so we can more easily debug | 03:15 |
| EmilienM | clarkb: any change to revert to the previous image? | 03:15 |
| EmilienM | all our gate is stuck now | 03:15 |
| *** salv-orlando has joined #openstack-infra | 03:16 | |
| EmilienM | if we can debug in RDO CI with new image in the meantime | 03:16 |
| weshay|ruck | https://logs.rdoproject.org/openstack-periodic-4hr/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset005-master/3c6f838/console.txt.gz#_2017-10-20_20_12_03_238 | 03:16 |
| clarkb | EmilienM: are we sure it us the image? | 03:16 |
| EmilienM | weshay|ruck: perfect | 03:16 |
| EmilienM | clarkb: probably the same packages | 03:16 |
| clarkb | we only have yesterdays image | 03:16 |
| EmilienM | perfect | 03:16 |
| clarkb | so unsure if that goes back far enough | 03:16 |
| EmilienM | it broke this morning | 03:16 |
| weshay|ruck | they build the upstream image in rdo and use it.. same thing | 03:16 |
| clarkb | (I havent checked) | 03:17 |
| weshay|ruck | ya.. it broke today for sure | 03:17 |
| clarkb | weshay|ruck: that doesnt mean it is the image though | 03:17 |
| EmilienM | this morning, around 11am PST | 03:17 |
| clarkb | is my point | 03:17 |
| EmilienM | right | 03:17 |
| EmilienM | in our investigation, we saw iptables updated | 03:17 |
| EmilienM | and a new package: iptables-services | 03:17 |
| weshay|ruck | clarkb, right.. I can go find where they build the image, it's the upstream dib | 03:17 |
| EmilienM | so maybe we miss something here | 03:17 |
| EmilienM | weshay|ruck: RDO CI image uses iptables-1.4.21-18.2.el7_4.x86_64 as well | 03:18 |
| EmilienM | let's compare all rpms | 03:18 |
| weshay|ruck | iptables-services.x86_64 1.4.21-18.2.el7_4 @quickstart-centos-updates | 03:19 |
| EmilienM | https://www.diffchecker.com/8Dd11LlG | 03:19 |
| *** ianychoi_ has joined #openstack-infra | 03:19 | |
| EmilienM | look the diff ^ | 03:19 |
| EmilienM | I don't see rpm that could be break networking | 03:19 |
| EmilienM | so either it's something in zuul v3 playbooks or it's something in the image | 03:20 |
| *** salv-orlando has quit IRC | 03:20 | |
| clarkb | or something i tripleo :) | 03:20 |
| *** fanzhang_ has quit IRC | 03:20 | |
| EmilienM | clarkb: yeah | 03:21 |
| EmilienM | clarkb: but it broke this morning suddenly | 03:21 |
| EmilienM | clarkb: and tbh we didn't merge stuffs today I think | 03:21 |
| EmilienM | let me check that as well | 03:21 |
| *** ianychoi has quit IRC | 03:21 | |
| weshay|ruck | not much has merged | 03:22 |
| EmilienM | yeah | 03:22 |
| EmilienM | and fyi it also affect stable branches | 03:22 |
| EmilienM | and we didn't merge backports today | 03:22 |
| clarkb | keep in mind if it is new packages reverting the image wont help as we install those new packages | 03:25 |
| EmilienM | yum update | 03:26 |
| EmilienM | ok | 03:26 |
| EmilienM | clarkb: what's your patch in zuul you mentionned? | 03:26 |
| clarkb | 513850, the tests are failing | 03:27 |
| *** lamt has quit IRC | 03:27 | |
| clarkb | ok I've got to go family calls | 03:28 |
| jeblair | clarkb, EmilienM: it's the weekend -- you could probably just tell zuul to hold 20 nodes and see what it catches | 03:28 |
| EmilienM | clarkb: no worries, family first | 03:32 |
| EmilienM | jeblair: that would be great | 03:32 |
| EmilienM | I think it's iptables or systemd | 03:32 |
| EmilienM | both upgrades | 03:32 |
| EmilienM | upgraded* | 03:32 |
| EmilienM | I'm reading git diff now for both pkg upgrades | 03:32 |
| jeblair | EmilienM: what job and project? | 03:33 |
| EmilienM | jeblair: all tripleo multinode jobs | 03:33 |
| *** lamt has joined #openstack-infra | 03:33 | |
| EmilienM | jeblair: tripleo-heat-templates, tripleo-ci, every project | 03:33 |
| jeblair | looks like clarkb did ('openstack', 'git.openstack.org/openstack/tripleo-heat-templates', 'legacy-tripleo-ci-centos-7-nonha-multinode-oooq') before | 03:34 |
| openstackgerrit | wes hayutin proposed openstack-infra/tripleo-ci master: DEBUG: downgrade iptables https://review.openstack.org/513891 | 03:34 |
| EmilienM | legacy-tripleo-ci-centos-7-nonha-multinode-oooq for example | 03:34 |
| jeblair | i have to pick one -- it doesn't regex | 03:34 |
| jeblair | EmilienM: should i just do that again? | 03:34 |
| EmilienM | yeah good | 03:34 |
| *** lamt is now known as Guest90416 | 03:34 | |
| EmilienM | or wait | 03:34 |
| EmilienM | jeblair: openstack-infra/tripleo-ci | 03:34 |
| EmilienM | jeblair: it's better so we can make change in this repo to debug stuffs with our framework | 03:35 |
| EmilienM | thanks | 03:35 |
| jeblair | EmilienM, clarkb: http://paste.openstack.org/show/624239/ | 03:36 |
| EmilienM | perfect | 03:36 |
| EmilienM | jeblair: ok so I can just send patches now, and when it fail just SSH? | 03:37 |
| EmilienM | (my key is missing) | 03:37 |
| jeblair | so the next 20 runs of that job on that project should get held | 03:37 |
| jeblair | EmilienM: i guess an infra-root will have to add your ssh key | 03:38 |
| EmilienM | ok | 03:38 |
| EmilienM | this change in iptables worries me https://git.centos.org/blobdiff/rpms!iptables.git/4d0bb22ecdc68576e7a4a7afd966207c50b664d2/SPECS!iptables.spec | 03:44 |
| *** markvoelker has quit IRC | 03:48 | |
| *** nicolasbock has quit IRC | 03:59 | |
| *** jamesmcarthur has joined #openstack-infra | 04:03 | |
| *** mriedem has quit IRC | 04:04 | |
| openstackgerrit | wes hayutin proposed openstack-infra/tripleo-ci master: DEBUG: downgrade iptables https://review.openstack.org/513891 | 04:06 |
| *** jamesmcarthur has quit IRC | 04:07 | |
| SamYaple | EmilienM: how so? | 04:09 |
| EmilienM | SamYaple: we're debugging multinode centos7 failures | 04:09 |
| EmilienM | SamYaple: the latest centos7 image seems not working in the current state of things | 04:09 |
| EmilienM | subnode-2 isn't reachable from main node | 04:10 |
| *** salv-orlando has joined #openstack-infra | 04:17 | |
| *** slaweq has quit IRC | 04:17 | |
| *** salv-orlando has quit IRC | 04:21 | |
| SamYaple | oh i see. that wasnt just an acedemic concern. youre looking for the cause | 04:23 |
| SamYaple | thanks EmilienM | 04:23 |
| SamYaple | good info to know | 04:23 |
| EmilienM | SamYaple: yeah, tripleo jobs are all failing now | 04:25 |
| EmilienM | SamYaple: do you run multinode on centos7, | 04:25 |
| EmilienM | ? | 04:25 |
| EmilienM | in OSA | 04:25 |
| SamYaple | EmilienM: i dont know. i dont work on OSA | 04:26 |
| SamYaple | pretty sure kolla does though | 04:26 |
| SamYaple | but they are just setting up those gates | 04:26 |
| EmilienM | oh right, kolla | 04:26 |
| *** slaweq has joined #openstack-infra | 04:26 | |
| EmilienM | sorry, late | 04:26 |
| *** bhavik1 has joined #openstack-infra | 04:26 | |
| SamYaple | the confusion is i used to work at RAX ;) | 04:26 |
| EmilienM | SamYaple: oh where are you now? | 04:27 |
| SamYaple | 1099 for Verizon Digital Media Services | 04:27 |
| EmilienM | wouhou, nice ! | 04:28 |
| EmilienM | huge clouds as well :D | 04:28 |
| *** bhavik1 has quit IRC | 04:32 | |
| SamYaple | not the same Verizon, its a CDN formerly know as Edgecast | 04:34 |
| *** claudiub|2 has joined #openstack-infra | 04:38 | |
| *** markvoelker has joined #openstack-infra | 04:38 | |
| *** markvoelker has quit IRC | 04:45 | |
| *** Kevin_Zheng has quit IRC | 04:45 | |
| *** edmondsw has joined #openstack-infra | 04:48 | |
| openstackgerrit | wes hayutin proposed openstack-infra/tripleo-ci master: DNM, testing fix for vxlan ping failure https://review.openstack.org/513896 | 04:48 |
| *** bobh has joined #openstack-infra | 04:49 | |
| *** markvoelker has joined #openstack-infra | 04:50 | |
| *** markvoelker has quit IRC | 04:51 | |
| *** edmondsw has quit IRC | 04:52 | |
| *** thorst has joined #openstack-infra | 04:53 | |
| *** armax has quit IRC | 04:59 | |
| *** armax has joined #openstack-infra | 04:59 | |
| *** armax has quit IRC | 04:59 | |
| *** thorst has quit IRC | 05:00 | |
| *** armax has joined #openstack-infra | 05:00 | |
| *** armax has quit IRC | 05:00 | |
| *** armax has joined #openstack-infra | 05:01 | |
| *** armax has quit IRC | 05:01 | |
| *** armax has joined #openstack-infra | 05:02 | |
| *** armax has quit IRC | 05:02 | |
| *** armax has joined #openstack-infra | 05:02 | |
| *** armax has quit IRC | 05:03 | |
| *** florianf has quit IRC | 05:07 | |
| *** florianf has joined #openstack-infra | 05:08 | |
| *** bobh has quit IRC | 05:09 | |
| openstackgerrit | wes hayutin proposed openstack-infra/tripleo-ci master: DEBUG: downgrade iptables https://review.openstack.org/513891 | 05:16 |
| *** salv-orlando has joined #openstack-infra | 05:17 | |
| *** florianf has quit IRC | 05:18 | |
| *** florianf has joined #openstack-infra | 05:18 | |
| *** salv-orlando has quit IRC | 05:21 | |
| *** gouthamr has joined #openstack-infra | 05:22 | |
| *** slaweq has quit IRC | 05:31 | |
| *** slaweq has joined #openstack-infra | 05:35 | |
| *** armaan has joined #openstack-infra | 05:42 | |
| *** armaan has quit IRC | 05:49 | |
| *** armaan has joined #openstack-infra | 05:50 | |
| *** armaan has quit IRC | 05:51 | |
| *** gouthamr has quit IRC | 05:53 | |
| *** florianf has quit IRC | 06:05 | |
| *** florianf has joined #openstack-infra | 06:05 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack-infra/project-config master: Normalize projects.yaml https://review.openstack.org/513899 | 06:07 |
| *** slaweq has quit IRC | 06:08 | |
| openstackgerrit | Anastasia Kravets proposed openstack-infra/project-config master: [ec2-api] Increase timeout for rally job https://review.openstack.org/513802 | 06:09 |
| *** xarses has quit IRC | 06:14 | |
| *** slaweq has joined #openstack-infra | 06:15 | |
| openstackgerrit | wes hayutin proposed openstack-infra/tripleo-ci master: DEBUG: downgrade iptables https://review.openstack.org/513891 | 06:15 |
| *** salv-orlando has joined #openstack-infra | 06:18 | |
| *** salv-orlando has quit IRC | 06:22 | |
| openstackgerrit | Andreas Florath proposed openstack/diskimage-builder master: mk_build_dir: fix and refactor https://review.openstack.org/510344 | 06:23 |
| *** makowals has quit IRC | 06:25 | |
| *** claudiub|2 has quit IRC | 06:29 | |
| *** edmondsw has joined #openstack-infra | 06:36 | |
| *** edmondsw has quit IRC | 06:40 | |
| *** slaweq has quit IRC | 06:47 | |
| *** ijw has joined #openstack-infra | 06:48 | |
| *** yamamoto has joined #openstack-infra | 06:50 | |
| *** claudiub|2 has joined #openstack-infra | 06:51 | |
| *** ijw has quit IRC | 06:52 | |
| *** armaan has joined #openstack-infra | 06:55 | |
| *** yamamoto has quit IRC | 06:55 | |
| *** yamamoto has joined #openstack-infra | 06:56 | |
| *** thorst has joined #openstack-infra | 06:56 | |
| *** baoli has joined #openstack-infra | 06:58 | |
| *** yamamoto has quit IRC | 07:00 | |
| *** thorst has quit IRC | 07:01 | |
| *** slaweq has joined #openstack-infra | 07:01 | |
| *** armaan has quit IRC | 07:02 | |
| *** baoli has quit IRC | 07:03 | |
| *** salv-orlando has joined #openstack-infra | 07:19 | |
| *** Guest90416 is now known as lamt___ | 07:21 | |
| *** salv-orlando has quit IRC | 07:23 | |
| *** claudiub|2 has quit IRC | 07:34 | |
| *** claudiub|2 has joined #openstack-infra | 07:35 | |
| *** slaweq has quit IRC | 07:39 | |
| *** florianf has quit IRC | 07:41 | |
| *** slaweq has joined #openstack-infra | 07:42 | |
| *** priteau has joined #openstack-infra | 07:43 | |
| *** florianf has joined #openstack-infra | 07:43 | |
| *** priteau has quit IRC | 07:47 | |
| *** thorre has quit IRC | 07:57 | |
| AJaeger | infra-root, first time a proposal job running with v3: https://review.openstack.org/#/c/513899/ . Woot! | 08:00 |
| *** thorre has joined #openstack-infra | 08:03 | |
| *** jamesmcarthur has joined #openstack-infra | 08:03 | |
| *** hashar has joined #openstack-infra | 08:04 | |
| *** jamesmcarthur has quit IRC | 08:08 | |
| SamYaple | nice! | 08:13 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Remove now redundant git review setup https://review.openstack.org/513908 | 08:13 |
| *** armaan has joined #openstack-infra | 08:13 | |
| openstackgerrit | Merged openstack-infra/project-config master: Normalize projects.yaml https://review.openstack.org/513899 | 08:14 |
| AJaeger | and we have first translation proposal job - https://review.openstack.org/#/c/513903/ ! | 08:16 |
| *** psachin has joined #openstack-infra | 08:16 | |
| *** salv-orlando has joined #openstack-infra | 08:19 | |
| *** hashar is now known as hasharAway | 08:20 | |
| openstackgerrit | Antoine Musso proposed openstack/diskimage-builder master: dib-lint: test elements have README.rst file https://review.openstack.org/177832 | 08:23 |
| openstackgerrit | Antoine Musso proposed openstack/diskimage-builder master: dib-lint: test elements have README.rst file https://review.openstack.org/177832 | 08:23 |
| AJaeger | infra-root, we now have all periodic translation jobs running in parallel - and get "[ERROR] Operation failed: operation returned 429 (null): <html><head><title>Error</title></head><body>Too many concurrent requests for client API key (maximum is 6)</body></html>" | 08:24 |
| *** edmondsw has joined #openstack-infra | 08:24 | |
| *** salv-orlando has quit IRC | 08:24 | |
| *** slaweq has quit IRC | 08:25 | |
| AJaeger | #status log increased Zanata limit for concurrent requests to 20 using Zanata UI | 08:26 |
| openstackstatus | AJaeger: finished logging | 08:26 |
| *** edmondsw has quit IRC | 08:28 | |
| *** slaweq has joined #openstack-infra | 08:28 | |
| *** slaweq has quit IRC | 08:29 | |
| *** yolanda has quit IRC | 08:33 | |
| *** sshnaidm has joined #openstack-infra | 08:35 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Remove zuul v2 and jjb content https://review.openstack.org/507180 | 08:37 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Remove jenkins Frozen tests https://review.openstack.org/513910 | 08:37 |
| AJaeger | mordred: rebased and added a patch to allow it to merge ^ | 08:37 |
| *** slaweq has joined #openstack-infra | 08:52 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Rename oslo.messaging legacy jobs https://review.openstack.org/513911 | 08:55 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Remove legacy oslo.messaging jobs https://review.openstack.org/513912 | 08:55 |
| *** thorst has joined #openstack-infra | 08:57 | |
| *** thorst has quit IRC | 09:02 | |
| *** armaan has quit IRC | 09:05 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Remove Ceilometer legacy jobs https://review.openstack.org/510720 | 09:13 |
| mordred | AJaeger: morning! | 09:15 |
| *** jtomasek has joined #openstack-infra | 09:15 | |
| mordred | AJaeger: awesome - have we talked about when we want to merge that? | 09:16 |
| *** salv-orlando has joined #openstack-infra | 09:20 | |
| mordred | AJaeger: yay on proposal bot!!! | 09:23 |
| *** salv-orlando has quit IRC | 09:25 | |
| *** yamamoto has joined #openstack-infra | 09:27 | |
| *** slaweq has quit IRC | 09:28 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Remove Ceilometer legacy jobs https://review.openstack.org/510720 | 09:29 |
| *** tosky has joined #openstack-infra | 09:31 | |
| *** claudiub has joined #openstack-infra | 09:35 | |
| *** yamamoto has quit IRC | 09:36 | |
| *** claudiub|2 has quit IRC | 09:37 | |
| openstackgerrit | Merged openstack-infra/project-config master: Rename oslo.messaging legacy jobs https://review.openstack.org/513911 | 09:44 |
| *** slaweq has joined #openstack-infra | 09:46 | |
| *** yamamoto has joined #openstack-infra | 09:47 | |
| *** armaan has joined #openstack-infra | 09:55 | |
| *** dizquierdo has joined #openstack-infra | 09:56 | |
| *** claudiub has quit IRC | 09:57 | |
| *** claudiub has joined #openstack-infra | 09:57 | |
| *** jtomasek has quit IRC | 10:04 | |
| *** slaweq has quit IRC | 10:06 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Add unittest tips jobs https://review.openstack.org/513913 | 10:12 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Add support for configured NAT source variable https://review.openstack.org/513914 | 10:12 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Add unittest tips jobs https://review.openstack.org/513913 | 10:15 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Add support for configured NAT source variable https://review.openstack.org/513914 | 10:15 |
| *** salv-orlando has joined #openstack-infra | 10:21 | |
| *** yamamoto has quit IRC | 10:25 | |
| *** salv-orlando has quit IRC | 10:25 | |
| AJaeger | mordred: morning! We discussed at last infra team meeting "next week", so I expect the next few days... | 10:28 |
| *** claudiub has quit IRC | 10:30 | |
| *** claudiub has joined #openstack-infra | 10:30 | |
| *** pbourke has quit IRC | 10:33 | |
| *** armaan has quit IRC | 10:34 | |
| *** pbourke has joined #openstack-infra | 10:34 | |
| *** claudiub|2 has joined #openstack-infra | 10:35 | |
| *** yamamoto has joined #openstack-infra | 10:35 | |
| mordred | AJaeger: sweet | 10:36 |
| *** claudiub has quit IRC | 10:36 | |
| *** claudiub has joined #openstack-infra | 10:37 | |
| AJaeger | mordred: remaing publish docs publishing job with problems - that I know of - is placement-api, care to review the stack starting at https://review.openstack.org/513822 ? | 10:37 |
| *** dizquierdo has quit IRC | 10:38 | |
| AJaeger | another open issue is still the propose-update job, our grep fails - I've put something in the etherpad and haven't had time to dig deeper | 10:38 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Add unittest tips jobs https://review.openstack.org/513913 | 10:39 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Add support for configured NAT source variable https://review.openstack.org/513914 | 10:39 |
| openstackgerrit | Monty Taylor proposed openstack/os-client-config master: Added nat_source flag for networks. https://review.openstack.org/513751 | 10:39 |
| openstackgerrit | Monty Taylor proposed openstack/os-client-config master: Fix doc typo https://review.openstack.org/513836 | 10:39 |
| openstackgerrit | Monty Taylor proposed openstack/os-client-config master: Add shade-tox-tips jobs https://review.openstack.org/513915 | 10:39 |
| *** claudiub|2 has quit IRC | 10:41 | |
| *** claudiub|2 has joined #openstack-infra | 10:42 | |
| *** claudiub has quit IRC | 10:46 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: WIP: Remove some unused jobs and templates https://review.openstack.org/513916 | 10:49 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: WIP: Remove some unused jobs and templates https://review.openstack.org/513916 | 10:52 |
| *** ijw has joined #openstack-infra | 10:53 | |
| *** yamamoto has quit IRC | 10:56 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Add devstack jobs for zuul v3 https://review.openstack.org/500365 | 10:57 |
| *** ijw has quit IRC | 10:57 | |
| *** thorst has joined #openstack-infra | 10:58 | |
| *** thorst has quit IRC | 11:02 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/openstack-zuul-jobs master: Use fetch-cover-output role in openstack-tox-cover https://review.openstack.org/513917 | 11:04 |
| mordred | AJaeger: ^^ just noticed - we fixed tox-cover but not openstack-tox-cover | 11:04 |
| openstackgerrit | Monty Taylor proposed openstack-infra/openstack-zuul-jobs master: Set success-url on coverage jobs to cover/ https://review.openstack.org/513918 | 11:15 |
| *** smcginnis has quit IRC | 11:16 | |
| *** thorst has joined #openstack-infra | 11:21 | |
| *** thorst has quit IRC | 11:21 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Support upper-constraints in tox-siblings https://review.openstack.org/513199 | 11:21 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Change tox-siblings disablement to allow opt-in https://review.openstack.org/513919 | 11:21 |
| *** smcginnis has joined #openstack-infra | 11:22 | |
| *** salv-orlando has joined #openstack-infra | 11:22 | |
| mordred | infra-root: ^^ if anyone is around on a saturday, https://review.openstack.org/513919 should keep tox-siblings disabled until such a time as we're ready to re-enable it - but also let jobs that need it opt-in to the behavior in the meantime | 11:24 |
| *** salv-orlando has quit IRC | 11:26 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Add unittest tips jobs https://review.openstack.org/513913 | 11:27 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Add devstack jobs for zuul v3 https://review.openstack.org/500365 | 11:27 |
| mordred | such as https://review.openstack.org/513913 | 11:27 |
| *** armaan has joined #openstack-infra | 11:27 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Add unittest tips jobs https://review.openstack.org/513913 | 11:28 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Add devstack jobs for zuul v3 https://review.openstack.org/500365 | 11:28 |
| *** armaan has quit IRC | 11:31 | |
| *** dave-mccowan has joined #openstack-infra | 11:35 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Add support for configured NAT source variable https://review.openstack.org/513914 | 11:37 |
| mordred | project-config-core: https://review.openstack.org/#/c/513913/ shows https://review.openstack.org/513919 and https://review.openstack.org/513918 both working | 11:38 |
| openstackgerrit | Monty Taylor proposed openstack/os-client-config master: Add shade-tox-tips jobs https://review.openstack.org/513915 | 11:43 |
| openstackgerrit | Monty Taylor proposed openstack/os-client-config master: Added nat_source flag for networks. https://review.openstack.org/513751 | 11:43 |
| openstackgerrit | Monty Taylor proposed openstack/os-client-config master: Fix doc typo https://review.openstack.org/513836 | 11:43 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Add support for configured NAT source variable https://review.openstack.org/513914 | 11:50 |
| *** yamamoto has joined #openstack-infra | 11:56 | |
| *** priteau has joined #openstack-infra | 12:01 | |
| *** armaan has joined #openstack-infra | 12:04 | |
| *** yamamoto has quit IRC | 12:06 | |
| *** nicolasbock has joined #openstack-infra | 12:17 | |
| *** salv-orlando has joined #openstack-infra | 12:23 | |
| *** yamamoto has joined #openstack-infra | 12:27 | |
| *** salv-orlando has quit IRC | 12:27 | |
| *** yamamoto has quit IRC | 12:29 | |
| *** bhavik1 has joined #openstack-infra | 12:31 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Make build-python-release job https://review.openstack.org/513925 | 12:36 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Remove old python-sdist job https://review.openstack.org/513926 | 12:36 |
| openstackgerrit | Monty Taylor proposed openstack-infra/openstack-zuul-jobs master: Stop using tox to build python tarballs https://review.openstack.org/513927 | 12:36 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Stop using tox for releasing python tarballs https://review.openstack.org/513928 | 12:36 |
| *** bhavik1 has quit IRC | 12:39 | |
| *** wolverineav has joined #openstack-infra | 12:52 | |
| openstackgerrit | David Moreau Simard proposed openstack-infra/openstack-zuul-jobs master: Add missing required-projects for puppet-openstackci https://review.openstack.org/513930 | 13:17 |
| pabelanger | mordred: +2 all around | 13:18 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: WIP: Remove some unused jobs and templates https://review.openstack.org/513916 | 13:18 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/puppet-openstackci master: Add support for installing ARA wsgi middleware for sqlite databases https://review.openstack.org/513866 | 13:18 |
| *** yamamoto has joined #openstack-infra | 13:19 | |
| dmsimard | https://review.openstack.org/513930 fixes the broken gate for puppet-openstackci | 13:19 |
| AJaeger | mordred: see https://review.openstack.org/#/c/513737/2 and compare with https://review.openstack.org/513917 | 13:20 |
| *** psachin has quit IRC | 13:21 | |
| dmsimard | mordred: why am I seeing messages about Ansible 2.4 in gate jobs ? | 13:21 |
| dmsimard | http://logs.openstack.org/68/513868/1/check/legacy-infra-puppet-apply-3-ubuntu-trusty/c2828a7/job-output.txt.gz#_2017-10-20_23_07_56_507330 | 13:21 |
| mordred | lookie there - it's a pabelanger and a dmsimard | 13:22 |
| *** thorst has joined #openstack-infra | 13:22 | |
| mordred | AJaeger: looking | 13:22 |
| dmsimard | mostly trying to make progress on ara which I've been neglecting (in favor of other important things, but still) | 13:23 |
| mordred | AJaeger: let's give https://review.openstack.org/#/c/513737 a try | 13:23 |
| AJaeger | could you review the stack starting with https://review.openstack.org/513822 some review, please? | 13:23 |
| *** salv-orlando has joined #openstack-infra | 13:24 | |
| dmsimard | Oh, so it looks like system-config doesn't pin Ansible and ends up with 2.4: tools/prep-apply.sh:/tmp/apply-ansible-env/bin/pip install ansible | 13:24 |
| mordred | yup. on it | 13:24 |
| mordred | AJaeger: look great | 13:26 |
| *** thorst has quit IRC | 13:27 | |
| AJaeger | thanks | 13:27 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: make openstack-tox-cover inherit from tox-cover https://review.openstack.org/513737 | 13:27 |
| *** salv-orlando has quit IRC | 13:28 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/openstack-zuul-jobs master: Set success-url on coverage jobs to cover/ https://review.openstack.org/513918 | 13:28 |
| mordred | AJaeger: while you're here - what do you think about https://review.openstack.org/#/c/513919/ ? | 13:30 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: WIP: Remove some unused jobs and templates https://review.openstack.org/513916 | 13:30 |
| mordred | (that'll unblock a big stack of shade and occ changes) | 13:30 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Add build-placement-api-ref https://review.openstack.org/513822 | 13:30 |
| mordred | btw - new-style v3 devstack jobs for shade are green and ready to land :) | 13:31 |
| AJaeger | \o/ | 13:31 |
| mordred | (the legacy job is bong - but dtroyer and I need to make a new version of that that installs old openstack anyway - so I'm not particuarly interested in fixing the old version) | 13:31 |
| dmsimard | AJaeger: typos in https://review.openstack.org/#/c/513824/ | 13:31 |
| AJaeger | mordred: +3A | 13:32 |
| mordred | AJaeger: thanks! | 13:32 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Fix placement-api jobs https://review.openstack.org/513824 | 13:33 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/project-config master: Add Zuul merge-check template. https://review.openstack.org/511339 | 13:33 |
| AJaeger | dmsimard: quickly fixed - thanks | 13:33 |
| AJaeger | dmsimard: merge-check? We removed that one... | 13:33 |
| dmsimard | AJaeger: it's just a rebase | 13:34 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Remove networking-ovn legacy jobs https://review.openstack.org/513675 | 13:34 |
| AJaeger | dmsimard: I'll fix... | 13:34 |
| dmsimard | AJaeger: sure | 13:34 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Add Zuul system-required template https://review.openstack.org/511339 | 13:35 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Fix doc typo https://review.openstack.org/513932 | 13:35 |
| openstackgerrit | wes hayutin proposed openstack-infra/tripleo-ci master: DEBUG: downgrade iptables https://review.openstack.org/513891 | 13:36 |
| AJaeger | cleanup change for review: https://review.openstack.org/#/c/511432 | 13:37 |
| dmsimard | AJaeger: lol openstack/tatu doesn't even exist ? | 13:38 |
| openstackgerrit | Merged openstack-infra/project-config master: Remove manila legacy jobs https://review.openstack.org/513076 | 13:38 |
| dmsimard | AJaeger: ah: https://review.openstack.org/#/c/511335/ | 13:38 |
| dmsimard | SSH as a service ? | 13:38 |
| dmsimard | ಠ_ಠ| 13:39 |
| *** mriedem has joined #openstack-infra | 13:39 | |
| AJaeger | dmsimard: ah ;) | 13:39 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/project-config master: Add Zuul system-required template https://review.openstack.org/511339 | 13:39 |
| openstackgerrit | Merged openstack-infra/project-config master: Fix neutron-lbaas-dashboard to use -horizon jobs https://review.openstack.org/513561 | 13:39 |
| openstackgerrit | Merged openstack-infra/project-config master: [ec2-api] Increase timeout for rally job https://review.openstack.org/513802 | 13:39 |
| openstackgerrit | Merged openstack-infra/project-config master: Fix placement-api jobs https://review.openstack.org/513824 | 13:40 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: WIP: Remove some unused jobs and templates https://review.openstack.org/513916 | 13:41 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Set success-url on coverage jobs to cover/ https://review.openstack.org/513918 | 13:43 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Remove legacy jobs in Karbor https://review.openstack.org/511432 | 13:44 |
| *** priteau has quit IRC | 13:45 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Change domain in test fixture to example.com https://review.openstack.org/513448 | 13:45 |
| dmsimard | AJaeger: how are you finding unusued jobs/playbooks ? | 13:45 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Change tox-siblings disablement to allow opt-in https://review.openstack.org/513919 | 13:45 |
| dmsimard | AJaeger: or do you just know ? | 13:45 |
| AJaeger | dmsimard: for i in `echo playbooks/legacy/*` ; do j=`basename $i`;echo $j ;grep $j ../project-config/zuul.d/projects.yaml ; done | 13:46 |
| AJaeger | dmsimard: called from openstack-zuul-jobs and then verifying ... | 13:46 |
| mordred | dmsimard: I always just assume AJaeger has a perfect mental map of all jobs in his head | 13:46 |
| AJaeger | there might be better ways... | 13:46 |
| dmsimard | AJaeger: maybe we should add a job that checks for orphaned content | 13:46 |
| AJaeger | dmsimard: when should it run? Could be non-voting... Or just a better script | 13:47 |
| dmsimard | AJaeger: I was thinking about this when I was writing my "role/playbook" finder, Zuul probably knows all of this in real time. For example when you add a job that tries to use a playbook that doesn't exist, it'll yield a syntax error iirc | 13:49 |
| dmsimard | Then again, it might be okay to have a playbook or a role that isn't used -- it *could* be used in the future | 13:50 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Specify duplicate project defn behavior https://review.openstack.org/511025 | 13:50 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Ignore duplicate project-template definitions https://review.openstack.org/513853 | 13:50 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Add support for result data in child jobs https://review.openstack.org/504808 | 13:52 |
| *** slaweq has joined #openstack-infra | 13:53 | |
| openstackgerrit | Merged openstack-infra/shade master: Add unittest tips jobs https://review.openstack.org/513913 | 13:58 |
| *** ramishra has joined #openstack-infra | 13:59 | |
| AJaeger | and one more cleanup: https://review.openstack.org/#/c/510431/ | 14:00 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Remove some unused jobs and templates https://review.openstack.org/513916 | 14:01 |
| weshay|ruck | dmsimard, k.. issue was iptables. This is passing tripleo-ci https://review.openstack.org/#/c/513891/ | 14:01 |
| mordred | jeblair: I just clicked approve on adding new-style devstack jobs to shade woot | 14:01 |
| mordred | weshay|ruck: excellent! | 14:02 |
| weshay|ruck | :) | 14:02 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/puppet-openstackci master: Add support for installing ARA wsgi middleware for sqlite databases https://review.openstack.org/513866 | 14:02 |
| mordred | weshay|ruck: I mena, not great that centos released an iptables update that broke things ... but good that you have a fix for your things ... AND that they actually caught a real error in the world | 14:02 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Remove legacy oslo.messaging jobs https://review.openstack.org/513912 | 14:03 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Remove now unused legacy placement jobs https://review.openstack.org/513823 | 14:03 |
| weshay|ruck | ya | 14:03 |
| weshay|ruck | such is life | 14:03 |
| dmsimard | weshay|ruck: ok so now we need to understand why this is happening | 14:03 |
| mordred | yup | 14:03 |
| dmsimard | weshay|ruck: excluding a base package like that sucks a lot | 14:03 |
| dmsimard | do we have a clue ? | 14:04 |
| weshay|ruck | https://git.centos.org/tag/rpms!iptables.git/imports!c7!iptables-1.4.21-18.2.el7_4 | 14:04 |
| weshay|ruck | oh weird. that link redirects | 14:05 |
| *** Jeffrey4l has quit IRC | 14:05 | |
| dmsimard | weshay|ruck: so https://git.centos.org/blobdiff/rpms!iptables.git/4d0bb22ecdc68576e7a4a7afd966207c50b664d2/SPECS!iptables.spec I guess | 14:05 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Remove ironic legacy jobs https://review.openstack.org/511264 | 14:06 |
| *** ramishra has quit IRC | 14:06 | |
| *** Jeffrey4l has joined #openstack-infra | 14:06 | |
| weshay|ruck | ya.. the two things that EmilienM and I were discussing last night.. was line 219 | 14:06 |
| weshay|ruck | and 165 | 14:06 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Remove legacy OSA role tests https://review.openstack.org/512635 | 14:07 |
| AJaeger | 511264 is rebased, let's get it in, please | 14:07 |
| dmsimard | weshay|ruck: so what's really happening ? Something around restarts or iptables+ip6tables running simultaneously ? | 14:08 |
| *** slaweq has quit IRC | 14:08 | |
| AJaeger | oh, the ironic and OSA changes need some more work - then let's get https://review.openstack.org/513916 in, please | 14:08 |
| dmsimard | I've seen iptables and ip6tables running simultaneously, is that really an issue ? :/ | 14:08 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Remove legacy jobs in Vitrage https://review.openstack.org/510431 | 14:08 |
| * AJaeger will rebase the other two once that is in | 14:09 | |
| mordred | AJaeger: oh wow - that's an exciting patch :) | 14:09 |
| dmsimard | AJaeger: that's a bit hard to review, I'll take some glances but I have faith in yo | 14:09 |
| dmsimard | you* | 14:09 |
| weshay|ruck | dmsimard, the only thing I've been able to 100% determine so far, is w/ the newer version of the rpm we see iptables dropping/rejecting connections during the ci run | 14:09 |
| AJaeger | dmsimard: good that zuul is self-testing ;) | 14:10 |
| mordred | AJaeger: yah - zuul syntax testing itself is pretty amazeballs | 14:10 |
| dmsimard | weshay|ruck: we can try and reproduce this outside of a tripleo context, do you know what kind of connections are being dropped ? some ports ? | 14:10 |
| AJaeger | now we need to improve ansible linting... | 14:10 |
| dmsimard | weshay|ruck: we have a job that is able to test multinode setup | 14:10 |
| AJaeger | pabelanger: want to get your linting change updated? | 14:11 |
| weshay|ruck | dmsimard, which job are you referring to? | 14:11 |
| dmsimard | weshay|ruck: let me submit an example that you can take over if you want, hang on | 14:12 |
| weshay|ruck | k.. thanks | 14:12 |
| dmsimard | weshay|ruck: what's the launchpad id for your bug ? | 14:13 |
| jeblair | AJaeger, jlk, dmsimard, mordred, pabelanger: current release of ansible-lint should have jlk's fix for the 'cmd' issue, so we should be able to use it again now. | 14:14 |
| openstackgerrit | Monty Taylor proposed openstack-infra/openstack-zuul-jobs master: Stop using tox to build python tarballs https://review.openstack.org/513927 | 14:14 |
| openstackgerrit | Monty Taylor proposed openstack-infra/openstack-zuul-jobs master: Enable tox_install_siblings for tox-linters https://review.openstack.org/513933 | 14:14 |
| mordred | jeblair: woot | 14:15 |
| jeblair | https://github.com/willthames/ansible-lint/commit/0cedf2ebf25295bd678d74933e58e3932e59b291 is the merged pr for that and is in the 3.4.16 release | 14:15 |
| jeblair | sorry that was the commit, the pr is https://github.com/willthames/ansible-lint/pull/284 | 14:15 |
| jeblair | i like the last comment: "I actually think the cmd argument should be the better practice, tempted to add a rule for ansible-review to enforce it (but not ansible-lint, as not using it isn't a terrible thing)" | 14:16 |
| jeblair | (i very much agree) | 14:16 |
| dmsimard | yeah, the shell and command modules are a bit weird | 14:16 |
| dmsimard | and then you have the 'args' argument for other parameters | 14:16 |
| dmsimard | everything should be at the first "depth" and 'cmd' is the actual command to run | 14:16 |
| dmsimard | those two modules are probably the oldest ones in Ansible and never grew out of that design | 14:17 |
| *** wolverineav has quit IRC | 14:18 | |
| dmsimard | imagine the outrage if they deprecated that usage :) | 14:18 |
| jeblair | ya. would be nice to have a new version and put the old ones on a long deprecation plan. | 14:18 |
| jeblair | meh, it's easy to deprecate things when you can just add new modules. | 14:19 |
| jeblair | i mean, after you decide on a name it's easy | 14:19 |
| dmsimard | weshay|ruck: nevermind, found it https://bugs.launchpad.net/tripleo/+bug/1725451 | 14:19 |
| openstack | Launchpad bug 1725451 in tripleo "CI: new centos7 image contains a regression that prevents multinode networking setup to work" [Critical,Triaged] - Assigned to Emilien Macchi (emilienm) | 14:19 |
| weshay|ruck | dmsimard, ya.. in alerts on #tripleo | 14:20 |
| weshay|ruck | I'm going to update the commit message and doc on this | 14:20 |
| dmsimard | weshay|ruck: hmm... I *think* I found a flaw in our multinode firewall set up | 14:22 |
| dmsimard | weshay|ruck: my test will confirm it | 14:22 |
| weshay|ruck | k | 14:22 |
| *** salv-orlando has joined #openstack-infra | 14:24 | |
| openstackgerrit | wes hayutin proposed openstack-infra/tripleo-ci master: downgrade iptables to iptables-services-1.4.21-18.0.1.el7 https://review.openstack.org/513891 | 14:25 |
| weshay|ruck | dmsimard, ok I'm going to go feed my kids, but I'll be back later. Please let me know what you found | 14:25 |
| openstackgerrit | wes hayutin proposed openstack-infra/tripleo-ci master: downgrade iptables to iptables-services-1.4.21-18.0.1.el7 https://review.openstack.org/513891 | 14:26 |
| dmsimard | weshay|ruck: so we're not adding the bridge IPs on iptables, but I'm not sure if that matters | 14:26 |
| dmsimard | weshay|ruck: we have a test that checks if we can ping the bridge addresses and it works so... anyway, we'll see. | 14:27 |
| weshay|ruck | k.. :) back in a bit, thanks for poking at it | 14:27 |
| *** weshay|ruck is now known as weshay|bbiab | 14:27 | |
| openstackgerrit | Andreas Florath proposed openstack/diskimage-builder master: [WIP] disk-usage refactoring https://review.openstack.org/512085 | 14:27 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/openstack-zuul-jobs master: Do not merge: Try to reproduce #1725451 https://review.openstack.org/513934 | 14:27 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Remove some unused jobs and templates https://review.openstack.org/513916 | 14:28 |
| *** salv-orlando has quit IRC | 14:29 | |
| dmsimard | weshay|bbiab: wow, jlk created the iptables distgit repo for iptables in fedora :) | 14:30 |
| dmsimard | small world | 14:30 |
| *** dave-mccowan has quit IRC | 14:32 | |
| dmsimard | man, rhel and centos packages... iptables version 1.4.21 BUT release 18.2 | 14:32 |
| dmsimard | like kernel packages.... 2.6.32-1037 | 14:33 |
| dmsimard | EmilienM, weshay|bbiab: ohhhhhhhhhhhhh https://bugzilla.redhat.com/show_bug.cgi?id=1504647 | 14:36 |
| openstack | bugzilla.redhat.com bug 1504647 in iptables "Fix invalid syntax in ip6tables.service systemd unit file" [High,Assigned] - Assigned to psutter | 14:36 |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config master: Grafana: scale zuul executor load correctly https://review.openstack.org/513936 | 14:38 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Remove ironic legacy jobs https://review.openstack.org/511264 | 14:39 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Remove ironic legacy jobs https://review.openstack.org/511264 | 14:43 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Remove ironic legacy jobs https://review.openstack.org/511264 | 14:45 |
| dmsimard | hm, persisting our iptables rules will be tricky, it's different on every distro :/ | 14:46 |
| dmsimard | AJaeger: do you know for sure how to persist iptables rules on suse ? seeing different answers on stackoverflow :) | 14:50 |
| *** priteau has joined #openstack-infra | 14:54 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Removal of kolla legacy jobs https://review.openstack.org/508950 | 14:58 |
| AJaeger | dmsimard: not really. Maybe dirk does? | 14:58 |
| dmsimard | AJaeger: I'll spin up an opensuse instances and try to figure it out | 14:59 |
| AJaeger | dmsimard: I'm running opensuse here as well - do you have something for me to check? | 15:01 |
| dmsimard | AJaeger: mine is spawning right now, was mostly interested in the content of /etc/sysconfig/SuSEfirewall2 | 15:02 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Remove manila legacy jobs https://review.openstack.org/513075 | 15:03 |
| *** priteau has quit IRC | 15:03 | |
| dirk | dmsimard there is a way to do custom rules with susefirewall | 15:03 |
| dirk | Newer distros of opensuse use firewalld | 15:03 |
| dirk | dmsimard afk | 15:04 |
| dmsimard | AJaeger: rhel/centos/fedora use /etc/sysconfig/iptables and /etc/sysconfig/ip6tables (output from iptables-save), debian/ubuntu use iptables-persistent which use /etc/iptables/rules.v4 and /etc/iptables/rules.v6 (content from iptables-save) | 15:04 |
| dmsimard | dirk: is there a way to do like the above ^ on suse ? | 15:04 |
| AJaeger | this one is ready for review now https://review.openstack.org/508950 | 15:04 |
| AJaeger | dmsimard: could you +2A https://review.openstack.org/#/c/513278 , please? | 15:05 |
| dmsimard | oh man SuseFirewall2 is another format entirely | 15:06 |
| dmsimard | looks more like firewalld kind of thing | 15:06 |
| cmurphy | this is how we persist iptables rules with susefirewall2 http://git.openstack.org/cgit/openstack-infra/project-config/tree/nodepool/elements/nodepool-base/install.d/20-iptables#n90 | 15:08 |
| dmsimard | cmurphy: oh cool so same thing as rhel/centos then, let me try that | 15:10 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Remove legacy-cross-*-* jobs related to global requirements https://review.openstack.org/513278 | 15:14 |
| dmsimard | cmurphy: do you know what loads iptables in the first place ? I'm able to list rules (i.e, iptables -vnL) but closest service I'm finding is "SuSEfirewall2.service" but it's disabled. | 15:14 |
| AJaeger | and https://review.openstack.org/#/c/513075/2 is ready for review as well | 15:14 |
| AJaeger | dmsimard: there's also SuSEfirewall2_init.service | 15:15 |
| dmsimard | AJaeger: but that's also disabled | 15:15 |
| dmsimard | there's no 'iptables' service as far as I can tell either | 15:15 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Remove legacy OSA role tests https://review.openstack.org/512635 | 15:16 |
| dirk | dmsimard: that's correct | 15:17 |
| mordred | jeblair: looking at your scale zuul load average patch for grafana got me looking at the zuul status grafana dashboard | 15:17 |
| cmurphy | dmsimard: SuSEfirewall2 is the service, it should load the rules on start when you have that FW_CUSTOMRULES setting and loading function like at the bottom of the file i linked | 15:17 |
| dirk | Unfortunately there isn't anything equivalently simple on opensuse. Either Susefirewall or firewalld (starting with the upcoming release) | 15:18 |
| mordred | jeblair: "Zuul Jobs Launched (per Hour)" and "Gerrit Events (per hour)" both stop having data after 16 Oct | 15:18 |
| mordred | jeblair: is that expected? | 15:18 |
| dmsimard | cmurphy: ok, at least on my test instance that service is disabled and not running, I'll fiddle with it a bit.. | 15:19 |
| dirk | dmsimard: that's nodepool image? | 15:19 |
| dmsimard | dirk: an opensuse leap image, closest I've got :/ | 15:19 |
| openstackgerrit | Merged openstack/os-client-config master: Update make_rest_client to work with version discovery https://review.openstack.org/513013 | 15:19 |
| dmsimard | dirk: maybe there's some stuff handling that in the nodepool image | 15:20 |
| AJaeger | mordred, could you +2 https://review.openstack.org/508950 , please? | 15:20 |
| dirk | dmsimard: it is possible that we forgot to enable the service | 15:20 |
| dirk | cmurphy: congratulations, btw! | 15:20 |
| cmurphy | dirk: ty | 15:20 |
| dmsimard | dirk: I'll blindly try cmurphy's suggestion on the integration job which runs on the nodepool image and we'll see what happens :) | 15:21 |
| AJaeger | cmurphy: congratulations from me as well | 15:21 |
| mordred | ++ | 15:21 |
| dmsimard | oh, yeah, congrats cmurphy :) | 15:21 |
| dirk | dmsimard: on suse services normally don't get enabled just by installing, they need to be sustemctl enable'd explicitly. That's a difference to most other distros | 15:22 |
| dmsimard | dirk: pretty sure debian and red hat does that a lot too, but probably not universally | 15:23 |
| dmsimard | like yum install httpd; systemctl enable --now httpd :) | 15:23 |
| *** thorst has joined #openstack-infra | 15:23 | |
| *** claudiub|2 has quit IRC | 15:23 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Add stats for executor and merger count https://review.openstack.org/513940 | 15:25 |
| *** salv-orlando has joined #openstack-infra | 15:25 | |
| jeblair | mordred: yeah, it's not in a great state is it? :) | 15:25 |
| *** fultonj has joined #openstack-infra | 15:27 | |
| *** fultonj has left #openstack-infra | 15:27 | |
| *** thorst has quit IRC | 15:27 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Remove unused legacy templates https://review.openstack.org/513941 | 15:29 |
| jeblair | mordred: i *think* we're probably emitting the right stats for that still, but just with a different key. i haven't looked it up yet. | 15:29 |
| *** salv-orlando has quit IRC | 15:30 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Use write-root-marker role https://review.openstack.org/513942 | 15:32 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/zuul-jobs master: Persist iptables rules https://review.openstack.org/513943 | 15:32 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/openstack-zuul-jobs master: Do not merge: Try to reproduce #1725451 https://review.openstack.org/513934 | 15:33 |
| mordred | jeblair: ok. cool- so it's a thing that's worth investigating, rather than a thing we stopped emitting on purpose and should clean up the graphs | 15:33 |
| jeblair | mordred: yup | 15:33 |
| dmsimard | AJaeger, dirk, cmurphy: I think https://review.openstack.org/#/c/513943/1/roles/multi-node-firewall/tasks/persist/Suse.yaml should do it | 15:33 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/zuul-jobs master: Persist iptables rules https://review.openstack.org/513943 | 15:34 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/zuul-jobs master: Persist iptables rules https://review.openstack.org/513943 | 15:37 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Revert "Fix branch check for proposal script" https://review.openstack.org/513944 | 15:37 |
| AJaeger | mordred, jeblair, dmsimard the above should fix our proposal script ^ | 15:37 |
| dmsimard | ok signing off for now, see ya later | 15:38 |
| AJaeger | thanks, dmsimard ! Enjoy the weekend | 15:38 |
| *** fultonj has joined #openstack-infra | 15:39 | |
| mordred | AJaeger: +2 | 15:43 |
| *** armax has joined #openstack-infra | 15:46 | |
| *** armax_ has joined #openstack-infra | 15:49 | |
| mordred | infra-root, project-config-root: I have a VERY small sample size so far - but I've had 2 devstack runs fail shade functional tests in the last 4 hours in a very weird way (timeout waiting for a VM to boot) - both were in OVH | 15:50 |
| mordred | gah. project-config-core I meant | 15:50 |
| *** armax has quit IRC | 15:50 | |
| *** armax_ is now known as armax | 15:50 | |
| mordred | I mention it just so folks can keep an eye out to see if there is a real something or if I just got unluckyu | 15:51 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Fix wheel/release https://review.openstack.org/513945 | 15:55 |
| mordred | AJaeger: ^^ whoops :) | 15:55 |
| *** caphrim007 has joined #openstack-infra | 15:56 | |
| EmilienM | dmsimard: yeah | 15:56 |
| EmilienM | dmsimard: I said it was iptables :D | 15:56 |
| EmilienM | dmsimard: https://review.openstack.org/#/c/513891/ | 15:57 |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config master: Grafana: fix zuul job launch graph https://review.openstack.org/513946 | 15:58 |
| *** fultonj has quit IRC | 15:58 | |
| EmilienM | weshay|bbiab: https://review.openstack.org/513943 | 15:59 |
| *** fultonj has joined #openstack-infra | 15:59 | |
| dmsimard | EmilienM: I updated the launchpad | 15:59 |
| AJaeger | mordred: currently going through jlk's https://review.openstack.org/#/c/503751 and looking for nuggets ;) | 15:59 |
| AJaeger | we need ansible-linter | 15:59 |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config master: Grafana: fix gerrit event graph https://review.openstack.org/513947 | 15:59 |
| *** armax has quit IRC | 16:00 | |
| EmilienM | dmsimard: ok thanks | 16:01 |
| clarkb | mordred: are you disabling kvm and using qemu? | 16:01 |
| *** fultonj has quit IRC | 16:01 | |
| clarkb | mordred: devstack-gate does this in the legacy jobs | 16:02 |
| *** fultonj has joined #openstack-infra | 16:02 | |
| clarkb | (for this reason, weird vm boot failures and crashes) | 16:02 |
| mordred | clarkb: I'll look - I don't know if the new devstack basejobs do thator not | 16:03 |
| *** nhandler has quit IRC | 16:04 | |
| mordred | clarkb: booyah. great call! https://review.openstack.org/513948 Disable nested virt in base devstack job | 16:05 |
| jeblair | looks like we need to add LIBVIRT_TYPE: qemu | 16:05 |
| jeblair | which is what that patch does! +2 | 16:06 |
| mordred | also - in case anyone is wondering - the failure rate is approx 1/8 of the time ... so it IS still a good idea that we do it :) | 16:06 |
| *** nhandler has joined #openstack-infra | 16:06 | |
| *** armax has joined #openstack-infra | 16:08 | |
| *** claudiub|2 has joined #openstack-infra | 16:10 | |
| *** yamamoto has quit IRC | 16:10 | |
| *** caphrim007 has quit IRC | 16:10 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Report initial pipeline changes under correct key https://review.openstack.org/513949 | 16:10 |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config master: Grafana: fix zuul pipeline gauges https://review.openstack.org/513950 | 16:12 |
| *** reed has quit IRC | 16:12 | |
| EmilienM | dmsimard: if the workaround works, https://review.openstack.org/#/c/513891/ - we'll merge it until we can use your patch | 16:12 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Re-enable shade-functional-devstack-legacy https://review.openstack.org/513951 | 16:12 |
| jeblair | heh, i wondered why those were working -- turns out we were only emitting them on reconfiguration. which happens a lot. so... | 16:12 |
| *** reed has joined #openstack-infra | 16:13 | |
| *** abishop has joined #openstack-infra | 16:13 | |
| *** fultonj has quit IRC | 16:13 | |
| dmsimard | EmilienM: I'm not sure my patch fixes your issue, but it's something we need to be doing regardless | 16:14 |
| dmsimard | EmilienM: I'm trying to do a reproducer outside of tripleo and understand what is going on | 16:15 |
| EmilienM | dmsimard: are you able to reproduce it outside of tripleo? we did reproduce the same problem in RDO CI | 16:16 |
| openstackgerrit | Merged openstack-infra/project-config master: Grafana: scale zuul executor load correctly https://review.openstack.org/513936 | 16:16 |
| dmsimard | EmilienM: haven't been able yet, but I found the persist issue so I'm taking care of that first | 16:17 |
| EmilienM | dmsimard: wes's patch confirms it's iptables anyway | 16:18 |
| EmilienM | dmsimard: and looking at the git diff in iptables, I confirm the persistence changed | 16:18 |
| EmilienM | https://git.centos.org/blob/rpms!iptables.git/4d0bb22ecdc68576e7a4a7afd966207c50b664d2/SPECS!iptables.spec#L278 | 16:18 |
| dmsimard | EmilienM: look at the bugzilla I linked in the launchpad, there are some issue with unit files too | 16:19 |
| EmilienM | sorry this link: https://git.centos.org/commitdiff/rpms!iptables.git/4d0bb22ecdc68576e7a4a7afd966207c50b664d2 | 16:19 |
| EmilienM | dmsimard: yeah indeed :/ | 16:19 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Fix flake8 https://review.openstack.org/513952 | 16:19 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Run pep8 everywhere https://review.openstack.org/513953 | 16:19 |
| openstackgerrit | Merged openstack-infra/project-config master: Grafana: fix zuul job launch graph https://review.openstack.org/513946 | 16:20 |
| openstackgerrit | Merged openstack-infra/project-config master: Grafana: fix gerrit event graph https://review.openstack.org/513947 | 16:20 |
| openstackgerrit | Merged openstack-infra/project-config master: Grafana: fix zuul pipeline gauges https://review.openstack.org/513950 | 16:20 |
| *** slaweq has joined #openstack-infra | 16:22 | |
| *** slaweq has quit IRC | 16:23 | |
| mordred | AJaeger: don't know if you were aware - but you can totally do depends-on with patches in the same repo - so if you wanted you could depends-on that second pep8 patch on the v2 removal patch (also, it's totally not necessary) | 16:23 |
| *** armax has quit IRC | 16:23 | |
| *** salv-orlando has joined #openstack-infra | 16:26 | |
| AJaeger | mordred: didn#t think about it - good idea... | 16:26 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Run pep8 everywhere https://review.openstack.org/513953 | 16:27 |
| AJaeger | mordred: added dependency... | 16:28 |
| mordred | AJaeger: +2 from me | 16:29 |
| *** salv-orlando has quit IRC | 16:30 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Fix flake8 https://review.openstack.org/513952 | 16:31 |
| AJaeger | mordred: thanks. Need to fix the first one - now dib linter complains... | 16:32 |
| *** fultonj has joined #openstack-infra | 16:32 | |
| *** nhandler has quit IRC | 16:35 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Fix flake8 https://review.openstack.org/513952 | 16:37 |
| AJaeger | what is this: http://logs.openstack.org/ec/ec22e46a850af1871cbb331c6a679474a3bbf965/post/publish-openstack-python-branch-tarball/dbf3ac0/ara/result/246a9359-8641-4a1b-a831-cc40d797a59d/ - Operatoin not permitted for the publish job? | 16:38 |
| *** nhandler has joined #openstack-infra | 16:40 | |
| *** eharney has quit IRC | 16:41 | |
| mordred | AJaeger: looking | 16:41 |
| mordred | AJaeger: that file (/srv/static/tarballs/trove/images/README) is owned by root | 16:43 |
| jlk | dmsimard: I created every Fedora dist-git repo (as of 2010 ish I think). I did the migration from cvs to git for all of Fedora (and RHEL). | 16:44 |
| mordred | clarkb: if you swing back through, feel like +Ain https://review.openstack.org/#/c/513948/ ? (It's the nested virt patch) | 16:45 |
| SpamapS | mmmmmm csv to git | 16:45 |
| SpamapS | that's a happy day | 16:45 |
| mordred | bzr to git has nothing on cvs to git | 16:46 |
| * SpamapS once took a company from {nothing} to csv to svn to perforce to bzr over a 6 year period... | 16:46 | |
| SpamapS | csv? cvs.. thanks | 16:46 |
| *** dave-mccowan has joined #openstack-infra | 16:46 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Add stats for executor and merger count https://review.openstack.org/513940 | 16:49 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Report initial pipeline changes under correct key https://review.openstack.org/513949 | 16:49 |
| jeblair | SpamapS: assumed you just meant they were programming in excel | 16:49 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Run pep8 everywhere https://review.openstack.org/513953 | 16:50 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/zuul-jobs master: Persist iptables rules https://review.openstack.org/513943 | 16:53 |
| SpamapS | jeblair: that excel to svn migration was hell. | 16:55 |
| SpamapS | Had to write it all in VBscript | 16:55 |
| *** mriedem has quit IRC | 16:56 | |
| AJaeger | mordred: and we should not overwrite it I guess - why is it pushed? | 16:56 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Enable ansible syntax test https://review.openstack.org/513955 | 16:56 |
| AJaeger | or change permissions to allow pushing? | 16:56 |
| *** Adri2000 has joined #openstack-infra | 16:57 | |
| *** Adri2000 has joined #openstack-infra | 16:57 | |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config master: Ignore some more pep8 errors https://review.openstack.org/513956 | 17:01 |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config master: Ignore some more pep8 errors https://review.openstack.org/513956 | 17:03 |
| jeblair | AJaeger: some of the things you fixed are things we typically ignore ^ | 17:03 |
| jeblair | 'fixed' should have been in quotes | 17:03 |
| jeblair | (the 'fixes' are fine, after all, they're ignored :) | 17:04 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/project-config master: Add multinode-integration jobs to zuul-jobs https://review.openstack.org/513957 | 17:05 |
| dmsimard | project-config-core: ^ can we get that in, they're not running on z-j see https://review.openstack.org/#/c/513943/ | 17:06 |
| mordred | AJaeger: I'm not sure which things are right | 17:07 |
| *** yamamoto has joined #openstack-infra | 17:11 | |
| *** gmann is now known as gmann_afk | 17:14 | |
| *** yamamoto has quit IRC | 17:19 | |
| pabelanger | AJaeger: yah, I'll push up a change this afternoon | 17:20 |
| AJaeger | jeblair: ok, understood... | 17:26 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Enable ansible syntax test https://review.openstack.org/513955 | 17:26 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Enable ansible syntax test https://review.openstack.org/513955 | 17:32 |
| *** bhavik1 has joined #openstack-infra | 17:33 | |
| *** bhavik1 has quit IRC | 17:34 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Create infra-zuul-jobs-linters job https://review.openstack.org/513958 | 17:35 |
| pabelanger | AJaeger: ^suggestions for job name welcome | 17:35 |
| AJaeger | pabelanger: I'll abandon 513955 ;) | 17:37 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/zuul-jobs master: Persist iptables rules https://review.openstack.org/513943 | 17:37 |
| AJaeger | pabelanger: care to +2A https://review.openstack.org/513952 , please? | 17:37 |
| *** salv-orlando has joined #openstack-infra | 17:37 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Create infra-zuul-jobs-linters job https://review.openstack.org/513958 | 17:38 |
| mordred | pabelanger: does tox even give us any value in that job? I kinda wonder if we shouldn't just have playbook/role that does that the linters env in each repo does - but targetting all three repos content too | 17:39 |
| mordred | pabelanger: (I think that patch looks great - just wondering out loud) | 17:39 |
| AJaeger | pabelanger: what about adding a new job for now - ansible-lint, have that non-voting and once we make it voting, merge it with linters? | 17:39 |
| pabelanger | mordred: yah, we discussed dropping tox and writing playbooks directly | 17:40 |
| AJaeger | I don't want to have regression come in for pep8 or bashate | 17:40 |
| mordred | pabelanger: kk. maybe we do tha as a follow up | 17:40 |
| pabelanger | so, if we want to do that, we can iterate in ozj first, then promote to project-config (because main.yaml sort things) | 17:40 |
| AJaeger | pabelanger: leave the job name, just call tox differently... | 17:41 |
| pabelanger | mordred: AJaeger: k | 17:41 |
| *** salv-orl_ has joined #openstack-infra | 17:41 | |
| mordred | pabelanger: well - if we keep tox-linters in each repo for pep8 and main.yaml sort - we could do infra-zuul-jobs-linters in ozj and leave it there - or does that seem like too much? | 17:41 |
| *** salv-orl_ has quit IRC | 17:42 | |
| pabelanger | mordred: yah, that would be fine I think | 17:42 |
| AJaeger | yes, we could also do it first for ozj and zuul-jobs - and once those two are voting, add to project-config and fix jobs at the same time | 17:42 |
| pabelanger | right, just keep in mind, we can start in ozj, but no other project will be able to run the job, due to sort ording in zuul-jobs. But once it is working, we can put it back into project-config | 17:43 |
| *** salv-orlando has quit IRC | 17:44 | |
| pabelanger | I'll iterate on 512715 | 17:44 |
| pabelanger | and abandon the other | 17:44 |
| mordred | pabelanger: I'm not following what you mean about sort ordering? | 17:44 |
| pabelanger | mordred: project-config, wont see the job in ozj | 17:44 |
| pabelanger | mordred: same with zuul-jobs | 17:44 |
| pabelanger | because ozj is listed after those projects in main.yaml | 17:45 |
| mordred | that's only for parent/child relations | 17:45 |
| mordred | pabelanger: we can totally add it to projet pipelines of all three projects | 17:45 |
| pabelanger | Oh, yes, you are correct | 17:45 |
| pabelanger | nevermind then :D | 17:45 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/openstack-zuul-jobs master: Add integration test coverage for iptables persistence https://review.openstack.org/513934 | 17:45 |
| mordred | woot! I was worried for a sec that jetlag was messing with me :) | 17:45 |
| pabelanger | no, just my brain not working again :) | 17:46 |
| mordred | pabelanger: it's been a long few weeks :) | 17:46 |
| mordred | pabelanger: congrats (or condolences) btw! | 17:46 |
| AJaeger | pabelanger: congratulations ! | 17:47 |
| AJaeger | mordred: let's talk about condolences after a few weeks ;) | 17:47 |
| pabelanger | mordred: AJaeger: Thanks. Lots of new stuff to learn | 17:47 |
| *** jamesmcarthur has joined #openstack-infra | 17:48 | |
| AJaeger | ;) | 17:48 |
| *** nicolasbock has quit IRC | 17:49 | |
| AJaeger | pabelanger: did your tests catch https://review.openstack.org/513945 ? jlk found that with ansible-lint, I just pushed it forward | 17:49 |
| *** dave-mccowan has quit IRC | 17:50 | |
| openstackgerrit | Merged openstack-infra/project-config master: Fix flake8 https://review.openstack.org/513952 | 17:50 |
| openstackgerrit | Merged openstack-infra/project-config master: Ignore some more pep8 errors https://review.openstack.org/513956 | 17:50 |
| pabelanger | AJaeger: no, the job did run on project-config yet | 17:52 |
| pabelanger | but would have | 17:52 |
| * AJaeger signs off for today, have a great Saturday! | 17:54 | |
| *** fultonj has quit IRC | 17:57 | |
| *** jamesmcarthur has quit IRC | 18:04 | |
| *** thorst has joined #openstack-infra | 18:05 | |
| *** thorst has quit IRC | 18:06 | |
| openstackgerrit | Merged openstack-infra/shade master: Add devstack jobs for zuul v3 https://review.openstack.org/500365 | 18:22 |
| smcginnis | Am I right thinking we'll need this to land before we try another lib update release? https://review.openstack.org/#/c/513944/ | 18:23 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Use new targets for Test Nodes panel https://review.openstack.org/513959 | 18:25 |
| pabelanger | anybody know why we have 16 nodes held? http://grafana.openstack.org/dashboard/db/nodepool | 18:26 |
| pabelanger | seems to be climbing too | 18:26 |
| smcginnis | pabelanger: Hopefully not leftover from when we were debugging release failures. | 18:26 |
| smcginnis | pabelanger: What's the time range on that data? | 18:27 |
| clarkb | pabelanger I think jeblair set a hold 20 for EmilienM to debug multinode overlay in tripleo | 18:27 |
| clarkb | because it is the weekend | 18:27 |
| pabelanger | ah | 18:27 |
| pabelanger | yah, I see tripleo iptables debug | 18:27 |
| pabelanger | kewl | 18:27 |
| pabelanger | smcginnis: top right shows the default, which is 6 hours | 18:28 |
| * smcginnis finally notices the 6 hour message in the top corner | 18:28 | |
| EmilienM | https://review.openstack.org/#/c/513891/ | 18:28 |
| smcginnis | :) | 18:28 |
| mordred | \o/ | 18:28 |
| mordred | native devstack jobs for shade landed | 18:28 |
| EmilienM | this is a workaround for now - we needed to make sure it was IPtables. | 18:28 |
| pabelanger | mordred: Nice | 18:28 |
| EmilienM | pabelanger: and dmsimard is working on https://review.openstack.org/513943 | 18:29 |
| smcginnis | I believe this one also needs to land before we can get the proposal-bot working: https://review.openstack.org/#/c/513864/ | 18:29 |
| pabelanger | EmilienM: wait, you are using wget to fetch an RPM? | 18:29 |
| pabelanger | EmilienM: doesn't the old version exist in the repo? | 18:29 |
| *** yamahata has joined #openstack-infra | 18:29 | |
| openstackgerrit | Merged openstack-infra/project-config master: Revert "Fix branch check for proposal script" https://review.openstack.org/513944 | 18:30 |
| EmilienM | pabelanger: I tried in my env, and no I didn't have the old pkg | 18:30 |
| *** xarses has joined #openstack-infra | 18:30 | |
| pabelanger | http://mirror.dfw.rax.openstack.org/centos/7/os/x86_64/Packages/iptables-1.4.21-18.0.1.el7.centos.i686.rpm | 18:31 |
| pabelanger | we still mirror it | 18:31 |
| EmilienM | for how long | 18:31 |
| EmilienM | but ok | 18:31 |
| EmilienM | weshay|bbiab: ^ | 18:31 |
| pabelanger | so, you could just be able to yum install==<that version | 18:31 |
| EmilienM | right | 18:31 |
| pabelanger | well, it should be there until centos purges it | 18:31 |
| pabelanger | I don't know how often that is | 18:31 |
| pabelanger | wait | 18:32 |
| pabelanger | that will be there forever | 18:32 |
| pabelanger | that is the base OS package | 18:32 |
| weshay|bbiab | we call yum update, etc.. so we need to handle that too | 18:32 |
| pabelanger | so, that was the release of 7.4 then | 18:32 |
| weshay|bbiab | I assume this will be fixed in an upcoming release of iptables | 18:32 |
| EmilienM | pabelanger: no, it was iptables upgrade 2 days ago | 18:32 |
| SamYaple | mordred: nice! | 18:33 |
| pabelanger | yah, so just uninstall iptable from updates repo | 18:33 |
| pabelanger | you'll fall back to base OS version of iptables | 18:33 |
| EmilienM | https://git.centos.org/blob/rpms!iptables.git/4d0bb22ecdc68576e7a4a7afd966207c50b664d2/SPECS!iptables.spec#L278 | 18:33 |
| EmilienM | dmsimard also found a BZ https://bugzilla.redhat.com/show_bug.cgi?id=1504647 | 18:33 |
| openstack | bugzilla.redhat.com bug 1504647 in iptables "Fix invalid syntax in ip6tables.service systemd unit file" [High,Assigned] - Assigned to psutter | 18:33 |
| *** andreww has joined #openstack-infra | 18:33 | |
| pabelanger | EmilienM: right, if you install centos 7.4 with out any updates / security iptable will work. It's only after you install package from updates repo, so you can uninstall and blacklist the updates version, and by defualt it should install iptables from OS (7.4 release version) | 18:34 |
| weshay|bbiab | ya.. the bug indicates it should not have passed tests | 18:34 |
| SamYaple | if its just a service file issue, you could override that, even before iptables update is installed | 18:35 |
| SamYaple | in the short term | 18:35 |
| *** xarses has quit IRC | 18:35 | |
| weshay|bbiab | odly I noticed this didn't do the trick sudo yum-config-manager --save --setopt=exclude=iptables* | 18:35 |
| weshay|bbiab | anyone see something I did wrong there | 18:36 |
| clarkb | you shouldn't require people to depend on an unpatched platform to use your software. Ideally the fix would be to the updated packages | 18:36 |
| weshay|bbiab | ya.. agree | 18:36 |
| pabelanger | clarkb: yah, that too | 18:37 |
| pabelanger | when did centos-7 break? | 18:41 |
| pabelanger | EmilienM: | 18:42 |
| EmilienM | yesterday morning 11am | 18:42 |
| pabelanger | yah, we could have rolled the image back | 18:43 |
| EmilienM | FYI i'm totally unresponsive today, doing customer support all day and stuck on that. | 18:43 |
| EmilienM | talk with weshay|bbiab please | 18:43 |
| pabelanger | but, looks like we've already upload images for today | 18:43 |
| EmilienM | pabelanger: I asked roll back | 18:43 |
| pabelanger | so, we've deleted the good images | 18:43 |
| weshay|bbiab | pabelanger, so you are suggesting "yum downgrade --allowerasing iptables iptables-services" | 18:45 |
| pabelanger | weshay|bbiab: yah, if you downgrade to working iptables, everything should be fine right? | 18:45 |
| mordred | yah. I think that's the workaround for now | 18:45 |
| weshay|bbiab | pabelanger, as long as we save the rules ya | 18:46 |
| *** rwsu has quit IRC | 18:46 | |
| mordred | and should allow breathing room to figure out what the right way to deal with the new packages is | 18:46 |
| pabelanger | yah, I don't think you need to wget rpms from the internet, since this seems to be the first update for iptable in 7.4. | 18:46 |
| pabelanger | and os package works as expected | 18:47 |
| weshay|bbiab | k k.. so just remove the wget | 18:47 |
| pabelanger | does centos-7 on devstack fail too? | 18:47 |
| pabelanger | I would assume so | 18:47 |
| openstackgerrit | wes hayutin proposed openstack-infra/tripleo-ci master: downgrade iptables to iptables-services-1.4.21-18.0.1.el7 https://review.openstack.org/513891 | 18:50 |
| weshay|bbiab | hrm.. I may need remove .rpm | 18:51 |
| mordred | pabelanger: I think the devstack and tripleo-ci iptables rules and setup are different - I believe dmsimard started looking to see what was different between them, but the focus has been on stopping the bleeding first | 18:52 |
| pabelanger | mordred: Yah, bleeding for sure. I'll wait to see what dmsimard finds out | 18:52 |
| openstackgerrit | wes hayutin proposed openstack-infra/tripleo-ci master: downgrade iptables to iptables-services-1.4.21-18.0.1.el7 https://review.openstack.org/513891 | 18:53 |
| weshay|bbiab | pabelanger, does that look right to you now? | 18:54 |
| clarkb | jeblair: in trying to fix my tests for 513850 I think at least some of the zuul tests are buggy in how they use getJobFromHistory | 18:55 |
| openstackgerrit | Andreas Florath proposed openstack/diskimage-builder master: [WIP] disk-usage refactoring https://review.openstack.org/512085 | 18:55 |
| clarkb | basically we don't reset the history when checking subsequent jobs so you can match against the original results | 18:56 |
| mordred | weshay|bbiab: looks sane to me ... fwiw, you don't need those ; at the ends of the lines - but they also don't hurt anything | 18:57 |
| clarkb | test_merger_repack attempts to reset that history but I think it is operating on the wrong object | 18:57 |
| clarkb | (should be self.executor_server) | 18:57 |
| weshay|bbiab | mordred, thanks.. I think it's just for good luck :) | 18:57 |
| mordred | weshay|bbiab: ++ :) | 18:57 |
| pabelanger | mordred: weshay|bbiab yah, looks correct | 18:57 |
| weshay|bbiab | thanks guys | 18:57 |
| mordred | weshay|bbiab: fingers crossed! | 18:58 |
| *** rwsu has joined #openstack-infra | 19:00 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Avoid tox_install.sh for constraints support https://review.openstack.org/508061 | 19:01 |
| openstackgerrit | Clark Boylan proposed openstack-infra/zuul feature/zuulv3: Only autohold failed builds https://review.openstack.org/513850 | 19:01 |
| mordred | tonyb: ^^ I rebased that and removed the WIP comment - I'm using both constraints and tox-siblings in shade, so it should be a decent guinea pig for the non-neutron case | 19:02 |
| clarkb | jeblair: I'll probably dig into fixing those tests on monday (for now have jsut fixed the tests on my change) | 19:03 |
| openstackgerrit | Monty Taylor proposed openstack/os-client-config master: Add shade-tox-tips jobs https://review.openstack.org/513915 | 19:04 |
| openstackgerrit | Monty Taylor proposed openstack/os-client-config master: Added nat_source flag for networks. https://review.openstack.org/513751 | 19:04 |
| openstackgerrit | Monty Taylor proposed openstack/os-client-config master: Fix doc typo https://review.openstack.org/513836 | 19:04 |
| *** bobh has joined #openstack-infra | 19:13 | |
| *** bobh has quit IRC | 19:14 | |
| *** bobh has joined #openstack-infra | 19:16 | |
| *** e0ne has joined #openstack-infra | 19:16 | |
| jlk | AJaeger: I don't think I had caught that | 19:21 |
| AJaeger | jlk: https://review.openstack.org/503751 - I split it off to move forward... | 19:22 |
| openstackgerrit | Merged openstack-infra/project-config master: Fix wheel/release https://review.openstack.org/513945 | 19:26 |
| *** spectr has joined #openstack-infra | 19:29 | |
| *** slaweq has joined #openstack-infra | 19:31 | |
| *** bobh has quit IRC | 19:32 | |
| fungi | i have ~500 lines of scrollback... anything which needs my urgent attention while i have a few minutes between weekend chores? | 19:34 |
| fungi | fwiw, if the tripleo failures are due to some distro releasing new broken packages i don't think image rollbacks are really an appropriate workaround, considering the message to users in general then is to not upgrade their packages. if jobs need workarounds, they should either shift focus to help fix those broken packages or be employing the same solutions as all those poor schmucks whose clouds | 19:41 |
| fungi | just broke because of lax qa on distro packages | 19:41 |
| clarkb | fungi: I don't think there is anything urgent | 19:43 |
| clarkb | were there changes to statsd reporting in zuul recently? I don't think http://logs.openstack.org/50/513850/3/check/tox-py35/4f8140d/job-output.txt.gz#_2017-10-21_19_10_05_267890 is related to my change | 19:46 |
| clarkb | I;ve rechecked it to see if it is consistent but need to get back to the weekend | 19:48 |
| dmsimard | clarkb: if you're still around, could you +A https://review.openstack.org/#/c/513957/ ? Otherwise we don't have multinode coverage on zuul-jobs | 20:03 |
| *** jamesmcarthur has joined #openstack-infra | 20:05 | |
| *** thorst has joined #openstack-infra | 20:07 | |
| *** jamesmcarthur has quit IRC | 20:09 | |
| *** thorst has quit IRC | 20:11 | |
| *** e0ne has quit IRC | 20:12 | |
| *** nicolasbock has joined #openstack-infra | 20:22 | |
| *** rhallisey has quit IRC | 20:23 | |
| *** d0ugal has quit IRC | 20:25 | |
| *** spectr has quit IRC | 20:37 | |
| *** d0ugal has joined #openstack-infra | 20:37 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add multinode-integration jobs to zuul-jobs https://review.openstack.org/513957 | 20:55 |
| *** hasharAway has quit IRC | 20:58 | |
| *** edmondsw has joined #openstack-infra | 21:01 | |
| *** edmondsw has quit IRC | 21:05 | |
| *** hongbin has joined #openstack-infra | 21:12 | |
| smcginnis | If anyone's around or checks in, would love to see https://review.openstack.org/#/c/513864/ go through. I would then try another release to see if we've cleared everything up. | 21:13 |
| *** felipemonteiro has joined #openstack-infra | 21:26 | |
| openstackgerrit | Merged openstack-infra/project-config master: move git configuration for release jobs to ansible tasks https://review.openstack.org/513864 | 21:32 |
| openstackgerrit | Merged openstack-infra/project-config master: Remove now redundant git review setup https://review.openstack.org/513908 | 21:32 |
| tonyb | mordred: Thanks. I'll check out the tox-siblings work | 21:41 |
| *** nicolasbock has quit IRC | 21:46 | |
| *** hongbin_ has joined #openstack-infra | 21:47 | |
| *** hongbin__ has joined #openstack-infra | 21:49 | |
| *** hongbin has quit IRC | 21:49 | |
| *** hongbin_ has quit IRC | 21:51 | |
| *** leyal has quit IRC | 22:03 | |
| *** leyal has joined #openstack-infra | 22:03 | |
| *** dimak has quit IRC | 22:04 | |
| *** lihi has quit IRC | 22:04 | |
| *** markvoelker has joined #openstack-infra | 22:04 | |
| *** dimak has joined #openstack-infra | 22:04 | |
| *** lihi has joined #openstack-infra | 22:04 | |
| openstackgerrit | Merged openstack/os-client-config master: Add shade-tox-tips jobs https://review.openstack.org/513915 | 22:04 |
| openstackgerrit | Merged openstack/os-client-config master: Added nat_source flag for networks. https://review.openstack.org/513751 | 22:04 |
| openstackgerrit | Merged openstack/os-client-config master: Fix doc typo https://review.openstack.org/513836 | 22:04 |
| *** thorst has joined #openstack-infra | 22:08 | |
| *** felipemonteiro has quit IRC | 22:08 | |
| smcginnis | Release-test job was a success. Released, announced, and proposed to requirements. | 22:12 |
| *** thorst has quit IRC | 22:12 | |
| *** yamahata has quit IRC | 22:13 | |
| *** jbadiapa has quit IRC | 22:15 | |
| tonyb | smcginnis: \o/ | 22:16 |
| *** jbadiapa has joined #openstack-infra | 22:31 | |
| *** slaweq has quit IRC | 22:37 | |
| *** hongbin__ has quit IRC | 22:39 | |
| *** markvoelker has quit IRC | 22:48 | |
| *** edmondsw has joined #openstack-infra | 22:48 | |
| *** markvoelker has joined #openstack-infra | 22:53 | |
| *** edmondsw has quit IRC | 22:53 | |
| *** jamesmcarthur has joined #openstack-infra | 23:05 | |
| *** markvoelker has quit IRC | 23:06 | |
| *** thorst has joined #openstack-infra | 23:08 | |
| *** thorst has quit IRC | 23:09 | |
| *** jamesmcarthur has quit IRC | 23:09 | |
| *** slaweq has joined #openstack-infra | 23:10 | |
| mordred | smcginnis, tonyb: woot | 23:11 |
| *** armaan has quit IRC | 23:12 | |
| inc0 | hey guys, I might need some help with debugging jobs | 23:14 |
| inc0 | https://review.openstack.org/#/c/512779/ <- this thing - galera can't build cluster because messages times out | 23:15 |
| inc0 | I used these multinode playbooks that's supposed tu build tunnels, but didn't help | 23:15 |
| mordred | inc0: I don't see multinode in that one - is that in a previous patchset? | 23:20 |
| mordred | ah - I see whatyou're doing - one sec | 23:20 |
| inc0 | ceph one should be multinode | 23:20 |
| inc0 | thanks mordred | 23:20 |
| SamYaple | inc0: but you arent using the tunnels | 23:23 |
| SamYaple | inc0: the logs show you trying to connect to the public ip address | 23:23 |
| inc0 | 172.24.4.0/24 is tunnel | 23:24 |
| inc0 | actually it's 172.24.4.0/23 I believe | 23:24 |
| inc0 | https://github.com/openstack-infra/zuul-jobs/blob/master/roles/multi-node-bridge/defaults/main.yaml#L6 | 23:24 |
| SamYaple | ah i see your point | 23:25 |
| SamYaple | lets see what configs youre dropping | 23:25 |
| inc0 | http://logs.openstack.org/79/512779/13/check/kolla-ansible-ubuntu-source-ceph/d609bd6/secondary1/logs/kolla_configs/mariadb/galera.cnf | 23:26 |
| inc0 | fyi without tunnels I had same issue - that's why I wanted to do tunnels in the first place | 23:26 |
| SamYaple | well you should definetely be doing the tunnels, i agree | 23:27 |
| inc0 | and tunnels I do | 23:27 |
| inc0 | now when I look at it, tunnel mtu is 1450, which can be issue if nodepool uses vxlan too | 23:28 |
| inc0 | let's try lowering it to 1400 | 23:28 |
| SamYaple | inc0: http://logs.openstack.org/79/512779/13/check/kolla-ansible-ubuntu-source-ceph/d609bd6/primary/logs/kolla/mariadb/mariadb.txt.gz | 23:28 |
| SamYaple | thats showing your issue | 23:28 |
| SamYaple | looks like multinode mariadb bootstrap might be wonky | 23:29 |
| SamYaple | any recent changes? | 23:29 |
| inc0 | not really | 23:29 |
| inc0 | to me it looks like it can't build cluster because everything times out | 23:29 |
| SamYaple | well thats just it though, the *first* node should be started just fine | 23:30 |
| SamYaple | because it doesnt connect to anyhtign else | 23:30 |
| SamYaple | but thats failing because its trying to connect to other things | 23:30 |
| inc0 | well it tries to connect to 2 and 3....but you're right it shouldn | 23:30 |
| SamYaple | inc0: you know in the #kolla channel there were alot of complaints about getting past mariadb for 5.0.0 release multinode | 23:32 |
| SamYaple | inc0: are you sure its not busted? | 23:32 |
| inc0 | well I deployed 5.0 many times now... but idk really | 23:32 |
| inc0 | we had multinode before zuulv3 and it worked | 23:32 |
| inc0 | it was 2 node, but well | 23:33 |
| SamYaple | ok well im jsut saying, its popped up alot in the channel | 23:33 |
| SamYaple | you guys did a massive rework of the mariadb bootstrap stuff right before 5.0.0 | 23:33 |
| inc0 | yeah, I know, although I've seen rabbitmq more | 23:33 |
| *** tosky has quit IRC | 23:34 | |
| inc0 | not really...I mean besides become | 23:34 |
| inc0 | but we run become here on all tasks | 23:34 |
| SamYaple | inc0: i was talkign about this https://github.com/openstack/kolla-ansible/commit/5cd55bf236adc4c271418bda7306ec2336fa8816 | 23:35 |
| SamYaple | that was a few days before 5.0.0 | 23:35 |
| inc0 | hmm, well let's try to rever it then | 23:37 |
| SamYaple | looks like youd have to revert 3 patches in total to rollback that far | 23:39 |
| SamYaple | inc0: when was the last time you did a from-scratch 3 node deploy? | 23:39 |
| SamYaple | (by hand) | 23:39 |
| inc0 | can't remember, few weeks ago, but I'll try it | 23:39 |
| *** slaweq has quit IRC | 23:41 | |
| * mordred hands SamYaple a pie | 23:43 | |
| SamYaple | mordred: fun fact, i am known as sampie around my office because of my love of pies | 23:44 |
| mordred | well! I handed you the right thing then | 23:45 |
| SamYaple | :) | 23:45 |
| *** slaweq has joined #openstack-infra | 23:45 | |
| inc0 | not sure tho, I mean we did have multinode gates:/ | 23:46 |
| * inc0 sad | 23:46 | |
| inc0 | anyway, I kicked off fresh local master deploy | 23:46 |
| SamYaple | inc0: that patch might be fine, but maybe this 3 node gate is exposing a bug that your users have been hitting. eitherway, its good news (i think) | 23:46 |
| inc0 | well I'll try and see if I can reproduce it locally | 23:47 |
| SamYaple | yea good first step. agreed | 23:47 |
| *** bobh has joined #openstack-infra | 23:49 | |
| *** bobh has quit IRC | 23:55 | |
| pabelanger | interesting | 23:56 |
| pabelanger | I had a type in a playbook, for the wrong host, zuul couldn't run the playbook, and my job passed: http://logs.openstack.org/62/513962/3/check/ansible-role-shade-ubuntu-xenial/2128b00/job-output.txt.gz#_2017-10-21_20_20_29_561915 | 23:56 |
| pabelanger | typo | 23:56 |
| pabelanger | wouldn't have expected -1 on that run | 23:56 |
| pabelanger | will check executor why | 23:57 |
| pabelanger | 2017-10-21 20:20:29,572 DEBUG zuul.AnsibleJob: [build: 2128b003ba1847d4b4b3b89f3a5baa5d] Ansible output: b'skipping: no hosts matched' | 23:57 |
| pabelanger | ansible exit code 0 | 23:58 |
| clarkb | yes we abuse that behavior in d-g | 23:58 |
| pabelanger | ah | 23:58 |
| clarkb | I think there is a new flag to make it error that zuul can use | 23:58 |
| *** bobh has joined #openstack-infra | 23:58 | |
| pabelanger | cool | 23:58 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!