| *** marst_ has quit IRC | 00:02 | |
| *** john-davidge has quit IRC | 00:02 | |
| *** wolverineav has quit IRC | 00:04 | |
| *** salv-orlando has quit IRC | 00:09 | |
| *** esikachev has joined #openstack-infra | 00:11 | |
| *** aeng has joined #openstack-infra | 00:12 | |
| *** beslemon_ has quit IRC | 00:16 | |
| *** esikachev has quit IRC | 00:18 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config master: layout/tripleo: improve scenarios coverage for glance https://review.openstack.org/445250 | 00:19 |
|---|---|---|
| *** armax has quit IRC | 00:20 | |
| *** marst has joined #openstack-infra | 00:20 | |
| *** sbezverk has quit IRC | 00:24 | |
| *** smarcet has quit IRC | 00:26 | |
| *** dingyichen has joined #openstack-infra | 00:26 | |
| *** yujunz has joined #openstack-infra | 00:33 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Use sphinx warning-is-error https://review.openstack.org/445254 | 00:35 |
| *** Julien-zte has joined #openstack-infra | 00:38 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Use sphinx warning-is-error https://review.openstack.org/445254 | 00:40 |
| *** esikachev has joined #openstack-infra | 00:40 | |
| *** wolverineav has joined #openstack-infra | 00:42 | |
| *** esikachev has quit IRC | 00:45 | |
| *** funzo has joined #openstack-infra | 00:46 | |
| *** funzo has quit IRC | 00:50 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Use sphinx warning-is-error https://review.openstack.org/445254 | 00:57 |
| *** sdague has quit IRC | 00:57 | |
| *** jamielennox is now known as jamielennox|away | 00:58 | |
| *** rhallisey has quit IRC | 00:59 | |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/infra-specs master: Zuulv3 Launcher Security Enhancement https://review.openstack.org/444495 | 01:03 |
| *** smarcet has joined #openstack-infra | 01:04 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: Updated cache evict policy on data updates https://review.openstack.org/445261 | 01:05 |
| openstackgerrit | Merged openstack-infra/openstackid-resources master: Updated cache evict policy on data updates https://review.openstack.org/445261 | 01:08 |
| *** cody-somerville has quit IRC | 01:09 | |
| *** cody-somerville has joined #openstack-infra | 01:09 | |
| *** cody-somerville has joined #openstack-infra | 01:09 | |
| *** smarcet has left #openstack-infra | 01:10 | |
| *** jamielennox|away is now known as jamielennox | 01:12 | |
| *** esikachev has joined #openstack-infra | 01:14 | |
| *** tuanluong has joined #openstack-infra | 01:17 | |
| openstackgerrit | fumihiko kakuma proposed openstack-infra/project-config master: Add neutron job with ryu master https://review.openstack.org/445262 | 01:18 |
| *** esikachev has quit IRC | 01:18 | |
| *** Apoorva has quit IRC | 01:19 | |
| openstackgerrit | Merged openstack/diskimage-builder master: blockdevice.py: python3 fixes https://review.openstack.org/444758 | 01:22 |
| *** cody-somerville has quit IRC | 01:23 | |
| *** Sukhdev has quit IRC | 01:26 | |
| *** dhinesh has quit IRC | 01:26 | |
| *** dhinesh has joined #openstack-infra | 01:26 | |
| *** wolverineav has quit IRC | 01:30 | |
| *** zhurong has joined #openstack-infra | 01:31 | |
| *** esikachev has joined #openstack-infra | 01:31 | |
| *** sbezverk has joined #openstack-infra | 01:34 | |
| *** sbezverk is now known as Guest69780 | 01:34 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Make our virtualenv source py3 safe https://review.openstack.org/444715 | 01:35 |
| openstackgerrit | Merged openstack/diskimage-builder master: squash-package-install to use the correct python https://review.openstack.org/443336 | 01:35 |
| *** Guest69780 is now known as sbezverk_ | 01:36 | |
| *** yanyanhu has joined #openstack-infra | 01:36 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: Install packages with roles in multinode https://review.openstack.org/443687 | 01:39 |
| *** wolverineav has joined #openstack-infra | 01:41 | |
| *** yamamoto has joined #openstack-infra | 01:43 | |
| *** esikachev has quit IRC | 01:44 | |
| *** wolverineav has quit IRC | 01:46 | |
| *** funzo has joined #openstack-infra | 01:47 | |
| *** esikachev has joined #openstack-infra | 01:48 | |
| *** liujiong has joined #openstack-infra | 01:48 | |
| *** knangia has quit IRC | 01:51 | |
| *** funzo has quit IRC | 01:51 | |
| *** esikachev has quit IRC | 01:53 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: DONT REVIEW: testing new ovb oooq on branches https://review.openstack.org/445227 | 01:56 |
| *** sshnaidm is now known as sshnaidm|off | 01:56 | |
| *** zz_dimtruck is now known as dimtruck | 01:57 | |
| *** john-davidge has joined #openstack-infra | 01:58 | |
| *** esikachev has joined #openstack-infra | 01:59 | |
| *** hongbin has joined #openstack-infra | 02:00 | |
| *** tphummel has quit IRC | 02:00 | |
| *** wolverineav has joined #openstack-infra | 02:02 | |
| *** john-davidge has quit IRC | 02:03 | |
| *** esikachev has quit IRC | 02:03 | |
| *** wolverineav has quit IRC | 02:06 | |
| *** esikachev has joined #openstack-infra | 02:08 | |
| *** iOx has joined #openstack-infra | 02:10 | |
| *** wolverineav has joined #openstack-infra | 02:11 | |
| rm_work | ianw: thanks for the help. this looks good! | 02:12 |
| *** esikachev has quit IRC | 02:12 | |
| *** iOx has quit IRC | 02:13 | |
| *** wolverineav has quit IRC | 02:16 | |
| *** esikachev has joined #openstack-infra | 02:17 | |
| *** wolverineav has joined #openstack-infra | 02:21 | |
| *** esikachev has quit IRC | 02:21 | |
| *** wolverineav has quit IRC | 02:25 | |
| *** esikachev has joined #openstack-infra | 02:29 | |
| *** zzzeek has quit IRC | 02:30 | |
| *** zzzeek has joined #openstack-infra | 02:31 | |
| *** dhinesh has quit IRC | 02:33 | |
| *** gouthamr has quit IRC | 02:33 | |
| *** esikachev has quit IRC | 02:34 | |
| *** esikachev has joined #openstack-infra | 02:36 | |
| *** esikachev has quit IRC | 02:40 | |
| *** gongysh has joined #openstack-infra | 02:40 | |
| *** yamahata has quit IRC | 02:42 | |
| *** baoli has joined #openstack-infra | 02:43 | |
| ianw | rm_work: cool, yeah it seems to work | 02:43 |
| rm_work | yep, hoping to merge that tomorrow | 02:44 |
| adriant | notmorgan, yeah, subtree_as_list does not appear to actually work... :( | 02:44 |
| rm_work | of course now it's harder to prove since the fix merged in DIB | 02:44 |
| adriant | ugh... wrong channel | 02:44 |
| rm_work | oh huh, maybe it is still possible to prove somehow, wut | 02:44 |
| rm_work | or something else is broken in neutron-lbaas config | 02:45 |
| rm_work | yeah anyway, thanks :) | 02:45 |
| ianw | rm_work: oh well the prior change did show the install coming from git. i'm thinking on how we can improve dib python3 coverage for more than unit tests ... it was in the wrapping that isn't unit-tested we hit the issues | 02:45 |
| *** hongbin has quit IRC | 02:46 | |
| *** hongbin has joined #openstack-infra | 02:48 | |
| *** wolverineav has joined #openstack-infra | 02:51 | |
| *** masber has quit IRC | 02:51 | |
| *** masber has joined #openstack-infra | 02:51 | |
| *** cody-somerville has joined #openstack-infra | 02:52 | |
| *** MasterOfBugs has quit IRC | 02:54 | |
| *** wolverineav has quit IRC | 02:55 | |
| *** wolverineav has joined #openstack-infra | 03:01 | |
| *** john-davidge has joined #openstack-infra | 03:01 | |
| *** beslemon has joined #openstack-infra | 03:01 | |
| *** yujunz has quit IRC | 03:02 | |
| *** BharatK has joined #openstack-infra | 03:04 | |
| *** wolverineav has quit IRC | 03:05 | |
| *** john-davidge has quit IRC | 03:06 | |
| *** yujunz has joined #openstack-infra | 03:09 | |
| *** salv-orlando has joined #openstack-infra | 03:10 | |
| *** priteau has joined #openstack-infra | 03:11 | |
| *** ihrachys has joined #openstack-infra | 03:12 | |
| *** nicolasbock has quit IRC | 03:13 | |
| *** salv-orlando has quit IRC | 03:15 | |
| *** priteau has quit IRC | 03:15 | |
| *** cody-somerville has quit IRC | 03:15 | |
| *** ihrachys has quit IRC | 03:17 | |
| *** esikachev has joined #openstack-infra | 03:17 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci master: Quickstart transition: replace toci_gate_test.sh https://review.openstack.org/431029 | 03:20 |
| *** wolverineav has joined #openstack-infra | 03:20 | |
| *** esikachev has quit IRC | 03:22 | |
| *** ihrachys has joined #openstack-infra | 03:23 | |
| *** wolverineav has quit IRC | 03:24 | |
| *** esikachev has joined #openstack-infra | 03:26 | |
| *** sbezverk has joined #openstack-infra | 03:29 | |
| *** esikachev has quit IRC | 03:30 | |
| *** sbezverk has quit IRC | 03:33 | |
| *** dhinesh has joined #openstack-infra | 03:33 | |
| openstackgerrit | Ihar Hrachyshka proposed openstack-infra/devstack-gate master: Switch from lib/neutron-legacy to lib/neutron https://review.openstack.org/436798 | 03:35 |
| *** stakeda has quit IRC | 03:35 | |
| *** ihrachys has quit IRC | 03:44 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Use sphinx warning-is-error https://review.openstack.org/445254 | 03:45 |
| *** BharatK_ has joined #openstack-infra | 03:46 | |
| *** BharatK has quit IRC | 03:46 | |
| *** Julien-zte has quit IRC | 03:48 | |
| *** Julien-zte has joined #openstack-infra | 03:48 | |
| *** funzo has joined #openstack-infra | 03:48 | |
| *** wolverineav has joined #openstack-infra | 03:49 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Use sphinx warning-is-error https://review.openstack.org/445254 | 03:50 |
| *** funzo has quit IRC | 03:53 | |
| *** esikachev has joined #openstack-infra | 03:53 | |
| *** nmathew has joined #openstack-infra | 03:53 | |
| *** wolverineav has quit IRC | 03:54 | |
| *** Sukhdev has joined #openstack-infra | 03:56 | |
| *** esikachev has quit IRC | 03:57 | |
| *** hongbin has quit IRC | 03:59 | |
| *** yujunz has quit IRC | 04:01 | |
| *** sree has joined #openstack-infra | 04:01 | |
| *** sree has quit IRC | 04:01 | |
| *** sree has joined #openstack-infra | 04:02 | |
| *** esikachev has joined #openstack-infra | 04:04 | |
| *** gildub has quit IRC | 04:04 | |
| *** baoli has quit IRC | 04:04 | |
| *** rlandy has quit IRC | 04:04 | |
| jlvillal | Is something wrong with Gerrit? | 04:06 |
| jlvillal | I'm seeing 500 server errors | 04:06 |
| loquacities | yep, same here | 04:06 |
| jlvillal | Any infra folks around? | 04:07 |
| jlvillal | clarkb, In case you are still awake/online ^^^ | 04:08 |
| *** asselin has quit IRC | 04:08 | |
| *** esikachev has quit IRC | 04:09 | |
| *** rabel_b1 has joined #openstack-infra | 04:12 | |
| rabel_b1 | hello. seems that review.openstack.org is having problems again | 04:13 |
| *** ihrachys has joined #openstack-infra | 04:15 | |
| Jeffrey4l | seem error here. | 04:15 |
| *** rabel__b1 has joined #openstack-infra | 04:16 | |
| Jeffrey4l | infra-root around? | 04:16 |
| jhesketh | yep | 04:16 |
| jhesketh | just taking a look at gerrit | 04:17 |
| gmann | jhesketh: thanks | 04:17 |
| *** sbezverk_ has quit IRC | 04:19 | |
| *** wolverineav has joined #openstack-infra | 04:19 | |
| *** rabel_b1 has quit IRC | 04:19 | |
| jhesketh | Jeffrey4l, gmann: can you confirm it's back up for you please | 04:21 |
| Jeffrey4l | jhesketh, gerrit is alive. thanks. | 04:21 |
| jhesketh | no worries | 04:21 |
| gmann | jhesketh: its up, thanks for quick help. | 04:22 |
| jhesketh | #status log gerrit under load throwing 503 errors. Service restart fixed symptoms and appears to be running smoothly | 04:22 |
| openstackstatus | jhesketh: finished logging | 04:22 |
| *** gildub has joined #openstack-infra | 04:22 | |
| *** wolverineav has quit IRC | 04:24 | |
| *** psachin has joined #openstack-infra | 04:25 | |
| *** ihrachys has quit IRC | 04:25 | |
| *** Sukhdev has quit IRC | 04:28 | |
| *** wolverineav has joined #openstack-infra | 04:29 | |
| *** esikachev has joined #openstack-infra | 04:30 | |
| *** wolverineav has quit IRC | 04:33 | |
| *** esikachev has quit IRC | 04:34 | |
| *** links has joined #openstack-infra | 04:35 | |
| *** ihrachys has joined #openstack-infra | 04:36 | |
| *** esikachev has joined #openstack-infra | 04:39 | |
| *** gongysh has quit IRC | 04:40 | |
| *** wolverineav has joined #openstack-infra | 04:40 | |
| *** wolverineav has quit IRC | 04:45 | |
| *** esikachev has quit IRC | 04:46 | |
| *** cody-somerville has joined #openstack-infra | 04:47 | |
| *** MasterOfBugs has joined #openstack-infra | 04:47 | |
| *** funzo has joined #openstack-infra | 04:49 | |
| *** gongysh has joined #openstack-infra | 04:53 | |
| *** funzo has quit IRC | 04:54 | |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Support for dependent pipelines with github https://review.openstack.org/445292 | 04:55 |
| *** gongysh has quit IRC | 04:56 | |
| *** knangia has joined #openstack-infra | 04:59 | |
| openstackgerrit | Ihar Hrachyshka proposed openstack-infra/devstack-gate master: Switch from lib/neutron-legacy to lib/neutron https://review.openstack.org/436798 | 05:00 |
| openstackgerrit | Chandan Kumar proposed openstack-infra/tripleo-ci master: DON'T MERGE: test tempest all branches https://review.openstack.org/436923 | 05:01 |
| *** john-davidge has joined #openstack-infra | 05:02 | |
| *** vsaienko has quit IRC | 05:04 | |
| openstackgerrit | Ihar Hrachyshka proposed openstack-infra/devstack-gate master: Switch from lib/neutron-legacy to lib/neutron https://review.openstack.org/436798 | 05:04 |
| *** john-davidge has quit IRC | 05:06 | |
| *** salv-orlando has joined #openstack-infra | 05:11 | |
| *** dimtruck is now known as zz_dimtruck | 05:12 | |
| openstackgerrit | Ihar Hrachyshka proposed openstack-infra/devstack-gate master: Switch from lib/neutron-legacy to lib/neutron https://review.openstack.org/436798 | 05:12 |
| openstackgerrit | Ihar Hrachyshka proposed openstack-infra/devstack-gate master: Switch from lib/neutron-legacy to lib/neutron https://review.openstack.org/436798 | 05:13 |
| *** esikachev has joined #openstack-infra | 05:14 | |
| *** salv-orlando has quit IRC | 05:15 | |
| openstackgerrit | Jamie Lennox proposed openstack-infra/nodepool feature/zuulv3: Don't try and delete nodes with no external_id https://review.openstack.org/445299 | 05:20 |
| *** eroux has joined #openstack-infra | 05:22 | |
| *** Julien-zte has quit IRC | 05:24 | |
| *** Julien-zte has joined #openstack-infra | 05:25 | |
| *** rabel__b1 has quit IRC | 05:27 | |
| *** yujunz has joined #openstack-infra | 05:31 | |
| *** prateek has joined #openstack-infra | 05:33 | |
| *** BharatK__ has joined #openstack-infra | 05:35 | |
| *** vikrant has joined #openstack-infra | 05:36 | |
| *** BharatK_ has quit IRC | 05:37 | |
| openstackgerrit | Varun Gadiraju proposed openstack-infra/project-config master: Run 'enable_plugin' in project-config for grenade jobs https://review.openstack.org/443348 | 05:40 |
| *** pgadiya has joined #openstack-infra | 05:42 | |
| *** csomerville has joined #openstack-infra | 05:49 | |
| *** funzo has joined #openstack-infra | 05:50 | |
| *** gongysh has joined #openstack-infra | 05:51 | |
| *** claudiub has joined #openstack-infra | 05:51 | |
| *** cody-somerville has quit IRC | 05:52 | |
| *** pahuang has quit IRC | 05:52 | |
| *** yujunz has quit IRC | 05:52 | |
| *** funzo has quit IRC | 05:55 | |
| *** ihrachys has quit IRC | 05:56 | |
| *** yujunz has joined #openstack-infra | 05:57 | |
| *** adriant has quit IRC | 06:11 | |
| *** salv-orlando has joined #openstack-infra | 06:12 | |
| *** pahuang has joined #openstack-infra | 06:14 | |
| *** yamahata has joined #openstack-infra | 06:15 | |
| *** prateek has quit IRC | 06:15 | |
| *** prateek has joined #openstack-infra | 06:16 | |
| *** salv-orlando has quit IRC | 06:17 | |
| *** nadya has joined #openstack-infra | 06:17 | |
| *** eroux has quit IRC | 06:25 | |
| *** salv-orlando has joined #openstack-infra | 06:26 | |
| openstackgerrit | Merged openstack-infra/project-config master: Revert "Mark Oslo backward compat jobs as non-voting" https://review.openstack.org/444922 | 06:26 |
| openstackgerrit | Rui Chen proposed openstack-infra/release-tools master: Add --stop option for bugs-fixed-since https://review.openstack.org/445315 | 06:27 |
| *** david-lyle has quit IRC | 06:27 | |
| *** beslemon has quit IRC | 06:28 | |
| *** beslemon has joined #openstack-infra | 06:29 | |
| *** anilvenkata has joined #openstack-infra | 06:30 | |
| anilvenkata | devstack-gate cores, this patch https://review.openstack.org/#/c/383827/ is in review for a long time and neutron reviewers are actively reviewing it, can devstack-gate reviewers also please review it? | 06:31 |
| *** beslemon has quit IRC | 06:34 | |
| *** RuiChen has joined #openstack-infra | 06:35 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add barbican-tempest-plugin tests to Nova https://review.openstack.org/443167 | 06:35 |
| *** faizy has joined #openstack-infra | 06:35 | |
| *** vsaienko has joined #openstack-infra | 06:36 | |
| *** dhinesh has quit IRC | 06:39 | |
| *** nadya has quit IRC | 06:40 | |
| *** vsaienko has quit IRC | 06:42 | |
| *** vsaienko has joined #openstack-infra | 06:44 | |
| *** andreas_s has joined #openstack-infra | 06:47 | |
| *** gyee has joined #openstack-infra | 06:49 | |
| *** gyee has quit IRC | 06:50 | |
| *** wolverineav has joined #openstack-infra | 06:50 | |
| *** funzo has joined #openstack-infra | 06:50 | |
| *** e0ne has joined #openstack-infra | 06:51 | |
| *** yujunz has quit IRC | 06:54 | |
| *** funzo has quit IRC | 06:55 | |
| *** wolverineav has quit IRC | 06:55 | |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/nodepool feature/zuulv3: Merge branch 'master' into feature/zuulv3 https://review.openstack.org/445325 | 06:59 |
| *** BharatK__ has quit IRC | 06:59 | |
| *** gildub has quit IRC | 07:01 | |
| *** john-davidge has joined #openstack-infra | 07:03 | |
| *** gildub has joined #openstack-infra | 07:05 | |
| *** john-davidge has quit IRC | 07:08 | |
| *** alexchadin has joined #openstack-infra | 07:10 | |
| *** wolverineav has joined #openstack-infra | 07:10 | |
| *** wolverineav has quit IRC | 07:15 | |
| *** yujunz has joined #openstack-infra | 07:15 | |
| *** mriedem has quit IRC | 07:17 | |
| vsaienk0 | AJaeger: yolanda: please add to your review queue https://review.openstack.org/#/c/443348/ small patch that removes workarounds around enable_plugin ironic. Thanks in advance! | 07:17 |
| *** tesseract has joined #openstack-infra | 07:20 | |
| *** knangia has quit IRC | 07:21 | |
| *** dhajare has joined #openstack-infra | 07:23 | |
| *** salv-orl_ has joined #openstack-infra | 07:23 | |
| *** salv-orlando has quit IRC | 07:26 | |
| *** yuval has joined #openstack-infra | 07:27 | |
| *** e0ne has quit IRC | 07:28 | |
| *** wolverineav has joined #openstack-infra | 07:30 | |
| *** slaweq__ has quit IRC | 07:33 | |
| *** YorikSar has quit IRC | 07:33 | |
| *** slaweq has joined #openstack-infra | 07:34 | |
| *** wolverineav has quit IRC | 07:34 | |
| *** YorikSar has joined #openstack-infra | 07:35 | |
| *** BharatK has joined #openstack-infra | 07:39 | |
| openstackgerrit | yatin proposed openstack-infra/project-config master: [magnum] Fix multinode setup https://review.openstack.org/445339 | 07:40 |
| *** markus_z has joined #openstack-infra | 07:45 | |
| *** stakeda has joined #openstack-infra | 07:46 | |
| *** MasterOfBugs has quit IRC | 07:48 | |
| openstackgerrit | Andreas Florath proposed openstack/diskimage-builder master: Refactor: block-device filesystem creation, mount and fstab https://review.openstack.org/444586 | 07:49 |
| *** wolverineav has joined #openstack-infra | 07:49 | |
| *** yujunz-zte has joined #openstack-infra | 07:50 | |
| *** pcaruana has joined #openstack-infra | 07:50 | |
| *** priteau has joined #openstack-infra | 07:51 | |
| *** matrohon has joined #openstack-infra | 07:52 | |
| *** yujunz has quit IRC | 07:52 | |
| openstackgerrit | Shu Muto proposed openstack-infra/infra-manual master: Add UI Cookiecutter into template selections https://review.openstack.org/444254 | 07:53 |
| *** wolverineav has quit IRC | 07:54 | |
| *** claudiub has quit IRC | 07:55 | |
| *** zzzeek has quit IRC | 08:00 | |
| *** wolverineav has joined #openstack-infra | 08:00 | |
| *** zzzeek has joined #openstack-infra | 08:00 | |
| *** isaacb has joined #openstack-infra | 08:01 | |
| *** yujunz-zte has quit IRC | 08:02 | |
| *** nadya has joined #openstack-infra | 08:03 | |
| *** shardy has joined #openstack-infra | 08:04 | |
| *** wolverineav has quit IRC | 08:04 | |
| *** garciadeblas has joined #openstack-infra | 08:08 | |
| *** wolverineav has joined #openstack-infra | 08:09 | |
| *** salv-orl_ has quit IRC | 08:10 | |
| *** hashar has joined #openstack-infra | 08:13 | |
| *** wolverineav has quit IRC | 08:14 | |
| *** florianf has joined #openstack-infra | 08:19 | |
| *** wolverineav has joined #openstack-infra | 08:19 | |
| *** ccamacho has joined #openstack-infra | 08:19 | |
| *** wolverineav has quit IRC | 08:23 | |
| *** jaosorior has joined #openstack-infra | 08:24 | |
| *** dingyichen has quit IRC | 08:25 | |
| *** martinkopec has joined #openstack-infra | 08:25 | |
| *** dizquierdo has joined #openstack-infra | 08:26 | |
| openstackgerrit | Merged openstack-infra/project-config master: Adding two other combinations of cinder multinode grenade job https://review.openstack.org/384836 | 08:28 |
| *** kzaitsev_mb has joined #openstack-infra | 08:32 | |
| *** aarefiev_afk is now known as aarefiev | 08:32 | |
| *** sandanar has joined #openstack-infra | 08:34 | |
| *** furlongm has quit IRC | 08:35 | |
| *** yolanda has quit IRC | 08:37 | |
| *** wolverineav has joined #openstack-infra | 08:39 | |
| *** amoralej|off is now known as amoralej | 08:39 | |
| *** rabel__b1 has joined #openstack-infra | 08:42 | |
| *** yolanda has joined #openstack-infra | 08:42 | |
| *** wolverineav has quit IRC | 08:43 | |
| *** gongysh has quit IRC | 08:45 | |
| *** ssbarnea has joined #openstack-infra | 08:46 | |
| *** ralonsoh has joined #openstack-infra | 08:46 | |
| *** gildub has quit IRC | 08:47 | |
| *** e0ne has joined #openstack-infra | 08:50 | |
| *** gildub has joined #openstack-infra | 08:51 | |
| *** sshnaidm|off is now known as sshnaidm | 08:52 | |
| *** funzo has joined #openstack-infra | 08:52 | |
| *** salv-orlando has joined #openstack-infra | 08:54 | |
| *** shu-mutou-AWAY is now known as shu-mutou | 08:56 | |
| *** funzo has quit IRC | 08:56 | |
| *** jpich has joined #openstack-infra | 08:58 | |
| *** wolverineav has joined #openstack-infra | 08:59 | |
| *** pgadiya has quit IRC | 08:59 | |
| *** esikachev has quit IRC | 09:00 | |
| *** wolverineav has quit IRC | 09:03 | |
| *** john-davidge has joined #openstack-infra | 09:04 | |
| *** john-davidge has quit IRC | 09:08 | |
| *** chmouel_ is now known as chmouel | 09:08 | |
| *** wolverineav has joined #openstack-infra | 09:09 | |
| *** gongysh has joined #openstack-infra | 09:12 | |
| *** wolverineav has quit IRC | 09:13 | |
| *** BharatK_ has joined #openstack-infra | 09:13 | |
| *** lucas-afk is now known as lucasagomes | 09:15 | |
| *** pgadiya has joined #openstack-infra | 09:15 | |
| *** esikachev has joined #openstack-infra | 09:16 | |
| *** BharatK has quit IRC | 09:17 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: DONT MERGE: test tempest all branches https://review.openstack.org/436923 | 09:19 |
| *** yee379 has quit IRC | 09:20 | |
| *** yee379 has joined #openstack-infra | 09:20 | |
| *** esikachev has quit IRC | 09:20 | |
| *** bhavik1 has joined #openstack-infra | 09:25 | |
| cmurphy | o docs dot o y r u so slow :( | 09:29 |
| *** Julien-zte has quit IRC | 09:30 | |
| *** pblaho has quit IRC | 09:31 | |
| *** Julien-zte has joined #openstack-infra | 09:31 | |
| *** gongysh has quit IRC | 09:33 | |
| *** tosky has joined #openstack-infra | 09:34 | |
| *** wolverineav has joined #openstack-infra | 09:38 | |
| *** alexchadin has quit IRC | 09:39 | |
| frickler | yolanda: I can confirm docs.o.o being sluggish again, could you take a look at it? | 09:40 |
| *** derekh has joined #openstack-infra | 09:42 | |
| *** wolverineav has quit IRC | 09:42 | |
| *** john-davidge has joined #openstack-infra | 09:46 | |
| *** pblaho has joined #openstack-infra | 09:46 | |
| *** Julien-zte has quit IRC | 09:50 | |
| *** Julien-zte has joined #openstack-infra | 09:51 | |
| *** funzo has joined #openstack-infra | 09:53 | |
| *** zhurong has quit IRC | 09:54 | |
| *** funzo has quit IRC | 09:57 | |
| *** dtantsur|afk is now known as dtantsur | 09:59 | |
| *** gildub has quit IRC | 10:01 | |
| *** nadya has quit IRC | 10:01 | |
| *** Julien-zte has quit IRC | 10:03 | |
| *** Julien-zte has joined #openstack-infra | 10:04 | |
| *** nicolasbock has joined #openstack-infra | 10:04 | |
| *** lennyb has quit IRC | 10:06 | |
| openstackgerrit | Merged openstack-infra/irc-meetings master: Move the Mistral meeting one hour earlier https://review.openstack.org/445167 | 10:07 |
| *** Julien-zte has quit IRC | 10:07 | |
| *** wolverineav has joined #openstack-infra | 10:07 | |
| *** Julien-zte has joined #openstack-infra | 10:07 | |
| *** Julien-zte has quit IRC | 10:08 | |
| ianychoi | cmurphy, same as me. docs.o.o is slow.. | 10:08 |
| *** Julien-zte has joined #openstack-infra | 10:09 | |
| *** pblaho has quit IRC | 10:10 | |
| *** Julien-zte has quit IRC | 10:10 | |
| *** Julien-zte has joined #openstack-infra | 10:11 | |
| *** Julien-zte has quit IRC | 10:11 | |
| *** lennyb has joined #openstack-infra | 10:11 | |
| *** wolverineav has quit IRC | 10:12 | |
| *** yanyanhu has quit IRC | 10:14 | |
| *** salv-orlando has quit IRC | 10:18 | |
| *** Julien-zte has joined #openstack-infra | 10:20 | |
| *** liujiong has quit IRC | 10:21 | |
| *** ecksun has left #openstack-infra | 10:23 | |
| *** pblaho has joined #openstack-infra | 10:23 | |
| *** martinkopec has quit IRC | 10:26 | |
| ianychoi | Would it be also cinder read-only problem for slow or 404 docs.o.o pages? | 10:26 |
| *** wolverineav has joined #openstack-infra | 10:27 | |
| *** rabel__b1 has quit IRC | 10:31 | |
| *** wolverineav has quit IRC | 10:31 | |
| *** rabel__b1 has joined #openstack-infra | 10:32 | |
| *** rabel_b1 has joined #openstack-infra | 10:36 | |
| *** kelepirci has joined #openstack-infra | 10:37 | |
| kelepirci | hello all | 10:38 |
| *** rabel__b1 has quit IRC | 10:38 | |
| *** jkilpatr has quit IRC | 10:39 | |
| kelepirci | is there any problem with docs.openstack.org? | 10:39 |
| *** alexchadin has joined #openstack-infra | 10:40 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: Install tripleo-admin ssh keys on CI nodes https://review.openstack.org/363630 | 10:44 |
| *** nadya has joined #openstack-infra | 10:45 | |
| *** salv-orlando has joined #openstack-infra | 10:45 | |
| *** salv-orl_ has joined #openstack-infra | 10:46 | |
| *** nicolasbock has quit IRC | 10:46 | |
| *** askb has quit IRC | 10:46 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: Close testenv-client before zuul finishes https://review.openstack.org/410470 | 10:47 |
| *** BharatK_ has quit IRC | 10:47 | |
| *** kzaitsev_mb has quit IRC | 10:47 | |
| openstackgerrit | Shu Muto proposed openstack-infra/project-config master: Fix zun-ui nodejs jobs and change lint to voting https://review.openstack.org/445423 | 10:47 |
| *** sdague has joined #openstack-infra | 10:47 | |
| *** nicolasbock has joined #openstack-infra | 10:48 | |
| *** askb has joined #openstack-infra | 10:48 | |
| *** shu-mutou is now known as shu-mutou-AWAY | 10:49 | |
| *** salv-orlando has quit IRC | 10:49 | |
| *** BharatK_ has joined #openstack-infra | 10:50 | |
| *** salv-orl_ has quit IRC | 10:51 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: TEST: DONT RECHECK or REVIEW: periodic jobs https://review.openstack.org/359215 | 10:51 |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: Close testenv-client before zuul finishes https://review.openstack.org/410470 | 10:53 |
| *** funzo has joined #openstack-infra | 10:54 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: Install tripleo-admin ssh keys on CI nodes https://review.openstack.org/363630 | 10:55 |
| *** BharatK_ has quit IRC | 10:56 | |
| *** BharatK_ has joined #openstack-infra | 10:57 | |
| *** funzo has quit IRC | 10:58 | |
| *** isaacb has quit IRC | 11:00 | |
| *** rabel_b1 has quit IRC | 11:03 | |
| *** rabel has joined #openstack-infra | 11:05 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: Compress logs to tar.gz archive https://review.openstack.org/434143 | 11:07 |
| *** BharatK_ has quit IRC | 11:08 | |
| *** sree has quit IRC | 11:08 | |
| *** BharatK has joined #openstack-infra | 11:08 | |
| *** sree has joined #openstack-infra | 11:08 | |
| *** dizquierdo has quit IRC | 11:10 | |
| *** bhavik1 has quit IRC | 11:13 | |
| *** BharatK_ has joined #openstack-infra | 11:14 | |
| *** BharatK has quit IRC | 11:17 | |
| *** jkilpatr has joined #openstack-infra | 11:19 | |
| *** BharatK_ has quit IRC | 11:22 | |
| *** jkilpatr has quit IRC | 11:27 | |
| *** sree has quit IRC | 11:27 | |
| *** sree has joined #openstack-infra | 11:27 | |
| *** yuval has quit IRC | 11:27 | |
| *** sree has quit IRC | 11:28 | |
| *** sree has joined #openstack-infra | 11:29 | |
| *** sree has quit IRC | 11:33 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: WIP: Conntainers with multi-nic and net-iso https://review.openstack.org/422448 | 11:35 |
| *** links has quit IRC | 11:36 | |
| *** makowals_ has joined #openstack-infra | 11:39 | |
| *** jkilpatr has joined #openstack-infra | 11:40 | |
| *** faizy has quit IRC | 11:40 | |
| *** rhallisey has joined #openstack-infra | 11:41 | |
| *** makowals has quit IRC | 11:43 | |
| *** gordc has joined #openstack-infra | 11:43 | |
| *** makowals_ has quit IRC | 11:44 | |
| *** pgadiya has quit IRC | 11:45 | |
| *** nmathew has quit IRC | 11:46 | |
| *** isaacb has joined #openstack-infra | 11:48 | |
| *** links has joined #openstack-infra | 11:48 | |
| *** wolverineav has joined #openstack-infra | 11:52 | |
| *** funzo has joined #openstack-infra | 11:54 | |
| *** Julien-zte has quit IRC | 11:55 | |
| *** ldnunes has joined #openstack-infra | 11:56 | |
| *** wolverineav has quit IRC | 11:56 | |
| *** funzo has quit IRC | 11:58 | |
| *** dave-mccowan has joined #openstack-infra | 12:00 | |
| *** _Cyclone_ has joined #openstack-infra | 12:01 | |
| *** askb has quit IRC | 12:01 | |
| *** yuval has joined #openstack-infra | 12:04 | |
| *** tuanluong has quit IRC | 12:05 | |
| *** nadya has quit IRC | 12:05 | |
| *** yuval has quit IRC | 12:06 | |
| *** yuval has joined #openstack-infra | 12:07 | |
| *** aysyd has joined #openstack-infra | 12:07 | |
| *** nadya has joined #openstack-infra | 12:10 | |
| *** jcoufal has joined #openstack-infra | 12:11 | |
| *** rvba` has quit IRC | 12:11 | |
| *** hashar is now known as hasharLunch | 12:12 | |
| *** kzaitsev_mb has joined #openstack-infra | 12:13 | |
| *** dizquierdo has joined #openstack-infra | 12:15 | |
| *** trown|outtypewww is now known as trown | 12:15 | |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci master: [WIP] Quickstart transition: fallback to old method on stable branches https://review.openstack.org/445461 | 12:16 |
| *** rvba has joined #openstack-infra | 12:16 | |
| *** rvba has quit IRC | 12:17 | |
| *** rvba has joined #openstack-infra | 12:17 | |
| *** ociuhandu has quit IRC | 12:22 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: Install packages with roles in multinode https://review.openstack.org/443687 | 12:23 |
| *** faizy has joined #openstack-infra | 12:25 | |
| *** rfolco has joined #openstack-infra | 12:29 | |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci master: [WIP] quickstart transition: add promotion logic https://review.openstack.org/445467 | 12:29 |
| *** yuval has quit IRC | 12:31 | |
| *** yuval has joined #openstack-infra | 12:31 | |
| *** makowals has joined #openstack-infra | 12:32 | |
| *** nadya has quit IRC | 12:32 | |
| *** nadya has joined #openstack-infra | 12:32 | |
| *** faizy has quit IRC | 12:34 | |
| *** rlandy has joined #openstack-infra | 12:36 | |
| *** gongysh has joined #openstack-infra | 12:37 | |
| *** mdrabe has joined #openstack-infra | 12:38 | |
| *** csomerville has quit IRC | 12:41 | |
| *** isaacb has quit IRC | 12:43 | |
| *** isaacb has joined #openstack-infra | 12:44 | |
| *** wolverineav has joined #openstack-infra | 12:46 | |
| *** makowals_ has joined #openstack-infra | 12:48 | |
| *** dprince has joined #openstack-infra | 12:49 | |
| *** makowals has quit IRC | 12:50 | |
| *** kgiusti has joined #openstack-infra | 12:51 | |
| *** panda|zZ is now known as panda | 12:51 | |
| *** gouthamr has joined #openstack-infra | 12:51 | |
| *** links has quit IRC | 12:52 | |
| *** baoli has joined #openstack-infra | 12:52 | |
| *** links has joined #openstack-infra | 12:53 | |
| *** amoralej is now known as amoralej|lunch | 12:53 | |
| *** amotoki has joined #openstack-infra | 12:54 | |
| *** lucasagomes is now known as lucas-hungry | 12:55 | |
| *** funzo has joined #openstack-infra | 12:55 | |
| *** Goneri has joined #openstack-infra | 12:55 | |
| *** baoli has quit IRC | 12:55 | |
| *** baoli has joined #openstack-infra | 12:56 | |
| *** vikrant has quit IRC | 12:56 | |
| *** vikrant has joined #openstack-infra | 12:58 | |
| *** funzo has quit IRC | 12:59 | |
| *** baoli has quit IRC | 13:00 | |
| *** baoli has joined #openstack-infra | 13:02 | |
| *** Julien-zte has joined #openstack-infra | 13:03 | |
| *** prazumovsky has joined #openstack-infra | 13:03 | |
| *** gongysh has quit IRC | 13:03 | |
| *** anilvenkata is now known as anilvenkata_afk | 13:04 | |
| *** chlong has joined #openstack-infra | 13:05 | |
| *** ociuhandu has joined #openstack-infra | 13:06 | |
| *** kzaitsev_mb has quit IRC | 13:08 | |
| *** nadya has quit IRC | 13:11 | |
| *** markvoelker has quit IRC | 13:11 | |
| *** stakeda has quit IRC | 13:11 | |
| *** dizquierdo has quit IRC | 13:13 | |
| *** markvoelker has joined #openstack-infra | 13:16 | |
| *** claudiub has joined #openstack-infra | 13:18 | |
| *** krtaylor has joined #openstack-infra | 13:19 | |
| claudiub | hello infra people. can I get into the python-hnvclient core / release teams? https://review.openstack.org/#/admin/groups/1726,members project proposal patch: https://review.openstack.org/#/c/441810/ | 13:20 |
| *** trown is now known as trown|brb | 13:21 | |
| *** faizy has joined #openstack-infra | 13:21 | |
| *** prazumovsky has quit IRC | 13:23 | |
| *** prateek has quit IRC | 13:25 | |
| *** esikachev has joined #openstack-infra | 13:25 | |
| *** trown|brb is now known as trown | 13:27 | |
| dasm | o/ folks | 13:27 |
| dasm | i'm seeing strange behavior with gerrit | 13:27 |
| dasm | on My reviews page i have other results than when i'll go into details of patch | 13:28 |
| dasm | Patch which currently has -1 doesn't have it in details | 13:28 |
| dasm | anyone else sees this thing? | 13:28 |
| *** felipemonteiro has joined #openstack-infra | 13:29 | |
| dasm | oh. it's probably some kind of gerrit problem. | 13:29 |
| *** rabel has quit IRC | 13:29 | |
| dasm | on details page, i've briefly "Conflicts with: 500 error" | 13:29 |
| *** esikachev has quit IRC | 13:29 | |
| odyssey4me | dasm yeah, I'm getting rejections for uploading new patches | 13:30 |
| odyssey4me | git fetch --all | 13:30 |
| dasm | and now i have 502 proxy error | 13:30 |
| johnthetubaguy | same here | 13:30 |
| strigazi | same | 13:30 |
| dasm | i'm not alone \o/ | 13:30 |
| dasm | :) | 13:30 |
| *** thomasem has joined #openstack-infra | 13:31 | |
| strigazi | working again | 13:31 |
| *** mattmceuen has joined #openstack-infra | 13:32 | |
| *** esikachev has joined #openstack-infra | 13:32 | |
| odyssey4me | nope, not for me | 13:32 |
| odyssey4me | it seems that gerrit has given up the ghost | 13:32 |
| odyssey4me | a good time for me to lunch :p | 13:32 |
| strigazi | bon app | 13:33 |
| *** kzaitsev_mb has joined #openstack-infra | 13:34 | |
| * johnthetubaguy wishes he had not decided to spend this post lunch time doing reviews | 13:34 | |
| dasm | johnthetubaguy: go for a lunch again :) | 13:34 |
| johnthetubaguy | heh | 13:34 |
| thomasem | :( | 13:34 |
| strigazi | ok, now i get 502 and 500 | 13:34 |
| yolanda | hi, looking at that, i'll check and reboot | 13:35 |
| dasm | sounds like numbers for bingo | 13:35 |
| dasm | yolanda: thanks | 13:35 |
| *** mriedem has joined #openstack-infra | 13:35 | |
| thomasem | 502 for me, assuming we're talking about Gerrit | 13:35 |
| strigazi | yes | 13:35 |
| yolanda | yep, it needs a reboot | 13:35 |
| yolanda | restart | 13:35 |
| thomasem | So, I keep seeing reboots coming in for Gerrit performance issues and such. Is there any idea what the root cause of these consistent problems are? | 13:36 |
| yolanda | #status alert Gerrit is going to be restarted due to performance problems | 13:36 |
| openstackstatus | yolanda: sending alert | 13:36 |
| *** sree has joined #openstack-infra | 13:36 | |
| *** esikachev has quit IRC | 13:37 | |
| yolanda | this time is a java error: java.io.IOException: Too many open files | 13:37 |
| thomasem | Hmmm... ran out of descriptors? | 13:37 |
| yolanda | that's what the error suggests yes | 13:37 |
| *** kzaitsev_mb has quit IRC | 13:38 | |
| thomasem | Wonder if something's not cleaning up after itself? | 13:38 |
| thomasem | or timing out properly | 13:38 |
| -openstackstatus- NOTICE: Gerrit is going to be restarted due to performance problems | 13:39 | |
| yolanda | we have 4096 max open files on that server | 13:39 |
| *** ChanServ changes topic to "Gerrit is going to be restarted due to performance problems" | 13:39 | |
| pabelanger | yolanda: /var/log/apache2/error.log shows some interesting errors on review.o.o | 13:39 |
| pabelanger | we should likely look into that | 13:40 |
| yolanda | pabelanger, the gerrit error.log is also full of that too many open files error | 13:40 |
| yolanda | #status alert Gerrit has been successfully restarted | 13:40 |
| *** dizquierdo has joined #openstack-infra | 13:40 | |
| pabelanger | yolanda: these look to be apache2 crashing along with various other errors | 13:40 |
| pabelanger | [Tue Mar 14 07:03:29.133281 2017] [core:notice] [pid 93500:tid 140617752352640] AH00052: child pid 97008 exit signal Segmentation fault (11) | 13:41 |
| yolanda | yep | 13:41 |
| thomasem | Any way to know if there were that many legitimate open files? Wondering if it's an actual bug in Gerrit where it's leaking connections or something. | 13:41 |
| *** matrohon has quit IRC | 13:42 | |
| yolanda | pabelanger, i finished the gerrit restart, do you think i shall restart apache as well? | 13:42 |
| yolanda | i see gerrit working now | 13:42 |
| thomasem | Same | 13:42 |
| yolanda | ah yes, again | 13:42 |
| yolanda | it worked for me in a pair of calls, but breaks again | 13:42 |
| openstackstatus | yolanda: finished sending alert | 13:42 |
| openstackstatus | yolanda: sending alert | 13:42 |
| pabelanger | yolanda: maybe just monitor apache2 for the moment | 13:43 |
| fungi | infra-root: three restarts in 24 hours? have people been double-checking the garbage collection time graphs in javamelody before restarting? | 13:43 |
| *** esikachev has joined #openstack-infra | 13:43 | |
| pabelanger | GC just now looked fine | 13:43 |
| dasm | yolanda: pabelanger gerrit seems to be working, although on my reviews i still see outdated info. | 13:43 |
| pabelanger | so, something else is up with gerrit this morning | 13:43 |
| dasm | before it started crashing with 502, i had the same issues | 13:43 |
| *** matrohon has joined #openstack-infra | 13:44 | |
| yolanda | fails again, segmentation fault errors | 13:44 |
| yolanda | AH00052: child pid 84376 exit signal Segmentation fault (11) | 13:44 |
| fungi | usually it survives a week or so before we need to restart it for that, so three times in a 24 hour period seems suspiciously like something else | 13:44 |
| pabelanger | right, this doesn't look to be GC related, something else | 13:44 |
| pabelanger | fungi: agree | 13:44 |
| yolanda | yes | 13:44 |
| *** knangia has joined #openstack-infra | 13:44 | |
| yolanda | not the usual one | 13:44 |
| *** kzaitsev_mb has joined #openstack-infra | 13:45 | |
| -openstackstatus- NOTICE: Gerrit has been successfully restarted | 13:45 | |
| *** ChanServ changes topic to "Gerrit has been successfully restarted" | 13:45 | |
| fungi | yolanda: did you mean to use #status okay to end the alert? | 13:46 |
| fungi | er, #status ok i guess | 13:46 |
| yolanda | oh yes... silly thing | 13:46 |
| yolanda | #status ok Gerrit has been successfully restarted | 13:46 |
| *** eharney has joined #openstack-infra | 13:48 | |
| openstackstatus | yolanda: finished sending alert | 13:49 |
| openstackstatus | yolanda: sending ok | 13:49 |
| *** bh526r has joined #openstack-infra | 13:49 | |
| *** electrofelix has joined #openstack-infra | 13:49 | |
| *** ChanServ changes topic to "Discussion of OpenStack Developer and Community Infrastructure | docs http://docs.openstack.org/infra/ | bugs https://storyboard.openstack.org/ | source https://git.openstack.org/cgit/openstack-infra/ | channel logs http://eavesdrop.openstack.org/irclogs/%23openstack-infra/" | 13:51 | |
| -openstackstatus- NOTICE: Gerrit has been successfully restarted | 13:51 | |
| *** lucas-hungry is now known as lucasagomes | 13:53 | |
| *** matrohon has quit IRC | 13:54 | |
| *** wklm has joined #openstack-infra | 13:54 | |
| *** anilvenkata_afk is now known as anilvenkata | 13:55 | |
| *** mattmceuen has quit IRC | 13:55 | |
| *** mattmceuen__ has joined #openstack-infra | 13:55 | |
| *** matrohon has joined #openstack-infra | 13:55 | |
| *** funzo has joined #openstack-infra | 13:56 | |
| *** links has quit IRC | 13:57 | |
| pabelanger | looks like epel mirror fixed itself over night too | 13:59 |
| *** funzo has quit IRC | 14:00 | |
| *** vsaienko has quit IRC | 14:04 | |
| *** ihrachys has joined #openstack-infra | 14:05 | |
| *** rbrndt__ has joined #openstack-infra | 14:06 | |
| *** vsaienko has joined #openstack-infra | 14:06 | |
| *** tonytan4ever has joined #openstack-infra | 14:08 | |
| *** amoralej|lunch is now known as amoralej | 14:08 | |
| *** tonytan4ever has quit IRC | 14:10 | |
| *** tonytan4ever has joined #openstack-infra | 14:10 | |
| *** Hal has joined #openstack-infra | 14:12 | |
| openstackgerrit | Adam Spiers proposed openstack-infra/jenkins-job-builder master: add two new options to Log Parser Plugin https://review.openstack.org/445509 | 14:13 |
| *** Hal has quit IRC | 14:15 | |
| *** Hal1 has joined #openstack-infra | 14:15 | |
| *** vsaienko has quit IRC | 14:18 | |
| *** zz_dimtruck is now known as dimtruck | 14:18 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Add request-list nodepool command https://review.openstack.org/445169 | 14:21 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Remove AllocatorTestCase and RoundRobinTestCase https://review.openstack.org/445175 | 14:21 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Fix min-ready/max-servers in test configs https://review.openstack.org/445512 | 14:21 |
| *** vsaienko has joined #openstack-infra | 14:22 | |
| isaacb | pabelanger: Hi, do you know if the kernel version can be controlled in ubuntu-minimal element of DIB | 14:26 |
| *** hongbin has joined #openstack-infra | 14:26 | |
| EmilienM | project-config cores: please review https://review.openstack.org/441186 which is for us a change we would like to apply today. Thanks a lot! | 14:26 |
| *** alexchadin has quit IRC | 14:27 | |
| *** sree has quit IRC | 14:29 | |
| *** vsaienko has quit IRC | 14:30 | |
| *** sree has joined #openstack-infra | 14:30 | |
| *** eharney_ has joined #openstack-infra | 14:32 | |
| *** eharney has quit IRC | 14:33 | |
| *** sree has quit IRC | 14:35 | |
| *** eharney_ is now known as eharney | 14:35 | |
| *** vsaienko has joined #openstack-infra | 14:35 | |
| *** dave-mccowan has quit IRC | 14:36 | |
| *** shardy has quit IRC | 14:37 | |
| *** shardy has joined #openstack-infra | 14:37 | |
| *** edmondsw has joined #openstack-infra | 14:40 | |
| pabelanger | isaacb: I don't believe so. IIRC, you'd have to write a element to handle that. FWIW: I've never needed to change or select a specific kernel. perhaps ianw would be better to ask | 14:42 |
| *** vikrant has quit IRC | 14:42 | |
| *** pblaho has quit IRC | 14:43 | |
| *** esberglu has joined #openstack-infra | 14:45 | |
| isaacb | pabelanger: ok. thanks | 14:46 |
| openstackgerrit | Armando Migliaccio proposed openstack-infra/project-config master: Add gate-hook to tempest job for networking-sfc https://review.openstack.org/445037 | 14:47 |
| *** markus_z has quit IRC | 14:48 | |
| *** burgerk has joined #openstack-infra | 14:49 | |
| *** vsaienko has quit IRC | 14:50 | |
| *** dave-mccowan has joined #openstack-infra | 14:51 | |
| *** makowals_ has quit IRC | 14:51 | |
| sshnaidm | AJaeger, hi, can you please take a look in your time at urgent patch for tripleo jobs? https://review.openstack.org/#/c/441186/ thanks | 14:52 |
| *** vsaienko has joined #openstack-infra | 14:53 | |
| *** edmondsw has quit IRC | 14:53 | |
| *** kelepirci has quit IRC | 14:53 | |
| *** marst has quit IRC | 14:55 | |
| *** funzo has joined #openstack-infra | 14:57 | |
| *** faizy_ has joined #openstack-infra | 14:58 | |
| clarkb | isaacb: pabelanger ya it will just install the newest "normal" kernel iirc. But should be easy to add an element that installs arbitrary kernels like hardware enablement kernels | 14:59 |
| *** vsaienko has quit IRC | 15:00 | |
| *** prateek has joined #openstack-infra | 15:00 | |
| *** faizy has quit IRC | 15:00 | |
| *** funzo has quit IRC | 15:01 | |
| isaacb | clarkb: I found in the documentation: DIB_ARM_KERNEL. in both debian-minimal and debootstrap. I'm currently trying to use it. (Maybe it's name is misleading) | 15:02 |
| *** cody-somerville has joined #openstack-infra | 15:03 | |
| clarkb | isaacb: my understanding is that debootstrap is really constrained in kernel management, and the dib element has a separate step to update it to something modern at the end | 15:03 |
| *** esikachev has quit IRC | 15:03 | |
| *** vsaienko has joined #openstack-infra | 15:04 | |
| *** armax has joined #openstack-infra | 15:04 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: Create run-cover role https://review.openstack.org/441332 | 15:05 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: Create zuul_workspace_root job variable https://review.openstack.org/441441 | 15:05 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: Organize playbooks folder https://review.openstack.org/441547 | 15:05 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: Rename prepare-workspace role to bootstrap https://review.openstack.org/441440 | 15:05 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: Add run-docs role and tox-docs job https://review.openstack.org/441345 | 15:05 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: Add net-info to bootstrap role https://review.openstack.org/441617 | 15:05 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: Add revoke-sudo role and update tox jobs https://review.openstack.org/441467 | 15:05 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: Create tox-tarball job https://review.openstack.org/441609 | 15:05 |
| *** pblaho has joined #openstack-infra | 15:08 | |
| *** matrohon has quit IRC | 15:09 | |
| *** marst has joined #openstack-infra | 15:10 | |
| *** esikachev has joined #openstack-infra | 15:10 | |
| *** vsaienko has quit IRC | 15:11 | |
| *** john-dav_ has joined #openstack-infra | 15:12 | |
| *** john-davidge has quit IRC | 15:12 | |
| *** matrohon has joined #openstack-infra | 15:12 | |
| *** john-dav_ is now known as john-davidge | 15:13 | |
| *** saibarspeis has joined #openstack-infra | 15:14 | |
| *** adrian_otto has joined #openstack-infra | 15:15 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: Create run-cover role https://review.openstack.org/441332 | 15:16 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: Create zuul_workspace_root job variable https://review.openstack.org/441441 | 15:16 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: Organize playbooks folder https://review.openstack.org/441547 | 15:16 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: Rename prepare-workspace role to bootstrap https://review.openstack.org/441440 | 15:16 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: Add run-docs role and tox-docs job https://review.openstack.org/441345 | 15:17 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: Add net-info to bootstrap role https://review.openstack.org/441617 | 15:17 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: Add revoke-sudo role and update tox jobs https://review.openstack.org/441467 | 15:17 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: Create tox-tarball job https://review.openstack.org/441609 | 15:17 |
| *** vsaienko has joined #openstack-infra | 15:17 | |
| *** andreas_s has quit IRC | 15:20 | |
| *** xyang1 has joined #openstack-infra | 15:23 | |
| *** hasharLunch is now known as hashar | 15:24 | |
| *** vsaienko has quit IRC | 15:24 | |
| clarkb | fungi: yolanda pabelanger looking at java melody for gerrit, the restart yesterday at ~1800UTC was GC related | 15:25 |
| *** psachin has quit IRC | 15:25 | |
| clarkb | any subsequent restarts wouldn't have been (according to java melody) | 15:25 |
| fungi | that was my suspicion as well | 15:26 |
| yolanda | clarkb, today there were errors about "Too many files opened" in gerrit (reaching ulimit?) and segfaults on apache | 15:27 |
| fungi | should probably start analyzing gerrit logs looking for some sort of repeated requests which could put it under stress | 15:27 |
| clarkb | yolanda: gerrit itself had too many files opened errors? | 15:28 |
| yolanda | yep, see gerrit error.log | 15:28 |
| clarkb | I think that is already configureable in the init script so we could try bumping up that number | 15:28 |
| *** vsaienko has joined #openstack-infra | 15:28 | |
| *** adarazs has joined #openstack-infra | 15:29 | |
| *** ociuhandu has quit IRC | 15:29 | |
| *** dhajare has quit IRC | 15:29 | |
| *** mtanino has quit IRC | 15:29 | |
| fungi | looking at the ssh log around that time, there is a flood of gerrit queries being performed by Hitachi-Manila-HNAS-CI a/17623 | 15:31 |
| adarazs | can an infra core push this change please: https://review.openstack.org/441186 | 15:32 |
| adarazs | it already has a +2 and a +1 from the TripleO PTL and we really need it to move forward. | 15:32 |
| *** saibarsp_ has joined #openstack-infra | 15:34 | |
| *** esikachev has quit IRC | 15:36 | |
| *** Julien-zte has quit IRC | 15:37 | |
| *** saibarspeis has quit IRC | 15:37 | |
| *** Julien-zte has joined #openstack-infra | 15:38 | |
| *** Julien-zte has quit IRC | 15:38 | |
| *** vsaienko has quit IRC | 15:38 | |
| *** Julien-zte has joined #openstack-infra | 15:38 | |
| *** anilvenkata has quit IRC | 15:38 | |
| *** jesusaur has quit IRC | 15:38 | |
| *** Julien-zte has quit IRC | 15:39 | |
| *** Julien-zte has joined #openstack-infra | 15:39 | |
| *** Julien-zte has quit IRC | 15:39 | |
| *** Julien-zte has joined #openstack-infra | 15:40 | |
| *** Julien-zte has quit IRC | 15:40 | |
| *** vsaienko has joined #openstack-infra | 15:41 | |
| *** Julien-zte has joined #openstack-infra | 15:41 | |
| *** Julien-zte has quit IRC | 15:41 | |
| *** erikmwilson has quit IRC | 15:41 | |
| *** ociuhandu has joined #openstack-infra | 15:41 | |
| *** saibarsp_ has quit IRC | 15:41 | |
| *** Julien-zte has joined #openstack-infra | 15:42 | |
| *** jesusaur has joined #openstack-infra | 15:42 | |
| *** erikmwilson has joined #openstack-infra | 15:42 | |
| *** Julien-zte has quit IRC | 15:42 | |
| openstackgerrit | Michael Johnson proposed openstack-infra/project-config master: Adding new repositories for Octavia https://review.openstack.org/444057 | 15:43 |
| *** ociuhandu has quit IRC | 15:45 | |
| *** nadya has joined #openstack-infra | 15:53 | |
| openstackgerrit | Merged openstack-infra/project-config master: Run 'enable_plugin' in project-config for grenade jobs https://review.openstack.org/443348 | 15:54 |
| *** vsaienko has quit IRC | 15:54 | |
| *** isaacb has quit IRC | 15:54 | |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder master: Add the Slave Prerequisites properties plugin https://review.openstack.org/441893 | 15:55 |
| *** markvoelker has quit IRC | 15:56 | |
| openstackgerrit | Alex Schultz proposed openstack/diskimage-builder master: Allow ELEMENTS_DIR to be configurable https://review.openstack.org/445545 | 15:57 |
| openstackgerrit | Merged openstack-infra/puppet-openstackid master: Update functions to force on production https://review.openstack.org/444323 | 15:57 |
| *** funzo has joined #openstack-infra | 15:57 | |
| *** markvoelker has joined #openstack-infra | 15:58 | |
| *** vsaienko has joined #openstack-infra | 15:58 | |
| openstackgerrit | Merged openstack-infra/system-config master: Add octavia logs to jenkins-log-client.yaml https://review.openstack.org/408246 | 15:58 |
| *** funzo has quit IRC | 16:02 | |
| *** salv-orlando has joined #openstack-infra | 16:03 | |
| openstackgerrit | Dean Troyer proposed openstack-infra/project-config master: Re-enable Image v1 for OSC functional jobs https://review.openstack.org/445553 | 16:05 |
| *** salv-orlando has quit IRC | 16:05 | |
| *** felipemonteiro has quit IRC | 16:05 | |
| openstackgerrit | Kyrylo Shatskyy proposed openstack-infra/jenkins-job-builder master: Add support for 'Groovy Label Assignment Plugin' https://review.openstack.org/428419 | 16:05 |
| *** salv-orlando has joined #openstack-infra | 16:05 | |
| openstackgerrit | John L. Villalovos proposed openstack-infra/project-config master: ironic: Make grenade multi-node job non-voting https://review.openstack.org/445555 | 16:05 |
| *** xarses_ has joined #openstack-infra | 16:06 | |
| openstackgerrit | Merged openstack-infra/system-config master: Update OpenStackId puppet settings https://review.openstack.org/444342 | 16:06 |
| *** hashar has quit IRC | 16:06 | |
| *** garciadeblas has quit IRC | 16:07 | |
| *** erikmwilson has quit IRC | 16:07 | |
| fungi | #status log Changes https://review.openstack.org/444323 and https://review.openstack.org/444342 have been approved, upgrading https://openstackid.org/ production to what's been running and tested on https://openstackid-dev.openstack.org/ | 16:07 |
| openstackstatus | fungi: finished logging | 16:07 |
| *** andreww has joined #openstack-infra | 16:07 | |
| *** felipemonteiro has joined #openstack-infra | 16:07 | |
| fungi | heads up ^ in case anybody spots anything unusual with authenticating there | 16:08 |
| *** xarses_ has quit IRC | 16:09 | |
| *** erikmwilson has joined #openstack-infra | 16:10 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Add request-list nodepool command https://review.openstack.org/445169 | 16:10 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Remove AllocatorTestCase and RoundRobinTestCase https://review.openstack.org/445175 | 16:10 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Fix min-ready/max-servers in test configs https://review.openstack.org/445512 | 16:10 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Fix race on node state check in node cleanup https://review.openstack.org/445557 | 16:10 |
| *** jkilpatr has quit IRC | 16:10 | |
| *** xarses_ has joined #openstack-infra | 16:11 | |
| *** andreww has quit IRC | 16:11 | |
| *** vsaienko has quit IRC | 16:11 | |
| clarkb | fungi: yolanda pabelanger so if gerrit has a sad again we should check if there are file limit errors in the gerrit error log and do a connection listing via gerrit ssh api? | 16:12 |
| *** andreww has joined #openstack-infra | 16:12 | |
| clarkb | maybe we can try killing "bad" looking connections as a first step? | 16:12 |
| yolanda | clarkb, agree | 16:13 |
| *** salv-orlando has quit IRC | 16:13 | |
| *** aarefiev is now known as aarefiev_afk | 16:14 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: WIP: Conntainers with multi-nic and net-iso https://review.openstack.org/422448 | 16:15 |
| *** xarses_ has quit IRC | 16:15 | |
| *** sandanar has quit IRC | 16:16 | |
| *** vsaienko has joined #openstack-infra | 16:16 | |
| *** andreww has quit IRC | 16:16 | |
| *** xarses_ has joined #openstack-infra | 16:16 | |
| clarkb | ianychoi: AJaeger: good morning/evening. I'd like to start cleaning up the older zanata hosts (running trusty and 3.7.3 zanata). Is the new host still happy? any reason for me to not delete the old ones soon? | 16:18 |
| clarkb | I guess I need to double check backups are working on new server first | 16:18 |
| clarkb | but other than that, let me know if you can think of anything | 16:18 |
| *** tphummel has joined #openstack-infra | 16:20 | |
| yolanda | clarkb, pabelanger , apache error logs are full of Segmentation faults, not from today but from the previous days as well | 16:21 |
| *** annegentle has joined #openstack-infra | 16:22 | |
| *** jkilpatr has joined #openstack-infra | 16:23 | |
| openstackgerrit | John L. Villalovos proposed openstack-infra/project-config master: ironic: Make grenade multi-node job non-voting https://review.openstack.org/445555 | 16:26 |
| *** annegentle has quit IRC | 16:26 | |
| *** annegentle has joined #openstack-infra | 16:27 | |
| *** trown is now known as trown|lunch | 16:29 | |
| clarkb | yolanda: I wonder if thats just noise when related to this problem (or perhaps a result of the other things) | 16:29 |
| fungi | clarkb: yes, would also help to check whether there is a similar flood of queries from the same ci account again | 16:30 |
| *** dhinesh has joined #openstack-infra | 16:30 | |
| *** bhavik1 has joined #openstack-infra | 16:31 | |
| *** e0ne has quit IRC | 16:31 | |
| *** yamahata has quit IRC | 16:32 | |
| *** annegentle has quit IRC | 16:32 | |
| *** Apoorva has joined #openstack-infra | 16:33 | |
| *** Apoorva has quit IRC | 16:33 | |
| *** esikachev has joined #openstack-infra | 16:35 | |
| *** tonytan_brb has joined #openstack-infra | 16:35 | |
| *** tonytan_brb has quit IRC | 16:35 | |
| *** prateek has quit IRC | 16:35 | |
| *** tonytan4ever has quit IRC | 16:36 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/nodepool feature/zuulv3: Remove ready-script support https://review.openstack.org/445567 | 16:36 |
| *** tonytan_brb has joined #openstack-infra | 16:36 | |
| *** prateek has joined #openstack-infra | 16:39 | |
| *** dizquierdo has quit IRC | 16:39 | |
| *** panda is now known as panda|bbl | 16:39 | |
| *** claudiub has quit IRC | 16:40 | |
| *** esikachev has quit IRC | 16:41 | |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder master: Enable options and use convert_mapping_to_xml for groovy plugin https://review.openstack.org/435974 | 16:41 |
| *** esikachev has joined #openstack-infra | 16:42 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/nodepool feature/zuulv3: Stop writing nodepool bash variable on nodes https://review.openstack.org/445572 | 16:42 |
| *** ccamacho has quit IRC | 16:42 | |
| *** david-lyle has joined #openstack-infra | 16:42 | |
| *** 21WAAS6DN has joined #openstack-infra | 16:42 | |
| clarkb | can someone double check me. The translate-dev servers to delete are 5cfc10d8-4ec1-45b3-8605-ee3d36a4c1df, b3cd5715-5cf4-4d2f-8550-5e4b0a9144ae, and afd4a8d9-98a7-4a21-a827-33106abeeb8a. The first two are failed attempts at building new 01 servers (that I thought launch node cleaned up but apparently didn't) and the third should be the old translate-dev.openstack.org server | 16:44 |
| *** pblaho has quit IRC | 16:45 | |
| *** esikachev has quit IRC | 16:46 | |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci master: Use custom network-isolation-absolute.yaml in ipv4 net-iso jobs https://review.openstack.org/422297 | 16:47 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci master: Test with bonded public interface https://review.openstack.org/385562 | 16:47 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci master: Add storage-mgmt network to compute nodes https://review.openstack.org/419049 | 16:47 |
| *** dhinesh has quit IRC | 16:50 | |
| *** dtantsur is now known as dtantsur|brb | 16:51 | |
| clarkb | I'm going to run a manual bup run on translate01 now to make sure backups are working there (in prep for deleting old server) | 16:51 |
| *** hashar has joined #openstack-infra | 16:51 | |
| *** adrian_otto has quit IRC | 16:52 | |
| clarkb | oh and getting puppet ansible to run there first to make sure bup excludes list is up to date | 16:53 |
| *** timello has quit IRC | 16:54 | |
| clarkb | hrm ansible may be stuck on its own? | 16:54 |
| *** tmorin has joined #openstack-infra | 16:55 | |
| *** d0ugal has quit IRC | 16:55 | |
| *** esikachev has joined #openstack-infra | 16:56 | |
| *** funzo has joined #openstack-infra | 16:58 | |
| openstackgerrit | Adam Spiers proposed openstack-infra/jenkins-job-builder master: add two new options to Log Parser Plugin https://review.openstack.org/445509 | 17:01 |
| tmorin | hi AJaeger, we've had neutron liaison +1, can you proceed with https://review.openstack.org/#/c/441725/ ? | 17:02 |
| clarkb | if 2017-03-14 16:09:10,639 p=10062 u=root | skipping: [elasticsearch04.openstack.org] is the lsat thing logged by the ansible puppet runner should I be manually killing the ansible processes? | 17:02 |
| clarkb | pabelanger: fungi mordred ^ | 17:02 |
| *** timello has joined #openstack-infra | 17:02 | |
| *** funzo has quit IRC | 17:03 | |
| *** erikmwilson has quit IRC | 17:07 | |
| *** bh526r has quit IRC | 17:08 | |
| *** erikmwilson has joined #openstack-infra | 17:09 | |
| *** arxcruz has quit IRC | 17:09 | |
| openstackgerrit | Ken Dreyer proposed openstack-infra/jenkins-job-builder master: builders: fix docker-build-publish docstring link https://review.openstack.org/445586 | 17:10 |
| *** mhickey has joined #openstack-infra | 17:11 | |
| *** arxcruz has joined #openstack-infra | 17:12 | |
| *** eharney has quit IRC | 17:12 | |
| clarkb | ok according to timestamps its been an hour that logs haven't moved | 17:12 |
| clarkb | if I don't hear objections I am going to kill some ansible processes | 17:12 |
| *** david-lyle_ has joined #openstack-infra | 17:13 | |
| *** david-lyle has quit IRC | 17:13 | |
| *** nadya has quit IRC | 17:14 | |
| *** yamahata has joined #openstack-infra | 17:14 | |
| *** matrohon has quit IRC | 17:16 | |
| fungi | clarkb: sounds like what i would try | 17:18 |
| clarkb | do we run the infracloud puppeting out of band with the rest of it? we must, looks like there are two sets of things running | 17:18 |
| *** tmorin has quit IRC | 17:18 | |
| fungi | sorry, been pulled in a bunch of directions and need to go grab lunch before our 1900z meeting | 17:18 |
| fungi | yeah, tehre's a separate playbook for the infracloud machines afair | 17:19 |
| * fungi will be back soonish | 17:19 | |
| *** eroux has joined #openstack-infra | 17:20 | |
| openstackgerrit | Merged openstack-infra/project-config master: Change TripleO multinode jobs to be quickstart based https://review.openstack.org/441186 | 17:20 |
| pabelanger | clarkb: yes, infracloud is a different crontab | 17:20 |
| *** dhinesh has joined #openstack-infra | 17:20 | |
| clarkb | that seems to have done it, I killed the lowest non defunct ssh process in the non infracloud ps tree output | 17:21 |
| *** pblaho has joined #openstack-infra | 17:22 | |
| *** hashar has quit IRC | 17:23 | |
| *** eharney has joined #openstack-infra | 17:26 | |
| *** lucasagomes is now known as lucas-afk | 17:26 | |
| *** trown|lunch is now known as trown | 17:28 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: Rename zuul-launcher to zuul-executor https://review.openstack.org/445594 | 17:29 |
| *** thingee has joined #openstack-infra | 17:34 | |
| *** esikachev has quit IRC | 17:34 | |
| jlvillal | pabelanger: What does zuul kill now? ;) | 17:35 |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder master: builders: fix docker-build-publish docstring link https://review.openstack.org/445586 | 17:35 |
| jlvillal | Your gate jobs worst nightmare, the Zuul Executor is coming to kill them all... | 17:36 |
| openstackgerrit | Ihar Hrachyshka proposed openstack-infra/elastic-recheck master: Removed the query for 1532809 https://review.openstack.org/445596 | 17:36 |
| *** esikachev has joined #openstack-infra | 17:38 | |
| pabelanger | jlvillal: we'll execute all the thing | 17:40 |
| jlvillal | :) | 17:40 |
| *** esikachev has quit IRC | 17:42 | |
| *** esikachev has joined #openstack-infra | 17:44 | |
| *** jaosorior has quit IRC | 17:45 | |
| *** mhickey has quit IRC | 17:45 | |
| *** jpich has quit IRC | 17:45 | |
| clarkb | ok puppet ran and I am not running a manual bup run | 17:45 |
| *** kelepirci has joined #openstack-infra | 17:46 | |
| *** vsaienko has quit IRC | 17:47 | |
| *** kelepirci has quit IRC | 17:47 | |
| *** Hal1 has quit IRC | 17:48 | |
| *** esikachev has quit IRC | 17:48 | |
| *** cody-somerville has quit IRC | 17:48 | |
| *** csomerville has joined #openstack-infra | 17:48 | |
| *** smarcet has joined #openstack-infra | 17:49 | |
| *** tesseract has quit IRC | 17:49 | |
| *** rbrndt__ has quit IRC | 17:50 | |
| *** esikachev has joined #openstack-infra | 17:50 | |
| *** cody-somerville has joined #openstack-infra | 17:50 | |
| openstackgerrit | Paul Belanger proposed openstack/diskimage-builder master: Split debootstrap into debootstrap-minimal https://review.openstack.org/445601 | 17:52 |
| openstackgerrit | Paul Belanger proposed openstack/diskimage-builder master: Split debootstrap into debootstrap-minimal https://review.openstack.org/413103 | 17:53 |
| *** csomerville has quit IRC | 17:53 | |
| clarkb | pabelanger: we might want to exclude /var/lib/lxcfs from bup too? | 17:54 |
| clarkb | looks to be systemd related so would be getting ahead of xenial updates | 17:54 |
| *** esikachev has quit IRC | 17:54 | |
| pabelanger | clarkb: sure | 17:55 |
| *** derekh has quit IRC | 17:55 | |
| *** nadya has joined #openstack-infra | 17:57 | |
| *** panda|bbl is now known as panda | 17:58 | |
| openstackgerrit | Paul Belanger proposed openstack/diskimage-builder master: Add ubuntu-rootfs operating-system element https://review.openstack.org/413115 | 17:58 |
| *** d0ugal has joined #openstack-infra | 17:58 | |
| *** funzo has joined #openstack-infra | 17:59 | |
| clarkb | #status log translate01.o.o is now being backed up to backup server | 18:00 |
| *** faizy_ has quit IRC | 18:00 | |
| *** pvaneck has joined #openstack-infra | 18:01 | |
| *** ralonsoh has quit IRC | 18:02 | |
| openstackgerrit | Paul Belanger proposed openstack/diskimage-builder master: Clean up cache for ubuntu-minimal https://review.openstack.org/414728 | 18:02 |
| *** nadya has quit IRC | 18:02 | |
| *** Gravious has joined #openstack-infra | 18:02 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: WIP Add per-repo public and private keys https://review.openstack.org/406382 | 18:02 |
| Gravious | howdy | 18:02 |
| *** dtantsur|brb is now known as dtantsur | 18:03 | |
| *** funzo has quit IRC | 18:03 | |
| *** ociuhandu has joined #openstack-infra | 18:03 | |
| *** pcaruana has quit IRC | 18:03 | |
| Gravious | fungi, ping | 18:04 |
| *** d0ugal has quit IRC | 18:04 | |
| *** mgagne_ is now known as mgagne | 18:05 | |
| clarkb | looks like ansible loop got stuck again :/ | 18:07 |
| clarkb | ok going to delete the extra zanata-dev servers now | 18:08 |
| clarkb | and will wait for ianychoi to confirm he is ready for deleting old prod server later today | 18:08 |
| *** bhavik1 has quit IRC | 18:10 | |
| *** Sukhdev has joined #openstack-infra | 18:11 | |
| *** jkilpatr has quit IRC | 18:12 | |
| *** jkilpatr has joined #openstack-infra | 18:13 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: WIP Add per-repo public and private keys https://review.openstack.org/406382 | 18:14 |
| *** florianf has quit IRC | 18:15 | |
| *** dizquierdo has joined #openstack-infra | 18:16 | |
| *** annegentle has joined #openstack-infra | 18:17 | |
| *** dhajare has joined #openstack-infra | 18:19 | |
| *** annegentle has quit IRC | 18:20 | |
| *** annegentle has joined #openstack-infra | 18:20 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config master: layout/tripleo: run upgrade job in tripleo-common https://review.openstack.org/445610 | 18:23 |
| zaro | clarkb: gerrit packedGitOpenFiles is currently set to 4096 which seems kinda low to me. too low can cause data corruption. | 18:24 |
| *** annegentle has quit IRC | 18:25 | |
| clarkb | zaro: that sounds like a fun error case. What do you think it should be set to? maybe you can push a patch? | 18:25 |
| zaro | https://groups.google.com/d/msg/repo-discuss/mGwI3yHnGWQ/gZxwHLDZDAAJ | 18:26 |
| zaro | clarkb: i’m not sure have you taken a look at the open files graph on javamelody? | 18:26 |
| clarkb | zaro: I haven't, but I can look now | 18:28 |
| clarkb | zaro: we peak at 8k | 18:28 |
| clarkb | and sure enough each time we hvae had to restart in the lsat day or so we hit that limit | 18:29 |
| clarkb | so our effective limit must be 8k with half of that for git | 18:29 |
| zaro | so maybe up packedGitOpenFiles to 8K? not sure what the ulimit is for the system though. probably want to coordinate those. | 18:30 |
| clarkb | ya they should be in sync, guessin 2x packagesgitopenfiles for ulimit | 18:30 |
| clarkb | zaro: actually the ulimit value is set to that value in the config | 18:31 |
| clarkb | oh wait no they double it | 18:32 |
| clarkb | zaro: so all we need to do is set packedGitOpenFiles and that will set the ulimit limit to double that | 18:32 |
| *** claudiub has joined #openstack-infra | 18:32 | |
| *** spotz is now known as spotz_zzz | 18:32 | |
| zaro | ahh cool. i can push that change if you like | 18:32 |
| clarkb | zaro: that would be great | 18:33 |
| *** spotz_zzz is now known as spotz | 18:33 | |
| *** cody-somerville has quit IRC | 18:34 | |
| *** cody-somerville has joined #openstack-infra | 18:34 | |
| *** mkoderer has quit IRC | 18:37 | |
| *** dprince has quit IRC | 18:38 | |
| *** flwang has quit IRC | 18:38 | |
| *** electrofelix has quit IRC | 18:39 | |
| *** john-davidge has quit IRC | 18:40 | |
| *** tosky has quit IRC | 18:40 | |
| *** Gravious has quit IRC | 18:40 | |
| *** Gravious has joined #openstack-infra | 18:40 | |
| *** john-davidge has joined #openstack-infra | 18:40 | |
| *** panda is now known as panda|dinner | 18:40 | |
| *** rbrndt__ has joined #openstack-infra | 18:41 | |
| *** aysyd has quit IRC | 18:41 | |
| *** timello has quit IRC | 18:41 | |
| openstackgerrit | Khai Do proposed openstack-infra/system-config master: Increase git open files and ulimit https://review.openstack.org/445616 | 18:42 |
| zaro | clarkb: ^ | 18:42 |
| clarkb | zaro: thanks | 18:44 |
| *** john-davidge has quit IRC | 18:44 | |
| zaro | np | 18:44 |
| *** shardy is now known as shardy_afk | 18:45 | |
| clarkb | fungi: yolanda pabelanger I have +2'd ^ https://review.openstack.org/monitoring?part=graph&graph=fileDescriptors shows our fd useage | 18:45 |
| clarkb | would be curious to see if it levels out if we bump the limit or if we are aggressively leaking | 18:45 |
| clarkb | but hard to tell until we bump the limit | 18:45 |
| *** timello has joined #openstack-infra | 18:45 | |
| clarkb | applying that change will require another gerrit restart too fwiw | 18:46 |
| clarkb | something we'll have to do in a couple hours looks like | 18:46 |
| *** pvaneck has quit IRC | 18:46 | |
| *** pvaneck has joined #openstack-infra | 18:47 | |
| clarkb | fungi: smarcet is openstackid.org running newer version now? | 18:49 |
| smarcet | clarkb | 18:49 |
| smarcet | i dont think so | 18:49 |
| smarcet | is not deployed so far from what i can see | 18:50 |
| zaro | clarkb: was wondering whether zulu-dev is setup and working with review-dev and a nodepool yet? | 18:50 |
| fungi | we probably need to collect puppet stdout from syslog | 18:50 |
| fungi | on openstackid.org | 18:50 |
| *** flwang has joined #openstack-infra | 18:50 | |
| fungi | prepping for the infra and tc meetings or i'd look into it now | 18:50 |
| smarcet | clarkb i think that puppet is disabled for openstackid.org | 18:50 |
| clarkb | fungi: smarcet its in the ansible emergency file | 18:50 |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci master: multinode settings: add multinode_mode variable https://review.openstack.org/445618 | 18:50 |
| * fungi checks | 18:50 | |
| clarkb | smarcet: yup | 18:50 |
| fungi | aha, yes. i guess we need to undo that | 18:50 |
| clarkb | and ansible is stuck again. I will work on killing the right pstree again | 18:51 |
| fungi | clarkb: most of the time when i see that, it's because it wants to connect to a server which is unreachable | 18:51 |
| *** 21WAAS6DN has quit IRC | 18:51 | |
| *** esikachev has joined #openstack-infra | 18:51 | |
| clarkb | ya I hvae an IP addr, currently trying to figure out what it is | 18:52 |
| fungi | ansible has a _really_ long timeout for deciding it can't connect to a remote system | 18:53 |
| clarkb | dig timing out trying to get reverse ptr lookup | 18:53 |
| fungi | eek | 18:54 |
| clarkb | of course I can never remember where we cache the openstack data json file | 18:54 |
| jeblair | clarkb, zaro: looking at the month and week forms of the open files graph, it seems to be a recent change. maybe the 3rd party ci fungi found started behaving poorly yesterday. | 18:54 |
| fungi | i haven't had time to perform any in-depth analysis of its particular trends, but it looked like it spammed the server with a bunch of potentially slow queries around the time of the most recent restart | 18:55 |
| clarkb | fungi: mirror.nyj01.internap.openstack.org is the host | 18:55 |
| clarkb | is that a region that we turned off? | 18:55 |
| clarkb | its not in nodepool. So now how do we get it out of the cache? just rm the file? | 18:56 |
| *** esikachev has quit IRC | 18:56 | |
| fungi | i don't remember turning it off, but maybe that happened last week while i was otherwise occupied | 18:56 |
| *** amotoki has quit IRC | 18:56 | |
| jhesketh | Morning | 18:58 |
| Gravious | fungi, howdy | 18:58 |
| clarkb | fungi: I think they had us move to a newer region a while back | 18:59 |
| clarkb | they must've finally killed it properly in the cloud though and now the mirror host unpings | 18:59 |
| Gravious | fungi, do you have any thoughts on dulwich, wrt to dulwich vs gitpython for openstack stuff? | 19:00 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: Add scenario005 job (4nodes) https://review.openstack.org/418098 | 19:00 |
| fungi | Gravious: i honestly haven't looked at dulwich. i know we currently have gitpython implemented in other parts of our toolchain, but beyond that i'm pretty unaware of the landscape there | 19:01 |
| clarkb | does anyone know if it is safe to rm the ansible inventory cache file? | 19:01 |
| clarkb | mordred: jeblair pabelanger ^ you may know | 19:01 |
| fungi | clarkb: yes, it is | 19:01 |
| clarkb | cool removin it now, then will kill the ssh connection attempt | 19:01 |
| clarkb | that should get us unstick | 19:01 |
| fungi | clarkb: i usually truncate it just to be safe because there used to be bugs, but i understand deletion should be safe now | 19:01 |
| clarkb | kk | 19:01 |
| fungi | it's that (weekly infra team meeting) time again! find us in #openstack-meeting for the next hour | 19:01 |
| Gravious | fungi, ah, well so far as i can tell their both reasonably healthy projects, and dulwich has a dual license (gplv2, apache). my issue is for some of why i'm trying to do the dulwich api seems like a much better fit. it also has *optional* C extensions so a possible performance advantage. as well as that it doesn't require git to be installed. | 19:02 |
| Gravious | *they're | 19:02 |
| *** dtantsur is now known as dtantsur|afk | 19:02 | |
| clarkb | Gravious: zuul has used gitpython for years. Its not perfect but it works reasonably well | 19:04 |
| clarkb | I don't think we have used dulwich anywhere | 19:04 |
| *** sslypushenko_ has joined #openstack-infra | 19:06 | |
| *** csomerville has joined #openstack-infra | 19:07 | |
| Gravious | a | 19:07 |
| *** esikachev has joined #openstack-infra | 19:07 | |
| Gravious | i don't know maybe i'm being dumb, my issue is i want to create trees directly without mucking about with index (even if only a temporary one) and i don't see a way to do that with gitpython | 19:07 |
| *** dizquierdo has quit IRC | 19:07 | |
| clarkb | Gravious: I think the zuul test suite does that | 19:08 |
| clarkb | Gravious: you might look there for examples? | 19:08 |
| Gravious | ok, will be quite happy to be wrong on this | 19:09 |
| clarkb | I think you just git init it | 19:09 |
| Gravious | git init it? | 19:09 |
| *** ociuhandu has quit IRC | 19:09 | |
| clarkb | sorry, create repo via init, make changes, add and comit | 19:09 |
| clarkb | its very analagous to cli because its largely just a passthrough | 19:09 |
| *** lseki has joined #openstack-infra | 19:09 | |
| clarkb | which is nice at times and terrible at times | 19:09 |
| Gravious | ah right, yeah i'm sorry but i think for what i'm doing that is really pretty suboptimal :< | 19:10 |
| *** cody-somerville has quit IRC | 19:10 | |
| clarkb | ok, still not sure what yo uwant t odo :) | 19:10 |
| *** esikachev has quit IRC | 19:12 | |
| Gravious | pretty much i'm just inserting a blob into a repo, but there is no corresponding file in the worktree (the ref i'm inserting the blob into will not be checked out), but doing so in such a way as to also not involve the index, mainly so that the repo is left clean if any part of the operation fails. | 19:12 |
| *** vsaienko has joined #openstack-infra | 19:13 | |
| *** vsaienko has quit IRC | 19:15 | |
| *** eroux has quit IRC | 19:17 | |
| clarkb | I guess I'd weigh the likelyhood of failure against the simplicity of except: repo.reset(HEAD) | 19:17 |
| clarkb | that said the use case seems odd? why are you checking things into git that won't ever reasonably be checked out? | 19:18 |
| openstackgerrit | Merged openstack-infra/infra-specs master: Zuul v3: Silly whitespace fixes https://review.openstack.org/443983 | 19:18 |
| *** vhosakot has joined #openstack-infra | 19:19 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Allow ELEMENTS_DIR to be configurable https://review.openstack.org/445545 | 19:19 |
| *** nadya has joined #openstack-infra | 19:21 | |
| Gravious | clarkb, Zara & I are working on a python interface to gerrit's notedb. so any updating of metadata should be handled in the background and not screw up a user's checkout (in the case that review metadata and code happen to be in the same repo) | 19:22 |
| clarkb | aha!, and notedb doesn't use notes for that? | 19:23 |
| clarkb | you should be ablt to git python with notes like any other git user I think | 19:23 |
| *** aysyd has joined #openstack-infra | 19:24 | |
| Gravious | clarkb, it's loosely based on notes, but the notes api for gitpython isn't very complete i believe | 19:24 |
| clarkb | that said looking at dulwich it seems fine. Just havne't used it | 19:24 |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Allow github trigger to match on branches/refs https://review.openstack.org/445625 | 19:25 |
| openstackgerrit | Felipe Monteiro proposed openstack-infra/project-config master: Remove non-voting glare gates from murano. https://review.openstack.org/445626 | 19:26 |
| *** dave-mccowan has quit IRC | 19:26 | |
| Gravious | clarkb, yeah they both seem to be pretty active, and dulwich unlike pygit2 has appropriate license | 19:28 |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: Clean up oooq jobs https://review.openstack.org/445628 | 19:30 |
| *** esikachev has joined #openstack-infra | 19:32 | |
| *** panda|dinner is now known as panda | 19:34 | |
| *** ociuhandu has joined #openstack-infra | 19:35 | |
| *** esikachev has quit IRC | 19:36 | |
| *** Dennis_ has joined #openstack-infra | 19:38 | |
| *** vhosakot_ has joined #openstack-infra | 19:40 | |
| Dennis_ | I am trying to sign the OpenStack Individual Contributor License Agreement on https://review.openstack.org/#/settings/new-agreement and I am getting a server error: Cannot store contact information. Can someone help please? | 19:40 |
| *** mattmceuen__ has quit IRC | 19:40 | |
| *** john-davidge has joined #openstack-infra | 19:41 | |
| Gravious | Dennis_, it seems folks on in a meeting, could be a while before you get a response | 19:41 |
| Gravious | *are in a | 19:41 |
| *** vsaienko has joined #openstack-infra | 19:42 | |
| Dennis_ | Thanks for the response Gravious. | 19:42 |
| * Gravious bows | 19:42 | |
| *** vhosakot has quit IRC | 19:42 | |
| *** amoralej is now known as amoralej|off | 19:43 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Fix for unpaused request handlers https://review.openstack.org/445632 | 19:45 |
| *** john-davidge has quit IRC | 19:46 | |
| openstackgerrit | Kendall Nelson proposed openstack-infra/storyboard master: Handle Branches from Launchpad in Migration https://review.openstack.org/437469 | 19:47 |
| ianychoi | clarkb, hello! I am fine to delete the old zanata production instance, but can I tell you later after I18n IRC meeting this week (Thu 13:00 UTC)? | 19:47 |
| clarkb | ianychoi: sure, I can wiat for then. I have already cleaned out the old dev server | 19:47 |
| *** esikachev has joined #openstack-infra | 19:47 | |
| ianychoi | clarkb, yep I have just seen your message in this channel. Fine :) | 19:49 |
| Dennis_ | It went through now. The server error seems to have disappeared. | 19:50 |
| Gravious | :) | 19:50 |
| Gravious | clarkb, fwiw, do you see any use of dulwich being any problem wrt other openstack aspects? | 19:51 |
| * Gravious doesn't want to use something if it's going to cause other problems | 19:52 | |
| *** esikachev has quit IRC | 19:52 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Fix for unpaused request handlers https://review.openstack.org/445632 | 19:56 |
| *** ihrachys has quit IRC | 19:59 | |
| *** hashar has joined #openstack-infra | 20:00 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: Add scenario005 job (4nodes) https://review.openstack.org/418098 | 20:00 |
| ianw | i hit an issue with vexxhost and []'s in the image names when using the bring-up scripts | 20:00 |
| clarkb | ianw: we don't quote properly? | 20:00 |
| *** funzo_ has joined #openstack-infra | 20:00 | |
| clarkb | I think you can use the uuid to get around weirdness like that | 20:00 |
| openstackgerrit | Ken'ichi Ohmichi proposed openstack-infra/project-config master: Fix regex for gate-puppet-openstack-integration-4.. https://review.openstack.org/445639 | 20:01 |
| ianw | https://review.openstack.org/#/c/440914/ | 20:01 |
| ianw | clarkb: yeah, i haven't got back to it for workaround purposes | 20:01 |
| clarkb | I have more info on the ansible loop getting stuck. the host shows up in nova list properly we just can't hit it and nyj01 is not in use by nodepool any longer | 20:01 |
| clarkb | any objection to me deleting the nyj01 mirror host via nova to clear that up and get ansible happy again? | 20:01 |
| jeblair | clarkb: no objection | 20:02 |
| * Gravious doesn't know whether to assume the question he asked was dumb or not | 20:02 | |
| Gravious | apologies, quite new here | 20:02 |
| openstackgerrit | Lubosz Kosnik (diltram) proposed openstack-infra/project-config master: Enable DPDK testing on gate https://review.openstack.org/445640 | 20:02 |
| clarkb | Gravious: its not sorry, just juggling multiple things | 20:02 |
| ianw | i have a change to the launch script to help with multiple hosts; it's a little WIP but comments welcome https://review.openstack.org/440928 | 20:02 |
| Gravious | a, no worries :) | 20:02 |
| clarkb | Gravious: I think my biggest concern would be needing to know how multiple git libs work | 20:02 |
| clarkb | Gravious: so slight preference for gitpython just to avoid needing to page in extra infos when switching between projects. That said license wise dulwich looks fine and if it works better for the use case then I don't see why you shouldn't use it | 20:03 |
| *** ociuhandu has quit IRC | 20:03 | |
| Gravious | clarkb, ah yes good point, hrm | 20:03 |
| jeblair | Gravious: you said pygit2 license is inappropriate? how so? | 20:03 |
| Gravious | jeblair, well it was up for debate what linking exception means in a python context iirc | 20:04 |
| *** ihrachys has joined #openstack-infra | 20:04 | |
| Gravious | (libgit2 being gplv2 with linking exception, pygit2 essentially copy-pasting that stmt) | 20:04 |
| *** funzo_ has quit IRC | 20:05 | |
| jeblair | Gravious: ah, concern that the exception would not apply since it's not a 'combined executable'. thanks. | 20:06 |
| Gravious | jeblair, yes i guess so | 20:07 |
| *** armax has quit IRC | 20:07 | |
| clarkb | Dennis_: typically that is caused by not hvaing the right foundation account type or having email addresses that don't match between gerrit and foundation account. But sonds like it got sorted out so maybe it was just temporary failure in server validating the gerrit request | 20:07 |
| *** esikachev has joined #openstack-infra | 20:08 | |
| jeblair | Gravious: https://github.com/libgit2/pygit2/issues/15 | 20:08 |
| Dennis_ | Thanks clarkb. All seems to be working great now | 20:08 |
| *** ihrachys has quit IRC | 20:08 | |
| clarkb | #status log internap nyj01 mirror host deleted because it is no longer reachable and was causing ansible runs to spin on waiting for ssh timeout | 20:09 |
| mgagne | clarkb: haven't it been decommissioned already? | 20:10 |
| Gravious | jeblair, ah, well, that's good to know | 20:11 |
| clarkb | mgagne: I think it may have effectively been yes, but nova was still reporting it as present so ansible tried to talk to it | 20:11 |
| clarkb | mgagne: so this is just syncing the nova state with reality so our tooling is happy again | 20:11 |
| openstackgerrit | Leehom Li proposed openstack/diskimage-builder master: Fix problem unable disable iptables service https://review.openstack.org/419703 | 20:11 |
| Gravious | jeblair, i expect dulwich may still be preferable here though, since it's pure python (with optional c extensions) | 20:12 |
| *** david-lyle_ is now known as david-lyle | 20:12 | |
| *** esikachev has quit IRC | 20:12 | |
| mgagne | that's concerning | 20:12 |
| mgagne | instance shouldn't go away without reasons | 20:12 |
| *** e0ne has joined #openstack-infra | 20:13 | |
| clarkb | mgagne: well its networking went away | 20:13 |
| clarkb | mgagne: not sure if instance itself was gone | 20:13 |
| clarkb | (but to us it effectively was) | 20:13 |
| clarkb | mgagne: | 66f4ed5c-1bd6-492a-b3be-a987aa55a595 | mirror.nyj01.internap.openstack.org | ACTIVE | inap-17301-WAN2344=74.217.28.58 | Ubuntu 14.04.5 LTS (Trusty Tahr) Daily Build [20160816] | was the host if you want to followup further | 20:14 |
| mgagne | deleted_at: 2017-03-14 20:05:51 | 20:14 |
| jeblair | Gravious: as for other aspects of your question, part of why we chose gitpython for zuul is *because* it used the cgit command line to perform merge operations, so we could be compatible with what devs used on their command lines (we can also get pretty close to what jgit does in gerrit, but that's straying from the point a bit). that was the main reason we focused on it rather than dulwich or pygit2. all that to say that since you ... | 20:14 |
| jeblair | ... want the opposite, it sounds like either of those packages may be more suitable to what you're doing. i suspect if we were to move away from gitpython, we would probably look at pygit2 first in an attempt to maintain maximum compatibility with cgit. or perhaps dulwich and just run os.system("git merge") when we need. | 20:14 |
| clarkb | mgagne: ya I just deleted it | 20:15 |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Better merge message for GitHub pull reqeusts https://review.openstack.org/445644 | 20:15 |
| clarkb | mgagne: but it was gone since at least 16:09UTC today | 20:15 |
| mgagne | can't do much now ¯\_(ツ)_/¯ | 20:15 |
| mgagne | hmm | 20:15 |
| mgagne | wait a second.. | 20:15 |
| *** vsaienko has quit IRC | 20:15 | |
| mgagne | ok, unrelated to something I did ~3 hours ago | 20:16 |
| mgagne | oh well.. | 20:16 |
| clarkb | this was cleanup we needed anyways. Mostly just awnted to make sure I unwinded the ansible blockage properly | 20:17 |
| clarkb | currently watching it | 20:17 |
| *** e0ne has quit IRC | 20:18 | |
| *** r-daneel has joined #openstack-infra | 20:18 | |
| *** vsaienko has joined #openstack-infra | 20:19 | |
| *** vhosakot has joined #openstack-infra | 20:20 | |
| Gravious | jeblair, ah i see, i had a leaning towards pygit2 at the start of this but chose gitpython due to licensing and other concerns. so yeah, seems like i might need to rethink... | 20:20 |
| Gravious | jeblair, thanks for the information though, that's pretty helpful :) | 20:21 |
| *** e0ne has joined #openstack-infra | 20:21 | |
| *** nadya has quit IRC | 20:22 | |
| *** esikachev has joined #openstack-infra | 20:23 | |
| *** vhosakot_ has quit IRC | 20:24 | |
| openstackgerrit | Merged openstack-infra/nodepool feature/zuulv3: Remove the --no-delete option from nodepool https://review.openstack.org/445241 | 20:24 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Add per-repo public and private keys https://review.openstack.org/406382 | 20:24 |
| *** matrohon has joined #openstack-infra | 20:24 | |
| *** gildub has joined #openstack-infra | 20:25 | |
| openstackgerrit | Lucio Seki proposed openstack-infra/puppet-openstackci master: Update single node CI's JJB to 1.6.2 https://review.openstack.org/445646 | 20:25 |
| jeblair | Gravious: cool, i hope that helps, though don't take my word for it -- i haven't thought about using gitpython for that in a few years, so it's still possible it may be usable for what you want and i'm not aware. i'm not expert in the parts of it i don't use. maybe also not in the parts of it i do use. :) | 20:26 |
| *** ociuhandu has joined #openstack-infra | 20:27 | |
| *** esikachev has quit IRC | 20:28 | |
| Gravious | jeblair, *nod* no it makes sense, i've looked through some of the gitpython source as well as the docs, i'm reasonably confident that other libs are better suited to this task | 20:28 |
| *** Dennis_ has quit IRC | 20:29 | |
| jeblair | ok good | 20:29 |
| *** EricGonczer_ has joined #openstack-infra | 20:33 | |
| *** esikachev has joined #openstack-infra | 20:33 | |
| *** dprince has joined #openstack-infra | 20:35 | |
| *** vsaienko has quit IRC | 20:36 | |
| *** vsaienko has joined #openstack-infra | 20:37 | |
| *** jkilpatr has quit IRC | 20:37 | |
| dmsimard | infra-root: I notice some of the channels didn't get their topics changed bac k | 20:37 |
| *** dhinesh has quit IRC | 20:37 | |
| *** esikachev has quit IRC | 20:37 | |
| dmsimard | And are still "Gerrit is going to be restarted due to performance problems" | 20:38 |
| *** dhinesh has joined #openstack-infra | 20:38 | |
| dmsimard | Ex real fast: #puppet-openstack, #rdo, #ara | 20:38 |
| *** dimtruck is now known as zz_dimtruck | 20:38 | |
| dmsimard | pabelanger, clarkb ^ | 20:38 |
| *** prateek has quit IRC | 20:38 | |
| clarkb | dmsimard: hrm, guessing the bot waas not in channel at the time? | 20:39 |
| clarkb | or maybe our anti flood measures are insufficient? | 20:39 |
| * clarkb pulls up logs | 20:39 | |
| dmsimard | clarkb: dunno, I'll change those back since I have +o in them but might worth looking if there is an issue | 20:39 |
| clarkb | oh meetbot filters joins parts quits in the logs :/ | 20:40 |
| clarkb | dmsimard: if you have logs with joins parts quits would be interesting to grab any for openstackstatus | 20:40 |
| clarkb | and I guess I can look and see what the bot itself has | 20:40 |
| dmsimard | clarkb: hum, yes, I have raw logs -- hang on | 20:41 |
| *** vsaienko has quit IRC | 20:43 | |
| *** jaypipes has quit IRC | 20:43 | |
| *** MasterOfBugs has joined #openstack-infra | 20:43 | |
| dmsimard | clarkb: #puppet-openstack http://paste.openstack.org/raw/602747/ | 20:44 |
| *** esikachev has joined #openstack-infra | 20:44 | |
| *** mhickey has joined #openstack-infra | 20:45 | |
| *** Goneri has quit IRC | 20:46 | |
| clarkb | dmsimard: yolanda fungi http://paste.openstack.org/show/602748/ I think that sending that same notice to twitter twice crashed the second update | 20:47 |
| clarkb | so bug in the twitter implementation | 20:47 |
| dmsimard | I was mistaken about #ara, topic is fine there -- definitely had to change #rdo back as well though -- looks like #tripleo was affected too | 20:47 |
| dmsimard | #storyboard too -- I guess it's by alphabetical order or something :) | 20:48 |
| clarkb | dmsimard: ya wouldn't surprise me | 20:48 |
| clarkb | but looks like it may have bailed out early because of twitter complaining | 20:48 |
| *** vsaienko has joined #openstack-infra | 20:48 | |
| *** armax has joined #openstack-infra | 20:48 | |
| *** esikachev has quit IRC | 20:48 | |
| clarkb | woo ansible running on its own now \o/ | 20:48 |
| *** askb has joined #openstack-infra | 20:49 | |
| *** dhajare has quit IRC | 20:49 | |
| openstackgerrit | Paul Belanger proposed openstack/diskimage-builder master: Clean up cache for ubuntu-minimal https://review.openstack.org/414728 | 20:52 |
| *** coreyob_ has joined #openstack-infra | 20:52 | |
| clarkb | is that an anti flood twitter error? | 20:55 |
| clarkb | I don't twitter so don't know much about how it protects itself against bad users | 20:55 |
| clarkb | we are up to 7k ish FDs on review.o.o now | 20:56 |
| *** vsaienko has quit IRC | 20:57 | |
| bkero | 7k open FDs? | 20:58 |
| openstackgerrit | Merged openstack-infra/nodepool feature/zuulv3: Fix race on node state check in node cleanup https://review.openstack.org/445557 | 20:58 |
| bkero | I hope we upped that limit beyond 8k | 20:58 |
| openstackgerrit | Merged openstack-infra/nodepool feature/zuulv3: Fix min-ready/max-servers in test configs https://review.openstack.org/445512 | 20:58 |
| *** jkilpatr has joined #openstack-infra | 20:58 | |
| openstackgerrit | Merged openstack-infra/nodepool feature/zuulv3: Fix for unpaused request handlers https://review.openstack.org/445632 | 20:58 |
| *** trown is now known as trown|outtypewww | 20:59 | |
| clarkb | bkero: we haven't yet | 21:00 |
| bkero | awesome | 21:00 |
| * bkero closes his gerrit FDs | 21:00 | |
| *** matrohon has quit IRC | 21:00 | |
| clarkb | 4200 ish fds appear to be connections from one IP | 21:00 |
| fungi | clarkb: can you map that back against gerrit show-connections? | 21:01 |
| bkero | Wonder who that is | 21:01 |
| openstackgerrit | Merged openstack-infra/nodepool feature/zuulv3: Add request-list nodepool command https://review.openstack.org/445169 | 21:01 |
| clarkb | I don't see user info associated with those connections in gerrit show-connections | 21:01 |
| *** funzo has joined #openstack-infra | 21:01 | |
| clarkb | fungi: ya I am just looking there | 21:01 |
| openstackgerrit | Merged openstack-infra/nodepool feature/zuulv3: Remove AllocatorTestCase and RoundRobinTestCase https://review.openstack.org/445175 | 21:01 |
| clarkb | fungi: there is no user in the user oclumn | 21:01 |
| fungi | check its ssh log maybe? | 21:01 |
| fungi | clarkb: oh, it will be a/NNNNNN for their account id number | 21:02 |
| fungi | i think? | 21:02 |
| clarkb | Hitachi-Manila-HNAS-CI | 21:02 |
| clarkb | fungi: ya but its not there under show-connections for some reason | 21:02 |
| clarkb | sshd log has it though | 21:02 |
| clarkb | should I go ahead and disable that host? | 21:02 |
| clarkb | er user | 21:02 |
| fungi | Hitachi-Manila-HNAS-CI is the same offender i saw earlier, so yes definitely | 21:03 |
| *** zz_dimtruck is now known as dimtruck | 21:03 | |
| *** vsaienko has joined #openstack-infra | 21:03 | |
| fungi | that's what was doing lots of broad change queries in parallel according to the ssh log | 21:03 |
| *** esikachev has joined #openstack-infra | 21:03 | |
| *** ociuhandu has quit IRC | 21:04 | |
| clarkb | ok its inactive now | 21:04 |
| clarkb | will send email in a minute | 21:04 |
| *** funzo has quit IRC | 21:05 | |
| fungi | clarkb: https://wiki.openstack.org/wiki/ThirdPartySystems/Hitachi_Manila_HNAS_CI says marcusvrn and ganso are contacts, maybe see if they're in the third-party channel? | 21:06 |
| clarkb | fungi: thanks | 21:06 |
| *** Apoorva has joined #openstack-infra | 21:06 | |
| clarkb | they aren't but I can send email | 21:07 |
| * fungi can't remember that channel name, should probably get added to https://wiki.openstack.org/wiki/IRC | 21:07 | |
| *** erikmwilson has quit IRC | 21:07 | |
| bkero | Sounds like some 3rd party CI | 21:07 |
| clarkb | ugh now I have to remember the rules about third part and email. maybe just cc openstack-infra and send to them | 21:07 |
| clarkb | but then that creates loda for ml admins | 21:08 |
| *** esikachev has quit IRC | 21:08 | |
| fungi | on that note, any volunteers to pitch in on moderating non-subscriber posts to the infra ml? ;) | 21:08 |
| *** erikmwilson has joined #openstack-infra | 21:08 | |
| fungi | we get one or two a week on average. plus a dozen or so spams a day you can ignore and mass discard when you go in to approve a legit posting | 21:09 |
| clarkb | third party announce thats the one | 21:09 |
| *** vsaienko has quit IRC | 21:10 | |
| fungi | Gravious: that issue thread jeblair found about pygit2 does indeed seem to clarify the licensing situation. i don't have any license-related concerns with it if you determine it's the best fit for what you're writing (there's still the non-pure-python downside, but that's tractable and not a show-stopper) | 21:11 |
| fungi | the license documentation for pynotedb should probably link to and maybe even quote some of that, though | 21:12 |
| clarkb | ok sent, hopefully I don't get moderated (pretty sure I joined that list way back when) | 21:12 |
| fungi | just in case the question comes up again | 21:12 |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: WIP: Conntainers with multi-nic and net-iso https://review.openstack.org/422448 | 21:12 |
| clarkb | fungi: I'm going to attempt closing their connections now | 21:14 |
| clarkb | via gerrit close-connection | 21:14 |
| fungi | that's worked for me on multiple occasions, but curious how long it takes to dump that many | 21:14 |
| *** e0ne has quit IRC | 21:15 | |
| openstackgerrit | Jamie Lennox proposed openstack-infra/nodepool feature/zuulv3: Allow loading logging config from yaml https://review.openstack.org/445656 | 21:15 |
| *** esikachev has joined #openstack-infra | 21:16 | |
| clarkb | 16660 and 22236 also show up | 21:16 |
| *** vsaienko has joined #openstack-infra | 21:16 | |
| clarkb | er as users with that ip addre | 21:16 |
| fungi | hitachi probably has them all behind a common nas | 21:16 |
| fungi | er, nat | 21:17 |
| fungi | (maybe nas too though!) | 21:17 |
| *** e0ne has joined #openstack-infra | 21:17 | |
| clarkb | Hitachi HBSD2 CI and Hitachi Manila HSP CI | 21:17 |
| clarkb | tempted to disable those too | 21:18 |
| jeblair | fungi, clarkb, bkero: is anyone working on fixing the statusbot twitter thing? | 21:19 |
| clarkb | jeblair: I am not, trying to fix gerrit | 21:19 |
| jeblair | i'd like to suggest that we either find someone willing to fix it quickly, or disable it. | 21:19 |
| *** ldnunes has quit IRC | 21:20 | |
| bkero | The openstack statusbot irc interface? | 21:20 |
| bkero | I didn't realize it was broken | 21:20 |
| fungi | initial triage suggests that it aborted resetting irc channel topics partway through when it received an error from the twittersphere | 21:20 |
| openstackgerrit | Morgan Fainberg proposed openstack-infra/shade master: change test_endpoints to use requests mock https://review.openstack.org/444541 | 21:21 |
| clarkb | ok I disabled those other two accounts too, will send more emails and details | 21:21 |
| fungi | clarkb: were they also opening a large number of connections? | 21:21 |
| bkero | fungi: sounds like it could just need a kick? | 21:21 |
| fungi | bkero: the bigger concern is that poor error handling in the twitter feature leads to a mess of incorrect irc channel topics someone now has to manually clean up | 21:22 |
| *** vsaienko has quit IRC | 21:22 | |
| clarkb | fungi: well show-connections only shows the accounts associated with like 5 of the total connections | 21:22 |
| clarkb | fungi: so I can't really tell | 21:22 |
| bkero | fungi: That is a problem. Let me take a look at the twitter code. | 21:22 |
| fungi | clarkb: ahh, got it | 21:22 |
| fungi | bkero: thanks for digging into the code side | 21:23 |
| *** esikachev has quit IRC | 21:23 | |
| fungi | i can try to piece together a list of channels that need their topics fixed and what they used to be (i think we have that data in the statusbot logs) | 21:23 |
| fungi | would have jumped on it sooner, but just now getting caught up after the tc meeting | 21:24 |
| *** aysyd has quit IRC | 21:25 | |
| bkero | fungi: If you still have it open, can you link me to the error log with the twitter hang? | 21:25 |
| *** gouthamr has quit IRC | 21:25 | |
| clarkb | fungi: also connection count seems to still be going up making me wonder if maybe its another account entirely at fault... | 21:26 |
| clarkb | though possibly also stale caches? I can try a flush too | 21:27 |
| *** vsaienko has joined #openstack-infra | 21:28 | |
| *** e0ne has quit IRC | 21:28 | |
| fungi | bkero: http://paste.openstack.org/show/602748/ was the paste from earlier | 21:28 |
| fungi | the theory is that twitter chokes if you try to send two identical messages in a row (perhaps within a short period of time?) | 21:29 |
| clarkb | any objection to flushing caches? | 21:29 |
| bkero | Oh okay | 21:29 |
| bkero | fungi: Yep. https://twittercommunity.com/t/status-is-a-duplicate/16928/4 | 21:29 |
| bkero | I think we can wrap that in a try: catch: | 21:29 |
| fungi | a #status notice was sent when it was intended to be a #status ok, and then the #status ok was sent immediately afterward with the same following text | 21:30 |
| *** pradk has quit IRC | 21:30 | |
| clarkb | bkero: fungi I think a simple fix would be to just catch all twitter errors, log and move on | 21:30 |
| clarkb | twitter is a secondary feature here | 21:30 |
| fungi | the twitter error hit in the middle of when statusbot was trying to finish resetting channel topics from the #status ok and so left a lot of them in the lurch when it aborted | 21:30 |
| bkero | clarkb: That's basically what I'm doing | 21:31 |
| fungi | right, i think if the bot fails to send a message to twitter, that's perfectly fine it should just move on | 21:31 |
| bkero | There is some exception handling. http://git.openstack.org/cgit/openstack-infra/statusbot/tree/statusbot/bot.py#n357 | 21:31 |
| *** felipemonteiro has quit IRC | 21:32 | |
| bkero | Ok, let me try a fix | 21:32 |
| *** e0ne has joined #openstack-infra | 21:32 | |
| clarkb | I don't hear opposition to flushing caches so doing that now | 21:34 |
| *** Jeffrey4l has quit IRC | 21:35 | |
| *** esikachev has joined #openstack-infra | 21:35 | |
| *** Jeffrey4l has joined #openstack-infra | 21:35 | |
| *** jcoufal has quit IRC | 21:36 | |
| *** tonytan_brb has quit IRC | 21:36 | |
| *** dimtruck is now known as zz_dimtruck | 21:36 | |
| fungi | clarkb: it's usually the first thing i try | 21:36 |
| *** Sukhdev has quit IRC | 21:37 | |
| *** vsaienko has quit IRC | 21:37 | |
| *** zz_dimtruck is now known as dimtruck | 21:37 | |
| *** eharney has quit IRC | 21:38 | |
| *** mhickey has quit IRC | 21:38 | |
| bkero | clarkb, jeblair, fungi: patch incoming | 21:39 |
| openstackgerrit | Ben Kero proposed openstack-infra/statusbot master: Catch exceptions from posting to Twitter https://review.openstack.org/445660 | 21:39 |
| bkero | ^ | 21:39 |
| *** esikachev has quit IRC | 21:40 | |
| clarkb | fungi: ok connection count is still climbing, I wonder if no account is associated with it because its not getting far enough to do auth properly? | 21:41 |
| *** Julien-zte has joined #openstack-infra | 21:41 | |
| openstackgerrit | Ben Kero proposed openstack-infra/statusbot master: Catch exceptions from posting to Twitter https://review.openstack.org/445660 | 21:41 |
| *** vsaienko has joined #openstack-infra | 21:41 | |
| fungi | clarkb: yeah, it's possible we need to drop that into iptables for now | 21:41 |
| clarkb | so tcp handshake is valid, now we have fd, but ssh auth is half way done | 21:41 |
| *** esikachev has joined #openstack-infra | 21:41 | |
| fungi | there are probably already some specific block rules for earlier offenders in the table | 21:42 |
| *** e0ne has quit IRC | 21:42 | |
| *** john-davidge has joined #openstack-infra | 21:42 | |
| clarkb | I don't see any current rules but I can propose a puppet change for this ip | 21:42 |
| fungi | in the past i think we've just temporarily inserted via cli | 21:43 |
| fungi | since the odds of rebooting the server or changing the ruleset while this issue is ongoing are slim | 21:43 |
| fungi | but i'm fine with it in puppet as well | 21:43 |
| clarkb | fungi: puppet will remove them | 21:43 |
| clarkb | or do you mean in a different input table? | 21:44 |
| fungi | it will? i thought it only did that if it thought the rules file changed | 21:44 |
| clarkb | fungi: oh right | 21:44 |
| clarkb | ya I got you | 21:44 |
| bkero | jeblair: So with this I'm just calling Tweet.log() as part of the exception catching, but that just passes for now. To fix that we might have to pass StatusBot's 'log' object to Tweet and stop stubbing out that function. Should I do that? | 21:44 |
| fungi | grr, it looks like maybe statusbot's debug log doesn't say what the topics were being changed from, just what they're changed to | 21:45 |
| jeblair | bkero: well Foo.log() means respend to "#status log" | 21:45 |
| *** esikachev has quit IRC | 21:46 | |
| jeblair | bkero: so it's not related to application level logging, but confusingly named | 21:46 |
| fungi | most recent successful run was on feb 28, so i'll crib from the ones it used then | 21:46 |
| bkero | jeblair: I thought it was this definition of log: http://git.openstack.org/cgit/openstack-infra/statusbot/tree/statusbot/bot.py#n188 | 21:46 |
| jeblair | fungi: i think it logs on startup and when they are changed. | 21:46 |
| *** esikachev has joined #openstack-infra | 21:46 | |
| jeblair | bkero: yeah, but that's inherited from http://git.openstack.org/cgit/openstack-infra/statusbot/tree/statusbot/bot.py#n151 | 21:46 |
| jeblair | bkero: which is the interface that the irc, wiki, and twitter updaters are implementing | 21:47 |
| *** john-davidge has quit IRC | 21:47 | |
| bkero | okay | 21:47 |
| clarkb | `iptables -I openstack-INPUT 3 -s 201.48.5.193 -j DROP` how does that look? | 21:47 |
| jeblair | bkero: so if that gets implemented, then it will be called any time someone says "#status log" | 21:47 |
| robputt_ | good evening / morning / afternoon OS infra | 21:47 |
| robputt_ | does the openstack/requirements repo come under infra ownership? | 21:47 |
| fungi | robputt_: there is a separate requirements team | 21:48 |
| robputt_ | hey thanks fungi | 21:48 |
| robputt_ | are you familiar with the repo at all? | 21:48 |
| openstackgerrit | Merged openstack-infra/tripleo-ci master: Install packages with roles in multinode https://review.openstack.org/443687 | 21:48 |
| robputt_ | basically i'd like to use some item from PyPi which is not already included... | 21:48 |
| *** kzaitsev_mb has quit IRC | 21:48 | |
| robputt_ | can I simply add it to requirements with the appropriate license or is there more to it? | 21:48 |
| fungi | robputt_: https://governance.openstack.org/tc/reference/projects/requirements.html | 21:48 |
| robputt_ | thanks fungi | 21:49 |
| fungi | robputt_: the readme in the requirements repo probably has answers to most of those questions | 21:49 |
| fungi | robputt_: also https://docs.openstack.org/developer/requirements/ | 21:50 |
| *** esikachev has quit IRC | 21:50 | |
| bkero | jeblair: okay | 21:51 |
| fungi | clarkb: lgtm | 21:51 |
| openstackgerrit | Ben Kero proposed openstack-infra/statusbot master: Catch exceptions from posting to Twitter https://review.openstack.org/445660 | 21:51 |
| bkero | jeblair: new rev | 21:51 |
| clarkb | fungi: you good with me running that now? (only slightly paranoid I might break connectivity to review.o.o forcing a reboot >_>) | 21:51 |
| *** Sukhdev has joined #openstack-infra | 21:51 | |
| fungi | clarkb: as long as its after the established allowance, it shouldn't kill your ssh session no matter what else happens | 21:52 |
| clarkb | mostly want to make sure at least one other person is around in case something derps | 21:52 |
| fungi | i'll ssh in to | 21:52 |
| fungi | too | 21:52 |
| clarkb | fungi: its before (intentionally to kill the 4500 connections we have) | 21:52 |
| clarkb | fungi: I can move the index to 4 to move it after | 21:53 |
| *** ZZelle has joined #openstack-infra | 21:53 | |
| *** rfolco has quit IRC | 21:53 | |
| *** xyang1 has quit IRC | 21:54 | |
| clarkb | though I guess gerrit may not notice those connections are dead if I just stop allowing packets to flow forcing use of close-connection anyways | 21:54 |
| clarkb | was trying to avoid that because there are 4500 | 21:54 |
| fungi | clarkb: does close-connections not work on them? | 21:54 |
| clarkb | fungi: it seems to | 21:55 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Add per-repo public and private keys https://review.openstack.org/406382 | 21:55 |
| clarkb | fungi: its jus ta lot of them to close one at a time via ssh | 21:55 |
| * clarkb not looking forward to hitting enter 4500 times for use of ssh key | 21:55 | |
| fungi | seems safer to stuff it down around 9 so it's after everything except the 29418/tcp rule | 21:55 |
| clarkb | fungi: can do | 21:56 |
| fungi | oh, hrm... i have my gerrit api ssh key agented if you'd rather i loop over those to close them | 21:56 |
| *** vsaienko has quit IRC | 21:56 | |
| *** dprince has quit IRC | 21:56 | |
| clarkb | fungi: mine is agented too, but I use -c so that I don't have t oreload keys if I need ot fowrad or do something funny | 21:56 |
| fungi | i only do that from a local machine of course | 21:56 |
| jeblair | clarkb: http://giphy.com/gifs/simpsons-dippy-bird-drinking-l41lUJ1YoZB1lHVPG | 21:56 |
| fungi | bwahahahaha | 21:57 |
| *** pradk has joined #openstack-infra | 21:57 | |
| clarkb | `iptables -I openstack-INPUT 10 -s 201.48.5.193 -j DROP` | 21:57 |
| clarkb | how is that? | 21:57 |
| fungi | i think it needs to be 9 or smaller yeah? or am i miscounting? | 21:58 |
| clarkb | fungi: they are 1 indexed iirc | 21:58 |
| *** wolverineav has quit IRC | 21:58 | |
| * clarkb checks man page | 21:58 | |
| clarkb | oh right ya | 21:58 |
| clarkb | I miscounted | 21:58 |
| clarkb | `iptables -I openstack-INPUT 9 -s 201.48.5.193 -j DROP` | 21:58 |
| *** wolverineav has joined #openstack-infra | 21:58 | |
| bkero | jeblair: isn't that why linux has /usr/bin/yes? | 21:58 |
| bkero | (Which I obviously alias to 'dippybird') | 21:59 |
| bkero | 201-048-005-193.static.ctbctelecom.com.br. wat | 21:59 |
| clarkb | ya I dunno | 21:59 |
| *** burgerk_ has joined #openstack-infra | 21:59 | |
| fungi | clarkb: yep, lgtm now | 21:59 |
| clarkb | but its going bye bye | 21:59 |
| clarkb | fungi: ok running now | 21:59 |
| clarkb | and done | 22:00 |
| fungi | it didn't kick me out of my terminal at least, so gj | 22:00 |
| clarkb | ya I'm still in. ready for you to kill connections for that IP via gerrit ssh api | 22:00 |
| fungi | on it | 22:00 |
| clarkb | I will also go ahead and reenable the accounts I disabled since dsibaling them didn't change anything and respond to the email thread | 22:01 |
| openstackgerrit | Morgan Fainberg proposed openstack-infra/shade master: change test_endpoints to use requests mock https://review.openstack.org/444541 | 22:01 |
| *** wolverin_ has joined #openstack-infra | 22:02 | |
| *** funzo has joined #openstack-infra | 22:02 | |
| fungi | loop underway. should take ~1hr at this speed | 22:02 |
| *** burgerk has quit IRC | 22:02 | |
| *** ociuhandu has joined #openstack-infra | 22:03 | |
| fungi | i guess i could have done it with the magic "Gerrit Code Review" account from localhost instead, for extra fastness | 22:03 |
| clarkb | accounts are reenabled and email is sent | 22:03 |
| *** ssbarnea has quit IRC | 22:04 | |
| *** burgerk_ has quit IRC | 22:04 | |
| *** wolverineav has quit IRC | 22:04 | |
| *** ssbarnea has joined #openstack-infra | 22:05 | |
| *** ssbarnea has quit IRC | 22:05 | |
| *** sslypushenko_ has quit IRC | 22:05 | |
| fungi | for s in `ssh -i review_site/etc/ssh_host_rsa_key -p 29418 'Ge | 22:05 |
| fungi | rrit Code Review'@127.0.0.1 gerrit show-connections -n|grep 201.48.5.193|cut -d' | 22:05 |
| fungi | ' -f1`;do ssh -i review_site/etc/ssh_host_rsa_key -p 29418 'Gerrit Code Review'@127.0.0.1 gerrit close-connection $s;donefor s in `ssh -i review_site/etc/ssh_host_rsa_key -p 29418 'Ge | 22:05 |
| fungi | rrit Code Review'@127.0.0.1 gerrit show-connections -n|grep 201.48.5.193|cut -d' | 22:05 |
| fungi | ' -f1`;do ssh -i review_site/etc/ssh_host_rsa_key -p 29418 'Gerrit Code Review'@127.0.0.1 gerrit close-connection $s;donefor s in `ssh -i review_site/etc/ssh_host_rsa_key -p 29418 'Ge | 22:05 |
| fungi | rrit Code Review'@127.0.0.1 gerrit show-connections -n|grep 201.48.5.193|cut -d' | 22:06 |
| fungi | gah, mispaste | 22:06 |
| fungi | and my paste button is bouncing | 22:06 |
| fungi | sorry | 22:06 |
| fungi | anyway, that's going, like, 10x as fast | 22:06 |
| *** wolverin_ has quit IRC | 22:06 | |
| *** funzo has quit IRC | 22:06 | |
| *** ihrachys has joined #openstack-infra | 22:08 | |
| clarkb | down to 3.1k connections so its working | 22:09 |
| *** claudiub has quit IRC | 22:11 | |
| *** kgiusti has left #openstack-infra | 22:14 | |
| fungi | finished | 22:15 |
| *** esberglu has quit IRC | 22:15 | |
| fungi | that was way faster | 22:16 |
| *** ihrachys has quit IRC | 22:16 | |
| fungi | for future reference, `ssh -i review_site/etc/ssh_host_rsa_key -p 29418 'Gerrit Code Review'@127.0.0.1 ...` | 22:16 |
| fungi | (su'd to the gerrit2 user) | 22:17 |
| clarkb | with that we are down to 250 ish connections | 22:17 |
| fungi | very useful for running gerrit api commands from the server | 22:17 |
| *** gouthamr has joined #openstack-infra | 22:19 | |
| *** hashar has quit IRC | 22:19 | |
| *** Julien-zte has quit IRC | 22:19 | |
| *** Julien-z_ has joined #openstack-infra | 22:19 | |
| clarkb | and open files graph has plummetted to | 22:20 |
| *** esikachev has joined #openstack-infra | 22:20 | |
| *** erhudy has quit IRC | 22:20 | |
| *** priteau has quit IRC | 22:20 | |
| *** hashar has joined #openstack-infra | 22:20 | |
| *** wolverineav has joined #openstack-infra | 22:23 | |
| *** jcoufal has joined #openstack-infra | 22:24 | |
| *** esikachev has quit IRC | 22:24 | |
| *** Goneri has joined #openstack-infra | 22:25 | |
| *** ssbarnea has joined #openstack-infra | 22:27 | |
| *** wolverineav has quit IRC | 22:27 | |
| clarkb | #status log review.openstack.org is blocking 201.48.5.193 in iptables as it was the source of thousands of stale ssh connections causing the server to crash. | 22:27 |
| openstackgerrit | Felipe Monteiro proposed openstack-infra/project-config master: Remove non-voting glare gates from murano. https://review.openstack.org/445626 | 22:32 |
| openstackgerrit | Morgan Fainberg proposed openstack-infra/shade master: Convert simple role crud to requests mock https://review.openstack.org/445673 | 22:32 |
| *** EricGonczer_ has quit IRC | 22:33 | |
| *** EricGonczer_ has joined #openstack-infra | 22:33 | |
| *** esikachev has joined #openstack-infra | 22:34 | |
| *** EricGonczer_ has quit IRC | 22:38 | |
| *** rhallisey has quit IRC | 22:38 | |
| *** wolverineav has joined #openstack-infra | 22:38 | |
| *** esikachev has quit IRC | 22:39 | |
| *** ym200 has joined #openstack-infra | 22:41 | |
| *** wolverineav has quit IRC | 22:42 | |
| *** pradk has quit IRC | 22:44 | |
| *** ijw has joined #openstack-infra | 22:45 | |
| *** gordc has quit IRC | 22:51 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: Add scenario005 job (4nodes) https://review.openstack.org/418098 | 22:52 |
| *** beslemon has joined #openstack-infra | 22:52 | |
| *** rfolco has joined #openstack-infra | 22:54 | |
| *** beslemon has quit IRC | 22:55 | |
| *** esikachev has joined #openstack-infra | 22:56 | |
| fungi | okay, the remaining irc topic cleanup has been applied semi-manually | 22:56 |
| fungi | the only one i couldn't manage was rally, throwing a syntax error and i can't figure out why | 22:56 |
| fungi | might have to do with their utf-8 cloud symbol separators, i'm not sure | 22:57 |
| *** beslemon has joined #openstack-infra | 22:57 | |
| *** smarcet has quit IRC | 22:57 | |
| openstackgerrit | Jamie Lennox proposed openstack-infra/nodepool feature/zuulv3: Refactor nodepool apps into base app https://review.openstack.org/445674 | 22:58 |
| openstackgerrit | Jamie Lennox proposed openstack-infra/nodepool feature/zuulv3: Split webapp into its own nodepool application https://review.openstack.org/445675 | 22:58 |
| fungi | aha, it was working, just too long | 22:58 |
| fungi | well, they can clean it up more than what i've done if they need to | 22:59 |
| *** esikachev has quit IRC | 23:00 | |
| *** beslemon has quit IRC | 23:01 | |
| *** smarcet has joined #openstack-infra | 23:01 | |
| *** rbrndt__ has quit IRC | 23:01 | |
| openstackgerrit | Jamie Lennox proposed openstack-infra/nodepool feature/zuulv3: Split webapp into its own nodepool application https://review.openstack.org/445675 | 23:02 |
| openstackgerrit | Jamie Lennox proposed openstack-infra/nodepool feature/zuulv3: Refactor nodepool apps into base app https://review.openstack.org/445674 | 23:02 |
| *** Julien-z_ has quit IRC | 23:02 | |
| *** funzo has joined #openstack-infra | 23:03 | |
| *** Julien-zte has joined #openstack-infra | 23:03 | |
| *** Julien-zte has quit IRC | 23:03 | |
| *** Julien-zte has joined #openstack-infra | 23:04 | |
| *** Julien-zte has quit IRC | 23:04 | |
| *** Julien-zte has joined #openstack-infra | 23:04 | |
| *** rhallisey has joined #openstack-infra | 23:04 | |
| *** Julien-zte has quit IRC | 23:05 | |
| *** esikachev has joined #openstack-infra | 23:06 | |
| *** funzo has quit IRC | 23:08 | |
| *** beslemon has joined #openstack-infra | 23:10 | |
| *** beslemon has quit IRC | 23:11 | |
| *** beslemon has joined #openstack-infra | 23:11 | |
| *** Goneri has quit IRC | 23:12 | |
| *** kzaitsev_mb has joined #openstack-infra | 23:14 | |
| *** beslemon has quit IRC | 23:14 | |
| *** beslemon has joined #openstack-infra | 23:14 | |
| *** esikachev has quit IRC | 23:15 | |
| clarkb | fungi: or other infra-root want to reivew https://review.openstack.org/#/c/427362/2 to help neutron with test log indexing | 23:17 |
| *** beslemon has quit IRC | 23:21 | |
| *** beslemon has joined #openstack-infra | 23:22 | |
| *** beslemon has quit IRC | 23:23 | |
| *** baoli has quit IRC | 23:26 | |
| *** sdague has quit IRC | 23:29 | |
| *** dimtruck is now known as zz_dimtruck | 23:31 | |
| *** erikmwilson has quit IRC | 23:37 | |
| *** erikmwilson has joined #openstack-infra | 23:38 | |
| *** beslemon has joined #openstack-infra | 23:39 | |
| *** panda is now known as panda|zZ | 23:42 | |
| *** EricGonczer_ has joined #openstack-infra | 23:42 | |
| *** beslemon has quit IRC | 23:44 | |
| *** hongbin has quit IRC | 23:45 | |
| *** beslemon has joined #openstack-infra | 23:47 | |
| *** dtrainor has joined #openstack-infra | 23:48 | |
| *** wolverineav has joined #openstack-infra | 23:49 | |
| dtrainor | Howdy. I'm having connectivity issues connecting to a few subdomains under openstack.org. I've been able to conclude with confidence that it's just my ISP so I think it will just magically resolve itself, but I did want to advertise this fact. | 23:50 |
| *** marst has quit IRC | 23:50 | |
| clarkb | dtrainor: is one of the hosts review.openstack.org? | 23:51 |
| *** hashar has quit IRC | 23:51 | |
| dtrainor | It is, paste.openstack.org is another one. | 23:51 |
| clarkb | (we had to block an IP earlier today that was DoSing us) | 23:51 |
| dtrainor | oh no | 23:51 |
| clarkb | ah we didn't block paste.o.o though | 23:51 |
| *** jcoufal has quit IRC | 23:51 | |
| clarkb | dtrainor: the other thing to check (because it seems frequent enough) is sometimes ipv6 isn't reliable on the remote end and forcing ipv4 magically solves things | 23:51 |
| openstackgerrit | Merged openstack-infra/system-config master: Index neutron functional and fullstack index logs in logstash https://review.openstack.org/427362 | 23:52 |
| dtrainor | i've tried various -4/-6 tests and end up with the same result, just a boring Network is unreachable error | 23:52 |
| clarkb | dtrainor: 201.48.5.193 is the source IP we had to block if that is you | 23:53 |
| *** wolverineav has quit IRC | 23:53 | |
| dtrainor | nope, i'm 98.165.225.115 | 23:54 |
| clarkb | ok we can rule that cause out then. | 23:54 |
| clarkb | just wanted to make sure it wasn't you and if it was work on fixing whatever insisted on having thousands of connections open | 23:54 |
| clarkb | :) | 23:54 |
| dtrainor | na, i'm not a great developer but i'm not *that* bad | 23:56 |
| dtrainor | i'll give it overnight and see if it clears up, otherwise i'll get on the phone with my isp | 23:56 |
| clarkb | mtr is a useful tool for seeing where packets are going away | 23:57 |
| dtrainor | EmilienM suggested i let you guys know, just in case | 23:57 |
| dtrainor | yep, i like that one | 23:57 |
| *** adriant has joined #openstack-infra | 23:58 | |
| *** wolverineav has joined #openstack-infra | 23:58 | |
| *** wolverin_ has joined #openstack-infra | 23:59 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!