| *** markvoelker has quit IRC | 00:06 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: [wip] fix opensuse pip-and-virtualenv https://review.openstack.org/638594 | 00:09 |
|---|---|---|
| *** pcrews has joined #openstack-infra | 00:31 | |
| *** yamamoto has joined #openstack-infra | 00:36 | |
| *** yamamoto has quit IRC | 00:41 | |
| *** sdake has joined #openstack-infra | 00:46 | |
| *** gregoryo has joined #openstack-infra | 00:46 | |
| *** sdake has quit IRC | 01:03 | |
| *** sdake has joined #openstack-infra | 01:06 | |
| *** wolverineav has joined #openstack-infra | 01:06 | |
| ianw | i think gentoo is rebuilding enough stuff that it might be in danger of hitting timeouts | 01:07 |
| ianw | it seems to be rebuilding gcc | 01:08 |
| *** hwoarang has quit IRC | 01:14 | |
| *** hwoarang has joined #openstack-infra | 01:16 | |
| *** sdake has quit IRC | 01:19 | |
| *** armax has quit IRC | 01:28 | |
| *** bhavikdbavishi has joined #openstack-infra | 01:28 | |
| *** dave-mccowan has joined #openstack-infra | 01:30 | |
| *** dave-mccowan has quit IRC | 01:35 | |
| *** wolverineav has quit IRC | 01:36 | |
| *** armax has joined #openstack-infra | 01:42 | |
| *** yamamoto has joined #openstack-infra | 02:06 | |
| *** markvoelker has joined #openstack-infra | 02:07 | |
| *** yamamoto has quit IRC | 02:09 | |
| *** rkukura has quit IRC | 02:11 | |
| ianw | took 1+h ... i guess over time we'll see this in the graphs now | 02:17 |
| *** jamesmcarthur has joined #openstack-infra | 02:17 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: [wip] fix opensuse pip-and-virtualenv https://review.openstack.org/638594 | 02:19 |
| *** armax has quit IRC | 02:23 | |
| *** dave-mccowan has joined #openstack-infra | 02:24 | |
| *** rkukura has joined #openstack-infra | 02:32 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: [wip] fix opensuse pip-and-virtualenv https://review.openstack.org/638594 | 02:34 |
| *** markvoelker has quit IRC | 02:37 | |
| *** bhavikdbavishi has quit IRC | 02:50 | |
| *** hongbin has joined #openstack-infra | 02:52 | |
| *** wolverineav has joined #openstack-infra | 02:53 | |
| *** dave-mccowan has quit IRC | 02:53 | |
| *** jamesmcarthur has quit IRC | 02:59 | |
| *** jamesmcarthur has joined #openstack-infra | 03:00 | |
| *** sdake has joined #openstack-infra | 03:02 | |
| *** wolverineav has quit IRC | 03:02 | |
| *** jamesmcarthur has quit IRC | 03:05 | |
| *** whoami-rajat has joined #openstack-infra | 03:12 | |
| *** yamamoto has joined #openstack-infra | 03:14 | |
| *** ykarel|away has joined #openstack-infra | 03:18 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Fix opensuse 42.3 pip-and-virtualenv https://review.openstack.org/638594 | 03:20 |
| *** ykarel|away is now known as ykarel | 03:20 | |
| *** ramishra has joined #openstack-infra | 03:21 | |
| *** hongbin has quit IRC | 03:30 | |
| *** jamesmcarthur has joined #openstack-infra | 03:30 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: executor: add zuul_release_nodeset Ansible action https://review.openstack.org/639002 | 03:32 |
| *** markvoelker has joined #openstack-infra | 03:35 | |
| *** janki has joined #openstack-infra | 03:35 | |
| *** sdake has quit IRC | 03:36 | |
| *** jamesmcarthur has quit IRC | 03:36 | |
| *** sdake has joined #openstack-infra | 03:42 | |
| *** jamesmcarthur has joined #openstack-infra | 03:45 | |
| *** jamesmcarthur has quit IRC | 03:51 | |
| *** udesale has joined #openstack-infra | 03:51 | |
| *** jamesmcarthur has joined #openstack-infra | 03:53 | |
| *** sdake has quit IRC | 03:56 | |
| *** bhavikdbavishi has joined #openstack-infra | 03:56 | |
| *** yamamoto has quit IRC | 03:57 | |
| *** markvoelker has quit IRC | 04:03 | |
| *** yamamoto has joined #openstack-infra | 04:03 | |
| *** jamesmcarthur has quit IRC | 04:09 | |
| *** jamesmcarthur has joined #openstack-infra | 04:12 | |
| *** wolverineav has joined #openstack-infra | 04:12 | |
| *** wolverineav has quit IRC | 04:17 | |
| *** udesale has quit IRC | 04:21 | |
| *** sdake has joined #openstack-infra | 04:22 | |
| *** udesale has joined #openstack-infra | 04:22 | |
| *** spsurya has joined #openstack-infra | 04:25 | |
| *** sdake has quit IRC | 04:25 | |
| *** sdake has joined #openstack-infra | 04:28 | |
| *** jamesmcarthur has quit IRC | 04:30 | |
| *** sdake has quit IRC | 04:30 | |
| *** sdake has joined #openstack-infra | 04:31 | |
| *** jamesmcarthur has joined #openstack-infra | 04:31 | |
| *** sdake has quit IRC | 04:35 | |
| *** sdake has joined #openstack-infra | 04:36 | |
| *** sdake has quit IRC | 04:40 | |
| *** sdake has joined #openstack-infra | 04:41 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config master: DIB grafana dashboard small fixes https://review.openstack.org/639007 | 04:44 |
| *** sdake has quit IRC | 04:45 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Fix opensuse 42.3 pip-and-virtualenv https://review.openstack.org/638594 | 04:45 |
| *** sdake has joined #openstack-infra | 04:51 | |
| *** sdake has quit IRC | 04:55 | |
| *** sdake has joined #openstack-infra | 04:56 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config master: DIB grafana dashboard small fixes https://review.openstack.org/639007 | 04:59 |
| *** yamamoto has quit IRC | 04:59 | |
| *** sdake has quit IRC | 05:00 | |
| *** markvoelker has joined #openstack-infra | 05:00 | |
| *** janki has quit IRC | 05:02 | |
| *** sdake has joined #openstack-infra | 05:02 | |
| *** jamesmcarthur has quit IRC | 05:05 | |
| *** sdake has quit IRC | 05:10 | |
| *** sdake_ has joined #openstack-infra | 05:12 | |
| *** bhavikdbavishi has quit IRC | 05:12 | |
| *** bhavikdbavishi has joined #openstack-infra | 05:13 | |
| *** sdake_ has quit IRC | 05:15 | |
| *** sdake has joined #openstack-infra | 05:16 | |
| openstackgerrit | Merged openstack-infra/project-config master: DIB grafana dashboard small fixes https://review.openstack.org/639007 | 05:18 |
| *** sdake has quit IRC | 05:21 | |
| *** sdake has joined #openstack-infra | 05:22 | |
| *** sdake has quit IRC | 05:26 | |
| *** sdake has joined #openstack-infra | 05:29 | |
| *** sdake has quit IRC | 05:30 | |
| *** sdake_ has joined #openstack-infra | 05:31 | |
| *** markvoelker has quit IRC | 05:33 | |
| *** jamesmcarthur has joined #openstack-infra | 05:35 | |
| *** sdake_ has quit IRC | 05:35 | |
| openstackgerrit | Merged openstack/os-performance-tools master: Update home-page https://review.openstack.org/630342 | 05:36 |
| *** sdake has joined #openstack-infra | 05:36 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: Add python-path option to label https://review.openstack.org/637338 | 05:37 |
| openstackgerrit | Ian Wienand proposed openstack-infra/nodepool master: [dnm] testing rhel8 images https://review.openstack.org/639013 | 05:39 |
| *** jamesmcarthur has quit IRC | 05:40 | |
| *** sdake has quit IRC | 05:45 | |
| *** bhavikdbavishi has quit IRC | 05:45 | |
| *** wolverineav has joined #openstack-infra | 05:47 | |
| *** wolverineav has quit IRC | 05:48 | |
| *** sdake has joined #openstack-infra | 05:48 | |
| *** wolverineav has joined #openstack-infra | 05:48 | |
| *** sdake has quit IRC | 05:51 | |
| *** sdake has joined #openstack-infra | 05:53 | |
| *** sdake has quit IRC | 05:55 | |
| *** sdake has joined #openstack-infra | 05:57 | |
| *** sdake has quit IRC | 06:00 | |
| *** dpawlik has joined #openstack-infra | 06:01 | |
| *** sdake has joined #openstack-infra | 06:02 | |
| *** larainema has quit IRC | 06:02 | |
| *** sdake has quit IRC | 06:06 | |
| *** sdake_ has joined #openstack-infra | 06:06 | |
| *** sdake_ has quit IRC | 06:10 | |
| *** jamesmcarthur has joined #openstack-infra | 06:11 | |
| *** jamesmcarthur has quit IRC | 06:16 | |
| *** jtomasek has joined #openstack-infra | 06:17 | |
| openstackgerrit | Noam Angel proposed openstack/diskimage-builder master: Add option to skip update packages https://review.openstack.org/636060 | 06:31 |
| openstackgerrit | Ghanshyam Mann proposed openstack-infra/openstack-zuul-jobs master: Add bionic nodeset for legacy job to switch to bionic https://review.openstack.org/639018 | 06:35 |
| *** chandankumar has quit IRC | 06:39 | |
| *** chandankumar has joined #openstack-infra | 06:40 | |
| *** quiquell|off is now known as quiquell | 06:48 | |
| *** jamesmcarthur has joined #openstack-infra | 06:53 | |
| *** ykarel has quit IRC | 06:57 | |
| *** jamesmcarthur has quit IRC | 06:58 | |
| *** ykarel has joined #openstack-infra | 06:58 | |
| *** AJaeger has quit IRC | 07:05 | |
| *** slaweq has joined #openstack-infra | 07:11 | |
| openstackgerrit | Chandan Kumar proposed openstack-infra/project-config master: Add github gnocchixyz/gnocchi to project list https://review.openstack.org/636093 | 07:15 |
| *** aojea has joined #openstack-infra | 07:20 | |
| *** pgaxatte has joined #openstack-infra | 07:23 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Fix opensuse 42.3 pip-and-virtualenv https://review.openstack.org/638594 | 07:24 |
| *** AJaeger has joined #openstack-infra | 07:24 | |
| *** udesale has quit IRC | 07:27 | |
| *** udesale has joined #openstack-infra | 07:27 | |
| *** bhavikdbavishi has joined #openstack-infra | 07:30 | |
| *** yamamoto has joined #openstack-infra | 07:30 | |
| *** markvoelker has joined #openstack-infra | 07:30 | |
| *** jamesmcarthur has joined #openstack-infra | 07:33 | |
| *** yamamoto has quit IRC | 07:34 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: Add depends-on support to frozen jobs API https://review.openstack.org/639022 | 07:35 |
| *** jamesmcarthur has quit IRC | 07:37 | |
| *** florianf has joined #openstack-infra | 07:39 | |
| *** yamamoto has joined #openstack-infra | 07:41 | |
| *** yamamoto has quit IRC | 07:41 | |
| *** yamamoto has joined #openstack-infra | 07:44 | |
| *** ykarel is now known as ykarel|lunch | 07:45 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Fix missing semaphore release around dequeue https://review.openstack.org/639023 | 07:46 |
| *** bhavikdbavishi has quit IRC | 07:54 | |
| *** wolverineav has quit IRC | 07:57 | |
| *** kjackal has joined #openstack-infra | 08:01 | |
| *** ginopc has joined #openstack-infra | 08:03 | |
| *** markvoelker has quit IRC | 08:03 | |
| *** kopecmartin|off is now known as kopecmartin | 08:06 | |
| *** ykarel_ has joined #openstack-infra | 08:09 | |
| *** ykarel|lunch has quit IRC | 08:11 | |
| *** ykarel_ is now known as ykarel | 08:11 | |
| openstackgerrit | Merged openstack/diskimage-builder master: set rhel minor release https://review.openstack.org/636063 | 08:11 |
| *** kjackal has quit IRC | 08:12 | |
| *** kjackal has joined #openstack-infra | 08:13 | |
| kashyap | Hey, folks -- indexing is behind by 76 hours: http://status.openstack.org/elastic-recheck/ | 08:14 |
| *** tkajinam has quit IRC | 08:14 | |
| kashyap | Anyone know when will it start indexing again? | 08:14 |
| *** takamatsu_ has quit IRC | 08:19 | |
| *** dtantsur|afk is now known as dtantsur | 08:21 | |
| *** takamatsu_ has joined #openstack-infra | 08:21 | |
| *** bhavikdbavishi has joined #openstack-infra | 08:23 | |
| *** florianf has quit IRC | 08:30 | |
| *** electrofelix has joined #openstack-infra | 08:31 | |
| *** quiquell is now known as quiquell|brb | 08:31 | |
| *** takamatsu_ has quit IRC | 08:32 | |
| *** jamesmcarthur has joined #openstack-infra | 08:33 | |
| *** zbr|ssbarnea has joined #openstack-infra | 08:34 | |
| *** jamesmcarthur has quit IRC | 08:38 | |
| *** takamatsu_ has joined #openstack-infra | 08:38 | |
| *** ccamacho has joined #openstack-infra | 08:45 | |
| openstackgerrit | Helena proposed openstack-infra/project-config master: Add rsd-virt-for-nova project https://review.openstack.org/638633 | 08:46 |
| *** iurygregory has joined #openstack-infra | 08:46 | |
| *** jpena|off is now known as jpena | 08:48 | |
| openstackgerrit | Merged openstack/diskimage-builder master: update spelling errors https://review.openstack.org/629096 | 08:49 |
| *** e0ne has joined #openstack-infra | 08:51 | |
| *** stakeda has joined #openstack-infra | 08:52 | |
| *** tosky has joined #openstack-infra | 08:55 | |
| kashyap | clarkb: When you're about: https://review.openstack.org/#/c/638757/ ("Capture the content of 'audit.log' file) | 08:56 |
| *** yboaron has joined #openstack-infra | 08:57 | |
| *** rcernin has quit IRC | 09:00 | |
| *** markvoelker has joined #openstack-infra | 09:01 | |
| *** roman_g has joined #openstack-infra | 09:02 | |
| *** yboaron_ has joined #openstack-infra | 09:02 | |
| *** jpich has joined #openstack-infra | 09:03 | |
| *** yboaron has quit IRC | 09:05 | |
| *** quiquell|brb is now known as quiquell | 09:07 | |
| *** shardy has joined #openstack-infra | 09:08 | |
| *** gregoryo has quit IRC | 09:08 | |
| *** jamesmcarthur has joined #openstack-infra | 09:10 | |
| *** cmorpheus is now known as cmurphy | 09:12 | |
| *** zbr|ssbarnea is now known as zbr|out | 09:13 | |
| *** jamesmcarthur has quit IRC | 09:14 | |
| *** shardy has quit IRC | 09:23 | |
| *** shardy has joined #openstack-infra | 09:23 | |
| *** owalsh_ is now known as owalsh | 09:23 | |
| *** derekh has joined #openstack-infra | 09:30 | |
| *** yamamoto has quit IRC | 09:32 | |
| *** markvoelker has quit IRC | 09:34 | |
| *** wolverineav has joined #openstack-infra | 09:36 | |
| *** ccamacho has quit IRC | 09:38 | |
| *** wolverineav has quit IRC | 09:41 | |
| *** jamesmcarthur has joined #openstack-infra | 09:49 | |
| *** gfidente has joined #openstack-infra | 09:52 | |
| *** jamesmcarthur has quit IRC | 09:54 | |
| *** jamesmcarthur has joined #openstack-infra | 09:59 | |
| *** sshnaidm|off is now known as sshnaidm | 10:05 | |
| *** ykarel_ has joined #openstack-infra | 10:12 | |
| *** ccamacho has joined #openstack-infra | 10:13 | |
| *** ykarel has quit IRC | 10:14 | |
| *** yamamoto has joined #openstack-infra | 10:19 | |
| *** yamamoto has quit IRC | 10:20 | |
| frickler | kashyap: there seems to be an issue with the logstash-workers indeed | 10:20 |
| kashyap | Nod | 10:20 |
| frickler | infra-root: on ls-worker01 all logs end more than two weeks ago with a connection refused message similar to this: http://paste.openstack.org/show/746020/ | 10:21 |
| *** ykarel_ is now known as ykarel | 10:21 | |
| frickler | I'll leave them in this state for now in case somebody wants to investigate further. maybe they all just need a restart | 10:21 |
| *** ginopc has quit IRC | 10:22 | |
| *** helenafm has joined #openstack-infra | 10:23 | |
| *** markvoelker has joined #openstack-infra | 10:31 | |
| *** yamamoto has joined #openstack-infra | 10:34 | |
| *** jaosorior has joined #openstack-infra | 10:37 | |
| *** yamamoto has quit IRC | 10:39 | |
| *** ccamacho has quit IRC | 10:43 | |
| *** ccamacho has joined #openstack-infra | 10:43 | |
| *** ccamacho has quit IRC | 10:45 | |
| *** trident has quit IRC | 10:47 | |
| *** yamamoto has joined #openstack-infra | 10:49 | |
| *** trident has joined #openstack-infra | 10:50 | |
| *** stakeda has quit IRC | 10:50 | |
| *** jamesmcarthur has quit IRC | 10:50 | |
| *** yamamoto has quit IRC | 10:51 | |
| *** markvoelker has quit IRC | 11:04 | |
| *** yamamoto has joined #openstack-infra | 11:04 | |
| openstackgerrit | Merged openstack-infra/glean master: write dns info to networkd https://review.openstack.org/610107 | 11:05 |
| *** yamamoto has quit IRC | 11:09 | |
| *** ccamacho has joined #openstack-infra | 11:12 | |
| *** ccamacho has quit IRC | 11:13 | |
| *** ccamacho has joined #openstack-infra | 11:14 | |
| *** udesale has quit IRC | 11:18 | |
| openstackgerrit | Frode Nordahl proposed openstack-infra/project-config master: Add ``ceph-rbd-mirror`` charm and dependencies https://review.openstack.org/639066 | 11:21 |
| *** snapiri has quit IRC | 11:23 | |
| *** bhavikdbavishi has quit IRC | 11:27 | |
| *** ccamacho has quit IRC | 11:28 | |
| *** kjackal has quit IRC | 11:34 | |
| *** finucannot is now known as stephenfin | 11:40 | |
| *** sdake has joined #openstack-infra | 11:46 | |
| *** sdake has quit IRC | 11:50 | |
| *** sdake has joined #openstack-infra | 11:51 | |
| *** sdake has quit IRC | 11:55 | |
| *** sdake has joined #openstack-infra | 11:56 | |
| *** ccamacho has joined #openstack-infra | 11:59 | |
| *** sdake has quit IRC | 12:00 | |
| *** markvoelker has joined #openstack-infra | 12:01 | |
| *** sdake has joined #openstack-infra | 12:01 | |
| *** sdake has quit IRC | 12:02 | |
| *** panda|ruck is now known as panda|ruck|lunch | 12:02 | |
| *** jamesmcarthur has joined #openstack-infra | 12:02 | |
| *** jamesmcarthur has quit IRC | 12:08 | |
| *** yamamoto has joined #openstack-infra | 12:08 | |
| *** quiquell is now known as quiquell|lunch | 12:13 | |
| *** jamesmcarthur has joined #openstack-infra | 12:13 | |
| *** jamesmcarthur has quit IRC | 12:19 | |
| *** jamesmcarthur has joined #openstack-infra | 12:22 | |
| *** kjackal has joined #openstack-infra | 12:23 | |
| *** quiquell|lunch is now known as quiquell | 12:28 | |
| *** jpena is now known as jpena|lunch | 12:31 | |
| *** bhavikdbavishi has joined #openstack-infra | 12:33 | |
| *** markvoelker has quit IRC | 12:35 | |
| *** yamamoto has quit IRC | 12:37 | |
| *** yamamoto has joined #openstack-infra | 12:37 | |
| *** udesale has joined #openstack-infra | 12:42 | |
| *** jamesmcarthur has quit IRC | 12:43 | |
| *** electrofelix has quit IRC | 12:44 | |
| *** rh-jelabarre has joined #openstack-infra | 12:46 | |
| *** e0ne has quit IRC | 12:47 | |
| *** jamesmcarthur has joined #openstack-infra | 12:53 | |
| *** luizbag has joined #openstack-infra | 12:57 | |
| *** jamesmcarthur has quit IRC | 13:01 | |
| *** jamesmcarthur has joined #openstack-infra | 13:05 | |
| *** jistr is now known as jistr|mtg | 13:07 | |
| *** jistr|mtg is now known as jistr | 13:08 | |
| *** jamesmcarthur has quit IRC | 13:10 | |
| *** rlandy has joined #openstack-infra | 13:15 | |
| *** jamesmcarthur has joined #openstack-infra | 13:18 | |
| *** bhavikdbavishi has quit IRC | 13:22 | |
| *** panda|ruck|lunch is now known as panda|ruck | 13:26 | |
| *** jpena|lunch is now known as jpena | 13:28 | |
| *** jamesmcarthur has quit IRC | 13:29 | |
| *** markvoelker has joined #openstack-infra | 13:32 | |
| *** e0ne has joined #openstack-infra | 13:35 | |
| *** snapiri has joined #openstack-infra | 13:37 | |
| *** udesale has quit IRC | 13:37 | |
| *** udesale has joined #openstack-infra | 13:38 | |
| openstackgerrit | Ghanshyam Mann proposed openstack-infra/openstack-zuul-jobs master: Add bionic nodeset for legacy job to switch to bionic https://review.openstack.org/639018 | 13:46 |
| *** jamesmcarthur has joined #openstack-infra | 13:50 | |
| openstackgerrit | Ghanshyam Mann proposed openstack-infra/openstack-zuul-jobs master: Migrating 'legacy-dsvm-base' from Xenial to Bionic https://review.openstack.org/639096 | 13:58 |
| openstackgerrit | Ghanshyam Mann proposed openstack-infra/openstack-zuul-jobs master: WIP: Migrating 'legacy-dsvm-base' from Xenial to Bionic https://review.openstack.org/639096 | 13:59 |
| *** jcoufal has joined #openstack-infra | 14:01 | |
| *** fried_rice is now known as efried | 14:04 | |
| *** ykarel_ has joined #openstack-infra | 14:04 | |
| *** markvoelker has quit IRC | 14:05 | |
| *** dpawlik has quit IRC | 14:06 | |
| *** ykarel has quit IRC | 14:07 | |
| *** dave-mccowan has joined #openstack-infra | 14:07 | |
| *** jamesmcarthur has quit IRC | 14:08 | |
| *** owalsh_ has joined #openstack-infra | 14:08 | |
| *** owalsh has quit IRC | 14:09 | |
| openstackgerrit | Ghanshyam Mann proposed openstack-infra/openstack-zuul-jobs master: WIP: Migrating 'legacy-dsvm-base' from Xenial to Bionic https://review.openstack.org/639096 | 14:19 |
| *** owalsh has joined #openstack-infra | 14:19 | |
| *** owalsh_ has quit IRC | 14:20 | |
| *** jamesmcarthur has joined #openstack-infra | 14:25 | |
| *** zul has joined #openstack-infra | 14:32 | |
| *** kgiusti has joined #openstack-infra | 14:40 | |
| *** eharney has joined #openstack-infra | 14:40 | |
| *** jesusaur has quit IRC | 14:42 | |
| *** mriedem has joined #openstack-infra | 14:47 | |
| *** jesusaur has joined #openstack-infra | 14:47 | |
| *** ekultails has joined #openstack-infra | 14:51 | |
| *** ykarel_ is now known as ykarel|away | 14:54 | |
| *** bhavikdbavishi has joined #openstack-infra | 14:55 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Fix rare semaphore leak during reconfiguration https://review.openstack.org/639118 | 14:57 |
| *** udesale has quit IRC | 15:02 | |
| *** e0ne has quit IRC | 15:03 | |
| *** e0ne has joined #openstack-infra | 15:05 | |
| *** kjackal has quit IRC | 15:05 | |
| *** ykarel|away has quit IRC | 15:11 | |
| *** beekneemech is now known as bnemec | 15:21 | |
| *** bhavikdbavishi has quit IRC | 15:31 | |
| *** armax has joined #openstack-infra | 15:33 | |
| *** agopi has quit IRC | 15:40 | |
| openstackgerrit | Logan V proposed openstack/diskimage-builder master: Fix cloud-init element for Fedora https://review.openstack.org/639136 | 15:42 |
| *** chandankumar is now known as raukadah | 15:51 | |
| *** kjackal has joined #openstack-infra | 15:52 | |
| openstackgerrit | Merged openstack-infra/zuul master: config: add playbooks to job.toDict() https://review.openstack.org/621343 | 15:52 |
| *** yamamoto has quit IRC | 15:52 | |
| *** yamamoto has joined #openstack-infra | 15:54 | |
| *** yamamoto has quit IRC | 15:54 | |
| openstackgerrit | Logan V proposed openstack/diskimage-builder master: Allow specification of filesystem journal size https://review.openstack.org/633368 | 15:55 |
| *** sreejithp has joined #openstack-infra | 15:57 | |
| *** bhavikdbavishi has joined #openstack-infra | 15:57 | |
| *** diablo_rojo has joined #openstack-infra | 15:58 | |
| clarkb | morning | 16:05 |
| clarkb | frickler: I'll take a look at that as it was next on my list (along with finishing the kdc/admind flipover | 16:05 |
| *** markvoelker has joined #openstack-infra | 16:05 | |
| *** jamesmcarthur_ has joined #openstack-infra | 16:06 | |
| clarkb | infra-root OOM Killer had killed our java logstash process on that worker01 node | 16:07 |
| clarkb | I've restarted it and the other services. My hunch is that we've got another very large/memory expensive log file sneaking it which triggers the OOMkiller | 16:07 |
| *** jamesmcarthur has quit IRC | 16:08 | |
| fungi | how did we track those down in the past? | 16:09 |
| *** gfidente has quit IRC | 16:10 | |
| openstackgerrit | Hervé Beraud proposed openstack-dev/pbr master: Resolve ``ValueError`` when mapping value contains a literal ``=``. https://review.openstack.org/625372 | 16:11 |
| *** gfidente has joined #openstack-infra | 16:11 | |
| clarkb | fungi: you can pull up the worker logfile where it stops and infer that one of the last handful of logfiles processed was the trigger | 16:11 |
| clarkb | fungi: then open those up in the log server and see which ones are in the multi hundred megabyte range | 16:11 |
| clarkb | I think I'm going to take this as an opportunity to reboot those servers | 16:11 |
| clarkb | may as well | 16:12 |
| clarkb | infra-root can you review https://review.openstack.org/#/c/638793/ which is my change that flips kadmind/kdcmaster to kdc03 from kdc01? The commit message also has a service flip over plan | 16:16 |
| clarkb | fungi: I usually go back through the logs after I've restarted thigns and try to dig out which log files are making it unhappy. I can walk others through that today if there is interest | 16:17 |
| fungi | not sure if i have time today but that's probably enough detail i could work it out myself | 16:18 |
| fungi | basically just knowing that the worker logs are verbose enough you can find out what build is being processed at the time it dies | 16:19 |
| fungi | from there it's presumably not hard to go digging in the archived artifacts for that build and see if any of the logs we want to index are massive? | 16:20 |
| clarkb | yup and then referring back to the log server for size data. It seems that anything over a hundred megabytes or so (uncompressed) gets us into danger territory | 16:20 |
| *** yboaron_ has quit IRC | 16:20 | |
| clarkb | the logstash queue is almost 300k entries long. We've also seen the logstash server become unreliably slow when ti gets a very large queue like that. I'll watch it to see if its size decreases once all workers are back and if it doesn't we may need to restart it too | 16:23 |
| fungi | given that zuul now limits the size of data a build can archive, i guess it's something which is compressing really well | 16:25 |
| *** e0ne has quit IRC | 16:26 | |
| *** ginopc has joined #openstack-infra | 16:28 | |
| *** agopi has joined #openstack-infra | 16:28 | |
| clarkb | 295820 is the job queue number now. Checking again in an hour is probably a reasonable window to gauge if its rolling downhill or picking up after itself | 16:28 |
| *** e0ne has joined #openstack-infra | 16:31 | |
| *** ginopc has quit IRC | 16:34 | |
| *** markvoelker has quit IRC | 16:34 | |
| clarkb | http://logs.openstack.org/45/560445/242/check/tripleo-ci-centos-7-containerized-undercloud-upgrades/2b1385d/logs/undercloud/var/log/extra/errors.txt.gz?level=INFO is still wgetting for me but well over 200MB at this point and was being processed at the end of a worker log file | 16:37 |
| clarkb | http://logs.openstack.org/70/630970/4/check/tripleo-ci-centos-7-standalone-upgrade/62a4783/logs/undercloud/var/log/extra/errors.txt.gz?level=INFO as well but I've not had a chance to pull it and check uncompressed size yet | 16:37 |
| clarkb | (I figure one big download at a time is nicer for the logs server | 16:37 |
| *** gyee has joined #openstack-infra | 16:37 | |
| clarkb | wget says 402.37M for that first one | 16:39 |
| *** takamatsu_ has quit IRC | 16:39 | |
| *** aojea has quit IRC | 16:39 | |
| clarkb | zbr|out is apparently out. Any idea of who we should talk to about ^ mwhahaha EmilienM? basically errors log file is quite large in some instances and is suspected of OOMing our log processing nodes | 16:40 |
| *** dklyle_ has quit IRC | 16:40 | |
| mwhahaha | lovely | 16:40 |
| *** takamatsu_ has joined #openstack-infra | 16:41 | |
| *** yamamoto has joined #openstack-infra | 16:41 | |
| *** ginopc has joined #openstack-infra | 16:41 | |
| mwhahaha | is it just teh upgrades job? | 16:41 |
| clarkb | ya looks like both containerized and standalone upgrade jobs based on the urls | 16:41 |
| mwhahaha | weshay -^ | 16:42 |
| clarkb | 279.41M on the second file | 16:43 |
| clarkb | probably not an emergency but with feature freeze coming up it would be good to try and stabilize the log processing pipeline so that we can use it to find and track bugs more effectively | 16:44 |
| mwhahaha | sure | 16:44 |
| *** raukadah is now known as chandankumar | 16:44 | |
| *** helenafm has quit IRC | 16:44 | |
| mwhahaha | i'll file a bug to get visibility | 16:44 |
| clarkb | thank you | 16:44 |
| *** takamatsu_ has quit IRC | 16:45 | |
| *** yamamoto has quit IRC | 16:46 | |
| mwhahaha | Bug 1817602 | 16:47 |
| openstack | bug 1817602 in tripleo "excessive errors in upgrade jobs causing logstash issues" [Critical,Triaged] https://launchpad.net/bugs/1817602 | 16:47 |
| *** kjackal has quit IRC | 16:48 | |
| *** kopecmartin is now known as kopecmartin|off | 16:48 | |
| *** pgaxatte has quit IRC | 16:49 | |
| *** takamatsu_ has joined #openstack-infra | 16:51 | |
| fungi | clarkb: should we stop indexing that errors.txt file temporarily? | 16:52 |
| fungi | presumably that's going to continue to crash the workers until fixed? | 16:52 |
| clarkb | fungi: ya if it isn't a one off issue that would be one way to address it. Another might be to take the head or tail of the file and store the remainder of the log in a file we don't index | 16:54 |
| clarkb | fungi: we can see if the workers OOM again | 16:55 |
| fungi | was it only that worker which crashed? | 16:55 |
| *** dklyle has joined #openstack-infra | 16:55 | |
| clarkb | fungi: no 03 had as well and at that point I decided to just reboot all of them | 16:55 |
| clarkb | can check the logs to see if more had crashed but I haven't looked yet | 16:55 |
| *** ginopc has quit IRC | 16:59 | |
| *** ramishra has quit IRC | 17:01 | |
| *** kjackal has joined #openstack-infra | 17:03 | |
| *** mudpuppy has joined #openstack-infra | 17:06 | |
| *** takamatsu_ has quit IRC | 17:06 | |
| *** takamatsu_ has joined #openstack-infra | 17:07 | |
| *** spotz has joined #openstack-infra | 17:10 | |
| *** luizbag has quit IRC | 17:16 | |
| *** eharney has quit IRC | 17:17 | |
| *** e0ne has quit IRC | 17:18 | |
| *** jpich has quit IRC | 17:20 | |
| *** takamatsu_ has quit IRC | 17:21 | |
| *** takamatsu_ has joined #openstack-infra | 17:22 | |
| *** dtantsur is now known as dtantsur|afk | 17:23 | |
| *** takamatsu_ has quit IRC | 17:25 | |
| *** sdake has joined #openstack-infra | 17:26 | |
| *** takamatsu_ has joined #openstack-infra | 17:31 | |
| *** markvoelker has joined #openstack-infra | 17:32 | |
| *** chandankumar is now known as raukadah | 17:32 | |
| clarkb | 294446 is the logstash queue size now. | 17:40 |
| clarkb | I think I will go ahead and restart it to bring the queue size down so geard performs better | 17:40 |
| clarkb | most of the backlog is from the weekend and with feature freeze happening in the near future I think we are more insterested in the newer data | 17:41 |
| *** dpawlik has joined #openstack-infra | 17:43 | |
| clarkb | infra-root I'm ready to begin the steps planned at https://review.openstack.org/#/c/638793/ once I get a positive review or two indicating the plan looks good | 17:45 |
| *** takamatsu_ has quit IRC | 17:46 | |
| clarkb | fungi: smarcet: I'd also like to approve https://review.openstack.org/#/c/616002/ and get openstackid switched over to the future parser today | 17:47 |
| clarkb | if you can review that one I'd be happy to approve and watch it | 17:47 |
| *** takamatsu_ has joined #openstack-infra | 17:48 | |
| *** dpawlik has quit IRC | 17:52 | |
| *** derekh has quit IRC | 18:01 | |
| *** trown is now known as trown|lunch | 18:04 | |
| *** takamatsu_ has quit IRC | 18:05 | |
| *** markvoelker has quit IRC | 18:05 | |
| *** sdake_ has joined #openstack-infra | 18:05 | |
| *** sdake has quit IRC | 18:07 | |
| *** sdake_ has quit IRC | 18:10 | |
| *** takamatsu_ has joined #openstack-infra | 18:10 | |
| *** sdake has joined #openstack-infra | 18:10 | |
| *** jpena is now known as jpena|off | 18:11 | |
| *** takamatsu_ has quit IRC | 18:14 | |
| *** Swami has joined #openstack-infra | 18:15 | |
| *** sdake has quit IRC | 18:16 | |
| *** sdake_ has joined #openstack-infra | 18:16 | |
| *** shardy has quit IRC | 18:18 | |
| *** takamatsu_ has joined #openstack-infra | 18:20 | |
| *** sdake_ has quit IRC | 18:20 | |
| *** sdake has joined #openstack-infra | 18:22 | |
| *** sdake has quit IRC | 18:26 | |
| *** kjackal has quit IRC | 18:26 | |
| *** raissa has joined #openstack-infra | 18:26 | |
| *** wolverineav has joined #openstack-infra | 18:27 | |
| *** sdake_ has joined #openstack-infra | 18:27 | |
| *** gfidente has quit IRC | 18:29 | |
| *** yamamoto has joined #openstack-infra | 18:29 | |
| *** gfidente has joined #openstack-infra | 18:29 | |
| weshay | mwhahaha clarkb sagi and I are looking into it now.. thanks for the heads up | 18:30 |
| *** yamamoto has quit IRC | 18:34 | |
| *** ijw has joined #openstack-infra | 18:35 | |
| *** wolverineav has quit IRC | 18:35 | |
| *** sdake_ has quit IRC | 18:35 | |
| *** sdake has joined #openstack-infra | 18:36 | |
| *** ijw has quit IRC | 18:36 | |
| *** ijw has joined #openstack-infra | 18:36 | |
| *** wolverineav has joined #openstack-infra | 18:37 | |
| *** bnemec has quit IRC | 18:40 | |
| clarkb | fungi: thoughts on single core approving the kdc update? | 18:41 |
| *** sdake has quit IRC | 18:41 | |
| *** sdake_ has joined #openstack-infra | 18:41 | |
| fungi | clarkb: i'm happy to do it if you're ready for this step. you had it marked wip though | 18:41 |
| mordred | clarkb: +2 from me - go ahead whenever you want | 18:42 |
| clarkb | fungi: I had it marked WIP so that it wouldn't merge before I did the steps before merging in the commit message | 18:42 |
| fungi | right | 18:42 |
| clarkb | mordred: cool thanks. I'll start with the steps I listed there and approve when those are done | 18:42 |
| fungi | sounds great! | 18:42 |
| *** e0ne has joined #openstack-infra | 18:44 | |
| clarkb | I've got kdc01 in the emergency file. I expect it will actually run pupept in a minute since it has been almost 45 minute since the last run | 18:44 |
| clarkb | once that run is over I'll stop services and continue with the plan | 18:44 |
| *** bnemec has joined #openstack-infra | 18:45 | |
| *** sdake_ has quit IRC | 18:45 | |
| *** sdake has joined #openstack-infra | 18:46 | |
| clarkb | I've stopped krb5-admin-server and krb5-kdc services on kdc01 | 18:49 |
| clarkb | and commented out the run-kprop.sh cron entry on kdc01 | 18:49 |
| clarkb | kprop is failing | 18:50 |
| * clarkb debugs | 18:50 | |
| *** takamatsu_ has quit IRC | 18:50 | |
| *** sdake has quit IRC | 18:50 | |
| clarkb | beh needs more sudo | 18:50 |
| clarkb | ok I actually have to leave the krb5-kdc service running on kdc01 to have kprop work. So I'm turning that service back on | 18:51 |
| *** sdake has joined #openstack-infra | 18:51 | |
| clarkb | I have approved the change that makes kdc03 the master | 18:52 |
| fungi | awesome | 18:55 |
| *** sdake has quit IRC | 18:55 | |
| *** sdake has joined #openstack-infra | 18:56 | |
| *** takamatsu_ has joined #openstack-infra | 18:56 | |
| *** sdake has quit IRC | 19:01 | |
| *** sdake has joined #openstack-infra | 19:02 | |
| openstackgerrit | Melvin Hillsman proposed openstack-infra/irc-meetings master: Update UC Meeting time. https://review.openstack.org/639176 | 19:03 |
| *** sdake has quit IRC | 19:05 | |
| *** sdake has joined #openstack-infra | 19:06 | |
| *** sdake has quit IRC | 19:11 | |
| *** sshnaidm is now known as sshnaidm|afk | 19:11 | |
| *** sdake has joined #openstack-infra | 19:14 | |
| *** yolanda has joined #openstack-infra | 19:15 | |
| *** sdake has quit IRC | 19:15 | |
| *** sdake_ has joined #openstack-infra | 19:15 | |
| *** eharney has joined #openstack-infra | 19:16 | |
| openstackgerrit | Merged openstack-infra/system-config master: Turn on the future parser for openstackid https://review.openstack.org/616002 | 19:17 |
| *** trown|lunch is now known as trown | 19:19 | |
| *** sdake_ has quit IRC | 19:20 | |
| clarkb | fungi: smarcet fyi there is a package install error on openstackid01 | 19:20 |
| fungi | looking | 19:21 |
| fungi | update-alternatives: error: cannot stat file '/usr/bin/node': Too many levels of symbolic links | 19:22 |
| fungi | that's certainly weird | 19:22 |
| openstackgerrit | Merged openstack-infra/system-config master: Make kdc03 the master kerberos kdc and admin server https://review.openstack.org/638793 | 19:23 |
| fungi | /usr/bin/node -> /usr/bin/nodejs | 19:23 |
| fungi | /usr/bin/nodejs -> /etc/alternatives/nodejs | 19:23 |
| fungi | /etc/alternatives/nodejs -> /usr/bin/node | 19:23 |
| fungi | well that's just dandy | 19:23 |
| clarkb | fungi: I'm guessing puppet writes one of those symlinks and it conflicts with a symlink that the packaging writes? | 19:23 |
| fungi | yeah, likely | 19:24 |
| fungi | the nodejs package (installed from nodesource) seems to want to provide an executable at /usr/bin/node | 19:26 |
| fungi | 2019-02-19 20:24:36 status installed nodejs:amd64 10.15.1-1nodesource1 | 19:26 |
| fungi | that's from the dpkg log | 19:27 |
| fungi | basically when the server was built | 19:27 |
| fungi | so i don't think an upgrade caused this | 19:27 |
| mordred | fungi: is our puppet trying to make the symlink? | 19:29 |
| fungi | i'm not sure yet | 19:29 |
| mordred | nod | 19:29 |
| fungi | i've just done a `sudo apt install --reinstall nodejs` which seems to have replaced the symlink at /usr/bin/node with an executable binary again | 19:30 |
| *** luizbag has joined #openstack-infra | 19:31 | |
| fungi | on openstackid-dev01 we have /usr/bin/node -> /usr/bin/nodejs but /usr/bin/nodejs is an executable binary provided by the nodejs package from ubuntu xenial | 19:32 |
| fungi | and on -dev01 there's a /usr/local/bin/node which seems to have been installed with npm instead. there is no /usr/local/bin/node on openstackid01 | 19:33 |
| fungi | both servers seem to have a /etc/apt/sources.list.d/nodesource.list present | 19:34 |
| clarkb | next puppet run in about 40 minuets should do the kdc swapo | 19:39 |
| *** kjackal has joined #openstack-infra | 19:50 | |
| openstackgerrit | Melvin Hillsman proposed openstack-infra/irc-meetings master: Update UC Meeting time. https://review.openstack.org/639176 | 19:54 |
| *** raissa has quit IRC | 19:58 | |
| *** jesusaur has quit IRC | 19:59 | |
| *** yamamoto has joined #openstack-infra | 20:00 | |
| *** jesusaur has joined #openstack-infra | 20:02 | |
| *** markvoelker has joined #openstack-infra | 20:02 | |
| openstackgerrit | Ivoline Ngong proposed openstack-infra/storyboard master: updated_at field in projects now contains a date that signifies the most recent date a task or story related to the project was last updated/created. Task: 24368 https://review.openstack.org/638840 | 20:02 |
| *** yamamoto has quit IRC | 20:04 | |
| clarkb | fungi: Feb 25 19:31:07 openstackid01 puppet-user[1263]: (/Stage[main]/Nodejs::Install/File[/usr/bin/node]/ensure) ensure changed 'file' to 'link' | 20:14 |
| fungi | yep, that seems to be the cause | 20:14 |
| clarkb | I expect puppet to run on kdc03 and make it master kerberos node in just a minute or two | 20:14 |
| openstackgerrit | Amy Marrich (spotz) proposed openstack-infra/irc-meetings master: Update UC Meeting time. https://review.openstack.org/639176 | 20:15 |
| clarkb | infra-root http://paste.openstack.org/show/746073/ that seems to have gone as expected | 20:16 |
| clarkb | Next in my list is to udpate dns then confirm that kadmin tasks work as expected. Anyone have ideas for how to do that effectively? mordred ? | 20:17 |
| clarkb | the testing part, not the dns part | 20:17 |
| clarkb | fungi: openstackid puppet application looks good to me | 20:22 |
| corvus | clarkb: create and delete a principal? | 20:25 |
| clarkb | corvus: oh right that will go into the db. Ya I'll createa clarkb-test principal and delete it | 20:26 |
| *** bhavikdbavishi has quit IRC | 20:31 | |
| *** dosaboy has quit IRC | 20:34 | |
| *** dosaboy has joined #openstack-infra | 20:36 | |
| *** markvoelker has quit IRC | 20:36 | |
| *** dosaboy has quit IRC | 20:37 | |
| *** wolverineav has quit IRC | 20:37 | |
| *** wolverineav has joined #openstack-infra | 20:37 | |
| *** jtomasek has quit IRC | 20:39 | |
| *** ccamacho has quit IRC | 20:39 | |
| *** wolverineav has quit IRC | 20:39 | |
| clarkb | dns records should all be updated | 20:40 |
| *** jtomasek has joined #openstack-infra | 20:41 | |
| *** jtomasek has quit IRC | 20:42 | |
| clarkb | new names all resolve for me | 20:42 |
| *** jtomasek has joined #openstack-infra | 20:43 | |
| *** dosaboy has joined #openstack-infra | 20:46 | |
| clarkb | now if only I understood the error messages kadmin is giving me on my desktop | 20:48 |
| clarkb | ok you have to configure krb5.conf with a real and admin server, it doesn't seem to look that up in dns | 20:53 |
| openstackgerrit | Ivoline Ngong proposed openstack-infra/storyboard master: Updated_at content modified https://review.openstack.org/638840 | 20:56 |
| clarkb | infra-root I am able to create and delete a test principal | 20:56 |
| clarkb | I'm going to turn off krb5-kdc on kdc01 now since it isn't getting replicated to. | 20:58 |
| clarkb | This is the point in time which I would expect any problems if we are going to have them | 20:58 |
| *** agopi has quit IRC | 20:58 | |
| corvus | clarkb: ack. i can stand by to authententicate from my workstation to verify correct operation after you do so. | 20:58 |
| clarkb | kdc01 is no longer running a krb5-kdc. Going to try a local kinit now | 20:59 |
| *** jrist has quit IRC | 20:59 | |
| *** agopi has joined #openstack-infra | 20:59 | |
| corvus | kinit: Cannot contact any KDC for realm 'OPENSTACK.ORG' while getting initial credentials | 20:59 |
| *** weshay has quit IRC | 20:59 | |
| *** weshay has joined #openstack-infra | 21:00 | |
| clarkb | hrm it worked for me from here but I also edit /etc/krb5.conf for kadmin | 21:00 |
| *** wolverineav has joined #openstack-infra | 21:00 | |
| corvus | oh, i have a local krb5.conf, sorry. | 21:01 |
| corvus | my test is invalid. | 21:01 |
| *** lucasagomes has quit IRC | 21:01 | |
| clarkb | ok | 21:01 |
| corvus | we should probably not be changing the numbers all the time :) | 21:01 |
| clarkb | fwiw I set it to use kdc.openstack.org | 21:01 |
| clarkb | whcih is a cname to the master (that I updated when I updated dns) | 21:01 |
| corvus | ok. wfm now. | 21:02 |
| corvus | i removed openstack.org completely from my krb5.conf file | 21:02 |
| corvus | (so i'm using only dns) | 21:02 |
| clarkb | great | 21:02 |
| *** jrist has joined #openstack-infra | 21:03 | |
| clarkb | for whatever reason kadmin did not like that locally | 21:03 |
| clarkb | it complained about missing config items in the file | 21:03 |
| corvus | yeah, kadmin may require it | 21:03 |
| corvus | that may be a good reason to consider keeping the hostnames stable | 21:04 |
| clarkb | we can use kdc.openstack.org or similar? then cname to the actual master node? | 21:04 |
| clarkb | we don't do that in the SRV records | 21:04 |
| corvus | clarkb: it's the non kadmin case i'm worried about | 21:04 |
| clarkb | ah ok | 21:04 |
| clarkb | _kerberos._udp.openstack.org SRV records then | 21:05 |
| corvus | i'm sure there's a solution, but i don't know what it is | 21:05 |
| clarkb | in any case this seems to be working. I'll keep my eyes and ears open for afs unhappyness | 21:05 |
| corvus | all i can say is that we named kdc01 and kdc02 before we established our hostname policy; i don't think we expected them to change | 21:05 |
| clarkb | and if this is happy after a day or two we should be able to delete the kdc01 server | 21:06 |
| clarkb | corvus: gotcha | 21:06 |
| clarkb | for extra safety I've removed kdc03 and kdc04 from kdc01's kprop.sh script | 21:06 |
| clarkb | so even if it tries to propogate it should talk to no downstream servers | 21:06 |
| *** agopi_ has joined #openstack-infra | 21:07 | |
| clarkb | #status log kdc03 is now our kerberos master/admin server. kdc01 is not yet deleted but is not running any services or the propogation cron. Will cleanup kdc01 after a day or two of happy afs services. | 21:07 |
| openstackstatus | clarkb: finished logging | 21:07 |
| *** jrist has quit IRC | 21:09 | |
| *** jpena|off has quit IRC | 21:09 | |
| *** otherwiseguy has quit IRC | 21:09 | |
| *** pabelanger has quit IRC | 21:09 | |
| *** jtomasek has quit IRC | 21:09 | |
| *** jpena|off has joined #openstack-infra | 21:10 | |
| *** agopi has quit IRC | 21:10 | |
| *** kashyap has quit IRC | 21:10 | |
| *** jrist has joined #openstack-infra | 21:10 | |
| *** otherwiseguy has joined #openstack-infra | 21:11 | |
| *** agopi__ has joined #openstack-infra | 21:13 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/system-config master: Updated openstackid prod node https://review.openstack.org/639190 | 21:14 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Run zuul-preview https://review.openstack.org/637654 | 21:15 |
| corvus | clarkb, fungi, mordred: ^ that patchset removes the speculative docker stuff, so the run-zuul-preview job is just like the rest of the run-base jobs for now. | 21:15 |
| corvus | that should let us start up the service, and we can swing back to speculative docker images after making changes to zuul+nodepool | 21:15 |
| clarkb | looking | 21:16 |
| *** agopi_ has quit IRC | 21:16 | |
| *** agopi__ is now known as agopi | 21:16 | |
| *** jcoufal has quit IRC | 21:16 | |
| clarkb | corvus: change lgtm. Is that something we might consider running on zuul01 instead of a new host? | 21:19 |
| clarkb | I guess zuul01 often has its memory spoken for so more services might not be a good idea? | 21:19 |
| corvus | clarkb: it's something we could colocate with other services (would be a great thing to throw into a k8s cluster), but not something i'd want to put on zuul01 since it's a spof. | 21:20 |
| corvus | we could probably spin it up as a 2G host | 21:21 |
| *** e0ne has quit IRC | 21:21 | |
| clarkb | re k8s maybe we ought to take an inventory of things that would work in a k8s then we can have a todo list peopel can work on to k8sify them into our k8s cluster (though maybe we need to finish the story on k8s deployments first?) | 21:22 |
| corvus | clarkb: ++ i think we have a lot | 21:23 |
| clarkb | ok gonna pop out for lunch now. Back in a bit | 21:24 |
| *** sdake has joined #openstack-infra | 21:27 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/nodepool master: [dnm] testing rhel8 images https://review.openstack.org/639013 | 21:28 |
| *** kgiusti has left #openstack-infra | 21:32 | |
| *** markvoelker has joined #openstack-infra | 21:33 | |
| *** luizbag has quit IRC | 21:34 | |
| openstackgerrit | Merged openstack-infra/system-config master: Turn on the future parser for refstack https://review.openstack.org/628153 | 21:35 |
| *** wolverineav has quit IRC | 21:42 | |
| *** wolverineav has joined #openstack-infra | 21:42 | |
| *** rcernin has joined #openstack-infra | 21:46 | |
| *** whoami-rajat has quit IRC | 21:47 | |
| *** yamamoto has joined #openstack-infra | 21:48 | |
| *** sdake has quit IRC | 21:48 | |
| *** yamamoto has quit IRC | 21:52 | |
| *** e0ne has joined #openstack-infra | 21:54 | |
| mordred | clarkb, corvus: I agree - I think we have a good amount of thigns we can k8s-ify - but also we should finish that story before we set people off working | 21:54 |
| *** e0ne has quit IRC | 21:55 | |
| *** sdake has joined #openstack-infra | 21:59 | |
| *** trown is now known as trown|outtypewww | 22:00 | |
| *** agopi has quit IRC | 22:00 | |
| *** sdake has quit IRC | 22:01 | |
| *** sdake has joined #openstack-infra | 22:03 | |
| *** markvoelker has quit IRC | 22:04 | |
| *** e0ne has joined #openstack-infra | 22:05 | |
| *** sdake has quit IRC | 22:05 | |
| *** kjackal has quit IRC | 22:07 | |
| *** sdake has joined #openstack-infra | 22:07 | |
| *** e0ne has quit IRC | 22:08 | |
| *** rkukura_ has joined #openstack-infra | 22:08 | |
| *** sdake has quit IRC | 22:10 | |
| *** rkukura has quit IRC | 22:10 | |
| *** rkukura_ is now known as rkukura | 22:10 | |
| *** zerick has quit IRC | 22:11 | |
| *** zerick has joined #openstack-infra | 22:11 | |
| *** wolverineav has quit IRC | 22:12 | |
| *** wolverineav has joined #openstack-infra | 22:13 | |
| *** sdake has joined #openstack-infra | 22:13 | |
| *** sdake has quit IRC | 22:13 | |
| openstackgerrit | Ivoline Ngong proposed openstack-infra/storyboard master: Updated_at content modified https://review.openstack.org/638840 | 22:17 |
| openstackgerrit | Merged openstack-infra/system-config master: Run zuul-preview https://review.openstack.org/637654 | 22:18 |
| *** agopi has joined #openstack-infra | 22:19 | |
| *** sshnaidm|afk has quit IRC | 22:24 | |
| clarkb | ok the refstack futureparsering resulted in a different apache config. Apache is still running so it didn't result in a broken config. I'm now trying to find where puppet stores those diffs | 22:35 |
| clarkb | old file had ServerAdmin undef in it | 22:36 |
| clarkb | so I think the old side was broken? new side doesn't have any ServerAdmin lines. Looking at what the intention is there | 22:37 |
| *** sshnaidm has joined #openstack-infra | 22:39 | |
| openstackgerrit | Merged openstack-dev/pbr master: Resolve ``ValueError`` when mapping value contains a literal ``=``. https://review.openstack.org/625372 | 22:43 |
| clarkb | ok current code is doing the right thing. We don't set a server admin value | 22:44 |
| mordred | cool | 22:44 |
| clarkb | cmurphy: I think that means we've done all of the futureparser conversions? | 22:45 |
| clarkb | and now we can continue to do puppet4 upgrades | 22:46 |
| clarkb | though I'm probably going to pick off another trusty host to upgrade before I look at that | 22:46 |
| cmurphy | clarkb: ask is the last one | 22:51 |
| clarkb | ah | 22:51 |
| clarkb | corvus: mordred: I'm looking at what is required to upgrade the afs volume and db servers. DB servers look simpler so maybe I'll start there. If I make a new db server what is the process for joining it to the group so that it can be synced to and old server(s) deleted? | 22:52 |
| mordred | clarkb: I'm going to have to defer to corvus on that one | 22:56 |
| *** tkajinam has joined #openstack-infra | 22:57 | |
| *** markvoelker has joined #openstack-infra | 23:01 | |
| clarkb | https://lists.openafs.org/pipermail/openafs-info/2013-February/039264.html is not instilling much confidence | 23:01 |
| clarkb | First of all we apparently shouldn't mix db versions | 23:02 |
| clarkb | and second IP addrs matter | 23:02 |
| clarkb | :/ | 23:02 |
| corvus | i would seriously consider apt-get dist-upgrade | 23:03 |
| clarkb | ya that may be easiest. We'd still need a full shutdown/outage of the cluster sounds like but an inplace upgrade would address the IP addr situation | 23:04 |
| corvus | i've never performed a complete replacement and don't know off the top of my head what's involved. | 23:04 |
| clarkb | solaris 8 to rhel 6 though, if they can make that jump we can do trusty to xenial right? :) | 23:05 |
| clarkb | corvus: do you expect that would be easiest for the volume servers too? | 23:06 |
| clarkb | I might page this out and try to pick off more of the easier options first. health.o.o I think you are the next lucky contestant | 23:07 |
| corvus | clarkb: for giant volumes, probably so. if we could move volumes between servers quickly, i would probably just do it entirely at the client level. | 23:10 |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config master: Set health service's vhost hostname explicitly https://review.openstack.org/639217 | 23:12 |
| clarkb | corvus: I was thinking that maybe we'd be able to move the cinder volumes but I'm not sure what reenrolling into a new afs server looks like when you do that | 23:12 |
| clarkb | infra-root https://review.openstack.org/639217 is a step zero for spinning up a new health server. If we can get that in. I'll boot a new instance and sned a followup change that adds things to inventory | 23:13 |
| *** pabelanger has joined #openstack-infra | 23:14 | |
| *** jamesmcarthur_ has quit IRC | 23:17 | |
| *** jamesmcarthur has joined #openstack-infra | 23:19 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: [wip] shortcut pip-and-virtualenv for later redhat distros https://review.openstack.org/639220 | 23:19 |
| openstackgerrit | Ian Wienand proposed openstack-infra/nodepool master: [dnm] testing rhel8 images https://review.openstack.org/639013 | 23:20 |
| ianw | cgoncalves /dmsimard: re prior discussions ^ ... i'm wondering if we just shortcut out all pip-and-virtualenv for later distros and use the inbuilt packages and assume they get it right | 23:21 |
| ianw | the only thing is linking /usr/bin/pip2 & /usr/bin/virtualenv-2 back to "pip" and "virtualenv" because a bunch of stuff assumes that | 23:22 |
| ianw | most python3 stuff explicitly calls "pip3" ... and virtualenv python3 is just a mess, so maybe that's all bases covered | 23:22 |
| ianw | the only certainty is it will break *something* | 23:23 |
| *** jamesmcarthur has quit IRC | 23:30 | |
| *** hwoarang has quit IRC | 23:32 | |
| *** hwoarang has joined #openstack-infra | 23:33 | |
| *** markvoelker has quit IRC | 23:34 | |
| *** jamesmcarthur has joined #openstack-infra | 23:36 | |
| *** yamamoto has joined #openstack-infra | 23:36 | |
| fungi | ianw: ideally we would just stop polluting our systems with preinstalled pip and virtualenv and instead require jobs which need those to install them | 23:40 |
| *** sreejithp has quit IRC | 23:40 | |
| fungi | last i heard most of our tox-using jobs try to, but no-op because it's already there | 23:41 |
| clarkb | we still use it for the zuul-cloner shim install iirc | 23:41 |
| clarkb | but could remove it after bootstrapping like that I usppose | 23:41 |
| *** yamamoto has quit IRC | 23:41 | |
| fungi | yeah, that was the last hold-out i was aware of, but we _could_ still install zuul-cloner without putting pip and virtualenv on the images | 23:41 |
| ianw | fungi: yes ... i think that falls under "only certainty is it will break something" :) it would be a firefighting period, but maybe one we should accept | 23:42 |
| ianw | i'd also have to re-audit if the distro versions have fixed all the pip bugs we started installing from source to fix too | 23:43 |
| clarkb | fungi: well you need a virtualenv to store the zuul-cloner deps | 23:43 |
| clarkb | so it has to get there somehow | 23:44 |
| *** gfidente has quit IRC | 23:44 | |
| ianw | yeah, i think having just the system packages installed would be a good simplification | 23:45 |
| fungi | clarkb: right but we can avoid installing virtualenv itself in the executable path (and clean it up after, or even use it from outside the chroot when putting the zuul-cloner virtualenv on the image) | 23:46 |
| clarkb | ya | 23:46 |
| ianw | nodepool-base is also setting up an os-testr virtualenv | 23:46 |
| *** jamesmcarthur has quit IRC | 23:47 | |
| ianw | and bindep for the infra-package-needs | 23:47 |
| fungi | er, actually the lack of support for moving virtualenv paths probably means we'd have to do it inside the chroot, but still wouldn't need to install virtualenv into the default execution path | 23:47 |
| fungi | ianw: i *think* the bindep role also tries to install bindep if it doesn't exist | 23:48 |
| fungi | but regardless, the same solution for all three could be used, or better still the jobs which need those can make sure to install them | 23:48 |
| ianw | fungi: well the build-essential the idea is to pre-install the packages to save time in the gate, i think that's valuable although i see the argument of moving it all into jobs | 23:49 |
| fungi | not saying i expect we'll do any of that any time soon, but it's a good goal to keep in mind as we work through pip and virtualenv related stuff on our images | 23:49 |
| fungi | problem with anything we preinstall, *especially* if we preinstall it with a package manager other than the distro's, is that there will be jobs which don't want whatever it is we're preinstalling and cleanly uninstalling that stuff is even harder | 23:50 |
| dmsimard | ianw: I was wondering if we should test with vanilla cloud images, we have quite a delta and it was frustrating (as an end user) that an issue wasn't reproduceable in "production" using the cloud images. | 23:51 |
| clarkb | dmsimard: we explicitly don't want vanilla cloud images though | 23:51 |
| clarkb | dmsimard: they change often and break and are generally unuseable | 23:51 |
| dmsimard | For caching ? | 23:51 |
| clarkb | dmsimard: for reliability | 23:51 |
| clarkb | one day cloud init updates and nothing works | 23:51 |
| dmsimard | We can gate them | 23:51 |
| fungi | vanilla cloud images similarly preinstall lots of stuff we may not want | 23:51 |
| dmsimard | fungi: true | 23:52 |
| ianw | dmsimard / clarkb: yeah ... although a *lot* has changed since the snapshot days | 23:52 |
| clarkb | like when hpcloud had cloud init set up to do weird things with the ephemeral disk | 23:52 |
| clarkb | all of a sudden nothing worked becuse this disk was over here now because your clodu image knows better | 23:52 |
| fungi | for example, and i don't know whether this is still the case, but for the longest time rhel shipped with quite a lot of python libraries installed as required deps of their system tools | 23:52 |
| clarkb | dmsimard: fwiw we do publish our images and provide the means for you to build them yourself | 23:53 |
| ianw | fungi: yep .. but e.g. rhel8 now has this idea of a "system" python and then user python is a different thing | 23:53 |
| fungi | and this broke our ability to test projects with newer versions of those python modules, especially if they also needed to interact with python modules that had to be installed outside a venv | 23:53 |
| clarkb | dmsimard: so in theory its not really much different than using upstream cloud images. You just have to download them from somewhere else | 23:53 |
| *** diablo_rojo has quit IRC | 23:53 | |
| ianw | clarkb: but i think, in practice, things would be much more stable today given the overall trend of minimising those images, their much wider use | 23:54 |
| clarkb | ianw: that is possible | 23:54 |
| clarkb | ianw: I'd still grump about the username choices though | 23:54 |
| ianw | clarkb: the cloud provider images we were using ~4/5 years ago, yes they managed to change in weird and wonderful ways all the time | 23:54 |
| clarkb | (seriously we don't need every distro to use a different username ) | 23:54 |
| ianw | but yeah, a lot of things built ontop of our customisations. i never really defend dib ... it has a lot of issues and they're getting worse (e.g. use of yum/zypper on ubuntu hosts is fragile) | 23:56 |
| fungi | i'd prefer if they all used the one we already have: root | 23:56 |
| fungi | but that seems to send clueless security auditors into a panic | 23:56 |
| clarkb | dmsimard: out of curiousity do you know what the difference was that caused problems for you? it is entirely possible we'd still need that delta if we built on top of the vanilla images | 23:56 |
| clarkb | like maybe it was something to handle rax networking | 23:56 |
| clarkb | which does not work with the vanilla images | 23:57 |
| clarkb | (as an example) | 23:57 |
| fungi | somehow having a standard username with root privileges on all your images is so much more secure than having a standard username which *is* root on all your images | 23:57 |
| clarkb | ianw: dib also does a lot of things right though. Like building without needing a virtualization layer (because nested virt still hard to find) and local caching of intermedaite artifacts | 23:58 |
| clarkb | ianw: finding image build tools that do both of those things outside of dib is basically impossible | 23:58 |
| clarkb | (granted that is also some of the source of the yum/zypper woes) | 23:58 |
| fungi | speaking of building without needing virtualization, i found mmdebstrap over the weekend. it's quite a lot faster than debootstrap and can work as a non-root user with fakechroot, even do cross-architecture filesystem builds | 23:59 |
| ianw | yeah, and one tool that produces ubuntu/debian/fedora/centos/rhel/gentoo/*suse on a daily basis :) | 23:59 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!