| jeblair | clarkb: though it means a delay before new things are available | 00:00 |
|---|---|---|
| clarkb | ya | 00:00 |
| jeblair | clarkb: one idea might be to release less frequently, unless there is a change to a package in g-r. | 00:00 |
| *** thorst_afk has joined #openstack-infra | 00:00 | |
| clarkb | jeblair: will vos release nop if nothing changed? eg do we already handle the trivial case when bandersnatch doesn't update anything? I guess thats part of what I can test tomorrow | 00:00 |
| jeblair | clarkb: i'm not certain. good thing to test. | 00:01 |
| jeblair | clarkb: (but also, i don't know if our script short-circuits in that case either) | 00:02 |
| *** jamielennox|away is now known as jamielennox | 00:02 | |
| *** gomarivera has joined #openstack-infra | 00:02 | |
| jeblair | clarkb: i do know that we release almost every 5 minutes because *something* is always changing on pypi. | 00:02 |
| clarkb | ya pypi is chatty | 00:03 |
| *** gildub has joined #openstack-infra | 00:05 | |
| *** gomarivera has quit IRC | 00:06 | |
| *** armax has quit IRC | 00:11 | |
| *** sflanigan has joined #openstack-infra | 00:13 | |
| *** dfflanders has joined #openstack-infra | 00:17 | |
| *** spzala has joined #openstack-infra | 00:21 | |
| *** spzala has quit IRC | 00:26 | |
| *** dfflanders has quit IRC | 00:27 | |
| *** esberglu has joined #openstack-infra | 00:28 | |
| *** hamzy has joined #openstack-infra | 00:28 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Don't make image & loopdev functions static https://review.openstack.org/470845 | 00:32 |
| *** esberglu has quit IRC | 00:32 | |
| *** thorst_afk has quit IRC | 00:33 | |
| *** sbezverk has joined #openstack-infra | 00:34 | |
| *** deafult has joined #openstack-infra | 00:36 | |
| *** spzala has joined #openstack-infra | 00:38 | |
| *** rkukura has quit IRC | 00:41 | |
| *** thorst_afk has joined #openstack-infra | 00:42 | |
| *** thorst_afk has quit IRC | 00:43 | |
| *** hongbin has quit IRC | 00:44 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Use picked nodes for later cmd_* calls https://review.openstack.org/469330 | 00:49 |
| *** zhurong has joined #openstack-infra | 00:49 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Add state to NodeBase class https://review.openstack.org/469749 | 00:52 |
| *** sree has joined #openstack-infra | 00:53 | |
| *** sflanigan has quit IRC | 00:53 | |
| *** cuongnv has joined #openstack-infra | 00:54 | |
| openstackgerrit | Jamie Lennox proposed openstack-infra/zuul feature/zuulv3: Store cache expiry times per status object https://review.openstack.org/461330 | 00:57 |
| openstackgerrit | Jamie Lennox proposed openstack-infra/zuul feature/zuulv3: Use routes for URL handling in webapp https://review.openstack.org/461331 | 00:57 |
| openstackgerrit | Jamie Lennox proposed openstack-infra/zuul feature/zuulv3: Use tenant_name to look up project key https://review.openstack.org/461332 | 00:57 |
| *** jamielennox is now known as jamielennox|away | 01:00 | |
| *** aeng has quit IRC | 01:03 | |
| *** jamielennox|away is now known as jamielennox | 01:10 | |
| *** sflanigan has joined #openstack-infra | 01:10 | |
| *** sflanigan has joined #openstack-infra | 01:11 | |
| *** thorst_afk has joined #openstack-infra | 01:14 | |
| *** aeng has joined #openstack-infra | 01:20 | |
| *** liujiong has joined #openstack-infra | 01:22 | |
| *** sree has quit IRC | 01:23 | |
| *** sbezverk has quit IRC | 01:23 | |
| *** LindaWang has joined #openstack-infra | 01:24 | |
| *** bobh has joined #openstack-infra | 01:25 | |
| *** Apoorva_ has joined #openstack-infra | 01:29 | |
| *** zhurong has quit IRC | 01:29 | |
| *** Apoorva_ has quit IRC | 01:29 | |
| *** spzala has quit IRC | 01:30 | |
| *** nunchuck has joined #openstack-infra | 01:30 | |
| *** thorst_afk has quit IRC | 01:31 | |
| *** armax has joined #openstack-infra | 01:32 | |
| *** Apoorva has quit IRC | 01:32 | |
| *** bobh has quit IRC | 01:34 | |
| *** gardlt has joined #openstack-infra | 01:34 | |
| *** aeng_ has joined #openstack-infra | 01:35 | |
| *** markvoelker has joined #openstack-infra | 01:35 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Remove 'state' argument from later cmd_* calls https://review.openstack.org/469750 | 01:36 |
| *** aeng has quit IRC | 01:37 | |
| *** namnh has joined #openstack-infra | 01:37 | |
| *** thorst_afk has joined #openstack-infra | 01:38 | |
| *** dave-mccowan has joined #openstack-infra | 01:38 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Use global state to check for duplicate fs labels https://review.openstack.org/469751 | 01:41 |
| *** tuanluong has joined #openstack-infra | 01:43 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Move global mount tracking into state https://review.openstack.org/469752 | 01:44 |
| *** thorst_afk has quit IRC | 01:44 | |
| *** r-daneel has quit IRC | 01:47 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Move rollback into NodeBase object https://review.openstack.org/464105 | 01:48 |
| openstackgerrit | Merged openstack/diskimage-builder master: Pass all blockdevices to bootloader https://review.openstack.org/471183 | 01:49 |
| *** ramishra has joined #openstack-infra | 01:52 | |
| *** Julien-zte has joined #openstack-infra | 01:53 | |
| openstackgerrit | Merged openstack/diskimage-builder master: PPC bootloader; install to boot partition https://review.openstack.org/471525 | 01:53 |
| *** rlandy has quit IRC | 01:54 | |
| *** dhinesh has quit IRC | 01:56 | |
| *** EricGonczer_ has quit IRC | 01:59 | |
| *** zhurong has joined #openstack-infra | 02:04 | |
| *** esberglu has joined #openstack-infra | 02:06 | |
| *** thorst_afk has joined #openstack-infra | 02:06 | |
| *** thorst_afk has quit IRC | 02:06 | |
| *** markvoelker has quit IRC | 02:09 | |
| *** esberglu has quit IRC | 02:10 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: DONT REVIEW: test image building https://review.openstack.org/459786 | 02:20 |
| *** hongbin has joined #openstack-infra | 02:23 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: DONT REVIEW: test image building https://review.openstack.org/459786 | 02:25 |
| *** jamesmcarthur has joined #openstack-infra | 02:26 | |
| *** hongbin has quit IRC | 02:27 | |
| *** hongbin has joined #openstack-infra | 02:28 | |
| *** jamesmcarthur has quit IRC | 02:31 | |
| *** dimtruck is now known as zz_dimtruck | 02:31 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/project-config master: Add tempest OOOQ job to experimental for tripleo https://review.openstack.org/472499 | 02:33 |
| *** deafult has quit IRC | 02:33 | |
| openstackgerrit | Saad Zaher proposed openstack-infra/project-config master: Added OpenStack-ansible repos for freezer https://review.openstack.org/469938 | 02:33 |
| *** mriedem has joined #openstack-infra | 02:34 | |
| *** thorst_afk has joined #openstack-infra | 02:37 | |
| *** knangia_ has quit IRC | 02:38 | |
| *** markvoelker has joined #openstack-infra | 02:44 | |
| openstackgerrit | Saad Zaher proposed openstack-infra/project-config master: Added Freezer OpenStack-ansible repo https://review.openstack.org/469938 | 02:47 |
| *** thorst_afk has quit IRC | 02:48 | |
| *** zz_dimtruck is now known as dimtruck | 02:52 | |
| *** spzala has joined #openstack-infra | 02:53 | |
| *** spzala_ has joined #openstack-infra | 02:53 | |
| *** spzala has quit IRC | 02:53 | |
| *** sflanigan is now known as sflanigan_afk | 02:53 | |
| *** larainema has quit IRC | 03:02 | |
| *** yamamoto has joined #openstack-infra | 03:07 | |
| *** sflanigan_afk is now known as sflanigan | 03:10 | |
| *** dimtruck is now known as zz_dimtruck | 03:12 | |
| *** larainema has joined #openstack-infra | 03:13 | |
| namnh | hi everyone, could you please review this patch: https://review.openstack.org/#/c/472153/ | 03:14 |
| *** xarses_ has quit IRC | 03:18 | |
| *** rfolco has joined #openstack-infra | 03:19 | |
| *** rfolco has quit IRC | 03:19 | |
| *** salv-orlando has joined #openstack-infra | 03:21 | |
| *** salv-orlando has quit IRC | 03:26 | |
| *** jcoufal has joined #openstack-infra | 03:26 | |
| *** bobh has joined #openstack-infra | 03:29 | |
| *** zz_dimtruck is now known as dimtruck | 03:34 | |
| *** sree has joined #openstack-infra | 03:41 | |
| *** sree has quit IRC | 03:43 | |
| *** bobh has quit IRC | 03:43 | |
| *** sree has joined #openstack-infra | 03:43 | |
| *** Julien-zte has quit IRC | 03:44 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/diskimage-builder master: Updated from global requirements https://review.openstack.org/472077 | 03:44 |
| *** bobh has joined #openstack-infra | 03:45 | |
| *** thorst_afk has joined #openstack-infra | 03:45 | |
| *** bobh has quit IRC | 03:45 | |
| *** bobh has joined #openstack-infra | 03:47 | |
| *** dave-mccowan has quit IRC | 03:47 | |
| *** bobh has quit IRC | 03:50 | |
| *** esberglu has joined #openstack-infra | 03:54 | |
| *** dimtruck is now known as zz_dimtruck | 03:54 | |
| *** esberglu has quit IRC | 03:59 | |
| *** spzala_ has quit IRC | 04:00 | |
| *** dingyichen has joined #openstack-infra | 04:03 | |
| *** thorst_afk has quit IRC | 04:04 | |
| *** hongbin has quit IRC | 04:11 | |
| *** zz_dimtruck is now known as dimtruck | 04:16 | |
| *** spzala has joined #openstack-infra | 04:17 | |
| *** links has joined #openstack-infra | 04:20 | |
| *** jcoufal has quit IRC | 04:20 | |
| *** spzala has quit IRC | 04:21 | |
| *** salv-orlando has joined #openstack-infra | 04:22 | |
| *** zhurong has quit IRC | 04:25 | |
| *** salv-orlando has quit IRC | 04:26 | |
| *** mriedem has quit IRC | 04:27 | |
| *** psachin has joined #openstack-infra | 04:31 | |
| *** zhurong has joined #openstack-infra | 04:34 | |
| *** vikrant has joined #openstack-infra | 04:37 | |
| *** dhajare|busy has joined #openstack-infra | 04:42 | |
| *** Hal has joined #openstack-infra | 04:42 | |
| *** yamamoto_ has joined #openstack-infra | 04:43 | |
| *** salv-orlando has joined #openstack-infra | 04:44 | |
| *** yamamoto has quit IRC | 04:47 | |
| *** gyee has quit IRC | 04:49 | |
| *** dimtruck is now known as zz_dimtruck | 04:50 | |
| *** dhinesh has joined #openstack-infra | 04:53 | |
| *** ramishra has quit IRC | 04:55 | |
| *** claudiub has joined #openstack-infra | 04:55 | |
| *** dhajare|busy has quit IRC | 04:56 | |
| *** ramishra has joined #openstack-infra | 04:57 | |
| *** spzala has joined #openstack-infra | 04:58 | |
| *** spzala has quit IRC | 05:03 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: [WIP] LVM support for dib-block-device https://review.openstack.org/472065 | 05:03 |
| *** gardlt has quit IRC | 05:05 | |
| *** Guest72612 has quit IRC | 05:06 | |
| *** med_ has joined #openstack-infra | 05:06 | |
| *** gardlt has joined #openstack-infra | 05:06 | |
| *** med_ is now known as Guest6511 | 05:06 | |
| *** ykarel has joined #openstack-infra | 05:07 | |
| *** links has quit IRC | 05:10 | |
| *** xarses has joined #openstack-infra | 05:10 | |
| *** gardlt has quit IRC | 05:10 | |
| *** zz_dimtruck is now known as dimtruck | 05:12 | |
| *** prateek has joined #openstack-infra | 05:12 | |
| *** xarses has quit IRC | 05:16 | |
| *** gomarivera has joined #openstack-infra | 05:24 | |
| *** links has joined #openstack-infra | 05:26 | |
| *** dimtruck is now known as zz_dimtruck | 05:27 | |
| *** gomarivera has quit IRC | 05:28 | |
| *** yamahata has joined #openstack-infra | 05:29 | |
| *** armax has quit IRC | 05:29 | |
| *** dhajare|busy has joined #openstack-infra | 05:30 | |
| *** spzala has joined #openstack-infra | 05:33 | |
| *** spzala has quit IRC | 05:38 | |
| *** pgadiya has joined #openstack-infra | 05:41 | |
| *** aeng_ has quit IRC | 05:42 | |
| *** esberglu has joined #openstack-infra | 05:42 | |
| *** pgadiya has quit IRC | 05:44 | |
| *** puiterwijk is now known as puiterwijk|pto | 05:44 | |
| *** dhinesh has quit IRC | 05:46 | |
| *** esberglu has quit IRC | 05:46 | |
| *** cshastri has joined #openstack-infra | 05:48 | |
| *** puiterwijk|pto is now known as puiterwijk | 05:51 | |
| *** Julien-zte has joined #openstack-infra | 05:54 | |
| *** nicolasbock has joined #openstack-infra | 05:57 | |
| *** dingyichen has quit IRC | 05:59 | |
| *** thorst_afk has joined #openstack-infra | 06:02 | |
| *** nicolasbock has quit IRC | 06:02 | |
| *** thorst_afk has quit IRC | 06:06 | |
| *** pgadiya has joined #openstack-infra | 06:07 | |
| *** spzala has joined #openstack-infra | 06:13 | |
| *** Hal has quit IRC | 06:15 | |
| EmilienM | yolanda: hey, sounds like gerrit is sending 502 proxy errors | 06:16 |
| EmilienM | ianw: ^ fyi | 06:16 |
| *** spzala has quit IRC | 06:18 | |
| *** jaosorior has joined #openstack-infra | 06:18 | |
| *** gildub has quit IRC | 06:25 | |
| EmilienM | infra-root: gerrit server is unavailable | 06:26 |
| *** jamesmcarthur has joined #openstack-infra | 06:27 | |
| *** eumel8 has joined #openstack-infra | 06:27 | |
| *** jbadiapa_ has quit IRC | 06:28 | |
| eumel8 | good morning. It seems https://review.openstack.org is down | 06:31 |
| *** jamesmcarthur has quit IRC | 06:31 | |
| *** iogg has joined #openstack-infra | 06:31 | |
| ianw | taking a look | 06:32 |
| eumel8 | thx | 06:33 |
| ianw | it's pegged at 100% cpu | 06:33 |
| eumel8 | proxy error 502 | 06:33 |
| ianw | java.io.IOException: Too many open files | 06:34 |
| *** JudeC has joined #openstack-infra | 06:35 | |
| *** ltomasbo|away is now known as ltomasbo | 06:35 | |
| EmilienM | maybe a restart? | 06:35 |
| ianw | ok, restarted it ... not sure what the root cause was | 06:37 |
| EmilienM | thanks ianw ! | 06:37 |
| ianw | openstack-ubuntu-testing-bot ... if anyone owns that, it's failing to connect every few seconds | 06:37 |
| eumel8 | service is back | 06:38 |
| ianw | castanos@us.ibm.com also | 06:38 |
| eumel8 | quick running | 06:38 |
| ianw | the error log is up to 15gb, so ... yeah ... unhappy for a while i guess | 06:39 |
| *** xarses has joined #openstack-infra | 06:40 | |
| *** xarses has quit IRC | 06:41 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config master: enable voting on tripleo-containers-ovb https://review.openstack.org/472536 | 06:41 |
| *** xarses has joined #openstack-infra | 06:41 | |
| ianw | there is a rather worrying consistent "java.io.IOException: Connection reset by peer" coming up in the logs ever few seconds | 06:41 |
| *** florianf has joined #openstack-infra | 06:42 | |
| *** yamahata has quit IRC | 06:43 | |
| ianw | russellb: do you have something in a tight loop polling gerrit every few seconds too? | 06:45 |
| *** salv-orlando has quit IRC | 06:52 | |
| EmilienM | ianw: yes, I think he still runs http://russellbryant.net/openstack-stats/ | 06:52 |
| EmilienM | looking at file timestamps, it's running | 06:52 |
| *** salv-orlando has joined #openstack-infra | 06:52 | |
| *** tesseract has joined #openstack-infra | 06:53 | |
| ianw | ahh, that would explain it i guess | 06:53 |
| EmilienM | it sounds like it runs ~ every hour or so | 06:53 |
| *** zz_dimtruck is now known as dimtruck | 06:53 | |
| ianw | i may have just caught it when it was busy | 06:54 |
| *** zhurong has quit IRC | 06:55 | |
| *** spzala has joined #openstack-infra | 06:55 | |
| *** zhurong has joined #openstack-infra | 06:56 | |
| *** shardy has joined #openstack-infra | 06:56 | |
| *** zhurong has quit IRC | 06:57 | |
| *** jascott1_ has quit IRC | 06:59 | |
| *** spzala has quit IRC | 06:59 | |
| *** jascott1 has joined #openstack-infra | 07:00 | |
| *** pgadiya has quit IRC | 07:02 | |
| *** thorst_afk has joined #openstack-infra | 07:03 | |
| *** dimtruck is now known as zz_dimtruck | 07:03 | |
| *** jascott1 has quit IRC | 07:04 | |
| *** pcaruana has joined #openstack-infra | 07:05 | |
| *** thorst_afk has quit IRC | 07:07 | |
| *** amoralej|off is now known as amoralej | 07:07 | |
| *** niska has quit IRC | 07:07 | |
| *** jani has left #openstack-infra | 07:08 | |
| *** markvoelker has quit IRC | 07:08 | |
| *** links has quit IRC | 07:09 | |
| *** rcernin has joined #openstack-infra | 07:09 | |
| *** spzala has joined #openstack-infra | 07:11 | |
| *** niska has joined #openstack-infra | 07:15 | |
| *** spzala has quit IRC | 07:16 | |
| *** ralonsoh has joined #openstack-infra | 07:16 | |
| *** ihrachys has quit IRC | 07:17 | |
| *** pgadiya has joined #openstack-infra | 07:18 | |
| *** aviau has quit IRC | 07:19 | |
| *** pahuang has quit IRC | 07:19 | |
| *** aviau has joined #openstack-infra | 07:19 | |
| *** links has joined #openstack-infra | 07:25 | |
| *** jpena|off is now known as jpena | 07:30 | |
| *** rossella__ has joined #openstack-infra | 07:30 | |
| *** esberglu has joined #openstack-infra | 07:30 | |
| *** rossella_ has joined #openstack-infra | 07:33 | |
| *** esberglu has quit IRC | 07:34 | |
| *** rossella__ has quit IRC | 07:34 | |
| *** yamamoto_ has quit IRC | 07:38 | |
| *** rossella__ has joined #openstack-infra | 07:39 | |
| *** rossella_ has quit IRC | 07:40 | |
| jamespage | ianw: hmm I own that account (openstack-ubuntu-testing-bot) | 07:47 |
| *** JudeC has quit IRC | 07:49 | |
| *** mhickey has joined #openstack-infra | 07:50 | |
| *** mandre is now known as mandre|away | 07:51 | |
| *** rossella__ has quit IRC | 07:52 | |
| *** zz_dimtruck is now known as dimtruck | 07:54 | |
| *** jpich has joined #openstack-infra | 07:55 | |
| *** yamamoto has joined #openstack-infra | 07:57 | |
| *** namnh has quit IRC | 07:58 | |
| *** yamamoto has quit IRC | 07:59 | |
| *** yamamoto has joined #openstack-infra | 07:59 | |
| *** zzzeek has quit IRC | 08:00 | |
| *** zzzeek has joined #openstack-infra | 08:00 | |
| *** markus_z has joined #openstack-infra | 08:01 | |
| openstackgerrit | Alfredo Moralejo proposed openstack-infra/tripleo-ci master: DNM: testing something for gate https://review.openstack.org/471361 | 08:03 |
| *** spzala has joined #openstack-infra | 08:03 | |
| *** thorst_afk has joined #openstack-infra | 08:03 | |
| *** dimtruck is now known as zz_dimtruck | 08:04 | |
| *** jbadiapa_ has joined #openstack-infra | 08:05 | |
| *** rossella__ has joined #openstack-infra | 08:06 | |
| *** eumel8 has quit IRC | 08:06 | |
| *** spzala has quit IRC | 08:07 | |
| *** lucas-afk is now known as lucasagomes | 08:11 | |
| *** rossella__ has quit IRC | 08:13 | |
| *** sflanigan has quit IRC | 08:13 | |
| *** ramishra has quit IRC | 08:18 | |
| *** ramishra has joined #openstack-infra | 08:19 | |
| *** ramishra has quit IRC | 08:20 | |
| *** thorst_afk has quit IRC | 08:23 | |
| *** ramishra has joined #openstack-infra | 08:24 | |
| *** ramishra has quit IRC | 08:25 | |
| *** panda|off is now known as panda | 08:26 | |
| *** pgadiya has quit IRC | 08:26 | |
| *** electrofelix has joined #openstack-infra | 08:27 | |
| *** spzala has joined #openstack-infra | 08:29 | |
| *** Julien-zte has quit IRC | 08:32 | |
| *** spzala has quit IRC | 08:34 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/openstack-zuul-jobs master: Add tox/py34 job https://review.openstack.org/472583 | 08:40 |
| *** yamamoto_ has joined #openstack-infra | 08:46 | |
| *** tosky has joined #openstack-infra | 08:46 | |
| *** derekh has joined #openstack-infra | 08:47 | |
| *** yamamoto has quit IRC | 08:47 | |
| *** dtantsur|sick is now known as dtantsur | 08:48 | |
| *** bauzas is now known as bauwser | 08:52 | |
| *** zz_dimtruck is now known as dimtruck | 08:55 | |
| *** dalvarezs is now known as dalvarez | 09:00 | |
| *** spzala has joined #openstack-infra | 09:00 | |
| *** yamamoto_ has quit IRC | 09:03 | |
| *** dimtruck is now known as zz_dimtruck | 09:05 | |
| *** spzala has quit IRC | 09:05 | |
| *** e0ne has joined #openstack-infra | 09:05 | |
| *** yamamoto has joined #openstack-infra | 09:07 | |
| *** neiljerram has joined #openstack-infra | 09:07 | |
| *** hashar has joined #openstack-infra | 09:09 | |
| *** mika has quit IRC | 09:12 | |
| *** mika has joined #openstack-infra | 09:12 | |
| *** mika has joined #openstack-infra | 09:12 | |
| *** markvoelker has joined #openstack-infra | 09:13 | |
| openstackgerrit | Spyros Trigazis (strigazi) proposed openstack-infra/project-config master: [magnum] Add experimental dcos job https://review.openstack.org/471776 | 09:15 |
| strigazi | ianw Can you have a look at ^^ | 09:15 |
| strigazi | any other infra core ^^ | 09:15 |
| *** yamamoto has quit IRC | 09:16 | |
| *** dizquierdo has joined #openstack-infra | 09:17 | |
| *** esberglu has joined #openstack-infra | 09:18 | |
| *** thorst_afk has joined #openstack-infra | 09:20 | |
| *** ltomasbo is now known as ltomasbo|away | 09:20 | |
| *** ltomasbo|away is now known as ltomasbo | 09:22 | |
| *** esberglu has quit IRC | 09:23 | |
| *** hashar has quit IRC | 09:23 | |
| *** mandre|away is now known as mandre | 09:23 | |
| *** neiljerram has quit IRC | 09:24 | |
| *** thorst_afk has quit IRC | 09:24 | |
| *** jamesmcarthur has joined #openstack-infra | 09:27 | |
| *** spzala has joined #openstack-infra | 09:27 | |
| *** florianf has quit IRC | 09:28 | |
| *** jamesmcarthur has quit IRC | 09:32 | |
| *** spzala has quit IRC | 09:32 | |
| *** MatthewLi has quit IRC | 09:34 | |
| *** yamamoto has joined #openstack-infra | 09:37 | |
| *** LindaWang has quit IRC | 09:37 | |
| *** yamamoto_ has joined #openstack-infra | 09:38 | |
| *** yamamoto has quit IRC | 09:38 | |
| *** Julien-zte has joined #openstack-infra | 09:39 | |
| *** hashar has joined #openstack-infra | 09:40 | |
| *** ramishra has joined #openstack-infra | 09:40 | |
| *** markvoelker has quit IRC | 09:42 | |
| *** florianf has joined #openstack-infra | 09:44 | |
| *** Julien-zte has quit IRC | 09:48 | |
| *** Julien-zte has joined #openstack-infra | 09:49 | |
| *** links has quit IRC | 09:51 | |
| *** liujiong has quit IRC | 09:52 | |
| *** sree has quit IRC | 09:52 | |
| *** neiljerram has joined #openstack-infra | 09:54 | |
| *** zhurong has joined #openstack-infra | 09:55 | |
| *** zz_dimtruck is now known as dimtruck | 09:56 | |
| *** spzala has joined #openstack-infra | 09:59 | |
| *** Julien-zte has quit IRC | 09:59 | |
| *** sree has joined #openstack-infra | 10:01 | |
| *** namnh has joined #openstack-infra | 10:03 | |
| *** links has joined #openstack-infra | 10:03 | |
| *** spzala has quit IRC | 10:04 | |
| *** sambetts|afk is now known as sambetts | 10:04 | |
| *** sree has quit IRC | 10:05 | |
| *** dimtruck is now known as zz_dimtruck | 10:05 | |
| Dinesh_Bhor | Hi all, can anyone point me to the doc which explains how we can create an external CI job and trigger it if any patch submitted to the gerrit? | 10:06 |
| *** dizquierdo has quit IRC | 10:06 | |
| *** zhurong has quit IRC | 10:06 | |
| *** dizquierdo has joined #openstack-infra | 10:10 | |
| *** wolverineav has joined #openstack-infra | 10:14 | |
| *** dizquierdo has quit IRC | 10:14 | |
| *** Julien-zte has joined #openstack-infra | 10:17 | |
| *** hashar has quit IRC | 10:18 | |
| *** sdague has joined #openstack-infra | 10:20 | |
| *** vikrant has quit IRC | 10:21 | |
| openstackgerrit | Alfredo Moralejo proposed openstack-infra/tripleo-ci master: DNM: testing something for gate https://review.openstack.org/471361 | 10:25 |
| *** neiljerram has quit IRC | 10:25 | |
| *** spzala has joined #openstack-infra | 10:27 | |
| amoralej | pabelanger, let me know when you are online please | 10:28 |
| *** spzala has quit IRC | 10:32 | |
| *** zz_dimtruck is now known as dimtruck | 10:35 | |
| *** markvoelker has joined #openstack-infra | 10:39 | |
| *** mpaolino has joined #openstack-infra | 10:41 | |
| *** salv-orlando has quit IRC | 10:42 | |
| *** salv-orlando has joined #openstack-infra | 10:43 | |
| *** pgadiya has joined #openstack-infra | 10:43 | |
| *** mpaolino has quit IRC | 10:45 | |
| *** dimtruck is now known as zz_dimtruck | 10:45 | |
| *** jamesdenton has joined #openstack-infra | 10:45 | |
| *** nicolasbock has joined #openstack-infra | 10:45 | |
| *** salv-orlando has quit IRC | 10:47 | |
| tushar | Dinesh_Bhor : https://docs.openstack.org/infra/openstackci/third_party_ci.html | 10:51 |
| jamespage | anyone around who could review https://review.openstack.org/#/c/472270/ for me? can't move forward on tempest testing for nova-lxd right now without that | 10:54 |
| Dinesh_Bhor | tushar: great, thanks a lot | 10:56 |
| *** spzala has joined #openstack-infra | 10:58 | |
| *** jkilpatr has joined #openstack-infra | 10:59 | |
| *** jascott1 has joined #openstack-infra | 11:01 | |
| *** spzala has quit IRC | 11:02 | |
| *** gomarivera has joined #openstack-infra | 11:03 | |
| *** tuanluong has quit IRC | 11:04 | |
| *** lucasagomes is now known as lucas-hungry | 11:05 | |
| *** jascott1 has quit IRC | 11:06 | |
| *** gomarivera has quit IRC | 11:07 | |
| *** mat128 has joined #openstack-infra | 11:08 | |
| *** psachin has quit IRC | 11:13 | |
| *** markvoelker has quit IRC | 11:13 | |
| openstackgerrit | Daniel Mellado proposed openstack-infra/project-config master: Add kuryr-tempest-plugin project https://review.openstack.org/472645 | 11:22 |
| *** jamesdenton has quit IRC | 11:23 | |
| *** jamesdenton has joined #openstack-infra | 11:24 | |
| *** spzala has joined #openstack-infra | 11:29 | |
| *** iogg has quit IRC | 11:30 | |
| *** iogg has joined #openstack-infra | 11:30 | |
| *** ccamacho is now known as ccamacho|lunch | 11:33 | |
| *** mugsie has quit IRC | 11:34 | |
| *** spzala has quit IRC | 11:34 | |
| *** zz_dimtruck is now known as dimtruck | 11:36 | |
| *** mugsie has joined #openstack-infra | 11:36 | |
| *** mugsie has quit IRC | 11:36 | |
| *** mugsie has joined #openstack-infra | 11:36 | |
| *** pgadiya has quit IRC | 11:38 | |
| *** Julien-zte has quit IRC | 11:39 | |
| *** dizquierdo has joined #openstack-infra | 11:40 | |
| *** yamamoto_ has quit IRC | 11:42 | |
| *** yamamoto has joined #openstack-infra | 11:43 | |
| *** abishop has joined #openstack-infra | 11:43 | |
| openstackgerrit | Merged openstack-infra/project-config master: Increase nova-lxd functional test timeout https://review.openstack.org/472270 | 11:45 |
| *** dimtruck is now known as zz_dimtruck | 11:45 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add seperate pep8 job for Browbeat https://review.openstack.org/470953 | 11:48 |
| openstackgerrit | Merged openstack-infra/project-config master: Add neutron periodic job with ryu master https://review.openstack.org/467885 | 11:51 |
| *** ldnunes has joined #openstack-infra | 11:54 | |
| *** namnh has quit IRC | 11:55 | |
| ianw | jamespage: thanks. i don't really think gerrit feels it at all, just when a human tails the logs you see these constant repeated login attempts :) | 11:55 |
| jamespage | ianw: the jenkins running that activity was not actually using the gerrit stream so I turned it off | 11:55 |
| jamespage | ianw: I think it was a leftover from when we had pre-landing package build tests running for core openstack projects | 11:56 |
| jamespage | ianw: but that generated alot of noise for developers, and alot of load for the supporting infra | 11:56 |
| *** gcb has quit IRC | 11:57 | |
| *** thorst_afk has joined #openstack-infra | 11:59 | |
| *** spzala has joined #openstack-infra | 12:00 | |
| *** katkapilatova has joined #openstack-infra | 12:02 | |
| *** askb has quit IRC | 12:02 | |
| *** xarses has quit IRC | 12:03 | |
| *** spzala has quit IRC | 12:04 | |
| *** lucas-hungry is now known as lucasagomes | 12:05 | |
| amoralej | hi infra | 12:05 |
| amoralej | i need some help to debug an issue we have been working on for some days without success. | 12:05 |
| amoralej | i'd like to get a node hold and get access into it | 12:06 |
| amoralej | pabelanger, helped me yesterday, but it seems he is not online yet | 12:06 |
| amoralej | anyone could help me? | 12:06 |
| *** shardy has quit IRC | 12:08 | |
| EmilienM | amoralej: you would likely get some help when fungi or pabelanger are online | 12:08 |
| amoralej | thanks EmilienM | 12:09 |
| *** markvoelker has joined #openstack-infra | 12:10 | |
| *** mugsie has quit IRC | 12:10 | |
| *** ldnunes has quit IRC | 12:12 | |
| *** mugsie has joined #openstack-infra | 12:12 | |
| *** mugsie has quit IRC | 12:12 | |
| *** mugsie has joined #openstack-infra | 12:12 | |
| *** trown|outtypewww is now known as trown | 12:12 | |
| *** xarses has joined #openstack-infra | 12:13 | |
| *** jcoufal has joined #openstack-infra | 12:14 | |
| *** xarses_ has joined #openstack-infra | 12:14 | |
| *** xarses_ has quit IRC | 12:14 | |
| *** xarses_ has joined #openstack-infra | 12:14 | |
| *** shardy has joined #openstack-infra | 12:17 | |
| *** xarses has quit IRC | 12:18 | |
| *** bobh has joined #openstack-infra | 12:18 | |
| *** dizquierdo has quit IRC | 12:22 | |
| *** ramishra has quit IRC | 12:23 | |
| *** HeOS has joined #openstack-infra | 12:24 | |
| *** abishop has left #openstack-infra | 12:24 | |
| *** ldnunes has joined #openstack-infra | 12:25 | |
| *** rlandy has joined #openstack-infra | 12:25 | |
| *** bobh has quit IRC | 12:27 | |
| *** spzala has joined #openstack-infra | 12:28 | |
| *** jpena is now known as jpena|lunch | 12:28 | |
| *** amoralej is now known as amoralej|lunch | 12:29 | |
| *** yland has joined #openstack-infra | 12:31 | |
| *** spzala has quit IRC | 12:33 | |
| *** markvoelker has quit IRC | 12:36 | |
| *** markvoelker has joined #openstack-infra | 12:36 | |
| *** zz_dimtruck is now known as dimtruck | 12:36 | |
| *** jaosorior has quit IRC | 12:37 | |
| *** gordc has joined #openstack-infra | 12:38 | |
| *** trown is now known as trown|afk | 12:40 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool feature/zuulv3: Add webapp port and listen_address configuration https://review.openstack.org/472128 | 12:41 |
| *** Julien-zte has joined #openstack-infra | 12:42 | |
| *** dave-mccowan has joined #openstack-infra | 12:43 | |
| *** EricGonczer_ has joined #openstack-infra | 12:43 | |
| openstackgerrit | Daniel Mellado proposed openstack-infra/project-config master: Add kuryr-tempest-plugin project https://review.openstack.org/472645 | 12:43 |
| *** shardy has quit IRC | 12:43 | |
| *** dimtruck is now known as zz_dimtruck | 12:46 | |
| pabelanger | amoralej|lunch: EmilienM: usually just better to ask for something, then waiting for a person to come online. Everybody mostly reads backscoll. | 12:48 |
| pabelanger | what do you need help with today | 12:48 |
| *** links has quit IRC | 12:48 | |
| *** ccamacho|lunch is now known as ccamacho | 12:49 | |
| tushar | Pabelanger : Hi | 12:50 |
| tushar | Pablanger : In nodepool creation getting failed for me. error log -https://gist.github.com/anonymous/c77476be4c7b55657269225041314474 | 12:51 |
| *** baoli has joined #openstack-infra | 12:51 | |
| *** baoli_ has joined #openstack-infra | 12:52 | |
| pabelanger | tushar: are you using the initialize-urandom / runtime-ssh-host-keys elements? | 12:53 |
| pabelanger | tushar: that error to me means there is something wrong with SSH | 12:53 |
| tushar | Pabelanger : yes, using initialize-urandom | 12:53 |
| pabelanger | tushar: you might need to get the console log from openstack command to see what services have been properly started | 12:54 |
| pabelanger | openstack console log show <instance> | 12:54 |
| pabelanger | newer versions of nodepool we have a console-log: True setting that make things a little easier | 12:54 |
| tushar | Pabelanger : console log should be taken from the provider or the server on which nodepool is running? | 12:56 |
| *** baoli has quit IRC | 12:56 | |
| *** shardy has joined #openstack-infra | 12:56 | |
| *** cuongnv has quit IRC | 12:57 | |
| *** ihrachys has joined #openstack-infra | 12:59 | |
| *** spzala has joined #openstack-infra | 12:59 | |
| *** kgiusti has joined #openstack-infra | 13:03 | |
| *** spzala has quit IRC | 13:03 | |
| pabelanger | tushar: either will work. The console log will be from the instance that nodepool is failing to launch | 13:03 |
| *** yamamoto has quit IRC | 13:04 | |
| *** yamamoto has joined #openstack-infra | 13:04 | |
| tushar | Pabelnager : Yes, I got that log from the provider | 13:06 |
| *** Goneri has joined #openstack-infra | 13:07 | |
| *** mugsie has quit IRC | 13:08 | |
| tushar | Pabelanger : what is the signficance of initialize-urandom, is there any documenation where description of all the nodepool elements is there? | 13:08 |
| *** mugsie has joined #openstack-infra | 13:11 | |
| *** mugsie has quit IRC | 13:11 | |
| *** mugsie has joined #openstack-infra | 13:11 | |
| *** mdrabe has joined #openstack-infra | 13:11 | |
| *** bobh has joined #openstack-infra | 13:11 | |
| *** dizquierdo has joined #openstack-infra | 13:11 | |
| *** cshen has quit IRC | 13:16 | |
| tushar | Pabelanger : console log of instance - https://gist.github.com/anonymous/b3d14d7a5cd599c0636a3d43a0eb2726 | 13:18 |
| pabelanger | tushar: we had to write initialize-urandom to work around a kernel bug, it just pumps entropy to urandom. That helps SSHd generate keys | 13:21 |
| pabelanger | tushar: are you using ubuntu or ubuntu-minimal elements? | 13:22 |
| pabelanger | tushar: also, you should be able to manually boot the DIB from your clouds and see if you can log in | 13:22 |
| pabelanger | tushar: are you using openssh-server element too? | 13:23 |
| tushar | Pabelanger : We are using ubuntu | 13:23 |
| tushar | not using openssh-server | 13:24 |
| *** jcoufal_ has joined #openstack-infra | 13:24 | |
| *** cshen has joined #openstack-infra | 13:25 | |
| pabelanger | k, you likley need that too | 13:25 |
| pabelanger | I should have been a dependency of elements however | 13:25 |
| pabelanger | tushar: can you pastebin / gist your build log again | 13:25 |
| pabelanger | tushar: you also can switch to ubuntu-minimal that is more inline with the images we build | 13:25 |
| *** jcoufal has quit IRC | 13:27 | |
| *** sbezverk has joined #openstack-infra | 13:28 | |
| *** vhosakot has joined #openstack-infra | 13:28 | |
| *** esberglu has joined #openstack-infra | 13:28 | |
| *** spzala has joined #openstack-infra | 13:29 | |
| *** xyang1 has joined #openstack-infra | 13:29 | |
| *** spzala has quit IRC | 13:31 | |
| *** spzala has joined #openstack-infra | 13:31 | |
| clarkb | I'm going to work on addressing the translate.openstack.org disk issue frank pointed out on the mailing list | 13:34 |
| tushar | Pabelanger : Build log - https://gist.github.com/anonymous/e8c769925839726f6eefa0cc57fef046 | 13:35 |
| tushar | Pabelanger : these are the elements which we are using currently - https://gist.github.com/anonymous/eb1616c76267c2369ddcdb1dd9a82757 | 13:36 |
| tushar | Pabelange : I am going to replace ubuntu with ubuntu-minimal, any another change ? | 13:36 |
| *** trown|afk is now known as trown | 13:37 | |
| *** dansmith is now known as superdan | 13:37 | |
| *** zz_dimtruck is now known as dimtruck | 13:37 | |
| *** dhajare|busy has quit IRC | 13:37 | |
| *** amoralej|lunch is now known as amoralej | 13:38 | |
| *** jpena|lunch is now known as jpena | 13:38 | |
| pabelanger | tushar: yes that, you can remove stackviz and need to add openssh-server | 13:38 |
| pabelanger | let me see why that isn't default now | 13:38 |
| pabelanger | Oh, it is there | 13:39 |
| pabelanger | so ya, ubuntu-minimal and stackviz can be removed | 13:39 |
| tushar | okk | 13:40 |
| pabelanger | clarkb: ack | 13:40 |
| tushar | Pabelanger : I will try with this | 13:40 |
| fungi | thanks clarkb! | 13:40 |
| *** mriedem has joined #openstack-infra | 13:40 | |
| pabelanger | tushar: ack | 13:40 |
| openstackgerrit | Clark Boylan proposed openstack-infra/puppet-zanata master: Add cron to clean out logs https://review.openstack.org/472683 | 13:40 |
| clarkb | pabelanger: fungi I ran the find in ^ manually which has us back to ~55% disk free. Getting that change in should make it all automagic from now on | 13:41 |
| clarkb | the server seemed to not be completely full on its disk this time around and the zanata service is running so I think that may be all we need to do. Will ask frank to confirm | 13:42 |
| pabelanger | +2 | 13:42 |
| *** jamesdenton has quit IRC | 13:43 | |
| amoralej | pabelanger, i'm going to need a new hold today, sorry but we are totally lost | 13:43 |
| amoralej | is that ok? | 13:43 |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Use openstack status.json file for zuulv3.o.o https://review.openstack.org/472475 | 13:44 |
| pabelanger | clarkb: fungi: ^ and depends-on would be helpful this morning | 13:44 |
| pabelanger | amoralej: sure, have you tried reproducing the issue locally? We try not rely on holding nodes too much for developer workflows. | 13:44 |
| pabelanger | amoralej: do you have a node id? | 13:45 |
| *** ericyoung has joined #openstack-infra | 13:45 | |
| amoralej | pabelanger, yeah, we've tried but we couldn't reproduced it | 13:46 |
| pabelanger | infra-root: does anybody know anything about the autohold on gate-tempest-dsvm-neutron-full-ubuntu-xenial jobs? We have 3 of them 7 days old, can they be cleaned up? | 13:46 |
| pabelanger | fungi: can we also delete your post_failure debug with dmellado too? | 13:46 |
| amoralej | i know it's not a good practice, but it's the only way i can think of to investigate it | 13:46 |
| amoralej | i need to start a new job and patch it to not delete the overcloud automatically | 13:47 |
| dmellado | pabelanger: feel free to burn it out | 13:47 |
| dmellado | at least from my side ;) | 13:47 |
| pabelanger | amoralej: we can hold it no problem, just don't want to depend on it too much. Since it takes capacity away | 13:47 |
| pabelanger | dmellado: thanks | 13:48 |
| clarkb | pabelanger: I want to say that job autohold may have been to try and debug the cinder lvm problems we had. Then they mysteriously went away (guessing distro packaging fixes) | 13:48 |
| ianychoi | clarkb, I thought you adjusted the partition but it seems that the log size incresing ratio is too high? Could u check which kind of log messages are mainly in the log file? | 13:48 |
| clarkb | pabelanger: so yes I think you can remove that hold | 13:48 |
| pabelanger | clarkb: thanks | 13:48 |
| *** jcoufal_ has quit IRC | 13:48 | |
| clarkb | ianychoi: the problem is we never delete logs | 13:48 |
| clarkb | ianychoi: wildfly's logging config rotates daily but doesn't delete anything | 13:49 |
| clarkb | ianychoi: and it is very chatty because it reloads its config every 30 seconds or something | 13:49 |
| clarkb | ianychoi: https://review.openstack.org/472683 should address that for us | 13:49 |
| pabelanger | clarkb: okay to delete the citycloud-kna1 node you used to debug private IP addresses? | 13:50 |
| *** jcoufal has joined #openstack-infra | 13:50 | |
| fungi | pabelanger: i don't see the held node for the post_failure debugging, but i don't even remember what that was now | 13:51 |
| *** chlong has joined #openstack-infra | 13:51 | |
| clarkb | pabelanger: which nodeid? I have one in kna1 I am using to test the afs stuff from yesterday that I still need | 13:51 |
| pabelanger | clarkb: 9085392 | 13:52 |
| fungi | pabelanger: i don't know about the autohold either. i didn't set it but maybe we can find it in the sudo logs? | 13:52 |
| pabelanger | fungi: ack, I've cleaned it up based on dmellado reply | 13:52 |
| clarkb | pabelanger: yes that one can be deleted | 13:52 |
| pabelanger | fungi: I think clarkb was the autohold, which is also removed now | 13:52 |
| fungi | cool, thanks! | 13:52 |
| *** dimtruck is now known as zz_dimtruck | 13:53 | |
| *** sbezverk has quit IRC | 13:53 | |
| fungi | clarkb: i think puppet is unhappy with your escaping in 472683 | 13:53 |
| openstackgerrit | Rosario Di Somma proposed openstack-infra/shade master: Convert data from raw clients to dict or list of dicts https://review.openstack.org/472690 | 13:53 |
| pabelanger | clarkb: is 9191322 / 9190088 / 9190626 also yours for testing? They are 1:22:56 old but didn't have a reason | 13:53 |
| pabelanger | jeblair: do you need 9165083 anymore also? That is currently held in bluebox-sjc1 | 13:54 |
| clarkb | pabelanger: no I've got three I am using for afs testing and all three have my name in the reason field | 13:54 |
| clarkb | not sure what those others are | 13:54 |
| fungi | clarkb: oh, also puppet lint has introduced new rules since the last time we did anything with that repo, looks like | 13:55 |
| pabelanger | thanks, I'll see look and see who held them | 13:55 |
| clarkb | fungi: yay! | 13:55 |
| pabelanger | funtime | 13:55 |
| clarkb | fungi: I work on getting it cleaned up | 13:55 |
| clarkb | I've got to take kids to a doctor's appointment mid morning then afterwards is when I intend on doing the additional testing with vos release on the pypi volume | 13:56 |
| *** jaosorior has joined #openstack-infra | 13:56 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool feature/zuulv3: Add webapp port and listen_address configuration https://review.openstack.org/472128 | 13:56 |
| fungi | clarkb: eep, mutual dependency cycle noted between Cron[cleanup-wildfly-logs] and Class[Zanata::Wildfly] | 13:56 |
| clarkb | fungi: how | 13:56 |
| clarkb | Oh I know | 13:56 |
| clarkb | ugh | 13:56 |
| clarkb | puppet | 13:56 |
| *** rbrndt has joined #openstack-infra | 13:58 | |
| *** jcoufal has quit IRC | 13:59 | |
| bauwser | gerrit is in the weeds | 14:00 |
| bauwser | at least when uploading | 14:00 |
| fungi | again!?! ianw also saw what looked like connection pileups on gerrit overnight | 14:00 |
| fungi | o | 14:00 |
| pabelanger | mordred: Shrews: possible that one of you might be using the 3 held nodes in rax for zuul stream testing? | 14:00 |
| clarkb | ya git review just told me I may be mitm'd | 14:00 |
| fungi | i'll start trying to hunt down and block the new offender | 14:00 |
| clarkb | fungi: thanks | 14:00 |
| bauwser | fungi: "Received disconnect from 104.130.246.91: 9: sendKexInit() none of the keys appears in supported list: [ssh-dss, ssh-rsa, ecdsa-sha2-nistp256, ecdsa-sha2-nistp384, ecdsa-sha2-nistp521]" | 14:01 |
| Shrews | pabelanger: not I | 14:01 |
| fungi | 169.48.164.163 has 4852 open connections to the api. adding to iptables now | 14:01 |
| bauwser | I can try connecting thru HTTP tho | 14:01 |
| fungi | blocked, i'm going to try to start closing the open connections from it | 14:01 |
| fungi | SpamapS: 169.48.164.163 seems to be another softlayer host? | 14:02 |
| clarkb | that is what whois says | 14:03 |
| fungi | mestery doesn't lurk in here any longer | 14:03 |
| fungi | i' | 14:03 |
| fungi | i'm tempted to block the entire netblock | 14:03 |
| clarkb | it would be nice if we can get them to confirm their zuul version and maybe we need to make a release to include my fix for this paramiko behavior | 14:03 |
| *** jamesdenton has joined #openstack-infra | 14:03 | |
| *** nicolasbock has quit IRC | 14:03 | |
| pabelanger | fungi: ++ | 14:03 |
| clarkb | we had whoever it was last time (I don't remember) test that the fix actually fixed it for them but guessing that softlayer is grabbing the last release which doesn't include my fix? | 14:04 |
| pabelanger | if they are puppet-openstackci, it should be using master branch of zuul | 14:04 |
| pabelanger | and 2.5.2 also includes it | 14:04 |
| clarkb | ah ok | 14:04 |
| clarkb | so either the fix isn't sufficient or they are not deploying what we expect? | 14:04 |
| fungi | earlier this week we confirmed the paramiko bump appeared in 2.5.2, but SpamapS also said they were deploying via puppet-openstackci which installs zuul master branch tip unless you override that | 14:04 |
| pabelanger | clarkb: agree | 14:04 |
| clarkb | I was fairly happy with the fix at the time though because I know we tested it but maybe the testing didn't test what I expected | 14:05 |
| *** mhickey has quit IRC | 14:06 | |
| fungi | #status log Blocked 169.48.164.163 in iptables on review.o.o temporarily for excessive connection counts | 14:06 |
| openstackstatus | fungi: finished logging | 14:06 |
| clarkb | fungi: fwiw I would be fine blocking the entire range if this is a persistent issue and we can't get them here or via email to work through it | 14:06 |
| clarkb | what I did last time was blocked the IPs until I had interactive session in irc with individuals where we worked through the problem and confirmed that the issue was fixed | 14:07 |
| fungi | looks like the three softlayer ip addresses were from completely different first octets even, so unless we work them all out from bgp announcements i don't think that's a viable approach after all | 14:07 |
| clarkb | ah | 14:07 |
| *** jcoufal has joined #openstack-infra | 14:07 | |
| fungi | and it's not like we can revoke an account to prevent this... the misconfigured hosts don't even have valid authentication set up | 14:08 |
| clarkb | also you can disable the accounts | 14:08 |
| clarkb | but ya that | 14:08 |
| *** markus_z has quit IRC | 14:08 | |
| clarkb | fungi: maybe we want to set up a global iptables -m connlimit rule on review.o.o? | 14:10 |
| fungi | i've closed over 10% of the connections from that address already, so hopefully errors are ceasing | 14:10 |
| clarkb | or at least for port 29418, limit to say 100 connections? | 14:10 |
| fungi | clarkb: i wonder if the kernel actually sees that many or if it's just the jvm... checking | 14:10 |
| *** zz_dimtruck is now known as dimtruck | 14:10 | |
| pabelanger | Didn't SpamapS say it was from people testing bonnyCI things? Maybe they are defaulting some thing to review.o.o by mistake | 14:11 |
| fungi | oh, yep, kernel sees them too. so that might work | 14:11 |
| fungi | pabelanger: yes, the assumption is that they're using a default they shouldn't be. he confirmed they shouldn't even be trying to connect to our gerrit from those at all | 14:11 |
| *** felipemonteiro has joined #openstack-infra | 14:12 | |
| fungi | clarkb: i like the connlimit idea. happen to have the syntax handy? | 14:12 |
| clarkb | I think a connlimit would mitigate this well as its repeated bad connections (due to client error) from a single source that hurts us. If this was a distributed attack it would be less useful but thankfully it is single source | 14:12 |
| clarkb | fungi: google says https://unix.stackexchange.com/questions/139285/limit-max-connections-per-ip-address-and-new-connections-per-second-with-iptable | 14:13 |
| clarkb | fungi: I'm reading manpages now to see if there is a better construction for us | 14:13 |
| fungi | right, this is not an intentional attack, so our options are much broader | 14:13 |
| *** felipemonteiro__ has joined #openstack-infra | 14:13 | |
| clarkb | `iptables -A INPUT -p tcp --syn --dport 23 -m connlimit --connlimit-above 2 -j REJECT` is the manpage example for limiting telnet to 2 connections | 14:13 |
| clarkb | so I think you can s/23/29418/ s/2/100/ and that will limit a single client to 100 gerrit git connections which should be plenty | 14:14 |
| fungi | nice. i guess we need that earlier in the chain than the 29418/tcp allow? | 14:14 |
| clarkb | fungi: I think so since first match wins in iptbales | 14:15 |
| *** jcoufal has quit IRC | 14:15 | |
| clarkb | s/git/ssh/ | 14:15 |
| *** mriedem has quit IRC | 14:15 | |
| *** jcoufal has joined #openstack-infra | 14:15 | |
| *** tesseract has quit IRC | 14:16 | |
| fungi | once i'm done shutting down these offending connections, i'll add that and then remove the block for 169.48.164.163 and see if it gets past 100 | 14:16 |
| mordred | pabelanger: this makes me think that maybe the mqtt/fedmsg trigger for v3 may be more important :) | 14:16 |
| *** felipemonteiro has quit IRC | 14:17 | |
| clarkb | # allow 2 telnet connections per client host is the comment for that command too. So it is per client host/ip | 14:17 |
| clarkb | won't limit us to 100 total from all clients. Though we may consider putting such a rule in place for whatever our ssh max is in total too | 14:18 |
| clarkb | I'm trying to push to puppet-zanata and still failing... git review is out to lunch. I shall be patient | 14:18 |
| *** ldnunes has quit IRC | 14:19 | |
| clarkb | remote: error: internal error while processing changes | 14:19 |
| *** panda is now known as panda|bbl | 14:19 | |
| clarkb | fungi: ^ fyi | 14:19 |
| fungi | i've already closed 1k connections from that address... i guess it wants lower still | 14:19 |
| pabelanger | mordred: Ya, not a good week for gerrit, that is for sure | 14:19 |
| fungi | and it's a milestone week... unfortunate timing | 14:20 |
| clarkb | possibly worth a bug to mina ssh too? | 14:20 |
| clarkb | or gerrit maybe | 14:20 |
| *** xyang1 has quit IRC | 14:20 | |
| *** mhickey has joined #openstack-infra | 14:21 | |
| pabelanger | connlimit sounds like a good step if we can get that working | 14:21 |
| *** rfolco has joined #openstack-infra | 14:21 | |
| *** tesseract has joined #openstack-infra | 14:23 | |
| *** xyang1 has joined #openstack-infra | 14:24 | |
| *** ykarel has quit IRC | 14:25 | |
| sdague | is gerrit gone all crazy again? | 14:26 |
| sdague | ah... yeh, seems like | 14:26 |
| sdague | host 169.48.164.163 | 14:26 |
| sdague | oh, it's another softlayer system? | 14:27 |
| *** gardlt has joined #openstack-infra | 14:27 | |
| lbragstad | i think i'm seeing some strange behavior with gerrit, too? | 14:28 |
| mordred | lbragstad: yah - we've have another DOS | 14:28 |
| *** liusheng has quit IRC | 14:29 | |
| lbragstad | mordred: ah | 14:29 |
| mordred | clarkb: do you think we need the limit to be as high as 100? | 14:29 |
| *** liusheng has joined #openstack-infra | 14:30 | |
| mordred | (I mean, 100 is obviously an improvement if it works - more just bikeshedding on what we think the limit _should_ be) | 14:30 |
| *** cshastri has quit IRC | 14:30 | |
| *** jamesdenton has quit IRC | 14:30 | |
| mordred | I guess it might be problematic for folks behind a corporate firewall if we lower it too much | 14:30 |
| fungi | looking at the counts, i saw others up in the neighborhood of 20 | 14:31 |
| *** ldnunes has joined #openstack-infra | 14:31 | |
| *** prateek has quit IRC | 14:31 | |
| fungi | so 100 is probably high enough to only be hit if something is really, really gone awry | 14:31 |
| fungi | all the offenders i've blocked this week were well over 500 | 14:31 |
| fungi | most in the thousands | 14:32 |
| clarkb | mordred: fungi ya figured its high enough no one doing legit stuff should notice but low enough to still be less than 1/10 of the aggregate limit | 14:32 |
| fungi | and at this point i've closed 50% of the connections from 169.48.164.163 so i'm starting to wonder if this just leaves mina-sshd indefinitely broken | 14:32 |
| clarkb | we can probably go lower, ~25? | 14:33 |
| clarkb | mordred: I mostyly worry about NAT | 14:33 |
| clarkb | 100 should avoid NAT problems (I hope0 | 14:33 |
| fungi | right, i assume a number of the higher counts are from multiple users behind a common nat | 14:33 |
| sdague | yeh, the NAT question is a problem. I know I hit that before when being on a corp firewall. | 14:34 |
| fungi | similarly, when we all get together at conferences... | 14:34 |
| sdague | is there any way to determine the connections / ip / user | 14:34 |
| sdague | or are we kind of disconnected there? | 14:34 |
| fungi | sdague: in this case there is no user. these are preauthentication | 14:34 |
| sdague | that would like you whitelist known NATs | 14:34 |
| *** EricGonc_ has joined #openstack-infra | 14:34 | |
| *** EricGonczer_ has quit IRC | 14:35 | |
| *** e0ne has quit IRC | 14:35 | |
| fungi | oh, i get you. yes we can scrape gerrit's ssh log for details on which account names connect from what ip addresses. also active connections have their account and ip address reported in the show-connections api output | 14:36 |
| clarkb | sdague: fungi mordred thankfully most nat connections are transient, you push code or list projects or whatever | 14:36 |
| pabelanger | fail2ban could do that too | 14:36 |
| fungi | right. it's mainly event stream connections which persist | 14:36 |
| pabelanger | used to use it for asterisk systems a lot | 14:36 |
| clarkb | sdague: fungi so you need to have 100 people all making an ssh connection at once (possibly conceivable at at summit/ptg but even then not too worried if limit is ~100) | 14:37 |
| clarkb | pabelanger: I've never had good luck with fail2ban | 14:37 |
| *** jamesmcarthur has joined #openstack-infra | 14:37 | |
| jeblair | pabelanger: i have deleted node 9165083 | 14:37 |
| clarkb | I fiddled with it back when we "had" to allow ssh with passwords | 14:37 |
| clarkb | turns out people get passwords wrong a lot (not the case here but ya) | 14:38 |
| fungi | right, i think 100 is probably a fine compromise and if we get reports of issues during summit or ptg we can always whitelist those nat pools manually | 14:38 |
| pabelanger | clarkb: it worked okay when you had a rouge phone on your network, not sure I ever used it on public web | 14:38 |
| *** jamesmcarthur has quit IRC | 14:38 | |
| pabelanger | jeblair: great, thanks | 14:38 |
| *** jamesmcarthur has joined #openstack-infra | 14:38 | |
| pabelanger | also, rogue* | 14:38 |
| mordred | yah. and I think medium-term getting an mqtt option for folks for 3rd party ci will be a great choice for mitigation | 14:38 |
| mordred | since we can then limit acl access to stream-events and konw that ssh connections are all from users - but that's ont a this-month thing | 14:39 |
| *** claudiub has quit IRC | 14:40 | |
| pabelanger | for some reason we have a 16+hr noop job in check pipeline | 14:41 |
| pabelanger | looking to see why | 14:41 |
| *** yland has quit IRC | 14:42 | |
| *** salv-orlando has joined #openstack-infra | 14:42 | |
| pabelanger | odd | 14:42 |
| pabelanger | https://review.openstack.org/#/c/472459/ | 14:42 |
| pabelanger | refs/meta/config | 14:42 |
| pabelanger | https://review.openstack.org/#/c/472460/ too | 14:43 |
| pabelanger | I guess somebody used the gerrit webui to try and update something? | 14:43 |
| *** yland has joined #openstack-infra | 14:43 | |
| fungi | pabelanger: i suspect that repo is waiting for a job that no launchers have registered | 14:43 |
| *** yland is now known as Guest22695 | 14:44 | |
| *** salv-orl_ has joined #openstack-infra | 14:44 | |
| *** Dinesh_Bhor has quit IRC | 14:44 | |
| fungi | pabelanger: but yes, also, the acl editor is an attractive nuisance | 14:44 |
| pabelanger | ya, I think we can safely abandon them, that would clear the job in check | 14:44 |
| fungi | pabelanger: mind abandoning those reviews with a pointer to the appropriate infra-manual section? | 14:44 |
| pabelanger | fungi: yes, mentioned openstack-infra but manual is a good idea too | 14:46 |
| fungi | oh weird. the openstack/mors project is getting those jobs via the noop-jobs project-template so i guess there's no layout misconfiguration. wonder if zuul-merger is failing to create the merge commit because it can't checkout the "branch" there? | 14:46 |
| clarkb | fungi: do you expect gerrit to be happier now? | 14:48 |
| pabelanger | ya, will check logs shortly | 14:48 |
| *** salv-orlando has quit IRC | 14:48 | |
| fungi | clarkb: dunno, we're down to 1.5k open connections from the offending system now, but i don't know how many sends mina-sshd into crazytown | 14:48 |
| fungi | approaching 1k open | 14:50 |
| *** jbadiapa_ has quit IRC | 14:51 | |
| fungi | i'm curious to see whether closing the connections gets it back into a working state, or if that's futile and we just have to restart gerrit regardless when this happens | 14:51 |
| *** xarses has joined #openstack-infra | 14:51 | |
| dtantsur | hi folks! is gerrit down/has problems? | 14:52 |
| clarkb | fungi: I'm beginning to think restarting gerrit may be required based on the number of ^ reports | 14:52 |
| odyssey4me | completely busted for me for a few mins now | 14:53 |
| *** andreww has joined #openstack-infra | 14:53 | |
| fungi | yeah, we're down to only 1k open connections from softlayer at this stage | 14:53 |
| dtantsur | also seeing 'internal server error' from 'git review' command | 14:53 |
| pabelanger | zuul also hasn't launched anything in 45mins now | 14:54 |
| fungi | starting to agree once mina-sshd tips over into broken from a connection flood there may be no recovering short of restarting gerrit completely | 14:54 |
| pabelanger | ++ | 14:55 |
| fungi | #status notice The Gerrit service on review.openstack.org is being restarted now to clear an issue arising from an unanticipated SSH API connection flood | 14:55 |
| openstackstatus | fungi: sending notice | 14:55 |
| *** bnemec is now known as beekneemech | 14:55 | |
| beekneemech | Good timing. It just fell over completely for me. :-) | 14:55 |
| *** xarses has quit IRC | 14:56 | |
| *** jpena is now known as jpena|off | 14:56 | |
| -openstackstatus- NOTICE: The Gerrit service on review.openstack.org is being restarted now to clear an issue arising from an unanticipated SSH API connection flood | 14:56 | |
| *** xarses has joined #openstack-infra | 14:57 | |
| *** andreww has quit IRC | 14:57 | |
| openstackstatus | fungi: finished sending notice | 14:58 |
| *** makowals has quit IRC | 14:58 | |
| *** jpena|off is now known as jpena | 14:58 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: WIP: Add reporter for Federated Message Bus (fedmsg) https://review.openstack.org/426861 | 15:00 |
| *** mdrabe has quit IRC | 15:00 | |
| amoralej | pabelanger, this is the node to hold "[Zuul] Building remotely on tripleo-centos-7-tripleo-test-cloud-rh1-9232499" | 15:02 |
| *** armax has joined #openstack-infra | 15:02 | |
| dtantsur | just FYI: this outage resulted in a few patches lost after "git review". may be worth asking people to double-check. | 15:02 |
| pabelanger | amoralej: ssh public key again please | 15:03 |
| openstackgerrit | Clark Boylan proposed openstack-infra/puppet-zanata master: Add cron to clean out logs https://review.openstack.org/472683 | 15:03 |
| openstackgerrit | Clark Boylan proposed openstack-infra/puppet-zanata master: Fix lint issues https://review.openstack.org/472722 | 15:03 |
| *** HeOS has quit IRC | 15:03 | |
| amoralej | pabelanger, https://github.com/amoralej.keys | 15:03 |
| clarkb | fungi: pabelanger ^ I think that should address the puppet issues on my first patch | 15:03 |
| clarkb | dtantsur: ya I noticed that with the above changes. The parent never got created (would've been a new change) and the child got a new patchset that pointed to the missing change | 15:04 |
| pabelanger | amoralej: ssh jenkins@66.187.229.137 | 15:04 |
| clarkb | dtantsur: so I think it only affected new changes, but newp atchsets are ok (at least with a sample size of one) | 15:04 |
| dtantsur | good to knwo | 15:04 |
| amoralej | pabelanger, thx | 15:05 |
| *** tmorin has joined #openstack-infra | 15:05 | |
| dtantsur | yeah, I did manage to update a few patches as well, despite the internal error | 15:05 |
| fungi | clarkb: i've tried inserting the connlimit reject at the start of the INPUT chain and removing all the host-specific drop rules. watching connection counts now to see what happens | 15:05 |
| fungi | 169.48.164.163 is already up to 21 | 15:06 |
| fungi | going to see if it manages to push past 100 | 15:06 |
| pabelanger | okay, mors stuck jobs removed after abandoning the patches | 15:07 |
| tmorin | hi guys, seems like no test job has been triggered for some changes submitted 10 minutes ago: are the test job going to start, or were the triggers lost and we need to do rechecks ? | 15:07 |
| *** cshen has quit IRC | 15:07 | |
| pabelanger | tmorin: yes, feel free to recheck. We had to restart gerrit | 15:08 |
| *** rcernin has quit IRC | 15:08 | |
| tmorin | pabelanger: thanks, I didn't want to overload the thing uselessly :) | 15:08 |
| openstackgerrit | Merged openstack-infra/puppet-zuul master: Expose zuul_status_url for zuulv3 https://review.openstack.org/472474 | 15:09 |
| szaher | pabelanger: Can I post new changes now ? | 15:09 |
| fungi | three of the addresses we've blocked this week are already up over 20 connections now | 15:09 |
| *** jaosorior has quit IRC | 15:09 | |
| pabelanger | szaher: yes, gerrit is ready to be used again | 15:09 |
| fungi | so this will be a good test (as long as i keep a close eye on things) | 15:09 |
| szaher | pabelanger: Thanks :) | 15:09 |
| clarkb | fungi: so they never even turned them off? | 15:10 |
| *** mdrabe has joined #openstack-infra | 15:10 | |
| fungi | clarkb: correct | 15:10 |
| fungi | yjeu | 15:10 |
| fungi | grrr | 15:10 |
| szaher | pabelanger: well for me, I still can't push anything to project-config | 15:10 |
| fungi | they've just been retrying )brokenly) all week ;) | 15:10 |
| openstackgerrit | Merged openstack-infra/system-config master: Use openstack status.json file for zuulv3.o.o https://review.openstack.org/472475 | 15:11 |
| clarkb | szaher: how is it failing and when did you try? | 15:11 |
| pabelanger | szaher: do you get an error? | 15:11 |
| szaher | pabelanger: To ssh://szaher@review.openstack.org:29418/openstack-infra/project-config.git | 15:11 |
| szaher | ! [remote rejected] HEAD -> refs/publish/master/freezer-tempest-plugin (no new changes) | 15:11 |
| szaher | no new changes | 15:11 |
| *** hongbin has joined #openstack-infra | 15:11 | |
| clarkb | szaher: right so the error is in the (), no new changes. that is the problem that dtantsur was talking about | 15:11 |
| clarkb | szaher: step 1 is checking if it actually did push succesffully the first time | 15:12 |
| clarkb | szaher: lookup the change by change id in gerrit | 15:12 |
| clarkb | if it did push then you are done. If it didn't then you will need to make some sort of chagne to your chagn and repush it so that the commit is different and gerrit stops complaining | 15:13 |
| *** rkukura has joined #openstack-infra | 15:13 | |
| szaher | clarkb: Ok, Thanks :) | 15:13 |
| *** jamesmcarthur has quit IRC | 15:13 | |
| openstackgerrit | Saad Zaher proposed openstack-infra/project-config master: Added Freezer tempest plugin repo https://review.openstack.org/472732 | 15:15 |
| *** salv-orl_ has quit IRC | 15:16 | |
| *** salv-orlando has joined #openstack-infra | 15:17 | |
| *** rderose has joined #openstack-infra | 15:18 | |
| *** dtantsur is now known as dtantsur|afk | 15:19 | |
| fungi | okay, now we have four misconfigured and previously blocked addresses with connection counts over 20 | 15:20 |
| fungi | some seem to climb faster than others | 15:20 |
| *** salv-orlando has quit IRC | 15:20 | |
| *** rkukura has quit IRC | 15:21 | |
| *** jbadiapa has joined #openstack-infra | 15:21 | |
| clarkb | they must have more cpus to negotiate the initial diffie hellmann portion of the ssh connection :P | 15:21 |
| clarkb | s/more/faster/ | 15:21 |
| fungi | indeed | 15:21 |
| *** claudiub has joined #openstack-infra | 15:23 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/puppet-zanata master: Fix lint issues https://review.openstack.org/472722 | 15:23 |
| openstackgerrit | Clark Boylan proposed openstack-infra/puppet-zanata master: Add cron to clean out logs https://review.openstack.org/472683 | 15:23 |
| fungi | also the connlimit-above doesn't seem to quite manage to the limit you give it, but comes close from what i can tell. so far a couple of them (just barely) exceeded 100 but only made it to 101 or 102 and are holding steady for a while | 15:23 |
| clarkb | fungi: pabelanger sorry, missed a linter error | 15:23 |
| clarkb | fungi: "its not going to the moon" - my aircraft mechanic brother (now how do you feel about flying) | 15:24 |
| fungi | as long as i don't need to fly to the moon any time soon, not so bad | 15:24 |
| openstackgerrit | Rosario Di Somma proposed openstack-infra/shade master: Convert data from raw clients to dict or list of dicts https://review.openstack.org/472690 | 15:25 |
| *** mhickey has quit IRC | 15:28 | |
| rcarrillocruz | helo helo | 15:28 |
| rcarrillocruz | question folks | 15:28 |
| rcarrillocruz | is nodepool a thing in centos? | 15:28 |
| rcarrillocruz | do we gate for nodepool on centos | 15:28 |
| rcarrillocruz | ? | 15:28 |
| rcarrillocruz | about to put nodepool for testing some stuff | 15:29 |
| clarkb | rcarrillocruz: do you mean "is nodepool tested on centos?" | 15:29 |
| rcarrillocruz | yes | 15:29 |
| fungi | clarkb: looks like it's holding up nicely! http://paste.openstack.org/show/612145/ | 15:29 |
| *** Goneri has quit IRC | 15:29 | |
| clarkb | rcarrillocruz: it isn't. We have jobs that test nodepool can build centos images though. That said the only real system interaction nodepool has is with dib and that is tested so it should just work | 15:30 |
| clarkb | rcarrillocruz: dib tests that dib can run on centos | 15:30 |
| rcarrillocruz | k, will give it a shot then, thx, cos I'm not going to need dib builds anyway | 15:30 |
| *** EricGonc_ has quit IRC | 15:31 | |
| * fungi is curious how you plan to use nodepool without dib | 15:31 | |
| clarkb | rcarrillocruz: I don't know that current nodepool is happy without the dib builds | 15:31 |
| clarkb | fungi: ya that :) | 15:31 |
| openstackgerrit | Rosario Di Somma proposed openstack-infra/shade master: Convert data from raw clients to Munch objects https://review.openstack.org/472690 | 15:31 |
| *** EricGonczer_ has joined #openstack-infra | 15:31 | |
| fungi | i suppose you could fake out teh dib calls and have them just return the pre-baked image you plan to use or something? | 15:31 |
| clarkb | fungi: cool so its holding at 1 or 2 above max, not too bad | 15:32 |
| fungi | clarkb: yeah, i need to step away for some errands shortly but will look into puppeting the rule addition when i get back | 15:32 |
| rcarrillocruz | i'm thinking of using it to test out network gear (IOS and the likes) | 15:32 |
| *** rderose has left #openstack-infra | 15:32 | |
| clarkb | fungi: rcarrillocruz you could write records into zookeeper that point at preuploaded images. It would just be aa manual step to do that I think | 15:32 |
| rcarrillocruz | ah wait | 15:32 |
| clarkb | fungi: I too have to step away to get kids ready for the doctor will be back later today to debug afs more | 15:32 |
| fungi | oh, yeah that's not a terrible idea | 15:32 |
| rcarrillocruz | was talking about nodepool 2.x something | 15:32 |
| clarkb | rcarrillocruz: yes that uses zookeeper | 15:33 |
| rcarrillocruz | that's non zookeeper, right? | 15:33 |
| rcarrillocruz | OH | 15:33 |
| rcarrillocruz | i thought that was only on feature/zuulv3 | 15:33 |
| clarkb | you need 0.ancient to not have zookeeper | 15:33 |
| rcarrillocruz | i was confused then | 15:33 |
| rcarrillocruz | thx! | 15:33 |
| pabelanger | rcarrillocruz: I make sure ansible-role-nodepool works on centos-7, that is gating | 15:33 |
| clarkb | nope the nodepool zookeepering is independent of the zuul zookeepering today. We run zuul 2.5 without zookeeper with nodepool whatever.latest with zookeeper | 15:33 |
| rcarrillocruz | gotcha | 15:34 |
| rcarrillocruz | pabelanger: neat! cos i had your role repo just in front of me right now :-) | 15:34 |
| fungi | clarkb: the connlimit is still holding steady there. total ssh api connections to gerrit are hovering between 620-630 steady for a while now | 15:36 |
| clarkb | fungi: we should investogate a total limit too to protect gerrit | 15:37 |
| pabelanger | rcarrillocruz: I'm trying to push rdo to package zookeeper but for now I am just depending on fedora testing | 15:38 |
| rcarrillocruz | erm | 15:38 |
| rcarrillocruz | that's be awesome | 15:38 |
| rcarrillocruz | cos as the chat we had the other day | 15:38 |
| rcarrillocruz | the only thing i could find was the cloudera package | 15:38 |
| rcarrillocruz | :/ | 15:38 |
| clarkb | it runs out of a tarball just fine | 15:38 |
| clarkb | that is how I install locally for testing | 15:38 |
| fungi | clarkb: agreed, those four abusers are presently making up ~2/3 of the total connection count to the api | 15:38 |
| *** jamesmcarthur has joined #openstack-infra | 15:39 | |
| fungi | so we likely still need to couple this with some manual blocking | 15:39 |
| pabelanger | ya, I need to update an ansible role I have for zookeepr to support the tarball option | 15:39 |
| pabelanger | might work on the flight to London | 15:39 |
| clarkb | fungi: ++ | 15:39 |
| *** mhickey has joined #openstack-infra | 15:41 | |
| fungi | i've manually re-blocked the four offenders using drop rules at the top of the input chain and am closing all their active connections to gerrit now | 15:41 |
| *** jcoufal_ has joined #openstack-infra | 15:43 | |
| *** jamesmcarthur has quit IRC | 15:43 | |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack-infra/project-config master: Make multinode gates regular https://review.openstack.org/471428 | 15:43 |
| *** mhickey has quit IRC | 15:43 | |
| *** jcoufal has quit IRC | 15:45 | |
| SpamapS | pabelanger fungi indeed, there's a group inside IBM/SoftLayer trying to stand up a Zuul. They're not very experienced, and so they may do silly things. Apologies if it broke something. | 15:49 |
| * SpamapS is not coordinating with them | 15:50 | |
| SpamapS | fungi: I'll pass that IP along to them | 15:50 |
| fungi | SpamapS: just amusing that even after you brought it to their attention earlier in the week they kept connecting (even from the old addresses we had blocked) | 15:50 |
| fungi | we've now, similar to your suggestion, added a connection limit via iptables | 15:50 |
| jeblair | SpamapS: if you could pass along the suggestion that they're welcome to stop by this channel and we can help 'em out, that would be mighty helpful :) | 15:51 |
| fungi | hopefully the next one, when it crops up, won't be quite so impactful | 15:51 |
| *** annegentle has joined #openstack-infra | 15:51 | |
| openstackgerrit | Danny Abukalam proposed openstack-infra/system-config master: Adding #openstack-uk to statusbot https://review.openstack.org/472741 | 15:51 |
| SpamapS | I think the problem is they're experimenting with something they don't realy understand and not keeping it well contained. | 15:52 |
| *** Apoorva has joined #openstack-infra | 15:52 | |
| SpamapS | jeblair: I will remind them that this is a public channel and they can and should introduce themselves if they're going to Zuul. :) | 15:52 |
| *** gyee has joined #openstack-infra | 15:53 | |
| fungi | it would actually help us if they got in touch, because we _think_ they're hitting a bug clarkb patched in the most current zuul release from a couple months ago | 15:53 |
| *** jpich has quit IRC | 15:53 | |
| SpamapS | Invitation extended. | 15:54 |
| fungi | so if they're running zuul new enough to have the "Close paramiko connections explicitly" commit (2.5.2 release or maybe master branch tip) then maybe there's another bug we don't know about | 15:54 |
| *** jamesmcarthur has joined #openstack-infra | 15:54 | |
| jeblair | it's a demigod and minion of Gozer, the Destructor. what could possibly go wrong if it escapes containment! | 15:55 |
| *** salv-orlando has joined #openstack-infra | 15:55 | |
| SpamapS | They're using puppet-openstackci | 15:55 |
| *** jascott1 has joined #openstack-infra | 15:55 | |
| SpamapS | but who knows when they pulled | 15:55 |
| *** gardlt has quit IRC | 15:55 | |
| fungi | SpamapS: puppet-openstackci uses zuul master branch tip unless overridden in configuration | 15:56 |
| *** gardlt has joined #openstack-infra | 15:56 | |
| fungi | which makes me suspect there could be another bug lurking | 15:56 |
| SpamapS | fungi: when does it git pull? Every puppet run? | 15:56 |
| *** felipemonteiro__ has quit IRC | 15:56 | |
| *** dougwig has joined #openstack-infra | 15:58 | |
| *** jcoufal has joined #openstack-infra | 15:58 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: WIP: Add reporter for Federated Message Bus (fedmsg) https://review.openstack.org/426861 | 15:58 |
| *** gardlt_ has joined #openstack-infra | 15:58 | |
| pabelanger | jeblair: I am ready to start up zuuv3.o.o and ze01.o.o again, if you are | 15:59 |
| fungi | SpamapS: i believe so... http://git.openstack.org/cgit/openstack-infra/puppet-zuul/tree/manifests/init.pp#n156 | 15:59 |
| fungi | http://git.openstack.org/cgit/openstack-infra/puppet-openstackci/tree/manifests/zuul_scheduler.pp#n55 | 16:00 |
| *** gardlt__ has joined #openstack-infra | 16:00 | |
| fungi | but possible they pass some explicit $revision into openstackci::zuul_scheduler | 16:00 |
| *** jcoufal_ has quit IRC | 16:00 | |
| *** rkukura has joined #openstack-infra | 16:01 | |
| *** jpena is now known as jpena|away | 16:01 | |
| *** gardlt has quit IRC | 16:01 | |
| *** r-daneel has joined #openstack-infra | 16:01 | |
| *** jbadiapa has quit IRC | 16:01 | |
| *** pblaho has quit IRC | 16:02 | |
| *** amoralej is now known as amoralej|brb | 16:02 | |
| *** gardlt__ has quit IRC | 16:02 | |
| *** gardlt has joined #openstack-infra | 16:02 | |
| SpamapS | fungi: meet dougwig ... dougwig.. meet fungi. | 16:02 |
| pabelanger | I believe you need to kick off puppet manually however | 16:02 |
| *** jascott1 has quit IRC | 16:02 | |
| pabelanger | https://docs.openstack.org/infra/openstackci/third_party_ci.html#run-masterless-puppet | 16:02 |
| fungi | oh, i've spoken to dougwig many times in the past | 16:02 |
| dougwig | fungi: SpamapS: hiya, and we know each other. :) | 16:02 |
| *** lucasagomes is now known as lucas-afk | 16:02 | |
| SpamapS | dougwig: fungi is the tech lead for the infra team and can help direct you to any resources to make your puppet-openstackci and Zuul experience a positive one. | 16:02 |
| pabelanger | https://docs.openstack.org/infra/openstackci/third_party_ci.html#updating-your-masterless-puppet-hosts too | 16:02 |
| *** gardlt_ has quit IRC | 16:03 | |
| fungi | so if using openstackci::single_node_ci it still defaults to zuul master per http://git.openstack.org/cgit/openstack-infra/puppet-openstackci/tree/manifests/single_node_ci.pp#n187 | 16:03 |
| jeblair | pabelanger: yep! let's do it! | 16:05 |
| fungi | dougwig: when you get a chance, can you check what installed version `pip list|grep zuul` reports? | 16:05 |
| clarkb | pbr freeze will give you the short sha1 of the version of zuul installed which may also be helpful. Also version of paramiko | 16:05 |
| clarkb | since I think older paramiko doesn't work with the fix? | 16:05 |
| fungi | dougwig: if it's really the tip of master, it'll hopefully be 2.5.3.dev1 | 16:05 |
| dougwig | fungi: I'm still trying to track down who was running that; I definitely will when I find it. | 16:05 |
| fungi | dougwig: awesome. i can give you ip addresses we're seeing if that helps, but i suspect there's a nat or something | 16:06 |
| jeblair | pabelanger: i'll try starting z-e | 16:06 |
| *** tosky has quit IRC | 16:06 | |
| dougwig | it's a float from the ocata cloud that I administer. trying to see who was using it; it's gone already | 16:06 |
| clarkb | ok I've got to pop out for real now. back in a few hours | 16:06 |
| pabelanger | jeblair: Oh, I just did | 16:06 |
| pabelanger | jeblair: I see hostkey failures still, let me make sure we added that properly | 16:07 |
| jeblair | pabelanger: cool, the lockfile did its thing, there's only one running :) | 16:07 |
| jeblair | pabelanger: yep | 16:07 |
| clarkb | pabelanger: fungi btw https://review.openstack.org/#/c/472722/ and its child pass tests now | 16:07 |
| pabelanger | known_hosts is setup | 16:07 |
| fungi | dougwig: if "already" was just in the last few minutes maybe, but we were still seeing connections coming from there as of 25 minutes ago (some addresses were ones we saw trying to connect on wednesday as well) | 16:07 |
| dougwig | fungi: aye, I dumped the entire float list when I logged in here, and 163 had been deleted. | 16:09 |
| openstackgerrit | Danny Abukalam proposed openstack-infra/system-config master: Adding #openstack-uk to statusbot https://review.openstack.org/472741 | 16:09 |
| dougwig | fungi: if you see it now, I've got a rogue on my network. | 16:10 |
| fungi | oh, i stand corrected, the ones from sl-reverse.com earlier this week weren't reconnecting, it was the other ones from sparqnet.net, hinet.net and one from "Ji Tong Communications Co.,Ltd" with no reverse entry | 16:10 |
| *** jcoufal_ has joined #openstack-infra | 16:10 | |
| pabelanger | jeblair: oh, looks like we might have old data for ssh host keys on review.o.o | 16:10 |
| fungi | dougwig: yep, i concur, seems to have stopped per tcpdump | 16:11 |
| *** annegentle has quit IRC | 16:11 | |
| jeblair | hrm, how do i manually compare the entry in known_hosts to the fingerprint ssh gives? | 16:12 |
| jeblair | ssh-keygen -l -f ~/.ssh/known_hosts | 16:12 |
| jeblair | pabelanger: i agree; ^ that does not match | 16:12 |
| *** jcoufal has quit IRC | 16:13 | |
| pabelanger | I think we are using old IP address and public key | 16:13 |
| pabelanger | which is odd, because I too that from zm01.o.o puppet info | 16:13 |
| fungi | pabelanger: are you using the gerrit ssh api host key from public hiera? | 16:13 |
| *** jcoufal__ has joined #openstack-infra | 16:13 | |
| pabelanger | fungi: It was hiera, but private | 16:14 |
| pabelanger | let me confirm again | 16:14 |
| *** jascott1 has joined #openstack-infra | 16:14 | |
| jeblair | that's weird, why would the host key be private? | 16:14 |
| jeblair | pabelanger: gerrit-code-review@review-dev.openstack.org | 16:15 |
| pabelanger | not sure, we likely could move it public | 16:15 |
| jeblair | pabelanger: right ip address, wrong host key | 16:15 |
| pabelanger | Ah | 16:15 |
| *** jcoufal_ has quit IRC | 16:15 | |
| dabukalam | Hi, we've started a new channel #openstack-uk for the UK OpenStack community. Wiki says I should ask in here to ensure it gets registered properly. I've started that process by adding openstackinfra to list of users and adding the channel to the list of channels here - https://review.openstack.org/#/c/472741/. Have I done this right? | 16:16 |
| pabelanger | jeblair: do we want to move it public now? | 16:16 |
| fungi | i could have sworn we put gerrit_ssh_rsa_pubkey_contents into public hiera. did my change for that never merge? | 16:16 |
| fungi | oh, i had it wip for some reason https://review.openstack.org/350709 | 16:17 |
| fungi | was depending on https://review.openstack.org/350663 | 16:17 |
| fungi | aha, failed testing and i didn't have time to sort out why, then didn't get around to revisiting it :/ | 16:18 |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Add gerrit_ssh_rsa_pubkey_contents to zuul-executor hiera https://review.openstack.org/472748 | 16:19 |
| pabelanger | we could do ^ or update private hieradata | 16:19 |
| fungi | pabelanger: mind looking at the implementation i already had? | 16:19 |
| fungi | or should i just abandon? | 16:19 |
| pabelanger | fungi: ya, just looking now | 16:19 |
| *** katkapilatova has left #openstack-infra | 16:20 | |
| fungi | i guess my idea was more grand. make one known_hosts file and we can keep it updated everywhere rather than needing to work out which specific known_hosts entries each system needs | 16:20 |
| pabelanger | fungi: we could go that route too. I don't really have a preference | 16:21 |
| pabelanger | fungi: if we went that route, we can use your hiera configuration, and update ze01.o.o today to use it. Then migrate other servers as we have time | 16:22 |
| *** tmorin has quit IRC | 16:22 | |
| fungi | i saw it as a sort of stop-gap until we can rely on dnssec with sshfp records for a global known host keys solution | 16:22 |
| *** sbezverk has joined #openstack-infra | 16:23 | |
| fungi | follow-on plan was to start just installing it at /etc/ssh/ssh_known_hosts on all our systems | 16:23 |
| *** annegentle has joined #openstack-infra | 16:23 | |
| fungi | rather than trying to hack maintaining user-specific ~/.ssh/known_hosts files | 16:23 |
| fungi | anyway, i need to go run some errands but should return in a couple hours | 16:25 |
| pabelanger | okay, I've also changed private hieradata for now too. To keep us moving forward, until we come up with a plan to sync known_hosts everwhere | 16:25 |
| fungi | pabelanger: sounds good | 16:25 |
| jeblair | pabelanger: what did you change in private hieradata? | 16:25 |
| pabelanger | jeblair: gerrit_ssh_rsa_pubkey_contents for zuul-executor.yaml | 16:26 |
| pabelanger | to match contents from zuul-merger.yaml | 16:26 |
| jeblair | gotcha | 16:27 |
| *** annegentle has quit IRC | 16:29 | |
| *** annegentle has joined #openstack-infra | 16:30 | |
| *** ltomasbo is now known as ltomasbo|away | 16:32 | |
| *** tesseract has quit IRC | 16:32 | |
| *** pcaruana has quit IRC | 16:34 | |
| *** esberglu has quit IRC | 16:39 | |
| *** esberglu has joined #openstack-infra | 16:39 | |
| *** esberglu has quit IRC | 16:43 | |
| *** rcernin has joined #openstack-infra | 16:45 | |
| *** ericyoung has quit IRC | 16:46 | |
| *** Guest22695 has quit IRC | 16:50 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci master: ovb playbook: add task to delete overcloud https://review.openstack.org/463740 | 16:52 |
| jeblair | RSA host key for IP address '2001:4800:7819:103:be76:4eff:fe05:8525' not in list of known hosts. | 16:53 |
| jeblair | Permission denied (publickey). | 16:53 |
| jeblair | pabelanger: ^ | 16:53 |
| jeblair | pabelanger: which is strange, because "ssh -i /var/lib/zuul/ssh/id_rsa zuul@review.openstack.org -p29418" seems to work | 16:53 |
| pabelanger | I think it is because we have the wrong IPv6 address | 16:54 |
| pabelanger | 2001:4800:7815:104:3bc3:d7f6:ff03:bf5d is what I added | 16:54 |
| jeblair | i agree, 2001:4800:7819:103:be76:4eff:fe05:8525 is the right one. | 16:54 |
| pabelanger | zuul_merger.pp must be outdated | 16:55 |
| *** Guest22695 has joined #openstack-infra | 16:55 | |
| pabelanger | will fix ze01.o.o first | 16:55 |
| pabelanger | 23.253.232.87 seems incorrect too | 16:56 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci master: Ping test horizon https://review.openstack.org/472758 | 16:56 |
| *** Guest22695 has quit IRC | 16:56 | |
| pabelanger | jeblair: 104.130.246.91 looks to be the ipv4 address also | 16:57 |
| pabelanger | should we change that too? | 16:57 |
| jeblair | pabelanger: yep. i'm not sure why we don't see warnings on the mergers. | 16:57 |
| jeblair | pabelanger: it may only log that warning on error | 16:58 |
| jeblair | pabelanger: the ip address being wrong is causing a warning, but ssh continues | 16:58 |
| jeblair | "Permission denied (publickey)" is the actual error | 16:58 |
| *** rkukura has quit IRC | 16:58 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Update ze01.o.o known_host file with proper IP addresses https://review.openstack.org/472759 | 16:59 |
| *** derekh has quit IRC | 16:59 | |
| pabelanger | jeblair: Hmm, agree publickey should not be failing | 16:59 |
| *** yamahata has joined #openstack-infra | 16:59 | |
| pabelanger | let me confirm zuul.conf section | 16:59 |
| openstackgerrit | Bao proposed openstack-infra/jenkins-job-builder master: Add `do-not-fetch-tags` to CloneOption for Git https://review.openstack.org/472761 | 17:00 |
| jkilpatr | pabelanger, any idea why the test vm's are missing openssl-devel? http://logs.openstack.org/48/470748/5/check/gate-browbeat-pep8-ubuntu-xenial/9596318/console.html | 17:02 |
| pabelanger | jkilpatr: the dependency was recently removed when we stopped installing npm / nodejs. You'll need to add them into your bindep.txt file now. There is also a post on ML about it | 17:03 |
| jkilpatr | bindep that's what I was trying to remember. | 17:04 |
| jkilpatr | thanks! | 17:04 |
| pabelanger | jeblair: it is possible we use ssh config to setup the location to our private key? | 17:06 |
| pabelanger | actually, I think we use environmental variable for it now | 17:09 |
| jeblair | pabelanger: i don't follow | 17:09 |
| pabelanger | jeblair: I am wondering if we are not passing our private key properly to ssh to auth with | 17:09 |
| jeblair | pabelanger: yeah, there may be a problem with the new ssh context manager stuff | 17:09 |
| jeblair | i'm setting up a test case | 17:09 |
| pabelanger | ack | 17:09 |
| *** Apoorva_ has joined #openstack-infra | 17:14 | |
| openstackgerrit | Merged openstack-infra/system-config master: Update ze01.o.o known_host file with proper IP addresses https://review.openstack.org/472759 | 17:14 |
| *** pblaho has joined #openstack-infra | 17:14 | |
| *** dizquierdo has quit IRC | 17:14 | |
| *** Apoorva has quit IRC | 17:18 | |
| *** esberglu has joined #openstack-infra | 17:19 | |
| *** amoralej|brb is now known as amoralej | 17:20 | |
| *** felipemonteiro has joined #openstack-infra | 17:20 | |
| *** esberglu has quit IRC | 17:21 | |
| *** armax has quit IRC | 17:21 | |
| *** esberglu has joined #openstack-infra | 17:21 | |
| *** jpena|away is now known as jpena | 17:22 | |
| *** jpena is now known as jpena|off | 17:23 | |
| *** jamesmcarthur has quit IRC | 17:25 | |
| *** jamesmcarthur has joined #openstack-infra | 17:26 | |
| *** dhinesh has joined #openstack-infra | 17:30 | |
| *** trown is now known as trown|lunch | 17:34 | |
| *** rbrndt has quit IRC | 17:34 | |
| *** jamesmcarthur has quit IRC | 17:37 | |
| *** mat128 has quit IRC | 17:39 | |
| *** Goneri has joined #openstack-infra | 17:39 | |
| openstackgerrit | Bao proposed openstack-infra/jenkins-job-builder master: Add `do-not-fetch-tags` to CloneOption for Git https://review.openstack.org/472761 | 17:40 |
| *** mat128 has joined #openstack-infra | 17:41 | |
| annegentle | if you have a moment, could I get eyes on https://review.openstack.org/#/c/461280/ -- it's a change to reviewstats and it looks like the -infra core team are reviewers on that repo. | 17:41 |
| *** ericyoung has joined #openstack-infra | 17:41 | |
| *** mat128 has quit IRC | 17:41 | |
| *** rkukura has joined #openstack-infra | 17:42 | |
| *** prad_ has joined #openstack-infra | 17:43 | |
| *** makowals has joined #openstack-infra | 17:45 | |
| *** makowals has quit IRC | 17:49 | |
| *** prateek has joined #openstack-infra | 17:50 | |
| *** liusheng has quit IRC | 17:50 | |
| *** salv-orlando has quit IRC | 17:50 | |
| *** salv-orlando has joined #openstack-infra | 17:51 | |
| *** liusheng has joined #openstack-infra | 17:51 | |
| *** prateek_ has joined #openstack-infra | 17:54 | |
| *** salv-orlando has quit IRC | 17:55 | |
| *** sbezverk has quit IRC | 17:56 | |
| *** sbezverk has joined #openstack-infra | 17:57 | |
| *** prateek has quit IRC | 17:57 | |
| *** vhosakot has quit IRC | 18:04 | |
| *** e0ne has joined #openstack-infra | 18:04 | |
| *** yland has joined #openstack-infra | 18:06 | |
| *** yland is now known as Guest55763 | 18:06 | |
| *** iogg has quit IRC | 18:07 | |
| inc0 | hey guys, can we manually trigger merge publisher job? | 18:07 |
| *** vhosakot has joined #openstack-infra | 18:13 | |
| *** electrofelix has quit IRC | 18:13 | |
| inc0 | I need to run this job https://github.com/openstack-infra/project-config/blob/master/jenkins/jobs/kolla.yaml#L28 | 18:15 |
| pabelanger | you can put the job in the experimental pipeline, then run 'check experimental' on your patchset | 18:16 |
| inc0 | pabelanger: I need to do it once | 18:17 |
| inc0 | we released, tag changed, tarballs images are obsolete | 18:17 |
| inc0 | I need to refresh them, our deploy gates are down now | 18:17 |
| *** trown|lunch is now known as trown | 18:18 | |
| pabelanger | not sure I am following | 18:20 |
| inc0 | our images are tagged with pbr-generated version | 18:20 |
| inc0 | so when you build image, it will get tag 5.0.0 today | 18:20 |
| inc0 | it was 4.0.2 yesterday | 18:20 |
| inc0 | and ansible will expect 5.0.0 | 18:20 |
| inc0 | so we need to refresh tarballs.o.o | 18:21 |
| pabelanger | how does it work today, a post job IIRC? | 18:22 |
| openstackgerrit | Bao proposed openstack-infra/jenkins-job-builder master: Add `do-not-fetch-tags` to CloneOption for Git https://review.openstack.org/472761 | 18:22 |
| *** sbezverk has quit IRC | 18:22 | |
| inc0 | I think so, https://github.com/openstack-infra/project-config/blob/master/jenkins/jobs/kolla.yaml#L51 | 18:22 |
| pabelanger | http://logs.openstack.org/14/142c3b839ef9188197b5b5c968266c1ae7dce22d/post/ should have been the last job to run | 18:25 |
| pabelanger | did something there change or break? | 18:25 |
| *** rbrndt has joined #openstack-infra | 18:26 | |
| *** vhosakot has quit IRC | 18:26 | |
| inc0 | pabelanger: https://review.openstack.org/#/c/472671/ | 18:26 |
| *** vhosakot has joined #openstack-infra | 18:27 | |
| *** felipemonteiro has quit IRC | 18:27 | |
| pabelanger | okay, so you had a release, 5.0.0.0b2 but images are older then that? And some how you are expecting 5.0.0.0b2 right? | 18:27 |
| inc0 | yes | 18:28 |
| inc0 | exactly | 18:28 |
| pabelanger | how have you fixed this issue before? | 18:28 |
| inc0 | frankly I can't remember, it'll fix itself when first thing merges to Kolla | 18:28 |
| pabelanger | right, or some periodic job | 18:29 |
| inc0 | but kolla-ansible gates are down till that happens | 18:29 |
| inc0 | I don't think we have periodic job for that | 18:29 |
| inc0 | we will when we're done with dockerhub publisher | 18:29 |
| *** ericyoung has quit IRC | 18:29 | |
| inc0 | that being said it would be cool to have any way to trigger these jobs manually | 18:29 |
| inc0 | in case of sth like this | 18:29 |
| *** e0ne has quit IRC | 18:30 | |
| pabelanger | we tend not to do manually triggers because they depend on humans, and basically block your development processes | 18:30 |
| pabelanger | let me see if I can figure out the syntax | 18:30 |
| inc0 | yeah I understand, maybe trigger this automatically on release at some point | 18:31 |
| inc0 | that would be ideal | 18:31 |
| *** Guest55763 has quit IRC | 18:31 | |
| pabelanger | right, you can do that today using the release pipeline | 18:32 |
| pabelanger | which will kick off a build each time a release happens | 18:32 |
| *** felipemonteiro has joined #openstack-infra | 18:32 | |
| inc0 | worst case scenerio I'll just make stub change and workflow it.. | 18:32 |
| inc0 | good to know | 18:33 |
| *** ralonsoh has quit IRC | 18:33 | |
| pabelanger | inc0: that is the fastest right now honestly. I need to construct the syntax for trigger | 18:33 |
| pabelanger | and how you've likely done it to fix before | 18:33 |
| *** annegentle has quit IRC | 18:34 | |
| *** felipemonteiro__ has joined #openstack-infra | 18:35 | |
| pabelanger | inc0: look to manila-image-elements jobs in zuul/layout.yaml. You can do the same for kolla | 18:35 |
| inc0 | thanks pabelanger I'll look at tit | 18:37 |
| *** annegentle has joined #openstack-infra | 18:37 | |
| inc0 | which also made a funny typo | 18:37 |
| fungi | yeah, _if_ there's a release (or pre-release) pipeline job, we can reenqueue the relevant tag if its jobs need to be rerun because something broke | 18:38 |
| *** felipemonteiro has quit IRC | 18:38 | |
| inc0 | fungi: good to know, I'll include it in our publisher jobs | 18:38 |
| *** rkukura has quit IRC | 18:39 | |
| *** amoralej is now known as amoralej|off | 18:40 | |
| fungi | inc0: these are the things which ran in the pre-release pipeline when that tag was pushed: http://logs.openstack.org/7c/7c040a11dd3eabd93940db44f358a04bd711145c/pre-release/ | 18:42 |
| *** yolanda has quit IRC | 18:43 | |
| fungi | (note that the pre-release pipeline matches pre-release (alpha, beta, release candidate) tags, and release matches release versioned tags (as their names would imply) | 18:45 |
| *** yolanda has joined #openstack-infra | 18:45 | |
| fungi | so you can choose to run different jobs for pre-releases and releases if you want | 18:45 |
| *** florianf has quit IRC | 18:46 | |
| *** rkukura has joined #openstack-infra | 18:47 | |
| *** EricGonczer_ has quit IRC | 18:51 | |
| *** sdague has quit IRC | 18:51 | |
| *** EricGonczer_ has joined #openstack-infra | 18:52 | |
| *** jcoufal__ has quit IRC | 18:53 | |
| openstackgerrit | wes hayutin proposed openstack-infra/tripleo-ci master: WIP: organize the tripleo ci configurations and playbooks https://review.openstack.org/472777 | 18:54 |
| *** gardlt has quit IRC | 19:05 | |
| *** gardlt has joined #openstack-infra | 19:07 | |
| *** sdague has joined #openstack-infra | 19:09 | |
| *** gardlt has quit IRC | 19:11 | |
| *** makowals has joined #openstack-infra | 19:23 | |
| *** EricGonczer_ has quit IRC | 19:24 | |
| *** EricGonc_ has joined #openstack-infra | 19:24 | |
| *** rossella__ has joined #openstack-infra | 19:25 | |
| *** prateek_ has quit IRC | 19:28 | |
| *** makowals has quit IRC | 19:30 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul feature/zuulv3: WIP: add log streaming test https://review.openstack.org/471079 | 19:30 |
| *** amoralej|off is now known as amoralej | 19:36 | |
| *** rkukura has quit IRC | 19:41 | |
| *** EricGonc_ has quit IRC | 19:42 | |
| clarkb | I have taken the pypi update lock on mirror-update.o.o in order to test additioanl afs things | 19:43 |
| clarkb | basically preventing any new vos releases from happening so I can control that and test the effects | 19:43 |
| *** EricGonczer_ has joined #openstack-infra | 19:44 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: WIP: add log streaming test https://review.openstack.org/471079 | 19:46 |
| *** rossella__ has quit IRC | 19:47 | |
| *** gardlt has joined #openstack-infra | 19:48 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: WIP: add log streaming test https://review.openstack.org/471079 | 19:48 |
| *** prad_ has quit IRC | 19:52 | |
| *** spzala has quit IRC | 19:54 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Fix git ssh env in merger https://review.openstack.org/472788 | 19:55 |
| clarkb | jeblair: a noop vos release is sufficient to create the slow pip install behavior in kna1 | 19:56 |
| clarkb | so I think what is happening is afs is saying "I got updates" even if it didn't. Which then invalidates the cache metadata forcing subsequent reads to go out and update at least that information but not necessary the complete files | 19:57 |
| jeblair | clarkb: ok, i think that pretty strongly points to file version checks after vos release causing that slowdown. | 19:57 |
| *** salv-orlando has joined #openstack-infra | 19:57 | |
| clarkb | I like the idea of only doing a vos release if we updated a global requirements req. then maybe update otherwise once every 4 hours or something? | 19:58 |
| jeblair | clarkb: that's the best mitigation i can come up with atm. | 19:58 |
| clarkb | if ^ sounds like a reasonable plan to infra-root I can start working on the log parsing to do that | 19:58 |
| clarkb | and also do a quick audit of our update cycle in the other repos | 19:59 |
| jeblair | clarkb: i think the docs vos release script may have some code to query for the most recent release of a volume, that may also come in handy here. | 19:59 |
| jeblair | clarkb: oh, sorry, it's for the most recent update, but probably prett adaptable. | 19:59 |
| clarkb | jeblair: ya since I will need to fall back on doing a vos release in X time period even if global reqs hasn't updated | 20:00 |
| *** e0ne has joined #openstack-infra | 20:01 | |
| *** annegentle has quit IRC | 20:01 | |
| openstackgerrit | Pete Birley proposed openstack-infra/project-config master: New project - OpenStack-Spaceport https://review.openstack.org/472791 | 20:03 |
| *** rossella__ has joined #openstack-infra | 20:03 | |
| *** ldnunes has quit IRC | 20:03 | |
| *** yland has joined #openstack-infra | 20:04 | |
| *** rossella__ has quit IRC | 20:05 | |
| *** panda|bbl is now known as panda | 20:07 | |
| fungi | clarkb: no objection on my end. maybe upper-constraints.txt instead of global-requirements.txt (on all branches of openstack/requirements for extra credit)? | 20:08 |
| clarkb | oh right that will get transitive deps | 20:09 |
| clarkb | oh we have to check all branches ugh | 20:09 |
| fungi | i said "extra credit" after all | 20:11 |
| clarkb | ya I'm just realizing this is more complicated than it first looked :) but I should be able to get something out today | 20:11 |
| fungi | if stable branch changes don't get new deps straight away, maybe not the end of the world | 20:11 |
| *** Apoorva_ has quit IRC | 20:12 | |
| *** Apoorva has joined #openstack-infra | 20:13 | |
| *** JudeC has joined #openstack-infra | 20:13 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul feature/zuulv3: WIP: add log streaming test https://review.openstack.org/471079 | 20:15 |
| *** ldnunes has joined #openstack-infra | 20:16 | |
| *** thorst_afk has quit IRC | 20:18 | |
| *** thorst_afk has joined #openstack-infra | 20:20 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Fix git ssh env in merger https://review.openstack.org/472788 | 20:21 |
| *** jamesdenton has joined #openstack-infra | 20:21 | |
| *** thorst_afk has quit IRC | 20:25 | |
| dirk | this should be ready for +A : https://review.openstack.org/#/c/471071/ | 20:25 |
| openstackgerrit | Pete Birley proposed openstack-infra/project-config master: New project - OpenStack-Spaceport https://review.openstack.org/472791 | 20:26 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/project-config master: Start using split out plugin for all keystone jobs https://review.openstack.org/472797 | 20:28 |
| *** spzala has joined #openstack-infra | 20:28 | |
| *** annegentle has joined #openstack-infra | 20:30 | |
| pabelanger | dirk: how are the job runtimes without wheels? | 20:31 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul feature/zuulv3: WIP: add log streaming test https://review.openstack.org/471079 | 20:32 |
| dirk | pabelanger: 5% faster than ubuntu: https://review.openstack.org/#/c/471489/ | 20:32 |
| dirk | pabelanger: (comparing dsvm-neutron-full times) | 20:33 |
| *** e0ne has quit IRC | 20:33 | |
| openstackgerrit | wes hayutin proposed openstack-infra/tripleo-ci master: WIP: organize the tripleo ci configurations and playbooks https://review.openstack.org/472777 | 20:34 |
| pabelanger | dirk: +2, will let fungi +3 since ptl :) | 20:36 |
| *** thorst_afk has joined #openstack-infra | 20:39 | |
| amoralej | pabelanger, you can unhold the server | 20:41 |
| amoralej | we already found the issue, it was bad dns + new eventlet | 20:41 |
| pabelanger | amoralej: thanks | 20:41 |
| amoralej | thanks for your help | 20:41 |
| pabelanger | amoralej: what was the dns issue? | 20:42 |
| amoralej | i know it's not a good thing, but i can tell you we were desperate | 20:42 |
| amoralej | well a mix of dns not working and new eventlet not managing it properly | 20:42 |
| *** jamesdenton has quit IRC | 20:42 | |
| *** thorst_afk has quit IRC | 20:43 | |
| pabelanger | amoralej: ya, you should consider using dns server from nodepool image we create, it is running unbound already and setup to forward requests properly. You'd just need to make a few changes I think | 20:43 |
| *** sdague has quit IRC | 20:43 | |
| amoralej | I'll talk to EmilienM about that next week | 20:44 |
| *** salv-orl_ has joined #openstack-infra | 20:44 | |
| *** salv-orlando has quit IRC | 20:46 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul feature/zuulv3: Fix zuul_streamer send() call for py35 https://review.openstack.org/472816 | 20:48 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/project-config master: Start using split out plugin for all keystone jobs https://review.openstack.org/472797 | 20:49 |
| openstackgerrit | wes hayutin proposed openstack-infra/tripleo-ci master: WIP: organize the tripleo ci configurations and playbooks https://review.openstack.org/472777 | 20:49 |
| inc0 | pabelanger: so this just merged but images are still old https://review.openstack.org/#/c/466738/ | 20:50 |
| inc0 | does it take time for tarballs to update? | 20:50 |
| inc0 | (by time I mean <1hr) | 20:50 |
| inc0 | >1hr | 20:50 |
| inc0 | Friday.. | 20:50 |
| pabelanger | inc0: all depends on how long your post jobs take to run | 20:51 |
| openstackgerrit | wes hayutin proposed openstack-infra/tripleo-ci master: WIP: organize the tripleo ci configurations and playbooks https://review.openstack.org/472777 | 20:51 |
| pabelanger | I see some still running on http://status.openstack.org/zuul | 20:51 |
| inc0 | ahh right, I see it now, sorry | 20:51 |
| *** jbadiapa has joined #openstack-infra | 20:52 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul feature/zuulv3: WIP: add log streaming test https://review.openstack.org/471079 | 20:54 |
| openstackgerrit | Merged openstack-infra/project-config master: Upgrade experimental tempest-dsvm-neutron-full-opensuse-422-nv to non-voting https://review.openstack.org/471071 | 20:55 |
| fungi | dirk: ^ | 20:56 |
| openstackgerrit | Brianna Poulos proposed openstack-infra/project-config master: Change Barbican Tempest Plugin Gate to Voting https://review.openstack.org/472821 | 20:56 |
| *** bobh has quit IRC | 20:56 | |
| *** jamesmcarthur has joined #openstack-infra | 20:57 | |
| *** annegentle has quit IRC | 20:58 | |
| *** annegentle has joined #openstack-infra | 20:58 | |
| jeblair | fungi, clarkb, mordred, pabelanger, SpamapS: what git repo/pypi package name should i use for a sphinx plugin for zuul job documentation? "zuul-sphinx"? | 20:59 |
| *** Apoorva_ has joined #openstack-infra | 20:59 | |
| *** annegentle has quit IRC | 20:59 | |
| fungi | sphinx-extension-zuul? not sure what the convention is | 21:00 |
| pabelanger | sphinxcontrib-foobar seems to be a thing | 21:01 |
| pabelanger | as does sphinx-foobar | 21:01 |
| jeblair | there's 'oslosphinx' and 'sphinxcontrib-programoutput' | 21:01 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Fix git ssh env in merger https://review.openstack.org/472788 | 21:01 |
| *** Goneri has quit IRC | 21:01 | |
| jeblair | sphinxcontrib.foo seems like a problematic pattern though (packages sharing a namespace?) | 21:02 |
| *** jamesmcarthur has quit IRC | 21:02 | |
| jeblair | https://sphinxext-survey.readthedocs.io/en/latest/domains.html | 21:03 |
| pabelanger | ya, zuul-sphinx is fine by me. Will defer to group | 21:03 |
| *** Apoorva has quit IRC | 21:03 | |
| jeblair | https://bitbucket.org/birkenfeld/sphinx-contrib/ | 21:03 |
| *** gordc has quit IRC | 21:03 | |
| mordred | jeblair: zuul-sphinx works for me - as does sphinx-extension-zuul | 21:04 |
| *** gardlt has quit IRC | 21:06 | |
| *** trown is now known as trown|outtypewww | 21:06 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Fix zuul_streamer send() call for py35 https://review.openstack.org/472816 | 21:06 |
| *** gardlt has joined #openstack-infra | 21:07 | |
| fungi | oh, right, sphinxcontrib-zuul could be a thing | 21:07 |
| fungi | so many color opportunities for this bikeshed | 21:07 |
| *** gordc has joined #openstack-infra | 21:08 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul feature/zuulv3: WIP: add log streaming test https://review.openstack.org/471079 | 21:10 |
| *** gardlt has quit IRC | 21:11 | |
| clarkb | my terminal can support 256 | 21:12 |
| *** pblaho has quit IRC | 21:12 | |
| openstackgerrit | Pete Birley proposed openstack-infra/project-config master: New project - OpenStack-Spaceport https://review.openstack.org/472791 | 21:12 |
| *** shardy has quit IRC | 21:13 | |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config master: Add zuul-sphinx project https://review.openstack.org/472823 | 21:14 |
| *** ldnunes has quit IRC | 21:15 | |
| jeblair | some can support 16 million | 21:16 |
| *** makowals has joined #openstack-infra | 21:16 | |
| *** armax has joined #openstack-infra | 21:16 | |
| jeblair | clarkb, fungi: can one of you ack https://review.openstack.org/472487 ? | 21:16 |
| fungi | my terminal does 8 colors with two intensities | 21:19 |
| fungi | pabelanger: sudo failure as zuul on ze01 is you? | 21:20 |
| *** amoralej is now known as amoralej|off | 21:21 | |
| *** dimtruck is now known as zz_dimtruck | 21:25 | |
| jeblair | fungi: me | 21:27 |
| fungi | jeblair: cool--thanks! | 21:27 |
| *** jkilpatr has quit IRC | 21:28 | |
| *** panda is now known as panda|off | 21:30 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Don't wait for forever to join streamer https://review.openstack.org/472839 | 21:34 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Use display.display for executor debug messages https://review.openstack.org/472840 | 21:34 |
| *** jiaohaolin1 has joined #openstack-infra | 21:36 | |
| *** askb has joined #openstack-infra | 21:39 | |
| *** jiaohaolin has quit IRC | 21:40 | |
| *** felipemonteiro__ has quit IRC | 21:40 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul feature/zuulv3: WIP: add log streaming test https://review.openstack.org/471079 | 21:41 |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack-infra/project-config master: Multinode gates, dockerhub publisher prework https://review.openstack.org/471428 | 21:43 |
| *** EricGonczer_ has quit IRC | 21:44 | |
| *** esberglu has quit IRC | 21:44 | |
| *** JudeC has quit IRC | 21:45 | |
| jeblair | starting zuul-executor on ze01 and restarting zuul on zuulv3 | 21:46 |
| *** EricGonczer_ has joined #openstack-infra | 21:46 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config master: Run vos release only when necessary https://review.openstack.org/472842 | 21:47 |
| *** JudeC has joined #openstack-infra | 21:47 | |
| clarkb | jeblair: fungi ^ ok thats a fun bit of bash, I think I commented it well enough. I have tested the individual bits locally as I've gone along but not in aggregate. THinking maybe easiest way is to just run that script in foreground on mirror-update after I get some quick review on it? | 21:48 |
| clarkb | I already see a bug :) | 21:50 |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config master: Run vos release only when necessary https://review.openstack.org/472842 | 21:50 |
| *** makowals has quit IRC | 21:50 | |
| openstackgerrit | James E. Blair proposed openstack-infra/openstack-zuul-jobs master: Change image to label https://review.openstack.org/472845 | 21:52 |
| jeblair | i'm self-approving that ^ | 21:52 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Change image to label https://review.openstack.org/472845 | 21:52 |
| *** jamesdenton has joined #openstack-infra | 21:53 | |
| *** mdrabe has quit IRC | 21:53 | |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack-infra/project-config master: Multinode gates, dockerhub publisher prework https://review.openstack.org/471428 | 21:54 |
| dirk | fungi: woohoo, thanks! | 21:56 |
| dirk | fungi: (or others) quick question: I've seen the "integrated-gate" template in the zuul config, which defines a consistent set of voting check and gate jobs for all the core set of projects. Interestingly it also defines experimental gate jobs, but no non-voting jobs. is that by design? I was thinking about adding a non-voting opensuse check either in the experimental or even in the check queue there | 21:57 |
| *** chlong has quit IRC | 21:58 | |
| clarkb | dirk: all experimental jobs are nonvoting because that pipeline can't vote. I think it would be fine to add the suse job there. Typically we want broader consensus before adding it to check/gate for projects though | 21:58 |
| fungi | dirk: i don't know that we have much precedent for adding jobs to that, but my gut says we've mostly deferred to the qa team on that template | 21:58 |
| *** jamesmcarthur has joined #openstack-infra | 21:59 | |
| *** kphatak has joined #openstack-infra | 21:59 | |
| *** gardlt has joined #openstack-infra | 21:59 | |
| kphatak | I am referring to openstack/mors project. I am an admin for launchpad team mors-drivers for the project. I am not able to merge any code commits on gerrit. I understand that I need to be part of mors-core and mors-release group on gerrit. But neither me nor owner of mors project Roopak Parikh are able to add anything into those groups https://review.openstack.org/#/admin/groups/1556,members | 22:02 |
| clarkb | kphatak: do you have a link to the change that added the new project? as part of the project creation a gerrit admin has to seed the groups with an initial member. I'm happy to do that now if you can point me to the chagne that created the project | 22:04 |
| *** jamesmcarthur has quit IRC | 22:04 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Use display.display for executor debug messages https://review.openstack.org/472840 | 22:04 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Don't wait for forever to join streamer https://review.openstack.org/472839 | 22:04 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Use threads instead of processes in zuul_stream https://review.openstack.org/472850 | 22:04 |
| *** gardlt has quit IRC | 22:04 | |
| *** dave-mccowan has quit IRC | 22:06 | |
| *** dave-mccowan has joined #openstack-infra | 22:07 | |
| jeblair | pabelanger: ze01 cannot ssh into nodes with the key that it has | 22:07 |
| jeblair | moreover, i cannot ssh into those nodes as root | 22:07 |
| *** rfolco has quit IRC | 22:08 | |
| *** EricGonczer_ has quit IRC | 22:10 | |
| jeblair | i'll shut down zuulv3 for now | 22:11 |
| *** EricGonczer_ has joined #openstack-infra | 22:11 | |
| *** wolverineav has quit IRC | 22:14 | |
| *** wolverin_ has joined #openstack-infra | 22:14 | |
| *** esberglu has joined #openstack-infra | 22:15 | |
| *** xarses has quit IRC | 22:16 | |
| clarkb | I don't see the issue with my shell script that bashate is complaining about | 22:16 |
| clarkb | E040: Syntax error: syntax error: unexpected end of file: '' | 22:17 |
| pabelanger | jeblair: okay, I'll poke around and see why. Make sure we have the correct private key for the zuul user | 22:17 |
| *** rcernin has quit IRC | 22:19 | |
| jeblair | pabelanger: yeah, that's one question, and why is there no ~root/.ssh/authorized_keys is another | 22:19 |
| jeblair | are these the same images we're using in prod, or does zuulv3 have its own images now? | 22:19 |
| pabelanger | jeblair: I think because we are mirring infra-root key for nl01.o.o? | 22:19 |
| pabelanger | they are the same images | 22:20 |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config master: Run vos release only when necessary https://review.openstack.org/472842 | 22:20 |
| *** esberglu has quit IRC | 22:20 | |
| pabelanger | missing* | 22:20 |
| clarkb | ok I think that is it, error was like 80 lines before the specified line. Ya syntax errors | 22:20 |
| pabelanger | jeblair: ya, nodepoolv3.yaml is missing our key-name setting. So we are not booting these images with the keypair from cloud | 22:21 |
| jeblair | pabelanger: oh, right, that's *launch* time now | 22:21 |
| jeblair | makes sense | 22:21 |
| *** xyang1 has quit IRC | 22:21 | |
| mordred | clarkb: nice idea! | 22:21 |
| jeblair | pabelanger: cool. so the other one is probably just another wrong key | 22:21 |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack-infra/project-config master: Multinode gates, dockerhub publisher prework https://review.openstack.org/471428 | 22:21 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Run vos release only when necessary https://review.openstack.org/472842 | 22:22 |
| jeblair | clarkb: sorry, i had to edit the commit message | 22:22 |
| pabelanger | jeblair: looks like we placed the jenkins user SSH private key on ze01.o.o, using jenkins user works | 22:22 |
| jeblair | I have learned recently: There is no force so great as the desire to edit someone else's copy. | 22:23 |
| pabelanger | should be able to update zuul-executor private hieradata with proper keys | 22:23 |
| jeblair | pabelanger: cool | 22:23 |
| pabelanger | give me a few minutes and I can update our configuration in hiera | 22:23 |
| *** ihrachys has quit IRC | 22:25 | |
| *** rbrndt has quit IRC | 22:26 | |
| pabelanger | jeblair: keys updated, ze01.o.o kicked. Was able to login with zuul user now | 22:29 |
| jeblair | clarkb: left some suggestions on 472842; generally lgtm | 22:30 |
| jeblair | pabelanger: cool, i'll restart | 22:30 |
| pabelanger | ack | 22:30 |
| pabelanger | I'll work on nodepoolv3.yaml change too | 22:30 |
| clarkb | jeblair: thanks will update shortly | 22:31 |
| *** r-daneel has quit IRC | 22:34 | |
| *** claudiub has quit IRC | 22:34 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Switch to flavor-name for nodepoolv3.yaml https://review.openstack.org/472853 | 22:41 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Add infra-root-keys to nodepoolv3.yaml https://review.openstack.org/472854 | 22:41 |
| kphatak | clarkb: https://review.openstack.org/#/c/351956/ change which created openstack-mors project | 22:43 |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config master: Run vos release only when necessary https://review.openstack.org/472842 | 22:45 |
| clarkb | jeblair: ^ updated | 22:45 |
| clarkb | kphatak: ok I have added rparikh to the two groups created by that change as rparikh was the new project creator. They should now be able to add who they like to the gerrit groups | 22:47 |
| kphatak | cool. thanks | 22:47 |
| *** rlandy has quit IRC | 22:47 | |
| *** bobh has joined #openstack-infra | 22:49 | |
| *** bobh has quit IRC | 22:49 | |
| *** kphatak has quit IRC | 22:50 | |
| *** jamesmcarthur has joined #openstack-infra | 22:56 | |
| *** salv-orl_ has quit IRC | 22:56 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config master: Run vos release only when necessary https://review.openstack.org/472842 | 22:57 |
| *** salv-orlando has joined #openstack-infra | 22:57 | |
| clarkb | I'm going to release the bandersnatch lock now. I don't think I will get around to testing ^ before weekending but will regrab lock early next week to test it and hopefully get it running | 22:58 |
| *** dougshelley66 has joined #openstack-infra | 22:59 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/openstack-zuul-roles master: Remove params for zuul_console https://review.openstack.org/472859 | 23:00 |
| pabelanger | jeblair: mordred ^ | 23:00 |
| *** amoralej|off is now known as amoralej | 23:00 | |
| openstackgerrit | Yolande Amate proposed openstack-infra/jenkins-job-builder master: Update artifactory_common_details to make use of convert_mapping_to_xml https://review.openstack.org/472860 | 23:01 |
| *** jamesmcarthur has quit IRC | 23:01 | |
| *** salv-orlando has quit IRC | 23:01 | |
| *** jamesdenton has quit IRC | 23:01 | |
| jeblair | pabelanger: is it valid to have nothing after the : ? | 23:02 |
| pabelanger | I _think_ so | 23:02 |
| mordred | jeblair: yes it is | 23:02 |
| mordred | I have a local playbook I use that does that | 23:02 |
| pabelanger | don't use that syntax much my self | 23:02 |
| pabelanger | finger b82946f7a81a496bbfa45451606c34b2@ze01.openstack.org | 23:03 |
| jeblair | huh. ends up as None. | 23:03 |
| pabelanger | seems better | 23:03 |
| jeblair | ok. | 23:03 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-roles master: Remove params for zuul_console https://review.openstack.org/472859 | 23:03 |
| *** EricGonczer_ has quit IRC | 23:07 | |
| *** EricGonczer_ has joined #openstack-infra | 23:08 | |
| *** kgiusti has left #openstack-infra | 23:09 | |
| *** EricGonczer_ has quit IRC | 23:09 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/openstack-zuul-roles master: Add libffi development headers for ansible / ansible-lint https://review.openstack.org/472861 | 23:09 |
| *** dizquierdo has joined #openstack-infra | 23:13 | |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack-infra/project-config master: Multinode gates, dockerhub publisher prework https://review.openstack.org/471428 | 23:15 |
| *** zz_dimtruck is now known as dimtruck | 23:17 | |
| *** tommylikehu has quit IRC | 23:23 | |
| *** tommylikehu has joined #openstack-infra | 23:24 | |
| *** jamesmcarthur has joined #openstack-infra | 23:26 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/openstack-zuul-jobs master: Stop console log streaming on remote host https://review.openstack.org/472866 | 23:28 |
| *** jamesmcarthur has quit IRC | 23:30 | |
| *** jamesdenton has joined #openstack-infra | 23:32 | |
| *** gardlt has joined #openstack-infra | 23:36 | |
| *** wolverin_ has quit IRC | 23:36 | |
| *** wolverineav has joined #openstack-infra | 23:36 | |
| *** gordc has quit IRC | 23:40 | |
| *** armax has quit IRC | 23:41 | |
| *** wolverineav has quit IRC | 23:42 | |
| *** armax has joined #openstack-infra | 23:42 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Add shutdown option for zuul_console https://review.openstack.org/472867 | 23:43 |
| *** dizquierdo has quit IRC | 23:44 | |
| *** armax has quit IRC | 23:45 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Add shutdown option for zuul_console https://review.openstack.org/472867 | 23:47 |
| openstackgerrit | Monty Taylor proposed openstack-infra/openstack-zuul-jobs master: Stop console log streaming on remote host https://review.openstack.org/472866 | 23:47 |
| mordred | clarkb, jeblair, pabelanger: ^^ and there's using it | 23:47 |
| *** gaelL has quit IRC | 23:48 | |
| pabelanger | mordred: noticed http://paste.openstack.org/show/612170/ on ze01.o.o also | 23:54 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!