| mnaser | hey -infra, we're doing some work moving things around from local storage to ceph (still have a few vms to finish that up).. one of them is mirror.ca-ymq-1.vexxhost.openstack.org | 00:01 |
|---|---|---|
| mnaser | i dont think that's being used right now as there are no ci slaves, but could it be temporarily safely shutdown for us to do the migration? | 00:01 |
| openstackgerrit | Merged openstack-infra/elastic-recheck master: Add query for rebuild vif plugging timeout bug 1694371 https://review.openstack.org/471471 | 00:01 |
| openstack | bug 1694371 in OpenStack Compute (nova) "Timeout while waiting for network-vif-plugged event during server rebuild" [Undecided,Confirmed] https://launchpad.net/bugs/1694371 | 00:01 |
| mnaser | (dont mind leaving it up) | 00:01 |
| clarkb | mnaser: you can shut it down | 00:01 |
| mnaser | so i can issue a nova stop from our side (it'll try to do acpi and then shut down after) | 00:02 |
| mnaser | that's okay? | 00:02 |
| mnaser | if it's okay, then we'll just do it when we have to do the migration | 00:03 |
| *** sflanigan has quit IRC | 00:04 | |
| *** stakeda has joined #openstack-infra | 00:05 | |
| *** markvoelker has quit IRC | 00:07 | |
| *** stakeda_ has joined #openstack-infra | 00:08 | |
| *** prometheanfire has joined #openstack-infra | 00:08 | |
| prometheanfire | is mitaka dvsm broken? | 00:08 |
| prometheanfire | fails look unrelated https://review.openstack.org/471169 | 00:08 |
| *** pahuang has quit IRC | 00:09 | |
| *** stakeda has quit IRC | 00:10 | |
| *** ihrachys has quit IRC | 00:12 | |
| *** rlandy has quit IRC | 00:17 | |
| *** r-daneel has quit IRC | 00:17 | |
| *** tuanluong has joined #openstack-infra | 00:18 | |
| *** adriant has joined #openstack-infra | 00:19 | |
| *** gardlt_ has joined #openstack-infra | 00:21 | |
| *** sflanigan has joined #openstack-infra | 00:23 | |
| *** pahuang has joined #openstack-infra | 00:26 | |
| jeblair | mnaser: yep that's fine | 00:26 |
| *** rloo has quit IRC | 00:34 | |
| *** stakeda has joined #openstack-infra | 00:35 | |
| *** stakeda_ has quit IRC | 00:36 | |
| *** yamamoto_ has joined #openstack-infra | 00:38 | |
| *** Julien-zte has joined #openstack-infra | 00:41 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul feature/zuulv3: executor: add support for custom ansible_port https://review.openstack.org/468710 | 00:42 |
| *** Julien-zte has quit IRC | 00:42 | |
| *** Julien-zte has joined #openstack-infra | 00:44 | |
| *** yamamoto_ has quit IRC | 00:44 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Log and collect stats for events consistently https://review.openstack.org/468664 | 00:49 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Add support for defining groups in nodesets https://review.openstack.org/467611 | 00:49 |
| *** vhosakot has quit IRC | 00:51 | |
| *** jbadiapa has quit IRC | 00:52 | |
| *** LindaWang has joined #openstack-infra | 00:52 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Start at block-device mounting/unmounting cleanup https://review.openstack.org/471183 | 00:54 |
| *** stakeda_ has joined #openstack-infra | 00:54 | |
| *** stakeda has quit IRC | 00:56 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Write inventory as yaml not ini https://review.openstack.org/467634 | 00:56 |
| *** rhallisey has quit IRC | 00:58 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Put variables into the inventory https://review.openstack.org/467635 | 00:58 |
| *** jbadiapa has joined #openstack-infra | 00:59 | |
| *** cuongnv has joined #openstack-infra | 00:59 | |
| *** Apoorva_ has joined #openstack-infra | 01:01 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: sql-connection: make _setup_tables staticmethod https://review.openstack.org/466455 | 01:01 |
| *** markvoelker has joined #openstack-infra | 01:03 | |
| *** Apoorva has quit IRC | 01:04 | |
| *** Apoorva_ has quit IRC | 01:05 | |
| *** thorst_afk has joined #openstack-infra | 01:06 | |
| *** namnh has joined #openstack-infra | 01:07 | |
| *** priteau has joined #openstack-infra | 01:08 | |
| *** stakeda has joined #openstack-infra | 01:11 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: PPC bootloader; install to root partition https://review.openstack.org/471525 | 01:11 |
| *** thorst_afk has quit IRC | 01:11 | |
| *** thorst_afk has joined #openstack-infra | 01:11 | |
| *** stakeda_ has quit IRC | 01:12 | |
| *** priteau has quit IRC | 01:12 | |
| *** xarses_ has joined #openstack-infra | 01:13 | |
| *** cuongnv_ has joined #openstack-infra | 01:14 | |
| *** LindaWang1 has joined #openstack-infra | 01:16 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: PPC bootloader; install to boot partition https://review.openstack.org/471525 | 01:16 |
| *** gcb has quit IRC | 01:17 | |
| *** andreww has quit IRC | 01:17 | |
| *** sflanigan has quit IRC | 01:17 | |
| *** LindaWang has quit IRC | 01:17 | |
| *** LindaWang1 is now known as LindaWang | 01:17 | |
| *** sflanigan has joined #openstack-infra | 01:17 | |
| *** gcb has joined #openstack-infra | 01:17 | |
| *** thorst_afk has quit IRC | 01:20 | |
| *** cuongnv has quit IRC | 01:21 | |
| *** stakeda has quit IRC | 01:23 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: PPC bootloader; install to boot partition https://review.openstack.org/471525 | 01:24 |
| *** gyee has quit IRC | 01:25 | |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck master: Add query for sync traits nova functional test bug 1696267 https://review.openstack.org/471528 | 01:27 |
| openstack | bug 1696267 in OpenStack Compute (nova) "Functional test test_sync_standard_traits intermittently fails" [High,In progress] https://launchpad.net/bugs/1696267 - Assigned to Chris Dent (cdent) | 01:27 |
| *** zhurong has joined #openstack-infra | 01:28 | |
| *** asselin_ has joined #openstack-infra | 01:29 | |
| *** Julien-zte has quit IRC | 01:30 | |
| *** asselin has quit IRC | 01:31 | |
| *** Julien-zte has joined #openstack-infra | 01:31 | |
| *** liujiong has joined #openstack-infra | 01:34 | |
| openstackgerrit | Merged openstack-infra/elastic-recheck master: Add query for sync traits nova functional test bug 1696267 https://review.openstack.org/471528 | 01:36 |
| openstack | bug 1696267 in OpenStack Compute (nova) "Functional test test_sync_standard_traits intermittently fails" [High,In progress] https://launchpad.net/bugs/1696267 - Assigned to Chris Dent (cdent) | 01:36 |
| *** dhinesh has quit IRC | 01:38 | |
| *** yamamoto_ has joined #openstack-infra | 01:40 | |
| *** gongysh has joined #openstack-infra | 01:41 | |
| *** cuongnv_ is now known as cuongnv | 01:42 | |
| *** yamamoto_ has quit IRC | 01:45 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Add a keep-output flag for functional tests https://review.openstack.org/471532 | 01:48 |
| *** rkukura has quit IRC | 01:49 | |
| *** chlong has quit IRC | 01:50 | |
| *** mattmceuen has joined #openstack-infra | 01:51 | |
| *** lbragstad has joined #openstack-infra | 01:51 | |
| *** thorst_afk has joined #openstack-infra | 01:56 | |
| *** gardlt_ has quit IRC | 01:57 | |
| *** yamahata has quit IRC | 01:59 | |
| *** gardlt has joined #openstack-infra | 01:59 | |
| openstackgerrit | zhurong proposed openstack-infra/project-config master: Add congress-dashboard to $PROJECTS variable in murano-congress-job https://review.openstack.org/471538 | 02:00 |
| *** salv-orlando has joined #openstack-infra | 02:05 | |
| *** salv-orl_ has quit IRC | 02:08 | |
| *** guyk has joined #openstack-infra | 02:09 | |
| *** mriedem has quit IRC | 02:10 | |
| guyk | Hi everyone, for our CI, we always have one test that keeps failing (tempest.api.compute.servers.test_server_rescue_negative.ServerRescueNegativeTestJSON). The issue is due to the timeout to detach the volume from the instance. We upgraded nodepool element and it is using xenial. Anyone has met this issue or have any clue about it? Thank you for your help. | 02:16 |
| *** yamamoto_ has joined #openstack-infra | 02:16 | |
| *** Shrews has quit IRC | 02:20 | |
| clarkb | guyk: you mean your test jobs run on xenial images built by nodepool? | 02:20 |
| *** sflanigan has quit IRC | 02:20 | |
| clarkb | guyk: I think we have noticed lvm related volume issues on xenial but I am not aware of them.being root caused yet | 02:21 |
| guyk | clarkb: yes test jobs are running on xenial | 02:22 |
| *** Shrews has joined #openstack-infra | 02:23 | |
| *** rkukura has joined #openstack-infra | 02:24 | |
| *** r-daneel has joined #openstack-infra | 02:26 | |
| guyk | clarkb: but we are not using lvm volumes in our third party CI for the tests | 02:27 |
| *** marst has joined #openstack-infra | 02:27 | |
| *** lbragstad has quit IRC | 02:30 | |
| *** zhurong has quit IRC | 02:30 | |
| *** rkukura_ has joined #openstack-infra | 02:31 | |
| *** rkukura has quit IRC | 02:31 | |
| *** rkukura_ is now known as rkukura | 02:31 | |
| *** thorst_afk has quit IRC | 02:32 | |
| *** thorst_afk has joined #openstack-infra | 02:33 | |
| *** r-daneel has quit IRC | 02:33 | |
| *** sflanigan has joined #openstack-infra | 02:34 | |
| *** sflanigan has joined #openstack-infra | 02:34 | |
| *** mat128 has quit IRC | 02:36 | |
| *** thorst_afk has quit IRC | 02:37 | |
| *** mat128 has joined #openstack-infra | 02:39 | |
| *** dimtruck is now known as zz_dimtruck | 02:47 | |
| *** zz_dimtruck is now known as dimtruck | 02:47 | |
| *** mat128 has quit IRC | 02:47 | |
| *** mat128 has joined #openstack-infra | 02:48 | |
| *** links has joined #openstack-infra | 02:52 | |
| *** mat128 has quit IRC | 02:52 | |
| *** dimtruck is now known as zz_dimtruck | 02:56 | |
| *** gomarivera has joined #openstack-infra | 02:58 | |
| *** vikrant has joined #openstack-infra | 03:02 | |
| *** rfolco has quit IRC | 03:03 | |
| *** gomarivera has quit IRC | 03:03 | |
| *** jamesmcarthur has joined #openstack-infra | 03:04 | |
| *** jamesmcarthur has quit IRC | 03:08 | |
| *** jamesmcarthur has joined #openstack-infra | 03:17 | |
| *** jamesmcarthur has quit IRC | 03:19 | |
| *** jamesmcarthur has joined #openstack-infra | 03:20 | |
| *** hongbin has joined #openstack-infra | 03:20 | |
| *** rfolco has joined #openstack-infra | 03:23 | |
| openstackgerrit | fumihiko kakuma proposed openstack-infra/project-config master: Add neutron periodic job with ryu master https://review.openstack.org/467885 | 03:28 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Update tracing in block_device_create_config_file https://review.openstack.org/471163 | 03:31 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Move ppc block-device default to right $ARCH https://review.openstack.org/471170 | 03:31 |
| *** aeng has quit IRC | 03:34 | |
| *** winggundamth has joined #openstack-infra | 03:34 | |
| *** sflanigan has quit IRC | 03:34 | |
| *** rkukura has quit IRC | 03:34 | |
| *** winggundamth has quit IRC | 03:37 | |
| *** winggundamth has joined #openstack-infra | 03:42 | |
| *** thorst_afk has joined #openstack-infra | 03:43 | |
| *** psachin has joined #openstack-infra | 03:44 | |
| *** jamesdenton has joined #openstack-infra | 03:45 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Pass all blockdevices to bootloader https://review.openstack.org/471183 | 03:49 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: PPC bootloader; install to boot partition https://review.openstack.org/471525 | 03:49 |
| *** marst has quit IRC | 03:49 | |
| *** sflanigan has joined #openstack-infra | 03:51 | |
| *** sflanigan has joined #openstack-infra | 03:51 | |
| *** aeng has joined #openstack-infra | 03:51 | |
| *** rfolco has quit IRC | 03:52 | |
| *** namnh_ has joined #openstack-infra | 03:54 | |
| *** namnh has quit IRC | 03:56 | |
| *** nicolasbock has joined #openstack-infra | 03:57 | |
| *** namnh has joined #openstack-infra | 03:57 | |
| *** ykarel_ has joined #openstack-infra | 03:59 | |
| *** ramishra has quit IRC | 03:59 | |
| *** namnh_ has quit IRC | 03:59 | |
| *** zhurong has joined #openstack-infra | 04:00 | |
| *** thorst_afk has quit IRC | 04:02 | |
| *** jamesdenton has quit IRC | 04:06 | |
| *** aeng has quit IRC | 04:09 | |
| *** jascott1 has quit IRC | 04:11 | |
| *** jascott1 has joined #openstack-infra | 04:12 | |
| *** sflanigan has quit IRC | 04:15 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Updated from global requirements https://review.openstack.org/470451 | 04:15 |
| openstackgerrit | Merged openstack/diskimage-builder master: Use https in docs links https://review.openstack.org/470791 | 04:16 |
| *** jascott1 has quit IRC | 04:16 | |
| *** jamesdenton has joined #openstack-infra | 04:21 | |
| *** jamesmcarthur has quit IRC | 04:22 | |
| *** aeng has joined #openstack-infra | 04:23 | |
| *** dhinesh has joined #openstack-infra | 04:23 | |
| *** Julien-zte has quit IRC | 04:27 | |
| *** Julien-zte has joined #openstack-infra | 04:28 | |
| *** sflanigan has joined #openstack-infra | 04:28 | |
| *** zhurong has quit IRC | 04:30 | |
| *** pahuang has quit IRC | 04:30 | |
| *** jascott1 has joined #openstack-infra | 04:32 | |
| *** adriant has quit IRC | 04:35 | |
| *** adriant has joined #openstack-infra | 04:39 | |
| *** zhurong has joined #openstack-infra | 04:40 | |
| *** adriant has quit IRC | 04:40 | |
| *** liujiong has quit IRC | 04:43 | |
| *** pahuang has joined #openstack-infra | 04:43 | |
| *** yamamoto_ has quit IRC | 04:44 | |
| *** zz_dimtruck is now known as dimtruck | 04:47 | |
| *** jascott1 has quit IRC | 04:47 | |
| *** hongbin has quit IRC | 04:47 | |
| *** jascott1 has joined #openstack-infra | 04:48 | |
| *** tumbarka has quit IRC | 04:49 | |
| *** florianf has joined #openstack-infra | 04:52 | |
| *** jascott1 has quit IRC | 04:52 | |
| *** pcaruana has joined #openstack-infra | 04:56 | |
| *** dimtruck is now known as zz_dimtruck | 04:57 | |
| *** rkukura has joined #openstack-infra | 04:59 | |
| *** Julien-zte has quit IRC | 05:06 | |
| *** sree has joined #openstack-infra | 05:08 | |
| *** priteau has joined #openstack-infra | 05:08 | |
| *** dims has quit IRC | 05:09 | |
| *** Julien-zte has joined #openstack-infra | 05:09 | |
| *** dhinesh has quit IRC | 05:10 | |
| *** priteau has quit IRC | 05:12 | |
| *** pgadiya has joined #openstack-infra | 05:12 | |
| *** pcaruana has quit IRC | 05:14 | |
| *** prateek has joined #openstack-infra | 05:15 | |
| *** ykarel__ has joined #openstack-infra | 05:16 | |
| *** Julien-z_ has joined #openstack-infra | 05:17 | |
| *** ykarel_ has quit IRC | 05:18 | |
| *** Julien-zte has quit IRC | 05:19 | |
| *** claudiub has joined #openstack-infra | 05:22 | |
| *** Julien-z_ has quit IRC | 05:25 | |
| *** pgadiya has quit IRC | 05:26 | |
| *** Julien-zte has joined #openstack-infra | 05:27 | |
| *** gildub_ has quit IRC | 05:27 | |
| *** Julien-zte has quit IRC | 05:28 | |
| *** Julien-zte has joined #openstack-infra | 05:31 | |
| *** yamamoto has joined #openstack-infra | 05:31 | |
| *** namnh_ has joined #openstack-infra | 05:38 | |
| *** namnh has quit IRC | 05:39 | |
| *** dims has joined #openstack-infra | 05:41 | |
| *** cshastri has joined #openstack-infra | 05:43 | |
| *** asselin has joined #openstack-infra | 05:46 | |
| *** Julien-zte has quit IRC | 05:46 | |
| *** zz_dimtruck is now known as dimtruck | 05:47 | |
| *** Julien-zte has joined #openstack-infra | 05:47 | |
| *** asselin_ has quit IRC | 05:48 | |
| ianw | http://logs.openstack.org/63/471163/5/gate/gate-dsvm-nodepool-redhat-src/10be690/ <- weird POST_FAILURE ... empty | 05:51 |
| *** dimtruck is now known as zz_dimtruck | 05:56 | |
| *** thorst_afk has joined #openstack-infra | 05:59 | |
| *** aeng has quit IRC | 06:01 | |
| openstackgerrit | Jamie Lennox proposed openstack-infra/nodepool feature/zuulv3: Allow loading logging config from yaml https://review.openstack.org/445656 | 06:01 |
| openstackgerrit | Merged openstack-infra/project-config master: Use new env vars for tripleo yaml build https://review.openstack.org/471481 | 06:01 |
| *** thorst_afk has quit IRC | 06:04 | |
| *** winggundamth has quit IRC | 06:05 | |
| *** dhajare|busy has joined #openstack-infra | 06:06 | |
| *** gildub has joined #openstack-infra | 06:10 | |
| *** Julien-zte has quit IRC | 06:10 | |
| *** Julien-zte has joined #openstack-infra | 06:11 | |
| *** jaosorior_away is now known as jaosorior | 06:11 | |
| openstackgerrit | zhurong proposed openstack-infra/project-config master: Add congress-dashboard to $PROJECTS variable in murano-congress-job https://review.openstack.org/471538 | 06:14 |
| *** markus_z has joined #openstack-infra | 06:19 | |
| *** rcernin has joined #openstack-infra | 06:21 | |
| *** jamesmcarthur has joined #openstack-infra | 06:23 | |
| *** isaacb has joined #openstack-infra | 06:24 | |
| *** salv-orlando has quit IRC | 06:25 | |
| *** jamesmcarthur has quit IRC | 06:27 | |
| *** ltomasbo|away is now known as ltomasbo | 06:30 | |
| *** gardlt has quit IRC | 06:31 | |
| *** cshastri has quit IRC | 06:33 | |
| *** hrubi has quit IRC | 06:35 | |
| *** tushar has joined #openstack-infra | 06:35 | |
| *** armax has quit IRC | 06:36 | |
| *** MatthewLi has quit IRC | 06:37 | |
| *** MatthewLi has joined #openstack-infra | 06:38 | |
| *** hrubi has joined #openstack-infra | 06:39 | |
| *** MatthewLi1 has joined #openstack-infra | 06:44 | |
| *** MatthewLi has quit IRC | 06:45 | |
| *** MatthewLi1 is now known as MatthewLi | 06:45 | |
| *** gildub has quit IRC | 06:46 | |
| *** dhajare|busy has quit IRC | 06:47 | |
| *** zz_dimtruck is now known as dimtruck | 06:47 | |
| *** cshastri has joined #openstack-infra | 06:49 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Add a keep-output flag for functional tests https://review.openstack.org/471532 | 06:50 |
| *** dimtruck is now known as zz_dimtruck | 06:57 | |
| *** zz_dimtruck is now known as dimtruck | 07:00 | |
| *** thorst_afk has joined #openstack-infra | 07:01 | |
| *** afazekas has quit IRC | 07:01 | |
| *** afazekas has joined #openstack-infra | 07:01 | |
| *** tuanluong_ has joined #openstack-infra | 07:01 | |
| *** pcaruana has joined #openstack-infra | 07:03 | |
| openstackgerrit | aditi sharma proposed openstack-infra/project-config master: Add install-guide-jobs for watcher https://review.openstack.org/471601 | 07:03 |
| *** tuanluong has quit IRC | 07:04 | |
| *** lewo has quit IRC | 07:05 | |
| *** thorst_afk has quit IRC | 07:05 | |
| *** lewo has joined #openstack-infra | 07:05 | |
| *** liujiong has joined #openstack-infra | 07:07 | |
| *** gardlt has joined #openstack-infra | 07:07 | |
| tushar | isa | 07:07 |
| *** salv-orlando has joined #openstack-infra | 07:09 | |
| *** dimtruck is now known as zz_dimtruck | 07:10 | |
| *** tesseract has joined #openstack-infra | 07:11 | |
| *** gardlt has quit IRC | 07:11 | |
| *** dhajare|busy has joined #openstack-infra | 07:12 | |
| *** stakeda has joined #openstack-infra | 07:14 | |
| *** flepied has quit IRC | 07:15 | |
| *** cshastri has quit IRC | 07:17 | |
| *** amoralej|off is now known as amoralej | 07:18 | |
| *** shardy has joined #openstack-infra | 07:21 | |
| *** Julien-zte has quit IRC | 07:23 | |
| openstackgerrit | Colleen Murphy proposed openstack-infra/project-config master: Use TEMPEST_REGEX in new keystone job https://review.openstack.org/471608 | 07:28 |
| *** rkukura has quit IRC | 07:28 | |
| openstackgerrit | Alfredo Moralejo proposed openstack-infra/tripleo-ci master: DNM: testing something for gate https://review.openstack.org/471361 | 07:28 |
| *** rkukura has joined #openstack-infra | 07:28 | |
| *** esikachev has joined #openstack-infra | 07:29 | |
| *** ccamacho has quit IRC | 07:30 | |
| *** salv-orl_ has joined #openstack-infra | 07:32 | |
| *** ccamacho has joined #openstack-infra | 07:33 | |
| *** namnh_ has quit IRC | 07:33 | |
| *** namnh has joined #openstack-infra | 07:33 | |
| *** jpena|off is now known as jpena | 07:34 | |
| *** cshastri has joined #openstack-infra | 07:35 | |
| *** alexchadin has joined #openstack-infra | 07:36 | |
| *** salv-orlando has quit IRC | 07:37 | |
| *** hashar has joined #openstack-infra | 07:38 | |
| *** hashar has quit IRC | 07:38 | |
| *** hashar has joined #openstack-infra | 07:39 | |
| *** pas-ha has left #openstack-infra | 07:40 | |
| EmilienM | clarkb: yes, me. I'll review it | 07:47 |
| openstackgerrit | Merged openstack-infra/project-config master: Switch to bash script for getting tempest plugin list https://review.openstack.org/470286 | 07:51 |
| *** flepied has joined #openstack-infra | 07:51 | |
| *** rcernin has quit IRC | 07:52 | |
| *** priteau has joined #openstack-infra | 07:52 | |
| *** rcernin has joined #openstack-infra | 07:52 | |
| *** tuanluong has joined #openstack-infra | 07:53 | |
| *** jpich has joined #openstack-infra | 07:53 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add experimental OVB ha job with tempest https://review.openstack.org/465030 | 07:55 |
| *** efoley has joined #openstack-infra | 07:55 | |
| *** tuanluong_ has quit IRC | 07:55 | |
| openstackgerrit | Merged openstack-infra/project-config master: vmware-nsx: return check-requirements to gating https://review.openstack.org/470966 | 07:56 |
| openstackgerrit | Merged openstack-infra/project-config master: do not use the remote prefix name for release notes scans https://review.openstack.org/471098 | 07:56 |
| openstackgerrit | Merged openstack-infra/project-config master: Add Multinode gate for OpenStack-Helm project https://review.openstack.org/471476 | 07:56 |
| openstackgerrit | Merged openstack-infra/project-config master: [sahara] introduce py3 scenario jobs https://review.openstack.org/471350 | 07:59 |
| *** zzzeek has quit IRC | 08:00 | |
| openstackgerrit | Merged openstack-infra/project-config master: jenkins: propose_update.sh: Use a single active repository for testing https://review.openstack.org/471326 | 08:00 |
| *** zzzeek has joined #openstack-infra | 08:00 | |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack-infra/project-config master: bifrost: Add experimental openSUSE 42.2 and CentOS 7 jobs https://review.openstack.org/471618 | 08:01 |
| *** zz_dimtruck is now known as dimtruck | 08:01 | |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack-infra/project-config master: bifrost: Add experimental openSUSE 42.2 and CentOS 7 jobs https://review.openstack.org/471618 | 08:01 |
| *** rossella_ has joined #openstack-infra | 08:01 | |
| *** thorst_afk has joined #openstack-infra | 08:01 | |
| openstackgerrit | Merged openstack-infra/project-config master: [sahara] put saharaclient in the projects list https://review.openstack.org/471336 | 08:02 |
| openstackgerrit | Merged openstack-infra/project-config master: Add devstack-plugin-container to solum gate https://review.openstack.org/471208 | 08:02 |
| openstackgerrit | zhurong proposed openstack-infra/project-config master: Add congress-dashboard to $PROJECTS variable in murano-congress-job https://review.openstack.org/471538 | 08:05 |
| rm_work | requirements proposal bot has gone a bit crazy... it's proposing new CRs constantly against some old tag instead of master... i went and abandoned a bunch today but it's super weird and still going: https://review.openstack.org/#/q/project:openstack/neutron-lbaas-dashboard | 08:07 |
| rm_work | any ideas? | 08:07 |
| openstackgerrit | chenhb proposed openstack-infra/project-config master: [rally]increase timeout value for cli job https://review.openstack.org/471622 | 08:09 |
| *** rossella_ has quit IRC | 08:09 | |
| *** tinwood has quit IRC | 08:11 | |
| *** rossella_ has joined #openstack-infra | 08:12 | |
| *** dimtruck is now known as zz_dimtruck | 08:12 | |
| *** tinwood has joined #openstack-infra | 08:12 | |
| *** Julien-zte has joined #openstack-infra | 08:15 | |
| *** ramishra has joined #openstack-infra | 08:16 | |
| *** lucas-afk is now known as lucasagomes | 08:16 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Update tracing in block_device_create_config_file https://review.openstack.org/471163 | 08:18 |
| *** thorst_afk has quit IRC | 08:21 | |
| *** rossella_ has quit IRC | 08:21 | |
| *** numans has joined #openstack-infra | 08:22 | |
| *** Julien-zte has quit IRC | 08:24 | |
| *** gongysh has quit IRC | 08:25 | |
| *** alexchadin has quit IRC | 08:26 | |
| *** gongysh has joined #openstack-infra | 08:33 | |
| *** florianf has quit IRC | 08:34 | |
| *** alexchadin has joined #openstack-infra | 08:35 | |
| *** ggnel_t has joined #openstack-infra | 08:36 | |
| *** derekh has joined #openstack-infra | 08:37 | |
| *** florianf has joined #openstack-infra | 08:38 | |
| *** cshastri has quit IRC | 08:40 | |
| *** tosky has joined #openstack-infra | 08:40 | |
| *** Julien-zte has joined #openstack-infra | 08:41 | |
| *** rossella_ has joined #openstack-infra | 08:41 | |
| *** neil__ has joined #openstack-infra | 08:44 | |
| *** zhurong has quit IRC | 08:47 | |
| *** rossella_ has quit IRC | 08:49 | |
| *** rossella__ has joined #openstack-infra | 08:49 | |
| *** jamielennox is now known as jamielennox|away | 08:50 | |
| *** rossella_ has joined #openstack-infra | 08:54 | |
| *** cshastri has joined #openstack-infra | 08:54 | |
| *** rossella__ has quit IRC | 08:55 | |
| *** cshastri has quit IRC | 08:55 | |
| *** cshastri has joined #openstack-infra | 08:56 | |
| *** pbourke has joined #openstack-infra | 08:56 | |
| *** gomarivera has joined #openstack-infra | 08:59 | |
| *** rossella__ has joined #openstack-infra | 09:01 | |
| *** zhurong has joined #openstack-infra | 09:01 | |
| *** rossella_ has quit IRC | 09:01 | |
| *** gomarivera has quit IRC | 09:03 | |
| *** zz_dimtruck is now known as dimtruck | 09:03 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/project-config master: Fix buildimage jobs to use new process https://review.openstack.org/471647 | 09:04 |
| openstackgerrit | yolanda.robla proposed openstack-infra/project-config master: Fix buildimage jobs to use new process https://review.openstack.org/471647 | 09:06 |
| *** sshnaidm|afk is now known as sshnaidm | 09:09 | |
| *** neil__ has quit IRC | 09:09 | |
| *** rossella_ has joined #openstack-infra | 09:10 | |
| *** jamielennox|away is now known as jamielennox | 09:10 | |
| *** florianf has quit IRC | 09:11 | |
| *** rossella__ has quit IRC | 09:11 | |
| *** dimtruck is now known as zz_dimtruck | 09:13 | |
| *** rossella_ has quit IRC | 09:15 | |
| *** thorst_afk has joined #openstack-infra | 09:18 | |
| *** rossella_ has joined #openstack-infra | 09:20 | |
| *** thorst_afk has quit IRC | 09:22 | |
| *** namnh_ has joined #openstack-infra | 09:22 | |
| *** florianf has joined #openstack-infra | 09:22 | |
| *** jamesmcarthur has joined #openstack-infra | 09:23 | |
| *** namnh has quit IRC | 09:25 | |
| *** markmcd has quit IRC | 09:25 | |
| *** markmcd has joined #openstack-infra | 09:27 | |
| *** jamesmcarthur has quit IRC | 09:27 | |
| *** markmcd has quit IRC | 09:31 | |
| *** sdague has joined #openstack-infra | 09:31 | |
| *** e0ne has joined #openstack-infra | 09:31 | |
| *** markmcd has joined #openstack-infra | 09:33 | |
| *** stakeda has quit IRC | 09:34 | |
| *** rossella_ has quit IRC | 09:35 | |
| *** neil__ has joined #openstack-infra | 09:38 | |
| *** mriedem has joined #openstack-infra | 09:44 | |
| *** rossella_ has joined #openstack-infra | 09:47 | |
| *** zhurong has quit IRC | 09:51 | |
| openstackgerrit | Merged openstack-infra/project-config master: Fix buildimage jobs to use new process https://review.openstack.org/471647 | 09:55 |
| *** ralonsoh has joined #openstack-infra | 09:55 | |
| openstackgerrit | Modestas Vainius proposed openstack-infra/jenkins-job-builder master: Support specifing user for querying plugin info https://review.openstack.org/471697 | 09:59 |
| openstackgerrit | Merged openstack/diskimage-builder master: Move ppc block-device default to right $ARCH https://review.openstack.org/471170 | 10:02 |
| *** nicolasbock_ has joined #openstack-infra | 10:04 | |
| *** alexchadin has quit IRC | 10:04 | |
| *** zz_dimtruck is now known as dimtruck | 10:04 | |
| *** alexchadin has joined #openstack-infra | 10:04 | |
| *** nicolasbock has quit IRC | 10:06 | |
| openstackgerrit | Modestas Vainius proposed openstack-infra/jenkins-job-builder master: Support taking slack token from Jenkins credentials. https://review.openstack.org/471415 | 10:06 |
| *** psachin has quit IRC | 10:08 | |
| *** psachin has joined #openstack-infra | 10:09 | |
| *** liujiong has quit IRC | 10:09 | |
| *** psachin has quit IRC | 10:09 | |
| *** dimtruck is now known as zz_dimtruck | 10:14 | |
| *** hrubi has quit IRC | 10:14 | |
| *** hrubi has joined #openstack-infra | 10:14 | |
| *** lucasagomes has quit IRC | 10:15 | |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci master: quickstart transition: add promotion logic https://review.openstack.org/445467 | 10:18 |
| openstackgerrit | yolanda.robla proposed openstack-infra/project-config master: Fix permissions on the buildimage job https://review.openstack.org/471707 | 10:19 |
| *** lucasagomes has joined #openstack-infra | 10:19 | |
| openstackgerrit | Modestas Vainius proposed openstack-infra/jenkins-job-builder master: Support specifing user for querying plugin info https://review.openstack.org/471697 | 10:21 |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci master: quickstart transition: add promotion logic https://review.openstack.org/445467 | 10:23 |
| *** gcb has quit IRC | 10:24 | |
| *** wolverineav has joined #openstack-infra | 10:27 | |
| *** hrubi has quit IRC | 10:28 | |
| *** hrubi has joined #openstack-infra | 10:28 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: DONT REVIEW: test ci patch https://review.openstack.org/471710 | 10:28 |
| *** wfoster has quit IRC | 10:29 | |
| *** rook has quit IRC | 10:29 | |
| *** alexchadin has quit IRC | 10:29 | |
| *** rook has joined #openstack-infra | 10:30 | |
| *** hrubi has quit IRC | 10:30 | |
| *** rook is now known as Guest52590 | 10:30 | |
| *** wfoster has joined #openstack-infra | 10:30 | |
| *** hrubi has joined #openstack-infra | 10:30 | |
| *** namnh has joined #openstack-infra | 10:30 | |
| *** alexchadin has joined #openstack-infra | 10:31 | |
| *** wolverineav has quit IRC | 10:31 | |
| *** sambetts|afk is now known as sambetts | 10:32 | |
| openstackgerrit | Bogdan Dobrelya proposed openstack-infra/tripleo-ci master: DO NOT MERGE testing 450792 https://review.openstack.org/466322 | 10:33 |
| *** namnh_ has quit IRC | 10:33 | |
| *** tushar has quit IRC | 10:33 | |
| *** zz_dimtruck is now known as dimtruck | 10:35 | |
| *** alexchadin has quit IRC | 10:36 | |
| openstackgerrit | Merged openstack-infra/project-config master: Fix permissions on the buildimage job https://review.openstack.org/471707 | 10:37 |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/project-config master: tripleo: transition periodic master job to quickstart https://review.openstack.org/463945 | 10:37 |
| *** alexchadin has joined #openstack-infra | 10:38 | |
| *** wolverineav has joined #openstack-infra | 10:39 | |
| *** wolverineav has quit IRC | 10:40 | |
| *** wolverin_ has joined #openstack-infra | 10:40 | |
| *** sree has quit IRC | 10:40 | |
| *** wolverineav has joined #openstack-infra | 10:42 | |
| *** wolverin_ has quit IRC | 10:42 | |
| *** rossella_ has quit IRC | 10:43 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul feature/zuulv3: executor: add support for custom ansible_port https://review.openstack.org/468710 | 10:43 |
| *** hashar has quit IRC | 10:44 | |
| *** dimtruck is now known as zz_dimtruck | 10:45 | |
| *** tushar has joined #openstack-infra | 10:50 | |
| *** elajkat_ has joined #openstack-infra | 10:55 | |
| elajkat_ | Hi Guys | 10:56 |
| *** rhallisey has joined #openstack-infra | 10:56 | |
| elajkat_ | We are working on a Heat feature to enable the trunk (VLAN aware VMs) feature from neutron in Heat | 10:56 |
| *** namnh has quit IRC | 10:56 | |
| openstackgerrit | Ricardo Noriega proposed openstack-infra/project-config master: Configuring services for L2GW experimental job https://review.openstack.org/471719 | 10:56 |
| elajkat_ | For that we need devstack gate jobs to run with the trunk extension enabled | 10:57 |
| *** Julien-zte has quit IRC | 10:57 | |
| elajkat_ | Do you have some idea what we need for that, with whom we should ask for help? | 10:57 |
| *** Julien-zte has joined #openstack-infra | 10:58 | |
| *** rossella_ has joined #openstack-infra | 10:59 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul feature/zuulv3: executor: add support for custom ansible_port https://review.openstack.org/468710 | 11:04 |
| *** e0ne has quit IRC | 11:04 | |
| *** lucasagomes is now known as lucas-hungry | 11:04 | |
| *** shardy is now known as shardy_lunch | 11:06 | |
| *** yamamoto has quit IRC | 11:10 | |
| *** jkilpatr has joined #openstack-infra | 11:10 | |
| *** rfolco has joined #openstack-infra | 11:17 | |
| *** LindaWang has quit IRC | 11:20 | |
| *** jamesmcarthur has joined #openstack-infra | 11:24 | |
| *** jamesmcarthur has quit IRC | 11:29 | |
| *** ldnunes has joined #openstack-infra | 11:31 | |
| openstackgerrit | Merged openstack-infra/project-config master: Configuring services for L2GW experimental job https://review.openstack.org/471719 | 11:32 |
| *** LindaWang has joined #openstack-infra | 11:32 | |
| *** zz_dimtruck is now known as dimtruck | 11:36 | |
| *** oanson has quit IRC | 11:40 | |
| *** psachin has joined #openstack-infra | 11:40 | |
| *** Guest52590 is now known as rook | 11:40 | |
| *** LindaWang has quit IRC | 11:41 | |
| *** LindaWang has joined #openstack-infra | 11:41 | |
| *** rossella_ has quit IRC | 11:41 | |
| *** oanson has joined #openstack-infra | 11:42 | |
| *** jpena is now known as jpena|lunch | 11:42 | |
| *** shardy_lunch is now known as shardy | 11:43 | |
| *** Julien-zte has quit IRC | 11:45 | |
| *** dimtruck is now known as zz_dimtruck | 11:45 | |
| *** cuongnv has quit IRC | 11:46 | |
| *** thorst_afk has joined #openstack-infra | 11:49 | |
| *** abishop has joined #openstack-infra | 11:54 | |
| *** tuanluong has quit IRC | 11:55 | |
| *** florianf has quit IRC | 11:56 | |
| *** alexchadin has quit IRC | 12:02 | |
| *** chlong has joined #openstack-infra | 12:03 | |
| *** ccamacho is now known as ccamacho|lunch | 12:06 | |
| *** mat128 has joined #openstack-infra | 12:08 | |
| *** lucas-hungry is now known as lucasagomes | 12:09 | |
| *** trown|outtypewww is now known as trown | 12:13 | |
| *** hashar has joined #openstack-infra | 12:15 | |
| *** isaacb has quit IRC | 12:16 | |
| *** florianf has joined #openstack-infra | 12:16 | |
| *** gongysh has quit IRC | 12:19 | |
| *** gongysh has joined #openstack-infra | 12:19 | |
| *** isaacb has joined #openstack-infra | 12:21 | |
| *** jiaohaolin has joined #openstack-infra | 12:22 | |
| *** gongysh has quit IRC | 12:23 | |
| *** rlandy has joined #openstack-infra | 12:26 | |
| *** rossella_ has joined #openstack-infra | 12:27 | |
| *** LindaWang has quit IRC | 12:30 | |
| *** LindaWang has joined #openstack-infra | 12:31 | |
| *** rossella_ has quit IRC | 12:31 | |
| jiaohaolin | Hello guys, my CI just setup ok, and it only worked for 5 hours, then the debug-log shows "Looking for the lost building", so I restart the service of zuul. BOOM, the zuul can't trigger the jobs . And the zuul.log shows | 12:32 |
| jiaohaolin | "2017-06-07 20:26:29,413 WARNING zuul.GerritEventConnector: Received unrecognized event type 'ref-replicated' from Gerrit. Can not get account information." | 12:32 |
| jiaohaolin | No matter how many times I restart the zuul, it just does not work | 12:32 |
| jiaohaolin | It drive me crazy | 12:32 |
| jiaohaolin | I'm working on this for months. Please help me | 12:33 |
| *** mattmceuen has quit IRC | 12:34 | |
| openstackgerrit | Merged openstack-infra/nodepool feature/zuulv3: Add boot-from-volume support for nodes https://review.openstack.org/464283 | 12:35 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Re-add the ability to set username on zuul-executor https://review.openstack.org/446308 | 12:36 |
| *** yamamoto has joined #openstack-infra | 12:36 | |
| *** zz_dimtruck is now known as dimtruck | 12:36 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Remove pipeline argument from various report fncts https://review.openstack.org/464252 | 12:37 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Increase file permissions around generate keys https://review.openstack.org/461218 | 12:38 |
| *** gardlt has joined #openstack-infra | 12:38 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Use ssh for git-upload-pack https://review.openstack.org/436802 | 12:38 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Bail with an error on a non-existent jobroot_dir https://review.openstack.org/469524 | 12:39 |
| *** Julien-zte has joined #openstack-infra | 12:42 | |
| openstackgerrit | Brad P. Crochet proposed openstack-infra/project-config master: Rename agent-ramdisk job to python-ironic-agent https://review.openstack.org/471758 | 12:42 |
| *** hashar has quit IRC | 12:42 | |
| *** gardlt has quit IRC | 12:43 | |
| *** rossella_ has joined #openstack-infra | 12:43 | |
| *** askb has quit IRC | 12:44 | |
| *** markus_z has quit IRC | 12:46 | |
| *** esikachev has quit IRC | 12:46 | |
| *** vikrant has quit IRC | 12:46 | |
| *** dimtruck is now known as zz_dimtruck | 12:46 | |
| *** yamamoto has quit IRC | 12:47 | |
| *** jpena|lunch is now known as jpena | 12:48 | |
| *** salv-orl_ has quit IRC | 12:49 | |
| *** salv-orlando has joined #openstack-infra | 12:50 | |
| *** alexchadin has joined #openstack-infra | 12:50 | |
| *** salv-orlando has quit IRC | 12:54 | |
| *** eharney has joined #openstack-infra | 12:56 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: TEST: DONT RECHECK or REVIEW: periodic jobs https://review.openstack.org/359215 | 12:57 |
| *** ccamacho|lunch is now known as ccamacho | 12:57 | |
| pabelanger | clarkb: Oh, thanks. I did the wrong thing for zookeeper, fixing | 12:57 |
| *** trown is now known as trown|brb | 12:58 | |
| openstackgerrit | Brad P. Crochet proposed openstack-infra/project-config master: Rename agent-ramdisk job to ironic-python-agent https://review.openstack.org/471758 | 12:58 |
| openstackgerrit | Justin Kilpatrick proposed openstack-infra/project-config master: Add seperate pep8 job for Browbeat https://review.openstack.org/470953 | 12:59 |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Allow ze01.o.o access to zookeeper / add to cacti.o.o https://review.openstack.org/471450 | 13:02 |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Disable webui for zuul-executors https://review.openstack.org/471458 | 13:02 |
| *** ihrachys has joined #openstack-infra | 13:06 | |
| *** e0ne has joined #openstack-infra | 13:06 | |
| *** yamamoto has joined #openstack-infra | 13:08 | |
| *** esikachev has joined #openstack-infra | 13:08 | |
| *** esikachev has quit IRC | 13:12 | |
| *** sree has joined #openstack-infra | 13:13 | |
| *** trown|brb is now known as trown | 13:13 | |
| *** mdrabe has joined #openstack-infra | 13:13 | |
| *** bobh has joined #openstack-infra | 13:14 | |
| *** felipemonteiro has joined #openstack-infra | 13:15 | |
| *** felipemonteiro_ has joined #openstack-infra | 13:16 | |
| *** markus_z has joined #openstack-infra | 13:19 | |
| *** felipemonteiro has quit IRC | 13:20 | |
| *** baoli has joined #openstack-infra | 13:23 | |
| *** esikachev has joined #openstack-infra | 13:24 | |
| *** e0ne has quit IRC | 13:26 | |
| *** pradk has quit IRC | 13:27 | |
| *** hashar has joined #openstack-infra | 13:27 | |
| *** e0ne has joined #openstack-infra | 13:28 | |
| *** esikachev has quit IRC | 13:28 | |
| *** jcoufal has joined #openstack-infra | 13:29 | |
| *** links has quit IRC | 13:30 | |
| *** esikachev has joined #openstack-infra | 13:30 | |
| openstackgerrit | Ricardo Noriega proposed openstack-infra/project-config master: Add enable_service parameter for l2gw_plugin and l2gw_agent https://review.openstack.org/471775 | 13:30 |
| *** eharney has quit IRC | 13:31 | |
| *** amoralej is now known as amoralej|lunch | 13:31 | |
| *** r-daneel has joined #openstack-infra | 13:31 | |
| *** r-daneel has quit IRC | 13:33 | |
| *** r-daneel has joined #openstack-infra | 13:33 | |
| *** eharney has joined #openstack-infra | 13:34 | |
| openstackgerrit | Merged openstack-infra/project-config master: Rename agent-ramdisk job to ironic-python-agent https://review.openstack.org/471758 | 13:35 |
| *** dave-mccowan has joined #openstack-infra | 13:36 | |
| openstackgerrit | Spyros Trigazis (strigazi) proposed openstack-infra/project-config master: [magnum] Add experimental dcos job https://review.openstack.org/471776 | 13:37 |
| *** zz_dimtruck is now known as dimtruck | 13:37 | |
| *** Goneri has joined #openstack-infra | 13:39 | |
| openstackgerrit | Spyros Trigazis (strigazi) proposed openstack-infra/project-config master: [magnum] Add experimental dcos job https://review.openstack.org/471776 | 13:40 |
| *** dave-mccowan has quit IRC | 13:41 | |
| *** florianf has quit IRC | 13:41 | |
| openstackgerrit | Bulat Gaifullin proposed openstack-infra/jenkins-job-builder master: Ensure jenkins url and credentials from config is instance of str https://review.openstack.org/463554 | 13:42 |
| *** esikachev has quit IRC | 13:42 | |
| *** chlong has quit IRC | 13:43 | |
| openstackgerrit | Yan Xing an proposed openstack-infra/project-config master: Add barbican to tacker localrc for jenkins job https://review.openstack.org/471780 | 13:44 |
| *** esikachev has joined #openstack-infra | 13:44 | |
| *** hashar has quit IRC | 13:44 | |
| *** dave-mccowan has joined #openstack-infra | 13:45 | |
| *** esikachev has quit IRC | 13:48 | |
| *** dimtruck is now known as zz_dimtruck | 13:50 | |
| hwoarang | could someone help me retrieve a log from a propose job? i am looking for the logs from the propose-openstack-ansible-update-osa-test-scripts job. This is a post-merge job so it leaves no traces in gerrit if it does nothing and logstash.openstack.org can't find the log i am looking for. i must be using the wrong filter or something | 13:50 |
| *** esikachev has joined #openstack-infra | 13:51 | |
| tushar | Hi All, DIB image creation failed with error - https://pastebin.com/5LCyB4eh | 13:53 |
| openstackgerrit | Merged openstack-infra/project-config master: Add enable_service parameter for l2gw_plugin and l2gw_agent https://review.openstack.org/471775 | 13:53 |
| tushar | Any one have any solution for this | 13:53 |
| *** zhurong has joined #openstack-infra | 13:54 | |
| pabelanger | tushar: /opt/cache should be created by devstack-cache | 13:54 |
| *** esikachev has quit IRC | 13:55 | |
| tushar | Pabelanger : Okay | 13:58 |
| *** florianf has joined #openstack-infra | 13:58 | |
| *** gomarivera has joined #openstack-infra | 13:59 | |
| hwoarang | nm | 13:59 |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack-infra/project-config master: jenkins: scripts: propose_update: Pass project directory as first argument https://review.openstack.org/471785 | 13:59 |
| *** rossella__ has joined #openstack-infra | 14:00 | |
| openstackgerrit | Gergely Csatari proposed openstack-infra/project-config master: Adding Vitrage install guide https://review.openstack.org/471786 | 14:00 |
| *** alexchadin has quit IRC | 14:01 | |
| *** yamamoto has quit IRC | 14:03 | |
| *** rossella_ has quit IRC | 14:03 | |
| *** gardlt has joined #openstack-infra | 14:03 | |
| *** gardlt has quit IRC | 14:03 | |
| *** yamamoto has joined #openstack-infra | 14:03 | |
| *** gardlt has joined #openstack-infra | 14:04 | |
| *** amoralej|lunch is now known as amoralej | 14:06 | |
| *** yamamoto has quit IRC | 14:08 | |
| *** esikachev has joined #openstack-infra | 14:08 | |
| tushar | Pabelanger : you mean cache-devstack element? | 14:08 |
| pabelanger | tushar: Ya, got them backwards | 14:09 |
| *** ykarel__ has quit IRC | 14:09 | |
| *** armax has joined #openstack-infra | 14:10 | |
| tushar | Pabelanger : ok | 14:11 |
| *** zz_dimtruck is now known as dimtruck | 14:12 | |
| *** esikachev has quit IRC | 14:12 | |
| *** dave-mccowan has quit IRC | 14:12 | |
| *** esikachev has joined #openstack-infra | 14:15 | |
| *** sree has quit IRC | 14:16 | |
| *** zhurong has quit IRC | 14:17 | |
| sdague | are we sure that multinode is working on citycloud - http://logs.openstack.org/16/449416/2/gate/gate-tempest-dsvm-multinode-live-migration-ubuntu-xenial/992306b/console.html ? | 14:19 |
| pabelanger | sure work, I know we there was an issue with networking in sto2 in the beginning | 14:22 |
| *** jamesmcarthur has joined #openstack-infra | 14:22 | |
| *** prateek has quit IRC | 14:22 | |
| *** esikachev has quit IRC | 14:23 | |
| *** pradk has joined #openstack-infra | 14:24 | |
| sdague | yeh, that is a pretty early setup fail with subnode connectivity | 14:25 |
| *** esikachev has joined #openstack-infra | 14:26 | |
| *** pradk has quit IRC | 14:28 | |
| pabelanger | IIRC, there was an issue with the private IP address but clarkb knows more | 14:28 |
| fungi | you mean with the instances that were getting an extra dead ip address in the nova api? | 14:29 |
| clarkb | ya that should be fixed | 14:29 |
| pabelanger | Oh, I thought it was something else | 14:29 |
| clarkb | since we only give out the private ip associated with the floating ip which is used for ssh connectivity checking | 14:29 |
| fungi | or at least shouldn't be impacting multi-node job network connectivity any longer, right? i don't think we ever heard back from them on whether the environment is broken on their end | 14:30 |
| *** pradk has joined #openstack-infra | 14:31 | |
| clarkb | fungi: correct, we worked around it on our end by only using private IPs with a floating IP. Which have to work in order to pass the ssh connectivity test that nodepool performs | 14:31 |
| *** jaosorior is now known as jaosorior_away | 14:31 | |
| clarkb | one region (kna1) continues to provide 2 private IPs | 14:31 |
| *** esikachev has quit IRC | 14:32 | |
| clarkb | sdague: if you look at that log it is successfully talking to the subnode for many tasks | 14:32 |
| fungi | this job was in sto2 anyway | 14:32 |
| clarkb | and then fails, so could possibly be a new thing | 14:32 |
| fungi | yeah, looks like sshd died/stopped at some point | 14:32 |
| *** esikachev has joined #openstack-infra | 14:33 | |
| fungi | we seem to have a syslog from the subnode too | 14:33 |
| mgagne | I'm having an issue with pbr and PBR_VERSION env variable where a dependency (oslo.messaging) requires pbr and installs it. But PBR_VERSION ends up overriding the version of pbr that dependency tries to install and obviously, it fails miserably with pkg_resources.DistributionNotFound. | 14:33 |
| mgagne | has anyone encountered that issue and how did you fix it? | 14:34 |
| fungi | mgagne: ooh, like you're setting PBR_VERSION and doing dependency installs during an sdist build? | 14:34 |
| *** rloo has joined #openstack-infra | 14:35 | |
| mgagne | well, I think it happens during pip install -r requirements.txt | 14:35 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: WIP Make log streaming to executor per-task https://review.openstack.org/471799 | 14:35 |
| clarkb | I think PBR_VERSION is generally only used by people building pacakges. I would work around that by install all of the requirements first then installing the thing you want a different version on | 14:35 |
| fungi | mgagne: i think the majority of users for PBR_VERSION are distro packagers who are building in the system environment with all python dependencies already preinstalled from system packages | 14:35 |
| mgagne | yes, I'm that guy atm | 14:35 |
| mgagne | well | 14:35 |
| mgagne | using virtualenv with .dev | 14:35 |
| mgagne | .deb* | 14:36 |
| *** gordc has joined #openstack-infra | 14:36 | |
| fungi | mgagne: are you preinstalling the debs for all the dependencies? may just need to enable system sitepackages for that venv | 14:36 |
| mgagne | there is no debs | 14:36 |
| mgagne | all virtualenv with dh-virtualenv | 14:36 |
| mgagne | so you are saying that PBR_VERSION must be set after pip install -r requirements.txt but before pip install . ? | 14:37 |
| fungi | oh, i haven't played with that particular debhelper extension before | 14:37 |
| fungi | does it try to autopackage your dependencies for you or something? | 14:37 |
| fungi | mgagne: yes, that would be one way to solve it | 14:37 |
| mgagne | working *very* fine until you try to use python3.4 and it needs to install oslo.messaging from sources instead of wheel :D | 14:37 |
| *** esikachev has quit IRC | 14:37 | |
| mgagne | it creates *one* package (by default) with your venv | 14:38 |
| fungi | mgagne: we don't publish universal wheels of oslo.messaging?!? i find that shocking | 14:38 |
| mgagne | fungi: lets be shocked together then :D | 14:38 |
| *** esikachev has joined #openstack-infra | 14:38 | |
| fungi | mgagne: got it, so i think other package maintainers start by packaging their dependencies and work their way up the dependency chain instead of building with dependendencies installed via pip | 14:38 |
| mgagne | oslo.messaging-5.14.0.tar.gz with python3.4, no wheel unless I misconfigured pip | 14:38 |
| clarkb | https://pypi.python.org/pypi/oslo.messaging/5.26.0 has a wheel | 14:39 |
| mgagne | fungi: yea but no, that's not gonna happen, we moved away from system packages for deps, much easier =) | 14:39 |
| fungi | mgagne: aha, yep we're publishing a py2 wheel currently | 14:39 |
| clarkb | oh py3.4 | 14:39 |
| mgagne | ok, so bumping to .26 should "fix" it | 14:39 |
| clarkb | well the wheel is only for py2 | 14:39 |
| mgagne | hmm no py34 :D | 14:39 |
| fungi | does latest master of oslo.messaging sypport py3k? | 14:39 |
| mgagne | haaaa | 14:39 |
| clarkb | sdague: logstash says that is the only occurence of that error (granted networking problems can manifest with many different logged errors) | 14:40 |
| mgagne | 3.5 according to pypi | 14:40 |
| fungi | mgagne: compare to the wheel for, say, oslo.db which is a py2.py3 (so-called "universal") wheel | 14:40 |
| clarkb | sdague: I don't think this is a case of multinode not working in particular cloud and more "something hiccuped" in a cloud | 14:40 |
| mgagne | ouch, I'm running trusty which only provides python3.4, not 3.5 | 14:41 |
| clarkb | fungi: iirc the wheel publishing jobs look at the trove data | 14:41 |
| fungi | mgagne: if it works for one, then probably works for the other | 14:41 |
| SamYaple | can I get some love on https://review.openstack.org/#/c/452547/ ? its been a few weeks | 14:41 |
| openstackgerrit | Tin Lam proposed openstack-infra/project-config master: Add new repo openstack-helm-infra under OSH https://review.openstack.org/469079 | 14:41 |
| fungi | mgagne: i strongly recommend checking with #openstack-oslo about whether oslo.messaging should have its setup.cfg adjusted to turn on universal wheels | 14:41 |
| *** rossella__ has quit IRC | 14:41 | |
| fungi | mgagne: maybe they can get a point release cut quickly | 14:42 |
| mgagne | fungi: so 3 options: downgrade to python 2.7, package whl for python 3.4, sequence pip to define PBR_VERSION *after* reqs installation | 14:42 |
| fungi | mgagne: i'll go ahead and propose the universal setting for oslo.messaging now | 14:43 |
| mgagne | fungi: ok, so there is nothing for me to do to get this done? except finding a tmp solution until it happens? | 14:43 |
| *** mdrabe has quit IRC | 14:44 | |
| fungi | mgagne: https://review.openstack.org/471807 should fix this on their next tag | 14:45 |
| *** rossella__ has joined #openstack-infra | 14:45 | |
| mgagne | =) thanks! | 14:45 |
| *** cshastri has quit IRC | 14:45 | |
| fungi | (assuming they merge it anyway) | 14:46 |
| *** elajkat_ has quit IRC | 14:46 | |
| fungi | mgagne: you could `python setup.py bdist_wheel` with that applied in the meantime and stash that wheel in your wheelhouse or something | 14:47 |
| mgagne | fungi: yea, will try to figure out something, maybe I can undefine PBR_VERSION during deps installation and define it during pip install . | 14:48 |
| fungi | that would be one solution, sure | 14:48 |
| tushar | Pabelanger : one more problem in DIB image creation - https://pastebin.com/XtJ91eFN | 14:50 |
| *** rossella__ has quit IRC | 14:50 | |
| tushar | chown: invalid user: 'jenkins:jenkins' | 14:50 |
| *** hongbin has joined #openstack-infra | 14:50 | |
| fungi | clarkb: as to your question last night (sorry, date night started right after the tc meeting) i just need to comb through the meetnig minutes and turn all the agreeds and actions into specs repo patches | 14:50 |
| *** Marx314 has joined #openstack-infra | 14:51 | |
| fungi | which i'm starting on now | 14:51 |
| pabelanger | tushar: have you copied the latest nodepool elements from project-config? That is setup in puppet element now directly | 14:51 |
| fungi | s/puppet/nodepool/ ? | 14:51 |
| *** lbragstad has joined #openstack-infra | 14:52 | |
| tushar | yes I am using the latest project-config, https://pastebin.com/bMCC9xxP | 14:53 |
| pabelanger | sorry, jenkins-slave element | 14:54 |
| tushar | puppet is not here under element list | 14:54 |
| tushar | yes jenkins-slave is there | 14:54 |
| pabelanger | okay, jenkins user should be created there | 14:54 |
| *** mdrabe has joined #openstack-infra | 14:54 | |
| pabelanger | if you PB your full log, we can see what is going on | 14:54 |
| openstackgerrit | Merged openstack-infra/system-config master: Add Debian Jessie to ceph mirror list https://review.openstack.org/452547 | 14:55 |
| clarkb | sdague: and lookingat failures for that job they are distributed across osic, internap, citycloud, and rax for the most part. So I don't think there is systemic citycloud failures there? | 14:55 |
| tushar | Pabelanger : If I am runnig command - chown -R jenkins:jenkins /home/jenkins manually then its executing without error, but while creating img its failing | 14:56 |
| pabelanger | tushar: the build log will explain why it is failing. Did the jenkins user get created? | 14:57 |
| *** marst has joined #openstack-infra | 14:57 | |
| sdague | clarkb: ok, cool | 14:57 |
| openstackgerrit | Gergely Csatari proposed openstack-infra/project-config master: Adding Vitrage install guide https://review.openstack.org/471786 | 14:57 |
| tushar | Pablenger : yes jenkins user is created | 14:58 |
| *** rloo has quit IRC | 14:59 | |
| *** mat128 has quit IRC | 15:00 | |
| openstackgerrit | Hongbin Lu proposed openstack-infra/project-config master: [Zun] Remove nova-docker non-voting job https://review.openstack.org/470424 | 15:00 |
| *** jmorgan1 has quit IRC | 15:00 | |
| pabelanger | tushar: please post the build.log | 15:01 |
| *** isq has quit IRC | 15:01 | |
| *** psachin has quit IRC | 15:01 | |
| *** jmorgan1 has joined #openstack-infra | 15:02 | |
| pabelanger | clarkb: fungi: jeblair: do you mind reviewing https://review.openstack.org/#/c/471450/ again. Fixes comment about zookeeper firewall port | 15:04 |
| *** jamesdenton has quit IRC | 15:04 | |
| *** ralonsoh_ has joined #openstack-infra | 15:04 | |
| *** rossella__ has joined #openstack-infra | 15:06 | |
| *** rbrndt has joined #openstack-infra | 15:06 | |
| *** ralonsoh has quit IRC | 15:07 | |
| *** ralonsoh_ is now known as ralonsoh | 15:07 | |
| clarkb | pabelanger: question about that. I thought we still execute jobs over gearman. Doesn't that mean ze01 will need to be in the gearman access list too? | 15:07 |
| clarkb | (thats why I thought commit message was typo, not realizing you really wanted it in the zk list too) | 15:07 |
| *** gomarivera has quit IRC | 15:08 | |
| pabelanger | clarkb: Hmm, I believe you are correct. jeblair can you confirm? | 15:11 |
| *** LindaWang has quit IRC | 15:11 | |
| pabelanger | in that case, we'd need to allow access to zuulv3.o.o for executors | 15:11 |
| *** rossella__ has quit IRC | 15:11 | |
| *** rcernin has quit IRC | 15:11 | |
| openstackgerrit | Modestas Vainius proposed openstack-infra/jenkins-job-builder master: Support taking slack token from Jenkins credentials. https://review.openstack.org/471415 | 15:12 |
| tushar | Pabelanger : build log - http://paste.openstack.org/show/611731/ | 15:12 |
| pabelanger | tushar: looks like paste.o.o truncated it | 15:13 |
| *** xarses has joined #openstack-infra | 15:14 | |
| *** xarses has quit IRC | 15:15 | |
| *** xarses has joined #openstack-infra | 15:15 | |
| *** felipemonteiro_ has quit IRC | 15:15 | |
| johnsom | Would anyone have any ideas as to why the updated requirements bot is spaming neutron-lbaas-dashboard with updates against an old commit that are already merged on master? https://review.openstack.org/#/q/openstack/neutron-lbaas-dashboard+status:open,n,z | 15:18 |
| johnsom | I just don't get why it's using an old commit and not head on master | 15:18 |
| tushar | Pabelanger : not allowing me to paste log file having big size | 15:20 |
| tushar | wait I am trying to paste using other tool | 15:20 |
| *** dave-mccowan has joined #openstack-infra | 15:21 | |
| clarkb | tushar: github's gist service is usually good about file sizees | 15:21 |
| *** salv-orlando has joined #openstack-infra | 15:22 | |
| jeblair | pabelanger, clarkb: oh, yeah, gearman, not zk. | 15:22 |
| tushar | clark : yes tried the same, thanks | 15:23 |
| pabelanger | kk, let me rework | 15:23 |
| tushar | Pabelanger : https://gist.github.com/anonymous/053c5ec1c8373a6fc20e44a15bfe6637 | 15:23 |
| *** jamesdenton has joined #openstack-infra | 15:23 | |
| *** rossella__ has joined #openstack-infra | 15:24 | |
| johnsom | install.d/95-chown-jenkins | 15:25 |
| johnsom | chown: invalid user: 'jenkins:jenkins' | 15:25 |
| johnsom | Looks like the element expects a jenkins user and group to already exist inside the image | 15:25 |
| fungi | could there be an ordering issue/missing dependency expression with the element? | 15:25 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: WIP Make log streaming to executor per-task https://review.openstack.org/471799 | 15:26 |
| fungi | johnsom: not sure yet, but the log from the latest proposal is at http://logs.openstack.org/b5/b50dc796db882a24abb4c326c79089d74c71deed/post/propose-requirements-updates/bddbe65/console.html#_2017-06-07_13_07_15_246164 | 15:27 |
| johnsom | Late to the party so not sure what the image should have, but if you expect jenkins to be there I bet there is a missing element dependency | 15:27 |
| *** chlong has joined #openstack-infra | 15:27 | |
| johnsom | Cool, thanks for the link, I will have a look | 15:28 |
| pabelanger | tushar: you're elements are not correct they are outdated | 15:28 |
| *** prometheanfire has left #openstack-infra | 15:28 | |
| pabelanger | tushar: you'll need to import the lastest elements from openstack-infra/project-config in nodepool/elements folder | 15:28 |
| tushar | Pabelanger : yeh , already downloded the latest elements | 15:29 |
| fungi | johnsom: trying to digest that log, i think the latest one may have been proposed because the job got confused by there being more than one open when it expects only one or none. might make more sense for me to dig up the previous one and see what caused the second to be created | 15:29 |
| openstackgerrit | Yan Xing an proposed openstack-infra/project-config master: Add barbican to tacker localrc for jenkins job https://review.openstack.org/471780 | 15:30 |
| johnsom | fungi We had like seven yesterday, so abandoned a bunch and merged one (just to see) to clean it out, but overnight three more came in.... | 15:30 |
| *** dhajare|busy has quit IRC | 15:31 | |
| fungi | johnsom: oh, i wonder if someone introduced a regression into that script | 15:31 |
| *** dteselkin has quit IRC | 15:31 | |
| *** dteselkin has joined #openstack-infra | 15:34 | |
| *** rderose has joined #openstack-infra | 15:34 | |
| isaacb | infra-cores: can anyone look in https://review.openstack.org/#/c/421253/ | 15:38 |
| *** rderose has left #openstack-infra | 15:39 | |
| *** gyee has joined #openstack-infra | 15:41 | |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Remove ansible files at startup before copy https://review.openstack.org/471518 | 15:42 |
| pabelanger | tushar: you are missing jenkins-slave and still have puppet elements in the build.log | 15:42 |
| *** dizquierdo has joined #openstack-infra | 15:42 | |
| *** rossella__ has quit IRC | 15:43 | |
| *** panda is now known as panda|bbl | 15:45 | |
| tushar | Pabelanger : jenkins-slave is present in /etc/nodepool/elements | 15:45 |
| pabelanger | tushar: it is missing from your gist, so something is happening | 15:46 |
| pabelanger | 2017-06-07 06:10:34,208 INFO nodepool.image.build.dpc: Expanded element dependencies to: install-bin dib-run-parts simple-init vm manifests puppet growroot source-repositories package-installs dkms dib-python infra-package-needs initialize-urandom dib-init-system install-types install-static openssh-server nodepool-base cache-devstack openstack-repos ubuntu bootloader dpkg runtime-ssh-host-keys | 15:46 |
| pabelanger | pip-and-virtualenv ubuntu-common base pkg-map cache-url cloud-init-datasources | 15:46 |
| pabelanger | puppet is still there, that can be removed | 15:46 |
| *** felipemonteiro has joined #openstack-infra | 15:47 | |
| *** felipemonteiro_ has joined #openstack-infra | 15:48 | |
| johnsom | fungi I think I figured it out | 15:49 |
| tushar | Pabelanger : okay. got your point | 15:49 |
| fungi | johnsom: do tell. i'm still staring at earlier logs trying to spot the issue | 15:49 |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Allow ze01.o.o access to gearman / add to cacti.o.o https://review.openstack.org/471450 | 15:49 |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Disable webui for zuul-executors https://review.openstack.org/471458 | 15:49 |
| johnsom | fungi When we duplicated neutron-lbaas-dashboard over to octavia-dashboard for the split out of neutron the .gitreview didn't get updated | 15:49 |
| fungi | johnsom: ohhh... | 15:50 |
| *** rkukura has quit IRC | 15:50 | |
| johnsom | fungi So it's pushing the patch for octavia-dashboard into neutron-lbaas-dashboard | 15:50 |
| fungi | and so it's proposing changes to the wrong repo! | 15:50 |
| fungi | that explains why 1. it's not seeing the correct requirements lists, and 2. it's not finding an existing proposal to update | 15:50 |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Allow ze01.o.o access to gearman / add to cacti.o.o https://review.openstack.org/471450 | 15:51 |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Disable webui for zuul-executors https://review.openstack.org/471458 | 15:51 |
| johnsom | Yep, the logs showed too many hits for neutron-lbaas-dashboard commits | 15:51 |
| fungi | johnsom: here it is! i should have searched through more of the log file :/ | 15:51 |
| fungi | http://logs.openstack.org/1e/1e0d149bbb115f388b3c0904d1aefee531df633d/post/propose-requirements-updates/a3d57fd/console.html#_2017-06-07_11_55_57_615890 | 15:51 |
| fungi | confirmed your suspicion | 15:52 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Fix race in test_crd_gate_unknown https://review.openstack.org/471833 | 15:52 |
| johnsom | fungi Yep, thanks for the link to the log! | 15:52 |
| *** felipemonteiro has quit IRC | 15:52 | |
| fungi | it's doing project:openstack/octavia-dashboard and then creates a gerrit remote for neutron-lbass-dashboard and queries that | 15:52 |
| fungi | glad that mystery's solved. i was worried we had a regression somewhere in one of the scripts i wasn't able to track down | 15:53 |
| *** Rockyg has joined #openstack-infra | 15:53 | |
| *** rossella__ has joined #openstack-infra | 15:54 | |
| *** jascott1 has joined #openstack-infra | 15:54 | |
| openstackgerrit | Martin André proposed openstack-infra/tripleo-ci master: Workaround for limited disk space on undercloud https://review.openstack.org/471836 | 15:55 |
| *** gomarivera has joined #openstack-infra | 15:57 | |
| *** hieulq_ has joined #openstack-infra | 15:57 | |
| openstackgerrit | Hongbin Lu proposed openstack-infra/project-config master: [Zun] Move etcd job to experiemental https://review.openstack.org/471840 | 15:57 |
| fungi | i guess we could add a sanity check to skip any repo where the gerrit remote doesn't match 'ssh://proposal-bot@review.openstack.org:29418/$PROJECT.git$' | 15:57 |
| *** Sukhdev has joined #openstack-infra | 15:58 | |
| *** Apoorva has joined #openstack-infra | 15:59 | |
| *** sambetts is now known as sambetts|afk | 16:00 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Reject some untrusted config objects https://review.openstack.org/470442 | 16:01 |
| openstackgerrit | Hongbin Lu proposed openstack-infra/project-config master: [Zun] Turn dsvm job to voting https://review.openstack.org/471842 | 16:02 |
| *** isaacb has quit IRC | 16:02 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Factor out duplicate code in executor repo tests https://review.openstack.org/471492 | 16:02 |
| *** dave-mccowan has quit IRC | 16:05 | |
| *** david-lyle has joined #openstack-infra | 16:07 | |
| openstackgerrit | Matt Riedemann proposed openstack-infra/project-config master: Run gate-tempest-dsvm-neutron-multinode-full in devstack check https://review.openstack.org/471016 | 16:08 |
| mriedem | sdague: ^ there you go | 16:08 |
| mriedem | not sure if the trusty job needed to be in there, but assumed they need to move together | 16:08 |
| openstackgerrit | Matt Riedemann proposed openstack-infra/project-config master: Run gate-tempest-dsvm-neutron-multinode-full in devstack check https://review.openstack.org/471016 | 16:08 |
| *** esikachev has quit IRC | 16:09 | |
| *** markus_z has quit IRC | 16:10 | |
| *** jamesdenton has quit IRC | 16:11 | |
| *** isaacb has joined #openstack-infra | 16:12 | |
| *** esikachev has joined #openstack-infra | 16:13 | |
| pabelanger | clarkb: jeblair: okay, I have updated 471450 to use gearman on zuulv3.o.o | 16:14 |
| *** florianf has quit IRC | 16:17 | |
| dhellmann | fungi : asettle and I are working on a docs spec to migrate a bunch of the content out of openstack-manuals and into the project repos | 16:17 |
| dhellmann | part of that will involve changes to the doc builds, and the publishing URLs | 16:17 |
| *** esikachev has quit IRC | 16:18 | |
| dhellmann | we're going to get andreas to walk through it with us, but I wonder if you want an infra spec for the related changes, too, or if we can use the docs spec to discuss those parts? | 16:18 |
| fungi | dhellmann: if there's a docs spec, that's plenty sufficient i expect | 16:19 |
| dhellmann | the end goal is to have 1 doc job per repo and have it publish all of the docs out of one sphinx build to one place like docs.o.o/$name | 16:19 |
| dhellmann | and do away with /developer, /releasenotes, etc. | 16:19 |
| dhellmann | that would be phased in, since the most pressing part is getting things like the install and config reference guides out of openstack-manuals | 16:19 |
| fungi | dhellmann: it should end up being pretty straightforward, unless there are aspects of it i'm missing. but happy to review the docs spec | 16:19 |
| *** esikachev has joined #openstack-infra | 16:19 | |
| dhellmann | ok, when we have that up I'll make sure we point it out to you | 16:20 |
| jeblair | pabelanger: small -1 on that | 16:20 |
| fungi | thanks dhellmann! | 16:20 |
| pabelanger | jeblair: ack, thanks | 16:20 |
| dhellmann | there are some knock-on effects from the move to translations and the "archiving" plans but if we uncover huge issues in the docs spec I can work with asettle on a separate infra spec | 16:20 |
| dhellmann | thanks, fungi | 16:20 |
| jeblair | dhellmann: do you have a vague hand-wavey timeline for this yet? | 16:21 |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Allow ze01.o.o access to gearman / add to cacti.o.o https://review.openstack.org/471450 | 16:21 |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Disable webui for zuul-executors https://review.openstack.org/471458 | 16:21 |
| asettle | Thanks fungi :) | 16:21 |
| asettle | Thansk dhq | 16:21 |
| asettle | dhellmann: | 16:21 |
| asettle | Frick | 16:21 |
| asettle | Typing is hard | 16:21 |
| asettle | I'm leaving | 16:21 |
| pabelanger | jeblair: updated | 16:21 |
| fungi | asettle: we no longer kick people out of the channel for typos ;) | 16:21 |
| dhellmann | jeblair : I'm working on a new job-template this afternoon, but it's likely to be next week before that's ready. We'll have the doc spec up soon (asettle?) too | 16:22 |
| pabelanger | thank goodness | 16:22 |
| * fungi notes that the channel would be empty if we did | 16:22 | |
| asettle | fungi: oh jeez I should have been kicked out ages ago | 16:22 |
| asettle | dhellmann: yep that's the hope. Depends on a few reviews, of course | 16:22 |
| dhellmann | jeblair : ideally we'd get the moving done before the end of this cycle, and the consolidate the rest of the builds like releasenotes and api-ref | 16:22 |
| jeblair | dhellmann: that helps, thanks. :) | 16:23 |
| dhellmann | though that's a fairly aggressive schedule, so we'll see | 16:23 |
| jeblair | pabelanger, clarkb: lgtm | 16:23 |
| dhellmann | asettle : ok, it sounds like we can proceed without a separate infra spec for the time being | 16:24 |
| *** esikachev has quit IRC | 16:24 | |
| *** esikachev has joined #openstack-infra | 16:25 | |
| asettle | \o/ huzzah | 16:27 |
| asettle | LIttle win | 16:27 |
| *** mat128 has joined #openstack-infra | 16:27 | |
| fungi | we're usually pretty flexible about not requiring specs unnecessarily, and this doesn't sound likely to alter the publishing framework we're using, just some adjustments to job definitions and maybe redirects in the vhost config if redirects you've been doing with .htaccess files previously aren't still preferred | 16:29 |
| *** esikachev has quit IRC | 16:29 | |
| *** mat128 has quit IRC | 16:29 | |
| *** trown is now known as trown|lunch | 16:30 | |
| *** vhosakot has joined #openstack-infra | 16:31 | |
| *** tesseract has quit IRC | 16:34 | |
| dhellmann | yep, we're trying to make this as low impact as possible | 16:35 |
| dhellmann | if I have a job-group in jenkins/jobs/somefile.yaml with a "branch" setting, does that restrict the jobs to patches on that branch? or do I have to also do something in the layout file to tell zuul to only run the job for a given branch? | 16:36 |
| dhellmann | I know I can do the latter, I'm just not clear if I have to | 16:36 |
| *** jpich has quit IRC | 16:36 | |
| *** jkilpatr has quit IRC | 16:36 | |
| dhellmann | for example translation-jobs-newton in translation-jobs.yaml has "branch: stable/newton" and I don't see any additional settings for {name}-propose-translation-update-newton in layout.yaml except where it's added to the template translation-jobs-newton | 16:37 |
| *** rossella__ has quit IRC | 16:38 | |
| *** lucasagomes is now known as lucas-afk | 16:38 | |
| fungi | dhellmann: you can restrict jobs (by name or regex match on their name) to only run on specific branches (by name or regex match on/excluding the branch names) | 16:39 |
| fungi | dhellmann: but you do that in zuul/layout.yaml | 16:39 |
| dhellmann | ok. I don't see how that's being done for that translation job | 16:39 |
| dhellmann | oh, it's a periodic job | 16:40 |
| dhellmann | hrm | 16:40 |
| fungi | dhellmann: is it a periodic job? in that case the job isn't triggered by changes or commits merging, but just on a schedule | 16:40 |
| fungi | yeah, that | 16:40 |
| dhellmann | yeah | 16:40 |
| dhellmann | maybe you can give me an example to look at if I describe what I want | 16:40 |
| dhellmann | I need a job in the post queue to publish updates to the docs | 16:40 |
| *** jkilpatr has joined #openstack-infra | 16:40 | |
| *** ramishra has quit IRC | 16:41 | |
| fungi | for periodic jobs, since there's no change or commit context from qhich to infer a branch, it gets hard-coded into the job | 16:41 |
| dhellmann | and the job needs to publish to a different location based on the branch | 16:41 |
| *** esikachev has joined #openstack-infra | 16:41 | |
| dhellmann | yeah, that makes sense now that I see that it's a periodic job | 16:41 |
| dhellmann | for master, we want to publish to /$name/latest | 16:41 |
| fungi | for your post job, could you just infer the branch from the envvars? | 16:41 |
| dhellmann | and for stable/$foo we want to publish to /$name/$foo | 16:41 |
| *** Sukhdev has quit IRC | 16:42 | |
| dhellmann | looking at openstack-publish-jobs.yaml I see {name}-docs-{node} which is what is publishing docs now to /developer | 16:42 |
| fungi | you can made decisions on $ZUUL_REFNAME | 16:42 |
| dhellmann | I was going to make a copy of that with a new name and adjust it, but I don't know how to get the branch-specific bit into the target value | 16:42 |
| dhellmann | for the afs publisher | 16:42 |
| *** dave-mccowan has joined #openstack-infra | 16:42 | |
| fungi | we already have plenty of jobs making decisions based on that envvar's value: http://codesearch.openstack.org/?q=ZUUL_REFNAME | 16:44 |
| dhellmann | that will be useful for passing the value to the actual build, but I was worried about the publisher step | 16:44 |
| dhellmann | I mean, I guess I can write a new publisher, but I was hoping to use the afs publisher as-is | 16:45 |
| dhellmann | I suppose I could have 2 jobs, one for use on master and one for use on a stable branch, and use the suffix pattern that I see elsewhere | 16:45 |
| *** ediardo has quit IRC | 16:45 | |
| fungi | oh, i get you... trying to adjust parameters going to the scp publisher | 16:46 |
| dhellmann | yeah | 16:46 |
| dhellmann | well, it says "afs" but I don't know how that's implemented | 16:46 |
| fungi | you can embed job envvars in them, but trying to munge those in place is probably harder | 16:46 |
| dhellmann | there's a "target" parameter that appears to be the output directory name | 16:46 |
| dhellmann | yeah, I think defining 2 templates will be simpler | 16:46 |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Remove ansible files at startup before copy https://review.openstack.org/471518 | 16:47 |
| fungi | right, sorry, the afs publisher but same story basically | 16:47 |
| dhellmann | right | 16:47 |
| *** esikachev has quit IRC | 16:47 | |
| dhellmann | I'm trying to be accurate because I'm re-figuring this stuff out and don't want my misspeaking to lead to bad help from you | 16:48 |
| *** flepied has quit IRC | 16:48 | |
| dhellmann | I think I see what to do now, though | 16:48 |
| jeblair | yeah, because of the difficulty in passing that kind of info to "jenkins", most of the publishing jobs set up the directory tree locally, then publish to some slightly more common root. so the job put the output in "target/master" or "target/stable/foo" as appropriate, and then publish "target/" to wherever. | 16:48 |
| *** rbrndt has quit IRC | 16:49 | |
| jeblair | putting a "root stamp" file in target/master or target/stable/foo helps the afs publisher know exactly what to rsync | 16:49 |
| fungi | http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/infra-publications.yaml is an example | 16:49 |
| fungi | it publishes different paths depending on the branch name | 16:49 |
| fungi | in fact, that's probably almost exactly what you're describing needing | 16:50 |
| jeblair | oh yeah, root-marker is what we call them | 16:50 |
| dhellmann | is it? | 16:50 |
| jeblair | fungi: ++ that looks pretty succinct | 16:50 |
| *** Swami has joined #openstack-infra | 16:50 | |
| dhellmann | literally the only thing I want to change about the current job is to publish to {name}/latest instead of developer/{name} | 16:50 |
| dhellmann | and then I need another job for stable branches that publishes to {name}/{branch} | 16:51 |
| *** salv-orlando has quit IRC | 16:51 | |
| dhellmann | except not stable/newton just newton for the branch | 16:51 |
| jeblair | dhellmann: yeah, you can do that all in one job | 16:51 |
| dhellmann | I think all I need to change is this line? http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/openstack-publish-jobs.yaml#n17 | 16:52 |
| dhellmann | oh, so you build to a different directory and do it that way? | 16:52 |
| *** rbrndt has joined #openstack-infra | 16:52 | |
| * dhellmann smacks forehead | 16:52 | |
| fungi | dhellmann: right, so for the publications repo we have different documents in different branches. the infra-publications-publish job puts content in the appropriate subtree for whatever change merged based on the branch name in $ZUUL_REFNAME and then the infra-publications-publish-index job updates a top-level index based on a script in the master branch | 16:52 |
| jeblair | dhellmann: yeah, target: {name}, then in the jobs, build ".../latest" or ".../{branch}" | 16:53 |
| jeblair | dhellmann: (at least in broad strokes; i haven't double checked these paths) | 16:53 |
| dhellmann | so I would run the sphinx build like normal, the move doc/build/html to something like doc/publish/latest or doc/publish/$NAME and then set source to doc/build/publish/** and remove-prefix to doc/build/publish ? | 16:53 |
| *** rkukura has joined #openstack-infra | 16:54 | |
| jeblair | dhellmann: yep | 16:54 |
| dhellmann | clever | 16:54 |
| *** rossella__ has joined #openstack-infra | 16:55 | |
| fungi | and as mentioned previously, creating .root-marker files is critical under each of those subtrees because the afs publisher is smart enough to remove old content your job no longer publishes and looks for those markers to let it know what it should or shouldn't delete | 16:55 |
| jeblair | yeah, let me dig up some further resources... | 16:55 |
| dhellmann | what goes into a .root-marker file? | 16:55 |
| *** derekh has quit IRC | 16:56 | |
| dhellmann | oh, I see there's an add-docs-root-marker build stpe | 16:56 |
| dhellmann | step | 16:56 |
| fungi | dhellmann: http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/macros.yaml#n320 | 16:56 |
| *** ltomasbo is now known as ltomasbo|away | 16:56 | |
| jeblair | dhellmann: content doesn't matter (it's mostly for human debugging). the presence of the file is key. | 16:56 |
| jeblair | dhellmann: http://specs.openstack.org/openstack-infra/infra-specs/specs/doc-publishing.html#rsync-process | 16:56 |
| jeblair | dhellmann: that may help explain how the rsync happens and what the root-markers are for | 16:57 |
| dhellmann | yes, I'll read through all of that | 16:57 |
| fungi | right, it basically says "oh, there's a root marker in this subtree, i should assume it belongs to another job and not blow it away" | 16:57 |
| dhellmann | cool, thank you both, this is going to be a bit easier but also more complicated than I expected :-) | 16:57 |
| *** e0ne has quit IRC | 16:57 | |
| jeblair | dhellmann: other things that may be handy: http://files.openstack.org/ will let you browse the afs tree easily, however, i don't know how to tell it not to do autoindex | 16:57 |
| jeblair | dhellmann: for very easy debugging and exploration, you can install the afs client locally and browse the whole tree: https://docs.openstack.org/infra/system-config/afs.html#client-configuration | 16:58 |
| dhellmann | slick | 16:59 |
| dhellmann | ok, I'm going to grab some lunch and then sit down and read those docs before I go any further with this, but I think I have a good picture of what I need to do now | 16:59 |
| dhellmann | thank you both again, fungi & jeblair | 16:59 |
| fungi | any time! | 17:00 |
| jeblair | np | 17:01 |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Remove ansible files at startup before copy https://review.openstack.org/471518 | 17:04 |
| *** ralonsoh has quit IRC | 17:06 | |
| *** yamamoto has joined #openstack-infra | 17:06 | |
| *** isaacb has quit IRC | 17:08 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul feature/zuulv3: WIP: add log streaming test https://review.openstack.org/471079 | 17:08 |
| openstackgerrit | Kendall Nelson proposed openstack-infra/infra-specs master: Update Task Tracker Spec https://review.openstack.org/471859 | 17:08 |
| pabelanger | clarkb: fungi: do you mind a +3 on https://review.openstack.org/#/c/471450/ should be last step for new ze01.o.o server | 17:10 |
| *** yamamoto has quit IRC | 17:11 | |
| *** salv-orlando has joined #openstack-infra | 17:14 | |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder master: Builders: conditional-step fix https://review.openstack.org/456684 | 17:14 |
| openstackgerrit | Luigi Toscano proposed openstack-infra/project-config master: Gerritbot: track current branches for sahara repositories https://review.openstack.org/471860 | 17:15 |
| *** esikachev has joined #openstack-infra | 17:15 | |
| *** pbourke has quit IRC | 17:17 | |
| *** shardy has quit IRC | 17:18 | |
| openstackgerrit | Merged openstack-infra/nodepool feature/zuulv3: Allow loading logging config from yaml https://review.openstack.org/445656 | 17:18 |
| *** Rockyg has quit IRC | 17:20 | |
| openstackgerrit | Merged openstack-infra/system-config master: Allow ze01.o.o access to gearman / add to cacti.o.o https://review.openstack.org/471450 | 17:20 |
| *** jpena is now known as jpena|off | 17:21 | |
| *** wolverineav has quit IRC | 17:26 | |
| *** mat128 has joined #openstack-infra | 17:27 | |
| *** rcernin has joined #openstack-infra | 17:30 | |
| *** wolverineav has joined #openstack-infra | 17:32 | |
| *** efoley has quit IRC | 17:36 | |
| openstackgerrit | Yolande Amate proposed openstack-infra/jenkins-job-builder master: Update GitLab Plugin to make use of convert_mapping_to_xml https://review.openstack.org/471867 | 17:37 |
| *** jamesdenton has joined #openstack-infra | 17:37 | |
| *** trown|lunch is now known as trown | 17:37 | |
| *** esikachev has quit IRC | 17:38 | |
| *** e0ne has joined #openstack-infra | 17:39 | |
| *** esikachev has joined #openstack-infra | 17:39 | |
| *** rossella__ has quit IRC | 17:41 | |
| *** dhinesh has joined #openstack-infra | 17:43 | |
| *** esikachev has quit IRC | 17:44 | |
| *** rbrndt has quit IRC | 17:45 | |
| *** gardlt has quit IRC | 17:46 | |
| *** gardlt has joined #openstack-infra | 17:47 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/puppet-zuul master: Fix typo with executor logging in zuul.conf https://review.openstack.org/471873 | 17:48 |
| *** wolverineav has quit IRC | 17:50 | |
| *** wolverineav has joined #openstack-infra | 17:50 | |
| *** rbrndt has joined #openstack-infra | 17:50 | |
| *** dtantsur is now known as dtantsur|afk | 17:50 | |
| *** gardlt has quit IRC | 17:51 | |
| *** rbrndt has quit IRC | 17:53 | |
| *** jamesmcarthur has quit IRC | 17:56 | |
| *** esikachev has joined #openstack-infra | 18:01 | |
| *** e0ne has quit IRC | 18:02 | |
| *** electrofelix has quit IRC | 18:02 | |
| *** gardlt has joined #openstack-infra | 18:05 | |
| *** esikachev has quit IRC | 18:06 | |
| *** esikachev has joined #openstack-infra | 18:06 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Add gerrit information to ze01.o.o https://review.openstack.org/471877 | 18:08 |
| *** xyang1 has joined #openstack-infra | 18:08 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/puppet-zuul master: Start executor as root https://review.openstack.org/471879 | 18:10 |
| *** jamesdenton has quit IRC | 18:11 | |
| *** pvaneck has joined #openstack-infra | 18:11 | |
| *** jamesmcarthur has joined #openstack-infra | 18:11 | |
| pabelanger | jeblair: fungi: clarkb: 3 more changes for ze01.o.o we need^ | 18:11 |
| *** esikachev has quit IRC | 18:12 | |
| *** dave-mccowan has quit IRC | 18:12 | |
| *** esikachev has joined #openstack-infra | 18:13 | |
| *** jamesmcarthur has quit IRC | 18:13 | |
| dhellmann | jeblair : the zuul docs say that ZUUL_BRANCH is the "target branch". Does that look like origin/stable/ocata or just stable/ocata? | 18:14 |
| *** jamesmcarthur has joined #openstack-infra | 18:15 | |
| *** jamesmcarthur has quit IRC | 18:15 | |
| *** jamesmcarthur has joined #openstack-infra | 18:16 | |
| *** dave-mccowan has joined #openstack-infra | 18:16 | |
| *** jamesmcarthur has quit IRC | 18:16 | |
| *** jamesmcarthur has joined #openstack-infra | 18:16 | |
| fungi | dhellmann: you can look at a stable branch job's _zuul_ansible/vars.yaml but it's just the bare branch name which came in the gerrit event metadata | 18:17 |
| *** jamesmcarthur has quit IRC | 18:17 | |
| fungi | e.g. master or stable/ocata | 18:17 |
| dhellmann | ok, perfect, thanks | 18:17 |
| *** jamesmcarthur has joined #openstack-infra | 18:17 | |
| *** jamesmcarthur has quit IRC | 18:18 | |
| *** jamesmcarthur has joined #openstack-infra | 18:18 | |
| *** jamesmcarthur has quit IRC | 18:18 | |
| *** jamesmcarthur has joined #openstack-infra | 18:19 | |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config master: define a unified project documentation build job https://review.openstack.org/471881 | 18:19 |
| *** jamesmcarthur has quit IRC | 18:19 | |
| dhellmann | fungi : ^^ | 18:19 |
| *** e0ne has joined #openstack-infra | 18:19 | |
| *** jamesmcarthur has joined #openstack-infra | 18:19 | |
| fungi | dhellmann: example http://logs.openstack.org/47/47774935ac0145f6ccb058163248bfeefef93e8d/post/nova-releasenotes/1fcbd71/_zuul_ansible/vars.yaml | 18:20 |
| *** jamesmcarthur has quit IRC | 18:20 | |
| dhellmann | there's no rush to land that. I'll hold it until we have the spec approved. | 18:20 |
| *** dave-mccowan has quit IRC | 18:20 | |
| *** esikachev has quit IRC | 18:20 | |
| fungi | those are the variables set for a stable/ocata job which ran in the post pipeline | 18:20 |
| *** jamesmcarthur has joined #openstack-infra | 18:20 | |
| *** salv-orlando has quit IRC | 18:21 | |
| *** jamesmcarthur has quit IRC | 18:21 | |
| *** jamesmcarthur has joined #openstack-infra | 18:21 | |
| dhellmann | hmm | 18:21 |
| *** salv-orlando has joined #openstack-infra | 18:21 | |
| dhellmann | branch probably isn't set when a tag is applied, though, is it | 18:21 |
| *** jamesmcarthur has quit IRC | 18:21 | |
| fungi | that's why i was suggesting to use $ZUUL_REFNAME | 18:22 |
| dhellmann | yeah, but I don't want the tag to show up in the url | 18:22 |
| dhellmann | at least I don't think I do | 18:22 |
| fungi | it's how other post pipeline jobs figure out what branch their triggering commit merged on | 18:22 |
| dhellmann | we really only talked about keeping one set of docs per series | 18:22 |
| *** jamesmcarthur has joined #openstack-infra | 18:23 | |
| *** jamesmcarthur has quit IRC | 18:23 | |
| dhellmann | if someone tags on stable/ocata, I want the build to publish to /{name}/ocata | 18:23 |
| fungi | yeah, it sounds like you want slightly different jobs for branch tip and tag | 18:23 |
| fungi | or are you only updating when there's a tag? | 18:24 |
| dhellmann | how icky would it be to look in the .gitreview file for the branch | 18:24 |
| fungi | pretty icky for projects that miss updating their .gitreview files on a per-branch basis | 18:24 |
| dhellmann | I guess that's not canonical | 18:24 |
| dhellmann | yeah | 18:24 |
| *** jamesmcarthur has joined #openstack-infra | 18:24 | |
| dhellmann | I suppose if it's a tag I can do "git branch --contains" | 18:24 |
| fungi | also there's support in git-review for no longer setting defaultbranch and inferring the branch name based on cues from git | 18:25 |
| *** jamesmcarthur has quit IRC | 18:25 | |
| *** jamesmcarthur has joined #openstack-infra | 18:25 | |
| fungi | yeah, that's also an option | 18:25 |
| *** salv-orlando has quit IRC | 18:25 | |
| *** jamesmcarthur has quit IRC | 18:25 | |
| *** esikachev has joined #openstack-infra | 18:26 | |
| *** jamesmcarthur has joined #openstack-infra | 18:26 | |
| fungi | tags don't map to one and only one branch (you can push a tag for a commit which appears in two branches, and even for a commit which is not on a branch at all though that's kinda gross and almost always a mistake) | 18:26 |
| *** jamesmcarthur has quit IRC | 18:26 | |
| *** jamesmcarthur has joined #openstack-infra | 18:27 | |
| *** prometheanfire has joined #openstack-infra | 18:27 | |
| fungi | so you need some logic to decide how you handle that anyway. zuul definitely can't tell you | 18:27 |
| dhellmann | yeah | 18:27 |
| *** jamesmcarthur has quit IRC | 18:27 | |
| dhellmann | if I get more than one branch, I guess I can copy to both locations | 18:27 |
| prometheanfire | is mitaka gate broken? https://review.openstack.org/471744 and https://review.openstack.org/471169 for example | 18:27 |
| dhellmann | that would come up when we tag and branch at the same time, when we first create the stable branch | 18:28 |
| fungi | prometheanfire: probably. i think once we reached scheduled eol people stopped maintaining it | 18:28 |
| dhellmann | or I guess I could only use the stable branch, since we're more likely to land another commit on master | 18:28 |
| prometheanfire | that's what I thought | 18:28 |
| fungi | prometheanfire: and the vmt has stopped considering mitaka supported as of the scheduled eol date too | 18:28 |
| prometheanfire | we only have those two reviews | 18:28 |
| *** jamesmcarthur has joined #openstack-infra | 18:28 | |
| prometheanfire | should we just abandon them? | 18:28 |
| *** jamesmcarthur has quit IRC | 18:28 | |
| prometheanfire | dhellmann: how are releases handled for things like that? (after EOL) | 18:29 |
| *** jamesmcarthur has joined #openstack-infra | 18:29 | |
| dhellmann | prometheanfire : yeah, I think it's ok to abandon those. | 18:29 |
| fungi | prometheanfire: sure i would just abandon. i mean confer with the stable team but i think we're just treating stable/mitaka as eol even though the stable team lacks teh resources to do the work to close down the branches for now | 18:29 |
| *** jamesmcarthur has quit IRC | 18:29 | |
| dhellmann | right | 18:30 |
| *** esikachev has quit IRC | 18:30 | |
| prometheanfire | I can't abandon | 18:30 |
| prometheanfire | dhellmann: mind doing the honors :P | 18:30 |
| *** jamesmcarthur has joined #openstack-infra | 18:30 | |
| prometheanfire | that is, unless we get https://review.openstack.org/470419 | 18:31 |
| *** Apoorva_ has joined #openstack-infra | 18:31 | |
| *** jamesmcarthur has quit IRC | 18:31 | |
| dhellmann | done | 18:31 |
| *** jamesmcarthur has joined #openstack-infra | 18:31 | |
| *** esikachev has joined #openstack-infra | 18:31 | |
| *** hieulq_ has quit IRC | 18:31 | |
| *** jamesmcarthur has quit IRC | 18:32 | |
| prometheanfire | thanks | 18:32 |
| *** jamesmcarthur has joined #openstack-infra | 18:33 | |
| *** jamesmcarthur has quit IRC | 18:33 | |
| *** jamesmcarthur has joined #openstack-infra | 18:34 | |
| *** Apoorva has quit IRC | 18:34 | |
| *** jamesmcarthur has quit IRC | 18:34 | |
| *** dizquierdo has quit IRC | 18:34 | |
| *** jamesmcarthur has joined #openstack-infra | 18:35 | |
| *** jamesmcarthur has quit IRC | 18:36 | |
| *** jamesmcarthur has joined #openstack-infra | 18:36 | |
| *** esikachev has quit IRC | 18:36 | |
| *** jamesmcarthur has quit IRC | 18:36 | |
| *** Sukhdev has joined #openstack-infra | 18:37 | |
| *** jamesmcarthur has joined #openstack-infra | 18:37 | |
| *** jamesmcarthur has quit IRC | 18:38 | |
| openstackgerrit | Matthew Thode proposed openstack-infra/project-config master: Create the requirements stable maintence team https://review.openstack.org/470419 | 18:38 |
| *** jamesmcarthur has joined #openstack-infra | 18:38 | |
| prometheanfire | dhellmann: dims dirk ^ | 18:39 |
| dhellmann | prometheanfire : lgtm | 18:40 |
| prometheanfire | neat | 18:40 |
| clarkb | pabelanger: +2'd | 18:40 |
| *** trown is now known as trown|mtg | 18:41 | |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config master: define a unified project documentation build job https://review.openstack.org/471881 | 18:42 |
| *** esikachev has joined #openstack-infra | 18:42 | |
| dirk | prometheanfire: thanks | 18:42 |
| *** jamesmcarthur has quit IRC | 18:43 | |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config master: define a unified project documentation build job https://review.openstack.org/471881 | 18:44 |
| *** Swami has quit IRC | 18:46 | |
| *** esikachev has quit IRC | 18:47 | |
| lbragstad | hey folks - curious if i could get a second set of eyes on https://review.openstack.org/#/c/471419/ whenever possible, thanks in advance! | 18:48 |
| *** rbrndt has joined #openstack-infra | 18:48 | |
| *** esikachev has joined #openstack-infra | 18:49 | |
| *** gardlt has quit IRC | 18:49 | |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config master: define a unified project documentation build job https://review.openstack.org/471881 | 18:50 |
| *** gardlt has joined #openstack-infra | 18:50 | |
| *** gardlt has quit IRC | 18:51 | |
| dhellmann | fungi : ok, I think ^^ is ready for review now | 18:51 |
| *** gardlt has joined #openstack-infra | 18:51 | |
| *** jamesmcarthur has joined #openstack-infra | 18:53 | |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Remove ansible files at startup before copy https://review.openstack.org/471518 | 18:53 |
| *** amoralej is now known as amoralej|off | 18:54 | |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config master: define a unified project documentation build job https://review.openstack.org/471881 | 18:56 |
| *** esikachev has quit IRC | 18:57 | |
| openstackgerrit | Adam Harwell proposed openstack-infra/project-config master: Octavia: dashboard jobs need to be slightly normalized https://review.openstack.org/471887 | 18:58 |
| *** Kaiyan has joined #openstack-infra | 18:59 | |
| *** esikachev has joined #openstack-infra | 19:04 | |
| clarkb | dhellmann: dtroyer I've cornered greghaynes and we are looking into osc slowness more. And it looks like table formatting in cliff is a big part of the cost now too | 19:05 |
| clarkb | dhellmann: dtroyer: specifically in determining terminal width and associated formatting. Using the json output is significantly faster | 19:06 |
| dhellmann | do we use table formatting in devstack? | 19:06 |
| clarkb | so two things, we may want to further profile cliff and try to reduce that cost because people use this on their terminal. And second maybe devstack should use json output? | 19:06 |
| clarkb | dhellmann: I think so, I don't see it passing -f when using osc | 19:06 |
| *** prometheanfire has left #openstack-infra | 19:06 | |
| dhellmann | ah | 19:06 |
| dhellmann | oh, maybe that's for human log readers | 19:07 |
| dhellmann | I assumed most uses were going to be for parsing output somehow | 19:07 |
| dtroyer | that %$^#$$!$@#$#% terminal width stuff needs to get turned off by default for a couple of reasons, that's good enough for me | 19:07 |
| dhellmann | dtroyer : yep, I'm fine with that | 19:07 |
| clarkb | I'm going to work on a devstack patch to try json output | 19:07 |
| clarkb | or can I just turn off the terminal width stuff in existing osc? | 19:08 |
| dhellmann | clarkb : if you or greghaynes want to propose a patch to cliff to turn off the terminal width stuff by default, dtroyer and I could probably land it this week | 19:08 |
| dtroyer | we're changing our functional test to all use JSON for multiple reasons, I think doing so in devstack is reasonable too | 19:08 |
| clarkb | dtroyer: dhellmann roger, thanks | 19:08 |
| dhellmann | clarkb : thanks for looking into this :-) | 19:08 |
| *** yamamoto has joined #openstack-infra | 19:08 | |
| pabelanger | clarkb: do you mind https://review.openstack.org/#/c/471877/ too. As for /var/lib/zuul/ssh, that has been like that for some time. I don't know why originally, but we'd have to update production servers to change that now | 19:09 |
| *** tinwood has quit IRC | 19:10 | |
| *** thorst_afk has quit IRC | 19:11 | |
| clarkb | dhellmann: dtroyer I think I will likely start with the devstack change as figuring out all the plumbing between cliff and osc will likely take longer | 19:12 |
| dhellmann | clarkb : sounds good. I'll look at the width thing | 19:12 |
| *** yamamoto has quit IRC | 19:12 | |
| *** tinwood has joined #openstack-infra | 19:13 | |
| clarkb | dhellmann: fwiw I used vprof to get a visual sense of where the cost was going | 19:14 |
| openstackgerrit | Merged openstack-infra/infra-specs master: Update Task Tracker Spec https://review.openstack.org/471859 | 19:14 |
| dhellmann | cool, I'll check that out, too | 19:14 |
| *** thorst_afk has joined #openstack-infra | 19:15 | |
| dhellmann | clarkb : it looks like the terminal width stuff only applies if the terminal is a tty. did you test with output redirected to a log? | 19:15 |
| clarkb | dhellmann: I didn't just on my local console | 19:17 |
| clarkb | so we may not hit that in devstack? | 19:17 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Bail with an error on a non-existent jobroot_dir https://review.openstack.org/469524 | 19:17 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Make log streaming to executor per-task https://review.openstack.org/471799 | 19:17 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Strip double-timestamps from output log https://review.openstack.org/471890 | 19:17 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Update log to include expected data on task results https://review.openstack.org/471891 | 19:17 |
| *** rcernin has quit IRC | 19:17 | |
| clarkb | it definitely ran faster locally when I hceked out the width calculations | 19:17 |
| clarkb | *hacked out the width | 19:17 |
| dhellmann | clarkb : ok. that would still apply for devstack in some cases I guess | 19:17 |
| dhellmann | maybe not in the gate, though? | 19:18 |
| clarkb | dhellmann: ya its possible that with all the log redirection we do in the gate it wouldn't affect it there | 19:18 |
| clarkb | but would definitely be a cost for foreground devstacking so should be fixed regardless I think | 19:18 |
| dhellmann | sure | 19:18 |
| fungi | though don't we go to some extreme lengths to fake a tty in our jobs? or maybe that's no longer the case | 19:19 |
| dhellmann | oh, well, if we're doing that then it would help regardless | 19:19 |
| *** thorst_afk has quit IRC | 19:20 | |
| clarkb | also looks like the -f jsob/value/table/etc applies to the subcommand not the high level openstack command like --os-cloud does. Thats a little annoying (notsure what is involved in making it support both | 19:21 |
| *** dave-mccowan has joined #openstack-infra | 19:23 | |
| clarkb | also looks like some of the openstack calls already pass -f value | 19:24 |
| *** wolverineav has quit IRC | 19:24 | |
| *** esikache1 has joined #openstack-infra | 19:24 | |
| dhellmann | not all sub commands support formatters | 19:25 |
| dhellmann | this vprof output is colorful, but I'm not sure how to make sense of the flame graph | 19:25 |
| dtroyer | clarkb: I'm not sure what you mean there, —os-cloud is a global option and produces nothing on its own | 19:25 |
| *** salv-orlando has joined #openstack-infra | 19:26 | |
| dhellmann | dtroyer : he wants to be able to say "openstack -f json whatever" instead of "openstack whatever -f json" | 19:26 |
| dtroyer | only cliff-subclass commands (which all built-in ones are) support the formatting so that's where those options show up | 19:26 |
| dhellmann | right | 19:26 |
| dtroyer | ah, make it global... | 19:26 |
| dhellmann | right | 19:26 |
| dtroyer | uh, yeah, we should look in to that | 19:27 |
| clarkb | ya that | 19:27 |
| dtroyer | except it might play havoc with things | 19:27 |
| dhellmann | s/might/will | 19:27 |
| clarkb | its just easier to think about flags that belong to all the commands in that way than takcing them on to the sub commands (its a minor thing) | 19:27 |
| *** esikachev has quit IRC | 19:27 | |
| dtroyer | 'might' implying that it can be worked around without much pain :) | 19:27 |
| dhellmann | heh | 19:28 |
| *** trown|mtg is now known as trown | 19:28 | |
| *** thorst_afk has joined #openstack-infra | 19:29 | |
| *** flepied has joined #openstack-infra | 19:30 | |
| openstackgerrit | Yolande Amate proposed openstack-infra/jenkins-job-builder master: Update artifactory_repository to make use of convert_mapping_to_xml https://review.openstack.org/471894 | 19:30 |
| *** baoli_ has joined #openstack-infra | 19:31 | |
| *** mat128 has quit IRC | 19:32 | |
| dhellmann | clarkb : what part of the profiler output led you to believe the table width stuff was significant? | 19:33 |
| dhellmann | I see 0.04xx% | 19:33 |
| *** baoli has quit IRC | 19:33 | |
| dhellmann | the pluign loading is way more expensive as far as I can tell | 19:33 |
| *** rcernin has joined #openstack-infra | 19:35 | |
| rm_work | help? https://review.openstack.org/#/c/471611/4 this change is having issues with checks, says the depends-on is failing to merge? but it isn't... | 19:35 |
| rm_work | at least not as far as I can tell | 19:36 |
| clarkb | dhellmann: my flamegraph says 56% is in cliff formatters/table.py | 19:36 |
| fungi | pabelanger: we stopped mirroring npm a couple months ago right? i still see the data in afs... wondering whether/when we should clean that up | 19:36 |
| clarkb | dhellmann: and assign_max_widths is ~31% | 19:36 |
| dhellmann | which sub command did you run? | 19:36 |
| clarkb | dhellmann: csv/value run in about half the time as default (table) | 19:36 |
| clarkb | dhellmann: just catalo list | 19:36 |
| clarkb | *catalog list | 19:36 |
| *** panda|bbl is now known as panda | 19:37 | |
| pabelanger | fungi: yes, we no longer mirror npm. Sure, we can delete that | 19:37 |
| dhellmann | ok, different results for server list | 19:37 |
| pabelanger | fungi: turn down the mirror for it should also be fine, we removed the setup from configure_mirror.sh | 19:38 |
| clarkb | dhellmann: I think the size of the table may be realted there and catalog list is a larger table | 19:38 |
| clarkb | as are commands that emit lots of resources | 19:39 |
| dhellmann | yeah, that would make sense | 19:39 |
| fungi | rm_work: if the change is being tested and the dependency gets a new patchset while those jobs are running, zuul aborts the jobs and reports that the dependency (the particular patchset it started testing) will be unable to merge (because now there's a new patchset). in this case the owner of the dependency keeps updating teh commit message | 19:39 |
| fungi | which creates a new patchset | 19:39 |
| dhellmann | I'm still seeing way more time in pkg_resources than anywhere else | 19:39 |
| dhellmann | although I could also be misreading this graph -- these colors are giving me a headache :-) | 19:39 |
| clarkb | ya its not the best tool, I just found it hard to dig through the pstats from cProfile | 19:40 |
| rm_work | fungi: lol | 19:40 |
| rm_work | fungi: ok thanks | 19:40 |
| dhellmann | definitely | 19:40 |
| fungi | rm_work: i agree it's a mildly confusing behavior if people are iterating on a change and its declared dependencies at roughly the same time. we've talked about maybe having zuul restart testing silently each time that happens instead | 19:40 |
| SpamapS | http://paste.openstack.org/show/611762/ <-- getting weird stuff trying to rebase and submit a stack of patches to feature/zuulv3 on zuul | 19:42 |
| rm_work | i see... yeah | 19:42 |
| rm_work | fungi: well, i told him to cut it out :P thanks | 19:42 |
| *** trown is now known as trown|brb | 19:42 | |
| SpamapS | Received disconnect from 2001:4800:7819:103:be76:4eff:fe05:8525 port 29418:9: sendKexInit() none of the keys appears in supported list: [ssh-dss, ssh-rsa, ecdsa-sha2-nistp256, ecdsa-sha2-nistp384, ecdsa-sha2-nistp521] | 19:42 |
| fungi | rm_work: heh, nice ;) | 19:42 |
| SpamapS | that one seems like the relevant line :-P | 19:42 |
| rm_work | having this merge would be super helpful for us: https://review.openstack.org/#/c/471887/ | 19:43 |
| rm_work | we're trying to unblock our dashboard gates | 19:43 |
| fungi | SpamapS: i got a couple of 5xx errors trying to pull up changes on the webui too, and then they worked when i retried. i wonder if we've got something going wrong on review.o.o just now | 19:43 |
| rm_work | yeah same fungi ^^ | 19:43 |
| rm_work | refresh seemed fine | 19:43 |
| *** flepied has quit IRC | 19:43 | |
| fungi | load average seems normal | 19:44 |
| fungi | nothing frightening in dmesg | 19:44 |
| SpamapS | Unable to negotiate with 2001:4800:7819:103:be76:4eff:fe05:8525 port 29418: no matching host key type found. Their offer: ssh-dss | 19:44 |
| SpamapS | seems like the ssh server is bong | 19:44 |
| rm_work | oh, I forgot to say -- project-config change for octavia: https://review.openstack.org/#/c/471887/ | 19:45 |
| fungi | yeah, that's going to be the java-based "mina-sshd" running inside its jvm | 19:45 |
| fungi | javamelody is failing to give me any graphs at all | 19:45 |
| rm_work | oh also getting 502s fungi SpamapS | 19:46 |
| fungi | seeing this in gerrit's error log http://paste.openstack.org/show/611763/ | 19:47 |
| fungi | "... org.apache.mina.util.DefaultExceptionMonitor : Unexpected exception. java.io.IOException: Too many open files | 19:47 |
| jeblair | wow that's a new one | 19:47 |
| mordred | my favorite error | 19:47 |
| inc0 | lol, but that's just a WARN | 19:48 |
| inc0 | nobody cares about WARN | 19:48 |
| fungi | srsly | 19:48 |
| jeblair | inc0: oh true, that couldn't possibly be related then | 19:48 |
| inc0 | I'm kidding | 19:48 |
| inc0 | :( | 19:48 |
| fungi | half my ssh attempts fail, the other half work | 19:49 |
| jeblair | fungi: anything useful out of melody yet? | 19:49 |
| fungi | trying to use show-connections to look for any offenders | 19:49 |
| fungi | jeblair: still no graphs from javamelody, and reloading the page doesn't seem to help | 19:49 |
| jeblair | fungi: http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=30&rra_id=all is interesting | 19:50 |
| fungi | wow, we have a crazy connection count | 19:50 |
| jeblair | (but as always, could be related to the cause or just another symptom) | 19:50 |
| *** JudeC has joined #openstack-infra | 19:50 | |
| fungi | 240 from "?", 502 from 210.12.16.251, 697 from 113.196.154.248, 1175 from 169.47.209.133 and 2590 from 169.47.209.131 | 19:51 |
| SpamapS | fungi: 169.47.209.131 is softlayer | 19:51 |
| SpamapS | and 133 too | 19:51 |
| fungi | the top two, yeah | 19:51 |
| SpamapS | let me make sure those aren't in BonnyCI's range | 19:51 |
| SpamapS | but I don't think they are | 19:51 |
| SpamapS | k they're not | 19:52 |
| jeblair | fungi: what's the javamelody url? | 19:52 |
| fungi | jeblair: https://review.openstack.org/monitoring | 19:52 |
| fungi | i'm going to iptables block the top four there for the moment while we continue looking | 19:52 |
| jeblair | i have some (not all) graphs. gc time is trending up | 19:53 |
| jeblair | the # open files graph mirrors the tcp connection count | 19:53 |
| SpamapS | It's possible those hosts are overactive in retrying and something else is causing fail deeper | 19:55 |
| jeblair | only other interesting graph i see is active threads, which is unusually high starting around 19:30 | 19:55 |
| jeblair | typically it's 1.0 or less threads. currently 14.0. | 19:55 |
| *** trown|brb is now known as trown | 19:55 | |
| *** jcoufal has quit IRC | 19:56 | |
| Kaiyan | Hi infra team, I need to add "libssl-dev" into bindep.txt, should I add it under the project itself or add to the one in project-config/jenkins/data/bindep-fallback.txt? Thanks! | 19:56 |
| jeblair | Kaiyan: generally the project itself; we're trying to reduce usage of the fallback file | 19:56 |
| Kaiyan | ok! Thanks! | 19:57 |
| pabelanger | Kaiyan: http://lists.openstack.org/pipermail/openstack-dev/2017-June/117951.html should help too | 19:57 |
| * SpamapS is chasing down a lead on who owns those two SL ips. | 19:57 | |
| Kaiyan | Very helpful! Thank you! | 19:58 |
| jeblair | most of the active threads are for requests that have been running for a long time. my guess is that hitting the too many files error has started causing follow-on failures (like transactions which can not commit) | 19:58 |
| fungi | #status log Blocked 169.47.209.131, 169.47.209.133, 113.196.154.248 and 210.12.16.251 in iptables on review.o.o temporarily while debugging excessive connection counts | 19:59 |
| openstackstatus | fungi: finished logging | 19:59 |
| jeblair | probably we should finish identifying the cause, ban all the ips, then restart to try to clean up. | 19:59 |
| jeblair | i wonder if we should git fsck. | 19:59 |
| fungi | i concur. seeing if connection counts are dropping yet | 20:00 |
| smcginnis | Kaiyan: You can probably copy what I did here: https://review.openstack.org/#/c/471502/ | 20:00 |
| jeblair | i can at least identify all the changes with outstanding requests, so we can focus on making sure just those git repos are okay. | 20:00 |
| fungi | should i try to close the open connections for the addresses i blocked? | 20:01 |
| fungi | only one of them seems to be decreasing, and that's going slowly | 20:01 |
| jeblair | fungi: i think that would be fine, but maybe if we've learned all we will, we should just restart now? | 20:01 |
| SpamapS | so, the two SL ips belong to an IBM team, they're investigating. | 20:01 |
| fungi | it's possible the high connection counts are a misbehavior (maybe even in zuul) resulting from some other problem happening in gerrit too | 20:02 |
| *** Sukhdev has quit IRC | 20:02 | |
| *** Kaiyan has quit IRC | 20:03 | |
| jeblair | these are the outstanding requests, if we want to check their git repos later: http://paste.openstack.org/show/611764/ | 20:03 |
| fungi | jeblair: yeah, i'm cool restarting gerrit now. do we want to do any cleanup/checking with the service stopped? | 20:03 |
| fungi | or just restart it as quickly as we can? | 20:03 |
| jeblair | fungi: i vote restart, then perform an rsync and offline git gc on any repos we're concerned about | 20:03 |
| jeblair | (we have at least 9 current backups of all the git repos which should be safe from any corruption) | 20:04 |
| * mordred also votes restart | 20:04 | |
| fungi | #status notice The Gerrit service on review.openstack.org is being restarted now to clear some excessive connection counts while we debug the intermittent request failures reported over the past few minutes | 20:04 |
| openstackstatus | fungi: sending notice | 20:04 |
| fungi | restart underway | 20:04 |
| *** esikache1 has quit IRC | 20:04 | |
| -openstackstatus- NOTICE: The Gerrit service on review.openstack.org is being restarted now to clear some excessive connection counts while we debug the intermittent request failures reported over the past few minutes | 20:05 | |
| fungi | webui seems to be returning again now | 20:06 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Convert some leftover changeish mentions to ref https://review.openstack.org/465718 | 20:06 |
| fungi | (just in time for zuul to merge a change apparently!) | 20:06 |
| jeblair | 210.12.16.251 seems to be a very misconfigured ci system | 20:06 |
| inc0 | fungi: definetly helped | 20:06 |
| jeblair | [2017-06-07 19:58:00,521 +0000] 8c9d2aa7 your-gerrit-user - AUTH FAILURE FROM 210.12.16.251 user-not-found | 20:06 |
| *** esikache1 has joined #openstack-infra | 20:06 | |
| jeblair | (that's a login attempt for "your-gerrit-user") | 20:07 |
| openstackstatus | fungi: finished sending notice | 20:07 |
| SpamapS | confirmed, IBM team was cloning only from review.o.o, they're changing to git.o.o | 20:08 |
| pabelanger | jeblair: looks that way. That seems to be from puppet-openstackci | 20:08 |
| jeblair | unfortunately, i have to run to an appointment now, so i can't help with any corruption checks | 20:08 |
| smcginnis | That IP is somewhere in China. | 20:08 |
| fungi | thanks jeblair! | 20:08 |
| fungi | i need to work on dinner for a few minutes but can start doing git fsck on copies of the repos for those failed changes shortly if nobody beats me to it | 20:09 |
| *** jamesmcarthur has quit IRC | 20:11 | |
| SpamapS | The SL ips were apparently a nodepool and a zuul FYI | 20:11 |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack-infra/release-tools master: update semver module for python 3.5 https://review.openstack.org/470350 | 20:11 |
| *** esikache1 has quit IRC | 20:12 | |
| *** jamesmcarthur has joined #openstack-infra | 20:13 | |
| *** nicolasbock_ has quit IRC | 20:14 | |
| fungi | SpamapS: why would nodepool be connecting to gerrit's ssh api? | 20:17 |
| fungi | that strikes me as odd. a zuul i can understand | 20:17 |
| *** e0ne_ has joined #openstack-infra | 20:19 | |
| openstackgerrit | Merged openstack-infra/puppet-zuul master: Fix typo with executor logging in zuul.conf https://review.openstack.org/471873 | 20:19 |
| SpamapS | fungi: image builds? | 20:19 |
| SpamapS | fungi: I'm not entirely sure. | 20:20 |
| *** marst_ has joined #openstack-infra | 20:20 | |
| *** wolverineav has joined #openstack-infra | 20:20 | |
| fungi | 113.196.154.248 seems to be used by a couple different users in gerrit... disco.ci and itri-neutron | 20:20 |
| *** Goneri has quit IRC | 20:20 | |
| *** esikache1 has joined #openstack-infra | 20:21 | |
| clarkb | dhellmann: dtroyer https://review.openstack.org/471900 is a first stab at cleaning up devstack. There are a few things I've missed htat are more complicated | 20:21 |
| *** gardlt has quit IRC | 20:21 | |
| *** flepied has joined #openstack-infra | 20:21 | |
| clarkb | dhellmann: dtroyer I also pushed a noopy change to devstack to have some hope of deltaing runtimes | 20:22 |
| pabelanger | ya, cache-devstack makes sense if they are hitting review.o.o | 20:22 |
| pabelanger | but, we default to git.o.o today | 20:22 |
| fungi | SpamapS: those softlayer ip addresses don't show up in the ssh log for gerrit at all though... wondering if they were connecting but never authenticating | 20:22 |
| *** e0ne has quit IRC | 20:22 | |
| SpamapS | Apparently the zuul in that SL pair of IPs was configured with a key that could not auth, and so it retried every 2 minutes, but there may be a bug in zuul where it doesn't properly close connections on auth fail. | 20:22 |
| dhellmann | clarkb : ack | 20:22 |
| SpamapS | fungi: ^^ | 20:22 |
| *** gardlt has joined #openstack-infra | 20:22 | |
| clarkb | SpamapS: I believe that zuul bug was fixed | 20:22 |
| clarkb | I want to say I wrote the change | 20:22 |
| openstackgerrit | Merged openstack-infra/tripleo-ci master: Use bool with ansible boolean variables https://review.openstack.org/465111 | 20:22 |
| fungi | yeah, already bit us with another third-party ci | 20:23 |
| fungi | involved bumping the paramiko version i think? | 20:23 |
| clarkb | SpamapS: I713cb704ac24db5d38cd9c0c2d60962dca647f75 | 20:23 |
| SpamapS | clarkb: ah, is it possible puppet-openstackci users wouldn't get that automatically? | 20:23 |
| *** marst has quit IRC | 20:23 | |
| clarkb | maybe we haven't released? not sure, thats the changeid to chekc for | 20:23 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: TEST: DONT RECHECK or REVIEW: periodic jobs https://review.openstack.org/359215 | 20:24 |
| *** gardlt_ has joined #openstack-infra | 20:24 | |
| pabelanger | 2.5.2 should have the fix | 20:25 |
| fungi | yep, i just confirmed that as well | 20:25 |
| *** gardlt has quit IRC | 20:26 | |
| *** esikache1 has quit IRC | 20:26 | |
| SpamapS | OK, so they actually were playing with Github support and a fork of BonnyCI.. they never intended to even attempt connection to review.o.o :-P | 20:26 |
| pabelanger | puppet-openstackci looks to clone from master branch too | 20:26 |
| SpamapS | They send their apologies. | 20:27 |
| fungi | SpamapS: just glad we figured it out | 20:27 |
| fungi | pabelanger: i concur http://git.openstack.org/cgit/openstack-infra/puppet-openstackci/tree/manifests/single_node_ci.pp#n187 | 20:28 |
| SpamapS | indeed | 20:28 |
| fungi | (unless overridden by the caller) | 20:28 |
| *** gardlt_ has quit IRC | 20:29 | |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Extend in-repo config update support to github [wip] https://review.openstack.org/471477 | 20:30 |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack-infra/release-tools master: Fix for missing OpenSSL headers https://review.openstack.org/471903 | 20:30 |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack-infra/release-tools master: update semver module for python 3.5 https://review.openstack.org/470350 | 20:31 |
| *** gardlt has joined #openstack-infra | 20:32 | |
| clarkb | dhellmann: dtroyer why do some commands like openstack role add not have cliff table output options? | 20:33 |
| openstackgerrit | Ronelle Landy proposed openstack-infra/project-config master: Add RDO Cloud-related definitions to project-config https://review.openstack.org/471909 | 20:35 |
| openstackgerrit | Merged openstack-infra/puppet-zuul master: Start executor as root https://review.openstack.org/471879 | 20:35 |
| dtroyer | because they have no output | 20:38 |
| dtroyer | they'll only generate output on an error | 20:38 |
| clarkb | dtroyer: gotcha | 20:38 |
| ihrachys | launchpad also dead? getting timeouts. | 20:38 |
| fungi | 60.251.195.198 is also opening a fair number of connections (approaching 200 now) and is being used for the same accounts as 113.196.154.248 | 20:39 |
| dtroyer | that would be another consistency smoothed out by making the output options global | 20:39 |
| fungi | #status log Blocked 113.196.154.248 in iptables on review.o.o temporarily for excessive connection counts | 20:39 |
| openstackstatus | fungi: finished logging | 20:39 |
| fungi | er, that's redundant | 20:40 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Remove ansible files at startup before copy https://review.openstack.org/471518 | 20:40 |
| fungi | #status log Blocked 60.251.195.198 in iptables on review.o.o temporarily for excessive connection counts | 20:40 |
| openstackstatus | fungi: finished logging | 20:40 |
| fungi | is the one i meant to add | 20:40 |
| *** e0ne has joined #openstack-infra | 20:40 | |
| fungi | i'm closing the connections from that one in a loop now since they weren't going away on their own | 20:42 |
| *** e0ne has quit IRC | 20:42 | |
| *** e0ne_ has quit IRC | 20:42 | |
| *** salv-orl_ has joined #openstack-infra | 20:43 | |
| *** e0ne has joined #openstack-infra | 20:44 | |
| *** salv-orlando has quit IRC | 20:46 | |
| fungi | infra-root: here's the list of 13 unique repos corresponding to the errors jeblair grepped out of the gerrit logs: http://paste.openstack.org/show/611769/ | 20:50 |
| *** trown is now known as trown|outtypewww | 20:52 | |
| *** esikache1 has joined #openstack-infra | 20:52 | |
| *** pcaruana has quit IRC | 20:53 | |
| *** pcaruana has joined #openstack-infra | 20:53 | |
| sdague | so, gerrit seemed to just make a ton of things as merge conflict that don't seem like they should be | 20:53 |
| sdague | https://review.openstack.org/#/c/393695/8 is a good example | 20:55 |
| sdague | it all happend 48 minutes ago | 20:55 |
| sdague | https://review.openstack.org/#/c/470676/ is another | 20:55 |
| sdague | it's a jenkins account merge conflict, which means I think it was zuul specifying it, right? | 20:55 |
| sdague | https://review.openstack.org/#/c/431149/ is a 3rd (I've got a dozen or so up in tabs) | 20:56 |
| *** jamesdenton has joined #openstack-infra | 20:57 | |
| fungi | sdague: i'm going through the impacted repos right now trying to figure out if any are corrupt... nova is in that list | 20:57 |
| *** esikache1 has quit IRC | 20:57 | |
| sdague | fungi: ok, cool | 20:57 |
| sdague | yeh, people have been just hitting rebase buttons, but it seemed worth reporting | 20:58 |
| sdague | as it seems odd | 20:58 |
| fungi | apparently gerrit got really unhappy after some misconfigured ci systems opened a total of >5k ssh connections to it, and so a number of git operations failed | 20:58 |
| *** priteau has quit IRC | 20:58 | |
| panda | where is the configuration (URL for example) for the nodepool providers stored ? | 21:00 |
| clarkb | dtroyer: dhellmann digging in more it does also appear that the actual make a connection and do ssl and read/write over it is a big part of the cost, at least against the cloud I am talking to | 21:01 |
| clarkb | maybe I should talk to a cloud that is more local to me to reduce that rtt overhead | 21:01 |
| fungi | panda: http://git.openstack.org/cgit/openstack-infra/project-config/tree/nodepool/nodepool.yaml | 21:01 |
| *** Apoorva_ has quit IRC | 21:02 | |
| fungi | panda: or do you mean http://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/templates/nodepool/clouds.yaml.erb | 21:02 |
| *** Apoorva has joined #openstack-infra | 21:03 | |
| sdague | fungi: gotcha | 21:03 |
| panda | fungi: exaclty the second one thanks | 21:03 |
| panda | fungi: how are the password and username passed ? | 21:05 |
| clarkb | panda: the cloud parameter in nodepool.yaml maps to a cloud entry in cloudsyaml | 21:06 |
| dhellmann | clarkb : https://review.openstack.org/471917 | 21:07 |
| panda | clarkb: I mean, how do we communicate the credentials to infra ? | 21:07 |
| panda | clarkb: in case we want to add a new provider | 21:07 |
| clarkb | panda: in the past we've just negotiated auth creds over email and then updated all the passwords on our end afterwards | 21:08 |
| clarkb | panda: then ew add them to the private hiera data store | 21:08 |
| panda | clarkb: ok, which email ? | 21:09 |
| clarkb | panda: I've done them in the past though not in a great spot to deal with it today. pabelanger or fungi might be? | 21:11 |
| clarkb | panda: also any more info on what you are trying to do? sounds like donate cloud resources? | 21:11 |
| panda | ok, personal email then, not an alias for infra | 21:11 |
| clarkb | panda: hvae you read https://docs.openstack.org/infra/system-config/contribute-cloud.html ? | 21:11 |
| fungi | trying to juggle making dinner and investigating teh gerrit explosion aftermath, and should be going offline for the evening already, so not really in a good spot to handle it today but perhaps tomorrow | 21:12 |
| panda | nothing urgent | 21:12 |
| fungi | panda: but always happy to have more resources for jobs--thanks for the interest! | 21:13 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Update log to include expected data on task results https://review.openstack.org/471891 | 21:13 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Strip double-timestamps from output log https://review.openstack.org/471890 | 21:13 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Move log streaming to share with zuul_console https://review.openstack.org/471921 | 21:13 |
| panda | fungi: I think you'll be disapponted :/ | 21:13 |
| pabelanger | panda: is this RDO cloud for tripleo-jobs? | 21:14 |
| *** ldnunes has quit IRC | 21:14 | |
| panda | pabelanger: yes | 21:15 |
| *** askb has joined #openstack-infra | 21:15 | |
| pabelanger | ya, I have discussions with weshay tomorrow about it. I don't think we should be adding it to nodepool.o.o since this would be the perfect time to move the 3rd-party CI | 21:15 |
| weshay | panda, at the tripleo-ci-squad sync | 21:16 |
| pabelanger | we've been trying to do this for a while now, and likely time to just do it. rdoproject would be a great place to run this from IMO | 21:16 |
| *** sdague has quit IRC | 21:16 | |
| *** pcaruana has quit IRC | 21:16 | |
| panda | weshay: pabelanger ok, didn't know | 21:17 |
| pabelanger | I would be great if we could run any job on the cloud but as I understand it, this is just tripleo OVB stuff | 21:17 |
| *** sree has joined #openstack-infra | 21:18 | |
| weshay | pabelanger, it's not just for ovb | 21:18 |
| *** Swami has joined #openstack-infra | 21:19 | |
| *** e0ne has quit IRC | 21:19 | |
| panda | it might be just tripleo stuff though | 21:20 |
| *** esikache1 has joined #openstack-infra | 21:20 | |
| *** eharney has quit IRC | 21:21 | |
| weshay | well.. I'll say this.. we hope it's not just ovb.. if multinode nodepool jobs can not run on that cloud for $reason that is good to know | 21:21 |
| *** dimtruck is now known as zz_dimtruck | 21:22 | |
| *** abishop has quit IRC | 21:22 | |
| *** funzo_ has joined #openstack-infra | 21:22 | |
| *** sree has quit IRC | 21:22 | |
| *** Apoorva_ has joined #openstack-infra | 21:23 | |
| *** kornica has joined #openstack-infra | 21:23 | |
| *** vhosakot_ has joined #openstack-infra | 21:24 | |
| *** berendt_ has joined #openstack-infra | 21:24 | |
| *** tommylikehu_ has joined #openstack-infra | 21:24 | |
| *** seongsoocho_ has joined #openstack-infra | 21:24 | |
| *** jbernard has quit IRC | 21:24 | |
| *** sweston_ has joined #openstack-infra | 21:24 | |
| *** betherly_ has joined #openstack-infra | 21:24 | |
| *** cargonza_ has joined #openstack-infra | 21:24 | |
| *** esikache1 has quit IRC | 21:24 | |
| *** Sukhdev has joined #openstack-infra | 21:25 | |
| *** zerick_ has joined #openstack-infra | 21:25 | |
| *** hwoarang_ has joined #openstack-infra | 21:26 | |
| *** esikache1 has joined #openstack-infra | 21:26 | |
| *** gorba_ has joined #openstack-infra | 21:26 | |
| *** frickler_ has joined #openstack-infra | 21:26 | |
| *** SamYaple_ has joined #openstack-infra | 21:27 | |
| *** aviau_ has joined #openstack-infra | 21:27 | |
| *** odyssey4me_ has joined #openstack-infra | 21:28 | |
| *** wolverineav has quit IRC | 21:29 | |
| *** waynr1 has joined #openstack-infra | 21:29 | |
| *** jbernard has joined #openstack-infra | 21:29 | |
| *** jamesmcarthur has quit IRC | 21:29 | |
| *** mtreinish has quit IRC | 21:29 | |
| *** karthikp has quit IRC | 21:29 | |
| *** michaelxin has quit IRC | 21:29 | |
| *** tommylikehu has quit IRC | 21:29 | |
| *** cargonza has quit IRC | 21:29 | |
| *** berendt has quit IRC | 21:29 | |
| *** betherly has quit IRC | 21:29 | |
| *** sweston has quit IRC | 21:29 | |
| *** seongsoocho has quit IRC | 21:29 | |
| *** rockstar has quit IRC | 21:29 | |
| *** funzo has quit IRC | 21:29 | |
| *** calebb has quit IRC | 21:29 | |
| *** waynr has quit IRC | 21:29 | |
| *** vhosakot has quit IRC | 21:29 | |
| *** fbouliane has quit IRC | 21:29 | |
| *** SamYaple has quit IRC | 21:29 | |
| *** gorba has quit IRC | 21:29 | |
| *** frickler has quit IRC | 21:29 | |
| *** pahuang has quit IRC | 21:29 | |
| *** kornicameister has quit IRC | 21:29 | |
| *** jgriffith has quit IRC | 21:29 | |
| *** Kevin_Zheng has quit IRC | 21:29 | |
| *** tomhambleton_ has quit IRC | 21:29 | |
| *** Apoorva has quit IRC | 21:29 | |
| *** aviau has quit IRC | 21:29 | |
| *** hwoarang has quit IRC | 21:29 | |
| *** zerick has quit IRC | 21:29 | |
| *** odyssey4me has quit IRC | 21:29 | |
| *** berendt_ is now known as berendt | 21:29 | |
| *** betherly_ is now known as betherly | 21:29 | |
| *** sweston_ is now known as sweston | 21:29 | |
| *** jgriffith_ has joined #openstack-infra | 21:29 | |
| *** tommylikehu_ is now known as tommylikehu | 21:29 | |
| *** seongsoocho_ is now known as seongsoocho | 21:29 | |
| *** cargonza_ is now known as cargonza | 21:29 | |
| *** karthikp has joined #openstack-infra | 21:29 | |
| *** jamesmcarthur has joined #openstack-infra | 21:30 | |
| *** zerick_ is now known as zerick | 21:30 | |
| *** mtreinish has joined #openstack-infra | 21:30 | |
| *** pahuang has joined #openstack-infra | 21:30 | |
| *** fbouliane has joined #openstack-infra | 21:30 | |
| *** calebb has joined #openstack-infra | 21:30 | |
| *** esikache1 has quit IRC | 21:31 | |
| clarkb | pabelanger: fungi is nodepool not launching nodes now? | 21:31 |
| *** vipul- has quit IRC | 21:32 | |
| *** spiffxp has quit IRC | 21:32 | |
| *** ryotagami has quit IRC | 21:32 | |
| *** ryotagami has joined #openstack-infra | 21:34 | |
| *** esberglu has quit IRC | 21:34 | |
| *** esberglu has joined #openstack-infra | 21:35 | |
| *** vipul has joined #openstack-infra | 21:37 | |
| *** mdrabe has quit IRC | 21:38 | |
| clarkb | I am relocating so afk for about 45 minutes but will try to look into ^ | 21:38 |
| clarkb | did aomething change config wise? | 21:38 |
| pabelanger | clarkb: no, but I see 330 nodes buildings now | 21:38 |
| pabelanger | huge upspike | 21:38 |
| clarkb | pabelanger: are they actually building? | 21:39 |
| clarkb | I saw that that was the demand but nothing was in building state | 21:39 |
| *** esberglu has quit IRC | 21:39 | |
| pabelanger | ya, I see them building in CLI | 21:39 |
| pabelanger | and grafana | 21:39 |
| clarkb | huh I wonder what the delay was | 21:40 |
| *** spiffxp has joined #openstack-infra | 21:40 | |
| pabelanger | but we are getting a large amount of failures in ovh still | 21:40 |
| clarkb | basically seems like we were lagging really hard | 21:40 |
| clarkb | since demand has been there for at least 15 minutes | 21:40 |
| fungi | i'm still waiting for my copy of the nova repo to complete | 21:41 |
| pabelanger | clarkb: not sure, we are bursty today. There was a large downtrend at 20:00UTC, with a large launch attempt | 21:41 |
| SpamapS | a thought bout misconfigured CI systems crashing gerrit... | 21:42 |
| pabelanger | seems to be happening every hour today | 21:42 |
| pabelanger | http://grafana.openstack.org/dashboard/db/nodepool | 21:42 |
| SpamapS | perhaps we should put that ssh port behind something that limits per-ip to something sane like 200? | 21:42 |
| pabelanger | used nodes is spiking up every hour it seems | 21:42 |
| fungi | SpamapS: i'd like to see mqtt client support after zuul v3 and then we can have them all point to firehose.o.o instead of hitting gerrit to get the event stream | 21:43 |
| *** rcernin has quit IRC | 21:44 | |
| SpamapS | fungi: neat | 21:44 |
| *** esikache1 has joined #openstack-infra | 21:44 | |
| * SpamapS would really like to be at "after zuulv3" | 21:44 | |
| fungi | the event stream is basically the only thing you can't do over gerrit's rest api now | 21:45 |
| *** zz_dimtruck is now known as dimtruck | 21:45 | |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/zuul feature/zuulv3: Re-enable test_worker_update_metadata https://review.openstack.org/468667 | 21:46 |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/zuul feature/zuulv3: Re-enable test_merge_conflict_reports https://review.openstack.org/468670 | 21:46 |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/zuul feature/zuulv3: Re-enable test_merge_failure_reporters https://review.openstack.org/468668 | 21:46 |
| SpamapS | fungi: another question ... we don't have alerting for n_files on a critical process? | 21:47 |
| *** hongbin has quit IRC | 21:47 | |
| *** panda is now known as panda|off | 21:47 | |
| *** yland has joined #openstack-infra | 21:47 | |
| * SpamapS is woefully unfamiliar with how we operate infra :-P | 21:47 | |
| *** gordc has quit IRC | 21:47 | |
| *** esikache1 has quit IRC | 21:48 | |
| *** flepied has quit IRC | 21:48 | |
| *** esberglu has joined #openstack-infra | 21:48 | |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config master: define a unified project documentation build job https://review.openstack.org/471881 | 21:49 |
| *** yamahata has joined #openstack-infra | 21:49 | |
| *** jamesmcarthur has quit IRC | 21:50 | |
| *** rfolco has quit IRC | 21:51 | |
| *** hongbin has joined #openstack-infra | 21:51 | |
| *** esberglu has quit IRC | 21:53 | |
| *** thorst_afk has quit IRC | 21:53 | |
| fungi | SpamapS: we don't, but also this was in the context of a jvm so a system-level check on that wouldn't have helped much anyway | 21:53 |
| fungi | this was the jvm refusing one of its own threads access to more file descriptors | 21:53 |
| *** gordc has joined #openstack-infra | 21:54 | |
| SpamapS | fungi: not sure it's worth the effort, but we should be able to monitor that process and know the nfiles level where the jvm starts refusing. | 21:55 |
| fungi | we could probably set something up (and we've discussed in the past at least finding a means of bridging javamelody stats into cacti via maybe a custom mib backend to net-snmpd) but ideas are cheaper than people | 21:55 |
| SpamapS | yeah | 21:55 |
| SpamapS | that last point is the kicker :- | 21:55 |
| SpamapS | :-P | 21:55 |
| * SpamapS will go back to zuul'ing | 21:56 | |
| *** michaelxin has joined #openstack-infra | 21:56 | |
| fungi | ideally, we're looking toward a rosy future where we can get rid of public access to gerrit's ssh-based api | 21:57 |
| *** SamYaple_ has quit IRC | 21:57 | |
| *** SamYaple has joined #openstack-infra | 21:57 | |
| SpamapS | fungi: seems like there'd still be the need for it for git-review's git operations, no? | 21:58 |
| fungi | those all work over https | 21:58 |
| fungi | you can configure git review now to use https | 21:58 |
| smcginnis | fungi: Don't all of the third party CIs use the SSH access? | 21:58 |
| fungi | smcginnis: right, that's why we wouldn't lock it down until we had some similar support provided via a proxy queue | 21:59 |
| SpamapS | but how do you push? | 21:59 |
| fungi | SpamapS: push over https works with gerrit | 21:59 |
| SpamapS | oh with auth'd https? I kinda like my ssh key for that ;) | 21:59 |
| fungi | yeah, the api key in gerrit is the annoying part of that | 21:59 |
| openstackgerrit | Ronelle Landy proposed openstack-infra/system-config master: Add credentials for RDO Cloud https://review.openstack.org/471944 | 22:00 |
| *** Julien-zte has quit IRC | 22:00 | |
| fungi | anyway, the socket proxy idea isn't terrible, but someone would need to research a good solution and then do a fair amount of testing | 22:01 |
| *** Julien-zte has joined #openstack-infra | 22:02 | |
| *** bobh has quit IRC | 22:02 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Move log streaming to share with zuul_console https://review.openstack.org/471921 | 22:02 |
| *** Julien-zte has quit IRC | 22:03 | |
| *** Julien-zte has joined #openstack-infra | 22:05 | |
| clarkb | the problem with using https is passowrdanagement is bad | 22:07 |
| clarkb | ssh much much better. but we can turn off access to things that arent push code | 22:07 |
| *** Swami has quit IRC | 22:07 | |
| *** rfolco has joined #openstack-infra | 22:07 | |
| mordred | ++ | 22:07 |
| fungi | that's a good point too | 22:09 |
| fungi | honestly, though, i expect if we get the openstackci default settings switched to use mqtt and https for zuul's interactions, that'll cut down on most of the problem | 22:10 |
| *** aeng has joined #openstack-infra | 22:11 | |
| *** baoli_ has quit IRC | 22:12 | |
| *** jamesdenton has quit IRC | 22:13 | |
| *** esikache1 has joined #openstack-infra | 22:14 | |
| mordred | fungi: ++ | 22:15 |
| pabelanger | would like to get some reviews on: https://review.openstack.org/#/c/471877/ next step needed for ze01.o.o | 22:17 |
| *** kaisers has quit IRC | 22:18 | |
| *** tosky has quit IRC | 22:19 | |
| fungi | du says i'm nearly done copying the nova repo now | 22:19 |
| fungi | 4.4gb for the full bare repo on disk | 22:19 |
| *** eranrom has quit IRC | 22:19 | |
| fungi | (way better than it used to be before we upped our repo repacking) | 22:20 |
| *** esikache1 has quit IRC | 22:20 | |
| *** eranrom has joined #openstack-infra | 22:20 | |
| *** pradk has quit IRC | 22:20 | |
| *** pvaneck has quit IRC | 22:20 | |
| *** Sukhdev has quit IRC | 22:20 | |
| clarkb | fungi: what are we copying nova to? | 22:26 |
| clarkb | I missed something when doing osc things | 22:26 |
| fungi | clarkb: my homedir (on the rootfs) | 22:27 |
| fungi | so still faster than going over the network at least | 22:27 |
| fungi | copying all 13 of the repos which had at least one git operation failure logged by gerrit during the incident | 22:27 |
| fungi | so i can do a git fsck on them | 22:27 |
| clarkb | ah | 22:27 |
| fungi | and hopefully identify any brokenness introduced there before it bites us badly | 22:28 |
| *** thorst_afk has joined #openstack-infra | 22:28 | |
| *** xyang1 has quit IRC | 22:30 | |
| openstackgerrit | Merged openstack-infra/system-config master: Log #openstack-tc channel https://review.openstack.org/471054 | 22:30 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/infra-specs master: Clean up implemented/abandoned specs https://review.openstack.org/471948 | 22:31 |
| *** thorst_afk has quit IRC | 22:32 | |
| *** chlong has quit IRC | 22:32 | |
| *** kaisers has joined #openstack-infra | 22:33 | |
| fungi | dangling blobs are okay in a git fsck i guess? | 22:34 |
| clarkb | they get GC'd I think but otherwise dont affect integrity | 22:34 |
| fungi | yeah, we seem to have plenty in every repo i check, so assuming that's just normal | 22:34 |
| *** jkilpatr has quit IRC | 22:34 | |
| *** rlandy has quit IRC | 22:35 | |
| *** salv-orl_ has quit IRC | 22:37 | |
| *** salv-orlando has joined #openstack-infra | 22:37 | |
| jeblair | fungi: yeah, i think they'd be fine. i think, at the most, they might indicate a patchset that didn't get fully added to the repo. | 22:39 |
| *** waynr1 is now known as waynr | 22:40 | |
| fungi | jeblair: i suppose we can check for that next by attempting to retrieve the refs mentioned in your paste, but that's presumably less urgent | 22:40 |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Extend in-repo config update support to github https://review.openstack.org/471477 | 22:40 |
| *** salv-orlando has quit IRC | 22:41 | |
| *** esikache1 has joined #openstack-infra | 22:42 | |
| fungi | infra-root: okay, i've finished a `git fsck` on direct filesystem copies of all 13 git repos concerned on review.o.o and aside from dangling commits and dangling objects there is nothing to report (no errors) | 22:43 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/infra-specs master: Clean up implemented specs https://review.openstack.org/471948 | 22:45 |
| *** esikache1 has quit IRC | 22:47 | |
| *** jascott1 has quit IRC | 22:49 | |
| *** jascott1 has joined #openstack-infra | 22:50 | |
| clarkb | dtroyer: dhellmann also doesn't look like csv is a universally available output format? | 22:50 |
| *** rbrndt has quit IRC | 22:51 | |
| dtroyer | clarkb: as in it is only available for list commands? | 22:51 |
| *** jascott1 has quit IRC | 22:51 | |
| *** marst_ has quit IRC | 22:52 | |
| *** esikache1 has joined #openstack-infra | 22:52 | |
| *** felipemonteiro_ has quit IRC | 22:52 | |
| *** gardlt has quit IRC | 22:52 | |
| *** gardlt has joined #openstack-infra | 22:53 | |
| *** gomarivera has quit IRC | 22:53 | |
| *** gomarivera has joined #openstack-infra | 22:54 | |
| clarkb | dtroyer: ya subnet show doesn't have it | 22:55 |
| clarkb | even though the pretty table is just a fancy form of csv with \s|\s delimiters | 22:55 |
| *** esikache1 has quit IRC | 22:56 | |
| *** vhosakot_ has quit IRC | 22:56 | |
| dtroyer | show commands are always 2 columns: attribute and value, list commands are rows of values, only list commands do CSV. | 22:57 |
| *** vhosakot has joined #openstack-infra | 22:57 | |
| clarkb | right but why wouldn't you have csv in show commands too? | 22:57 |
| clarkb | or create commands? | 22:57 |
| *** vhosakot has quit IRC | 22:57 | |
| *** gordc has quit IRC | 22:58 | |
| *** gardlt has quit IRC | 22:58 | |
| clarkb | as a user its painful to need to reason about that and have different parsing tools for everything | 22:58 |
| clarkb | I guess json is valid everywhere? | 22:58 |
| clarkb | maybe I just need to give in and use jq | 22:58 |
| dtroyer | it is, and jq is a wonderful thing | 22:58 |
| *** gomarivera has quit IRC | 22:58 | |
| clarkb | dtroyer: I really think if one of the thing osc wants to do is make everything consistent then that has to follow through to output formats too | 22:59 |
| *** dhinesh has quit IRC | 22:59 | |
| *** dhinesh has joined #openstack-infra | 22:59 | |
| dtroyer | a row-oriented show command output of any size will wrap very wuickly and be very hard to read | 22:59 |
| dtroyer | *quickly | 23:00 |
| fungi | q (for using simple sql syntax with csv) is also pretty wonderful | 23:00 |
| clarkb | dtroyer: you do it exactly the same way as the prettytable, but don't worry about whitespace | 23:00 |
| clarkb | dtroyer: since the pretty table is almost a csv on its own | 23:00 |
| *** lbragstad has quit IRC | 23:00 | |
| dtroyer | the difference is simply one object or many objects. I suppose adding a row-oriented format to show commands is do-able, it's a cliff thing | 23:00 |
| clarkb | dtroyer: field,value\nid,$uuid\ncreate_at,$timestamp\n | 23:01 |
| dtroyer | internally, the take_action() method returns the results totally differently because prettytable doens't know the difference, the comamnds are hard-coded that way | 23:01 |
| *** thorst_afk has joined #openstack-infra | 23:02 | |
| dtroyer | I'll admit the output of OSC is still rather primitive, there are a lot of things we could do to help ourselves. I do wish prettytable had never existed, I think it boxes us into some pre-concieved notions early (yay puns!) | 23:02 |
| *** vhosakot has joined #openstack-infra | 23:03 | |
| dtroyer | representing structured data in these forms is impossible, you will have to resort to json or yaml to do anything really interesting | 23:04 |
| clarkb | the problem is that json and yaml are terrible for humans, and to an extent terrible for scripting (jq is really not a fun tool to use ime, different versions behave differently and the syntax is really clunky) | 23:05 |
| fungi | seems like most of it doesn't necessarily need "structured" data if you accept that you can have very wide rows with each value being in its own column. then batch activity makes for some very useful rdbms-like tabular data | 23:06 |
| dtroyer | shell scripting and structured data do not mix well, no | 23:06 |
| fungi | (every key in your key,value pairs is a column of its own) | 23:07 |
| clarkb | dtroyer: is the shell output as ubiquitous as the json output? that might be better in devstack | 23:07 |
| dtroyer | shell is only in create and show commands | 23:07 |
| clarkb | no you can't shell on lists | 23:07 |
| clarkb | ok | 23:07 |
| dtroyer | I suppose adding the list forms to the show-style output is doable, but I wouldn't want it as the default based on 5 years of using this | 23:08 |
| *** esikache1 has joined #openstack-infra | 23:08 | |
| *** bobh has joined #openstack-infra | 23:10 | |
| clarkb | dtroyer: its already the default (table) | 23:10 |
| dtroyer | I'm talking about the attributes being rows (list) or columns (show) | 23:10 |
| clarkb | I'm fine with then being columns just render the csv like the table is and have them in columns | 23:11 |
| *** yamamoto_ has joined #openstack-infra | 23:11 | |
| dtroyer | you mean a CSv with the first line being "Fielf","Value" | 23:12 |
| dtroyer | "Field" | 23:12 |
| *** openstack has joined #openstack-infra | 23:13 | |
| clarkb | yup, currently converting to use that :) | 23:13 |
| dtroyer | and doesn't have a header | 23:13 |
| clarkb | I can't actually source the values because of conflicting var names :/ | 23:14 |
| dtroyer | Somewhere I did a thing adding a prefix to those nmaes | 23:14 |
| dtroyer | I don't recall if it merged, doesn't look like it did | 23:14 |
| *** yamamoto_ has quit IRC | 23:16 | |
| *** claudiub has quit IRC | 23:17 | |
| dtroyer | actually I think it did —prefix | 23:17 |
| *** thorst_afk has quit IRC | 23:17 | |
| dtroyer | --f shell --prefix q_ | 23:17 |
| dtroyer | although the eval() part of that makes me wince | 23:18 |
| dtroyer | there is _no_ sanitization on returned data, ie a badly-formed image description could wreak havoc | 23:18 |
| *** tomhambleton_ has joined #openstack-infra | 23:21 | |
| * jeblair waves hi to the future cve authors reading this! | 23:21 | |
| *** Kevin_Zheng has joined #openstack-infra | 23:21 | |
| *** rockstar has joined #openstack-infra | 23:21 | |
| *** jascott1 has joined #openstack-infra | 23:25 | |
| *** esikache1 has joined #openstack-infra | 23:25 | |
| openstackgerrit | Merged openstack-infra/release-tools master: Fix for missing OpenSSL headers https://review.openstack.org/471903 | 23:25 |
| openstackgerrit | Merged openstack-infra/release-tools master: update semver module for python 3.5 https://review.openstack.org/470350 | 23:26 |
| *** xarses has quit IRC | 23:26 | |
| clarkb | dtroyer: fun, I think what I just pushed in https://review.openstack.org/471900 is half sane and avoids evals | 23:27 |
| clarkb | so no cves :) | 23:27 |
| *** esikache1 has quit IRC | 23:29 | |
| *** JudeC has quit IRC | 23:30 | |
| *** gardlt has joined #openstack-infra | 23:30 | |
| *** gomarivera has joined #openstack-infra | 23:30 | |
| *** Julien-zte has quit IRC | 23:32 | |
| *** gomarivera has quit IRC | 23:34 | |
| *** gardlt has quit IRC | 23:34 | |
| *** Julien-zte has joined #openstack-infra | 23:36 | |
| *** esikache1 has joined #openstack-infra | 23:38 | |
| *** jiaohaolin has quit IRC | 23:40 | |
| *** Julien-zte has quit IRC | 23:41 | |
| *** yland has quit IRC | 23:41 | |
| *** jiaohaolin has joined #openstack-infra | 23:42 | |
| *** esikache1 has quit IRC | 23:42 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Use class as super() argument https://review.openstack.org/471961 | 23:44 |
| *** yamamoto_ has joined #openstack-infra | 23:47 | |
| *** yland has joined #openstack-infra | 23:53 | |
| *** yland is now known as Guest80957 | 23:54 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!