| *** salv-orlando has joined #openstack-infra | 00:00 | |
| *** bobh has quit IRC | 00:04 | |
| *** Guest80957 has quit IRC | 00:04 | |
| *** salv-orlando has quit IRC | 00:05 | |
| *** esikache1 has joined #openstack-infra | 00:10 | |
| *** esikache1 has quit IRC | 00:15 | |
| *** esikache1 has joined #openstack-infra | 00:25 | |
| *** r-daneel has quit IRC | 00:26 | |
| *** esikache1 has quit IRC | 00:29 | |
| *** masber has quit IRC | 00:32 | |
| *** masber has joined #openstack-infra | 00:32 | |
| *** lbragstad has joined #openstack-infra | 00:37 | |
| *** rfolco has quit IRC | 00:44 | |
| *** Julien-zte has joined #openstack-infra | 00:45 | |
| *** LindaWang has joined #openstack-infra | 00:45 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: Preparing infra servers in RDO cloud https://review.openstack.org/468047 | 00:46 |
|---|---|---|
| *** cuongnv has joined #openstack-infra | 00:47 | |
| *** armax has quit IRC | 00:48 | |
| *** vhosakot has quit IRC | 00:50 | |
| *** armax has joined #openstack-infra | 00:56 | |
| *** gardlt has joined #openstack-infra | 00:59 | |
| *** armax has quit IRC | 00:59 | |
| *** dpaterson has quit IRC | 01:00 | |
| *** salv-orlando has joined #openstack-infra | 01:01 | |
| *** salv-orlando has quit IRC | 01:06 | |
| *** dpaterson has joined #openstack-infra | 01:06 | |
| openstackgerrit | Jamie Lennox proposed openstack-infra/zuul feature/zuulv3: Expose buildset to the executor and url formatter https://review.openstack.org/463457 | 01:06 |
| openstackgerrit | Jamie Lennox proposed openstack-infra/zuul feature/zuulv3: Add tenant to url formatting. https://review.openstack.org/463628 | 01:06 |
| *** jamesdenton has joined #openstack-infra | 01:07 | |
| *** liujiong has joined #openstack-infra | 01:08 | |
| *** rfolco has joined #openstack-infra | 01:08 | |
| *** rhallisey has quit IRC | 01:09 | |
| *** stakeda has joined #openstack-infra | 01:13 | |
| *** thorst_afk has joined #openstack-infra | 01:14 | |
| *** sree has joined #openstack-infra | 01:18 | |
| *** csomerville has joined #openstack-infra | 01:19 | |
| *** thorst_afk has quit IRC | 01:19 | |
| *** cody-somerville has quit IRC | 01:20 | |
| *** mriedem has quit IRC | 01:22 | |
| *** sree has quit IRC | 01:22 | |
| jiaohaolin | (20:32:00) jiaohaolin: Hello guys, my CI just setup ok, and it only worked for 5 hours, then the debug-log shows "Looking for the lost building", so I restart the service of zuul. BOOM, the zuul can't trigger the jobs . And the zuul.log shows "2017-06-07 20:26:29,413 WARNING zuul.GerritEventConnector: Received unrecognized event type 'ref-replicated' from Gerrit. Can not get account information." | 01:23 |
| jiaohaolin | No matter how many times I restart the zuul, it just does not work It drive me crazy | 01:24 |
| jiaohaolin | help | 01:24 |
| *** rfolco has quit IRC | 01:25 | |
| *** jamesdenton has quit IRC | 01:25 | |
| *** dhinesh has quit IRC | 01:32 | |
| *** JudeC has joined #openstack-infra | 01:34 | |
| *** Julien-zte has quit IRC | 01:35 | |
| *** Apoorva has joined #openstack-infra | 01:36 | |
| *** Apoorva_ has quit IRC | 01:40 | |
| *** jiaohaolin1 has joined #openstack-infra | 01:40 | |
| *** Apoorva has quit IRC | 01:40 | |
| *** yamahata has quit IRC | 01:40 | |
| *** jiaohaolin has quit IRC | 01:41 | |
| *** lbragstad has quit IRC | 01:42 | |
| *** thorst_afk has joined #openstack-infra | 01:47 | |
| *** thorst_afk has quit IRC | 01:47 | |
| *** waynr has quit IRC | 01:48 | |
| *** zhurong has joined #openstack-infra | 01:48 | |
| *** waynr has joined #openstack-infra | 01:49 | |
| *** csomerville has quit IRC | 01:57 | |
| *** xinliang has quit IRC | 02:00 | |
| *** armax has joined #openstack-infra | 02:02 | |
| *** jamielennox is now known as jamielennox|away | 02:04 | |
| *** gomarivera has joined #openstack-infra | 02:09 | |
| jeblair | jiaohaolin1: 'looking for lost builds' is a normal message, you don't need to restart it for that. | 02:12 |
| *** thorst_afk has joined #openstack-infra | 02:12 | |
| jeblair | jiaohaolin1: the message about 'ref-replicated' is normal too, you can ignore it. | 02:12 |
| *** thorst_afk has quit IRC | 02:12 | |
| *** xinliang has joined #openstack-infra | 02:12 | |
| *** xinliang has quit IRC | 02:12 | |
| *** xinliang has joined #openstack-infra | 02:12 | |
| *** yamamoto_ has quit IRC | 02:14 | |
| *** jamielennox|away is now known as jamielennox | 02:15 | |
| *** JudeC has quit IRC | 02:16 | |
| *** yamamoto_ has joined #openstack-infra | 02:17 | |
| *** gildub has joined #openstack-infra | 02:17 | |
| *** gomarivera has quit IRC | 02:27 | |
| *** gyee has quit IRC | 02:27 | |
| jiaohaolin1 | jeblair: But the zuul log only shows 'ref-replicated' ,and when the looking for lost building come up ,the zuul can't not receive the event flow from gerrit | 02:27 |
| jiaohaolin1 | And the zuul is not working ,it can't trigger any jobs | 02:28 |
| *** hongbin has quit IRC | 02:36 | |
| *** Sukhdev has joined #openstack-infra | 02:36 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Allow per-repo selection of configuration classes to load https://review.openstack.org/472009 | 02:36 |
| *** Rockyg has joined #openstack-infra | 02:37 | |
| *** Rockyg has quit IRC | 02:37 | |
| *** bakbad has joined #openstack-infra | 02:38 | |
| jeblair | jiaohaolin1: if you are seeing ref-replicated messages, then you are receiving gerrit events | 02:38 |
| jeblair | jiaohaolin1: can you copy your layout.yaml to paste.openstack.org and paste the link here? | 02:39 |
| *** Julien-zte has joined #openstack-infra | 02:39 | |
| jiaohaolin1 | ok | 02:40 |
| jiaohaolin1 | jeblair: wait a second | 02:40 |
| *** gomarivera has joined #openstack-infra | 02:44 | |
| jiaohaolin1 | jeblair: http://paste.openstack.org/show/611791/ | 02:44 |
| *** liusheng has quit IRC | 02:46 | |
| *** liusheng has joined #openstack-infra | 02:46 | |
| jeblair | jiaohaolin1: can you copy the debug.log to paste.openstack.org too? | 02:48 |
| jiaohaolin1 | jeblair: When I use this command ‘sudo -u zuul zuul enqueue --trigger gerrit --pipeline check --project openstack/cinder --change 439488,18’to trigger job ,it will be blocked | 02:49 |
| jiaohaolin1 | jeblair: ok | 02:49 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Allow per-repo selection of configuration classes to load https://review.openstack.org/472009 | 02:49 |
| jiaohaolin1 | jeblair: http://paste.openstack.org/show/611792/ | 02:50 |
| *** jamesmcarthur has joined #openstack-infra | 02:50 | |
| jeblair | jiaohaolin1: how about the first two minutes of logs after zuul starts? | 02:52 |
| jiaohaolin1 | jeblair: http://paste.openstack.org/show/611793/ This is a piece of zuul.log, | 02:52 |
| *** dingyichen has joined #openstack-infra | 02:53 | |
| jiaohaolin1 | jeblair: wait for a minute ,let me restart it | 02:55 |
| *** jamesmcarthur has quit IRC | 02:55 | |
| *** Julien-zte has quit IRC | 02:57 | |
| jiaohaolin1 | jeblair: http://paste.openstack.org/show/611794/ This is the new log after restart zuul. | 02:57 |
| *** dingyichen has quit IRC | 03:02 | |
| jeblair | jiaohaolin1: is that from the beginning of the log after restart? i would expect some log lines i don't see there. | 03:02 |
| jiaohaolin1 | jeblair: I delete the old debug.log, then restart zuul | 03:03 |
| *** yamamoto_ has quit IRC | 03:04 | |
| jeblair | jiaohaolin1: are you sure that it stopped when you restarted it? can you stop it, double check that no zuul processes are running, delete the debug log, then start again and paste the debug log? | 03:08 |
| jiaohaolin1 | jeblair: Ok ,wait for a moment | 03:09 |
| *** Julien-zte has joined #openstack-infra | 03:10 | |
| *** rfolco has joined #openstack-infra | 03:11 | |
| *** pahuang has quit IRC | 03:12 | |
| *** rfolco has quit IRC | 03:12 | |
| *** rkukura has quit IRC | 03:12 | |
| *** thorst_afk has joined #openstack-infra | 03:13 | |
| jeblair | jiaohaolin1: for your reference, this is what zuul logged the last time we restarted it: http://paste.openstack.org/show/611796/ | 03:14 |
| *** bakbad_ has joined #openstack-infra | 03:14 | |
| jeblair | jiaohaolin1: comparing that to yours suggests some things to me: | 03:15 |
| *** yamahata has joined #openstack-infra | 03:15 | |
| jeblair | jiaohaolin1: you may have multiple zuul-scheduler processes running, and you may not have a gearman server running. | 03:15 |
| *** brucet has joined #openstack-infra | 03:15 | |
| *** brucet has quit IRC | 03:16 | |
| *** brucet has joined #openstack-infra | 03:16 | |
| *** bakbad has quit IRC | 03:17 | |
| jiaohaolin1 | jeblair: the zuul seems can not be stop | 03:18 |
| jeblair | jiaohaolin1: you may need to "kill -9" the PID, then be sure to remove the pidfile from /var/run/zuul/zuul.pid | 03:19 |
| *** bakbad__ has joined #openstack-infra | 03:19 | |
| jiaohaolin1 | ok ,let me try it | 03:20 |
| *** bakbad_ has quit IRC | 03:22 | |
| *** dimtruck is now known as zz_dimtruck | 03:23 | |
| jeblair | jiaohaolin1: i have to go now. my advice is make sure that you have stopped all of the zuul services, removed any pid files, and have "start=true" in the "[gearman_server]" section of zuul.conf. you should see debug log entries that look like http://paste.openstack.org/show/611796/ when it starts. | 03:24 |
| jiaohaolin1 | jeblair: http://paste.openstack.org/show/611797/ It's the new one | 03:24 |
| jiaohaolin1 | jeblair: ok ,thank | 03:26 |
| jiaohaolin1 | jeblair: see you tomorrow | 03:26 |
| *** rfolco has joined #openstack-infra | 03:27 | |
| *** rfolco has quit IRC | 03:27 | |
| *** brucet has quit IRC | 03:27 | |
| jiaohaolin1 | jeblair: it seem the zuul not working is due to python | 03:28 |
| *** brucet has joined #openstack-infra | 03:28 | |
| *** gomarivera has quit IRC | 03:30 | |
| *** gomarivera has joined #openstack-infra | 03:31 | |
| *** gomarivera has quit IRC | 03:31 | |
| *** gomarivera has joined #openstack-infra | 03:32 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Allow per-repo selection of configuration classes to load https://review.openstack.org/472009 | 03:32 |
| *** thorst_afk has quit IRC | 03:32 | |
| *** brucet has quit IRC | 03:33 | |
| *** links has joined #openstack-infra | 03:34 | |
| *** brucet has joined #openstack-infra | 03:40 | |
| *** ramishra has joined #openstack-infra | 03:43 | |
| *** brucet has quit IRC | 03:43 | |
| *** ykarel__ has joined #openstack-infra | 03:43 | |
| *** zhurong has quit IRC | 03:45 | |
| *** brucet has joined #openstack-infra | 03:46 | |
| *** bruce has joined #openstack-infra | 03:49 | |
| *** bruce is now known as Guest42119 | 03:49 | |
| *** brucet has quit IRC | 03:51 | |
| openstackgerrit | Jamie Lennox proposed openstack-infra/zuul feature/zuulv3: Expose buildset to the executor and url formatter https://review.openstack.org/463457 | 03:53 |
| openstackgerrit | Jamie Lennox proposed openstack-infra/zuul feature/zuulv3: Add tenant to url formatting. https://review.openstack.org/463628 | 03:53 |
| *** gomarivera has quit IRC | 03:56 | |
| *** gomarivera has joined #openstack-infra | 03:57 | |
| *** tojuvone has joined #openstack-infra | 03:57 | |
| *** tojuvone has left #openstack-infra | 03:58 | |
| *** gomarivera has quit IRC | 04:01 | |
| *** psachin has joined #openstack-infra | 04:02 | |
| *** namnh has joined #openstack-infra | 04:04 | |
| *** pahuang has joined #openstack-infra | 04:04 | |
| *** stakeda has quit IRC | 04:11 | |
| *** stakeda has joined #openstack-infra | 04:14 | |
| *** ramishra has quit IRC | 04:25 | |
| *** dave-mccowan has quit IRC | 04:26 | |
| *** Julien-zte has quit IRC | 04:27 | |
| *** bakbad__ is now known as bakbad | 04:27 | |
| *** Julien-zte has joined #openstack-infra | 04:28 | |
| *** thorst_afk has joined #openstack-infra | 04:29 | |
| *** ihrachys has quit IRC | 04:29 | |
| *** pcaruana has joined #openstack-infra | 04:31 | |
| *** zhurong has joined #openstack-infra | 04:33 | |
| *** thorst_afk has quit IRC | 04:37 | |
| *** salv-orlando has joined #openstack-infra | 04:39 | |
| *** yamamoto has joined #openstack-infra | 04:48 | |
| *** sree has joined #openstack-infra | 04:51 | |
| *** deafult has joined #openstack-infra | 04:53 | |
| *** dpaterson has quit IRC | 04:56 | |
| *** pcaruana has quit IRC | 04:57 | |
| *** rkukura has joined #openstack-infra | 05:00 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Add env var to dump config graph https://review.openstack.org/472064 | 05:04 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: [WIP] LVM support for dib-block-device https://review.openstack.org/472065 | 05:04 |
| openstackgerrit | Jamie Lennox proposed openstack-infra/zuul feature/zuulv3: Strip not split the untrusted_wrapper https://review.openstack.org/472066 | 05:04 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Add env var to dump config graph https://review.openstack.org/472064 | 05:04 |
| *** jaosorior_away is now known as jaosorior | 05:05 | |
| *** dpaterson has joined #openstack-infra | 05:05 | |
| *** salv-orlando has quit IRC | 05:05 | |
| *** salv-orlando has joined #openstack-infra | 05:06 | |
| *** aeng has quit IRC | 05:06 | |
| *** ltomasbo|away is now known as ltomasbo | 05:09 | |
| *** ramishra has joined #openstack-infra | 05:11 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Use class as super() argument https://review.openstack.org/471961 | 05:11 |
| *** jamielennox is now known as jamielennox|away | 05:12 | |
| *** prateek has joined #openstack-infra | 05:12 | |
| *** eranrom has quit IRC | 05:17 | |
| *** jamielennox|away is now known as jamielennox | 05:19 | |
| *** aeng has joined #openstack-infra | 05:23 | |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder master: Add option to make test output file layout match Jenkins https://review.openstack.org/126143 | 05:24 |
| *** ykarel_ has joined #openstack-infra | 05:26 | |
| *** claudiub has joined #openstack-infra | 05:26 | |
| *** gomarivera has joined #openstack-infra | 05:27 | |
| *** ykarel__ has quit IRC | 05:28 | |
| *** Sukhdev has quit IRC | 05:35 | |
| *** gildub has quit IRC | 05:36 | |
| *** cshastri has joined #openstack-infra | 05:44 | |
| *** flepied has joined #openstack-infra | 05:48 | |
| *** dpaterson has quit IRC | 05:50 | |
| *** aeng has quit IRC | 05:51 | |
| *** gildub has joined #openstack-infra | 05:53 | |
| *** dhajare|busy has joined #openstack-infra | 05:53 | |
| *** jamielennox is now known as jamielennox|away | 05:56 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/diskimage-builder master: Updated from global requirements https://review.openstack.org/472077 | 05:58 |
| *** jamielennox|away is now known as jamielennox | 06:03 | |
| *** aeng has joined #openstack-infra | 06:04 | |
| *** armax has quit IRC | 06:08 | |
| *** armax has joined #openstack-infra | 06:08 | |
| *** armax has quit IRC | 06:09 | |
| *** esikache1 has joined #openstack-infra | 06:11 | |
| *** rcernin has joined #openstack-infra | 06:15 | |
| *** florianf has joined #openstack-infra | 06:16 | |
| *** esikache1 has quit IRC | 06:18 | |
| *** oanson has quit IRC | 06:20 | |
| *** eranrom has joined #openstack-infra | 06:20 | |
| *** oanson has joined #openstack-infra | 06:21 | |
| *** esikache1 has joined #openstack-infra | 06:23 | |
| *** Julien-zte has quit IRC | 06:26 | |
| *** esikache1 has quit IRC | 06:28 | |
| *** Guest42119 has quit IRC | 06:29 | |
| *** thorst_afk has joined #openstack-infra | 06:29 | |
| *** Guest42119 has joined #openstack-infra | 06:29 | |
| *** Guest42119 is now known as brucet | 06:31 | |
| *** zhurong has quit IRC | 06:32 | |
| *** thorst_afk has quit IRC | 06:34 | |
| *** makowals has joined #openstack-infra | 06:35 | |
| *** Julien-zte has joined #openstack-infra | 06:36 | |
| *** deafult has quit IRC | 06:37 | |
| *** guyk has quit IRC | 06:40 | |
| *** gomarivera has quit IRC | 06:41 | |
| *** gomarivera has joined #openstack-infra | 06:42 | |
| *** dizquierdo has joined #openstack-infra | 06:42 | |
| *** namnh_ has joined #openstack-infra | 06:45 | |
| *** eranrom has quit IRC | 06:45 | |
| *** gomarivera has quit IRC | 06:47 | |
| *** namnh has quit IRC | 06:47 | |
| *** esberglu has joined #openstack-infra | 06:49 | |
| *** dpaterson has joined #openstack-infra | 06:51 | |
| *** namnh_ has quit IRC | 06:51 | |
| *** jamesmcarthur has joined #openstack-infra | 06:51 | |
| *** gardlt has quit IRC | 06:52 | |
| *** gardlt has joined #openstack-infra | 06:53 | |
| *** esberglu has quit IRC | 06:54 | |
| *** jamesmcarthur has quit IRC | 06:56 | |
| *** pcaruana has joined #openstack-infra | 06:57 | |
| *** gardlt has quit IRC | 06:57 | |
| *** JudeC has joined #openstack-infra | 06:58 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Add env var to dump config graph https://review.openstack.org/472064 | 06:59 |
| *** rossella__ has joined #openstack-infra | 07:02 | |
| *** rossella__ has quit IRC | 07:07 | |
| *** pahuang has quit IRC | 07:10 | |
| *** tesseract has joined #openstack-infra | 07:12 | |
| *** elajkat_ has joined #openstack-infra | 07:12 | |
| *** shardy has joined #openstack-infra | 07:14 | |
| *** markus_z has joined #openstack-infra | 07:14 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Use picked nodes for later cmd_* calls https://review.openstack.org/469330 | 07:14 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Pass all blockdevices to bootloader https://review.openstack.org/471183 | 07:14 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Don't make image & loopdev functions static https://review.openstack.org/470845 | 07:14 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Move global mount tracking into state https://review.openstack.org/469752 | 07:14 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Move rollback into NodeBase object https://review.openstack.org/464105 | 07:14 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Use global state to check for duplicate fs labels https://review.openstack.org/469751 | 07:15 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Remove 'state' argument from later cmd_* calls https://review.openstack.org/469750 | 07:15 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Add state to NodeBase class https://review.openstack.org/469749 | 07:15 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: PPC bootloader; install to boot partition https://review.openstack.org/471525 | 07:15 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: [WIP] LVM support for dib-block-device https://review.openstack.org/472065 | 07:15 |
| *** esikache1 has joined #openstack-infra | 07:16 | |
| *** amoralej|off is now known as amoralej | 07:17 | |
| *** aviau_ has quit IRC | 07:19 | |
| *** rossella__ has joined #openstack-infra | 07:19 | |
| *** aviau has joined #openstack-infra | 07:19 | |
| *** oanson has quit IRC | 07:20 | |
| *** esikache1 has quit IRC | 07:20 | |
| *** oanson has joined #openstack-infra | 07:21 | |
| *** JudeC has quit IRC | 07:21 | |
| sileht | Hi, we have an issue with this review: https://review.openstack.org/#/c/469806/ zuul doesn't want to process it, I have try to 'recheck', to rebase, to reapprove, none want to trigger the gate jobs | 07:25 |
| *** esikache1 has joined #openstack-infra | 07:26 | |
| *** rkukura has quit IRC | 07:26 | |
| *** elajkat_ has quit IRC | 07:28 | |
| *** rkukura has joined #openstack-infra | 07:28 | |
| *** e0ne has joined #openstack-infra | 07:29 | |
| *** thorst_afk has joined #openstack-infra | 07:30 | |
| *** esikache1 has quit IRC | 07:30 | |
| *** oanson has quit IRC | 07:31 | |
| *** oanson has joined #openstack-infra | 07:31 | |
| *** jpich has joined #openstack-infra | 07:33 | |
| *** jpena|off is now known as jpena | 07:34 | |
| *** Julien-zte has quit IRC | 07:34 | |
| *** thorst_afk has quit IRC | 07:35 | |
| *** esikache1 has joined #openstack-infra | 07:35 | |
| *** jascott1 has quit IRC | 07:36 | |
| *** alexchadin has joined #openstack-infra | 07:36 | |
| *** jascott1 has joined #openstack-infra | 07:37 | |
| *** gcb has joined #openstack-infra | 07:39 | |
| *** esikache1 has quit IRC | 07:41 | |
| *** jaosorior has quit IRC | 07:42 | |
| *** jascott1 has quit IRC | 07:42 | |
| *** dizquierdo has quit IRC | 07:44 | |
| *** nicolasbock has joined #openstack-infra | 07:45 | |
| *** zhurong has joined #openstack-infra | 07:46 | |
| *** esikache1 has joined #openstack-infra | 07:47 | |
| *** shen has joined #openstack-infra | 07:48 | |
| *** ralonsoh has joined #openstack-infra | 07:48 | |
| *** ralonsoh has quit IRC | 07:49 | |
| *** jascott1 has joined #openstack-infra | 07:50 | |
| *** esikache1 has quit IRC | 07:51 | |
| *** jaosorior has joined #openstack-infra | 07:54 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool feature/zuulv3: Add webapp port and listen_address configuration https://review.openstack.org/472128 | 07:54 |
| shen | excuse me, can some one help me, please? When I setting up my CI test environment, I run 'sudo service nodepool start', the nodepool automate to create instance on my Openstack cloud, but when it try to ssh to the just created instance, I get a timeout error. I have also tried to create an instance myself, but I still can not ping to the instance. | 07:55 |
| *** ralonsoh has joined #openstack-infra | 07:56 | |
| *** tuanluong has joined #openstack-infra | 07:57 | |
| shen | can I create an instance with a public ip, and the instance can be ssh from my cloud host? | 07:59 |
| *** zzzeek has quit IRC | 08:00 | |
| *** zzzeek has joined #openstack-infra | 08:01 | |
| *** rossella__ has quit IRC | 08:04 | |
| *** e0ne has quit IRC | 08:04 | |
| *** esikache1 has joined #openstack-infra | 08:07 | |
| *** jascott1 has quit IRC | 08:09 | |
| *** sflanigan has quit IRC | 08:09 | |
| *** jascott1 has joined #openstack-infra | 08:09 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add congress-dashboard to $PROJECTS variable in murano-congress-job https://review.openstack.org/471538 | 08:09 |
| *** hashar has joined #openstack-infra | 08:11 | |
| *** esikache1 has quit IRC | 08:11 | |
| *** zhurong has quit IRC | 08:13 | |
| *** salv-orlando has quit IRC | 08:13 | |
| *** salv-orlando has joined #openstack-infra | 08:14 | |
| *** jascott1 has quit IRC | 08:14 | |
| *** esikache1 has joined #openstack-infra | 08:14 | |
| *** zhurong has joined #openstack-infra | 08:15 | |
| *** elajkat_ has joined #openstack-infra | 08:16 | |
| *** salv-orlando has quit IRC | 08:17 | |
| *** lucas-afk is now known as lucasagomes | 08:18 | |
| *** esikache1 has quit IRC | 08:19 | |
| *** rossella__ has joined #openstack-infra | 08:20 | |
| elajkat_ | Hi Guys, We are working on a feature in Heat to enable the handling of neutron trunks (aka VLAN aware VMs) | 08:20 |
| elajkat_ | For this we need devstack gate job(s) with trunk extension enabled. Who can help us to have that in Heat gate jobs, first as perhaps non-voting? | 08:22 |
| *** namnh has joined #openstack-infra | 08:25 | |
| *** e0ne has joined #openstack-infra | 08:25 | |
| *** links has quit IRC | 08:25 | |
| *** esikache1 has joined #openstack-infra | 08:26 | |
| *** jzerebecki has quit IRC | 08:28 | |
| *** jzerebecki has joined #openstack-infra | 08:28 | |
| *** prateek_ has joined #openstack-infra | 08:28 | |
| *** Julien-zte has joined #openstack-infra | 08:29 | |
| *** prateek has quit IRC | 08:30 | |
| *** Julien-zte has quit IRC | 08:30 | |
| *** thorst_afk has joined #openstack-infra | 08:31 | |
| *** eranrom has joined #openstack-infra | 08:31 | |
| *** Julien-zte has joined #openstack-infra | 08:31 | |
| *** prateek_ has quit IRC | 08:32 | |
| *** prateek has joined #openstack-infra | 08:33 | |
| *** prateek has quit IRC | 08:33 | |
| *** prateek has joined #openstack-infra | 08:34 | |
| *** isaacb has joined #openstack-infra | 08:34 | |
| *** prateek has quit IRC | 08:35 | |
| *** esikache1 has quit IRC | 08:35 | |
| *** prateek has joined #openstack-infra | 08:35 | |
| *** frickler_ is now known as frickler | 08:36 | |
| *** esberglu has joined #openstack-infra | 08:37 | |
| *** tosky has joined #openstack-infra | 08:38 | |
| openstackgerrit | Martin André proposed openstack-infra/tripleo-ci master: Workaround for limited disk space on undercloud https://review.openstack.org/471836 | 08:38 |
| openstackgerrit | Martin André proposed openstack-infra/tripleo-ci master: Remove leftover containers_minimal file https://review.openstack.org/472151 | 08:38 |
| *** links has joined #openstack-infra | 08:38 | |
| openstackgerrit | Nam Nguyen Hoai proposed openstack-infra/project-config master: Update barbican gate from experimental to non-voting https://review.openstack.org/472153 | 08:41 |
| *** esberglu has quit IRC | 08:42 | |
| *** prateek has quit IRC | 08:44 | |
| *** salv-orlando has joined #openstack-infra | 08:45 | |
| *** efoley has joined #openstack-infra | 08:47 | |
| *** electrofelix has joined #openstack-infra | 08:47 | |
| *** odyssey4me_ is now known as odyssey4me | 08:49 | |
| *** thorst_afk has quit IRC | 08:50 | |
| *** eranrom has quit IRC | 08:51 | |
| openstackgerrit | Nam Nguyen Hoai proposed openstack-infra/project-config master: Update barbican gate from experimental to non-voting https://review.openstack.org/472153 | 08:51 |
| *** winggundamth has joined #openstack-infra | 08:53 | |
| tushar | shen : the instance which is created has public and private ip both ? | 08:55 |
| *** yamamoto has quit IRC | 08:58 | |
| *** winggundamth has quit IRC | 08:58 | |
| tushar | Hi All | 08:59 |
| tushar | In our ci setup the DIB image creation is failing with error | 09:00 |
| tushar | 2017-06-08 02:30:02,294 INFO nodepool.image.build.dpc: Cloning into '/opt/dib_cache/source-repositories/devstack_9383d75c34a24bf0cd2ce50da1f4b74274dfb57b.tmp'... | 09:00 |
| tushar | 2017-06-08 02:34:21,936 INFO nodepool.image.build.dpc: fatal: unable to connect to git.openstack.org: | 09:00 |
| *** winggundamth has joined #openstack-infra | 09:00 | |
| tushar | Any one can help me to solve this? | 09:00 |
| *** stakeda has quit IRC | 09:02 | |
| *** rossella__ has quit IRC | 09:02 | |
| *** guyk has joined #openstack-infra | 09:06 | |
| guyk | Hi everyone, for our CI, we always have one test that keeps failing (tempest.api.compute.servers.test_server_rescue_negative.ServerRescueNegativeTestJSON). The issue is due to the timeout to detach the volume from the instance. We upgraded nodepool element and it is using xenial. Anyone has met this issue or have any clue about it? Thank you for your help. | 09:07 |
| *** esikache1 has joined #openstack-infra | 09:11 | |
| tushar | guyk : for which component you are running the CI ? | 09:14 |
| *** esikache1 has quit IRC | 09:19 | |
| *** rossella__ has joined #openstack-infra | 09:19 | |
| openstackgerrit | Ricardo Noriega proposed openstack-infra/project-config master: Skipping OVSDB configuration for now https://review.openstack.org/472175 | 09:20 |
| *** esikache1 has joined #openstack-infra | 09:21 | |
| guyk | tushar: we are running it for cinder. | 09:24 |
| *** prateek has joined #openstack-infra | 09:25 | |
| tushar | guyk : not sure | 09:29 |
| tushar | It is passing for me | 09:29 |
| tushar | we are running with Xenial | 09:30 |
| guyk | tushar: sometimes it passed too | 09:31 |
| *** alexchadin has quit IRC | 09:31 | |
| *** alexchadin has joined #openstack-infra | 09:31 | |
| tushar | guyk : you need to look into your storage backend | 09:32 |
| *** gongysh has joined #openstack-infra | 09:32 | |
| *** cshastri has quit IRC | 09:33 | |
| *** esikache1 has quit IRC | 09:33 | |
| guyk | tushar: we have already checked our storage backend. The detach operation takes few seconds. Besides, after looking in nova-compute logs, we found this error: "DeviceDetachFailed: Device detach failed for vdb: Unable to detach from guest transient domain." | 09:34 |
| *** efoley has quit IRC | 09:34 | |
| *** dizquierdo has joined #openstack-infra | 09:34 | |
| *** esberglu has joined #openstack-infra | 09:35 | |
| *** MatthewLi has quit IRC | 09:36 | |
| *** sambetts|afk is now known as sambetts | 09:38 | |
| *** alexchadin has quit IRC | 09:40 | |
| *** esberglu has quit IRC | 09:40 | |
| *** cshastri has joined #openstack-infra | 09:43 | |
| openstackgerrit | Alfredo Moralejo proposed openstack-infra/tripleo-ci master: DNM: testing something for gate https://review.openstack.org/471361 | 09:44 |
| openstackgerrit | Merged openstack-infra/project-config master: Skipping OVSDB configuration for now https://review.openstack.org/472175 | 09:45 |
| *** esikache1 has joined #openstack-infra | 09:46 | |
| *** rwsu has quit IRC | 09:46 | |
| *** thorst_afk has joined #openstack-infra | 09:47 | |
| *** panda|off is now known as panda | 09:49 | |
| *** jamesmcarthur has joined #openstack-infra | 09:51 | |
| *** thorst_afk has quit IRC | 09:51 | |
| *** esikache1 has quit IRC | 09:52 | |
| *** bakbad has quit IRC | 09:53 | |
| *** esikache1 has joined #openstack-infra | 09:54 | |
| *** jamesmcarthur has quit IRC | 09:55 | |
| *** esikache1 has quit IRC | 09:59 | |
| *** kaisers has quit IRC | 10:00 | |
| *** mat128 has joined #openstack-infra | 10:05 | |
| *** liujiong has quit IRC | 10:06 | |
| *** sdague has joined #openstack-infra | 10:11 | |
| *** esikache1 has joined #openstack-infra | 10:15 | |
| *** kaisers has joined #openstack-infra | 10:16 | |
| *** esikache1 has quit IRC | 10:20 | |
| jamespage | anyone have some time to help me figure out why the nova-lxd devstack gate experimental check bails really early - example - gate-tempest-dsvm-lxd-nv | 10:24 |
| jamespage | https://review.openstack.org/#/c/469556 | 10:24 |
| jamespage | or that even.. | 10:24 |
| jamespage | devstack-early indicates "./stack.sh: line 506: generate-subunit: command not found" | 10:25 |
| jamespage | but I'm not clear on why that's required or why its not installed... | 10:25 |
| jamespage | oh that's coming from the exit trap, so its something else actually failing | 10:27 |
| *** markvoelker has quit IRC | 10:27 | |
| *** MatthewLi has joined #openstack-infra | 10:27 | |
| *** markvoelker has joined #openstack-infra | 10:27 | |
| *** flepied has quit IRC | 10:28 | |
| openstackgerrit | Andrea Frittoli proposed openstack-infra/irc-meetings master: Moving QA meeting to 8:00UTC https://review.openstack.org/472194 | 10:31 |
| *** dtantsur|afk is now known as dtantsur | 10:31 | |
| *** med_ has quit IRC | 10:31 | |
| *** med_ has joined #openstack-infra | 10:35 | |
| *** wolverineav has joined #openstack-infra | 10:35 | |
| *** med_ is now known as Guest72612 | 10:35 | |
| openstackgerrit | Alfredo Moralejo proposed openstack-infra/tripleo-ci master: DNM: testing something for gate https://review.openstack.org/471361 | 10:36 |
| ianw | jamespage: http://logs.openstack.org/56/469556/4/experimental/gate-tempest-dsvm-lxd-nv/d2ad4d4/logs/devstack-early.txt.gz ... none of that should be happening. it seems like there's missing local configuration | 10:37 |
| jamespage | ianw: hmm yes - I'm struggling to figure out why its missing; let me dig further | 10:38 |
| jamespage | I wonder if the nova-lxd pre-test script is foobaring it | 10:38 |
| jamespage | ah yes | 10:38 |
| jamespage | I think it see it | 10:38 |
| jamespage | https://www.irccloud.com/pastebin/TIpNVVDz/ | 10:39 |
| *** dizquierdo has quit IRC | 10:39 | |
| jamespage | that's probably overriding the local.conf generated by devstack-gate | 10:39 |
| * jamespage goes to resolve that | 10:39 | |
| *** sree has quit IRC | 10:39 | |
| ianw | jamespage: i think it's the localrc ... where's that coming from? | 10:44 |
| *** salv-orlando has quit IRC | 10:44 | |
| jamespage | ianw: hook provided by nova-lxd - re-writing that now | 10:44 |
| *** salv-orlando has joined #openstack-infra | 10:44 | |
| ianw | yeah, your link there. use local.conf and i think it will all just work | 10:45 |
| jamespage | ianw: yeah - irccloud is sometime to helpful :-) | 10:46 |
| *** pahuang has joined #openstack-infra | 10:48 | |
| *** jkilpatr has joined #openstack-infra | 11:00 | |
| *** rhallisey has joined #openstack-infra | 11:02 | |
| *** amoralej is now known as amoralej|lunch | 11:05 | |
| *** shen has quit IRC | 11:06 | |
| *** kaisers has quit IRC | 11:06 | |
| *** jpena is now known as jpena|lunch | 11:07 | |
| *** gildub has quit IRC | 11:08 | |
| *** rhallisey has quit IRC | 11:10 | |
| *** rhallisey has joined #openstack-infra | 11:10 | |
| *** mugsie has quit IRC | 11:12 | |
| *** mugsie has joined #openstack-infra | 11:14 | |
| *** mugsie has joined #openstack-infra | 11:14 | |
| *** Julien-zte has quit IRC | 11:15 | |
| *** kaisers has joined #openstack-infra | 11:23 | |
| *** lucasagomes is now known as lucas-hungry | 11:26 | |
| *** ccamacho is now known as ccamacho|lunch | 11:30 | |
| *** pgadiya has joined #openstack-infra | 11:30 | |
| *** pgadiya has quit IRC | 11:30 | |
| *** kgiusti has quit IRC | 11:33 | |
| *** eranrom has joined #openstack-infra | 11:36 | |
| *** gongysh has quit IRC | 11:37 | |
| *** dtantsur is now known as dtantsur|sick | 11:37 | |
| mordred | tristanC: your nodepool patch for webapp port had a sad: http://logs.openstack.org/28/472128/1/check/gate-dsvm-nodepool/25ac413/logs/screen-nodepool-launcher.txt.gz | 11:38 |
| *** mikal has quit IRC | 11:41 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Extend in-repo config update support to github https://review.openstack.org/471477 | 11:41 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Re-enable test_worker_update_metadata https://review.openstack.org/468667 | 11:42 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Re-enable test_merge_failure_reporters https://review.openstack.org/468668 | 11:42 |
| *** gomarivera has joined #openstack-infra | 11:43 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Fix race in test_crd_gate_unknown https://review.openstack.org/471833 | 11:43 |
| *** mikal has joined #openstack-infra | 11:43 | |
| *** esikache1 has joined #openstack-infra | 11:44 | |
| *** jani has joined #openstack-infra | 11:44 | |
| *** cuongnv has quit IRC | 11:45 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Move log streaming to share with zuul_console https://review.openstack.org/471921 | 11:45 |
| *** thorst_afk has joined #openstack-infra | 11:46 | |
| *** gomarivera has quit IRC | 11:47 | |
| *** mugsie has quit IRC | 11:47 | |
| *** esikache1 has quit IRC | 11:48 | |
| *** mugsie has joined #openstack-infra | 11:49 | |
| *** mugsie has quit IRC | 11:49 | |
| *** mugsie has joined #openstack-infra | 11:49 | |
| *** wolverineav has quit IRC | 11:52 | |
| *** eranrom has quit IRC | 11:52 | |
| *** wolverineav has joined #openstack-infra | 11:52 | |
| *** abishop has joined #openstack-infra | 11:53 | |
| openstackgerrit | Attila Darazs proposed openstack-infra/project-config master: tripleo: replace obsolete nonha job with oooq variant https://review.openstack.org/472226 | 11:55 |
| *** esikache1 has joined #openstack-infra | 11:56 | |
| *** hwoarang_ is now known as hwoarang | 11:56 | |
| *** ldnunes has joined #openstack-infra | 11:56 | |
| *** jpena|lunch is now known as jpena | 11:57 | |
| *** aeng_ has joined #openstack-infra | 11:58 | |
| *** lucas-hungry is now known as lucasagomes | 12:10 | |
| *** jascott1 has joined #openstack-infra | 12:11 | |
| sileht | Hi, we have an issue with this review: https://review.openstack.org/#/c/469806/ zuul doesn't want to process it, I have try to 'recheck', to rebase, to reapprove, none want to trigger the gate jobs | 12:12 |
| *** amoralej|lunch is now known as amoralej | 12:12 | |
| *** ldnunes has quit IRC | 12:13 | |
| *** mriedem has joined #openstack-infra | 12:13 | |
| *** esikache1 has quit IRC | 12:14 | |
| *** ldnunes has joined #openstack-infra | 12:14 | |
| *** esikachev has joined #openstack-infra | 12:14 | |
| *** jascott1 has quit IRC | 12:15 | |
| *** wolverineav has quit IRC | 12:20 | |
| *** wolverineav has joined #openstack-infra | 12:20 | |
| frickler | sileht: https://review.openstack.org/#/c/470759/ depends on the same change-id, so it is blocking | 12:21 |
| *** jamesdenton has joined #openstack-infra | 12:21 | |
| sileht | frickler, but it's abandonned and not on same branch | 12:21 |
| frickler | sileht: zuul doesn't care about branches in that case and it seems also not about abandoned, though the latter may be considered a bug | 12:22 |
| *** winggundamth has quit IRC | 12:22 | |
| *** tuanluong has quit IRC | 12:23 | |
| frickler | sileht: but I had a similar issue some time ago and removing the depends-on solved it | 12:23 |
| sileht | ok I see | 12:23 |
| sileht | it's because of DependOn: + same change id | 12:24 |
| *** lewo has quit IRC | 12:24 | |
| frickler | yes | 12:24 |
| *** lewo has joined #openstack-infra | 12:24 | |
| frickler | and zuul notices the cyclic dependency and goes belly up | 12:24 |
| *** wolverineav has quit IRC | 12:25 | |
| *** krtaylor has quit IRC | 12:25 | |
| sileht | frickler, thx, I haven't noticed the DependsOn pointing to itself | 12:26 |
| sileht | frickler, I removed it and recheck and it works now ! | 12:26 |
| sileht | frickler, thx a lot | 12:26 |
| frickler | yw | 12:27 |
| *** ianychoi has quit IRC | 12:27 | |
| *** ianychoi has joined #openstack-infra | 12:28 | |
| *** rossella__ has quit IRC | 12:28 | |
| *** wolverineav has joined #openstack-infra | 12:29 | |
| *** derekh has joined #openstack-infra | 12:31 | |
| *** namnh has quit IRC | 12:33 | |
| *** rlandy has joined #openstack-infra | 12:33 | |
| *** mat128 has quit IRC | 12:34 | |
| *** sree has joined #openstack-infra | 12:35 | |
| *** gordc has joined #openstack-infra | 12:35 | |
| *** zhurong has quit IRC | 12:36 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool feature/zuulv3: Add webapp port and listen_address configuration https://review.openstack.org/472128 | 12:36 |
| *** krtaylor has joined #openstack-infra | 12:38 | |
| *** flepied has joined #openstack-infra | 12:39 | |
| tristanC | mordred: oops, thanks for the link! | 12:40 |
| *** rfolco has joined #openstack-infra | 12:41 | |
| *** bh526r has joined #openstack-infra | 12:42 | |
| *** trown|outtypewww is now known as trown | 12:42 | |
| *** rossella__ has joined #openstack-infra | 12:43 | |
| *** pradk has joined #openstack-infra | 12:45 | |
| *** jroll has left #openstack-infra | 12:46 | |
| *** flepied has quit IRC | 12:50 | |
| *** askb has quit IRC | 12:50 | |
| *** kgiusti has joined #openstack-infra | 12:51 | |
| *** ihrachys has joined #openstack-infra | 12:53 | |
| pabelanger | ovh-bhs1 back to normal, some of our compute hosts were in maintenance. New ones have been added now | 12:53 |
| pabelanger | am seeing a fair bit of launch failures in citcy-cloud-sto2 | 12:54 |
| pabelanger | looking into that now | 12:54 |
| amoralej | pabelanger, we are having problems to reproduce and debug a issue we are hitting in a gate, job, would it be possible to not destroy the instances after the job and get access into it? | 12:54 |
| *** yamamoto has joined #openstack-infra | 12:55 | |
| pabelanger | amoralej: usually, we can but an autohold on the job which prevents nodepool from deleting it. Then we ask for your SSH public key | 12:55 |
| jkilpatr | can someone explain to me how I'm duplicating a docs job? | 12:55 |
| jkilpatr | https://review.openstack.org/#/c/470953/4 | 12:55 |
| *** flepied has joined #openstack-infra | 12:56 | |
| pabelanger | jkilpatr: IIRC, it already gets defined in ansible-role-jobs | 12:57 |
| jkilpatr | so remove docs on rtfd? | 12:57 |
| pabelanger | jkilpatr: so you likley want to drop python-jobs and inlcude the specific job template | 12:57 |
| amoralej | pabelanger, that autohold would affect to all instances of a specific job, in this case gate-tripleo-ci-centos-7-ovb-ha , or for a specific review? | 12:57 |
| pabelanger | jkilpatr: no, that is how you publish docs to rtfd | 12:57 |
| amoralej | in this case 471361,5 | 12:57 |
| *** dizquierdo has joined #openstack-infra | 12:58 | |
| pabelanger | amoralej: right, any failure would trigger the hold | 12:58 |
| pabelanger | amoralej: if there is a specific node you want to hold, we need to nodepool ID of the server | 12:58 |
| *** esikachev has quit IRC | 12:58 | |
| *** mriedem1 has joined #openstack-infra | 12:59 | |
| openstackgerrit | Justin Kilpatrick proposed openstack-infra/project-config master: Add seperate pep8 job for Browbeat https://review.openstack.org/470953 | 12:59 |
| pabelanger | jkilpatr: left comment | 13:00 |
| *** mriedem has quit IRC | 13:00 | |
| amoralej | pabelanger, like http://paste.openstack.org/show/611902/? | 13:02 |
| *** mriedem1 is now known as mriedem | 13:02 | |
| amoralej | however this is multihost | 13:02 |
| *** esikachev has joined #openstack-infra | 13:02 | |
| *** bobh has joined #openstack-infra | 13:02 | |
| amoralej | i'd need all instances to be on hold | 13:02 |
| pabelanger | amoralej: right, 9209046 is the id we use in nodepool | 13:02 |
| *** baoli has joined #openstack-infra | 13:02 | |
| pabelanger | amoralej: okay, held. We'll need your publish SSH key | 13:03 |
| *** gongysh has joined #openstack-infra | 13:03 | |
| *** gongysh has quit IRC | 13:03 | |
| *** LindaWang has quit IRC | 13:03 | |
| amoralej | pabelanger did you hold only that or all instances assigned to the job? | 13:04 |
| amoralej | because of the multihost | 13:04 |
| pabelanger | amoralej: we'll only hold the instance that nodepool directly creates, so if OVB or heat deletes things, we have no control over that | 13:04 |
| amoralej | ok, thanks | 13:04 |
| amoralej | that's good | 13:04 |
| pabelanger | amoralej: you'll have to update your jobs to not delete things | 13:04 |
| *** mdrabe has joined #openstack-infra | 13:04 | |
| amoralej | pabelanger https://github.com/amoralej.keys | 13:04 |
| *** trown is now known as trown|brb | 13:05 | |
| pabelanger | amoralej: ssh jenkins@66.187.229.142 | 13:05 |
| amoralej | thanks pabelanger++ | 13:05 |
| pabelanger | amoralej: if you can let me know when you are finished in a few hours, we can unhold and release the node back to the cloud | 13:05 |
| amoralej | sure | 13:06 |
| *** esberglu has joined #openstack-infra | 13:06 | |
| *** esikachev has quit IRC | 13:07 | |
| *** mat128 has joined #openstack-infra | 13:09 | |
| *** mat128 has quit IRC | 13:10 | |
| *** esikachev has joined #openstack-infra | 13:10 | |
| *** jamesdenton has quit IRC | 13:10 | |
| *** yamamoto has quit IRC | 13:10 | |
| *** wolverineav has quit IRC | 13:11 | |
| *** dave-mccowan has joined #openstack-infra | 13:12 | |
| *** yamamoto has joined #openstack-infra | 13:14 | |
| *** esikachev has quit IRC | 13:14 | |
| *** mat128 has joined #openstack-infra | 13:15 | |
| *** gardlt has joined #openstack-infra | 13:15 | |
| *** gardlt has quit IRC | 13:15 | |
| *** links has quit IRC | 13:15 | |
| *** gardlt has joined #openstack-infra | 13:16 | |
| *** atarakt has quit IRC | 13:19 | |
| *** funzo_ is now known as funzo | 13:19 | |
| *** atarakt has joined #openstack-infra | 13:19 | |
| *** Goneri has joined #openstack-infra | 13:19 | |
| *** wolverineav has joined #openstack-infra | 13:20 | |
| *** dhajare|busy has quit IRC | 13:23 | |
| *** esikachev has joined #openstack-infra | 13:24 | |
| *** sbezverk has joined #openstack-infra | 13:25 | |
| *** esikachev has quit IRC | 13:28 | |
| *** trown|brb is now known as trown | 13:29 | |
| *** jcoufal has joined #openstack-infra | 13:30 | |
| *** esikachev has joined #openstack-infra | 13:33 | |
| *** sbezverk has quit IRC | 13:34 | |
| *** gomarivera has joined #openstack-infra | 13:36 | |
| *** gomarivera has quit IRC | 13:36 | |
| *** esikachev has quit IRC | 13:37 | |
| *** gomarivera has joined #openstack-infra | 13:37 | |
| *** asselin_ has joined #openstack-infra | 13:38 | |
| *** esikachev has joined #openstack-infra | 13:38 | |
| *** eranrom has joined #openstack-infra | 13:38 | |
| *** dhajare|busy has joined #openstack-infra | 13:39 | |
| *** prateek has quit IRC | 13:40 | |
| *** asselin has quit IRC | 13:40 | |
| *** asselin__ has joined #openstack-infra | 13:42 | |
| *** esikachev has quit IRC | 13:43 | |
| *** mhickey has joined #openstack-infra | 13:43 | |
| *** asselin_ has quit IRC | 13:45 | |
| openstackgerrit | Rosario Di Somma proposed openstack-infra/shade master: Don't remove top-container element for volume REST API calls https://review.openstack.org/472268 | 13:47 |
| *** dizquierdo has quit IRC | 13:49 | |
| *** flepied has quit IRC | 13:52 | |
| *** ianychoi_ has joined #openstack-infra | 13:54 | |
| *** dhajare|busy has quit IRC | 13:55 | |
| openstackgerrit | James Page proposed openstack-infra/project-config master: Increase nova-lxd functional test timeout https://review.openstack.org/472270 | 13:55 |
| openstackgerrit | Merged openstack-infra/tripleo-ci master: Remove leftover containers_minimal file https://review.openstack.org/472151 | 13:56 |
| openstackgerrit | Merged openstack-infra/tripleo-ci master: Workaround for limited disk space on undercloud https://review.openstack.org/471836 | 13:56 |
| *** ianychoi has quit IRC | 13:57 | |
| *** ianychoi_ is now known as ianychoi | 13:57 | |
| *** EricGonczer_ has joined #openstack-infra | 13:57 | |
| *** jamesmcarthur has joined #openstack-infra | 13:57 | |
| *** matt-borland has joined #openstack-infra | 13:57 | |
| *** aeng_ has quit IRC | 13:58 | |
| *** jamesdenton has joined #openstack-infra | 13:59 | |
| openstackgerrit | Merged openstack-infra/system-config master: Add gerrit information to ze01.o.o https://review.openstack.org/471877 | 14:00 |
| *** ramishra has quit IRC | 14:00 | |
| *** ggnel_t has quit IRC | 14:00 | |
| *** pahuang has quit IRC | 14:00 | |
| *** krtaylor has quit IRC | 14:01 | |
| *** dhajare|busy has joined #openstack-infra | 14:02 | |
| *** chlong has joined #openstack-infra | 14:02 | |
| *** LindaWang has joined #openstack-infra | 14:03 | |
| *** jcoufal has quit IRC | 14:03 | |
| *** ZZelle has quit IRC | 14:03 | |
| pabelanger | fungi: clarkb: do we want to remove the track-upstream field for debian packaging projects? I am not sure anybody is working on them any more | 14:03 |
| *** zz_dimtruck is now known as dimtruck | 14:03 | |
| *** ZZelle has joined #openstack-infra | 14:03 | |
| *** spzala has joined #openstack-infra | 14:04 | |
| fungi | pabelanger: probably worth checking with zigo and wendar on that | 14:04 |
| pabelanger | fungi: agree, was going to send out a ML post, but wanted to check first on our side | 14:04 |
| fungi | i don't have an opinion on the matter | 14:05 |
| wendar | pabelanger: at the moment, as far as I know, no one is working on it | 14:05 |
| wendar | pabelanger: zigo told me to stop, that he was taking it back on | 14:05 |
| wendar | pabelanger: so, for now it's stalled. I may run for PTL next cycle and take it back on | 14:05 |
| wendar | but... I think we'd want to do the setup differently | 14:05 |
| pabelanger | fungi: ya, only noticed it yesteday when manage-project ran. Figured if nobody is using it, we could save some resources, HDD space | 14:05 |
| wendar | (as we talked about at the PTG) | 14:05 |
| pabelanger | wendar: right, that would mean no more needed for track-upstream too | 14:06 |
| wendar | pabelanger: I'm fine if you want to stop it :) | 14:06 |
| pabelanger | okay, I'll get a ML post up first | 14:06 |
| *** sbezverk has joined #openstack-infra | 14:07 | |
| *** gardlt_ has joined #openstack-infra | 14:07 | |
| *** gardlt has quit IRC | 14:08 | |
| *** florianf is now known as florianf|afk | 14:09 | |
| *** rbrndt has joined #openstack-infra | 14:09 | |
| *** spzala has quit IRC | 14:10 | |
| *** ykarel_ has quit IRC | 14:12 | |
| *** ZZelle has quit IRC | 14:12 | |
| *** jamesdenton has quit IRC | 14:12 | |
| *** ZZelle has joined #openstack-infra | 14:12 | |
| *** lbragstad has joined #openstack-infra | 14:14 | |
| *** ZZelle has quit IRC | 14:14 | |
| *** panda is now known as panda|bbl | 14:14 | |
| *** eharney has joined #openstack-infra | 14:16 | |
| *** r-daneel has joined #openstack-infra | 14:17 | |
| *** felipemonteiro_ has joined #openstack-infra | 14:18 | |
| *** yolanda has quit IRC | 14:24 | |
| *** jamesdenton has joined #openstack-infra | 14:25 | |
| *** sbezverk has quit IRC | 14:26 | |
| *** LindaWang has quit IRC | 14:27 | |
| *** LindaWang has joined #openstack-infra | 14:27 | |
| *** yolanda has joined #openstack-infra | 14:28 | |
| *** hieulq_ has joined #openstack-infra | 14:29 | |
| *** Julien-zte has joined #openstack-infra | 14:29 | |
| *** felipemonteiro__ has joined #openstack-infra | 14:30 | |
| *** bobh has quit IRC | 14:31 | |
| *** felipemonteiro_ has quit IRC | 14:34 | |
| *** jamesdenton has quit IRC | 14:35 | |
| *** ykarel has joined #openstack-infra | 14:35 | |
| *** xarses has joined #openstack-infra | 14:39 | |
| *** rhallisey has quit IRC | 14:40 | |
| *** andreww has joined #openstack-infra | 14:41 | |
| *** sree has quit IRC | 14:43 | |
| *** claudiub has quit IRC | 14:43 | |
| *** mat128 has quit IRC | 14:43 | |
| *** xarses has quit IRC | 14:44 | |
| *** salv-orl_ has joined #openstack-infra | 14:44 | |
| *** xarses has joined #openstack-infra | 14:45 | |
| *** andreww has quit IRC | 14:45 | |
| *** andreww has joined #openstack-infra | 14:46 | |
| *** LindaWang has quit IRC | 14:47 | |
| *** salv-orlando has quit IRC | 14:47 | |
| *** armax has joined #openstack-infra | 14:47 | |
| *** cshastri has quit IRC | 14:49 | |
| *** xarses has quit IRC | 14:50 | |
| *** hongbin has joined #openstack-infra | 14:50 | |
| *** andreww has quit IRC | 14:51 | |
| openstackgerrit | Igor Duarte Cardoso proposed openstack-infra/project-config master: Add gerritbot for networking-sfc https://review.openstack.org/472282 | 14:51 |
| *** andreww has joined #openstack-infra | 14:52 | |
| *** xarses has joined #openstack-infra | 14:53 | |
| *** spzala has joined #openstack-infra | 14:54 | |
| *** rbrndt has quit IRC | 14:55 | |
| *** andreww has quit IRC | 14:56 | |
| *** psachin has quit IRC | 14:57 | |
| *** andreww has joined #openstack-infra | 14:57 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Add a test which exercises a speculative role checkout https://review.openstack.org/462677 | 14:57 |
| *** xarses has quit IRC | 14:58 | |
| *** dhajare|busy has quit IRC | 14:58 | |
| *** xarses has joined #openstack-infra | 14:58 | |
| *** sree has joined #openstack-infra | 14:59 | |
| amoralej | pabelanger, you can destroy the node when you want | 14:59 |
| amoralej | an thanks for your help | 14:59 |
| pabelanger | amoralej: thanks | 15:01 |
| *** sree has quit IRC | 15:02 | |
| *** andreww has quit IRC | 15:02 | |
| *** andreww has joined #openstack-infra | 15:03 | |
| *** xarses has quit IRC | 15:03 | |
| *** xarses has joined #openstack-infra | 15:04 | |
| *** LindaWang has joined #openstack-infra | 15:06 | |
| *** andreww has quit IRC | 15:07 | |
| *** lbragstad has quit IRC | 15:08 | |
| *** xarses has quit IRC | 15:08 | |
| *** andreww has joined #openstack-infra | 15:08 | |
| pabelanger | mordred: maybe if you have a moment, could you help me figure out why ze01.o.o does not have the group zuul-executor on puppetmaster? It is failing to copy our hieradata files and unable to figure out why | 15:09 |
| *** xarses has joined #openstack-infra | 15:09 | |
| pabelanger | I believe I have the group variable setup properly in site.pp | 15:10 |
| *** yamahata has quit IRC | 15:11 | |
| mordred | pabelanger: do you have it in ./modules/openstack_project/files/puppetmaster/groups.txt ? | 15:11 |
| fungi | right, needs to be set in both files (for now at least( | 15:12 |
| pabelanger | Oh, I don't think so | 15:12 |
| pabelanger | that is likely it | 15:12 |
| *** jascott1 has joined #openstack-infra | 15:12 | |
| pabelanger | danke | 15:12 |
| *** andreww has quit IRC | 15:13 | |
| *** mat128 has joined #openstack-infra | 15:13 | |
| *** xarses has quit IRC | 15:14 | |
| *** andreww has joined #openstack-infra | 15:14 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Add zuul-executor to groups.txt https://review.openstack.org/472288 | 15:14 |
| pabelanger | mordred: fungi: ^ if you don't mind a review | 15:15 |
| *** xarses has joined #openstack-infra | 15:16 | |
| *** rcernin has quit IRC | 15:16 | |
| openstackgerrit | Igor Duarte Cardoso proposed openstack-infra/project-config master: Add the #networking-sfc channel https://review.openstack.org/472282 | 15:16 |
| *** hashar has quit IRC | 15:17 | |
| *** jascott1 has quit IRC | 15:17 | |
| asettle | fungi: as per conversation with Doug yesterday, spec is up: https://review.openstack.org/#/c/472275/ | 15:18 |
| asettle | Ah! And I've already added you to the reviewers list. Ugh. I'm so scrambled. | 15:18 |
| *** andreww has quit IRC | 15:18 | |
| fungi | asettle: no problem, i saw the post to both mailing lists too | 15:19 |
| asettle | fungi: I'm an email happy soul | 15:20 |
| * asettle throws email confetti | 15:20 | |
| fungi | it's good to be a happy soul | 15:20 |
| fungi | e-mail or otherwise ;) | 15:20 |
| asettle | I'm happier now the spec is written? :p | 15:20 |
| fungi | accomplishments! | 15:20 |
| asettle | Exactly! | 15:20 |
| * asettle is ignoring the fact that she just threw the spec into the big black hole that is 'the community' | 15:21 | |
| fungi | hopefully we'll get to it before the hawking radiation does | 15:23 |
| asettle | Aw that's promising. | 15:23 |
| *** ccamacho|lunch is now known as ccamacho | 15:24 | |
| pabelanger | stepping out for an errand should be able in 2 hours | 15:25 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Add tenant to url formatting. https://review.openstack.org/463628 | 15:25 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Expose buildset to the executor and url formatter https://review.openstack.org/463457 | 15:26 |
| *** yamamoto has quit IRC | 15:30 | |
| openstackgerrit | Felipe Monteiro proposed openstack-infra/project-config master: Use keystone uuid tokens to optimize test run time https://review.openstack.org/470399 | 15:31 |
| openstackgerrit | Merged openstack-infra/system-config master: Add zuul-executor to groups.txt https://review.openstack.org/472288 | 15:33 |
| *** makowals has quit IRC | 15:34 | |
| *** yamamoto has joined #openstack-infra | 15:35 | |
| *** makowals has joined #openstack-infra | 15:35 | |
| *** bobh has joined #openstack-infra | 15:35 | |
| *** jgriffith_ is now known as jgriffith | 15:36 | |
| *** bobh has quit IRC | 15:37 | |
| *** bobh has joined #openstack-infra | 15:37 | |
| *** ykarel has quit IRC | 15:38 | |
| *** yamamoto has quit IRC | 15:39 | |
| *** armax has quit IRC | 15:40 | |
| openstackgerrit | Ian Y. Choi proposed openstack-infra/infra-specs master: New spec for translation_check_site (Ansible AIO) https://review.openstack.org/440825 | 15:40 |
| *** gyee has joined #openstack-infra | 15:42 | |
| *** mdrabe has quit IRC | 15:44 | |
| openstackgerrit | Tom Barron proposed openstack-infra/project-config master: Promote manila lvm centos 7 job to voting https://review.openstack.org/472302 | 15:45 |
| *** LindaWang has quit IRC | 15:47 | |
| *** rwsu has joined #openstack-infra | 15:48 | |
| *** jcoufal has joined #openstack-infra | 15:48 | |
| *** e0ne has quit IRC | 15:48 | |
| *** Julien-zte has quit IRC | 15:49 | |
| openstackgerrit | Ian Y. Choi proposed openstack-infra/infra-specs master: New spec for translation_check_site (Ansible AIO) https://review.openstack.org/440825 | 15:49 |
| *** mdrabe has joined #openstack-infra | 15:50 | |
| *** Julien-zte has joined #openstack-infra | 15:51 | |
| *** Julien-zte has quit IRC | 15:51 | |
| *** clayton has quit IRC | 15:51 | |
| *** markus_z has quit IRC | 15:51 | |
| *** Apoorva has joined #openstack-infra | 15:52 | |
| *** Apoorva has quit IRC | 15:52 | |
| *** Julien-zte has joined #openstack-infra | 15:53 | |
| *** Julien-zte has quit IRC | 15:53 | |
| *** clayton has joined #openstack-infra | 15:54 | |
| *** eharney has quit IRC | 15:57 | |
| *** lbragstad has joined #openstack-infra | 15:57 | |
| *** gyee has quit IRC | 15:58 | |
| *** jaosorior has quit IRC | 16:00 | |
| *** Apoorva has joined #openstack-infra | 16:01 | |
| *** gomarivera has quit IRC | 16:01 | |
| *** rlandy is now known as rlandy|brb | 16:05 | |
| *** EricGonc_ has joined #openstack-infra | 16:06 | |
| *** jamesmcarthur has quit IRC | 16:06 | |
| *** isaacb has quit IRC | 16:07 | |
| openstackgerrit | Merged openstack-infra/shade master: Add links to user list dict https://review.openstack.org/471474 | 16:07 |
| *** EricGonczer_ has quit IRC | 16:07 | |
| *** jamesmcarthur has joined #openstack-infra | 16:08 | |
| *** mat128 has quit IRC | 16:08 | |
| *** eharney has joined #openstack-infra | 16:08 | |
| *** ykarel has joined #openstack-infra | 16:08 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack-infra/shade master: Updated from global requirements https://review.openstack.org/472310 | 16:15 |
| *** EricGonczer_ has joined #openstack-infra | 16:16 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/diskimage-builder master: Updated from global requirements https://review.openstack.org/472077 | 16:16 |
| openstackgerrit | Merged openstack-infra/shade master: Use get_discovery from keystoneauth https://review.openstack.org/471501 | 16:16 |
| *** gyee has joined #openstack-infra | 16:16 | |
| *** EricGonc_ has quit IRC | 16:17 | |
| *** jamesdenton has joined #openstack-infra | 16:17 | |
| *** trown is now known as trown|lunch | 16:18 | |
| *** ykarel has quit IRC | 16:20 | |
| *** ykarel has joined #openstack-infra | 16:21 | |
| *** rlandy|brb is now known as rlandy | 16:21 | |
| *** esikachev has joined #openstack-infra | 16:22 | |
| Ng | hey folks | 16:24 |
| Ng | how do I figure out when a periodic CI job runs? | 16:24 |
| Ng | (I'm looking to get the last ~week of results from TripleO's centos-7-scenario002-multinode-oooq) | 16:25 |
| *** esikachev has quit IRC | 16:26 | |
| *** lucasagomes is now known as lucas-afk | 16:26 | |
| openstackgerrit | Merged openstack-infra/shade master: Don't remove top-container element for volume REST API calls https://review.openstack.org/472268 | 16:28 |
| *** jpich has quit IRC | 16:29 | |
| *** sbezverk has joined #openstack-infra | 16:29 | |
| *** esikachev has joined #openstack-infra | 16:29 | |
| *** dizquierdo has joined #openstack-infra | 16:31 | |
| *** shardy has quit IRC | 16:31 | |
| *** rhallisey has joined #openstack-infra | 16:31 | |
| *** shardy has joined #openstack-infra | 16:32 | |
| *** tesseract has quit IRC | 16:32 | |
| *** jkilpatr_ has joined #openstack-infra | 16:33 | |
| *** jamesmcarthur has quit IRC | 16:33 | |
| *** rbrndt has joined #openstack-infra | 16:34 | |
| *** jamesmcarthur has joined #openstack-infra | 16:34 | |
| *** jkilpatr has quit IRC | 16:36 | |
| *** jamesmcarthur has quit IRC | 16:36 | |
| fungi | Ng: go to http://logs.openstack.org/periodic/<jobname>/ and sort by date | 16:36 |
| *** yamamoto has joined #openstack-infra | 16:37 | |
| clarkb | also the health dashboard if you produce subunit results | 16:37 |
| *** jamesmcarthur has joined #openstack-infra | 16:37 | |
| *** esikachev has quit IRC | 16:38 | |
| Ng | fungi: clarkb: awesome, thanks! | 16:38 |
| fungi | Ng: so either http://logs.openstack.org/periodic/periodic-tripleo-ci-centos-7-scenario002-multinode-oooq/?C=M;O=A or http://status.openstack.org/openstack-health/#/job/periodic-tripleo-ci-centos-7-scenario002-multinode-oooq | 16:39 |
| * fungi is not sure how to get the health dashboard to show more history for that one | 16:40 | |
| fungi | maybe it only just started emitting subunit a couple days ago? | 16:40 |
| *** Hal has quit IRC | 16:40 | |
| fungi | or maybe it was failing too early to get that far until a couple days ago | 16:40 |
| Ng | I think I have seen it running earlier than that | 16:40 |
| Ng | but thanks again, this is collectively enough information for what I need :) | 16:41 |
| fungi | you can also get trending for job status via graphite.openstack.org | 16:41 |
| *** derekh has quit IRC | 16:42 | |
| *** armax has joined #openstack-infra | 16:43 | |
| *** yamamoto has quit IRC | 16:43 | |
| fungi | Ng: http://graphite.openstack.org/render/?width=586&height=308&_salt=1496940212.254&target=stats.zuul.pipeline.periodic.job.periodic-tripleo-ci-centos-7-scenario002-multinode-oooq.FAILURE&target=stats.zuul.pipeline.periodic.job.periodic-tripleo-ci-centos-7-scenario002-multinode-oooq.SUCCESS&from=00%3A00_20170508&until=23%3A59_20170608&lineMode=connected | 16:43 |
| *** clayton has quit IRC | 16:43 | |
| Ng | fungi: it's consistently 0.10 successful? ;D | 16:44 |
| fungi | hrm, no, connected line seems to be a poor choice for something that runs the same number of times a day | 16:44 |
| *** mhickey has quit IRC | 16:45 | |
| fungi | Ng: maybe this is more readable... http://graphite.openstack.org/render/?width=586&height=308&_salt=1496940340.63&from=00%3A00_20170508&until=23%3A59_20170608&target=stats.zuul.pipeline.periodic.job.periodic-tripleo-ci-centos-7-scenario002-multinode-oooq.FAILURE&target=stats.zuul.pipeline.periodic.job.periodic-tripleo-ci-centos-7-scenario002-multinode-oooq.SUCCESS&connectedLimit=&areaMode=stacked | 16:46 |
| *** ykarel has quit IRC | 16:46 | |
| Ng | fungi: nice! | 16:46 |
| *** clayton has joined #openstack-infra | 16:46 | |
| fungi | definitely some failures and successes interspersed | 16:46 |
| fungi | that's the actual job statuses reported by zuul (as opposed to openstack health, which will only know about a job if it got far enough to run tests and create a subunit stream) | 16:46 |
| fungi | i'm not really sure why its units are reported as 0.1, but whatevs. i'd just ignore the y axis for this purpose | 16:48 |
| openstackgerrit | Pavlo Shchelokovskyy proposed openstack-infra/project-config master: Add etcd3 to networking-generic-switch job https://review.openstack.org/472343 | 16:48 |
| *** mat128 has joined #openstack-infra | 16:50 | |
| *** EricGonczer_ has quit IRC | 16:53 | |
| *** vhosakot has joined #openstack-infra | 16:54 | |
| *** EricGonczer_ has joined #openstack-infra | 16:54 | |
| *** sambetts is now known as sambetts|afk | 16:58 | |
| *** rderose has joined #openstack-infra | 16:59 | |
| *** rkukura has quit IRC | 17:00 | |
| *** pcaruana has quit IRC | 17:00 | |
| *** mat128 has quit IRC | 17:03 | |
| *** ralonsoh has quit IRC | 17:03 | |
| *** rbrndt has quit IRC | 17:04 | |
| *** baoli has quit IRC | 17:04 | |
| *** baoli has joined #openstack-infra | 17:05 | |
| *** esikachev has joined #openstack-infra | 17:06 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Strip not split the untrusted_wrapper https://review.openstack.org/472066 | 17:07 |
| *** rderose has quit IRC | 17:07 | |
| *** mat128 has joined #openstack-infra | 17:09 | |
| openstackgerrit | Merged openstack-infra/shade master: De-client-ify User Ops https://review.openstack.org/467784 | 17:12 |
| *** rkukura has joined #openstack-infra | 17:13 | |
| *** ltomasbo is now known as ltomasbo|away | 17:14 | |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/zuul feature/zuulv3: Add coverage artifacts to .gitignore https://review.openstack.org/472353 | 17:16 |
| *** esikachev has quit IRC | 17:16 | |
| *** ericyoung has quit IRC | 17:16 | |
| dhellmann | fungi, jeblair : I wanted to make sure you saw that asettle had posted the docs migration spec: https://review.openstack.org/#/c/471881/ | 17:22 |
| *** trown|lunch is now known as trown | 17:22 | |
| dhellmann | oops, wrong link | 17:22 |
| fungi | dhellmann: yep! we were discussing just above (15:18 utc) | 17:22 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul feature/zuulv3: Correct location of ansible log file https://review.openstack.org/472355 | 17:22 |
| fungi | thanks | 17:23 |
| dhellmann | ah, good | 17:23 |
| *** jpena is now known as jpena|off | 17:23 | |
| *** baoli has quit IRC | 17:23 | |
| *** baoli has joined #openstack-infra | 17:24 | |
| *** baoli has quit IRC | 17:25 | |
| *** amotoki is now known as amotoki_away | 17:28 | |
| *** rkukura has quit IRC | 17:33 | |
| *** baoli has joined #openstack-infra | 17:34 | |
| *** amotoki_away is now known as amotoki | 17:35 | |
| *** dhinesh has joined #openstack-infra | 17:37 | |
| *** nicolasbock has quit IRC | 17:37 | |
| *** hieulq_ has quit IRC | 17:39 | |
| *** hieulq_ has joined #openstack-infra | 17:39 | |
| *** hieulq_ has quit IRC | 17:40 | |
| *** jamesmcarthur has quit IRC | 17:40 | |
| *** andreww has joined #openstack-infra | 17:42 | |
| *** jamesdenton has quit IRC | 17:45 | |
| *** elajkat_ has quit IRC | 17:46 | |
| *** xarses has quit IRC | 17:46 | |
| *** andreww has quit IRC | 17:47 | |
| *** xarses has joined #openstack-infra | 17:47 | |
| *** sshnaidm is now known as sshnaidm|off | 17:48 | |
| *** neil__ has quit IRC | 17:48 | |
| *** jascott1 has joined #openstack-infra | 17:49 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Pass zuul_connections to puppet-zuul https://review.openstack.org/472360 | 17:49 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Allow per-repo selection of configuration classes to load https://review.openstack.org/472009 | 17:51 |
| *** nicolasbock has joined #openstack-infra | 17:53 | |
| *** jbadiapa_ has joined #openstack-infra | 17:56 | |
| openstackgerrit | Rosario Di Somma proposed openstack-infra/shade master: Remove unneeded calls to shade_exceptions https://review.openstack.org/472363 | 17:57 |
| pabelanger | fungi: clarkb: do you mind a +3 on https://review.openstack.org/#/c/472360/ We didn't actually have ze01.o.o use the new connection data in hiera | 17:58 |
| *** jbadiapa has quit IRC | 17:59 | |
| clarkb | pabelanger: I'm not sure I understand what that change is doing | 18:00 |
| clarkb | is that what tells it to gearman to zuul? | 18:00 |
| pabelanger | clarkb: it tells zuul-executor (zuul.conf) our gerrit connection information | 18:00 |
| pabelanger | that is it hiera now | 18:01 |
| clarkb | not to bikeshed that but maybe connections and zuul_connections isn't the best way to express that? I'll approve the change | 18:01 |
| fungi | clarkb: http://git.openstack.org/cgit/openstack-infra/system-config/tree/hiera/group/zuul-executor.yaml | 18:01 |
| *** amoralej is now known as amoralej|off | 18:01 | |
| clarkb | oh its the full on zuul.conf entry | 18:01 |
| clarkb | oh I know my confusion, we need it in executor now for the merger functionality ya? | 18:02 |
| pabelanger | Ya | 18:02 |
| clarkb | I keep thinking zuul executor connects to zuul via gearman and worker nodes via ssh, so zuul connection must be gearman. Got it | 18:02 |
| pabelanger | I completely forgot about it when doing the initial setup | 18:03 |
| fungi | lots of moving parts | 18:03 |
| clarkb | I would've too clearly ^ | 18:03 |
| clarkb | :) | 18:03 |
| pabelanger | but finger jobs@ze01.openstack.org does work! I know jobs isn't valid, but not sure how to list users over finger | 18:03 |
| pabelanger | Build ID jobs is not valid is retunred | 18:04 |
| *** rbrndt has joined #openstack-infra | 18:04 | |
| pabelanger | returned* | 18:04 |
| pabelanger | I also still need to update puppet-zuul with systemctl daemon-reload logic, will post a patch shortly for that | 18:04 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Log a little extra debug info on log streaming https://review.openstack.org/472367 | 18:06 |
| *** jkilpatr_ has quit IRC | 18:08 | |
| *** rbrndt has quit IRC | 18:09 | |
| *** shardy has quit IRC | 18:10 | |
| *** jamesdenton has joined #openstack-infra | 18:10 | |
| openstackgerrit | Merged openstack-infra/system-config master: Pass zuul_connections to puppet-zuul https://review.openstack.org/472360 | 18:11 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Bail with an error on a non-existent jobroot_dir https://review.openstack.org/469524 | 18:15 |
| *** jkilpatr has joined #openstack-infra | 18:15 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Make log streaming to executor per-task https://review.openstack.org/471799 | 18:16 |
| *** Sukhdev has joined #openstack-infra | 18:16 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Strip double-timestamps from output log https://review.openstack.org/471890 | 18:18 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Update log to include expected data on task results https://review.openstack.org/471891 | 18:19 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Move log streaming to share with zuul_console https://review.openstack.org/471921 | 18:19 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Add friendly error messages and tests for nodeset dupes https://review.openstack.org/472370 | 18:24 |
| *** electrofelix has quit IRC | 18:25 | |
| *** salv-orl_ has quit IRC | 18:25 | |
| *** salv-orlando has joined #openstack-infra | 18:26 | |
| *** spzala has quit IRC | 18:29 | |
| *** spzala has joined #openstack-infra | 18:29 | |
| rm_work | anyone around for a project-config change? https://review.openstack.org/#/c/471887/ | 18:30 |
| rm_work | we really would like to unblock our gate | 18:30 |
| *** salv-orlando has quit IRC | 18:30 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Move log streaming to share with zuul_console https://review.openstack.org/471921 | 18:32 |
| clarkb | rm_work: reviewed | 18:32 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Change node 'image' to 'label' https://review.openstack.org/472372 | 18:33 |
| *** spzala has quit IRC | 18:33 | |
| cmurphy | can i jump on that bandwagon too https://review.openstack.org/#/c/471608/ :) | 18:33 |
| clarkb | sure | 18:34 |
| fungi | clarkb: rm_work: oh, i probably left the same comments | 18:34 |
| *** jamesdenton has quit IRC | 18:34 | |
| *** dizquierdo is now known as dizquierdo_afk | 18:34 | |
| *** jamesdenton has joined #openstack-infra | 18:35 | |
| openstackgerrit | Adam Harwell proposed openstack-infra/project-config master: Octavia: dashboard jobs need to be slightly normalized https://review.openstack.org/471887 | 18:35 |
| rm_work | clarkb / fungi ^^ done | 18:35 |
| cmurphy | thanks fungi and clarkb | 18:36 |
| fungi | sorry we seem to be more behind than usual on project-config reviews | 18:37 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Allow per-repo selection of configuration classes to load https://review.openstack.org/472009 | 18:38 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Change node 'image' to 'label' https://review.openstack.org/472372 | 18:38 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Add friendly error messages and tests for nodeset dupes https://review.openstack.org/472370 | 18:38 |
| cmurphy | really makes you appreciate ajaeger :) | 18:39 |
| openstackgerrit | Merged openstack-infra/project-config master: Use TEMPEST_REGEX in new keystone job https://review.openstack.org/471608 | 18:41 |
| *** hashar has joined #openstack-infra | 18:44 | |
| *** nicolasbock has quit IRC | 18:45 | |
| fungi | cmurphy: i already did! | 18:45 |
| jeblair | cmurphy: indeed, but my appreciation for ajaeger has never waned :) | 18:45 |
| fungi | we'll have to remind him that he doesn't have to take vacation to get appreciated ;) | 18:46 |
| * mordred appreciates AJaeger a lot | 18:47 | |
| *** nicolasbock has joined #openstack-infra | 18:48 | |
| * clarkb piles on, maybe we can get cmurphy to buy AJaeger a beer from the channel | 18:48 | |
| cmurphy | i can probably manage that | 18:50 |
| *** nicolasbock has quit IRC | 18:50 | |
| *** nicolasbock has joined #openstack-infra | 18:50 | |
| clarkb | cmurphy: I can trade you brisket | 18:52 |
| *** bh526r has quit IRC | 18:52 | |
| openstackgerrit | Merged openstack-infra/project-config master: Octavia: dashboard jobs need to be slightly normalized https://review.openstack.org/471887 | 18:52 |
| cmurphy | clarkb: deal | 18:52 |
| * fungi needs to figure out how to barter for brisket now | 18:53 | |
| *** spzala has joined #openstack-infra | 18:53 | |
| *** mrmartin has quit IRC | 18:54 | |
| *** mrmartin has joined #openstack-infra | 18:54 | |
| *** amotoki is now known as amotoki_away | 18:55 | |
| *** dave-mccowan has quit IRC | 18:58 | |
| *** eharney has quit IRC | 18:58 | |
| *** jamesmcarthur has joined #openstack-infra | 18:59 | |
| *** jamesmcarthur has quit IRC | 19:00 | |
| *** jamesmcarthur has joined #openstack-infra | 19:00 | |
| *** nicolasbock has quit IRC | 19:00 | |
| *** dave-mccowan has joined #openstack-infra | 19:01 | |
| fungi | clarkb: you were asking tuesday night about the specs cleanup work... i've got https://review.openstack.org/471948 up for review if you're still interested | 19:02 |
| *** gardlt_ has quit IRC | 19:03 | |
| clarkb | fungi: done | 19:04 |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck master: Add query for KeyError in nova functional tests bug 1696834 https://review.openstack.org/472379 | 19:05 |
| openstack | bug 1696834 in OpenStack Compute (nova) "Intermittent "KeyError: 'allocations'" in functional tests" [Medium,Confirmed] https://launchpad.net/bugs/1696834 | 19:05 |
| *** nicolasbock has joined #openstack-infra | 19:05 | |
| fungi | gratci | 19:05 |
| *** rbrndt has joined #openstack-infra | 19:07 | |
| * jeblair hopes cmurphy likes brisket | 19:09 | |
| jeblair | the way this is going, cmurphy may end up with 20 briskets, and ajaeger with 20 beers | 19:09 |
| mordred | funny, I was planning on smoking a brisket this weekend already | 19:11 |
| openstackgerrit | Merged openstack-infra/elastic-recheck master: Add query for KeyError in nova functional tests bug 1696834 https://review.openstack.org/472379 | 19:12 |
| openstack | bug 1696834 in OpenStack Compute (nova) "Intermittent "KeyError: 'allocations'" in functional tests" [Medium,Confirmed] https://launchpad.net/bugs/1696834 | 19:12 |
| *** e0ne has joined #openstack-infra | 19:14 | |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck master: Simplify query for bug 1696834 https://review.openstack.org/472383 | 19:15 |
| openstack | bug 1696834 in OpenStack Compute (nova) "Intermittent "KeyError: 'allocations'" in functional tests" [Medium,Confirmed] https://launchpad.net/bugs/1696834 | 19:15 |
| fungi | i'm planning to have the smoker going tomorrow, saturday and sunday but no brisket on the agenda this time | 19:16 |
| *** e0ne has quit IRC | 19:17 | |
| *** yamahata has joined #openstack-infra | 19:17 | |
| *** yland has joined #openstack-infra | 19:18 | |
| *** nicolasbock has quit IRC | 19:18 | |
| *** nicolasbock has joined #openstack-infra | 19:18 | |
| *** amoralej|off is now known as amoralej | 19:23 | |
| *** nicolasbock has quit IRC | 19:23 | |
| *** bobh has quit IRC | 19:24 | |
| *** jiaohaolin has joined #openstack-infra | 19:28 | |
| *** jiaohaolin1 has quit IRC | 19:29 | |
| *** yland has quit IRC | 19:29 | |
| *** yland has joined #openstack-infra | 19:29 | |
| mordred | fungi: that's a lot of days to run a smoker with no brisket | 19:29 |
| *** yland is now known as Guest12239 | 19:30 | |
| clarkb | must be a north carolina thing | 19:30 |
| fungi | yeah, i'm coming to the same conclusion | 19:30 |
| fungi | no, it's more that i'm smoking things to use up what's in our deep freezer and avoid venturing out into the sea of tourists | 19:31 |
| fungi | but clearly, there should have been brisket in the freezer | 19:31 |
| openstackgerrit | Merged openstack-infra/shade master: Remove unneeded calls to shade_exceptions https://review.openstack.org/472363 | 19:32 |
| *** sbezverk has quit IRC | 19:32 | |
| fungi | speaking of which, a few weekends ago i smoked salmon bellies. now that was some tasty, tasty fish | 19:32 |
| mordred | fungi: mmmmm | 19:33 |
| fungi | need to get my hands on more of those | 19:33 |
| mordred | fungi: we just recently got a chest-freezer to facilitate having more briskets and pork shoulders on hand | 19:33 |
| fungi | i'm going to start calling it "salmon brisket" | 19:33 |
| clarkb | general fyi to channle I've held a couple nodepool nodes to try and further debug this devstack runtime situation | 19:34 |
| fungi | yeah, we have a goodly sized chest freezer in the garage for just such reasons | 19:34 |
| clarkb | mordred: you don't call them butts? | 19:34 |
| *** florianf|afk is now known as florianf | 19:34 | |
| mordred | clarkb: I try not to call them butts | 19:34 |
| *** Guest12239 has quit IRC | 19:35 | |
| *** e0ne has joined #openstack-infra | 19:36 | |
| fungi | clarkb: around these parts we usually we use a "picnic" (shoulder cut with the hock/shank) | 19:36 |
| clarkb | EmilienM: bkero I want to follow up on https://review.openstack.org/#/c/447176/ where is a good place to look at logs for the container jobs now that that has merged? | 19:36 |
| fungi | clarkb: the "boston butt" usually refers to a cut above the shoulder | 19:36 |
| mordred | fungi, clarkb: also, I do not get my pork packed in barrels | 19:36 |
| *** panda|bbl is now known as panda | 19:36 | |
| fungi | clarkb: which devstack runtime situation? i feel like i missed some actual important details in the cooking show commercial break | 19:37 |
| clarkb | TIL that butt is the butt end of the shoulder. "pork butt" is how everything seems to be labeled out here | 19:37 |
| clarkb | fungi: it takes ~half an hour or more in various clouds | 19:37 |
| clarkb | fungi: this is after we got runtime down to like 8 minutes with wheel mirrors and all that a year ago or whenever it was | 19:38 |
| mordred | clarkb: that is not why it's called butt | 19:38 |
| fungi | clarkb: oh. that issue. yeah so you were focusing on openstackclient performance improvements, is that the bulk of it do you think? | 19:38 |
| clarkb | mordred: thats what the googles said | 19:38 |
| mordred | clarkb: http://mentalfloss.com/article/22382/how-9-cuts-meat-got-their-names | 19:38 |
| *** pradk has quit IRC | 19:38 | |
| clarkb | mordred: neat | 19:38 |
| clarkb | fungi: I think its a chunk of it, but I want to interactively poke at things in envs that are slow including openstackclient | 19:39 |
| clarkb | fungi: of course if osc is slow that could be the code is slow, the disk is slow, the network is slow, the apis are slow and so on | 19:39 |
| fungi | sure | 19:41 |
| clarkb | so wanted to poke directly at it | 19:41 |
| jeblair | pabelanger: what's the zuulv3 status? | 19:42 |
| *** claudiub has joined #openstack-infra | 19:44 | |
| *** yamahata has quit IRC | 19:44 | |
| clarkb | one thing I should do is compare tempest runtimes again as tempest is hitting all the same apis and such across clouds so should indiicate if its a problem in openstack slowing down deployment of if deployment is just slow | 19:45 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Run ansible-lint under pep8 envlist https://review.openstack.org/466895 | 19:45 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Gather facts for run-bindep role https://review.openstack.org/466918 | 19:46 |
| *** asselin__ has quit IRC | 19:46 | |
| fungi | clarkb: we have stable branch jobs too... i wonder have you looked to see if runtimes differ dramatically between those and master? | 19:46 |
| fungi | if so, could probably checkpoint particular phases of devstack setup and see if it's possible to spot where most of the increase happened | 19:47 |
| clarkb | also brisket is too expensive to stock out here :/ | 19:47 |
| *** jamesmcarthur has quit IRC | 19:47 | |
| clarkb | fungi: thats a good idea | 19:47 |
| *** amoralej is now known as amoralej|off | 19:47 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-roles master: Remove tox -epy27 envlist https://review.openstack.org/466897 | 19:48 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-roles master: Run ansible-lint under pep8 envlist https://review.openstack.org/466898 | 19:48 |
| *** rossella__ has quit IRC | 19:50 | |
| *** xarses has quit IRC | 19:50 | |
| *** xarses has joined #openstack-infra | 19:51 | |
| *** jamesmcarthur has joined #openstack-infra | 19:51 | |
| *** gardlt has joined #openstack-infra | 19:51 | |
| *** bobh has joined #openstack-infra | 19:51 | |
| *** jamesmcarthur has quit IRC | 19:51 | |
| *** JudeC has joined #openstack-infra | 19:52 | |
| *** bobh has quit IRC | 19:52 | |
| EmilienM | clarkb: close to 10 pm here, I'll let bkero help out maybe | 19:53 |
| mordred | clarkb: wow really? I get it here for $4/pound for high quality organic ... of course I'm getting it from restaurant supplier so not paying retail markup | 19:54 |
| clarkb | mordred: $6/lb for local ranged brisket | 19:55 |
| mordred | that is much more per pound | 19:55 |
| clarkb | so thats like $200 for two briskets (depending on size) | 19:55 |
| mordred | clarkb: is that restaurant supply or retail butcher? | 19:56 |
| clarkb | mordred: its a local butcher that does both, I pay what I assume is retial though | 19:56 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul feature/zuulv3: Correct location of ansible log file https://review.openstack.org/472355 | 19:56 |
| clarkb | mordred: The Meating Place | 19:56 |
| mordred | mmm. meating | 19:56 |
| *** jkilpatr has quit IRC | 19:56 | |
| *** rbrndt has quit IRC | 19:57 | |
| *** salv-orlando has joined #openstack-infra | 19:57 | |
| *** rbrndt has joined #openstack-infra | 19:57 | |
| clarkb | but the brisket is so good from there, which cuts down on cook time etc etc etc probably worth it in the end because I'm impatient | 19:57 |
| *** rbrndt has quit IRC | 19:57 | |
| mordred | clarkb: oh yah. samehere - the stuff I get is so good it basically does all the work for me just by being good meat | 19:58 |
| mordred | clarkb: amusingly enough, my supplier's name is "Clark" | 19:58 |
| clarkb | infra-root any idea why we have multiple jobs per day at http://logs.openstack.org/periodic-stable/periodic-tempest-dsvm-neutron-full-ubuntu-xenial-ocata/ ? | 19:58 |
| clarkb | we also seem to skip days | 19:59 |
| mordred | clarkb: nope. that doesn't seem great though | 19:59 |
| fungi | clarkb: maybe delayed running on days when we had a backlog since periodic gets lowest priority? | 20:00 |
| *** rbrndt has joined #openstack-infra | 20:01 | |
| *** gardlt_ has joined #openstack-infra | 20:02 | |
| mordred | and we don't de-dupe multiple jobs | 20:02 |
| mordred | fungi: that makes sense to me | 20:02 |
| jeblair | it's on my post-zuulv3 list | 20:03 |
| *** gardlt has quit IRC | 20:03 | |
| *** jamesmcarthur has joined #openstack-infra | 20:03 | |
| *** harlowja has quit IRC | 20:08 | |
| fungi | inc0: were you still wanting to have a discussion today to go into detail on your image building/testing/publishing jobs? | 20:09 |
| clarkb | fungi: small sample rough comparisons of periodic job runs in 1:1 cloud regions shows that ocata and master aren't much difference. There may be a slight difference between master/ocata and newton though | 20:10 |
| *** gardlt_ has quit IRC | 20:11 | |
| fungi | i expect mitaka is relatively broken at this point (i've seen the failure reports from zuul to the stable ml daily) | 20:11 |
| fungi | so probably hard to get any useful data there | 20:11 |
| clarkb | http://logs.openstack.org/periodic/periodic-tempest-dsvm-all-master/e33f28f/console.html#_2017-06-04_06_38_13_775863 vs http://logs.openstack.org/periodic-stable/periodic-tempest-dsvm-neutron-full-ubuntu-xenial-newton/737ed9d/console.html.gz#_2017-06-02_06_10_39_335069 | 20:11 |
| clarkb | thats 28 vs 23 minutes roughly | 20:11 |
| fungi | though maybe we still have a working mitaka run from long enough ago that it was actually working if it only broke in recent weeks | 20:12 |
| clarkb | we have added new services since then (etcd and nova placement api) | 20:12 |
| clarkb | our images don't have vi(m) on them anymore | 20:13 |
| clarkb | the tragedy | 20:13 |
| fungi | i noticed that recently too | 20:13 |
| fungi | and yet i survived ;) | 20:13 |
| *** jkilpatr has joined #openstack-infra | 20:14 | |
| *** jkilpatr has quit IRC | 20:14 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Correct location of ansible log file https://review.openstack.org/472355 | 20:16 |
| *** dangers_away has quit IRC | 20:16 | |
| *** dangers has joined #openstack-infra | 20:16 | |
| *** matt-borland has quit IRC | 20:17 | |
| *** matt-borland_ has joined #openstack-infra | 20:17 | |
| *** hashar has quit IRC | 20:19 | |
| *** sbezverk has joined #openstack-infra | 20:19 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/project-config master: Templatize Laravel jobs https://review.openstack.org/472402 | 20:22 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/project-config master: Add jobs for openstackid-resources https://review.openstack.org/472403 | 20:22 |
| *** borland__ has joined #openstack-infra | 20:27 | |
| *** Goneri has quit IRC | 20:28 | |
| *** matt-borland_ has quit IRC | 20:28 | |
| *** flepied has joined #openstack-infra | 20:29 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Allow per-repo selection of configuration classes to load https://review.openstack.org/472009 | 20:29 |
| *** e0ne has quit IRC | 20:30 | |
| *** jzerebec1i has joined #openstack-infra | 20:32 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Add friendly error messages and tests for nodeset dupes https://review.openstack.org/472370 | 20:32 |
| inc0 | fungi: yes, sorry, I was on meetings | 20:32 |
| inc0 | do you have time now? | 20:33 |
| fungi | sure | 20:33 |
| *** jzerebecki has quit IRC | 20:33 | |
| *** Apoorva_ has joined #openstack-infra | 20:33 | |
| inc0 | ok, so let me grab an etherpad | 20:33 |
| inc0 | I think I created one but forgot the link:( I'll do it again | 20:33 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Change node 'image' to 'label' https://review.openstack.org/472372 | 20:34 |
| fungi | i can find it for you quickly | 20:34 |
| inc0 | https://etherpad.openstack.org/p/kolla-publisher-jobs | 20:34 |
| inc0 | we can use this since it was empty anyway really | 20:34 |
| fungi | https://etherpad.openstack.org/p/kolla-publisher-job-design was the one you had before | 20:34 |
| *** jascott1 has quit IRC | 20:35 | |
| *** mrmartin has quit IRC | 20:35 | |
| inc0 | ok let's use https://etherpad.openstack.org/p/kolla-publisher-job-design | 20:35 |
| inc0 | thank you fungi | 20:35 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Log a little extra debug info on log streaming https://review.openstack.org/472367 | 20:35 |
| inc0 | so, I understand we can run gate for kolla to build+publish | 20:35 |
| *** jascott1 has joined #openstack-infra | 20:36 | |
| inc0 | to tarballs/kolla/images/staging | 20:36 |
| inc0 | (path doesn't matter) | 20:36 |
| inc0 | or tmp | 20:36 |
| inc0 | I think I used tmp in my change | 20:36 |
| inc0 | so we build and publish | 20:36 |
| *** Apoorva has quit IRC | 20:36 | |
| fungi | yeah, something along those lines should be fine as long as the same filename gets used each time | 20:36 |
| inc0 | now what I don't understand is how we want to schedule x jobs to sign them off | 20:36 |
| inc0 | let say we have 3 different deploy scenerios | 20:37 |
| fungi | you can define job dependencies in zuul's layout.yaml file | 20:37 |
| fungi | lemme find you a good example | 20:37 |
| inc0 | they all have to add +1 to file and after that we kick off publish job | 20:37 |
| *** hamzy has quit IRC | 20:38 | |
| *** mrmartin has joined #openstack-infra | 20:38 | |
| fungi | here's an excellent example: http://git.openstack.org/cgit/openstack-infra/project-config/tree/zuul/layout.yaml#n7221 | 20:39 |
| fungi | notice the nesting there, each of those jobs only runs if the job before it ran and passed | 20:39 |
| *** esikachev has joined #openstack-infra | 20:40 | |
| fungi | as i was saying earlier in the week, the down side (for now with zuul 2.x) is that this effectively serializes the jobs, so there's no parallelism of multiple validation jobs | 20:40 |
| inc0 | I assume most nested first right? | 20:40 |
| fungi | most nested runs last | 20:40 |
| *** pradk has joined #openstack-infra | 20:40 | |
| inc0 | ok | 20:40 |
| inc0 | but with zuul v3 we can specify x jobs in pararell? | 20:41 |
| *** Apoorva_ has quit IRC | 20:41 | |
| fungi | so in that example the pkgdeb-publish-auto-backports job runs first, if it passes then reprepro-import-debian-openstack-jessie-backports will run and if that passes reprepro-sign-jessie-backports and then if that passes reprepro-release-deb-openstack finally runs | 20:41 |
| *** Apoorva has joined #openstack-infra | 20:41 | |
| fungi | and yeah, in zuul v3 jobs can have multiple dependencies specified (the syntax changes to accommodate that) and it creates a dag | 20:42 |
| inc0 | pararellism is optimization, we can handle it later | 20:42 |
| fungi | that's what i figured as well | 20:42 |
| inc0 | time is not that much of a factor anyway | 20:42 |
| *** jamesdenton has quit IRC | 20:42 | |
| inc0 | another question, let say last job is one that will copy it to tarballs.o.o proper (not tmp) | 20:42 |
| fungi | right, if it takes a few hours to complete the entire process it's not that big a deal since it runs once a day | 20:42 |
| inc0 | is there any way this last job can get all previous jobs ids? | 20:43 |
| inc0 | so it can tag images with history of gate jobs it passed before sign off | 20:43 |
| inc0 | I guess alternatively we can have all jobs publish to same file and override one after another | 20:44 |
| fungi | what i suggested earlier in the week is to upload and retrieve an additional temporary file alongside the staging image, and have each job append its details to that file and reupload it on completion | 20:44 |
| inc0 | appending it's id to the thing | 20:44 |
| inc0 | we can use image metadata rather than file so it will carry on to dockerhub too | 20:44 |
| inc0 | but that's detail | 20:44 |
| *** salv-orl_ has joined #openstack-infra | 20:44 | |
| fungi | can include things like urls to logs of the previous jobs from that batch, for example | 20:44 |
| inc0 | what would be problem is that this approach effectively means pararell will never be possible | 20:45 |
| *** dizquierdo_afk is now known as dizquierdo | 20:45 | |
| fungi | to avoid reuploading the image over and over, you could have the final job that needs to download and reupload it anyway incorporate the metadata from the earlier jobs | 20:45 |
| inc0 | so maybe good feature for zuulv3 pararell execution, all job ids in history? | 20:45 |
| inc0 | that too | 20:45 |
| *** esikachev has quit IRC | 20:45 | |
| inc0 | in fact that would even make it easier with pararell | 20:45 |
| inc0 | as echo my-id >> all-ids is safe | 20:46 |
| fungi | and yeah, when zuul v3 is in production, we should look at implementing additional features in zuul to provide additional context about earlier job runs to later jobs in a build if that's something that seems useful | 20:46 |
| inc0 | (there might be some race conditions but really rare and with very low impact) | 20:46 |
| *** salv-orlando has quit IRC | 20:47 | |
| inc0 | so today it would be wget tarballs.o.o/kolla/images/tmp/job-ids; echo my-id >> job-ids; publish new job-ids | 20:47 |
| inc0 | and build job would create blank job id | 20:48 |
| *** esikachev has joined #openstack-infra | 20:48 | |
| fungi | yep, or would initialize that file with its id rather than appending | 20:48 |
| inc0 | well, right, either way it will clear up last one | 20:48 |
| fungi | agreed, that's the important bit | 20:49 |
| inc0 | and publish job can delete stuff from tarbals.o.o/tmp to avoid having wrong images with files | 20:49 |
| inc0 | again, optimization for later | 20:49 |
| sdague | oddly, all the fails on this are because the mirrors don't have the package yet - https://review.openstack.org/#/c/472341 - is that normal for packages to not be ready by the time upper-constraints jobs run? | 20:50 |
| fungi | publish job can't delete anything, but if the sequence never gets to the end then the staging path provides you with the broken tarball and a list of the jobs which ran against it, so you can even retrieve that for easier troubleshooting | 20:50 |
| inc0 | ok, I think i have all for now | 20:51 |
| inc0 | I'll try to implement it | 20:52 |
| fungi | sdague: would need to look at the timing for the release jobs and compare | 20:52 |
| fungi | sdague: also, it _can_ happen more easily if our pypi mirroring has broken so i'm checking that now | 20:52 |
| *** kgiusti has quit IRC | 20:52 | |
| fungi | inc0: you're welcome! let us know if you get stuck and need input | 20:53 |
| *** EricGonc_ has joined #openstack-infra | 20:53 | |
| inc0 | https://review.openstack.org/#/c/471428/ I'll modify this change accordingly | 20:53 |
| inc0 | so just keep an eye on it plz | 20:53 |
| *** EricGonczer_ has quit IRC | 20:54 | |
| *** esikachev has quit IRC | 20:54 | |
| *** wolverineav has quit IRC | 20:54 | |
| *** wolverineav has joined #openstack-infra | 20:54 | |
| fungi | sdague: bandersnatch found/retrieved that release and then vos release pushed it to the mirrors at 2017-06-08T16:46:56,496453368+0000 | 20:55 |
| fungi | according to the log on mirror-update.o.o | 20:55 |
| sdague | ah, the jobs ran at 16:45 | 20:56 |
| fungi | about 10 seconds too late for that job to find it apparently | 20:56 |
| *** gardlt has joined #openstack-infra | 20:56 | |
| inc0 | fungi: check https://etherpad.openstack.org/p/kolla-publisher-job-design I summarized this plan, check if you agree | 20:56 |
| sdague | fungi: yeh | 20:56 |
| *** florianf has quit IRC | 20:57 | |
| *** rbrndt has quit IRC | 20:58 | |
| fungi | inc0: that looks good. might be able to optimize bandwidth utilization further by making #3 and #4 the same job (retrieve staging image, publish to both dockerhub and known-working image location on tarballs.o.o) | 20:58 |
| *** Apoorva_ has joined #openstack-infra | 20:59 | |
| fungi | inc0: that way you don't pull down the image an extra time | 20:59 |
| inc0 | yeah, this is rev1 | 20:59 |
| inc0 | we'll work on it when we see real problems | 20:59 |
| *** trown is now known as trown|outtypewww | 20:59 | |
| *** gardlt has quit IRC | 21:00 | |
| *** EricGonczer_ has joined #openstack-infra | 21:00 | |
| *** florianf has joined #openstack-infra | 21:00 | |
| *** rbrndt has joined #openstack-infra | 21:00 | |
| fungi | sdague: given that bandersnatch runs every 5 minutes, i guess we _can_ race if a job that expects a package on our mirror runs within 5 minutes of the upload triggering it | 21:01 |
| fungi | i suppose we could make the pypi publish job busywait and poll to see it appear on a mirror before exiting | 21:02 |
| *** EricGonc_ has quit IRC | 21:02 | |
| *** Apoorva has quit IRC | 21:02 | |
| *** thorst_afk has quit IRC | 21:02 | |
| fungi | dhellmann: ^ i can't remember whether we discussed solutions to that in the past | 21:02 |
| fungi | i mean, it's not super critical, should succeed on a recheck | 21:03 |
| *** borland__ has quit IRC | 21:03 | |
| *** thorst_afk has joined #openstack-infra | 21:04 | |
| *** xyang1 has joined #openstack-infra | 21:05 | |
| *** gardlt has joined #openstack-infra | 21:07 | |
| *** thorst_afk has quit IRC | 21:09 | |
| *** gardlt has quit IRC | 21:12 | |
| *** Sukhdev has quit IRC | 21:14 | |
| *** Sukhdev has joined #openstack-infra | 21:17 | |
| *** Sukhdev has quit IRC | 21:21 | |
| *** abishop has quit IRC | 21:22 | |
| *** jcoufal has quit IRC | 21:22 | |
| pabelanger | jeblair: I believe we are ready for jobs to be run on zuulv3.o.o | 21:24 |
| pabelanger | jeblair: I believe we need to validate that mergers are working properly, I am not sure if we properly loaded tennant configuration on zuulv3.o.o | 21:25 |
| *** jamesmcarthur has quit IRC | 21:25 | |
| *** rfolco has quit IRC | 21:26 | |
| jeblair | pabelanger: seems to be there | 21:27 |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack-infra/project-config master: Make multinode gates regular https://review.openstack.org/471428 | 21:27 |
| *** esberglu has quit IRC | 21:27 | |
| *** thorst_afk has joined #openstack-infra | 21:27 | |
| *** esberglu has joined #openstack-infra | 21:27 | |
| *** vhosakot has quit IRC | 21:28 | |
| pabelanger | jeblair: okay, great. Working to fix up http://zuulv3.openstack.org/ since we don't have the right path to status.json currently | 21:28 |
| *** vhosakot has joined #openstack-infra | 21:28 | |
| *** thorst_afk has quit IRC | 21:31 | |
| *** esberglu has quit IRC | 21:32 | |
| *** ldnunes has quit IRC | 21:32 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: UTF8 encode commands sent to executor socket https://review.openstack.org/472419 | 21:32 |
| openstackgerrit | Paul Belanger proposed openstack-infra/puppet-zuul master: Clean up zuul-scheduler init script https://review.openstack.org/470414 | 21:38 |
| pabelanger | jeblair: clarkb: ^ updates for zuul-scheduler init script this will only affect zuulv3.o.o. Also added daemon-reload hacks | 21:39 |
| *** mat128 has quit IRC | 21:41 | |
| *** adam_g has quit IRC | 21:42 | |
| jeblair | pabelanger: /var/lib/zuul/ssh/id_rsa on ze01 is empty | 21:43 |
| *** adam_g has joined #openstack-infra | 21:43 | |
| pabelanger | jeblair: I see the issue | 21:44 |
| *** spzala has quit IRC | 21:44 | |
| pabelanger | typo in site.pp | 21:44 |
| *** Goneri has joined #openstack-infra | 21:44 | |
| pabelanger | Oh, maybe not | 21:44 |
| pabelanger | checking hiera now | 21:44 |
| mordred | pabelanger: did the executor group thing land? | 21:44 |
| pabelanger | mordred: yes I believe so | 21:45 |
| *** yland has joined #openstack-infra | 21:45 | |
| pabelanger | our hieradata is correct | 21:45 |
| mordred | kk | 21:45 |
| pabelanger | puppet just didn't see it populate the file | 21:45 |
| *** yland is now known as Guest90667 | 21:45 | |
| *** mriedem is now known as mriedem_afk | 21:48 | |
| *** mriedem_afk has quit IRC | 21:48 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: UTF8 encode commands sent to executor socket https://review.openstack.org/472419 | 21:50 |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Fix typo with gerrit ssh private key for ze01.o.o https://review.openstack.org/472452 | 21:50 |
| *** askb has joined #openstack-infra | 21:50 | |
| pabelanger | jeblair: mordred: ^it was a typo, and we defaulted to an empty string in puppet-zuul | 21:51 |
| pabelanger | hieradata is correct | 21:51 |
| pabelanger | it is setup on 1097 | 21:51 |
| openstackgerrit | Merged openstack-infra/elastic-recheck master: Simplify query for bug 1696834 https://review.openstack.org/472383 | 21:51 |
| openstack | bug 1696834 in OpenStack Compute (nova) "Intermittent "KeyError: 'allocations'" in functional tests" [Medium,Confirmed] https://launchpad.net/bugs/1696834 | 21:51 |
| *** Guest90667 has quit IRC | 21:53 | |
| jeblair | pabelanger: +3 | 21:53 |
| *** chlong has quit IRC | 21:53 | |
| openstackgerrit | Rosario Di Somma proposed openstack-infra/shade master: Don't remove top-container element for network REST API calls https://review.openstack.org/472453 | 21:55 |
| *** xyang1 has quit IRC | 21:57 | |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack-infra/project-config master: Make multinode gates regular https://review.openstack.org/471428 | 21:59 |
| *** xyang1 has joined #openstack-infra | 21:59 | |
| *** baoli has quit IRC | 22:02 | |
| *** reed has quit IRC | 22:02 | |
| *** baoli has joined #openstack-infra | 22:02 | |
| *** reed has joined #openstack-infra | 22:03 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Add ssh known_hosts to ze01.o.o https://review.openstack.org/472456 | 22:03 |
| *** gardlt has joined #openstack-infra | 22:05 | |
| pabelanger | jeblair: that should fix our missing known_hosts^. The contents is from our zuul_merger.pp file today | 22:05 |
| *** dave-mccowan has quit IRC | 22:05 | |
| jeblair | pabelanger: isn't that in hiera? | 22:06 |
| *** jamesmcarthur has joined #openstack-infra | 22:06 | |
| pabelanger | jeblair: just the public key pair is. We construct the rest today: http://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/manifests/zuul_merger.pp#n21 | 22:06 |
| pabelanger | we could move the whole thing into hiera if you'd like | 22:07 |
| *** baoli has quit IRC | 22:07 | |
| jeblair | pabelanger: nah, when we make openstack_project::zuul_merger, it'll be fine to put it in there | 22:08 |
| jeblair | pabelanger: hiera('gerrit_zuul_user_ssh_key_contents') doesn't look right though | 22:08 |
| jeblair | pabelanger: should be gerrit_ssh_host_key => hiera('gerrit_ssh_rsa_pubkey_contents') | 22:08 |
| pabelanger | ack | 22:09 |
| pabelanger | fixing | 22:09 |
| *** gardlt has quit IRC | 22:09 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Add ssh known_hosts to ze01.o.o https://review.openstack.org/472456 | 22:10 |
| *** jamesmcarthur has quit IRC | 22:11 | |
| *** mdrabe has quit IRC | 22:11 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Add ssh known_hosts to ze01.o.o https://review.openstack.org/472456 | 22:11 |
| *** flepied has left #openstack-infra | 22:12 | |
| *** jbadiapa_ has quit IRC | 22:14 | |
| *** tosky has quit IRC | 22:18 | |
| *** salv-orl_ has quit IRC | 22:19 | |
| *** salv-orlando has joined #openstack-infra | 22:20 | |
| *** Apoorva_ has quit IRC | 22:21 | |
| *** Goneri has quit IRC | 22:21 | |
| *** Apoorva has joined #openstack-infra | 22:21 | |
| clarkb | ok https://etherpad.openstack.org/p/debugging-devstack-slowness has info in it | 22:23 |
| clarkb | jeblair: mordred ^ you may be particularly interested as AFS seems to be at least part of the problem | 22:23 |
| *** salv-orlando has quit IRC | 22:24 | |
| clarkb | jeblair: mordred: what I'd like to be able to measure is our cache miss rate, do you know if afs provides that information? | 22:26 |
| jeblair | clarkb: i agree that cold caches are bad. how did you confirm they are causing pip to be slow? | 22:28 |
| clarkb | also wondering if we bump the modtimes on bandersnatch updates such that afs would treat the entire mirror as possibly not cached? | 22:28 |
| clarkb | jeblair: well thats what I'd like to actually confirm but I don't see a way to have afs provide miss rates? | 22:28 |
| *** esberglu has joined #openstack-infra | 22:29 | |
| jeblair | clarkb: sorry, i think i misunderstood your etherpad | 22:29 |
| jeblair | Based on this it appears that cold afs caches are really painful, particularly across the Atlantic. It also seems to show that pip is a victim of this | 22:29 |
| jeblair | far more than distro package mirrors. | 22:29 |
| clarkb | jeblair: its mostly just inference now based on difference in runtimes and the cost of cold vs warm hits | 22:29 |
| jeblair | ^ sounds like that's the *hypothesis* not the *conclusion*. | 22:29 |
| clarkb | right I'm inferring that based on the first table | 22:29 |
| clarkb | and yes hypothesis upfront | 22:29 |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack-infra/project-config master: Make multinode gates regular https://review.openstack.org/471428 | 22:30 |
| clarkb | basically pip install takes almost twice as long in europe as in north america. The warm data fetch rates within those clouds are all really good (and best in citycloud even) | 22:30 |
| *** jbadiapa_ has joined #openstack-infra | 22:30 | |
| clarkb | so I'm making a leap that to be slower (particularly in citycloud) it is hitting cold cache enough to significantly tank the pip install time | 22:31 |
| clarkb | I also separately ran fio random writes test for 512MB in 8 passes on ovh and citycloud and disk write speed is ~200-350MBps | 22:31 |
| *** dizquierdo has quit IRC | 22:31 | |
| clarkb | so I don't think its a local disk bottneck at install. Though there will be some of that overhead too | 22:32 |
| *** florianf has quit IRC | 22:33 | |
| *** esberglu has quit IRC | 22:33 | |
| *** jascott1 has quit IRC | 22:34 | |
| *** jascott1 has joined #openstack-infra | 22:35 | |
| mordred | clarkb: the cache is all in /var/cache/openafs | 22:36 |
| *** jascott1 has quit IRC | 22:37 | |
| mordred | uhm | 22:37 |
| *** jascott1 has joined #openstack-infra | 22:37 | |
| mordred | /var/cache/openafs/D760 on mirror.bhs1.ovh.openstack.org seems to be the first one with entries for today | 22:38 |
| *** wolverineav has quit IRC | 22:39 | |
| *** xyang1 has quit IRC | 22:39 | |
| mordred | clarkb: but it's lots of small files - so I'd certainly _hope_ we're not invalidating the entire pypi just because the top level index changed | 22:39 |
| *** xarses has quit IRC | 22:40 | |
| clarkb | mordred: well if we update the modtime on the dirs/files (not sure if bandersnatch does that but if it did) would that be enough for afs to treat the cache as invalid? | 22:40 |
| clarkb | I'm not sure how it knows what is valid or not | 22:40 |
| mordred | clarkb: hrm. that might well - I'm also not sure | 22:40 |
| *** jascott1 has quit IRC | 22:41 | |
| clarkb | usually you have to explicitly turn that off on ext4 for example as it will update things and make stuff sad (thought access time is worse than mod time there) | 22:41 |
| jeblair | it's a ro volume, so the volume is invalidated as a whole; i don't *think* that means all the individual files are also invalidated (just that the cache manager has to ask the fileserver about them individually again, but shouldn't have to fetch the data). i'm not sure about that. | 22:41 |
| mordred | clarkb: oh - there's cache files there from a while back | 22:41 |
| mordred | -rw------- 1 root root 1048576 Apr 27 19:40 V1562256 | 22:41 |
| mordred | for intance | 22:41 |
| mordred | including some going back to Dec 7 | 22:42 |
| jeblair | it's possible that the cost of asking whether each file is up to date is expensive enough with transatlantic round trips that we notice | 22:42 |
| jeblair | i'd suggest a different term to cover that though; it's not a cold cache, it's just a cold index. | 22:43 |
| clarkb | jeblair: though I get ~40MBps hitting that file a second time | 22:43 |
| mordred | jeblair: oh - because since these are ro volumes, the client isn't getting active invalidation callbacks is it? | 22:43 |
| clarkb | mordred: and in citycloud its NAT'd (not NAT'd in ovh though) | 22:43 |
| jeblair | mordred: it does -- for the whole volume (it's a single invalidation call) | 22:43 |
| mordred | nod | 22:45 |
| mordred | http://docs.openafs.org/AdminGuide/HDRWQ402.html <-- entry on cache management, fwiw | 22:45 |
| clarkb | my next step was to run pip install of a fixed set of deps (probably global requirements) a few times per region to see if that provides useful info | 22:45 |
| jeblair | clarkb: i think correlating that with vos release times would be useful | 22:45 |
| clarkb | but also hopeful for more afs side information gathering if that is possible | 22:46 |
| mordred | clarkb: and mentions a tool called "xstat_cm_test" which is a tool to analyze cache misses | 22:46 |
| clarkb | jeblair: oh good point like if it spikes every 5 minutes | 22:46 |
| mordred | or, rather: | 22:46 |
| mordred | xstat_cm_test - Displays data collections from the Cache Manager | 22:46 |
| mordred | anyway - I recommend browsing that link above - there's some good reading on that page | 22:46 |
| *** EricGonczer_ has quit IRC | 22:46 | |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack-infra/project-config master: Make multinode gates regular https://review.openstack.org/471428 | 22:47 |
| jeblair | afsmonitor is supposed to supply cache hit/miss info, but it's not working for me for some reason | 22:47 |
| openstackgerrit | Eric Kao proposed openstack-infra/project-config master: add release job to congress-dashboard https://review.openstack.org/472467 | 22:47 |
| *** esikachev has joined #openstack-infra | 22:48 | |
| mordred | http://paste.openstack.org/show/612062/ <-- clarkb, jeblair there is the xstat_cm_test output for ovh bhs1 | 22:48 |
| mordred | both our vcache and dcache miss ratio are below recommended values | 22:51 |
| mordred | (which is good) | 22:51 |
| openstackgerrit | Rosario Di Somma proposed openstack-infra/shade master: Don't remove top-container element for network REST API calls https://review.openstack.org/472453 | 22:51 |
| jeblair | mordred, clarkb: yeah, that's a 99.97% cache hit rate | 22:51 |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Implement a cache of github change objects https://review.openstack.org/472468 | 22:51 |
| jeblair | for dcache | 22:52 |
| *** jascott1 has joined #openstack-infra | 22:52 | |
| *** esikachev has quit IRC | 22:52 | |
| clarkb | ya kna1 looks even better | 22:52 |
| mordred | maybe it's just that the performance cost is super bad on first fetch, but we're getting a TON of hits overall? | 22:52 |
| jeblair | clarkb: what are those devstack self-reporting numbers? random individual runs, or averages? | 22:53 |
| clarkb | jeblair: periodic job runs against master except for the outlier (that was a random individual run) | 22:54 |
| *** gordc has quit IRC | 22:54 | |
| clarkb | jeblair: we should be able to query logstash for numbers in aggregate though | 22:54 |
| clarkb | let me try sorting that out | 22:54 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack-infra/shade master: Updated from global requirements https://review.openstack.org/472310 | 22:54 |
| *** rbrndt has quit IRC | 22:55 | |
| dhinesh | is a third party CI setup necessary to evaluate and merge code to openstack-cinder? | 22:57 |
| *** panda is now known as panda|off | 22:57 | |
| fungi | dhinesh: no idea, you'd need to ask the cinder team what requirements they have | 22:57 |
| dhinesh | fungi: ok thanks | 22:58 |
| clarkb | jeblair: 454.45s is the average total pip_install time reported by devstack in ovh-gra1 for the gate-tempest-dsvm-neutron-full-ubuntu-xenial job | 22:59 |
| clarkb | jeblair: I dropped a couple data points ebcause they looked like they were indexed funny | 22:59 |
| clarkb | jeblair: oh sorry thats average for last 24 hours. 11 runs | 23:00 |
| jeblair | clarkb: what's the deviation on that? | 23:00 |
| clarkb | 353 is min, max is 551 | 23:00 |
| * clarkb sorts out st dev | 23:00 | |
| jeblair | that's enough variation at least to think that the invalidation cycle may be at play. | 23:01 |
| pabelanger | jeblair: how did you want to handle the URL change for status.json? On zuulv3-dev.o.o we just updated the apache2 rewriterule, are we okay to do the same in puppet-zuul? I am not sure what it would look like with more then 1 tenant and our current webui | 23:02 |
| clarkb | ~67.6 | 23:02 |
| clarkb | whihc ya is large | 23:02 |
| jeblair | pabelanger: yeah, the current ui won't handle more than one tenant. we need to fix that eventually. but for now, adding it as an argument to puppet-zuul sounds good. | 23:03 |
| pabelanger | great | 23:03 |
| jeblair | clarkb: i think running installs and correlating with vos release times will be helpful | 23:03 |
| clarkb | jeblair: ok I will work on that next, do you think we need a large install like installing all of global requirements? | 23:03 |
| *** felipemonteiro__ has quit IRC | 23:05 | |
| clarkb | I will fiddle with it to try and find something that looks right | 23:07 |
| *** jascott1_ has joined #openstack-infra | 23:08 | |
| *** sdague has quit IRC | 23:08 | |
| jeblair | clarkb: not necessarily that huge; i'd aim for something that can be installed within about a minute best-case, that way it's easy to be deliberate about completing the install within a vos release cycle, but still big enough the effect should be easily noticable. | 23:08 |
| *** jascott1 has quit IRC | 23:10 | |
| *** armax_ has joined #openstack-infra | 23:12 | |
| clarkb | for X in `seq 1 30`; do echo $X ; rm -rf venv ; virtualenv venv ; time venv/bin/pip install ./neutron -c requirements/upper-constraints.txt --no-cache-dir -q --log piplogging.log ; done is what I am starting with | 23:13 |
| openstackgerrit | Paul Belanger proposed openstack-infra/puppet-zuul master: Expose zuul_status_url for zuulv3 https://review.openstack.org/472474 | 23:13 |
| clarkb | runs each iteration in about 40 seconds | 23:13 |
| *** armax has quit IRC | 23:14 | |
| *** armax_ is now known as armax | 23:14 | |
| openstackgerrit | Merged openstack-infra/system-config master: Fix typo with gerrit ssh private key for ze01.o.o https://review.openstack.org/472452 | 23:15 |
| *** claudiub has quit IRC | 23:15 | |
| openstackgerrit | Merged openstack-infra/system-config master: Add ssh known_hosts to ze01.o.o https://review.openstack.org/472456 | 23:16 |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack-infra/project-config master: Make multinode gates regular https://review.openstack.org/471428 | 23:16 |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Use openstack status.json file for ze01.o.o https://review.openstack.org/472475 | 23:17 |
| clarkb | jeblair: and one just took 57 seconds | 23:18 |
| clarkb | jeblair: the user and system cpu time took the same amount of time roughly though so just wall increased (blocking on reads?) | 23:18 |
| *** knangia_ has joined #openstack-infra | 23:18 | |
| clarkb | the download rates that pip prints appear roughly the same though | 23:19 |
| clarkb | hrm kna1 is 2.5 ish minutes on first pass through | 23:19 |
| *** rkukura has joined #openstack-infra | 23:20 | |
| clarkb | and then 54 on second | 23:20 |
| clarkb | so thats possibly our cache warming up. Now to see if it goes back to 2.5 ish at some point | 23:20 |
| clarkb | jeblair: right around 2321-2322 UTC they all took longer than their normal | 23:23 |
| clarkb | jeblair: I am thinking I should restart with timestamp printing | 23:23 |
| *** Apoorva has quit IRC | 23:25 | |
| *** dhinesh has quit IRC | 23:29 | |
| *** dhinesh has joined #openstack-infra | 23:30 | |
| *** armax_ has joined #openstack-infra | 23:33 | |
| *** armax has quit IRC | 23:33 | |
| *** armax_ is now known as armax | 23:33 | |
| *** nunchuck has quit IRC | 23:34 | |
| *** markvoelker has quit IRC | 23:34 | |
| clarkb | jeblair: http://paste.openstack.org/show/612065/ seems to be a good illustration of what is going on | 23:35 |
| clarkb | jeblair: I think that does correlate it to our every 5 minutes we vos release | 23:35 |
| clarkb | that ran on ovh gra1 I think | 23:36 |
| *** sbezverk has quit IRC | 23:38 | |
| jeblair | when did the vos releases happen? | 23:38 |
| clarkb | jeblair: they happen every 5 minutes via cron, about to check that we ran one during that 2 minute interval. But trying to capture this in ehterpad first | 23:39 |
| jeblair | clarkb: i think we would want to avoid one happening *during* a run. so i think you'll want 2 complete runs before a vos release, then 2 complete runs after. | 23:39 |
| jeblair | (it may also be instructive to have one during a run as well, but as additional data) | 23:40 |
| *** Apoorva has joined #openstack-infra | 23:40 | |
| clarkb | jeblair: http://paste.openstack.org/show/612068/ is the logs for that vos release | 23:41 |
| clarkb | that one happened during the run | 23:41 |
| clarkb | the two previous happened before the run | 23:41 |
| clarkb | so let me make this a bit more manual to do what you descirbe | 23:41 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Add 'description' field to jobs https://review.openstack.org/472483 | 23:43 |
| *** EricGonczer_ has joined #openstack-infra | 23:45 | |
| *** pahuang has joined #openstack-infra | 23:48 | |
| jeblair | huh, we should listen to zuul-jobs in here | 23:49 |
| *** JudeC has quit IRC | 23:49 | |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config master: Subscribe -infra to zuul-jobs and zuul-base-jobs https://review.openstack.org/472487 | 23:50 |
| clarkb | jeblair: http://paste.openstack.org/show/612069/ thats what you asked for and seems to hold up that theory | 23:50 |
| clarkb | jeblair: I interleaved mirror-update logs with output from my test machine in kna1 | 23:51 |
| clarkb | pip's log file doesn't appear to do timestamps | 23:53 |
| jeblair | clarkb: a way to test whether it's downloading files or not might be to touch all of the files you're downloading before the vos release and see if the next install is even slower. | 23:54 |
| *** vhosakot has quit IRC | 23:55 | |
| clarkb | jeblair: you mean touch them in the afs volume? | 23:55 |
| jeblair | clarkb: yeah | 23:55 |
| jeblair | clarkb: if it isn't slower, then we're downloading all the files again, if it is slower, then we're at least not downloading everything after release. | 23:55 |
| clarkb | jeblair: what I've noticed just watching the pip output (that I pruned in that paste) is that it will say its downloading something then pause for a long time before it actually shows the progress bar download it | 23:56 |
| clarkb | jeblair: I'm beginning to think it is the metadata more so than the file contents that we are blocking on | 23:56 |
| clarkb | like apache can't serve the content until afs makes sure the mod time and all that are up to date because new vos release happened (but tahts theory more than evidence) | 23:56 |
| clarkb | I saw similar with wget | 23:57 |
| jeblair | yeah. it's not inconsistent with the idea that after each vos release, we have to ask the fileserver what the latest version is. | 23:57 |
| clarkb | basically it sends the request, then you wait before actually downloading anything | 23:57 |
| clarkb | the actual downlaods according to pip are at like 20MBps or so | 23:57 |
| *** harlowja has joined #openstack-infra | 23:57 | |
| *** jamielennox is now known as jamielennox|away | 23:57 | |
| jeblair | clarkb: ah, that points very strongly to that theory then. | 23:57 |
| jeblair | clarkb: that after a release, it has to verify it has the correct version in the cache, then it can serve that version. | 23:58 |
| jeblair | clarkb: a vos release with no changes from bandersnatch would be a good test too -- you should see the same behavior. | 23:58 |
| clarkb | good point. I am running out of time to do that now. But tomorrow I can stop the cron, then do a noop release and check that behavior | 23:59 |
| clarkb | if that is the case do we want to scale back to releasing like once every few hours or so? | 23:59 |
| clarkb | I think that should be sufficient to make most jobs much happier (though likely won't actually know until its tested) | 23:59 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!