| *** lindycoder has joined #openstack-infra | 00:01 | |
| *** tqtran has quit IRC | 00:01 | |
| pabelanger | rcarrillocruz: clarkb: okay, so I restarted nova services on controller00.chocolate.ic.openstack.org and things are coming back online again | 00:02 |
|---|---|---|
| pabelanger | I have no idea why that is | 00:02 |
| clarkb | background noise? | 00:03 |
| clarkb | I don't have any better ideas | 00:03 |
| *** devkulkarni1 has quit IRC | 00:03 | |
| pabelanger | same | 00:04 |
| pabelanger | k, enough yak shaving of infracloud | 00:04 |
| * clarkb is doing more paperwork | 00:04 | |
| clarkb | its that time of year again | 00:04 |
| ianw | that reminds me, i have to do my ethics training | 00:05 |
| pabelanger | same | 00:05 |
| pabelanger | been pushing it off | 00:05 |
| *** annegentle has quit IRC | 00:05 | |
| *** harlowja has quit IRC | 00:06 | |
| ianw | i think the reminder emails follow an exponential curve as the end-date approaches :) | 00:06 |
| *** Rahul has joined #openstack-infra | 00:06 | |
| *** devkulkarni has joined #openstack-infra | 00:06 | |
| *** Rahul is now known as Guest25334 | 00:07 | |
| *** maeker has quit IRC | 00:07 | |
| *** Guest25334 has quit IRC | 00:09 | |
| *** trown|outtypewww has quit IRC | 00:09 | |
| *** maeker has joined #openstack-infra | 00:09 | |
| *** Rahul has joined #openstack-infra | 00:10 | |
| *** Rahul is now known as Guest55618 | 00:11 | |
| *** devkulkarni has quit IRC | 00:11 | |
| *** maeker has quit IRC | 00:11 | |
| *** maeker has joined #openstack-infra | 00:12 | |
| *** esberglu has joined #openstack-infra | 00:12 | |
| *** sflanigan has joined #openstack-infra | 00:13 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Install dracut-generic-config package https://review.openstack.org/411540 | 00:16 |
| openstackgerrit | Mikhail S Medvedev proposed openstack/diskimage-builder: Fix bootloader element on ppc https://review.openstack.org/411541 | 00:16 |
| *** esberglu has quit IRC | 00:16 | |
| ianw | pabelanger / greghaynes : ^^ not tested, but i think it's right. building image now | 00:17 |
| *** rbuzatu has joined #openstack-infra | 00:17 | |
| pabelanger | #status log nova services restart on controller00.chocolate.ic.openstack.org to fix nodes failing to launch, unsure why this fixed our issue | 00:17 |
| openstackstatus | pabelanger: finished logging | 00:17 |
| *** Guest55618 has quit IRC | 00:18 | |
| *** scarab_ has joined #openstack-infra | 00:18 | |
| *** trown has joined #openstack-infra | 00:19 | |
| openstackgerrit | Joshua Harlow proposed openstack-infra/project-config: Initiate end of life of openstack/anvil https://review.openstack.org/411542 | 00:19 |
| *** marst has quit IRC | 00:20 | |
| *** rbuzatu has quit IRC | 00:22 | |
| ianw | harlowja: ^ i thought yahoo used the output of that? | 00:22 |
| *** mtanino has quit IRC | 00:23 | |
| *** scarab__ has joined #openstack-infra | 00:23 | |
| *** scarab_ has quit IRC | 00:23 | |
| *** wolverineav has joined #openstack-infra | 00:24 | |
| openstackgerrit | Jamie Lennox proposed openstack-infra/zuul: Use integration_id with github https://review.openstack.org/411544 | 00:25 |
| *** gomarivera has joined #openstack-infra | 00:25 | |
| *** scarab__ has quit IRC | 00:27 | |
| *** sdague has quit IRC | 00:28 | |
| *** Jeffrey4l_ is now known as Jeffrey4l | 00:28 | |
| *** wolverineav has quit IRC | 00:28 | |
| *** gouthamr has joined #openstack-infra | 00:29 | |
| *** scarab_ has joined #openstack-infra | 00:31 | |
| *** rwsu has quit IRC | 00:32 | |
| *** harlowja has joined #openstack-infra | 00:34 | |
| *** scarab_ has quit IRC | 00:35 | |
| *** scarab_ has joined #openstack-infra | 00:35 | |
| *** scarab_ has quit IRC | 00:37 | |
| *** scarab_ has joined #openstack-infra | 00:38 | |
| *** Julien-zte has joined #openstack-infra | 00:38 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Remove swapfile from undercloud https://review.openstack.org/410323 | 00:38 |
| *** Julien-zte has quit IRC | 00:38 | |
| *** sflanigan is now known as sflanigan_afk | 00:38 | |
| *** Julien-zte has joined #openstack-infra | 00:39 | |
| openstackgerrit | Joshua Harlow proposed openstack-infra/project-config: Initiate end of life of openstack/anvil https://review.openstack.org/411542 | 00:40 |
| *** scarab_ has quit IRC | 00:42 | |
| *** scarab_ has joined #openstack-infra | 00:42 | |
| *** wolverineav has joined #openstack-infra | 00:43 | |
| *** rwsu has joined #openstack-infra | 00:44 | |
| *** maeker has quit IRC | 00:46 | |
| *** scarab_ has quit IRC | 00:47 | |
| *** wolverineav has quit IRC | 00:48 | |
| *** scarab_ has joined #openstack-infra | 00:48 | |
| *** Julien-zte has quit IRC | 00:49 | |
| *** scarab_ has quit IRC | 00:53 | |
| *** wolverineav has joined #openstack-infra | 00:54 | |
| *** sflanigan_afk is now known as sflanigan | 00:55 | |
| *** jamielennox is now known as jamielennox|away | 00:57 | |
| *** wolverineav has quit IRC | 00:58 | |
| *** sdake_ has quit IRC | 01:00 | |
| *** jamielennox|away is now known as jamielennox | 01:01 | |
| *** tuanluong has joined #openstack-infra | 01:05 | |
| *** esberglu has joined #openstack-infra | 01:06 | |
| *** asettle has joined #openstack-infra | 01:08 | |
| *** esberglu has quit IRC | 01:11 | |
| *** claudiub has joined #openstack-infra | 01:12 | |
| *** scarab_ has joined #openstack-infra | 01:12 | |
| *** asettle has quit IRC | 01:13 | |
| *** markvoelker has joined #openstack-infra | 01:14 | |
| *** sdake has joined #openstack-infra | 01:14 | |
| *** scarab__ has joined #openstack-infra | 01:16 | |
| *** scarab_ has quit IRC | 01:17 | |
| *** mriedem has joined #openstack-infra | 01:17 | |
| *** esberglu has joined #openstack-infra | 01:17 | |
| *** tphummel has quit IRC | 01:17 | |
| *** markvoelker has quit IRC | 01:19 | |
| *** scarab__ has quit IRC | 01:21 | |
| *** scarab_ has joined #openstack-infra | 01:22 | |
| *** wolverineav has joined #openstack-infra | 01:23 | |
| *** harlowja has quit IRC | 01:25 | |
| *** tuanluong has quit IRC | 01:26 | |
| *** scarab_ has quit IRC | 01:27 | |
| *** Julien-zte has joined #openstack-infra | 01:27 | |
| *** Julien-zte has quit IRC | 01:27 | |
| *** wolverineav has quit IRC | 01:27 | |
| *** nadya has joined #openstack-infra | 01:29 | |
| *** lindycoder has quit IRC | 01:30 | |
| *** scarab_ has joined #openstack-infra | 01:32 | |
| *** nadya has quit IRC | 01:34 | |
| *** scarab_ has quit IRC | 01:35 | |
| *** scarab_ has joined #openstack-infra | 01:35 | |
| *** sdake has quit IRC | 01:35 | |
| *** esikachev has joined #openstack-infra | 01:37 | |
| *** sdake has joined #openstack-infra | 01:38 | |
| *** scarab_ has quit IRC | 01:39 | |
| *** scarab_ has joined #openstack-infra | 01:40 | |
| *** jamielennox is now known as jamielennox|away | 01:41 | |
| *** esikachev has quit IRC | 01:41 | |
| *** wolverineav has joined #openstack-infra | 01:42 | |
| *** scarab_ has quit IRC | 01:44 | |
| *** claudiub|2 has joined #openstack-infra | 01:44 | |
| *** yamamoto has joined #openstack-infra | 01:45 | |
| *** yanyanhu_ has joined #openstack-infra | 01:46 | |
| *** jamielennox|away is now known as jamielennox | 01:47 | |
| *** wolverineav has quit IRC | 01:47 | |
| *** claudiub has quit IRC | 01:47 | |
| *** scarab_ has joined #openstack-infra | 01:48 | |
| *** scarab_ has quit IRC | 01:53 | |
| *** yamamoto has quit IRC | 01:53 | |
| *** bradm has quit IRC | 01:55 | |
| *** tphummel has joined #openstack-infra | 01:58 | |
| *** bradm has joined #openstack-infra | 01:58 | |
| *** sshnaidm is now known as sshnaidm|afk | 01:58 | |
| *** jeckersb is now known as jeckersb_gone | 01:59 | |
| openstackgerrit | Fei Long Wang proposed openstack-infra/irc-meetings: Propose to change Zaqar meeting time https://review.openstack.org/411564 | 02:00 |
| *** jeckersb_gone is now known as jeckersb | 02:00 | |
| *** sdake has quit IRC | 02:00 | |
| *** scarab_ has joined #openstack-infra | 02:00 | |
| *** claudiub|2 has quit IRC | 02:00 | |
| *** scarab_ has quit IRC | 02:04 | |
| *** scarab_ has joined #openstack-infra | 02:05 | |
| *** scarab_ has quit IRC | 02:09 | |
| *** asettle has joined #openstack-infra | 02:09 | |
| *** scarab_ has joined #openstack-infra | 02:09 | |
| *** scarab_ has quit IRC | 02:13 | |
| *** tuanluong has joined #openstack-infra | 02:13 | |
| *** asettle has quit IRC | 02:13 | |
| *** scarab_ has joined #openstack-infra | 02:14 | |
| openstackgerrit | Jamie Lennox proposed openstack-infra/nodepool: Accept user-home in config validator https://review.openstack.org/404519 | 02:15 |
| openstackgerrit | Merged openstack-infra/shade: Convert glance parts of task test to requests_mock https://review.openstack.org/410425 | 02:15 |
| openstackgerrit | Merged openstack-infra/shade: Pass md5 and sha256 to create_object sanely. https://review.openstack.org/411315 | 02:15 |
| openstackgerrit | Merged openstack-infra/shade: Stop double-reporting extra_data in exceptions https://review.openstack.org/411368 | 02:15 |
| openstackgerrit | Merged openstack-infra/shade: Clear the exception stack when we catch and continue https://review.openstack.org/411369 | 02:15 |
| *** aidin has joined #openstack-infra | 02:16 | |
| *** sreeram_v has joined #openstack-infra | 02:17 | |
| *** scarab_ has quit IRC | 02:17 | |
| *** scarab_ has joined #openstack-infra | 02:17 | |
| *** rbuzatu has joined #openstack-infra | 02:18 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Install dracut-generic-config package https://review.openstack.org/411540 | 02:19 |
| *** scarab_ has quit IRC | 02:22 | |
| *** sdake has joined #openstack-infra | 02:22 | |
| *** aeng has quit IRC | 02:23 | |
| *** rbuzatu has quit IRC | 02:23 | |
| *** scarab_ has joined #openstack-infra | 02:23 | |
| openstackgerrit | Merged openstack-infra/shade: Add total image import time to debug log https://review.openstack.org/411383 | 02:26 |
| openstackgerrit | Merged openstack-infra/shade: Magnum's keystone id is container-infra, not container https://review.openstack.org/411399 | 02:26 |
| *** gomarivera has quit IRC | 02:26 | |
| *** armax has quit IRC | 02:27 | |
| *** scarab_ has quit IRC | 02:28 | |
| *** Jeffrey4l has quit IRC | 02:31 | |
| *** Jeffrey4l has joined #openstack-infra | 02:31 | |
| *** markvoelker has joined #openstack-infra | 02:31 | |
| *** EricGonczer_ has quit IRC | 02:31 | |
| *** scarab_ has joined #openstack-infra | 02:32 | |
| openstackgerrit | wangxiyuan proposed openstack-infra/project-config: Add doc CI to zaqarclient https://review.openstack.org/411174 | 02:32 |
| *** Jeffrey4l has quit IRC | 02:33 | |
| *** Jeffrey4l has joined #openstack-infra | 02:33 | |
| *** EricGonczer_ has joined #openstack-infra | 02:34 | |
| *** markvoelker has quit IRC | 02:36 | |
| *** scarab_ has quit IRC | 02:36 | |
| *** sreeram_v has quit IRC | 02:36 | |
| *** gongysh has joined #openstack-infra | 02:37 | |
| *** sreeram_v has joined #openstack-infra | 02:37 | |
| *** sreeram_v has quit IRC | 02:41 | |
| *** scarab_ has joined #openstack-infra | 02:42 | |
| *** sdake has quit IRC | 02:42 | |
| *** sdake has joined #openstack-infra | 02:43 | |
| *** wolverineav has joined #openstack-infra | 02:43 | |
| *** kjorgensen has quit IRC | 02:44 | |
| *** scarab_ has quit IRC | 02:46 | |
| *** mmandel has joined #openstack-infra | 02:46 | |
| *** wolverineav has quit IRC | 02:48 | |
| *** scarab_ has joined #openstack-infra | 02:49 | |
| *** esberglu has quit IRC | 02:50 | |
| *** kjorgensen has joined #openstack-infra | 02:50 | |
| *** esberglu has joined #openstack-infra | 02:50 | |
| *** mmandel has quit IRC | 02:51 | |
| *** kjorgens_ has joined #openstack-infra | 02:51 | |
| *** yamamoto has joined #openstack-infra | 02:54 | |
| *** scarab_ has quit IRC | 02:54 | |
| *** kjorgensen has quit IRC | 02:54 | |
| *** esberglu has quit IRC | 02:55 | |
| *** scarab_ has joined #openstack-infra | 02:55 | |
| *** kjorgens_ has quit IRC | 02:55 | |
| *** yamahata has quit IRC | 02:58 | |
| *** scarab__ has joined #openstack-infra | 02:59 | |
| *** scarab_ has quit IRC | 02:59 | |
| *** sflanigan is now known as sflanigan_afk | 03:02 | |
| *** nherciu has quit IRC | 03:02 | |
| *** zhiyan has quit IRC | 03:02 | |
| *** scarab_ has joined #openstack-infra | 03:03 | |
| *** nherciu has joined #openstack-infra | 03:03 | |
| *** scarab__ has quit IRC | 03:03 | |
| *** sdake has quit IRC | 03:03 | |
| *** skelso has joined #openstack-infra | 03:05 | |
| *** yamamoto has quit IRC | 03:05 | |
| *** sekelso has joined #openstack-infra | 03:07 | |
| *** scarab_ has quit IRC | 03:07 | |
| *** scarab_ has joined #openstack-infra | 03:08 | |
| *** gouthamr has quit IRC | 03:10 | |
| *** asettle has joined #openstack-infra | 03:10 | |
| *** skelso has quit IRC | 03:10 | |
| *** scarab_ has quit IRC | 03:12 | |
| *** scarab_ has joined #openstack-infra | 03:13 | |
| *** asettle has quit IRC | 03:14 | |
| *** sdake has joined #openstack-infra | 03:15 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config: Fedora 23 omnibus removal https://review.openstack.org/411605 | 03:17 |
| *** scarab_ has quit IRC | 03:18 | |
| *** rbuzatu has joined #openstack-infra | 03:19 | |
| *** shu-mutou-AWAY is now known as shu-mutou | 03:22 | |
| *** mmandel has joined #openstack-infra | 03:23 | |
| *** scarab_ has joined #openstack-infra | 03:23 | |
| *** rbuzatu has quit IRC | 03:24 | |
| *** sflanigan_afk is now known as sflanigan | 03:26 | |
| *** mmandel has quit IRC | 03:27 | |
| *** scarab_ has quit IRC | 03:27 | |
| *** scarab__ has joined #openstack-infra | 03:27 | |
| *** dimtruck is now known as zz_dimtruck | 03:29 | |
| *** aeng has joined #openstack-infra | 03:32 | |
| *** scarab__ has quit IRC | 03:32 | |
| *** aidin has quit IRC | 03:32 | |
| *** tqtran has joined #openstack-infra | 03:33 | |
| *** amotoki has quit IRC | 03:33 | |
| *** amotoki has joined #openstack-infra | 03:34 | |
| *** rkukura has quit IRC | 03:34 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config: Update tools/build-image.sh with some useful options https://review.openstack.org/411608 | 03:35 |
| *** scarab_ has joined #openstack-infra | 03:35 | |
| *** sdake has quit IRC | 03:35 | |
| *** amotoki has quit IRC | 03:35 | |
| *** amotoki has joined #openstack-infra | 03:36 | |
| *** sdake has joined #openstack-infra | 03:36 | |
| *** scarab_ has quit IRC | 03:39 | |
| *** vikrant has joined #openstack-infra | 03:40 | |
| *** tqtran has quit IRC | 03:40 | |
| *** scarab_ has joined #openstack-infra | 03:41 | |
| *** Douhet has quit IRC | 03:42 | |
| *** amotoki has quit IRC | 03:42 | |
| *** baoli has joined #openstack-infra | 03:44 | |
| *** yamahata has joined #openstack-infra | 03:45 | |
| *** scarab_ has quit IRC | 03:45 | |
| *** scarab_ has joined #openstack-infra | 03:46 | |
| *** ramishra has quit IRC | 03:47 | |
| *** scarab_ has quit IRC | 03:50 | |
| *** jerryz has quit IRC | 03:52 | |
| *** snarwade has joined #openstack-infra | 03:52 | |
| *** scarab_ has joined #openstack-infra | 03:54 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Revert "Revert "Pause diskimage builds for centos-7"" https://review.openstack.org/411613 | 03:55 |
| pabelanger | ianw: jhesketh: ^we should land that again, so I can remove the broken centos-7 DIBs. I guess I didn't test well enough today | 03:56 |
| * jhesketh looks | 03:56 | |
| *** vladani has quit IRC | 03:56 | |
| *** psachin has joined #openstack-infra | 03:58 | |
| *** scarab_ has quit IRC | 03:58 | |
| pabelanger | jhesketh: ianw: once approved and merged, we'll need to run: nodepool dib-image-delete centos-7-0000000134 on nodepool.o.o to remove the broken image from all clouds | 03:59 |
| pabelanger | if you don't mind handling that | 03:59 |
| ianw | pabelanger: i can do that | 03:59 |
| ianw | i got about an hour though, so i don't really have time to rebuild | 04:00 |
| *** scarab_ has joined #openstack-infra | 04:00 | |
| pabelanger | ianw: np, rolling back gives us more time | 04:01 |
| jhesketh | I'm around too so if you need to leave ianw I can take over | 04:04 |
| ianw | jhesketh: heh, just a date with rogue one with the kids. so very important, obviously :) | 04:04 |
| jhesketh | that is clearly a priority :-) | 04:05 |
| *** scarab_ has quit IRC | 04:05 | |
| *** scarab__ has joined #openstack-infra | 04:05 | |
| openstackgerrit | Merged openstack-infra/project-config: Revert "Revert "Pause diskimage builds for centos-7"" https://review.openstack.org/411613 | 04:06 |
| pabelanger | nice, okay. | 04:07 |
| pabelanger | and I'm offline, thanks again | 04:07 |
| jhesketh | thanks pabelanger | 04:08 |
| jhesketh | ianw: it's merged if you wanted to kick the image? | 04:08 |
| ianw | yep, will do in a sec | 04:08 |
| *** scarab__ has quit IRC | 04:09 | |
| ianw | i get rid of all the building ones | 04:09 |
| *** aidin1 has joined #openstack-infra | 04:10 | |
| *** asettle has joined #openstack-infra | 04:10 | |
| *** maeker has joined #openstack-infra | 04:12 | |
| *** scarab_ has joined #openstack-infra | 04:13 | |
| *** zz_dimtruck is now known as dimtruck | 04:14 | |
| *** asettle has quit IRC | 04:15 | |
| *** scarab_ has quit IRC | 04:17 | |
| *** baoli has quit IRC | 04:17 | |
| *** scarab_ has joined #openstack-infra | 04:18 | |
| *** thorst has joined #openstack-infra | 04:18 | |
| ianw | ok, old build cleared out, and old building nodes cleared | 04:19 |
| *** rbuzatu has joined #openstack-infra | 04:21 | |
| *** sekelso has quit IRC | 04:21 | |
| *** scarab_ has quit IRC | 04:22 | |
| *** aidin1 has quit IRC | 04:24 | |
| *** dkehn__ has joined #openstack-infra | 04:25 | |
| *** rbuzatu has quit IRC | 04:25 | |
| *** scarab_ has joined #openstack-infra | 04:25 | |
| *** dkehn_ has quit IRC | 04:26 | |
| *** dave-mccowan has quit IRC | 04:26 | |
| *** john-davidge has joined #openstack-infra | 04:27 | |
| *** s-shiono has joined #openstack-infra | 04:29 | |
| *** scarab_ has quit IRC | 04:30 | |
| *** nadya has joined #openstack-infra | 04:31 | |
| *** john-davidge has quit IRC | 04:31 | |
| *** scarab_ has joined #openstack-infra | 04:31 | |
| *** maeker has quit IRC | 04:34 | |
| *** nicolasbock has quit IRC | 04:34 | |
| *** scarab_ has quit IRC | 04:35 | |
| *** scarab_ has joined #openstack-infra | 04:35 | |
| *** nadya has quit IRC | 04:35 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config: Fedora 23 omnibus removal https://review.openstack.org/411605 | 04:36 |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config: Pause & unready Fedora 23 in preparation for removal https://review.openstack.org/411620 | 04:36 |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config: Remove Fedora 23 from nodepool https://review.openstack.org/411621 | 04:36 |
| *** vladani has joined #openstack-infra | 04:36 | |
| jhesketh | thanks ianw | 04:37 |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config: Fedora 23 omnibus removal https://review.openstack.org/411605 | 04:38 |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config: Remove Fedora 23 from nodepool https://review.openstack.org/411621 | 04:38 |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config: Pause & unready Fedora 23 in preparation for removal https://review.openstack.org/411620 | 04:38 |
| *** scarab_ has quit IRC | 04:39 | |
| *** scarab_ has joined #openstack-infra | 04:40 | |
| *** hongbin has quit IRC | 04:44 | |
| *** scarab_ has quit IRC | 04:45 | |
| *** thorst has quit IRC | 04:45 | |
| *** scarab_ has joined #openstack-infra | 04:47 | |
| *** thorst has joined #openstack-infra | 04:49 | |
| *** scarab_ has quit IRC | 04:52 | |
| *** scarab_ has joined #openstack-infra | 04:53 | |
| *** wolverineav has joined #openstack-infra | 04:54 | |
| *** tphummel has quit IRC | 04:56 | |
| *** scarab_ has quit IRC | 04:56 | |
| *** scarab_ has joined #openstack-infra | 04:56 | |
| *** thorst has quit IRC | 04:57 | |
| *** wolverineav has quit IRC | 04:59 | |
| *** scarab_ has quit IRC | 05:01 | |
| *** wolverineav has joined #openstack-infra | 05:05 | |
| *** NileshSawant has joined #openstack-infra | 05:06 | |
| *** sreeram_v has joined #openstack-infra | 05:08 | |
| *** wolverineav has quit IRC | 05:09 | |
| *** asettle has joined #openstack-infra | 05:11 | |
| NileshSawant | Hello All, we have created CI setup, but default tests are also failing Can someone please help me out Log link : http://54.169.104.103//$LOG_PATH/console.html | 05:13 |
| NileshSawant | am not sure where we went worng | 05:14 |
| NileshSawant | default test for project are also faling | 05:14 |
| *** wolverineav has joined #openstack-infra | 05:14 | |
| NileshSawant | we are using project_config_repo: https://github.com/Kaminario/openstack-ci-configuration.git | 05:15 |
| NileshSawant | in /etc/puppet/environments/common.yaml | 05:15 |
| *** asettle has quit IRC | 05:15 | |
| NileshSawant | is this repo is correct ? or there is any misconfiguration ? | 05:16 |
| NileshSawant | Please help | 05:16 |
| *** rkukura has joined #openstack-infra | 05:18 | |
| *** wolverineav has quit IRC | 05:19 | |
| *** links has joined #openstack-infra | 05:21 | |
| *** sdake_ has joined #openstack-infra | 05:24 | |
| *** wolverineav has joined #openstack-infra | 05:24 | |
| *** sdake has quit IRC | 05:25 | |
| *** wolverineav has quit IRC | 05:28 | |
| *** scarab_ has joined #openstack-infra | 05:30 | |
| *** hurgleburgler has quit IRC | 05:33 | |
| *** scarab_ has quit IRC | 05:34 | |
| *** wolverineav has joined #openstack-infra | 05:34 | |
| *** tlian has quit IRC | 05:35 | |
| *** gomarivera has joined #openstack-infra | 05:37 | |
| *** tqtran has joined #openstack-infra | 05:37 | |
| *** wolverineav has quit IRC | 05:39 | |
| *** kjorgensen has joined #openstack-infra | 05:42 | |
| *** tqtran has quit IRC | 05:42 | |
| *** ramishra has joined #openstack-infra | 05:43 | |
| *** thorst has joined #openstack-infra | 05:44 | |
| *** wolverineav has joined #openstack-infra | 05:44 | |
| *** savihou has joined #openstack-infra | 05:46 | |
| *** kjorgensen has quit IRC | 05:46 | |
| *** wolverineav has quit IRC | 05:48 | |
| *** rbergeron has quit IRC | 05:51 | |
| *** scarab_ has joined #openstack-infra | 05:52 | |
| *** thorst has quit IRC | 05:52 | |
| *** camunoz has quit IRC | 05:53 | |
| *** vladani_ has joined #openstack-infra | 05:54 | |
| *** vladani has quit IRC | 05:54 | |
| *** scarab_ has quit IRC | 05:56 | |
| *** savihou has quit IRC | 05:56 | |
| *** vladani_ has quit IRC | 05:59 | |
| *** scarab_ has joined #openstack-infra | 06:01 | |
| *** jaosorior has joined #openstack-infra | 06:02 | |
| *** vladani has joined #openstack-infra | 06:03 | |
| *** scarab_ has quit IRC | 06:05 | |
| *** scarab_ has joined #openstack-infra | 06:06 | |
| *** scarab_ has quit IRC | 06:10 | |
| openstackgerrit | wangxiyuan proposed openstack-infra/project-config: Add doc CI to zaqarclient https://review.openstack.org/411174 | 06:11 |
| *** pgadiya has joined #openstack-infra | 06:12 | |
| *** rcernin has quit IRC | 06:13 | |
| *** gomarivera has quit IRC | 06:15 | |
| *** vladani has quit IRC | 06:20 | |
| *** scarab_ has joined #openstack-infra | 06:21 | |
| *** rbuzatu has joined #openstack-infra | 06:22 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/project-config: Make py35 jobs voting for tripleo jobs https://review.openstack.org/410674 | 06:22 |
| *** nadya has joined #openstack-infra | 06:22 | |
| *** jimbaker has quit IRC | 06:22 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/project-config: Make py35 jobs voting for tripleo-related jobs https://review.openstack.org/410674 | 06:23 |
| *** wolverineav has joined #openstack-infra | 06:23 | |
| *** rbergeron has joined #openstack-infra | 06:24 | |
| *** nadya has quit IRC | 06:24 | |
| *** scarab_ has quit IRC | 06:25 | |
| *** jimbaker has joined #openstack-infra | 06:26 | |
| *** gildub has quit IRC | 06:26 | |
| *** rbuzatu has quit IRC | 06:27 | |
| *** wolverineav has quit IRC | 06:27 | |
| *** nadya has joined #openstack-infra | 06:29 | |
| *** scarab_ has joined #openstack-infra | 06:29 | |
| *** vladani has joined #openstack-infra | 06:30 | |
| *** gomarivera has joined #openstack-infra | 06:31 | |
| *** scarab__ has joined #openstack-infra | 06:32 | |
| *** scarab_ has quit IRC | 06:32 | |
| *** sdake_ has quit IRC | 06:32 | |
| *** sdake has joined #openstack-infra | 06:34 | |
| *** esikachev has joined #openstack-infra | 06:34 | |
| *** scarab_ has joined #openstack-infra | 06:36 | |
| *** aeng has quit IRC | 06:36 | |
| *** rcernin has joined #openstack-infra | 06:37 | |
| *** scarab__ has quit IRC | 06:37 | |
| *** Mmike has quit IRC | 06:37 | |
| *** armax has joined #openstack-infra | 06:38 | |
| *** thorst has joined #openstack-infra | 06:39 | |
| *** scarab__ has joined #openstack-infra | 06:40 | |
| *** scarab_ has quit IRC | 06:41 | |
| *** markus_z has joined #openstack-infra | 06:42 | |
| *** rcernin has quit IRC | 06:43 | |
| *** wolverineav has joined #openstack-infra | 06:43 | |
| *** scarab__ has quit IRC | 06:45 | |
| *** scarab_ has joined #openstack-infra | 06:46 | |
| *** thorst has quit IRC | 06:46 | |
| *** wolverineav has quit IRC | 06:47 | |
| *** scarab_ has quit IRC | 06:51 | |
| *** matbu is now known as matbu|halfpto | 06:51 | |
| *** Mmike has joined #openstack-infra | 06:51 | |
| *** Mmike has joined #openstack-infra | 06:51 | |
| *** scarab_ has joined #openstack-infra | 06:52 | |
| *** askb has quit IRC | 06:53 | |
| *** rcernin has joined #openstack-infra | 06:54 | |
| *** john-davidge has joined #openstack-infra | 06:54 | |
| *** scarab_ has quit IRC | 06:57 | |
| *** scarab_ has joined #openstack-infra | 06:58 | |
| *** john-davidge has quit IRC | 06:59 | |
| *** gomarivera has quit IRC | 07:00 | |
| *** ramishra has quit IRC | 07:01 | |
| *** scarab_ has quit IRC | 07:03 | |
| *** sdake has quit IRC | 07:03 | |
| *** andreas_s has joined #openstack-infra | 07:03 | |
| *** scarab_ has joined #openstack-infra | 07:05 | |
| *** ramishra has joined #openstack-infra | 07:06 | |
| openstackgerrit | Tuan Luong-Anh proposed openstack-infra/zuul: Replace assertEquals with assertEqual https://review.openstack.org/400119 | 07:09 |
| *** vladani has quit IRC | 07:09 | |
| *** scarab_ has quit IRC | 07:10 | |
| *** vladani has joined #openstack-infra | 07:10 | |
| *** nadya has quit IRC | 07:10 | |
| openstackgerrit | Anastasia Kravets proposed openstack-infra/project-config: ec2-api: set up ec2-api project api-ref job https://review.openstack.org/411322 | 07:11 |
| *** rkukura has quit IRC | 07:11 | |
| *** asettle has joined #openstack-infra | 07:12 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul: Fix 'Add hostname to merger path' https://review.openstack.org/411664 | 07:16 |
| *** asettle has quit IRC | 07:17 | |
| *** adriant has quit IRC | 07:20 | |
| *** adreznec has quit IRC | 07:21 | |
| *** bnemec has quit IRC | 07:21 | |
| *** gongysh has quit IRC | 07:22 | |
| *** abregman has joined #openstack-infra | 07:23 | |
| *** aidin has joined #openstack-infra | 07:24 | |
| *** aidin has quit IRC | 07:29 | |
| *** scarab_ has joined #openstack-infra | 07:31 | |
| *** tphummel has joined #openstack-infra | 07:31 | |
| AJaeger | NileshSawant: better ask on the #openstack-third-party-ci channel | 07:31 |
| *** scarab_ has quit IRC | 07:34 | |
| *** dimtruck is now known as zz_dimtruck | 07:34 | |
| *** scarab_ has joined #openstack-infra | 07:34 | |
| *** jpena|off is now known as jpena | 07:35 | |
| *** bnemec has joined #openstack-infra | 07:37 | |
| *** scarab__ has joined #openstack-infra | 07:38 | |
| *** shu-mutou is now known as shu-mutou-AWAY | 07:38 | |
| *** thorst has joined #openstack-infra | 07:38 | |
| *** tqtran has joined #openstack-infra | 07:39 | |
| *** abregman has quit IRC | 07:39 | |
| *** mmandel has joined #openstack-infra | 07:39 | |
| *** scarab_ has quit IRC | 07:39 | |
| *** derekh has joined #openstack-infra | 07:40 | |
| AJaeger | ianw: could you change your WIP to +A in https://review.openstack.org/#/c/408330/ , please? | 07:41 |
| *** scarab__ has quit IRC | 07:43 | |
| *** tqtran has quit IRC | 07:43 | |
| *** mmandel has quit IRC | 07:43 | |
| *** thorst has quit IRC | 07:44 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Add Fuxi job to run fullstack tests https://review.openstack.org/408330 | 07:44 |
| *** derekh has quit IRC | 07:47 | |
| *** scarab_ has joined #openstack-infra | 07:49 | |
| *** e0ne has joined #openstack-infra | 07:52 | |
| *** scarab_ has quit IRC | 07:53 | |
| *** pcaruana has joined #openstack-infra | 07:56 | |
| *** florianf has joined #openstack-infra | 07:57 | |
| *** tesseract has joined #openstack-infra | 07:57 | |
| *** tesseract is now known as Guest31304 | 07:58 | |
| *** scarab_ has joined #openstack-infra | 07:59 | |
| *** scarab_ has quit IRC | 08:04 | |
| *** ihrachys has joined #openstack-infra | 08:04 | |
| *** witek has joined #openstack-infra | 08:05 | |
| openstackgerrit | Dougal Matthews proposed openstack-infra/tripleo-ci: Update my feed to use the tripleo tag https://review.openstack.org/411687 | 08:06 |
| *** jamielennox is now known as jamielennox|away | 08:07 | |
| openstackgerrit | Dougal Matthews proposed openstack-infra/tripleo-ci: Use a more specific feed for my blog https://review.openstack.org/411687 | 08:07 |
| *** scarab_ has joined #openstack-infra | 08:08 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack-infra/project-config: Normalize projects.yaml https://review.openstack.org/411688 | 08:09 |
| *** adreznec has joined #openstack-infra | 08:09 | |
| *** nadya has joined #openstack-infra | 08:11 | |
| *** adam_g has quit IRC | 08:12 | |
| *** adam_g has joined #openstack-infra | 08:12 | |
| *** scarab_ has quit IRC | 08:12 | |
| *** scarab_ has joined #openstack-infra | 08:13 | |
| *** dtardivel has joined #openstack-infra | 08:14 | |
| *** jamielennox|away is now known as jamielennox | 08:14 | |
| *** ralonsoh has joined #openstack-infra | 08:15 | |
| *** vladani has quit IRC | 08:15 | |
| *** vladani has joined #openstack-infra | 08:16 | |
| yolanda | good morning | 08:16 |
| *** shardy has joined #openstack-infra | 08:16 | |
| *** scarab_ has quit IRC | 08:18 | |
| *** arxcruz has quit IRC | 08:20 | |
| *** scarab_ has joined #openstack-infra | 08:21 | |
| openstackgerrit | Steven Hardy proposed openstack-infra/tripleo-ci: Implement major upgrade for Newton to Ocata https://review.openstack.org/404831 | 08:21 |
| *** vladani has quit IRC | 08:23 | |
| *** rbuzatu has joined #openstack-infra | 08:23 | |
| *** matrohon has joined #openstack-infra | 08:23 | |
| *** scarab_ has quit IRC | 08:24 | |
| *** scarab_ has joined #openstack-infra | 08:24 | |
| AJaeger | morning, yolanda ! | 08:26 |
| *** rbuzatu has quit IRC | 08:28 | |
| *** scarab_ has quit IRC | 08:28 | |
| *** e0ne has quit IRC | 08:29 | |
| *** s-shiono has quit IRC | 08:31 | |
| *** amoralej|off is now known as amoralej | 08:31 | |
| *** rlandy|bbl is now known as rlandy | 08:31 | |
| *** rlandy has quit IRC | 08:31 | |
| *** jaosorior has quit IRC | 08:32 | |
| *** ramishra has quit IRC | 08:32 | |
| *** _Cyclone_ has quit IRC | 08:33 | |
| *** jaosorior has joined #openstack-infra | 08:33 | |
| *** flepied has joined #openstack-infra | 08:33 | |
| *** asettle has joined #openstack-infra | 08:34 | |
| openstackgerrit | Merged openstack-infra/project-config: Normalize projects.yaml https://review.openstack.org/411688 | 08:34 |
| *** rbuzatu has joined #openstack-infra | 08:35 | |
| *** asettle has quit IRC | 08:39 | |
| *** thorst has joined #openstack-infra | 08:39 | |
| *** ihrachys has quit IRC | 08:39 | |
| openstackgerrit | Merged openstack-infra/project-config: Add Fuxi job to run fullstack tests https://review.openstack.org/408330 | 08:40 |
| *** claudiub|2 has joined #openstack-infra | 08:40 | |
| *** pblaho has quit IRC | 08:42 | |
| *** hogepodge has quit IRC | 08:42 | |
| witek | AJaeger: could you please merge this one for us https://review.openstack.org/411411/ | 08:43 |
| witek | our tempest tests are blocked at the moment :( | 08:43 |
| *** thorst has quit IRC | 08:44 | |
| openstackgerrit | Merged openstack-infra/project-config: Implement Experimental Infra jobs for Puppet 4 testing https://review.openstack.org/410483 | 08:44 |
| *** hogepodge has joined #openstack-infra | 08:44 | |
| openstackgerrit | Merged openstack-infra/project-config: Add doc CI to zaqarclient https://review.openstack.org/411174 | 08:44 |
| openstackgerrit | Merged openstack-infra/project-config: Make py35 jobs voting for tripleo-related jobs https://review.openstack.org/410674 | 08:44 |
| openstackgerrit | Merged openstack-infra/project-config: ec2-api: set up ec2-api project api-ref job https://review.openstack.org/411322 | 08:44 |
| openstackgerrit | Merged openstack-infra/project-config: Run Cinder in-tree tests: devstack-plugin-nfs-nv https://review.openstack.org/409169 | 08:45 |
| *** _ody has quit IRC | 08:45 | |
| openstackgerrit | Merged openstack-infra/project-config: Run Cinder in-tree tests: full-bdd https://review.openstack.org/409170 | 08:48 |
| openstackgerrit | Merged openstack-infra/project-config: cleanup: cannot break out of case https://review.openstack.org/411532 | 08:49 |
| *** flepied has quit IRC | 08:49 | |
| *** gongysh has joined #openstack-infra | 08:49 | |
| openstackgerrit | Merged openstack-infra/project-config: Initiate end of life of openstack/anvil https://review.openstack.org/411542 | 08:50 |
| openstackgerrit | Merged openstack-infra/project-config: Update tools/build-image.sh with some useful options https://review.openstack.org/411608 | 08:50 |
| *** sflanigan has quit IRC | 08:51 | |
| *** yanyanhu_ has quit IRC | 08:52 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Fedora 23 omnibus removal https://review.openstack.org/411605 | 08:52 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Pause & unready Fedora 23 in preparation for removal https://review.openstack.org/411620 | 08:53 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Remove Fedora 23 from nodepool https://review.openstack.org/411621 | 08:53 |
| *** med_ has quit IRC | 08:53 | |
| *** _Cyclone_ has joined #openstack-infra | 08:53 | |
| *** vladani has joined #openstack-infra | 08:54 | |
| *** ccamacho|out is now known as ccamacho | 08:55 | |
| *** jaosorior has quit IRC | 08:55 | |
| openstackgerrit | Merged openstack-infra/project-config: Drop Java Implemention codebase https://review.openstack.org/410728 | 08:56 |
| *** med_ has joined #openstack-infra | 08:57 | |
| *** med_ is now known as Guest67717 | 08:57 | |
| *** zzzeek has quit IRC | 09:00 | |
| *** ihrachys has joined #openstack-infra | 09:00 | |
| openstackgerrit | Merged openstack-infra/project-config: Monasca Gate Setup for ORM+PostgreSQL https://review.openstack.org/366617 | 09:01 |
| *** zzzeek has joined #openstack-infra | 09:02 | |
| openstackgerrit | Merged openstack-infra/project-config: python-openstackclient: Make Python 3.5 check voting https://review.openstack.org/411349 | 09:02 |
| openstackgerrit | Merged openstack-infra/project-config: [mistral] Fix for failing dsvm gates https://review.openstack.org/411283 | 09:03 |
| *** lucas-afk is now known as lucasagomes | 09:04 | |
| *** ihrachys has quit IRC | 09:05 | |
| *** ihrachys has joined #openstack-infra | 09:10 | |
| amoralej | hi, yolanda, it seems that centos7 jobs are not stuck in queued status | 09:11 |
| amoralej | *are stuck* i mean | 09:11 |
| *** rossella_s has joined #openstack-infra | 09:12 | |
| *** kaisers1 has joined #openstack-infra | 09:13 | |
| *** _ody has joined #openstack-infra | 09:14 | |
| openstackgerrit | Witold Bedyk proposed openstack-infra/project-config: Set temporarily Monasca Java tempest tests as -nv https://review.openstack.org/411411 | 09:14 |
| *** itisha has joined #openstack-infra | 09:17 | |
| yolanda | hi amoralej let me look | 09:19 |
| *** lucasagomes is now known as lucas-brb | 09:19 | |
| *** flepied has joined #openstack-infra | 09:19 | |
| *** knangia has quit IRC | 09:20 | |
| openstackgerrit | Alfredo Moralejo proposed openstack-infra/tripleo-ci: Update packates after configuring openstack repos https://review.openstack.org/411725 | 09:24 |
| yolanda | amoralej, looks as high demand | 09:24 |
| *** yamahata has quit IRC | 09:24 | |
| amoralej | ok, i'll wait then, tkx!! | 09:24 |
| yolanda | there are lots of nodes in building state | 09:24 |
| openstackgerrit | Alfredo Moralejo proposed openstack-infra/tripleo-ci: Update packages after configuring openstack repos https://review.openstack.org/411725 | 09:25 |
| openstackgerrit | Merged openstack-infra/project-config: Fix regex to run keystone plugin tests https://review.openstack.org/410295 | 09:25 |
| openstackgerrit | Witold Bedyk proposed openstack-infra/project-config: Set temporarily Monasca Java tempest tests as -nv https://review.openstack.org/411411 | 09:25 |
| *** john-davidge has joined #openstack-infra | 09:29 | |
| *** john-davidge has quit IRC | 09:29 | |
| *** john-davidge has joined #openstack-infra | 09:29 | |
| *** martinkopec has joined #openstack-infra | 09:30 | |
| *** markus_z has quit IRC | 09:31 | |
| *** panda|Zz is now known as panda | 09:33 | |
| *** asettle has joined #openstack-infra | 09:33 | |
| *** pahuang has quit IRC | 09:33 | |
| *** thorst has joined #openstack-infra | 09:39 | |
| *** tqtran has joined #openstack-infra | 09:41 | |
| *** derekh has joined #openstack-infra | 09:41 | |
| openstackgerrit | Witold Bedyk proposed openstack-infra/project-config: Set temporarily Monasca Java tempest tests as -nv https://review.openstack.org/411411 | 09:43 |
| *** thorst has quit IRC | 09:44 | |
| *** esikachev has quit IRC | 09:44 | |
| *** tqtran has quit IRC | 09:45 | |
| *** markus_z has joined #openstack-infra | 09:46 | |
| *** jamielennox is now known as jamielennox|away | 09:48 | |
| derekh | the tripleo cloud is having trouble booting the latest centos image from nodepool, are any other clouds having problems with it? | 09:49 |
| openstackgerrit | Merged openstack-infra/project-config: remove retired cookbooks from infrastructure system https://review.openstack.org/411216 | 09:51 |
| *** dizquierdo_ has joined #openstack-infra | 09:52 | |
| *** e0ne has joined #openstack-infra | 09:52 | |
| AJaeger | witek: once it's ready and has another +2 ;) | 09:53 |
| AJaeger | yolanda: could you review https://review.openstack.org/#/c/411465/ and https://review.openstack.org/411411/ , please? | 09:54 |
| *** hashar has joined #openstack-infra | 09:54 | |
| AJaeger | derekh: thre was some discussion about broken centos images in the last 12 hours, see backscroll (or read on eavesdrop.o.o). I don't know whether it was fully solved ;( | 09:55 |
| derekh | AJaeger: ok, will find it on eavesdropp, thanks | 09:55 |
| *** jamielennox|away is now known as jamielennox | 09:55 | |
| AJaeger | rcarrillocruz: thanks! | 09:56 |
| AJaeger | rcarrillocruz: ther's also https://review.openstack.org/411476, please | 09:57 |
| *** rbuzatu has quit IRC | 09:57 | |
| yolanda | ah, ricky was faster :) | 09:57 |
| AJaeger | ;) | 09:58 |
| *** chem has joined #openstack-infra | 09:59 | |
| *** jordanP has joined #openstack-infra | 10:00 | |
| *** mmandel has joined #openstack-infra | 10:00 | |
| *** pgadiya has quit IRC | 10:01 | |
| rcarrillocruz | lgtm | 10:01 |
| *** rbuzatu has joined #openstack-infra | 10:01 | |
| *** rbuzatu has quit IRC | 10:01 | |
| openstackgerrit | Merged openstack-infra/irc-meetings: Propose to change Zaqar meeting time https://review.openstack.org/411564 | 10:01 |
| derekh | AJaeger: ok, as best I can tell they were trying to solve the same problem last night, and ran this command somewhere to remove a old image "nodepool dib-image-delete centos-7-0000000134" | 10:02 |
| derekh | AJaeger: on rh1 we are now using nodepool_build_id == 135 , which also seem broke | 10:02 |
| derekh | AJaeger: would it also make sense to delete that? | 10:02 |
| openstackgerrit | Merged openstack-infra/project-config: Fedora 23 omnibus removal https://review.openstack.org/411605 | 10:02 |
| *** e0ne has quit IRC | 10:02 | |
| openstackgerrit | Jan Klare proposed openstack-infra/infra-manual: add retired projects to 'reference/legacy.yml' https://review.openstack.org/411738 | 10:03 |
| *** markus_z has quit IRC | 10:03 | |
| openstackgerrit | Merged openstack-infra/project-config: Update octavia projects for octavia spinout https://review.openstack.org/410997 | 10:03 |
| openstackgerrit | Merged openstack-infra/project-config: Set temporarily Monasca Java tempest tests as -nv https://review.openstack.org/411411 | 10:04 |
| openstackgerrit | Merged openstack-infra/project-config: Always clone quickstart extras with quickstart https://review.openstack.org/401282 | 10:04 |
| openstackgerrit | Merged openstack-infra/project-config: Add mistral coverage job https://review.openstack.org/411136 | 10:04 |
| openstackgerrit | Merged openstack-infra/project-config: charm-ceph-fs: add python35-jobs https://review.openstack.org/411265 | 10:04 |
| *** mmandel has quit IRC | 10:05 | |
| openstackgerrit | Merged openstack-infra/project-config: Revert "networking-midonet: Remove newton bitrot jobs" https://review.openstack.org/409561 | 10:05 |
| openstackgerrit | Merged openstack-infra/project-config: Pause & unready Fedora 23 in preparation for removal https://review.openstack.org/411620 | 10:05 |
| AJaeger | derekh: sorry, cannot help with that. Going back to an older image looks like a good solution but no idea about numbers. | 10:06 |
| derekh | AJaeger: ok | 10:06 |
| *** rfolco has joined #openstack-infra | 10:07 | |
| *** chem has quit IRC | 10:08 | |
| *** chem has joined #openstack-infra | 10:08 | |
| *** rossella_s has quit IRC | 10:11 | |
| openstackgerrit | Oksana Voshchana proposed openstack-infra/devstack-gate: TEST for multinode grenade devstack https://review.openstack.org/396679 | 10:11 |
| openstackgerrit | Merged openstack-infra/project-config: Trusty/py34 cleanup, part 2 https://review.openstack.org/411476 | 10:11 |
| *** rossella_s has joined #openstack-infra | 10:12 | |
| openstackgerrit | Merged openstack-infra/project-config: Trusty/py34 cleanup https://review.openstack.org/411465 | 10:12 |
| *** esikachev has joined #openstack-infra | 10:13 | |
| *** pgadiya has joined #openstack-infra | 10:13 | |
| *** tinwood is now known as tinwood_holiday | 10:14 | |
| *** markus_z has joined #openstack-infra | 10:15 | |
| *** openstackgerrit has quit IRC | 10:18 | |
| *** rossella_s has quit IRC | 10:18 | |
| *** rossella_s has joined #openstack-infra | 10:18 | |
| *** yamamoto has joined #openstack-infra | 10:22 | |
| ttx | fungi: ok, so I tracked down the issue with the governance-website publication. The issue is that for some reason ZUUL_REFNAME is set to "refs/heads/master" (instead of "master") which triggers some logic in ./jenkins/scripts/run-docs.sh to handle tagged releases | 10:22 |
| ttx | That's pretty obvious if you compare http://logs.openstack.org/9a/9a5461467385b75a5118ca6f09e2c6a63db91e3f/post/static-governance-publish/e0473db/console.html with http://logs.openstack.org/d3/d3eb09d1c075764c7ba2be03099a04bf0949e69f/post/static-governance-website-publish/74e7f98/console.html | 10:23 |
| ttx | now I'm not sure how to best fix that | 10:25 |
| *** openstackgerrit has joined #openstack-infra | 10:29 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Trusty/py34 cleanup, part3 https://review.openstack.org/411745 | 10:29 |
| *** priteau has joined #openstack-infra | 10:30 | |
| *** lucas-brb is now known as lucasagomes | 10:31 | |
| *** rbuzatu has joined #openstack-infra | 10:31 | |
| *** arxcruz has joined #openstack-infra | 10:32 | |
| AJaeger | ttx, fungi: did we change ZUUL_REFNAME from master to refs/heads/master ? That might cause some other problems as well... | 10:32 |
| *** thorst has joined #openstack-infra | 10:33 | |
| *** florianf has quit IRC | 10:36 | |
| *** rbuzatu has quit IRC | 10:37 | |
| AJaeger | rcarrillocruz, yolanda: and next part of trusty cleanup - in case you have some review time, please: https://review.openstack.org/411745 | 10:37 |
| ttx | AJaeger: I think in most (all) repos it's properly set to "master". Just this one (openstack/governance-website) ends up with the wrong one | 10:37 |
| dulek | Any idea how to exactly pinpoint the first occurrence when job started to fail if it happened earlier than logstash limit? | 10:38 |
| *** snarwade is now known as snarwade|afk | 10:41 | |
| *** florianf has joined #openstack-infra | 10:41 | |
| *** thorst has quit IRC | 10:41 | |
| *** saibarspeis has joined #openstack-infra | 10:50 | |
| openstackgerrit | Alfredo Moralejo proposed openstack-infra/tripleo-ci: Update packages after configuring openstack repos https://review.openstack.org/411725 | 10:51 |
| *** sreeram_v has quit IRC | 10:54 | |
| *** pblaho has joined #openstack-infra | 10:55 | |
| *** ociuhandu has joined #openstack-infra | 10:55 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/zuul: WIP Add per-repo public and private keys https://review.openstack.org/406382 | 10:56 |
| openstackgerrit | Julia Varlamova proposed openstack-infra/devstack-gate: Add MULTI_KEYSTONE flag and new role https://review.openstack.org/394895 | 10:56 |
| *** rkukura has joined #openstack-infra | 11:01 | |
| *** sreeram_v has joined #openstack-infra | 11:02 | |
| *** tphummel has quit IRC | 11:02 | |
| *** yamamoto has quit IRC | 11:03 | |
| BobBall | How are we meant to use devstack plugins in conjuction with devstack-gate in a third party CI? ERROR_ON_CLONE must be set to True in "gate" jobs to ensure the git trees come from Zuul, but that prevents devstack from doing a git clone of the repo with our devstack plugin in. Do all devstack plugins have to be added to http://git.openstack.org/cgit/openstack-infra/devstack-gate/tree/devstack-vm-gate-wrap.sh ? | 11:04 |
| *** rkukura has quit IRC | 11:06 | |
| *** sreeram_v has quit IRC | 11:06 | |
| BobBall | *sigh* And just after posting that question I find the right google terms... | 11:06 |
| BobBall | https://github.com/openstack-dev/devstack/blob/master/doc/source/plugins.rst | 11:06 |
| *** tpsilva has joined #openstack-infra | 11:06 | |
| *** vladani has quit IRC | 11:07 | |
| *** e0ne has joined #openstack-infra | 11:07 | |
| openstackgerrit | Oksana Voshchana proposed openstack-infra/devstack-gate: TEST for multinode grenade devstack https://review.openstack.org/396679 | 11:09 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Fix python-heatclient docs publishing https://review.openstack.org/411757 | 11:11 |
| AJaeger | argh, we miss documents for heat, the above fixed it. Team, please review ^ | 11:13 |
| *** chem has quit IRC | 11:18 | |
| *** vladani has joined #openstack-infra | 11:18 | |
| *** tosky has joined #openstack-infra | 11:19 | |
| *** chem has joined #openstack-infra | 11:19 | |
| *** gongysh has quit IRC | 11:19 | |
| *** lseki has joined #openstack-infra | 11:20 | |
| *** sdague has joined #openstack-infra | 11:21 | |
| *** dgonzalez has quit IRC | 11:22 | |
| *** dgonzalez has joined #openstack-infra | 11:24 | |
| *** thorst has joined #openstack-infra | 11:29 | |
| openstackgerrit | Sean Dague proposed openstack-infra/devstack-gate: Introduce roles into the feature matrix https://review.openstack.org/386072 | 11:29 |
| openstackgerrit | Sean Dague proposed openstack-infra/devstack-gate: enable placement api by default for ocata and beyond https://review.openstack.org/409871 | 11:29 |
| openstackgerrit | Ihar Hrachyshka proposed openstack-infra/project-config: neutron: move grenade multinode linuxbridge job into check queue https://review.openstack.org/411263 | 11:31 |
| *** thorst has quit IRC | 11:36 | |
| openstackgerrit | Valeriy Ponomaryov proposed openstack-infra/project-config: [Manila] Disable tempest for grenade job https://review.openstack.org/411768 | 11:36 |
| *** jaosorior has joined #openstack-infra | 11:38 | |
| *** jkilpatr has quit IRC | 11:44 | |
| *** nicolasbock has joined #openstack-infra | 11:46 | |
| *** aidin has joined #openstack-infra | 11:46 | |
| *** sekelso has joined #openstack-infra | 11:52 | |
| lseki | hey guys, is there any way to limit the number of nodes that nodepool spawns, for a specific label? | 11:52 |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci: Postci tests trial and error https://review.openstack.org/411189 | 11:55 |
| EmilienM | rcarrillocruz, yolanda: we're having a serious issue in tripleo CI, would it be possible to revert the latest centos image in nodepool? cc derekh for more details | 11:59 |
| *** _Cyclone_ has quit IRC | 12:00 | |
| yolanda | EmilienM, that's problem you mentioned with qemu? | 12:00 |
| EmilienM | yolanda: no | 12:01 |
| EmilienM | yolanda: https://bugs.launchpad.net/tripleo/+bug/1650503 | 12:01 |
| openstack | Launchpad bug 1650503 in tripleo "nodepool slaves failing to boot" [Critical,Triaged] | 12:01 |
| lseki | It seems that nodepool creates a node for a specific label depending on the demand (e.g. queue for build A > queue for build B), but I'd like to run as many builds B as possible. | 12:02 |
| *** yamamoto has joined #openstack-infra | 12:03 | |
| derekh | yolanda: from eavesdrop is looks like they deleted a broken image lastnight, but I guess nodepool built and uploaded a new one | 12:04 |
| yolanda | yep, each day the images are updated | 12:05 |
| yolanda | i see two images with 4 hours of difference only | 12:06 |
| yolanda | can you wait for other cores to join? i honestly have not been involved in image building from a long time | 12:06 |
| derekh | I beleive they did this last night "nodepool dib-image-delete centos-7-0000000134" | 12:07 |
| derekh | but nodepool is now using 135 | 12:07 |
| yolanda | EmilienM, derekh , problems are longer than 4 hours? | 12:07 |
| yolanda | ah, 4 days | 12:07 |
| yolanda | it's ok | 12:07 |
| *** nadya has quit IRC | 12:07 | |
| yolanda | there is one 6 hours long, another 4 days, that shall be the working one | 12:07 |
| derekh | yup, sounds like it, | 12:08 |
| derekh | centos jobs are queing up everywhere not just the tripleo jobs | 12:08 |
| yolanda | what i'm not sure is about the upload to providers, because it has been already uploaded | 12:09 |
| yolanda | so even if i remove the original one, each provider has its own | 12:09 |
| derekh | yolanda: ok, /me has no idea either | 12:09 |
| yolanda | yep, i hope people like pabelanger, fungi... will join soon from US | 12:09 |
| derekh | ok, thanks for looking | 12:10 |
| yolanda | i have not been involved in uploads so i prefer for them to do the operation, and get more knowledge for the next time to help myself | 12:10 |
| derekh | ok | 12:10 |
| *** savihou has joined #openstack-infra | 12:11 | |
| *** yamamoto has quit IRC | 12:13 | |
| *** dtantsur|afk is now known as dtantsur | 12:16 | |
| *** ramishra has joined #openstack-infra | 12:16 | |
| *** jkilpatr has joined #openstack-infra | 12:16 | |
| *** kjorgensen has joined #openstack-infra | 12:17 | |
| *** savihou has quit IRC | 12:20 | |
| *** savihou has joined #openstack-infra | 12:20 | |
| *** yujunz has joined #openstack-infra | 12:20 | |
| *** yujunz is now known as Guest38540 | 12:21 | |
| *** kjorgensen has quit IRC | 12:22 | |
| EmilienM | fungi, pabelanger: when you're around, we need your help asap | 12:22 |
| openstackgerrit | Ivan Udovichenko proposed openstack-infra/puppet-apps_site: Ensure that memcached service is installed and running https://review.openstack.org/407100 | 12:22 |
| *** Guest38540 has quit IRC | 12:23 | |
| *** chem has quit IRC | 12:23 | |
| *** chem has joined #openstack-infra | 12:24 | |
| *** thorst has joined #openstack-infra | 12:25 | |
| yolanda | so in the past there was a nodepool image-upload command that allowed to upload desired image to the providers. But now is different, and I don't have enough knowledge for dealing with that :( | 12:28 |
| EmilienM | yolanda: no worries, you already helped a lot | 12:28 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Fix python-heatclient docs publishing https://review.openstack.org/411757 | 12:29 |
| *** pgadiya has quit IRC | 12:31 | |
| *** thorst has quit IRC | 12:33 | |
| *** jeckersb is now known as jeckersb_gone | 12:33 | |
| *** dave-mccowan has joined #openstack-infra | 12:33 | |
| *** rbuzatu has joined #openstack-infra | 12:34 | |
| *** tuanluong has quit IRC | 12:35 | |
| *** lucasagomes is now known as lucas-hungry | 12:35 | |
| *** rbuzatu has quit IRC | 12:39 | |
| *** mhickey has joined #openstack-infra | 12:39 | |
| *** wolverineav has joined #openstack-infra | 12:40 | |
| *** nadya has joined #openstack-infra | 12:40 | |
| AJaeger | rcarrillocruz: could you review again, please? Now tests pass ^ | 12:40 |
| *** links has quit IRC | 12:42 | |
| *** markvoelker has joined #openstack-infra | 12:45 | |
| *** gouthamr has joined #openstack-infra | 12:46 | |
| *** dprince has joined #openstack-infra | 12:46 | |
| *** pgadiya has joined #openstack-infra | 12:47 | |
| *** jamesdenton has joined #openstack-infra | 12:47 | |
| *** thorst has joined #openstack-infra | 12:49 | |
| *** jamesden_ has joined #openstack-infra | 12:50 | |
| *** jamesdenton has quit IRC | 12:52 | |
| *** gomarivera has joined #openstack-infra | 12:53 | |
| *** dizquierdo_ has quit IRC | 12:55 | |
| *** priteau has quit IRC | 12:56 | |
| *** thorst has quit IRC | 12:56 | |
| *** sekelso has quit IRC | 12:56 | |
| *** thorst has joined #openstack-infra | 12:57 | |
| *** gomarivera has quit IRC | 12:57 | |
| *** matbu|halfpto is now known as matbu | 13:00 | |
| *** weshay_bbiab is now known as weshay | 13:01 | |
| *** vikrant has quit IRC | 13:02 | |
| *** thorst has quit IRC | 13:05 | |
| *** esberglu has joined #openstack-infra | 13:06 | |
| *** sekelso has joined #openstack-infra | 13:09 | |
| *** baoli has joined #openstack-infra | 13:09 | |
| *** esberglu has quit IRC | 13:10 | |
| *** jamesden_ has quit IRC | 13:12 | |
| strigazi | clarkb, yt? | 13:13 |
| *** aidin has quit IRC | 13:13 | |
| strigazi | mordred, yt? | 13:13 |
| *** xyang1 has joined #openstack-infra | 13:15 | |
| *** sekelso has quit IRC | 13:15 | |
| *** sreeram_v has joined #openstack-infra | 13:18 | |
| *** EricGonc_ has joined #openstack-infra | 13:19 | |
| *** jcoufal has joined #openstack-infra | 13:20 | |
| *** EricGonc_ has quit IRC | 13:20 | |
| *** EricGonczer_ has quit IRC | 13:21 | |
| openstackgerrit | Witold Bedyk proposed openstack-infra/project-config: Revert "Set temporarily Monasca Java tempest tests as -nv" https://review.openstack.org/411798 | 13:21 |
| openstackgerrit | Javier Peña proposed openstack-infra/project-config: Add missing closing " in tripleo job https://review.openstack.org/411800 | 13:22 |
| *** EricGonczer_ has joined #openstack-infra | 13:23 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: tripleo: add experimental job for containerized undercloud https://review.openstack.org/411802 | 13:24 |
| EmilienM | dprince, flaper87 ^ | 13:24 |
| openstackgerrit | Oksana Voshchana proposed openstack-infra/devstack-gate: TEST for multinode grenade devstack https://review.openstack.org/396679 | 13:24 |
| *** pradk has joined #openstack-infra | 13:25 | |
| openstackgerrit | Oksana Voshchana proposed openstack-infra/devstack-gate: TEST for multinode grenade devstack https://review.openstack.org/396679 | 13:26 |
| *** mkoderer has joined #openstack-infra | 13:27 | |
| *** rbuzatu has joined #openstack-infra | 13:27 | |
| amoralej | could you take a look into https://review.openstack.org/#/c/411800/ ?, it's making some jobs to fail | 13:28 |
| *** armax has quit IRC | 13:28 | |
| *** EricGonczer_ has quit IRC | 13:29 | |
| *** EricGonczer_ has joined #openstack-infra | 13:29 | |
| *** pgadiya has quit IRC | 13:30 | |
| *** amotoki has joined #openstack-infra | 13:32 | |
| *** rlandy has joined #openstack-infra | 13:33 | |
| *** diana_clarke has joined #openstack-infra | 13:36 | |
| *** bhunter71 has quit IRC | 13:42 | |
| *** jpena is now known as jpena|lunch | 13:45 | |
| *** gongysh has joined #openstack-infra | 13:46 | |
| *** amoralej is now known as amoralej|lunch | 13:47 | |
| openstackgerrit | Bernard Cafarelli proposed openstack-infra/project-config: Update networking-sfc grafana dashboard https://review.openstack.org/411821 | 13:47 |
| *** rhallisey has joined #openstack-infra | 13:49 | |
| *** ldnunes has joined #openstack-infra | 13:49 | |
| *** lucas-hungry is now known as lucasagomes | 13:49 | |
| *** lindycoder has joined #openstack-infra | 13:50 | |
| *** chlong has quit IRC | 14:00 | |
| *** ralonsoh_ has joined #openstack-infra | 14:00 | |
| *** lVathan[m] has quit IRC | 14:02 | |
| *** ralonsoh__ has joined #openstack-infra | 14:03 | |
| *** ralonsoh has quit IRC | 14:03 | |
| *** nadya has quit IRC | 14:04 | |
| *** ralonsoh has joined #openstack-infra | 14:04 | |
| *** baoli has quit IRC | 14:04 | |
| *** asselin has joined #openstack-infra | 14:05 | |
| *** ralonsoh_ has quit IRC | 14:06 | |
| *** ralonsoh__ has quit IRC | 14:06 | |
| *** Goneri has joined #openstack-infra | 14:07 | |
| openstackgerrit | Valeriy Ponomaryov proposed openstack-infra/project-config: [Manila] Disable tempest for grenade job https://review.openstack.org/411768 | 14:07 |
| *** asselin_ has quit IRC | 14:07 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Add release notes and an error message for release https://review.openstack.org/411829 | 14:08 |
| *** amotoki has quit IRC | 14:08 | |
| *** gongysh has quit IRC | 14:09 | |
| *** diana_clarke has left #openstack-infra | 14:10 | |
| mordred | yolanda, EmilienM: there was a dracut related issue that was theoretically fixed yesterday. I'm guessing it is now once again not fixed? | 14:10 |
| *** rossella_s has quit IRC | 14:11 | |
| *** rossella_s has joined #openstack-infra | 14:12 | |
| *** lezbar__ has joined #openstack-infra | 14:12 | |
| *** lezbar has quit IRC | 14:13 | |
| *** dave-mccowan has quit IRC | 14:14 | |
| *** baoli has joined #openstack-infra | 14:15 | |
| EmilienM | mordred, derekh: it doesn't seem to be the case | 14:15 |
| *** M-docaedo_vector has joined #openstack-infra | 14:15 | |
| EmilienM | mordred: we filed a bug https://bugs.launchpad.net/tripleo/+bug/1650503 | 14:15 |
| openstack | Launchpad bug 1650503 in tripleo "nodepool slaves failing to boot" [Critical,Triaged] | 14:15 |
| EmilienM | yeah dracut is failing | 14:15 |
| mordred | hrm. it looks like action was taken on this already ... I'd like to wait until pabelanger is up so that I don't accidentally fight a half-in-progress something | 14:16 |
| mordred | he should be up soon | 14:16 |
| derekh | mordred: I believe they deleted image centos-7-0000000134 last night which probably sortyed things out for a bit, but 135 was then uploaded by nodepool so the problem came back | 14:16 |
| mordred | derekh: yah - I'm confused about 135 being uploaded becaues we reverted the revert of the pause -so centos-7 should have been on pause | 14:17 |
| derekh | mordred: at least thats my understanding of what happened, | 14:17 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Trusty/py34 cleanup, part4 https://review.openstack.org/411833 | 14:18 |
| derekh | mordred: ok, affraid I don't have much more insight then that | 14:18 |
| mordred | I'm going to go ahead and delete 135 for now | 14:18 |
| openstackgerrit | Ihar Hrachyshka proposed openstack-infra/project-config: Migrate -platform- devstack jobs to -neutron-full- https://review.openstack.org/411328 | 14:20 |
| *** e0ne has quit IRC | 14:20 | |
| ihrachys | ianw: tell me what you think about ^ | 14:20 |
| *** e0ne has joined #openstack-infra | 14:21 | |
| mordred | derekh, EmilienM: ok- the image list is now showing we're back to 005 for centos-7 - so hopefully that is fixed for now | 14:21 |
| *** trown is now known as trown|brb | 14:22 | |
| derekh | mordred: thanks, I'll let you know to confirm | 14:22 |
| *** fguillot has joined #openstack-infra | 14:23 | |
| EmilienM | mordred: ++ merci beaucoup | 14:23 |
| derekh | mordred: yup, latest instances started has metadata nodepool_build_id ==0 000000005 | 14:24 |
| derekh | and I can ping it so thats an improvement ;-) | 14:24 |
| derekh | mordred: thanks | 14:24 |
| *** amotoki has joined #openstack-infra | 14:25 | |
| yolanda | mordred, so how you will upload a new image to providers now? i miss the image-upload command | 14:25 |
| mordred | woot! | 14:25 |
| *** tlian has joined #openstack-infra | 14:27 | |
| mordred | yolanda: new nodepool builders are pretty aggressive with trying to upload an image if there is a new one and it hasn't been uploaded | 14:27 |
| mordred | yolanda: so the control is really a pause attribute in nodepool.yaml | 14:27 |
| mordred | which tells nodepool to please don't do that righ tnow | 14:28 |
| *** jamesdenton has joined #openstack-infra | 14:28 | |
| mordred | yolanda: http://docs.openstack.org/infra/system-config/nodepool.html#bad-images updated docs on this | 14:29 |
| mordred | yolanda: also - hi! haven't been online same time as you in a while - good to see you | 14:30 |
| *** e0ne has quit IRC | 14:30 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Trusty/py34 cleanup, part4 https://review.openstack.org/411833 | 14:31 |
| *** amotoki has quit IRC | 14:32 | |
| yolanda | ah mordred, so just deleting the bad images, from dib and provider, will cause to upload a new one | 14:32 |
| *** rbuzatu has quit IRC | 14:32 | |
| yolanda | and yes, hi... we haven't talked in a while. I'm also not very present on the channel now | 14:33 |
| Shrews | mordred: hrm, that doc is wrong. the example should use dib-image-delete, not image-delete | 14:34 |
| yolanda | Shrews, shall not be both? dib-image for disk, image-delete for providers? | 14:35 |
| mordred | Shrews: indeed - and I ran dib-image-delete when I deleted | 14:35 |
| mordred | Shrews: you wanna fix or want me to? | 14:36 |
| *** wolverineav has quit IRC | 14:36 | |
| Shrews | yolanda: no. dib-image-delete will delete from providers before deleting the dib image. image-delete only deletes from the provider, keeping the dib | 14:36 |
| Shrews | mordred: i'll grab it | 14:36 |
| yolanda | ah ok | 14:36 |
| *** amoralej|lunch is now known as amoralej | 14:36 | |
| *** wolverineav has joined #openstack-infra | 14:36 | |
| mordred | Shrews: cool. also - if you have time before you roadtrip - I pushed up a release notes patch for shade - would like to cut a release today | 14:36 |
| Shrews | yolanda: and if the image were not paused before image-delete, it would immediately reupload | 14:36 |
| Shrews | mordred: ack | 14:37 |
| yolanda | that pause command is new to me | 14:37 |
| yolanda | btw, can you look at https://review.openstack.org/411800 ? | 14:37 |
| yolanda | it breaks tripleo changes | 14:37 |
| yolanda | mordred ^ | 14:37 |
| mordred | yyup. done | 14:37 |
| *** trown|brb is now known as trown | 14:37 | |
| yolanda | ++ | 14:37 |
| *** dansmith is now known as superdan | 14:38 | |
| mordred | greghaynes, pabelanger: it's possible that newer qemu-img can produce working vhd images and we might be able to get rid of the use of vhd-util | 14:38 |
| *** e0ne has joined #openstack-infra | 14:38 | |
| *** claudiub|2 has quit IRC | 14:38 | |
| *** thorst_ has joined #openstack-infra | 14:41 | |
| *** marst has joined #openstack-infra | 14:41 | |
| openstackgerrit | Andrey Shestakov proposed openstack/diskimage-builder: Fix dhcp-all-interfaces for ubuntu-minimal xenial https://review.openstack.org/407725 | 14:42 |
| *** jeckersb_gone is now known as jeckersb | 14:43 | |
| *** NileshSawant has quit IRC | 14:44 | |
| openstackgerrit | Merged openstack-infra/project-config: Add missing closing " in tripleo job https://review.openstack.org/411800 | 14:44 |
| *** wolverineav has quit IRC | 14:45 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/system-config: Correct bad image delete example https://review.openstack.org/411844 | 14:45 |
| *** tosky has quit IRC | 14:45 | |
| Shrews | mordred: ^^^ | 14:45 |
| mordred | Shrews: ossum. thanks! | 14:46 |
| *** baoli has quit IRC | 14:46 | |
| *** thorst_ has quit IRC | 14:46 | |
| fungi | ttx: oh, that issue with the publish job was probably my fault. i'll respin it again with just "master" (for tags we end up with refs/tags/whatever and i'm too used to requeuing refs for tags that i didn't have an example for a post pipeline ref handy and misremembered the ref name format gerrit uses in those events) | 14:48 |
| fungi | AJaeger: ^ | 14:48 |
| openstackgerrit | Merged openstack-infra/system-config: Document documenting rotated signing keys https://review.openstack.org/409905 | 14:48 |
| AJaeger | fungi: Ah! I wondered already what happend... | 14:48 |
| fungi | when we reenqueue refs we have to supply --ref on the command line | 14:48 |
| fungi | so almost definitely my fault | 14:49 |
| AJaeger | speaking about docs, python-heatclient is not published properly - could some project-config cores please review https://review.openstack.org/411757 ? | 14:49 |
| fungi | oof, lots of neutron unit test failures in the gate suddenly | 14:50 |
| openstackgerrit | Alfredo Moralejo proposed openstack-infra/tripleo-ci: Update packages after configuring openstack repos https://review.openstack.org/411725 | 14:50 |
| *** sreeram_v has quit IRC | 14:51 | |
| fungi | wonder if one of the other neutron changes ahead of those is introducing an incompatibility or maybe a nondeterministic failure | 14:51 |
| openstackgerrit | Steven Hardy proposed openstack-infra/tripleo-ci: Implement major upgrade for Newton to Ocata https://review.openstack.org/404831 | 14:51 |
| AJaeger | clarkb: two more cleanup for trusty/py34: https://review.openstack.org/411833 and https://review.openstack.org/411745 - reviews are welcome | 14:51 |
| openstackgerrit | Steven Hardy proposed openstack-infra/tripleo-ci: Add basic pre/post upgrade sanity checks https://review.openstack.org/411846 | 14:51 |
| *** thorst_ has joined #openstack-infra | 14:51 | |
| *** sreeram_v has joined #openstack-infra | 14:52 | |
| clarkb | AJaeger: noted, though I am in my crawl space clearing an ice jam on my fyrnace condensation line and cyrrenrly have no home heat aside from the small fireplace. tldr dont expect much fron me today | 14:52 |
| *** dave-mccowan has joined #openstack-infra | 14:52 | |
| openstackgerrit | Steven Hardy proposed openstack-infra/tripleo-ci: Add basic pre/post upgrade sanity checks https://review.openstack.org/411846 | 14:52 |
| *** jamesllondonlapt has joined #openstack-infra | 14:53 | |
| fungi | clarkb: oof, that one's never fun | 14:53 |
| AJaeger | fungi, looking at http://logs.openstack.org/17/385617/29/gate/gate-neutron-python35/720352f/console.html and https://review.openstack.org/#/c/410065/ the errors and changes are db related. Do you want to -2 the job to see what happens? | 14:53 |
| *** flepied has quit IRC | 14:53 | |
| ttx | fungi: could you re-run it with ref = master ? it may fail in another way ;) | 14:53 |
| AJaeger | clarkb: take care of heat, that's far more important! Hope you get it fixed soon! | 14:53 |
| fungi | ttx: it's already queued for the past ~7 minutes | 14:53 |
| fungi | should know shortly | 14:53 |
| *** hurgleburgler has joined #openstack-infra | 14:54 | |
| * ttx watches | 14:54 | |
| *** HeOS has quit IRC | 14:54 | |
| AJaeger | ihrachys, njohnston see above comments regarding neutron, please^ | 14:54 |
| *** gomarivera has joined #openstack-infra | 14:55 | |
| AJaeger | bbl | 14:55 |
| openstackgerrit | Merged openstack-infra/infra-specs: Artifact signing is now implemented https://review.openstack.org/409906 | 14:55 |
| fungi | yeah, the random gate failures for neutron's python 2.7, 3.5 and coverage (which probably runs 2.7) unit tests make me worry | 14:55 |
| ihrachys | fungi: it's probably a neutron patch landed | 14:56 |
| ihrachys | fungi: may I ask you to report a bug real quick? I may get to it in 30 mins | 14:57 |
| fungi | ihrachys: i haven't looked into the failure pattern, but there's 7 neutron changes passing in the gate right now and then three more right behind those all failing | 14:58 |
| ihrachys | fungi: ack, will check asap | 14:58 |
| *** mmandel has joined #openstack-infra | 14:58 | |
| *** jpena|lunch is now known as jpena | 14:58 | |
| ihrachys | fungi: I think I know the patch that broke it, so we either revert or make a quick fix | 14:58 |
| *** gomarivera has quit IRC | 14:59 | |
| fungi | so it's likely one of the succeeding ones in the gate introducing the issue (410954, 410953, 410444, 407728, 409806, 410065, 410406) | 14:59 |
| *** coolsvap has quit IRC | 14:59 | |
| fungi | 385617, 392183 and 392183 right behind them are all failing on unit tests (some py27, some py35, some coverage) | 15:00 |
| sdague | fungi / clarkb could we get some reviews on https://review.openstack.org/#/c/409871/ (d-g enable placement api) - and https://review.openstack.org/#/c/386072/22 (d-g roles support in feature matrix) | 15:01 |
| sdague | because those are needed to unlock some things we need to do in nova this cycle | 15:02 |
| *** itisha has quit IRC | 15:02 | |
| *** andymaier has joined #openstack-infra | 15:02 | |
| *** dingyichen has quit IRC | 15:02 | |
| *** hongbin has joined #openstack-infra | 15:03 | |
| *** nadya has joined #openstack-infra | 15:04 | |
| *** john-davidge has quit IRC | 15:05 | |
| *** john-davidge has joined #openstack-infra | 15:06 | |
| *** trown is now known as trown|afk | 15:08 | |
| *** nadya has quit IRC | 15:09 | |
| *** flepied has joined #openstack-infra | 15:09 | |
| fungi | ttx: https://governance.openstack.org/ looks as intended now | 15:10 |
| ttx | fungi: yes, looks like it works. Thanks!* | 15:11 |
| fungi | ttx: my pleasure, and sorry about the earlier confusion | 15:11 |
| ihrachys | fungi: fyi I have a grasp of what happens with the test failure you showed, I will bake a patch in next hour | 15:12 |
| ihrachys | thanks for reaching our | 15:12 |
| ihrachys | *out | 15:12 |
| fungi | ihrachys: you're welcome. it looked worrisome so figured you'd want to know as soon as possible | 15:13 |
| ihrachys | yeah, I see the spike in our grafana it's scary | 15:13 |
| *** bhunter71 has joined #openstack-infra | 15:13 | |
| *** thorst_ has quit IRC | 15:14 | |
| *** jaosorior has quit IRC | 15:15 | |
| *** derekh has quit IRC | 15:15 | |
| *** esberglu has joined #openstack-infra | 15:15 | |
| *** eharney has joined #openstack-infra | 15:15 | |
| *** lVathan[m] has joined #openstack-infra | 15:21 | |
| *** chlong has joined #openstack-infra | 15:21 | |
| *** mtanino has joined #openstack-infra | 15:22 | |
| *** jgriffith is now known as jgriffith_AutoAw | 15:22 | |
| *** jgriffith_AutoAw is now known as jgriffith | 15:22 | |
| *** dtantsur is now known as dtantsur|afk | 15:24 | |
| *** armax has joined #openstack-infra | 15:25 | |
| *** wolverineav has joined #openstack-infra | 15:26 | |
| *** jeckersb is now known as jeckersb_gone | 15:26 | |
| *** zz_dimtruck is now known as dimtruck | 15:26 | |
| *** dave-mccowan has quit IRC | 15:27 | |
| *** rbuzatu has joined #openstack-infra | 15:27 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/zuul: WIP Add per-repo public and private keys https://review.openstack.org/406382 | 15:28 |
| *** wolverineav has quit IRC | 15:29 | |
| *** rbrndt has joined #openstack-infra | 15:30 | |
| *** jeckersb_gone is now known as jeckersb | 15:30 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/zuul: WIP Add per-repo public and private keys https://review.openstack.org/406382 | 15:32 |
| *** rbuzatu has quit IRC | 15:32 | |
| *** mmandel has quit IRC | 15:32 | |
| *** flepied has quit IRC | 15:32 | |
| fungi | yay, my bandersnatch fix merged so hopefully when they get around to releasing 1.12 we can start trusting its exit code in our wrapper script again! https://bitbucket.org/pypa/bandersnatch/pull-requests/23 | 15:33 |
| *** esikachev has quit IRC | 15:33 | |
| *** m1st3rwr0ng has joined #openstack-infra | 15:34 | |
| *** thorst_ has joined #openstack-infra | 15:35 | |
| *** markus_z has quit IRC | 15:35 | |
| pabelanger | morning | 15:37 |
| pabelanger | some excitement | 15:37 |
| pabelanger | mordred: EmilienM: I think centos-7 DIB was deleted last night before https://review.openstack.org/#/c/411613/ landed on disk | 15:38 |
| pabelanger | so a new DIB was build and uploaded | 15:38 |
| *** sreeram_v has quit IRC | 15:38 | |
| *** dave-mccowan has joined #openstack-infra | 15:40 | |
| mordred | pabelanger: ah. that makes sense to me | 15:41 |
| pabelanger | mordred: Shrews: I wonder if we should also not upload if a diskimage is paused, to avoid this issue. | 15:42 |
| pabelanger | currently we don't upload if a provider image is paused | 15:42 |
| *** ihrachys has quit IRC | 15:42 | |
| openstackgerrit | Merged openstack/os-client-config: Magnum's service_type is container_infra https://review.openstack.org/411398 | 15:44 |
| pabelanger | Wow, now ubuntu-precise DIB is no longer building | 15:44 |
| pabelanger | checking why | 15:44 |
| pabelanger | ianw: AJaeger: currently removing fedora-23 DIBs from clouds and disk | 15:45 |
| *** tqtran has joined #openstack-infra | 15:45 | |
| *** thorst_ has quit IRC | 15:46 | |
| pabelanger | mordred: any thoughts on this shade exception: http://paste.openstack.org/show/592651/ | 15:47 |
| pabelanger | AJaeger: https://review.openstack.org/#/c/411621 is ready to be merged now, I just did the clean up in nodepool | 15:49 |
| *** szaher has quit IRC | 15:49 | |
| *** flepied has joined #openstack-infra | 15:49 | |
| mordred | pabelanger: it may just be an expired token or something - swiftclient doens't use Session as of right now, so it's tough to say | 15:49 |
| mordred | pabelanger: the next set of patches I'm working on should make it easier to know what the issue is there | 15:50 |
| *** tqtran has quit IRC | 15:50 | |
| mordred | pabelanger: are you seeing that fequently or just once? | 15:50 |
| EmilienM | we are seeing some multinode job problems with networking, it seems like subnode-1 can't ping subnode-2 | 15:50 |
| EmilienM | http://logs.openstack.org/25/411725/4/check/gate-tripleo-ci-centos-7-nonha-multinode/6260e1c/console.html#_2016-12-16_15_34_22_354472 | 15:50 |
| jeblair | mordred, pabelanger: hrm -- there's no nodepool context for that exception :( | 15:50 |
| EmilienM | it might be something in tripleo scripts though we didn't change anything lately (since everything is broken) | 15:50 |
| jeblair | mordred, pabelanger: how can that be? how do we figure out where in nodepool that happened? | 15:51 |
| EmilienM | http://logs.openstack.org/94/124794/3/check/gate-tripleo-ci-centos-7-nonha-multinode/5f3a474/console.html#_2016-12-16_14_25_52_901666 | 15:52 |
| EmilienM | it's worth mentionning ipv6 jobs fail after 35 seconds of run | 15:52 |
| EmilienM | it might affect all multinode CI jobs | 15:52 |
| mordred | jeblair: my guess is that it's an exception that happened in shade so it logged the exception but it's something that's handled with a retry so the exception did not propagate to nodepool | 15:53 |
| mordred | jeblair: looking at that now | 15:53 |
| pabelanger | mordred: I seen it when I deleted fedora-23 images, I can see how often it happens | 15:53 |
| *** armax has quit IRC | 15:53 | |
| pabelanger | jeblair: sorry, I did dib-image-delete fedora-23-0001, and got that from rax | 15:54 |
| jeblair | pabelanger: sure, but it shows up in the logs a lot from what i see... i agree it *probably* comes from the rax image delete just because it seems to usually show up right after a deleting line -- but that's not enough in a multi-threaded program like nodepool -- we need something attached to that exception that says what thread was running it and against what provider | 15:55 |
| *** maeker has joined #openstack-infra | 15:55 | |
| jeblair | otherwise, as operators, we're left with "one of our 11 providers keeps throwing errors" | 15:55 |
| mordred | jeblair: nope. I'm now more confused as to why we don't get a nodepool exception there | 15:55 |
| jeblair | ya, i'm trying to track it down too | 15:56 |
| EmilienM | the last link I posted seems related to a patch that we merged a few hours in tripleo zuul layout, nevermind. Though we still have networking issue with multinode | 15:56 |
| mordred | jeblair: it's most-likely coming from the delete_object call in delete_image - but that's not a handled/squashed exceptoin | 15:56 |
| pabelanger | jeblair: I too just noticed all the exceptions, I was just tailing the file before. | 15:56 |
| *** john-davidge has quit IRC | 15:56 | |
| *** arxcruz has quit IRC | 15:56 | |
| *** john-davidge has joined #openstack-infra | 15:57 | |
| *** mmandel has joined #openstack-infra | 15:57 | |
| *** thorst_ has joined #openstack-infra | 15:57 | |
| *** esikachev has joined #openstack-infra | 15:57 | |
| *** SergeyLukjanov is now known as SergeyLukjanov__ | 15:57 | |
| *** andreas_s has quit IRC | 15:58 | |
| jeblair | mordred, pabelanger: i think i see a problem | 15:58 |
| openstackgerrit | James E. Blair proposed openstack-infra/nodepool: Fix image delete exception logging https://review.openstack.org/411881 | 16:00 |
| jeblair | mordred, pabelanger: ^ would that have caused us to log the exception without the preceding nodepool context? | 16:00 |
| mordred | jeblair: good catch. yes, I think so. we log the shade exception in shade_inner_exception | 16:01 |
| jeblair | (i'm not super familiar with what happens when you pass the wrong # of args to a logger -- i'm more familiar with interpolating into the string directly) | 16:01 |
| mordred | jeblair: but without that log line, we'd be missing the context line | 16:01 |
| *** john-davidge has quit IRC | 16:01 | |
| jeblair | mordred: aha right | 16:01 |
| *** john-dav_ has joined #openstack-infra | 16:01 | |
| *** thorst_ has quit IRC | 16:01 | |
| mordred | jeblair: I mean, I don't know for sure if that's how that fails - but it failing certainly would lead to us not having the line we don't have :) | 16:01 |
| jeblair | mordred: so really the nodepool exception handler is doing nothing, it's all from the inner handler | 16:01 |
| mordred | yah | 16:01 |
| jeblair | ok | 16:02 |
| jeblair | would be nice for python logging to mention something :) | 16:02 |
| jeblair | maybe at DERP level or something | 16:02 |
| *** esikachev has quit IRC | 16:02 | |
| mordred | jeblair: if that is a swift auth exceptoin that just needs a reauth - I'd love to catch and reauth/retry | 16:02 |
| *** wolverineav has joined #openstack-infra | 16:02 | |
| *** Guest31304 has quit IRC | 16:02 | |
| *** sdake has joined #openstack-infra | 16:03 | |
| mordred | jeblair: EXCEPT - I'm literally about to replace that function with one that uses the keystoneauth Session - so I do not expect to have such an issue shortly | 16:03 |
| mordred | which is my way of saying, I think shade should handle that error more gracefully if it's the error I think it is | 16:03 |
| pabelanger | jeblair: +3 | 16:03 |
| *** john-dav_ has quit IRC | 16:05 | |
| *** wolverineav has quit IRC | 16:06 | |
| clarkb | I have heat againanddidnt have to pay outrageous prices (fixed it myself) \o/ | 16:08 |
| mordred | clarkb: \o/ | 16:08 |
| clarkb | but now etired | 16:08 |
| mordred | clarkb: fix with beer? | 16:08 |
| pabelanger | and here is the reason for our ubuntu-precise DIB failures today: http://paste.openstack.org/show/592655/ | 16:08 |
| pabelanger | exception in get-pip.py | 16:09 |
| fungi | clarkb: kettle of hot water and a funnel? | 16:09 |
| *** baoli has joined #openstack-infra | 16:10 | |
| clarkb | fungi: I actually pulled the line in and got ice plug warm enough by hand to use mouth pumo to remove it | 16:10 |
| clarkb | then completely drained system | 16:10 |
| fungi | awesome | 16:11 |
| fungi | i've used the hot water trick before, but in that case i had fairly easy access to the top of the condensate drain | 16:11 |
| *** esikachev has joined #openstack-infra | 16:12 | |
| *** aidin has joined #openstack-infra | 16:13 | |
| *** maeker has quit IRC | 16:16 | |
| *** maeker has joined #openstack-infra | 16:16 | |
| *** aidin has quit IRC | 16:18 | |
| openstackgerrit | Sergey Slipushenko proposed openstack-infra/project-config: RefStack: add coverage job https://review.openstack.org/410283 | 16:20 |
| *** martinkopec has quit IRC | 16:21 | |
| openstackgerrit | Paul Belanger proposed openstack/diskimage-builder: Revert "Fix pip-and-virtualenv to work with python3" https://review.openstack.org/411892 | 16:22 |
| pabelanger | clarkb: ianw: greghaynes: ^ looks to be breaking ubuntu-precise builds, we are forcing DIB_PYTHON_VERSION=3 | 16:23 |
| greghaynes | pabelanger: hrm? | 16:23 |
| *** rcernin has quit IRC | 16:23 | |
| AJaeger | pabelanger: +2A on the final f23 change | 16:23 |
| pabelanger | greghaynes: see http://paste.openstack.org/show/592655/ for example | 16:23 |
| *** andymaier has quit IRC | 16:25 | |
| pabelanger | greghaynes: 50-dib-python-version is missing precise support | 16:25 |
| *** pcaruana has quit IRC | 16:25 | |
| EmilienM | https://bugs.launchpad.net/tripleo/+bug/1650612 | 16:25 |
| openstack | Launchpad bug 1650612 in tripleo "nodepool subnodes can't ping each others" [Critical,Triaged] | 16:25 |
| greghaynes | ah that should be easy to fix | 16:25 |
| EmilienM | I would like to request an access on a multinode node by ssh or something, we can't debug the bug without that access | 16:26 |
| fungi | pabelanger: oh, yes there was no python 3 installed by default on precise, and if you installed the provided package from the distro it was 3.2.x | 16:26 |
| jeblair | EmilienM: i'll help | 16:26 |
| EmilienM | jeblair: thanks | 16:27 |
| jeblair | EmilienM: are any jobs running on the kind of node you want right now? | 16:27 |
| *** mhickey has quit IRC | 16:27 | |
| EmilienM | jeblair: looking | 16:27 |
| openstackgerrit | Paul Belanger proposed openstack/diskimage-builder: Add ubuntu-precise support to dib-python https://review.openstack.org/411898 | 16:27 |
| pabelanger | fungi: clarkb: greghaynes: ^ should be our fix | 16:27 |
| EmilienM | amoralej: look 411846 in zuul, it passed multijob | 16:28 |
| EmilienM | multinode job* | 16:28 |
| jeblair | EmilienM: so the issue is that gate-tripleo-ci-centos7-nonha-multinode sometimes but does not always fail? | 16:29 |
| amoralej | i'm checking http://logs.openstack.org/09/409809/1/check/gate-tripleo-ci-centos-7-nonha-multinode/fb7d781/console.html that passed EmilienM | 16:29 |
| EmilienM | jeblair: yes, and it's very new | 16:29 |
| *** chlong has quit IRC | 16:29 | |
| jeblair | EmilienM: do you have a logstash query? maybe it correlates to a specific cloud provider? | 16:30 |
| openstackgerrit | Andrea Frittoli proposed openstack-infra/project-config: Align the dsvm-neutron-full-ssh job https://review.openstack.org/411899 | 16:30 |
| EmilienM | amoralej: https://review.openstack.org/#/c/411340/ | 16:30 |
| greghaynes | pabelanger: LGTM | 16:30 |
| EmilienM | this patch has all jobs passing :/ | 16:30 |
| EmilienM | jeblair: let me find a logstash query | 16:30 |
| openstackgerrit | Merged openstack-infra/project-config: Remove Fedora 23 from nodepool https://review.openstack.org/411621 | 16:31 |
| jeblair | EmilienM: maybe: message:"still failed after restarting openvswitch" | 16:31 |
| andreaf | sdague: tiny project-config patch https://review.openstack.org/411899 to enable TLS on the -ssh job | 16:31 |
| *** vaidy has quit IRC | 16:31 | |
| *** adrian_otto has joined #openstack-infra | 16:32 | |
| *** isviridov_away has quit IRC | 16:32 | |
| AJaeger | andreaf: that is part of several repos, I sugest you ask nova and neutron PTLs or liasons to review this. | 16:32 |
| jeblair | EmilienM: i see 3 of those in the last 24h; 2 on iad and one os osic | 16:32 |
| pabelanger | ianw: clarkb: greghaynes: fungi: I think what I would like to do, is update our nodepool dsvm job, and increase the amount of images we attempt to launch. Logic being, if we are actually launching these images in the gate, we'll be catching more of these issues (dib-python, dracut) before code is merged. I believe we'll need to expose a way for the -dsvm-nodepool-src-diskimage-builder job to use | 16:32 |
| pabelanger | another nodepool.yaml file with more diskimages, so we don't run all the image builds on nodepool changes. Today we only have a ubuntu-xenial minimal | 16:32 |
| AJaeger | andreaf: I'd like to see some buyin on it | 16:32 |
| EmilienM | jeblair: rax & cloud1 but only 3 runs had this thing | 16:32 |
| EmilienM | jeblair: it sounds like a transient issue | 16:33 |
| greghaynes | pabelanger: that wouldnt hurt, but we also have dib functional testing which would be a better way for catching this - check out the test-elements dir in the ubuntu element | 16:33 |
| openstackgerrit | Merged openstack-infra/nodepool: Fix image delete exception logging https://review.openstack.org/411881 | 16:33 |
| greghaynes | is there an infra mirror for precise? | 16:33 |
| AJaeger | andreaf: sorry, that's a new one - I mixed up changes. Still reviews by others would be nice | 16:34 |
| jeblair | greghaynes: negative http://mirror.dfw.rax.openstack.org/ubuntu/dists/ | 16:34 |
| fungi | greghaynes: nope http://mirror.iad.rax.openstack.org/ubuntu/dists/ | 16:34 |
| * fungi lost that shootout | 16:34 | |
| greghaynes | ah, so thats not a deal breaker but definitely is a downside to adding precise as something we gate on | 16:34 |
| *** claudiub|2 has joined #openstack-infra | 16:35 | |
| EmilienM | amoralej, fungi, bnemec: though the ovs thing is only for https://review.openstack.org/#/c/411725/ if you look at logstash | 16:35 |
| EmilienM | http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22still%20failed%20after%20restarting%20openvswitch%5C%22 | 16:35 |
| fungi | greghaynes: we don't want to support precise for much longer and only have a handful of servers still relying on it, so probably safe to have as a nonvoting job | 16:35 |
| EmilienM | fungi: sorry, wrong ping | 16:35 |
| EmilienM | amoralej, jeblair, bnemec: it might be related to the patch itself, like you said Ben (we're updating packages) | 16:35 |
| amoralej | EmilienM, maybe something is being broken by the update | 16:36 |
| bnemec | Yeah, then I stand by my theory that 7.3 broke us again. :-) | 16:36 |
| amoralej | i can try a more conservative alternative | 16:36 |
| jeblair | EmilienM: do you still need something from me, or should i stand down? | 16:36 |
| bnemec | Well, at some point we've got to get it working with the latest bits. | 16:36 |
| EmilienM | jeblair: sorry for noise, thanks it's good | 16:37 |
| EmilienM | jeblair: I thought it would affect all infra, hopefully not. | 16:37 |
| jeblair | EmilienM: ok. we have an auto-hold feature, so if you don't make progress, i can ask nodepool to hold the next node(s) that fail(s) gate-tripleo-ci-centos7-nonha-multinode | 16:37 |
| jeblair | EmilienM: but i'll wait to hear from you before doing that | 16:38 |
| EmilienM | jeblair: oh that would be awesome | 16:38 |
| EmilienM | jeblair: we can hold jobs for 411725 | 16:38 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config: move release step for updating constraints to after the PyPI upload https://review.openstack.org/411495 | 16:38 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config: add update_constraints.sh for use by new release job https://review.openstack.org/411907 | 16:38 |
| *** tjones has joined #openstack-infra | 16:38 | |
| EmilienM | jeblair: I can run a recheck so it triggers multinode jobs and you can hold the node | 16:38 |
| EmilienM | jeblair: let me know when I can run it | 16:38 |
| openstackgerrit | Matthieu Huin proposed openstack/python-jenkins: url-proof scripts passed to run_script https://review.openstack.org/407479 | 16:39 |
| jeblair | EmilienM: it's set up now | 16:40 |
| *** baoli has quit IRC | 16:40 | |
| EmilienM | let's go | 16:40 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci: Update packages after configuring openstack repos https://review.openstack.org/411725 | 16:40 |
| greghaynes | pabelanger: If you want to do the nonvoting job thing we have a list of test-elements which only run in our nonvoting job - check out tests/run_functests.sh and the projects.yaml for dib (you explicitly list the test there) | 16:40 |
| EmilienM | gate-tripleo-ci-centos-7-nonha-multinode for 411725 is queued | 16:41 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Add release notes and an error message for release https://review.openstack.org/411829 | 16:42 |
| EmilienM | jeblair: it's running on an ipv6. I guess I need an ipv6 to ssh? | 16:43 |
| openstackgerrit | Paul Belanger proposed openstack/diskimage-builder: Increase func testing for ubuntu-minimal element https://review.openstack.org/411910 | 16:43 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Update test of object metadata to mock requests https://review.openstack.org/411911 | 16:44 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Change get_object_metadata to use REST https://review.openstack.org/411912 | 16:44 |
| pabelanger | greghaynes: ya, looking now. I'll add some more coverage, but will also look into nodepool changes, since we can actually boot images in that job | 16:44 |
| *** m1st3rwr0ng has quit IRC | 16:44 | |
| *** baoli has joined #openstack-infra | 16:44 | |
| jeblair | mordred: does osc have deps on all the clients? | 16:45 |
| *** baoli has quit IRC | 16:45 | |
| jeblair | mordred: see comment on 411829 | 16:46 |
| jeblair | pabelanger, mordred: 410853 and 411360 lgtm -- do one or both of you want to review/land? maybe if #openstack-release is chill we can do that today? | 16:47 |
| pabelanger | looking | 16:47 |
| jeblair | "do that" == "restart" | 16:47 |
| *** rhallisey has quit IRC | 16:47 | |
| pabelanger | jeblair: Ya, I'm able to support this effort today | 16:48 |
| pabelanger | will review | 16:48 |
| *** savihou has quit IRC | 16:49 | |
| *** achevychalov has quit IRC | 16:49 | |
| *** baoli has joined #openstack-infra | 16:50 | |
| pabelanger | jeblair: Shrews: left a question on 411360 | 16:51 |
| *** vaidy has joined #openstack-infra | 16:51 | |
| *** matrohon has quit IRC | 16:51 | |
| *** isviridov_away has joined #openstack-infra | 16:52 | |
| fungi | ouch, that glance liberty tarball build issue highlights a fairly serious blind-spot in our ci model | 16:53 |
| mordred | jeblair: not on all of them - but yes all of them we depend on | 16:53 |
| *** chlong has joined #openstack-infra | 16:53 | |
| *** ralonsoh has quit IRC | 16:53 | |
| mordred | jeblair: yes - I like the warning idea | 16:54 |
| fungi | since tags are branchless (at least from a gerrit trigger perspective), we try to run release jobs for stable/liberty and stable/mitaka tags on xenial rather than precise | 16:54 |
| jeblair | mordred: cool, and cool | 16:54 |
| *** rpodolyaka is now known as rpodolyaka_afk | 16:54 | |
| fungi | i can't think of an easy way to solve that dilemma | 16:54 |
| jeblair | fungi: no, the closest thing to what we want intellectually is "this tag is on a commit that is only in this branch" | 16:55 |
| jeblair | which is far from guaranteed :( | 16:55 |
| *** maeker has quit IRC | 16:55 | |
| fungi | right | 16:55 |
| *** soliosg has joined #openstack-infra | 16:55 | |
| fungi | or a way of being able to declare in-repo what node to use (zuul v3!) | 16:55 |
| jeblair | however, maybe the commit searching is what we should implement | 16:56 |
| *** gomarivera has joined #openstack-infra | 16:56 | |
| clarkb | fungi: glance liberty is eol though you mean mitaka? | 16:56 |
| jeblair | we could have a tag ref match any branch that it's on.... and then push the complexity on to the case where the commit is on more than one branch | 16:56 |
| fungi | if the tag was on a stable/liberty commit and the zuul configuration in that repo's stable/liberty branch said to run tarball jobs on ubuntu-trusty, zuul v3 would presumably just dtrt | 16:56 |
| soliosg | can the gerrit username (https://review.openstack.org/) be updated? without creating a hassle for sysadmins; currently I don't have remote git branches under that name | 16:57 |
| jeblair | fungi: yeah, but i think i was hoping that some of the more basic jobs would not be in repo | 16:57 |
| fungi | true | 16:57 |
| fungi | tarball jobs are otherwise a clear candidate for being in our central configuration corpus | 16:57 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Add release notes and an error message for release https://review.openstack.org/411829 | 16:58 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Update test of object metadata to mock requests https://review.openstack.org/411911 | 16:58 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Change get_object_metadata to use REST https://review.openstack.org/411912 | 16:58 |
| mordred | jeblair: ok. updated the stuff in 411829 - also split the other two out to not depend on it | 16:58 |
| fungi | soliosg: there's no change username feature implemented in gerrit, even for sysadmins. it requires editing a table in the backend database for gerrit | 16:58 |
| *** hashar has quit IRC | 16:59 | |
| fungi | soliosg: you can set your gerrit username in ~/.ssh/config however, and then git-review should just use that automatically | 16:59 |
| *** Swami has joined #openstack-infra | 16:59 | |
| clarkb | or in the git config for gitreview | 16:59 |
| soliosg | fungi: understand, noticed that even gerrit set-account allows to change some settings but not the username | 17:00 |
| fungi | true, it can be set in a gitreview section of ~/.git/config or if you use git-review with multiple gerrits you can set it per-host with a "Host review.openstack.org" section in ~/.ssh/config | 17:00 |
| *** gomarivera has quit IRC | 17:01 | |
| *** kjorgensen has joined #openstack-infra | 17:01 | |
| jeblair | pabelanger: re your comment -- i think so, mostly for consistency, but i think what's there is still safe so we can do it in a followup. | 17:01 |
| *** kjorgensen has quit IRC | 17:01 | |
| soliosg | fungi: used the company's active directory username, but would prefer to match the irc handle and launchpad users | 17:02 |
| *** knangia has joined #openstack-infra | 17:02 | |
| pabelanger | jeblair: sure, WFM | 17:02 |
| *** kjorgens_ has joined #openstack-infra | 17:03 | |
| pabelanger | +2 | 17:03 |
| fungi | soliosg: the gerrit username is also not displayed directly to anyone besides you (though it can be used as a shortcut in queries/searches in gerrit if people happen to know/guess it) | 17:03 |
| *** rcernin has joined #openstack-infra | 17:04 | |
| soliosg | fungi: thanks, then no need to change, can live with that | 17:05 |
| openstackgerrit | Paul Belanger proposed openstack-infra/nodepool: Rename ubuntu-dib to ubuntu-trusty https://review.openstack.org/411920 | 17:05 |
| openstackgerrit | Paul Belanger proposed openstack-infra/nodepool: Start templating nodepool.yaml https://review.openstack.org/411921 | 17:05 |
| openstackgerrit | Merged openstack-infra/devstack-gate: Introduce roles into the feature matrix https://review.openstack.org/386072 | 17:06 |
| EmilienM | we (tripleo) also noticed that we have different versions of images in different jobs (centos7.2 in some, 7.3 in others) | 17:06 |
| *** tjones has left #openstack-infra | 17:06 | |
| * EmilienM looking for details with cloud providers | 17:06 | |
| pabelanger | EmilienM: example? | 17:06 |
| jeblair | pabelanger: do we have a graph of nodes by state + provider? like "graph of ready nodes by provider"? so we can see how much of each provider we're using? | 17:06 |
| EmilienM | pabelanger: yeah I'm looking at it | 17:06 |
| *** gomarivera has joined #openstack-infra | 17:07 | |
| fungi | centos 7.3 introduced a major version bump on libvirt out of nowhere? that's rather unexpected | 17:08 |
| *** adrian_otto has quit IRC | 17:08 | |
| pabelanger | jeblair: we should graph that info at the top of each providers dashboard | 17:08 |
| EmilienM | bnemec, amoralej: for the centos version: it sounds like nodepool has 7.3 but rh1 7.2 if I look at ovb jobs | 17:08 |
| *** savihou has joined #openstack-infra | 17:08 | |
| jeblair | pabelanger: well, i was looking for one graph to illustrate the diversity of providers... 'ready node launch attempts' is probably closest; i'll see about adding one | 17:08 |
| amoralej | it seems osci is 7.2 also | 17:09 |
| pabelanger | jeblair: agreed, it would be nice to have it as a single graph | 17:09 |
| amoralej | http://logs.openstack.org/40/411340/8/check/gate-tripleo-ci-centos-7-nonha-multinode/966cbe7/console.html#_2016-12-16_15_13_13_099677 | 17:09 |
| fungi | EmilienM: amoralej: yeah, while we try to upload new images to all our providers every day, it's often the case that an updated image we build only makes it onto some providers and gets caught by transient upload errors in others | 17:09 |
| EmilienM | centos-7-2-node-osic | 17:09 |
| fungi | so we don't guarantee that the image in all providers we're using is equally fresh | 17:10 |
| amoralej | thanks fungi, that explain some weird behavior we were seeing | 17:10 |
| pabelanger | EmilienM: keep in mind, we don't manage the ovb image, in fact pretty sure it is using the centos element, where we use centos-minimal | 17:10 |
| EmilienM | fungi: ok. In this case, could we check all providers and try to sync them? | 17:10 |
| jeblair | pabelanger: ha, the nodepool daemon handling the webapp is still running the old version; i'll see about restarting it | 17:11 |
| fungi | EmilienM: well, nodepool already does try to do that (though it doesn't upload them all simultaneously either). odds are we're failing uploads somewhere | 17:11 |
| pabelanger | EmilienM: actually, I would need to check. You are using heat to launch ovb, where does heat get that image from? | 17:11 |
| pabelanger | amoralej: do you have a log showing 7.3? | 17:12 |
| jeblair | EmilienM, amoralej: and *now* http://nodepool.openstack.org/image-list is up to date | 17:13 |
| fungi | looks like all our centos-7 images are currently from ~4.5 days ago, so should be consistent now? | 17:13 |
| pabelanger | yes | 17:13 |
| *** jpena is now known as jpena|off | 17:13 | |
| *** jgriffith is now known as jgriffith_AutoAw | 17:13 | |
| clarkb | because 7.3 broke dib | 17:14 |
| fungi | i'm guessing the jobs which ran against 7.3 ran on nodes booted from newer images we've since deleted? | 17:14 |
| Shrews | pabelanger: replied to your comment. tl;dr, don't have to do it in nodepool.py b/c it's one thread | 17:14 |
| bnemec | pabelanger: We don't use heat to launch the undercloud node, which is what we're looking at right now. | 17:14 |
| *** panda is now known as panda|off | 17:14 | |
| *** jgriffith_AutoAw is now known as jgriffith | 17:14 | |
| EmilienM | fungi, pabelanger: ok, we'll try again 'recheck' | 17:14 |
| bnemec | That should be a stock nodepool image. | 17:14 |
| amoralej | pabelanger http://logs.openstack.org/09/409809/1/gate/gate-tripleo-ci-centos-7-nonha-multinode/43ed9c4/console.html#_2016-12-16_08_20_53_132555 | 17:14 |
| bnemec | That log is from earlier this morning, maybe before the newer image had been deleted? | 17:14 |
| amoralej | 3.10.0-514.2.2.el7.x86_64 | 17:14 |
| mordred | wow - centos 7.3 increased the _major_ version of libvirt? | 17:15 |
| fungi | mordred: yep, to 2.0.0 | 17:15 |
| amoralej | 2.0, yes | 17:15 |
| mordred | wow | 17:15 |
| mordred | like | 17:15 |
| bnemec | To a .0 release, no less | 17:15 |
| EmilienM | you can say it, it's Friday, please go ahead | 17:15 |
| pabelanger | amoralej: http://logs.openstack.org/09/409809/1/gate/gate-tripleo-ci-centos-7-nonha-multinode/43ed9c4/console.html#_2016-12-16_08_20_53_131683 | 17:15 |
| pabelanger | that is a new image | 17:15 |
| EmilienM | bnemec: I didn't touch 311514 it's moving to gate anyway | 17:15 |
| jeblair | Shrews: well, actually, it's used by about 1000 threads. but it's still safe for now because there's no locking involved (it's read only for the moment) | 17:15 |
| pabelanger | for some reason, 7.3 DIB worked | 17:15 |
| fungi | mordred: for support mordred http://lists.openstack.org/pipermail/openstack-dev/2016-December/109080.html | 17:15 |
| amoralej | it shouldn't pabelanger? | 17:16 |
| fungi | mordred: er, no idea where the "for support mordred" came from | 17:16 |
| fungi | i fully support mordred though | 17:16 |
| jeblair | Shrews: (and yeah, a reconfiguration might throw some errors right now, and using resetHosts should alleviate that) | 17:16 |
| bnemec | EmilienM: Not yet. The updates-nv job has to finish first. | 17:16 |
| mordred | fungi: yay! I like being supported | 17:16 |
| pabelanger | amoralej: well, when I tested 7.3 DIB yesterday it did work | 17:16 |
| pabelanger | but after we built an uploaded, it failed again | 17:17 |
| pabelanger | so we must be hitting some race or something | 17:17 |
| pabelanger | either way, we've rolled back to 7.2 images, moving forward they'll be the same across all clouds now | 17:17 |
| Shrews | jeblair: 1000 threads? wha? | 17:17 |
| pabelanger | bnemec: right, thanks | 17:17 |
| jeblair | Shrews: yeah, in nodepoold each node launched gets its own thread | 17:18 |
| *** yamahata has joined #openstack-infra | 17:18 | |
| amoralej | ok pabelanger, when do you expect to have 7.3 images distributed again? | 17:18 |
| pabelanger | amoralej: like next week at this point, we need to confirm https://review.openstack.org/#/c/411540/ is the fix | 17:19 |
| *** saibarspeis has quit IRC | 17:19 | |
| jeblair | launching a node in openstack is *hard* | 17:19 |
| *** maeker has joined #openstack-infra | 17:19 | |
| amoralej | that gives us some time to get some fixes to get ready for it | 17:19 |
| *** saibarspeis has joined #openstack-infra | 17:19 | |
| *** arxcruz has joined #openstack-infra | 17:20 | |
| *** wolverineav has joined #openstack-infra | 17:20 | |
| *** psachin has quit IRC | 17:20 | |
| *** tqtran has joined #openstack-infra | 17:20 | |
| *** rbuzatu has joined #openstack-infra | 17:20 | |
| amoralej | pabelanger, where can i see what are the current distributed and logs of how they were built? | 17:20 |
| jeblair | amoralej: for one, that link i pointed you at earlier: http://nodepool.openstack.org/image-list | 17:21 |
| *** jordanP has quit IRC | 17:22 | |
| jeblair | amoralej: and for the other: http://nb01.openstack.org/ and http://nb02.openstack.org/ | 17:22 |
| pabelanger | yup | 17:22 |
| amoralej | thx jeblair, pabelanger | 17:22 |
| jeblair | amoralej: oh, you'll also want: http://nodepool.openstack.org/dib-image-list | 17:22 |
| *** maeker has quit IRC | 17:23 | |
| jeblair | amoralej: with image image-list, you can see what build of what image is current in each provider; with dib-image-list you can see where each build was built (nb01 or nb02); then on that host, i guess you'll have to look for image name and timestamp to find the right log, sorry about that last bit. :( | 17:23 |
| *** maeker_ has joined #openstack-infra | 17:24 | |
| amoralej | good enough jeblair | 17:24 |
| *** wolverineav has quit IRC | 17:24 | |
| openstackgerrit | Merged openstack/diskimage-builder: Add ubuntu-precise support to dib-python https://review.openstack.org/411898 | 17:27 |
| *** sdake has quit IRC | 17:28 | |
| *** maeker_ has quit IRC | 17:29 | |
| *** armax has joined #openstack-infra | 17:29 | |
| *** esberglu has quit IRC | 17:30 | |
| *** maeker has joined #openstack-infra | 17:30 | |
| *** esberglu has joined #openstack-infra | 17:30 | |
| openstackgerrit | Paul Belanger proposed openstack/diskimage-builder: Start func testing on centos-minimal again https://review.openstack.org/411926 | 17:31 |
| *** Guest36050 has quit IRC | 17:33 | |
| *** maeker has quit IRC | 17:33 | |
| *** esberglu has quit IRC | 17:35 | |
| openstackgerrit | Merged openstack-infra/nodepool: Remove ZK lock variables https://review.openstack.org/410853 | 17:35 |
| EmilienM | clarkb: https://review.openstack.org/410823 is still triggering tripleo multinode jobs even with https://review.openstack.org/#/c/410863/1/zuul/layout.yaml -- I'm wondering if we need to also add the branch in line 1589 | 17:36 |
| *** wolverineav has joined #openstack-infra | 17:36 | |
| *** savihou has quit IRC | 17:37 | |
| *** adrian_otto has joined #openstack-infra | 17:38 | |
| EmilienM | clarkb: nevermind | 17:38 |
| openstackgerrit | Paul Belanger proposed openstack-infra/nodepool: Start templating nodepool.yaml https://review.openstack.org/411921 | 17:39 |
| openstackgerrit | Merged openstack-infra/nodepool: Share single ZooKeeper object among workers https://review.openstack.org/411360 | 17:39 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: Do not run tripleo jobs for puppet-ceph stable/hammer https://review.openstack.org/411928 | 17:40 |
| *** wolverineav has quit IRC | 17:40 | |
| EmilienM | clarkb: that's what I missed ^ | 17:40 |
| openstackgerrit | Paul Belanger proposed openstack-infra/nodepool: WIP support more images for dsvm job https://review.openstack.org/411929 | 17:42 |
| *** vhosakot has joined #openstack-infra | 17:44 | |
| *** xyang1 has quit IRC | 17:44 | |
| *** maeker has joined #openstack-infra | 17:45 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/nodepool: WIP support more images for dsvm jobs https://review.openstack.org/411929 | 17:47 |
| *** rossella_s has quit IRC | 17:47 | |
| *** maeker has quit IRC | 17:48 | |
| *** wolverineav has joined #openstack-infra | 17:48 | |
| *** wolverin_ has joined #openstack-infra | 17:49 | |
| openstackgerrit | Paul Belanger proposed openstack/diskimage-builder: Start func testing on centos-minimal again https://review.openstack.org/411926 | 17:49 |
| *** eharney has quit IRC | 17:51 | |
| *** wolverineav has quit IRC | 17:52 | |
| *** wolverin_ has quit IRC | 17:54 | |
| pabelanger | greghaynes: do you mind cutting another release of diskimage-builder for the precise fix | 17:54 |
| *** tphummel has joined #openstack-infra | 17:56 | |
| *** rhallisey has joined #openstack-infra | 17:57 | |
| *** maeker has joined #openstack-infra | 17:59 | |
| openstackgerrit | Jay Faulkner proposed openstack-infra/devstack-gate: Update documentation for ironic-bm-logs https://review.openstack.org/411938 | 17:59 |
| *** wolverineav has joined #openstack-infra | 17:59 | |
| *** rkukura has joined #openstack-infra | 18:00 | |
| *** asettle has quit IRC | 18:01 | |
| *** asettle has joined #openstack-infra | 18:01 | |
| *** dprince has quit IRC | 18:02 | |
| *** kgiusti has joined #openstack-infra | 18:03 | |
| *** wolverineav has quit IRC | 18:04 | |
| *** asettle has quit IRC | 18:06 | |
| *** eharney has joined #openstack-infra | 18:07 | |
| *** xarses has joined #openstack-infra | 18:08 | |
| openstackgerrit | Derek Schultz proposed openstack-infra/project-config: add new-project Picasso https://review.openstack.org/411940 | 18:08 |
| *** rbuzatu has quit IRC | 18:12 | |
| *** rkukura has quit IRC | 18:14 | |
| *** rbuzatu has joined #openstack-infra | 18:15 | |
| openstackgerrit | Paul Belanger proposed openstack/diskimage-builder: Start func testing on centos-minimal again https://review.openstack.org/411926 | 18:16 |
| openstackgerrit | Derek Schultz proposed openstack-infra/project-config: minimal zuul config: merge-check and noop-jobs https://review.openstack.org/411944 | 18:17 |
| *** flepied has quit IRC | 18:18 | |
| EmilienM | hey infra/project-config, can someone review this change? https://review.openstack.org/#/c/411928/ -- so we'll be able to unblock puppet-ceph stable/hammer CI and therefore Puppet OpenStack stable/mitaka CI | 18:21 |
| *** lucasagomes is now known as lucas-afk | 18:23 | |
| *** ociuhandu has quit IRC | 18:23 | |
| *** ianychoi has quit IRC | 18:30 | |
| *** adrian_otto has quit IRC | 18:30 | |
| *** trown|afk is now known as trown | 18:31 | |
| AJaeger | EmilienM: on it... | 18:31 |
| *** asettle has joined #openstack-infra | 18:31 | |
| AJaeger | EmilienM: one +2 | 18:31 |
| EmilienM | AJaeger: danke | 18:32 |
| *** asettle has quit IRC | 18:32 | |
| *** jgriffith is now known as jgriffith_AutoAw | 18:33 | |
| *** sdake has joined #openstack-infra | 18:33 | |
| *** esberglu has joined #openstack-infra | 18:33 | |
| *** vhosakot has quit IRC | 18:34 | |
| *** nadya has joined #openstack-infra | 18:35 | |
| openstackgerrit | Derek Schultz proposed openstack-infra/project-config: add new-project Picasso https://review.openstack.org/411940 | 18:36 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Update test of object metadata to mock requests https://review.openstack.org/411911 | 18:37 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Change get_object_metadata to use REST https://review.openstack.org/411912 | 18:37 |
| mordred | ok. those should work now | 18:37 |
| mordred | jeblair, pabelanger: those ^^ should have an effect on the specific error we are seeing in our logs | 18:37 |
| mordred | jeblair, pabelanger: of course, the next two patches (replace the other operations) will likely be needed to make it go away completely if the cause is what I think it is | 18:38 |
| mordred | but if the cause _is_ what I think it is, those patches will cause the error message to shift to not be on the get_object_metadata call but on a different swift call | 18:38 |
| mordred | although I expect us to get roughly the same error | 18:38 |
| EmilienM | mordred: do you mind review https://review.openstack.org/#/c/411928/ please? (thx a lot) | 18:39 |
| openstackgerrit | Paul Belanger proposed openstack-infra/nodepool: WIP support more images for dsvm jobs https://review.openstack.org/411929 | 18:40 |
| *** nadya has quit IRC | 18:40 | |
| mordred | EmilienM: done | 18:40 |
| jeblair | mordred: does _object_store_client.head do magic taskmanager stuff? | 18:40 |
| EmilienM | mordred: merci beaucoup | 18:40 |
| *** dprince has joined #openstack-infra | 18:41 | |
| mordred | jeblair: yes. all of the direct REST stuff magically run themselves through taskmanager | 18:41 |
| jeblair | mordred: cool | 18:41 |
| mordred | jeblair: shade/_adapter.py is where you want to look to see that, if you want to see that | 18:41 |
| jeblair | mordred: yeah, i know we talked about it | 18:42 |
| jeblair | don't recall seeing it yet | 18:42 |
| mordred | well - enjoy it in this patch! :) | 18:42 |
| jeblair | i am! | 18:42 |
| *** vhosakot has joined #openstack-infra | 18:43 | |
| *** yamahata has quit IRC | 18:44 | |
| openstackgerrit | Merged openstack-infra/project-config: Do not run tripleo jobs for puppet-ceph stable/hammer https://review.openstack.org/411928 | 18:45 |
| openstackgerrit | Jim Rollenhagen proposed openstack-infra/project-config: Use ironic-vars-early in grenade job https://review.openstack.org/411949 | 18:46 |
| jroll | ^ stupid patch, easy review, if someone would be so kind, this is (again) the last patch in the way of moving ironic code out of tempest's tree | 18:47 |
| openstackgerrit | Derek Schultz proposed openstack-infra/project-config: add new-project Picasso https://review.openstack.org/411940 | 18:47 |
| *** maeker has quit IRC | 18:48 | |
| openstackgerrit | Derek Schultz proposed openstack-infra/project-config: add new-project Picasso https://review.openstack.org/411940 | 18:48 |
| *** maeker has joined #openstack-infra | 18:50 | |
| *** itisha has joined #openstack-infra | 18:52 | |
| *** maeker has quit IRC | 18:52 | |
| *** rbrndt has quit IRC | 18:52 | |
| *** maeker_ has joined #openstack-infra | 18:52 | |
| *** baoli has quit IRC | 18:52 | |
| *** john-davidge has joined #openstack-infra | 18:52 | |
| amrith | hi jeremy, could I please ssh into 198.72.124.177 to take a look at what it is doing | 18:53 |
| amrith | it is runnign a CI job for change 411848 | 18:53 |
| fungi | amrith: i've held that node so nodepool won't delete it, just a sec and i'll get you set up with access | 18:53 |
| amrith | and I'm wondering what it is doing right now | 18:53 |
| fungi | amrith: where can i find a public ssh key for you? | 18:54 |
| *** saibarspeis has quit IRC | 18:54 | |
| *** sdake has quit IRC | 18:54 | |
| *** flepied has joined #openstack-infra | 18:54 | |
| amrith | https://launchpad.net/~amrith | 18:54 |
| amrith | amrith-rsa-key | 18:54 |
| amrith | please lmk the user name I should access | 18:55 |
| *** maeker_ has quit IRC | 18:55 | |
| *** jgriffith_AutoAw is now known as jgriffith | 18:55 | |
| *** maeker has joined #openstack-infra | 18:56 | |
| fungi | amrith: ssh jenkins@198.72.124.177 | 18:56 |
| *** sdake has joined #openstack-infra | 18:56 | |
| amrith | thank you I am on | 18:56 |
| fungi | have fun, and let me know when you're done with it so i can release the hold | 18:56 |
| *** gomarivera has quit IRC | 18:57 | |
| openstackgerrit | John L. Villalovos proposed openstack-infra/project-config: ironic: Have the grenade builder use devstack-virtual-ironic https://review.openstack.org/411956 | 18:58 |
| *** jerryz has joined #openstack-infra | 18:58 | |
| amrith | thanks jeremy | 18:58 |
| fungi | any time! | 18:58 |
| *** liusheng has quit IRC | 18:58 | |
| pabelanger | jeblair: when are you looking to restart nodepool things for 411360? | 18:58 |
| *** clif_h has quit IRC | 19:01 | |
| openstackgerrit | John L. Villalovos proposed openstack-infra/project-config: ironic: Have the grenade builder use devstack-virtual-ironic https://review.openstack.org/411956 | 19:03 |
| pabelanger | mordred: greghaynes: I do see qemu-img on fedora24 supports vhd, I'll dig more into ti | 19:03 |
| pabelanger | it* | 19:03 |
| openstackgerrit | Derek Schultz proposed openstack-infra/project-config: add new-project Picasso https://review.openstack.org/411940 | 19:04 |
| *** hurgleburgler has quit IRC | 19:04 | |
| AJaeger | EmilienM: tripleo-incubator and tripleo-image-elements have stable/icehouse branches. Are they active? | 19:05 |
| EmilienM | AJaeger: no they should EOL | 19:06 |
| EmilienM | AJaeger: I'll look at that when i'm back on keyboard | 19:07 |
| AJaeger | EmilienM: Ok, then I remove some trusty jobs from them... | 19:07 |
| EmilienM | go for it | 19:07 |
| openstackgerrit | John L. Villalovos proposed openstack-infra/project-config: ironic: Have the grenade builder use devstack-virtual-ironic https://review.openstack.org/411956 | 19:10 |
| *** bhunter71 has quit IRC | 19:10 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/nodepool: Expose the ability to build all infra diskimages https://review.openstack.org/411929 | 19:14 |
| openstackgerrit | Paul Belanger proposed openstack-infra/nodepool: WIP Show ubuntu-precise failure https://review.openstack.org/411962 | 19:14 |
| *** rkukura has joined #openstack-infra | 19:15 | |
| *** rkukura has left #openstack-infra | 19:15 | |
| *** baoli has joined #openstack-infra | 19:15 | |
| *** sdake has quit IRC | 19:16 | |
| *** sdake has joined #openstack-infra | 19:17 | |
| *** e0ne has quit IRC | 19:17 | |
| openstackgerrit | Merged openstack/python-jenkins: url-proof scripts passed to run_script https://review.openstack.org/407479 | 19:18 |
| *** thorst_ has joined #openstack-infra | 19:20 | |
| *** liusheng has joined #openstack-infra | 19:21 | |
| *** florianf has quit IRC | 19:21 | |
| *** yamahata has joined #openstack-infra | 19:22 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Trusty/py34 cleanup, part5 https://review.openstack.org/411964 | 19:24 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Move comment for z/tempest correctly https://review.openstack.org/411965 | 19:24 |
| AJaeger | team, do we still need " z/tempest" as project? It's really short now and it looks useless, see also ^ | 19:24 |
| *** dtardivel has quit IRC | 19:24 | |
| fungi | we at least used to rely on it to tie together different projects that used some related but not necessarily identical jobs into a common gate queue | 19:25 |
| AJaeger | right now all those jobs have been removed and we are only left with gate-tempest-dsvm-full-ubuntu-trusty and gate-tempest-dsvm-full-ubuntu-xenial in it. | 19:25 |
| AJaeger | so, that looks useless to me but I don't have that history... | 19:26 |
| *** rkukura has joined #openstack-infra | 19:26 | |
| *** thorst_ has quit IRC | 19:27 | |
| fungi | i agree it seems relatively pointless now | 19:27 |
| fungi | also though, we may need to pay closer attention to the one job pattern which is marked with queue-name: integrated | 19:27 |
| fungi | since it presently is restricted to just branch: stable/newton | 19:28 |
| AJaeger | oh ;( | 19:28 |
| fungi | we probably want to move that queue-name to a job which is run more commonly (like the successor to that one) | 19:29 |
| AJaeger | shared queue is now openstack-dev/devstack, openstack-dev/grenade, openstack-infra/devstack-gate, openstack-infra/pypi-mirror, openstack/cinder, openstack/devstack-plugin-ceph, openstack/glance, openstack/keystone, openstack/neutron, openstack/nova, openstack/python-swiftclient, openstack/requirements, openstack/swift, openstack/tempest, z/tempest | 19:29 |
| fungi | i guess that would be the -neutron version now? | 19:29 |
| *** vhosakot has quit IRC | 19:30 | |
| fungi | it's more that we want to make sure we don't dissolve the integrated gate queue when newton reaches eol | 19:30 |
| AJaeger | ;) | 19:31 |
| amrith | fungi, I'm logged out from 198.72.124.177, you can release it the hold. tests are still running on that machine now. thanks much | 19:31 |
| fungi | amrith: thanks for following up. did it yield any helpful details? | 19:32 |
| amrith | greghaynes, ping ... | 19:32 |
| AJaeger | fungi: I would take gate-tempest-dsvm-neutron-full-ubuntu-xenial - let me clean that up... | 19:32 |
| amrith | fungi, a couple of questions for greghaynes to begin with. and some changes I'm going to try | 19:32 |
| fungi | AJaeger: right, something that actually runs against master and is shared by those same projects (especially if we're removing the fake z/tempest one) | 19:32 |
| *** asettle has joined #openstack-infra | 19:33 | |
| jeblair | pabelanger: i can devote some time to it after lunch, in a couple of hours; if you want to do so sooner, feel free | 19:33 |
| jeblair | pabelanger: let's go ahead and check with -release | 19:33 |
| amrith | fungi, since greghaynes may not be here right now, let me pose the question here; maybe someone else can help. when you launch an instance for devstack-gate, I believe that you launch a vm using some prebuilt image. Is that image a raw image or a qcow2, and if it is a qcow2, what are the image creation options specified to qemu-img-create? | 19:34 |
| amrith | fungi, greghaynes specifically I'd like to know whether preallocation is specified or not. | 19:34 |
| amrith | fungi, greghaynes if it is a qcow2 that is | 19:34 |
| fungi | amrith: it can be either, depending on what each of our donated providers accepts (some are qemu, some raw). as to what image creation options are used, that i'm less sure about | 19:35 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Remove z/tempest https://review.openstack.org/411965 | 19:35 |
| clarkb | and some are vhd | 19:35 |
| amrith | clarkb, ok | 19:35 |
| clarkb | fungi: amrith the only option we pass is to force the qcow2 image version to aomething really old | 19:36 |
| fungi | we basically create one image and convert to all those formats, and then upload whichever format is needed | 19:36 |
| clarkb | otherwise some clouds cant boot them | 19:36 |
| *** asettle has quit IRC | 19:36 | |
| *** florianf has joined #openstack-infra | 19:36 | |
| *** asettle has joined #openstack-infra | 19:36 | |
| *** yamahata has quit IRC | 19:36 | |
| amrith | clarkb, fungi so we do use an old qcow2 with compat = 0.10? | 19:36 |
| AJaeger | project-config cores, I've now gone through the whole layout.yaml, removed unused trusty and python34 jobs and made py35 voting in case I removed py34 voting jobs. See https://review.openstack.org/#/q/status:open+project:openstack-infra/project-config+branch:master+topic:trusty-cleanup for the four open changes (including the z/tempest one). Please review to avoid merge conflicts and rebases | 19:37 |
| clarkb | amrith: somethibg like that yes | 19:37 |
| *** stevelle has left #openstack-infra | 19:37 | |
| *** electrofelix has quit IRC | 19:37 | |
| clarkb | I forget the exact compat version | 19:37 |
| *** yamahata has joined #openstack-infra | 19:37 | |
| *** sdake has quit IRC | 19:37 | |
| fungi | amrith: exactly that, in fact http://git.openstack.org/cgit/openstack-infra/project-config/tree/nodepool/nodepool.yaml#n988 | 19:37 |
| fungi | QEMU_IMG_OPTIONS: compat=0.10 | 19:37 |
| *** kjorgens_ has quit IRC | 19:37 | |
| fungi | yeah, some providers require very old qemu format | 19:38 |
| amrith | yikes, thx fungi. that could explain some of the slow performance I'm seeing in the gate. | 19:38 |
| clarkb | its pretty easy to do side by side comparison | 19:39 |
| fungi | amrith: have you correlated slowness to specific clouds/providers? we could probably cross-reference to which ones require what image formats | 19:39 |
| clarkb | conver from one to the other and do io thtouput tests on the same hypervisor | 19:39 |
| *** asettle has quit IRC | 19:39 | |
| jeblair | amrith: just to make sure i'm clear -- you are asking about the instances that we launch in real clouds (eg rackspace) upon which we run devstack-gate jobs. you are not asking about instances launched inside of the devstack-based clouds that are set up within *those* instances. is that all correct? | 19:40 |
| amrith | fungi, I have been tabulating them but given that the workloads are also very variable at different times of the day, there is a lot of noise. | 19:40 |
| amrith | clarkb, yes, I'm doing that locally on a quiet machine | 19:40 |
| amrith | want to compare in a less noisy environment | 19:40 |
| *** abompard has quit IRC | 19:41 | |
| fungi | yeah, in some cases we actually spike up to the point where our workload reduces the performance of the providers we're testing in | 19:41 |
| amrith | jeblair, I pushed up a change for the latter; I was inquiring about the former. | 19:41 |
| jeblair | pabelanger: -release folks say they're done with the milestone work, so we should be good to restart the builders any time. | 19:41 |
| *** rhallisey has quit IRC | 19:41 | |
| amrith | jeblair, you can see what I'm playing with in https://review.openstack.org/#/c/411848/ | 19:41 |
| jeblair | amrith: gotcha -- you were comparing what you're seeing in gate tests with what we see in "reality". :) | 19:42 |
| *** sdake has joined #openstack-infra | 19:42 | |
| amrith | jeblair, we run a CI in rax; we test on machines with 32gb RAM. we don't see swap, and our IO performance is better. When I run on my laptop with image type RAW I get much better IO performance than when I run with qcow2 and QEMU_IMG_OPTIONS pinned at 0.01 | 19:43 |
| amrith | our gate tests are setup in multiple streams with dependencies. as a result, the test framework often attempts to run multiple streams in parallel | 19:44 |
| amrith | we sometimes end up with a number of nested VM's | 19:44 |
| *** rbrndt has joined #openstack-infra | 19:44 | |
| clarkb | raw will be better than qcow | 19:44 |
| jeblair | amrith: ah. it's also worth noting that i believe it's possible that some clouds may transform images on the backend, so what we upload is not necessarily what they run | 19:44 |
| amrith | and when all of them are doing IO (to their cinder volumes) which are really part of the file system of the CI VM, the IO performance of QCOW2 is a bad sitation | 19:44 |
| clarkb | because ut doesnt have to uncompress blocks and write them back | 19:44 |
| clarkb | jeblair: ++ that too | 19:45 |
| amrith | and so as clarkb says, raw is much much better than qcow2 | 19:45 |
| fungi | on the other hand, uploading raw means much more time/bandwidth | 19:45 |
| amrith | but I hear that preallocation makes qcow2 very good | 19:45 |
| jeblair | (i recall that in infracloud, we had to stop it doing qcow->qcow translations :) | 19:45 |
| clarkb | jeblair: it was qcow to raw we stopped | 19:45 |
| clarkb | because it killed boots timing out on conversion | 19:45 |
| jeblair | ah, right | 19:45 |
| clarkb | and our trsting showed it didnt really slow us done | 19:45 |
| fungi | and yeah, some providers convert qcow2 to raw on the backend or somewhat inlined with the upload process, which introduces other fun delays | 19:46 |
| amrith | clarkb, oh? long boot time issues? | 19:46 |
| clarkb | possibly because we do unsafe writes in kvm | 19:46 |
| clarkb | amrith: yes because nova was converting qcow2 to raw before booting | 19:46 |
| clarkb | that took longer than novas timeout | 19:46 |
| jeblair | yeah, unsafe writes can be a big help here, but also battery-backed cache, etc... | 19:47 |
| amrith | jeblair, I hear ssd's are good too :) | 19:47 |
| pabelanger | jeblair: ack, ready when you are | 19:48 |
| amrith | apparently all the spinning rust ... | 19:48 |
| *** amoralej is now known as amoralej|off | 19:48 | |
| AJaeger | pabelanger, jeblair: If you're restarting builders, do you want to restart gerrit as well to get https://review.openstack.org/409344 - the reuqirements link change? Or should we wait for the usual restart because of java gc? ;) | 19:50 |
| *** HeOS has joined #openstack-infra | 19:50 | |
| *** jamesllondonlapt has quit IRC | 19:51 | |
| *** jamesdenton has quit IRC | 19:52 | |
| EmilienM | AJaeger: for some reasons, gate-tripleo-ci-centos-7-scenario001-multinode-puppet is still run in puppet-ceph stable/hammer, I might have missed something | 19:53 |
| *** john-davidge has quit IRC | 19:53 | |
| AJaeger | EmilienM: when did the change merge? | 19:53 |
| *** jamesdenton has joined #openstack-infra | 19:53 | |
| amrith | fungi, would you be able to get me access to 158.69.64.97 please. last one for today, I promise :) | 19:54 |
| AJaeger | EmilienM: and when was your job started? | 19:54 |
| EmilienM | AJaeger: 2h ago | 19:54 |
| EmilienM | AJaeger: now | 19:54 |
| *** florianf has quit IRC | 19:54 | |
| fungi | amrith: sure, working on it now | 19:54 |
| amrith | fungi, thx | 19:54 |
| AJaeger | EmilienM: which change? | 19:55 |
| EmilienM | AJaeger: but i looked at the layout and I see don't see the rule for stable/hammer I wonder if it's because of https://github.com/openstack-infra/project-config/blob/master/zuul/layout.yaml#L1590 | 19:55 |
| AJaeger | EmilienM: it should be fine, zuul was updated an hour ago | 19:55 |
| EmilienM | AJaeger: 410823 | 19:55 |
| EmilienM | AJaeger: mhh ok | 19:55 |
| *** clenimar has quit IRC | 19:55 | |
| EmilienM | AJaeger: I'm wondering if https://github.com/openstack-infra/project-config/blob/master/zuul/layout.yaml#L1590 could override https://github.com/openstack-infra/project-config/blob/master/zuul/layout.yaml#L1580 | 19:55 |
| AJaeger | EmilienM: your timing is fine I mean - let me double check the change | 19:55 |
| EmilienM | I never remember in which order zuul layout is grabbed | 19:56 |
| fungi | amrith: ssh jenkins@158.69.64.97 | 19:56 |
| amrith | thx fungi | 19:56 |
| fungi | you bet | 19:56 |
| *** claudiub|2 has quit IRC | 19:58 | |
| *** jamesdenton has quit IRC | 19:58 | |
| AJaeger | EmilienM: line 1590 comes later and thus overrides 1580. | 19:59 |
| AJaeger | EmilienM: check the output of the layout job and see what the regex is for it. | 19:59 |
| EmilienM | AJaeger: indeed, my bad. Let me submit a patch | 19:59 |
| * AJaeger just killed his X session ;( | 19:59 | |
| *** rhallisey has joined #openstack-infra | 19:59 | |
| AJaeger | bbl | 19:59 |
| EmilienM | AJaeger: changing order would help? | 20:00 |
| AJaeger | EmilienM: yes | 20:00 |
| EmilienM | (let's try) | 20:00 |
| EmilienM | ok | 20:00 |
| *** rbuzatu has quit IRC | 20:01 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: Fix zuul layout order for tripleo scenarios https://review.openstack.org/411976 | 20:02 |
| *** sdake has quit IRC | 20:02 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: Fix zuul layout order for tripleo scenarios https://review.openstack.org/411976 | 20:03 |
| *** sdake has joined #openstack-infra | 20:04 | |
| *** chem has quit IRC | 20:06 | |
| *** e0ne has joined #openstack-infra | 20:09 | |
| EmilienM | AJaeger: ok this time I see it http://logs.openstack.org/76/411976/2/check/gate-project-config-layout/f45d48e/console.html#_2016-12-16_20_10_18_966571 | 20:12 |
| *** sdague has quit IRC | 20:13 | |
| EmilienM | if someone can review https://review.openstack.org/411976 before the week-end :-) (I promess to stop after this one :)) | 20:13 |
| openstackgerrit | Joshua White proposed openstack-infra/devstack-gate: Enable Swift multinode grenade testing https://review.openstack.org/411980 | 20:13 |
| AJaeger | EmilienM: Is mistyping "promise" a way out? ;) | 20:14 |
| *** rfolco has quit IRC | 20:15 | |
| AJaeger | EmilienM: LGTM, +2 | 20:15 |
| *** ldnunes has quit IRC | 20:15 | |
| *** rfolco has joined #openstack-infra | 20:15 | |
| EmilienM | AJaeger: :D | 20:15 |
| EmilienM | AJaeger: you should be offline at this time! thanks for your help and enjoy the week-end | 20:15 |
| *** rfolco has quit IRC | 20:15 | |
| AJaeger | EmilienM: yeah, on my way to sign off ;) | 20:16 |
| *** sdague has joined #openstack-infra | 20:16 | |
| AJaeger | just one LARGE weekend wish - please review https://review.openstack.org/#/q/status:open+project:openstack-infra/project-config+branch:master+topic:trusty-cleanup and https://review.openstack.org/411757 | 20:17 |
| * AJaeger wishes everybody a great weekend | 20:17 | |
| fungi | have a good weekend, AJaeger! | 20:17 |
| amrith | AJaeger, have a good weekend! | 20:18 |
| *** esikachev has quit IRC | 20:18 | |
| AJaeger | thanks! | 20:18 |
| EmilienM | AJaeger: tchüss! | 20:18 |
| EmilienM | or tschüss even | 20:18 |
| pabelanger | clarkb: mind adding https://review.openstack.org/#/c/411929 to your review list. I'd like to use it to update the diskimage-builder dsvm nodepool job to build all the images we use in the gate. You can see an example of the failure we'd be able to catch for ubuntu-precise: | 20:19 |
| pabelanger | http://logs.openstack.org/62/411962/1/check/gate-dsvm-nodepool/48442be/logs/screen-nodepool-builder.txt.gz#_2016-12-16_19_34_35_698 | 20:19 |
| pabelanger | working on centos-7 dib example too | 20:19 |
| clarkb | pabelanger: fwiw it has never built any of the images we use in the gate | 20:19 |
| *** jgriffith is now known as jgriffith_AutoAw | 20:20 | |
| pabelanger | clarkb: ya, I'm also trying to fix that in there func tests too | 20:20 |
| pabelanger | clarkb: https://review.openstack.org/#/c/411910/ and https://review.openstack.org/#/c/411926 | 20:20 |
| clarkb | pabelanger: I don't think we have enough ram to do that | 20:20 |
| clarkb | thats 5GB of VM memory on a devstack host with 8GB ram will be close if it comes under | 20:21 |
| jeblair | pabelanger: i have some time now, how about i restart the builders? | 20:21 |
| pabelanger | clarkb: Ya, I haven't tested the launching of vms yet, was maybe thinking of grouping them by 2 or 3 | 20:21 |
| pabelanger | jeblair: yes | 20:21 |
| jeblair | pabelanger: are we currently in an ubuntu-precise build loop? | 20:22 |
| pabelanger | jeblair: yes, we can abort | 20:22 |
| pabelanger | it has been fixed but we need a dib release | 20:22 |
| jeblair | gotcha | 20:22 |
| *** maeker has quit IRC | 20:22 | |
| clarkb | pabelanger: its possible we can use smaller falvors too though I think ubuntu at least wants a bit of memory to boot into by default | 20:23 |
| clarkb | not sure about centos or fedora though | 20:23 |
| jeblair | i'll restart nb01 which is *not* builting it first (that way we don't have 2 aborts in a row) | 20:23 |
| *** asselin_ has joined #openstack-infra | 20:23 | |
| *** jgriffith_AutoAw is now known as jgriffith | 20:23 | |
| pabelanger | clarkb: agreed, let me work on that but wanted to get your feedback first before diving more into it | 20:23 |
| openstackgerrit | Joshua White proposed openstack-infra/project-config: Swift experimental multinode grenade job https://review.openstack.org/411982 | 20:24 |
| jeblair | oh, nb01 is uploading a bunch of xenial images right now | 20:24 |
| *** sdake has quit IRC | 20:24 | |
| clarkb | pabelanger: I'm ok with it as long as we don't try to build and boot all of the actual gate images. the minimal + devuser + simple init is plenty for this type of job I think | 20:24 |
| clarkb | (which is what current proposed diff does) | 20:24 |
| jeblair | i guess we'll test aborted uploads :) | 20:25 |
| pabelanger | clarkb: ack | 20:25 |
| clarkb | pabelanger: I think that adding too much stuff will make the job flaky just due to all the moving parts | 20:25 |
| clarkb | and the constrained size of the test env | 20:25 |
| *** rkukura has quit IRC | 20:25 | |
| *** asselin has quit IRC | 20:25 | |
| openstackgerrit | Merged openstack-infra/project-config: Fix zuul layout order for tripleo scenarios https://review.openstack.org/411976 | 20:26 |
| *** sdake has joined #openstack-infra | 20:26 | |
| pabelanger | clarkb: agreed, but I'd like to try, at least with diskimage-builder jobs. See how it goes, and if it is too painful scale it back | 20:26 |
| *** rbrndt has quit IRC | 20:27 | |
| *** maeker_ has joined #openstack-infra | 20:27 | |
| *** csomerville has quit IRC | 20:29 | |
| *** rkukura has joined #openstack-infra | 20:29 | |
| clarkb | pabelanger: I did leave one comment that should be addressed before we merge that | 20:29 |
| pabelanger | clarkb: ack, I can clean that up | 20:31 |
| jeblair | pabelanger: restarted. they seem to be working. they are both uploading and one of them is building trusty. so we'll have to wait a bit till we can see their idle cpu usage. | 20:31 |
| amrith | clarkb, fungi some good news. please take a look at https://etherpad.openstack.org/p/SO6BOreFLB | 20:31 |
| *** rbrndt has joined #openstack-infra | 20:32 | |
| amrith | instance launch time down from 300+s to 20s | 20:32 |
| *** maeker_ has quit IRC | 20:32 | |
| amrith | was NOT related to qcow2 and options there | 20:32 |
| clarkb | amrith: what changed? | 20:32 |
| amrith | turns out that the nova on the machines | 20:32 |
| amrith | was setup for virt type = qemu | 20:32 |
| amrith | before devstack ran | 20:32 |
| amrith | I changed it to kvm | 20:32 |
| amrith | added the changed lines to the etherpad | 20:33 |
| *** eharney has quit IRC | 20:33 | |
| *** rhallisey has quit IRC | 20:33 | |
| amrith | cpu_mode = host-passthrough, virt_type=kvm | 20:33 |
| fungi | i wonder if that causes nested virt for kvm to kick in on your test platform? | 20:33 |
| amrith | I don't know whether this is a good change or not | 20:33 |
| amrith | it makes nested virt much faster (from my experience on local machines) | 20:33 |
| *** jtomasek has quit IRC | 20:33 | |
| amrith | but it may not work on all clouds | 20:33 |
| amrith | We'll have to look at things like sys/module/kvm_intel/parameters/nested | 20:34 |
| fungi | yeah, we disable nested virt acceleration across the board for a couple of reasons. one is consistency, the other stability | 20:34 |
| amrith | which is Y on this mahcine | 20:34 |
| amrith | but I don't know what it means on the other clouds that infra uses | 20:34 |
| amrith | for trove, nested virt is an important thing | 20:34 |
| sdague | clarkb: can you take a look at this - https://review.openstack.org/#/c/409871/2 - d-g review | 20:34 |
| amrith | and qemu is certainly way slower | 20:34 |
| amrith | I'm not proposing what I found as a solution | 20:34 |
| amrith | but it is a solid data point indicating why our CI (tesora's) is different from the openstack one | 20:35 |
| clarkb | right we don't have nested virt everywhere | 20:35 |
| amrith | we use kvm | 20:35 |
| clarkb | and where we do have it it crashes | 20:35 |
| clarkb | amrith: but you run on rax you said? | 20:35 |
| amrith | clarkb, it is running now :( | 20:35 |
| amrith | sorry if I crash this instance | 20:35 |
| clarkb | there is no nested kvm on rax aiaui | 20:35 |
| amrith | yes clarkb in rax | 20:35 |
| clarkb | amrith: how? they run xen | 20:35 |
| fungi | amrith: agreed. it's pretty well established that nested virt acceleration has a significant positive performance impact if you're doing virtual machines inside virtual machines, as long as the hypervisor has available support | 20:35 |
| clarkb | my undersatnding was you could not nest kvm on top of xen | 20:35 |
| amrith | clarkb, we may have dedicated machines | 20:36 |
| amrith | I am not sure | 20:36 |
| fungi | amrith: rackspace public cloud, or something else in rackspace? | 20:36 |
| amrith | rax is the provider | 20:36 |
| amrith | I don't know what kind of machine we get | 20:36 |
| amrith | but I will find out | 20:36 |
| fungi | rackspace _public_cloud_ is xen, but if you have your own dedicated environment it could very well be kvm | 20:36 |
| amrith | fungi, we are doing vm's inside vm's | 20:36 |
| amrith | fungi, could be, I will find out | 20:36 |
| fungi | yes, i'm quite aware you're running test workloads in side virtual machines within other virtual machines ;) | 20:37 |
| amrith | multiple vm's, replication, clustering ... | 20:37 |
| clarkb | amrith: re fail rate with nested kvm its not consistent and likely depends on all sorts of factors like what cpu the hypervisor has, what version of kvm/kernel the hypervisor has, and what version of kernel the guest has | 20:37 |
| amrith | whole busloads of VM's at a time :) | 20:37 |
| fungi | just saying i'm not surprised turning on nested virt acceleration made that way faster | 20:37 |
| clarkb | its a mess. I did some research semi recently and amd cpus apparently are much much better than intel for nested virt stability | 20:38 |
| clarkb | it also messes with live migration (which would be a reasonable reason to turn it off in clouds even if it otherwise worked fine) | 20:38 |
| fungi | what's more fun is that in some of our providers you could get an instance on an intel-backed hypervisor with one request and the very next one gives you an amd-backed one | 20:38 |
| mordred | pabelanger: so - the biggest issue is that older qemu doesn't write things out in a way that rax will accept | 20:38 |
| clarkb | (the linux kernel turns it on by default on AMD but not on Intel) | 20:38 |
| clarkb | fungi: yup | 20:39 |
| mordred | pabelanger: I did a bunch of digging this morning - will braindump where I'm at on that in a little bit | 20:39 |
| fungi | amrith: http://docs.openstack.org/infra/manual/testing.html#known-differences-to-watch-out-for has a bit of info | 20:40 |
| pabelanger | clarkb: what about using the new openssh-server element in diskimage-builder? I can just add an elements-dep on it and ssh server will now be setup | 20:40 |
| pabelanger | mordred: ah, too bad | 20:41 |
| clarkb | pabelanger: that'll work too, then you just remove the apt-get stuff from the setup-nodepool element (leave the mkdir /etc/nodepool in there) | 20:41 |
| pabelanger | clarkb: yup | 20:41 |
| amrith | fungi, thx. issue we have now is that it is virtually impossible to get a change through check (or gate) because of the number of jobs and the high false-positive failure rate | 20:42 |
| amrith | we split jobs in half to reduce the time for each | 20:42 |
| amrith | but that has helped only a little | 20:42 |
| *** gouthamr has quit IRC | 20:44 | |
| amrith | fungi, I have logged out of 158.69.64.97 tests are still running (and will fail). many thanks | 20:44 |
| fungi | turning on nested virt unfortunately means jobs would still run slow at random depending on which provider allocated the test instance, and we'd probably observe a higher rate of other random (nested-virt-related) job failures/crashes where it was technically supported | 20:45 |
| openstackgerrit | Paul Belanger proposed openstack-infra/nodepool: Expose the ability to build all infra diskimages https://review.openstack.org/411929 | 20:45 |
| clarkb | and not be able to debug them | 20:45 |
| clarkb | thats the most frustrating part, they just disappear into the ether | 20:45 |
| *** rbrndt has quit IRC | 20:45 | |
| fungi | right, there's some panic recorded in the hypervisor's log, but we don't have access to the bottom-layer hypervisor | 20:46 |
| *** sdake has quit IRC | 20:46 | |
| *** csomerville has joined #openstack-infra | 20:46 | |
| pabelanger | fungi: clarkb: jeblair: any objections on asking somebody from the tripleo-release team to tag our precise fix for DIB? Or do we want to wait for ianw or greghaynes? | 20:47 |
| clarkb | I think we can wait for ianw or greghaynes | 20:47 |
| *** gouthamr has joined #openstack-infra | 20:48 | |
| fungi | pabelanger: if we run with stale precise images for a while i think that's fine. they're only used for a handful of infra tests at this point | 20:48 |
| *** chlong has quit IRC | 20:48 | |
| *** jcoufal has quit IRC | 20:48 | |
| pabelanger | k, let me pause then, so we don't loop | 20:49 |
| *** sdake has joined #openstack-infra | 20:49 | |
| * pabelanger really likes the pause flag :) | 20:49 | |
| amrith | fungi, clarkb bbiab | 20:50 |
| greghaynes | pabelanger: I can tag it, one sec | 20:51 |
| pabelanger | greghaynes: Ah, great! | 20:52 |
| greghaynes | its just the one trivial fix so hopefully it doesnt come to bite us that its a friday release | 20:52 |
| mordred | pabelanger: that patch to the devstack plugin looks like it's now always going to build 5 images instead of just one | 20:53 |
| mordred | pabelanger: OH! I see - it has pause values. nice | 20:54 |
| pabelanger | Ya, figured we could use that to control which images to build | 20:54 |
| *** gomarivera has joined #openstack-infra | 20:55 | |
| mordred | yah - so - before we land that, I'd like to add variables to shade's gate to pause more of them (we actually hard-gate on the nodepool job now, and I'm concernd adding image builds will break things) | 20:55 |
| mordred | one sec - patch coming | 20:55 |
| pabelanger | mordred: ya, I was planning on adding an experimental job to diskimage-builder, to toggle the different images | 20:56 |
| clarkb | mordred: its default to only building one in that change | 20:56 |
| clarkb | mordred: so you should be fine, I think we keep that default and force specific jobs to opt into more | 20:56 |
| pabelanger | still need to play with RAM settings | 20:56 |
| mordred | oh - ok. cool | 20:56 |
| mordred | yay! | 20:56 |
| *** mkoderer has quit IRC | 20:56 | |
| *** esberglu has quit IRC | 20:56 | |
| greghaynes | pabelanger: 1.25.1 pushed | 20:57 |
| *** esberglu has joined #openstack-infra | 20:57 | |
| *** eharney has joined #openstack-infra | 20:57 | |
| pabelanger | greghaynes: thank you | 20:58 |
| greghaynes | np | 20:58 |
| *** asettle has joined #openstack-infra | 20:59 | |
| *** gomarivera has quit IRC | 20:59 | |
| *** nadya has joined #openstack-infra | 20:59 | |
| *** asettle has quit IRC | 21:00 | |
| *** jcoufal has joined #openstack-infra | 21:01 | |
| *** jcoufal has quit IRC | 21:01 | |
| *** esberglu has quit IRC | 21:01 | |
| *** itisha has quit IRC | 21:02 | |
| *** rbrndt has joined #openstack-infra | 21:04 | |
| openstackgerrit | James E. Blair proposed openstack-infra/nodepool: Fix image delete exception logging https://review.openstack.org/411990 | 21:05 |
| *** askb has joined #openstack-infra | 21:05 | |
| jeblair | i'm instapproving that to the master branch ^ | 21:05 |
| jeblair | (i accidentally proposed it to v3 earlier, oops) | 21:05 |
| *** rbrndt has quit IRC | 21:05 | |
| pabelanger | ++ | 21:06 |
| *** esberglu has joined #openstack-infra | 21:06 | |
| mordred | jeblair: I +2'd it anyway | 21:07 |
| *** ihrachys has joined #openstack-infra | 21:07 | |
| *** rbuzatu has joined #openstack-infra | 21:08 | |
| *** dprince has quit IRC | 21:08 | |
| *** sdake has quit IRC | 21:09 | |
| *** sdake has joined #openstack-infra | 21:10 | |
| *** esberglu has quit IRC | 21:11 | |
| *** rbuzatu has quit IRC | 21:12 | |
| *** bhunter71 has joined #openstack-infra | 21:12 | |
| *** tlian has quit IRC | 21:13 | |
| *** denismakogon has joined #openstack-infra | 21:14 | |
| openstackgerrit | Tomasz Trębski proposed openstack-infra/project-config: java-mysql gate for monasca-api should be voting https://review.openstack.org/411993 | 21:16 |
| *** john-davidge has joined #openstack-infra | 21:17 | |
| *** csomerville has quit IRC | 21:18 | |
| *** e0ne has quit IRC | 21:18 | |
| *** esikachev has joined #openstack-infra | 21:18 | |
| *** nadya has quit IRC | 21:18 | |
| *** john-davidge has quit IRC | 21:19 | |
| *** rlandy has quit IRC | 21:21 | |
| *** esikachev has quit IRC | 21:23 | |
| *** ihrachys has quit IRC | 21:24 | |
| *** gouthamr has quit IRC | 21:25 | |
| *** gouthamr has joined #openstack-infra | 21:26 | |
| *** thorst_ has joined #openstack-infra | 21:26 | |
| *** gouthamr has quit IRC | 21:27 | |
| *** lseki has quit IRC | 21:30 | |
| fungi | need to pop out for sushi, but will be back in a while | 21:30 |
| *** sdake has quit IRC | 21:30 | |
| *** john-davidge has joined #openstack-infra | 21:30 | |
| *** rbrndt has joined #openstack-infra | 21:32 | |
| *** sdake has joined #openstack-infra | 21:32 | |
| openstackgerrit | Merged openstack-infra/nodepool: Fix image delete exception logging https://review.openstack.org/411990 | 21:33 |
| *** john-davidge has quit IRC | 21:34 | |
| *** Jeffrey4l has quit IRC | 21:35 | |
| pabelanger | fungi: yummy | 21:35 |
| *** thorst_ has quit IRC | 21:35 | |
| *** Jeffrey4l has joined #openstack-infra | 21:36 | |
| *** thorst_ has joined #openstack-infra | 21:36 | |
| *** tjones has joined #openstack-infra | 21:39 | |
| *** soliosg has quit IRC | 21:40 | |
| *** fguillot has quit IRC | 21:41 | |
| *** adrian_otto has joined #openstack-infra | 21:42 | |
| openstackgerrit | James E. Blair proposed openstack-infra/nodepool: Update documentation on diskimages/images https://review.openstack.org/411998 | 21:42 |
| openstackgerrit | James E. Blair proposed openstack-infra/nodepool: Remove script-dir https://review.openstack.org/411999 | 21:42 |
| jeblair | clarkb, mordred, pabelanger, Shrews: ^ starting some doc/cleanup stuff | 21:43 |
| *** rbrndt has quit IRC | 21:43 | |
| *** john-davidge has joined #openstack-infra | 21:44 | |
| *** tlian has joined #openstack-infra | 21:46 | |
| clarkb | we still use script dir for ready script | 21:48 |
| *** john-davidge has quit IRC | 21:48 | |
| *** esikachev has joined #openstack-infra | 21:49 | |
| *** adrian_otto has quit IRC | 21:49 | |
| pabelanger | isn't that /opt? | 21:52 |
| *** esikachev has quit IRC | 21:53 | |
| *** vhosakot has joined #openstack-infra | 21:54 | |
| jeblair | clarkb: there's a blurb in commit msg about that | 21:55 |
| jeblair | to be fair, i didn't say 'ready script' in the blurb. :) but the dib element to copy over the scripts for use in ready-script is what i was referring to. | 21:56 |
| *** eharney has quit IRC | 21:57 | |
| clarkb | ya it was used to tell it where to find the scripts to bake into the image iirc | 21:57 |
| clarkb | but I guess that can be set directly like you mention | 21:57 |
| clarkb | pabelanger: and then yes they end up in /opt/nodepool-scripts on our images | 21:58 |
| jeblair | http://git.openstack.org/cgit/openstack-infra/project-config/tree/nodepool/elements/nodepool-base/extra-data.d/99-copy-nodepool-scripts | 21:58 |
| *** kiran-r has joined #openstack-infra | 21:58 | |
| jeblair | clarkb: it defaults to the in-image copy of our project-config, so we can even do nothing | 21:58 |
| *** john-davidge has joined #openstack-infra | 21:59 | |
| jeblair | (which is one of my favorite things to do) | 21:59 |
| *** wolverineav has joined #openstack-infra | 22:00 | |
| *** esberglu has joined #openstack-infra | 22:01 | |
| *** john-davidge has quit IRC | 22:03 | |
| *** esikachev has joined #openstack-infra | 22:04 | |
| *** wolverineav has quit IRC | 22:04 | |
| *** trown is now known as trown|outtypewww | 22:05 | |
| *** esberglu has quit IRC | 22:05 | |
| *** kiran-r has quit IRC | 22:07 | |
| *** lindycoder has quit IRC | 22:07 | |
| *** esikachev has quit IRC | 22:09 | |
| *** jgriffith is now known as jgriffith_AutoAw | 22:11 | |
| *** wolverin_ has joined #openstack-infra | 22:11 | |
| *** wolveri__ has joined #openstack-infra | 22:11 | |
| *** adrian_otto has joined #openstack-infra | 22:12 | |
| *** jgriffith_AutoAw is now known as jgriffith | 22:12 | |
| *** wolverin_ has quit IRC | 22:15 | |
| *** wolveri__ has quit IRC | 22:15 | |
| *** adrianofr_ has quit IRC | 22:19 | |
| *** jeckersb is now known as jeckersb_gone | 22:19 | |
| pabelanger | greghaynes: jeblair: clarkb: ubuntu-precise DIB builds finished, uploading to clouds now | 22:21 |
| *** tphummel has quit IRC | 22:21 | |
| *** xarses_ has joined #openstack-infra | 22:22 | |
| *** rbrndt has joined #openstack-infra | 22:22 | |
| *** andreww has joined #openstack-infra | 22:23 | |
| *** xarses has quit IRC | 22:23 | |
| *** dave-mccowan has quit IRC | 22:25 | |
| *** tphummel has joined #openstack-infra | 22:25 | |
| *** xarses_ has quit IRC | 22:26 | |
| *** esberglu has joined #openstack-infra | 22:27 | |
| *** jkilpatr has quit IRC | 22:27 | |
| *** mrtenio has quit IRC | 22:28 | |
| *** ricardoas has quit IRC | 22:28 | |
| *** adrian_otto has quit IRC | 22:31 | |
| *** weshay is now known as weshay_lata | 22:32 | |
| *** jamesdenton has joined #openstack-infra | 22:34 | |
| *** esberglu has quit IRC | 22:34 | |
| *** toure has quit IRC | 22:34 | |
| *** esberglu has joined #openstack-infra | 22:34 | |
| *** toure has joined #openstack-infra | 22:36 | |
| *** jamesden_ has joined #openstack-infra | 22:37 | |
| *** jgriffith is now known as jgriffith_AutoAw | 22:38 | |
| *** jamesdenton has quit IRC | 22:38 | |
| *** esberglu has quit IRC | 22:39 | |
| *** mtanin___ has joined #openstack-infra | 22:40 | |
| *** flepied has quit IRC | 22:40 | |
| *** mtanino has quit IRC | 22:42 | |
| *** xarses_ has joined #openstack-infra | 22:48 | |
| *** jkilpatr has joined #openstack-infra | 22:50 | |
| *** wolverineav has joined #openstack-infra | 22:50 | |
| *** dave-mccowan has joined #openstack-infra | 22:51 | |
| *** andreww has quit IRC | 22:51 | |
| *** rbrndt has quit IRC | 22:53 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/nodepool: Expose the ability to build all infra diskimages https://review.openstack.org/411929 | 22:54 |
| *** wolverineav has quit IRC | 22:55 | |
| *** dave-mccowan has quit IRC | 22:56 | |
| *** gomarivera has joined #openstack-infra | 22:56 | |
| *** Swami has quit IRC | 22:57 | |
| *** wolverineav has joined #openstack-infra | 23:00 | |
| *** wolverin_ has joined #openstack-infra | 23:00 | |
| *** gomarivera has quit IRC | 23:01 | |
| *** mmandel has quit IRC | 23:02 | |
| *** hongbin has quit IRC | 23:02 | |
| *** mmandel has joined #openstack-infra | 23:02 | |
| *** wolverineav has quit IRC | 23:04 | |
| *** wolverin_ has quit IRC | 23:05 | |
| * fungi raises a glass to doing nothing on a friday night | 23:06 | |
| *** mmandel has quit IRC | 23:07 | |
| amrith | fungi, lucky you; I have the glass, I have the stuff in the glass, but still have to get the freaking CI to do my bidding | 23:08 |
| *** sdague has quit IRC | 23:11 | |
| *** esikachev has joined #openstack-infra | 23:11 | |
| *** adrian_otto has joined #openstack-infra | 23:12 | |
| fungi | well, was more in response to jeblair's favorite thing to do | 23:12 |
| *** maeker has joined #openstack-infra | 23:15 | |
| *** esikachev has quit IRC | 23:15 | |
| *** vhosakot has quit IRC | 23:16 | |
| *** abompard has joined #openstack-infra | 23:16 | |
| *** abompard has joined #openstack-infra | 23:16 | |
| *** wolverineav has joined #openstack-infra | 23:20 | |
| *** marst has quit IRC | 23:21 | |
| *** denismakogon has quit IRC | 23:22 | |
| *** wolverineav has quit IRC | 23:24 | |
| *** maeker has quit IRC | 23:26 | |
| *** maeker has joined #openstack-infra | 23:28 | |
| *** rkukura has quit IRC | 23:28 | |
| fungi | i only wish i was doing nothing, but instead i'm doing paperwork | 23:29 |
| *** wolverineav has joined #openstack-infra | 23:30 | |
| *** jgriffith_AutoAw is now known as jgriffith | 23:31 | |
| *** esikachev has joined #openstack-infra | 23:33 | |
| *** rkukura has joined #openstack-infra | 23:33 | |
| *** wolverineav has quit IRC | 23:34 | |
| *** baoli has quit IRC | 23:34 | |
| openstackgerrit | James E. Blair proposed openstack-infra/nodepool: Docs: add Zookeeper installation section https://review.openstack.org/412009 | 23:34 |
| jeblair | mordred: could i impose upon you to update the docs here: file:///home/corvus/git/openstack-infra/nodepool/doc/build/html/operation.html#metadata | 23:35 |
| jeblair | ga | 23:35 |
| jeblair | you get the idea :) | 23:35 |
| jeblair | mordred: http://docs.openstack.org/infra/nodepool/operation.html#metadata | 23:36 |
| *** maeker has quit IRC | 23:37 | |
| *** esikachev has quit IRC | 23:37 | |
| *** maeker has joined #openstack-infra | 23:40 | |
| *** armax has quit IRC | 23:42 | |
| *** dimtruck is now known as zz_dimtruck | 23:44 | |
| jeblair | mordred: oh, that patch hasn't landed yet; i'll leave a review comment | 23:49 |
| *** wolverineav has joined #openstack-infra | 23:49 | |
| *** tphummel has quit IRC | 23:50 | |
| *** wolverineav has quit IRC | 23:53 | |
| *** kiran-r has joined #openstack-infra | 23:54 | |
| *** maeker has quit IRC | 23:57 | |
| *** gomarivera has joined #openstack-infra | 23:57 | |
| *** rbuzatu has joined #openstack-infra | 23:59 | |
| *** rbuzatu has quit IRC | 23:59 | |
| *** wolverineav has joined #openstack-infra | 23:59 | |
| *** rbuzatu has joined #openstack-infra | 23:59 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!